2024-08-03T20:32:05.152 INFO:root:teuthology version: 0.0.1.dev386+g7d7f5fe 2024-08-03T20:32:05.174 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-08-03T20:32:05.266 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/vshankar-2024-08-02_18:35:56-fs-wip-vshankar-testing-20240802.091913-debug-testing-default-smithi/7833836 branch: wip-vshankar-testing-20240802.091913-debug description: fs/functional/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-4c-client conf/{client mds mgr mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} subvol_versions/create_subvol_version_v1 tasks/damage} email: vshankar@redhat.com first_in_suite: false job_id: '7833836' kernel: client: branch: testing flavor: default kdb: 1 sha1: ef231aaae597a12bd3e2479b633f31ed1fcc23a8 ktype: upstream last_in_suite: false log-rotate: ceph-mds: 10G ceph-osd: 10G machine_type: smithi mgrmodules: sequential: - print: Enabling mgr modules name: vshankar-2024-08-02_18:35:56-fs-wip-vshankar-testing-20240802.091913-debug-testing-default-smithi no_nested_subset: false openstack: - volumes: count: 4 size: 20 - machine: disk: 200 os_type: centos os_version: 9.stream overrides: admin_socket: branch: wip-vshankar-testing-20240802.091913-debug ceph: cluster-conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 rados mon op timeout: 900 rados osd op timeout: 900 mds: debug mds: 20 debug mds balancer: 20 debug ms: 2 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mon: mon op complaint time: 120 mon warn on pool no app: false osd: osd op complaint time: 180 conf: client: pid file: '' global: ms die on skipped message: false mgr: client mount timeout: 30 debug client: 20 debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon down mkfs grace: 300 osd: bdev async discard: true bdev enable discard: true bluestore allocator: bitmap bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug ms: 1 debug osd: 20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - FS_DEGRADED - FS_INLINE_DATA_DEPRECATED - FS_WITH_FAILED_MDS - MDS_ALL_DOWN - MDS_DAMAGE - MDS_DEGRADED - MDS_FAILED - MDS_INSUFFICIENT_STANDBY - MDS_UP_LESS_THAN_MAX - online, but wants - filesystem is online with fewer MDS than max_mds - POOL_APP_NOT_ENABLED - do not have an application enabled - overall HEALTH_ - Replacing daemon - deprecated feature inline_data - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding - PG_AVAILABILITY - PG_DEGRADED - Reduced data availability - Degraded data redundancy - bad backtrace - object missing on disk - error reading table object - error reading sessionmap - Error loading MDS rank - missing journal object - Error recovering journal - error decoding table object - failed to read JournalPointer - Corrupt directory entry - Corrupt fnode header - corrupt sessionmap header - Corrupt dentry - Scrub error on inode - Metadata damage detected - MDS_READ_ONLY - force file system read-only - with standby daemon mds - MDS abort because newly corrupt dentry sha1: 3792a0a4a40f645dee271188eab388b1c22c10f8 ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs install: ceph: flavor: default sha1: 3792a0a4a40f645dee271188eab388b1c22c10f8 kclient: syntax: v1 selinux: allowlist: - scontext=system_u:system_r:getty_t:s0 subvolume_version: 1 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-vshankar-testing-20240802.091913-debug sha1: 3792a0a4a40f645dee271188eab388b1c22c10f8 owner: scheduled_vshankar@teuthology priority: 120 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mgr.y - mds.a - mds.c - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - client.2 - - mon.b - mon.c - mgr.x - mds.b - mds.d - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - client.3 seed: 8569 sha1: 3792a0a4a40f645dee271188eab388b1c22c10f8 sleep_before_teardown: 0 subset: 921/1024 suite: fs suite_branch: wip-vshankar-testing-20240802.091913-debug suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_3792a0a4a40f645dee271188eab388b1c22c10f8/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: 3792a0a4a40f645dee271188eab388b1c22c10f8 syslog: ignorelist: - WARNING*.*check_session_state - WARNING*.*__ceph_remove_cap targets: smithi032.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBBEjgnfLSs9lqOXHOa/Qij0TO92ti1seO/TFIkruZ7njoFUZwK0ns3l3qN/K4jsYdwKxnrYeISPVKfBlHe+Tl4g= smithi118.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPifbzPloH1zU8jOISrQRk2LVFiPLW4lgOKvMElkNf3OB3J+1zb2PHttd0EMJuM3OtfV2Jt+UnFjAi4XIV4wTQ4= tasks: - install: extra_packages: deb: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror rpm: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror extra_system_packages: deb: - bison - flex - libelf-dev - libssl-dev - network-manager - iproute2 - util-linux - dump - indent - libaio-dev - libtool-bin - uuid-dev - xfslibs-dev - postgresql - postgresql-client - postgresql-common - postgresql-contrib rpm: - bison - flex - elfutils-libelf-devel - openssl-devel - NetworkManager - iproute - util-linux - libacl-devel - libaio-devel - libattr-devel - libtool - libuuid-devel - xfsdump - xfsprogs - xfsprogs-devel - libaio-devel - libtool - libuuid-devel - xfsprogs-devel - postgresql - postgresql-server - postgresql-contrib - ceph: create_rbd_pool: false - sequential: - mgrmodules - kclient: null - cephfs_test_runner: modules: - tasks.cephfs.test_damage teuthology: fragments_dropped: [] meta: {} postmerge: - local function is_kupstream() return yaml.ktype == 'upstream' end - local function is_kdistro() return yaml.ktype == 'distro' end - local function is_fuse() return false end - local function syntax_version() return yaml.overrides.kclient.syntax end teuthology_branch: main teuthology_sha1: 7d7f5fe79c7b6b5080402153d07fb1acc4ce00a2 timestamp: 2024-08-02_18:35:56 tube: smithi user: vshankar verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.4074525 2024-08-03T20:32:05.266 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_3792a0a4a40f645dee271188eab388b1c22c10f8/qa; will attempt to use it 2024-08-03T20:32:05.267 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_3792a0a4a40f645dee271188eab388b1c22c10f8/qa/tasks 2024-08-03T20:32:05.267 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-08-03T20:32:05.269 INFO:teuthology.task.internal:Checking packages... 2024-08-03T20:32:05.288 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '3792a0a4a40f645dee271188eab388b1c22c10f8' 2024-08-03T20:32:05.289 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-08-03T20:32:05.289 INFO:teuthology.packaging:ref: None 2024-08-03T20:32:05.289 INFO:teuthology.packaging:tag: None 2024-08-03T20:32:05.289 INFO:teuthology.packaging:branch: wip-vshankar-testing-20240802.091913-debug 2024-08-03T20:32:05.289 INFO:teuthology.packaging:sha1: 3792a0a4a40f645dee271188eab388b1c22c10f8 2024-08-03T20:32:05.289 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=wip-vshankar-testing-20240802.091913-debug 2024-08-03T20:32:05.457 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-5476.g3792a0a4 2024-08-03T20:32:05.459 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-08-03T20:32:05.465 INFO:teuthology.task.internal:no buildpackages task found 2024-08-03T20:32:05.466 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-08-03T20:32:05.473 INFO:teuthology.task.internal:Saving configuration 2024-08-03T20:32:05.489 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-08-03T20:32:05.495 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-08-03T20:32:05.520 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi032.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/vshankar-2024-08-02_18:35:56-fs-wip-vshankar-testing-20240802.091913-debug-testing-default-smithi/7833836', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-08-03 20:26:24.007505', 'locked_by': 'scheduled_vshankar@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBBEjgnfLSs9lqOXHOa/Qij0TO92ti1seO/TFIkruZ7njoFUZwK0ns3l3qN/K4jsYdwKxnrYeISPVKfBlHe+Tl4g='} 2024-08-03T20:32:05.542 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi118.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/vshankar-2024-08-02_18:35:56-fs-wip-vshankar-testing-20240802.091913-debug-testing-default-smithi/7833836', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-08-03 20:26:24.005403', 'locked_by': 'scheduled_vshankar@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPifbzPloH1zU8jOISrQRk2LVFiPLW4lgOKvMElkNf3OB3J+1zb2PHttd0EMJuM3OtfV2Jt+UnFjAi4XIV4wTQ4='} 2024-08-03T20:32:05.542 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-08-03T20:32:05.548 INFO:teuthology.task.internal:roles: ubuntu@smithi032.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'client.2'] 2024-08-03T20:32:05.548 INFO:teuthology.task.internal:roles: ubuntu@smithi118.front.sepia.ceph.com - ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'client.3'] 2024-08-03T20:32:05.548 INFO:teuthology.run_tasks:Running task console_log... 2024-08-03T20:32:05.604 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f73286e7ac0>, signals=[15]) 2024-08-03T20:32:05.604 INFO:teuthology.run_tasks:Running task internal.connect... 2024-08-03T20:32:05.610 INFO:teuthology.task.internal:Opening connections... 2024-08-03T20:32:05.610 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi032.front.sepia.ceph.com 2024-08-03T20:32:05.611 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi032.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-08-03T20:32:05.686 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi118.front.sepia.ceph.com 2024-08-03T20:32:05.687 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi118.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-08-03T20:32:05.757 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-08-03T20:32:05.764 DEBUG:teuthology.orchestra.run.smithi032:> uname -m 2024-08-03T20:32:05.783 INFO:teuthology.orchestra.run.smithi032.stdout:x86_64 2024-08-03T20:32:05.783 DEBUG:teuthology.orchestra.run.smithi032:> cat /etc/os-release 2024-08-03T20:32:05.841 INFO:teuthology.orchestra.run.smithi032.stdout:NAME="CentOS Stream" 2024-08-03T20:32:05.841 INFO:teuthology.orchestra.run.smithi032.stdout:VERSION="9" 2024-08-03T20:32:05.841 INFO:teuthology.orchestra.run.smithi032.stdout:ID="centos" 2024-08-03T20:32:05.841 INFO:teuthology.orchestra.run.smithi032.stdout:ID_LIKE="rhel fedora" 2024-08-03T20:32:05.841 INFO:teuthology.orchestra.run.smithi032.stdout:VERSION_ID="9" 2024-08-03T20:32:05.841 INFO:teuthology.orchestra.run.smithi032.stdout:PLATFORM_ID="platform:el9" 2024-08-03T20:32:05.842 INFO:teuthology.orchestra.run.smithi032.stdout:PRETTY_NAME="CentOS Stream 9" 2024-08-03T20:32:05.842 INFO:teuthology.orchestra.run.smithi032.stdout:ANSI_COLOR="0;31" 2024-08-03T20:32:05.842 INFO:teuthology.orchestra.run.smithi032.stdout:LOGO="fedora-logo-icon" 2024-08-03T20:32:05.842 INFO:teuthology.orchestra.run.smithi032.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-08-03T20:32:05.842 INFO:teuthology.orchestra.run.smithi032.stdout:HOME_URL="https://centos.org/" 2024-08-03T20:32:05.842 INFO:teuthology.orchestra.run.smithi032.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-08-03T20:32:05.842 INFO:teuthology.orchestra.run.smithi032.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-08-03T20:32:05.842 INFO:teuthology.orchestra.run.smithi032.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-08-03T20:32:05.843 INFO:teuthology.lock.ops:Updating smithi032.front.sepia.ceph.com on lock server 2024-08-03T20:32:05.862 DEBUG:teuthology.orchestra.run.smithi118:> uname -m 2024-08-03T20:32:05.879 INFO:teuthology.orchestra.run.smithi118.stdout:x86_64 2024-08-03T20:32:05.879 DEBUG:teuthology.orchestra.run.smithi118:> cat /etc/os-release 2024-08-03T20:32:05.935 INFO:teuthology.orchestra.run.smithi118.stdout:NAME="CentOS Stream" 2024-08-03T20:32:05.935 INFO:teuthology.orchestra.run.smithi118.stdout:VERSION="9" 2024-08-03T20:32:05.935 INFO:teuthology.orchestra.run.smithi118.stdout:ID="centos" 2024-08-03T20:32:05.935 INFO:teuthology.orchestra.run.smithi118.stdout:ID_LIKE="rhel fedora" 2024-08-03T20:32:05.935 INFO:teuthology.orchestra.run.smithi118.stdout:VERSION_ID="9" 2024-08-03T20:32:05.935 INFO:teuthology.orchestra.run.smithi118.stdout:PLATFORM_ID="platform:el9" 2024-08-03T20:32:05.935 INFO:teuthology.orchestra.run.smithi118.stdout:PRETTY_NAME="CentOS Stream 9" 2024-08-03T20:32:05.936 INFO:teuthology.orchestra.run.smithi118.stdout:ANSI_COLOR="0;31" 2024-08-03T20:32:05.936 INFO:teuthology.orchestra.run.smithi118.stdout:LOGO="fedora-logo-icon" 2024-08-03T20:32:05.936 INFO:teuthology.orchestra.run.smithi118.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-08-03T20:32:05.936 INFO:teuthology.orchestra.run.smithi118.stdout:HOME_URL="https://centos.org/" 2024-08-03T20:32:05.936 INFO:teuthology.orchestra.run.smithi118.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-08-03T20:32:05.936 INFO:teuthology.orchestra.run.smithi118.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-08-03T20:32:05.936 INFO:teuthology.orchestra.run.smithi118.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-08-03T20:32:05.936 INFO:teuthology.lock.ops:Updating smithi118.front.sepia.ceph.com on lock server 2024-08-03T20:32:05.953 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-08-03T20:32:05.963 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-08-03T20:32:05.971 INFO:teuthology.task.internal:Checking for old test directory... 2024-08-03T20:32:05.971 DEBUG:teuthology.orchestra.run.smithi032:> test '!' -e /home/ubuntu/cephtest 2024-08-03T20:32:05.973 DEBUG:teuthology.orchestra.run.smithi118:> test '!' -e /home/ubuntu/cephtest 2024-08-03T20:32:05.991 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-08-03T20:32:05.998 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-08-03T20:32:05.998 DEBUG:teuthology.orchestra.run.smithi032:> test -z $(ls -A /var/lib/ceph) 2024-08-03T20:32:06.031 DEBUG:teuthology.orchestra.run.smithi118:> test -z $(ls -A /var/lib/ceph) 2024-08-03T20:32:06.057 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-08-03T20:32:06.110 INFO:teuthology.run_tasks:Running task kernel... 2024-08-03T20:32:06.125 INFO:teuthology.task.kernel:normalize config orig: {'client': {'branch': 'testing'}, 'flavor': 'default', 'kdb': 1, 'sha1': 'ef231aaae597a12bd3e2479b633f31ed1fcc23a8'} 2024-08-03T20:32:06.125 INFO:teuthology.task.kernel:normalize config final: {'client.0': {'branch': 'testing'}, 'client.2': {'branch': 'testing'}, 'client.1': {'branch': 'testing'}, 'client.3': {'branch': 'testing'}} 2024-08-03T20:32:06.125 INFO:teuthology.task.kernel:config {'client.0': {'branch': 'testing'}, 'client.1': {'branch': 'testing'}}, timeout 300 2024-08-03T20:32:06.125 DEBUG:teuthology.orchestra.run.smithi032:> test -f /run/.containerenv -o -f /.dockerenv 2024-08-03T20:32:06.126 DEBUG:teuthology.orchestra.run.smithi118:> test -f /run/.containerenv -o -f /.dockerenv 2024-08-03T20:32:06.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-03T20:32:06.142 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=centos%2F9%2Fx86_64&ref=testing 2024-08-03T20:32:06.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-03T20:32:06.147 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=centos%2F9%2Fx86_64&ref=testing 2024-08-03T20:32:06.314 DEBUG:teuthology.task.kernel:sha1 for client.1 is ef231aaae597a12bd3e2479b633f31ed1fcc23a8 2024-08-03T20:32:06.314 INFO:teuthology.task.kernel:Checking kernel version of client.1, want "ef231aaae597a12bd3e2479b633f31ed1fcc23a8"... 2024-08-03T20:32:06.315 DEBUG:teuthology.orchestra.run.smithi118:> uname -r 2024-08-03T20:32:06.332 INFO:teuthology.orchestra.run.smithi118.stdout:5.14.0-437.el9.x86_64 2024-08-03T20:32:06.332 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-437.el9.x86_64 vs ef231aaae597a12bd3e2479b633f31ed1fcc23a8 2024-08-03T20:32:06.333 DEBUG:teuthology.task.kernel:failed to parse current kernel version 2024-08-03T20:32:06.333 DEBUG:teuthology.orchestra.run.smithi118:> sudo yum upgrade -y linux-firmware 2024-08-03T20:32:06.338 DEBUG:teuthology.task.kernel:sha1 for client.0 is ef231aaae597a12bd3e2479b633f31ed1fcc23a8 2024-08-03T20:32:06.339 INFO:teuthology.task.kernel:Checking kernel version of client.0, want "ef231aaae597a12bd3e2479b633f31ed1fcc23a8"... 2024-08-03T20:32:06.339 DEBUG:teuthology.orchestra.run.smithi032:> uname -r 2024-08-03T20:32:06.356 INFO:teuthology.orchestra.run.smithi032.stdout:5.14.0-437.el9.x86_64 2024-08-03T20:32:06.357 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-437.el9.x86_64 vs ef231aaae597a12bd3e2479b633f31ed1fcc23a8 2024-08-03T20:32:06.357 DEBUG:teuthology.task.kernel:failed to parse current kernel version 2024-08-03T20:32:06.357 DEBUG:teuthology.orchestra.run.smithi032:> sudo yum upgrade -y linux-firmware 2024-08-03T20:32:09.929 INFO:teuthology.orchestra.run.smithi118.stdout:CentOS Stream 9 - BaseOS 4.3 MB/s | 8.2 MB 00:01 2024-08-03T20:32:10.604 INFO:teuthology.orchestra.run.smithi032.stdout:CentOS Stream 9 - BaseOS 3.0 MB/s | 8.2 MB 00:02 2024-08-03T20:32:14.797 INFO:teuthology.orchestra.run.smithi032.stdout:CentOS Stream 9 - AppStream 6.5 MB/s | 20 MB 00:03 2024-08-03T20:32:20.280 INFO:teuthology.orchestra.run.smithi032.stdout:CentOS Stream 9 - CRB 4.2 MB/s | 6.6 MB 00:01 2024-08-03T20:32:22.491 INFO:teuthology.orchestra.run.smithi032.stdout:CentOS Stream 9 - Extras packages 20 kB/s | 18 kB 00:00 2024-08-03T20:32:23.494 INFO:teuthology.orchestra.run.smithi032.stdout:Extra Packages for Enterprise Linux 26 MB/s | 22 MB 00:00 2024-08-03T20:32:28.002 INFO:teuthology.orchestra.run.smithi118.stdout:CentOS Stream 9 - AppStream 1.2 MB/s | 20 MB 00:16 2024-08-03T20:32:29.016 INFO:teuthology.orchestra.run.smithi032.stdout:lab-extras 26 kB/s | 1.7 kB 00:00 2024-08-03T20:32:31.196 INFO:teuthology.orchestra.run.smithi032.stdout:Dependencies resolved. 2024-08-03T20:32:31.204 INFO:teuthology.orchestra.run.smithi032.stdout:================================================================================ 2024-08-03T20:32:31.204 INFO:teuthology.orchestra.run.smithi032.stdout: Package Architecture Version Repository Size 2024-08-03T20:32:31.204 INFO:teuthology.orchestra.run.smithi032.stdout:================================================================================ 2024-08-03T20:32:31.204 INFO:teuthology.orchestra.run.smithi032.stdout:Upgrading: 2024-08-03T20:32:31.204 INFO:teuthology.orchestra.run.smithi032.stdout: linux-firmware noarch 20240716-145.el9 baseos 383 M 2024-08-03T20:32:31.204 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-03T20:32:31.204 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction Summary 2024-08-03T20:32:31.204 INFO:teuthology.orchestra.run.smithi032.stdout:================================================================================ 2024-08-03T20:32:31.205 INFO:teuthology.orchestra.run.smithi032.stdout:Upgrade 1 Package 2024-08-03T20:32:31.205 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-03T20:32:31.205 INFO:teuthology.orchestra.run.smithi032.stdout:Total download size: 383 M 2024-08-03T20:32:31.205 INFO:teuthology.orchestra.run.smithi032.stdout:Downloading Packages: 2024-08-03T20:32:33.561 INFO:teuthology.orchestra.run.smithi118.stdout:CentOS Stream 9 - CRB 4.4 MB/s | 6.6 MB 00:01 2024-08-03T20:32:35.953 INFO:teuthology.orchestra.run.smithi118.stdout:CentOS Stream 9 - Extras packages 17 kB/s | 18 kB 00:01 2024-08-03T20:32:36.907 INFO:teuthology.orchestra.run.smithi118.stdout:Extra Packages for Enterprise Linux 27 MB/s | 22 MB 00:00 2024-08-03T20:32:42.479 INFO:teuthology.orchestra.run.smithi118.stdout:lab-extras 45 kB/s | 1.7 kB 00:00 2024-08-03T20:32:44.529 INFO:teuthology.orchestra.run.smithi118.stdout:Dependencies resolved. 2024-08-03T20:32:44.535 INFO:teuthology.orchestra.run.smithi118.stdout:================================================================================ 2024-08-03T20:32:44.535 INFO:teuthology.orchestra.run.smithi118.stdout: Package Architecture Version Repository Size 2024-08-03T20:32:44.535 INFO:teuthology.orchestra.run.smithi118.stdout:================================================================================ 2024-08-03T20:32:44.536 INFO:teuthology.orchestra.run.smithi118.stdout:Upgrading: 2024-08-03T20:32:44.536 INFO:teuthology.orchestra.run.smithi118.stdout: linux-firmware noarch 20240716-145.el9 baseos 383 M 2024-08-03T20:32:44.536 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-08-03T20:32:44.536 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction Summary 2024-08-03T20:32:44.536 INFO:teuthology.orchestra.run.smithi118.stdout:================================================================================ 2024-08-03T20:32:44.536 INFO:teuthology.orchestra.run.smithi118.stdout:Upgrade 1 Package 2024-08-03T20:32:44.536 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-08-03T20:32:44.536 INFO:teuthology.orchestra.run.smithi118.stdout:Total download size: 383 M 2024-08-03T20:32:44.537 INFO:teuthology.orchestra.run.smithi118.stdout:Downloading Packages: 2024-08-03T20:33:09.382 INFO:teuthology.orchestra.run.smithi032.stdout:linux-firmware-20240716-145.el9.noarch.rpm 10 MB/s | 383 MB 00:37 2024-08-03T20:33:09.382 INFO:teuthology.orchestra.run.smithi032.stdout:-------------------------------------------------------------------------------- 2024-08-03T20:33:09.383 INFO:teuthology.orchestra.run.smithi032.stdout:Total 10 MB/s | 383 MB 00:38 2024-08-03T20:33:12.609 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction check 2024-08-03T20:33:12.710 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction check succeeded. 2024-08-03T20:33:12.710 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction test 2024-08-03T20:33:16.128 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction test succeeded. 2024-08-03T20:33:16.128 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction 2024-08-03T20:33:25.442 INFO:teuthology.orchestra.run.smithi032.stdout: Preparing : 1/1 2024-08-03T20:33:26.157 INFO:teuthology.orchestra.run.smithi032.stdout: Upgrading : linux-firmware-20240716-145.el9.noarch 1/2 2024-08-03T20:33:26.247 INFO:teuthology.orchestra.run.smithi032.stdout: Cleanup : linux-firmware-20240219-143.el9.noarch 2/2 2024-08-03T20:33:26.767 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: linux-firmware-20240219-143.el9.noarch 2/2 2024-08-03T20:33:26.767 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : linux-firmware-20240716-145.el9.noarch 1/2 2024-08-03T20:33:27.543 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : linux-firmware-20240219-143.el9.noarch 2/2 2024-08-03T20:33:27.544 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-03T20:33:27.544 INFO:teuthology.orchestra.run.smithi032.stdout:Upgraded: 2024-08-03T20:33:27.544 INFO:teuthology.orchestra.run.smithi032.stdout: linux-firmware-20240716-145.el9.noarch 2024-08-03T20:33:27.544 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-03T20:33:27.544 INFO:teuthology.orchestra.run.smithi032.stdout:Complete! 2024-08-03T20:33:27.878 INFO:teuthology.task.kernel:Downloading kernel ef231aaae597a12bd3e2479b633f31ed1fcc23a8 on client.0... 2024-08-03T20:33:27.879 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=centos%2F9%2Fx86_64&sha1=ef231aaae597a12bd3e2479b633f31ed1fcc23a8 2024-08-03T20:33:28.034 INFO:teuthology.task.kernel:fetching, builder baseurl is https://4.chacra.ceph.com/r/kernel/testing/ef231aaae597a12bd3e2479b633f31ed1fcc23a8/centos/9/flavors/default/x86_64/ 2024-08-03T20:33:28.034 DEBUG:teuthology.orchestra.run.smithi032:> rm -f /tmp/kernel.x86_64.rpm && echo kernel-6.11.0_rc1_gef231aaae597-1.x86_64.rpm | wget -nv -O /tmp/kernel.x86_64.rpm --base=https://4.chacra.ceph.com/r/kernel/testing/ef231aaae597a12bd3e2479b633f31ed1fcc23a8/centos/9/flavors/default/x86_64/ --input-file=- 2024-08-03T20:33:28.037 DEBUG:teuthology.task.kernel:Waiting for download/copy to ubuntu@smithi032.front.sepia.ceph.com to complete... 2024-08-03T20:33:34.326 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03 20:33:34 URL:https://4.chacra.ceph.com/r/kernel/testing/ef231aaae597a12bd3e2479b633f31ed1fcc23a8/centos/9/flavors/default/x86_64/kernel-6.11.0_rc1_gef231aaae597-1.x86_64.rpm [359268674/359268674] -> "/tmp/kernel.x86_64.rpm" [1] 2024-08-03T20:33:34.326 INFO:teuthology.orchestra.run.smithi032.stderr:FINISHED --2024-08-03 20:33:34-- 2024-08-03T20:33:34.326 INFO:teuthology.orchestra.run.smithi032.stderr:Total wall clock time: 6.3s 2024-08-03T20:33:34.326 INFO:teuthology.orchestra.run.smithi032.stderr:Downloaded: 1 files, 343M in 6.0s (57.2 MB/s) 2024-08-03T20:33:34.330 INFO:teuthology.task.kernel:Installing kernel ef231aaae597a12bd3e2479b633f31ed1fcc23a8 on client.0... 2024-08-03T20:33:34.330 DEBUG:teuthology.orchestra.run.smithi032:> sudo rpm -ivh --oldpackage --replacefiles --replacepkgs /tmp/kernel.x86_64.rpm 2024-08-03T20:33:36.491 INFO:teuthology.orchestra.run.smithi032.stdout:Verifying... ######################################## 2024-08-03T20:33:36.541 INFO:teuthology.orchestra.run.smithi032.stdout:Preparing... ######################################## 2024-08-03T20:33:36.595 INFO:teuthology.orchestra.run.smithi032.stdout:Updating / installing... 2024-08-03T20:33:44.956 INFO:teuthology.orchestra.run.smithi032.stdout:kernel-6.11.0_rc1_gef231aaae597-1 ######################################## 2024-08-03T20:34:25.562 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi032.front.sepia.ceph.com, path=/tmp/kernel.x86_64.rpm, version=None) 2024-08-03T20:34:25.562 DEBUG:teuthology.orchestra.run.smithi032:> rpm -qlp /tmp/kernel.x86_64.rpm 2024-08-03T20:34:25.633 INFO:teuthology.orchestra.run.smithi032.stdout:/boot/System.map-6.11.0-rc1-gef231aaae597 2024-08-03T20:34:25.633 INFO:teuthology.orchestra.run.smithi032.stdout:/boot/config-6.11.0-rc1-gef231aaae597 2024-08-03T20:34:25.633 INFO:teuthology.orchestra.run.smithi032.stdout:/boot/vmlinuz-6.11.0-rc1-gef231aaae597 2024-08-03T20:34:25.633 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597 2024-08-03T20:34:25.633 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/System.map 2024-08-03T20:34:25.633 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/config 2024-08-03T20:34:25.633 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel 2024-08-03T20:34:25.633 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/arch 2024-08-03T20:34:25.633 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/arch/x86 2024-08-03T20:34:25.633 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/arch/x86/crypto 2024-08-03T20:34:25.633 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/arch/x86/crypto/blowfish-x86_64.ko 2024-08-03T20:34:25.634 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/arch/x86/crypto/camellia-aesni-avx-x86_64.ko 2024-08-03T20:34:25.634 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/arch/x86/crypto/camellia-aesni-avx2.ko 2024-08-03T20:34:25.634 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/arch/x86/crypto/camellia-x86_64.ko 2024-08-03T20:34:25.634 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/arch/x86/crypto/cast5-avx-x86_64.ko 2024-08-03T20:34:25.634 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/arch/x86/crypto/cast6-avx-x86_64.ko 2024-08-03T20:34:25.634 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/arch/x86/crypto/crc32-pclmul.ko 2024-08-03T20:34:25.634 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/arch/x86/crypto/crc32c-intel.ko 2024-08-03T20:34:25.634 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/arch/x86/crypto/crct10dif-pclmul.ko 2024-08-03T20:34:25.634 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/arch/x86/crypto/des3_ede-x86_64.ko 2024-08-03T20:34:25.634 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/arch/x86/crypto/ghash-clmulni-intel.ko 2024-08-03T20:34:25.634 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/arch/x86/crypto/serpent-avx-x86_64.ko 2024-08-03T20:34:25.634 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/arch/x86/crypto/serpent-avx2.ko 2024-08-03T20:34:25.634 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/arch/x86/crypto/serpent-sse2-x86_64.ko 2024-08-03T20:34:25.635 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/arch/x86/crypto/sha1-ssse3.ko 2024-08-03T20:34:25.635 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/arch/x86/crypto/sha256-ssse3.ko 2024-08-03T20:34:25.635 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/arch/x86/crypto/sha512-ssse3.ko 2024-08-03T20:34:25.635 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/arch/x86/crypto/twofish-avx-x86_64.ko 2024-08-03T20:34:25.635 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/arch/x86/crypto/twofish-x86_64-3way.ko 2024-08-03T20:34:25.635 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/arch/x86/crypto/twofish-x86_64.ko 2024-08-03T20:34:25.635 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/arch/x86/kernel 2024-08-03T20:34:25.635 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/arch/x86/kernel/cpu 2024-08-03T20:34:25.635 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/arch/x86/kernel/cpu/mce 2024-08-03T20:34:25.635 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/arch/x86/kernel/cpu/mce/mce-inject.ko 2024-08-03T20:34:25.635 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/arch/x86/kvm 2024-08-03T20:34:25.635 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/arch/x86/kvm/kvm-amd.ko 2024-08-03T20:34:25.635 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/arch/x86/kvm/kvm-intel.ko 2024-08-03T20:34:25.636 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/arch/x86/kvm/kvm.ko 2024-08-03T20:34:25.636 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/crypto 2024-08-03T20:34:25.636 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/crypto/ansi_cprng.ko 2024-08-03T20:34:25.636 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/crypto/anubis.ko 2024-08-03T20:34:25.636 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/crypto/arc4.ko 2024-08-03T20:34:25.636 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/crypto/async_tx 2024-08-03T20:34:25.636 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/crypto/async_tx/async_memcpy.ko 2024-08-03T20:34:25.636 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/crypto/async_tx/async_pq.ko 2024-08-03T20:34:25.636 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/crypto/async_tx/async_raid6_recov.ko 2024-08-03T20:34:25.636 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/crypto/async_tx/async_tx.ko 2024-08-03T20:34:25.636 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/crypto/async_tx/async_xor.ko 2024-08-03T20:34:25.636 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/crypto/async_tx/raid6test.ko 2024-08-03T20:34:25.636 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/crypto/authenc.ko 2024-08-03T20:34:25.637 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/crypto/authencesn.ko 2024-08-03T20:34:25.637 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/crypto/blake2b_generic.ko 2024-08-03T20:34:25.637 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/crypto/blowfish_common.ko 2024-08-03T20:34:25.637 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/crypto/blowfish_generic.ko 2024-08-03T20:34:25.637 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/crypto/camellia_generic.ko 2024-08-03T20:34:25.637 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/crypto/cast5_generic.ko 2024-08-03T20:34:25.637 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/crypto/cast6_generic.ko 2024-08-03T20:34:25.637 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/crypto/cast_common.ko 2024-08-03T20:34:25.637 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/crypto/ccm.ko 2024-08-03T20:34:25.637 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/crypto/cmac.ko 2024-08-03T20:34:25.637 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/crypto/crc32_generic.ko 2024-08-03T20:34:25.637 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/crypto/crypto_engine.ko 2024-08-03T20:34:25.637 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/crypto/crypto_user.ko 2024-08-03T20:34:25.637 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/crypto/cts.ko 2024-08-03T20:34:25.638 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/crypto/des_generic.ko 2024-08-03T20:34:25.638 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/crypto/dh_generic.ko 2024-08-03T20:34:25.638 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/crypto/ecc.ko 2024-08-03T20:34:25.638 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/crypto/ecdh_generic.ko 2024-08-03T20:34:25.638 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/crypto/echainiv.ko 2024-08-03T20:34:25.638 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/crypto/essiv.ko 2024-08-03T20:34:25.638 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/crypto/fcrypt.ko 2024-08-03T20:34:25.638 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/crypto/khazad.ko 2024-08-03T20:34:25.638 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/crypto/lz4.ko 2024-08-03T20:34:25.638 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/crypto/lz4hc.ko 2024-08-03T20:34:25.638 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/crypto/md4.ko 2024-08-03T20:34:25.638 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/crypto/michael_mic.ko 2024-08-03T20:34:25.638 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/crypto/pcbc.ko 2024-08-03T20:34:25.638 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/crypto/pcrypt.ko 2024-08-03T20:34:25.639 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/crypto/rmd160.ko 2024-08-03T20:34:25.639 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/crypto/seed.ko 2024-08-03T20:34:25.639 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/crypto/serpent_generic.ko 2024-08-03T20:34:25.639 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/crypto/tcrypt.ko 2024-08-03T20:34:25.639 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/crypto/tea.ko 2024-08-03T20:34:25.639 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/crypto/twofish_common.ko 2024-08-03T20:34:25.639 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/crypto/twofish_generic.ko 2024-08-03T20:34:25.639 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/crypto/vmac.ko 2024-08-03T20:34:25.639 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/crypto/wp512.ko 2024-08-03T20:34:25.639 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/crypto/xcbc.ko 2024-08-03T20:34:25.639 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/crypto/xor.ko 2024-08-03T20:34:25.639 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/crypto/xxhash_generic.ko 2024-08-03T20:34:25.639 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers 2024-08-03T20:34:25.639 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/acpi 2024-08-03T20:34:25.639 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/acpi/acpi_ipmi.ko 2024-08-03T20:34:25.640 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/acpi/acpi_pad.ko 2024-08-03T20:34:25.640 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/acpi/ec_sys.ko 2024-08-03T20:34:25.640 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/acpi/platform_profile.ko 2024-08-03T20:34:25.640 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/acpi/sbs.ko 2024-08-03T20:34:25.640 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/acpi/sbshc.ko 2024-08-03T20:34:25.640 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/acpi/video.ko 2024-08-03T20:34:25.640 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/ata 2024-08-03T20:34:25.640 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/ata/acard-ahci.ko 2024-08-03T20:34:25.640 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/ata/ahci_platform.ko 2024-08-03T20:34:25.640 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/ata/ata_generic.ko 2024-08-03T20:34:25.640 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/ata/libahci_platform.ko 2024-08-03T20:34:25.640 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/ata/pata_acpi.ko 2024-08-03T20:34:25.640 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/ata/pata_ali.ko 2024-08-03T20:34:25.640 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/ata/pata_amd.ko 2024-08-03T20:34:25.640 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/ata/pata_artop.ko 2024-08-03T20:34:25.641 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/ata/pata_atiixp.ko 2024-08-03T20:34:25.641 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/ata/pata_atp867x.ko 2024-08-03T20:34:25.641 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/ata/pata_cmd640.ko 2024-08-03T20:34:25.641 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/ata/pata_cmd64x.ko 2024-08-03T20:34:25.641 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/ata/pata_cypress.ko 2024-08-03T20:34:25.641 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/ata/pata_efar.ko 2024-08-03T20:34:25.641 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/ata/pata_hpt366.ko 2024-08-03T20:34:25.641 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/ata/pata_hpt37x.ko 2024-08-03T20:34:25.641 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/ata/pata_hpt3x2n.ko 2024-08-03T20:34:25.641 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/ata/pata_hpt3x3.ko 2024-08-03T20:34:25.641 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/ata/pata_it8213.ko 2024-08-03T20:34:25.641 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/ata/pata_it821x.ko 2024-08-03T20:34:25.641 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/ata/pata_jmicron.ko 2024-08-03T20:34:25.641 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/ata/pata_marvell.ko 2024-08-03T20:34:25.642 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/ata/pata_mpiix.ko 2024-08-03T20:34:25.642 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/ata/pata_netcell.ko 2024-08-03T20:34:25.642 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/ata/pata_ninja32.ko 2024-08-03T20:34:25.642 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/ata/pata_ns87410.ko 2024-08-03T20:34:25.642 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/ata/pata_ns87415.ko 2024-08-03T20:34:25.642 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/ata/pata_oldpiix.ko 2024-08-03T20:34:25.642 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/ata/pata_opti.ko 2024-08-03T20:34:25.642 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/ata/pata_optidma.ko 2024-08-03T20:34:25.642 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/ata/pata_pcmcia.ko 2024-08-03T20:34:25.642 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/ata/pata_pdc2027x.ko 2024-08-03T20:34:25.642 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/ata/pata_pdc202xx_old.ko 2024-08-03T20:34:25.642 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/ata/pata_piccolo.ko 2024-08-03T20:34:25.642 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/ata/pata_rdc.ko 2024-08-03T20:34:25.642 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/ata/pata_sch.ko 2024-08-03T20:34:25.643 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/ata/pata_serverworks.ko 2024-08-03T20:34:25.643 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/ata/pata_sil680.ko 2024-08-03T20:34:25.643 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/ata/pata_sis.ko 2024-08-03T20:34:25.643 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/ata/pata_sl82c105.ko 2024-08-03T20:34:25.643 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/ata/pata_triflex.ko 2024-08-03T20:34:25.643 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/ata/pata_via.ko 2024-08-03T20:34:25.643 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/ata/pdc_adma.ko 2024-08-03T20:34:25.643 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/ata/sata_inic162x.ko 2024-08-03T20:34:25.643 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/ata/sata_mv.ko 2024-08-03T20:34:25.643 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/ata/sata_nv.ko 2024-08-03T20:34:25.643 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/ata/sata_promise.ko 2024-08-03T20:34:25.644 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/ata/sata_qstor.ko 2024-08-03T20:34:25.644 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/ata/sata_sil.ko 2024-08-03T20:34:25.644 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/ata/sata_sil24.ko 2024-08-03T20:34:25.644 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/ata/sata_sis.ko 2024-08-03T20:34:25.644 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/ata/sata_svw.ko 2024-08-03T20:34:25.644 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/ata/sata_sx4.ko 2024-08-03T20:34:25.644 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/ata/sata_uli.ko 2024-08-03T20:34:25.645 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/ata/sata_via.ko 2024-08-03T20:34:25.645 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/ata/sata_vsc.ko 2024-08-03T20:34:25.645 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/auxdisplay 2024-08-03T20:34:25.645 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/auxdisplay/ks0108.ko 2024-08-03T20:34:25.645 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/base 2024-08-03T20:34:25.645 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/base/regmap 2024-08-03T20:34:25.645 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/base/regmap/regmap-i2c.ko 2024-08-03T20:34:25.645 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/base/regmap/regmap-mmio.ko 2024-08-03T20:34:25.645 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/bcma 2024-08-03T20:34:25.645 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/bcma/bcma.ko 2024-08-03T20:34:25.645 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/block 2024-08-03T20:34:25.645 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/block/aoe 2024-08-03T20:34:25.645 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/block/aoe/aoe.ko 2024-08-03T20:34:25.645 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/block/brd.ko 2024-08-03T20:34:25.645 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/block/drbd 2024-08-03T20:34:25.646 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/block/drbd/drbd.ko 2024-08-03T20:34:25.646 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/block/floppy.ko 2024-08-03T20:34:25.646 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/block/loop.ko 2024-08-03T20:34:25.646 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/block/mtip32xx 2024-08-03T20:34:25.646 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/block/mtip32xx/mtip32xx.ko 2024-08-03T20:34:25.646 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/block/nbd.ko 2024-08-03T20:34:25.646 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/block/null_blk 2024-08-03T20:34:25.646 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/block/null_blk/null_blk.ko 2024-08-03T20:34:25.646 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/block/pktcdvd.ko 2024-08-03T20:34:25.646 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/block/rbd.ko 2024-08-03T20:34:25.646 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/block/virtio_blk.ko 2024-08-03T20:34:25.646 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/block/xen-blkback 2024-08-03T20:34:25.646 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/block/xen-blkback/xen-blkback.ko 2024-08-03T20:34:25.646 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/block/xen-blkfront.ko 2024-08-03T20:34:25.647 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/block/zram 2024-08-03T20:34:25.647 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/block/zram/zram.ko 2024-08-03T20:34:25.647 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/char 2024-08-03T20:34:25.647 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/char/hangcheck-timer.ko 2024-08-03T20:34:25.647 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/char/hw_random 2024-08-03T20:34:25.647 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/char/hw_random/amd-rng.ko 2024-08-03T20:34:25.647 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/char/hw_random/intel-rng.ko 2024-08-03T20:34:25.647 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/char/hw_random/timeriomem-rng.ko 2024-08-03T20:34:25.647 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/char/hw_random/via-rng.ko 2024-08-03T20:34:25.647 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/char/hw_random/virtio-rng.ko 2024-08-03T20:34:25.647 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/char/ipmi 2024-08-03T20:34:25.647 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/char/ipmi/ipmi_devintf.ko 2024-08-03T20:34:25.647 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/char/ipmi/ipmi_msghandler.ko 2024-08-03T20:34:25.647 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/char/ipmi/ipmi_poweroff.ko 2024-08-03T20:34:25.648 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/char/ipmi/ipmi_si.ko 2024-08-03T20:34:25.648 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/char/ipmi/ipmi_ssif.ko 2024-08-03T20:34:25.648 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/char/ipmi/ipmi_watchdog.ko 2024-08-03T20:34:25.648 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/char/lp.ko 2024-08-03T20:34:25.648 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/char/mwave 2024-08-03T20:34:25.648 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/char/mwave/mwave.ko 2024-08-03T20:34:25.648 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/char/ppdev.ko 2024-08-03T20:34:25.648 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/char/tlclk.ko 2024-08-03T20:34:25.648 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/char/tpm 2024-08-03T20:34:25.648 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/char/tpm/tpm.ko 2024-08-03T20:34:25.648 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/char/tpm/tpm_atmel.ko 2024-08-03T20:34:25.648 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/char/tpm/tpm_infineon.ko 2024-08-03T20:34:25.648 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/char/tpm/tpm_nsc.ko 2024-08-03T20:34:25.648 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/char/tpm/tpm_tis.ko 2024-08-03T20:34:25.648 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/char/tpm/tpm_tis_core.ko 2024-08-03T20:34:25.649 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/char/uv_mmtimer.ko 2024-08-03T20:34:25.649 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/char/virtio_console.ko 2024-08-03T20:34:25.649 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/cpufreq 2024-08-03T20:34:25.649 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/cpufreq/acpi-cpufreq.ko 2024-08-03T20:34:25.649 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/cpufreq/amd_freq_sensitivity.ko 2024-08-03T20:34:25.649 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/cpufreq/p4-clockmod.ko 2024-08-03T20:34:25.649 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/cpufreq/pcc-cpufreq.ko 2024-08-03T20:34:25.649 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/cpufreq/powernow-k8.ko 2024-08-03T20:34:25.649 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/cpufreq/speedstep-lib.ko 2024-08-03T20:34:25.649 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/crypto 2024-08-03T20:34:25.649 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/crypto/ccp 2024-08-03T20:34:25.649 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/crypto/ccp/ccp-crypto.ko 2024-08-03T20:34:25.649 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/crypto/ccp/ccp.ko 2024-08-03T20:34:25.650 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/crypto/intel 2024-08-03T20:34:25.650 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/crypto/intel/qat 2024-08-03T20:34:25.650 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/crypto/intel/qat/qat_common 2024-08-03T20:34:25.650 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/crypto/intel/qat/qat_common/intel_qat.ko 2024-08-03T20:34:25.650 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/crypto/intel/qat/qat_dh895xcc 2024-08-03T20:34:25.650 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/crypto/intel/qat/qat_dh895xcc/qat_dh895xcc.ko 2024-08-03T20:34:25.650 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/crypto/padlock-aes.ko 2024-08-03T20:34:25.650 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/crypto/padlock-sha.ko 2024-08-03T20:34:25.650 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/crypto/virtio 2024-08-03T20:34:25.650 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/crypto/virtio/virtio_crypto.ko 2024-08-03T20:34:25.650 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/dca 2024-08-03T20:34:25.650 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/dca/dca.ko 2024-08-03T20:34:25.650 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/devfreq 2024-08-03T20:34:25.651 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/devfreq/governor_simpleondemand.ko 2024-08-03T20:34:25.651 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/dma 2024-08-03T20:34:25.651 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/dma/dw 2024-08-03T20:34:25.651 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/dma/dw/dw_dmac.ko 2024-08-03T20:34:25.651 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/dma/ioat 2024-08-03T20:34:25.651 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/dma/ioat/ioatdma.ko 2024-08-03T20:34:25.651 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/edac 2024-08-03T20:34:25.651 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/edac/amd64_edac.ko 2024-08-03T20:34:25.651 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/edac/e752x_edac.ko 2024-08-03T20:34:25.651 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/edac/edac_mce_amd.ko 2024-08-03T20:34:25.651 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/edac/i3000_edac.ko 2024-08-03T20:34:25.652 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/edac/i3200_edac.ko 2024-08-03T20:34:25.652 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/edac/i5100_edac.ko 2024-08-03T20:34:25.652 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/edac/i5400_edac.ko 2024-08-03T20:34:25.652 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/edac/i7300_edac.ko 2024-08-03T20:34:25.652 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/edac/i7core_edac.ko 2024-08-03T20:34:25.652 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/edac/i82975x_edac.ko 2024-08-03T20:34:25.652 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/edac/ie31200_edac.ko 2024-08-03T20:34:25.652 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/edac/sb_edac.ko 2024-08-03T20:34:25.652 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/edac/x38_edac.ko 2024-08-03T20:34:25.652 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/firmware 2024-08-03T20:34:25.652 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/firmware/edd.ko 2024-08-03T20:34:25.652 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/firmware/iscsi_ibft.ko 2024-08-03T20:34:25.652 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/gpio 2024-08-03T20:34:25.653 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/gpio/gpio-ich.ko 2024-08-03T20:34:25.653 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/gpio/gpio-viperboard.ko 2024-08-03T20:34:25.653 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/hid 2024-08-03T20:34:25.653 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/hid/hid-appleir.ko 2024-08-03T20:34:25.653 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/hid/hid-aureal.ko 2024-08-03T20:34:25.653 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/hid/hid-axff.ko 2024-08-03T20:34:25.653 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/hid/hid-dr.ko 2024-08-03T20:34:25.653 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/hid/hid-elecom.ko 2024-08-03T20:34:25.653 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/hid/hid-elo.ko 2024-08-03T20:34:25.653 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/hid/hid-emsff.ko 2024-08-03T20:34:25.653 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/hid/hid-gaff.ko 2024-08-03T20:34:25.653 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/hid/hid-gt683r.ko 2024-08-03T20:34:25.653 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/hid/hid-gyration.ko 2024-08-03T20:34:25.653 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/hid/hid-holtek-kbd.ko 2024-08-03T20:34:25.654 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/hid/hid-holtek-mouse.ko 2024-08-03T20:34:25.654 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/hid/hid-holtekff.ko 2024-08-03T20:34:25.654 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/hid/hid-hyperv.ko 2024-08-03T20:34:25.654 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/hid/hid-icade.ko 2024-08-03T20:34:25.654 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/hid/hid-keytouch.ko 2024-08-03T20:34:25.654 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/hid/hid-kye.ko 2024-08-03T20:34:25.654 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/hid/hid-lcpower.ko 2024-08-03T20:34:25.654 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/hid/hid-led.ko 2024-08-03T20:34:25.654 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/hid/hid-lenovo.ko 2024-08-03T20:34:25.654 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/hid/hid-logitech-dj.ko 2024-08-03T20:34:25.654 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/hid/hid-logitech-hidpp.ko 2024-08-03T20:34:25.654 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/hid/hid-multitouch.ko 2024-08-03T20:34:25.654 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/hid/hid-ortek.ko 2024-08-03T20:34:25.655 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/hid/hid-penmount.ko 2024-08-03T20:34:25.655 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/hid/hid-petalynx.ko 2024-08-03T20:34:25.655 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/hid/hid-picolcd.ko 2024-08-03T20:34:25.655 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/hid/hid-pl.ko 2024-08-03T20:34:25.655 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/hid/hid-plantronics.ko 2024-08-03T20:34:25.655 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/hid/hid-primax.ko 2024-08-03T20:34:25.655 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/hid/hid-rmi.ko 2024-08-03T20:34:25.655 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/hid/hid-roccat-arvo.ko 2024-08-03T20:34:25.655 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/hid/hid-roccat-common.ko 2024-08-03T20:34:25.655 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/hid/hid-roccat-isku.ko 2024-08-03T20:34:25.655 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/hid/hid-roccat-kone.ko 2024-08-03T20:34:25.655 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/hid/hid-roccat-koneplus.ko 2024-08-03T20:34:25.655 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/hid/hid-roccat-konepure.ko 2024-08-03T20:34:25.655 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/hid/hid-roccat-kovaplus.ko 2024-08-03T20:34:25.656 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/hid/hid-roccat-lua.ko 2024-08-03T20:34:25.656 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/hid/hid-roccat-pyra.ko 2024-08-03T20:34:25.656 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/hid/hid-roccat-ryos.ko 2024-08-03T20:34:25.656 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/hid/hid-roccat-savu.ko 2024-08-03T20:34:25.656 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/hid/hid-roccat.ko 2024-08-03T20:34:25.656 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/hid/hid-saitek.ko 2024-08-03T20:34:25.656 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/hid/hid-samsung.ko 2024-08-03T20:34:25.656 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/hid/hid-sensor-hub.ko 2024-08-03T20:34:25.656 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/hid/hid-sjoy.ko 2024-08-03T20:34:25.656 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/hid/hid-sony.ko 2024-08-03T20:34:25.656 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/hid/hid-speedlink.ko 2024-08-03T20:34:25.656 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/hid/hid-steelseries.ko 2024-08-03T20:34:25.656 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/hid/hid-sunplus.ko 2024-08-03T20:34:25.656 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/hid/hid-thrustmaster.ko 2024-08-03T20:34:25.657 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/hid/hid-tivo.ko 2024-08-03T20:34:25.657 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/hid/hid-tmff.ko 2024-08-03T20:34:25.657 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/hid/hid-topseed.ko 2024-08-03T20:34:25.657 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/hid/hid-twinhan.ko 2024-08-03T20:34:25.657 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/hid/hid-uclogic.ko 2024-08-03T20:34:25.657 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/hid/hid-waltop.ko 2024-08-03T20:34:25.657 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/hid/hid-wiimote.ko 2024-08-03T20:34:25.657 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/hid/hid-xinmo.ko 2024-08-03T20:34:25.657 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/hid/hid-zpff.ko 2024-08-03T20:34:25.657 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/hid/hid-zydacron.ko 2024-08-03T20:34:25.657 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/hid/uhid.ko 2024-08-03T20:34:25.657 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/hid/wacom.ko 2024-08-03T20:34:25.657 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/hv 2024-08-03T20:34:25.657 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/hv/hv_balloon.ko 2024-08-03T20:34:25.658 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/hv/hv_utils.ko 2024-08-03T20:34:25.658 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/hv/hv_vmbus.ko 2024-08-03T20:34:25.658 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/hwmon 2024-08-03T20:34:25.658 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/hwmon/abituguru.ko 2024-08-03T20:34:25.658 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/hwmon/abituguru3.ko 2024-08-03T20:34:25.658 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/hwmon/acpi_power_meter.ko 2024-08-03T20:34:25.658 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/hwmon/ad7414.ko 2024-08-03T20:34:25.658 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/hwmon/ad7418.ko 2024-08-03T20:34:25.658 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/hwmon/adc128d818.ko 2024-08-03T20:34:25.658 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/hwmon/adm1025.ko 2024-08-03T20:34:25.658 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/hwmon/adm1026.ko 2024-08-03T20:34:25.658 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/hwmon/adm1029.ko 2024-08-03T20:34:25.659 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/hwmon/adm1031.ko 2024-08-03T20:34:25.659 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/hwmon/adm9240.ko 2024-08-03T20:34:25.659 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/hwmon/ads7828.ko 2024-08-03T20:34:25.659 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/hwmon/adt7410.ko 2024-08-03T20:34:25.659 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/hwmon/adt7411.ko 2024-08-03T20:34:25.659 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/hwmon/adt7462.ko 2024-08-03T20:34:25.659 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/hwmon/adt7470.ko 2024-08-03T20:34:25.659 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/hwmon/adt7475.ko 2024-08-03T20:34:25.659 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/hwmon/adt7x10.ko 2024-08-03T20:34:25.659 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/hwmon/amc6821.ko 2024-08-03T20:34:25.659 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/hwmon/applesmc.ko 2024-08-03T20:34:25.659 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/hwmon/asb100.ko 2024-08-03T20:34:25.659 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/hwmon/asc7621.ko 2024-08-03T20:34:25.660 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/hwmon/asus_atk0110.ko 2024-08-03T20:34:25.660 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/hwmon/atxp1.ko 2024-08-03T20:34:25.660 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/hwmon/coretemp.ko 2024-08-03T20:34:25.660 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/hwmon/dell-smm-hwmon.ko 2024-08-03T20:34:25.660 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/hwmon/dme1737.ko 2024-08-03T20:34:25.660 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/hwmon/ds1621.ko 2024-08-03T20:34:25.660 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/hwmon/ds620.ko 2024-08-03T20:34:25.660 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/hwmon/emc1403.ko 2024-08-03T20:34:25.660 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/hwmon/emc6w201.ko 2024-08-03T20:34:25.660 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/hwmon/f71805f.ko 2024-08-03T20:34:25.660 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/hwmon/f71882fg.ko 2024-08-03T20:34:25.660 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/hwmon/f75375s.ko 2024-08-03T20:34:25.660 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/hwmon/fam15h_power.ko 2024-08-03T20:34:25.660 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/hwmon/fschmd.ko 2024-08-03T20:34:25.661 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/hwmon/g760a.ko 2024-08-03T20:34:25.661 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/hwmon/g762.ko 2024-08-03T20:34:25.661 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/hwmon/gl518sm.ko 2024-08-03T20:34:25.661 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/hwmon/gl520sm.ko 2024-08-03T20:34:25.661 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/hwmon/hwmon-vid.ko 2024-08-03T20:34:25.661 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/hwmon/i5500_temp.ko 2024-08-03T20:34:25.661 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/hwmon/i5k_amb.ko 2024-08-03T20:34:25.661 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/hwmon/ibmaem.ko 2024-08-03T20:34:25.661 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/hwmon/ibmpex.ko 2024-08-03T20:34:25.661 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/hwmon/ina209.ko 2024-08-03T20:34:25.661 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/hwmon/ina2xx.ko 2024-08-03T20:34:25.662 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/hwmon/it87.ko 2024-08-03T20:34:25.662 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/hwmon/k10temp.ko 2024-08-03T20:34:25.662 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/hwmon/k8temp.ko 2024-08-03T20:34:25.662 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/hwmon/lineage-pem.ko 2024-08-03T20:34:25.662 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/hwmon/lm63.ko 2024-08-03T20:34:25.662 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/hwmon/lm73.ko 2024-08-03T20:34:25.662 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/hwmon/lm75.ko 2024-08-03T20:34:25.662 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/hwmon/lm77.ko 2024-08-03T20:34:25.662 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/hwmon/lm78.ko 2024-08-03T20:34:25.662 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/hwmon/lm80.ko 2024-08-03T20:34:25.662 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/hwmon/lm83.ko 2024-08-03T20:34:25.662 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/hwmon/lm85.ko 2024-08-03T20:34:25.663 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/hwmon/lm87.ko 2024-08-03T20:34:25.663 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/hwmon/lm90.ko 2024-08-03T20:34:25.663 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/hwmon/lm92.ko 2024-08-03T20:34:25.663 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/hwmon/lm93.ko 2024-08-03T20:34:25.663 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/hwmon/lm95234.ko 2024-08-03T20:34:25.663 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/hwmon/lm95241.ko 2024-08-03T20:34:25.663 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/hwmon/lm95245.ko 2024-08-03T20:34:25.663 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/hwmon/ltc2945.ko 2024-08-03T20:34:25.663 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/hwmon/ltc4151.ko 2024-08-03T20:34:25.663 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/hwmon/ltc4215.ko 2024-08-03T20:34:25.663 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/hwmon/ltc4222.ko 2024-08-03T20:34:25.663 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/hwmon/ltc4245.ko 2024-08-03T20:34:25.663 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/hwmon/ltc4260.ko 2024-08-03T20:34:25.663 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/hwmon/ltc4261.ko 2024-08-03T20:34:25.663 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/hwmon/max16065.ko 2024-08-03T20:34:25.664 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/hwmon/max1619.ko 2024-08-03T20:34:25.664 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/hwmon/max1668.ko 2024-08-03T20:34:25.664 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/hwmon/max197.ko 2024-08-03T20:34:25.664 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/hwmon/max6639.ko 2024-08-03T20:34:25.664 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/hwmon/max6650.ko 2024-08-03T20:34:25.664 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/hwmon/max6697.ko 2024-08-03T20:34:25.664 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/hwmon/mcp3021.ko 2024-08-03T20:34:25.664 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/hwmon/nct6683.ko 2024-08-03T20:34:25.664 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/hwmon/nct6775-core.ko 2024-08-03T20:34:25.664 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/hwmon/nct6775.ko 2024-08-03T20:34:25.664 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/hwmon/nct7802.ko 2024-08-03T20:34:25.664 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/hwmon/pc87360.ko 2024-08-03T20:34:25.664 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/hwmon/pc87427.ko 2024-08-03T20:34:25.665 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/hwmon/pcf8591.ko 2024-08-03T20:34:25.665 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/hwmon/pmbus 2024-08-03T20:34:25.665 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/hwmon/pmbus/adm1275.ko 2024-08-03T20:34:25.665 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/hwmon/pmbus/lm25066.ko 2024-08-03T20:34:25.665 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/hwmon/pmbus/ltc2978.ko 2024-08-03T20:34:25.665 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/hwmon/pmbus/max16064.ko 2024-08-03T20:34:25.665 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/hwmon/pmbus/max34440.ko 2024-08-03T20:34:25.665 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/hwmon/pmbus/max8688.ko 2024-08-03T20:34:25.665 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/hwmon/pmbus/pmbus.ko 2024-08-03T20:34:25.665 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/hwmon/pmbus/pmbus_core.ko 2024-08-03T20:34:25.665 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/hwmon/pmbus/tps40422.ko 2024-08-03T20:34:25.665 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/hwmon/pmbus/ucd9000.ko 2024-08-03T20:34:25.665 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/hwmon/pmbus/ucd9200.ko 2024-08-03T20:34:25.665 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/hwmon/pmbus/zl6100.ko 2024-08-03T20:34:25.666 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/hwmon/powr1220.ko 2024-08-03T20:34:25.666 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/hwmon/sch5627.ko 2024-08-03T20:34:25.666 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/hwmon/sch5636.ko 2024-08-03T20:34:25.666 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/hwmon/sch56xx-common.ko 2024-08-03T20:34:25.666 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/hwmon/sht15.ko 2024-08-03T20:34:25.666 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/hwmon/sht21.ko 2024-08-03T20:34:25.666 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/hwmon/shtc1.ko 2024-08-03T20:34:25.666 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/hwmon/sis5595.ko 2024-08-03T20:34:25.666 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/hwmon/smsc47b397.ko 2024-08-03T20:34:25.666 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/hwmon/smsc47m1.ko 2024-08-03T20:34:25.666 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/hwmon/smsc47m192.ko 2024-08-03T20:34:25.666 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/hwmon/thmc50.ko 2024-08-03T20:34:25.666 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/hwmon/tmp102.ko 2024-08-03T20:34:25.666 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/hwmon/tmp103.ko 2024-08-03T20:34:25.667 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/hwmon/tmp401.ko 2024-08-03T20:34:25.667 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/hwmon/tmp421.ko 2024-08-03T20:34:25.667 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/hwmon/via-cputemp.ko 2024-08-03T20:34:25.667 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/hwmon/via686a.ko 2024-08-03T20:34:25.667 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/hwmon/vt1211.ko 2024-08-03T20:34:25.667 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/hwmon/vt8231.ko 2024-08-03T20:34:25.667 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/hwmon/w83627ehf.ko 2024-08-03T20:34:25.667 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/hwmon/w83627hf.ko 2024-08-03T20:34:25.667 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/hwmon/w83781d.ko 2024-08-03T20:34:25.667 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/hwmon/w83791d.ko 2024-08-03T20:34:25.667 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/hwmon/w83792d.ko 2024-08-03T20:34:25.667 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/hwmon/w83793.ko 2024-08-03T20:34:25.667 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/hwmon/w83795.ko 2024-08-03T20:34:25.668 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/hwmon/w83l785ts.ko 2024-08-03T20:34:25.668 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/hwmon/w83l786ng.ko 2024-08-03T20:34:25.668 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/i2c 2024-08-03T20:34:25.668 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/i2c/algos 2024-08-03T20:34:25.668 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/i2c/algos/i2c-algo-bit.ko 2024-08-03T20:34:25.668 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/i2c/algos/i2c-algo-pca.ko 2024-08-03T20:34:25.668 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/i2c/busses 2024-08-03T20:34:25.668 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/i2c/busses/i2c-amd756-s4882.ko 2024-08-03T20:34:25.668 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/i2c/busses/i2c-amd756.ko 2024-08-03T20:34:25.668 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/i2c/busses/i2c-amd8111.ko 2024-08-03T20:34:25.668 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/i2c/busses/i2c-ccgx-ucsi.ko 2024-08-03T20:34:25.668 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/i2c/busses/i2c-designware-core.ko 2024-08-03T20:34:25.668 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/i2c/busses/i2c-designware-pci.ko 2024-08-03T20:34:25.668 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/i2c/busses/i2c-designware-platform.ko 2024-08-03T20:34:25.669 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/i2c/busses/i2c-diolan-u2c.ko 2024-08-03T20:34:25.669 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/i2c/busses/i2c-i801.ko 2024-08-03T20:34:25.669 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/i2c/busses/i2c-isch.ko 2024-08-03T20:34:25.669 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/i2c/busses/i2c-ismt.ko 2024-08-03T20:34:25.669 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/i2c/busses/i2c-nforce2-s4985.ko 2024-08-03T20:34:25.669 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/i2c/busses/i2c-nforce2.ko 2024-08-03T20:34:25.669 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/i2c/busses/i2c-parport.ko 2024-08-03T20:34:25.669 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/i2c/busses/i2c-pca-platform.ko 2024-08-03T20:34:25.669 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/i2c/busses/i2c-piix4.ko 2024-08-03T20:34:25.669 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/i2c/busses/i2c-scmi.ko 2024-08-03T20:34:25.669 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/i2c/busses/i2c-simtec.ko 2024-08-03T20:34:25.669 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/i2c/busses/i2c-sis96x.ko 2024-08-03T20:34:25.669 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/i2c/busses/i2c-tiny-usb.ko 2024-08-03T20:34:25.669 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/i2c/busses/i2c-via.ko 2024-08-03T20:34:25.670 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/i2c/busses/i2c-viapro.ko 2024-08-03T20:34:25.670 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/i2c/busses/i2c-viperboard.ko 2024-08-03T20:34:25.670 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/i2c/i2c-dev.ko 2024-08-03T20:34:25.670 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/i2c/i2c-mux.ko 2024-08-03T20:34:25.670 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/i2c/i2c-slave-eeprom.ko 2024-08-03T20:34:25.670 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/i2c/i2c-smbus.ko 2024-08-03T20:34:25.670 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/i2c/i2c-stub.ko 2024-08-03T20:34:25.670 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/infiniband 2024-08-03T20:34:25.670 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/infiniband/core 2024-08-03T20:34:25.670 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/infiniband/core/ib_cm.ko 2024-08-03T20:34:25.670 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/infiniband/core/ib_core.ko 2024-08-03T20:34:25.670 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/infiniband/core/ib_umad.ko 2024-08-03T20:34:25.670 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/infiniband/core/ib_uverbs.ko 2024-08-03T20:34:25.671 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/infiniband/core/iw_cm.ko 2024-08-03T20:34:25.671 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/infiniband/core/rdma_cm.ko 2024-08-03T20:34:25.671 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/infiniband/core/rdma_ucm.ko 2024-08-03T20:34:25.671 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/infiniband/hw 2024-08-03T20:34:25.671 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/infiniband/hw/cxgb4 2024-08-03T20:34:25.671 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/infiniband/hw/cxgb4/iw_cxgb4.ko 2024-08-03T20:34:25.671 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/infiniband/hw/mlx4 2024-08-03T20:34:25.671 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/infiniband/hw/mlx4/mlx4_ib.ko 2024-08-03T20:34:25.671 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/infiniband/hw/mlx5 2024-08-03T20:34:25.671 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/infiniband/hw/mlx5/mlx5_ib.ko 2024-08-03T20:34:25.671 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/infiniband/hw/mthca 2024-08-03T20:34:25.671 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/infiniband/hw/mthca/ib_mthca.ko 2024-08-03T20:34:25.671 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/infiniband/ulp 2024-08-03T20:34:25.671 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/infiniband/ulp/ipoib 2024-08-03T20:34:25.672 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/infiniband/ulp/ipoib/ib_ipoib.ko 2024-08-03T20:34:25.672 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/infiniband/ulp/iser 2024-08-03T20:34:25.672 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/infiniband/ulp/iser/ib_iser.ko 2024-08-03T20:34:25.672 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/infiniband/ulp/isert 2024-08-03T20:34:25.672 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/infiniband/ulp/isert/ib_isert.ko 2024-08-03T20:34:25.672 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/infiniband/ulp/srp 2024-08-03T20:34:25.672 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/infiniband/ulp/srp/ib_srp.ko 2024-08-03T20:34:25.672 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/infiniband/ulp/srpt 2024-08-03T20:34:25.672 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/infiniband/ulp/srpt/ib_srpt.ko 2024-08-03T20:34:25.672 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/input 2024-08-03T20:34:25.672 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/input/gameport 2024-08-03T20:34:25.672 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/input/gameport/emu10k1-gp.ko 2024-08-03T20:34:25.672 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/input/gameport/fm801-gp.ko 2024-08-03T20:34:25.672 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/input/gameport/gameport.ko 2024-08-03T20:34:25.673 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/input/joydev.ko 2024-08-03T20:34:25.673 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/input/keyboard 2024-08-03T20:34:25.673 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/input/keyboard/gpio_keys.ko 2024-08-03T20:34:25.673 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/input/mouse 2024-08-03T20:34:25.673 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/input/mouse/appletouch.ko 2024-08-03T20:34:25.673 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/input/mouse/bcm5974.ko 2024-08-03T20:34:25.673 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/input/mouse/cyapatp.ko 2024-08-03T20:34:25.673 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/input/mouse/elan_i2c.ko 2024-08-03T20:34:25.673 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/input/mouse/sermouse.ko 2024-08-03T20:34:25.673 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/input/mouse/synaptics_i2c.ko 2024-08-03T20:34:25.673 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/input/mouse/synaptics_usb.ko 2024-08-03T20:34:25.673 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/input/mouse/vsxxxaa.ko 2024-08-03T20:34:25.673 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/input/rmi4 2024-08-03T20:34:25.674 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/input/rmi4/rmi_core.ko 2024-08-03T20:34:25.674 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/input/serio 2024-08-03T20:34:25.674 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/input/serio/altera_ps2.ko 2024-08-03T20:34:25.674 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/input/serio/arc_ps2.ko 2024-08-03T20:34:25.674 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/input/serio/hyperv-keyboard.ko 2024-08-03T20:34:25.674 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/input/serio/serio_raw.ko 2024-08-03T20:34:25.674 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/input/sparse-keymap.ko 2024-08-03T20:34:25.674 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/leds 2024-08-03T20:34:25.674 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/leds/leds-blinkm.ko 2024-08-03T20:34:25.674 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/leds/leds-lm3530.ko 2024-08-03T20:34:25.674 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/leds/leds-lp3944.ko 2024-08-03T20:34:25.674 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/leds/leds-ss4200.ko 2024-08-03T20:34:25.674 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/leds/trigger 2024-08-03T20:34:25.674 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/leds/trigger/ledtrig-backlight.ko 2024-08-03T20:34:25.675 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/leds/trigger/ledtrig-camera.ko 2024-08-03T20:34:25.675 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/leds/trigger/ledtrig-default-on.ko 2024-08-03T20:34:25.675 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/leds/trigger/ledtrig-gpio.ko 2024-08-03T20:34:25.675 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/leds/trigger/ledtrig-heartbeat.ko 2024-08-03T20:34:25.675 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/leds/trigger/ledtrig-oneshot.ko 2024-08-03T20:34:25.675 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/leds/trigger/ledtrig-timer.ko 2024-08-03T20:34:25.675 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/leds/trigger/ledtrig-transient.ko 2024-08-03T20:34:25.675 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/md 2024-08-03T20:34:25.675 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/md/bcache 2024-08-03T20:34:25.675 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/md/bcache/bcache.ko 2024-08-03T20:34:25.675 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/md/dm-bio-prison.ko 2024-08-03T20:34:25.675 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/md/dm-cache-smq.ko 2024-08-03T20:34:25.675 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/md/dm-cache.ko 2024-08-03T20:34:25.675 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/md/dm-crypt.ko 2024-08-03T20:34:25.676 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/md/dm-delay.ko 2024-08-03T20:34:25.676 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/md/dm-flakey.ko 2024-08-03T20:34:25.676 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/md/dm-log-userspace.ko 2024-08-03T20:34:25.676 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/md/dm-multipath.ko 2024-08-03T20:34:25.676 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/md/dm-queue-length.ko 2024-08-03T20:34:25.676 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/md/dm-raid.ko 2024-08-03T20:34:25.676 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/md/dm-round-robin.ko 2024-08-03T20:34:25.676 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/md/dm-service-time.ko 2024-08-03T20:34:25.676 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/md/dm-switch.ko 2024-08-03T20:34:25.676 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/md/dm-thin-pool.ko 2024-08-03T20:34:25.676 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/md/dm-verity.ko 2024-08-03T20:34:25.676 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/md/persistent-data 2024-08-03T20:34:25.676 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/md/persistent-data/dm-persistent-data.ko 2024-08-03T20:34:25.677 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/md/raid0.ko 2024-08-03T20:34:25.677 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/md/raid1.ko 2024-08-03T20:34:25.677 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/md/raid10.ko 2024-08-03T20:34:25.677 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/md/raid456.ko 2024-08-03T20:34:25.677 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/message 2024-08-03T20:34:25.677 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/message/fusion 2024-08-03T20:34:25.677 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/message/fusion/mptbase.ko 2024-08-03T20:34:25.677 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/message/fusion/mptctl.ko 2024-08-03T20:34:25.677 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/message/fusion/mptfc.ko 2024-08-03T20:34:25.677 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/message/fusion/mptlan.ko 2024-08-03T20:34:25.677 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/message/fusion/mptsas.ko 2024-08-03T20:34:25.677 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/message/fusion/mptscsih.ko 2024-08-03T20:34:25.678 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/message/fusion/mptspi.ko 2024-08-03T20:34:25.678 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/mfd 2024-08-03T20:34:25.678 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/mfd/lpc_ich.ko 2024-08-03T20:34:25.678 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/mfd/lpc_sch.ko 2024-08-03T20:34:25.678 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/mfd/mfd-core.ko 2024-08-03T20:34:25.678 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/mfd/sm501.ko 2024-08-03T20:34:25.678 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/mfd/viperboard.ko 2024-08-03T20:34:25.678 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/mfd/vx855.ko 2024-08-03T20:34:25.678 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/mfd/wl1273-core.ko 2024-08-03T20:34:25.678 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/misc 2024-08-03T20:34:25.678 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/misc/altera-stapl 2024-08-03T20:34:25.678 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/misc/altera-stapl/altera-stapl.ko 2024-08-03T20:34:25.679 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/misc/apds9802als.ko 2024-08-03T20:34:25.679 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/misc/apds990x.ko 2024-08-03T20:34:25.679 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/misc/bh1770glc.ko 2024-08-03T20:34:25.679 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/misc/cb710 2024-08-03T20:34:25.679 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/misc/cb710/cb710.ko 2024-08-03T20:34:25.679 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/misc/eeprom 2024-08-03T20:34:25.679 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/misc/eeprom/at24.ko 2024-08-03T20:34:25.679 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/misc/eeprom/eeprom_93cx6.ko 2024-08-03T20:34:25.679 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/misc/eeprom/max6875.ko 2024-08-03T20:34:25.679 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/misc/enclosure.ko 2024-08-03T20:34:25.679 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/misc/hpilo.ko 2024-08-03T20:34:25.679 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/misc/isl29003.ko 2024-08-03T20:34:25.680 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/misc/isl29020.ko 2024-08-03T20:34:25.680 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/misc/lis3lv02d 2024-08-03T20:34:25.680 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/misc/lis3lv02d/lis3lv02d.ko 2024-08-03T20:34:25.680 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/misc/lis3lv02d/lis3lv02d_i2c.ko 2024-08-03T20:34:25.680 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/misc/mei 2024-08-03T20:34:25.680 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/misc/mei/mei-me.ko 2024-08-03T20:34:25.680 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/misc/mei/mei-txe.ko 2024-08-03T20:34:25.680 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/misc/mei/mei.ko 2024-08-03T20:34:25.680 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/misc/sgi-gru 2024-08-03T20:34:25.680 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/misc/sgi-gru/gru.ko 2024-08-03T20:34:25.680 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/misc/sgi-xp 2024-08-03T20:34:25.680 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/misc/sgi-xp/xp.ko 2024-08-03T20:34:25.680 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/misc/sgi-xp/xpc.ko 2024-08-03T20:34:25.680 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/misc/sgi-xp/xpnet.ko 2024-08-03T20:34:25.681 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/misc/tifm_7xx1.ko 2024-08-03T20:34:25.681 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/misc/tifm_core.ko 2024-08-03T20:34:25.681 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/misc/tsl2550.ko 2024-08-03T20:34:25.681 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/misc/vmw_balloon.ko 2024-08-03T20:34:25.681 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/misc/vmw_vmci 2024-08-03T20:34:25.681 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/misc/vmw_vmci/vmw_vmci.ko 2024-08-03T20:34:25.681 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net 2024-08-03T20:34:25.681 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/bonding 2024-08-03T20:34:25.681 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/bonding/bonding.ko 2024-08-03T20:34:25.681 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/dsa 2024-08-03T20:34:25.681 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/dsa/b53 2024-08-03T20:34:25.681 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/dsa/b53/b53_common.ko 2024-08-03T20:34:25.681 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/dsa/bcm-sf2.ko 2024-08-03T20:34:25.682 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/dsa/mv88e6060.ko 2024-08-03T20:34:25.682 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/dsa/mv88e6xxx 2024-08-03T20:34:25.682 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/dsa/mv88e6xxx/mv88e6xxx.ko 2024-08-03T20:34:25.682 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/dummy.ko 2024-08-03T20:34:25.682 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/eql.ko 2024-08-03T20:34:25.682 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/ethernet 2024-08-03T20:34:25.682 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/ethernet/3com 2024-08-03T20:34:25.682 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/ethernet/3com/3c574_cs.ko 2024-08-03T20:34:25.682 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/ethernet/3com/3c589_cs.ko 2024-08-03T20:34:25.682 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/ethernet/3com/3c59x.ko 2024-08-03T20:34:25.682 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/ethernet/3com/typhoon.ko 2024-08-03T20:34:25.682 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/ethernet/8390 2024-08-03T20:34:25.682 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/ethernet/8390/8390.ko 2024-08-03T20:34:25.682 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/ethernet/8390/axnet_cs.ko 2024-08-03T20:34:25.683 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/ethernet/8390/ne2k-pci.ko 2024-08-03T20:34:25.683 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/ethernet/8390/pcnet_cs.ko 2024-08-03T20:34:25.683 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/ethernet/adaptec 2024-08-03T20:34:25.683 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/ethernet/adaptec/starfire.ko 2024-08-03T20:34:25.683 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/ethernet/agere 2024-08-03T20:34:25.683 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/ethernet/agere/et131x.ko 2024-08-03T20:34:25.683 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/ethernet/alteon 2024-08-03T20:34:25.683 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/ethernet/alteon/acenic.ko 2024-08-03T20:34:25.683 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/ethernet/altera 2024-08-03T20:34:25.683 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/ethernet/altera/altera_tse.ko 2024-08-03T20:34:25.683 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/ethernet/amd 2024-08-03T20:34:25.683 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/ethernet/amd/amd8111e.ko 2024-08-03T20:34:25.683 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/ethernet/amd/nmclan_cs.ko 2024-08-03T20:34:25.683 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/ethernet/amd/pcnet32.ko 2024-08-03T20:34:25.683 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/ethernet/atheros 2024-08-03T20:34:25.684 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/ethernet/atheros/alx 2024-08-03T20:34:25.684 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/ethernet/atheros/alx/alx.ko 2024-08-03T20:34:25.684 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/ethernet/atheros/atl1c 2024-08-03T20:34:25.684 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/ethernet/atheros/atl1c/atl1c.ko 2024-08-03T20:34:25.684 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/ethernet/atheros/atl1e 2024-08-03T20:34:25.684 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/ethernet/atheros/atl1e/atl1e.ko 2024-08-03T20:34:25.684 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/ethernet/atheros/atlx 2024-08-03T20:34:25.684 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/ethernet/atheros/atlx/atl1.ko 2024-08-03T20:34:25.684 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/ethernet/atheros/atlx/atl2.ko 2024-08-03T20:34:25.684 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/ethernet/broadcom 2024-08-03T20:34:25.684 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/ethernet/broadcom/b44.ko 2024-08-03T20:34:25.684 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/ethernet/broadcom/bnx2.ko 2024-08-03T20:34:25.684 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/ethernet/broadcom/bnx2x 2024-08-03T20:34:25.684 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/ethernet/broadcom/bnx2x/bnx2x.ko 2024-08-03T20:34:25.685 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/ethernet/broadcom/cnic.ko 2024-08-03T20:34:25.685 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/ethernet/broadcom/genet 2024-08-03T20:34:25.685 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/ethernet/broadcom/genet/genet.ko 2024-08-03T20:34:25.685 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/ethernet/broadcom/tg3.ko 2024-08-03T20:34:25.685 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/ethernet/brocade 2024-08-03T20:34:25.685 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/ethernet/brocade/bna 2024-08-03T20:34:25.685 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/ethernet/brocade/bna/bna.ko 2024-08-03T20:34:25.685 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/ethernet/chelsio 2024-08-03T20:34:25.685 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/ethernet/chelsio/cxgb 2024-08-03T20:34:25.685 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/ethernet/chelsio/cxgb/cxgb.ko 2024-08-03T20:34:25.685 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/ethernet/chelsio/cxgb3 2024-08-03T20:34:25.685 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/ethernet/chelsio/cxgb3/cxgb3.ko 2024-08-03T20:34:25.685 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/ethernet/chelsio/cxgb4 2024-08-03T20:34:25.685 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/ethernet/chelsio/cxgb4/cxgb4.ko 2024-08-03T20:34:25.686 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/ethernet/chelsio/cxgb4vf 2024-08-03T20:34:25.686 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/ethernet/chelsio/cxgb4vf/cxgb4vf.ko 2024-08-03T20:34:25.686 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/ethernet/chelsio/libcxgb 2024-08-03T20:34:25.686 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/ethernet/chelsio/libcxgb/libcxgb.ko 2024-08-03T20:34:25.686 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/ethernet/cisco 2024-08-03T20:34:25.686 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/ethernet/cisco/enic 2024-08-03T20:34:25.686 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/ethernet/cisco/enic/enic.ko 2024-08-03T20:34:25.686 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/ethernet/dec 2024-08-03T20:34:25.686 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/ethernet/dec/tulip 2024-08-03T20:34:25.686 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/ethernet/dec/tulip/de2104x.ko 2024-08-03T20:34:25.686 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/ethernet/dec/tulip/dmfe.ko 2024-08-03T20:34:25.686 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/ethernet/dec/tulip/tulip.ko 2024-08-03T20:34:25.686 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/ethernet/dec/tulip/uli526x.ko 2024-08-03T20:34:25.687 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/ethernet/dec/tulip/winbond-840.ko 2024-08-03T20:34:25.687 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/ethernet/dec/tulip/xircom_cb.ko 2024-08-03T20:34:25.687 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/ethernet/dlink 2024-08-03T20:34:25.687 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/ethernet/dlink/dl2k.ko 2024-08-03T20:34:25.687 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/ethernet/dlink/sundance.ko 2024-08-03T20:34:25.687 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/ethernet/dnet.ko 2024-08-03T20:34:25.687 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/ethernet/emulex 2024-08-03T20:34:25.687 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/ethernet/emulex/benet 2024-08-03T20:34:25.687 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/ethernet/emulex/benet/be2net.ko 2024-08-03T20:34:25.687 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/ethernet/ethoc.ko 2024-08-03T20:34:25.687 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/ethernet/fealnx.ko 2024-08-03T20:34:25.687 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/ethernet/intel 2024-08-03T20:34:25.687 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/ethernet/intel/e100.ko 2024-08-03T20:34:25.688 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/ethernet/intel/e1000 2024-08-03T20:34:25.688 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/ethernet/intel/e1000/e1000.ko 2024-08-03T20:34:25.688 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/ethernet/intel/e1000e 2024-08-03T20:34:25.688 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/ethernet/intel/e1000e/e1000e.ko 2024-08-03T20:34:25.688 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/ethernet/intel/fm10k 2024-08-03T20:34:25.688 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/ethernet/intel/fm10k/fm10k.ko 2024-08-03T20:34:25.688 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/ethernet/intel/i40e 2024-08-03T20:34:25.688 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/ethernet/intel/i40e/i40e.ko 2024-08-03T20:34:25.688 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/ethernet/intel/iavf 2024-08-03T20:34:25.688 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/ethernet/intel/iavf/iavf.ko 2024-08-03T20:34:25.688 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/ethernet/intel/igb 2024-08-03T20:34:25.688 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/ethernet/intel/igb/igb.ko 2024-08-03T20:34:25.688 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/ethernet/intel/igbvf 2024-08-03T20:34:25.688 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/ethernet/intel/igbvf/igbvf.ko 2024-08-03T20:34:25.689 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/ethernet/intel/ixgbe 2024-08-03T20:34:25.689 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/ethernet/intel/ixgbe/ixgbe.ko 2024-08-03T20:34:25.689 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/ethernet/intel/ixgbevf 2024-08-03T20:34:25.689 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/ethernet/intel/ixgbevf/ixgbevf.ko 2024-08-03T20:34:25.689 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/ethernet/intel/libeth 2024-08-03T20:34:25.689 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/ethernet/intel/libeth/libeth.ko 2024-08-03T20:34:25.689 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/ethernet/intel/libie 2024-08-03T20:34:25.689 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/ethernet/intel/libie/libie.ko 2024-08-03T20:34:25.689 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/ethernet/jme.ko 2024-08-03T20:34:25.689 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/ethernet/marvell 2024-08-03T20:34:25.689 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/ethernet/marvell/mvmdio.ko 2024-08-03T20:34:25.689 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/ethernet/marvell/skge.ko 2024-08-03T20:34:25.690 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/ethernet/marvell/sky2.ko 2024-08-03T20:34:25.690 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/ethernet/mellanox 2024-08-03T20:34:25.690 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/ethernet/mellanox/mlx4 2024-08-03T20:34:25.690 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_core.ko 2024-08-03T20:34:25.690 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_en.ko 2024-08-03T20:34:25.690 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/ethernet/mellanox/mlx5 2024-08-03T20:34:25.690 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/ethernet/mellanox/mlx5/core 2024-08-03T20:34:25.690 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/ethernet/mellanox/mlx5/core/mlx5_core.ko 2024-08-03T20:34:25.690 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/ethernet/micrel 2024-08-03T20:34:25.690 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/ethernet/micrel/ksz884x.ko 2024-08-03T20:34:25.690 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/ethernet/myricom 2024-08-03T20:34:25.690 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/ethernet/myricom/myri10ge 2024-08-03T20:34:25.690 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/ethernet/myricom/myri10ge/myri10ge.ko 2024-08-03T20:34:25.690 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/ethernet/natsemi 2024-08-03T20:34:25.691 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/ethernet/natsemi/natsemi.ko 2024-08-03T20:34:25.691 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/ethernet/natsemi/ns83820.ko 2024-08-03T20:34:25.691 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/ethernet/neterion 2024-08-03T20:34:25.691 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/ethernet/neterion/s2io.ko 2024-08-03T20:34:25.691 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/ethernet/nvidia 2024-08-03T20:34:25.691 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/ethernet/nvidia/forcedeth.ko 2024-08-03T20:34:25.691 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/ethernet/packetengines 2024-08-03T20:34:25.691 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/ethernet/packetengines/hamachi.ko 2024-08-03T20:34:25.691 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/ethernet/packetengines/yellowfin.ko 2024-08-03T20:34:25.691 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/ethernet/qlogic 2024-08-03T20:34:25.691 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/ethernet/qlogic/netxen 2024-08-03T20:34:25.691 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/ethernet/qlogic/netxen/netxen_nic.ko 2024-08-03T20:34:25.691 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/ethernet/qlogic/qla3xxx.ko 2024-08-03T20:34:25.692 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/ethernet/qlogic/qlcnic 2024-08-03T20:34:25.692 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/ethernet/qlogic/qlcnic/qlcnic.ko 2024-08-03T20:34:25.692 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/ethernet/rdc 2024-08-03T20:34:25.692 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/ethernet/rdc/r6040.ko 2024-08-03T20:34:25.692 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/ethernet/realtek 2024-08-03T20:34:25.692 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/ethernet/realtek/8139cp.ko 2024-08-03T20:34:25.692 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/ethernet/realtek/8139too.ko 2024-08-03T20:34:25.692 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/ethernet/realtek/atp.ko 2024-08-03T20:34:25.692 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/ethernet/realtek/r8169.ko 2024-08-03T20:34:25.692 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/ethernet/rocker 2024-08-03T20:34:25.692 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/ethernet/rocker/rocker.ko 2024-08-03T20:34:25.692 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/ethernet/sfc 2024-08-03T20:34:25.692 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/ethernet/sfc/sfc.ko 2024-08-03T20:34:25.693 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/ethernet/silan 2024-08-03T20:34:25.693 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/ethernet/silan/sc92031.ko 2024-08-03T20:34:25.693 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/ethernet/sis 2024-08-03T20:34:25.693 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/ethernet/sis/sis190.ko 2024-08-03T20:34:25.693 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/ethernet/sis/sis900.ko 2024-08-03T20:34:25.693 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/ethernet/smsc 2024-08-03T20:34:25.693 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/ethernet/smsc/epic100.ko 2024-08-03T20:34:25.693 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/ethernet/smsc/smc91c92_cs.ko 2024-08-03T20:34:25.693 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/ethernet/smsc/smsc911x.ko 2024-08-03T20:34:25.693 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/ethernet/smsc/smsc9420.ko 2024-08-03T20:34:25.693 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/ethernet/stmicro 2024-08-03T20:34:25.693 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/ethernet/stmicro/stmmac 2024-08-03T20:34:25.694 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/ethernet/stmicro/stmmac/dwmac-intel.ko 2024-08-03T20:34:25.694 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/ethernet/stmicro/stmmac/stmmac.ko 2024-08-03T20:34:25.694 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/ethernet/sun 2024-08-03T20:34:25.694 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/ethernet/sun/cassini.ko 2024-08-03T20:34:25.694 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/ethernet/sun/niu.ko 2024-08-03T20:34:25.694 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/ethernet/sun/sungem.ko 2024-08-03T20:34:25.694 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/ethernet/sun/sunhme.ko 2024-08-03T20:34:25.694 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/ethernet/tehuti 2024-08-03T20:34:25.694 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/ethernet/tehuti/tehuti.ko 2024-08-03T20:34:25.695 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/ethernet/ti 2024-08-03T20:34:25.695 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/ethernet/ti/tlan.ko 2024-08-03T20:34:25.695 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/ethernet/via 2024-08-03T20:34:25.695 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/ethernet/via/via-rhine.ko 2024-08-03T20:34:25.695 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/ethernet/via/via-velocity.ko 2024-08-03T20:34:25.695 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/ethernet/wiznet 2024-08-03T20:34:25.695 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/ethernet/wiznet/w5100.ko 2024-08-03T20:34:25.695 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/ethernet/wiznet/w5300.ko 2024-08-03T20:34:25.695 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/ethernet/xircom 2024-08-03T20:34:25.695 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/ethernet/xircom/xirc2ps_cs.ko 2024-08-03T20:34:25.695 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/geneve.ko 2024-08-03T20:34:25.695 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/hyperv 2024-08-03T20:34:25.695 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/hyperv/hv_netvsc.ko 2024-08-03T20:34:25.696 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/ieee802154 2024-08-03T20:34:25.696 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/ieee802154/fakelb.ko 2024-08-03T20:34:25.696 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/ifb.ko 2024-08-03T20:34:25.696 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/ipvlan 2024-08-03T20:34:25.696 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/ipvlan/ipvlan.ko 2024-08-03T20:34:25.696 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/macvlan.ko 2024-08-03T20:34:25.696 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/macvtap.ko 2024-08-03T20:34:25.696 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/mdio 2024-08-03T20:34:25.696 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/mdio.ko 2024-08-03T20:34:25.696 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/mdio/mdio-bcm-unimac.ko 2024-08-03T20:34:25.696 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/mdio/mdio-bitbang.ko 2024-08-03T20:34:25.696 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/mdio/mdio-regmap.ko 2024-08-03T20:34:25.696 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/mii.ko 2024-08-03T20:34:25.697 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/net_failover.ko 2024-08-03T20:34:25.697 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/netconsole.ko 2024-08-03T20:34:25.697 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/nlmon.ko 2024-08-03T20:34:25.697 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/pcs 2024-08-03T20:34:25.697 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/pcs/pcs-lynx.ko 2024-08-03T20:34:25.697 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/pcs/pcs_xpcs.ko 2024-08-03T20:34:25.697 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/phy 2024-08-03T20:34:25.697 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/phy/amd.ko 2024-08-03T20:34:25.697 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/phy/bcm-phy-lib.ko 2024-08-03T20:34:25.697 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/phy/bcm-phy-ptp.ko 2024-08-03T20:34:25.697 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/phy/bcm7xxx.ko 2024-08-03T20:34:25.697 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/phy/bcm87xx.ko 2024-08-03T20:34:25.697 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/phy/broadcom.ko 2024-08-03T20:34:25.698 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/phy/cicada.ko 2024-08-03T20:34:25.698 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/phy/davicom.ko 2024-08-03T20:34:25.698 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/phy/dp83640.ko 2024-08-03T20:34:25.698 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/phy/et1011c.ko 2024-08-03T20:34:25.698 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/phy/icplus.ko 2024-08-03T20:34:25.698 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/phy/lxt.ko 2024-08-03T20:34:25.698 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/phy/marvell.ko 2024-08-03T20:34:25.698 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/phy/micrel.ko 2024-08-03T20:34:25.698 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/phy/national.ko 2024-08-03T20:34:25.698 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/phy/phylink.ko 2024-08-03T20:34:25.698 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/phy/qsemi.ko 2024-08-03T20:34:25.698 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/phy/realtek.ko 2024-08-03T20:34:25.698 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/phy/smsc.ko 2024-08-03T20:34:25.699 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/phy/ste10Xp.ko 2024-08-03T20:34:25.699 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/phy/vitesse.ko 2024-08-03T20:34:25.699 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/ppp 2024-08-03T20:34:25.699 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/ppp/bsd_comp.ko 2024-08-03T20:34:25.699 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/ppp/ppp_async.ko 2024-08-03T20:34:25.699 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/ppp/ppp_deflate.ko 2024-08-03T20:34:25.699 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/ppp/ppp_generic.ko 2024-08-03T20:34:25.699 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/ppp/ppp_mppe.ko 2024-08-03T20:34:25.699 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/ppp/ppp_synctty.ko 2024-08-03T20:34:25.699 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/ppp/pppoe.ko 2024-08-03T20:34:25.699 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/ppp/pppox.ko 2024-08-03T20:34:25.699 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/ppp/pptp.ko 2024-08-03T20:34:25.699 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/slip 2024-08-03T20:34:25.699 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/slip/slhc.ko 2024-08-03T20:34:25.699 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/slip/slip.ko 2024-08-03T20:34:25.700 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/sungem_phy.ko 2024-08-03T20:34:25.700 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/tap.ko 2024-08-03T20:34:25.700 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/team 2024-08-03T20:34:25.700 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/team/team.ko 2024-08-03T20:34:25.700 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/team/team_mode_activebackup.ko 2024-08-03T20:34:25.700 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/team/team_mode_broadcast.ko 2024-08-03T20:34:25.700 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/team/team_mode_loadbalance.ko 2024-08-03T20:34:25.700 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/team/team_mode_random.ko 2024-08-03T20:34:25.700 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/team/team_mode_roundrobin.ko 2024-08-03T20:34:25.700 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/tun.ko 2024-08-03T20:34:25.700 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/veth.ko 2024-08-03T20:34:25.700 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/virtio_net.ko 2024-08-03T20:34:25.700 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/vmxnet3 2024-08-03T20:34:25.700 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/vmxnet3/vmxnet3.ko 2024-08-03T20:34:25.701 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/vxlan 2024-08-03T20:34:25.701 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/vxlan/vxlan.ko 2024-08-03T20:34:25.701 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/xen-netback 2024-08-03T20:34:25.701 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/xen-netback/xen-netback.ko 2024-08-03T20:34:25.701 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/xen-netfront.ko 2024-08-03T20:34:25.701 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/ntb 2024-08-03T20:34:25.701 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/ntb/ntb.ko 2024-08-03T20:34:25.701 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/nvme 2024-08-03T20:34:25.701 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/nvme/host 2024-08-03T20:34:25.701 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/nvme/host/nvme-core.ko 2024-08-03T20:34:25.701 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/nvme/host/nvme.ko 2024-08-03T20:34:25.701 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/parport 2024-08-03T20:34:25.701 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/parport/parport.ko 2024-08-03T20:34:25.701 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/parport/parport_cs.ko 2024-08-03T20:34:25.701 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/parport/parport_pc.ko 2024-08-03T20:34:25.702 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/parport/parport_serial.ko 2024-08-03T20:34:25.702 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/pci 2024-08-03T20:34:25.702 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/pci/hotplug 2024-08-03T20:34:25.702 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/pci/hotplug/acpiphp_ibm.ko 2024-08-03T20:34:25.702 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/pci/pcie 2024-08-03T20:34:25.702 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/pci/pcie/aer_inject.ko 2024-08-03T20:34:25.702 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/pci/xen-pcifront.ko 2024-08-03T20:34:25.702 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/pcmcia 2024-08-03T20:34:25.702 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/pcmcia/i82092.ko 2024-08-03T20:34:25.702 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/pcmcia/pd6729.ko 2024-08-03T20:34:25.702 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/pcmcia/yenta_socket.ko 2024-08-03T20:34:25.702 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/pinctrl 2024-08-03T20:34:25.702 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/pinctrl/intel 2024-08-03T20:34:25.703 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/pinctrl/intel/pinctrl-cherryview.ko 2024-08-03T20:34:25.703 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/platform 2024-08-03T20:34:25.703 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/platform/x86 2024-08-03T20:34:25.703 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/platform/x86/acer-wmi.ko 2024-08-03T20:34:25.703 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/platform/x86/acerhdf.ko 2024-08-03T20:34:25.703 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/platform/x86/apple-gmux.ko 2024-08-03T20:34:25.703 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/platform/x86/asus-laptop.ko 2024-08-03T20:34:25.703 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/platform/x86/asus-nb-wmi.ko 2024-08-03T20:34:25.703 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/platform/x86/asus-wmi.ko 2024-08-03T20:34:25.703 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/platform/x86/classmate-laptop.ko 2024-08-03T20:34:25.703 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/platform/x86/eeepc-laptop.ko 2024-08-03T20:34:25.703 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/platform/x86/eeepc-wmi.ko 2024-08-03T20:34:25.703 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/platform/x86/fujitsu-laptop.ko 2024-08-03T20:34:25.703 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/platform/x86/fujitsu-tablet.ko 2024-08-03T20:34:25.704 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/platform/x86/hdaps.ko 2024-08-03T20:34:25.704 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/platform/x86/intel 2024-08-03T20:34:25.704 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/platform/x86/intel/intel-rst.ko 2024-08-03T20:34:25.704 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/platform/x86/intel_ips.ko 2024-08-03T20:34:25.704 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/platform/x86/msi-wmi.ko 2024-08-03T20:34:25.704 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/platform/x86/mxm-wmi.ko 2024-08-03T20:34:25.704 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/platform/x86/panasonic-laptop.ko 2024-08-03T20:34:25.704 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/platform/x86/samsung-laptop.ko 2024-08-03T20:34:25.704 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/platform/x86/samsung-q10.ko 2024-08-03T20:34:25.704 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/platform/x86/topstar-laptop.ko 2024-08-03T20:34:25.704 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/platform/x86/toshiba_bluetooth.ko 2024-08-03T20:34:25.704 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/platform/x86/toshiba_haps.ko 2024-08-03T20:34:25.705 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/platform/x86/wmi-bmof.ko 2024-08-03T20:34:25.705 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/platform/x86/wmi.ko 2024-08-03T20:34:25.705 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/powercap 2024-08-03T20:34:25.705 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/powercap/intel_rapl_common.ko 2024-08-03T20:34:25.705 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/powercap/intel_rapl_msr.ko 2024-08-03T20:34:25.705 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/pps 2024-08-03T20:34:25.705 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/pps/clients 2024-08-03T20:34:25.705 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/pps/clients/pps-gpio.ko 2024-08-03T20:34:25.705 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/pps/clients/pps-ldisc.ko 2024-08-03T20:34:25.705 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/pps/clients/pps_parport.ko 2024-08-03T20:34:25.705 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/pps/pps_core.ko 2024-08-03T20:34:25.705 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/ptp 2024-08-03T20:34:25.705 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/ptp/ptp.ko 2024-08-03T20:34:25.705 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/ptp/ptp_kvm.ko 2024-08-03T20:34:25.706 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/ras 2024-08-03T20:34:25.706 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/ras/amd 2024-08-03T20:34:25.706 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/ras/amd/atl 2024-08-03T20:34:25.706 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/ras/amd/atl/amd_atl.ko 2024-08-03T20:34:25.706 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/ras/amd/fmpm.ko 2024-08-03T20:34:25.706 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/rtc 2024-08-03T20:34:25.706 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/rtc/rtc-bq32k.ko 2024-08-03T20:34:25.706 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/rtc/rtc-ds1286.ko 2024-08-03T20:34:25.706 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/rtc/rtc-ds1307.ko 2024-08-03T20:34:25.706 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/rtc/rtc-ds1374.ko 2024-08-03T20:34:25.706 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/rtc/rtc-ds1511.ko 2024-08-03T20:34:25.706 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/rtc/rtc-ds1553.ko 2024-08-03T20:34:25.706 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/rtc/rtc-ds1672.ko 2024-08-03T20:34:25.706 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/rtc/rtc-ds1742.ko 2024-08-03T20:34:25.707 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/rtc/rtc-ds2404.ko 2024-08-03T20:34:25.707 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/rtc/rtc-ds3232.ko 2024-08-03T20:34:25.707 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/rtc/rtc-em3027.ko 2024-08-03T20:34:25.707 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/rtc/rtc-fm3130.ko 2024-08-03T20:34:25.707 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/rtc/rtc-isl12022.ko 2024-08-03T20:34:25.707 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/rtc/rtc-isl1208.ko 2024-08-03T20:34:25.707 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/rtc/rtc-m41t80.ko 2024-08-03T20:34:25.707 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/rtc/rtc-m48t35.ko 2024-08-03T20:34:25.707 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/rtc/rtc-m48t59.ko 2024-08-03T20:34:25.707 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/rtc/rtc-max6900.ko 2024-08-03T20:34:25.707 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/rtc/rtc-msm6242.ko 2024-08-03T20:34:25.707 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/rtc/rtc-pcf2127.ko 2024-08-03T20:34:25.707 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/rtc/rtc-pcf85063.ko 2024-08-03T20:34:25.707 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/rtc/rtc-pcf8523.ko 2024-08-03T20:34:25.708 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/rtc/rtc-pcf8563.ko 2024-08-03T20:34:25.708 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/rtc/rtc-pcf8583.ko 2024-08-03T20:34:25.708 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/rtc/rtc-rp5c01.ko 2024-08-03T20:34:25.708 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/rtc/rtc-rs5c372.ko 2024-08-03T20:34:25.708 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/rtc/rtc-rv3029c2.ko 2024-08-03T20:34:25.708 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/rtc/rtc-rx8025.ko 2024-08-03T20:34:25.708 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/rtc/rtc-rx8581.ko 2024-08-03T20:34:25.708 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/rtc/rtc-stk17ta8.ko 2024-08-03T20:34:25.708 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/rtc/rtc-x1205.ko 2024-08-03T20:34:25.708 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/scsi 2024-08-03T20:34:25.708 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/scsi/3w-9xxx.ko 2024-08-03T20:34:25.708 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/scsi/3w-sas.ko 2024-08-03T20:34:25.708 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/scsi/3w-xxxx.ko 2024-08-03T20:34:25.708 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/scsi/BusLogic.ko 2024-08-03T20:34:25.709 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/scsi/a100u2w.ko 2024-08-03T20:34:25.709 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/scsi/aacraid 2024-08-03T20:34:25.709 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/scsi/aacraid/aacraid.ko 2024-08-03T20:34:25.709 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/scsi/advansys.ko 2024-08-03T20:34:25.709 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/scsi/aic7xxx 2024-08-03T20:34:25.709 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/scsi/aic7xxx/aic79xx.ko 2024-08-03T20:34:25.709 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/scsi/aic7xxx/aic7xxx.ko 2024-08-03T20:34:25.709 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/scsi/am53c974.ko 2024-08-03T20:34:25.709 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/scsi/arcmsr 2024-08-03T20:34:25.709 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/scsi/arcmsr/arcmsr.ko 2024-08-03T20:34:25.709 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/scsi/atp870u.ko 2024-08-03T20:34:25.709 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/scsi/be2iscsi 2024-08-03T20:34:25.710 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/scsi/be2iscsi/be2iscsi.ko 2024-08-03T20:34:25.710 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/scsi/bfa 2024-08-03T20:34:25.710 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/scsi/bfa/bfa.ko 2024-08-03T20:34:25.710 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/scsi/bnx2fc 2024-08-03T20:34:25.710 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/scsi/bnx2fc/bnx2fc.ko 2024-08-03T20:34:25.710 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/scsi/bnx2i 2024-08-03T20:34:25.710 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/scsi/bnx2i/bnx2i.ko 2024-08-03T20:34:25.710 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/scsi/ch.ko 2024-08-03T20:34:25.710 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/scsi/csiostor 2024-08-03T20:34:25.710 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/scsi/csiostor/csiostor.ko 2024-08-03T20:34:25.710 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/scsi/cxgbi 2024-08-03T20:34:25.710 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/scsi/cxgbi/cxgb3i 2024-08-03T20:34:25.710 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/scsi/cxgbi/cxgb3i/cxgb3i.ko 2024-08-03T20:34:25.710 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/scsi/cxgbi/cxgb4i 2024-08-03T20:34:25.711 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/scsi/cxgbi/cxgb4i/cxgb4i.ko 2024-08-03T20:34:25.711 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/scsi/cxgbi/libcxgbi.ko 2024-08-03T20:34:25.711 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/scsi/dc395x.ko 2024-08-03T20:34:25.711 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/scsi/device_handler 2024-08-03T20:34:25.711 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/scsi/device_handler/scsi_dh_alua.ko 2024-08-03T20:34:25.711 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/scsi/device_handler/scsi_dh_emc.ko 2024-08-03T20:34:25.711 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/scsi/device_handler/scsi_dh_hp_sw.ko 2024-08-03T20:34:25.711 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/scsi/device_handler/scsi_dh_rdac.ko 2024-08-03T20:34:25.711 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/scsi/dmx3191d.ko 2024-08-03T20:34:25.711 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/scsi/esas2r 2024-08-03T20:34:25.711 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/scsi/esas2r/esas2r.ko 2024-08-03T20:34:25.712 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/scsi/esp_scsi.ko 2024-08-03T20:34:25.712 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/scsi/fcoe 2024-08-03T20:34:25.712 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/scsi/fcoe/fcoe.ko 2024-08-03T20:34:25.712 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/scsi/fcoe/libfcoe.ko 2024-08-03T20:34:25.712 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/scsi/fnic 2024-08-03T20:34:25.712 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/scsi/fnic/fnic.ko 2024-08-03T20:34:25.712 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/scsi/hpsa.ko 2024-08-03T20:34:25.712 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/scsi/hptiop.ko 2024-08-03T20:34:25.712 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/scsi/hv_storvsc.ko 2024-08-03T20:34:25.712 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/scsi/initio.ko 2024-08-03T20:34:25.712 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/scsi/ipr.ko 2024-08-03T20:34:25.712 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/scsi/ips.ko 2024-08-03T20:34:25.713 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/scsi/isci 2024-08-03T20:34:25.713 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/scsi/isci/isci.ko 2024-08-03T20:34:25.713 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/scsi/iscsi_boot_sysfs.ko 2024-08-03T20:34:25.713 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/scsi/iscsi_tcp.ko 2024-08-03T20:34:25.713 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/scsi/libfc 2024-08-03T20:34:25.713 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/scsi/libfc/libfc.ko 2024-08-03T20:34:25.713 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/scsi/libiscsi.ko 2024-08-03T20:34:25.713 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/scsi/libiscsi_tcp.ko 2024-08-03T20:34:25.713 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/scsi/libsas 2024-08-03T20:34:25.713 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/scsi/libsas/libsas.ko 2024-08-03T20:34:25.713 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/scsi/lpfc 2024-08-03T20:34:25.713 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/scsi/lpfc/lpfc.ko 2024-08-03T20:34:25.713 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/scsi/megaraid 2024-08-03T20:34:25.714 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/scsi/megaraid.ko 2024-08-03T20:34:25.714 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/scsi/megaraid/megaraid_mbox.ko 2024-08-03T20:34:25.714 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/scsi/megaraid/megaraid_mm.ko 2024-08-03T20:34:25.714 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/scsi/megaraid/megaraid_sas.ko 2024-08-03T20:34:25.714 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/scsi/mpt3sas 2024-08-03T20:34:25.714 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/scsi/mpt3sas/mpt3sas.ko 2024-08-03T20:34:25.714 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/scsi/mvsas 2024-08-03T20:34:25.714 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/scsi/mvsas/mvsas.ko 2024-08-03T20:34:25.714 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/scsi/mvumi.ko 2024-08-03T20:34:25.714 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/scsi/pm8001 2024-08-03T20:34:25.714 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/scsi/pm8001/pm80xx.ko 2024-08-03T20:34:25.714 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/scsi/pmcraid.ko 2024-08-03T20:34:25.714 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/scsi/qla1280.ko 2024-08-03T20:34:25.715 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/scsi/qla2xxx 2024-08-03T20:34:25.715 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/scsi/qla2xxx/qla2xxx.ko 2024-08-03T20:34:25.715 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/scsi/qla2xxx/tcm_qla2xxx.ko 2024-08-03T20:34:25.715 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/scsi/qla4xxx 2024-08-03T20:34:25.715 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/scsi/qla4xxx/qla4xxx.ko 2024-08-03T20:34:25.715 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/scsi/raid_class.ko 2024-08-03T20:34:25.715 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/scsi/scsi_debug.ko 2024-08-03T20:34:25.715 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/scsi/scsi_transport_fc.ko 2024-08-03T20:34:25.715 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/scsi/scsi_transport_iscsi.ko 2024-08-03T20:34:25.715 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/scsi/scsi_transport_sas.ko 2024-08-03T20:34:25.715 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/scsi/scsi_transport_spi.ko 2024-08-03T20:34:25.715 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/scsi/scsi_transport_srp.ko 2024-08-03T20:34:25.715 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/scsi/ses.ko 2024-08-03T20:34:25.716 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/scsi/st.ko 2024-08-03T20:34:25.716 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/scsi/stex.ko 2024-08-03T20:34:25.716 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/scsi/sym53c8xx_2 2024-08-03T20:34:25.716 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/scsi/sym53c8xx_2/sym53c8xx.ko 2024-08-03T20:34:25.716 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/scsi/virtio_scsi.ko 2024-08-03T20:34:25.716 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/scsi/vmw_pvscsi.ko 2024-08-03T20:34:25.716 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/scsi/wd719x.ko 2024-08-03T20:34:25.716 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/ssb 2024-08-03T20:34:25.716 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/ssb/ssb.ko 2024-08-03T20:34:25.716 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/target 2024-08-03T20:34:25.716 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/target/iscsi 2024-08-03T20:34:25.716 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/target/iscsi/iscsi_target_mod.ko 2024-08-03T20:34:25.716 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/target/loopback 2024-08-03T20:34:25.716 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/target/loopback/tcm_loop.ko 2024-08-03T20:34:25.717 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/target/target_core_file.ko 2024-08-03T20:34:25.717 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/target/target_core_iblock.ko 2024-08-03T20:34:25.717 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/target/target_core_mod.ko 2024-08-03T20:34:25.717 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/target/target_core_pscsi.ko 2024-08-03T20:34:25.717 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/target/target_core_user.ko 2024-08-03T20:34:25.717 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/target/tcm_fc 2024-08-03T20:34:25.717 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/target/tcm_fc/tcm_fc.ko 2024-08-03T20:34:25.717 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/thermal 2024-08-03T20:34:25.717 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/thermal/intel 2024-08-03T20:34:25.717 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/thermal/intel/int340x_thermal 2024-08-03T20:34:25.717 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/thermal/intel/int340x_thermal/acpi_thermal_rel.ko 2024-08-03T20:34:25.717 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/thermal/intel/int340x_thermal/int3400_thermal.ko 2024-08-03T20:34:25.718 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/thermal/intel/int340x_thermal/int3401_thermal.ko 2024-08-03T20:34:25.718 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/thermal/intel/int340x_thermal/int3402_thermal.ko 2024-08-03T20:34:25.718 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/thermal/intel/int340x_thermal/int3403_thermal.ko 2024-08-03T20:34:25.718 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/thermal/intel/int340x_thermal/int340x_thermal_zone.ko 2024-08-03T20:34:25.718 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device.ko 2024-08-03T20:34:25.718 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device_pci.ko 2024-08-03T20:34:25.718 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device_pci_legacy.ko 2024-08-03T20:34:25.718 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_mbox.ko 2024-08-03T20:34:25.718 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_power_floor.ko 2024-08-03T20:34:25.718 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_rapl.ko 2024-08-03T20:34:25.718 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_rfim.ko 2024-08-03T20:34:25.718 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_wt_hint.ko 2024-08-03T20:34:25.718 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_wt_req.ko 2024-08-03T20:34:25.719 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/thermal/intel/intel_soc_dts_iosf.ko 2024-08-03T20:34:25.719 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/thermal/intel/intel_soc_dts_thermal.ko 2024-08-03T20:34:25.719 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/thermal/intel/x86_pkg_temp_thermal.ko 2024-08-03T20:34:25.719 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/tty 2024-08-03T20:34:25.719 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/tty/ipwireless 2024-08-03T20:34:25.719 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/tty/ipwireless/ipwireless.ko 2024-08-03T20:34:25.719 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/tty/n_gsm.ko 2024-08-03T20:34:25.719 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/tty/n_hdlc.ko 2024-08-03T20:34:25.719 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/tty/nozomi.ko 2024-08-03T20:34:25.719 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/tty/serial 2024-08-03T20:34:25.719 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/tty/serial/8250 2024-08-03T20:34:25.719 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/tty/serial/8250/serial_cs.ko 2024-08-03T20:34:25.719 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/tty/serial/arc_uart.ko 2024-08-03T20:34:25.719 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/tty/serial/jsm 2024-08-03T20:34:25.720 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/tty/serial/jsm/jsm.ko 2024-08-03T20:34:25.720 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/ufs 2024-08-03T20:34:25.720 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/ufs/core 2024-08-03T20:34:25.720 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/ufs/core/ufshcd-core.ko 2024-08-03T20:34:25.720 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/ufs/host 2024-08-03T20:34:25.720 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/ufs/host/ufshcd-pci.ko 2024-08-03T20:34:25.720 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/uio 2024-08-03T20:34:25.720 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/uio/uio.ko 2024-08-03T20:34:25.720 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/uio/uio_aec.ko 2024-08-03T20:34:25.720 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/uio/uio_cif.ko 2024-08-03T20:34:25.720 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/uio/uio_pci_generic.ko 2024-08-03T20:34:25.720 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/uio/uio_sercos3.ko 2024-08-03T20:34:25.720 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/usb 2024-08-03T20:34:25.720 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/usb/class 2024-08-03T20:34:25.721 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/usb/class/cdc-acm.ko 2024-08-03T20:34:25.721 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/usb/class/cdc-wdm.ko 2024-08-03T20:34:25.721 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/usb/class/usblp.ko 2024-08-03T20:34:25.721 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/usb/class/usbtmc.ko 2024-08-03T20:34:25.721 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/usb/host 2024-08-03T20:34:25.721 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/usb/host/sl811-hcd.ko 2024-08-03T20:34:25.721 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/usb/image 2024-08-03T20:34:25.721 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/usb/image/mdc800.ko 2024-08-03T20:34:25.721 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/usb/image/microtek.ko 2024-08-03T20:34:25.721 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/usb/misc 2024-08-03T20:34:25.721 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/usb/misc/adutux.ko 2024-08-03T20:34:25.721 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/usb/misc/appledisplay.ko 2024-08-03T20:34:25.721 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/usb/misc/emi26.ko 2024-08-03T20:34:25.721 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/usb/misc/emi62.ko 2024-08-03T20:34:25.722 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/usb/misc/ezusb.ko 2024-08-03T20:34:25.722 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/usb/misc/idmouse.ko 2024-08-03T20:34:25.722 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/usb/misc/iowarrior.ko 2024-08-03T20:34:25.722 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/usb/misc/isight_firmware.ko 2024-08-03T20:34:25.722 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/usb/misc/ldusb.ko 2024-08-03T20:34:25.722 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/usb/misc/legousbtower.ko 2024-08-03T20:34:25.722 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/usb/misc/sisusbvga 2024-08-03T20:34:25.722 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/usb/misc/sisusbvga/sisusbvga.ko 2024-08-03T20:34:25.722 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/usb/misc/trancevibrator.ko 2024-08-03T20:34:25.722 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/usb/misc/usb3503.ko 2024-08-03T20:34:25.722 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/usb/misc/usblcd.ko 2024-08-03T20:34:25.722 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/usb/misc/usbsevseg.ko 2024-08-03T20:34:25.722 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/usb/misc/uss720.ko 2024-08-03T20:34:25.722 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/usb/misc/yurex.ko 2024-08-03T20:34:25.723 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/usb/phy 2024-08-03T20:34:25.723 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/usb/phy/phy-generic.ko 2024-08-03T20:34:25.723 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/usb/serial 2024-08-03T20:34:25.723 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/usb/serial/aircable.ko 2024-08-03T20:34:25.723 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/usb/serial/ark3116.ko 2024-08-03T20:34:25.723 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/usb/serial/belkin_sa.ko 2024-08-03T20:34:25.723 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/usb/serial/ch341.ko 2024-08-03T20:34:25.723 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/usb/serial/cp210x.ko 2024-08-03T20:34:25.723 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/usb/serial/cyberjack.ko 2024-08-03T20:34:25.723 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/usb/serial/cypress_m8.ko 2024-08-03T20:34:25.723 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/usb/serial/digi_acceleport.ko 2024-08-03T20:34:25.723 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/usb/serial/empeg.ko 2024-08-03T20:34:25.723 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/usb/serial/ftdi_sio.ko 2024-08-03T20:34:25.724 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/usb/serial/garmin_gps.ko 2024-08-03T20:34:25.724 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/usb/serial/io_edgeport.ko 2024-08-03T20:34:25.724 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/usb/serial/io_ti.ko 2024-08-03T20:34:25.724 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/usb/serial/ipaq.ko 2024-08-03T20:34:25.724 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/usb/serial/ipw.ko 2024-08-03T20:34:25.724 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/usb/serial/ir-usb.ko 2024-08-03T20:34:25.724 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/usb/serial/iuu_phoenix.ko 2024-08-03T20:34:25.724 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/usb/serial/keyspan.ko 2024-08-03T20:34:25.724 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/usb/serial/keyspan_pda.ko 2024-08-03T20:34:25.724 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/usb/serial/kl5kusb105.ko 2024-08-03T20:34:25.724 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/usb/serial/kobil_sct.ko 2024-08-03T20:34:25.724 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/usb/serial/mct_u232.ko 2024-08-03T20:34:25.724 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/usb/serial/mos7720.ko 2024-08-03T20:34:25.724 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/usb/serial/mos7840.ko 2024-08-03T20:34:25.724 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/usb/serial/navman.ko 2024-08-03T20:34:25.725 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/usb/serial/omninet.ko 2024-08-03T20:34:25.725 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/usb/serial/opticon.ko 2024-08-03T20:34:25.725 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/usb/serial/option.ko 2024-08-03T20:34:25.725 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/usb/serial/oti6858.ko 2024-08-03T20:34:25.725 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/usb/serial/pl2303.ko 2024-08-03T20:34:25.725 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/usb/serial/qcaux.ko 2024-08-03T20:34:25.725 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/usb/serial/qcserial.ko 2024-08-03T20:34:25.725 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/usb/serial/quatech2.ko 2024-08-03T20:34:25.725 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/usb/serial/safe_serial.ko 2024-08-03T20:34:25.725 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/usb/serial/sierra.ko 2024-08-03T20:34:25.725 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/usb/serial/spcp8x5.ko 2024-08-03T20:34:25.725 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/usb/serial/ssu100.ko 2024-08-03T20:34:25.725 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/usb/serial/symbolserial.ko 2024-08-03T20:34:25.726 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/usb/serial/ti_usb_3410_5052.ko 2024-08-03T20:34:25.726 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/usb/serial/usb-serial-simple.ko 2024-08-03T20:34:25.726 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/usb/serial/usb_debug.ko 2024-08-03T20:34:25.726 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/usb/serial/usb_wwan.ko 2024-08-03T20:34:25.726 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/usb/serial/visor.ko 2024-08-03T20:34:25.726 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/usb/serial/whiteheat.ko 2024-08-03T20:34:25.726 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/usb/serial/xsens_mt.ko 2024-08-03T20:34:25.726 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/usb/storage 2024-08-03T20:34:25.726 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/usb/storage/uas.ko 2024-08-03T20:34:25.726 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/usb/storage/ums-alauda.ko 2024-08-03T20:34:25.726 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/usb/storage/ums-cypress.ko 2024-08-03T20:34:25.726 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/usb/storage/ums-datafab.ko 2024-08-03T20:34:25.726 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/usb/storage/ums-eneub6250.ko 2024-08-03T20:34:25.727 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/usb/storage/ums-freecom.ko 2024-08-03T20:34:25.727 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/usb/storage/ums-isd200.ko 2024-08-03T20:34:25.727 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/usb/storage/ums-jumpshot.ko 2024-08-03T20:34:25.727 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/usb/storage/ums-karma.ko 2024-08-03T20:34:25.727 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/usb/storage/ums-onetouch.ko 2024-08-03T20:34:25.727 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/usb/storage/ums-realtek.ko 2024-08-03T20:34:25.727 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/usb/storage/ums-sddr09.ko 2024-08-03T20:34:25.727 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/usb/storage/ums-sddr55.ko 2024-08-03T20:34:25.727 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/usb/storage/ums-usbat.ko 2024-08-03T20:34:25.727 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/usb/storage/usb-storage.ko 2024-08-03T20:34:25.727 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/usb/usbip 2024-08-03T20:34:25.727 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/usb/usbip/usbip-core.ko 2024-08-03T20:34:25.727 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/usb/usbip/usbip-host.ko 2024-08-03T20:34:25.727 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/usb/usbip/vhci-hcd.ko 2024-08-03T20:34:25.727 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/vfio 2024-08-03T20:34:25.728 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/vfio/pci 2024-08-03T20:34:25.728 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/vfio/pci/vfio-pci-core.ko 2024-08-03T20:34:25.728 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/vfio/pci/vfio-pci.ko 2024-08-03T20:34:25.728 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/vfio/vfio.ko 2024-08-03T20:34:25.728 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/vfio/vfio_iommu_type1.ko 2024-08-03T20:34:25.728 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/vhost 2024-08-03T20:34:25.728 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/vhost/vhost.ko 2024-08-03T20:34:25.728 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/vhost/vhost_iotlb.ko 2024-08-03T20:34:25.728 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/vhost/vhost_net.ko 2024-08-03T20:34:25.728 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/vhost/vhost_scsi.ko 2024-08-03T20:34:25.728 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/video 2024-08-03T20:34:25.728 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/video/backlight 2024-08-03T20:34:25.728 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/video/backlight/apple_bl.ko 2024-08-03T20:34:25.728 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/video/backlight/lcd.ko 2024-08-03T20:34:25.729 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/video/backlight/platform_lcd.ko 2024-08-03T20:34:25.729 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/virtio 2024-08-03T20:34:25.729 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/virtio/virtio.ko 2024-08-03T20:34:25.729 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/virtio/virtio_balloon.ko 2024-08-03T20:34:25.729 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/virtio/virtio_mmio.ko 2024-08-03T20:34:25.729 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/virtio/virtio_pci.ko 2024-08-03T20:34:25.729 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/virtio/virtio_pci_legacy_dev.ko 2024-08-03T20:34:25.729 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/virtio/virtio_pci_modern_dev.ko 2024-08-03T20:34:25.729 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/virtio/virtio_ring.ko 2024-08-03T20:34:25.729 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/w1 2024-08-03T20:34:25.729 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/w1/masters 2024-08-03T20:34:25.729 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/w1/masters/ds2482.ko 2024-08-03T20:34:25.729 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/w1/masters/ds2490.ko 2024-08-03T20:34:25.729 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/w1/slaves 2024-08-03T20:34:25.729 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/w1/slaves/w1_ds2406.ko 2024-08-03T20:34:25.730 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/w1/slaves/w1_ds2408.ko 2024-08-03T20:34:25.730 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/w1/slaves/w1_ds2413.ko 2024-08-03T20:34:25.730 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/w1/slaves/w1_ds2423.ko 2024-08-03T20:34:25.730 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/w1/slaves/w1_ds2431.ko 2024-08-03T20:34:25.730 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/w1/slaves/w1_ds2433.ko 2024-08-03T20:34:25.730 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/w1/slaves/w1_ds2780.ko 2024-08-03T20:34:25.730 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/w1/slaves/w1_ds2781.ko 2024-08-03T20:34:25.730 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/w1/slaves/w1_ds28e04.ko 2024-08-03T20:34:25.730 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/w1/slaves/w1_smem.ko 2024-08-03T20:34:25.730 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/w1/slaves/w1_therm.ko 2024-08-03T20:34:25.730 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/w1/wire.ko 2024-08-03T20:34:25.730 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/watchdog 2024-08-03T20:34:25.730 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/watchdog/alim1535_wdt.ko 2024-08-03T20:34:25.731 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/watchdog/alim7101_wdt.ko 2024-08-03T20:34:25.731 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/watchdog/f71808e_wdt.ko 2024-08-03T20:34:25.731 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/watchdog/hpwdt.ko 2024-08-03T20:34:25.731 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/watchdog/i6300esb.ko 2024-08-03T20:34:25.731 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/watchdog/iTCO_vendor_support.ko 2024-08-03T20:34:25.731 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/watchdog/iTCO_wdt.ko 2024-08-03T20:34:25.731 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/watchdog/ib700wdt.ko 2024-08-03T20:34:25.731 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/watchdog/ibmasr.ko 2024-08-03T20:34:25.731 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/watchdog/ie6xx_wdt.ko 2024-08-03T20:34:25.731 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/watchdog/it8712f_wdt.ko 2024-08-03T20:34:25.731 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/watchdog/it87_wdt.ko 2024-08-03T20:34:25.731 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/watchdog/machzwd.ko 2024-08-03T20:34:25.731 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/watchdog/nv_tco.ko 2024-08-03T20:34:25.731 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/watchdog/pcwd_pci.ko 2024-08-03T20:34:25.732 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/watchdog/pcwd_usb.ko 2024-08-03T20:34:25.732 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/watchdog/sbc_fitpc2_wdt.ko 2024-08-03T20:34:25.732 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/watchdog/sch311x_wdt.ko 2024-08-03T20:34:25.732 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/watchdog/softdog.ko 2024-08-03T20:34:25.732 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/watchdog/sp5100_tco.ko 2024-08-03T20:34:25.732 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/watchdog/via_wdt.ko 2024-08-03T20:34:25.732 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/watchdog/w83627hf_wdt.ko 2024-08-03T20:34:25.732 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/watchdog/w83877f_wdt.ko 2024-08-03T20:34:25.732 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/watchdog/w83977f_wdt.ko 2024-08-03T20:34:25.732 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/watchdog/wdt_pci.ko 2024-08-03T20:34:25.732 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/watchdog/xen_wdt.ko 2024-08-03T20:34:25.732 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/xen 2024-08-03T20:34:25.732 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/xen/xen-acpi-processor.ko 2024-08-03T20:34:25.732 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/xen/xen-evtchn.ko 2024-08-03T20:34:25.733 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/xen/xen-gntalloc.ko 2024-08-03T20:34:25.733 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/xen/xen-gntdev.ko 2024-08-03T20:34:25.733 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/xen/xen-pciback 2024-08-03T20:34:25.733 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/xen/xen-pciback/xen-pciback.ko 2024-08-03T20:34:25.733 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/xen/xen-privcmd.ko 2024-08-03T20:34:25.733 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/xen/xenfs 2024-08-03T20:34:25.733 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/xen/xenfs/xenfs.ko 2024-08-03T20:34:25.733 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/fs 2024-08-03T20:34:25.733 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/fs/9p 2024-08-03T20:34:25.733 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/fs/9p/9p.ko 2024-08-03T20:34:25.733 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/fs/affs 2024-08-03T20:34:25.733 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/fs/affs/affs.ko 2024-08-03T20:34:25.733 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/fs/befs 2024-08-03T20:34:25.733 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/fs/befs/befs.ko 2024-08-03T20:34:25.734 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/fs/binfmt_misc.ko 2024-08-03T20:34:25.734 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/fs/btrfs 2024-08-03T20:34:25.734 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/fs/btrfs/btrfs.ko 2024-08-03T20:34:25.734 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/fs/cachefiles 2024-08-03T20:34:25.734 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/fs/cachefiles/cachefiles.ko 2024-08-03T20:34:25.734 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/fs/ceph 2024-08-03T20:34:25.734 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/fs/ceph/ceph.ko 2024-08-03T20:34:25.734 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/fs/coda 2024-08-03T20:34:25.734 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/fs/coda/coda.ko 2024-08-03T20:34:25.734 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/fs/cramfs 2024-08-03T20:34:25.734 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/fs/cramfs/cramfs.ko 2024-08-03T20:34:25.734 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/fs/dlm 2024-08-03T20:34:25.734 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/fs/dlm/dlm.ko 2024-08-03T20:34:25.734 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/fs/ecryptfs 2024-08-03T20:34:25.735 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/fs/ecryptfs/ecryptfs.ko 2024-08-03T20:34:25.735 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/fs/f2fs 2024-08-03T20:34:25.735 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/fs/f2fs/f2fs.ko 2024-08-03T20:34:25.735 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/fs/fat 2024-08-03T20:34:25.735 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/fs/fat/fat.ko 2024-08-03T20:34:25.735 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/fs/fat/msdos.ko 2024-08-03T20:34:25.735 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/fs/fat/vfat.ko 2024-08-03T20:34:25.735 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/fs/fuse 2024-08-03T20:34:25.735 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/fs/fuse/cuse.ko 2024-08-03T20:34:25.735 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/fs/fuse/fuse.ko 2024-08-03T20:34:25.735 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/fs/gfs2 2024-08-03T20:34:25.735 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/fs/gfs2/gfs2.ko 2024-08-03T20:34:25.735 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/fs/hfs 2024-08-03T20:34:25.735 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/fs/hfs/hfs.ko 2024-08-03T20:34:25.736 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/fs/hfsplus 2024-08-03T20:34:25.736 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/fs/hfsplus/hfsplus.ko 2024-08-03T20:34:25.736 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/fs/isofs 2024-08-03T20:34:25.736 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/fs/isofs/isofs.ko 2024-08-03T20:34:25.736 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/fs/jfs 2024-08-03T20:34:25.736 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/fs/jfs/jfs.ko 2024-08-03T20:34:25.736 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/fs/lockd 2024-08-03T20:34:25.736 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/fs/lockd/lockd.ko 2024-08-03T20:34:25.736 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/fs/minix 2024-08-03T20:34:25.736 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/fs/minix/minix.ko 2024-08-03T20:34:25.736 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/fs/netfs 2024-08-03T20:34:25.736 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/fs/netfs/netfs.ko 2024-08-03T20:34:25.736 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/fs/nfs 2024-08-03T20:34:25.736 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/fs/nfs/blocklayout 2024-08-03T20:34:25.736 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/fs/nfs/blocklayout/blocklayoutdriver.ko 2024-08-03T20:34:25.737 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/fs/nfs/filelayout 2024-08-03T20:34:25.737 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/fs/nfs/filelayout/nfs_layout_nfsv41_files.ko 2024-08-03T20:34:25.737 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/fs/nfs/flexfilelayout 2024-08-03T20:34:25.737 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/fs/nfs/flexfilelayout/nfs_layout_flexfiles.ko 2024-08-03T20:34:25.737 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/fs/nfs/nfs.ko 2024-08-03T20:34:25.737 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/fs/nfs/nfsv3.ko 2024-08-03T20:34:25.737 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/fs/nfs/nfsv4.ko 2024-08-03T20:34:25.737 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/fs/nfs_common 2024-08-03T20:34:25.737 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/fs/nfs_common/grace.ko 2024-08-03T20:34:25.737 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/fs/nfs_common/nfs_acl.ko 2024-08-03T20:34:25.737 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/fs/nfsd 2024-08-03T20:34:25.737 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/fs/nfsd/nfsd.ko 2024-08-03T20:34:25.737 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/fs/nilfs2 2024-08-03T20:34:25.737 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/fs/nilfs2/nilfs2.ko 2024-08-03T20:34:25.738 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/fs/nls 2024-08-03T20:34:25.738 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/fs/nls/mac-celtic.ko 2024-08-03T20:34:25.738 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/fs/nls/mac-centeuro.ko 2024-08-03T20:34:25.738 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/fs/nls/mac-croatian.ko 2024-08-03T20:34:25.738 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/fs/nls/mac-cyrillic.ko 2024-08-03T20:34:25.738 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/fs/nls/mac-gaelic.ko 2024-08-03T20:34:25.738 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/fs/nls/mac-greek.ko 2024-08-03T20:34:25.738 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/fs/nls/mac-iceland.ko 2024-08-03T20:34:25.738 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/fs/nls/mac-inuit.ko 2024-08-03T20:34:25.738 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/fs/nls/mac-roman.ko 2024-08-03T20:34:25.738 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/fs/nls/mac-romanian.ko 2024-08-03T20:34:25.738 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/fs/nls/mac-turkish.ko 2024-08-03T20:34:25.738 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/fs/nls/nls_cp1250.ko 2024-08-03T20:34:25.739 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/fs/nls/nls_cp1251.ko 2024-08-03T20:34:25.739 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/fs/nls/nls_cp1255.ko 2024-08-03T20:34:25.739 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/fs/nls/nls_cp737.ko 2024-08-03T20:34:25.739 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/fs/nls/nls_cp775.ko 2024-08-03T20:34:25.739 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/fs/nls/nls_cp850.ko 2024-08-03T20:34:25.739 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/fs/nls/nls_cp852.ko 2024-08-03T20:34:25.739 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/fs/nls/nls_cp855.ko 2024-08-03T20:34:25.739 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/fs/nls/nls_cp857.ko 2024-08-03T20:34:25.739 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/fs/nls/nls_cp860.ko 2024-08-03T20:34:25.739 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/fs/nls/nls_cp861.ko 2024-08-03T20:34:25.739 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/fs/nls/nls_cp862.ko 2024-08-03T20:34:25.739 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/fs/nls/nls_cp863.ko 2024-08-03T20:34:25.739 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/fs/nls/nls_cp864.ko 2024-08-03T20:34:25.739 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/fs/nls/nls_cp865.ko 2024-08-03T20:34:25.739 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/fs/nls/nls_cp866.ko 2024-08-03T20:34:25.740 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/fs/nls/nls_cp869.ko 2024-08-03T20:34:25.740 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/fs/nls/nls_cp874.ko 2024-08-03T20:34:25.740 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/fs/nls/nls_cp932.ko 2024-08-03T20:34:25.740 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/fs/nls/nls_cp936.ko 2024-08-03T20:34:25.740 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/fs/nls/nls_cp949.ko 2024-08-03T20:34:25.740 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/fs/nls/nls_cp950.ko 2024-08-03T20:34:25.740 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/fs/nls/nls_euc-jp.ko 2024-08-03T20:34:25.740 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/fs/nls/nls_iso8859-1.ko 2024-08-03T20:34:25.740 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/fs/nls/nls_iso8859-13.ko 2024-08-03T20:34:25.740 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/fs/nls/nls_iso8859-14.ko 2024-08-03T20:34:25.740 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/fs/nls/nls_iso8859-15.ko 2024-08-03T20:34:25.740 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/fs/nls/nls_iso8859-2.ko 2024-08-03T20:34:25.740 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/fs/nls/nls_iso8859-3.ko 2024-08-03T20:34:25.740 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/fs/nls/nls_iso8859-4.ko 2024-08-03T20:34:25.740 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/fs/nls/nls_iso8859-5.ko 2024-08-03T20:34:25.740 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/fs/nls/nls_iso8859-6.ko 2024-08-03T20:34:25.741 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/fs/nls/nls_iso8859-7.ko 2024-08-03T20:34:25.741 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/fs/nls/nls_iso8859-9.ko 2024-08-03T20:34:25.741 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/fs/nls/nls_koi8-r.ko 2024-08-03T20:34:25.741 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/fs/nls/nls_koi8-ru.ko 2024-08-03T20:34:25.741 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/fs/nls/nls_koi8-u.ko 2024-08-03T20:34:25.741 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/fs/nls/nls_ucs2_utils.ko 2024-08-03T20:34:25.741 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/fs/nls/nls_utf8.ko 2024-08-03T20:34:25.741 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/fs/ocfs2 2024-08-03T20:34:25.741 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/fs/ocfs2/cluster 2024-08-03T20:34:25.741 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/fs/ocfs2/cluster/ocfs2_nodemanager.ko 2024-08-03T20:34:25.741 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/fs/ocfs2/dlm 2024-08-03T20:34:25.741 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/fs/ocfs2/dlm/ocfs2_dlm.ko 2024-08-03T20:34:25.741 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/fs/ocfs2/dlmfs 2024-08-03T20:34:25.741 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/fs/ocfs2/dlmfs/ocfs2_dlmfs.ko 2024-08-03T20:34:25.742 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/fs/ocfs2/ocfs2.ko 2024-08-03T20:34:25.742 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/fs/ocfs2/ocfs2_stack_o2cb.ko 2024-08-03T20:34:25.742 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/fs/ocfs2/ocfs2_stack_user.ko 2024-08-03T20:34:25.742 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/fs/ocfs2/ocfs2_stackglue.ko 2024-08-03T20:34:25.742 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/fs/overlayfs 2024-08-03T20:34:25.742 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/fs/overlayfs/overlay.ko 2024-08-03T20:34:25.742 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/fs/pstore 2024-08-03T20:34:25.742 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/fs/pstore/ramoops.ko 2024-08-03T20:34:25.742 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/fs/reiserfs 2024-08-03T20:34:25.742 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/fs/reiserfs/reiserfs.ko 2024-08-03T20:34:25.742 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/fs/romfs 2024-08-03T20:34:25.742 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/fs/romfs/romfs.ko 2024-08-03T20:34:25.742 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/fs/smb 2024-08-03T20:34:25.742 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/fs/smb/client 2024-08-03T20:34:25.743 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/fs/smb/client/cifs.ko 2024-08-03T20:34:25.743 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/fs/smb/common 2024-08-03T20:34:25.743 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/fs/smb/common/cifs_arc4.ko 2024-08-03T20:34:25.743 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/fs/smb/common/cifs_md4.ko 2024-08-03T20:34:25.743 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/fs/squashfs 2024-08-03T20:34:25.743 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/fs/squashfs/squashfs.ko 2024-08-03T20:34:25.743 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/fs/sysv 2024-08-03T20:34:25.743 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/fs/sysv/sysv.ko 2024-08-03T20:34:25.743 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/fs/udf 2024-08-03T20:34:25.743 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/fs/udf/udf.ko 2024-08-03T20:34:25.743 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/fs/ufs 2024-08-03T20:34:25.743 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/fs/ufs/ufs.ko 2024-08-03T20:34:25.743 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/fs/xfs 2024-08-03T20:34:25.743 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/fs/xfs/xfs.ko 2024-08-03T20:34:25.743 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/kernel 2024-08-03T20:34:25.744 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/kernel/rcu 2024-08-03T20:34:25.744 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/kernel/rcu/rcutorture.ko 2024-08-03T20:34:25.744 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/kernel/torture.ko 2024-08-03T20:34:25.744 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/kernel/trace 2024-08-03T20:34:25.744 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/kernel/trace/ring_buffer_benchmark.ko 2024-08-03T20:34:25.744 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/lib 2024-08-03T20:34:25.744 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/lib/asn1_encoder.ko 2024-08-03T20:34:25.744 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/lib/crc-itu-t.ko 2024-08-03T20:34:25.744 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/lib/crc8.ko 2024-08-03T20:34:25.744 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/lib/crypto 2024-08-03T20:34:25.744 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/lib/crypto/libaescfb.ko 2024-08-03T20:34:25.744 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/lib/crypto/libarc4.ko 2024-08-03T20:34:25.744 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/lib/crypto/libdes.ko 2024-08-03T20:34:25.744 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/lib/libcrc32c.ko 2024-08-03T20:34:25.744 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/lib/lru_cache.ko 2024-08-03T20:34:25.745 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/lib/lz4 2024-08-03T20:34:25.745 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/lib/lz4/lz4_compress.ko 2024-08-03T20:34:25.745 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/lib/lz4/lz4hc_compress.ko 2024-08-03T20:34:25.745 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/lib/math 2024-08-03T20:34:25.745 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/lib/math/cordic.ko 2024-08-03T20:34:25.745 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/lib/raid6 2024-08-03T20:34:25.746 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/lib/raid6/raid6_pq.ko 2024-08-03T20:34:25.746 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/lib/reed_solomon 2024-08-03T20:34:25.746 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/lib/reed_solomon/reed_solomon.ko 2024-08-03T20:34:25.746 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/lib/ts_bm.ko 2024-08-03T20:34:25.746 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/lib/ts_fsm.ko 2024-08-03T20:34:25.746 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/lib/ts_kmp.ko 2024-08-03T20:34:25.747 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/lib/zstd 2024-08-03T20:34:25.747 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/lib/zstd/zstd_compress.ko 2024-08-03T20:34:25.747 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/mm 2024-08-03T20:34:25.747 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/mm/hwpoison-inject.ko 2024-08-03T20:34:25.747 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net 2024-08-03T20:34:25.747 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/6lowpan 2024-08-03T20:34:25.747 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/6lowpan/6lowpan.ko 2024-08-03T20:34:25.747 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/6lowpan/nhc_dest.ko 2024-08-03T20:34:25.747 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/6lowpan/nhc_fragment.ko 2024-08-03T20:34:25.747 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/6lowpan/nhc_hop.ko 2024-08-03T20:34:25.747 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/6lowpan/nhc_ipv6.ko 2024-08-03T20:34:25.747 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/6lowpan/nhc_mobility.ko 2024-08-03T20:34:25.747 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/6lowpan/nhc_routing.ko 2024-08-03T20:34:25.748 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/6lowpan/nhc_udp.ko 2024-08-03T20:34:25.748 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/802 2024-08-03T20:34:25.748 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/802/garp.ko 2024-08-03T20:34:25.748 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/802/mrp.ko 2024-08-03T20:34:25.748 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/802/p8022.ko 2024-08-03T20:34:25.748 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/802/psnap.ko 2024-08-03T20:34:25.749 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/802/stp.ko 2024-08-03T20:34:25.749 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/8021q 2024-08-03T20:34:25.749 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/8021q/8021q.ko 2024-08-03T20:34:25.749 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/9p 2024-08-03T20:34:25.749 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/9p/9pnet.ko 2024-08-03T20:34:25.749 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/9p/9pnet_fd.ko 2024-08-03T20:34:25.749 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/9p/9pnet_rdma.ko 2024-08-03T20:34:25.749 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/9p/9pnet_virtio.ko 2024-08-03T20:34:25.749 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/batman-adv 2024-08-03T20:34:25.749 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/batman-adv/batman-adv.ko 2024-08-03T20:34:25.749 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/bridge 2024-08-03T20:34:25.749 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/bridge/br_netfilter.ko 2024-08-03T20:34:25.749 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/bridge/bridge.ko 2024-08-03T20:34:25.750 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/bridge/netfilter 2024-08-03T20:34:25.750 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/bridge/netfilter/ebt_802_3.ko 2024-08-03T20:34:25.750 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/bridge/netfilter/ebt_among.ko 2024-08-03T20:34:25.750 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/bridge/netfilter/ebt_arp.ko 2024-08-03T20:34:25.750 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/bridge/netfilter/ebt_arpreply.ko 2024-08-03T20:34:25.750 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/bridge/netfilter/ebt_dnat.ko 2024-08-03T20:34:25.750 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/bridge/netfilter/ebt_ip.ko 2024-08-03T20:34:25.750 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/bridge/netfilter/ebt_ip6.ko 2024-08-03T20:34:25.750 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/bridge/netfilter/ebt_limit.ko 2024-08-03T20:34:25.750 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/bridge/netfilter/ebt_log.ko 2024-08-03T20:34:25.750 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/bridge/netfilter/ebt_mark.ko 2024-08-03T20:34:25.750 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/bridge/netfilter/ebt_mark_m.ko 2024-08-03T20:34:25.751 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/bridge/netfilter/ebt_nflog.ko 2024-08-03T20:34:25.751 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/bridge/netfilter/ebt_pkttype.ko 2024-08-03T20:34:25.751 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/bridge/netfilter/ebt_redirect.ko 2024-08-03T20:34:25.751 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/bridge/netfilter/ebt_snat.ko 2024-08-03T20:34:25.751 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/bridge/netfilter/ebt_stp.ko 2024-08-03T20:34:25.751 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/bridge/netfilter/ebt_vlan.ko 2024-08-03T20:34:25.751 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/bridge/netfilter/ebtable_broute.ko 2024-08-03T20:34:25.751 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/bridge/netfilter/ebtable_filter.ko 2024-08-03T20:34:25.751 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/bridge/netfilter/ebtable_nat.ko 2024-08-03T20:34:25.751 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/bridge/netfilter/ebtables.ko 2024-08-03T20:34:25.751 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/bridge/netfilter/nf_conntrack_bridge.ko 2024-08-03T20:34:25.751 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/bridge/netfilter/nft_meta_bridge.ko 2024-08-03T20:34:25.751 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/bridge/netfilter/nft_reject_bridge.ko 2024-08-03T20:34:25.752 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/ceph 2024-08-03T20:34:25.752 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/ceph/libceph.ko 2024-08-03T20:34:25.752 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/core 2024-08-03T20:34:25.752 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/core/failover.ko 2024-08-03T20:34:25.752 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/core/pktgen.ko 2024-08-03T20:34:25.752 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/dccp 2024-08-03T20:34:25.752 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/dccp/dccp.ko 2024-08-03T20:34:25.752 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/dccp/dccp_diag.ko 2024-08-03T20:34:25.752 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/dccp/dccp_ipv4.ko 2024-08-03T20:34:25.752 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/dccp/dccp_ipv6.ko 2024-08-03T20:34:25.752 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/dns_resolver 2024-08-03T20:34:25.752 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/dns_resolver/dns_resolver.ko 2024-08-03T20:34:25.752 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/dsa 2024-08-03T20:34:25.752 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/dsa/dsa_core.ko 2024-08-03T20:34:25.753 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/dsa/tag_brcm.ko 2024-08-03T20:34:25.753 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/dsa/tag_dsa.ko 2024-08-03T20:34:25.753 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/dsa/tag_none.ko 2024-08-03T20:34:25.753 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/dsa/tag_trailer.ko 2024-08-03T20:34:25.753 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/ieee802154 2024-08-03T20:34:25.753 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/ieee802154/6lowpan 2024-08-03T20:34:25.753 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/ieee802154/6lowpan/ieee802154_6lowpan.ko 2024-08-03T20:34:25.753 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/ieee802154/ieee802154.ko 2024-08-03T20:34:25.753 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/ieee802154/ieee802154_socket.ko 2024-08-03T20:34:25.753 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/ipv4 2024-08-03T20:34:25.753 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/ipv4/ah4.ko 2024-08-03T20:34:25.753 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/ipv4/esp4.ko 2024-08-03T20:34:25.753 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/ipv4/fou.ko 2024-08-03T20:34:25.753 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/ipv4/gre.ko 2024-08-03T20:34:25.754 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/ipv4/inet_diag.ko 2024-08-03T20:34:25.754 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/ipv4/ip_gre.ko 2024-08-03T20:34:25.754 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/ipv4/ip_tunnel.ko 2024-08-03T20:34:25.754 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/ipv4/ip_vti.ko 2024-08-03T20:34:25.754 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/ipv4/ipcomp.ko 2024-08-03T20:34:25.754 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/ipv4/ipip.ko 2024-08-03T20:34:25.754 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/ipv4/netfilter 2024-08-03T20:34:25.754 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/ipv4/netfilter/arp_tables.ko 2024-08-03T20:34:25.754 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/ipv4/netfilter/arpt_mangle.ko 2024-08-03T20:34:25.754 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/ipv4/netfilter/arptable_filter.ko 2024-08-03T20:34:25.754 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/ipv4/netfilter/ipt_ECN.ko 2024-08-03T20:34:25.754 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/ipv4/netfilter/ipt_SYNPROXY.ko 2024-08-03T20:34:25.754 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/ipv4/netfilter/ipt_ah.ko 2024-08-03T20:34:25.755 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/ipv4/netfilter/ipt_rpfilter.ko 2024-08-03T20:34:25.755 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/ipv4/netfilter/iptable_mangle.ko 2024-08-03T20:34:25.755 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/ipv4/netfilter/iptable_nat.ko 2024-08-03T20:34:25.755 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/ipv4/netfilter/iptable_raw.ko 2024-08-03T20:34:25.755 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/ipv4/netfilter/iptable_security.ko 2024-08-03T20:34:25.755 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/ipv4/netfilter/nf_defrag_ipv4.ko 2024-08-03T20:34:25.755 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/ipv4/netfilter/nf_dup_ipv4.ko 2024-08-03T20:34:25.755 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/ipv4/netfilter/nf_nat_h323.ko 2024-08-03T20:34:25.755 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/ipv4/netfilter/nf_nat_pptp.ko 2024-08-03T20:34:25.755 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/ipv4/netfilter/nf_nat_snmp_basic.ko 2024-08-03T20:34:25.755 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/ipv4/netfilter/nf_socket_ipv4.ko 2024-08-03T20:34:25.755 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/ipv4/netfilter/nf_tproxy_ipv4.ko 2024-08-03T20:34:25.755 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/ipv4/netfilter/nft_dup_ipv4.ko 2024-08-03T20:34:25.755 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/ipv4/netfilter/nft_fib_ipv4.ko 2024-08-03T20:34:25.756 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/ipv4/netfilter/nft_reject_ipv4.ko 2024-08-03T20:34:25.756 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/ipv4/tcp_bic.ko 2024-08-03T20:34:25.756 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/ipv4/tcp_dctcp.ko 2024-08-03T20:34:25.756 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/ipv4/tcp_diag.ko 2024-08-03T20:34:25.756 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/ipv4/tcp_highspeed.ko 2024-08-03T20:34:25.756 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/ipv4/tcp_htcp.ko 2024-08-03T20:34:25.756 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/ipv4/tcp_hybla.ko 2024-08-03T20:34:25.756 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/ipv4/tcp_illinois.ko 2024-08-03T20:34:25.756 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/ipv4/tcp_lp.ko 2024-08-03T20:34:25.756 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/ipv4/tcp_scalable.ko 2024-08-03T20:34:25.756 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/ipv4/tcp_vegas.ko 2024-08-03T20:34:25.756 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/ipv4/tcp_veno.ko 2024-08-03T20:34:25.756 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/ipv4/tcp_westwood.ko 2024-08-03T20:34:25.756 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/ipv4/tcp_yeah.ko 2024-08-03T20:34:25.757 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/ipv4/tunnel4.ko 2024-08-03T20:34:25.757 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/ipv4/udp_diag.ko 2024-08-03T20:34:25.757 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/ipv4/udp_tunnel.ko 2024-08-03T20:34:25.757 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/ipv4/xfrm4_tunnel.ko 2024-08-03T20:34:25.757 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/ipv6 2024-08-03T20:34:25.757 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/ipv6/ah6.ko 2024-08-03T20:34:25.757 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/ipv6/esp6.ko 2024-08-03T20:34:25.757 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/ipv6/fou6.ko 2024-08-03T20:34:25.757 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/ipv6/ip6_tunnel.ko 2024-08-03T20:34:25.757 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/ipv6/ip6_udp_tunnel.ko 2024-08-03T20:34:25.757 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/ipv6/ip6_vti.ko 2024-08-03T20:34:25.757 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/ipv6/ipcomp6.ko 2024-08-03T20:34:25.757 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/ipv6/netfilter 2024-08-03T20:34:25.757 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/ipv6/netfilter/ip6_tables.ko 2024-08-03T20:34:25.758 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/ipv6/netfilter/ip6t_REJECT.ko 2024-08-03T20:34:25.758 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/ipv6/netfilter/ip6t_SYNPROXY.ko 2024-08-03T20:34:25.758 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/ipv6/netfilter/ip6t_ah.ko 2024-08-03T20:34:25.758 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/ipv6/netfilter/ip6t_eui64.ko 2024-08-03T20:34:25.758 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/ipv6/netfilter/ip6t_frag.ko 2024-08-03T20:34:25.758 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/ipv6/netfilter/ip6t_hbh.ko 2024-08-03T20:34:25.758 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/ipv6/netfilter/ip6t_ipv6header.ko 2024-08-03T20:34:25.758 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/ipv6/netfilter/ip6t_mh.ko 2024-08-03T20:34:25.758 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/ipv6/netfilter/ip6t_rpfilter.ko 2024-08-03T20:34:25.758 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/ipv6/netfilter/ip6t_rt.ko 2024-08-03T20:34:25.758 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/ipv6/netfilter/ip6table_filter.ko 2024-08-03T20:34:25.758 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/ipv6/netfilter/ip6table_mangle.ko 2024-08-03T20:34:25.758 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/ipv6/netfilter/ip6table_nat.ko 2024-08-03T20:34:25.758 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/ipv6/netfilter/ip6table_raw.ko 2024-08-03T20:34:25.759 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/ipv6/netfilter/ip6table_security.ko 2024-08-03T20:34:25.759 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/ipv6/netfilter/nf_defrag_ipv6.ko 2024-08-03T20:34:25.759 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/ipv6/netfilter/nf_dup_ipv6.ko 2024-08-03T20:34:25.759 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/ipv6/netfilter/nf_reject_ipv6.ko 2024-08-03T20:34:25.759 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/ipv6/netfilter/nf_socket_ipv6.ko 2024-08-03T20:34:25.759 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/ipv6/netfilter/nf_tproxy_ipv6.ko 2024-08-03T20:34:25.759 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/ipv6/netfilter/nft_dup_ipv6.ko 2024-08-03T20:34:25.759 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/ipv6/netfilter/nft_fib_ipv6.ko 2024-08-03T20:34:25.759 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/ipv6/netfilter/nft_reject_ipv6.ko 2024-08-03T20:34:25.759 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/ipv6/sit.ko 2024-08-03T20:34:25.759 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/ipv6/tunnel6.ko 2024-08-03T20:34:25.759 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/ipv6/xfrm6_tunnel.ko 2024-08-03T20:34:25.759 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/key 2024-08-03T20:34:25.760 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/key/af_key.ko 2024-08-03T20:34:25.760 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/l2tp 2024-08-03T20:34:25.760 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/l2tp/l2tp_core.ko 2024-08-03T20:34:25.760 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/l2tp/l2tp_debugfs.ko 2024-08-03T20:34:25.760 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/l2tp/l2tp_eth.ko 2024-08-03T20:34:25.760 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/l2tp/l2tp_ip.ko 2024-08-03T20:34:25.760 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/l2tp/l2tp_ip6.ko 2024-08-03T20:34:25.760 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/l2tp/l2tp_netlink.ko 2024-08-03T20:34:25.760 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/l2tp/l2tp_ppp.ko 2024-08-03T20:34:25.760 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/llc 2024-08-03T20:34:25.760 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/llc/llc.ko 2024-08-03T20:34:25.760 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/mac802154 2024-08-03T20:34:25.760 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/mac802154/mac802154.ko 2024-08-03T20:34:25.760 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/mpls 2024-08-03T20:34:25.760 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/mpls/mpls_gso.ko 2024-08-03T20:34:25.761 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/netfilter 2024-08-03T20:34:25.761 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/netfilter/ipset 2024-08-03T20:34:25.761 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/netfilter/ipset/ip_set.ko 2024-08-03T20:34:25.761 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/netfilter/ipset/ip_set_bitmap_ip.ko 2024-08-03T20:34:25.761 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/netfilter/ipset/ip_set_bitmap_ipmac.ko 2024-08-03T20:34:25.761 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/netfilter/ipset/ip_set_bitmap_port.ko 2024-08-03T20:34:25.761 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/netfilter/ipset/ip_set_hash_ip.ko 2024-08-03T20:34:25.761 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/netfilter/ipset/ip_set_hash_ipmark.ko 2024-08-03T20:34:25.761 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/netfilter/ipset/ip_set_hash_ipport.ko 2024-08-03T20:34:25.761 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/netfilter/ipset/ip_set_hash_ipportip.ko 2024-08-03T20:34:25.761 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/netfilter/ipset/ip_set_hash_ipportnet.ko 2024-08-03T20:34:25.762 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/netfilter/ipset/ip_set_hash_mac.ko 2024-08-03T20:34:25.762 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/netfilter/ipset/ip_set_hash_net.ko 2024-08-03T20:34:25.762 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/netfilter/ipset/ip_set_hash_netiface.ko 2024-08-03T20:34:25.762 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/netfilter/ipset/ip_set_hash_netnet.ko 2024-08-03T20:34:25.762 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/netfilter/ipset/ip_set_hash_netport.ko 2024-08-03T20:34:25.762 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/netfilter/ipset/ip_set_hash_netportnet.ko 2024-08-03T20:34:25.762 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/netfilter/ipset/ip_set_list_set.ko 2024-08-03T20:34:25.762 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/netfilter/ipvs 2024-08-03T20:34:25.762 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/netfilter/ipvs/ip_vs.ko 2024-08-03T20:34:25.762 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/netfilter/ipvs/ip_vs_dh.ko 2024-08-03T20:34:25.762 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/netfilter/ipvs/ip_vs_fo.ko 2024-08-03T20:34:25.762 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/netfilter/ipvs/ip_vs_ftp.ko 2024-08-03T20:34:25.763 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/netfilter/ipvs/ip_vs_lblc.ko 2024-08-03T20:34:25.763 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/netfilter/ipvs/ip_vs_lblcr.ko 2024-08-03T20:34:25.763 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/netfilter/ipvs/ip_vs_lc.ko 2024-08-03T20:34:25.763 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/netfilter/ipvs/ip_vs_nq.ko 2024-08-03T20:34:25.763 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/netfilter/ipvs/ip_vs_pe_sip.ko 2024-08-03T20:34:25.763 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/netfilter/ipvs/ip_vs_rr.ko 2024-08-03T20:34:25.763 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/netfilter/ipvs/ip_vs_sed.ko 2024-08-03T20:34:25.763 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/netfilter/ipvs/ip_vs_sh.ko 2024-08-03T20:34:25.763 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/netfilter/ipvs/ip_vs_wlc.ko 2024-08-03T20:34:25.764 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/netfilter/ipvs/ip_vs_wrr.ko 2024-08-03T20:34:25.764 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/netfilter/nf_conncount.ko 2024-08-03T20:34:25.764 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/netfilter/nf_conntrack.ko 2024-08-03T20:34:25.764 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/netfilter/nf_conntrack_amanda.ko 2024-08-03T20:34:25.764 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/netfilter/nf_conntrack_broadcast.ko 2024-08-03T20:34:25.764 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/netfilter/nf_conntrack_ftp.ko 2024-08-03T20:34:25.764 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/netfilter/nf_conntrack_h323.ko 2024-08-03T20:34:25.764 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/netfilter/nf_conntrack_irc.ko 2024-08-03T20:34:25.765 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/netfilter/nf_conntrack_netbios_ns.ko 2024-08-03T20:34:25.765 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/netfilter/nf_conntrack_netlink.ko 2024-08-03T20:34:25.765 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/netfilter/nf_conntrack_pptp.ko 2024-08-03T20:34:25.765 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/netfilter/nf_conntrack_sane.ko 2024-08-03T20:34:25.765 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/netfilter/nf_conntrack_sip.ko 2024-08-03T20:34:25.765 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/netfilter/nf_conntrack_snmp.ko 2024-08-03T20:34:25.765 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/netfilter/nf_conntrack_tftp.ko 2024-08-03T20:34:25.765 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/netfilter/nf_dup_netdev.ko 2024-08-03T20:34:25.765 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/netfilter/nf_flow_table.ko 2024-08-03T20:34:25.765 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/netfilter/nf_flow_table_inet.ko 2024-08-03T20:34:25.765 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/netfilter/nf_log_syslog.ko 2024-08-03T20:34:25.765 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/netfilter/nf_nat.ko 2024-08-03T20:34:25.765 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/netfilter/nf_nat_amanda.ko 2024-08-03T20:34:25.765 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/netfilter/nf_nat_ftp.ko 2024-08-03T20:34:25.766 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/netfilter/nf_nat_irc.ko 2024-08-03T20:34:25.766 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/netfilter/nf_nat_sip.ko 2024-08-03T20:34:25.766 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/netfilter/nf_nat_tftp.ko 2024-08-03T20:34:25.766 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/netfilter/nf_synproxy_core.ko 2024-08-03T20:34:25.766 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/netfilter/nf_tables.ko 2024-08-03T20:34:25.766 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/netfilter/nfnetlink.ko 2024-08-03T20:34:25.766 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/netfilter/nfnetlink_acct.ko 2024-08-03T20:34:25.766 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/netfilter/nfnetlink_log.ko 2024-08-03T20:34:25.766 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/netfilter/nfnetlink_osf.ko 2024-08-03T20:34:25.766 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/netfilter/nfnetlink_queue.ko 2024-08-03T20:34:25.766 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/netfilter/nft_chain_nat.ko 2024-08-03T20:34:25.766 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/netfilter/nft_compat.ko 2024-08-03T20:34:25.766 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/netfilter/nft_ct.ko 2024-08-03T20:34:25.766 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/netfilter/nft_dup_netdev.ko 2024-08-03T20:34:25.767 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/netfilter/nft_fib.ko 2024-08-03T20:34:25.767 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/netfilter/nft_fwd_netdev.ko 2024-08-03T20:34:25.767 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/netfilter/nft_hash.ko 2024-08-03T20:34:25.767 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/netfilter/nft_limit.ko 2024-08-03T20:34:25.767 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/netfilter/nft_log.ko 2024-08-03T20:34:25.767 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/netfilter/nft_masq.ko 2024-08-03T20:34:25.767 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/netfilter/nft_nat.ko 2024-08-03T20:34:25.767 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/netfilter/nft_numgen.ko 2024-08-03T20:34:25.767 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/netfilter/nft_queue.ko 2024-08-03T20:34:25.767 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/netfilter/nft_quota.ko 2024-08-03T20:34:25.767 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/netfilter/nft_redir.ko 2024-08-03T20:34:25.767 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/netfilter/nft_reject.ko 2024-08-03T20:34:25.767 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/netfilter/nft_reject_inet.ko 2024-08-03T20:34:25.767 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/netfilter/nft_socket.ko 2024-08-03T20:34:25.768 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/netfilter/nft_tunnel.ko 2024-08-03T20:34:25.768 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/netfilter/xt_AUDIT.ko 2024-08-03T20:34:25.768 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/netfilter/xt_CHECKSUM.ko 2024-08-03T20:34:25.768 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/netfilter/xt_CLASSIFY.ko 2024-08-03T20:34:25.768 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/netfilter/xt_CONNSECMARK.ko 2024-08-03T20:34:25.768 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/netfilter/xt_CT.ko 2024-08-03T20:34:25.768 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/netfilter/xt_DSCP.ko 2024-08-03T20:34:25.768 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/netfilter/xt_HL.ko 2024-08-03T20:34:25.768 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/netfilter/xt_HMARK.ko 2024-08-03T20:34:25.768 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/netfilter/xt_IDLETIMER.ko 2024-08-03T20:34:25.768 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/netfilter/xt_LED.ko 2024-08-03T20:34:25.768 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/netfilter/xt_LOG.ko 2024-08-03T20:34:25.768 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/netfilter/xt_MASQUERADE.ko 2024-08-03T20:34:25.769 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/netfilter/xt_NETMAP.ko 2024-08-03T20:34:25.769 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/netfilter/xt_NFLOG.ko 2024-08-03T20:34:25.769 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/netfilter/xt_NFQUEUE.ko 2024-08-03T20:34:25.769 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/netfilter/xt_RATEEST.ko 2024-08-03T20:34:25.769 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/netfilter/xt_REDIRECT.ko 2024-08-03T20:34:25.769 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/netfilter/xt_SECMARK.ko 2024-08-03T20:34:25.769 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/netfilter/xt_TCPMSS.ko 2024-08-03T20:34:25.769 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/netfilter/xt_TCPOPTSTRIP.ko 2024-08-03T20:34:25.769 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/netfilter/xt_TEE.ko 2024-08-03T20:34:25.769 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/netfilter/xt_TPROXY.ko 2024-08-03T20:34:25.769 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/netfilter/xt_TRACE.ko 2024-08-03T20:34:25.769 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/netfilter/xt_addrtype.ko 2024-08-03T20:34:25.769 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/netfilter/xt_bpf.ko 2024-08-03T20:34:25.769 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/netfilter/xt_cgroup.ko 2024-08-03T20:34:25.769 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/netfilter/xt_cluster.ko 2024-08-03T20:34:25.770 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/netfilter/xt_comment.ko 2024-08-03T20:34:25.770 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/netfilter/xt_connbytes.ko 2024-08-03T20:34:25.770 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/netfilter/xt_connlabel.ko 2024-08-03T20:34:25.770 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/netfilter/xt_connlimit.ko 2024-08-03T20:34:25.770 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/netfilter/xt_connmark.ko 2024-08-03T20:34:25.770 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/netfilter/xt_conntrack.ko 2024-08-03T20:34:25.770 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/netfilter/xt_cpu.ko 2024-08-03T20:34:25.770 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/netfilter/xt_dccp.ko 2024-08-03T20:34:25.770 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/netfilter/xt_devgroup.ko 2024-08-03T20:34:25.770 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/netfilter/xt_dscp.ko 2024-08-03T20:34:25.770 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/netfilter/xt_ecn.ko 2024-08-03T20:34:25.770 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/netfilter/xt_esp.ko 2024-08-03T20:34:25.770 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/netfilter/xt_hashlimit.ko 2024-08-03T20:34:25.771 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/netfilter/xt_helper.ko 2024-08-03T20:34:25.771 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/netfilter/xt_hl.ko 2024-08-03T20:34:25.771 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/netfilter/xt_ipcomp.ko 2024-08-03T20:34:25.771 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/netfilter/xt_iprange.ko 2024-08-03T20:34:25.771 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/netfilter/xt_ipvs.ko 2024-08-03T20:34:25.771 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/netfilter/xt_l2tp.ko 2024-08-03T20:34:25.771 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/netfilter/xt_length.ko 2024-08-03T20:34:25.771 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/netfilter/xt_limit.ko 2024-08-03T20:34:25.771 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/netfilter/xt_mac.ko 2024-08-03T20:34:25.771 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/netfilter/xt_mark.ko 2024-08-03T20:34:25.771 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/netfilter/xt_multiport.ko 2024-08-03T20:34:25.771 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/netfilter/xt_nat.ko 2024-08-03T20:34:25.771 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/netfilter/xt_nfacct.ko 2024-08-03T20:34:25.772 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/netfilter/xt_osf.ko 2024-08-03T20:34:25.772 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/netfilter/xt_owner.ko 2024-08-03T20:34:25.772 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/netfilter/xt_physdev.ko 2024-08-03T20:34:25.772 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/netfilter/xt_pkttype.ko 2024-08-03T20:34:25.772 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/netfilter/xt_policy.ko 2024-08-03T20:34:25.772 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/netfilter/xt_quota.ko 2024-08-03T20:34:25.772 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/netfilter/xt_rateest.ko 2024-08-03T20:34:25.772 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/netfilter/xt_realm.ko 2024-08-03T20:34:25.772 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/netfilter/xt_recent.ko 2024-08-03T20:34:25.772 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/netfilter/xt_sctp.ko 2024-08-03T20:34:25.772 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/netfilter/xt_set.ko 2024-08-03T20:34:25.772 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/netfilter/xt_socket.ko 2024-08-03T20:34:25.772 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/netfilter/xt_state.ko 2024-08-03T20:34:25.772 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/netfilter/xt_statistic.ko 2024-08-03T20:34:25.773 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/netfilter/xt_string.ko 2024-08-03T20:34:25.773 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/netfilter/xt_tcpmss.ko 2024-08-03T20:34:25.773 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/netfilter/xt_time.ko 2024-08-03T20:34:25.773 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/netfilter/xt_u32.ko 2024-08-03T20:34:25.773 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/netlink 2024-08-03T20:34:25.773 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/netlink/netlink_diag.ko 2024-08-03T20:34:25.773 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/nsh 2024-08-03T20:34:25.773 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/nsh/nsh.ko 2024-08-03T20:34:25.773 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/openvswitch 2024-08-03T20:34:25.773 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/openvswitch/openvswitch.ko 2024-08-03T20:34:25.773 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/openvswitch/vport-geneve.ko 2024-08-03T20:34:25.773 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/openvswitch/vport-gre.ko 2024-08-03T20:34:25.773 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/openvswitch/vport-vxlan.ko 2024-08-03T20:34:25.774 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/packet 2024-08-03T20:34:25.774 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/packet/af_packet_diag.ko 2024-08-03T20:34:25.774 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/rds 2024-08-03T20:34:25.774 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/rds/rds.ko 2024-08-03T20:34:25.774 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/rds/rds_rdma.ko 2024-08-03T20:34:25.774 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/rds/rds_tcp.ko 2024-08-03T20:34:25.774 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/sched 2024-08-03T20:34:25.774 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/sched/act_csum.ko 2024-08-03T20:34:25.774 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/sched/act_gact.ko 2024-08-03T20:34:25.774 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/sched/act_mirred.ko 2024-08-03T20:34:25.774 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/sched/act_nat.ko 2024-08-03T20:34:25.774 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/sched/act_pedit.ko 2024-08-03T20:34:25.774 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/sched/act_police.ko 2024-08-03T20:34:25.777 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/sched/act_simple.ko 2024-08-03T20:34:25.777 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/sched/act_skbedit.ko 2024-08-03T20:34:25.777 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/sched/act_vlan.ko 2024-08-03T20:34:25.778 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/sched/cls_basic.ko 2024-08-03T20:34:25.778 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/sched/cls_bpf.ko 2024-08-03T20:34:25.778 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/sched/cls_flow.ko 2024-08-03T20:34:25.778 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/sched/cls_fw.ko 2024-08-03T20:34:25.778 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/sched/cls_route.ko 2024-08-03T20:34:25.778 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/sched/cls_u32.ko 2024-08-03T20:34:25.778 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/sched/em_cmp.ko 2024-08-03T20:34:25.778 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/sched/em_ipset.ko 2024-08-03T20:34:25.778 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/sched/em_meta.ko 2024-08-03T20:34:25.778 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/sched/em_nbyte.ko 2024-08-03T20:34:25.779 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/sched/em_text.ko 2024-08-03T20:34:25.779 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/sched/em_u32.ko 2024-08-03T20:34:25.779 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/sched/sch_choke.ko 2024-08-03T20:34:25.779 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/sched/sch_codel.ko 2024-08-03T20:34:25.779 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/sched/sch_drr.ko 2024-08-03T20:34:25.779 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/sched/sch_fq.ko 2024-08-03T20:34:25.779 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/sched/sch_gred.ko 2024-08-03T20:34:25.779 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/sched/sch_hfsc.ko 2024-08-03T20:34:25.779 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/sched/sch_hhf.ko 2024-08-03T20:34:25.779 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/sched/sch_htb.ko 2024-08-03T20:34:25.779 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/sched/sch_ingress.ko 2024-08-03T20:34:25.779 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/sched/sch_mqprio.ko 2024-08-03T20:34:25.779 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/sched/sch_mqprio_lib.ko 2024-08-03T20:34:25.780 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/sched/sch_multiq.ko 2024-08-03T20:34:25.780 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/sched/sch_netem.ko 2024-08-03T20:34:25.780 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/sched/sch_pie.ko 2024-08-03T20:34:25.780 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/sched/sch_plug.ko 2024-08-03T20:34:25.780 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/sched/sch_prio.ko 2024-08-03T20:34:25.780 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/sched/sch_qfq.ko 2024-08-03T20:34:25.780 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/sched/sch_red.ko 2024-08-03T20:34:25.780 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/sched/sch_sfb.ko 2024-08-03T20:34:25.780 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/sched/sch_sfq.ko 2024-08-03T20:34:25.780 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/sched/sch_tbf.ko 2024-08-03T20:34:25.780 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/sched/sch_teql.ko 2024-08-03T20:34:25.780 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/sctp 2024-08-03T20:34:25.780 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/sctp/sctp.ko 2024-08-03T20:34:25.780 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/sctp/sctp_diag.ko 2024-08-03T20:34:25.781 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/sunrpc 2024-08-03T20:34:25.781 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/sunrpc/auth_gss 2024-08-03T20:34:25.781 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/sunrpc/auth_gss/auth_rpcgss.ko 2024-08-03T20:34:25.781 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/sunrpc/auth_gss/rpcsec_gss_krb5.ko 2024-08-03T20:34:25.781 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/sunrpc/sunrpc.ko 2024-08-03T20:34:25.781 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/sunrpc/xprtrdma 2024-08-03T20:34:25.781 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/sunrpc/xprtrdma/rpcrdma.ko 2024-08-03T20:34:25.781 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/tipc 2024-08-03T20:34:25.781 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/tipc/tipc.ko 2024-08-03T20:34:25.781 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/tipc/tipc_diag.ko 2024-08-03T20:34:25.781 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/unix 2024-08-03T20:34:25.781 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/unix/unix_diag.ko 2024-08-03T20:34:25.781 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/vmw_vsock 2024-08-03T20:34:25.782 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/vmw_vsock/vmw_vsock_virtio_transport_common.ko 2024-08-03T20:34:25.782 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/vmw_vsock/vmw_vsock_vmci_transport.ko 2024-08-03T20:34:25.782 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/vmw_vsock/vsock.ko 2024-08-03T20:34:25.782 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/vmw_vsock/vsock_diag.ko 2024-08-03T20:34:25.782 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/vmw_vsock/vsock_loopback.ko 2024-08-03T20:34:25.782 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/xfrm 2024-08-03T20:34:25.782 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/xfrm/xfrm_ipcomp.ko 2024-08-03T20:34:25.782 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/security 2024-08-03T20:34:25.782 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/security/keys 2024-08-03T20:34:25.782 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/security/keys/encrypted-keys 2024-08-03T20:34:25.782 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/security/keys/encrypted-keys/encrypted-keys.ko 2024-08-03T20:34:25.782 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/security/keys/trusted-keys 2024-08-03T20:34:25.782 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/security/keys/trusted-keys/trusted.ko 2024-08-03T20:34:25.782 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/modules.alias 2024-08-03T20:34:25.783 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/modules.alias.bin 2024-08-03T20:34:25.783 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/modules.builtin 2024-08-03T20:34:25.783 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/modules.builtin.alias.bin 2024-08-03T20:34:25.783 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/modules.builtin.bin 2024-08-03T20:34:25.783 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/modules.builtin.modinfo 2024-08-03T20:34:25.783 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/modules.dep 2024-08-03T20:34:25.783 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/modules.dep.bin 2024-08-03T20:34:25.783 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/modules.devname 2024-08-03T20:34:25.783 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/modules.order 2024-08-03T20:34:25.783 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/modules.softdep 2024-08-03T20:34:25.783 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/modules.symbols 2024-08-03T20:34:25.783 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/modules.symbols.bin 2024-08-03T20:34:25.783 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/modules.weakdep 2024-08-03T20:34:25.783 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/vmlinuz 2024-08-03T20:34:25.788 DEBUG:teuthology.task.kernel:get_image_version: 6.11.0-rc1-gef231aaae597 2024-08-03T20:34:25.788 DEBUG:teuthology.orchestra.run.smithi032:> rpm --scripts -qp /tmp/kernel.x86_64.rpm 2024-08-03T20:34:25.812 INFO:teuthology.orchestra.run.smithi032.stdout:postinstall scriptlet (using /bin/sh): 2024-08-03T20:34:25.812 INFO:teuthology.orchestra.run.smithi032.stdout:if [ -x /usr/bin/kernel-install ]; then 2024-08-03T20:34:25.812 INFO:teuthology.orchestra.run.smithi032.stdout: /usr/bin/kernel-install add 6.11.0-rc1-gef231aaae597 /lib/modules/6.11.0-rc1-gef231aaae597/vmlinuz 2024-08-03T20:34:25.812 INFO:teuthology.orchestra.run.smithi032.stdout:fi 2024-08-03T20:34:25.813 INFO:teuthology.orchestra.run.smithi032.stdout:for file in vmlinuz System.map config; do 2024-08-03T20:34:25.813 INFO:teuthology.orchestra.run.smithi032.stdout: if ! cmp --silent "/lib/modules/6.11.0-rc1-gef231aaae597/${file}" "/boot/${file}-6.11.0-rc1-gef231aaae597"; then 2024-08-03T20:34:25.813 INFO:teuthology.orchestra.run.smithi032.stdout: cp "/lib/modules/6.11.0-rc1-gef231aaae597/${file}" "/boot/${file}-6.11.0-rc1-gef231aaae597" 2024-08-03T20:34:25.813 INFO:teuthology.orchestra.run.smithi032.stdout: fi 2024-08-03T20:34:25.813 INFO:teuthology.orchestra.run.smithi032.stdout:done 2024-08-03T20:34:25.813 INFO:teuthology.orchestra.run.smithi032.stdout:if [ -d "/lib/modules/6.11.0-rc1-gef231aaae597/dtb" ] && \ 2024-08-03T20:34:25.813 INFO:teuthology.orchestra.run.smithi032.stdout: ! diff -rq "/lib/modules/6.11.0-rc1-gef231aaae597/dtb" "/boot/dtb-6.11.0-rc1-gef231aaae597" >/dev/null 2>&1; then 2024-08-03T20:34:25.813 INFO:teuthology.orchestra.run.smithi032.stdout: rm -rf "/boot/dtb-6.11.0-rc1-gef231aaae597" 2024-08-03T20:34:25.813 INFO:teuthology.orchestra.run.smithi032.stdout: cp -r "/lib/modules/6.11.0-rc1-gef231aaae597/dtb" "/boot/dtb-6.11.0-rc1-gef231aaae597" 2024-08-03T20:34:25.813 INFO:teuthology.orchestra.run.smithi032.stdout:fi 2024-08-03T20:34:25.813 INFO:teuthology.orchestra.run.smithi032.stdout:if [ ! -e "/lib/modules/6.11.0-rc1-gef231aaae597/modules.dep" ]; then 2024-08-03T20:34:25.813 INFO:teuthology.orchestra.run.smithi032.stdout: /usr/sbin/depmod 6.11.0-rc1-gef231aaae597 2024-08-03T20:34:25.813 INFO:teuthology.orchestra.run.smithi032.stdout:fi 2024-08-03T20:34:25.814 INFO:teuthology.orchestra.run.smithi032.stdout:preuninstall scriptlet (using /bin/sh): 2024-08-03T20:34:25.814 INFO:teuthology.orchestra.run.smithi032.stdout:if [ -x /usr/bin/kernel-install ]; then 2024-08-03T20:34:25.814 INFO:teuthology.orchestra.run.smithi032.stdout:kernel-install remove 6.11.0-rc1-gef231aaae597 2024-08-03T20:34:25.814 INFO:teuthology.orchestra.run.smithi032.stdout:fi 2024-08-03T20:34:25.815 DEBUG:teuthology.orchestra.run.smithi032:> sudo rpm -qi grub2-tools 2024-08-03T20:34:25.889 INFO:teuthology.orchestra.run.smithi032.stdout:Name : grub2-tools 2024-08-03T20:34:25.889 INFO:teuthology.orchestra.run.smithi032.stdout:Epoch : 1 2024-08-03T20:34:25.890 INFO:teuthology.orchestra.run.smithi032.stdout:Version : 2.06 2024-08-03T20:34:25.890 INFO:teuthology.orchestra.run.smithi032.stdout:Release : 68.el9 2024-08-03T20:34:25.890 INFO:teuthology.orchestra.run.smithi032.stdout:Architecture: x86_64 2024-08-03T20:34:25.890 INFO:teuthology.orchestra.run.smithi032.stdout:Install Date: Fri 19 Apr 2024 09:32:39 PM UTC 2024-08-03T20:34:25.890 INFO:teuthology.orchestra.run.smithi032.stdout:Group : Unspecified 2024-08-03T20:34:25.890 INFO:teuthology.orchestra.run.smithi032.stdout:Size : 8264254 2024-08-03T20:34:25.890 INFO:teuthology.orchestra.run.smithi032.stdout:License : GPLv3+ 2024-08-03T20:34:25.890 INFO:teuthology.orchestra.run.smithi032.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-08-03T20:34:25.890 INFO:teuthology.orchestra.run.smithi032.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-08-03T20:34:25.890 INFO:teuthology.orchestra.run.smithi032.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-08-03T20:34:25.890 INFO:teuthology.orchestra.run.smithi032.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-08-03T20:34:25.890 INFO:teuthology.orchestra.run.smithi032.stdout:Packager : builder@centos.org 2024-08-03T20:34:25.890 INFO:teuthology.orchestra.run.smithi032.stdout:Vendor : CentOS 2024-08-03T20:34:25.891 INFO:teuthology.orchestra.run.smithi032.stdout:URL : http://www.gnu.org/software/grub/ 2024-08-03T20:34:25.891 INFO:teuthology.orchestra.run.smithi032.stdout:Summary : Support tools for GRUB. 2024-08-03T20:34:25.891 INFO:teuthology.orchestra.run.smithi032.stdout:Description : 2024-08-03T20:34:25.891 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-03T20:34:25.891 INFO:teuthology.orchestra.run.smithi032.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-08-03T20:34:25.891 INFO:teuthology.orchestra.run.smithi032.stdout:customizable bootloader with modular architecture. It supports a rich 2024-08-03T20:34:25.891 INFO:teuthology.orchestra.run.smithi032.stdout:variety of kernel formats, file systems, computer architectures and 2024-08-03T20:34:25.891 INFO:teuthology.orchestra.run.smithi032.stdout:hardware devices. 2024-08-03T20:34:25.891 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-03T20:34:25.891 INFO:teuthology.orchestra.run.smithi032.stdout:This subpackage provides tools for support of all platforms. 2024-08-03T20:34:25.892 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-08-03T20:34:25.892 INFO:teuthology.task.kernel:Updating grub on smithi032 to boot 6.11.0-rc1-gef231aaae597 2024-08-03T20:34:25.892 DEBUG:teuthology.orchestra.run.smithi032:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-08-03T20:34:26.568 INFO:teuthology.orchestra.run.smithi032.stderr:Generating grub configuration file ... 2024-08-03T20:34:27.955 INFO:teuthology.orchestra.run.smithi032.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-08-03T20:34:27.988 INFO:teuthology.orchestra.run.smithi032.stderr:done 2024-08-03T20:34:27.991 DEBUG:teuthology.orchestra.run.smithi032:> mktemp 2024-08-03T20:34:28.011 INFO:teuthology.orchestra.run.smithi032.stdout:/tmp/tmp.TrmuJAQ2px 2024-08-03T20:34:28.012 DEBUG:teuthology.orchestra.run.smithi032:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.TrmuJAQ2px 2024-08-03T20:34:28.077 DEBUG:teuthology.orchestra.run.smithi032:> sudo chmod 0666 /tmp/tmp.TrmuJAQ2px 2024-08-03T20:34:28.217 DEBUG:teuthology.orchestra.remote:smithi032:/tmp/tmp.TrmuJAQ2px is 6KB 2024-08-03T20:34:28.265 DEBUG:teuthology.orchestra.run.smithi032:> rm -fr /tmp/tmp.TrmuJAQ2px 2024-08-03T20:34:28.280 DEBUG:teuthology.orchestra.run.smithi032:> sudo /bin/ls /boot/loader/entries || true 2024-08-03T20:34:28.343 INFO:teuthology.orchestra.run.smithi032.stdout:7fc8828e06d0482694bcf4a447d13cb6-0-rescue.conf 2024-08-03T20:34:28.343 INFO:teuthology.orchestra.run.smithi032.stdout:7fc8828e06d0482694bcf4a447d13cb6-5.14.0-239.el9.x86_64.conf 2024-08-03T20:34:28.343 INFO:teuthology.orchestra.run.smithi032.stdout:7fc8828e06d0482694bcf4a447d13cb6-5.14.0-437.el9.x86_64.conf 2024-08-03T20:34:28.343 INFO:teuthology.orchestra.run.smithi032.stdout:7fc8828e06d0482694bcf4a447d13cb6-6.11.0-rc1-gef231aaae597.conf 2024-08-03T20:34:28.344 DEBUG:teuthology.orchestra.run.smithi032:> sudo grub2-set-default 7fc8828e06d0482694bcf4a447d13cb6-6.11.0-rc1-gef231aaae597 2024-08-03T20:34:28.520 DEBUG:teuthology.orchestra.run.smithi032:> sudo shutdown -r now 2024-08-03T20:34:58.523 INFO:teuthology.task.kernel:Checking client client.0 for new kernel version... 2024-08-03T20:34:58.524 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi032.front.sepia.ceph.com' 2024-08-03T20:34:58.525 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi032.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-08-03T20:35:17.086 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.32 2024-08-03T20:35:26.088 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi032.front.sepia.ceph.com' 2024-08-03T20:35:26.089 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi032.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-08-03T20:35:36.233 INFO:teuthology.orchestra.run.smithi118.stdout:linux-firmware-20240716-145.el9.noarch.rpm 2.2 MB/s | 383 MB 02:51 2024-08-03T20:35:36.233 INFO:teuthology.orchestra.run.smithi118.stdout:-------------------------------------------------------------------------------- 2024-08-03T20:35:36.233 INFO:teuthology.orchestra.run.smithi118.stdout:Total 2.2 MB/s | 383 MB 02:51 2024-08-03T20:35:39.324 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction check 2024-08-03T20:35:39.416 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction check succeeded. 2024-08-03T20:35:39.416 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction test 2024-08-03T20:35:42.688 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction test succeeded. 2024-08-03T20:35:42.689 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction 2024-08-03T20:35:44.482 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.32 2024-08-03T20:35:52.424 INFO:teuthology.orchestra.run.smithi118.stdout: Preparing : 1/1 2024-08-03T20:35:53.364 INFO:teuthology.orchestra.run.smithi118.stdout: Upgrading : linux-firmware-20240716-145.el9.noarch 1/2 2024-08-03T20:35:53.438 INFO:teuthology.orchestra.run.smithi118.stdout: Cleanup : linux-firmware-20240219-143.el9.noarch 2/2 2024-08-03T20:35:54.007 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: linux-firmware-20240219-143.el9.noarch 2/2 2024-08-03T20:35:54.007 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : linux-firmware-20240716-145.el9.noarch 1/2 2024-08-03T20:35:54.771 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : linux-firmware-20240219-143.el9.noarch 2/2 2024-08-03T20:35:54.771 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-08-03T20:35:54.771 INFO:teuthology.orchestra.run.smithi118.stdout:Upgraded: 2024-08-03T20:35:54.771 INFO:teuthology.orchestra.run.smithi118.stdout: linux-firmware-20240716-145.el9.noarch 2024-08-03T20:35:54.772 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-08-03T20:35:54.772 INFO:teuthology.orchestra.run.smithi118.stdout:Complete! 2024-08-03T20:35:55.085 INFO:teuthology.task.kernel:Downloading kernel ef231aaae597a12bd3e2479b633f31ed1fcc23a8 on client.1... 2024-08-03T20:35:55.086 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=centos%2F9%2Fx86_64&sha1=ef231aaae597a12bd3e2479b633f31ed1fcc23a8 2024-08-03T20:35:55.240 INFO:teuthology.task.kernel:fetching, builder baseurl is https://4.chacra.ceph.com/r/kernel/testing/ef231aaae597a12bd3e2479b633f31ed1fcc23a8/centos/9/flavors/default/x86_64/ 2024-08-03T20:35:55.241 DEBUG:teuthology.orchestra.run.smithi118:> rm -f /tmp/kernel.x86_64.rpm && echo kernel-6.11.0_rc1_gef231aaae597-1.x86_64.rpm | wget -nv -O /tmp/kernel.x86_64.rpm --base=https://4.chacra.ceph.com/r/kernel/testing/ef231aaae597a12bd3e2479b633f31ed1fcc23a8/centos/9/flavors/default/x86_64/ --input-file=- 2024-08-03T20:35:55.243 DEBUG:teuthology.task.kernel:Waiting for download/copy to ubuntu@smithi118.front.sepia.ceph.com to complete... 2024-08-03T20:35:56.483 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi032.front.sepia.ceph.com' 2024-08-03T20:35:56.484 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi032.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-08-03T20:35:57.015 DEBUG:teuthology.orchestra.run.smithi032:> true 2024-08-03T20:35:57.618 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi032.front.sepia.ceph.com' 2024-08-03T20:35:57.618 INFO:teuthology.task.kernel:Checking kernel version of client.0, want "6.11.0-rc1-gef231aaae597"... 2024-08-03T20:35:57.618 DEBUG:teuthology.orchestra.run.smithi032:> uname -r 2024-08-03T20:35:57.652 INFO:teuthology.orchestra.run.smithi032.stdout:6.11.0-rc1-gef231aaae597 2024-08-03T20:35:57.653 DEBUG:teuthology.task.kernel:current kernel version is 6.11.0-rc1-gef231aaae597 vs 6.11.0-rc1-gef231aaae597 2024-08-03T20:35:57.653 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-08-03T20:35:57.653 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-08-03T20:35:58.656 DEBUG:teuthology.parallel:result is None 2024-08-03T20:36:02.755 INFO:teuthology.orchestra.run.smithi118.stderr:2024-08-03 20:36:02 URL:https://4.chacra.ceph.com/r/kernel/testing/ef231aaae597a12bd3e2479b633f31ed1fcc23a8/centos/9/flavors/default/x86_64/kernel-6.11.0_rc1_gef231aaae597-1.x86_64.rpm [359268674/359268674] -> "/tmp/kernel.x86_64.rpm" [1] 2024-08-03T20:36:02.852 INFO:teuthology.orchestra.run.smithi118.stderr:FINISHED --2024-08-03 20:36:02-- 2024-08-03T20:36:02.852 INFO:teuthology.orchestra.run.smithi118.stderr:Total wall clock time: 7.5s 2024-08-03T20:36:02.852 INFO:teuthology.orchestra.run.smithi118.stderr:Downloaded: 1 files, 343M in 7.2s (47.9 MB/s) 2024-08-03T20:36:02.855 INFO:teuthology.task.kernel:Installing kernel ef231aaae597a12bd3e2479b633f31ed1fcc23a8 on client.1... 2024-08-03T20:36:02.856 DEBUG:teuthology.orchestra.run.smithi118:> sudo rpm -ivh --oldpackage --replacefiles --replacepkgs /tmp/kernel.x86_64.rpm 2024-08-03T20:36:04.913 INFO:teuthology.orchestra.run.smithi118.stdout:Verifying... ######################################## 2024-08-03T20:36:05.120 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing... ######################################## 2024-08-03T20:36:05.169 INFO:teuthology.orchestra.run.smithi118.stdout:Updating / installing... 2024-08-03T20:36:13.151 INFO:teuthology.orchestra.run.smithi118.stdout:kernel-6.11.0_rc1_gef231aaae597-1 ######################################## 2024-08-03T20:36:57.654 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi118.front.sepia.ceph.com, path=/tmp/kernel.x86_64.rpm, version=None) 2024-08-03T20:36:57.655 DEBUG:teuthology.orchestra.run.smithi118:> rpm -qlp /tmp/kernel.x86_64.rpm 2024-08-03T20:36:57.733 INFO:teuthology.orchestra.run.smithi118.stdout:/boot/System.map-6.11.0-rc1-gef231aaae597 2024-08-03T20:36:57.733 INFO:teuthology.orchestra.run.smithi118.stdout:/boot/config-6.11.0-rc1-gef231aaae597 2024-08-03T20:36:57.733 INFO:teuthology.orchestra.run.smithi118.stdout:/boot/vmlinuz-6.11.0-rc1-gef231aaae597 2024-08-03T20:36:57.734 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597 2024-08-03T20:36:57.734 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/System.map 2024-08-03T20:36:57.734 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/config 2024-08-03T20:36:57.734 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel 2024-08-03T20:36:57.734 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/arch 2024-08-03T20:36:57.734 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/arch/x86 2024-08-03T20:36:57.734 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/arch/x86/crypto 2024-08-03T20:36:57.734 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/arch/x86/crypto/blowfish-x86_64.ko 2024-08-03T20:36:57.734 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/arch/x86/crypto/camellia-aesni-avx-x86_64.ko 2024-08-03T20:36:57.734 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/arch/x86/crypto/camellia-aesni-avx2.ko 2024-08-03T20:36:57.734 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/arch/x86/crypto/camellia-x86_64.ko 2024-08-03T20:36:57.734 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/arch/x86/crypto/cast5-avx-x86_64.ko 2024-08-03T20:36:57.735 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/arch/x86/crypto/cast6-avx-x86_64.ko 2024-08-03T20:36:57.735 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/arch/x86/crypto/crc32-pclmul.ko 2024-08-03T20:36:57.735 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/arch/x86/crypto/crc32c-intel.ko 2024-08-03T20:36:57.735 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/arch/x86/crypto/crct10dif-pclmul.ko 2024-08-03T20:36:57.735 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/arch/x86/crypto/des3_ede-x86_64.ko 2024-08-03T20:36:57.735 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/arch/x86/crypto/ghash-clmulni-intel.ko 2024-08-03T20:36:57.735 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/arch/x86/crypto/serpent-avx-x86_64.ko 2024-08-03T20:36:57.735 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/arch/x86/crypto/serpent-avx2.ko 2024-08-03T20:36:57.735 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/arch/x86/crypto/serpent-sse2-x86_64.ko 2024-08-03T20:36:57.735 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/arch/x86/crypto/sha1-ssse3.ko 2024-08-03T20:36:57.735 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/arch/x86/crypto/sha256-ssse3.ko 2024-08-03T20:36:57.735 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/arch/x86/crypto/sha512-ssse3.ko 2024-08-03T20:36:57.736 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/arch/x86/crypto/twofish-avx-x86_64.ko 2024-08-03T20:36:57.736 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/arch/x86/crypto/twofish-x86_64-3way.ko 2024-08-03T20:36:57.736 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/arch/x86/crypto/twofish-x86_64.ko 2024-08-03T20:36:57.736 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/arch/x86/kernel 2024-08-03T20:36:57.736 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/arch/x86/kernel/cpu 2024-08-03T20:36:57.736 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/arch/x86/kernel/cpu/mce 2024-08-03T20:36:57.736 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/arch/x86/kernel/cpu/mce/mce-inject.ko 2024-08-03T20:36:57.736 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/arch/x86/kvm 2024-08-03T20:36:57.736 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/arch/x86/kvm/kvm-amd.ko 2024-08-03T20:36:57.736 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/arch/x86/kvm/kvm-intel.ko 2024-08-03T20:36:57.736 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/arch/x86/kvm/kvm.ko 2024-08-03T20:36:57.736 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/crypto 2024-08-03T20:36:57.737 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/crypto/ansi_cprng.ko 2024-08-03T20:36:57.737 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/crypto/anubis.ko 2024-08-03T20:36:57.737 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/crypto/arc4.ko 2024-08-03T20:36:57.737 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/crypto/async_tx 2024-08-03T20:36:57.737 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/crypto/async_tx/async_memcpy.ko 2024-08-03T20:36:57.737 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/crypto/async_tx/async_pq.ko 2024-08-03T20:36:57.737 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/crypto/async_tx/async_raid6_recov.ko 2024-08-03T20:36:57.737 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/crypto/async_tx/async_tx.ko 2024-08-03T20:36:57.737 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/crypto/async_tx/async_xor.ko 2024-08-03T20:36:57.737 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/crypto/async_tx/raid6test.ko 2024-08-03T20:36:57.737 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/crypto/authenc.ko 2024-08-03T20:36:57.738 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/crypto/authencesn.ko 2024-08-03T20:36:57.738 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/crypto/blake2b_generic.ko 2024-08-03T20:36:57.738 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/crypto/blowfish_common.ko 2024-08-03T20:36:57.738 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/crypto/blowfish_generic.ko 2024-08-03T20:36:57.738 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/crypto/camellia_generic.ko 2024-08-03T20:36:57.738 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/crypto/cast5_generic.ko 2024-08-03T20:36:57.738 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/crypto/cast6_generic.ko 2024-08-03T20:36:57.738 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/crypto/cast_common.ko 2024-08-03T20:36:57.738 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/crypto/ccm.ko 2024-08-03T20:36:57.738 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/crypto/cmac.ko 2024-08-03T20:36:57.738 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/crypto/crc32_generic.ko 2024-08-03T20:36:57.738 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/crypto/crypto_engine.ko 2024-08-03T20:36:57.738 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/crypto/crypto_user.ko 2024-08-03T20:36:57.739 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/crypto/cts.ko 2024-08-03T20:36:57.739 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/crypto/des_generic.ko 2024-08-03T20:36:57.739 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/crypto/dh_generic.ko 2024-08-03T20:36:57.739 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/crypto/ecc.ko 2024-08-03T20:36:57.739 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/crypto/ecdh_generic.ko 2024-08-03T20:36:57.739 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/crypto/echainiv.ko 2024-08-03T20:36:57.739 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/crypto/essiv.ko 2024-08-03T20:36:57.739 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/crypto/fcrypt.ko 2024-08-03T20:36:57.739 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/crypto/khazad.ko 2024-08-03T20:36:57.739 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/crypto/lz4.ko 2024-08-03T20:36:57.739 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/crypto/lz4hc.ko 2024-08-03T20:36:57.739 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/crypto/md4.ko 2024-08-03T20:36:57.739 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/crypto/michael_mic.ko 2024-08-03T20:36:57.739 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/crypto/pcbc.ko 2024-08-03T20:36:57.740 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/crypto/pcrypt.ko 2024-08-03T20:36:57.740 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/crypto/rmd160.ko 2024-08-03T20:36:57.740 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/crypto/seed.ko 2024-08-03T20:36:57.740 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/crypto/serpent_generic.ko 2024-08-03T20:36:57.740 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/crypto/tcrypt.ko 2024-08-03T20:36:57.740 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/crypto/tea.ko 2024-08-03T20:36:57.740 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/crypto/twofish_common.ko 2024-08-03T20:36:57.740 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/crypto/twofish_generic.ko 2024-08-03T20:36:57.740 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/crypto/vmac.ko 2024-08-03T20:36:57.740 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/crypto/wp512.ko 2024-08-03T20:36:57.740 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/crypto/xcbc.ko 2024-08-03T20:36:57.740 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/crypto/xor.ko 2024-08-03T20:36:57.740 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/crypto/xxhash_generic.ko 2024-08-03T20:36:57.741 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers 2024-08-03T20:36:57.741 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/acpi 2024-08-03T20:36:57.741 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/acpi/acpi_ipmi.ko 2024-08-03T20:36:57.741 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/acpi/acpi_pad.ko 2024-08-03T20:36:57.741 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/acpi/ec_sys.ko 2024-08-03T20:36:57.741 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/acpi/platform_profile.ko 2024-08-03T20:36:57.741 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/acpi/sbs.ko 2024-08-03T20:36:57.741 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/acpi/sbshc.ko 2024-08-03T20:36:57.741 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/acpi/video.ko 2024-08-03T20:36:57.741 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/ata 2024-08-03T20:36:57.741 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/ata/acard-ahci.ko 2024-08-03T20:36:57.741 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/ata/ahci_platform.ko 2024-08-03T20:36:57.741 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/ata/ata_generic.ko 2024-08-03T20:36:57.741 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/ata/libahci_platform.ko 2024-08-03T20:36:57.741 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/ata/pata_acpi.ko 2024-08-03T20:36:57.742 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/ata/pata_ali.ko 2024-08-03T20:36:57.742 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/ata/pata_amd.ko 2024-08-03T20:36:57.742 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/ata/pata_artop.ko 2024-08-03T20:36:57.742 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/ata/pata_atiixp.ko 2024-08-03T20:36:57.742 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/ata/pata_atp867x.ko 2024-08-03T20:36:57.742 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/ata/pata_cmd640.ko 2024-08-03T20:36:57.742 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/ata/pata_cmd64x.ko 2024-08-03T20:36:57.742 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/ata/pata_cypress.ko 2024-08-03T20:36:57.742 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/ata/pata_efar.ko 2024-08-03T20:36:57.742 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/ata/pata_hpt366.ko 2024-08-03T20:36:57.742 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/ata/pata_hpt37x.ko 2024-08-03T20:36:57.742 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/ata/pata_hpt3x2n.ko 2024-08-03T20:36:57.742 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/ata/pata_hpt3x3.ko 2024-08-03T20:36:57.743 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/ata/pata_it8213.ko 2024-08-03T20:36:57.743 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/ata/pata_it821x.ko 2024-08-03T20:36:57.743 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/ata/pata_jmicron.ko 2024-08-03T20:36:57.743 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/ata/pata_marvell.ko 2024-08-03T20:36:57.743 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/ata/pata_mpiix.ko 2024-08-03T20:36:57.743 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/ata/pata_netcell.ko 2024-08-03T20:36:57.743 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/ata/pata_ninja32.ko 2024-08-03T20:36:57.743 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/ata/pata_ns87410.ko 2024-08-03T20:36:57.743 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/ata/pata_ns87415.ko 2024-08-03T20:36:57.743 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/ata/pata_oldpiix.ko 2024-08-03T20:36:57.743 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/ata/pata_opti.ko 2024-08-03T20:36:57.743 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/ata/pata_optidma.ko 2024-08-03T20:36:57.743 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/ata/pata_pcmcia.ko 2024-08-03T20:36:57.743 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/ata/pata_pdc2027x.ko 2024-08-03T20:36:57.744 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/ata/pata_pdc202xx_old.ko 2024-08-03T20:36:57.744 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/ata/pata_piccolo.ko 2024-08-03T20:36:57.744 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/ata/pata_rdc.ko 2024-08-03T20:36:57.744 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/ata/pata_sch.ko 2024-08-03T20:36:57.744 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/ata/pata_serverworks.ko 2024-08-03T20:36:57.744 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/ata/pata_sil680.ko 2024-08-03T20:36:57.744 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/ata/pata_sis.ko 2024-08-03T20:36:57.744 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/ata/pata_sl82c105.ko 2024-08-03T20:36:57.744 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/ata/pata_triflex.ko 2024-08-03T20:36:57.744 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/ata/pata_via.ko 2024-08-03T20:36:57.744 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/ata/pdc_adma.ko 2024-08-03T20:36:57.744 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/ata/sata_inic162x.ko 2024-08-03T20:36:57.744 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/ata/sata_mv.ko 2024-08-03T20:36:57.745 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/ata/sata_nv.ko 2024-08-03T20:36:57.745 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/ata/sata_promise.ko 2024-08-03T20:36:57.745 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/ata/sata_qstor.ko 2024-08-03T20:36:57.745 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/ata/sata_sil.ko 2024-08-03T20:36:57.745 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/ata/sata_sil24.ko 2024-08-03T20:36:57.745 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/ata/sata_sis.ko 2024-08-03T20:36:57.745 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/ata/sata_svw.ko 2024-08-03T20:36:57.745 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/ata/sata_sx4.ko 2024-08-03T20:36:57.745 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/ata/sata_uli.ko 2024-08-03T20:36:57.745 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/ata/sata_via.ko 2024-08-03T20:36:57.745 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/ata/sata_vsc.ko 2024-08-03T20:36:57.745 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/auxdisplay 2024-08-03T20:36:57.745 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/auxdisplay/ks0108.ko 2024-08-03T20:36:57.746 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/base 2024-08-03T20:36:57.746 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/base/regmap 2024-08-03T20:36:57.746 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/base/regmap/regmap-i2c.ko 2024-08-03T20:36:57.746 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/base/regmap/regmap-mmio.ko 2024-08-03T20:36:57.746 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/bcma 2024-08-03T20:36:57.746 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/bcma/bcma.ko 2024-08-03T20:36:57.746 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/block 2024-08-03T20:36:57.746 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/block/aoe 2024-08-03T20:36:57.746 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/block/aoe/aoe.ko 2024-08-03T20:36:57.746 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/block/brd.ko 2024-08-03T20:36:57.746 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/block/drbd 2024-08-03T20:36:57.746 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/block/drbd/drbd.ko 2024-08-03T20:36:57.746 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/block/floppy.ko 2024-08-03T20:36:57.747 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/block/loop.ko 2024-08-03T20:36:57.747 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/block/mtip32xx 2024-08-03T20:36:57.747 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/block/mtip32xx/mtip32xx.ko 2024-08-03T20:36:57.747 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/block/nbd.ko 2024-08-03T20:36:57.747 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/block/null_blk 2024-08-03T20:36:57.747 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/block/null_blk/null_blk.ko 2024-08-03T20:36:57.747 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/block/pktcdvd.ko 2024-08-03T20:36:57.747 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/block/rbd.ko 2024-08-03T20:36:57.747 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/block/virtio_blk.ko 2024-08-03T20:36:57.747 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/block/xen-blkback 2024-08-03T20:36:57.747 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/block/xen-blkback/xen-blkback.ko 2024-08-03T20:36:57.747 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/block/xen-blkfront.ko 2024-08-03T20:36:57.747 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/block/zram 2024-08-03T20:36:57.748 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/block/zram/zram.ko 2024-08-03T20:36:57.748 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/char 2024-08-03T20:36:57.748 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/char/hangcheck-timer.ko 2024-08-03T20:36:57.748 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/char/hw_random 2024-08-03T20:36:57.748 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/char/hw_random/amd-rng.ko 2024-08-03T20:36:57.748 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/char/hw_random/intel-rng.ko 2024-08-03T20:36:57.748 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/char/hw_random/timeriomem-rng.ko 2024-08-03T20:36:57.748 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/char/hw_random/via-rng.ko 2024-08-03T20:36:57.748 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/char/hw_random/virtio-rng.ko 2024-08-03T20:36:57.748 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/char/ipmi 2024-08-03T20:36:57.748 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/char/ipmi/ipmi_devintf.ko 2024-08-03T20:36:57.749 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/char/ipmi/ipmi_msghandler.ko 2024-08-03T20:36:57.749 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/char/ipmi/ipmi_poweroff.ko 2024-08-03T20:36:57.749 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/char/ipmi/ipmi_si.ko 2024-08-03T20:36:57.749 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/char/ipmi/ipmi_ssif.ko 2024-08-03T20:36:57.749 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/char/ipmi/ipmi_watchdog.ko 2024-08-03T20:36:57.749 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/char/lp.ko 2024-08-03T20:36:57.749 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/char/mwave 2024-08-03T20:36:57.749 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/char/mwave/mwave.ko 2024-08-03T20:36:57.749 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/char/ppdev.ko 2024-08-03T20:36:57.749 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/char/tlclk.ko 2024-08-03T20:36:57.749 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/char/tpm 2024-08-03T20:36:57.749 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/char/tpm/tpm.ko 2024-08-03T20:36:57.749 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/char/tpm/tpm_atmel.ko 2024-08-03T20:36:57.749 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/char/tpm/tpm_infineon.ko 2024-08-03T20:36:57.750 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/char/tpm/tpm_nsc.ko 2024-08-03T20:36:57.750 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/char/tpm/tpm_tis.ko 2024-08-03T20:36:57.750 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/char/tpm/tpm_tis_core.ko 2024-08-03T20:36:57.750 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/char/uv_mmtimer.ko 2024-08-03T20:36:57.750 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/char/virtio_console.ko 2024-08-03T20:36:57.750 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/cpufreq 2024-08-03T20:36:57.750 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/cpufreq/acpi-cpufreq.ko 2024-08-03T20:36:57.750 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/cpufreq/amd_freq_sensitivity.ko 2024-08-03T20:36:57.750 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/cpufreq/p4-clockmod.ko 2024-08-03T20:36:57.750 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/cpufreq/pcc-cpufreq.ko 2024-08-03T20:36:57.750 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/cpufreq/powernow-k8.ko 2024-08-03T20:36:57.751 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/cpufreq/speedstep-lib.ko 2024-08-03T20:36:57.751 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/crypto 2024-08-03T20:36:57.751 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/crypto/ccp 2024-08-03T20:36:57.751 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/crypto/ccp/ccp-crypto.ko 2024-08-03T20:36:57.751 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/crypto/ccp/ccp.ko 2024-08-03T20:36:57.751 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/crypto/intel 2024-08-03T20:36:57.751 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/crypto/intel/qat 2024-08-03T20:36:57.751 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/crypto/intel/qat/qat_common 2024-08-03T20:36:57.751 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/crypto/intel/qat/qat_common/intel_qat.ko 2024-08-03T20:36:57.752 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/crypto/intel/qat/qat_dh895xcc 2024-08-03T20:36:57.752 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/crypto/intel/qat/qat_dh895xcc/qat_dh895xcc.ko 2024-08-03T20:36:57.752 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/crypto/padlock-aes.ko 2024-08-03T20:36:57.752 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/crypto/padlock-sha.ko 2024-08-03T20:36:57.752 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/crypto/virtio 2024-08-03T20:36:57.752 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/crypto/virtio/virtio_crypto.ko 2024-08-03T20:36:57.752 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/dca 2024-08-03T20:36:57.752 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/dca/dca.ko 2024-08-03T20:36:57.752 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/devfreq 2024-08-03T20:36:57.752 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/devfreq/governor_simpleondemand.ko 2024-08-03T20:36:57.752 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/dma 2024-08-03T20:36:57.752 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/dma/dw 2024-08-03T20:36:57.752 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/dma/dw/dw_dmac.ko 2024-08-03T20:36:57.753 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/dma/ioat 2024-08-03T20:36:57.753 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/dma/ioat/ioatdma.ko 2024-08-03T20:36:57.753 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/edac 2024-08-03T20:36:57.753 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/edac/amd64_edac.ko 2024-08-03T20:36:57.753 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/edac/e752x_edac.ko 2024-08-03T20:36:57.753 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/edac/edac_mce_amd.ko 2024-08-03T20:36:57.753 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/edac/i3000_edac.ko 2024-08-03T20:36:57.753 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/edac/i3200_edac.ko 2024-08-03T20:36:57.753 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/edac/i5100_edac.ko 2024-08-03T20:36:57.754 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/edac/i5400_edac.ko 2024-08-03T20:36:57.754 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/edac/i7300_edac.ko 2024-08-03T20:36:57.754 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/edac/i7core_edac.ko 2024-08-03T20:36:57.754 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/edac/i82975x_edac.ko 2024-08-03T20:36:57.754 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/edac/ie31200_edac.ko 2024-08-03T20:36:57.754 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/edac/sb_edac.ko 2024-08-03T20:36:57.755 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/edac/x38_edac.ko 2024-08-03T20:36:57.755 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/firmware 2024-08-03T20:36:57.755 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/firmware/edd.ko 2024-08-03T20:36:57.755 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/firmware/iscsi_ibft.ko 2024-08-03T20:36:57.755 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/gpio 2024-08-03T20:36:57.755 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/gpio/gpio-ich.ko 2024-08-03T20:36:57.755 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/gpio/gpio-viperboard.ko 2024-08-03T20:36:57.755 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/hid 2024-08-03T20:36:57.756 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/hid/hid-appleir.ko 2024-08-03T20:36:57.756 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/hid/hid-aureal.ko 2024-08-03T20:36:57.756 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/hid/hid-axff.ko 2024-08-03T20:36:57.756 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/hid/hid-dr.ko 2024-08-03T20:36:57.756 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/hid/hid-elecom.ko 2024-08-03T20:36:57.756 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/hid/hid-elo.ko 2024-08-03T20:36:57.756 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/hid/hid-emsff.ko 2024-08-03T20:36:57.756 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/hid/hid-gaff.ko 2024-08-03T20:36:57.756 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/hid/hid-gt683r.ko 2024-08-03T20:36:57.756 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/hid/hid-gyration.ko 2024-08-03T20:36:57.756 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/hid/hid-holtek-kbd.ko 2024-08-03T20:36:57.756 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/hid/hid-holtek-mouse.ko 2024-08-03T20:36:57.757 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/hid/hid-holtekff.ko 2024-08-03T20:36:57.757 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/hid/hid-hyperv.ko 2024-08-03T20:36:57.757 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/hid/hid-icade.ko 2024-08-03T20:36:57.757 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/hid/hid-keytouch.ko 2024-08-03T20:36:57.757 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/hid/hid-kye.ko 2024-08-03T20:36:57.757 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/hid/hid-lcpower.ko 2024-08-03T20:36:57.757 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/hid/hid-led.ko 2024-08-03T20:36:57.757 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/hid/hid-lenovo.ko 2024-08-03T20:36:57.757 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/hid/hid-logitech-dj.ko 2024-08-03T20:36:57.757 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/hid/hid-logitech-hidpp.ko 2024-08-03T20:36:57.757 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/hid/hid-multitouch.ko 2024-08-03T20:36:57.757 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/hid/hid-ortek.ko 2024-08-03T20:36:57.758 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/hid/hid-penmount.ko 2024-08-03T20:36:57.758 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/hid/hid-petalynx.ko 2024-08-03T20:36:57.758 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/hid/hid-picolcd.ko 2024-08-03T20:36:57.758 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/hid/hid-pl.ko 2024-08-03T20:36:57.758 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/hid/hid-plantronics.ko 2024-08-03T20:36:57.758 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/hid/hid-primax.ko 2024-08-03T20:36:57.759 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/hid/hid-rmi.ko 2024-08-03T20:36:57.759 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/hid/hid-roccat-arvo.ko 2024-08-03T20:36:57.759 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/hid/hid-roccat-common.ko 2024-08-03T20:36:57.759 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/hid/hid-roccat-isku.ko 2024-08-03T20:36:57.759 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/hid/hid-roccat-kone.ko 2024-08-03T20:36:57.759 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/hid/hid-roccat-koneplus.ko 2024-08-03T20:36:57.759 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/hid/hid-roccat-konepure.ko 2024-08-03T20:36:57.759 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/hid/hid-roccat-kovaplus.ko 2024-08-03T20:36:57.759 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/hid/hid-roccat-lua.ko 2024-08-03T20:36:57.759 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/hid/hid-roccat-pyra.ko 2024-08-03T20:36:57.759 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/hid/hid-roccat-ryos.ko 2024-08-03T20:36:57.760 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/hid/hid-roccat-savu.ko 2024-08-03T20:36:57.760 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/hid/hid-roccat.ko 2024-08-03T20:36:57.760 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/hid/hid-saitek.ko 2024-08-03T20:36:57.760 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/hid/hid-samsung.ko 2024-08-03T20:36:57.760 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/hid/hid-sensor-hub.ko 2024-08-03T20:36:57.760 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/hid/hid-sjoy.ko 2024-08-03T20:36:57.760 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/hid/hid-sony.ko 2024-08-03T20:36:57.760 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/hid/hid-speedlink.ko 2024-08-03T20:36:57.760 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/hid/hid-steelseries.ko 2024-08-03T20:36:57.760 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/hid/hid-sunplus.ko 2024-08-03T20:36:57.760 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/hid/hid-thrustmaster.ko 2024-08-03T20:36:57.760 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/hid/hid-tivo.ko 2024-08-03T20:36:57.760 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/hid/hid-tmff.ko 2024-08-03T20:36:57.761 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/hid/hid-topseed.ko 2024-08-03T20:36:57.761 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/hid/hid-twinhan.ko 2024-08-03T20:36:57.761 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/hid/hid-uclogic.ko 2024-08-03T20:36:57.761 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/hid/hid-waltop.ko 2024-08-03T20:36:57.761 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/hid/hid-wiimote.ko 2024-08-03T20:36:57.761 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/hid/hid-xinmo.ko 2024-08-03T20:36:57.761 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/hid/hid-zpff.ko 2024-08-03T20:36:57.761 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/hid/hid-zydacron.ko 2024-08-03T20:36:57.761 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/hid/uhid.ko 2024-08-03T20:36:57.761 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/hid/wacom.ko 2024-08-03T20:36:57.761 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/hv 2024-08-03T20:36:57.761 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/hv/hv_balloon.ko 2024-08-03T20:36:57.762 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/hv/hv_utils.ko 2024-08-03T20:36:57.762 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/hv/hv_vmbus.ko 2024-08-03T20:36:57.762 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/hwmon 2024-08-03T20:36:57.762 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/hwmon/abituguru.ko 2024-08-03T20:36:57.762 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/hwmon/abituguru3.ko 2024-08-03T20:36:57.762 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/hwmon/acpi_power_meter.ko 2024-08-03T20:36:57.762 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/hwmon/ad7414.ko 2024-08-03T20:36:57.762 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/hwmon/ad7418.ko 2024-08-03T20:36:57.762 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/hwmon/adc128d818.ko 2024-08-03T20:36:57.763 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/hwmon/adm1025.ko 2024-08-03T20:36:57.763 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/hwmon/adm1026.ko 2024-08-03T20:36:57.763 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/hwmon/adm1029.ko 2024-08-03T20:36:57.763 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/hwmon/adm1031.ko 2024-08-03T20:36:57.763 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/hwmon/adm9240.ko 2024-08-03T20:36:57.763 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/hwmon/ads7828.ko 2024-08-03T20:36:57.763 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/hwmon/adt7410.ko 2024-08-03T20:36:57.763 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/hwmon/adt7411.ko 2024-08-03T20:36:57.763 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/hwmon/adt7462.ko 2024-08-03T20:36:57.763 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/hwmon/adt7470.ko 2024-08-03T20:36:57.763 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/hwmon/adt7475.ko 2024-08-03T20:36:57.764 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/hwmon/adt7x10.ko 2024-08-03T20:36:57.764 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/hwmon/amc6821.ko 2024-08-03T20:36:57.764 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/hwmon/applesmc.ko 2024-08-03T20:36:57.764 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/hwmon/asb100.ko 2024-08-03T20:36:57.764 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/hwmon/asc7621.ko 2024-08-03T20:36:57.764 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/hwmon/asus_atk0110.ko 2024-08-03T20:36:57.764 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/hwmon/atxp1.ko 2024-08-03T20:36:57.764 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/hwmon/coretemp.ko 2024-08-03T20:36:57.764 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/hwmon/dell-smm-hwmon.ko 2024-08-03T20:36:57.764 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/hwmon/dme1737.ko 2024-08-03T20:36:57.764 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/hwmon/ds1621.ko 2024-08-03T20:36:57.764 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/hwmon/ds620.ko 2024-08-03T20:36:57.764 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/hwmon/emc1403.ko 2024-08-03T20:36:57.764 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/hwmon/emc6w201.ko 2024-08-03T20:36:57.765 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/hwmon/f71805f.ko 2024-08-03T20:36:57.765 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/hwmon/f71882fg.ko 2024-08-03T20:36:57.765 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/hwmon/f75375s.ko 2024-08-03T20:36:57.765 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/hwmon/fam15h_power.ko 2024-08-03T20:36:57.765 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/hwmon/fschmd.ko 2024-08-03T20:36:57.765 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/hwmon/g760a.ko 2024-08-03T20:36:57.765 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/hwmon/g762.ko 2024-08-03T20:36:57.765 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/hwmon/gl518sm.ko 2024-08-03T20:36:57.765 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/hwmon/gl520sm.ko 2024-08-03T20:36:57.765 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/hwmon/hwmon-vid.ko 2024-08-03T20:36:57.765 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/hwmon/i5500_temp.ko 2024-08-03T20:36:57.766 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/hwmon/i5k_amb.ko 2024-08-03T20:36:57.766 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/hwmon/ibmaem.ko 2024-08-03T20:36:57.766 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/hwmon/ibmpex.ko 2024-08-03T20:36:57.766 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/hwmon/ina209.ko 2024-08-03T20:36:57.766 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/hwmon/ina2xx.ko 2024-08-03T20:36:57.766 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/hwmon/it87.ko 2024-08-03T20:36:57.766 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/hwmon/k10temp.ko 2024-08-03T20:36:57.766 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/hwmon/k8temp.ko 2024-08-03T20:36:57.766 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/hwmon/lineage-pem.ko 2024-08-03T20:36:57.766 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/hwmon/lm63.ko 2024-08-03T20:36:57.766 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/hwmon/lm73.ko 2024-08-03T20:36:57.766 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/hwmon/lm75.ko 2024-08-03T20:36:57.767 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/hwmon/lm77.ko 2024-08-03T20:36:57.767 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/hwmon/lm78.ko 2024-08-03T20:36:57.767 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/hwmon/lm80.ko 2024-08-03T20:36:57.767 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/hwmon/lm83.ko 2024-08-03T20:36:57.767 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/hwmon/lm85.ko 2024-08-03T20:36:57.767 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/hwmon/lm87.ko 2024-08-03T20:36:57.767 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/hwmon/lm90.ko 2024-08-03T20:36:57.767 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/hwmon/lm92.ko 2024-08-03T20:36:57.767 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/hwmon/lm93.ko 2024-08-03T20:36:57.767 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/hwmon/lm95234.ko 2024-08-03T20:36:57.767 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/hwmon/lm95241.ko 2024-08-03T20:36:57.767 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/hwmon/lm95245.ko 2024-08-03T20:36:57.767 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/hwmon/ltc2945.ko 2024-08-03T20:36:57.768 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/hwmon/ltc4151.ko 2024-08-03T20:36:57.768 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/hwmon/ltc4215.ko 2024-08-03T20:36:57.768 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/hwmon/ltc4222.ko 2024-08-03T20:36:57.768 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/hwmon/ltc4245.ko 2024-08-03T20:36:57.768 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/hwmon/ltc4260.ko 2024-08-03T20:36:57.768 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/hwmon/ltc4261.ko 2024-08-03T20:36:57.768 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/hwmon/max16065.ko 2024-08-03T20:36:57.768 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/hwmon/max1619.ko 2024-08-03T20:36:57.768 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/hwmon/max1668.ko 2024-08-03T20:36:57.768 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/hwmon/max197.ko 2024-08-03T20:36:57.768 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/hwmon/max6639.ko 2024-08-03T20:36:57.768 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/hwmon/max6650.ko 2024-08-03T20:36:57.769 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/hwmon/max6697.ko 2024-08-03T20:36:57.769 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/hwmon/mcp3021.ko 2024-08-03T20:36:57.769 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/hwmon/nct6683.ko 2024-08-03T20:36:57.769 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/hwmon/nct6775-core.ko 2024-08-03T20:36:57.769 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/hwmon/nct6775.ko 2024-08-03T20:36:57.769 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/hwmon/nct7802.ko 2024-08-03T20:36:57.769 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/hwmon/pc87360.ko 2024-08-03T20:36:57.769 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/hwmon/pc87427.ko 2024-08-03T20:36:57.769 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/hwmon/pcf8591.ko 2024-08-03T20:36:57.769 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/hwmon/pmbus 2024-08-03T20:36:57.769 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/hwmon/pmbus/adm1275.ko 2024-08-03T20:36:57.769 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/hwmon/pmbus/lm25066.ko 2024-08-03T20:36:57.769 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/hwmon/pmbus/ltc2978.ko 2024-08-03T20:36:57.770 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/hwmon/pmbus/max16064.ko 2024-08-03T20:36:57.770 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/hwmon/pmbus/max34440.ko 2024-08-03T20:36:57.770 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/hwmon/pmbus/max8688.ko 2024-08-03T20:36:57.770 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/hwmon/pmbus/pmbus.ko 2024-08-03T20:36:57.770 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/hwmon/pmbus/pmbus_core.ko 2024-08-03T20:36:57.770 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/hwmon/pmbus/tps40422.ko 2024-08-03T20:36:57.770 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/hwmon/pmbus/ucd9000.ko 2024-08-03T20:36:57.770 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/hwmon/pmbus/ucd9200.ko 2024-08-03T20:36:57.770 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/hwmon/pmbus/zl6100.ko 2024-08-03T20:36:57.770 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/hwmon/powr1220.ko 2024-08-03T20:36:57.770 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/hwmon/sch5627.ko 2024-08-03T20:36:57.770 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/hwmon/sch5636.ko 2024-08-03T20:36:57.771 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/hwmon/sch56xx-common.ko 2024-08-03T20:36:57.771 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/hwmon/sht15.ko 2024-08-03T20:36:57.771 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/hwmon/sht21.ko 2024-08-03T20:36:57.771 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/hwmon/shtc1.ko 2024-08-03T20:36:57.771 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/hwmon/sis5595.ko 2024-08-03T20:36:57.771 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/hwmon/smsc47b397.ko 2024-08-03T20:36:57.771 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/hwmon/smsc47m1.ko 2024-08-03T20:36:57.771 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/hwmon/smsc47m192.ko 2024-08-03T20:36:57.771 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/hwmon/thmc50.ko 2024-08-03T20:36:57.771 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/hwmon/tmp102.ko 2024-08-03T20:36:57.771 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/hwmon/tmp103.ko 2024-08-03T20:36:57.771 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/hwmon/tmp401.ko 2024-08-03T20:36:57.771 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/hwmon/tmp421.ko 2024-08-03T20:36:57.771 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/hwmon/via-cputemp.ko 2024-08-03T20:36:57.772 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/hwmon/via686a.ko 2024-08-03T20:36:57.772 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/hwmon/vt1211.ko 2024-08-03T20:36:57.772 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/hwmon/vt8231.ko 2024-08-03T20:36:57.772 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/hwmon/w83627ehf.ko 2024-08-03T20:36:57.772 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/hwmon/w83627hf.ko 2024-08-03T20:36:57.772 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/hwmon/w83781d.ko 2024-08-03T20:36:57.772 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/hwmon/w83791d.ko 2024-08-03T20:36:57.772 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/hwmon/w83792d.ko 2024-08-03T20:36:57.772 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/hwmon/w83793.ko 2024-08-03T20:36:57.772 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/hwmon/w83795.ko 2024-08-03T20:36:57.772 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/hwmon/w83l785ts.ko 2024-08-03T20:36:57.773 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/hwmon/w83l786ng.ko 2024-08-03T20:36:57.773 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/i2c 2024-08-03T20:36:57.773 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/i2c/algos 2024-08-03T20:36:57.773 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/i2c/algos/i2c-algo-bit.ko 2024-08-03T20:36:57.773 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/i2c/algos/i2c-algo-pca.ko 2024-08-03T20:36:57.773 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/i2c/busses 2024-08-03T20:36:57.773 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/i2c/busses/i2c-amd756-s4882.ko 2024-08-03T20:36:57.773 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/i2c/busses/i2c-amd756.ko 2024-08-03T20:36:57.773 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/i2c/busses/i2c-amd8111.ko 2024-08-03T20:36:57.773 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/i2c/busses/i2c-ccgx-ucsi.ko 2024-08-03T20:36:57.773 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/i2c/busses/i2c-designware-core.ko 2024-08-03T20:36:57.773 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/i2c/busses/i2c-designware-pci.ko 2024-08-03T20:36:57.774 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/i2c/busses/i2c-designware-platform.ko 2024-08-03T20:36:57.774 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/i2c/busses/i2c-diolan-u2c.ko 2024-08-03T20:36:57.774 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/i2c/busses/i2c-i801.ko 2024-08-03T20:36:57.774 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/i2c/busses/i2c-isch.ko 2024-08-03T20:36:57.774 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/i2c/busses/i2c-ismt.ko 2024-08-03T20:36:57.774 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/i2c/busses/i2c-nforce2-s4985.ko 2024-08-03T20:36:57.774 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/i2c/busses/i2c-nforce2.ko 2024-08-03T20:36:57.774 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/i2c/busses/i2c-parport.ko 2024-08-03T20:36:57.774 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/i2c/busses/i2c-pca-platform.ko 2024-08-03T20:36:57.774 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/i2c/busses/i2c-piix4.ko 2024-08-03T20:36:57.774 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/i2c/busses/i2c-scmi.ko 2024-08-03T20:36:57.774 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/i2c/busses/i2c-simtec.ko 2024-08-03T20:36:57.775 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/i2c/busses/i2c-sis96x.ko 2024-08-03T20:36:57.775 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/i2c/busses/i2c-tiny-usb.ko 2024-08-03T20:36:57.775 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/i2c/busses/i2c-via.ko 2024-08-03T20:36:57.775 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/i2c/busses/i2c-viapro.ko 2024-08-03T20:36:57.775 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/i2c/busses/i2c-viperboard.ko 2024-08-03T20:36:57.775 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/i2c/i2c-dev.ko 2024-08-03T20:36:57.775 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/i2c/i2c-mux.ko 2024-08-03T20:36:57.775 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/i2c/i2c-slave-eeprom.ko 2024-08-03T20:36:57.775 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/i2c/i2c-smbus.ko 2024-08-03T20:36:57.775 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/i2c/i2c-stub.ko 2024-08-03T20:36:57.775 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/infiniband 2024-08-03T20:36:57.775 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/infiniband/core 2024-08-03T20:36:57.775 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/infiniband/core/ib_cm.ko 2024-08-03T20:36:57.775 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/infiniband/core/ib_core.ko 2024-08-03T20:36:57.776 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/infiniband/core/ib_umad.ko 2024-08-03T20:36:57.776 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/infiniband/core/ib_uverbs.ko 2024-08-03T20:36:57.776 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/infiniband/core/iw_cm.ko 2024-08-03T20:36:57.776 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/infiniband/core/rdma_cm.ko 2024-08-03T20:36:57.776 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/infiniband/core/rdma_ucm.ko 2024-08-03T20:36:57.776 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/infiniband/hw 2024-08-03T20:36:57.776 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/infiniband/hw/cxgb4 2024-08-03T20:36:57.776 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/infiniband/hw/cxgb4/iw_cxgb4.ko 2024-08-03T20:36:57.776 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/infiniband/hw/mlx4 2024-08-03T20:36:57.776 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/infiniband/hw/mlx4/mlx4_ib.ko 2024-08-03T20:36:57.776 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/infiniband/hw/mlx5 2024-08-03T20:36:57.776 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/infiniband/hw/mlx5/mlx5_ib.ko 2024-08-03T20:36:57.776 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/infiniband/hw/mthca 2024-08-03T20:36:57.776 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/infiniband/hw/mthca/ib_mthca.ko 2024-08-03T20:36:57.777 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/infiniband/ulp 2024-08-03T20:36:57.777 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/infiniband/ulp/ipoib 2024-08-03T20:36:57.777 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/infiniband/ulp/ipoib/ib_ipoib.ko 2024-08-03T20:36:57.777 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/infiniband/ulp/iser 2024-08-03T20:36:57.777 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/infiniband/ulp/iser/ib_iser.ko 2024-08-03T20:36:57.777 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/infiniband/ulp/isert 2024-08-03T20:36:57.777 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/infiniband/ulp/isert/ib_isert.ko 2024-08-03T20:36:57.777 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/infiniband/ulp/srp 2024-08-03T20:36:57.777 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/infiniband/ulp/srp/ib_srp.ko 2024-08-03T20:36:57.777 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/infiniband/ulp/srpt 2024-08-03T20:36:57.777 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/infiniband/ulp/srpt/ib_srpt.ko 2024-08-03T20:36:57.777 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/input 2024-08-03T20:36:57.777 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/input/gameport 2024-08-03T20:36:57.777 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/input/gameport/emu10k1-gp.ko 2024-08-03T20:36:57.778 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/input/gameport/fm801-gp.ko 2024-08-03T20:36:57.778 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/input/gameport/gameport.ko 2024-08-03T20:36:57.778 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/input/joydev.ko 2024-08-03T20:36:57.778 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/input/keyboard 2024-08-03T20:36:57.778 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/input/keyboard/gpio_keys.ko 2024-08-03T20:36:57.778 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/input/mouse 2024-08-03T20:36:57.778 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/input/mouse/appletouch.ko 2024-08-03T20:36:57.778 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/input/mouse/bcm5974.ko 2024-08-03T20:36:57.778 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/input/mouse/cyapatp.ko 2024-08-03T20:36:57.778 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/input/mouse/elan_i2c.ko 2024-08-03T20:36:57.778 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/input/mouse/sermouse.ko 2024-08-03T20:36:57.779 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/input/mouse/synaptics_i2c.ko 2024-08-03T20:36:57.779 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/input/mouse/synaptics_usb.ko 2024-08-03T20:36:57.779 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/input/mouse/vsxxxaa.ko 2024-08-03T20:36:57.779 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/input/rmi4 2024-08-03T20:36:57.779 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/input/rmi4/rmi_core.ko 2024-08-03T20:36:57.779 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/input/serio 2024-08-03T20:36:57.779 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/input/serio/altera_ps2.ko 2024-08-03T20:36:57.779 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/input/serio/arc_ps2.ko 2024-08-03T20:36:57.779 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/input/serio/hyperv-keyboard.ko 2024-08-03T20:36:57.779 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/input/serio/serio_raw.ko 2024-08-03T20:36:57.780 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/input/sparse-keymap.ko 2024-08-03T20:36:57.780 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/leds 2024-08-03T20:36:57.780 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/leds/leds-blinkm.ko 2024-08-03T20:36:57.780 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/leds/leds-lm3530.ko 2024-08-03T20:36:57.780 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/leds/leds-lp3944.ko 2024-08-03T20:36:57.782 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/leds/leds-ss4200.ko 2024-08-03T20:36:57.783 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/leds/trigger 2024-08-03T20:36:57.783 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/leds/trigger/ledtrig-backlight.ko 2024-08-03T20:36:57.783 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/leds/trigger/ledtrig-camera.ko 2024-08-03T20:36:57.783 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/leds/trigger/ledtrig-default-on.ko 2024-08-03T20:36:57.783 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/leds/trigger/ledtrig-gpio.ko 2024-08-03T20:36:57.783 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/leds/trigger/ledtrig-heartbeat.ko 2024-08-03T20:36:57.783 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/leds/trigger/ledtrig-oneshot.ko 2024-08-03T20:36:57.783 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/leds/trigger/ledtrig-timer.ko 2024-08-03T20:36:57.783 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/leds/trigger/ledtrig-transient.ko 2024-08-03T20:36:57.783 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/md 2024-08-03T20:36:57.783 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/md/bcache 2024-08-03T20:36:57.784 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/md/bcache/bcache.ko 2024-08-03T20:36:57.784 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/md/dm-bio-prison.ko 2024-08-03T20:36:57.784 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/md/dm-cache-smq.ko 2024-08-03T20:36:57.784 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/md/dm-cache.ko 2024-08-03T20:36:57.784 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/md/dm-crypt.ko 2024-08-03T20:36:57.784 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/md/dm-delay.ko 2024-08-03T20:36:57.784 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/md/dm-flakey.ko 2024-08-03T20:36:57.784 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/md/dm-log-userspace.ko 2024-08-03T20:36:57.784 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/md/dm-multipath.ko 2024-08-03T20:36:57.784 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/md/dm-queue-length.ko 2024-08-03T20:36:57.784 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/md/dm-raid.ko 2024-08-03T20:36:57.784 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/md/dm-round-robin.ko 2024-08-03T20:36:57.784 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/md/dm-service-time.ko 2024-08-03T20:36:57.785 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/md/dm-switch.ko 2024-08-03T20:36:57.785 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/md/dm-thin-pool.ko 2024-08-03T20:36:57.785 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/md/dm-verity.ko 2024-08-03T20:36:57.785 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/md/persistent-data 2024-08-03T20:36:57.785 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/md/persistent-data/dm-persistent-data.ko 2024-08-03T20:36:57.785 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/md/raid0.ko 2024-08-03T20:36:57.785 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/md/raid1.ko 2024-08-03T20:36:57.785 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/md/raid10.ko 2024-08-03T20:36:57.785 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/md/raid456.ko 2024-08-03T20:36:57.785 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/message 2024-08-03T20:36:57.785 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/message/fusion 2024-08-03T20:36:57.785 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/message/fusion/mptbase.ko 2024-08-03T20:36:57.785 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/message/fusion/mptctl.ko 2024-08-03T20:36:57.786 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/message/fusion/mptfc.ko 2024-08-03T20:36:57.786 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/message/fusion/mptlan.ko 2024-08-03T20:36:57.786 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/message/fusion/mptsas.ko 2024-08-03T20:36:57.786 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/message/fusion/mptscsih.ko 2024-08-03T20:36:57.786 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/message/fusion/mptspi.ko 2024-08-03T20:36:57.786 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/mfd 2024-08-03T20:36:57.786 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/mfd/lpc_ich.ko 2024-08-03T20:36:57.786 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/mfd/lpc_sch.ko 2024-08-03T20:36:57.786 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/mfd/mfd-core.ko 2024-08-03T20:36:57.786 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/mfd/sm501.ko 2024-08-03T20:36:57.786 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/mfd/viperboard.ko 2024-08-03T20:36:57.786 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/mfd/vx855.ko 2024-08-03T20:36:57.786 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/mfd/wl1273-core.ko 2024-08-03T20:36:57.787 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/misc 2024-08-03T20:36:57.787 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/misc/altera-stapl 2024-08-03T20:36:57.787 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/misc/altera-stapl/altera-stapl.ko 2024-08-03T20:36:57.787 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/misc/apds9802als.ko 2024-08-03T20:36:57.787 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/misc/apds990x.ko 2024-08-03T20:36:57.787 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/misc/bh1770glc.ko 2024-08-03T20:36:57.787 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/misc/cb710 2024-08-03T20:36:57.787 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/misc/cb710/cb710.ko 2024-08-03T20:36:57.787 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/misc/eeprom 2024-08-03T20:36:57.787 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/misc/eeprom/at24.ko 2024-08-03T20:36:57.787 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/misc/eeprom/eeprom_93cx6.ko 2024-08-03T20:36:57.787 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/misc/eeprom/max6875.ko 2024-08-03T20:36:57.787 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/misc/enclosure.ko 2024-08-03T20:36:57.787 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/misc/hpilo.ko 2024-08-03T20:36:57.788 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/misc/isl29003.ko 2024-08-03T20:36:57.788 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/misc/isl29020.ko 2024-08-03T20:36:57.788 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/misc/lis3lv02d 2024-08-03T20:36:57.788 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/misc/lis3lv02d/lis3lv02d.ko 2024-08-03T20:36:57.788 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/misc/lis3lv02d/lis3lv02d_i2c.ko 2024-08-03T20:36:57.788 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/misc/mei 2024-08-03T20:36:57.788 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/misc/mei/mei-me.ko 2024-08-03T20:36:57.788 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/misc/mei/mei-txe.ko 2024-08-03T20:36:57.788 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/misc/mei/mei.ko 2024-08-03T20:36:57.788 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/misc/sgi-gru 2024-08-03T20:36:57.788 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/misc/sgi-gru/gru.ko 2024-08-03T20:36:57.789 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/misc/sgi-xp 2024-08-03T20:36:57.789 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/misc/sgi-xp/xp.ko 2024-08-03T20:36:57.789 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/misc/sgi-xp/xpc.ko 2024-08-03T20:36:57.789 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/misc/sgi-xp/xpnet.ko 2024-08-03T20:36:57.789 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/misc/tifm_7xx1.ko 2024-08-03T20:36:57.789 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/misc/tifm_core.ko 2024-08-03T20:36:57.789 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/misc/tsl2550.ko 2024-08-03T20:36:57.789 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/misc/vmw_balloon.ko 2024-08-03T20:36:57.789 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/misc/vmw_vmci 2024-08-03T20:36:57.789 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/misc/vmw_vmci/vmw_vmci.ko 2024-08-03T20:36:57.789 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net 2024-08-03T20:36:57.789 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/bonding 2024-08-03T20:36:57.789 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/bonding/bonding.ko 2024-08-03T20:36:57.790 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/dsa 2024-08-03T20:36:57.790 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/dsa/b53 2024-08-03T20:36:57.790 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/dsa/b53/b53_common.ko 2024-08-03T20:36:57.790 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/dsa/bcm-sf2.ko 2024-08-03T20:36:57.790 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/dsa/mv88e6060.ko 2024-08-03T20:36:57.790 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/dsa/mv88e6xxx 2024-08-03T20:36:57.790 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/dsa/mv88e6xxx/mv88e6xxx.ko 2024-08-03T20:36:57.790 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/dummy.ko 2024-08-03T20:36:57.790 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/eql.ko 2024-08-03T20:36:57.790 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/ethernet 2024-08-03T20:36:57.790 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/ethernet/3com 2024-08-03T20:36:57.790 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/ethernet/3com/3c574_cs.ko 2024-08-03T20:36:57.790 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/ethernet/3com/3c589_cs.ko 2024-08-03T20:36:57.790 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/ethernet/3com/3c59x.ko 2024-08-03T20:36:57.791 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/ethernet/3com/typhoon.ko 2024-08-03T20:36:57.791 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/ethernet/8390 2024-08-03T20:36:57.791 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/ethernet/8390/8390.ko 2024-08-03T20:36:57.791 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/ethernet/8390/axnet_cs.ko 2024-08-03T20:36:57.791 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/ethernet/8390/ne2k-pci.ko 2024-08-03T20:36:57.791 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/ethernet/8390/pcnet_cs.ko 2024-08-03T20:36:57.791 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/ethernet/adaptec 2024-08-03T20:36:57.791 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/ethernet/adaptec/starfire.ko 2024-08-03T20:36:57.791 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/ethernet/agere 2024-08-03T20:36:57.791 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/ethernet/agere/et131x.ko 2024-08-03T20:36:57.791 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/ethernet/alteon 2024-08-03T20:36:57.791 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/ethernet/alteon/acenic.ko 2024-08-03T20:36:57.791 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/ethernet/altera 2024-08-03T20:36:57.792 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/ethernet/altera/altera_tse.ko 2024-08-03T20:36:57.792 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/ethernet/amd 2024-08-03T20:36:57.792 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/ethernet/amd/amd8111e.ko 2024-08-03T20:36:57.792 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/ethernet/amd/nmclan_cs.ko 2024-08-03T20:36:57.792 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/ethernet/amd/pcnet32.ko 2024-08-03T20:36:57.792 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/ethernet/atheros 2024-08-03T20:36:57.792 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/ethernet/atheros/alx 2024-08-03T20:36:57.792 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/ethernet/atheros/alx/alx.ko 2024-08-03T20:36:57.792 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/ethernet/atheros/atl1c 2024-08-03T20:36:57.792 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/ethernet/atheros/atl1c/atl1c.ko 2024-08-03T20:36:57.792 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/ethernet/atheros/atl1e 2024-08-03T20:36:57.793 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/ethernet/atheros/atl1e/atl1e.ko 2024-08-03T20:36:57.793 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/ethernet/atheros/atlx 2024-08-03T20:36:57.793 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/ethernet/atheros/atlx/atl1.ko 2024-08-03T20:36:57.793 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/ethernet/atheros/atlx/atl2.ko 2024-08-03T20:36:57.793 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/ethernet/broadcom 2024-08-03T20:36:57.793 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/ethernet/broadcom/b44.ko 2024-08-03T20:36:57.793 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/ethernet/broadcom/bnx2.ko 2024-08-03T20:36:57.793 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/ethernet/broadcom/bnx2x 2024-08-03T20:36:57.793 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/ethernet/broadcom/bnx2x/bnx2x.ko 2024-08-03T20:36:57.793 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/ethernet/broadcom/cnic.ko 2024-08-03T20:36:57.793 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/ethernet/broadcom/genet 2024-08-03T20:36:57.793 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/ethernet/broadcom/genet/genet.ko 2024-08-03T20:36:57.793 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/ethernet/broadcom/tg3.ko 2024-08-03T20:36:57.794 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/ethernet/brocade 2024-08-03T20:36:57.794 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/ethernet/brocade/bna 2024-08-03T20:36:57.794 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/ethernet/brocade/bna/bna.ko 2024-08-03T20:36:57.794 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/ethernet/chelsio 2024-08-03T20:36:57.794 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/ethernet/chelsio/cxgb 2024-08-03T20:36:57.794 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/ethernet/chelsio/cxgb/cxgb.ko 2024-08-03T20:36:57.794 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/ethernet/chelsio/cxgb3 2024-08-03T20:36:57.794 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/ethernet/chelsio/cxgb3/cxgb3.ko 2024-08-03T20:36:57.794 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/ethernet/chelsio/cxgb4 2024-08-03T20:36:57.794 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/ethernet/chelsio/cxgb4/cxgb4.ko 2024-08-03T20:36:57.794 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/ethernet/chelsio/cxgb4vf 2024-08-03T20:36:57.794 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/ethernet/chelsio/cxgb4vf/cxgb4vf.ko 2024-08-03T20:36:57.794 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/ethernet/chelsio/libcxgb 2024-08-03T20:36:57.795 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/ethernet/chelsio/libcxgb/libcxgb.ko 2024-08-03T20:36:57.795 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/ethernet/cisco 2024-08-03T20:36:57.795 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/ethernet/cisco/enic 2024-08-03T20:36:57.795 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/ethernet/cisco/enic/enic.ko 2024-08-03T20:36:57.795 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/ethernet/dec 2024-08-03T20:36:57.795 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/ethernet/dec/tulip 2024-08-03T20:36:57.795 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/ethernet/dec/tulip/de2104x.ko 2024-08-03T20:36:57.795 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/ethernet/dec/tulip/dmfe.ko 2024-08-03T20:36:57.795 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/ethernet/dec/tulip/tulip.ko 2024-08-03T20:36:57.795 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/ethernet/dec/tulip/uli526x.ko 2024-08-03T20:36:57.795 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/ethernet/dec/tulip/winbond-840.ko 2024-08-03T20:36:57.795 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/ethernet/dec/tulip/xircom_cb.ko 2024-08-03T20:36:57.795 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/ethernet/dlink 2024-08-03T20:36:57.795 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/ethernet/dlink/dl2k.ko 2024-08-03T20:36:57.796 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/ethernet/dlink/sundance.ko 2024-08-03T20:36:57.796 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/ethernet/dnet.ko 2024-08-03T20:36:57.796 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/ethernet/emulex 2024-08-03T20:36:57.796 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/ethernet/emulex/benet 2024-08-03T20:36:57.796 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/ethernet/emulex/benet/be2net.ko 2024-08-03T20:36:57.796 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/ethernet/ethoc.ko 2024-08-03T20:36:57.796 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/ethernet/fealnx.ko 2024-08-03T20:36:57.796 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/ethernet/intel 2024-08-03T20:36:57.796 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/ethernet/intel/e100.ko 2024-08-03T20:36:57.796 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/ethernet/intel/e1000 2024-08-03T20:36:57.796 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/ethernet/intel/e1000/e1000.ko 2024-08-03T20:36:57.796 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/ethernet/intel/e1000e 2024-08-03T20:36:57.796 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/ethernet/intel/e1000e/e1000e.ko 2024-08-03T20:36:57.797 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/ethernet/intel/fm10k 2024-08-03T20:36:57.797 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/ethernet/intel/fm10k/fm10k.ko 2024-08-03T20:36:57.797 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/ethernet/intel/i40e 2024-08-03T20:36:57.797 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/ethernet/intel/i40e/i40e.ko 2024-08-03T20:36:57.797 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/ethernet/intel/iavf 2024-08-03T20:36:57.797 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/ethernet/intel/iavf/iavf.ko 2024-08-03T20:36:57.797 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/ethernet/intel/igb 2024-08-03T20:36:57.797 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/ethernet/intel/igb/igb.ko 2024-08-03T20:36:57.797 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/ethernet/intel/igbvf 2024-08-03T20:36:57.797 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/ethernet/intel/igbvf/igbvf.ko 2024-08-03T20:36:57.797 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/ethernet/intel/ixgbe 2024-08-03T20:36:57.798 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/ethernet/intel/ixgbe/ixgbe.ko 2024-08-03T20:36:57.798 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/ethernet/intel/ixgbevf 2024-08-03T20:36:57.798 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/ethernet/intel/ixgbevf/ixgbevf.ko 2024-08-03T20:36:57.798 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/ethernet/intel/libeth 2024-08-03T20:36:57.798 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/ethernet/intel/libeth/libeth.ko 2024-08-03T20:36:57.798 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/ethernet/intel/libie 2024-08-03T20:36:57.798 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/ethernet/intel/libie/libie.ko 2024-08-03T20:36:57.798 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/ethernet/jme.ko 2024-08-03T20:36:57.798 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/ethernet/marvell 2024-08-03T20:36:57.798 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/ethernet/marvell/mvmdio.ko 2024-08-03T20:36:57.799 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/ethernet/marvell/skge.ko 2024-08-03T20:36:57.799 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/ethernet/marvell/sky2.ko 2024-08-03T20:36:57.799 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/ethernet/mellanox 2024-08-03T20:36:57.799 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/ethernet/mellanox/mlx4 2024-08-03T20:36:57.799 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_core.ko 2024-08-03T20:36:57.799 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_en.ko 2024-08-03T20:36:57.799 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/ethernet/mellanox/mlx5 2024-08-03T20:36:57.799 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/ethernet/mellanox/mlx5/core 2024-08-03T20:36:57.799 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/ethernet/mellanox/mlx5/core/mlx5_core.ko 2024-08-03T20:36:57.800 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/ethernet/micrel 2024-08-03T20:36:57.800 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/ethernet/micrel/ksz884x.ko 2024-08-03T20:36:57.800 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/ethernet/myricom 2024-08-03T20:36:57.800 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/ethernet/myricom/myri10ge 2024-08-03T20:36:57.800 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/ethernet/myricom/myri10ge/myri10ge.ko 2024-08-03T20:36:57.800 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/ethernet/natsemi 2024-08-03T20:36:57.800 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/ethernet/natsemi/natsemi.ko 2024-08-03T20:36:57.800 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/ethernet/natsemi/ns83820.ko 2024-08-03T20:36:57.801 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/ethernet/neterion 2024-08-03T20:36:57.801 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/ethernet/neterion/s2io.ko 2024-08-03T20:36:57.801 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/ethernet/nvidia 2024-08-03T20:36:57.801 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/ethernet/nvidia/forcedeth.ko 2024-08-03T20:36:57.801 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/ethernet/packetengines 2024-08-03T20:36:57.801 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/ethernet/packetengines/hamachi.ko 2024-08-03T20:36:57.801 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/ethernet/packetengines/yellowfin.ko 2024-08-03T20:36:57.801 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/ethernet/qlogic 2024-08-03T20:36:57.801 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/ethernet/qlogic/netxen 2024-08-03T20:36:57.802 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/ethernet/qlogic/netxen/netxen_nic.ko 2024-08-03T20:36:57.802 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/ethernet/qlogic/qla3xxx.ko 2024-08-03T20:36:57.802 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/ethernet/qlogic/qlcnic 2024-08-03T20:36:57.802 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/ethernet/qlogic/qlcnic/qlcnic.ko 2024-08-03T20:36:57.802 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/ethernet/rdc 2024-08-03T20:36:57.802 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/ethernet/rdc/r6040.ko 2024-08-03T20:36:57.802 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/ethernet/realtek 2024-08-03T20:36:57.802 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/ethernet/realtek/8139cp.ko 2024-08-03T20:36:57.803 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/ethernet/realtek/8139too.ko 2024-08-03T20:36:57.803 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/ethernet/realtek/atp.ko 2024-08-03T20:36:57.803 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/ethernet/realtek/r8169.ko 2024-08-03T20:36:57.803 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/ethernet/rocker 2024-08-03T20:36:57.803 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/ethernet/rocker/rocker.ko 2024-08-03T20:36:57.803 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/ethernet/sfc 2024-08-03T20:36:57.803 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/ethernet/sfc/sfc.ko 2024-08-03T20:36:57.803 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/ethernet/silan 2024-08-03T20:36:57.804 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/ethernet/silan/sc92031.ko 2024-08-03T20:36:57.804 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/ethernet/sis 2024-08-03T20:36:57.804 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/ethernet/sis/sis190.ko 2024-08-03T20:36:57.804 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/ethernet/sis/sis900.ko 2024-08-03T20:36:57.804 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/ethernet/smsc 2024-08-03T20:36:57.804 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/ethernet/smsc/epic100.ko 2024-08-03T20:36:57.804 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/ethernet/smsc/smc91c92_cs.ko 2024-08-03T20:36:57.804 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/ethernet/smsc/smsc911x.ko 2024-08-03T20:36:57.805 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/ethernet/smsc/smsc9420.ko 2024-08-03T20:36:57.805 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/ethernet/stmicro 2024-08-03T20:36:57.805 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/ethernet/stmicro/stmmac 2024-08-03T20:36:57.805 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/ethernet/stmicro/stmmac/dwmac-intel.ko 2024-08-03T20:36:57.805 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/ethernet/stmicro/stmmac/stmmac.ko 2024-08-03T20:36:57.805 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/ethernet/sun 2024-08-03T20:36:57.805 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/ethernet/sun/cassini.ko 2024-08-03T20:36:57.805 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/ethernet/sun/niu.ko 2024-08-03T20:36:57.805 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/ethernet/sun/sungem.ko 2024-08-03T20:36:57.806 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/ethernet/sun/sunhme.ko 2024-08-03T20:36:57.806 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/ethernet/tehuti 2024-08-03T20:36:57.806 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/ethernet/tehuti/tehuti.ko 2024-08-03T20:36:57.806 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/ethernet/ti 2024-08-03T20:36:57.806 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/ethernet/ti/tlan.ko 2024-08-03T20:36:57.806 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/ethernet/via 2024-08-03T20:36:57.806 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/ethernet/via/via-rhine.ko 2024-08-03T20:36:57.807 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/ethernet/via/via-velocity.ko 2024-08-03T20:36:57.807 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/ethernet/wiznet 2024-08-03T20:36:57.807 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/ethernet/wiznet/w5100.ko 2024-08-03T20:36:57.807 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/ethernet/wiznet/w5300.ko 2024-08-03T20:36:57.807 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/ethernet/xircom 2024-08-03T20:36:57.807 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/ethernet/xircom/xirc2ps_cs.ko 2024-08-03T20:36:57.807 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/geneve.ko 2024-08-03T20:36:57.807 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/hyperv 2024-08-03T20:36:57.808 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/hyperv/hv_netvsc.ko 2024-08-03T20:36:57.808 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/ieee802154 2024-08-03T20:36:57.808 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/ieee802154/fakelb.ko 2024-08-03T20:36:57.808 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/ifb.ko 2024-08-03T20:36:57.808 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/ipvlan 2024-08-03T20:36:57.808 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/ipvlan/ipvlan.ko 2024-08-03T20:36:57.808 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/macvlan.ko 2024-08-03T20:36:57.808 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/macvtap.ko 2024-08-03T20:36:57.808 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/mdio 2024-08-03T20:36:57.809 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/mdio.ko 2024-08-03T20:36:57.809 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/mdio/mdio-bcm-unimac.ko 2024-08-03T20:36:57.809 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/mdio/mdio-bitbang.ko 2024-08-03T20:36:57.809 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/mdio/mdio-regmap.ko 2024-08-03T20:36:57.809 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/mii.ko 2024-08-03T20:36:57.809 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/net_failover.ko 2024-08-03T20:36:57.809 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/netconsole.ko 2024-08-03T20:36:57.809 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/nlmon.ko 2024-08-03T20:36:57.810 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/pcs 2024-08-03T20:36:57.810 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/pcs/pcs-lynx.ko 2024-08-03T20:36:57.810 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/pcs/pcs_xpcs.ko 2024-08-03T20:36:57.810 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/phy 2024-08-03T20:36:57.810 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/phy/amd.ko 2024-08-03T20:36:57.810 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/phy/bcm-phy-lib.ko 2024-08-03T20:36:57.810 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/phy/bcm-phy-ptp.ko 2024-08-03T20:36:57.810 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/phy/bcm7xxx.ko 2024-08-03T20:36:57.811 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/phy/bcm87xx.ko 2024-08-03T20:36:57.811 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/phy/broadcom.ko 2024-08-03T20:36:57.811 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/phy/cicada.ko 2024-08-03T20:36:57.811 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/phy/davicom.ko 2024-08-03T20:36:57.811 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/phy/dp83640.ko 2024-08-03T20:36:57.811 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/phy/et1011c.ko 2024-08-03T20:36:57.811 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/phy/icplus.ko 2024-08-03T20:36:57.811 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/phy/lxt.ko 2024-08-03T20:36:57.811 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/phy/marvell.ko 2024-08-03T20:36:57.812 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/phy/micrel.ko 2024-08-03T20:36:57.812 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/phy/national.ko 2024-08-03T20:36:57.812 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/phy/phylink.ko 2024-08-03T20:36:57.812 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/phy/qsemi.ko 2024-08-03T20:36:57.812 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/phy/realtek.ko 2024-08-03T20:36:57.812 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/phy/smsc.ko 2024-08-03T20:36:57.812 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/phy/ste10Xp.ko 2024-08-03T20:36:57.812 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/phy/vitesse.ko 2024-08-03T20:36:57.813 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/ppp 2024-08-03T20:36:57.813 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/ppp/bsd_comp.ko 2024-08-03T20:36:57.813 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/ppp/ppp_async.ko 2024-08-03T20:36:57.813 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/ppp/ppp_deflate.ko 2024-08-03T20:36:57.813 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/ppp/ppp_generic.ko 2024-08-03T20:36:57.813 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/ppp/ppp_mppe.ko 2024-08-03T20:36:57.813 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/ppp/ppp_synctty.ko 2024-08-03T20:36:57.813 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/ppp/pppoe.ko 2024-08-03T20:36:57.814 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/ppp/pppox.ko 2024-08-03T20:36:57.814 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/ppp/pptp.ko 2024-08-03T20:36:57.814 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/slip 2024-08-03T20:36:57.814 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/slip/slhc.ko 2024-08-03T20:36:57.814 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/slip/slip.ko 2024-08-03T20:36:57.814 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/sungem_phy.ko 2024-08-03T20:36:57.814 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/tap.ko 2024-08-03T20:36:57.814 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/team 2024-08-03T20:36:57.814 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/team/team.ko 2024-08-03T20:36:57.814 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/team/team_mode_activebackup.ko 2024-08-03T20:36:57.814 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/team/team_mode_broadcast.ko 2024-08-03T20:36:57.814 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/team/team_mode_loadbalance.ko 2024-08-03T20:36:57.815 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/team/team_mode_random.ko 2024-08-03T20:36:57.815 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/team/team_mode_roundrobin.ko 2024-08-03T20:36:57.815 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/tun.ko 2024-08-03T20:36:57.815 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/veth.ko 2024-08-03T20:36:57.815 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/virtio_net.ko 2024-08-03T20:36:57.815 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/vmxnet3 2024-08-03T20:36:57.815 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/vmxnet3/vmxnet3.ko 2024-08-03T20:36:57.815 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/vxlan 2024-08-03T20:36:57.815 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/vxlan/vxlan.ko 2024-08-03T20:36:57.815 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/xen-netback 2024-08-03T20:36:57.815 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/xen-netback/xen-netback.ko 2024-08-03T20:36:57.815 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/net/xen-netfront.ko 2024-08-03T20:36:57.815 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/ntb 2024-08-03T20:36:57.816 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/ntb/ntb.ko 2024-08-03T20:36:57.816 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/nvme 2024-08-03T20:36:57.816 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/nvme/host 2024-08-03T20:36:57.816 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/nvme/host/nvme-core.ko 2024-08-03T20:36:57.816 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/nvme/host/nvme.ko 2024-08-03T20:36:57.816 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/parport 2024-08-03T20:36:57.816 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/parport/parport.ko 2024-08-03T20:36:57.816 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/parport/parport_cs.ko 2024-08-03T20:36:57.816 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/parport/parport_pc.ko 2024-08-03T20:36:57.816 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/parport/parport_serial.ko 2024-08-03T20:36:57.816 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/pci 2024-08-03T20:36:57.816 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/pci/hotplug 2024-08-03T20:36:57.816 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/pci/hotplug/acpiphp_ibm.ko 2024-08-03T20:36:57.816 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/pci/pcie 2024-08-03T20:36:57.817 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/pci/pcie/aer_inject.ko 2024-08-03T20:36:57.817 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/pci/xen-pcifront.ko 2024-08-03T20:36:57.817 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/pcmcia 2024-08-03T20:36:57.817 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/pcmcia/i82092.ko 2024-08-03T20:36:57.817 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/pcmcia/pd6729.ko 2024-08-03T20:36:57.817 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/pcmcia/yenta_socket.ko 2024-08-03T20:36:57.817 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/pinctrl 2024-08-03T20:36:57.817 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/pinctrl/intel 2024-08-03T20:36:57.817 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/pinctrl/intel/pinctrl-cherryview.ko 2024-08-03T20:36:57.817 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/platform 2024-08-03T20:36:57.817 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/platform/x86 2024-08-03T20:36:57.817 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/platform/x86/acer-wmi.ko 2024-08-03T20:36:57.817 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/platform/x86/acerhdf.ko 2024-08-03T20:36:57.817 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/platform/x86/apple-gmux.ko 2024-08-03T20:36:57.818 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/platform/x86/asus-laptop.ko 2024-08-03T20:36:57.818 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/platform/x86/asus-nb-wmi.ko 2024-08-03T20:36:57.818 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/platform/x86/asus-wmi.ko 2024-08-03T20:36:57.818 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/platform/x86/classmate-laptop.ko 2024-08-03T20:36:57.818 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/platform/x86/eeepc-laptop.ko 2024-08-03T20:36:57.818 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/platform/x86/eeepc-wmi.ko 2024-08-03T20:36:57.818 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/platform/x86/fujitsu-laptop.ko 2024-08-03T20:36:57.818 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/platform/x86/fujitsu-tablet.ko 2024-08-03T20:36:57.818 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/platform/x86/hdaps.ko 2024-08-03T20:36:57.818 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/platform/x86/intel 2024-08-03T20:36:57.818 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/platform/x86/intel/intel-rst.ko 2024-08-03T20:36:57.818 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/platform/x86/intel_ips.ko 2024-08-03T20:36:57.818 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/platform/x86/msi-wmi.ko 2024-08-03T20:36:57.819 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/platform/x86/mxm-wmi.ko 2024-08-03T20:36:57.819 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/platform/x86/panasonic-laptop.ko 2024-08-03T20:36:57.819 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/platform/x86/samsung-laptop.ko 2024-08-03T20:36:57.819 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/platform/x86/samsung-q10.ko 2024-08-03T20:36:57.819 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/platform/x86/topstar-laptop.ko 2024-08-03T20:36:57.819 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/platform/x86/toshiba_bluetooth.ko 2024-08-03T20:36:57.819 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/platform/x86/toshiba_haps.ko 2024-08-03T20:36:57.819 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/platform/x86/wmi-bmof.ko 2024-08-03T20:36:57.819 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/platform/x86/wmi.ko 2024-08-03T20:36:57.819 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/powercap 2024-08-03T20:36:57.819 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/powercap/intel_rapl_common.ko 2024-08-03T20:36:57.819 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/powercap/intel_rapl_msr.ko 2024-08-03T20:36:57.819 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/pps 2024-08-03T20:36:57.819 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/pps/clients 2024-08-03T20:36:57.820 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/pps/clients/pps-gpio.ko 2024-08-03T20:36:57.820 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/pps/clients/pps-ldisc.ko 2024-08-03T20:36:57.820 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/pps/clients/pps_parport.ko 2024-08-03T20:36:57.820 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/pps/pps_core.ko 2024-08-03T20:36:57.820 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/ptp 2024-08-03T20:36:57.820 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/ptp/ptp.ko 2024-08-03T20:36:57.820 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/ptp/ptp_kvm.ko 2024-08-03T20:36:57.820 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/ras 2024-08-03T20:36:57.820 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/ras/amd 2024-08-03T20:36:57.820 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/ras/amd/atl 2024-08-03T20:36:57.820 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/ras/amd/atl/amd_atl.ko 2024-08-03T20:36:57.820 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/ras/amd/fmpm.ko 2024-08-03T20:36:57.821 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/rtc 2024-08-03T20:36:57.821 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/rtc/rtc-bq32k.ko 2024-08-03T20:36:57.821 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/rtc/rtc-ds1286.ko 2024-08-03T20:36:57.821 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/rtc/rtc-ds1307.ko 2024-08-03T20:36:57.821 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/rtc/rtc-ds1374.ko 2024-08-03T20:36:57.821 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/rtc/rtc-ds1511.ko 2024-08-03T20:36:57.821 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/rtc/rtc-ds1553.ko 2024-08-03T20:36:57.821 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/rtc/rtc-ds1672.ko 2024-08-03T20:36:57.821 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/rtc/rtc-ds1742.ko 2024-08-03T20:36:57.821 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/rtc/rtc-ds2404.ko 2024-08-03T20:36:57.821 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/rtc/rtc-ds3232.ko 2024-08-03T20:36:57.821 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/rtc/rtc-em3027.ko 2024-08-03T20:36:57.821 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/rtc/rtc-fm3130.ko 2024-08-03T20:36:57.821 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/rtc/rtc-isl12022.ko 2024-08-03T20:36:57.822 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/rtc/rtc-isl1208.ko 2024-08-03T20:36:57.822 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/rtc/rtc-m41t80.ko 2024-08-03T20:36:57.822 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/rtc/rtc-m48t35.ko 2024-08-03T20:36:57.822 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/rtc/rtc-m48t59.ko 2024-08-03T20:36:57.822 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/rtc/rtc-max6900.ko 2024-08-03T20:36:57.822 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/rtc/rtc-msm6242.ko 2024-08-03T20:36:57.822 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/rtc/rtc-pcf2127.ko 2024-08-03T20:36:57.822 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/rtc/rtc-pcf85063.ko 2024-08-03T20:36:57.822 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/rtc/rtc-pcf8523.ko 2024-08-03T20:36:57.822 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/rtc/rtc-pcf8563.ko 2024-08-03T20:36:57.822 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/rtc/rtc-pcf8583.ko 2024-08-03T20:36:57.822 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/rtc/rtc-rp5c01.ko 2024-08-03T20:36:57.822 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/rtc/rtc-rs5c372.ko 2024-08-03T20:36:57.822 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/rtc/rtc-rv3029c2.ko 2024-08-03T20:36:57.823 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/rtc/rtc-rx8025.ko 2024-08-03T20:36:57.823 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/rtc/rtc-rx8581.ko 2024-08-03T20:36:57.823 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/rtc/rtc-stk17ta8.ko 2024-08-03T20:36:57.823 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/rtc/rtc-x1205.ko 2024-08-03T20:36:57.823 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/scsi 2024-08-03T20:36:57.823 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/scsi/3w-9xxx.ko 2024-08-03T20:36:57.823 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/scsi/3w-sas.ko 2024-08-03T20:36:57.823 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/scsi/3w-xxxx.ko 2024-08-03T20:36:57.823 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/scsi/BusLogic.ko 2024-08-03T20:36:57.823 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/scsi/a100u2w.ko 2024-08-03T20:36:57.823 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/scsi/aacraid 2024-08-03T20:36:57.823 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/scsi/aacraid/aacraid.ko 2024-08-03T20:36:57.823 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/scsi/advansys.ko 2024-08-03T20:36:57.824 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/scsi/aic7xxx 2024-08-03T20:36:57.824 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/scsi/aic7xxx/aic79xx.ko 2024-08-03T20:36:57.824 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/scsi/aic7xxx/aic7xxx.ko 2024-08-03T20:36:57.824 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/scsi/am53c974.ko 2024-08-03T20:36:57.824 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/scsi/arcmsr 2024-08-03T20:36:57.824 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/scsi/arcmsr/arcmsr.ko 2024-08-03T20:36:57.824 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/scsi/atp870u.ko 2024-08-03T20:36:57.824 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/scsi/be2iscsi 2024-08-03T20:36:57.824 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/scsi/be2iscsi/be2iscsi.ko 2024-08-03T20:36:57.824 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/scsi/bfa 2024-08-03T20:36:57.824 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/scsi/bfa/bfa.ko 2024-08-03T20:36:57.824 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/scsi/bnx2fc 2024-08-03T20:36:57.824 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/scsi/bnx2fc/bnx2fc.ko 2024-08-03T20:36:57.825 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/scsi/bnx2i 2024-08-03T20:36:57.825 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/scsi/bnx2i/bnx2i.ko 2024-08-03T20:36:57.825 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/scsi/ch.ko 2024-08-03T20:36:57.825 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/scsi/csiostor 2024-08-03T20:36:57.825 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/scsi/csiostor/csiostor.ko 2024-08-03T20:36:57.825 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/scsi/cxgbi 2024-08-03T20:36:57.825 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/scsi/cxgbi/cxgb3i 2024-08-03T20:36:57.825 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/scsi/cxgbi/cxgb3i/cxgb3i.ko 2024-08-03T20:36:57.825 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/scsi/cxgbi/cxgb4i 2024-08-03T20:36:57.825 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/scsi/cxgbi/cxgb4i/cxgb4i.ko 2024-08-03T20:36:57.826 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/scsi/cxgbi/libcxgbi.ko 2024-08-03T20:36:57.826 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/scsi/dc395x.ko 2024-08-03T20:36:57.826 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/scsi/device_handler 2024-08-03T20:36:57.826 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/scsi/device_handler/scsi_dh_alua.ko 2024-08-03T20:36:57.826 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/scsi/device_handler/scsi_dh_emc.ko 2024-08-03T20:36:57.826 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/scsi/device_handler/scsi_dh_hp_sw.ko 2024-08-03T20:36:57.826 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/scsi/device_handler/scsi_dh_rdac.ko 2024-08-03T20:36:57.826 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/scsi/dmx3191d.ko 2024-08-03T20:36:57.826 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/scsi/esas2r 2024-08-03T20:36:57.826 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/scsi/esas2r/esas2r.ko 2024-08-03T20:36:57.826 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/scsi/esp_scsi.ko 2024-08-03T20:36:57.826 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/scsi/fcoe 2024-08-03T20:36:57.826 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/scsi/fcoe/fcoe.ko 2024-08-03T20:36:57.827 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/scsi/fcoe/libfcoe.ko 2024-08-03T20:36:57.827 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/scsi/fnic 2024-08-03T20:36:57.827 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/scsi/fnic/fnic.ko 2024-08-03T20:36:57.827 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/scsi/hpsa.ko 2024-08-03T20:36:57.827 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/scsi/hptiop.ko 2024-08-03T20:36:57.827 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/scsi/hv_storvsc.ko 2024-08-03T20:36:57.827 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/scsi/initio.ko 2024-08-03T20:36:57.827 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/scsi/ipr.ko 2024-08-03T20:36:57.827 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/scsi/ips.ko 2024-08-03T20:36:57.827 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/scsi/isci 2024-08-03T20:36:57.827 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/scsi/isci/isci.ko 2024-08-03T20:36:57.827 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/scsi/iscsi_boot_sysfs.ko 2024-08-03T20:36:57.827 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/scsi/iscsi_tcp.ko 2024-08-03T20:36:57.827 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/scsi/libfc 2024-08-03T20:36:57.827 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/scsi/libfc/libfc.ko 2024-08-03T20:36:57.828 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/scsi/libiscsi.ko 2024-08-03T20:36:57.828 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/scsi/libiscsi_tcp.ko 2024-08-03T20:36:57.828 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/scsi/libsas 2024-08-03T20:36:57.828 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/scsi/libsas/libsas.ko 2024-08-03T20:36:57.828 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/scsi/lpfc 2024-08-03T20:36:57.828 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/scsi/lpfc/lpfc.ko 2024-08-03T20:36:57.828 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/scsi/megaraid 2024-08-03T20:36:57.828 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/scsi/megaraid.ko 2024-08-03T20:36:57.828 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/scsi/megaraid/megaraid_mbox.ko 2024-08-03T20:36:57.828 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/scsi/megaraid/megaraid_mm.ko 2024-08-03T20:36:57.828 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/scsi/megaraid/megaraid_sas.ko 2024-08-03T20:36:57.828 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/scsi/mpt3sas 2024-08-03T20:36:57.828 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/scsi/mpt3sas/mpt3sas.ko 2024-08-03T20:36:57.828 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/scsi/mvsas 2024-08-03T20:36:57.829 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/scsi/mvsas/mvsas.ko 2024-08-03T20:36:57.829 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/scsi/mvumi.ko 2024-08-03T20:36:57.829 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/scsi/pm8001 2024-08-03T20:36:57.829 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/scsi/pm8001/pm80xx.ko 2024-08-03T20:36:57.829 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/scsi/pmcraid.ko 2024-08-03T20:36:57.829 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/scsi/qla1280.ko 2024-08-03T20:36:57.829 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/scsi/qla2xxx 2024-08-03T20:36:57.829 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/scsi/qla2xxx/qla2xxx.ko 2024-08-03T20:36:57.829 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/scsi/qla2xxx/tcm_qla2xxx.ko 2024-08-03T20:36:57.829 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/scsi/qla4xxx 2024-08-03T20:36:57.829 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/scsi/qla4xxx/qla4xxx.ko 2024-08-03T20:36:57.829 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/scsi/raid_class.ko 2024-08-03T20:36:57.829 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/scsi/scsi_debug.ko 2024-08-03T20:36:57.829 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/scsi/scsi_transport_fc.ko 2024-08-03T20:36:57.830 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/scsi/scsi_transport_iscsi.ko 2024-08-03T20:36:57.830 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/scsi/scsi_transport_sas.ko 2024-08-03T20:36:57.830 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/scsi/scsi_transport_spi.ko 2024-08-03T20:36:57.830 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/scsi/scsi_transport_srp.ko 2024-08-03T20:36:57.830 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/scsi/ses.ko 2024-08-03T20:36:57.830 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/scsi/st.ko 2024-08-03T20:36:57.830 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/scsi/stex.ko 2024-08-03T20:36:57.830 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/scsi/sym53c8xx_2 2024-08-03T20:36:57.830 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/scsi/sym53c8xx_2/sym53c8xx.ko 2024-08-03T20:36:57.830 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/scsi/virtio_scsi.ko 2024-08-03T20:36:57.830 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/scsi/vmw_pvscsi.ko 2024-08-03T20:36:57.830 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/scsi/wd719x.ko 2024-08-03T20:36:57.830 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/ssb 2024-08-03T20:36:57.830 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/ssb/ssb.ko 2024-08-03T20:36:57.831 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/target 2024-08-03T20:36:57.831 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/target/iscsi 2024-08-03T20:36:57.831 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/target/iscsi/iscsi_target_mod.ko 2024-08-03T20:36:57.831 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/target/loopback 2024-08-03T20:36:57.831 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/target/loopback/tcm_loop.ko 2024-08-03T20:36:57.831 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/target/target_core_file.ko 2024-08-03T20:36:57.831 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/target/target_core_iblock.ko 2024-08-03T20:36:57.831 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/target/target_core_mod.ko 2024-08-03T20:36:57.831 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/target/target_core_pscsi.ko 2024-08-03T20:36:57.831 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/target/target_core_user.ko 2024-08-03T20:36:57.831 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/target/tcm_fc 2024-08-03T20:36:57.831 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/target/tcm_fc/tcm_fc.ko 2024-08-03T20:36:57.831 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/thermal 2024-08-03T20:36:57.831 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/thermal/intel 2024-08-03T20:36:57.832 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/thermal/intel/int340x_thermal 2024-08-03T20:36:57.832 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/thermal/intel/int340x_thermal/acpi_thermal_rel.ko 2024-08-03T20:36:57.832 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/thermal/intel/int340x_thermal/int3400_thermal.ko 2024-08-03T20:36:57.832 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/thermal/intel/int340x_thermal/int3401_thermal.ko 2024-08-03T20:36:57.832 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/thermal/intel/int340x_thermal/int3402_thermal.ko 2024-08-03T20:36:57.832 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/thermal/intel/int340x_thermal/int3403_thermal.ko 2024-08-03T20:36:57.832 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/thermal/intel/int340x_thermal/int340x_thermal_zone.ko 2024-08-03T20:36:57.832 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device.ko 2024-08-03T20:36:57.832 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device_pci.ko 2024-08-03T20:36:57.832 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device_pci_legacy.ko 2024-08-03T20:36:57.832 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_mbox.ko 2024-08-03T20:36:57.832 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_power_floor.ko 2024-08-03T20:36:57.832 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_rapl.ko 2024-08-03T20:36:57.833 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_rfim.ko 2024-08-03T20:36:57.833 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_wt_hint.ko 2024-08-03T20:36:57.833 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_wt_req.ko 2024-08-03T20:36:57.833 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/thermal/intel/intel_soc_dts_iosf.ko 2024-08-03T20:36:57.833 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/thermal/intel/intel_soc_dts_thermal.ko 2024-08-03T20:36:57.833 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/thermal/intel/x86_pkg_temp_thermal.ko 2024-08-03T20:36:57.833 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/tty 2024-08-03T20:36:57.833 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/tty/ipwireless 2024-08-03T20:36:57.833 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/tty/ipwireless/ipwireless.ko 2024-08-03T20:36:57.833 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/tty/n_gsm.ko 2024-08-03T20:36:57.833 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/tty/n_hdlc.ko 2024-08-03T20:36:57.833 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/tty/nozomi.ko 2024-08-03T20:36:57.833 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/tty/serial 2024-08-03T20:36:57.833 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/tty/serial/8250 2024-08-03T20:36:57.834 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/tty/serial/8250/serial_cs.ko 2024-08-03T20:36:57.834 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/tty/serial/arc_uart.ko 2024-08-03T20:36:57.834 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/tty/serial/jsm 2024-08-03T20:36:57.834 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/tty/serial/jsm/jsm.ko 2024-08-03T20:36:57.834 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/ufs 2024-08-03T20:36:57.834 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/ufs/core 2024-08-03T20:36:57.834 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/ufs/core/ufshcd-core.ko 2024-08-03T20:36:57.834 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/ufs/host 2024-08-03T20:36:57.834 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/ufs/host/ufshcd-pci.ko 2024-08-03T20:36:57.834 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/uio 2024-08-03T20:36:57.834 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/uio/uio.ko 2024-08-03T20:36:57.834 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/uio/uio_aec.ko 2024-08-03T20:36:57.834 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/uio/uio_cif.ko 2024-08-03T20:36:57.835 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/uio/uio_pci_generic.ko 2024-08-03T20:36:57.835 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/uio/uio_sercos3.ko 2024-08-03T20:36:57.835 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/usb 2024-08-03T20:36:57.835 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/usb/class 2024-08-03T20:36:57.835 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/usb/class/cdc-acm.ko 2024-08-03T20:36:57.835 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/usb/class/cdc-wdm.ko 2024-08-03T20:36:57.835 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/usb/class/usblp.ko 2024-08-03T20:36:57.835 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/usb/class/usbtmc.ko 2024-08-03T20:36:57.835 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/usb/host 2024-08-03T20:36:57.835 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/usb/host/sl811-hcd.ko 2024-08-03T20:36:57.835 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/usb/image 2024-08-03T20:36:57.835 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/usb/image/mdc800.ko 2024-08-03T20:36:57.835 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/usb/image/microtek.ko 2024-08-03T20:36:57.835 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/usb/misc 2024-08-03T20:36:57.836 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/usb/misc/adutux.ko 2024-08-03T20:36:57.836 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/usb/misc/appledisplay.ko 2024-08-03T20:36:57.836 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/usb/misc/emi26.ko 2024-08-03T20:36:57.836 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/usb/misc/emi62.ko 2024-08-03T20:36:57.836 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/usb/misc/ezusb.ko 2024-08-03T20:36:57.836 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/usb/misc/idmouse.ko 2024-08-03T20:36:57.836 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/usb/misc/iowarrior.ko 2024-08-03T20:36:57.836 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/usb/misc/isight_firmware.ko 2024-08-03T20:36:57.836 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/usb/misc/ldusb.ko 2024-08-03T20:36:57.836 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/usb/misc/legousbtower.ko 2024-08-03T20:36:57.836 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/usb/misc/sisusbvga 2024-08-03T20:36:57.836 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/usb/misc/sisusbvga/sisusbvga.ko 2024-08-03T20:36:57.836 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/usb/misc/trancevibrator.ko 2024-08-03T20:36:57.836 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/usb/misc/usb3503.ko 2024-08-03T20:36:57.837 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/usb/misc/usblcd.ko 2024-08-03T20:36:57.837 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/usb/misc/usbsevseg.ko 2024-08-03T20:36:57.837 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/usb/misc/uss720.ko 2024-08-03T20:36:57.837 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/usb/misc/yurex.ko 2024-08-03T20:36:57.837 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/usb/phy 2024-08-03T20:36:57.837 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/usb/phy/phy-generic.ko 2024-08-03T20:36:57.837 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/usb/serial 2024-08-03T20:36:57.837 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/usb/serial/aircable.ko 2024-08-03T20:36:57.837 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/usb/serial/ark3116.ko 2024-08-03T20:36:57.837 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/usb/serial/belkin_sa.ko 2024-08-03T20:36:57.837 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/usb/serial/ch341.ko 2024-08-03T20:36:57.837 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/usb/serial/cp210x.ko 2024-08-03T20:36:57.837 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/usb/serial/cyberjack.ko 2024-08-03T20:36:57.837 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/usb/serial/cypress_m8.ko 2024-08-03T20:36:57.838 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/usb/serial/digi_acceleport.ko 2024-08-03T20:36:57.838 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/usb/serial/empeg.ko 2024-08-03T20:36:57.838 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/usb/serial/ftdi_sio.ko 2024-08-03T20:36:57.838 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/usb/serial/garmin_gps.ko 2024-08-03T20:36:57.838 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/usb/serial/io_edgeport.ko 2024-08-03T20:36:57.838 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/usb/serial/io_ti.ko 2024-08-03T20:36:57.838 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/usb/serial/ipaq.ko 2024-08-03T20:36:57.838 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/usb/serial/ipw.ko 2024-08-03T20:36:57.838 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/usb/serial/ir-usb.ko 2024-08-03T20:36:57.838 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/usb/serial/iuu_phoenix.ko 2024-08-03T20:36:57.838 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/usb/serial/keyspan.ko 2024-08-03T20:36:57.838 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/usb/serial/keyspan_pda.ko 2024-08-03T20:36:57.838 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/usb/serial/kl5kusb105.ko 2024-08-03T20:36:57.838 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/usb/serial/kobil_sct.ko 2024-08-03T20:36:57.839 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/usb/serial/mct_u232.ko 2024-08-03T20:36:57.839 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/usb/serial/mos7720.ko 2024-08-03T20:36:57.839 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/usb/serial/mos7840.ko 2024-08-03T20:36:57.839 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/usb/serial/navman.ko 2024-08-03T20:36:57.839 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/usb/serial/omninet.ko 2024-08-03T20:36:57.839 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/usb/serial/opticon.ko 2024-08-03T20:36:57.839 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/usb/serial/option.ko 2024-08-03T20:36:57.839 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/usb/serial/oti6858.ko 2024-08-03T20:36:57.839 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/usb/serial/pl2303.ko 2024-08-03T20:36:57.839 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/usb/serial/qcaux.ko 2024-08-03T20:36:57.839 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/usb/serial/qcserial.ko 2024-08-03T20:36:57.839 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/usb/serial/quatech2.ko 2024-08-03T20:36:57.839 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/usb/serial/safe_serial.ko 2024-08-03T20:36:57.840 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/usb/serial/sierra.ko 2024-08-03T20:36:57.840 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/usb/serial/spcp8x5.ko 2024-08-03T20:36:57.840 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/usb/serial/ssu100.ko 2024-08-03T20:36:57.840 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/usb/serial/symbolserial.ko 2024-08-03T20:36:57.840 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/usb/serial/ti_usb_3410_5052.ko 2024-08-03T20:36:57.840 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/usb/serial/usb-serial-simple.ko 2024-08-03T20:36:57.840 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/usb/serial/usb_debug.ko 2024-08-03T20:36:57.840 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/usb/serial/usb_wwan.ko 2024-08-03T20:36:57.840 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/usb/serial/visor.ko 2024-08-03T20:36:57.840 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/usb/serial/whiteheat.ko 2024-08-03T20:36:57.840 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/usb/serial/xsens_mt.ko 2024-08-03T20:36:57.840 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/usb/storage 2024-08-03T20:36:57.840 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/usb/storage/uas.ko 2024-08-03T20:36:57.840 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/usb/storage/ums-alauda.ko 2024-08-03T20:36:57.841 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/usb/storage/ums-cypress.ko 2024-08-03T20:36:57.841 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/usb/storage/ums-datafab.ko 2024-08-03T20:36:57.841 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/usb/storage/ums-eneub6250.ko 2024-08-03T20:36:57.841 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/usb/storage/ums-freecom.ko 2024-08-03T20:36:57.841 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/usb/storage/ums-isd200.ko 2024-08-03T20:36:57.841 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/usb/storage/ums-jumpshot.ko 2024-08-03T20:36:57.841 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/usb/storage/ums-karma.ko 2024-08-03T20:36:57.841 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/usb/storage/ums-onetouch.ko 2024-08-03T20:36:57.841 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/usb/storage/ums-realtek.ko 2024-08-03T20:36:57.841 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/usb/storage/ums-sddr09.ko 2024-08-03T20:36:57.841 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/usb/storage/ums-sddr55.ko 2024-08-03T20:36:57.841 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/usb/storage/ums-usbat.ko 2024-08-03T20:36:57.841 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/usb/storage/usb-storage.ko 2024-08-03T20:36:57.841 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/usb/usbip 2024-08-03T20:36:57.842 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/usb/usbip/usbip-core.ko 2024-08-03T20:36:57.842 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/usb/usbip/usbip-host.ko 2024-08-03T20:36:57.842 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/usb/usbip/vhci-hcd.ko 2024-08-03T20:36:57.842 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/vfio 2024-08-03T20:36:57.842 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/vfio/pci 2024-08-03T20:36:57.842 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/vfio/pci/vfio-pci-core.ko 2024-08-03T20:36:57.842 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/vfio/pci/vfio-pci.ko 2024-08-03T20:36:57.842 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/vfio/vfio.ko 2024-08-03T20:36:57.842 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/vfio/vfio_iommu_type1.ko 2024-08-03T20:36:57.842 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/vhost 2024-08-03T20:36:57.842 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/vhost/vhost.ko 2024-08-03T20:36:57.842 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/vhost/vhost_iotlb.ko 2024-08-03T20:36:57.843 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/vhost/vhost_net.ko 2024-08-03T20:36:57.843 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/vhost/vhost_scsi.ko 2024-08-03T20:36:57.843 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/video 2024-08-03T20:36:57.843 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/video/backlight 2024-08-03T20:36:57.843 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/video/backlight/apple_bl.ko 2024-08-03T20:36:57.843 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/video/backlight/lcd.ko 2024-08-03T20:36:57.843 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/video/backlight/platform_lcd.ko 2024-08-03T20:36:57.843 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/virtio 2024-08-03T20:36:57.843 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/virtio/virtio.ko 2024-08-03T20:36:57.843 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/virtio/virtio_balloon.ko 2024-08-03T20:36:57.843 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/virtio/virtio_mmio.ko 2024-08-03T20:36:57.843 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/virtio/virtio_pci.ko 2024-08-03T20:36:57.843 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/virtio/virtio_pci_legacy_dev.ko 2024-08-03T20:36:57.844 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/virtio/virtio_pci_modern_dev.ko 2024-08-03T20:36:57.844 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/virtio/virtio_ring.ko 2024-08-03T20:36:57.844 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/w1 2024-08-03T20:36:57.844 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/w1/masters 2024-08-03T20:36:57.844 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/w1/masters/ds2482.ko 2024-08-03T20:36:57.844 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/w1/masters/ds2490.ko 2024-08-03T20:36:57.844 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/w1/slaves 2024-08-03T20:36:57.844 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/w1/slaves/w1_ds2406.ko 2024-08-03T20:36:57.844 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/w1/slaves/w1_ds2408.ko 2024-08-03T20:36:57.844 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/w1/slaves/w1_ds2413.ko 2024-08-03T20:36:57.844 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/w1/slaves/w1_ds2423.ko 2024-08-03T20:36:57.844 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/w1/slaves/w1_ds2431.ko 2024-08-03T20:36:57.844 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/w1/slaves/w1_ds2433.ko 2024-08-03T20:36:57.844 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/w1/slaves/w1_ds2780.ko 2024-08-03T20:36:57.845 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/w1/slaves/w1_ds2781.ko 2024-08-03T20:36:57.845 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/w1/slaves/w1_ds28e04.ko 2024-08-03T20:36:57.845 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/w1/slaves/w1_smem.ko 2024-08-03T20:36:57.845 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/w1/slaves/w1_therm.ko 2024-08-03T20:36:57.845 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/w1/wire.ko 2024-08-03T20:36:57.845 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/watchdog 2024-08-03T20:36:57.845 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/watchdog/alim1535_wdt.ko 2024-08-03T20:36:57.845 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/watchdog/alim7101_wdt.ko 2024-08-03T20:36:57.845 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/watchdog/f71808e_wdt.ko 2024-08-03T20:36:57.845 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/watchdog/hpwdt.ko 2024-08-03T20:36:57.845 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/watchdog/i6300esb.ko 2024-08-03T20:36:57.845 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/watchdog/iTCO_vendor_support.ko 2024-08-03T20:36:57.845 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/watchdog/iTCO_wdt.ko 2024-08-03T20:36:57.845 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/watchdog/ib700wdt.ko 2024-08-03T20:36:57.846 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/watchdog/ibmasr.ko 2024-08-03T20:36:57.846 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/watchdog/ie6xx_wdt.ko 2024-08-03T20:36:57.846 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/watchdog/it8712f_wdt.ko 2024-08-03T20:36:57.846 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/watchdog/it87_wdt.ko 2024-08-03T20:36:57.846 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/watchdog/machzwd.ko 2024-08-03T20:36:57.846 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/watchdog/nv_tco.ko 2024-08-03T20:36:57.846 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/watchdog/pcwd_pci.ko 2024-08-03T20:36:57.846 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/watchdog/pcwd_usb.ko 2024-08-03T20:36:57.846 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/watchdog/sbc_fitpc2_wdt.ko 2024-08-03T20:36:57.846 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/watchdog/sch311x_wdt.ko 2024-08-03T20:36:57.846 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/watchdog/softdog.ko 2024-08-03T20:36:57.846 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/watchdog/sp5100_tco.ko 2024-08-03T20:36:57.846 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/watchdog/via_wdt.ko 2024-08-03T20:36:57.847 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/watchdog/w83627hf_wdt.ko 2024-08-03T20:36:57.847 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/watchdog/w83877f_wdt.ko 2024-08-03T20:36:57.847 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/watchdog/w83977f_wdt.ko 2024-08-03T20:36:57.847 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/watchdog/wdt_pci.ko 2024-08-03T20:36:57.847 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/watchdog/xen_wdt.ko 2024-08-03T20:36:57.847 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/xen 2024-08-03T20:36:57.847 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/xen/xen-acpi-processor.ko 2024-08-03T20:36:57.847 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/xen/xen-evtchn.ko 2024-08-03T20:36:57.847 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/xen/xen-gntalloc.ko 2024-08-03T20:36:57.847 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/xen/xen-gntdev.ko 2024-08-03T20:36:57.847 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/xen/xen-pciback 2024-08-03T20:36:57.847 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/xen/xen-pciback/xen-pciback.ko 2024-08-03T20:36:57.847 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/xen/xen-privcmd.ko 2024-08-03T20:36:57.847 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/xen/xenfs 2024-08-03T20:36:57.848 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/drivers/xen/xenfs/xenfs.ko 2024-08-03T20:36:57.848 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/fs 2024-08-03T20:36:57.848 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/fs/9p 2024-08-03T20:36:57.848 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/fs/9p/9p.ko 2024-08-03T20:36:57.848 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/fs/affs 2024-08-03T20:36:57.848 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/fs/affs/affs.ko 2024-08-03T20:36:57.848 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/fs/befs 2024-08-03T20:36:57.848 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/fs/befs/befs.ko 2024-08-03T20:36:57.848 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/fs/binfmt_misc.ko 2024-08-03T20:36:57.848 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/fs/btrfs 2024-08-03T20:36:57.849 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/fs/btrfs/btrfs.ko 2024-08-03T20:36:57.849 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/fs/cachefiles 2024-08-03T20:36:57.849 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/fs/cachefiles/cachefiles.ko 2024-08-03T20:36:57.849 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/fs/ceph 2024-08-03T20:36:57.849 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/fs/ceph/ceph.ko 2024-08-03T20:36:57.849 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/fs/coda 2024-08-03T20:36:57.849 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/fs/coda/coda.ko 2024-08-03T20:36:57.849 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/fs/cramfs 2024-08-03T20:36:57.849 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/fs/cramfs/cramfs.ko 2024-08-03T20:36:57.849 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/fs/dlm 2024-08-03T20:36:57.849 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/fs/dlm/dlm.ko 2024-08-03T20:36:57.849 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/fs/ecryptfs 2024-08-03T20:36:57.849 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/fs/ecryptfs/ecryptfs.ko 2024-08-03T20:36:57.849 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/fs/f2fs 2024-08-03T20:36:57.850 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/fs/f2fs/f2fs.ko 2024-08-03T20:36:57.850 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/fs/fat 2024-08-03T20:36:57.850 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/fs/fat/fat.ko 2024-08-03T20:36:57.850 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/fs/fat/msdos.ko 2024-08-03T20:36:57.850 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/fs/fat/vfat.ko 2024-08-03T20:36:57.850 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/fs/fuse 2024-08-03T20:36:57.850 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/fs/fuse/cuse.ko 2024-08-03T20:36:57.850 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/fs/fuse/fuse.ko 2024-08-03T20:36:57.850 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/fs/gfs2 2024-08-03T20:36:57.850 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/fs/gfs2/gfs2.ko 2024-08-03T20:36:57.850 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/fs/hfs 2024-08-03T20:36:57.850 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/fs/hfs/hfs.ko 2024-08-03T20:36:57.850 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/fs/hfsplus 2024-08-03T20:36:57.850 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/fs/hfsplus/hfsplus.ko 2024-08-03T20:36:57.851 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/fs/isofs 2024-08-03T20:36:57.851 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/fs/isofs/isofs.ko 2024-08-03T20:36:57.851 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/fs/jfs 2024-08-03T20:36:57.851 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/fs/jfs/jfs.ko 2024-08-03T20:36:57.851 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/fs/lockd 2024-08-03T20:36:57.851 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/fs/lockd/lockd.ko 2024-08-03T20:36:57.851 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/fs/minix 2024-08-03T20:36:57.851 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/fs/minix/minix.ko 2024-08-03T20:36:57.851 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/fs/netfs 2024-08-03T20:36:57.851 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/fs/netfs/netfs.ko 2024-08-03T20:36:57.851 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/fs/nfs 2024-08-03T20:36:57.851 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/fs/nfs/blocklayout 2024-08-03T20:36:57.851 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/fs/nfs/blocklayout/blocklayoutdriver.ko 2024-08-03T20:36:57.851 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/fs/nfs/filelayout 2024-08-03T20:36:57.852 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/fs/nfs/filelayout/nfs_layout_nfsv41_files.ko 2024-08-03T20:36:57.852 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/fs/nfs/flexfilelayout 2024-08-03T20:36:57.852 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/fs/nfs/flexfilelayout/nfs_layout_flexfiles.ko 2024-08-03T20:36:57.852 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/fs/nfs/nfs.ko 2024-08-03T20:36:57.852 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/fs/nfs/nfsv3.ko 2024-08-03T20:36:57.852 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/fs/nfs/nfsv4.ko 2024-08-03T20:36:57.852 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/fs/nfs_common 2024-08-03T20:36:57.852 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/fs/nfs_common/grace.ko 2024-08-03T20:36:57.852 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/fs/nfs_common/nfs_acl.ko 2024-08-03T20:36:57.852 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/fs/nfsd 2024-08-03T20:36:57.852 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/fs/nfsd/nfsd.ko 2024-08-03T20:36:57.852 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/fs/nilfs2 2024-08-03T20:36:57.852 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/fs/nilfs2/nilfs2.ko 2024-08-03T20:36:57.852 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/fs/nls 2024-08-03T20:36:57.853 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/fs/nls/mac-celtic.ko 2024-08-03T20:36:57.853 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/fs/nls/mac-centeuro.ko 2024-08-03T20:36:57.853 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/fs/nls/mac-croatian.ko 2024-08-03T20:36:57.853 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/fs/nls/mac-cyrillic.ko 2024-08-03T20:36:57.853 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/fs/nls/mac-gaelic.ko 2024-08-03T20:36:57.853 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/fs/nls/mac-greek.ko 2024-08-03T20:36:57.853 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/fs/nls/mac-iceland.ko 2024-08-03T20:36:57.853 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/fs/nls/mac-inuit.ko 2024-08-03T20:36:57.853 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/fs/nls/mac-roman.ko 2024-08-03T20:36:57.853 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/fs/nls/mac-romanian.ko 2024-08-03T20:36:57.853 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/fs/nls/mac-turkish.ko 2024-08-03T20:36:57.853 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/fs/nls/nls_cp1250.ko 2024-08-03T20:36:57.853 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/fs/nls/nls_cp1251.ko 2024-08-03T20:36:57.854 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/fs/nls/nls_cp1255.ko 2024-08-03T20:36:57.854 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/fs/nls/nls_cp737.ko 2024-08-03T20:36:57.854 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/fs/nls/nls_cp775.ko 2024-08-03T20:36:57.854 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/fs/nls/nls_cp850.ko 2024-08-03T20:36:57.854 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/fs/nls/nls_cp852.ko 2024-08-03T20:36:57.854 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/fs/nls/nls_cp855.ko 2024-08-03T20:36:57.854 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/fs/nls/nls_cp857.ko 2024-08-03T20:36:57.854 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/fs/nls/nls_cp860.ko 2024-08-03T20:36:57.854 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/fs/nls/nls_cp861.ko 2024-08-03T20:36:57.854 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/fs/nls/nls_cp862.ko 2024-08-03T20:36:57.854 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/fs/nls/nls_cp863.ko 2024-08-03T20:36:57.855 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/fs/nls/nls_cp864.ko 2024-08-03T20:36:57.855 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/fs/nls/nls_cp865.ko 2024-08-03T20:36:57.855 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/fs/nls/nls_cp866.ko 2024-08-03T20:36:57.855 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/fs/nls/nls_cp869.ko 2024-08-03T20:36:57.855 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/fs/nls/nls_cp874.ko 2024-08-03T20:36:57.855 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/fs/nls/nls_cp932.ko 2024-08-03T20:36:57.855 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/fs/nls/nls_cp936.ko 2024-08-03T20:36:57.855 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/fs/nls/nls_cp949.ko 2024-08-03T20:36:57.855 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/fs/nls/nls_cp950.ko 2024-08-03T20:36:57.855 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/fs/nls/nls_euc-jp.ko 2024-08-03T20:36:57.855 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/fs/nls/nls_iso8859-1.ko 2024-08-03T20:36:57.855 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/fs/nls/nls_iso8859-13.ko 2024-08-03T20:36:57.855 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/fs/nls/nls_iso8859-14.ko 2024-08-03T20:36:57.855 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/fs/nls/nls_iso8859-15.ko 2024-08-03T20:36:57.855 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/fs/nls/nls_iso8859-2.ko 2024-08-03T20:36:57.856 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/fs/nls/nls_iso8859-3.ko 2024-08-03T20:36:57.856 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/fs/nls/nls_iso8859-4.ko 2024-08-03T20:36:57.856 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/fs/nls/nls_iso8859-5.ko 2024-08-03T20:36:57.856 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/fs/nls/nls_iso8859-6.ko 2024-08-03T20:36:57.856 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/fs/nls/nls_iso8859-7.ko 2024-08-03T20:36:57.856 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/fs/nls/nls_iso8859-9.ko 2024-08-03T20:36:57.856 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/fs/nls/nls_koi8-r.ko 2024-08-03T20:36:57.856 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/fs/nls/nls_koi8-ru.ko 2024-08-03T20:36:57.856 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/fs/nls/nls_koi8-u.ko 2024-08-03T20:36:57.856 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/fs/nls/nls_ucs2_utils.ko 2024-08-03T20:36:57.856 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/fs/nls/nls_utf8.ko 2024-08-03T20:36:57.856 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/fs/ocfs2 2024-08-03T20:36:57.856 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/fs/ocfs2/cluster 2024-08-03T20:36:57.856 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/fs/ocfs2/cluster/ocfs2_nodemanager.ko 2024-08-03T20:36:57.857 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/fs/ocfs2/dlm 2024-08-03T20:36:57.857 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/fs/ocfs2/dlm/ocfs2_dlm.ko 2024-08-03T20:36:57.857 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/fs/ocfs2/dlmfs 2024-08-03T20:36:57.857 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/fs/ocfs2/dlmfs/ocfs2_dlmfs.ko 2024-08-03T20:36:57.857 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/fs/ocfs2/ocfs2.ko 2024-08-03T20:36:57.857 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/fs/ocfs2/ocfs2_stack_o2cb.ko 2024-08-03T20:36:57.857 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/fs/ocfs2/ocfs2_stack_user.ko 2024-08-03T20:36:57.857 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/fs/ocfs2/ocfs2_stackglue.ko 2024-08-03T20:36:57.857 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/fs/overlayfs 2024-08-03T20:36:57.857 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/fs/overlayfs/overlay.ko 2024-08-03T20:36:57.857 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/fs/pstore 2024-08-03T20:36:57.857 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/fs/pstore/ramoops.ko 2024-08-03T20:36:57.857 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/fs/reiserfs 2024-08-03T20:36:57.858 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/fs/reiserfs/reiserfs.ko 2024-08-03T20:36:57.858 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/fs/romfs 2024-08-03T20:36:57.858 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/fs/romfs/romfs.ko 2024-08-03T20:36:57.858 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/fs/smb 2024-08-03T20:36:57.858 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/fs/smb/client 2024-08-03T20:36:57.858 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/fs/smb/client/cifs.ko 2024-08-03T20:36:57.858 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/fs/smb/common 2024-08-03T20:36:57.858 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/fs/smb/common/cifs_arc4.ko 2024-08-03T20:36:57.858 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/fs/smb/common/cifs_md4.ko 2024-08-03T20:36:57.858 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/fs/squashfs 2024-08-03T20:36:57.858 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/fs/squashfs/squashfs.ko 2024-08-03T20:36:57.858 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/fs/sysv 2024-08-03T20:36:57.858 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/fs/sysv/sysv.ko 2024-08-03T20:36:57.858 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/fs/udf 2024-08-03T20:36:57.859 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/fs/udf/udf.ko 2024-08-03T20:36:57.859 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/fs/ufs 2024-08-03T20:36:57.859 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/fs/ufs/ufs.ko 2024-08-03T20:36:57.859 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/fs/xfs 2024-08-03T20:36:57.859 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/fs/xfs/xfs.ko 2024-08-03T20:36:57.859 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/kernel 2024-08-03T20:36:57.859 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/kernel/rcu 2024-08-03T20:36:57.859 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/kernel/rcu/rcutorture.ko 2024-08-03T20:36:57.859 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/kernel/torture.ko 2024-08-03T20:36:57.859 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/kernel/trace 2024-08-03T20:36:57.859 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/kernel/trace/ring_buffer_benchmark.ko 2024-08-03T20:36:57.859 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/lib 2024-08-03T20:36:57.859 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/lib/asn1_encoder.ko 2024-08-03T20:36:57.859 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/lib/crc-itu-t.ko 2024-08-03T20:36:57.860 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/lib/crc8.ko 2024-08-03T20:36:57.860 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/lib/crypto 2024-08-03T20:36:57.860 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/lib/crypto/libaescfb.ko 2024-08-03T20:36:57.860 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/lib/crypto/libarc4.ko 2024-08-03T20:36:57.860 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/lib/crypto/libdes.ko 2024-08-03T20:36:57.860 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/lib/libcrc32c.ko 2024-08-03T20:36:57.860 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/lib/lru_cache.ko 2024-08-03T20:36:57.860 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/lib/lz4 2024-08-03T20:36:57.860 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/lib/lz4/lz4_compress.ko 2024-08-03T20:36:57.860 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/lib/lz4/lz4hc_compress.ko 2024-08-03T20:36:57.860 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/lib/math 2024-08-03T20:36:57.860 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/lib/math/cordic.ko 2024-08-03T20:36:57.860 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/lib/raid6 2024-08-03T20:36:57.860 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/lib/raid6/raid6_pq.ko 2024-08-03T20:36:57.861 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/lib/reed_solomon 2024-08-03T20:36:57.861 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/lib/reed_solomon/reed_solomon.ko 2024-08-03T20:36:57.861 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/lib/ts_bm.ko 2024-08-03T20:36:57.861 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/lib/ts_fsm.ko 2024-08-03T20:36:57.861 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/lib/ts_kmp.ko 2024-08-03T20:36:57.861 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/lib/zstd 2024-08-03T20:36:57.861 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/lib/zstd/zstd_compress.ko 2024-08-03T20:36:57.861 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/mm 2024-08-03T20:36:57.861 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/mm/hwpoison-inject.ko 2024-08-03T20:36:57.861 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net 2024-08-03T20:36:57.861 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/6lowpan 2024-08-03T20:36:57.861 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/6lowpan/6lowpan.ko 2024-08-03T20:36:57.861 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/6lowpan/nhc_dest.ko 2024-08-03T20:36:57.861 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/6lowpan/nhc_fragment.ko 2024-08-03T20:36:57.862 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/6lowpan/nhc_hop.ko 2024-08-03T20:36:57.862 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/6lowpan/nhc_ipv6.ko 2024-08-03T20:36:57.862 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/6lowpan/nhc_mobility.ko 2024-08-03T20:36:57.862 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/6lowpan/nhc_routing.ko 2024-08-03T20:36:57.862 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/6lowpan/nhc_udp.ko 2024-08-03T20:36:57.862 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/802 2024-08-03T20:36:57.862 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/802/garp.ko 2024-08-03T20:36:57.862 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/802/mrp.ko 2024-08-03T20:36:57.862 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/802/p8022.ko 2024-08-03T20:36:57.862 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/802/psnap.ko 2024-08-03T20:36:57.862 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/802/stp.ko 2024-08-03T20:36:57.862 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/8021q 2024-08-03T20:36:57.862 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/8021q/8021q.ko 2024-08-03T20:36:57.862 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/9p 2024-08-03T20:36:57.863 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/9p/9pnet.ko 2024-08-03T20:36:57.863 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/9p/9pnet_fd.ko 2024-08-03T20:36:57.863 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/9p/9pnet_rdma.ko 2024-08-03T20:36:57.863 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/9p/9pnet_virtio.ko 2024-08-03T20:36:57.863 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/batman-adv 2024-08-03T20:36:57.863 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/batman-adv/batman-adv.ko 2024-08-03T20:36:57.864 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/bridge 2024-08-03T20:36:57.864 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/bridge/br_netfilter.ko 2024-08-03T20:36:57.864 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/bridge/bridge.ko 2024-08-03T20:36:57.864 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/bridge/netfilter 2024-08-03T20:36:57.864 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/bridge/netfilter/ebt_802_3.ko 2024-08-03T20:36:57.864 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/bridge/netfilter/ebt_among.ko 2024-08-03T20:36:57.864 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/bridge/netfilter/ebt_arp.ko 2024-08-03T20:36:57.864 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/bridge/netfilter/ebt_arpreply.ko 2024-08-03T20:36:57.864 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/bridge/netfilter/ebt_dnat.ko 2024-08-03T20:36:57.864 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/bridge/netfilter/ebt_ip.ko 2024-08-03T20:36:57.864 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/bridge/netfilter/ebt_ip6.ko 2024-08-03T20:36:57.864 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/bridge/netfilter/ebt_limit.ko 2024-08-03T20:36:57.864 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/bridge/netfilter/ebt_log.ko 2024-08-03T20:36:57.864 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/bridge/netfilter/ebt_mark.ko 2024-08-03T20:36:57.865 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/bridge/netfilter/ebt_mark_m.ko 2024-08-03T20:36:57.865 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/bridge/netfilter/ebt_nflog.ko 2024-08-03T20:36:57.865 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/bridge/netfilter/ebt_pkttype.ko 2024-08-03T20:36:57.865 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/bridge/netfilter/ebt_redirect.ko 2024-08-03T20:36:57.865 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/bridge/netfilter/ebt_snat.ko 2024-08-03T20:36:57.865 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/bridge/netfilter/ebt_stp.ko 2024-08-03T20:36:57.865 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/bridge/netfilter/ebt_vlan.ko 2024-08-03T20:36:57.865 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/bridge/netfilter/ebtable_broute.ko 2024-08-03T20:36:57.865 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/bridge/netfilter/ebtable_filter.ko 2024-08-03T20:36:57.865 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/bridge/netfilter/ebtable_nat.ko 2024-08-03T20:36:57.865 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/bridge/netfilter/ebtables.ko 2024-08-03T20:36:57.865 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/bridge/netfilter/nf_conntrack_bridge.ko 2024-08-03T20:36:57.865 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/bridge/netfilter/nft_meta_bridge.ko 2024-08-03T20:36:57.865 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/bridge/netfilter/nft_reject_bridge.ko 2024-08-03T20:36:57.866 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/ceph 2024-08-03T20:36:57.866 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/ceph/libceph.ko 2024-08-03T20:36:57.866 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/core 2024-08-03T20:36:57.866 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/core/failover.ko 2024-08-03T20:36:57.866 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/core/pktgen.ko 2024-08-03T20:36:57.866 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/dccp 2024-08-03T20:36:57.866 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/dccp/dccp.ko 2024-08-03T20:36:57.866 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/dccp/dccp_diag.ko 2024-08-03T20:36:57.866 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/dccp/dccp_ipv4.ko 2024-08-03T20:36:57.866 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/dccp/dccp_ipv6.ko 2024-08-03T20:36:57.867 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/dns_resolver 2024-08-03T20:36:57.867 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/dns_resolver/dns_resolver.ko 2024-08-03T20:36:57.867 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/dsa 2024-08-03T20:36:57.867 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/dsa/dsa_core.ko 2024-08-03T20:36:57.867 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/dsa/tag_brcm.ko 2024-08-03T20:36:57.867 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/dsa/tag_dsa.ko 2024-08-03T20:36:57.867 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/dsa/tag_none.ko 2024-08-03T20:36:57.867 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/dsa/tag_trailer.ko 2024-08-03T20:36:57.867 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/ieee802154 2024-08-03T20:36:57.867 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/ieee802154/6lowpan 2024-08-03T20:36:57.867 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/ieee802154/6lowpan/ieee802154_6lowpan.ko 2024-08-03T20:36:57.867 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/ieee802154/ieee802154.ko 2024-08-03T20:36:57.867 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/ieee802154/ieee802154_socket.ko 2024-08-03T20:36:57.867 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/ipv4 2024-08-03T20:36:57.868 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/ipv4/ah4.ko 2024-08-03T20:36:57.868 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/ipv4/esp4.ko 2024-08-03T20:36:57.868 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/ipv4/fou.ko 2024-08-03T20:36:57.868 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/ipv4/gre.ko 2024-08-03T20:36:57.868 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/ipv4/inet_diag.ko 2024-08-03T20:36:57.868 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/ipv4/ip_gre.ko 2024-08-03T20:36:57.868 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/ipv4/ip_tunnel.ko 2024-08-03T20:36:57.868 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/ipv4/ip_vti.ko 2024-08-03T20:36:57.868 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/ipv4/ipcomp.ko 2024-08-03T20:36:57.868 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/ipv4/ipip.ko 2024-08-03T20:36:57.868 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/ipv4/netfilter 2024-08-03T20:36:57.868 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/ipv4/netfilter/arp_tables.ko 2024-08-03T20:36:57.868 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/ipv4/netfilter/arpt_mangle.ko 2024-08-03T20:36:57.868 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/ipv4/netfilter/arptable_filter.ko 2024-08-03T20:36:57.869 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/ipv4/netfilter/ipt_ECN.ko 2024-08-03T20:36:57.869 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/ipv4/netfilter/ipt_SYNPROXY.ko 2024-08-03T20:36:57.869 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/ipv4/netfilter/ipt_ah.ko 2024-08-03T20:36:57.869 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/ipv4/netfilter/ipt_rpfilter.ko 2024-08-03T20:36:57.869 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/ipv4/netfilter/iptable_mangle.ko 2024-08-03T20:36:57.869 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/ipv4/netfilter/iptable_nat.ko 2024-08-03T20:36:57.869 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/ipv4/netfilter/iptable_raw.ko 2024-08-03T20:36:57.869 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/ipv4/netfilter/iptable_security.ko 2024-08-03T20:36:57.869 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/ipv4/netfilter/nf_defrag_ipv4.ko 2024-08-03T20:36:57.869 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/ipv4/netfilter/nf_dup_ipv4.ko 2024-08-03T20:36:57.870 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/ipv4/netfilter/nf_nat_h323.ko 2024-08-03T20:36:57.870 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/ipv4/netfilter/nf_nat_pptp.ko 2024-08-03T20:36:57.870 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/ipv4/netfilter/nf_nat_snmp_basic.ko 2024-08-03T20:36:57.870 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/ipv4/netfilter/nf_socket_ipv4.ko 2024-08-03T20:36:57.870 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/ipv4/netfilter/nf_tproxy_ipv4.ko 2024-08-03T20:36:57.870 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/ipv4/netfilter/nft_dup_ipv4.ko 2024-08-03T20:36:57.870 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/ipv4/netfilter/nft_fib_ipv4.ko 2024-08-03T20:36:57.870 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/ipv4/netfilter/nft_reject_ipv4.ko 2024-08-03T20:36:57.870 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/ipv4/tcp_bic.ko 2024-08-03T20:36:57.870 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/ipv4/tcp_dctcp.ko 2024-08-03T20:36:57.870 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/ipv4/tcp_diag.ko 2024-08-03T20:36:57.870 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/ipv4/tcp_highspeed.ko 2024-08-03T20:36:57.870 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/ipv4/tcp_htcp.ko 2024-08-03T20:36:57.871 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/ipv4/tcp_hybla.ko 2024-08-03T20:36:57.871 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/ipv4/tcp_illinois.ko 2024-08-03T20:36:57.871 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/ipv4/tcp_lp.ko 2024-08-03T20:36:57.871 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/ipv4/tcp_scalable.ko 2024-08-03T20:36:57.871 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/ipv4/tcp_vegas.ko 2024-08-03T20:36:57.871 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/ipv4/tcp_veno.ko 2024-08-03T20:36:57.871 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/ipv4/tcp_westwood.ko 2024-08-03T20:36:57.871 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/ipv4/tcp_yeah.ko 2024-08-03T20:36:57.871 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/ipv4/tunnel4.ko 2024-08-03T20:36:57.871 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/ipv4/udp_diag.ko 2024-08-03T20:36:57.871 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/ipv4/udp_tunnel.ko 2024-08-03T20:36:57.871 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/ipv4/xfrm4_tunnel.ko 2024-08-03T20:36:57.871 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/ipv6 2024-08-03T20:36:57.872 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/ipv6/ah6.ko 2024-08-03T20:36:57.872 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/ipv6/esp6.ko 2024-08-03T20:36:57.872 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/ipv6/fou6.ko 2024-08-03T20:36:57.872 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/ipv6/ip6_tunnel.ko 2024-08-03T20:36:57.872 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/ipv6/ip6_udp_tunnel.ko 2024-08-03T20:36:57.872 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/ipv6/ip6_vti.ko 2024-08-03T20:36:57.872 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/ipv6/ipcomp6.ko 2024-08-03T20:36:57.872 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/ipv6/netfilter 2024-08-03T20:36:57.872 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/ipv6/netfilter/ip6_tables.ko 2024-08-03T20:36:57.872 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/ipv6/netfilter/ip6t_REJECT.ko 2024-08-03T20:36:57.872 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/ipv6/netfilter/ip6t_SYNPROXY.ko 2024-08-03T20:36:57.872 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/ipv6/netfilter/ip6t_ah.ko 2024-08-03T20:36:57.872 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/ipv6/netfilter/ip6t_eui64.ko 2024-08-03T20:36:57.872 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/ipv6/netfilter/ip6t_frag.ko 2024-08-03T20:36:57.873 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/ipv6/netfilter/ip6t_hbh.ko 2024-08-03T20:36:57.873 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/ipv6/netfilter/ip6t_ipv6header.ko 2024-08-03T20:36:57.873 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/ipv6/netfilter/ip6t_mh.ko 2024-08-03T20:36:57.873 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/ipv6/netfilter/ip6t_rpfilter.ko 2024-08-03T20:36:57.873 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/ipv6/netfilter/ip6t_rt.ko 2024-08-03T20:36:57.873 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/ipv6/netfilter/ip6table_filter.ko 2024-08-03T20:36:57.873 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/ipv6/netfilter/ip6table_mangle.ko 2024-08-03T20:36:57.873 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/ipv6/netfilter/ip6table_nat.ko 2024-08-03T20:36:57.873 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/ipv6/netfilter/ip6table_raw.ko 2024-08-03T20:36:57.873 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/ipv6/netfilter/ip6table_security.ko 2024-08-03T20:36:57.873 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/ipv6/netfilter/nf_defrag_ipv6.ko 2024-08-03T20:36:57.874 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/ipv6/netfilter/nf_dup_ipv6.ko 2024-08-03T20:36:57.874 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/ipv6/netfilter/nf_reject_ipv6.ko 2024-08-03T20:36:57.874 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/ipv6/netfilter/nf_socket_ipv6.ko 2024-08-03T20:36:57.874 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/ipv6/netfilter/nf_tproxy_ipv6.ko 2024-08-03T20:36:57.874 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/ipv6/netfilter/nft_dup_ipv6.ko 2024-08-03T20:36:57.874 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/ipv6/netfilter/nft_fib_ipv6.ko 2024-08-03T20:36:57.874 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/ipv6/netfilter/nft_reject_ipv6.ko 2024-08-03T20:36:57.874 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/ipv6/sit.ko 2024-08-03T20:36:57.874 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/ipv6/tunnel6.ko 2024-08-03T20:36:57.874 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/ipv6/xfrm6_tunnel.ko 2024-08-03T20:36:57.874 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/key 2024-08-03T20:36:57.874 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/key/af_key.ko 2024-08-03T20:36:57.874 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/l2tp 2024-08-03T20:36:57.875 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/l2tp/l2tp_core.ko 2024-08-03T20:36:57.875 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/l2tp/l2tp_debugfs.ko 2024-08-03T20:36:57.875 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/l2tp/l2tp_eth.ko 2024-08-03T20:36:57.875 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/l2tp/l2tp_ip.ko 2024-08-03T20:36:57.875 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/l2tp/l2tp_ip6.ko 2024-08-03T20:36:57.875 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/l2tp/l2tp_netlink.ko 2024-08-03T20:36:57.875 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/l2tp/l2tp_ppp.ko 2024-08-03T20:36:57.875 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/llc 2024-08-03T20:36:57.875 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/llc/llc.ko 2024-08-03T20:36:57.875 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/mac802154 2024-08-03T20:36:57.875 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/mac802154/mac802154.ko 2024-08-03T20:36:57.875 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/mpls 2024-08-03T20:36:57.875 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/mpls/mpls_gso.ko 2024-08-03T20:36:57.875 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/netfilter 2024-08-03T20:36:57.876 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/netfilter/ipset 2024-08-03T20:36:57.876 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/netfilter/ipset/ip_set.ko 2024-08-03T20:36:57.876 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/netfilter/ipset/ip_set_bitmap_ip.ko 2024-08-03T20:36:57.876 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/netfilter/ipset/ip_set_bitmap_ipmac.ko 2024-08-03T20:36:57.876 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/netfilter/ipset/ip_set_bitmap_port.ko 2024-08-03T20:36:57.876 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/netfilter/ipset/ip_set_hash_ip.ko 2024-08-03T20:36:57.876 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/netfilter/ipset/ip_set_hash_ipmark.ko 2024-08-03T20:36:57.876 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/netfilter/ipset/ip_set_hash_ipport.ko 2024-08-03T20:36:57.876 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/netfilter/ipset/ip_set_hash_ipportip.ko 2024-08-03T20:36:57.876 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/netfilter/ipset/ip_set_hash_ipportnet.ko 2024-08-03T20:36:57.876 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/netfilter/ipset/ip_set_hash_mac.ko 2024-08-03T20:36:57.876 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/netfilter/ipset/ip_set_hash_net.ko 2024-08-03T20:36:57.876 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/netfilter/ipset/ip_set_hash_netiface.ko 2024-08-03T20:36:57.877 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/netfilter/ipset/ip_set_hash_netnet.ko 2024-08-03T20:36:57.877 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/netfilter/ipset/ip_set_hash_netport.ko 2024-08-03T20:36:57.877 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/netfilter/ipset/ip_set_hash_netportnet.ko 2024-08-03T20:36:57.877 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/netfilter/ipset/ip_set_list_set.ko 2024-08-03T20:36:57.877 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/netfilter/ipvs 2024-08-03T20:36:57.877 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/netfilter/ipvs/ip_vs.ko 2024-08-03T20:36:57.877 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/netfilter/ipvs/ip_vs_dh.ko 2024-08-03T20:36:57.877 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/netfilter/ipvs/ip_vs_fo.ko 2024-08-03T20:36:57.877 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/netfilter/ipvs/ip_vs_ftp.ko 2024-08-03T20:36:57.877 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/netfilter/ipvs/ip_vs_lblc.ko 2024-08-03T20:36:57.877 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/netfilter/ipvs/ip_vs_lblcr.ko 2024-08-03T20:36:57.877 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/netfilter/ipvs/ip_vs_lc.ko 2024-08-03T20:36:57.877 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/netfilter/ipvs/ip_vs_nq.ko 2024-08-03T20:36:57.877 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/netfilter/ipvs/ip_vs_pe_sip.ko 2024-08-03T20:36:57.877 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/netfilter/ipvs/ip_vs_rr.ko 2024-08-03T20:36:57.878 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/netfilter/ipvs/ip_vs_sed.ko 2024-08-03T20:36:57.878 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/netfilter/ipvs/ip_vs_sh.ko 2024-08-03T20:36:57.878 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/netfilter/ipvs/ip_vs_wlc.ko 2024-08-03T20:36:57.878 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/netfilter/ipvs/ip_vs_wrr.ko 2024-08-03T20:36:57.878 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/netfilter/nf_conncount.ko 2024-08-03T20:36:57.878 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/netfilter/nf_conntrack.ko 2024-08-03T20:36:57.878 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/netfilter/nf_conntrack_amanda.ko 2024-08-03T20:36:57.878 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/netfilter/nf_conntrack_broadcast.ko 2024-08-03T20:36:57.878 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/netfilter/nf_conntrack_ftp.ko 2024-08-03T20:36:57.878 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/netfilter/nf_conntrack_h323.ko 2024-08-03T20:36:57.878 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/netfilter/nf_conntrack_irc.ko 2024-08-03T20:36:57.878 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/netfilter/nf_conntrack_netbios_ns.ko 2024-08-03T20:36:57.878 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/netfilter/nf_conntrack_netlink.ko 2024-08-03T20:36:57.879 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/netfilter/nf_conntrack_pptp.ko 2024-08-03T20:36:57.879 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/netfilter/nf_conntrack_sane.ko 2024-08-03T20:36:57.879 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/netfilter/nf_conntrack_sip.ko 2024-08-03T20:36:57.879 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/netfilter/nf_conntrack_snmp.ko 2024-08-03T20:36:57.879 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/netfilter/nf_conntrack_tftp.ko 2024-08-03T20:36:57.879 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/netfilter/nf_dup_netdev.ko 2024-08-03T20:36:57.879 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/netfilter/nf_flow_table.ko 2024-08-03T20:36:57.879 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/netfilter/nf_flow_table_inet.ko 2024-08-03T20:36:57.879 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/netfilter/nf_log_syslog.ko 2024-08-03T20:36:57.879 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/netfilter/nf_nat.ko 2024-08-03T20:36:57.879 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/netfilter/nf_nat_amanda.ko 2024-08-03T20:36:57.879 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/netfilter/nf_nat_ftp.ko 2024-08-03T20:36:57.879 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/netfilter/nf_nat_irc.ko 2024-08-03T20:36:57.879 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/netfilter/nf_nat_sip.ko 2024-08-03T20:36:57.880 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/netfilter/nf_nat_tftp.ko 2024-08-03T20:36:57.880 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/netfilter/nf_synproxy_core.ko 2024-08-03T20:36:57.880 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/netfilter/nf_tables.ko 2024-08-03T20:36:57.880 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/netfilter/nfnetlink.ko 2024-08-03T20:36:57.880 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/netfilter/nfnetlink_acct.ko 2024-08-03T20:36:57.880 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/netfilter/nfnetlink_log.ko 2024-08-03T20:36:57.880 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/netfilter/nfnetlink_osf.ko 2024-08-03T20:36:57.880 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/netfilter/nfnetlink_queue.ko 2024-08-03T20:36:57.880 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/netfilter/nft_chain_nat.ko 2024-08-03T20:36:57.880 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/netfilter/nft_compat.ko 2024-08-03T20:36:57.880 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/netfilter/nft_ct.ko 2024-08-03T20:36:57.880 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/netfilter/nft_dup_netdev.ko 2024-08-03T20:36:57.880 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/netfilter/nft_fib.ko 2024-08-03T20:36:57.881 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/netfilter/nft_fwd_netdev.ko 2024-08-03T20:36:57.881 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/netfilter/nft_hash.ko 2024-08-03T20:36:57.881 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/netfilter/nft_limit.ko 2024-08-03T20:36:57.881 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/netfilter/nft_log.ko 2024-08-03T20:36:57.881 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/netfilter/nft_masq.ko 2024-08-03T20:36:57.881 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/netfilter/nft_nat.ko 2024-08-03T20:36:57.881 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/netfilter/nft_numgen.ko 2024-08-03T20:36:57.881 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/netfilter/nft_queue.ko 2024-08-03T20:36:57.881 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/netfilter/nft_quota.ko 2024-08-03T20:36:57.881 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/netfilter/nft_redir.ko 2024-08-03T20:36:57.881 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/netfilter/nft_reject.ko 2024-08-03T20:36:57.881 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/netfilter/nft_reject_inet.ko 2024-08-03T20:36:57.881 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/netfilter/nft_socket.ko 2024-08-03T20:36:57.881 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/netfilter/nft_tunnel.ko 2024-08-03T20:36:57.882 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/netfilter/xt_AUDIT.ko 2024-08-03T20:36:57.882 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/netfilter/xt_CHECKSUM.ko 2024-08-03T20:36:57.882 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/netfilter/xt_CLASSIFY.ko 2024-08-03T20:36:57.882 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/netfilter/xt_CONNSECMARK.ko 2024-08-03T20:36:57.882 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/netfilter/xt_CT.ko 2024-08-03T20:36:57.882 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/netfilter/xt_DSCP.ko 2024-08-03T20:36:57.882 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/netfilter/xt_HL.ko 2024-08-03T20:36:57.882 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/netfilter/xt_HMARK.ko 2024-08-03T20:36:57.882 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/netfilter/xt_IDLETIMER.ko 2024-08-03T20:36:57.882 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/netfilter/xt_LED.ko 2024-08-03T20:36:57.882 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/netfilter/xt_LOG.ko 2024-08-03T20:36:57.882 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/netfilter/xt_MASQUERADE.ko 2024-08-03T20:36:57.882 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/netfilter/xt_NETMAP.ko 2024-08-03T20:36:57.882 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/netfilter/xt_NFLOG.ko 2024-08-03T20:36:57.883 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/netfilter/xt_NFQUEUE.ko 2024-08-03T20:36:57.883 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/netfilter/xt_RATEEST.ko 2024-08-03T20:36:57.883 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/netfilter/xt_REDIRECT.ko 2024-08-03T20:36:57.883 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/netfilter/xt_SECMARK.ko 2024-08-03T20:36:57.883 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/netfilter/xt_TCPMSS.ko 2024-08-03T20:36:57.883 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/netfilter/xt_TCPOPTSTRIP.ko 2024-08-03T20:36:57.883 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/netfilter/xt_TEE.ko 2024-08-03T20:36:57.883 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/netfilter/xt_TPROXY.ko 2024-08-03T20:36:57.883 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/netfilter/xt_TRACE.ko 2024-08-03T20:36:57.883 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/netfilter/xt_addrtype.ko 2024-08-03T20:36:57.883 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/netfilter/xt_bpf.ko 2024-08-03T20:36:57.884 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/netfilter/xt_cgroup.ko 2024-08-03T20:36:57.884 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/netfilter/xt_cluster.ko 2024-08-03T20:36:57.884 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/netfilter/xt_comment.ko 2024-08-03T20:36:57.884 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/netfilter/xt_connbytes.ko 2024-08-03T20:36:57.884 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/netfilter/xt_connlabel.ko 2024-08-03T20:36:57.884 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/netfilter/xt_connlimit.ko 2024-08-03T20:36:57.884 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/netfilter/xt_connmark.ko 2024-08-03T20:36:57.884 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/netfilter/xt_conntrack.ko 2024-08-03T20:36:57.884 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/netfilter/xt_cpu.ko 2024-08-03T20:36:57.884 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/netfilter/xt_dccp.ko 2024-08-03T20:36:57.884 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/netfilter/xt_devgroup.ko 2024-08-03T20:36:57.884 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/netfilter/xt_dscp.ko 2024-08-03T20:36:57.884 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/netfilter/xt_ecn.ko 2024-08-03T20:36:57.884 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/netfilter/xt_esp.ko 2024-08-03T20:36:57.885 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/netfilter/xt_hashlimit.ko 2024-08-03T20:36:57.885 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/netfilter/xt_helper.ko 2024-08-03T20:36:57.885 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/netfilter/xt_hl.ko 2024-08-03T20:36:57.885 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/netfilter/xt_ipcomp.ko 2024-08-03T20:36:57.885 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/netfilter/xt_iprange.ko 2024-08-03T20:36:57.885 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/netfilter/xt_ipvs.ko 2024-08-03T20:36:57.885 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/netfilter/xt_l2tp.ko 2024-08-03T20:36:57.885 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/netfilter/xt_length.ko 2024-08-03T20:36:57.885 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/netfilter/xt_limit.ko 2024-08-03T20:36:57.885 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/netfilter/xt_mac.ko 2024-08-03T20:36:57.885 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/netfilter/xt_mark.ko 2024-08-03T20:36:57.885 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/netfilter/xt_multiport.ko 2024-08-03T20:36:57.885 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/netfilter/xt_nat.ko 2024-08-03T20:36:57.885 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/netfilter/xt_nfacct.ko 2024-08-03T20:36:57.886 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/netfilter/xt_osf.ko 2024-08-03T20:36:57.886 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/netfilter/xt_owner.ko 2024-08-03T20:36:57.886 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/netfilter/xt_physdev.ko 2024-08-03T20:36:57.886 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/netfilter/xt_pkttype.ko 2024-08-03T20:36:57.886 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/netfilter/xt_policy.ko 2024-08-03T20:36:57.886 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/netfilter/xt_quota.ko 2024-08-03T20:36:57.886 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/netfilter/xt_rateest.ko 2024-08-03T20:36:57.886 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/netfilter/xt_realm.ko 2024-08-03T20:36:57.886 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/netfilter/xt_recent.ko 2024-08-03T20:36:57.886 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/netfilter/xt_sctp.ko 2024-08-03T20:36:57.886 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/netfilter/xt_set.ko 2024-08-03T20:36:57.886 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/netfilter/xt_socket.ko 2024-08-03T20:36:57.886 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/netfilter/xt_state.ko 2024-08-03T20:36:57.887 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/netfilter/xt_statistic.ko 2024-08-03T20:36:57.887 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/netfilter/xt_string.ko 2024-08-03T20:36:57.887 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/netfilter/xt_tcpmss.ko 2024-08-03T20:36:57.887 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/netfilter/xt_time.ko 2024-08-03T20:36:57.887 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/netfilter/xt_u32.ko 2024-08-03T20:36:57.887 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/netlink 2024-08-03T20:36:57.887 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/netlink/netlink_diag.ko 2024-08-03T20:36:57.887 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/nsh 2024-08-03T20:36:57.887 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/nsh/nsh.ko 2024-08-03T20:36:57.887 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/openvswitch 2024-08-03T20:36:57.887 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/openvswitch/openvswitch.ko 2024-08-03T20:36:57.887 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/openvswitch/vport-geneve.ko 2024-08-03T20:36:57.887 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/openvswitch/vport-gre.ko 2024-08-03T20:36:57.888 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/openvswitch/vport-vxlan.ko 2024-08-03T20:36:57.888 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/packet 2024-08-03T20:36:57.888 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/packet/af_packet_diag.ko 2024-08-03T20:36:57.888 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/rds 2024-08-03T20:36:57.888 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/rds/rds.ko 2024-08-03T20:36:57.888 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/rds/rds_rdma.ko 2024-08-03T20:36:57.888 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/rds/rds_tcp.ko 2024-08-03T20:36:57.888 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/sched 2024-08-03T20:36:57.888 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/sched/act_csum.ko 2024-08-03T20:36:57.888 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/sched/act_gact.ko 2024-08-03T20:36:57.888 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/sched/act_mirred.ko 2024-08-03T20:36:57.888 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/sched/act_nat.ko 2024-08-03T20:36:57.888 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/sched/act_pedit.ko 2024-08-03T20:36:57.888 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/sched/act_police.ko 2024-08-03T20:36:57.889 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/sched/act_simple.ko 2024-08-03T20:36:57.889 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/sched/act_skbedit.ko 2024-08-03T20:36:57.889 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/sched/act_vlan.ko 2024-08-03T20:36:57.889 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/sched/cls_basic.ko 2024-08-03T20:36:57.889 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/sched/cls_bpf.ko 2024-08-03T20:36:57.889 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/sched/cls_flow.ko 2024-08-03T20:36:57.889 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/sched/cls_fw.ko 2024-08-03T20:36:57.889 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/sched/cls_route.ko 2024-08-03T20:36:57.889 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/sched/cls_u32.ko 2024-08-03T20:36:57.889 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/sched/em_cmp.ko 2024-08-03T20:36:57.889 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/sched/em_ipset.ko 2024-08-03T20:36:57.889 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/sched/em_meta.ko 2024-08-03T20:36:57.889 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/sched/em_nbyte.ko 2024-08-03T20:36:57.889 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/sched/em_text.ko 2024-08-03T20:36:57.890 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/sched/em_u32.ko 2024-08-03T20:36:57.890 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/sched/sch_choke.ko 2024-08-03T20:36:57.890 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/sched/sch_codel.ko 2024-08-03T20:36:57.890 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/sched/sch_drr.ko 2024-08-03T20:36:57.890 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/sched/sch_fq.ko 2024-08-03T20:36:57.890 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/sched/sch_gred.ko 2024-08-03T20:36:57.890 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/sched/sch_hfsc.ko 2024-08-03T20:36:57.890 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/sched/sch_hhf.ko 2024-08-03T20:36:57.890 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/sched/sch_htb.ko 2024-08-03T20:36:57.890 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/sched/sch_ingress.ko 2024-08-03T20:36:57.890 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/sched/sch_mqprio.ko 2024-08-03T20:36:57.890 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/sched/sch_mqprio_lib.ko 2024-08-03T20:36:57.890 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/sched/sch_multiq.ko 2024-08-03T20:36:57.890 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/sched/sch_netem.ko 2024-08-03T20:36:57.890 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/sched/sch_pie.ko 2024-08-03T20:36:57.891 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/sched/sch_plug.ko 2024-08-03T20:36:57.891 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/sched/sch_prio.ko 2024-08-03T20:36:57.891 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/sched/sch_qfq.ko 2024-08-03T20:36:57.891 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/sched/sch_red.ko 2024-08-03T20:36:57.891 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/sched/sch_sfb.ko 2024-08-03T20:36:57.891 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/sched/sch_sfq.ko 2024-08-03T20:36:57.891 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/sched/sch_tbf.ko 2024-08-03T20:36:57.891 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/sched/sch_teql.ko 2024-08-03T20:36:57.891 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/sctp 2024-08-03T20:36:57.891 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/sctp/sctp.ko 2024-08-03T20:36:57.891 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/sctp/sctp_diag.ko 2024-08-03T20:36:57.891 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/sunrpc 2024-08-03T20:36:57.891 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/sunrpc/auth_gss 2024-08-03T20:36:57.892 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/sunrpc/auth_gss/auth_rpcgss.ko 2024-08-03T20:36:57.892 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/sunrpc/auth_gss/rpcsec_gss_krb5.ko 2024-08-03T20:36:57.892 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/sunrpc/sunrpc.ko 2024-08-03T20:36:57.892 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/sunrpc/xprtrdma 2024-08-03T20:36:57.892 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/sunrpc/xprtrdma/rpcrdma.ko 2024-08-03T20:36:57.892 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/tipc 2024-08-03T20:36:57.892 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/tipc/tipc.ko 2024-08-03T20:36:57.892 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/tipc/tipc_diag.ko 2024-08-03T20:36:57.892 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/unix 2024-08-03T20:36:57.892 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/unix/unix_diag.ko 2024-08-03T20:36:57.892 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/vmw_vsock 2024-08-03T20:36:57.892 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/vmw_vsock/vmw_vsock_virtio_transport_common.ko 2024-08-03T20:36:57.892 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/vmw_vsock/vmw_vsock_vmci_transport.ko 2024-08-03T20:36:57.892 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/vmw_vsock/vsock.ko 2024-08-03T20:36:57.893 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/vmw_vsock/vsock_diag.ko 2024-08-03T20:36:57.893 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/vmw_vsock/vsock_loopback.ko 2024-08-03T20:36:57.893 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/xfrm 2024-08-03T20:36:57.893 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/net/xfrm/xfrm_ipcomp.ko 2024-08-03T20:36:57.893 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/security 2024-08-03T20:36:57.893 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/security/keys 2024-08-03T20:36:57.893 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/security/keys/encrypted-keys 2024-08-03T20:36:57.893 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/security/keys/encrypted-keys/encrypted-keys.ko 2024-08-03T20:36:57.893 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/security/keys/trusted-keys 2024-08-03T20:36:57.893 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/kernel/security/keys/trusted-keys/trusted.ko 2024-08-03T20:36:57.893 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/modules.alias 2024-08-03T20:36:57.893 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/modules.alias.bin 2024-08-03T20:36:57.894 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/modules.builtin 2024-08-03T20:36:57.894 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/modules.builtin.alias.bin 2024-08-03T20:36:57.894 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/modules.builtin.bin 2024-08-03T20:36:57.894 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/modules.builtin.modinfo 2024-08-03T20:36:57.894 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/modules.dep 2024-08-03T20:36:57.894 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/modules.dep.bin 2024-08-03T20:36:57.894 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/modules.devname 2024-08-03T20:36:57.894 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/modules.order 2024-08-03T20:36:57.894 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/modules.softdep 2024-08-03T20:36:57.894 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/modules.symbols 2024-08-03T20:36:57.894 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/modules.symbols.bin 2024-08-03T20:36:57.894 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/modules.weakdep 2024-08-03T20:36:57.894 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.11.0-rc1-gef231aaae597/vmlinuz 2024-08-03T20:36:57.898 DEBUG:teuthology.task.kernel:get_image_version: 6.11.0-rc1-gef231aaae597 2024-08-03T20:36:57.899 DEBUG:teuthology.orchestra.run.smithi118:> rpm --scripts -qp /tmp/kernel.x86_64.rpm 2024-08-03T20:36:57.921 INFO:teuthology.orchestra.run.smithi118.stdout:postinstall scriptlet (using /bin/sh): 2024-08-03T20:36:57.922 INFO:teuthology.orchestra.run.smithi118.stdout:if [ -x /usr/bin/kernel-install ]; then 2024-08-03T20:36:57.922 INFO:teuthology.orchestra.run.smithi118.stdout: /usr/bin/kernel-install add 6.11.0-rc1-gef231aaae597 /lib/modules/6.11.0-rc1-gef231aaae597/vmlinuz 2024-08-03T20:36:57.922 INFO:teuthology.orchestra.run.smithi118.stdout:fi 2024-08-03T20:36:57.922 INFO:teuthology.orchestra.run.smithi118.stdout:for file in vmlinuz System.map config; do 2024-08-03T20:36:57.922 INFO:teuthology.orchestra.run.smithi118.stdout: if ! cmp --silent "/lib/modules/6.11.0-rc1-gef231aaae597/${file}" "/boot/${file}-6.11.0-rc1-gef231aaae597"; then 2024-08-03T20:36:57.922 INFO:teuthology.orchestra.run.smithi118.stdout: cp "/lib/modules/6.11.0-rc1-gef231aaae597/${file}" "/boot/${file}-6.11.0-rc1-gef231aaae597" 2024-08-03T20:36:57.922 INFO:teuthology.orchestra.run.smithi118.stdout: fi 2024-08-03T20:36:57.922 INFO:teuthology.orchestra.run.smithi118.stdout:done 2024-08-03T20:36:57.922 INFO:teuthology.orchestra.run.smithi118.stdout:if [ -d "/lib/modules/6.11.0-rc1-gef231aaae597/dtb" ] && \ 2024-08-03T20:36:57.922 INFO:teuthology.orchestra.run.smithi118.stdout: ! diff -rq "/lib/modules/6.11.0-rc1-gef231aaae597/dtb" "/boot/dtb-6.11.0-rc1-gef231aaae597" >/dev/null 2>&1; then 2024-08-03T20:36:57.922 INFO:teuthology.orchestra.run.smithi118.stdout: rm -rf "/boot/dtb-6.11.0-rc1-gef231aaae597" 2024-08-03T20:36:57.922 INFO:teuthology.orchestra.run.smithi118.stdout: cp -r "/lib/modules/6.11.0-rc1-gef231aaae597/dtb" "/boot/dtb-6.11.0-rc1-gef231aaae597" 2024-08-03T20:36:57.923 INFO:teuthology.orchestra.run.smithi118.stdout:fi 2024-08-03T20:36:57.923 INFO:teuthology.orchestra.run.smithi118.stdout:if [ ! -e "/lib/modules/6.11.0-rc1-gef231aaae597/modules.dep" ]; then 2024-08-03T20:36:57.923 INFO:teuthology.orchestra.run.smithi118.stdout: /usr/sbin/depmod 6.11.0-rc1-gef231aaae597 2024-08-03T20:36:57.923 INFO:teuthology.orchestra.run.smithi118.stdout:fi 2024-08-03T20:36:57.923 INFO:teuthology.orchestra.run.smithi118.stdout:preuninstall scriptlet (using /bin/sh): 2024-08-03T20:36:57.923 INFO:teuthology.orchestra.run.smithi118.stdout:if [ -x /usr/bin/kernel-install ]; then 2024-08-03T20:36:57.923 INFO:teuthology.orchestra.run.smithi118.stdout:kernel-install remove 6.11.0-rc1-gef231aaae597 2024-08-03T20:36:57.923 INFO:teuthology.orchestra.run.smithi118.stdout:fi 2024-08-03T20:36:57.924 DEBUG:teuthology.orchestra.run.smithi118:> sudo rpm -qi grub2-tools 2024-08-03T20:36:57.996 INFO:teuthology.orchestra.run.smithi118.stdout:Name : grub2-tools 2024-08-03T20:36:57.996 INFO:teuthology.orchestra.run.smithi118.stdout:Epoch : 1 2024-08-03T20:36:57.996 INFO:teuthology.orchestra.run.smithi118.stdout:Version : 2.06 2024-08-03T20:36:57.996 INFO:teuthology.orchestra.run.smithi118.stdout:Release : 68.el9 2024-08-03T20:36:57.996 INFO:teuthology.orchestra.run.smithi118.stdout:Architecture: x86_64 2024-08-03T20:36:57.996 INFO:teuthology.orchestra.run.smithi118.stdout:Install Date: Fri 19 Apr 2024 09:32:39 PM UTC 2024-08-03T20:36:57.996 INFO:teuthology.orchestra.run.smithi118.stdout:Group : Unspecified 2024-08-03T20:36:57.997 INFO:teuthology.orchestra.run.smithi118.stdout:Size : 8264254 2024-08-03T20:36:57.997 INFO:teuthology.orchestra.run.smithi118.stdout:License : GPLv3+ 2024-08-03T20:36:57.997 INFO:teuthology.orchestra.run.smithi118.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-08-03T20:36:57.997 INFO:teuthology.orchestra.run.smithi118.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-08-03T20:36:57.997 INFO:teuthology.orchestra.run.smithi118.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-08-03T20:36:57.997 INFO:teuthology.orchestra.run.smithi118.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-08-03T20:36:57.997 INFO:teuthology.orchestra.run.smithi118.stdout:Packager : builder@centos.org 2024-08-03T20:36:57.997 INFO:teuthology.orchestra.run.smithi118.stdout:Vendor : CentOS 2024-08-03T20:36:57.997 INFO:teuthology.orchestra.run.smithi118.stdout:URL : http://www.gnu.org/software/grub/ 2024-08-03T20:36:57.997 INFO:teuthology.orchestra.run.smithi118.stdout:Summary : Support tools for GRUB. 2024-08-03T20:36:57.997 INFO:teuthology.orchestra.run.smithi118.stdout:Description : 2024-08-03T20:36:57.997 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-08-03T20:36:57.997 INFO:teuthology.orchestra.run.smithi118.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-08-03T20:36:57.997 INFO:teuthology.orchestra.run.smithi118.stdout:customizable bootloader with modular architecture. It supports a rich 2024-08-03T20:36:57.998 INFO:teuthology.orchestra.run.smithi118.stdout:variety of kernel formats, file systems, computer architectures and 2024-08-03T20:36:57.998 INFO:teuthology.orchestra.run.smithi118.stdout:hardware devices. 2024-08-03T20:36:57.998 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-08-03T20:36:57.998 INFO:teuthology.orchestra.run.smithi118.stdout:This subpackage provides tools for support of all platforms. 2024-08-03T20:36:57.999 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-08-03T20:36:57.999 INFO:teuthology.task.kernel:Updating grub on smithi118 to boot 6.11.0-rc1-gef231aaae597 2024-08-03T20:36:57.999 DEBUG:teuthology.orchestra.run.smithi118:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-08-03T20:36:58.543 INFO:teuthology.orchestra.run.smithi118.stderr:Generating grub configuration file ... 2024-08-03T20:36:59.939 INFO:teuthology.orchestra.run.smithi118.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-08-03T20:36:59.976 INFO:teuthology.orchestra.run.smithi118.stderr:done 2024-08-03T20:36:59.979 DEBUG:teuthology.orchestra.run.smithi118:> mktemp 2024-08-03T20:36:59.994 INFO:teuthology.orchestra.run.smithi118.stdout:/tmp/tmp.WFPzPw2U4c 2024-08-03T20:36:59.995 DEBUG:teuthology.orchestra.run.smithi118:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.WFPzPw2U4c 2024-08-03T20:37:00.060 DEBUG:teuthology.orchestra.run.smithi118:> sudo chmod 0666 /tmp/tmp.WFPzPw2U4c 2024-08-03T20:37:00.196 DEBUG:teuthology.orchestra.remote:smithi118:/tmp/tmp.WFPzPw2U4c is 6KB 2024-08-03T20:37:00.244 DEBUG:teuthology.orchestra.run.smithi118:> rm -fr /tmp/tmp.WFPzPw2U4c 2024-08-03T20:37:00.258 DEBUG:teuthology.orchestra.run.smithi118:> sudo /bin/ls /boot/loader/entries || true 2024-08-03T20:37:00.319 INFO:teuthology.orchestra.run.smithi118.stdout:7fc8828e06d0482694bcf4a447d13cb6-0-rescue.conf 2024-08-03T20:37:00.320 INFO:teuthology.orchestra.run.smithi118.stdout:7fc8828e06d0482694bcf4a447d13cb6-5.14.0-239.el9.x86_64.conf 2024-08-03T20:37:00.320 INFO:teuthology.orchestra.run.smithi118.stdout:7fc8828e06d0482694bcf4a447d13cb6-5.14.0-437.el9.x86_64.conf 2024-08-03T20:37:00.320 INFO:teuthology.orchestra.run.smithi118.stdout:7fc8828e06d0482694bcf4a447d13cb6-6.11.0-rc1-gef231aaae597.conf 2024-08-03T20:37:00.321 DEBUG:teuthology.orchestra.run.smithi118:> sudo grub2-set-default 7fc8828e06d0482694bcf4a447d13cb6-6.11.0-rc1-gef231aaae597 2024-08-03T20:37:00.466 DEBUG:teuthology.orchestra.run.smithi118:> sudo shutdown -r now 2024-08-03T20:37:30.469 INFO:teuthology.task.kernel:Checking client client.1 for new kernel version... 2024-08-03T20:37:30.470 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi118.front.sepia.ceph.com' 2024-08-03T20:37:30.471 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi118.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-08-03T20:37:48.898 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.118 2024-08-03T20:37:57.899 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi118.front.sepia.ceph.com' 2024-08-03T20:37:57.900 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi118.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-08-03T20:38:00.958 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.118 2024-08-03T20:38:12.960 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi118.front.sepia.ceph.com' 2024-08-03T20:38:12.961 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi118.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-08-03T20:38:16.030 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.118 2024-08-03T20:38:31.032 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi118.front.sepia.ceph.com' 2024-08-03T20:38:31.033 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi118.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-08-03T20:39:02.979 DEBUG:teuthology.orchestra.run.smithi118:> true 2024-08-03T20:39:03.554 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi118.front.sepia.ceph.com' 2024-08-03T20:39:03.554 INFO:teuthology.task.kernel:Checking kernel version of client.1, want "6.11.0-rc1-gef231aaae597"... 2024-08-03T20:39:03.554 DEBUG:teuthology.orchestra.run.smithi118:> uname -r 2024-08-03T20:39:03.588 INFO:teuthology.orchestra.run.smithi118.stdout:6.11.0-rc1-gef231aaae597 2024-08-03T20:39:03.589 DEBUG:teuthology.task.kernel:current kernel version is 6.11.0-rc1-gef231aaae597 vs 6.11.0-rc1-gef231aaae597 2024-08-03T20:39:03.589 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-08-03T20:39:03.589 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-08-03T20:39:04.590 DEBUG:teuthology.parallel:result is None 2024-08-03T20:39:04.590 INFO:teuthology.run_tasks:Running task internal.base... 2024-08-03T20:39:04.599 INFO:teuthology.task.internal:Creating test directory... 2024-08-03T20:39:04.599 DEBUG:teuthology.orchestra.run.smithi032:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-08-03T20:39:04.604 DEBUG:teuthology.orchestra.run.smithi118:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-08-03T20:39:04.640 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-08-03T20:39:04.646 INFO:teuthology.run_tasks:Running task internal.archive... 2024-08-03T20:39:04.654 INFO:teuthology.task.internal:Creating archive directory... 2024-08-03T20:39:04.654 DEBUG:teuthology.orchestra.run.smithi032:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-08-03T20:39:04.684 DEBUG:teuthology.orchestra.run.smithi118:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-08-03T20:39:04.737 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-08-03T20:39:04.744 INFO:teuthology.task.internal:Enabling coredump saving... 2024-08-03T20:39:04.744 DEBUG:teuthology.orchestra.run.smithi032:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-08-03T20:39:04.768 DEBUG:teuthology.orchestra.run.smithi118:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-08-03T20:39:04.896 INFO:teuthology.orchestra.run.smithi032.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-03T20:39:04.901 INFO:teuthology.orchestra.run.smithi118.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-03T20:39:04.938 INFO:teuthology.orchestra.run.smithi032.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-03T20:39:04.982 INFO:teuthology.orchestra.run.smithi118.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-03T20:39:04.985 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-08-03T20:39:04.992 INFO:teuthology.task.internal:Configuring sudo... 2024-08-03T20:39:04.992 DEBUG:teuthology.orchestra.run.smithi032:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-08-03T20:39:04.996 DEBUG:teuthology.orchestra.run.smithi118:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-08-03T20:39:05.052 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-08-03T20:39:05.062 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-08-03T20:39:05.062 DEBUG:teuthology.orchestra.run.smithi032:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-08-03T20:39:05.066 DEBUG:teuthology.orchestra.run.smithi118:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-08-03T20:39:05.099 DEBUG:teuthology.orchestra.run.smithi032:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-08-03T20:39:05.202 DEBUG:teuthology.orchestra.run.smithi032:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-08-03T20:39:05.283 DEBUG:teuthology.orchestra.run.smithi032:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-08-03T20:39:05.346 DEBUG:teuthology.orchestra.run.smithi032:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-08-03T20:39:05.416 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-08-03T20:39:05.416 DEBUG:teuthology.orchestra.run.smithi032:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-08-03T20:39:05.470 DEBUG:teuthology.orchestra.run.smithi118:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-08-03T20:39:05.534 DEBUG:teuthology.orchestra.run.smithi118:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-08-03T20:39:05.607 DEBUG:teuthology.orchestra.run.smithi118:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-08-03T20:39:05.669 DEBUG:teuthology.orchestra.run.smithi118:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-08-03T20:39:05.745 DEBUG:teuthology.orchestra.run.smithi118:> set -ex 2024-08-03T20:39:05.745 DEBUG:teuthology.orchestra.run.smithi118:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-08-03T20:39:05.800 DEBUG:teuthology.orchestra.run.smithi032:> sudo service rsyslog restart 2024-08-03T20:39:05.804 DEBUG:teuthology.orchestra.run.smithi118:> sudo service rsyslog restart 2024-08-03T20:39:05.887 INFO:teuthology.orchestra.run.smithi032.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-08-03T20:39:05.929 INFO:teuthology.orchestra.run.smithi118.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-08-03T20:39:06.447 INFO:teuthology.run_tasks:Running task internal.timer... 2024-08-03T20:39:06.455 INFO:teuthology.task.internal:Starting timer... 2024-08-03T20:39:06.455 INFO:teuthology.run_tasks:Running task pcp... 2024-08-03T20:39:06.470 INFO:teuthology.run_tasks:Running task selinux... 2024-08-03T20:39:06.482 DEBUG:teuthology.task:Applying overrides for task selinux: {'allowlist': ['scontext=system_u:system_r:getty_t:s0']} 2024-08-03T20:39:06.482 DEBUG:teuthology.orchestra.run.smithi032:> sudo service auditd rotate 2024-08-03T20:39:06.568 INFO:teuthology.orchestra.run.smithi032.stdout:Rotating logs: 2024-08-03T20:39:06.571 DEBUG:teuthology.orchestra.run.smithi118:> sudo service auditd rotate 2024-08-03T20:39:06.701 INFO:teuthology.orchestra.run.smithi118.stdout:Rotating logs: 2024-08-03T20:39:06.705 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-08-03T20:39:06.705 DEBUG:teuthology.orchestra.run.smithi032:> /usr/sbin/getenforce 2024-08-03T20:39:06.749 INFO:teuthology.orchestra.run.smithi032.stdout:Permissive 2024-08-03T20:39:06.750 DEBUG:teuthology.orchestra.run.smithi118:> /usr/sbin/getenforce 2024-08-03T20:39:06.792 INFO:teuthology.orchestra.run.smithi118.stdout:Permissive 2024-08-03T20:39:06.792 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi032.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi118.front.sepia.ceph.com': 'permissive'} 2024-08-03T20:39:06.793 DEBUG:teuthology.orchestra.run.smithi032:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-08-03T20:39:06.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-03T20:39:06.849 DEBUG:teuthology.orchestra.run.smithi118:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-08-03T20:39:06.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-03T20:39:06.908 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-08-03T20:39:06.908 DEBUG:teuthology.orchestra.run.smithi032:> sudo /usr/sbin/setenforce permissive 2024-08-03T20:39:06.967 DEBUG:teuthology.orchestra.run.smithi118:> sudo /usr/sbin/setenforce permissive 2024-08-03T20:39:07.022 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-08-03T20:39:07.032 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-08-03T20:39:07.042 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-08-03T20:39:07.043 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi032.front.sepia.ceph.com,smithi118.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-08-03T20:44:41.830 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi032.front.sepia.ceph.com'), Remote(name='ubuntu@smithi118.front.sepia.ceph.com')] 2024-08-03T20:44:41.832 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi032.front.sepia.ceph.com' 2024-08-03T20:44:41.833 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi032.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-08-03T20:44:41.926 DEBUG:teuthology.orchestra.run.smithi032:> true 2024-08-03T20:44:42.032 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi032.front.sepia.ceph.com' 2024-08-03T20:44:42.033 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi118.front.sepia.ceph.com' 2024-08-03T20:44:42.033 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi118.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-08-03T20:44:42.123 DEBUG:teuthology.orchestra.run.smithi118:> true 2024-08-03T20:44:42.234 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi118.front.sepia.ceph.com' 2024-08-03T20:44:42.235 INFO:teuthology.run_tasks:Running task clock... 2024-08-03T20:44:42.248 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-08-03T20:44:42.248 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-08-03T20:44:42.248 DEBUG:teuthology.orchestra.run.smithi032:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-08-03T20:44:42.252 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-08-03T20:44:42.252 DEBUG:teuthology.orchestra.run.smithi118:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-08-03T20:44:42.309 INFO:teuthology.orchestra.run.smithi032.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-08-03T20:44:42.337 INFO:teuthology.orchestra.run.smithi118.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-08-03T20:44:42.339 INFO:teuthology.orchestra.run.smithi032.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-08-03T20:44:42.368 INFO:teuthology.orchestra.run.smithi118.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-08-03T20:44:42.390 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: ntpd: command not found 2024-08-03T20:44:42.414 INFO:teuthology.orchestra.run.smithi032.stdout:506 Cannot talk to daemon 2024-08-03T20:44:42.418 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: ntpd: command not found 2024-08-03T20:44:42.442 INFO:teuthology.orchestra.run.smithi118.stdout:506 Cannot talk to daemon 2024-08-03T20:44:42.444 INFO:teuthology.orchestra.run.smithi032.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-08-03T20:44:42.472 INFO:teuthology.orchestra.run.smithi118.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-08-03T20:44:42.474 INFO:teuthology.orchestra.run.smithi032.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-08-03T20:44:42.502 INFO:teuthology.orchestra.run.smithi118.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-08-03T20:44:42.569 INFO:teuthology.orchestra.run.smithi032.stderr:bash: line 1: ntpq: command not found 2024-08-03T20:44:42.575 INFO:teuthology.orchestra.run.smithi032.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-08-03T20:44:42.575 INFO:teuthology.orchestra.run.smithi032.stdout:=============================================================================== 2024-08-03T20:44:42.575 INFO:teuthology.orchestra.run.smithi032.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-08-03T20:44:42.576 INFO:teuthology.orchestra.run.smithi032.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-08-03T20:44:42.576 INFO:teuthology.orchestra.run.smithi032.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-08-03T20:44:42.576 INFO:teuthology.orchestra.run.smithi032.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-08-03T20:44:42.603 INFO:teuthology.orchestra.run.smithi118.stderr:bash: line 1: ntpq: command not found 2024-08-03T20:44:42.609 INFO:teuthology.orchestra.run.smithi118.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-08-03T20:44:42.609 INFO:teuthology.orchestra.run.smithi118.stdout:=============================================================================== 2024-08-03T20:44:42.609 INFO:teuthology.orchestra.run.smithi118.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-08-03T20:44:42.609 INFO:teuthology.orchestra.run.smithi118.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-08-03T20:44:42.609 INFO:teuthology.orchestra.run.smithi118.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-08-03T20:44:42.609 INFO:teuthology.orchestra.run.smithi118.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-08-03T20:44:42.610 INFO:teuthology.run_tasks:Running task install... 2024-08-03T20:44:42.619 DEBUG:teuthology.task.install:project ceph 2024-08-03T20:44:42.619 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '3792a0a4a40f645dee271188eab388b1c22c10f8'}} 2024-08-03T20:44:42.620 DEBUG:teuthology.task.install:config {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': '3792a0a4a40f645dee271188eab388b1c22c10f8'} 2024-08-03T20:44:42.620 INFO:teuthology.task.install:Using flavor: default 2024-08-03T20:44:42.625 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-08-03T20:44:42.625 INFO:teuthology.task.install:extra packages: {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']} 2024-08-03T20:44:42.626 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '3792a0a4a40f645dee271188eab388b1c22c10f8', 'tag': None, 'wait_for_package': False} 2024-08-03T20:44:42.626 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=3792a0a4a40f645dee271188eab388b1c22c10f8 2024-08-03T20:44:42.628 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '3792a0a4a40f645dee271188eab388b1c22c10f8', 'tag': None, 'wait_for_package': False} 2024-08-03T20:44:42.629 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=3792a0a4a40f645dee271188eab388b1c22c10f8 2024-08-03T20:44:42.777 INFO:teuthology.task.install.rpm:Pulling from https://1.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240802.091913-debug/3792a0a4a40f645dee271188eab388b1c22c10f8/centos/9/flavors/default/ 2024-08-03T20:44:42.777 INFO:teuthology.task.install.rpm:Package version is 19.0.0-5476.g3792a0a4 2024-08-03T20:44:42.785 INFO:teuthology.task.install.rpm:Pulling from https://1.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240802.091913-debug/3792a0a4a40f645dee271188eab388b1c22c10f8/centos/9/flavors/default/ 2024-08-03T20:44:42.785 INFO:teuthology.task.install.rpm:Package version is 19.0.0-5476.g3792a0a4 2024-08-03T20:44:42.896 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://1.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240802.091913-debug/3792a0a4a40f645dee271188eab388b1c22c10f8/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240802.091913-debug/3792a0a4a40f645dee271188eab388b1c22c10f8/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240802.091913-debug/3792a0a4a40f645dee271188eab388b1c22c10f8/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-08-03T20:44:42.896 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-08-03T20:44:42.896 DEBUG:teuthology.orchestra.run.smithi032:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-08-03T20:44:42.900 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://1.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240802.091913-debug/3792a0a4a40f645dee271188eab388b1c22c10f8/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240802.091913-debug/3792a0a4a40f645dee271188eab388b1c22c10f8/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240802.091913-debug/3792a0a4a40f645dee271188eab388b1c22c10f8/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-08-03T20:44:42.900 DEBUG:teuthology.orchestra.run.smithi118:> set -ex 2024-08-03T20:44:42.900 DEBUG:teuthology.orchestra.run.smithi118:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-08-03T20:44:42.953 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-08-03T20:44:42.954 DEBUG:teuthology.orchestra.run.smithi032:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/3792a0a4a40f645dee271188eab388b1c22c10f8/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-08-03T20:44:42.957 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-08-03T20:44:42.957 DEBUG:teuthology.orchestra.run.smithi118:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/3792a0a4a40f645dee271188eab388b1c22c10f8/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-08-03T20:44:43.051 DEBUG:teuthology.orchestra.run.smithi032:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-08-03T20:44:43.054 DEBUG:teuthology.orchestra.run.smithi118:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-08-03T20:44:43.130 DEBUG:teuthology.orchestra.run.smithi032:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-08-03T20:44:43.132 DEBUG:teuthology.orchestra.run.smithi118:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-08-03T20:44:43.185 INFO:teuthology.orchestra.run.smithi032.stdout:check_obsoletes = 1 2024-08-03T20:44:43.187 INFO:teuthology.orchestra.run.smithi118.stdout:check_obsoletes = 1 2024-08-03T20:44:43.189 DEBUG:teuthology.orchestra.run.smithi032:> sudo yum clean all 2024-08-03T20:44:43.191 DEBUG:teuthology.orchestra.run.smithi118:> sudo yum clean all 2024-08-03T20:44:43.461 INFO:teuthology.orchestra.run.smithi032.stdout:45 files removed 2024-08-03T20:44:43.462 INFO:teuthology.orchestra.run.smithi118.stdout:45 files removed 2024-08-03T20:44:43.494 DEBUG:teuthology.orchestra.run.smithi118:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-08-03T20:44:43.496 DEBUG:teuthology.orchestra.run.smithi032:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-08-03T20:44:44.187 INFO:teuthology.orchestra.run.smithi118.stdout:ceph packages for x86_64 264 kB/s | 84 kB 00:00 2024-08-03T20:44:44.216 INFO:teuthology.orchestra.run.smithi032.stdout:ceph packages for x86_64 245 kB/s | 84 kB 00:00 2024-08-03T20:44:44.537 INFO:teuthology.orchestra.run.smithi118.stdout:ceph noarch packages 78 kB/s | 24 kB 00:00 2024-08-03T20:44:44.574 INFO:teuthology.orchestra.run.smithi032.stdout:ceph noarch packages 77 kB/s | 24 kB 00:00 2024-08-03T20:44:44.871 INFO:teuthology.orchestra.run.smithi118.stdout:ceph source packages 6.5 kB/s | 1.9 kB 00:00 2024-08-03T20:44:45.034 INFO:teuthology.orchestra.run.smithi032.stdout:ceph source packages 4.6 kB/s | 1.9 kB 00:00 2024-08-03T20:44:46.892 INFO:teuthology.orchestra.run.smithi032.stdout:CentOS Stream 9 - BaseOS 4.5 MB/s | 8.2 MB 00:01 2024-08-03T20:44:47.180 INFO:teuthology.orchestra.run.smithi118.stdout:CentOS Stream 9 - BaseOS 3.6 MB/s | 8.2 MB 00:02 2024-08-03T20:44:50.716 INFO:teuthology.orchestra.run.smithi118.stdout:CentOS Stream 9 - AppStream 8.4 MB/s | 20 MB 00:02 2024-08-03T20:44:51.718 INFO:teuthology.orchestra.run.smithi032.stdout:CentOS Stream 9 - AppStream 5.5 MB/s | 20 MB 00:03 2024-08-03T20:44:55.306 INFO:teuthology.orchestra.run.smithi118.stdout:CentOS Stream 9 - CRB 10 MB/s | 6.6 MB 00:00 2024-08-03T20:44:56.807 INFO:teuthology.orchestra.run.smithi118.stdout:CentOS Stream 9 - Extras packages 95 kB/s | 18 kB 00:00 2024-08-03T20:44:57.336 INFO:teuthology.orchestra.run.smithi032.stdout:CentOS Stream 9 - CRB 4.3 MB/s | 6.6 MB 00:01 2024-08-03T20:44:57.744 INFO:teuthology.orchestra.run.smithi118.stdout:Extra Packages for Enterprise Linux 28 MB/s | 22 MB 00:00 2024-08-03T20:44:59.161 INFO:teuthology.orchestra.run.smithi032.stdout:CentOS Stream 9 - Extras packages 40 kB/s | 18 kB 00:00 2024-08-03T20:45:00.174 INFO:teuthology.orchestra.run.smithi032.stdout:Extra Packages for Enterprise Linux 25 MB/s | 22 MB 00:00 2024-08-03T20:45:04.000 INFO:teuthology.orchestra.run.smithi118.stdout:lab-extras 29 kB/s | 1.7 kB 00:00 2024-08-03T20:45:06.505 INFO:teuthology.orchestra.run.smithi118.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-08-03T20:45:06.505 INFO:teuthology.orchestra.run.smithi118.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-08-03T20:45:06.518 INFO:teuthology.orchestra.run.smithi118.stdout:Package NetworkManager-1:1.47.4-1.el9.x86_64 is already installed. 2024-08-03T20:45:06.519 INFO:teuthology.orchestra.run.smithi118.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-08-03T20:45:06.519 INFO:teuthology.orchestra.run.smithi118.stdout:Package util-linux-2.37.4-18.el9.x86_64 is already installed. 2024-08-03T20:45:06.520 INFO:teuthology.orchestra.run.smithi118.stdout:Package libacl-devel-2.3.1-4.el9.x86_64 is already installed. 2024-08-03T20:45:06.520 INFO:teuthology.orchestra.run.smithi118.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-08-03T20:45:06.523 INFO:teuthology.orchestra.run.smithi118.stdout:Package xfsdump-3.1.12-4.el9.x86_64 is already installed. 2024-08-03T20:45:06.523 INFO:teuthology.orchestra.run.smithi118.stdout:Package xfsprogs-6.3.0-1.el9.x86_64 is already installed. 2024-08-03T20:45:06.589 INFO:teuthology.orchestra.run.smithi118.stdout:Dependencies resolved. 2024-08-03T20:45:06.612 INFO:teuthology.orchestra.run.smithi118.stdout:======================================================================================= 2024-08-03T20:45:06.613 INFO:teuthology.orchestra.run.smithi118.stdout: Package Arch Version Repository Size 2024-08-03T20:45:06.613 INFO:teuthology.orchestra.run.smithi118.stdout:======================================================================================= 2024-08-03T20:45:06.613 INFO:teuthology.orchestra.run.smithi118.stdout:Installing: 2024-08-03T20:45:06.613 INFO:teuthology.orchestra.run.smithi118.stdout: bison x86_64 3.7.4-5.el9 appstream 943 k 2024-08-03T20:45:06.613 INFO:teuthology.orchestra.run.smithi118.stdout: ceph x86_64 2:19.0.0-5476.g3792a0a4.el9 ceph 6.6 k 2024-08-03T20:45:06.613 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-base x86_64 2:19.0.0-5476.g3792a0a4.el9 ceph 5.5 M 2024-08-03T20:45:06.614 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-fuse x86_64 2:19.0.0-5476.g3792a0a4.el9 ceph 872 k 2024-08-03T20:45:06.614 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-5476.g3792a0a4.el9 ceph 148 k 2024-08-03T20:45:06.614 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-mgr x86_64 2:19.0.0-5476.g3792a0a4.el9 ceph 1.5 M 2024-08-03T20:45:06.614 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-mgr-cephadm noarch 2:19.0.0-5476.g3792a0a4.el9 ceph-noarch 150 k 2024-08-03T20:45:06.614 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-mgr-dashboard noarch 2:19.0.0-5476.g3792a0a4.el9 ceph-noarch 14 M 2024-08-03T20:45:06.614 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-5476.g3792a0a4.el9 ceph-noarch 7.4 M 2024-08-03T20:45:06.615 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-mgr-rook noarch 2:19.0.0-5476.g3792a0a4.el9 ceph-noarch 49 k 2024-08-03T20:45:06.615 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-radosgw x86_64 2:19.0.0-5476.g3792a0a4.el9 ceph 10 M 2024-08-03T20:45:06.615 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-test x86_64 2:19.0.0-5476.g3792a0a4.el9 ceph 44 M 2024-08-03T20:45:06.615 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-volume noarch 2:19.0.0-5476.g3792a0a4.el9 ceph-noarch 276 k 2024-08-03T20:45:06.615 INFO:teuthology.orchestra.run.smithi118.stdout: cephadm noarch 2:19.0.0-5476.g3792a0a4.el9 ceph-noarch 771 k 2024-08-03T20:45:06.615 INFO:teuthology.orchestra.run.smithi118.stdout: cephfs-mirror x86_64 2:19.0.0-5476.g3792a0a4.el9 ceph 225 k 2024-08-03T20:45:06.615 INFO:teuthology.orchestra.run.smithi118.stdout: cephfs-shell x86_64 2:19.0.0-5476.g3792a0a4.el9 ceph 27 k 2024-08-03T20:45:06.615 INFO:teuthology.orchestra.run.smithi118.stdout: cephfs-top noarch 2:19.0.0-5476.g3792a0a4.el9 ceph-noarch 18 k 2024-08-03T20:45:06.615 INFO:teuthology.orchestra.run.smithi118.stdout: elfutils-libelf-devel x86_64 0.191-4.el9 appstream 22 k 2024-08-03T20:45:06.615 INFO:teuthology.orchestra.run.smithi118.stdout: flex x86_64 2.6.4-9.el9 appstream 313 k 2024-08-03T20:45:06.616 INFO:teuthology.orchestra.run.smithi118.stdout: libaio-devel x86_64 0.3.111-13.el9 appstream 11 k 2024-08-03T20:45:06.616 INFO:teuthology.orchestra.run.smithi118.stdout: libcephfs-devel x86_64 2:19.0.0-5476.g3792a0a4.el9 ceph 32 k 2024-08-03T20:45:06.616 INFO:teuthology.orchestra.run.smithi118.stdout: libcephfs2 x86_64 2:19.0.0-5476.g3792a0a4.el9 ceph 767 k 2024-08-03T20:45:06.616 INFO:teuthology.orchestra.run.smithi118.stdout: librados-devel x86_64 2:19.0.0-5476.g3792a0a4.el9 ceph 127 k 2024-08-03T20:45:06.616 INFO:teuthology.orchestra.run.smithi118.stdout: libtool x86_64 2.4.6-46.el9 appstream 581 k 2024-08-03T20:45:06.616 INFO:teuthology.orchestra.run.smithi118.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-08-03T20:45:06.616 INFO:teuthology.orchestra.run.smithi118.stdout: openssl-devel x86_64 1:3.2.2-2.el9 appstream 4.4 M 2024-08-03T20:45:06.616 INFO:teuthology.orchestra.run.smithi118.stdout: postgresql x86_64 13.14-1.el9 appstream 1.6 M 2024-08-03T20:45:06.617 INFO:teuthology.orchestra.run.smithi118.stdout: postgresql-contrib x86_64 13.14-1.el9 appstream 911 k 2024-08-03T20:45:06.617 INFO:teuthology.orchestra.run.smithi118.stdout: postgresql-server x86_64 13.14-1.el9 appstream 5.8 M 2024-08-03T20:45:06.617 INFO:teuthology.orchestra.run.smithi118.stdout: python3-cephfs x86_64 2:19.0.0-5476.g3792a0a4.el9 ceph 188 k 2024-08-03T20:45:06.617 INFO:teuthology.orchestra.run.smithi118.stdout: python3-rados x86_64 2:19.0.0-5476.g3792a0a4.el9 ceph 371 k 2024-08-03T20:45:06.617 INFO:teuthology.orchestra.run.smithi118.stdout: python3-rbd x86_64 2:19.0.0-5476.g3792a0a4.el9 ceph 355 k 2024-08-03T20:45:06.617 INFO:teuthology.orchestra.run.smithi118.stdout: python3-rgw x86_64 2:19.0.0-5476.g3792a0a4.el9 ceph 116 k 2024-08-03T20:45:06.617 INFO:teuthology.orchestra.run.smithi118.stdout: rbd-fuse x86_64 2:19.0.0-5476.g3792a0a4.el9 ceph 86 k 2024-08-03T20:45:06.617 INFO:teuthology.orchestra.run.smithi118.stdout: rbd-mirror x86_64 2:19.0.0-5476.g3792a0a4.el9 ceph 3.1 M 2024-08-03T20:45:06.618 INFO:teuthology.orchestra.run.smithi118.stdout: rbd-nbd x86_64 2:19.0.0-5476.g3792a0a4.el9 ceph 170 k 2024-08-03T20:45:06.618 INFO:teuthology.orchestra.run.smithi118.stdout: xfsprogs-devel x86_64 6.4.0-3.el9 appstream 96 k 2024-08-03T20:45:06.618 INFO:teuthology.orchestra.run.smithi118.stdout:Upgrading: 2024-08-03T20:45:06.618 INFO:teuthology.orchestra.run.smithi118.stdout: NetworkManager x86_64 1:1.48.2-2.el9 baseos 2.3 M 2024-08-03T20:45:06.618 INFO:teuthology.orchestra.run.smithi118.stdout: NetworkManager-libnm x86_64 1:1.48.2-2.el9 baseos 1.8 M 2024-08-03T20:45:06.618 INFO:teuthology.orchestra.run.smithi118.stdout: NetworkManager-team x86_64 1:1.48.2-2.el9 baseos 41 k 2024-08-03T20:45:06.618 INFO:teuthology.orchestra.run.smithi118.stdout: NetworkManager-tui x86_64 1:1.48.2-2.el9 baseos 247 k 2024-08-03T20:45:06.618 INFO:teuthology.orchestra.run.smithi118.stdout: elfutils-debuginfod-client x86_64 0.191-4.el9 baseos 37 k 2024-08-03T20:45:06.619 INFO:teuthology.orchestra.run.smithi118.stdout: elfutils-libelf x86_64 0.191-4.el9 baseos 207 k 2024-08-03T20:45:06.619 INFO:teuthology.orchestra.run.smithi118.stdout: elfutils-libs x86_64 0.191-4.el9 baseos 260 k 2024-08-03T20:45:06.619 INFO:teuthology.orchestra.run.smithi118.stdout: librados2 x86_64 2:19.0.0-5476.g3792a0a4.el9 ceph 3.4 M 2024-08-03T20:45:06.619 INFO:teuthology.orchestra.run.smithi118.stdout: librbd1 x86_64 2:19.0.0-5476.g3792a0a4.el9 ceph 3.1 M 2024-08-03T20:45:06.619 INFO:teuthology.orchestra.run.smithi118.stdout: openssl x86_64 1:3.2.2-2.el9 baseos 1.4 M 2024-08-03T20:45:06.619 INFO:teuthology.orchestra.run.smithi118.stdout: openssl-libs x86_64 1:3.2.2-2.el9 baseos 2.4 M 2024-08-03T20:45:06.619 INFO:teuthology.orchestra.run.smithi118.stdout: xfsprogs x86_64 6.4.0-3.el9 baseos 1.1 M 2024-08-03T20:45:06.619 INFO:teuthology.orchestra.run.smithi118.stdout:Installing dependencies: 2024-08-03T20:45:06.620 INFO:teuthology.orchestra.run.smithi118.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-08-03T20:45:06.620 INFO:teuthology.orchestra.run.smithi118.stdout: automake noarch 1.16.2-8.el9 appstream 689 k 2024-08-03T20:45:06.620 INFO:teuthology.orchestra.run.smithi118.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-08-03T20:45:06.620 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-common x86_64 2:19.0.0-5476.g3792a0a4.el9 ceph 20 M 2024-08-03T20:45:06.620 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-grafana-dashboards noarch 2:19.0.0-5476.g3792a0a4.el9 ceph-noarch 34 k 2024-08-03T20:45:06.620 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-mds x86_64 2:19.0.0-5476.g3792a0a4.el9 ceph 2.3 M 2024-08-03T20:45:06.620 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-mgr-modules-core noarch 2:19.0.0-5476.g3792a0a4.el9 ceph-noarch 284 k 2024-08-03T20:45:06.621 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-mon x86_64 2:19.0.0-5476.g3792a0a4.el9 ceph 4.9 M 2024-08-03T20:45:06.621 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-osd x86_64 2:19.0.0-5476.g3792a0a4.el9 ceph 18 M 2024-08-03T20:45:06.621 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-prometheus-alerts noarch 2:19.0.0-5476.g3792a0a4.el9 ceph-noarch 16 k 2024-08-03T20:45:06.621 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-selinux x86_64 2:19.0.0-5476.g3792a0a4.el9 ceph 25 k 2024-08-03T20:45:06.621 INFO:teuthology.orchestra.run.smithi118.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-08-03T20:45:06.621 INFO:teuthology.orchestra.run.smithi118.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-08-03T20:45:06.621 INFO:teuthology.orchestra.run.smithi118.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-08-03T20:45:06.622 INFO:teuthology.orchestra.run.smithi118.stdout: fuse x86_64 2.9.9-16.el9 baseos 80 k 2024-08-03T20:45:06.622 INFO:teuthology.orchestra.run.smithi118.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-08-03T20:45:06.622 INFO:teuthology.orchestra.run.smithi118.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-08-03T20:45:06.622 INFO:teuthology.orchestra.run.smithi118.stdout: libarrow x86_64 9.0.0-11.el9 epel 4.4 M 2024-08-03T20:45:06.622 INFO:teuthology.orchestra.run.smithi118.stdout: libarrow-doc noarch 9.0.0-11.el9 epel 26 k 2024-08-03T20:45:06.622 INFO:teuthology.orchestra.run.smithi118.stdout: libcephsqlite x86_64 2:19.0.0-5476.g3792a0a4.el9 ceph 158 k 2024-08-03T20:45:06.622 INFO:teuthology.orchestra.run.smithi118.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-08-03T20:45:06.622 INFO:teuthology.orchestra.run.smithi118.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-08-03T20:45:06.623 INFO:teuthology.orchestra.run.smithi118.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-08-03T20:45:06.623 INFO:teuthology.orchestra.run.smithi118.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-08-03T20:45:06.623 INFO:teuthology.orchestra.run.smithi118.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-08-03T20:45:06.623 INFO:teuthology.orchestra.run.smithi118.stdout: libradosstriper1 x86_64 2:19.0.0-5476.g3792a0a4.el9 ceph 504 k 2024-08-03T20:45:06.623 INFO:teuthology.orchestra.run.smithi118.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-08-03T20:45:06.623 INFO:teuthology.orchestra.run.smithi118.stdout: librgw2 x86_64 2:19.0.0-5476.g3792a0a4.el9 ceph 5.3 M 2024-08-03T20:45:06.623 INFO:teuthology.orchestra.run.smithi118.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-08-03T20:45:06.624 INFO:teuthology.orchestra.run.smithi118.stdout: libzstd-devel x86_64 1.5.1-2.el9 appstream 47 k 2024-08-03T20:45:06.624 INFO:teuthology.orchestra.run.smithi118.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-08-03T20:45:06.624 INFO:teuthology.orchestra.run.smithi118.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-08-03T20:45:06.624 INFO:teuthology.orchestra.run.smithi118.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-08-03T20:45:06.624 INFO:teuthology.orchestra.run.smithi118.stdout: luarocks noarch 3.9.2-5.el9 epel 151 k 2024-08-03T20:45:06.624 INFO:teuthology.orchestra.run.smithi118.stdout: openblas x86_64 0.3.21-2.el9 appstream 35 k 2024-08-03T20:45:06.624 INFO:teuthology.orchestra.run.smithi118.stdout: openblas-openmp x86_64 0.3.21-2.el9 appstream 4.7 M 2024-08-03T20:45:06.624 INFO:teuthology.orchestra.run.smithi118.stdout: parquet-libs x86_64 9.0.0-11.el9 epel 839 k 2024-08-03T20:45:06.624 INFO:teuthology.orchestra.run.smithi118.stdout: postgresql-private-libs x86_64 13.14-1.el9 appstream 133 k 2024-08-03T20:45:06.624 INFO:teuthology.orchestra.run.smithi118.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-08-03T20:45:06.625 INFO:teuthology.orchestra.run.smithi118.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-08-03T20:45:06.625 INFO:teuthology.orchestra.run.smithi118.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-08-03T20:45:06.625 INFO:teuthology.orchestra.run.smithi118.stdout: python3-attrs noarch 20.3.0-7.el9 appstream 87 k 2024-08-03T20:45:06.625 INFO:teuthology.orchestra.run.smithi118.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-08-03T20:45:06.625 INFO:teuthology.orchestra.run.smithi118.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-08-03T20:45:06.625 INFO:teuthology.orchestra.run.smithi118.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-08-03T20:45:06.626 INFO:teuthology.orchestra.run.smithi118.stdout: python3-ceph-argparse x86_64 2:19.0.0-5476.g3792a0a4.el9 ceph 45 k 2024-08-03T20:45:06.626 INFO:teuthology.orchestra.run.smithi118.stdout: python3-ceph-common x86_64 2:19.0.0-5476.g3792a0a4.el9 ceph 138 k 2024-08-03T20:45:06.626 INFO:teuthology.orchestra.run.smithi118.stdout: python3-certifi noarch 2023.05.07-4.el9 epel 14 k 2024-08-03T20:45:06.626 INFO:teuthology.orchestra.run.smithi118.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-08-03T20:45:06.626 INFO:teuthology.orchestra.run.smithi118.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-08-03T20:45:06.626 INFO:teuthology.orchestra.run.smithi118.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-08-03T20:45:06.626 INFO:teuthology.orchestra.run.smithi118.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-08-03T20:45:06.626 INFO:teuthology.orchestra.run.smithi118.stdout: python3-colorama noarch 0.4.6-3.el9 appstream 61 k 2024-08-03T20:45:06.626 INFO:teuthology.orchestra.run.smithi118.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-08-03T20:45:06.627 INFO:teuthology.orchestra.run.smithi118.stdout: python3-devel x86_64 3.9.18-3.el9 appstream 244 k 2024-08-03T20:45:06.627 INFO:teuthology.orchestra.run.smithi118.stdout: python3-google-auth noarch 1:2.32.0-1.el9 epel 216 k 2024-08-03T20:45:06.627 INFO:teuthology.orchestra.run.smithi118.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-08-03T20:45:06.627 INFO:teuthology.orchestra.run.smithi118.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-08-03T20:45:06.627 INFO:teuthology.orchestra.run.smithi118.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-08-03T20:45:06.627 INFO:teuthology.orchestra.run.smithi118.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-08-03T20:45:06.627 INFO:teuthology.orchestra.run.smithi118.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-08-03T20:45:06.627 INFO:teuthology.orchestra.run.smithi118.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-08-03T20:45:06.627 INFO:teuthology.orchestra.run.smithi118.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-08-03T20:45:06.628 INFO:teuthology.orchestra.run.smithi118.stdout: python3-jinja2 noarch 2.11.3-6.el9 appstream 248 k 2024-08-03T20:45:06.628 INFO:teuthology.orchestra.run.smithi118.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-08-03T20:45:06.628 INFO:teuthology.orchestra.run.smithi118.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-08-03T20:45:06.628 INFO:teuthology.orchestra.run.smithi118.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-08-03T20:45:06.628 INFO:teuthology.orchestra.run.smithi118.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-08-03T20:45:06.628 INFO:teuthology.orchestra.run.smithi118.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-08-03T20:45:06.628 INFO:teuthology.orchestra.run.smithi118.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-08-03T20:45:06.629 INFO:teuthology.orchestra.run.smithi118.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-08-03T20:45:06.629 INFO:teuthology.orchestra.run.smithi118.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-08-03T20:45:06.629 INFO:teuthology.orchestra.run.smithi118.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-08-03T20:45:06.629 INFO:teuthology.orchestra.run.smithi118.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-08-03T20:45:06.629 INFO:teuthology.orchestra.run.smithi118.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-08-03T20:45:06.629 INFO:teuthology.orchestra.run.smithi118.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-08-03T20:45:06.630 INFO:teuthology.orchestra.run.smithi118.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-08-03T20:45:06.630 INFO:teuthology.orchestra.run.smithi118.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-08-03T20:45:06.630 INFO:teuthology.orchestra.run.smithi118.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-08-03T20:45:06.630 INFO:teuthology.orchestra.run.smithi118.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-08-03T20:45:06.630 INFO:teuthology.orchestra.run.smithi118.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-08-03T20:45:06.630 INFO:teuthology.orchestra.run.smithi118.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-08-03T20:45:06.630 INFO:teuthology.orchestra.run.smithi118.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-08-03T20:45:06.630 INFO:teuthology.orchestra.run.smithi118.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-08-03T20:45:06.630 INFO:teuthology.orchestra.run.smithi118.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-08-03T20:45:06.631 INFO:teuthology.orchestra.run.smithi118.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-08-03T20:45:06.631 INFO:teuthology.orchestra.run.smithi118.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-08-03T20:45:06.631 INFO:teuthology.orchestra.run.smithi118.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-08-03T20:45:06.631 INFO:teuthology.orchestra.run.smithi118.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-08-03T20:45:06.631 INFO:teuthology.orchestra.run.smithi118.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-08-03T20:45:06.631 INFO:teuthology.orchestra.run.smithi118.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-08-03T20:45:06.631 INFO:teuthology.orchestra.run.smithi118.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-08-03T20:45:06.631 INFO:teuthology.orchestra.run.smithi118.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-08-03T20:45:06.631 INFO:teuthology.orchestra.run.smithi118.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-08-03T20:45:06.631 INFO:teuthology.orchestra.run.smithi118.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-08-03T20:45:06.632 INFO:teuthology.orchestra.run.smithi118.stdout: python3-typing-extensions noarch 4.12.2-2.el9 epel 74 k 2024-08-03T20:45:06.632 INFO:teuthology.orchestra.run.smithi118.stdout: python3-urllib3 noarch 1.26.5-6.el9 baseos 215 k 2024-08-03T20:45:06.632 INFO:teuthology.orchestra.run.smithi118.stdout: python3-wcwidth noarch 0.2.5-8.el9 appstream 44 k 2024-08-03T20:45:06.632 INFO:teuthology.orchestra.run.smithi118.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-08-03T20:45:06.632 INFO:teuthology.orchestra.run.smithi118.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-08-03T20:45:06.633 INFO:teuthology.orchestra.run.smithi118.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-08-03T20:45:06.633 INFO:teuthology.orchestra.run.smithi118.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-08-03T20:45:06.633 INFO:teuthology.orchestra.run.smithi118.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-08-03T20:45:06.633 INFO:teuthology.orchestra.run.smithi118.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-08-03T20:45:06.633 INFO:teuthology.orchestra.run.smithi118.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-08-03T20:45:06.634 INFO:teuthology.orchestra.run.smithi118.stdout: socat x86_64 1.7.4.1-6.el9 appstream 304 k 2024-08-03T20:45:06.634 INFO:teuthology.orchestra.run.smithi118.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-08-03T20:45:06.634 INFO:teuthology.orchestra.run.smithi118.stdout: uuid x86_64 1.6.2-55.el9 appstream 58 k 2024-08-03T20:45:06.634 INFO:teuthology.orchestra.run.smithi118.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-08-03T20:45:06.634 INFO:teuthology.orchestra.run.smithi118.stdout: zlib-devel x86_64 1.2.11-41.el9 appstream 45 k 2024-08-03T20:45:06.634 INFO:teuthology.orchestra.run.smithi118.stdout:Installing weak dependencies: 2024-08-03T20:45:06.634 INFO:teuthology.orchestra.run.smithi118.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-08-03T20:45:06.635 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-08-03T20:45:06.635 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction Summary 2024-08-03T20:45:06.635 INFO:teuthology.orchestra.run.smithi118.stdout:======================================================================================= 2024-08-03T20:45:06.635 INFO:teuthology.orchestra.run.smithi118.stdout:Install 148 Packages 2024-08-03T20:45:06.635 INFO:teuthology.orchestra.run.smithi118.stdout:Upgrade 12 Packages 2024-08-03T20:45:06.635 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-08-03T20:45:06.636 INFO:teuthology.orchestra.run.smithi118.stdout:Total download size: 233 M 2024-08-03T20:45:06.636 INFO:teuthology.orchestra.run.smithi118.stdout:Downloading Packages: 2024-08-03T20:45:06.664 INFO:teuthology.orchestra.run.smithi032.stdout:lab-extras 30 kB/s | 1.7 kB 00:00 2024-08-03T20:45:07.343 INFO:teuthology.orchestra.run.smithi118.stdout:(1/160): ceph-19.0.0-5476.g3792a0a4.el9.x86_64. 34 kB/s | 6.6 kB 00:00 2024-08-03T20:45:07.594 INFO:teuthology.orchestra.run.smithi118.stdout:(2/160): ceph-fuse-19.0.0-5476.g3792a0a4.el9.x8 3.4 MB/s | 872 kB 00:00 2024-08-03T20:45:07.661 INFO:teuthology.orchestra.run.smithi118.stdout:(3/160): ceph-base-19.0.0-5476.g3792a0a4.el9.x8 11 MB/s | 5.5 MB 00:00 2024-08-03T20:45:07.686 INFO:teuthology.orchestra.run.smithi118.stdout:(4/160): ceph-immutable-object-cache-19.0.0-547 1.6 MB/s | 148 kB 00:00 2024-08-03T20:45:07.786 INFO:teuthology.orchestra.run.smithi118.stdout:(5/160): ceph-mds-19.0.0-5476.g3792a0a4.el9.x86 18 MB/s | 2.3 MB 00:00 2024-08-03T20:45:07.995 INFO:teuthology.orchestra.run.smithi118.stdout:(6/160): ceph-mgr-19.0.0-5476.g3792a0a4.el9.x86 4.9 MB/s | 1.5 MB 00:00 2024-08-03T20:45:08.367 INFO:teuthology.orchestra.run.smithi118.stdout:(7/160): ceph-mon-19.0.0-5476.g3792a0a4.el9.x86 8.4 MB/s | 4.9 MB 00:00 2024-08-03T20:45:08.637 INFO:teuthology.orchestra.run.smithi118.stdout:(8/160): ceph-common-19.0.0-5476.g3792a0a4.el9. 13 MB/s | 20 MB 00:01 2024-08-03T20:45:08.679 INFO:teuthology.orchestra.run.smithi118.stdout:(9/160): ceph-selinux-19.0.0-5476.g3792a0a4.el9 611 kB/s | 25 kB 00:00 2024-08-03T20:45:08.971 INFO:teuthology.orchestra.run.smithi118.stdout:(10/160): ceph-radosgw-19.0.0-5476.g3792a0a4.el 17 MB/s | 10 MB 00:00 2024-08-03T20:45:09.029 INFO:teuthology.orchestra.run.smithi118.stdout:(11/160): cephfs-mirror-19.0.0-5476.g3792a0a4.e 3.8 MB/s | 225 kB 00:00 2024-08-03T20:45:09.096 INFO:teuthology.orchestra.run.smithi118.stdout:(12/160): cephfs-shell-19.0.0-5476.g3792a0a4.el 407 kB/s | 27 kB 00:00 2024-08-03T20:45:09.172 INFO:teuthology.orchestra.run.smithi118.stdout:(13/160): libcephfs-devel-19.0.0-5476.g3792a0a4 424 kB/s | 32 kB 00:00 2024-08-03T20:45:09.242 INFO:teuthology.orchestra.run.smithi032.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-08-03T20:45:09.243 INFO:teuthology.orchestra.run.smithi032.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-08-03T20:45:09.255 INFO:teuthology.orchestra.run.smithi032.stdout:Package NetworkManager-1:1.47.4-1.el9.x86_64 is already installed. 2024-08-03T20:45:09.256 INFO:teuthology.orchestra.run.smithi032.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-08-03T20:45:09.256 INFO:teuthology.orchestra.run.smithi032.stdout:Package util-linux-2.37.4-18.el9.x86_64 is already installed. 2024-08-03T20:45:09.257 INFO:teuthology.orchestra.run.smithi032.stdout:Package libacl-devel-2.3.1-4.el9.x86_64 is already installed. 2024-08-03T20:45:09.259 INFO:teuthology.orchestra.run.smithi032.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-08-03T20:45:09.261 INFO:teuthology.orchestra.run.smithi032.stdout:Package xfsdump-3.1.12-4.el9.x86_64 is already installed. 2024-08-03T20:45:09.262 INFO:teuthology.orchestra.run.smithi032.stdout:Package xfsprogs-6.3.0-1.el9.x86_64 is already installed. 2024-08-03T20:45:09.272 INFO:teuthology.orchestra.run.smithi118.stdout:(14/160): libcephfs2-19.0.0-5476.g3792a0a4.el9. 7.5 MB/s | 767 kB 00:00 2024-08-03T20:45:09.332 INFO:teuthology.orchestra.run.smithi032.stdout:Dependencies resolved. 2024-08-03T20:45:09.352 INFO:teuthology.orchestra.run.smithi032.stdout:======================================================================================= 2024-08-03T20:45:09.352 INFO:teuthology.orchestra.run.smithi032.stdout: Package Arch Version Repository Size 2024-08-03T20:45:09.352 INFO:teuthology.orchestra.run.smithi032.stdout:======================================================================================= 2024-08-03T20:45:09.353 INFO:teuthology.orchestra.run.smithi032.stdout:Installing: 2024-08-03T20:45:09.353 INFO:teuthology.orchestra.run.smithi032.stdout: bison x86_64 3.7.4-5.el9 appstream 943 k 2024-08-03T20:45:09.353 INFO:teuthology.orchestra.run.smithi032.stdout: ceph x86_64 2:19.0.0-5476.g3792a0a4.el9 ceph 6.6 k 2024-08-03T20:45:09.353 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-base x86_64 2:19.0.0-5476.g3792a0a4.el9 ceph 5.5 M 2024-08-03T20:45:09.353 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-fuse x86_64 2:19.0.0-5476.g3792a0a4.el9 ceph 872 k 2024-08-03T20:45:09.353 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-5476.g3792a0a4.el9 ceph 148 k 2024-08-03T20:45:09.353 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-mgr x86_64 2:19.0.0-5476.g3792a0a4.el9 ceph 1.5 M 2024-08-03T20:45:09.353 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-mgr-cephadm noarch 2:19.0.0-5476.g3792a0a4.el9 ceph-noarch 150 k 2024-08-03T20:45:09.353 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-mgr-dashboard noarch 2:19.0.0-5476.g3792a0a4.el9 ceph-noarch 14 M 2024-08-03T20:45:09.353 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-5476.g3792a0a4.el9 ceph-noarch 7.4 M 2024-08-03T20:45:09.353 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-mgr-rook noarch 2:19.0.0-5476.g3792a0a4.el9 ceph-noarch 49 k 2024-08-03T20:45:09.353 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-radosgw x86_64 2:19.0.0-5476.g3792a0a4.el9 ceph 10 M 2024-08-03T20:45:09.353 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-test x86_64 2:19.0.0-5476.g3792a0a4.el9 ceph 44 M 2024-08-03T20:45:09.354 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-volume noarch 2:19.0.0-5476.g3792a0a4.el9 ceph-noarch 276 k 2024-08-03T20:45:09.354 INFO:teuthology.orchestra.run.smithi032.stdout: cephadm noarch 2:19.0.0-5476.g3792a0a4.el9 ceph-noarch 771 k 2024-08-03T20:45:09.354 INFO:teuthology.orchestra.run.smithi032.stdout: cephfs-mirror x86_64 2:19.0.0-5476.g3792a0a4.el9 ceph 225 k 2024-08-03T20:45:09.354 INFO:teuthology.orchestra.run.smithi032.stdout: cephfs-shell x86_64 2:19.0.0-5476.g3792a0a4.el9 ceph 27 k 2024-08-03T20:45:09.354 INFO:teuthology.orchestra.run.smithi032.stdout: cephfs-top noarch 2:19.0.0-5476.g3792a0a4.el9 ceph-noarch 18 k 2024-08-03T20:45:09.354 INFO:teuthology.orchestra.run.smithi032.stdout: elfutils-libelf-devel x86_64 0.191-4.el9 appstream 22 k 2024-08-03T20:45:09.354 INFO:teuthology.orchestra.run.smithi032.stdout: flex x86_64 2.6.4-9.el9 appstream 313 k 2024-08-03T20:45:09.354 INFO:teuthology.orchestra.run.smithi032.stdout: libaio-devel x86_64 0.3.111-13.el9 appstream 11 k 2024-08-03T20:45:09.354 INFO:teuthology.orchestra.run.smithi032.stdout: libcephfs-devel x86_64 2:19.0.0-5476.g3792a0a4.el9 ceph 32 k 2024-08-03T20:45:09.354 INFO:teuthology.orchestra.run.smithi032.stdout: libcephfs2 x86_64 2:19.0.0-5476.g3792a0a4.el9 ceph 767 k 2024-08-03T20:45:09.354 INFO:teuthology.orchestra.run.smithi032.stdout: librados-devel x86_64 2:19.0.0-5476.g3792a0a4.el9 ceph 127 k 2024-08-03T20:45:09.354 INFO:teuthology.orchestra.run.smithi032.stdout: libtool x86_64 2.4.6-46.el9 appstream 581 k 2024-08-03T20:45:09.354 INFO:teuthology.orchestra.run.smithi032.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-08-03T20:45:09.355 INFO:teuthology.orchestra.run.smithi032.stdout: openssl-devel x86_64 1:3.2.2-2.el9 appstream 4.4 M 2024-08-03T20:45:09.355 INFO:teuthology.orchestra.run.smithi032.stdout: postgresql x86_64 13.14-1.el9 appstream 1.6 M 2024-08-03T20:45:09.355 INFO:teuthology.orchestra.run.smithi032.stdout: postgresql-contrib x86_64 13.14-1.el9 appstream 911 k 2024-08-03T20:45:09.355 INFO:teuthology.orchestra.run.smithi032.stdout: postgresql-server x86_64 13.14-1.el9 appstream 5.8 M 2024-08-03T20:45:09.355 INFO:teuthology.orchestra.run.smithi032.stdout: python3-cephfs x86_64 2:19.0.0-5476.g3792a0a4.el9 ceph 188 k 2024-08-03T20:45:09.355 INFO:teuthology.orchestra.run.smithi032.stdout: python3-rados x86_64 2:19.0.0-5476.g3792a0a4.el9 ceph 371 k 2024-08-03T20:45:09.355 INFO:teuthology.orchestra.run.smithi032.stdout: python3-rbd x86_64 2:19.0.0-5476.g3792a0a4.el9 ceph 355 k 2024-08-03T20:45:09.355 INFO:teuthology.orchestra.run.smithi032.stdout: python3-rgw x86_64 2:19.0.0-5476.g3792a0a4.el9 ceph 116 k 2024-08-03T20:45:09.355 INFO:teuthology.orchestra.run.smithi032.stdout: rbd-fuse x86_64 2:19.0.0-5476.g3792a0a4.el9 ceph 86 k 2024-08-03T20:45:09.355 INFO:teuthology.orchestra.run.smithi032.stdout: rbd-mirror x86_64 2:19.0.0-5476.g3792a0a4.el9 ceph 3.1 M 2024-08-03T20:45:09.355 INFO:teuthology.orchestra.run.smithi032.stdout: rbd-nbd x86_64 2:19.0.0-5476.g3792a0a4.el9 ceph 170 k 2024-08-03T20:45:09.355 INFO:teuthology.orchestra.run.smithi032.stdout: xfsprogs-devel x86_64 6.4.0-3.el9 appstream 96 k 2024-08-03T20:45:09.355 INFO:teuthology.orchestra.run.smithi032.stdout:Upgrading: 2024-08-03T20:45:09.356 INFO:teuthology.orchestra.run.smithi032.stdout: NetworkManager x86_64 1:1.48.2-2.el9 baseos 2.3 M 2024-08-03T20:45:09.356 INFO:teuthology.orchestra.run.smithi032.stdout: NetworkManager-libnm x86_64 1:1.48.2-2.el9 baseos 1.8 M 2024-08-03T20:45:09.356 INFO:teuthology.orchestra.run.smithi032.stdout: NetworkManager-team x86_64 1:1.48.2-2.el9 baseos 41 k 2024-08-03T20:45:09.356 INFO:teuthology.orchestra.run.smithi032.stdout: NetworkManager-tui x86_64 1:1.48.2-2.el9 baseos 247 k 2024-08-03T20:45:09.356 INFO:teuthology.orchestra.run.smithi032.stdout: elfutils-debuginfod-client x86_64 0.191-4.el9 baseos 37 k 2024-08-03T20:45:09.356 INFO:teuthology.orchestra.run.smithi032.stdout: elfutils-libelf x86_64 0.191-4.el9 baseos 207 k 2024-08-03T20:45:09.356 INFO:teuthology.orchestra.run.smithi032.stdout: elfutils-libs x86_64 0.191-4.el9 baseos 260 k 2024-08-03T20:45:09.356 INFO:teuthology.orchestra.run.smithi032.stdout: librados2 x86_64 2:19.0.0-5476.g3792a0a4.el9 ceph 3.4 M 2024-08-03T20:45:09.356 INFO:teuthology.orchestra.run.smithi032.stdout: librbd1 x86_64 2:19.0.0-5476.g3792a0a4.el9 ceph 3.1 M 2024-08-03T20:45:09.356 INFO:teuthology.orchestra.run.smithi032.stdout: openssl x86_64 1:3.2.2-2.el9 baseos 1.4 M 2024-08-03T20:45:09.356 INFO:teuthology.orchestra.run.smithi032.stdout: openssl-libs x86_64 1:3.2.2-2.el9 baseos 2.4 M 2024-08-03T20:45:09.356 INFO:teuthology.orchestra.run.smithi032.stdout: xfsprogs x86_64 6.4.0-3.el9 baseos 1.1 M 2024-08-03T20:45:09.356 INFO:teuthology.orchestra.run.smithi032.stdout:Installing dependencies: 2024-08-03T20:45:09.357 INFO:teuthology.orchestra.run.smithi032.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-08-03T20:45:09.357 INFO:teuthology.orchestra.run.smithi032.stdout: automake noarch 1.16.2-8.el9 appstream 689 k 2024-08-03T20:45:09.357 INFO:teuthology.orchestra.run.smithi032.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-08-03T20:45:09.357 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-common x86_64 2:19.0.0-5476.g3792a0a4.el9 ceph 20 M 2024-08-03T20:45:09.357 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-grafana-dashboards noarch 2:19.0.0-5476.g3792a0a4.el9 ceph-noarch 34 k 2024-08-03T20:45:09.357 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-mds x86_64 2:19.0.0-5476.g3792a0a4.el9 ceph 2.3 M 2024-08-03T20:45:09.357 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-mgr-modules-core noarch 2:19.0.0-5476.g3792a0a4.el9 ceph-noarch 284 k 2024-08-03T20:45:09.357 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-mon x86_64 2:19.0.0-5476.g3792a0a4.el9 ceph 4.9 M 2024-08-03T20:45:09.357 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-osd x86_64 2:19.0.0-5476.g3792a0a4.el9 ceph 18 M 2024-08-03T20:45:09.357 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-prometheus-alerts noarch 2:19.0.0-5476.g3792a0a4.el9 ceph-noarch 16 k 2024-08-03T20:45:09.357 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-selinux x86_64 2:19.0.0-5476.g3792a0a4.el9 ceph 25 k 2024-08-03T20:45:09.357 INFO:teuthology.orchestra.run.smithi032.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-08-03T20:45:09.357 INFO:teuthology.orchestra.run.smithi032.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-08-03T20:45:09.358 INFO:teuthology.orchestra.run.smithi032.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-08-03T20:45:09.358 INFO:teuthology.orchestra.run.smithi032.stdout: fuse x86_64 2.9.9-16.el9 baseos 80 k 2024-08-03T20:45:09.358 INFO:teuthology.orchestra.run.smithi032.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-08-03T20:45:09.358 INFO:teuthology.orchestra.run.smithi032.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-08-03T20:45:09.358 INFO:teuthology.orchestra.run.smithi032.stdout: libarrow x86_64 9.0.0-11.el9 epel 4.4 M 2024-08-03T20:45:09.358 INFO:teuthology.orchestra.run.smithi032.stdout: libarrow-doc noarch 9.0.0-11.el9 epel 26 k 2024-08-03T20:45:09.358 INFO:teuthology.orchestra.run.smithi032.stdout: libcephsqlite x86_64 2:19.0.0-5476.g3792a0a4.el9 ceph 158 k 2024-08-03T20:45:09.358 INFO:teuthology.orchestra.run.smithi032.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-08-03T20:45:09.358 INFO:teuthology.orchestra.run.smithi032.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-08-03T20:45:09.358 INFO:teuthology.orchestra.run.smithi032.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-08-03T20:45:09.359 INFO:teuthology.orchestra.run.smithi032.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-08-03T20:45:09.359 INFO:teuthology.orchestra.run.smithi032.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-08-03T20:45:09.359 INFO:teuthology.orchestra.run.smithi032.stdout: libradosstriper1 x86_64 2:19.0.0-5476.g3792a0a4.el9 ceph 504 k 2024-08-03T20:45:09.359 INFO:teuthology.orchestra.run.smithi032.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-08-03T20:45:09.359 INFO:teuthology.orchestra.run.smithi032.stdout: librgw2 x86_64 2:19.0.0-5476.g3792a0a4.el9 ceph 5.3 M 2024-08-03T20:45:09.359 INFO:teuthology.orchestra.run.smithi032.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-08-03T20:45:09.359 INFO:teuthology.orchestra.run.smithi032.stdout: libzstd-devel x86_64 1.5.1-2.el9 appstream 47 k 2024-08-03T20:45:09.359 INFO:teuthology.orchestra.run.smithi032.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-08-03T20:45:09.359 INFO:teuthology.orchestra.run.smithi032.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-08-03T20:45:09.359 INFO:teuthology.orchestra.run.smithi032.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-08-03T20:45:09.362 INFO:teuthology.orchestra.run.smithi032.stdout: luarocks noarch 3.9.2-5.el9 epel 151 k 2024-08-03T20:45:09.362 INFO:teuthology.orchestra.run.smithi032.stdout: openblas x86_64 0.3.21-2.el9 appstream 35 k 2024-08-03T20:45:09.362 INFO:teuthology.orchestra.run.smithi032.stdout: openblas-openmp x86_64 0.3.21-2.el9 appstream 4.7 M 2024-08-03T20:45:09.362 INFO:teuthology.orchestra.run.smithi032.stdout: parquet-libs x86_64 9.0.0-11.el9 epel 839 k 2024-08-03T20:45:09.362 INFO:teuthology.orchestra.run.smithi032.stdout: postgresql-private-libs x86_64 13.14-1.el9 appstream 133 k 2024-08-03T20:45:09.362 INFO:teuthology.orchestra.run.smithi032.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-08-03T20:45:09.362 INFO:teuthology.orchestra.run.smithi032.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-08-03T20:45:09.363 INFO:teuthology.orchestra.run.smithi032.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-08-03T20:45:09.363 INFO:teuthology.orchestra.run.smithi032.stdout: python3-attrs noarch 20.3.0-7.el9 appstream 87 k 2024-08-03T20:45:09.363 INFO:teuthology.orchestra.run.smithi032.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-08-03T20:45:09.363 INFO:teuthology.orchestra.run.smithi032.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-08-03T20:45:09.363 INFO:teuthology.orchestra.run.smithi032.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-08-03T20:45:09.363 INFO:teuthology.orchestra.run.smithi032.stdout: python3-ceph-argparse x86_64 2:19.0.0-5476.g3792a0a4.el9 ceph 45 k 2024-08-03T20:45:09.363 INFO:teuthology.orchestra.run.smithi032.stdout: python3-ceph-common x86_64 2:19.0.0-5476.g3792a0a4.el9 ceph 138 k 2024-08-03T20:45:09.363 INFO:teuthology.orchestra.run.smithi032.stdout: python3-certifi noarch 2023.05.07-4.el9 epel 14 k 2024-08-03T20:45:09.363 INFO:teuthology.orchestra.run.smithi032.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-08-03T20:45:09.363 INFO:teuthology.orchestra.run.smithi032.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-08-03T20:45:09.363 INFO:teuthology.orchestra.run.smithi032.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-08-03T20:45:09.363 INFO:teuthology.orchestra.run.smithi032.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-08-03T20:45:09.363 INFO:teuthology.orchestra.run.smithi032.stdout: python3-colorama noarch 0.4.6-3.el9 appstream 61 k 2024-08-03T20:45:09.363 INFO:teuthology.orchestra.run.smithi032.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-08-03T20:45:09.364 INFO:teuthology.orchestra.run.smithi032.stdout: python3-devel x86_64 3.9.18-3.el9 appstream 244 k 2024-08-03T20:45:09.364 INFO:teuthology.orchestra.run.smithi032.stdout: python3-google-auth noarch 1:2.32.0-1.el9 epel 216 k 2024-08-03T20:45:09.364 INFO:teuthology.orchestra.run.smithi032.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-08-03T20:45:09.364 INFO:teuthology.orchestra.run.smithi032.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-08-03T20:45:09.364 INFO:teuthology.orchestra.run.smithi032.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-08-03T20:45:09.364 INFO:teuthology.orchestra.run.smithi032.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-08-03T20:45:09.364 INFO:teuthology.orchestra.run.smithi032.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-08-03T20:45:09.364 INFO:teuthology.orchestra.run.smithi032.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-08-03T20:45:09.364 INFO:teuthology.orchestra.run.smithi032.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-08-03T20:45:09.364 INFO:teuthology.orchestra.run.smithi032.stdout: python3-jinja2 noarch 2.11.3-6.el9 appstream 248 k 2024-08-03T20:45:09.364 INFO:teuthology.orchestra.run.smithi032.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-08-03T20:45:09.364 INFO:teuthology.orchestra.run.smithi032.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-08-03T20:45:09.365 INFO:teuthology.orchestra.run.smithi032.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-08-03T20:45:09.365 INFO:teuthology.orchestra.run.smithi032.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-08-03T20:45:09.365 INFO:teuthology.orchestra.run.smithi032.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-08-03T20:45:09.365 INFO:teuthology.orchestra.run.smithi032.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-08-03T20:45:09.365 INFO:teuthology.orchestra.run.smithi032.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-08-03T20:45:09.365 INFO:teuthology.orchestra.run.smithi032.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-08-03T20:45:09.365 INFO:teuthology.orchestra.run.smithi032.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-08-03T20:45:09.365 INFO:teuthology.orchestra.run.smithi032.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-08-03T20:45:09.365 INFO:teuthology.orchestra.run.smithi032.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-08-03T20:45:09.366 INFO:teuthology.orchestra.run.smithi032.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-08-03T20:45:09.366 INFO:teuthology.orchestra.run.smithi032.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-08-03T20:45:09.366 INFO:teuthology.orchestra.run.smithi032.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-08-03T20:45:09.366 INFO:teuthology.orchestra.run.smithi032.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-08-03T20:45:09.366 INFO:teuthology.orchestra.run.smithi032.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-08-03T20:45:09.366 INFO:teuthology.orchestra.run.smithi032.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-08-03T20:45:09.366 INFO:teuthology.orchestra.run.smithi032.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-08-03T20:45:09.366 INFO:teuthology.orchestra.run.smithi032.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-08-03T20:45:09.366 INFO:teuthology.orchestra.run.smithi032.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-08-03T20:45:09.366 INFO:teuthology.orchestra.run.smithi032.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-08-03T20:45:09.366 INFO:teuthology.orchestra.run.smithi032.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-08-03T20:45:09.366 INFO:teuthology.orchestra.run.smithi032.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-08-03T20:45:09.366 INFO:teuthology.orchestra.run.smithi032.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-08-03T20:45:09.367 INFO:teuthology.orchestra.run.smithi032.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-08-03T20:45:09.367 INFO:teuthology.orchestra.run.smithi032.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-08-03T20:45:09.367 INFO:teuthology.orchestra.run.smithi032.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-08-03T20:45:09.367 INFO:teuthology.orchestra.run.smithi032.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-08-03T20:45:09.367 INFO:teuthology.orchestra.run.smithi032.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-08-03T20:45:09.367 INFO:teuthology.orchestra.run.smithi032.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-08-03T20:45:09.367 INFO:teuthology.orchestra.run.smithi032.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-08-03T20:45:09.367 INFO:teuthology.orchestra.run.smithi032.stdout: python3-typing-extensions noarch 4.12.2-2.el9 epel 74 k 2024-08-03T20:45:09.367 INFO:teuthology.orchestra.run.smithi032.stdout: python3-urllib3 noarch 1.26.5-6.el9 baseos 215 k 2024-08-03T20:45:09.367 INFO:teuthology.orchestra.run.smithi032.stdout: python3-wcwidth noarch 0.2.5-8.el9 appstream 44 k 2024-08-03T20:45:09.367 INFO:teuthology.orchestra.run.smithi032.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-08-03T20:45:09.367 INFO:teuthology.orchestra.run.smithi032.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-08-03T20:45:09.368 INFO:teuthology.orchestra.run.smithi032.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-08-03T20:45:09.368 INFO:teuthology.orchestra.run.smithi032.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-08-03T20:45:09.368 INFO:teuthology.orchestra.run.smithi032.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-08-03T20:45:09.368 INFO:teuthology.orchestra.run.smithi032.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-08-03T20:45:09.368 INFO:teuthology.orchestra.run.smithi032.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-08-03T20:45:09.368 INFO:teuthology.orchestra.run.smithi032.stdout: socat x86_64 1.7.4.1-6.el9 appstream 304 k 2024-08-03T20:45:09.368 INFO:teuthology.orchestra.run.smithi032.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-08-03T20:45:09.368 INFO:teuthology.orchestra.run.smithi032.stdout: uuid x86_64 1.6.2-55.el9 appstream 58 k 2024-08-03T20:45:09.368 INFO:teuthology.orchestra.run.smithi032.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-08-03T20:45:09.368 INFO:teuthology.orchestra.run.smithi032.stdout: zlib-devel x86_64 1.2.11-41.el9 appstream 45 k 2024-08-03T20:45:09.368 INFO:teuthology.orchestra.run.smithi032.stdout:Installing weak dependencies: 2024-08-03T20:45:09.369 INFO:teuthology.orchestra.run.smithi032.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-08-03T20:45:09.369 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-03T20:45:09.369 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction Summary 2024-08-03T20:45:09.369 INFO:teuthology.orchestra.run.smithi032.stdout:======================================================================================= 2024-08-03T20:45:09.369 INFO:teuthology.orchestra.run.smithi032.stdout:Install 148 Packages 2024-08-03T20:45:09.369 INFO:teuthology.orchestra.run.smithi032.stdout:Upgrade 12 Packages 2024-08-03T20:45:09.369 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-03T20:45:09.370 INFO:teuthology.orchestra.run.smithi032.stdout:Total download size: 233 M 2024-08-03T20:45:09.370 INFO:teuthology.orchestra.run.smithi032.stdout:Downloading Packages: 2024-08-03T20:45:09.506 INFO:teuthology.orchestra.run.smithi118.stdout:(15/160): ceph-osd-19.0.0-5476.g3792a0a4.el9.x8 12 MB/s | 18 MB 00:01 2024-08-03T20:45:09.531 INFO:teuthology.orchestra.run.smithi118.stdout:(16/160): libcephsqlite-19.0.0-5476.g3792a0a4.e 611 kB/s | 158 kB 00:00 2024-08-03T20:45:09.556 INFO:teuthology.orchestra.run.smithi118.stdout:(17/160): librados-devel-19.0.0-5476.g3792a0a4. 2.5 MB/s | 127 kB 00:00 2024-08-03T20:45:09.590 INFO:teuthology.orchestra.run.smithi118.stdout:(18/160): libradosstriper1-19.0.0-5476.g3792a0a 8.4 MB/s | 504 kB 00:00 2024-08-03T20:45:09.657 INFO:teuthology.orchestra.run.smithi118.stdout:(19/160): python3-ceph-argparse-19.0.0-5476.g37 679 kB/s | 45 kB 00:00 2024-08-03T20:45:09.749 INFO:teuthology.orchestra.run.smithi118.stdout:(20/160): python3-ceph-common-19.0.0-5476.g3792 1.5 MB/s | 138 kB 00:00 2024-08-03T20:45:09.852 INFO:teuthology.orchestra.run.smithi118.stdout:(21/160): librgw2-19.0.0-5476.g3792a0a4.el9.x86 18 MB/s | 5.3 MB 00:00 2024-08-03T20:45:09.883 INFO:teuthology.orchestra.run.smithi118.stdout:(22/160): python3-cephfs-19.0.0-5476.g3792a0a4. 1.4 MB/s | 188 kB 00:00 2024-08-03T20:45:09.916 INFO:teuthology.orchestra.run.smithi118.stdout:(23/160): python3-rados-19.0.0-5476.g3792a0a4.e 5.7 MB/s | 371 kB 00:00 2024-08-03T20:45:09.941 INFO:teuthology.orchestra.run.smithi118.stdout:(24/160): python3-rbd-19.0.0-5476.g3792a0a4.el9 6.0 MB/s | 355 kB 00:00 2024-08-03T20:45:09.967 INFO:teuthology.orchestra.run.smithi118.stdout:(25/160): python3-rgw-19.0.0-5476.g3792a0a4.el9 2.2 MB/s | 116 kB 00:00 2024-08-03T20:45:09.992 INFO:teuthology.orchestra.run.smithi118.stdout:(26/160): rbd-fuse-19.0.0-5476.g3792a0a4.el9.x8 1.7 MB/s | 86 kB 00:00 2024-08-03T20:45:10.051 INFO:teuthology.orchestra.run.smithi118.stdout:(27/160): rbd-nbd-19.0.0-5476.g3792a0a4.el9.x86 2.9 MB/s | 170 kB 00:00 2024-08-03T20:45:10.118 INFO:teuthology.orchestra.run.smithi118.stdout:(28/160): rbd-mirror-19.0.0-5476.g3792a0a4.el9. 20 MB/s | 3.1 MB 00:00 2024-08-03T20:45:10.143 INFO:teuthology.orchestra.run.smithi118.stdout:(29/160): ceph-grafana-dashboards-19.0.0-5476.g 374 kB/s | 34 kB 00:00 2024-08-03T20:45:10.168 INFO:teuthology.orchestra.run.smithi118.stdout:(30/160): ceph-mgr-cephadm-19.0.0-5476.g3792a0a 2.9 MB/s | 150 kB 00:00 2024-08-03T20:45:10.581 INFO:teuthology.orchestra.run.smithi032.stdout:(1/160): ceph-19.0.0-5476.g3792a0a4.el9.x86_64. 19 kB/s | 6.6 kB 00:00 2024-08-03T20:45:10.644 INFO:teuthology.orchestra.run.smithi118.stdout:(31/160): ceph-mgr-diskprediction-local-19.0.0- 16 MB/s | 7.4 MB 00:00 2024-08-03T20:45:10.798 INFO:teuthology.orchestra.run.smithi032.stdout:(2/160): ceph-fuse-19.0.0-5476.g3792a0a4.el9.x8 3.9 MB/s | 872 kB 00:00 2024-08-03T20:45:10.849 INFO:teuthology.orchestra.run.smithi032.stdout:(3/160): ceph-immutable-object-cache-19.0.0-547 2.9 MB/s | 148 kB 00:00 2024-08-03T20:45:11.069 INFO:teuthology.orchestra.run.smithi118.stdout:(32/160): ceph-test-19.0.0-5476.g3792a0a4.el9.x 18 MB/s | 44 MB 00:02 2024-08-03T20:45:11.094 INFO:teuthology.orchestra.run.smithi118.stdout:(33/160): ceph-mgr-modules-core-19.0.0-5476.g37 632 kB/s | 284 kB 00:00 2024-08-03T20:45:11.116 INFO:teuthology.orchestra.run.smithi032.stdout:(4/160): ceph-mds-19.0.0-5476.g3792a0a4.el9.x86 8.5 MB/s | 2.3 MB 00:00 2024-08-03T20:45:11.119 INFO:teuthology.orchestra.run.smithi118.stdout:(34/160): ceph-mgr-rook-19.0.0-5476.g3792a0a4.e 991 kB/s | 49 kB 00:00 2024-08-03T20:45:11.145 INFO:teuthology.orchestra.run.smithi118.stdout:(35/160): ceph-prometheus-alerts-19.0.0-5476.g3 329 kB/s | 16 kB 00:00 2024-08-03T20:45:11.179 INFO:teuthology.orchestra.run.smithi118.stdout:(36/160): ceph-volume-19.0.0-5476.g3792a0a4.el9 4.6 MB/s | 276 kB 00:00 2024-08-03T20:45:11.254 INFO:teuthology.orchestra.run.smithi118.stdout:(37/160): cephfs-top-19.0.0-5476.g3792a0a4.el9. 244 kB/s | 18 kB 00:00 2024-08-03T20:45:11.299 INFO:teuthology.orchestra.run.smithi032.stdout:(5/160): ceph-base-19.0.0-5476.g3792a0a4.el9.x8 5.1 MB/s | 5.5 MB 00:01 2024-08-03T20:45:11.366 INFO:teuthology.orchestra.run.smithi032.stdout:(6/160): ceph-mgr-19.0.0-5476.g3792a0a4.el9.x86 6.1 MB/s | 1.5 MB 00:00 2024-08-03T20:45:11.414 INFO:teuthology.orchestra.run.smithi118.stdout:(38/160): ceph-mgr-dashboard-19.0.0-5476.g3792a 11 MB/s | 14 MB 00:01 2024-08-03T20:45:11.446 INFO:teuthology.orchestra.run.smithi118.stdout:(39/160): cephadm-19.0.0-5476.g3792a0a4.el9.noa 2.5 MB/s | 771 kB 00:00 2024-08-03T20:45:11.496 INFO:teuthology.orchestra.run.smithi118.stdout:(40/160): fuse-2.9.9-16.el9.x86_64.rpm 330 kB/s | 80 kB 00:00 2024-08-03T20:45:11.572 INFO:teuthology.orchestra.run.smithi118.stdout:(41/160): python3-cffi-1.14.5-5.el9.x86_64.rpm 3.3 MB/s | 253 kB 00:00 2024-08-03T20:45:11.597 INFO:teuthology.orchestra.run.smithi118.stdout:(42/160): libquadmath-11.4.1-3.el9.x86_64.rpm 1.3 MB/s | 195 kB 00:00 2024-08-03T20:45:11.600 INFO:teuthology.orchestra.run.smithi032.stdout:(7/160): ceph-common-19.0.0-5476.g3792a0a4.el9. 14 MB/s | 20 MB 00:01 2024-08-03T20:45:11.639 INFO:teuthology.orchestra.run.smithi118.stdout:(43/160): libgfortran-11.4.1-3.el9.x86_64.rpm 3.5 MB/s | 803 kB 00:00 2024-08-03T20:45:11.664 INFO:teuthology.orchestra.run.smithi118.stdout:(44/160): python3-ply-3.11-14.el9.noarch.rpm 1.6 MB/s | 106 kB 00:00 2024-08-03T20:45:11.690 INFO:teuthology.orchestra.run.smithi118.stdout:(45/160): python3-pycparser-2.20-6.el9.noarch.r 2.6 MB/s | 135 kB 00:00 2024-08-03T20:45:11.715 INFO:teuthology.orchestra.run.smithi118.stdout:(46/160): python3-requests-2.25.1-8.el9.noarch. 2.4 MB/s | 125 kB 00:00 2024-08-03T20:45:11.740 INFO:teuthology.orchestra.run.smithi118.stdout:(47/160): python3-urllib3-1.26.5-6.el9.noarch.r 4.2 MB/s | 215 kB 00:00 2024-08-03T20:45:11.774 INFO:teuthology.orchestra.run.smithi118.stdout:(48/160): python3-cryptography-36.0.1-4.el9.x86 6.1 MB/s | 1.2 MB 00:00 2024-08-03T20:45:11.800 INFO:teuthology.orchestra.run.smithi032.stdout:(8/160): ceph-mon-19.0.0-5476.g3792a0a4.el9.x86 9.8 MB/s | 4.9 MB 00:00 2024-08-03T20:45:11.867 INFO:teuthology.orchestra.run.smithi032.stdout:(9/160): ceph-selinux-19.0.0-5476.g3792a0a4.el9 380 kB/s | 25 kB 00:00 2024-08-03T20:45:11.915 INFO:teuthology.orchestra.run.smithi118.stdout:(49/160): boost-program-options-1.75.0-8.el9.x8 757 kB/s | 107 kB 00:00 2024-08-03T20:45:11.941 INFO:teuthology.orchestra.run.smithi118.stdout:(50/160): automake-1.16.2-8.el9.noarch.rpm 3.0 MB/s | 689 kB 00:00 2024-08-03T20:45:11.966 INFO:teuthology.orchestra.run.smithi118.stdout:(51/160): elfutils-libelf-devel-0.191-4.el9.x86 433 kB/s | 22 kB 00:00 2024-08-03T20:45:11.991 INFO:teuthology.orchestra.run.smithi118.stdout:(52/160): flex-2.6.4-9.el9.x86_64.rpm 6.1 MB/s | 313 kB 00:00 2024-08-03T20:45:12.050 INFO:teuthology.orchestra.run.smithi118.stdout:(53/160): flexiblas-3.0.4-8.el9.x86_64.rpm 391 kB/s | 33 kB 00:00 2024-08-03T20:45:12.092 INFO:teuthology.orchestra.run.smithi118.stdout:(54/160): bison-3.7.4-5.el9.x86_64.rpm 2.6 MB/s | 943 kB 00:00 2024-08-03T20:45:12.117 INFO:teuthology.orchestra.run.smithi118.stdout:(55/160): flexiblas-openblas-openmp-3.0.4-8.el9 272 kB/s | 18 kB 00:00 2024-08-03T20:45:12.118 INFO:teuthology.orchestra.run.smithi032.stdout:(10/160): ceph-radosgw-19.0.0-5476.g3792a0a4.el 20 MB/s | 10 MB 00:00 2024-08-03T20:45:12.143 INFO:teuthology.orchestra.run.smithi118.stdout:(56/160): libaio-devel-0.3.111-13.el9.x86_64.rp 224 kB/s | 11 kB 00:00 2024-08-03T20:45:12.168 INFO:teuthology.orchestra.run.smithi118.stdout:(57/160): libpmemobj-1.12.1-1.el9.x86_64.rpm 3.1 MB/s | 160 kB 00:00 2024-08-03T20:45:12.185 INFO:teuthology.orchestra.run.smithi032.stdout:(11/160): cephfs-mirror-19.0.0-5476.g3792a0a4.e 3.3 MB/s | 225 kB 00:00 2024-08-03T20:45:12.193 INFO:teuthology.orchestra.run.smithi118.stdout:(58/160): librabbitmq-0.11.0-7.el9.x86_64.rpm 901 kB/s | 45 kB 00:00 2024-08-03T20:45:12.235 INFO:teuthology.orchestra.run.smithi032.stdout:(12/160): cephfs-shell-19.0.0-5476.g3792a0a4.el 541 kB/s | 27 kB 00:00 2024-08-03T20:45:12.236 INFO:teuthology.orchestra.run.smithi118.stdout:(59/160): librdkafka-1.6.1-102.el9.x86_64.rpm 9.7 MB/s | 662 kB 00:00 2024-08-03T20:45:12.293 INFO:teuthology.orchestra.run.smithi032.stdout:(13/160): libcephfs-devel-19.0.0-5476.g3792a0a4 548 kB/s | 32 kB 00:00 2024-08-03T20:45:12.294 INFO:teuthology.orchestra.run.smithi118.stdout:(60/160): flexiblas-netlib-3.0.4-8.el9.x86_64.r 9.9 MB/s | 3.0 MB 00:00 2024-08-03T20:45:12.319 INFO:teuthology.orchestra.run.smithi118.stdout:(61/160): libuuid-devel-2.37.4-18.el9.x86_64.rp 305 kB/s | 25 kB 00:00 2024-08-03T20:45:12.345 INFO:teuthology.orchestra.run.smithi118.stdout:(62/160): libzstd-devel-1.5.1-2.el9.x86_64.rpm 944 kB/s | 47 kB 00:00 2024-08-03T20:45:12.370 INFO:teuthology.orchestra.run.smithi118.stdout:(63/160): lttng-ust-2.12.0-6.el9.x86_64.rpm 5.7 MB/s | 292 kB 00:00 2024-08-03T20:45:12.386 INFO:teuthology.orchestra.run.smithi032.stdout:(14/160): libcephfs2-19.0.0-5476.g3792a0a4.el9. 8.2 MB/s | 767 kB 00:00 2024-08-03T20:45:12.403 INFO:teuthology.orchestra.run.smithi118.stdout:(64/160): libtool-2.4.6-46.el9.x86_64.rpm 2.7 MB/s | 581 kB 00:00 2024-08-03T20:45:12.429 INFO:teuthology.orchestra.run.smithi118.stdout:(65/160): openblas-0.3.21-2.el9.x86_64.rpm 603 kB/s | 35 kB 00:00 2024-08-03T20:45:12.454 INFO:teuthology.orchestra.run.smithi118.stdout:(66/160): lua-5.4.4-4.el9.x86_64.rpm 1.7 MB/s | 188 kB 00:00 2024-08-03T20:45:12.588 INFO:teuthology.orchestra.run.smithi118.stdout:(67/160): openssl-devel-3.2.2-2.el9.x86_64.rpm 28 MB/s | 4.4 MB 00:00 2024-08-03T20:45:12.619 INFO:teuthology.orchestra.run.smithi032.stdout:(15/160): ceph-osd-19.0.0-5476.g3792a0a4.el9.x8 14 MB/s | 18 MB 00:01 2024-08-03T20:45:12.629 INFO:teuthology.orchestra.run.smithi118.stdout:(68/160): postgresql-13.14-1.el9.x86_64.rpm 9.1 MB/s | 1.6 MB 00:00 2024-08-03T20:45:12.644 INFO:teuthology.orchestra.run.smithi032.stdout:(16/160): libcephsqlite-19.0.0-5476.g3792a0a4.e 611 kB/s | 158 kB 00:00 2024-08-03T20:45:12.670 INFO:teuthology.orchestra.run.smithi032.stdout:(17/160): librados-devel-19.0.0-5476.g3792a0a4. 2.5 MB/s | 127 kB 00:00 2024-08-03T20:45:12.705 INFO:teuthology.orchestra.run.smithi118.stdout:(69/160): openblas-openmp-0.3.21-2.el9.x86_64.r 16 MB/s | 4.7 MB 00:00 2024-08-03T20:45:12.721 INFO:teuthology.orchestra.run.smithi032.stdout:(18/160): libradosstriper1-19.0.0-5476.g3792a0a 6.5 MB/s | 504 kB 00:00 2024-08-03T20:45:12.730 INFO:teuthology.orchestra.run.smithi118.stdout:(70/160): postgresql-private-libs-13.14-1.el9.x 1.3 MB/s | 133 kB 00:00 2024-08-03T20:45:12.772 INFO:teuthology.orchestra.run.smithi118.stdout:(71/160): postgresql-contrib-13.14-1.el9.x86_64 4.8 MB/s | 911 kB 00:00 2024-08-03T20:45:12.814 INFO:teuthology.orchestra.run.smithi118.stdout:(72/160): protobuf-3.14.0-13.el9.x86_64.rpm 12 MB/s | 1.0 MB 00:00 2024-08-03T20:45:12.820 INFO:teuthology.orchestra.run.smithi032.stdout:(19/160): python3-ceph-argparse-19.0.0-5476.g37 452 kB/s | 45 kB 00:00 2024-08-03T20:45:12.842 INFO:teuthology.orchestra.run.smithi118.stdout:(73/160): python3-attrs-20.3.0-7.el9.noarch.rpm 1.3 MB/s | 87 kB 00:00 2024-08-03T20:45:12.881 INFO:teuthology.orchestra.run.smithi118.stdout:(74/160): python3-colorama-0.4.6-3.el9.noarch.r 1.5 MB/s | 61 kB 00:00 2024-08-03T20:45:12.912 INFO:teuthology.orchestra.run.smithi032.stdout:(20/160): python3-ceph-common-19.0.0-5476.g3792 1.5 MB/s | 138 kB 00:00 2024-08-03T20:45:12.965 INFO:teuthology.orchestra.run.smithi118.stdout:(75/160): postgresql-server-13.14-1.el9.x86_64. 22 MB/s | 5.8 MB 00:00 2024-08-03T20:45:12.990 INFO:teuthology.orchestra.run.smithi118.stdout:(76/160): python3-devel-3.9.18-3.el9.x86_64.rpm 2.2 MB/s | 244 kB 00:00 2024-08-03T20:45:12.996 INFO:teuthology.orchestra.run.smithi032.stdout:(21/160): python3-cephfs-19.0.0-5476.g3792a0a4. 2.2 MB/s | 188 kB 00:00 2024-08-03T20:45:13.046 INFO:teuthology.orchestra.run.smithi118.stdout:(77/160): python3-jinja2-2.11.3-6.el9.noarch.rp 3.1 MB/s | 248 kB 00:00 2024-08-03T20:45:13.071 INFO:teuthology.orchestra.run.smithi032.stdout:(22/160): librgw2-19.0.0-5476.g3792a0a4.el9.x86 13 MB/s | 5.3 MB 00:00 2024-08-03T20:45:13.074 INFO:teuthology.orchestra.run.smithi118.stdout:(78/160): python3-jsonpatch-1.21-16.el9.noarch. 307 kB/s | 26 kB 00:00 2024-08-03T20:45:13.099 INFO:teuthology.orchestra.run.smithi118.stdout:(79/160): python3-jsonpointer-2.0-4.el9.noarch. 353 kB/s | 19 kB 00:00 2024-08-03T20:45:13.105 INFO:teuthology.orchestra.run.smithi032.stdout:(23/160): python3-rados-19.0.0-5476.g3792a0a4.e 3.3 MB/s | 371 kB 00:00 2024-08-03T20:45:13.124 INFO:teuthology.orchestra.run.smithi118.stdout:(80/160): python3-mako-1.1.4-6.el9.noarch.rpm 3.4 MB/s | 172 kB 00:00 2024-08-03T20:45:13.138 INFO:teuthology.orchestra.run.smithi032.stdout:(24/160): python3-rbd-19.0.0-5476.g3792a0a4.el9 5.2 MB/s | 355 kB 00:00 2024-08-03T20:45:13.150 INFO:teuthology.orchestra.run.smithi118.stdout:(81/160): python3-markupsafe-1.1.1-12.el9.x86_6 697 kB/s | 35 kB 00:00 2024-08-03T20:45:13.180 INFO:teuthology.orchestra.run.smithi032.stdout:(25/160): python3-rgw-19.0.0-5476.g3792a0a4.el9 1.5 MB/s | 116 kB 00:00 2024-08-03T20:45:13.201 INFO:teuthology.orchestra.run.smithi118.stdout:(82/160): python3-numpy-f2py-1.20.1-5.el9.x86_6 4.9 MB/s | 250 kB 00:00 2024-08-03T20:45:13.206 INFO:teuthology.orchestra.run.smithi032.stdout:(26/160): rbd-fuse-19.0.0-5476.g3792a0a4.el9.x8 1.3 MB/s | 86 kB 00:00 2024-08-03T20:45:13.281 INFO:teuthology.orchestra.run.smithi032.stdout:(27/160): rbd-nbd-19.0.0-5476.g3792a0a4.el9.x86 2.2 MB/s | 170 kB 00:00 2024-08-03T20:45:13.292 INFO:teuthology.orchestra.run.smithi118.stdout:(83/160): python3-babel-2.9.1-2.el9.noarch.rpm 12 MB/s | 6.0 MB 00:00 2024-08-03T20:45:13.318 INFO:teuthology.orchestra.run.smithi118.stdout:(84/160): python3-oauthlib-3.1.1-5.el9.noarch.r 1.9 MB/s | 222 kB 00:00 2024-08-03T20:45:13.343 INFO:teuthology.orchestra.run.smithi118.stdout:(85/160): python3-packaging-20.9-5.el9.noarch.r 1.5 MB/s | 77 kB 00:00 2024-08-03T20:45:13.368 INFO:teuthology.orchestra.run.smithi118.stdout:(86/160): python3-prettytable-0.7.2-27.el9.noar 843 kB/s | 42 kB 00:00 2024-08-03T20:45:13.403 INFO:teuthology.orchestra.run.smithi118.stdout:(87/160): python3-protobuf-3.14.0-13.el9.noarch 4.5 MB/s | 269 kB 00:00 2024-08-03T20:45:13.406 INFO:teuthology.orchestra.run.smithi032.stdout:(28/160): rbd-mirror-19.0.0-5476.g3792a0a4.el9. 14 MB/s | 3.1 MB 00:00 2024-08-03T20:45:13.427 INFO:teuthology.orchestra.run.smithi118.stdout:(88/160): python3-pyasn1-0.4.8-6.el9.noarch.rpm 2.7 MB/s | 159 kB 00:00 2024-08-03T20:45:13.432 INFO:teuthology.orchestra.run.smithi032.stdout:(29/160): ceph-grafana-dashboards-19.0.0-5476.g 229 kB/s | 34 kB 00:00 2024-08-03T20:45:13.461 INFO:teuthology.orchestra.run.smithi118.stdout:(89/160): python3-pyasn1-modules-0.4.8-6.el9.no 4.7 MB/s | 279 kB 00:00 2024-08-03T20:45:13.486 INFO:teuthology.orchestra.run.smithi118.stdout:(90/160): python3-pytz-2021.1-5.el9.noarch.rpm 875 kB/s | 51 kB 00:00 2024-08-03T20:45:13.490 INFO:teuthology.orchestra.run.smithi032.stdout:(30/160): ceph-mgr-cephadm-19.0.0-5476.g3792a0a 1.8 MB/s | 150 kB 00:00 2024-08-03T20:45:13.531 INFO:teuthology.orchestra.run.smithi118.stdout:(91/160): python3-requests-oauthlib-1.3.0-12.el 766 kB/s | 54 kB 00:00 2024-08-03T20:45:13.612 INFO:teuthology.orchestra.run.smithi118.stdout:(92/160): python3-numpy-1.20.1-5.el9.x86_64.rpm 10 MB/s | 5.1 MB 00:00 2024-08-03T20:45:13.637 INFO:teuthology.orchestra.run.smithi118.stdout:(93/160): python3-toml-0.10.2-6.el9.noarch.rpm 395 kB/s | 42 kB 00:00 2024-08-03T20:45:13.662 INFO:teuthology.orchestra.run.smithi118.stdout:(94/160): python3-wcwidth-0.2.5-8.el9.noarch.rp 888 kB/s | 44 kB 00:00 2024-08-03T20:45:13.687 INFO:teuthology.orchestra.run.smithi118.stdout:(95/160): qatlib-24.02.0-1.el9.x86_64.rpm 4.3 MB/s | 221 kB 00:00 2024-08-03T20:45:13.713 INFO:teuthology.orchestra.run.smithi118.stdout:(96/160): qatlib-service-24.02.0-1.el9.x86_64.r 716 kB/s | 36 kB 00:00 2024-08-03T20:45:13.738 INFO:teuthology.orchestra.run.smithi118.stdout:(97/160): qatzip-libs-1.2.0-1.el9.x86_64.rpm 948 kB/s | 47 kB 00:00 2024-08-03T20:45:13.763 INFO:teuthology.orchestra.run.smithi118.stdout:(98/160): socat-1.7.4.1-6.el9.x86_64.rpm 5.9 MB/s | 304 kB 00:00 2024-08-03T20:45:13.789 INFO:teuthology.orchestra.run.smithi118.stdout:(99/160): uuid-1.6.2-55.el9.x86_64.rpm 1.1 MB/s | 58 kB 00:00 2024-08-03T20:45:13.814 INFO:teuthology.orchestra.run.smithi118.stdout:(100/160): xfsprogs-devel-6.4.0-3.el9.x86_64.rp 1.9 MB/s | 96 kB 00:00 2024-08-03T20:45:13.839 INFO:teuthology.orchestra.run.smithi118.stdout:(101/160): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.2 MB/s | 64 kB 00:00 2024-08-03T20:45:13.865 INFO:teuthology.orchestra.run.smithi118.stdout:(102/160): zlib-devel-1.2.11-41.el9.x86_64.rpm 893 kB/s | 45 kB 00:00 2024-08-03T20:45:13.898 INFO:teuthology.orchestra.run.smithi118.stdout:(103/160): lua-devel-5.4.4-4.el9.x86_64.rpm 382 kB/s | 22 kB 00:00 2024-08-03T20:45:13.998 INFO:teuthology.orchestra.run.smithi118.stdout:(104/160): abseil-cpp-20211102.0-3.el9.x86_64.r 5.4 MB/s | 548 kB 00:00 2024-08-03T20:45:14.057 INFO:teuthology.orchestra.run.smithi118.stdout:(105/160): protobuf-compiler-3.14.0-13.el9.x86_ 4.4 MB/s | 863 kB 00:00 2024-08-03T20:45:14.116 INFO:teuthology.orchestra.run.smithi032.stdout:(31/160): ceph-mgr-diskprediction-local-19.0.0- 12 MB/s | 7.4 MB 00:00 2024-08-03T20:45:14.116 INFO:teuthology.orchestra.run.smithi118.stdout:(106/160): gperftools-libs-2.9.1-2.el9.x86_64.r 2.6 MB/s | 309 kB 00:00 2024-08-03T20:45:14.149 INFO:teuthology.orchestra.run.smithi118.stdout:(107/160): grpc-data-1.46.7-10.el9.noarch.rpm 212 kB/s | 19 kB 00:00 2024-08-03T20:45:14.175 INFO:teuthology.orchestra.run.smithi118.stdout:(108/160): libarrow-doc-9.0.0-11.el9.noarch.rpm 1.0 MB/s | 26 kB 00:00 2024-08-03T20:45:14.208 INFO:teuthology.orchestra.run.smithi118.stdout:(109/160): liboath-2.6.7-2.el9.x86_64.rpm 1.4 MB/s | 49 kB 00:00 2024-08-03T20:45:14.216 INFO:teuthology.orchestra.run.smithi032.stdout:(32/160): ceph-mgr-modules-core-19.0.0-5476.g37 2.8 MB/s | 284 kB 00:00 2024-08-03T20:45:14.234 INFO:teuthology.orchestra.run.smithi118.stdout:(110/160): libunwind-1.6.2-1.el9.x86_64.rpm 2.6 MB/s | 67 kB 00:00 2024-08-03T20:45:14.259 INFO:teuthology.orchestra.run.smithi118.stdout:(111/160): luarocks-3.9.2-5.el9.noarch.rpm 6.0 MB/s | 151 kB 00:00 2024-08-03T20:45:14.283 INFO:teuthology.orchestra.run.smithi032.stdout:(33/160): ceph-mgr-rook-19.0.0-5476.g3792a0a4.e 738 kB/s | 49 kB 00:00 2024-08-03T20:45:14.309 INFO:teuthology.orchestra.run.smithi118.stdout:(112/160): parquet-libs-9.0.0-11.el9.x86_64.rpm 16 MB/s | 839 kB 00:00 2024-08-03T20:45:14.351 INFO:teuthology.orchestra.run.smithi118.stdout:(113/160): python3-asyncssh-2.13.2-5.el9.noarch 13 MB/s | 548 kB 00:00 2024-08-03T20:45:14.377 INFO:teuthology.orchestra.run.smithi118.stdout:(114/160): python3-bcrypt-3.2.2-1.el9.x86_64.rp 1.7 MB/s | 43 kB 00:00 2024-08-03T20:45:14.402 INFO:teuthology.orchestra.run.smithi118.stdout:(115/160): python3-cachetools-4.2.4-1.el9.noarc 1.3 MB/s | 32 kB 00:00 2024-08-03T20:45:14.427 INFO:teuthology.orchestra.run.smithi118.stdout:(116/160): python3-certifi-2023.05.07-4.el9.noa 563 kB/s | 14 kB 00:00 2024-08-03T20:45:14.461 INFO:teuthology.orchestra.run.smithi118.stdout:(117/160): python3-cheroot-8.6.0-4.el9.noarch.r 5.1 MB/s | 172 kB 00:00 2024-08-03T20:45:14.492 INFO:teuthology.orchestra.run.smithi032.stdout:(34/160): ceph-mgr-dashboard-19.0.0-5476.g3792a 13 MB/s | 14 MB 00:01 2024-08-03T20:45:14.503 INFO:teuthology.orchestra.run.smithi118.stdout:(118/160): python3-cherrypy-18.6.1-2.el9.noarch 8.5 MB/s | 358 kB 00:00 2024-08-03T20:45:14.517 INFO:teuthology.orchestra.run.smithi032.stdout:(35/160): ceph-prometheus-alerts-19.0.0-5476.g3 70 kB/s | 16 kB 00:00 2024-08-03T20:45:14.551 INFO:teuthology.orchestra.run.smithi032.stdout:(36/160): ceph-volume-19.0.0-5476.g3792a0a4.el9 4.6 MB/s | 276 kB 00:00 2024-08-03T20:45:14.561 INFO:teuthology.orchestra.run.smithi118.stdout:(119/160): python3-cmd2-2.4.2-2.el9.noarch.rpm 5.1 MB/s | 303 kB 00:00 2024-08-03T20:45:14.595 INFO:teuthology.orchestra.run.smithi118.stdout:(120/160): python3-google-auth-2.32.0-1.el9.noa 6.3 MB/s | 216 kB 00:00 2024-08-03T20:45:14.691 INFO:teuthology.orchestra.run.smithi118.stdout:(121/160): libarrow-9.0.0-11.el9.x86_64.rpm 7.8 MB/s | 4.4 MB 00:00 2024-08-03T20:45:14.712 INFO:teuthology.orchestra.run.smithi118.stdout:(122/160): python3-grpcio-tools-1.46.7-10.el9.x 5.7 MB/s | 144 kB 00:00 2024-08-03T20:45:14.738 INFO:teuthology.orchestra.run.smithi118.stdout:(123/160): python3-jaraco-8.2.1-3.el9.noarch.rp 434 kB/s | 11 kB 00:00 2024-08-03T20:45:14.938 INFO:teuthology.orchestra.run.smithi118.stdout:(124/160): python3-scipy-1.6.2-8.el9.x86_64.rpm 11 MB/s | 16 MB 00:01 2024-08-03T20:45:14.964 INFO:teuthology.orchestra.run.smithi118.stdout:(125/160): python3-jaraco-classes-3.2.1-5.el9.n 78 kB/s | 18 kB 00:00 2024-08-03T20:45:14.968 INFO:teuthology.orchestra.run.smithi032.stdout:(37/160): ceph-test-19.0.0-5476.g3792a0a4.el9.x 14 MB/s | 44 MB 00:03 2024-08-03T20:45:14.993 INFO:teuthology.orchestra.run.smithi032.stdout:(38/160): cephfs-top-19.0.0-5476.g3792a0a4.el9. 41 kB/s | 18 kB 00:00 2024-08-03T20:45:15.035 INFO:teuthology.orchestra.run.smithi032.stdout:(39/160): cephadm-19.0.0-5476.g3792a0a4.el9.noa 1.5 MB/s | 771 kB 00:00 2024-08-03T20:45:15.043 INFO:teuthology.orchestra.run.smithi118.stdout:(126/160): python3-jaraco-functools-3.5.0-2.el9 381 kB/s | 19 kB 00:00 2024-08-03T20:45:15.072 INFO:teuthology.orchestra.run.smithi118.stdout:(127/160): python3-jaraco-text-3.2.0-6.el9.noar 734 kB/s | 20 kB 00:00 2024-08-03T20:45:15.098 INFO:teuthology.orchestra.run.smithi118.stdout:(128/160): python3-jaraco-collections-3.0.0-8.e 146 kB/s | 23 kB 00:00 2024-08-03T20:45:15.131 INFO:teuthology.orchestra.run.smithi118.stdout:(129/160): python3-logutils-0.3.5-21.el9.noarch 1.4 MB/s | 46 kB 00:00 2024-08-03T20:45:15.144 INFO:teuthology.orchestra.run.smithi032.stdout:(40/160): fuse-2.9.9-16.el9.x86_64.rpm 456 kB/s | 80 kB 00:00 2024-08-03T20:45:15.165 INFO:teuthology.orchestra.run.smithi118.stdout:(130/160): python3-more-itertools-8.12.0-2.el9. 2.3 MB/s | 79 kB 00:00 2024-08-03T20:45:15.190 INFO:teuthology.orchestra.run.smithi118.stdout:(131/160): python3-natsort-7.1.1-5.el9.noarch.r 2.3 MB/s | 58 kB 00:00 2024-08-03T20:45:15.202 INFO:teuthology.orchestra.run.smithi032.stdout:(41/160): libquadmath-11.4.1-3.el9.x86_64.rpm 1.1 MB/s | 195 kB 00:00 2024-08-03T20:45:15.224 INFO:teuthology.orchestra.run.smithi118.stdout:(132/160): python3-kubernetes-26.1.0-2.el9.noar 6.8 MB/s | 1.0 MB 00:00 2024-08-03T20:45:15.244 INFO:teuthology.orchestra.run.smithi032.stdout:(42/160): libgfortran-11.4.1-3.el9.x86_64.rpm 3.1 MB/s | 803 kB 00:00 2024-08-03T20:45:15.266 INFO:teuthology.orchestra.run.smithi118.stdout:(133/160): python3-grpcio-1.46.7-10.el9.x86_64. 3.0 MB/s | 2.0 MB 00:00 2024-08-03T20:45:15.269 INFO:teuthology.orchestra.run.smithi032.stdout:(43/160): python3-cffi-1.14.5-5.el9.x86_64.rpm 2.0 MB/s | 253 kB 00:00 2024-08-03T20:45:15.291 INFO:teuthology.orchestra.run.smithi118.stdout:(134/160): python3-portend-3.1.0-2.el9.noarch.r 246 kB/s | 16 kB 00:00 2024-08-03T20:45:15.295 INFO:teuthology.orchestra.run.smithi032.stdout:(44/160): python3-ply-3.11-14.el9.noarch.rpm 2.1 MB/s | 106 kB 00:00 2024-08-03T20:45:15.316 INFO:teuthology.orchestra.run.smithi118.stdout:(135/160): python3-pecan-1.4.2-3.el9.noarch.rpm 2.1 MB/s | 272 kB 00:00 2024-08-03T20:45:15.328 INFO:teuthology.orchestra.run.smithi032.stdout:(45/160): python3-pycparser-2.20-6.el9.noarch.r 2.3 MB/s | 135 kB 00:00 2024-08-03T20:45:15.341 INFO:teuthology.orchestra.run.smithi118.stdout:(136/160): python3-pyperclip-1.8.2-2.el9.noarch 562 kB/s | 28 kB 00:00 2024-08-03T20:45:15.354 INFO:teuthology.orchestra.run.smithi032.stdout:(46/160): python3-requests-2.25.1-8.el9.noarch. 2.1 MB/s | 125 kB 00:00 2024-08-03T20:45:15.367 INFO:teuthology.orchestra.run.smithi118.stdout:(137/160): python3-pyOpenSSL-21.0.0-1.el9.noarc 893 kB/s | 90 kB 00:00 2024-08-03T20:45:15.387 INFO:teuthology.orchestra.run.smithi032.stdout:(47/160): python3-cryptography-36.0.1-4.el9.x86 6.6 MB/s | 1.2 MB 00:00 2024-08-03T20:45:15.400 INFO:teuthology.orchestra.run.smithi118.stdout:(138/160): python3-repoze-lru-0.7-16.el9.noarch 367 kB/s | 31 kB 00:00 2024-08-03T20:45:15.413 INFO:teuthology.orchestra.run.smithi032.stdout:(48/160): python3-urllib3-1.26.5-6.el9.noarch.r 2.5 MB/s | 215 kB 00:00 2024-08-03T20:45:15.426 INFO:teuthology.orchestra.run.smithi118.stdout:(139/160): python3-rsa-4.9-2.el9.noarch.rpm 2.3 MB/s | 59 kB 00:00 2024-08-03T20:45:15.451 INFO:teuthology.orchestra.run.smithi118.stdout:(140/160): python3-tempora-5.0.0-2.el9.noarch.r 717 kB/s | 36 kB 00:00 2024-08-03T20:45:15.476 INFO:teuthology.orchestra.run.smithi118.stdout:(141/160): python3-typing-extensions-4.12.2-2.e 1.4 MB/s | 74 kB 00:00 2024-08-03T20:45:15.502 INFO:teuthology.orchestra.run.smithi118.stdout:(142/160): python3-routes-2.5.1-5.el9.noarch.rp 1.2 MB/s | 188 kB 00:00 2024-08-03T20:45:15.530 INFO:teuthology.orchestra.run.smithi032.stdout:(49/160): automake-1.16.2-8.el9.noarch.rpm 3.8 MB/s | 689 kB 00:00 2024-08-03T20:45:15.552 INFO:teuthology.orchestra.run.smithi118.stdout:(143/160): python3-websocket-client-1.2.3-2.el9 1.2 MB/s | 90 kB 00:00 2024-08-03T20:45:15.555 INFO:teuthology.orchestra.run.smithi032.stdout:(50/160): boost-program-options-1.75.0-8.el9.x8 753 kB/s | 107 kB 00:00 2024-08-03T20:45:15.577 INFO:teuthology.orchestra.run.smithi118.stdout:(144/160): python3-webob-1.8.7-6.el9.noarch.rpm 1.8 MB/s | 230 kB 00:00 2024-08-03T20:45:15.580 INFO:teuthology.orchestra.run.smithi032.stdout:(51/160): elfutils-libelf-devel-0.191-4.el9.x86 434 kB/s | 22 kB 00:00 2024-08-03T20:45:15.602 INFO:teuthology.orchestra.run.smithi118.stdout:(145/160): python3-zc-lockfile-2.0-10.el9.noarc 399 kB/s | 20 kB 00:00 2024-08-03T20:45:15.622 INFO:teuthology.orchestra.run.smithi032.stdout:(52/160): flex-2.6.4-9.el9.x86_64.rpm 4.6 MB/s | 313 kB 00:00 2024-08-03T20:45:15.628 INFO:teuthology.orchestra.run.smithi118.stdout:(146/160): re2-20211101-3.el9.x86_64.rpm 3.7 MB/s | 192 kB 00:00 2024-08-03T20:45:15.647 INFO:teuthology.orchestra.run.smithi032.stdout:(53/160): flexiblas-3.0.4-8.el9.x86_64.rpm 491 kB/s | 33 kB 00:00 2024-08-03T20:45:15.653 INFO:teuthology.orchestra.run.smithi118.stdout:(147/160): python3-werkzeug-2.0.3-3.el9.1.noarc 2.8 MB/s | 427 kB 00:00 2024-08-03T20:45:15.681 INFO:teuthology.orchestra.run.smithi032.stdout:(54/160): bison-3.7.4-5.el9.x86_64.rpm 3.1 MB/s | 943 kB 00:00 2024-08-03T20:45:15.706 INFO:teuthology.orchestra.run.smithi032.stdout:(55/160): flexiblas-openblas-openmp-3.0.4-8.el9 311 kB/s | 18 kB 00:00 2024-08-03T20:45:15.711 INFO:teuthology.orchestra.run.smithi118.stdout:(148/160): thrift-0.15.0-2.el9.x86_64.rpm 15 MB/s | 1.6 MB 00:00 2024-08-03T20:45:15.732 INFO:teuthology.orchestra.run.smithi032.stdout:(56/160): libaio-devel-0.3.111-13.el9.x86_64.rp 222 kB/s | 11 kB 00:00 2024-08-03T20:45:15.757 INFO:teuthology.orchestra.run.smithi032.stdout:(57/160): libpmemobj-1.12.1-1.el9.x86_64.rpm 3.1 MB/s | 160 kB 00:00 2024-08-03T20:45:15.782 INFO:teuthology.orchestra.run.smithi032.stdout:(58/160): librabbitmq-0.11.0-7.el9.x86_64.rpm 902 kB/s | 45 kB 00:00 2024-08-03T20:45:15.816 INFO:teuthology.orchestra.run.smithi032.stdout:(59/160): librdkafka-1.6.1-102.el9.x86_64.rpm 11 MB/s | 662 kB 00:00 2024-08-03T20:45:15.850 INFO:teuthology.orchestra.run.smithi032.stdout:(60/160): libtool-2.4.6-46.el9.x86_64.rpm 8.5 MB/s | 581 kB 00:00 2024-08-03T20:45:15.875 INFO:teuthology.orchestra.run.smithi032.stdout:(61/160): libuuid-devel-2.37.4-18.el9.x86_64.rp 433 kB/s | 25 kB 00:00 2024-08-03T20:45:15.901 INFO:teuthology.orchestra.run.smithi032.stdout:(62/160): libzstd-devel-1.5.1-2.el9.x86_64.rpm 944 kB/s | 47 kB 00:00 2024-08-03T20:45:15.912 INFO:teuthology.orchestra.run.smithi118.stdout:(149/160): NetworkManager-1.48.2-2.el9.x86_64.r 12 MB/s | 2.3 MB 00:00 2024-08-03T20:45:15.926 INFO:teuthology.orchestra.run.smithi032.stdout:(63/160): lttng-ust-2.12.0-6.el9.x86_64.rpm 5.7 MB/s | 292 kB 00:00 2024-08-03T20:45:15.951 INFO:teuthology.orchestra.run.smithi032.stdout:(64/160): lua-5.4.4-4.el9.x86_64.rpm 3.7 MB/s | 188 kB 00:00 2024-08-03T20:45:15.976 INFO:teuthology.orchestra.run.smithi032.stdout:(65/160): openblas-0.3.21-2.el9.x86_64.rpm 705 kB/s | 35 kB 00:00 2024-08-03T20:45:15.979 INFO:teuthology.orchestra.run.smithi118.stdout:(150/160): NetworkManager-libnm-1.48.2-2.el9.x8 27 MB/s | 1.8 MB 00:00 2024-08-03T20:45:16.035 INFO:teuthology.orchestra.run.smithi032.stdout:(66/160): flexiblas-netlib-3.0.4-8.el9.x86_64.r 7.2 MB/s | 3.0 MB 00:00 2024-08-03T20:45:16.041 INFO:teuthology.orchestra.run.smithi118.stdout:(151/160): NetworkManager-team-1.48.2-2.el9.x86 663 kB/s | 41 kB 00:00 2024-08-03T20:45:16.096 INFO:teuthology.orchestra.run.smithi118.stdout:(152/160): NetworkManager-tui-1.48.2-2.el9.x86_ 4.4 MB/s | 247 kB 00:00 2024-08-03T20:45:16.110 INFO:teuthology.orchestra.run.smithi032.stdout:(67/160): postgresql-13.14-1.el9.x86_64.rpm 21 MB/s | 1.6 MB 00:00 2024-08-03T20:45:16.155 INFO:teuthology.orchestra.run.smithi118.stdout:(153/160): librados2-19.0.0-5476.g3792a0a4.el9. 6.4 MB/s | 3.4 MB 00:00 2024-08-03T20:45:16.177 INFO:teuthology.orchestra.run.smithi032.stdout:(68/160): openblas-openmp-0.3.21-2.el9.x86_64.r 21 MB/s | 4.7 MB 00:00 2024-08-03T20:45:16.180 INFO:teuthology.orchestra.run.smithi118.stdout:(154/160): elfutils-debuginfod-client-0.191-4.e 440 kB/s | 37 kB 00:00 2024-08-03T20:45:16.244 INFO:teuthology.orchestra.run.smithi032.stdout:(69/160): openssl-devel-3.2.2-2.el9.x86_64.rpm 17 MB/s | 4.4 MB 00:00 2024-08-03T20:45:16.247 INFO:teuthology.orchestra.run.smithi118.stdout:(155/160): librbd1-19.0.0-5476.g3792a0a4.el9.x8 5.3 MB/s | 3.1 MB 00:00 2024-08-03T20:45:16.270 INFO:teuthology.orchestra.run.smithi032.stdout:(70/160): postgresql-private-libs-13.14-1.el9.x 1.4 MB/s | 133 kB 00:00 2024-08-03T20:45:16.272 INFO:teuthology.orchestra.run.smithi118.stdout:(156/160): elfutils-libelf-0.191-4.el9.x86_64.r 1.7 MB/s | 207 kB 00:00 2024-08-03T20:45:16.297 INFO:teuthology.orchestra.run.smithi118.stdout:(157/160): elfutils-libs-0.191-4.el9.x86_64.rpm 2.2 MB/s | 260 kB 00:00 2024-08-03T20:45:16.311 INFO:teuthology.orchestra.run.smithi032.stdout:(71/160): postgresql-contrib-13.14-1.el9.x86_64 4.4 MB/s | 911 kB 00:00 2024-08-03T20:45:16.356 INFO:teuthology.orchestra.run.smithi118.stdout:(158/160): xfsprogs-6.4.0-3.el9.x86_64.rpm 19 MB/s | 1.1 MB 00:00 2024-08-03T20:45:16.362 INFO:teuthology.orchestra.run.smithi032.stdout:(72/160): protobuf-3.14.0-13.el9.x86_64.rpm 11 MB/s | 1.0 MB 00:00 2024-08-03T20:45:16.387 INFO:teuthology.orchestra.run.smithi032.stdout:(73/160): python3-attrs-20.3.0-7.el9.noarch.rpm 1.1 MB/s | 87 kB 00:00 2024-08-03T20:45:16.429 INFO:teuthology.orchestra.run.smithi032.stdout:(74/160): python3-colorama-0.4.6-3.el9.noarch.r 1.4 MB/s | 61 kB 00:00 2024-08-03T20:45:16.431 INFO:teuthology.orchestra.run.smithi118.stdout:(159/160): openssl-libs-3.2.2-2.el9.x86_64.rpm 15 MB/s | 2.4 MB 00:00 2024-08-03T20:45:16.490 INFO:teuthology.orchestra.run.smithi118.stdout:(160/160): openssl-3.2.2-2.el9.x86_64.rpm 5.6 MB/s | 1.4 MB 00:00 2024-08-03T20:45:16.495 INFO:teuthology.orchestra.run.smithi118.stdout:-------------------------------------------------------------------------------- 2024-08-03T20:45:16.495 INFO:teuthology.orchestra.run.smithi118.stdout:Total 24 MB/s | 233 MB 00:09 2024-08-03T20:45:16.521 INFO:teuthology.orchestra.run.smithi032.stdout:(75/160): postgresql-server-13.14-1.el9.x86_64. 21 MB/s | 5.8 MB 00:00 2024-08-03T20:45:16.547 INFO:teuthology.orchestra.run.smithi032.stdout:(76/160): python3-devel-3.9.18-3.el9.x86_64.rpm 2.0 MB/s | 244 kB 00:00 2024-08-03T20:45:16.572 INFO:teuthology.orchestra.run.smithi032.stdout:(77/160): python3-jinja2-2.11.3-6.el9.noarch.rp 4.9 MB/s | 248 kB 00:00 2024-08-03T20:45:16.597 INFO:teuthology.orchestra.run.smithi032.stdout:(78/160): python3-jsonpatch-1.21-16.el9.noarch. 513 kB/s | 26 kB 00:00 2024-08-03T20:45:16.622 INFO:teuthology.orchestra.run.smithi032.stdout:(79/160): python3-jsonpointer-2.0-4.el9.noarch. 382 kB/s | 19 kB 00:00 2024-08-03T20:45:16.647 INFO:teuthology.orchestra.run.smithi032.stdout:(80/160): python3-mako-1.1.4-6.el9.noarch.rpm 3.4 MB/s | 172 kB 00:00 2024-08-03T20:45:16.692 INFO:teuthology.orchestra.run.smithi032.stdout:(81/160): python3-markupsafe-1.1.1-12.el9.x86_6 695 kB/s | 35 kB 00:00 2024-08-03T20:45:16.773 INFO:teuthology.orchestra.run.smithi032.stdout:(82/160): python3-babel-2.9.1-2.el9.noarch.rpm 15 MB/s | 6.0 MB 00:00 2024-08-03T20:45:16.798 INFO:teuthology.orchestra.run.smithi032.stdout:(83/160): python3-numpy-f2py-1.20.1-5.el9.x86_6 2.0 MB/s | 250 kB 00:00 2024-08-03T20:45:16.824 INFO:teuthology.orchestra.run.smithi032.stdout:(84/160): python3-oauthlib-3.1.1-5.el9.noarch.r 4.3 MB/s | 222 kB 00:00 2024-08-03T20:45:16.849 INFO:teuthology.orchestra.run.smithi032.stdout:(85/160): python3-packaging-20.9-5.el9.noarch.r 1.5 MB/s | 77 kB 00:00 2024-08-03T20:45:16.874 INFO:teuthology.orchestra.run.smithi032.stdout:(86/160): python3-prettytable-0.7.2-27.el9.noar 847 kB/s | 42 kB 00:00 2024-08-03T20:45:16.900 INFO:teuthology.orchestra.run.smithi032.stdout:(87/160): python3-protobuf-3.14.0-13.el9.noarch 5.2 MB/s | 269 kB 00:00 2024-08-03T20:45:16.925 INFO:teuthology.orchestra.run.smithi032.stdout:(88/160): python3-pyasn1-0.4.8-6.el9.noarch.rpm 3.1 MB/s | 159 kB 00:00 2024-08-03T20:45:16.951 INFO:teuthology.orchestra.run.smithi032.stdout:(89/160): python3-pyasn1-modules-0.4.8-6.el9.no 5.4 MB/s | 279 kB 00:00 2024-08-03T20:45:16.976 INFO:teuthology.orchestra.run.smithi032.stdout:(90/160): python3-pytz-2021.1-5.el9.noarch.rpm 1.0 MB/s | 51 kB 00:00 2024-08-03T20:45:17.001 INFO:teuthology.orchestra.run.smithi032.stdout:(91/160): python3-requests-oauthlib-1.3.0-12.el 1.0 MB/s | 54 kB 00:00 2024-08-03T20:45:17.043 INFO:teuthology.orchestra.run.smithi032.stdout:(92/160): python3-toml-0.10.2-6.el9.noarch.rpm 1.0 MB/s | 42 kB 00:00 2024-08-03T20:45:17.119 INFO:teuthology.orchestra.run.smithi032.stdout:(93/160): python3-numpy-1.20.1-5.el9.x86_64.rpm 11 MB/s | 5.1 MB 00:00 2024-08-03T20:45:17.144 INFO:teuthology.orchestra.run.smithi032.stdout:(94/160): python3-wcwidth-0.2.5-8.el9.noarch.rp 441 kB/s | 44 kB 00:00 2024-08-03T20:45:17.169 INFO:teuthology.orchestra.run.smithi032.stdout:(95/160): qatlib-24.02.0-1.el9.x86_64.rpm 4.3 MB/s | 221 kB 00:00 2024-08-03T20:45:17.194 INFO:teuthology.orchestra.run.smithi032.stdout:(96/160): qatlib-service-24.02.0-1.el9.x86_64.r 715 kB/s | 36 kB 00:00 2024-08-03T20:45:17.245 INFO:teuthology.orchestra.run.smithi032.stdout:(97/160): qatzip-libs-1.2.0-1.el9.x86_64.rpm 629 kB/s | 47 kB 00:00 2024-08-03T20:45:17.270 INFO:teuthology.orchestra.run.smithi032.stdout:(98/160): socat-1.7.4.1-6.el9.x86_64.rpm 3.9 MB/s | 304 kB 00:00 2024-08-03T20:45:17.295 INFO:teuthology.orchestra.run.smithi032.stdout:(99/160): uuid-1.6.2-55.el9.x86_64.rpm 1.1 MB/s | 58 kB 00:00 2024-08-03T20:45:17.321 INFO:teuthology.orchestra.run.smithi032.stdout:(100/160): xfsprogs-devel-6.4.0-3.el9.x86_64.rp 1.9 MB/s | 96 kB 00:00 2024-08-03T20:45:17.346 INFO:teuthology.orchestra.run.smithi032.stdout:(101/160): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.2 MB/s | 64 kB 00:00 2024-08-03T20:45:17.371 INFO:teuthology.orchestra.run.smithi032.stdout:(102/160): zlib-devel-1.2.11-41.el9.x86_64.rpm 892 kB/s | 45 kB 00:00 2024-08-03T20:45:17.438 INFO:teuthology.orchestra.run.smithi032.stdout:(103/160): lua-devel-5.4.4-4.el9.x86_64.rpm 243 kB/s | 22 kB 00:00 2024-08-03T20:45:17.522 INFO:teuthology.orchestra.run.smithi032.stdout:(104/160): abseil-cpp-20211102.0-3.el9.x86_64.r 6.5 MB/s | 548 kB 00:00 2024-08-03T20:45:17.558 INFO:teuthology.orchestra.run.smithi032.stdout:(105/160): gperftools-libs-2.9.1-2.el9.x86_64.r 9.1 MB/s | 309 kB 00:00 2024-08-03T20:45:17.573 INFO:teuthology.orchestra.run.smithi032.stdout:(106/160): grpc-data-1.46.7-10.el9.noarch.rpm 1.1 MB/s | 19 kB 00:00 2024-08-03T20:45:17.606 INFO:teuthology.orchestra.run.smithi032.stdout:(107/160): protobuf-compiler-3.14.0-13.el9.x86_ 3.6 MB/s | 863 kB 00:00 2024-08-03T20:45:17.815 INFO:teuthology.orchestra.run.smithi032.stdout:(108/160): python3-scipy-1.6.2-8.el9.x86_64.rpm 19 MB/s | 16 MB 00:00 2024-08-03T20:45:17.865 INFO:teuthology.orchestra.run.smithi032.stdout:(109/160): libarrow-doc-9.0.0-11.el9.noarch.rpm 100 kB/s | 26 kB 00:00 2024-08-03T20:45:17.891 INFO:teuthology.orchestra.run.smithi032.stdout:(110/160): libunwind-1.6.2-1.el9.x86_64.rpm 2.9 MB/s | 67 kB 00:00 2024-08-03T20:45:17.916 INFO:teuthology.orchestra.run.smithi032.stdout:(111/160): liboath-2.6.7-2.el9.x86_64.rpm 485 kB/s | 49 kB 00:00 2024-08-03T20:45:17.941 INFO:teuthology.orchestra.run.smithi032.stdout:(112/160): luarocks-3.9.2-5.el9.noarch.rpm 2.9 MB/s | 151 kB 00:00 2024-08-03T20:45:17.984 INFO:teuthology.orchestra.run.smithi032.stdout:(113/160): python3-asyncssh-2.13.2-5.el9.noarch 13 MB/s | 548 kB 00:00 2024-08-03T20:45:18.009 INFO:teuthology.orchestra.run.smithi032.stdout:(114/160): python3-bcrypt-3.2.2-1.el9.x86_64.rp 1.7 MB/s | 43 kB 00:00 2024-08-03T20:45:18.042 INFO:teuthology.orchestra.run.smithi032.stdout:(115/160): parquet-libs-9.0.0-11.el9.x86_64.rpm 6.5 MB/s | 839 kB 00:00 2024-08-03T20:45:18.068 INFO:teuthology.orchestra.run.smithi032.stdout:(116/160): python3-cachetools-4.2.4-1.el9.noarc 550 kB/s | 32 kB 00:00 2024-08-03T20:45:18.093 INFO:teuthology.orchestra.run.smithi032.stdout:(117/160): python3-certifi-2023.05.07-4.el9.noa 281 kB/s | 14 kB 00:00 2024-08-03T20:45:18.118 INFO:teuthology.orchestra.run.smithi032.stdout:(118/160): python3-cheroot-8.6.0-4.el9.noarch.r 3.3 MB/s | 172 kB 00:00 2024-08-03T20:45:18.152 INFO:teuthology.orchestra.run.smithi032.stdout:(119/160): python3-cherrypy-18.6.1-2.el9.noarch 6.0 MB/s | 358 kB 00:00 2024-08-03T20:45:18.177 INFO:teuthology.orchestra.run.smithi032.stdout:(120/160): python3-cmd2-2.4.2-2.el9.noarch.rpm 5.1 MB/s | 303 kB 00:00 2024-08-03T20:45:18.203 INFO:teuthology.orchestra.run.smithi032.stdout:(121/160): python3-google-auth-2.32.0-1.el9.noa 4.2 MB/s | 216 kB 00:00 2024-08-03T20:45:18.228 INFO:teuthology.orchestra.run.smithi032.stdout:(122/160): python3-grpcio-tools-1.46.7-10.el9.x 5.7 MB/s | 144 kB 00:00 2024-08-03T20:45:18.253 INFO:teuthology.orchestra.run.smithi032.stdout:(123/160): python3-jaraco-8.2.1-3.el9.noarch.rp 429 kB/s | 11 kB 00:00 2024-08-03T20:45:18.279 INFO:teuthology.orchestra.run.smithi032.stdout:(124/160): python3-jaraco-classes-3.2.1-5.el9.n 713 kB/s | 18 kB 00:00 2024-08-03T20:45:18.304 INFO:teuthology.orchestra.run.smithi032.stdout:(125/160): python3-jaraco-collections-3.0.0-8.e 931 kB/s | 23 kB 00:00 2024-08-03T20:45:18.331 INFO:teuthology.orchestra.run.smithi032.stdout:(126/160): python3-jaraco-functools-3.5.0-2.el9 774 kB/s | 19 kB 00:00 2024-08-03T20:45:18.355 INFO:teuthology.orchestra.run.smithi032.stdout:(127/160): python3-jaraco-text-3.2.0-6.el9.noar 795 kB/s | 20 kB 00:00 2024-08-03T20:45:18.422 INFO:teuthology.orchestra.run.smithi032.stdout:(128/160): python3-kubernetes-26.1.0-2.el9.noar 15 MB/s | 1.0 MB 00:00 2024-08-03T20:45:18.447 INFO:teuthology.orchestra.run.smithi032.stdout:(129/160): python3-logutils-0.3.5-21.el9.noarch 1.8 MB/s | 46 kB 00:00 2024-08-03T20:45:18.497 INFO:teuthology.orchestra.run.smithi032.stdout:(130/160): python3-grpcio-1.46.7-10.el9.x86_64. 6.4 MB/s | 2.0 MB 00:00 2024-08-03T20:45:18.523 INFO:teuthology.orchestra.run.smithi032.stdout:(131/160): python3-more-itertools-8.12.0-2.el9. 1.0 MB/s | 79 kB 00:00 2024-08-03T20:45:18.548 INFO:teuthology.orchestra.run.smithi032.stdout:(132/160): python3-natsort-7.1.1-5.el9.noarch.r 1.1 MB/s | 58 kB 00:00 2024-08-03T20:45:18.623 INFO:teuthology.orchestra.run.smithi032.stdout:(133/160): libarrow-9.0.0-11.el9.x86_64.rpm 4.2 MB/s | 4.4 MB 00:01 2024-08-03T20:45:18.649 INFO:teuthology.orchestra.run.smithi032.stdout:(134/160): python3-portend-3.1.0-2.el9.noarch.r 164 kB/s | 16 kB 00:00 2024-08-03T20:45:18.673 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction check 2024-08-03T20:45:18.674 INFO:teuthology.orchestra.run.smithi032.stdout:(135/160): python3-pecan-1.4.2-3.el9.noarch.rpm 1.8 MB/s | 272 kB 00:00 2024-08-03T20:45:18.699 INFO:teuthology.orchestra.run.smithi032.stdout:(136/160): python3-pyperclip-1.8.2-2.el9.noarch 562 kB/s | 28 kB 00:00 2024-08-03T20:45:18.725 INFO:teuthology.orchestra.run.smithi032.stdout:(137/160): python3-repoze-lru-0.7-16.el9.noarch 614 kB/s | 31 kB 00:00 2024-08-03T20:45:18.759 INFO:teuthology.orchestra.run.smithi032.stdout:(138/160): python3-pyOpenSSL-21.0.0-1.el9.noarc 667 kB/s | 90 kB 00:00 2024-08-03T20:45:18.775 INFO:teuthology.orchestra.run.smithi032.stdout:(139/160): python3-rsa-4.9-2.el9.noarch.rpm 1.1 MB/s | 59 kB 00:00 2024-08-03T20:45:18.792 INFO:teuthology.orchestra.run.smithi032.stdout:(140/160): python3-tempora-5.0.0-2.el9.noarch.r 1.0 MB/s | 36 kB 00:00 2024-08-03T20:45:18.818 INFO:teuthology.orchestra.run.smithi032.stdout:(141/160): python3-typing-extensions-4.12.2-2.e 1.7 MB/s | 74 kB 00:00 2024-08-03T20:45:18.843 INFO:teuthology.orchestra.run.smithi032.stdout:(142/160): python3-routes-2.5.1-5.el9.noarch.rp 1.3 MB/s | 188 kB 00:00 2024-08-03T20:45:18.868 INFO:teuthology.orchestra.run.smithi032.stdout:(143/160): python3-websocket-client-1.2.3-2.el9 1.7 MB/s | 90 kB 00:00 2024-08-03T20:45:18.894 INFO:teuthology.orchestra.run.smithi032.stdout:(144/160): python3-zc-lockfile-2.0-10.el9.noarc 801 kB/s | 20 kB 00:00 2024-08-03T20:45:18.899 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction check succeeded. 2024-08-03T20:45:18.899 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction test 2024-08-03T20:45:18.919 INFO:teuthology.orchestra.run.smithi032.stdout:(145/160): python3-webob-1.8.7-6.el9.noarch.rpm 1.8 MB/s | 230 kB 00:00 2024-08-03T20:45:18.953 INFO:teuthology.orchestra.run.smithi032.stdout:(146/160): re2-20211101-3.el9.x86_64.rpm 3.2 MB/s | 192 kB 00:00 2024-08-03T20:45:18.978 INFO:teuthology.orchestra.run.smithi032.stdout:(147/160): python3-werkzeug-2.0.3-3.el9.1.noarc 3.1 MB/s | 427 kB 00:00 2024-08-03T20:45:19.037 INFO:teuthology.orchestra.run.smithi032.stdout:(148/160): thrift-0.15.0-2.el9.x86_64.rpm 14 MB/s | 1.6 MB 00:00 2024-08-03T20:45:19.270 INFO:teuthology.orchestra.run.smithi032.stdout:(149/160): NetworkManager-1.48.2-2.el9.x86_64.r 11 MB/s | 2.3 MB 00:00 2024-08-03T20:45:19.346 INFO:teuthology.orchestra.run.smithi032.stdout:(150/160): NetworkManager-libnm-1.48.2-2.el9.x8 24 MB/s | 1.8 MB 00:00 2024-08-03T20:45:19.398 INFO:teuthology.orchestra.run.smithi032.stdout:(151/160): NetworkManager-team-1.48.2-2.el9.x86 820 kB/s | 41 kB 00:00 2024-08-03T20:45:19.463 INFO:teuthology.orchestra.run.smithi032.stdout:(152/160): librados2-19.0.0-5476.g3792a0a4.el9. 6.6 MB/s | 3.4 MB 00:00 2024-08-03T20:45:19.488 INFO:teuthology.orchestra.run.smithi032.stdout:(153/160): NetworkManager-tui-1.48.2-2.el9.x86_ 2.6 MB/s | 247 kB 00:00 2024-08-03T20:45:19.514 INFO:teuthology.orchestra.run.smithi032.stdout:(154/160): elfutils-debuginfod-client-0.191-4.e 729 kB/s | 37 kB 00:00 2024-08-03T20:45:19.539 INFO:teuthology.orchestra.run.smithi032.stdout:(155/160): elfutils-libelf-0.191-4.el9.x86_64.r 4.0 MB/s | 207 kB 00:00 2024-08-03T20:45:19.564 INFO:teuthology.orchestra.run.smithi032.stdout:(156/160): elfutils-libs-0.191-4.el9.x86_64.rpm 5.1 MB/s | 260 kB 00:00 2024-08-03T20:45:19.615 INFO:teuthology.orchestra.run.smithi032.stdout:(157/160): openssl-3.2.2-2.el9.x86_64.rpm 18 MB/s | 1.4 MB 00:00 2024-08-03T20:45:19.682 INFO:teuthology.orchestra.run.smithi032.stdout:(158/160): librbd1-19.0.0-5476.g3792a0a4.el9.x8 4.5 MB/s | 3.1 MB 00:00 2024-08-03T20:45:19.740 INFO:teuthology.orchestra.run.smithi032.stdout:(159/160): xfsprogs-6.4.0-3.el9.x86_64.rpm 8.7 MB/s | 1.1 MB 00:00 2024-08-03T20:45:19.790 INFO:teuthology.orchestra.run.smithi032.stdout:(160/160): openssl-libs-3.2.2-2.el9.x86_64.rpm 11 MB/s | 2.4 MB 00:00 2024-08-03T20:45:19.796 INFO:teuthology.orchestra.run.smithi032.stdout:-------------------------------------------------------------------------------- 2024-08-03T20:45:19.796 INFO:teuthology.orchestra.run.smithi032.stdout:Total 22 MB/s | 233 MB 00:10 2024-08-03T20:45:21.266 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction test succeeded. 2024-08-03T20:45:21.267 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction 2024-08-03T20:45:21.977 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction check 2024-08-03T20:45:22.222 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction check succeeded. 2024-08-03T20:45:22.222 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction test 2024-08-03T20:45:24.703 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction test succeeded. 2024-08-03T20:45:24.704 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction 2024-08-03T20:45:28.197 INFO:teuthology.orchestra.run.smithi118.stdout: Preparing : 1/1 2024-08-03T20:45:28.287 INFO:teuthology.orchestra.run.smithi118.stdout: Upgrading : openssl-libs-1:3.2.2-2.el9.x86_64 1/172 2024-08-03T20:45:28.347 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 2/172 2024-08-03T20:45:28.419 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 3/172 2024-08-03T20:45:28.489 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : thrift-0.15.0-2.el9.x86_64 4/172 2024-08-03T20:45:28.521 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 5/172 2024-08-03T20:45:28.583 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 6/172 2024-08-03T20:45:28.969 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 7/172 2024-08-03T20:45:29.137 INFO:teuthology.orchestra.run.smithi118.stdout: Upgrading : librados2-2:19.0.0-5476.g3792a0a4.el9.x86_64 8/172 2024-08-03T20:45:29.286 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: librados2-2:19.0.0-5476.g3792a0a4.el9.x86_64 8/172 2024-08-03T20:45:29.314 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : libcephfs2-2:19.0.0-5476.g3792a0a4.el9.x86_64 9/172 2024-08-03T20:45:29.422 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: libcephfs2-2:19.0.0-5476.g3792a0a4.el9.x86_64 9/172 2024-08-03T20:45:29.478 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-rados-2:19.0.0-5476.g3792a0a4.el9.x86_64 10/172 2024-08-03T20:45:29.529 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : postgresql-private-libs-13.14-1.el9.x86_64 11/172 2024-08-03T20:45:29.767 INFO:teuthology.orchestra.run.smithi118.stdout: Upgrading : elfutils-libelf-0.191-4.el9.x86_64 12/172 2024-08-03T20:45:29.834 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : postgresql-13.14-1.el9.x86_64 13/172 2024-08-03T20:45:29.861 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : libcephsqlite-2:19.0.0-5476.g3792a0a4.el9.x86_64 14/172 2024-08-03T20:45:29.999 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: libcephsqlite-2:19.0.0-5476.g3792a0a4.el9.x86_64 14/172 2024-08-03T20:45:30.036 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : libradosstriper1-2:19.0.0-5476.g3792a0a4.el9.x86 15/172 2024-08-03T20:45:30.110 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: libradosstriper1-2:19.0.0-5476.g3792a0a4.el9.x86 15/172 2024-08-03T20:45:30.146 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 16/172 2024-08-03T20:45:30.185 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 16/172 2024-08-03T20:45:30.421 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 17/172 2024-08-03T20:45:30.554 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 17/172 2024-08-03T20:45:30.818 INFO:teuthology.orchestra.run.smithi118.stdout: Upgrading : xfsprogs-6.4.0-3.el9.x86_64 18/172 2024-08-03T20:45:31.037 INFO:teuthology.orchestra.run.smithi118.stdout: Upgrading : NetworkManager-libnm-1:1.48.2-2.el9.x86_64 19/172 2024-08-03T20:45:31.115 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: NetworkManager-libnm-1:1.48.2-2.el9.x86_64 19/172 2024-08-03T20:45:31.222 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: NetworkManager-1:1.48.2-2.el9.x86_64 20/172 2024-08-03T20:45:31.348 INFO:teuthology.orchestra.run.smithi118.stdout: Upgrading : NetworkManager-1:1.48.2-2.el9.x86_64 20/172 2024-08-03T20:45:31.555 INFO:teuthology.orchestra.run.smithi032.stdout: Preparing : 1/1 2024-08-03T20:45:31.999 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: NetworkManager-1:1.48.2-2.el9.x86_64 20/172 2024-08-03T20:45:32.000 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : re2-1:20211101-3.el9.x86_64 21/172 2024-08-03T20:45:32.000 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : libarrow-9.0.0-11.el9.x86_64 22/172 2024-08-03T20:45:32.000 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : liboath-2.6.7-2.el9.x86_64 23/172 2024-08-03T20:45:32.000 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 24/172 2024-08-03T20:45:32.000 INFO:teuthology.orchestra.run.smithi032.stdout: Upgrading : openssl-libs-1:3.2.2-2.el9.x86_64 1/172 2024-08-03T20:45:32.000 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 2/172 2024-08-03T20:45:32.000 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 3/172 2024-08-03T20:45:32.001 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : thrift-0.15.0-2.el9.x86_64 4/172 2024-08-03T20:45:32.001 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 5/172 2024-08-03T20:45:32.001 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 6/172 2024-08-03T20:45:32.042 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 25/172 2024-08-03T20:45:32.087 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-packaging-20.9-5.el9.noarch 26/172 2024-08-03T20:45:32.142 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 27/172 2024-08-03T20:45:32.180 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 28/172 2024-08-03T20:45:32.214 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : lua-5.4.4-4.el9.x86_64 29/172 2024-08-03T20:45:32.293 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 7/172 2024-08-03T20:45:32.303 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 30/172 2024-08-03T20:45:32.367 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-urllib3-1.26.5-6.el9.noarch 31/172 2024-08-03T20:45:32.402 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 32/172 2024-08-03T20:45:32.414 INFO:teuthology.orchestra.run.smithi032.stdout: Upgrading : librados2-2:19.0.0-5476.g3792a0a4.el9.x86_64 8/172 2024-08-03T20:45:32.448 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 33/172 2024-08-03T20:45:32.528 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 34/172 2024-08-03T20:45:32.572 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-ceph-common-2:19.0.0-5476.g3792a0a4.el9. 35/172 2024-08-03T20:45:32.601 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: librados2-2:19.0.0-5476.g3792a0a4.el9.x86_64 8/172 2024-08-03T20:45:32.619 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-ceph-argparse-2:19.0.0-5476.g3792a0a4.el 36/172 2024-08-03T20:45:32.632 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : libcephfs2-2:19.0.0-5476.g3792a0a4.el9.x86_64 9/172 2024-08-03T20:45:32.652 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-cephfs-2:19.0.0-5476.g3792a0a4.el9.x86_6 37/172 2024-08-03T20:45:32.734 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: libcephfs2-2:19.0.0-5476.g3792a0a4.el9.x86_64 9/172 2024-08-03T20:45:32.735 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 38/172 2024-08-03T20:45:32.763 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-rados-2:19.0.0-5476.g3792a0a4.el9.x86_64 10/172 2024-08-03T20:45:32.788 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : luarocks-3.9.2-5.el9.noarch 39/172 2024-08-03T20:45:32.815 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : postgresql-private-libs-13.14-1.el9.x86_64 11/172 2024-08-03T20:45:32.880 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 40/172 2024-08-03T20:45:33.041 INFO:teuthology.orchestra.run.smithi032.stdout: Upgrading : elfutils-libelf-0.191-4.el9.x86_64 12/172 2024-08-03T20:45:33.052 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 41/172 2024-08-03T20:45:33.095 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : postgresql-13.14-1.el9.x86_64 13/172 2024-08-03T20:45:33.112 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : libcephsqlite-2:19.0.0-5476.g3792a0a4.el9.x86_64 14/172 2024-08-03T20:45:33.117 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 42/172 2024-08-03T20:45:33.243 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: libcephsqlite-2:19.0.0-5476.g3792a0a4.el9.x86_64 14/172 2024-08-03T20:45:33.263 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : libradosstriper1-2:19.0.0-5476.g3792a0a4.el9.x86 15/172 2024-08-03T20:45:33.346 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: libradosstriper1-2:19.0.0-5476.g3792a0a4.el9.x86 15/172 2024-08-03T20:45:33.365 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 16/172 2024-08-03T20:45:33.403 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 16/172 2024-08-03T20:45:33.668 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 17/172 2024-08-03T20:45:33.764 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-rsa-4.9-2.el9.noarch 43/172 2024-08-03T20:45:33.770 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 17/172 2024-08-03T20:45:33.859 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 44/172 2024-08-03T20:45:33.904 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-jinja2-2.11.3-6.el9.noarch 45/172 2024-08-03T20:45:33.941 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 46/172 2024-08-03T20:45:33.990 INFO:teuthology.orchestra.run.smithi118.stdout: Upgrading : elfutils-libs-0.191-4.el9.x86_64 47/172 2024-08-03T20:45:34.045 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : librados-devel-2:19.0.0-5476.g3792a0a4.el9.x86_6 48/172 2024-08-03T20:45:34.058 INFO:teuthology.orchestra.run.smithi032.stdout: Upgrading : xfsprogs-6.4.0-3.el9.x86_64 18/172 2024-08-03T20:45:34.099 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 49/172 2024-08-03T20:45:34.278 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : socat-1.7.4.1-6.el9.x86_64 50/172 2024-08-03T20:45:34.282 INFO:teuthology.orchestra.run.smithi032.stdout: Upgrading : NetworkManager-libnm-1:1.48.2-2.el9.x86_64 19/172 2024-08-03T20:45:34.343 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: NetworkManager-libnm-1:1.48.2-2.el9.x86_64 19/172 2024-08-03T20:45:34.365 INFO:teuthology.orchestra.run.smithi118.stdout: Upgrading : openssl-1:3.2.2-2.el9.x86_64 51/172 2024-08-03T20:45:34.468 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: NetworkManager-1:1.48.2-2.el9.x86_64 20/172 2024-08-03T20:45:34.471 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 52/172 2024-08-03T20:45:34.566 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 53/172 2024-08-03T20:45:34.568 INFO:teuthology.orchestra.run.smithi032.stdout: Upgrading : NetworkManager-1:1.48.2-2.el9.x86_64 20/172 2024-08-03T20:45:34.653 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 54/172 2024-08-03T20:45:34.701 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 55/172 2024-08-03T20:45:34.738 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-typing-extensions-4.12.2-2.el9.noarch 56/172 2024-08-03T20:45:34.795 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 57/172 2024-08-03T20:45:34.823 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: NetworkManager-1:1.48.2-2.el9.x86_64 20/172 2024-08-03T20:45:34.836 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 58/172 2024-08-03T20:45:34.882 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 59/172 2024-08-03T20:45:34.932 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 60/172 2024-08-03T20:45:34.978 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : re2-1:20211101-3.el9.x86_64 21/172 2024-08-03T20:45:35.007 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : libarrow-9.0.0-11.el9.x86_64 22/172 2024-08-03T20:45:35.062 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : liboath-2.6.7-2.el9.x86_64 23/172 2024-08-03T20:45:35.140 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 61/172 2024-08-03T20:45:35.147 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 24/172 2024-08-03T20:45:35.206 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 25/172 2024-08-03T20:45:35.244 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-packaging-20.9-5.el9.noarch 26/172 2024-08-03T20:45:35.299 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 27/172 2024-08-03T20:45:35.337 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 28/172 2024-08-03T20:45:35.373 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : lua-5.4.4-4.el9.x86_64 29/172 2024-08-03T20:45:35.465 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 30/172 2024-08-03T20:45:35.474 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 62/172 2024-08-03T20:45:35.526 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-certifi-2023.05.07-4.el9.noarch 63/172 2024-08-03T20:45:35.526 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-urllib3-1.26.5-6.el9.noarch 31/172 2024-08-03T20:45:35.560 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 32/172 2024-08-03T20:45:35.607 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 33/172 2024-08-03T20:45:35.646 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 64/172 2024-08-03T20:45:35.682 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-google-auth-1:2.32.0-1.el9.noarch 65/172 2024-08-03T20:45:35.690 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 34/172 2024-08-03T20:45:35.714 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-ceph-common-2:19.0.0-5476.g3792a0a4.el9. 35/172 2024-08-03T20:45:35.752 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-ceph-argparse-2:19.0.0-5476.g3792a0a4.el 36/172 2024-08-03T20:45:35.768 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 66/172 2024-08-03T20:45:35.777 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-cephfs-2:19.0.0-5476.g3792a0a4.el9.x86_6 37/172 2024-08-03T20:45:35.805 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 67/172 2024-08-03T20:45:35.829 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : libarrow-doc-9.0.0-11.el9.noarch 68/172 2024-08-03T20:45:35.871 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 38/172 2024-08-03T20:45:35.922 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : luarocks-3.9.2-5.el9.noarch 39/172 2024-08-03T20:45:35.935 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 69/172 2024-08-03T20:45:36.016 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 40/172 2024-08-03T20:45:36.099 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 70/172 2024-08-03T20:45:36.138 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 71/172 2024-08-03T20:45:36.174 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : zlib-devel-1.2.11-41.el9.x86_64 72/172 2024-08-03T20:45:36.188 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 41/172 2024-08-03T20:45:36.233 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 73/172 2024-08-03T20:45:36.259 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 42/172 2024-08-03T20:45:36.275 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : uuid-1.6.2-55.el9.x86_64 74/172 2024-08-03T20:45:36.315 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 75/172 2024-08-03T20:45:36.352 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 76/172 2024-08-03T20:45:36.387 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 77/172 2024-08-03T20:45:36.420 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 78/172 2024-08-03T20:45:36.463 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 79/172 2024-08-03T20:45:36.496 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 80/172 2024-08-03T20:45:36.596 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 81/172 2024-08-03T20:45:36.646 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 82/172 2024-08-03T20:45:36.683 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 83/172 2024-08-03T20:45:36.830 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 84/172 2024-08-03T20:45:36.905 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 85/172 2024-08-03T20:45:36.951 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-rsa-4.9-2.el9.noarch 43/172 2024-08-03T20:45:37.054 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 44/172 2024-08-03T20:45:37.095 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-jinja2-2.11.3-6.el9.noarch 45/172 2024-08-03T20:45:37.414 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 46/172 2024-08-03T20:45:37.474 INFO:teuthology.orchestra.run.smithi032.stdout: Upgrading : elfutils-libs-0.191-4.el9.x86_64 47/172 2024-08-03T20:45:37.512 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : librados-devel-2:19.0.0-5476.g3792a0a4.el9.x86_6 48/172 2024-08-03T20:45:37.567 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 49/172 2024-08-03T20:45:37.746 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : socat-1.7.4.1-6.el9.x86_64 50/172 2024-08-03T20:45:37.804 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 86/172 2024-08-03T20:45:37.823 INFO:teuthology.orchestra.run.smithi032.stdout: Upgrading : openssl-1:3.2.2-2.el9.x86_64 51/172 2024-08-03T20:45:37.857 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 87/172 2024-08-03T20:45:37.891 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 88/172 2024-08-03T20:45:37.932 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 52/172 2024-08-03T20:45:38.008 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 89/172 2024-08-03T20:45:38.010 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 53/172 2024-08-03T20:45:38.082 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-devel-3.9.18-3.el9.x86_64 90/172 2024-08-03T20:45:38.104 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 54/172 2024-08-03T20:45:38.136 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-colorama-0.4.6-3.el9.noarch 91/172 2024-08-03T20:45:38.149 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 55/172 2024-08-03T20:45:38.185 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-typing-extensions-4.12.2-2.el9.noarch 56/172 2024-08-03T20:45:38.234 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 92/172 2024-08-03T20:45:38.244 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 57/172 2024-08-03T20:45:38.277 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 93/172 2024-08-03T20:45:38.285 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 58/172 2024-08-03T20:45:38.331 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 59/172 2024-08-03T20:45:38.379 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 60/172 2024-08-03T20:45:38.498 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : openblas-0.3.21-2.el9.x86_64 94/172 2024-08-03T20:45:38.532 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 95/172 2024-08-03T20:45:38.560 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 61/172 2024-08-03T20:45:38.613 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 62/172 2024-08-03T20:45:38.656 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-certifi-2023.05.07-4.el9.noarch 63/172 2024-08-03T20:45:38.740 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 96/172 2024-08-03T20:45:38.782 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 64/172 2024-08-03T20:45:38.820 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-google-auth-1:2.32.0-1.el9.noarch 65/172 2024-08-03T20:45:38.864 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 66/172 2024-08-03T20:45:38.901 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 67/172 2024-08-03T20:45:38.926 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : libarrow-doc-9.0.0-11.el9.noarch 68/172 2024-08-03T20:45:39.031 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 69/172 2024-08-03T20:45:39.202 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 70/172 2024-08-03T20:45:39.242 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 71/172 2024-08-03T20:45:39.279 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : zlib-devel-1.2.11-41.el9.x86_64 72/172 2024-08-03T20:45:39.320 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 73/172 2024-08-03T20:45:39.363 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : uuid-1.6.2-55.el9.x86_64 74/172 2024-08-03T20:45:39.402 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 75/172 2024-08-03T20:45:39.439 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 76/172 2024-08-03T20:45:39.482 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 77/172 2024-08-03T20:45:39.515 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 78/172 2024-08-03T20:45:39.559 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 79/172 2024-08-03T20:45:39.573 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 97/172 2024-08-03T20:45:39.591 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 80/172 2024-08-03T20:45:39.695 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 81/172 2024-08-03T20:45:39.713 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 98/172 2024-08-03T20:45:39.759 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 82/172 2024-08-03T20:45:39.803 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 83/172 2024-08-03T20:45:39.932 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 84/172 2024-08-03T20:45:40.008 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 85/172 2024-08-03T20:45:40.947 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 86/172 2024-08-03T20:45:41.009 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 87/172 2024-08-03T20:45:41.043 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 88/172 2024-08-03T20:45:41.163 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 89/172 2024-08-03T20:45:41.228 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-devel-3.9.18-3.el9.x86_64 90/172 2024-08-03T20:45:41.281 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-colorama-0.4.6-3.el9.noarch 91/172 2024-08-03T20:45:41.309 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 99/172 2024-08-03T20:45:41.390 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 92/172 2024-08-03T20:45:41.413 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 100/172 2024-08-03T20:45:41.439 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 93/172 2024-08-03T20:45:41.450 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 101/172 2024-08-03T20:45:41.483 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 102/172 2024-08-03T20:45:41.674 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : openblas-0.3.21-2.el9.x86_64 94/172 2024-08-03T20:45:41.700 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 95/172 2024-08-03T20:45:41.799 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 96/172 2024-08-03T20:45:41.844 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 103/172 2024-08-03T20:45:41.903 INFO:teuthology.orchestra.run.smithi118.stdout: Upgrading : librbd1-2:19.0.0-5476.g3792a0a4.el9.x86_64 104/172 2024-08-03T20:45:42.020 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: librbd1-2:19.0.0-5476.g3792a0a4.el9.x86_64 104/172 2024-08-03T20:45:42.056 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-rbd-2:19.0.0-5476.g3792a0a4.el9.x86_64 105/172 2024-08-03T20:45:42.109 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 106/172 2024-08-03T20:45:42.648 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 97/172 2024-08-03T20:45:42.658 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : parquet-libs-9.0.0-11.el9.x86_64 107/172 2024-08-03T20:45:42.690 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : librgw2-2:19.0.0-5476.g3792a0a4.el9.x86_64 108/172 2024-08-03T20:45:42.765 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: librgw2-2:19.0.0-5476.g3792a0a4.el9.x86_64 108/172 2024-08-03T20:45:42.792 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-rgw-2:19.0.0-5476.g3792a0a4.el9.x86_64 109/172 2024-08-03T20:45:42.800 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 98/172 2024-08-03T20:45:44.469 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 99/172 2024-08-03T20:45:44.572 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 100/172 2024-08-03T20:45:44.608 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 101/172 2024-08-03T20:45:44.641 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 102/172 2024-08-03T20:45:45.018 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 103/172 2024-08-03T20:45:45.077 INFO:teuthology.orchestra.run.smithi032.stdout: Upgrading : librbd1-2:19.0.0-5476.g3792a0a4.el9.x86_64 104/172 2024-08-03T20:45:45.212 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: librbd1-2:19.0.0-5476.g3792a0a4.el9.x86_64 104/172 2024-08-03T20:45:45.242 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-rbd-2:19.0.0-5476.g3792a0a4.el9.x86_64 105/172 2024-08-03T20:45:45.299 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 106/172 2024-08-03T20:45:45.339 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: ceph-common-2:19.0.0-5476.g3792a0a4.el9.x86_64 110/172 2024-08-03T20:45:45.380 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : ceph-common-2:19.0.0-5476.g3792a0a4.el9.x86_64 110/172 2024-08-03T20:45:45.883 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : parquet-libs-9.0.0-11.el9.x86_64 107/172 2024-08-03T20:45:45.914 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : librgw2-2:19.0.0-5476.g3792a0a4.el9.x86_64 108/172 2024-08-03T20:45:46.006 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: librgw2-2:19.0.0-5476.g3792a0a4.el9.x86_64 108/172 2024-08-03T20:45:46.032 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-rgw-2:19.0.0-5476.g3792a0a4.el9.x86_64 109/172 2024-08-03T20:45:46.044 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: ceph-common-2:19.0.0-5476.g3792a0a4.el9.x86_64 110/172 2024-08-03T20:45:46.084 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : ceph-base-2:19.0.0-5476.g3792a0a4.el9.x86_64 111/172 2024-08-03T20:45:46.245 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: ceph-base-2:19.0.0-5476.g3792a0a4.el9.x86_64 111/172 2024-08-03T20:45:46.245 INFO:teuthology.orchestra.run.smithi118.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-08-03T20:45:46.246 INFO:teuthology.orchestra.run.smithi118.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-08-03T20:45:46.246 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-08-03T20:45:46.276 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : ceph-selinux-2:19.0.0-5476.g3792a0a4.el9.x86_64 112/172 2024-08-03T20:45:48.648 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: ceph-common-2:19.0.0-5476.g3792a0a4.el9.x86_64 110/172 2024-08-03T20:45:48.695 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : ceph-common-2:19.0.0-5476.g3792a0a4.el9.x86_64 110/172 2024-08-03T20:45:49.395 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: ceph-common-2:19.0.0-5476.g3792a0a4.el9.x86_64 110/172 2024-08-03T20:45:49.440 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : ceph-base-2:19.0.0-5476.g3792a0a4.el9.x86_64 111/172 2024-08-03T20:45:49.553 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: ceph-base-2:19.0.0-5476.g3792a0a4.el9.x86_64 111/172 2024-08-03T20:45:49.553 INFO:teuthology.orchestra.run.smithi032.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-08-03T20:45:49.553 INFO:teuthology.orchestra.run.smithi032.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-08-03T20:45:49.553 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-03T20:45:49.583 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : ceph-selinux-2:19.0.0-5476.g3792a0a4.el9.x86_64 112/172 2024-08-03T20:46:00.354 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: ceph-selinux-2:19.0.0-5476.g3792a0a4.el9.x86_64 112/172 2024-08-03T20:46:00.354 INFO:teuthology.orchestra.run.smithi118.stdout:skipping the directory /sys 2024-08-03T20:46:00.354 INFO:teuthology.orchestra.run.smithi118.stdout:skipping the directory /proc 2024-08-03T20:46:00.354 INFO:teuthology.orchestra.run.smithi118.stdout:skipping the directory /mnt 2024-08-03T20:46:00.354 INFO:teuthology.orchestra.run.smithi118.stdout:skipping the directory /var/tmp 2024-08-03T20:46:00.354 INFO:teuthology.orchestra.run.smithi118.stdout:skipping the directory /home 2024-08-03T20:46:00.354 INFO:teuthology.orchestra.run.smithi118.stdout:skipping the directory /root 2024-08-03T20:46:00.354 INFO:teuthology.orchestra.run.smithi118.stdout:skipping the directory /tmp 2024-08-03T20:46:00.354 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-08-03T20:46:00.565 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : ceph-volume-2:19.0.0-5476.g3792a0a4.el9.noarch 113/172 2024-08-03T20:46:00.602 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: ceph-volume-2:19.0.0-5476.g3792a0a4.el9.noarch 113/172 2024-08-03T20:46:00.602 INFO:teuthology.orchestra.run.smithi118.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-08-03T20:46:00.602 INFO:teuthology.orchestra.run.smithi118.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-08-03T20:46:00.602 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-08-03T20:46:02.284 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : ceph-osd-2:19.0.0-5476.g3792a0a4.el9.x86_64 114/172 2024-08-03T20:46:02.342 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: ceph-osd-2:19.0.0-5476.g3792a0a4.el9.x86_64 114/172 2024-08-03T20:46:02.342 INFO:teuthology.orchestra.run.smithi118.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-08-03T20:46:02.342 INFO:teuthology.orchestra.run.smithi118.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-08-03T20:46:02.342 INFO:teuthology.orchestra.run.smithi118.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-08-03T20:46:02.343 INFO:teuthology.orchestra.run.smithi118.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-08-03T20:46:02.343 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-08-03T20:46:02.528 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: ceph-selinux-2:19.0.0-5476.g3792a0a4.el9.x86_64 112/172 2024-08-03T20:46:02.528 INFO:teuthology.orchestra.run.smithi032.stdout:skipping the directory /sys 2024-08-03T20:46:02.528 INFO:teuthology.orchestra.run.smithi032.stdout:skipping the directory /proc 2024-08-03T20:46:02.528 INFO:teuthology.orchestra.run.smithi032.stdout:skipping the directory /mnt 2024-08-03T20:46:02.528 INFO:teuthology.orchestra.run.smithi032.stdout:skipping the directory /var/tmp 2024-08-03T20:46:02.528 INFO:teuthology.orchestra.run.smithi032.stdout:skipping the directory /home 2024-08-03T20:46:02.528 INFO:teuthology.orchestra.run.smithi032.stdout:skipping the directory /root 2024-08-03T20:46:02.529 INFO:teuthology.orchestra.run.smithi032.stdout:skipping the directory /tmp 2024-08-03T20:46:02.529 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-03T20:46:02.602 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : ceph-mds-2:19.0.0-5476.g3792a0a4.el9.x86_64 115/172 2024-08-03T20:46:02.652 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: ceph-mds-2:19.0.0-5476.g3792a0a4.el9.x86_64 115/172 2024-08-03T20:46:02.652 INFO:teuthology.orchestra.run.smithi118.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-08-03T20:46:02.653 INFO:teuthology.orchestra.run.smithi118.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-08-03T20:46:02.653 INFO:teuthology.orchestra.run.smithi118.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-08-03T20:46:02.653 INFO:teuthology.orchestra.run.smithi118.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-08-03T20:46:02.653 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-08-03T20:46:02.763 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : ceph-volume-2:19.0.0-5476.g3792a0a4.el9.noarch 113/172 2024-08-03T20:46:02.806 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: ceph-volume-2:19.0.0-5476.g3792a0a4.el9.noarch 113/172 2024-08-03T20:46:02.806 INFO:teuthology.orchestra.run.smithi032.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-08-03T20:46:02.807 INFO:teuthology.orchestra.run.smithi032.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-08-03T20:46:02.807 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-03T20:46:03.161 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : ceph-mon-2:19.0.0-5476.g3792a0a4.el9.x86_64 116/172 2024-08-03T20:46:03.212 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: ceph-mon-2:19.0.0-5476.g3792a0a4.el9.x86_64 116/172 2024-08-03T20:46:03.212 INFO:teuthology.orchestra.run.smithi118.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-08-03T20:46:03.212 INFO:teuthology.orchestra.run.smithi118.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-08-03T20:46:03.212 INFO:teuthology.orchestra.run.smithi118.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-08-03T20:46:03.212 INFO:teuthology.orchestra.run.smithi118.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-08-03T20:46:03.212 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-08-03T20:46:03.353 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : automake-1.16.2-8.el9.noarch 117/172 2024-08-03T20:46:03.438 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-ply-3.11-14.el9.noarch 118/172 2024-08-03T20:46:03.503 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 119/172 2024-08-03T20:46:03.738 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 120/172 2024-08-03T20:46:03.799 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 121/172 2024-08-03T20:46:03.885 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 122/172 2024-08-03T20:46:04.011 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 123/172 2024-08-03T20:46:04.187 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 124/172 2024-08-03T20:46:04.240 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 125/172 2024-08-03T20:46:04.280 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 126/172 2024-08-03T20:46:04.306 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : fuse-2.9.9-16.el9.x86_64 127/172 2024-08-03T20:46:04.574 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : ceph-osd-2:19.0.0-5476.g3792a0a4.el9.x86_64 114/172 2024-08-03T20:46:04.632 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: ceph-osd-2:19.0.0-5476.g3792a0a4.el9.x86_64 114/172 2024-08-03T20:46:04.632 INFO:teuthology.orchestra.run.smithi032.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-08-03T20:46:04.632 INFO:teuthology.orchestra.run.smithi032.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-08-03T20:46:04.632 INFO:teuthology.orchestra.run.smithi032.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-08-03T20:46:04.632 INFO:teuthology.orchestra.run.smithi032.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-08-03T20:46:04.633 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-03T20:46:04.874 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: cephadm-2:19.0.0-5476.g3792a0a4.el9.noarch 128/172 2024-08-03T20:46:04.891 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : ceph-mds-2:19.0.0-5476.g3792a0a4.el9.x86_64 115/172 2024-08-03T20:46:04.931 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : cephadm-2:19.0.0-5476.g3792a0a4.el9.noarch 128/172 2024-08-03T20:46:04.943 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: ceph-mds-2:19.0.0-5476.g3792a0a4.el9.x86_64 115/172 2024-08-03T20:46:04.943 INFO:teuthology.orchestra.run.smithi032.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-08-03T20:46:04.943 INFO:teuthology.orchestra.run.smithi032.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-08-03T20:46:04.943 INFO:teuthology.orchestra.run.smithi032.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-08-03T20:46:04.943 INFO:teuthology.orchestra.run.smithi032.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-08-03T20:46:04.943 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-03T20:46:04.965 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-5476.g3792a0a4.e 129/172 2024-08-03T20:46:05.043 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-5476.g3792a0a4. 130/172 2024-08-03T20:46:05.070 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-5476.g3792a0a4.el9.noa 131/172 2024-08-03T20:46:05.483 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : ceph-mon-2:19.0.0-5476.g3792a0a4.el9.x86_64 116/172 2024-08-03T20:46:05.534 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: ceph-mon-2:19.0.0-5476.g3792a0a4.el9.x86_64 116/172 2024-08-03T20:46:05.535 INFO:teuthology.orchestra.run.smithi032.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-08-03T20:46:05.535 INFO:teuthology.orchestra.run.smithi032.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-08-03T20:46:05.535 INFO:teuthology.orchestra.run.smithi032.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-08-03T20:46:05.535 INFO:teuthology.orchestra.run.smithi032.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-08-03T20:46:05.535 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-03T20:46:05.667 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : automake-1.16.2-8.el9.noarch 117/172 2024-08-03T20:46:05.728 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-ply-3.11-14.el9.noarch 118/172 2024-08-03T20:46:05.802 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 119/172 2024-08-03T20:46:06.038 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 120/172 2024-08-03T20:46:06.096 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 121/172 2024-08-03T20:46:06.186 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 122/172 2024-08-03T20:46:06.313 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 123/172 2024-08-03T20:46:06.490 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 124/172 2024-08-03T20:46:06.537 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 125/172 2024-08-03T20:46:06.577 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 126/172 2024-08-03T20:46:06.610 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : fuse-2.9.9-16.el9.x86_64 127/172 2024-08-03T20:46:07.130 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: cephadm-2:19.0.0-5476.g3792a0a4.el9.noarch 128/172 2024-08-03T20:46:07.159 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : cephadm-2:19.0.0-5476.g3792a0a4.el9.noarch 128/172 2024-08-03T20:46:07.194 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-5476.g3792a0a4.e 129/172 2024-08-03T20:46:07.275 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-5476.g3792a0a4. 130/172 2024-08-03T20:46:07.306 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-5476.g3792a0a4.el9.noa 131/172 2024-08-03T20:46:08.700 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-5476.g3792a0a4.el9.noa 131/172 2024-08-03T20:46:08.768 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-5476.g3792a0a4.el9.n 132/172 2024-08-03T20:46:09.848 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-5476.g3792a0a4.el9.n 132/172 2024-08-03T20:46:09.877 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-5476.g379 133/172 2024-08-03T20:46:10.026 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-5476.g379 133/172 2024-08-03T20:46:10.213 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-5476.g3792a0a4.el 134/172 2024-08-03T20:46:10.246 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : ceph-mgr-2:19.0.0-5476.g3792a0a4.el9.x86_64 135/172 2024-08-03T20:46:10.298 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: ceph-mgr-2:19.0.0-5476.g3792a0a4.el9.x86_64 135/172 2024-08-03T20:46:10.298 INFO:teuthology.orchestra.run.smithi118.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-08-03T20:46:10.298 INFO:teuthology.orchestra.run.smithi118.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-08-03T20:46:10.298 INFO:teuthology.orchestra.run.smithi118.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-08-03T20:46:10.298 INFO:teuthology.orchestra.run.smithi118.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-08-03T20:46:10.298 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-08-03T20:46:10.355 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : ceph-mgr-rook-2:19.0.0-5476.g3792a0a4.el9.noarch 136/172 2024-08-03T20:46:10.382 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-5476.g3792a0a4.el9.noarch 136/172 2024-08-03T20:46:10.493 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : ceph-2:19.0.0-5476.g3792a0a4.el9.x86_64 137/172 2024-08-03T20:46:10.561 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : ceph-fuse-2:19.0.0-5476.g3792a0a4.el9.x86_64 138/172 2024-08-03T20:46:11.144 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-5476.g3792a0a4.el9.noa 131/172 2024-08-03T20:46:11.213 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-5476.g3792a0a4.el9.n 132/172 2024-08-03T20:46:11.631 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : libtool-2.4.6-46.el9.x86_64 139/172 2024-08-03T20:46:11.666 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : ceph-radosgw-2:19.0.0-5476.g3792a0a4.el9.x86_64 140/172 2024-08-03T20:46:11.717 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-5476.g3792a0a4.el9.x86_64 140/172 2024-08-03T20:46:11.717 INFO:teuthology.orchestra.run.smithi118.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-08-03T20:46:11.717 INFO:teuthology.orchestra.run.smithi118.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-08-03T20:46:11.717 INFO:teuthology.orchestra.run.smithi118.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-08-03T20:46:11.717 INFO:teuthology.orchestra.run.smithi118.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-08-03T20:46:11.717 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-08-03T20:46:11.775 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-5476.g3792a 141/172 2024-08-03T20:46:11.825 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-5476.g3792a 141/172 2024-08-03T20:46:11.825 INFO:teuthology.orchestra.run.smithi118.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-08-03T20:46:11.825 INFO:teuthology.orchestra.run.smithi118.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-08-03T20:46:11.825 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-08-03T20:46:11.884 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : cephfs-mirror-2:19.0.0-5476.g3792a0a4.el9.x86_64 142/172 2024-08-03T20:46:11.935 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-5476.g3792a0a4.el9.x86_64 142/172 2024-08-03T20:46:11.935 INFO:teuthology.orchestra.run.smithi118.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-08-03T20:46:11.935 INFO:teuthology.orchestra.run.smithi118.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-08-03T20:46:11.935 INFO:teuthology.orchestra.run.smithi118.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-08-03T20:46:11.935 INFO:teuthology.orchestra.run.smithi118.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-08-03T20:46:11.935 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-08-03T20:46:12.325 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : rbd-mirror-2:19.0.0-5476.g3792a0a4.el9.x86_64 143/172 2024-08-03T20:46:12.362 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-5476.g3792a0a4.el9.n 132/172 2024-08-03T20:46:12.377 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: rbd-mirror-2:19.0.0-5476.g3792a0a4.el9.x86_64 143/172 2024-08-03T20:46:12.378 INFO:teuthology.orchestra.run.smithi118.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-08-03T20:46:12.378 INFO:teuthology.orchestra.run.smithi118.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-08-03T20:46:12.378 INFO:teuthology.orchestra.run.smithi118.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-08-03T20:46:12.378 INFO:teuthology.orchestra.run.smithi118.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-08-03T20:46:12.378 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-08-03T20:46:12.387 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-5476.g379 133/172 2024-08-03T20:46:12.544 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-5476.g379 133/172 2024-08-03T20:46:12.742 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-5476.g3792a0a4.el 134/172 2024-08-03T20:46:12.773 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : ceph-mgr-2:19.0.0-5476.g3792a0a4.el9.x86_64 135/172 2024-08-03T20:46:12.831 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: ceph-mgr-2:19.0.0-5476.g3792a0a4.el9.x86_64 135/172 2024-08-03T20:46:12.831 INFO:teuthology.orchestra.run.smithi032.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-08-03T20:46:12.831 INFO:teuthology.orchestra.run.smithi032.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-08-03T20:46:12.832 INFO:teuthology.orchestra.run.smithi032.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-08-03T20:46:12.832 INFO:teuthology.orchestra.run.smithi032.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-08-03T20:46:12.832 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-03T20:46:12.890 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : ceph-mgr-rook-2:19.0.0-5476.g3792a0a4.el9.noarch 136/172 2024-08-03T20:46:12.917 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-5476.g3792a0a4.el9.noarch 136/172 2024-08-03T20:46:13.030 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : ceph-2:19.0.0-5476.g3792a0a4.el9.x86_64 137/172 2024-08-03T20:46:13.097 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : ceph-fuse-2:19.0.0-5476.g3792a0a4.el9.x86_64 138/172 2024-08-03T20:46:14.223 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : libtool-2.4.6-46.el9.x86_64 139/172 2024-08-03T20:46:14.259 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : ceph-radosgw-2:19.0.0-5476.g3792a0a4.el9.x86_64 140/172 2024-08-03T20:46:14.312 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-5476.g3792a0a4.el9.x86_64 140/172 2024-08-03T20:46:14.312 INFO:teuthology.orchestra.run.smithi032.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-08-03T20:46:14.312 INFO:teuthology.orchestra.run.smithi032.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-08-03T20:46:14.312 INFO:teuthology.orchestra.run.smithi032.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-08-03T20:46:14.312 INFO:teuthology.orchestra.run.smithi032.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-08-03T20:46:14.313 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-03T20:46:14.368 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-5476.g3792a 141/172 2024-08-03T20:46:14.419 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-5476.g3792a 141/172 2024-08-03T20:46:14.419 INFO:teuthology.orchestra.run.smithi032.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-08-03T20:46:14.419 INFO:teuthology.orchestra.run.smithi032.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-08-03T20:46:14.419 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-03T20:46:14.486 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : cephfs-mirror-2:19.0.0-5476.g3792a0a4.el9.x86_64 142/172 2024-08-03T20:46:14.538 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-5476.g3792a0a4.el9.x86_64 142/172 2024-08-03T20:46:14.538 INFO:teuthology.orchestra.run.smithi032.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-08-03T20:46:14.538 INFO:teuthology.orchestra.run.smithi032.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-08-03T20:46:14.538 INFO:teuthology.orchestra.run.smithi032.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-08-03T20:46:14.539 INFO:teuthology.orchestra.run.smithi032.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-08-03T20:46:14.539 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-03T20:46:14.911 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : rbd-mirror-2:19.0.0-5476.g3792a0a4.el9.x86_64 143/172 2024-08-03T20:46:14.964 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: rbd-mirror-2:19.0.0-5476.g3792a0a4.el9.x86_64 143/172 2024-08-03T20:46:14.964 INFO:teuthology.orchestra.run.smithi032.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-08-03T20:46:14.964 INFO:teuthology.orchestra.run.smithi032.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-08-03T20:46:14.964 INFO:teuthology.orchestra.run.smithi032.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-08-03T20:46:14.964 INFO:teuthology.orchestra.run.smithi032.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-08-03T20:46:14.964 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-03T20:46:17.082 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : ceph-test-2:19.0.0-5476.g3792a0a4.el9.x86_64 144/172 2024-08-03T20:46:17.123 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : rbd-fuse-2:19.0.0-5476.g3792a0a4.el9.x86_64 145/172 2024-08-03T20:46:17.175 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : rbd-nbd-2:19.0.0-5476.g3792a0a4.el9.x86_64 146/172 2024-08-03T20:46:17.204 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : xfsprogs-devel-6.4.0-3.el9.x86_64 147/172 2024-08-03T20:46:17.239 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : elfutils-libelf-devel-0.191-4.el9.x86_64 148/172 2024-08-03T20:46:17.492 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : cephfs-shell-2:19.0.0-5476.g3792a0a4.el9.x86_64 149/172 2024-08-03T20:46:17.553 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : postgresql-contrib-13.14-1.el9.x86_64 150/172 2024-08-03T20:46:17.603 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : libcephfs-devel-2:19.0.0-5476.g3792a0a4.el9.x86_ 151/172 2024-08-03T20:46:17.642 INFO:teuthology.orchestra.run.smithi118.stdout: Upgrading : elfutils-debuginfod-client-0.191-4.el9.x86_64 152/172 2024-08-03T20:46:17.681 INFO:teuthology.orchestra.run.smithi118.stdout: Upgrading : NetworkManager-team-1:1.48.2-2.el9.x86_64 153/172 2024-08-03T20:46:17.726 INFO:teuthology.orchestra.run.smithi118.stdout: Upgrading : NetworkManager-tui-1:1.48.2-2.el9.x86_64 154/172 2024-08-03T20:46:18.440 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 155/172 2024-08-03T20:46:18.482 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : postgresql-server-13.14-1.el9.x86_64 155/172 2024-08-03T20:46:18.524 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 155/172 2024-08-03T20:46:19.787 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : cephfs-top-2:19.0.0-5476.g3792a0a4.el9.noarch 156/172 2024-08-03T20:46:19.900 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : openssl-devel-1:3.2.2-2.el9.x86_64 157/172 2024-08-03T20:46:19.954 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 158/172 2024-08-03T20:46:19.975 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : ceph-test-2:19.0.0-5476.g3792a0a4.el9.x86_64 144/172 2024-08-03T20:46:20.017 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : rbd-fuse-2:19.0.0-5476.g3792a0a4.el9.x86_64 145/172 2024-08-03T20:46:20.056 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : flex-2.6.4-9.el9.x86_64 159/172 2024-08-03T20:46:20.068 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : rbd-nbd-2:19.0.0-5476.g3792a0a4.el9.x86_64 146/172 2024-08-03T20:46:20.094 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : bison-3.7.4-5.el9.x86_64 160/172 2024-08-03T20:46:20.105 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : xfsprogs-devel-6.4.0-3.el9.x86_64 147/172 2024-08-03T20:46:20.119 INFO:teuthology.orchestra.run.smithi118.stdout: Cleanup : NetworkManager-tui-1:1.47.4-1.el9.x86_64 161/172 2024-08-03T20:46:20.141 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : elfutils-libelf-devel-0.191-4.el9.x86_64 148/172 2024-08-03T20:46:20.144 INFO:teuthology.orchestra.run.smithi118.stdout: Cleanup : elfutils-debuginfod-client-0.190-2.el9.x86_64 162/172 2024-08-03T20:46:20.183 INFO:teuthology.orchestra.run.smithi118.stdout: Cleanup : elfutils-libs-0.190-2.el9.x86_64 163/172 2024-08-03T20:46:20.183 INFO:teuthology.orchestra.run.smithi118.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 164/172 2024-08-03T20:46:20.316 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 164/172 2024-08-03T20:46:20.348 INFO:teuthology.orchestra.run.smithi118.stdout: Cleanup : openssl-1:3.0.7-27.el9.x86_64 165/172 2024-08-03T20:46:20.348 INFO:teuthology.orchestra.run.smithi118.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 166/172 2024-08-03T20:46:20.382 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : cephfs-shell-2:19.0.0-5476.g3792a0a4.el9.x86_64 149/172 2024-08-03T20:46:20.441 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 166/172 2024-08-03T20:46:20.445 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : postgresql-contrib-13.14-1.el9.x86_64 150/172 2024-08-03T20:46:20.465 INFO:teuthology.orchestra.run.smithi118.stdout: Cleanup : NetworkManager-team-1:1.47.4-1.el9.x86_64 167/172 2024-08-03T20:46:20.479 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : libcephfs-devel-2:19.0.0-5476.g3792a0a4.el9.x86_ 151/172 2024-08-03T20:46:20.481 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: NetworkManager-1:1.47.4-1.el9.x86_64 168/172 2024-08-03T20:46:20.482 INFO:teuthology.orchestra.run.smithi118.stdout: Cleanup : NetworkManager-1:1.47.4-1.el9.x86_64 168/172 2024-08-03T20:46:20.509 INFO:teuthology.orchestra.run.smithi032.stdout: Upgrading : elfutils-debuginfod-client-0.191-4.el9.x86_64 152/172 2024-08-03T20:46:20.541 INFO:teuthology.orchestra.run.smithi032.stdout: Upgrading : NetworkManager-team-1:1.48.2-2.el9.x86_64 153/172 2024-08-03T20:46:20.584 INFO:teuthology.orchestra.run.smithi032.stdout: Upgrading : NetworkManager-tui-1:1.48.2-2.el9.x86_64 154/172 2024-08-03T20:46:20.730 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: NetworkManager-1:1.47.4-1.el9.x86_64 168/172 2024-08-03T20:46:20.731 INFO:teuthology.orchestra.run.smithi118.stdout: Cleanup : NetworkManager-libnm-1:1.47.4-1.el9.x86_64 169/172 2024-08-03T20:46:20.842 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: NetworkManager-libnm-1:1.47.4-1.el9.x86_64 169/172 2024-08-03T20:46:20.874 INFO:teuthology.orchestra.run.smithi118.stdout: Cleanup : openssl-libs-1:3.0.7-27.el9.x86_64 170/172 2024-08-03T20:46:20.910 INFO:teuthology.orchestra.run.smithi118.stdout: Cleanup : elfutils-libelf-0.190-2.el9.x86_64 171/172 2024-08-03T20:46:20.991 INFO:teuthology.orchestra.run.smithi118.stdout: Cleanup : xfsprogs-6.3.0-1.el9.x86_64 172/172 2024-08-03T20:46:21.203 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: xfsprogs-6.3.0-1.el9.x86_64 172/172 2024-08-03T20:46:21.204 INFO:teuthology.orchestra.run.smithi118.stdout:Couldn't write '0' to 'kernel/yama/ptrace_scope', ignoring: No such file or directory 2024-08-03T20:46:21.204 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-08-03T20:46:21.361 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 155/172 2024-08-03T20:46:21.407 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : postgresql-server-13.14-1.el9.x86_64 155/172 2024-08-03T20:46:21.450 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 155/172 2024-08-03T20:46:22.808 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : cephfs-top-2:19.0.0-5476.g3792a0a4.el9.noarch 156/172 2024-08-03T20:46:22.923 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : openssl-devel-1:3.2.2-2.el9.x86_64 157/172 2024-08-03T20:46:22.970 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 158/172 2024-08-03T20:46:23.091 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : flex-2.6.4-9.el9.x86_64 159/172 2024-08-03T20:46:23.125 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : bison-3.7.4-5.el9.x86_64 160/172 2024-08-03T20:46:23.158 INFO:teuthology.orchestra.run.smithi032.stdout: Cleanup : NetworkManager-tui-1:1.47.4-1.el9.x86_64 161/172 2024-08-03T20:46:23.217 INFO:teuthology.orchestra.run.smithi032.stdout: Cleanup : elfutils-debuginfod-client-0.190-2.el9.x86_64 162/172 2024-08-03T20:46:23.245 INFO:teuthology.orchestra.run.smithi032.stdout: Cleanup : elfutils-libs-0.190-2.el9.x86_64 163/172 2024-08-03T20:46:23.245 INFO:teuthology.orchestra.run.smithi032.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 164/172 2024-08-03T20:46:23.353 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 164/172 2024-08-03T20:46:23.385 INFO:teuthology.orchestra.run.smithi032.stdout: Cleanup : openssl-1:3.0.7-27.el9.x86_64 165/172 2024-08-03T20:46:23.385 INFO:teuthology.orchestra.run.smithi032.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 166/172 2024-08-03T20:46:23.504 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 166/172 2024-08-03T20:46:23.531 INFO:teuthology.orchestra.run.smithi032.stdout: Cleanup : NetworkManager-team-1:1.47.4-1.el9.x86_64 167/172 2024-08-03T20:46:23.545 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: NetworkManager-1:1.47.4-1.el9.x86_64 168/172 2024-08-03T20:46:23.545 INFO:teuthology.orchestra.run.smithi032.stdout: Cleanup : NetworkManager-1:1.47.4-1.el9.x86_64 168/172 2024-08-03T20:46:23.789 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: NetworkManager-1:1.47.4-1.el9.x86_64 168/172 2024-08-03T20:46:23.789 INFO:teuthology.orchestra.run.smithi032.stdout: Cleanup : NetworkManager-libnm-1:1.47.4-1.el9.x86_64 169/172 2024-08-03T20:46:23.899 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: NetworkManager-libnm-1:1.47.4-1.el9.x86_64 169/172 2024-08-03T20:46:23.931 INFO:teuthology.orchestra.run.smithi032.stdout: Cleanup : openssl-libs-1:3.0.7-27.el9.x86_64 170/172 2024-08-03T20:46:23.971 INFO:teuthology.orchestra.run.smithi032.stdout: Cleanup : elfutils-libelf-0.190-2.el9.x86_64 171/172 2024-08-03T20:46:24.059 INFO:teuthology.orchestra.run.smithi032.stdout: Cleanup : xfsprogs-6.3.0-1.el9.x86_64 172/172 2024-08-03T20:46:24.257 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: xfsprogs-6.3.0-1.el9.x86_64 172/172 2024-08-03T20:46:24.258 INFO:teuthology.orchestra.run.smithi032.stdout:Couldn't write '0' to 'kernel/yama/ptrace_scope', ignoring: No such file or directory 2024-08-03T20:46:24.258 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-03T20:46:29.438 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : ceph-2:19.0.0-5476.g3792a0a4.el9.x86_64 1/172 2024-08-03T20:46:29.438 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : ceph-base-2:19.0.0-5476.g3792a0a4.el9.x86_64 2/172 2024-08-03T20:46:29.438 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : ceph-common-2:19.0.0-5476.g3792a0a4.el9.x86_64 3/172 2024-08-03T20:46:29.438 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : ceph-fuse-2:19.0.0-5476.g3792a0a4.el9.x86_64 4/172 2024-08-03T20:46:29.438 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-5476.g3792a 5/172 2024-08-03T20:46:29.438 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : ceph-mds-2:19.0.0-5476.g3792a0a4.el9.x86_64 6/172 2024-08-03T20:46:29.439 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : ceph-mgr-2:19.0.0-5476.g3792a0a4.el9.x86_64 7/172 2024-08-03T20:46:29.439 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : ceph-mon-2:19.0.0-5476.g3792a0a4.el9.x86_64 8/172 2024-08-03T20:46:29.439 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : ceph-osd-2:19.0.0-5476.g3792a0a4.el9.x86_64 9/172 2024-08-03T20:46:29.439 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : ceph-radosgw-2:19.0.0-5476.g3792a0a4.el9.x86_64 10/172 2024-08-03T20:46:29.439 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : ceph-selinux-2:19.0.0-5476.g3792a0a4.el9.x86_64 11/172 2024-08-03T20:46:29.439 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : ceph-test-2:19.0.0-5476.g3792a0a4.el9.x86_64 12/172 2024-08-03T20:46:29.439 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : cephfs-mirror-2:19.0.0-5476.g3792a0a4.el9.x86_64 13/172 2024-08-03T20:46:29.439 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : cephfs-shell-2:19.0.0-5476.g3792a0a4.el9.x86_64 14/172 2024-08-03T20:46:29.440 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : libcephfs-devel-2:19.0.0-5476.g3792a0a4.el9.x86_ 15/172 2024-08-03T20:46:29.440 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : libcephfs2-2:19.0.0-5476.g3792a0a4.el9.x86_64 16/172 2024-08-03T20:46:29.440 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : libcephsqlite-2:19.0.0-5476.g3792a0a4.el9.x86_64 17/172 2024-08-03T20:46:29.440 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : librados-devel-2:19.0.0-5476.g3792a0a4.el9.x86_6 18/172 2024-08-03T20:46:29.440 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : libradosstriper1-2:19.0.0-5476.g3792a0a4.el9.x86 19/172 2024-08-03T20:46:29.440 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : librgw2-2:19.0.0-5476.g3792a0a4.el9.x86_64 20/172 2024-08-03T20:46:29.440 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-ceph-argparse-2:19.0.0-5476.g3792a0a4.el 21/172 2024-08-03T20:46:29.441 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-ceph-common-2:19.0.0-5476.g3792a0a4.el9. 22/172 2024-08-03T20:46:29.441 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-cephfs-2:19.0.0-5476.g3792a0a4.el9.x86_6 23/172 2024-08-03T20:46:29.441 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-rados-2:19.0.0-5476.g3792a0a4.el9.x86_64 24/172 2024-08-03T20:46:29.441 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-rbd-2:19.0.0-5476.g3792a0a4.el9.x86_64 25/172 2024-08-03T20:46:29.454 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-rgw-2:19.0.0-5476.g3792a0a4.el9.x86_64 26/172 2024-08-03T20:46:29.454 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : rbd-fuse-2:19.0.0-5476.g3792a0a4.el9.x86_64 27/172 2024-08-03T20:46:29.454 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : rbd-mirror-2:19.0.0-5476.g3792a0a4.el9.x86_64 28/172 2024-08-03T20:46:29.454 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : rbd-nbd-2:19.0.0-5476.g3792a0a4.el9.x86_64 29/172 2024-08-03T20:46:29.454 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-5476.g3792a0a4. 30/172 2024-08-03T20:46:29.455 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-5476.g3792a0a4.el9.noa 31/172 2024-08-03T20:46:29.455 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-5476.g3792a0a4.el9.n 32/172 2024-08-03T20:46:29.455 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-5476.g379 33/172 2024-08-03T20:46:29.455 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-5476.g3792a0a4.el 34/172 2024-08-03T20:46:29.455 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : ceph-mgr-rook-2:19.0.0-5476.g3792a0a4.el9.noarch 35/172 2024-08-03T20:46:29.455 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-5476.g3792a0a4.e 36/172 2024-08-03T20:46:29.455 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : ceph-volume-2:19.0.0-5476.g3792a0a4.el9.noarch 37/172 2024-08-03T20:46:29.455 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : cephadm-2:19.0.0-5476.g3792a0a4.el9.noarch 38/172 2024-08-03T20:46:29.455 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : cephfs-top-2:19.0.0-5476.g3792a0a4.el9.noarch 39/172 2024-08-03T20:46:29.455 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : fuse-2.9.9-16.el9.x86_64 40/172 2024-08-03T20:46:29.455 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 41/172 2024-08-03T20:46:29.455 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 42/172 2024-08-03T20:46:29.455 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 43/172 2024-08-03T20:46:29.456 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 44/172 2024-08-03T20:46:29.456 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-ply-3.11-14.el9.noarch 45/172 2024-08-03T20:46:29.456 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 46/172 2024-08-03T20:46:29.456 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 47/172 2024-08-03T20:46:29.456 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-urllib3-1.26.5-6.el9.noarch 48/172 2024-08-03T20:46:29.456 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : automake-1.16.2-8.el9.noarch 49/172 2024-08-03T20:46:29.456 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : bison-3.7.4-5.el9.x86_64 50/172 2024-08-03T20:46:29.456 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 51/172 2024-08-03T20:46:29.456 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : elfutils-libelf-devel-0.191-4.el9.x86_64 52/172 2024-08-03T20:46:29.456 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : flex-2.6.4-9.el9.x86_64 53/172 2024-08-03T20:46:29.456 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 54/172 2024-08-03T20:46:29.457 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 55/172 2024-08-03T20:46:29.457 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 56/172 2024-08-03T20:46:29.457 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 57/172 2024-08-03T20:46:29.457 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 58/172 2024-08-03T20:46:29.457 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 59/172 2024-08-03T20:46:29.457 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 60/172 2024-08-03T20:46:29.457 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : libtool-2.4.6-46.el9.x86_64 61/172 2024-08-03T20:46:29.457 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 62/172 2024-08-03T20:46:29.457 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 63/172 2024-08-03T20:46:29.457 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 64/172 2024-08-03T20:46:29.457 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : lua-5.4.4-4.el9.x86_64 65/172 2024-08-03T20:46:29.457 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 66/172 2024-08-03T20:46:29.458 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 67/172 2024-08-03T20:46:29.458 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : openssl-devel-1:3.2.2-2.el9.x86_64 68/172 2024-08-03T20:46:29.458 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : postgresql-13.14-1.el9.x86_64 69/172 2024-08-03T20:46:29.458 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : postgresql-contrib-13.14-1.el9.x86_64 70/172 2024-08-03T20:46:29.458 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : postgresql-private-libs-13.14-1.el9.x86_64 71/172 2024-08-03T20:46:29.458 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : postgresql-server-13.14-1.el9.x86_64 72/172 2024-08-03T20:46:29.458 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 73/172 2024-08-03T20:46:29.458 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 74/172 2024-08-03T20:46:29.458 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 75/172 2024-08-03T20:46:29.458 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-colorama-0.4.6-3.el9.noarch 76/172 2024-08-03T20:46:29.458 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-devel-3.9.18-3.el9.x86_64 77/172 2024-08-03T20:46:29.458 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-jinja2-2.11.3-6.el9.noarch 78/172 2024-08-03T20:46:29.459 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 79/172 2024-08-03T20:46:29.459 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 80/172 2024-08-03T20:46:29.459 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 81/172 2024-08-03T20:46:29.459 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 82/172 2024-08-03T20:46:29.459 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 83/172 2024-08-03T20:46:29.459 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 84/172 2024-08-03T20:46:29.459 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 85/172 2024-08-03T20:46:29.459 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 86/172 2024-08-03T20:46:29.459 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 87/172 2024-08-03T20:46:29.459 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 88/172 2024-08-03T20:46:29.459 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 89/172 2024-08-03T20:46:29.459 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 90/172 2024-08-03T20:46:29.460 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 91/172 2024-08-03T20:46:29.460 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 92/172 2024-08-03T20:46:29.460 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 93/172 2024-08-03T20:46:29.460 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 94/172 2024-08-03T20:46:29.460 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 95/172 2024-08-03T20:46:29.460 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 96/172 2024-08-03T20:46:29.460 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 97/172 2024-08-03T20:46:29.460 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 98/172 2024-08-03T20:46:29.460 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : socat-1.7.4.1-6.el9.x86_64 99/172 2024-08-03T20:46:29.460 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 100/172 2024-08-03T20:46:29.460 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : xfsprogs-devel-6.4.0-3.el9.x86_64 101/172 2024-08-03T20:46:29.460 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 102/172 2024-08-03T20:46:29.460 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : zlib-devel-1.2.11-41.el9.x86_64 103/172 2024-08-03T20:46:29.460 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 104/172 2024-08-03T20:46:29.461 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 105/172 2024-08-03T20:46:29.461 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 106/172 2024-08-03T20:46:29.461 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 107/172 2024-08-03T20:46:29.461 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 108/172 2024-08-03T20:46:29.461 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : libarrow-9.0.0-11.el9.x86_64 109/172 2024-08-03T20:46:29.461 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : libarrow-doc-9.0.0-11.el9.noarch 110/172 2024-08-03T20:46:29.461 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 111/172 2024-08-03T20:46:29.461 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 112/172 2024-08-03T20:46:29.461 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : luarocks-3.9.2-5.el9.noarch 113/172 2024-08-03T20:46:29.461 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : parquet-libs-9.0.0-11.el9.x86_64 114/172 2024-08-03T20:46:29.461 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 115/172 2024-08-03T20:46:29.461 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 116/172 2024-08-03T20:46:29.461 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 117/172 2024-08-03T20:46:29.462 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-certifi-2023.05.07-4.el9.noarch 118/172 2024-08-03T20:46:29.462 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 119/172 2024-08-03T20:46:29.462 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 120/172 2024-08-03T20:46:29.462 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 121/172 2024-08-03T20:46:29.462 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-google-auth-1:2.32.0-1.el9.noarch 122/172 2024-08-03T20:46:29.462 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 123/172 2024-08-03T20:46:29.462 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 124/172 2024-08-03T20:46:29.462 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 125/172 2024-08-03T20:46:29.462 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 126/172 2024-08-03T20:46:29.462 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 127/172 2024-08-03T20:46:29.463 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 128/172 2024-08-03T20:46:29.463 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 129/172 2024-08-03T20:46:29.463 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 130/172 2024-08-03T20:46:29.463 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 131/172 2024-08-03T20:46:29.463 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 132/172 2024-08-03T20:46:29.463 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 133/172 2024-08-03T20:46:29.463 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 134/172 2024-08-03T20:46:29.463 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 135/172 2024-08-03T20:46:29.463 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 136/172 2024-08-03T20:46:29.463 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 137/172 2024-08-03T20:46:29.463 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 138/172 2024-08-03T20:46:29.463 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 139/172 2024-08-03T20:46:29.464 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 140/172 2024-08-03T20:46:29.464 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 141/172 2024-08-03T20:46:29.464 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-typing-extensions-4.12.2-2.el9.noarch 142/172 2024-08-03T20:46:29.464 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 143/172 2024-08-03T20:46:29.464 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 144/172 2024-08-03T20:46:29.464 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 145/172 2024-08-03T20:46:29.464 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 146/172 2024-08-03T20:46:29.464 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : re2-1:20211101-3.el9.x86_64 147/172 2024-08-03T20:46:29.464 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 148/172 2024-08-03T20:46:29.464 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : librados2-2:19.0.0-5476.g3792a0a4.el9.x86_64 149/172 2024-08-03T20:46:29.464 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 150/172 2024-08-03T20:46:29.465 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : librbd1-2:19.0.0-5476.g3792a0a4.el9.x86_64 151/172 2024-08-03T20:46:29.465 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 152/172 2024-08-03T20:46:29.465 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : NetworkManager-1:1.48.2-2.el9.x86_64 153/172 2024-08-03T20:46:29.465 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : NetworkManager-1:1.47.4-1.el9.x86_64 154/172 2024-08-03T20:46:29.465 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : NetworkManager-libnm-1:1.48.2-2.el9.x86_64 155/172 2024-08-03T20:46:29.465 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : NetworkManager-libnm-1:1.47.4-1.el9.x86_64 156/172 2024-08-03T20:46:29.465 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : NetworkManager-team-1:1.48.2-2.el9.x86_64 157/172 2024-08-03T20:46:29.465 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : NetworkManager-team-1:1.47.4-1.el9.x86_64 158/172 2024-08-03T20:46:29.465 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : NetworkManager-tui-1:1.48.2-2.el9.x86_64 159/172 2024-08-03T20:46:29.465 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : NetworkManager-tui-1:1.47.4-1.el9.x86_64 160/172 2024-08-03T20:46:29.466 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : elfutils-debuginfod-client-0.191-4.el9.x86_64 161/172 2024-08-03T20:46:29.466 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : elfutils-debuginfod-client-0.190-2.el9.x86_64 162/172 2024-08-03T20:46:29.466 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : elfutils-libelf-0.191-4.el9.x86_64 163/172 2024-08-03T20:46:29.466 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : elfutils-libelf-0.190-2.el9.x86_64 164/172 2024-08-03T20:46:29.466 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : elfutils-libs-0.191-4.el9.x86_64 165/172 2024-08-03T20:46:29.466 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : elfutils-libs-0.190-2.el9.x86_64 166/172 2024-08-03T20:46:29.466 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : openssl-1:3.2.2-2.el9.x86_64 167/172 2024-08-03T20:46:29.466 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 168/172 2024-08-03T20:46:29.466 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : openssl-libs-1:3.2.2-2.el9.x86_64 169/172 2024-08-03T20:46:29.466 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 170/172 2024-08-03T20:46:29.466 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : xfsprogs-6.4.0-3.el9.x86_64 171/172 2024-08-03T20:46:30.744 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : ceph-2:19.0.0-5476.g3792a0a4.el9.x86_64 1/172 2024-08-03T20:46:30.744 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : ceph-base-2:19.0.0-5476.g3792a0a4.el9.x86_64 2/172 2024-08-03T20:46:30.744 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : ceph-common-2:19.0.0-5476.g3792a0a4.el9.x86_64 3/172 2024-08-03T20:46:30.744 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : ceph-fuse-2:19.0.0-5476.g3792a0a4.el9.x86_64 4/172 2024-08-03T20:46:30.744 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-5476.g3792a 5/172 2024-08-03T20:46:30.744 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : ceph-mds-2:19.0.0-5476.g3792a0a4.el9.x86_64 6/172 2024-08-03T20:46:30.745 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : ceph-mgr-2:19.0.0-5476.g3792a0a4.el9.x86_64 7/172 2024-08-03T20:46:30.745 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : ceph-mon-2:19.0.0-5476.g3792a0a4.el9.x86_64 8/172 2024-08-03T20:46:30.745 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : ceph-osd-2:19.0.0-5476.g3792a0a4.el9.x86_64 9/172 2024-08-03T20:46:30.745 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : ceph-radosgw-2:19.0.0-5476.g3792a0a4.el9.x86_64 10/172 2024-08-03T20:46:30.745 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : ceph-selinux-2:19.0.0-5476.g3792a0a4.el9.x86_64 11/172 2024-08-03T20:46:30.745 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : ceph-test-2:19.0.0-5476.g3792a0a4.el9.x86_64 12/172 2024-08-03T20:46:30.745 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : cephfs-mirror-2:19.0.0-5476.g3792a0a4.el9.x86_64 13/172 2024-08-03T20:46:30.745 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : cephfs-shell-2:19.0.0-5476.g3792a0a4.el9.x86_64 14/172 2024-08-03T20:46:30.745 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : libcephfs-devel-2:19.0.0-5476.g3792a0a4.el9.x86_ 15/172 2024-08-03T20:46:30.745 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : libcephfs2-2:19.0.0-5476.g3792a0a4.el9.x86_64 16/172 2024-08-03T20:46:30.745 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : libcephsqlite-2:19.0.0-5476.g3792a0a4.el9.x86_64 17/172 2024-08-03T20:46:30.745 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : librados-devel-2:19.0.0-5476.g3792a0a4.el9.x86_6 18/172 2024-08-03T20:46:30.745 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : libradosstriper1-2:19.0.0-5476.g3792a0a4.el9.x86 19/172 2024-08-03T20:46:30.745 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : librgw2-2:19.0.0-5476.g3792a0a4.el9.x86_64 20/172 2024-08-03T20:46:30.746 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-ceph-argparse-2:19.0.0-5476.g3792a0a4.el 21/172 2024-08-03T20:46:30.746 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-ceph-common-2:19.0.0-5476.g3792a0a4.el9. 22/172 2024-08-03T20:46:30.746 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-cephfs-2:19.0.0-5476.g3792a0a4.el9.x86_6 23/172 2024-08-03T20:46:30.746 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-rados-2:19.0.0-5476.g3792a0a4.el9.x86_64 24/172 2024-08-03T20:46:30.746 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-rbd-2:19.0.0-5476.g3792a0a4.el9.x86_64 25/172 2024-08-03T20:46:30.746 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-rgw-2:19.0.0-5476.g3792a0a4.el9.x86_64 26/172 2024-08-03T20:46:30.746 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : rbd-fuse-2:19.0.0-5476.g3792a0a4.el9.x86_64 27/172 2024-08-03T20:46:30.746 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : rbd-mirror-2:19.0.0-5476.g3792a0a4.el9.x86_64 28/172 2024-08-03T20:46:30.746 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : rbd-nbd-2:19.0.0-5476.g3792a0a4.el9.x86_64 29/172 2024-08-03T20:46:30.746 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-5476.g3792a0a4. 30/172 2024-08-03T20:46:30.746 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-5476.g3792a0a4.el9.noa 31/172 2024-08-03T20:46:30.746 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-5476.g3792a0a4.el9.n 32/172 2024-08-03T20:46:30.746 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-5476.g379 33/172 2024-08-03T20:46:30.747 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-5476.g3792a0a4.el 34/172 2024-08-03T20:46:30.747 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : ceph-mgr-rook-2:19.0.0-5476.g3792a0a4.el9.noarch 35/172 2024-08-03T20:46:30.747 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-5476.g3792a0a4.e 36/172 2024-08-03T20:46:30.747 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : ceph-volume-2:19.0.0-5476.g3792a0a4.el9.noarch 37/172 2024-08-03T20:46:30.747 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : cephadm-2:19.0.0-5476.g3792a0a4.el9.noarch 38/172 2024-08-03T20:46:30.747 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : cephfs-top-2:19.0.0-5476.g3792a0a4.el9.noarch 39/172 2024-08-03T20:46:30.747 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : fuse-2.9.9-16.el9.x86_64 40/172 2024-08-03T20:46:30.747 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 41/172 2024-08-03T20:46:30.747 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 42/172 2024-08-03T20:46:30.747 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 43/172 2024-08-03T20:46:30.747 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 44/172 2024-08-03T20:46:30.747 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-ply-3.11-14.el9.noarch 45/172 2024-08-03T20:46:30.747 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 46/172 2024-08-03T20:46:30.748 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 47/172 2024-08-03T20:46:30.748 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-urllib3-1.26.5-6.el9.noarch 48/172 2024-08-03T20:46:30.748 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : automake-1.16.2-8.el9.noarch 49/172 2024-08-03T20:46:30.748 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : bison-3.7.4-5.el9.x86_64 50/172 2024-08-03T20:46:30.748 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 51/172 2024-08-03T20:46:30.748 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : elfutils-libelf-devel-0.191-4.el9.x86_64 52/172 2024-08-03T20:46:30.748 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : flex-2.6.4-9.el9.x86_64 53/172 2024-08-03T20:46:30.748 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 54/172 2024-08-03T20:46:30.748 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 55/172 2024-08-03T20:46:30.748 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 56/172 2024-08-03T20:46:30.748 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 57/172 2024-08-03T20:46:30.748 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 58/172 2024-08-03T20:46:30.748 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 59/172 2024-08-03T20:46:30.749 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 60/172 2024-08-03T20:46:30.749 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : libtool-2.4.6-46.el9.x86_64 61/172 2024-08-03T20:46:30.749 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 62/172 2024-08-03T20:46:30.749 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 63/172 2024-08-03T20:46:30.749 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 64/172 2024-08-03T20:46:30.749 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : lua-5.4.4-4.el9.x86_64 65/172 2024-08-03T20:46:30.749 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 66/172 2024-08-03T20:46:30.749 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 67/172 2024-08-03T20:46:30.749 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : openssl-devel-1:3.2.2-2.el9.x86_64 68/172 2024-08-03T20:46:30.749 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : postgresql-13.14-1.el9.x86_64 69/172 2024-08-03T20:46:30.749 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : postgresql-contrib-13.14-1.el9.x86_64 70/172 2024-08-03T20:46:30.749 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : postgresql-private-libs-13.14-1.el9.x86_64 71/172 2024-08-03T20:46:30.749 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : postgresql-server-13.14-1.el9.x86_64 72/172 2024-08-03T20:46:30.749 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 73/172 2024-08-03T20:46:30.749 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 74/172 2024-08-03T20:46:30.750 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 75/172 2024-08-03T20:46:30.750 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-colorama-0.4.6-3.el9.noarch 76/172 2024-08-03T20:46:30.750 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-devel-3.9.18-3.el9.x86_64 77/172 2024-08-03T20:46:30.750 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-jinja2-2.11.3-6.el9.noarch 78/172 2024-08-03T20:46:30.750 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 79/172 2024-08-03T20:46:30.750 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 80/172 2024-08-03T20:46:30.750 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 81/172 2024-08-03T20:46:30.750 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 82/172 2024-08-03T20:46:30.750 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 83/172 2024-08-03T20:46:30.750 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 84/172 2024-08-03T20:46:30.750 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 85/172 2024-08-03T20:46:30.750 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 86/172 2024-08-03T20:46:30.750 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 87/172 2024-08-03T20:46:30.751 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 88/172 2024-08-03T20:46:30.751 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 89/172 2024-08-03T20:46:30.751 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 90/172 2024-08-03T20:46:30.751 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 91/172 2024-08-03T20:46:30.751 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 92/172 2024-08-03T20:46:30.751 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 93/172 2024-08-03T20:46:30.751 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 94/172 2024-08-03T20:46:30.751 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 95/172 2024-08-03T20:46:30.751 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 96/172 2024-08-03T20:46:30.751 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 97/172 2024-08-03T20:46:30.751 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 98/172 2024-08-03T20:46:30.751 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : socat-1.7.4.1-6.el9.x86_64 99/172 2024-08-03T20:46:30.751 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 100/172 2024-08-03T20:46:30.752 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : xfsprogs-devel-6.4.0-3.el9.x86_64 101/172 2024-08-03T20:46:30.752 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 102/172 2024-08-03T20:46:30.752 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : zlib-devel-1.2.11-41.el9.x86_64 103/172 2024-08-03T20:46:30.752 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 104/172 2024-08-03T20:46:30.752 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 105/172 2024-08-03T20:46:30.752 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 106/172 2024-08-03T20:46:30.752 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 107/172 2024-08-03T20:46:30.752 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 108/172 2024-08-03T20:46:30.752 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : libarrow-9.0.0-11.el9.x86_64 109/172 2024-08-03T20:46:30.752 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : libarrow-doc-9.0.0-11.el9.noarch 110/172 2024-08-03T20:46:30.752 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 111/172 2024-08-03T20:46:30.752 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 112/172 2024-08-03T20:46:30.752 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : luarocks-3.9.2-5.el9.noarch 113/172 2024-08-03T20:46:30.753 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : parquet-libs-9.0.0-11.el9.x86_64 114/172 2024-08-03T20:46:30.753 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 115/172 2024-08-03T20:46:30.753 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 116/172 2024-08-03T20:46:30.753 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 117/172 2024-08-03T20:46:30.753 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-certifi-2023.05.07-4.el9.noarch 118/172 2024-08-03T20:46:30.753 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 119/172 2024-08-03T20:46:30.753 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 120/172 2024-08-03T20:46:30.753 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 121/172 2024-08-03T20:46:30.753 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-google-auth-1:2.32.0-1.el9.noarch 122/172 2024-08-03T20:46:30.753 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 123/172 2024-08-03T20:46:30.753 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 124/172 2024-08-03T20:46:30.753 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 125/172 2024-08-03T20:46:30.753 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 126/172 2024-08-03T20:46:30.754 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 127/172 2024-08-03T20:46:30.754 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 128/172 2024-08-03T20:46:30.754 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 129/172 2024-08-03T20:46:30.754 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 130/172 2024-08-03T20:46:30.754 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 131/172 2024-08-03T20:46:30.754 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 132/172 2024-08-03T20:46:30.754 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 133/172 2024-08-03T20:46:30.754 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 134/172 2024-08-03T20:46:30.754 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 135/172 2024-08-03T20:46:30.754 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 136/172 2024-08-03T20:46:30.754 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 137/172 2024-08-03T20:46:30.754 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 138/172 2024-08-03T20:46:30.755 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 139/172 2024-08-03T20:46:30.755 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 140/172 2024-08-03T20:46:30.755 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 141/172 2024-08-03T20:46:30.755 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-typing-extensions-4.12.2-2.el9.noarch 142/172 2024-08-03T20:46:30.755 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 143/172 2024-08-03T20:46:30.755 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 144/172 2024-08-03T20:46:30.755 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 145/172 2024-08-03T20:46:30.755 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 146/172 2024-08-03T20:46:30.755 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : re2-1:20211101-3.el9.x86_64 147/172 2024-08-03T20:46:30.755 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 148/172 2024-08-03T20:46:30.755 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : librados2-2:19.0.0-5476.g3792a0a4.el9.x86_64 149/172 2024-08-03T20:46:30.755 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 150/172 2024-08-03T20:46:30.755 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : librbd1-2:19.0.0-5476.g3792a0a4.el9.x86_64 151/172 2024-08-03T20:46:30.755 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 152/172 2024-08-03T20:46:30.756 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : NetworkManager-1:1.48.2-2.el9.x86_64 153/172 2024-08-03T20:46:30.756 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : NetworkManager-1:1.47.4-1.el9.x86_64 154/172 2024-08-03T20:46:30.756 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : NetworkManager-libnm-1:1.48.2-2.el9.x86_64 155/172 2024-08-03T20:46:30.756 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : NetworkManager-libnm-1:1.47.4-1.el9.x86_64 156/172 2024-08-03T20:46:30.756 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : NetworkManager-team-1:1.48.2-2.el9.x86_64 157/172 2024-08-03T20:46:30.756 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : NetworkManager-team-1:1.47.4-1.el9.x86_64 158/172 2024-08-03T20:46:30.756 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : NetworkManager-tui-1:1.48.2-2.el9.x86_64 159/172 2024-08-03T20:46:30.756 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : NetworkManager-tui-1:1.47.4-1.el9.x86_64 160/172 2024-08-03T20:46:30.756 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : elfutils-debuginfod-client-0.191-4.el9.x86_64 161/172 2024-08-03T20:46:30.756 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : elfutils-debuginfod-client-0.190-2.el9.x86_64 162/172 2024-08-03T20:46:30.756 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : elfutils-libelf-0.191-4.el9.x86_64 163/172 2024-08-03T20:46:30.756 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : elfutils-libelf-0.190-2.el9.x86_64 164/172 2024-08-03T20:46:30.756 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : elfutils-libs-0.191-4.el9.x86_64 165/172 2024-08-03T20:46:30.756 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : elfutils-libs-0.190-2.el9.x86_64 166/172 2024-08-03T20:46:30.757 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : openssl-1:3.2.2-2.el9.x86_64 167/172 2024-08-03T20:46:30.757 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 168/172 2024-08-03T20:46:30.757 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : openssl-libs-1:3.2.2-2.el9.x86_64 169/172 2024-08-03T20:46:30.757 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 170/172 2024-08-03T20:46:30.757 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : xfsprogs-6.4.0-3.el9.x86_64 171/172 2024-08-03T20:46:31.766 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : xfsprogs-6.3.0-1.el9.x86_64 172/172 2024-08-03T20:46:31.767 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-08-03T20:46:31.767 INFO:teuthology.orchestra.run.smithi118.stdout:Upgraded: 2024-08-03T20:46:31.767 INFO:teuthology.orchestra.run.smithi118.stdout: NetworkManager-1:1.48.2-2.el9.x86_64 2024-08-03T20:46:31.767 INFO:teuthology.orchestra.run.smithi118.stdout: NetworkManager-libnm-1:1.48.2-2.el9.x86_64 2024-08-03T20:46:31.767 INFO:teuthology.orchestra.run.smithi118.stdout: NetworkManager-team-1:1.48.2-2.el9.x86_64 2024-08-03T20:46:31.767 INFO:teuthology.orchestra.run.smithi118.stdout: NetworkManager-tui-1:1.48.2-2.el9.x86_64 2024-08-03T20:46:31.767 INFO:teuthology.orchestra.run.smithi118.stdout: elfutils-debuginfod-client-0.191-4.el9.x86_64 2024-08-03T20:46:31.767 INFO:teuthology.orchestra.run.smithi118.stdout: elfutils-libelf-0.191-4.el9.x86_64 2024-08-03T20:46:31.767 INFO:teuthology.orchestra.run.smithi118.stdout: elfutils-libs-0.191-4.el9.x86_64 2024-08-03T20:46:31.767 INFO:teuthology.orchestra.run.smithi118.stdout: librados2-2:19.0.0-5476.g3792a0a4.el9.x86_64 2024-08-03T20:46:31.767 INFO:teuthology.orchestra.run.smithi118.stdout: librbd1-2:19.0.0-5476.g3792a0a4.el9.x86_64 2024-08-03T20:46:31.768 INFO:teuthology.orchestra.run.smithi118.stdout: openssl-1:3.2.2-2.el9.x86_64 2024-08-03T20:46:31.768 INFO:teuthology.orchestra.run.smithi118.stdout: openssl-libs-1:3.2.2-2.el9.x86_64 2024-08-03T20:46:31.768 INFO:teuthology.orchestra.run.smithi118.stdout: xfsprogs-6.4.0-3.el9.x86_64 2024-08-03T20:46:31.768 INFO:teuthology.orchestra.run.smithi118.stdout:Installed: 2024-08-03T20:46:31.768 INFO:teuthology.orchestra.run.smithi118.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-08-03T20:46:31.768 INFO:teuthology.orchestra.run.smithi118.stdout: automake-1.16.2-8.el9.noarch 2024-08-03T20:46:31.768 INFO:teuthology.orchestra.run.smithi118.stdout: bison-3.7.4-5.el9.x86_64 2024-08-03T20:46:31.768 INFO:teuthology.orchestra.run.smithi118.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-08-03T20:46:31.768 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-2:19.0.0-5476.g3792a0a4.el9.x86_64 2024-08-03T20:46:31.768 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-base-2:19.0.0-5476.g3792a0a4.el9.x86_64 2024-08-03T20:46:31.768 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-common-2:19.0.0-5476.g3792a0a4.el9.x86_64 2024-08-03T20:46:31.768 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-fuse-2:19.0.0-5476.g3792a0a4.el9.x86_64 2024-08-03T20:46:31.768 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-grafana-dashboards-2:19.0.0-5476.g3792a0a4.el9.noarch 2024-08-03T20:46:31.769 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-immutable-object-cache-2:19.0.0-5476.g3792a0a4.el9.x86_64 2024-08-03T20:46:31.769 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-mds-2:19.0.0-5476.g3792a0a4.el9.x86_64 2024-08-03T20:46:31.769 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-mgr-2:19.0.0-5476.g3792a0a4.el9.x86_64 2024-08-03T20:46:31.769 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-mgr-cephadm-2:19.0.0-5476.g3792a0a4.el9.noarch 2024-08-03T20:46:31.769 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-mgr-dashboard-2:19.0.0-5476.g3792a0a4.el9.noarch 2024-08-03T20:46:31.769 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-mgr-diskprediction-local-2:19.0.0-5476.g3792a0a4.el9.noarch 2024-08-03T20:46:31.769 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-mgr-modules-core-2:19.0.0-5476.g3792a0a4.el9.noarch 2024-08-03T20:46:31.769 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-mgr-rook-2:19.0.0-5476.g3792a0a4.el9.noarch 2024-08-03T20:46:31.769 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-mon-2:19.0.0-5476.g3792a0a4.el9.x86_64 2024-08-03T20:46:31.769 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-osd-2:19.0.0-5476.g3792a0a4.el9.x86_64 2024-08-03T20:46:31.769 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-prometheus-alerts-2:19.0.0-5476.g3792a0a4.el9.noarch 2024-08-03T20:46:31.769 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-radosgw-2:19.0.0-5476.g3792a0a4.el9.x86_64 2024-08-03T20:46:31.769 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-selinux-2:19.0.0-5476.g3792a0a4.el9.x86_64 2024-08-03T20:46:31.769 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-test-2:19.0.0-5476.g3792a0a4.el9.x86_64 2024-08-03T20:46:31.770 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-volume-2:19.0.0-5476.g3792a0a4.el9.noarch 2024-08-03T20:46:31.770 INFO:teuthology.orchestra.run.smithi118.stdout: cephadm-2:19.0.0-5476.g3792a0a4.el9.noarch 2024-08-03T20:46:31.770 INFO:teuthology.orchestra.run.smithi118.stdout: cephfs-mirror-2:19.0.0-5476.g3792a0a4.el9.x86_64 2024-08-03T20:46:31.770 INFO:teuthology.orchestra.run.smithi118.stdout: cephfs-shell-2:19.0.0-5476.g3792a0a4.el9.x86_64 2024-08-03T20:46:31.770 INFO:teuthology.orchestra.run.smithi118.stdout: cephfs-top-2:19.0.0-5476.g3792a0a4.el9.noarch 2024-08-03T20:46:31.770 INFO:teuthology.orchestra.run.smithi118.stdout: elfutils-libelf-devel-0.191-4.el9.x86_64 2024-08-03T20:46:31.770 INFO:teuthology.orchestra.run.smithi118.stdout: flex-2.6.4-9.el9.x86_64 2024-08-03T20:46:31.770 INFO:teuthology.orchestra.run.smithi118.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-08-03T20:46:31.770 INFO:teuthology.orchestra.run.smithi118.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-08-03T20:46:31.770 INFO:teuthology.orchestra.run.smithi118.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-08-03T20:46:31.770 INFO:teuthology.orchestra.run.smithi118.stdout: fuse-2.9.9-16.el9.x86_64 2024-08-03T20:46:31.770 INFO:teuthology.orchestra.run.smithi118.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-08-03T20:46:31.771 INFO:teuthology.orchestra.run.smithi118.stdout: grpc-data-1.46.7-10.el9.noarch 2024-08-03T20:46:31.771 INFO:teuthology.orchestra.run.smithi118.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-08-03T20:46:31.771 INFO:teuthology.orchestra.run.smithi118.stdout: libarrow-9.0.0-11.el9.x86_64 2024-08-03T20:46:31.771 INFO:teuthology.orchestra.run.smithi118.stdout: libarrow-doc-9.0.0-11.el9.noarch 2024-08-03T20:46:31.771 INFO:teuthology.orchestra.run.smithi118.stdout: libcephfs-devel-2:19.0.0-5476.g3792a0a4.el9.x86_64 2024-08-03T20:46:31.771 INFO:teuthology.orchestra.run.smithi118.stdout: libcephfs2-2:19.0.0-5476.g3792a0a4.el9.x86_64 2024-08-03T20:46:31.771 INFO:teuthology.orchestra.run.smithi118.stdout: libcephsqlite-2:19.0.0-5476.g3792a0a4.el9.x86_64 2024-08-03T20:46:31.771 INFO:teuthology.orchestra.run.smithi118.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-08-03T20:46:31.771 INFO:teuthology.orchestra.run.smithi118.stdout: liboath-2.6.7-2.el9.x86_64 2024-08-03T20:46:31.771 INFO:teuthology.orchestra.run.smithi118.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-08-03T20:46:31.771 INFO:teuthology.orchestra.run.smithi118.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-08-03T20:46:31.771 INFO:teuthology.orchestra.run.smithi118.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-08-03T20:46:31.771 INFO:teuthology.orchestra.run.smithi118.stdout: librados-devel-2:19.0.0-5476.g3792a0a4.el9.x86_64 2024-08-03T20:46:31.771 INFO:teuthology.orchestra.run.smithi118.stdout: libradosstriper1-2:19.0.0-5476.g3792a0a4.el9.x86_64 2024-08-03T20:46:31.772 INFO:teuthology.orchestra.run.smithi118.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-08-03T20:46:31.772 INFO:teuthology.orchestra.run.smithi118.stdout: librgw2-2:19.0.0-5476.g3792a0a4.el9.x86_64 2024-08-03T20:46:31.772 INFO:teuthology.orchestra.run.smithi118.stdout: libtool-2.4.6-46.el9.x86_64 2024-08-03T20:46:31.772 INFO:teuthology.orchestra.run.smithi118.stdout: libunwind-1.6.2-1.el9.x86_64 2024-08-03T20:46:31.772 INFO:teuthology.orchestra.run.smithi118.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-08-03T20:46:31.772 INFO:teuthology.orchestra.run.smithi118.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-08-03T20:46:31.772 INFO:teuthology.orchestra.run.smithi118.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-08-03T20:46:31.772 INFO:teuthology.orchestra.run.smithi118.stdout: lua-5.4.4-4.el9.x86_64 2024-08-03T20:46:31.772 INFO:teuthology.orchestra.run.smithi118.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-08-03T20:46:31.772 INFO:teuthology.orchestra.run.smithi118.stdout: luarocks-3.9.2-5.el9.noarch 2024-08-03T20:46:31.772 INFO:teuthology.orchestra.run.smithi118.stdout: openblas-0.3.21-2.el9.x86_64 2024-08-03T20:46:31.772 INFO:teuthology.orchestra.run.smithi118.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-08-03T20:46:31.772 INFO:teuthology.orchestra.run.smithi118.stdout: openssl-devel-1:3.2.2-2.el9.x86_64 2024-08-03T20:46:31.773 INFO:teuthology.orchestra.run.smithi118.stdout: parquet-libs-9.0.0-11.el9.x86_64 2024-08-03T20:46:31.773 INFO:teuthology.orchestra.run.smithi118.stdout: postgresql-13.14-1.el9.x86_64 2024-08-03T20:46:31.773 INFO:teuthology.orchestra.run.smithi118.stdout: postgresql-contrib-13.14-1.el9.x86_64 2024-08-03T20:46:31.773 INFO:teuthology.orchestra.run.smithi118.stdout: postgresql-private-libs-13.14-1.el9.x86_64 2024-08-03T20:46:31.773 INFO:teuthology.orchestra.run.smithi118.stdout: postgresql-server-13.14-1.el9.x86_64 2024-08-03T20:46:31.773 INFO:teuthology.orchestra.run.smithi118.stdout: protobuf-3.14.0-13.el9.x86_64 2024-08-03T20:46:31.773 INFO:teuthology.orchestra.run.smithi118.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-08-03T20:46:31.773 INFO:teuthology.orchestra.run.smithi118.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-08-03T20:46:31.773 INFO:teuthology.orchestra.run.smithi118.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-08-03T20:46:31.773 INFO:teuthology.orchestra.run.smithi118.stdout: python3-babel-2.9.1-2.el9.noarch 2024-08-03T20:46:31.773 INFO:teuthology.orchestra.run.smithi118.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-08-03T20:46:31.773 INFO:teuthology.orchestra.run.smithi118.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-08-03T20:46:31.774 INFO:teuthology.orchestra.run.smithi118.stdout: python3-ceph-argparse-2:19.0.0-5476.g3792a0a4.el9.x86_64 2024-08-03T20:46:31.774 INFO:teuthology.orchestra.run.smithi118.stdout: python3-ceph-common-2:19.0.0-5476.g3792a0a4.el9.x86_64 2024-08-03T20:46:31.774 INFO:teuthology.orchestra.run.smithi118.stdout: python3-cephfs-2:19.0.0-5476.g3792a0a4.el9.x86_64 2024-08-03T20:46:31.774 INFO:teuthology.orchestra.run.smithi118.stdout: python3-certifi-2023.05.07-4.el9.noarch 2024-08-03T20:46:31.774 INFO:teuthology.orchestra.run.smithi118.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-08-03T20:46:31.774 INFO:teuthology.orchestra.run.smithi118.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-08-03T20:46:31.774 INFO:teuthology.orchestra.run.smithi118.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-08-03T20:46:31.774 INFO:teuthology.orchestra.run.smithi118.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-08-03T20:46:31.774 INFO:teuthology.orchestra.run.smithi118.stdout: python3-colorama-0.4.6-3.el9.noarch 2024-08-03T20:46:31.774 INFO:teuthology.orchestra.run.smithi118.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-08-03T20:46:31.774 INFO:teuthology.orchestra.run.smithi118.stdout: python3-devel-3.9.18-3.el9.x86_64 2024-08-03T20:46:31.774 INFO:teuthology.orchestra.run.smithi118.stdout: python3-google-auth-1:2.32.0-1.el9.noarch 2024-08-03T20:46:31.774 INFO:teuthology.orchestra.run.smithi118.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-08-03T20:46:31.775 INFO:teuthology.orchestra.run.smithi118.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-08-03T20:46:31.775 INFO:teuthology.orchestra.run.smithi118.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-08-03T20:46:31.775 INFO:teuthology.orchestra.run.smithi118.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-08-03T20:46:31.775 INFO:teuthology.orchestra.run.smithi118.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-08-03T20:46:31.775 INFO:teuthology.orchestra.run.smithi118.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-08-03T20:46:31.775 INFO:teuthology.orchestra.run.smithi118.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-08-03T20:46:31.775 INFO:teuthology.orchestra.run.smithi118.stdout: python3-jinja2-2.11.3-6.el9.noarch 2024-08-03T20:46:31.775 INFO:teuthology.orchestra.run.smithi118.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-08-03T20:46:31.775 INFO:teuthology.orchestra.run.smithi118.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-08-03T20:46:31.775 INFO:teuthology.orchestra.run.smithi118.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-08-03T20:46:31.775 INFO:teuthology.orchestra.run.smithi118.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-08-03T20:46:31.775 INFO:teuthology.orchestra.run.smithi118.stdout: python3-mako-1.1.4-6.el9.noarch 2024-08-03T20:46:31.776 INFO:teuthology.orchestra.run.smithi118.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-08-03T20:46:31.776 INFO:teuthology.orchestra.run.smithi118.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-08-03T20:46:31.776 INFO:teuthology.orchestra.run.smithi118.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-08-03T20:46:31.776 INFO:teuthology.orchestra.run.smithi118.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-08-03T20:46:31.776 INFO:teuthology.orchestra.run.smithi118.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-08-03T20:46:31.776 INFO:teuthology.orchestra.run.smithi118.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-08-03T20:46:31.776 INFO:teuthology.orchestra.run.smithi118.stdout: python3-packaging-20.9-5.el9.noarch 2024-08-03T20:46:31.776 INFO:teuthology.orchestra.run.smithi118.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-08-03T20:46:31.776 INFO:teuthology.orchestra.run.smithi118.stdout: python3-ply-3.11-14.el9.noarch 2024-08-03T20:46:31.776 INFO:teuthology.orchestra.run.smithi118.stdout: python3-portend-3.1.0-2.el9.noarch 2024-08-03T20:46:31.776 INFO:teuthology.orchestra.run.smithi118.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-08-03T20:46:31.776 INFO:teuthology.orchestra.run.smithi118.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-08-03T20:46:31.776 INFO:teuthology.orchestra.run.smithi118.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-08-03T20:46:31.777 INFO:teuthology.orchestra.run.smithi118.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-08-03T20:46:31.777 INFO:teuthology.orchestra.run.smithi118.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-08-03T20:46:31.777 INFO:teuthology.orchestra.run.smithi118.stdout: python3-pycparser-2.20-6.el9.noarch 2024-08-03T20:46:31.777 INFO:teuthology.orchestra.run.smithi118.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-08-03T20:46:31.777 INFO:teuthology.orchestra.run.smithi118.stdout: python3-pytz-2021.1-5.el9.noarch 2024-08-03T20:46:31.777 INFO:teuthology.orchestra.run.smithi118.stdout: python3-rados-2:19.0.0-5476.g3792a0a4.el9.x86_64 2024-08-03T20:46:31.777 INFO:teuthology.orchestra.run.smithi118.stdout: python3-rbd-2:19.0.0-5476.g3792a0a4.el9.x86_64 2024-08-03T20:46:31.777 INFO:teuthology.orchestra.run.smithi118.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-08-03T20:46:31.777 INFO:teuthology.orchestra.run.smithi118.stdout: python3-requests-2.25.1-8.el9.noarch 2024-08-03T20:46:31.777 INFO:teuthology.orchestra.run.smithi118.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-08-03T20:46:31.777 INFO:teuthology.orchestra.run.smithi118.stdout: python3-rgw-2:19.0.0-5476.g3792a0a4.el9.x86_64 2024-08-03T20:46:31.777 INFO:teuthology.orchestra.run.smithi118.stdout: python3-routes-2.5.1-5.el9.noarch 2024-08-03T20:46:31.778 INFO:teuthology.orchestra.run.smithi118.stdout: python3-rsa-4.9-2.el9.noarch 2024-08-03T20:46:31.778 INFO:teuthology.orchestra.run.smithi118.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-08-03T20:46:31.778 INFO:teuthology.orchestra.run.smithi118.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-08-03T20:46:31.778 INFO:teuthology.orchestra.run.smithi118.stdout: python3-toml-0.10.2-6.el9.noarch 2024-08-03T20:46:31.778 INFO:teuthology.orchestra.run.smithi118.stdout: python3-typing-extensions-4.12.2-2.el9.noarch 2024-08-03T20:46:31.778 INFO:teuthology.orchestra.run.smithi118.stdout: python3-urllib3-1.26.5-6.el9.noarch 2024-08-03T20:46:31.778 INFO:teuthology.orchestra.run.smithi118.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-08-03T20:46:31.778 INFO:teuthology.orchestra.run.smithi118.stdout: python3-webob-1.8.7-6.el9.noarch 2024-08-03T20:46:31.778 INFO:teuthology.orchestra.run.smithi118.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-08-03T20:46:31.778 INFO:teuthology.orchestra.run.smithi118.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-08-03T20:46:31.778 INFO:teuthology.orchestra.run.smithi118.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-08-03T20:46:31.778 INFO:teuthology.orchestra.run.smithi118.stdout: qatlib-24.02.0-1.el9.x86_64 2024-08-03T20:46:31.778 INFO:teuthology.orchestra.run.smithi118.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-08-03T20:46:31.779 INFO:teuthology.orchestra.run.smithi118.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-08-03T20:46:31.779 INFO:teuthology.orchestra.run.smithi118.stdout: rbd-fuse-2:19.0.0-5476.g3792a0a4.el9.x86_64 2024-08-03T20:46:31.779 INFO:teuthology.orchestra.run.smithi118.stdout: rbd-mirror-2:19.0.0-5476.g3792a0a4.el9.x86_64 2024-08-03T20:46:31.779 INFO:teuthology.orchestra.run.smithi118.stdout: rbd-nbd-2:19.0.0-5476.g3792a0a4.el9.x86_64 2024-08-03T20:46:31.779 INFO:teuthology.orchestra.run.smithi118.stdout: re2-1:20211101-3.el9.x86_64 2024-08-03T20:46:31.779 INFO:teuthology.orchestra.run.smithi118.stdout: socat-1.7.4.1-6.el9.x86_64 2024-08-03T20:46:31.779 INFO:teuthology.orchestra.run.smithi118.stdout: thrift-0.15.0-2.el9.x86_64 2024-08-03T20:46:31.779 INFO:teuthology.orchestra.run.smithi118.stdout: uuid-1.6.2-55.el9.x86_64 2024-08-03T20:46:31.779 INFO:teuthology.orchestra.run.smithi118.stdout: xfsprogs-devel-6.4.0-3.el9.x86_64 2024-08-03T20:46:31.779 INFO:teuthology.orchestra.run.smithi118.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-08-03T20:46:31.779 INFO:teuthology.orchestra.run.smithi118.stdout: zlib-devel-1.2.11-41.el9.x86_64 2024-08-03T20:46:31.779 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-08-03T20:46:31.779 INFO:teuthology.orchestra.run.smithi118.stdout:Complete! 2024-08-03T20:46:32.476 DEBUG:teuthology.parallel:result is None 2024-08-03T20:46:33.122 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : xfsprogs-6.3.0-1.el9.x86_64 172/172 2024-08-03T20:46:33.123 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-03T20:46:33.123 INFO:teuthology.orchestra.run.smithi032.stdout:Upgraded: 2024-08-03T20:46:33.123 INFO:teuthology.orchestra.run.smithi032.stdout: NetworkManager-1:1.48.2-2.el9.x86_64 2024-08-03T20:46:33.123 INFO:teuthology.orchestra.run.smithi032.stdout: NetworkManager-libnm-1:1.48.2-2.el9.x86_64 2024-08-03T20:46:33.123 INFO:teuthology.orchestra.run.smithi032.stdout: NetworkManager-team-1:1.48.2-2.el9.x86_64 2024-08-03T20:46:33.123 INFO:teuthology.orchestra.run.smithi032.stdout: NetworkManager-tui-1:1.48.2-2.el9.x86_64 2024-08-03T20:46:33.123 INFO:teuthology.orchestra.run.smithi032.stdout: elfutils-debuginfod-client-0.191-4.el9.x86_64 2024-08-03T20:46:33.123 INFO:teuthology.orchestra.run.smithi032.stdout: elfutils-libelf-0.191-4.el9.x86_64 2024-08-03T20:46:33.123 INFO:teuthology.orchestra.run.smithi032.stdout: elfutils-libs-0.191-4.el9.x86_64 2024-08-03T20:46:33.123 INFO:teuthology.orchestra.run.smithi032.stdout: librados2-2:19.0.0-5476.g3792a0a4.el9.x86_64 2024-08-03T20:46:33.123 INFO:teuthology.orchestra.run.smithi032.stdout: librbd1-2:19.0.0-5476.g3792a0a4.el9.x86_64 2024-08-03T20:46:33.123 INFO:teuthology.orchestra.run.smithi032.stdout: openssl-1:3.2.2-2.el9.x86_64 2024-08-03T20:46:33.123 INFO:teuthology.orchestra.run.smithi032.stdout: openssl-libs-1:3.2.2-2.el9.x86_64 2024-08-03T20:46:33.124 INFO:teuthology.orchestra.run.smithi032.stdout: xfsprogs-6.4.0-3.el9.x86_64 2024-08-03T20:46:33.124 INFO:teuthology.orchestra.run.smithi032.stdout:Installed: 2024-08-03T20:46:33.124 INFO:teuthology.orchestra.run.smithi032.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-08-03T20:46:33.124 INFO:teuthology.orchestra.run.smithi032.stdout: automake-1.16.2-8.el9.noarch 2024-08-03T20:46:33.124 INFO:teuthology.orchestra.run.smithi032.stdout: bison-3.7.4-5.el9.x86_64 2024-08-03T20:46:33.124 INFO:teuthology.orchestra.run.smithi032.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-08-03T20:46:33.124 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-2:19.0.0-5476.g3792a0a4.el9.x86_64 2024-08-03T20:46:33.124 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-base-2:19.0.0-5476.g3792a0a4.el9.x86_64 2024-08-03T20:46:33.124 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-common-2:19.0.0-5476.g3792a0a4.el9.x86_64 2024-08-03T20:46:33.124 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-fuse-2:19.0.0-5476.g3792a0a4.el9.x86_64 2024-08-03T20:46:33.124 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-grafana-dashboards-2:19.0.0-5476.g3792a0a4.el9.noarch 2024-08-03T20:46:33.124 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-immutable-object-cache-2:19.0.0-5476.g3792a0a4.el9.x86_64 2024-08-03T20:46:33.124 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-mds-2:19.0.0-5476.g3792a0a4.el9.x86_64 2024-08-03T20:46:33.125 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-mgr-2:19.0.0-5476.g3792a0a4.el9.x86_64 2024-08-03T20:46:33.125 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-mgr-cephadm-2:19.0.0-5476.g3792a0a4.el9.noarch 2024-08-03T20:46:33.125 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-mgr-dashboard-2:19.0.0-5476.g3792a0a4.el9.noarch 2024-08-03T20:46:33.125 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-mgr-diskprediction-local-2:19.0.0-5476.g3792a0a4.el9.noarch 2024-08-03T20:46:33.125 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-mgr-modules-core-2:19.0.0-5476.g3792a0a4.el9.noarch 2024-08-03T20:46:33.125 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-mgr-rook-2:19.0.0-5476.g3792a0a4.el9.noarch 2024-08-03T20:46:33.125 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-mon-2:19.0.0-5476.g3792a0a4.el9.x86_64 2024-08-03T20:46:33.125 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-osd-2:19.0.0-5476.g3792a0a4.el9.x86_64 2024-08-03T20:46:33.125 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-prometheus-alerts-2:19.0.0-5476.g3792a0a4.el9.noarch 2024-08-03T20:46:33.125 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-radosgw-2:19.0.0-5476.g3792a0a4.el9.x86_64 2024-08-03T20:46:33.125 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-selinux-2:19.0.0-5476.g3792a0a4.el9.x86_64 2024-08-03T20:46:33.125 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-test-2:19.0.0-5476.g3792a0a4.el9.x86_64 2024-08-03T20:46:33.125 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-volume-2:19.0.0-5476.g3792a0a4.el9.noarch 2024-08-03T20:46:33.126 INFO:teuthology.orchestra.run.smithi032.stdout: cephadm-2:19.0.0-5476.g3792a0a4.el9.noarch 2024-08-03T20:46:33.126 INFO:teuthology.orchestra.run.smithi032.stdout: cephfs-mirror-2:19.0.0-5476.g3792a0a4.el9.x86_64 2024-08-03T20:46:33.126 INFO:teuthology.orchestra.run.smithi032.stdout: cephfs-shell-2:19.0.0-5476.g3792a0a4.el9.x86_64 2024-08-03T20:46:33.126 INFO:teuthology.orchestra.run.smithi032.stdout: cephfs-top-2:19.0.0-5476.g3792a0a4.el9.noarch 2024-08-03T20:46:33.126 INFO:teuthology.orchestra.run.smithi032.stdout: elfutils-libelf-devel-0.191-4.el9.x86_64 2024-08-03T20:46:33.126 INFO:teuthology.orchestra.run.smithi032.stdout: flex-2.6.4-9.el9.x86_64 2024-08-03T20:46:33.126 INFO:teuthology.orchestra.run.smithi032.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-08-03T20:46:33.126 INFO:teuthology.orchestra.run.smithi032.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-08-03T20:46:33.126 INFO:teuthology.orchestra.run.smithi032.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-08-03T20:46:33.126 INFO:teuthology.orchestra.run.smithi032.stdout: fuse-2.9.9-16.el9.x86_64 2024-08-03T20:46:33.126 INFO:teuthology.orchestra.run.smithi032.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-08-03T20:46:33.126 INFO:teuthology.orchestra.run.smithi032.stdout: grpc-data-1.46.7-10.el9.noarch 2024-08-03T20:46:33.126 INFO:teuthology.orchestra.run.smithi032.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-08-03T20:46:33.127 INFO:teuthology.orchestra.run.smithi032.stdout: libarrow-9.0.0-11.el9.x86_64 2024-08-03T20:46:33.127 INFO:teuthology.orchestra.run.smithi032.stdout: libarrow-doc-9.0.0-11.el9.noarch 2024-08-03T20:46:33.127 INFO:teuthology.orchestra.run.smithi032.stdout: libcephfs-devel-2:19.0.0-5476.g3792a0a4.el9.x86_64 2024-08-03T20:46:33.127 INFO:teuthology.orchestra.run.smithi032.stdout: libcephfs2-2:19.0.0-5476.g3792a0a4.el9.x86_64 2024-08-03T20:46:33.127 INFO:teuthology.orchestra.run.smithi032.stdout: libcephsqlite-2:19.0.0-5476.g3792a0a4.el9.x86_64 2024-08-03T20:46:33.127 INFO:teuthology.orchestra.run.smithi032.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-08-03T20:46:33.127 INFO:teuthology.orchestra.run.smithi032.stdout: liboath-2.6.7-2.el9.x86_64 2024-08-03T20:46:33.127 INFO:teuthology.orchestra.run.smithi032.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-08-03T20:46:33.127 INFO:teuthology.orchestra.run.smithi032.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-08-03T20:46:33.127 INFO:teuthology.orchestra.run.smithi032.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-08-03T20:46:33.127 INFO:teuthology.orchestra.run.smithi032.stdout: librados-devel-2:19.0.0-5476.g3792a0a4.el9.x86_64 2024-08-03T20:46:33.127 INFO:teuthology.orchestra.run.smithi032.stdout: libradosstriper1-2:19.0.0-5476.g3792a0a4.el9.x86_64 2024-08-03T20:46:33.128 INFO:teuthology.orchestra.run.smithi032.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-08-03T20:46:33.128 INFO:teuthology.orchestra.run.smithi032.stdout: librgw2-2:19.0.0-5476.g3792a0a4.el9.x86_64 2024-08-03T20:46:33.128 INFO:teuthology.orchestra.run.smithi032.stdout: libtool-2.4.6-46.el9.x86_64 2024-08-03T20:46:33.128 INFO:teuthology.orchestra.run.smithi032.stdout: libunwind-1.6.2-1.el9.x86_64 2024-08-03T20:46:33.128 INFO:teuthology.orchestra.run.smithi032.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-08-03T20:46:33.128 INFO:teuthology.orchestra.run.smithi032.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-08-03T20:46:33.128 INFO:teuthology.orchestra.run.smithi032.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-08-03T20:46:33.128 INFO:teuthology.orchestra.run.smithi032.stdout: lua-5.4.4-4.el9.x86_64 2024-08-03T20:46:33.128 INFO:teuthology.orchestra.run.smithi032.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-08-03T20:46:33.128 INFO:teuthology.orchestra.run.smithi032.stdout: luarocks-3.9.2-5.el9.noarch 2024-08-03T20:46:33.129 INFO:teuthology.orchestra.run.smithi032.stdout: openblas-0.3.21-2.el9.x86_64 2024-08-03T20:46:33.129 INFO:teuthology.orchestra.run.smithi032.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-08-03T20:46:33.129 INFO:teuthology.orchestra.run.smithi032.stdout: openssl-devel-1:3.2.2-2.el9.x86_64 2024-08-03T20:46:33.129 INFO:teuthology.orchestra.run.smithi032.stdout: parquet-libs-9.0.0-11.el9.x86_64 2024-08-03T20:46:33.129 INFO:teuthology.orchestra.run.smithi032.stdout: postgresql-13.14-1.el9.x86_64 2024-08-03T20:46:33.129 INFO:teuthology.orchestra.run.smithi032.stdout: postgresql-contrib-13.14-1.el9.x86_64 2024-08-03T20:46:33.129 INFO:teuthology.orchestra.run.smithi032.stdout: postgresql-private-libs-13.14-1.el9.x86_64 2024-08-03T20:46:33.129 INFO:teuthology.orchestra.run.smithi032.stdout: postgresql-server-13.14-1.el9.x86_64 2024-08-03T20:46:33.129 INFO:teuthology.orchestra.run.smithi032.stdout: protobuf-3.14.0-13.el9.x86_64 2024-08-03T20:46:33.129 INFO:teuthology.orchestra.run.smithi032.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-08-03T20:46:33.129 INFO:teuthology.orchestra.run.smithi032.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-08-03T20:46:33.129 INFO:teuthology.orchestra.run.smithi032.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-08-03T20:46:33.129 INFO:teuthology.orchestra.run.smithi032.stdout: python3-babel-2.9.1-2.el9.noarch 2024-08-03T20:46:33.129 INFO:teuthology.orchestra.run.smithi032.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-08-03T20:46:33.130 INFO:teuthology.orchestra.run.smithi032.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-08-03T20:46:33.130 INFO:teuthology.orchestra.run.smithi032.stdout: python3-ceph-argparse-2:19.0.0-5476.g3792a0a4.el9.x86_64 2024-08-03T20:46:33.130 INFO:teuthology.orchestra.run.smithi032.stdout: python3-ceph-common-2:19.0.0-5476.g3792a0a4.el9.x86_64 2024-08-03T20:46:33.130 INFO:teuthology.orchestra.run.smithi032.stdout: python3-cephfs-2:19.0.0-5476.g3792a0a4.el9.x86_64 2024-08-03T20:46:33.130 INFO:teuthology.orchestra.run.smithi032.stdout: python3-certifi-2023.05.07-4.el9.noarch 2024-08-03T20:46:33.130 INFO:teuthology.orchestra.run.smithi032.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-08-03T20:46:33.130 INFO:teuthology.orchestra.run.smithi032.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-08-03T20:46:33.130 INFO:teuthology.orchestra.run.smithi032.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-08-03T20:46:33.130 INFO:teuthology.orchestra.run.smithi032.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-08-03T20:46:33.130 INFO:teuthology.orchestra.run.smithi032.stdout: python3-colorama-0.4.6-3.el9.noarch 2024-08-03T20:46:33.130 INFO:teuthology.orchestra.run.smithi032.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-08-03T20:46:33.130 INFO:teuthology.orchestra.run.smithi032.stdout: python3-devel-3.9.18-3.el9.x86_64 2024-08-03T20:46:33.130 INFO:teuthology.orchestra.run.smithi032.stdout: python3-google-auth-1:2.32.0-1.el9.noarch 2024-08-03T20:46:33.131 INFO:teuthology.orchestra.run.smithi032.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-08-03T20:46:33.131 INFO:teuthology.orchestra.run.smithi032.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-08-03T20:46:33.131 INFO:teuthology.orchestra.run.smithi032.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-08-03T20:46:33.131 INFO:teuthology.orchestra.run.smithi032.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-08-03T20:46:33.131 INFO:teuthology.orchestra.run.smithi032.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-08-03T20:46:33.131 INFO:teuthology.orchestra.run.smithi032.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-08-03T20:46:33.131 INFO:teuthology.orchestra.run.smithi032.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-08-03T20:46:33.131 INFO:teuthology.orchestra.run.smithi032.stdout: python3-jinja2-2.11.3-6.el9.noarch 2024-08-03T20:46:33.131 INFO:teuthology.orchestra.run.smithi032.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-08-03T20:46:33.131 INFO:teuthology.orchestra.run.smithi032.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-08-03T20:46:33.131 INFO:teuthology.orchestra.run.smithi032.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-08-03T20:46:33.131 INFO:teuthology.orchestra.run.smithi032.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-08-03T20:46:33.131 INFO:teuthology.orchestra.run.smithi032.stdout: python3-mako-1.1.4-6.el9.noarch 2024-08-03T20:46:33.131 INFO:teuthology.orchestra.run.smithi032.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-08-03T20:46:33.132 INFO:teuthology.orchestra.run.smithi032.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-08-03T20:46:33.132 INFO:teuthology.orchestra.run.smithi032.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-08-03T20:46:33.132 INFO:teuthology.orchestra.run.smithi032.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-08-03T20:46:33.132 INFO:teuthology.orchestra.run.smithi032.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-08-03T20:46:33.132 INFO:teuthology.orchestra.run.smithi032.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-08-03T20:46:33.132 INFO:teuthology.orchestra.run.smithi032.stdout: python3-packaging-20.9-5.el9.noarch 2024-08-03T20:46:33.132 INFO:teuthology.orchestra.run.smithi032.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-08-03T20:46:33.132 INFO:teuthology.orchestra.run.smithi032.stdout: python3-ply-3.11-14.el9.noarch 2024-08-03T20:46:33.132 INFO:teuthology.orchestra.run.smithi032.stdout: python3-portend-3.1.0-2.el9.noarch 2024-08-03T20:46:33.132 INFO:teuthology.orchestra.run.smithi032.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-08-03T20:46:33.132 INFO:teuthology.orchestra.run.smithi032.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-08-03T20:46:33.132 INFO:teuthology.orchestra.run.smithi032.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-08-03T20:46:33.132 INFO:teuthology.orchestra.run.smithi032.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-08-03T20:46:33.132 INFO:teuthology.orchestra.run.smithi032.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-08-03T20:46:33.133 INFO:teuthology.orchestra.run.smithi032.stdout: python3-pycparser-2.20-6.el9.noarch 2024-08-03T20:46:33.133 INFO:teuthology.orchestra.run.smithi032.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-08-03T20:46:33.133 INFO:teuthology.orchestra.run.smithi032.stdout: python3-pytz-2021.1-5.el9.noarch 2024-08-03T20:46:33.133 INFO:teuthology.orchestra.run.smithi032.stdout: python3-rados-2:19.0.0-5476.g3792a0a4.el9.x86_64 2024-08-03T20:46:33.133 INFO:teuthology.orchestra.run.smithi032.stdout: python3-rbd-2:19.0.0-5476.g3792a0a4.el9.x86_64 2024-08-03T20:46:33.133 INFO:teuthology.orchestra.run.smithi032.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-08-03T20:46:33.133 INFO:teuthology.orchestra.run.smithi032.stdout: python3-requests-2.25.1-8.el9.noarch 2024-08-03T20:46:33.133 INFO:teuthology.orchestra.run.smithi032.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-08-03T20:46:33.133 INFO:teuthology.orchestra.run.smithi032.stdout: python3-rgw-2:19.0.0-5476.g3792a0a4.el9.x86_64 2024-08-03T20:46:33.133 INFO:teuthology.orchestra.run.smithi032.stdout: python3-routes-2.5.1-5.el9.noarch 2024-08-03T20:46:33.133 INFO:teuthology.orchestra.run.smithi032.stdout: python3-rsa-4.9-2.el9.noarch 2024-08-03T20:46:33.133 INFO:teuthology.orchestra.run.smithi032.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-08-03T20:46:33.133 INFO:teuthology.orchestra.run.smithi032.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-08-03T20:46:33.133 INFO:teuthology.orchestra.run.smithi032.stdout: python3-toml-0.10.2-6.el9.noarch 2024-08-03T20:46:33.133 INFO:teuthology.orchestra.run.smithi032.stdout: python3-typing-extensions-4.12.2-2.el9.noarch 2024-08-03T20:46:33.134 INFO:teuthology.orchestra.run.smithi032.stdout: python3-urllib3-1.26.5-6.el9.noarch 2024-08-03T20:46:33.134 INFO:teuthology.orchestra.run.smithi032.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-08-03T20:46:33.134 INFO:teuthology.orchestra.run.smithi032.stdout: python3-webob-1.8.7-6.el9.noarch 2024-08-03T20:46:33.134 INFO:teuthology.orchestra.run.smithi032.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-08-03T20:46:33.134 INFO:teuthology.orchestra.run.smithi032.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-08-03T20:46:33.134 INFO:teuthology.orchestra.run.smithi032.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-08-03T20:46:33.134 INFO:teuthology.orchestra.run.smithi032.stdout: qatlib-24.02.0-1.el9.x86_64 2024-08-03T20:46:33.134 INFO:teuthology.orchestra.run.smithi032.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-08-03T20:46:33.134 INFO:teuthology.orchestra.run.smithi032.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-08-03T20:46:33.134 INFO:teuthology.orchestra.run.smithi032.stdout: rbd-fuse-2:19.0.0-5476.g3792a0a4.el9.x86_64 2024-08-03T20:46:33.134 INFO:teuthology.orchestra.run.smithi032.stdout: rbd-mirror-2:19.0.0-5476.g3792a0a4.el9.x86_64 2024-08-03T20:46:33.134 INFO:teuthology.orchestra.run.smithi032.stdout: rbd-nbd-2:19.0.0-5476.g3792a0a4.el9.x86_64 2024-08-03T20:46:33.134 INFO:teuthology.orchestra.run.smithi032.stdout: re2-1:20211101-3.el9.x86_64 2024-08-03T20:46:33.135 INFO:teuthology.orchestra.run.smithi032.stdout: socat-1.7.4.1-6.el9.x86_64 2024-08-03T20:46:33.135 INFO:teuthology.orchestra.run.smithi032.stdout: thrift-0.15.0-2.el9.x86_64 2024-08-03T20:46:33.135 INFO:teuthology.orchestra.run.smithi032.stdout: uuid-1.6.2-55.el9.x86_64 2024-08-03T20:46:33.135 INFO:teuthology.orchestra.run.smithi032.stdout: xfsprogs-devel-6.4.0-3.el9.x86_64 2024-08-03T20:46:33.135 INFO:teuthology.orchestra.run.smithi032.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-08-03T20:46:33.135 INFO:teuthology.orchestra.run.smithi032.stdout: zlib-devel-1.2.11-41.el9.x86_64 2024-08-03T20:46:33.135 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-03T20:46:33.135 INFO:teuthology.orchestra.run.smithi032.stdout:Complete! 2024-08-03T20:46:33.812 DEBUG:teuthology.parallel:result is None 2024-08-03T20:46:33.813 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=3792a0a4a40f645dee271188eab388b1c22c10f8 2024-08-03T20:46:33.970 DEBUG:teuthology.orchestra.run.smithi032:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-08-03T20:46:34.019 INFO:teuthology.orchestra.run.smithi032.stdout:19.0.0-5476.g3792a0a4.el9 2024-08-03T20:46:34.020 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-5476.g3792a0a4.el9 2024-08-03T20:46:34.020 INFO:teuthology.task.install:The correct ceph version 19.0.0-5476.g3792a0a4 is installed. 2024-08-03T20:46:34.022 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=3792a0a4a40f645dee271188eab388b1c22c10f8 2024-08-03T20:46:34.175 DEBUG:teuthology.orchestra.run.smithi118:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-08-03T20:46:34.222 INFO:teuthology.orchestra.run.smithi118.stdout:19.0.0-5476.g3792a0a4.el9 2024-08-03T20:46:34.223 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-5476.g3792a0a4.el9 2024-08-03T20:46:34.223 INFO:teuthology.task.install:The correct ceph version 19.0.0-5476.g3792a0a4 is installed. 2024-08-03T20:46:34.225 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-08-03T20:46:34.225 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-08-03T20:46:34.225 DEBUG:teuthology.orchestra.run.smithi032:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-08-03T20:46:34.286 DEBUG:teuthology.orchestra.run.smithi118:> set -ex 2024-08-03T20:46:34.286 DEBUG:teuthology.orchestra.run.smithi118:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-08-03T20:46:34.350 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-08-03T20:46:34.351 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-08-03T20:46:34.351 DEBUG:teuthology.orchestra.run.smithi032:> sudo dd of=/usr/bin/daemon-helper 2024-08-03T20:46:34.409 DEBUG:teuthology.orchestra.run.smithi032:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-08-03T20:46:34.506 DEBUG:teuthology.orchestra.run.smithi118:> set -ex 2024-08-03T20:46:34.506 DEBUG:teuthology.orchestra.run.smithi118:> sudo dd of=/usr/bin/daemon-helper 2024-08-03T20:46:34.562 DEBUG:teuthology.orchestra.run.smithi118:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-08-03T20:46:34.658 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-08-03T20:46:34.658 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-08-03T20:46:34.658 DEBUG:teuthology.orchestra.run.smithi032:> sudo dd of=/usr/bin/adjust-ulimits 2024-08-03T20:46:34.716 DEBUG:teuthology.orchestra.run.smithi032:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-08-03T20:46:34.814 DEBUG:teuthology.orchestra.run.smithi118:> set -ex 2024-08-03T20:46:34.814 DEBUG:teuthology.orchestra.run.smithi118:> sudo dd of=/usr/bin/adjust-ulimits 2024-08-03T20:46:34.869 DEBUG:teuthology.orchestra.run.smithi118:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-08-03T20:46:34.966 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-08-03T20:46:34.966 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-08-03T20:46:34.966 DEBUG:teuthology.orchestra.run.smithi032:> sudo dd of=/usr/bin/stdin-killer 2024-08-03T20:46:35.025 DEBUG:teuthology.orchestra.run.smithi032:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-08-03T20:46:35.126 DEBUG:teuthology.orchestra.run.smithi118:> set -ex 2024-08-03T20:46:35.126 DEBUG:teuthology.orchestra.run.smithi118:> sudo dd of=/usr/bin/stdin-killer 2024-08-03T20:46:35.213 DEBUG:teuthology.orchestra.run.smithi118:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-08-03T20:46:35.310 INFO:teuthology.run_tasks:Running task ceph... 2024-08-03T20:46:35.396 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-08-03T20:46:35.397 DEBUG:teuthology.orchestra.run.smithi032:> sudo chmod 777 /var/log/ceph 2024-08-03T20:46:35.402 DEBUG:teuthology.orchestra.run.smithi118:> sudo chmod 777 /var/log/ceph 2024-08-03T20:46:35.457 INFO:tasks.ceph:Disabling ceph logrotate... 2024-08-03T20:46:35.458 DEBUG:teuthology.orchestra.run.smithi032:> sudo rm -f -- /etc/logrotate.d/ceph 2024-08-03T20:46:35.461 DEBUG:teuthology.orchestra.run.smithi118:> sudo rm -f -- /etc/logrotate.d/ceph 2024-08-03T20:46:35.517 INFO:tasks.ceph:Creating extra log directories... 2024-08-03T20:46:35.517 DEBUG:teuthology.orchestra.run.smithi032:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-08-03T20:46:35.521 DEBUG:teuthology.orchestra.run.smithi118:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-08-03T20:46:35.581 INFO:tasks.ceph:Setting up log rotation with {'ceph-mds': '10G', 'ceph-osd': '10G'} 2024-08-03T20:46:35.582 INFO:tasks.ceph:writing logrotate stanza for ceph-mds 2024-08-03T20:46:35.582 INFO:tasks.ceph:writing logrotate stanza for ceph-osd 2024-08-03T20:46:35.582 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-08-03T20:46:35.582 DEBUG:teuthology.orchestra.run.smithi032:> dd of=/home/ubuntu/cephtest/logrotate.ceph-test.conf 2024-08-03T20:46:35.617 DEBUG:teuthology.orchestra.run.smithi032:> sudo mv /home/ubuntu/cephtest/logrotate.ceph-test.conf /etc/logrotate.d/ceph-test.conf && sudo chmod 0644 /etc/logrotate.d/ceph-test.conf && sudo chown root.root /etc/logrotate.d/ceph-test.conf 2024-08-03T20:46:35.787 DEBUG:teuthology.orchestra.run.smithi032:> sudo chcon system_u:object_r:etc_t:s0 /etc/logrotate.d/ceph-test.conf 2024-08-03T20:46:35.843 DEBUG:teuthology.orchestra.run.smithi118:> set -ex 2024-08-03T20:46:35.843 DEBUG:teuthology.orchestra.run.smithi118:> dd of=/home/ubuntu/cephtest/logrotate.ceph-test.conf 2024-08-03T20:46:35.878 DEBUG:teuthology.orchestra.run.smithi118:> sudo mv /home/ubuntu/cephtest/logrotate.ceph-test.conf /etc/logrotate.d/ceph-test.conf && sudo chmod 0644 /etc/logrotate.d/ceph-test.conf && sudo chown root.root /etc/logrotate.d/ceph-test.conf 2024-08-03T20:46:36.046 DEBUG:teuthology.orchestra.run.smithi118:> sudo chcon system_u:object_r:etc_t:s0 /etc/logrotate.d/ceph-test.conf 2024-08-03T20:46:36.106 INFO:tasks.ceph:Creating ceph cluster ceph... 2024-08-03T20:46:36.106 INFO:tasks.ceph:config {'conf': {'client': {'pid file': ''}, 'global': {'ms die on skipped message': False}, 'mgr': {'client mount timeout': 30, 'debug client': 20, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon down mkfs grace': 300}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'online, but wants', 'filesystem is online with fewer MDS than max_mds', 'POOL_APP_NOT_ENABLED', 'do not have an application enabled', 'overall HEALTH_', 'Replacing daemon', 'deprecated feature inline_data', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'PG_AVAILABILITY', 'PG_DEGRADED', 'Reduced data availability', 'Degraded data redundancy', 'bad backtrace', 'object missing on disk', 'error reading table object', 'error reading sessionmap', 'Error loading MDS rank', 'missing journal object', 'Error recovering journal', 'error decoding table object', 'failed to read JournalPointer', 'Corrupt directory entry', 'Corrupt fnode header', 'corrupt sessionmap header', 'Corrupt dentry', 'Scrub error on inode', 'Metadata damage detected', 'MDS_READ_ONLY', 'force file system read-only', 'with standby daemon mds', 'MDS abort because newly corrupt dentry'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2024-08-03T20:46:36.106 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/vshankar-2024-08-02_18:35:56-fs-wip-vshankar-testing-20240802.091913-debug-testing-default-smithi/7833836', 'branch': 'wip-vshankar-testing-20240802.091913-debug', 'description': 'fs/functional/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-4c-client conf/{client mds mgr mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} subvol_versions/create_subvol_version_v1 tasks/damage}', 'email': 'vshankar@redhat.com', 'first_in_suite': False, 'job_id': '7833836', 'kernel': {'client': {'branch': 'testing'}, 'flavor': 'default', 'kdb': 1, 'sha1': 'ef231aaae597a12bd3e2479b633f31ed1fcc23a8'}, 'ktype': 'upstream', 'last_in_suite': False, 'log-rotate': {'ceph-mds': '10G', 'ceph-osd': '10G'}, 'machine_type': 'smithi', 'mgrmodules': {'sequential': [{'print': 'Enabling mgr modules'}]}, 'name': 'vshankar-2024-08-02_18:35:56-fs-wip-vshankar-testing-20240802.091913-debug-testing-default-smithi', 'no_nested_subset': False, 'openstack': [{'volumes': {'count': 4, 'size': 20}}, {'machine': {'disk': 200}}], 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'wip-vshankar-testing-20240802.091913-debug'}, 'ceph': {'cluster-conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 2, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mon': {'mon op complaint time': 120, 'mon warn on pool no app': False}, 'osd': {'osd op complaint time': 180}}, 'conf': {'client': {'pid file': ''}, 'global': {'ms die on skipped message': False}, 'mgr': {'client mount timeout': 30, 'debug client': 20, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon down mkfs grace': 300}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'online, but wants', 'filesystem is online with fewer MDS than max_mds', 'POOL_APP_NOT_ENABLED', 'do not have an application enabled', 'overall HEALTH_', 'Replacing daemon', 'deprecated feature inline_data', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'PG_AVAILABILITY', 'PG_DEGRADED', 'Reduced data availability', 'Degraded data redundancy', 'bad backtrace', 'object missing on disk', 'error reading table object', 'error reading sessionmap', 'Error loading MDS rank', 'missing journal object', 'Error recovering journal', 'error decoding table object', 'failed to read JournalPointer', 'Corrupt directory entry', 'Corrupt fnode header', 'corrupt sessionmap header', 'Corrupt dentry', 'Scrub error on inode', 'Metadata damage detected', 'MDS_READ_ONLY', 'force file system read-only', 'with standby daemon mds', 'MDS abort because newly corrupt dentry'], 'sha1': '3792a0a4a40f645dee271188eab388b1c22c10f8'}, 'ceph-deploy': {'bluestore': True, 'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs'}, 'install': {'ceph': {'flavor': 'default', 'sha1': '3792a0a4a40f645dee271188eab388b1c22c10f8'}}, 'kclient': {'syntax': 'v1'}, 'selinux': {'allowlist': ['scontext=system_u:system_r:getty_t:s0']}, 'subvolume_version': 1, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'wip-vshankar-testing-20240802.091913-debug', 'sha1': '3792a0a4a40f645dee271188eab388b1c22c10f8'}}, 'owner': 'scheduled_vshankar@teuthology', 'priority': 120, 'repo': 'https://git.ceph.com/ceph-ci.git', 'roles': [['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'client.2'], ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'client.3']], 'seed': 8569, 'sha1': '3792a0a4a40f645dee271188eab388b1c22c10f8', 'sleep_before_teardown': 0, 'subset': '921/1024', 'suite': 'fs', 'suite_branch': 'wip-vshankar-testing-20240802.091913-debug', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph-c_3792a0a4a40f645dee271188eab388b1c22c10f8/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://git.ceph.com/ceph-ci.git', 'suite_sha1': '3792a0a4a40f645dee271188eab388b1c22c10f8', 'syslog': {'ignorelist': ['WARNING*.*check_session_state', 'WARNING*.*__ceph_remove_cap']}, 'targets': {'smithi032.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBBEjgnfLSs9lqOXHOa/Qij0TO92ti1seO/TFIkruZ7njoFUZwK0ns3l3qN/K4jsYdwKxnrYeISPVKfBlHe+Tl4g=', 'smithi118.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPifbzPloH1zU8jOISrQRk2LVFiPLW4lgOKvMElkNf3OB3J+1zb2PHttd0EMJuM3OtfV2Jt+UnFjAi4XIV4wTQ4='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'client': {'branch': 'testing'}, 'flavor': 'default', 'kdb': 1, 'sha1': 'ef231aaae597a12bd3e2479b633f31ed1fcc23a8'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': '3792a0a4a40f645dee271188eab388b1c22c10f8'}}, {'ceph': {'create_rbd_pool': False, 'cluster-conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 2, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mon': {'mon op complaint time': 120, 'mon warn on pool no app': False}, 'osd': {'osd op complaint time': 180}}, 'conf': {'client': {'pid file': ''}, 'global': {'ms die on skipped message': False}, 'mgr': {'client mount timeout': 30, 'debug client': 20, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon down mkfs grace': 300}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'online, but wants', 'filesystem is online with fewer MDS than max_mds', 'POOL_APP_NOT_ENABLED', 'do not have an application enabled', 'overall HEALTH_', 'Replacing daemon', 'deprecated feature inline_data', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'PG_AVAILABILITY', 'PG_DEGRADED', 'Reduced data availability', 'Degraded data redundancy', 'bad backtrace', 'object missing on disk', 'error reading table object', 'error reading sessionmap', 'Error loading MDS rank', 'missing journal object', 'Error recovering journal', 'error decoding table object', 'failed to read JournalPointer', 'Corrupt directory entry', 'Corrupt fnode header', 'corrupt sessionmap header', 'Corrupt dentry', 'Scrub error on inode', 'Metadata damage detected', 'MDS_READ_ONLY', 'force file system read-only', 'with standby daemon mds', 'MDS abort because newly corrupt dentry'], 'sha1': '3792a0a4a40f645dee271188eab388b1c22c10f8', 'cluster': 'ceph'}}, {'sequential': ['mgrmodules']}, {'kclient': None}, {'cephfs_test_runner': {'modules': ['tasks.cephfs.test_damage']}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': ["local function is_kupstream() return yaml.ktype == 'upstream' end", "local function is_kdistro() return yaml.ktype == 'distro' end", 'local function is_fuse() return false end', 'local function syntax_version() return yaml.overrides.kclient.syntax end']}, 'teuthology_branch': 'main', 'teuthology_sha1': '7d7f5fe79c7b6b5080402153d07fb1acc4ce00a2', 'timestamp': '2024-08-02_18:35:56', 'tube': 'smithi', 'user': 'vshankar', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.4074525'} 2024-08-03T20:46:36.107 DEBUG:teuthology.orchestra.run.smithi032:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-08-03T20:46:36.111 DEBUG:teuthology.orchestra.run.smithi118:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-08-03T20:46:36.146 DEBUG:teuthology.orchestra.run.smithi032:> sudo install -d -m0777 -- /var/run/ceph 2024-08-03T20:46:36.187 DEBUG:teuthology.orchestra.run.smithi118:> sudo install -d -m0777 -- /var/run/ceph 2024-08-03T20:46:36.243 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-08-03T20:46:36.243 DEBUG:teuthology.orchestra.run.smithi032:> dd if=/scratch_devs of=/dev/stdout 2024-08-03T20:46:36.278 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-08-03T20:46:36.278 DEBUG:teuthology.orchestra.run.smithi032:> stat /dev/vg_nvme/lv_1 2024-08-03T20:46:36.353 INFO:teuthology.orchestra.run.smithi032.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-08-03T20:46:36.354 INFO:teuthology.orchestra.run.smithi032.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-08-03T20:46:36.354 INFO:teuthology.orchestra.run.smithi032.stdout:Device: 6h/6d Inode: 638 Links: 1 2024-08-03T20:46:36.354 INFO:teuthology.orchestra.run.smithi032.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-08-03T20:46:36.354 INFO:teuthology.orchestra.run.smithi032.stdout:Context: system_u:object_r:device_t:s0 2024-08-03T20:46:36.354 INFO:teuthology.orchestra.run.smithi032.stdout:Access: 2024-08-03 20:46:24.617402179 +0000 2024-08-03T20:46:36.354 INFO:teuthology.orchestra.run.smithi032.stdout:Modify: 2024-08-03 20:43:49.771019178 +0000 2024-08-03T20:46:36.354 INFO:teuthology.orchestra.run.smithi032.stdout:Change: 2024-08-03 20:43:49.771019178 +0000 2024-08-03T20:46:36.354 INFO:teuthology.orchestra.run.smithi032.stdout: Birth: 2024-08-03 20:43:49.771019178 +0000 2024-08-03T20:46:36.354 DEBUG:teuthology.orchestra.run.smithi032:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-08-03T20:46:36.447 INFO:teuthology.orchestra.run.smithi032.stderr:1+0 records in 2024-08-03T20:46:36.447 INFO:teuthology.orchestra.run.smithi032.stderr:1+0 records out 2024-08-03T20:46:36.447 INFO:teuthology.orchestra.run.smithi032.stderr:512 bytes copied, 0.000243703 s, 2.1 MB/s 2024-08-03T20:46:36.449 DEBUG:teuthology.orchestra.run.smithi032:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-08-03T20:46:36.486 DEBUG:teuthology.orchestra.run.smithi032:> stat /dev/vg_nvme/lv_2 2024-08-03T20:46:36.563 INFO:teuthology.orchestra.run.smithi032.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-08-03T20:46:36.563 INFO:teuthology.orchestra.run.smithi032.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-08-03T20:46:36.563 INFO:teuthology.orchestra.run.smithi032.stdout:Device: 6h/6d Inode: 656 Links: 1 2024-08-03T20:46:36.564 INFO:teuthology.orchestra.run.smithi032.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-08-03T20:46:36.564 INFO:teuthology.orchestra.run.smithi032.stdout:Context: system_u:object_r:device_t:s0 2024-08-03T20:46:36.564 INFO:teuthology.orchestra.run.smithi032.stdout:Access: 2024-08-03 20:46:24.618402159 +0000 2024-08-03T20:46:36.564 INFO:teuthology.orchestra.run.smithi032.stdout:Modify: 2024-08-03 20:43:50.224010138 +0000 2024-08-03T20:46:36.564 INFO:teuthology.orchestra.run.smithi032.stdout:Change: 2024-08-03 20:43:50.224010138 +0000 2024-08-03T20:46:36.564 INFO:teuthology.orchestra.run.smithi032.stdout: Birth: 2024-08-03 20:43:50.224010138 +0000 2024-08-03T20:46:36.564 DEBUG:teuthology.orchestra.run.smithi032:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-08-03T20:46:36.656 INFO:teuthology.orchestra.run.smithi032.stderr:1+0 records in 2024-08-03T20:46:36.656 INFO:teuthology.orchestra.run.smithi032.stderr:1+0 records out 2024-08-03T20:46:36.656 INFO:teuthology.orchestra.run.smithi032.stderr:512 bytes copied, 0.000170318 s, 3.0 MB/s 2024-08-03T20:46:36.658 DEBUG:teuthology.orchestra.run.smithi032:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-08-03T20:46:36.695 DEBUG:teuthology.orchestra.run.smithi032:> stat /dev/vg_nvme/lv_3 2024-08-03T20:46:36.770 INFO:teuthology.orchestra.run.smithi032.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-08-03T20:46:36.770 INFO:teuthology.orchestra.run.smithi032.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-08-03T20:46:36.770 INFO:teuthology.orchestra.run.smithi032.stdout:Device: 6h/6d Inode: 677 Links: 1 2024-08-03T20:46:36.770 INFO:teuthology.orchestra.run.smithi032.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-08-03T20:46:36.770 INFO:teuthology.orchestra.run.smithi032.stdout:Context: system_u:object_r:device_t:s0 2024-08-03T20:46:36.770 INFO:teuthology.orchestra.run.smithi032.stdout:Access: 2024-08-03 20:46:24.618402159 +0000 2024-08-03T20:46:36.771 INFO:teuthology.orchestra.run.smithi032.stdout:Modify: 2024-08-03 20:43:50.684000958 +0000 2024-08-03T20:46:36.771 INFO:teuthology.orchestra.run.smithi032.stdout:Change: 2024-08-03 20:43:50.684000958 +0000 2024-08-03T20:46:36.771 INFO:teuthology.orchestra.run.smithi032.stdout: Birth: 2024-08-03 20:43:50.684000958 +0000 2024-08-03T20:46:36.771 DEBUG:teuthology.orchestra.run.smithi032:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-08-03T20:46:36.862 INFO:teuthology.orchestra.run.smithi032.stderr:1+0 records in 2024-08-03T20:46:36.863 INFO:teuthology.orchestra.run.smithi032.stderr:1+0 records out 2024-08-03T20:46:36.863 INFO:teuthology.orchestra.run.smithi032.stderr:512 bytes copied, 0.000169653 s, 3.0 MB/s 2024-08-03T20:46:36.865 DEBUG:teuthology.orchestra.run.smithi032:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-08-03T20:46:36.901 DEBUG:teuthology.orchestra.run.smithi032:> stat /dev/vg_nvme/lv_4 2024-08-03T20:46:36.977 INFO:teuthology.orchestra.run.smithi032.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-08-03T20:46:36.977 INFO:teuthology.orchestra.run.smithi032.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-08-03T20:46:36.977 INFO:teuthology.orchestra.run.smithi032.stdout:Device: 6h/6d Inode: 695 Links: 1 2024-08-03T20:46:36.977 INFO:teuthology.orchestra.run.smithi032.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-08-03T20:46:36.977 INFO:teuthology.orchestra.run.smithi032.stdout:Context: system_u:object_r:device_t:s0 2024-08-03T20:46:36.977 INFO:teuthology.orchestra.run.smithi032.stdout:Access: 2024-08-03 20:46:24.619402138 +0000 2024-08-03T20:46:36.977 INFO:teuthology.orchestra.run.smithi032.stdout:Modify: 2024-08-03 20:43:51.153991578 +0000 2024-08-03T20:46:36.977 INFO:teuthology.orchestra.run.smithi032.stdout:Change: 2024-08-03 20:43:51.153991578 +0000 2024-08-03T20:46:36.978 INFO:teuthology.orchestra.run.smithi032.stdout: Birth: 2024-08-03 20:43:51.153991578 +0000 2024-08-03T20:46:36.978 DEBUG:teuthology.orchestra.run.smithi032:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-08-03T20:46:37.070 INFO:teuthology.orchestra.run.smithi032.stderr:1+0 records in 2024-08-03T20:46:37.070 INFO:teuthology.orchestra.run.smithi032.stderr:1+0 records out 2024-08-03T20:46:37.070 INFO:teuthology.orchestra.run.smithi032.stderr:512 bytes copied, 0.000165572 s, 3.1 MB/s 2024-08-03T20:46:37.072 DEBUG:teuthology.orchestra.run.smithi032:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-08-03T20:46:37.109 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-08-03T20:46:37.109 DEBUG:teuthology.orchestra.run.smithi118:> set -ex 2024-08-03T20:46:37.109 DEBUG:teuthology.orchestra.run.smithi118:> dd if=/scratch_devs of=/dev/stdout 2024-08-03T20:46:37.148 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-08-03T20:46:37.148 DEBUG:teuthology.orchestra.run.smithi118:> stat /dev/vg_nvme/lv_1 2024-08-03T20:46:37.225 INFO:teuthology.orchestra.run.smithi118.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-08-03T20:46:37.225 INFO:teuthology.orchestra.run.smithi118.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-08-03T20:46:37.225 INFO:teuthology.orchestra.run.smithi118.stdout:Device: 6h/6d Inode: 631 Links: 1 2024-08-03T20:46:37.225 INFO:teuthology.orchestra.run.smithi118.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-08-03T20:46:37.225 INFO:teuthology.orchestra.run.smithi118.stdout:Context: system_u:object_r:device_t:s0 2024-08-03T20:46:37.226 INFO:teuthology.orchestra.run.smithi118.stdout:Access: 2024-08-03 20:46:21.562787917 +0000 2024-08-03T20:46:37.226 INFO:teuthology.orchestra.run.smithi118.stdout:Modify: 2024-08-03 20:43:46.828862990 +0000 2024-08-03T20:46:37.226 INFO:teuthology.orchestra.run.smithi118.stdout:Change: 2024-08-03 20:43:46.828862990 +0000 2024-08-03T20:46:37.226 INFO:teuthology.orchestra.run.smithi118.stdout: Birth: 2024-08-03 20:43:46.828862990 +0000 2024-08-03T20:46:37.226 DEBUG:teuthology.orchestra.run.smithi118:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-08-03T20:46:37.320 INFO:teuthology.orchestra.run.smithi118.stderr:1+0 records in 2024-08-03T20:46:37.320 INFO:teuthology.orchestra.run.smithi118.stderr:1+0 records out 2024-08-03T20:46:37.320 INFO:teuthology.orchestra.run.smithi118.stderr:512 bytes copied, 0.000183067 s, 2.8 MB/s 2024-08-03T20:46:37.323 DEBUG:teuthology.orchestra.run.smithi118:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-08-03T20:46:37.361 DEBUG:teuthology.orchestra.run.smithi118:> stat /dev/vg_nvme/lv_2 2024-08-03T20:46:37.436 INFO:teuthology.orchestra.run.smithi118.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-08-03T20:46:37.436 INFO:teuthology.orchestra.run.smithi118.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-08-03T20:46:37.436 INFO:teuthology.orchestra.run.smithi118.stdout:Device: 6h/6d Inode: 651 Links: 1 2024-08-03T20:46:37.436 INFO:teuthology.orchestra.run.smithi118.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-08-03T20:46:37.436 INFO:teuthology.orchestra.run.smithi118.stdout:Context: system_u:object_r:device_t:s0 2024-08-03T20:46:37.436 INFO:teuthology.orchestra.run.smithi118.stdout:Access: 2024-08-03 20:46:21.563787895 +0000 2024-08-03T20:46:37.436 INFO:teuthology.orchestra.run.smithi118.stdout:Modify: 2024-08-03 20:43:47.272853339 +0000 2024-08-03T20:46:37.437 INFO:teuthology.orchestra.run.smithi118.stdout:Change: 2024-08-03 20:43:47.272853339 +0000 2024-08-03T20:46:37.437 INFO:teuthology.orchestra.run.smithi118.stdout: Birth: 2024-08-03 20:43:47.272853339 +0000 2024-08-03T20:46:37.437 DEBUG:teuthology.orchestra.run.smithi118:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-08-03T20:46:37.528 INFO:teuthology.orchestra.run.smithi118.stderr:1+0 records in 2024-08-03T20:46:37.528 INFO:teuthology.orchestra.run.smithi118.stderr:1+0 records out 2024-08-03T20:46:37.529 INFO:teuthology.orchestra.run.smithi118.stderr:512 bytes copied, 0.0001714 s, 3.0 MB/s 2024-08-03T20:46:37.531 DEBUG:teuthology.orchestra.run.smithi118:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-08-03T20:46:37.567 DEBUG:teuthology.orchestra.run.smithi118:> stat /dev/vg_nvme/lv_3 2024-08-03T20:46:37.643 INFO:teuthology.orchestra.run.smithi118.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-08-03T20:46:37.643 INFO:teuthology.orchestra.run.smithi118.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-08-03T20:46:37.643 INFO:teuthology.orchestra.run.smithi118.stdout:Device: 6h/6d Inode: 671 Links: 1 2024-08-03T20:46:37.643 INFO:teuthology.orchestra.run.smithi118.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-08-03T20:46:37.643 INFO:teuthology.orchestra.run.smithi118.stdout:Context: system_u:object_r:device_t:s0 2024-08-03T20:46:37.643 INFO:teuthology.orchestra.run.smithi118.stdout:Access: 2024-08-03 20:46:21.563787895 +0000 2024-08-03T20:46:37.643 INFO:teuthology.orchestra.run.smithi118.stdout:Modify: 2024-08-03 20:43:47.754842861 +0000 2024-08-03T20:46:37.643 INFO:teuthology.orchestra.run.smithi118.stdout:Change: 2024-08-03 20:43:47.754842861 +0000 2024-08-03T20:46:37.643 INFO:teuthology.orchestra.run.smithi118.stdout: Birth: 2024-08-03 20:43:47.754842861 +0000 2024-08-03T20:46:37.644 DEBUG:teuthology.orchestra.run.smithi118:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-08-03T20:46:37.735 INFO:teuthology.orchestra.run.smithi118.stderr:1+0 records in 2024-08-03T20:46:37.735 INFO:teuthology.orchestra.run.smithi118.stderr:1+0 records out 2024-08-03T20:46:37.735 INFO:teuthology.orchestra.run.smithi118.stderr:512 bytes copied, 0.000181266 s, 2.8 MB/s 2024-08-03T20:46:37.737 DEBUG:teuthology.orchestra.run.smithi118:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-08-03T20:46:37.774 DEBUG:teuthology.orchestra.run.smithi118:> stat /dev/vg_nvme/lv_4 2024-08-03T20:46:37.849 INFO:teuthology.orchestra.run.smithi118.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-08-03T20:46:37.849 INFO:teuthology.orchestra.run.smithi118.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-08-03T20:46:37.849 INFO:teuthology.orchestra.run.smithi118.stdout:Device: 6h/6d Inode: 690 Links: 1 2024-08-03T20:46:37.850 INFO:teuthology.orchestra.run.smithi118.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-08-03T20:46:37.850 INFO:teuthology.orchestra.run.smithi118.stdout:Context: system_u:object_r:device_t:s0 2024-08-03T20:46:37.850 INFO:teuthology.orchestra.run.smithi118.stdout:Access: 2024-08-03 20:46:21.564787873 +0000 2024-08-03T20:46:37.850 INFO:teuthology.orchestra.run.smithi118.stdout:Modify: 2024-08-03 20:43:48.234832427 +0000 2024-08-03T20:46:37.850 INFO:teuthology.orchestra.run.smithi118.stdout:Change: 2024-08-03 20:43:48.234832427 +0000 2024-08-03T20:46:37.850 INFO:teuthology.orchestra.run.smithi118.stdout: Birth: 2024-08-03 20:43:48.234832427 +0000 2024-08-03T20:46:37.850 DEBUG:teuthology.orchestra.run.smithi118:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-08-03T20:46:37.942 INFO:teuthology.orchestra.run.smithi118.stderr:1+0 records in 2024-08-03T20:46:37.942 INFO:teuthology.orchestra.run.smithi118.stderr:1+0 records out 2024-08-03T20:46:37.942 INFO:teuthology.orchestra.run.smithi118.stderr:512 bytes copied, 0.000172944 s, 3.0 MB/s 2024-08-03T20:46:37.945 DEBUG:teuthology.orchestra.run.smithi118:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-08-03T20:46:37.981 INFO:tasks.ceph:osd dev map: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-08-03T20:46:37.982 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi032.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi118.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-08-03T20:46:37.982 INFO:tasks.ceph:Generating config... 2024-08-03T20:46:37.983 INFO:tasks.ceph:[client] pid file = 2024-08-03T20:46:37.983 INFO:tasks.ceph:[global] ms die on skipped message = False 2024-08-03T20:46:37.983 INFO:tasks.ceph:[mgr] client mount timeout = 30 2024-08-03T20:46:37.983 INFO:tasks.ceph:[mgr] debug client = 20 2024-08-03T20:46:37.983 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-08-03T20:46:37.983 INFO:tasks.ceph:[mgr] debug ms = 1 2024-08-03T20:46:37.983 INFO:tasks.ceph:[mon] debug mon = 20 2024-08-03T20:46:37.984 INFO:tasks.ceph:[mon] debug ms = 1 2024-08-03T20:46:37.984 INFO:tasks.ceph:[mon] debug paxos = 20 2024-08-03T20:46:37.984 INFO:tasks.ceph:[mon] mon down mkfs grace = 300 2024-08-03T20:46:37.984 INFO:tasks.ceph:[osd] bdev async discard = True 2024-08-03T20:46:37.984 INFO:tasks.ceph:[osd] bdev enable discard = True 2024-08-03T20:46:37.984 INFO:tasks.ceph:[osd] bluestore allocator = bitmap 2024-08-03T20:46:37.984 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2024-08-03T20:46:37.984 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2024-08-03T20:46:37.984 INFO:tasks.ceph:[osd] debug bluefs = 1/20 2024-08-03T20:46:37.984 INFO:tasks.ceph:[osd] debug bluestore = 1/20 2024-08-03T20:46:37.984 INFO:tasks.ceph:[osd] debug ms = 1 2024-08-03T20:46:37.984 INFO:tasks.ceph:[osd] debug osd = 20 2024-08-03T20:46:37.984 INFO:tasks.ceph:[osd] debug rocksdb = 4/10 2024-08-03T20:46:37.984 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2024-08-03T20:46:37.985 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2024-08-03T20:46:37.985 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2024-08-03T20:46:37.985 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2024-08-03T20:46:37.985 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2024-08-03T20:46:37.985 INFO:tasks.ceph:Setting up mon.a... 2024-08-03T20:46:37.985 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2024-08-03T20:46:38.145 INFO:teuthology.orchestra.run.smithi032.stdout:creating /etc/ceph/ceph.keyring 2024-08-03T20:46:38.158 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2024-08-03T20:46:38.316 DEBUG:teuthology.orchestra.run.smithi032:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-08-03T20:46:38.371 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.32'), ('mon.b', '172.21.15.118'), ('mon.c', '[v2:172.21.15.118:3301,v1:172.21.15.118:6790]')] 2024-08-03T20:46:38.371 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.32,172.21.15.118,[v2:172.21.15.118:3301,v1:172.21.15.118:6790]', 'ms die on skipped message': False}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false', 'client mount timeout': 30, 'debug client': 20}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon down mkfs grace': 300}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'pid file': ''}, 'mon.a': {}, 'mon.b': {}, 'mon.c': {}, 'mds.a': {}, 'mds.c': {}, 'mds.b': {}, 'mds.d': {}} 2024-08-03T20:46:38.372 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-08-03T20:46:38.372 DEBUG:teuthology.orchestra.run.smithi032:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-08-03T20:46:38.409 DEBUG:teuthology.orchestra.run.smithi032:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.32 --add b 172.21.15.118 --addv c '[v2:172.21.15.118:3301,v1:172.21.15.118:6790]' --print /home/ubuntu/cephtest/ceph.monmap 2024-08-03T20:46:38.597 INFO:teuthology.orchestra.run.smithi032.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2024-08-03T20:46:38.597 INFO:teuthology.orchestra.run.smithi032.stdout:monmaptool: generated fsid 40ab141a-01d7-49b9-ae3c-2edec9876e95 2024-08-03T20:46:38.598 INFO:teuthology.orchestra.run.smithi032.stdout:setting min_mon_release = quincy 2024-08-03T20:46:38.598 INFO:teuthology.orchestra.run.smithi032.stdout:epoch 0 2024-08-03T20:46:38.598 INFO:teuthology.orchestra.run.smithi032.stdout:fsid 40ab141a-01d7-49b9-ae3c-2edec9876e95 2024-08-03T20:46:38.598 INFO:teuthology.orchestra.run.smithi032.stdout:last_changed 2024-08-03T20:46:38.597614+0000 2024-08-03T20:46:38.598 INFO:teuthology.orchestra.run.smithi032.stdout:created 2024-08-03T20:46:38.597614+0000 2024-08-03T20:46:38.598 INFO:teuthology.orchestra.run.smithi032.stdout:min_mon_release 17 (quincy) 2024-08-03T20:46:38.598 INFO:teuthology.orchestra.run.smithi032.stdout:election_strategy: 1 2024-08-03T20:46:38.599 INFO:teuthology.orchestra.run.smithi032.stdout:0: [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] mon.a 2024-08-03T20:46:38.599 INFO:teuthology.orchestra.run.smithi032.stdout:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2024-08-03T20:46:38.599 INFO:teuthology.orchestra.run.smithi032.stdout:2: [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] mon.c 2024-08-03T20:46:38.599 INFO:teuthology.orchestra.run.smithi032.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2024-08-03T20:46:38.607 DEBUG:teuthology.orchestra.run.smithi032:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-08-03T20:46:38.644 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID 40ab141a-01d7-49b9-ae3c-2edec9876e95... 2024-08-03T20:46:38.645 DEBUG:teuthology.orchestra.run.smithi032:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-08-03T20:46:38.694 DEBUG:teuthology.orchestra.run.smithi118:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-08-03T20:46:38.784 INFO:teuthology.orchestra.run.smithi032.stdout:[global] 2024-08-03T20:46:38.785 INFO:teuthology.orchestra.run.smithi032.stdout: chdir = "" 2024-08-03T20:46:38.785 INFO:teuthology.orchestra.run.smithi032.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-08-03T20:46:38.785 INFO:teuthology.orchestra.run.smithi032.stdout: auth supported = cephx 2024-08-03T20:46:38.785 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-03T20:46:38.785 INFO:teuthology.orchestra.run.smithi032.stdout: filestore xattr use omap = true 2024-08-03T20:46:38.785 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-03T20:46:38.785 INFO:teuthology.orchestra.run.smithi032.stdout: mon clock drift allowed = 1.000 2024-08-03T20:46:38.785 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-03T20:46:38.785 INFO:teuthology.orchestra.run.smithi032.stdout: osd crush chooseleaf type = 0 2024-08-03T20:46:38.786 INFO:teuthology.orchestra.run.smithi032.stdout: auth debug = true 2024-08-03T20:46:38.786 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-03T20:46:38.786 INFO:teuthology.orchestra.run.smithi032.stdout: ms die on old message = true 2024-08-03T20:46:38.786 INFO:teuthology.orchestra.run.smithi032.stdout: ms die on bug = true 2024-08-03T20:46:38.786 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-03T20:46:38.786 INFO:teuthology.orchestra.run.smithi032.stdout: mon max pg per osd = 10000 # >= luminous 2024-08-03T20:46:38.786 INFO:teuthology.orchestra.run.smithi032.stdout: mon pg warn max object skew = 0 2024-08-03T20:46:38.786 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-03T20:46:38.786 INFO:teuthology.orchestra.run.smithi032.stdout: # disable pg_autoscaler by default for new pools 2024-08-03T20:46:38.786 INFO:teuthology.orchestra.run.smithi032.stdout: osd_pool_default_pg_autoscale_mode = off 2024-08-03T20:46:38.787 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-03T20:46:38.787 INFO:teuthology.orchestra.run.smithi032.stdout: osd pool default size = 2 2024-08-03T20:46:38.787 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-03T20:46:38.787 INFO:teuthology.orchestra.run.smithi032.stdout: mon osd allow primary affinity = true 2024-08-03T20:46:38.787 INFO:teuthology.orchestra.run.smithi032.stdout: mon osd allow pg remap = true 2024-08-03T20:46:38.787 INFO:teuthology.orchestra.run.smithi032.stdout: mon warn on legacy crush tunables = false 2024-08-03T20:46:38.787 INFO:teuthology.orchestra.run.smithi032.stdout: mon warn on crush straw calc version zero = false 2024-08-03T20:46:38.787 INFO:teuthology.orchestra.run.smithi032.stdout: mon warn on no sortbitwise = false 2024-08-03T20:46:38.787 INFO:teuthology.orchestra.run.smithi032.stdout: mon warn on osd down out interval zero = false 2024-08-03T20:46:38.787 INFO:teuthology.orchestra.run.smithi032.stdout: mon warn on too few osds = false 2024-08-03T20:46:38.787 INFO:teuthology.orchestra.run.smithi032.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-08-03T20:46:38.788 INFO:teuthology.orchestra.run.smithi032.stdout: mon_warn_on_pool_no_redundancy = false 2024-08-03T20:46:38.788 INFO:teuthology.orchestra.run.smithi032.stdout: mon_allow_pool_size_one = true 2024-08-03T20:46:38.788 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-03T20:46:38.788 INFO:teuthology.orchestra.run.smithi032.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-08-03T20:46:38.788 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-03T20:46:38.788 INFO:teuthology.orchestra.run.smithi032.stdout: osd default data pool replay window = 5 2024-08-03T20:46:38.788 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-03T20:46:38.788 INFO:teuthology.orchestra.run.smithi032.stdout: mon allow pool delete = true 2024-08-03T20:46:38.788 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-03T20:46:38.788 INFO:teuthology.orchestra.run.smithi032.stdout: mon cluster log file level = debug 2024-08-03T20:46:38.788 INFO:teuthology.orchestra.run.smithi032.stdout: debug asserts on shutdown = true 2024-08-03T20:46:38.789 INFO:teuthology.orchestra.run.smithi032.stdout: mon health detail to clog = false 2024-08-03T20:46:38.789 INFO:teuthology.orchestra.run.smithi032.stdout: mon host = "172.21.15.32,172.21.15.118,[v2:172.21.15.118:3301,v1:172.21.15.118:6790]" 2024-08-03T20:46:38.789 INFO:teuthology.orchestra.run.smithi032.stdout: ms die on skipped message = False 2024-08-03T20:46:38.789 INFO:teuthology.orchestra.run.smithi032.stdout: fsid = 40ab141a-01d7-49b9-ae3c-2edec9876e95 2024-08-03T20:46:38.789 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-03T20:46:38.789 INFO:teuthology.orchestra.run.smithi032.stdout:[osd] 2024-08-03T20:46:38.789 INFO:teuthology.orchestra.run.smithi032.stdout: osd journal size = 100 2024-08-03T20:46:38.789 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-03T20:46:38.789 INFO:teuthology.orchestra.run.smithi032.stdout: osd scrub load threshold = 5.0 2024-08-03T20:46:38.789 INFO:teuthology.orchestra.run.smithi032.stdout: osd scrub max interval = 600 2024-08-03T20:46:38.790 INFO:teuthology.orchestra.run.smithi032.stdout: osd mclock profile = high_recovery_ops 2024-08-03T20:46:38.790 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-03T20:46:38.790 INFO:teuthology.orchestra.run.smithi032.stdout: osd recover clone overlap = true 2024-08-03T20:46:38.790 INFO:teuthology.orchestra.run.smithi032.stdout: osd recovery max chunk = 1048576 2024-08-03T20:46:38.790 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-03T20:46:38.790 INFO:teuthology.orchestra.run.smithi032.stdout: osd debug shutdown = true 2024-08-03T20:46:38.790 INFO:teuthology.orchestra.run.smithi032.stdout: osd debug op order = true 2024-08-03T20:46:38.790 INFO:teuthology.orchestra.run.smithi032.stdout: osd debug verify stray on activate = true 2024-08-03T20:46:38.790 INFO:teuthology.orchestra.run.smithi032.stdout: osd debug trim objects = true 2024-08-03T20:46:38.790 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-03T20:46:38.790 INFO:teuthology.orchestra.run.smithi032.stdout: osd open classes on start = true 2024-08-03T20:46:38.790 INFO:teuthology.orchestra.run.smithi032.stdout: osd debug pg log writeout = true 2024-08-03T20:46:38.790 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-03T20:46:38.791 INFO:teuthology.orchestra.run.smithi032.stdout: osd deep scrub update digest min age = 30 2024-08-03T20:46:38.791 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-03T20:46:38.791 INFO:teuthology.orchestra.run.smithi032.stdout: osd map max advance = 10 2024-08-03T20:46:38.791 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-03T20:46:38.791 INFO:teuthology.orchestra.run.smithi032.stdout: journal zero on create = true 2024-08-03T20:46:38.791 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-03T20:46:38.791 INFO:teuthology.orchestra.run.smithi032.stdout: filestore ondisk finisher threads = 3 2024-08-03T20:46:38.792 INFO:teuthology.orchestra.run.smithi032.stdout: filestore apply finisher threads = 3 2024-08-03T20:46:38.792 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-03T20:46:38.792 INFO:teuthology.orchestra.run.smithi032.stdout: bdev debug aio = true 2024-08-03T20:46:38.792 INFO:teuthology.orchestra.run.smithi032.stdout: osd debug misdirected ops = true 2024-08-03T20:46:38.792 INFO:teuthology.orchestra.run.smithi032.stdout: bdev async discard = True 2024-08-03T20:46:38.793 INFO:teuthology.orchestra.run.smithi032.stdout: bdev enable discard = True 2024-08-03T20:46:38.793 INFO:teuthology.orchestra.run.smithi032.stdout: bluestore allocator = bitmap 2024-08-03T20:46:38.793 INFO:teuthology.orchestra.run.smithi032.stdout: bluestore block size = 96636764160 2024-08-03T20:46:38.793 INFO:teuthology.orchestra.run.smithi032.stdout: bluestore fsck on mount = True 2024-08-03T20:46:38.793 INFO:teuthology.orchestra.run.smithi032.stdout: debug bluefs = 1/20 2024-08-03T20:46:38.793 INFO:teuthology.orchestra.run.smithi032.stdout: debug bluestore = 1/20 2024-08-03T20:46:38.793 INFO:teuthology.orchestra.run.smithi032.stdout: debug ms = 1 2024-08-03T20:46:38.793 INFO:teuthology.orchestra.run.smithi032.stdout: debug osd = 20 2024-08-03T20:46:38.794 INFO:teuthology.orchestra.run.smithi032.stdout: debug rocksdb = 4/10 2024-08-03T20:46:38.794 INFO:teuthology.orchestra.run.smithi032.stdout: mon osd backfillfull_ratio = 0.85 2024-08-03T20:46:38.794 INFO:teuthology.orchestra.run.smithi032.stdout: mon osd full ratio = 0.9 2024-08-03T20:46:38.794 INFO:teuthology.orchestra.run.smithi032.stdout: mon osd nearfull ratio = 0.8 2024-08-03T20:46:38.794 INFO:teuthology.orchestra.run.smithi032.stdout: osd failsafe full ratio = 0.95 2024-08-03T20:46:38.794 INFO:teuthology.orchestra.run.smithi032.stdout: osd objectstore = bluestore 2024-08-03T20:46:38.794 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-03T20:46:38.794 INFO:teuthology.orchestra.run.smithi032.stdout:[mgr] 2024-08-03T20:46:38.795 INFO:teuthology.orchestra.run.smithi032.stdout: debug ms = 1 2024-08-03T20:46:38.795 INFO:teuthology.orchestra.run.smithi032.stdout: debug mgr = 20 2024-08-03T20:46:38.795 INFO:teuthology.orchestra.run.smithi032.stdout: debug mon = 20 2024-08-03T20:46:38.795 INFO:teuthology.orchestra.run.smithi032.stdout: debug auth = 20 2024-08-03T20:46:38.795 INFO:teuthology.orchestra.run.smithi032.stdout: mon reweight min pgs per osd = 4 2024-08-03T20:46:38.795 INFO:teuthology.orchestra.run.smithi032.stdout: mon reweight min bytes per osd = 10 2024-08-03T20:46:38.795 INFO:teuthology.orchestra.run.smithi032.stdout: mgr/telemetry/nag = false 2024-08-03T20:46:38.795 INFO:teuthology.orchestra.run.smithi032.stdout: client mount timeout = 30 2024-08-03T20:46:38.795 INFO:teuthology.orchestra.run.smithi032.stdout: debug client = 20 2024-08-03T20:46:38.795 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-03T20:46:38.795 INFO:teuthology.orchestra.run.smithi032.stdout:[mon] 2024-08-03T20:46:38.795 INFO:teuthology.orchestra.run.smithi032.stdout: debug ms = 1 2024-08-03T20:46:38.795 INFO:teuthology.orchestra.run.smithi032.stdout: debug mon = 20 2024-08-03T20:46:38.795 INFO:teuthology.orchestra.run.smithi032.stdout: debug paxos = 20 2024-08-03T20:46:38.796 INFO:teuthology.orchestra.run.smithi032.stdout: debug auth = 20 2024-08-03T20:46:38.796 INFO:teuthology.orchestra.run.smithi032.stdout: mon data avail warn = 5 2024-08-03T20:46:38.796 INFO:teuthology.orchestra.run.smithi032.stdout: mon mgr mkfs grace = 240 2024-08-03T20:46:38.796 INFO:teuthology.orchestra.run.smithi032.stdout: mon reweight min pgs per osd = 4 2024-08-03T20:46:38.796 INFO:teuthology.orchestra.run.smithi032.stdout: mon osd reporter subtree level = osd 2024-08-03T20:46:38.796 INFO:teuthology.orchestra.run.smithi032.stdout: mon osd prime pg temp = true 2024-08-03T20:46:38.796 INFO:teuthology.orchestra.run.smithi032.stdout: mon reweight min bytes per osd = 10 2024-08-03T20:46:38.796 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-03T20:46:38.796 INFO:teuthology.orchestra.run.smithi032.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-08-03T20:46:38.796 INFO:teuthology.orchestra.run.smithi032.stdout: auth mon ticket ttl = 660 # 11m 2024-08-03T20:46:38.797 INFO:teuthology.orchestra.run.smithi032.stdout: auth service ticket ttl = 240 # 4m 2024-08-03T20:46:38.797 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-03T20:46:38.797 INFO:teuthology.orchestra.run.smithi032.stdout: # don't complain about insecure global_id in the test suite 2024-08-03T20:46:38.797 INFO:teuthology.orchestra.run.smithi032.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-08-03T20:46:38.797 INFO:teuthology.orchestra.run.smithi032.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-08-03T20:46:38.797 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-03T20:46:38.797 INFO:teuthology.orchestra.run.smithi032.stdout: # 1m isn't quite enough 2024-08-03T20:46:38.797 INFO:teuthology.orchestra.run.smithi032.stdout: mon_down_mkfs_grace = 2m 2024-08-03T20:46:38.797 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-03T20:46:38.797 INFO:teuthology.orchestra.run.smithi032.stdout: mon_warn_on_filestore_osds = false 2024-08-03T20:46:38.797 INFO:teuthology.orchestra.run.smithi032.stdout: mon down mkfs grace = 300 2024-08-03T20:46:38.797 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-03T20:46:38.797 INFO:teuthology.orchestra.run.smithi032.stdout:[client] 2024-08-03T20:46:38.797 INFO:teuthology.orchestra.run.smithi032.stdout: rgw cache enabled = true 2024-08-03T20:46:38.797 INFO:teuthology.orchestra.run.smithi032.stdout: rgw enable ops log = true 2024-08-03T20:46:38.798 INFO:teuthology.orchestra.run.smithi032.stdout: rgw enable usage log = true 2024-08-03T20:46:38.798 INFO:teuthology.orchestra.run.smithi032.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-08-03T20:46:38.798 INFO:teuthology.orchestra.run.smithi032.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-08-03T20:46:38.798 INFO:teuthology.orchestra.run.smithi032.stdout: pid file = "" 2024-08-03T20:46:38.798 INFO:teuthology.orchestra.run.smithi032.stdout:[mon.a] 2024-08-03T20:46:38.798 INFO:teuthology.orchestra.run.smithi032.stdout:[mon.b] 2024-08-03T20:46:38.798 INFO:teuthology.orchestra.run.smithi032.stdout:[mon.c] 2024-08-03T20:46:38.798 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.a] 2024-08-03T20:46:38.798 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.c] 2024-08-03T20:46:38.798 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.b] 2024-08-03T20:46:38.798 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.d] 2024-08-03T20:46:38.799 INFO:teuthology.orchestra.run.smithi118.stdout:[global] 2024-08-03T20:46:38.799 INFO:teuthology.orchestra.run.smithi118.stdout: chdir = "" 2024-08-03T20:46:38.800 INFO:teuthology.orchestra.run.smithi118.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-08-03T20:46:38.800 INFO:teuthology.orchestra.run.smithi118.stdout: auth supported = cephx 2024-08-03T20:46:38.800 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-08-03T20:46:38.800 INFO:teuthology.orchestra.run.smithi118.stdout: filestore xattr use omap = true 2024-08-03T20:46:38.800 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-08-03T20:46:38.800 INFO:teuthology.orchestra.run.smithi118.stdout: mon clock drift allowed = 1.000 2024-08-03T20:46:38.800 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-08-03T20:46:38.800 INFO:teuthology.orchestra.run.smithi118.stdout: osd crush chooseleaf type = 0 2024-08-03T20:46:38.800 INFO:teuthology.orchestra.run.smithi118.stdout: auth debug = true 2024-08-03T20:46:38.800 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-08-03T20:46:38.800 INFO:teuthology.orchestra.run.smithi118.stdout: ms die on old message = true 2024-08-03T20:46:38.800 INFO:teuthology.orchestra.run.smithi118.stdout: ms die on bug = true 2024-08-03T20:46:38.801 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-08-03T20:46:38.801 INFO:teuthology.orchestra.run.smithi118.stdout: mon max pg per osd = 10000 # >= luminous 2024-08-03T20:46:38.801 INFO:teuthology.orchestra.run.smithi118.stdout: mon pg warn max object skew = 0 2024-08-03T20:46:38.801 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-08-03T20:46:38.801 INFO:teuthology.orchestra.run.smithi118.stdout: # disable pg_autoscaler by default for new pools 2024-08-03T20:46:38.801 INFO:teuthology.orchestra.run.smithi118.stdout: osd_pool_default_pg_autoscale_mode = off 2024-08-03T20:46:38.801 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-08-03T20:46:38.801 INFO:teuthology.orchestra.run.smithi118.stdout: osd pool default size = 2 2024-08-03T20:46:38.801 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-08-03T20:46:38.801 INFO:teuthology.orchestra.run.smithi118.stdout: mon osd allow primary affinity = true 2024-08-03T20:46:38.801 INFO:teuthology.orchestra.run.smithi118.stdout: mon osd allow pg remap = true 2024-08-03T20:46:38.801 INFO:teuthology.orchestra.run.smithi118.stdout: mon warn on legacy crush tunables = false 2024-08-03T20:46:38.801 INFO:teuthology.orchestra.run.smithi118.stdout: mon warn on crush straw calc version zero = false 2024-08-03T20:46:38.801 INFO:teuthology.orchestra.run.smithi118.stdout: mon warn on no sortbitwise = false 2024-08-03T20:46:38.802 INFO:teuthology.orchestra.run.smithi118.stdout: mon warn on osd down out interval zero = false 2024-08-03T20:46:38.802 INFO:teuthology.orchestra.run.smithi118.stdout: mon warn on too few osds = false 2024-08-03T20:46:38.802 INFO:teuthology.orchestra.run.smithi118.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-08-03T20:46:38.802 INFO:teuthology.orchestra.run.smithi118.stdout: mon_warn_on_pool_no_redundancy = false 2024-08-03T20:46:38.802 INFO:teuthology.orchestra.run.smithi118.stdout: mon_allow_pool_size_one = true 2024-08-03T20:46:38.802 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-08-03T20:46:38.802 INFO:teuthology.orchestra.run.smithi118.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-08-03T20:46:38.802 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-08-03T20:46:38.802 INFO:teuthology.orchestra.run.smithi118.stdout: osd default data pool replay window = 5 2024-08-03T20:46:38.802 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-08-03T20:46:38.802 INFO:teuthology.orchestra.run.smithi118.stdout: mon allow pool delete = true 2024-08-03T20:46:38.803 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-08-03T20:46:38.803 INFO:teuthology.orchestra.run.smithi118.stdout: mon cluster log file level = debug 2024-08-03T20:46:38.803 INFO:teuthology.orchestra.run.smithi118.stdout: debug asserts on shutdown = true 2024-08-03T20:46:38.803 INFO:teuthology.orchestra.run.smithi118.stdout: mon health detail to clog = false 2024-08-03T20:46:38.803 INFO:teuthology.orchestra.run.smithi118.stdout: mon host = "172.21.15.32,172.21.15.118,[v2:172.21.15.118:3301,v1:172.21.15.118:6790]" 2024-08-03T20:46:38.803 INFO:teuthology.orchestra.run.smithi118.stdout: ms die on skipped message = False 2024-08-03T20:46:38.803 INFO:teuthology.orchestra.run.smithi118.stdout: fsid = 40ab141a-01d7-49b9-ae3c-2edec9876e95 2024-08-03T20:46:38.804 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-08-03T20:46:38.804 INFO:teuthology.orchestra.run.smithi118.stdout:[osd] 2024-08-03T20:46:38.804 INFO:teuthology.orchestra.run.smithi118.stdout: osd journal size = 100 2024-08-03T20:46:38.804 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-08-03T20:46:38.804 INFO:teuthology.orchestra.run.smithi118.stdout: osd scrub load threshold = 5.0 2024-08-03T20:46:38.804 INFO:teuthology.orchestra.run.smithi118.stdout: osd scrub max interval = 600 2024-08-03T20:46:38.804 INFO:teuthology.orchestra.run.smithi118.stdout: osd mclock profile = high_recovery_ops 2024-08-03T20:46:38.804 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-08-03T20:46:38.804 INFO:teuthology.orchestra.run.smithi118.stdout: osd recover clone overlap = true 2024-08-03T20:46:38.804 INFO:teuthology.orchestra.run.smithi118.stdout: osd recovery max chunk = 1048576 2024-08-03T20:46:38.804 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-08-03T20:46:38.805 INFO:teuthology.orchestra.run.smithi118.stdout: osd debug shutdown = true 2024-08-03T20:46:38.805 INFO:teuthology.orchestra.run.smithi118.stdout: osd debug op order = true 2024-08-03T20:46:38.805 INFO:teuthology.orchestra.run.smithi118.stdout: osd debug verify stray on activate = true 2024-08-03T20:46:38.805 INFO:teuthology.orchestra.run.smithi118.stdout: osd debug trim objects = true 2024-08-03T20:46:38.805 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-08-03T20:46:38.805 INFO:teuthology.orchestra.run.smithi118.stdout: osd open classes on start = true 2024-08-03T20:46:38.805 INFO:teuthology.orchestra.run.smithi118.stdout: osd debug pg log writeout = true 2024-08-03T20:46:38.805 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-08-03T20:46:38.805 INFO:teuthology.orchestra.run.smithi118.stdout: osd deep scrub update digest min age = 30 2024-08-03T20:46:38.805 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-08-03T20:46:38.805 INFO:teuthology.orchestra.run.smithi118.stdout: osd map max advance = 10 2024-08-03T20:46:38.805 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-08-03T20:46:38.805 INFO:teuthology.orchestra.run.smithi118.stdout: journal zero on create = true 2024-08-03T20:46:38.805 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-08-03T20:46:38.806 INFO:teuthology.orchestra.run.smithi118.stdout: filestore ondisk finisher threads = 3 2024-08-03T20:46:38.806 INFO:teuthology.orchestra.run.smithi118.stdout: filestore apply finisher threads = 3 2024-08-03T20:46:38.806 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-08-03T20:46:38.806 INFO:teuthology.orchestra.run.smithi118.stdout: bdev debug aio = true 2024-08-03T20:46:38.806 INFO:teuthology.orchestra.run.smithi118.stdout: osd debug misdirected ops = true 2024-08-03T20:46:38.806 INFO:teuthology.orchestra.run.smithi118.stdout: bdev async discard = True 2024-08-03T20:46:38.806 INFO:teuthology.orchestra.run.smithi118.stdout: bdev enable discard = True 2024-08-03T20:46:38.806 INFO:teuthology.orchestra.run.smithi118.stdout: bluestore allocator = bitmap 2024-08-03T20:46:38.806 INFO:teuthology.orchestra.run.smithi118.stdout: bluestore block size = 96636764160 2024-08-03T20:46:38.806 INFO:teuthology.orchestra.run.smithi118.stdout: bluestore fsck on mount = True 2024-08-03T20:46:38.806 INFO:teuthology.orchestra.run.smithi118.stdout: debug bluefs = 1/20 2024-08-03T20:46:38.806 INFO:teuthology.orchestra.run.smithi118.stdout: debug bluestore = 1/20 2024-08-03T20:46:38.806 INFO:teuthology.orchestra.run.smithi118.stdout: debug ms = 1 2024-08-03T20:46:38.806 INFO:teuthology.orchestra.run.smithi118.stdout: debug osd = 20 2024-08-03T20:46:38.807 INFO:teuthology.orchestra.run.smithi118.stdout: debug rocksdb = 4/10 2024-08-03T20:46:38.807 INFO:teuthology.orchestra.run.smithi118.stdout: mon osd backfillfull_ratio = 0.85 2024-08-03T20:46:38.807 INFO:teuthology.orchestra.run.smithi118.stdout: mon osd full ratio = 0.9 2024-08-03T20:46:38.807 INFO:teuthology.orchestra.run.smithi118.stdout: mon osd nearfull ratio = 0.8 2024-08-03T20:46:38.807 INFO:teuthology.orchestra.run.smithi118.stdout: osd failsafe full ratio = 0.95 2024-08-03T20:46:38.807 INFO:teuthology.orchestra.run.smithi118.stdout: osd objectstore = bluestore 2024-08-03T20:46:38.807 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-08-03T20:46:38.807 INFO:teuthology.orchestra.run.smithi118.stdout:[mgr] 2024-08-03T20:46:38.807 INFO:teuthology.orchestra.run.smithi118.stdout: debug ms = 1 2024-08-03T20:46:38.807 INFO:teuthology.orchestra.run.smithi118.stdout: debug mgr = 20 2024-08-03T20:46:38.807 INFO:teuthology.orchestra.run.smithi118.stdout: debug mon = 20 2024-08-03T20:46:38.807 INFO:teuthology.orchestra.run.smithi118.stdout: debug auth = 20 2024-08-03T20:46:38.808 INFO:teuthology.orchestra.run.smithi118.stdout: mon reweight min pgs per osd = 4 2024-08-03T20:46:38.808 INFO:teuthology.orchestra.run.smithi118.stdout: mon reweight min bytes per osd = 10 2024-08-03T20:46:38.808 INFO:teuthology.orchestra.run.smithi118.stdout: mgr/telemetry/nag = false 2024-08-03T20:46:38.808 INFO:teuthology.orchestra.run.smithi118.stdout: client mount timeout = 30 2024-08-03T20:46:38.808 INFO:teuthology.orchestra.run.smithi118.stdout: debug client = 20 2024-08-03T20:46:38.808 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-08-03T20:46:38.808 INFO:teuthology.orchestra.run.smithi118.stdout:[mon] 2024-08-03T20:46:38.808 INFO:teuthology.orchestra.run.smithi118.stdout: debug ms = 1 2024-08-03T20:46:38.808 INFO:teuthology.orchestra.run.smithi118.stdout: debug mon = 20 2024-08-03T20:46:38.809 INFO:teuthology.orchestra.run.smithi118.stdout: debug paxos = 20 2024-08-03T20:46:38.809 INFO:teuthology.orchestra.run.smithi118.stdout: debug auth = 20 2024-08-03T20:46:38.809 INFO:teuthology.orchestra.run.smithi118.stdout: mon data avail warn = 5 2024-08-03T20:46:38.809 INFO:teuthology.orchestra.run.smithi118.stdout: mon mgr mkfs grace = 240 2024-08-03T20:46:38.809 INFO:teuthology.orchestra.run.smithi118.stdout: mon reweight min pgs per osd = 4 2024-08-03T20:46:38.809 INFO:teuthology.orchestra.run.smithi118.stdout: mon osd reporter subtree level = osd 2024-08-03T20:46:38.809 INFO:teuthology.orchestra.run.smithi118.stdout: mon osd prime pg temp = true 2024-08-03T20:46:38.809 INFO:teuthology.orchestra.run.smithi118.stdout: mon reweight min bytes per osd = 10 2024-08-03T20:46:38.809 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-08-03T20:46:38.809 INFO:teuthology.orchestra.run.smithi118.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-08-03T20:46:38.809 INFO:teuthology.orchestra.run.smithi118.stdout: auth mon ticket ttl = 660 # 11m 2024-08-03T20:46:38.809 INFO:teuthology.orchestra.run.smithi118.stdout: auth service ticket ttl = 240 # 4m 2024-08-03T20:46:38.809 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-08-03T20:46:38.810 INFO:teuthology.orchestra.run.smithi118.stdout: # don't complain about insecure global_id in the test suite 2024-08-03T20:46:38.810 INFO:teuthology.orchestra.run.smithi118.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-08-03T20:46:38.810 INFO:teuthology.orchestra.run.smithi118.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-08-03T20:46:38.810 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-08-03T20:46:38.810 INFO:teuthology.orchestra.run.smithi118.stdout: # 1m isn't quite enough 2024-08-03T20:46:38.810 INFO:teuthology.orchestra.run.smithi118.stdout: mon_down_mkfs_grace = 2m 2024-08-03T20:46:38.811 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-08-03T20:46:38.811 INFO:teuthology.orchestra.run.smithi118.stdout: mon_warn_on_filestore_osds = false 2024-08-03T20:46:38.811 INFO:teuthology.orchestra.run.smithi118.stdout: mon down mkfs grace = 300 2024-08-03T20:46:38.811 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-08-03T20:46:38.811 INFO:teuthology.orchestra.run.smithi118.stdout:[client] 2024-08-03T20:46:38.811 INFO:teuthology.orchestra.run.smithi118.stdout: rgw cache enabled = true 2024-08-03T20:46:38.811 INFO:teuthology.orchestra.run.smithi118.stdout: rgw enable ops log = true 2024-08-03T20:46:38.811 INFO:teuthology.orchestra.run.smithi118.stdout: rgw enable usage log = true 2024-08-03T20:46:38.811 INFO:teuthology.orchestra.run.smithi118.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-08-03T20:46:38.811 INFO:teuthology.orchestra.run.smithi118.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-08-03T20:46:38.811 INFO:teuthology.orchestra.run.smithi118.stdout: pid file = "" 2024-08-03T20:46:38.811 INFO:teuthology.orchestra.run.smithi118.stdout:[mon.a] 2024-08-03T20:46:38.812 INFO:teuthology.orchestra.run.smithi118.stdout:[mon.b] 2024-08-03T20:46:38.812 INFO:teuthology.orchestra.run.smithi118.stdout:[mon.c] 2024-08-03T20:46:38.812 INFO:teuthology.orchestra.run.smithi118.stdout:[mds.a] 2024-08-03T20:46:38.812 INFO:teuthology.orchestra.run.smithi118.stdout:[mds.c] 2024-08-03T20:46:38.812 INFO:teuthology.orchestra.run.smithi118.stdout:[mds.b] 2024-08-03T20:46:38.812 INFO:teuthology.orchestra.run.smithi118.stdout:[mds.d] 2024-08-03T20:46:38.820 INFO:tasks.ceph:Creating admin key on mon.a... 2024-08-03T20:46:38.820 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2024-08-03T20:46:38.990 INFO:tasks.ceph:Copying monmap to all nodes... 2024-08-03T20:46:38.990 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-08-03T20:46:38.991 DEBUG:teuthology.orchestra.run.smithi032:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2024-08-03T20:46:39.026 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-08-03T20:46:39.026 DEBUG:teuthology.orchestra.run.smithi032:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2024-08-03T20:46:39.100 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi032.front.sepia.ceph.com 2024-08-03T20:46:39.101 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-08-03T20:46:39.101 DEBUG:teuthology.orchestra.run.smithi032:> sudo dd of=/etc/ceph/ceph.keyring 2024-08-03T20:46:39.101 DEBUG:teuthology.orchestra.run.smithi032:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-08-03T20:46:39.218 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-08-03T20:46:39.218 DEBUG:teuthology.orchestra.run.smithi032:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-08-03T20:46:39.293 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi118.front.sepia.ceph.com 2024-08-03T20:46:39.293 DEBUG:teuthology.orchestra.run.smithi118:> set -ex 2024-08-03T20:46:39.293 DEBUG:teuthology.orchestra.run.smithi118:> sudo dd of=/etc/ceph/ceph.keyring 2024-08-03T20:46:39.293 DEBUG:teuthology.orchestra.run.smithi118:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-08-03T20:46:39.373 DEBUG:teuthology.orchestra.run.smithi118:> set -ex 2024-08-03T20:46:39.373 DEBUG:teuthology.orchestra.run.smithi118:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-08-03T20:46:39.448 INFO:tasks.ceph:Setting up mon nodes... 2024-08-03T20:46:39.448 INFO:tasks.ceph:Setting up mgr nodes... 2024-08-03T20:46:39.449 DEBUG:teuthology.orchestra.run.smithi032:> sudo mkdir -p /var/lib/ceph/mgr/ceph-y && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.y /var/lib/ceph/mgr/ceph-y/keyring 2024-08-03T20:46:39.616 INFO:teuthology.orchestra.run.smithi032.stdout:creating /var/lib/ceph/mgr/ceph-y/keyring 2024-08-03T20:46:39.627 DEBUG:teuthology.orchestra.run.smithi118:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2024-08-03T20:46:39.801 INFO:teuthology.orchestra.run.smithi118.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2024-08-03T20:46:39.815 INFO:tasks.ceph:Setting up mds nodes... 2024-08-03T20:46:39.815 DEBUG:teuthology.orchestra.run.smithi032:> sudo mkdir -p /var/lib/ceph/mds/ceph-a && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.a /var/lib/ceph/mds/ceph-a/keyring 2024-08-03T20:46:39.983 INFO:teuthology.orchestra.run.smithi032.stdout:creating /var/lib/ceph/mds/ceph-a/keyring 2024-08-03T20:46:39.994 DEBUG:teuthology.orchestra.run.smithi032:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-a 2024-08-03T20:46:40.049 DEBUG:teuthology.orchestra.run.smithi032:> sudo mkdir -p /var/lib/ceph/mds/ceph-c && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.c /var/lib/ceph/mds/ceph-c/keyring 2024-08-03T20:46:40.218 INFO:teuthology.orchestra.run.smithi032.stdout:creating /var/lib/ceph/mds/ceph-c/keyring 2024-08-03T20:46:40.229 DEBUG:teuthology.orchestra.run.smithi032:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-c 2024-08-03T20:46:40.285 DEBUG:teuthology.orchestra.run.smithi118:> sudo mkdir -p /var/lib/ceph/mds/ceph-b && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.b /var/lib/ceph/mds/ceph-b/keyring 2024-08-03T20:46:40.454 INFO:teuthology.orchestra.run.smithi118.stdout:creating /var/lib/ceph/mds/ceph-b/keyring 2024-08-03T20:46:40.465 DEBUG:teuthology.orchestra.run.smithi118:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-b 2024-08-03T20:46:40.521 DEBUG:teuthology.orchestra.run.smithi118:> sudo mkdir -p /var/lib/ceph/mds/ceph-d && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.d /var/lib/ceph/mds/ceph-d/keyring 2024-08-03T20:46:40.685 INFO:teuthology.orchestra.run.smithi118.stdout:creating /var/lib/ceph/mds/ceph-d/keyring 2024-08-03T20:46:40.696 DEBUG:teuthology.orchestra.run.smithi118:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-d 2024-08-03T20:46:40.751 INFO:tasks.ceph_client:Setting up client nodes... 2024-08-03T20:46:40.752 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-08-03T20:46:40.901 INFO:teuthology.orchestra.run.smithi032.stdout:creating /etc/ceph/ceph.client.0.keyring 2024-08-03T20:46:40.935 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.2 /etc/ceph/ceph.client.2.keyring && sudo chmod 0644 /etc/ceph/ceph.client.2.keyring 2024-08-03T20:46:41.082 INFO:teuthology.orchestra.run.smithi032.stdout:creating /etc/ceph/ceph.client.2.keyring 2024-08-03T20:46:41.115 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/ceph.client.1.keyring && sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-08-03T20:46:41.259 INFO:teuthology.orchestra.run.smithi118.stdout:creating /etc/ceph/ceph.client.1.keyring 2024-08-03T20:46:41.293 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.3 /etc/ceph/ceph.client.3.keyring && sudo chmod 0644 /etc/ceph/ceph.client.3.keyring 2024-08-03T20:46:41.435 INFO:teuthology.orchestra.run.smithi118.stdout:creating /etc/ceph/ceph.client.3.keyring 2024-08-03T20:46:41.469 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-08-03T20:46:41.470 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi032.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi118.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-08-03T20:46:41.470 DEBUG:teuthology.orchestra.run.smithi032:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2024-08-03T20:46:41.526 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-08-03T20:46:41.526 INFO:tasks.ceph:role: osd.0 2024-08-03T20:46:41.526 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi032.front.sepia.ceph.com 2024-08-03T20:46:41.527 DEBUG:teuthology.orchestra.run.smithi032:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-08-03T20:46:41.581 INFO:teuthology.orchestra.run.smithi032.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-08-03T20:46:41.582 INFO:teuthology.orchestra.run.smithi032.stdout: = sectsz=512 attr=2, projid32bit=1 2024-08-03T20:46:41.582 INFO:teuthology.orchestra.run.smithi032.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-08-03T20:46:41.582 INFO:teuthology.orchestra.run.smithi032.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-08-03T20:46:41.582 INFO:teuthology.orchestra.run.smithi032.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-08-03T20:46:41.582 INFO:teuthology.orchestra.run.smithi032.stdout: = sunit=0 swidth=0 blks 2024-08-03T20:46:41.582 INFO:teuthology.orchestra.run.smithi032.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-08-03T20:46:41.582 INFO:teuthology.orchestra.run.smithi032.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-08-03T20:46:41.582 INFO:teuthology.orchestra.run.smithi032.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-08-03T20:46:41.582 INFO:teuthology.orchestra.run.smithi032.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-08-03T20:46:41.616 INFO:teuthology.orchestra.run.smithi032.stdout:Discarding blocks...Done. 2024-08-03T20:46:41.619 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi032.front.sepia.ceph.com -o noatime 2024-08-03T20:46:41.619 DEBUG:teuthology.orchestra.run.smithi032:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2024-08-03T20:46:41.701 DEBUG:teuthology.orchestra.run.smithi032:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2024-08-03T20:46:41.763 DEBUG:teuthology.orchestra.run.smithi032:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2024-08-03T20:46:41.819 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-08-03T20:46:41.819 INFO:tasks.ceph:role: osd.1 2024-08-03T20:46:41.820 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi032.front.sepia.ceph.com 2024-08-03T20:46:41.820 DEBUG:teuthology.orchestra.run.smithi032:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-08-03T20:46:41.875 INFO:teuthology.orchestra.run.smithi032.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-08-03T20:46:41.875 INFO:teuthology.orchestra.run.smithi032.stdout: = sectsz=512 attr=2, projid32bit=1 2024-08-03T20:46:41.875 INFO:teuthology.orchestra.run.smithi032.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-08-03T20:46:41.876 INFO:teuthology.orchestra.run.smithi032.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-08-03T20:46:41.876 INFO:teuthology.orchestra.run.smithi032.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-08-03T20:46:41.876 INFO:teuthology.orchestra.run.smithi032.stdout: = sunit=0 swidth=0 blks 2024-08-03T20:46:41.876 INFO:teuthology.orchestra.run.smithi032.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-08-03T20:46:41.876 INFO:teuthology.orchestra.run.smithi032.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-08-03T20:46:41.876 INFO:teuthology.orchestra.run.smithi032.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-08-03T20:46:41.877 INFO:teuthology.orchestra.run.smithi032.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-08-03T20:46:41.908 INFO:teuthology.orchestra.run.smithi032.stdout:Discarding blocks...Done. 2024-08-03T20:46:41.912 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi032.front.sepia.ceph.com -o noatime 2024-08-03T20:46:41.912 DEBUG:teuthology.orchestra.run.smithi032:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2024-08-03T20:46:41.991 DEBUG:teuthology.orchestra.run.smithi032:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2024-08-03T20:46:42.052 DEBUG:teuthology.orchestra.run.smithi032:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2024-08-03T20:46:42.108 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-08-03T20:46:42.108 INFO:tasks.ceph:role: osd.2 2024-08-03T20:46:42.108 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi032.front.sepia.ceph.com 2024-08-03T20:46:42.108 DEBUG:teuthology.orchestra.run.smithi032:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-08-03T20:46:42.163 INFO:teuthology.orchestra.run.smithi032.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-08-03T20:46:42.163 INFO:teuthology.orchestra.run.smithi032.stdout: = sectsz=512 attr=2, projid32bit=1 2024-08-03T20:46:42.164 INFO:teuthology.orchestra.run.smithi032.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-08-03T20:46:42.164 INFO:teuthology.orchestra.run.smithi032.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-08-03T20:46:42.164 INFO:teuthology.orchestra.run.smithi032.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-08-03T20:46:42.164 INFO:teuthology.orchestra.run.smithi032.stdout: = sunit=0 swidth=0 blks 2024-08-03T20:46:42.164 INFO:teuthology.orchestra.run.smithi032.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-08-03T20:46:42.164 INFO:teuthology.orchestra.run.smithi032.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-08-03T20:46:42.164 INFO:teuthology.orchestra.run.smithi032.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-08-03T20:46:42.164 INFO:teuthology.orchestra.run.smithi032.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-08-03T20:46:42.196 INFO:teuthology.orchestra.run.smithi032.stdout:Discarding blocks...Done. 2024-08-03T20:46:42.200 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi032.front.sepia.ceph.com -o noatime 2024-08-03T20:46:42.200 DEBUG:teuthology.orchestra.run.smithi032:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2024-08-03T20:46:42.279 DEBUG:teuthology.orchestra.run.smithi032:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2024-08-03T20:46:42.340 DEBUG:teuthology.orchestra.run.smithi032:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2024-08-03T20:46:42.396 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-08-03T20:46:42.396 INFO:tasks.ceph:role: osd.3 2024-08-03T20:46:42.396 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi032.front.sepia.ceph.com 2024-08-03T20:46:42.396 DEBUG:teuthology.orchestra.run.smithi032:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-08-03T20:46:42.451 INFO:teuthology.orchestra.run.smithi032.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-08-03T20:46:42.451 INFO:teuthology.orchestra.run.smithi032.stdout: = sectsz=512 attr=2, projid32bit=1 2024-08-03T20:46:42.451 INFO:teuthology.orchestra.run.smithi032.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-08-03T20:46:42.451 INFO:teuthology.orchestra.run.smithi032.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-08-03T20:46:42.452 INFO:teuthology.orchestra.run.smithi032.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-08-03T20:46:42.452 INFO:teuthology.orchestra.run.smithi032.stdout: = sunit=0 swidth=0 blks 2024-08-03T20:46:42.452 INFO:teuthology.orchestra.run.smithi032.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-08-03T20:46:42.452 INFO:teuthology.orchestra.run.smithi032.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-08-03T20:46:42.452 INFO:teuthology.orchestra.run.smithi032.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-08-03T20:46:42.452 INFO:teuthology.orchestra.run.smithi032.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-08-03T20:46:42.485 INFO:teuthology.orchestra.run.smithi032.stdout:Discarding blocks...Done. 2024-08-03T20:46:42.488 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi032.front.sepia.ceph.com -o noatime 2024-08-03T20:46:42.488 DEBUG:teuthology.orchestra.run.smithi032:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-3 2024-08-03T20:46:42.568 DEBUG:teuthology.orchestra.run.smithi032:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2024-08-03T20:46:42.629 DEBUG:teuthology.orchestra.run.smithi032:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-08-03T20:46:42.791 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:46:42.791+0000 7f9309723640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2024-08-03T20:46:42.792 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:46:42.791+0000 7f9309723640 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2024-08-03T20:46:42.792 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:46:42.792+0000 7f9309723640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-08-03T20:46:42.792 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:46:42.792+0000 7f9309723640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-08-03T20:46:42.793 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:46:42.792+0000 7f9309723640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-08-03T20:46:42.794 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:46:42.794+0000 7f9309723640 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2024-08-03T20:46:45.764 DEBUG:teuthology.orchestra.run.smithi032:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-08-03T20:46:45.821 DEBUG:teuthology.orchestra.run.smithi032:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-08-03T20:46:45.978 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:46:45.976+0000 7fed7d8dd640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2024-08-03T20:46:45.979 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:46:45.977+0000 7fed7d8dd640 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2024-08-03T20:46:45.979 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:46:45.977+0000 7fed7d8dd640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-08-03T20:46:45.979 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:46:45.977+0000 7fed7d8dd640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-08-03T20:46:45.979 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:46:45.977+0000 7fed7d8dd640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-08-03T20:46:45.979 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:46:45.978+0000 7fed7d8dd640 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2024-08-03T20:46:48.911 DEBUG:teuthology.orchestra.run.smithi032:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-08-03T20:46:48.971 DEBUG:teuthology.orchestra.run.smithi032:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-08-03T20:46:49.130 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:46:49.128+0000 7f51da733640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2024-08-03T20:46:49.130 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:46:49.129+0000 7f51da733640 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2024-08-03T20:46:49.130 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:46:49.129+0000 7f51da733640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-08-03T20:46:49.131 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:46:49.129+0000 7f51da733640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-08-03T20:46:49.131 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:46:49.129+0000 7f51da733640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-08-03T20:46:49.131 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:46:49.130+0000 7f51da733640 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2024-08-03T20:46:52.076 DEBUG:teuthology.orchestra.run.smithi032:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-08-03T20:46:52.136 DEBUG:teuthology.orchestra.run.smithi032:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-08-03T20:46:52.296 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:46:52.294+0000 7f669520d640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2024-08-03T20:46:52.296 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:46:52.294+0000 7f669520d640 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2024-08-03T20:46:52.296 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:46:52.294+0000 7f669520d640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-08-03T20:46:52.297 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:46:52.294+0000 7f669520d640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-08-03T20:46:52.297 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:46:52.294+0000 7f669520d640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-08-03T20:46:52.297 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:46:52.296+0000 7f669520d640 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2024-08-03T20:46:55.215 DEBUG:teuthology.orchestra.run.smithi032:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-08-03T20:46:55.275 DEBUG:teuthology.orchestra.run.smithi118:> sudo mkdir -p /var/lib/ceph/osd/ceph-4 2024-08-03T20:46:55.336 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-08-03T20:46:55.336 INFO:tasks.ceph:role: osd.4 2024-08-03T20:46:55.336 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi118.front.sepia.ceph.com 2024-08-03T20:46:55.336 DEBUG:teuthology.orchestra.run.smithi118:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-08-03T20:46:55.392 INFO:teuthology.orchestra.run.smithi118.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-08-03T20:46:55.393 INFO:teuthology.orchestra.run.smithi118.stdout: = sectsz=512 attr=2, projid32bit=1 2024-08-03T20:46:55.393 INFO:teuthology.orchestra.run.smithi118.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-08-03T20:46:55.393 INFO:teuthology.orchestra.run.smithi118.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-08-03T20:46:55.393 INFO:teuthology.orchestra.run.smithi118.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-08-03T20:46:55.393 INFO:teuthology.orchestra.run.smithi118.stdout: = sunit=0 swidth=0 blks 2024-08-03T20:46:55.393 INFO:teuthology.orchestra.run.smithi118.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-08-03T20:46:55.393 INFO:teuthology.orchestra.run.smithi118.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-08-03T20:46:55.393 INFO:teuthology.orchestra.run.smithi118.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-08-03T20:46:55.393 INFO:teuthology.orchestra.run.smithi118.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-08-03T20:46:55.425 INFO:teuthology.orchestra.run.smithi118.stdout:Discarding blocks...Done. 2024-08-03T20:46:55.429 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi118.front.sepia.ceph.com -o noatime 2024-08-03T20:46:55.429 DEBUG:teuthology.orchestra.run.smithi118:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-4 2024-08-03T20:46:55.510 DEBUG:teuthology.orchestra.run.smithi118:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-4 2024-08-03T20:46:55.572 DEBUG:teuthology.orchestra.run.smithi118:> sudo mkdir -p /var/lib/ceph/osd/ceph-5 2024-08-03T20:46:55.628 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-08-03T20:46:55.628 INFO:tasks.ceph:role: osd.5 2024-08-03T20:46:55.628 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi118.front.sepia.ceph.com 2024-08-03T20:46:55.628 DEBUG:teuthology.orchestra.run.smithi118:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-08-03T20:46:55.683 INFO:teuthology.orchestra.run.smithi118.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-08-03T20:46:55.683 INFO:teuthology.orchestra.run.smithi118.stdout: = sectsz=512 attr=2, projid32bit=1 2024-08-03T20:46:55.683 INFO:teuthology.orchestra.run.smithi118.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-08-03T20:46:55.683 INFO:teuthology.orchestra.run.smithi118.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-08-03T20:46:55.683 INFO:teuthology.orchestra.run.smithi118.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-08-03T20:46:55.684 INFO:teuthology.orchestra.run.smithi118.stdout: = sunit=0 swidth=0 blks 2024-08-03T20:46:55.684 INFO:teuthology.orchestra.run.smithi118.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-08-03T20:46:55.684 INFO:teuthology.orchestra.run.smithi118.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-08-03T20:46:55.684 INFO:teuthology.orchestra.run.smithi118.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-08-03T20:46:55.684 INFO:teuthology.orchestra.run.smithi118.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-08-03T20:46:55.716 INFO:teuthology.orchestra.run.smithi118.stdout:Discarding blocks...Done. 2024-08-03T20:46:55.720 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi118.front.sepia.ceph.com -o noatime 2024-08-03T20:46:55.720 DEBUG:teuthology.orchestra.run.smithi118:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-5 2024-08-03T20:46:55.799 DEBUG:teuthology.orchestra.run.smithi118:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-5 2024-08-03T20:46:55.861 DEBUG:teuthology.orchestra.run.smithi118:> sudo mkdir -p /var/lib/ceph/osd/ceph-6 2024-08-03T20:46:55.918 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-08-03T20:46:55.918 INFO:tasks.ceph:role: osd.6 2024-08-03T20:46:55.918 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi118.front.sepia.ceph.com 2024-08-03T20:46:55.919 DEBUG:teuthology.orchestra.run.smithi118:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-08-03T20:46:55.974 INFO:teuthology.orchestra.run.smithi118.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-08-03T20:46:55.974 INFO:teuthology.orchestra.run.smithi118.stdout: = sectsz=512 attr=2, projid32bit=1 2024-08-03T20:46:55.974 INFO:teuthology.orchestra.run.smithi118.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-08-03T20:46:55.974 INFO:teuthology.orchestra.run.smithi118.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-08-03T20:46:55.975 INFO:teuthology.orchestra.run.smithi118.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-08-03T20:46:55.975 INFO:teuthology.orchestra.run.smithi118.stdout: = sunit=0 swidth=0 blks 2024-08-03T20:46:55.975 INFO:teuthology.orchestra.run.smithi118.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-08-03T20:46:55.975 INFO:teuthology.orchestra.run.smithi118.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-08-03T20:46:55.975 INFO:teuthology.orchestra.run.smithi118.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-08-03T20:46:55.975 INFO:teuthology.orchestra.run.smithi118.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-08-03T20:46:56.008 INFO:teuthology.orchestra.run.smithi118.stdout:Discarding blocks...Done. 2024-08-03T20:46:56.012 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi118.front.sepia.ceph.com -o noatime 2024-08-03T20:46:56.012 DEBUG:teuthology.orchestra.run.smithi118:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-6 2024-08-03T20:46:56.094 DEBUG:teuthology.orchestra.run.smithi118:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-6 2024-08-03T20:46:56.156 DEBUG:teuthology.orchestra.run.smithi118:> sudo mkdir -p /var/lib/ceph/osd/ceph-7 2024-08-03T20:46:56.212 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-08-03T20:46:56.212 INFO:tasks.ceph:role: osd.7 2024-08-03T20:46:56.212 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi118.front.sepia.ceph.com 2024-08-03T20:46:56.212 DEBUG:teuthology.orchestra.run.smithi118:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-08-03T20:46:56.268 INFO:teuthology.orchestra.run.smithi118.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-08-03T20:46:56.268 INFO:teuthology.orchestra.run.smithi118.stdout: = sectsz=512 attr=2, projid32bit=1 2024-08-03T20:46:56.268 INFO:teuthology.orchestra.run.smithi118.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-08-03T20:46:56.268 INFO:teuthology.orchestra.run.smithi118.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-08-03T20:46:56.268 INFO:teuthology.orchestra.run.smithi118.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-08-03T20:46:56.268 INFO:teuthology.orchestra.run.smithi118.stdout: = sunit=0 swidth=0 blks 2024-08-03T20:46:56.268 INFO:teuthology.orchestra.run.smithi118.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-08-03T20:46:56.268 INFO:teuthology.orchestra.run.smithi118.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-08-03T20:46:56.268 INFO:teuthology.orchestra.run.smithi118.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-08-03T20:46:56.268 INFO:teuthology.orchestra.run.smithi118.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-08-03T20:46:56.301 INFO:teuthology.orchestra.run.smithi118.stdout:Discarding blocks...Done. 2024-08-03T20:46:56.305 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi118.front.sepia.ceph.com -o noatime 2024-08-03T20:46:56.305 DEBUG:teuthology.orchestra.run.smithi118:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-7 2024-08-03T20:46:56.385 DEBUG:teuthology.orchestra.run.smithi118:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-7 2024-08-03T20:46:56.446 DEBUG:teuthology.orchestra.run.smithi118:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-08-03T20:46:56.626 INFO:teuthology.orchestra.run.smithi118.stderr:2024-08-03T20:46:56.626+0000 7fef757cb640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2024-08-03T20:46:56.626 INFO:teuthology.orchestra.run.smithi118.stderr:2024-08-03T20:46:56.626+0000 7fef757cb640 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2024-08-03T20:46:56.626 INFO:teuthology.orchestra.run.smithi118.stderr:2024-08-03T20:46:56.626+0000 7fef757cb640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-08-03T20:46:56.627 INFO:teuthology.orchestra.run.smithi118.stderr:2024-08-03T20:46:56.626+0000 7fef757cb640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-08-03T20:46:56.627 INFO:teuthology.orchestra.run.smithi118.stderr:2024-08-03T20:46:56.626+0000 7fef757cb640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-08-03T20:46:56.629 INFO:teuthology.orchestra.run.smithi118.stderr:2024-08-03T20:46:56.629+0000 7fef757cb640 -1 bluestore(/var/lib/ceph/osd/ceph-4) _read_fsid unparsable uuid 2024-08-03T20:46:59.540 DEBUG:teuthology.orchestra.run.smithi118:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-08-03T20:46:59.595 DEBUG:teuthology.orchestra.run.smithi118:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-08-03T20:46:59.751 INFO:teuthology.orchestra.run.smithi118.stderr:2024-08-03T20:46:59.750+0000 7fd1a6d57640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2024-08-03T20:46:59.752 INFO:teuthology.orchestra.run.smithi118.stderr:2024-08-03T20:46:59.750+0000 7fd1a6d57640 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2024-08-03T20:46:59.752 INFO:teuthology.orchestra.run.smithi118.stderr:2024-08-03T20:46:59.750+0000 7fd1a6d57640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-08-03T20:46:59.752 INFO:teuthology.orchestra.run.smithi118.stderr:2024-08-03T20:46:59.750+0000 7fd1a6d57640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-08-03T20:46:59.752 INFO:teuthology.orchestra.run.smithi118.stderr:2024-08-03T20:46:59.750+0000 7fd1a6d57640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-08-03T20:46:59.753 INFO:teuthology.orchestra.run.smithi118.stderr:2024-08-03T20:46:59.752+0000 7fd1a6d57640 -1 bluestore(/var/lib/ceph/osd/ceph-5) _read_fsid unparsable uuid 2024-08-03T20:47:02.681 DEBUG:teuthology.orchestra.run.smithi118:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-08-03T20:47:02.739 DEBUG:teuthology.orchestra.run.smithi118:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 6 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-08-03T20:47:02.894 INFO:teuthology.orchestra.run.smithi118.stderr:2024-08-03T20:47:02.893+0000 7f8726c38640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-6/keyring: can't open /var/lib/ceph/osd/ceph-6/keyring: (2) No such file or directory 2024-08-03T20:47:02.894 INFO:teuthology.orchestra.run.smithi118.stderr:2024-08-03T20:47:02.893+0000 7f8726c38640 -1 created new key in keyring /var/lib/ceph/osd/ceph-6/keyring 2024-08-03T20:47:02.895 INFO:teuthology.orchestra.run.smithi118.stderr:2024-08-03T20:47:02.893+0000 7f8726c38640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-08-03T20:47:02.895 INFO:teuthology.orchestra.run.smithi118.stderr:2024-08-03T20:47:02.893+0000 7f8726c38640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-08-03T20:47:02.895 INFO:teuthology.orchestra.run.smithi118.stderr:2024-08-03T20:47:02.893+0000 7f8726c38640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-08-03T20:47:02.896 INFO:teuthology.orchestra.run.smithi118.stderr:2024-08-03T20:47:02.895+0000 7f8726c38640 -1 bluestore(/var/lib/ceph/osd/ceph-6) _read_fsid unparsable uuid 2024-08-03T20:47:05.798 DEBUG:teuthology.orchestra.run.smithi118:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-08-03T20:47:05.855 DEBUG:teuthology.orchestra.run.smithi118:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 7 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-08-03T20:47:06.010 INFO:teuthology.orchestra.run.smithi118.stderr:2024-08-03T20:47:06.009+0000 7f158fc43640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-7/keyring: can't open /var/lib/ceph/osd/ceph-7/keyring: (2) No such file or directory 2024-08-03T20:47:06.011 INFO:teuthology.orchestra.run.smithi118.stderr:2024-08-03T20:47:06.009+0000 7f158fc43640 -1 created new key in keyring /var/lib/ceph/osd/ceph-7/keyring 2024-08-03T20:47:06.011 INFO:teuthology.orchestra.run.smithi118.stderr:2024-08-03T20:47:06.009+0000 7f158fc43640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-08-03T20:47:06.011 INFO:teuthology.orchestra.run.smithi118.stderr:2024-08-03T20:47:06.009+0000 7f158fc43640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-08-03T20:47:06.011 INFO:teuthology.orchestra.run.smithi118.stderr:2024-08-03T20:47:06.009+0000 7f158fc43640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-08-03T20:47:06.011 INFO:teuthology.orchestra.run.smithi118.stderr:2024-08-03T20:47:06.011+0000 7f158fc43640 -1 bluestore(/var/lib/ceph/osd/ceph-7) _read_fsid unparsable uuid 2024-08-03T20:47:06.108 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-08-03T20:47:06.113 DEBUG:teuthology.orchestra.run.smithi118:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-08-03T20:47:08.950 DEBUG:teuthology.orchestra.run.smithi118:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-08-03T20:47:09.006 INFO:tasks.ceph:Reading keys from all nodes... 2024-08-03T20:47:09.006 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-08-03T20:47:09.006 DEBUG:teuthology.orchestra.run.smithi032:> sudo dd if=/var/lib/ceph/mgr/ceph-y/keyring of=/dev/stdout 2024-08-03T20:47:09.069 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-08-03T20:47:09.069 DEBUG:teuthology.orchestra.run.smithi032:> sudo dd if=/var/lib/ceph/mds/ceph-a/keyring of=/dev/stdout 2024-08-03T20:47:09.169 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-08-03T20:47:09.169 DEBUG:teuthology.orchestra.run.smithi032:> sudo dd if=/var/lib/ceph/mds/ceph-c/keyring of=/dev/stdout 2024-08-03T20:47:09.267 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-08-03T20:47:09.267 DEBUG:teuthology.orchestra.run.smithi032:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2024-08-03T20:47:09.363 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-08-03T20:47:09.363 DEBUG:teuthology.orchestra.run.smithi032:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2024-08-03T20:47:09.462 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-08-03T20:47:09.462 DEBUG:teuthology.orchestra.run.smithi032:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2024-08-03T20:47:09.557 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-08-03T20:47:09.557 DEBUG:teuthology.orchestra.run.smithi032:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2024-08-03T20:47:09.652 DEBUG:teuthology.orchestra.run.smithi118:> set -ex 2024-08-03T20:47:09.652 DEBUG:teuthology.orchestra.run.smithi118:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2024-08-03T20:47:09.708 DEBUG:teuthology.orchestra.run.smithi118:> set -ex 2024-08-03T20:47:09.708 DEBUG:teuthology.orchestra.run.smithi118:> sudo dd if=/var/lib/ceph/mds/ceph-b/keyring of=/dev/stdout 2024-08-03T20:47:09.804 DEBUG:teuthology.orchestra.run.smithi118:> set -ex 2024-08-03T20:47:09.805 DEBUG:teuthology.orchestra.run.smithi118:> sudo dd if=/var/lib/ceph/mds/ceph-d/keyring of=/dev/stdout 2024-08-03T20:47:09.900 DEBUG:teuthology.orchestra.run.smithi118:> set -ex 2024-08-03T20:47:09.900 DEBUG:teuthology.orchestra.run.smithi118:> sudo dd if=/var/lib/ceph/osd/ceph-4/keyring of=/dev/stdout 2024-08-03T20:47:09.997 DEBUG:teuthology.orchestra.run.smithi118:> set -ex 2024-08-03T20:47:09.997 DEBUG:teuthology.orchestra.run.smithi118:> sudo dd if=/var/lib/ceph/osd/ceph-5/keyring of=/dev/stdout 2024-08-03T20:47:10.092 DEBUG:teuthology.orchestra.run.smithi118:> set -ex 2024-08-03T20:47:10.092 DEBUG:teuthology.orchestra.run.smithi118:> sudo dd if=/var/lib/ceph/osd/ceph-6/keyring of=/dev/stdout 2024-08-03T20:47:10.187 DEBUG:teuthology.orchestra.run.smithi118:> set -ex 2024-08-03T20:47:10.187 DEBUG:teuthology.orchestra.run.smithi118:> sudo dd if=/var/lib/ceph/osd/ceph-7/keyring of=/dev/stdout 2024-08-03T20:47:10.281 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-08-03T20:47:10.281 DEBUG:teuthology.orchestra.run.smithi032:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2024-08-03T20:47:10.321 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-08-03T20:47:10.321 DEBUG:teuthology.orchestra.run.smithi032:> dd if=/etc/ceph/ceph.client.2.keyring of=/dev/stdout 2024-08-03T20:47:10.398 DEBUG:teuthology.orchestra.run.smithi118:> set -ex 2024-08-03T20:47:10.398 DEBUG:teuthology.orchestra.run.smithi118:> dd if=/etc/ceph/ceph.client.1.keyring of=/dev/stdout 2024-08-03T20:47:10.433 DEBUG:teuthology.orchestra.run.smithi118:> set -ex 2024-08-03T20:47:10.433 DEBUG:teuthology.orchestra.run.smithi118:> dd if=/etc/ceph/ceph.client.3.keyring of=/dev/stdout 2024-08-03T20:47:10.508 INFO:tasks.ceph:Adding keys to all mons... 2024-08-03T20:47:10.508 DEBUG:teuthology.orchestra.run.smithi032:> sudo tee -a /etc/ceph/ceph.keyring 2024-08-03T20:47:10.512 DEBUG:teuthology.orchestra.run.smithi118:> sudo tee -a /etc/ceph/ceph.keyring 2024-08-03T20:47:10.564 INFO:teuthology.orchestra.run.smithi032.stdout:[mgr.y] 2024-08-03T20:47:10.564 INFO:teuthology.orchestra.run.smithi032.stdout: key = AQAvl65m5Gq4JBAA306Ge1zHF3Z0WIzUrset4A== 2024-08-03T20:47:10.564 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.a] 2024-08-03T20:47:10.564 INFO:teuthology.orchestra.run.smithi032.stdout: key = AQAvl65mfaKbOhAAUgCH82U0SB7owF5Cds3XYA== 2024-08-03T20:47:10.565 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.c] 2024-08-03T20:47:10.565 INFO:teuthology.orchestra.run.smithi032.stdout: key = AQAwl65mEUAFDRAAGdw1cymfvfKxGeJS/Od6Rw== 2024-08-03T20:47:10.565 INFO:teuthology.orchestra.run.smithi032.stdout:[osd.0] 2024-08-03T20:47:10.565 INFO:teuthology.orchestra.run.smithi032.stdout: key = AQAyl65m4kIyLxAAdCh0D2MjJNPPu5b7jNkrbw== 2024-08-03T20:47:10.565 INFO:teuthology.orchestra.run.smithi032.stdout:[osd.1] 2024-08-03T20:47:10.565 INFO:teuthology.orchestra.run.smithi032.stdout: key = AQA1l65mLzVOOhAA081V1cUyYFPz17L0nny7gA== 2024-08-03T20:47:10.565 INFO:teuthology.orchestra.run.smithi032.stdout:[osd.2] 2024-08-03T20:47:10.565 INFO:teuthology.orchestra.run.smithi032.stdout: key = AQA5l65mBnXDBxAA9/lZVkwD2ykMTOIysoYODg== 2024-08-03T20:47:10.565 INFO:teuthology.orchestra.run.smithi032.stdout:[osd.3] 2024-08-03T20:47:10.565 INFO:teuthology.orchestra.run.smithi032.stdout: key = AQA8l65mAriiERAAfzV8An67+PnA3EzeAutIbA== 2024-08-03T20:47:10.565 INFO:teuthology.orchestra.run.smithi032.stdout:[mgr.x] 2024-08-03T20:47:10.565 INFO:teuthology.orchestra.run.smithi032.stdout: key = AQAvl65mUaPKLxAAnll74Gmk8nSuN/HuX6rTxA== 2024-08-03T20:47:10.565 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.b] 2024-08-03T20:47:10.565 INFO:teuthology.orchestra.run.smithi032.stdout: key = AQAwl65mNcoVGxAAlRXNjZhxVu8Vjslk9Msrxw== 2024-08-03T20:47:10.566 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.d] 2024-08-03T20:47:10.566 INFO:teuthology.orchestra.run.smithi032.stdout: key = AQAwl65mjUHZKBAAqOCnIopBK8Rhk36hrlNnzw== 2024-08-03T20:47:10.566 INFO:teuthology.orchestra.run.smithi032.stdout:[osd.4] 2024-08-03T20:47:10.566 INFO:teuthology.orchestra.run.smithi032.stdout: key = AQBAl65m5uJSJRAA8S2IjJpxuqHY8YK4tTDMPg== 2024-08-03T20:47:10.566 INFO:teuthology.orchestra.run.smithi032.stdout:[osd.5] 2024-08-03T20:47:10.566 INFO:teuthology.orchestra.run.smithi032.stdout: key = AQBDl65mBP3JLBAADgK2X3zNyfnjjZADX1PFcA== 2024-08-03T20:47:10.566 INFO:teuthology.orchestra.run.smithi032.stdout:[osd.6] 2024-08-03T20:47:10.566 INFO:teuthology.orchestra.run.smithi032.stdout: key = AQBGl65m8DBNNRAANeOUfW70S5JfqFlpuMF9KQ== 2024-08-03T20:47:10.566 INFO:teuthology.orchestra.run.smithi032.stdout:[osd.7] 2024-08-03T20:47:10.566 INFO:teuthology.orchestra.run.smithi032.stdout: key = AQBKl65mUnOcABAA41wylzC1Ec9Fj01Ke0HhTw== 2024-08-03T20:47:10.566 INFO:teuthology.orchestra.run.smithi032.stdout:[client.0] 2024-08-03T20:47:10.566 INFO:teuthology.orchestra.run.smithi032.stdout: key = AQAwl65msgO3NRAAHG/WuMTK9c4lDXocXvqWnA== 2024-08-03T20:47:10.566 INFO:teuthology.orchestra.run.smithi032.stdout:[client.2] 2024-08-03T20:47:10.567 INFO:teuthology.orchestra.run.smithi032.stdout: key = AQAxl65mBnbkBBAAjor60WWLTKXidiBMgtKuEA== 2024-08-03T20:47:10.567 INFO:teuthology.orchestra.run.smithi032.stdout:[client.1] 2024-08-03T20:47:10.567 INFO:teuthology.orchestra.run.smithi032.stdout: key = AQAxl65mbe19DxAA+sepc9ucXk2Bqy7C90mfbA== 2024-08-03T20:47:10.567 INFO:teuthology.orchestra.run.smithi032.stdout:[client.3] 2024-08-03T20:47:10.567 INFO:teuthology.orchestra.run.smithi032.stdout: key = AQAxl65munz8GRAA/cRl3ghwQE42FOizurcW3g== 2024-08-03T20:47:10.599 INFO:teuthology.orchestra.run.smithi118.stdout:[mgr.y] 2024-08-03T20:47:10.599 INFO:teuthology.orchestra.run.smithi118.stdout: key = AQAvl65m5Gq4JBAA306Ge1zHF3Z0WIzUrset4A== 2024-08-03T20:47:10.599 INFO:teuthology.orchestra.run.smithi118.stdout:[mds.a] 2024-08-03T20:47:10.600 INFO:teuthology.orchestra.run.smithi118.stdout: key = AQAvl65mfaKbOhAAUgCH82U0SB7owF5Cds3XYA== 2024-08-03T20:47:10.600 INFO:teuthology.orchestra.run.smithi118.stdout:[mds.c] 2024-08-03T20:47:10.600 INFO:teuthology.orchestra.run.smithi118.stdout: key = AQAwl65mEUAFDRAAGdw1cymfvfKxGeJS/Od6Rw== 2024-08-03T20:47:10.600 INFO:teuthology.orchestra.run.smithi118.stdout:[osd.0] 2024-08-03T20:47:10.600 INFO:teuthology.orchestra.run.smithi118.stdout: key = AQAyl65m4kIyLxAAdCh0D2MjJNPPu5b7jNkrbw== 2024-08-03T20:47:10.600 INFO:teuthology.orchestra.run.smithi118.stdout:[osd.1] 2024-08-03T20:47:10.600 INFO:teuthology.orchestra.run.smithi118.stdout: key = AQA1l65mLzVOOhAA081V1cUyYFPz17L0nny7gA== 2024-08-03T20:47:10.600 INFO:teuthology.orchestra.run.smithi118.stdout:[osd.2] 2024-08-03T20:47:10.600 INFO:teuthology.orchestra.run.smithi118.stdout: key = AQA5l65mBnXDBxAA9/lZVkwD2ykMTOIysoYODg== 2024-08-03T20:47:10.600 INFO:teuthology.orchestra.run.smithi118.stdout:[osd.3] 2024-08-03T20:47:10.600 INFO:teuthology.orchestra.run.smithi118.stdout: key = AQA8l65mAriiERAAfzV8An67+PnA3EzeAutIbA== 2024-08-03T20:47:10.600 INFO:teuthology.orchestra.run.smithi118.stdout:[mgr.x] 2024-08-03T20:47:10.600 INFO:teuthology.orchestra.run.smithi118.stdout: key = AQAvl65mUaPKLxAAnll74Gmk8nSuN/HuX6rTxA== 2024-08-03T20:47:10.600 INFO:teuthology.orchestra.run.smithi118.stdout:[mds.b] 2024-08-03T20:47:10.600 INFO:teuthology.orchestra.run.smithi118.stdout: key = AQAwl65mNcoVGxAAlRXNjZhxVu8Vjslk9Msrxw== 2024-08-03T20:47:10.601 INFO:teuthology.orchestra.run.smithi118.stdout:[mds.d] 2024-08-03T20:47:10.601 INFO:teuthology.orchestra.run.smithi118.stdout: key = AQAwl65mjUHZKBAAqOCnIopBK8Rhk36hrlNnzw== 2024-08-03T20:47:10.601 INFO:teuthology.orchestra.run.smithi118.stdout:[osd.4] 2024-08-03T20:47:10.601 INFO:teuthology.orchestra.run.smithi118.stdout: key = AQBAl65m5uJSJRAA8S2IjJpxuqHY8YK4tTDMPg== 2024-08-03T20:47:10.601 INFO:teuthology.orchestra.run.smithi118.stdout:[osd.5] 2024-08-03T20:47:10.601 INFO:teuthology.orchestra.run.smithi118.stdout: key = AQBDl65mBP3JLBAADgK2X3zNyfnjjZADX1PFcA== 2024-08-03T20:47:10.601 INFO:teuthology.orchestra.run.smithi118.stdout:[osd.6] 2024-08-03T20:47:10.601 INFO:teuthology.orchestra.run.smithi118.stdout: key = AQBGl65m8DBNNRAANeOUfW70S5JfqFlpuMF9KQ== 2024-08-03T20:47:10.601 INFO:teuthology.orchestra.run.smithi118.stdout:[osd.7] 2024-08-03T20:47:10.601 INFO:teuthology.orchestra.run.smithi118.stdout: key = AQBKl65mUnOcABAA41wylzC1Ec9Fj01Ke0HhTw== 2024-08-03T20:47:10.601 INFO:teuthology.orchestra.run.smithi118.stdout:[client.0] 2024-08-03T20:47:10.601 INFO:teuthology.orchestra.run.smithi118.stdout: key = AQAwl65msgO3NRAAHG/WuMTK9c4lDXocXvqWnA== 2024-08-03T20:47:10.601 INFO:teuthology.orchestra.run.smithi118.stdout:[client.2] 2024-08-03T20:47:10.602 INFO:teuthology.orchestra.run.smithi118.stdout: key = AQAxl65mBnbkBBAAjor60WWLTKXidiBMgtKuEA== 2024-08-03T20:47:10.602 INFO:teuthology.orchestra.run.smithi118.stdout:[client.1] 2024-08-03T20:47:10.602 INFO:teuthology.orchestra.run.smithi118.stdout: key = AQAxl65mbe19DxAA+sepc9ucXk2Bqy7C90mfbA== 2024-08-03T20:47:10.602 INFO:teuthology.orchestra.run.smithi118.stdout:[client.3] 2024-08-03T20:47:10.602 INFO:teuthology.orchestra.run.smithi118.stdout: key = AQAxl65munz8GRAA/cRl3ghwQE42FOizurcW3g== 2024-08-03T20:47:10.603 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-08-03T20:47:10.613 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-08-03T20:47:10.803 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.a --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-08-03T20:47:10.807 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.a --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-08-03T20:47:10.972 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.c --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-08-03T20:47:10.976 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.c --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-08-03T20:47:11.140 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-08-03T20:47:11.144 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-08-03T20:47:11.308 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-08-03T20:47:11.312 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-08-03T20:47:11.473 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-08-03T20:47:11.477 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-08-03T20:47:11.638 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-08-03T20:47:11.642 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-08-03T20:47:11.806 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-08-03T20:47:11.810 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-08-03T20:47:11.972 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.b --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-08-03T20:47:11.976 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.b --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-08-03T20:47:12.142 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.d --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-08-03T20:47:12.146 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.d --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-08-03T20:47:12.311 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-08-03T20:47:12.316 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-08-03T20:47:12.478 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-08-03T20:47:12.482 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-08-03T20:47:12.645 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-08-03T20:47:12.649 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-08-03T20:47:12.811 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-08-03T20:47:12.815 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-08-03T20:47:12.978 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-08-03T20:47:12.982 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-08-03T20:47:13.146 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.2 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-08-03T20:47:13.151 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.2 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-08-03T20:47:13.316 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-08-03T20:47:13.320 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-08-03T20:47:13.484 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.3 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-08-03T20:47:13.487 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.3 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-08-03T20:47:13.652 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-08-03T20:47:13.652 DEBUG:teuthology.orchestra.run.smithi032:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2024-08-03T20:47:13.708 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-08-03T20:47:13.887 DEBUG:teuthology.orchestra.run.smithi032:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2024-08-03T20:47:13.943 DEBUG:teuthology.orchestra.run.smithi118:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2024-08-03T20:47:14.000 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-08-03T20:47:14.176 DEBUG:teuthology.orchestra.run.smithi118:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2024-08-03T20:47:14.247 DEBUG:teuthology.orchestra.run.smithi118:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2024-08-03T20:47:14.305 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-08-03T20:47:14.482 DEBUG:teuthology.orchestra.run.smithi118:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2024-08-03T20:47:14.539 DEBUG:teuthology.orchestra.run.smithi032:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-08-03T20:47:14.544 DEBUG:teuthology.orchestra.run.smithi118:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-08-03T20:47:14.580 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2024-08-03T20:47:14.581 INFO:tasks.ceph.mon.a:Restarting daemon 2024-08-03T20:47:14.581 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-08-03T20:47:14.623 INFO:tasks.ceph.mon.a:Started 2024-08-03T20:47:14.623 INFO:tasks.ceph.mon.b:Restarting daemon 2024-08-03T20:47:14.624 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2024-08-03T20:47:14.627 INFO:tasks.ceph.mon.b:Started 2024-08-03T20:47:14.627 INFO:tasks.ceph.mon.c:Restarting daemon 2024-08-03T20:47:14.627 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2024-08-03T20:47:14.631 INFO:tasks.ceph.mon.c:Started 2024-08-03T20:47:14.631 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2024-08-03T20:47:14.631 INFO:tasks.ceph.mgr.y:Restarting daemon 2024-08-03T20:47:14.632 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i y 2024-08-03T20:47:14.635 INFO:tasks.ceph.mgr.y:Started 2024-08-03T20:47:14.635 INFO:tasks.ceph.mgr.x:Restarting daemon 2024-08-03T20:47:14.635 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2024-08-03T20:47:14.640 INFO:tasks.ceph.mgr.x:Started 2024-08-03T20:47:14.640 INFO:tasks.ceph:setting config [client] client_mount_timeout = 600 2024-08-03T20:47:14.640 DEBUG:teuthology.orchestra.run.smithi032:> sudo ceph --cluster ceph config set client client_mount_timeout 600 2024-08-03T20:47:14.643 INFO:tasks.ceph:setting config [client] debug_client = 20 2024-08-03T20:47:14.644 DEBUG:teuthology.orchestra.run.smithi032:> sudo ceph --cluster ceph config set client debug_client 20 2024-08-03T20:47:14.689 INFO:tasks.ceph:setting config [client] debug_ms = 1 2024-08-03T20:47:14.689 DEBUG:teuthology.orchestra.run.smithi032:> sudo ceph --cluster ceph config set client debug_ms 1 2024-08-03T20:47:14.735 INFO:tasks.ceph:setting config [client] rados_mon_op_timeout = 900 2024-08-03T20:47:14.735 DEBUG:teuthology.orchestra.run.smithi032:> sudo ceph --cluster ceph config set client rados_mon_op_timeout 900 2024-08-03T20:47:14.782 INFO:tasks.ceph:setting config [client] rados_osd_op_timeout = 900 2024-08-03T20:47:14.782 DEBUG:teuthology.orchestra.run.smithi032:> sudo ceph --cluster ceph config set client rados_osd_op_timeout 900 2024-08-03T20:47:14.828 INFO:tasks.ceph:setting config [mds] debug_mds = 20 2024-08-03T20:47:14.829 DEBUG:teuthology.orchestra.run.smithi032:> sudo ceph --cluster ceph config set mds debug_mds 20 2024-08-03T20:47:14.878 INFO:tasks.ceph:setting config [mds] debug_mds_balancer = 20 2024-08-03T20:47:14.878 DEBUG:teuthology.orchestra.run.smithi032:> sudo ceph --cluster ceph config set mds debug_mds_balancer 20 2024-08-03T20:47:14.928 INFO:tasks.ceph:setting config [mds] debug_ms = 2 2024-08-03T20:47:14.928 DEBUG:teuthology.orchestra.run.smithi032:> sudo ceph --cluster ceph config set mds debug_ms 2 2024-08-03T20:47:14.976 INFO:tasks.ceph:setting config [mds] mds_debug_frag = True 2024-08-03T20:47:14.976 DEBUG:teuthology.orchestra.run.smithi032:> sudo ceph --cluster ceph config set mds mds_debug_frag True 2024-08-03T20:47:15.028 INFO:tasks.ceph:setting config [mds] mds_debug_scatterstat = True 2024-08-03T20:47:15.028 DEBUG:teuthology.orchestra.run.smithi032:> sudo ceph --cluster ceph config set mds mds_debug_scatterstat True 2024-08-03T20:47:15.080 INFO:tasks.ceph:setting config [mds] mds_op_complaint_time = 180 2024-08-03T20:47:15.081 DEBUG:teuthology.orchestra.run.smithi032:> sudo ceph --cluster ceph config set mds mds_op_complaint_time 180 2024-08-03T20:47:15.137 INFO:tasks.ceph:setting config [mds] mds_verify_scatter = True 2024-08-03T20:47:15.137 DEBUG:teuthology.orchestra.run.smithi032:> sudo ceph --cluster ceph config set mds mds_verify_scatter True 2024-08-03T20:47:15.187 INFO:tasks.ceph:setting config [mds] osd_op_complaint_time = 180 2024-08-03T20:47:15.187 DEBUG:teuthology.orchestra.run.smithi032:> sudo ceph --cluster ceph config set mds osd_op_complaint_time 180 2024-08-03T20:47:15.257 INFO:tasks.ceph:setting config [mds] rados_mon_op_timeout = 900 2024-08-03T20:47:15.257 DEBUG:teuthology.orchestra.run.smithi032:> sudo ceph --cluster ceph config set mds rados_mon_op_timeout 900 2024-08-03T20:47:15.315 INFO:tasks.ceph:setting config [mds] rados_osd_op_timeout = 900 2024-08-03T20:47:15.315 DEBUG:teuthology.orchestra.run.smithi032:> sudo ceph --cluster ceph config set mds rados_osd_op_timeout 900 2024-08-03T20:47:15.366 INFO:tasks.ceph:setting config [mon] mon_op_complaint_time = 120 2024-08-03T20:47:15.367 DEBUG:teuthology.orchestra.run.smithi032:> sudo ceph --cluster ceph config set mon mon_op_complaint_time 120 2024-08-03T20:47:15.419 INFO:tasks.ceph:setting config [mon] mon_warn_on_pool_no_app = False 2024-08-03T20:47:15.420 DEBUG:teuthology.orchestra.run.smithi032:> sudo ceph --cluster ceph config set mon mon_warn_on_pool_no_app False 2024-08-03T20:47:15.440 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:15.440+0000 7f22dca00640 1 -- 172.21.15.32:0/477661341 <== mon.2 v2:172.21.15.118:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22d0006330 con 0x7f22c4004800 2024-08-03T20:47:15.455 INFO:tasks.ceph:setting config [osd] osd_op_complaint_time = 180 2024-08-03T20:47:15.455 DEBUG:teuthology.orchestra.run.smithi032:> sudo ceph --cluster ceph config set osd osd_op_complaint_time 180 2024-08-03T20:47:15.456 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:15.443+0000 7ffb53400640 1 -- 172.21.15.32:0/1320847939 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ffb64073940 msgr2=0x7ffb64136980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:47:15.456 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:15.443+0000 7ffb53400640 1 --2- 172.21.15.32:0/1320847939 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ffb64073940 0x7ffb64136980 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7ffb5800b5c0 tx=0x7ffb5800ba80 comp rx=0 tx=0).stop 2024-08-03T20:47:15.456 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:15.443+0000 7ffb53400640 1 -- 172.21.15.32:0/1320847939 shutdown_connections 2024-08-03T20:47:15.457 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:15.443+0000 7ffb53400640 1 --2- 172.21.15.32:0/1320847939 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ffb64073940 0x7ffb64136980 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:15.457 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:15.443+0000 7ffb53400640 1 --2- 172.21.15.32:0/1320847939 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ffb640af080 0x7ffb6414acf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:15.457 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:15.443+0000 7ffb53400640 1 --2- 172.21.15.32:0/1320847939 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7ffb64070120 0x7ffb641462c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:15.457 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:15.443+0000 7ffb53400640 1 -- 172.21.15.32:0/1320847939 >> 172.21.15.32:0/1320847939 conn(0x7ffb640cff10 msgr2=0x7ffb640b4360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:47:15.457 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:15.443+0000 7ffb53400640 1 -- 172.21.15.32:0/1320847939 shutdown_connections 2024-08-03T20:47:15.457 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:15.443+0000 7ffb53400640 1 -- 172.21.15.32:0/1320847939 wait complete. 2024-08-03T20:47:15.457 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:15.441+0000 7f22df200640 1 -- 172.21.15.32:0/477661341 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f22c4004800 msgr2=0x7f22c4004c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:47:15.457 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:15.441+0000 7f22df200640 1 --2- 172.21.15.32:0/477661341 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f22c4004800 0x7f22c4004c10 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f22d000c6f0 tx=0x7f22c4005830 comp rx=0 tx=0).stop 2024-08-03T20:47:15.458 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:15.441+0000 7f22df200640 1 -- 172.21.15.32:0/477661341 shutdown_connections 2024-08-03T20:47:15.458 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:15.441+0000 7f22df200640 1 --2- 172.21.15.32:0/477661341 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f22c4004800 0x7f22c4004c10 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:15.458 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:15.441+0000 7f22df200640 1 -- 172.21.15.32:0/477661341 >> 172.21.15.32:0/477661341 conn(0x7f22d801db80 msgr2=0x7f22d8039840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:47:15.458 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:15.452+0000 7f22df200640 1 -- 172.21.15.32:0/477661341 shutdown_connections 2024-08-03T20:47:15.458 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:15.452+0000 7f22df200640 1 -- 172.21.15.32:0/477661341 wait complete. 2024-08-03T20:47:15.468 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:15.465+0000 7f22df200640 1 Processor -- start 2024-08-03T20:47:15.491 DEBUG:tasks.ceph:set 18 configs 2024-08-03T20:47:15.492 DEBUG:tasks.ceph:waiting for RemoteProcess(client=, args=['sudo', 'ceph', '--cluster', 'ceph', 'config', 'set', 'client', 'client_mount_timeout', '600'], check_status=True, hostname='smithi032') 2024-08-03T20:47:15.492 DEBUG:tasks.ceph:waiting for RemoteProcess(client=, args=['sudo', 'ceph', '--cluster', 'ceph', 'config', 'set', 'client', 'debug_client', '20'], check_status=True, hostname='smithi032') 2024-08-03T20:47:15.493 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:15.487+0000 7f3f60200640 1 -- 172.21.15.32:0/1886482409 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command([{prefix=config set, name=debug_client}] v 0) -- 0x7f3f2c0050e0 con 0x7f3f400099b0 2024-08-03T20:47:15.493 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:15.465+0000 7f22df200640 1 -- start start 2024-08-03T20:47:15.493 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:15.468+0000 7f22df200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f22c4004800 0x7f22d804a6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:47:15.493 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:15.468+0000 7f22df200640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f22d80d9ff0 0x7f22d80dbd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:47:15.493 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:15.468+0000 7f22df200640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f22d80dc290 0x7f22d8178f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:47:15.493 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:15.468+0000 7f22df200640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f22d8007540 con 0x7f22c4004800 2024-08-03T20:47:15.493 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:15.468+0000 7f22df200640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f22d80073c0 con 0x7f22d80dc290 2024-08-03T20:47:15.494 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:15.468+0000 7f22df200640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7f22d8039f50 con 0x7f22d80d9ff0 2024-08-03T20:47:15.494 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:15.479+0000 7f22e4c00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f22d80dc290 0x7f22d8178f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:47:15.494 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:15.479+0000 7f22e4c00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f22d80dc290 0x7f22d8178f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.32:43792/0 (socket says 172.21.15.32:43792) 2024-08-03T20:47:15.494 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:15.479+0000 7f22e4c00640 1 -- 172.21.15.32:0/2527973483 learned_addr learned my addr 172.21.15.32:0/2527973483 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:47:15.494 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:15.480+0000 7f22e4c00640 1 -- 172.21.15.32:0/2527973483 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f22d80d9ff0 msgr2=0x7f22d80dbd50 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-08-03T20:47:15.494 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:15.480+0000 7f22e4c00640 1 --2- 172.21.15.32:0/2527973483 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f22d80d9ff0 0x7f22d80dbd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:15.494 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:15.480+0000 7f22e4c00640 1 -- 172.21.15.32:0/2527973483 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f22c4004800 msgr2=0x7f22d804a6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:47:15.494 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:15.480+0000 7f22e4c00640 1 --2- 172.21.15.32:0/2527973483 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f22c4004800 0x7f22d804a6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:15.494 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:15.480+0000 7f22e4c00640 1 -- 172.21.15.32:0/2527973483 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f22d80484a0 con 0x7f22d80dc290 2024-08-03T20:47:15.494 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:15.481+0000 7f22e4c00640 1 --2- 172.21.15.32:0/2527973483 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f22d80dc290 0x7f22d8178f50 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f22cc000f60 tx=0x7f22cc00fe20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:47:15.494 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:15.482+0000 7f22d6000640 1 -- 172.21.15.32:0/2527973483 <== mon.1 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22cc005440 con 0x7f22d80dc290 2024-08-03T20:47:15.494 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:15.482+0000 7f22d6000640 1 -- 172.21.15.32:0/2527973483 <== mon.1 v2:172.21.15.118:3300/0 2 ==== config(2 keys) ==== 54+0+0 (secure 0 0 0) 0x7f22cc018070 con 0x7f22d80dc290 2024-08-03T20:47:15.495 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:15.482+0000 7f22d6000640 1 -- 172.21.15.32:0/2527973483 <== mon.1 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22cc00db00 con 0x7f22d80dc290 2024-08-03T20:47:15.495 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:15.487+0000 7f22df200640 1 -- 172.21.15.32:0/2527973483 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f22d804ae70 con 0x7f22d80dc290 2024-08-03T20:47:15.495 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:15.487+0000 7f22df200640 1 -- 172.21.15.32:0/2527973483 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f22d804b430 con 0x7f22d80dc290 2024-08-03T20:47:15.495 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:15.489+0000 7f22d6000640 1 -- 172.21.15.32:0/2527973483 <== mon.1 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f22cc00dca0 con 0x7f22d80dc290 2024-08-03T20:47:15.498 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:15.491+0000 7f22d6000640 1 -- 172.21.15.32:0/2527973483 <== mon.1 v2:172.21.15.118:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f22cc01d700 con 0x7f22d80dc290 2024-08-03T20:47:15.498 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:15.495+0000 7f22df200640 1 -- 172.21.15.32:0/2527973483 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f22d8005de0 con 0x7f22d80dc290 2024-08-03T20:47:15.498 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:15.497+0000 7f22d6000640 1 -- 172.21.15.32:0/2527973483 <== mon.1 v2:172.21.15.118:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+74906 (secure 0 0 0) 0x7f22d8006730 con 0x7f22d80dc290 2024-08-03T20:47:15.502 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:15.502+0000 7f3f55600640 1 -- 172.21.15.32:0/1886482409 <== mon.1 v2:172.21.15.118:3300/0 9 ==== config(3 keys) ==== 79+0+0 (secure 0 0 0) 0x7f3f5001ad10 con 0x7f3f400099b0 2024-08-03T20:47:15.505 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:15.502+0000 7fd40be00640 1 -- 172.21.15.32:0/1538312743 <== mon.1 v2:172.21.15.118:3300/0 9 ==== config(3 keys) ==== 79+0+0 (secure 0 0 0) 0x7fd404046450 con 0x7fd414160350 2024-08-03T20:47:15.508 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:15.507+0000 7fd418a00640 1 -- 172.21.15.32:0/1538312743 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command([{prefix=config set, name=rados_mon_op_timeout}] v 0) -- 0x7fd3e4005100 con 0x7fd414160350 2024-08-03T20:47:15.508 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:15.508+0000 7f22d6000640 1 -- 172.21.15.32:0/2527973483 <== mon.1 v2:172.21.15.118:3300/0 7 ==== config(3 keys) ==== 79+0+0 (secure 0 0 0) 0x7f22cc0035f0 con 0x7f22d80dc290 2024-08-03T20:47:15.508 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:15.507+0000 7f3f55600640 1 -- 172.21.15.32:0/1886482409 <== mon.1 v2:172.21.15.118:3300/0 10 ==== mon_command_ack([{prefix=config set, name=debug_client}]=0 v4)=0 v4) ==== 114+0+0 (secure 0 0 0) 0x7f3f500109e0 con 0x7f3f400099b0 2024-08-03T20:47:15.510 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:15.509+0000 7f3f60200640 1 -- 172.21.15.32:0/1886482409 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f3f400099b0 msgr2=0x7f3f58136270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:47:15.510 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:15.509+0000 7f3f60200640 1 --2- 172.21.15.32:0/1886482409 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f3f400099b0 0x7f3f58136270 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f3f580ae8a0 tx=0x7f3f50006430 comp rx=0 tx=0).stop 2024-08-03T20:47:15.512 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:15.511+0000 7f3f60200640 1 -- 172.21.15.32:0/1886482409 shutdown_connections 2024-08-03T20:47:15.512 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:15.511+0000 7f3f60200640 1 --2- 172.21.15.32:0/1886482409 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f3f58151200 0x7f3f5814f980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:15.512 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:15.511+0000 7f3f60200640 1 --2- 172.21.15.32:0/1886482409 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3f58136c00 0x7f3f58150cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:15.512 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:15.511+0000 7f3f60200640 1 --2- 172.21.15.32:0/1886482409 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f3f400099b0 0x7f3f58136270 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:15.512 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:15.511+0000 7f3f60200640 1 -- 172.21.15.32:0/1886482409 >> 172.21.15.32:0/1886482409 conn(0x7f3f580cff10 msgr2=0x7f3f580cf760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:47:15.513 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:15.511+0000 7f92f8c00640 1 -- 172.21.15.32:0/1965028869 <== mon.1 v2:172.21.15.118:3300/0 8 ==== config(3 keys) ==== 79+0+0 (secure 0 0 0) 0x7f92ec03dd20 con 0x7f92fc0d45d0 2024-08-03T20:47:15.513 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:15.511+0000 7f3f60200640 1 -- 172.21.15.32:0/1886482409 shutdown_connections 2024-08-03T20:47:15.513 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:15.511+0000 7f3f60200640 1 -- 172.21.15.32:0/1886482409 wait complete. 2024-08-03T20:47:15.518 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:15.516+0000 7f2c1fe00640 1 -- 172.21.15.32:0/4259006080 <== mon.0 v2:172.21.15.32:3300/0 8 ==== config(3 keys) ==== 79+0+0 (secure 0 0 0) 0x7f2c200047e0 con 0x7f2c28158f60 2024-08-03T20:47:15.526 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:15.522+0000 7fd40be00640 1 -- 172.21.15.32:0/1538312743 <== mon.1 v2:172.21.15.118:3300/0 10 ==== config(4 keys) ==== 110+0+0 (secure 0 0 0) 0x7fd404055070 con 0x7fd414160350 2024-08-03T20:47:15.526 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:15.523+0000 7f92f8c00640 1 -- 172.21.15.32:0/1965028869 <== mon.1 v2:172.21.15.118:3300/0 9 ==== config(4 keys) ==== 110+0+0 (secure 0 0 0) 0x7f92ec003be0 con 0x7f92fc0d45d0 2024-08-03T20:47:15.526 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:15.523+0000 7f22d6000640 1 -- 172.21.15.32:0/2527973483 <== mon.1 v2:172.21.15.118:3300/0 8 ==== config(4 keys) ==== 110+0+0 (secure 0 0 0) 0x7f22cc0038f0 con 0x7f22d80dc290 2024-08-03T20:47:15.527 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:15.527+0000 7fd40be00640 1 -- 172.21.15.32:0/1538312743 <== mon.1 v2:172.21.15.118:3300/0 11 ==== mon_command_ack([{prefix=config set, name=rados_mon_op_timeout}]=0 v5)=0 v5) ==== 123+0+0 (secure 0 0 0) 0x7fd4141555c0 con 0x7fd414160350 2024-08-03T20:47:15.533 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:15.530+0000 7f2c1fe00640 1 -- 172.21.15.32:0/4259006080 <== mon.0 v2:172.21.15.32:3300/0 9 ==== config(4 keys) ==== 110+0+0 (secure 0 0 0) 0x7f2c20000fb0 con 0x7f2c28158f60 2024-08-03T20:47:15.538 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:15.538+0000 7fd418a00640 1 -- 172.21.15.32:0/1538312743 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fd414160350 msgr2=0x7fd4140b54a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:47:15.539 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:15.538+0000 7fd418a00640 1 --2- 172.21.15.32:0/1538312743 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fd414160350 0x7fd4140b54a0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fd404009230 tx=0x7fd4040046c0 comp rx=0 tx=0).stop 2024-08-03T20:47:15.539 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:15.538+0000 7fd418a00640 1 -- 172.21.15.32:0/1538312743 shutdown_connections 2024-08-03T20:47:15.539 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:15.539+0000 7fd418a00640 1 --2- 172.21.15.32:0/1538312743 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fd414160350 0x7fd4140b54a0 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:15.540 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:15.539+0000 7fd418a00640 1 --2- 172.21.15.32:0/1538312743 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fd4141609d0 0x7fd414156900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:15.540 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:15.539+0000 7fd418a00640 1 --2- 172.21.15.32:0/1538312743 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd414159ff0 0x7fd4141f3360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:15.540 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:15.539+0000 7fd418a00640 1 -- 172.21.15.32:0/1538312743 >> 172.21.15.32:0/1538312743 conn(0x7fd4140cfd80 msgr2=0x7fd41415ec60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:47:15.543 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:15.542+0000 7fd418a00640 1 -- 172.21.15.32:0/1538312743 shutdown_connections 2024-08-03T20:47:15.543 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:15.542+0000 7fd418a00640 1 -- 172.21.15.32:0/1538312743 wait complete. 2024-08-03T20:47:15.592 DEBUG:tasks.ceph:waiting for RemoteProcess(client=, args=['sudo', 'ceph', '--cluster', 'ceph', 'config', 'set', 'client', 'debug_ms', '1'], check_status=True, hostname='smithi032') 2024-08-03T20:47:15.592 DEBUG:tasks.ceph:waiting for RemoteProcess(client=, args=['sudo', 'ceph', '--cluster', 'ceph', 'config', 'set', 'client', 'rados_mon_op_timeout', '900'], check_status=True, hostname='smithi032') 2024-08-03T20:47:15.635 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:15.634+0000 7f2c2f200640 1 -- 172.21.15.32:0/4259006080 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command([{prefix=config set, name=rados_osd_op_timeout}] v 0) -- 0x7f2bfc005310 con 0x7f2c28158f60 2024-08-03T20:47:15.639 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:15.636+0000 7fcfb9600640 1 -- 172.21.15.32:0/3404327945 <== mon.1 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfac007330 con 0x7fcfa0003a00 2024-08-03T20:47:15.639 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:15.636+0000 7fcfc1e00640 1 -- 172.21.15.32:0/3404327945 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fcfa0003a00 msgr2=0x7fcfa0003e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:47:15.639 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:15.636+0000 7fcfc1e00640 1 --2- 172.21.15.32:0/3404327945 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fcfa0003a00 0x7fcfa0003e30 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fcfac001bd0 tx=0x7fcfa0005830 comp rx=0 tx=0).stop 2024-08-03T20:47:15.639 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:15.636+0000 7fcfc1e00640 1 -- 172.21.15.32:0/3404327945 shutdown_connections 2024-08-03T20:47:15.639 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:15.636+0000 7fcfc1e00640 1 --2- 172.21.15.32:0/3404327945 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fcfa0003a00 0x7fcfa0003e30 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:15.639 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:15.636+0000 7fcfc1e00640 1 -- 172.21.15.32:0/3404327945 >> 172.21.15.32:0/3404327945 conn(0x7fcfbc09d620 msgr2=0x7fcfbc0d4320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:47:15.639 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:15.636+0000 7fcfc1e00640 1 -- 172.21.15.32:0/3404327945 shutdown_connections 2024-08-03T20:47:15.639 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:15.637+0000 7fcfc1e00640 1 -- 172.21.15.32:0/3404327945 wait complete. 2024-08-03T20:47:15.640 DEBUG:tasks.ceph:waiting for RemoteProcess(client=, args=['sudo', 'ceph', '--cluster', 'ceph', 'config', 'set', 'client', 'rados_osd_op_timeout', '900'], check_status=True, hostname='smithi032') 2024-08-03T20:47:15.647 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:15.643+0000 7f2c1fe00640 1 -- 172.21.15.32:0/4259006080 <== mon.0 v2:172.21.15.32:3300/0 10 ==== mon_command_ack([{prefix=config set, name=rados_osd_op_timeout}]=0 v6)=0 v6) ==== 123+0+0 (secure 0 0 0) 0x7f2c280b3e60 con 0x7f2c28158f60 2024-08-03T20:47:15.648 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:15.648+0000 7f22d6000640 1 -- 172.21.15.32:0/2527973483 <== mon.1 v2:172.21.15.118:3300/0 9 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f22cc003c20 con 0x7f22d80dc290 2024-08-03T20:47:15.650 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:15.649+0000 7fcfc1e00640 1 Processor -- start 2024-08-03T20:47:15.651 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:15.650+0000 7fcfc1e00640 1 -- start start 2024-08-03T20:47:15.652 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:15.652+0000 7f92f8c00640 1 -- 172.21.15.32:0/1965028869 <== mon.1 v2:172.21.15.118:3300/0 10 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f92ec03e420 con 0x7f92fc0d45d0 2024-08-03T20:47:15.654 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:15.654+0000 7f2c2f200640 1 -- 172.21.15.32:0/4259006080 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2c28158f60 msgr2=0x7f2c280cdc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:47:15.654 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:15.654+0000 7f2c2f200640 1 --2- 172.21.15.32:0/4259006080 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2c28158f60 0x7f2c280cdc80 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f2c28072160 tx=0x7f2c20004a40 comp rx=0 tx=0).stop 2024-08-03T20:47:15.663 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:15.663+0000 7fcfc1e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcfa0003a00 0x7fcfbc15a070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:47:15.664 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:15.663+0000 7fcfc1e00640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fcfbc15a5b0 0x7fcfbc0b1ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:47:15.665 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:15.663+0000 7fcfc1e00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fcfbc14e190 0x7fcfbc0b23e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:47:15.665 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:15.664+0000 7fcfc1e00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fcfbc15d790 con 0x7fcfa0003a00 2024-08-03T20:47:15.665 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:15.665+0000 7f2c2f200640 1 -- 172.21.15.32:0/4259006080 shutdown_connections 2024-08-03T20:47:15.665 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:15.665+0000 7f2c2f200640 1 --2- 172.21.15.32:0/4259006080 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2c28158f60 0x7f2c280cdc80 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:15.665 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:15.665+0000 7f2c2f200640 1 --2- 172.21.15.32:0/4259006080 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f2c280cf360 0x7f2c2815ca30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:15.665 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:15.665+0000 7f2c2f200640 1 --2- 172.21.15.32:0/4259006080 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f2c280b3040 0x7f2c2813c520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:15.665 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:15.665+0000 7f2c2f200640 1 -- 172.21.15.32:0/4259006080 >> 172.21.15.32:0/4259006080 conn(0x7f2c280cff10 msgr2=0x7f2c28073800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:47:15.666 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:15.664+0000 7fcfc1e00640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7fcfbc15d610 con 0x7fcfbc14e190 2024-08-03T20:47:15.666 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:15.664+0000 7fcfc1e00640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7fcfbc0d5250 con 0x7fcfbc15a5b0 2024-08-03T20:47:15.666 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:15.664+0000 7fcfbb400640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fcfbc14e190 0x7fcfbc0b23e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:47:15.666 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:15.664+0000 7fcfbb400640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fcfbc14e190 0x7fcfbc0b23e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.32:43832/0 (socket says 172.21.15.32:43832) 2024-08-03T20:47:15.666 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:15.664+0000 7fcfbb400640 1 -- 172.21.15.32:0/1014609552 learned_addr learned my addr 172.21.15.32:0/1014609552 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:47:15.666 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:15.664+0000 7fcfba000640 1 --2- 172.21.15.32:0/1014609552 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fcfbc15a5b0 0x7fcfbc0b1ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:47:15.667 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:15.665+0000 7f2c2f200640 1 -- 172.21.15.32:0/4259006080 shutdown_connections 2024-08-03T20:47:15.667 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:15.665+0000 7f2c2f200640 1 -- 172.21.15.32:0/4259006080 wait complete. 2024-08-03T20:47:15.668 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:15.665+0000 7fcfbb400640 1 -- 172.21.15.32:0/1014609552 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fcfbc15a5b0 msgr2=0x7fcfbc0b1ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:47:15.668 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:15.665+0000 7fcfbb400640 1 --2- 172.21.15.32:0/1014609552 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fcfbc15a5b0 0x7fcfbc0b1ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:15.668 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:15.665+0000 7fcfbb400640 1 -- 172.21.15.32:0/1014609552 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcfa0003a00 msgr2=0x7fcfbc15a070 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-08-03T20:47:15.668 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:15.666+0000 7fcfbaa00640 1 --2- 172.21.15.32:0/1014609552 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcfa0003a00 0x7fcfbc15a070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:47:15.668 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:15.666+0000 7fcfbb400640 1 --2- 172.21.15.32:0/1014609552 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcfa0003a00 0x7fcfbc15a070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:15.668 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:15.666+0000 7fcfbb400640 1 -- 172.21.15.32:0/1014609552 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcfbc157640 con 0x7fcfbc14e190 2024-08-03T20:47:15.668 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:15.666+0000 7fcfbaa00640 1 --2- 172.21.15.32:0/1014609552 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcfa0003a00 0x7fcfbc15a070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-03T20:47:15.668 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:15.667+0000 7fcfbb400640 1 --2- 172.21.15.32:0/1014609552 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fcfbc14e190 0x7fcfbc0b23e0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fcfbc0d4270 tx=0x7fcfb4006720 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:47:15.669 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:15.667+0000 7fcfb3400640 1 -- 172.21.15.32:0/1014609552 <== mon.1 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfb4001a60 con 0x7fcfbc14e190 2024-08-03T20:47:15.669 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:15.668+0000 7fcfb3400640 1 -- 172.21.15.32:0/1014609552 <== mon.1 v2:172.21.15.118:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fcfb4018070 con 0x7fcfbc14e190 2024-08-03T20:47:15.671 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:15.668+0000 7fcfb3400640 1 -- 172.21.15.32:0/1014609552 <== mon.1 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfb4006850 con 0x7fcfbc14e190 2024-08-03T20:47:15.674 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:15.671+0000 7fcfc1e00640 1 -- 172.21.15.32:0/1014609552 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcfbc157920 con 0x7fcfbc14e190 2024-08-03T20:47:15.674 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:15.671+0000 7fcfc1e00640 1 -- 172.21.15.32:0/1014609552 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcfbc0b16a0 con 0x7fcfbc14e190 2024-08-03T20:47:15.674 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:15.674+0000 7fcfc1e00640 1 -- 172.21.15.32:0/1014609552 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcfbc15e000 con 0x7fcfbc14e190 2024-08-03T20:47:15.677 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:15.674+0000 7fcfb3400640 1 -- 172.21.15.32:0/1014609552 <== mon.1 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fcfb40117c0 con 0x7fcfbc14e190 2024-08-03T20:47:15.677 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:15.674+0000 7fcfb3400640 1 -- 172.21.15.32:0/1014609552 <== mon.1 v2:172.21.15.118:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7fcfb401caa0 con 0x7fcfbc14e190 2024-08-03T20:47:15.679 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:15.679+0000 7fcfb3400640 1 -- 172.21.15.32:0/1014609552 <== mon.1 v2:172.21.15.118:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+74906 (secure 0 0 0) 0x7fcfb401c410 con 0x7fcfbc14e190 2024-08-03T20:47:15.756 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:15.755+0000 7f73bbe00640 1 -- 172.21.15.32:0/3938829127 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f73bc0d2450 msgr2=0x7f73bc0cf6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:47:15.756 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:15.755+0000 7f73bbe00640 1 --2- 172.21.15.32:0/3938829127 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f73bc0d2450 0x7f73bc0cf6e0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f73bc0d2a00 tx=0x7f73b402a7a0 comp rx=0 tx=0).stop 2024-08-03T20:47:15.757 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:15.755+0000 7f73bbe00640 1 -- 172.21.15.32:0/3938829127 shutdown_connections 2024-08-03T20:47:15.757 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:15.755+0000 7f73bbe00640 1 --2- 172.21.15.32:0/3938829127 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f73bc0d2450 0x7f73bc0cf6e0 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:15.757 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:15.755+0000 7f73bbe00640 1 --2- 172.21.15.32:0/3938829127 >> v2:172.21.15.32:3300/0 conn(0x7f73bc0b9300 0x7f73bc0d1f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:15.757 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:15.755+0000 7f73bbe00640 1 -- 172.21.15.32:0/3938829127 >> 172.21.15.32:0/3938829127 conn(0x7f73bc0d4160 msgr2=0x7f73bc0d4570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:47:15.757 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:15.756+0000 7f73bbe00640 1 -- 172.21.15.32:0/3938829127 shutdown_connections 2024-08-03T20:47:15.757 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:15.756+0000 7f73bbe00640 1 -- 172.21.15.32:0/3938829127 wait complete. 2024-08-03T20:47:15.762 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:15.762+0000 7f73bbe00640 1 Processor -- start 2024-08-03T20:47:15.762 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:15.762+0000 7f73bbe00640 1 -- start start 2024-08-03T20:47:15.770 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:15.768+0000 7f73bbe00640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f73bc0b9300 0x7f73bc0b2230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:47:15.770 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:15.769+0000 7f73bbe00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f73bc0d2450 0x7f73bc0c68b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:47:15.770 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:15.769+0000 7f73baa00640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f73bc0b9300 0x7f73bc0b2230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:47:15.771 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:15.769+0000 7f73baa00640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f73bc0b9300 0x7f73bc0b2230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3301/0 says I am v2:172.21.15.32:49450/0 (socket says 172.21.15.32:49450) 2024-08-03T20:47:15.771 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:15.769+0000 7f73baa00640 1 -- 172.21.15.32:0/3244316475 learned_addr learned my addr 172.21.15.32:0/3244316475 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:47:15.771 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:15.769+0000 7f73ba000640 1 --2- 172.21.15.32:0/3244316475 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f73bc0d2450 0x7f73bc0c68b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:47:15.771 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:15.769+0000 7f73bbe00640 1 --2- 172.21.15.32:0/3244316475 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f73bc1531e0 0x7f73bc1535f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:47:15.771 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:15.770+0000 7f73bbe00640 1 -- 172.21.15.32:0/3244316475 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f73bc0cfe30 con 0x7f73bc1531e0 2024-08-03T20:47:15.772 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:15.770+0000 7f73bbe00640 1 -- 172.21.15.32:0/3244316475 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f73bc0d4ed0 con 0x7f73bc0d2450 2024-08-03T20:47:15.772 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:15.770+0000 7f73bb400640 1 --2- 172.21.15.32:0/3244316475 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f73bc1531e0 0x7f73bc1535f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:47:15.772 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:15.771+0000 7f73bbe00640 1 -- 172.21.15.32:0/3244316475 --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7f73bc0cffb0 con 0x7f73bc0b9300 2024-08-03T20:47:15.772 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:15.772+0000 7f73ba000640 1 -- 172.21.15.32:0/3244316475 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f73bc0b9300 msgr2=0x7f73bc0b2230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:47:15.773 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:15.772+0000 7f73ba000640 1 --2- 172.21.15.32:0/3244316475 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f73bc0b9300 0x7f73bc0b2230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:15.773 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:15.772+0000 7f73ba000640 1 -- 172.21.15.32:0/3244316475 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f73bc1531e0 msgr2=0x7f73bc1535f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:47:15.773 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:15.772+0000 7f73ba000640 1 --2- 172.21.15.32:0/3244316475 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f73bc1531e0 0x7f73bc1535f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:15.773 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:15.772+0000 7f73ba000640 1 -- 172.21.15.32:0/3244316475 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f73bc0c6df0 con 0x7f73bc0d2450 2024-08-03T20:47:15.773 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:15.772+0000 7f73bb400640 1 --2- 172.21.15.32:0/3244316475 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f73bc1531e0 0x7f73bc1535f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-03T20:47:15.774 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:15.773+0000 7f73ba000640 1 --2- 172.21.15.32:0/3244316475 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f73bc0d2450 0x7f73bc0c68b0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f73a800d660 tx=0x7f73a800db20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:47:15.774 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:15.774+0000 7f73b2a00640 1 -- 172.21.15.32:0/3244316475 <== mon.1 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73a8017070 con 0x7f73bc0d2450 2024-08-03T20:47:15.775 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:15.774+0000 7f73b2a00640 1 -- 172.21.15.32:0/3244316475 <== mon.1 v2:172.21.15.118:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f73a8004310 con 0x7f73bc0d2450 2024-08-03T20:47:15.775 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:15.774+0000 7f73bbe00640 1 -- 172.21.15.32:0/3244316475 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f73bc0b1190 con 0x7f73bc0d2450 2024-08-03T20:47:15.782 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:15.780+0000 7f73b2a00640 1 -- 172.21.15.32:0/3244316475 <== mon.1 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73a8005260 con 0x7f73bc0d2450 2024-08-03T20:47:15.782 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:15.780+0000 7f73bbe00640 1 -- 172.21.15.32:0/3244316475 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f73bc0b1750 con 0x7f73bc0d2450 2024-08-03T20:47:15.782 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:15.780+0000 7f73b2a00640 1 -- 172.21.15.32:0/3244316475 <== mon.1 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f73a8009820 con 0x7f73bc0d2450 2024-08-03T20:47:15.783 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:15.782+0000 7f73b2a00640 1 -- 172.21.15.32:0/3244316475 <== mon.1 v2:172.21.15.118:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f73a8009d40 con 0x7f73bc0d2450 2024-08-03T20:47:15.783 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:15.782+0000 7f73bbe00640 1 -- 172.21.15.32:0/3244316475 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f73bc0b0f00 con 0x7f73bc0d2450 2024-08-03T20:47:15.785 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:15.785+0000 7f73b2a00640 1 -- 172.21.15.32:0/3244316475 <== mon.1 v2:172.21.15.118:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+74906 (secure 0 0 0) 0x7f73a8020020 con 0x7f73bc0d2450 2024-08-03T20:47:15.802 DEBUG:tasks.ceph:waiting for RemoteProcess(client=, args=['sudo', 'ceph', '--cluster', 'ceph', 'config', 'set', 'mds', 'debug_mds', '20'], check_status=True, hostname='smithi032') 2024-08-03T20:47:15.853 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:15.852+0000 7f22df200640 1 -- 172.21.15.32:0/2527973483 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command([{prefix=config set, name=debug_mds_balancer}] v 0) -- 0x7f22b40052f0 con 0x7f22d80dc290 2024-08-03T20:47:15.868 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:15.867+0000 7f4e42e00640 1 -- 172.21.15.32:0/3668243099 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f4e3c152350 msgr2=0x7f4e3c0cf450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:47:15.869 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:15.869+0000 7f9302600640 1 -- 172.21.15.32:0/1965028869 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command([{prefix=config set, name=debug_mds}] v 0) -- 0x7f92d00052f0 con 0x7f92fc0d45d0 2024-08-03T20:47:15.871 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:15.871+0000 7f4e42e00640 1 --2- 172.21.15.32:0/3668243099 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f4e3c152350 0x7f4e3c0cf450 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f4e34005fd0 tx=0x7f4e3402aa50 comp rx=0 tx=0).stop 2024-08-03T20:47:15.871 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:15.871+0000 7f4e42e00640 1 -- 172.21.15.32:0/3668243099 shutdown_connections 2024-08-03T20:47:15.872 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:15.871+0000 7f4e42e00640 1 --2- 172.21.15.32:0/3668243099 >> v2:172.21.15.32:3300/0 conn(0x7f4e3c073150 0x7f4e3c073560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:15.872 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:15.871+0000 7f4e42e00640 1 --2- 172.21.15.32:0/3668243099 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f4e3c152350 0x7f4e3c0cf450 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:15.872 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:15.871+0000 7f4e42e00640 1 -- 172.21.15.32:0/3668243099 >> 172.21.15.32:0/3668243099 conn(0x7f4e3c0b1a70 msgr2=0x7f4e3c0b1e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:47:15.872 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:15.869+0000 7f22d6000640 1 -- 172.21.15.32:0/2527973483 <== mon.1 v2:172.21.15.118:3300/0 10 ==== mon_command_ack([{prefix=config set, name=debug_mds_balancer}]=0 v7)=0 v7) ==== 117+0+0 (secure 0 0 0) 0x7f22d8006730 con 0x7f22d80dc290 2024-08-03T20:47:15.872 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:15.872+0000 7f4e42e00640 1 -- 172.21.15.32:0/3668243099 shutdown_connections 2024-08-03T20:47:15.873 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:15.872+0000 7f9c9a600640 1 -- 172.21.15.32:0/3411091056 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f9c940d26e0 msgr2=0x7f9c940d4090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:47:15.873 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:15.872+0000 7f9c9a600640 1 --2- 172.21.15.32:0/3411091056 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f9c940d26e0 0x7f9c940d4090 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f9c940d1b90 tx=0x7f9c8802a580 comp rx=0 tx=0).stop 2024-08-03T20:47:15.873 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:15.872+0000 7f9c9a600640 1 -- 172.21.15.32:0/3411091056 shutdown_connections 2024-08-03T20:47:15.873 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:15.872+0000 7f9c9a600640 1 --2- 172.21.15.32:0/3411091056 >> v2:172.21.15.32:3300/0 conn(0x7f9c940cfb40 0x7f9c94162820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:15.873 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:15.872+0000 7f9c9a600640 1 --2- 172.21.15.32:0/3411091056 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f9c940d26e0 0x7f9c940d4090 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:15.874 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:15.872+0000 7f9c9a600640 1 -- 172.21.15.32:0/3411091056 >> 172.21.15.32:0/3411091056 conn(0x7f9c9408d7e0 msgr2=0x7f9c9409d410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:47:15.874 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:15.872+0000 7f4e42e00640 1 -- 172.21.15.32:0/3668243099 wait complete. 2024-08-03T20:47:15.874 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:15.872+0000 7f9c9a600640 1 -- 172.21.15.32:0/3411091056 shutdown_connections 2024-08-03T20:47:15.874 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:15.872+0000 7f9c9a600640 1 -- 172.21.15.32:0/3411091056 wait complete. 2024-08-03T20:47:15.875 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:15.873+0000 7f22df200640 1 -- 172.21.15.32:0/2527973483 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f22d80dc290 msgr2=0x7f22d8178f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:47:15.875 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:15.873+0000 7f22df200640 1 --2- 172.21.15.32:0/2527973483 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f22d80dc290 0x7f22d8178f50 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f22cc000f60 tx=0x7f22cc00fe20 comp rx=0 tx=0).stop 2024-08-03T20:47:15.875 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:15.874+0000 7f22df200640 1 -- 172.21.15.32:0/2527973483 shutdown_connections 2024-08-03T20:47:15.875 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:15.874+0000 7f22df200640 1 --2- 172.21.15.32:0/2527973483 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f22d80dc290 0x7f22d8178f50 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:15.875 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:15.874+0000 7f22df200640 1 --2- 172.21.15.32:0/2527973483 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f22d80d9ff0 0x7f22d80dbd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:15.875 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:15.874+0000 7f22df200640 1 --2- 172.21.15.32:0/2527973483 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f22c4004800 0x7f22d804a6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:15.876 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:15.874+0000 7f22df200640 1 -- 172.21.15.32:0/2527973483 >> 172.21.15.32:0/2527973483 conn(0x7f22d801db80 msgr2=0x7f22d800a620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:47:15.876 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:15.875+0000 7f22df200640 1 -- 172.21.15.32:0/2527973483 shutdown_connections 2024-08-03T20:47:15.877 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:15.877+0000 7f22df200640 1 -- 172.21.15.32:0/2527973483 wait complete. 2024-08-03T20:47:15.883 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:15.882+0000 7f4e42e00640 1 Processor -- start 2024-08-03T20:47:15.883 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:15.882+0000 7f4e42e00640 1 -- start start 2024-08-03T20:47:15.892 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:15.889+0000 7f92f8c00640 1 -- 172.21.15.32:0/1965028869 <== mon.1 v2:172.21.15.118:3300/0 11 ==== mon_command_ack([{prefix=config set, name=debug_mds}]=0 v8)=0 v8) ==== 108+0+0 (secure 0 0 0) 0x7f92ec04c030 con 0x7f92fc0d45d0 2024-08-03T20:47:15.892 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:15.890+0000 7f4e42e00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f4e3c06f740 0x7f4e3c0b3c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:47:15.894 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:15.894+0000 7f8090e00640 1 -- 172.21.15.32:0/327661306 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f808c162150 msgr2=0x7f808c162560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:47:15.894 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:15.894+0000 7f8090e00640 1 --2- 172.21.15.32:0/327661306 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f808c162150 0x7f808c162560 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f8084009f60 tx=0x7f808402a7e0 comp rx=0 tx=0).stop 2024-08-03T20:47:15.895 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:15.894+0000 7f92f2000640 1 -- 172.21.15.32:0/1965028869 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f92fc0d45d0 msgr2=0x7f92fc160050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:47:15.895 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:15.894+0000 7f92f2000640 1 --2- 172.21.15.32:0/1965028869 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f92fc0d45d0 0x7f92fc160050 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f92ec031000 tx=0x7f92ec0046c0 comp rx=0 tx=0).stop 2024-08-03T20:47:15.896 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:15.894+0000 7f92f2000640 1 -- 172.21.15.32:0/1965028869 shutdown_connections 2024-08-03T20:47:15.896 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:15.894+0000 7f92f2000640 1 --2- 172.21.15.32:0/1965028869 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f92fc0d45d0 0x7f92fc160050 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:15.896 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:15.894+0000 7f92f2000640 1 --2- 172.21.15.32:0/1965028869 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f92fc162650 0x7f92fc160590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:15.896 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:15.894+0000 7f92f2000640 1 --2- 172.21.15.32:0/1965028869 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f92fc0d26e0 0x7f92fc1532c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:15.896 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:15.894+0000 7f92f2000640 1 -- 172.21.15.32:0/1965028869 >> 172.21.15.32:0/1965028869 conn(0x7f92fc08d7e0 msgr2=0x7f92fc06e4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:47:15.897 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:15.891+0000 7f4e42e00640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f4e3c073150 0x7f4e3c1f5a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:47:15.897 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:15.891+0000 7f4e42e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4e3c152350 0x7f4e3c1f5fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:47:15.897 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:15.891+0000 7f4e41000640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f4e3c073150 0x7f4e3c1f5a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:47:15.897 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:15.891+0000 7f4e41000640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f4e3c073150 0x7f4e3c1f5a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3301/0 says I am v2:172.21.15.32:49486/0 (socket says 172.21.15.32:49486) 2024-08-03T20:47:15.897 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:15.892+0000 7f4e41000640 1 -- 172.21.15.32:0/854577433 learned_addr learned my addr 172.21.15.32:0/854577433 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:47:15.897 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:15.892+0000 7f4e42e00640 1 -- 172.21.15.32:0/854577433 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f4e3c073e30 con 0x7f4e3c152350 2024-08-03T20:47:15.898 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:15.892+0000 7f4e42e00640 1 -- 172.21.15.32:0/854577433 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f4e3c073cb0 con 0x7f4e3c06f740 2024-08-03T20:47:15.898 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:15.892+0000 7f4e42e00640 1 -- 172.21.15.32:0/854577433 --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7f4e3c073fb0 con 0x7f4e3c073150 2024-08-03T20:47:15.899 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:15.895+0000 7f92f2000640 1 -- 172.21.15.32:0/1965028869 shutdown_connections 2024-08-03T20:47:15.899 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:15.895+0000 7f92f2000640 1 -- 172.21.15.32:0/1965028869 wait complete. 2024-08-03T20:47:15.899 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:15.895+0000 7f4e41a00640 1 --2- 172.21.15.32:0/854577433 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f4e3c06f740 0x7f4e3c0b3c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:47:15.899 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:15.896+0000 7f4e42400640 1 --2- 172.21.15.32:0/854577433 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4e3c152350 0x7f4e3c1f5fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:47:15.899 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:15.896+0000 7f9c9a600640 1 Processor -- start 2024-08-03T20:47:15.900 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:15.896+0000 7f4e41000640 1 -- 172.21.15.32:0/854577433 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f4e3c06f740 msgr2=0x7f4e3c0b3c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:47:15.900 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:15.896+0000 7f4e41000640 1 --2- 172.21.15.32:0/854577433 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f4e3c06f740 0x7f4e3c0b3c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:15.900 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:15.896+0000 7f4e41000640 1 -- 172.21.15.32:0/854577433 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4e3c152350 msgr2=0x7f4e3c1f5fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:47:15.900 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:15.896+0000 7f4e41000640 1 --2- 172.21.15.32:0/854577433 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4e3c152350 0x7f4e3c1f5fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:15.900 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:15.896+0000 7f4e41000640 1 -- 172.21.15.32:0/854577433 --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4e3c0b4850 con 0x7f4e3c073150 2024-08-03T20:47:15.900 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:15.898+0000 7f4e41000640 1 --2- 172.21.15.32:0/854577433 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f4e3c073150 0x7f4e3c1f5a60 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f4e2800c3a0 tx=0x7f4e2800c860 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:47:15.901 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:15.896+0000 7f8090e00640 1 -- 172.21.15.32:0/327661306 shutdown_connections 2024-08-03T20:47:15.901 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:15.896+0000 7f8090e00640 1 --2- 172.21.15.32:0/327661306 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f808c162150 0x7f808c162560 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:15.901 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:15.896+0000 7f8090e00640 1 --2- 172.21.15.32:0/327661306 >> v2:172.21.15.32:3300/0 conn(0x7f808c06d430 0x7f808c06d840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:15.901 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:15.896+0000 7f8090e00640 1 -- 172.21.15.32:0/327661306 >> 172.21.15.32:0/327661306 conn(0x7f808c0d4280 msgr2=0x7f808c0d4690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:47:15.901 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:15.899+0000 7f8090e00640 1 -- 172.21.15.32:0/327661306 shutdown_connections 2024-08-03T20:47:15.901 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:15.899+0000 7f8090e00640 1 -- 172.21.15.32:0/327661306 wait complete. 2024-08-03T20:47:15.902 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:15.898+0000 7f4e39600640 1 -- 172.21.15.32:0/854577433 <== mon.2 v2:172.21.15.118:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e28017070 con 0x7f4e3c073150 2024-08-03T20:47:15.902 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:15.902+0000 7f4e39600640 1 -- 172.21.15.32:0/854577433 <== mon.2 v2:172.21.15.118:3301/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f4e280053c0 con 0x7f4e3c073150 2024-08-03T20:47:15.902 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:15.902+0000 7f4e39600640 1 -- 172.21.15.32:0/854577433 <== mon.2 v2:172.21.15.118:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e28003850 con 0x7f4e3c073150 2024-08-03T20:47:15.904 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:15.900+0000 7f9c9a600640 1 -- start start 2024-08-03T20:47:15.911 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:15.911+0000 7f4e42e00640 1 -- 172.21.15.32:0/854577433 --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4e3c0b4b30 con 0x7f4e3c073150 2024-08-03T20:47:15.911 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:15.911+0000 7f4e42e00640 1 -- 172.21.15.32:0/854577433 --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f4e3c0b50f0 con 0x7f4e3c073150 2024-08-03T20:47:15.911 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:15.911+0000 7f9c9a600640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f9c940cfb40 0x7f9c94202170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:47:15.912 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:15.912+0000 7f9c9a600640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f9c940d26e0 0x7f9c940c9500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:47:15.912 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:15.912+0000 7f9c99200640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f9c940cfb40 0x7f9c94202170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:47:15.912 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:15.912+0000 7f9c99200640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f9c940cfb40 0x7f9c94202170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.32:43888/0 (socket says 172.21.15.32:43888) 2024-08-03T20:47:15.913 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:15.913+0000 7f9c93e00640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f9c940d26e0 0x7f9c940c9500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:47:15.913 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:15.913+0000 7f9c93e00640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f9c940d26e0 0x7f9c940c9500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3301/0 says I am v2:172.21.15.32:49500/0 (socket says 172.21.15.32:49500) 2024-08-03T20:47:15.913 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:15.913+0000 7f9c99200640 1 -- 172.21.15.32:0/2445254085 learned_addr learned my addr 172.21.15.32:0/2445254085 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:47:15.914 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:15.913+0000 7f9c9a600640 1 --2- 172.21.15.32:0/2445254085 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9c940d45d0 0x7f9c940c9a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:47:15.914 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:15.913+0000 7f9c9a600640 1 -- 172.21.15.32:0/2445254085 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f9c940b53a0 con 0x7f9c940d45d0 2024-08-03T20:47:15.914 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:15.913+0000 7f9c9a600640 1 -- 172.21.15.32:0/2445254085 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f9c940b5220 con 0x7f9c940cfb40 2024-08-03T20:47:15.915 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:15.915+0000 7f8090e00640 1 Processor -- start 2024-08-03T20:47:15.917 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:15.915+0000 7f9c9a600640 1 -- 172.21.15.32:0/2445254085 --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7f9c94162d60 con 0x7f9c940d26e0 2024-08-03T20:47:15.917 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:15.916+0000 7f9c99c00640 1 --2- 172.21.15.32:0/2445254085 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9c940d45d0 0x7f9c940c9a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:47:15.917 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:15.916+0000 7f8090e00640 1 -- start start 2024-08-03T20:47:15.920 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:15.919+0000 7f4e39600640 1 -- 172.21.15.32:0/854577433 <== mon.2 v2:172.21.15.118:3301/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f4e2801a070 con 0x7f4e3c073150 2024-08-03T20:47:15.920 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:15.919+0000 7f4e39600640 1 -- 172.21.15.32:0/854577433 <== mon.2 v2:172.21.15.118:3301/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f4e2801c440 con 0x7f4e3c073150 2024-08-03T20:47:15.920 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:15.919+0000 7f9c99200640 1 -- 172.21.15.32:0/2445254085 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f9c940d26e0 msgr2=0x7f9c940c9500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:47:15.920 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:15.919+0000 7f9c99200640 1 --2- 172.21.15.32:0/2445254085 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f9c940d26e0 0x7f9c940c9500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:15.920 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:15.919+0000 7f9c99200640 1 -- 172.21.15.32:0/2445254085 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9c940d45d0 msgr2=0x7f9c940c9a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:47:15.922 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:15.920+0000 7f9c99200640 1 --2- 172.21.15.32:0/2445254085 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9c940d45d0 0x7f9c940c9a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:15.922 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:15.920+0000 7f9c99200640 1 -- 172.21.15.32:0/2445254085 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9c942069a0 con 0x7f9c940cfb40 2024-08-03T20:47:15.922 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:15.921+0000 7f9c99c00640 1 --2- 172.21.15.32:0/2445254085 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9c940d45d0 0x7f9c940c9a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-03T20:47:15.922 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:15.922+0000 7f4e42e00640 1 -- 172.21.15.32:0/854577433 --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4e3c0d0ed0 con 0x7f4e3c073150 2024-08-03T20:47:15.923 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:15.922+0000 7f9c99200640 1 --2- 172.21.15.32:0/2445254085 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f9c940cfb40 0x7f9c94202170 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f9c88033040 tx=0x7f9c88004340 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:47:15.925 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:15.923+0000 7f9c90c00640 1 -- 172.21.15.32:0/2445254085 <== mon.1 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c88041070 con 0x7f9c940cfb40 2024-08-03T20:47:15.925 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:15.925+0000 7f9c90c00640 1 -- 172.21.15.32:0/2445254085 <== mon.1 v2:172.21.15.118:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f9c88031b00 con 0x7f9c940cfb40 2024-08-03T20:47:15.925 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:15.925+0000 7f9c9a600640 1 -- 172.21.15.32:0/2445254085 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9c94206c80 con 0x7f9c940cfb40 2024-08-03T20:47:15.926 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:15.926+0000 7f8090e00640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f808c06d430 0x7f808c153240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:47:15.927 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:15.926+0000 7f8090e00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f808c0cf990 0x7f808c0c9cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:47:15.927 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:15.926+0000 7f8090e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f808c162150 0x7f808c157640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:47:15.927 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:15.926+0000 7f8090e00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f808c0d29f0 con 0x7f808c162150 2024-08-03T20:47:15.927 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:15.926+0000 7f8090e00640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f808c0d2870 con 0x7f808c0cf990 2024-08-03T20:47:15.927 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:15.926+0000 7f8090e00640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7f808c0d2b70 con 0x7f808c06d430 2024-08-03T20:47:15.928 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:15.926+0000 7f8089e00640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f808c06d430 0x7f808c153240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:47:15.928 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:15.927+0000 7f8089e00640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f808c06d430 0x7f808c153240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3301/0 says I am v2:172.21.15.32:49512/0 (socket says 172.21.15.32:49512) 2024-08-03T20:47:15.928 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:15.927+0000 7f8089e00640 1 -- 172.21.15.32:0/2437226764 learned_addr learned my addr 172.21.15.32:0/2437226764 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:47:15.930 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:15.927+0000 7f4e39600640 1 -- 172.21.15.32:0/854577433 <== mon.2 v2:172.21.15.118:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+74906 (secure 0 0 0) 0x7f4e3c0d0ed0 con 0x7f4e3c073150 2024-08-03T20:47:15.930 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:15.927+0000 7f8089400640 1 --2- 172.21.15.32:0/2437226764 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f808c0cf990 0x7f808c0c9cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:47:15.931 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:15.928+0000 7f8089e00640 1 -- 172.21.15.32:0/2437226764 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f808c0cf990 msgr2=0x7f808c0c9cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:47:15.931 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:15.928+0000 7f8089e00640 1 --2- 172.21.15.32:0/2437226764 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f808c0cf990 0x7f808c0c9cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:15.931 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:15.928+0000 7f8089e00640 1 -- 172.21.15.32:0/2437226764 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f808c162150 msgr2=0x7f808c157640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:47:15.931 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:15.928+0000 7f8089e00640 1 --2- 172.21.15.32:0/2437226764 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f808c162150 0x7f808c157640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:15.931 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:15.928+0000 7f8089e00640 1 -- 172.21.15.32:0/2437226764 --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f808c155920 con 0x7f808c06d430 2024-08-03T20:47:15.932 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:15.925+0000 7f9c9a600640 1 -- 172.21.15.32:0/2445254085 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9c940b8950 con 0x7f9c940cfb40 2024-08-03T20:47:15.932 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:15.925+0000 7f9c90c00640 1 -- 172.21.15.32:0/2445254085 <== mon.1 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c8803b430 con 0x7f9c940cfb40 2024-08-03T20:47:15.936 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:15.932+0000 7f8089e00640 1 --2- 172.21.15.32:0/2437226764 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f808c06d430 0x7f808c153240 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f807c00c430 tx=0x7f807c00c8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:47:15.936 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:15.932+0000 7f8082000640 1 -- 172.21.15.32:0/2437226764 <== mon.2 v2:172.21.15.118:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f807c011790 con 0x7f808c06d430 2024-08-03T20:47:15.936 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:15.933+0000 7f8082000640 1 -- 172.21.15.32:0/2437226764 <== mon.2 v2:172.21.15.118:3301/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f807c017070 con 0x7f808c06d430 2024-08-03T20:47:15.937 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:15.934+0000 7f8082000640 1 -- 172.21.15.32:0/2437226764 <== mon.2 v2:172.21.15.118:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f807c011790 con 0x7f808c06d430 2024-08-03T20:47:15.937 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:15.934+0000 7f8090e00640 1 -- 172.21.15.32:0/2437226764 --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f808c155bc0 con 0x7f808c06d430 2024-08-03T20:47:15.937 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:15.934+0000 7f8090e00640 1 -- 172.21.15.32:0/2437226764 --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f808c15a940 con 0x7f808c06d430 2024-08-03T20:47:15.937 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:15.935+0000 7f8082000640 1 -- 172.21.15.32:0/2437226764 <== mon.2 v2:172.21.15.118:3301/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f807c011a40 con 0x7f808c06d430 2024-08-03T20:47:15.937 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:15.935+0000 7f8082000640 1 -- 172.21.15.32:0/2437226764 <== mon.2 v2:172.21.15.118:3301/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f807c012590 con 0x7f808c06d430 2024-08-03T20:47:15.937 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:15.934+0000 7f9c90c00640 1 -- 172.21.15.32:0/2445254085 <== mon.1 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f9c88044070 con 0x7f9c940cfb40 2024-08-03T20:47:15.938 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:15.934+0000 7f9c90c00640 1 -- 172.21.15.32:0/2445254085 <== mon.1 v2:172.21.15.118:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f9c88046790 con 0x7f9c940cfb40 2024-08-03T20:47:15.941 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:15.939+0000 7f9c9a600640 1 -- 172.21.15.32:0/2445254085 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9c70005290 con 0x7f9c940cfb40 2024-08-03T20:47:15.945 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:15.945+0000 7f9c90c00640 1 -- 172.21.15.32:0/2445254085 <== mon.1 v2:172.21.15.118:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+74906 (secure 0 0 0) 0x7f9c940d49e0 con 0x7f9c940cfb40 2024-08-03T20:47:15.953 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:15.953+0000 7fcfc1e00640 1 -- 172.21.15.32:0/1014609552 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command([{prefix=config set, name=debug_ms}] v 0) -- 0x7fcfbc158830 con 0x7fcfbc14e190 2024-08-03T20:47:15.958 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:15.957+0000 7f8090e00640 1 -- 172.21.15.32:0/2437226764 --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f808c0d3ff0 con 0x7f808c06d430 2024-08-03T20:47:15.966 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:15.964+0000 7fcfb3400640 1 -- 172.21.15.32:0/1014609552 <== mon.1 v2:172.21.15.118:3300/0 7 ==== mon_command_ack([{prefix=config set, name=debug_ms}]=0 v9)=0 v9) ==== 106+0+0 (secure 0 0 0) 0x7fcfb401c5f0 con 0x7fcfbc14e190 2024-08-03T20:47:15.973 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:15.973+0000 7fcfc1e00640 1 -- 172.21.15.32:0/1014609552 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fcfbc14e190 msgr2=0x7fcfbc0b23e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:47:15.973 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:15.973+0000 7fcfc1e00640 1 --2- 172.21.15.32:0/1014609552 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fcfbc14e190 0x7fcfbc0b23e0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fcfbc0d4270 tx=0x7fcfb4006720 comp rx=0 tx=0).stop 2024-08-03T20:47:15.976 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:15.975+0000 7fcfc1e00640 1 -- 172.21.15.32:0/1014609552 shutdown_connections 2024-08-03T20:47:15.976 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:15.975+0000 7fcfc1e00640 1 --2- 172.21.15.32:0/1014609552 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fcfbc14e190 0x7fcfbc0b23e0 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:15.976 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:15.975+0000 7fcfc1e00640 1 --2- 172.21.15.32:0/1014609552 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fcfbc15a5b0 0x7fcfbc0b1ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:15.976 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:15.975+0000 7fcfc1e00640 1 --2- 172.21.15.32:0/1014609552 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcfa0003a00 0x7fcfbc15a070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:15.976 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:15.975+0000 7fcfc1e00640 1 -- 172.21.15.32:0/1014609552 >> 172.21.15.32:0/1014609552 conn(0x7fcfbc09d620 msgr2=0x7fcfbc0b4ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:47:15.976 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:15.975+0000 7fcfc1e00640 1 -- 172.21.15.32:0/1014609552 shutdown_connections 2024-08-03T20:47:15.976 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:15.975+0000 7fcfc1e00640 1 -- 172.21.15.32:0/1014609552 wait complete. 2024-08-03T20:47:15.978 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:15.978+0000 7f8082000640 1 -- 172.21.15.32:0/2437226764 <== mon.2 v2:172.21.15.118:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+74906 (secure 0 0 0) 0x7f808c06dd80 con 0x7f808c06d430 2024-08-03T20:47:15.981 DEBUG:tasks.ceph:waiting for RemoteProcess(client=, args=['sudo', 'ceph', '--cluster', 'ceph', 'config', 'set', 'mds', 'debug_mds_balancer', '20'], check_status=True, hostname='smithi032') 2024-08-03T20:47:15.982 DEBUG:tasks.ceph:waiting for RemoteProcess(client=, args=['sudo', 'ceph', '--cluster', 'ceph', 'config', 'set', 'mds', 'debug_ms', '2'], check_status=True, hostname='smithi032') 2024-08-03T20:47:15.995 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:15.995+0000 7f7c47e00640 1 -- 172.21.15.32:0/5278860 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f7c480d47a0 msgr2=0x7f7c480ceb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:47:15.995 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:15.995+0000 7f7c47e00640 1 --2- 172.21.15.32:0/5278860 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f7c480d47a0 0x7f7c480ceb30 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f7c4808e670 tx=0x7f7c3402a4f0 comp rx=0 tx=0).stop 2024-08-03T20:47:15.997 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:15.997+0000 7f7c47e00640 1 -- 172.21.15.32:0/5278860 shutdown_connections 2024-08-03T20:47:15.997 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:15.997+0000 7f7c47e00640 1 --2- 172.21.15.32:0/5278860 >> v2:172.21.15.32:3300/0 conn(0x7f7c480cf070 0x7f7c480cf480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:15.997 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:15.997+0000 7f7c47e00640 1 --2- 172.21.15.32:0/5278860 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f7c480d47a0 0x7f7c480ceb30 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:15.998 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:15.998+0000 7f7c47e00640 1 -- 172.21.15.32:0/5278860 >> 172.21.15.32:0/5278860 conn(0x7f7c4808d7e0 msgr2=0x7f7c4809d410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:47:16.003 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:16.003+0000 7f7c47e00640 1 -- 172.21.15.32:0/5278860 shutdown_connections 2024-08-03T20:47:16.003 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:16.003+0000 7f7c47e00640 1 -- 172.21.15.32:0/5278860 wait complete. 2024-08-03T20:47:16.010 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:16.010+0000 7f1a62200640 1 -- 172.21.15.32:0/3500766741 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f1a54054190 msgr2=0x7f1a54007550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:47:16.010 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:16.010+0000 7f1a62200640 1 --2- 172.21.15.32:0/3500766741 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f1a54054190 0x7f1a54007550 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f1a48009830 tx=0x7f1a4802a4d0 comp rx=0 tx=0).stop 2024-08-03T20:47:16.014 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:16.014+0000 7f1a62200640 1 -- 172.21.15.32:0/3500766741 shutdown_connections 2024-08-03T20:47:16.014 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:16.014+0000 7f1a62200640 1 --2- 172.21.15.32:0/3500766741 >> v2:172.21.15.118:3300/0 conn(0x7f1a540e3200 0x7f1a54053150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:16.015 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:16.014+0000 7f1a62200640 1 --2- 172.21.15.32:0/3500766741 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f1a54054190 0x7f1a54007550 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:16.015 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:16.014+0000 7f1a62200640 1 -- 172.21.15.32:0/3500766741 >> 172.21.15.32:0/3500766741 conn(0x7f1a5403a680 msgr2=0x7f1a5403aa90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:47:16.017 DEBUG:tasks.ceph:waiting for RemoteProcess(client=, args=['sudo', 'ceph', '--cluster', 'ceph', 'config', 'set', 'mds', 'mds_debug_frag', 'True'], check_status=True, hostname='smithi032') 2024-08-03T20:47:16.018 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:16.017+0000 7f1a62200640 1 -- 172.21.15.32:0/3500766741 shutdown_connections 2024-08-03T20:47:16.018 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:16.018+0000 7f1a62200640 1 -- 172.21.15.32:0/3500766741 wait complete. 2024-08-03T20:47:16.021 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:16.021+0000 7f73bbe00640 1 -- 172.21.15.32:0/3244316475 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command([{prefix=config set, name=mds_debug_frag}] v 0) -- 0x7f73bc0d18f0 con 0x7f73bc0d2450 2024-08-03T20:47:16.021 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:16.021+0000 7f7c47e00640 1 Processor -- start 2024-08-03T20:47:16.023 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:16.022+0000 7f7c47e00640 1 -- start start 2024-08-03T20:47:16.025 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:16.024+0000 7f1a62200640 1 Processor -- start 2024-08-03T20:47:16.025 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:16.024+0000 7f1a62200640 1 -- start start 2024-08-03T20:47:16.026 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:16.025+0000 7f7c47e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7c480cf070 0x7f7c481f93d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:47:16.026 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:16.025+0000 7f7c47e00640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f7c480d26e0 0x7f7c480cdc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:47:16.026 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:16.025+0000 7f7c47e00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f7c481556c0 0x7f7c4806d4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:47:16.026 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:16.026+0000 7f7c47e00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f7c480cfb40 con 0x7f7c480cf070 2024-08-03T20:47:16.027 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:16.026+0000 7f7c47e00640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f7c480cf9c0 con 0x7f7c481556c0 2024-08-03T20:47:16.027 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:16.027+0000 7f7c47400640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f7c481556c0 0x7f7c4806d4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:47:16.027 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:16.027+0000 7f7c47e00640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7f7c480cfcc0 con 0x7f7c480d26e0 2024-08-03T20:47:16.027 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:16.027+0000 7f7c46a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7c480cf070 0x7f7c481f93d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:47:16.028 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:16.028+0000 7f7c47400640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f7c481556c0 0x7f7c4806d4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.32:43936/0 (socket says 172.21.15.32:43936) 2024-08-03T20:47:16.028 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:16.028+0000 7f7c46a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7c480cf070 0x7f7c481f93d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:43226/0 (socket says 172.21.15.32:43226) 2024-08-03T20:47:16.028 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:16.028+0000 7f7c46a00640 1 -- 172.21.15.32:0/284859340 learned_addr learned my addr 172.21.15.32:0/284859340 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:47:16.029 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:16.028+0000 7f7c46000640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f7c480d26e0 0x7f7c480cdc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:47:16.036 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:16.030+0000 7f7c46a00640 1 -- 172.21.15.32:0/284859340 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f7c480d26e0 msgr2=0x7f7c480cdc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:47:16.036 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:16.030+0000 7f7c46a00640 1 --2- 172.21.15.32:0/284859340 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f7c480d26e0 0x7f7c480cdc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:16.036 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:16.030+0000 7f7c46a00640 1 -- 172.21.15.32:0/284859340 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f7c481556c0 msgr2=0x7f7c4806d4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:47:16.036 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:16.030+0000 7f7c46a00640 1 --2- 172.21.15.32:0/284859340 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f7c481556c0 0x7f7c4806d4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:16.037 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:16.030+0000 7f7c46a00640 1 -- 172.21.15.32:0/284859340 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7c480cb0e0 con 0x7f7c480cf070 2024-08-03T20:47:16.037 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:16.032+0000 7f7c46a00640 1 --2- 172.21.15.32:0/284859340 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7c480cf070 0x7f7c481f93d0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f7c3800c7f0 tx=0x7f7c3800ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:47:16.037 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:16.032+0000 7f7c3ea00640 1 -- 172.21.15.32:0/284859340 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c38017070 con 0x7f7c480cf070 2024-08-03T20:47:16.037 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:16.033+0000 7f7c47e00640 1 -- 172.21.15.32:0/284859340 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7c480cb3c0 con 0x7f7c480cf070 2024-08-03T20:47:16.037 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:16.033+0000 7f7c47e00640 1 -- 172.21.15.32:0/284859340 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7c48155ad0 con 0x7f7c480cf070 2024-08-03T20:47:16.037 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:16.035+0000 7f73b2a00640 1 -- 172.21.15.32:0/3244316475 <== mon.1 v2:172.21.15.118:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mds_debug_frag}]=0 v10)=0 v10) ==== 115+0+0 (secure 0 0 0) 0x7f73bc0c6df0 con 0x7f73bc0d2450 2024-08-03T20:47:16.038 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:16.033+0000 7f7c3ea00640 1 -- 172.21.15.32:0/284859340 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f7c38005360 con 0x7f7c480cf070 2024-08-03T20:47:16.038 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:16.033+0000 7f7c3ea00640 1 -- 172.21.15.32:0/284859340 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c38017070 con 0x7f7c480cf070 2024-08-03T20:47:16.038 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:16.037+0000 7f7c3ea00640 1 -- 172.21.15.32:0/284859340 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f7c3801a070 con 0x7f7c480cf070 2024-08-03T20:47:16.038 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:16.037+0000 7f7c3ea00640 1 -- 172.21.15.32:0/284859340 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f7c38003ca0 con 0x7f7c480cf070 2024-08-03T20:47:16.038 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:16.038+0000 7f7c47e00640 1 -- 172.21.15.32:0/284859340 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7c20005290 con 0x7f7c480cf070 2024-08-03T20:47:16.039 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:16.039+0000 7f1a62200640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f1a54053690 0x7f1a54187200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:47:16.039 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:16.039+0000 7f1a62200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1a54054190 0x7f1a540d9d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:47:16.043 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:16.040+0000 7f1a62200640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f1a540e3200 0x7f1a5404ceb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:47:16.043 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:16.043+0000 7f1a5b400640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f1a54053690 0x7f1a54187200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:47:16.044 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:16.043+0000 7f1a62200640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f1a54053f80 con 0x7f1a54054190 2024-08-03T20:47:16.044 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:16.043+0000 7f1a62200640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f1a54053e00 con 0x7f1a540e3200 2024-08-03T20:47:16.044 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:16.043+0000 7f1a62200640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7f1a540e7510 con 0x7f1a54053690 2024-08-03T20:47:16.045 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:16.044+0000 7f1a5be00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f1a540e3200 0x7f1a5404ceb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:47:16.046 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:16.046+0000 7f1a5be00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f1a540e3200 0x7f1a5404ceb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.32:43950/0 (socket says 172.21.15.32:43950) 2024-08-03T20:47:16.046 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:16.046+0000 7f1a5be00640 1 -- 172.21.15.32:0/2519591636 learned_addr learned my addr 172.21.15.32:0/2519591636 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:47:16.046 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:16.046+0000 7f1a5aa00640 1 --2- 172.21.15.32:0/2519591636 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1a54054190 0x7f1a540d9d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:47:16.047 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:16.047+0000 7f73bbe00640 1 -- 172.21.15.32:0/3244316475 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f73bc0d2450 msgr2=0x7f73bc0c68b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:47:16.048 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:16.047+0000 7f73bbe00640 1 --2- 172.21.15.32:0/3244316475 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f73bc0d2450 0x7f73bc0c68b0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f73a800d660 tx=0x7f73a800db20 comp rx=0 tx=0).stop 2024-08-03T20:47:16.048 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:16.047+0000 7f73bbe00640 1 -- 172.21.15.32:0/3244316475 shutdown_connections 2024-08-03T20:47:16.048 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:16.047+0000 7f73bbe00640 1 --2- 172.21.15.32:0/3244316475 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f73bc0d2450 0x7f73bc0c68b0 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:16.048 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:16.047+0000 7f73bbe00640 1 --2- 172.21.15.32:0/3244316475 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f73bc1531e0 0x7f73bc1535f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:16.048 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:16.047+0000 7f73bbe00640 1 --2- 172.21.15.32:0/3244316475 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f73bc0b9300 0x7f73bc0b2230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:16.048 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:16.047+0000 7f73bbe00640 1 -- 172.21.15.32:0/3244316475 >> 172.21.15.32:0/3244316475 conn(0x7f73bc0d4160 msgr2=0x7f73bc0d4570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:47:16.048 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:16.047+0000 7f73bbe00640 1 -- 172.21.15.32:0/3244316475 shutdown_connections 2024-08-03T20:47:16.049 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:16.047+0000 7f73bbe00640 1 -- 172.21.15.32:0/3244316475 wait complete. 2024-08-03T20:47:16.049 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:16.048+0000 7f1a5be00640 1 -- 172.21.15.32:0/2519591636 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f1a54053690 msgr2=0x7f1a54187200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:47:16.049 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:16.048+0000 7f1a5be00640 1 --2- 172.21.15.32:0/2519591636 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f1a54053690 0x7f1a54187200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:16.049 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:16.048+0000 7f1a5be00640 1 -- 172.21.15.32:0/2519591636 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1a54054190 msgr2=0x7f1a540d9d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:47:16.049 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:16.049+0000 7f1a5be00640 1 --2- 172.21.15.32:0/2519591636 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1a54054190 0x7f1a540d9d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:16.050 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:16.049+0000 7f1a5be00640 1 -- 172.21.15.32:0/2519591636 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1a540dd670 con 0x7f1a540e3200 2024-08-03T20:47:16.050 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:16.049+0000 7f1a5aa00640 1 --2- 172.21.15.32:0/2519591636 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1a54054190 0x7f1a540d9d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-03T20:47:16.054 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:16.050+0000 7f7c3ea00640 1 -- 172.21.15.32:0/284859340 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+74906 (secure 0 0 0) 0x7f7c38021020 con 0x7f7c480cf070 2024-08-03T20:47:16.054 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:16.053+0000 7f1a5be00640 1 --2- 172.21.15.32:0/2519591636 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f1a540e3200 0x7f1a5404ceb0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f1a5c066470 tx=0x7f1a5c06ef40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:47:16.054 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:16.053+0000 7f1a4f400640 1 -- 172.21.15.32:0/2519591636 <== mon.1 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a5c06aad0 con 0x7f1a540e3200 2024-08-03T20:47:16.054 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:16.053+0000 7f1a4f400640 1 -- 172.21.15.32:0/2519591636 <== mon.1 v2:172.21.15.118:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f1a5c073470 con 0x7f1a540e3200 2024-08-03T20:47:16.054 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:16.053+0000 7f1a4f400640 1 -- 172.21.15.32:0/2519591636 <== mon.1 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a5c073900 con 0x7f1a540e3200 2024-08-03T20:47:16.057 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:16.057+0000 7f1a62200640 1 -- 172.21.15.32:0/2519591636 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1a540da5c0 con 0x7f1a540e3200 2024-08-03T20:47:16.058 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:16.057+0000 7f1a62200640 1 -- 172.21.15.32:0/2519591636 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1a54187af0 con 0x7f1a540e3200 2024-08-03T20:47:16.059 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:16.059+0000 7f1a4f400640 1 -- 172.21.15.32:0/2519591636 <== mon.1 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f1a5c097450 con 0x7f1a540e3200 2024-08-03T20:47:16.059 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:16.059+0000 7f1a4f400640 1 -- 172.21.15.32:0/2519591636 <== mon.1 v2:172.21.15.118:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f1a5c096790 con 0x7f1a540e3200 2024-08-03T20:47:16.064 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:16.063+0000 7f1a62200640 1 -- 172.21.15.32:0/2519591636 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1a540dd840 con 0x7f1a540e3200 2024-08-03T20:47:16.067 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:16.066+0000 7f9f99c00640 1 -- 172.21.15.32:0/2300322792 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f9f940ccc50 msgr2=0x7f9f940cd060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:47:16.067 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:16.066+0000 7f9f99c00640 1 --2- 172.21.15.32:0/2300322792 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f9f940ccc50 0x7f9f940cd060 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f9f9406fd70 tx=0x7f9f8c02bad0 comp rx=0 tx=0).stop 2024-08-03T20:47:16.067 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:16.066+0000 7f9f99c00640 1 -- 172.21.15.32:0/2300322792 shutdown_connections 2024-08-03T20:47:16.067 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:16.066+0000 7f9f99c00640 1 --2- 172.21.15.32:0/2300322792 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f9f940ccc50 0x7f9f940cd060 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:16.067 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:16.066+0000 7f9f99c00640 1 --2- 172.21.15.32:0/2300322792 >> v2:172.21.15.32:3300/0 conn(0x7f9f940cdb10 0x7f9f940cad10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:16.067 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:16.066+0000 7f9f99c00640 1 -- 172.21.15.32:0/2300322792 >> 172.21.15.32:0/2300322792 conn(0x7f9f940b1a70 msgr2=0x7f9f940b1e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:47:16.067 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:16.066+0000 7f9f99c00640 1 -- 172.21.15.32:0/2300322792 shutdown_connections 2024-08-03T20:47:16.067 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:16.066+0000 7f9f99c00640 1 -- 172.21.15.32:0/2300322792 wait complete. 2024-08-03T20:47:16.071 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:16.069+0000 7f1a4f400640 1 -- 172.21.15.32:0/2519591636 <== mon.1 v2:172.21.15.118:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+74906 (secure 0 0 0) 0x7f1a5c09d020 con 0x7f1a540e3200 2024-08-03T20:47:16.078 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:16.078+0000 7f9f99c00640 1 Processor -- start 2024-08-03T20:47:16.079 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:16.079+0000 7f9f99c00640 1 -- start start 2024-08-03T20:47:16.083 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:16.083+0000 7f9f99c00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f9f940ccc50 0x7f9f940c96c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:47:16.083 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:16.083+0000 7f9f99c00640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f9f940cdb10 0x7f9f94158270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:47:16.084 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:16.083+0000 7f9f99c00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9f9415e020 0x7f9f941587b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:47:16.084 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:16.083+0000 7f9f99c00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f9f940cc980 con 0x7f9f9415e020 2024-08-03T20:47:16.084 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:16.083+0000 7f9f99c00640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f9f940cc800 con 0x7f9f940ccc50 2024-08-03T20:47:16.084 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:16.083+0000 7f9f99c00640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7f9f940ce0e0 con 0x7f9f940cdb10 2024-08-03T20:47:16.087 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:16.087+0000 7f9f93e00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f9f940ccc50 0x7f9f940c96c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:47:16.087 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:16.087+0000 7f9f93e00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f9f940ccc50 0x7f9f940c96c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.32:43968/0 (socket says 172.21.15.32:43968) 2024-08-03T20:47:16.087 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:16.087+0000 7f9f93e00640 1 -- 172.21.15.32:0/670129250 learned_addr learned my addr 172.21.15.32:0/670129250 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:47:16.088 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:16.087+0000 7f9f99200640 1 --2- 172.21.15.32:0/670129250 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9f9415e020 0x7f9f941587b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:47:16.088 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:16.087+0000 7f9f93400640 1 --2- 172.21.15.32:0/670129250 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f9f940cdb10 0x7f9f94158270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:47:16.088 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:16.087+0000 7f9f93e00640 1 -- 172.21.15.32:0/670129250 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f9f940cdb10 msgr2=0x7f9f94158270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:47:16.088 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:16.087+0000 7f9f93e00640 1 --2- 172.21.15.32:0/670129250 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f9f940cdb10 0x7f9f94158270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:16.088 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:16.088+0000 7f9f93e00640 1 -- 172.21.15.32:0/670129250 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9f9415e020 msgr2=0x7f9f941587b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:47:16.089 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:16.088+0000 7f9f93e00640 1 --2- 172.21.15.32:0/670129250 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9f9415e020 0x7f9f941587b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:16.089 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:16.088+0000 7f9f93e00640 1 -- 172.21.15.32:0/670129250 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9f940b0720 con 0x7f9f940ccc50 2024-08-03T20:47:16.089 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:16.088+0000 7f9f99200640 1 --2- 172.21.15.32:0/670129250 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9f9415e020 0x7f9f941587b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-03T20:47:16.089 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:16.088+0000 7f9f93e00640 1 --2- 172.21.15.32:0/670129250 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f9f940ccc50 0x7f9f940c96c0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f9f8000d7a0 tx=0x7f9f8000dc60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:47:16.089 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:16.089+0000 7f9f8be00640 1 -- 172.21.15.32:0/670129250 <== mon.1 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f80017070 con 0x7f9f940ccc50 2024-08-03T20:47:16.089 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:16.089+0000 7f9f8be00640 1 -- 172.21.15.32:0/670129250 <== mon.1 v2:172.21.15.118:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f9f80009d20 con 0x7f9f940ccc50 2024-08-03T20:47:16.090 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:16.089+0000 7f9f8be00640 1 -- 172.21.15.32:0/670129250 <== mon.1 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f800106c0 con 0x7f9f940ccc50 2024-08-03T20:47:16.090 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:16.089+0000 7f9f99c00640 1 -- 172.21.15.32:0/670129250 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9f940ae570 con 0x7f9f940ccc50 2024-08-03T20:47:16.090 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:16.089+0000 7f9f99c00640 1 -- 172.21.15.32:0/670129250 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9f940aeb30 con 0x7f9f940ccc50 2024-08-03T20:47:16.095 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:16.094+0000 7f9f8be00640 1 -- 172.21.15.32:0/670129250 <== mon.1 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f9f8001a070 con 0x7f9f940ccc50 2024-08-03T20:47:16.095 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:16.094+0000 7f9f8be00640 1 -- 172.21.15.32:0/670129250 <== mon.1 v2:172.21.15.118:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f9f8001cb00 con 0x7f9f940ccc50 2024-08-03T20:47:16.096 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:16.095+0000 7f9f99c00640 1 -- 172.21.15.32:0/670129250 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9f940cd060 con 0x7f9f940ccc50 2024-08-03T20:47:16.099 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:16.099+0000 7f9f8be00640 1 -- 172.21.15.32:0/670129250 <== mon.1 v2:172.21.15.118:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+74906 (secure 0 0 0) 0x7f9f80010860 con 0x7f9f940ccc50 2024-08-03T20:47:16.109 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:16.108+0000 7fee3a800640 1 -- 172.21.15.32:0/2728548953 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fee2c053e60 msgr2=0x7fee2c0361e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:47:16.109 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:16.108+0000 7fee3a800640 1 --2- 172.21.15.32:0/2728548953 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fee2c053e60 0x7fee2c0361e0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fee2c00ecb0 tx=0x7fee2002a390 comp rx=0 tx=0).stop 2024-08-03T20:47:16.111 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:16.111+0000 7fee3a800640 1 -- 172.21.15.32:0/2728548953 shutdown_connections 2024-08-03T20:47:16.111 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:16.111+0000 7fee3a800640 1 --2- 172.21.15.32:0/2728548953 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fee2c053e60 0x7fee2c0361e0 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:16.111 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:16.111+0000 7fee3a800640 1 --2- 172.21.15.32:0/2728548953 >> v2:172.21.15.118:3300/0 conn(0x7fee2c057050 0x7fee2c0547b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:16.112 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:16.111+0000 7fee3a800640 1 -- 172.21.15.32:0/2728548953 >> 172.21.15.32:0/2728548953 conn(0x7fee2c01fbe0 msgr2=0x7fee2c056940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:47:16.118 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:16.111+0000 7fee3a800640 1 -- 172.21.15.32:0/2728548953 shutdown_connections 2024-08-03T20:47:16.118 DEBUG:tasks.ceph:waiting for RemoteProcess(client=, args=['sudo', 'ceph', '--cluster', 'ceph', 'config', 'set', 'mds', 'mds_debug_scatterstat', 'True'], check_status=True, hostname='smithi032') 2024-08-03T20:47:16.120 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:16.111+0000 7fee3a800640 1 -- 172.21.15.32:0/2728548953 wait complete. 2024-08-03T20:47:16.120 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:16.119+0000 7fee3a800640 1 Processor -- start 2024-08-03T20:47:16.121 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:16.120+0000 7fee3a800640 1 -- start start 2024-08-03T20:47:16.125 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:16.125+0000 7f63b0e00640 1 -- 172.21.15.32:0/3858798480 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f63ac0d0fb0 msgr2=0x7f63ac0b7d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:47:16.125 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:16.125+0000 7f63b0e00640 1 --2- 172.21.15.32:0/3858798480 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f63ac0d0fb0 0x7f63ac0b7d00 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f63a4005fd0 tx=0x7f63a402ab70 comp rx=0 tx=0).stop 2024-08-03T20:47:16.126 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:16.125+0000 7fee3a800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fee2c053a50 0x7fee2c17b560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:47:16.126 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:16.125+0000 7fee3a800640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fee2c053e60 0x7fee2c17e070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:47:16.126 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:16.126+0000 7fee39400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fee2c053a50 0x7fee2c17b560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:47:16.130 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:16.128+0000 7f63b0e00640 1 -- 172.21.15.32:0/3858798480 shutdown_connections 2024-08-03T20:47:16.130 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:16.128+0000 7f63b0e00640 1 --2- 172.21.15.32:0/3858798480 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f63ac0cf9c0 0x7f63ac0d0a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:16.130 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:16.128+0000 7f63b0e00640 1 --2- 172.21.15.32:0/3858798480 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f63ac0d0fb0 0x7f63ac0b7d00 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:16.130 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:16.128+0000 7f63b0e00640 1 -- 172.21.15.32:0/3858798480 >> 172.21.15.32:0/3858798480 conn(0x7f63ac09d660 msgr2=0x7f63ac0d4330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:47:16.130 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:16.128+0000 7f63b0e00640 1 -- 172.21.15.32:0/3858798480 shutdown_connections 2024-08-03T20:47:16.130 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:16.128+0000 7f63b0e00640 1 -- 172.21.15.32:0/3858798480 wait complete. 2024-08-03T20:47:16.130 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:16.126+0000 7fee39400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fee2c053a50 0x7fee2c17b560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:43276/0 (socket says 172.21.15.32:43276) 2024-08-03T20:47:16.131 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:16.126+0000 7fee39400640 1 -- 172.21.15.32:0/2090010868 learned_addr learned my addr 172.21.15.32:0/2090010868 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:47:16.131 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:16.127+0000 7fee38a00640 1 --2- 172.21.15.32:0/2090010868 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fee2c053e60 0x7fee2c17e070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:47:16.131 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:16.127+0000 7fee3a800640 1 --2- 172.21.15.32:0/2090010868 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fee2c057050 0x7fee2c04e360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:47:16.131 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:16.127+0000 7fee3a800640 1 -- 172.21.15.32:0/2090010868 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fee2c0376e0 con 0x7fee2c053a50 2024-08-03T20:47:16.131 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:16.127+0000 7fee39e00640 1 --2- 172.21.15.32:0/2090010868 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fee2c057050 0x7fee2c04e360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:47:16.131 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:16.127+0000 7fee3a800640 1 -- 172.21.15.32:0/2090010868 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7fee2c051a50 con 0x7fee2c053e60 2024-08-03T20:47:16.131 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:16.127+0000 7fee3a800640 1 -- 172.21.15.32:0/2090010868 --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7fee2c054cf0 con 0x7fee2c057050 2024-08-03T20:47:16.132 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:16.128+0000 7fee39400640 1 -- 172.21.15.32:0/2090010868 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fee2c057050 msgr2=0x7fee2c04e360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:47:16.132 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:16.128+0000 7fee39400640 1 --2- 172.21.15.32:0/2090010868 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fee2c057050 0x7fee2c04e360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:16.132 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:16.128+0000 7fee39400640 1 -- 172.21.15.32:0/2090010868 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fee2c053e60 msgr2=0x7fee2c17e070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:47:16.132 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:16.128+0000 7fee39400640 1 --2- 172.21.15.32:0/2090010868 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fee2c053e60 0x7fee2c17e070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:16.132 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:16.128+0000 7fee39400640 1 -- 172.21.15.32:0/2090010868 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fee2c17baa0 con 0x7fee2c053a50 2024-08-03T20:47:16.132 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:16.129+0000 7fee39400640 1 --2- 172.21.15.32:0/2090010868 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fee2c053a50 0x7fee2c17b560 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fee34051e10 tx=0x7fee340512a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:47:16.132 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:16.129+0000 7fee31600640 1 -- 172.21.15.32:0/2090010868 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee34091070 con 0x7fee2c053a50 2024-08-03T20:47:16.133 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:16.129+0000 7fee31600640 1 -- 172.21.15.32:0/2090010868 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fee34096750 con 0x7fee2c053a50 2024-08-03T20:47:16.133 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:16.129+0000 7fee31600640 1 -- 172.21.15.32:0/2090010868 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee34096b40 con 0x7fee2c053a50 2024-08-03T20:47:16.133 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:16.129+0000 7fee3a800640 1 -- 172.21.15.32:0/2090010868 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fee2c17e5b0 con 0x7fee2c053a50 2024-08-03T20:47:16.133 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:16.129+0000 7fee3a800640 1 -- 172.21.15.32:0/2090010868 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fee2c17eda0 con 0x7fee2c053a50 2024-08-03T20:47:16.133 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:16.131+0000 7fee31600640 1 -- 172.21.15.32:0/2090010868 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fee34096ce0 con 0x7fee2c053a50 2024-08-03T20:47:16.133 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:16.131+0000 7fee38a00640 1 --2- 172.21.15.32:0/2090010868 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fee2c053e60 0x7fee2c17e070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-03T20:47:16.133 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:16.131+0000 7fee39e00640 1 --2- 172.21.15.32:0/2090010868 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fee2c057050 0x7fee2c04e360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-03T20:47:16.135 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:16.135+0000 7f63b0e00640 1 Processor -- start 2024-08-03T20:47:16.135 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:16.135+0000 7f63b0e00640 1 -- start start 2024-08-03T20:47:16.136 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:16.136+0000 7fee31600640 1 -- 172.21.15.32:0/2090010868 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7fee340953c0 con 0x7fee2c053a50 2024-08-03T20:47:16.138 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:16.138+0000 7f63b0e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f63ac0b59f0 0x7f63ac1547d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:47:16.140 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:16.139+0000 7f63a9e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f63ac0b59f0 0x7f63ac1547d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:47:16.140 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:16.139+0000 7f63a9e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f63ac0b59f0 0x7f63ac1547d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:43278/0 (socket says 172.21.15.32:43278) 2024-08-03T20:47:16.140 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:16.139+0000 7f63a9e00640 1 -- 172.21.15.32:0/2997032322 learned_addr learned my addr 172.21.15.32:0/2997032322 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:47:16.140 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:16.139+0000 7f63b0e00640 1 --2- 172.21.15.32:0/2997032322 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f63ac0cf9c0 0x7f63ac0c8f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:47:16.141 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:16.140+0000 7f63b0e00640 1 --2- 172.21.15.32:0/2997032322 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f63ac0d0fb0 0x7f63ac0c9b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:47:16.141 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:16.140+0000 7f63a9400640 1 --2- 172.21.15.32:0/2997032322 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f63ac0cf9c0 0x7f63ac0c8f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:47:16.141 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:16.140+0000 7f63b0e00640 1 -- 172.21.15.32:0/2997032322 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f63ac0b88d0 con 0x7f63ac0b59f0 2024-08-03T20:47:16.142 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:16.140+0000 7f8090e00640 1 -- 172.21.15.32:0/2437226764 --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_command([{prefix=config set, name=mds_op_complaint_time}] v 0) -- 0x7f808c162560 con 0x7f808c06d430 2024-08-03T20:47:16.142 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:16.140+0000 7f63aa800640 1 --2- 172.21.15.32:0/2997032322 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f63ac0d0fb0 0x7f63ac0c9b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:47:16.142 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:16.140+0000 7f63b0e00640 1 -- 172.21.15.32:0/2997032322 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f63ac0b8750 con 0x7f63ac0cf9c0 2024-08-03T20:47:16.142 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:16.140+0000 7f63b0e00640 1 -- 172.21.15.32:0/2997032322 --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7f63ac0b8a50 con 0x7f63ac0d0fb0 2024-08-03T20:47:16.143 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:16.141+0000 7fee3a800640 1 -- 172.21.15.32:0/2090010868 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fee2c17efe0 con 0x7fee2c053a50 2024-08-03T20:47:16.143 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:16.141+0000 7f9c9a600640 1 -- 172.21.15.32:0/2445254085 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command([{prefix=config set, name=osd_op_complaint_time}] v 0) -- 0x7f9c70005580 con 0x7f9c940cfb40 2024-08-03T20:47:16.144 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:16.141+0000 7f63a9400640 1 -- 172.21.15.32:0/2997032322 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f63ac0d0fb0 msgr2=0x7f63ac0c9b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:47:16.144 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:16.141+0000 7f63a9400640 1 --2- 172.21.15.32:0/2997032322 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f63ac0d0fb0 0x7f63ac0c9b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:16.144 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:16.141+0000 7f63a9400640 1 -- 172.21.15.32:0/2997032322 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f63ac0b59f0 msgr2=0x7f63ac1547d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:47:16.144 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:16.141+0000 7f63a9400640 1 --2- 172.21.15.32:0/2997032322 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f63ac0b59f0 0x7f63ac1547d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:16.144 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:16.141+0000 7f63a9400640 1 -- 172.21.15.32:0/2997032322 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f63ac154d10 con 0x7f63ac0cf9c0 2024-08-03T20:47:16.144 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:16.142+0000 7f63a9400640 1 --2- 172.21.15.32:0/2997032322 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f63ac0cf9c0 0x7f63ac0c8f60 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f639c00d680 tx=0x7f639c00db40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:47:16.144 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:16.143+0000 7f63a2000640 1 -- 172.21.15.32:0/2997032322 <== mon.1 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f639c017070 con 0x7f63ac0cf9c0 2024-08-03T20:47:16.144 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:16.143+0000 7f63b0e00640 1 -- 172.21.15.32:0/2997032322 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f63ac0c7f00 con 0x7f63ac0cf9c0 2024-08-03T20:47:16.144 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:16.143+0000 7f63b0e00640 1 -- 172.21.15.32:0/2997032322 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f63ac1f6230 con 0x7f63ac0cf9c0 2024-08-03T20:47:16.144 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:16.143+0000 7f63a2000640 1 -- 172.21.15.32:0/2997032322 <== mon.1 v2:172.21.15.118:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f639c005310 con 0x7f63ac0cf9c0 2024-08-03T20:47:16.145 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:16.143+0000 7f63a2000640 1 -- 172.21.15.32:0/2997032322 <== mon.1 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f639c009820 con 0x7f63ac0cf9c0 2024-08-03T20:47:16.145 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:16.145+0000 7f63a2000640 1 -- 172.21.15.32:0/2997032322 <== mon.1 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f639c0099c0 con 0x7f63ac0cf9c0 2024-08-03T20:47:16.147 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:16.146+0000 7f63a2000640 1 -- 172.21.15.32:0/2997032322 <== mon.1 v2:172.21.15.118:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f639c0105a0 con 0x7f63ac0cf9c0 2024-08-03T20:47:16.147 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:16.147+0000 7f63b0e00640 1 -- 172.21.15.32:0/2997032322 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6384005290 con 0x7f63ac0cf9c0 2024-08-03T20:47:16.150 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:16.150+0000 7fee31600640 1 -- 172.21.15.32:0/2090010868 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+74906 (secure 0 0 0) 0x7fee2c17efe0 con 0x7fee2c053a50 2024-08-03T20:47:16.151 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:16.151+0000 7f4d72a00640 1 -- 172.21.15.32:0/3742093140 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f4d6c0d26e0 msgr2=0x7f4d6c0d4090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:47:16.151 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:16.151+0000 7f4d72a00640 1 --2- 172.21.15.32:0/3742093140 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f4d6c0d26e0 0x7f4d6c0d4090 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f4d60005fd0 tx=0x7f4d6002a530 comp rx=0 tx=0).stop 2024-08-03T20:47:16.152 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:16.152+0000 7f4d72a00640 1 -- 172.21.15.32:0/3742093140 shutdown_connections 2024-08-03T20:47:16.152 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:16.152+0000 7f4d72a00640 1 --2- 172.21.15.32:0/3742093140 >> v2:172.21.15.118:3300/0 conn(0x7f4d6c0d45d0 0x7f4d6c162110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:16.152 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:16.152+0000 7f4d72a00640 1 --2- 172.21.15.32:0/3742093140 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f4d6c0d26e0 0x7f4d6c0d4090 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:16.152 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:16.152+0000 7f4d72a00640 1 -- 172.21.15.32:0/3742093140 >> 172.21.15.32:0/3742093140 conn(0x7f4d6c08d7e0 msgr2=0x7f4d6c09d410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:47:16.153 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:16.153+0000 7f8082000640 1 -- 172.21.15.32:0/2437226764 <== mon.2 v2:172.21.15.118:3301/0 7 ==== mon_command_ack([{prefix=config set, name=mds_op_complaint_time}]=0 v11)=0 v11) ==== 121+0+0 (secure 0 0 0) 0x7f807c011d30 con 0x7f808c06d430 2024-08-03T20:47:16.154 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:16.150+0000 7f63a2000640 1 -- 172.21.15.32:0/2997032322 <== mon.1 v2:172.21.15.118:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+74906 (secure 0 0 0) 0x7f63ac154d10 con 0x7f63ac0cf9c0 2024-08-03T20:47:16.155 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:16.155+0000 7f4d72a00640 1 -- 172.21.15.32:0/3742093140 shutdown_connections 2024-08-03T20:47:16.155 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:16.155+0000 7f4d72a00640 1 -- 172.21.15.32:0/3742093140 wait complete. 2024-08-03T20:47:16.156 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:16.156+0000 7f8077400640 1 -- 172.21.15.32:0/2437226764 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f808c06d430 msgr2=0x7f808c153240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:47:16.156 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:16.156+0000 7f8077400640 1 --2- 172.21.15.32:0/2437226764 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f808c06d430 0x7f808c153240 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f807c00c430 tx=0x7f807c00c8f0 comp rx=0 tx=0).stop 2024-08-03T20:47:16.157 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:16.156+0000 7f8077400640 1 -- 172.21.15.32:0/2437226764 shutdown_connections 2024-08-03T20:47:16.157 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:16.156+0000 7f8077400640 1 --2- 172.21.15.32:0/2437226764 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f808c0cf990 0x7f808c0c9cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:16.157 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:16.156+0000 7f8077400640 1 --2- 172.21.15.32:0/2437226764 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f808c162150 0x7f808c157640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:16.157 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:16.156+0000 7f8077400640 1 --2- 172.21.15.32:0/2437226764 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f808c06d430 0x7f808c153240 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:16.157 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:16.157+0000 7f8077400640 1 -- 172.21.15.32:0/2437226764 >> 172.21.15.32:0/2437226764 conn(0x7f808c0d4280 msgr2=0x7f808c15d680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:47:16.157 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:16.157+0000 7f8077400640 1 -- 172.21.15.32:0/2437226764 shutdown_connections 2024-08-03T20:47:16.158 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:16.157+0000 7f8077400640 1 -- 172.21.15.32:0/2437226764 wait complete. 2024-08-03T20:47:16.159 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:16.159+0000 7f4d72a00640 1 Processor -- start 2024-08-03T20:47:16.159 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:16.159+0000 7f4d72a00640 1 -- start start 2024-08-03T20:47:16.160 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:16.159+0000 7f4e42e00640 1 -- 172.21.15.32:0/854577433 --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_command([{prefix=config set, name=mds_debug_scatterstat}] v 0) -- 0x7f4e3c0cb680 con 0x7f4e3c073150 2024-08-03T20:47:16.164 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:16.162+0000 7f4d72a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4d6c0d26e0 0x7f4d6c2025e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:47:16.164 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:16.162+0000 7f4d72a00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f4d6c0d45d0 0x7f4d6c203710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:47:16.164 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:16.162+0000 7f4d72a00640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f4d6c162650 0x7f4d6c0b83b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:47:16.164 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:16.162+0000 7f4d72a00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f4d6c162be0 con 0x7f4d6c0d26e0 2024-08-03T20:47:16.164 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:16.162+0000 7f4d72a00640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f4d6c162a60 con 0x7f4d6c0d45d0 2024-08-03T20:47:16.164 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:16.162+0000 7f4d72a00640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7f4d6c162d60 con 0x7f4d6c162650 2024-08-03T20:47:16.164 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:16.162+0000 7f4d72000640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f4d6c162650 0x7f4d6c0b83b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:47:16.164 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:16.162+0000 7f4d72000640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f4d6c162650 0x7f4d6c0b83b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3301/0 says I am v2:172.21.15.32:49592/0 (socket says 172.21.15.32:49592) 2024-08-03T20:47:16.164 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:16.162+0000 7f4d72000640 1 -- 172.21.15.32:0/3530116150 learned_addr learned my addr 172.21.15.32:0/3530116150 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:47:16.164 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:16.162+0000 7f4d70c00640 1 --2- 172.21.15.32:0/3530116150 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f4d6c0d45d0 0x7f4d6c203710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:47:16.165 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:16.163+0000 7f4d71600640 1 --2- 172.21.15.32:0/3530116150 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4d6c0d26e0 0x7f4d6c2025e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:47:16.165 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:16.163+0000 7f4d72000640 1 -- 172.21.15.32:0/3530116150 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f4d6c0d45d0 msgr2=0x7f4d6c203710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:47:16.165 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:16.163+0000 7f4d72000640 1 --2- 172.21.15.32:0/3530116150 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f4d6c0d45d0 0x7f4d6c203710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:16.165 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:16.163+0000 7f4d72000640 1 -- 172.21.15.32:0/3530116150 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4d6c0d26e0 msgr2=0x7f4d6c2025e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:47:16.165 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:16.163+0000 7f4d72000640 1 --2- 172.21.15.32:0/3530116150 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4d6c0d26e0 0x7f4d6c2025e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:16.165 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:16.163+0000 7f4d72000640 1 -- 172.21.15.32:0/3530116150 --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4d6c202b20 con 0x7f4d6c162650 2024-08-03T20:47:16.165 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:16.163+0000 7f4d70c00640 1 --2- 172.21.15.32:0/3530116150 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f4d6c0d45d0 0x7f4d6c203710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-03T20:47:16.165 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:16.163+0000 7f4d72000640 1 --2- 172.21.15.32:0/3530116150 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f4d6c162650 0x7f4d6c0b83b0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f4d5800b440 tx=0x7f4d5800b900 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:47:16.165 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:16.164+0000 7f9c90c00640 1 -- 172.21.15.32:0/2445254085 <== mon.1 v2:172.21.15.118:3300/0 7 ==== mon_command_ack([{prefix=config set, name=osd_op_complaint_time}]=0 v12)=0 v12) ==== 121+0+0 (secure 0 0 0) 0x7f9c940d49e0 con 0x7f9c940cfb40 2024-08-03T20:47:16.166 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:16.164+0000 7f4d69600640 1 -- 172.21.15.32:0/3530116150 <== mon.2 v2:172.21.15.118:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d58011830 con 0x7f4d6c162650 2024-08-03T20:47:16.166 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:16.164+0000 7f4d69600640 1 -- 172.21.15.32:0/3530116150 <== mon.2 v2:172.21.15.118:3301/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f4d58016070 con 0x7f4d6c162650 2024-08-03T20:47:16.166 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:16.165+0000 7f4d69600640 1 -- 172.21.15.32:0/3530116150 <== mon.2 v2:172.21.15.118:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d58011d60 con 0x7f4d6c162650 2024-08-03T20:47:16.166 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:16.165+0000 7f4d72a00640 1 -- 172.21.15.32:0/3530116150 --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4d6c203030 con 0x7f4d6c162650 2024-08-03T20:47:16.167 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:16.166+0000 7f9c9a600640 1 -- 172.21.15.32:0/2445254085 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f9c940cfb40 msgr2=0x7f9c94202170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:47:16.167 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:16.166+0000 7f9c9a600640 1 --2- 172.21.15.32:0/2445254085 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f9c940cfb40 0x7f9c94202170 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f9c88033040 tx=0x7f9c88004340 comp rx=0 tx=0).stop 2024-08-03T20:47:16.167 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:16.166+0000 7f9c9a600640 1 -- 172.21.15.32:0/2445254085 shutdown_connections 2024-08-03T20:47:16.167 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:16.166+0000 7f9c9a600640 1 --2- 172.21.15.32:0/2445254085 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9c940d45d0 0x7f9c940c9a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:16.168 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:16.166+0000 7f9c9a600640 1 --2- 172.21.15.32:0/2445254085 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f9c940d26e0 0x7f9c940c9500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:16.168 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:16.166+0000 7f9c9a600640 1 --2- 172.21.15.32:0/2445254085 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f9c940cfb40 0x7f9c94202170 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:16.168 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:16.166+0000 7f9c9a600640 1 -- 172.21.15.32:0/2445254085 >> 172.21.15.32:0/2445254085 conn(0x7f9c9408d7e0 msgr2=0x7f9c94161be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:47:16.168 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:16.167+0000 7f9c9a600640 1 -- 172.21.15.32:0/2445254085 shutdown_connections 2024-08-03T20:47:16.168 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:16.167+0000 7f9c9a600640 1 -- 172.21.15.32:0/2445254085 wait complete. 2024-08-03T20:47:16.168 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:16.165+0000 7f4d72a00640 1 -- 172.21.15.32:0/3530116150 --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f4d6c15a3d0 con 0x7f4d6c162650 2024-08-03T20:47:16.169 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:16.168+0000 7f4d69600640 1 -- 172.21.15.32:0/3530116150 <== mon.2 v2:172.21.15.118:3301/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f4d5801b460 con 0x7f4d6c162650 2024-08-03T20:47:16.169 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:16.168+0000 7f4d69600640 1 -- 172.21.15.32:0/3530116150 <== mon.2 v2:172.21.15.118:3301/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f4d5801a8b0 con 0x7f4d6c162650 2024-08-03T20:47:16.169 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:16.168+0000 7f4d72a00640 1 -- 172.21.15.32:0/3530116150 --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4d48005290 con 0x7f4d6c162650 2024-08-03T20:47:16.173 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:16.173+0000 7f4d69600640 1 -- 172.21.15.32:0/3530116150 <== mon.2 v2:172.21.15.118:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+74906 (secure 0 0 0) 0x7f4d5801a3c0 con 0x7f4d6c162650 2024-08-03T20:47:16.175 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:16.175+0000 7f4e39600640 1 -- 172.21.15.32:0/854577433 <== mon.2 v2:172.21.15.118:3301/0 7 ==== mon_command_ack([{prefix=config set, name=mds_debug_scatterstat}]=0 v13)=0 v13) ==== 122+0+0 (secure 0 0 0) 0x7f4e3c0cb680 con 0x7f4e3c073150 2024-08-03T20:47:16.178 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:16.178+0000 7f4e42e00640 1 -- 172.21.15.32:0/854577433 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f4e3c073150 msgr2=0x7f4e3c1f5a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:47:16.178 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:16.178+0000 7f4e42e00640 1 --2- 172.21.15.32:0/854577433 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f4e3c073150 0x7f4e3c1f5a60 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f4e2800c3a0 tx=0x7f4e2800c860 comp rx=0 tx=0).stop 2024-08-03T20:47:16.184 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:16.184+0000 7f4e42e00640 1 -- 172.21.15.32:0/854577433 shutdown_connections 2024-08-03T20:47:16.184 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:16.184+0000 7f4e42e00640 1 --2- 172.21.15.32:0/854577433 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4e3c152350 0x7f4e3c1f5fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:16.184 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:16.184+0000 7f4e42e00640 1 --2- 172.21.15.32:0/854577433 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f4e3c073150 0x7f4e3c1f5a60 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:16.184 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:16.184+0000 7f4e42e00640 1 --2- 172.21.15.32:0/854577433 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f4e3c06f740 0x7f4e3c0b3c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:16.185 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:16.184+0000 7f4e42e00640 1 -- 172.21.15.32:0/854577433 >> 172.21.15.32:0/854577433 conn(0x7f4e3c0b1a70 msgr2=0x7f4e3c15daa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:47:16.185 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:16.184+0000 7f4e42e00640 1 -- 172.21.15.32:0/854577433 shutdown_connections 2024-08-03T20:47:16.185 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:16.184+0000 7f4e42e00640 1 -- 172.21.15.32:0/854577433 wait complete. 2024-08-03T20:47:16.220 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:16.219+0000 7f7c47e00640 1 -- 172.21.15.32:0/284859340 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command([{prefix=config set, name=mds_verify_scatter}] v 0) -- 0x7f7c20005580 con 0x7f7c480cf070 2024-08-03T20:47:16.226 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:16.225+0000 7f1a62200640 1 -- 172.21.15.32:0/2519591636 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command([{prefix=config set, name=rados_osd_op_timeout}] v 0) -- 0x7f1a540e79d0 con 0x7f1a540e3200 2024-08-03T20:47:16.226 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:16.225+0000 7f7c3ea00640 1 -- 172.21.15.32:0/284859340 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mds_verify_scatter}]=0 v14)=0 v14) ==== 119+0+0 (secure 0 0 0) 0x7f7c3800a6e0 con 0x7f7c480cf070 2024-08-03T20:47:16.227 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:16.227+0000 7f7c47e00640 1 -- 172.21.15.32:0/284859340 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7c480cf070 msgr2=0x7f7c481f93d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:47:16.227 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:16.227+0000 7f7c47e00640 1 --2- 172.21.15.32:0/284859340 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7c480cf070 0x7f7c481f93d0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f7c3800c7f0 tx=0x7f7c3800ccb0 comp rx=0 tx=0).stop 2024-08-03T20:47:16.228 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:16.227+0000 7f7c47e00640 1 -- 172.21.15.32:0/284859340 shutdown_connections 2024-08-03T20:47:16.228 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:16.227+0000 7f7c47e00640 1 --2- 172.21.15.32:0/284859340 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f7c481556c0 0x7f7c4806d4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:16.228 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:16.227+0000 7f7c47e00640 1 --2- 172.21.15.32:0/284859340 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f7c480d26e0 0x7f7c480cdc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:16.228 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:16.227+0000 7f7c47e00640 1 --2- 172.21.15.32:0/284859340 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7c480cf070 0x7f7c481f93d0 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:16.228 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:16.227+0000 7f7c47e00640 1 -- 172.21.15.32:0/284859340 >> 172.21.15.32:0/284859340 conn(0x7f7c4808d7e0 msgr2=0x7f7c480b8f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:47:16.228 DEBUG:tasks.ceph:waiting for RemoteProcess(client=, args=['sudo', 'ceph', '--cluster', 'ceph', 'config', 'set', 'mds', 'mds_op_complaint_time', '180'], check_status=True, hostname='smithi032') 2024-08-03T20:47:16.229 DEBUG:tasks.ceph:waiting for RemoteProcess(client=, args=['sudo', 'ceph', '--cluster', 'ceph', 'config', 'set', 'mds', 'mds_verify_scatter', 'True'], check_status=True, hostname='smithi032') 2024-08-03T20:47:16.229 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:16.228+0000 7f7c47e00640 1 -- 172.21.15.32:0/284859340 shutdown_connections 2024-08-03T20:47:16.229 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:16.228+0000 7f7c47e00640 1 -- 172.21.15.32:0/284859340 wait complete. 2024-08-03T20:47:16.236 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:16.235+0000 7f1a4f400640 1 -- 172.21.15.32:0/2519591636 <== mon.1 v2:172.21.15.118:3300/0 7 ==== mon_command_ack([{prefix=config set, name=rados_osd_op_timeout}]=0 v15)=0 v15) ==== 120+0+0 (secure 0 0 0) 0x7f1a5c096a90 con 0x7f1a540e3200 2024-08-03T20:47:16.243 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:16.242+0000 7f1a62200640 1 -- 172.21.15.32:0/2519591636 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f1a540e3200 msgr2=0x7f1a5404ceb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:47:16.243 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:16.243+0000 7f1a62200640 1 --2- 172.21.15.32:0/2519591636 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f1a540e3200 0x7f1a5404ceb0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f1a5c066470 tx=0x7f1a5c06ef40 comp rx=0 tx=0).stop 2024-08-03T20:47:16.244 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:16.243+0000 7f1a62200640 1 -- 172.21.15.32:0/2519591636 shutdown_connections 2024-08-03T20:47:16.244 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:16.243+0000 7f1a62200640 1 --2- 172.21.15.32:0/2519591636 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f1a540e3200 0x7f1a5404ceb0 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:16.244 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:16.243+0000 7f1a62200640 1 --2- 172.21.15.32:0/2519591636 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1a54054190 0x7f1a540d9d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:16.244 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:16.243+0000 7f1a62200640 1 --2- 172.21.15.32:0/2519591636 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f1a54053690 0x7f1a54187200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:16.244 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:16.243+0000 7f1a62200640 1 -- 172.21.15.32:0/2519591636 >> 172.21.15.32:0/2519591636 conn(0x7f1a5403a680 msgr2=0x7f1a540074c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:47:16.245 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:16.244+0000 7f1a62200640 1 -- 172.21.15.32:0/2519591636 shutdown_connections 2024-08-03T20:47:16.245 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:16.244+0000 7f1a62200640 1 -- 172.21.15.32:0/2519591636 wait complete. 2024-08-03T20:47:16.267 DEBUG:tasks.ceph:waiting for RemoteProcess(client=, args=['sudo', 'ceph', '--cluster', 'ceph', 'config', 'set', 'mds', 'osd_op_complaint_time', '180'], check_status=True, hostname='smithi032') 2024-08-03T20:47:16.268 DEBUG:tasks.ceph:waiting for RemoteProcess(client=, args=['sudo', 'ceph', '--cluster', 'ceph', 'config', 'set', 'mds', 'rados_mon_op_timeout', '900'], check_status=True, hostname='smithi032') 2024-08-03T20:47:16.268 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:16.267+0000 7f9f99c00640 1 -- 172.21.15.32:0/670129250 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command([{prefix=config set, name=mon_op_complaint_time}] v 0) -- 0x7f9f94073740 con 0x7f9f940ccc50 2024-08-03T20:47:16.272 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:16.272+0000 7f9f8be00640 1 -- 172.21.15.32:0/670129250 <== mon.1 v2:172.21.15.118:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_op_complaint_time}]=0 v16)=0 v16) ==== 121+0+0 (secure 0 0 0) 0x7f9f80010a40 con 0x7f9f940ccc50 2024-08-03T20:47:16.274 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:16.273+0000 7f9f99c00640 1 -- 172.21.15.32:0/670129250 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f9f940ccc50 msgr2=0x7f9f940c96c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:47:16.274 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:16.273+0000 7f9f99c00640 1 --2- 172.21.15.32:0/670129250 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f9f940ccc50 0x7f9f940c96c0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f9f8000d7a0 tx=0x7f9f8000dc60 comp rx=0 tx=0).stop 2024-08-03T20:47:16.274 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:16.274+0000 7f9f99c00640 1 -- 172.21.15.32:0/670129250 shutdown_connections 2024-08-03T20:47:16.274 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:16.274+0000 7f9f99c00640 1 --2- 172.21.15.32:0/670129250 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9f9415e020 0x7f9f941587b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:16.274 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:16.274+0000 7f9f99c00640 1 --2- 172.21.15.32:0/670129250 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f9f940cdb10 0x7f9f94158270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:16.274 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:16.274+0000 7f9f99c00640 1 --2- 172.21.15.32:0/670129250 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f9f940ccc50 0x7f9f940c96c0 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:16.275 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:16.274+0000 7f9f99c00640 1 -- 172.21.15.32:0/670129250 >> 172.21.15.32:0/670129250 conn(0x7f9f940b1a70 msgr2=0x7f9f9415bcf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:47:16.275 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:16.274+0000 7f9f99c00640 1 -- 172.21.15.32:0/670129250 shutdown_connections 2024-08-03T20:47:16.275 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:16.274+0000 7f9f99c00640 1 -- 172.21.15.32:0/670129250 wait complete. 2024-08-03T20:47:16.297 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:16.297+0000 7f63b0e00640 1 -- 172.21.15.32:0/2997032322 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command([{prefix=config set, name=rados_mon_op_timeout}] v 0) -- 0x7f6384005580 con 0x7f63ac0cf9c0 2024-08-03T20:47:16.300 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:16.300+0000 7f4d72a00640 1 -- 172.21.15.32:0/3530116150 --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_command([{prefix=config set, name=mon_warn_on_pool_no_app}] v 0) -- 0x7f4d48005580 con 0x7f4d6c162650 2024-08-03T20:47:16.301 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:16.301+0000 7f63a2000640 1 -- 172.21.15.32:0/2997032322 <== mon.1 v2:172.21.15.118:3300/0 7 ==== mon_command_ack([{prefix=config set, name=rados_mon_op_timeout}]=0 v17)=0 v17) ==== 120+0+0 (secure 0 0 0) 0x7f639c0047a0 con 0x7f63ac0cf9c0 2024-08-03T20:47:16.303 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:16.302+0000 7f63b0e00640 1 -- 172.21.15.32:0/2997032322 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f63ac0cf9c0 msgr2=0x7f63ac0c8f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:47:16.303 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:16.302+0000 7f63b0e00640 1 --2- 172.21.15.32:0/2997032322 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f63ac0cf9c0 0x7f63ac0c8f60 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f639c00d680 tx=0x7f639c00db40 comp rx=0 tx=0).stop 2024-08-03T20:47:16.303 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:16.302+0000 7f63b0e00640 1 -- 172.21.15.32:0/2997032322 shutdown_connections 2024-08-03T20:47:16.303 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:16.302+0000 7f63b0e00640 1 --2- 172.21.15.32:0/2997032322 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f63ac0cf9c0 0x7f63ac0c8f60 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:16.303 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:16.302+0000 7f63b0e00640 1 --2- 172.21.15.32:0/2997032322 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f63ac0d0fb0 0x7f63ac0c9b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:16.303 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:16.302+0000 7f63b0e00640 1 --2- 172.21.15.32:0/2997032322 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f63ac0b59f0 0x7f63ac1547d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:16.303 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:16.302+0000 7f63b0e00640 1 -- 172.21.15.32:0/2997032322 >> 172.21.15.32:0/2997032322 conn(0x7f63ac09d660 msgr2=0x7f63ac161740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:47:16.303 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:16.302+0000 7f63b0e00640 1 -- 172.21.15.32:0/2997032322 shutdown_connections 2024-08-03T20:47:16.303 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:16.303+0000 7f63b0e00640 1 -- 172.21.15.32:0/2997032322 wait complete. 2024-08-03T20:47:16.306 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:16.305+0000 7f4d69600640 1 -- 172.21.15.32:0/3530116150 <== mon.2 v2:172.21.15.118:3301/0 7 ==== mon_command_ack([{prefix=config set, name=mon_warn_on_pool_no_app}]=0 v18)=0 v18) ==== 125+0+0 (secure 0 0 0) 0x7f4d5801ab00 con 0x7f4d6c162650 2024-08-03T20:47:16.307 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:16.307+0000 7f4d72a00640 1 -- 172.21.15.32:0/3530116150 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f4d6c162650 msgr2=0x7f4d6c0b83b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:47:16.308 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:16.307+0000 7f4d72a00640 1 --2- 172.21.15.32:0/3530116150 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f4d6c162650 0x7f4d6c0b83b0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f4d5800b440 tx=0x7f4d5800b900 comp rx=0 tx=0).stop 2024-08-03T20:47:16.308 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:16.307+0000 7f4d72a00640 1 -- 172.21.15.32:0/3530116150 shutdown_connections 2024-08-03T20:47:16.308 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:16.307+0000 7f4d72a00640 1 --2- 172.21.15.32:0/3530116150 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f4d6c0d45d0 0x7f4d6c203710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:16.308 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:16.307+0000 7f4d72a00640 1 --2- 172.21.15.32:0/3530116150 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f4d6c162650 0x7f4d6c0b83b0 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:16.308 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:16.307+0000 7f4d72a00640 1 --2- 172.21.15.32:0/3530116150 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4d6c0d26e0 0x7f4d6c2025e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:16.308 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:16.307+0000 7f4d72a00640 1 -- 172.21.15.32:0/3530116150 >> 172.21.15.32:0/3530116150 conn(0x7f4d6c08d7e0 msgr2=0x7f4d6c0d0720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:47:16.308 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:16.307+0000 7f4d72a00640 1 -- 172.21.15.32:0/3530116150 shutdown_connections 2024-08-03T20:47:16.308 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:16.307+0000 7f4d72a00640 1 -- 172.21.15.32:0/3530116150 wait complete. 2024-08-03T20:47:16.311 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:16.311+0000 7fee3a800640 1 -- 172.21.15.32:0/2090010868 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command([{prefix=config set, name=osd_op_complaint_time}] v 0) -- 0x7fee2c0335e0 con 0x7fee2c053a50 2024-08-03T20:47:16.314 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:16.314+0000 7fee31600640 1 -- 172.21.15.32:0/2090010868 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{prefix=config set, name=osd_op_complaint_time}]=0 v19)=0 v19) ==== 121+0+0 (secure 0 0 0) 0x7fee340955c0 con 0x7fee2c053a50 2024-08-03T20:47:16.316 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:16.315+0000 7fee3a800640 1 -- 172.21.15.32:0/2090010868 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fee2c053a50 msgr2=0x7fee2c17b560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:47:16.316 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:16.315+0000 7fee3a800640 1 --2- 172.21.15.32:0/2090010868 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fee2c053a50 0x7fee2c17b560 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fee34051e10 tx=0x7fee340512a0 comp rx=0 tx=0).stop 2024-08-03T20:47:16.316 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:16.315+0000 7fee3a800640 1 -- 172.21.15.32:0/2090010868 shutdown_connections 2024-08-03T20:47:16.316 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:16.315+0000 7fee3a800640 1 --2- 172.21.15.32:0/2090010868 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fee2c053e60 0x7fee2c17e070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:16.316 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:16.316+0000 7fee3a800640 1 --2- 172.21.15.32:0/2090010868 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fee2c057050 0x7fee2c04e360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:16.316 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:16.316+0000 7fee3a800640 1 --2- 172.21.15.32:0/2090010868 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fee2c053a50 0x7fee2c17b560 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:16.317 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:16.316+0000 7fee3a800640 1 -- 172.21.15.32:0/2090010868 >> 172.21.15.32:0/2090010868 conn(0x7fee2c01fbe0 msgr2=0x7fee2c034e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:47:16.317 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:16.316+0000 7fee3a800640 1 -- 172.21.15.32:0/2090010868 shutdown_connections 2024-08-03T20:47:16.317 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:16.316+0000 7fee3a800640 1 -- 172.21.15.32:0/2090010868 wait complete. 2024-08-03T20:47:16.332 DEBUG:tasks.ceph:waiting for RemoteProcess(client=, args=['sudo', 'ceph', '--cluster', 'ceph', 'config', 'set', 'mds', 'rados_osd_op_timeout', '900'], check_status=True, hostname='smithi032') 2024-08-03T20:47:16.332 DEBUG:tasks.ceph:waiting for RemoteProcess(client=, args=['sudo', 'ceph', '--cluster', 'ceph', 'config', 'set', 'mon', 'mon_op_complaint_time', '120'], check_status=True, hostname='smithi032') 2024-08-03T20:47:16.332 DEBUG:tasks.ceph:waiting for RemoteProcess(client=, args=['sudo', 'ceph', '--cluster', 'ceph', 'config', 'set', 'mon', 'mon_warn_on_pool_no_app', 'False'], check_status=True, hostname='smithi032') 2024-08-03T20:47:16.333 DEBUG:tasks.ceph:waiting for RemoteProcess(client=, args=['sudo', 'ceph', '--cluster', 'ceph', 'config', 'set', 'osd', 'osd_op_complaint_time', '180'], check_status=True, hostname='smithi032') 2024-08-03T20:47:16.337 INFO:tasks.ceph:Setting crush tunables to default 2024-08-03T20:47:16.337 DEBUG:teuthology.orchestra.run.smithi032:> sudo ceph --cluster ceph osd crush tunables default 2024-08-03T20:47:16.602 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:16.600+0000 7ff2ee600640 1 -- 172.21.15.32:0/3153759777 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ff2e80b59f0 msgr2=0x7ff2e80d4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:47:16.602 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:16.600+0000 7ff2ee600640 1 --2- 172.21.15.32:0/3153759777 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ff2e80b59f0 0x7ff2e80d4cc0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7ff2e808d2b0 tx=0x7ff2d802a520 comp rx=0 tx=0).stop 2024-08-03T20:47:16.603 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:16.601+0000 7ff2ee600640 1 -- 172.21.15.32:0/3153759777 shutdown_connections 2024-08-03T20:47:16.603 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:16.601+0000 7ff2ee600640 1 --2- 172.21.15.32:0/3153759777 >> v2:172.21.15.32:3300/0 conn(0x7ff2e80d0fb0 0x7ff2e80d2230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:16.603 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:16.601+0000 7ff2ee600640 1 --2- 172.21.15.32:0/3153759777 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7ff2e80cf9c0 0x7ff2e80d0a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:16.603 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:16.601+0000 7ff2ee600640 1 --2- 172.21.15.32:0/3153759777 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ff2e80b59f0 0x7ff2e80d4cc0 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:16.603 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:16.601+0000 7ff2ee600640 1 -- 172.21.15.32:0/3153759777 >> 172.21.15.32:0/3153759777 conn(0x7ff2e809d660 msgr2=0x7ff2e80d4330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:47:16.604 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:16.601+0000 7ff2ee600640 1 -- 172.21.15.32:0/3153759777 shutdown_connections 2024-08-03T20:47:16.604 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:16.601+0000 7ff2ee600640 1 -- 172.21.15.32:0/3153759777 wait complete. 2024-08-03T20:47:16.608 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:16.603+0000 7ff2ee600640 1 Processor -- start 2024-08-03T20:47:16.609 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:16.603+0000 7ff2ee600640 1 -- start start 2024-08-03T20:47:16.609 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:16.605+0000 7ff2ee600640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ff2e80b59f0 0x7ff2e815c890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:47:16.609 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:16.605+0000 7ff2ee600640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7ff2e80cf9c0 0x7ff2e815c120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:47:16.609 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:16.605+0000 7ff2ee600640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff2e80d0fb0 0x7ff2e815e6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:47:16.609 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:16.605+0000 7ff2ee600640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7ff2e80d2bb0 con 0x7ff2e80d0fb0 2024-08-03T20:47:16.610 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:16.605+0000 7ff2ee600640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7ff2e80d2a30 con 0x7ff2e80b59f0 2024-08-03T20:47:16.610 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:16.605+0000 7ff2ee600640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7ff2e80d2d30 con 0x7ff2e80cf9c0 2024-08-03T20:47:16.610 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:16.605+0000 7ff2e7400640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ff2e80b59f0 0x7ff2e815c890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:47:16.610 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:16.605+0000 7ff2e7e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff2e80d0fb0 0x7ff2e815e6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:47:16.610 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:16.605+0000 7ff2e7400640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ff2e80b59f0 0x7ff2e815c890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.32:44038/0 (socket says 172.21.15.32:44038) 2024-08-03T20:47:16.610 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:16.605+0000 7ff2e7400640 1 -- 172.21.15.32:0/2133458975 learned_addr learned my addr 172.21.15.32:0/2133458975 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:47:16.610 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:16.605+0000 7ff2dfe00640 1 --2- 172.21.15.32:0/2133458975 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7ff2e80cf9c0 0x7ff2e815c120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:47:16.610 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:16.605+0000 7ff2e7400640 1 -- 172.21.15.32:0/2133458975 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7ff2e80cf9c0 msgr2=0x7ff2e815c120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:47:16.610 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:16.605+0000 7ff2e7400640 1 --2- 172.21.15.32:0/2133458975 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7ff2e80cf9c0 0x7ff2e815c120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:16.611 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:16.605+0000 7ff2e7400640 1 -- 172.21.15.32:0/2133458975 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff2e80d0fb0 msgr2=0x7ff2e815e6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:47:16.611 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:16.605+0000 7ff2e7400640 1 --2- 172.21.15.32:0/2133458975 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff2e80d0fb0 0x7ff2e815e6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:16.611 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:16.605+0000 7ff2e7400640 1 -- 172.21.15.32:0/2133458975 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff2e815c660 con 0x7ff2e80b59f0 2024-08-03T20:47:16.611 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:16.605+0000 7ff2dfe00640 1 --2- 172.21.15.32:0/2133458975 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7ff2e80cf9c0 0x7ff2e815c120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-03T20:47:16.611 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:16.606+0000 7ff2e7400640 1 --2- 172.21.15.32:0/2133458975 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ff2e80b59f0 0x7ff2e815c890 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7ff2d8009920 tx=0x7ff2d8033c20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:47:16.611 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:16.606+0000 7ff2df400640 1 -- 172.21.15.32:0/2133458975 <== mon.1 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff2d8033e10 con 0x7ff2e80b59f0 2024-08-03T20:47:16.611 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:16.606+0000 7ff2df400640 1 -- 172.21.15.32:0/2133458975 <== mon.1 v2:172.21.15.118:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7ff2d8031b60 con 0x7ff2e80b59f0 2024-08-03T20:47:16.612 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:16.606+0000 7ff2df400640 1 -- 172.21.15.32:0/2133458975 <== mon.1 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff2d8005020 con 0x7ff2e80b59f0 2024-08-03T20:47:16.612 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:16.606+0000 7ff2ee600640 1 -- 172.21.15.32:0/2133458975 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff2e81f8e70 con 0x7ff2e80b59f0 2024-08-03T20:47:16.612 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:16.606+0000 7ff2ee600640 1 -- 172.21.15.32:0/2133458975 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff2e81f9430 con 0x7ff2e80b59f0 2024-08-03T20:47:16.612 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:16.607+0000 7ff2df400640 1 -- 172.21.15.32:0/2133458975 <== mon.1 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7ff2d8045020 con 0x7ff2e80b59f0 2024-08-03T20:47:16.612 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:16.607+0000 7ff2df400640 1 -- 172.21.15.32:0/2133458975 <== mon.1 v2:172.21.15.118:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7ff2d8004e10 con 0x7ff2e80b59f0 2024-08-03T20:47:16.612 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:16.607+0000 7ff2ee600640 1 -- 172.21.15.32:0/2133458975 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff2e80d0a70 con 0x7ff2e80b59f0 2024-08-03T20:47:16.612 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:16.610+0000 7ff2df400640 1 -- 172.21.15.32:0/2133458975 <== mon.1 v2:172.21.15.118:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+74906 (secure 0 0 0) 0x7ff2d8031770 con 0x7ff2e80b59f0 2024-08-03T20:47:16.682 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:16.681+0000 7ff2ee600640 1 -- 172.21.15.32:0/2133458975 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) -- 0x7ff2e806f070 con 0x7ff2e80b59f0 2024-08-03T20:47:16.686 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:16.686+0000 7ff2df400640 1 -- 172.21.15.32:0/2133458975 <== mon.1 v2:172.21.15.118:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v2) ==== 124+0+0 (secure 0 0 0) 0x7ff2d80055f0 con 0x7ff2e80b59f0 2024-08-03T20:47:16.686 INFO:teuthology.orchestra.run.smithi032.stderr:adjusted tunables profile to default 2024-08-03T20:47:16.687 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:16.687+0000 7ff2ee600640 1 -- 172.21.15.32:0/2133458975 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ff2e80b59f0 msgr2=0x7ff2e815c890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:47:16.687 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:16.687+0000 7ff2ee600640 1 --2- 172.21.15.32:0/2133458975 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ff2e80b59f0 0x7ff2e815c890 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7ff2d8009920 tx=0x7ff2d8033c20 comp rx=0 tx=0).stop 2024-08-03T20:47:16.688 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:16.687+0000 7ff2ee600640 1 -- 172.21.15.32:0/2133458975 shutdown_connections 2024-08-03T20:47:16.688 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:16.687+0000 7ff2ee600640 1 --2- 172.21.15.32:0/2133458975 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff2e80d0fb0 0x7ff2e815e6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:16.688 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:16.687+0000 7ff2ee600640 1 --2- 172.21.15.32:0/2133458975 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7ff2e80cf9c0 0x7ff2e815c120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:16.688 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:16.687+0000 7ff2ee600640 1 --2- 172.21.15.32:0/2133458975 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ff2e80b59f0 0x7ff2e815c890 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:16.688 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:16.687+0000 7ff2ee600640 1 -- 172.21.15.32:0/2133458975 >> 172.21.15.32:0/2133458975 conn(0x7ff2e809d660 msgr2=0x7ff2e81618c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:47:16.688 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:16.687+0000 7ff2ee600640 1 -- 172.21.15.32:0/2133458975 shutdown_connections 2024-08-03T20:47:16.688 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:16.687+0000 7ff2ee600640 1 -- 172.21.15.32:0/2133458975 wait complete. 2024-08-03T20:47:16.712 INFO:tasks.ceph:check_enable_crimson: False 2024-08-03T20:47:16.712 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2024-08-03T20:47:16.712 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-08-03T20:47:16.712 DEBUG:teuthology.orchestra.run.smithi032:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2024-08-03T20:47:16.769 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-08-03T20:47:16.769 DEBUG:teuthology.orchestra.run.smithi032:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2024-08-03T20:47:16.864 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-08-03T20:47:16.864 DEBUG:teuthology.orchestra.run.smithi032:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2024-08-03T20:47:16.962 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-08-03T20:47:16.962 DEBUG:teuthology.orchestra.run.smithi032:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2024-08-03T20:47:17.058 DEBUG:teuthology.orchestra.run.smithi118:> set -ex 2024-08-03T20:47:17.058 DEBUG:teuthology.orchestra.run.smithi118:> sudo dd if=/var/lib/ceph/osd/ceph-4/fsid of=/dev/stdout 2024-08-03T20:47:17.115 DEBUG:teuthology.orchestra.run.smithi118:> set -ex 2024-08-03T20:47:17.115 DEBUG:teuthology.orchestra.run.smithi118:> sudo dd if=/var/lib/ceph/osd/ceph-5/fsid of=/dev/stdout 2024-08-03T20:47:17.211 DEBUG:teuthology.orchestra.run.smithi118:> set -ex 2024-08-03T20:47:17.211 DEBUG:teuthology.orchestra.run.smithi118:> sudo dd if=/var/lib/ceph/osd/ceph-6/fsid of=/dev/stdout 2024-08-03T20:47:17.307 DEBUG:teuthology.orchestra.run.smithi118:> set -ex 2024-08-03T20:47:17.307 DEBUG:teuthology.orchestra.run.smithi118:> sudo dd if=/var/lib/ceph/osd/ceph-7/fsid of=/dev/stdout 2024-08-03T20:47:17.402 DEBUG:teuthology.orchestra.run.smithi118:> sudo ceph --cluster ceph osd new 5a83e759-9035-438b-889a-0a6af1f0a0d2 0 2024-08-03T20:47:17.661 INFO:teuthology.orchestra.run.smithi118.stderr:2024-08-03T20:47:17.660+0000 7efedda00640 1 -- 172.21.15.118:0/643970623 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7efed80b59f0 msgr2=0x7efed80d4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:47:17.661 INFO:teuthology.orchestra.run.smithi118.stderr:2024-08-03T20:47:17.660+0000 7efedda00640 1 --2- 172.21.15.118:0/643970623 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7efed80b59f0 0x7efed80d4cc0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7efed808d2b0 tx=0x7efec802a5f0 comp rx=0 tx=0).stop 2024-08-03T20:47:17.661 INFO:teuthology.orchestra.run.smithi118.stderr:2024-08-03T20:47:17.660+0000 7efedda00640 1 -- 172.21.15.118:0/643970623 shutdown_connections 2024-08-03T20:47:17.661 INFO:teuthology.orchestra.run.smithi118.stderr:2024-08-03T20:47:17.660+0000 7efedda00640 1 --2- 172.21.15.118:0/643970623 >> v2:172.21.15.32:3300/0 conn(0x7efed80d0fb0 0x7efed80d2230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:17.662 INFO:teuthology.orchestra.run.smithi118.stderr:2024-08-03T20:47:17.660+0000 7efedda00640 1 --2- 172.21.15.118:0/643970623 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7efed80cf9c0 0x7efed80d0a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:17.662 INFO:teuthology.orchestra.run.smithi118.stderr:2024-08-03T20:47:17.660+0000 7efedda00640 1 --2- 172.21.15.118:0/643970623 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7efed80b59f0 0x7efed80d4cc0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:17.662 INFO:teuthology.orchestra.run.smithi118.stderr:2024-08-03T20:47:17.660+0000 7efedda00640 1 -- 172.21.15.118:0/643970623 >> 172.21.15.118:0/643970623 conn(0x7efed809d660 msgr2=0x7efed80d4330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:47:17.662 INFO:teuthology.orchestra.run.smithi118.stderr:2024-08-03T20:47:17.660+0000 7efedda00640 1 -- 172.21.15.118:0/643970623 shutdown_connections 2024-08-03T20:47:17.662 INFO:teuthology.orchestra.run.smithi118.stderr:2024-08-03T20:47:17.660+0000 7efedda00640 1 -- 172.21.15.118:0/643970623 wait complete. 2024-08-03T20:47:17.663 INFO:teuthology.orchestra.run.smithi118.stderr:2024-08-03T20:47:17.662+0000 7efedda00640 1 Processor -- start 2024-08-03T20:47:17.663 INFO:teuthology.orchestra.run.smithi118.stderr:2024-08-03T20:47:17.662+0000 7efedda00640 1 -- start start 2024-08-03T20:47:17.666 INFO:teuthology.orchestra.run.smithi118.stderr:2024-08-03T20:47:17.664+0000 7efedda00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efed80b59f0 0x7efed815bae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:47:17.666 INFO:teuthology.orchestra.run.smithi118.stderr:2024-08-03T20:47:17.664+0000 7efedda00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7efed80cf9c0 0x7efed81f2710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:47:17.666 INFO:teuthology.orchestra.run.smithi118.stderr:2024-08-03T20:47:17.664+0000 7efedda00640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7efed80d0fb0 0x7efed81f8820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:47:17.666 INFO:teuthology.orchestra.run.smithi118.stderr:2024-08-03T20:47:17.664+0000 7efedda00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7efed80d2b30 con 0x7efed80b59f0 2024-08-03T20:47:17.666 INFO:teuthology.orchestra.run.smithi118.stderr:2024-08-03T20:47:17.664+0000 7efedda00640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7efed80d29b0 con 0x7efed80cf9c0 2024-08-03T20:47:17.666 INFO:teuthology.orchestra.run.smithi118.stderr:2024-08-03T20:47:17.664+0000 7efedda00640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7efed80d2cb0 con 0x7efed80d0fb0 2024-08-03T20:47:17.666 INFO:teuthology.orchestra.run.smithi118.stderr:2024-08-03T20:47:17.664+0000 7efecfe00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7efed80cf9c0 0x7efed81f2710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:47:17.666 INFO:teuthology.orchestra.run.smithi118.stderr:2024-08-03T20:47:17.664+0000 7efecfe00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7efed80cf9c0 0x7efed81f2710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.118:45586/0 (socket says 172.21.15.118:45586) 2024-08-03T20:47:17.666 INFO:teuthology.orchestra.run.smithi118.stderr:2024-08-03T20:47:17.664+0000 7efecfe00640 1 -- 172.21.15.118:0/1689670242 learned_addr learned my addr 172.21.15.118:0/1689670242 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-08-03T20:47:17.666 INFO:teuthology.orchestra.run.smithi118.stderr:2024-08-03T20:47:17.664+0000 7efed6a00640 1 --2- 172.21.15.118:0/1689670242 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efed80b59f0 0x7efed815bae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:47:17.666 INFO:teuthology.orchestra.run.smithi118.stderr:2024-08-03T20:47:17.664+0000 7efed7400640 1 --2- 172.21.15.118:0/1689670242 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7efed80d0fb0 0x7efed81f8820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:47:17.666 INFO:teuthology.orchestra.run.smithi118.stderr:2024-08-03T20:47:17.665+0000 7efecfe00640 1 -- 172.21.15.118:0/1689670242 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7efed80d0fb0 msgr2=0x7efed81f8820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:47:17.667 INFO:teuthology.orchestra.run.smithi118.stderr:2024-08-03T20:47:17.665+0000 7efecfe00640 1 --2- 172.21.15.118:0/1689670242 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7efed80d0fb0 0x7efed81f8820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:17.667 INFO:teuthology.orchestra.run.smithi118.stderr:2024-08-03T20:47:17.665+0000 7efecfe00640 1 -- 172.21.15.118:0/1689670242 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efed80b59f0 msgr2=0x7efed815bae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:47:17.667 INFO:teuthology.orchestra.run.smithi118.stderr:2024-08-03T20:47:17.665+0000 7efecfe00640 1 --2- 172.21.15.118:0/1689670242 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efed80b59f0 0x7efed815bae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:17.667 INFO:teuthology.orchestra.run.smithi118.stderr:2024-08-03T20:47:17.665+0000 7efecfe00640 1 -- 172.21.15.118:0/1689670242 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efed815c8e0 con 0x7efed80cf9c0 2024-08-03T20:47:17.667 INFO:teuthology.orchestra.run.smithi118.stderr:2024-08-03T20:47:17.665+0000 7efed6a00640 1 --2- 172.21.15.118:0/1689670242 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efed80b59f0 0x7efed815bae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-03T20:47:17.667 INFO:teuthology.orchestra.run.smithi118.stderr:2024-08-03T20:47:17.665+0000 7efecfe00640 1 --2- 172.21.15.118:0/1689670242 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7efed80cf9c0 0x7efed81f2710 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7efec000e9f0 tx=0x7efec000eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:47:17.668 INFO:teuthology.orchestra.run.smithi118.stderr:2024-08-03T20:47:17.665+0000 7efecea00640 1 -- 172.21.15.118:0/1689670242 <== mon.1 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efec000cd80 con 0x7efed80cf9c0 2024-08-03T20:47:17.668 INFO:teuthology.orchestra.run.smithi118.stderr:2024-08-03T20:47:17.665+0000 7efecea00640 1 -- 172.21.15.118:0/1689670242 <== mon.1 v2:172.21.15.118:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7efec0004020 con 0x7efed80cf9c0 2024-08-03T20:47:17.668 INFO:teuthology.orchestra.run.smithi118.stderr:2024-08-03T20:47:17.665+0000 7efedda00640 1 -- 172.21.15.118:0/1689670242 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efed81f2c50 con 0x7efed80cf9c0 2024-08-03T20:47:17.668 INFO:teuthology.orchestra.run.smithi118.stderr:2024-08-03T20:47:17.665+0000 7efecea00640 1 -- 172.21.15.118:0/1689670242 <== mon.1 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efec0004470 con 0x7efed80cf9c0 2024-08-03T20:47:17.668 INFO:teuthology.orchestra.run.smithi118.stderr:2024-08-03T20:47:17.665+0000 7efedda00640 1 -- 172.21.15.118:0/1689670242 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efed815ce60 con 0x7efed80cf9c0 2024-08-03T20:47:17.668 INFO:teuthology.orchestra.run.smithi118.stderr:2024-08-03T20:47:17.666+0000 7efecea00640 1 -- 172.21.15.118:0/1689670242 <== mon.1 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7efec00053b0 con 0x7efed80cf9c0 2024-08-03T20:47:17.668 INFO:teuthology.orchestra.run.smithi118.stderr:2024-08-03T20:47:17.666+0000 7efecea00640 1 -- 172.21.15.118:0/1689670242 <== mon.1 v2:172.21.15.118:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 725+0+0 (secure 0 0 0) 0x7efec0003af0 con 0x7efed80cf9c0 2024-08-03T20:47:17.668 INFO:teuthology.orchestra.run.smithi118.stderr:2024-08-03T20:47:17.666+0000 7efedda00640 1 -- 172.21.15.118:0/1689670242 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efed80d0a70 con 0x7efed80cf9c0 2024-08-03T20:47:17.670 INFO:teuthology.orchestra.run.smithi118.stderr:2024-08-03T20:47:17.669+0000 7efecea00640 1 -- 172.21.15.118:0/1689670242 <== mon.1 v2:172.21.15.118:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+74906 (secure 0 0 0) 0x7efec0003cf0 con 0x7efed80cf9c0 2024-08-03T20:47:17.744 INFO:teuthology.orchestra.run.smithi118.stderr:2024-08-03T20:47:17.743+0000 7efedda00640 1 -- 172.21.15.118:0/1689670242 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command({"prefix": "osd new", "uuid": "5a83e759-9035-438b-889a-0a6af1f0a0d2", "id": 0} v 0) -- 0x7efed806f260 con 0x7efed80cf9c0 2024-08-03T20:47:17.749 INFO:teuthology.orchestra.run.smithi118.stderr:2024-08-03T20:47:17.748+0000 7efecea00640 1 -- 172.21.15.118:0/1689670242 <== mon.1 v2:172.21.15.118:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "5a83e759-9035-438b-889a-0a6af1f0a0d2", "id": 0}]=0 v3) ==== 112+0+2 (secure 0 0 0) 0x7efec00047c0 con 0x7efed80cf9c0 2024-08-03T20:47:17.750 INFO:teuthology.orchestra.run.smithi118.stdout:0 2024-08-03T20:47:17.751 INFO:teuthology.orchestra.run.smithi118.stderr:2024-08-03T20:47:17.749+0000 7efedda00640 1 -- 172.21.15.118:0/1689670242 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7efed80cf9c0 msgr2=0x7efed81f2710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:47:17.751 INFO:teuthology.orchestra.run.smithi118.stderr:2024-08-03T20:47:17.749+0000 7efedda00640 1 --2- 172.21.15.118:0/1689670242 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7efed80cf9c0 0x7efed81f2710 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7efec000e9f0 tx=0x7efec000eeb0 comp rx=0 tx=0).stop 2024-08-03T20:47:17.751 INFO:teuthology.orchestra.run.smithi118.stderr:2024-08-03T20:47:17.750+0000 7efedda00640 1 -- 172.21.15.118:0/1689670242 shutdown_connections 2024-08-03T20:47:17.751 INFO:teuthology.orchestra.run.smithi118.stderr:2024-08-03T20:47:17.750+0000 7efedda00640 1 --2- 172.21.15.118:0/1689670242 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7efed80cf9c0 0x7efed81f2710 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:17.751 INFO:teuthology.orchestra.run.smithi118.stderr:2024-08-03T20:47:17.750+0000 7efedda00640 1 --2- 172.21.15.118:0/1689670242 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7efed80d0fb0 0x7efed81f8820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:17.751 INFO:teuthology.orchestra.run.smithi118.stderr:2024-08-03T20:47:17.750+0000 7efedda00640 1 --2- 172.21.15.118:0/1689670242 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efed80b59f0 0x7efed815bae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:17.751 INFO:teuthology.orchestra.run.smithi118.stderr:2024-08-03T20:47:17.750+0000 7efedda00640 1 -- 172.21.15.118:0/1689670242 >> 172.21.15.118:0/1689670242 conn(0x7efed809d660 msgr2=0x7efed815fcc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:47:17.751 INFO:teuthology.orchestra.run.smithi118.stderr:2024-08-03T20:47:17.750+0000 7efedda00640 1 -- 172.21.15.118:0/1689670242 shutdown_connections 2024-08-03T20:47:17.752 INFO:teuthology.orchestra.run.smithi118.stderr:2024-08-03T20:47:17.750+0000 7efedda00640 1 -- 172.21.15.118:0/1689670242 wait complete. 2024-08-03T20:47:17.771 DEBUG:teuthology.orchestra.run.smithi118:> sudo ceph --cluster ceph osd new d7c5c210-da4f-49f9-b9e7-1dfb02daa4bb 1 2024-08-03T20:47:17.992 INFO:teuthology.orchestra.run.smithi118.stderr:2024-08-03T20:47:17.986+0000 7f9fb4e00640 1 -- 172.21.15.118:0/1016636976 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f9fb00b9300 msgr2=0x7f9fb00d1f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:47:17.992 INFO:teuthology.orchestra.run.smithi118.stderr:2024-08-03T20:47:17.986+0000 7f9fb4e00640 1 --2- 172.21.15.118:0/1016636976 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f9fb00b9300 0x7f9fb00d1f10 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f9f98005fd0 tx=0x7f9f9802a540 comp rx=0 tx=0).stop 2024-08-03T20:47:17.992 INFO:teuthology.orchestra.run.smithi118.stderr:2024-08-03T20:47:17.986+0000 7f9fb4e00640 1 -- 172.21.15.118:0/1016636976 shutdown_connections 2024-08-03T20:47:17.993 INFO:teuthology.orchestra.run.smithi118.stderr:2024-08-03T20:47:17.986+0000 7f9fb4e00640 1 --2- 172.21.15.118:0/1016636976 >> v2:172.21.15.32:3300/0 conn(0x7f9fb00d2450 0x7f9fb00cf6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:17.993 INFO:teuthology.orchestra.run.smithi118.stderr:2024-08-03T20:47:17.986+0000 7f9fb4e00640 1 --2- 172.21.15.118:0/1016636976 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f9fb00b9300 0x7f9fb00d1f10 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:17.993 INFO:teuthology.orchestra.run.smithi118.stderr:2024-08-03T20:47:17.986+0000 7f9fb4e00640 1 -- 172.21.15.118:0/1016636976 >> 172.21.15.118:0/1016636976 conn(0x7f9fb00d4160 msgr2=0x7f9fb00d4570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:47:17.993 INFO:teuthology.orchestra.run.smithi118.stderr:2024-08-03T20:47:17.986+0000 7f9fb4e00640 1 -- 172.21.15.118:0/1016636976 shutdown_connections 2024-08-03T20:47:17.993 INFO:teuthology.orchestra.run.smithi118.stderr:2024-08-03T20:47:17.986+0000 7f9fb4e00640 1 -- 172.21.15.118:0/1016636976 wait complete. 2024-08-03T20:47:17.993 INFO:teuthology.orchestra.run.smithi118.stderr:2024-08-03T20:47:17.988+0000 7f9fb4e00640 1 Processor -- start 2024-08-03T20:47:17.993 INFO:teuthology.orchestra.run.smithi118.stderr:2024-08-03T20:47:17.988+0000 7f9fb4e00640 1 -- start start 2024-08-03T20:47:17.993 INFO:teuthology.orchestra.run.smithi118.stderr:2024-08-03T20:47:17.990+0000 7f9fb4e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9fb00b9300 0x7f9fb0162910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:47:17.993 INFO:teuthology.orchestra.run.smithi118.stderr:2024-08-03T20:47:17.990+0000 7f9fb4e00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f9fb00d2450 0x7f9fb0150400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:47:17.993 INFO:teuthology.orchestra.run.smithi118.stderr:2024-08-03T20:47:17.990+0000 7f9fb4e00640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f9fb0168120 0x7f9fb009df30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:47:17.993 INFO:teuthology.orchestra.run.smithi118.stderr:2024-08-03T20:47:17.990+0000 7f9fb4e00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f9fb00cffb0 con 0x7f9fb00b9300 2024-08-03T20:47:17.994 INFO:teuthology.orchestra.run.smithi118.stderr:2024-08-03T20:47:17.990+0000 7f9fb4e00640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f9fb00cfe30 con 0x7f9fb00d2450 2024-08-03T20:47:17.994 INFO:teuthology.orchestra.run.smithi118.stderr:2024-08-03T20:47:17.990+0000 7f9fb4e00640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7f9fb00d4e70 con 0x7f9fb0168120 2024-08-03T20:47:17.994 INFO:teuthology.orchestra.run.smithi118.stderr:2024-08-03T20:47:17.990+0000 7f9fafe00640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f9fb0168120 0x7f9fb009df30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:47:17.994 INFO:teuthology.orchestra.run.smithi118.stderr:2024-08-03T20:47:17.990+0000 7f9faf400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9fb00b9300 0x7f9fb0162910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:47:17.994 INFO:teuthology.orchestra.run.smithi118.stderr:2024-08-03T20:47:17.990+0000 7f9faf400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9fb00b9300 0x7f9fb0162910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.118:53306/0 (socket says 172.21.15.118:53306) 2024-08-03T20:47:17.994 INFO:teuthology.orchestra.run.smithi118.stderr:2024-08-03T20:47:17.990+0000 7f9fafe00640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f9fb0168120 0x7f9fb009df30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3301/0 says I am v2:172.21.15.118:47934/0 (socket says 172.21.15.118:47934) 2024-08-03T20:47:17.994 INFO:teuthology.orchestra.run.smithi118.stderr:2024-08-03T20:47:17.990+0000 7f9faf400640 1 -- 172.21.15.118:0/2780247487 learned_addr learned my addr 172.21.15.118:0/2780247487 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-08-03T20:47:17.994 INFO:teuthology.orchestra.run.smithi118.stderr:2024-08-03T20:47:17.990+0000 7f9fa7e00640 1 --2- 172.21.15.118:0/2780247487 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f9fb00d2450 0x7f9fb0150400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:47:17.994 INFO:teuthology.orchestra.run.smithi118.stderr:2024-08-03T20:47:17.991+0000 7f9faf400640 1 -- 172.21.15.118:0/2780247487 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f9fb0168120 msgr2=0x7f9fb009df30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:47:17.994 INFO:teuthology.orchestra.run.smithi118.stderr:2024-08-03T20:47:17.991+0000 7f9faf400640 1 --2- 172.21.15.118:0/2780247487 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f9fb0168120 0x7f9fb009df30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:17.994 INFO:teuthology.orchestra.run.smithi118.stderr:2024-08-03T20:47:17.991+0000 7f9faf400640 1 -- 172.21.15.118:0/2780247487 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f9fb00d2450 msgr2=0x7f9fb0150400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:47:17.995 INFO:teuthology.orchestra.run.smithi118.stderr:2024-08-03T20:47:17.991+0000 7f9faf400640 1 --2- 172.21.15.118:0/2780247487 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f9fb00d2450 0x7f9fb0150400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:17.995 INFO:teuthology.orchestra.run.smithi118.stderr:2024-08-03T20:47:17.991+0000 7f9faf400640 1 -- 172.21.15.118:0/2780247487 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9fb0161b50 con 0x7f9fb00b9300 2024-08-03T20:47:17.995 INFO:teuthology.orchestra.run.smithi118.stderr:2024-08-03T20:47:17.991+0000 7f9faf400640 1 --2- 172.21.15.118:0/2780247487 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9fb00b9300 0x7f9fb0162910 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f9fa000c3a0 tx=0x7f9fa000c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:47:17.995 INFO:teuthology.orchestra.run.smithi118.stderr:2024-08-03T20:47:17.991+0000 7f9fa7400640 1 -- 172.21.15.118:0/2780247487 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9fa0017070 con 0x7f9fb00b9300 2024-08-03T20:47:17.995 INFO:teuthology.orchestra.run.smithi118.stderr:2024-08-03T20:47:17.991+0000 7f9fa7400640 1 -- 172.21.15.118:0/2780247487 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f9fa00053c0 con 0x7f9fb00b9300 2024-08-03T20:47:17.995 INFO:teuthology.orchestra.run.smithi118.stderr:2024-08-03T20:47:17.991+0000 7f9fa7400640 1 -- 172.21.15.118:0/2780247487 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9fa0003850 con 0x7f9fb00b9300 2024-08-03T20:47:17.995 INFO:teuthology.orchestra.run.smithi118.stderr:2024-08-03T20:47:17.991+0000 7f9fb4e00640 1 -- 172.21.15.118:0/2780247487 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9fb0161e30 con 0x7f9fb00b9300 2024-08-03T20:47:17.996 INFO:teuthology.orchestra.run.smithi118.stderr:2024-08-03T20:47:17.992+0000 7f9fa7400640 1 -- 172.21.15.118:0/2780247487 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f9fa001a070 con 0x7f9fb00b9300 2024-08-03T20:47:17.996 INFO:teuthology.orchestra.run.smithi118.stderr:2024-08-03T20:47:17.992+0000 7f9fb4e00640 1 -- 172.21.15.118:0/2780247487 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9fb009ea70 con 0x7f9fb00b9300 2024-08-03T20:47:17.996 INFO:teuthology.orchestra.run.smithi118.stderr:2024-08-03T20:47:17.992+0000 7f9fa7400640 1 -- 172.21.15.118:0/2780247487 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 840+0+0 (secure 0 0 0) 0x7f9fa0011d70 con 0x7f9fb00b9300 2024-08-03T20:47:17.996 INFO:teuthology.orchestra.run.smithi118.stderr:2024-08-03T20:47:17.993+0000 7f9fb4e00640 1 -- 172.21.15.118:0/2780247487 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9fb00d4aa0 con 0x7f9fb00b9300 2024-08-03T20:47:17.996 INFO:teuthology.orchestra.run.smithi118.stderr:2024-08-03T20:47:17.995+0000 7f9fa7400640 1 -- 172.21.15.118:0/2780247487 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+74906 (secure 0 0 0) 0x7f9fb00d4aa0 con 0x7f9fb00b9300 2024-08-03T20:47:18.073 INFO:teuthology.orchestra.run.smithi118.stderr:2024-08-03T20:47:18.072+0000 7f9fb4e00640 1 -- 172.21.15.118:0/2780247487 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd new", "uuid": "d7c5c210-da4f-49f9-b9e7-1dfb02daa4bb", "id": 1} v 0) -- 0x7f9fb00d18f0 con 0x7f9fb00b9300 2024-08-03T20:47:18.074 INFO:teuthology.orchestra.run.smithi118.stderr:2024-08-03T20:47:18.074+0000 7f9fa7400640 1 -- 172.21.15.118:0/2780247487 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "d7c5c210-da4f-49f9-b9e7-1dfb02daa4bb", "id": 1}]=0 v4) ==== 112+0+2 (secure 0 0 0) 0x7f9fa0022070 con 0x7f9fb00b9300 2024-08-03T20:47:18.075 INFO:teuthology.orchestra.run.smithi118.stdout:1 2024-08-03T20:47:18.076 INFO:teuthology.orchestra.run.smithi118.stderr:2024-08-03T20:47:18.075+0000 7f9fb4e00640 1 -- 172.21.15.118:0/2780247487 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9fb00b9300 msgr2=0x7f9fb0162910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:47:18.076 INFO:teuthology.orchestra.run.smithi118.stderr:2024-08-03T20:47:18.075+0000 7f9fb4e00640 1 --2- 172.21.15.118:0/2780247487 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9fb00b9300 0x7f9fb0162910 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f9fa000c3a0 tx=0x7f9fa000c860 comp rx=0 tx=0).stop 2024-08-03T20:47:18.076 INFO:teuthology.orchestra.run.smithi118.stderr:2024-08-03T20:47:18.075+0000 7f9fb4e00640 1 -- 172.21.15.118:0/2780247487 shutdown_connections 2024-08-03T20:47:18.076 INFO:teuthology.orchestra.run.smithi118.stderr:2024-08-03T20:47:18.075+0000 7f9fb4e00640 1 --2- 172.21.15.118:0/2780247487 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f9fb00d2450 0x7f9fb0150400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:18.076 INFO:teuthology.orchestra.run.smithi118.stderr:2024-08-03T20:47:18.075+0000 7f9fb4e00640 1 --2- 172.21.15.118:0/2780247487 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f9fb0168120 0x7f9fb009df30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:18.076 INFO:teuthology.orchestra.run.smithi118.stderr:2024-08-03T20:47:18.075+0000 7f9fb4e00640 1 --2- 172.21.15.118:0/2780247487 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9fb00b9300 0x7f9fb0162910 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:18.076 INFO:teuthology.orchestra.run.smithi118.stderr:2024-08-03T20:47:18.075+0000 7f9fb4e00640 1 -- 172.21.15.118:0/2780247487 >> 172.21.15.118:0/2780247487 conn(0x7f9fb00d4160 msgr2=0x7f9fb00b8990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:47:18.077 INFO:teuthology.orchestra.run.smithi118.stderr:2024-08-03T20:47:18.075+0000 7f9fb4e00640 1 -- 172.21.15.118:0/2780247487 shutdown_connections 2024-08-03T20:47:18.077 INFO:teuthology.orchestra.run.smithi118.stderr:2024-08-03T20:47:18.075+0000 7f9fb4e00640 1 -- 172.21.15.118:0/2780247487 wait complete. 2024-08-03T20:47:18.098 DEBUG:teuthology.orchestra.run.smithi118:> sudo ceph --cluster ceph osd new 57f2e6de-a201-45c2-8981-8e56a4875ebd 2 2024-08-03T20:47:18.319 INFO:teuthology.orchestra.run.smithi118.stderr:2024-08-03T20:47:18.318+0000 7f9477600640 1 -- 172.21.15.118:0/1548983438 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f94700cf3b0 msgr2=0x7f94700cf7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:47:18.319 INFO:teuthology.orchestra.run.smithi118.stderr:2024-08-03T20:47:18.318+0000 7f9477600640 1 --2- 172.21.15.118:0/1548983438 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f94700cf3b0 0x7f94700cf7c0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f947008e320 tx=0x7f946802a8b0 comp rx=0 tx=0).stop 2024-08-03T20:47:18.319 INFO:teuthology.orchestra.run.smithi118.stderr:2024-08-03T20:47:18.318+0000 7f9477600640 1 -- 172.21.15.118:0/1548983438 shutdown_connections 2024-08-03T20:47:18.319 INFO:teuthology.orchestra.run.smithi118.stderr:2024-08-03T20:47:18.318+0000 7f9477600640 1 --2- 172.21.15.118:0/1548983438 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f94700cf3b0 0x7f94700cf7c0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:18.320 INFO:teuthology.orchestra.run.smithi118.stderr:2024-08-03T20:47:18.318+0000 7f9477600640 1 --2- 172.21.15.118:0/1548983438 >> v2:172.21.15.32:3300/0 conn(0x7f94700d4e00 0x7f94700b2cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:18.320 INFO:teuthology.orchestra.run.smithi118.stderr:2024-08-03T20:47:18.318+0000 7f9477600640 1 -- 172.21.15.118:0/1548983438 >> 172.21.15.118:0/1548983438 conn(0x7f94700d42e0 msgr2=0x7f94700d46f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:47:18.320 INFO:teuthology.orchestra.run.smithi118.stderr:2024-08-03T20:47:18.318+0000 7f9477600640 1 -- 172.21.15.118:0/1548983438 shutdown_connections 2024-08-03T20:47:18.320 INFO:teuthology.orchestra.run.smithi118.stderr:2024-08-03T20:47:18.318+0000 7f9477600640 1 -- 172.21.15.118:0/1548983438 wait complete. 2024-08-03T20:47:18.321 INFO:teuthology.orchestra.run.smithi118.stderr:2024-08-03T20:47:18.320+0000 7f9477600640 1 Processor -- start 2024-08-03T20:47:18.321 INFO:teuthology.orchestra.run.smithi118.stderr:2024-08-03T20:47:18.320+0000 7f9477600640 1 -- start start 2024-08-03T20:47:18.323 INFO:teuthology.orchestra.run.smithi118.stderr:2024-08-03T20:47:18.322+0000 7f9477600640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f94700cf3b0 0x7f947015f8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:47:18.323 INFO:teuthology.orchestra.run.smithi118.stderr:2024-08-03T20:47:18.322+0000 7f9477600640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f94700d1eb0 0x7f947015b8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:47:18.323 INFO:teuthology.orchestra.run.smithi118.stderr:2024-08-03T20:47:18.322+0000 7f9477600640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f94700d4e00 0x7f947015bde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:47:18.323 INFO:teuthology.orchestra.run.smithi118.stderr:2024-08-03T20:47:18.322+0000 7f9477600640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f94700cfd00 con 0x7f94700cf3b0 2024-08-03T20:47:18.323 INFO:teuthology.orchestra.run.smithi118.stderr:2024-08-03T20:47:18.322+0000 7f9477600640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f94700cfe80 con 0x7f94700d1eb0 2024-08-03T20:47:18.323 INFO:teuthology.orchestra.run.smithi118.stderr:2024-08-03T20:47:18.322+0000 7f9477600640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7f94700d0000 con 0x7f94700d4e00 2024-08-03T20:47:18.323 INFO:teuthology.orchestra.run.smithi118.stderr:2024-08-03T20:47:18.322+0000 7f9467e00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f94700d1eb0 0x7f947015b8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:47:18.324 INFO:teuthology.orchestra.run.smithi118.stderr:2024-08-03T20:47:18.322+0000 7f9474c00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f94700cf3b0 0x7f947015f8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:47:18.324 INFO:teuthology.orchestra.run.smithi118.stderr:2024-08-03T20:47:18.322+0000 7f9475600640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f94700d4e00 0x7f947015bde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:47:18.324 INFO:teuthology.orchestra.run.smithi118.stderr:2024-08-03T20:47:18.322+0000 7f9467e00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f94700d1eb0 0x7f947015b8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.118:45606/0 (socket says 172.21.15.118:45606) 2024-08-03T20:47:18.324 INFO:teuthology.orchestra.run.smithi118.stderr:2024-08-03T20:47:18.322+0000 7f9467e00640 1 -- 172.21.15.118:0/3173861245 learned_addr learned my addr 172.21.15.118:0/3173861245 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-08-03T20:47:18.324 INFO:teuthology.orchestra.run.smithi118.stderr:2024-08-03T20:47:18.322+0000 7f9474c00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f94700cf3b0 0x7f947015f8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.118:53324/0 (socket says 172.21.15.118:53324) 2024-08-03T20:47:18.325 INFO:teuthology.orchestra.run.smithi118.stderr:2024-08-03T20:47:18.322+0000 7f9467e00640 1 -- 172.21.15.118:0/3173861245 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f94700d4e00 msgr2=0x7f947015bde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:47:18.325 INFO:teuthology.orchestra.run.smithi118.stderr:2024-08-03T20:47:18.322+0000 7f9467e00640 1 --2- 172.21.15.118:0/3173861245 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f94700d4e00 0x7f947015bde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:18.325 INFO:teuthology.orchestra.run.smithi118.stderr:2024-08-03T20:47:18.322+0000 7f9467e00640 1 -- 172.21.15.118:0/3173861245 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f94700cf3b0 msgr2=0x7f947015f8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:47:18.326 INFO:teuthology.orchestra.run.smithi118.stderr:2024-08-03T20:47:18.322+0000 7f9467e00640 1 --2- 172.21.15.118:0/3173861245 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f94700cf3b0 0x7f947015f8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:18.326 INFO:teuthology.orchestra.run.smithi118.stderr:2024-08-03T20:47:18.322+0000 7f9467e00640 1 -- 172.21.15.118:0/3173861245 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f947015a800 con 0x7f94700d1eb0 2024-08-03T20:47:18.326 INFO:teuthology.orchestra.run.smithi118.stderr:2024-08-03T20:47:18.322+0000 7f9474c00640 1 --2- 172.21.15.118:0/3173861245 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f94700cf3b0 0x7f947015f8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-03T20:47:18.326 INFO:teuthology.orchestra.run.smithi118.stderr:2024-08-03T20:47:18.323+0000 7f9467e00640 1 --2- 172.21.15.118:0/3173861245 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f94700d1eb0 0x7f947015b8a0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f945800c3d0 tx=0x7f945800c890 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:47:18.326 INFO:teuthology.orchestra.run.smithi118.stderr:2024-08-03T20:47:18.323+0000 7f946d600640 1 -- 172.21.15.118:0/3173861245 <== mon.1 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9458011790 con 0x7f94700d1eb0 2024-08-03T20:47:18.326 INFO:teuthology.orchestra.run.smithi118.stderr:2024-08-03T20:47:18.323+0000 7f946d600640 1 -- 172.21.15.118:0/3173861245 <== mon.1 v2:172.21.15.118:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f9458017070 con 0x7f94700d1eb0 2024-08-03T20:47:18.326 INFO:teuthology.orchestra.run.smithi118.stderr:2024-08-03T20:47:18.323+0000 7f9477600640 1 -- 172.21.15.118:0/3173861245 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f947015aae0 con 0x7f94700d1eb0 2024-08-03T20:47:18.326 INFO:teuthology.orchestra.run.smithi118.stderr:2024-08-03T20:47:18.323+0000 7f946d600640 1 -- 172.21.15.118:0/3173861245 <== mon.1 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9458011cc0 con 0x7f94700d1eb0 2024-08-03T20:47:18.326 INFO:teuthology.orchestra.run.smithi118.stderr:2024-08-03T20:47:18.323+0000 7f9477600640 1 -- 172.21.15.118:0/3173861245 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f947015b0a0 con 0x7f94700d1eb0 2024-08-03T20:47:18.326 INFO:teuthology.orchestra.run.smithi118.stderr:2024-08-03T20:47:18.323+0000 7f946d600640 1 -- 172.21.15.118:0/3173861245 <== mon.1 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f945801c460 con 0x7f94700d1eb0 2024-08-03T20:47:18.326 INFO:teuthology.orchestra.run.smithi118.stderr:2024-08-03T20:47:18.323+0000 7f946d600640 1 -- 172.21.15.118:0/3173861245 <== mon.1 v2:172.21.15.118:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 955+0+0 (secure 0 0 0) 0x7f9458012b10 con 0x7f94700d1eb0 2024-08-03T20:47:18.326 INFO:teuthology.orchestra.run.smithi118.stderr:2024-08-03T20:47:18.324+0000 7f9477600640 1 -- 172.21.15.118:0/3173861245 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f947015ffa0 con 0x7f94700d1eb0 2024-08-03T20:47:18.327 INFO:teuthology.orchestra.run.smithi118.stderr:2024-08-03T20:47:18.326+0000 7f946d600640 1 -- 172.21.15.118:0/3173861245 <== mon.1 v2:172.21.15.118:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+74906 (secure 0 0 0) 0x7f9458012de0 con 0x7f94700d1eb0 2024-08-03T20:47:18.404 INFO:teuthology.orchestra.run.smithi118.stderr:2024-08-03T20:47:18.403+0000 7f9477600640 1 -- 172.21.15.118:0/3173861245 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command({"prefix": "osd new", "uuid": "57f2e6de-a201-45c2-8981-8e56a4875ebd", "id": 2} v 0) -- 0x7f94700cf7c0 con 0x7f94700d1eb0 2024-08-03T20:47:18.409 INFO:teuthology.orchestra.run.smithi118.stderr:2024-08-03T20:47:18.408+0000 7f946d600640 1 -- 172.21.15.118:0/3173861245 <== mon.1 v2:172.21.15.118:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "57f2e6de-a201-45c2-8981-8e56a4875ebd", "id": 2}]=0 v5) ==== 112+0+2 (secure 0 0 0) 0x7f94580126f0 con 0x7f94700d1eb0 2024-08-03T20:47:18.409 INFO:teuthology.orchestra.run.smithi118.stdout:2 2024-08-03T20:47:18.410 INFO:teuthology.orchestra.run.smithi118.stderr:2024-08-03T20:47:18.409+0000 7f9477600640 1 -- 172.21.15.118:0/3173861245 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f94700d1eb0 msgr2=0x7f947015b8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:47:18.410 INFO:teuthology.orchestra.run.smithi118.stderr:2024-08-03T20:47:18.409+0000 7f9477600640 1 --2- 172.21.15.118:0/3173861245 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f94700d1eb0 0x7f947015b8a0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f945800c3d0 tx=0x7f945800c890 comp rx=0 tx=0).stop 2024-08-03T20:47:18.410 INFO:teuthology.orchestra.run.smithi118.stderr:2024-08-03T20:47:18.409+0000 7f9477600640 1 -- 172.21.15.118:0/3173861245 shutdown_connections 2024-08-03T20:47:18.410 INFO:teuthology.orchestra.run.smithi118.stderr:2024-08-03T20:47:18.409+0000 7f9477600640 1 --2- 172.21.15.118:0/3173861245 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f94700d1eb0 0x7f947015b8a0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:18.410 INFO:teuthology.orchestra.run.smithi118.stderr:2024-08-03T20:47:18.409+0000 7f9477600640 1 --2- 172.21.15.118:0/3173861245 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f94700d4e00 0x7f947015bde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:18.411 INFO:teuthology.orchestra.run.smithi118.stderr:2024-08-03T20:47:18.409+0000 7f9477600640 1 --2- 172.21.15.118:0/3173861245 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f94700cf3b0 0x7f947015f8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:18.411 INFO:teuthology.orchestra.run.smithi118.stderr:2024-08-03T20:47:18.409+0000 7f9477600640 1 -- 172.21.15.118:0/3173861245 >> 172.21.15.118:0/3173861245 conn(0x7f94700d42e0 msgr2=0x7f94700b9120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:47:18.411 INFO:teuthology.orchestra.run.smithi118.stderr:2024-08-03T20:47:18.409+0000 7f9477600640 1 -- 172.21.15.118:0/3173861245 shutdown_connections 2024-08-03T20:47:18.411 INFO:teuthology.orchestra.run.smithi118.stderr:2024-08-03T20:47:18.409+0000 7f9477600640 1 -- 172.21.15.118:0/3173861245 wait complete. 2024-08-03T20:47:18.432 DEBUG:teuthology.orchestra.run.smithi118:> sudo ceph --cluster ceph osd new 6439a280-da45-4148-8130-f16323935ff3 3 2024-08-03T20:47:18.650 INFO:teuthology.orchestra.run.smithi118.stderr:2024-08-03T20:47:18.648+0000 7f7497200640 1 -- 172.21.15.118:0/1962693310 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f749806f660 msgr2=0x7f749806fa70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:47:18.650 INFO:teuthology.orchestra.run.smithi118.stderr:2024-08-03T20:47:18.648+0000 7f7497200640 1 --2- 172.21.15.118:0/1962693310 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f749806f660 0x7f749806fa70 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f7490009ca0 tx=0x7f749002ab10 comp rx=0 tx=0).stop 2024-08-03T20:47:18.650 INFO:teuthology.orchestra.run.smithi118.stderr:2024-08-03T20:47:18.648+0000 7f7497200640 1 -- 172.21.15.118:0/1962693310 shutdown_connections 2024-08-03T20:47:18.650 INFO:teuthology.orchestra.run.smithi118.stderr:2024-08-03T20:47:18.648+0000 7f7497200640 1 --2- 172.21.15.118:0/1962693310 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f749806f660 0x7f749806fa70 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:18.650 INFO:teuthology.orchestra.run.smithi118.stderr:2024-08-03T20:47:18.648+0000 7f7497200640 1 -- 172.21.15.118:0/1962693310 >> 172.21.15.118:0/1962693310 conn(0x7f74980d0dd0 msgr2=0x7f74980d11e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:47:18.650 INFO:teuthology.orchestra.run.smithi118.stderr:2024-08-03T20:47:18.648+0000 7f7497200640 1 -- 172.21.15.118:0/1962693310 shutdown_connections 2024-08-03T20:47:18.650 INFO:teuthology.orchestra.run.smithi118.stderr:2024-08-03T20:47:18.648+0000 7f7497200640 1 -- 172.21.15.118:0/1962693310 wait complete. 2024-08-03T20:47:18.651 INFO:teuthology.orchestra.run.smithi118.stderr:2024-08-03T20:47:18.650+0000 7f7497200640 1 Processor -- start 2024-08-03T20:47:18.651 INFO:teuthology.orchestra.run.smithi118.stderr:2024-08-03T20:47:18.651+0000 7f7497200640 1 -- start start 2024-08-03T20:47:18.653 INFO:teuthology.orchestra.run.smithi118.stderr:2024-08-03T20:47:18.652+0000 7f7497200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f749806c9e0 0x7f74981f2da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:47:18.653 INFO:teuthology.orchestra.run.smithi118.stderr:2024-08-03T20:47:18.652+0000 7f7497200640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f749806f660 0x7f74980b66c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:47:18.653 INFO:teuthology.orchestra.run.smithi118.stderr:2024-08-03T20:47:18.652+0000 7f7497200640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f74980b0170 0x7f74981f8de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:47:18.654 INFO:teuthology.orchestra.run.smithi118.stderr:2024-08-03T20:47:18.652+0000 7f7497200640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f749815f480 con 0x7f749806c9e0 2024-08-03T20:47:18.654 INFO:teuthology.orchestra.run.smithi118.stderr:2024-08-03T20:47:18.652+0000 7f7497200640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f749815f300 con 0x7f74980b0170 2024-08-03T20:47:18.654 INFO:teuthology.orchestra.run.smithi118.stderr:2024-08-03T20:47:18.652+0000 7f7497200640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7f749815f600 con 0x7f749806f660 2024-08-03T20:47:18.654 INFO:teuthology.orchestra.run.smithi118.stderr:2024-08-03T20:47:18.652+0000 7f7495e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f749806c9e0 0x7f74981f2da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:47:18.654 INFO:teuthology.orchestra.run.smithi118.stderr:2024-08-03T20:47:18.652+0000 7f7495400640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f749806f660 0x7f74980b66c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:47:18.654 INFO:teuthology.orchestra.run.smithi118.stderr:2024-08-03T20:47:18.652+0000 7f7496800640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f74980b0170 0x7f74981f8de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:47:18.654 INFO:teuthology.orchestra.run.smithi118.stderr:2024-08-03T20:47:18.652+0000 7f7495e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f749806c9e0 0x7f74981f2da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.118:53328/0 (socket says 172.21.15.118:53328) 2024-08-03T20:47:18.654 INFO:teuthology.orchestra.run.smithi118.stderr:2024-08-03T20:47:18.652+0000 7f7495e00640 1 -- 172.21.15.118:0/727898819 learned_addr learned my addr 172.21.15.118:0/727898819 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-08-03T20:47:18.656 INFO:teuthology.orchestra.run.smithi118.stderr:2024-08-03T20:47:18.653+0000 7f7495e00640 1 -- 172.21.15.118:0/727898819 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f749806f660 msgr2=0x7f74980b66c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:47:18.656 INFO:teuthology.orchestra.run.smithi118.stderr:2024-08-03T20:47:18.653+0000 7f7495e00640 1 --2- 172.21.15.118:0/727898819 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f749806f660 0x7f74980b66c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:18.656 INFO:teuthology.orchestra.run.smithi118.stderr:2024-08-03T20:47:18.653+0000 7f7495e00640 1 -- 172.21.15.118:0/727898819 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f74980b0170 msgr2=0x7f74981f8de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:47:18.656 INFO:teuthology.orchestra.run.smithi118.stderr:2024-08-03T20:47:18.653+0000 7f7495e00640 1 --2- 172.21.15.118:0/727898819 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f74980b0170 0x7f74981f8de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:18.656 INFO:teuthology.orchestra.run.smithi118.stderr:2024-08-03T20:47:18.653+0000 7f7495e00640 1 -- 172.21.15.118:0/727898819 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f74980c6e20 con 0x7f749806c9e0 2024-08-03T20:47:18.656 INFO:teuthology.orchestra.run.smithi118.stderr:2024-08-03T20:47:18.653+0000 7f7495e00640 1 --2- 172.21.15.118:0/727898819 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f749806c9e0 0x7f74981f2da0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f748800c3d0 tx=0x7f748800c890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:47:18.656 INFO:teuthology.orchestra.run.smithi118.stderr:2024-08-03T20:47:18.653+0000 7f748e000640 1 -- 172.21.15.118:0/727898819 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74880052d0 con 0x7f749806c9e0 2024-08-03T20:47:18.656 INFO:teuthology.orchestra.run.smithi118.stderr:2024-08-03T20:47:18.653+0000 7f748e000640 1 -- 172.21.15.118:0/727898819 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f7488007020 con 0x7f749806c9e0 2024-08-03T20:47:18.656 INFO:teuthology.orchestra.run.smithi118.stderr:2024-08-03T20:47:18.653+0000 7f7497200640 1 -- 172.21.15.118:0/727898819 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f74980c7100 con 0x7f749806c9e0 2024-08-03T20:47:18.656 INFO:teuthology.orchestra.run.smithi118.stderr:2024-08-03T20:47:18.653+0000 7f748e000640 1 -- 172.21.15.118:0/727898819 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7488007470 con 0x7f749806c9e0 2024-08-03T20:47:18.656 INFO:teuthology.orchestra.run.smithi118.stderr:2024-08-03T20:47:18.653+0000 7f7497200640 1 -- 172.21.15.118:0/727898819 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f74980c9b90 con 0x7f749806c9e0 2024-08-03T20:47:18.657 INFO:teuthology.orchestra.run.smithi118.stderr:2024-08-03T20:47:18.654+0000 7f748e000640 1 -- 172.21.15.118:0/727898819 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f74880067d0 con 0x7f749806c9e0 2024-08-03T20:47:18.657 INFO:teuthology.orchestra.run.smithi118.stderr:2024-08-03T20:47:18.654+0000 7f748e000640 1 -- 172.21.15.118:0/727898819 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(5..5 src has 1..5) ==== 1070+0+0 (secure 0 0 0) 0x7f7488007610 con 0x7f749806c9e0 2024-08-03T20:47:18.657 INFO:teuthology.orchestra.run.smithi118.stderr:2024-08-03T20:47:18.655+0000 7f7497200640 1 -- 172.21.15.118:0/727898819 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f749806d770 con 0x7f749806c9e0 2024-08-03T20:47:18.658 INFO:teuthology.orchestra.run.smithi118.stderr:2024-08-03T20:47:18.657+0000 7f748e000640 1 -- 172.21.15.118:0/727898819 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+74906 (secure 0 0 0) 0x7f749806d770 con 0x7f749806c9e0 2024-08-03T20:47:18.735 INFO:teuthology.orchestra.run.smithi118.stderr:2024-08-03T20:47:18.734+0000 7f7497200640 1 -- 172.21.15.118:0/727898819 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd new", "uuid": "6439a280-da45-4148-8130-f16323935ff3", "id": 3} v 0) -- 0x7f749806fa70 con 0x7f749806c9e0 2024-08-03T20:47:18.737 INFO:teuthology.orchestra.run.smithi118.stderr:2024-08-03T20:47:18.736+0000 7f748e000640 1 -- 172.21.15.118:0/727898819 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "6439a280-da45-4148-8130-f16323935ff3", "id": 3}]=0 v6) ==== 112+0+2 (secure 0 0 0) 0x7f7488003650 con 0x7f749806c9e0 2024-08-03T20:47:18.737 INFO:teuthology.orchestra.run.smithi118.stdout:3 2024-08-03T20:47:18.738 INFO:teuthology.orchestra.run.smithi118.stderr:2024-08-03T20:47:18.737+0000 7f7497200640 1 -- 172.21.15.118:0/727898819 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f749806c9e0 msgr2=0x7f74981f2da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:47:18.738 INFO:teuthology.orchestra.run.smithi118.stderr:2024-08-03T20:47:18.737+0000 7f7497200640 1 --2- 172.21.15.118:0/727898819 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f749806c9e0 0x7f74981f2da0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f748800c3d0 tx=0x7f748800c890 comp rx=0 tx=0).stop 2024-08-03T20:47:18.738 INFO:teuthology.orchestra.run.smithi118.stderr:2024-08-03T20:47:18.737+0000 7f7497200640 1 -- 172.21.15.118:0/727898819 shutdown_connections 2024-08-03T20:47:18.738 INFO:teuthology.orchestra.run.smithi118.stderr:2024-08-03T20:47:18.737+0000 7f7497200640 1 --2- 172.21.15.118:0/727898819 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f74980b0170 0x7f74981f8de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:18.738 INFO:teuthology.orchestra.run.smithi118.stderr:2024-08-03T20:47:18.737+0000 7f7497200640 1 --2- 172.21.15.118:0/727898819 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f749806f660 0x7f74980b66c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:18.739 INFO:teuthology.orchestra.run.smithi118.stderr:2024-08-03T20:47:18.737+0000 7f7497200640 1 --2- 172.21.15.118:0/727898819 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f749806c9e0 0x7f74981f2da0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:18.739 INFO:teuthology.orchestra.run.smithi118.stderr:2024-08-03T20:47:18.737+0000 7f7497200640 1 -- 172.21.15.118:0/727898819 >> 172.21.15.118:0/727898819 conn(0x7f74980d0dd0 msgr2=0x7f74980cb8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:47:18.739 INFO:teuthology.orchestra.run.smithi118.stderr:2024-08-03T20:47:18.737+0000 7f7497200640 1 -- 172.21.15.118:0/727898819 shutdown_connections 2024-08-03T20:47:18.739 INFO:teuthology.orchestra.run.smithi118.stderr:2024-08-03T20:47:18.737+0000 7f7497200640 1 -- 172.21.15.118:0/727898819 wait complete. 2024-08-03T20:47:18.759 DEBUG:teuthology.orchestra.run.smithi118:> sudo ceph --cluster ceph osd new 4f902eff-1e18-418d-82c9-2151c69524fa 4 2024-08-03T20:47:18.982 INFO:teuthology.orchestra.run.smithi118.stderr:2024-08-03T20:47:18.980+0000 7fb431600640 1 -- 172.21.15.118:0/3108836263 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb42c0cf510 msgr2=0x7fb42c0cf920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:47:18.982 INFO:teuthology.orchestra.run.smithi118.stderr:2024-08-03T20:47:18.980+0000 7fb431600640 1 --2- 172.21.15.118:0/3108836263 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb42c0cf510 0x7fb42c0cf920 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fb424005fd0 tx=0x7fb42402a940 comp rx=0 tx=0).stop 2024-08-03T20:47:18.982 INFO:teuthology.orchestra.run.smithi118.stderr:2024-08-03T20:47:18.980+0000 7fb431600640 1 -- 172.21.15.118:0/3108836263 shutdown_connections 2024-08-03T20:47:18.982 INFO:teuthology.orchestra.run.smithi118.stderr:2024-08-03T20:47:18.980+0000 7fb431600640 1 --2- 172.21.15.118:0/3108836263 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb42c0cf510 0x7fb42c0cf920 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:18.983 INFO:teuthology.orchestra.run.smithi118.stderr:2024-08-03T20:47:18.980+0000 7fb431600640 1 -- 172.21.15.118:0/3108836263 >> 172.21.15.118:0/3108836263 conn(0x7fb42c08d7a0 msgr2=0x7fb42c09d3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:47:18.983 INFO:teuthology.orchestra.run.smithi118.stderr:2024-08-03T20:47:18.980+0000 7fb431600640 1 -- 172.21.15.118:0/3108836263 shutdown_connections 2024-08-03T20:47:18.983 INFO:teuthology.orchestra.run.smithi118.stderr:2024-08-03T20:47:18.980+0000 7fb431600640 1 -- 172.21.15.118:0/3108836263 wait complete. 2024-08-03T20:47:18.983 INFO:teuthology.orchestra.run.smithi118.stderr:2024-08-03T20:47:18.983+0000 7fb431600640 1 Processor -- start 2024-08-03T20:47:18.983 INFO:teuthology.orchestra.run.smithi118.stderr:2024-08-03T20:47:18.983+0000 7fb431600640 1 -- start start 2024-08-03T20:47:18.988 INFO:teuthology.orchestra.run.smithi118.stderr:2024-08-03T20:47:18.984+0000 7fb431600640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb42c0cf510 0x7fb42c1eb200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:47:18.988 INFO:teuthology.orchestra.run.smithi118.stderr:2024-08-03T20:47:18.984+0000 7fb431600640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fb42c0cffe0 0x7fb42c0b0c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:47:18.988 INFO:teuthology.orchestra.run.smithi118.stderr:2024-08-03T20:47:18.984+0000 7fb431600640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb42c0d26e0 0x7fb42c0b1170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:47:18.988 INFO:teuthology.orchestra.run.smithi118.stderr:2024-08-03T20:47:18.984+0000 7fb431600640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fb42c0d4310 con 0x7fb42c0d26e0 2024-08-03T20:47:18.988 INFO:teuthology.orchestra.run.smithi118.stderr:2024-08-03T20:47:18.984+0000 7fb431600640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7fb42c0d4190 con 0x7fb42c0cf510 2024-08-03T20:47:18.989 INFO:teuthology.orchestra.run.smithi118.stderr:2024-08-03T20:47:18.984+0000 7fb431600640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7fb42c0d4490 con 0x7fb42c0cffe0 2024-08-03T20:47:18.989 INFO:teuthology.orchestra.run.smithi118.stderr:2024-08-03T20:47:18.984+0000 7fb42be00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb42c0cf510 0x7fb42c1eb200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:47:18.989 INFO:teuthology.orchestra.run.smithi118.stderr:2024-08-03T20:47:18.984+0000 7fb42be00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb42c0cf510 0x7fb42c1eb200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.118:45618/0 (socket says 172.21.15.118:45618) 2024-08-03T20:47:18.989 INFO:teuthology.orchestra.run.smithi118.stderr:2024-08-03T20:47:18.984+0000 7fb42b400640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fb42c0cffe0 0x7fb42c0b0c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:47:18.989 INFO:teuthology.orchestra.run.smithi118.stderr:2024-08-03T20:47:18.984+0000 7fb42be00640 1 -- 172.21.15.118:0/1764454649 learned_addr learned my addr 172.21.15.118:0/1764454649 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-08-03T20:47:18.989 INFO:teuthology.orchestra.run.smithi118.stderr:2024-08-03T20:47:18.984+0000 7fb430c00640 1 --2- 172.21.15.118:0/1764454649 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb42c0d26e0 0x7fb42c0b1170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:47:18.989 INFO:teuthology.orchestra.run.smithi118.stderr:2024-08-03T20:47:18.985+0000 7fb42be00640 1 -- 172.21.15.118:0/1764454649 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fb42c0cffe0 msgr2=0x7fb42c0b0c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:47:18.989 INFO:teuthology.orchestra.run.smithi118.stderr:2024-08-03T20:47:18.985+0000 7fb42be00640 1 --2- 172.21.15.118:0/1764454649 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fb42c0cffe0 0x7fb42c0b0c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:18.989 INFO:teuthology.orchestra.run.smithi118.stderr:2024-08-03T20:47:18.985+0000 7fb42be00640 1 -- 172.21.15.118:0/1764454649 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb42c0d26e0 msgr2=0x7fb42c0b1170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:47:18.989 INFO:teuthology.orchestra.run.smithi118.stderr:2024-08-03T20:47:18.985+0000 7fb42be00640 1 --2- 172.21.15.118:0/1764454649 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb42c0d26e0 0x7fb42c0b1170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:18.989 INFO:teuthology.orchestra.run.smithi118.stderr:2024-08-03T20:47:18.985+0000 7fb42be00640 1 -- 172.21.15.118:0/1764454649 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb42c146cc0 con 0x7fb42c0cf510 2024-08-03T20:47:18.989 INFO:teuthology.orchestra.run.smithi118.stderr:2024-08-03T20:47:18.985+0000 7fb430c00640 1 --2- 172.21.15.118:0/1764454649 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb42c0d26e0 0x7fb42c0b1170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-03T20:47:18.989 INFO:teuthology.orchestra.run.smithi118.stderr:2024-08-03T20:47:18.985+0000 7fb42be00640 1 --2- 172.21.15.118:0/1764454649 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb42c0cf510 0x7fb42c1eb200 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fb41c00c430 tx=0x7fb41c00c8f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:47:18.990 INFO:teuthology.orchestra.run.smithi118.stderr:2024-08-03T20:47:18.985+0000 7fb423e00640 1 -- 172.21.15.118:0/1764454649 <== mon.1 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb41c011790 con 0x7fb42c0cf510 2024-08-03T20:47:18.990 INFO:teuthology.orchestra.run.smithi118.stderr:2024-08-03T20:47:18.986+0000 7fb423e00640 1 -- 172.21.15.118:0/1764454649 <== mon.1 v2:172.21.15.118:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fb41c017070 con 0x7fb42c0cf510 2024-08-03T20:47:18.990 INFO:teuthology.orchestra.run.smithi118.stderr:2024-08-03T20:47:18.986+0000 7fb431600640 1 -- 172.21.15.118:0/1764454649 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb42c146fa0 con 0x7fb42c0cf510 2024-08-03T20:47:18.990 INFO:teuthology.orchestra.run.smithi118.stderr:2024-08-03T20:47:18.986+0000 7fb423e00640 1 -- 172.21.15.118:0/1764454649 <== mon.1 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb41c011ce0 con 0x7fb42c0cf510 2024-08-03T20:47:18.990 INFO:teuthology.orchestra.run.smithi118.stderr:2024-08-03T20:47:18.986+0000 7fb431600640 1 -- 172.21.15.118:0/1764454649 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb42c145c50 con 0x7fb42c0cf510 2024-08-03T20:47:18.990 INFO:teuthology.orchestra.run.smithi118.stderr:2024-08-03T20:47:18.986+0000 7fb423e00640 1 -- 172.21.15.118:0/1764454649 <== mon.1 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fb41c01c460 con 0x7fb42c0cf510 2024-08-03T20:47:18.990 INFO:teuthology.orchestra.run.smithi118.stderr:2024-08-03T20:47:18.986+0000 7fb423e00640 1 -- 172.21.15.118:0/1764454649 <== mon.1 v2:172.21.15.118:3300/0 5 ==== osd_map(6..6 src has 1..6) ==== 1185+0+0 (secure 0 0 0) 0x7fb41c012ca0 con 0x7fb42c0cf510 2024-08-03T20:47:18.990 INFO:teuthology.orchestra.run.smithi118.stderr:2024-08-03T20:47:18.987+0000 7fb431600640 1 -- 172.21.15.118:0/1764454649 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb42c145eb0 con 0x7fb42c0cf510 2024-08-03T20:47:18.990 INFO:teuthology.orchestra.run.smithi118.stderr:2024-08-03T20:47:18.989+0000 7fb423e00640 1 -- 172.21.15.118:0/1764454649 <== mon.1 v2:172.21.15.118:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+74906 (secure 0 0 0) 0x7fb41c012790 con 0x7fb42c0cf510 2024-08-03T20:47:19.069 INFO:teuthology.orchestra.run.smithi118.stderr:2024-08-03T20:47:19.068+0000 7fb431600640 1 -- 172.21.15.118:0/1764454649 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command({"prefix": "osd new", "uuid": "4f902eff-1e18-418d-82c9-2151c69524fa", "id": 4} v 0) -- 0x7fb42c0d4940 con 0x7fb42c0cf510 2024-08-03T20:47:19.073 INFO:teuthology.orchestra.run.smithi118.stderr:2024-08-03T20:47:19.072+0000 7fb423e00640 1 -- 172.21.15.118:0/1764454649 <== mon.1 v2:172.21.15.118:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "4f902eff-1e18-418d-82c9-2151c69524fa", "id": 4}]=0 v7) ==== 112+0+2 (secure 0 0 0) 0x7fb41c012970 con 0x7fb42c0cf510 2024-08-03T20:47:19.073 INFO:teuthology.orchestra.run.smithi118.stdout:4 2024-08-03T20:47:19.074 INFO:teuthology.orchestra.run.smithi118.stderr:2024-08-03T20:47:19.073+0000 7fb431600640 1 -- 172.21.15.118:0/1764454649 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb42c0cf510 msgr2=0x7fb42c1eb200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:47:19.075 INFO:teuthology.orchestra.run.smithi118.stderr:2024-08-03T20:47:19.073+0000 7fb431600640 1 --2- 172.21.15.118:0/1764454649 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb42c0cf510 0x7fb42c1eb200 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fb41c00c430 tx=0x7fb41c00c8f0 comp rx=0 tx=0).stop 2024-08-03T20:47:19.075 INFO:teuthology.orchestra.run.smithi118.stderr:2024-08-03T20:47:19.073+0000 7fb431600640 1 -- 172.21.15.118:0/1764454649 shutdown_connections 2024-08-03T20:47:19.075 INFO:teuthology.orchestra.run.smithi118.stderr:2024-08-03T20:47:19.073+0000 7fb431600640 1 --2- 172.21.15.118:0/1764454649 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb42c0d26e0 0x7fb42c0b1170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:19.075 INFO:teuthology.orchestra.run.smithi118.stderr:2024-08-03T20:47:19.073+0000 7fb431600640 1 --2- 172.21.15.118:0/1764454649 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fb42c0cffe0 0x7fb42c0b0c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:19.075 INFO:teuthology.orchestra.run.smithi118.stderr:2024-08-03T20:47:19.073+0000 7fb431600640 1 --2- 172.21.15.118:0/1764454649 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb42c0cf510 0x7fb42c1eb200 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:19.075 INFO:teuthology.orchestra.run.smithi118.stderr:2024-08-03T20:47:19.073+0000 7fb431600640 1 -- 172.21.15.118:0/1764454649 >> 172.21.15.118:0/1764454649 conn(0x7fb42c08d7a0 msgr2=0x7fb42c0b9090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:47:19.075 INFO:teuthology.orchestra.run.smithi118.stderr:2024-08-03T20:47:19.073+0000 7fb431600640 1 -- 172.21.15.118:0/1764454649 shutdown_connections 2024-08-03T20:47:19.075 INFO:teuthology.orchestra.run.smithi118.stderr:2024-08-03T20:47:19.073+0000 7fb431600640 1 -- 172.21.15.118:0/1764454649 wait complete. 2024-08-03T20:47:19.097 DEBUG:teuthology.orchestra.run.smithi118:> sudo ceph --cluster ceph osd new bfd9feac-dd06-458c-91b9-af7f175f612c 5 2024-08-03T20:47:19.317 INFO:teuthology.orchestra.run.smithi118.stderr:2024-08-03T20:47:19.316+0000 7f721a800640 1 -- 172.21.15.118:0/3863137113 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f72140cf510 msgr2=0x7f72140cf920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:47:19.317 INFO:teuthology.orchestra.run.smithi118.stderr:2024-08-03T20:47:19.316+0000 7f721a800640 1 --2- 172.21.15.118:0/3863137113 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f72140cf510 0x7f72140cf920 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f720c005fd0 tx=0x7f720c02a940 comp rx=0 tx=0).stop 2024-08-03T20:47:19.318 INFO:teuthology.orchestra.run.smithi118.stderr:2024-08-03T20:47:19.316+0000 7f721a800640 1 -- 172.21.15.118:0/3863137113 shutdown_connections 2024-08-03T20:47:19.318 INFO:teuthology.orchestra.run.smithi118.stderr:2024-08-03T20:47:19.316+0000 7f721a800640 1 --2- 172.21.15.118:0/3863137113 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f72140cf510 0x7f72140cf920 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:19.318 INFO:teuthology.orchestra.run.smithi118.stderr:2024-08-03T20:47:19.316+0000 7f721a800640 1 -- 172.21.15.118:0/3863137113 >> 172.21.15.118:0/3863137113 conn(0x7f721408d7a0 msgr2=0x7f721409d3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:47:19.318 INFO:teuthology.orchestra.run.smithi118.stderr:2024-08-03T20:47:19.316+0000 7f721a800640 1 -- 172.21.15.118:0/3863137113 shutdown_connections 2024-08-03T20:47:19.318 INFO:teuthology.orchestra.run.smithi118.stderr:2024-08-03T20:47:19.316+0000 7f721a800640 1 -- 172.21.15.118:0/3863137113 wait complete. 2024-08-03T20:47:19.319 INFO:teuthology.orchestra.run.smithi118.stderr:2024-08-03T20:47:19.318+0000 7f721a800640 1 Processor -- start 2024-08-03T20:47:19.319 INFO:teuthology.orchestra.run.smithi118.stderr:2024-08-03T20:47:19.318+0000 7f721a800640 1 -- start start 2024-08-03T20:47:19.321 INFO:teuthology.orchestra.run.smithi118.stderr:2024-08-03T20:47:19.320+0000 7f721a800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f72140cf510 0x7f72141e6890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:47:19.321 INFO:teuthology.orchestra.run.smithi118.stderr:2024-08-03T20:47:19.320+0000 7f721a800640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f72140cffe0 0x7f721409dd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:47:19.321 INFO:teuthology.orchestra.run.smithi118.stderr:2024-08-03T20:47:19.320+0000 7f721a800640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f72140d26e0 0x7f7214152700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:47:19.321 INFO:teuthology.orchestra.run.smithi118.stderr:2024-08-03T20:47:19.320+0000 7f721a800640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f72140d4310 con 0x7f72140cf510 2024-08-03T20:47:19.322 INFO:teuthology.orchestra.run.smithi118.stderr:2024-08-03T20:47:19.320+0000 7f721a800640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f72140d4190 con 0x7f72140d26e0 2024-08-03T20:47:19.322 INFO:teuthology.orchestra.run.smithi118.stderr:2024-08-03T20:47:19.320+0000 7f721a800640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7f72140d4490 con 0x7f72140cffe0 2024-08-03T20:47:19.322 INFO:teuthology.orchestra.run.smithi118.stderr:2024-08-03T20:47:19.320+0000 7f7219e00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f72140d26e0 0x7f7214152700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:47:19.322 INFO:teuthology.orchestra.run.smithi118.stderr:2024-08-03T20:47:19.320+0000 7f7218a00640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f72140cffe0 0x7f721409dd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:47:19.322 INFO:teuthology.orchestra.run.smithi118.stderr:2024-08-03T20:47:19.320+0000 7f7219400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f72140cf510 0x7f72141e6890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:47:19.322 INFO:teuthology.orchestra.run.smithi118.stderr:2024-08-03T20:47:19.320+0000 7f7219e00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f72140d26e0 0x7f7214152700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.118:45640/0 (socket says 172.21.15.118:45640) 2024-08-03T20:47:19.322 INFO:teuthology.orchestra.run.smithi118.stderr:2024-08-03T20:47:19.320+0000 7f7219e00640 1 -- 172.21.15.118:0/3618809571 learned_addr learned my addr 172.21.15.118:0/3618809571 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-08-03T20:47:19.323 INFO:teuthology.orchestra.run.smithi118.stderr:2024-08-03T20:47:19.321+0000 7f7218a00640 1 -- 172.21.15.118:0/3618809571 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f72140d26e0 msgr2=0x7f7214152700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:47:19.324 INFO:teuthology.orchestra.run.smithi118.stderr:2024-08-03T20:47:19.321+0000 7f7218a00640 1 --2- 172.21.15.118:0/3618809571 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f72140d26e0 0x7f7214152700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:19.324 INFO:teuthology.orchestra.run.smithi118.stderr:2024-08-03T20:47:19.321+0000 7f7218a00640 1 -- 172.21.15.118:0/3618809571 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f72140cf510 msgr2=0x7f72141e6890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:47:19.324 INFO:teuthology.orchestra.run.smithi118.stderr:2024-08-03T20:47:19.321+0000 7f7218a00640 1 --2- 172.21.15.118:0/3618809571 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f72140cf510 0x7f72141e6890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:19.324 INFO:teuthology.orchestra.run.smithi118.stderr:2024-08-03T20:47:19.321+0000 7f7218a00640 1 -- 172.21.15.118:0/3618809571 --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f72140a0260 con 0x7f72140cffe0 2024-08-03T20:47:19.324 INFO:teuthology.orchestra.run.smithi118.stderr:2024-08-03T20:47:19.321+0000 7f7219e00640 1 --2- 172.21.15.118:0/3618809571 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f72140d26e0 0x7f7214152700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-03T20:47:19.324 INFO:teuthology.orchestra.run.smithi118.stderr:2024-08-03T20:47:19.321+0000 7f7219400640 1 --2- 172.21.15.118:0/3618809571 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f72140cf510 0x7f72141e6890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-03T20:47:19.324 INFO:teuthology.orchestra.run.smithi118.stderr:2024-08-03T20:47:19.321+0000 7f7218a00640 1 --2- 172.21.15.118:0/3618809571 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f72140cffe0 0x7f721409dd40 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f720000c3c0 tx=0x7f720000c880 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:47:19.324 INFO:teuthology.orchestra.run.smithi118.stderr:2024-08-03T20:47:19.321+0000 7f7211600640 1 -- 172.21.15.118:0/3618809571 <== mon.2 v2:172.21.15.118:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7200017070 con 0x7f72140cffe0 2024-08-03T20:47:19.324 INFO:teuthology.orchestra.run.smithi118.stderr:2024-08-03T20:47:19.321+0000 7f7211600640 1 -- 172.21.15.118:0/3618809571 <== mon.2 v2:172.21.15.118:3301/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f72000053c0 con 0x7f72140cffe0 2024-08-03T20:47:19.325 INFO:teuthology.orchestra.run.smithi118.stderr:2024-08-03T20:47:19.321+0000 7f7211600640 1 -- 172.21.15.118:0/3618809571 <== mon.2 v2:172.21.15.118:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7200003850 con 0x7f72140cffe0 2024-08-03T20:47:19.325 INFO:teuthology.orchestra.run.smithi118.stderr:2024-08-03T20:47:19.321+0000 7f721a800640 1 -- 172.21.15.118:0/3618809571 --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f72140a0540 con 0x7f72140cffe0 2024-08-03T20:47:19.325 INFO:teuthology.orchestra.run.smithi118.stderr:2024-08-03T20:47:19.321+0000 7f721a800640 1 -- 172.21.15.118:0/3618809571 --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f72140a0b00 con 0x7f72140cffe0 2024-08-03T20:47:19.325 INFO:teuthology.orchestra.run.smithi118.stderr:2024-08-03T20:47:19.322+0000 7f7211600640 1 -- 172.21.15.118:0/3618809571 <== mon.2 v2:172.21.15.118:3301/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f720001a070 con 0x7f72140cffe0 2024-08-03T20:47:19.325 INFO:teuthology.orchestra.run.smithi118.stderr:2024-08-03T20:47:19.322+0000 7f7211600640 1 -- 172.21.15.118:0/3618809571 <== mon.2 v2:172.21.15.118:3301/0 5 ==== osd_map(7..7 src has 1..7) ==== 1300+0+0 (secure 0 0 0) 0x7f7200011cc0 con 0x7f72140cffe0 2024-08-03T20:47:19.325 INFO:teuthology.orchestra.run.smithi118.stderr:2024-08-03T20:47:19.322+0000 7f721a800640 1 -- 172.21.15.118:0/3618809571 --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f72140d4940 con 0x7f72140cffe0 2024-08-03T20:47:19.327 INFO:teuthology.orchestra.run.smithi118.stderr:2024-08-03T20:47:19.327+0000 7f7211600640 1 -- 172.21.15.118:0/3618809571 <== mon.2 v2:172.21.15.118:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+74906 (secure 0 0 0) 0x7f7200021020 con 0x7f72140cffe0 2024-08-03T20:47:19.403 INFO:teuthology.orchestra.run.smithi118.stderr:2024-08-03T20:47:19.402+0000 7f721a800640 1 -- 172.21.15.118:0/3618809571 --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_command({"prefix": "osd new", "uuid": "bfd9feac-dd06-458c-91b9-af7f175f612c", "id": 5} v 0) -- 0x7f7214160460 con 0x7f72140cffe0 2024-08-03T20:47:19.408 INFO:teuthology.orchestra.run.smithi118.stderr:2024-08-03T20:47:19.407+0000 7f7211600640 1 -- 172.21.15.118:0/3618809571 <== mon.2 v2:172.21.15.118:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "bfd9feac-dd06-458c-91b9-af7f175f612c", "id": 5}]=0 v8) ==== 112+0+2 (secure 0 0 0) 0x7f7200011590 con 0x7f72140cffe0 2024-08-03T20:47:19.408 INFO:teuthology.orchestra.run.smithi118.stdout:5 2024-08-03T20:47:19.410 INFO:teuthology.orchestra.run.smithi118.stderr:2024-08-03T20:47:19.408+0000 7f721a800640 1 -- 172.21.15.118:0/3618809571 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f72140cffe0 msgr2=0x7f721409dd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:47:19.410 INFO:teuthology.orchestra.run.smithi118.stderr:2024-08-03T20:47:19.408+0000 7f721a800640 1 --2- 172.21.15.118:0/3618809571 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f72140cffe0 0x7f721409dd40 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f720000c3c0 tx=0x7f720000c880 comp rx=0 tx=0).stop 2024-08-03T20:47:19.410 INFO:teuthology.orchestra.run.smithi118.stderr:2024-08-03T20:47:19.409+0000 7f721a800640 1 -- 172.21.15.118:0/3618809571 shutdown_connections 2024-08-03T20:47:19.410 INFO:teuthology.orchestra.run.smithi118.stderr:2024-08-03T20:47:19.409+0000 7f721a800640 1 --2- 172.21.15.118:0/3618809571 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f72140d26e0 0x7f7214152700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:19.410 INFO:teuthology.orchestra.run.smithi118.stderr:2024-08-03T20:47:19.409+0000 7f721a800640 1 --2- 172.21.15.118:0/3618809571 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f72140cffe0 0x7f721409dd40 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:19.410 INFO:teuthology.orchestra.run.smithi118.stderr:2024-08-03T20:47:19.409+0000 7f721a800640 1 --2- 172.21.15.118:0/3618809571 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f72140cf510 0x7f72141e6890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:19.410 INFO:teuthology.orchestra.run.smithi118.stderr:2024-08-03T20:47:19.409+0000 7f721a800640 1 -- 172.21.15.118:0/3618809571 >> 172.21.15.118:0/3618809571 conn(0x7f721408d7a0 msgr2=0x7f72140b5a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:47:19.410 INFO:teuthology.orchestra.run.smithi118.stderr:2024-08-03T20:47:19.409+0000 7f721a800640 1 -- 172.21.15.118:0/3618809571 shutdown_connections 2024-08-03T20:47:19.410 INFO:teuthology.orchestra.run.smithi118.stderr:2024-08-03T20:47:19.409+0000 7f721a800640 1 -- 172.21.15.118:0/3618809571 wait complete. 2024-08-03T20:47:19.433 DEBUG:teuthology.orchestra.run.smithi118:> sudo ceph --cluster ceph osd new 5ed5a279-9d8e-416d-bbc7-7edd3e6d5705 6 2024-08-03T20:47:19.655 INFO:teuthology.orchestra.run.smithi118.stderr:2024-08-03T20:47:19.653+0000 7fe63ea00640 1 -- 172.21.15.118:0/3972674400 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fe6380b43d0 msgr2=0x7fe6380cf510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:47:19.655 INFO:teuthology.orchestra.run.smithi118.stderr:2024-08-03T20:47:19.653+0000 7fe63ea00640 1 --2- 172.21.15.118:0/3972674400 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fe6380b43d0 0x7fe6380cf510 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fe6380d1b20 tx=0x7fe63002aac0 comp rx=0 tx=0).stop 2024-08-03T20:47:19.655 INFO:teuthology.orchestra.run.smithi118.stderr:2024-08-03T20:47:19.654+0000 7fe63ea00640 1 -- 172.21.15.118:0/3972674400 shutdown_connections 2024-08-03T20:47:19.655 INFO:teuthology.orchestra.run.smithi118.stderr:2024-08-03T20:47:19.654+0000 7fe63ea00640 1 --2- 172.21.15.118:0/3972674400 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fe6380b43d0 0x7fe6380cf510 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:19.656 INFO:teuthology.orchestra.run.smithi118.stderr:2024-08-03T20:47:19.654+0000 7fe63ea00640 1 --2- 172.21.15.118:0/3972674400 >> v2:172.21.15.32:3300/0 conn(0x7fe6380cffe0 0x7fe6380b3e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:19.656 INFO:teuthology.orchestra.run.smithi118.stderr:2024-08-03T20:47:19.654+0000 7fe63ea00640 1 -- 172.21.15.118:0/3972674400 >> 172.21.15.118:0/3972674400 conn(0x7fe63808d7a0 msgr2=0x7fe63809d3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:47:19.656 INFO:teuthology.orchestra.run.smithi118.stderr:2024-08-03T20:47:19.654+0000 7fe63ea00640 1 -- 172.21.15.118:0/3972674400 shutdown_connections 2024-08-03T20:47:19.656 INFO:teuthology.orchestra.run.smithi118.stderr:2024-08-03T20:47:19.654+0000 7fe63ea00640 1 -- 172.21.15.118:0/3972674400 wait complete. 2024-08-03T20:47:19.658 INFO:teuthology.orchestra.run.smithi118.stderr:2024-08-03T20:47:19.657+0000 7fe63ea00640 1 Processor -- start 2024-08-03T20:47:19.659 INFO:teuthology.orchestra.run.smithi118.stderr:2024-08-03T20:47:19.658+0000 7fe63ea00640 1 -- start start 2024-08-03T20:47:19.663 INFO:teuthology.orchestra.run.smithi118.stderr:2024-08-03T20:47:19.660+0000 7fe63ea00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe6380b43d0 0x7fe6381f5310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:47:19.663 INFO:teuthology.orchestra.run.smithi118.stderr:2024-08-03T20:47:19.660+0000 7fe63ea00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fe6380cffe0 0x7fe6380b1db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:47:19.663 INFO:teuthology.orchestra.run.smithi118.stderr:2024-08-03T20:47:19.660+0000 7fe63ea00640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fe6380d26e0 0x7fe6380b22f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:47:19.664 INFO:teuthology.orchestra.run.smithi118.stderr:2024-08-03T20:47:19.660+0000 7fe63ea00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fe6380d4100 con 0x7fe6380b43d0 2024-08-03T20:47:19.664 INFO:teuthology.orchestra.run.smithi118.stderr:2024-08-03T20:47:19.660+0000 7fe63ea00640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7fe6380d3f80 con 0x7fe6380cffe0 2024-08-03T20:47:19.664 INFO:teuthology.orchestra.run.smithi118.stderr:2024-08-03T20:47:19.660+0000 7fe63ea00640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7fe6380d4280 con 0x7fe6380d26e0 2024-08-03T20:47:19.664 INFO:teuthology.orchestra.run.smithi118.stderr:2024-08-03T20:47:19.660+0000 7fe63cc00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fe6380cffe0 0x7fe6380b1db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:47:19.664 INFO:teuthology.orchestra.run.smithi118.stderr:2024-08-03T20:47:19.660+0000 7fe63d600640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe6380b43d0 0x7fe6381f5310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:47:19.664 INFO:teuthology.orchestra.run.smithi118.stderr:2024-08-03T20:47:19.660+0000 7fe63cc00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fe6380cffe0 0x7fe6380b1db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.118:45656/0 (socket says 172.21.15.118:45656) 2024-08-03T20:47:19.664 INFO:teuthology.orchestra.run.smithi118.stderr:2024-08-03T20:47:19.660+0000 7fe63cc00640 1 -- 172.21.15.118:0/3647979980 learned_addr learned my addr 172.21.15.118:0/3647979980 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-08-03T20:47:19.664 INFO:teuthology.orchestra.run.smithi118.stderr:2024-08-03T20:47:19.660+0000 7fe63e000640 1 --2- 172.21.15.118:0/3647979980 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fe6380d26e0 0x7fe6380b22f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:47:19.665 INFO:teuthology.orchestra.run.smithi118.stderr:2024-08-03T20:47:19.661+0000 7fe63d600640 1 -- 172.21.15.118:0/3647979980 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fe6380d26e0 msgr2=0x7fe6380b22f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:47:19.665 INFO:teuthology.orchestra.run.smithi118.stderr:2024-08-03T20:47:19.661+0000 7fe63d600640 1 --2- 172.21.15.118:0/3647979980 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fe6380d26e0 0x7fe6380b22f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:19.665 INFO:teuthology.orchestra.run.smithi118.stderr:2024-08-03T20:47:19.661+0000 7fe63d600640 1 -- 172.21.15.118:0/3647979980 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fe6380cffe0 msgr2=0x7fe6380b1db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:47:19.665 INFO:teuthology.orchestra.run.smithi118.stderr:2024-08-03T20:47:19.661+0000 7fe63d600640 1 --2- 172.21.15.118:0/3647979980 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fe6380cffe0 0x7fe6380b1db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:19.665 INFO:teuthology.orchestra.run.smithi118.stderr:2024-08-03T20:47:19.661+0000 7fe63d600640 1 -- 172.21.15.118:0/3647979980 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe6381f96b0 con 0x7fe6380b43d0 2024-08-03T20:47:19.665 INFO:teuthology.orchestra.run.smithi118.stderr:2024-08-03T20:47:19.661+0000 7fe63cc00640 1 --2- 172.21.15.118:0/3647979980 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fe6380cffe0 0x7fe6380b1db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-03T20:47:19.665 INFO:teuthology.orchestra.run.smithi118.stderr:2024-08-03T20:47:19.661+0000 7fe63d600640 1 --2- 172.21.15.118:0/3647979980 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe6380b43d0 0x7fe6381f5310 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fe62c00c3c0 tx=0x7fe62c00c7c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:47:19.665 INFO:teuthology.orchestra.run.smithi118.stderr:2024-08-03T20:47:19.661+0000 7fe635600640 1 -- 172.21.15.118:0/3647979980 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe62c017070 con 0x7fe6380b43d0 2024-08-03T20:47:19.665 INFO:teuthology.orchestra.run.smithi118.stderr:2024-08-03T20:47:19.662+0000 7fe635600640 1 -- 172.21.15.118:0/3647979980 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fe62c0035f0 con 0x7fe6380b43d0 2024-08-03T20:47:19.666 INFO:teuthology.orchestra.run.smithi118.stderr:2024-08-03T20:47:19.662+0000 7fe635600640 1 -- 172.21.15.118:0/3647979980 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe62c003a40 con 0x7fe6380b43d0 2024-08-03T20:47:19.666 INFO:teuthology.orchestra.run.smithi118.stderr:2024-08-03T20:47:19.662+0000 7fe63ea00640 1 -- 172.21.15.118:0/3647979980 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe6381f9990 con 0x7fe6380b43d0 2024-08-03T20:47:19.666 INFO:teuthology.orchestra.run.smithi118.stderr:2024-08-03T20:47:19.662+0000 7fe63ea00640 1 -- 172.21.15.118:0/3647979980 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe6381f9f50 con 0x7fe6380b43d0 2024-08-03T20:47:19.666 INFO:teuthology.orchestra.run.smithi118.stderr:2024-08-03T20:47:19.662+0000 7fe635600640 1 -- 172.21.15.118:0/3647979980 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fe62c01a070 con 0x7fe6380b43d0 2024-08-03T20:47:19.666 INFO:teuthology.orchestra.run.smithi118.stderr:2024-08-03T20:47:19.662+0000 7fe635600640 1 -- 172.21.15.118:0/3647979980 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(8..8 src has 1..8) ==== 1415+0+0 (secure 0 0 0) 0x7fe62c003be0 con 0x7fe6380b43d0 2024-08-03T20:47:19.667 INFO:teuthology.orchestra.run.smithi118.stderr:2024-08-03T20:47:19.663+0000 7fe63ea00640 1 -- 172.21.15.118:0/3647979980 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe6380d47a0 con 0x7fe6380b43d0 2024-08-03T20:47:19.667 INFO:teuthology.orchestra.run.smithi118.stderr:2024-08-03T20:47:19.665+0000 7fe635600640 1 -- 172.21.15.118:0/3647979980 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+74906 (secure 0 0 0) 0x7fe62c021020 con 0x7fe6380b43d0 2024-08-03T20:47:19.743 INFO:teuthology.orchestra.run.smithi118.stderr:2024-08-03T20:47:19.742+0000 7fe63ea00640 1 -- 172.21.15.118:0/3647979980 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd new", "uuid": "5ed5a279-9d8e-416d-bbc7-7edd3e6d5705", "id": 6} v 0) -- 0x7fe6380cf7b0 con 0x7fe6380b43d0 2024-08-03T20:47:19.745 INFO:teuthology.orchestra.run.smithi118.stderr:2024-08-03T20:47:19.744+0000 7fe635600640 1 -- 172.21.15.118:0/3647979980 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "5ed5a279-9d8e-416d-bbc7-7edd3e6d5705", "id": 6}]=0 v9) ==== 112+0+2 (secure 0 0 0) 0x7fe62c01c440 con 0x7fe6380b43d0 2024-08-03T20:47:19.745 INFO:teuthology.orchestra.run.smithi118.stdout:6 2024-08-03T20:47:19.746 INFO:teuthology.orchestra.run.smithi118.stderr:2024-08-03T20:47:19.745+0000 7fe63ea00640 1 -- 172.21.15.118:0/3647979980 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe6380b43d0 msgr2=0x7fe6381f5310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:47:19.746 INFO:teuthology.orchestra.run.smithi118.stderr:2024-08-03T20:47:19.745+0000 7fe63ea00640 1 --2- 172.21.15.118:0/3647979980 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe6380b43d0 0x7fe6381f5310 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fe62c00c3c0 tx=0x7fe62c00c7c0 comp rx=0 tx=0).stop 2024-08-03T20:47:19.746 INFO:teuthology.orchestra.run.smithi118.stderr:2024-08-03T20:47:19.745+0000 7fe63ea00640 1 -- 172.21.15.118:0/3647979980 shutdown_connections 2024-08-03T20:47:19.746 INFO:teuthology.orchestra.run.smithi118.stderr:2024-08-03T20:47:19.745+0000 7fe63ea00640 1 --2- 172.21.15.118:0/3647979980 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fe6380cffe0 0x7fe6380b1db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:19.747 INFO:teuthology.orchestra.run.smithi118.stderr:2024-08-03T20:47:19.745+0000 7fe63ea00640 1 --2- 172.21.15.118:0/3647979980 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fe6380d26e0 0x7fe6380b22f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:19.747 INFO:teuthology.orchestra.run.smithi118.stderr:2024-08-03T20:47:19.745+0000 7fe63ea00640 1 --2- 172.21.15.118:0/3647979980 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe6380b43d0 0x7fe6381f5310 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:19.747 INFO:teuthology.orchestra.run.smithi118.stderr:2024-08-03T20:47:19.745+0000 7fe63ea00640 1 -- 172.21.15.118:0/3647979980 >> 172.21.15.118:0/3647979980 conn(0x7fe63808d7a0 msgr2=0x7fe63806eda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:47:19.747 INFO:teuthology.orchestra.run.smithi118.stderr:2024-08-03T20:47:19.745+0000 7fe63ea00640 1 -- 172.21.15.118:0/3647979980 shutdown_connections 2024-08-03T20:47:19.747 INFO:teuthology.orchestra.run.smithi118.stderr:2024-08-03T20:47:19.745+0000 7fe63ea00640 1 -- 172.21.15.118:0/3647979980 wait complete. 2024-08-03T20:47:19.770 DEBUG:teuthology.orchestra.run.smithi118:> sudo ceph --cluster ceph osd new f2086c47-875e-4451-a436-2008d1ea16af 7 2024-08-03T20:47:19.993 INFO:teuthology.orchestra.run.smithi118.stderr:2024-08-03T20:47:19.992+0000 7f0da7000640 1 -- 172.21.15.118:0/117312192 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f0da00b59f0 msgr2=0x7f0da00d4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:47:19.993 INFO:teuthology.orchestra.run.smithi118.stderr:2024-08-03T20:47:19.992+0000 7f0da7000640 1 --2- 172.21.15.118:0/117312192 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f0da00b59f0 0x7f0da00d4cc0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f0d90005fd0 tx=0x7f0d9002a670 comp rx=0 tx=0).stop 2024-08-03T20:47:19.993 INFO:teuthology.orchestra.run.smithi118.stderr:2024-08-03T20:47:19.992+0000 7f0da7000640 1 -- 172.21.15.118:0/117312192 shutdown_connections 2024-08-03T20:47:19.993 INFO:teuthology.orchestra.run.smithi118.stderr:2024-08-03T20:47:19.992+0000 7f0da7000640 1 --2- 172.21.15.118:0/117312192 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f0da00cf9c0 0x7f0da00d0a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:19.994 INFO:teuthology.orchestra.run.smithi118.stderr:2024-08-03T20:47:19.992+0000 7f0da7000640 1 --2- 172.21.15.118:0/117312192 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f0da00b59f0 0x7f0da00d4cc0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:19.994 INFO:teuthology.orchestra.run.smithi118.stderr:2024-08-03T20:47:19.992+0000 7f0da7000640 1 -- 172.21.15.118:0/117312192 >> 172.21.15.118:0/117312192 conn(0x7f0da009d660 msgr2=0x7f0da00d4330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:47:19.994 INFO:teuthology.orchestra.run.smithi118.stderr:2024-08-03T20:47:19.992+0000 7f0da7000640 1 -- 172.21.15.118:0/117312192 shutdown_connections 2024-08-03T20:47:19.994 INFO:teuthology.orchestra.run.smithi118.stderr:2024-08-03T20:47:19.992+0000 7f0da7000640 1 -- 172.21.15.118:0/117312192 wait complete. 2024-08-03T20:47:19.995 INFO:teuthology.orchestra.run.smithi118.stderr:2024-08-03T20:47:19.994+0000 7f0da7000640 1 Processor -- start 2024-08-03T20:47:19.995 INFO:teuthology.orchestra.run.smithi118.stderr:2024-08-03T20:47:19.994+0000 7f0da7000640 1 -- start start 2024-08-03T20:47:19.998 INFO:teuthology.orchestra.run.smithi118.stderr:2024-08-03T20:47:19.996+0000 7f0da7000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0da00b59f0 0x7f0da01577f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:47:19.998 INFO:teuthology.orchestra.run.smithi118.stderr:2024-08-03T20:47:19.996+0000 7f0da7000640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f0da00cf9c0 0x7f0da014be60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:47:19.998 INFO:teuthology.orchestra.run.smithi118.stderr:2024-08-03T20:47:19.996+0000 7f0da7000640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f0da00d0fb0 0x7f0da014c3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:47:19.998 INFO:teuthology.orchestra.run.smithi118.stderr:2024-08-03T20:47:19.996+0000 7f0da7000640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f0da00d2940 con 0x7f0da00b59f0 2024-08-03T20:47:19.998 INFO:teuthology.orchestra.run.smithi118.stderr:2024-08-03T20:47:19.996+0000 7f0da7000640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f0da00b9030 con 0x7f0da00cf9c0 2024-08-03T20:47:19.998 INFO:teuthology.orchestra.run.smithi118.stderr:2024-08-03T20:47:19.996+0000 7f0da7000640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7f0da00d2ac0 con 0x7f0da00d0fb0 2024-08-03T20:47:19.998 INFO:teuthology.orchestra.run.smithi118.stderr:2024-08-03T20:47:19.996+0000 7f0d97e00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f0da00cf9c0 0x7f0da014be60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:47:19.998 INFO:teuthology.orchestra.run.smithi118.stderr:2024-08-03T20:47:19.996+0000 7f0da5000640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f0da00d0fb0 0x7f0da014c3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:47:19.998 INFO:teuthology.orchestra.run.smithi118.stderr:2024-08-03T20:47:19.996+0000 7f0d97e00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f0da00cf9c0 0x7f0da014be60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.118:45676/0 (socket says 172.21.15.118:45676) 2024-08-03T20:47:19.998 INFO:teuthology.orchestra.run.smithi118.stderr:2024-08-03T20:47:19.996+0000 7f0d97e00640 1 -- 172.21.15.118:0/2362089564 learned_addr learned my addr 172.21.15.118:0/2362089564 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-08-03T20:47:19.998 INFO:teuthology.orchestra.run.smithi118.stderr:2024-08-03T20:47:19.996+0000 7f0d9fe00640 1 --2- 172.21.15.118:0/2362089564 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0da00b59f0 0x7f0da01577f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:47:19.998 INFO:teuthology.orchestra.run.smithi118.stderr:2024-08-03T20:47:19.996+0000 7f0d97e00640 1 -- 172.21.15.118:0/2362089564 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f0da00d0fb0 msgr2=0x7f0da014c3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:47:19.999 INFO:teuthology.orchestra.run.smithi118.stderr:2024-08-03T20:47:19.996+0000 7f0d97e00640 1 --2- 172.21.15.118:0/2362089564 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f0da00d0fb0 0x7f0da014c3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:19.999 INFO:teuthology.orchestra.run.smithi118.stderr:2024-08-03T20:47:19.996+0000 7f0d97e00640 1 -- 172.21.15.118:0/2362089564 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0da00b59f0 msgr2=0x7f0da01577f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:47:19.999 INFO:teuthology.orchestra.run.smithi118.stderr:2024-08-03T20:47:19.996+0000 7f0d97e00640 1 --2- 172.21.15.118:0/2362089564 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0da00b59f0 0x7f0da01577f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:19.999 INFO:teuthology.orchestra.run.smithi118.stderr:2024-08-03T20:47:19.996+0000 7f0d97e00640 1 -- 172.21.15.118:0/2362089564 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0da015fbe0 con 0x7f0da00cf9c0 2024-08-03T20:47:19.999 INFO:teuthology.orchestra.run.smithi118.stderr:2024-08-03T20:47:19.996+0000 7f0da5000640 1 --2- 172.21.15.118:0/2362089564 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f0da00d0fb0 0x7f0da014c3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-03T20:47:19.999 INFO:teuthology.orchestra.run.smithi118.stderr:2024-08-03T20:47:19.997+0000 7f0d97e00640 1 --2- 172.21.15.118:0/2362089564 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f0da00cf9c0 0x7f0da014be60 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f0d8800e7c0 tx=0x7f0d8800ec80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:47:20.000 INFO:teuthology.orchestra.run.smithi118.stderr:2024-08-03T20:47:19.997+0000 7f0d9cc00640 1 -- 172.21.15.118:0/2362089564 <== mon.1 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d8800ccc0 con 0x7f0da00cf9c0 2024-08-03T20:47:20.000 INFO:teuthology.orchestra.run.smithi118.stderr:2024-08-03T20:47:19.997+0000 7f0d9cc00640 1 -- 172.21.15.118:0/2362089564 <== mon.1 v2:172.21.15.118:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f0d88010430 con 0x7f0da00cf9c0 2024-08-03T20:47:20.000 INFO:teuthology.orchestra.run.smithi118.stderr:2024-08-03T20:47:19.997+0000 7f0d9cc00640 1 -- 172.21.15.118:0/2362089564 <== mon.1 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d88010860 con 0x7f0da00cf9c0 2024-08-03T20:47:20.000 INFO:teuthology.orchestra.run.smithi118.stderr:2024-08-03T20:47:19.997+0000 7f0da7000640 1 -- 172.21.15.118:0/2362089564 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0da00b1e20 con 0x7f0da00cf9c0 2024-08-03T20:47:20.000 INFO:teuthology.orchestra.run.smithi118.stderr:2024-08-03T20:47:19.997+0000 7f0da7000640 1 -- 172.21.15.118:0/2362089564 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0da00b23e0 con 0x7f0da00cf9c0 2024-08-03T20:47:20.000 INFO:teuthology.orchestra.run.smithi118.stderr:2024-08-03T20:47:19.997+0000 7f0d9cc00640 1 -- 172.21.15.118:0/2362089564 <== mon.1 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f0d88010a00 con 0x7f0da00cf9c0 2024-08-03T20:47:20.000 INFO:teuthology.orchestra.run.smithi118.stderr:2024-08-03T20:47:19.998+0000 7f0d9cc00640 1 -- 172.21.15.118:0/2362089564 <== mon.1 v2:172.21.15.118:3300/0 5 ==== osd_map(9..9 src has 1..9) ==== 1530+0+0 (secure 0 0 0) 0x7f0d88010cf0 con 0x7f0da00cf9c0 2024-08-03T20:47:20.000 INFO:teuthology.orchestra.run.smithi118.stderr:2024-08-03T20:47:19.998+0000 7f0da7000640 1 -- 172.21.15.118:0/2362089564 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0da00d2fe0 con 0x7f0da00cf9c0 2024-08-03T20:47:20.001 INFO:teuthology.orchestra.run.smithi118.stderr:2024-08-03T20:47:20.001+0000 7f0d9cc00640 1 -- 172.21.15.118:0/2362089564 <== mon.1 v2:172.21.15.118:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+74906 (secure 0 0 0) 0x7f0d88018c40 con 0x7f0da00cf9c0 2024-08-03T20:47:20.078 INFO:teuthology.orchestra.run.smithi118.stderr:2024-08-03T20:47:20.077+0000 7f0da7000640 1 -- 172.21.15.118:0/2362089564 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command({"prefix": "osd new", "uuid": "f2086c47-875e-4451-a436-2008d1ea16af", "id": 7} v 0) -- 0x7f0da0161b70 con 0x7f0da00cf9c0 2024-08-03T20:47:20.082 INFO:teuthology.orchestra.run.smithi118.stderr:2024-08-03T20:47:20.081+0000 7f0d9cc00640 1 -- 172.21.15.118:0/2362089564 <== mon.1 v2:172.21.15.118:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "f2086c47-875e-4451-a436-2008d1ea16af", "id": 7}]=0 v10) ==== 112+0+2 (secure 0 0 0) 0x7f0d88018e20 con 0x7f0da00cf9c0 2024-08-03T20:47:20.082 INFO:teuthology.orchestra.run.smithi118.stdout:7 2024-08-03T20:47:20.083 INFO:teuthology.orchestra.run.smithi118.stderr:2024-08-03T20:47:20.082+0000 7f0da7000640 1 -- 172.21.15.118:0/2362089564 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f0da00cf9c0 msgr2=0x7f0da014be60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:47:20.083 INFO:teuthology.orchestra.run.smithi118.stderr:2024-08-03T20:47:20.082+0000 7f0da7000640 1 --2- 172.21.15.118:0/2362089564 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f0da00cf9c0 0x7f0da014be60 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f0d8800e7c0 tx=0x7f0d8800ec80 comp rx=0 tx=0).stop 2024-08-03T20:47:20.084 INFO:teuthology.orchestra.run.smithi118.stderr:2024-08-03T20:47:20.082+0000 7f0da7000640 1 -- 172.21.15.118:0/2362089564 shutdown_connections 2024-08-03T20:47:20.084 INFO:teuthology.orchestra.run.smithi118.stderr:2024-08-03T20:47:20.082+0000 7f0da7000640 1 --2- 172.21.15.118:0/2362089564 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f0da00cf9c0 0x7f0da014be60 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:20.084 INFO:teuthology.orchestra.run.smithi118.stderr:2024-08-03T20:47:20.082+0000 7f0da7000640 1 --2- 172.21.15.118:0/2362089564 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f0da00d0fb0 0x7f0da014c3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:20.084 INFO:teuthology.orchestra.run.smithi118.stderr:2024-08-03T20:47:20.082+0000 7f0da7000640 1 --2- 172.21.15.118:0/2362089564 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0da00b59f0 0x7f0da01577f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:20.084 INFO:teuthology.orchestra.run.smithi118.stderr:2024-08-03T20:47:20.082+0000 7f0da7000640 1 -- 172.21.15.118:0/2362089564 >> 172.21.15.118:0/2362089564 conn(0x7f0da009d660 msgr2=0x7f0da00cefb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:47:20.084 INFO:teuthology.orchestra.run.smithi118.stderr:2024-08-03T20:47:20.082+0000 7f0da7000640 1 -- 172.21.15.118:0/2362089564 shutdown_connections 2024-08-03T20:47:20.084 INFO:teuthology.orchestra.run.smithi118.stderr:2024-08-03T20:47:20.082+0000 7f0da7000640 1 -- 172.21.15.118:0/2362089564 wait complete. 2024-08-03T20:47:20.107 INFO:tasks.ceph.osd.0:Restarting daemon 2024-08-03T20:47:20.108 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2024-08-03T20:47:20.112 INFO:tasks.ceph.osd.0:Started 2024-08-03T20:47:20.112 INFO:tasks.ceph.osd.1:Restarting daemon 2024-08-03T20:47:20.113 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2024-08-03T20:47:20.116 INFO:tasks.ceph.osd.1:Started 2024-08-03T20:47:20.117 INFO:tasks.ceph.osd.2:Restarting daemon 2024-08-03T20:47:20.117 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2024-08-03T20:47:20.120 INFO:tasks.ceph.osd.2:Started 2024-08-03T20:47:20.120 INFO:tasks.ceph.osd.3:Restarting daemon 2024-08-03T20:47:20.121 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2024-08-03T20:47:20.125 INFO:tasks.ceph.osd.3:Started 2024-08-03T20:47:20.125 INFO:tasks.ceph.osd.4:Restarting daemon 2024-08-03T20:47:20.125 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 4 2024-08-03T20:47:20.129 INFO:tasks.ceph.osd.4:Started 2024-08-03T20:47:20.129 INFO:tasks.ceph.osd.5:Restarting daemon 2024-08-03T20:47:20.129 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 5 2024-08-03T20:47:20.133 INFO:tasks.ceph.osd.5:Started 2024-08-03T20:47:20.133 INFO:tasks.ceph.osd.6:Restarting daemon 2024-08-03T20:47:20.133 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 6 2024-08-03T20:47:20.137 INFO:tasks.ceph.osd.6:Started 2024-08-03T20:47:20.137 INFO:tasks.ceph.osd.7:Restarting daemon 2024-08-03T20:47:20.137 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 7 2024-08-03T20:47:20.141 INFO:tasks.ceph.osd.7:Started 2024-08-03T20:47:20.142 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-08-03T20:47:20.470 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:20.469+0000 7f2e6b000640 1 -- 172.21.15.32:0/3335801533 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2e640cffe0 msgr2=0x7f2e640b3e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:47:20.470 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:20.469+0000 7f2e6b000640 1 --2- 172.21.15.32:0/3335801533 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2e640cffe0 0x7f2e640b3e90 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f2e50005fd0 tx=0x7f2e5002a5d0 comp rx=0 tx=0).stop 2024-08-03T20:47:20.470 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:20.469+0000 7f2e6b000640 1 -- 172.21.15.32:0/3335801533 shutdown_connections 2024-08-03T20:47:20.470 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:20.469+0000 7f2e6b000640 1 --2- 172.21.15.32:0/3335801533 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2e640cffe0 0x7f2e640b3e90 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:20.470 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:20.469+0000 7f2e6b000640 1 --2- 172.21.15.32:0/3335801533 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f2e640d26e0 0x7f2e64161b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:20.470 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:20.469+0000 7f2e6b000640 1 -- 172.21.15.32:0/3335801533 >> 172.21.15.32:0/3335801533 conn(0x7f2e6408d7a0 msgr2=0x7f2e6409d3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:47:20.470 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:20.469+0000 7f2e6b000640 1 -- 172.21.15.32:0/3335801533 shutdown_connections 2024-08-03T20:47:20.470 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:20.469+0000 7f2e6b000640 1 -- 172.21.15.32:0/3335801533 wait complete. 2024-08-03T20:47:20.472 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:20.471+0000 7f2e6b000640 1 Processor -- start 2024-08-03T20:47:20.472 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:20.472+0000 7f2e6b000640 1 -- start start 2024-08-03T20:47:20.475 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:20.473+0000 7f2e6b000640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f2e640b43d0 0x7f2e6415e370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:47:20.475 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:20.473+0000 7f2e6b000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2e640cffe0 0x7f2e640c8160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:47:20.475 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:20.473+0000 7f2e6b000640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f2e640d26e0 0x7f2e640c86a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:47:20.475 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:20.473+0000 7f2e6b000640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f2e640d4310 con 0x7f2e640cffe0 2024-08-03T20:47:20.475 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:20.473+0000 7f2e6b000640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f2e640d4190 con 0x7f2e640d26e0 2024-08-03T20:47:20.475 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:20.473+0000 7f2e6b000640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7f2e640d4490 con 0x7f2e640b43d0 2024-08-03T20:47:20.476 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:20.473+0000 7f2e69200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2e640cffe0 0x7f2e640c8160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:47:20.476 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:20.473+0000 7f2e69200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2e640cffe0 0x7f2e640c8160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:51666/0 (socket says 172.21.15.32:51666) 2024-08-03T20:47:20.476 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:20.473+0000 7f2e69200640 1 -- 172.21.15.32:0/2359503583 learned_addr learned my addr 172.21.15.32:0/2359503583 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:47:20.476 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:20.474+0000 7f2e69c00640 1 --2- 172.21.15.32:0/2359503583 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f2e640b43d0 0x7f2e6415e370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:47:20.476 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:20.474+0000 7f2e6a600640 1 --2- 172.21.15.32:0/2359503583 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f2e640d26e0 0x7f2e640c86a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:47:20.476 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:20.474+0000 7f2e69200640 1 -- 172.21.15.32:0/2359503583 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f2e640b43d0 msgr2=0x7f2e6415e370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:47:20.476 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:20.474+0000 7f2e69200640 1 --2- 172.21.15.32:0/2359503583 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f2e640b43d0 0x7f2e6415e370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:20.476 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:20.474+0000 7f2e69200640 1 -- 172.21.15.32:0/2359503583 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f2e640d26e0 msgr2=0x7f2e640c86a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:47:20.476 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:20.474+0000 7f2e69200640 1 --2- 172.21.15.32:0/2359503583 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f2e640d26e0 0x7f2e640c86a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:20.476 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:20.474+0000 7f2e69200640 1 -- 172.21.15.32:0/2359503583 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2e640c8e70 con 0x7f2e640cffe0 2024-08-03T20:47:20.476 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:20.474+0000 7f2e69200640 1 --2- 172.21.15.32:0/2359503583 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2e640cffe0 0x7f2e640c8160 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f2e50034040 tx=0x7f2e500038a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:47:20.476 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:20.475+0000 7f2e61600640 1 -- 172.21.15.32:0/2359503583 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e5003e030 con 0x7f2e640cffe0 2024-08-03T20:47:20.476 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:20.475+0000 7f2e61600640 1 -- 172.21.15.32:0/2359503583 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f2e50003ca0 con 0x7f2e640cffe0 2024-08-03T20:47:20.476 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:20.475+0000 7f2e61600640 1 -- 172.21.15.32:0/2359503583 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e50035040 con 0x7f2e640cffe0 2024-08-03T20:47:20.477 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:20.475+0000 7f2e6b000640 1 -- 172.21.15.32:0/2359503583 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2e6415e8b0 con 0x7f2e640cffe0 2024-08-03T20:47:20.477 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:20.475+0000 7f2e6b000640 1 -- 172.21.15.32:0/2359503583 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2e641f5a70 con 0x7f2e640cffe0 2024-08-03T20:47:20.477 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:20.476+0000 7f2e6b000640 1 -- 172.21.15.32:0/2359503583 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2e640c8be0 con 0x7f2e640cffe0 2024-08-03T20:47:20.477 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:20.476+0000 7f2e61600640 1 -- 172.21.15.32:0/2359503583 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 2) ==== 49591+0+0 (secure 0 0 0) 0x7f2e50041030 con 0x7f2e640cffe0 2024-08-03T20:47:20.477 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:20.476+0000 7f2e61600640 1 -- 172.21.15.32:0/2359503583 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(10..10 src has 1..10) ==== 1645+0+0 (secure 0 0 0) 0x7f2e50071750 con 0x7f2e640cffe0 2024-08-03T20:47:20.478 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:20.478+0000 7f2e61600640 1 -- 172.21.15.32:0/2359503583 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+74906 (secure 0 0 0) 0x7f2e6415e8b0 con 0x7f2e640cffe0 2024-08-03T20:47:20.545 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:20.545+0000 7f2e6b000640 1 -- 172.21.15.32:0/2359503583 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f2e640d4940 con 0x7f2e640cffe0 2024-08-03T20:47:20.546 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:20.546+0000 7f2e61600640 1 -- 172.21.15.32:0/2359503583 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) ==== 74+0+7058 (secure 0 0 0) 0x7f2e640c8e70 con 0x7f2e640cffe0 2024-08-03T20:47:20.546 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-03T20:47:20.547 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":10,"fsid":"40ab141a-01d7-49b9-ae3c-2edec9876e95","created":"2024-08-03T20:47:14.957362+0000","modified":"2024-08-03T20:47:20.079442+0000","last_up_change":"0.000000","last_in_change":"2024-08-03T20:47:20.079442+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"5a83e759-9035-438b-889a-0a6af1f0a0d2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"d7c5c210-da4f-49f9-b9e7-1dfb02daa4bb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"57f2e6de-a201-45c2-8981-8e56a4875ebd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"6439a280-da45-4148-8130-f16323935ff3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"4f902eff-1e18-418d-82c9-2151c69524fa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bfd9feac-dd06-458c-91b9-af7f175f612c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"5ed5a279-9d8e-416d-bbc7-7edd3e6d5705","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"f2086c47-875e-4451-a436-2008d1ea16af","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-08-03T20:47:20.548 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:20.547+0000 7f2e6b000640 1 -- 172.21.15.32:0/2359503583 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2e640cffe0 msgr2=0x7f2e640c8160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:47:20.548 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:20.547+0000 7f2e6b000640 1 --2- 172.21.15.32:0/2359503583 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2e640cffe0 0x7f2e640c8160 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f2e50034040 tx=0x7f2e500038a0 comp rx=0 tx=0).stop 2024-08-03T20:47:20.548 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:20.547+0000 7f2e6b000640 1 -- 172.21.15.32:0/2359503583 shutdown_connections 2024-08-03T20:47:20.548 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:20.547+0000 7f2e6b000640 1 --2- 172.21.15.32:0/2359503583 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f2e640d26e0 0x7f2e640c86a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:20.548 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:20.547+0000 7f2e6b000640 1 --2- 172.21.15.32:0/2359503583 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2e640cffe0 0x7f2e640c8160 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:20.548 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:20.547+0000 7f2e6b000640 1 --2- 172.21.15.32:0/2359503583 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f2e640b43d0 0x7f2e6415e370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:20.548 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:20.547+0000 7f2e6b000640 1 -- 172.21.15.32:0/2359503583 >> 172.21.15.32:0/2359503583 conn(0x7f2e6408d7a0 msgr2=0x7f2e640b5a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:47:20.548 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:20.547+0000 7f2e6b000640 1 -- 172.21.15.32:0/2359503583 shutdown_connections 2024-08-03T20:47:20.548 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:20.547+0000 7f2e6b000640 1 -- 172.21.15.32:0/2359503583 wait complete. 2024-08-03T20:47:20.569 INFO:tasks.ceph.ceph_manager.ceph:[] 2024-08-03T20:47:20.569 INFO:tasks.ceph:Waiting for OSDs to come up 2024-08-03T20:47:20.875 INFO:tasks.ceph.osd.4.smithi118.stderr:2024-08-03T20:47:20.874+0000 7f7def23a640 -1 Falling back to public interface 2024-08-03T20:47:20.876 INFO:tasks.ceph.osd.3.smithi032.stderr:2024-08-03T20:47:20.875+0000 7f11ae615640 -1 Falling back to public interface 2024-08-03T20:47:20.889 INFO:tasks.ceph.osd.5.smithi118.stderr:2024-08-03T20:47:20.888+0000 7f1b64a3f640 -1 Falling back to public interface 2024-08-03T20:47:20.897 INFO:tasks.ceph.osd.2.smithi032.stderr:2024-08-03T20:47:20.897+0000 7f601b2ff640 -1 Falling back to public interface 2024-08-03T20:47:20.898 INFO:tasks.ceph.osd.7.smithi118.stderr:2024-08-03T20:47:20.897+0000 7f557fa1a640 -1 Falling back to public interface 2024-08-03T20:47:20.904 INFO:tasks.ceph.osd.6.smithi118.stderr:2024-08-03T20:47:20.903+0000 7f7e5ba12640 -1 Falling back to public interface 2024-08-03T20:47:20.921 INFO:tasks.ceph.osd.0.smithi032.stderr:2024-08-03T20:47:20.921+0000 7f859b23d640 -1 Falling back to public interface 2024-08-03T20:47:20.959 INFO:tasks.ceph.osd.1.smithi032.stderr:2024-08-03T20:47:20.959+0000 7f4275e32640 -1 Falling back to public interface 2024-08-03T20:47:21.226 INFO:tasks.ceph.mgr.x.smithi118.stderr:2024-08-03T20:47:21.225+0000 7fecbf800640 -1 mgr.server handle_report got status from non-daemon mon.c 2024-08-03T20:47:21.227 INFO:tasks.ceph.mgr.x.smithi118.stderr:2024-08-03T20:47:21.225+0000 7fecbf800640 -1 mgr.server handle_report got status from non-daemon mon.b 2024-08-03T20:47:21.374 DEBUG:teuthology.orchestra.run.smithi032:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-08-03T20:47:21.585 INFO:teuthology.misc.health.smithi032.stderr:2024-08-03T20:47:21.584+0000 7fea72600640 1 -- 172.21.15.32:0/3726580304 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fea6c0d4d80 msgr2=0x7fea6c0b4240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:47:21.585 INFO:teuthology.misc.health.smithi032.stderr:2024-08-03T20:47:21.584+0000 7fea72600640 1 --2- 172.21.15.32:0/3726580304 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fea6c0d4d80 0x7fea6c0b4240 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fea64005fd0 tx=0x7fea6402a840 comp rx=0 tx=0).stop 2024-08-03T20:47:21.585 INFO:teuthology.misc.health.smithi032.stderr:2024-08-03T20:47:21.584+0000 7fea72600640 1 -- 172.21.15.32:0/3726580304 shutdown_connections 2024-08-03T20:47:21.585 INFO:teuthology.misc.health.smithi032.stderr:2024-08-03T20:47:21.584+0000 7fea72600640 1 --2- 172.21.15.32:0/3726580304 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fea6c0d4d80 0x7fea6c0b4240 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:21.585 INFO:teuthology.misc.health.smithi032.stderr:2024-08-03T20:47:21.584+0000 7fea72600640 1 --2- 172.21.15.32:0/3726580304 >> v2:172.21.15.118:3300/0 conn(0x7fea6c0b4d90 0x7fea6c0d0d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:21.585 INFO:teuthology.misc.health.smithi032.stderr:2024-08-03T20:47:21.584+0000 7fea72600640 1 -- 172.21.15.32:0/3726580304 >> 172.21.15.32:0/3726580304 conn(0x7fea6c0d41d0 msgr2=0x7fea6c0d45e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:47:21.586 INFO:teuthology.misc.health.smithi032.stderr:2024-08-03T20:47:21.584+0000 7fea72600640 1 -- 172.21.15.32:0/3726580304 shutdown_connections 2024-08-03T20:47:21.586 INFO:teuthology.misc.health.smithi032.stderr:2024-08-03T20:47:21.584+0000 7fea72600640 1 -- 172.21.15.32:0/3726580304 wait complete. 2024-08-03T20:47:21.587 INFO:teuthology.misc.health.smithi032.stderr:2024-08-03T20:47:21.586+0000 7fea72600640 1 Processor -- start 2024-08-03T20:47:21.587 INFO:teuthology.misc.health.smithi032.stderr:2024-08-03T20:47:21.587+0000 7fea72600640 1 -- start start 2024-08-03T20:47:21.590 INFO:teuthology.misc.health.smithi032.stderr:2024-08-03T20:47:21.588+0000 7fea72600640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fea6c0d1270 0x7fea6c168790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:47:21.590 INFO:teuthology.misc.health.smithi032.stderr:2024-08-03T20:47:21.588+0000 7fea72600640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fea6c0d4d80 0x7fea6c160bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:47:21.590 INFO:teuthology.misc.health.smithi032.stderr:2024-08-03T20:47:21.588+0000 7fea72600640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fea6c166460 0x7fea6c0cd450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:47:21.590 INFO:teuthology.misc.health.smithi032.stderr:2024-08-03T20:47:21.588+0000 7fea72600640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fea6c0b5340 con 0x7fea6c0d4d80 2024-08-03T20:47:21.590 INFO:teuthology.misc.health.smithi032.stderr:2024-08-03T20:47:21.588+0000 7fea72600640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7fea6c0b4990 con 0x7fea6c0d1270 2024-08-03T20:47:21.590 INFO:teuthology.misc.health.smithi032.stderr:2024-08-03T20:47:21.588+0000 7fea72600640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7fea6c0b54c0 con 0x7fea6c166460 2024-08-03T20:47:21.590 INFO:teuthology.misc.health.smithi032.stderr:2024-08-03T20:47:21.588+0000 7fea63e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fea6c0d4d80 0x7fea6c160bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:47:21.590 INFO:teuthology.misc.health.smithi032.stderr:2024-08-03T20:47:21.588+0000 7fea63e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fea6c0d4d80 0x7fea6c160bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:51676/0 (socket says 172.21.15.32:51676) 2024-08-03T20:47:21.591 INFO:teuthology.misc.health.smithi032.stderr:2024-08-03T20:47:21.588+0000 7fea63e00640 1 -- 172.21.15.32:0/2196379504 learned_addr learned my addr 172.21.15.32:0/2196379504 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:47:21.591 INFO:teuthology.misc.health.smithi032.stderr:2024-08-03T20:47:21.588+0000 7fea71200640 1 --2- 172.21.15.32:0/2196379504 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fea6c0d1270 0x7fea6c168790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:47:21.591 INFO:teuthology.misc.health.smithi032.stderr:2024-08-03T20:47:21.588+0000 7fea71c00640 1 --2- 172.21.15.32:0/2196379504 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fea6c166460 0x7fea6c0cd450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:47:21.591 INFO:teuthology.misc.health.smithi032.stderr:2024-08-03T20:47:21.589+0000 7fea63e00640 1 -- 172.21.15.32:0/2196379504 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fea6c166460 msgr2=0x7fea6c0cd450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:47:21.591 INFO:teuthology.misc.health.smithi032.stderr:2024-08-03T20:47:21.589+0000 7fea63e00640 1 --2- 172.21.15.32:0/2196379504 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fea6c166460 0x7fea6c0cd450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:21.591 INFO:teuthology.misc.health.smithi032.stderr:2024-08-03T20:47:21.589+0000 7fea63e00640 1 -- 172.21.15.32:0/2196379504 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fea6c0d1270 msgr2=0x7fea6c168790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:47:21.591 INFO:teuthology.misc.health.smithi032.stderr:2024-08-03T20:47:21.589+0000 7fea63e00640 1 --2- 172.21.15.32:0/2196379504 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fea6c0d1270 0x7fea6c168790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:21.591 INFO:teuthology.misc.health.smithi032.stderr:2024-08-03T20:47:21.589+0000 7fea63e00640 1 -- 172.21.15.32:0/2196379504 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fea6c162450 con 0x7fea6c0d4d80 2024-08-03T20:47:21.591 INFO:teuthology.misc.health.smithi032.stderr:2024-08-03T20:47:21.589+0000 7fea63e00640 1 --2- 172.21.15.32:0/2196379504 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fea6c0d4d80 0x7fea6c160bf0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fea5400c530 tx=0x7fea5400c9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:47:21.591 INFO:teuthology.misc.health.smithi032.stderr:2024-08-03T20:47:21.589+0000 7fea69600640 1 -- 172.21.15.32:0/2196379504 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea54016020 con 0x7fea6c0d4d80 2024-08-03T20:47:21.591 INFO:teuthology.misc.health.smithi032.stderr:2024-08-03T20:47:21.589+0000 7fea69600640 1 -- 172.21.15.32:0/2196379504 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fea54012940 con 0x7fea6c0d4d80 2024-08-03T20:47:21.591 INFO:teuthology.misc.health.smithi032.stderr:2024-08-03T20:47:21.589+0000 7fea72600640 1 -- 172.21.15.32:0/2196379504 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fea6c0cb3b0 con 0x7fea6c0d4d80 2024-08-03T20:47:21.591 INFO:teuthology.misc.health.smithi032.stderr:2024-08-03T20:47:21.589+0000 7fea69600640 1 -- 172.21.15.32:0/2196379504 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea54012d90 con 0x7fea6c0d4d80 2024-08-03T20:47:21.592 INFO:teuthology.misc.health.smithi032.stderr:2024-08-03T20:47:21.589+0000 7fea72600640 1 -- 172.21.15.32:0/2196379504 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fea6c166910 con 0x7fea6c0d4d80 2024-08-03T20:47:21.592 INFO:teuthology.misc.health.smithi032.stderr:2024-08-03T20:47:21.590+0000 7fea72600640 1 -- 172.21.15.32:0/2196379504 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fea6c0b4d90 con 0x7fea6c0d4d80 2024-08-03T20:47:21.593 INFO:teuthology.misc.health.smithi032.stderr:2024-08-03T20:47:21.591+0000 7fea69600640 1 -- 172.21.15.32:0/2196379504 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) ==== 98977+0+0 (secure 0 0 0) 0x7fea54019020 con 0x7fea6c0d4d80 2024-08-03T20:47:21.593 INFO:teuthology.misc.health.smithi032.stderr:2024-08-03T20:47:21.592+0000 7fea69600640 1 --2- 172.21.15.32:0/2196379504 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fea48076e60 0x7fea48079370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:47:21.593 INFO:teuthology.misc.health.smithi032.stderr:2024-08-03T20:47:21.592+0000 7fea69600640 1 -- 172.21.15.32:0/2196379504 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(10..10 src has 1..10) ==== 1645+0+0 (secure 0 0 0) 0x7fea54097510 con 0x7fea6c0d4d80 2024-08-03T20:47:21.593 INFO:teuthology.misc.health.smithi032.stderr:2024-08-03T20:47:21.593+0000 7fea71200640 1 --2- 172.21.15.32:0/2196379504 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fea48076e60 0x7fea48079370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:47:21.594 INFO:teuthology.misc.health.smithi032.stderr:2024-08-03T20:47:21.593+0000 7fea71200640 1 --2- 172.21.15.32:0/2196379504 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fea48076e60 0x7fea48079370 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fea5c009590 tx=0x7fea5c005f30 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:47:21.597 INFO:teuthology.misc.health.smithi032.stderr:2024-08-03T20:47:21.597+0000 7fea69600640 1 -- 172.21.15.32:0/2196379504 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7fea54020020 con 0x7fea6c0d4d80 2024-08-03T20:47:21.801 INFO:teuthology.misc.health.smithi032.stderr:2024-08-03T20:47:21.801+0000 7fea72600640 1 -- 172.21.15.32:0/2196379504 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fea6c1621e0 con 0x7fea6c0d4d80 2024-08-03T20:47:21.802 INFO:teuthology.misc.health.smithi032.stderr:2024-08-03T20:47:21.802+0000 7fea69600640 1 -- 172.21.15.32:0/2196379504 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) ==== 74+0+7058 (secure 0 0 0) 0x7fea6c0d32d0 con 0x7fea6c0d4d80 2024-08-03T20:47:21.802 INFO:teuthology.misc.health.smithi032.stdout: 2024-08-03T20:47:21.802 INFO:teuthology.misc.health.smithi032.stdout:{"epoch":10,"fsid":"40ab141a-01d7-49b9-ae3c-2edec9876e95","created":"2024-08-03T20:47:14.957362+0000","modified":"2024-08-03T20:47:20.079442+0000","last_up_change":"0.000000","last_in_change":"2024-08-03T20:47:20.079442+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"5a83e759-9035-438b-889a-0a6af1f0a0d2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"d7c5c210-da4f-49f9-b9e7-1dfb02daa4bb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"57f2e6de-a201-45c2-8981-8e56a4875ebd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"6439a280-da45-4148-8130-f16323935ff3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"4f902eff-1e18-418d-82c9-2151c69524fa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bfd9feac-dd06-458c-91b9-af7f175f612c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"5ed5a279-9d8e-416d-bbc7-7edd3e6d5705","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"f2086c47-875e-4451-a436-2008d1ea16af","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-08-03T20:47:21.805 INFO:teuthology.misc.health.smithi032.stderr:2024-08-03T20:47:21.804+0000 7fea72600640 1 -- 172.21.15.32:0/2196379504 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fea48076e60 msgr2=0x7fea48079370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:47:21.805 INFO:teuthology.misc.health.smithi032.stderr:2024-08-03T20:47:21.804+0000 7fea72600640 1 --2- 172.21.15.32:0/2196379504 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fea48076e60 0x7fea48079370 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fea5c009590 tx=0x7fea5c005f30 comp rx=0 tx=0).stop 2024-08-03T20:47:21.805 INFO:teuthology.misc.health.smithi032.stderr:2024-08-03T20:47:21.804+0000 7fea72600640 1 -- 172.21.15.32:0/2196379504 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fea6c0d4d80 msgr2=0x7fea6c160bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:47:21.805 INFO:teuthology.misc.health.smithi032.stderr:2024-08-03T20:47:21.804+0000 7fea72600640 1 --2- 172.21.15.32:0/2196379504 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fea6c0d4d80 0x7fea6c160bf0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fea5400c530 tx=0x7fea5400c9f0 comp rx=0 tx=0).stop 2024-08-03T20:47:21.805 INFO:teuthology.misc.health.smithi032.stderr:2024-08-03T20:47:21.804+0000 7fea72600640 1 -- 172.21.15.32:0/2196379504 shutdown_connections 2024-08-03T20:47:21.805 INFO:teuthology.misc.health.smithi032.stderr:2024-08-03T20:47:21.804+0000 7fea72600640 1 --2- 172.21.15.32:0/2196379504 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fea48076e60 0x7fea48079370 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:21.805 INFO:teuthology.misc.health.smithi032.stderr:2024-08-03T20:47:21.804+0000 7fea72600640 1 --2- 172.21.15.32:0/2196379504 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fea6c166460 0x7fea6c0cd450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:21.805 INFO:teuthology.misc.health.smithi032.stderr:2024-08-03T20:47:21.804+0000 7fea72600640 1 --2- 172.21.15.32:0/2196379504 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fea6c0d4d80 0x7fea6c160bf0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:21.806 INFO:teuthology.misc.health.smithi032.stderr:2024-08-03T20:47:21.804+0000 7fea72600640 1 --2- 172.21.15.32:0/2196379504 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fea6c0d1270 0x7fea6c168790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:21.806 INFO:teuthology.misc.health.smithi032.stderr:2024-08-03T20:47:21.804+0000 7fea72600640 1 -- 172.21.15.32:0/2196379504 >> 172.21.15.32:0/2196379504 conn(0x7fea6c0d41d0 msgr2=0x7fea6c165cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:47:21.806 INFO:teuthology.misc.health.smithi032.stderr:2024-08-03T20:47:21.804+0000 7fea72600640 1 -- 172.21.15.32:0/2196379504 shutdown_connections 2024-08-03T20:47:21.806 INFO:teuthology.misc.health.smithi032.stderr:2024-08-03T20:47:21.804+0000 7fea72600640 1 -- 172.21.15.32:0/2196379504 wait complete. 2024-08-03T20:47:21.827 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-08-03T20:47:24.401 INFO:tasks.ceph.osd.3.smithi032.stderr:2024-08-03T20:47:24.401+0000 7f11ae615640 -1 osd.3 0 log_to_monitors true 2024-08-03T20:47:24.405 INFO:tasks.ceph.osd.4.smithi118.stderr:2024-08-03T20:47:24.404+0000 7f7def23a640 -1 osd.4 0 log_to_monitors true 2024-08-03T20:47:24.446 INFO:tasks.ceph.osd.2.smithi032.stderr:2024-08-03T20:47:24.445+0000 7f601b2ff640 -1 osd.2 0 log_to_monitors true 2024-08-03T20:47:24.508 INFO:tasks.ceph.osd.7.smithi118.stderr:2024-08-03T20:47:24.507+0000 7f557fa1a640 -1 osd.7 0 log_to_monitors true 2024-08-03T20:47:24.517 INFO:tasks.ceph.osd.0.smithi032.stderr:2024-08-03T20:47:24.516+0000 7f859b23d640 -1 osd.0 0 log_to_monitors true 2024-08-03T20:47:24.546 INFO:tasks.ceph.osd.5.smithi118.stderr:2024-08-03T20:47:24.545+0000 7f1b64a3f640 -1 osd.5 0 log_to_monitors true 2024-08-03T20:47:24.547 INFO:tasks.ceph.osd.6.smithi118.stderr:2024-08-03T20:47:24.546+0000 7f7e5ba12640 -1 osd.6 0 log_to_monitors true 2024-08-03T20:47:24.571 INFO:tasks.ceph.osd.1.smithi032.stderr:2024-08-03T20:47:24.571+0000 7f4275e32640 -1 osd.1 0 log_to_monitors true 2024-08-03T20:47:26.474 INFO:tasks.ceph.osd.4.smithi118.stderr:2024-08-03T20:47:26.472+0000 7f7deaa00640 -1 osd.4 0 waiting for initial osdmap 2024-08-03T20:47:26.486 INFO:tasks.ceph.osd.4.smithi118.stderr:2024-08-03T20:47:26.484+0000 7f7de0000640 -1 osd.4 13 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-08-03T20:47:26.836 INFO:tasks.ceph.osd.3.smithi032.stderr:2024-08-03T20:47:26.836+0000 7f11a8e00640 -1 osd.3 0 waiting for initial osdmap 2024-08-03T20:47:26.843 INFO:tasks.ceph.osd.2.smithi032.stderr:2024-08-03T20:47:26.843+0000 7f6016200640 -1 osd.2 0 waiting for initial osdmap 2024-08-03T20:47:26.844 INFO:tasks.ceph.osd.3.smithi032.stderr:2024-08-03T20:47:26.844+0000 7f119ee00640 -1 osd.3 13 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-08-03T20:47:26.852 INFO:tasks.ceph.osd.2.smithi032.stderr:2024-08-03T20:47:26.852+0000 7f600c200640 -1 osd.2 13 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-08-03T20:47:27.872 INFO:tasks.ceph.osd.0.smithi032.stderr:2024-08-03T20:47:27.872+0000 7f8596a00640 -1 osd.0 0 waiting for initial osdmap 2024-08-03T20:47:27.879 INFO:tasks.ceph.osd.0.smithi032.stderr:2024-08-03T20:47:27.879+0000 7f858c000640 -1 osd.0 14 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-08-03T20:47:27.923 INFO:tasks.ceph.osd.1.smithi032.stderr:2024-08-03T20:47:27.922+0000 7f4270c00640 -1 osd.1 0 waiting for initial osdmap 2024-08-03T20:47:27.929 INFO:tasks.ceph.osd.1.smithi032.stderr:2024-08-03T20:47:27.929+0000 7f4266c00640 -1 osd.1 14 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-08-03T20:47:27.969 INFO:tasks.ceph.osd.6.smithi118.stderr:2024-08-03T20:47:27.968+0000 7f7e56400640 -1 osd.6 0 waiting for initial osdmap 2024-08-03T20:47:27.976 INFO:tasks.ceph.osd.6.smithi118.stderr:2024-08-03T20:47:27.975+0000 7f7e4c400640 -1 osd.6 14 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-08-03T20:47:27.998 INFO:tasks.ceph.osd.7.smithi118.stderr:2024-08-03T20:47:27.997+0000 7f557a400640 -1 osd.7 0 waiting for initial osdmap 2024-08-03T20:47:28.008 INFO:tasks.ceph.osd.7.smithi118.stderr:2024-08-03T20:47:28.007+0000 7f5570400640 -1 osd.7 14 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-08-03T20:47:28.018 INFO:tasks.ceph.osd.5.smithi118.stderr:2024-08-03T20:47:28.018+0000 7f1b60200640 -1 osd.5 0 waiting for initial osdmap 2024-08-03T20:47:28.023 INFO:tasks.ceph.osd.5.smithi118.stderr:2024-08-03T20:47:28.022+0000 7f1b55800640 -1 osd.5 14 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-08-03T20:47:28.633 DEBUG:teuthology.orchestra.run.smithi032:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-08-03T20:47:28.850 INFO:teuthology.misc.health.smithi032.stderr:2024-08-03T20:47:28.849+0000 7f6752600640 1 -- 172.21.15.32:0/1575064320 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f674c15e810 msgr2=0x7f674c0d08a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:47:28.850 INFO:teuthology.misc.health.smithi032.stderr:2024-08-03T20:47:28.849+0000 7f6752600640 1 --2- 172.21.15.32:0/1575064320 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f674c15e810 0x7f674c0d08a0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f673c009830 tx=0x7f673c02a480 comp rx=0 tx=0).stop 2024-08-03T20:47:28.850 INFO:teuthology.misc.health.smithi032.stderr:2024-08-03T20:47:28.849+0000 7f6752600640 1 -- 172.21.15.32:0/1575064320 shutdown_connections 2024-08-03T20:47:28.850 INFO:teuthology.misc.health.smithi032.stderr:2024-08-03T20:47:28.849+0000 7f6752600640 1 --2- 172.21.15.32:0/1575064320 >> v2:172.21.15.118:3300/0 conn(0x7f674c06d0e0 0x7f674c0cc4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:28.851 INFO:teuthology.misc.health.smithi032.stderr:2024-08-03T20:47:28.849+0000 7f6752600640 1 --2- 172.21.15.32:0/1575064320 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f674c15e810 0x7f674c0d08a0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:28.851 INFO:teuthology.misc.health.smithi032.stderr:2024-08-03T20:47:28.849+0000 7f6752600640 1 -- 172.21.15.32:0/1575064320 >> 172.21.15.32:0/1575064320 conn(0x7f674c0cdb00 msgr2=0x7f674c0cdf10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:47:28.851 INFO:teuthology.misc.health.smithi032.stderr:2024-08-03T20:47:28.849+0000 7f6752600640 1 -- 172.21.15.32:0/1575064320 shutdown_connections 2024-08-03T20:47:28.851 INFO:teuthology.misc.health.smithi032.stderr:2024-08-03T20:47:28.849+0000 7f6752600640 1 -- 172.21.15.32:0/1575064320 wait complete. 2024-08-03T20:47:28.852 INFO:teuthology.misc.health.smithi032.stderr:2024-08-03T20:47:28.851+0000 7f6752600640 1 Processor -- start 2024-08-03T20:47:28.852 INFO:teuthology.misc.health.smithi032.stderr:2024-08-03T20:47:28.852+0000 7f6752600640 1 -- start start 2024-08-03T20:47:28.854 INFO:teuthology.misc.health.smithi032.stderr:2024-08-03T20:47:28.853+0000 7f6752600640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f674c0cc0e0 0x7f674c1538d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:47:28.854 INFO:teuthology.misc.health.smithi032.stderr:2024-08-03T20:47:28.853+0000 7f6752600640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f674c15e810 0x7f674c1f8bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:47:28.854 INFO:teuthology.misc.health.smithi032.stderr:2024-08-03T20:47:28.853+0000 7f6752600640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f674c1f9130 0x7f674c0b42a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:47:28.854 INFO:teuthology.misc.health.smithi032.stderr:2024-08-03T20:47:28.853+0000 7f6752600640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f674c070220 con 0x7f674c0cc0e0 2024-08-03T20:47:28.854 INFO:teuthology.misc.health.smithi032.stderr:2024-08-03T20:47:28.853+0000 7f6752600640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f674c0700a0 con 0x7f674c1f9130 2024-08-03T20:47:28.854 INFO:teuthology.misc.health.smithi032.stderr:2024-08-03T20:47:28.853+0000 7f6752600640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7f674c0703a0 con 0x7f674c15e810 2024-08-03T20:47:28.854 INFO:teuthology.misc.health.smithi032.stderr:2024-08-03T20:47:28.853+0000 7f674b400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f674c0cc0e0 0x7f674c1538d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:47:28.854 INFO:teuthology.misc.health.smithi032.stderr:2024-08-03T20:47:28.853+0000 7f674b400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f674c0cc0e0 0x7f674c1538d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:51720/0 (socket says 172.21.15.32:51720) 2024-08-03T20:47:28.854 INFO:teuthology.misc.health.smithi032.stderr:2024-08-03T20:47:28.853+0000 7f674b400640 1 -- 172.21.15.32:0/1031415512 learned_addr learned my addr 172.21.15.32:0/1031415512 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:47:28.855 INFO:teuthology.misc.health.smithi032.stderr:2024-08-03T20:47:28.853+0000 7f674be00640 1 --2- 172.21.15.32:0/1031415512 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f674c1f9130 0x7f674c0b42a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:47:28.855 INFO:teuthology.misc.health.smithi032.stderr:2024-08-03T20:47:28.853+0000 7f674aa00640 1 --2- 172.21.15.32:0/1031415512 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f674c15e810 0x7f674c1f8bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:47:28.856 INFO:teuthology.misc.health.smithi032.stderr:2024-08-03T20:47:28.854+0000 7f674b400640 1 -- 172.21.15.32:0/1031415512 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f674c15e810 msgr2=0x7f674c1f8bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:47:28.856 INFO:teuthology.misc.health.smithi032.stderr:2024-08-03T20:47:28.854+0000 7f674b400640 1 --2- 172.21.15.32:0/1031415512 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f674c15e810 0x7f674c1f8bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:28.856 INFO:teuthology.misc.health.smithi032.stderr:2024-08-03T20:47:28.854+0000 7f674b400640 1 -- 172.21.15.32:0/1031415512 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f674c1f9130 msgr2=0x7f674c0b42a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:47:28.856 INFO:teuthology.misc.health.smithi032.stderr:2024-08-03T20:47:28.854+0000 7f674b400640 1 --2- 172.21.15.32:0/1031415512 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f674c1f9130 0x7f674c0b42a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:28.856 INFO:teuthology.misc.health.smithi032.stderr:2024-08-03T20:47:28.854+0000 7f674b400640 1 -- 172.21.15.32:0/1031415512 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f674c1f9aa0 con 0x7f674c0cc0e0 2024-08-03T20:47:28.856 INFO:teuthology.misc.health.smithi032.stderr:2024-08-03T20:47:28.854+0000 7f674b400640 1 --2- 172.21.15.32:0/1031415512 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f674c0cc0e0 0x7f674c1538d0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f673c000c00 tx=0x7f673c033c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:47:28.856 INFO:teuthology.misc.health.smithi032.stderr:2024-08-03T20:47:28.854+0000 7f6743400640 1 -- 172.21.15.32:0/1031415512 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f673c005c90 con 0x7f674c0cc0e0 2024-08-03T20:47:28.856 INFO:teuthology.misc.health.smithi032.stderr:2024-08-03T20:47:28.854+0000 7f6743400640 1 -- 172.21.15.32:0/1031415512 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f673c031e50 con 0x7f674c0cc0e0 2024-08-03T20:47:28.857 INFO:teuthology.misc.health.smithi032.stderr:2024-08-03T20:47:28.855+0000 7f6752600640 1 -- 172.21.15.32:0/1031415512 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f674c1f9540 con 0x7f674c0cc0e0 2024-08-03T20:47:28.857 INFO:teuthology.misc.health.smithi032.stderr:2024-08-03T20:47:28.855+0000 7f6743400640 1 -- 172.21.15.32:0/1031415512 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f673c033310 con 0x7f674c0cc0e0 2024-08-03T20:47:28.857 INFO:teuthology.misc.health.smithi032.stderr:2024-08-03T20:47:28.855+0000 7f6752600640 1 -- 172.21.15.32:0/1031415512 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f674c1f97c0 con 0x7f674c0cc0e0 2024-08-03T20:47:28.857 INFO:teuthology.misc.health.smithi032.stderr:2024-08-03T20:47:28.856+0000 7f6752600640 1 -- 172.21.15.32:0/1031415512 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f674c070850 con 0x7f674c0cc0e0 2024-08-03T20:47:28.857 INFO:teuthology.misc.health.smithi032.stderr:2024-08-03T20:47:28.857+0000 7f6743400640 1 -- 172.21.15.32:0/1031415512 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 4) ==== 99083+0+0 (secure 0 0 0) 0x7f673c0334b0 con 0x7f674c0cc0e0 2024-08-03T20:47:28.858 INFO:teuthology.misc.health.smithi032.stderr:2024-08-03T20:47:28.858+0000 7f6743400640 1 --2- 172.21.15.32:0/1031415512 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f672c076f80 0x7f672c079490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:47:28.858 INFO:teuthology.misc.health.smithi032.stderr:2024-08-03T20:47:28.858+0000 7f6743400640 1 -- 172.21.15.32:0/1031415512 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(14..14 src has 1..14) ==== 3071+0+0 (secure 0 0 0) 0x7f673c0b9650 con 0x7f674c0cc0e0 2024-08-03T20:47:28.859 INFO:teuthology.misc.health.smithi032.stderr:2024-08-03T20:47:28.859+0000 7f674aa00640 1 --2- 172.21.15.32:0/1031415512 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f672c076f80 0x7f672c079490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:47:28.859 INFO:teuthology.misc.health.smithi032.stderr:2024-08-03T20:47:28.859+0000 7f674aa00640 1 --2- 172.21.15.32:0/1031415512 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f672c076f80 0x7f672c079490 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f6738007fd0 tx=0x7f6738001390 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:47:28.862 INFO:teuthology.misc.health.smithi032.stderr:2024-08-03T20:47:28.862+0000 7f6743400640 1 -- 172.21.15.32:0/1031415512 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7f673c051020 con 0x7f674c0cc0e0 2024-08-03T20:47:29.076 INFO:teuthology.misc.health.smithi032.stderr:2024-08-03T20:47:29.075+0000 7f6752600640 1 -- 172.21.15.32:0/1031415512 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f674c1e9ff0 con 0x7f674c0cc0e0 2024-08-03T20:47:29.076 INFO:teuthology.misc.health.smithi032.stderr:2024-08-03T20:47:29.076+0000 7f6743400640 1 -- 172.21.15.32:0/1031415512 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v15) ==== 74+0+12713 (secure 0 0 0) 0x7f674c1f9540 con 0x7f674c0cc0e0 2024-08-03T20:47:29.077 INFO:teuthology.misc.health.smithi032.stdout: 2024-08-03T20:47:29.077 INFO:teuthology.misc.health.smithi032.stdout:{"epoch":15,"fsid":"40ab141a-01d7-49b9-ae3c-2edec9876e95","created":"2024-08-03T20:47:14.957362+0000","modified":"2024-08-03T20:47:28.874451+0000","last_up_change":"2024-08-03T20:47:28.874451+0000","last_in_change":"2024-08-03T20:47:20.079442+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-08-03T20:47:28.261300+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5a83e759-9035-438b-889a-0a6af1f0a0d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6816","nonce":3066156116},{"type":"v1","addr":"172.21.15.32:6817","nonce":3066156116}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6818","nonce":3066156116},{"type":"v1","addr":"172.21.15.32:6819","nonce":3066156116}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6822","nonce":3066156116},{"type":"v1","addr":"172.21.15.32:6823","nonce":3066156116}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6820","nonce":3066156116},{"type":"v1","addr":"172.21.15.32:6821","nonce":3066156116}]},"public_addr":"172.21.15.32:6817/3066156116","cluster_addr":"172.21.15.32:6819/3066156116","heartbeat_back_addr":"172.21.15.32:6823/3066156116","heartbeat_front_addr":"172.21.15.32:6821/3066156116","state":["exists","up"]},{"osd":1,"uuid":"d7c5c210-da4f-49f9-b9e7-1dfb02daa4bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6824","nonce":1551241759},{"type":"v1","addr":"172.21.15.32:6825","nonce":1551241759}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6826","nonce":1551241759},{"type":"v1","addr":"172.21.15.32:6827","nonce":1551241759}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6830","nonce":1551241759},{"type":"v1","addr":"172.21.15.32:6831","nonce":1551241759}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6828","nonce":1551241759},{"type":"v1","addr":"172.21.15.32:6829","nonce":1551241759}]},"public_addr":"172.21.15.32:6825/1551241759","cluster_addr":"172.21.15.32:6827/1551241759","heartbeat_back_addr":"172.21.15.32:6831/1551241759","heartbeat_front_addr":"172.21.15.32:6829/1551241759","state":["exists","up"]},{"osd":2,"uuid":"57f2e6de-a201-45c2-8981-8e56a4875ebd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6808","nonce":3806708944},{"type":"v1","addr":"172.21.15.32:6809","nonce":3806708944}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6810","nonce":3806708944},{"type":"v1","addr":"172.21.15.32:6811","nonce":3806708944}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6814","nonce":3806708944},{"type":"v1","addr":"172.21.15.32:6815","nonce":3806708944}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6812","nonce":3806708944},{"type":"v1","addr":"172.21.15.32:6813","nonce":3806708944}]},"public_addr":"172.21.15.32:6809/3806708944","cluster_addr":"172.21.15.32:6811/3806708944","heartbeat_back_addr":"172.21.15.32:6815/3806708944","heartbeat_front_addr":"172.21.15.32:6813/3806708944","state":["exists","up"]},{"osd":3,"uuid":"6439a280-da45-4148-8130-f16323935ff3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6800","nonce":9088303},{"type":"v1","addr":"172.21.15.32:6801","nonce":9088303}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6802","nonce":9088303},{"type":"v1","addr":"172.21.15.32:6803","nonce":9088303}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6806","nonce":9088303},{"type":"v1","addr":"172.21.15.32:6807","nonce":9088303}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6804","nonce":9088303},{"type":"v1","addr":"172.21.15.32:6805","nonce":9088303}]},"public_addr":"172.21.15.32:6801/9088303","cluster_addr":"172.21.15.32:6803/9088303","heartbeat_back_addr":"172.21.15.32:6807/9088303","heartbeat_front_addr":"172.21.15.32:6805/9088303","state":["exists","up"]},{"osd":4,"uuid":"4f902eff-1e18-418d-82c9-2151c69524fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6802","nonce":864752275},{"type":"v1","addr":"172.21.15.118:6803","nonce":864752275}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6804","nonce":864752275},{"type":"v1","addr":"172.21.15.118:6805","nonce":864752275}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6808","nonce":864752275},{"type":"v1","addr":"172.21.15.118:6809","nonce":864752275}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6806","nonce":864752275},{"type":"v1","addr":"172.21.15.118:6807","nonce":864752275}]},"public_addr":"172.21.15.118:6803/864752275","cluster_addr":"172.21.15.118:6805/864752275","heartbeat_back_addr":"172.21.15.118:6809/864752275","heartbeat_front_addr":"172.21.15.118:6807/864752275","state":["exists","up"]},{"osd":5,"uuid":"bfd9feac-dd06-458c-91b9-af7f175f612c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6810","nonce":2724891770},{"type":"v1","addr":"172.21.15.118:6811","nonce":2724891770}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6812","nonce":2724891770},{"type":"v1","addr":"172.21.15.118:6813","nonce":2724891770}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6816","nonce":2724891770},{"type":"v1","addr":"172.21.15.118:6817","nonce":2724891770}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6814","nonce":2724891770},{"type":"v1","addr":"172.21.15.118:6815","nonce":2724891770}]},"public_addr":"172.21.15.118:6811/2724891770","cluster_addr":"172.21.15.118:6813/2724891770","heartbeat_back_addr":"172.21.15.118:6817/2724891770","heartbeat_front_addr":"172.21.15.118:6815/2724891770","state":["exists","up"]},{"osd":6,"uuid":"5ed5a279-9d8e-416d-bbc7-7edd3e6d5705","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6825","nonce":801947386},{"type":"v1","addr":"172.21.15.118:6827","nonce":801947386}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6828","nonce":801947386},{"type":"v1","addr":"172.21.15.118:6829","nonce":801947386}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6832","nonce":801947386},{"type":"v1","addr":"172.21.15.118:6833","nonce":801947386}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6830","nonce":801947386},{"type":"v1","addr":"172.21.15.118:6831","nonce":801947386}]},"public_addr":"172.21.15.118:6827/801947386","cluster_addr":"172.21.15.118:6829/801947386","heartbeat_back_addr":"172.21.15.118:6833/801947386","heartbeat_front_addr":"172.21.15.118:6831/801947386","state":["exists","up"]},{"osd":7,"uuid":"f2086c47-875e-4451-a436-2008d1ea16af","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6818","nonce":653000048},{"type":"v1","addr":"172.21.15.118:6819","nonce":653000048}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6820","nonce":653000048},{"type":"v1","addr":"172.21.15.118:6821","nonce":653000048}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6824","nonce":653000048},{"type":"v1","addr":"172.21.15.118:6826","nonce":653000048}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6822","nonce":653000048},{"type":"v1","addr":"172.21.15.118:6823","nonce":653000048}]},"public_addr":"172.21.15.118:6819/653000048","cluster_addr":"172.21.15.118:6821/653000048","heartbeat_back_addr":"172.21.15.118:6826/653000048","heartbeat_front_addr":"172.21.15.118:6823/653000048","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-03T20:47:25.454175+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-03T20:47:25.377734+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-03T20:47:25.412351+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-08-03T20:47:29.079 INFO:teuthology.misc.health.smithi032.stderr:2024-08-03T20:47:29.078+0000 7f6752600640 1 -- 172.21.15.32:0/1031415512 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f672c076f80 msgr2=0x7f672c079490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:47:29.079 INFO:teuthology.misc.health.smithi032.stderr:2024-08-03T20:47:29.078+0000 7f6752600640 1 --2- 172.21.15.32:0/1031415512 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f672c076f80 0x7f672c079490 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f6738007fd0 tx=0x7f6738001390 comp rx=0 tx=0).stop 2024-08-03T20:47:29.079 INFO:teuthology.misc.health.smithi032.stderr:2024-08-03T20:47:29.078+0000 7f6752600640 1 -- 172.21.15.32:0/1031415512 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f674c0cc0e0 msgr2=0x7f674c1538d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:47:29.079 INFO:teuthology.misc.health.smithi032.stderr:2024-08-03T20:47:29.078+0000 7f6752600640 1 --2- 172.21.15.32:0/1031415512 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f674c0cc0e0 0x7f674c1538d0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f673c000c00 tx=0x7f673c033c20 comp rx=0 tx=0).stop 2024-08-03T20:47:29.079 INFO:teuthology.misc.health.smithi032.stderr:2024-08-03T20:47:29.079+0000 7f6752600640 1 -- 172.21.15.32:0/1031415512 shutdown_connections 2024-08-03T20:47:29.080 INFO:teuthology.misc.health.smithi032.stderr:2024-08-03T20:47:29.079+0000 7f6752600640 1 --2- 172.21.15.32:0/1031415512 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f672c076f80 0x7f672c079490 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:29.080 INFO:teuthology.misc.health.smithi032.stderr:2024-08-03T20:47:29.079+0000 7f6752600640 1 --2- 172.21.15.32:0/1031415512 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f674c1f9130 0x7f674c0b42a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:29.080 INFO:teuthology.misc.health.smithi032.stderr:2024-08-03T20:47:29.079+0000 7f6752600640 1 --2- 172.21.15.32:0/1031415512 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f674c15e810 0x7f674c1f8bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:29.080 INFO:teuthology.misc.health.smithi032.stderr:2024-08-03T20:47:29.079+0000 7f6752600640 1 --2- 172.21.15.32:0/1031415512 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f674c0cc0e0 0x7f674c1538d0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:29.080 INFO:teuthology.misc.health.smithi032.stderr:2024-08-03T20:47:29.079+0000 7f6752600640 1 -- 172.21.15.32:0/1031415512 >> 172.21.15.32:0/1031415512 conn(0x7f674c0cdb00 msgr2=0x7f674c068a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:47:29.080 INFO:teuthology.misc.health.smithi032.stderr:2024-08-03T20:47:29.079+0000 7f6752600640 1 -- 172.21.15.32:0/1031415512 shutdown_connections 2024-08-03T20:47:29.080 INFO:teuthology.misc.health.smithi032.stderr:2024-08-03T20:47:29.079+0000 7f6752600640 1 -- 172.21.15.32:0/1031415512 wait complete. 2024-08-03T20:47:29.098 DEBUG:teuthology.misc:8 of 8 OSDs are up 2024-08-03T20:47:29.099 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2024-08-03T20:47:29.099 INFO:tasks.ceph.mds.a:Restarting daemon 2024-08-03T20:47:29.099 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-08-03T20:47:29.103 INFO:tasks.ceph.mds.a:Started 2024-08-03T20:47:29.103 INFO:tasks.ceph.mds.c:Restarting daemon 2024-08-03T20:47:29.103 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-08-03T20:47:29.106 INFO:tasks.ceph.mds.c:Started 2024-08-03T20:47:29.107 INFO:tasks.ceph.mds.b:Restarting daemon 2024-08-03T20:47:29.107 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-08-03T20:47:29.111 INFO:tasks.ceph.mds.b:Started 2024-08-03T20:47:29.111 INFO:tasks.ceph.mds.d:Restarting daemon 2024-08-03T20:47:29.111 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-08-03T20:47:29.115 INFO:tasks.ceph.mds.d:Started 2024-08-03T20:47:29.115 INFO:tasks.ceph:Setting up CephFS filesystem(s)... 2024-08-03T20:47:29.116 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-08-03T20:47:29.301 INFO:tasks.ceph.mds.a.smithi032.stdout:starting mds.a at 2024-08-03T20:47:29.303 INFO:tasks.ceph.mds.c.smithi032.stdout:starting mds.c at 2024-08-03T20:47:29.309 INFO:tasks.ceph.mds.d.smithi118.stdout:starting mds.d at 2024-08-03T20:47:29.310 INFO:tasks.ceph.mds.b.smithi118.stdout:starting mds.b at 2024-08-03T20:47:29.354 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:29.353+0000 7f9921200640 1 -- 172.21.15.32:0/677086421 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f991c0b7760 msgr2=0x7f991c0cdba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:47:29.354 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:29.353+0000 7f9921200640 1 --2- 172.21.15.32:0/677086421 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f991c0b7760 0x7f991c0cdba0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f991c0a0060 tx=0x7f990c02a4d0 comp rx=0 tx=0).stop 2024-08-03T20:47:29.354 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:29.353+0000 7f9921200640 1 -- 172.21.15.32:0/677086421 shutdown_connections 2024-08-03T20:47:29.354 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:29.353+0000 7f9921200640 1 --2- 172.21.15.32:0/677086421 >> v2:172.21.15.118:3300/0 conn(0x7f991c09ac30 0x7f991c09b040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:29.354 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:29.353+0000 7f9921200640 1 --2- 172.21.15.32:0/677086421 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f991c0b7760 0x7f991c0cdba0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:29.354 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:29.353+0000 7f9921200640 1 -- 172.21.15.32:0/677086421 >> 172.21.15.32:0/677086421 conn(0x7f991c0b3bb0 msgr2=0x7f991c0b3fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:47:29.354 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:29.353+0000 7f9921200640 1 -- 172.21.15.32:0/677086421 shutdown_connections 2024-08-03T20:47:29.354 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:29.353+0000 7f9921200640 1 -- 172.21.15.32:0/677086421 wait complete. 2024-08-03T20:47:29.356 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:29.355+0000 7f9921200640 1 Processor -- start 2024-08-03T20:47:29.356 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:29.356+0000 7f9921200640 1 -- start start 2024-08-03T20:47:29.358 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:29.357+0000 7f9921200640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f991c09ac30 0x7f991c14e620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:47:29.358 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:29.357+0000 7f9921200640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f991c0b7760 0x7f991c161190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:47:29.358 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:29.357+0000 7f9921200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f991c0cfc90 0x7f991c15f7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:47:29.358 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:29.357+0000 7f9921200640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f991c0b29f0 con 0x7f991c0cfc90 2024-08-03T20:47:29.358 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:29.357+0000 7f9921200640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f991c0d09c0 con 0x7f991c0b7760 2024-08-03T20:47:29.358 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:29.357+0000 7f9921200640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7f991c0b2b70 con 0x7f991c09ac30 2024-08-03T20:47:29.359 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:29.357+0000 7f991a800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f991c0cfc90 0x7f991c15f7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:47:29.359 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:29.357+0000 7f9913e00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f991c0b7760 0x7f991c161190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:47:29.359 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:29.357+0000 7f9913e00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f991c0b7760 0x7f991c161190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.32:52280/0 (socket says 172.21.15.32:52280) 2024-08-03T20:47:29.359 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:29.357+0000 7f9913e00640 1 -- 172.21.15.32:0/4208009412 learned_addr learned my addr 172.21.15.32:0/4208009412 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:47:29.359 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:29.357+0000 7f991a800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f991c0cfc90 0x7f991c15f7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:51772/0 (socket says 172.21.15.32:51772) 2024-08-03T20:47:29.359 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:29.357+0000 7f9919e00640 1 --2- 172.21.15.32:0/4208009412 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f991c09ac30 0x7f991c14e620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:47:29.359 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:29.358+0000 7f9913e00640 1 -- 172.21.15.32:0/4208009412 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f991c09ac30 msgr2=0x7f991c14e620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:47:29.359 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:29.358+0000 7f9913e00640 1 --2- 172.21.15.32:0/4208009412 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f991c09ac30 0x7f991c14e620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:29.359 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:29.358+0000 7f9913e00640 1 -- 172.21.15.32:0/4208009412 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f991c0cfc90 msgr2=0x7f991c15f7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:47:29.359 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:29.358+0000 7f9913e00640 1 --2- 172.21.15.32:0/4208009412 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f991c0cfc90 0x7f991c15f7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:29.359 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:29.358+0000 7f9913e00640 1 -- 172.21.15.32:0/4208009412 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f991c0b00d0 con 0x7f991c0b7760 2024-08-03T20:47:29.360 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:29.358+0000 7f9913e00640 1 --2- 172.21.15.32:0/4208009412 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f991c0b7760 0x7f991c161190 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f990400e790 tx=0x7f990400ec50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:47:29.360 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:29.358+0000 7f9912000640 1 -- 172.21.15.32:0/4208009412 <== mon.1 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f990400eea0 con 0x7f991c0b7760 2024-08-03T20:47:29.360 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:29.358+0000 7f9912000640 1 -- 172.21.15.32:0/4208009412 <== mon.1 v2:172.21.15.118:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f990400cdb0 con 0x7f991c0b7760 2024-08-03T20:47:29.360 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:29.358+0000 7f9912000640 1 -- 172.21.15.32:0/4208009412 <== mon.1 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9904010690 con 0x7f991c0b7760 2024-08-03T20:47:29.361 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:29.358+0000 7f9921200640 1 -- 172.21.15.32:0/4208009412 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f991c0b03b0 con 0x7f991c0b7760 2024-08-03T20:47:29.361 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:29.359+0000 7f9921200640 1 -- 172.21.15.32:0/4208009412 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f991c1616d0 con 0x7f991c0b7760 2024-08-03T20:47:29.361 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:29.360+0000 7f9921200640 1 -- 172.21.15.32:0/4208009412 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f991c164710 con 0x7f991c0b7760 2024-08-03T20:47:29.361 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:29.361+0000 7f9912000640 1 -- 172.21.15.32:0/4208009412 <== mon.1 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 4) ==== 99083+0+0 (secure 0 0 0) 0x7f9904010830 con 0x7f991c0b7760 2024-08-03T20:47:29.362 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:29.362+0000 7f9912000640 1 --2- 172.21.15.32:0/4208009412 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f98f8076fb0 0x7f98f80794c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:47:29.362 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:29.362+0000 7f9912000640 1 -- 172.21.15.32:0/4208009412 <== mon.1 v2:172.21.15.118:3300/0 5 ==== osd_map(15..15 src has 1..15) ==== 4846+0+0 (secure 0 0 0) 0x7f9904098d50 con 0x7f991c0b7760 2024-08-03T20:47:29.362 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:29.362+0000 7f9919e00640 1 --2- 172.21.15.32:0/4208009412 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f98f8076fb0 0x7f98f80794c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:47:29.363 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:29.362+0000 7f9919e00640 1 --2- 172.21.15.32:0/4208009412 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f98f8076fb0 0x7f98f80794c0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f990c034000 tx=0x7f990c004730 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:47:29.367 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:29.367+0000 7f9912000640 1 -- 172.21.15.32:0/4208009412 <== mon.1 v2:172.21.15.118:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7f9904061d30 con 0x7f991c0b7760 2024-08-03T20:47:29.610 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:29.610+0000 7f9921200640 1 -- 172.21.15.32:0/4208009412 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f991c0b0580 con 0x7f991c0b7760 2024-08-03T20:47:29.611 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:29.610+0000 7f9912000640 1 -- 172.21.15.32:0/4208009412 <== mon.1 v2:172.21.15.118:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 2 v2) ==== 93+0+1434 (secure 0 0 0) 0x7f9904066280 con 0x7f991c0b7760 2024-08-03T20:47:29.611 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-03T20:47:29.611 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":2,"btime":"2024-08-03T20:47:29:310479+0000","default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4338,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6834/3026095815","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":3026095815},{"type":"v1","addr":"172.21.15.32:6834","nonce":3026095815}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[]} 2024-08-03T20:47:29.612 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 2 2024-08-03T20:47:29.614 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:29.613+0000 7f9921200640 1 -- 172.21.15.32:0/4208009412 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f98f8076fb0 msgr2=0x7f98f80794c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:47:29.614 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:29.613+0000 7f9921200640 1 --2- 172.21.15.32:0/4208009412 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f98f8076fb0 0x7f98f80794c0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f990c034000 tx=0x7f990c004730 comp rx=0 tx=0).stop 2024-08-03T20:47:29.614 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:29.613+0000 7f9921200640 1 -- 172.21.15.32:0/4208009412 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f991c0b7760 msgr2=0x7f991c161190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:47:29.614 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:29.613+0000 7f9921200640 1 --2- 172.21.15.32:0/4208009412 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f991c0b7760 0x7f991c161190 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f990400e790 tx=0x7f990400ec50 comp rx=0 tx=0).stop 2024-08-03T20:47:29.614 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:29.613+0000 7f9921200640 1 -- 172.21.15.32:0/4208009412 shutdown_connections 2024-08-03T20:47:29.614 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:29.613+0000 7f9921200640 1 --2- 172.21.15.32:0/4208009412 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f98f8076fb0 0x7f98f80794c0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:29.615 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:29.613+0000 7f9921200640 1 --2- 172.21.15.32:0/4208009412 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f991c0b7760 0x7f991c161190 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:29.615 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:29.613+0000 7f9921200640 1 --2- 172.21.15.32:0/4208009412 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f991c0cfc90 0x7f991c15f7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:29.615 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:29.613+0000 7f9921200640 1 --2- 172.21.15.32:0/4208009412 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f991c09ac30 0x7f991c14e620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:29.615 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:29.613+0000 7f9921200640 1 -- 172.21.15.32:0/4208009412 >> 172.21.15.32:0/4208009412 conn(0x7f991c0b3bb0 msgr2=0x7f991c0cead0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:47:29.615 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:29.613+0000 7f9921200640 1 -- 172.21.15.32:0/4208009412 shutdown_connections 2024-08-03T20:47:29.615 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:29.613+0000 7f9921200640 1 -- 172.21.15.32:0/4208009412 wait complete. 2024-08-03T20:47:31.638 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-08-03T20:47:31.870 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:31.869+0000 7f4a13800640 1 -- 172.21.15.32:0/1227981174 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4a0c06e930 msgr2=0x7f4a0c144f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:47:31.871 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:31.869+0000 7f4a13800640 1 --2- 172.21.15.32:0/1227981174 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4a0c06e930 0x7f4a0c144f90 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f4a0c0afaf0 tx=0x7f49fc02a4f0 comp rx=0 tx=0).stop 2024-08-03T20:47:31.871 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:31.869+0000 7f4a13800640 1 -- 172.21.15.32:0/1227981174 shutdown_connections 2024-08-03T20:47:31.871 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:31.869+0000 7f4a13800640 1 --2- 172.21.15.32:0/1227981174 >> v2:172.21.15.118:3300/0 conn(0x7f4a0c073e60 0x7f4a0c074270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:31.871 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:31.869+0000 7f4a13800640 1 --2- 172.21.15.32:0/1227981174 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4a0c06e930 0x7f4a0c144f90 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:31.871 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:31.869+0000 7f4a13800640 1 -- 172.21.15.32:0/1227981174 >> 172.21.15.32:0/1227981174 conn(0x7f4a0c0b1bd0 msgr2=0x7f4a0c0b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:47:31.872 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:31.869+0000 7f4a13800640 1 -- 172.21.15.32:0/1227981174 shutdown_connections 2024-08-03T20:47:31.872 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:31.869+0000 7f4a13800640 1 -- 172.21.15.32:0/1227981174 wait complete. 2024-08-03T20:47:31.873 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:31.871+0000 7f4a13800640 1 Processor -- start 2024-08-03T20:47:31.873 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:31.871+0000 7f4a13800640 1 -- start start 2024-08-03T20:47:31.877 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:31.873+0000 7f4a13800640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f4a0c06e930 0x7f4a0c13c060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:47:31.877 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:31.873+0000 7f4a13800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4a0c073e60 0x7f4a0c1435b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:47:31.877 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:31.873+0000 7f4a13800640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f4a0c0747b0 0x7f4a0c1f8a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:47:31.877 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:31.873+0000 7f4a13800640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f4a0c0cf9c0 con 0x7f4a0c073e60 2024-08-03T20:47:31.878 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:31.873+0000 7f4a13800640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f4a0c0cf840 con 0x7f4a0c0747b0 2024-08-03T20:47:31.878 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:31.873+0000 7f4a13800640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7f4a0c0ccff0 con 0x7f4a0c06e930 2024-08-03T20:47:31.878 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:31.873+0000 7f4a03e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4a0c073e60 0x7f4a0c1435b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:47:31.878 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:31.873+0000 7f4a03e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4a0c073e60 0x7f4a0c1435b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:40526/0 (socket says 172.21.15.32:40526) 2024-08-03T20:47:31.878 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:31.873+0000 7f4a03e00640 1 -- 172.21.15.32:0/254465047 learned_addr learned my addr 172.21.15.32:0/254465047 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:47:31.879 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:31.873+0000 7f4a10e00640 1 --2- 172.21.15.32:0/254465047 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f4a0c06e930 0x7f4a0c13c060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:47:31.879 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:31.873+0000 7f4a11800640 1 --2- 172.21.15.32:0/254465047 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f4a0c0747b0 0x7f4a0c1f8a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:47:31.879 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:31.874+0000 7f4a03e00640 1 -- 172.21.15.32:0/254465047 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f4a0c06e930 msgr2=0x7f4a0c13c060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:47:31.879 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:31.874+0000 7f4a03e00640 1 --2- 172.21.15.32:0/254465047 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f4a0c06e930 0x7f4a0c13c060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:31.879 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:31.874+0000 7f4a03e00640 1 -- 172.21.15.32:0/254465047 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f4a0c0747b0 msgr2=0x7f4a0c1f8a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:47:31.879 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:31.874+0000 7f4a03e00640 1 --2- 172.21.15.32:0/254465047 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f4a0c0747b0 0x7f4a0c1f8a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:31.880 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:31.874+0000 7f4a03e00640 1 -- 172.21.15.32:0/254465047 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4a0c144320 con 0x7f4a0c073e60 2024-08-03T20:47:31.880 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:31.874+0000 7f4a03e00640 1 --2- 172.21.15.32:0/254465047 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4a0c073e60 0x7f4a0c1435b0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f49f400b410 tx=0x7f49f4009f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:47:31.880 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:31.874+0000 7f4a09600640 1 -- 172.21.15.32:0/254465047 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49f4015020 con 0x7f4a0c073e60 2024-08-03T20:47:31.880 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:31.874+0000 7f4a09600640 1 -- 172.21.15.32:0/254465047 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f49f4011940 con 0x7f4a0c073e60 2024-08-03T20:47:31.880 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:31.874+0000 7f4a13800640 1 -- 172.21.15.32:0/254465047 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4a0c1e5eb0 con 0x7f4a0c073e60 2024-08-03T20:47:31.880 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:31.874+0000 7f4a09600640 1 -- 172.21.15.32:0/254465047 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49f4011d90 con 0x7f4a0c073e60 2024-08-03T20:47:31.881 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:31.874+0000 7f4a13800640 1 -- 172.21.15.32:0/254465047 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4a0c1e6470 con 0x7f4a0c073e60 2024-08-03T20:47:31.881 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:31.876+0000 7f4a13800640 1 -- 172.21.15.32:0/254465047 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4a0c144090 con 0x7f4a0c073e60 2024-08-03T20:47:31.882 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:31.877+0000 7f4a09600640 1 -- 172.21.15.32:0/254465047 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 4) ==== 99083+0+0 (secure 0 0 0) 0x7f49f4018020 con 0x7f4a0c073e60 2024-08-03T20:47:31.883 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:31.878+0000 7f4a09600640 1 --2- 172.21.15.32:0/254465047 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f49e8076fb0 0x7f49e80794c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:47:31.883 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:31.878+0000 7f4a09600640 1 -- 172.21.15.32:0/254465047 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(17..17 src has 1..17) ==== 4857+0+0 (secure 0 0 0) 0x7f49f4098810 con 0x7f4a0c073e60 2024-08-03T20:47:31.883 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:31.879+0000 7f4a10e00640 1 --2- 172.21.15.32:0/254465047 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f49e8076fb0 0x7f49e80794c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:47:31.883 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:31.879+0000 7f4a10e00640 1 --2- 172.21.15.32:0/254465047 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f49e8076fb0 0x7f49e80794c0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f49fc033000 tx=0x7f49fc004730 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:47:31.883 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:31.882+0000 7f4a09600640 1 -- 172.21.15.32:0/254465047 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7f49f401f020 con 0x7f4a0c073e60 2024-08-03T20:47:32.126 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:32.125+0000 7f4a13800640 1 -- 172.21.15.32:0/254465047 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f4a0c0cd600 con 0x7f4a0c073e60 2024-08-03T20:47:32.126 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:32.126+0000 7f4a09600640 1 -- 172.21.15.32:0/254465047 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 3 v3) ==== 93+0+3909 (secure 0 0 0) 0x7f49f40617d0 con 0x7f4a0c073e60 2024-08-03T20:47:32.127 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-03T20:47:32.127 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":3,"btime":"2024-08-03T20:47:30:311782+0000","default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4289,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6835/1753256117","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6834","nonce":1753256117},{"type":"v1","addr":"172.21.15.118:6835","nonce":1753256117}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4292,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6837/3055588578","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6836","nonce":3055588578},{"type":"v1","addr":"172.21.15.118:6837","nonce":3055588578}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4338,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6834/3026095815","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":3026095815},{"type":"v1","addr":"172.21.15.32:6834","nonce":3026095815}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4341,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6835/246278108","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6833","nonce":246278108},{"type":"v1","addr":"172.21.15.32:6835","nonce":246278108}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[]} 2024-08-03T20:47:32.127 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 3 2024-08-03T20:47:32.129 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:32.129+0000 7f4a13800640 1 -- 172.21.15.32:0/254465047 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f49e8076fb0 msgr2=0x7f49e80794c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:47:32.129 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:32.129+0000 7f4a13800640 1 --2- 172.21.15.32:0/254465047 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f49e8076fb0 0x7f49e80794c0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f49fc033000 tx=0x7f49fc004730 comp rx=0 tx=0).stop 2024-08-03T20:47:32.129 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:32.129+0000 7f4a13800640 1 -- 172.21.15.32:0/254465047 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4a0c073e60 msgr2=0x7f4a0c1435b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:47:32.130 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:32.129+0000 7f4a13800640 1 --2- 172.21.15.32:0/254465047 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4a0c073e60 0x7f4a0c1435b0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f49f400b410 tx=0x7f49f4009f90 comp rx=0 tx=0).stop 2024-08-03T20:47:32.130 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:32.129+0000 7f4a13800640 1 -- 172.21.15.32:0/254465047 shutdown_connections 2024-08-03T20:47:32.130 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:32.129+0000 7f4a13800640 1 --2- 172.21.15.32:0/254465047 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f49e8076fb0 0x7f49e80794c0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:32.130 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:32.129+0000 7f4a13800640 1 --2- 172.21.15.32:0/254465047 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f4a0c0747b0 0x7f4a0c1f8a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:32.130 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:32.129+0000 7f4a13800640 1 --2- 172.21.15.32:0/254465047 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4a0c073e60 0x7f4a0c1435b0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:32.130 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:32.129+0000 7f4a13800640 1 --2- 172.21.15.32:0/254465047 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f4a0c06e930 0x7f4a0c13c060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:32.130 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:32.129+0000 7f4a13800640 1 -- 172.21.15.32:0/254465047 >> 172.21.15.32:0/254465047 conn(0x7f4a0c0b1bd0 msgr2=0x7f4a0c0b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:47:32.130 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:32.129+0000 7f4a13800640 1 -- 172.21.15.32:0/254465047 shutdown_connections 2024-08-03T20:47:32.130 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:32.129+0000 7f4a13800640 1 -- 172.21.15.32:0/254465047 wait complete. 2024-08-03T20:47:32.153 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-08-03T20:47:32.381 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:32.380+0000 7f6acba00640 1 -- 172.21.15.32:0/1064046739 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6ac415f440 msgr2=0x7f6ac415b910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:47:32.381 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:32.380+0000 7f6acba00640 1 --2- 172.21.15.32:0/1064046739 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6ac415f440 0x7f6ac415b910 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f6ac40d1030 tx=0x7f6abc02aa10 comp rx=0 tx=0).stop 2024-08-03T20:47:32.381 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:32.380+0000 7f6acba00640 1 -- 172.21.15.32:0/1064046739 shutdown_connections 2024-08-03T20:47:32.381 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:32.380+0000 7f6acba00640 1 --2- 172.21.15.32:0/1064046739 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6ac415f440 0x7f6ac415b910 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:32.381 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:32.380+0000 7f6acba00640 1 --2- 172.21.15.32:0/1064046739 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f6ac40b4ce0 0x7f6ac415ef00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:32.381 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:32.380+0000 7f6acba00640 1 --2- 172.21.15.32:0/1064046739 >> v2:172.21.15.118:3300/0 conn(0x7f6ac40b38b0 0x7f6ac41523f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:32.382 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:32.380+0000 7f6acba00640 1 -- 172.21.15.32:0/1064046739 >> 172.21.15.32:0/1064046739 conn(0x7f6ac40b1bd0 msgr2=0x7f6ac40b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:47:32.382 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:32.380+0000 7f6acba00640 1 -- 172.21.15.32:0/1064046739 shutdown_connections 2024-08-03T20:47:32.382 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:32.381+0000 7f6acba00640 1 -- 172.21.15.32:0/1064046739 wait complete. 2024-08-03T20:47:32.383 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:32.383+0000 7f6acba00640 1 Processor -- start 2024-08-03T20:47:32.383 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:32.383+0000 7f6acba00640 1 -- start start 2024-08-03T20:47:32.386 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:32.384+0000 7f6acba00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f6ac40b38b0 0x7f6ac414cf10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:47:32.386 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:32.384+0000 7f6acba00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6ac40b4ce0 0x7f6ac4166e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:47:32.386 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:32.384+0000 7f6acba00640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f6ac415f440 0x7f6ac415dbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:47:32.387 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:32.384+0000 7f6acba00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f6ac415c1e0 con 0x7f6ac40b4ce0 2024-08-03T20:47:32.387 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:32.384+0000 7f6acba00640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f6ac415c060 con 0x7f6ac40b38b0 2024-08-03T20:47:32.387 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:32.384+0000 7f6acba00640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7f6ac415c360 con 0x7f6ac415f440 2024-08-03T20:47:32.387 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:32.385+0000 7f6ac3e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6ac40b4ce0 0x7f6ac4166e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:47:32.387 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:32.385+0000 7f6ac3e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6ac40b4ce0 0x7f6ac4166e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:40550/0 (socket says 172.21.15.32:40550) 2024-08-03T20:47:32.387 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:32.385+0000 7f6ac3e00640 1 -- 172.21.15.32:0/2322912074 learned_addr learned my addr 172.21.15.32:0/2322912074 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:47:32.387 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:32.385+0000 7f6ac9000640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f6ac40b38b0 0x7f6ac414cf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:47:32.387 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:32.385+0000 7f6ac9a00640 1 --2- 172.21.15.32:0/2322912074 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f6ac415f440 0x7f6ac415dbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:47:32.387 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:32.385+0000 7f6ac3e00640 1 -- 172.21.15.32:0/2322912074 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f6ac415f440 msgr2=0x7f6ac415dbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:47:32.388 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:32.385+0000 7f6ac3e00640 1 --2- 172.21.15.32:0/2322912074 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f6ac415f440 0x7f6ac415dbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:32.388 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:32.385+0000 7f6ac3e00640 1 -- 172.21.15.32:0/2322912074 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f6ac40b38b0 msgr2=0x7f6ac414cf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:47:32.388 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:32.385+0000 7f6ac3e00640 1 --2- 172.21.15.32:0/2322912074 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f6ac40b38b0 0x7f6ac414cf10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:32.388 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:32.385+0000 7f6ac3e00640 1 -- 172.21.15.32:0/2322912074 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6ac41f5be0 con 0x7f6ac40b4ce0 2024-08-03T20:47:32.388 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:32.385+0000 7f6ac9000640 1 --2- 172.21.15.32:0/2322912074 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f6ac40b38b0 0x7f6ac414cf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-03T20:47:32.388 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:32.385+0000 7f6ac3e00640 1 --2- 172.21.15.32:0/2322912074 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6ac40b4ce0 0x7f6ac4166e70 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f6ab000e790 tx=0x7f6ab000ec50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:47:32.388 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:32.386+0000 7f6ac0c00640 1 -- 172.21.15.32:0/2322912074 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ab000eea0 con 0x7f6ac40b4ce0 2024-08-03T20:47:32.388 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:32.386+0000 7f6ac0c00640 1 -- 172.21.15.32:0/2322912074 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f6ab000cd50 con 0x7f6ac40b4ce0 2024-08-03T20:47:32.388 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:32.386+0000 7f6ac0c00640 1 -- 172.21.15.32:0/2322912074 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ab0010690 con 0x7f6ac40b4ce0 2024-08-03T20:47:32.388 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:32.386+0000 7f6acba00640 1 -- 172.21.15.32:0/2322912074 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6ac41f5ec0 con 0x7f6ac40b4ce0 2024-08-03T20:47:32.389 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:32.386+0000 7f6acba00640 1 -- 172.21.15.32:0/2322912074 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6ac41f6480 con 0x7f6ac40b4ce0 2024-08-03T20:47:32.389 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:32.387+0000 7f6acba00640 1 -- 172.21.15.32:0/2322912074 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6ac4070130 con 0x7f6ac40b4ce0 2024-08-03T20:47:32.389 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:32.388+0000 7f6ac0c00640 1 -- 172.21.15.32:0/2322912074 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 4) ==== 99083+0+0 (secure 0 0 0) 0x7f6ab0010830 con 0x7f6ac40b4ce0 2024-08-03T20:47:32.390 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:32.390+0000 7f6ac0c00640 1 --2- 172.21.15.32:0/2322912074 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f6aa4076fb0 0x7f6aa40794c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:47:32.390 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:32.390+0000 7f6ac0c00640 1 -- 172.21.15.32:0/2322912074 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(17..17 src has 1..17) ==== 4857+0+0 (secure 0 0 0) 0x7f6ab0098cb0 con 0x7f6ac40b4ce0 2024-08-03T20:47:32.390 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:32.390+0000 7f6ac9000640 1 --2- 172.21.15.32:0/2322912074 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f6aa4076fb0 0x7f6aa40794c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:47:32.391 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:32.391+0000 7f6ac9000640 1 --2- 172.21.15.32:0/2322912074 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f6aa4076fb0 0x7f6aa40794c0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f6ab8009ea0 tx=0x7f6ab8005f90 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:47:32.394 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:32.394+0000 7f6ac0c00640 1 -- 172.21.15.32:0/2322912074 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7f6ac4070130 con 0x7f6ac40b4ce0 2024-08-03T20:47:32.598 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:32.597+0000 7f6acba00640 1 -- 172.21.15.32:0/2322912074 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) -- 0x7f6ac41f5970 con 0x7f6ac40b4ce0 2024-08-03T20:47:32.598 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:32.598+0000 7f6ac0c00640 1 -- 172.21.15.32:0/2322912074 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v17) ==== 84+0+65 (secure 0 0 0) 0x7f6ab0061c80 con 0x7f6ac40b4ce0 2024-08-03T20:47:32.599 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-03T20:47:32.599 INFO:teuthology.orchestra.run.smithi032.stdout:[ 2024-08-03T20:47:32.599 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-03T20:47:32.599 INFO:teuthology.orchestra.run.smithi032.stdout: "poolnum": 1, 2024-08-03T20:47:32.599 INFO:teuthology.orchestra.run.smithi032.stdout: "poolname": ".mgr" 2024-08-03T20:47:32.599 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-03T20:47:32.599 INFO:teuthology.orchestra.run.smithi032.stdout:] 2024-08-03T20:47:32.601 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:32.600+0000 7f6acba00640 1 -- 172.21.15.32:0/2322912074 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f6aa4076fb0 msgr2=0x7f6aa40794c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:47:32.601 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:32.600+0000 7f6acba00640 1 --2- 172.21.15.32:0/2322912074 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f6aa4076fb0 0x7f6aa40794c0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f6ab8009ea0 tx=0x7f6ab8005f90 comp rx=0 tx=0).stop 2024-08-03T20:47:32.601 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:32.600+0000 7f6acba00640 1 -- 172.21.15.32:0/2322912074 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6ac40b4ce0 msgr2=0x7f6ac4166e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:47:32.601 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:32.600+0000 7f6acba00640 1 --2- 172.21.15.32:0/2322912074 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6ac40b4ce0 0x7f6ac4166e70 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f6ab000e790 tx=0x7f6ab000ec50 comp rx=0 tx=0).stop 2024-08-03T20:47:32.601 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:32.600+0000 7f6acba00640 1 -- 172.21.15.32:0/2322912074 shutdown_connections 2024-08-03T20:47:32.601 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:32.600+0000 7f6acba00640 1 --2- 172.21.15.32:0/2322912074 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f6aa4076fb0 0x7f6aa40794c0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:32.601 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:32.600+0000 7f6acba00640 1 --2- 172.21.15.32:0/2322912074 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f6ac415f440 0x7f6ac415dbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:32.602 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:32.600+0000 7f6acba00640 1 --2- 172.21.15.32:0/2322912074 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6ac40b4ce0 0x7f6ac4166e70 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:32.602 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:32.600+0000 7f6acba00640 1 --2- 172.21.15.32:0/2322912074 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f6ac40b38b0 0x7f6ac414cf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:32.602 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:32.600+0000 7f6acba00640 1 -- 172.21.15.32:0/2322912074 >> 172.21.15.32:0/2322912074 conn(0x7f6ac40b1bd0 msgr2=0x7f6ac406f140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:47:32.602 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:32.600+0000 7f6acba00640 1 -- 172.21.15.32:0/2322912074 shutdown_connections 2024-08-03T20:47:32.602 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:32.600+0000 7f6acba00640 1 -- 172.21.15.32:0/2322912074 wait complete. 2024-08-03T20:47:32.622 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-08-03T20:47:32.622 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-08-03T20:47:32.855 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:32.853+0000 7fa660c00640 1 -- 172.21.15.32:0/1952171066 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fa65c0ced30 msgr2=0x7fa640003a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:47:32.855 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:32.853+0000 7fa660c00640 1 --2- 172.21.15.32:0/1952171066 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fa65c0ced30 0x7fa640003a50 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fa64c00c310 tx=0x7fa640005830 comp rx=0 tx=0).stop 2024-08-03T20:47:32.855 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:32.853+0000 7fa65a800640 1 -- 172.21.15.32:0/1952171066 reap_dead start 2024-08-03T20:47:32.855 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:32.853+0000 7fa660c00640 1 -- 172.21.15.32:0/1952171066 shutdown_connections 2024-08-03T20:47:32.856 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:32.853+0000 7fa660c00640 1 -- 172.21.15.32:0/1952171066 >> 172.21.15.32:0/1952171066 conn(0x7fa65c0b1bd0 msgr2=0x7fa65c0b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:47:32.856 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:32.854+0000 7fa660c00640 1 -- 172.21.15.32:0/1952171066 shutdown_connections 2024-08-03T20:47:32.856 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:32.854+0000 7fa660c00640 1 -- 172.21.15.32:0/1952171066 wait complete. 2024-08-03T20:47:32.857 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:32.856+0000 7fa660c00640 1 Processor -- start 2024-08-03T20:47:32.857 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:32.856+0000 7fa660c00640 1 -- start start 2024-08-03T20:47:32.861 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:32.857+0000 7fa660c00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa65c0ced30 0x7fa65c1f5740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:47:32.862 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:32.857+0000 7fa660c00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa65c140a30 0x7fa65c134720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:47:32.862 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:32.857+0000 7fa660c00640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fa65c0ca490 0x7fa65c0ca8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:47:32.862 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:32.857+0000 7fa660c00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fa65c0b3ce0 con 0x7fa65c0ced30 2024-08-03T20:47:32.862 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:32.857+0000 7fa660c00640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7fa65c0cf730 con 0x7fa65c140a30 2024-08-03T20:47:32.862 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:32.857+0000 7fa660c00640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7fa65c0b3e60 con 0x7fa65c0ca490 2024-08-03T20:47:32.862 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:32.858+0000 7fa659e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa65c0ced30 0x7fa65c1f5740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:47:32.863 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:32.858+0000 7fa659e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa65c0ced30 0x7fa65c1f5740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:40566/0 (socket says 172.21.15.32:40566) 2024-08-03T20:47:32.863 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:32.858+0000 7fa659e00640 1 -- 172.21.15.32:0/2180800195 learned_addr learned my addr 172.21.15.32:0/2180800195 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:47:32.863 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:32.858+0000 7fa659400640 1 --2- 172.21.15.32:0/2180800195 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa65c140a30 0x7fa65c134720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:47:32.863 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:32.858+0000 7fa65a800640 1 --2- 172.21.15.32:0/2180800195 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fa65c0ca490 0x7fa65c0ca8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:47:32.863 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:32.858+0000 7fa659e00640 1 -- 172.21.15.32:0/2180800195 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fa65c0ca490 msgr2=0x7fa65c0ca8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:47:32.863 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:32.858+0000 7fa659e00640 1 --2- 172.21.15.32:0/2180800195 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fa65c0ca490 0x7fa65c0ca8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:32.864 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:32.858+0000 7fa659e00640 1 -- 172.21.15.32:0/2180800195 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa65c140a30 msgr2=0x7fa65c134720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:47:32.864 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:32.858+0000 7fa659e00640 1 --2- 172.21.15.32:0/2180800195 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa65c140a30 0x7fa65c134720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:32.864 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:32.858+0000 7fa659e00640 1 -- 172.21.15.32:0/2180800195 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa65c0c93f0 con 0x7fa65c0ced30 2024-08-03T20:47:32.864 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:32.858+0000 7fa659e00640 1 --2- 172.21.15.32:0/2180800195 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa65c0ced30 0x7fa65c1f5740 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fa64c00c430 tx=0x7fa64c001d90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:47:32.864 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:32.859+0000 7fa652000640 1 -- 172.21.15.32:0/2180800195 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa64c031040 con 0x7fa65c0ced30 2024-08-03T20:47:32.864 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:32.859+0000 7fa652000640 1 -- 172.21.15.32:0/2180800195 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fa64c0035f0 con 0x7fa65c0ced30 2024-08-03T20:47:32.865 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:32.859+0000 7fa660c00640 1 -- 172.21.15.32:0/2180800195 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa65c0c96d0 con 0x7fa65c0ced30 2024-08-03T20:47:32.865 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:32.859+0000 7fa652000640 1 -- 172.21.15.32:0/2180800195 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa64c0039e0 con 0x7fa65c0ced30 2024-08-03T20:47:32.865 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:32.859+0000 7fa660c00640 1 -- 172.21.15.32:0/2180800195 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa65c0c9c90 con 0x7fa65c0ced30 2024-08-03T20:47:32.865 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:32.860+0000 7fa660c00640 1 -- 172.21.15.32:0/2180800195 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa65c0b43c0 con 0x7fa65c0ced30 2024-08-03T20:47:32.865 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:32.861+0000 7fa652000640 1 -- 172.21.15.32:0/2180800195 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 4) ==== 99083+0+0 (secure 0 0 0) 0x7fa64c034070 con 0x7fa65c0ced30 2024-08-03T20:47:32.866 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:32.862+0000 7fa652000640 1 --2- 172.21.15.32:0/2180800195 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fa63c076fb0 0x7fa63c0794c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:47:32.867 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:32.862+0000 7fa652000640 1 -- 172.21.15.32:0/2180800195 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(17..17 src has 1..17) ==== 4857+0+0 (secure 0 0 0) 0x7fa64c0b8490 con 0x7fa65c0ced30 2024-08-03T20:47:32.867 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:32.862+0000 7fa659400640 1 --2- 172.21.15.32:0/2180800195 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fa63c076fb0 0x7fa63c0794c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:47:32.867 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:32.863+0000 7fa659400640 1 --2- 172.21.15.32:0/2180800195 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fa63c076fb0 0x7fa63c0794c0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fa65c0cc830 tx=0x7fa648006b70 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:47:32.868 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:32.866+0000 7fa652000640 1 -- 172.21.15.32:0/2180800195 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7fa64c081450 con 0x7fa65c0ced30 2024-08-03T20:47:32.902 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:32.902+0000 7fa652000640 1 -- 172.21.15.32:0/2180800195 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7fa64c07d6e0 con 0x7fa65c0ced30 2024-08-03T20:47:33.073 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:33.073+0000 7fa660c00640 1 -- 172.21.15.32:0/2180800195 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) -- 0x7fa65c0cc860 con 0x7fa65c0ced30 2024-08-03T20:47:33.904 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:33.902+0000 7fa652000640 1 -- 172.21.15.32:0/2180800195 <== mon.0 v2:172.21.15.32:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v18) ==== 138+0+0 (secure 0 0 0) 0x7fa64c065a50 con 0x7fa65c0ced30 2024-08-03T20:47:33.904 INFO:teuthology.orchestra.run.smithi032.stderr:pool 'cephfs_metadata' created 2024-08-03T20:47:33.907 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:33.905+0000 7fa660c00640 1 -- 172.21.15.32:0/2180800195 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fa63c076fb0 msgr2=0x7fa63c0794c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:47:33.907 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:33.905+0000 7fa660c00640 1 --2- 172.21.15.32:0/2180800195 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fa63c076fb0 0x7fa63c0794c0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fa65c0cc830 tx=0x7fa648006b70 comp rx=0 tx=0).stop 2024-08-03T20:47:33.907 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:33.905+0000 7fa660c00640 1 -- 172.21.15.32:0/2180800195 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa65c0ced30 msgr2=0x7fa65c1f5740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:47:33.907 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:33.905+0000 7fa660c00640 1 --2- 172.21.15.32:0/2180800195 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa65c0ced30 0x7fa65c1f5740 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fa64c00c430 tx=0x7fa64c001d90 comp rx=0 tx=0).stop 2024-08-03T20:47:33.907 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:33.905+0000 7fa660c00640 1 -- 172.21.15.32:0/2180800195 shutdown_connections 2024-08-03T20:47:33.907 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:33.905+0000 7fa660c00640 1 --2- 172.21.15.32:0/2180800195 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fa63c076fb0 0x7fa63c0794c0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:33.907 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:33.905+0000 7fa660c00640 1 --2- 172.21.15.32:0/2180800195 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa65c140a30 0x7fa65c134720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:33.907 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:33.905+0000 7fa660c00640 1 --2- 172.21.15.32:0/2180800195 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fa65c0ca490 0x7fa65c0ca8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:33.907 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:33.905+0000 7fa660c00640 1 --2- 172.21.15.32:0/2180800195 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa65c0ced30 0x7fa65c1f5740 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:33.907 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:33.905+0000 7fa660c00640 1 -- 172.21.15.32:0/2180800195 >> 172.21.15.32:0/2180800195 conn(0x7fa65c0b1bd0 msgr2=0x7fa65c15b7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:47:33.908 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:33.906+0000 7fa660c00640 1 -- 172.21.15.32:0/2180800195 shutdown_connections 2024-08-03T20:47:33.908 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:33.906+0000 7fa660c00640 1 -- 172.21.15.32:0/2180800195 wait complete. 2024-08-03T20:47:33.938 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-08-03T20:47:34.179 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:34.177+0000 7f67c5000640 1 -- 172.21.15.32:0/3271696112 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f67a4004db0 msgr2=0x7f67a4005280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:47:34.179 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:34.177+0000 7f67c5000640 1 --2- 172.21.15.32:0/3271696112 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f67a4004db0 0x7f67a4005280 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f67b8009ed0 tx=0x7f67a4005830 comp rx=0 tx=0).stop 2024-08-03T20:47:34.179 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:34.177+0000 7f67be800640 1 -- 172.21.15.32:0/3271696112 reap_dead start 2024-08-03T20:47:34.179 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:34.177+0000 7f67c5000640 1 -- 172.21.15.32:0/3271696112 shutdown_connections 2024-08-03T20:47:34.179 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:34.177+0000 7f67c5000640 1 -- 172.21.15.32:0/3271696112 >> 172.21.15.32:0/3271696112 conn(0x7f67c00b1bd0 msgr2=0x7f67c00b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:47:34.179 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:34.177+0000 7f67c5000640 1 -- 172.21.15.32:0/3271696112 shutdown_connections 2024-08-03T20:47:34.179 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:34.177+0000 7f67c5000640 1 -- 172.21.15.32:0/3271696112 wait complete. 2024-08-03T20:47:34.180 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:34.179+0000 7f67c5000640 1 Processor -- start 2024-08-03T20:47:34.180 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:34.179+0000 7f67c5000640 1 -- start start 2024-08-03T20:47:34.182 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:34.181+0000 7f67c5000640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f67a4004db0 0x7f67c0157bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:47:34.183 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:34.181+0000 7f67c5000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f67c01f6340 0x7f67c0150c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:47:34.183 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:34.181+0000 7f67c5000640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f67c01545f0 0x7f67c0154a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:47:34.183 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:34.181+0000 7f67c5000640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f67c015d8b0 con 0x7f67c01f6340 2024-08-03T20:47:34.183 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:34.181+0000 7f67c5000640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f67c00ccc80 con 0x7f67a4004db0 2024-08-03T20:47:34.183 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:34.181+0000 7f67c5000640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7f67c00cce00 con 0x7f67c01545f0 2024-08-03T20:47:34.183 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:34.181+0000 7f67bd400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f67c01f6340 0x7f67c0150c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:47:34.183 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:34.181+0000 7f67bd400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f67c01f6340 0x7f67c0150c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:40592/0 (socket says 172.21.15.32:40592) 2024-08-03T20:47:34.183 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:34.181+0000 7f67bd400640 1 -- 172.21.15.32:0/1069270091 learned_addr learned my addr 172.21.15.32:0/1069270091 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:47:34.183 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:34.181+0000 7f67bd400640 1 -- 172.21.15.32:0/1069270091 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f67c01545f0 msgr2=0x7f67c0154a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:47:34.183 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:34.181+0000 7f67bd400640 1 --2- 172.21.15.32:0/1069270091 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f67c01545f0 0x7f67c0154a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:34.183 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:34.181+0000 7f67bd400640 1 -- 172.21.15.32:0/1069270091 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f67a4004db0 msgr2=0x7f67c0157bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:47:34.183 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:34.181+0000 7f67bd400640 1 --2- 172.21.15.32:0/1069270091 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f67a4004db0 0x7f67c0157bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:34.184 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:34.181+0000 7f67bd400640 1 -- 172.21.15.32:0/1069270091 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f67c01f6750 con 0x7f67c01f6340 2024-08-03T20:47:34.184 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:34.181+0000 7f67bd400640 1 --2- 172.21.15.32:0/1069270091 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f67c01f6340 0x7f67c0150c20 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f67ac003f50 tx=0x7f67ac0018c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:47:34.184 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:34.182+0000 7f67b6000640 1 -- 172.21.15.32:0/1069270091 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67ac015070 con 0x7f67c01f6340 2024-08-03T20:47:34.184 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:34.182+0000 7f67b6000640 1 -- 172.21.15.32:0/1069270091 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f67ac011900 con 0x7f67c01f6340 2024-08-03T20:47:34.185 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:34.182+0000 7f67b6000640 1 -- 172.21.15.32:0/1069270091 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67ac011d50 con 0x7f67c01f6340 2024-08-03T20:47:34.185 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:34.182+0000 7f67c5000640 1 -- 172.21.15.32:0/1069270091 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f67c0154f40 con 0x7f67c01f6340 2024-08-03T20:47:34.185 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:34.182+0000 7f67c5000640 1 -- 172.21.15.32:0/1069270091 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f67c0158370 con 0x7f67c01f6340 2024-08-03T20:47:34.185 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:34.183+0000 7f67c5000640 1 -- 172.21.15.32:0/1069270091 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f67c00cd360 con 0x7f67c01f6340 2024-08-03T20:47:34.186 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:34.184+0000 7f67b6000640 1 -- 172.21.15.32:0/1069270091 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f67ac018070 con 0x7f67c01f6340 2024-08-03T20:47:34.187 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:34.185+0000 7f67b6000640 1 --2- 172.21.15.32:0/1069270091 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f67a0077040 0x7f67a0079550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:47:34.187 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:34.185+0000 7f67b6000640 1 -- 172.21.15.32:0/1069270091 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(18..18 src has 1..18) ==== 5227+0+0 (secure 0 0 0) 0x7f67ac0990d0 con 0x7f67c01f6340 2024-08-03T20:47:34.187 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:34.186+0000 7f67bde00640 1 --2- 172.21.15.32:0/1069270091 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f67a0077040 0x7f67a0079550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:47:34.188 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:34.186+0000 7f67bde00640 1 --2- 172.21.15.32:0/1069270091 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f67a0077040 0x7f67a0079550 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f67b0001c20 tx=0x7f67b0001b70 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:47:34.192 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:34.190+0000 7f67b6000640 1 -- 172.21.15.32:0/1069270091 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7f67c0154f40 con 0x7f67c01f6340 2024-08-03T20:47:34.394 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:34.393+0000 7f67c5000640 1 -- 172.21.15.32:0/1069270091 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) -- 0x7f67c0158100 con 0x7f67c01f6340 2024-08-03T20:47:34.911 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:34.909+0000 7f67b6000640 1 -- 172.21.15.32:0/1069270091 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v19) ==== 170+0+0 (secure 0 0 0) 0x7f67ac062410 con 0x7f67c01f6340 2024-08-03T20:47:34.911 INFO:teuthology.orchestra.run.smithi032.stderr:pool 'cephfs_data' created 2024-08-03T20:47:34.914 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:34.912+0000 7f67c5000640 1 -- 172.21.15.32:0/1069270091 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f67a0077040 msgr2=0x7f67a0079550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:47:34.914 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:34.912+0000 7f67c5000640 1 --2- 172.21.15.32:0/1069270091 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f67a0077040 0x7f67a0079550 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f67b0001c20 tx=0x7f67b0001b70 comp rx=0 tx=0).stop 2024-08-03T20:47:34.914 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:34.912+0000 7f67c5000640 1 -- 172.21.15.32:0/1069270091 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f67c01f6340 msgr2=0x7f67c0150c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:47:34.914 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:34.912+0000 7f67c5000640 1 --2- 172.21.15.32:0/1069270091 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f67c01f6340 0x7f67c0150c20 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f67ac003f50 tx=0x7f67ac0018c0 comp rx=0 tx=0).stop 2024-08-03T20:47:34.915 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:34.913+0000 7f67c5000640 1 -- 172.21.15.32:0/1069270091 shutdown_connections 2024-08-03T20:47:34.915 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:34.913+0000 7f67c5000640 1 --2- 172.21.15.32:0/1069270091 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f67a0077040 0x7f67a0079550 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:34.915 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:34.913+0000 7f67c5000640 1 --2- 172.21.15.32:0/1069270091 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f67c01545f0 0x7f67c0154a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:34.915 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:34.913+0000 7f67c5000640 1 --2- 172.21.15.32:0/1069270091 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f67c01f6340 0x7f67c0150c20 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:34.915 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:34.913+0000 7f67c5000640 1 --2- 172.21.15.32:0/1069270091 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f67a4004db0 0x7f67c0157bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:34.915 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:34.913+0000 7f67c5000640 1 -- 172.21.15.32:0/1069270091 >> 172.21.15.32:0/1069270091 conn(0x7f67c00b1bd0 msgr2=0x7f67c015dba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:47:34.915 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:34.913+0000 7f67c5000640 1 -- 172.21.15.32:0/1069270091 shutdown_connections 2024-08-03T20:47:34.915 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:34.913+0000 7f67c5000640 1 -- 172.21.15.32:0/1069270091 wait complete. 2024-08-03T20:47:34.953 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-08-03T20:47:35.184 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:35.182+0000 7fef62800640 1 -- 172.21.15.32:0/1555830599 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fef5c160350 msgr2=0x7fef5c06e630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:47:35.184 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:35.182+0000 7fef62800640 1 --2- 172.21.15.32:0/1555830599 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fef5c160350 0x7fef5c06e630 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fef54005fd0 tx=0x7fef5402a8b0 comp rx=0 tx=0).stop 2024-08-03T20:47:35.184 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:35.182+0000 7fef62800640 1 -- 172.21.15.32:0/1555830599 shutdown_connections 2024-08-03T20:47:35.184 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:35.182+0000 7fef62800640 1 --2- 172.21.15.32:0/1555830599 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fef5c160350 0x7fef5c06e630 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:35.184 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:35.182+0000 7fef62800640 1 --2- 172.21.15.32:0/1555830599 >> v2:172.21.15.118:3300/0 conn(0x7fef5c159ff0 0x7fef5c15fe10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:35.184 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:35.182+0000 7fef62800640 1 -- 172.21.15.32:0/1555830599 >> 172.21.15.32:0/1555830599 conn(0x7fef5c0cfd80 msgr2=0x7fef5c0d0190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:47:35.184 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:35.182+0000 7fef62800640 1 -- 172.21.15.32:0/1555830599 shutdown_connections 2024-08-03T20:47:35.184 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:35.182+0000 7fef62800640 1 -- 172.21.15.32:0/1555830599 wait complete. 2024-08-03T20:47:35.186 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:35.184+0000 7fef62800640 1 Processor -- start 2024-08-03T20:47:35.186 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:35.184+0000 7fef62800640 1 -- start start 2024-08-03T20:47:35.188 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:35.186+0000 7fef62800640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fef5c159ff0 0x7fef5c1f55a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:47:35.189 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:35.186+0000 7fef62800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fef5c160350 0x7fef5c0b5300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:47:35.189 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:35.186+0000 7fef62800640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fef5c1609d0 0x7fef5c15c820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:47:35.189 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:35.186+0000 7fef62800640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fef5c06ef00 con 0x7fef5c160350 2024-08-03T20:47:35.189 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:35.186+0000 7fef62800640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7fef5c06ed80 con 0x7fef5c1609d0 2024-08-03T20:47:35.189 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:35.186+0000 7fef62800640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7fef5c06f080 con 0x7fef5c159ff0 2024-08-03T20:47:35.189 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:35.186+0000 7fef60a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fef5c160350 0x7fef5c0b5300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:47:35.189 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:35.186+0000 7fef60a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fef5c160350 0x7fef5c0b5300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:40606/0 (socket says 172.21.15.32:40606) 2024-08-03T20:47:35.189 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:35.186+0000 7fef60a00640 1 -- 172.21.15.32:0/1512272399 learned_addr learned my addr 172.21.15.32:0/1512272399 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:47:35.189 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:35.186+0000 7fef61e00640 1 --2- 172.21.15.32:0/1512272399 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fef5c1609d0 0x7fef5c15c820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:47:35.189 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:35.186+0000 7fef61400640 1 --2- 172.21.15.32:0/1512272399 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fef5c159ff0 0x7fef5c1f55a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:47:35.189 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:35.187+0000 7fef60a00640 1 -- 172.21.15.32:0/1512272399 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fef5c159ff0 msgr2=0x7fef5c1f55a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:47:35.189 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:35.187+0000 7fef60a00640 1 --2- 172.21.15.32:0/1512272399 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fef5c159ff0 0x7fef5c1f55a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:35.190 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:35.187+0000 7fef60a00640 1 -- 172.21.15.32:0/1512272399 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fef5c1609d0 msgr2=0x7fef5c15c820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:47:35.190 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:35.187+0000 7fef60a00640 1 --2- 172.21.15.32:0/1512272399 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fef5c1609d0 0x7fef5c15c820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:35.190 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:35.187+0000 7fef60a00640 1 -- 172.21.15.32:0/1512272399 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fef5c1f5ef0 con 0x7fef5c160350 2024-08-03T20:47:35.190 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:35.187+0000 7fef60a00640 1 --2- 172.21.15.32:0/1512272399 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fef5c160350 0x7fef5c0b5300 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fef4800e9f0 tx=0x7fef4800eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:47:35.190 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:35.187+0000 7fef59600640 1 -- 172.21.15.32:0/1512272399 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef4800cd80 con 0x7fef5c160350 2024-08-03T20:47:35.190 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:35.187+0000 7fef59600640 1 -- 172.21.15.32:0/1512272399 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fef48004020 con 0x7fef5c160350 2024-08-03T20:47:35.190 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:35.187+0000 7fef62800640 1 -- 172.21.15.32:0/1512272399 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fef5c1f61d0 con 0x7fef5c160350 2024-08-03T20:47:35.190 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:35.187+0000 7fef59600640 1 -- 172.21.15.32:0/1512272399 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef48004470 con 0x7fef5c160350 2024-08-03T20:47:35.190 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:35.187+0000 7fef62800640 1 -- 172.21.15.32:0/1512272399 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fef5c0c8550 con 0x7fef5c160350 2024-08-03T20:47:35.191 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:35.188+0000 7fef62800640 1 -- 172.21.15.32:0/1512272399 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fef5c06f5a0 con 0x7fef5c160350 2024-08-03T20:47:35.191 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:35.190+0000 7fef59600640 1 -- 172.21.15.32:0/1512272399 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7fef480035f0 con 0x7fef5c160350 2024-08-03T20:47:35.192 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:35.191+0000 7fef59600640 1 --2- 172.21.15.32:0/1512272399 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fef3c077010 0x7fef3c079520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:47:35.192 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:35.191+0000 7fef59600640 1 -- 172.21.15.32:0/1512272399 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(19..19 src has 1..19) ==== 5609+0+0 (secure 0 0 0) 0x7fef480990e0 con 0x7fef5c160350 2024-08-03T20:47:35.192 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:35.191+0000 7fef61400640 1 --2- 172.21.15.32:0/1512272399 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fef3c077010 0x7fef3c079520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:47:35.193 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:35.192+0000 7fef61400640 1 --2- 172.21.15.32:0/1512272399 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fef3c077010 0x7fef3c079520 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fef50007d60 tx=0x7fef50007cf0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:47:35.196 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:35.195+0000 7fef59600640 1 -- 172.21.15.32:0/1512272399 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7fef48061c50 con 0x7fef5c160350 2024-08-03T20:47:35.441 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:35.440+0000 7fef62800640 1 -- 172.21.15.32:0/1512272399 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) -- 0x7fef5c0d09e0 con 0x7fef5c160350 2024-08-03T20:47:35.927 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:35.925+0000 7fef59600640 1 -- 172.21.15.32:0/1512272399 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 2 and data pool 3 v4) ==== 172+0+0 (secure 0 0 0) 0x7fef480661a0 con 0x7fef5c160350 2024-08-03T20:47:35.927 INFO:teuthology.orchestra.run.smithi032.stderr:new fs with metadata pool 2 and data pool 3 2024-08-03T20:47:35.944 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:35.942+0000 7fef62800640 1 -- 172.21.15.32:0/1512272399 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fef3c077010 msgr2=0x7fef3c079520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:47:35.944 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:35.942+0000 7fef62800640 1 --2- 172.21.15.32:0/1512272399 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fef3c077010 0x7fef3c079520 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fef50007d60 tx=0x7fef50007cf0 comp rx=0 tx=0).stop 2024-08-03T20:47:35.944 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:35.942+0000 7fef62800640 1 -- 172.21.15.32:0/1512272399 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fef5c160350 msgr2=0x7fef5c0b5300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:47:35.944 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:35.942+0000 7fef62800640 1 --2- 172.21.15.32:0/1512272399 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fef5c160350 0x7fef5c0b5300 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fef4800e9f0 tx=0x7fef4800eeb0 comp rx=0 tx=0).stop 2024-08-03T20:47:35.944 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:35.942+0000 7fef62800640 1 -- 172.21.15.32:0/1512272399 shutdown_connections 2024-08-03T20:47:35.944 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:35.942+0000 7fef62800640 1 --2- 172.21.15.32:0/1512272399 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fef3c077010 0x7fef3c079520 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:35.944 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:35.942+0000 7fef62800640 1 --2- 172.21.15.32:0/1512272399 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fef5c1609d0 0x7fef5c15c820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:35.944 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:35.942+0000 7fef62800640 1 --2- 172.21.15.32:0/1512272399 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fef5c160350 0x7fef5c0b5300 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:35.944 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:35.942+0000 7fef62800640 1 --2- 172.21.15.32:0/1512272399 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fef5c159ff0 0x7fef5c1f55a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:35.944 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:35.942+0000 7fef62800640 1 -- 172.21.15.32:0/1512272399 >> 172.21.15.32:0/1512272399 conn(0x7fef5c0cfd80 msgr2=0x7fef5c0ad8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:47:35.945 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:35.943+0000 7fef62800640 1 -- 172.21.15.32:0/1512272399 shutdown_connections 2024-08-03T20:47:35.945 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:35.943+0000 7fef62800640 1 -- 172.21.15.32:0/1512272399 wait complete. 2024-08-03T20:47:35.979 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-08-03T20:47:36.207 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:36.205+0000 7f6002400640 1 -- 172.21.15.32:0/1683866144 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f5ffc144f90 msgr2=0x7f5ffc074570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:47:36.207 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:36.205+0000 7f6002400640 1 --2- 172.21.15.32:0/1683866144 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f5ffc144f90 0x7f5ffc074570 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f5fe8009830 tx=0x7f5fe802a510 comp rx=0 tx=0).stop 2024-08-03T20:47:36.207 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:36.205+0000 7f6002400640 1 -- 172.21.15.32:0/1683866144 shutdown_connections 2024-08-03T20:47:36.207 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:36.205+0000 7f6002400640 1 --2- 172.21.15.32:0/1683866144 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f5ffc15fff0 0x7f5ffc160400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:36.207 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:36.205+0000 7f6002400640 1 --2- 172.21.15.32:0/1683866144 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f5ffc144f90 0x7f5ffc074570 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:36.208 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:36.205+0000 7f6002400640 1 -- 172.21.15.32:0/1683866144 >> 172.21.15.32:0/1683866144 conn(0x7f5ffc0b1bd0 msgr2=0x7f5ffc0b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:47:36.208 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:36.205+0000 7f6002400640 1 -- 172.21.15.32:0/1683866144 shutdown_connections 2024-08-03T20:47:36.208 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:36.205+0000 7f6002400640 1 -- 172.21.15.32:0/1683866144 wait complete. 2024-08-03T20:47:36.209 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:36.207+0000 7f6002400640 1 Processor -- start 2024-08-03T20:47:36.209 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:36.208+0000 7f6002400640 1 -- start start 2024-08-03T20:47:36.211 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:36.209+0000 7f6002400640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f5ffc06e930 0x7f5ffc15bce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:47:36.211 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:36.209+0000 7f6002400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5ffc144f90 0x7f5ffc0c9bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:47:36.211 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:36.209+0000 7f6002400640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f5ffc15fff0 0x7f5ffc0b4280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:47:36.211 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:36.209+0000 7f6002400640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f5ffc0cf840 con 0x7f5ffc144f90 2024-08-03T20:47:36.211 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:36.209+0000 7f6002400640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f5ffc145f30 con 0x7f5ffc06e930 2024-08-03T20:47:36.212 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:36.209+0000 7f6002400640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7f5ffc0cf9c0 con 0x7f5ffc15fff0 2024-08-03T20:47:36.212 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:36.209+0000 7f5ffaa00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5ffc144f90 0x7f5ffc0c9bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:47:36.212 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:36.209+0000 7f5ffaa00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5ffc144f90 0x7f5ffc0c9bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:40610/0 (socket says 172.21.15.32:40610) 2024-08-03T20:47:36.212 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:36.209+0000 7f5ffaa00640 1 -- 172.21.15.32:0/2155240908 learned_addr learned my addr 172.21.15.32:0/2155240908 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:47:36.212 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:36.209+0000 7f5ffb400640 1 --2- 172.21.15.32:0/2155240908 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f5ffc06e930 0x7f5ffc15bce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:47:36.212 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:36.209+0000 7f5ffbe00640 1 --2- 172.21.15.32:0/2155240908 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f5ffc15fff0 0x7f5ffc0b4280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:47:36.212 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:36.210+0000 7f5ffaa00640 1 -- 172.21.15.32:0/2155240908 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f5ffc15fff0 msgr2=0x7f5ffc0b4280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:47:36.212 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:36.210+0000 7f5ffaa00640 1 --2- 172.21.15.32:0/2155240908 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f5ffc15fff0 0x7f5ffc0b4280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:36.212 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:36.210+0000 7f5ffaa00640 1 -- 172.21.15.32:0/2155240908 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f5ffc06e930 msgr2=0x7f5ffc15bce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:47:36.212 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:36.210+0000 7f5ffaa00640 1 --2- 172.21.15.32:0/2155240908 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f5ffc06e930 0x7f5ffc15bce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:36.212 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:36.210+0000 7f5ffaa00640 1 -- 172.21.15.32:0/2155240908 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5ffc15c220 con 0x7f5ffc144f90 2024-08-03T20:47:36.212 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:36.210+0000 7f5ffaa00640 1 --2- 172.21.15.32:0/2155240908 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5ffc144f90 0x7f5ffc0c9bb0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f5fe802ab00 tx=0x7f5fe8033c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:47:36.213 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:36.210+0000 7f5ff3400640 1 -- 172.21.15.32:0/2155240908 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5fe8005c90 con 0x7f5ffc144f90 2024-08-03T20:47:36.213 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:36.210+0000 7f5ff3400640 1 -- 172.21.15.32:0/2155240908 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f5fe8031e50 con 0x7f5ffc144f90 2024-08-03T20:47:36.213 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:36.210+0000 7f6002400640 1 -- 172.21.15.32:0/2155240908 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5ffc1e6cd0 con 0x7f5ffc144f90 2024-08-03T20:47:36.214 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:36.210+0000 7f5ff3400640 1 -- 172.21.15.32:0/2155240908 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5fe8005260 con 0x7f5ffc144f90 2024-08-03T20:47:36.214 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:36.211+0000 7f6002400640 1 -- 172.21.15.32:0/2155240908 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5ffc1e7290 con 0x7f5ffc144f90 2024-08-03T20:47:36.214 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:36.212+0000 7f6002400640 1 -- 172.21.15.32:0/2155240908 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5ffc160cc0 con 0x7f5ffc144f90 2024-08-03T20:47:36.214 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:36.213+0000 7f5ff3400640 1 -- 172.21.15.32:0/2155240908 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f5fe8005400 con 0x7f5ffc144f90 2024-08-03T20:47:36.216 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:36.214+0000 7f5ff3400640 1 --2- 172.21.15.32:0/2155240908 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f5fdc077040 0x7f5fdc079550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:47:36.216 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:36.215+0000 7f5ff3400640 1 -- 172.21.15.32:0/2155240908 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f5fe80b98a0 con 0x7f5ffc144f90 2024-08-03T20:47:36.216 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:36.215+0000 7f5ffb400640 1 --2- 172.21.15.32:0/2155240908 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f5fdc077040 0x7f5fdc079550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:47:36.216 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:36.215+0000 7f5ffb400640 1 --2- 172.21.15.32:0/2155240908 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f5fdc077040 0x7f5fdc079550 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f5fec007fd0 tx=0x7f5fec009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:47:36.220 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:36.219+0000 7f5ff3400640 1 -- 172.21.15.32:0/2155240908 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7f5ffc15c220 con 0x7f5ffc144f90 2024-08-03T20:47:36.228 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-08-03T20:47:36.232 DEBUG:teuthology.orchestra.run.smithi118:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-08-03T20:47:36.431 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:36.430+0000 7f6002400640 1 -- 172.21.15.32:0/2155240908 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f5ffc069090 con 0x7f5ffc144f90 2024-08-03T20:47:36.432 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:36.431+0000 7f5ff3400640 1 -- 172.21.15.32:0/2155240908 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) ==== 74+0+16558 (secure 0 0 0) 0x7f5fe803f030 con 0x7f5ffc144f90 2024-08-03T20:47:36.433 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-03T20:47:36.433 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":20,"fsid":"40ab141a-01d7-49b9-ae3c-2edec9876e95","created":"2024-08-03T20:47:14.957362+0000","modified":"2024-08-03T20:47:35.442537+0000","last_up_change":"2024-08-03T20:47:28.874451+0000","last_in_change":"2024-08-03T20:47:20.079442+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-08-03T20:47:28.261300+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-08-03T20:47:33.085381+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-08-03T20:47:34.405408+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5a83e759-9035-438b-889a-0a6af1f0a0d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6816","nonce":3066156116},{"type":"v1","addr":"172.21.15.32:6817","nonce":3066156116}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6818","nonce":3066156116},{"type":"v1","addr":"172.21.15.32:6819","nonce":3066156116}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6822","nonce":3066156116},{"type":"v1","addr":"172.21.15.32:6823","nonce":3066156116}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6820","nonce":3066156116},{"type":"v1","addr":"172.21.15.32:6821","nonce":3066156116}]},"public_addr":"172.21.15.32:6817/3066156116","cluster_addr":"172.21.15.32:6819/3066156116","heartbeat_back_addr":"172.21.15.32:6823/3066156116","heartbeat_front_addr":"172.21.15.32:6821/3066156116","state":["exists","up"]},{"osd":1,"uuid":"d7c5c210-da4f-49f9-b9e7-1dfb02daa4bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6824","nonce":1551241759},{"type":"v1","addr":"172.21.15.32:6825","nonce":1551241759}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6826","nonce":1551241759},{"type":"v1","addr":"172.21.15.32:6827","nonce":1551241759}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6830","nonce":1551241759},{"type":"v1","addr":"172.21.15.32:6831","nonce":1551241759}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6828","nonce":1551241759},{"type":"v1","addr":"172.21.15.32:6829","nonce":1551241759}]},"public_addr":"172.21.15.32:6825/1551241759","cluster_addr":"172.21.15.32:6827/1551241759","heartbeat_back_addr":"172.21.15.32:6831/1551241759","heartbeat_front_addr":"172.21.15.32:6829/1551241759","state":["exists","up"]},{"osd":2,"uuid":"57f2e6de-a201-45c2-8981-8e56a4875ebd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6808","nonce":3806708944},{"type":"v1","addr":"172.21.15.32:6809","nonce":3806708944}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6810","nonce":3806708944},{"type":"v1","addr":"172.21.15.32:6811","nonce":3806708944}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6814","nonce":3806708944},{"type":"v1","addr":"172.21.15.32:6815","nonce":3806708944}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6812","nonce":3806708944},{"type":"v1","addr":"172.21.15.32:6813","nonce":3806708944}]},"public_addr":"172.21.15.32:6809/3806708944","cluster_addr":"172.21.15.32:6811/3806708944","heartbeat_back_addr":"172.21.15.32:6815/3806708944","heartbeat_front_addr":"172.21.15.32:6813/3806708944","state":["exists","up"]},{"osd":3,"uuid":"6439a280-da45-4148-8130-f16323935ff3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6800","nonce":9088303},{"type":"v1","addr":"172.21.15.32:6801","nonce":9088303}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6802","nonce":9088303},{"type":"v1","addr":"172.21.15.32:6803","nonce":9088303}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6806","nonce":9088303},{"type":"v1","addr":"172.21.15.32:6807","nonce":9088303}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6804","nonce":9088303},{"type":"v1","addr":"172.21.15.32:6805","nonce":9088303}]},"public_addr":"172.21.15.32:6801/9088303","cluster_addr":"172.21.15.32:6803/9088303","heartbeat_back_addr":"172.21.15.32:6807/9088303","heartbeat_front_addr":"172.21.15.32:6805/9088303","state":["exists","up"]},{"osd":4,"uuid":"4f902eff-1e18-418d-82c9-2151c69524fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6802","nonce":864752275},{"type":"v1","addr":"172.21.15.118:6803","nonce":864752275}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6804","nonce":864752275},{"type":"v1","addr":"172.21.15.118:6805","nonce":864752275}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6808","nonce":864752275},{"type":"v1","addr":"172.21.15.118:6809","nonce":864752275}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6806","nonce":864752275},{"type":"v1","addr":"172.21.15.118:6807","nonce":864752275}]},"public_addr":"172.21.15.118:6803/864752275","cluster_addr":"172.21.15.118:6805/864752275","heartbeat_back_addr":"172.21.15.118:6809/864752275","heartbeat_front_addr":"172.21.15.118:6807/864752275","state":["exists","up"]},{"osd":5,"uuid":"bfd9feac-dd06-458c-91b9-af7f175f612c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6810","nonce":2724891770},{"type":"v1","addr":"172.21.15.118:6811","nonce":2724891770}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6812","nonce":2724891770},{"type":"v1","addr":"172.21.15.118:6813","nonce":2724891770}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6816","nonce":2724891770},{"type":"v1","addr":"172.21.15.118:6817","nonce":2724891770}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6814","nonce":2724891770},{"type":"v1","addr":"172.21.15.118:6815","nonce":2724891770}]},"public_addr":"172.21.15.118:6811/2724891770","cluster_addr":"172.21.15.118:6813/2724891770","heartbeat_back_addr":"172.21.15.118:6817/2724891770","heartbeat_front_addr":"172.21.15.118:6815/2724891770","state":["exists","up"]},{"osd":6,"uuid":"5ed5a279-9d8e-416d-bbc7-7edd3e6d5705","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6825","nonce":801947386},{"type":"v1","addr":"172.21.15.118:6827","nonce":801947386}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6828","nonce":801947386},{"type":"v1","addr":"172.21.15.118:6829","nonce":801947386}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6832","nonce":801947386},{"type":"v1","addr":"172.21.15.118:6833","nonce":801947386}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6830","nonce":801947386},{"type":"v1","addr":"172.21.15.118:6831","nonce":801947386}]},"public_addr":"172.21.15.118:6827/801947386","cluster_addr":"172.21.15.118:6829/801947386","heartbeat_back_addr":"172.21.15.118:6833/801947386","heartbeat_front_addr":"172.21.15.118:6831/801947386","state":["exists","up"]},{"osd":7,"uuid":"f2086c47-875e-4451-a436-2008d1ea16af","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6818","nonce":653000048},{"type":"v1","addr":"172.21.15.118:6819","nonce":653000048}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6820","nonce":653000048},{"type":"v1","addr":"172.21.15.118:6821","nonce":653000048}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6824","nonce":653000048},{"type":"v1","addr":"172.21.15.118:6826","nonce":653000048}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6822","nonce":653000048},{"type":"v1","addr":"172.21.15.118:6823","nonce":653000048}]},"public_addr":"172.21.15.118:6819/653000048","cluster_addr":"172.21.15.118:6821/653000048","heartbeat_back_addr":"172.21.15.118:6826/653000048","heartbeat_front_addr":"172.21.15.118:6823/653000048","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-03T20:47:25.491994+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-03T20:47:25.587358+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-03T20:47:25.454175+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-03T20:47:25.377734+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-03T20:47:25.412351+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-03T20:47:25.499815+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-03T20:47:25.575044+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-03T20:47:25.552383+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-08-03T20:47:36.435 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:36.433+0000 7f6002400640 1 -- 172.21.15.32:0/2155240908 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f5fdc077040 msgr2=0x7f5fdc079550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:47:36.435 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:36.433+0000 7f6002400640 1 --2- 172.21.15.32:0/2155240908 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f5fdc077040 0x7f5fdc079550 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f5fec007fd0 tx=0x7f5fec009040 comp rx=0 tx=0).stop 2024-08-03T20:47:36.435 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:36.433+0000 7f6002400640 1 -- 172.21.15.32:0/2155240908 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5ffc144f90 msgr2=0x7f5ffc0c9bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:47:36.435 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:36.433+0000 7f6002400640 1 --2- 172.21.15.32:0/2155240908 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5ffc144f90 0x7f5ffc0c9bb0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f5fe802ab00 tx=0x7f5fe8033c20 comp rx=0 tx=0).stop 2024-08-03T20:47:36.435 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:36.433+0000 7f6002400640 1 -- 172.21.15.32:0/2155240908 shutdown_connections 2024-08-03T20:47:36.435 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:36.433+0000 7f6002400640 1 --2- 172.21.15.32:0/2155240908 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f5fdc077040 0x7f5fdc079550 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:36.435 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:36.433+0000 7f6002400640 1 --2- 172.21.15.32:0/2155240908 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f5ffc15fff0 0x7f5ffc0b4280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:36.435 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:36.433+0000 7f6002400640 1 --2- 172.21.15.32:0/2155240908 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5ffc144f90 0x7f5ffc0c9bb0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:36.436 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:36.433+0000 7f6002400640 1 --2- 172.21.15.32:0/2155240908 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f5ffc06e930 0x7f5ffc15bce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:36.436 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:36.433+0000 7f6002400640 1 -- 172.21.15.32:0/2155240908 >> 172.21.15.32:0/2155240908 conn(0x7f5ffc0b1bd0 msgr2=0x7f5ffc0b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:47:36.436 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:36.433+0000 7f6002400640 1 -- 172.21.15.32:0/2155240908 shutdown_connections 2024-08-03T20:47:36.436 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:36.434+0000 7f6002400640 1 -- 172.21.15.32:0/2155240908 wait complete. 2024-08-03T20:47:36.457 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-08-03T20:47:36.687 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:36.685+0000 7fc33aa00640 1 -- 172.21.15.32:0/2314440546 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fc3340b3d70 msgr2=0x7fc3340b4180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:47:36.687 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:36.685+0000 7fc33aa00640 1 --2- 172.21.15.32:0/2314440546 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fc3340b3d70 0x7fc3340b4180 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fc31c009830 tx=0x7fc31c02a510 comp rx=0 tx=0).stop 2024-08-03T20:47:36.688 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:36.685+0000 7fc33aa00640 1 -- 172.21.15.32:0/2314440546 shutdown_connections 2024-08-03T20:47:36.688 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:36.685+0000 7fc33aa00640 1 --2- 172.21.15.32:0/2314440546 >> v2:172.21.15.118:3300/0 conn(0x7fc3340b46c0 0x7fc3340b4ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:36.688 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:36.685+0000 7fc33aa00640 1 --2- 172.21.15.32:0/2314440546 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fc3340b3d70 0x7fc3340b4180 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:36.688 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:36.685+0000 7fc33aa00640 1 -- 172.21.15.32:0/2314440546 >> 172.21.15.32:0/2314440546 conn(0x7fc33409d640 msgr2=0x7fc3340d4310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:47:36.688 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:36.686+0000 7fc33aa00640 1 -- 172.21.15.32:0/2314440546 shutdown_connections 2024-08-03T20:47:36.688 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:36.686+0000 7fc33aa00640 1 -- 172.21.15.32:0/2314440546 wait complete. 2024-08-03T20:47:36.689 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:36.688+0000 7fc33aa00640 1 Processor -- start 2024-08-03T20:47:36.689 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:36.688+0000 7fc33aa00640 1 -- start start 2024-08-03T20:47:36.692 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:36.689+0000 7fc33aa00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc3340b3d70 0x7fc3341f6470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:47:36.692 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:36.689+0000 7fc33aa00640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fc3340b46c0 0x7fc33415ea80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:47:36.692 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:36.689+0000 7fc33aa00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc3340d0d30 0x7fc33415efc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:47:36.692 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:36.689+0000 7fc33aa00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fc3340d4cc0 con 0x7fc3340d0d30 2024-08-03T20:47:36.692 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:36.689+0000 7fc33aa00640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7fc3340d4b40 con 0x7fc3340b3d70 2024-08-03T20:47:36.692 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:36.689+0000 7fc33aa00640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7fc3340b5340 con 0x7fc3340b46c0 2024-08-03T20:47:36.693 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:36.690+0000 7fc338a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc3340d0d30 0x7fc33415efc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:47:36.693 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:36.690+0000 7fc338a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc3340d0d30 0x7fc33415efc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:40618/0 (socket says 172.21.15.32:40618) 2024-08-03T20:47:36.693 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:36.690+0000 7fc338a00640 1 -- 172.21.15.32:0/4266390621 learned_addr learned my addr 172.21.15.32:0/4266390621 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:47:36.693 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:36.690+0000 7fc333e00640 1 --2- 172.21.15.32:0/4266390621 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc3340b3d70 0x7fc3341f6470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:47:36.693 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:36.690+0000 7fc32b400640 1 --2- 172.21.15.32:0/4266390621 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fc3340b46c0 0x7fc33415ea80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:47:36.693 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:36.690+0000 7fc338a00640 1 -- 172.21.15.32:0/4266390621 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fc3340b46c0 msgr2=0x7fc33415ea80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:47:36.693 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:36.690+0000 7fc338a00640 1 --2- 172.21.15.32:0/4266390621 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fc3340b46c0 0x7fc33415ea80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:36.693 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:36.690+0000 7fc338a00640 1 -- 172.21.15.32:0/4266390621 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc3340b3d70 msgr2=0x7fc3341f6470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:47:36.693 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:36.690+0000 7fc338a00640 1 --2- 172.21.15.32:0/4266390621 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc3340b3d70 0x7fc3341f6470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:36.693 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:36.690+0000 7fc338a00640 1 -- 172.21.15.32:0/4266390621 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc3340cb060 con 0x7fc3340d0d30 2024-08-03T20:47:36.693 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:36.690+0000 7fc338a00640 1 --2- 172.21.15.32:0/4266390621 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc3340d0d30 0x7fc33415efc0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fc32c00ec20 tx=0x7fc32c00c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:47:36.693 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:36.691+0000 7fc330c00640 1 -- 172.21.15.32:0/4266390621 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc32c003b70 con 0x7fc3340d0d30 2024-08-03T20:47:36.693 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:36.691+0000 7fc330c00640 1 -- 172.21.15.32:0/4266390621 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fc32c003e00 con 0x7fc3340d0d30 2024-08-03T20:47:36.693 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:36.691+0000 7fc33aa00640 1 -- 172.21.15.32:0/4266390621 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc3341f5990 con 0x7fc3340d0d30 2024-08-03T20:47:36.694 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:36.691+0000 7fc330c00640 1 -- 172.21.15.32:0/4266390621 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc32c004280 con 0x7fc3340d0d30 2024-08-03T20:47:36.694 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:36.691+0000 7fc33aa00640 1 -- 172.21.15.32:0/4266390621 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc33415f580 con 0x7fc3340d0d30 2024-08-03T20:47:36.694 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:36.692+0000 7fc33aa00640 1 -- 172.21.15.32:0/4266390621 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc3340cadd0 con 0x7fc3340d0d30 2024-08-03T20:47:36.694 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:36.693+0000 7fc330c00640 1 -- 172.21.15.32:0/4266390621 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7fc32c004420 con 0x7fc3340d0d30 2024-08-03T20:47:36.695 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:36.694+0000 7fc330c00640 1 --2- 172.21.15.32:0/4266390621 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fc310077010 0x7fc310079520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:47:36.695 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:36.694+0000 7fc330c00640 1 -- 172.21.15.32:0/4266390621 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7fc32c0996c0 con 0x7fc3340d0d30 2024-08-03T20:47:36.696 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:36.695+0000 7fc333e00640 1 --2- 172.21.15.32:0/4266390621 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fc310077010 0x7fc310079520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:47:36.696 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:36.695+0000 7fc333e00640 1 --2- 172.21.15.32:0/4266390621 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fc310077010 0x7fc310079520 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fc324007fd0 tx=0x7fc324001370 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:47:36.703 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:36.702+0000 7fc330c00640 1 -- 172.21.15.32:0/4266390621 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7fc32c0621d0 con 0x7fc3340d0d30 2024-08-03T20:47:36.909 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:36.907+0000 7fc33aa00640 1 -- 172.21.15.32:0/4266390621 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fc33415f7e0 con 0x7fc3340d0d30 2024-08-03T20:47:36.910 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:36.909+0000 7fc330c00640 1 -- 172.21.15.32:0/4266390621 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) ==== 74+0+16558 (secure 0 0 0) 0x7fc32c066720 con 0x7fc3340d0d30 2024-08-03T20:47:36.911 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-03T20:47:36.911 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":20,"fsid":"40ab141a-01d7-49b9-ae3c-2edec9876e95","created":"2024-08-03T20:47:14.957362+0000","modified":"2024-08-03T20:47:35.442537+0000","last_up_change":"2024-08-03T20:47:28.874451+0000","last_in_change":"2024-08-03T20:47:20.079442+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-08-03T20:47:28.261300+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-08-03T20:47:33.085381+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-08-03T20:47:34.405408+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5a83e759-9035-438b-889a-0a6af1f0a0d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6816","nonce":3066156116},{"type":"v1","addr":"172.21.15.32:6817","nonce":3066156116}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6818","nonce":3066156116},{"type":"v1","addr":"172.21.15.32:6819","nonce":3066156116}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6822","nonce":3066156116},{"type":"v1","addr":"172.21.15.32:6823","nonce":3066156116}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6820","nonce":3066156116},{"type":"v1","addr":"172.21.15.32:6821","nonce":3066156116}]},"public_addr":"172.21.15.32:6817/3066156116","cluster_addr":"172.21.15.32:6819/3066156116","heartbeat_back_addr":"172.21.15.32:6823/3066156116","heartbeat_front_addr":"172.21.15.32:6821/3066156116","state":["exists","up"]},{"osd":1,"uuid":"d7c5c210-da4f-49f9-b9e7-1dfb02daa4bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6824","nonce":1551241759},{"type":"v1","addr":"172.21.15.32:6825","nonce":1551241759}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6826","nonce":1551241759},{"type":"v1","addr":"172.21.15.32:6827","nonce":1551241759}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6830","nonce":1551241759},{"type":"v1","addr":"172.21.15.32:6831","nonce":1551241759}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6828","nonce":1551241759},{"type":"v1","addr":"172.21.15.32:6829","nonce":1551241759}]},"public_addr":"172.21.15.32:6825/1551241759","cluster_addr":"172.21.15.32:6827/1551241759","heartbeat_back_addr":"172.21.15.32:6831/1551241759","heartbeat_front_addr":"172.21.15.32:6829/1551241759","state":["exists","up"]},{"osd":2,"uuid":"57f2e6de-a201-45c2-8981-8e56a4875ebd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6808","nonce":3806708944},{"type":"v1","addr":"172.21.15.32:6809","nonce":3806708944}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6810","nonce":3806708944},{"type":"v1","addr":"172.21.15.32:6811","nonce":3806708944}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6814","nonce":3806708944},{"type":"v1","addr":"172.21.15.32:6815","nonce":3806708944}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6812","nonce":3806708944},{"type":"v1","addr":"172.21.15.32:6813","nonce":3806708944}]},"public_addr":"172.21.15.32:6809/3806708944","cluster_addr":"172.21.15.32:6811/3806708944","heartbeat_back_addr":"172.21.15.32:6815/3806708944","heartbeat_front_addr":"172.21.15.32:6813/3806708944","state":["exists","up"]},{"osd":3,"uuid":"6439a280-da45-4148-8130-f16323935ff3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6800","nonce":9088303},{"type":"v1","addr":"172.21.15.32:6801","nonce":9088303}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6802","nonce":9088303},{"type":"v1","addr":"172.21.15.32:6803","nonce":9088303}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6806","nonce":9088303},{"type":"v1","addr":"172.21.15.32:6807","nonce":9088303}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6804","nonce":9088303},{"type":"v1","addr":"172.21.15.32:6805","nonce":9088303}]},"public_addr":"172.21.15.32:6801/9088303","cluster_addr":"172.21.15.32:6803/9088303","heartbeat_back_addr":"172.21.15.32:6807/9088303","heartbeat_front_addr":"172.21.15.32:6805/9088303","state":["exists","up"]},{"osd":4,"uuid":"4f902eff-1e18-418d-82c9-2151c69524fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6802","nonce":864752275},{"type":"v1","addr":"172.21.15.118:6803","nonce":864752275}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6804","nonce":864752275},{"type":"v1","addr":"172.21.15.118:6805","nonce":864752275}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6808","nonce":864752275},{"type":"v1","addr":"172.21.15.118:6809","nonce":864752275}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6806","nonce":864752275},{"type":"v1","addr":"172.21.15.118:6807","nonce":864752275}]},"public_addr":"172.21.15.118:6803/864752275","cluster_addr":"172.21.15.118:6805/864752275","heartbeat_back_addr":"172.21.15.118:6809/864752275","heartbeat_front_addr":"172.21.15.118:6807/864752275","state":["exists","up"]},{"osd":5,"uuid":"bfd9feac-dd06-458c-91b9-af7f175f612c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6810","nonce":2724891770},{"type":"v1","addr":"172.21.15.118:6811","nonce":2724891770}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6812","nonce":2724891770},{"type":"v1","addr":"172.21.15.118:6813","nonce":2724891770}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6816","nonce":2724891770},{"type":"v1","addr":"172.21.15.118:6817","nonce":2724891770}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6814","nonce":2724891770},{"type":"v1","addr":"172.21.15.118:6815","nonce":2724891770}]},"public_addr":"172.21.15.118:6811/2724891770","cluster_addr":"172.21.15.118:6813/2724891770","heartbeat_back_addr":"172.21.15.118:6817/2724891770","heartbeat_front_addr":"172.21.15.118:6815/2724891770","state":["exists","up"]},{"osd":6,"uuid":"5ed5a279-9d8e-416d-bbc7-7edd3e6d5705","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6825","nonce":801947386},{"type":"v1","addr":"172.21.15.118:6827","nonce":801947386}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6828","nonce":801947386},{"type":"v1","addr":"172.21.15.118:6829","nonce":801947386}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6832","nonce":801947386},{"type":"v1","addr":"172.21.15.118:6833","nonce":801947386}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6830","nonce":801947386},{"type":"v1","addr":"172.21.15.118:6831","nonce":801947386}]},"public_addr":"172.21.15.118:6827/801947386","cluster_addr":"172.21.15.118:6829/801947386","heartbeat_back_addr":"172.21.15.118:6833/801947386","heartbeat_front_addr":"172.21.15.118:6831/801947386","state":["exists","up"]},{"osd":7,"uuid":"f2086c47-875e-4451-a436-2008d1ea16af","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6818","nonce":653000048},{"type":"v1","addr":"172.21.15.118:6819","nonce":653000048}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6820","nonce":653000048},{"type":"v1","addr":"172.21.15.118:6821","nonce":653000048}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6824","nonce":653000048},{"type":"v1","addr":"172.21.15.118:6826","nonce":653000048}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6822","nonce":653000048},{"type":"v1","addr":"172.21.15.118:6823","nonce":653000048}]},"public_addr":"172.21.15.118:6819/653000048","cluster_addr":"172.21.15.118:6821/653000048","heartbeat_back_addr":"172.21.15.118:6826/653000048","heartbeat_front_addr":"172.21.15.118:6823/653000048","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-03T20:47:25.491994+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-03T20:47:25.587358+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-03T20:47:25.454175+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-03T20:47:25.377734+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-03T20:47:25.412351+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-03T20:47:25.499815+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-03T20:47:25.575044+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-03T20:47:25.552383+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-08-03T20:47:36.913 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:36.912+0000 7fc33aa00640 1 -- 172.21.15.32:0/4266390621 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fc310077010 msgr2=0x7fc310079520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:47:36.914 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:36.912+0000 7fc33aa00640 1 --2- 172.21.15.32:0/4266390621 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fc310077010 0x7fc310079520 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fc324007fd0 tx=0x7fc324001370 comp rx=0 tx=0).stop 2024-08-03T20:47:36.914 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:36.912+0000 7fc33aa00640 1 -- 172.21.15.32:0/4266390621 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc3340d0d30 msgr2=0x7fc33415efc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:47:36.914 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:36.912+0000 7fc33aa00640 1 --2- 172.21.15.32:0/4266390621 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc3340d0d30 0x7fc33415efc0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fc32c00ec20 tx=0x7fc32c00c6a0 comp rx=0 tx=0).stop 2024-08-03T20:47:36.914 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:36.912+0000 7fc33aa00640 1 -- 172.21.15.32:0/4266390621 shutdown_connections 2024-08-03T20:47:36.914 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:36.912+0000 7fc33aa00640 1 --2- 172.21.15.32:0/4266390621 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fc310077010 0x7fc310079520 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:36.914 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:36.912+0000 7fc33aa00640 1 --2- 172.21.15.32:0/4266390621 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc3340d0d30 0x7fc33415efc0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:36.914 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:36.912+0000 7fc33aa00640 1 --2- 172.21.15.32:0/4266390621 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fc3340b46c0 0x7fc33415ea80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:36.914 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:36.912+0000 7fc33aa00640 1 --2- 172.21.15.32:0/4266390621 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc3340b3d70 0x7fc3341f6470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:36.914 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:36.912+0000 7fc33aa00640 1 -- 172.21.15.32:0/4266390621 >> 172.21.15.32:0/4266390621 conn(0x7fc33409d640 msgr2=0x7fc33406e6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:47:36.914 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:36.912+0000 7fc33aa00640 1 -- 172.21.15.32:0/4266390621 shutdown_connections 2024-08-03T20:47:36.914 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:36.912+0000 7fc33aa00640 1 -- 172.21.15.32:0/4266390621 wait complete. 2024-08-03T20:47:36.937 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-08-03T20:47:37.171 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:37.169+0000 7f9b91e00640 1 -- 172.21.15.32:0/4158804174 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9b8c0cf070 msgr2=0x7f9b8c0cf480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:47:37.171 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:37.169+0000 7f9b91e00640 1 --2- 172.21.15.32:0/4158804174 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9b8c0cf070 0x7f9b8c0cf480 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f9b84005fd0 tx=0x7f9b8402aa90 comp rx=0 tx=0).stop 2024-08-03T20:47:37.171 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:37.169+0000 7f9b91e00640 1 -- 172.21.15.32:0/4158804174 shutdown_connections 2024-08-03T20:47:37.171 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:37.169+0000 7f9b91e00640 1 --2- 172.21.15.32:0/4158804174 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f9b8c0d47a0 0x7f9b8c0ceb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:37.172 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:37.169+0000 7f9b91e00640 1 --2- 172.21.15.32:0/4158804174 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9b8c0cf070 0x7f9b8c0cf480 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:37.172 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:37.169+0000 7f9b91e00640 1 -- 172.21.15.32:0/4158804174 >> 172.21.15.32:0/4158804174 conn(0x7f9b8c08d7e0 msgr2=0x7f9b8c09d410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:47:37.172 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:37.169+0000 7f9b91e00640 1 -- 172.21.15.32:0/4158804174 shutdown_connections 2024-08-03T20:47:37.172 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:37.169+0000 7f9b91e00640 1 -- 172.21.15.32:0/4158804174 wait complete. 2024-08-03T20:47:37.173 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:37.171+0000 7f9b91e00640 1 Processor -- start 2024-08-03T20:47:37.173 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:37.172+0000 7f9b91e00640 1 -- start start 2024-08-03T20:47:37.175 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:37.173+0000 7f9b91e00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f9b8c0cf070 0x7f9b8c1f8bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:47:37.175 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:37.173+0000 7f9b91e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9b8c0d26e0 0x7f9b8c0ca3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:47:37.175 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:37.173+0000 7f9b91e00640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f9b8c1f6230 0x7f9b8c1f5620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:47:37.175 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:37.173+0000 7f9b91e00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f9b8c0cfb40 con 0x7f9b8c0d26e0 2024-08-03T20:47:37.176 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:37.173+0000 7f9b91e00640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f9b8c0cf9c0 con 0x7f9b8c0cf070 2024-08-03T20:47:37.176 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:37.173+0000 7f9b91e00640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7f9b8c0cfcc0 con 0x7f9b8c1f6230 2024-08-03T20:47:37.176 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:37.173+0000 7f9b8be00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9b8c0d26e0 0x7f9b8c0ca3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:47:37.176 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:37.173+0000 7f9b8be00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9b8c0d26e0 0x7f9b8c0ca3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:40646/0 (socket says 172.21.15.32:40646) 2024-08-03T20:47:37.176 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:37.173+0000 7f9b8be00640 1 -- 172.21.15.32:0/1662270801 learned_addr learned my addr 172.21.15.32:0/1662270801 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:47:37.176 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:37.174+0000 7f9b91400640 1 --2- 172.21.15.32:0/1662270801 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f9b8c1f6230 0x7f9b8c1f5620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:47:37.176 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:37.174+0000 7f9b90a00640 1 --2- 172.21.15.32:0/1662270801 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f9b8c0cf070 0x7f9b8c1f8bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:47:37.176 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:37.174+0000 7f9b8be00640 1 -- 172.21.15.32:0/1662270801 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f9b8c1f6230 msgr2=0x7f9b8c1f5620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:47:37.176 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:37.174+0000 7f9b8be00640 1 --2- 172.21.15.32:0/1662270801 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f9b8c1f6230 0x7f9b8c1f5620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:37.176 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:37.174+0000 7f9b8be00640 1 -- 172.21.15.32:0/1662270801 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f9b8c0cf070 msgr2=0x7f9b8c1f8bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:47:37.176 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:37.174+0000 7f9b8be00640 1 --2- 172.21.15.32:0/1662270801 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f9b8c0cf070 0x7f9b8c1f8bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:37.176 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:37.174+0000 7f9b8be00640 1 -- 172.21.15.32:0/1662270801 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9b8c0cc480 con 0x7f9b8c0d26e0 2024-08-03T20:47:37.176 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:37.174+0000 7f9b91400640 1 --2- 172.21.15.32:0/1662270801 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f9b8c1f6230 0x7f9b8c1f5620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-03T20:47:37.177 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:37.174+0000 7f9b8be00640 1 --2- 172.21.15.32:0/1662270801 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9b8c0d26e0 0x7f9b8c0ca3b0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f9b7800e790 tx=0x7f9b7800ec50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:47:37.177 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:37.174+0000 7f9b88c00640 1 -- 172.21.15.32:0/1662270801 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b7800cac0 con 0x7f9b8c0d26e0 2024-08-03T20:47:37.177 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:37.174+0000 7f9b88c00640 1 -- 172.21.15.32:0/1662270801 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f9b7800cc60 con 0x7f9b8c0d26e0 2024-08-03T20:47:37.178 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:37.174+0000 7f9b88c00640 1 -- 172.21.15.32:0/1662270801 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b78004280 con 0x7f9b8c0d26e0 2024-08-03T20:47:37.178 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:37.175+0000 7f9b91e00640 1 -- 172.21.15.32:0/1662270801 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9b8c1f9420 con 0x7f9b8c0d26e0 2024-08-03T20:47:37.178 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:37.175+0000 7f9b91e00640 1 -- 172.21.15.32:0/1662270801 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9b8c1f99e0 con 0x7f9b8c0d26e0 2024-08-03T20:47:37.178 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:37.176+0000 7f9b91e00640 1 -- 172.21.15.32:0/1662270801 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9b8c0d5250 con 0x7f9b8c0d26e0 2024-08-03T20:47:37.178 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:37.177+0000 7f9b88c00640 1 -- 172.21.15.32:0/1662270801 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f9b780035f0 con 0x7f9b8c0d26e0 2024-08-03T20:47:37.179 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:37.178+0000 7f9b88c00640 1 --2- 172.21.15.32:0/1662270801 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f9b6c077040 0x7f9b6c079550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:47:37.179 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:37.178+0000 7f9b88c00640 1 -- 172.21.15.32:0/1662270801 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f9b780991d0 con 0x7f9b8c0d26e0 2024-08-03T20:47:37.180 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:37.179+0000 7f9b90a00640 1 --2- 172.21.15.32:0/1662270801 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f9b6c077040 0x7f9b6c079550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:47:37.180 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:37.179+0000 7f9b90a00640 1 --2- 172.21.15.32:0/1662270801 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f9b6c077040 0x7f9b6c079550 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f9b80007fd0 tx=0x7f9b80009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:47:37.188 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:37.187+0000 7f9b88c00640 1 -- 172.21.15.32:0/1662270801 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7f9b8c0d5250 con 0x7f9b8c0d26e0 2024-08-03T20:47:37.433 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:37.432+0000 7f9b91e00640 1 -- 172.21.15.32:0/1662270801 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) -- 0x7f9b8c1f91b0 con 0x7f9b8c0d26e0 2024-08-03T20:47:37.939 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:37.937+0000 7f9b88c00640 1 -- 172.21.15.32:0/1662270801 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v7) ==== 118+0+0 (secure 0 0 0) 0x7f9b78062330 con 0x7f9b8c0d26e0 2024-08-03T20:47:37.942 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:37.940+0000 7f9b91e00640 1 -- 172.21.15.32:0/1662270801 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f9b6c077040 msgr2=0x7f9b6c079550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:47:37.942 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:37.940+0000 7f9b91e00640 1 --2- 172.21.15.32:0/1662270801 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f9b6c077040 0x7f9b6c079550 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f9b80007fd0 tx=0x7f9b80009040 comp rx=0 tx=0).stop 2024-08-03T20:47:37.942 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:37.940+0000 7f9b91e00640 1 -- 172.21.15.32:0/1662270801 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9b8c0d26e0 msgr2=0x7f9b8c0ca3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:47:37.942 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:37.940+0000 7f9b91e00640 1 --2- 172.21.15.32:0/1662270801 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9b8c0d26e0 0x7f9b8c0ca3b0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f9b7800e790 tx=0x7f9b7800ec50 comp rx=0 tx=0).stop 2024-08-03T20:47:37.942 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:37.940+0000 7f9b91e00640 1 -- 172.21.15.32:0/1662270801 shutdown_connections 2024-08-03T20:47:37.942 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:37.940+0000 7f9b91e00640 1 --2- 172.21.15.32:0/1662270801 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f9b6c077040 0x7f9b6c079550 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:37.942 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:37.940+0000 7f9b91e00640 1 --2- 172.21.15.32:0/1662270801 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f9b8c1f6230 0x7f9b8c1f5620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:37.942 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:37.940+0000 7f9b91e00640 1 --2- 172.21.15.32:0/1662270801 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9b8c0d26e0 0x7f9b8c0ca3b0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:37.942 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:37.940+0000 7f9b91e00640 1 --2- 172.21.15.32:0/1662270801 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f9b8c0cf070 0x7f9b8c1f8bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:37.942 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:37.940+0000 7f9b91e00640 1 -- 172.21.15.32:0/1662270801 >> 172.21.15.32:0/1662270801 conn(0x7f9b8c08d7e0 msgr2=0x7f9b8c06da00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:47:37.942 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:37.940+0000 7f9b91e00640 1 -- 172.21.15.32:0/1662270801 shutdown_connections 2024-08-03T20:47:37.943 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:37.940+0000 7f9b91e00640 1 -- 172.21.15.32:0/1662270801 wait complete. 2024-08-03T20:47:37.965 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-08-03T20:47:37.966 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-08-03T20:47:38.198 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:38.196+0000 7f16cf200640 1 -- 172.21.15.32:0/2216433518 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f16d0159ff0 msgr2=0x7f16d015fe10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:47:38.198 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:38.196+0000 7f16cf200640 1 --2- 172.21.15.32:0/2216433518 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f16d0159ff0 0x7f16d015fe10 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f16b8009830 tx=0x7f16b802a510 comp rx=0 tx=0).stop 2024-08-03T20:47:38.198 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:38.196+0000 7f16cf200640 1 -- 172.21.15.32:0/2216433518 shutdown_connections 2024-08-03T20:47:38.198 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:38.196+0000 7f16cf200640 1 --2- 172.21.15.32:0/2216433518 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f16d0159ff0 0x7f16d015fe10 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:38.198 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:38.196+0000 7f16cf200640 1 --2- 172.21.15.32:0/2216433518 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f16d01609d0 0x7f16d00cc050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:38.199 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:38.196+0000 7f16cf200640 1 -- 172.21.15.32:0/2216433518 >> 172.21.15.32:0/2216433518 conn(0x7f16d00cfd80 msgr2=0x7f16d00d0190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:47:38.199 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:38.196+0000 7f16cf200640 1 -- 172.21.15.32:0/2216433518 shutdown_connections 2024-08-03T20:47:38.199 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:38.196+0000 7f16cf200640 1 -- 172.21.15.32:0/2216433518 wait complete. 2024-08-03T20:47:38.199 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:38.198+0000 7f16cf200640 1 Processor -- start 2024-08-03T20:47:38.200 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:38.198+0000 7f16cf200640 1 -- start start 2024-08-03T20:47:38.202 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:38.200+0000 7f16cf200640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f16d0159ff0 0x7f16d01e5c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:47:38.202 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:38.200+0000 7f16cf200640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f16d0160350 0x7f16d014e380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:47:38.202 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:38.200+0000 7f16cf200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f16d01609d0 0x7f16d00ce9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:47:38.202 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:38.200+0000 7f16cf200640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f16d015d0b0 con 0x7f16d01609d0 2024-08-03T20:47:38.202 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:38.200+0000 7f16cf200640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f16d015cf30 con 0x7f16d0159ff0 2024-08-03T20:47:38.202 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:38.200+0000 7f16cf200640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7f16d015d230 con 0x7f16d0160350 2024-08-03T20:47:38.202 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:38.200+0000 7f16ce800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f16d01609d0 0x7f16d00ce9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:47:38.203 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:38.200+0000 7f16ce800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f16d01609d0 0x7f16d00ce9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:40666/0 (socket says 172.21.15.32:40666) 2024-08-03T20:47:38.203 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:38.200+0000 7f16ce800640 1 -- 172.21.15.32:0/2631623129 learned_addr learned my addr 172.21.15.32:0/2631623129 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:47:38.203 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:38.200+0000 7f16c7e00640 1 --2- 172.21.15.32:0/2631623129 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f16d0160350 0x7f16d014e380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:47:38.203 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:38.200+0000 7f16cde00640 1 --2- 172.21.15.32:0/2631623129 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f16d0159ff0 0x7f16d01e5c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:47:38.203 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:38.201+0000 7f16ce800640 1 -- 172.21.15.32:0/2631623129 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f16d0160350 msgr2=0x7f16d014e380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:47:38.203 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:38.201+0000 7f16ce800640 1 --2- 172.21.15.32:0/2631623129 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f16d0160350 0x7f16d014e380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:38.203 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:38.201+0000 7f16ce800640 1 -- 172.21.15.32:0/2631623129 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f16d0159ff0 msgr2=0x7f16d01e5c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:47:38.203 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:38.201+0000 7f16ce800640 1 --2- 172.21.15.32:0/2631623129 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f16d0159ff0 0x7f16d01e5c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:38.203 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:38.201+0000 7f16ce800640 1 -- 172.21.15.32:0/2631623129 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f16d01e6530 con 0x7f16d01609d0 2024-08-03T20:47:38.203 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:38.201+0000 7f16c7e00640 1 --2- 172.21.15.32:0/2631623129 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f16d0160350 0x7f16d014e380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-03T20:47:38.204 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:38.201+0000 7f16ce800640 1 --2- 172.21.15.32:0/2631623129 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f16d01609d0 0x7f16d00ce9a0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f16c800c760 tx=0x7f16c800cc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:47:38.204 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:38.201+0000 7f16c6000640 1 -- 172.21.15.32:0/2631623129 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16c8003da0 con 0x7f16d01609d0 2024-08-03T20:47:38.204 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:38.201+0000 7f16c6000640 1 -- 172.21.15.32:0/2631623129 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f16c800ce10 con 0x7f16d01609d0 2024-08-03T20:47:38.204 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:38.201+0000 7f16c6000640 1 -- 172.21.15.32:0/2631623129 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16c80052d0 con 0x7f16d01609d0 2024-08-03T20:47:38.205 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:38.201+0000 7f16cf200640 1 -- 172.21.15.32:0/2631623129 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f16d01e6810 con 0x7f16d01609d0 2024-08-03T20:47:38.205 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:38.201+0000 7f16cf200640 1 -- 172.21.15.32:0/2631623129 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f16d01e7010 con 0x7f16d01609d0 2024-08-03T20:47:38.205 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:38.202+0000 7f16cf200640 1 -- 172.21.15.32:0/2631623129 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f16d015fe10 con 0x7f16d01609d0 2024-08-03T20:47:38.205 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:38.204+0000 7f16c6000640 1 -- 172.21.15.32:0/2631623129 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f16c8005470 con 0x7f16d01609d0 2024-08-03T20:47:38.206 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:38.205+0000 7f16c6000640 1 --2- 172.21.15.32:0/2631623129 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f16ac077010 0x7f16ac079520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:47:38.206 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:38.205+0000 7f16c6000640 1 -- 172.21.15.32:0/2631623129 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f16c8098740 con 0x7f16d01609d0 2024-08-03T20:47:38.206 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:38.205+0000 7f16cde00640 1 --2- 172.21.15.32:0/2631623129 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f16ac077010 0x7f16ac079520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:47:38.207 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:38.206+0000 7f16cde00640 1 --2- 172.21.15.32:0/2631623129 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f16ac077010 0x7f16ac079520 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f16c0009b30 tx=0x7f16c0005f90 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:47:38.211 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:38.209+0000 7f16c6000640 1 -- 172.21.15.32:0/2631623129 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7f16d0160760 con 0x7f16d01609d0 2024-08-03T20:47:38.460 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:38.459+0000 7f16cf200640 1 -- 172.21.15.32:0/2631623129 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) -- 0x7f16d00d09e0 con 0x7f16d01609d0 2024-08-03T20:47:38.945 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:38.943+0000 7f16c6000640 1 -- 172.21.15.32:0/2631623129 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v8) ==== 122+0+0 (secure 0 0 0) 0x7f16c80618a0 con 0x7f16d01609d0 2024-08-03T20:47:38.948 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:38.946+0000 7f16cf200640 1 -- 172.21.15.32:0/2631623129 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f16ac077010 msgr2=0x7f16ac079520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:47:38.948 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:38.946+0000 7f16cf200640 1 --2- 172.21.15.32:0/2631623129 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f16ac077010 0x7f16ac079520 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f16c0009b30 tx=0x7f16c0005f90 comp rx=0 tx=0).stop 2024-08-03T20:47:38.948 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:38.946+0000 7f16cf200640 1 -- 172.21.15.32:0/2631623129 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f16d01609d0 msgr2=0x7f16d00ce9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:47:38.948 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:38.946+0000 7f16cf200640 1 --2- 172.21.15.32:0/2631623129 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f16d01609d0 0x7f16d00ce9a0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f16c800c760 tx=0x7f16c800cc20 comp rx=0 tx=0).stop 2024-08-03T20:47:38.949 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:38.946+0000 7f16cf200640 1 -- 172.21.15.32:0/2631623129 shutdown_connections 2024-08-03T20:47:38.949 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:38.946+0000 7f16cf200640 1 --2- 172.21.15.32:0/2631623129 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f16ac077010 0x7f16ac079520 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:38.949 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:38.946+0000 7f16cf200640 1 --2- 172.21.15.32:0/2631623129 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f16d01609d0 0x7f16d00ce9a0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:38.949 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:38.946+0000 7f16cf200640 1 --2- 172.21.15.32:0/2631623129 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f16d0160350 0x7f16d014e380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:38.949 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:38.946+0000 7f16cf200640 1 --2- 172.21.15.32:0/2631623129 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f16d0159ff0 0x7f16d01e5c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:38.949 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:38.946+0000 7f16cf200640 1 -- 172.21.15.32:0/2631623129 >> 172.21.15.32:0/2631623129 conn(0x7f16d00cfd80 msgr2=0x7f16d015e9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:47:38.949 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:38.947+0000 7f16cf200640 1 -- 172.21.15.32:0/2631623129 shutdown_connections 2024-08-03T20:47:38.949 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:38.947+0000 7f16cf200640 1 -- 172.21.15.32:0/2631623129 wait complete. 2024-08-03T20:47:38.973 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-08-03T20:47:38.973 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-08-03T20:47:39.206 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:39.204+0000 7f2e71600640 1 -- 172.21.15.32:0/1032134217 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f2e6c165160 msgr2=0x7f2e6c167580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:47:39.206 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:39.204+0000 7f2e71600640 1 --2- 172.21.15.32:0/1032134217 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f2e6c165160 0x7f2e6c167580 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f2e64009be0 tx=0x7f2e6402a890 comp rx=0 tx=0).stop 2024-08-03T20:47:39.206 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:39.204+0000 7f2e71600640 1 -- 172.21.15.32:0/1032134217 shutdown_connections 2024-08-03T20:47:39.206 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:39.204+0000 7f2e71600640 1 --2- 172.21.15.32:0/1032134217 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f2e6c165160 0x7f2e6c167580 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:39.206 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:39.204+0000 7f2e71600640 1 -- 172.21.15.32:0/1032134217 >> 172.21.15.32:0/1032134217 conn(0x7f2e6c0b1bd0 msgr2=0x7f2e6c0b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:47:39.206 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:39.204+0000 7f2e71600640 1 -- 172.21.15.32:0/1032134217 shutdown_connections 2024-08-03T20:47:39.207 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:39.204+0000 7f2e71600640 1 -- 172.21.15.32:0/1032134217 wait complete. 2024-08-03T20:47:39.208 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:39.206+0000 7f2e71600640 1 Processor -- start 2024-08-03T20:47:39.208 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:39.207+0000 7f2e71600640 1 -- start start 2024-08-03T20:47:39.211 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:39.208+0000 7f2e71600640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2e6c06e930 0x7f2e6c140200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:47:39.211 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:39.208+0000 7f2e71600640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f2e6c144f90 0x7f2e6c0b48f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:47:39.211 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:39.208+0000 7f2e71600640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f2e6c165160 0x7f2e6c13b4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:47:39.211 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:39.208+0000 7f2e71600640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f2e6c167ac0 con 0x7f2e6c06e930 2024-08-03T20:47:39.211 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:39.208+0000 7f2e71600640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f2e6c0cfa50 con 0x7f2e6c144f90 2024-08-03T20:47:39.211 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:39.208+0000 7f2e71600640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7f2e6c167c40 con 0x7f2e6c165160 2024-08-03T20:47:39.212 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:39.208+0000 7f2e6aa00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2e6c06e930 0x7f2e6c140200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:47:39.212 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:39.208+0000 7f2e6aa00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2e6c06e930 0x7f2e6c140200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:40680/0 (socket says 172.21.15.32:40680) 2024-08-03T20:47:39.212 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:39.208+0000 7f2e6aa00640 1 -- 172.21.15.32:0/158039115 learned_addr learned my addr 172.21.15.32:0/158039115 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:47:39.212 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:39.208+0000 7f2e6b400640 1 --2- 172.21.15.32:0/158039115 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f2e6c165160 0x7f2e6c13b4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:47:39.212 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:39.208+0000 7f2e62000640 1 --2- 172.21.15.32:0/158039115 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f2e6c144f90 0x7f2e6c0b48f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:47:39.212 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:39.209+0000 7f2e6aa00640 1 -- 172.21.15.32:0/158039115 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f2e6c165160 msgr2=0x7f2e6c13b4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:47:39.212 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:39.209+0000 7f2e6aa00640 1 --2- 172.21.15.32:0/158039115 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f2e6c165160 0x7f2e6c13b4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:39.212 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:39.209+0000 7f2e6aa00640 1 -- 172.21.15.32:0/158039115 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f2e6c144f90 msgr2=0x7f2e6c0b48f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:47:39.212 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:39.209+0000 7f2e6aa00640 1 --2- 172.21.15.32:0/158039115 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f2e6c144f90 0x7f2e6c0b48f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:39.212 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:39.209+0000 7f2e6aa00640 1 -- 172.21.15.32:0/158039115 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2e6c1f98d0 con 0x7f2e6c06e930 2024-08-03T20:47:39.212 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:39.209+0000 7f2e62000640 1 --2- 172.21.15.32:0/158039115 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f2e6c144f90 0x7f2e6c0b48f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-03T20:47:39.212 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:39.209+0000 7f2e6aa00640 1 --2- 172.21.15.32:0/158039115 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2e6c06e930 0x7f2e6c140200 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f2e5c00c390 tx=0x7f2e5c00c850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:47:39.212 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:39.209+0000 7f2e63400640 1 -- 172.21.15.32:0/158039115 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e5c011790 con 0x7f2e6c06e930 2024-08-03T20:47:39.212 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:39.209+0000 7f2e63400640 1 -- 172.21.15.32:0/158039115 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f2e5c017070 con 0x7f2e6c06e930 2024-08-03T20:47:39.213 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:39.209+0000 7f2e63400640 1 -- 172.21.15.32:0/158039115 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e5c011ce0 con 0x7f2e6c06e930 2024-08-03T20:47:39.213 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:39.209+0000 7f2e71600640 1 -- 172.21.15.32:0/158039115 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2e6c1f8d50 con 0x7f2e6c06e930 2024-08-03T20:47:39.213 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:39.210+0000 7f2e71600640 1 -- 172.21.15.32:0/158039115 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2e6c1f9310 con 0x7f2e6c06e930 2024-08-03T20:47:39.213 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:39.211+0000 7f2e71600640 1 -- 172.21.15.32:0/158039115 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2e6c073e60 con 0x7f2e6c06e930 2024-08-03T20:47:39.213 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:39.212+0000 7f2e63400640 1 -- 172.21.15.32:0/158039115 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f2e5c01c460 con 0x7f2e6c06e930 2024-08-03T20:47:39.214 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:39.213+0000 7f2e63400640 1 --2- 172.21.15.32:0/158039115 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f2e48077010 0x7f2e48079520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:47:39.214 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:39.213+0000 7f2e63400640 1 -- 172.21.15.32:0/158039115 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f2e5c0986f0 con 0x7f2e6c06e930 2024-08-03T20:47:39.215 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:39.213+0000 7f2e62000640 1 --2- 172.21.15.32:0/158039115 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f2e48077010 0x7f2e48079520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:47:39.215 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:39.214+0000 7f2e62000640 1 --2- 172.21.15.32:0/158039115 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f2e48077010 0x7f2e48079520 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f2e54007fd0 tx=0x7f2e54009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:47:39.218 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:39.217+0000 7f2e63400640 1 -- 172.21.15.32:0/158039115 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7f2e5c0618e0 con 0x7f2e6c06e930 2024-08-03T20:47:39.462 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:39.461+0000 7f2e71600640 1 -- 172.21.15.32:0/158039115 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f2e6c145990 con 0x7f2e6c06e930 2024-08-03T20:47:39.463 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:39.461+0000 7f2e63400640 1 -- 172.21.15.32:0/158039115 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) ==== 93+0+5220 (secure 0 0 0) 0x7f2e5c0611e0 con 0x7f2e6c06e930 2024-08-03T20:47:39.463 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-03T20:47:39.463 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":8,"btime":"2024-08-03T20:47:38:939479+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4289,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6835/1753256117","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6834","nonce":1753256117},{"type":"v1","addr":"172.21.15.118:6835","nonce":1753256117}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4292,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6837/3055588578","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6836","nonce":3055588578},{"type":"v1","addr":"172.21.15.118:6837","nonce":3055588578}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4338,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6834/3026095815","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":3026095815},{"type":"v1","addr":"172.21.15.32:6834","nonce":3026095815}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-08-03T20:47:35.442482+0000","modified":"2024-08-03T20:47:38.460928+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4341},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4341":{"gid":4341,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.32:6835/246278108","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6833","nonce":246278108},{"type":"v1","addr":"172.21.15.32:6835","nonce":246278108}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4341,"qdb_cluster":[4341]},"id":1}]} 2024-08-03T20:47:39.463 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 8 2024-08-03T20:47:39.465 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:39.464+0000 7f2e71600640 1 -- 172.21.15.32:0/158039115 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f2e48077010 msgr2=0x7f2e48079520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:47:39.465 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:39.464+0000 7f2e71600640 1 --2- 172.21.15.32:0/158039115 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f2e48077010 0x7f2e48079520 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f2e54007fd0 tx=0x7f2e54009040 comp rx=0 tx=0).stop 2024-08-03T20:47:39.466 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:39.464+0000 7f2e71600640 1 -- 172.21.15.32:0/158039115 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2e6c06e930 msgr2=0x7f2e6c140200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:47:39.466 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:39.464+0000 7f2e71600640 1 --2- 172.21.15.32:0/158039115 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2e6c06e930 0x7f2e6c140200 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f2e5c00c390 tx=0x7f2e5c00c850 comp rx=0 tx=0).stop 2024-08-03T20:47:39.466 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:39.464+0000 7f2e71600640 1 -- 172.21.15.32:0/158039115 shutdown_connections 2024-08-03T20:47:39.466 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:39.464+0000 7f2e71600640 1 --2- 172.21.15.32:0/158039115 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f2e48077010 0x7f2e48079520 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:39.466 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:39.464+0000 7f2e71600640 1 --2- 172.21.15.32:0/158039115 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f2e6c144f90 0x7f2e6c0b48f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:39.466 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:39.464+0000 7f2e71600640 1 --2- 172.21.15.32:0/158039115 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f2e6c165160 0x7f2e6c13b4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:39.466 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:39.464+0000 7f2e71600640 1 --2- 172.21.15.32:0/158039115 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2e6c06e930 0x7f2e6c140200 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:39.466 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:39.464+0000 7f2e71600640 1 -- 172.21.15.32:0/158039115 >> 172.21.15.32:0/158039115 conn(0x7f2e6c0b1bd0 msgr2=0x7f2e6c0d0920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:47:39.466 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:39.464+0000 7f2e71600640 1 -- 172.21.15.32:0/158039115 shutdown_connections 2024-08-03T20:47:39.467 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:39.464+0000 7f2e71600640 1 -- 172.21.15.32:0/158039115 wait complete. 2024-08-03T20:47:39.490 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-08-03T20:47:39.719 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:39.717+0000 7f621e600640 1 -- 172.21.15.32:0/3606777373 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f62180cee10 msgr2=0x7f621815c0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:47:39.719 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:39.717+0000 7f621e600640 1 --2- 172.21.15.32:0/3606777373 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f62180cee10 0x7f621815c0b0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f62180701e0 tx=0x7f620c02a560 comp rx=0 tx=0).stop 2024-08-03T20:47:39.719 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:39.717+0000 7f621e600640 1 -- 172.21.15.32:0/3606777373 shutdown_connections 2024-08-03T20:47:39.719 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:39.717+0000 7f621e600640 1 --2- 172.21.15.32:0/3606777373 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f62180af450 0x7f621806f9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:39.719 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:39.717+0000 7f621e600640 1 --2- 172.21.15.32:0/3606777373 >> v2:172.21.15.118:3300/0 conn(0x7f62180af040 0x7f621806f480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:39.719 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:39.717+0000 7f621e600640 1 --2- 172.21.15.32:0/3606777373 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f62180cee10 0x7f621815c0b0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:39.719 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:39.717+0000 7f621e600640 1 -- 172.21.15.32:0/3606777373 >> 172.21.15.32:0/3606777373 conn(0x7f62180b2120 msgr2=0x7f62180cfd60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:47:39.719 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:39.717+0000 7f621e600640 1 -- 172.21.15.32:0/3606777373 shutdown_connections 2024-08-03T20:47:39.719 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:39.717+0000 7f621e600640 1 -- 172.21.15.32:0/3606777373 wait complete. 2024-08-03T20:47:39.720 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:39.719+0000 7f621e600640 1 Processor -- start 2024-08-03T20:47:39.721 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:39.719+0000 7f621e600640 1 -- start start 2024-08-03T20:47:39.723 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:39.721+0000 7f621e600640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f62180af040 0x7f621809d440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:47:39.723 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:39.721+0000 7f621e600640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f62180af450 0x7f62181515d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:47:39.724 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:39.721+0000 7f621e600640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f62180cee10 0x7f621815d950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:47:39.724 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:39.721+0000 7f621e600640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f62180ca740 con 0x7f62180af450 2024-08-03T20:47:39.724 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:39.721+0000 7f621e600640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f62180ca5c0 con 0x7f62180af040 2024-08-03T20:47:39.724 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:39.721+0000 7f621e600640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7f62180ca8c0 con 0x7f62180cee10 2024-08-03T20:47:39.724 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:39.721+0000 7f6217e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f62180af450 0x7f62181515d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:47:39.724 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:39.721+0000 7f6217e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f62180af450 0x7f62181515d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:40704/0 (socket says 172.21.15.32:40704) 2024-08-03T20:47:39.725 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:39.721+0000 7f6217e00640 1 -- 172.21.15.32:0/1342720270 learned_addr learned my addr 172.21.15.32:0/1342720270 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:47:39.725 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:39.721+0000 7f621d200640 1 --2- 172.21.15.32:0/1342720270 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f62180af040 0x7f621809d440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:47:39.725 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:39.721+0000 7f621dc00640 1 --2- 172.21.15.32:0/1342720270 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f62180cee10 0x7f621815d950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:47:39.725 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:39.721+0000 7f6217e00640 1 -- 172.21.15.32:0/1342720270 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f62180cee10 msgr2=0x7f621815d950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:47:39.725 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:39.721+0000 7f6217e00640 1 --2- 172.21.15.32:0/1342720270 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f62180cee10 0x7f621815d950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:39.725 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:39.721+0000 7f6217e00640 1 -- 172.21.15.32:0/1342720270 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f62180af040 msgr2=0x7f621809d440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:47:39.726 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:39.721+0000 7f6217e00640 1 --2- 172.21.15.32:0/1342720270 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f62180af040 0x7f621809d440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:39.726 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:39.721+0000 7f6217e00640 1 -- 172.21.15.32:0/1342720270 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f62181f62e0 con 0x7f62180af450 2024-08-03T20:47:39.726 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:39.722+0000 7f6217e00640 1 --2- 172.21.15.32:0/1342720270 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f62180af450 0x7f62181515d0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f620400e710 tx=0x7f620400ebd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:47:39.726 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:39.722+0000 7f6214c00640 1 -- 172.21.15.32:0/1342720270 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f620400cab0 con 0x7f62180af450 2024-08-03T20:47:39.726 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:39.722+0000 7f6214c00640 1 -- 172.21.15.32:0/1342720270 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f620400cd40 con 0x7f62180af450 2024-08-03T20:47:39.727 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:39.722+0000 7f621e600640 1 -- 172.21.15.32:0/1342720270 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f621809b150 con 0x7f62180af450 2024-08-03T20:47:39.727 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:39.722+0000 7f6214c00640 1 -- 172.21.15.32:0/1342720270 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6204010690 con 0x7f62180af450 2024-08-03T20:47:39.727 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:39.722+0000 7f621e600640 1 -- 172.21.15.32:0/1342720270 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f62181f8dc0 con 0x7f62180af450 2024-08-03T20:47:39.729 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:39.723+0000 7f621e600640 1 -- 172.21.15.32:0/1342720270 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f62180cae50 con 0x7f62180af450 2024-08-03T20:47:39.729 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:39.724+0000 7f6214c00640 1 -- 172.21.15.32:0/1342720270 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f6204010830 con 0x7f62180af450 2024-08-03T20:47:39.729 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:39.725+0000 7f6214c00640 1 --2- 172.21.15.32:0/1342720270 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f61f8077010 0x7f61f8079520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:47:39.729 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:39.725+0000 7f6214c00640 1 -- 172.21.15.32:0/1342720270 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f62040991c0 con 0x7f62180af450 2024-08-03T20:47:39.729 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:39.726+0000 7f621d200640 1 --2- 172.21.15.32:0/1342720270 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f61f8077010 0x7f61f8079520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:47:39.729 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:39.726+0000 7f621d200640 1 --2- 172.21.15.32:0/1342720270 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f61f8077010 0x7f61f8079520 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f620c033000 tx=0x7f620c004730 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:47:39.731 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:39.729+0000 7f6214c00640 1 -- 172.21.15.32:0/1342720270 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7f6204061d60 con 0x7f62180af450 2024-08-03T20:47:39.938 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:39.935+0000 7f621e600640 1 -- 172.21.15.32:0/1342720270 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f621809aee0 con 0x7f62180af450 2024-08-03T20:47:39.938 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:39.936+0000 7f6214c00640 1 -- 172.21.15.32:0/1342720270 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) ==== 74+0+16545 (secure 0 0 0) 0x7f62040662b0 con 0x7f62180af450 2024-08-03T20:47:39.938 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-03T20:47:39.938 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":21,"fsid":"40ab141a-01d7-49b9-ae3c-2edec9876e95","created":"2024-08-03T20:47:14.957362+0000","modified":"2024-08-03T20:47:36.927196+0000","last_up_change":"2024-08-03T20:47:28.874451+0000","last_in_change":"2024-08-03T20:47:20.079442+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-08-03T20:47:28.261300+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-08-03T20:47:33.085381+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-08-03T20:47:34.405408+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5a83e759-9035-438b-889a-0a6af1f0a0d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6816","nonce":3066156116},{"type":"v1","addr":"172.21.15.32:6817","nonce":3066156116}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6818","nonce":3066156116},{"type":"v1","addr":"172.21.15.32:6819","nonce":3066156116}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6822","nonce":3066156116},{"type":"v1","addr":"172.21.15.32:6823","nonce":3066156116}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6820","nonce":3066156116},{"type":"v1","addr":"172.21.15.32:6821","nonce":3066156116}]},"public_addr":"172.21.15.32:6817/3066156116","cluster_addr":"172.21.15.32:6819/3066156116","heartbeat_back_addr":"172.21.15.32:6823/3066156116","heartbeat_front_addr":"172.21.15.32:6821/3066156116","state":["exists","up"]},{"osd":1,"uuid":"d7c5c210-da4f-49f9-b9e7-1dfb02daa4bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6824","nonce":1551241759},{"type":"v1","addr":"172.21.15.32:6825","nonce":1551241759}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6826","nonce":1551241759},{"type":"v1","addr":"172.21.15.32:6827","nonce":1551241759}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6830","nonce":1551241759},{"type":"v1","addr":"172.21.15.32:6831","nonce":1551241759}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6828","nonce":1551241759},{"type":"v1","addr":"172.21.15.32:6829","nonce":1551241759}]},"public_addr":"172.21.15.32:6825/1551241759","cluster_addr":"172.21.15.32:6827/1551241759","heartbeat_back_addr":"172.21.15.32:6831/1551241759","heartbeat_front_addr":"172.21.15.32:6829/1551241759","state":["exists","up"]},{"osd":2,"uuid":"57f2e6de-a201-45c2-8981-8e56a4875ebd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6808","nonce":3806708944},{"type":"v1","addr":"172.21.15.32:6809","nonce":3806708944}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6810","nonce":3806708944},{"type":"v1","addr":"172.21.15.32:6811","nonce":3806708944}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6814","nonce":3806708944},{"type":"v1","addr":"172.21.15.32:6815","nonce":3806708944}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6812","nonce":3806708944},{"type":"v1","addr":"172.21.15.32:6813","nonce":3806708944}]},"public_addr":"172.21.15.32:6809/3806708944","cluster_addr":"172.21.15.32:6811/3806708944","heartbeat_back_addr":"172.21.15.32:6815/3806708944","heartbeat_front_addr":"172.21.15.32:6813/3806708944","state":["exists","up"]},{"osd":3,"uuid":"6439a280-da45-4148-8130-f16323935ff3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6800","nonce":9088303},{"type":"v1","addr":"172.21.15.32:6801","nonce":9088303}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6802","nonce":9088303},{"type":"v1","addr":"172.21.15.32:6803","nonce":9088303}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6806","nonce":9088303},{"type":"v1","addr":"172.21.15.32:6807","nonce":9088303}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6804","nonce":9088303},{"type":"v1","addr":"172.21.15.32:6805","nonce":9088303}]},"public_addr":"172.21.15.32:6801/9088303","cluster_addr":"172.21.15.32:6803/9088303","heartbeat_back_addr":"172.21.15.32:6807/9088303","heartbeat_front_addr":"172.21.15.32:6805/9088303","state":["exists","up"]},{"osd":4,"uuid":"4f902eff-1e18-418d-82c9-2151c69524fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6802","nonce":864752275},{"type":"v1","addr":"172.21.15.118:6803","nonce":864752275}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6804","nonce":864752275},{"type":"v1","addr":"172.21.15.118:6805","nonce":864752275}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6808","nonce":864752275},{"type":"v1","addr":"172.21.15.118:6809","nonce":864752275}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6806","nonce":864752275},{"type":"v1","addr":"172.21.15.118:6807","nonce":864752275}]},"public_addr":"172.21.15.118:6803/864752275","cluster_addr":"172.21.15.118:6805/864752275","heartbeat_back_addr":"172.21.15.118:6809/864752275","heartbeat_front_addr":"172.21.15.118:6807/864752275","state":["exists","up"]},{"osd":5,"uuid":"bfd9feac-dd06-458c-91b9-af7f175f612c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6810","nonce":2724891770},{"type":"v1","addr":"172.21.15.118:6811","nonce":2724891770}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6812","nonce":2724891770},{"type":"v1","addr":"172.21.15.118:6813","nonce":2724891770}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6816","nonce":2724891770},{"type":"v1","addr":"172.21.15.118:6817","nonce":2724891770}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6814","nonce":2724891770},{"type":"v1","addr":"172.21.15.118:6815","nonce":2724891770}]},"public_addr":"172.21.15.118:6811/2724891770","cluster_addr":"172.21.15.118:6813/2724891770","heartbeat_back_addr":"172.21.15.118:6817/2724891770","heartbeat_front_addr":"172.21.15.118:6815/2724891770","state":["exists","up"]},{"osd":6,"uuid":"5ed5a279-9d8e-416d-bbc7-7edd3e6d5705","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6825","nonce":801947386},{"type":"v1","addr":"172.21.15.118:6827","nonce":801947386}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6828","nonce":801947386},{"type":"v1","addr":"172.21.15.118:6829","nonce":801947386}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6832","nonce":801947386},{"type":"v1","addr":"172.21.15.118:6833","nonce":801947386}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6830","nonce":801947386},{"type":"v1","addr":"172.21.15.118:6831","nonce":801947386}]},"public_addr":"172.21.15.118:6827/801947386","cluster_addr":"172.21.15.118:6829/801947386","heartbeat_back_addr":"172.21.15.118:6833/801947386","heartbeat_front_addr":"172.21.15.118:6831/801947386","state":["exists","up"]},{"osd":7,"uuid":"f2086c47-875e-4451-a436-2008d1ea16af","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6818","nonce":653000048},{"type":"v1","addr":"172.21.15.118:6819","nonce":653000048}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6820","nonce":653000048},{"type":"v1","addr":"172.21.15.118:6821","nonce":653000048}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6824","nonce":653000048},{"type":"v1","addr":"172.21.15.118:6826","nonce":653000048}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6822","nonce":653000048},{"type":"v1","addr":"172.21.15.118:6823","nonce":653000048}]},"public_addr":"172.21.15.118:6819/653000048","cluster_addr":"172.21.15.118:6821/653000048","heartbeat_back_addr":"172.21.15.118:6826/653000048","heartbeat_front_addr":"172.21.15.118:6823/653000048","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-03T20:47:25.491994+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-03T20:47:25.587358+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-03T20:47:25.454175+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-03T20:47:25.377734+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-03T20:47:25.412351+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-03T20:47:25.499815+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-03T20:47:25.575044+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-03T20:47:25.552383+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-08-03T20:47:39.940 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:39.938+0000 7f621e600640 1 -- 172.21.15.32:0/1342720270 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f61f8077010 msgr2=0x7f61f8079520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:47:39.940 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:39.938+0000 7f621e600640 1 --2- 172.21.15.32:0/1342720270 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f61f8077010 0x7f61f8079520 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f620c033000 tx=0x7f620c004730 comp rx=0 tx=0).stop 2024-08-03T20:47:39.940 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:39.938+0000 7f621e600640 1 -- 172.21.15.32:0/1342720270 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f62180af450 msgr2=0x7f62181515d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:47:39.940 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:39.938+0000 7f621e600640 1 --2- 172.21.15.32:0/1342720270 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f62180af450 0x7f62181515d0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f620400e710 tx=0x7f620400ebd0 comp rx=0 tx=0).stop 2024-08-03T20:47:39.940 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:39.938+0000 7f621e600640 1 -- 172.21.15.32:0/1342720270 shutdown_connections 2024-08-03T20:47:39.940 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:39.938+0000 7f621e600640 1 --2- 172.21.15.32:0/1342720270 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f61f8077010 0x7f61f8079520 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:39.940 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:39.938+0000 7f621e600640 1 --2- 172.21.15.32:0/1342720270 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f62180cee10 0x7f621815d950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:39.941 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:39.938+0000 7f621e600640 1 --2- 172.21.15.32:0/1342720270 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f62180af450 0x7f62181515d0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:39.941 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:39.938+0000 7f621e600640 1 --2- 172.21.15.32:0/1342720270 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f62180af040 0x7f621809d440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:39.941 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:39.938+0000 7f621e600640 1 -- 172.21.15.32:0/1342720270 >> 172.21.15.32:0/1342720270 conn(0x7f62180b2120 msgr2=0x7f6218157820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:47:39.941 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:39.938+0000 7f621e600640 1 -- 172.21.15.32:0/1342720270 shutdown_connections 2024-08-03T20:47:39.941 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:39.938+0000 7f621e600640 1 -- 172.21.15.32:0/1342720270 wait complete. 2024-08-03T20:47:39.962 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-08-03T20:47:39.962 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-08-03T20:47:40.188 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:40.186+0000 7f4337200640 1 -- 172.21.15.32:0/417903017 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f433806d0f0 msgr2=0x7f43380711f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:47:40.188 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:40.186+0000 7f4337200640 1 --2- 172.21.15.32:0/417903017 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f433806d0f0 0x7f43380711f0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f4320009830 tx=0x7f432002a3f0 comp rx=0 tx=0).stop 2024-08-03T20:47:40.188 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:40.186+0000 7f4337200640 1 -- 172.21.15.32:0/417903017 shutdown_connections 2024-08-03T20:47:40.188 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:40.186+0000 7f4337200640 1 --2- 172.21.15.32:0/417903017 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f433815d310 0x7f43381594d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:40.188 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:40.186+0000 7f4337200640 1 --2- 172.21.15.32:0/417903017 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f433806d0f0 0x7f43380711f0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:40.188 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:40.186+0000 7f4337200640 1 -- 172.21.15.32:0/417903017 >> 172.21.15.32:0/417903017 conn(0x7f43380d0dd0 msgr2=0x7f43380d11e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:47:40.188 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:40.186+0000 7f4337200640 1 -- 172.21.15.32:0/417903017 shutdown_connections 2024-08-03T20:47:40.188 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:40.186+0000 7f4337200640 1 -- 172.21.15.32:0/417903017 wait complete. 2024-08-03T20:47:40.189 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:40.188+0000 7f4337200640 1 Processor -- start 2024-08-03T20:47:40.189 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:40.188+0000 7f4337200640 1 -- start start 2024-08-03T20:47:40.193 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:40.190+0000 7f4337200640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f43380cb420 0x7f43381e7ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:47:40.193 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:40.190+0000 7f4337200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f433815d310 0x7f43381e4080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:47:40.193 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:40.190+0000 7f4337200640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f43380cfb20 0x7f43380cff30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:47:40.193 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:40.190+0000 7f4337200640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f4338159b90 con 0x7f433815d310 2024-08-03T20:47:40.193 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:40.190+0000 7f4337200640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f4338159a10 con 0x7f43380cb420 2024-08-03T20:47:40.194 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:40.190+0000 7f4337200640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7f4338159d10 con 0x7f43380cfb20 2024-08-03T20:47:40.194 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:40.190+0000 7f432fe00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f433815d310 0x7f43381e4080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:47:40.194 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:40.190+0000 7f432fe00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f433815d310 0x7f43381e4080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:57230/0 (socket says 172.21.15.32:57230) 2024-08-03T20:47:40.194 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:40.190+0000 7f432fe00640 1 -- 172.21.15.32:0/634261035 learned_addr learned my addr 172.21.15.32:0/634261035 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:47:40.194 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:40.190+0000 7f432fe00640 1 -- 172.21.15.32:0/634261035 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f43380cfb20 msgr2=0x7f43380cff30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:47:40.194 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:40.190+0000 7f432fe00640 1 --2- 172.21.15.32:0/634261035 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f43380cfb20 0x7f43380cff30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:40.194 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:40.190+0000 7f432fe00640 1 -- 172.21.15.32:0/634261035 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f43380cb420 msgr2=0x7f43381e7ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:47:40.194 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:40.190+0000 7f432fe00640 1 --2- 172.21.15.32:0/634261035 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f43380cb420 0x7f43381e7ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:40.194 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:40.190+0000 7f432fe00640 1 -- 172.21.15.32:0/634261035 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f43380d0470 con 0x7f433815d310 2024-08-03T20:47:40.194 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:40.191+0000 7f432fe00640 1 --2- 172.21.15.32:0/634261035 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f433815d310 0x7f43381e4080 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f4320009950 tx=0x7f4320033c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:47:40.194 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:40.191+0000 7f432e000640 1 -- 172.21.15.32:0/634261035 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4320005c90 con 0x7f433815d310 2024-08-03T20:47:40.195 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:40.191+0000 7f432e000640 1 -- 172.21.15.32:0/634261035 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f4320031e50 con 0x7f433815d310 2024-08-03T20:47:40.195 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:40.191+0000 7f4337200640 1 -- 172.21.15.32:0/634261035 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f43381e5260 con 0x7f433815d310 2024-08-03T20:47:40.195 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:40.191+0000 7f432e000640 1 -- 172.21.15.32:0/634261035 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4320005260 con 0x7f433815d310 2024-08-03T20:47:40.195 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:40.191+0000 7f4337200640 1 -- 172.21.15.32:0/634261035 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f43381e5820 con 0x7f433815d310 2024-08-03T20:47:40.195 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:40.192+0000 7f4337200640 1 -- 172.21.15.32:0/634261035 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f433815a1c0 con 0x7f433815d310 2024-08-03T20:47:40.196 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:40.193+0000 7f432e000640 1 -- 172.21.15.32:0/634261035 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f4320005400 con 0x7f433815d310 2024-08-03T20:47:40.196 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:40.194+0000 7f432e000640 1 --2- 172.21.15.32:0/634261035 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f4314077010 0x7f4314079520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:47:40.196 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:40.194+0000 7f432e000640 1 -- 172.21.15.32:0/634261035 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f43200b9ab0 con 0x7f433815d310 2024-08-03T20:47:40.196 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:40.195+0000 7f4335e00640 1 --2- 172.21.15.32:0/634261035 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f4314077010 0x7f4314079520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:47:40.197 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:40.195+0000 7f4335e00640 1 --2- 172.21.15.32:0/634261035 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f4314077010 0x7f4314079520 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f4328007fd0 tx=0x7f4328007900 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:47:40.200 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:40.199+0000 7f432e000640 1 -- 172.21.15.32:0/634261035 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7f43380d0470 con 0x7f433815d310 2024-08-03T20:47:40.321 INFO:tasks.ceph.mds.c.smithi032.stderr:2024-08-03T20:47:40.320+0000 7f8eee600640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-08-03T20:47:40.402 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:40.401+0000 7f4337200640 1 -- 172.21.15.32:0/634261035 --> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f433806d820 con 0x7f4314077010 2024-08-03T20:47:40.408 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:40.407+0000 7f432e000640 1 -- 172.21.15.32:0/634261035 <== mgr.4106 v2:172.21.15.118:6800/4205950699 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+251936 (secure 0 0 0) 0x7f433806d820 con 0x7f4314077010 2024-08-03T20:47:40.412 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-03T20:47:40.413 INFO:teuthology.orchestra.run.smithi032.stderr:dumped all 2024-08-03T20:47:40.415 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:40.411+0000 7f4337200640 1 -- 172.21.15.32:0/634261035 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f4314077010 msgr2=0x7f4314079520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:47:40.415 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:40.411+0000 7f4337200640 1 --2- 172.21.15.32:0/634261035 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f4314077010 0x7f4314079520 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f4328007fd0 tx=0x7f4328007900 comp rx=0 tx=0).stop 2024-08-03T20:47:40.415 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:40.411+0000 7f4337200640 1 -- 172.21.15.32:0/634261035 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f433815d310 msgr2=0x7f43381e4080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:47:40.415 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:40.411+0000 7f4337200640 1 --2- 172.21.15.32:0/634261035 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f433815d310 0x7f43381e4080 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f4320009950 tx=0x7f4320033c20 comp rx=0 tx=0).stop 2024-08-03T20:47:40.415 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:40.411+0000 7f4337200640 1 -- 172.21.15.32:0/634261035 shutdown_connections 2024-08-03T20:47:40.416 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:40.411+0000 7f4337200640 1 --2- 172.21.15.32:0/634261035 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f4314077010 0x7f4314079520 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:40.416 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:40.411+0000 7f4337200640 1 --2- 172.21.15.32:0/634261035 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f43380cfb20 0x7f43380cff30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:40.416 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:40.411+0000 7f4337200640 1 --2- 172.21.15.32:0/634261035 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f433815d310 0x7f43381e4080 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:40.416 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:40.411+0000 7f4337200640 1 --2- 172.21.15.32:0/634261035 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f43380cb420 0x7f43381e7ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:40.416 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:40.411+0000 7f4337200640 1 -- 172.21.15.32:0/634261035 >> 172.21.15.32:0/634261035 conn(0x7f43380d0dd0 msgr2=0x7f43380b5e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:47:40.416 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:40.411+0000 7f4337200640 1 -- 172.21.15.32:0/634261035 shutdown_connections 2024-08-03T20:47:40.416 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:40.411+0000 7f4337200640 1 -- 172.21.15.32:0/634261035 wait complete. 2024-08-03T20:47:40.438 INFO:teuthology.orchestra.run.smithi032.stdout:{"pg_ready":true,"pg_map":{"version":20,"stamp":"2024-08-03T20:47:40.222169+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":146,"num_write_kb":1642,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":111,"ondisk_log_size":111,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":168,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":252316,"kb_used_data":2856,"kb_used_omap":12,"kb_used_meta":249331,"kb_avail":754722404,"statfs":{"total":773094113280,"available":772835741696,"internally_reserved":0,"allocated":2924544,"data_stored":1765884,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":255315505},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":28,"apply_latency_ms":28,"commit_latency_ns":28000000,"apply_latency_ns":28000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.310435"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-03T20:47:37.317266+0000","last_change":"2024-08-03T20:47:35.962230+0000","last_active":"2024-08-03T20:47:37.317266+0000","last_peered":"2024-08-03T20:47:37.317266+0000","last_clean":"2024-08-03T20:47:37.317266+0000","last_became_active":"2024-08-03T20:47:35.958864+0000","last_became_peered":"2024-08-03T20:47:35.958864+0000","last_unstale":"2024-08-03T20:47:37.317266+0000","last_undegraded":"2024-08-03T20:47:37.317266+0000","last_fullsized":"2024-08-03T20:47:37.317266+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_clean_scrub_stamp":"2024-08-03T20:47:34.904132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T03:01:41.602990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-03T20:47:36.935685+0000","last_change":"2024-08-03T20:47:35.941982+0000","last_active":"2024-08-03T20:47:36.935685+0000","last_peered":"2024-08-03T20:47:36.935685+0000","last_clean":"2024-08-03T20:47:36.935685+0000","last_became_active":"2024-08-03T20:47:35.941614+0000","last_became_peered":"2024-08-03T20:47:35.941614+0000","last_unstale":"2024-08-03T20:47:36.935685+0000","last_undegraded":"2024-08-03T20:47:36.935685+0000","last_fullsized":"2024-08-03T20:47:36.935685+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_clean_scrub_stamp":"2024-08-03T20:47:34.904132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T01:08:56.085650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-03T20:47:37.979574+0000","last_change":"2024-08-03T20:47:35.926803+0000","last_active":"2024-08-03T20:47:37.979574+0000","last_peered":"2024-08-03T20:47:37.979574+0000","last_clean":"2024-08-03T20:47:37.979574+0000","last_became_active":"2024-08-03T20:47:35.926198+0000","last_became_peered":"2024-08-03T20:47:35.926198+0000","last_unstale":"2024-08-03T20:47:37.979574+0000","last_undegraded":"2024-08-03T20:47:37.979574+0000","last_fullsized":"2024-08-03T20:47:37.979574+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_clean_scrub_stamp":"2024-08-03T20:47:34.904132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T04:57:58.179012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-03T20:47:37.979134+0000","last_change":"2024-08-03T20:47:35.960607+0000","last_active":"2024-08-03T20:47:37.979134+0000","last_peered":"2024-08-03T20:47:37.979134+0000","last_clean":"2024-08-03T20:47:37.979134+0000","last_became_active":"2024-08-03T20:47:35.956844+0000","last_became_peered":"2024-08-03T20:47:35.956844+0000","last_unstale":"2024-08-03T20:47:37.979134+0000","last_undegraded":"2024-08-03T20:47:37.979134+0000","last_fullsized":"2024-08-03T20:47:37.979134+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_clean_scrub_stamp":"2024-08-03T20:47:34.904132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T07:48:20.679795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-03T20:47:37.316984+0000","last_change":"2024-08-03T20:47:35.962693+0000","last_active":"2024-08-03T20:47:37.316984+0000","last_peered":"2024-08-03T20:47:37.316984+0000","last_clean":"2024-08-03T20:47:37.316984+0000","last_became_active":"2024-08-03T20:47:35.959911+0000","last_became_peered":"2024-08-03T20:47:35.959911+0000","last_unstale":"2024-08-03T20:47:37.316984+0000","last_undegraded":"2024-08-03T20:47:37.316984+0000","last_fullsized":"2024-08-03T20:47:37.316984+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_clean_scrub_stamp":"2024-08-03T20:47:34.904132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T04:51:27.371858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-03T20:47:37.979016+0000","last_change":"2024-08-03T20:47:35.947517+0000","last_active":"2024-08-03T20:47:37.979016+0000","last_peered":"2024-08-03T20:47:37.979016+0000","last_clean":"2024-08-03T20:47:37.979016+0000","last_became_active":"2024-08-03T20:47:35.947167+0000","last_became_peered":"2024-08-03T20:47:35.947167+0000","last_unstale":"2024-08-03T20:47:37.979016+0000","last_undegraded":"2024-08-03T20:47:37.979016+0000","last_fullsized":"2024-08-03T20:47:37.979016+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_clean_scrub_stamp":"2024-08-03T20:47:34.904132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-04T22:33:13.028920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-03T20:47:37.316673+0000","last_change":"2024-08-03T20:47:35.947270+0000","last_active":"2024-08-03T20:47:37.316673+0000","last_peered":"2024-08-03T20:47:37.316673+0000","last_clean":"2024-08-03T20:47:37.316673+0000","last_became_active":"2024-08-03T20:47:35.946647+0000","last_became_peered":"2024-08-03T20:47:35.946647+0000","last_unstale":"2024-08-03T20:47:37.316673+0000","last_undegraded":"2024-08-03T20:47:37.316673+0000","last_fullsized":"2024-08-03T20:47:37.316673+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_clean_scrub_stamp":"2024-08-03T20:47:34.904132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T00:10:43.343156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-03T20:47:37.978113+0000","last_change":"2024-08-03T20:47:35.946701+0000","last_active":"2024-08-03T20:47:37.978113+0000","last_peered":"2024-08-03T20:47:37.978113+0000","last_clean":"2024-08-03T20:47:37.978113+0000","last_became_active":"2024-08-03T20:47:35.946340+0000","last_became_peered":"2024-08-03T20:47:35.946340+0000","last_unstale":"2024-08-03T20:47:37.978113+0000","last_undegraded":"2024-08-03T20:47:37.978113+0000","last_fullsized":"2024-08-03T20:47:37.978113+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_clean_scrub_stamp":"2024-08-03T20:47:34.904132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T07:21:44.279582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-03T20:47:37.317505+0000","last_change":"2024-08-03T20:47:35.963774+0000","last_active":"2024-08-03T20:47:37.317505+0000","last_peered":"2024-08-03T20:47:37.317505+0000","last_clean":"2024-08-03T20:47:37.317505+0000","last_became_active":"2024-08-03T20:47:35.958408+0000","last_became_peered":"2024-08-03T20:47:35.958408+0000","last_unstale":"2024-08-03T20:47:37.317505+0000","last_undegraded":"2024-08-03T20:47:37.317505+0000","last_fullsized":"2024-08-03T20:47:37.317505+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_clean_scrub_stamp":"2024-08-03T20:47:34.904132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T00:58:45.652167+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-03T20:47:36.941791+0000","last_change":"2024-08-03T20:47:35.943147+0000","last_active":"2024-08-03T20:47:36.941791+0000","last_peered":"2024-08-03T20:47:36.941791+0000","last_clean":"2024-08-03T20:47:36.941791+0000","last_became_active":"2024-08-03T20:47:35.942253+0000","last_became_peered":"2024-08-03T20:47:35.942253+0000","last_unstale":"2024-08-03T20:47:36.941791+0000","last_undegraded":"2024-08-03T20:47:36.941791+0000","last_fullsized":"2024-08-03T20:47:36.941791+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_clean_scrub_stamp":"2024-08-03T20:47:34.904132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T01:08:56.085650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-03T20:47:36.941585+0000","last_change":"2024-08-03T20:47:35.938819+0000","last_active":"2024-08-03T20:47:36.941585+0000","last_peered":"2024-08-03T20:47:36.941585+0000","last_clean":"2024-08-03T20:47:36.941585+0000","last_became_active":"2024-08-03T20:47:35.938499+0000","last_became_peered":"2024-08-03T20:47:35.938499+0000","last_unstale":"2024-08-03T20:47:36.941585+0000","last_undegraded":"2024-08-03T20:47:36.941585+0000","last_fullsized":"2024-08-03T20:47:36.941585+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_clean_scrub_stamp":"2024-08-03T20:47:34.904132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T04:57:58.179012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-03T20:47:36.936210+0000","last_change":"2024-08-03T20:47:35.936873+0000","last_active":"2024-08-03T20:47:36.936210+0000","last_peered":"2024-08-03T20:47:36.936210+0000","last_clean":"2024-08-03T20:47:36.936210+0000","last_became_active":"2024-08-03T20:47:35.936005+0000","last_became_peered":"2024-08-03T20:47:35.936005+0000","last_unstale":"2024-08-03T20:47:36.936210+0000","last_undegraded":"2024-08-03T20:47:36.936210+0000","last_fullsized":"2024-08-03T20:47:36.936210+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_clean_scrub_stamp":"2024-08-03T20:47:34.904132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T01:42:26.761950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-03T20:47:37.978994+0000","last_change":"2024-08-03T20:47:35.950146+0000","last_active":"2024-08-03T20:47:37.978994+0000","last_peered":"2024-08-03T20:47:37.978994+0000","last_clean":"2024-08-03T20:47:37.978994+0000","last_became_active":"2024-08-03T20:47:35.949842+0000","last_became_peered":"2024-08-03T20:47:35.949842+0000","last_unstale":"2024-08-03T20:47:37.978994+0000","last_undegraded":"2024-08-03T20:47:37.978994+0000","last_fullsized":"2024-08-03T20:47:37.978994+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_clean_scrub_stamp":"2024-08-03T20:47:34.904132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-04T22:48:57.612773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-03T20:47:37.979037+0000","last_change":"2024-08-03T20:47:35.940356+0000","last_active":"2024-08-03T20:47:37.979037+0000","last_peered":"2024-08-03T20:47:37.979037+0000","last_clean":"2024-08-03T20:47:37.979037+0000","last_became_active":"2024-08-03T20:47:35.940143+0000","last_became_peered":"2024-08-03T20:47:35.940143+0000","last_unstale":"2024-08-03T20:47:37.979037+0000","last_undegraded":"2024-08-03T20:47:37.979037+0000","last_fullsized":"2024-08-03T20:47:37.979037+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_clean_scrub_stamp":"2024-08-03T20:47:34.904132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T04:01:08.373245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-03T20:47:36.935545+0000","last_change":"2024-08-03T20:47:35.951009+0000","last_active":"2024-08-03T20:47:36.935545+0000","last_peered":"2024-08-03T20:47:36.935545+0000","last_clean":"2024-08-03T20:47:36.935545+0000","last_became_active":"2024-08-03T20:47:35.950658+0000","last_became_peered":"2024-08-03T20:47:35.950658+0000","last_unstale":"2024-08-03T20:47:36.935545+0000","last_undegraded":"2024-08-03T20:47:36.935545+0000","last_fullsized":"2024-08-03T20:47:36.935545+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_clean_scrub_stamp":"2024-08-03T20:47:34.904132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-04T20:49:19.998935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-03T20:47:37.978861+0000","last_change":"2024-08-03T20:47:35.948379+0000","last_active":"2024-08-03T20:47:37.978861+0000","last_peered":"2024-08-03T20:47:37.978861+0000","last_clean":"2024-08-03T20:47:37.978861+0000","last_became_active":"2024-08-03T20:47:35.947812+0000","last_became_peered":"2024-08-03T20:47:35.947812+0000","last_unstale":"2024-08-03T20:47:37.978861+0000","last_undegraded":"2024-08-03T20:47:37.978861+0000","last_fullsized":"2024-08-03T20:47:37.978861+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_clean_scrub_stamp":"2024-08-03T20:47:34.904132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T01:08:56.085650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-03T20:47:37.979587+0000","last_change":"2024-08-03T20:47:35.939474+0000","last_active":"2024-08-03T20:47:37.979587+0000","last_peered":"2024-08-03T20:47:37.979587+0000","last_clean":"2024-08-03T20:47:37.979587+0000","last_became_active":"2024-08-03T20:47:35.938914+0000","last_became_peered":"2024-08-03T20:47:35.938914+0000","last_unstale":"2024-08-03T20:47:37.979587+0000","last_undegraded":"2024-08-03T20:47:37.979587+0000","last_fullsized":"2024-08-03T20:47:37.979587+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_clean_scrub_stamp":"2024-08-03T20:47:34.904132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T01:52:59.081172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-03T20:47:37.352899+0000","last_change":"2024-08-03T20:47:35.933829+0000","last_active":"2024-08-03T20:47:37.352899+0000","last_peered":"2024-08-03T20:47:37.352899+0000","last_clean":"2024-08-03T20:47:37.352899+0000","last_became_active":"2024-08-03T20:47:35.933515+0000","last_became_peered":"2024-08-03T20:47:35.933515+0000","last_unstale":"2024-08-03T20:47:37.352899+0000","last_undegraded":"2024-08-03T20:47:37.352899+0000","last_fullsized":"2024-08-03T20:47:37.352899+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_clean_scrub_stamp":"2024-08-03T20:47:34.904132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T02:10:57.293691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-03T20:47:37.317301+0000","last_change":"2024-08-03T20:47:35.952999+0000","last_active":"2024-08-03T20:47:37.317301+0000","last_peered":"2024-08-03T20:47:37.317301+0000","last_clean":"2024-08-03T20:47:37.317301+0000","last_became_active":"2024-08-03T20:47:35.952281+0000","last_became_peered":"2024-08-03T20:47:35.952281+0000","last_unstale":"2024-08-03T20:47:37.317301+0000","last_undegraded":"2024-08-03T20:47:37.317301+0000","last_fullsized":"2024-08-03T20:47:37.317301+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_clean_scrub_stamp":"2024-08-03T20:47:34.904132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T01:15:48.374663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-03T20:47:36.941659+0000","last_change":"2024-08-03T20:47:35.927412+0000","last_active":"2024-08-03T20:47:36.941659+0000","last_peered":"2024-08-03T20:47:36.941659+0000","last_clean":"2024-08-03T20:47:36.941659+0000","last_became_active":"2024-08-03T20:47:35.927069+0000","last_became_peered":"2024-08-03T20:47:35.927069+0000","last_unstale":"2024-08-03T20:47:36.941659+0000","last_undegraded":"2024-08-03T20:47:36.941659+0000","last_fullsized":"2024-08-03T20:47:36.941659+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_clean_scrub_stamp":"2024-08-03T20:47:34.904132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-04T20:49:19.998935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-03T20:47:37.979382+0000","last_change":"2024-08-03T20:47:35.936665+0000","last_active":"2024-08-03T20:47:37.979382+0000","last_peered":"2024-08-03T20:47:37.979382+0000","last_clean":"2024-08-03T20:47:37.979382+0000","last_became_active":"2024-08-03T20:47:35.936298+0000","last_became_peered":"2024-08-03T20:47:35.936298+0000","last_unstale":"2024-08-03T20:47:37.979382+0000","last_undegraded":"2024-08-03T20:47:37.979382+0000","last_fullsized":"2024-08-03T20:47:37.979382+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_clean_scrub_stamp":"2024-08-03T20:47:34.904132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-04T22:12:15.293757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-03T20:47:37.979017+0000","last_change":"2024-08-03T20:47:35.938468+0000","last_active":"2024-08-03T20:47:37.979017+0000","last_peered":"2024-08-03T20:47:37.979017+0000","last_clean":"2024-08-03T20:47:37.979017+0000","last_became_active":"2024-08-03T20:47:35.937757+0000","last_became_peered":"2024-08-03T20:47:35.937757+0000","last_unstale":"2024-08-03T20:47:37.979017+0000","last_undegraded":"2024-08-03T20:47:37.979017+0000","last_fullsized":"2024-08-03T20:47:37.979017+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_clean_scrub_stamp":"2024-08-03T20:47:34.904132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-04T23:07:18.318169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-03T20:47:37.978241+0000","last_change":"2024-08-03T20:47:35.953502+0000","last_active":"2024-08-03T20:47:37.978241+0000","last_peered":"2024-08-03T20:47:37.978241+0000","last_clean":"2024-08-03T20:47:37.978241+0000","last_became_active":"2024-08-03T20:47:35.953152+0000","last_became_peered":"2024-08-03T20:47:35.953152+0000","last_unstale":"2024-08-03T20:47:37.978241+0000","last_undegraded":"2024-08-03T20:47:37.978241+0000","last_fullsized":"2024-08-03T20:47:37.978241+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_clean_scrub_stamp":"2024-08-03T20:47:34.904132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T01:56:58.984458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-03T20:47:37.979235+0000","last_change":"2024-08-03T20:47:35.937927+0000","last_active":"2024-08-03T20:47:37.979235+0000","last_peered":"2024-08-03T20:47:37.979235+0000","last_clean":"2024-08-03T20:47:37.979235+0000","last_became_active":"2024-08-03T20:47:35.937579+0000","last_became_peered":"2024-08-03T20:47:35.937579+0000","last_unstale":"2024-08-03T20:47:37.979235+0000","last_undegraded":"2024-08-03T20:47:37.979235+0000","last_fullsized":"2024-08-03T20:47:37.979235+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_clean_scrub_stamp":"2024-08-03T20:47:34.904132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T03:44:29.605859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-03T20:47:37.979020+0000","last_change":"2024-08-03T20:47:35.953835+0000","last_active":"2024-08-03T20:47:37.979020+0000","last_peered":"2024-08-03T20:47:37.979020+0000","last_clean":"2024-08-03T20:47:37.979020+0000","last_became_active":"2024-08-03T20:47:35.953520+0000","last_became_peered":"2024-08-03T20:47:35.953520+0000","last_unstale":"2024-08-03T20:47:37.979020+0000","last_undegraded":"2024-08-03T20:47:37.979020+0000","last_fullsized":"2024-08-03T20:47:37.979020+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_clean_scrub_stamp":"2024-08-03T20:47:34.904132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T06:11:09.513212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-03T20:47:36.941856+0000","last_change":"2024-08-03T20:47:35.938228+0000","last_active":"2024-08-03T20:47:36.941856+0000","last_peered":"2024-08-03T20:47:36.941856+0000","last_clean":"2024-08-03T20:47:36.941856+0000","last_became_active":"2024-08-03T20:47:35.937893+0000","last_became_peered":"2024-08-03T20:47:35.937893+0000","last_unstale":"2024-08-03T20:47:36.941856+0000","last_undegraded":"2024-08-03T20:47:36.941856+0000","last_fullsized":"2024-08-03T20:47:36.941856+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_clean_scrub_stamp":"2024-08-03T20:47:34.904132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-04T22:33:13.028920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-03T20:47:36.935344+0000","last_change":"2024-08-03T20:47:35.937679+0000","last_active":"2024-08-03T20:47:36.935344+0000","last_peered":"2024-08-03T20:47:36.935344+0000","last_clean":"2024-08-03T20:47:36.935344+0000","last_became_active":"2024-08-03T20:47:35.937328+0000","last_became_peered":"2024-08-03T20:47:35.937328+0000","last_unstale":"2024-08-03T20:47:36.935344+0000","last_undegraded":"2024-08-03T20:47:36.935344+0000","last_fullsized":"2024-08-03T20:47:36.935344+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_clean_scrub_stamp":"2024-08-03T20:47:34.904132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T04:57:58.179012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-03T20:47:37.978822+0000","last_change":"2024-08-03T20:47:35.954328+0000","last_active":"2024-08-03T20:47:37.978822+0000","last_peered":"2024-08-03T20:47:37.978822+0000","last_clean":"2024-08-03T20:47:37.978822+0000","last_became_active":"2024-08-03T20:47:35.951285+0000","last_became_peered":"2024-08-03T20:47:35.951285+0000","last_unstale":"2024-08-03T20:47:37.978822+0000","last_undegraded":"2024-08-03T20:47:37.978822+0000","last_fullsized":"2024-08-03T20:47:37.978822+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_clean_scrub_stamp":"2024-08-03T20:47:34.904132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T01:01:23.625299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-03T20:47:37.979494+0000","last_change":"2024-08-03T20:47:35.940504+0000","last_active":"2024-08-03T20:47:37.979494+0000","last_peered":"2024-08-03T20:47:37.979494+0000","last_clean":"2024-08-03T20:47:37.979494+0000","last_became_active":"2024-08-03T20:47:35.938638+0000","last_became_peered":"2024-08-03T20:47:35.938638+0000","last_unstale":"2024-08-03T20:47:37.979494+0000","last_undegraded":"2024-08-03T20:47:37.979494+0000","last_fullsized":"2024-08-03T20:47:37.979494+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_clean_scrub_stamp":"2024-08-03T20:47:34.904132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-04T23:57:18.840519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-03T20:47:37.979441+0000","last_change":"2024-08-03T20:47:35.938131+0000","last_active":"2024-08-03T20:47:37.979441+0000","last_peered":"2024-08-03T20:47:37.979441+0000","last_clean":"2024-08-03T20:47:37.979441+0000","last_became_active":"2024-08-03T20:47:35.937270+0000","last_became_peered":"2024-08-03T20:47:35.937270+0000","last_unstale":"2024-08-03T20:47:37.979441+0000","last_undegraded":"2024-08-03T20:47:37.979441+0000","last_fullsized":"2024-08-03T20:47:37.979441+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_clean_scrub_stamp":"2024-08-03T20:47:34.904132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-04T21:57:09.430415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-03T20:47:37.352726+0000","last_change":"2024-08-03T20:47:35.937957+0000","last_active":"2024-08-03T20:47:37.352726+0000","last_peered":"2024-08-03T20:47:37.352726+0000","last_clean":"2024-08-03T20:47:37.352726+0000","last_became_active":"2024-08-03T20:47:35.937656+0000","last_became_peered":"2024-08-03T20:47:35.937656+0000","last_unstale":"2024-08-03T20:47:37.352726+0000","last_undegraded":"2024-08-03T20:47:37.352726+0000","last_fullsized":"2024-08-03T20:47:37.352726+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_clean_scrub_stamp":"2024-08-03T20:47:34.904132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T05:51:29.523373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-03T20:47:36.941128+0000","last_change":"2024-08-03T20:47:35.932459+0000","last_active":"2024-08-03T20:47:36.941128+0000","last_peered":"2024-08-03T20:47:36.941128+0000","last_clean":"2024-08-03T20:47:36.941128+0000","last_became_active":"2024-08-03T20:47:35.932103+0000","last_became_peered":"2024-08-03T20:47:35.932103+0000","last_unstale":"2024-08-03T20:47:36.941128+0000","last_undegraded":"2024-08-03T20:47:36.941128+0000","last_fullsized":"2024-08-03T20:47:36.941128+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_clean_scrub_stamp":"2024-08-03T20:47:34.904132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T08:34:04.057775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-03T20:47:36.935757+0000","last_change":"2024-08-03T20:47:35.940544+0000","last_active":"2024-08-03T20:47:36.935757+0000","last_peered":"2024-08-03T20:47:36.935757+0000","last_clean":"2024-08-03T20:47:36.935757+0000","last_became_active":"2024-08-03T20:47:35.940218+0000","last_became_peered":"2024-08-03T20:47:35.940218+0000","last_unstale":"2024-08-03T20:47:36.935757+0000","last_undegraded":"2024-08-03T20:47:36.935757+0000","last_fullsized":"2024-08-03T20:47:36.935757+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_clean_scrub_stamp":"2024-08-03T20:47:34.904132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T07:58:42.869702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"20'1","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-03T20:47:37.978814+0000","last_change":"2024-08-03T20:47:34.939881+0000","last_active":"2024-08-03T20:47:37.978814+0000","last_peered":"2024-08-03T20:47:37.978814+0000","last_clean":"2024-08-03T20:47:37.978814+0000","last_became_active":"2024-08-03T20:47:34.936765+0000","last_became_peered":"2024-08-03T20:47:34.936765+0000","last_unstale":"2024-08-03T20:47:37.978814+0000","last_undegraded":"2024-08-03T20:47:37.978814+0000","last_fullsized":"2024-08-03T20:47:37.978814+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:47:33.895731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:47:33.895731+0000","last_clean_scrub_stamp":"2024-08-03T20:47:33.895731+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T02:29:41.523344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-03T20:47:37.979599+0000","last_change":"2024-08-03T20:47:35.942088+0000","last_active":"2024-08-03T20:47:37.979599+0000","last_peered":"2024-08-03T20:47:37.979599+0000","last_clean":"2024-08-03T20:47:37.979599+0000","last_became_active":"2024-08-03T20:47:35.941800+0000","last_became_peered":"2024-08-03T20:47:35.941800+0000","last_unstale":"2024-08-03T20:47:37.979599+0000","last_undegraded":"2024-08-03T20:47:37.979599+0000","last_fullsized":"2024-08-03T20:47:37.979599+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_clean_scrub_stamp":"2024-08-03T20:47:34.904132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-04T20:49:19.998935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"20'1","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-03T20:47:36.941066+0000","last_change":"2024-08-03T20:47:34.938348+0000","last_active":"2024-08-03T20:47:36.941066+0000","last_peered":"2024-08-03T20:47:36.941066+0000","last_clean":"2024-08-03T20:47:36.941066+0000","last_became_active":"2024-08-03T20:47:34.936169+0000","last_became_peered":"2024-08-03T20:47:34.936169+0000","last_unstale":"2024-08-03T20:47:36.941066+0000","last_undegraded":"2024-08-03T20:47:36.941066+0000","last_fullsized":"2024-08-03T20:47:36.941066+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:47:33.895731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:47:33.895731+0000","last_clean_scrub_stamp":"2024-08-03T20:47:33.895731+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T06:44:37.944928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-03T20:47:37.978344+0000","last_change":"2024-08-03T20:47:34.935991+0000","last_active":"2024-08-03T20:47:37.978344+0000","last_peered":"2024-08-03T20:47:37.978344+0000","last_clean":"2024-08-03T20:47:37.978344+0000","last_became_active":"2024-08-03T20:47:34.930541+0000","last_became_peered":"2024-08-03T20:47:34.930541+0000","last_unstale":"2024-08-03T20:47:37.978344+0000","last_undegraded":"2024-08-03T20:47:37.978344+0000","last_fullsized":"2024-08-03T20:47:37.978344+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:47:33.895731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:47:33.895731+0000","last_clean_scrub_stamp":"2024-08-03T20:47:33.895731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T04:37:55.336635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-03T20:47:37.978164+0000","last_change":"2024-08-03T20:47:35.934722+0000","last_active":"2024-08-03T20:47:37.978164+0000","last_peered":"2024-08-03T20:47:37.978164+0000","last_clean":"2024-08-03T20:47:37.978164+0000","last_became_active":"2024-08-03T20:47:35.934183+0000","last_became_peered":"2024-08-03T20:47:35.934183+0000","last_unstale":"2024-08-03T20:47:37.978164+0000","last_undegraded":"2024-08-03T20:47:37.978164+0000","last_fullsized":"2024-08-03T20:47:37.978164+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_clean_scrub_stamp":"2024-08-03T20:47:34.904132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T06:29:11.740575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-03T20:47:37.978945+0000","last_change":"2024-08-03T20:47:34.934591+0000","last_active":"2024-08-03T20:47:37.978945+0000","last_peered":"2024-08-03T20:47:37.978945+0000","last_clean":"2024-08-03T20:47:37.978945+0000","last_became_active":"2024-08-03T20:47:34.934155+0000","last_became_peered":"2024-08-03T20:47:34.934155+0000","last_unstale":"2024-08-03T20:47:37.978945+0000","last_undegraded":"2024-08-03T20:47:37.978945+0000","last_fullsized":"2024-08-03T20:47:37.978945+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:47:33.895731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:47:33.895731+0000","last_clean_scrub_stamp":"2024-08-03T20:47:33.895731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-04T22:30:28.119577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-03T20:47:36.934971+0000","last_change":"2024-08-03T20:47:35.947874+0000","last_active":"2024-08-03T20:47:36.934971+0000","last_peered":"2024-08-03T20:47:36.934971+0000","last_clean":"2024-08-03T20:47:36.934971+0000","last_became_active":"2024-08-03T20:47:35.947570+0000","last_became_peered":"2024-08-03T20:47:35.947570+0000","last_unstale":"2024-08-03T20:47:36.934971+0000","last_undegraded":"2024-08-03T20:47:36.934971+0000","last_fullsized":"2024-08-03T20:47:36.934971+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_clean_scrub_stamp":"2024-08-03T20:47:34.904132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-04T22:12:15.293757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-03T20:47:36.941786+0000","last_change":"2024-08-03T20:47:34.937955+0000","last_active":"2024-08-03T20:47:36.941786+0000","last_peered":"2024-08-03T20:47:36.941786+0000","last_clean":"2024-08-03T20:47:36.941786+0000","last_became_active":"2024-08-03T20:47:34.937613+0000","last_became_peered":"2024-08-03T20:47:34.937613+0000","last_unstale":"2024-08-03T20:47:36.941786+0000","last_undegraded":"2024-08-03T20:47:36.941786+0000","last_fullsized":"2024-08-03T20:47:36.941786+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:47:33.895731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:47:33.895731+0000","last_clean_scrub_stamp":"2024-08-03T20:47:33.895731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-04T22:30:28.119577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-03T20:47:37.979527+0000","last_change":"2024-08-03T20:47:35.941948+0000","last_active":"2024-08-03T20:47:37.979527+0000","last_peered":"2024-08-03T20:47:37.979527+0000","last_clean":"2024-08-03T20:47:37.979527+0000","last_became_active":"2024-08-03T20:47:35.941651+0000","last_became_peered":"2024-08-03T20:47:35.941651+0000","last_unstale":"2024-08-03T20:47:37.979527+0000","last_undegraded":"2024-08-03T20:47:37.979527+0000","last_fullsized":"2024-08-03T20:47:37.979527+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_clean_scrub_stamp":"2024-08-03T20:47:34.904132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T06:29:37.397869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-03T20:47:37.979267+0000","last_change":"2024-08-03T20:47:34.936965+0000","last_active":"2024-08-03T20:47:37.979267+0000","last_peered":"2024-08-03T20:47:37.979267+0000","last_clean":"2024-08-03T20:47:37.979267+0000","last_became_active":"2024-08-03T20:47:34.936667+0000","last_became_peered":"2024-08-03T20:47:34.936667+0000","last_unstale":"2024-08-03T20:47:37.979267+0000","last_undegraded":"2024-08-03T20:47:37.979267+0000","last_fullsized":"2024-08-03T20:47:37.979267+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:47:33.895731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:47:33.895731+0000","last_clean_scrub_stamp":"2024-08-03T20:47:33.895731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T07:46:07.162679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-03T20:47:37.979788+0000","last_change":"2024-08-03T20:47:35.940917+0000","last_active":"2024-08-03T20:47:37.979788+0000","last_peered":"2024-08-03T20:47:37.979788+0000","last_clean":"2024-08-03T20:47:37.979788+0000","last_became_active":"2024-08-03T20:47:35.940218+0000","last_became_peered":"2024-08-03T20:47:35.940218+0000","last_unstale":"2024-08-03T20:47:37.979788+0000","last_undegraded":"2024-08-03T20:47:37.979788+0000","last_fullsized":"2024-08-03T20:47:37.979788+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_clean_scrub_stamp":"2024-08-03T20:47:34.904132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-04T23:07:18.318169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"20'3","reported_seq":18,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-03T20:47:36.935425+0000","last_change":"2024-08-03T20:47:34.939100+0000","last_active":"2024-08-03T20:47:36.935425+0000","last_peered":"2024-08-03T20:47:36.935425+0000","last_clean":"2024-08-03T20:47:36.935425+0000","last_became_active":"2024-08-03T20:47:34.935869+0000","last_became_peered":"2024-08-03T20:47:34.935869+0000","last_unstale":"2024-08-03T20:47:36.935425+0000","last_undegraded":"2024-08-03T20:47:36.935425+0000","last_fullsized":"2024-08-03T20:47:36.935425+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:47:33.895731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:47:33.895731+0000","last_clean_scrub_stamp":"2024-08-03T20:47:33.895731+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T06:44:37.944928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-03T20:47:36.936329+0000","last_change":"2024-08-03T20:47:35.944428+0000","last_active":"2024-08-03T20:47:36.936329+0000","last_peered":"2024-08-03T20:47:36.936329+0000","last_clean":"2024-08-03T20:47:36.936329+0000","last_became_active":"2024-08-03T20:47:35.943608+0000","last_became_peered":"2024-08-03T20:47:35.943608+0000","last_unstale":"2024-08-03T20:47:36.936329+0000","last_undegraded":"2024-08-03T20:47:36.936329+0000","last_fullsized":"2024-08-03T20:47:36.936329+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_clean_scrub_stamp":"2024-08-03T20:47:34.904132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T05:51:29.523373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"20'6","reported_seq":21,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-03T20:47:36.941564+0000","last_change":"2024-08-03T20:47:34.931832+0000","last_active":"2024-08-03T20:47:36.941564+0000","last_peered":"2024-08-03T20:47:36.941564+0000","last_clean":"2024-08-03T20:47:36.941564+0000","last_became_active":"2024-08-03T20:47:34.931150+0000","last_became_peered":"2024-08-03T20:47:34.931150+0000","last_unstale":"2024-08-03T20:47:36.941564+0000","last_undegraded":"2024-08-03T20:47:36.941564+0000","last_fullsized":"2024-08-03T20:47:36.941564+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:47:33.895731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:47:33.895731+0000","last_clean_scrub_stamp":"2024-08-03T20:47:33.895731+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T05:01:48.922382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-03T20:47:37.316866+0000","last_change":"2024-08-03T20:47:35.953367+0000","last_active":"2024-08-03T20:47:37.316866+0000","last_peered":"2024-08-03T20:47:37.316866+0000","last_clean":"2024-08-03T20:47:37.316866+0000","last_became_active":"2024-08-03T20:47:35.952666+0000","last_became_peered":"2024-08-03T20:47:35.952666+0000","last_unstale":"2024-08-03T20:47:37.316866+0000","last_undegraded":"2024-08-03T20:47:37.316866+0000","last_fullsized":"2024-08-03T20:47:37.316866+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_clean_scrub_stamp":"2024-08-03T20:47:34.904132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-04T21:57:09.430415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"20'2","reported_seq":17,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-03T20:47:37.979652+0000","last_change":"2024-08-03T20:47:34.932411+0000","last_active":"2024-08-03T20:47:37.979652+0000","last_peered":"2024-08-03T20:47:37.979652+0000","last_clean":"2024-08-03T20:47:37.979652+0000","last_became_active":"2024-08-03T20:47:34.931336+0000","last_became_peered":"2024-08-03T20:47:34.931336+0000","last_unstale":"2024-08-03T20:47:37.979652+0000","last_undegraded":"2024-08-03T20:47:37.979652+0000","last_fullsized":"2024-08-03T20:47:37.979652+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:47:33.895731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:47:33.895731+0000","last_clean_scrub_stamp":"2024-08-03T20:47:33.895731+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T05:59:33.076657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-03T20:47:37.979070+0000","last_change":"2024-08-03T20:47:35.932875+0000","last_active":"2024-08-03T20:47:37.979070+0000","last_peered":"2024-08-03T20:47:37.979070+0000","last_clean":"2024-08-03T20:47:37.979070+0000","last_became_active":"2024-08-03T20:47:35.932516+0000","last_became_peered":"2024-08-03T20:47:35.932516+0000","last_unstale":"2024-08-03T20:47:37.979070+0000","last_undegraded":"2024-08-03T20:47:37.979070+0000","last_fullsized":"2024-08-03T20:47:37.979070+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_clean_scrub_stamp":"2024-08-03T20:47:34.904132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T08:34:04.057775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-03T20:47:37.979194+0000","last_change":"2024-08-03T20:47:34.947236+0000","last_active":"2024-08-03T20:47:37.979194+0000","last_peered":"2024-08-03T20:47:37.979194+0000","last_clean":"2024-08-03T20:47:37.979194+0000","last_became_active":"2024-08-03T20:47:34.946894+0000","last_became_peered":"2024-08-03T20:47:34.946894+0000","last_unstale":"2024-08-03T20:47:37.979194+0000","last_undegraded":"2024-08-03T20:47:37.979194+0000","last_fullsized":"2024-08-03T20:47:37.979194+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:47:33.895731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:47:33.895731+0000","last_clean_scrub_stamp":"2024-08-03T20:47:33.895731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T05:01:48.922382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-03T20:47:37.317004+0000","last_change":"2024-08-03T20:47:35.950815+0000","last_active":"2024-08-03T20:47:37.317004+0000","last_peered":"2024-08-03T20:47:37.317004+0000","last_clean":"2024-08-03T20:47:37.317004+0000","last_became_active":"2024-08-03T20:47:35.950494+0000","last_became_peered":"2024-08-03T20:47:35.950494+0000","last_unstale":"2024-08-03T20:47:37.317004+0000","last_undegraded":"2024-08-03T20:47:37.317004+0000","last_fullsized":"2024-08-03T20:47:37.317004+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_clean_scrub_stamp":"2024-08-03T20:47:34.904132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T05:49:42.422083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-03T20:47:36.941067+0000","last_change":"2024-08-03T20:47:34.939132+0000","last_active":"2024-08-03T20:47:36.941067+0000","last_peered":"2024-08-03T20:47:36.941067+0000","last_clean":"2024-08-03T20:47:36.941067+0000","last_became_active":"2024-08-03T20:47:34.937563+0000","last_became_peered":"2024-08-03T20:47:34.937563+0000","last_unstale":"2024-08-03T20:47:36.941067+0000","last_undegraded":"2024-08-03T20:47:36.941067+0000","last_fullsized":"2024-08-03T20:47:36.941067+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:47:33.895731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:47:33.895731+0000","last_clean_scrub_stamp":"2024-08-03T20:47:33.895731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T06:32:14.208950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-03T20:47:36.941315+0000","last_change":"2024-08-03T20:47:35.944827+0000","last_active":"2024-08-03T20:47:36.941315+0000","last_peered":"2024-08-03T20:47:36.941315+0000","last_clean":"2024-08-03T20:47:36.941315+0000","last_became_active":"2024-08-03T20:47:35.944517+0000","last_became_peered":"2024-08-03T20:47:35.944517+0000","last_unstale":"2024-08-03T20:47:36.941315+0000","last_undegraded":"2024-08-03T20:47:36.941315+0000","last_fullsized":"2024-08-03T20:47:36.941315+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_clean_scrub_stamp":"2024-08-03T20:47:34.904132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-04T23:57:18.840519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"20'8","reported_seq":23,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-03T20:47:37.352795+0000","last_change":"2024-08-03T20:47:34.936194+0000","last_active":"2024-08-03T20:47:37.352795+0000","last_peered":"2024-08-03T20:47:37.352795+0000","last_clean":"2024-08-03T20:47:37.352795+0000","last_became_active":"2024-08-03T20:47:34.935835+0000","last_became_peered":"2024-08-03T20:47:34.935835+0000","last_unstale":"2024-08-03T20:47:37.352795+0000","last_undegraded":"2024-08-03T20:47:37.352795+0000","last_fullsized":"2024-08-03T20:47:37.352795+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:47:33.895731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:47:33.895731+0000","last_clean_scrub_stamp":"2024-08-03T20:47:33.895731+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T06:13:41.904985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-03T20:47:37.316755+0000","last_change":"2024-08-03T20:47:35.950421+0000","last_active":"2024-08-03T20:47:37.316755+0000","last_peered":"2024-08-03T20:47:37.316755+0000","last_clean":"2024-08-03T20:47:37.316755+0000","last_became_active":"2024-08-03T20:47:35.950109+0000","last_became_peered":"2024-08-03T20:47:35.950109+0000","last_unstale":"2024-08-03T20:47:37.316755+0000","last_undegraded":"2024-08-03T20:47:37.316755+0000","last_fullsized":"2024-08-03T20:47:37.316755+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_clean_scrub_stamp":"2024-08-03T20:47:34.904132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T02:17:30.747723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"20'1","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-03T20:47:36.936564+0000","last_change":"2024-08-03T20:47:34.935417+0000","last_active":"2024-08-03T20:47:36.936564+0000","last_peered":"2024-08-03T20:47:36.936564+0000","last_clean":"2024-08-03T20:47:36.936564+0000","last_became_active":"2024-08-03T20:47:34.932211+0000","last_became_peered":"2024-08-03T20:47:34.932211+0000","last_unstale":"2024-08-03T20:47:36.936564+0000","last_undegraded":"2024-08-03T20:47:36.936564+0000","last_fullsized":"2024-08-03T20:47:36.936564+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:47:33.895731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:47:33.895731+0000","last_clean_scrub_stamp":"2024-08-03T20:47:33.895731+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T04:43:47.155950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-03T20:47:37.317197+0000","last_change":"2024-08-03T20:47:35.958660+0000","last_active":"2024-08-03T20:47:37.317197+0000","last_peered":"2024-08-03T20:47:37.317197+0000","last_clean":"2024-08-03T20:47:37.317197+0000","last_became_active":"2024-08-03T20:47:35.958296+0000","last_became_peered":"2024-08-03T20:47:35.958296+0000","last_unstale":"2024-08-03T20:47:37.317197+0000","last_undegraded":"2024-08-03T20:47:37.317197+0000","last_fullsized":"2024-08-03T20:47:37.317197+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_clean_scrub_stamp":"2024-08-03T20:47:34.904132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T01:52:59.081172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"20'2","reported_seq":17,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-03T20:47:37.352928+0000","last_change":"2024-08-03T20:47:34.934762+0000","last_active":"2024-08-03T20:47:37.352928+0000","last_peered":"2024-08-03T20:47:37.352928+0000","last_clean":"2024-08-03T20:47:37.352928+0000","last_became_active":"2024-08-03T20:47:34.933194+0000","last_became_peered":"2024-08-03T20:47:34.933194+0000","last_unstale":"2024-08-03T20:47:37.352928+0000","last_undegraded":"2024-08-03T20:47:37.352928+0000","last_fullsized":"2024-08-03T20:47:37.352928+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:47:33.895731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:47:33.895731+0000","last_clean_scrub_stamp":"2024-08-03T20:47:33.895731+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T07:58:41.861301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-03T20:47:37.317479+0000","last_change":"2024-08-03T20:47:35.952102+0000","last_active":"2024-08-03T20:47:37.317479+0000","last_peered":"2024-08-03T20:47:37.317479+0000","last_clean":"2024-08-03T20:47:37.317479+0000","last_became_active":"2024-08-03T20:47:35.951350+0000","last_became_peered":"2024-08-03T20:47:35.951350+0000","last_unstale":"2024-08-03T20:47:37.317479+0000","last_undegraded":"2024-08-03T20:47:37.317479+0000","last_fullsized":"2024-08-03T20:47:37.317479+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_clean_scrub_stamp":"2024-08-03T20:47:34.904132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-04T23:37:39.394391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"20'1","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-03T20:47:36.935542+0000","last_change":"2024-08-03T20:47:34.932568+0000","last_active":"2024-08-03T20:47:36.935542+0000","last_peered":"2024-08-03T20:47:36.935542+0000","last_clean":"2024-08-03T20:47:36.935542+0000","last_became_active":"2024-08-03T20:47:34.932223+0000","last_became_peered":"2024-08-03T20:47:34.932223+0000","last_unstale":"2024-08-03T20:47:36.935542+0000","last_undegraded":"2024-08-03T20:47:36.935542+0000","last_fullsized":"2024-08-03T20:47:36.935542+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:47:33.895731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:47:33.895731+0000","last_clean_scrub_stamp":"2024-08-03T20:47:33.895731+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T05:01:48.922382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-03T20:47:37.979523+0000","last_change":"2024-08-03T20:47:35.957384+0000","last_active":"2024-08-03T20:47:37.979523+0000","last_peered":"2024-08-03T20:47:37.979523+0000","last_clean":"2024-08-03T20:47:37.979523+0000","last_became_active":"2024-08-03T20:47:35.956464+0000","last_became_peered":"2024-08-03T20:47:35.956464+0000","last_unstale":"2024-08-03T20:47:37.979523+0000","last_undegraded":"2024-08-03T20:47:37.979523+0000","last_fullsized":"2024-08-03T20:47:37.979523+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_clean_scrub_stamp":"2024-08-03T20:47:34.904132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T05:49:42.422083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-03T20:47:37.352724+0000","last_change":"2024-08-03T20:47:34.934154+0000","last_active":"2024-08-03T20:47:37.352724+0000","last_peered":"2024-08-03T20:47:37.352724+0000","last_clean":"2024-08-03T20:47:37.352724+0000","last_became_active":"2024-08-03T20:47:34.933257+0000","last_became_peered":"2024-08-03T20:47:34.933257+0000","last_unstale":"2024-08-03T20:47:37.352724+0000","last_undegraded":"2024-08-03T20:47:37.352724+0000","last_fullsized":"2024-08-03T20:47:37.352724+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:47:33.895731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:47:33.895731+0000","last_clean_scrub_stamp":"2024-08-03T20:47:33.895731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-04T21:10:14.859583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-03T20:47:37.317012+0000","last_change":"2024-08-03T20:47:35.957705+0000","last_active":"2024-08-03T20:47:37.317012+0000","last_peered":"2024-08-03T20:47:37.317012+0000","last_clean":"2024-08-03T20:47:37.317012+0000","last_became_active":"2024-08-03T20:47:35.951141+0000","last_became_peered":"2024-08-03T20:47:35.951141+0000","last_unstale":"2024-08-03T20:47:37.317012+0000","last_undegraded":"2024-08-03T20:47:37.317012+0000","last_fullsized":"2024-08-03T20:47:37.317012+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_clean_scrub_stamp":"2024-08-03T20:47:34.904132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T06:29:37.397869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"16'72","reported_seq":111,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-03T20:47:37.316600+0000","last_change":"2024-08-03T20:47:30.547632+0000","last_active":"2024-08-03T20:47:37.316600+0000","last_peered":"2024-08-03T20:47:37.316600+0000","last_clean":"2024-08-03T20:47:37.316600+0000","last_became_active":"2024-08-03T20:47:30.546255+0000","last_became_peered":"2024-08-03T20:47:30.546255+0000","last_unstale":"2024-08-03T20:47:37.316600+0000","last_undegraded":"2024-08-03T20:47:37.316600+0000","last_fullsized":"2024-08-03T20:47:37.316600+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:47:28.874451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:47:28.874451+0000","last_clean_scrub_stamp":"2024-08-03T20:47:28.874451+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T00:19:40.548812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-03T20:47:37.978786+0000","last_change":"2024-08-03T20:47:34.941906+0000","last_active":"2024-08-03T20:47:37.978786+0000","last_peered":"2024-08-03T20:47:37.978786+0000","last_clean":"2024-08-03T20:47:37.978786+0000","last_became_active":"2024-08-03T20:47:34.941492+0000","last_became_peered":"2024-08-03T20:47:34.941492+0000","last_unstale":"2024-08-03T20:47:37.978786+0000","last_undegraded":"2024-08-03T20:47:37.978786+0000","last_fullsized":"2024-08-03T20:47:37.978786+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:47:33.895731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:47:33.895731+0000","last_clean_scrub_stamp":"2024-08-03T20:47:33.895731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T06:32:14.208950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-03T20:47:37.978467+0000","last_change":"2024-08-03T20:47:35.953945+0000","last_active":"2024-08-03T20:47:37.978467+0000","last_peered":"2024-08-03T20:47:37.978467+0000","last_clean":"2024-08-03T20:47:37.978467+0000","last_became_active":"2024-08-03T20:47:35.953580+0000","last_became_peered":"2024-08-03T20:47:35.953580+0000","last_unstale":"2024-08-03T20:47:37.978467+0000","last_undegraded":"2024-08-03T20:47:37.978467+0000","last_fullsized":"2024-08-03T20:47:37.978467+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_clean_scrub_stamp":"2024-08-03T20:47:34.904132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T03:08:26.700491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-03T20:47:37.316444+0000","last_change":"2024-08-03T20:47:34.944839+0000","last_active":"2024-08-03T20:47:37.316444+0000","last_peered":"2024-08-03T20:47:37.316444+0000","last_clean":"2024-08-03T20:47:37.316444+0000","last_became_active":"2024-08-03T20:47:34.944530+0000","last_became_peered":"2024-08-03T20:47:34.944530+0000","last_unstale":"2024-08-03T20:47:37.316444+0000","last_undegraded":"2024-08-03T20:47:37.316444+0000","last_fullsized":"2024-08-03T20:47:37.316444+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:47:33.895731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:47:33.895731+0000","last_clean_scrub_stamp":"2024-08-03T20:47:33.895731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T02:40:00.976532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-03T20:47:37.978347+0000","last_change":"2024-08-03T20:47:35.926904+0000","last_active":"2024-08-03T20:47:37.978347+0000","last_peered":"2024-08-03T20:47:37.978347+0000","last_clean":"2024-08-03T20:47:37.978347+0000","last_became_active":"2024-08-03T20:47:35.926525+0000","last_became_peered":"2024-08-03T20:47:35.926525+0000","last_unstale":"2024-08-03T20:47:37.978347+0000","last_undegraded":"2024-08-03T20:47:37.978347+0000","last_fullsized":"2024-08-03T20:47:37.978347+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_clean_scrub_stamp":"2024-08-03T20:47:34.904132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T05:57:44.170587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-03T20:47:37.352725+0000","last_change":"2024-08-03T20:47:34.939125+0000","last_active":"2024-08-03T20:47:37.352725+0000","last_peered":"2024-08-03T20:47:37.352725+0000","last_clean":"2024-08-03T20:47:37.352725+0000","last_became_active":"2024-08-03T20:47:34.935161+0000","last_became_peered":"2024-08-03T20:47:34.935161+0000","last_unstale":"2024-08-03T20:47:37.352725+0000","last_undegraded":"2024-08-03T20:47:37.352725+0000","last_fullsized":"2024-08-03T20:47:37.352725+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:47:33.895731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:47:33.895731+0000","last_clean_scrub_stamp":"2024-08-03T20:47:33.895731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T04:37:55.336635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-03T20:47:36.934972+0000","last_change":"2024-08-03T20:47:35.933618+0000","last_active":"2024-08-03T20:47:36.934972+0000","last_peered":"2024-08-03T20:47:36.934972+0000","last_clean":"2024-08-03T20:47:36.934972+0000","last_became_active":"2024-08-03T20:47:35.932873+0000","last_became_peered":"2024-08-03T20:47:35.932873+0000","last_unstale":"2024-08-03T20:47:36.934972+0000","last_undegraded":"2024-08-03T20:47:36.934972+0000","last_fullsized":"2024-08-03T20:47:36.934972+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_clean_scrub_stamp":"2024-08-03T20:47:34.904132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-04T23:57:18.840519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-03T20:47:37.979331+0000","last_change":"2024-08-03T20:47:34.940602+0000","last_active":"2024-08-03T20:47:37.979331+0000","last_peered":"2024-08-03T20:47:37.979331+0000","last_clean":"2024-08-03T20:47:37.979331+0000","last_became_active":"2024-08-03T20:47:34.940156+0000","last_became_peered":"2024-08-03T20:47:34.940156+0000","last_unstale":"2024-08-03T20:47:37.979331+0000","last_undegraded":"2024-08-03T20:47:37.979331+0000","last_fullsized":"2024-08-03T20:47:37.979331+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:47:33.895731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:47:33.895731+0000","last_clean_scrub_stamp":"2024-08-03T20:47:33.895731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T06:44:37.944928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-03T20:47:36.935959+0000","last_change":"2024-08-03T20:47:35.938536+0000","last_active":"2024-08-03T20:47:36.935959+0000","last_peered":"2024-08-03T20:47:36.935959+0000","last_clean":"2024-08-03T20:47:36.935959+0000","last_became_active":"2024-08-03T20:47:35.938281+0000","last_became_peered":"2024-08-03T20:47:35.938281+0000","last_unstale":"2024-08-03T20:47:36.935959+0000","last_undegraded":"2024-08-03T20:47:36.935959+0000","last_fullsized":"2024-08-03T20:47:36.935959+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_clean_scrub_stamp":"2024-08-03T20:47:34.904132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T06:17:22.118393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-03T20:47:36.935901+0000","last_change":"2024-08-03T20:47:34.936859+0000","last_active":"2024-08-03T20:47:36.935901+0000","last_peered":"2024-08-03T20:47:36.935901+0000","last_clean":"2024-08-03T20:47:36.935901+0000","last_became_active":"2024-08-03T20:47:34.936017+0000","last_became_peered":"2024-08-03T20:47:34.936017+0000","last_unstale":"2024-08-03T20:47:36.935901+0000","last_undegraded":"2024-08-03T20:47:36.935901+0000","last_fullsized":"2024-08-03T20:47:36.935901+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:47:33.895731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:47:33.895731+0000","last_clean_scrub_stamp":"2024-08-03T20:47:33.895731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T02:40:00.976532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-03T20:47:37.979373+0000","last_change":"2024-08-03T20:47:35.940325+0000","last_active":"2024-08-03T20:47:37.979373+0000","last_peered":"2024-08-03T20:47:37.979373+0000","last_clean":"2024-08-03T20:47:37.979373+0000","last_became_active":"2024-08-03T20:47:35.939888+0000","last_became_peered":"2024-08-03T20:47:35.939888+0000","last_unstale":"2024-08-03T20:47:37.979373+0000","last_undegraded":"2024-08-03T20:47:37.979373+0000","last_fullsized":"2024-08-03T20:47:37.979373+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_clean_scrub_stamp":"2024-08-03T20:47:34.904132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T02:17:30.747723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-03T20:47:37.317158+0000","last_change":"2024-08-03T20:47:34.946657+0000","last_active":"2024-08-03T20:47:37.317158+0000","last_peered":"2024-08-03T20:47:37.317158+0000","last_clean":"2024-08-03T20:47:37.317158+0000","last_became_active":"2024-08-03T20:47:34.945719+0000","last_became_peered":"2024-08-03T20:47:34.945719+0000","last_unstale":"2024-08-03T20:47:37.317158+0000","last_undegraded":"2024-08-03T20:47:37.317158+0000","last_fullsized":"2024-08-03T20:47:37.317158+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:47:33.895731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:47:33.895731+0000","last_clean_scrub_stamp":"2024-08-03T20:47:33.895731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-04T21:30:40.295081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-03T20:47:36.935812+0000","last_change":"2024-08-03T20:47:35.936144+0000","last_active":"2024-08-03T20:47:36.935812+0000","last_peered":"2024-08-03T20:47:36.935812+0000","last_clean":"2024-08-03T20:47:36.935812+0000","last_became_active":"2024-08-03T20:47:35.935842+0000","last_became_peered":"2024-08-03T20:47:35.935842+0000","last_unstale":"2024-08-03T20:47:36.935812+0000","last_undegraded":"2024-08-03T20:47:36.935812+0000","last_fullsized":"2024-08-03T20:47:36.935812+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_clean_scrub_stamp":"2024-08-03T20:47:34.904132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-04T22:33:13.028920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"21'1","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-03T20:47:36.935790+0000","last_change":"2024-08-03T20:47:34.934773+0000","last_active":"2024-08-03T20:47:36.935790+0000","last_peered":"2024-08-03T20:47:36.935790+0000","last_clean":"2024-08-03T20:47:36.935790+0000","last_became_active":"2024-08-03T20:47:34.934473+0000","last_became_peered":"2024-08-03T20:47:34.934473+0000","last_unstale":"2024-08-03T20:47:36.935790+0000","last_undegraded":"2024-08-03T20:47:36.935790+0000","last_fullsized":"2024-08-03T20:47:36.935790+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:47:33.895731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:47:33.895731+0000","last_clean_scrub_stamp":"2024-08-03T20:47:33.895731+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-04T22:30:28.119577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-03T20:47:37.979065+0000","last_change":"2024-08-03T20:47:35.933622+0000","last_active":"2024-08-03T20:47:37.979065+0000","last_peered":"2024-08-03T20:47:37.979065+0000","last_clean":"2024-08-03T20:47:37.979065+0000","last_became_active":"2024-08-03T20:47:35.933133+0000","last_became_peered":"2024-08-03T20:47:35.933133+0000","last_unstale":"2024-08-03T20:47:37.979065+0000","last_undegraded":"2024-08-03T20:47:37.979065+0000","last_fullsized":"2024-08-03T20:47:37.979065+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_clean_scrub_stamp":"2024-08-03T20:47:34.904132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-04T22:43:36.335718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"20'2","reported_seq":17,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-03T20:47:36.941891+0000","last_change":"2024-08-03T20:47:34.944805+0000","last_active":"2024-08-03T20:47:36.941891+0000","last_peered":"2024-08-03T20:47:36.941891+0000","last_clean":"2024-08-03T20:47:36.941891+0000","last_became_active":"2024-08-03T20:47:34.943207+0000","last_became_peered":"2024-08-03T20:47:34.943207+0000","last_unstale":"2024-08-03T20:47:36.941891+0000","last_undegraded":"2024-08-03T20:47:36.941891+0000","last_fullsized":"2024-08-03T20:47:36.941891+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:47:33.895731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:47:33.895731+0000","last_clean_scrub_stamp":"2024-08-03T20:47:33.895731+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T07:46:07.162679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-03T20:47:37.979614+0000","last_change":"2024-08-03T20:47:35.941163+0000","last_active":"2024-08-03T20:47:37.979614+0000","last_peered":"2024-08-03T20:47:37.979614+0000","last_clean":"2024-08-03T20:47:37.979614+0000","last_became_active":"2024-08-03T20:47:35.940955+0000","last_became_peered":"2024-08-03T20:47:35.940955+0000","last_unstale":"2024-08-03T20:47:37.979614+0000","last_undegraded":"2024-08-03T20:47:37.979614+0000","last_fullsized":"2024-08-03T20:47:37.979614+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_clean_scrub_stamp":"2024-08-03T20:47:34.904132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T01:08:56.085650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-03T20:47:37.316412+0000","last_change":"2024-08-03T20:47:34.943492+0000","last_active":"2024-08-03T20:47:37.316412+0000","last_peered":"2024-08-03T20:47:37.316412+0000","last_clean":"2024-08-03T20:47:37.316412+0000","last_became_active":"2024-08-03T20:47:34.943121+0000","last_became_peered":"2024-08-03T20:47:34.943121+0000","last_unstale":"2024-08-03T20:47:37.316412+0000","last_undegraded":"2024-08-03T20:47:37.316412+0000","last_fullsized":"2024-08-03T20:47:37.316412+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:47:33.895731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:47:33.895731+0000","last_clean_scrub_stamp":"2024-08-03T20:47:33.895731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T07:46:07.162679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-03T20:47:36.935830+0000","last_change":"2024-08-03T20:47:35.924518+0000","last_active":"2024-08-03T20:47:36.935830+0000","last_peered":"2024-08-03T20:47:36.935830+0000","last_clean":"2024-08-03T20:47:36.935830+0000","last_became_active":"2024-08-03T20:47:35.923938+0000","last_became_peered":"2024-08-03T20:47:35.923938+0000","last_unstale":"2024-08-03T20:47:36.935830+0000","last_undegraded":"2024-08-03T20:47:36.935830+0000","last_fullsized":"2024-08-03T20:47:36.935830+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_clean_scrub_stamp":"2024-08-03T20:47:34.904132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T04:26:33.175697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-03T20:47:36.935369+0000","last_change":"2024-08-03T20:47:34.941712+0000","last_active":"2024-08-03T20:47:36.935369+0000","last_peered":"2024-08-03T20:47:36.935369+0000","last_clean":"2024-08-03T20:47:36.935369+0000","last_became_active":"2024-08-03T20:47:34.941138+0000","last_became_peered":"2024-08-03T20:47:34.941138+0000","last_unstale":"2024-08-03T20:47:36.935369+0000","last_undegraded":"2024-08-03T20:47:36.935369+0000","last_fullsized":"2024-08-03T20:47:36.935369+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:47:33.895731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:47:33.895731+0000","last_clean_scrub_stamp":"2024-08-03T20:47:33.895731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T07:46:07.162679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-03T20:47:37.978318+0000","last_change":"2024-08-03T20:47:35.941992+0000","last_active":"2024-08-03T20:47:37.978318+0000","last_peered":"2024-08-03T20:47:37.978318+0000","last_clean":"2024-08-03T20:47:37.978318+0000","last_became_active":"2024-08-03T20:47:35.941633+0000","last_became_peered":"2024-08-03T20:47:35.941633+0000","last_unstale":"2024-08-03T20:47:37.978318+0000","last_undegraded":"2024-08-03T20:47:37.978318+0000","last_fullsized":"2024-08-03T20:47:37.978318+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_clean_scrub_stamp":"2024-08-03T20:47:34.904132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T03:06:49.510875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-03T20:47:36.935863+0000","last_change":"2024-08-03T20:47:34.932831+0000","last_active":"2024-08-03T20:47:36.935863+0000","last_peered":"2024-08-03T20:47:36.935863+0000","last_clean":"2024-08-03T20:47:36.935863+0000","last_became_active":"2024-08-03T20:47:34.932520+0000","last_became_peered":"2024-08-03T20:47:34.932520+0000","last_unstale":"2024-08-03T20:47:36.935863+0000","last_undegraded":"2024-08-03T20:47:36.935863+0000","last_fullsized":"2024-08-03T20:47:36.935863+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:47:33.895731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:47:33.895731+0000","last_clean_scrub_stamp":"2024-08-03T20:47:33.895731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T00:43:30.770268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-03T20:47:37.978692+0000","last_change":"2024-08-03T20:47:35.942701+0000","last_active":"2024-08-03T20:47:37.978692+0000","last_peered":"2024-08-03T20:47:37.978692+0000","last_clean":"2024-08-03T20:47:37.978692+0000","last_became_active":"2024-08-03T20:47:35.942261+0000","last_became_peered":"2024-08-03T20:47:35.942261+0000","last_unstale":"2024-08-03T20:47:37.978692+0000","last_undegraded":"2024-08-03T20:47:37.978692+0000","last_fullsized":"2024-08-03T20:47:37.978692+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_clean_scrub_stamp":"2024-08-03T20:47:34.904132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-04T22:01:37.009450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"20'2","reported_seq":17,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-03T20:47:37.317111+0000","last_change":"2024-08-03T20:47:34.947233+0000","last_active":"2024-08-03T20:47:37.317111+0000","last_peered":"2024-08-03T20:47:37.317111+0000","last_clean":"2024-08-03T20:47:37.317111+0000","last_became_active":"2024-08-03T20:47:34.946866+0000","last_became_peered":"2024-08-03T20:47:34.946866+0000","last_unstale":"2024-08-03T20:47:37.317111+0000","last_undegraded":"2024-08-03T20:47:37.317111+0000","last_fullsized":"2024-08-03T20:47:37.317111+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:47:33.895731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:47:33.895731+0000","last_clean_scrub_stamp":"2024-08-03T20:47:33.895731+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T03:46:04.675513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-03T20:47:36.941158+0000","last_change":"2024-08-03T20:47:35.936142+0000","last_active":"2024-08-03T20:47:36.941158+0000","last_peered":"2024-08-03T20:47:36.941158+0000","last_clean":"2024-08-03T20:47:36.941158+0000","last_became_active":"2024-08-03T20:47:35.935764+0000","last_became_peered":"2024-08-03T20:47:35.935764+0000","last_unstale":"2024-08-03T20:47:36.941158+0000","last_undegraded":"2024-08-03T20:47:36.941158+0000","last_fullsized":"2024-08-03T20:47:36.941158+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_clean_scrub_stamp":"2024-08-03T20:47:34.904132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-04T22:12:15.293757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-03T20:47:37.979240+0000","last_change":"2024-08-03T20:47:35.934358+0000","last_active":"2024-08-03T20:47:37.979240+0000","last_peered":"2024-08-03T20:47:37.979240+0000","last_clean":"2024-08-03T20:47:37.979240+0000","last_became_active":"2024-08-03T20:47:35.933874+0000","last_became_peered":"2024-08-03T20:47:35.933874+0000","last_unstale":"2024-08-03T20:47:37.979240+0000","last_undegraded":"2024-08-03T20:47:37.979240+0000","last_fullsized":"2024-08-03T20:47:37.979240+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_clean_scrub_stamp":"2024-08-03T20:47:34.904132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T00:10:43.343156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"20'2","reported_seq":17,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-03T20:47:37.316590+0000","last_change":"2024-08-03T20:47:34.950578+0000","last_active":"2024-08-03T20:47:37.316590+0000","last_peered":"2024-08-03T20:47:37.316590+0000","last_clean":"2024-08-03T20:47:37.316590+0000","last_became_active":"2024-08-03T20:47:34.950187+0000","last_became_peered":"2024-08-03T20:47:34.950187+0000","last_unstale":"2024-08-03T20:47:37.316590+0000","last_undegraded":"2024-08-03T20:47:37.316590+0000","last_fullsized":"2024-08-03T20:47:37.316590+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:47:33.895731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:47:33.895731+0000","last_clean_scrub_stamp":"2024-08-03T20:47:33.895731+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T00:43:30.770268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-03T20:47:37.979438+0000","last_change":"2024-08-03T20:47:35.934586+0000","last_active":"2024-08-03T20:47:37.979438+0000","last_peered":"2024-08-03T20:47:37.979438+0000","last_clean":"2024-08-03T20:47:37.979438+0000","last_became_active":"2024-08-03T20:47:35.934289+0000","last_became_peered":"2024-08-03T20:47:35.934289+0000","last_unstale":"2024-08-03T20:47:37.979438+0000","last_undegraded":"2024-08-03T20:47:37.979438+0000","last_fullsized":"2024-08-03T20:47:37.979438+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_clean_scrub_stamp":"2024-08-03T20:47:34.904132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T04:51:27.371858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"20'2","reported_seq":17,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-03T20:47:37.316998+0000","last_change":"2024-08-03T20:47:34.946365+0000","last_active":"2024-08-03T20:47:37.316998+0000","last_peered":"2024-08-03T20:47:37.316998+0000","last_clean":"2024-08-03T20:47:37.316998+0000","last_became_active":"2024-08-03T20:47:34.945922+0000","last_became_peered":"2024-08-03T20:47:34.945922+0000","last_unstale":"2024-08-03T20:47:37.316998+0000","last_undegraded":"2024-08-03T20:47:37.316998+0000","last_fullsized":"2024-08-03T20:47:37.316998+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:47:33.895731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:47:33.895731+0000","last_clean_scrub_stamp":"2024-08-03T20:47:33.895731+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T06:59:03.089784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-03T20:47:37.979781+0000","last_change":"2024-08-03T20:47:35.941147+0000","last_active":"2024-08-03T20:47:37.979781+0000","last_peered":"2024-08-03T20:47:37.979781+0000","last_clean":"2024-08-03T20:47:37.979781+0000","last_became_active":"2024-08-03T20:47:35.940605+0000","last_became_peered":"2024-08-03T20:47:35.940605+0000","last_unstale":"2024-08-03T20:47:37.979781+0000","last_undegraded":"2024-08-03T20:47:37.979781+0000","last_fullsized":"2024-08-03T20:47:37.979781+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_clean_scrub_stamp":"2024-08-03T20:47:34.904132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-04T22:33:13.028920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"20'1","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-03T20:47:37.978906+0000","last_change":"2024-08-03T20:47:34.934888+0000","last_active":"2024-08-03T20:47:37.978906+0000","last_peered":"2024-08-03T20:47:37.978906+0000","last_clean":"2024-08-03T20:47:37.978906+0000","last_became_active":"2024-08-03T20:47:34.934595+0000","last_became_peered":"2024-08-03T20:47:34.934595+0000","last_unstale":"2024-08-03T20:47:37.978906+0000","last_undegraded":"2024-08-03T20:47:37.978906+0000","last_fullsized":"2024-08-03T20:47:37.978906+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:47:33.895731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:47:33.895731+0000","last_clean_scrub_stamp":"2024-08-03T20:47:33.895731+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T06:13:41.904985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-03T20:47:37.978652+0000","last_change":"2024-08-03T20:47:35.953798+0000","last_active":"2024-08-03T20:47:37.978652+0000","last_peered":"2024-08-03T20:47:37.978652+0000","last_clean":"2024-08-03T20:47:37.978652+0000","last_became_active":"2024-08-03T20:47:35.950186+0000","last_became_peered":"2024-08-03T20:47:35.950186+0000","last_unstale":"2024-08-03T20:47:37.978652+0000","last_undegraded":"2024-08-03T20:47:37.978652+0000","last_fullsized":"2024-08-03T20:47:37.978652+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_clean_scrub_stamp":"2024-08-03T20:47:34.904132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T07:58:55.643722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"20'4","reported_seq":19,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-03T20:47:37.979858+0000","last_change":"2024-08-03T20:47:34.939409+0000","last_active":"2024-08-03T20:47:37.979858+0000","last_peered":"2024-08-03T20:47:37.979858+0000","last_clean":"2024-08-03T20:47:37.979858+0000","last_became_active":"2024-08-03T20:47:34.939057+0000","last_became_peered":"2024-08-03T20:47:34.939057+0000","last_unstale":"2024-08-03T20:47:37.979858+0000","last_undegraded":"2024-08-03T20:47:37.979858+0000","last_fullsized":"2024-08-03T20:47:37.979858+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:47:33.895731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:47:33.895731+0000","last_clean_scrub_stamp":"2024-08-03T20:47:33.895731+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T05:49:05.605066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509444,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31972,"kb_used_data":788,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339868,"statfs":{"total":96636764160,"available":96604024832,"internally_reserved":0,"allocated":806912,"data_stored":663491,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":6,"up_from":15,"seq":64424509445,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31396,"kb_used_data":216,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340444,"statfs":{"total":96636764160,"available":96604614656,"internally_reserved":0,"allocated":221184,"data_stored":74153,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":5,"up_from":15,"seq":64424509445,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31388,"kb_used_data":208,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340452,"statfs":{"total":96636764160,"available":96604622848,"internally_reserved":0,"allocated":212992,"data_stored":72631,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509444,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31392,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340448,"statfs":{"total":96636764160,"available":96604618752,"internally_reserved":0,"allocated":217088,"data_stored":72701,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509446,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31980,"kb_used_data":796,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339860,"statfs":{"total":96636764160,"available":96604016640,"internally_reserved":0,"allocated":815104,"data_stored":663547,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542149,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31400,"kb_used_data":216,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340440,"statfs":{"total":96636764160,"available":96604610560,"internally_reserved":0,"allocated":221184,"data_stored":72667,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":3,"up_from":14,"seq":60129542147,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31400,"kb_used_data":216,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340440,"statfs":{"total":96636764160,"available":96604610560,"internally_reserved":0,"allocated":221184,"data_stored":74153,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":14,"seq":60129542148,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31388,"kb_used_data":204,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340452,"statfs":{"total":96636764160,"available":96604622848,"internally_reserved":0,"allocated":208896,"data_stored":72541,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-08-03T20:47:40.443 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-08-03T20:47:40.706 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:40.704+0000 7f75f6400640 1 -- 172.21.15.32:0/2238630508 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f75f00708c0 msgr2=0x7f75f00b4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:47:40.706 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:40.704+0000 7f75f6400640 1 --2- 172.21.15.32:0/2238630508 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f75f00708c0 0x7f75f00b4de0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f75f00b3080 tx=0x7f75e002a4b0 comp rx=0 tx=0).stop 2024-08-03T20:47:40.707 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:40.704+0000 7f75f6400640 1 -- 172.21.15.32:0/2238630508 shutdown_connections 2024-08-03T20:47:40.707 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:40.704+0000 7f75f6400640 1 --2- 172.21.15.32:0/2238630508 >> v2:172.21.15.118:3300/0 conn(0x7f75f01515c0 0x7f75f015e170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:40.707 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:40.704+0000 7f75f6400640 1 --2- 172.21.15.32:0/2238630508 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f75f00708c0 0x7f75f00b4de0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:40.707 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:40.704+0000 7f75f6400640 1 -- 172.21.15.32:0/2238630508 >> 172.21.15.32:0/2238630508 conn(0x7f75f00b1bd0 msgr2=0x7f75f00b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:47:40.707 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:40.704+0000 7f75f6400640 1 -- 172.21.15.32:0/2238630508 shutdown_connections 2024-08-03T20:47:40.707 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:40.705+0000 7f75f6400640 1 -- 172.21.15.32:0/2238630508 wait complete. 2024-08-03T20:47:40.708 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:40.707+0000 7f75f6400640 1 Processor -- start 2024-08-03T20:47:40.708 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:40.707+0000 7f75f6400640 1 -- start start 2024-08-03T20:47:40.711 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:40.708+0000 7f75f6400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f75f00708c0 0x7f75f014d580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:47:40.711 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:40.708+0000 7f75f6400640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f75f01515c0 0x7f75f00c5220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:47:40.711 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:40.708+0000 7f75f6400640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f75f015e6b0 0x7f75f00c5760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:47:40.711 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:40.708+0000 7f75f6400640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f75f006f1c0 con 0x7f75f00708c0 2024-08-03T20:47:40.712 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:40.708+0000 7f75f6400640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f75f006f040 con 0x7f75f015e6b0 2024-08-03T20:47:40.712 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:40.708+0000 7f75f6400640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7f75f006f340 con 0x7f75f01515c0 2024-08-03T20:47:40.712 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:40.709+0000 7f75ef400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f75f00708c0 0x7f75f014d580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:47:40.712 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:40.709+0000 7f75ef400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f75f00708c0 0x7f75f014d580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:57246/0 (socket says 172.21.15.32:57246) 2024-08-03T20:47:40.712 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:40.709+0000 7f75ef400640 1 -- 172.21.15.32:0/13217643 learned_addr learned my addr 172.21.15.32:0/13217643 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:47:40.712 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:40.709+0000 7f75efe00640 1 --2- 172.21.15.32:0/13217643 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f75f015e6b0 0x7f75f00c5760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:47:40.712 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:40.709+0000 7f75eea00640 1 --2- 172.21.15.32:0/13217643 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f75f01515c0 0x7f75f00c5220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:47:40.712 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:40.709+0000 7f75ef400640 1 -- 172.21.15.32:0/13217643 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f75f01515c0 msgr2=0x7f75f00c5220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:47:40.712 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:40.709+0000 7f75ef400640 1 --2- 172.21.15.32:0/13217643 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f75f01515c0 0x7f75f00c5220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:40.712 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:40.709+0000 7f75ef400640 1 -- 172.21.15.32:0/13217643 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f75f015e6b0 msgr2=0x7f75f00c5760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:47:40.712 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:40.709+0000 7f75ef400640 1 --2- 172.21.15.32:0/13217643 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f75f015e6b0 0x7f75f00c5760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:40.712 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:40.709+0000 7f75ef400640 1 -- 172.21.15.32:0/13217643 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f75f014e030 con 0x7f75f00708c0 2024-08-03T20:47:40.713 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:40.709+0000 7f75eea00640 1 --2- 172.21.15.32:0/13217643 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f75f01515c0 0x7f75f00c5220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-03T20:47:40.713 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:40.709+0000 7f75ef400640 1 --2- 172.21.15.32:0/13217643 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f75f00708c0 0x7f75f014d580 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f75e0009780 tx=0x7f75e0033c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:47:40.713 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:40.710+0000 7f75e7400640 1 -- 172.21.15.32:0/13217643 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75e0031d60 con 0x7f75f00708c0 2024-08-03T20:47:40.713 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:40.710+0000 7f75e7400640 1 -- 172.21.15.32:0/13217643 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f75e0033e10 con 0x7f75f00708c0 2024-08-03T20:47:40.713 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:40.710+0000 7f75e7400640 1 -- 172.21.15.32:0/13217643 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75e0005260 con 0x7f75f00708c0 2024-08-03T20:47:40.713 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:40.710+0000 7f75f6400640 1 -- 172.21.15.32:0/13217643 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f75f015a590 con 0x7f75f00708c0 2024-08-03T20:47:40.713 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:40.710+0000 7f75f6400640 1 -- 172.21.15.32:0/13217643 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f75f00d0090 con 0x7f75f00708c0 2024-08-03T20:47:40.713 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:40.711+0000 7f75f6400640 1 -- 172.21.15.32:0/13217643 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f75f014e200 con 0x7f75f00708c0 2024-08-03T20:47:40.714 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:40.712+0000 7f75e7400640 1 -- 172.21.15.32:0/13217643 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f75e0005400 con 0x7f75f00708c0 2024-08-03T20:47:40.714 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:40.713+0000 7f75e7400640 1 --2- 172.21.15.32:0/13217643 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f75d0077010 0x7f75d0079520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:47:40.715 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:40.713+0000 7f75e7400640 1 -- 172.21.15.32:0/13217643 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f75e00b9a90 con 0x7f75f00708c0 2024-08-03T20:47:40.715 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:40.714+0000 7f75eea00640 1 --2- 172.21.15.32:0/13217643 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f75d0077010 0x7f75d0079520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:47:40.715 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:40.714+0000 7f75eea00640 1 --2- 172.21.15.32:0/13217643 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f75d0077010 0x7f75d0079520 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f75f00b0bc0 tx=0x7f75dc005f10 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:47:40.719 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:40.718+0000 7f75e7400640 1 -- 172.21.15.32:0/13217643 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7f75e003f030 con 0x7f75f00708c0 2024-08-03T20:47:40.917 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:40.916+0000 7f75f6400640 1 -- 172.21.15.32:0/13217643 --> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f75f00cea80 con 0x7f75d0077010 2024-08-03T20:47:40.922 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:40.921+0000 7f75e7400640 1 -- 172.21.15.32:0/13217643 <== mgr.4106 v2:172.21.15.118:6800/4205950699 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+251936 (secure 0 0 0) 0x7f75dc00bd20 con 0x7f75d0077010 2024-08-03T20:47:40.926 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-03T20:47:40.927 INFO:teuthology.orchestra.run.smithi032.stderr:dumped all 2024-08-03T20:47:40.928 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:40.925+0000 7f75f6400640 1 -- 172.21.15.32:0/13217643 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f75d0077010 msgr2=0x7f75d0079520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:47:40.928 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:40.925+0000 7f75f6400640 1 --2- 172.21.15.32:0/13217643 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f75d0077010 0x7f75d0079520 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f75f00b0bc0 tx=0x7f75dc005f10 comp rx=0 tx=0).stop 2024-08-03T20:47:40.929 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:40.925+0000 7f75f6400640 1 -- 172.21.15.32:0/13217643 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f75f00708c0 msgr2=0x7f75f014d580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:47:40.929 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:40.925+0000 7f75f6400640 1 --2- 172.21.15.32:0/13217643 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f75f00708c0 0x7f75f014d580 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f75e0009780 tx=0x7f75e0033c20 comp rx=0 tx=0).stop 2024-08-03T20:47:40.929 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:40.925+0000 7f75f6400640 1 -- 172.21.15.32:0/13217643 shutdown_connections 2024-08-03T20:47:40.929 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:40.925+0000 7f75f6400640 1 --2- 172.21.15.32:0/13217643 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f75d0077010 0x7f75d0079520 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:40.929 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:40.925+0000 7f75f6400640 1 --2- 172.21.15.32:0/13217643 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f75f015e6b0 0x7f75f00c5760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:40.929 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:40.925+0000 7f75f6400640 1 --2- 172.21.15.32:0/13217643 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f75f01515c0 0x7f75f00c5220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:40.929 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:40.925+0000 7f75f6400640 1 --2- 172.21.15.32:0/13217643 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f75f00708c0 0x7f75f014d580 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:40.929 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:40.925+0000 7f75f6400640 1 -- 172.21.15.32:0/13217643 >> 172.21.15.32:0/13217643 conn(0x7f75f00b1bd0 msgr2=0x7f75f0150c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:47:40.929 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:40.925+0000 7f75f6400640 1 -- 172.21.15.32:0/13217643 shutdown_connections 2024-08-03T20:47:40.929 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:40.925+0000 7f75f6400640 1 -- 172.21.15.32:0/13217643 wait complete. 2024-08-03T20:47:40.948 INFO:teuthology.orchestra.run.smithi032.stdout:{"pg_ready":true,"pg_map":{"version":20,"stamp":"2024-08-03T20:47:40.222169+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":146,"num_write_kb":1642,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":111,"ondisk_log_size":111,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":168,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":252316,"kb_used_data":2856,"kb_used_omap":12,"kb_used_meta":249331,"kb_avail":754722404,"statfs":{"total":773094113280,"available":772835741696,"internally_reserved":0,"allocated":2924544,"data_stored":1765884,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":255315505},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":28,"apply_latency_ms":28,"commit_latency_ns":28000000,"apply_latency_ns":28000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.310435"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-03T20:47:37.317266+0000","last_change":"2024-08-03T20:47:35.962230+0000","last_active":"2024-08-03T20:47:37.317266+0000","last_peered":"2024-08-03T20:47:37.317266+0000","last_clean":"2024-08-03T20:47:37.317266+0000","last_became_active":"2024-08-03T20:47:35.958864+0000","last_became_peered":"2024-08-03T20:47:35.958864+0000","last_unstale":"2024-08-03T20:47:37.317266+0000","last_undegraded":"2024-08-03T20:47:37.317266+0000","last_fullsized":"2024-08-03T20:47:37.317266+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_clean_scrub_stamp":"2024-08-03T20:47:34.904132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T03:01:41.602990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-03T20:47:36.935685+0000","last_change":"2024-08-03T20:47:35.941982+0000","last_active":"2024-08-03T20:47:36.935685+0000","last_peered":"2024-08-03T20:47:36.935685+0000","last_clean":"2024-08-03T20:47:36.935685+0000","last_became_active":"2024-08-03T20:47:35.941614+0000","last_became_peered":"2024-08-03T20:47:35.941614+0000","last_unstale":"2024-08-03T20:47:36.935685+0000","last_undegraded":"2024-08-03T20:47:36.935685+0000","last_fullsized":"2024-08-03T20:47:36.935685+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_clean_scrub_stamp":"2024-08-03T20:47:34.904132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T01:08:56.085650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-03T20:47:37.979574+0000","last_change":"2024-08-03T20:47:35.926803+0000","last_active":"2024-08-03T20:47:37.979574+0000","last_peered":"2024-08-03T20:47:37.979574+0000","last_clean":"2024-08-03T20:47:37.979574+0000","last_became_active":"2024-08-03T20:47:35.926198+0000","last_became_peered":"2024-08-03T20:47:35.926198+0000","last_unstale":"2024-08-03T20:47:37.979574+0000","last_undegraded":"2024-08-03T20:47:37.979574+0000","last_fullsized":"2024-08-03T20:47:37.979574+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_clean_scrub_stamp":"2024-08-03T20:47:34.904132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T04:57:58.179012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-03T20:47:37.979134+0000","last_change":"2024-08-03T20:47:35.960607+0000","last_active":"2024-08-03T20:47:37.979134+0000","last_peered":"2024-08-03T20:47:37.979134+0000","last_clean":"2024-08-03T20:47:37.979134+0000","last_became_active":"2024-08-03T20:47:35.956844+0000","last_became_peered":"2024-08-03T20:47:35.956844+0000","last_unstale":"2024-08-03T20:47:37.979134+0000","last_undegraded":"2024-08-03T20:47:37.979134+0000","last_fullsized":"2024-08-03T20:47:37.979134+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_clean_scrub_stamp":"2024-08-03T20:47:34.904132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T07:48:20.679795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-03T20:47:37.316984+0000","last_change":"2024-08-03T20:47:35.962693+0000","last_active":"2024-08-03T20:47:37.316984+0000","last_peered":"2024-08-03T20:47:37.316984+0000","last_clean":"2024-08-03T20:47:37.316984+0000","last_became_active":"2024-08-03T20:47:35.959911+0000","last_became_peered":"2024-08-03T20:47:35.959911+0000","last_unstale":"2024-08-03T20:47:37.316984+0000","last_undegraded":"2024-08-03T20:47:37.316984+0000","last_fullsized":"2024-08-03T20:47:37.316984+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_clean_scrub_stamp":"2024-08-03T20:47:34.904132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T04:51:27.371858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-03T20:47:37.979016+0000","last_change":"2024-08-03T20:47:35.947517+0000","last_active":"2024-08-03T20:47:37.979016+0000","last_peered":"2024-08-03T20:47:37.979016+0000","last_clean":"2024-08-03T20:47:37.979016+0000","last_became_active":"2024-08-03T20:47:35.947167+0000","last_became_peered":"2024-08-03T20:47:35.947167+0000","last_unstale":"2024-08-03T20:47:37.979016+0000","last_undegraded":"2024-08-03T20:47:37.979016+0000","last_fullsized":"2024-08-03T20:47:37.979016+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_clean_scrub_stamp":"2024-08-03T20:47:34.904132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-04T22:33:13.028920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-03T20:47:37.316673+0000","last_change":"2024-08-03T20:47:35.947270+0000","last_active":"2024-08-03T20:47:37.316673+0000","last_peered":"2024-08-03T20:47:37.316673+0000","last_clean":"2024-08-03T20:47:37.316673+0000","last_became_active":"2024-08-03T20:47:35.946647+0000","last_became_peered":"2024-08-03T20:47:35.946647+0000","last_unstale":"2024-08-03T20:47:37.316673+0000","last_undegraded":"2024-08-03T20:47:37.316673+0000","last_fullsized":"2024-08-03T20:47:37.316673+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_clean_scrub_stamp":"2024-08-03T20:47:34.904132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T00:10:43.343156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-03T20:47:37.978113+0000","last_change":"2024-08-03T20:47:35.946701+0000","last_active":"2024-08-03T20:47:37.978113+0000","last_peered":"2024-08-03T20:47:37.978113+0000","last_clean":"2024-08-03T20:47:37.978113+0000","last_became_active":"2024-08-03T20:47:35.946340+0000","last_became_peered":"2024-08-03T20:47:35.946340+0000","last_unstale":"2024-08-03T20:47:37.978113+0000","last_undegraded":"2024-08-03T20:47:37.978113+0000","last_fullsized":"2024-08-03T20:47:37.978113+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_clean_scrub_stamp":"2024-08-03T20:47:34.904132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T07:21:44.279582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-03T20:47:37.317505+0000","last_change":"2024-08-03T20:47:35.963774+0000","last_active":"2024-08-03T20:47:37.317505+0000","last_peered":"2024-08-03T20:47:37.317505+0000","last_clean":"2024-08-03T20:47:37.317505+0000","last_became_active":"2024-08-03T20:47:35.958408+0000","last_became_peered":"2024-08-03T20:47:35.958408+0000","last_unstale":"2024-08-03T20:47:37.317505+0000","last_undegraded":"2024-08-03T20:47:37.317505+0000","last_fullsized":"2024-08-03T20:47:37.317505+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_clean_scrub_stamp":"2024-08-03T20:47:34.904132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T00:58:45.652167+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-03T20:47:36.941791+0000","last_change":"2024-08-03T20:47:35.943147+0000","last_active":"2024-08-03T20:47:36.941791+0000","last_peered":"2024-08-03T20:47:36.941791+0000","last_clean":"2024-08-03T20:47:36.941791+0000","last_became_active":"2024-08-03T20:47:35.942253+0000","last_became_peered":"2024-08-03T20:47:35.942253+0000","last_unstale":"2024-08-03T20:47:36.941791+0000","last_undegraded":"2024-08-03T20:47:36.941791+0000","last_fullsized":"2024-08-03T20:47:36.941791+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_clean_scrub_stamp":"2024-08-03T20:47:34.904132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T01:08:56.085650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-03T20:47:36.941585+0000","last_change":"2024-08-03T20:47:35.938819+0000","last_active":"2024-08-03T20:47:36.941585+0000","last_peered":"2024-08-03T20:47:36.941585+0000","last_clean":"2024-08-03T20:47:36.941585+0000","last_became_active":"2024-08-03T20:47:35.938499+0000","last_became_peered":"2024-08-03T20:47:35.938499+0000","last_unstale":"2024-08-03T20:47:36.941585+0000","last_undegraded":"2024-08-03T20:47:36.941585+0000","last_fullsized":"2024-08-03T20:47:36.941585+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_clean_scrub_stamp":"2024-08-03T20:47:34.904132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T04:57:58.179012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-03T20:47:36.936210+0000","last_change":"2024-08-03T20:47:35.936873+0000","last_active":"2024-08-03T20:47:36.936210+0000","last_peered":"2024-08-03T20:47:36.936210+0000","last_clean":"2024-08-03T20:47:36.936210+0000","last_became_active":"2024-08-03T20:47:35.936005+0000","last_became_peered":"2024-08-03T20:47:35.936005+0000","last_unstale":"2024-08-03T20:47:36.936210+0000","last_undegraded":"2024-08-03T20:47:36.936210+0000","last_fullsized":"2024-08-03T20:47:36.936210+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_clean_scrub_stamp":"2024-08-03T20:47:34.904132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T01:42:26.761950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-03T20:47:37.978994+0000","last_change":"2024-08-03T20:47:35.950146+0000","last_active":"2024-08-03T20:47:37.978994+0000","last_peered":"2024-08-03T20:47:37.978994+0000","last_clean":"2024-08-03T20:47:37.978994+0000","last_became_active":"2024-08-03T20:47:35.949842+0000","last_became_peered":"2024-08-03T20:47:35.949842+0000","last_unstale":"2024-08-03T20:47:37.978994+0000","last_undegraded":"2024-08-03T20:47:37.978994+0000","last_fullsized":"2024-08-03T20:47:37.978994+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_clean_scrub_stamp":"2024-08-03T20:47:34.904132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-04T22:48:57.612773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-03T20:47:37.979037+0000","last_change":"2024-08-03T20:47:35.940356+0000","last_active":"2024-08-03T20:47:37.979037+0000","last_peered":"2024-08-03T20:47:37.979037+0000","last_clean":"2024-08-03T20:47:37.979037+0000","last_became_active":"2024-08-03T20:47:35.940143+0000","last_became_peered":"2024-08-03T20:47:35.940143+0000","last_unstale":"2024-08-03T20:47:37.979037+0000","last_undegraded":"2024-08-03T20:47:37.979037+0000","last_fullsized":"2024-08-03T20:47:37.979037+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_clean_scrub_stamp":"2024-08-03T20:47:34.904132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T04:01:08.373245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-03T20:47:36.935545+0000","last_change":"2024-08-03T20:47:35.951009+0000","last_active":"2024-08-03T20:47:36.935545+0000","last_peered":"2024-08-03T20:47:36.935545+0000","last_clean":"2024-08-03T20:47:36.935545+0000","last_became_active":"2024-08-03T20:47:35.950658+0000","last_became_peered":"2024-08-03T20:47:35.950658+0000","last_unstale":"2024-08-03T20:47:36.935545+0000","last_undegraded":"2024-08-03T20:47:36.935545+0000","last_fullsized":"2024-08-03T20:47:36.935545+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_clean_scrub_stamp":"2024-08-03T20:47:34.904132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-04T20:49:19.998935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-03T20:47:37.978861+0000","last_change":"2024-08-03T20:47:35.948379+0000","last_active":"2024-08-03T20:47:37.978861+0000","last_peered":"2024-08-03T20:47:37.978861+0000","last_clean":"2024-08-03T20:47:37.978861+0000","last_became_active":"2024-08-03T20:47:35.947812+0000","last_became_peered":"2024-08-03T20:47:35.947812+0000","last_unstale":"2024-08-03T20:47:37.978861+0000","last_undegraded":"2024-08-03T20:47:37.978861+0000","last_fullsized":"2024-08-03T20:47:37.978861+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_clean_scrub_stamp":"2024-08-03T20:47:34.904132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T01:08:56.085650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-03T20:47:37.979587+0000","last_change":"2024-08-03T20:47:35.939474+0000","last_active":"2024-08-03T20:47:37.979587+0000","last_peered":"2024-08-03T20:47:37.979587+0000","last_clean":"2024-08-03T20:47:37.979587+0000","last_became_active":"2024-08-03T20:47:35.938914+0000","last_became_peered":"2024-08-03T20:47:35.938914+0000","last_unstale":"2024-08-03T20:47:37.979587+0000","last_undegraded":"2024-08-03T20:47:37.979587+0000","last_fullsized":"2024-08-03T20:47:37.979587+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_clean_scrub_stamp":"2024-08-03T20:47:34.904132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T01:52:59.081172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-03T20:47:37.352899+0000","last_change":"2024-08-03T20:47:35.933829+0000","last_active":"2024-08-03T20:47:37.352899+0000","last_peered":"2024-08-03T20:47:37.352899+0000","last_clean":"2024-08-03T20:47:37.352899+0000","last_became_active":"2024-08-03T20:47:35.933515+0000","last_became_peered":"2024-08-03T20:47:35.933515+0000","last_unstale":"2024-08-03T20:47:37.352899+0000","last_undegraded":"2024-08-03T20:47:37.352899+0000","last_fullsized":"2024-08-03T20:47:37.352899+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_clean_scrub_stamp":"2024-08-03T20:47:34.904132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T02:10:57.293691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-03T20:47:37.317301+0000","last_change":"2024-08-03T20:47:35.952999+0000","last_active":"2024-08-03T20:47:37.317301+0000","last_peered":"2024-08-03T20:47:37.317301+0000","last_clean":"2024-08-03T20:47:37.317301+0000","last_became_active":"2024-08-03T20:47:35.952281+0000","last_became_peered":"2024-08-03T20:47:35.952281+0000","last_unstale":"2024-08-03T20:47:37.317301+0000","last_undegraded":"2024-08-03T20:47:37.317301+0000","last_fullsized":"2024-08-03T20:47:37.317301+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_clean_scrub_stamp":"2024-08-03T20:47:34.904132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T01:15:48.374663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-03T20:47:36.941659+0000","last_change":"2024-08-03T20:47:35.927412+0000","last_active":"2024-08-03T20:47:36.941659+0000","last_peered":"2024-08-03T20:47:36.941659+0000","last_clean":"2024-08-03T20:47:36.941659+0000","last_became_active":"2024-08-03T20:47:35.927069+0000","last_became_peered":"2024-08-03T20:47:35.927069+0000","last_unstale":"2024-08-03T20:47:36.941659+0000","last_undegraded":"2024-08-03T20:47:36.941659+0000","last_fullsized":"2024-08-03T20:47:36.941659+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_clean_scrub_stamp":"2024-08-03T20:47:34.904132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-04T20:49:19.998935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-03T20:47:37.979382+0000","last_change":"2024-08-03T20:47:35.936665+0000","last_active":"2024-08-03T20:47:37.979382+0000","last_peered":"2024-08-03T20:47:37.979382+0000","last_clean":"2024-08-03T20:47:37.979382+0000","last_became_active":"2024-08-03T20:47:35.936298+0000","last_became_peered":"2024-08-03T20:47:35.936298+0000","last_unstale":"2024-08-03T20:47:37.979382+0000","last_undegraded":"2024-08-03T20:47:37.979382+0000","last_fullsized":"2024-08-03T20:47:37.979382+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_clean_scrub_stamp":"2024-08-03T20:47:34.904132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-04T22:12:15.293757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-03T20:47:37.979017+0000","last_change":"2024-08-03T20:47:35.938468+0000","last_active":"2024-08-03T20:47:37.979017+0000","last_peered":"2024-08-03T20:47:37.979017+0000","last_clean":"2024-08-03T20:47:37.979017+0000","last_became_active":"2024-08-03T20:47:35.937757+0000","last_became_peered":"2024-08-03T20:47:35.937757+0000","last_unstale":"2024-08-03T20:47:37.979017+0000","last_undegraded":"2024-08-03T20:47:37.979017+0000","last_fullsized":"2024-08-03T20:47:37.979017+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_clean_scrub_stamp":"2024-08-03T20:47:34.904132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-04T23:07:18.318169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-03T20:47:37.978241+0000","last_change":"2024-08-03T20:47:35.953502+0000","last_active":"2024-08-03T20:47:37.978241+0000","last_peered":"2024-08-03T20:47:37.978241+0000","last_clean":"2024-08-03T20:47:37.978241+0000","last_became_active":"2024-08-03T20:47:35.953152+0000","last_became_peered":"2024-08-03T20:47:35.953152+0000","last_unstale":"2024-08-03T20:47:37.978241+0000","last_undegraded":"2024-08-03T20:47:37.978241+0000","last_fullsized":"2024-08-03T20:47:37.978241+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_clean_scrub_stamp":"2024-08-03T20:47:34.904132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T01:56:58.984458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-03T20:47:37.979235+0000","last_change":"2024-08-03T20:47:35.937927+0000","last_active":"2024-08-03T20:47:37.979235+0000","last_peered":"2024-08-03T20:47:37.979235+0000","last_clean":"2024-08-03T20:47:37.979235+0000","last_became_active":"2024-08-03T20:47:35.937579+0000","last_became_peered":"2024-08-03T20:47:35.937579+0000","last_unstale":"2024-08-03T20:47:37.979235+0000","last_undegraded":"2024-08-03T20:47:37.979235+0000","last_fullsized":"2024-08-03T20:47:37.979235+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_clean_scrub_stamp":"2024-08-03T20:47:34.904132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T03:44:29.605859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-03T20:47:37.979020+0000","last_change":"2024-08-03T20:47:35.953835+0000","last_active":"2024-08-03T20:47:37.979020+0000","last_peered":"2024-08-03T20:47:37.979020+0000","last_clean":"2024-08-03T20:47:37.979020+0000","last_became_active":"2024-08-03T20:47:35.953520+0000","last_became_peered":"2024-08-03T20:47:35.953520+0000","last_unstale":"2024-08-03T20:47:37.979020+0000","last_undegraded":"2024-08-03T20:47:37.979020+0000","last_fullsized":"2024-08-03T20:47:37.979020+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_clean_scrub_stamp":"2024-08-03T20:47:34.904132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T06:11:09.513212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-03T20:47:36.941856+0000","last_change":"2024-08-03T20:47:35.938228+0000","last_active":"2024-08-03T20:47:36.941856+0000","last_peered":"2024-08-03T20:47:36.941856+0000","last_clean":"2024-08-03T20:47:36.941856+0000","last_became_active":"2024-08-03T20:47:35.937893+0000","last_became_peered":"2024-08-03T20:47:35.937893+0000","last_unstale":"2024-08-03T20:47:36.941856+0000","last_undegraded":"2024-08-03T20:47:36.941856+0000","last_fullsized":"2024-08-03T20:47:36.941856+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_clean_scrub_stamp":"2024-08-03T20:47:34.904132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-04T22:33:13.028920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-03T20:47:36.935344+0000","last_change":"2024-08-03T20:47:35.937679+0000","last_active":"2024-08-03T20:47:36.935344+0000","last_peered":"2024-08-03T20:47:36.935344+0000","last_clean":"2024-08-03T20:47:36.935344+0000","last_became_active":"2024-08-03T20:47:35.937328+0000","last_became_peered":"2024-08-03T20:47:35.937328+0000","last_unstale":"2024-08-03T20:47:36.935344+0000","last_undegraded":"2024-08-03T20:47:36.935344+0000","last_fullsized":"2024-08-03T20:47:36.935344+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_clean_scrub_stamp":"2024-08-03T20:47:34.904132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T04:57:58.179012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-03T20:47:37.978822+0000","last_change":"2024-08-03T20:47:35.954328+0000","last_active":"2024-08-03T20:47:37.978822+0000","last_peered":"2024-08-03T20:47:37.978822+0000","last_clean":"2024-08-03T20:47:37.978822+0000","last_became_active":"2024-08-03T20:47:35.951285+0000","last_became_peered":"2024-08-03T20:47:35.951285+0000","last_unstale":"2024-08-03T20:47:37.978822+0000","last_undegraded":"2024-08-03T20:47:37.978822+0000","last_fullsized":"2024-08-03T20:47:37.978822+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_clean_scrub_stamp":"2024-08-03T20:47:34.904132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T01:01:23.625299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-03T20:47:37.979494+0000","last_change":"2024-08-03T20:47:35.940504+0000","last_active":"2024-08-03T20:47:37.979494+0000","last_peered":"2024-08-03T20:47:37.979494+0000","last_clean":"2024-08-03T20:47:37.979494+0000","last_became_active":"2024-08-03T20:47:35.938638+0000","last_became_peered":"2024-08-03T20:47:35.938638+0000","last_unstale":"2024-08-03T20:47:37.979494+0000","last_undegraded":"2024-08-03T20:47:37.979494+0000","last_fullsized":"2024-08-03T20:47:37.979494+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_clean_scrub_stamp":"2024-08-03T20:47:34.904132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-04T23:57:18.840519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-03T20:47:37.979441+0000","last_change":"2024-08-03T20:47:35.938131+0000","last_active":"2024-08-03T20:47:37.979441+0000","last_peered":"2024-08-03T20:47:37.979441+0000","last_clean":"2024-08-03T20:47:37.979441+0000","last_became_active":"2024-08-03T20:47:35.937270+0000","last_became_peered":"2024-08-03T20:47:35.937270+0000","last_unstale":"2024-08-03T20:47:37.979441+0000","last_undegraded":"2024-08-03T20:47:37.979441+0000","last_fullsized":"2024-08-03T20:47:37.979441+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_clean_scrub_stamp":"2024-08-03T20:47:34.904132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-04T21:57:09.430415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-03T20:47:37.352726+0000","last_change":"2024-08-03T20:47:35.937957+0000","last_active":"2024-08-03T20:47:37.352726+0000","last_peered":"2024-08-03T20:47:37.352726+0000","last_clean":"2024-08-03T20:47:37.352726+0000","last_became_active":"2024-08-03T20:47:35.937656+0000","last_became_peered":"2024-08-03T20:47:35.937656+0000","last_unstale":"2024-08-03T20:47:37.352726+0000","last_undegraded":"2024-08-03T20:47:37.352726+0000","last_fullsized":"2024-08-03T20:47:37.352726+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_clean_scrub_stamp":"2024-08-03T20:47:34.904132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T05:51:29.523373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-03T20:47:36.941128+0000","last_change":"2024-08-03T20:47:35.932459+0000","last_active":"2024-08-03T20:47:36.941128+0000","last_peered":"2024-08-03T20:47:36.941128+0000","last_clean":"2024-08-03T20:47:36.941128+0000","last_became_active":"2024-08-03T20:47:35.932103+0000","last_became_peered":"2024-08-03T20:47:35.932103+0000","last_unstale":"2024-08-03T20:47:36.941128+0000","last_undegraded":"2024-08-03T20:47:36.941128+0000","last_fullsized":"2024-08-03T20:47:36.941128+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_clean_scrub_stamp":"2024-08-03T20:47:34.904132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T08:34:04.057775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-03T20:47:36.935757+0000","last_change":"2024-08-03T20:47:35.940544+0000","last_active":"2024-08-03T20:47:36.935757+0000","last_peered":"2024-08-03T20:47:36.935757+0000","last_clean":"2024-08-03T20:47:36.935757+0000","last_became_active":"2024-08-03T20:47:35.940218+0000","last_became_peered":"2024-08-03T20:47:35.940218+0000","last_unstale":"2024-08-03T20:47:36.935757+0000","last_undegraded":"2024-08-03T20:47:36.935757+0000","last_fullsized":"2024-08-03T20:47:36.935757+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_clean_scrub_stamp":"2024-08-03T20:47:34.904132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T07:58:42.869702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"20'1","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-03T20:47:37.978814+0000","last_change":"2024-08-03T20:47:34.939881+0000","last_active":"2024-08-03T20:47:37.978814+0000","last_peered":"2024-08-03T20:47:37.978814+0000","last_clean":"2024-08-03T20:47:37.978814+0000","last_became_active":"2024-08-03T20:47:34.936765+0000","last_became_peered":"2024-08-03T20:47:34.936765+0000","last_unstale":"2024-08-03T20:47:37.978814+0000","last_undegraded":"2024-08-03T20:47:37.978814+0000","last_fullsized":"2024-08-03T20:47:37.978814+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:47:33.895731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:47:33.895731+0000","last_clean_scrub_stamp":"2024-08-03T20:47:33.895731+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T02:29:41.523344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-03T20:47:37.979599+0000","last_change":"2024-08-03T20:47:35.942088+0000","last_active":"2024-08-03T20:47:37.979599+0000","last_peered":"2024-08-03T20:47:37.979599+0000","last_clean":"2024-08-03T20:47:37.979599+0000","last_became_active":"2024-08-03T20:47:35.941800+0000","last_became_peered":"2024-08-03T20:47:35.941800+0000","last_unstale":"2024-08-03T20:47:37.979599+0000","last_undegraded":"2024-08-03T20:47:37.979599+0000","last_fullsized":"2024-08-03T20:47:37.979599+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_clean_scrub_stamp":"2024-08-03T20:47:34.904132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-04T20:49:19.998935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"20'1","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-03T20:47:36.941066+0000","last_change":"2024-08-03T20:47:34.938348+0000","last_active":"2024-08-03T20:47:36.941066+0000","last_peered":"2024-08-03T20:47:36.941066+0000","last_clean":"2024-08-03T20:47:36.941066+0000","last_became_active":"2024-08-03T20:47:34.936169+0000","last_became_peered":"2024-08-03T20:47:34.936169+0000","last_unstale":"2024-08-03T20:47:36.941066+0000","last_undegraded":"2024-08-03T20:47:36.941066+0000","last_fullsized":"2024-08-03T20:47:36.941066+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:47:33.895731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:47:33.895731+0000","last_clean_scrub_stamp":"2024-08-03T20:47:33.895731+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T06:44:37.944928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-03T20:47:37.978344+0000","last_change":"2024-08-03T20:47:34.935991+0000","last_active":"2024-08-03T20:47:37.978344+0000","last_peered":"2024-08-03T20:47:37.978344+0000","last_clean":"2024-08-03T20:47:37.978344+0000","last_became_active":"2024-08-03T20:47:34.930541+0000","last_became_peered":"2024-08-03T20:47:34.930541+0000","last_unstale":"2024-08-03T20:47:37.978344+0000","last_undegraded":"2024-08-03T20:47:37.978344+0000","last_fullsized":"2024-08-03T20:47:37.978344+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:47:33.895731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:47:33.895731+0000","last_clean_scrub_stamp":"2024-08-03T20:47:33.895731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T04:37:55.336635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-03T20:47:37.978164+0000","last_change":"2024-08-03T20:47:35.934722+0000","last_active":"2024-08-03T20:47:37.978164+0000","last_peered":"2024-08-03T20:47:37.978164+0000","last_clean":"2024-08-03T20:47:37.978164+0000","last_became_active":"2024-08-03T20:47:35.934183+0000","last_became_peered":"2024-08-03T20:47:35.934183+0000","last_unstale":"2024-08-03T20:47:37.978164+0000","last_undegraded":"2024-08-03T20:47:37.978164+0000","last_fullsized":"2024-08-03T20:47:37.978164+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_clean_scrub_stamp":"2024-08-03T20:47:34.904132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T06:29:11.740575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-03T20:47:37.978945+0000","last_change":"2024-08-03T20:47:34.934591+0000","last_active":"2024-08-03T20:47:37.978945+0000","last_peered":"2024-08-03T20:47:37.978945+0000","last_clean":"2024-08-03T20:47:37.978945+0000","last_became_active":"2024-08-03T20:47:34.934155+0000","last_became_peered":"2024-08-03T20:47:34.934155+0000","last_unstale":"2024-08-03T20:47:37.978945+0000","last_undegraded":"2024-08-03T20:47:37.978945+0000","last_fullsized":"2024-08-03T20:47:37.978945+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:47:33.895731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:47:33.895731+0000","last_clean_scrub_stamp":"2024-08-03T20:47:33.895731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-04T22:30:28.119577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-03T20:47:36.934971+0000","last_change":"2024-08-03T20:47:35.947874+0000","last_active":"2024-08-03T20:47:36.934971+0000","last_peered":"2024-08-03T20:47:36.934971+0000","last_clean":"2024-08-03T20:47:36.934971+0000","last_became_active":"2024-08-03T20:47:35.947570+0000","last_became_peered":"2024-08-03T20:47:35.947570+0000","last_unstale":"2024-08-03T20:47:36.934971+0000","last_undegraded":"2024-08-03T20:47:36.934971+0000","last_fullsized":"2024-08-03T20:47:36.934971+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_clean_scrub_stamp":"2024-08-03T20:47:34.904132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-04T22:12:15.293757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-03T20:47:36.941786+0000","last_change":"2024-08-03T20:47:34.937955+0000","last_active":"2024-08-03T20:47:36.941786+0000","last_peered":"2024-08-03T20:47:36.941786+0000","last_clean":"2024-08-03T20:47:36.941786+0000","last_became_active":"2024-08-03T20:47:34.937613+0000","last_became_peered":"2024-08-03T20:47:34.937613+0000","last_unstale":"2024-08-03T20:47:36.941786+0000","last_undegraded":"2024-08-03T20:47:36.941786+0000","last_fullsized":"2024-08-03T20:47:36.941786+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:47:33.895731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:47:33.895731+0000","last_clean_scrub_stamp":"2024-08-03T20:47:33.895731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-04T22:30:28.119577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-03T20:47:37.979527+0000","last_change":"2024-08-03T20:47:35.941948+0000","last_active":"2024-08-03T20:47:37.979527+0000","last_peered":"2024-08-03T20:47:37.979527+0000","last_clean":"2024-08-03T20:47:37.979527+0000","last_became_active":"2024-08-03T20:47:35.941651+0000","last_became_peered":"2024-08-03T20:47:35.941651+0000","last_unstale":"2024-08-03T20:47:37.979527+0000","last_undegraded":"2024-08-03T20:47:37.979527+0000","last_fullsized":"2024-08-03T20:47:37.979527+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_clean_scrub_stamp":"2024-08-03T20:47:34.904132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T06:29:37.397869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-03T20:47:37.979267+0000","last_change":"2024-08-03T20:47:34.936965+0000","last_active":"2024-08-03T20:47:37.979267+0000","last_peered":"2024-08-03T20:47:37.979267+0000","last_clean":"2024-08-03T20:47:37.979267+0000","last_became_active":"2024-08-03T20:47:34.936667+0000","last_became_peered":"2024-08-03T20:47:34.936667+0000","last_unstale":"2024-08-03T20:47:37.979267+0000","last_undegraded":"2024-08-03T20:47:37.979267+0000","last_fullsized":"2024-08-03T20:47:37.979267+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:47:33.895731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:47:33.895731+0000","last_clean_scrub_stamp":"2024-08-03T20:47:33.895731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T07:46:07.162679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-03T20:47:37.979788+0000","last_change":"2024-08-03T20:47:35.940917+0000","last_active":"2024-08-03T20:47:37.979788+0000","last_peered":"2024-08-03T20:47:37.979788+0000","last_clean":"2024-08-03T20:47:37.979788+0000","last_became_active":"2024-08-03T20:47:35.940218+0000","last_became_peered":"2024-08-03T20:47:35.940218+0000","last_unstale":"2024-08-03T20:47:37.979788+0000","last_undegraded":"2024-08-03T20:47:37.979788+0000","last_fullsized":"2024-08-03T20:47:37.979788+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_clean_scrub_stamp":"2024-08-03T20:47:34.904132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-04T23:07:18.318169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"20'3","reported_seq":18,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-03T20:47:36.935425+0000","last_change":"2024-08-03T20:47:34.939100+0000","last_active":"2024-08-03T20:47:36.935425+0000","last_peered":"2024-08-03T20:47:36.935425+0000","last_clean":"2024-08-03T20:47:36.935425+0000","last_became_active":"2024-08-03T20:47:34.935869+0000","last_became_peered":"2024-08-03T20:47:34.935869+0000","last_unstale":"2024-08-03T20:47:36.935425+0000","last_undegraded":"2024-08-03T20:47:36.935425+0000","last_fullsized":"2024-08-03T20:47:36.935425+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:47:33.895731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:47:33.895731+0000","last_clean_scrub_stamp":"2024-08-03T20:47:33.895731+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T06:44:37.944928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-03T20:47:36.936329+0000","last_change":"2024-08-03T20:47:35.944428+0000","last_active":"2024-08-03T20:47:36.936329+0000","last_peered":"2024-08-03T20:47:36.936329+0000","last_clean":"2024-08-03T20:47:36.936329+0000","last_became_active":"2024-08-03T20:47:35.943608+0000","last_became_peered":"2024-08-03T20:47:35.943608+0000","last_unstale":"2024-08-03T20:47:36.936329+0000","last_undegraded":"2024-08-03T20:47:36.936329+0000","last_fullsized":"2024-08-03T20:47:36.936329+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_clean_scrub_stamp":"2024-08-03T20:47:34.904132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T05:51:29.523373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"20'6","reported_seq":21,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-03T20:47:36.941564+0000","last_change":"2024-08-03T20:47:34.931832+0000","last_active":"2024-08-03T20:47:36.941564+0000","last_peered":"2024-08-03T20:47:36.941564+0000","last_clean":"2024-08-03T20:47:36.941564+0000","last_became_active":"2024-08-03T20:47:34.931150+0000","last_became_peered":"2024-08-03T20:47:34.931150+0000","last_unstale":"2024-08-03T20:47:36.941564+0000","last_undegraded":"2024-08-03T20:47:36.941564+0000","last_fullsized":"2024-08-03T20:47:36.941564+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:47:33.895731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:47:33.895731+0000","last_clean_scrub_stamp":"2024-08-03T20:47:33.895731+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T05:01:48.922382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-03T20:47:37.316866+0000","last_change":"2024-08-03T20:47:35.953367+0000","last_active":"2024-08-03T20:47:37.316866+0000","last_peered":"2024-08-03T20:47:37.316866+0000","last_clean":"2024-08-03T20:47:37.316866+0000","last_became_active":"2024-08-03T20:47:35.952666+0000","last_became_peered":"2024-08-03T20:47:35.952666+0000","last_unstale":"2024-08-03T20:47:37.316866+0000","last_undegraded":"2024-08-03T20:47:37.316866+0000","last_fullsized":"2024-08-03T20:47:37.316866+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_clean_scrub_stamp":"2024-08-03T20:47:34.904132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-04T21:57:09.430415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"20'2","reported_seq":17,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-03T20:47:37.979652+0000","last_change":"2024-08-03T20:47:34.932411+0000","last_active":"2024-08-03T20:47:37.979652+0000","last_peered":"2024-08-03T20:47:37.979652+0000","last_clean":"2024-08-03T20:47:37.979652+0000","last_became_active":"2024-08-03T20:47:34.931336+0000","last_became_peered":"2024-08-03T20:47:34.931336+0000","last_unstale":"2024-08-03T20:47:37.979652+0000","last_undegraded":"2024-08-03T20:47:37.979652+0000","last_fullsized":"2024-08-03T20:47:37.979652+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:47:33.895731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:47:33.895731+0000","last_clean_scrub_stamp":"2024-08-03T20:47:33.895731+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T05:59:33.076657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-03T20:47:37.979070+0000","last_change":"2024-08-03T20:47:35.932875+0000","last_active":"2024-08-03T20:47:37.979070+0000","last_peered":"2024-08-03T20:47:37.979070+0000","last_clean":"2024-08-03T20:47:37.979070+0000","last_became_active":"2024-08-03T20:47:35.932516+0000","last_became_peered":"2024-08-03T20:47:35.932516+0000","last_unstale":"2024-08-03T20:47:37.979070+0000","last_undegraded":"2024-08-03T20:47:37.979070+0000","last_fullsized":"2024-08-03T20:47:37.979070+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_clean_scrub_stamp":"2024-08-03T20:47:34.904132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T08:34:04.057775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-03T20:47:37.979194+0000","last_change":"2024-08-03T20:47:34.947236+0000","last_active":"2024-08-03T20:47:37.979194+0000","last_peered":"2024-08-03T20:47:37.979194+0000","last_clean":"2024-08-03T20:47:37.979194+0000","last_became_active":"2024-08-03T20:47:34.946894+0000","last_became_peered":"2024-08-03T20:47:34.946894+0000","last_unstale":"2024-08-03T20:47:37.979194+0000","last_undegraded":"2024-08-03T20:47:37.979194+0000","last_fullsized":"2024-08-03T20:47:37.979194+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:47:33.895731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:47:33.895731+0000","last_clean_scrub_stamp":"2024-08-03T20:47:33.895731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T05:01:48.922382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-03T20:47:37.317004+0000","last_change":"2024-08-03T20:47:35.950815+0000","last_active":"2024-08-03T20:47:37.317004+0000","last_peered":"2024-08-03T20:47:37.317004+0000","last_clean":"2024-08-03T20:47:37.317004+0000","last_became_active":"2024-08-03T20:47:35.950494+0000","last_became_peered":"2024-08-03T20:47:35.950494+0000","last_unstale":"2024-08-03T20:47:37.317004+0000","last_undegraded":"2024-08-03T20:47:37.317004+0000","last_fullsized":"2024-08-03T20:47:37.317004+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_clean_scrub_stamp":"2024-08-03T20:47:34.904132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T05:49:42.422083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-03T20:47:36.941067+0000","last_change":"2024-08-03T20:47:34.939132+0000","last_active":"2024-08-03T20:47:36.941067+0000","last_peered":"2024-08-03T20:47:36.941067+0000","last_clean":"2024-08-03T20:47:36.941067+0000","last_became_active":"2024-08-03T20:47:34.937563+0000","last_became_peered":"2024-08-03T20:47:34.937563+0000","last_unstale":"2024-08-03T20:47:36.941067+0000","last_undegraded":"2024-08-03T20:47:36.941067+0000","last_fullsized":"2024-08-03T20:47:36.941067+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:47:33.895731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:47:33.895731+0000","last_clean_scrub_stamp":"2024-08-03T20:47:33.895731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T06:32:14.208950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-03T20:47:36.941315+0000","last_change":"2024-08-03T20:47:35.944827+0000","last_active":"2024-08-03T20:47:36.941315+0000","last_peered":"2024-08-03T20:47:36.941315+0000","last_clean":"2024-08-03T20:47:36.941315+0000","last_became_active":"2024-08-03T20:47:35.944517+0000","last_became_peered":"2024-08-03T20:47:35.944517+0000","last_unstale":"2024-08-03T20:47:36.941315+0000","last_undegraded":"2024-08-03T20:47:36.941315+0000","last_fullsized":"2024-08-03T20:47:36.941315+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_clean_scrub_stamp":"2024-08-03T20:47:34.904132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-04T23:57:18.840519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"20'8","reported_seq":23,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-03T20:47:37.352795+0000","last_change":"2024-08-03T20:47:34.936194+0000","last_active":"2024-08-03T20:47:37.352795+0000","last_peered":"2024-08-03T20:47:37.352795+0000","last_clean":"2024-08-03T20:47:37.352795+0000","last_became_active":"2024-08-03T20:47:34.935835+0000","last_became_peered":"2024-08-03T20:47:34.935835+0000","last_unstale":"2024-08-03T20:47:37.352795+0000","last_undegraded":"2024-08-03T20:47:37.352795+0000","last_fullsized":"2024-08-03T20:47:37.352795+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:47:33.895731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:47:33.895731+0000","last_clean_scrub_stamp":"2024-08-03T20:47:33.895731+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T06:13:41.904985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-03T20:47:37.316755+0000","last_change":"2024-08-03T20:47:35.950421+0000","last_active":"2024-08-03T20:47:37.316755+0000","last_peered":"2024-08-03T20:47:37.316755+0000","last_clean":"2024-08-03T20:47:37.316755+0000","last_became_active":"2024-08-03T20:47:35.950109+0000","last_became_peered":"2024-08-03T20:47:35.950109+0000","last_unstale":"2024-08-03T20:47:37.316755+0000","last_undegraded":"2024-08-03T20:47:37.316755+0000","last_fullsized":"2024-08-03T20:47:37.316755+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_clean_scrub_stamp":"2024-08-03T20:47:34.904132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T02:17:30.747723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"20'1","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-03T20:47:36.936564+0000","last_change":"2024-08-03T20:47:34.935417+0000","last_active":"2024-08-03T20:47:36.936564+0000","last_peered":"2024-08-03T20:47:36.936564+0000","last_clean":"2024-08-03T20:47:36.936564+0000","last_became_active":"2024-08-03T20:47:34.932211+0000","last_became_peered":"2024-08-03T20:47:34.932211+0000","last_unstale":"2024-08-03T20:47:36.936564+0000","last_undegraded":"2024-08-03T20:47:36.936564+0000","last_fullsized":"2024-08-03T20:47:36.936564+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:47:33.895731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:47:33.895731+0000","last_clean_scrub_stamp":"2024-08-03T20:47:33.895731+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T04:43:47.155950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-03T20:47:37.317197+0000","last_change":"2024-08-03T20:47:35.958660+0000","last_active":"2024-08-03T20:47:37.317197+0000","last_peered":"2024-08-03T20:47:37.317197+0000","last_clean":"2024-08-03T20:47:37.317197+0000","last_became_active":"2024-08-03T20:47:35.958296+0000","last_became_peered":"2024-08-03T20:47:35.958296+0000","last_unstale":"2024-08-03T20:47:37.317197+0000","last_undegraded":"2024-08-03T20:47:37.317197+0000","last_fullsized":"2024-08-03T20:47:37.317197+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_clean_scrub_stamp":"2024-08-03T20:47:34.904132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T01:52:59.081172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"20'2","reported_seq":17,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-03T20:47:37.352928+0000","last_change":"2024-08-03T20:47:34.934762+0000","last_active":"2024-08-03T20:47:37.352928+0000","last_peered":"2024-08-03T20:47:37.352928+0000","last_clean":"2024-08-03T20:47:37.352928+0000","last_became_active":"2024-08-03T20:47:34.933194+0000","last_became_peered":"2024-08-03T20:47:34.933194+0000","last_unstale":"2024-08-03T20:47:37.352928+0000","last_undegraded":"2024-08-03T20:47:37.352928+0000","last_fullsized":"2024-08-03T20:47:37.352928+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:47:33.895731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:47:33.895731+0000","last_clean_scrub_stamp":"2024-08-03T20:47:33.895731+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T07:58:41.861301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-03T20:47:37.317479+0000","last_change":"2024-08-03T20:47:35.952102+0000","last_active":"2024-08-03T20:47:37.317479+0000","last_peered":"2024-08-03T20:47:37.317479+0000","last_clean":"2024-08-03T20:47:37.317479+0000","last_became_active":"2024-08-03T20:47:35.951350+0000","last_became_peered":"2024-08-03T20:47:35.951350+0000","last_unstale":"2024-08-03T20:47:37.317479+0000","last_undegraded":"2024-08-03T20:47:37.317479+0000","last_fullsized":"2024-08-03T20:47:37.317479+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_clean_scrub_stamp":"2024-08-03T20:47:34.904132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-04T23:37:39.394391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"20'1","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-03T20:47:36.935542+0000","last_change":"2024-08-03T20:47:34.932568+0000","last_active":"2024-08-03T20:47:36.935542+0000","last_peered":"2024-08-03T20:47:36.935542+0000","last_clean":"2024-08-03T20:47:36.935542+0000","last_became_active":"2024-08-03T20:47:34.932223+0000","last_became_peered":"2024-08-03T20:47:34.932223+0000","last_unstale":"2024-08-03T20:47:36.935542+0000","last_undegraded":"2024-08-03T20:47:36.935542+0000","last_fullsized":"2024-08-03T20:47:36.935542+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:47:33.895731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:47:33.895731+0000","last_clean_scrub_stamp":"2024-08-03T20:47:33.895731+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T05:01:48.922382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-03T20:47:37.979523+0000","last_change":"2024-08-03T20:47:35.957384+0000","last_active":"2024-08-03T20:47:37.979523+0000","last_peered":"2024-08-03T20:47:37.979523+0000","last_clean":"2024-08-03T20:47:37.979523+0000","last_became_active":"2024-08-03T20:47:35.956464+0000","last_became_peered":"2024-08-03T20:47:35.956464+0000","last_unstale":"2024-08-03T20:47:37.979523+0000","last_undegraded":"2024-08-03T20:47:37.979523+0000","last_fullsized":"2024-08-03T20:47:37.979523+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_clean_scrub_stamp":"2024-08-03T20:47:34.904132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T05:49:42.422083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-03T20:47:37.352724+0000","last_change":"2024-08-03T20:47:34.934154+0000","last_active":"2024-08-03T20:47:37.352724+0000","last_peered":"2024-08-03T20:47:37.352724+0000","last_clean":"2024-08-03T20:47:37.352724+0000","last_became_active":"2024-08-03T20:47:34.933257+0000","last_became_peered":"2024-08-03T20:47:34.933257+0000","last_unstale":"2024-08-03T20:47:37.352724+0000","last_undegraded":"2024-08-03T20:47:37.352724+0000","last_fullsized":"2024-08-03T20:47:37.352724+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:47:33.895731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:47:33.895731+0000","last_clean_scrub_stamp":"2024-08-03T20:47:33.895731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-04T21:10:14.859583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-03T20:47:37.317012+0000","last_change":"2024-08-03T20:47:35.957705+0000","last_active":"2024-08-03T20:47:37.317012+0000","last_peered":"2024-08-03T20:47:37.317012+0000","last_clean":"2024-08-03T20:47:37.317012+0000","last_became_active":"2024-08-03T20:47:35.951141+0000","last_became_peered":"2024-08-03T20:47:35.951141+0000","last_unstale":"2024-08-03T20:47:37.317012+0000","last_undegraded":"2024-08-03T20:47:37.317012+0000","last_fullsized":"2024-08-03T20:47:37.317012+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_clean_scrub_stamp":"2024-08-03T20:47:34.904132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T06:29:37.397869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"16'72","reported_seq":111,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-03T20:47:37.316600+0000","last_change":"2024-08-03T20:47:30.547632+0000","last_active":"2024-08-03T20:47:37.316600+0000","last_peered":"2024-08-03T20:47:37.316600+0000","last_clean":"2024-08-03T20:47:37.316600+0000","last_became_active":"2024-08-03T20:47:30.546255+0000","last_became_peered":"2024-08-03T20:47:30.546255+0000","last_unstale":"2024-08-03T20:47:37.316600+0000","last_undegraded":"2024-08-03T20:47:37.316600+0000","last_fullsized":"2024-08-03T20:47:37.316600+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:47:28.874451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:47:28.874451+0000","last_clean_scrub_stamp":"2024-08-03T20:47:28.874451+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T00:19:40.548812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-03T20:47:37.978786+0000","last_change":"2024-08-03T20:47:34.941906+0000","last_active":"2024-08-03T20:47:37.978786+0000","last_peered":"2024-08-03T20:47:37.978786+0000","last_clean":"2024-08-03T20:47:37.978786+0000","last_became_active":"2024-08-03T20:47:34.941492+0000","last_became_peered":"2024-08-03T20:47:34.941492+0000","last_unstale":"2024-08-03T20:47:37.978786+0000","last_undegraded":"2024-08-03T20:47:37.978786+0000","last_fullsized":"2024-08-03T20:47:37.978786+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:47:33.895731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:47:33.895731+0000","last_clean_scrub_stamp":"2024-08-03T20:47:33.895731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T06:32:14.208950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-03T20:47:37.978467+0000","last_change":"2024-08-03T20:47:35.953945+0000","last_active":"2024-08-03T20:47:37.978467+0000","last_peered":"2024-08-03T20:47:37.978467+0000","last_clean":"2024-08-03T20:47:37.978467+0000","last_became_active":"2024-08-03T20:47:35.953580+0000","last_became_peered":"2024-08-03T20:47:35.953580+0000","last_unstale":"2024-08-03T20:47:37.978467+0000","last_undegraded":"2024-08-03T20:47:37.978467+0000","last_fullsized":"2024-08-03T20:47:37.978467+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_clean_scrub_stamp":"2024-08-03T20:47:34.904132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T03:08:26.700491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-03T20:47:37.316444+0000","last_change":"2024-08-03T20:47:34.944839+0000","last_active":"2024-08-03T20:47:37.316444+0000","last_peered":"2024-08-03T20:47:37.316444+0000","last_clean":"2024-08-03T20:47:37.316444+0000","last_became_active":"2024-08-03T20:47:34.944530+0000","last_became_peered":"2024-08-03T20:47:34.944530+0000","last_unstale":"2024-08-03T20:47:37.316444+0000","last_undegraded":"2024-08-03T20:47:37.316444+0000","last_fullsized":"2024-08-03T20:47:37.316444+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:47:33.895731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:47:33.895731+0000","last_clean_scrub_stamp":"2024-08-03T20:47:33.895731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T02:40:00.976532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-03T20:47:37.978347+0000","last_change":"2024-08-03T20:47:35.926904+0000","last_active":"2024-08-03T20:47:37.978347+0000","last_peered":"2024-08-03T20:47:37.978347+0000","last_clean":"2024-08-03T20:47:37.978347+0000","last_became_active":"2024-08-03T20:47:35.926525+0000","last_became_peered":"2024-08-03T20:47:35.926525+0000","last_unstale":"2024-08-03T20:47:37.978347+0000","last_undegraded":"2024-08-03T20:47:37.978347+0000","last_fullsized":"2024-08-03T20:47:37.978347+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_clean_scrub_stamp":"2024-08-03T20:47:34.904132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T05:57:44.170587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-03T20:47:37.352725+0000","last_change":"2024-08-03T20:47:34.939125+0000","last_active":"2024-08-03T20:47:37.352725+0000","last_peered":"2024-08-03T20:47:37.352725+0000","last_clean":"2024-08-03T20:47:37.352725+0000","last_became_active":"2024-08-03T20:47:34.935161+0000","last_became_peered":"2024-08-03T20:47:34.935161+0000","last_unstale":"2024-08-03T20:47:37.352725+0000","last_undegraded":"2024-08-03T20:47:37.352725+0000","last_fullsized":"2024-08-03T20:47:37.352725+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:47:33.895731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:47:33.895731+0000","last_clean_scrub_stamp":"2024-08-03T20:47:33.895731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T04:37:55.336635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-03T20:47:36.934972+0000","last_change":"2024-08-03T20:47:35.933618+0000","last_active":"2024-08-03T20:47:36.934972+0000","last_peered":"2024-08-03T20:47:36.934972+0000","last_clean":"2024-08-03T20:47:36.934972+0000","last_became_active":"2024-08-03T20:47:35.932873+0000","last_became_peered":"2024-08-03T20:47:35.932873+0000","last_unstale":"2024-08-03T20:47:36.934972+0000","last_undegraded":"2024-08-03T20:47:36.934972+0000","last_fullsized":"2024-08-03T20:47:36.934972+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_clean_scrub_stamp":"2024-08-03T20:47:34.904132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-04T23:57:18.840519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-03T20:47:37.979331+0000","last_change":"2024-08-03T20:47:34.940602+0000","last_active":"2024-08-03T20:47:37.979331+0000","last_peered":"2024-08-03T20:47:37.979331+0000","last_clean":"2024-08-03T20:47:37.979331+0000","last_became_active":"2024-08-03T20:47:34.940156+0000","last_became_peered":"2024-08-03T20:47:34.940156+0000","last_unstale":"2024-08-03T20:47:37.979331+0000","last_undegraded":"2024-08-03T20:47:37.979331+0000","last_fullsized":"2024-08-03T20:47:37.979331+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:47:33.895731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:47:33.895731+0000","last_clean_scrub_stamp":"2024-08-03T20:47:33.895731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T06:44:37.944928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-03T20:47:36.935959+0000","last_change":"2024-08-03T20:47:35.938536+0000","last_active":"2024-08-03T20:47:36.935959+0000","last_peered":"2024-08-03T20:47:36.935959+0000","last_clean":"2024-08-03T20:47:36.935959+0000","last_became_active":"2024-08-03T20:47:35.938281+0000","last_became_peered":"2024-08-03T20:47:35.938281+0000","last_unstale":"2024-08-03T20:47:36.935959+0000","last_undegraded":"2024-08-03T20:47:36.935959+0000","last_fullsized":"2024-08-03T20:47:36.935959+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_clean_scrub_stamp":"2024-08-03T20:47:34.904132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T06:17:22.118393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-03T20:47:36.935901+0000","last_change":"2024-08-03T20:47:34.936859+0000","last_active":"2024-08-03T20:47:36.935901+0000","last_peered":"2024-08-03T20:47:36.935901+0000","last_clean":"2024-08-03T20:47:36.935901+0000","last_became_active":"2024-08-03T20:47:34.936017+0000","last_became_peered":"2024-08-03T20:47:34.936017+0000","last_unstale":"2024-08-03T20:47:36.935901+0000","last_undegraded":"2024-08-03T20:47:36.935901+0000","last_fullsized":"2024-08-03T20:47:36.935901+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:47:33.895731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:47:33.895731+0000","last_clean_scrub_stamp":"2024-08-03T20:47:33.895731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T02:40:00.976532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-03T20:47:37.979373+0000","last_change":"2024-08-03T20:47:35.940325+0000","last_active":"2024-08-03T20:47:37.979373+0000","last_peered":"2024-08-03T20:47:37.979373+0000","last_clean":"2024-08-03T20:47:37.979373+0000","last_became_active":"2024-08-03T20:47:35.939888+0000","last_became_peered":"2024-08-03T20:47:35.939888+0000","last_unstale":"2024-08-03T20:47:37.979373+0000","last_undegraded":"2024-08-03T20:47:37.979373+0000","last_fullsized":"2024-08-03T20:47:37.979373+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_clean_scrub_stamp":"2024-08-03T20:47:34.904132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T02:17:30.747723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-03T20:47:37.317158+0000","last_change":"2024-08-03T20:47:34.946657+0000","last_active":"2024-08-03T20:47:37.317158+0000","last_peered":"2024-08-03T20:47:37.317158+0000","last_clean":"2024-08-03T20:47:37.317158+0000","last_became_active":"2024-08-03T20:47:34.945719+0000","last_became_peered":"2024-08-03T20:47:34.945719+0000","last_unstale":"2024-08-03T20:47:37.317158+0000","last_undegraded":"2024-08-03T20:47:37.317158+0000","last_fullsized":"2024-08-03T20:47:37.317158+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:47:33.895731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:47:33.895731+0000","last_clean_scrub_stamp":"2024-08-03T20:47:33.895731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-04T21:30:40.295081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-03T20:47:36.935812+0000","last_change":"2024-08-03T20:47:35.936144+0000","last_active":"2024-08-03T20:47:36.935812+0000","last_peered":"2024-08-03T20:47:36.935812+0000","last_clean":"2024-08-03T20:47:36.935812+0000","last_became_active":"2024-08-03T20:47:35.935842+0000","last_became_peered":"2024-08-03T20:47:35.935842+0000","last_unstale":"2024-08-03T20:47:36.935812+0000","last_undegraded":"2024-08-03T20:47:36.935812+0000","last_fullsized":"2024-08-03T20:47:36.935812+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_clean_scrub_stamp":"2024-08-03T20:47:34.904132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-04T22:33:13.028920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"21'1","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-03T20:47:36.935790+0000","last_change":"2024-08-03T20:47:34.934773+0000","last_active":"2024-08-03T20:47:36.935790+0000","last_peered":"2024-08-03T20:47:36.935790+0000","last_clean":"2024-08-03T20:47:36.935790+0000","last_became_active":"2024-08-03T20:47:34.934473+0000","last_became_peered":"2024-08-03T20:47:34.934473+0000","last_unstale":"2024-08-03T20:47:36.935790+0000","last_undegraded":"2024-08-03T20:47:36.935790+0000","last_fullsized":"2024-08-03T20:47:36.935790+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:47:33.895731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:47:33.895731+0000","last_clean_scrub_stamp":"2024-08-03T20:47:33.895731+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-04T22:30:28.119577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-03T20:47:37.979065+0000","last_change":"2024-08-03T20:47:35.933622+0000","last_active":"2024-08-03T20:47:37.979065+0000","last_peered":"2024-08-03T20:47:37.979065+0000","last_clean":"2024-08-03T20:47:37.979065+0000","last_became_active":"2024-08-03T20:47:35.933133+0000","last_became_peered":"2024-08-03T20:47:35.933133+0000","last_unstale":"2024-08-03T20:47:37.979065+0000","last_undegraded":"2024-08-03T20:47:37.979065+0000","last_fullsized":"2024-08-03T20:47:37.979065+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_clean_scrub_stamp":"2024-08-03T20:47:34.904132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-04T22:43:36.335718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"20'2","reported_seq":17,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-03T20:47:36.941891+0000","last_change":"2024-08-03T20:47:34.944805+0000","last_active":"2024-08-03T20:47:36.941891+0000","last_peered":"2024-08-03T20:47:36.941891+0000","last_clean":"2024-08-03T20:47:36.941891+0000","last_became_active":"2024-08-03T20:47:34.943207+0000","last_became_peered":"2024-08-03T20:47:34.943207+0000","last_unstale":"2024-08-03T20:47:36.941891+0000","last_undegraded":"2024-08-03T20:47:36.941891+0000","last_fullsized":"2024-08-03T20:47:36.941891+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:47:33.895731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:47:33.895731+0000","last_clean_scrub_stamp":"2024-08-03T20:47:33.895731+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T07:46:07.162679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-03T20:47:37.979614+0000","last_change":"2024-08-03T20:47:35.941163+0000","last_active":"2024-08-03T20:47:37.979614+0000","last_peered":"2024-08-03T20:47:37.979614+0000","last_clean":"2024-08-03T20:47:37.979614+0000","last_became_active":"2024-08-03T20:47:35.940955+0000","last_became_peered":"2024-08-03T20:47:35.940955+0000","last_unstale":"2024-08-03T20:47:37.979614+0000","last_undegraded":"2024-08-03T20:47:37.979614+0000","last_fullsized":"2024-08-03T20:47:37.979614+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_clean_scrub_stamp":"2024-08-03T20:47:34.904132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T01:08:56.085650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-03T20:47:37.316412+0000","last_change":"2024-08-03T20:47:34.943492+0000","last_active":"2024-08-03T20:47:37.316412+0000","last_peered":"2024-08-03T20:47:37.316412+0000","last_clean":"2024-08-03T20:47:37.316412+0000","last_became_active":"2024-08-03T20:47:34.943121+0000","last_became_peered":"2024-08-03T20:47:34.943121+0000","last_unstale":"2024-08-03T20:47:37.316412+0000","last_undegraded":"2024-08-03T20:47:37.316412+0000","last_fullsized":"2024-08-03T20:47:37.316412+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:47:33.895731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:47:33.895731+0000","last_clean_scrub_stamp":"2024-08-03T20:47:33.895731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T07:46:07.162679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-03T20:47:36.935830+0000","last_change":"2024-08-03T20:47:35.924518+0000","last_active":"2024-08-03T20:47:36.935830+0000","last_peered":"2024-08-03T20:47:36.935830+0000","last_clean":"2024-08-03T20:47:36.935830+0000","last_became_active":"2024-08-03T20:47:35.923938+0000","last_became_peered":"2024-08-03T20:47:35.923938+0000","last_unstale":"2024-08-03T20:47:36.935830+0000","last_undegraded":"2024-08-03T20:47:36.935830+0000","last_fullsized":"2024-08-03T20:47:36.935830+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_clean_scrub_stamp":"2024-08-03T20:47:34.904132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T04:26:33.175697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-03T20:47:36.935369+0000","last_change":"2024-08-03T20:47:34.941712+0000","last_active":"2024-08-03T20:47:36.935369+0000","last_peered":"2024-08-03T20:47:36.935369+0000","last_clean":"2024-08-03T20:47:36.935369+0000","last_became_active":"2024-08-03T20:47:34.941138+0000","last_became_peered":"2024-08-03T20:47:34.941138+0000","last_unstale":"2024-08-03T20:47:36.935369+0000","last_undegraded":"2024-08-03T20:47:36.935369+0000","last_fullsized":"2024-08-03T20:47:36.935369+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:47:33.895731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:47:33.895731+0000","last_clean_scrub_stamp":"2024-08-03T20:47:33.895731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T07:46:07.162679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-03T20:47:37.978318+0000","last_change":"2024-08-03T20:47:35.941992+0000","last_active":"2024-08-03T20:47:37.978318+0000","last_peered":"2024-08-03T20:47:37.978318+0000","last_clean":"2024-08-03T20:47:37.978318+0000","last_became_active":"2024-08-03T20:47:35.941633+0000","last_became_peered":"2024-08-03T20:47:35.941633+0000","last_unstale":"2024-08-03T20:47:37.978318+0000","last_undegraded":"2024-08-03T20:47:37.978318+0000","last_fullsized":"2024-08-03T20:47:37.978318+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_clean_scrub_stamp":"2024-08-03T20:47:34.904132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T03:06:49.510875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-03T20:47:36.935863+0000","last_change":"2024-08-03T20:47:34.932831+0000","last_active":"2024-08-03T20:47:36.935863+0000","last_peered":"2024-08-03T20:47:36.935863+0000","last_clean":"2024-08-03T20:47:36.935863+0000","last_became_active":"2024-08-03T20:47:34.932520+0000","last_became_peered":"2024-08-03T20:47:34.932520+0000","last_unstale":"2024-08-03T20:47:36.935863+0000","last_undegraded":"2024-08-03T20:47:36.935863+0000","last_fullsized":"2024-08-03T20:47:36.935863+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:47:33.895731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:47:33.895731+0000","last_clean_scrub_stamp":"2024-08-03T20:47:33.895731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T00:43:30.770268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-03T20:47:37.978692+0000","last_change":"2024-08-03T20:47:35.942701+0000","last_active":"2024-08-03T20:47:37.978692+0000","last_peered":"2024-08-03T20:47:37.978692+0000","last_clean":"2024-08-03T20:47:37.978692+0000","last_became_active":"2024-08-03T20:47:35.942261+0000","last_became_peered":"2024-08-03T20:47:35.942261+0000","last_unstale":"2024-08-03T20:47:37.978692+0000","last_undegraded":"2024-08-03T20:47:37.978692+0000","last_fullsized":"2024-08-03T20:47:37.978692+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_clean_scrub_stamp":"2024-08-03T20:47:34.904132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-04T22:01:37.009450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"20'2","reported_seq":17,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-03T20:47:37.317111+0000","last_change":"2024-08-03T20:47:34.947233+0000","last_active":"2024-08-03T20:47:37.317111+0000","last_peered":"2024-08-03T20:47:37.317111+0000","last_clean":"2024-08-03T20:47:37.317111+0000","last_became_active":"2024-08-03T20:47:34.946866+0000","last_became_peered":"2024-08-03T20:47:34.946866+0000","last_unstale":"2024-08-03T20:47:37.317111+0000","last_undegraded":"2024-08-03T20:47:37.317111+0000","last_fullsized":"2024-08-03T20:47:37.317111+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:47:33.895731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:47:33.895731+0000","last_clean_scrub_stamp":"2024-08-03T20:47:33.895731+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T03:46:04.675513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-03T20:47:36.941158+0000","last_change":"2024-08-03T20:47:35.936142+0000","last_active":"2024-08-03T20:47:36.941158+0000","last_peered":"2024-08-03T20:47:36.941158+0000","last_clean":"2024-08-03T20:47:36.941158+0000","last_became_active":"2024-08-03T20:47:35.935764+0000","last_became_peered":"2024-08-03T20:47:35.935764+0000","last_unstale":"2024-08-03T20:47:36.941158+0000","last_undegraded":"2024-08-03T20:47:36.941158+0000","last_fullsized":"2024-08-03T20:47:36.941158+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_clean_scrub_stamp":"2024-08-03T20:47:34.904132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-04T22:12:15.293757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-03T20:47:37.979240+0000","last_change":"2024-08-03T20:47:35.934358+0000","last_active":"2024-08-03T20:47:37.979240+0000","last_peered":"2024-08-03T20:47:37.979240+0000","last_clean":"2024-08-03T20:47:37.979240+0000","last_became_active":"2024-08-03T20:47:35.933874+0000","last_became_peered":"2024-08-03T20:47:35.933874+0000","last_unstale":"2024-08-03T20:47:37.979240+0000","last_undegraded":"2024-08-03T20:47:37.979240+0000","last_fullsized":"2024-08-03T20:47:37.979240+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_clean_scrub_stamp":"2024-08-03T20:47:34.904132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T00:10:43.343156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"20'2","reported_seq":17,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-03T20:47:37.316590+0000","last_change":"2024-08-03T20:47:34.950578+0000","last_active":"2024-08-03T20:47:37.316590+0000","last_peered":"2024-08-03T20:47:37.316590+0000","last_clean":"2024-08-03T20:47:37.316590+0000","last_became_active":"2024-08-03T20:47:34.950187+0000","last_became_peered":"2024-08-03T20:47:34.950187+0000","last_unstale":"2024-08-03T20:47:37.316590+0000","last_undegraded":"2024-08-03T20:47:37.316590+0000","last_fullsized":"2024-08-03T20:47:37.316590+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:47:33.895731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:47:33.895731+0000","last_clean_scrub_stamp":"2024-08-03T20:47:33.895731+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T00:43:30.770268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-03T20:47:37.979438+0000","last_change":"2024-08-03T20:47:35.934586+0000","last_active":"2024-08-03T20:47:37.979438+0000","last_peered":"2024-08-03T20:47:37.979438+0000","last_clean":"2024-08-03T20:47:37.979438+0000","last_became_active":"2024-08-03T20:47:35.934289+0000","last_became_peered":"2024-08-03T20:47:35.934289+0000","last_unstale":"2024-08-03T20:47:37.979438+0000","last_undegraded":"2024-08-03T20:47:37.979438+0000","last_fullsized":"2024-08-03T20:47:37.979438+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_clean_scrub_stamp":"2024-08-03T20:47:34.904132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T04:51:27.371858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"20'2","reported_seq":17,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-03T20:47:37.316998+0000","last_change":"2024-08-03T20:47:34.946365+0000","last_active":"2024-08-03T20:47:37.316998+0000","last_peered":"2024-08-03T20:47:37.316998+0000","last_clean":"2024-08-03T20:47:37.316998+0000","last_became_active":"2024-08-03T20:47:34.945922+0000","last_became_peered":"2024-08-03T20:47:34.945922+0000","last_unstale":"2024-08-03T20:47:37.316998+0000","last_undegraded":"2024-08-03T20:47:37.316998+0000","last_fullsized":"2024-08-03T20:47:37.316998+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:47:33.895731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:47:33.895731+0000","last_clean_scrub_stamp":"2024-08-03T20:47:33.895731+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T06:59:03.089784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-03T20:47:37.979781+0000","last_change":"2024-08-03T20:47:35.941147+0000","last_active":"2024-08-03T20:47:37.979781+0000","last_peered":"2024-08-03T20:47:37.979781+0000","last_clean":"2024-08-03T20:47:37.979781+0000","last_became_active":"2024-08-03T20:47:35.940605+0000","last_became_peered":"2024-08-03T20:47:35.940605+0000","last_unstale":"2024-08-03T20:47:37.979781+0000","last_undegraded":"2024-08-03T20:47:37.979781+0000","last_fullsized":"2024-08-03T20:47:37.979781+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_clean_scrub_stamp":"2024-08-03T20:47:34.904132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-04T22:33:13.028920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"20'1","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-03T20:47:37.978906+0000","last_change":"2024-08-03T20:47:34.934888+0000","last_active":"2024-08-03T20:47:37.978906+0000","last_peered":"2024-08-03T20:47:37.978906+0000","last_clean":"2024-08-03T20:47:37.978906+0000","last_became_active":"2024-08-03T20:47:34.934595+0000","last_became_peered":"2024-08-03T20:47:34.934595+0000","last_unstale":"2024-08-03T20:47:37.978906+0000","last_undegraded":"2024-08-03T20:47:37.978906+0000","last_fullsized":"2024-08-03T20:47:37.978906+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:47:33.895731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:47:33.895731+0000","last_clean_scrub_stamp":"2024-08-03T20:47:33.895731+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T06:13:41.904985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-03T20:47:37.978652+0000","last_change":"2024-08-03T20:47:35.953798+0000","last_active":"2024-08-03T20:47:37.978652+0000","last_peered":"2024-08-03T20:47:37.978652+0000","last_clean":"2024-08-03T20:47:37.978652+0000","last_became_active":"2024-08-03T20:47:35.950186+0000","last_became_peered":"2024-08-03T20:47:35.950186+0000","last_unstale":"2024-08-03T20:47:37.978652+0000","last_undegraded":"2024-08-03T20:47:37.978652+0000","last_fullsized":"2024-08-03T20:47:37.978652+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_clean_scrub_stamp":"2024-08-03T20:47:34.904132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T07:58:55.643722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"20'4","reported_seq":19,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-03T20:47:37.979858+0000","last_change":"2024-08-03T20:47:34.939409+0000","last_active":"2024-08-03T20:47:37.979858+0000","last_peered":"2024-08-03T20:47:37.979858+0000","last_clean":"2024-08-03T20:47:37.979858+0000","last_became_active":"2024-08-03T20:47:34.939057+0000","last_became_peered":"2024-08-03T20:47:34.939057+0000","last_unstale":"2024-08-03T20:47:37.979858+0000","last_undegraded":"2024-08-03T20:47:37.979858+0000","last_fullsized":"2024-08-03T20:47:37.979858+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:47:33.895731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:47:33.895731+0000","last_clean_scrub_stamp":"2024-08-03T20:47:33.895731+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T05:49:05.605066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509444,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31972,"kb_used_data":788,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339868,"statfs":{"total":96636764160,"available":96604024832,"internally_reserved":0,"allocated":806912,"data_stored":663491,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":6,"up_from":15,"seq":64424509445,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31396,"kb_used_data":216,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340444,"statfs":{"total":96636764160,"available":96604614656,"internally_reserved":0,"allocated":221184,"data_stored":74153,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":5,"up_from":15,"seq":64424509445,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31388,"kb_used_data":208,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340452,"statfs":{"total":96636764160,"available":96604622848,"internally_reserved":0,"allocated":212992,"data_stored":72631,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509444,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31392,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340448,"statfs":{"total":96636764160,"available":96604618752,"internally_reserved":0,"allocated":217088,"data_stored":72701,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509446,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31980,"kb_used_data":796,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339860,"statfs":{"total":96636764160,"available":96604016640,"internally_reserved":0,"allocated":815104,"data_stored":663547,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542149,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31400,"kb_used_data":216,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340440,"statfs":{"total":96636764160,"available":96604610560,"internally_reserved":0,"allocated":221184,"data_stored":72667,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":3,"up_from":14,"seq":60129542147,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31400,"kb_used_data":216,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340440,"statfs":{"total":96636764160,"available":96604610560,"internally_reserved":0,"allocated":221184,"data_stored":74153,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":14,"seq":60129542148,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31388,"kb_used_data":204,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340452,"statfs":{"total":96636764160,"available":96604622848,"internally_reserved":0,"allocated":208896,"data_stored":72541,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-08-03T20:47:40.952 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-08-03T20:47:40.952 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config log 1 --format=json 2024-08-03T20:47:40.953 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2024-08-03T20:47:41.216 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:41.215+0000 7fb983600640 1 -- 172.21.15.32:0/3939843059 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb97c073e60 msgr2=0x7fb97c074270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:47:41.217 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:41.215+0000 7fb983600640 1 --2- 172.21.15.32:0/3939843059 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb97c073e60 0x7fb97c074270 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fb968009830 tx=0x7fb96802a480 comp rx=0 tx=0).stop 2024-08-03T20:47:41.217 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:41.215+0000 7fb983600640 1 -- 172.21.15.32:0/3939843059 shutdown_connections 2024-08-03T20:47:41.217 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:41.215+0000 7fb983600640 1 --2- 172.21.15.32:0/3939843059 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb97c073e60 0x7fb97c074270 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:41.217 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:41.215+0000 7fb983600640 1 --2- 172.21.15.32:0/3939843059 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fb97c06e930 0x7fb97c144f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:41.217 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:41.215+0000 7fb983600640 1 -- 172.21.15.32:0/3939843059 >> 172.21.15.32:0/3939843059 conn(0x7fb97c0b1bd0 msgr2=0x7fb97c0b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:47:41.217 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:41.215+0000 7fb983600640 1 -- 172.21.15.32:0/3939843059 shutdown_connections 2024-08-03T20:47:41.217 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:41.215+0000 7fb983600640 1 -- 172.21.15.32:0/3939843059 wait complete. 2024-08-03T20:47:41.218 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:41.217+0000 7fb983600640 1 Processor -- start 2024-08-03T20:47:41.219 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:41.217+0000 7fb983600640 1 -- start start 2024-08-03T20:47:41.222 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:41.219+0000 7fb983600640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fb97c06e930 0x7fb97c13dbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:47:41.222 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:41.219+0000 7fb983600640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb97c073e60 0x7fb97c1f8fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:47:41.222 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:41.219+0000 7fb983600640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb97c0747b0 0x7fb97c153860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:47:41.222 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:41.219+0000 7fb983600640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fb97c0cf9c0 con 0x7fb97c073e60 2024-08-03T20:47:41.222 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:41.219+0000 7fb983600640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7fb97c0cf840 con 0x7fb97c0747b0 2024-08-03T20:47:41.223 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:41.219+0000 7fb983600640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7fb97c0ccff0 con 0x7fb97c06e930 2024-08-03T20:47:41.223 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:41.219+0000 7fb97be00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb97c073e60 0x7fb97c1f8fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:47:41.223 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:41.219+0000 7fb97be00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb97c073e60 0x7fb97c1f8fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:57270/0 (socket says 172.21.15.32:57270) 2024-08-03T20:47:41.223 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:41.219+0000 7fb97be00640 1 -- 172.21.15.32:0/3546718213 learned_addr learned my addr 172.21.15.32:0/3546718213 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:47:41.223 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:41.219+0000 7fb980c00640 1 --2- 172.21.15.32:0/3546718213 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fb97c06e930 0x7fb97c13dbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:47:41.223 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:41.219+0000 7fb981600640 1 --2- 172.21.15.32:0/3546718213 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb97c0747b0 0x7fb97c153860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:47:41.223 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:41.219+0000 7fb97be00640 1 -- 172.21.15.32:0/3546718213 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fb97c06e930 msgr2=0x7fb97c13dbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:47:41.224 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:41.219+0000 7fb97be00640 1 --2- 172.21.15.32:0/3546718213 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fb97c06e930 0x7fb97c13dbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:41.224 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:41.219+0000 7fb97be00640 1 -- 172.21.15.32:0/3546718213 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb97c0747b0 msgr2=0x7fb97c153860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:47:41.224 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:41.219+0000 7fb97be00640 1 --2- 172.21.15.32:0/3546718213 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb97c0747b0 0x7fb97c153860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:41.224 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:41.219+0000 7fb97be00640 1 -- 172.21.15.32:0/3546718213 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb97c1f94f0 con 0x7fb97c073e60 2024-08-03T20:47:41.225 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:41.219+0000 7fb981600640 1 --2- 172.21.15.32:0/3546718213 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb97c0747b0 0x7fb97c153860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-03T20:47:41.225 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:41.220+0000 7fb97be00640 1 --2- 172.21.15.32:0/3546718213 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb97c073e60 0x7fb97c1f8fb0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fb968009950 tx=0x7fb968033c90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:47:41.225 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:41.220+0000 7fb978c00640 1 -- 172.21.15.32:0/3546718213 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb968005c90 con 0x7fb97c073e60 2024-08-03T20:47:41.225 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:41.220+0000 7fb978c00640 1 -- 172.21.15.32:0/3546718213 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fb968031e50 con 0x7fb97c073e60 2024-08-03T20:47:41.225 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:41.220+0000 7fb983600640 1 -- 172.21.15.32:0/3546718213 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb97c1434c0 con 0x7fb97c073e60 2024-08-03T20:47:41.225 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:41.220+0000 7fb978c00640 1 -- 172.21.15.32:0/3546718213 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb968033310 con 0x7fb97c073e60 2024-08-03T20:47:41.225 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:41.220+0000 7fb983600640 1 -- 172.21.15.32:0/3546718213 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb97c143a80 con 0x7fb97c073e60 2024-08-03T20:47:41.226 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:41.221+0000 7fb983600640 1 -- 172.21.15.32:0/3546718213 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb97c073070 con 0x7fb97c073e60 2024-08-03T20:47:41.226 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:41.222+0000 7fb978c00640 1 -- 172.21.15.32:0/3546718213 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7fb9680334b0 con 0x7fb97c073e60 2024-08-03T20:47:41.226 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:41.223+0000 7fb978c00640 1 --2- 172.21.15.32:0/3546718213 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fb95c077040 0x7fb95c079550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:47:41.226 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:41.223+0000 7fb978c00640 1 -- 172.21.15.32:0/3546718213 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7fb9680b9cf0 con 0x7fb97c073e60 2024-08-03T20:47:41.226 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:41.224+0000 7fb980c00640 1 --2- 172.21.15.32:0/3546718213 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fb95c077040 0x7fb95c079550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:47:41.226 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:41.224+0000 7fb980c00640 1 --2- 172.21.15.32:0/3546718213 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fb95c077040 0x7fb95c079550 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fb970005fd0 tx=0x7fb970005c50 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:47:41.229 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:41.228+0000 7fb978c00640 1 -- 172.21.15.32:0/3546718213 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7fb97c073070 con 0x7fb97c073e60 2024-08-03T20:47:41.436 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:41.435+0000 7fb983600640 1 -- 172.21.15.32:0/3546718213 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "config log", "num": 1, "format": "json"} v 0) -- 0x7fb97c0cd530 con 0x7fb97c073e60 2024-08-03T20:47:41.436 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:41.435+0000 7fb978c00640 1 -- 172.21.15.32:0/3546718213 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "config log", "num": 1, "format": "json"}]=0 v27) ==== 86+0+160 (secure 0 0 0) 0x7fb97c1f94f0 con 0x7fb97c073e60 2024-08-03T20:47:41.437 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-03T20:47:41.439 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:41.437+0000 7fb983600640 1 -- 172.21.15.32:0/3546718213 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fb95c077040 msgr2=0x7fb95c079550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:47:41.439 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:41.437+0000 7fb983600640 1 --2- 172.21.15.32:0/3546718213 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fb95c077040 0x7fb95c079550 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fb970005fd0 tx=0x7fb970005c50 comp rx=0 tx=0).stop 2024-08-03T20:47:41.439 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:41.437+0000 7fb983600640 1 -- 172.21.15.32:0/3546718213 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb97c073e60 msgr2=0x7fb97c1f8fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:47:41.439 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:41.437+0000 7fb983600640 1 --2- 172.21.15.32:0/3546718213 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb97c073e60 0x7fb97c1f8fb0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fb968009950 tx=0x7fb968033c90 comp rx=0 tx=0).stop 2024-08-03T20:47:41.439 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:41.437+0000 7fb983600640 1 -- 172.21.15.32:0/3546718213 shutdown_connections 2024-08-03T20:47:41.439 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:41.437+0000 7fb983600640 1 --2- 172.21.15.32:0/3546718213 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fb95c077040 0x7fb95c079550 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:41.439 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:41.437+0000 7fb983600640 1 --2- 172.21.15.32:0/3546718213 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb97c0747b0 0x7fb97c153860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:41.440 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:41.437+0000 7fb983600640 1 --2- 172.21.15.32:0/3546718213 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb97c073e60 0x7fb97c1f8fb0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:41.440 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:41.437+0000 7fb983600640 1 --2- 172.21.15.32:0/3546718213 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fb97c06e930 0x7fb97c13dbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:41.440 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:41.437+0000 7fb983600640 1 -- 172.21.15.32:0/3546718213 >> 172.21.15.32:0/3546718213 conn(0x7fb97c0b1bd0 msgr2=0x7fb97c0b1470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:47:41.440 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:41.438+0000 7fb983600640 1 -- 172.21.15.32:0/3546718213 shutdown_connections 2024-08-03T20:47:41.440 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:41.438+0000 7fb983600640 1 -- 172.21.15.32:0/3546718213 wait complete. 2024-08-03T20:47:41.460 INFO:teuthology.orchestra.run.smithi032.stdout:[{"version":27,"timestamp":"2024-08-03T20:47:28.011925+0000","name":"","changes":[{"name":"osd.5/osd_mclock_max_capacity_iops_ssd","new_value":"3764.678226"}]}] 2024-08-03T20:47:41.461 INFO:tasks.ceph_manager:config epoch is 27 2024-08-03T20:47:41.461 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-08-03T20:47:41.461 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2024-08-03T20:47:41.461 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2024-08-03T20:47:41.689 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:41.687+0000 7f662a000640 1 -- 172.21.15.32:0/3126143558 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f662415f610 msgr2=0x7f66241523f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:47:41.689 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:41.687+0000 7f662a000640 1 --2- 172.21.15.32:0/3126143558 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f662415f610 0x7f66241523f0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f6610005fd0 tx=0x7f661002a5d0 comp rx=0 tx=0).stop 2024-08-03T20:47:41.689 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:41.687+0000 7f662a000640 1 -- 172.21.15.32:0/3126143558 shutdown_connections 2024-08-03T20:47:41.689 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:41.687+0000 7f662a000640 1 --2- 172.21.15.32:0/3126143558 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f66240b4ce0 0x7f66240cdf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:41.690 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:41.687+0000 7f662a000640 1 --2- 172.21.15.32:0/3126143558 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f662415f610 0x7f66241523f0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:41.690 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:41.687+0000 7f662a000640 1 -- 172.21.15.32:0/3126143558 >> 172.21.15.32:0/3126143558 conn(0x7f66240b1bd0 msgr2=0x7f66240b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:47:41.690 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:41.688+0000 7f662a000640 1 -- 172.21.15.32:0/3126143558 shutdown_connections 2024-08-03T20:47:41.690 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:41.688+0000 7f662a000640 1 -- 172.21.15.32:0/3126143558 wait complete. 2024-08-03T20:47:41.691 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:41.690+0000 7f662a000640 1 Processor -- start 2024-08-03T20:47:41.691 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:41.690+0000 7f662a000640 1 -- start start 2024-08-03T20:47:41.694 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:41.691+0000 7f662a000640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f66240b38b0 0x7f662414f4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:47:41.695 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:41.691+0000 7f662a000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f66240b4ce0 0x7f66241472f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:47:41.695 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:41.691+0000 7f662a000640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f66241589e0 0x7f662406f1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:47:41.695 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:41.691+0000 7f662a000640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f66240cea90 con 0x7f66240b4ce0 2024-08-03T20:47:41.695 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:41.691+0000 7f662a000640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f66240ce910 con 0x7f66240b38b0 2024-08-03T20:47:41.695 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:41.691+0000 7f662a000640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7f66240cec10 con 0x7f66241589e0 2024-08-03T20:47:41.695 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:41.692+0000 7f6622a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f66240b4ce0 0x7f66241472f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:47:41.695 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:41.692+0000 7f6622a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f66240b4ce0 0x7f66241472f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:57282/0 (socket says 172.21.15.32:57282) 2024-08-03T20:47:41.695 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:41.692+0000 7f6622a00640 1 -- 172.21.15.32:0/2370181089 learned_addr learned my addr 172.21.15.32:0/2370181089 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:47:41.695 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:41.692+0000 7f6623400640 1 --2- 172.21.15.32:0/2370181089 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f66240b38b0 0x7f662414f4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:47:41.695 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:41.692+0000 7f6623e00640 1 --2- 172.21.15.32:0/2370181089 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f66241589e0 0x7f662406f1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:47:41.695 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:41.692+0000 7f6622a00640 1 -- 172.21.15.32:0/2370181089 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f66241589e0 msgr2=0x7f662406f1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:47:41.695 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:41.692+0000 7f6622a00640 1 --2- 172.21.15.32:0/2370181089 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f66241589e0 0x7f662406f1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:41.695 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:41.692+0000 7f6622a00640 1 -- 172.21.15.32:0/2370181089 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f66240b38b0 msgr2=0x7f662414f4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:47:41.696 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:41.692+0000 7f6622a00640 1 --2- 172.21.15.32:0/2370181089 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f66240b38b0 0x7f662414f4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:41.696 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:41.692+0000 7f6622a00640 1 -- 172.21.15.32:0/2370181089 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f662414fce0 con 0x7f66240b4ce0 2024-08-03T20:47:41.696 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:41.692+0000 7f6622a00640 1 --2- 172.21.15.32:0/2370181089 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f66240b4ce0 0x7f66241472f0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f6610006010 tx=0x7f66100038a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:47:41.696 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:41.693+0000 7f661b400640 1 -- 172.21.15.32:0/2370181089 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f661003e030 con 0x7f66240b4ce0 2024-08-03T20:47:41.696 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:41.693+0000 7f661b400640 1 -- 172.21.15.32:0/2370181089 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f6610003ca0 con 0x7f66240b4ce0 2024-08-03T20:47:41.696 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:41.693+0000 7f662a000640 1 -- 172.21.15.32:0/2370181089 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f662414ffc0 con 0x7f66240b4ce0 2024-08-03T20:47:41.696 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:41.693+0000 7f661b400640 1 -- 172.21.15.32:0/2370181089 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6610035040 con 0x7f66240b4ce0 2024-08-03T20:47:41.696 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:41.693+0000 7f662a000640 1 -- 172.21.15.32:0/2370181089 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f662414d990 con 0x7f66240b4ce0 2024-08-03T20:47:41.696 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:41.694+0000 7f662a000640 1 -- 172.21.15.32:0/2370181089 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f66241504e0 con 0x7f66240b4ce0 2024-08-03T20:47:41.697 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:41.695+0000 7f661b400640 1 -- 172.21.15.32:0/2370181089 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f6610041030 con 0x7f66240b4ce0 2024-08-03T20:47:41.697 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:41.696+0000 7f661b400640 1 --2- 172.21.15.32:0/2370181089 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f6604077010 0x7f6604079520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:47:41.697 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:41.696+0000 7f661b400640 1 -- 172.21.15.32:0/2370181089 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f66100b8ea0 con 0x7f66240b4ce0 2024-08-03T20:47:41.697 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:41.696+0000 7f6623400640 1 --2- 172.21.15.32:0/2370181089 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f6604077010 0x7f6604079520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:47:41.698 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:41.697+0000 7f6623400640 1 --2- 172.21.15.32:0/2370181089 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f6604077010 0x7f6604079520 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f6614001720 tx=0x7f6614009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:47:41.701 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:41.700+0000 7f661b400640 1 -- 172.21.15.32:0/2370181089 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7f66241504e0 con 0x7f66240b4ce0 2024-08-03T20:47:41.956 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:41.954+0000 7f662a000640 1 -- 172.21.15.32:0/2370181089 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) -- 0x7f662414fa70 con 0x7f66240b4ce0 2024-08-03T20:47:41.958 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:41.957+0000 7f661b400640 1 -- 172.21.15.32:0/2370181089 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v5) ==== 74+0+191109 (secure 0 0 0) 0x7f66100819b0 con 0x7f66240b4ce0 2024-08-03T20:47:41.961 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-03T20:47:41.963 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:41.960+0000 7f662a000640 1 -- 172.21.15.32:0/2370181089 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f6604077010 msgr2=0x7f6604079520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:47:41.963 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:41.960+0000 7f662a000640 1 --2- 172.21.15.32:0/2370181089 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f6604077010 0x7f6604079520 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f6614001720 tx=0x7f6614009040 comp rx=0 tx=0).stop 2024-08-03T20:47:41.963 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:41.960+0000 7f662a000640 1 -- 172.21.15.32:0/2370181089 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f66240b4ce0 msgr2=0x7f66241472f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:47:41.964 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:41.960+0000 7f662a000640 1 --2- 172.21.15.32:0/2370181089 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f66240b4ce0 0x7f66241472f0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f6610006010 tx=0x7f66100038a0 comp rx=0 tx=0).stop 2024-08-03T20:47:41.964 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:41.961+0000 7f662a000640 1 -- 172.21.15.32:0/2370181089 shutdown_connections 2024-08-03T20:47:41.964 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:41.961+0000 7f662a000640 1 --2- 172.21.15.32:0/2370181089 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f6604077010 0x7f6604079520 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:41.964 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:41.961+0000 7f662a000640 1 --2- 172.21.15.32:0/2370181089 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f66241589e0 0x7f662406f1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:41.964 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:41.961+0000 7f662a000640 1 --2- 172.21.15.32:0/2370181089 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f66240b4ce0 0x7f66241472f0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:41.964 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:41.961+0000 7f662a000640 1 --2- 172.21.15.32:0/2370181089 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f66240b38b0 0x7f662414f4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:41.964 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:41.961+0000 7f662a000640 1 -- 172.21.15.32:0/2370181089 >> 172.21.15.32:0/2370181089 conn(0x7f66240b1bd0 msgr2=0x7f662415abe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:47:41.964 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:41.961+0000 7f662a000640 1 -- 172.21.15.32:0/2370181089 shutdown_connections 2024-08-03T20:47:41.964 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:41.961+0000 7f662a000640 1 -- 172.21.15.32:0/2370181089 wait complete. 2024-08-03T20:47:41.983 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":5,"flags":0,"active_gid":4106,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6800","nonce":4205950699},{"type":"v1","addr":"172.21.15.118:6801","nonce":4205950699}]},"active_addr":"172.21.15.118:6801/4205950699","active_change":"2024-08-03T20:47:20.206038+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[{"gid":4102,"name":"y","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.27.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:10.4.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_nginx":{"name":"container_image_nginx","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nginx:1.26.1","min":"","max":"","enum_allowed":[],"desc":"Nginx container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.7.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.5","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.51.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"secondary_zone_period_retry_limit":{"name":"secondary_zone_period_retry_limit","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"RGW module period update retry limit for secondary site","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"smb","can_run":true,"error_string":"","module_options":{"internal_store_backend":{"name":"internal_store_backend","type":"str","level":"dev","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"set internal store backend. for develoment and testing only","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"update_orchestration":{"name":"update_orchestration","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically update orchestration when smb resources are changed","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.27.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:10.4.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_nginx":{"name":"container_image_nginx","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nginx:1.26.1","min":"","max":"","enum_allowed":[],"desc":"Nginx container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.7.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.5","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.51.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"secondary_zone_period_retry_limit":{"name":"secondary_zone_period_retry_limit","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"RGW module period update retry limit for secondary site","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"smb","can_run":true,"error_string":"","module_options":{"internal_store_backend":{"name":"internal_store_backend","type":"str","level":"dev","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"set internal store backend. for develoment and testing only","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"update_orchestration":{"name":"update_orchestration","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically update orchestration when smb resources are changed","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.118:0","nonce":2721355872}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.118:0","nonce":1553241256}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.118:0","nonce":643517886}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.118:0","nonce":1846113063}]}]} 2024-08-03T20:47:41.987 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2024-08-03T20:47:41.988 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2024-08-03T20:47:41.988 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-08-03T20:47:42.216 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:42.214+0000 7f53adc00640 1 -- 172.21.15.32:0/2147631231 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f53a80b7760 msgr2=0x7f53a80cdba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:47:42.216 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:42.214+0000 7f53adc00640 1 --2- 172.21.15.32:0/2147631231 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f53a80b7760 0x7f53a80cdba0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f53a80742d0 tx=0x7f539802a3f0 comp rx=0 tx=0).stop 2024-08-03T20:47:42.216 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:42.214+0000 7f53adc00640 1 -- 172.21.15.32:0/2147631231 shutdown_connections 2024-08-03T20:47:42.217 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:42.214+0000 7f53adc00640 1 --2- 172.21.15.32:0/2147631231 >> v2:172.21.15.118:3300/0 conn(0x7f53a80b29f0 0x7f53a809b040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:42.217 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:42.214+0000 7f53adc00640 1 --2- 172.21.15.32:0/2147631231 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f53a80b7760 0x7f53a80cdba0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:42.217 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:42.214+0000 7f53adc00640 1 -- 172.21.15.32:0/2147631231 >> 172.21.15.32:0/2147631231 conn(0x7f53a80b3bb0 msgr2=0x7f53a80b3fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:47:42.217 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:42.214+0000 7f53adc00640 1 -- 172.21.15.32:0/2147631231 shutdown_connections 2024-08-03T20:47:42.217 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:42.214+0000 7f53adc00640 1 -- 172.21.15.32:0/2147631231 wait complete. 2024-08-03T20:47:42.218 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:42.217+0000 7f53adc00640 1 Processor -- start 2024-08-03T20:47:42.218 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:42.217+0000 7f53adc00640 1 -- start start 2024-08-03T20:47:42.221 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:42.218+0000 7f53adc00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f53a809ac30 0x7f53a8162a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:47:42.221 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:42.218+0000 7f53adc00640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f53a80b29f0 0x7f53a80b63d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:47:42.221 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:42.218+0000 7f53adc00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f53a80b7760 0x7f53a8070b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:47:42.221 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:42.218+0000 7f53adc00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f53a80d0730 con 0x7f53a80b7760 2024-08-03T20:47:42.222 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:42.218+0000 7f53adc00640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f53a80d05b0 con 0x7f53a809ac30 2024-08-03T20:47:42.222 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:42.218+0000 7f53adc00640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7f53a80d08b0 con 0x7f53a80b29f0 2024-08-03T20:47:42.222 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:42.219+0000 7f53a7400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f53a80b7760 0x7f53a8070b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:47:42.222 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:42.219+0000 7f53a7400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f53a80b7760 0x7f53a8070b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:57288/0 (socket says 172.21.15.32:57288) 2024-08-03T20:47:42.222 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:42.219+0000 7f53a7400640 1 -- 172.21.15.32:0/1834878253 learned_addr learned my addr 172.21.15.32:0/1834878253 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:47:42.222 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:42.219+0000 7f53a6a00640 1 --2- 172.21.15.32:0/1834878253 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f53a809ac30 0x7f53a8162a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:47:42.222 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:42.219+0000 7f53a6000640 1 --2- 172.21.15.32:0/1834878253 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f53a80b29f0 0x7f53a80b63d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:47:42.222 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:42.219+0000 7f53a7400640 1 -- 172.21.15.32:0/1834878253 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f53a80b29f0 msgr2=0x7f53a80b63d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:47:42.222 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:42.219+0000 7f53a7400640 1 --2- 172.21.15.32:0/1834878253 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f53a80b29f0 0x7f53a80b63d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:42.223 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:42.219+0000 7f53a7400640 1 -- 172.21.15.32:0/1834878253 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f53a809ac30 msgr2=0x7f53a8162a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:47:42.223 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:42.219+0000 7f53a7400640 1 --2- 172.21.15.32:0/1834878253 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f53a809ac30 0x7f53a8162a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:42.223 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:42.219+0000 7f53a7400640 1 -- 172.21.15.32:0/1834878253 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f53a80b6910 con 0x7f53a80b7760 2024-08-03T20:47:42.223 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:42.219+0000 7f53a6a00640 1 --2- 172.21.15.32:0/1834878253 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f53a809ac30 0x7f53a8162a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-03T20:47:42.223 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:42.219+0000 7f53a7400640 1 --2- 172.21.15.32:0/1834878253 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f53a80b7760 0x7f53a8070b10 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f53a000e9c0 tx=0x7f53a000ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:47:42.223 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:42.220+0000 7f539ea00640 1 -- 172.21.15.32:0/1834878253 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53a0004d20 con 0x7f53a80b7760 2024-08-03T20:47:42.223 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:42.220+0000 7f539ea00640 1 -- 172.21.15.32:0/1834878253 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f53a000cb20 con 0x7f53a80b7760 2024-08-03T20:47:42.223 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:42.220+0000 7f53adc00640 1 -- 172.21.15.32:0/1834878253 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f53a80b6e20 con 0x7f53a80b7760 2024-08-03T20:47:42.223 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:42.220+0000 7f539ea00640 1 -- 172.21.15.32:0/1834878253 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53a0005020 con 0x7f53a80b7760 2024-08-03T20:47:42.223 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:42.220+0000 7f53adc00640 1 -- 172.21.15.32:0/1834878253 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f53a80b73e0 con 0x7f53a80b7760 2024-08-03T20:47:42.224 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:42.221+0000 7f53adc00640 1 -- 172.21.15.32:0/1834878253 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f53a80b6b90 con 0x7f53a80b7760 2024-08-03T20:47:42.224 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:42.222+0000 7f539ea00640 1 -- 172.21.15.32:0/1834878253 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f53a00051c0 con 0x7f53a80b7760 2024-08-03T20:47:42.225 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:42.223+0000 7f539ea00640 1 --2- 172.21.15.32:0/1834878253 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f5388077040 0x7f5388079550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:47:42.225 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:42.223+0000 7f539ea00640 1 -- 172.21.15.32:0/1834878253 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f53a0099120 con 0x7f53a80b7760 2024-08-03T20:47:42.225 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:42.223+0000 7f53a6a00640 1 --2- 172.21.15.32:0/1834878253 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f5388077040 0x7f5388079550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:47:42.225 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:42.224+0000 7f53a6a00640 1 --2- 172.21.15.32:0/1834878253 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f5388077040 0x7f5388079550 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f5398034000 tx=0x7f5398004730 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:47:42.228 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:42.227+0000 7f539ea00640 1 -- 172.21.15.32:0/1834878253 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7f53a0016030 con 0x7f53a80b7760 2024-08-03T20:47:42.440 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:42.438+0000 7f53adc00640 1 -- 172.21.15.32:0/1834878253 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f53a809b040 con 0x7f53a80b7760 2024-08-03T20:47:42.441 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:42.440+0000 7f539ea00640 1 -- 172.21.15.32:0/1834878253 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) ==== 74+0+16545 (secure 0 0 0) 0x7f53a0062310 con 0x7f53a80b7760 2024-08-03T20:47:42.441 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-03T20:47:42.442 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":21,"fsid":"40ab141a-01d7-49b9-ae3c-2edec9876e95","created":"2024-08-03T20:47:14.957362+0000","modified":"2024-08-03T20:47:36.927196+0000","last_up_change":"2024-08-03T20:47:28.874451+0000","last_in_change":"2024-08-03T20:47:20.079442+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-08-03T20:47:28.261300+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-08-03T20:47:33.085381+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-08-03T20:47:34.405408+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5a83e759-9035-438b-889a-0a6af1f0a0d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6816","nonce":3066156116},{"type":"v1","addr":"172.21.15.32:6817","nonce":3066156116}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6818","nonce":3066156116},{"type":"v1","addr":"172.21.15.32:6819","nonce":3066156116}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6822","nonce":3066156116},{"type":"v1","addr":"172.21.15.32:6823","nonce":3066156116}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6820","nonce":3066156116},{"type":"v1","addr":"172.21.15.32:6821","nonce":3066156116}]},"public_addr":"172.21.15.32:6817/3066156116","cluster_addr":"172.21.15.32:6819/3066156116","heartbeat_back_addr":"172.21.15.32:6823/3066156116","heartbeat_front_addr":"172.21.15.32:6821/3066156116","state":["exists","up"]},{"osd":1,"uuid":"d7c5c210-da4f-49f9-b9e7-1dfb02daa4bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6824","nonce":1551241759},{"type":"v1","addr":"172.21.15.32:6825","nonce":1551241759}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6826","nonce":1551241759},{"type":"v1","addr":"172.21.15.32:6827","nonce":1551241759}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6830","nonce":1551241759},{"type":"v1","addr":"172.21.15.32:6831","nonce":1551241759}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6828","nonce":1551241759},{"type":"v1","addr":"172.21.15.32:6829","nonce":1551241759}]},"public_addr":"172.21.15.32:6825/1551241759","cluster_addr":"172.21.15.32:6827/1551241759","heartbeat_back_addr":"172.21.15.32:6831/1551241759","heartbeat_front_addr":"172.21.15.32:6829/1551241759","state":["exists","up"]},{"osd":2,"uuid":"57f2e6de-a201-45c2-8981-8e56a4875ebd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6808","nonce":3806708944},{"type":"v1","addr":"172.21.15.32:6809","nonce":3806708944}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6810","nonce":3806708944},{"type":"v1","addr":"172.21.15.32:6811","nonce":3806708944}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6814","nonce":3806708944},{"type":"v1","addr":"172.21.15.32:6815","nonce":3806708944}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6812","nonce":3806708944},{"type":"v1","addr":"172.21.15.32:6813","nonce":3806708944}]},"public_addr":"172.21.15.32:6809/3806708944","cluster_addr":"172.21.15.32:6811/3806708944","heartbeat_back_addr":"172.21.15.32:6815/3806708944","heartbeat_front_addr":"172.21.15.32:6813/3806708944","state":["exists","up"]},{"osd":3,"uuid":"6439a280-da45-4148-8130-f16323935ff3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6800","nonce":9088303},{"type":"v1","addr":"172.21.15.32:6801","nonce":9088303}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6802","nonce":9088303},{"type":"v1","addr":"172.21.15.32:6803","nonce":9088303}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6806","nonce":9088303},{"type":"v1","addr":"172.21.15.32:6807","nonce":9088303}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6804","nonce":9088303},{"type":"v1","addr":"172.21.15.32:6805","nonce":9088303}]},"public_addr":"172.21.15.32:6801/9088303","cluster_addr":"172.21.15.32:6803/9088303","heartbeat_back_addr":"172.21.15.32:6807/9088303","heartbeat_front_addr":"172.21.15.32:6805/9088303","state":["exists","up"]},{"osd":4,"uuid":"4f902eff-1e18-418d-82c9-2151c69524fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6802","nonce":864752275},{"type":"v1","addr":"172.21.15.118:6803","nonce":864752275}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6804","nonce":864752275},{"type":"v1","addr":"172.21.15.118:6805","nonce":864752275}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6808","nonce":864752275},{"type":"v1","addr":"172.21.15.118:6809","nonce":864752275}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6806","nonce":864752275},{"type":"v1","addr":"172.21.15.118:6807","nonce":864752275}]},"public_addr":"172.21.15.118:6803/864752275","cluster_addr":"172.21.15.118:6805/864752275","heartbeat_back_addr":"172.21.15.118:6809/864752275","heartbeat_front_addr":"172.21.15.118:6807/864752275","state":["exists","up"]},{"osd":5,"uuid":"bfd9feac-dd06-458c-91b9-af7f175f612c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6810","nonce":2724891770},{"type":"v1","addr":"172.21.15.118:6811","nonce":2724891770}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6812","nonce":2724891770},{"type":"v1","addr":"172.21.15.118:6813","nonce":2724891770}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6816","nonce":2724891770},{"type":"v1","addr":"172.21.15.118:6817","nonce":2724891770}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6814","nonce":2724891770},{"type":"v1","addr":"172.21.15.118:6815","nonce":2724891770}]},"public_addr":"172.21.15.118:6811/2724891770","cluster_addr":"172.21.15.118:6813/2724891770","heartbeat_back_addr":"172.21.15.118:6817/2724891770","heartbeat_front_addr":"172.21.15.118:6815/2724891770","state":["exists","up"]},{"osd":6,"uuid":"5ed5a279-9d8e-416d-bbc7-7edd3e6d5705","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6825","nonce":801947386},{"type":"v1","addr":"172.21.15.118:6827","nonce":801947386}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6828","nonce":801947386},{"type":"v1","addr":"172.21.15.118:6829","nonce":801947386}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6832","nonce":801947386},{"type":"v1","addr":"172.21.15.118:6833","nonce":801947386}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6830","nonce":801947386},{"type":"v1","addr":"172.21.15.118:6831","nonce":801947386}]},"public_addr":"172.21.15.118:6827/801947386","cluster_addr":"172.21.15.118:6829/801947386","heartbeat_back_addr":"172.21.15.118:6833/801947386","heartbeat_front_addr":"172.21.15.118:6831/801947386","state":["exists","up"]},{"osd":7,"uuid":"f2086c47-875e-4451-a436-2008d1ea16af","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6818","nonce":653000048},{"type":"v1","addr":"172.21.15.118:6819","nonce":653000048}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6820","nonce":653000048},{"type":"v1","addr":"172.21.15.118:6821","nonce":653000048}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6824","nonce":653000048},{"type":"v1","addr":"172.21.15.118:6826","nonce":653000048}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6822","nonce":653000048},{"type":"v1","addr":"172.21.15.118:6823","nonce":653000048}]},"public_addr":"172.21.15.118:6819/653000048","cluster_addr":"172.21.15.118:6821/653000048","heartbeat_back_addr":"172.21.15.118:6826/653000048","heartbeat_front_addr":"172.21.15.118:6823/653000048","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-03T20:47:25.491994+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-03T20:47:25.587358+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-03T20:47:25.454175+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-03T20:47:25.377734+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-03T20:47:25.412351+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-03T20:47:25.499815+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-03T20:47:25.575044+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-03T20:47:25.552383+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-08-03T20:47:42.444 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:42.442+0000 7f53adc00640 1 -- 172.21.15.32:0/1834878253 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f5388077040 msgr2=0x7f5388079550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:47:42.444 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:42.442+0000 7f53adc00640 1 --2- 172.21.15.32:0/1834878253 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f5388077040 0x7f5388079550 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f5398034000 tx=0x7f5398004730 comp rx=0 tx=0).stop 2024-08-03T20:47:42.444 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:42.442+0000 7f53adc00640 1 -- 172.21.15.32:0/1834878253 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f53a80b7760 msgr2=0x7f53a8070b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:47:42.444 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:42.442+0000 7f53adc00640 1 --2- 172.21.15.32:0/1834878253 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f53a80b7760 0x7f53a8070b10 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f53a000e9c0 tx=0x7f53a000ee80 comp rx=0 tx=0).stop 2024-08-03T20:47:42.444 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:42.442+0000 7f53adc00640 1 -- 172.21.15.32:0/1834878253 shutdown_connections 2024-08-03T20:47:42.444 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:42.442+0000 7f53adc00640 1 --2- 172.21.15.32:0/1834878253 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f5388077040 0x7f5388079550 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:42.444 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:42.442+0000 7f53adc00640 1 --2- 172.21.15.32:0/1834878253 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f53a80b7760 0x7f53a8070b10 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:42.445 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:42.442+0000 7f53adc00640 1 --2- 172.21.15.32:0/1834878253 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f53a80b29f0 0x7f53a80b63d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:42.445 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:42.442+0000 7f53adc00640 1 --2- 172.21.15.32:0/1834878253 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f53a809ac30 0x7f53a8162a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:42.445 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:42.442+0000 7f53adc00640 1 -- 172.21.15.32:0/1834878253 >> 172.21.15.32:0/1834878253 conn(0x7f53a80b3bb0 msgr2=0x7f53a80d0ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:47:42.445 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:42.442+0000 7f53adc00640 1 -- 172.21.15.32:0/1834878253 shutdown_connections 2024-08-03T20:47:42.445 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:42.443+0000 7f53adc00640 1 -- 172.21.15.32:0/1834878253 wait complete. 2024-08-03T20:47:42.466 INFO:tasks.ceph.ceph_manager.ceph:all up! 2024-08-03T20:47:42.466 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-08-03T20:47:42.693 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:42.692+0000 7f6c46a00640 1 -- 172.21.15.32:0/1012100304 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6c40144f90 msgr2=0x7f6c40074570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:47:42.693 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:42.692+0000 7f6c46a00640 1 --2- 172.21.15.32:0/1012100304 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6c40144f90 0x7f6c40074570 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f6c2c009830 tx=0x7f6c2c02a510 comp rx=0 tx=0).stop 2024-08-03T20:47:42.694 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:42.692+0000 7f6c46a00640 1 -- 172.21.15.32:0/1012100304 shutdown_connections 2024-08-03T20:47:42.694 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:42.692+0000 7f6c46a00640 1 --2- 172.21.15.32:0/1012100304 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f6c4015fff0 0x7f6c40160400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:42.694 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:42.692+0000 7f6c46a00640 1 --2- 172.21.15.32:0/1012100304 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6c40144f90 0x7f6c40074570 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:42.694 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:42.692+0000 7f6c46a00640 1 -- 172.21.15.32:0/1012100304 >> 172.21.15.32:0/1012100304 conn(0x7f6c400b1bd0 msgr2=0x7f6c400b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:47:42.694 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:42.692+0000 7f6c46a00640 1 -- 172.21.15.32:0/1012100304 shutdown_connections 2024-08-03T20:47:42.694 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:42.692+0000 7f6c46a00640 1 -- 172.21.15.32:0/1012100304 wait complete. 2024-08-03T20:47:42.695 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:42.694+0000 7f6c46a00640 1 Processor -- start 2024-08-03T20:47:42.695 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:42.694+0000 7f6c46a00640 1 -- start start 2024-08-03T20:47:42.698 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:42.695+0000 7f6c46a00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f6c4006e930 0x7f6c401f66f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:47:42.698 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:42.695+0000 7f6c46a00640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f6c40144f90 0x7f6c4013d960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:47:42.698 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:42.695+0000 7f6c46a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6c4015fff0 0x7f6c4006ef70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:47:42.698 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:42.695+0000 7f6c46a00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f6c400cf840 con 0x7f6c4015fff0 2024-08-03T20:47:42.698 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:42.695+0000 7f6c46a00640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f6c40145f30 con 0x7f6c4006e930 2024-08-03T20:47:42.698 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:42.695+0000 7f6c46a00640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7f6c400cf9c0 con 0x7f6c40144f90 2024-08-03T20:47:42.699 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:42.696+0000 7f6c44a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6c4015fff0 0x7f6c4006ef70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:47:42.699 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:42.696+0000 7f6c44a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6c4015fff0 0x7f6c4006ef70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:57308/0 (socket says 172.21.15.32:57308) 2024-08-03T20:47:42.699 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:42.696+0000 7f6c44a00640 1 -- 172.21.15.32:0/3788116375 learned_addr learned my addr 172.21.15.32:0/3788116375 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:47:42.699 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:42.696+0000 7f6c3f400640 1 --2- 172.21.15.32:0/3788116375 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f6c40144f90 0x7f6c4013d960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:47:42.699 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:42.696+0000 7f6c3fe00640 1 --2- 172.21.15.32:0/3788116375 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f6c4006e930 0x7f6c401f66f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:47:42.699 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:42.696+0000 7f6c44a00640 1 -- 172.21.15.32:0/3788116375 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f6c40144f90 msgr2=0x7f6c4013d960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:47:42.699 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:42.696+0000 7f6c44a00640 1 --2- 172.21.15.32:0/3788116375 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f6c40144f90 0x7f6c4013d960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:42.699 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:42.696+0000 7f6c44a00640 1 -- 172.21.15.32:0/3788116375 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f6c4006e930 msgr2=0x7f6c401f66f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:47:42.699 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:42.696+0000 7f6c44a00640 1 --2- 172.21.15.32:0/3788116375 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f6c4006e930 0x7f6c401f66f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:42.699 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:42.696+0000 7f6c44a00640 1 -- 172.21.15.32:0/3788116375 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6c401f7030 con 0x7f6c4015fff0 2024-08-03T20:47:42.699 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:42.696+0000 7f6c3fe00640 1 --2- 172.21.15.32:0/3788116375 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f6c4006e930 0x7f6c401f66f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-03T20:47:42.699 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:42.696+0000 7f6c44a00640 1 --2- 172.21.15.32:0/3788116375 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6c4015fff0 0x7f6c4006ef70 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f6c3800ea80 tx=0x7f6c3800ef40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:47:42.699 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:42.697+0000 7f6c37e00640 1 -- 172.21.15.32:0/3788116375 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c38004b20 con 0x7f6c4015fff0 2024-08-03T20:47:42.700 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:42.697+0000 7f6c37e00640 1 -- 172.21.15.32:0/3788116375 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f6c38004ce0 con 0x7f6c4015fff0 2024-08-03T20:47:42.700 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:42.697+0000 7f6c37e00640 1 -- 172.21.15.32:0/3788116375 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c38005280 con 0x7f6c4015fff0 2024-08-03T20:47:42.700 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:42.697+0000 7f6c46a00640 1 -- 172.21.15.32:0/3788116375 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6c401f7310 con 0x7f6c4015fff0 2024-08-03T20:47:42.700 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:42.697+0000 7f6c46a00640 1 -- 172.21.15.32:0/3788116375 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6c400ca290 con 0x7f6c4015fff0 2024-08-03T20:47:42.700 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:42.698+0000 7f6c46a00640 1 -- 172.21.15.32:0/3788116375 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6c401f6da0 con 0x7f6c4015fff0 2024-08-03T20:47:42.700 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:42.699+0000 7f6c37e00640 1 -- 172.21.15.32:0/3788116375 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f6c38005420 con 0x7f6c4015fff0 2024-08-03T20:47:42.701 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:42.700+0000 7f6c37e00640 1 --2- 172.21.15.32:0/3788116375 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f6c20077010 0x7f6c20079520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:47:42.701 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:42.700+0000 7f6c37e00640 1 -- 172.21.15.32:0/3788116375 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f6c380995b0 con 0x7f6c4015fff0 2024-08-03T20:47:42.702 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:42.700+0000 7f6c3fe00640 1 --2- 172.21.15.32:0/3788116375 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f6c20077010 0x7f6c20079520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:47:42.702 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:42.701+0000 7f6c3fe00640 1 --2- 172.21.15.32:0/3788116375 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f6c20077010 0x7f6c20079520 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f6c30000f60 tx=0x7f6c30009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:47:42.705 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:42.704+0000 7f6c37e00640 1 -- 172.21.15.32:0/3788116375 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7f6c38062710 con 0x7f6c4015fff0 2024-08-03T20:47:42.912 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:42.911+0000 7f6c46a00640 1 -- 172.21.15.32:0/3788116375 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f6c40160cc0 con 0x7f6c4015fff0 2024-08-03T20:47:42.913 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:42.912+0000 7f6c37e00640 1 -- 172.21.15.32:0/3788116375 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) ==== 74+0+16545 (secure 0 0 0) 0x7f6c38062010 con 0x7f6c4015fff0 2024-08-03T20:47:42.913 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-03T20:47:42.914 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":21,"fsid":"40ab141a-01d7-49b9-ae3c-2edec9876e95","created":"2024-08-03T20:47:14.957362+0000","modified":"2024-08-03T20:47:36.927196+0000","last_up_change":"2024-08-03T20:47:28.874451+0000","last_in_change":"2024-08-03T20:47:20.079442+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-08-03T20:47:28.261300+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-08-03T20:47:33.085381+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-08-03T20:47:34.405408+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5a83e759-9035-438b-889a-0a6af1f0a0d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6816","nonce":3066156116},{"type":"v1","addr":"172.21.15.32:6817","nonce":3066156116}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6818","nonce":3066156116},{"type":"v1","addr":"172.21.15.32:6819","nonce":3066156116}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6822","nonce":3066156116},{"type":"v1","addr":"172.21.15.32:6823","nonce":3066156116}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6820","nonce":3066156116},{"type":"v1","addr":"172.21.15.32:6821","nonce":3066156116}]},"public_addr":"172.21.15.32:6817/3066156116","cluster_addr":"172.21.15.32:6819/3066156116","heartbeat_back_addr":"172.21.15.32:6823/3066156116","heartbeat_front_addr":"172.21.15.32:6821/3066156116","state":["exists","up"]},{"osd":1,"uuid":"d7c5c210-da4f-49f9-b9e7-1dfb02daa4bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6824","nonce":1551241759},{"type":"v1","addr":"172.21.15.32:6825","nonce":1551241759}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6826","nonce":1551241759},{"type":"v1","addr":"172.21.15.32:6827","nonce":1551241759}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6830","nonce":1551241759},{"type":"v1","addr":"172.21.15.32:6831","nonce":1551241759}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6828","nonce":1551241759},{"type":"v1","addr":"172.21.15.32:6829","nonce":1551241759}]},"public_addr":"172.21.15.32:6825/1551241759","cluster_addr":"172.21.15.32:6827/1551241759","heartbeat_back_addr":"172.21.15.32:6831/1551241759","heartbeat_front_addr":"172.21.15.32:6829/1551241759","state":["exists","up"]},{"osd":2,"uuid":"57f2e6de-a201-45c2-8981-8e56a4875ebd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6808","nonce":3806708944},{"type":"v1","addr":"172.21.15.32:6809","nonce":3806708944}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6810","nonce":3806708944},{"type":"v1","addr":"172.21.15.32:6811","nonce":3806708944}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6814","nonce":3806708944},{"type":"v1","addr":"172.21.15.32:6815","nonce":3806708944}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6812","nonce":3806708944},{"type":"v1","addr":"172.21.15.32:6813","nonce":3806708944}]},"public_addr":"172.21.15.32:6809/3806708944","cluster_addr":"172.21.15.32:6811/3806708944","heartbeat_back_addr":"172.21.15.32:6815/3806708944","heartbeat_front_addr":"172.21.15.32:6813/3806708944","state":["exists","up"]},{"osd":3,"uuid":"6439a280-da45-4148-8130-f16323935ff3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6800","nonce":9088303},{"type":"v1","addr":"172.21.15.32:6801","nonce":9088303}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6802","nonce":9088303},{"type":"v1","addr":"172.21.15.32:6803","nonce":9088303}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6806","nonce":9088303},{"type":"v1","addr":"172.21.15.32:6807","nonce":9088303}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6804","nonce":9088303},{"type":"v1","addr":"172.21.15.32:6805","nonce":9088303}]},"public_addr":"172.21.15.32:6801/9088303","cluster_addr":"172.21.15.32:6803/9088303","heartbeat_back_addr":"172.21.15.32:6807/9088303","heartbeat_front_addr":"172.21.15.32:6805/9088303","state":["exists","up"]},{"osd":4,"uuid":"4f902eff-1e18-418d-82c9-2151c69524fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6802","nonce":864752275},{"type":"v1","addr":"172.21.15.118:6803","nonce":864752275}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6804","nonce":864752275},{"type":"v1","addr":"172.21.15.118:6805","nonce":864752275}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6808","nonce":864752275},{"type":"v1","addr":"172.21.15.118:6809","nonce":864752275}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6806","nonce":864752275},{"type":"v1","addr":"172.21.15.118:6807","nonce":864752275}]},"public_addr":"172.21.15.118:6803/864752275","cluster_addr":"172.21.15.118:6805/864752275","heartbeat_back_addr":"172.21.15.118:6809/864752275","heartbeat_front_addr":"172.21.15.118:6807/864752275","state":["exists","up"]},{"osd":5,"uuid":"bfd9feac-dd06-458c-91b9-af7f175f612c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6810","nonce":2724891770},{"type":"v1","addr":"172.21.15.118:6811","nonce":2724891770}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6812","nonce":2724891770},{"type":"v1","addr":"172.21.15.118:6813","nonce":2724891770}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6816","nonce":2724891770},{"type":"v1","addr":"172.21.15.118:6817","nonce":2724891770}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6814","nonce":2724891770},{"type":"v1","addr":"172.21.15.118:6815","nonce":2724891770}]},"public_addr":"172.21.15.118:6811/2724891770","cluster_addr":"172.21.15.118:6813/2724891770","heartbeat_back_addr":"172.21.15.118:6817/2724891770","heartbeat_front_addr":"172.21.15.118:6815/2724891770","state":["exists","up"]},{"osd":6,"uuid":"5ed5a279-9d8e-416d-bbc7-7edd3e6d5705","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6825","nonce":801947386},{"type":"v1","addr":"172.21.15.118:6827","nonce":801947386}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6828","nonce":801947386},{"type":"v1","addr":"172.21.15.118:6829","nonce":801947386}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6832","nonce":801947386},{"type":"v1","addr":"172.21.15.118:6833","nonce":801947386}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6830","nonce":801947386},{"type":"v1","addr":"172.21.15.118:6831","nonce":801947386}]},"public_addr":"172.21.15.118:6827/801947386","cluster_addr":"172.21.15.118:6829/801947386","heartbeat_back_addr":"172.21.15.118:6833/801947386","heartbeat_front_addr":"172.21.15.118:6831/801947386","state":["exists","up"]},{"osd":7,"uuid":"f2086c47-875e-4451-a436-2008d1ea16af","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6818","nonce":653000048},{"type":"v1","addr":"172.21.15.118:6819","nonce":653000048}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6820","nonce":653000048},{"type":"v1","addr":"172.21.15.118:6821","nonce":653000048}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6824","nonce":653000048},{"type":"v1","addr":"172.21.15.118:6826","nonce":653000048}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6822","nonce":653000048},{"type":"v1","addr":"172.21.15.118:6823","nonce":653000048}]},"public_addr":"172.21.15.118:6819/653000048","cluster_addr":"172.21.15.118:6821/653000048","heartbeat_back_addr":"172.21.15.118:6826/653000048","heartbeat_front_addr":"172.21.15.118:6823/653000048","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-03T20:47:25.491994+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-03T20:47:25.587358+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-03T20:47:25.454175+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-03T20:47:25.377734+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-03T20:47:25.412351+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-03T20:47:25.499815+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-03T20:47:25.575044+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-03T20:47:25.552383+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-08-03T20:47:42.916 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:42.914+0000 7f6c46a00640 1 -- 172.21.15.32:0/3788116375 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f6c20077010 msgr2=0x7f6c20079520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:47:42.916 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:42.914+0000 7f6c46a00640 1 --2- 172.21.15.32:0/3788116375 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f6c20077010 0x7f6c20079520 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f6c30000f60 tx=0x7f6c30009040 comp rx=0 tx=0).stop 2024-08-03T20:47:42.916 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:42.914+0000 7f6c46a00640 1 -- 172.21.15.32:0/3788116375 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6c4015fff0 msgr2=0x7f6c4006ef70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:47:42.916 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:42.914+0000 7f6c46a00640 1 --2- 172.21.15.32:0/3788116375 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6c4015fff0 0x7f6c4006ef70 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f6c3800ea80 tx=0x7f6c3800ef40 comp rx=0 tx=0).stop 2024-08-03T20:47:42.916 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:42.914+0000 7f6c46a00640 1 -- 172.21.15.32:0/3788116375 shutdown_connections 2024-08-03T20:47:42.916 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:42.914+0000 7f6c46a00640 1 --2- 172.21.15.32:0/3788116375 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f6c20077010 0x7f6c20079520 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:42.916 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:42.914+0000 7f6c46a00640 1 --2- 172.21.15.32:0/3788116375 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6c4015fff0 0x7f6c4006ef70 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:42.916 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:42.914+0000 7f6c46a00640 1 --2- 172.21.15.32:0/3788116375 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f6c40144f90 0x7f6c4013d960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:42.916 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:42.914+0000 7f6c46a00640 1 --2- 172.21.15.32:0/3788116375 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f6c4006e930 0x7f6c401f66f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:42.916 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:42.914+0000 7f6c46a00640 1 -- 172.21.15.32:0/3788116375 >> 172.21.15.32:0/3788116375 conn(0x7f6c400b1bd0 msgr2=0x7f6c400b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:47:42.917 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:42.914+0000 7f6c46a00640 1 -- 172.21.15.32:0/3788116375 shutdown_connections 2024-08-03T20:47:42.917 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:42.914+0000 7f6c46a00640 1 -- 172.21.15.32:0/3788116375 wait complete. 2024-08-03T20:47:42.940 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2024-08-03T20:47:42.940 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2024-08-03T20:47:42.941 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2024-08-03T20:47:42.941 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2024-08-03T20:47:42.941 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 flush_pg_stats 2024-08-03T20:47:42.942 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 flush_pg_stats 2024-08-03T20:47:42.942 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 flush_pg_stats 2024-08-03T20:47:42.943 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 flush_pg_stats 2024-08-03T20:47:43.360 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.358+0000 7f4b65e00640 1 -- 172.21.15.32:0/848154993 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f4b580e3200 msgr2=0x7f4b580edf80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:47:43.361 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.358+0000 7f4b65e00640 1 --2- 172.21.15.32:0/848154993 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f4b580e3200 0x7f4b580edf80 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f4b6006c4e0 tx=0x7f4b600a5f30 comp rx=0 tx=0).stop 2024-08-03T20:47:43.361 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.359+0000 7f4b65e00640 1 -- 172.21.15.32:0/848154993 shutdown_connections 2024-08-03T20:47:43.361 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.359+0000 7f4b65e00640 1 --2- 172.21.15.32:0/848154993 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f4b580e3200 0x7f4b580edf80 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:43.361 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.359+0000 7f4b65e00640 1 -- 172.21.15.32:0/848154993 >> 172.21.15.32:0/848154993 conn(0x7f4b5803a680 msgr2=0x7f4b5803aa90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:47:43.361 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.359+0000 7f4b65e00640 1 -- 172.21.15.32:0/848154993 shutdown_connections 2024-08-03T20:47:43.361 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.359+0000 7f4b65e00640 1 -- 172.21.15.32:0/848154993 wait complete. 2024-08-03T20:47:43.362 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.358+0000 7f6691c00640 1 -- 172.21.15.32:0/100473314 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6684052360 msgr2=0x7f66840e7280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:47:43.362 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.358+0000 7f6691c00640 1 --2- 172.21.15.32:0/100473314 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6684052360 0x7f66840e7280 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f6684055860 tx=0x7f668002a4d0 comp rx=0 tx=0).stop 2024-08-03T20:47:43.362 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.360+0000 7f6691c00640 1 -- 172.21.15.32:0/100473314 shutdown_connections 2024-08-03T20:47:43.362 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.360+0000 7f6691c00640 1 --2- 172.21.15.32:0/100473314 >> v2:172.21.15.118:3300/0 conn(0x7f6684054720 0x7f6684054b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:43.362 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.360+0000 7f6691c00640 1 --2- 172.21.15.32:0/100473314 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6684052360 0x7f66840e7280 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:43.362 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.360+0000 7f6691c00640 1 -- 172.21.15.32:0/100473314 >> 172.21.15.32:0/100473314 conn(0x7f668401d9a0 msgr2=0x7f6684039660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:47:43.362 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.360+0000 7f6691c00640 1 -- 172.21.15.32:0/100473314 shutdown_connections 2024-08-03T20:47:43.362 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.361+0000 7f6691c00640 1 -- 172.21.15.32:0/100473314 wait complete. 2024-08-03T20:47:43.364 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.363+0000 7f4b65e00640 1 Processor -- start 2024-08-03T20:47:43.364 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.363+0000 7f4b65e00640 1 -- start start 2024-08-03T20:47:43.366 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.365+0000 7f6691c00640 1 Processor -- start 2024-08-03T20:47:43.366 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.365+0000 7f6691c00640 1 -- start start 2024-08-03T20:47:43.367 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.365+0000 7f4b65e00640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f4b58007550 0x7f4b580d7910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:47:43.367 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.366+0000 7f4b65e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4b58054190 0x7f4b581820a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:47:43.367 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.366+0000 7f4b65e00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f4b580e3200 0x7f4b58038db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:47:43.367 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.366+0000 7f4b65e00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f4b580ee850 con 0x7f4b58054190 2024-08-03T20:47:43.368 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.366+0000 7f4b64a00640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f4b58007550 0x7f4b580d7910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:47:43.368 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.366+0000 7f4b64a00640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f4b58007550 0x7f4b580d7910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3301/0 says I am v2:172.21.15.32:36350/0 (socket says 172.21.15.32:36350) 2024-08-03T20:47:43.368 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.366+0000 7f4b64a00640 1 -- 172.21.15.32:0/2531059969 learned_addr learned my addr 172.21.15.32:0/2531059969 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:47:43.368 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.366+0000 7f4b5fe00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4b58054190 0x7f4b581820a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:47:43.368 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.367+0000 7f4b65e00640 1 -- 172.21.15.32:0/2531059969 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f4b580ee6d0 con 0x7f4b580e3200 2024-08-03T20:47:43.368 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.367+0000 7f4b65e00640 1 -- 172.21.15.32:0/2531059969 --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7f4b580ee9d0 con 0x7f4b58007550 2024-08-03T20:47:43.369 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.367+0000 7f4b65400640 1 --2- 172.21.15.32:0/2531059969 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f4b580e3200 0x7f4b58038db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:47:43.369 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.367+0000 7ff99be00640 1 -- 172.21.15.32:0/2497686533 <== mon.2 v2:172.21.15.118:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9880047a0 con 0x7ff99c0b93d0 2024-08-03T20:47:43.369 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.368+0000 7f6691c00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6684051630 0x7f66840e4180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:47:43.369 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.368+0000 7f6691c00640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f6684052360 0x7f6684050010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:47:43.369 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.368+0000 7f6691c00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f6684054720 0x7f6684050550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:47:43.370 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.368+0000 7f6691c00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f6684033ec0 con 0x7f6684051630 2024-08-03T20:47:43.370 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.368+0000 7f6691c00640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f6684052770 con 0x7f6684054720 2024-08-03T20:47:43.370 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.368+0000 7f6691c00640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7f66840dd100 con 0x7f6684052360 2024-08-03T20:47:43.370 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.368+0000 7f4b5fe00640 1 -- 172.21.15.32:0/2531059969 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f4b58007550 msgr2=0x7f4b580d7910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:47:43.370 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.368+0000 7f4b5fe00640 1 --2- 172.21.15.32:0/2531059969 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f4b58007550 0x7f4b580d7910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:43.370 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.368+0000 7f4b5fe00640 1 -- 172.21.15.32:0/2531059969 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f4b580e3200 msgr2=0x7f4b58038db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:47:43.370 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.368+0000 7f4b5fe00640 1 --2- 172.21.15.32:0/2531059969 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f4b580e3200 0x7f4b58038db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:43.371 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.368+0000 7f668be00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6684051630 0x7f66840e4180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:47:43.371 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.368+0000 7f4b5fe00640 1 -- 172.21.15.32:0/2531059969 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4b580d9660 con 0x7f4b58054190 2024-08-03T20:47:43.371 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.368+0000 7f4b65400640 1 --2- 172.21.15.32:0/2531059969 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f4b580e3200 0x7f4b58038db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-03T20:47:43.371 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.368+0000 7f668be00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6684051630 0x7f66840e4180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:57348/0 (socket says 172.21.15.32:57348) 2024-08-03T20:47:43.372 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.368+0000 7f668be00640 1 -- 172.21.15.32:0/3729796857 learned_addr learned my addr 172.21.15.32:0/3729796857 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:47:43.372 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.368+0000 7f6691200640 1 --2- 172.21.15.32:0/3729796857 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f6684054720 0x7f6684050550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:47:43.372 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.369+0000 7ff9a2c00640 1 -- 172.21.15.32:0/2497686533 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7ff99c0b93d0 msgr2=0x7ff99c0b39e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:47:43.372 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.369+0000 7ff9a2c00640 1 --2- 172.21.15.32:0/2497686533 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7ff99c0b93d0 0x7ff99c0b39e0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7ff988005fd0 tx=0x7ff98802a5d0 comp rx=0 tx=0).stop 2024-08-03T20:47:43.372 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.369+0000 7f4b5fe00640 1 --2- 172.21.15.32:0/2531059969 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4b58054190 0x7f4b581820a0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f4b5000b3e0 tx=0x7f4b5000b8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:47:43.373 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.369+0000 7ff9a2c00640 1 -- 172.21.15.32:0/2497686533 shutdown_connections 2024-08-03T20:47:43.373 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.369+0000 7ff9a2c00640 1 --2- 172.21.15.32:0/2497686533 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7ff99c0b93d0 0x7ff99c0b39e0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:43.373 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.369+0000 7ff9a2c00640 1 --2- 172.21.15.32:0/2497686533 >> v2:172.21.15.32:3300/0 conn(0x7ff99c0d4ba0 0x7ff99c0b3240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:43.373 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.369+0000 7ff9a2c00640 1 -- 172.21.15.32:0/2497686533 >> 172.21.15.32:0/2497686533 conn(0x7ff99c0d4080 msgr2=0x7ff99c0d4490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:47:43.374 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.370+0000 7f4b5cc00640 1 -- 172.21.15.32:0/2531059969 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b50015020 con 0x7f4b58054190 2024-08-03T20:47:43.374 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.370+0000 7f4b5cc00640 1 -- 172.21.15.32:0/2531059969 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f4b50011940 con 0x7f4b58054190 2024-08-03T20:47:43.374 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.370+0000 7f4b5cc00640 1 -- 172.21.15.32:0/2531059969 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b50011d90 con 0x7f4b58054190 2024-08-03T20:47:43.374 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.370+0000 7f668be00640 1 -- 172.21.15.32:0/3729796857 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f6684052360 msgr2=0x7f6684050010 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-08-03T20:47:43.374 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.370+0000 7f668be00640 1 --2- 172.21.15.32:0/3729796857 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f6684052360 0x7f6684050010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:43.374 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.370+0000 7f668be00640 1 -- 172.21.15.32:0/3729796857 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f6684054720 msgr2=0x7f6684050550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:47:43.375 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.370+0000 7f668be00640 1 --2- 172.21.15.32:0/3729796857 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f6684054720 0x7f6684050550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:43.375 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.370+0000 7f668be00640 1 -- 172.21.15.32:0/3729796857 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f66840dcc30 con 0x7f6684051630 2024-08-03T20:47:43.375 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.370+0000 7ff9a2c00640 1 -- 172.21.15.32:0/2497686533 shutdown_connections 2024-08-03T20:47:43.375 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.370+0000 7ff9a2c00640 1 -- 172.21.15.32:0/2497686533 wait complete. 2024-08-03T20:47:43.375 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.370+0000 7f4b65e00640 1 -- 172.21.15.32:0/2531059969 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4b580e2500 con 0x7f4b58054190 2024-08-03T20:47:43.376 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.371+0000 7f668be00640 1 --2- 172.21.15.32:0/3729796857 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6684051630 0x7f66840e4180 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f6680000c00 tx=0x7f6680005270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:47:43.376 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.370+0000 7f4b65e00640 1 -- 172.21.15.32:0/2531059969 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4b580e21d0 con 0x7f4b58054190 2024-08-03T20:47:43.376 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.371+0000 7ff45e400640 1 -- 172.21.15.32:0/922888438 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7ff450056da0 msgr2=0x7ff450057c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:47:43.376 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.371+0000 7ff45e400640 1 --2- 172.21.15.32:0/922888438 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7ff450056da0 0x7ff450057c80 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7ff45000fcc0 tx=0x7ff44c02a560 comp rx=0 tx=0).stop 2024-08-03T20:47:43.377 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.372+0000 7ff45e400640 1 -- 172.21.15.32:0/922888438 shutdown_connections 2024-08-03T20:47:43.377 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.372+0000 7ff45e400640 1 --2- 172.21.15.32:0/922888438 >> v2:172.21.15.32:3300/0 conn(0x7ff4500e2f40 0x7ff45003c1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:43.377 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.372+0000 7ff45e400640 1 --2- 172.21.15.32:0/922888438 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7ff450056da0 0x7ff450057c80 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:43.377 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.372+0000 7ff45e400640 1 -- 172.21.15.32:0/922888438 >> 172.21.15.32:0/922888438 conn(0x7ff45003a5f0 msgr2=0x7ff45003aa00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:47:43.377 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.372+0000 7f4b65e00640 1 -- 172.21.15.32:0/2531059969 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f4b580e2a20 con 0x7f4b58054190 2024-08-03T20:47:43.378 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.373+0000 7ff45e400640 1 -- 172.21.15.32:0/922888438 shutdown_connections 2024-08-03T20:47:43.378 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.373+0000 7ff45e400640 1 -- 172.21.15.32:0/922888438 wait complete. 2024-08-03T20:47:43.378 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.374+0000 7f876d600640 1 -- 172.21.15.32:0/1279906054 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f87680b4de0 msgr2=0x7f87681515c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:47:43.379 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.374+0000 7f876d600640 1 --2- 172.21.15.32:0/1279906054 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f87680b4de0 0x7f87681515c0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f8768072280 tx=0x7f875802a4d0 comp rx=0 tx=0).stop 2024-08-03T20:47:43.379 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.375+0000 7f876d600640 1 -- 172.21.15.32:0/1279906054 shutdown_connections 2024-08-03T20:47:43.379 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.375+0000 7f876d600640 1 --2- 172.21.15.32:0/1279906054 >> v2:172.21.15.32:3300/0 conn(0x7f876815e880 0x7f8768073800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:43.379 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.375+0000 7f876d600640 1 --2- 172.21.15.32:0/1279906054 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f87680b4de0 0x7f87681515c0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:43.379 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.375+0000 7f876d600640 1 -- 172.21.15.32:0/1279906054 >> 172.21.15.32:0/1279906054 conn(0x7f87680b1bd0 msgr2=0x7f87680b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:47:43.379 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.375+0000 7f876d600640 1 -- 172.21.15.32:0/1279906054 shutdown_connections 2024-08-03T20:47:43.379 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.375+0000 7f876d600640 1 -- 172.21.15.32:0/1279906054 wait complete. 2024-08-03T20:47:43.380 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.375+0000 7f667fe00640 1 -- 172.21.15.32:0/3729796857 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6680005670 con 0x7f6684051630 2024-08-03T20:47:43.380 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.375+0000 7f667fe00640 1 -- 172.21.15.32:0/3729796857 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f6680004220 con 0x7f6684051630 2024-08-03T20:47:43.380 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.375+0000 7f667fe00640 1 -- 172.21.15.32:0/3729796857 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6680031950 con 0x7f6684051630 2024-08-03T20:47:43.380 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.377+0000 7f6691c00640 1 -- 172.21.15.32:0/3729796857 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f668404c8f0 con 0x7f6684051630 2024-08-03T20:47:43.380 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.377+0000 7f6691c00640 1 -- 172.21.15.32:0/3729796857 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f668404cfd0 con 0x7f6684051630 2024-08-03T20:47:43.380 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.377+0000 7ff9a2c00640 1 Processor -- start 2024-08-03T20:47:43.381 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.377+0000 7ff9a2c00640 1 -- start start 2024-08-03T20:47:43.381 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.377+0000 7f4b5cc00640 1 -- 172.21.15.32:0/2531059969 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f4b50018020 con 0x7f4b58054190 2024-08-03T20:47:43.381 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.378+0000 7f6691c00640 1 -- 172.21.15.32:0/3729796857 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f668404cb30 con 0x7f6684051630 2024-08-03T20:47:43.382 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.378+0000 7ff45e400640 1 Processor -- start 2024-08-03T20:47:43.382 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.379+0000 7ff45e400640 1 -- start start 2024-08-03T20:47:43.382 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.379+0000 7f876d600640 1 Processor -- start 2024-08-03T20:47:43.382 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.379+0000 7fc50e400640 1 -- 172.21.15.32:0/1056868023 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fc4ec004800 msgr2=0x7fc4ec004c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:47:43.382 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.379+0000 7fc50e400640 1 --2- 172.21.15.32:0/1056868023 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fc4ec004800 0x7fc4ec004c10 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fc5080cbfc0 tx=0x7fc4ec005830 comp rx=0 tx=0).stop 2024-08-03T20:47:43.383 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.379+0000 7f876d600640 1 -- start start 2024-08-03T20:47:43.383 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.379+0000 7fc50e400640 1 -- 172.21.15.32:0/1056868023 shutdown_connections 2024-08-03T20:47:43.383 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.379+0000 7fc50e400640 1 --2- 172.21.15.32:0/1056868023 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fc4ec004800 0x7fc4ec004c10 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:43.383 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.379+0000 7fc50e400640 1 -- 172.21.15.32:0/1056868023 >> 172.21.15.32:0/1056868023 conn(0x7fc5080b1d50 msgr2=0x7fc5080b2160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:47:43.384 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.379+0000 7f4b5cc00640 1 --2- 172.21.15.32:0/2531059969 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f4b44077060 0x7f4b44079570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:47:43.384 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.379+0000 7f4b5cc00640 1 -- 172.21.15.32:0/2531059969 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f4b50098e30 con 0x7f4b58054190 2024-08-03T20:47:43.384 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.379+0000 7f4b5cc00640 1 --2- 172.21.15.32:0/2531059969 >> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] conn(0x7f4b4407f330 0x7f4b44081780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:47:43.384 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.379+0000 7f4b5cc00640 1 -- 172.21.15.32:0/2531059969 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f4b44081e50 con 0x7f4b4407f330 2024-08-03T20:47:43.384 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.379+0000 7f4b5cc00640 1 -- 172.21.15.32:0/2531059969 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f4b500991e0 con 0x7f4b58054190 2024-08-03T20:47:43.384 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.380+0000 7f4b64a00640 1 --2- 172.21.15.32:0/2531059969 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f4b44077060 0x7f4b44079570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:47:43.384 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.380+0000 7f4b65400640 1 --2- 172.21.15.32:0/2531059969 >> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] conn(0x7f4b4407f330 0x7f4b44081780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:47:43.384 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.381+0000 7f3b48c00640 1 -- 172.21.15.32:0/497242544 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f3b440d2100 msgr2=0x7f3b440b9660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:47:43.385 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.381+0000 7f3b48c00640 1 --2- 172.21.15.32:0/497242544 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f3b440d2100 0x7f3b440b9660 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f3b440b9ba0 tx=0x7f3b3402aa30 comp rx=0 tx=0).stop 2024-08-03T20:47:43.385 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.381+0000 7f3b48c00640 1 -- 172.21.15.32:0/497242544 shutdown_connections 2024-08-03T20:47:43.385 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.381+0000 7fdcc6000640 1 -- 172.21.15.32:0/4057436414 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdcc0009be0 con 0x7fdcac005350 2024-08-03T20:47:43.386 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.381+0000 7f3b48c00640 1 --2- 172.21.15.32:0/497242544 >> v2:172.21.15.118:3300/0 conn(0x7f3b440d1220 0x7f3b440cef30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:43.386 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.381+0000 7f3b48c00640 1 --2- 172.21.15.32:0/497242544 >> v2:172.21.15.32:3300/0 conn(0x7f3b440b34d0 0x7f3b440d0ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:43.386 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.381+0000 7f3b48c00640 1 --2- 172.21.15.32:0/497242544 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f3b440d2100 0x7f3b440b9660 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:43.386 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.381+0000 7f3b48c00640 1 -- 172.21.15.32:0/497242544 >> 172.21.15.32:0/497242544 conn(0x7f3b4409d620 msgr2=0x7f3b440d4320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:47:43.386 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.382+0000 7fdcce000640 1 -- 172.21.15.32:0/4057436414 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdcac005350 msgr2=0x7fdcac00b9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:47:43.386 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.382+0000 7ff45e400640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ff450053c50 0x7ff45017e840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:47:43.387 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.382+0000 7f3b48c00640 1 -- 172.21.15.32:0/497242544 shutdown_connections 2024-08-03T20:47:43.387 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.382+0000 7ff45e400640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7ff450056da0 0x7ff4500dbe40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:47:43.387 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.380+0000 7fc50e400640 1 -- 172.21.15.32:0/1056868023 shutdown_connections 2024-08-03T20:47:43.387 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.382+0000 7fdcce000640 1 --2- 172.21.15.32:0/4057436414 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdcac005350 0x7fdcac00b9e0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fdcc0009f60 tx=0x7fdcac005830 comp rx=0 tx=0).stop 2024-08-03T20:47:43.388 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.382+0000 7f876d600640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f87680733f0 0x7f87681fcca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:47:43.388 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.382+0000 7fc50e400640 1 -- 172.21.15.32:0/1056868023 wait complete. 2024-08-03T20:47:43.388 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.382+0000 7ff45e400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff4500e2f40 0x7ff450051040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:47:43.389 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.381+0000 7f4b65400640 1 --2- 172.21.15.32:0/2531059969 >> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] conn(0x7f4b4407f330 0x7f4b44081780 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:47:43.389 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.382+0000 7ff45d000640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ff450053c50 0x7ff45017e840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:47:43.389 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.382+0000 7ff45d000640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ff450053c50 0x7ff45017e840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.32:45874/0 (socket says 172.21.15.32:45874) 2024-08-03T20:47:43.389 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.382+0000 7f876d600640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f87680b4de0 0x7f87681404a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:47:43.390 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.382+0000 7f876d600640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f876813f3b0 0x7f87681428b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:47:43.390 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.382+0000 7f876d600640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f876806eab0 con 0x7f876813f3b0 2024-08-03T20:47:43.390 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.382+0000 7f876d600640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f876806e930 con 0x7f87680733f0 2024-08-03T20:47:43.390 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.382+0000 7f4b5cc00640 1 -- 172.21.15.32:0/2531059969 <== osd.1 v2:172.21.15.32:6824/1551241759 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f4b44081e50 con 0x7f4b4407f330 2024-08-03T20:47:43.390 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.380+0000 7ff9a2c00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff99c0b3f20 0x7ff99c09f560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:47:43.390 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.380+0000 7ff9a2c00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ff99c0b93d0 0x7ff99c0a0e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:47:43.391 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.382+0000 7f876d600640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7f876806ec30 con 0x7f87680b4de0 2024-08-03T20:47:43.391 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.380+0000 7ff9a2c00640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7ff99c0d4ba0 0x7ff99c159120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:47:43.391 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.380+0000 7ff9a2c00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7ff99c0d08f0 con 0x7ff99c0b3f20 2024-08-03T20:47:43.391 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.380+0000 7ff9a2c00640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7ff99c0d0770 con 0x7ff99c0b93d0 2024-08-03T20:47:43.391 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.380+0000 7ff9a2c00640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7ff99c0d0a70 con 0x7ff99c0d4ba0 2024-08-03T20:47:43.392 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.380+0000 7ff9a1800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff99c0b3f20 0x7ff99c09f560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:47:43.392 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.380+0000 7ff9a1800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff99c0b3f20 0x7ff99c09f560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:57406/0 (socket says 172.21.15.32:57406) 2024-08-03T20:47:43.392 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.380+0000 7ff9a1800640 1 -- 172.21.15.32:0/521504658 learned_addr learned my addr 172.21.15.32:0/521504658 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:47:43.392 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.381+0000 7ff9a1800640 1 -- 172.21.15.32:0/521504658 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7ff99c0d4ba0 msgr2=0x7ff99c159120 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-08-03T20:47:43.392 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.381+0000 7ff9a1800640 1 --2- 172.21.15.32:0/521504658 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7ff99c0d4ba0 0x7ff99c159120 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:43.392 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.381+0000 7ff9a1800640 1 -- 172.21.15.32:0/521504658 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ff99c0b93d0 msgr2=0x7ff99c0a0e30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-08-03T20:47:43.392 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.381+0000 7ff9a1800640 1 --2- 172.21.15.32:0/521504658 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ff99c0b93d0 0x7ff99c0a0e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:43.392 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.382+0000 7fdcce000640 1 -- 172.21.15.32:0/4057436414 shutdown_connections 2024-08-03T20:47:43.393 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.381+0000 7ff9a1800640 1 -- 172.21.15.32:0/521504658 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff99c159660 con 0x7ff99c0b3f20 2024-08-03T20:47:43.393 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.382+0000 7ff9a1800640 1 --2- 172.21.15.32:0/521504658 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff99c0b3f20 0x7ff99c09f560 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7ff99000e9a0 tx=0x7ff99000ee60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:47:43.393 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.383+0000 7fdcce000640 1 --2- 172.21.15.32:0/4057436414 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdcac005350 0x7fdcac00b9e0 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:43.393 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.383+0000 7fdcce000640 1 -- 172.21.15.32:0/4057436414 >> 172.21.15.32:0/4057436414 conn(0x7fdcc809d660 msgr2=0x7fdcc80d4330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:47:43.394 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.383+0000 7ff45d000640 1 -- 172.21.15.32:0/2349670721 learned_addr learned my addr 172.21.15.32:0/2349670721 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:47:43.394 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.383+0000 7ff999600640 1 -- 172.21.15.32:0/521504658 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff99000cd80 con 0x7ff99c0b3f20 2024-08-03T20:47:43.394 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.383+0000 7f4b64a00640 1 --2- 172.21.15.32:0/2531059969 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f4b44077060 0x7f4b44079570 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f4b54001220 tx=0x7f4b54004640 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:47:43.394 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.383+0000 7ff45e400640 1 -- 172.21.15.32:0/2349670721 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7ff450007180 con 0x7ff4500e2f40 2024-08-03T20:47:43.394 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.383+0000 7ff45e400640 1 -- 172.21.15.32:0/2349670721 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7ff450007000 con 0x7ff450053c50 2024-08-03T20:47:43.395 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.383+0000 7ff45e400640 1 -- 172.21.15.32:0/2349670721 --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7ff450007300 con 0x7ff450056da0 2024-08-03T20:47:43.395 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.383+0000 7ff999600640 1 -- 172.21.15.32:0/521504658 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7ff990004020 con 0x7ff99c0b3f20 2024-08-03T20:47:43.395 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.383+0000 7ff999600640 1 -- 172.21.15.32:0/521504658 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff990004430 con 0x7ff99c0b3f20 2024-08-03T20:47:43.395 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.382+0000 7f667fe00640 1 -- 172.21.15.32:0/3729796857 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f6680031b10 con 0x7f6684051630 2024-08-03T20:47:43.395 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.383+0000 7ff9a2c00640 1 -- 172.21.15.32:0/521504658 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff99c09fd90 con 0x7ff99c0b3f20 2024-08-03T20:47:43.396 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.383+0000 7fdcce000640 1 -- 172.21.15.32:0/4057436414 shutdown_connections 2024-08-03T20:47:43.396 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.383+0000 7fdcce000640 1 -- 172.21.15.32:0/4057436414 wait complete. 2024-08-03T20:47:43.396 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.382+0000 7f3b48c00640 1 -- 172.21.15.32:0/497242544 wait complete. 2024-08-03T20:47:43.396 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.384+0000 7ff9a2c00640 1 -- 172.21.15.32:0/521504658 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff99c0a0350 con 0x7ff99c0b3f20 2024-08-03T20:47:43.397 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.383+0000 7f8766a00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f87680733f0 0x7f87681fcca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:47:43.397 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.383+0000 7f8766a00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f87680733f0 0x7f87681fcca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.32:45878/0 (socket says 172.21.15.32:45878) 2024-08-03T20:47:43.397 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.383+0000 7f8766a00640 1 -- 172.21.15.32:0/4247644840 learned_addr learned my addr 172.21.15.32:0/4247644840 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:47:43.397 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.383+0000 7f8766000640 1 --2- 172.21.15.32:0/4247644840 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f87680b4de0 0x7f87681404a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:47:43.397 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.384+0000 7f8767400640 1 --2- 172.21.15.32:0/4247644840 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f876813f3b0 0x7f87681428b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:47:43.397 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.383+0000 7ff457e00640 1 --2- 172.21.15.32:0/2349670721 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7ff450056da0 0x7ff4500dbe40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:47:43.397 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.383+0000 7ff45da00640 1 --2- 172.21.15.32:0/2349670721 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff4500e2f40 0x7ff450051040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:47:43.398 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.385+0000 7f667fe00640 1 --2- 172.21.15.32:0/3729796857 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f666c077090 0x7f666c0795a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:47:43.398 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.385+0000 7ff9a2c00640 1 -- 172.21.15.32:0/521504658 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7ff99c0d0f90 con 0x7ff99c0b3f20 2024-08-03T20:47:43.398 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.385+0000 7f8766a00640 1 -- 172.21.15.32:0/4247644840 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f87680b4de0 msgr2=0x7f87681404a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:47:43.398 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.385+0000 7f8766a00640 1 --2- 172.21.15.32:0/4247644840 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f87680b4de0 0x7f87681404a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:43.398 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.385+0000 7f8766a00640 1 -- 172.21.15.32:0/4247644840 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f876813f3b0 msgr2=0x7f87681428b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:47:43.399 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.385+0000 7f667fe00640 1 -- 172.21.15.32:0/3729796857 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f66800b9f40 con 0x7f6684051630 2024-08-03T20:47:43.399 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.385+0000 7f667fe00640 1 --2- 172.21.15.32:0/3729796857 >> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] conn(0x7f666c07f360 0x7f666c0817b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:47:43.399 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.385+0000 7f667fe00640 1 -- 172.21.15.32:0/3729796857 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f666c081e80 con 0x7f666c07f360 2024-08-03T20:47:43.399 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.385+0000 7f667fe00640 1 -- 172.21.15.32:0/3729796857 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f66800ba2f0 con 0x7f6684051630 2024-08-03T20:47:43.400 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.385+0000 7ff45d000640 1 -- 172.21.15.32:0/2349670721 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7ff450056da0 msgr2=0x7ff4500dbe40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:47:43.400 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.385+0000 7ff45d000640 1 --2- 172.21.15.32:0/2349670721 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7ff450056da0 0x7ff4500dbe40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:43.400 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.385+0000 7ff45d000640 1 -- 172.21.15.32:0/2349670721 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff4500e2f40 msgr2=0x7ff450051040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:47:43.400 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.385+0000 7ff45d000640 1 --2- 172.21.15.32:0/2349670721 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff4500e2f40 0x7ff450051040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:43.400 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.385+0000 7ff45d000640 1 -- 172.21.15.32:0/2349670721 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff4500dc380 con 0x7ff450053c50 2024-08-03T20:47:43.400 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.385+0000 7ff457e00640 1 --2- 172.21.15.32:0/2349670721 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7ff450056da0 0x7ff4500dbe40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-03T20:47:43.400 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.385+0000 7f8766a00640 1 --2- 172.21.15.32:0/4247644840 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f876813f3b0 0x7f87681428b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:43.400 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.385+0000 7f8766a00640 1 -- 172.21.15.32:0/4247644840 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f87681fd530 con 0x7f87680733f0 2024-08-03T20:47:43.400 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.385+0000 7f8767400640 1 --2- 172.21.15.32:0/4247644840 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f876813f3b0 0x7f87681428b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-03T20:47:43.401 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.386+0000 7f668b400640 1 --2- 172.21.15.32:0/3729796857 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f666c077090 0x7f666c0795a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:47:43.401 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.386+0000 7f3b48c00640 1 Processor -- start 2024-08-03T20:47:43.401 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.386+0000 7f8766a00640 1 --2- 172.21.15.32:0/4247644840 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f87680733f0 0x7f87681fcca0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f8758034040 tx=0x7f8758003960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:47:43.402 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.386+0000 7f6691200640 1 --2- 172.21.15.32:0/3729796857 >> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] conn(0x7f666c07f360 0x7f666c0817b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:47:43.402 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.386+0000 7ff45da00640 1 --2- 172.21.15.32:0/2349670721 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff4500e2f40 0x7ff450051040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-03T20:47:43.402 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.386+0000 7f3b48c00640 1 -- start start 2024-08-03T20:47:43.403 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.386+0000 7f875ea00640 1 -- 172.21.15.32:0/4247644840 <== mon.1 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f875803e030 con 0x7f87680733f0 2024-08-03T20:47:43.403 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.387+0000 7f875ea00640 1 -- 172.21.15.32:0/4247644840 <== mon.1 v2:172.21.15.118:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f8758003e50 con 0x7f87680733f0 2024-08-03T20:47:43.403 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.387+0000 7f876d600640 1 -- 172.21.15.32:0/4247644840 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f87681fd810 con 0x7f87680733f0 2024-08-03T20:47:43.403 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.387+0000 7f875ea00640 1 -- 172.21.15.32:0/4247644840 <== mon.1 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8758035040 con 0x7f87680733f0 2024-08-03T20:47:43.403 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.387+0000 7f876d600640 1 -- 172.21.15.32:0/4247644840 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8768142390 con 0x7f87680733f0 2024-08-03T20:47:43.403 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.387+0000 7fc50e400640 1 Processor -- start 2024-08-03T20:47:43.404 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.388+0000 7fdcce000640 1 Processor -- start 2024-08-03T20:47:43.404 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.388+0000 7fdcce000640 1 -- start start 2024-08-03T20:47:43.404 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.388+0000 7f876d600640 1 -- 172.21.15.32:0/4247644840 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f8740001030 con 0x7f87680733f0 2024-08-03T20:47:43.404 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.389+0000 7ff999600640 1 -- 172.21.15.32:0/521504658 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7ff9900035f0 con 0x7ff99c0b3f20 2024-08-03T20:47:43.405 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.388+0000 7f6691200640 1 --2- 172.21.15.32:0/3729796857 >> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] conn(0x7f666c07f360 0x7f666c0817b0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:47:43.405 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.389+0000 7f667fe00640 1 -- 172.21.15.32:0/3729796857 <== osd.0 v2:172.21.15.32:6816/3066156116 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f666c081e80 con 0x7f666c07f360 2024-08-03T20:47:43.405 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.389+0000 7f3b48c00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f3b440b34d0 0x7f3b441583e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:47:43.405 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.389+0000 7f3b48c00640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f3b440d1220 0x7f3b4415c300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:47:43.405 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.389+0000 7f3b48c00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3b440d2100 0x7f3b441faeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:47:43.405 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.389+0000 7f3b48c00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f3b440cf910 con 0x7f3b440d2100 2024-08-03T20:47:43.405 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.389+0000 7f3b48c00640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f3b440cf790 con 0x7f3b440b34d0 2024-08-03T20:47:43.405 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.389+0000 7f3b48c00640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7f3b440cfa90 con 0x7f3b440d1220 2024-08-03T20:47:43.406 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.388+0000 7fc50e400640 1 -- start start 2024-08-03T20:47:43.406 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.390+0000 7ff45d000640 1 --2- 172.21.15.32:0/2349670721 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ff450053c50 0x7ff45017e840 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7ff44c000c00 tx=0x7ff44c003b90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:47:43.406 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.390+0000 7ff999600640 1 --2- 172.21.15.32:0/521504658 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7ff97c0770e0 0x7ff97c0795f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:47:43.406 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.391+0000 7ff999600640 1 -- 172.21.15.32:0/521504658 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7ff9900aa520 con 0x7ff99c0b3f20 2024-08-03T20:47:43.407 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.391+0000 7ff454c00640 1 -- 172.21.15.32:0/2349670721 <== mon.1 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff44c03e030 con 0x7ff450053c50 2024-08-03T20:47:43.407 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.391+0000 7ff454c00640 1 -- 172.21.15.32:0/2349670721 <== mon.1 v2:172.21.15.118:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7ff44c0055f0 con 0x7ff450053c50 2024-08-03T20:47:43.407 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.391+0000 7ff454c00640 1 -- 172.21.15.32:0/2349670721 <== mon.1 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff44c035040 con 0x7ff450053c50 2024-08-03T20:47:43.407 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.391+0000 7ff9a0e00640 1 --2- 172.21.15.32:0/521504658 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7ff97c0770e0 0x7ff97c0795f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:47:43.407 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.391+0000 7ff45e400640 1 -- 172.21.15.32:0/2349670721 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff45016db40 con 0x7ff450053c50 2024-08-03T20:47:43.408 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.390+0000 7f3b3be00640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f3b440d1220 0x7f3b4415c300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:47:43.408 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.390+0000 7f3b3be00640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f3b440d1220 0x7f3b4415c300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3301/0 says I am v2:172.21.15.32:36416/0 (socket says 172.21.15.32:36416) 2024-08-03T20:47:43.408 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.390+0000 7f3b3be00640 1 -- 172.21.15.32:0/1634897216 learned_addr learned my addr 172.21.15.32:0/1634897216 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:47:43.408 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.390+0000 7f3b42800640 1 --2- 172.21.15.32:0/1634897216 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3b440d2100 0x7f3b441faeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:47:43.408 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.390+0000 7f3b3be00640 1 -- 172.21.15.32:0/1634897216 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f3b440b34d0 msgr2=0x7f3b441583e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-08-03T20:47:43.408 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.391+0000 7ff45e400640 1 -- 172.21.15.32:0/2349670721 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff45016e2a0 con 0x7ff450053c50 2024-08-03T20:47:43.409 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.391+0000 7f3b3be00640 1 --2- 172.21.15.32:0/1634897216 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f3b440b34d0 0x7f3b441583e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:43.409 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.391+0000 7f3b3be00640 1 -- 172.21.15.32:0/1634897216 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3b440d2100 msgr2=0x7f3b441faeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:47:43.409 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.391+0000 7f3b3be00640 1 --2- 172.21.15.32:0/1634897216 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3b440d2100 0x7f3b441faeb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:43.409 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.391+0000 7f3b3be00640 1 -- 172.21.15.32:0/1634897216 --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3b4415c840 con 0x7f3b440d1220 2024-08-03T20:47:43.409 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.391+0000 7f3b42800640 1 --2- 172.21.15.32:0/1634897216 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3b440d2100 0x7f3b441faeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-03T20:47:43.409 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.392+0000 7fdcce000640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fdcac005350 0x7fdcc814d9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:47:43.409 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.391+0000 7f668b400640 1 --2- 172.21.15.32:0/3729796857 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f666c077090 0x7f666c0795a0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f66841818c0 tx=0x7f6678006b30 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:47:43.410 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.391+0000 7f875ea00640 1 -- 172.21.15.32:0/4247644840 <== mon.1 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f8758041030 con 0x7f87680733f0 2024-08-03T20:47:43.410 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.392+0000 7fdcce000640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fdcc81f88f0 0x7fdcc81f8d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:47:43.410 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.392+0000 7f3b3be00640 1 --2- 172.21.15.32:0/1634897216 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f3b440d1220 0x7f3b4415c300 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f3b2c00d780 tx=0x7f3b2c00dc40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:47:43.411 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.392+0000 7fdcc7400640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fdcac005350 0x7fdcc814d9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:47:43.411 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.391+0000 7fc50e400640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc4ec004800 0x7fc508157770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:47:43.411 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.391+0000 7fc50e400640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fc5080ae3e0 0x7fc508152af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:47:43.411 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.392+0000 7fc50e400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc5081f3cf0 0x7fc5081f4100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:47:43.411 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.392+0000 7fc50e400640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fc508158b80 con 0x7fc5081f3cf0 2024-08-03T20:47:43.411 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.392+0000 7fc50e400640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7fc5080ce1b0 con 0x7fc4ec004800 2024-08-03T20:47:43.411 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.392+0000 7fc50e400640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7fc508158d00 con 0x7fc5080ae3e0 2024-08-03T20:47:43.412 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.392+0000 7fdcce000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdcc81f9240 0x7fdcc814e620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:47:43.412 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.392+0000 7fdcc7400640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fdcac005350 0x7fdcc814d9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.32:45896/0 (socket says 172.21.15.32:45896) 2024-08-03T20:47:43.412 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.392+0000 7fdcc7400640 1 -- 172.21.15.32:0/3010512042 learned_addr learned my addr 172.21.15.32:0/3010512042 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:47:43.412 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.392+0000 7fc50da00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc5081f3cf0 0x7fc5081f4100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:47:43.412 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.392+0000 7fc50d000640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc4ec004800 0x7fc508157770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:47:43.412 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.392+0000 7ff999600640 1 --2- 172.21.15.32:0/521504658 >> [v2:172.21.15.32:6808/3806708944,v1:172.21.15.32:6809/3806708944] conn(0x7ff97c07f350 0x7ff97c0817a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:47:43.412 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.392+0000 7ff999600640 1 -- 172.21.15.32:0/521504658 --> [v2:172.21.15.32:6808/3806708944,v1:172.21.15.32:6809/3806708944] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7ff99002bec0 con 0x7ff97c07f350 2024-08-03T20:47:43.413 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.392+0000 7ff999600640 1 -- 172.21.15.32:0/521504658 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7ff9900aa910 con 0x7ff99c0b3f20 2024-08-03T20:47:43.413 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.392+0000 7fc50d000640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc4ec004800 0x7fc508157770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.32:45912/0 (socket says 172.21.15.32:45912) 2024-08-03T20:47:43.413 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.392+0000 7fc50d000640 1 -- 172.21.15.32:0/4288518052 learned_addr learned my addr 172.21.15.32:0/4288518052 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:47:43.413 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.392+0000 7fc507e00640 1 --2- 172.21.15.32:0/4288518052 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fc5080ae3e0 0x7fc508152af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:47:43.413 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.392+0000 7f3b3a000640 1 -- 172.21.15.32:0/1634897216 <== mon.2 v2:172.21.15.118:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b2c017070 con 0x7f3b440d1220 2024-08-03T20:47:43.414 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.393+0000 7fdcc6a00640 1 --2- 172.21.15.32:0/3010512042 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fdcc81f88f0 0x7fdcc81f8d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:47:43.414 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.393+0000 7fdcc7e00640 1 --2- 172.21.15.32:0/3010512042 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdcc81f9240 0x7fdcc814e620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:47:43.414 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.393+0000 7fdcce000640 1 -- 172.21.15.32:0/3010512042 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fdcc80d0bf0 con 0x7fdcc81f9240 2024-08-03T20:47:43.414 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.393+0000 7ff9a2200640 1 --2- 172.21.15.32:0/521504658 >> [v2:172.21.15.32:6808/3806708944,v1:172.21.15.32:6809/3806708944] conn(0x7ff97c07f350 0x7ff97c0817a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:47:43.414 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.393+0000 7fc507e00640 1 -- 172.21.15.32:0/4288518052 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc4ec004800 msgr2=0x7fc508157770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:47:43.414 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.393+0000 7fc507e00640 1 --2- 172.21.15.32:0/4288518052 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc4ec004800 0x7fc508157770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:43.414 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.393+0000 7fc507e00640 1 -- 172.21.15.32:0/4288518052 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc5081f3cf0 msgr2=0x7fc5081f4100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:47:43.415 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.393+0000 7f3b3a000640 1 -- 172.21.15.32:0/1634897216 <== mon.2 v2:172.21.15.118:3301/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f3b2c00de90 con 0x7f3b440d1220 2024-08-03T20:47:43.415 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.393+0000 7f3b3a000640 1 -- 172.21.15.32:0/1634897216 <== mon.2 v2:172.21.15.118:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b2c0099d0 con 0x7f3b440d1220 2024-08-03T20:47:43.415 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.393+0000 7f3b48c00640 1 -- 172.21.15.32:0/1634897216 --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3b4415cd50 con 0x7f3b440d1220 2024-08-03T20:47:43.415 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.393+0000 7fdcce000640 1 -- 172.21.15.32:0/3010512042 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7fdcc80d0a70 con 0x7fdcac005350 2024-08-03T20:47:43.415 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.393+0000 7fdcce000640 1 -- 172.21.15.32:0/3010512042 --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7fdcc80d0d70 con 0x7fdcc81f88f0 2024-08-03T20:47:43.416 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.393+0000 7f875ea00640 1 --2- 172.21.15.32:0/4247644840 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f8748077010 0x7f8748079520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:47:43.416 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.394+0000 7f875ea00640 1 -- 172.21.15.32:0/4247644840 <== mon.1 v2:172.21.15.118:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f87580b95c0 con 0x7f87680733f0 2024-08-03T20:47:43.416 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.394+0000 7f875ea00640 1 --2- 172.21.15.32:0/4247644840 >> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] conn(0x7f874807f2e0 0x7f8748081730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:47:43.416 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.394+0000 7f875ea00640 1 -- 172.21.15.32:0/4247644840 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f8758005310 con 0x7f874807f2e0 2024-08-03T20:47:43.417 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.394+0000 7f875ea00640 1 -- 172.21.15.32:0/4247644840 <== mon.1 v2:172.21.15.118:3300/0 6 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f87580b99b0 con 0x7f87680733f0 2024-08-03T20:47:43.417 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.394+0000 7ff45e400640 1 -- 172.21.15.32:0/2349670721 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7ff45016e8f0 con 0x7ff450053c50 2024-08-03T20:47:43.417 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.394+0000 7f8766000640 1 --2- 172.21.15.32:0/4247644840 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f8748077010 0x7f8748079520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:47:43.418 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.394+0000 7ff9a2200640 1 --2- 172.21.15.32:0/521504658 >> [v2:172.21.15.32:6808/3806708944,v1:172.21.15.32:6809/3806708944] conn(0x7ff97c07f350 0x7ff97c0817a0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:47:43.418 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.394+0000 7ff9a0e00640 1 --2- 172.21.15.32:0/521504658 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7ff97c0770e0 0x7ff97c0795f0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7ff988033000 tx=0x7ff988004730 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:47:43.418 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.394+0000 7fc507e00640 1 --2- 172.21.15.32:0/4288518052 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc5081f3cf0 0x7fc5081f4100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:43.418 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.395+0000 7fc507e00640 1 -- 172.21.15.32:0/4288518052 --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc5081f4640 con 0x7fc5080ae3e0 2024-08-03T20:47:43.418 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.395+0000 7fc50da00640 1 --2- 172.21.15.32:0/4288518052 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc5081f3cf0 0x7fc5081f4100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-03T20:47:43.418 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.395+0000 7fdcc7e00640 1 -- 172.21.15.32:0/3010512042 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fdcc81f88f0 msgr2=0x7fdcc81f8d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:47:43.419 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.395+0000 7fdcc7e00640 1 --2- 172.21.15.32:0/3010512042 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fdcc81f88f0 0x7fdcc81f8d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:43.419 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.395+0000 7fdcc7e00640 1 -- 172.21.15.32:0/3010512042 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fdcac005350 msgr2=0x7fdcc814d9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:47:43.422 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.395+0000 7fdcc7e00640 1 --2- 172.21.15.32:0/3010512042 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fdcac005350 0x7fdcc814d9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:43.422 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.395+0000 7fdcc7e00640 1 -- 172.21.15.32:0/3010512042 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdcc80b2a20 con 0x7fdcc81f9240 2024-08-03T20:47:43.422 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.395+0000 7f8767400640 1 --2- 172.21.15.32:0/4247644840 >> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] conn(0x7f874807f2e0 0x7f8748081730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:47:43.423 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.395+0000 7fdcc7400640 1 --2- 172.21.15.32:0/3010512042 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fdcac005350 0x7fdcc814d9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-03T20:47:43.423 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.395+0000 7fc507e00640 1 --2- 172.21.15.32:0/4288518052 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fc5080ae3e0 0x7fc508152af0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fc4f4037000 tx=0x7fc4f4004e30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:47:43.423 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.395+0000 7f8766000640 1 --2- 172.21.15.32:0/4247644840 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f8748077010 0x7f8748079520 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f876813fde0 tx=0x7f8754009060 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:47:43.424 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.395+0000 7f3b48c00640 1 -- 172.21.15.32:0/1634897216 --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f3b440c8110 con 0x7f3b440d1220 2024-08-03T20:47:43.424 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.396+0000 7fdcc7e00640 1 --2- 172.21.15.32:0/3010512042 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdcc81f9240 0x7fdcc814e620 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fdcc808d790 tx=0x7fdcc00319e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:47:43.424 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.396+0000 7fc504c00640 1 -- 172.21.15.32:0/4288518052 <== mon.2 v2:172.21.15.118:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4f40064e0 con 0x7fc5080ae3e0 2024-08-03T20:47:43.424 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.396+0000 7fc504c00640 1 -- 172.21.15.32:0/4288518052 <== mon.2 v2:172.21.15.118:3301/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fc4f4047070 con 0x7fc5080ae3e0 2024-08-03T20:47:43.424 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.396+0000 7fc504c00640 1 -- 172.21.15.32:0/4288518052 <== mon.2 v2:172.21.15.118:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4f4043860 con 0x7fc5080ae3e0 2024-08-03T20:47:43.424 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.396+0000 7fc50e400640 1 -- 172.21.15.32:0/4288518052 --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc508153030 con 0x7fc5080ae3e0 2024-08-03T20:47:43.425 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.396+0000 7ff999600640 1 -- 172.21.15.32:0/521504658 <== osd.2 v2:172.21.15.32:6808/3806708944 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7ff99002bec0 con 0x7ff97c07f350 2024-08-03T20:47:43.425 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.397+0000 7fdcbf400640 1 -- 172.21.15.32:0/3010512042 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdcc0036040 con 0x7fdcc81f9240 2024-08-03T20:47:43.425 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.397+0000 7fdcbf400640 1 -- 172.21.15.32:0/3010512042 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fdcc0031b50 con 0x7fdcc81f9240 2024-08-03T20:47:43.425 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.397+0000 7fdcbf400640 1 -- 172.21.15.32:0/3010512042 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdcc0036040 con 0x7fdcc81f9240 2024-08-03T20:47:43.425 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.397+0000 7fdcce000640 1 -- 172.21.15.32:0/3010512042 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdcc80b2ec0 con 0x7fdcc81f9240 2024-08-03T20:47:43.426 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.397+0000 7f8767400640 1 --2- 172.21.15.32:0/4247644840 >> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] conn(0x7f874807f2e0 0x7f8748081730 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:47:43.426 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.397+0000 7fdcce000640 1 -- 172.21.15.32:0/3010512042 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdcc80b3480 con 0x7fdcc81f9240 2024-08-03T20:47:43.426 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.398+0000 7ff454c00640 1 -- 172.21.15.32:0/2349670721 <== mon.1 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7ff44c041030 con 0x7ff450053c50 2024-08-03T20:47:43.427 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.397+0000 7f3b48c00640 1 -- 172.21.15.32:0/1634897216 --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f3b440cef30 con 0x7f3b440d1220 2024-08-03T20:47:43.427 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.398+0000 7f3b3a000640 1 -- 172.21.15.32:0/1634897216 <== mon.2 v2:172.21.15.118:3301/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f3b2c009b70 con 0x7f3b440d1220 2024-08-03T20:47:43.427 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.396+0000 7fc50e400640 1 -- 172.21.15.32:0/4288518052 --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fc5080ad620 con 0x7fc5080ae3e0 2024-08-03T20:47:43.427 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.398+0000 7fc4fe000640 1 -- 172.21.15.32:0/4288518052 --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fc4dc001030 con 0x7fc5080ae3e0 2024-08-03T20:47:43.427 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.400+0000 7f3b3a000640 1 --2- 172.21.15.32:0/1634897216 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f3b20077040 0x7f3b20079550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:47:43.428 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.400+0000 7fdcce000640 1 -- 172.21.15.32:0/3010512042 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fdca0001030 con 0x7fdcc81f9240 2024-08-03T20:47:43.428 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.400+0000 7f875ea00640 1 -- 172.21.15.32:0/4247644840 <== osd.6 v2:172.21.15.118:6825/801947386 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f8758005310 con 0x7f874807f2e0 2024-08-03T20:47:43.428 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.400+0000 7f3b3a000640 1 -- 172.21.15.32:0/1634897216 <== mon.2 v2:172.21.15.118:3301/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f3b2c097e60 con 0x7f3b440d1220 2024-08-03T20:47:43.428 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.400+0000 7f3b3a000640 1 --2- 172.21.15.32:0/1634897216 >> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] conn(0x7f3b2007f230 0x7f3b20081680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:47:43.429 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.400+0000 7f3b3a000640 1 -- 172.21.15.32:0/1634897216 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f3b20081d50 con 0x7f3b2007f230 2024-08-03T20:47:43.429 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.400+0000 7f3b41e00640 1 --2- 172.21.15.32:0/1634897216 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f3b20077040 0x7f3b20079550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:47:43.429 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.401+0000 7f3b3a000640 1 -- 172.21.15.32:0/1634897216 <== mon.2 v2:172.21.15.118:3301/0 6 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f3b2c09d060 con 0x7f3b440d1220 2024-08-03T20:47:43.429 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.400+0000 7ff454c00640 1 --2- 172.21.15.32:0/2349670721 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7ff438077060 0x7ff438079570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:47:43.429 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.400+0000 7ff454c00640 1 -- 172.21.15.32:0/2349670721 <== mon.1 v2:172.21.15.118:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7ff44c0b8f70 con 0x7ff450053c50 2024-08-03T20:47:43.429 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.401+0000 7ff454c00640 1 --2- 172.21.15.32:0/2349670721 >> [v2:172.21.15.32:6800/9088303,v1:172.21.15.32:6801/9088303] conn(0x7ff43807f330 0x7ff438081780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:47:43.429 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.401+0000 7ff454c00640 1 -- 172.21.15.32:0/2349670721 --> [v2:172.21.15.32:6800/9088303,v1:172.21.15.32:6801/9088303] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7ff44c009080 con 0x7ff43807f330 2024-08-03T20:47:43.429 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.401+0000 7ff454c00640 1 -- 172.21.15.32:0/2349670721 <== mon.1 v2:172.21.15.118:3300/0 6 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7ff44c0bb590 con 0x7ff450053c50 2024-08-03T20:47:43.430 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.401+0000 7f3b42800640 1 --2- 172.21.15.32:0/1634897216 >> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] conn(0x7f3b2007f230 0x7f3b20081680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:47:43.430 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.401+0000 7ff457e00640 1 --2- 172.21.15.32:0/2349670721 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7ff438077060 0x7ff438079570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:47:43.430 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.401+0000 7ff45da00640 1 --2- 172.21.15.32:0/2349670721 >> [v2:172.21.15.32:6800/9088303,v1:172.21.15.32:6801/9088303] conn(0x7ff43807f330 0x7ff438081780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:47:43.430 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.402+0000 7f3b41e00640 1 --2- 172.21.15.32:0/1634897216 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f3b20077040 0x7f3b20079550 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f3b34035000 tx=0x7f3b34004ab0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:47:43.431 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.401+0000 7fdcbf400640 1 -- 172.21.15.32:0/3010512042 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7fdcc0044030 con 0x7fdcc81f9240 2024-08-03T20:47:43.431 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.402+0000 7fc504c00640 1 -- 172.21.15.32:0/4288518052 <== mon.2 v2:172.21.15.118:3301/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7fc4f4043a00 con 0x7fc5080ae3e0 2024-08-03T20:47:43.431 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.402+0000 7f3b42800640 1 --2- 172.21.15.32:0/1634897216 >> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] conn(0x7f3b2007f230 0x7f3b20081680 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:47:43.431 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.402+0000 7ff457e00640 1 --2- 172.21.15.32:0/2349670721 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7ff438077060 0x7ff438079570 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7ff4500dc8d0 tx=0x7ff444008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:47:43.432 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.403+0000 7fdcbf400640 1 --2- 172.21.15.32:0/3010512042 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fdca8077210 0x7fdca8079720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:47:43.432 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.403+0000 7fdcbf400640 1 -- 172.21.15.32:0/3010512042 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7fdcc00b83d0 con 0x7fdcc81f9240 2024-08-03T20:47:43.432 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.403+0000 7fc504c00640 1 --2- 172.21.15.32:0/4288518052 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fc4e8077110 0x7fc4e8079620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:47:43.432 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.403+0000 7fdcbf400640 1 --2- 172.21.15.32:0/3010512042 >> [v2:172.21.15.118:6802/864752275,v1:172.21.15.118:6803/864752275] conn(0x7fdca807f460 0x7fdca80818b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:47:43.433 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.403+0000 7fdcbf400640 1 -- 172.21.15.32:0/3010512042 --> [v2:172.21.15.118:6802/864752275,v1:172.21.15.118:6803/864752275] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fdcc0041e50 con 0x7fdca807f460 2024-08-03T20:47:43.433 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.403+0000 7fdcbf400640 1 -- 172.21.15.32:0/3010512042 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7fdcc00bf050 con 0x7fdcc81f9240 2024-08-03T20:47:43.433 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.404+0000 7f3b3a000640 1 -- 172.21.15.32:0/1634897216 <== osd.5 v2:172.21.15.118:6810/2724891770 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f3b20081d50 con 0x7f3b2007f230 2024-08-03T20:47:43.433 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.403+0000 7fc504c00640 1 -- 172.21.15.32:0/4288518052 <== mon.2 v2:172.21.15.118:3301/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7fc4f40c26e0 con 0x7fc5080ae3e0 2024-08-03T20:47:43.433 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.404+0000 7fc504c00640 1 --2- 172.21.15.32:0/4288518052 >> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] conn(0x7fc4e807f380 0x7fc4e80817d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:47:43.433 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.404+0000 7fc504c00640 1 -- 172.21.15.32:0/4288518052 --> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fc4f4006330 con 0x7fc4e807f380 2024-08-03T20:47:43.434 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.404+0000 7fc504c00640 1 -- 172.21.15.32:0/4288518052 <== mon.2 v2:172.21.15.118:3301/0 6 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7fc4f40c2ad0 con 0x7fc5080ae3e0 2024-08-03T20:47:43.434 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.404+0000 7fc50da00640 1 --2- 172.21.15.32:0/4288518052 >> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] conn(0x7fc4e807f380 0x7fc4e80817d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:47:43.434 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.404+0000 7fc50d000640 1 --2- 172.21.15.32:0/4288518052 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fc4e8077110 0x7fc4e8079620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:47:43.434 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.404+0000 7fdcc6a00640 1 --2- 172.21.15.32:0/3010512042 >> [v2:172.21.15.118:6802/864752275,v1:172.21.15.118:6803/864752275] conn(0x7fdca807f460 0x7fdca80818b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:47:43.434 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.405+0000 7fc50da00640 1 --2- 172.21.15.32:0/4288518052 >> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] conn(0x7fc4e807f380 0x7fc4e80817d0 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:47:43.435 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.404+0000 7ff45da00640 1 --2- 172.21.15.32:0/2349670721 >> [v2:172.21.15.32:6800/9088303,v1:172.21.15.32:6801/9088303] conn(0x7ff43807f330 0x7ff438081780 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:47:43.435 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.405+0000 7fdcc6a00640 1 --2- 172.21.15.32:0/3010512042 >> [v2:172.21.15.118:6802/864752275,v1:172.21.15.118:6803/864752275] conn(0x7fdca807f460 0x7fdca80818b0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:47:43.435 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.405+0000 7fdcc7400640 1 --2- 172.21.15.32:0/3010512042 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fdca8077210 0x7fdca8079720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:47:43.435 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.406+0000 7fc50d000640 1 --2- 172.21.15.32:0/4288518052 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fc4e8077110 0x7fc4e8079620 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fc4f8007ba0 tx=0x7fc4f8009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:47:43.435 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.406+0000 7ff454c00640 1 -- 172.21.15.32:0/2349670721 <== osd.3 v2:172.21.15.32:6800/9088303 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7ff44c009080 con 0x7ff43807f330 2024-08-03T20:47:43.436 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.406+0000 7fdcc7400640 1 --2- 172.21.15.32:0/3010512042 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fdca8077210 0x7fdca8079720 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fdcb8004360 tx=0x7fdcb8007e70 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:47:43.436 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.406+0000 7fc504c00640 1 -- 172.21.15.32:0/4288518052 <== osd.7 v2:172.21.15.118:6818/653000048 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fc4f4006330 con 0x7fc4e807f380 2024-08-03T20:47:43.436 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.410+0000 7fdcbf400640 1 -- 172.21.15.32:0/3010512042 <== osd.4 v2:172.21.15.118:6802/864752275 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fdcc0041e50 con 0x7fdca807f460 2024-08-03T20:47:43.437 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.424+0000 7ff45e400640 1 -- 172.21.15.32:0/2349670721 --> [v2:172.21.15.32:6800/9088303,v1:172.21.15.32:6801/9088303] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7ff450057c80 con 0x7ff43807f330 2024-08-03T20:47:43.437 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.425+0000 7ff454c00640 1 -- 172.21.15.32:0/2349670721 <== osd.3 v2:172.21.15.32:6800/9088303 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7ff450057c80 con 0x7ff43807f330 2024-08-03T20:47:43.437 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.426+0000 7ff44a000640 1 -- 172.21.15.32:0/2349670721 >> [v2:172.21.15.32:6800/9088303,v1:172.21.15.32:6801/9088303] conn(0x7ff43807f330 msgr2=0x7ff438081780 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:47:43.437 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.426+0000 7ff44a000640 1 --2- 172.21.15.32:0/2349670721 >> [v2:172.21.15.32:6800/9088303,v1:172.21.15.32:6801/9088303] conn(0x7ff43807f330 0x7ff438081780 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:43.438 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.426+0000 7ff44a000640 1 -- 172.21.15.32:0/2349670721 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7ff438077060 msgr2=0x7ff438079570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:47:43.438 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.427+0000 7f6691c00640 1 -- 172.21.15.32:0/3729796857 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f6684055200 con 0x7f666c07f360 2024-08-03T20:47:43.438 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.426+0000 7ff44a000640 1 --2- 172.21.15.32:0/2349670721 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7ff438077060 0x7ff438079570 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7ff4500dc8d0 tx=0x7ff444008040 comp rx=0 tx=0).stop 2024-08-03T20:47:43.438 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.426+0000 7ff44a000640 1 -- 172.21.15.32:0/2349670721 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ff450053c50 msgr2=0x7ff45017e840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:47:43.438 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.426+0000 7ff44a000640 1 --2- 172.21.15.32:0/2349670721 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ff450053c50 0x7ff45017e840 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7ff44c000c00 tx=0x7ff44c003b90 comp rx=0 tx=0).stop 2024-08-03T20:47:43.438 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.426+0000 7ff45da00640 1 -- 172.21.15.32:0/2349670721 reap_dead start 2024-08-03T20:47:43.439 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.427+0000 7ff44a000640 1 -- 172.21.15.32:0/2349670721 shutdown_connections 2024-08-03T20:47:43.439 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.427+0000 7ff44a000640 1 -- 172.21.15.32:0/2349670721 >> 172.21.15.32:0/2349670721 conn(0x7ff45003a5f0 msgr2=0x7ff45003aa00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:47:43.439 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.427+0000 7ff44a000640 1 -- 172.21.15.32:0/2349670721 shutdown_connections 2024-08-03T20:47:43.439 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.428+0000 7ff44a000640 1 -- 172.21.15.32:0/2349670721 wait complete. 2024-08-03T20:47:43.439 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.428+0000 7f667fe00640 1 -- 172.21.15.32:0/3729796857 <== osd.0 v2:172.21.15.32:6816/3066156116 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f6684055200 con 0x7f666c07f360 2024-08-03T20:47:43.440 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.430+0000 7f6691c00640 1 -- 172.21.15.32:0/3729796857 >> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] conn(0x7f666c07f360 msgr2=0x7f666c0817b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:47:43.440 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.430+0000 7f6691c00640 1 --2- 172.21.15.32:0/3729796857 >> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] conn(0x7f666c07f360 0x7f666c0817b0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:43.440 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.431+0000 7f6691c00640 1 -- 172.21.15.32:0/3729796857 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f666c077090 msgr2=0x7f666c0795a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:47:43.440 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.431+0000 7f6691c00640 1 --2- 172.21.15.32:0/3729796857 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f666c077090 0x7f666c0795a0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f66841818c0 tx=0x7f6678006b30 comp rx=0 tx=0).stop 2024-08-03T20:47:43.440 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.431+0000 7f6691c00640 1 -- 172.21.15.32:0/3729796857 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6684051630 msgr2=0x7f66840e4180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:47:43.440 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.431+0000 7f6691c00640 1 --2- 172.21.15.32:0/3729796857 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6684051630 0x7f66840e4180 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f6680000c00 tx=0x7f6680005270 comp rx=0 tx=0).stop 2024-08-03T20:47:43.440 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.430+0000 7ff9a2c00640 1 -- 172.21.15.32:0/521504658 --> [v2:172.21.15.32:6808/3806708944,v1:172.21.15.32:6809/3806708944] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7ff99c0d1630 con 0x7ff97c07f350 2024-08-03T20:47:43.441 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.431+0000 7ff999600640 1 -- 172.21.15.32:0/521504658 <== osd.2 v2:172.21.15.32:6808/3806708944 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7ff99c0d1630 con 0x7ff97c07f350 2024-08-03T20:47:43.441 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.431+0000 7f6691200640 1 -- 172.21.15.32:0/3729796857 reap_dead start 2024-08-03T20:47:43.441 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.431+0000 7ff9a2c00640 1 -- 172.21.15.32:0/521504658 >> [v2:172.21.15.32:6808/3806708944,v1:172.21.15.32:6809/3806708944] conn(0x7ff97c07f350 msgr2=0x7ff97c0817a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:47:43.441 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.431+0000 7ff9a2c00640 1 --2- 172.21.15.32:0/521504658 >> [v2:172.21.15.32:6808/3806708944,v1:172.21.15.32:6809/3806708944] conn(0x7ff97c07f350 0x7ff97c0817a0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:43.442 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.431+0000 7f4b65e00640 1 -- 172.21.15.32:0/2531059969 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f4b58053150 con 0x7f4b4407f330 2024-08-03T20:47:43.442 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.432+0000 7f6691c00640 1 -- 172.21.15.32:0/3729796857 shutdown_connections 2024-08-03T20:47:43.442 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.432+0000 7f6691c00640 1 -- 172.21.15.32:0/3729796857 >> 172.21.15.32:0/3729796857 conn(0x7f668401d9a0 msgr2=0x7f6684005a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:47:43.442 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.432+0000 7f6691c00640 1 -- 172.21.15.32:0/3729796857 shutdown_connections 2024-08-03T20:47:43.442 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.432+0000 7f6691c00640 1 -- 172.21.15.32:0/3729796857 wait complete. 2024-08-03T20:47:43.442 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.432+0000 7f4b5cc00640 1 -- 172.21.15.32:0/2531059969 <== osd.1 v2:172.21.15.32:6824/1551241759 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f4b58053150 con 0x7f4b4407f330 2024-08-03T20:47:43.443 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.432+0000 7ff9a2c00640 1 -- 172.21.15.32:0/521504658 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7ff97c0770e0 msgr2=0x7ff97c0795f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:47:43.443 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.432+0000 7ff9a2c00640 1 --2- 172.21.15.32:0/521504658 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7ff97c0770e0 0x7ff97c0795f0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7ff988033000 tx=0x7ff988004730 comp rx=0 tx=0).stop 2024-08-03T20:47:43.443 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.432+0000 7ff9a2c00640 1 -- 172.21.15.32:0/521504658 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff99c0b3f20 msgr2=0x7ff99c09f560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:47:43.443 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.432+0000 7ff9a2c00640 1 --2- 172.21.15.32:0/521504658 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff99c0b3f20 0x7ff99c09f560 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7ff99000e9a0 tx=0x7ff99000ee60 comp rx=0 tx=0).stop 2024-08-03T20:47:43.443 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.432+0000 7ff9a2200640 1 -- 172.21.15.32:0/521504658 reap_dead start 2024-08-03T20:47:43.443 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.432+0000 7ff9a2c00640 1 -- 172.21.15.32:0/521504658 shutdown_connections 2024-08-03T20:47:43.443 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.432+0000 7ff9a2c00640 1 -- 172.21.15.32:0/521504658 >> 172.21.15.32:0/521504658 conn(0x7ff99c0d4080 msgr2=0x7ff99c06e130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:47:43.444 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.433+0000 7f4b65e00640 1 -- 172.21.15.32:0/2531059969 >> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] conn(0x7f4b4407f330 msgr2=0x7f4b44081780 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:47:43.444 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.433+0000 7f4b65e00640 1 --2- 172.21.15.32:0/2531059969 >> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] conn(0x7f4b4407f330 0x7f4b44081780 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:43.444 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.434+0000 7f4b65e00640 1 -- 172.21.15.32:0/2531059969 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f4b44077060 msgr2=0x7f4b44079570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:47:43.444 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.434+0000 7f4b65e00640 1 --2- 172.21.15.32:0/2531059969 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f4b44077060 0x7f4b44079570 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f4b54001220 tx=0x7f4b54004640 comp rx=0 tx=0).stop 2024-08-03T20:47:43.445 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.434+0000 7f4b65e00640 1 -- 172.21.15.32:0/2531059969 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4b58054190 msgr2=0x7f4b581820a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:47:43.445 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.434+0000 7f4b65e00640 1 --2- 172.21.15.32:0/2531059969 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4b58054190 0x7f4b581820a0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f4b5000b3e0 tx=0x7f4b5000b8a0 comp rx=0 tx=0).stop 2024-08-03T20:47:43.445 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.434+0000 7f4b65400640 1 -- 172.21.15.32:0/2531059969 reap_dead start 2024-08-03T20:47:43.445 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.434+0000 7f4b65e00640 1 -- 172.21.15.32:0/2531059969 shutdown_connections 2024-08-03T20:47:43.445 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.434+0000 7f4b65e00640 1 -- 172.21.15.32:0/2531059969 >> 172.21.15.32:0/2531059969 conn(0x7f4b5803a680 msgr2=0x7f4b58039ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:47:43.445 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.435+0000 7f4b65e00640 1 -- 172.21.15.32:0/2531059969 shutdown_connections 2024-08-03T20:47:43.445 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.435+0000 7ff9a2c00640 1 -- 172.21.15.32:0/521504658 shutdown_connections 2024-08-03T20:47:43.445 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.435+0000 7f4b65e00640 1 -- 172.21.15.32:0/2531059969 wait complete. 2024-08-03T20:47:43.446 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.435+0000 7ff9a2c00640 1 -- 172.21.15.32:0/521504658 wait complete. 2024-08-03T20:47:43.446 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.436+0000 7f876d600640 1 -- 172.21.15.32:0/4247644840 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f8740002dd0 con 0x7f874807f2e0 2024-08-03T20:47:43.446 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.437+0000 7f875ea00640 1 -- 172.21.15.32:0/4247644840 <== osd.6 v2:172.21.15.118:6825/801947386 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f8740002dd0 con 0x7f874807f2e0 2024-08-03T20:47:43.448 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.437+0000 7f876d600640 1 -- 172.21.15.32:0/4247644840 >> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] conn(0x7f874807f2e0 msgr2=0x7f8748081730 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:47:43.448 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.437+0000 7f876d600640 1 --2- 172.21.15.32:0/4247644840 >> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] conn(0x7f874807f2e0 0x7f8748081730 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:43.448 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.438+0000 7f876d600640 1 -- 172.21.15.32:0/4247644840 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f8748077010 msgr2=0x7f8748079520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:47:43.448 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.438+0000 7f876d600640 1 --2- 172.21.15.32:0/4247644840 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f8748077010 0x7f8748079520 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f876813fde0 tx=0x7f8754009060 comp rx=0 tx=0).stop 2024-08-03T20:47:43.448 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.438+0000 7f876d600640 1 -- 172.21.15.32:0/4247644840 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f87680733f0 msgr2=0x7f87681fcca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:47:43.448 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.438+0000 7f876d600640 1 --2- 172.21.15.32:0/4247644840 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f87680733f0 0x7f87681fcca0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f8758034040 tx=0x7f8758003960 comp rx=0 tx=0).stop 2024-08-03T20:47:43.448 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.438+0000 7f8767400640 1 -- 172.21.15.32:0/4247644840 reap_dead start 2024-08-03T20:47:43.448 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.438+0000 7f876d600640 1 -- 172.21.15.32:0/4247644840 shutdown_connections 2024-08-03T20:47:43.448 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.438+0000 7f876d600640 1 -- 172.21.15.32:0/4247644840 >> 172.21.15.32:0/4247644840 conn(0x7f87680b1bd0 msgr2=0x7f87681496d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:47:43.448 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.439+0000 7f876d600640 1 -- 172.21.15.32:0/4247644840 shutdown_connections 2024-08-03T20:47:43.448 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.439+0000 7f876d600640 1 -- 172.21.15.32:0/4247644840 wait complete. 2024-08-03T20:47:43.449 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.440+0000 7fc4fe000640 1 -- 172.21.15.32:0/4288518052 --> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fc4dc002dd0 con 0x7fc4e807f380 2024-08-03T20:47:43.449 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.441+0000 7fc504c00640 1 -- 172.21.15.32:0/4288518052 <== osd.7 v2:172.21.15.118:6818/653000048 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7fc4dc002dd0 con 0x7fc4e807f380 2024-08-03T20:47:43.449 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.442+0000 7f3b48c00640 1 -- 172.21.15.32:0/1634897216 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f3b4415cb00 con 0x7f3b2007f230 2024-08-03T20:47:43.449 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.443+0000 7f3b3a000640 1 -- 172.21.15.32:0/1634897216 <== osd.5 v2:172.21.15.118:6810/2724891770 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f3b4415cb00 con 0x7f3b2007f230 2024-08-03T20:47:43.450 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.443+0000 7fc4fe000640 1 -- 172.21.15.32:0/4288518052 >> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] conn(0x7fc4e807f380 msgr2=0x7fc4e80817d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:47:43.450 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.443+0000 7fc4fe000640 1 --2- 172.21.15.32:0/4288518052 >> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] conn(0x7fc4e807f380 0x7fc4e80817d0 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:43.450 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.444+0000 7f3b48c00640 1 -- 172.21.15.32:0/1634897216 >> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] conn(0x7f3b2007f230 msgr2=0x7f3b20081680 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:47:43.450 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.444+0000 7f3b48c00640 1 --2- 172.21.15.32:0/1634897216 >> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] conn(0x7f3b2007f230 0x7f3b20081680 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:43.451 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.444+0000 7fc4fe000640 1 -- 172.21.15.32:0/4288518052 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fc4e8077110 msgr2=0x7fc4e8079620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:47:43.451 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.444+0000 7fc4fe000640 1 --2- 172.21.15.32:0/4288518052 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fc4e8077110 0x7fc4e8079620 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fc4f8007ba0 tx=0x7fc4f8009040 comp rx=0 tx=0).stop 2024-08-03T20:47:43.451 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.444+0000 7f3b48c00640 1 -- 172.21.15.32:0/1634897216 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f3b20077040 msgr2=0x7f3b20079550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:47:43.451 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.444+0000 7f3b48c00640 1 --2- 172.21.15.32:0/1634897216 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f3b20077040 0x7f3b20079550 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f3b34035000 tx=0x7f3b34004ab0 comp rx=0 tx=0).stop 2024-08-03T20:47:43.451 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.444+0000 7f3b48c00640 1 -- 172.21.15.32:0/1634897216 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f3b440d1220 msgr2=0x7f3b4415c300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:47:43.452 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.444+0000 7f3b48c00640 1 --2- 172.21.15.32:0/1634897216 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f3b440d1220 0x7f3b4415c300 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f3b2c00d780 tx=0x7f3b2c00dc40 comp rx=0 tx=0).stop 2024-08-03T20:47:43.452 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.444+0000 7fc4fe000640 1 -- 172.21.15.32:0/4288518052 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fc5080ae3e0 msgr2=0x7fc508152af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:47:43.452 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.444+0000 7fc4fe000640 1 --2- 172.21.15.32:0/4288518052 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fc5080ae3e0 0x7fc508152af0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fc4f4037000 tx=0x7fc4f4004e30 comp rx=0 tx=0).stop 2024-08-03T20:47:43.452 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.445+0000 7f3b42800640 1 -- 172.21.15.32:0/1634897216 reap_dead start 2024-08-03T20:47:43.453 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.445+0000 7f3b48c00640 1 -- 172.21.15.32:0/1634897216 shutdown_connections 2024-08-03T20:47:43.453 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.445+0000 7f3b48c00640 1 -- 172.21.15.32:0/1634897216 >> 172.21.15.32:0/1634897216 conn(0x7f3b4409d620 msgr2=0x7f3b440b7850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:47:43.453 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.445+0000 7fc50da00640 1 -- 172.21.15.32:0/4288518052 reap_dead start 2024-08-03T20:47:43.453 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.445+0000 7fc4fe000640 1 -- 172.21.15.32:0/4288518052 shutdown_connections 2024-08-03T20:47:43.453 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.445+0000 7fc4fe000640 1 -- 172.21.15.32:0/4288518052 >> 172.21.15.32:0/4288518052 conn(0x7fc5080b1d50 msgr2=0x7fc5080b0380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:47:43.453 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.446+0000 7fc4fe000640 1 -- 172.21.15.32:0/4288518052 shutdown_connections 2024-08-03T20:47:43.454 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.446+0000 7fc4fe000640 1 -- 172.21.15.32:0/4288518052 wait complete. 2024-08-03T20:47:43.454 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.446+0000 7f3b48c00640 1 -- 172.21.15.32:0/1634897216 shutdown_connections 2024-08-03T20:47:43.454 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.447+0000 7f3b48c00640 1 -- 172.21.15.32:0/1634897216 wait complete. 2024-08-03T20:47:43.454 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.450+0000 7fdcce000640 1 -- 172.21.15.32:0/3010512042 --> [v2:172.21.15.118:6802/864752275,v1:172.21.15.118:6803/864752275] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fdca0002dd0 con 0x7fdca807f460 2024-08-03T20:47:43.454 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.451+0000 7fdcbf400640 1 -- 172.21.15.32:0/3010512042 <== osd.4 v2:172.21.15.118:6802/864752275 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7fdca0002dd0 con 0x7fdca807f460 2024-08-03T20:47:43.456 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.452+0000 7fdcce000640 1 -- 172.21.15.32:0/3010512042 >> [v2:172.21.15.118:6802/864752275,v1:172.21.15.118:6803/864752275] conn(0x7fdca807f460 msgr2=0x7fdca80818b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:47:43.456 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.452+0000 7fdcce000640 1 --2- 172.21.15.32:0/3010512042 >> [v2:172.21.15.118:6802/864752275,v1:172.21.15.118:6803/864752275] conn(0x7fdca807f460 0x7fdca80818b0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:43.456 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.453+0000 7fdcce000640 1 -- 172.21.15.32:0/3010512042 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fdca8077210 msgr2=0x7fdca8079720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:47:43.456 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.453+0000 7fdcce000640 1 --2- 172.21.15.32:0/3010512042 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fdca8077210 0x7fdca8079720 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fdcb8004360 tx=0x7fdcb8007e70 comp rx=0 tx=0).stop 2024-08-03T20:47:43.456 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.453+0000 7fdcce000640 1 -- 172.21.15.32:0/3010512042 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdcc81f9240 msgr2=0x7fdcc814e620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:47:43.456 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.453+0000 7fdcce000640 1 --2- 172.21.15.32:0/3010512042 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdcc81f9240 0x7fdcc814e620 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fdcc808d790 tx=0x7fdcc00319e0 comp rx=0 tx=0).stop 2024-08-03T20:47:43.456 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.453+0000 7fdcc7e00640 1 -- 172.21.15.32:0/3010512042 reap_dead start 2024-08-03T20:47:43.456 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.454+0000 7fdcce000640 1 -- 172.21.15.32:0/3010512042 shutdown_connections 2024-08-03T20:47:43.456 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.454+0000 7fdcce000640 1 -- 172.21.15.32:0/3010512042 >> 172.21.15.32:0/3010512042 conn(0x7fdcc809d660 msgr2=0x7fdcc81612d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:47:43.457 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.456+0000 7fdcce000640 1 -- 172.21.15.32:0/3010512042 shutdown_connections 2024-08-03T20:47:43.457 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.456+0000 7fdcce000640 1 -- 172.21.15.32:0/3010512042 wait complete. 2024-08-03T20:47:43.463 INFO:teuthology.orchestra.run.smithi032.stdout:60129542148 2024-08-03T20:47:43.463 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-08-03T20:47:43.475 INFO:teuthology.orchestra.run.smithi032.stdout:64424509447 2024-08-03T20:47:43.476 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-08-03T20:47:43.477 INFO:teuthology.orchestra.run.smithi032.stdout:60129542149 2024-08-03T20:47:43.478 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-08-03T20:47:43.479 INFO:teuthology.orchestra.run.smithi032.stdout:64424509445 2024-08-03T20:47:43.479 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-08-03T20:47:43.483 INFO:teuthology.orchestra.run.smithi032.stdout:64424509446 2024-08-03T20:47:43.483 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2024-08-03T20:47:43.492 INFO:teuthology.orchestra.run.smithi032.stdout:64424509446 2024-08-03T20:47:43.492 INFO:teuthology.orchestra.run.smithi032.stdout:64424509445 2024-08-03T20:47:43.493 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-08-03T20:47:43.493 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2024-08-03T20:47:43.497 INFO:teuthology.orchestra.run.smithi032.stdout:60129542150 2024-08-03T20:47:43.497 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-08-03T20:47:43.880 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.878+0000 7f52db200640 1 -- 172.21.15.32:0/3096807412 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f52d4053b50 msgr2=0x7f52d4036240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:47:43.880 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.878+0000 7f52db200640 1 --2- 172.21.15.32:0/3096807412 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f52d4053b50 0x7f52d4036240 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f52cc009830 tx=0x7f52cc02a420 comp rx=0 tx=0).stop 2024-08-03T20:47:43.880 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.878+0000 7f52db200640 1 -- 172.21.15.32:0/3096807412 shutdown_connections 2024-08-03T20:47:43.880 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.878+0000 7f52db200640 1 --2- 172.21.15.32:0/3096807412 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f52d4008170 0x7f52d4053610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:43.880 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.878+0000 7f52db200640 1 --2- 172.21.15.32:0/3096807412 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f52d4053b50 0x7f52d4036240 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:43.880 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.878+0000 7f52db200640 1 -- 172.21.15.32:0/3096807412 >> 172.21.15.32:0/3096807412 conn(0x7f52d401fbe0 msgr2=0x7f52d4056940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:47:43.881 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.878+0000 7f52db200640 1 -- 172.21.15.32:0/3096807412 shutdown_connections 2024-08-03T20:47:43.881 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.878+0000 7f52db200640 1 -- 172.21.15.32:0/3096807412 wait complete. 2024-08-03T20:47:43.884 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.882+0000 7f26a3c00640 1 -- 172.21.15.32:0/766597492 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f26940e1000 msgr2=0x7f26940532a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:47:43.884 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.882+0000 7f26a3c00640 1 --2- 172.21.15.32:0/766597492 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f26940e1000 0x7f26940532a0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f269c06bb10 tx=0x7f269c0a5ea0 comp rx=0 tx=0).stop 2024-08-03T20:47:43.885 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.883+0000 7f26a3c00640 1 -- 172.21.15.32:0/766597492 shutdown_connections 2024-08-03T20:47:43.885 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.883+0000 7f26a3c00640 1 --2- 172.21.15.32:0/766597492 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f26940537e0 0x7f2694054650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:43.885 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.883+0000 7f26a3c00640 1 --2- 172.21.15.32:0/766597492 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f26940e1000 0x7f26940532a0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:43.885 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.883+0000 7f26a3c00640 1 -- 172.21.15.32:0/766597492 >> 172.21.15.32:0/766597492 conn(0x7f269401fbd0 msgr2=0x7f2694056930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:47:43.885 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.884+0000 7f26a3c00640 1 -- 172.21.15.32:0/766597492 shutdown_connections 2024-08-03T20:47:43.886 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.884+0000 7f52db200640 1 Processor -- start 2024-08-03T20:47:43.886 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.884+0000 7f26a3c00640 1 -- 172.21.15.32:0/766597492 wait complete. 2024-08-03T20:47:43.886 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.884+0000 7f52db200640 1 -- start start 2024-08-03T20:47:43.887 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.885+0000 7fae34600640 1 -- 172.21.15.32:0/3359969188 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fae2c06f070 msgr2=0x7fae2c0cedb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:47:43.887 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.886+0000 7fae34600640 1 --2- 172.21.15.32:0/3359969188 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fae2c06f070 0x7fae2c0cedb0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fae18005fd0 tx=0x7fae1802a5d0 comp rx=0 tx=0).stop 2024-08-03T20:47:43.887 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.886+0000 7fae34600640 1 -- 172.21.15.32:0/3359969188 shutdown_connections 2024-08-03T20:47:43.887 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.886+0000 7fae34600640 1 --2- 172.21.15.32:0/3359969188 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fae2c06f070 0x7fae2c0cedb0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:43.888 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.886+0000 7fae34600640 1 --2- 172.21.15.32:0/3359969188 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fae2c0b3bb0 0x7fae2c06eb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:43.888 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.886+0000 7fae34600640 1 -- 172.21.15.32:0/3359969188 >> 172.21.15.32:0/3359969188 conn(0x7fae2c0b1890 msgr2=0x7fae2c0b1ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:47:43.888 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.886+0000 7fae34600640 1 -- 172.21.15.32:0/3359969188 shutdown_connections 2024-08-03T20:47:43.888 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.887+0000 7fae34600640 1 -- 172.21.15.32:0/3359969188 wait complete. 2024-08-03T20:47:43.889 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.887+0000 7f26a3c00640 1 Processor -- start 2024-08-03T20:47:43.889 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.888+0000 7f26a3c00640 1 -- start start 2024-08-03T20:47:43.890 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.888+0000 7f52db200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f52d4007d60 0x7f52d40e2ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:47:43.890 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.888+0000 7f52db200640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f52d4008170 0x7f52d4020650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:47:43.890 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.888+0000 7f52db200640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f52d4053b50 0x7f52d4020b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:47:43.890 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.888+0000 7f52db200640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f52d4036780 con 0x7f52d4007d60 2024-08-03T20:47:43.890 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.888+0000 7f52db200640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f52d4037b10 con 0x7f52d4053b50 2024-08-03T20:47:43.890 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.888+0000 7f52db200640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7f52d4036900 con 0x7f52d4008170 2024-08-03T20:47:43.890 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.889+0000 7f52d9400640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f52d4008170 0x7f52d4020650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:47:43.890 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.889+0000 7f52d9400640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f52d4008170 0x7f52d4020650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3301/0 says I am v2:172.21.15.32:36490/0 (socket says 172.21.15.32:36490) 2024-08-03T20:47:43.891 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.889+0000 7f52d9400640 1 -- 172.21.15.32:0/591383566 learned_addr learned my addr 172.21.15.32:0/591383566 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:47:43.891 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.889+0000 7f52da800640 1 --2- 172.21.15.32:0/591383566 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f52d4053b50 0x7f52d4020b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:47:43.891 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.889+0000 7f52d9e00640 1 --2- 172.21.15.32:0/591383566 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f52d4007d60 0x7f52d40e2ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:47:43.891 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.889+0000 7f52d9400640 1 -- 172.21.15.32:0/591383566 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f52d4053b50 msgr2=0x7f52d4020b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:47:43.891 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.889+0000 7f52d9400640 1 --2- 172.21.15.32:0/591383566 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f52d4053b50 0x7f52d4020b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:43.892 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.889+0000 7f52d9400640 1 -- 172.21.15.32:0/591383566 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f52d4007d60 msgr2=0x7f52d40e2ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:47:43.892 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.889+0000 7f52d9400640 1 --2- 172.21.15.32:0/591383566 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f52d4007d60 0x7f52d40e2ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:43.892 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.889+0000 7f52d9400640 1 -- 172.21.15.32:0/591383566 --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f52d40e6790 con 0x7f52d4008170 2024-08-03T20:47:43.892 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.890+0000 7fae34600640 1 Processor -- start 2024-08-03T20:47:43.892 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.890+0000 7f52d9e00640 1 --2- 172.21.15.32:0/591383566 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f52d4007d60 0x7f52d40e2ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-03T20:47:43.892 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.890+0000 7f52d9400640 1 --2- 172.21.15.32:0/591383566 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f52d4008170 0x7f52d4020650 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f52c800d6a0 tx=0x7f52c800db60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:47:43.893 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.890+0000 7f26a3c00640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f26940537e0 0x7f2694008290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:47:43.893 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.890+0000 7f26a3c00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f2694054b90 0x7f2694036f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:47:43.893 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.891+0000 7f26a3c00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f26940e1000 0x7f26940359c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:47:43.893 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.891+0000 7f26a3c00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f26940e5400 con 0x7f26940e1000 2024-08-03T20:47:43.893 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.891+0000 7f26a3c00640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f26940e5280 con 0x7f2694054b90 2024-08-03T20:47:43.893 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.891+0000 7f26a3c00640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7f26940e5580 con 0x7f26940537e0 2024-08-03T20:47:43.894 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.890+0000 7fae34600640 1 -- start start 2024-08-03T20:47:43.894 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.891+0000 7f269be00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f2694054b90 0x7f2694036f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:47:43.894 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.891+0000 7f26a1200640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f26940537e0 0x7f2694008290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:47:43.894 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.891+0000 7f269be00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f2694054b90 0x7f2694036f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.32:45944/0 (socket says 172.21.15.32:45944) 2024-08-03T20:47:43.894 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.891+0000 7f269be00640 1 -- 172.21.15.32:0/244158632 learned_addr learned my addr 172.21.15.32:0/244158632 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:47:43.894 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.892+0000 7f26a1c00640 1 --2- 172.21.15.32:0/244158632 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f26940e1000 0x7f26940359c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:47:43.895 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.892+0000 7f269be00640 1 -- 172.21.15.32:0/244158632 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f26940537e0 msgr2=0x7f2694008290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:47:43.895 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.892+0000 7f269be00640 1 --2- 172.21.15.32:0/244158632 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f26940537e0 0x7f2694008290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:43.895 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.892+0000 7f269be00640 1 -- 172.21.15.32:0/244158632 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f26940e1000 msgr2=0x7f26940359c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:47:43.895 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.892+0000 7f269be00640 1 --2- 172.21.15.32:0/244158632 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f26940e1000 0x7f26940359c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:43.895 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.892+0000 7f269be00640 1 -- 172.21.15.32:0/244158632 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f26940374d0 con 0x7f2694054b90 2024-08-03T20:47:43.895 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.892+0000 7f26a1c00640 1 --2- 172.21.15.32:0/244158632 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f26940e1000 0x7f26940359c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-03T20:47:43.895 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.891+0000 7f52d2000640 1 -- 172.21.15.32:0/591383566 <== mon.2 v2:172.21.15.118:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52c8017070 con 0x7f52d4008170 2024-08-03T20:47:43.895 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.891+0000 7f52d2000640 1 -- 172.21.15.32:0/591383566 <== mon.2 v2:172.21.15.118:3301/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f52c8009b10 con 0x7f52d4008170 2024-08-03T20:47:43.895 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.891+0000 7f52d2000640 1 -- 172.21.15.32:0/591383566 <== mon.2 v2:172.21.15.118:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52c8010460 con 0x7f52d4008170 2024-08-03T20:47:43.896 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.893+0000 7f269be00640 1 --2- 172.21.15.32:0/244158632 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f2694054b90 0x7f2694036f90 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f2688009fd0 tx=0x7f268800ec20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:47:43.896 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.893+0000 7f2698c00640 1 -- 172.21.15.32:0/244158632 <== mon.1 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f268800ccc0 con 0x7f2694054b90 2024-08-03T20:47:43.896 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.893+0000 7f26a3c00640 1 -- 172.21.15.32:0/244158632 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2694008950 con 0x7f2694054b90 2024-08-03T20:47:43.896 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.893+0000 7f26a3c00640 1 -- 172.21.15.32:0/244158632 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f26940e3650 con 0x7f2694054b90 2024-08-03T20:47:43.896 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.893+0000 7f52db200640 1 -- 172.21.15.32:0/591383566 --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f52d40e3c00 con 0x7f52d4008170 2024-08-03T20:47:43.896 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.893+0000 7f52db200640 1 -- 172.21.15.32:0/591383566 --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f52d417f400 con 0x7f52d4008170 2024-08-03T20:47:43.897 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.894+0000 7f2698c00640 1 -- 172.21.15.32:0/244158632 <== mon.1 v2:172.21.15.118:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f268800ee70 con 0x7f2694054b90 2024-08-03T20:47:43.897 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.894+0000 7fae34600640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fae2c06f070 0x7fae2c148eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:47:43.897 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.894+0000 7f2698c00640 1 -- 172.21.15.32:0/244158632 <== mon.1 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f268800ce60 con 0x7f2694054b90 2024-08-03T20:47:43.898 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.894+0000 7fae34600640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fae2c1554a0 0x7fae2c1532f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:47:43.898 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.894+0000 7fae34600640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fae2c1493f0 0x7fae2c153830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:47:43.898 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.895+0000 7fae34600640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fae2c0cf7f0 con 0x7fae2c06f070 2024-08-03T20:47:43.898 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.895+0000 7fae34600640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7fae2c0cf670 con 0x7fae2c1554a0 2024-08-03T20:47:43.898 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.895+0000 7fae34600640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7fae2c0cf970 con 0x7fae2c1493f0 2024-08-03T20:47:43.898 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.895+0000 7fae31c00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fae2c06f070 0x7fae2c148eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:47:43.898 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.895+0000 7fae31c00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fae2c06f070 0x7fae2c148eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:57470/0 (socket says 172.21.15.32:57470) 2024-08-03T20:47:43.898 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.895+0000 7fae31c00640 1 -- 172.21.15.32:0/3624013074 learned_addr learned my addr 172.21.15.32:0/3624013074 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:47:43.899 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.896+0000 7fae31200640 1 --2- 172.21.15.32:0/3624013074 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fae2c1554a0 0x7fae2c1532f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:47:43.899 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.896+0000 7fae31c00640 1 -- 172.21.15.32:0/3624013074 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fae2c1493f0 msgr2=0x7fae2c153830 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-08-03T20:47:43.899 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.896+0000 7fae31c00640 1 --2- 172.21.15.32:0/3624013074 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fae2c1493f0 0x7fae2c153830 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:43.899 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.896+0000 7fae31c00640 1 -- 172.21.15.32:0/3624013074 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fae2c1554a0 msgr2=0x7fae2c1532f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:47:43.899 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.896+0000 7fae31c00640 1 --2- 172.21.15.32:0/3624013074 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fae2c1554a0 0x7fae2c1532f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:43.899 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.896+0000 7fae31c00640 1 -- 172.21.15.32:0/3624013074 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fae2c1fcfe0 con 0x7fae2c06f070 2024-08-03T20:47:43.899 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.897+0000 7fae31c00640 1 --2- 172.21.15.32:0/3624013074 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fae2c06f070 0x7fae2c148eb0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fae2000c780 tx=0x7fae2000cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:47:43.900 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.897+0000 7f774a000640 1 -- 172.21.15.32:0/3918632437 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f77440b4860 msgr2=0x7f77440b4c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:47:43.900 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.897+0000 7f774a000640 1 --2- 172.21.15.32:0/3918632437 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f77440b4860 0x7f77440b4c70 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f7744071de0 tx=0x7f773c02a9a0 comp rx=0 tx=0).stop 2024-08-03T20:47:43.900 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.898+0000 7fae29600640 1 -- 172.21.15.32:0/3624013074 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae20017070 con 0x7fae2c06f070 2024-08-03T20:47:43.900 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.898+0000 7fae29600640 1 -- 172.21.15.32:0/3624013074 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fae20005070 con 0x7fae2c06f070 2024-08-03T20:47:43.900 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.898+0000 7fae29600640 1 -- 172.21.15.32:0/3624013074 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae2000a640 con 0x7fae2c06f070 2024-08-03T20:47:43.901 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.898+0000 7f774a000640 1 -- 172.21.15.32:0/3918632437 shutdown_connections 2024-08-03T20:47:43.901 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.898+0000 7f774a000640 1 --2- 172.21.15.32:0/3918632437 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f77440b4860 0x7f77440b4c70 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:43.901 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.898+0000 7f774a000640 1 --2- 172.21.15.32:0/3918632437 >> v2:172.21.15.32:3300/0 conn(0x7f7744073b70 0x7f77440b3b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:43.901 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.898+0000 7f774a000640 1 --2- 172.21.15.32:0/3918632437 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f77440cf4a0 0x7f77440744a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:43.901 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.898+0000 7f774a000640 1 -- 172.21.15.32:0/3918632437 >> 172.21.15.32:0/3918632437 conn(0x7f77440b1c40 msgr2=0x7f77440b2050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:47:43.901 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.898+0000 7fae34600640 1 -- 172.21.15.32:0/3624013074 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fae2c1fd2c0 con 0x7fae2c06f070 2024-08-03T20:47:43.902 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.898+0000 7f774a000640 1 -- 172.21.15.32:0/3918632437 shutdown_connections 2024-08-03T20:47:43.902 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.898+0000 7fae34600640 1 -- 172.21.15.32:0/3624013074 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fae2c158230 con 0x7fae2c06f070 2024-08-03T20:47:43.902 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.896+0000 7f52db200640 1 -- 172.21.15.32:0/591383566 --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f52d4053610 con 0x7f52d4008170 2024-08-03T20:47:43.903 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.897+0000 7f52d2000640 1 -- 172.21.15.32:0/591383566 <== mon.2 v2:172.21.15.118:3301/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f52c801a070 con 0x7f52d4008170 2024-08-03T20:47:43.903 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.899+0000 7f52d2000640 1 --2- 172.21.15.32:0/591383566 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f52bc077110 0x7f52bc079620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:47:43.903 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.900+0000 7f52d2000640 1 -- 172.21.15.32:0/591383566 <== mon.2 v2:172.21.15.118:3301/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f52c8098720 con 0x7f52d4008170 2024-08-03T20:47:43.903 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.900+0000 7f52d9e00640 1 --2- 172.21.15.32:0/591383566 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f52bc077110 0x7f52bc079620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:47:43.903 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.901+0000 7f52d9e00640 1 --2- 172.21.15.32:0/591383566 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f52bc077110 0x7f52bc079620 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f52dc066470 tx=0x7f52dc070040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:47:43.903 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.901+0000 7fae34600640 1 -- 172.21.15.32:0/3624013074 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fae2c1fd7e0 con 0x7fae2c06f070 2024-08-03T20:47:43.903 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.898+0000 7f774a000640 1 -- 172.21.15.32:0/3918632437 wait complete. 2024-08-03T20:47:43.904 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.899+0000 7f2698c00640 1 -- 172.21.15.32:0/244158632 <== mon.1 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f2688005260 con 0x7f2694054b90 2024-08-03T20:47:43.904 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.900+0000 7f26a3c00640 1 -- 172.21.15.32:0/244158632 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2678005290 con 0x7f2694054b90 2024-08-03T20:47:43.904 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.901+0000 7f2698c00640 1 --2- 172.21.15.32:0/244158632 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f267c077010 0x7f267c079520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:47:43.904 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.901+0000 7f2698c00640 1 -- 172.21.15.32:0/244158632 <== mon.1 v2:172.21.15.118:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f2688099170 con 0x7f2694054b90 2024-08-03T20:47:43.904 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.901+0000 7f26a1200640 1 --2- 172.21.15.32:0/244158632 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f267c077010 0x7f267c079520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:47:43.904 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.902+0000 7f774a000640 1 Processor -- start 2024-08-03T20:47:43.905 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.902+0000 7f26a1200640 1 --2- 172.21.15.32:0/244158632 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f267c077010 0x7f267c079520 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f269400b7c0 tx=0x7f269c0a8040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:47:43.905 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.902+0000 7f774a000640 1 -- start start 2024-08-03T20:47:43.905 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.903+0000 7fae29600640 1 -- 172.21.15.32:0/3624013074 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7fae2001a070 con 0x7fae2c06f070 2024-08-03T20:47:43.906 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.905+0000 7f52d2000640 1 -- 172.21.15.32:0/591383566 <== mon.2 v2:172.21.15.118:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7f52d40e6790 con 0x7f52d4008170 2024-08-03T20:47:43.906 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.904+0000 7fae29600640 1 --2- 172.21.15.32:0/3624013074 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fae0c077010 0x7fae0c079520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:47:43.906 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.905+0000 7fae29600640 1 -- 172.21.15.32:0/3624013074 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7fae20097bb0 con 0x7fae2c06f070 2024-08-03T20:47:43.907 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.904+0000 7f774a000640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f7744073b70 0x7f7744148210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:47:43.907 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.904+0000 7f774a000640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f77440b4860 0x7f7744148750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:47:43.907 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.904+0000 7f774a000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f77440cf4a0 0x7f774414b500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:47:43.907 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.905+0000 7f774a000640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f77440cdd60 con 0x7f77440cf4a0 2024-08-03T20:47:43.907 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.905+0000 7f774a000640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f77440cdbe0 con 0x7f77440b4860 2024-08-03T20:47:43.907 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.905+0000 7f774a000640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7f77440cdee0 con 0x7f7744073b70 2024-08-03T20:47:43.907 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.905+0000 7f7743400640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f7744073b70 0x7f7744148210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:47:43.908 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.905+0000 7f7743400640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f7744073b70 0x7f7744148210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3301/0 says I am v2:172.21.15.32:36534/0 (socket says 172.21.15.32:36534) 2024-08-03T20:47:43.908 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.905+0000 7f7743400640 1 -- 172.21.15.32:0/120322178 learned_addr learned my addr 172.21.15.32:0/120322178 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:47:43.908 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.905+0000 7fae31200640 1 --2- 172.21.15.32:0/3624013074 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fae0c077010 0x7fae0c079520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:47:43.908 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.906+0000 7fae31200640 1 --2- 172.21.15.32:0/3624013074 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fae0c077010 0x7fae0c079520 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fae2c150080 tx=0x7fae2400a540 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:47:43.909 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.905+0000 7f7742a00640 1 --2- 172.21.15.32:0/120322178 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f77440b4860 0x7f7744148750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:47:43.909 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.906+0000 7f7742a00640 1 -- 172.21.15.32:0/120322178 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f7744073b70 msgr2=0x7f7744148210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:47:43.909 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.906+0000 7f7742a00640 1 --2- 172.21.15.32:0/120322178 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f7744073b70 0x7f7744148210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:43.909 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.906+0000 7f7742a00640 1 -- 172.21.15.32:0/120322178 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f77440cf4a0 msgr2=0x7f774414b500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:47:43.909 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.906+0000 7f7743e00640 1 --2- 172.21.15.32:0/120322178 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f77440cf4a0 0x7f774414b500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:47:43.909 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.906+0000 7f7742a00640 1 --2- 172.21.15.32:0/120322178 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f77440cf4a0 0x7f774414b500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:43.909 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.906+0000 7f7742a00640 1 -- 172.21.15.32:0/120322178 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f774414c5b0 con 0x7f77440b4860 2024-08-03T20:47:43.909 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.906+0000 7f7743e00640 1 --2- 172.21.15.32:0/120322178 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f77440cf4a0 0x7f774414b500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-03T20:47:43.909 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.907+0000 7f7742a00640 1 --2- 172.21.15.32:0/120322178 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f77440b4860 0x7f7744148750 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f773000b440 tx=0x7f773000b900 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:47:43.910 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.907+0000 7f2698c00640 1 -- 172.21.15.32:0/244158632 <== mon.1 v2:172.21.15.118:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7f26940374d0 con 0x7f2694054b90 2024-08-03T20:47:43.910 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.907+0000 7f773b400640 1 -- 172.21.15.32:0/120322178 <== mon.1 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7730011830 con 0x7f77440b4860 2024-08-03T20:47:43.910 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.907+0000 7f773b400640 1 -- 172.21.15.32:0/120322178 <== mon.1 v2:172.21.15.118:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f7730016070 con 0x7f77440b4860 2024-08-03T20:47:43.910 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.907+0000 7f773b400640 1 -- 172.21.15.32:0/120322178 <== mon.1 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7730011d60 con 0x7f77440b4860 2024-08-03T20:47:43.910 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.908+0000 7f774a000640 1 -- 172.21.15.32:0/120322178 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f774414c890 con 0x7f77440b4860 2024-08-03T20:47:43.910 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.909+0000 7f774a000640 1 -- 172.21.15.32:0/120322178 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f774414ce50 con 0x7f77440b4860 2024-08-03T20:47:43.911 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.910+0000 7f774a000640 1 -- 172.21.15.32:0/120322178 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f77440b4c70 con 0x7f77440b4860 2024-08-03T20:47:43.912 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.911+0000 7f07f4600640 1 -- 172.21.15.32:0/397322685 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f07ec09af90 msgr2=0x7f07ec0b5c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:47:43.913 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.911+0000 7f07f4600640 1 --2- 172.21.15.32:0/397322685 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f07ec09af90 0x7f07ec0b5c00 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f07ec0cc840 tx=0x7f07d802a620 comp rx=0 tx=0).stop 2024-08-03T20:47:43.913 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.912+0000 7f07f4600640 1 -- 172.21.15.32:0/397322685 shutdown_connections 2024-08-03T20:47:43.913 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.912+0000 7f07f4600640 1 --2- 172.21.15.32:0/397322685 >> v2:172.21.15.118:3300/0 conn(0x7f07ec0b6140 0x7f07ec0b6550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:43.913 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.912+0000 7f07f4600640 1 --2- 172.21.15.32:0/397322685 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f07ec09af90 0x7f07ec0b5c00 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:43.914 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.912+0000 7f07f4600640 1 --2- 172.21.15.32:0/397322685 >> v2:172.21.15.32:3300/0 conn(0x7f07ec0cf540 0x7f07ec09aa50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:43.914 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.912+0000 7f07f4600640 1 -- 172.21.15.32:0/397322685 >> 172.21.15.32:0/397322685 conn(0x7f07ec0d2160 msgr2=0x7f07ec0d2570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:47:43.914 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.912+0000 7f07f4600640 1 -- 172.21.15.32:0/397322685 shutdown_connections 2024-08-03T20:47:43.915 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.914+0000 7fae29600640 1 -- 172.21.15.32:0/3624013074 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7fae2c1fd7e0 con 0x7fae2c06f070 2024-08-03T20:47:43.916 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.915+0000 7f773b400640 1 -- 172.21.15.32:0/120322178 <== mon.1 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f773003a640 con 0x7f77440b4860 2024-08-03T20:47:43.917 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.912+0000 7f07f4600640 1 -- 172.21.15.32:0/397322685 wait complete. 2024-08-03T20:47:43.918 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.916+0000 7f773b400640 1 --2- 172.21.15.32:0/120322178 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f7724077060 0x7f7724079570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:47:43.918 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.917+0000 7f773b400640 1 -- 172.21.15.32:0/120322178 <== mon.1 v2:172.21.15.118:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f77300988a0 con 0x7f77440b4860 2024-08-03T20:47:43.918 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.917+0000 7f497a800640 1 -- 172.21.15.32:0/393919290 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f49740b4de0 msgr2=0x7f49741515c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:47:43.918 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.917+0000 7f497a800640 1 --2- 172.21.15.32:0/393919290 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f49740b4de0 0x7f49741515c0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f4964005fd0 tx=0x7f496402aa10 comp rx=0 tx=0).stop 2024-08-03T20:47:43.919 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.917+0000 7f7743400640 1 --2- 172.21.15.32:0/120322178 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f7724077060 0x7f7724079570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:47:43.919 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.917+0000 7f497a800640 1 -- 172.21.15.32:0/393919290 shutdown_connections 2024-08-03T20:47:43.919 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.917+0000 7f497a800640 1 --2- 172.21.15.32:0/393919290 >> v2:172.21.15.118:3300/0 conn(0x7f497415e880 0x7f4974073800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:43.919 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.917+0000 7f497a800640 1 --2- 172.21.15.32:0/393919290 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f49740b4de0 0x7f49741515c0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:43.919 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.917+0000 7f497a800640 1 -- 172.21.15.32:0/393919290 >> 172.21.15.32:0/393919290 conn(0x7f49740b1bd0 msgr2=0x7f49740b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:47:43.920 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.917+0000 7f773b400640 1 -- 172.21.15.32:0/120322178 <== mon.1 v2:172.21.15.118:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7f7730061b00 con 0x7f77440b4860 2024-08-03T20:47:43.920 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.918+0000 7f7743400640 1 --2- 172.21.15.32:0/120322178 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f7724077060 0x7f7724079570 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f7734005fd0 tx=0x7f7734005d30 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:47:43.920 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.919+0000 7f497a800640 1 -- 172.21.15.32:0/393919290 shutdown_connections 2024-08-03T20:47:43.920 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.919+0000 7f497a800640 1 -- 172.21.15.32:0/393919290 wait complete. 2024-08-03T20:47:43.921 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.916+0000 7f07f4600640 1 Processor -- start 2024-08-03T20:47:43.921 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.917+0000 7f07f4600640 1 -- start start 2024-08-03T20:47:43.922 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.920+0000 7fe9cbe00640 1 -- 172.21.15.32:0/2287464732 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fe9cc0cec50 msgr2=0x7fe9cc162640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:47:43.922 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.920+0000 7fe9cbe00640 1 --2- 172.21.15.32:0/2287464732 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fe9cc0cec50 0x7fe9cc162640 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fe9b4009830 tx=0x7fe9b402a5b0 comp rx=0 tx=0).stop 2024-08-03T20:47:43.922 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.921+0000 7fe9cbe00640 1 -- 172.21.15.32:0/2287464732 shutdown_connections 2024-08-03T20:47:43.922 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.921+0000 7fe9cbe00640 1 --2- 172.21.15.32:0/2287464732 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fe9cc0cec50 0x7fe9cc162640 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:43.922 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.921+0000 7fe9cbe00640 1 -- 172.21.15.32:0/2287464732 >> 172.21.15.32:0/2287464732 conn(0x7fe9cc0d4180 msgr2=0x7fe9cc0d4590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:47:43.922 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.921+0000 7fe9cbe00640 1 -- 172.21.15.32:0/2287464732 shutdown_connections 2024-08-03T20:47:43.923 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.921+0000 7fe9cbe00640 1 -- 172.21.15.32:0/2287464732 wait complete. 2024-08-03T20:47:43.924 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.923+0000 7f497a800640 1 Processor -- start 2024-08-03T20:47:43.925 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.923+0000 7f07f4600640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f07ec09af90 0x7f07ec0b0000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:47:43.925 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.923+0000 7f07f4600640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f07ec0b6140 0x7f07ec159e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:47:43.925 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.923+0000 7f07f4600640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f07ec0cf540 0x7f07ec15a3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:47:43.925 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.923+0000 7f07f4600640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f07ec0b6c10 con 0x7f07ec0cf540 2024-08-03T20:47:43.925 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.923+0000 7f07f4600640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f07ec0b6a90 con 0x7f07ec09af90 2024-08-03T20:47:43.925 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.923+0000 7f07f4600640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7f07ec0b6d90 con 0x7f07ec0b6140 2024-08-03T20:47:43.925 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.924+0000 7f497a800640 1 -- start start 2024-08-03T20:47:43.926 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.924+0000 7f07f1200640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f07ec0b6140 0x7f07ec159e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:47:43.926 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.924+0000 7f07f1200640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f07ec0b6140 0x7f07ec159e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3301/0 says I am v2:172.21.15.32:36564/0 (socket says 172.21.15.32:36564) 2024-08-03T20:47:43.926 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.924+0000 7f07f1200640 1 -- 172.21.15.32:0/3514988541 learned_addr learned my addr 172.21.15.32:0/3514988541 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:47:43.926 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.924+0000 7f07f1c00640 1 --2- 172.21.15.32:0/3514988541 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f07ec09af90 0x7f07ec0b0000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:47:43.927 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.924+0000 7f6445000640 1 -- 172.21.15.32:0/65753540 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f64400b79d0 msgr2=0x7f64400d02e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:47:43.927 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.924+0000 7f6445000640 1 --2- 172.21.15.32:0/65753540 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f64400b79d0 0x7f64400d02e0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f64400b4cd0 tx=0x7f642c02a4a0 comp rx=0 tx=0).stop 2024-08-03T20:47:43.927 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.924+0000 7f07f1200640 1 -- 172.21.15.32:0/3514988541 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f07ec09af90 msgr2=0x7f07ec0b0000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:47:43.927 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.924+0000 7f07f1200640 1 --2- 172.21.15.32:0/3514988541 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f07ec09af90 0x7f07ec0b0000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:43.927 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.924+0000 7f07f1200640 1 -- 172.21.15.32:0/3514988541 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f07ec0cf540 msgr2=0x7f07ec15a3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:47:43.927 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.924+0000 7f07f1200640 1 --2- 172.21.15.32:0/3514988541 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f07ec0cf540 0x7f07ec15a3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:43.927 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.924+0000 7f07f1200640 1 -- 172.21.15.32:0/3514988541 --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f07ec1599b0 con 0x7f07ec0b6140 2024-08-03T20:47:43.927 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.924+0000 7f07f1c00640 1 --2- 172.21.15.32:0/3514988541 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f07ec09af90 0x7f07ec0b0000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-03T20:47:43.928 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.925+0000 7f6445000640 1 -- 172.21.15.32:0/65753540 shutdown_connections 2024-08-03T20:47:43.928 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.925+0000 7f6445000640 1 --2- 172.21.15.32:0/65753540 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f644009aa50 0x7f644009ae60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:43.928 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.925+0000 7f6445000640 1 --2- 172.21.15.32:0/65753540 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f64400b79d0 0x7f64400d02e0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:43.928 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.925+0000 7f6445000640 1 -- 172.21.15.32:0/65753540 >> 172.21.15.32:0/65753540 conn(0x7f64400b3c20 msgr2=0x7f64400b4030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:47:43.928 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.926+0000 7f6445000640 1 -- 172.21.15.32:0/65753540 shutdown_connections 2024-08-03T20:47:43.929 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.926+0000 7fe9cbe00640 1 Processor -- start 2024-08-03T20:47:43.929 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.926+0000 7f6445000640 1 -- 172.21.15.32:0/65753540 wait complete. 2024-08-03T20:47:43.929 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.926+0000 7fe9cbe00640 1 -- start start 2024-08-03T20:47:43.930 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.925+0000 7f07f1200640 1 --2- 172.21.15.32:0/3514988541 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f07ec0b6140 0x7f07ec159e60 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f07d8034040 tx=0x7f07d8003890 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:47:43.930 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.927+0000 7f07e9600640 1 -- 172.21.15.32:0/3514988541 <== mon.2 v2:172.21.15.118:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07d803e030 con 0x7f07ec0b6140 2024-08-03T20:47:43.930 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.927+0000 7f07e9600640 1 -- 172.21.15.32:0/3514988541 <== mon.2 v2:172.21.15.118:3301/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f07d8003d80 con 0x7f07ec0b6140 2024-08-03T20:47:43.930 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.927+0000 7f497a800640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f49740733f0 0x7f497413f960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:47:43.931 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.927+0000 7f07e9600640 1 -- 172.21.15.32:0/3514988541 <== mon.2 v2:172.21.15.118:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07d8035040 con 0x7f07ec0b6140 2024-08-03T20:47:43.931 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.927+0000 7f497a800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f49740b4de0 0x7f4974152730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:47:43.931 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.927+0000 7f4973400640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f49740733f0 0x7f497413f960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:47:43.931 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.927+0000 7f4973400640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f49740733f0 0x7f497413f960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3301/0 says I am v2:172.21.15.32:36580/0 (socket says 172.21.15.32:36580) 2024-08-03T20:47:43.931 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.928+0000 7f07f4600640 1 -- 172.21.15.32:0/3514988541 --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f07ec15a8e0 con 0x7f07ec0b6140 2024-08-03T20:47:43.932 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.928+0000 7f4973400640 1 -- 172.21.15.32:0/3185224006 learned_addr learned my addr 172.21.15.32:0/3185224006 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:47:43.932 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.928+0000 7f07f4600640 1 -- 172.21.15.32:0/3514988541 --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f07ec1fa080 con 0x7f07ec0b6140 2024-08-03T20:47:43.933 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.928+0000 7f497a800640 1 --2- 172.21.15.32:0/3185224006 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f4974144f80 0x7f4974152c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:47:43.933 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.928+0000 7f4973e00640 1 --2- 172.21.15.32:0/3185224006 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f4974144f80 0x7f4974152c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:47:43.933 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.928+0000 7f496aa00640 1 --2- 172.21.15.32:0/3185224006 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f49740b4de0 0x7f4974152730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:47:43.933 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.929+0000 7f497a800640 1 -- 172.21.15.32:0/3185224006 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f497406e930 con 0x7f49740b4de0 2024-08-03T20:47:43.933 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.929+0000 7f497a800640 1 -- 172.21.15.32:0/3185224006 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f497406eab0 con 0x7f4974144f80 2024-08-03T20:47:43.933 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.929+0000 7f497a800640 1 -- 172.21.15.32:0/3185224006 --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7f497406ec30 con 0x7f49740733f0 2024-08-03T20:47:43.934 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.929+0000 7fe9cbe00640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fe9cc0cec50 0x7fe9cc0c7130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:47:43.934 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.929+0000 7fe9cbe00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fe9cc0d1d50 0x7fe9cc0a04a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:47:43.934 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.929+0000 7fe9cbe00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe9cc162b80 0x7fe9cc0a09e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:47:43.934 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.930+0000 7fe9caa00640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fe9cc0cec50 0x7fe9cc0c7130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:47:43.934 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.929+0000 7f07f4600640 1 -- 172.21.15.32:0/3514988541 --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f07ec0b6550 con 0x7f07ec0b6140 2024-08-03T20:47:43.934 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.930+0000 7fe9caa00640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fe9cc0cec50 0x7fe9cc0c7130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3301/0 says I am v2:172.21.15.32:36586/0 (socket says 172.21.15.32:36586) 2024-08-03T20:47:43.934 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.930+0000 7fe9caa00640 1 -- 172.21.15.32:0/4036997747 learned_addr learned my addr 172.21.15.32:0/4036997747 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:47:43.935 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.930+0000 7f4973400640 1 -- 172.21.15.32:0/3185224006 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f4974144f80 msgr2=0x7f4974152c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:47:43.935 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.930+0000 7f4973400640 1 --2- 172.21.15.32:0/3185224006 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f4974144f80 0x7f4974152c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:43.935 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.930+0000 7f4973400640 1 -- 172.21.15.32:0/3185224006 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f49740b4de0 msgr2=0x7f4974152730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:47:43.935 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.930+0000 7fe9cbe00640 1 -- 172.21.15.32:0/4036997747 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fe9cc0b76e0 con 0x7fe9cc162b80 2024-08-03T20:47:43.935 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.930+0000 7fe9cbe00640 1 -- 172.21.15.32:0/4036997747 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7fe9cc163390 con 0x7fe9cc0d1d50 2024-08-03T20:47:43.936 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.930+0000 7fe9cbe00640 1 -- 172.21.15.32:0/4036997747 --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7fe9cc0b7860 con 0x7fe9cc0cec50 2024-08-03T20:47:43.936 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.930+0000 7f6445000640 1 Processor -- start 2024-08-03T20:47:43.936 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.930+0000 7fe9cb400640 1 --2- 172.21.15.32:0/4036997747 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe9cc162b80 0x7fe9cc0a09e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:47:43.936 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.931+0000 7fe9c2000640 1 --2- 172.21.15.32:0/4036997747 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fe9cc0d1d50 0x7fe9cc0a04a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:47:43.937 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.931+0000 7f4973400640 1 --2- 172.21.15.32:0/3185224006 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f49740b4de0 0x7f4974152730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:43.937 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.931+0000 7f4973400640 1 -- 172.21.15.32:0/3185224006 --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f49741523b0 con 0x7f49740733f0 2024-08-03T20:47:43.937 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.931+0000 7f496aa00640 1 --2- 172.21.15.32:0/3185224006 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f49740b4de0 0x7f4974152730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-03T20:47:43.937 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.932+0000 7f4973400640 1 --2- 172.21.15.32:0/3185224006 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f49740733f0 0x7f497413f960 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f4964009d90 tx=0x7f49640045e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:47:43.937 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.932+0000 7f6445000640 1 -- start start 2024-08-03T20:47:43.938 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.932+0000 7fe9caa00640 1 -- 172.21.15.32:0/4036997747 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fe9cc0d1d50 msgr2=0x7fe9cc0a04a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:47:43.938 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.932+0000 7fe9caa00640 1 --2- 172.21.15.32:0/4036997747 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fe9cc0d1d50 0x7fe9cc0a04a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:43.938 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.932+0000 7f496be00640 1 -- 172.21.15.32:0/3185224006 <== mon.2 v2:172.21.15.118:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4964033070 con 0x7f49740733f0 2024-08-03T20:47:43.939 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.932+0000 7f07e9600640 1 -- 172.21.15.32:0/3514988541 <== mon.2 v2:172.21.15.118:3301/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f07d8041030 con 0x7f07ec0b6140 2024-08-03T20:47:43.939 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.933+0000 7fe9caa00640 1 -- 172.21.15.32:0/4036997747 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe9cc162b80 msgr2=0x7fe9cc0a09e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:47:43.939 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.933+0000 7fe9caa00640 1 --2- 172.21.15.32:0/4036997747 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe9cc162b80 0x7fe9cc0a09e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:43.939 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.933+0000 7fe9caa00640 1 -- 172.21.15.32:0/4036997747 --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe9cc073750 con 0x7fe9cc0cec50 2024-08-03T20:47:43.939 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.933+0000 7fe9c2000640 1 --2- 172.21.15.32:0/4036997747 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fe9cc0d1d50 0x7fe9cc0a04a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-03T20:47:43.940 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.933+0000 7f496be00640 1 -- 172.21.15.32:0/3185224006 <== mon.2 v2:172.21.15.118:3301/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f4964038450 con 0x7f49740733f0 2024-08-03T20:47:43.940 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.933+0000 7f497a800640 1 -- 172.21.15.32:0/3185224006 --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f49741444a0 con 0x7f49740733f0 2024-08-03T20:47:43.940 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.933+0000 7fe9caa00640 1 --2- 172.21.15.32:0/4036997747 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fe9cc0cec50 0x7fe9cc0c7130 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fe9bc00c730 tx=0x7fe9bc00cbf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:47:43.940 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.933+0000 7f497a800640 1 -- 172.21.15.32:0/3185224006 --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f4974144a60 con 0x7f49740733f0 2024-08-03T20:47:43.940 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.933+0000 7f496be00640 1 -- 172.21.15.32:0/3185224006 <== mon.2 v2:172.21.15.118:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4964033070 con 0x7f49740733f0 2024-08-03T20:47:43.941 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.934+0000 7f07e9600640 1 --2- 172.21.15.32:0/3514988541 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f07cc077060 0x7f07cc079570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:47:43.941 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.934+0000 7f07e9600640 1 -- 172.21.15.32:0/3514988541 <== mon.2 v2:172.21.15.118:3301/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f07d80b90f0 con 0x7f07ec0b6140 2024-08-03T20:47:43.941 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.935+0000 7f6445000640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f644006e500 0x7f644015c550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:47:43.941 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.935+0000 7f07f1c00640 1 --2- 172.21.15.32:0/3514988541 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f07cc077060 0x7f07cc079570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:47:43.942 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.935+0000 7f497a800640 1 -- 172.21.15.32:0/3185224006 --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f494c005290 con 0x7f49740733f0 2024-08-03T20:47:43.943 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.935+0000 7f6445000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f644009aa50 0x7f6440161ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:47:43.943 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.936+0000 7f6445000640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f64400b79d0 0x7f6440162220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:47:43.943 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.936+0000 7f6445000640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f64400b2db0 con 0x7f644009aa50 2024-08-03T20:47:43.943 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.936+0000 7f643e800640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f64400b79d0 0x7f6440162220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:47:43.944 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.936+0000 7f643e800640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f64400b79d0 0x7f6440162220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3301/0 says I am v2:172.21.15.32:36602/0 (socket says 172.21.15.32:36602) 2024-08-03T20:47:43.944 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.936+0000 7f643e800640 1 -- 172.21.15.32:0/2574166090 learned_addr learned my addr 172.21.15.32:0/2574166090 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:47:43.944 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.936+0000 7f643de00640 1 --2- 172.21.15.32:0/2574166090 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f644006e500 0x7f644015c550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:47:43.944 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.936+0000 7f6445000640 1 -- 172.21.15.32:0/2574166090 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f64400b2c30 con 0x7f644006e500 2024-08-03T20:47:43.944 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.936+0000 7f643d400640 1 --2- 172.21.15.32:0/2574166090 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f644009aa50 0x7f6440161ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:47:43.944 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.936+0000 7f6445000640 1 -- 172.21.15.32:0/2574166090 --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7f64400d14e0 con 0x7f64400b79d0 2024-08-03T20:47:43.944 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.937+0000 7f643de00640 1 -- 172.21.15.32:0/2574166090 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f64400b79d0 msgr2=0x7f6440162220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:47:43.944 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.937+0000 7f643de00640 1 --2- 172.21.15.32:0/2574166090 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f64400b79d0 0x7f6440162220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:43.944 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.937+0000 7f643de00640 1 -- 172.21.15.32:0/2574166090 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f644009aa50 msgr2=0x7f6440161ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:47:43.944 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.937+0000 7f643de00640 1 --2- 172.21.15.32:0/2574166090 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f644009aa50 0x7f6440161ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:43.944 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.937+0000 7f643de00640 1 -- 172.21.15.32:0/2574166090 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6440158a60 con 0x7f644006e500 2024-08-03T20:47:43.944 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.938+0000 7f643d400640 1 --2- 172.21.15.32:0/2574166090 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f644009aa50 0x7f6440161ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-03T20:47:43.944 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.938+0000 7f643e800640 1 --2- 172.21.15.32:0/2574166090 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f64400b79d0 0x7f6440162220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-03T20:47:43.945 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.938+0000 7f643de00640 1 --2- 172.21.15.32:0/2574166090 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f644006e500 0x7f644015c550 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f6440073ea0 tx=0x7f642c005270 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:47:43.945 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.938+0000 7f6436000640 1 -- 172.21.15.32:0/2574166090 <== mon.1 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f642c03f070 con 0x7f644006e500 2024-08-03T20:47:43.945 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.938+0000 7f6436000640 1 -- 172.21.15.32:0/2574166090 <== mon.1 v2:172.21.15.118:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f642c004360 con 0x7f644006e500 2024-08-03T20:47:43.945 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.938+0000 7f6445000640 1 -- 172.21.15.32:0/2574166090 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f644015cde0 con 0x7f644006e500 2024-08-03T20:47:43.945 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.938+0000 7f6436000640 1 -- 172.21.15.32:0/2574166090 <== mon.1 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f642c031bb0 con 0x7f644006e500 2024-08-03T20:47:43.945 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.938+0000 7f07f1c00640 1 --2- 172.21.15.32:0/3514988541 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f07cc077060 0x7f07cc079570 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f07e0005fd0 tx=0x7f07e0005d40 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:47:43.946 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.939+0000 7fe9c3e00640 1 -- 172.21.15.32:0/4036997747 <== mon.2 v2:172.21.15.118:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9bc016020 con 0x7fe9cc0cec50 2024-08-03T20:47:43.946 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.939+0000 7fe9cbe00640 1 -- 172.21.15.32:0/4036997747 --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe9cc073a10 con 0x7fe9cc0cec50 2024-08-03T20:47:43.946 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.939+0000 7fe9cbe00640 1 -- 172.21.15.32:0/4036997747 --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fe9cc071500 con 0x7fe9cc0cec50 2024-08-03T20:47:43.946 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.941+0000 7fe9c3e00640 1 -- 172.21.15.32:0/4036997747 <== mon.2 v2:172.21.15.118:3301/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fe9bc006b50 con 0x7fe9cc0cec50 2024-08-03T20:47:43.946 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.941+0000 7fe9c3e00640 1 -- 172.21.15.32:0/4036997747 <== mon.2 v2:172.21.15.118:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9bc00aa50 con 0x7fe9cc0cec50 2024-08-03T20:47:43.947 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.943+0000 7f496be00640 1 -- 172.21.15.32:0/3185224006 <== mon.2 v2:172.21.15.118:3301/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f496403f070 con 0x7f49740733f0 2024-08-03T20:47:43.948 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.939+0000 7f6445000640 1 -- 172.21.15.32:0/2574166090 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f644015d3a0 con 0x7f644006e500 2024-08-03T20:47:43.948 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.941+0000 7f642b400640 1 -- 172.21.15.32:0/2574166090 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6414005290 con 0x7f644006e500 2024-08-03T20:47:43.948 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.944+0000 7fe9cbe00640 1 -- 172.21.15.32:0/4036997747 --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe9a4005290 con 0x7fe9cc0cec50 2024-08-03T20:47:43.948 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.942+0000 7f07e9600640 1 -- 172.21.15.32:0/3514988541 <== mon.2 v2:172.21.15.118:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7f07ec09b3a0 con 0x7f07ec0b6140 2024-08-03T20:47:43.949 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.947+0000 7f6436000640 1 -- 172.21.15.32:0/2574166090 <== mon.1 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f642c042070 con 0x7f644006e500 2024-08-03T20:47:43.949 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.947+0000 7f496be00640 1 --2- 172.21.15.32:0/3185224006 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f4950077240 0x7f4950079750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:47:43.950 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.949+0000 7fe9c3e00640 1 -- 172.21.15.32:0/4036997747 <== mon.2 v2:172.21.15.118:3301/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7fe9bc019020 con 0x7fe9cc0cec50 2024-08-03T20:47:43.950 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.948+0000 7f496be00640 1 -- 172.21.15.32:0/3185224006 <== mon.2 v2:172.21.15.118:3301/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f49640b9900 con 0x7f49740733f0 2024-08-03T20:47:43.951 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.948+0000 7f496aa00640 1 --2- 172.21.15.32:0/3185224006 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f4950077240 0x7f4950079750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:47:43.951 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.948+0000 7f496be00640 1 -- 172.21.15.32:0/3185224006 <== mon.2 v2:172.21.15.118:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7f496407e800 con 0x7f49740733f0 2024-08-03T20:47:43.952 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.950+0000 7f496aa00640 1 --2- 172.21.15.32:0/3185224006 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f4950077240 0x7f4950079750 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f4974145b80 tx=0x7f495c009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:47:43.952 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.951+0000 7fe9c3e00640 1 --2- 172.21.15.32:0/4036997747 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fe9a8077010 0x7fe9a8079520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:47:43.952 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.951+0000 7fe9c3e00640 1 -- 172.21.15.32:0/4036997747 <== mon.2 v2:172.21.15.118:3301/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7fe9bc098570 con 0x7fe9cc0cec50 2024-08-03T20:47:43.953 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.951+0000 7f6436000640 1 --2- 172.21.15.32:0/2574166090 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f6420077040 0x7f6420079550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:47:43.953 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.951+0000 7f6436000640 1 -- 172.21.15.32:0/2574166090 <== mon.1 v2:172.21.15.118:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f642c0badf0 con 0x7f644006e500 2024-08-03T20:47:43.953 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.952+0000 7fe9c2000640 1 --2- 172.21.15.32:0/4036997747 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fe9a8077010 0x7fe9a8079520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:47:43.954 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.952+0000 7f643d400640 1 --2- 172.21.15.32:0/2574166090 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f6420077040 0x7f6420079550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:47:43.954 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.953+0000 7fe9c2000640 1 --2- 172.21.15.32:0/4036997747 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fe9a8077010 0x7fe9a8079520 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fe9b4034000 tx=0x7fe9b4004730 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:47:43.956 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.954+0000 7f643d400640 1 --2- 172.21.15.32:0/2574166090 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f6420077040 0x7f6420079550 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f6430006fd0 tx=0x7f6430008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:47:43.956 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.954+0000 7f6436000640 1 -- 172.21.15.32:0/2574166090 <== mon.1 v2:172.21.15.118:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7f642c031770 con 0x7f644006e500 2024-08-03T20:47:43.959 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:43.956+0000 7fe9c3e00640 1 -- 172.21.15.32:0/4036997747 <== mon.2 v2:172.21.15.118:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7fe9bc020070 con 0x7fe9cc0cec50 2024-08-03T20:47:44.362 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:44.361+0000 7f26a3c00640 1 -- 172.21.15.32:0/244158632 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) -- 0x7f2678005580 con 0x7f2694054b90 2024-08-03T20:47:44.363 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:44.362+0000 7f2698c00640 1 -- 172.21.15.32:0/244158632 <== mon.1 v2:172.21.15.118:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f26880622d0 con 0x7f2694054b90 2024-08-03T20:47:44.364 INFO:teuthology.orchestra.run.smithi032.stdout:64424509446 2024-08-03T20:47:44.366 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:44.365+0000 7f26a3c00640 1 -- 172.21.15.32:0/244158632 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f267c077010 msgr2=0x7f267c079520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:47:44.367 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:44.365+0000 7f26a3c00640 1 --2- 172.21.15.32:0/244158632 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f267c077010 0x7f267c079520 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f269400b7c0 tx=0x7f269c0a8040 comp rx=0 tx=0).stop 2024-08-03T20:47:44.367 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:44.366+0000 7f26a3c00640 1 -- 172.21.15.32:0/244158632 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f2694054b90 msgr2=0x7f2694036f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:47:44.367 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:44.366+0000 7f26a3c00640 1 --2- 172.21.15.32:0/244158632 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f2694054b90 0x7f2694036f90 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f2688009fd0 tx=0x7f268800ec20 comp rx=0 tx=0).stop 2024-08-03T20:47:44.368 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:44.366+0000 7f26a3c00640 1 -- 172.21.15.32:0/244158632 shutdown_connections 2024-08-03T20:47:44.368 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:44.366+0000 7f26a3c00640 1 --2- 172.21.15.32:0/244158632 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f267c077010 0x7f267c079520 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:44.368 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:44.366+0000 7f26a3c00640 1 --2- 172.21.15.32:0/244158632 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f2694054b90 0x7f2694036f90 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:44.368 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:44.366+0000 7f26a3c00640 1 --2- 172.21.15.32:0/244158632 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f26940e1000 0x7f26940359c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:44.368 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:44.366+0000 7f26a3c00640 1 --2- 172.21.15.32:0/244158632 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f26940537e0 0x7f2694008290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:44.368 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:44.366+0000 7f26a3c00640 1 -- 172.21.15.32:0/244158632 >> 172.21.15.32:0/244158632 conn(0x7f269401fbd0 msgr2=0x7f2694033ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:47:44.369 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:44.367+0000 7f26a3c00640 1 -- 172.21.15.32:0/244158632 shutdown_connections 2024-08-03T20:47:44.369 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:44.367+0000 7f26a3c00640 1 -- 172.21.15.32:0/244158632 wait complete. 2024-08-03T20:47:44.391 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:44.389+0000 7f52db200640 1 -- 172.21.15.32:0/591383566 --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) -- 0x7f52d4036340 con 0x7f52d4008170 2024-08-03T20:47:44.392 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:44.390+0000 7f52d2000640 1 -- 172.21.15.32:0/591383566 <== mon.2 v2:172.21.15.118:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f52c80612b0 con 0x7f52d4008170 2024-08-03T20:47:44.392 INFO:teuthology.orchestra.run.smithi032.stdout:60129542147 2024-08-03T20:47:44.396 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:44.395+0000 7f52db200640 1 -- 172.21.15.32:0/591383566 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f52bc077110 msgr2=0x7f52bc079620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:47:44.396 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:44.395+0000 7f52db200640 1 --2- 172.21.15.32:0/591383566 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f52bc077110 0x7f52bc079620 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f52dc066470 tx=0x7f52dc070040 comp rx=0 tx=0).stop 2024-08-03T20:47:44.396 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:44.395+0000 7f52db200640 1 -- 172.21.15.32:0/591383566 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f52d4008170 msgr2=0x7f52d4020650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:47:44.397 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:44.395+0000 7f52db200640 1 --2- 172.21.15.32:0/591383566 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f52d4008170 0x7f52d4020650 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f52c800d6a0 tx=0x7f52c800db60 comp rx=0 tx=0).stop 2024-08-03T20:47:44.397 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:44.395+0000 7f52db200640 1 -- 172.21.15.32:0/591383566 shutdown_connections 2024-08-03T20:47:44.397 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:44.395+0000 7f52db200640 1 --2- 172.21.15.32:0/591383566 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f52bc077110 0x7f52bc079620 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:44.397 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:44.395+0000 7f52db200640 1 --2- 172.21.15.32:0/591383566 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f52d4053b50 0x7f52d4020b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:44.397 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:44.395+0000 7f52db200640 1 --2- 172.21.15.32:0/591383566 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f52d4008170 0x7f52d4020650 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:44.397 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:44.395+0000 7f52db200640 1 --2- 172.21.15.32:0/591383566 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f52d4007d60 0x7f52d40e2ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:44.398 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:44.395+0000 7f52db200640 1 -- 172.21.15.32:0/591383566 >> 172.21.15.32:0/591383566 conn(0x7f52d401fbe0 msgr2=0x7f52d4055120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:47:44.398 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:44.396+0000 7f52db200640 1 -- 172.21.15.32:0/591383566 shutdown_connections 2024-08-03T20:47:44.398 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:44.396+0000 7f52db200640 1 -- 172.21.15.32:0/591383566 wait complete. 2024-08-03T20:47:44.399 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:44.397+0000 7fae34600640 1 -- 172.21.15.32:0/3624013074 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) -- 0x7fae2c0cedb0 con 0x7fae2c06f070 2024-08-03T20:47:44.399 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:44.398+0000 7fae29600640 1 -- 172.21.15.32:0/3624013074 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fae20069070 con 0x7fae2c06f070 2024-08-03T20:47:44.400 INFO:teuthology.orchestra.run.smithi032.stdout:60129542148 2024-08-03T20:47:44.405 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:44.403+0000 7fae34600640 1 -- 172.21.15.32:0/3624013074 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fae0c077010 msgr2=0x7fae0c079520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:47:44.405 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:44.403+0000 7fae34600640 1 --2- 172.21.15.32:0/3624013074 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fae0c077010 0x7fae0c079520 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fae2c150080 tx=0x7fae2400a540 comp rx=0 tx=0).stop 2024-08-03T20:47:44.405 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:44.403+0000 7fae34600640 1 -- 172.21.15.32:0/3624013074 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fae2c06f070 msgr2=0x7fae2c148eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:47:44.406 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:44.403+0000 7fae34600640 1 --2- 172.21.15.32:0/3624013074 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fae2c06f070 0x7fae2c148eb0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fae2000c780 tx=0x7fae2000cc40 comp rx=0 tx=0).stop 2024-08-03T20:47:44.406 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:44.404+0000 7fae34600640 1 -- 172.21.15.32:0/3624013074 shutdown_connections 2024-08-03T20:47:44.406 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:44.404+0000 7fae34600640 1 --2- 172.21.15.32:0/3624013074 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fae0c077010 0x7fae0c079520 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:44.406 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:44.404+0000 7fae34600640 1 --2- 172.21.15.32:0/3624013074 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fae2c1554a0 0x7fae2c1532f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:44.406 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:44.404+0000 7fae34600640 1 --2- 172.21.15.32:0/3624013074 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fae2c1493f0 0x7fae2c153830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:44.406 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:44.404+0000 7fae34600640 1 --2- 172.21.15.32:0/3624013074 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fae2c06f070 0x7fae2c148eb0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:44.407 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:44.404+0000 7fae34600640 1 -- 172.21.15.32:0/3624013074 >> 172.21.15.32:0/3624013074 conn(0x7fae2c0b1890 msgr2=0x7fae2c0ae040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:47:44.407 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:44.405+0000 7fae34600640 1 -- 172.21.15.32:0/3624013074 shutdown_connections 2024-08-03T20:47:44.407 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:44.405+0000 7fae34600640 1 -- 172.21.15.32:0/3624013074 wait complete. 2024-08-03T20:47:44.407 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:44.406+0000 7f774a000640 1 -- 172.21.15.32:0/120322178 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) -- 0x7f7744149060 con 0x7f77440b4860 2024-08-03T20:47:44.408 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509447 got 64424509446 for osd.0 2024-08-03T20:47:44.408 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:44.407+0000 7f773b400640 1 -- 172.21.15.32:0/120322178 <== mon.1 v2:172.21.15.118:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f7730061400 con 0x7f77440b4860 2024-08-03T20:47:44.409 INFO:teuthology.orchestra.run.smithi032.stdout:64424509444 2024-08-03T20:47:44.412 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:44.411+0000 7f774a000640 1 -- 172.21.15.32:0/120322178 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f7724077060 msgr2=0x7f7724079570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:47:44.413 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:44.411+0000 7f774a000640 1 --2- 172.21.15.32:0/120322178 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f7724077060 0x7f7724079570 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f7734005fd0 tx=0x7f7734005d30 comp rx=0 tx=0).stop 2024-08-03T20:47:44.413 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:44.411+0000 7f774a000640 1 -- 172.21.15.32:0/120322178 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f77440b4860 msgr2=0x7f7744148750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:47:44.413 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:44.411+0000 7f774a000640 1 --2- 172.21.15.32:0/120322178 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f77440b4860 0x7f7744148750 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f773000b440 tx=0x7f773000b900 comp rx=0 tx=0).stop 2024-08-03T20:47:44.413 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:44.411+0000 7f774a000640 1 -- 172.21.15.32:0/120322178 shutdown_connections 2024-08-03T20:47:44.413 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:44.411+0000 7f774a000640 1 --2- 172.21.15.32:0/120322178 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f7724077060 0x7f7724079570 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:44.413 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:44.411+0000 7f774a000640 1 --2- 172.21.15.32:0/120322178 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f77440b4860 0x7f7744148750 secure :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f773000b440 tx=0x7f773000b900 comp rx=0 tx=0).stop 2024-08-03T20:47:44.413 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:44.411+0000 7f774a000640 1 --2- 172.21.15.32:0/120322178 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f77440cf4a0 0x7f774414b500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:44.413 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:44.411+0000 7f774a000640 1 --2- 172.21.15.32:0/120322178 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f7744073b70 0x7f7744148210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:44.413 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:44.411+0000 7f774a000640 1 -- 172.21.15.32:0/120322178 >> 172.21.15.32:0/120322178 conn(0x7f77440b1c40 msgr2=0x7f7744072750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:47:44.413 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:44.411+0000 7f774a000640 1 -- 172.21.15.32:0/120322178 shutdown_connections 2024-08-03T20:47:44.413 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:44.411+0000 7f774a000640 1 -- 172.21.15.32:0/120322178 wait complete. 2024-08-03T20:47:44.420 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:44.419+0000 7f07f4600640 1 -- 172.21.15.32:0/3514988541 --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) -- 0x7f07ec0ce8f0 con 0x7f07ec0b6140 2024-08-03T20:47:44.421 INFO:teuthology.orchestra.run.smithi032.stdout:64424509444 2024-08-03T20:47:44.422 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:44.420+0000 7f07e9600640 1 -- 172.21.15.32:0/3514988541 <== mon.2 v2:172.21.15.118:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f07d80822d0 con 0x7f07ec0b6140 2024-08-03T20:47:44.422 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:44.420+0000 7fe9cbe00640 1 -- 172.21.15.32:0/4036997747 --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) -- 0x7fe9a4005580 con 0x7fe9cc0cec50 2024-08-03T20:47:44.422 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:44.421+0000 7fe9c3e00640 1 -- 172.21.15.32:0/4036997747 <== mon.2 v2:172.21.15.118:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fe9bc061760 con 0x7fe9cc0cec50 2024-08-03T20:47:44.423 INFO:teuthology.orchestra.run.smithi032.stdout:64424509445 2024-08-03T20:47:44.425 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:44.423+0000 7f07f4600640 1 -- 172.21.15.32:0/3514988541 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f07cc077060 msgr2=0x7f07cc079570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:47:44.425 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:44.424+0000 7fe9cbe00640 1 -- 172.21.15.32:0/4036997747 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fe9a8077010 msgr2=0x7fe9a8079520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:47:44.425 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:44.424+0000 7fe9cbe00640 1 --2- 172.21.15.32:0/4036997747 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fe9a8077010 0x7fe9a8079520 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fe9b4034000 tx=0x7fe9b4004730 comp rx=0 tx=0).stop 2024-08-03T20:47:44.425 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:44.424+0000 7fe9cbe00640 1 -- 172.21.15.32:0/4036997747 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fe9cc0cec50 msgr2=0x7fe9cc0c7130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:47:44.425 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:44.424+0000 7fe9cbe00640 1 --2- 172.21.15.32:0/4036997747 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fe9cc0cec50 0x7fe9cc0c7130 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fe9bc00c730 tx=0x7fe9bc00cbf0 comp rx=0 tx=0).stop 2024-08-03T20:47:44.426 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:44.423+0000 7f07f4600640 1 --2- 172.21.15.32:0/3514988541 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f07cc077060 0x7f07cc079570 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f07e0005fd0 tx=0x7f07e0005d40 comp rx=0 tx=0).stop 2024-08-03T20:47:44.426 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:44.423+0000 7f07f4600640 1 -- 172.21.15.32:0/3514988541 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f07ec0b6140 msgr2=0x7f07ec159e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:47:44.426 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:44.423+0000 7f07f4600640 1 --2- 172.21.15.32:0/3514988541 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f07ec0b6140 0x7f07ec159e60 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f07d8034040 tx=0x7f07d8003890 comp rx=0 tx=0).stop 2024-08-03T20:47:44.426 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:44.424+0000 7fe9cbe00640 1 -- 172.21.15.32:0/4036997747 shutdown_connections 2024-08-03T20:47:44.427 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:44.424+0000 7f07f4600640 1 -- 172.21.15.32:0/3514988541 shutdown_connections 2024-08-03T20:47:44.427 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:44.424+0000 7f07f4600640 1 --2- 172.21.15.32:0/3514988541 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f07cc077060 0x7f07cc079570 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:44.427 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:44.424+0000 7f07f4600640 1 --2- 172.21.15.32:0/3514988541 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f07ec0cf540 0x7f07ec15a3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:44.427 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:44.424+0000 7f07f4600640 1 --2- 172.21.15.32:0/3514988541 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f07ec0b6140 0x7f07ec159e60 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:44.427 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:44.424+0000 7f07f4600640 1 --2- 172.21.15.32:0/3514988541 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f07ec09af90 0x7f07ec0b0000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:44.427 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:44.424+0000 7f07f4600640 1 -- 172.21.15.32:0/3514988541 >> 172.21.15.32:0/3514988541 conn(0x7f07ec0d2160 msgr2=0x7f07ec0b2270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:47:44.428 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:44.424+0000 7fe9cbe00640 1 --2- 172.21.15.32:0/4036997747 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fe9a8077010 0x7fe9a8079520 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:44.428 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:44.424+0000 7fe9cbe00640 1 --2- 172.21.15.32:0/4036997747 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fe9cc0d1d50 0x7fe9cc0a04a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:44.428 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:44.424+0000 7fe9cbe00640 1 --2- 172.21.15.32:0/4036997747 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe9cc162b80 0x7fe9cc0a09e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:44.428 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:44.424+0000 7fe9cbe00640 1 --2- 172.21.15.32:0/4036997747 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fe9cc0cec50 0x7fe9cc0c7130 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:44.428 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:44.424+0000 7fe9cbe00640 1 -- 172.21.15.32:0/4036997747 >> 172.21.15.32:0/4036997747 conn(0x7fe9cc0d4180 msgr2=0x7fe9cc161660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:47:44.428 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:44.424+0000 7f07f4600640 1 -- 172.21.15.32:0/3514988541 shutdown_connections 2024-08-03T20:47:44.428 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:44.424+0000 7f07f4600640 1 -- 172.21.15.32:0/3514988541 wait complete. 2024-08-03T20:47:44.429 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:44.425+0000 7fe9cbe00640 1 -- 172.21.15.32:0/4036997747 shutdown_connections 2024-08-03T20:47:44.429 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:44.425+0000 7fe9cbe00640 1 -- 172.21.15.32:0/4036997747 wait complete. 2024-08-03T20:47:44.429 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:44.428+0000 7f642b400640 1 -- 172.21.15.32:0/2574166090 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) -- 0x7f6414005850 con 0x7f644006e500 2024-08-03T20:47:44.429 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:44.428+0000 7f497a800640 1 -- 172.21.15.32:0/3185224006 --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) -- 0x7f494c005850 con 0x7f49740733f0 2024-08-03T20:47:44.430 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:44.428+0000 7f6436000640 1 -- 172.21.15.32:0/2574166090 <== mon.1 v2:172.21.15.118:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f642c081720 con 0x7f644006e500 2024-08-03T20:47:44.430 INFO:teuthology.orchestra.run.smithi032.stdout:60129542149 2024-08-03T20:47:44.431 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:44.429+0000 7f496be00640 1 -- 172.21.15.32:0/3185224006 <== mon.2 v2:172.21.15.118:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f4964082c70 con 0x7f49740733f0 2024-08-03T20:47:44.431 INFO:teuthology.orchestra.run.smithi032.stdout:64424509445 2024-08-03T20:47:44.434 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:44.433+0000 7f642b400640 1 -- 172.21.15.32:0/2574166090 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f6420077040 msgr2=0x7f6420079550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:47:44.434 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:44.433+0000 7f642b400640 1 --2- 172.21.15.32:0/2574166090 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f6420077040 0x7f6420079550 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f6430006fd0 tx=0x7f6430008040 comp rx=0 tx=0).stop 2024-08-03T20:47:44.434 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:44.433+0000 7f642b400640 1 -- 172.21.15.32:0/2574166090 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f644006e500 msgr2=0x7f644015c550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:47:44.435 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:44.433+0000 7f497a800640 1 -- 172.21.15.32:0/3185224006 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f4950077240 msgr2=0x7f4950079750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:47:44.435 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:44.433+0000 7f497a800640 1 --2- 172.21.15.32:0/3185224006 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f4950077240 0x7f4950079750 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f4974145b80 tx=0x7f495c009040 comp rx=0 tx=0).stop 2024-08-03T20:47:44.435 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:44.433+0000 7f497a800640 1 -- 172.21.15.32:0/3185224006 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f49740733f0 msgr2=0x7f497413f960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:47:44.435 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:44.433+0000 7f642b400640 1 --2- 172.21.15.32:0/2574166090 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f644006e500 0x7f644015c550 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f6440073ea0 tx=0x7f642c005270 comp rx=0 tx=0).stop 2024-08-03T20:47:44.435 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:44.433+0000 7f497a800640 1 --2- 172.21.15.32:0/3185224006 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f49740733f0 0x7f497413f960 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f4964009d90 tx=0x7f49640045e0 comp rx=0 tx=0).stop 2024-08-03T20:47:44.436 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:44.433+0000 7f642b400640 1 -- 172.21.15.32:0/2574166090 shutdown_connections 2024-08-03T20:47:44.436 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:44.433+0000 7f642b400640 1 --2- 172.21.15.32:0/2574166090 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f6420077040 0x7f6420079550 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:44.436 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:44.433+0000 7f642b400640 1 --2- 172.21.15.32:0/2574166090 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f64400b79d0 0x7f6440162220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:44.436 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:44.433+0000 7f642b400640 1 --2- 172.21.15.32:0/2574166090 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f644009aa50 0x7f6440161ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:44.436 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:44.433+0000 7f642b400640 1 --2- 172.21.15.32:0/2574166090 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f644006e500 0x7f644015c550 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:44.436 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:44.433+0000 7f642b400640 1 -- 172.21.15.32:0/2574166090 >> 172.21.15.32:0/2574166090 conn(0x7f64400b3c20 msgr2=0x7f644015b4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:47:44.436 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:44.433+0000 7f642b400640 1 -- 172.21.15.32:0/2574166090 shutdown_connections 2024-08-03T20:47:44.436 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:44.433+0000 7f642b400640 1 -- 172.21.15.32:0/2574166090 wait complete. 2024-08-03T20:47:44.437 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:44.434+0000 7f497a800640 1 -- 172.21.15.32:0/3185224006 shutdown_connections 2024-08-03T20:47:44.437 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:44.434+0000 7f497a800640 1 --2- 172.21.15.32:0/3185224006 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f4950077240 0x7f4950079750 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:44.437 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:44.434+0000 7f497a800640 1 --2- 172.21.15.32:0/3185224006 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f4974144f80 0x7f4974152c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:44.437 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:44.434+0000 7f497a800640 1 --2- 172.21.15.32:0/3185224006 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f49740b4de0 0x7f4974152730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:44.437 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:44.434+0000 7f497a800640 1 --2- 172.21.15.32:0/3185224006 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f49740733f0 0x7f497413f960 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:44.438 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:44.434+0000 7f497a800640 1 -- 172.21.15.32:0/3185224006 >> 172.21.15.32:0/3185224006 conn(0x7f49740b1bd0 msgr2=0x7f49740cc250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:47:44.438 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:44.434+0000 7f497a800640 1 -- 172.21.15.32:0/3185224006 shutdown_connections 2024-08-03T20:47:44.438 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:44.434+0000 7f497a800640 1 -- 172.21.15.32:0/3185224006 wait complete. 2024-08-03T20:47:44.439 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542148 got 60129542147 for osd.3 2024-08-03T20:47:44.447 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542149 got 60129542148 for osd.2 2024-08-03T20:47:44.451 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509445 got 64424509444 for osd.1 2024-08-03T20:47:44.453 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509446 got 64424509445 for osd.5 2024-08-03T20:47:44.462 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509445 got 64424509444 for osd.7 2024-08-03T20:47:44.463 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509446 got 64424509445 for osd.6 2024-08-03T20:47:44.467 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542150 got 60129542149 for osd.4 2024-08-03T20:47:45.409 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-08-03T20:47:45.439 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-08-03T20:47:45.448 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-08-03T20:47:45.452 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-08-03T20:47:45.454 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-08-03T20:47:45.463 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2024-08-03T20:47:45.464 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2024-08-03T20:47:45.469 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-08-03T20:47:45.773 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:45.771+0000 7fbebac00640 1 -- 172.21.15.32:0/584926051 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fbeb4073760 msgr2=0x7fbeb4073b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:47:45.774 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:45.771+0000 7fbeb2a00640 1 -- 172.21.15.32:0/584926051 <== mon.2 v2:172.21.15.118:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbea40016d0 con 0x7fbeb4073760 2024-08-03T20:47:45.774 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:45.771+0000 7fbebac00640 1 --2- 172.21.15.32:0/584926051 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fbeb4073760 0x7fbeb4073b70 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fbea0009830 tx=0x7fbea002a4d0 comp rx=0 tx=0).stop 2024-08-03T20:47:45.774 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:45.772+0000 7fbebac00640 1 -- 172.21.15.32:0/584926051 shutdown_connections 2024-08-03T20:47:45.774 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:45.772+0000 7fbebac00640 1 --2- 172.21.15.32:0/584926051 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fbeb4073760 0x7fbeb4073b70 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:45.774 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:45.772+0000 7fbebac00640 1 --2- 172.21.15.32:0/584926051 >> v2:172.21.15.32:3300/0 conn(0x7fbeb415d660 0x7fbeb40cd340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:45.774 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:45.772+0000 7fbebac00640 1 -- 172.21.15.32:0/584926051 >> 172.21.15.32:0/584926051 conn(0x7fbeb40cff30 msgr2=0x7fbeb40d0340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:47:45.774 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:45.772+0000 7fbebac00640 1 -- 172.21.15.32:0/584926051 shutdown_connections 2024-08-03T20:47:45.776 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:45.772+0000 7fbebac00640 1 -- 172.21.15.32:0/584926051 wait complete. 2024-08-03T20:47:45.778 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:45.777+0000 7fbebac00640 1 Processor -- start 2024-08-03T20:47:45.778 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:45.777+0000 7fbebac00640 1 -- start start 2024-08-03T20:47:45.782 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:45.781+0000 7fbebac00640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fbeb4073760 0x7fbeb40ab1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:47:45.782 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:45.781+0000 7fbebac00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbeb40cee70 0x7fbeb40ab700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:47:45.783 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:45.781+0000 7fbeb3e00640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fbeb4073760 0x7fbeb40ab1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:47:45.783 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:45.781+0000 7fbeb3e00640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fbeb4073760 0x7fbeb40ab1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3301/0 says I am v2:172.21.15.32:36610/0 (socket says 172.21.15.32:36610) 2024-08-03T20:47:45.783 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:45.782+0000 7fbeb3e00640 1 -- 172.21.15.32:0/3538299134 learned_addr learned my addr 172.21.15.32:0/3538299134 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:47:45.784 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:45.782+0000 7fbebac00640 1 --2- 172.21.15.32:0/3538299134 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fbeb415d660 0x7fbeb40ad490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:47:45.784 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:45.783+0000 7fbebac00640 1 -- 172.21.15.32:0/3538299134 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fbeb415e090 con 0x7fbeb40cee70 2024-08-03T20:47:45.784 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:45.783+0000 7fbebac00640 1 -- 172.21.15.32:0/3538299134 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7fbeb415df10 con 0x7fbeb415d660 2024-08-03T20:47:45.784 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:45.783+0000 7fbebac00640 1 -- 172.21.15.32:0/3538299134 --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7fbeb415e210 con 0x7fbeb4073760 2024-08-03T20:47:45.785 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:45.783+0000 7fbeb3400640 1 --2- 172.21.15.32:0/3538299134 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbeb40cee70 0x7fbeb40ab700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:47:45.785 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:45.783+0000 7fbeb8c00640 1 --2- 172.21.15.32:0/3538299134 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fbeb415d660 0x7fbeb40ad490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:47:45.786 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:45.785+0000 7fbeb3400640 1 -- 172.21.15.32:0/3538299134 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fbeb4073760 msgr2=0x7fbeb40ab1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:47:45.786 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:45.785+0000 7fbeb3400640 1 --2- 172.21.15.32:0/3538299134 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fbeb4073760 0x7fbeb40ab1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:45.787 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:45.785+0000 7fbeb3400640 1 -- 172.21.15.32:0/3538299134 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fbeb415d660 msgr2=0x7fbeb40ad490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:47:45.787 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:45.785+0000 7fbeb3400640 1 --2- 172.21.15.32:0/3538299134 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fbeb415d660 0x7fbeb40ad490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:45.788 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:45.786+0000 7fbeb3400640 1 -- 172.21.15.32:0/3538299134 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbeb40abe80 con 0x7fbeb40cee70 2024-08-03T20:47:45.788 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:45.786+0000 7fbeb8c00640 1 --2- 172.21.15.32:0/3538299134 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fbeb415d660 0x7fbeb40ad490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-03T20:47:45.788 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:45.786+0000 7fbeb3e00640 1 --2- 172.21.15.32:0/3538299134 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fbeb4073760 0x7fbeb40ab1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-03T20:47:45.789 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:45.787+0000 7fbeb3400640 1 --2- 172.21.15.32:0/3538299134 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbeb40cee70 0x7fbeb40ab700 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fbeb40cca70 tx=0x7fbea0005850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:47:45.789 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:45.788+0000 7fbeabe00640 1 -- 172.21.15.32:0/3538299134 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbea0005020 con 0x7fbeb40cee70 2024-08-03T20:47:45.790 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:45.788+0000 7fbeabe00640 1 -- 172.21.15.32:0/3538299134 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fbea00052b0 con 0x7fbeb40cee70 2024-08-03T20:47:45.790 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:45.789+0000 7fbebac00640 1 -- 172.21.15.32:0/3538299134 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbeb4159a30 con 0x7fbeb40cee70 2024-08-03T20:47:45.790 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:45.789+0000 7fbebac00640 1 -- 172.21.15.32:0/3538299134 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbeb4159ff0 con 0x7fbeb40cee70 2024-08-03T20:47:45.790 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:45.789+0000 7fbeabe00640 1 -- 172.21.15.32:0/3538299134 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbea00047a0 con 0x7fbeb40cee70 2024-08-03T20:47:45.797 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:45.795+0000 7fbeabe00640 1 -- 172.21.15.32:0/3538299134 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7fbea0004940 con 0x7fbeb40cee70 2024-08-03T20:47:45.799 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:45.797+0000 7fbeabe00640 1 --2- 172.21.15.32:0/3538299134 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fbe94077010 0x7fbe94079520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:47:45.799 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:45.798+0000 7fbeabe00640 1 -- 172.21.15.32:0/3538299134 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7fbea00b9b40 con 0x7fbeb40cee70 2024-08-03T20:47:45.800 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:45.799+0000 7fbea9600640 1 -- 172.21.15.32:0/3538299134 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbeb4074ea0 con 0x7fbeb40cee70 2024-08-03T20:47:45.800 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:45.799+0000 7fbeb3e00640 1 --2- 172.21.15.32:0/3538299134 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fbe94077010 0x7fbe94079520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:47:45.805 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:45.804+0000 7fbeb3e00640 1 --2- 172.21.15.32:0/3538299134 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fbe94077010 0x7fbe94079520 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fbeac007e90 tx=0x7fbeac007870 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:47:45.812 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:45.811+0000 7fbeabe00640 1 -- 172.21.15.32:0/3538299134 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7fbeb4074ea0 con 0x7fbeb40cee70 2024-08-03T20:47:45.851 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:45.849+0000 7f71b3e00640 1 -- 172.21.15.32:0/3256820295 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f71b4163160 msgr2=0x7f71b4165580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:47:45.851 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:45.849+0000 7f71b3e00640 1 --2- 172.21.15.32:0/3256820295 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f71b4163160 0x7f71b4165580 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f71ac009be0 tx=0x7f71ac02a8b0 comp rx=0 tx=0).stop 2024-08-03T20:47:45.852 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:45.850+0000 7f71b3e00640 1 -- 172.21.15.32:0/3256820295 shutdown_connections 2024-08-03T20:47:45.852 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:45.850+0000 7f71b3e00640 1 --2- 172.21.15.32:0/3256820295 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f71b4163160 0x7f71b4165580 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:45.852 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:45.850+0000 7f71b3e00640 1 -- 172.21.15.32:0/3256820295 >> 172.21.15.32:0/3256820295 conn(0x7f71b40b1d50 msgr2=0x7f71b40b2160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:47:45.853 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:45.851+0000 7f71b3e00640 1 -- 172.21.15.32:0/3256820295 shutdown_connections 2024-08-03T20:47:45.854 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:45.852+0000 7f71b3e00640 1 -- 172.21.15.32:0/3256820295 wait complete. 2024-08-03T20:47:45.854 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:45.852+0000 7facdec00640 1 -- 172.21.15.32:0/1966699203 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7facd00072e0 msgr2=0x7facd0035cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:47:45.854 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:45.852+0000 7facdec00640 1 --2- 172.21.15.32:0/1966699203 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7facd00072e0 0x7facd0035cd0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7facd00e14c0 tx=0x7faccc02a560 comp rx=0 tx=0).stop 2024-08-03T20:47:45.854 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:45.853+0000 7facdec00640 1 -- 172.21.15.32:0/1966699203 shutdown_connections 2024-08-03T20:47:45.854 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:45.853+0000 7facdec00640 1 --2- 172.21.15.32:0/1966699203 >> v2:172.21.15.32:3300/0 conn(0x7facd0006140 0x7facd0054720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:45.855 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:45.853+0000 7facdec00640 1 --2- 172.21.15.32:0/1966699203 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7facd00e0e80 0x7facd0005c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:45.855 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:45.853+0000 7facdec00640 1 --2- 172.21.15.32:0/1966699203 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7facd00072e0 0x7facd0035cd0 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:45.855 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:45.853+0000 7facdec00640 1 -- 172.21.15.32:0/1966699203 >> 172.21.15.32:0/1966699203 conn(0x7facd001d9a0 msgr2=0x7facd0039660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:47:45.855 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:45.853+0000 7facdec00640 1 -- 172.21.15.32:0/1966699203 shutdown_connections 2024-08-03T20:47:45.855 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:45.853+0000 7facdec00640 1 -- 172.21.15.32:0/1966699203 wait complete. 2024-08-03T20:47:45.856 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:45.854+0000 7f04cfe00640 1 -- 172.21.15.32:0/802517813 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f04c800a970 msgr2=0x7f04c801c990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:47:45.856 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:45.854+0000 7f04cfe00640 1 --2- 172.21.15.32:0/802517813 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f04c800a970 0x7f04c801c990 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f04bc009830 tx=0x7f04bc02a460 comp rx=0 tx=0).stop 2024-08-03T20:47:45.857 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:45.854+0000 7f04cfe00640 1 -- 172.21.15.32:0/802517813 shutdown_connections 2024-08-03T20:47:45.857 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:45.854+0000 7f04cfe00640 1 --2- 172.21.15.32:0/802517813 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f04c800a970 0x7f04c801c990 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:45.857 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:45.854+0000 7f04cfe00640 1 --2- 172.21.15.32:0/802517813 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f04c8009da0 0x7f04c80366f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:45.857 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:45.854+0000 7f04cfe00640 1 -- 172.21.15.32:0/802517813 >> 172.21.15.32:0/802517813 conn(0x7f04c8051420 msgr2=0x7f04c8051860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:47:45.857 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:45.854+0000 7f04cfe00640 1 -- 172.21.15.32:0/802517813 shutdown_connections 2024-08-03T20:47:45.857 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:45.854+0000 7f04cfe00640 1 -- 172.21.15.32:0/802517813 wait complete. 2024-08-03T20:47:45.857 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:45.855+0000 7f71b3e00640 1 Processor -- start 2024-08-03T20:47:45.858 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:45.856+0000 7f71b3e00640 1 -- start start 2024-08-03T20:47:45.859 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:45.858+0000 7facdec00640 1 Processor -- start 2024-08-03T20:47:45.859 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:45.858+0000 7facdec00640 1 -- start start 2024-08-03T20:47:45.860 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:45.858+0000 7f71b3e00640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f71b40ca5c0 0x7f71b4164e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:47:45.861 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:45.859+0000 7f71b3e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f71b4157fc0 0x7f71b40b4c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:47:45.861 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:45.859+0000 7f71b2a00640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f71b40ca5c0 0x7f71b4164e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:47:45.861 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:45.859+0000 7f04cfe00640 1 Processor -- start 2024-08-03T20:47:45.861 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:45.859+0000 7f04cfe00640 1 -- start start 2024-08-03T20:47:45.862 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:45.859+0000 7f71b2a00640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f71b40ca5c0 0x7f71b4164e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3301/0 says I am v2:172.21.15.32:36646/0 (socket says 172.21.15.32:36646) 2024-08-03T20:47:45.862 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:45.859+0000 7f71b3e00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f71b4163160 0x7f71b4073540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:47:45.862 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:45.859+0000 7f71b3e00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f71b40cce10 con 0x7f71b4157fc0 2024-08-03T20:47:45.862 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:45.859+0000 7f71b3e00640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f71b415b8f0 con 0x7f71b4163160 2024-08-03T20:47:45.862 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:45.859+0000 7f71b3e00640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7f71b4165ac0 con 0x7f71b40ca5c0 2024-08-03T20:47:45.862 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:45.860+0000 7f71b2a00640 1 -- 172.21.15.32:0/3771676324 learned_addr learned my addr 172.21.15.32:0/3771676324 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:47:45.863 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:45.861+0000 7f71b3400640 1 --2- 172.21.15.32:0/3771676324 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f71b4163160 0x7f71b4073540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:47:45.863 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:45.861+0000 7f71b2000640 1 --2- 172.21.15.32:0/3771676324 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f71b4157fc0 0x7f71b40b4c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:47:45.863 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:45.861+0000 7f04cfe00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f04c8009da0 0x7f04c80dc8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:47:45.863 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:45.861+0000 7f04cfe00640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f04c800a970 0x7f04c804c5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:47:45.863 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:45.861+0000 7f04cfe00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f04c8036c30 0x7f04c8052360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:47:45.863 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:45.861+0000 7f04cfe00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f04c801d260 con 0x7f04c8036c30 2024-08-03T20:47:45.863 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:45.861+0000 7f04cfe00640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f04c801d0e0 con 0x7f04c8009da0 2024-08-03T20:47:45.864 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:45.861+0000 7f04cfe00640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7f04c801d3e0 con 0x7f04c800a970 2024-08-03T20:47:45.864 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:45.862+0000 7facdec00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7facd0006140 0x7facd00482c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:47:45.864 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:45.862+0000 7f04ce000640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f04c800a970 0x7f04c804c5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:47:45.864 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:45.862+0000 7f04ce000640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f04c800a970 0x7f04c804c5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3301/0 says I am v2:172.21.15.32:36656/0 (socket says 172.21.15.32:36656) 2024-08-03T20:47:45.865 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:45.862+0000 7f04ce000640 1 -- 172.21.15.32:0/317629542 learned_addr learned my addr 172.21.15.32:0/317629542 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:47:45.865 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:45.863+0000 7f04cea00640 1 --2- 172.21.15.32:0/317629542 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f04c8009da0 0x7f04c80dc8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:47:45.865 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:45.863+0000 7f04cf400640 1 --2- 172.21.15.32:0/317629542 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f04c8036c30 0x7f04c8052360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:47:45.865 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:45.863+0000 7facdec00640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7facd00072e0 0x7facd00dd770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:47:45.865 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:45.862+0000 7f71b2a00640 1 -- 172.21.15.32:0/3771676324 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f71b4163160 msgr2=0x7f71b4073540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:47:45.865 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:45.862+0000 7f71b2a00640 1 --2- 172.21.15.32:0/3771676324 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f71b4163160 0x7f71b4073540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:45.865 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:45.862+0000 7f71b2a00640 1 -- 172.21.15.32:0/3771676324 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f71b4157fc0 msgr2=0x7f71b40b4c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:47:45.865 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:45.862+0000 7f71b2a00640 1 --2- 172.21.15.32:0/3771676324 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f71b4157fc0 0x7f71b40b4c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:45.866 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:45.862+0000 7f71b2a00640 1 -- 172.21.15.32:0/3771676324 --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f71b41f9da0 con 0x7f71b40ca5c0 2024-08-03T20:47:45.866 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:45.863+0000 7f71b3400640 1 --2- 172.21.15.32:0/3771676324 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f71b4163160 0x7f71b4073540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-03T20:47:45.866 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:45.863+0000 7facdd800640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7facd0006140 0x7facd00482c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:47:45.866 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:45.863+0000 7f04ce000640 1 -- 172.21.15.32:0/317629542 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f04c8009da0 msgr2=0x7f04c80dc8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:47:45.866 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:45.863+0000 7f71b2000640 1 --2- 172.21.15.32:0/3771676324 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f71b4157fc0 0x7f71b40b4c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-03T20:47:45.867 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:45.863+0000 7facdd800640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7facd0006140 0x7facd00482c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.32:46056/0 (socket says 172.21.15.32:46056) 2024-08-03T20:47:45.867 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:45.864+0000 7f71b2a00640 1 --2- 172.21.15.32:0/3771676324 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f71b40ca5c0 0x7f71b4164e20 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f71a400c430 tx=0x7f71a400c8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:47:45.870 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:45.863+0000 7facdd800640 1 -- 172.21.15.32:0/2406000982 learned_addr learned my addr 172.21.15.32:0/2406000982 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:47:45.870 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:45.864+0000 7facdce00640 1 --2- 172.21.15.32:0/2406000982 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7facd00072e0 0x7facd00dd770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:47:45.870 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:45.864+0000 7f04ce000640 1 --2- 172.21.15.32:0/317629542 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f04c8009da0 0x7f04c80dc8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:45.870 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:45.864+0000 7facdec00640 1 --2- 172.21.15.32:0/2406000982 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7facd00e0e80 0x7facd00defc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:47:45.871 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:45.864+0000 7f04ce000640 1 -- 172.21.15.32:0/317629542 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f04c8036c30 msgr2=0x7f04c8052360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:47:45.871 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:45.864+0000 7f04ce000640 1 --2- 172.21.15.32:0/317629542 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f04c8036c30 0x7f04c8052360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:45.871 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:45.864+0000 7f04ce000640 1 -- 172.21.15.32:0/317629542 --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f04c8031b20 con 0x7f04c800a970 2024-08-03T20:47:45.871 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:45.864+0000 7f71aaa00640 1 -- 172.21.15.32:0/3771676324 <== mon.2 v2:172.21.15.118:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f71a4011850 con 0x7f71b40ca5c0 2024-08-03T20:47:45.871 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:45.864+0000 7f71aaa00640 1 -- 172.21.15.32:0/3771676324 <== mon.2 v2:172.21.15.118:3301/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f71a4017070 con 0x7f71b40ca5c0 2024-08-03T20:47:45.872 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:45.864+0000 7f71aaa00640 1 -- 172.21.15.32:0/3771676324 <== mon.2 v2:172.21.15.118:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f71a4011da0 con 0x7f71b40ca5c0 2024-08-03T20:47:45.872 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:45.864+0000 7f04cf400640 1 --2- 172.21.15.32:0/317629542 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f04c8036c30 0x7f04c8052360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-03T20:47:45.872 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:45.864+0000 7facdec00640 1 -- 172.21.15.32:0/2406000982 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7facd0054ff0 con 0x7facd00e0e80 2024-08-03T20:47:45.872 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:45.864+0000 7facdec00640 1 -- 172.21.15.32:0/2406000982 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7facd0054e70 con 0x7facd0006140 2024-08-03T20:47:45.873 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:45.865+0000 7f04ce000640 1 --2- 172.21.15.32:0/317629542 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f04c800a970 0x7f04c804c5f0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f04c000c430 tx=0x7f04c000c8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:47:45.873 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:45.864+0000 7f71b3e00640 1 -- 172.21.15.32:0/3771676324 --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f71b41fa080 con 0x7f71b40ca5c0 2024-08-03T20:47:45.873 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:45.865+0000 7f71b3e00640 1 -- 172.21.15.32:0/3771676324 --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f71b41fa640 con 0x7f71b40ca5c0 2024-08-03T20:47:45.873 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:45.864+0000 7facdec00640 1 -- 172.21.15.32:0/2406000982 --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7facd0055170 con 0x7facd00072e0 2024-08-03T20:47:45.873 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:45.864+0000 7facde200640 1 --2- 172.21.15.32:0/2406000982 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7facd00e0e80 0x7facd00defc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:47:45.874 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:45.864+0000 7f04cea00640 1 --2- 172.21.15.32:0/317629542 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f04c8009da0 0x7f04c80dc8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-03T20:47:45.874 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:45.865+0000 7f04c6a00640 1 -- 172.21.15.32:0/317629542 <== mon.2 v2:172.21.15.118:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04c0011790 con 0x7f04c800a970 2024-08-03T20:47:45.874 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:45.865+0000 7f04c6a00640 1 -- 172.21.15.32:0/317629542 <== mon.2 v2:172.21.15.118:3301/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f04c0017070 con 0x7f04c800a970 2024-08-03T20:47:45.874 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:45.865+0000 7f04c6a00640 1 -- 172.21.15.32:0/317629542 <== mon.2 v2:172.21.15.118:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04c0011cc0 con 0x7f04c800a970 2024-08-03T20:47:45.875 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:45.866+0000 7facde200640 1 -- 172.21.15.32:0/2406000982 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7facd00072e0 msgr2=0x7facd00dd770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:47:45.875 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:45.866+0000 7facde200640 1 --2- 172.21.15.32:0/2406000982 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7facd00072e0 0x7facd00dd770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:45.875 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:45.866+0000 7facde200640 1 -- 172.21.15.32:0/2406000982 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7facd0006140 msgr2=0x7facd00482c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:47:45.875 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:45.866+0000 7facde200640 1 --2- 172.21.15.32:0/2406000982 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7facd0006140 0x7facd00482c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:45.875 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:45.866+0000 7facde200640 1 -- 172.21.15.32:0/2406000982 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7facd004a820 con 0x7facd00e0e80 2024-08-03T20:47:45.875 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:45.867+0000 7facde200640 1 --2- 172.21.15.32:0/2406000982 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7facd00e0e80 0x7facd00defc0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7facd806b6f0 tx=0x7facd806ef40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:47:45.875 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:45.867+0000 7f04cfe00640 1 -- 172.21.15.32:0/317629542 --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f04c8031e00 con 0x7f04c800a970 2024-08-03T20:47:45.876 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:45.867+0000 7f71b3e00640 1 -- 172.21.15.32:0/3771676324 --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f71b4165360 con 0x7f71b40ca5c0 2024-08-03T20:47:45.876 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:45.868+0000 7facd5600640 1 -- 172.21.15.32:0/2406000982 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7facd8093600 con 0x7facd00e0e80 2024-08-03T20:47:45.876 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:45.868+0000 7f1e08c00640 1 -- 172.21.15.32:0/332755179 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1e040cf510 msgr2=0x7f1e040cf920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:47:45.877 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:45.868+0000 7facd5600640 1 -- 172.21.15.32:0/2406000982 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7facd80938b0 con 0x7facd00e0e80 2024-08-03T20:47:45.877 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:45.868+0000 7facdec00640 1 -- 172.21.15.32:0/2406000982 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7facd004ab00 con 0x7facd00e0e80 2024-08-03T20:47:45.877 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:45.868+0000 7facdec00640 1 -- 172.21.15.32:0/2406000982 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7facd004b0c0 con 0x7facd00e0e80 2024-08-03T20:47:45.877 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:45.868+0000 7f1e08c00640 1 --2- 172.21.15.32:0/332755179 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1e040cf510 0x7f1e040cf920 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f1e0408dd50 tx=0x7f1dfc02a5a0 comp rx=0 tx=0).stop 2024-08-03T20:47:45.877 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:45.868+0000 7facd5600640 1 -- 172.21.15.32:0/2406000982 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7facd8093dc0 con 0x7facd00e0e80 2024-08-03T20:47:45.878 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:45.868+0000 7f04cfe00640 1 -- 172.21.15.32:0/317629542 --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f04c80323c0 con 0x7f04c800a970 2024-08-03T20:47:45.878 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:45.869+0000 7f1e08c00640 1 -- 172.21.15.32:0/332755179 shutdown_connections 2024-08-03T20:47:45.878 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:45.869+0000 7f1e08c00640 1 --2- 172.21.15.32:0/332755179 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1e040cf510 0x7f1e040cf920 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:45.878 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:45.869+0000 7f1e08c00640 1 --2- 172.21.15.32:0/332755179 >> v2:172.21.15.118:3300/0 conn(0x7f1e040d26e0 0x7f1e04161b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:45.879 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:45.869+0000 7f1e08c00640 1 -- 172.21.15.32:0/332755179 >> 172.21.15.32:0/332755179 conn(0x7f1e0408d7a0 msgr2=0x7f1e0409d3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:47:45.879 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:45.870+0000 7facdec00640 1 -- 172.21.15.32:0/2406000982 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7facb4005290 con 0x7facd00e0e80 2024-08-03T20:47:45.879 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:45.870+0000 7f71aaa00640 1 -- 172.21.15.32:0/3771676324 <== mon.2 v2:172.21.15.118:3301/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f71a401c460 con 0x7f71b40ca5c0 2024-08-03T20:47:45.879 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:45.871+0000 7f71aaa00640 1 --2- 172.21.15.32:0/3771676324 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f71900770e0 0x7f71900795f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:47:45.879 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:45.872+0000 7f71aaa00640 1 -- 172.21.15.32:0/3771676324 <== mon.2 v2:172.21.15.118:3301/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f71a4098700 con 0x7f71b40ca5c0 2024-08-03T20:47:45.880 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:45.871+0000 7facd5600640 1 -- 172.21.15.32:0/2406000982 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7facd809c3f0 con 0x7facd00e0e80 2024-08-03T20:47:45.880 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:45.873+0000 7f1e08c00640 1 -- 172.21.15.32:0/332755179 shutdown_connections 2024-08-03T20:47:45.880 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:45.873+0000 7f71b2000640 1 --2- 172.21.15.32:0/3771676324 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f71900770e0 0x7f71900795f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:47:45.881 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:45.873+0000 7f04cfe00640 1 -- 172.21.15.32:0/317629542 --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f04c80366f0 con 0x7f04c800a970 2024-08-03T20:47:45.881 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:45.873+0000 7f1e08c00640 1 -- 172.21.15.32:0/332755179 wait complete. 2024-08-03T20:47:45.881 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:45.873+0000 7facd5600640 1 --2- 172.21.15.32:0/2406000982 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7facb8077090 0x7facb80795a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:47:45.881 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:45.874+0000 7facd5600640 1 -- 172.21.15.32:0/2406000982 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7facd8113790 con 0x7facd00e0e80 2024-08-03T20:47:45.882 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:45.874+0000 7facdd800640 1 --2- 172.21.15.32:0/2406000982 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7facb8077090 0x7facb80795a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:47:45.882 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:45.875+0000 7facdd800640 1 --2- 172.21.15.32:0/2406000982 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7facb8077090 0x7facb80795a0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7faccc009660 tx=0x7faccc004730 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:47:45.882 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:45.875+0000 7f71b2000640 1 --2- 172.21.15.32:0/3771676324 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f71900770e0 0x7f71900795f0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f71b415efc0 tx=0x7f719c009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:47:45.882 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:45.876+0000 7f71aaa00640 1 -- 172.21.15.32:0/3771676324 <== mon.2 v2:172.21.15.118:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7f71a40618e0 con 0x7f71b40ca5c0 2024-08-03T20:47:45.882 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:45.878+0000 7f1e08c00640 1 Processor -- start 2024-08-03T20:47:45.883 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:45.880+0000 7facd5600640 1 -- 172.21.15.32:0/2406000982 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7facd808f030 con 0x7facd00e0e80 2024-08-03T20:47:45.883 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:45.878+0000 7f1e08c00640 1 -- start start 2024-08-03T20:47:45.883 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:45.881+0000 7f1e08c00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f1e040cf510 0x7f1e040cd720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:47:45.883 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:45.881+0000 7f1e08c00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1e040cffe0 0x7f1e041f4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:47:45.883 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:45.881+0000 7f1e08c00640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f1e040d26e0 0x7f1e041f53a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:47:45.883 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:45.881+0000 7f1e08c00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f1e040d4100 con 0x7f1e040cffe0 2024-08-03T20:47:45.883 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:45.881+0000 7f1e08c00640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f1e040d3f80 con 0x7f1e040cf510 2024-08-03T20:47:45.884 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:45.881+0000 7f1e08c00640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7f1e040d4280 con 0x7f1e040d26e0 2024-08-03T20:47:45.884 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:45.879+0000 7f04c6a00640 1 -- 172.21.15.32:0/317629542 <== mon.2 v2:172.21.15.118:3301/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f04c001c460 con 0x7f04c800a970 2024-08-03T20:47:45.884 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:45.881+0000 7f04c6a00640 1 --2- 172.21.15.32:0/317629542 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f04b0077010 0x7f04b0079520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:47:45.884 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:45.881+0000 7f04c6a00640 1 -- 172.21.15.32:0/317629542 <== mon.2 v2:172.21.15.118:3301/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f04c005e3f0 con 0x7f04c800a970 2024-08-03T20:47:45.884 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:45.881+0000 7f1dfaa00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1e040cffe0 0x7f1e041f4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:47:45.884 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:45.881+0000 7f1e03400640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f1e040cf510 0x7f1e040cd720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:47:45.885 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:45.882+0000 7f1e03400640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f1e040cf510 0x7f1e040cd720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.32:46064/0 (socket says 172.21.15.32:46064) 2024-08-03T20:47:45.885 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:45.882+0000 7f1e03400640 1 -- 172.21.15.32:0/3140339105 learned_addr learned my addr 172.21.15.32:0/3140339105 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:47:45.885 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:45.882+0000 7f04cea00640 1 --2- 172.21.15.32:0/317629542 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f04b0077010 0x7f04b0079520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:47:45.885 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:45.882+0000 7f04c6a00640 1 -- 172.21.15.32:0/317629542 <== mon.2 v2:172.21.15.118:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7f04c0062500 con 0x7f04c800a970 2024-08-03T20:47:45.886 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:45.882+0000 7f1e03e00640 1 --2- 172.21.15.32:0/3140339105 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f1e040d26e0 0x7f1e041f53a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:47:45.887 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:45.882+0000 7f1dfaa00640 1 -- 172.21.15.32:0/3140339105 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f1e040d26e0 msgr2=0x7f1e041f53a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:47:45.887 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:45.883+0000 7f1dfaa00640 1 --2- 172.21.15.32:0/3140339105 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f1e040d26e0 0x7f1e041f53a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:45.887 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:45.883+0000 7f1dfaa00640 1 -- 172.21.15.32:0/3140339105 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f1e040cf510 msgr2=0x7f1e040cd720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:47:45.887 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:45.883+0000 7f1dfaa00640 1 --2- 172.21.15.32:0/3140339105 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f1e040cf510 0x7f1e040cd720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:45.887 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:45.883+0000 7f1dfaa00640 1 -- 172.21.15.32:0/3140339105 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1e041f5ba0 con 0x7f1e040cffe0 2024-08-03T20:47:45.887 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:45.883+0000 7f1e03400640 1 --2- 172.21.15.32:0/3140339105 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f1e040cf510 0x7f1e040cd720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-03T20:47:45.887 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:45.884+0000 7f1dfaa00640 1 --2- 172.21.15.32:0/3140339105 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1e040cffe0 0x7f1e041f4e60 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f1dec00c3c0 tx=0x7f1dec00c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:47:45.887 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:45.884+0000 7f1dfbe00640 1 -- 172.21.15.32:0/3140339105 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1dec011790 con 0x7f1e040cffe0 2024-08-03T20:47:45.887 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:45.885+0000 7f1dfbe00640 1 -- 172.21.15.32:0/3140339105 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f1dec017070 con 0x7f1e040cffe0 2024-08-03T20:47:45.887 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:45.885+0000 7f1dfbe00640 1 -- 172.21.15.32:0/3140339105 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1dec011cc0 con 0x7f1e040cffe0 2024-08-03T20:47:45.888 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:45.884+0000 7f04cea00640 1 --2- 172.21.15.32:0/317629542 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f04b0077010 0x7f04b0079520 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f04d00520f0 tx=0x7f04d0075430 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:47:45.888 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:45.886+0000 7f1e08c00640 1 -- 172.21.15.32:0/3140339105 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1e041f6140 con 0x7f1e040cffe0 2024-08-03T20:47:45.888 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:45.887+0000 7f1e08c00640 1 -- 172.21.15.32:0/3140339105 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1e041f7ca0 con 0x7f1e040cffe0 2024-08-03T20:47:45.890 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:45.889+0000 7f1e08c00640 1 -- 172.21.15.32:0/3140339105 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1e041f5910 con 0x7f1e040cffe0 2024-08-03T20:47:45.897 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:45.893+0000 7f1dfbe00640 1 -- 172.21.15.32:0/3140339105 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f1dec026430 con 0x7f1e040cffe0 2024-08-03T20:47:45.897 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:45.895+0000 7f1dfbe00640 1 --2- 172.21.15.32:0/3140339105 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f1de0077060 0x7f1de0079570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:47:45.897 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:45.896+0000 7f1e03400640 1 --2- 172.21.15.32:0/3140339105 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f1de0077060 0x7f1de0079570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:47:45.897 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:45.896+0000 7f1dfbe00640 1 -- 172.21.15.32:0/3140339105 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f1dec05e1b0 con 0x7f1e040cffe0 2024-08-03T20:47:45.899 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:45.898+0000 7f1e03400640 1 --2- 172.21.15.32:0/3140339105 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f1de0077060 0x7f1de0079570 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f1df400f670 tx=0x7f1df40075d0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:47:45.901 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:45.899+0000 7f1dfbe00640 1 -- 172.21.15.32:0/3140339105 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7f1dec061bc0 con 0x7f1e040cffe0 2024-08-03T20:47:45.919 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:45.917+0000 7f3611c00640 1 -- 172.21.15.32:0/300972238 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f360c161d90 msgr2=0x7f360c0d0b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:47:45.919 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:45.917+0000 7f3611c00640 1 --2- 172.21.15.32:0/300972238 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f360c161d90 0x7f360c0d0b10 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f35f8009900 tx=0x7f35f802a5c0 comp rx=0 tx=0).stop 2024-08-03T20:47:45.919 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:45.917+0000 7f3611c00640 1 -- 172.21.15.32:0/300972238 shutdown_connections 2024-08-03T20:47:45.920 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:45.917+0000 7f3611c00640 1 --2- 172.21.15.32:0/300972238 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f360c161d90 0x7f360c0d0b10 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:45.920 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:45.917+0000 7f3611c00640 1 -- 172.21.15.32:0/300972238 >> 172.21.15.32:0/300972238 conn(0x7f360c0d4120 msgr2=0x7f360c0d4530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:47:45.920 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:45.918+0000 7f3611c00640 1 -- 172.21.15.32:0/300972238 shutdown_connections 2024-08-03T20:47:45.920 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:45.918+0000 7f3611c00640 1 -- 172.21.15.32:0/300972238 wait complete. 2024-08-03T20:47:45.924 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:45.923+0000 7f3611c00640 1 Processor -- start 2024-08-03T20:47:45.925 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:45.923+0000 7f3611c00640 1 -- start start 2024-08-03T20:47:45.927 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:45.925+0000 7f673d000640 1 -- 172.21.15.32:0/739390316 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f67380b94d0 msgr2=0x7f67380b8d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:47:45.927 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:45.926+0000 7f673d000640 1 --2- 172.21.15.32:0/739390316 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f67380b94d0 0x7f67380b8d90 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f67380d3f60 tx=0x7f672802a510 comp rx=0 tx=0).stop 2024-08-03T20:47:45.928 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:45.926+0000 7f673d000640 1 -- 172.21.15.32:0/739390316 shutdown_connections 2024-08-03T20:47:45.928 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:45.926+0000 7f673d000640 1 --2- 172.21.15.32:0/739390316 >> v2:172.21.15.118:3300/0 conn(0x7f67380d26b0 0x7f67380d2ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:45.928 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:45.926+0000 7f673d000640 1 --2- 172.21.15.32:0/739390316 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f67380b94d0 0x7f67380b8d90 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:45.928 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:45.926+0000 7f673d000640 1 -- 172.21.15.32:0/739390316 >> 172.21.15.32:0/739390316 conn(0x7f673809d670 msgr2=0x7f67380d43f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:47:45.928 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:45.926+0000 7f3611c00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f360c0b94c0 0x7f360c09e850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:47:45.928 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:45.926+0000 7f673d000640 1 -- 172.21.15.32:0/739390316 shutdown_connections 2024-08-03T20:47:45.929 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:45.926+0000 7f3611c00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f360c0d4c40 0x7f360c15cb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:47:45.929 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:45.926+0000 7f673d000640 1 -- 172.21.15.32:0/739390316 wait complete. 2024-08-03T20:47:45.930 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:45.927+0000 7f3611c00640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f360c161d90 0x7f360c0a0190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:47:45.930 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:45.927+0000 7f3611c00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f360c0d2460 con 0x7f360c0b94c0 2024-08-03T20:47:45.930 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:45.927+0000 7f3611c00640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f360c0d13d0 con 0x7f360c0d4c40 2024-08-03T20:47:45.930 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:45.927+0000 7f3611c00640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7f360c0d25e0 con 0x7f360c161d90 2024-08-03T20:47:45.930 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:45.927+0000 7f360a000640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f360c0d4c40 0x7f360c15cb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:47:45.930 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:45.927+0000 7f360aa00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f360c0b94c0 0x7f360c09e850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:47:45.930 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:45.927+0000 7f360a000640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f360c0d4c40 0x7f360c15cb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.32:46082/0 (socket says 172.21.15.32:46082) 2024-08-03T20:47:45.930 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:45.927+0000 7f360a000640 1 -- 172.21.15.32:0/140235677 learned_addr learned my addr 172.21.15.32:0/140235677 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:47:45.930 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:45.928+0000 7f360b400640 1 --2- 172.21.15.32:0/140235677 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f360c161d90 0x7f360c0a0190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:47:45.931 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:45.928+0000 7f360a000640 1 -- 172.21.15.32:0/140235677 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f360c161d90 msgr2=0x7f360c0a0190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:47:45.931 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:45.928+0000 7f360a000640 1 --2- 172.21.15.32:0/140235677 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f360c161d90 0x7f360c0a0190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:45.931 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:45.928+0000 7f360a000640 1 -- 172.21.15.32:0/140235677 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f360c0b94c0 msgr2=0x7f360c09e850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:47:45.931 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:45.928+0000 7f360a000640 1 --2- 172.21.15.32:0/140235677 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f360c0b94c0 0x7f360c09e850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:45.931 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:45.928+0000 7f360a000640 1 -- 172.21.15.32:0/140235677 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f360c09f0f0 con 0x7f360c0d4c40 2024-08-03T20:47:45.931 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:45.928+0000 7f360aa00640 1 --2- 172.21.15.32:0/140235677 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f360c0b94c0 0x7f360c09e850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-03T20:47:45.931 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:45.928+0000 7f360a000640 1 --2- 172.21.15.32:0/140235677 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f360c0d4c40 0x7f360c15cb70 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f35fc00c3a0 tx=0x7f35fc00c860 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:47:45.932 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:45.929+0000 7f3602a00640 1 -- 172.21.15.32:0/140235677 <== mon.1 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35fc017070 con 0x7f360c0d4c40 2024-08-03T20:47:45.932 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:45.929+0000 7f3602a00640 1 -- 172.21.15.32:0/140235677 <== mon.1 v2:172.21.15.118:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f35fc0053c0 con 0x7f360c0d4c40 2024-08-03T20:47:45.932 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:45.929+0000 7f3611c00640 1 -- 172.21.15.32:0/140235677 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f360c09f3d0 con 0x7f360c0d4c40 2024-08-03T20:47:45.932 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:45.929+0000 7f3602a00640 1 -- 172.21.15.32:0/140235677 <== mon.1 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35fc003850 con 0x7f360c0d4c40 2024-08-03T20:47:45.932 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:45.929+0000 7f3611c00640 1 -- 172.21.15.32:0/140235677 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f360c09f990 con 0x7f360c0d4c40 2024-08-03T20:47:45.933 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:45.930+0000 7f673d000640 1 Processor -- start 2024-08-03T20:47:45.933 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:45.931+0000 7f673d000640 1 -- start start 2024-08-03T20:47:45.933 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:45.931+0000 7f3611c00640 1 -- 172.21.15.32:0/140235677 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f35e4005290 con 0x7f360c0d4c40 2024-08-03T20:47:45.935 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:45.933+0000 7f673d000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f67380b94d0 0x7f6738070ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:47:45.935 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:45.933+0000 7f673d000640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f67380d26b0 0x7f6738160fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:47:45.935 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:45.934+0000 7f673d000640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f67380d4d10 0x7f6738072600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:47:45.935 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:45.934+0000 7f673d000640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f67380b3d20 con 0x7f67380b94d0 2024-08-03T20:47:45.935 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:45.934+0000 7f673d000640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f67380b3ba0 con 0x7f67380d26b0 2024-08-03T20:47:45.935 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:45.934+0000 7f673d000640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7f67380b3ea0 con 0x7f67380d4d10 2024-08-03T20:47:45.936 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:45.934+0000 7f6735e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f67380b94d0 0x7f6738070ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:47:45.936 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:45.935+0000 7f6737200640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f67380d4d10 0x7f6738072600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:47:45.936 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:45.935+0000 7f6737200640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f67380d4d10 0x7f6738072600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3301/0 says I am v2:172.21.15.32:36708/0 (socket says 172.21.15.32:36708) 2024-08-03T20:47:45.936 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:45.935+0000 7f6737200640 1 -- 172.21.15.32:0/271008676 learned_addr learned my addr 172.21.15.32:0/271008676 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:47:45.936 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:45.935+0000 7f6735e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f67380b94d0 0x7f6738070ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:57642/0 (socket says 172.21.15.32:57642) 2024-08-03T20:47:45.937 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:45.935+0000 7f6735400640 1 --2- 172.21.15.32:0/271008676 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f67380d26b0 0x7f6738160fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:47:45.937 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:45.935+0000 7f6737200640 1 -- 172.21.15.32:0/271008676 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f67380d26b0 msgr2=0x7f6738160fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:47:45.937 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:45.935+0000 7f6737200640 1 --2- 172.21.15.32:0/271008676 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f67380d26b0 0x7f6738160fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:45.937 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:45.935+0000 7f6737200640 1 -- 172.21.15.32:0/271008676 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f67380b94d0 msgr2=0x7f6738070ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:47:45.937 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:45.935+0000 7f6737200640 1 --2- 172.21.15.32:0/271008676 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f67380b94d0 0x7f6738070ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:45.937 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:45.935+0000 7f6737200640 1 -- 172.21.15.32:0/271008676 --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6738071560 con 0x7f67380d4d10 2024-08-03T20:47:45.938 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:45.935+0000 7f3602a00640 1 -- 172.21.15.32:0/140235677 <== mon.1 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f35fc01a070 con 0x7f360c0d4c40 2024-08-03T20:47:45.938 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:45.936+0000 7f3602a00640 1 --2- 172.21.15.32:0/140235677 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f35ec077010 0x7f35ec079520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:47:45.938 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:45.936+0000 7f3602a00640 1 -- 172.21.15.32:0/140235677 <== mon.1 v2:172.21.15.118:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f35fc097cd0 con 0x7f360c0d4c40 2024-08-03T20:47:45.939 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:45.937+0000 7f6737200640 1 --2- 172.21.15.32:0/271008676 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f67380d4d10 0x7f6738072600 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f6730009f00 tx=0x7f673000c490 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:47:45.939 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:45.938+0000 7f360aa00640 1 --2- 172.21.15.32:0/140235677 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f35ec077010 0x7f35ec079520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:47:45.939 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:45.938+0000 7f672e000640 1 -- 172.21.15.32:0/271008676 <== mon.2 v2:172.21.15.118:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f673000ce50 con 0x7f67380d4d10 2024-08-03T20:47:45.939 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:45.938+0000 7f672e000640 1 -- 172.21.15.32:0/271008676 <== mon.2 v2:172.21.15.118:3301/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f67300090d0 con 0x7f67380d4d10 2024-08-03T20:47:45.940 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:45.938+0000 7f672e000640 1 -- 172.21.15.32:0/271008676 <== mon.2 v2:172.21.15.118:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6730004b50 con 0x7f67380d4d10 2024-08-03T20:47:45.940 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:45.938+0000 7f673d000640 1 -- 172.21.15.32:0/271008676 --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6738071840 con 0x7f67380d4d10 2024-08-03T20:47:45.940 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:45.939+0000 7f673d000640 1 -- 172.21.15.32:0/271008676 --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f6738071e00 con 0x7f67380d4d10 2024-08-03T20:47:45.941 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:45.939+0000 7f360aa00640 1 --2- 172.21.15.32:0/140235677 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f35ec077010 0x7f35ec079520 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f3604004500 tx=0x7f3604009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:47:45.943 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:45.940+0000 7f673d000640 1 -- 172.21.15.32:0/271008676 --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f67380712d0 con 0x7f67380d4d10 2024-08-03T20:47:45.944 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:45.942+0000 7f3602a00640 1 -- 172.21.15.32:0/140235677 <== mon.1 v2:172.21.15.118:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7f35fc09d020 con 0x7f360c0d4c40 2024-08-03T20:47:45.944 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:45.943+0000 7f672e000640 1 -- 172.21.15.32:0/271008676 <== mon.2 v2:172.21.15.118:3301/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f673001b030 con 0x7f67380d4d10 2024-08-03T20:47:45.947 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:45.945+0000 7f672e000640 1 --2- 172.21.15.32:0/271008676 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f67180770e0 0x7f67180795f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:47:45.947 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:45.945+0000 7f672e000640 1 -- 172.21.15.32:0/271008676 <== mon.2 v2:172.21.15.118:3301/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f6730098840 con 0x7f67380d4d10 2024-08-03T20:47:45.947 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:45.946+0000 7f6735e00640 1 --2- 172.21.15.32:0/271008676 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f67180770e0 0x7f67180795f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:47:45.948 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:45.947+0000 7f6735e00640 1 --2- 172.21.15.32:0/271008676 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f67180770e0 0x7f67180795f0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f6728034000 tx=0x7f6728004730 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:47:45.950 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:45.949+0000 7f672e000640 1 -- 172.21.15.32:0/271008676 <== mon.2 v2:172.21.15.118:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7f6730022060 con 0x7f67380d4d10 2024-08-03T20:47:45.975 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:45.973+0000 7f8071e00640 1 -- 172.21.15.32:0/2855429135 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f806c0d26e0 msgr2=0x7f806c161b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:47:45.976 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:45.975+0000 7f806b400640 1 -- 172.21.15.32:0/2855429135 <== mon.1 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8064033070 con 0x7f806c0d26e0 2024-08-03T20:47:45.976 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:45.975+0000 7f8071e00640 1 --2- 172.21.15.32:0/2855429135 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f806c0d26e0 0x7f806c161b80 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f806c08dd50 tx=0x7f806402aa10 comp rx=0 tx=0).stop 2024-08-03T20:47:45.977 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:45.976+0000 7f8071e00640 1 -- 172.21.15.32:0/2855429135 shutdown_connections 2024-08-03T20:47:45.977 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:45.976+0000 7f8071e00640 1 --2- 172.21.15.32:0/2855429135 >> v2:172.21.15.32:3300/0 conn(0x7f806c0cf510 0x7f806c0cf920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:45.977 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:45.976+0000 7f8071e00640 1 --2- 172.21.15.32:0/2855429135 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f806c0d26e0 0x7f806c161b80 secure :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f806c08dd50 tx=0x7f806402aa10 comp rx=0 tx=0).stop 2024-08-03T20:47:45.977 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:45.976+0000 7f8071e00640 1 -- 172.21.15.32:0/2855429135 >> 172.21.15.32:0/2855429135 conn(0x7f806c08d7a0 msgr2=0x7f806c09d3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:47:45.978 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:45.976+0000 7f8071e00640 1 -- 172.21.15.32:0/2855429135 shutdown_connections 2024-08-03T20:47:45.978 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:45.976+0000 7f8071e00640 1 -- 172.21.15.32:0/2855429135 wait complete. 2024-08-03T20:47:45.984 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:45.981+0000 7f8071e00640 1 Processor -- start 2024-08-03T20:47:45.984 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:45.981+0000 7f8071e00640 1 -- start start 2024-08-03T20:47:45.985 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:45.984+0000 7f8071e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f806c0cf510 0x7f806c0ca9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:47:45.987 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:45.984+0000 7f8070a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f806c0cf510 0x7f806c0ca9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:47:45.987 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:45.985+0000 7f8070a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f806c0cf510 0x7f806c0ca9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:57654/0 (socket says 172.21.15.32:57654) 2024-08-03T20:47:45.987 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:45.985+0000 7f8070a00640 1 -- 172.21.15.32:0/478202081 learned_addr learned my addr 172.21.15.32:0/478202081 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:47:45.987 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:45.985+0000 7f8071e00640 1 --2- 172.21.15.32:0/478202081 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f806c0cffe0 0x7f806c155600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:47:45.987 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:45.985+0000 7f8071e00640 1 --2- 172.21.15.32:0/478202081 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f806c1f5190 0x7f806c0b4830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:47:45.987 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:45.985+0000 7f8071e00640 1 -- 172.21.15.32:0/478202081 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f806c0d4100 con 0x7f806c0cf510 2024-08-03T20:47:45.987 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:45.985+0000 7f8071e00640 1 -- 172.21.15.32:0/478202081 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f806c0d3f80 con 0x7f806c0cffe0 2024-08-03T20:47:45.988 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:45.985+0000 7f8071e00640 1 -- 172.21.15.32:0/478202081 --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7f806c0d4280 con 0x7f806c1f5190 2024-08-03T20:47:45.988 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:45.985+0000 7f806be00640 1 --2- 172.21.15.32:0/478202081 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f806c0cffe0 0x7f806c155600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:47:45.988 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:45.986+0000 7f8071400640 1 --2- 172.21.15.32:0/478202081 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f806c1f5190 0x7f806c0b4830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:47:45.989 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:45.986+0000 7f806be00640 1 -- 172.21.15.32:0/478202081 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f806c1f5190 msgr2=0x7f806c0b4830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:47:45.989 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:45.986+0000 7f806be00640 1 --2- 172.21.15.32:0/478202081 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f806c1f5190 0x7f806c0b4830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:45.989 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:45.986+0000 7f806be00640 1 -- 172.21.15.32:0/478202081 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f806c0cf510 msgr2=0x7f806c0ca9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:47:45.989 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:45.986+0000 7f806be00640 1 --2- 172.21.15.32:0/478202081 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f806c0cf510 0x7f806c0ca9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:45.989 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:45.986+0000 7f806be00640 1 -- 172.21.15.32:0/478202081 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f806c0caf30 con 0x7f806c0cffe0 2024-08-03T20:47:45.989 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:45.986+0000 7f8070a00640 1 --2- 172.21.15.32:0/478202081 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f806c0cf510 0x7f806c0ca9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-03T20:47:45.989 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:45.987+0000 7f806be00640 1 --2- 172.21.15.32:0/478202081 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f806c0cffe0 0x7f806c155600 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f8058001dd0 tx=0x7f805800c7b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:47:45.990 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:45.988+0000 7f8068c00640 1 -- 172.21.15.32:0/478202081 <== mon.1 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8058015d70 con 0x7f806c0cffe0 2024-08-03T20:47:45.990 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:45.988+0000 7f8071e00640 1 -- 172.21.15.32:0/478202081 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f806c1f6110 con 0x7f806c0cffe0 2024-08-03T20:47:45.990 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:45.988+0000 7f8071e00640 1 -- 172.21.15.32:0/478202081 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f806c1f5b50 con 0x7f806c0cffe0 2024-08-03T20:47:45.992 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:45.989+0000 7f8068c00640 1 -- 172.21.15.32:0/478202081 <== mon.1 v2:172.21.15.118:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f8058005020 con 0x7f806c0cffe0 2024-08-03T20:47:45.992 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:45.989+0000 7f8068c00640 1 -- 172.21.15.32:0/478202081 <== mon.1 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8058015d70 con 0x7f806c0cffe0 2024-08-03T20:47:45.993 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:45.992+0000 7f8071e00640 1 -- 172.21.15.32:0/478202081 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8044005290 con 0x7f806c0cffe0 2024-08-03T20:47:45.995 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:45.994+0000 7f8068c00640 1 -- 172.21.15.32:0/478202081 <== mon.1 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f8058005250 con 0x7f806c0cffe0 2024-08-03T20:47:45.997 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:45.996+0000 7f8068c00640 1 --2- 172.21.15.32:0/478202081 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f804c077240 0x7f804c079750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:47:45.998 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:45.997+0000 7f8068c00640 1 -- 172.21.15.32:0/478202081 <== mon.1 v2:172.21.15.118:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f805809fe10 con 0x7f806c0cffe0 2024-08-03T20:47:45.998 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:45.997+0000 7f8070a00640 1 --2- 172.21.15.32:0/478202081 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f804c077240 0x7f804c079750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:47:45.999 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:45.998+0000 7f8070a00640 1 --2- 172.21.15.32:0/478202081 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f804c077240 0x7f804c079750 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f8064000f80 tx=0x7f8064004ab0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:47:46.002 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:46.000+0000 7f8068c00640 1 -- 172.21.15.32:0/478202081 <== mon.1 v2:172.21.15.118:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7f8058068920 con 0x7f806c0cffe0 2024-08-03T20:47:46.268 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:46.266+0000 7fbea9600640 1 -- 172.21.15.32:0/3538299134 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) -- 0x7fbeb415be00 con 0x7fbeb40cee70 2024-08-03T20:47:46.269 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:46.267+0000 7fbeabe00640 1 -- 172.21.15.32:0/3538299134 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fbea003f030 con 0x7fbeb40cee70 2024-08-03T20:47:46.270 INFO:teuthology.orchestra.run.smithi032.stdout:64424509447 2024-08-03T20:47:46.274 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:46.271+0000 7fbea9600640 1 -- 172.21.15.32:0/3538299134 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fbe94077010 msgr2=0x7fbe94079520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:47:46.275 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:46.271+0000 7fbea9600640 1 --2- 172.21.15.32:0/3538299134 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fbe94077010 0x7fbe94079520 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fbeac007e90 tx=0x7fbeac007870 comp rx=0 tx=0).stop 2024-08-03T20:47:46.275 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:46.271+0000 7fbea9600640 1 -- 172.21.15.32:0/3538299134 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbeb40cee70 msgr2=0x7fbeb40ab700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:47:46.275 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:46.271+0000 7fbea9600640 1 --2- 172.21.15.32:0/3538299134 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbeb40cee70 0x7fbeb40ab700 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fbeb40cca70 tx=0x7fbea0005850 comp rx=0 tx=0).stop 2024-08-03T20:47:46.275 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:46.272+0000 7fbea9600640 1 -- 172.21.15.32:0/3538299134 shutdown_connections 2024-08-03T20:47:46.275 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:46.272+0000 7fbea9600640 1 --2- 172.21.15.32:0/3538299134 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fbe94077010 0x7fbe94079520 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:46.275 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:46.272+0000 7fbea9600640 1 --2- 172.21.15.32:0/3538299134 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fbeb415d660 0x7fbeb40ad490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:46.275 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:46.272+0000 7fbea9600640 1 --2- 172.21.15.32:0/3538299134 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbeb40cee70 0x7fbeb40ab700 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:46.275 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:46.272+0000 7fbea9600640 1 --2- 172.21.15.32:0/3538299134 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fbeb4073760 0x7fbeb40ab1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:46.275 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:46.272+0000 7fbea9600640 1 -- 172.21.15.32:0/3538299134 >> 172.21.15.32:0/3538299134 conn(0x7fbeb40cff30 msgr2=0x7fbeb40afb90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:47:46.276 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:46.273+0000 7fbea9600640 1 -- 172.21.15.32:0/3538299134 shutdown_connections 2024-08-03T20:47:46.276 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:46.273+0000 7fbea9600640 1 -- 172.21.15.32:0/3538299134 wait complete. 2024-08-03T20:47:46.314 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509447 got 64424509447 for osd.0 2024-08-03T20:47:46.314 DEBUG:teuthology.parallel:result is None 2024-08-03T20:47:46.333 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:46.332+0000 7f71b3e00640 1 -- 172.21.15.32:0/3771676324 --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) -- 0x7f71b415a9f0 con 0x7f71b40ca5c0 2024-08-03T20:47:46.334 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:46.332+0000 7f71aaa00640 1 -- 172.21.15.32:0/3771676324 <== mon.2 v2:172.21.15.118:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f71a40611e0 con 0x7f71b40ca5c0 2024-08-03T20:47:46.334 INFO:teuthology.orchestra.run.smithi032.stdout:60129542149 2024-08-03T20:47:46.338 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:46.337+0000 7f71b3e00640 1 -- 172.21.15.32:0/3771676324 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f71900770e0 msgr2=0x7f71900795f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:47:46.338 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:46.337+0000 7f71b3e00640 1 --2- 172.21.15.32:0/3771676324 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f71900770e0 0x7f71900795f0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f71b415efc0 tx=0x7f719c009040 comp rx=0 tx=0).stop 2024-08-03T20:47:46.338 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:46.337+0000 7f71b3e00640 1 -- 172.21.15.32:0/3771676324 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f71b40ca5c0 msgr2=0x7f71b4164e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:47:46.338 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:46.337+0000 7f71b3e00640 1 --2- 172.21.15.32:0/3771676324 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f71b40ca5c0 0x7f71b4164e20 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f71a400c430 tx=0x7f71a400c8f0 comp rx=0 tx=0).stop 2024-08-03T20:47:46.339 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:46.337+0000 7f71b3e00640 1 -- 172.21.15.32:0/3771676324 shutdown_connections 2024-08-03T20:47:46.339 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:46.337+0000 7f71b3e00640 1 --2- 172.21.15.32:0/3771676324 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f71900770e0 0x7f71900795f0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:46.339 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:46.337+0000 7f71b3e00640 1 --2- 172.21.15.32:0/3771676324 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f71b4163160 0x7f71b4073540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:46.339 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:46.337+0000 7f71b3e00640 1 --2- 172.21.15.32:0/3771676324 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f71b4157fc0 0x7f71b40b4c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:46.339 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:46.337+0000 7f71b3e00640 1 --2- 172.21.15.32:0/3771676324 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f71b40ca5c0 0x7f71b4164e20 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:46.339 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:46.337+0000 7f71b3e00640 1 -- 172.21.15.32:0/3771676324 >> 172.21.15.32:0/3771676324 conn(0x7f71b40b1d50 msgr2=0x7f71b414fb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:47:46.340 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:46.337+0000 7f71b3e00640 1 -- 172.21.15.32:0/3771676324 shutdown_connections 2024-08-03T20:47:46.340 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:46.338+0000 7f71b3e00640 1 -- 172.21.15.32:0/3771676324 wait complete. 2024-08-03T20:47:46.359 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:46.358+0000 7f04cfe00640 1 -- 172.21.15.32:0/317629542 --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) -- 0x7f04c801de10 con 0x7f04c800a970 2024-08-03T20:47:46.360 INFO:teuthology.orchestra.run.smithi032.stdout:64424509445 2024-08-03T20:47:46.360 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:46.358+0000 7f04c6a00640 1 -- 172.21.15.32:0/317629542 <== mon.2 v2:172.21.15.118:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f04c0061e00 con 0x7f04c800a970 2024-08-03T20:47:46.363 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:46.362+0000 7f04cfe00640 1 -- 172.21.15.32:0/317629542 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f04b0077010 msgr2=0x7f04b0079520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:47:46.363 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:46.362+0000 7f04cfe00640 1 --2- 172.21.15.32:0/317629542 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f04b0077010 0x7f04b0079520 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f04d00520f0 tx=0x7f04d0075430 comp rx=0 tx=0).stop 2024-08-03T20:47:46.363 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:46.362+0000 7f04cfe00640 1 -- 172.21.15.32:0/317629542 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f04c800a970 msgr2=0x7f04c804c5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:47:46.363 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:46.362+0000 7f04cfe00640 1 --2- 172.21.15.32:0/317629542 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f04c800a970 0x7f04c804c5f0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f04c000c430 tx=0x7f04c000c8f0 comp rx=0 tx=0).stop 2024-08-03T20:47:46.364 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:46.362+0000 7f04cfe00640 1 -- 172.21.15.32:0/317629542 shutdown_connections 2024-08-03T20:47:46.364 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:46.362+0000 7f04cfe00640 1 --2- 172.21.15.32:0/317629542 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f04b0077010 0x7f04b0079520 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:46.364 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:46.362+0000 7f04cfe00640 1 --2- 172.21.15.32:0/317629542 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f04c8036c30 0x7f04c8052360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:46.364 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:46.362+0000 7f04cfe00640 1 --2- 172.21.15.32:0/317629542 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f04c800a970 0x7f04c804c5f0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:46.364 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:46.362+0000 7f04cfe00640 1 --2- 172.21.15.32:0/317629542 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f04c8009da0 0x7f04c80dc8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:46.364 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:46.362+0000 7f04cfe00640 1 -- 172.21.15.32:0/317629542 >> 172.21.15.32:0/317629542 conn(0x7f04c8051420 msgr2=0x7f04c80de690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:47:46.364 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:46.362+0000 7f04cfe00640 1 -- 172.21.15.32:0/317629542 shutdown_connections 2024-08-03T20:47:46.364 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:46.362+0000 7f04cfe00640 1 -- 172.21.15.32:0/317629542 wait complete. 2024-08-03T20:47:46.370 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:46.369+0000 7f3611c00640 1 -- 172.21.15.32:0/140235677 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) -- 0x7f35e4005580 con 0x7f360c0d4c40 2024-08-03T20:47:46.370 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:46.369+0000 7f3602a00640 1 -- 172.21.15.32:0/140235677 <== mon.1 v2:172.21.15.118:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f35fc060e30 con 0x7f360c0d4c40 2024-08-03T20:47:46.371 INFO:teuthology.orchestra.run.smithi032.stdout:64424509445 2024-08-03T20:47:46.373 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:46.372+0000 7f3611c00640 1 -- 172.21.15.32:0/140235677 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f35ec077010 msgr2=0x7f35ec079520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:47:46.373 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:46.372+0000 7f3611c00640 1 --2- 172.21.15.32:0/140235677 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f35ec077010 0x7f35ec079520 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f3604004500 tx=0x7f3604009040 comp rx=0 tx=0).stop 2024-08-03T20:47:46.374 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:46.372+0000 7f3611c00640 1 -- 172.21.15.32:0/140235677 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f360c0d4c40 msgr2=0x7f360c15cb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:47:46.374 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:46.372+0000 7f3611c00640 1 --2- 172.21.15.32:0/140235677 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f360c0d4c40 0x7f360c15cb70 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f35fc00c3a0 tx=0x7f35fc00c860 comp rx=0 tx=0).stop 2024-08-03T20:47:46.374 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:46.372+0000 7f3611c00640 1 -- 172.21.15.32:0/140235677 shutdown_connections 2024-08-03T20:47:46.374 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:46.372+0000 7f3611c00640 1 --2- 172.21.15.32:0/140235677 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f35ec077010 0x7f35ec079520 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:46.374 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:46.372+0000 7f3611c00640 1 --2- 172.21.15.32:0/140235677 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f360c0d4c40 0x7f360c15cb70 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:46.374 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:46.372+0000 7f3611c00640 1 --2- 172.21.15.32:0/140235677 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f360c161d90 0x7f360c0a0190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:46.374 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:46.372+0000 7f3611c00640 1 --2- 172.21.15.32:0/140235677 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f360c0b94c0 0x7f360c09e850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:46.374 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:46.372+0000 7f3611c00640 1 -- 172.21.15.32:0/140235677 >> 172.21.15.32:0/140235677 conn(0x7f360c0d4120 msgr2=0x7f360c0d2ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:47:46.375 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:46.372+0000 7f3611c00640 1 -- 172.21.15.32:0/140235677 shutdown_connections 2024-08-03T20:47:46.375 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:46.373+0000 7f3611c00640 1 -- 172.21.15.32:0/140235677 wait complete. 2024-08-03T20:47:46.375 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:46.374+0000 7facdec00640 1 -- 172.21.15.32:0/2406000982 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) -- 0x7facb4005580 con 0x7facd00e0e80 2024-08-03T20:47:46.376 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:46.374+0000 7facd5600640 1 -- 172.21.15.32:0/2406000982 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7facd80e0900 con 0x7facd00e0e80 2024-08-03T20:47:46.376 INFO:teuthology.orchestra.run.smithi032.stdout:60129542150 2024-08-03T20:47:46.379 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:46.378+0000 7f1e08c00640 1 -- 172.21.15.32:0/3140339105 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) -- 0x7f1e040d4750 con 0x7f1e040cffe0 2024-08-03T20:47:46.380 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:46.378+0000 7facdec00640 1 -- 172.21.15.32:0/2406000982 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7facb8077090 msgr2=0x7facb80795a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:47:46.380 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:46.379+0000 7facdec00640 1 --2- 172.21.15.32:0/2406000982 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7facb8077090 0x7facb80795a0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7faccc009660 tx=0x7faccc004730 comp rx=0 tx=0).stop 2024-08-03T20:47:46.380 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:46.379+0000 7facdec00640 1 -- 172.21.15.32:0/2406000982 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7facd00e0e80 msgr2=0x7facd00defc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:47:46.380 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:46.379+0000 7f1dfbe00640 1 -- 172.21.15.32:0/3140339105 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f1dec066110 con 0x7f1e040cffe0 2024-08-03T20:47:46.381 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:46.379+0000 7facdec00640 1 --2- 172.21.15.32:0/2406000982 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7facd00e0e80 0x7facd00defc0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7facd806b6f0 tx=0x7facd806ef40 comp rx=0 tx=0).stop 2024-08-03T20:47:46.381 INFO:teuthology.orchestra.run.smithi032.stdout:64424509446 2024-08-03T20:47:46.381 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:46.379+0000 7facdec00640 1 -- 172.21.15.32:0/2406000982 shutdown_connections 2024-08-03T20:47:46.382 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:46.379+0000 7facdec00640 1 --2- 172.21.15.32:0/2406000982 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7facb8077090 0x7facb80795a0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:46.382 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:46.379+0000 7facdec00640 1 --2- 172.21.15.32:0/2406000982 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7facd00e0e80 0x7facd00defc0 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:46.382 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:46.379+0000 7facdec00640 1 --2- 172.21.15.32:0/2406000982 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7facd00072e0 0x7facd00dd770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:46.382 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:46.379+0000 7facdec00640 1 --2- 172.21.15.32:0/2406000982 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7facd0006140 0x7facd00482c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:46.382 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:46.379+0000 7facdec00640 1 -- 172.21.15.32:0/2406000982 >> 172.21.15.32:0/2406000982 conn(0x7facd001d9a0 msgr2=0x7facd00e0340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:47:46.382 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:46.380+0000 7facdec00640 1 -- 172.21.15.32:0/2406000982 shutdown_connections 2024-08-03T20:47:46.382 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:46.380+0000 7facdec00640 1 -- 172.21.15.32:0/2406000982 wait complete. 2024-08-03T20:47:46.383 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:46.381+0000 7f1e08c00640 1 -- 172.21.15.32:0/3140339105 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f1de0077060 msgr2=0x7f1de0079570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:47:46.383 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:46.381+0000 7f1e08c00640 1 --2- 172.21.15.32:0/3140339105 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f1de0077060 0x7f1de0079570 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f1df400f670 tx=0x7f1df40075d0 comp rx=0 tx=0).stop 2024-08-03T20:47:46.383 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:46.381+0000 7f1e08c00640 1 -- 172.21.15.32:0/3140339105 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1e040cffe0 msgr2=0x7f1e041f4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:47:46.383 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:46.382+0000 7f1e08c00640 1 --2- 172.21.15.32:0/3140339105 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1e040cffe0 0x7f1e041f4e60 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f1dec00c3c0 tx=0x7f1dec00c880 comp rx=0 tx=0).stop 2024-08-03T20:47:46.383 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:46.382+0000 7f1e08c00640 1 -- 172.21.15.32:0/3140339105 shutdown_connections 2024-08-03T20:47:46.383 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:46.382+0000 7f1e08c00640 1 --2- 172.21.15.32:0/3140339105 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f1de0077060 0x7f1de0079570 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:46.384 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:46.382+0000 7f1e08c00640 1 --2- 172.21.15.32:0/3140339105 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f1e040d26e0 0x7f1e041f53a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:46.384 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:46.382+0000 7f1e08c00640 1 --2- 172.21.15.32:0/3140339105 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1e040cffe0 0x7f1e041f4e60 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:46.384 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:46.382+0000 7f1e08c00640 1 --2- 172.21.15.32:0/3140339105 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f1e040cf510 0x7f1e040cd720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:46.384 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:46.382+0000 7f1e08c00640 1 -- 172.21.15.32:0/3140339105 >> 172.21.15.32:0/3140339105 conn(0x7f1e0408d7a0 msgr2=0x7f1e04161b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:47:46.384 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:46.382+0000 7f1e08c00640 1 -- 172.21.15.32:0/3140339105 shutdown_connections 2024-08-03T20:47:46.384 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:46.382+0000 7f1e08c00640 1 -- 172.21.15.32:0/3140339105 wait complete. 2024-08-03T20:47:46.387 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542148 got 60129542149 for osd.3 2024-08-03T20:47:46.387 DEBUG:teuthology.parallel:result is None 2024-08-03T20:47:46.402 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:46.400+0000 7f8071e00640 1 -- 172.21.15.32:0/478202081 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) -- 0x7f8044005580 con 0x7f806c0cffe0 2024-08-03T20:47:46.402 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:46.400+0000 7f8068c00640 1 -- 172.21.15.32:0/478202081 <== mon.1 v2:172.21.15.118:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f805806ce70 con 0x7f806c0cffe0 2024-08-03T20:47:46.404 INFO:teuthology.orchestra.run.smithi032.stdout:60129542151 2024-08-03T20:47:46.404 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509445 got 64424509445 for osd.7 2024-08-03T20:47:46.404 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509445 got 64424509445 for osd.1 2024-08-03T20:47:46.405 DEBUG:teuthology.parallel:result is None 2024-08-03T20:47:46.405 DEBUG:teuthology.parallel:result is None 2024-08-03T20:47:46.405 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:46.403+0000 7f8071e00640 1 -- 172.21.15.32:0/478202081 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f804c077240 msgr2=0x7f804c079750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:47:46.405 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:46.403+0000 7f8071e00640 1 --2- 172.21.15.32:0/478202081 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f804c077240 0x7f804c079750 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f8064000f80 tx=0x7f8064004ab0 comp rx=0 tx=0).stop 2024-08-03T20:47:46.406 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:46.403+0000 7f8071e00640 1 -- 172.21.15.32:0/478202081 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f806c0cffe0 msgr2=0x7f806c155600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:47:46.406 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:46.403+0000 7f8071e00640 1 --2- 172.21.15.32:0/478202081 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f806c0cffe0 0x7f806c155600 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f8058001dd0 tx=0x7f805800c7b0 comp rx=0 tx=0).stop 2024-08-03T20:47:46.406 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:46.404+0000 7f673d000640 1 -- 172.21.15.32:0/271008676 --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) -- 0x7f67380c7cc0 con 0x7f67380d4d10 2024-08-03T20:47:46.407 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:46.404+0000 7f8071e00640 1 -- 172.21.15.32:0/478202081 shutdown_connections 2024-08-03T20:47:46.407 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:46.404+0000 7f8071e00640 1 --2- 172.21.15.32:0/478202081 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f804c077240 0x7f804c079750 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:46.407 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:46.404+0000 7f8071e00640 1 --2- 172.21.15.32:0/478202081 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f806c0cffe0 0x7f806c155600 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:46.407 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:46.404+0000 7f8071e00640 1 --2- 172.21.15.32:0/478202081 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f806c1f5190 0x7f806c0b4830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:46.407 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:46.404+0000 7f8071e00640 1 --2- 172.21.15.32:0/478202081 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f806c0cf510 0x7f806c0ca9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:46.407 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:46.404+0000 7f8071e00640 1 -- 172.21.15.32:0/478202081 >> 172.21.15.32:0/478202081 conn(0x7f806c08d7a0 msgr2=0x7f806c0d1300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:47:46.407 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:46.404+0000 7f8071e00640 1 -- 172.21.15.32:0/478202081 shutdown_connections 2024-08-03T20:47:46.407 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:46.404+0000 7f8071e00640 1 -- 172.21.15.32:0/478202081 wait complete. 2024-08-03T20:47:46.408 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:46.405+0000 7f672e000640 1 -- 172.21.15.32:0/271008676 <== mon.2 v2:172.21.15.118:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f6730010040 con 0x7f67380d4d10 2024-08-03T20:47:46.408 INFO:teuthology.orchestra.run.smithi032.stdout:64424509446 2024-08-03T20:47:46.410 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:46.407+0000 7f673d000640 1 -- 172.21.15.32:0/271008676 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f67180770e0 msgr2=0x7f67180795f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:47:46.410 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:46.407+0000 7f673d000640 1 --2- 172.21.15.32:0/271008676 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f67180770e0 0x7f67180795f0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f6728034000 tx=0x7f6728004730 comp rx=0 tx=0).stop 2024-08-03T20:47:46.410 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:46.407+0000 7f673d000640 1 -- 172.21.15.32:0/271008676 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f67380d4d10 msgr2=0x7f6738072600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:47:46.410 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:46.407+0000 7f673d000640 1 --2- 172.21.15.32:0/271008676 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f67380d4d10 0x7f6738072600 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f6730009f00 tx=0x7f673000c490 comp rx=0 tx=0).stop 2024-08-03T20:47:46.410 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:46.408+0000 7f673d000640 1 -- 172.21.15.32:0/271008676 shutdown_connections 2024-08-03T20:47:46.410 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:46.408+0000 7f673d000640 1 --2- 172.21.15.32:0/271008676 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f67180770e0 0x7f67180795f0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:46.410 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:46.408+0000 7f673d000640 1 --2- 172.21.15.32:0/271008676 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f67380d26b0 0x7f6738160fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:46.411 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:46.408+0000 7f673d000640 1 --2- 172.21.15.32:0/271008676 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f67380d4d10 0x7f6738072600 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:46.411 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:46.408+0000 7f673d000640 1 --2- 172.21.15.32:0/271008676 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f67380b94d0 0x7f6738070ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:46.411 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:46.408+0000 7f673d000640 1 -- 172.21.15.32:0/271008676 >> 172.21.15.32:0/271008676 conn(0x7f673809d670 msgr2=0x7f67380d1300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:47:46.411 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:46.408+0000 7f673d000640 1 -- 172.21.15.32:0/271008676 shutdown_connections 2024-08-03T20:47:46.411 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:46.408+0000 7f673d000640 1 -- 172.21.15.32:0/271008676 wait complete. 2024-08-03T20:47:46.412 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542149 got 60129542150 for osd.2 2024-08-03T20:47:46.412 DEBUG:teuthology.parallel:result is None 2024-08-03T20:47:46.417 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509446 got 64424509446 for osd.5 2024-08-03T20:47:46.417 DEBUG:teuthology.parallel:result is None 2024-08-03T20:47:46.441 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542150 got 60129542151 for osd.4 2024-08-03T20:47:46.441 DEBUG:teuthology.parallel:result is None 2024-08-03T20:47:46.446 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509446 got 64424509446 for osd.6 2024-08-03T20:47:46.446 DEBUG:teuthology.parallel:result is None 2024-08-03T20:47:46.446 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-08-03T20:47:46.446 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-08-03T20:47:46.713 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:46.711+0000 7f88bdc00640 1 -- 172.21.15.32:0/3702022157 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f88b8165160 msgr2=0x7f88b8167580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:47:46.713 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:46.711+0000 7f88bdc00640 1 --2- 172.21.15.32:0/3702022157 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f88b8165160 0x7f88b8167580 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f88b0005fd0 tx=0x7f88b002a8b0 comp rx=0 tx=0).stop 2024-08-03T20:47:46.713 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:46.711+0000 7f88bdc00640 1 -- 172.21.15.32:0/3702022157 shutdown_connections 2024-08-03T20:47:46.713 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:46.711+0000 7f88bdc00640 1 --2- 172.21.15.32:0/3702022157 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f88b8165160 0x7f88b8167580 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:46.713 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:46.711+0000 7f88bdc00640 1 -- 172.21.15.32:0/3702022157 >> 172.21.15.32:0/3702022157 conn(0x7f88b80b1bd0 msgr2=0x7f88b80b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:47:46.714 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:46.712+0000 7f88bdc00640 1 -- 172.21.15.32:0/3702022157 shutdown_connections 2024-08-03T20:47:46.714 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:46.712+0000 7f88bdc00640 1 -- 172.21.15.32:0/3702022157 wait complete. 2024-08-03T20:47:46.715 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:46.714+0000 7f88bdc00640 1 Processor -- start 2024-08-03T20:47:46.715 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:46.714+0000 7f88bdc00640 1 -- start start 2024-08-03T20:47:46.718 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:46.715+0000 7f88bdc00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f88b806e930 0x7f88b8143930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:47:46.718 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:46.715+0000 7f88bdc00640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f88b8144f90 0x7f88b81f8950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:47:46.718 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:46.715+0000 7f88bdc00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f88b8165160 0x7f88b81f8e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:47:46.718 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:46.715+0000 7f88bdc00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f88b80cf9c0 con 0x7f88b806e930 2024-08-03T20:47:46.718 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:46.715+0000 7f88bdc00640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f88b80cf840 con 0x7f88b8165160 2024-08-03T20:47:46.719 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:46.715+0000 7f88bdc00640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7f88b8167ac0 con 0x7f88b8144f90 2024-08-03T20:47:46.719 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:46.716+0000 7f88b6a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f88b806e930 0x7f88b8143930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:47:46.719 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:46.716+0000 7f88b6a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f88b806e930 0x7f88b8143930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:57666/0 (socket says 172.21.15.32:57666) 2024-08-03T20:47:46.719 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:46.716+0000 7f88b6a00640 1 -- 172.21.15.32:0/958466881 learned_addr learned my addr 172.21.15.32:0/958466881 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:47:46.719 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:46.716+0000 7f88b6000640 1 --2- 172.21.15.32:0/958466881 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f88b8144f90 0x7f88b81f8950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:47:46.719 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:46.716+0000 7f88b6a00640 1 -- 172.21.15.32:0/958466881 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f88b8144f90 msgr2=0x7f88b81f8950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:47:46.719 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:46.716+0000 7f88b6a00640 1 --2- 172.21.15.32:0/958466881 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f88b8144f90 0x7f88b81f8950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:46.719 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:46.716+0000 7f88b7400640 1 --2- 172.21.15.32:0/958466881 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f88b8165160 0x7f88b81f8e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:47:46.719 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:46.716+0000 7f88b6a00640 1 -- 172.21.15.32:0/958466881 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f88b8165160 msgr2=0x7f88b81f8e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:47:46.719 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:46.716+0000 7f88b6a00640 1 --2- 172.21.15.32:0/958466881 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f88b8165160 0x7f88b81f8e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:46.719 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:46.716+0000 7f88b6a00640 1 -- 172.21.15.32:0/958466881 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f88b8143e70 con 0x7f88b806e930 2024-08-03T20:47:46.719 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:46.716+0000 7f88b6000640 1 --2- 172.21.15.32:0/958466881 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f88b8144f90 0x7f88b81f8950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-03T20:47:46.719 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:46.716+0000 7f88b7400640 1 --2- 172.21.15.32:0/958466881 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f88b8165160 0x7f88b81f8e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-03T20:47:46.720 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:46.716+0000 7f88b6a00640 1 --2- 172.21.15.32:0/958466881 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f88b806e930 0x7f88b8143930 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f88a800c400 tx=0x7f88a800c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:47:46.720 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:46.717+0000 7f88aea00640 1 -- 172.21.15.32:0/958466881 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88a8017070 con 0x7f88b806e930 2024-08-03T20:47:46.720 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:46.717+0000 7f88aea00640 1 -- 172.21.15.32:0/958466881 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f88a80053c0 con 0x7f88b806e930 2024-08-03T20:47:46.720 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:46.717+0000 7f88bdc00640 1 -- 172.21.15.32:0/958466881 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f88b8144380 con 0x7f88b806e930 2024-08-03T20:47:46.720 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:46.717+0000 7f88aea00640 1 -- 172.21.15.32:0/958466881 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88a8003850 con 0x7f88b806e930 2024-08-03T20:47:46.720 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:46.717+0000 7f88bdc00640 1 -- 172.21.15.32:0/958466881 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f88b80ca4f0 con 0x7f88b806e930 2024-08-03T20:47:46.720 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:46.718+0000 7f88bdc00640 1 -- 172.21.15.32:0/958466881 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f88b8073e60 con 0x7f88b806e930 2024-08-03T20:47:46.720 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:46.719+0000 7f88aea00640 1 -- 172.21.15.32:0/958466881 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f88a801a070 con 0x7f88b806e930 2024-08-03T20:47:46.721 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:46.720+0000 7f88aea00640 1 --2- 172.21.15.32:0/958466881 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f8898077010 0x7f8898079520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:47:46.721 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:46.720+0000 7f88aea00640 1 -- 172.21.15.32:0/958466881 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f88a8097e00 con 0x7f88b806e930 2024-08-03T20:47:46.722 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:46.720+0000 7f88b6000640 1 --2- 172.21.15.32:0/958466881 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f8898077010 0x7f8898079520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:47:46.722 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:46.721+0000 7f88b6000640 1 --2- 172.21.15.32:0/958466881 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f8898077010 0x7f8898079520 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f88b8139500 tx=0x7f88a4009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:47:46.726 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:46.724+0000 7f88aea00640 1 -- 172.21.15.32:0/958466881 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7f88a8060f60 con 0x7f88b806e930 2024-08-03T20:47:46.925 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:46.924+0000 7f88bdc00640 1 -- 172.21.15.32:0/958466881 --> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f88b815c430 con 0x7f8898077010 2024-08-03T20:47:46.930 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:46.929+0000 7f88aea00640 1 -- 172.21.15.32:0/958466881 <== mgr.4106 v2:172.21.15.118:6800/4205950699 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+251826 (secure 0 0 0) 0x7f88b815c430 con 0x7f8898077010 2024-08-03T20:47:46.934 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-03T20:47:46.935 INFO:teuthology.orchestra.run.smithi032.stderr:dumped all 2024-08-03T20:47:46.937 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:46.933+0000 7f88bdc00640 1 -- 172.21.15.32:0/958466881 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f8898077010 msgr2=0x7f8898079520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:47:46.937 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:46.933+0000 7f88bdc00640 1 --2- 172.21.15.32:0/958466881 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f8898077010 0x7f8898079520 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f88b8139500 tx=0x7f88a4009040 comp rx=0 tx=0).stop 2024-08-03T20:47:46.937 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:46.933+0000 7f88bdc00640 1 -- 172.21.15.32:0/958466881 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f88b806e930 msgr2=0x7f88b8143930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:47:46.938 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:46.933+0000 7f88bdc00640 1 --2- 172.21.15.32:0/958466881 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f88b806e930 0x7f88b8143930 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f88a800c400 tx=0x7f88a800c8c0 comp rx=0 tx=0).stop 2024-08-03T20:47:46.938 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:46.934+0000 7f88bdc00640 1 -- 172.21.15.32:0/958466881 shutdown_connections 2024-08-03T20:47:46.938 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:46.934+0000 7f88bdc00640 1 --2- 172.21.15.32:0/958466881 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f8898077010 0x7f8898079520 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:46.938 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:46.934+0000 7f88bdc00640 1 --2- 172.21.15.32:0/958466881 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f88b8165160 0x7f88b81f8e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:46.938 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:46.934+0000 7f88bdc00640 1 --2- 172.21.15.32:0/958466881 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f88b8144f90 0x7f88b81f8950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:46.938 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:46.934+0000 7f88bdc00640 1 --2- 172.21.15.32:0/958466881 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f88b806e930 0x7f88b8143930 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:46.938 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:46.934+0000 7f88bdc00640 1 -- 172.21.15.32:0/958466881 >> 172.21.15.32:0/958466881 conn(0x7f88b80b1bd0 msgr2=0x7f88b8165de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:47:46.938 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:46.934+0000 7f88bdc00640 1 -- 172.21.15.32:0/958466881 shutdown_connections 2024-08-03T20:47:46.938 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:46.934+0000 7f88bdc00640 1 -- 172.21.15.32:0/958466881 wait complete. 2024-08-03T20:47:46.961 INFO:teuthology.orchestra.run.smithi032.stdout:{"pg_ready":true,"pg_map":{"version":23,"stamp":"2024-08-03T20:47:46.223784+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":146,"num_write_kb":1642,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":111,"ondisk_log_size":111,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":252316,"kb_used_data":2856,"kb_used_omap":12,"kb_used_meta":249331,"kb_avail":754722404,"statfs":{"total":773094113280,"available":772835741696,"internally_reserved":0,"allocated":2924544,"data_stored":1765884,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":255315505},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.002815"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-03T20:47:37.317266+0000","last_change":"2024-08-03T20:47:35.962230+0000","last_active":"2024-08-03T20:47:37.317266+0000","last_peered":"2024-08-03T20:47:37.317266+0000","last_clean":"2024-08-03T20:47:37.317266+0000","last_became_active":"2024-08-03T20:47:35.958864+0000","last_became_peered":"2024-08-03T20:47:35.958864+0000","last_unstale":"2024-08-03T20:47:37.317266+0000","last_undegraded":"2024-08-03T20:47:37.317266+0000","last_fullsized":"2024-08-03T20:47:37.317266+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_clean_scrub_stamp":"2024-08-03T20:47:34.904132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T03:01:41.602990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-03T20:47:36.935685+0000","last_change":"2024-08-03T20:47:35.941982+0000","last_active":"2024-08-03T20:47:36.935685+0000","last_peered":"2024-08-03T20:47:36.935685+0000","last_clean":"2024-08-03T20:47:36.935685+0000","last_became_active":"2024-08-03T20:47:35.941614+0000","last_became_peered":"2024-08-03T20:47:35.941614+0000","last_unstale":"2024-08-03T20:47:36.935685+0000","last_undegraded":"2024-08-03T20:47:36.935685+0000","last_fullsized":"2024-08-03T20:47:36.935685+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_clean_scrub_stamp":"2024-08-03T20:47:34.904132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T01:08:56.085650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-03T20:47:37.979574+0000","last_change":"2024-08-03T20:47:35.926803+0000","last_active":"2024-08-03T20:47:37.979574+0000","last_peered":"2024-08-03T20:47:37.979574+0000","last_clean":"2024-08-03T20:47:37.979574+0000","last_became_active":"2024-08-03T20:47:35.926198+0000","last_became_peered":"2024-08-03T20:47:35.926198+0000","last_unstale":"2024-08-03T20:47:37.979574+0000","last_undegraded":"2024-08-03T20:47:37.979574+0000","last_fullsized":"2024-08-03T20:47:37.979574+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_clean_scrub_stamp":"2024-08-03T20:47:34.904132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T04:57:58.179012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-03T20:47:37.979134+0000","last_change":"2024-08-03T20:47:35.960607+0000","last_active":"2024-08-03T20:47:37.979134+0000","last_peered":"2024-08-03T20:47:37.979134+0000","last_clean":"2024-08-03T20:47:37.979134+0000","last_became_active":"2024-08-03T20:47:35.956844+0000","last_became_peered":"2024-08-03T20:47:35.956844+0000","last_unstale":"2024-08-03T20:47:37.979134+0000","last_undegraded":"2024-08-03T20:47:37.979134+0000","last_fullsized":"2024-08-03T20:47:37.979134+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_clean_scrub_stamp":"2024-08-03T20:47:34.904132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T07:48:20.679795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-03T20:47:37.316984+0000","last_change":"2024-08-03T20:47:35.962693+0000","last_active":"2024-08-03T20:47:37.316984+0000","last_peered":"2024-08-03T20:47:37.316984+0000","last_clean":"2024-08-03T20:47:37.316984+0000","last_became_active":"2024-08-03T20:47:35.959911+0000","last_became_peered":"2024-08-03T20:47:35.959911+0000","last_unstale":"2024-08-03T20:47:37.316984+0000","last_undegraded":"2024-08-03T20:47:37.316984+0000","last_fullsized":"2024-08-03T20:47:37.316984+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_clean_scrub_stamp":"2024-08-03T20:47:34.904132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T04:51:27.371858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-03T20:47:37.979016+0000","last_change":"2024-08-03T20:47:35.947517+0000","last_active":"2024-08-03T20:47:37.979016+0000","last_peered":"2024-08-03T20:47:37.979016+0000","last_clean":"2024-08-03T20:47:37.979016+0000","last_became_active":"2024-08-03T20:47:35.947167+0000","last_became_peered":"2024-08-03T20:47:35.947167+0000","last_unstale":"2024-08-03T20:47:37.979016+0000","last_undegraded":"2024-08-03T20:47:37.979016+0000","last_fullsized":"2024-08-03T20:47:37.979016+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_clean_scrub_stamp":"2024-08-03T20:47:34.904132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-04T22:33:13.028920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-03T20:47:37.316673+0000","last_change":"2024-08-03T20:47:35.947270+0000","last_active":"2024-08-03T20:47:37.316673+0000","last_peered":"2024-08-03T20:47:37.316673+0000","last_clean":"2024-08-03T20:47:37.316673+0000","last_became_active":"2024-08-03T20:47:35.946647+0000","last_became_peered":"2024-08-03T20:47:35.946647+0000","last_unstale":"2024-08-03T20:47:37.316673+0000","last_undegraded":"2024-08-03T20:47:37.316673+0000","last_fullsized":"2024-08-03T20:47:37.316673+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_clean_scrub_stamp":"2024-08-03T20:47:34.904132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T00:10:43.343156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-03T20:47:37.978113+0000","last_change":"2024-08-03T20:47:35.946701+0000","last_active":"2024-08-03T20:47:37.978113+0000","last_peered":"2024-08-03T20:47:37.978113+0000","last_clean":"2024-08-03T20:47:37.978113+0000","last_became_active":"2024-08-03T20:47:35.946340+0000","last_became_peered":"2024-08-03T20:47:35.946340+0000","last_unstale":"2024-08-03T20:47:37.978113+0000","last_undegraded":"2024-08-03T20:47:37.978113+0000","last_fullsized":"2024-08-03T20:47:37.978113+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_clean_scrub_stamp":"2024-08-03T20:47:34.904132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T07:21:44.279582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-03T20:47:37.317505+0000","last_change":"2024-08-03T20:47:35.963774+0000","last_active":"2024-08-03T20:47:37.317505+0000","last_peered":"2024-08-03T20:47:37.317505+0000","last_clean":"2024-08-03T20:47:37.317505+0000","last_became_active":"2024-08-03T20:47:35.958408+0000","last_became_peered":"2024-08-03T20:47:35.958408+0000","last_unstale":"2024-08-03T20:47:37.317505+0000","last_undegraded":"2024-08-03T20:47:37.317505+0000","last_fullsized":"2024-08-03T20:47:37.317505+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_clean_scrub_stamp":"2024-08-03T20:47:34.904132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T00:58:45.652167+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-03T20:47:36.941791+0000","last_change":"2024-08-03T20:47:35.943147+0000","last_active":"2024-08-03T20:47:36.941791+0000","last_peered":"2024-08-03T20:47:36.941791+0000","last_clean":"2024-08-03T20:47:36.941791+0000","last_became_active":"2024-08-03T20:47:35.942253+0000","last_became_peered":"2024-08-03T20:47:35.942253+0000","last_unstale":"2024-08-03T20:47:36.941791+0000","last_undegraded":"2024-08-03T20:47:36.941791+0000","last_fullsized":"2024-08-03T20:47:36.941791+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_clean_scrub_stamp":"2024-08-03T20:47:34.904132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T01:08:56.085650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-03T20:47:36.941585+0000","last_change":"2024-08-03T20:47:35.938819+0000","last_active":"2024-08-03T20:47:36.941585+0000","last_peered":"2024-08-03T20:47:36.941585+0000","last_clean":"2024-08-03T20:47:36.941585+0000","last_became_active":"2024-08-03T20:47:35.938499+0000","last_became_peered":"2024-08-03T20:47:35.938499+0000","last_unstale":"2024-08-03T20:47:36.941585+0000","last_undegraded":"2024-08-03T20:47:36.941585+0000","last_fullsized":"2024-08-03T20:47:36.941585+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_clean_scrub_stamp":"2024-08-03T20:47:34.904132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T04:57:58.179012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-03T20:47:36.936210+0000","last_change":"2024-08-03T20:47:35.936873+0000","last_active":"2024-08-03T20:47:36.936210+0000","last_peered":"2024-08-03T20:47:36.936210+0000","last_clean":"2024-08-03T20:47:36.936210+0000","last_became_active":"2024-08-03T20:47:35.936005+0000","last_became_peered":"2024-08-03T20:47:35.936005+0000","last_unstale":"2024-08-03T20:47:36.936210+0000","last_undegraded":"2024-08-03T20:47:36.936210+0000","last_fullsized":"2024-08-03T20:47:36.936210+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_clean_scrub_stamp":"2024-08-03T20:47:34.904132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T01:42:26.761950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-03T20:47:37.978994+0000","last_change":"2024-08-03T20:47:35.950146+0000","last_active":"2024-08-03T20:47:37.978994+0000","last_peered":"2024-08-03T20:47:37.978994+0000","last_clean":"2024-08-03T20:47:37.978994+0000","last_became_active":"2024-08-03T20:47:35.949842+0000","last_became_peered":"2024-08-03T20:47:35.949842+0000","last_unstale":"2024-08-03T20:47:37.978994+0000","last_undegraded":"2024-08-03T20:47:37.978994+0000","last_fullsized":"2024-08-03T20:47:37.978994+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_clean_scrub_stamp":"2024-08-03T20:47:34.904132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-04T22:48:57.612773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-03T20:47:37.979037+0000","last_change":"2024-08-03T20:47:35.940356+0000","last_active":"2024-08-03T20:47:37.979037+0000","last_peered":"2024-08-03T20:47:37.979037+0000","last_clean":"2024-08-03T20:47:37.979037+0000","last_became_active":"2024-08-03T20:47:35.940143+0000","last_became_peered":"2024-08-03T20:47:35.940143+0000","last_unstale":"2024-08-03T20:47:37.979037+0000","last_undegraded":"2024-08-03T20:47:37.979037+0000","last_fullsized":"2024-08-03T20:47:37.979037+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_clean_scrub_stamp":"2024-08-03T20:47:34.904132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T04:01:08.373245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-03T20:47:36.935545+0000","last_change":"2024-08-03T20:47:35.951009+0000","last_active":"2024-08-03T20:47:36.935545+0000","last_peered":"2024-08-03T20:47:36.935545+0000","last_clean":"2024-08-03T20:47:36.935545+0000","last_became_active":"2024-08-03T20:47:35.950658+0000","last_became_peered":"2024-08-03T20:47:35.950658+0000","last_unstale":"2024-08-03T20:47:36.935545+0000","last_undegraded":"2024-08-03T20:47:36.935545+0000","last_fullsized":"2024-08-03T20:47:36.935545+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_clean_scrub_stamp":"2024-08-03T20:47:34.904132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-04T20:49:19.998935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-03T20:47:37.978861+0000","last_change":"2024-08-03T20:47:35.948379+0000","last_active":"2024-08-03T20:47:37.978861+0000","last_peered":"2024-08-03T20:47:37.978861+0000","last_clean":"2024-08-03T20:47:37.978861+0000","last_became_active":"2024-08-03T20:47:35.947812+0000","last_became_peered":"2024-08-03T20:47:35.947812+0000","last_unstale":"2024-08-03T20:47:37.978861+0000","last_undegraded":"2024-08-03T20:47:37.978861+0000","last_fullsized":"2024-08-03T20:47:37.978861+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_clean_scrub_stamp":"2024-08-03T20:47:34.904132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T01:08:56.085650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-03T20:47:37.979587+0000","last_change":"2024-08-03T20:47:35.939474+0000","last_active":"2024-08-03T20:47:37.979587+0000","last_peered":"2024-08-03T20:47:37.979587+0000","last_clean":"2024-08-03T20:47:37.979587+0000","last_became_active":"2024-08-03T20:47:35.938914+0000","last_became_peered":"2024-08-03T20:47:35.938914+0000","last_unstale":"2024-08-03T20:47:37.979587+0000","last_undegraded":"2024-08-03T20:47:37.979587+0000","last_fullsized":"2024-08-03T20:47:37.979587+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_clean_scrub_stamp":"2024-08-03T20:47:34.904132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T01:52:59.081172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-03T20:47:37.352899+0000","last_change":"2024-08-03T20:47:35.933829+0000","last_active":"2024-08-03T20:47:37.352899+0000","last_peered":"2024-08-03T20:47:37.352899+0000","last_clean":"2024-08-03T20:47:37.352899+0000","last_became_active":"2024-08-03T20:47:35.933515+0000","last_became_peered":"2024-08-03T20:47:35.933515+0000","last_unstale":"2024-08-03T20:47:37.352899+0000","last_undegraded":"2024-08-03T20:47:37.352899+0000","last_fullsized":"2024-08-03T20:47:37.352899+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_clean_scrub_stamp":"2024-08-03T20:47:34.904132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T02:10:57.293691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-03T20:47:37.317301+0000","last_change":"2024-08-03T20:47:35.952999+0000","last_active":"2024-08-03T20:47:37.317301+0000","last_peered":"2024-08-03T20:47:37.317301+0000","last_clean":"2024-08-03T20:47:37.317301+0000","last_became_active":"2024-08-03T20:47:35.952281+0000","last_became_peered":"2024-08-03T20:47:35.952281+0000","last_unstale":"2024-08-03T20:47:37.317301+0000","last_undegraded":"2024-08-03T20:47:37.317301+0000","last_fullsized":"2024-08-03T20:47:37.317301+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_clean_scrub_stamp":"2024-08-03T20:47:34.904132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T01:15:48.374663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-03T20:47:36.941659+0000","last_change":"2024-08-03T20:47:35.927412+0000","last_active":"2024-08-03T20:47:36.941659+0000","last_peered":"2024-08-03T20:47:36.941659+0000","last_clean":"2024-08-03T20:47:36.941659+0000","last_became_active":"2024-08-03T20:47:35.927069+0000","last_became_peered":"2024-08-03T20:47:35.927069+0000","last_unstale":"2024-08-03T20:47:36.941659+0000","last_undegraded":"2024-08-03T20:47:36.941659+0000","last_fullsized":"2024-08-03T20:47:36.941659+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_clean_scrub_stamp":"2024-08-03T20:47:34.904132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-04T20:49:19.998935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-03T20:47:37.979382+0000","last_change":"2024-08-03T20:47:35.936665+0000","last_active":"2024-08-03T20:47:37.979382+0000","last_peered":"2024-08-03T20:47:37.979382+0000","last_clean":"2024-08-03T20:47:37.979382+0000","last_became_active":"2024-08-03T20:47:35.936298+0000","last_became_peered":"2024-08-03T20:47:35.936298+0000","last_unstale":"2024-08-03T20:47:37.979382+0000","last_undegraded":"2024-08-03T20:47:37.979382+0000","last_fullsized":"2024-08-03T20:47:37.979382+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_clean_scrub_stamp":"2024-08-03T20:47:34.904132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-04T22:12:15.293757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-03T20:47:37.979017+0000","last_change":"2024-08-03T20:47:35.938468+0000","last_active":"2024-08-03T20:47:37.979017+0000","last_peered":"2024-08-03T20:47:37.979017+0000","last_clean":"2024-08-03T20:47:37.979017+0000","last_became_active":"2024-08-03T20:47:35.937757+0000","last_became_peered":"2024-08-03T20:47:35.937757+0000","last_unstale":"2024-08-03T20:47:37.979017+0000","last_undegraded":"2024-08-03T20:47:37.979017+0000","last_fullsized":"2024-08-03T20:47:37.979017+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_clean_scrub_stamp":"2024-08-03T20:47:34.904132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-04T23:07:18.318169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-03T20:47:37.978241+0000","last_change":"2024-08-03T20:47:35.953502+0000","last_active":"2024-08-03T20:47:37.978241+0000","last_peered":"2024-08-03T20:47:37.978241+0000","last_clean":"2024-08-03T20:47:37.978241+0000","last_became_active":"2024-08-03T20:47:35.953152+0000","last_became_peered":"2024-08-03T20:47:35.953152+0000","last_unstale":"2024-08-03T20:47:37.978241+0000","last_undegraded":"2024-08-03T20:47:37.978241+0000","last_fullsized":"2024-08-03T20:47:37.978241+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_clean_scrub_stamp":"2024-08-03T20:47:34.904132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T01:56:58.984458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-03T20:47:37.979235+0000","last_change":"2024-08-03T20:47:35.937927+0000","last_active":"2024-08-03T20:47:37.979235+0000","last_peered":"2024-08-03T20:47:37.979235+0000","last_clean":"2024-08-03T20:47:37.979235+0000","last_became_active":"2024-08-03T20:47:35.937579+0000","last_became_peered":"2024-08-03T20:47:35.937579+0000","last_unstale":"2024-08-03T20:47:37.979235+0000","last_undegraded":"2024-08-03T20:47:37.979235+0000","last_fullsized":"2024-08-03T20:47:37.979235+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_clean_scrub_stamp":"2024-08-03T20:47:34.904132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T03:44:29.605859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-03T20:47:37.979020+0000","last_change":"2024-08-03T20:47:35.953835+0000","last_active":"2024-08-03T20:47:37.979020+0000","last_peered":"2024-08-03T20:47:37.979020+0000","last_clean":"2024-08-03T20:47:37.979020+0000","last_became_active":"2024-08-03T20:47:35.953520+0000","last_became_peered":"2024-08-03T20:47:35.953520+0000","last_unstale":"2024-08-03T20:47:37.979020+0000","last_undegraded":"2024-08-03T20:47:37.979020+0000","last_fullsized":"2024-08-03T20:47:37.979020+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_clean_scrub_stamp":"2024-08-03T20:47:34.904132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T06:11:09.513212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-03T20:47:36.941856+0000","last_change":"2024-08-03T20:47:35.938228+0000","last_active":"2024-08-03T20:47:36.941856+0000","last_peered":"2024-08-03T20:47:36.941856+0000","last_clean":"2024-08-03T20:47:36.941856+0000","last_became_active":"2024-08-03T20:47:35.937893+0000","last_became_peered":"2024-08-03T20:47:35.937893+0000","last_unstale":"2024-08-03T20:47:36.941856+0000","last_undegraded":"2024-08-03T20:47:36.941856+0000","last_fullsized":"2024-08-03T20:47:36.941856+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_clean_scrub_stamp":"2024-08-03T20:47:34.904132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-04T22:33:13.028920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-03T20:47:36.935344+0000","last_change":"2024-08-03T20:47:35.937679+0000","last_active":"2024-08-03T20:47:36.935344+0000","last_peered":"2024-08-03T20:47:36.935344+0000","last_clean":"2024-08-03T20:47:36.935344+0000","last_became_active":"2024-08-03T20:47:35.937328+0000","last_became_peered":"2024-08-03T20:47:35.937328+0000","last_unstale":"2024-08-03T20:47:36.935344+0000","last_undegraded":"2024-08-03T20:47:36.935344+0000","last_fullsized":"2024-08-03T20:47:36.935344+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_clean_scrub_stamp":"2024-08-03T20:47:34.904132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T04:57:58.179012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-03T20:47:37.978822+0000","last_change":"2024-08-03T20:47:35.954328+0000","last_active":"2024-08-03T20:47:37.978822+0000","last_peered":"2024-08-03T20:47:37.978822+0000","last_clean":"2024-08-03T20:47:37.978822+0000","last_became_active":"2024-08-03T20:47:35.951285+0000","last_became_peered":"2024-08-03T20:47:35.951285+0000","last_unstale":"2024-08-03T20:47:37.978822+0000","last_undegraded":"2024-08-03T20:47:37.978822+0000","last_fullsized":"2024-08-03T20:47:37.978822+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_clean_scrub_stamp":"2024-08-03T20:47:34.904132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T01:01:23.625299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-03T20:47:37.979494+0000","last_change":"2024-08-03T20:47:35.940504+0000","last_active":"2024-08-03T20:47:37.979494+0000","last_peered":"2024-08-03T20:47:37.979494+0000","last_clean":"2024-08-03T20:47:37.979494+0000","last_became_active":"2024-08-03T20:47:35.938638+0000","last_became_peered":"2024-08-03T20:47:35.938638+0000","last_unstale":"2024-08-03T20:47:37.979494+0000","last_undegraded":"2024-08-03T20:47:37.979494+0000","last_fullsized":"2024-08-03T20:47:37.979494+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_clean_scrub_stamp":"2024-08-03T20:47:34.904132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-04T23:57:18.840519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-03T20:47:37.979441+0000","last_change":"2024-08-03T20:47:35.938131+0000","last_active":"2024-08-03T20:47:37.979441+0000","last_peered":"2024-08-03T20:47:37.979441+0000","last_clean":"2024-08-03T20:47:37.979441+0000","last_became_active":"2024-08-03T20:47:35.937270+0000","last_became_peered":"2024-08-03T20:47:35.937270+0000","last_unstale":"2024-08-03T20:47:37.979441+0000","last_undegraded":"2024-08-03T20:47:37.979441+0000","last_fullsized":"2024-08-03T20:47:37.979441+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_clean_scrub_stamp":"2024-08-03T20:47:34.904132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-04T21:57:09.430415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-03T20:47:37.352726+0000","last_change":"2024-08-03T20:47:35.937957+0000","last_active":"2024-08-03T20:47:37.352726+0000","last_peered":"2024-08-03T20:47:37.352726+0000","last_clean":"2024-08-03T20:47:37.352726+0000","last_became_active":"2024-08-03T20:47:35.937656+0000","last_became_peered":"2024-08-03T20:47:35.937656+0000","last_unstale":"2024-08-03T20:47:37.352726+0000","last_undegraded":"2024-08-03T20:47:37.352726+0000","last_fullsized":"2024-08-03T20:47:37.352726+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_clean_scrub_stamp":"2024-08-03T20:47:34.904132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T05:51:29.523373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-03T20:47:36.941128+0000","last_change":"2024-08-03T20:47:35.932459+0000","last_active":"2024-08-03T20:47:36.941128+0000","last_peered":"2024-08-03T20:47:36.941128+0000","last_clean":"2024-08-03T20:47:36.941128+0000","last_became_active":"2024-08-03T20:47:35.932103+0000","last_became_peered":"2024-08-03T20:47:35.932103+0000","last_unstale":"2024-08-03T20:47:36.941128+0000","last_undegraded":"2024-08-03T20:47:36.941128+0000","last_fullsized":"2024-08-03T20:47:36.941128+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_clean_scrub_stamp":"2024-08-03T20:47:34.904132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T08:34:04.057775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-03T20:47:36.935757+0000","last_change":"2024-08-03T20:47:35.940544+0000","last_active":"2024-08-03T20:47:36.935757+0000","last_peered":"2024-08-03T20:47:36.935757+0000","last_clean":"2024-08-03T20:47:36.935757+0000","last_became_active":"2024-08-03T20:47:35.940218+0000","last_became_peered":"2024-08-03T20:47:35.940218+0000","last_unstale":"2024-08-03T20:47:36.935757+0000","last_undegraded":"2024-08-03T20:47:36.935757+0000","last_fullsized":"2024-08-03T20:47:36.935757+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_clean_scrub_stamp":"2024-08-03T20:47:34.904132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T07:58:42.869702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"20'1","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-03T20:47:37.978814+0000","last_change":"2024-08-03T20:47:34.939881+0000","last_active":"2024-08-03T20:47:37.978814+0000","last_peered":"2024-08-03T20:47:37.978814+0000","last_clean":"2024-08-03T20:47:37.978814+0000","last_became_active":"2024-08-03T20:47:34.936765+0000","last_became_peered":"2024-08-03T20:47:34.936765+0000","last_unstale":"2024-08-03T20:47:37.978814+0000","last_undegraded":"2024-08-03T20:47:37.978814+0000","last_fullsized":"2024-08-03T20:47:37.978814+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:47:33.895731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:47:33.895731+0000","last_clean_scrub_stamp":"2024-08-03T20:47:33.895731+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T02:29:41.523344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-03T20:47:37.979599+0000","last_change":"2024-08-03T20:47:35.942088+0000","last_active":"2024-08-03T20:47:37.979599+0000","last_peered":"2024-08-03T20:47:37.979599+0000","last_clean":"2024-08-03T20:47:37.979599+0000","last_became_active":"2024-08-03T20:47:35.941800+0000","last_became_peered":"2024-08-03T20:47:35.941800+0000","last_unstale":"2024-08-03T20:47:37.979599+0000","last_undegraded":"2024-08-03T20:47:37.979599+0000","last_fullsized":"2024-08-03T20:47:37.979599+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_clean_scrub_stamp":"2024-08-03T20:47:34.904132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-04T20:49:19.998935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"20'1","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-03T20:47:36.941066+0000","last_change":"2024-08-03T20:47:34.938348+0000","last_active":"2024-08-03T20:47:36.941066+0000","last_peered":"2024-08-03T20:47:36.941066+0000","last_clean":"2024-08-03T20:47:36.941066+0000","last_became_active":"2024-08-03T20:47:34.936169+0000","last_became_peered":"2024-08-03T20:47:34.936169+0000","last_unstale":"2024-08-03T20:47:36.941066+0000","last_undegraded":"2024-08-03T20:47:36.941066+0000","last_fullsized":"2024-08-03T20:47:36.941066+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:47:33.895731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:47:33.895731+0000","last_clean_scrub_stamp":"2024-08-03T20:47:33.895731+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T06:44:37.944928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-03T20:47:37.978344+0000","last_change":"2024-08-03T20:47:34.935991+0000","last_active":"2024-08-03T20:47:37.978344+0000","last_peered":"2024-08-03T20:47:37.978344+0000","last_clean":"2024-08-03T20:47:37.978344+0000","last_became_active":"2024-08-03T20:47:34.930541+0000","last_became_peered":"2024-08-03T20:47:34.930541+0000","last_unstale":"2024-08-03T20:47:37.978344+0000","last_undegraded":"2024-08-03T20:47:37.978344+0000","last_fullsized":"2024-08-03T20:47:37.978344+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:47:33.895731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:47:33.895731+0000","last_clean_scrub_stamp":"2024-08-03T20:47:33.895731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T04:37:55.336635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-03T20:47:37.978164+0000","last_change":"2024-08-03T20:47:35.934722+0000","last_active":"2024-08-03T20:47:37.978164+0000","last_peered":"2024-08-03T20:47:37.978164+0000","last_clean":"2024-08-03T20:47:37.978164+0000","last_became_active":"2024-08-03T20:47:35.934183+0000","last_became_peered":"2024-08-03T20:47:35.934183+0000","last_unstale":"2024-08-03T20:47:37.978164+0000","last_undegraded":"2024-08-03T20:47:37.978164+0000","last_fullsized":"2024-08-03T20:47:37.978164+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_clean_scrub_stamp":"2024-08-03T20:47:34.904132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T06:29:11.740575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-03T20:47:37.978945+0000","last_change":"2024-08-03T20:47:34.934591+0000","last_active":"2024-08-03T20:47:37.978945+0000","last_peered":"2024-08-03T20:47:37.978945+0000","last_clean":"2024-08-03T20:47:37.978945+0000","last_became_active":"2024-08-03T20:47:34.934155+0000","last_became_peered":"2024-08-03T20:47:34.934155+0000","last_unstale":"2024-08-03T20:47:37.978945+0000","last_undegraded":"2024-08-03T20:47:37.978945+0000","last_fullsized":"2024-08-03T20:47:37.978945+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:47:33.895731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:47:33.895731+0000","last_clean_scrub_stamp":"2024-08-03T20:47:33.895731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-04T22:30:28.119577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-03T20:47:36.934971+0000","last_change":"2024-08-03T20:47:35.947874+0000","last_active":"2024-08-03T20:47:36.934971+0000","last_peered":"2024-08-03T20:47:36.934971+0000","last_clean":"2024-08-03T20:47:36.934971+0000","last_became_active":"2024-08-03T20:47:35.947570+0000","last_became_peered":"2024-08-03T20:47:35.947570+0000","last_unstale":"2024-08-03T20:47:36.934971+0000","last_undegraded":"2024-08-03T20:47:36.934971+0000","last_fullsized":"2024-08-03T20:47:36.934971+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_clean_scrub_stamp":"2024-08-03T20:47:34.904132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-04T22:12:15.293757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-03T20:47:36.941786+0000","last_change":"2024-08-03T20:47:34.937955+0000","last_active":"2024-08-03T20:47:36.941786+0000","last_peered":"2024-08-03T20:47:36.941786+0000","last_clean":"2024-08-03T20:47:36.941786+0000","last_became_active":"2024-08-03T20:47:34.937613+0000","last_became_peered":"2024-08-03T20:47:34.937613+0000","last_unstale":"2024-08-03T20:47:36.941786+0000","last_undegraded":"2024-08-03T20:47:36.941786+0000","last_fullsized":"2024-08-03T20:47:36.941786+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:47:33.895731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:47:33.895731+0000","last_clean_scrub_stamp":"2024-08-03T20:47:33.895731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-04T22:30:28.119577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-03T20:47:37.979527+0000","last_change":"2024-08-03T20:47:35.941948+0000","last_active":"2024-08-03T20:47:37.979527+0000","last_peered":"2024-08-03T20:47:37.979527+0000","last_clean":"2024-08-03T20:47:37.979527+0000","last_became_active":"2024-08-03T20:47:35.941651+0000","last_became_peered":"2024-08-03T20:47:35.941651+0000","last_unstale":"2024-08-03T20:47:37.979527+0000","last_undegraded":"2024-08-03T20:47:37.979527+0000","last_fullsized":"2024-08-03T20:47:37.979527+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_clean_scrub_stamp":"2024-08-03T20:47:34.904132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T06:29:37.397869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-03T20:47:37.979267+0000","last_change":"2024-08-03T20:47:34.936965+0000","last_active":"2024-08-03T20:47:37.979267+0000","last_peered":"2024-08-03T20:47:37.979267+0000","last_clean":"2024-08-03T20:47:37.979267+0000","last_became_active":"2024-08-03T20:47:34.936667+0000","last_became_peered":"2024-08-03T20:47:34.936667+0000","last_unstale":"2024-08-03T20:47:37.979267+0000","last_undegraded":"2024-08-03T20:47:37.979267+0000","last_fullsized":"2024-08-03T20:47:37.979267+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:47:33.895731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:47:33.895731+0000","last_clean_scrub_stamp":"2024-08-03T20:47:33.895731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T07:46:07.162679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-03T20:47:37.979788+0000","last_change":"2024-08-03T20:47:35.940917+0000","last_active":"2024-08-03T20:47:37.979788+0000","last_peered":"2024-08-03T20:47:37.979788+0000","last_clean":"2024-08-03T20:47:37.979788+0000","last_became_active":"2024-08-03T20:47:35.940218+0000","last_became_peered":"2024-08-03T20:47:35.940218+0000","last_unstale":"2024-08-03T20:47:37.979788+0000","last_undegraded":"2024-08-03T20:47:37.979788+0000","last_fullsized":"2024-08-03T20:47:37.979788+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_clean_scrub_stamp":"2024-08-03T20:47:34.904132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-04T23:07:18.318169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"20'3","reported_seq":18,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-03T20:47:36.935425+0000","last_change":"2024-08-03T20:47:34.939100+0000","last_active":"2024-08-03T20:47:36.935425+0000","last_peered":"2024-08-03T20:47:36.935425+0000","last_clean":"2024-08-03T20:47:36.935425+0000","last_became_active":"2024-08-03T20:47:34.935869+0000","last_became_peered":"2024-08-03T20:47:34.935869+0000","last_unstale":"2024-08-03T20:47:36.935425+0000","last_undegraded":"2024-08-03T20:47:36.935425+0000","last_fullsized":"2024-08-03T20:47:36.935425+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:47:33.895731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:47:33.895731+0000","last_clean_scrub_stamp":"2024-08-03T20:47:33.895731+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T06:44:37.944928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-03T20:47:36.936329+0000","last_change":"2024-08-03T20:47:35.944428+0000","last_active":"2024-08-03T20:47:36.936329+0000","last_peered":"2024-08-03T20:47:36.936329+0000","last_clean":"2024-08-03T20:47:36.936329+0000","last_became_active":"2024-08-03T20:47:35.943608+0000","last_became_peered":"2024-08-03T20:47:35.943608+0000","last_unstale":"2024-08-03T20:47:36.936329+0000","last_undegraded":"2024-08-03T20:47:36.936329+0000","last_fullsized":"2024-08-03T20:47:36.936329+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_clean_scrub_stamp":"2024-08-03T20:47:34.904132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T05:51:29.523373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"20'6","reported_seq":21,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-03T20:47:36.941564+0000","last_change":"2024-08-03T20:47:34.931832+0000","last_active":"2024-08-03T20:47:36.941564+0000","last_peered":"2024-08-03T20:47:36.941564+0000","last_clean":"2024-08-03T20:47:36.941564+0000","last_became_active":"2024-08-03T20:47:34.931150+0000","last_became_peered":"2024-08-03T20:47:34.931150+0000","last_unstale":"2024-08-03T20:47:36.941564+0000","last_undegraded":"2024-08-03T20:47:36.941564+0000","last_fullsized":"2024-08-03T20:47:36.941564+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:47:33.895731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:47:33.895731+0000","last_clean_scrub_stamp":"2024-08-03T20:47:33.895731+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T05:01:48.922382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-03T20:47:37.316866+0000","last_change":"2024-08-03T20:47:35.953367+0000","last_active":"2024-08-03T20:47:37.316866+0000","last_peered":"2024-08-03T20:47:37.316866+0000","last_clean":"2024-08-03T20:47:37.316866+0000","last_became_active":"2024-08-03T20:47:35.952666+0000","last_became_peered":"2024-08-03T20:47:35.952666+0000","last_unstale":"2024-08-03T20:47:37.316866+0000","last_undegraded":"2024-08-03T20:47:37.316866+0000","last_fullsized":"2024-08-03T20:47:37.316866+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_clean_scrub_stamp":"2024-08-03T20:47:34.904132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-04T21:57:09.430415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"20'2","reported_seq":17,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-03T20:47:37.979652+0000","last_change":"2024-08-03T20:47:34.932411+0000","last_active":"2024-08-03T20:47:37.979652+0000","last_peered":"2024-08-03T20:47:37.979652+0000","last_clean":"2024-08-03T20:47:37.979652+0000","last_became_active":"2024-08-03T20:47:34.931336+0000","last_became_peered":"2024-08-03T20:47:34.931336+0000","last_unstale":"2024-08-03T20:47:37.979652+0000","last_undegraded":"2024-08-03T20:47:37.979652+0000","last_fullsized":"2024-08-03T20:47:37.979652+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:47:33.895731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:47:33.895731+0000","last_clean_scrub_stamp":"2024-08-03T20:47:33.895731+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T05:59:33.076657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-03T20:47:37.979070+0000","last_change":"2024-08-03T20:47:35.932875+0000","last_active":"2024-08-03T20:47:37.979070+0000","last_peered":"2024-08-03T20:47:37.979070+0000","last_clean":"2024-08-03T20:47:37.979070+0000","last_became_active":"2024-08-03T20:47:35.932516+0000","last_became_peered":"2024-08-03T20:47:35.932516+0000","last_unstale":"2024-08-03T20:47:37.979070+0000","last_undegraded":"2024-08-03T20:47:37.979070+0000","last_fullsized":"2024-08-03T20:47:37.979070+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_clean_scrub_stamp":"2024-08-03T20:47:34.904132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T08:34:04.057775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-03T20:47:37.979194+0000","last_change":"2024-08-03T20:47:34.947236+0000","last_active":"2024-08-03T20:47:37.979194+0000","last_peered":"2024-08-03T20:47:37.979194+0000","last_clean":"2024-08-03T20:47:37.979194+0000","last_became_active":"2024-08-03T20:47:34.946894+0000","last_became_peered":"2024-08-03T20:47:34.946894+0000","last_unstale":"2024-08-03T20:47:37.979194+0000","last_undegraded":"2024-08-03T20:47:37.979194+0000","last_fullsized":"2024-08-03T20:47:37.979194+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:47:33.895731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:47:33.895731+0000","last_clean_scrub_stamp":"2024-08-03T20:47:33.895731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T05:01:48.922382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-03T20:47:37.317004+0000","last_change":"2024-08-03T20:47:35.950815+0000","last_active":"2024-08-03T20:47:37.317004+0000","last_peered":"2024-08-03T20:47:37.317004+0000","last_clean":"2024-08-03T20:47:37.317004+0000","last_became_active":"2024-08-03T20:47:35.950494+0000","last_became_peered":"2024-08-03T20:47:35.950494+0000","last_unstale":"2024-08-03T20:47:37.317004+0000","last_undegraded":"2024-08-03T20:47:37.317004+0000","last_fullsized":"2024-08-03T20:47:37.317004+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_clean_scrub_stamp":"2024-08-03T20:47:34.904132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T05:49:42.422083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-03T20:47:36.941067+0000","last_change":"2024-08-03T20:47:34.939132+0000","last_active":"2024-08-03T20:47:36.941067+0000","last_peered":"2024-08-03T20:47:36.941067+0000","last_clean":"2024-08-03T20:47:36.941067+0000","last_became_active":"2024-08-03T20:47:34.937563+0000","last_became_peered":"2024-08-03T20:47:34.937563+0000","last_unstale":"2024-08-03T20:47:36.941067+0000","last_undegraded":"2024-08-03T20:47:36.941067+0000","last_fullsized":"2024-08-03T20:47:36.941067+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:47:33.895731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:47:33.895731+0000","last_clean_scrub_stamp":"2024-08-03T20:47:33.895731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T06:32:14.208950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-03T20:47:36.941315+0000","last_change":"2024-08-03T20:47:35.944827+0000","last_active":"2024-08-03T20:47:36.941315+0000","last_peered":"2024-08-03T20:47:36.941315+0000","last_clean":"2024-08-03T20:47:36.941315+0000","last_became_active":"2024-08-03T20:47:35.944517+0000","last_became_peered":"2024-08-03T20:47:35.944517+0000","last_unstale":"2024-08-03T20:47:36.941315+0000","last_undegraded":"2024-08-03T20:47:36.941315+0000","last_fullsized":"2024-08-03T20:47:36.941315+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_clean_scrub_stamp":"2024-08-03T20:47:34.904132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-04T23:57:18.840519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"20'8","reported_seq":23,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-03T20:47:37.352795+0000","last_change":"2024-08-03T20:47:34.936194+0000","last_active":"2024-08-03T20:47:37.352795+0000","last_peered":"2024-08-03T20:47:37.352795+0000","last_clean":"2024-08-03T20:47:37.352795+0000","last_became_active":"2024-08-03T20:47:34.935835+0000","last_became_peered":"2024-08-03T20:47:34.935835+0000","last_unstale":"2024-08-03T20:47:37.352795+0000","last_undegraded":"2024-08-03T20:47:37.352795+0000","last_fullsized":"2024-08-03T20:47:37.352795+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:47:33.895731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:47:33.895731+0000","last_clean_scrub_stamp":"2024-08-03T20:47:33.895731+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T06:13:41.904985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-03T20:47:37.316755+0000","last_change":"2024-08-03T20:47:35.950421+0000","last_active":"2024-08-03T20:47:37.316755+0000","last_peered":"2024-08-03T20:47:37.316755+0000","last_clean":"2024-08-03T20:47:37.316755+0000","last_became_active":"2024-08-03T20:47:35.950109+0000","last_became_peered":"2024-08-03T20:47:35.950109+0000","last_unstale":"2024-08-03T20:47:37.316755+0000","last_undegraded":"2024-08-03T20:47:37.316755+0000","last_fullsized":"2024-08-03T20:47:37.316755+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_clean_scrub_stamp":"2024-08-03T20:47:34.904132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T02:17:30.747723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"20'1","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-03T20:47:36.936564+0000","last_change":"2024-08-03T20:47:34.935417+0000","last_active":"2024-08-03T20:47:36.936564+0000","last_peered":"2024-08-03T20:47:36.936564+0000","last_clean":"2024-08-03T20:47:36.936564+0000","last_became_active":"2024-08-03T20:47:34.932211+0000","last_became_peered":"2024-08-03T20:47:34.932211+0000","last_unstale":"2024-08-03T20:47:36.936564+0000","last_undegraded":"2024-08-03T20:47:36.936564+0000","last_fullsized":"2024-08-03T20:47:36.936564+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:47:33.895731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:47:33.895731+0000","last_clean_scrub_stamp":"2024-08-03T20:47:33.895731+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T04:43:47.155950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-03T20:47:37.317197+0000","last_change":"2024-08-03T20:47:35.958660+0000","last_active":"2024-08-03T20:47:37.317197+0000","last_peered":"2024-08-03T20:47:37.317197+0000","last_clean":"2024-08-03T20:47:37.317197+0000","last_became_active":"2024-08-03T20:47:35.958296+0000","last_became_peered":"2024-08-03T20:47:35.958296+0000","last_unstale":"2024-08-03T20:47:37.317197+0000","last_undegraded":"2024-08-03T20:47:37.317197+0000","last_fullsized":"2024-08-03T20:47:37.317197+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_clean_scrub_stamp":"2024-08-03T20:47:34.904132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T01:52:59.081172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"20'2","reported_seq":17,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-03T20:47:37.352928+0000","last_change":"2024-08-03T20:47:34.934762+0000","last_active":"2024-08-03T20:47:37.352928+0000","last_peered":"2024-08-03T20:47:37.352928+0000","last_clean":"2024-08-03T20:47:37.352928+0000","last_became_active":"2024-08-03T20:47:34.933194+0000","last_became_peered":"2024-08-03T20:47:34.933194+0000","last_unstale":"2024-08-03T20:47:37.352928+0000","last_undegraded":"2024-08-03T20:47:37.352928+0000","last_fullsized":"2024-08-03T20:47:37.352928+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:47:33.895731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:47:33.895731+0000","last_clean_scrub_stamp":"2024-08-03T20:47:33.895731+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T07:58:41.861301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-03T20:47:37.317479+0000","last_change":"2024-08-03T20:47:35.952102+0000","last_active":"2024-08-03T20:47:37.317479+0000","last_peered":"2024-08-03T20:47:37.317479+0000","last_clean":"2024-08-03T20:47:37.317479+0000","last_became_active":"2024-08-03T20:47:35.951350+0000","last_became_peered":"2024-08-03T20:47:35.951350+0000","last_unstale":"2024-08-03T20:47:37.317479+0000","last_undegraded":"2024-08-03T20:47:37.317479+0000","last_fullsized":"2024-08-03T20:47:37.317479+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_clean_scrub_stamp":"2024-08-03T20:47:34.904132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-04T23:37:39.394391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"20'1","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-03T20:47:36.935542+0000","last_change":"2024-08-03T20:47:34.932568+0000","last_active":"2024-08-03T20:47:36.935542+0000","last_peered":"2024-08-03T20:47:36.935542+0000","last_clean":"2024-08-03T20:47:36.935542+0000","last_became_active":"2024-08-03T20:47:34.932223+0000","last_became_peered":"2024-08-03T20:47:34.932223+0000","last_unstale":"2024-08-03T20:47:36.935542+0000","last_undegraded":"2024-08-03T20:47:36.935542+0000","last_fullsized":"2024-08-03T20:47:36.935542+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:47:33.895731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:47:33.895731+0000","last_clean_scrub_stamp":"2024-08-03T20:47:33.895731+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T05:01:48.922382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-03T20:47:37.979523+0000","last_change":"2024-08-03T20:47:35.957384+0000","last_active":"2024-08-03T20:47:37.979523+0000","last_peered":"2024-08-03T20:47:37.979523+0000","last_clean":"2024-08-03T20:47:37.979523+0000","last_became_active":"2024-08-03T20:47:35.956464+0000","last_became_peered":"2024-08-03T20:47:35.956464+0000","last_unstale":"2024-08-03T20:47:37.979523+0000","last_undegraded":"2024-08-03T20:47:37.979523+0000","last_fullsized":"2024-08-03T20:47:37.979523+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_clean_scrub_stamp":"2024-08-03T20:47:34.904132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T05:49:42.422083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-03T20:47:37.352724+0000","last_change":"2024-08-03T20:47:34.934154+0000","last_active":"2024-08-03T20:47:37.352724+0000","last_peered":"2024-08-03T20:47:37.352724+0000","last_clean":"2024-08-03T20:47:37.352724+0000","last_became_active":"2024-08-03T20:47:34.933257+0000","last_became_peered":"2024-08-03T20:47:34.933257+0000","last_unstale":"2024-08-03T20:47:37.352724+0000","last_undegraded":"2024-08-03T20:47:37.352724+0000","last_fullsized":"2024-08-03T20:47:37.352724+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:47:33.895731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:47:33.895731+0000","last_clean_scrub_stamp":"2024-08-03T20:47:33.895731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-04T21:10:14.859583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-03T20:47:37.317012+0000","last_change":"2024-08-03T20:47:35.957705+0000","last_active":"2024-08-03T20:47:37.317012+0000","last_peered":"2024-08-03T20:47:37.317012+0000","last_clean":"2024-08-03T20:47:37.317012+0000","last_became_active":"2024-08-03T20:47:35.951141+0000","last_became_peered":"2024-08-03T20:47:35.951141+0000","last_unstale":"2024-08-03T20:47:37.317012+0000","last_undegraded":"2024-08-03T20:47:37.317012+0000","last_fullsized":"2024-08-03T20:47:37.317012+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_clean_scrub_stamp":"2024-08-03T20:47:34.904132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T06:29:37.397869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"16'72","reported_seq":111,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-03T20:47:37.316600+0000","last_change":"2024-08-03T20:47:30.547632+0000","last_active":"2024-08-03T20:47:37.316600+0000","last_peered":"2024-08-03T20:47:37.316600+0000","last_clean":"2024-08-03T20:47:37.316600+0000","last_became_active":"2024-08-03T20:47:30.546255+0000","last_became_peered":"2024-08-03T20:47:30.546255+0000","last_unstale":"2024-08-03T20:47:37.316600+0000","last_undegraded":"2024-08-03T20:47:37.316600+0000","last_fullsized":"2024-08-03T20:47:37.316600+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:47:28.874451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:47:28.874451+0000","last_clean_scrub_stamp":"2024-08-03T20:47:28.874451+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T00:19:40.548812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-03T20:47:37.978786+0000","last_change":"2024-08-03T20:47:34.941906+0000","last_active":"2024-08-03T20:47:37.978786+0000","last_peered":"2024-08-03T20:47:37.978786+0000","last_clean":"2024-08-03T20:47:37.978786+0000","last_became_active":"2024-08-03T20:47:34.941492+0000","last_became_peered":"2024-08-03T20:47:34.941492+0000","last_unstale":"2024-08-03T20:47:37.978786+0000","last_undegraded":"2024-08-03T20:47:37.978786+0000","last_fullsized":"2024-08-03T20:47:37.978786+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:47:33.895731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:47:33.895731+0000","last_clean_scrub_stamp":"2024-08-03T20:47:33.895731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T06:32:14.208950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-03T20:47:37.978467+0000","last_change":"2024-08-03T20:47:35.953945+0000","last_active":"2024-08-03T20:47:37.978467+0000","last_peered":"2024-08-03T20:47:37.978467+0000","last_clean":"2024-08-03T20:47:37.978467+0000","last_became_active":"2024-08-03T20:47:35.953580+0000","last_became_peered":"2024-08-03T20:47:35.953580+0000","last_unstale":"2024-08-03T20:47:37.978467+0000","last_undegraded":"2024-08-03T20:47:37.978467+0000","last_fullsized":"2024-08-03T20:47:37.978467+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_clean_scrub_stamp":"2024-08-03T20:47:34.904132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T03:08:26.700491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-03T20:47:37.316444+0000","last_change":"2024-08-03T20:47:34.944839+0000","last_active":"2024-08-03T20:47:37.316444+0000","last_peered":"2024-08-03T20:47:37.316444+0000","last_clean":"2024-08-03T20:47:37.316444+0000","last_became_active":"2024-08-03T20:47:34.944530+0000","last_became_peered":"2024-08-03T20:47:34.944530+0000","last_unstale":"2024-08-03T20:47:37.316444+0000","last_undegraded":"2024-08-03T20:47:37.316444+0000","last_fullsized":"2024-08-03T20:47:37.316444+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:47:33.895731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:47:33.895731+0000","last_clean_scrub_stamp":"2024-08-03T20:47:33.895731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T02:40:00.976532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-03T20:47:37.978347+0000","last_change":"2024-08-03T20:47:35.926904+0000","last_active":"2024-08-03T20:47:37.978347+0000","last_peered":"2024-08-03T20:47:37.978347+0000","last_clean":"2024-08-03T20:47:37.978347+0000","last_became_active":"2024-08-03T20:47:35.926525+0000","last_became_peered":"2024-08-03T20:47:35.926525+0000","last_unstale":"2024-08-03T20:47:37.978347+0000","last_undegraded":"2024-08-03T20:47:37.978347+0000","last_fullsized":"2024-08-03T20:47:37.978347+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_clean_scrub_stamp":"2024-08-03T20:47:34.904132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T05:57:44.170587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-03T20:47:37.352725+0000","last_change":"2024-08-03T20:47:34.939125+0000","last_active":"2024-08-03T20:47:37.352725+0000","last_peered":"2024-08-03T20:47:37.352725+0000","last_clean":"2024-08-03T20:47:37.352725+0000","last_became_active":"2024-08-03T20:47:34.935161+0000","last_became_peered":"2024-08-03T20:47:34.935161+0000","last_unstale":"2024-08-03T20:47:37.352725+0000","last_undegraded":"2024-08-03T20:47:37.352725+0000","last_fullsized":"2024-08-03T20:47:37.352725+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:47:33.895731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:47:33.895731+0000","last_clean_scrub_stamp":"2024-08-03T20:47:33.895731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T04:37:55.336635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-03T20:47:36.934972+0000","last_change":"2024-08-03T20:47:35.933618+0000","last_active":"2024-08-03T20:47:36.934972+0000","last_peered":"2024-08-03T20:47:36.934972+0000","last_clean":"2024-08-03T20:47:36.934972+0000","last_became_active":"2024-08-03T20:47:35.932873+0000","last_became_peered":"2024-08-03T20:47:35.932873+0000","last_unstale":"2024-08-03T20:47:36.934972+0000","last_undegraded":"2024-08-03T20:47:36.934972+0000","last_fullsized":"2024-08-03T20:47:36.934972+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_clean_scrub_stamp":"2024-08-03T20:47:34.904132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-04T23:57:18.840519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-03T20:47:37.979331+0000","last_change":"2024-08-03T20:47:34.940602+0000","last_active":"2024-08-03T20:47:37.979331+0000","last_peered":"2024-08-03T20:47:37.979331+0000","last_clean":"2024-08-03T20:47:37.979331+0000","last_became_active":"2024-08-03T20:47:34.940156+0000","last_became_peered":"2024-08-03T20:47:34.940156+0000","last_unstale":"2024-08-03T20:47:37.979331+0000","last_undegraded":"2024-08-03T20:47:37.979331+0000","last_fullsized":"2024-08-03T20:47:37.979331+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:47:33.895731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:47:33.895731+0000","last_clean_scrub_stamp":"2024-08-03T20:47:33.895731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T06:44:37.944928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-03T20:47:36.935959+0000","last_change":"2024-08-03T20:47:35.938536+0000","last_active":"2024-08-03T20:47:36.935959+0000","last_peered":"2024-08-03T20:47:36.935959+0000","last_clean":"2024-08-03T20:47:36.935959+0000","last_became_active":"2024-08-03T20:47:35.938281+0000","last_became_peered":"2024-08-03T20:47:35.938281+0000","last_unstale":"2024-08-03T20:47:36.935959+0000","last_undegraded":"2024-08-03T20:47:36.935959+0000","last_fullsized":"2024-08-03T20:47:36.935959+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_clean_scrub_stamp":"2024-08-03T20:47:34.904132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T06:17:22.118393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-03T20:47:36.935901+0000","last_change":"2024-08-03T20:47:34.936859+0000","last_active":"2024-08-03T20:47:36.935901+0000","last_peered":"2024-08-03T20:47:36.935901+0000","last_clean":"2024-08-03T20:47:36.935901+0000","last_became_active":"2024-08-03T20:47:34.936017+0000","last_became_peered":"2024-08-03T20:47:34.936017+0000","last_unstale":"2024-08-03T20:47:36.935901+0000","last_undegraded":"2024-08-03T20:47:36.935901+0000","last_fullsized":"2024-08-03T20:47:36.935901+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:47:33.895731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:47:33.895731+0000","last_clean_scrub_stamp":"2024-08-03T20:47:33.895731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T02:40:00.976532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-03T20:47:37.979373+0000","last_change":"2024-08-03T20:47:35.940325+0000","last_active":"2024-08-03T20:47:37.979373+0000","last_peered":"2024-08-03T20:47:37.979373+0000","last_clean":"2024-08-03T20:47:37.979373+0000","last_became_active":"2024-08-03T20:47:35.939888+0000","last_became_peered":"2024-08-03T20:47:35.939888+0000","last_unstale":"2024-08-03T20:47:37.979373+0000","last_undegraded":"2024-08-03T20:47:37.979373+0000","last_fullsized":"2024-08-03T20:47:37.979373+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_clean_scrub_stamp":"2024-08-03T20:47:34.904132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T02:17:30.747723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-03T20:47:37.317158+0000","last_change":"2024-08-03T20:47:34.946657+0000","last_active":"2024-08-03T20:47:37.317158+0000","last_peered":"2024-08-03T20:47:37.317158+0000","last_clean":"2024-08-03T20:47:37.317158+0000","last_became_active":"2024-08-03T20:47:34.945719+0000","last_became_peered":"2024-08-03T20:47:34.945719+0000","last_unstale":"2024-08-03T20:47:37.317158+0000","last_undegraded":"2024-08-03T20:47:37.317158+0000","last_fullsized":"2024-08-03T20:47:37.317158+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:47:33.895731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:47:33.895731+0000","last_clean_scrub_stamp":"2024-08-03T20:47:33.895731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-04T21:30:40.295081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-03T20:47:36.935812+0000","last_change":"2024-08-03T20:47:35.936144+0000","last_active":"2024-08-03T20:47:36.935812+0000","last_peered":"2024-08-03T20:47:36.935812+0000","last_clean":"2024-08-03T20:47:36.935812+0000","last_became_active":"2024-08-03T20:47:35.935842+0000","last_became_peered":"2024-08-03T20:47:35.935842+0000","last_unstale":"2024-08-03T20:47:36.935812+0000","last_undegraded":"2024-08-03T20:47:36.935812+0000","last_fullsized":"2024-08-03T20:47:36.935812+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_clean_scrub_stamp":"2024-08-03T20:47:34.904132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-04T22:33:13.028920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"21'1","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-03T20:47:36.935790+0000","last_change":"2024-08-03T20:47:34.934773+0000","last_active":"2024-08-03T20:47:36.935790+0000","last_peered":"2024-08-03T20:47:36.935790+0000","last_clean":"2024-08-03T20:47:36.935790+0000","last_became_active":"2024-08-03T20:47:34.934473+0000","last_became_peered":"2024-08-03T20:47:34.934473+0000","last_unstale":"2024-08-03T20:47:36.935790+0000","last_undegraded":"2024-08-03T20:47:36.935790+0000","last_fullsized":"2024-08-03T20:47:36.935790+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:47:33.895731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:47:33.895731+0000","last_clean_scrub_stamp":"2024-08-03T20:47:33.895731+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-04T22:30:28.119577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-03T20:47:37.979065+0000","last_change":"2024-08-03T20:47:35.933622+0000","last_active":"2024-08-03T20:47:37.979065+0000","last_peered":"2024-08-03T20:47:37.979065+0000","last_clean":"2024-08-03T20:47:37.979065+0000","last_became_active":"2024-08-03T20:47:35.933133+0000","last_became_peered":"2024-08-03T20:47:35.933133+0000","last_unstale":"2024-08-03T20:47:37.979065+0000","last_undegraded":"2024-08-03T20:47:37.979065+0000","last_fullsized":"2024-08-03T20:47:37.979065+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_clean_scrub_stamp":"2024-08-03T20:47:34.904132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-04T22:43:36.335718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"20'2","reported_seq":17,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-03T20:47:36.941891+0000","last_change":"2024-08-03T20:47:34.944805+0000","last_active":"2024-08-03T20:47:36.941891+0000","last_peered":"2024-08-03T20:47:36.941891+0000","last_clean":"2024-08-03T20:47:36.941891+0000","last_became_active":"2024-08-03T20:47:34.943207+0000","last_became_peered":"2024-08-03T20:47:34.943207+0000","last_unstale":"2024-08-03T20:47:36.941891+0000","last_undegraded":"2024-08-03T20:47:36.941891+0000","last_fullsized":"2024-08-03T20:47:36.941891+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:47:33.895731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:47:33.895731+0000","last_clean_scrub_stamp":"2024-08-03T20:47:33.895731+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T07:46:07.162679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-03T20:47:37.979614+0000","last_change":"2024-08-03T20:47:35.941163+0000","last_active":"2024-08-03T20:47:37.979614+0000","last_peered":"2024-08-03T20:47:37.979614+0000","last_clean":"2024-08-03T20:47:37.979614+0000","last_became_active":"2024-08-03T20:47:35.940955+0000","last_became_peered":"2024-08-03T20:47:35.940955+0000","last_unstale":"2024-08-03T20:47:37.979614+0000","last_undegraded":"2024-08-03T20:47:37.979614+0000","last_fullsized":"2024-08-03T20:47:37.979614+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_clean_scrub_stamp":"2024-08-03T20:47:34.904132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T01:08:56.085650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-03T20:47:37.316412+0000","last_change":"2024-08-03T20:47:34.943492+0000","last_active":"2024-08-03T20:47:37.316412+0000","last_peered":"2024-08-03T20:47:37.316412+0000","last_clean":"2024-08-03T20:47:37.316412+0000","last_became_active":"2024-08-03T20:47:34.943121+0000","last_became_peered":"2024-08-03T20:47:34.943121+0000","last_unstale":"2024-08-03T20:47:37.316412+0000","last_undegraded":"2024-08-03T20:47:37.316412+0000","last_fullsized":"2024-08-03T20:47:37.316412+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:47:33.895731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:47:33.895731+0000","last_clean_scrub_stamp":"2024-08-03T20:47:33.895731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T07:46:07.162679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-03T20:47:36.935830+0000","last_change":"2024-08-03T20:47:35.924518+0000","last_active":"2024-08-03T20:47:36.935830+0000","last_peered":"2024-08-03T20:47:36.935830+0000","last_clean":"2024-08-03T20:47:36.935830+0000","last_became_active":"2024-08-03T20:47:35.923938+0000","last_became_peered":"2024-08-03T20:47:35.923938+0000","last_unstale":"2024-08-03T20:47:36.935830+0000","last_undegraded":"2024-08-03T20:47:36.935830+0000","last_fullsized":"2024-08-03T20:47:36.935830+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_clean_scrub_stamp":"2024-08-03T20:47:34.904132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T04:26:33.175697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-03T20:47:36.935369+0000","last_change":"2024-08-03T20:47:34.941712+0000","last_active":"2024-08-03T20:47:36.935369+0000","last_peered":"2024-08-03T20:47:36.935369+0000","last_clean":"2024-08-03T20:47:36.935369+0000","last_became_active":"2024-08-03T20:47:34.941138+0000","last_became_peered":"2024-08-03T20:47:34.941138+0000","last_unstale":"2024-08-03T20:47:36.935369+0000","last_undegraded":"2024-08-03T20:47:36.935369+0000","last_fullsized":"2024-08-03T20:47:36.935369+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:47:33.895731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:47:33.895731+0000","last_clean_scrub_stamp":"2024-08-03T20:47:33.895731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T07:46:07.162679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-03T20:47:37.978318+0000","last_change":"2024-08-03T20:47:35.941992+0000","last_active":"2024-08-03T20:47:37.978318+0000","last_peered":"2024-08-03T20:47:37.978318+0000","last_clean":"2024-08-03T20:47:37.978318+0000","last_became_active":"2024-08-03T20:47:35.941633+0000","last_became_peered":"2024-08-03T20:47:35.941633+0000","last_unstale":"2024-08-03T20:47:37.978318+0000","last_undegraded":"2024-08-03T20:47:37.978318+0000","last_fullsized":"2024-08-03T20:47:37.978318+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_clean_scrub_stamp":"2024-08-03T20:47:34.904132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T03:06:49.510875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-03T20:47:36.935863+0000","last_change":"2024-08-03T20:47:34.932831+0000","last_active":"2024-08-03T20:47:36.935863+0000","last_peered":"2024-08-03T20:47:36.935863+0000","last_clean":"2024-08-03T20:47:36.935863+0000","last_became_active":"2024-08-03T20:47:34.932520+0000","last_became_peered":"2024-08-03T20:47:34.932520+0000","last_unstale":"2024-08-03T20:47:36.935863+0000","last_undegraded":"2024-08-03T20:47:36.935863+0000","last_fullsized":"2024-08-03T20:47:36.935863+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:47:33.895731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:47:33.895731+0000","last_clean_scrub_stamp":"2024-08-03T20:47:33.895731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T00:43:30.770268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-03T20:47:37.978692+0000","last_change":"2024-08-03T20:47:35.942701+0000","last_active":"2024-08-03T20:47:37.978692+0000","last_peered":"2024-08-03T20:47:37.978692+0000","last_clean":"2024-08-03T20:47:37.978692+0000","last_became_active":"2024-08-03T20:47:35.942261+0000","last_became_peered":"2024-08-03T20:47:35.942261+0000","last_unstale":"2024-08-03T20:47:37.978692+0000","last_undegraded":"2024-08-03T20:47:37.978692+0000","last_fullsized":"2024-08-03T20:47:37.978692+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_clean_scrub_stamp":"2024-08-03T20:47:34.904132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-04T22:01:37.009450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"20'2","reported_seq":17,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-03T20:47:37.317111+0000","last_change":"2024-08-03T20:47:34.947233+0000","last_active":"2024-08-03T20:47:37.317111+0000","last_peered":"2024-08-03T20:47:37.317111+0000","last_clean":"2024-08-03T20:47:37.317111+0000","last_became_active":"2024-08-03T20:47:34.946866+0000","last_became_peered":"2024-08-03T20:47:34.946866+0000","last_unstale":"2024-08-03T20:47:37.317111+0000","last_undegraded":"2024-08-03T20:47:37.317111+0000","last_fullsized":"2024-08-03T20:47:37.317111+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:47:33.895731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:47:33.895731+0000","last_clean_scrub_stamp":"2024-08-03T20:47:33.895731+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T03:46:04.675513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-03T20:47:36.941158+0000","last_change":"2024-08-03T20:47:35.936142+0000","last_active":"2024-08-03T20:47:36.941158+0000","last_peered":"2024-08-03T20:47:36.941158+0000","last_clean":"2024-08-03T20:47:36.941158+0000","last_became_active":"2024-08-03T20:47:35.935764+0000","last_became_peered":"2024-08-03T20:47:35.935764+0000","last_unstale":"2024-08-03T20:47:36.941158+0000","last_undegraded":"2024-08-03T20:47:36.941158+0000","last_fullsized":"2024-08-03T20:47:36.941158+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_clean_scrub_stamp":"2024-08-03T20:47:34.904132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-04T22:12:15.293757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-03T20:47:37.979240+0000","last_change":"2024-08-03T20:47:35.934358+0000","last_active":"2024-08-03T20:47:37.979240+0000","last_peered":"2024-08-03T20:47:37.979240+0000","last_clean":"2024-08-03T20:47:37.979240+0000","last_became_active":"2024-08-03T20:47:35.933874+0000","last_became_peered":"2024-08-03T20:47:35.933874+0000","last_unstale":"2024-08-03T20:47:37.979240+0000","last_undegraded":"2024-08-03T20:47:37.979240+0000","last_fullsized":"2024-08-03T20:47:37.979240+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_clean_scrub_stamp":"2024-08-03T20:47:34.904132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T00:10:43.343156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"20'2","reported_seq":17,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-03T20:47:37.316590+0000","last_change":"2024-08-03T20:47:34.950578+0000","last_active":"2024-08-03T20:47:37.316590+0000","last_peered":"2024-08-03T20:47:37.316590+0000","last_clean":"2024-08-03T20:47:37.316590+0000","last_became_active":"2024-08-03T20:47:34.950187+0000","last_became_peered":"2024-08-03T20:47:34.950187+0000","last_unstale":"2024-08-03T20:47:37.316590+0000","last_undegraded":"2024-08-03T20:47:37.316590+0000","last_fullsized":"2024-08-03T20:47:37.316590+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:47:33.895731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:47:33.895731+0000","last_clean_scrub_stamp":"2024-08-03T20:47:33.895731+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T00:43:30.770268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-03T20:47:37.979438+0000","last_change":"2024-08-03T20:47:35.934586+0000","last_active":"2024-08-03T20:47:37.979438+0000","last_peered":"2024-08-03T20:47:37.979438+0000","last_clean":"2024-08-03T20:47:37.979438+0000","last_became_active":"2024-08-03T20:47:35.934289+0000","last_became_peered":"2024-08-03T20:47:35.934289+0000","last_unstale":"2024-08-03T20:47:37.979438+0000","last_undegraded":"2024-08-03T20:47:37.979438+0000","last_fullsized":"2024-08-03T20:47:37.979438+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_clean_scrub_stamp":"2024-08-03T20:47:34.904132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T04:51:27.371858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"20'2","reported_seq":17,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-03T20:47:37.316998+0000","last_change":"2024-08-03T20:47:34.946365+0000","last_active":"2024-08-03T20:47:37.316998+0000","last_peered":"2024-08-03T20:47:37.316998+0000","last_clean":"2024-08-03T20:47:37.316998+0000","last_became_active":"2024-08-03T20:47:34.945922+0000","last_became_peered":"2024-08-03T20:47:34.945922+0000","last_unstale":"2024-08-03T20:47:37.316998+0000","last_undegraded":"2024-08-03T20:47:37.316998+0000","last_fullsized":"2024-08-03T20:47:37.316998+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:47:33.895731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:47:33.895731+0000","last_clean_scrub_stamp":"2024-08-03T20:47:33.895731+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T06:59:03.089784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-03T20:47:37.979781+0000","last_change":"2024-08-03T20:47:35.941147+0000","last_active":"2024-08-03T20:47:37.979781+0000","last_peered":"2024-08-03T20:47:37.979781+0000","last_clean":"2024-08-03T20:47:37.979781+0000","last_became_active":"2024-08-03T20:47:35.940605+0000","last_became_peered":"2024-08-03T20:47:35.940605+0000","last_unstale":"2024-08-03T20:47:37.979781+0000","last_undegraded":"2024-08-03T20:47:37.979781+0000","last_fullsized":"2024-08-03T20:47:37.979781+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_clean_scrub_stamp":"2024-08-03T20:47:34.904132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-04T22:33:13.028920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"20'1","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-03T20:47:37.978906+0000","last_change":"2024-08-03T20:47:34.934888+0000","last_active":"2024-08-03T20:47:37.978906+0000","last_peered":"2024-08-03T20:47:37.978906+0000","last_clean":"2024-08-03T20:47:37.978906+0000","last_became_active":"2024-08-03T20:47:34.934595+0000","last_became_peered":"2024-08-03T20:47:34.934595+0000","last_unstale":"2024-08-03T20:47:37.978906+0000","last_undegraded":"2024-08-03T20:47:37.978906+0000","last_fullsized":"2024-08-03T20:47:37.978906+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:47:33.895731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:47:33.895731+0000","last_clean_scrub_stamp":"2024-08-03T20:47:33.895731+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T06:13:41.904985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-03T20:47:37.978652+0000","last_change":"2024-08-03T20:47:35.953798+0000","last_active":"2024-08-03T20:47:37.978652+0000","last_peered":"2024-08-03T20:47:37.978652+0000","last_clean":"2024-08-03T20:47:37.978652+0000","last_became_active":"2024-08-03T20:47:35.950186+0000","last_became_peered":"2024-08-03T20:47:35.950186+0000","last_unstale":"2024-08-03T20:47:37.978652+0000","last_undegraded":"2024-08-03T20:47:37.978652+0000","last_fullsized":"2024-08-03T20:47:37.978652+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_clean_scrub_stamp":"2024-08-03T20:47:34.904132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T07:58:55.643722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"20'4","reported_seq":19,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-03T20:47:37.979858+0000","last_change":"2024-08-03T20:47:34.939409+0000","last_active":"2024-08-03T20:47:37.979858+0000","last_peered":"2024-08-03T20:47:37.979858+0000","last_clean":"2024-08-03T20:47:37.979858+0000","last_became_active":"2024-08-03T20:47:34.939057+0000","last_became_peered":"2024-08-03T20:47:34.939057+0000","last_unstale":"2024-08-03T20:47:37.979858+0000","last_undegraded":"2024-08-03T20:47:37.979858+0000","last_fullsized":"2024-08-03T20:47:37.979858+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:47:33.895731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:47:33.895731+0000","last_clean_scrub_stamp":"2024-08-03T20:47:33.895731+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T05:49:05.605066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509446,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31972,"kb_used_data":788,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339868,"statfs":{"total":96636764160,"available":96604024832,"internally_reserved":0,"allocated":806912,"data_stored":663491,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":15,"seq":64424509447,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31396,"kb_used_data":216,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340444,"statfs":{"total":96636764160,"available":96604614656,"internally_reserved":0,"allocated":221184,"data_stored":74153,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":15,"seq":64424509447,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31388,"kb_used_data":208,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340452,"statfs":{"total":96636764160,"available":96604622848,"internally_reserved":0,"allocated":212992,"data_stored":72631,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509446,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31392,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340448,"statfs":{"total":96636764160,"available":96604618752,"internally_reserved":0,"allocated":217088,"data_stored":72701,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509448,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31980,"kb_used_data":796,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339860,"statfs":{"total":96636764160,"available":96604016640,"internally_reserved":0,"allocated":815104,"data_stored":663547,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542151,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31400,"kb_used_data":216,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340440,"statfs":{"total":96636764160,"available":96604610560,"internally_reserved":0,"allocated":221184,"data_stored":72667,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":14,"seq":60129542149,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31400,"kb_used_data":216,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340440,"statfs":{"total":96636764160,"available":96604610560,"internally_reserved":0,"allocated":221184,"data_stored":74153,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":14,"seq":60129542150,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31388,"kb_used_data":204,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340452,"statfs":{"total":96636764160,"available":96604622848,"internally_reserved":0,"allocated":208896,"data_stored":72541,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-08-03T20:47:46.965 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-08-03T20:47:47.232 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:47.230+0000 7fe981e00640 1 -- 172.21.15.32:0/924855885 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fe97c0b78d0 msgr2=0x7fe97c09ac30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:47:47.232 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:47.230+0000 7fe981e00640 1 --2- 172.21.15.32:0/924855885 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fe97c0b78d0 0x7fe97c09ac30 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fe964009830 tx=0x7fe96402a3f0 comp rx=0 tx=0).stop 2024-08-03T20:47:47.232 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:47.230+0000 7fe981e00640 1 -- 172.21.15.32:0/924855885 shutdown_connections 2024-08-03T20:47:47.232 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:47.230+0000 7fe981e00640 1 --2- 172.21.15.32:0/924855885 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fe97c0b78d0 0x7fe97c09ac30 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:47.232 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:47.230+0000 7fe981e00640 1 --2- 172.21.15.32:0/924855885 >> v2:172.21.15.32:3300/0 conn(0x7fe97c09b170 0x7fe97c0d0350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:47.232 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:47.230+0000 7fe981e00640 1 -- 172.21.15.32:0/924855885 >> 172.21.15.32:0/924855885 conn(0x7fe97c0b3bb0 msgr2=0x7fe97c0b3fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:47:47.232 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:47.230+0000 7fe981e00640 1 -- 172.21.15.32:0/924855885 shutdown_connections 2024-08-03T20:47:47.232 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:47.230+0000 7fe981e00640 1 -- 172.21.15.32:0/924855885 wait complete. 2024-08-03T20:47:47.235 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:47.234+0000 7fe981e00640 1 Processor -- start 2024-08-03T20:47:47.235 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:47.234+0000 7fe981e00640 1 -- start start 2024-08-03T20:47:47.238 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:47.235+0000 7fe981e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe97c09b170 0x7fe97c14dc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:47:47.238 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:47.235+0000 7fe981e00640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fe97c0b78d0 0x7fe97c14ef00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:47:47.238 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:47.235+0000 7fe981e00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fe97c0d1c30 0x7fe97c14f440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:47:47.238 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:47.235+0000 7fe981e00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fe97c0d0dd0 con 0x7fe97c09b170 2024-08-03T20:47:47.238 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:47.235+0000 7fe981e00640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7fe97c0d0c50 con 0x7fe97c0d1c30 2024-08-03T20:47:47.238 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:47.235+0000 7fe981e00640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7fe97c0d0f50 con 0x7fe97c0b78d0 2024-08-03T20:47:47.238 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:47.236+0000 7fe97aa00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe97c09b170 0x7fe97c14dc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:47:47.238 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:47.236+0000 7fe97aa00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe97c09b170 0x7fe97c14dc30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:57678/0 (socket says 172.21.15.32:57678) 2024-08-03T20:47:47.238 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:47.236+0000 7fe97aa00640 1 -- 172.21.15.32:0/2606599066 learned_addr learned my addr 172.21.15.32:0/2606599066 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:47:47.238 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:47.236+0000 7fe973e00640 1 --2- 172.21.15.32:0/2606599066 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fe97c0b78d0 0x7fe97c14ef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:47:47.239 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:47.236+0000 7fe97b400640 1 --2- 172.21.15.32:0/2606599066 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fe97c0d1c30 0x7fe97c14f440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:47:47.239 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:47.236+0000 7fe97aa00640 1 -- 172.21.15.32:0/2606599066 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fe97c0b78d0 msgr2=0x7fe97c14ef00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:47:47.239 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:47.236+0000 7fe97aa00640 1 --2- 172.21.15.32:0/2606599066 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fe97c0b78d0 0x7fe97c14ef00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:47.239 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:47.236+0000 7fe97aa00640 1 -- 172.21.15.32:0/2606599066 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fe97c0d1c30 msgr2=0x7fe97c14f440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:47:47.239 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:47.236+0000 7fe97aa00640 1 --2- 172.21.15.32:0/2606599066 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fe97c0d1c30 0x7fe97c14f440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:47.239 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:47.236+0000 7fe97aa00640 1 -- 172.21.15.32:0/2606599066 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe97c0b65e0 con 0x7fe97c09b170 2024-08-03T20:47:47.239 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:47.237+0000 7fe97aa00640 1 --2- 172.21.15.32:0/2606599066 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe97c09b170 0x7fe97c14dc30 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fe96c00c430 tx=0x7fe96c00c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:47:47.240 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:47.237+0000 7fe973400640 1 -- 172.21.15.32:0/2606599066 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe96c011790 con 0x7fe97c09b170 2024-08-03T20:47:47.240 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:47.237+0000 7fe973400640 1 -- 172.21.15.32:0/2606599066 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fe96c017070 con 0x7fe97c09b170 2024-08-03T20:47:47.240 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:47.237+0000 7fe981e00640 1 -- 172.21.15.32:0/2606599066 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe97c0b68c0 con 0x7fe97c09b170 2024-08-03T20:47:47.240 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:47.237+0000 7fe973400640 1 -- 172.21.15.32:0/2606599066 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe96c011d40 con 0x7fe97c09b170 2024-08-03T20:47:47.240 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:47.237+0000 7fe981e00640 1 -- 172.21.15.32:0/2606599066 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe97c14e700 con 0x7fe97c09b170 2024-08-03T20:47:47.240 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:47.239+0000 7fe981e00640 1 -- 172.21.15.32:0/2606599066 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe97c0d0350 con 0x7fe97c09b170 2024-08-03T20:47:47.242 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:47.240+0000 7fe973400640 1 -- 172.21.15.32:0/2606599066 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7fe96c01c460 con 0x7fe97c09b170 2024-08-03T20:47:47.243 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:47.241+0000 7fe973400640 1 --2- 172.21.15.32:0/2606599066 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fe958077040 0x7fe958079550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:47:47.243 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:47.241+0000 7fe973400640 1 -- 172.21.15.32:0/2606599066 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7fe96c098670 con 0x7fe97c09b170 2024-08-03T20:47:47.243 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:47.242+0000 7fe973e00640 1 --2- 172.21.15.32:0/2606599066 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fe958077040 0x7fe958079550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:47:47.244 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:47.243+0000 7fe973e00640 1 --2- 172.21.15.32:0/2606599066 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fe958077040 0x7fe958079550 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fe964034000 tx=0x7fe964004730 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:47:47.247 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:47.246+0000 7fe973400640 1 -- 172.21.15.32:0/2606599066 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7fe96c0617e0 con 0x7fe97c09b170 2024-08-03T20:47:47.449 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:47.448+0000 7fe981e00640 1 -- 172.21.15.32:0/2606599066 --> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fe97c14e170 con 0x7fe958077040 2024-08-03T20:47:47.454 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:47.453+0000 7fe973400640 1 -- 172.21.15.32:0/2606599066 <== mgr.4106 v2:172.21.15.118:6800/4205950699 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+251826 (secure 0 0 0) 0x7fe97c14e170 con 0x7fe958077040 2024-08-03T20:47:47.457 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-03T20:47:47.458 INFO:teuthology.orchestra.run.smithi032.stderr:dumped all 2024-08-03T20:47:47.460 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:47.457+0000 7fe981e00640 1 -- 172.21.15.32:0/2606599066 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fe958077040 msgr2=0x7fe958079550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:47:47.460 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:47.457+0000 7fe981e00640 1 --2- 172.21.15.32:0/2606599066 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fe958077040 0x7fe958079550 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fe964034000 tx=0x7fe964004730 comp rx=0 tx=0).stop 2024-08-03T20:47:47.460 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:47.457+0000 7fe981e00640 1 -- 172.21.15.32:0/2606599066 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe97c09b170 msgr2=0x7fe97c14dc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:47:47.461 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:47.457+0000 7fe981e00640 1 --2- 172.21.15.32:0/2606599066 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe97c09b170 0x7fe97c14dc30 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fe96c00c430 tx=0x7fe96c00c830 comp rx=0 tx=0).stop 2024-08-03T20:47:47.461 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:47.457+0000 7fe981e00640 1 -- 172.21.15.32:0/2606599066 shutdown_connections 2024-08-03T20:47:47.461 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:47.457+0000 7fe981e00640 1 --2- 172.21.15.32:0/2606599066 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fe958077040 0x7fe958079550 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:47.461 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:47.457+0000 7fe981e00640 1 --2- 172.21.15.32:0/2606599066 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fe97c0d1c30 0x7fe97c14f440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:47.461 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:47.457+0000 7fe981e00640 1 --2- 172.21.15.32:0/2606599066 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fe97c0b78d0 0x7fe97c14ef00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:47.461 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:47.457+0000 7fe981e00640 1 --2- 172.21.15.32:0/2606599066 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe97c09b170 0x7fe97c14dc30 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:47.461 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:47.457+0000 7fe981e00640 1 -- 172.21.15.32:0/2606599066 >> 172.21.15.32:0/2606599066 conn(0x7fe97c0b3bb0 msgr2=0x7fe97c0ccf20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:47:47.461 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:47.457+0000 7fe981e00640 1 -- 172.21.15.32:0/2606599066 shutdown_connections 2024-08-03T20:47:47.461 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:47.457+0000 7fe981e00640 1 -- 172.21.15.32:0/2606599066 wait complete. 2024-08-03T20:47:47.483 INFO:teuthology.orchestra.run.smithi032.stdout:{"pg_ready":true,"pg_map":{"version":23,"stamp":"2024-08-03T20:47:46.223784+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":146,"num_write_kb":1642,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":111,"ondisk_log_size":111,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":252316,"kb_used_data":2856,"kb_used_omap":12,"kb_used_meta":249331,"kb_avail":754722404,"statfs":{"total":773094113280,"available":772835741696,"internally_reserved":0,"allocated":2924544,"data_stored":1765884,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":255315505},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.002815"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-03T20:47:37.317266+0000","last_change":"2024-08-03T20:47:35.962230+0000","last_active":"2024-08-03T20:47:37.317266+0000","last_peered":"2024-08-03T20:47:37.317266+0000","last_clean":"2024-08-03T20:47:37.317266+0000","last_became_active":"2024-08-03T20:47:35.958864+0000","last_became_peered":"2024-08-03T20:47:35.958864+0000","last_unstale":"2024-08-03T20:47:37.317266+0000","last_undegraded":"2024-08-03T20:47:37.317266+0000","last_fullsized":"2024-08-03T20:47:37.317266+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_clean_scrub_stamp":"2024-08-03T20:47:34.904132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T03:01:41.602990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-03T20:47:36.935685+0000","last_change":"2024-08-03T20:47:35.941982+0000","last_active":"2024-08-03T20:47:36.935685+0000","last_peered":"2024-08-03T20:47:36.935685+0000","last_clean":"2024-08-03T20:47:36.935685+0000","last_became_active":"2024-08-03T20:47:35.941614+0000","last_became_peered":"2024-08-03T20:47:35.941614+0000","last_unstale":"2024-08-03T20:47:36.935685+0000","last_undegraded":"2024-08-03T20:47:36.935685+0000","last_fullsized":"2024-08-03T20:47:36.935685+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_clean_scrub_stamp":"2024-08-03T20:47:34.904132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T01:08:56.085650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-03T20:47:37.979574+0000","last_change":"2024-08-03T20:47:35.926803+0000","last_active":"2024-08-03T20:47:37.979574+0000","last_peered":"2024-08-03T20:47:37.979574+0000","last_clean":"2024-08-03T20:47:37.979574+0000","last_became_active":"2024-08-03T20:47:35.926198+0000","last_became_peered":"2024-08-03T20:47:35.926198+0000","last_unstale":"2024-08-03T20:47:37.979574+0000","last_undegraded":"2024-08-03T20:47:37.979574+0000","last_fullsized":"2024-08-03T20:47:37.979574+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_clean_scrub_stamp":"2024-08-03T20:47:34.904132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T04:57:58.179012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-03T20:47:37.979134+0000","last_change":"2024-08-03T20:47:35.960607+0000","last_active":"2024-08-03T20:47:37.979134+0000","last_peered":"2024-08-03T20:47:37.979134+0000","last_clean":"2024-08-03T20:47:37.979134+0000","last_became_active":"2024-08-03T20:47:35.956844+0000","last_became_peered":"2024-08-03T20:47:35.956844+0000","last_unstale":"2024-08-03T20:47:37.979134+0000","last_undegraded":"2024-08-03T20:47:37.979134+0000","last_fullsized":"2024-08-03T20:47:37.979134+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_clean_scrub_stamp":"2024-08-03T20:47:34.904132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T07:48:20.679795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-03T20:47:37.316984+0000","last_change":"2024-08-03T20:47:35.962693+0000","last_active":"2024-08-03T20:47:37.316984+0000","last_peered":"2024-08-03T20:47:37.316984+0000","last_clean":"2024-08-03T20:47:37.316984+0000","last_became_active":"2024-08-03T20:47:35.959911+0000","last_became_peered":"2024-08-03T20:47:35.959911+0000","last_unstale":"2024-08-03T20:47:37.316984+0000","last_undegraded":"2024-08-03T20:47:37.316984+0000","last_fullsized":"2024-08-03T20:47:37.316984+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_clean_scrub_stamp":"2024-08-03T20:47:34.904132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T04:51:27.371858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-03T20:47:37.979016+0000","last_change":"2024-08-03T20:47:35.947517+0000","last_active":"2024-08-03T20:47:37.979016+0000","last_peered":"2024-08-03T20:47:37.979016+0000","last_clean":"2024-08-03T20:47:37.979016+0000","last_became_active":"2024-08-03T20:47:35.947167+0000","last_became_peered":"2024-08-03T20:47:35.947167+0000","last_unstale":"2024-08-03T20:47:37.979016+0000","last_undegraded":"2024-08-03T20:47:37.979016+0000","last_fullsized":"2024-08-03T20:47:37.979016+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_clean_scrub_stamp":"2024-08-03T20:47:34.904132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-04T22:33:13.028920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-03T20:47:37.316673+0000","last_change":"2024-08-03T20:47:35.947270+0000","last_active":"2024-08-03T20:47:37.316673+0000","last_peered":"2024-08-03T20:47:37.316673+0000","last_clean":"2024-08-03T20:47:37.316673+0000","last_became_active":"2024-08-03T20:47:35.946647+0000","last_became_peered":"2024-08-03T20:47:35.946647+0000","last_unstale":"2024-08-03T20:47:37.316673+0000","last_undegraded":"2024-08-03T20:47:37.316673+0000","last_fullsized":"2024-08-03T20:47:37.316673+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_clean_scrub_stamp":"2024-08-03T20:47:34.904132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T00:10:43.343156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-03T20:47:37.978113+0000","last_change":"2024-08-03T20:47:35.946701+0000","last_active":"2024-08-03T20:47:37.978113+0000","last_peered":"2024-08-03T20:47:37.978113+0000","last_clean":"2024-08-03T20:47:37.978113+0000","last_became_active":"2024-08-03T20:47:35.946340+0000","last_became_peered":"2024-08-03T20:47:35.946340+0000","last_unstale":"2024-08-03T20:47:37.978113+0000","last_undegraded":"2024-08-03T20:47:37.978113+0000","last_fullsized":"2024-08-03T20:47:37.978113+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_clean_scrub_stamp":"2024-08-03T20:47:34.904132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T07:21:44.279582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-03T20:47:37.317505+0000","last_change":"2024-08-03T20:47:35.963774+0000","last_active":"2024-08-03T20:47:37.317505+0000","last_peered":"2024-08-03T20:47:37.317505+0000","last_clean":"2024-08-03T20:47:37.317505+0000","last_became_active":"2024-08-03T20:47:35.958408+0000","last_became_peered":"2024-08-03T20:47:35.958408+0000","last_unstale":"2024-08-03T20:47:37.317505+0000","last_undegraded":"2024-08-03T20:47:37.317505+0000","last_fullsized":"2024-08-03T20:47:37.317505+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_clean_scrub_stamp":"2024-08-03T20:47:34.904132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T00:58:45.652167+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-03T20:47:36.941791+0000","last_change":"2024-08-03T20:47:35.943147+0000","last_active":"2024-08-03T20:47:36.941791+0000","last_peered":"2024-08-03T20:47:36.941791+0000","last_clean":"2024-08-03T20:47:36.941791+0000","last_became_active":"2024-08-03T20:47:35.942253+0000","last_became_peered":"2024-08-03T20:47:35.942253+0000","last_unstale":"2024-08-03T20:47:36.941791+0000","last_undegraded":"2024-08-03T20:47:36.941791+0000","last_fullsized":"2024-08-03T20:47:36.941791+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_clean_scrub_stamp":"2024-08-03T20:47:34.904132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T01:08:56.085650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-03T20:47:36.941585+0000","last_change":"2024-08-03T20:47:35.938819+0000","last_active":"2024-08-03T20:47:36.941585+0000","last_peered":"2024-08-03T20:47:36.941585+0000","last_clean":"2024-08-03T20:47:36.941585+0000","last_became_active":"2024-08-03T20:47:35.938499+0000","last_became_peered":"2024-08-03T20:47:35.938499+0000","last_unstale":"2024-08-03T20:47:36.941585+0000","last_undegraded":"2024-08-03T20:47:36.941585+0000","last_fullsized":"2024-08-03T20:47:36.941585+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_clean_scrub_stamp":"2024-08-03T20:47:34.904132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T04:57:58.179012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-03T20:47:36.936210+0000","last_change":"2024-08-03T20:47:35.936873+0000","last_active":"2024-08-03T20:47:36.936210+0000","last_peered":"2024-08-03T20:47:36.936210+0000","last_clean":"2024-08-03T20:47:36.936210+0000","last_became_active":"2024-08-03T20:47:35.936005+0000","last_became_peered":"2024-08-03T20:47:35.936005+0000","last_unstale":"2024-08-03T20:47:36.936210+0000","last_undegraded":"2024-08-03T20:47:36.936210+0000","last_fullsized":"2024-08-03T20:47:36.936210+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_clean_scrub_stamp":"2024-08-03T20:47:34.904132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T01:42:26.761950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-03T20:47:37.978994+0000","last_change":"2024-08-03T20:47:35.950146+0000","last_active":"2024-08-03T20:47:37.978994+0000","last_peered":"2024-08-03T20:47:37.978994+0000","last_clean":"2024-08-03T20:47:37.978994+0000","last_became_active":"2024-08-03T20:47:35.949842+0000","last_became_peered":"2024-08-03T20:47:35.949842+0000","last_unstale":"2024-08-03T20:47:37.978994+0000","last_undegraded":"2024-08-03T20:47:37.978994+0000","last_fullsized":"2024-08-03T20:47:37.978994+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_clean_scrub_stamp":"2024-08-03T20:47:34.904132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-04T22:48:57.612773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-03T20:47:37.979037+0000","last_change":"2024-08-03T20:47:35.940356+0000","last_active":"2024-08-03T20:47:37.979037+0000","last_peered":"2024-08-03T20:47:37.979037+0000","last_clean":"2024-08-03T20:47:37.979037+0000","last_became_active":"2024-08-03T20:47:35.940143+0000","last_became_peered":"2024-08-03T20:47:35.940143+0000","last_unstale":"2024-08-03T20:47:37.979037+0000","last_undegraded":"2024-08-03T20:47:37.979037+0000","last_fullsized":"2024-08-03T20:47:37.979037+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_clean_scrub_stamp":"2024-08-03T20:47:34.904132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T04:01:08.373245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-03T20:47:36.935545+0000","last_change":"2024-08-03T20:47:35.951009+0000","last_active":"2024-08-03T20:47:36.935545+0000","last_peered":"2024-08-03T20:47:36.935545+0000","last_clean":"2024-08-03T20:47:36.935545+0000","last_became_active":"2024-08-03T20:47:35.950658+0000","last_became_peered":"2024-08-03T20:47:35.950658+0000","last_unstale":"2024-08-03T20:47:36.935545+0000","last_undegraded":"2024-08-03T20:47:36.935545+0000","last_fullsized":"2024-08-03T20:47:36.935545+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_clean_scrub_stamp":"2024-08-03T20:47:34.904132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-04T20:49:19.998935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-03T20:47:37.978861+0000","last_change":"2024-08-03T20:47:35.948379+0000","last_active":"2024-08-03T20:47:37.978861+0000","last_peered":"2024-08-03T20:47:37.978861+0000","last_clean":"2024-08-03T20:47:37.978861+0000","last_became_active":"2024-08-03T20:47:35.947812+0000","last_became_peered":"2024-08-03T20:47:35.947812+0000","last_unstale":"2024-08-03T20:47:37.978861+0000","last_undegraded":"2024-08-03T20:47:37.978861+0000","last_fullsized":"2024-08-03T20:47:37.978861+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_clean_scrub_stamp":"2024-08-03T20:47:34.904132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T01:08:56.085650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-03T20:47:37.979587+0000","last_change":"2024-08-03T20:47:35.939474+0000","last_active":"2024-08-03T20:47:37.979587+0000","last_peered":"2024-08-03T20:47:37.979587+0000","last_clean":"2024-08-03T20:47:37.979587+0000","last_became_active":"2024-08-03T20:47:35.938914+0000","last_became_peered":"2024-08-03T20:47:35.938914+0000","last_unstale":"2024-08-03T20:47:37.979587+0000","last_undegraded":"2024-08-03T20:47:37.979587+0000","last_fullsized":"2024-08-03T20:47:37.979587+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_clean_scrub_stamp":"2024-08-03T20:47:34.904132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T01:52:59.081172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-03T20:47:37.352899+0000","last_change":"2024-08-03T20:47:35.933829+0000","last_active":"2024-08-03T20:47:37.352899+0000","last_peered":"2024-08-03T20:47:37.352899+0000","last_clean":"2024-08-03T20:47:37.352899+0000","last_became_active":"2024-08-03T20:47:35.933515+0000","last_became_peered":"2024-08-03T20:47:35.933515+0000","last_unstale":"2024-08-03T20:47:37.352899+0000","last_undegraded":"2024-08-03T20:47:37.352899+0000","last_fullsized":"2024-08-03T20:47:37.352899+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_clean_scrub_stamp":"2024-08-03T20:47:34.904132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T02:10:57.293691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-03T20:47:37.317301+0000","last_change":"2024-08-03T20:47:35.952999+0000","last_active":"2024-08-03T20:47:37.317301+0000","last_peered":"2024-08-03T20:47:37.317301+0000","last_clean":"2024-08-03T20:47:37.317301+0000","last_became_active":"2024-08-03T20:47:35.952281+0000","last_became_peered":"2024-08-03T20:47:35.952281+0000","last_unstale":"2024-08-03T20:47:37.317301+0000","last_undegraded":"2024-08-03T20:47:37.317301+0000","last_fullsized":"2024-08-03T20:47:37.317301+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_clean_scrub_stamp":"2024-08-03T20:47:34.904132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T01:15:48.374663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-03T20:47:36.941659+0000","last_change":"2024-08-03T20:47:35.927412+0000","last_active":"2024-08-03T20:47:36.941659+0000","last_peered":"2024-08-03T20:47:36.941659+0000","last_clean":"2024-08-03T20:47:36.941659+0000","last_became_active":"2024-08-03T20:47:35.927069+0000","last_became_peered":"2024-08-03T20:47:35.927069+0000","last_unstale":"2024-08-03T20:47:36.941659+0000","last_undegraded":"2024-08-03T20:47:36.941659+0000","last_fullsized":"2024-08-03T20:47:36.941659+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_clean_scrub_stamp":"2024-08-03T20:47:34.904132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-04T20:49:19.998935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-03T20:47:37.979382+0000","last_change":"2024-08-03T20:47:35.936665+0000","last_active":"2024-08-03T20:47:37.979382+0000","last_peered":"2024-08-03T20:47:37.979382+0000","last_clean":"2024-08-03T20:47:37.979382+0000","last_became_active":"2024-08-03T20:47:35.936298+0000","last_became_peered":"2024-08-03T20:47:35.936298+0000","last_unstale":"2024-08-03T20:47:37.979382+0000","last_undegraded":"2024-08-03T20:47:37.979382+0000","last_fullsized":"2024-08-03T20:47:37.979382+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_clean_scrub_stamp":"2024-08-03T20:47:34.904132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-04T22:12:15.293757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-03T20:47:37.979017+0000","last_change":"2024-08-03T20:47:35.938468+0000","last_active":"2024-08-03T20:47:37.979017+0000","last_peered":"2024-08-03T20:47:37.979017+0000","last_clean":"2024-08-03T20:47:37.979017+0000","last_became_active":"2024-08-03T20:47:35.937757+0000","last_became_peered":"2024-08-03T20:47:35.937757+0000","last_unstale":"2024-08-03T20:47:37.979017+0000","last_undegraded":"2024-08-03T20:47:37.979017+0000","last_fullsized":"2024-08-03T20:47:37.979017+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_clean_scrub_stamp":"2024-08-03T20:47:34.904132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-04T23:07:18.318169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-03T20:47:37.978241+0000","last_change":"2024-08-03T20:47:35.953502+0000","last_active":"2024-08-03T20:47:37.978241+0000","last_peered":"2024-08-03T20:47:37.978241+0000","last_clean":"2024-08-03T20:47:37.978241+0000","last_became_active":"2024-08-03T20:47:35.953152+0000","last_became_peered":"2024-08-03T20:47:35.953152+0000","last_unstale":"2024-08-03T20:47:37.978241+0000","last_undegraded":"2024-08-03T20:47:37.978241+0000","last_fullsized":"2024-08-03T20:47:37.978241+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_clean_scrub_stamp":"2024-08-03T20:47:34.904132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T01:56:58.984458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-03T20:47:37.979235+0000","last_change":"2024-08-03T20:47:35.937927+0000","last_active":"2024-08-03T20:47:37.979235+0000","last_peered":"2024-08-03T20:47:37.979235+0000","last_clean":"2024-08-03T20:47:37.979235+0000","last_became_active":"2024-08-03T20:47:35.937579+0000","last_became_peered":"2024-08-03T20:47:35.937579+0000","last_unstale":"2024-08-03T20:47:37.979235+0000","last_undegraded":"2024-08-03T20:47:37.979235+0000","last_fullsized":"2024-08-03T20:47:37.979235+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_clean_scrub_stamp":"2024-08-03T20:47:34.904132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T03:44:29.605859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-03T20:47:37.979020+0000","last_change":"2024-08-03T20:47:35.953835+0000","last_active":"2024-08-03T20:47:37.979020+0000","last_peered":"2024-08-03T20:47:37.979020+0000","last_clean":"2024-08-03T20:47:37.979020+0000","last_became_active":"2024-08-03T20:47:35.953520+0000","last_became_peered":"2024-08-03T20:47:35.953520+0000","last_unstale":"2024-08-03T20:47:37.979020+0000","last_undegraded":"2024-08-03T20:47:37.979020+0000","last_fullsized":"2024-08-03T20:47:37.979020+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_clean_scrub_stamp":"2024-08-03T20:47:34.904132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T06:11:09.513212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-03T20:47:36.941856+0000","last_change":"2024-08-03T20:47:35.938228+0000","last_active":"2024-08-03T20:47:36.941856+0000","last_peered":"2024-08-03T20:47:36.941856+0000","last_clean":"2024-08-03T20:47:36.941856+0000","last_became_active":"2024-08-03T20:47:35.937893+0000","last_became_peered":"2024-08-03T20:47:35.937893+0000","last_unstale":"2024-08-03T20:47:36.941856+0000","last_undegraded":"2024-08-03T20:47:36.941856+0000","last_fullsized":"2024-08-03T20:47:36.941856+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_clean_scrub_stamp":"2024-08-03T20:47:34.904132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-04T22:33:13.028920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-03T20:47:36.935344+0000","last_change":"2024-08-03T20:47:35.937679+0000","last_active":"2024-08-03T20:47:36.935344+0000","last_peered":"2024-08-03T20:47:36.935344+0000","last_clean":"2024-08-03T20:47:36.935344+0000","last_became_active":"2024-08-03T20:47:35.937328+0000","last_became_peered":"2024-08-03T20:47:35.937328+0000","last_unstale":"2024-08-03T20:47:36.935344+0000","last_undegraded":"2024-08-03T20:47:36.935344+0000","last_fullsized":"2024-08-03T20:47:36.935344+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_clean_scrub_stamp":"2024-08-03T20:47:34.904132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T04:57:58.179012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-03T20:47:37.978822+0000","last_change":"2024-08-03T20:47:35.954328+0000","last_active":"2024-08-03T20:47:37.978822+0000","last_peered":"2024-08-03T20:47:37.978822+0000","last_clean":"2024-08-03T20:47:37.978822+0000","last_became_active":"2024-08-03T20:47:35.951285+0000","last_became_peered":"2024-08-03T20:47:35.951285+0000","last_unstale":"2024-08-03T20:47:37.978822+0000","last_undegraded":"2024-08-03T20:47:37.978822+0000","last_fullsized":"2024-08-03T20:47:37.978822+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_clean_scrub_stamp":"2024-08-03T20:47:34.904132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T01:01:23.625299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-03T20:47:37.979494+0000","last_change":"2024-08-03T20:47:35.940504+0000","last_active":"2024-08-03T20:47:37.979494+0000","last_peered":"2024-08-03T20:47:37.979494+0000","last_clean":"2024-08-03T20:47:37.979494+0000","last_became_active":"2024-08-03T20:47:35.938638+0000","last_became_peered":"2024-08-03T20:47:35.938638+0000","last_unstale":"2024-08-03T20:47:37.979494+0000","last_undegraded":"2024-08-03T20:47:37.979494+0000","last_fullsized":"2024-08-03T20:47:37.979494+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_clean_scrub_stamp":"2024-08-03T20:47:34.904132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-04T23:57:18.840519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-03T20:47:37.979441+0000","last_change":"2024-08-03T20:47:35.938131+0000","last_active":"2024-08-03T20:47:37.979441+0000","last_peered":"2024-08-03T20:47:37.979441+0000","last_clean":"2024-08-03T20:47:37.979441+0000","last_became_active":"2024-08-03T20:47:35.937270+0000","last_became_peered":"2024-08-03T20:47:35.937270+0000","last_unstale":"2024-08-03T20:47:37.979441+0000","last_undegraded":"2024-08-03T20:47:37.979441+0000","last_fullsized":"2024-08-03T20:47:37.979441+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_clean_scrub_stamp":"2024-08-03T20:47:34.904132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-04T21:57:09.430415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-03T20:47:37.352726+0000","last_change":"2024-08-03T20:47:35.937957+0000","last_active":"2024-08-03T20:47:37.352726+0000","last_peered":"2024-08-03T20:47:37.352726+0000","last_clean":"2024-08-03T20:47:37.352726+0000","last_became_active":"2024-08-03T20:47:35.937656+0000","last_became_peered":"2024-08-03T20:47:35.937656+0000","last_unstale":"2024-08-03T20:47:37.352726+0000","last_undegraded":"2024-08-03T20:47:37.352726+0000","last_fullsized":"2024-08-03T20:47:37.352726+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_clean_scrub_stamp":"2024-08-03T20:47:34.904132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T05:51:29.523373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-03T20:47:36.941128+0000","last_change":"2024-08-03T20:47:35.932459+0000","last_active":"2024-08-03T20:47:36.941128+0000","last_peered":"2024-08-03T20:47:36.941128+0000","last_clean":"2024-08-03T20:47:36.941128+0000","last_became_active":"2024-08-03T20:47:35.932103+0000","last_became_peered":"2024-08-03T20:47:35.932103+0000","last_unstale":"2024-08-03T20:47:36.941128+0000","last_undegraded":"2024-08-03T20:47:36.941128+0000","last_fullsized":"2024-08-03T20:47:36.941128+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_clean_scrub_stamp":"2024-08-03T20:47:34.904132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T08:34:04.057775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-03T20:47:36.935757+0000","last_change":"2024-08-03T20:47:35.940544+0000","last_active":"2024-08-03T20:47:36.935757+0000","last_peered":"2024-08-03T20:47:36.935757+0000","last_clean":"2024-08-03T20:47:36.935757+0000","last_became_active":"2024-08-03T20:47:35.940218+0000","last_became_peered":"2024-08-03T20:47:35.940218+0000","last_unstale":"2024-08-03T20:47:36.935757+0000","last_undegraded":"2024-08-03T20:47:36.935757+0000","last_fullsized":"2024-08-03T20:47:36.935757+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_clean_scrub_stamp":"2024-08-03T20:47:34.904132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T07:58:42.869702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"20'1","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-03T20:47:37.978814+0000","last_change":"2024-08-03T20:47:34.939881+0000","last_active":"2024-08-03T20:47:37.978814+0000","last_peered":"2024-08-03T20:47:37.978814+0000","last_clean":"2024-08-03T20:47:37.978814+0000","last_became_active":"2024-08-03T20:47:34.936765+0000","last_became_peered":"2024-08-03T20:47:34.936765+0000","last_unstale":"2024-08-03T20:47:37.978814+0000","last_undegraded":"2024-08-03T20:47:37.978814+0000","last_fullsized":"2024-08-03T20:47:37.978814+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:47:33.895731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:47:33.895731+0000","last_clean_scrub_stamp":"2024-08-03T20:47:33.895731+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T02:29:41.523344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-03T20:47:37.979599+0000","last_change":"2024-08-03T20:47:35.942088+0000","last_active":"2024-08-03T20:47:37.979599+0000","last_peered":"2024-08-03T20:47:37.979599+0000","last_clean":"2024-08-03T20:47:37.979599+0000","last_became_active":"2024-08-03T20:47:35.941800+0000","last_became_peered":"2024-08-03T20:47:35.941800+0000","last_unstale":"2024-08-03T20:47:37.979599+0000","last_undegraded":"2024-08-03T20:47:37.979599+0000","last_fullsized":"2024-08-03T20:47:37.979599+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_clean_scrub_stamp":"2024-08-03T20:47:34.904132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-04T20:49:19.998935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"20'1","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-03T20:47:36.941066+0000","last_change":"2024-08-03T20:47:34.938348+0000","last_active":"2024-08-03T20:47:36.941066+0000","last_peered":"2024-08-03T20:47:36.941066+0000","last_clean":"2024-08-03T20:47:36.941066+0000","last_became_active":"2024-08-03T20:47:34.936169+0000","last_became_peered":"2024-08-03T20:47:34.936169+0000","last_unstale":"2024-08-03T20:47:36.941066+0000","last_undegraded":"2024-08-03T20:47:36.941066+0000","last_fullsized":"2024-08-03T20:47:36.941066+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:47:33.895731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:47:33.895731+0000","last_clean_scrub_stamp":"2024-08-03T20:47:33.895731+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T06:44:37.944928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-03T20:47:37.978344+0000","last_change":"2024-08-03T20:47:34.935991+0000","last_active":"2024-08-03T20:47:37.978344+0000","last_peered":"2024-08-03T20:47:37.978344+0000","last_clean":"2024-08-03T20:47:37.978344+0000","last_became_active":"2024-08-03T20:47:34.930541+0000","last_became_peered":"2024-08-03T20:47:34.930541+0000","last_unstale":"2024-08-03T20:47:37.978344+0000","last_undegraded":"2024-08-03T20:47:37.978344+0000","last_fullsized":"2024-08-03T20:47:37.978344+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:47:33.895731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:47:33.895731+0000","last_clean_scrub_stamp":"2024-08-03T20:47:33.895731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T04:37:55.336635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-03T20:47:37.978164+0000","last_change":"2024-08-03T20:47:35.934722+0000","last_active":"2024-08-03T20:47:37.978164+0000","last_peered":"2024-08-03T20:47:37.978164+0000","last_clean":"2024-08-03T20:47:37.978164+0000","last_became_active":"2024-08-03T20:47:35.934183+0000","last_became_peered":"2024-08-03T20:47:35.934183+0000","last_unstale":"2024-08-03T20:47:37.978164+0000","last_undegraded":"2024-08-03T20:47:37.978164+0000","last_fullsized":"2024-08-03T20:47:37.978164+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_clean_scrub_stamp":"2024-08-03T20:47:34.904132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T06:29:11.740575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-03T20:47:37.978945+0000","last_change":"2024-08-03T20:47:34.934591+0000","last_active":"2024-08-03T20:47:37.978945+0000","last_peered":"2024-08-03T20:47:37.978945+0000","last_clean":"2024-08-03T20:47:37.978945+0000","last_became_active":"2024-08-03T20:47:34.934155+0000","last_became_peered":"2024-08-03T20:47:34.934155+0000","last_unstale":"2024-08-03T20:47:37.978945+0000","last_undegraded":"2024-08-03T20:47:37.978945+0000","last_fullsized":"2024-08-03T20:47:37.978945+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:47:33.895731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:47:33.895731+0000","last_clean_scrub_stamp":"2024-08-03T20:47:33.895731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-04T22:30:28.119577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-03T20:47:36.934971+0000","last_change":"2024-08-03T20:47:35.947874+0000","last_active":"2024-08-03T20:47:36.934971+0000","last_peered":"2024-08-03T20:47:36.934971+0000","last_clean":"2024-08-03T20:47:36.934971+0000","last_became_active":"2024-08-03T20:47:35.947570+0000","last_became_peered":"2024-08-03T20:47:35.947570+0000","last_unstale":"2024-08-03T20:47:36.934971+0000","last_undegraded":"2024-08-03T20:47:36.934971+0000","last_fullsized":"2024-08-03T20:47:36.934971+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_clean_scrub_stamp":"2024-08-03T20:47:34.904132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-04T22:12:15.293757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-03T20:47:36.941786+0000","last_change":"2024-08-03T20:47:34.937955+0000","last_active":"2024-08-03T20:47:36.941786+0000","last_peered":"2024-08-03T20:47:36.941786+0000","last_clean":"2024-08-03T20:47:36.941786+0000","last_became_active":"2024-08-03T20:47:34.937613+0000","last_became_peered":"2024-08-03T20:47:34.937613+0000","last_unstale":"2024-08-03T20:47:36.941786+0000","last_undegraded":"2024-08-03T20:47:36.941786+0000","last_fullsized":"2024-08-03T20:47:36.941786+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:47:33.895731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:47:33.895731+0000","last_clean_scrub_stamp":"2024-08-03T20:47:33.895731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-04T22:30:28.119577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-03T20:47:37.979527+0000","last_change":"2024-08-03T20:47:35.941948+0000","last_active":"2024-08-03T20:47:37.979527+0000","last_peered":"2024-08-03T20:47:37.979527+0000","last_clean":"2024-08-03T20:47:37.979527+0000","last_became_active":"2024-08-03T20:47:35.941651+0000","last_became_peered":"2024-08-03T20:47:35.941651+0000","last_unstale":"2024-08-03T20:47:37.979527+0000","last_undegraded":"2024-08-03T20:47:37.979527+0000","last_fullsized":"2024-08-03T20:47:37.979527+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_clean_scrub_stamp":"2024-08-03T20:47:34.904132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T06:29:37.397869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-03T20:47:37.979267+0000","last_change":"2024-08-03T20:47:34.936965+0000","last_active":"2024-08-03T20:47:37.979267+0000","last_peered":"2024-08-03T20:47:37.979267+0000","last_clean":"2024-08-03T20:47:37.979267+0000","last_became_active":"2024-08-03T20:47:34.936667+0000","last_became_peered":"2024-08-03T20:47:34.936667+0000","last_unstale":"2024-08-03T20:47:37.979267+0000","last_undegraded":"2024-08-03T20:47:37.979267+0000","last_fullsized":"2024-08-03T20:47:37.979267+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:47:33.895731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:47:33.895731+0000","last_clean_scrub_stamp":"2024-08-03T20:47:33.895731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T07:46:07.162679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-03T20:47:37.979788+0000","last_change":"2024-08-03T20:47:35.940917+0000","last_active":"2024-08-03T20:47:37.979788+0000","last_peered":"2024-08-03T20:47:37.979788+0000","last_clean":"2024-08-03T20:47:37.979788+0000","last_became_active":"2024-08-03T20:47:35.940218+0000","last_became_peered":"2024-08-03T20:47:35.940218+0000","last_unstale":"2024-08-03T20:47:37.979788+0000","last_undegraded":"2024-08-03T20:47:37.979788+0000","last_fullsized":"2024-08-03T20:47:37.979788+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_clean_scrub_stamp":"2024-08-03T20:47:34.904132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-04T23:07:18.318169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"20'3","reported_seq":18,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-03T20:47:36.935425+0000","last_change":"2024-08-03T20:47:34.939100+0000","last_active":"2024-08-03T20:47:36.935425+0000","last_peered":"2024-08-03T20:47:36.935425+0000","last_clean":"2024-08-03T20:47:36.935425+0000","last_became_active":"2024-08-03T20:47:34.935869+0000","last_became_peered":"2024-08-03T20:47:34.935869+0000","last_unstale":"2024-08-03T20:47:36.935425+0000","last_undegraded":"2024-08-03T20:47:36.935425+0000","last_fullsized":"2024-08-03T20:47:36.935425+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:47:33.895731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:47:33.895731+0000","last_clean_scrub_stamp":"2024-08-03T20:47:33.895731+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T06:44:37.944928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-03T20:47:36.936329+0000","last_change":"2024-08-03T20:47:35.944428+0000","last_active":"2024-08-03T20:47:36.936329+0000","last_peered":"2024-08-03T20:47:36.936329+0000","last_clean":"2024-08-03T20:47:36.936329+0000","last_became_active":"2024-08-03T20:47:35.943608+0000","last_became_peered":"2024-08-03T20:47:35.943608+0000","last_unstale":"2024-08-03T20:47:36.936329+0000","last_undegraded":"2024-08-03T20:47:36.936329+0000","last_fullsized":"2024-08-03T20:47:36.936329+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_clean_scrub_stamp":"2024-08-03T20:47:34.904132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T05:51:29.523373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"20'6","reported_seq":21,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-03T20:47:36.941564+0000","last_change":"2024-08-03T20:47:34.931832+0000","last_active":"2024-08-03T20:47:36.941564+0000","last_peered":"2024-08-03T20:47:36.941564+0000","last_clean":"2024-08-03T20:47:36.941564+0000","last_became_active":"2024-08-03T20:47:34.931150+0000","last_became_peered":"2024-08-03T20:47:34.931150+0000","last_unstale":"2024-08-03T20:47:36.941564+0000","last_undegraded":"2024-08-03T20:47:36.941564+0000","last_fullsized":"2024-08-03T20:47:36.941564+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:47:33.895731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:47:33.895731+0000","last_clean_scrub_stamp":"2024-08-03T20:47:33.895731+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T05:01:48.922382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-03T20:47:37.316866+0000","last_change":"2024-08-03T20:47:35.953367+0000","last_active":"2024-08-03T20:47:37.316866+0000","last_peered":"2024-08-03T20:47:37.316866+0000","last_clean":"2024-08-03T20:47:37.316866+0000","last_became_active":"2024-08-03T20:47:35.952666+0000","last_became_peered":"2024-08-03T20:47:35.952666+0000","last_unstale":"2024-08-03T20:47:37.316866+0000","last_undegraded":"2024-08-03T20:47:37.316866+0000","last_fullsized":"2024-08-03T20:47:37.316866+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_clean_scrub_stamp":"2024-08-03T20:47:34.904132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-04T21:57:09.430415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"20'2","reported_seq":17,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-03T20:47:37.979652+0000","last_change":"2024-08-03T20:47:34.932411+0000","last_active":"2024-08-03T20:47:37.979652+0000","last_peered":"2024-08-03T20:47:37.979652+0000","last_clean":"2024-08-03T20:47:37.979652+0000","last_became_active":"2024-08-03T20:47:34.931336+0000","last_became_peered":"2024-08-03T20:47:34.931336+0000","last_unstale":"2024-08-03T20:47:37.979652+0000","last_undegraded":"2024-08-03T20:47:37.979652+0000","last_fullsized":"2024-08-03T20:47:37.979652+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:47:33.895731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:47:33.895731+0000","last_clean_scrub_stamp":"2024-08-03T20:47:33.895731+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T05:59:33.076657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-03T20:47:37.979070+0000","last_change":"2024-08-03T20:47:35.932875+0000","last_active":"2024-08-03T20:47:37.979070+0000","last_peered":"2024-08-03T20:47:37.979070+0000","last_clean":"2024-08-03T20:47:37.979070+0000","last_became_active":"2024-08-03T20:47:35.932516+0000","last_became_peered":"2024-08-03T20:47:35.932516+0000","last_unstale":"2024-08-03T20:47:37.979070+0000","last_undegraded":"2024-08-03T20:47:37.979070+0000","last_fullsized":"2024-08-03T20:47:37.979070+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_clean_scrub_stamp":"2024-08-03T20:47:34.904132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T08:34:04.057775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-03T20:47:37.979194+0000","last_change":"2024-08-03T20:47:34.947236+0000","last_active":"2024-08-03T20:47:37.979194+0000","last_peered":"2024-08-03T20:47:37.979194+0000","last_clean":"2024-08-03T20:47:37.979194+0000","last_became_active":"2024-08-03T20:47:34.946894+0000","last_became_peered":"2024-08-03T20:47:34.946894+0000","last_unstale":"2024-08-03T20:47:37.979194+0000","last_undegraded":"2024-08-03T20:47:37.979194+0000","last_fullsized":"2024-08-03T20:47:37.979194+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:47:33.895731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:47:33.895731+0000","last_clean_scrub_stamp":"2024-08-03T20:47:33.895731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T05:01:48.922382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-03T20:47:37.317004+0000","last_change":"2024-08-03T20:47:35.950815+0000","last_active":"2024-08-03T20:47:37.317004+0000","last_peered":"2024-08-03T20:47:37.317004+0000","last_clean":"2024-08-03T20:47:37.317004+0000","last_became_active":"2024-08-03T20:47:35.950494+0000","last_became_peered":"2024-08-03T20:47:35.950494+0000","last_unstale":"2024-08-03T20:47:37.317004+0000","last_undegraded":"2024-08-03T20:47:37.317004+0000","last_fullsized":"2024-08-03T20:47:37.317004+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_clean_scrub_stamp":"2024-08-03T20:47:34.904132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T05:49:42.422083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-03T20:47:36.941067+0000","last_change":"2024-08-03T20:47:34.939132+0000","last_active":"2024-08-03T20:47:36.941067+0000","last_peered":"2024-08-03T20:47:36.941067+0000","last_clean":"2024-08-03T20:47:36.941067+0000","last_became_active":"2024-08-03T20:47:34.937563+0000","last_became_peered":"2024-08-03T20:47:34.937563+0000","last_unstale":"2024-08-03T20:47:36.941067+0000","last_undegraded":"2024-08-03T20:47:36.941067+0000","last_fullsized":"2024-08-03T20:47:36.941067+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:47:33.895731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:47:33.895731+0000","last_clean_scrub_stamp":"2024-08-03T20:47:33.895731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T06:32:14.208950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-03T20:47:36.941315+0000","last_change":"2024-08-03T20:47:35.944827+0000","last_active":"2024-08-03T20:47:36.941315+0000","last_peered":"2024-08-03T20:47:36.941315+0000","last_clean":"2024-08-03T20:47:36.941315+0000","last_became_active":"2024-08-03T20:47:35.944517+0000","last_became_peered":"2024-08-03T20:47:35.944517+0000","last_unstale":"2024-08-03T20:47:36.941315+0000","last_undegraded":"2024-08-03T20:47:36.941315+0000","last_fullsized":"2024-08-03T20:47:36.941315+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_clean_scrub_stamp":"2024-08-03T20:47:34.904132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-04T23:57:18.840519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"20'8","reported_seq":23,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-03T20:47:37.352795+0000","last_change":"2024-08-03T20:47:34.936194+0000","last_active":"2024-08-03T20:47:37.352795+0000","last_peered":"2024-08-03T20:47:37.352795+0000","last_clean":"2024-08-03T20:47:37.352795+0000","last_became_active":"2024-08-03T20:47:34.935835+0000","last_became_peered":"2024-08-03T20:47:34.935835+0000","last_unstale":"2024-08-03T20:47:37.352795+0000","last_undegraded":"2024-08-03T20:47:37.352795+0000","last_fullsized":"2024-08-03T20:47:37.352795+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:47:33.895731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:47:33.895731+0000","last_clean_scrub_stamp":"2024-08-03T20:47:33.895731+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T06:13:41.904985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-03T20:47:37.316755+0000","last_change":"2024-08-03T20:47:35.950421+0000","last_active":"2024-08-03T20:47:37.316755+0000","last_peered":"2024-08-03T20:47:37.316755+0000","last_clean":"2024-08-03T20:47:37.316755+0000","last_became_active":"2024-08-03T20:47:35.950109+0000","last_became_peered":"2024-08-03T20:47:35.950109+0000","last_unstale":"2024-08-03T20:47:37.316755+0000","last_undegraded":"2024-08-03T20:47:37.316755+0000","last_fullsized":"2024-08-03T20:47:37.316755+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_clean_scrub_stamp":"2024-08-03T20:47:34.904132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T02:17:30.747723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"20'1","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-03T20:47:36.936564+0000","last_change":"2024-08-03T20:47:34.935417+0000","last_active":"2024-08-03T20:47:36.936564+0000","last_peered":"2024-08-03T20:47:36.936564+0000","last_clean":"2024-08-03T20:47:36.936564+0000","last_became_active":"2024-08-03T20:47:34.932211+0000","last_became_peered":"2024-08-03T20:47:34.932211+0000","last_unstale":"2024-08-03T20:47:36.936564+0000","last_undegraded":"2024-08-03T20:47:36.936564+0000","last_fullsized":"2024-08-03T20:47:36.936564+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:47:33.895731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:47:33.895731+0000","last_clean_scrub_stamp":"2024-08-03T20:47:33.895731+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T04:43:47.155950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-03T20:47:37.317197+0000","last_change":"2024-08-03T20:47:35.958660+0000","last_active":"2024-08-03T20:47:37.317197+0000","last_peered":"2024-08-03T20:47:37.317197+0000","last_clean":"2024-08-03T20:47:37.317197+0000","last_became_active":"2024-08-03T20:47:35.958296+0000","last_became_peered":"2024-08-03T20:47:35.958296+0000","last_unstale":"2024-08-03T20:47:37.317197+0000","last_undegraded":"2024-08-03T20:47:37.317197+0000","last_fullsized":"2024-08-03T20:47:37.317197+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_clean_scrub_stamp":"2024-08-03T20:47:34.904132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T01:52:59.081172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"20'2","reported_seq":17,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-03T20:47:37.352928+0000","last_change":"2024-08-03T20:47:34.934762+0000","last_active":"2024-08-03T20:47:37.352928+0000","last_peered":"2024-08-03T20:47:37.352928+0000","last_clean":"2024-08-03T20:47:37.352928+0000","last_became_active":"2024-08-03T20:47:34.933194+0000","last_became_peered":"2024-08-03T20:47:34.933194+0000","last_unstale":"2024-08-03T20:47:37.352928+0000","last_undegraded":"2024-08-03T20:47:37.352928+0000","last_fullsized":"2024-08-03T20:47:37.352928+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:47:33.895731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:47:33.895731+0000","last_clean_scrub_stamp":"2024-08-03T20:47:33.895731+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T07:58:41.861301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-03T20:47:37.317479+0000","last_change":"2024-08-03T20:47:35.952102+0000","last_active":"2024-08-03T20:47:37.317479+0000","last_peered":"2024-08-03T20:47:37.317479+0000","last_clean":"2024-08-03T20:47:37.317479+0000","last_became_active":"2024-08-03T20:47:35.951350+0000","last_became_peered":"2024-08-03T20:47:35.951350+0000","last_unstale":"2024-08-03T20:47:37.317479+0000","last_undegraded":"2024-08-03T20:47:37.317479+0000","last_fullsized":"2024-08-03T20:47:37.317479+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_clean_scrub_stamp":"2024-08-03T20:47:34.904132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-04T23:37:39.394391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"20'1","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-03T20:47:36.935542+0000","last_change":"2024-08-03T20:47:34.932568+0000","last_active":"2024-08-03T20:47:36.935542+0000","last_peered":"2024-08-03T20:47:36.935542+0000","last_clean":"2024-08-03T20:47:36.935542+0000","last_became_active":"2024-08-03T20:47:34.932223+0000","last_became_peered":"2024-08-03T20:47:34.932223+0000","last_unstale":"2024-08-03T20:47:36.935542+0000","last_undegraded":"2024-08-03T20:47:36.935542+0000","last_fullsized":"2024-08-03T20:47:36.935542+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:47:33.895731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:47:33.895731+0000","last_clean_scrub_stamp":"2024-08-03T20:47:33.895731+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T05:01:48.922382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-03T20:47:37.979523+0000","last_change":"2024-08-03T20:47:35.957384+0000","last_active":"2024-08-03T20:47:37.979523+0000","last_peered":"2024-08-03T20:47:37.979523+0000","last_clean":"2024-08-03T20:47:37.979523+0000","last_became_active":"2024-08-03T20:47:35.956464+0000","last_became_peered":"2024-08-03T20:47:35.956464+0000","last_unstale":"2024-08-03T20:47:37.979523+0000","last_undegraded":"2024-08-03T20:47:37.979523+0000","last_fullsized":"2024-08-03T20:47:37.979523+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_clean_scrub_stamp":"2024-08-03T20:47:34.904132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T05:49:42.422083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-03T20:47:37.352724+0000","last_change":"2024-08-03T20:47:34.934154+0000","last_active":"2024-08-03T20:47:37.352724+0000","last_peered":"2024-08-03T20:47:37.352724+0000","last_clean":"2024-08-03T20:47:37.352724+0000","last_became_active":"2024-08-03T20:47:34.933257+0000","last_became_peered":"2024-08-03T20:47:34.933257+0000","last_unstale":"2024-08-03T20:47:37.352724+0000","last_undegraded":"2024-08-03T20:47:37.352724+0000","last_fullsized":"2024-08-03T20:47:37.352724+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:47:33.895731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:47:33.895731+0000","last_clean_scrub_stamp":"2024-08-03T20:47:33.895731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-04T21:10:14.859583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-03T20:47:37.317012+0000","last_change":"2024-08-03T20:47:35.957705+0000","last_active":"2024-08-03T20:47:37.317012+0000","last_peered":"2024-08-03T20:47:37.317012+0000","last_clean":"2024-08-03T20:47:37.317012+0000","last_became_active":"2024-08-03T20:47:35.951141+0000","last_became_peered":"2024-08-03T20:47:35.951141+0000","last_unstale":"2024-08-03T20:47:37.317012+0000","last_undegraded":"2024-08-03T20:47:37.317012+0000","last_fullsized":"2024-08-03T20:47:37.317012+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_clean_scrub_stamp":"2024-08-03T20:47:34.904132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T06:29:37.397869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"16'72","reported_seq":111,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-03T20:47:37.316600+0000","last_change":"2024-08-03T20:47:30.547632+0000","last_active":"2024-08-03T20:47:37.316600+0000","last_peered":"2024-08-03T20:47:37.316600+0000","last_clean":"2024-08-03T20:47:37.316600+0000","last_became_active":"2024-08-03T20:47:30.546255+0000","last_became_peered":"2024-08-03T20:47:30.546255+0000","last_unstale":"2024-08-03T20:47:37.316600+0000","last_undegraded":"2024-08-03T20:47:37.316600+0000","last_fullsized":"2024-08-03T20:47:37.316600+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:47:28.874451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:47:28.874451+0000","last_clean_scrub_stamp":"2024-08-03T20:47:28.874451+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T00:19:40.548812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-03T20:47:37.978786+0000","last_change":"2024-08-03T20:47:34.941906+0000","last_active":"2024-08-03T20:47:37.978786+0000","last_peered":"2024-08-03T20:47:37.978786+0000","last_clean":"2024-08-03T20:47:37.978786+0000","last_became_active":"2024-08-03T20:47:34.941492+0000","last_became_peered":"2024-08-03T20:47:34.941492+0000","last_unstale":"2024-08-03T20:47:37.978786+0000","last_undegraded":"2024-08-03T20:47:37.978786+0000","last_fullsized":"2024-08-03T20:47:37.978786+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:47:33.895731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:47:33.895731+0000","last_clean_scrub_stamp":"2024-08-03T20:47:33.895731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T06:32:14.208950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-03T20:47:37.978467+0000","last_change":"2024-08-03T20:47:35.953945+0000","last_active":"2024-08-03T20:47:37.978467+0000","last_peered":"2024-08-03T20:47:37.978467+0000","last_clean":"2024-08-03T20:47:37.978467+0000","last_became_active":"2024-08-03T20:47:35.953580+0000","last_became_peered":"2024-08-03T20:47:35.953580+0000","last_unstale":"2024-08-03T20:47:37.978467+0000","last_undegraded":"2024-08-03T20:47:37.978467+0000","last_fullsized":"2024-08-03T20:47:37.978467+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_clean_scrub_stamp":"2024-08-03T20:47:34.904132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T03:08:26.700491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-03T20:47:37.316444+0000","last_change":"2024-08-03T20:47:34.944839+0000","last_active":"2024-08-03T20:47:37.316444+0000","last_peered":"2024-08-03T20:47:37.316444+0000","last_clean":"2024-08-03T20:47:37.316444+0000","last_became_active":"2024-08-03T20:47:34.944530+0000","last_became_peered":"2024-08-03T20:47:34.944530+0000","last_unstale":"2024-08-03T20:47:37.316444+0000","last_undegraded":"2024-08-03T20:47:37.316444+0000","last_fullsized":"2024-08-03T20:47:37.316444+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:47:33.895731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:47:33.895731+0000","last_clean_scrub_stamp":"2024-08-03T20:47:33.895731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T02:40:00.976532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-03T20:47:37.978347+0000","last_change":"2024-08-03T20:47:35.926904+0000","last_active":"2024-08-03T20:47:37.978347+0000","last_peered":"2024-08-03T20:47:37.978347+0000","last_clean":"2024-08-03T20:47:37.978347+0000","last_became_active":"2024-08-03T20:47:35.926525+0000","last_became_peered":"2024-08-03T20:47:35.926525+0000","last_unstale":"2024-08-03T20:47:37.978347+0000","last_undegraded":"2024-08-03T20:47:37.978347+0000","last_fullsized":"2024-08-03T20:47:37.978347+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_clean_scrub_stamp":"2024-08-03T20:47:34.904132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T05:57:44.170587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-03T20:47:37.352725+0000","last_change":"2024-08-03T20:47:34.939125+0000","last_active":"2024-08-03T20:47:37.352725+0000","last_peered":"2024-08-03T20:47:37.352725+0000","last_clean":"2024-08-03T20:47:37.352725+0000","last_became_active":"2024-08-03T20:47:34.935161+0000","last_became_peered":"2024-08-03T20:47:34.935161+0000","last_unstale":"2024-08-03T20:47:37.352725+0000","last_undegraded":"2024-08-03T20:47:37.352725+0000","last_fullsized":"2024-08-03T20:47:37.352725+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:47:33.895731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:47:33.895731+0000","last_clean_scrub_stamp":"2024-08-03T20:47:33.895731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T04:37:55.336635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-03T20:47:36.934972+0000","last_change":"2024-08-03T20:47:35.933618+0000","last_active":"2024-08-03T20:47:36.934972+0000","last_peered":"2024-08-03T20:47:36.934972+0000","last_clean":"2024-08-03T20:47:36.934972+0000","last_became_active":"2024-08-03T20:47:35.932873+0000","last_became_peered":"2024-08-03T20:47:35.932873+0000","last_unstale":"2024-08-03T20:47:36.934972+0000","last_undegraded":"2024-08-03T20:47:36.934972+0000","last_fullsized":"2024-08-03T20:47:36.934972+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_clean_scrub_stamp":"2024-08-03T20:47:34.904132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-04T23:57:18.840519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-03T20:47:37.979331+0000","last_change":"2024-08-03T20:47:34.940602+0000","last_active":"2024-08-03T20:47:37.979331+0000","last_peered":"2024-08-03T20:47:37.979331+0000","last_clean":"2024-08-03T20:47:37.979331+0000","last_became_active":"2024-08-03T20:47:34.940156+0000","last_became_peered":"2024-08-03T20:47:34.940156+0000","last_unstale":"2024-08-03T20:47:37.979331+0000","last_undegraded":"2024-08-03T20:47:37.979331+0000","last_fullsized":"2024-08-03T20:47:37.979331+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:47:33.895731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:47:33.895731+0000","last_clean_scrub_stamp":"2024-08-03T20:47:33.895731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T06:44:37.944928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-03T20:47:36.935959+0000","last_change":"2024-08-03T20:47:35.938536+0000","last_active":"2024-08-03T20:47:36.935959+0000","last_peered":"2024-08-03T20:47:36.935959+0000","last_clean":"2024-08-03T20:47:36.935959+0000","last_became_active":"2024-08-03T20:47:35.938281+0000","last_became_peered":"2024-08-03T20:47:35.938281+0000","last_unstale":"2024-08-03T20:47:36.935959+0000","last_undegraded":"2024-08-03T20:47:36.935959+0000","last_fullsized":"2024-08-03T20:47:36.935959+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_clean_scrub_stamp":"2024-08-03T20:47:34.904132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T06:17:22.118393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-03T20:47:36.935901+0000","last_change":"2024-08-03T20:47:34.936859+0000","last_active":"2024-08-03T20:47:36.935901+0000","last_peered":"2024-08-03T20:47:36.935901+0000","last_clean":"2024-08-03T20:47:36.935901+0000","last_became_active":"2024-08-03T20:47:34.936017+0000","last_became_peered":"2024-08-03T20:47:34.936017+0000","last_unstale":"2024-08-03T20:47:36.935901+0000","last_undegraded":"2024-08-03T20:47:36.935901+0000","last_fullsized":"2024-08-03T20:47:36.935901+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:47:33.895731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:47:33.895731+0000","last_clean_scrub_stamp":"2024-08-03T20:47:33.895731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T02:40:00.976532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-03T20:47:37.979373+0000","last_change":"2024-08-03T20:47:35.940325+0000","last_active":"2024-08-03T20:47:37.979373+0000","last_peered":"2024-08-03T20:47:37.979373+0000","last_clean":"2024-08-03T20:47:37.979373+0000","last_became_active":"2024-08-03T20:47:35.939888+0000","last_became_peered":"2024-08-03T20:47:35.939888+0000","last_unstale":"2024-08-03T20:47:37.979373+0000","last_undegraded":"2024-08-03T20:47:37.979373+0000","last_fullsized":"2024-08-03T20:47:37.979373+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_clean_scrub_stamp":"2024-08-03T20:47:34.904132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T02:17:30.747723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-03T20:47:37.317158+0000","last_change":"2024-08-03T20:47:34.946657+0000","last_active":"2024-08-03T20:47:37.317158+0000","last_peered":"2024-08-03T20:47:37.317158+0000","last_clean":"2024-08-03T20:47:37.317158+0000","last_became_active":"2024-08-03T20:47:34.945719+0000","last_became_peered":"2024-08-03T20:47:34.945719+0000","last_unstale":"2024-08-03T20:47:37.317158+0000","last_undegraded":"2024-08-03T20:47:37.317158+0000","last_fullsized":"2024-08-03T20:47:37.317158+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:47:33.895731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:47:33.895731+0000","last_clean_scrub_stamp":"2024-08-03T20:47:33.895731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-04T21:30:40.295081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-03T20:47:36.935812+0000","last_change":"2024-08-03T20:47:35.936144+0000","last_active":"2024-08-03T20:47:36.935812+0000","last_peered":"2024-08-03T20:47:36.935812+0000","last_clean":"2024-08-03T20:47:36.935812+0000","last_became_active":"2024-08-03T20:47:35.935842+0000","last_became_peered":"2024-08-03T20:47:35.935842+0000","last_unstale":"2024-08-03T20:47:36.935812+0000","last_undegraded":"2024-08-03T20:47:36.935812+0000","last_fullsized":"2024-08-03T20:47:36.935812+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_clean_scrub_stamp":"2024-08-03T20:47:34.904132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-04T22:33:13.028920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"21'1","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-03T20:47:36.935790+0000","last_change":"2024-08-03T20:47:34.934773+0000","last_active":"2024-08-03T20:47:36.935790+0000","last_peered":"2024-08-03T20:47:36.935790+0000","last_clean":"2024-08-03T20:47:36.935790+0000","last_became_active":"2024-08-03T20:47:34.934473+0000","last_became_peered":"2024-08-03T20:47:34.934473+0000","last_unstale":"2024-08-03T20:47:36.935790+0000","last_undegraded":"2024-08-03T20:47:36.935790+0000","last_fullsized":"2024-08-03T20:47:36.935790+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:47:33.895731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:47:33.895731+0000","last_clean_scrub_stamp":"2024-08-03T20:47:33.895731+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-04T22:30:28.119577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-03T20:47:37.979065+0000","last_change":"2024-08-03T20:47:35.933622+0000","last_active":"2024-08-03T20:47:37.979065+0000","last_peered":"2024-08-03T20:47:37.979065+0000","last_clean":"2024-08-03T20:47:37.979065+0000","last_became_active":"2024-08-03T20:47:35.933133+0000","last_became_peered":"2024-08-03T20:47:35.933133+0000","last_unstale":"2024-08-03T20:47:37.979065+0000","last_undegraded":"2024-08-03T20:47:37.979065+0000","last_fullsized":"2024-08-03T20:47:37.979065+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_clean_scrub_stamp":"2024-08-03T20:47:34.904132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-04T22:43:36.335718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"20'2","reported_seq":17,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-03T20:47:36.941891+0000","last_change":"2024-08-03T20:47:34.944805+0000","last_active":"2024-08-03T20:47:36.941891+0000","last_peered":"2024-08-03T20:47:36.941891+0000","last_clean":"2024-08-03T20:47:36.941891+0000","last_became_active":"2024-08-03T20:47:34.943207+0000","last_became_peered":"2024-08-03T20:47:34.943207+0000","last_unstale":"2024-08-03T20:47:36.941891+0000","last_undegraded":"2024-08-03T20:47:36.941891+0000","last_fullsized":"2024-08-03T20:47:36.941891+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:47:33.895731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:47:33.895731+0000","last_clean_scrub_stamp":"2024-08-03T20:47:33.895731+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T07:46:07.162679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-03T20:47:37.979614+0000","last_change":"2024-08-03T20:47:35.941163+0000","last_active":"2024-08-03T20:47:37.979614+0000","last_peered":"2024-08-03T20:47:37.979614+0000","last_clean":"2024-08-03T20:47:37.979614+0000","last_became_active":"2024-08-03T20:47:35.940955+0000","last_became_peered":"2024-08-03T20:47:35.940955+0000","last_unstale":"2024-08-03T20:47:37.979614+0000","last_undegraded":"2024-08-03T20:47:37.979614+0000","last_fullsized":"2024-08-03T20:47:37.979614+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_clean_scrub_stamp":"2024-08-03T20:47:34.904132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T01:08:56.085650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-03T20:47:37.316412+0000","last_change":"2024-08-03T20:47:34.943492+0000","last_active":"2024-08-03T20:47:37.316412+0000","last_peered":"2024-08-03T20:47:37.316412+0000","last_clean":"2024-08-03T20:47:37.316412+0000","last_became_active":"2024-08-03T20:47:34.943121+0000","last_became_peered":"2024-08-03T20:47:34.943121+0000","last_unstale":"2024-08-03T20:47:37.316412+0000","last_undegraded":"2024-08-03T20:47:37.316412+0000","last_fullsized":"2024-08-03T20:47:37.316412+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:47:33.895731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:47:33.895731+0000","last_clean_scrub_stamp":"2024-08-03T20:47:33.895731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T07:46:07.162679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-03T20:47:36.935830+0000","last_change":"2024-08-03T20:47:35.924518+0000","last_active":"2024-08-03T20:47:36.935830+0000","last_peered":"2024-08-03T20:47:36.935830+0000","last_clean":"2024-08-03T20:47:36.935830+0000","last_became_active":"2024-08-03T20:47:35.923938+0000","last_became_peered":"2024-08-03T20:47:35.923938+0000","last_unstale":"2024-08-03T20:47:36.935830+0000","last_undegraded":"2024-08-03T20:47:36.935830+0000","last_fullsized":"2024-08-03T20:47:36.935830+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_clean_scrub_stamp":"2024-08-03T20:47:34.904132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T04:26:33.175697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-03T20:47:36.935369+0000","last_change":"2024-08-03T20:47:34.941712+0000","last_active":"2024-08-03T20:47:36.935369+0000","last_peered":"2024-08-03T20:47:36.935369+0000","last_clean":"2024-08-03T20:47:36.935369+0000","last_became_active":"2024-08-03T20:47:34.941138+0000","last_became_peered":"2024-08-03T20:47:34.941138+0000","last_unstale":"2024-08-03T20:47:36.935369+0000","last_undegraded":"2024-08-03T20:47:36.935369+0000","last_fullsized":"2024-08-03T20:47:36.935369+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:47:33.895731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:47:33.895731+0000","last_clean_scrub_stamp":"2024-08-03T20:47:33.895731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T07:46:07.162679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-03T20:47:37.978318+0000","last_change":"2024-08-03T20:47:35.941992+0000","last_active":"2024-08-03T20:47:37.978318+0000","last_peered":"2024-08-03T20:47:37.978318+0000","last_clean":"2024-08-03T20:47:37.978318+0000","last_became_active":"2024-08-03T20:47:35.941633+0000","last_became_peered":"2024-08-03T20:47:35.941633+0000","last_unstale":"2024-08-03T20:47:37.978318+0000","last_undegraded":"2024-08-03T20:47:37.978318+0000","last_fullsized":"2024-08-03T20:47:37.978318+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_clean_scrub_stamp":"2024-08-03T20:47:34.904132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T03:06:49.510875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-03T20:47:36.935863+0000","last_change":"2024-08-03T20:47:34.932831+0000","last_active":"2024-08-03T20:47:36.935863+0000","last_peered":"2024-08-03T20:47:36.935863+0000","last_clean":"2024-08-03T20:47:36.935863+0000","last_became_active":"2024-08-03T20:47:34.932520+0000","last_became_peered":"2024-08-03T20:47:34.932520+0000","last_unstale":"2024-08-03T20:47:36.935863+0000","last_undegraded":"2024-08-03T20:47:36.935863+0000","last_fullsized":"2024-08-03T20:47:36.935863+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:47:33.895731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:47:33.895731+0000","last_clean_scrub_stamp":"2024-08-03T20:47:33.895731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T00:43:30.770268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-03T20:47:37.978692+0000","last_change":"2024-08-03T20:47:35.942701+0000","last_active":"2024-08-03T20:47:37.978692+0000","last_peered":"2024-08-03T20:47:37.978692+0000","last_clean":"2024-08-03T20:47:37.978692+0000","last_became_active":"2024-08-03T20:47:35.942261+0000","last_became_peered":"2024-08-03T20:47:35.942261+0000","last_unstale":"2024-08-03T20:47:37.978692+0000","last_undegraded":"2024-08-03T20:47:37.978692+0000","last_fullsized":"2024-08-03T20:47:37.978692+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_clean_scrub_stamp":"2024-08-03T20:47:34.904132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-04T22:01:37.009450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"20'2","reported_seq":17,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-03T20:47:37.317111+0000","last_change":"2024-08-03T20:47:34.947233+0000","last_active":"2024-08-03T20:47:37.317111+0000","last_peered":"2024-08-03T20:47:37.317111+0000","last_clean":"2024-08-03T20:47:37.317111+0000","last_became_active":"2024-08-03T20:47:34.946866+0000","last_became_peered":"2024-08-03T20:47:34.946866+0000","last_unstale":"2024-08-03T20:47:37.317111+0000","last_undegraded":"2024-08-03T20:47:37.317111+0000","last_fullsized":"2024-08-03T20:47:37.317111+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:47:33.895731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:47:33.895731+0000","last_clean_scrub_stamp":"2024-08-03T20:47:33.895731+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T03:46:04.675513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-03T20:47:36.941158+0000","last_change":"2024-08-03T20:47:35.936142+0000","last_active":"2024-08-03T20:47:36.941158+0000","last_peered":"2024-08-03T20:47:36.941158+0000","last_clean":"2024-08-03T20:47:36.941158+0000","last_became_active":"2024-08-03T20:47:35.935764+0000","last_became_peered":"2024-08-03T20:47:35.935764+0000","last_unstale":"2024-08-03T20:47:36.941158+0000","last_undegraded":"2024-08-03T20:47:36.941158+0000","last_fullsized":"2024-08-03T20:47:36.941158+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_clean_scrub_stamp":"2024-08-03T20:47:34.904132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-04T22:12:15.293757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-03T20:47:37.979240+0000","last_change":"2024-08-03T20:47:35.934358+0000","last_active":"2024-08-03T20:47:37.979240+0000","last_peered":"2024-08-03T20:47:37.979240+0000","last_clean":"2024-08-03T20:47:37.979240+0000","last_became_active":"2024-08-03T20:47:35.933874+0000","last_became_peered":"2024-08-03T20:47:35.933874+0000","last_unstale":"2024-08-03T20:47:37.979240+0000","last_undegraded":"2024-08-03T20:47:37.979240+0000","last_fullsized":"2024-08-03T20:47:37.979240+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_clean_scrub_stamp":"2024-08-03T20:47:34.904132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T00:10:43.343156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"20'2","reported_seq":17,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-03T20:47:37.316590+0000","last_change":"2024-08-03T20:47:34.950578+0000","last_active":"2024-08-03T20:47:37.316590+0000","last_peered":"2024-08-03T20:47:37.316590+0000","last_clean":"2024-08-03T20:47:37.316590+0000","last_became_active":"2024-08-03T20:47:34.950187+0000","last_became_peered":"2024-08-03T20:47:34.950187+0000","last_unstale":"2024-08-03T20:47:37.316590+0000","last_undegraded":"2024-08-03T20:47:37.316590+0000","last_fullsized":"2024-08-03T20:47:37.316590+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:47:33.895731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:47:33.895731+0000","last_clean_scrub_stamp":"2024-08-03T20:47:33.895731+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T00:43:30.770268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-03T20:47:37.979438+0000","last_change":"2024-08-03T20:47:35.934586+0000","last_active":"2024-08-03T20:47:37.979438+0000","last_peered":"2024-08-03T20:47:37.979438+0000","last_clean":"2024-08-03T20:47:37.979438+0000","last_became_active":"2024-08-03T20:47:35.934289+0000","last_became_peered":"2024-08-03T20:47:35.934289+0000","last_unstale":"2024-08-03T20:47:37.979438+0000","last_undegraded":"2024-08-03T20:47:37.979438+0000","last_fullsized":"2024-08-03T20:47:37.979438+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_clean_scrub_stamp":"2024-08-03T20:47:34.904132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T04:51:27.371858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"20'2","reported_seq":17,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-03T20:47:37.316998+0000","last_change":"2024-08-03T20:47:34.946365+0000","last_active":"2024-08-03T20:47:37.316998+0000","last_peered":"2024-08-03T20:47:37.316998+0000","last_clean":"2024-08-03T20:47:37.316998+0000","last_became_active":"2024-08-03T20:47:34.945922+0000","last_became_peered":"2024-08-03T20:47:34.945922+0000","last_unstale":"2024-08-03T20:47:37.316998+0000","last_undegraded":"2024-08-03T20:47:37.316998+0000","last_fullsized":"2024-08-03T20:47:37.316998+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:47:33.895731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:47:33.895731+0000","last_clean_scrub_stamp":"2024-08-03T20:47:33.895731+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T06:59:03.089784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-03T20:47:37.979781+0000","last_change":"2024-08-03T20:47:35.941147+0000","last_active":"2024-08-03T20:47:37.979781+0000","last_peered":"2024-08-03T20:47:37.979781+0000","last_clean":"2024-08-03T20:47:37.979781+0000","last_became_active":"2024-08-03T20:47:35.940605+0000","last_became_peered":"2024-08-03T20:47:35.940605+0000","last_unstale":"2024-08-03T20:47:37.979781+0000","last_undegraded":"2024-08-03T20:47:37.979781+0000","last_fullsized":"2024-08-03T20:47:37.979781+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_clean_scrub_stamp":"2024-08-03T20:47:34.904132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-04T22:33:13.028920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"20'1","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-03T20:47:37.978906+0000","last_change":"2024-08-03T20:47:34.934888+0000","last_active":"2024-08-03T20:47:37.978906+0000","last_peered":"2024-08-03T20:47:37.978906+0000","last_clean":"2024-08-03T20:47:37.978906+0000","last_became_active":"2024-08-03T20:47:34.934595+0000","last_became_peered":"2024-08-03T20:47:34.934595+0000","last_unstale":"2024-08-03T20:47:37.978906+0000","last_undegraded":"2024-08-03T20:47:37.978906+0000","last_fullsized":"2024-08-03T20:47:37.978906+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:47:33.895731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:47:33.895731+0000","last_clean_scrub_stamp":"2024-08-03T20:47:33.895731+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T06:13:41.904985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-03T20:47:37.978652+0000","last_change":"2024-08-03T20:47:35.953798+0000","last_active":"2024-08-03T20:47:37.978652+0000","last_peered":"2024-08-03T20:47:37.978652+0000","last_clean":"2024-08-03T20:47:37.978652+0000","last_became_active":"2024-08-03T20:47:35.950186+0000","last_became_peered":"2024-08-03T20:47:35.950186+0000","last_unstale":"2024-08-03T20:47:37.978652+0000","last_undegraded":"2024-08-03T20:47:37.978652+0000","last_fullsized":"2024-08-03T20:47:37.978652+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:47:34.904132+0000","last_clean_scrub_stamp":"2024-08-03T20:47:34.904132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T07:58:55.643722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"20'4","reported_seq":19,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-03T20:47:37.979858+0000","last_change":"2024-08-03T20:47:34.939409+0000","last_active":"2024-08-03T20:47:37.979858+0000","last_peered":"2024-08-03T20:47:37.979858+0000","last_clean":"2024-08-03T20:47:37.979858+0000","last_became_active":"2024-08-03T20:47:34.939057+0000","last_became_peered":"2024-08-03T20:47:34.939057+0000","last_unstale":"2024-08-03T20:47:37.979858+0000","last_undegraded":"2024-08-03T20:47:37.979858+0000","last_fullsized":"2024-08-03T20:47:37.979858+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:47:33.895731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:47:33.895731+0000","last_clean_scrub_stamp":"2024-08-03T20:47:33.895731+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T05:49:05.605066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509446,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31972,"kb_used_data":788,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339868,"statfs":{"total":96636764160,"available":96604024832,"internally_reserved":0,"allocated":806912,"data_stored":663491,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":15,"seq":64424509447,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31396,"kb_used_data":216,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340444,"statfs":{"total":96636764160,"available":96604614656,"internally_reserved":0,"allocated":221184,"data_stored":74153,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":15,"seq":64424509447,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31388,"kb_used_data":208,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340452,"statfs":{"total":96636764160,"available":96604622848,"internally_reserved":0,"allocated":212992,"data_stored":72631,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509446,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31392,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340448,"statfs":{"total":96636764160,"available":96604618752,"internally_reserved":0,"allocated":217088,"data_stored":72701,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509448,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31980,"kb_used_data":796,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339860,"statfs":{"total":96636764160,"available":96604016640,"internally_reserved":0,"allocated":815104,"data_stored":663547,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542151,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31400,"kb_used_data":216,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340440,"statfs":{"total":96636764160,"available":96604610560,"internally_reserved":0,"allocated":221184,"data_stored":72667,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":14,"seq":60129542149,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31400,"kb_used_data":216,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340440,"statfs":{"total":96636764160,"available":96604610560,"internally_reserved":0,"allocated":221184,"data_stored":74153,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":14,"seq":60129542150,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31388,"kb_used_data":204,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340452,"statfs":{"total":96636764160,"available":96604622848,"internally_reserved":0,"allocated":208896,"data_stored":72541,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-08-03T20:47:47.487 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-08-03T20:47:47.487 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-08-03T20:47:47.487 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2024-08-03T20:47:47.487 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2024-08-03T20:47:47.750 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:47.748+0000 7fbac4400640 1 -- 172.21.15.32:0/2089789175 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbabc0cccf0 msgr2=0x7fbabc0cd100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:47:47.750 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:47.748+0000 7fbac4400640 1 --2- 172.21.15.32:0/2089789175 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbabc0cccf0 0x7fbabc0cd100 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fbabc0cca80 tx=0x7fbaa802a4a0 comp rx=0 tx=0).stop 2024-08-03T20:47:47.750 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:47.748+0000 7fbac4400640 1 -- 172.21.15.32:0/2089789175 shutdown_connections 2024-08-03T20:47:47.751 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:47.748+0000 7fbac4400640 1 --2- 172.21.15.32:0/2089789175 >> v2:172.21.15.118:3300/0 conn(0x7fbabc0ced30 0x7fbabc0cf140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:47.751 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:47.748+0000 7fbac4400640 1 --2- 172.21.15.32:0/2089789175 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbabc0cccf0 0x7fbabc0cd100 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:47.751 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:47.748+0000 7fbac4400640 1 -- 172.21.15.32:0/2089789175 >> 172.21.15.32:0/2089789175 conn(0x7fbabc0b1bd0 msgr2=0x7fbabc0b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:47:47.751 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:47.748+0000 7fbac4400640 1 -- 172.21.15.32:0/2089789175 shutdown_connections 2024-08-03T20:47:47.751 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:47.748+0000 7fbac4400640 1 -- 172.21.15.32:0/2089789175 wait complete. 2024-08-03T20:47:47.752 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:47.751+0000 7fbac4400640 1 Processor -- start 2024-08-03T20:47:47.752 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:47.751+0000 7fbac4400640 1 -- start start 2024-08-03T20:47:47.755 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:47.752+0000 7fbac4400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbabc0cccf0 0x7fbabc153b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:47:47.755 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:47.752+0000 7fbac4400640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fbabc0ced30 0x7fbabc0cb1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:47:47.755 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:47.752+0000 7fbac4400640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fbabc145c50 0x7fbabc0cff10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:47:47.755 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:47.752+0000 7fbac4400640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fbabc0b4070 con 0x7fbabc0cccf0 2024-08-03T20:47:47.755 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:47.752+0000 7fbac4400640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7fbabc0b3ef0 con 0x7fbabc0ced30 2024-08-03T20:47:47.755 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:47.752+0000 7fbac4400640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7fbabc0b41f0 con 0x7fbabc145c50 2024-08-03T20:47:47.755 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:47.752+0000 7fbac1a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbabc0cccf0 0x7fbabc153b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:47:47.755 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:47.753+0000 7fbac1a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbabc0cccf0 0x7fbabc153b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:57698/0 (socket says 172.21.15.32:57698) 2024-08-03T20:47:47.756 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:47.753+0000 7fbac1a00640 1 -- 172.21.15.32:0/680952390 learned_addr learned my addr 172.21.15.32:0/680952390 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:47:47.756 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:47.753+0000 7fbac2400640 1 --2- 172.21.15.32:0/680952390 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fbabc145c50 0x7fbabc0cff10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:47:47.756 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:47.753+0000 7fbac1000640 1 --2- 172.21.15.32:0/680952390 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fbabc0ced30 0x7fbabc0cb1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:47:47.756 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:47.753+0000 7fbac1a00640 1 -- 172.21.15.32:0/680952390 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fbabc145c50 msgr2=0x7fbabc0cff10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:47:47.756 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:47.753+0000 7fbac1a00640 1 --2- 172.21.15.32:0/680952390 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fbabc145c50 0x7fbabc0cff10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:47.756 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:47.753+0000 7fbac1a00640 1 -- 172.21.15.32:0/680952390 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fbabc0ced30 msgr2=0x7fbabc0cb1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:47:47.756 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:47.753+0000 7fbac1a00640 1 --2- 172.21.15.32:0/680952390 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fbabc0ced30 0x7fbabc0cb1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:47.756 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:47.753+0000 7fbac1a00640 1 -- 172.21.15.32:0/680952390 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbabc1f9450 con 0x7fbabc0cccf0 2024-08-03T20:47:47.756 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:47.753+0000 7fbac2400640 1 --2- 172.21.15.32:0/680952390 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fbabc145c50 0x7fbabc0cff10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-03T20:47:47.756 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:47.753+0000 7fbac1000640 1 --2- 172.21.15.32:0/680952390 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fbabc0ced30 0x7fbabc0cb1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-03T20:47:47.756 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:47.753+0000 7fbac1a00640 1 --2- 172.21.15.32:0/680952390 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbabc0cccf0 0x7fbabc153b10 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fbab000c460 tx=0x7fbab000c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:47:47.756 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:47.753+0000 7fbab9600640 1 -- 172.21.15.32:0/680952390 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbab0016020 con 0x7fbabc0cccf0 2024-08-03T20:47:47.756 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:47.754+0000 7fbab9600640 1 -- 172.21.15.32:0/680952390 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fbab0012940 con 0x7fbabc0cccf0 2024-08-03T20:47:47.757 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:47.754+0000 7fbab9600640 1 -- 172.21.15.32:0/680952390 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbab0012d90 con 0x7fbabc0cccf0 2024-08-03T20:47:47.757 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:47.754+0000 7fbac4400640 1 -- 172.21.15.32:0/680952390 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbabc1f8690 con 0x7fbabc0cccf0 2024-08-03T20:47:47.757 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:47.754+0000 7fbac4400640 1 -- 172.21.15.32:0/680952390 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbabc1f8c50 con 0x7fbabc0cccf0 2024-08-03T20:47:47.757 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:47.755+0000 7fbac4400640 1 -- 172.21.15.32:0/680952390 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbabc0b4710 con 0x7fbabc0cccf0 2024-08-03T20:47:47.757 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:47.756+0000 7fbab9600640 1 -- 172.21.15.32:0/680952390 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7fbab0019020 con 0x7fbabc0cccf0 2024-08-03T20:47:47.759 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:47.758+0000 7fbab9600640 1 --2- 172.21.15.32:0/680952390 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fba9c077010 0x7fba9c079520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:47:47.760 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:47.758+0000 7fbab9600640 1 -- 172.21.15.32:0/680952390 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7fbab00982f0 con 0x7fbabc0cccf0 2024-08-03T20:47:47.760 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:47.758+0000 7fbac1000640 1 --2- 172.21.15.32:0/680952390 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fba9c077010 0x7fba9c079520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:47:47.760 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:47.759+0000 7fbac1000640 1 --2- 172.21.15.32:0/680952390 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fba9c077010 0x7fba9c079520 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fbaa8034000 tx=0x7fbaa8004730 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:47:47.763 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:47.762+0000 7fbab9600640 1 -- 172.21.15.32:0/680952390 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7fbab0020070 con 0x7fbabc0cccf0 2024-08-03T20:47:48.010 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:48.008+0000 7fbac4400640 1 -- 172.21.15.32:0/680952390 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) -- 0x7fbabc06d9f0 con 0x7fbabc0cccf0 2024-08-03T20:47:48.010 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:48.009+0000 7fbab9600640 1 -- 172.21.15.32:0/680952390 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) ==== 72+0+46 (secure 0 0 0) 0x7fbab00614f0 con 0x7fbabc0cccf0 2024-08-03T20:47:48.010 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-03T20:47:48.011 INFO:teuthology.orchestra.run.smithi032.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-08-03T20:47:48.013 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:48.011+0000 7fbac4400640 1 -- 172.21.15.32:0/680952390 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fba9c077010 msgr2=0x7fba9c079520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:47:48.013 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:48.011+0000 7fbac4400640 1 --2- 172.21.15.32:0/680952390 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fba9c077010 0x7fba9c079520 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fbaa8034000 tx=0x7fbaa8004730 comp rx=0 tx=0).stop 2024-08-03T20:47:48.013 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:48.011+0000 7fbac4400640 1 -- 172.21.15.32:0/680952390 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbabc0cccf0 msgr2=0x7fbabc153b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:47:48.013 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:48.011+0000 7fbac4400640 1 --2- 172.21.15.32:0/680952390 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbabc0cccf0 0x7fbabc153b10 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fbab000c460 tx=0x7fbab000c920 comp rx=0 tx=0).stop 2024-08-03T20:47:48.013 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:48.011+0000 7fbac4400640 1 -- 172.21.15.32:0/680952390 shutdown_connections 2024-08-03T20:47:48.013 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:48.011+0000 7fbac4400640 1 --2- 172.21.15.32:0/680952390 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fba9c077010 0x7fba9c079520 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:48.013 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:48.011+0000 7fbac4400640 1 --2- 172.21.15.32:0/680952390 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fbabc0ced30 0x7fbabc0cb1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:48.013 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:48.011+0000 7fbac4400640 1 --2- 172.21.15.32:0/680952390 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fbabc145c50 0x7fbabc0cff10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:48.013 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:48.011+0000 7fbac4400640 1 --2- 172.21.15.32:0/680952390 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbabc0cccf0 0x7fbabc153b10 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:48.014 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:48.011+0000 7fbac4400640 1 -- 172.21.15.32:0/680952390 >> 172.21.15.32:0/680952390 conn(0x7fbabc0b1bd0 msgr2=0x7fbabc0cc3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:47:48.014 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:48.011+0000 7fbac4400640 1 -- 172.21.15.32:0/680952390 shutdown_connections 2024-08-03T20:47:48.014 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:48.011+0000 7fbac4400640 1 -- 172.21.15.32:0/680952390 wait complete. 2024-08-03T20:47:48.035 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2024-08-03T20:47:48.035 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-08-03T20:47:48.261 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:48.259+0000 7f922b400640 1 -- 172.21.15.32:0/2741619902 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9224144f90 msgr2=0x7f9224074570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:47:48.261 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:48.259+0000 7f922b400640 1 --2- 172.21.15.32:0/2741619902 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9224144f90 0x7f9224074570 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f9224073b50 tx=0x7f921002a510 comp rx=0 tx=0).stop 2024-08-03T20:47:48.262 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:48.259+0000 7f922b400640 1 -- 172.21.15.32:0/2741619902 shutdown_connections 2024-08-03T20:47:48.262 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:48.259+0000 7f922b400640 1 --2- 172.21.15.32:0/2741619902 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9224144f90 0x7f9224074570 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:48.262 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:48.259+0000 7f922b400640 1 -- 172.21.15.32:0/2741619902 >> 172.21.15.32:0/2741619902 conn(0x7f92240b1bd0 msgr2=0x7f92240b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:47:48.262 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:48.260+0000 7f922b400640 1 -- 172.21.15.32:0/2741619902 shutdown_connections 2024-08-03T20:47:48.262 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:48.260+0000 7f922b400640 1 -- 172.21.15.32:0/2741619902 wait complete. 2024-08-03T20:47:48.263 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:48.262+0000 7f922b400640 1 Processor -- start 2024-08-03T20:47:48.263 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:48.262+0000 7f922b400640 1 -- start start 2024-08-03T20:47:48.265 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:48.263+0000 7f922b400640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f922406e930 0x7f9224156d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:47:48.266 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:48.263+0000 7f922b400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9224144f90 0x7f92241414e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:47:48.266 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:48.263+0000 7f922b400640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f922415fff0 0x7f9224141a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:47:48.266 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:48.263+0000 7f922b400640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f92240cf840 con 0x7f9224144f90 2024-08-03T20:47:48.266 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:48.263+0000 7f922b400640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f9224145f30 con 0x7f922406e930 2024-08-03T20:47:48.266 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:48.263+0000 7f922b400640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7f92240cf9c0 con 0x7f922415fff0 2024-08-03T20:47:48.266 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:48.264+0000 7f9223e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9224144f90 0x7f92241414e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:47:48.266 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:48.264+0000 7f9223e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9224144f90 0x7f92241414e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:57716/0 (socket says 172.21.15.32:57716) 2024-08-03T20:47:48.266 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:48.264+0000 7f9223e00640 1 -- 172.21.15.32:0/2405631496 learned_addr learned my addr 172.21.15.32:0/2405631496 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:47:48.266 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:48.264+0000 7f9228a00640 1 --2- 172.21.15.32:0/2405631496 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f922406e930 0x7f9224156d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:47:48.266 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:48.264+0000 7f9229400640 1 --2- 172.21.15.32:0/2405631496 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f922415fff0 0x7f9224141a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:47:48.266 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:48.264+0000 7f9223e00640 1 -- 172.21.15.32:0/2405631496 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f922415fff0 msgr2=0x7f9224141a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:47:48.267 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:48.264+0000 7f9223e00640 1 --2- 172.21.15.32:0/2405631496 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f922415fff0 0x7f9224141a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:48.267 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:48.264+0000 7f9223e00640 1 -- 172.21.15.32:0/2405631496 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f922406e930 msgr2=0x7f9224156d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:47:48.267 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:48.264+0000 7f9223e00640 1 --2- 172.21.15.32:0/2405631496 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f922406e930 0x7f9224156d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:48.267 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:48.264+0000 7f9223e00640 1 -- 172.21.15.32:0/2405631496 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9224157260 con 0x7f9224144f90 2024-08-03T20:47:48.267 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:48.264+0000 7f9223e00640 1 --2- 172.21.15.32:0/2405631496 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9224144f90 0x7f92241414e0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f92100098f0 tx=0x7f9210033c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:47:48.268 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:48.265+0000 7f9220c00640 1 -- 172.21.15.32:0/2405631496 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9210005c90 con 0x7f9224144f90 2024-08-03T20:47:48.268 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:48.265+0000 7f9220c00640 1 -- 172.21.15.32:0/2405631496 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f9210031e50 con 0x7f9224144f90 2024-08-03T20:47:48.268 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:48.265+0000 7f9220c00640 1 -- 172.21.15.32:0/2405631496 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9210005260 con 0x7f9224144f90 2024-08-03T20:47:48.268 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:48.265+0000 7f922b400640 1 -- 172.21.15.32:0/2405631496 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f92241576b0 con 0x7f9224144f90 2024-08-03T20:47:48.268 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:48.265+0000 7f922b400640 1 -- 172.21.15.32:0/2405631496 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9224157c70 con 0x7f9224144f90 2024-08-03T20:47:48.268 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:48.266+0000 7f922b400640 1 -- 172.21.15.32:0/2405631496 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9224160400 con 0x7f9224144f90 2024-08-03T20:47:48.268 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:48.267+0000 7f9220c00640 1 -- 172.21.15.32:0/2405631496 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f9210005400 con 0x7f9224144f90 2024-08-03T20:47:48.270 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:48.268+0000 7f9220c00640 1 --2- 172.21.15.32:0/2405631496 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f9204077040 0x7f9204079550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:47:48.270 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:48.268+0000 7f9220c00640 1 -- 172.21.15.32:0/2405631496 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f92100b9950 con 0x7f9224144f90 2024-08-03T20:47:48.270 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:48.269+0000 7f9228a00640 1 --2- 172.21.15.32:0/2405631496 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f9204077040 0x7f9204079550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:47:48.270 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:48.269+0000 7f9228a00640 1 --2- 172.21.15.32:0/2405631496 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f9204077040 0x7f9204079550 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f9218007fd0 tx=0x7f9218001370 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:47:48.274 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:48.273+0000 7f9220c00640 1 -- 172.21.15.32:0/2405631496 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7f9224157260 con 0x7f9224144f90 2024-08-03T20:47:48.519 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:48.517+0000 7f922b400640 1 -- 172.21.15.32:0/2405631496 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f92241457e0 con 0x7f9224144f90 2024-08-03T20:47:48.519 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:48.517+0000 7f9220c00640 1 -- 172.21.15.32:0/2405631496 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) ==== 93+0+5220 (secure 0 0 0) 0x7f921003f030 con 0x7f9224144f90 2024-08-03T20:47:48.519 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-03T20:47:48.520 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":8,"btime":"2024-08-03T20:47:38:939479+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4289,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6835/1753256117","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6834","nonce":1753256117},{"type":"v1","addr":"172.21.15.118:6835","nonce":1753256117}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4292,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6837/3055588578","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6836","nonce":3055588578},{"type":"v1","addr":"172.21.15.118:6837","nonce":3055588578}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4338,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6834/3026095815","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":3026095815},{"type":"v1","addr":"172.21.15.32:6834","nonce":3026095815}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-08-03T20:47:35.442482+0000","modified":"2024-08-03T20:47:38.460928+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4341},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4341":{"gid":4341,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.32:6835/246278108","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6833","nonce":246278108},{"type":"v1","addr":"172.21.15.32:6835","nonce":246278108}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4341,"qdb_cluster":[4341]},"id":1}]} 2024-08-03T20:47:48.520 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 8 2024-08-03T20:47:48.521 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:48.519+0000 7f922b400640 1 -- 172.21.15.32:0/2405631496 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f9204077040 msgr2=0x7f9204079550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:47:48.521 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:48.519+0000 7f922b400640 1 --2- 172.21.15.32:0/2405631496 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f9204077040 0x7f9204079550 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f9218007fd0 tx=0x7f9218001370 comp rx=0 tx=0).stop 2024-08-03T20:47:48.521 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:48.519+0000 7f922b400640 1 -- 172.21.15.32:0/2405631496 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9224144f90 msgr2=0x7f92241414e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:47:48.521 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:48.519+0000 7f922b400640 1 --2- 172.21.15.32:0/2405631496 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9224144f90 0x7f92241414e0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f92100098f0 tx=0x7f9210033c20 comp rx=0 tx=0).stop 2024-08-03T20:47:48.521 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:48.520+0000 7f922b400640 1 -- 172.21.15.32:0/2405631496 shutdown_connections 2024-08-03T20:47:48.522 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:48.520+0000 7f922b400640 1 --2- 172.21.15.32:0/2405631496 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f9204077040 0x7f9204079550 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:48.522 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:48.520+0000 7f922b400640 1 --2- 172.21.15.32:0/2405631496 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f922415fff0 0x7f9224141a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:48.522 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:48.520+0000 7f922b400640 1 --2- 172.21.15.32:0/2405631496 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9224144f90 0x7f92241414e0 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:48.522 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:48.520+0000 7f922b400640 1 --2- 172.21.15.32:0/2405631496 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f922406e930 0x7f9224156d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:48.522 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:48.520+0000 7f922b400640 1 -- 172.21.15.32:0/2405631496 >> 172.21.15.32:0/2405631496 conn(0x7f92240b1bd0 msgr2=0x7f92240d0920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:47:48.522 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:48.520+0000 7f922b400640 1 -- 172.21.15.32:0/2405631496 shutdown_connections 2024-08-03T20:47:48.522 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:48.520+0000 7f922b400640 1 -- 172.21.15.32:0/2405631496 wait complete. 2024-08-03T20:47:48.543 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-08-03T20:47:48.771 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:48.769+0000 7f8e16200640 1 -- 172.21.15.32:0/500330550 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8e10159f70 msgr2=0x7f8e1015ff60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:47:48.771 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:48.769+0000 7f8e16200640 1 --2- 172.21.15.32:0/500330550 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8e10159f70 0x7f8e1015ff60 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f8e04009860 tx=0x7f8e0402a4b0 comp rx=0 tx=0).stop 2024-08-03T20:47:48.771 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:48.769+0000 7f8e16200640 1 -- 172.21.15.32:0/500330550 shutdown_connections 2024-08-03T20:47:48.771 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:48.769+0000 7f8e16200640 1 --2- 172.21.15.32:0/500330550 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f8e101604a0 0x7f8e10070750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:48.771 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:48.769+0000 7f8e16200640 1 --2- 172.21.15.32:0/500330550 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8e10159f70 0x7f8e1015ff60 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:48.771 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:48.769+0000 7f8e16200640 1 -- 172.21.15.32:0/500330550 >> 172.21.15.32:0/500330550 conn(0x7f8e100d00f0 msgr2=0x7f8e100d0500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:47:48.771 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:48.769+0000 7f8e16200640 1 -- 172.21.15.32:0/500330550 shutdown_connections 2024-08-03T20:47:48.771 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:48.769+0000 7f8e16200640 1 -- 172.21.15.32:0/500330550 wait complete. 2024-08-03T20:47:48.772 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:48.771+0000 7f8e16200640 1 Processor -- start 2024-08-03T20:47:48.772 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:48.771+0000 7f8e16200640 1 -- start start 2024-08-03T20:47:48.775 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:48.773+0000 7f8e16200640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8e10070c90 0x7f8e101f5150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:47:48.775 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:48.773+0000 7f8e16200640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f8e10159f70 0x7f8e1009f870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:47:48.775 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:48.773+0000 7f8e16200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8e101604a0 0x7f8e100b1d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:47:48.775 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:48.773+0000 7f8e16200640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f8e10071150 con 0x7f8e101604a0 2024-08-03T20:47:48.775 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:48.773+0000 7f8e16200640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f8e100cfe40 con 0x7f8e10070c90 2024-08-03T20:47:48.775 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:48.773+0000 7f8e16200640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7f8e1015d080 con 0x7f8e10159f70 2024-08-03T20:47:48.775 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:48.773+0000 7f8e15800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8e101604a0 0x7f8e100b1d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:47:48.775 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:48.773+0000 7f8e15800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8e101604a0 0x7f8e100b1d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:57738/0 (socket says 172.21.15.32:57738) 2024-08-03T20:47:48.775 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:48.773+0000 7f8e15800640 1 -- 172.21.15.32:0/1366487406 learned_addr learned my addr 172.21.15.32:0/1366487406 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:47:48.776 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:48.773+0000 7f8e0fe00640 1 --2- 172.21.15.32:0/1366487406 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f8e10159f70 0x7f8e1009f870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:47:48.776 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:48.773+0000 7f8e14e00640 1 --2- 172.21.15.32:0/1366487406 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8e10070c90 0x7f8e101f5150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:47:48.776 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:48.773+0000 7f8e15800640 1 -- 172.21.15.32:0/1366487406 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f8e10159f70 msgr2=0x7f8e1009f870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:47:48.776 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:48.773+0000 7f8e15800640 1 --2- 172.21.15.32:0/1366487406 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f8e10159f70 0x7f8e1009f870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:48.776 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:48.773+0000 7f8e15800640 1 -- 172.21.15.32:0/1366487406 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8e10070c90 msgr2=0x7f8e101f5150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:47:48.776 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:48.773+0000 7f8e15800640 1 --2- 172.21.15.32:0/1366487406 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8e10070c90 0x7f8e101f5150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:48.776 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:48.773+0000 7f8e15800640 1 -- 172.21.15.32:0/1366487406 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8e101f5690 con 0x7f8e101604a0 2024-08-03T20:47:48.776 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:48.774+0000 7f8e15800640 1 --2- 172.21.15.32:0/1366487406 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8e101604a0 0x7f8e100b1d00 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f8e0800cad0 tx=0x7f8e0800cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:47:48.777 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:48.774+0000 7f8e0cc00640 1 -- 172.21.15.32:0/1366487406 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e08016020 con 0x7f8e101604a0 2024-08-03T20:47:48.777 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:48.774+0000 7f8e0cc00640 1 -- 172.21.15.32:0/1366487406 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f8e0800ad20 con 0x7f8e101604a0 2024-08-03T20:47:48.777 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:48.774+0000 7f8e0cc00640 1 -- 172.21.15.32:0/1366487406 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e08012680 con 0x7f8e101604a0 2024-08-03T20:47:48.777 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:48.774+0000 7f8e16200640 1 -- 172.21.15.32:0/1366487406 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8e100a0800 con 0x7f8e101604a0 2024-08-03T20:47:48.777 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:48.774+0000 7f8e16200640 1 -- 172.21.15.32:0/1366487406 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8e100cab50 con 0x7f8e101604a0 2024-08-03T20:47:48.778 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:48.775+0000 7f8e16200640 1 -- 172.21.15.32:0/1366487406 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8e100a0570 con 0x7f8e101604a0 2024-08-03T20:47:48.778 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:48.776+0000 7f8e0cc00640 1 -- 172.21.15.32:0/1366487406 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f8e08019020 con 0x7f8e101604a0 2024-08-03T20:47:48.779 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:48.777+0000 7f8e0cc00640 1 --2- 172.21.15.32:0/1366487406 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f8df0077040 0x7f8df0079550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:47:48.779 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:48.777+0000 7f8e0cc00640 1 -- 172.21.15.32:0/1366487406 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f8e08098dc0 con 0x7f8e101604a0 2024-08-03T20:47:48.779 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:48.778+0000 7f8e14e00640 1 --2- 172.21.15.32:0/1366487406 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f8df0077040 0x7f8df0079550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:47:48.780 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:48.779+0000 7f8e14e00640 1 --2- 172.21.15.32:0/1366487406 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f8df0077040 0x7f8df0079550 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f8e04034000 tx=0x7f8e04004730 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:47:48.783 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:48.782+0000 7f8e0cc00640 1 -- 172.21.15.32:0/1366487406 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7f8e08020020 con 0x7f8e101604a0 2024-08-03T20:47:49.032 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:49.030+0000 7f8e16200640 1 -- 172.21.15.32:0/1366487406 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f8e1015d530 con 0x7f8e101604a0 2024-08-03T20:47:49.032 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:49.031+0000 7f8e0cc00640 1 -- 172.21.15.32:0/1366487406 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) ==== 93+0+5220 (secure 0 0 0) 0x7f8e101608b0 con 0x7f8e101604a0 2024-08-03T20:47:49.032 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-03T20:47:49.033 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":8,"btime":"2024-08-03T20:47:38:939479+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4289,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6835/1753256117","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6834","nonce":1753256117},{"type":"v1","addr":"172.21.15.118:6835","nonce":1753256117}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4292,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6837/3055588578","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6836","nonce":3055588578},{"type":"v1","addr":"172.21.15.118:6837","nonce":3055588578}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4338,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6834/3026095815","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":3026095815},{"type":"v1","addr":"172.21.15.32:6834","nonce":3026095815}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-08-03T20:47:35.442482+0000","modified":"2024-08-03T20:47:38.460928+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4341},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4341":{"gid":4341,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.32:6835/246278108","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6833","nonce":246278108},{"type":"v1","addr":"172.21.15.32:6835","nonce":246278108}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4341,"qdb_cluster":[4341]},"id":1}]} 2024-08-03T20:47:49.033 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 8 2024-08-03T20:47:49.035 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:49.033+0000 7f8e16200640 1 -- 172.21.15.32:0/1366487406 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f8df0077040 msgr2=0x7f8df0079550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:47:49.035 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:49.033+0000 7f8e16200640 1 --2- 172.21.15.32:0/1366487406 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f8df0077040 0x7f8df0079550 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f8e04034000 tx=0x7f8e04004730 comp rx=0 tx=0).stop 2024-08-03T20:47:49.035 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:49.033+0000 7f8e16200640 1 -- 172.21.15.32:0/1366487406 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8e101604a0 msgr2=0x7f8e100b1d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:47:49.035 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:49.033+0000 7f8e16200640 1 --2- 172.21.15.32:0/1366487406 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8e101604a0 0x7f8e100b1d00 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f8e0800cad0 tx=0x7f8e0800cf90 comp rx=0 tx=0).stop 2024-08-03T20:47:49.035 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:49.033+0000 7f8e16200640 1 -- 172.21.15.32:0/1366487406 shutdown_connections 2024-08-03T20:47:49.036 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:49.033+0000 7f8e16200640 1 --2- 172.21.15.32:0/1366487406 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f8df0077040 0x7f8df0079550 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:49.036 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:49.033+0000 7f8e16200640 1 --2- 172.21.15.32:0/1366487406 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8e101604a0 0x7f8e100b1d00 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:49.036 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:49.033+0000 7f8e16200640 1 --2- 172.21.15.32:0/1366487406 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f8e10159f70 0x7f8e1009f870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:49.036 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:49.033+0000 7f8e16200640 1 --2- 172.21.15.32:0/1366487406 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8e10070c90 0x7f8e101f5150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:49.036 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:49.033+0000 7f8e16200640 1 -- 172.21.15.32:0/1366487406 >> 172.21.15.32:0/1366487406 conn(0x7f8e100d00f0 msgr2=0x7f8e1015bc90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:47:49.036 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:49.034+0000 7f8e16200640 1 -- 172.21.15.32:0/1366487406 shutdown_connections 2024-08-03T20:47:49.036 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:49.034+0000 7f8e16200640 1 -- 172.21.15.32:0/1366487406 wait complete. 2024-08-03T20:47:49.058 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-08-03T20:47:49.287 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:49.285+0000 7f02b7000640 1 -- 172.21.15.32:0/4185728792 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f02b00b4de0 msgr2=0x7f02b01515c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:47:49.287 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:49.285+0000 7f02b7000640 1 --2- 172.21.15.32:0/4185728792 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f02b00b4de0 0x7f02b01515c0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f02a0009830 tx=0x7f02a002a410 comp rx=0 tx=0).stop 2024-08-03T20:47:49.287 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:49.285+0000 7f02b7000640 1 -- 172.21.15.32:0/4185728792 shutdown_connections 2024-08-03T20:47:49.287 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:49.285+0000 7f02b7000640 1 --2- 172.21.15.32:0/4185728792 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f02b015e880 0x7f02b0073800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:49.287 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:49.285+0000 7f02b7000640 1 --2- 172.21.15.32:0/4185728792 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f02b00b4de0 0x7f02b01515c0 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:49.287 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:49.285+0000 7f02b7000640 1 -- 172.21.15.32:0/4185728792 >> 172.21.15.32:0/4185728792 conn(0x7f02b00b1bd0 msgr2=0x7f02b00b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:47:49.287 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:49.285+0000 7f02b7000640 1 -- 172.21.15.32:0/4185728792 shutdown_connections 2024-08-03T20:47:49.287 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:49.285+0000 7f02b7000640 1 -- 172.21.15.32:0/4185728792 wait complete. 2024-08-03T20:47:49.288 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:49.287+0000 7f02b7000640 1 Processor -- start 2024-08-03T20:47:49.288 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:49.287+0000 7f02b7000640 1 -- start start 2024-08-03T20:47:49.291 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:49.289+0000 7f02b7000640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f02b00733f0 0x7f02b014cf60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:47:49.291 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:49.289+0000 7f02b7000640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f02b00b4de0 0x7f02b014b470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:47:49.291 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:49.289+0000 7f02b7000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f02b01544a0 0x7f02b01548b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:47:49.291 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:49.289+0000 7f02b7000640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f02b006e930 con 0x7f02b01544a0 2024-08-03T20:47:49.291 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:49.289+0000 7f02b7000640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f02b0070ad0 con 0x7f02b00733f0 2024-08-03T20:47:49.291 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:49.289+0000 7f02b7000640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7f02b006eab0 con 0x7f02b00b4de0 2024-08-03T20:47:49.291 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:49.289+0000 7f02b5000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f02b01544a0 0x7f02b01548b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:47:49.291 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:49.289+0000 7f02b5000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f02b01544a0 0x7f02b01548b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:57754/0 (socket says 172.21.15.32:57754) 2024-08-03T20:47:49.291 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:49.289+0000 7f02b5000640 1 -- 172.21.15.32:0/3125328825 learned_addr learned my addr 172.21.15.32:0/3125328825 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:47:49.291 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:49.289+0000 7f02afe00640 1 --2- 172.21.15.32:0/3125328825 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f02b00733f0 0x7f02b014cf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:47:49.292 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:49.289+0000 7f02b5000640 1 -- 172.21.15.32:0/3125328825 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f02b00b4de0 msgr2=0x7f02b014b470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:47:49.292 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:49.289+0000 7f02b5000640 1 --2- 172.21.15.32:0/3125328825 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f02b00b4de0 0x7f02b014b470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:49.292 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:49.289+0000 7f02b5000640 1 -- 172.21.15.32:0/3125328825 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f02b00733f0 msgr2=0x7f02b014cf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:47:49.292 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:49.289+0000 7f02b5000640 1 --2- 172.21.15.32:0/3125328825 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f02b00733f0 0x7f02b014cf60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:49.292 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:49.289+0000 7f02b5000640 1 -- 172.21.15.32:0/3125328825 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f02b0154df0 con 0x7f02b01544a0 2024-08-03T20:47:49.292 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:49.290+0000 7f02b5000640 1 --2- 172.21.15.32:0/3125328825 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f02b01544a0 0x7f02b01548b0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f02a800ea40 tx=0x7f02a800ef00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:47:49.293 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:49.290+0000 7f02a7e00640 1 -- 172.21.15.32:0/3125328825 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02a800ce40 con 0x7f02b01544a0 2024-08-03T20:47:49.293 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:49.290+0000 7f02a7e00640 1 -- 172.21.15.32:0/3125328825 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f02a8004c10 con 0x7f02b01544a0 2024-08-03T20:47:49.293 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:49.290+0000 7f02a7e00640 1 -- 172.21.15.32:0/3125328825 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02a8005020 con 0x7f02b01544a0 2024-08-03T20:47:49.293 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:49.290+0000 7f02b7000640 1 -- 172.21.15.32:0/3125328825 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f02b014b9b0 con 0x7f02b01544a0 2024-08-03T20:47:49.293 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:49.290+0000 7f02b7000640 1 -- 172.21.15.32:0/3125328825 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f02b0130a40 con 0x7f02b01544a0 2024-08-03T20:47:49.293 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:49.291+0000 7f02b7000640 1 -- 172.21.15.32:0/3125328825 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f02b01515c0 con 0x7f02b01544a0 2024-08-03T20:47:49.293 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:49.292+0000 7f02a7e00640 1 -- 172.21.15.32:0/3125328825 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f02a80051c0 con 0x7f02b01544a0 2024-08-03T20:47:49.294 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:49.293+0000 7f02a7e00640 1 --2- 172.21.15.32:0/3125328825 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f0290077010 0x7f0290079520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:47:49.294 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:49.293+0000 7f02a7e00640 1 -- 172.21.15.32:0/3125328825 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f02a8099770 con 0x7f02b01544a0 2024-08-03T20:47:49.295 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:49.293+0000 7f02afe00640 1 --2- 172.21.15.32:0/3125328825 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f0290077010 0x7f0290079520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:47:49.296 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:49.294+0000 7f02afe00640 1 --2- 172.21.15.32:0/3125328825 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f0290077010 0x7f0290079520 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f02a0034000 tx=0x7f02a0004730 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:47:49.299 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:49.298+0000 7f02a7e00640 1 -- 172.21.15.32:0/3125328825 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7f02b0154df0 con 0x7f02b01544a0 2024-08-03T20:47:49.504 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:49.503+0000 7f02b7000640 1 -- 172.21.15.32:0/3125328825 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f02b015c540 con 0x7f02b01544a0 2024-08-03T20:47:49.505 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:49.504+0000 7f02a7e00640 1 -- 172.21.15.32:0/3125328825 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) ==== 74+0+16545 (secure 0 0 0) 0x7f02a80628d0 con 0x7f02b01544a0 2024-08-03T20:47:49.506 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-03T20:47:49.506 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":21,"fsid":"40ab141a-01d7-49b9-ae3c-2edec9876e95","created":"2024-08-03T20:47:14.957362+0000","modified":"2024-08-03T20:47:36.927196+0000","last_up_change":"2024-08-03T20:47:28.874451+0000","last_in_change":"2024-08-03T20:47:20.079442+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-08-03T20:47:28.261300+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-08-03T20:47:33.085381+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-08-03T20:47:34.405408+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5a83e759-9035-438b-889a-0a6af1f0a0d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6816","nonce":3066156116},{"type":"v1","addr":"172.21.15.32:6817","nonce":3066156116}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6818","nonce":3066156116},{"type":"v1","addr":"172.21.15.32:6819","nonce":3066156116}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6822","nonce":3066156116},{"type":"v1","addr":"172.21.15.32:6823","nonce":3066156116}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6820","nonce":3066156116},{"type":"v1","addr":"172.21.15.32:6821","nonce":3066156116}]},"public_addr":"172.21.15.32:6817/3066156116","cluster_addr":"172.21.15.32:6819/3066156116","heartbeat_back_addr":"172.21.15.32:6823/3066156116","heartbeat_front_addr":"172.21.15.32:6821/3066156116","state":["exists","up"]},{"osd":1,"uuid":"d7c5c210-da4f-49f9-b9e7-1dfb02daa4bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6824","nonce":1551241759},{"type":"v1","addr":"172.21.15.32:6825","nonce":1551241759}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6826","nonce":1551241759},{"type":"v1","addr":"172.21.15.32:6827","nonce":1551241759}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6830","nonce":1551241759},{"type":"v1","addr":"172.21.15.32:6831","nonce":1551241759}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6828","nonce":1551241759},{"type":"v1","addr":"172.21.15.32:6829","nonce":1551241759}]},"public_addr":"172.21.15.32:6825/1551241759","cluster_addr":"172.21.15.32:6827/1551241759","heartbeat_back_addr":"172.21.15.32:6831/1551241759","heartbeat_front_addr":"172.21.15.32:6829/1551241759","state":["exists","up"]},{"osd":2,"uuid":"57f2e6de-a201-45c2-8981-8e56a4875ebd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6808","nonce":3806708944},{"type":"v1","addr":"172.21.15.32:6809","nonce":3806708944}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6810","nonce":3806708944},{"type":"v1","addr":"172.21.15.32:6811","nonce":3806708944}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6814","nonce":3806708944},{"type":"v1","addr":"172.21.15.32:6815","nonce":3806708944}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6812","nonce":3806708944},{"type":"v1","addr":"172.21.15.32:6813","nonce":3806708944}]},"public_addr":"172.21.15.32:6809/3806708944","cluster_addr":"172.21.15.32:6811/3806708944","heartbeat_back_addr":"172.21.15.32:6815/3806708944","heartbeat_front_addr":"172.21.15.32:6813/3806708944","state":["exists","up"]},{"osd":3,"uuid":"6439a280-da45-4148-8130-f16323935ff3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6800","nonce":9088303},{"type":"v1","addr":"172.21.15.32:6801","nonce":9088303}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6802","nonce":9088303},{"type":"v1","addr":"172.21.15.32:6803","nonce":9088303}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6806","nonce":9088303},{"type":"v1","addr":"172.21.15.32:6807","nonce":9088303}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6804","nonce":9088303},{"type":"v1","addr":"172.21.15.32:6805","nonce":9088303}]},"public_addr":"172.21.15.32:6801/9088303","cluster_addr":"172.21.15.32:6803/9088303","heartbeat_back_addr":"172.21.15.32:6807/9088303","heartbeat_front_addr":"172.21.15.32:6805/9088303","state":["exists","up"]},{"osd":4,"uuid":"4f902eff-1e18-418d-82c9-2151c69524fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6802","nonce":864752275},{"type":"v1","addr":"172.21.15.118:6803","nonce":864752275}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6804","nonce":864752275},{"type":"v1","addr":"172.21.15.118:6805","nonce":864752275}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6808","nonce":864752275},{"type":"v1","addr":"172.21.15.118:6809","nonce":864752275}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6806","nonce":864752275},{"type":"v1","addr":"172.21.15.118:6807","nonce":864752275}]},"public_addr":"172.21.15.118:6803/864752275","cluster_addr":"172.21.15.118:6805/864752275","heartbeat_back_addr":"172.21.15.118:6809/864752275","heartbeat_front_addr":"172.21.15.118:6807/864752275","state":["exists","up"]},{"osd":5,"uuid":"bfd9feac-dd06-458c-91b9-af7f175f612c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6810","nonce":2724891770},{"type":"v1","addr":"172.21.15.118:6811","nonce":2724891770}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6812","nonce":2724891770},{"type":"v1","addr":"172.21.15.118:6813","nonce":2724891770}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6816","nonce":2724891770},{"type":"v1","addr":"172.21.15.118:6817","nonce":2724891770}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6814","nonce":2724891770},{"type":"v1","addr":"172.21.15.118:6815","nonce":2724891770}]},"public_addr":"172.21.15.118:6811/2724891770","cluster_addr":"172.21.15.118:6813/2724891770","heartbeat_back_addr":"172.21.15.118:6817/2724891770","heartbeat_front_addr":"172.21.15.118:6815/2724891770","state":["exists","up"]},{"osd":6,"uuid":"5ed5a279-9d8e-416d-bbc7-7edd3e6d5705","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6825","nonce":801947386},{"type":"v1","addr":"172.21.15.118:6827","nonce":801947386}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6828","nonce":801947386},{"type":"v1","addr":"172.21.15.118:6829","nonce":801947386}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6832","nonce":801947386},{"type":"v1","addr":"172.21.15.118:6833","nonce":801947386}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6830","nonce":801947386},{"type":"v1","addr":"172.21.15.118:6831","nonce":801947386}]},"public_addr":"172.21.15.118:6827/801947386","cluster_addr":"172.21.15.118:6829/801947386","heartbeat_back_addr":"172.21.15.118:6833/801947386","heartbeat_front_addr":"172.21.15.118:6831/801947386","state":["exists","up"]},{"osd":7,"uuid":"f2086c47-875e-4451-a436-2008d1ea16af","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6818","nonce":653000048},{"type":"v1","addr":"172.21.15.118:6819","nonce":653000048}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6820","nonce":653000048},{"type":"v1","addr":"172.21.15.118:6821","nonce":653000048}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6824","nonce":653000048},{"type":"v1","addr":"172.21.15.118:6826","nonce":653000048}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6822","nonce":653000048},{"type":"v1","addr":"172.21.15.118:6823","nonce":653000048}]},"public_addr":"172.21.15.118:6819/653000048","cluster_addr":"172.21.15.118:6821/653000048","heartbeat_back_addr":"172.21.15.118:6826/653000048","heartbeat_front_addr":"172.21.15.118:6823/653000048","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-03T20:47:25.491994+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-03T20:47:25.587358+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-03T20:47:25.454175+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-03T20:47:25.377734+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-03T20:47:25.412351+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-03T20:47:25.499815+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-03T20:47:25.575044+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-03T20:47:25.552383+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-08-03T20:47:49.509 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:49.506+0000 7f02b7000640 1 -- 172.21.15.32:0/3125328825 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f0290077010 msgr2=0x7f0290079520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:47:49.509 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:49.506+0000 7f02b7000640 1 --2- 172.21.15.32:0/3125328825 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f0290077010 0x7f0290079520 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f02a0034000 tx=0x7f02a0004730 comp rx=0 tx=0).stop 2024-08-03T20:47:49.509 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:49.506+0000 7f02b7000640 1 -- 172.21.15.32:0/3125328825 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f02b01544a0 msgr2=0x7f02b01548b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:47:49.509 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:49.506+0000 7f02b7000640 1 --2- 172.21.15.32:0/3125328825 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f02b01544a0 0x7f02b01548b0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f02a800ea40 tx=0x7f02a800ef00 comp rx=0 tx=0).stop 2024-08-03T20:47:49.509 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:49.506+0000 7f02b7000640 1 -- 172.21.15.32:0/3125328825 shutdown_connections 2024-08-03T20:47:49.509 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:49.507+0000 7f02b7000640 1 --2- 172.21.15.32:0/3125328825 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f0290077010 0x7f0290079520 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:49.510 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:49.507+0000 7f02b7000640 1 --2- 172.21.15.32:0/3125328825 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f02b01544a0 0x7f02b01548b0 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:49.510 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:49.507+0000 7f02b7000640 1 --2- 172.21.15.32:0/3125328825 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f02b00b4de0 0x7f02b014b470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:49.510 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:49.507+0000 7f02b7000640 1 --2- 172.21.15.32:0/3125328825 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f02b00733f0 0x7f02b014cf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:49.510 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:49.507+0000 7f02b7000640 1 -- 172.21.15.32:0/3125328825 >> 172.21.15.32:0/3125328825 conn(0x7f02b00b1bd0 msgr2=0x7f02b015c820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:47:49.510 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:49.507+0000 7f02b7000640 1 -- 172.21.15.32:0/3125328825 shutdown_connections 2024-08-03T20:47:49.510 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:49.507+0000 7f02b7000640 1 -- 172.21.15.32:0/3125328825 wait complete. 2024-08-03T20:47:49.531 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-08-03T20:47:49.765 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:49.756+0000 7fce1c200640 1 -- 172.21.15.32:0/4168407495 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fce1415fff0 msgr2=0x7fce14160400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:47:49.765 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:49.756+0000 7fce1c200640 1 --2- 172.21.15.32:0/4168407495 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fce1415fff0 0x7fce14160400 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fce0c005fd0 tx=0x7fce0c02a8b0 comp rx=0 tx=0).stop 2024-08-03T20:47:49.765 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:49.757+0000 7fce1c200640 1 -- 172.21.15.32:0/4168407495 shutdown_connections 2024-08-03T20:47:49.765 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:49.757+0000 7fce1c200640 1 --2- 172.21.15.32:0/4168407495 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fce1415fff0 0x7fce14160400 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:49.765 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:49.757+0000 7fce1c200640 1 --2- 172.21.15.32:0/4168407495 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fce1406e930 0x7fce14144f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:49.765 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:49.757+0000 7fce1c200640 1 -- 172.21.15.32:0/4168407495 >> 172.21.15.32:0/4168407495 conn(0x7fce140b1bd0 msgr2=0x7fce140b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:47:49.766 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:49.757+0000 7fce1c200640 1 -- 172.21.15.32:0/4168407495 shutdown_connections 2024-08-03T20:47:49.766 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:49.757+0000 7fce1c200640 1 -- 172.21.15.32:0/4168407495 wait complete. 2024-08-03T20:47:49.766 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:49.759+0000 7fce1c200640 1 Processor -- start 2024-08-03T20:47:49.766 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:49.759+0000 7fce1c200640 1 -- start start 2024-08-03T20:47:49.766 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:49.760+0000 7fce1c200640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fce1406e930 0x7fce1413c9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:47:49.766 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:49.760+0000 7fce1c200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fce14073e60 0x7fce1406ef70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:47:49.766 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:49.760+0000 7fce1c200640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fce1415fff0 0x7fce1406f4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:47:49.766 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:49.760+0000 7fce1c200640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fce140cf840 con 0x7fce14073e60 2024-08-03T20:47:49.766 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:49.760+0000 7fce1c200640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7fce14145f30 con 0x7fce1415fff0 2024-08-03T20:47:49.766 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:49.760+0000 7fce1c200640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7fce140cf9c0 con 0x7fce1406e930 2024-08-03T20:47:49.767 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:49.761+0000 7fce18e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fce14073e60 0x7fce1406ef70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:47:49.767 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:49.761+0000 7fce18e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fce14073e60 0x7fce1406ef70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:57764/0 (socket says 172.21.15.32:57764) 2024-08-03T20:47:49.767 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:49.761+0000 7fce18e00640 1 -- 172.21.15.32:0/1070628375 learned_addr learned my addr 172.21.15.32:0/1070628375 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:47:49.767 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:49.761+0000 7fce19800640 1 --2- 172.21.15.32:0/1070628375 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fce1406e930 0x7fce1413c9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:47:49.767 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:49.761+0000 7fce1a200640 1 --2- 172.21.15.32:0/1070628375 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fce1415fff0 0x7fce1406f4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:47:49.767 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:49.761+0000 7fce18e00640 1 -- 172.21.15.32:0/1070628375 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fce1406e930 msgr2=0x7fce1413c9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:47:49.767 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:49.761+0000 7fce18e00640 1 --2- 172.21.15.32:0/1070628375 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fce1406e930 0x7fce1413c9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:49.767 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:49.761+0000 7fce18e00640 1 -- 172.21.15.32:0/1070628375 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fce1415fff0 msgr2=0x7fce1406f4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:47:49.767 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:49.761+0000 7fce18e00640 1 --2- 172.21.15.32:0/1070628375 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fce1415fff0 0x7fce1406f4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:49.767 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:49.761+0000 7fce18e00640 1 -- 172.21.15.32:0/1070628375 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fce141f5290 con 0x7fce14073e60 2024-08-03T20:47:49.767 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:49.761+0000 7fce18e00640 1 --2- 172.21.15.32:0/1070628375 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fce14073e60 0x7fce1406ef70 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fce0000c3f0 tx=0x7fce0000c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:47:49.767 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:49.762+0000 7fce11600640 1 -- 172.21.15.32:0/1070628375 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce00017070 con 0x7fce14073e60 2024-08-03T20:47:49.767 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:49.762+0000 7fce11600640 1 -- 172.21.15.32:0/1070628375 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fce000035f0 con 0x7fce14073e60 2024-08-03T20:47:49.768 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:49.762+0000 7fce11600640 1 -- 172.21.15.32:0/1070628375 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce00003a40 con 0x7fce14073e60 2024-08-03T20:47:49.768 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:49.762+0000 7fce1c200640 1 -- 172.21.15.32:0/1070628375 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fce141f5530 con 0x7fce14073e60 2024-08-03T20:47:49.768 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:49.762+0000 7fce1c200640 1 -- 172.21.15.32:0/1070628375 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fce141f5af0 con 0x7fce14073e60 2024-08-03T20:47:49.768 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:49.763+0000 7fce1c200640 1 -- 172.21.15.32:0/1070628375 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fce14160bc0 con 0x7fce14073e60 2024-08-03T20:47:49.769 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:49.764+0000 7fce11600640 1 -- 172.21.15.32:0/1070628375 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7fce0001a070 con 0x7fce14073e60 2024-08-03T20:47:49.769 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:49.765+0000 7fce11600640 1 --2- 172.21.15.32:0/1070628375 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fcdf4077010 0x7fcdf4079520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:47:49.769 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:49.765+0000 7fce11600640 1 -- 172.21.15.32:0/1070628375 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7fce00097c80 con 0x7fce14073e60 2024-08-03T20:47:49.769 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:49.766+0000 7fce19800640 1 --2- 172.21.15.32:0/1070628375 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fcdf4077010 0x7fcdf4079520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:47:49.769 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:49.766+0000 7fce19800640 1 --2- 172.21.15.32:0/1070628375 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fcdf4077010 0x7fcdf4079520 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fce08009b30 tx=0x7fce08005f90 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:47:49.771 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:49.770+0000 7fce11600640 1 -- 172.21.15.32:0/1070628375 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7fce00060820 con 0x7fce14073e60 2024-08-03T20:47:50.014 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:50.012+0000 7fce1c200640 1 -- 172.21.15.32:0/1070628375 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fce14145890 con 0x7fce14073e60 2024-08-03T20:47:50.015 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:50.013+0000 7fce11600640 1 -- 172.21.15.32:0/1070628375 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) ==== 93+0+5220 (secure 0 0 0) 0x7fce000690b0 con 0x7fce14073e60 2024-08-03T20:47:50.015 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-03T20:47:50.015 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":8,"btime":"2024-08-03T20:47:38:939479+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4289,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6835/1753256117","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6834","nonce":1753256117},{"type":"v1","addr":"172.21.15.118:6835","nonce":1753256117}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4292,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6837/3055588578","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6836","nonce":3055588578},{"type":"v1","addr":"172.21.15.118:6837","nonce":3055588578}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4338,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6834/3026095815","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":3026095815},{"type":"v1","addr":"172.21.15.32:6834","nonce":3026095815}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-08-03T20:47:35.442482+0000","modified":"2024-08-03T20:47:38.460928+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4341},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4341":{"gid":4341,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.32:6835/246278108","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6833","nonce":246278108},{"type":"v1","addr":"172.21.15.32:6835","nonce":246278108}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4341,"qdb_cluster":[4341]},"id":1}]} 2024-08-03T20:47:50.015 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 8 2024-08-03T20:47:50.017 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:50.015+0000 7fce1c200640 1 -- 172.21.15.32:0/1070628375 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fcdf4077010 msgr2=0x7fcdf4079520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:47:50.017 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:50.015+0000 7fce1c200640 1 --2- 172.21.15.32:0/1070628375 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fcdf4077010 0x7fcdf4079520 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fce08009b30 tx=0x7fce08005f90 comp rx=0 tx=0).stop 2024-08-03T20:47:50.017 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:50.015+0000 7fce1c200640 1 -- 172.21.15.32:0/1070628375 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fce14073e60 msgr2=0x7fce1406ef70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:47:50.017 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:50.015+0000 7fce1c200640 1 --2- 172.21.15.32:0/1070628375 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fce14073e60 0x7fce1406ef70 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fce0000c3f0 tx=0x7fce0000c8b0 comp rx=0 tx=0).stop 2024-08-03T20:47:50.017 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:50.015+0000 7fce1c200640 1 -- 172.21.15.32:0/1070628375 shutdown_connections 2024-08-03T20:47:50.017 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:50.015+0000 7fce1c200640 1 --2- 172.21.15.32:0/1070628375 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fcdf4077010 0x7fcdf4079520 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:50.018 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:50.015+0000 7fce1c200640 1 --2- 172.21.15.32:0/1070628375 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fce1415fff0 0x7fce1406f4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:50.018 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:50.015+0000 7fce1c200640 1 --2- 172.21.15.32:0/1070628375 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fce14073e60 0x7fce1406ef70 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:50.018 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:50.015+0000 7fce1c200640 1 --2- 172.21.15.32:0/1070628375 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fce1406e930 0x7fce1413c9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:50.018 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:50.015+0000 7fce1c200640 1 -- 172.21.15.32:0/1070628375 >> 172.21.15.32:0/1070628375 conn(0x7fce140b1bd0 msgr2=0x7fce140b39c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:47:50.018 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:50.015+0000 7fce1c200640 1 -- 172.21.15.32:0/1070628375 shutdown_connections 2024-08-03T20:47:50.018 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:50.015+0000 7fce1c200640 1 -- 172.21.15.32:0/1070628375 wait complete. 2024-08-03T20:47:50.442 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 8, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-08-03T20:47:35.442482+0000', 'modified': '2024-08-03T20:47:38.460928+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 4341}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4341': {'gid': 4341, 'name': 'c', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.32:6835/246278108', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6833', 'nonce': 246278108}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 246278108}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4341, 'qdb_cluster': [4341]} 2024-08-03T20:47:50.442 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-08-03T20:47:50.442 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-08-03T20:47:50.673 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:50.671+0000 7f7f79600640 1 -- 172.21.15.32:0/2528021699 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7f740d3960 msgr2=0x7f7f740cf710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:47:50.673 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:50.671+0000 7f7f79600640 1 --2- 172.21.15.32:0/2528021699 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7f740d3960 0x7f7f740cf710 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f7f64005fd0 tx=0x7f7f6402a560 comp rx=0 tx=0).stop 2024-08-03T20:47:50.673 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:50.671+0000 7f7f79600640 1 -- 172.21.15.32:0/2528021699 shutdown_connections 2024-08-03T20:47:50.674 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:50.671+0000 7f7f79600640 1 --2- 172.21.15.32:0/2528021699 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f7f740b6510 0x7f7f740b7850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:50.674 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:50.671+0000 7f7f79600640 1 --2- 172.21.15.32:0/2528021699 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7f740d3960 0x7f7f740cf710 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:50.674 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:50.671+0000 7f7f79600640 1 -- 172.21.15.32:0/2528021699 >> 172.21.15.32:0/2528021699 conn(0x7f7f740d0fe0 msgr2=0x7f7f740d13f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:47:50.674 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:50.671+0000 7f7f79600640 1 -- 172.21.15.32:0/2528021699 shutdown_connections 2024-08-03T20:47:50.674 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:50.672+0000 7f7f79600640 1 -- 172.21.15.32:0/2528021699 wait complete. 2024-08-03T20:47:50.675 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:50.674+0000 7f7f79600640 1 Processor -- start 2024-08-03T20:47:50.675 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:50.674+0000 7f7f79600640 1 -- start start 2024-08-03T20:47:50.678 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:50.675+0000 7f7f79600640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7f740cfc50 0x7f7f740b5840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:47:50.678 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:50.675+0000 7f7f79600640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f7f740d3960 0x7f7f7415c6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:47:50.678 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:50.675+0000 7f7f79600640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f7f741f98a0 0x7f7f7415cc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:47:50.678 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:50.675+0000 7f7f79600640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f7f740b7f10 con 0x7f7f740cfc50 2024-08-03T20:47:50.678 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:50.675+0000 7f7f79600640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f7f740b7d90 con 0x7f7f740d3960 2024-08-03T20:47:50.678 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:50.675+0000 7f7f79600640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7f7f740b8090 con 0x7f7f741f98a0 2024-08-03T20:47:50.678 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:50.675+0000 7f7f72a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7f740cfc50 0x7f7f740b5840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:47:50.679 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:50.675+0000 7f7f72a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7f740cfc50 0x7f7f740b5840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:52372/0 (socket says 172.21.15.32:52372) 2024-08-03T20:47:50.679 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:50.675+0000 7f7f72a00640 1 -- 172.21.15.32:0/262238558 learned_addr learned my addr 172.21.15.32:0/262238558 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:47:50.679 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:50.675+0000 7f7f72000640 1 --2- 172.21.15.32:0/262238558 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f7f740d3960 0x7f7f7415c6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:47:50.679 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:50.676+0000 7f7f73400640 1 --2- 172.21.15.32:0/262238558 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f7f741f98a0 0x7f7f7415cc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:47:50.679 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:50.676+0000 7f7f72a00640 1 -- 172.21.15.32:0/262238558 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f7f741f98a0 msgr2=0x7f7f7415cc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:47:50.679 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:50.676+0000 7f7f72a00640 1 --2- 172.21.15.32:0/262238558 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f7f741f98a0 0x7f7f7415cc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:50.679 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:50.676+0000 7f7f72a00640 1 -- 172.21.15.32:0/262238558 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f7f740d3960 msgr2=0x7f7f7415c6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:47:50.679 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:50.676+0000 7f7f72a00640 1 --2- 172.21.15.32:0/262238558 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f7f740d3960 0x7f7f7415c6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:50.679 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:50.676+0000 7f7f72a00640 1 -- 172.21.15.32:0/262238558 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7f740b5d80 con 0x7f7f740cfc50 2024-08-03T20:47:50.679 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:50.676+0000 7f7f72a00640 1 --2- 172.21.15.32:0/262238558 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7f740cfc50 0x7f7f740b5840 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f7f64034040 tx=0x7f7f640039e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:47:50.679 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:50.676+0000 7f7f6aa00640 1 -- 172.21.15.32:0/262238558 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f6403e030 con 0x7f7f740cfc50 2024-08-03T20:47:50.679 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:50.676+0000 7f7f6aa00640 1 -- 172.21.15.32:0/262238558 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f7f640047a0 con 0x7f7f740cfc50 2024-08-03T20:47:50.679 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:50.677+0000 7f7f79600640 1 -- 172.21.15.32:0/262238558 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7f740b6d20 con 0x7f7f740cfc50 2024-08-03T20:47:50.680 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:50.677+0000 7f7f6aa00640 1 -- 172.21.15.32:0/262238558 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f64035040 con 0x7f7f740cfc50 2024-08-03T20:47:50.680 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:50.677+0000 7f7f79600640 1 -- 172.21.15.32:0/262238558 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7f740b6a40 con 0x7f7f740cfc50 2024-08-03T20:47:50.680 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:50.679+0000 7f7f6aa00640 1 -- 172.21.15.32:0/262238558 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f7f64041030 con 0x7f7f740cfc50 2024-08-03T20:47:50.681 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:50.680+0000 7f7f6aa00640 1 --2- 172.21.15.32:0/262238558 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f7f54077040 0x7f7f54079550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:47:50.682 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:50.680+0000 7f7f6aa00640 1 -- 172.21.15.32:0/262238558 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f7f640b9100 con 0x7f7f740cfc50 2024-08-03T20:47:50.682 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:50.680+0000 7f7f72000640 1 --2- 172.21.15.32:0/262238558 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f7f54077040 0x7f7f54079550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:47:50.682 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:50.681+0000 7f7f807c7740 1 Processor -- start 2024-08-03T20:47:50.682 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:50.681+0000 7f7f807c7740 1 -- start start 2024-08-03T20:47:50.682 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:50.681+0000 7f7f72000640 1 --2- 172.21.15.32:0/262238558 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f7f54077040 0x7f7f54079550 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f7f740b8b10 tx=0x7f7f60009060 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:47:50.685 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:50.682+0000 7f7f807c7740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a7f7fcf270 0x55a7f83b9e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:47:50.685 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:50.682+0000 7f7f807c7740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55a7f7fa63b0 0x55a7f838ece0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:47:50.685 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:50.682+0000 7f7f807c7740 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55a7f7fae590 0x55a7f838f220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:47:50.685 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:50.682+0000 7f7f807c7740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55a7f82e7a60 con 0x55a7f7fcf270 2024-08-03T20:47:50.685 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:50.682+0000 7f7f807c7740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x55a7f82d6f60 con 0x55a7f7fa63b0 2024-08-03T20:47:50.685 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:50.682+0000 7f7f807c7740 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x55a7f82dafa0 con 0x55a7f7fae590 2024-08-03T20:47:50.685 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:50.683+0000 7f7f72a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a7f7fcf270 0x55a7f83b9e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:47:50.685 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:50.683+0000 7f7f72a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a7f7fcf270 0x55a7f83b9e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:52380/0 (socket says 172.21.15.32:52380) 2024-08-03T20:47:50.685 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:50.683+0000 7f7f72a00640 1 -- 172.21.15.32:0/2003699021 learned_addr learned my addr 172.21.15.32:0/2003699021 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:47:50.685 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:50.683+0000 7f7f72000640 1 --2- 172.21.15.32:0/2003699021 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55a7f7fa63b0 0x55a7f838ece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:47:50.685 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:50.683+0000 7f7f73400640 1 --2- 172.21.15.32:0/2003699021 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55a7f7fae590 0x55a7f838f220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:47:50.686 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:50.683+0000 7f7f72a00640 1 -- 172.21.15.32:0/2003699021 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55a7f7fae590 msgr2=0x55a7f838f220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:47:50.686 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:50.683+0000 7f7f72a00640 1 --2- 172.21.15.32:0/2003699021 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55a7f7fae590 0x55a7f838f220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:50.686 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:50.683+0000 7f7f72a00640 1 -- 172.21.15.32:0/2003699021 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55a7f7fa63b0 msgr2=0x55a7f838ece0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:47:50.686 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:50.683+0000 7f7f72a00640 1 --2- 172.21.15.32:0/2003699021 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55a7f7fa63b0 0x55a7f838ece0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:50.686 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:50.683+0000 7f7f72a00640 1 -- 172.21.15.32:0/2003699021 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a7f803a850 con 0x55a7f7fcf270 2024-08-03T20:47:50.686 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:50.683+0000 7f7f72a00640 1 --2- 172.21.15.32:0/2003699021 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a7f7fcf270 0x55a7f83b9e90 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f7f640819a0 tx=0x7f7f640815f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2bade6a21d4a40ee server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:47:50.686 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:50.683+0000 7f7f5f000640 1 -- 172.21.15.32:0/2003699021 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f64048040 con 0x55a7f7fcf270 2024-08-03T20:47:50.686 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:50.683+0000 7f7f5f000640 1 -- 172.21.15.32:0/2003699021 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f7f64081ca0 con 0x55a7f7fcf270 2024-08-03T20:47:50.686 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:50.683+0000 7f7f5f000640 1 -- 172.21.15.32:0/2003699021 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f6408a9e0 con 0x55a7f7fcf270 2024-08-03T20:47:50.687 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:50.684+0000 7f7f807c7740 1 -- 172.21.15.32:0/2003699021 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a7f7fcf270 msgr2=0x55a7f83b9e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:47:50.687 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:50.684+0000 7f7f807c7740 1 --2- 172.21.15.32:0/2003699021 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a7f7fcf270 0x55a7f83b9e90 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f7f640819a0 tx=0x7f7f640815f0 comp rx=0 tx=0).stop 2024-08-03T20:47:50.688 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:50.684+0000 7f7f807c7740 1 -- 172.21.15.32:0/2003699021 shutdown_connections 2024-08-03T20:47:50.688 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:50.684+0000 7f7f807c7740 1 --2- 172.21.15.32:0/2003699021 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55a7f7fa63b0 0x55a7f838ece0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:50.688 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:50.684+0000 7f7f807c7740 1 --2- 172.21.15.32:0/2003699021 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55a7f7fae590 0x55a7f838f220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:50.688 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:50.684+0000 7f7f807c7740 1 --2- 172.21.15.32:0/2003699021 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a7f7fcf270 0x55a7f83b9e90 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:50.688 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:50.684+0000 7f7f807c7740 1 -- 172.21.15.32:0/2003699021 >> 172.21.15.32:0/2003699021 conn(0x55a7f7fc33c0 msgr2=0x55a7f8375180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:47:50.688 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:50.684+0000 7f7f807c7740 1 -- 172.21.15.32:0/2003699021 shutdown_connections 2024-08-03T20:47:50.688 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:50.684+0000 7f7f807c7740 1 -- 172.21.15.32:0/2003699021 wait complete. 2024-08-03T20:47:50.688 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:50.685+0000 7f7f807c7740 1 -- start start 2024-08-03T20:47:50.688 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:50.686+0000 7f7f807c7740 1 Processor -- start 2024-08-03T20:47:50.691 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:50.687+0000 7f7f807c7740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55a7f7fa63b0 0x55a7f83b43d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:47:50.692 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:50.687+0000 7f7f807c7740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a7f7fae590 0x55a7f838d800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:47:50.692 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:50.687+0000 7f7f807c7740 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55a7f7fcf270 0x55a7f83ccab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:47:50.692 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:50.687+0000 7f7f807c7740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55a7f8049620 con 0x55a7f7fae590 2024-08-03T20:47:50.692 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:50.687+0000 7f7f807c7740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x55a7f8274160 con 0x55a7f7fa63b0 2024-08-03T20:47:50.692 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:50.687+0000 7f7f807c7740 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x55a7f82e7a60 con 0x55a7f7fcf270 2024-08-03T20:47:50.692 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:50.687+0000 7f7f807c7740 10 client.0 _ll_register_callbacks cb 0x55a7f82836b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-03T20:47:50.692 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:50.687+0000 7f7f72000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a7f7fae590 0x55a7f838d800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:47:50.692 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:50.688+0000 7f7f72000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a7f7fae590 0x55a7f838d800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:52396/0 (socket says 172.21.15.32:52396) 2024-08-03T20:47:50.692 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:50.688+0000 7f7f72000640 1 -- 172.21.15.32:0/2314475601 learned_addr learned my addr 172.21.15.32:0/2314475601 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:47:50.692 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:50.688+0000 7f7f72a00640 1 --2- 172.21.15.32:0/2314475601 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55a7f7fa63b0 0x55a7f83b43d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:47:50.692 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:50.688+0000 7f7f73400640 1 --2- 172.21.15.32:0/2314475601 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55a7f7fcf270 0x55a7f83ccab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:47:50.693 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:50.688+0000 7f7f72000640 1 -- 172.21.15.32:0/2314475601 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55a7f7fcf270 msgr2=0x55a7f83ccab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:47:50.693 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:50.688+0000 7f7f72000640 1 --2- 172.21.15.32:0/2314475601 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55a7f7fcf270 0x55a7f83ccab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:50.693 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:50.688+0000 7f7f72000640 1 -- 172.21.15.32:0/2314475601 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55a7f7fa63b0 msgr2=0x55a7f83b43d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:47:50.693 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:50.688+0000 7f7f72000640 1 --2- 172.21.15.32:0/2314475601 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55a7f7fa63b0 0x55a7f83b43d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:50.693 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:50.688+0000 7f7f72000640 1 -- 172.21.15.32:0/2314475601 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a7f80b16b0 con 0x55a7f7fae590 2024-08-03T20:47:50.693 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:50.688+0000 7f7f72000640 1 --2- 172.21.15.32:0/2314475601 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a7f7fae590 0x55a7f838d800 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f7f600154d0 tx=0x7f7f600046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=67de94766c51408b server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:47:50.693 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:50.688+0000 7f7f53400640 10 client.0 ms_handle_connect on v2:172.21.15.32:3300/0 2024-08-03T20:47:50.693 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:50.689+0000 7f7f53400640 1 -- 172.21.15.32:0/2314475601 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f60025070 con 0x55a7f7fae590 2024-08-03T20:47:50.693 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:50.689+0000 7f7f53400640 1 -- 172.21.15.32:0/2314475601 <== mon.0 v2:172.21.15.32:3300/0 2 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f7f60013720 con 0x55a7f7fae590 2024-08-03T20:47:50.693 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:50.689+0000 7f7f807c7740 1 -- 172.21.15.32:0/2314475601 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7f64081ca0 con 0x55a7f7fae590 2024-08-03T20:47:50.693 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:50.689+0000 7f7f53400640 1 -- 172.21.15.32:0/2314475601 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_get_version_reply(handle=1 version=8) ==== 24+0+0 (secure 0 0 0) 0x7f7f6001b040 con 0x55a7f7fae590 2024-08-03T20:47:50.693 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:50.689+0000 7f7f807c7740 10 client.4653 fetch_fsmap learned FSMap version 8 2024-08-03T20:47:50.694 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:50.689+0000 7f7f807c7740 1 -- 172.21.15.32:0/2314475601 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,fsmap=8,monmap=2+}) -- 0x55a7f83b4b20 con 0x55a7f7fae590 2024-08-03T20:47:50.694 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:50.689+0000 7f7f53400640 1 -- 172.21.15.32:0/2314475601 <== mon.0 v2:172.21.15.32:3300/0 4 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f7f64081ca0 con 0x55a7f7fae590 2024-08-03T20:47:50.694 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:50.689+0000 7f7f53400640 1 -- 172.21.15.32:0/2314475601 <== mon.0 v2:172.21.15.32:3300/0 5 ==== fsmap(e 8) ==== 3502+0+0 (secure 0 0 0) 0x7f7f60013a10 con 0x55a7f7fae590 2024-08-03T20:47:50.694 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:50.689+0000 7f7f807c7740 10 client.4653 fetch_fsmap finished waiting for FSMap version 8 2024-08-03T20:47:50.694 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:50.689+0000 7f7f807c7740 10 client.4653 resolve_mds: resolved name 'c' to daemon mds.c 2024-08-03T20:47:50.694 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:50.689+0000 7f7f807c7740 20 client.4653 populate_metadata read hostname 'smithi032' 2024-08-03T20:47:50.694 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:50.689+0000 7f7f807c7740 1 --2- 172.21.15.32:0/2314475601 >> [v2:172.21.15.32:6833/246278108,v1:172.21.15.32:6835/246278108] conn(0x55a7f7f8de40 0x55a7f83b0de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:47:50.694 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:50.689+0000 7f7f807c7740 4 client.4653 mds_command: new command op to 4341 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-08-03T20:47:50.694 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:50.689+0000 7f7f807c7740 1 -- 172.21.15.32:0/2314475601 --> [v2:172.21.15.32:6833/246278108,v1:172.21.15.32:6835/246278108] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a7f82e99b0 con 0x55a7f7f8de40 2024-08-03T20:47:50.694 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:50.690+0000 7f7f72a00640 1 --2- 172.21.15.32:0/2314475601 >> [v2:172.21.15.32:6833/246278108,v1:172.21.15.32:6835/246278108] conn(0x55a7f7f8de40 0x55a7f83b0de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:47:50.694 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:50.690+0000 7f7f72a00640 1 --2- 172.21.15.32:0/2314475601 >> [v2:172.21.15.32:6833/246278108,v1:172.21.15.32:6835/246278108] conn(0x55a7f7f8de40 0x55a7f83b0de0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3ba55e995d36a55a server_cookie=66393b2755d1bf5f in_seq=0 out_seq=0 2024-08-03T20:47:50.694 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:50.690+0000 7f7f53400640 10 client.4653 ms_handle_connect on v2:172.21.15.32:6833/246278108 2024-08-03T20:47:50.695 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:50.691+0000 7f7f53400640 1 -- 172.21.15.32:0/2314475601 <== mds.0 v2:172.21.15.32:6833/246278108 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55a7f82e99b0 con 0x55a7f7f8de40 2024-08-03T20:47:50.696 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:50.691+0000 7f7f53400640 10 client.4653 handle_command_reply: tid=0 2024-08-03T20:47:50.696 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:50.691+0000 7f7f807c7740 1 client.4653 shutdown 2024-08-03T20:47:50.696 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:50.691+0000 7f7f807c7740 1 -- 172.21.15.32:0/2314475601 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a7f7fae590 msgr2=0x55a7f838d800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:47:50.696 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:50.691+0000 7f7f807c7740 1 --2- 172.21.15.32:0/2314475601 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a7f7fae590 0x55a7f838d800 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f7f600154d0 tx=0x7f7f600046b0 comp rx=0 tx=0).stop 2024-08-03T20:47:50.696 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:50.691+0000 7f7f807c7740 1 -- 172.21.15.32:0/2314475601 shutdown_connections 2024-08-03T20:47:50.696 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:50.691+0000 7f7f807c7740 1 --2- 172.21.15.32:0/2314475601 >> [v2:172.21.15.32:6833/246278108,v1:172.21.15.32:6835/246278108] conn(0x55a7f7f8de40 0x55a7f83b0de0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:50.696 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:50.691+0000 7f7f807c7740 1 --2- 172.21.15.32:0/2314475601 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55a7f7fcf270 0x55a7f83ccab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:50.696 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:50.691+0000 7f7f807c7740 1 --2- 172.21.15.32:0/2314475601 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a7f7fae590 0x55a7f838d800 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:50.696 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:50.691+0000 7f7f53400640 0 client.4653 ms_handle_reset on v2:172.21.15.32:6833/246278108 2024-08-03T20:47:50.696 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:50.691+0000 7f7f807c7740 1 --2- 172.21.15.32:0/2314475601 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55a7f7fa63b0 0x55a7f83b43d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:50.696 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:50.691+0000 7f7f807c7740 1 -- 172.21.15.32:0/2314475601 >> 172.21.15.32:0/2314475601 conn(0x55a7f7fc33c0 msgr2=0x55a7f838f2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:47:50.696 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:50.691+0000 7f7f807c7740 1 -- 172.21.15.32:0/2314475601 shutdown_connections 2024-08-03T20:47:50.696 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:50.691+0000 7f7f807c7740 1 -- 172.21.15.32:0/2314475601 wait complete. 2024-08-03T20:47:50.696 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:50.692+0000 7f7f807c7740 20 client.4653 trim_cache size 0 max 16384 2024-08-03T20:47:50.713 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:50.712+0000 7f7f807c7740 1 Processor -- start 2024-08-03T20:47:50.713 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:50.712+0000 7f7f807c7740 1 -- start start 2024-08-03T20:47:50.715 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:50.713+0000 7f7f807c7740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a7f7fa63b0 0x55a7f83b1df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:47:50.716 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:50.713+0000 7f7f807c7740 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55a7f7fae590 0x55a7f83c82e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:47:50.716 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:50.713+0000 7f7f807c7740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55a7f7fcf270 0x55a7f8430ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:47:50.716 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:50.713+0000 7f7f807c7740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f7f7414ba60 con 0x55a7f7fa63b0 2024-08-03T20:47:50.716 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:50.713+0000 7f7f807c7740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f7f74149cb0 con 0x55a7f7fcf270 2024-08-03T20:47:50.716 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:50.713+0000 7f7f807c7740 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7f7f7414a300 con 0x55a7f7fae590 2024-08-03T20:47:50.716 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:50.713+0000 7f7f72a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a7f7fa63b0 0x55a7f83b1df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:47:50.716 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:50.713+0000 7f7f72a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a7f7fa63b0 0x55a7f83b1df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:52406/0 (socket says 172.21.15.32:52406) 2024-08-03T20:47:50.716 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:50.713+0000 7f7f72a00640 1 -- 172.21.15.32:0/3315497265 learned_addr learned my addr 172.21.15.32:0/3315497265 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:47:50.716 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:50.714+0000 7f7f72a00640 1 -- 172.21.15.32:0/3315497265 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55a7f7fae590 msgr2=0x55a7f83c82e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:47:50.716 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:50.714+0000 7f7f72a00640 1 --2- 172.21.15.32:0/3315497265 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55a7f7fae590 0x55a7f83c82e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:50.716 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:50.714+0000 7f7f73400640 1 --2- 172.21.15.32:0/3315497265 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55a7f7fcf270 0x55a7f8430ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:47:50.717 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:50.714+0000 7f7f72a00640 1 -- 172.21.15.32:0/3315497265 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55a7f7fcf270 msgr2=0x55a7f8430ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:47:50.717 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:50.714+0000 7f7f72a00640 1 --2- 172.21.15.32:0/3315497265 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55a7f7fcf270 0x55a7f8430ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:50.717 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:50.714+0000 7f7f72a00640 1 -- 172.21.15.32:0/3315497265 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a7f82e0260 con 0x55a7f7fa63b0 2024-08-03T20:47:50.717 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:50.714+0000 7f7f73400640 1 --2- 172.21.15.32:0/3315497265 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55a7f7fcf270 0x55a7f8430ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-03T20:47:50.717 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:50.714+0000 7f7f72a00640 1 --2- 172.21.15.32:0/3315497265 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a7f7fa63b0 0x55a7f83b1df0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f7f64005f70 tx=0x7f7f64009260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=166fd81837e95940 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:47:50.717 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:50.714+0000 7f7f5e600640 1 -- 172.21.15.32:0/3315497265 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f640b5070 con 0x55a7f7fa63b0 2024-08-03T20:47:50.717 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:50.714+0000 7f7f5e600640 1 -- 172.21.15.32:0/3315497265 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f7f640861f0 con 0x55a7f7fa63b0 2024-08-03T20:47:50.717 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:50.714+0000 7f7f5e600640 1 -- 172.21.15.32:0/3315497265 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f64048040 con 0x55a7f7fa63b0 2024-08-03T20:47:50.718 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:50.715+0000 7f7f807c7740 1 -- 172.21.15.32:0/3315497265 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a7f7fa63b0 msgr2=0x55a7f83b1df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:47:50.718 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:50.715+0000 7f7f807c7740 1 --2- 172.21.15.32:0/3315497265 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a7f7fa63b0 0x55a7f83b1df0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f7f64005f70 tx=0x7f7f64009260 comp rx=0 tx=0).stop 2024-08-03T20:47:50.718 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:50.715+0000 7f7f807c7740 1 -- 172.21.15.32:0/3315497265 shutdown_connections 2024-08-03T20:47:50.718 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:50.715+0000 7f7f807c7740 1 --2- 172.21.15.32:0/3315497265 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55a7f7fcf270 0x55a7f8430ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:50.718 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:50.715+0000 7f7f807c7740 1 --2- 172.21.15.32:0/3315497265 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55a7f7fae590 0x55a7f83c82e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:50.719 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:50.715+0000 7f7f807c7740 1 --2- 172.21.15.32:0/3315497265 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a7f7fa63b0 0x55a7f83b1df0 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:50.719 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:50.715+0000 7f7f807c7740 1 -- 172.21.15.32:0/3315497265 >> 172.21.15.32:0/3315497265 conn(0x55a7f7fc33c0 msgr2=0x55a7f838d1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:47:50.719 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:50.715+0000 7f7f807c7740 1 -- 172.21.15.32:0/3315497265 shutdown_connections 2024-08-03T20:47:50.719 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:50.715+0000 7f7f807c7740 1 -- 172.21.15.32:0/3315497265 wait complete. 2024-08-03T20:47:50.719 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:50.716+0000 7f7f807c7740 1 -- start start 2024-08-03T20:47:50.719 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:50.716+0000 7f7f807c7740 1 Processor -- start 2024-08-03T20:47:50.723 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:50.718+0000 7f7f807c7740 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55a7f7fa63b0 0x55a7f842c800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:47:50.723 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:50.718+0000 7f7f807c7740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a7f7fae590 0x55a7f83d0120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:47:50.723 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:50.718+0000 7f7f807c7740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55a7f7fcf270 0x55a7f83d0660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:47:50.723 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:50.718+0000 7f7f807c7740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55a7f83cd130 con 0x55a7f7fae590 2024-08-03T20:47:50.723 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:50.718+0000 7f7f807c7740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x55a7f838dd40 con 0x55a7f7fcf270 2024-08-03T20:47:50.723 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:50.718+0000 7f7f807c7740 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x55a7f83b6770 con 0x55a7f7fa63b0 2024-08-03T20:47:50.723 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:50.718+0000 7f7f807c7740 10 client.0 _ll_register_callbacks cb 0x7f7f6000ed30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-03T20:47:50.724 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:50.718+0000 7f7f72000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a7f7fae590 0x55a7f83d0120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:47:50.724 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:50.718+0000 7f7f72000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a7f7fae590 0x55a7f83d0120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:52422/0 (socket says 172.21.15.32:52422) 2024-08-03T20:47:50.724 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:50.718+0000 7f7f72000640 1 -- 172.21.15.32:0/144721741 learned_addr learned my addr 172.21.15.32:0/144721741 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:47:50.724 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:50.718+0000 7f7f72a00640 1 --2- 172.21.15.32:0/144721741 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55a7f7fa63b0 0x55a7f842c800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:47:50.724 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:50.718+0000 7f7f73400640 1 --2- 172.21.15.32:0/144721741 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55a7f7fcf270 0x55a7f83d0660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:47:50.724 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:50.718+0000 7f7f72000640 1 -- 172.21.15.32:0/144721741 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55a7f7fa63b0 msgr2=0x55a7f842c800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:47:50.724 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:50.718+0000 7f7f72000640 1 --2- 172.21.15.32:0/144721741 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55a7f7fa63b0 0x55a7f842c800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:50.724 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:50.718+0000 7f7f72000640 1 -- 172.21.15.32:0/144721741 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55a7f7fcf270 msgr2=0x55a7f83d0660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:47:50.724 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:50.718+0000 7f7f72000640 1 --2- 172.21.15.32:0/144721741 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55a7f7fcf270 0x55a7f83d0660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:50.724 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:50.718+0000 7f7f72000640 1 -- 172.21.15.32:0/144721741 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a7f83b11f0 con 0x55a7f7fae590 2024-08-03T20:47:50.724 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:50.718+0000 7f7f72000640 1 --2- 172.21.15.32:0/144721741 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a7f7fae590 0x55a7f83d0120 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f7f6002a460 tx=0x7f7f600216f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bde748f53f338773 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:47:50.725 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:50.718+0000 7f7f53400640 10 client.0 ms_handle_connect on v2:172.21.15.32:3300/0 2024-08-03T20:47:50.725 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:50.719+0000 7f7f53400640 1 -- 172.21.15.32:0/144721741 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f60025030 con 0x55a7f7fae590 2024-08-03T20:47:50.725 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:50.719+0000 7f7f53400640 1 -- 172.21.15.32:0/144721741 <== mon.0 v2:172.21.15.32:3300/0 2 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f7f60004480 con 0x55a7f7fae590 2024-08-03T20:47:50.725 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:50.719+0000 7f7f807c7740 1 -- 172.21.15.32:0/144721741 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7f640861f0 con 0x55a7f7fae590 2024-08-03T20:47:50.725 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:50.719+0000 7f7f53400640 1 -- 172.21.15.32:0/144721741 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_get_version_reply(handle=1 version=8) ==== 24+0+0 (secure 0 0 0) 0x7f7f60030020 con 0x55a7f7fae590 2024-08-03T20:47:50.725 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:50.719+0000 7f7f807c7740 10 client.4659 fetch_fsmap learned FSMap version 8 2024-08-03T20:47:50.725 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:50.719+0000 7f7f807c7740 1 -- 172.21.15.32:0/144721741 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,fsmap=8,monmap=2+}) -- 0x55a7f842cd40 con 0x55a7f7fae590 2024-08-03T20:47:50.725 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:50.719+0000 7f7f53400640 1 -- 172.21.15.32:0/144721741 <== mon.0 v2:172.21.15.32:3300/0 4 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f7f640861f0 con 0x55a7f7fae590 2024-08-03T20:47:50.725 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:50.719+0000 7f7f53400640 1 -- 172.21.15.32:0/144721741 <== mon.0 v2:172.21.15.32:3300/0 5 ==== fsmap(e 8) ==== 3502+0+0 (secure 0 0 0) 0x7f7f6003d050 con 0x55a7f7fae590 2024-08-03T20:47:50.725 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:50.719+0000 7f7f807c7740 10 client.4659 fetch_fsmap finished waiting for FSMap version 8 2024-08-03T20:47:50.725 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:50.719+0000 7f7f807c7740 10 client.4659 resolve_mds: resolved name 'c' to daemon mds.c 2024-08-03T20:47:50.725 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:50.719+0000 7f7f807c7740 20 client.4659 populate_metadata read hostname 'smithi032' 2024-08-03T20:47:50.725 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:50.719+0000 7f7f807c7740 1 --2- 172.21.15.32:0/144721741 >> [v2:172.21.15.32:6833/246278108,v1:172.21.15.32:6835/246278108] conn(0x55a7f839bd20 0x55a7f83b9600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:47:50.726 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:50.719+0000 7f7f807c7740 4 client.4659 mds_command: new command op to 4341 tid=0 multi_id=0 [{"prefix": "status"}] 2024-08-03T20:47:50.726 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:50.719+0000 7f7f807c7740 1 -- 172.21.15.32:0/144721741 --> [v2:172.21.15.32:6833/246278108,v1:172.21.15.32:6835/246278108] -- command(tid 0: {"prefix": "status"}) -- 0x55a7f82e5450 con 0x55a7f839bd20 2024-08-03T20:47:50.726 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:50.720+0000 7f7f72a00640 1 --2- 172.21.15.32:0/144721741 >> [v2:172.21.15.32:6833/246278108,v1:172.21.15.32:6835/246278108] conn(0x55a7f839bd20 0x55a7f83b9600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:47:50.726 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:50.720+0000 7f7f72a00640 1 --2- 172.21.15.32:0/144721741 >> [v2:172.21.15.32:6833/246278108,v1:172.21.15.32:6835/246278108] conn(0x55a7f839bd20 0x55a7f83b9600 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d4d20d70819bdecf server_cookie=422b2467b41ac6e3 in_seq=0 out_seq=0 2024-08-03T20:47:50.726 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:50.720+0000 7f7f53400640 10 client.4659 ms_handle_connect on v2:172.21.15.32:6833/246278108 2024-08-03T20:47:50.726 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:50.720+0000 7f7f53400640 1 -- 172.21.15.32:0/144721741 <== mds.0 v2:172.21.15.32:6833/246278108 1 ==== command_reply(tid 0: 0 ) ==== 8+0+324 (crc 0 0 0) 0x55a7f82e5450 con 0x55a7f839bd20 2024-08-03T20:47:50.726 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:50.720+0000 7f7f53400640 10 client.4659 handle_command_reply: tid=0 2024-08-03T20:47:50.726 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:50.720+0000 7f7f807c7740 1 client.4659 shutdown 2024-08-03T20:47:50.726 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:50.720+0000 7f7f807c7740 1 -- 172.21.15.32:0/144721741 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a7f7fae590 msgr2=0x55a7f83d0120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:47:50.726 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:50.720+0000 7f7f807c7740 1 --2- 172.21.15.32:0/144721741 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a7f7fae590 0x55a7f83d0120 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f7f6002a460 tx=0x7f7f600216f0 comp rx=0 tx=0).stop 2024-08-03T20:47:50.726 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:50.721+0000 7f7f807c7740 1 -- 172.21.15.32:0/144721741 shutdown_connections 2024-08-03T20:47:50.726 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:50.721+0000 7f7f807c7740 1 --2- 172.21.15.32:0/144721741 >> [v2:172.21.15.32:6833/246278108,v1:172.21.15.32:6835/246278108] conn(0x55a7f839bd20 0x55a7f83b9600 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:50.726 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:50.721+0000 7f7f807c7740 1 --2- 172.21.15.32:0/144721741 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55a7f7fcf270 0x55a7f83d0660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:50.727 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:50.721+0000 7f7f807c7740 1 --2- 172.21.15.32:0/144721741 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a7f7fae590 0x55a7f83d0120 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:50.727 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:50.721+0000 7f7f807c7740 1 --2- 172.21.15.32:0/144721741 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55a7f7fa63b0 0x55a7f842c800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:50.727 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:50.721+0000 7f7f53400640 0 client.4659 ms_handle_reset on v2:172.21.15.32:6833/246278108 2024-08-03T20:47:50.727 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:50.721+0000 7f7f807c7740 1 -- 172.21.15.32:0/144721741 >> 172.21.15.32:0/144721741 conn(0x55a7f7fc33c0 msgr2=0x55a7f8370ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:47:50.727 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:50.721+0000 7f7f807c7740 1 -- 172.21.15.32:0/144721741 shutdown_connections 2024-08-03T20:47:50.727 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:50.721+0000 7f7f807c7740 1 -- 172.21.15.32:0/144721741 wait complete. 2024-08-03T20:47:50.727 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:50.721+0000 7f7f807c7740 20 client.4659 trim_cache size 0 max 16384 2024-08-03T20:47:50.727 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-08-03T20:47:50.727 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_fsid": "40ab141a-01d7-49b9-ae3c-2edec9876e95", 2024-08-03T20:47:50.727 INFO:teuthology.orchestra.run.smithi032.stdout: "whoami": 0, 2024-08-03T20:47:50.728 INFO:teuthology.orchestra.run.smithi032.stdout: "id": 4341, 2024-08-03T20:47:50.728 INFO:teuthology.orchestra.run.smithi032.stdout: "want_state": "up:active", 2024-08-03T20:47:50.728 INFO:teuthology.orchestra.run.smithi032.stdout: "state": "up:active", 2024-08-03T20:47:50.728 INFO:teuthology.orchestra.run.smithi032.stdout: "fs_name": "cephfs", 2024-08-03T20:47:50.728 INFO:teuthology.orchestra.run.smithi032.stdout: "rank_uptime": 14.791429525, 2024-08-03T20:47:50.728 INFO:teuthology.orchestra.run.smithi032.stdout: "mdsmap_epoch": 8, 2024-08-03T20:47:50.728 INFO:teuthology.orchestra.run.smithi032.stdout: "osdmap_epoch": 21, 2024-08-03T20:47:50.728 INFO:teuthology.orchestra.run.smithi032.stdout: "osdmap_epoch_barrier": 21, 2024-08-03T20:47:50.728 INFO:teuthology.orchestra.run.smithi032.stdout: "uptime": 21.414213228000001 2024-08-03T20:47:50.728 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-08-03T20:47:50.729 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:50.721+0000 7f7f79600640 1 -- 172.21.15.32:0/262238558 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f7f54077040 msgr2=0x7f7f54079550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:47:50.729 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:50.721+0000 7f7f79600640 1 --2- 172.21.15.32:0/262238558 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f7f54077040 0x7f7f54079550 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f7f740b8b10 tx=0x7f7f60009060 comp rx=0 tx=0).stop 2024-08-03T20:47:50.729 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:50.721+0000 7f7f79600640 1 -- 172.21.15.32:0/262238558 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7f740cfc50 msgr2=0x7f7f740b5840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:47:50.729 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:50.721+0000 7f7f79600640 1 --2- 172.21.15.32:0/262238558 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7f740cfc50 0x7f7f740b5840 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f7f64034040 tx=0x7f7f640039e0 comp rx=0 tx=0).stop 2024-08-03T20:47:50.729 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:50.722+0000 7f7f79600640 1 -- 172.21.15.32:0/262238558 shutdown_connections 2024-08-03T20:47:50.730 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:50.722+0000 7f7f79600640 1 --2- 172.21.15.32:0/262238558 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f7f54077040 0x7f7f54079550 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:50.730 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:50.722+0000 7f7f79600640 1 --2- 172.21.15.32:0/262238558 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f7f740d3960 0x7f7f7415c6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:50.730 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:50.722+0000 7f7f79600640 1 --2- 172.21.15.32:0/262238558 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f7f741f98a0 0x7f7f7415cc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:50.730 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:50.722+0000 7f7f79600640 1 --2- 172.21.15.32:0/262238558 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7f740cfc50 0x7f7f740b5840 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:50.730 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:50.722+0000 7f7f79600640 1 -- 172.21.15.32:0/262238558 >> 172.21.15.32:0/262238558 conn(0x7f7f740d0fe0 msgr2=0x7f7f740ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:47:50.730 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:50.722+0000 7f7f79600640 1 -- 172.21.15.32:0/262238558 shutdown_connections 2024-08-03T20:47:50.730 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:50.722+0000 7f7f79600640 1 -- 172.21.15.32:0/262238558 wait complete. 2024-08-03T20:47:50.749 INFO:teuthology.run_tasks:Running task sequential... 2024-08-03T20:47:50.762 INFO:teuthology.task.sequential:In sequential, running task sequential... 2024-08-03T20:47:50.762 INFO:teuthology.task.sequential:In sequential, running task print... 2024-08-03T20:47:50.763 INFO:teuthology.task.print:Enabling mgr modules 2024-08-03T20:47:50.763 INFO:teuthology.run_tasks:Running task kclient... 2024-08-03T20:47:50.796 INFO:tasks.kclient:Mounting kernel clients... 2024-08-03T20:47:50.796 INFO:tasks.kclient:config is {'client.0': {}, 'client.2': {}, 'client.1': {}, 'client.3': {}} 2024-08-03T20:47:50.796 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-08-03T20:47:50.796 DEBUG:teuthology.orchestra.run.smithi032:> ip netns list 2024-08-03T20:47:50.833 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-08-03T20:47:50.833 DEBUG:teuthology.orchestra.run.smithi032:> sudo ip link delete ceph-brx 2024-08-03T20:47:50.932 INFO:teuthology.orchestra.run.smithi032.stderr:Cannot find device "ceph-brx" 2024-08-03T20:47:50.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-03T20:47:50.936 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-08-03T20:47:50.936 DEBUG:teuthology.orchestra.run.smithi032:> ip netns list 2024-08-03T20:47:50.972 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-08-03T20:47:50.972 DEBUG:teuthology.orchestra.run.smithi032:> sudo ip link delete ceph-brx 2024-08-03T20:47:51.071 INFO:teuthology.orchestra.run.smithi032.stderr:Cannot find device "ceph-brx" 2024-08-03T20:47:51.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-03T20:47:51.075 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-08-03T20:47:51.075 DEBUG:teuthology.orchestra.run.smithi118:> ip netns list 2024-08-03T20:47:51.115 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-08-03T20:47:51.116 DEBUG:teuthology.orchestra.run.smithi118:> sudo ip link delete ceph-brx 2024-08-03T20:47:51.219 INFO:teuthology.orchestra.run.smithi118.stderr:Cannot find device "ceph-brx" 2024-08-03T20:47:51.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-03T20:47:51.222 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-08-03T20:47:51.222 DEBUG:teuthology.orchestra.run.smithi118:> ip netns list 2024-08-03T20:47:51.258 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-08-03T20:47:51.258 DEBUG:teuthology.orchestra.run.smithi118:> sudo ip link delete ceph-brx 2024-08-03T20:47:51.358 INFO:teuthology.orchestra.run.smithi118.stderr:Cannot find device "ceph-brx" 2024-08-03T20:47:51.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-03T20:47:51.361 INFO:tasks.kclient:client.0 config is {'syntax': 'v1'} 2024-08-03T20:47:51.361 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-08-03T20:47:51.361 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-08-03T20:47:51.361 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-08-03T20:47:51.589 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:51.587+0000 7fc57d800640 1 -- 172.21.15.32:0/639449645 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc5780af020 msgr2=0x7fc5780742b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:47:51.589 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:51.587+0000 7fc57d800640 1 --2- 172.21.15.32:0/639449645 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc5780af020 0x7fc5780742b0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fc568009830 tx=0x7fc56802a530 comp rx=0 tx=0).stop 2024-08-03T20:47:51.589 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:51.587+0000 7fc57d800640 1 -- 172.21.15.32:0/639449645 shutdown_connections 2024-08-03T20:47:51.590 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:51.587+0000 7fc57d800640 1 --2- 172.21.15.32:0/639449645 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc5780af020 0x7fc5780742b0 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:51.590 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:51.587+0000 7fc57d800640 1 -- 172.21.15.32:0/639449645 >> 172.21.15.32:0/639449645 conn(0x7fc5780b2120 msgr2=0x7fc5780cfd60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:47:51.590 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:51.588+0000 7fc57d800640 1 -- 172.21.15.32:0/639449645 shutdown_connections 2024-08-03T20:47:51.590 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:51.588+0000 7fc57d800640 1 -- 172.21.15.32:0/639449645 wait complete. 2024-08-03T20:47:51.591 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:51.590+0000 7fc57d800640 1 Processor -- start 2024-08-03T20:47:51.591 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:51.590+0000 7fc57d800640 1 -- start start 2024-08-03T20:47:51.594 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:51.591+0000 7fc57d800640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fc57806fb50 0x7fc578154f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:47:51.594 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:51.591+0000 7fc57d800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc5780acd80 0x7fc57815ebb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:47:51.595 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:51.591+0000 7fc57d800640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc5780af020 0x7fc57815f0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:47:51.595 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:51.591+0000 7fc57d800640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fc5780ad310 con 0x7fc5780acd80 2024-08-03T20:47:51.595 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:51.591+0000 7fc57d800640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7fc5780ad190 con 0x7fc5780af020 2024-08-03T20:47:51.595 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:51.591+0000 7fc57d800640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7fc5780ad490 con 0x7fc57806fb50 2024-08-03T20:47:51.595 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:51.592+0000 7fc577400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc5780acd80 0x7fc57815ebb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:47:51.595 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:51.592+0000 7fc577e00640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fc57806fb50 0x7fc578154f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:47:51.595 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:51.592+0000 7fc577400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc5780acd80 0x7fc57815ebb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:52438/0 (socket says 172.21.15.32:52438) 2024-08-03T20:47:51.595 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:51.592+0000 7fc577e00640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fc57806fb50 0x7fc578154f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3301/0 says I am v2:172.21.15.32:36622/0 (socket says 172.21.15.32:36622) 2024-08-03T20:47:51.595 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:51.592+0000 7fc577400640 1 -- 172.21.15.32:0/605846584 learned_addr learned my addr 172.21.15.32:0/605846584 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:47:51.595 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:51.592+0000 7fc57ce00640 1 --2- 172.21.15.32:0/605846584 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc5780af020 0x7fc57815f0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:47:51.595 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:51.592+0000 7fc577400640 1 -- 172.21.15.32:0/605846584 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fc57806fb50 msgr2=0x7fc578154f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:47:51.595 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:51.592+0000 7fc577400640 1 --2- 172.21.15.32:0/605846584 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fc57806fb50 0x7fc578154f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:51.596 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:51.592+0000 7fc577400640 1 -- 172.21.15.32:0/605846584 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc5780af020 msgr2=0x7fc57815f0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:47:51.596 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:51.592+0000 7fc577400640 1 --2- 172.21.15.32:0/605846584 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc5780af020 0x7fc57815f0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:51.596 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:51.592+0000 7fc577400640 1 -- 172.21.15.32:0/605846584 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc57809dbb0 con 0x7fc5780acd80 2024-08-03T20:47:51.596 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:51.592+0000 7fc577400640 1 --2- 172.21.15.32:0/605846584 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc5780acd80 0x7fc57815ebb0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fc56400c400 tx=0x7fc56400c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:47:51.596 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:51.593+0000 7fc56fe00640 1 -- 172.21.15.32:0/605846584 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc564011790 con 0x7fc5780acd80 2024-08-03T20:47:51.596 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:51.593+0000 7fc56fe00640 1 -- 172.21.15.32:0/605846584 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fc564017070 con 0x7fc5780acd80 2024-08-03T20:47:51.596 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:51.593+0000 7fc57d800640 1 -- 172.21.15.32:0/605846584 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc57809d610 con 0x7fc5780acd80 2024-08-03T20:47:51.596 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:51.593+0000 7fc56fe00640 1 -- 172.21.15.32:0/605846584 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc564011cc0 con 0x7fc5780acd80 2024-08-03T20:47:51.596 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:51.593+0000 7fc57d800640 1 -- 172.21.15.32:0/605846584 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc5780c7690 con 0x7fc5780acd80 2024-08-03T20:47:51.597 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:51.594+0000 7fc57d800640 1 -- 172.21.15.32:0/605846584 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc5780c7b00 con 0x7fc5780acd80 2024-08-03T20:47:51.597 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:51.595+0000 7fc56fe00640 1 -- 172.21.15.32:0/605846584 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7fc56401c460 con 0x7fc5780acd80 2024-08-03T20:47:51.598 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:51.596+0000 7fc56fe00640 1 --2- 172.21.15.32:0/605846584 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fc558077010 0x7fc558079520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:47:51.598 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:51.597+0000 7fc56fe00640 1 -- 172.21.15.32:0/605846584 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7fc564098670 con 0x7fc5780acd80 2024-08-03T20:47:51.598 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:51.597+0000 7fc577e00640 1 --2- 172.21.15.32:0/605846584 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fc558077010 0x7fc558079520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:47:51.599 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:51.597+0000 7fc577e00640 1 --2- 172.21.15.32:0/605846584 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fc558077010 0x7fc558079520 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fc568034000 tx=0x7fc568004730 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:47:51.602 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:51.601+0000 7fc56fe00640 1 -- 172.21.15.32:0/605846584 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7fc564061210 con 0x7fc5780acd80 2024-08-03T20:47:51.846 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:51.845+0000 7fc57d800640 1 -- 172.21.15.32:0/605846584 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7fc5780742b0 con 0x7fc5780acd80 2024-08-03T20:47:51.847 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:51.845+0000 7fc56fe00640 1 -- 172.21.15.32:0/605846584 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v8) ==== 53+0+73 (secure 0 0 0) 0x7fc564065760 con 0x7fc5780acd80 2024-08-03T20:47:51.847 INFO:teuthology.orchestra.run.smithi032.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-08-03T20:47:51.849 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:51.848+0000 7fc57d800640 1 -- 172.21.15.32:0/605846584 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fc558077010 msgr2=0x7fc558079520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:47:51.849 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:51.848+0000 7fc57d800640 1 --2- 172.21.15.32:0/605846584 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fc558077010 0x7fc558079520 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fc568034000 tx=0x7fc568004730 comp rx=0 tx=0).stop 2024-08-03T20:47:51.850 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:51.848+0000 7fc57d800640 1 -- 172.21.15.32:0/605846584 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc5780acd80 msgr2=0x7fc57815ebb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:47:51.850 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:51.848+0000 7fc57d800640 1 --2- 172.21.15.32:0/605846584 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc5780acd80 0x7fc57815ebb0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fc56400c400 tx=0x7fc56400c8c0 comp rx=0 tx=0).stop 2024-08-03T20:47:51.850 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:51.848+0000 7fc57d800640 1 -- 172.21.15.32:0/605846584 shutdown_connections 2024-08-03T20:47:51.850 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:51.848+0000 7fc57d800640 1 --2- 172.21.15.32:0/605846584 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fc558077010 0x7fc558079520 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:51.850 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:51.848+0000 7fc57d800640 1 --2- 172.21.15.32:0/605846584 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc5780af020 0x7fc57815f0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:51.850 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:51.848+0000 7fc57d800640 1 --2- 172.21.15.32:0/605846584 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc5780acd80 0x7fc57815ebb0 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:51.850 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:51.848+0000 7fc57d800640 1 --2- 172.21.15.32:0/605846584 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fc57806fb50 0x7fc578154f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:51.850 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:51.848+0000 7fc57d800640 1 -- 172.21.15.32:0/605846584 >> 172.21.15.32:0/605846584 conn(0x7fc5780b2120 msgr2=0x7fc57815fd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:47:51.850 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:51.848+0000 7fc57d800640 1 -- 172.21.15.32:0/605846584 shutdown_connections 2024-08-03T20:47:51.850 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:51.848+0000 7fc57d800640 1 -- 172.21.15.32:0/605846584 wait complete. 2024-08-03T20:47:51.871 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-08-03T20:47:51.871 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-08-03T20:47:51.872 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi032.front.sepia.ceph.com 2024-08-03T20:47:51.872 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-08-03T20:47:51.872 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-08-03T20:47:51.872 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-08-03T20:47:51.872 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-08-03T20:47:51.872 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-08-03T20:47:51.872 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.0' 2024-08-03T20:47:51.872 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-08-03T20:47:51.872 DEBUG:teuthology.orchestra.run.smithi032:> ip addr 2024-08-03T20:47:51.908 INFO:teuthology.orchestra.run.smithi032.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-08-03T20:47:51.908 INFO:teuthology.orchestra.run.smithi032.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-08-03T20:47:51.908 INFO:teuthology.orchestra.run.smithi032.stdout: inet 127.0.0.1/8 scope host lo 2024-08-03T20:47:51.908 INFO:teuthology.orchestra.run.smithi032.stdout: valid_lft forever preferred_lft forever 2024-08-03T20:47:51.908 INFO:teuthology.orchestra.run.smithi032.stdout: inet6 ::1/128 scope host 2024-08-03T20:47:51.908 INFO:teuthology.orchestra.run.smithi032.stdout: valid_lft forever preferred_lft forever 2024-08-03T20:47:51.908 INFO:teuthology.orchestra.run.smithi032.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-08-03T20:47:51.908 INFO:teuthology.orchestra.run.smithi032.stdout: link/ether 0c:c4:7a:6b:fd:0c brd ff:ff:ff:ff:ff:ff 2024-08-03T20:47:51.908 INFO:teuthology.orchestra.run.smithi032.stdout: altname enp7s0f0 2024-08-03T20:47:51.908 INFO:teuthology.orchestra.run.smithi032.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-08-03T20:47:51.909 INFO:teuthology.orchestra.run.smithi032.stdout: link/ether 0c:c4:7a:6b:fd:0d brd ff:ff:ff:ff:ff:ff 2024-08-03T20:47:51.909 INFO:teuthology.orchestra.run.smithi032.stdout: altname enp7s0f1 2024-08-03T20:47:51.909 INFO:teuthology.orchestra.run.smithi032.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-08-03T20:47:51.909 INFO:teuthology.orchestra.run.smithi032.stdout: link/ether ac:1f:6b:f8:26:02 brd ff:ff:ff:ff:ff:ff 2024-08-03T20:47:51.909 INFO:teuthology.orchestra.run.smithi032.stdout: altname enp1s0f0 2024-08-03T20:47:51.909 INFO:teuthology.orchestra.run.smithi032.stdout: inet 172.21.15.32/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-08-03T20:47:51.909 INFO:teuthology.orchestra.run.smithi032.stdout: valid_lft 42484sec preferred_lft 42484sec 2024-08-03T20:47:51.909 INFO:teuthology.orchestra.run.smithi032.stdout: inet6 fe80::ae1f:6bff:fef8:2602/64 scope link noprefixroute 2024-08-03T20:47:51.909 INFO:teuthology.orchestra.run.smithi032.stdout: valid_lft forever preferred_lft forever 2024-08-03T20:47:51.909 INFO:teuthology.orchestra.run.smithi032.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-08-03T20:47:51.909 INFO:teuthology.orchestra.run.smithi032.stdout: link/ether ac:1f:6b:f8:26:03 brd ff:ff:ff:ff:ff:ff 2024-08-03T20:47:51.909 INFO:teuthology.orchestra.run.smithi032.stdout: altname enp1s0f1 2024-08-03T20:47:51.910 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-08-03T20:47:51.910 DEBUG:teuthology.orchestra.run.smithi032:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-08-03T20:47:51.910 DEBUG:teuthology.orchestra.run.smithi032:> set -e 2024-08-03T20:47:51.910 DEBUG:teuthology.orchestra.run.smithi032:> sudo ip link add name ceph-brx type bridge 2024-08-03T20:47:51.911 DEBUG:teuthology.orchestra.run.smithi032:> sudo ip addr flush dev ceph-brx 2024-08-03T20:47:51.911 DEBUG:teuthology.orchestra.run.smithi032:> sudo ip link set ceph-brx up 2024-08-03T20:47:51.911 DEBUG:teuthology.orchestra.run.smithi032:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-08-03T20:47:51.911 DEBUG:teuthology.orchestra.run.smithi032:> ') 2024-08-03T20:47:52.028 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:52 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-08-03T20:47:52.323 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:52 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-08-03T20:47:52.332 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-08-03T20:47:52.332 DEBUG:teuthology.orchestra.run.smithi032:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-08-03T20:47:52.389 INFO:teuthology.orchestra.run.smithi032.stdout:1 2024-08-03T20:47:52.392 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-08-03T20:47:52.392 DEBUG:teuthology.orchestra.run.smithi032:> route 2024-08-03T20:47:52.435 INFO:teuthology.orchestra.run.smithi032.stdout:Kernel IP routing table 2024-08-03T20:47:52.435 INFO:teuthology.orchestra.run.smithi032.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-08-03T20:47:52.435 INFO:teuthology.orchestra.run.smithi032.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-08-03T20:47:52.435 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-08-03T20:47:52.435 INFO:teuthology.orchestra.run.smithi032.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-08-03T20:47:52.436 DEBUG:teuthology.orchestra.run.smithi032:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-08-03T20:47:52.436 DEBUG:teuthology.orchestra.run.smithi032:> set -e 2024-08-03T20:47:52.436 DEBUG:teuthology.orchestra.run.smithi032:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-08-03T20:47:52.436 DEBUG:teuthology.orchestra.run.smithi032:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-08-03T20:47:52.436 DEBUG:teuthology.orchestra.run.smithi032:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-08-03T20:47:52.436 DEBUG:teuthology.orchestra.run.smithi032:> ') 2024-08-03T20:47:52.554 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:52 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-08-03T20:47:53.059 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:53 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-08-03T20:47:53.067 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-08-03T20:47:53.067 DEBUG:teuthology.orchestra.run.smithi032:> ip netns list 2024-08-03T20:47:53.103 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-08-03T20:47:53.104 DEBUG:teuthology.orchestra.run.smithi032:> ip netns list-id 2024-08-03T20:47:53.179 DEBUG:teuthology.orchestra.run.smithi032:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-08-03T20:47:53.179 DEBUG:teuthology.orchestra.run.smithi032:> set -e 2024-08-03T20:47:53.179 DEBUG:teuthology.orchestra.run.smithi032:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.0 2024-08-03T20:47:53.179 DEBUG:teuthology.orchestra.run.smithi032:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.0 0 2024-08-03T20:47:53.179 DEBUG:teuthology.orchestra.run.smithi032:> ') 2024-08-03T20:47:53.299 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:53 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-08-03T20:47:53.351 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:53 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-08-03T20:47:53.359 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.0' with 192.168.0.1/16 2024-08-03T20:47:53.360 DEBUG:teuthology.orchestra.run.smithi032:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-08-03T20:47:53.360 DEBUG:teuthology.orchestra.run.smithi032:> set -e 2024-08-03T20:47:53.360 DEBUG:teuthology.orchestra.run.smithi032:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.0 type veth peer name brx.0 2024-08-03T20:47:53.360 DEBUG:teuthology.orchestra.run.smithi032:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-08-03T20:47:53.360 DEBUG:teuthology.orchestra.run.smithi032:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set veth0 up 2024-08-03T20:47:53.360 DEBUG:teuthology.orchestra.run.smithi032:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set lo up 2024-08-03T20:47:53.360 DEBUG:teuthology.orchestra.run.smithi032:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip route add default via 192.168.255.254 2024-08-03T20:47:53.360 DEBUG:teuthology.orchestra.run.smithi032:> ') 2024-08-03T20:47:53.439 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:53 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-08-03T20:47:53.594 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:53 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-08-03T20:47:53.603 DEBUG:teuthology.orchestra.run.smithi032:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-08-03T20:47:53.603 DEBUG:teuthology.orchestra.run.smithi032:> set -e 2024-08-03T20:47:53.603 DEBUG:teuthology.orchestra.run.smithi032:> sudo ip link set brx.0 up 2024-08-03T20:47:53.603 DEBUG:teuthology.orchestra.run.smithi032:> sudo ip link set dev brx.0 master ceph-brx 2024-08-03T20:47:53.603 DEBUG:teuthology.orchestra.run.smithi032:> ') 2024-08-03T20:47:53.682 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:53 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-08-03T20:47:53.763 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:53 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-08-03T20:47:53.772 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-08-03T20:47:53.772 DEBUG:teuthology.orchestra.run.smithi032:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-08-03T20:47:53.808 INFO:teuthology.orchestra.run.smithi032.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-08-03T20:47:53.808 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-08-03T20:47:53.808 DEBUG:teuthology.orchestra.run.smithi032:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-08-03T20:47:53.883 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2024-08-03T20:47:53.883 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-08-03T20:47:53.883 DEBUG:teuthology.orchestra.run.smithi032:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-08-03T20:47:54.114 INFO:teuthology.orchestra.run.smithi032.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-08-03T20:47:54.114 INFO:teuthology.orchestra.run.smithi032.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-08-03T20:47:54.114 INFO:teuthology.orchestra.run.smithi032.stdout:invalid new device string format 2024-08-03T20:47:54.717 INFO:teuthology.orchestra.run.smithi032.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-08-03T20:47:54.717 INFO:teuthology.orchestra.run.smithi032.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-08-03T20:47:54.717 INFO:teuthology.orchestra.run.smithi032.stdout:invalid new device string format 2024-08-03T20:47:54.717 INFO:teuthology.orchestra.run.smithi032.stdout:mount.ceph: resolved to: "172.21.15.32:3300,172.21.15.118:3300,172.21.15.118:3301" 2024-08-03T20:47:54.717 INFO:teuthology.orchestra.run.smithi032.stdout:mount.ceph: switching to using v1 address with old syntax 2024-08-03T20:47:54.751 INFO:teuthology.orchestra.run.smithi032.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-08-03T20:47:54.751 INFO:teuthology.orchestra.run.smithi032.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-08-03T20:47:54.751 INFO:teuthology.orchestra.run.smithi032.stdout:invalid new device string format 2024-08-03T20:47:54.752 INFO:teuthology.orchestra.run.smithi032.stdout:mount.ceph: resolved to: "172.21.15.32:3300,172.21.15.118:3300,172.21.15.118:3301" 2024-08-03T20:47:54.752 INFO:teuthology.orchestra.run.smithi032.stdout:mount.ceph: switching to using v1 address with old syntax 2024-08-03T20:47:54.752 INFO:teuthology.orchestra.run.smithi032.stdout:mount.ceph: resolved to: "172.21.15.32:6789,172.21.15.118:6789,172.21.15.118:6790" 2024-08-03T20:47:54.752 INFO:teuthology.orchestra.run.smithi032.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.32:6789,172.21.15.118:6789,172.21.15.118:6790:/ 2024-08-03T20:47:54.752 INFO:teuthology.orchestra.run.smithi032.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,key=0,fsid=40ab141a-01d7-49b9-ae3c-2edec9876e95" will pass to kernel 2024-08-03T20:47:54.754 INFO:teuthology.orchestra.run.smithi032.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2024-08-03T20:47:54.754 INFO:teuthology.orchestra.run.smithi032.stdout: You just mounted a file system that supports labels which does not 2024-08-03T20:47:54.754 INFO:teuthology.orchestra.run.smithi032.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-08-03T20:47:54.754 INFO:teuthology.orchestra.run.smithi032.stdout: applications will generate AVC messages and not be allowed access to 2024-08-03T20:47:54.754 INFO:teuthology.orchestra.run.smithi032.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-08-03T20:47:54.757 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-08-03T20:47:54.757 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-08-03T20:47:54.757 DEBUG:teuthology.orchestra.run.smithi032:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-08-03T20:47:54.815 DEBUG:teuthology.orchestra.run.smithi032:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-08-03T20:47:54.897 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:54 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-08-03T20:47:54.905 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:54 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-08-03T20:47:54.913 INFO:teuthology.orchestra.run.smithi032.stdout:client4671 2024-08-03T20:47:54.913 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-08-03T20:47:54.913 DEBUG:teuthology.orchestra.run.smithi032:> dd if=/proc/self/mounts of=/dev/stdout 2024-08-03T20:47:54.949 DEBUG:teuthology.orchestra.run.smithi032:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-08-03T20:47:55.071 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:55 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-08-03T20:47:55.078 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:55 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-08-03T20:47:55.086 INFO:teuthology.orchestra.run.smithi032.stdout:client4671 2024-08-03T20:47:55.091 DEBUG:teuthology.orchestra.run.smithi032:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-08-03T20:47:55.172 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:55 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-08-03T20:47:55.179 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:55 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-08-03T20:47:55.187 INFO:teuthology.orchestra.run.smithi032.stdout:client4671 2024-08-03T20:47:55.187 DEBUG:teuthology.orchestra.run.smithi032:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=300 -- bash -c 'sudo dd if=/sys/kernel/debug/ceph/40ab141a-01d7-49b9-ae3c-2edec9876e95.client4671/status') 2024-08-03T20:47:55.268 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:55 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-08-03T20:47:55.292 INFO:teuthology.orchestra.run.smithi032.stdout:instance: client.4671 (3)192.168.0.1:0/1459475584 2024-08-03T20:47:55.292 INFO:teuthology.orchestra.run.smithi032.stdout:blocklisted: false 2024-08-03T20:47:55.292 INFO:teuthology.orchestra.run.smithi032.stderr:0+1 records in 2024-08-03T20:47:55.292 INFO:teuthology.orchestra.run.smithi032.stderr:0+1 records out 2024-08-03T20:47:55.293 INFO:teuthology.orchestra.run.smithi032.stderr:69 bytes copied, 6.9226e-05 s, 997 kB/s 2024-08-03T20:47:55.294 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:55 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-08-03T20:47:55.302 INFO:tasks.kclient:client.2 config is {'syntax': 'v1'} 2024-08-03T20:47:55.303 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-08-03T20:47:55.303 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.2 2024-08-03T20:47:55.303 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-08-03T20:47:55.532 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:55.530+0000 7f7605600640 1 -- 172.21.15.32:0/759330643 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f760015fff0 msgr2=0x7f7600160400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:47:55.532 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:55.530+0000 7f7605600640 1 --2- 172.21.15.32:0/759330643 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f760015fff0 0x7f7600160400 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f75f8005fd0 tx=0x7f75f802a8a0 comp rx=0 tx=0).stop 2024-08-03T20:47:55.532 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:55.530+0000 7f7605600640 1 -- 172.21.15.32:0/759330643 shutdown_connections 2024-08-03T20:47:55.532 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:55.530+0000 7f7605600640 1 --2- 172.21.15.32:0/759330643 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f760015fff0 0x7f7600160400 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:55.532 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:55.530+0000 7f7605600640 1 --2- 172.21.15.32:0/759330643 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f760006e930 0x7f7600144f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:55.532 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:55.530+0000 7f7605600640 1 -- 172.21.15.32:0/759330643 >> 172.21.15.32:0/759330643 conn(0x7f76000b1bd0 msgr2=0x7f76000b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:47:55.532 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:55.530+0000 7f7605600640 1 -- 172.21.15.32:0/759330643 shutdown_connections 2024-08-03T20:47:55.532 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:55.530+0000 7f7605600640 1 -- 172.21.15.32:0/759330643 wait complete. 2024-08-03T20:47:55.534 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:55.532+0000 7f7605600640 1 Processor -- start 2024-08-03T20:47:55.534 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:55.533+0000 7f7605600640 1 -- start start 2024-08-03T20:47:55.537 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:55.534+0000 7f7605600640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f760006e930 0x7f760013a570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:47:55.537 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:55.534+0000 7f7605600640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f7600073e60 0x7f76000cb9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:47:55.537 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:55.534+0000 7f7605600640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f760015fff0 0x7f76000c9610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:47:55.537 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:55.534+0000 7f7605600640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f76000cf840 con 0x7f760006e930 2024-08-03T20:47:55.537 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:55.534+0000 7f7605600640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f7600145f30 con 0x7f7600073e60 2024-08-03T20:47:55.537 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:55.534+0000 7f7605600640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7f76000cf9c0 con 0x7f760015fff0 2024-08-03T20:47:55.537 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:55.534+0000 7f75fea00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f760006e930 0x7f760013a570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:47:55.537 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:55.534+0000 7f75fea00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f760006e930 0x7f760013a570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:52450/0 (socket says 172.21.15.32:52450) 2024-08-03T20:47:55.538 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:55.534+0000 7f75fea00640 1 -- 172.21.15.32:0/3485247474 learned_addr learned my addr 172.21.15.32:0/3485247474 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:47:55.538 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:55.535+0000 7f75ff400640 1 --2- 172.21.15.32:0/3485247474 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f760015fff0 0x7f76000c9610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:47:55.538 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:55.535+0000 7f75f7e00640 1 --2- 172.21.15.32:0/3485247474 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f7600073e60 0x7f76000cb9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:47:55.538 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:55.535+0000 7f75fea00640 1 -- 172.21.15.32:0/3485247474 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f760015fff0 msgr2=0x7f76000c9610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:47:55.538 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:55.535+0000 7f75fea00640 1 --2- 172.21.15.32:0/3485247474 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f760015fff0 0x7f76000c9610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:55.538 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:55.535+0000 7f75fea00640 1 -- 172.21.15.32:0/3485247474 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f7600073e60 msgr2=0x7f76000cb9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:47:55.538 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:55.535+0000 7f75fea00640 1 --2- 172.21.15.32:0/3485247474 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f7600073e60 0x7f76000cb9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:55.538 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:55.535+0000 7f75fea00640 1 -- 172.21.15.32:0/3485247474 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f76000b49f0 con 0x7f760006e930 2024-08-03T20:47:55.538 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:55.535+0000 7f75f7e00640 1 --2- 172.21.15.32:0/3485247474 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f7600073e60 0x7f76000cb9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-03T20:47:55.538 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:55.535+0000 7f75ff400640 1 --2- 172.21.15.32:0/3485247474 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f760015fff0 0x7f76000c9610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-03T20:47:55.538 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:55.535+0000 7f75fea00640 1 --2- 172.21.15.32:0/3485247474 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f760006e930 0x7f760013a570 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f75f000e9c0 tx=0x7f75f000ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:47:55.538 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:55.535+0000 7f75f6a00640 1 -- 172.21.15.32:0/3485247474 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75f000cd80 con 0x7f760006e930 2024-08-03T20:47:55.539 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:55.535+0000 7f75f6a00640 1 -- 172.21.15.32:0/3485247474 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f75f0004020 con 0x7f760006e930 2024-08-03T20:47:55.539 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:55.535+0000 7f75f6a00640 1 -- 172.21.15.32:0/3485247474 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75f0004450 con 0x7f760006e930 2024-08-03T20:47:55.539 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:55.536+0000 7f7605600640 1 -- 172.21.15.32:0/3485247474 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f76001f6550 con 0x7f760006e930 2024-08-03T20:47:55.539 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:55.536+0000 7f7605600640 1 -- 172.21.15.32:0/3485247474 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f76000b3170 con 0x7f760006e930 2024-08-03T20:47:55.539 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:55.537+0000 7f7605600640 1 -- 172.21.15.32:0/3485247474 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7600160bc0 con 0x7f760006e930 2024-08-03T20:47:55.539 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:55.538+0000 7f75f6a00640 1 -- 172.21.15.32:0/3485247474 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f75f00035f0 con 0x7f760006e930 2024-08-03T20:47:55.541 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:55.539+0000 7f75f6a00640 1 --2- 172.21.15.32:0/3485247474 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f75dc077010 0x7f75dc079520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:47:55.541 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:55.540+0000 7f75f6a00640 1 -- 172.21.15.32:0/3485247474 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f75f0099390 con 0x7f760006e930 2024-08-03T20:47:55.541 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:55.540+0000 7f75f7e00640 1 --2- 172.21.15.32:0/3485247474 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f75dc077010 0x7f75dc079520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:47:55.542 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:55.541+0000 7f75f7e00640 1 --2- 172.21.15.32:0/3485247474 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f75dc077010 0x7f75dc079520 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f76001588e0 tx=0x7f75e8009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:47:55.545 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:55.544+0000 7f75f6a00640 1 -- 172.21.15.32:0/3485247474 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7f75f00624f0 con 0x7f760006e930 2024-08-03T20:47:55.793 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:55.792+0000 7f7605600640 1 -- 172.21.15.32:0/3485247474 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f760015e250 con 0x7f760006e930 2024-08-03T20:47:55.794 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:55.793+0000 7f75f6a00640 1 -- 172.21.15.32:0/3485247474 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v8) ==== 53+0+73 (secure 0 0 0) 0x7f75f0061df0 con 0x7f760006e930 2024-08-03T20:47:55.794 INFO:teuthology.orchestra.run.smithi032.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-08-03T20:47:55.797 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:55.795+0000 7f7605600640 1 -- 172.21.15.32:0/3485247474 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f75dc077010 msgr2=0x7f75dc079520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:47:55.797 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:55.795+0000 7f7605600640 1 --2- 172.21.15.32:0/3485247474 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f75dc077010 0x7f75dc079520 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f76001588e0 tx=0x7f75e8009040 comp rx=0 tx=0).stop 2024-08-03T20:47:55.797 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:55.795+0000 7f7605600640 1 -- 172.21.15.32:0/3485247474 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f760006e930 msgr2=0x7f760013a570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:47:55.797 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:55.795+0000 7f7605600640 1 --2- 172.21.15.32:0/3485247474 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f760006e930 0x7f760013a570 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f75f000e9c0 tx=0x7f75f000ee80 comp rx=0 tx=0).stop 2024-08-03T20:47:55.797 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:55.795+0000 7f7605600640 1 -- 172.21.15.32:0/3485247474 shutdown_connections 2024-08-03T20:47:55.797 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:55.795+0000 7f7605600640 1 --2- 172.21.15.32:0/3485247474 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f75dc077010 0x7f75dc079520 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:55.797 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:55.795+0000 7f7605600640 1 --2- 172.21.15.32:0/3485247474 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f7600073e60 0x7f76000cb9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:55.797 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:55.795+0000 7f7605600640 1 --2- 172.21.15.32:0/3485247474 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f760015fff0 0x7f76000c9610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:55.797 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:55.795+0000 7f7605600640 1 --2- 172.21.15.32:0/3485247474 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f760006e930 0x7f760013a570 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:55.797 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:55.795+0000 7f7605600640 1 -- 172.21.15.32:0/3485247474 >> 172.21.15.32:0/3485247474 conn(0x7f76000b1bd0 msgr2=0x7f760015aa70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:47:55.798 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:55.795+0000 7f7605600640 1 -- 172.21.15.32:0/3485247474 shutdown_connections 2024-08-03T20:47:55.798 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:55.795+0000 7f7605600640 1 -- 172.21.15.32:0/3485247474 wait complete. 2024-08-03T20:47:55.822 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-08-03T20:47:55.822 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-08-03T20:47:55.823 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi032.front.sepia.ceph.com 2024-08-03T20:47:55.823 INFO:tasks.cephfs.mount:self.client.name = client.2 2024-08-03T20:47:55.823 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.2 2024-08-03T20:47:55.823 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-08-03T20:47:55.823 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-08-03T20:47:55.823 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-08-03T20:47:55.823 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.2' 2024-08-03T20:47:55.823 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-08-03T20:47:55.824 DEBUG:teuthology.orchestra.run.smithi032:> ip addr 2024-08-03T20:47:55.860 INFO:teuthology.orchestra.run.smithi032.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-08-03T20:47:55.860 INFO:teuthology.orchestra.run.smithi032.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-08-03T20:47:55.860 INFO:teuthology.orchestra.run.smithi032.stdout: inet 127.0.0.1/8 scope host lo 2024-08-03T20:47:55.860 INFO:teuthology.orchestra.run.smithi032.stdout: valid_lft forever preferred_lft forever 2024-08-03T20:47:55.860 INFO:teuthology.orchestra.run.smithi032.stdout: inet6 ::1/128 scope host 2024-08-03T20:47:55.860 INFO:teuthology.orchestra.run.smithi032.stdout: valid_lft forever preferred_lft forever 2024-08-03T20:47:55.860 INFO:teuthology.orchestra.run.smithi032.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-08-03T20:47:55.860 INFO:teuthology.orchestra.run.smithi032.stdout: link/ether 0c:c4:7a:6b:fd:0c brd ff:ff:ff:ff:ff:ff 2024-08-03T20:47:55.860 INFO:teuthology.orchestra.run.smithi032.stdout: altname enp7s0f0 2024-08-03T20:47:55.860 INFO:teuthology.orchestra.run.smithi032.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-08-03T20:47:55.860 INFO:teuthology.orchestra.run.smithi032.stdout: link/ether 0c:c4:7a:6b:fd:0d brd ff:ff:ff:ff:ff:ff 2024-08-03T20:47:55.860 INFO:teuthology.orchestra.run.smithi032.stdout: altname enp7s0f1 2024-08-03T20:47:55.861 INFO:teuthology.orchestra.run.smithi032.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-08-03T20:47:55.861 INFO:teuthology.orchestra.run.smithi032.stdout: link/ether ac:1f:6b:f8:26:02 brd ff:ff:ff:ff:ff:ff 2024-08-03T20:47:55.861 INFO:teuthology.orchestra.run.smithi032.stdout: altname enp1s0f0 2024-08-03T20:47:55.861 INFO:teuthology.orchestra.run.smithi032.stdout: inet 172.21.15.32/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-08-03T20:47:55.861 INFO:teuthology.orchestra.run.smithi032.stdout: valid_lft 42480sec preferred_lft 42480sec 2024-08-03T20:47:55.861 INFO:teuthology.orchestra.run.smithi032.stdout: inet6 fe80::ae1f:6bff:fef8:2602/64 scope link noprefixroute 2024-08-03T20:47:55.861 INFO:teuthology.orchestra.run.smithi032.stdout: valid_lft forever preferred_lft forever 2024-08-03T20:47:55.861 INFO:teuthology.orchestra.run.smithi032.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-08-03T20:47:55.861 INFO:teuthology.orchestra.run.smithi032.stdout: link/ether ac:1f:6b:f8:26:03 brd ff:ff:ff:ff:ff:ff 2024-08-03T20:47:55.861 INFO:teuthology.orchestra.run.smithi032.stdout: altname enp1s0f1 2024-08-03T20:47:55.861 INFO:teuthology.orchestra.run.smithi032.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-08-03T20:47:55.861 INFO:teuthology.orchestra.run.smithi032.stdout: link/ether 22:86:28:30:4e:62 brd ff:ff:ff:ff:ff:ff 2024-08-03T20:47:55.861 INFO:teuthology.orchestra.run.smithi032.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-08-03T20:47:55.862 INFO:teuthology.orchestra.run.smithi032.stdout: valid_lft forever preferred_lft forever 2024-08-03T20:47:55.862 INFO:teuthology.orchestra.run.smithi032.stdout: inet6 fe80::d804:7cff:fed9:7ba4/64 scope link 2024-08-03T20:47:55.862 INFO:teuthology.orchestra.run.smithi032.stdout: valid_lft forever preferred_lft forever 2024-08-03T20:47:55.862 INFO:teuthology.orchestra.run.smithi032.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-08-03T20:47:55.862 INFO:teuthology.orchestra.run.smithi032.stdout: link/ether 22:86:28:30:4e:62 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-08-03T20:47:55.862 INFO:teuthology.orchestra.run.smithi032.stdout: inet6 fe80::2086:28ff:fe30:4e62/64 scope link 2024-08-03T20:47:55.862 INFO:teuthology.orchestra.run.smithi032.stdout: valid_lft forever preferred_lft forever 2024-08-03T20:47:55.862 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-08-03T20:47:55.862 DEBUG:teuthology.orchestra.run.smithi032:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-08-03T20:47:55.955 INFO:teuthology.orchestra.run.smithi032.stdout:1 2024-08-03T20:47:55.958 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-08-03T20:47:55.959 DEBUG:teuthology.orchestra.run.smithi032:> route 2024-08-03T20:47:55.995 INFO:teuthology.orchestra.run.smithi032.stdout:Kernel IP routing table 2024-08-03T20:47:55.995 INFO:teuthology.orchestra.run.smithi032.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-08-03T20:47:55.995 INFO:teuthology.orchestra.run.smithi032.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-08-03T20:47:55.995 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-08-03T20:47:55.996 INFO:teuthology.orchestra.run.smithi032.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-08-03T20:47:55.996 DEBUG:teuthology.orchestra.run.smithi032:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-08-03T20:47:55.996 DEBUG:teuthology.orchestra.run.smithi032:> set -e 2024-08-03T20:47:55.996 DEBUG:teuthology.orchestra.run.smithi032:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-08-03T20:47:55.996 DEBUG:teuthology.orchestra.run.smithi032:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-08-03T20:47:55.996 DEBUG:teuthology.orchestra.run.smithi032:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-08-03T20:47:55.996 DEBUG:teuthology.orchestra.run.smithi032:> ') 2024-08-03T20:47:56.115 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:56 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-08-03T20:47:56.186 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:56 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-08-03T20:47:56.194 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-08-03T20:47:56.194 DEBUG:teuthology.orchestra.run.smithi032:> ip netns list 2024-08-03T20:47:56.230 INFO:teuthology.orchestra.run.smithi032.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-08-03T20:47:56.231 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-08-03T20:47:56.231 DEBUG:teuthology.orchestra.run.smithi032:> ip netns list-id 2024-08-03T20:47:56.305 INFO:teuthology.orchestra.run.smithi032.stdout:nsid 0 (iproute2 netns name: ceph-ns--home-ubuntu-cephtest-mnt.0) 2024-08-03T20:47:56.306 DEBUG:teuthology.orchestra.run.smithi032:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-08-03T20:47:56.306 DEBUG:teuthology.orchestra.run.smithi032:> set -e 2024-08-03T20:47:56.306 DEBUG:teuthology.orchestra.run.smithi032:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.2 2024-08-03T20:47:56.306 DEBUG:teuthology.orchestra.run.smithi032:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.2 1 2024-08-03T20:47:56.306 DEBUG:teuthology.orchestra.run.smithi032:> ') 2024-08-03T20:47:56.425 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:56 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-08-03T20:47:56.477 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:56 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-08-03T20:47:56.486 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-08-03T20:47:56.486 DEBUG:teuthology.orchestra.run.smithi032:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr 2024-08-03T20:47:56.541 INFO:teuthology.orchestra.run.smithi032.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-08-03T20:47:56.542 INFO:teuthology.orchestra.run.smithi032.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-08-03T20:47:56.542 INFO:teuthology.orchestra.run.smithi032.stdout: inet 127.0.0.1/8 scope host lo 2024-08-03T20:47:56.542 INFO:teuthology.orchestra.run.smithi032.stdout: valid_lft forever preferred_lft forever 2024-08-03T20:47:56.542 INFO:teuthology.orchestra.run.smithi032.stdout: inet6 ::1/128 scope host 2024-08-03T20:47:56.542 INFO:teuthology.orchestra.run.smithi032.stdout: valid_lft forever preferred_lft forever 2024-08-03T20:47:56.542 INFO:teuthology.orchestra.run.smithi032.stdout:2: veth0@if7: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-08-03T20:47:56.542 INFO:teuthology.orchestra.run.smithi032.stdout: link/ether c2:06:19:14:61:93 brd ff:ff:ff:ff:ff:ff link-netnsid 0 2024-08-03T20:47:56.543 INFO:teuthology.orchestra.run.smithi032.stdout: inet 192.168.0.1/16 brd 192.168.255.255 scope global veth0 2024-08-03T20:47:56.543 INFO:teuthology.orchestra.run.smithi032.stdout: valid_lft forever preferred_lft forever 2024-08-03T20:47:56.543 INFO:teuthology.orchestra.run.smithi032.stdout: inet6 fe80::c006:19ff:fe14:6193/64 scope link 2024-08-03T20:47:56.543 INFO:teuthology.orchestra.run.smithi032.stdout: valid_lft forever preferred_lft forever 2024-08-03T20:47:56.544 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-08-03T20:47:56.544 DEBUG:teuthology.orchestra.run.smithi032:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr 2024-08-03T20:47:56.600 INFO:teuthology.orchestra.run.smithi032.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-08-03T20:47:56.600 INFO:teuthology.orchestra.run.smithi032.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-08-03T20:47:56.600 INFO:teuthology.orchestra.run.smithi032.stdout: inet 127.0.0.1/8 scope host lo 2024-08-03T20:47:56.601 INFO:teuthology.orchestra.run.smithi032.stdout: valid_lft forever preferred_lft forever 2024-08-03T20:47:56.601 INFO:teuthology.orchestra.run.smithi032.stdout: inet6 ::1/128 scope host 2024-08-03T20:47:56.601 INFO:teuthology.orchestra.run.smithi032.stdout: valid_lft forever preferred_lft forever 2024-08-03T20:47:56.601 INFO:teuthology.orchestra.run.smithi032.stdout:2: veth0@if7: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-08-03T20:47:56.601 INFO:teuthology.orchestra.run.smithi032.stdout: link/ether c2:06:19:14:61:93 brd ff:ff:ff:ff:ff:ff link-netnsid 0 2024-08-03T20:47:56.601 INFO:teuthology.orchestra.run.smithi032.stdout: inet 192.168.0.1/16 brd 192.168.255.255 scope global veth0 2024-08-03T20:47:56.601 INFO:teuthology.orchestra.run.smithi032.stdout: valid_lft forever preferred_lft forever 2024-08-03T20:47:56.601 INFO:teuthology.orchestra.run.smithi032.stdout: inet6 fe80::c006:19ff:fe14:6193/64 scope link 2024-08-03T20:47:56.601 INFO:teuthology.orchestra.run.smithi032.stdout: valid_lft forever preferred_lft forever 2024-08-03T20:47:56.603 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.2' with 192.168.0.2/16 2024-08-03T20:47:56.603 DEBUG:teuthology.orchestra.run.smithi032:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-08-03T20:47:56.603 DEBUG:teuthology.orchestra.run.smithi032:> set -e 2024-08-03T20:47:56.603 DEBUG:teuthology.orchestra.run.smithi032:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.2 type veth peer name brx.1 2024-08-03T20:47:56.603 DEBUG:teuthology.orchestra.run.smithi032:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.2 ip addr add 192.168.0.2/16 brd 192.168.255.255 dev veth0 2024-08-03T20:47:56.604 DEBUG:teuthology.orchestra.run.smithi032:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.2 ip link set veth0 up 2024-08-03T20:47:56.604 DEBUG:teuthology.orchestra.run.smithi032:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.2 ip link set lo up 2024-08-03T20:47:56.604 DEBUG:teuthology.orchestra.run.smithi032:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.2 ip route add default via 192.168.255.254 2024-08-03T20:47:56.604 DEBUG:teuthology.orchestra.run.smithi032:> ') 2024-08-03T20:47:56.683 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:56 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-08-03T20:47:56.815 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:56 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-08-03T20:47:56.824 DEBUG:teuthology.orchestra.run.smithi032:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-08-03T20:47:56.824 DEBUG:teuthology.orchestra.run.smithi032:> set -e 2024-08-03T20:47:56.824 DEBUG:teuthology.orchestra.run.smithi032:> sudo ip link set brx.1 up 2024-08-03T20:47:56.824 DEBUG:teuthology.orchestra.run.smithi032:> sudo ip link set dev brx.1 master ceph-brx 2024-08-03T20:47:56.824 DEBUG:teuthology.orchestra.run.smithi032:> ') 2024-08-03T20:47:56.903 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:56 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-08-03T20:47:56.984 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:56 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-08-03T20:47:56.993 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-08-03T20:47:56.993 DEBUG:teuthology.orchestra.run.smithi032:> mkdir -p -v /home/ubuntu/cephtest/mnt.2 2024-08-03T20:47:57.029 INFO:teuthology.orchestra.run.smithi032.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.2' 2024-08-03T20:47:57.029 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-08-03T20:47:57.029 DEBUG:teuthology.orchestra.run.smithi032:> chmod 0000 /home/ubuntu/cephtest/mnt.2 2024-08-03T20:47:57.104 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2024-08-03T20:47:57.104 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-08-03T20:47:57.104 DEBUG:teuthology.orchestra.run.smithi032:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.2 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=2,mds_namespace=cephfs 2024-08-03T20:47:57.300 INFO:teuthology.orchestra.run.smithi032.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=2,mds_namespace=cephfs 2024-08-03T20:47:57.301 INFO:teuthology.orchestra.run.smithi032.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=2,mds_namespace=cephfs". 2024-08-03T20:47:57.301 INFO:teuthology.orchestra.run.smithi032.stdout:invalid new device string format 2024-08-03T20:47:57.321 INFO:teuthology.orchestra.run.smithi032.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=2,mds_namespace=cephfs 2024-08-03T20:47:57.321 INFO:teuthology.orchestra.run.smithi032.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=2,mds_namespace=cephfs". 2024-08-03T20:47:57.321 INFO:teuthology.orchestra.run.smithi032.stdout:invalid new device string format 2024-08-03T20:47:57.321 INFO:teuthology.orchestra.run.smithi032.stdout:mount.ceph: resolved to: "172.21.15.32:3300,172.21.15.118:3300,172.21.15.118:3301" 2024-08-03T20:47:57.322 INFO:teuthology.orchestra.run.smithi032.stdout:mount.ceph: switching to using v1 address with old syntax 2024-08-03T20:47:57.350 INFO:teuthology.orchestra.run.smithi032.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=2,mds_namespace=cephfs 2024-08-03T20:47:57.350 INFO:teuthology.orchestra.run.smithi032.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=2,mds_namespace=cephfs". 2024-08-03T20:47:57.350 INFO:teuthology.orchestra.run.smithi032.stdout:invalid new device string format 2024-08-03T20:47:57.350 INFO:teuthology.orchestra.run.smithi032.stdout:mount.ceph: resolved to: "172.21.15.32:3300,172.21.15.118:3300,172.21.15.118:3301" 2024-08-03T20:47:57.351 INFO:teuthology.orchestra.run.smithi032.stdout:mount.ceph: switching to using v1 address with old syntax 2024-08-03T20:47:57.351 INFO:teuthology.orchestra.run.smithi032.stdout:mount.ceph: resolved to: "172.21.15.32:6789,172.21.15.118:6789,172.21.15.118:6790" 2024-08-03T20:47:57.351 INFO:teuthology.orchestra.run.smithi032.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.32:6789,172.21.15.118:6789,172.21.15.118:6790:/ 2024-08-03T20:47:57.351 INFO:teuthology.orchestra.run.smithi032.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=2,mds_namespace=cephfs,key=2,fsid=40ab141a-01d7-49b9-ae3c-2edec9876e95" will pass to kernel 2024-08-03T20:47:57.352 INFO:teuthology.orchestra.run.smithi032.stdout:mount: /home/ubuntu/cephtest/mnt.2 does not contain SELinux labels. 2024-08-03T20:47:57.352 INFO:teuthology.orchestra.run.smithi032.stdout: You just mounted a file system that supports labels which does not 2024-08-03T20:47:57.352 INFO:teuthology.orchestra.run.smithi032.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-08-03T20:47:57.352 INFO:teuthology.orchestra.run.smithi032.stdout: applications will generate AVC messages and not be allowed access to 2024-08-03T20:47:57.352 INFO:teuthology.orchestra.run.smithi032.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-08-03T20:47:57.355 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-08-03T20:47:57.355 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-08-03T20:47:57.355 DEBUG:teuthology.orchestra.run.smithi032:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.2 2024-08-03T20:47:57.411 DEBUG:teuthology.orchestra.run.smithi032:> (cd /home/ubuntu/cephtest/mnt.2 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-08-03T20:47:57.491 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:57 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-08-03T20:47:57.494 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:57 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-08-03T20:47:57.502 INFO:teuthology.orchestra.run.smithi032.stdout:client4657 2024-08-03T20:47:57.502 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-08-03T20:47:57.502 DEBUG:teuthology.orchestra.run.smithi032:> dd if=/proc/self/mounts of=/dev/stdout 2024-08-03T20:47:57.538 DEBUG:teuthology.orchestra.run.smithi032:> (cd /home/ubuntu/cephtest/mnt.2 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-08-03T20:47:57.656 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:57 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-08-03T20:47:57.660 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:57 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-08-03T20:47:57.667 INFO:teuthology.orchestra.run.smithi032.stdout:client4657 2024-08-03T20:47:57.668 DEBUG:teuthology.orchestra.run.smithi032:> (cd /home/ubuntu/cephtest/mnt.2 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-08-03T20:47:57.747 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:57 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-08-03T20:47:57.750 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:57 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-08-03T20:47:57.758 INFO:teuthology.orchestra.run.smithi032.stdout:client4657 2024-08-03T20:47:57.758 DEBUG:teuthology.orchestra.run.smithi032:> (cd /home/ubuntu/cephtest/mnt.2 && exec stdin-killer --timeout=300 -- bash -c 'sudo dd if=/sys/kernel/debug/ceph/40ab141a-01d7-49b9-ae3c-2edec9876e95.client4657/status') 2024-08-03T20:47:57.837 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:57 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-08-03T20:47:57.858 INFO:teuthology.orchestra.run.smithi032.stdout:instance: client.4657 (3)172.21.15.32:0/3527813429 2024-08-03T20:47:57.859 INFO:teuthology.orchestra.run.smithi032.stdout:blocklisted: false 2024-08-03T20:47:57.859 INFO:teuthology.orchestra.run.smithi032.stderr:0+1 records in 2024-08-03T20:47:57.859 INFO:teuthology.orchestra.run.smithi032.stderr:0+1 records out 2024-08-03T20:47:57.859 INFO:teuthology.orchestra.run.smithi032.stderr:70 bytes copied, 5.4823e-05 s, 1.3 MB/s 2024-08-03T20:47:57.861 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:57 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-08-03T20:47:57.870 INFO:tasks.kclient:client.1 config is {'syntax': 'v1'} 2024-08-03T20:47:57.870 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-08-03T20:47:57.870 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2024-08-03T20:47:57.870 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-08-03T20:47:58.099 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:58.097+0000 7fdd64a00640 1 -- 172.21.15.32:0/4030888502 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdd600b9300 msgr2=0x7fdd600d1f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:47:58.099 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:58.097+0000 7fdd64a00640 1 --2- 172.21.15.32:0/4030888502 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdd600b9300 0x7fdd600d1f10 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fdd600d2a00 tx=0x7fdd4802a560 comp rx=0 tx=0).stop 2024-08-03T20:47:58.099 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:58.097+0000 7fdd64a00640 1 -- 172.21.15.32:0/4030888502 shutdown_connections 2024-08-03T20:47:58.099 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:58.097+0000 7fdd64a00640 1 --2- 172.21.15.32:0/4030888502 >> v2:172.21.15.118:3300/0 conn(0x7fdd600d2450 0x7fdd600cf6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:58.099 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:58.097+0000 7fdd64a00640 1 --2- 172.21.15.32:0/4030888502 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdd600b9300 0x7fdd600d1f10 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:58.099 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:58.097+0000 7fdd64a00640 1 -- 172.21.15.32:0/4030888502 >> 172.21.15.32:0/4030888502 conn(0x7fdd600d4160 msgr2=0x7fdd600d4570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:47:58.099 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:58.097+0000 7fdd64a00640 1 -- 172.21.15.32:0/4030888502 shutdown_connections 2024-08-03T20:47:58.100 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:58.097+0000 7fdd64a00640 1 -- 172.21.15.32:0/4030888502 wait complete. 2024-08-03T20:47:58.100 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:58.099+0000 7fdd64a00640 1 Processor -- start 2024-08-03T20:47:58.100 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:58.099+0000 7fdd64a00640 1 -- start start 2024-08-03T20:47:58.103 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:58.101+0000 7fdd64a00640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fdd600b9300 0x7fdd6015ed80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:47:58.103 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:58.101+0000 7fdd64a00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fdd600d2450 0x7fdd6009df30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:47:58.103 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:58.101+0000 7fdd64a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdd60165470 0x7fdd60165880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:47:58.103 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:58.101+0000 7fdd64a00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fdd600cffb0 con 0x7fdd60165470 2024-08-03T20:47:58.103 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:58.101+0000 7fdd64a00640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7fdd600cfe30 con 0x7fdd600d2450 2024-08-03T20:47:58.103 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:58.101+0000 7fdd64a00640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7fdd600d4e70 con 0x7fdd600b9300 2024-08-03T20:47:58.103 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:58.101+0000 7fdd5fe00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdd60165470 0x7fdd60165880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:47:58.103 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:58.101+0000 7fdd5fe00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdd60165470 0x7fdd60165880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:52466/0 (socket says 172.21.15.32:52466) 2024-08-03T20:47:58.103 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:58.101+0000 7fdd5fe00640 1 -- 172.21.15.32:0/565955467 learned_addr learned my addr 172.21.15.32:0/565955467 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:47:58.104 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:58.101+0000 7fdd5f400640 1 --2- 172.21.15.32:0/565955467 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fdd600b9300 0x7fdd6015ed80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:47:58.104 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:58.101+0000 7fdd56a00640 1 --2- 172.21.15.32:0/565955467 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fdd600d2450 0x7fdd6009df30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:47:58.104 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:58.101+0000 7fdd5fe00640 1 -- 172.21.15.32:0/565955467 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fdd600b9300 msgr2=0x7fdd6015ed80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:47:58.104 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:58.101+0000 7fdd5fe00640 1 --2- 172.21.15.32:0/565955467 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fdd600b9300 0x7fdd6015ed80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:58.104 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:58.101+0000 7fdd5fe00640 1 -- 172.21.15.32:0/565955467 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fdd600d2450 msgr2=0x7fdd6009df30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:47:58.104 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:58.101+0000 7fdd5fe00640 1 --2- 172.21.15.32:0/565955467 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fdd600d2450 0x7fdd6009df30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:58.104 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:58.101+0000 7fdd5fe00640 1 -- 172.21.15.32:0/565955467 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdd60165dc0 con 0x7fdd60165470 2024-08-03T20:47:58.105 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:58.102+0000 7fdd5fe00640 1 --2- 172.21.15.32:0/565955467 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdd60165470 0x7fdd60165880 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fdd5800ea80 tx=0x7fdd5800ef40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:47:58.105 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:58.102+0000 7fdd57e00640 1 -- 172.21.15.32:0/565955467 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd58004b20 con 0x7fdd60165470 2024-08-03T20:47:58.105 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:58.102+0000 7fdd57e00640 1 -- 172.21.15.32:0/565955467 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fdd58004ce0 con 0x7fdd60165470 2024-08-03T20:47:58.105 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:58.102+0000 7fdd57e00640 1 -- 172.21.15.32:0/565955467 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd58005280 con 0x7fdd60165470 2024-08-03T20:47:58.106 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:58.102+0000 7fdd64a00640 1 -- 172.21.15.32:0/565955467 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdd600b2ea0 con 0x7fdd60165470 2024-08-03T20:47:58.106 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:58.102+0000 7fdd64a00640 1 -- 172.21.15.32:0/565955467 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdd601602a0 con 0x7fdd60165470 2024-08-03T20:47:58.106 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:58.103+0000 7fdd64a00640 1 -- 172.21.15.32:0/565955467 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdd600d18f0 con 0x7fdd60165470 2024-08-03T20:47:58.106 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:58.105+0000 7fdd57e00640 1 -- 172.21.15.32:0/565955467 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7fdd58005420 con 0x7fdd60165470 2024-08-03T20:47:58.107 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:58.106+0000 7fdd57e00640 1 --2- 172.21.15.32:0/565955467 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fdd3c077010 0x7fdd3c079520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:47:58.107 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:58.106+0000 7fdd57e00640 1 -- 172.21.15.32:0/565955467 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7fdd58099390 con 0x7fdd60165470 2024-08-03T20:47:58.108 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:58.107+0000 7fdd5f400640 1 --2- 172.21.15.32:0/565955467 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fdd3c077010 0x7fdd3c079520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:47:58.108 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:58.107+0000 7fdd5f400640 1 --2- 172.21.15.32:0/565955467 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fdd3c077010 0x7fdd3c079520 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fdd50001720 tx=0x7fdd50009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:47:58.112 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:58.110+0000 7fdd57e00640 1 -- 172.21.15.32:0/565955467 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7fdd60165dc0 con 0x7fdd60165470 2024-08-03T20:47:58.355 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:58.354+0000 7fdd64a00640 1 -- 172.21.15.32:0/565955467 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7fdd600b3ac0 con 0x7fdd60165470 2024-08-03T20:47:58.356 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:58.354+0000 7fdd57e00640 1 -- 172.21.15.32:0/565955467 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v8) ==== 53+0+73 (secure 0 0 0) 0x7fdd58061f30 con 0x7fdd60165470 2024-08-03T20:47:58.356 INFO:teuthology.orchestra.run.smithi032.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-08-03T20:47:58.358 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:58.356+0000 7fdd64a00640 1 -- 172.21.15.32:0/565955467 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fdd3c077010 msgr2=0x7fdd3c079520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:47:58.358 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:58.356+0000 7fdd64a00640 1 --2- 172.21.15.32:0/565955467 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fdd3c077010 0x7fdd3c079520 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fdd50001720 tx=0x7fdd50009040 comp rx=0 tx=0).stop 2024-08-03T20:47:58.358 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:58.357+0000 7fdd64a00640 1 -- 172.21.15.32:0/565955467 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdd60165470 msgr2=0x7fdd60165880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:47:58.359 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:58.357+0000 7fdd64a00640 1 --2- 172.21.15.32:0/565955467 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdd60165470 0x7fdd60165880 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fdd5800ea80 tx=0x7fdd5800ef40 comp rx=0 tx=0).stop 2024-08-03T20:47:58.359 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:58.357+0000 7fdd64a00640 1 -- 172.21.15.32:0/565955467 shutdown_connections 2024-08-03T20:47:58.359 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:58.357+0000 7fdd64a00640 1 --2- 172.21.15.32:0/565955467 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fdd3c077010 0x7fdd3c079520 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:58.359 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:58.357+0000 7fdd64a00640 1 --2- 172.21.15.32:0/565955467 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fdd600d2450 0x7fdd6009df30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:58.359 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:58.357+0000 7fdd64a00640 1 --2- 172.21.15.32:0/565955467 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdd60165470 0x7fdd60165880 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:58.359 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:58.357+0000 7fdd64a00640 1 --2- 172.21.15.32:0/565955467 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fdd600b9300 0x7fdd6015ed80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:47:58.359 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:58.357+0000 7fdd64a00640 1 -- 172.21.15.32:0/565955467 >> 172.21.15.32:0/565955467 conn(0x7fdd600d4160 msgr2=0x7fdd600cf420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:47:58.359 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:58.357+0000 7fdd64a00640 1 -- 172.21.15.32:0/565955467 shutdown_connections 2024-08-03T20:47:58.359 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:47:58.357+0000 7fdd64a00640 1 -- 172.21.15.32:0/565955467 wait complete. 2024-08-03T20:47:58.380 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-08-03T20:47:58.380 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-08-03T20:47:58.380 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi118.front.sepia.ceph.com 2024-08-03T20:47:58.380 INFO:tasks.cephfs.mount:self.client.name = client.1 2024-08-03T20:47:58.381 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2024-08-03T20:47:58.381 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-08-03T20:47:58.381 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-08-03T20:47:58.381 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-08-03T20:47:58.381 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.1' 2024-08-03T20:47:58.382 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-08-03T20:47:58.382 DEBUG:teuthology.orchestra.run.smithi118:> ip addr 2024-08-03T20:47:58.422 INFO:teuthology.orchestra.run.smithi118.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-08-03T20:47:58.423 INFO:teuthology.orchestra.run.smithi118.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-08-03T20:47:58.423 INFO:teuthology.orchestra.run.smithi118.stdout: inet 127.0.0.1/8 scope host lo 2024-08-03T20:47:58.423 INFO:teuthology.orchestra.run.smithi118.stdout: valid_lft forever preferred_lft forever 2024-08-03T20:47:58.423 INFO:teuthology.orchestra.run.smithi118.stdout: inet6 ::1/128 scope host 2024-08-03T20:47:58.423 INFO:teuthology.orchestra.run.smithi118.stdout: valid_lft forever preferred_lft forever 2024-08-03T20:47:58.423 INFO:teuthology.orchestra.run.smithi118.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-08-03T20:47:58.423 INFO:teuthology.orchestra.run.smithi118.stdout: link/ether 0c:c4:7a:d8:a4:cc brd ff:ff:ff:ff:ff:ff 2024-08-03T20:47:58.423 INFO:teuthology.orchestra.run.smithi118.stdout: altname enp7s0f0 2024-08-03T20:47:58.423 INFO:teuthology.orchestra.run.smithi118.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-08-03T20:47:58.423 INFO:teuthology.orchestra.run.smithi118.stdout: link/ether 0c:c4:7a:d8:a4:cd brd ff:ff:ff:ff:ff:ff 2024-08-03T20:47:58.423 INFO:teuthology.orchestra.run.smithi118.stdout: altname enp7s0f1 2024-08-03T20:47:58.423 INFO:teuthology.orchestra.run.smithi118.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-08-03T20:47:58.423 INFO:teuthology.orchestra.run.smithi118.stdout: link/ether 0c:c4:7a:8f:d7:24 brd ff:ff:ff:ff:ff:ff 2024-08-03T20:47:58.424 INFO:teuthology.orchestra.run.smithi118.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-08-03T20:47:58.424 INFO:teuthology.orchestra.run.smithi118.stdout: link/ether 0c:c4:7a:8f:d7:25 brd ff:ff:ff:ff:ff:ff 2024-08-03T20:47:58.424 INFO:teuthology.orchestra.run.smithi118.stdout: inet 172.21.15.118/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-08-03T20:47:58.424 INFO:teuthology.orchestra.run.smithi118.stdout: valid_lft 42656sec preferred_lft 42656sec 2024-08-03T20:47:58.424 INFO:teuthology.orchestra.run.smithi118.stdout: inet6 fe80::ec4:7aff:fe8f:d725/64 scope link 2024-08-03T20:47:58.424 INFO:teuthology.orchestra.run.smithi118.stdout: valid_lft forever preferred_lft forever 2024-08-03T20:47:58.424 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-08-03T20:47:58.424 DEBUG:teuthology.orchestra.run.smithi118:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-08-03T20:47:58.424 DEBUG:teuthology.orchestra.run.smithi118:> set -e 2024-08-03T20:47:58.424 DEBUG:teuthology.orchestra.run.smithi118:> sudo ip link add name ceph-brx type bridge 2024-08-03T20:47:58.425 DEBUG:teuthology.orchestra.run.smithi118:> sudo ip addr flush dev ceph-brx 2024-08-03T20:47:58.425 DEBUG:teuthology.orchestra.run.smithi118:> sudo ip link set ceph-brx up 2024-08-03T20:47:58.425 DEBUG:teuthology.orchestra.run.smithi118:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-08-03T20:47:58.425 DEBUG:teuthology.orchestra.run.smithi118:> ') 2024-08-03T20:47:58.545 INFO:teuthology.orchestra.run.smithi118.stderr:2024-08-03T20:47:58 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-08-03T20:47:58.864 INFO:teuthology.orchestra.run.smithi118.stderr:2024-08-03T20:47:58 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-08-03T20:47:58.874 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-08-03T20:47:58.874 DEBUG:teuthology.orchestra.run.smithi118:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-08-03T20:47:58.930 INFO:teuthology.orchestra.run.smithi118.stdout:1 2024-08-03T20:47:58.934 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-08-03T20:47:58.935 DEBUG:teuthology.orchestra.run.smithi118:> route 2024-08-03T20:47:58.987 INFO:teuthology.orchestra.run.smithi118.stdout:Kernel IP routing table 2024-08-03T20:47:58.987 INFO:teuthology.orchestra.run.smithi118.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-08-03T20:47:58.987 INFO:teuthology.orchestra.run.smithi118.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-08-03T20:47:58.987 INFO:teuthology.orchestra.run.smithi118.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-08-03T20:47:58.987 INFO:teuthology.orchestra.run.smithi118.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-08-03T20:47:58.988 DEBUG:teuthology.orchestra.run.smithi118:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-08-03T20:47:58.988 DEBUG:teuthology.orchestra.run.smithi118:> set -e 2024-08-03T20:47:58.988 DEBUG:teuthology.orchestra.run.smithi118:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-08-03T20:47:58.988 DEBUG:teuthology.orchestra.run.smithi118:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-08-03T20:47:58.988 DEBUG:teuthology.orchestra.run.smithi118:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-08-03T20:47:58.988 DEBUG:teuthology.orchestra.run.smithi118:> ') 2024-08-03T20:47:59.068 INFO:teuthology.orchestra.run.smithi118.stderr:2024-08-03T20:47:59 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-08-03T20:47:59.684 INFO:teuthology.orchestra.run.smithi118.stderr:2024-08-03T20:47:59 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-08-03T20:47:59.692 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-08-03T20:47:59.692 DEBUG:teuthology.orchestra.run.smithi118:> ip netns list 2024-08-03T20:47:59.731 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-08-03T20:47:59.731 DEBUG:teuthology.orchestra.run.smithi118:> ip netns list-id 2024-08-03T20:47:59.807 DEBUG:teuthology.orchestra.run.smithi118:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-08-03T20:47:59.807 DEBUG:teuthology.orchestra.run.smithi118:> set -e 2024-08-03T20:47:59.807 DEBUG:teuthology.orchestra.run.smithi118:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.1 2024-08-03T20:47:59.808 DEBUG:teuthology.orchestra.run.smithi118:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.1 0 2024-08-03T20:47:59.808 DEBUG:teuthology.orchestra.run.smithi118:> ') 2024-08-03T20:47:59.927 INFO:teuthology.orchestra.run.smithi118.stderr:2024-08-03T20:47:59 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-08-03T20:47:59.983 INFO:teuthology.orchestra.run.smithi118.stderr:2024-08-03T20:47:59 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-08-03T20:47:59.992 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.1' with 192.168.0.1/16 2024-08-03T20:47:59.993 DEBUG:teuthology.orchestra.run.smithi118:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-08-03T20:47:59.993 DEBUG:teuthology.orchestra.run.smithi118:> set -e 2024-08-03T20:47:59.993 DEBUG:teuthology.orchestra.run.smithi118:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.1 type veth peer name brx.0 2024-08-03T20:47:59.993 DEBUG:teuthology.orchestra.run.smithi118:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-08-03T20:47:59.993 DEBUG:teuthology.orchestra.run.smithi118:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip link set veth0 up 2024-08-03T20:47:59.993 DEBUG:teuthology.orchestra.run.smithi118:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip link set lo up 2024-08-03T20:47:59.993 DEBUG:teuthology.orchestra.run.smithi118:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip route add default via 192.168.255.254 2024-08-03T20:47:59.993 DEBUG:teuthology.orchestra.run.smithi118:> ') 2024-08-03T20:48:00.075 INFO:teuthology.orchestra.run.smithi118.stderr:2024-08-03T20:48:00 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-08-03T20:48:00.229 INFO:teuthology.orchestra.run.smithi118.stderr:2024-08-03T20:48:00 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-08-03T20:48:00.238 DEBUG:teuthology.orchestra.run.smithi118:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-08-03T20:48:00.238 DEBUG:teuthology.orchestra.run.smithi118:> set -e 2024-08-03T20:48:00.239 DEBUG:teuthology.orchestra.run.smithi118:> sudo ip link set brx.0 up 2024-08-03T20:48:00.239 DEBUG:teuthology.orchestra.run.smithi118:> sudo ip link set dev brx.0 master ceph-brx 2024-08-03T20:48:00.239 DEBUG:teuthology.orchestra.run.smithi118:> ') 2024-08-03T20:48:00.319 INFO:teuthology.orchestra.run.smithi118.stderr:2024-08-03T20:48:00 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-08-03T20:48:00.407 INFO:teuthology.orchestra.run.smithi118.stderr:2024-08-03T20:48:00 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-08-03T20:48:00.414 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-08-03T20:48:00.415 DEBUG:teuthology.orchestra.run.smithi118:> mkdir -p -v /home/ubuntu/cephtest/mnt.1 2024-08-03T20:48:00.449 INFO:teuthology.orchestra.run.smithi118.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.1' 2024-08-03T20:48:00.449 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-08-03T20:48:00.450 DEBUG:teuthology.orchestra.run.smithi118:> chmod 0000 /home/ubuntu/cephtest/mnt.1 2024-08-03T20:48:00.524 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2024-08-03T20:48:00.524 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-08-03T20:48:00.524 DEBUG:teuthology.orchestra.run.smithi118:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.1 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=1,mds_namespace=cephfs 2024-08-03T20:48:00.746 INFO:teuthology.orchestra.run.smithi118.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=1,mds_namespace=cephfs 2024-08-03T20:48:00.747 INFO:teuthology.orchestra.run.smithi118.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=1,mds_namespace=cephfs". 2024-08-03T20:48:00.747 INFO:teuthology.orchestra.run.smithi118.stdout:invalid new device string format 2024-08-03T20:48:01.408 INFO:teuthology.orchestra.run.smithi118.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=1,mds_namespace=cephfs 2024-08-03T20:48:01.408 INFO:teuthology.orchestra.run.smithi118.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=1,mds_namespace=cephfs". 2024-08-03T20:48:01.409 INFO:teuthology.orchestra.run.smithi118.stdout:invalid new device string format 2024-08-03T20:48:01.409 INFO:teuthology.orchestra.run.smithi118.stdout:mount.ceph: resolved to: "172.21.15.32:3300,172.21.15.118:3300,172.21.15.118:3301" 2024-08-03T20:48:01.409 INFO:teuthology.orchestra.run.smithi118.stdout:mount.ceph: switching to using v1 address with old syntax 2024-08-03T20:48:01.443 INFO:teuthology.orchestra.run.smithi118.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=1,mds_namespace=cephfs 2024-08-03T20:48:01.443 INFO:teuthology.orchestra.run.smithi118.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=1,mds_namespace=cephfs". 2024-08-03T20:48:01.443 INFO:teuthology.orchestra.run.smithi118.stdout:invalid new device string format 2024-08-03T20:48:01.443 INFO:teuthology.orchestra.run.smithi118.stdout:mount.ceph: resolved to: "172.21.15.32:3300,172.21.15.118:3300,172.21.15.118:3301" 2024-08-03T20:48:01.443 INFO:teuthology.orchestra.run.smithi118.stdout:mount.ceph: switching to using v1 address with old syntax 2024-08-03T20:48:01.443 INFO:teuthology.orchestra.run.smithi118.stdout:mount.ceph: resolved to: "172.21.15.32:6789,172.21.15.118:6789,172.21.15.118:6790" 2024-08-03T20:48:01.444 INFO:teuthology.orchestra.run.smithi118.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.32:6789,172.21.15.118:6789,172.21.15.118:6790:/ 2024-08-03T20:48:01.444 INFO:teuthology.orchestra.run.smithi118.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=1,mds_namespace=cephfs,key=1,fsid=40ab141a-01d7-49b9-ae3c-2edec9876e95" will pass to kernel 2024-08-03T20:48:01.445 INFO:teuthology.orchestra.run.smithi118.stdout:mount: /home/ubuntu/cephtest/mnt.1 does not contain SELinux labels. 2024-08-03T20:48:01.445 INFO:teuthology.orchestra.run.smithi118.stdout: You just mounted a file system that supports labels which does not 2024-08-03T20:48:01.445 INFO:teuthology.orchestra.run.smithi118.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-08-03T20:48:01.445 INFO:teuthology.orchestra.run.smithi118.stdout: applications will generate AVC messages and not be allowed access to 2024-08-03T20:48:01.446 INFO:teuthology.orchestra.run.smithi118.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-08-03T20:48:01.450 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-08-03T20:48:01.450 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-08-03T20:48:01.451 DEBUG:teuthology.orchestra.run.smithi118:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.1 2024-08-03T20:48:01.506 DEBUG:teuthology.orchestra.run.smithi118:> (cd /home/ubuntu/cephtest/mnt.1 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-08-03T20:48:01.586 INFO:teuthology.orchestra.run.smithi118.stderr:2024-08-03T20:48:01 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-08-03T20:48:01.590 INFO:teuthology.orchestra.run.smithi118.stderr:2024-08-03T20:48:01 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-08-03T20:48:01.598 INFO:teuthology.orchestra.run.smithi118.stdout:client4686 2024-08-03T20:48:01.598 DEBUG:teuthology.orchestra.run.smithi118:> set -ex 2024-08-03T20:48:01.598 DEBUG:teuthology.orchestra.run.smithi118:> dd if=/proc/self/mounts of=/dev/stdout 2024-08-03T20:48:01.634 DEBUG:teuthology.orchestra.run.smithi118:> (cd /home/ubuntu/cephtest/mnt.1 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-08-03T20:48:01.751 INFO:teuthology.orchestra.run.smithi118.stderr:2024-08-03T20:48:01 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-08-03T20:48:01.755 INFO:teuthology.orchestra.run.smithi118.stderr:2024-08-03T20:48:01 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-08-03T20:48:01.763 INFO:teuthology.orchestra.run.smithi118.stdout:client4686 2024-08-03T20:48:01.763 DEBUG:teuthology.orchestra.run.smithi118:> (cd /home/ubuntu/cephtest/mnt.1 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-08-03T20:48:01.840 INFO:teuthology.orchestra.run.smithi118.stderr:2024-08-03T20:48:01 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-08-03T20:48:01.844 INFO:teuthology.orchestra.run.smithi118.stderr:2024-08-03T20:48:01 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-08-03T20:48:01.852 INFO:teuthology.orchestra.run.smithi118.stdout:client4686 2024-08-03T20:48:01.853 DEBUG:teuthology.orchestra.run.smithi118:> (cd /home/ubuntu/cephtest/mnt.1 && exec stdin-killer --timeout=300 -- bash -c 'sudo dd if=/sys/kernel/debug/ceph/40ab141a-01d7-49b9-ae3c-2edec9876e95.client4686/status') 2024-08-03T20:48:01.930 INFO:teuthology.orchestra.run.smithi118.stderr:2024-08-03T20:48:01 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-08-03T20:48:01.951 INFO:teuthology.orchestra.run.smithi118.stdout:instance: client.4686 (3)172.21.15.118:0/4114532114 2024-08-03T20:48:01.951 INFO:teuthology.orchestra.run.smithi118.stdout:blocklisted: false 2024-08-03T20:48:01.952 INFO:teuthology.orchestra.run.smithi118.stderr:0+1 records in 2024-08-03T20:48:01.952 INFO:teuthology.orchestra.run.smithi118.stderr:0+1 records out 2024-08-03T20:48:01.952 INFO:teuthology.orchestra.run.smithi118.stderr:71 bytes copied, 5.4939e-05 s, 1.3 MB/s 2024-08-03T20:48:01.953 INFO:teuthology.orchestra.run.smithi118.stderr:2024-08-03T20:48:01 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-08-03T20:48:01.962 INFO:tasks.kclient:client.3 config is {'syntax': 'v1'} 2024-08-03T20:48:01.962 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-08-03T20:48:01.962 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.3 2024-08-03T20:48:01.962 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-08-03T20:48:02.202 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:02.201+0000 7f15bee00640 1 -- 172.21.15.32:0/1225543203 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f15b806f720 msgr2=0x7f15b806fb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:48:02.203 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:02.201+0000 7f15bee00640 1 --2- 172.21.15.32:0/1225543203 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f15b806f720 0x7f15b806fb30 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f15a8009830 tx=0x7f15a802a510 comp rx=0 tx=0).stop 2024-08-03T20:48:02.203 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:02.201+0000 7f15bee00640 1 -- 172.21.15.32:0/1225543203 shutdown_connections 2024-08-03T20:48:02.203 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:02.201+0000 7f15bee00640 1 --2- 172.21.15.32:0/1225543203 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f15b80ade80 0x7f15b80ae290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:02.203 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:02.201+0000 7f15bee00640 1 --2- 172.21.15.32:0/1225543203 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f15b806f720 0x7f15b806fb30 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:02.203 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:02.201+0000 7f15bee00640 1 -- 172.21.15.32:0/1225543203 >> 172.21.15.32:0/1225543203 conn(0x7f15b80b1bd0 msgr2=0x7f15b80b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:48:02.203 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:02.201+0000 7f15bee00640 1 -- 172.21.15.32:0/1225543203 shutdown_connections 2024-08-03T20:48:02.203 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:02.201+0000 7f15bee00640 1 -- 172.21.15.32:0/1225543203 wait complete. 2024-08-03T20:48:02.204 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:02.203+0000 7f15bee00640 1 Processor -- start 2024-08-03T20:48:02.204 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:02.203+0000 7f15bee00640 1 -- start start 2024-08-03T20:48:02.206 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:02.205+0000 7f15bee00640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f15b806f720 0x7f15b8148200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:48:02.206 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:02.205+0000 7f15bee00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f15b80ade80 0x7f15b815aa60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:48:02.207 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:02.205+0000 7f15bee00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f15b80b39a0 0x7f15b8072d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:48:02.207 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:02.205+0000 7f15bee00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f15b80aeb60 con 0x7f15b80b39a0 2024-08-03T20:48:02.207 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:02.205+0000 7f15bee00640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f15b80ae9e0 con 0x7f15b80ade80 2024-08-03T20:48:02.207 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:02.205+0000 7f15bee00640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7f15b80aece0 con 0x7f15b806f720 2024-08-03T20:48:02.207 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:02.205+0000 7f15bce00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f15b80b39a0 0x7f15b8072d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:48:02.207 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:02.205+0000 7f15bce00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f15b80b39a0 0x7f15b8072d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:55880/0 (socket says 172.21.15.32:55880) 2024-08-03T20:48:02.207 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:02.205+0000 7f15bce00640 1 -- 172.21.15.32:0/3172930198 learned_addr learned my addr 172.21.15.32:0/3172930198 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:48:02.207 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:02.205+0000 7f15b7e00640 1 --2- 172.21.15.32:0/3172930198 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f15b806f720 0x7f15b8148200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:48:02.207 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:02.205+0000 7f15b7400640 1 --2- 172.21.15.32:0/3172930198 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f15b80ade80 0x7f15b815aa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:48:02.209 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:02.206+0000 7f15b7e00640 1 -- 172.21.15.32:0/3172930198 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f15b80ade80 msgr2=0x7f15b815aa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:48:02.209 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:02.206+0000 7f15b7e00640 1 --2- 172.21.15.32:0/3172930198 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f15b80ade80 0x7f15b815aa60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:02.209 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:02.206+0000 7f15b7e00640 1 -- 172.21.15.32:0/3172930198 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f15b80b39a0 msgr2=0x7f15b8072d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:48:02.209 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:02.206+0000 7f15b7e00640 1 --2- 172.21.15.32:0/3172930198 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f15b80b39a0 0x7f15b8072d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:02.209 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:02.206+0000 7f15b7e00640 1 -- 172.21.15.32:0/3172930198 --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f15b8158d90 con 0x7f15b806f720 2024-08-03T20:48:02.209 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:02.206+0000 7f15bce00640 1 --2- 172.21.15.32:0/3172930198 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f15b80b39a0 0x7f15b8072d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-03T20:48:02.209 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:02.206+0000 7f15b7e00640 1 --2- 172.21.15.32:0/3172930198 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f15b806f720 0x7f15b8148200 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f15a8031000 tx=0x7f15a8033c20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:48:02.209 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:02.206+0000 7f15afe00640 1 -- 172.21.15.32:0/3172930198 <== mon.2 v2:172.21.15.118:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15a8005c90 con 0x7f15b806f720 2024-08-03T20:48:02.209 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:02.206+0000 7f15afe00640 1 -- 172.21.15.32:0/3172930198 <== mon.2 v2:172.21.15.118:3301/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f15a8031e50 con 0x7f15b806f720 2024-08-03T20:48:02.209 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:02.206+0000 7f15bee00640 1 -- 172.21.15.32:0/3172930198 --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f15b8148740 con 0x7f15b806f720 2024-08-03T20:48:02.209 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:02.206+0000 7f15afe00640 1 -- 172.21.15.32:0/3172930198 <== mon.2 v2:172.21.15.118:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15a8005280 con 0x7f15b806f720 2024-08-03T20:48:02.209 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:02.206+0000 7f15bee00640 1 -- 172.21.15.32:0/3172930198 --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f15b81f92a0 con 0x7f15b806f720 2024-08-03T20:48:02.210 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:02.207+0000 7f15bee00640 1 -- 172.21.15.32:0/3172930198 --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f15b806fb30 con 0x7f15b806f720 2024-08-03T20:48:02.210 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:02.209+0000 7f15afe00640 1 -- 172.21.15.32:0/3172930198 <== mon.2 v2:172.21.15.118:3301/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f15a8005420 con 0x7f15b806f720 2024-08-03T20:48:02.211 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:02.210+0000 7f15afe00640 1 --2- 172.21.15.32:0/3172930198 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f1598077010 0x7f1598079520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:48:02.211 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:02.210+0000 7f15afe00640 1 -- 172.21.15.32:0/3172930198 <== mon.2 v2:172.21.15.118:3301/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f15a80b98a0 con 0x7f15b806f720 2024-08-03T20:48:02.211 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:02.210+0000 7f15b7400640 1 --2- 172.21.15.32:0/3172930198 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f1598077010 0x7f1598079520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:48:02.212 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:02.211+0000 7f15b7400640 1 --2- 172.21.15.32:0/3172930198 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f1598077010 0x7f1598079520 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f15a4007fd0 tx=0x7f15a4001390 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:48:02.214 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:02.213+0000 7f15afe00640 1 -- 172.21.15.32:0/3172930198 <== mon.2 v2:172.21.15.118:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7f15b8148740 con 0x7f15b806f720 2024-08-03T20:48:02.460 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:02.459+0000 7f15bee00640 1 -- 172.21.15.32:0/3172930198 --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f15b8160f30 con 0x7f15b806f720 2024-08-03T20:48:02.461 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:02.459+0000 7f15afe00640 1 -- 172.21.15.32:0/3172930198 <== mon.2 v2:172.21.15.118:3301/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v8) ==== 53+0+73 (secure 0 0 0) 0x7f15a803f030 con 0x7f15b806f720 2024-08-03T20:48:02.461 INFO:teuthology.orchestra.run.smithi032.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-08-03T20:48:02.463 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:02.462+0000 7f15bee00640 1 -- 172.21.15.32:0/3172930198 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f1598077010 msgr2=0x7f1598079520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:48:02.464 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:02.462+0000 7f15bee00640 1 --2- 172.21.15.32:0/3172930198 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f1598077010 0x7f1598079520 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f15a4007fd0 tx=0x7f15a4001390 comp rx=0 tx=0).stop 2024-08-03T20:48:02.464 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:02.462+0000 7f15bee00640 1 -- 172.21.15.32:0/3172930198 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f15b806f720 msgr2=0x7f15b8148200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:48:02.464 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:02.462+0000 7f15bee00640 1 --2- 172.21.15.32:0/3172930198 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f15b806f720 0x7f15b8148200 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f15a8031000 tx=0x7f15a8033c20 comp rx=0 tx=0).stop 2024-08-03T20:48:02.464 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:02.462+0000 7f15bee00640 1 -- 172.21.15.32:0/3172930198 shutdown_connections 2024-08-03T20:48:02.464 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:02.462+0000 7f15bee00640 1 --2- 172.21.15.32:0/3172930198 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f1598077010 0x7f1598079520 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:02.464 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:02.462+0000 7f15bee00640 1 --2- 172.21.15.32:0/3172930198 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f15b80ade80 0x7f15b815aa60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:02.464 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:02.462+0000 7f15bee00640 1 --2- 172.21.15.32:0/3172930198 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f15b80b39a0 0x7f15b8072d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:02.464 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:02.462+0000 7f15bee00640 1 --2- 172.21.15.32:0/3172930198 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f15b806f720 0x7f15b8148200 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:02.464 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:02.462+0000 7f15bee00640 1 -- 172.21.15.32:0/3172930198 >> 172.21.15.32:0/3172930198 conn(0x7f15b80b1bd0 msgr2=0x7f15b8160520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:48:02.465 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:02.462+0000 7f15bee00640 1 -- 172.21.15.32:0/3172930198 shutdown_connections 2024-08-03T20:48:02.465 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:02.462+0000 7f15bee00640 1 -- 172.21.15.32:0/3172930198 wait complete. 2024-08-03T20:48:02.485 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-08-03T20:48:02.485 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-08-03T20:48:02.485 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi118.front.sepia.ceph.com 2024-08-03T20:48:02.485 INFO:tasks.cephfs.mount:self.client.name = client.3 2024-08-03T20:48:02.486 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.3 2024-08-03T20:48:02.486 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-08-03T20:48:02.486 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-08-03T20:48:02.486 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-08-03T20:48:02.486 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.3' 2024-08-03T20:48:02.486 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-08-03T20:48:02.486 DEBUG:teuthology.orchestra.run.smithi118:> ip addr 2024-08-03T20:48:02.523 INFO:teuthology.orchestra.run.smithi118.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-08-03T20:48:02.523 INFO:teuthology.orchestra.run.smithi118.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-08-03T20:48:02.523 INFO:teuthology.orchestra.run.smithi118.stdout: inet 127.0.0.1/8 scope host lo 2024-08-03T20:48:02.523 INFO:teuthology.orchestra.run.smithi118.stdout: valid_lft forever preferred_lft forever 2024-08-03T20:48:02.523 INFO:teuthology.orchestra.run.smithi118.stdout: inet6 ::1/128 scope host 2024-08-03T20:48:02.523 INFO:teuthology.orchestra.run.smithi118.stdout: valid_lft forever preferred_lft forever 2024-08-03T20:48:02.523 INFO:teuthology.orchestra.run.smithi118.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-08-03T20:48:02.523 INFO:teuthology.orchestra.run.smithi118.stdout: link/ether 0c:c4:7a:d8:a4:cc brd ff:ff:ff:ff:ff:ff 2024-08-03T20:48:02.524 INFO:teuthology.orchestra.run.smithi118.stdout: altname enp7s0f0 2024-08-03T20:48:02.524 INFO:teuthology.orchestra.run.smithi118.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-08-03T20:48:02.524 INFO:teuthology.orchestra.run.smithi118.stdout: link/ether 0c:c4:7a:d8:a4:cd brd ff:ff:ff:ff:ff:ff 2024-08-03T20:48:02.524 INFO:teuthology.orchestra.run.smithi118.stdout: altname enp7s0f1 2024-08-03T20:48:02.524 INFO:teuthology.orchestra.run.smithi118.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-08-03T20:48:02.524 INFO:teuthology.orchestra.run.smithi118.stdout: link/ether 0c:c4:7a:8f:d7:24 brd ff:ff:ff:ff:ff:ff 2024-08-03T20:48:02.524 INFO:teuthology.orchestra.run.smithi118.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-08-03T20:48:02.524 INFO:teuthology.orchestra.run.smithi118.stdout: link/ether 0c:c4:7a:8f:d7:25 brd ff:ff:ff:ff:ff:ff 2024-08-03T20:48:02.524 INFO:teuthology.orchestra.run.smithi118.stdout: inet 172.21.15.118/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-08-03T20:48:02.524 INFO:teuthology.orchestra.run.smithi118.stdout: valid_lft 42652sec preferred_lft 42652sec 2024-08-03T20:48:02.524 INFO:teuthology.orchestra.run.smithi118.stdout: inet6 fe80::ec4:7aff:fe8f:d725/64 scope link 2024-08-03T20:48:02.524 INFO:teuthology.orchestra.run.smithi118.stdout: valid_lft forever preferred_lft forever 2024-08-03T20:48:02.524 INFO:teuthology.orchestra.run.smithi118.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-08-03T20:48:02.525 INFO:teuthology.orchestra.run.smithi118.stdout: link/ether 36:40:c8:51:ce:95 brd ff:ff:ff:ff:ff:ff 2024-08-03T20:48:02.525 INFO:teuthology.orchestra.run.smithi118.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-08-03T20:48:02.525 INFO:teuthology.orchestra.run.smithi118.stdout: valid_lft forever preferred_lft forever 2024-08-03T20:48:02.525 INFO:teuthology.orchestra.run.smithi118.stdout: inet6 fe80::c04d:edff:fe67:196/64 scope link 2024-08-03T20:48:02.525 INFO:teuthology.orchestra.run.smithi118.stdout: valid_lft forever preferred_lft forever 2024-08-03T20:48:02.525 INFO:teuthology.orchestra.run.smithi118.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-08-03T20:48:02.525 INFO:teuthology.orchestra.run.smithi118.stdout: link/ether 36:40:c8:51:ce:95 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.1 2024-08-03T20:48:02.525 INFO:teuthology.orchestra.run.smithi118.stdout: inet6 fe80::3440:c8ff:fe51:ce95/64 scope link 2024-08-03T20:48:02.525 INFO:teuthology.orchestra.run.smithi118.stdout: valid_lft forever preferred_lft forever 2024-08-03T20:48:02.525 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-08-03T20:48:02.525 DEBUG:teuthology.orchestra.run.smithi118:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-08-03T20:48:02.616 INFO:teuthology.orchestra.run.smithi118.stdout:1 2024-08-03T20:48:02.620 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-08-03T20:48:02.620 DEBUG:teuthology.orchestra.run.smithi118:> route 2024-08-03T20:48:02.656 INFO:teuthology.orchestra.run.smithi118.stdout:Kernel IP routing table 2024-08-03T20:48:02.656 INFO:teuthology.orchestra.run.smithi118.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-08-03T20:48:02.656 INFO:teuthology.orchestra.run.smithi118.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-08-03T20:48:02.656 INFO:teuthology.orchestra.run.smithi118.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-08-03T20:48:02.656 INFO:teuthology.orchestra.run.smithi118.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-08-03T20:48:02.656 DEBUG:teuthology.orchestra.run.smithi118:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-08-03T20:48:02.656 DEBUG:teuthology.orchestra.run.smithi118:> set -e 2024-08-03T20:48:02.657 DEBUG:teuthology.orchestra.run.smithi118:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-08-03T20:48:02.657 DEBUG:teuthology.orchestra.run.smithi118:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-08-03T20:48:02.657 DEBUG:teuthology.orchestra.run.smithi118:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-08-03T20:48:02.657 DEBUG:teuthology.orchestra.run.smithi118:> ') 2024-08-03T20:48:02.774 INFO:teuthology.orchestra.run.smithi118.stderr:2024-08-03T20:48:02 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-08-03T20:48:02.845 INFO:teuthology.orchestra.run.smithi118.stderr:2024-08-03T20:48:02 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-08-03T20:48:02.853 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-08-03T20:48:02.853 DEBUG:teuthology.orchestra.run.smithi118:> ip netns list 2024-08-03T20:48:02.888 INFO:teuthology.orchestra.run.smithi118.stdout:ceph-ns--home-ubuntu-cephtest-mnt.1 (id: 0) 2024-08-03T20:48:02.889 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-08-03T20:48:02.889 DEBUG:teuthology.orchestra.run.smithi118:> ip netns list-id 2024-08-03T20:48:02.963 INFO:teuthology.orchestra.run.smithi118.stdout:nsid 0 (iproute2 netns name: ceph-ns--home-ubuntu-cephtest-mnt.1) 2024-08-03T20:48:02.964 DEBUG:teuthology.orchestra.run.smithi118:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-08-03T20:48:02.964 DEBUG:teuthology.orchestra.run.smithi118:> set -e 2024-08-03T20:48:02.964 DEBUG:teuthology.orchestra.run.smithi118:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.3 2024-08-03T20:48:02.964 DEBUG:teuthology.orchestra.run.smithi118:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.3 1 2024-08-03T20:48:02.964 DEBUG:teuthology.orchestra.run.smithi118:> ') 2024-08-03T20:48:03.082 INFO:teuthology.orchestra.run.smithi118.stderr:2024-08-03T20:48:03 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-08-03T20:48:03.133 INFO:teuthology.orchestra.run.smithi118.stderr:2024-08-03T20:48:03 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-08-03T20:48:03.142 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-08-03T20:48:03.142 DEBUG:teuthology.orchestra.run.smithi118:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip addr 2024-08-03T20:48:03.198 INFO:teuthology.orchestra.run.smithi118.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-08-03T20:48:03.198 INFO:teuthology.orchestra.run.smithi118.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-08-03T20:48:03.198 INFO:teuthology.orchestra.run.smithi118.stdout: inet 127.0.0.1/8 scope host lo 2024-08-03T20:48:03.198 INFO:teuthology.orchestra.run.smithi118.stdout: valid_lft forever preferred_lft forever 2024-08-03T20:48:03.198 INFO:teuthology.orchestra.run.smithi118.stdout: inet6 ::1/128 scope host 2024-08-03T20:48:03.198 INFO:teuthology.orchestra.run.smithi118.stdout: valid_lft forever preferred_lft forever 2024-08-03T20:48:03.199 INFO:teuthology.orchestra.run.smithi118.stdout:2: veth0@if7: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-08-03T20:48:03.199 INFO:teuthology.orchestra.run.smithi118.stdout: link/ether b6:1d:73:1b:bd:62 brd ff:ff:ff:ff:ff:ff link-netnsid 0 2024-08-03T20:48:03.199 INFO:teuthology.orchestra.run.smithi118.stdout: inet 192.168.0.1/16 brd 192.168.255.255 scope global veth0 2024-08-03T20:48:03.199 INFO:teuthology.orchestra.run.smithi118.stdout: valid_lft forever preferred_lft forever 2024-08-03T20:48:03.199 INFO:teuthology.orchestra.run.smithi118.stdout: inet6 fe80::b41d:73ff:fe1b:bd62/64 scope link 2024-08-03T20:48:03.199 INFO:teuthology.orchestra.run.smithi118.stdout: valid_lft forever preferred_lft forever 2024-08-03T20:48:03.201 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-08-03T20:48:03.201 DEBUG:teuthology.orchestra.run.smithi118:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip addr 2024-08-03T20:48:03.256 INFO:teuthology.orchestra.run.smithi118.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-08-03T20:48:03.256 INFO:teuthology.orchestra.run.smithi118.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-08-03T20:48:03.256 INFO:teuthology.orchestra.run.smithi118.stdout: inet 127.0.0.1/8 scope host lo 2024-08-03T20:48:03.256 INFO:teuthology.orchestra.run.smithi118.stdout: valid_lft forever preferred_lft forever 2024-08-03T20:48:03.256 INFO:teuthology.orchestra.run.smithi118.stdout: inet6 ::1/128 scope host 2024-08-03T20:48:03.256 INFO:teuthology.orchestra.run.smithi118.stdout: valid_lft forever preferred_lft forever 2024-08-03T20:48:03.256 INFO:teuthology.orchestra.run.smithi118.stdout:2: veth0@if7: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-08-03T20:48:03.256 INFO:teuthology.orchestra.run.smithi118.stdout: link/ether b6:1d:73:1b:bd:62 brd ff:ff:ff:ff:ff:ff link-netnsid 0 2024-08-03T20:48:03.257 INFO:teuthology.orchestra.run.smithi118.stdout: inet 192.168.0.1/16 brd 192.168.255.255 scope global veth0 2024-08-03T20:48:03.257 INFO:teuthology.orchestra.run.smithi118.stdout: valid_lft forever preferred_lft forever 2024-08-03T20:48:03.257 INFO:teuthology.orchestra.run.smithi118.stdout: inet6 fe80::b41d:73ff:fe1b:bd62/64 scope link 2024-08-03T20:48:03.257 INFO:teuthology.orchestra.run.smithi118.stdout: valid_lft forever preferred_lft forever 2024-08-03T20:48:03.259 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.3' with 192.168.0.2/16 2024-08-03T20:48:03.259 DEBUG:teuthology.orchestra.run.smithi118:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-08-03T20:48:03.259 DEBUG:teuthology.orchestra.run.smithi118:> set -e 2024-08-03T20:48:03.259 DEBUG:teuthology.orchestra.run.smithi118:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.3 type veth peer name brx.1 2024-08-03T20:48:03.259 DEBUG:teuthology.orchestra.run.smithi118:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.3 ip addr add 192.168.0.2/16 brd 192.168.255.255 dev veth0 2024-08-03T20:48:03.259 DEBUG:teuthology.orchestra.run.smithi118:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.3 ip link set veth0 up 2024-08-03T20:48:03.259 DEBUG:teuthology.orchestra.run.smithi118:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.3 ip link set lo up 2024-08-03T20:48:03.259 DEBUG:teuthology.orchestra.run.smithi118:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.3 ip route add default via 192.168.255.254 2024-08-03T20:48:03.259 DEBUG:teuthology.orchestra.run.smithi118:> ') 2024-08-03T20:48:03.338 INFO:teuthology.orchestra.run.smithi118.stderr:2024-08-03T20:48:03 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-08-03T20:48:03.466 INFO:teuthology.orchestra.run.smithi118.stderr:2024-08-03T20:48:03 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-08-03T20:48:03.475 DEBUG:teuthology.orchestra.run.smithi118:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-08-03T20:48:03.475 DEBUG:teuthology.orchestra.run.smithi118:> set -e 2024-08-03T20:48:03.475 DEBUG:teuthology.orchestra.run.smithi118:> sudo ip link set brx.1 up 2024-08-03T20:48:03.475 DEBUG:teuthology.orchestra.run.smithi118:> sudo ip link set dev brx.1 master ceph-brx 2024-08-03T20:48:03.475 DEBUG:teuthology.orchestra.run.smithi118:> ') 2024-08-03T20:48:03.553 INFO:teuthology.orchestra.run.smithi118.stderr:2024-08-03T20:48:03 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-08-03T20:48:03.632 INFO:teuthology.orchestra.run.smithi118.stderr:2024-08-03T20:48:03 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-08-03T20:48:03.640 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-08-03T20:48:03.641 DEBUG:teuthology.orchestra.run.smithi118:> mkdir -p -v /home/ubuntu/cephtest/mnt.3 2024-08-03T20:48:03.676 INFO:teuthology.orchestra.run.smithi118.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.3' 2024-08-03T20:48:03.676 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-08-03T20:48:03.676 DEBUG:teuthology.orchestra.run.smithi118:> chmod 0000 /home/ubuntu/cephtest/mnt.3 2024-08-03T20:48:03.751 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2024-08-03T20:48:03.751 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-08-03T20:48:03.752 DEBUG:teuthology.orchestra.run.smithi118:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.3 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=3,mds_namespace=cephfs 2024-08-03T20:48:03.946 INFO:teuthology.orchestra.run.smithi118.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=3,mds_namespace=cephfs 2024-08-03T20:48:03.946 INFO:teuthology.orchestra.run.smithi118.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=3,mds_namespace=cephfs". 2024-08-03T20:48:03.946 INFO:teuthology.orchestra.run.smithi118.stdout:invalid new device string format 2024-08-03T20:48:03.966 INFO:teuthology.orchestra.run.smithi118.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=3,mds_namespace=cephfs 2024-08-03T20:48:03.966 INFO:teuthology.orchestra.run.smithi118.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=3,mds_namespace=cephfs". 2024-08-03T20:48:03.966 INFO:teuthology.orchestra.run.smithi118.stdout:invalid new device string format 2024-08-03T20:48:03.966 INFO:teuthology.orchestra.run.smithi118.stdout:mount.ceph: resolved to: "172.21.15.32:3300,172.21.15.118:3300,172.21.15.118:3301" 2024-08-03T20:48:03.966 INFO:teuthology.orchestra.run.smithi118.stdout:mount.ceph: switching to using v1 address with old syntax 2024-08-03T20:48:03.997 INFO:teuthology.orchestra.run.smithi118.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=3,mds_namespace=cephfs 2024-08-03T20:48:03.997 INFO:teuthology.orchestra.run.smithi118.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=3,mds_namespace=cephfs". 2024-08-03T20:48:03.997 INFO:teuthology.orchestra.run.smithi118.stdout:invalid new device string format 2024-08-03T20:48:03.997 INFO:teuthology.orchestra.run.smithi118.stdout:mount.ceph: resolved to: "172.21.15.32:3300,172.21.15.118:3300,172.21.15.118:3301" 2024-08-03T20:48:03.997 INFO:teuthology.orchestra.run.smithi118.stdout:mount.ceph: switching to using v1 address with old syntax 2024-08-03T20:48:03.998 INFO:teuthology.orchestra.run.smithi118.stdout:mount.ceph: resolved to: "172.21.15.32:6789,172.21.15.118:6789,172.21.15.118:6790" 2024-08-03T20:48:03.998 INFO:teuthology.orchestra.run.smithi118.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.32:6789,172.21.15.118:6789,172.21.15.118:6790:/ 2024-08-03T20:48:03.998 INFO:teuthology.orchestra.run.smithi118.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=3,mds_namespace=cephfs,key=3,fsid=40ab141a-01d7-49b9-ae3c-2edec9876e95" will pass to kernel 2024-08-03T20:48:03.999 INFO:teuthology.orchestra.run.smithi118.stdout:mount: /home/ubuntu/cephtest/mnt.3 does not contain SELinux labels. 2024-08-03T20:48:03.999 INFO:teuthology.orchestra.run.smithi118.stdout: You just mounted a file system that supports labels which does not 2024-08-03T20:48:03.999 INFO:teuthology.orchestra.run.smithi118.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-08-03T20:48:03.999 INFO:teuthology.orchestra.run.smithi118.stdout: applications will generate AVC messages and not be allowed access to 2024-08-03T20:48:03.999 INFO:teuthology.orchestra.run.smithi118.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-08-03T20:48:04.002 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-08-03T20:48:04.002 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-08-03T20:48:04.002 DEBUG:teuthology.orchestra.run.smithi118:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.3 2024-08-03T20:48:04.058 DEBUG:teuthology.orchestra.run.smithi118:> (cd /home/ubuntu/cephtest/mnt.3 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-08-03T20:48:04.136 INFO:teuthology.orchestra.run.smithi118.stderr:2024-08-03T20:48:04 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-08-03T20:48:04.139 INFO:teuthology.orchestra.run.smithi118.stderr:2024-08-03T20:48:04 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-08-03T20:48:04.147 INFO:teuthology.orchestra.run.smithi118.stdout:client4541 2024-08-03T20:48:04.147 DEBUG:teuthology.orchestra.run.smithi118:> set -ex 2024-08-03T20:48:04.147 DEBUG:teuthology.orchestra.run.smithi118:> dd if=/proc/self/mounts of=/dev/stdout 2024-08-03T20:48:04.182 DEBUG:teuthology.orchestra.run.smithi118:> (cd /home/ubuntu/cephtest/mnt.3 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-08-03T20:48:04.300 INFO:teuthology.orchestra.run.smithi118.stderr:2024-08-03T20:48:04 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-08-03T20:48:04.303 INFO:teuthology.orchestra.run.smithi118.stderr:2024-08-03T20:48:04 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-08-03T20:48:04.314 INFO:teuthology.orchestra.run.smithi118.stdout:client4541 2024-08-03T20:48:04.314 DEBUG:teuthology.orchestra.run.smithi118:> (cd /home/ubuntu/cephtest/mnt.3 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-08-03T20:48:04.392 INFO:teuthology.orchestra.run.smithi118.stderr:2024-08-03T20:48:04 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-08-03T20:48:04.395 INFO:teuthology.orchestra.run.smithi118.stderr:2024-08-03T20:48:04 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-08-03T20:48:04.403 INFO:teuthology.orchestra.run.smithi118.stdout:client4541 2024-08-03T20:48:04.404 DEBUG:teuthology.orchestra.run.smithi118:> (cd /home/ubuntu/cephtest/mnt.3 && exec stdin-killer --timeout=300 -- bash -c 'sudo dd if=/sys/kernel/debug/ceph/40ab141a-01d7-49b9-ae3c-2edec9876e95.client4541/status') 2024-08-03T20:48:04.482 INFO:teuthology.orchestra.run.smithi118.stderr:2024-08-03T20:48:04 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-08-03T20:48:04.503 INFO:teuthology.orchestra.run.smithi118.stdout:instance: client.4541 (3)192.168.0.2:0/1092685974 2024-08-03T20:48:04.503 INFO:teuthology.orchestra.run.smithi118.stdout:blocklisted: false 2024-08-03T20:48:04.504 INFO:teuthology.orchestra.run.smithi118.stderr:0+1 records in 2024-08-03T20:48:04.504 INFO:teuthology.orchestra.run.smithi118.stderr:0+1 records out 2024-08-03T20:48:04.504 INFO:teuthology.orchestra.run.smithi118.stderr:69 bytes copied, 6.5581e-05 s, 1.1 MB/s 2024-08-03T20:48:04.506 INFO:teuthology.orchestra.run.smithi118.stderr:2024-08-03T20:48:04 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-08-03T20:48:04.514 INFO:teuthology.run_tasks:Running task cephfs_test_runner... 2024-08-03T20:48:04.573 INFO:tasks.cephfs_test_runner:Starting test: test_damaged_dentry (tasks.cephfs.test_damage.TestDamage) 2024-08-03T20:48:04.574 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-08-03T20:48:04.812 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:04.811+0000 7fb505400640 1 -- 172.21.15.32:0/1832082872 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb50015f610 msgr2=0x7fb5001523f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:48:04.812 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:04.811+0000 7fb505400640 1 --2- 172.21.15.32:0/1832082872 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb50015f610 0x7fb5001523f0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fb4ec009830 tx=0x7fb4ec02a510 comp rx=0 tx=0).stop 2024-08-03T20:48:04.812 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:04.812+0000 7fb505400640 1 -- 172.21.15.32:0/1832082872 shutdown_connections 2024-08-03T20:48:04.813 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:04.812+0000 7fb505400640 1 --2- 172.21.15.32:0/1832082872 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fb5000b4ce0 0x7fb5000cdf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:04.813 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:04.812+0000 7fb505400640 1 --2- 172.21.15.32:0/1832082872 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb50015f610 0x7fb5001523f0 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:04.813 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:04.812+0000 7fb505400640 1 -- 172.21.15.32:0/1832082872 >> 172.21.15.32:0/1832082872 conn(0x7fb5000b1bd0 msgr2=0x7fb5000b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:48:04.813 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:04.812+0000 7fb505400640 1 -- 172.21.15.32:0/1832082872 shutdown_connections 2024-08-03T20:48:04.813 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:04.812+0000 7fb505400640 1 -- 172.21.15.32:0/1832082872 wait complete. 2024-08-03T20:48:04.815 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:04.815+0000 7fb505400640 1 Processor -- start 2024-08-03T20:48:04.815 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:04.815+0000 7fb505400640 1 -- start start 2024-08-03T20:48:04.818 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:04.816+0000 7fb505400640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb5000b38b0 0x7fb5000b0d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:48:04.819 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:04.816+0000 7fb505400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb5000b4ce0 0x7fb500152cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:48:04.819 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:04.816+0000 7fb505400640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fb5000af050 0x7fb500153230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:48:04.819 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:04.816+0000 7fb505400640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fb5000cea90 con 0x7fb5000b4ce0 2024-08-03T20:48:04.819 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:04.816+0000 7fb505400640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7fb5000ce910 con 0x7fb5000b38b0 2024-08-03T20:48:04.819 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:04.816+0000 7fb505400640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7fb5000cec10 con 0x7fb5000af050 2024-08-03T20:48:04.819 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:04.816+0000 7fb4fde00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb5000b4ce0 0x7fb500152cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:48:04.819 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:04.816+0000 7fb4fde00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb5000b4ce0 0x7fb500152cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:55886/0 (socket says 172.21.15.32:55886) 2024-08-03T20:48:04.820 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:04.816+0000 7fb4fde00640 1 -- 172.21.15.32:0/3165183195 learned_addr learned my addr 172.21.15.32:0/3165183195 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:48:04.820 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:04.817+0000 7fb4ff200640 1 --2- 172.21.15.32:0/3165183195 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fb5000af050 0x7fb500153230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:48:04.820 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:04.817+0000 7fb4fe800640 1 --2- 172.21.15.32:0/3165183195 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb5000b38b0 0x7fb5000b0d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:48:04.820 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:04.817+0000 7fb4fde00640 1 -- 172.21.15.32:0/3165183195 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fb5000af050 msgr2=0x7fb500153230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:48:04.820 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:04.817+0000 7fb4fde00640 1 --2- 172.21.15.32:0/3165183195 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fb5000af050 0x7fb500153230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:04.820 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:04.817+0000 7fb4fde00640 1 -- 172.21.15.32:0/3165183195 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb5000b38b0 msgr2=0x7fb5000b0d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:48:04.820 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:04.817+0000 7fb4fde00640 1 --2- 172.21.15.32:0/3165183195 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb5000b38b0 0x7fb5000b0d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:04.820 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:04.817+0000 7fb4fde00640 1 -- 172.21.15.32:0/3165183195 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb500154da0 con 0x7fb5000b4ce0 2024-08-03T20:48:04.820 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:04.817+0000 7fb4ff200640 1 --2- 172.21.15.32:0/3165183195 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fb5000af050 0x7fb500153230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-03T20:48:04.820 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:04.817+0000 7fb4fe800640 1 --2- 172.21.15.32:0/3165183195 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb5000b38b0 0x7fb5000b0d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-03T20:48:04.821 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:04.817+0000 7fb4fde00640 1 --2- 172.21.15.32:0/3165183195 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb5000b4ce0 0x7fb500152cf0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fb4ec009950 tx=0x7fb4ec032c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:48:04.821 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:04.818+0000 7fb4f6a00640 1 -- 172.21.15.32:0/3165183195 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4ec040070 con 0x7fb5000b4ce0 2024-08-03T20:48:04.821 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:04.818+0000 7fb4f6a00640 1 -- 172.21.15.32:0/3165183195 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fb4ec005310 con 0x7fb5000b4ce0 2024-08-03T20:48:04.821 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:04.818+0000 7fb505400640 1 -- 172.21.15.32:0/3165183195 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb500155080 con 0x7fb5000b4ce0 2024-08-03T20:48:04.821 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:04.818+0000 7fb4f6a00640 1 -- 172.21.15.32:0/3165183195 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4ec005c90 con 0x7fb5000b4ce0 2024-08-03T20:48:04.821 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:04.818+0000 7fb505400640 1 -- 172.21.15.32:0/3165183195 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb5001f5480 con 0x7fb5000b4ce0 2024-08-03T20:48:04.822 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:04.819+0000 7fb505400640 1 -- 172.21.15.32:0/3165183195 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb500159140 con 0x7fb5000b4ce0 2024-08-03T20:48:04.823 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:04.820+0000 7fb4f6a00640 1 -- 172.21.15.32:0/3165183195 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7fb4ec039040 con 0x7fb5000b4ce0 2024-08-03T20:48:04.823 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:04.821+0000 7fb4f6a00640 1 --2- 172.21.15.32:0/3165183195 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fb4e0077040 0x7fb4e0079550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:48:04.823 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:04.821+0000 7fb4f6a00640 1 -- 172.21.15.32:0/3165183195 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7fb4ec0b97e0 con 0x7fb5000b4ce0 2024-08-03T20:48:04.823 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:04.822+0000 7fb4fe800640 1 --2- 172.21.15.32:0/3165183195 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fb4e0077040 0x7fb4e0079550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:48:04.823 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:04.822+0000 7fb4fe800640 1 --2- 172.21.15.32:0/3165183195 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fb4e0077040 0x7fb4e0079550 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fb4f0007f20 tx=0x7fb4f0007eb0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:48:04.826 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:04.826+0000 7fb4f6a00640 1 -- 172.21.15.32:0/3165183195 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7fb4ec082a90 con 0x7fb5000b4ce0 2024-08-03T20:48:05.033 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:05.032+0000 7fb505400640 1 -- 172.21.15.32:0/3165183195 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fb5000cf130 con 0x7fb5000b4ce0 2024-08-03T20:48:05.034 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:05.033+0000 7fb4f6a00640 1 -- 172.21.15.32:0/3165183195 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) ==== 74+0+16545 (secure 0 0 0) 0x7fb4ec082390 con 0x7fb5000b4ce0 2024-08-03T20:48:05.034 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-03T20:48:05.034 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":21,"fsid":"40ab141a-01d7-49b9-ae3c-2edec9876e95","created":"2024-08-03T20:47:14.957362+0000","modified":"2024-08-03T20:47:36.927196+0000","last_up_change":"2024-08-03T20:47:28.874451+0000","last_in_change":"2024-08-03T20:47:20.079442+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-08-03T20:47:28.261300+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-08-03T20:47:33.085381+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-08-03T20:47:34.405408+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5a83e759-9035-438b-889a-0a6af1f0a0d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6816","nonce":3066156116},{"type":"v1","addr":"172.21.15.32:6817","nonce":3066156116}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6818","nonce":3066156116},{"type":"v1","addr":"172.21.15.32:6819","nonce":3066156116}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6822","nonce":3066156116},{"type":"v1","addr":"172.21.15.32:6823","nonce":3066156116}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6820","nonce":3066156116},{"type":"v1","addr":"172.21.15.32:6821","nonce":3066156116}]},"public_addr":"172.21.15.32:6817/3066156116","cluster_addr":"172.21.15.32:6819/3066156116","heartbeat_back_addr":"172.21.15.32:6823/3066156116","heartbeat_front_addr":"172.21.15.32:6821/3066156116","state":["exists","up"]},{"osd":1,"uuid":"d7c5c210-da4f-49f9-b9e7-1dfb02daa4bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6824","nonce":1551241759},{"type":"v1","addr":"172.21.15.32:6825","nonce":1551241759}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6826","nonce":1551241759},{"type":"v1","addr":"172.21.15.32:6827","nonce":1551241759}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6830","nonce":1551241759},{"type":"v1","addr":"172.21.15.32:6831","nonce":1551241759}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6828","nonce":1551241759},{"type":"v1","addr":"172.21.15.32:6829","nonce":1551241759}]},"public_addr":"172.21.15.32:6825/1551241759","cluster_addr":"172.21.15.32:6827/1551241759","heartbeat_back_addr":"172.21.15.32:6831/1551241759","heartbeat_front_addr":"172.21.15.32:6829/1551241759","state":["exists","up"]},{"osd":2,"uuid":"57f2e6de-a201-45c2-8981-8e56a4875ebd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6808","nonce":3806708944},{"type":"v1","addr":"172.21.15.32:6809","nonce":3806708944}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6810","nonce":3806708944},{"type":"v1","addr":"172.21.15.32:6811","nonce":3806708944}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6814","nonce":3806708944},{"type":"v1","addr":"172.21.15.32:6815","nonce":3806708944}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6812","nonce":3806708944},{"type":"v1","addr":"172.21.15.32:6813","nonce":3806708944}]},"public_addr":"172.21.15.32:6809/3806708944","cluster_addr":"172.21.15.32:6811/3806708944","heartbeat_back_addr":"172.21.15.32:6815/3806708944","heartbeat_front_addr":"172.21.15.32:6813/3806708944","state":["exists","up"]},{"osd":3,"uuid":"6439a280-da45-4148-8130-f16323935ff3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6800","nonce":9088303},{"type":"v1","addr":"172.21.15.32:6801","nonce":9088303}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6802","nonce":9088303},{"type":"v1","addr":"172.21.15.32:6803","nonce":9088303}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6806","nonce":9088303},{"type":"v1","addr":"172.21.15.32:6807","nonce":9088303}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6804","nonce":9088303},{"type":"v1","addr":"172.21.15.32:6805","nonce":9088303}]},"public_addr":"172.21.15.32:6801/9088303","cluster_addr":"172.21.15.32:6803/9088303","heartbeat_back_addr":"172.21.15.32:6807/9088303","heartbeat_front_addr":"172.21.15.32:6805/9088303","state":["exists","up"]},{"osd":4,"uuid":"4f902eff-1e18-418d-82c9-2151c69524fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6802","nonce":864752275},{"type":"v1","addr":"172.21.15.118:6803","nonce":864752275}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6804","nonce":864752275},{"type":"v1","addr":"172.21.15.118:6805","nonce":864752275}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6808","nonce":864752275},{"type":"v1","addr":"172.21.15.118:6809","nonce":864752275}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6806","nonce":864752275},{"type":"v1","addr":"172.21.15.118:6807","nonce":864752275}]},"public_addr":"172.21.15.118:6803/864752275","cluster_addr":"172.21.15.118:6805/864752275","heartbeat_back_addr":"172.21.15.118:6809/864752275","heartbeat_front_addr":"172.21.15.118:6807/864752275","state":["exists","up"]},{"osd":5,"uuid":"bfd9feac-dd06-458c-91b9-af7f175f612c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6810","nonce":2724891770},{"type":"v1","addr":"172.21.15.118:6811","nonce":2724891770}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6812","nonce":2724891770},{"type":"v1","addr":"172.21.15.118:6813","nonce":2724891770}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6816","nonce":2724891770},{"type":"v1","addr":"172.21.15.118:6817","nonce":2724891770}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6814","nonce":2724891770},{"type":"v1","addr":"172.21.15.118:6815","nonce":2724891770}]},"public_addr":"172.21.15.118:6811/2724891770","cluster_addr":"172.21.15.118:6813/2724891770","heartbeat_back_addr":"172.21.15.118:6817/2724891770","heartbeat_front_addr":"172.21.15.118:6815/2724891770","state":["exists","up"]},{"osd":6,"uuid":"5ed5a279-9d8e-416d-bbc7-7edd3e6d5705","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6825","nonce":801947386},{"type":"v1","addr":"172.21.15.118:6827","nonce":801947386}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6828","nonce":801947386},{"type":"v1","addr":"172.21.15.118:6829","nonce":801947386}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6832","nonce":801947386},{"type":"v1","addr":"172.21.15.118:6833","nonce":801947386}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6830","nonce":801947386},{"type":"v1","addr":"172.21.15.118:6831","nonce":801947386}]},"public_addr":"172.21.15.118:6827/801947386","cluster_addr":"172.21.15.118:6829/801947386","heartbeat_back_addr":"172.21.15.118:6833/801947386","heartbeat_front_addr":"172.21.15.118:6831/801947386","state":["exists","up"]},{"osd":7,"uuid":"f2086c47-875e-4451-a436-2008d1ea16af","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6818","nonce":653000048},{"type":"v1","addr":"172.21.15.118:6819","nonce":653000048}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6820","nonce":653000048},{"type":"v1","addr":"172.21.15.118:6821","nonce":653000048}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6824","nonce":653000048},{"type":"v1","addr":"172.21.15.118:6826","nonce":653000048}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6822","nonce":653000048},{"type":"v1","addr":"172.21.15.118:6823","nonce":653000048}]},"public_addr":"172.21.15.118:6819/653000048","cluster_addr":"172.21.15.118:6821/653000048","heartbeat_back_addr":"172.21.15.118:6826/653000048","heartbeat_front_addr":"172.21.15.118:6823/653000048","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-03T20:47:25.491994+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-03T20:47:25.587358+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-03T20:47:25.454175+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-03T20:47:25.377734+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-03T20:47:25.412351+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-03T20:47:25.499815+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-03T20:47:25.575044+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-03T20:47:25.552383+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-08-03T20:48:05.037 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:05.036+0000 7fb505400640 1 -- 172.21.15.32:0/3165183195 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fb4e0077040 msgr2=0x7fb4e0079550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:48:05.037 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:05.036+0000 7fb505400640 1 --2- 172.21.15.32:0/3165183195 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fb4e0077040 0x7fb4e0079550 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fb4f0007f20 tx=0x7fb4f0007eb0 comp rx=0 tx=0).stop 2024-08-03T20:48:05.037 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:05.036+0000 7fb505400640 1 -- 172.21.15.32:0/3165183195 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb5000b4ce0 msgr2=0x7fb500152cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:48:05.037 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:05.036+0000 7fb505400640 1 --2- 172.21.15.32:0/3165183195 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb5000b4ce0 0x7fb500152cf0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fb4ec009950 tx=0x7fb4ec032c10 comp rx=0 tx=0).stop 2024-08-03T20:48:05.037 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:05.036+0000 7fb505400640 1 -- 172.21.15.32:0/3165183195 shutdown_connections 2024-08-03T20:48:05.037 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:05.036+0000 7fb505400640 1 --2- 172.21.15.32:0/3165183195 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fb4e0077040 0x7fb4e0079550 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:05.037 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:05.036+0000 7fb505400640 1 --2- 172.21.15.32:0/3165183195 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fb5000af050 0x7fb500153230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:05.037 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:05.036+0000 7fb505400640 1 --2- 172.21.15.32:0/3165183195 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb5000b4ce0 0x7fb500152cf0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:05.037 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:05.036+0000 7fb505400640 1 --2- 172.21.15.32:0/3165183195 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb5000b38b0 0x7fb5000b0d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:05.037 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:05.036+0000 7fb505400640 1 -- 172.21.15.32:0/3165183195 >> 172.21.15.32:0/3165183195 conn(0x7fb5000b1bd0 msgr2=0x7fb5001521a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:48:05.038 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:05.036+0000 7fb505400640 1 -- 172.21.15.32:0/3165183195 shutdown_connections 2024-08-03T20:48:05.038 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:05.036+0000 7fb505400640 1 -- 172.21.15.32:0/3165183195 wait complete. 2024-08-03T20:48:05.062 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-08-03T20:47:28.261300+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-08-03T20:47:33.085381+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-08-03T20:47:34.405408+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-08-03T20:48:05.062 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-08-03T20:48:05.295 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:05.294+0000 7ff07fe00640 1 -- 172.21.15.32:0/3192043688 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff060004d40 msgr2=0x7ff060005190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:48:05.295 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:05.294+0000 7ff07fe00640 1 --2- 172.21.15.32:0/3192043688 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff060004d40 0x7ff060005190 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7ff0780ccab0 tx=0x7ff060005830 comp rx=0 tx=0).stop 2024-08-03T20:48:05.295 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:05.294+0000 7ff07de00640 1 -- 172.21.15.32:0/3192043688 reap_dead start 2024-08-03T20:48:05.295 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:05.294+0000 7ff07fe00640 1 -- 172.21.15.32:0/3192043688 shutdown_connections 2024-08-03T20:48:05.295 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:05.294+0000 7ff07fe00640 1 -- 172.21.15.32:0/3192043688 >> 172.21.15.32:0/3192043688 conn(0x7ff0780b1bd0 msgr2=0x7ff0780b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:48:05.295 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:05.294+0000 7ff07fe00640 1 -- 172.21.15.32:0/3192043688 shutdown_connections 2024-08-03T20:48:05.295 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:05.294+0000 7ff07fe00640 1 -- 172.21.15.32:0/3192043688 wait complete. 2024-08-03T20:48:05.296 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:05.296+0000 7ff07fe00640 1 Processor -- start 2024-08-03T20:48:05.297 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:05.296+0000 7ff07fe00640 1 -- start start 2024-08-03T20:48:05.299 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:05.298+0000 7ff07fe00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff060004d40 0x7ff07813c060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:48:05.300 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:05.298+0000 7ff07fe00640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7ff0780c58e0 0x7ff078073840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:48:05.300 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:05.298+0000 7ff07fe00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ff078073d80 0x7ff0781f8850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:48:05.300 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:05.298+0000 7ff07fe00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7ff0780b3ce0 con 0x7ff060004d40 2024-08-03T20:48:05.300 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:05.298+0000 7ff07fe00640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7ff0780cf730 con 0x7ff078073d80 2024-08-03T20:48:05.300 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:05.298+0000 7ff07fe00640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7ff0780b3e60 con 0x7ff0780c58e0 2024-08-03T20:48:05.300 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:05.298+0000 7ff07d400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff060004d40 0x7ff07813c060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:48:05.300 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:05.298+0000 7ff07d400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff060004d40 0x7ff07813c060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:55896/0 (socket says 172.21.15.32:55896) 2024-08-03T20:48:05.300 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:05.298+0000 7ff07d400640 1 -- 172.21.15.32:0/2190870506 learned_addr learned my addr 172.21.15.32:0/2190870506 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:48:05.300 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:05.298+0000 7ff07ca00640 1 --2- 172.21.15.32:0/2190870506 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7ff0780c58e0 0x7ff078073840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:48:05.300 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:05.298+0000 7ff07de00640 1 --2- 172.21.15.32:0/2190870506 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ff078073d80 0x7ff0781f8850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:48:05.300 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:05.298+0000 7ff07d400640 1 -- 172.21.15.32:0/2190870506 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7ff0780c58e0 msgr2=0x7ff078073840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:48:05.301 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:05.298+0000 7ff07d400640 1 --2- 172.21.15.32:0/2190870506 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7ff0780c58e0 0x7ff078073840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:05.301 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:05.298+0000 7ff07d400640 1 -- 172.21.15.32:0/2190870506 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ff078073d80 msgr2=0x7ff0781f8850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:48:05.301 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:05.298+0000 7ff07d400640 1 --2- 172.21.15.32:0/2190870506 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ff078073d80 0x7ff0781f8850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:05.301 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:05.298+0000 7ff07d400640 1 -- 172.21.15.32:0/2190870506 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff078074190 con 0x7ff060004d40 2024-08-03T20:48:05.301 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:05.298+0000 7ff07de00640 1 --2- 172.21.15.32:0/2190870506 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ff078073d80 0x7ff0781f8850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-03T20:48:05.301 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:05.299+0000 7ff07d400640 1 --2- 172.21.15.32:0/2190870506 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff060004d40 0x7ff07813c060 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7ff06c003f50 tx=0x7ff06c0017d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:48:05.301 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:05.299+0000 7ff075600640 1 -- 172.21.15.32:0/2190870506 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff06c015070 con 0x7ff060004d40 2024-08-03T20:48:05.301 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:05.299+0000 7ff075600640 1 -- 172.21.15.32:0/2190870506 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7ff06c011900 con 0x7ff060004d40 2024-08-03T20:48:05.301 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:05.299+0000 7ff07fe00640 1 -- 172.21.15.32:0/2190870506 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff07813c5a0 con 0x7ff060004d40 2024-08-03T20:48:05.301 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:05.299+0000 7ff075600640 1 -- 172.21.15.32:0/2190870506 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff06c011d10 con 0x7ff060004d40 2024-08-03T20:48:05.301 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:05.299+0000 7ff07fe00640 1 -- 172.21.15.32:0/2190870506 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff0780cb950 con 0x7ff060004d40 2024-08-03T20:48:05.302 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:05.300+0000 7ff07fe00640 1 -- 172.21.15.32:0/2190870506 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff07815b680 con 0x7ff060004d40 2024-08-03T20:48:05.302 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:05.302+0000 7ff075600640 1 -- 172.21.15.32:0/2190870506 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7ff06c018070 con 0x7ff060004d40 2024-08-03T20:48:05.303 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:05.303+0000 7ff075600640 1 --2- 172.21.15.32:0/2190870506 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7ff058077040 0x7ff058079550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:48:05.303 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:05.303+0000 7ff075600640 1 -- 172.21.15.32:0/2190870506 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7ff06c099670 con 0x7ff060004d40 2024-08-03T20:48:05.304 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:05.303+0000 7ff07ca00640 1 --2- 172.21.15.32:0/2190870506 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7ff058077040 0x7ff058079550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:48:05.304 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:05.304+0000 7ff07ca00640 1 --2- 172.21.15.32:0/2190870506 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7ff058077040 0x7ff058079550 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7ff0780cc340 tx=0x7ff0640067a0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:48:05.307 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:05.307+0000 7ff075600640 1 -- 172.21.15.32:0/2190870506 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7ff07815b680 con 0x7ff060004d40 2024-08-03T20:48:05.512 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:05.512+0000 7ff07fe00640 1 -- 172.21.15.32:0/2190870506 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7ff07806a3f0 con 0x7ff060004d40 2024-08-03T20:48:05.520 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:05.513+0000 7ff075600640 1 -- 172.21.15.32:0/2190870506 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v21) ==== 93+0+10 (secure 0 0 0) 0x7ff07806a3f0 con 0x7ff060004d40 2024-08-03T20:48:05.521 INFO:teuthology.orchestra.run.smithi032.stdout:pg_num: 1 2024-08-03T20:48:05.522 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:05.515+0000 7ff07fe00640 1 -- 172.21.15.32:0/2190870506 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7ff058077040 msgr2=0x7ff058079550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:48:05.522 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:05.515+0000 7ff07fe00640 1 --2- 172.21.15.32:0/2190870506 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7ff058077040 0x7ff058079550 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7ff0780cc340 tx=0x7ff0640067a0 comp rx=0 tx=0).stop 2024-08-03T20:48:05.522 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:05.515+0000 7ff07fe00640 1 -- 172.21.15.32:0/2190870506 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff060004d40 msgr2=0x7ff07813c060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:48:05.522 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:05.515+0000 7ff07fe00640 1 --2- 172.21.15.32:0/2190870506 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff060004d40 0x7ff07813c060 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7ff06c003f50 tx=0x7ff06c0017d0 comp rx=0 tx=0).stop 2024-08-03T20:48:05.522 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:05.515+0000 7ff07fe00640 1 -- 172.21.15.32:0/2190870506 shutdown_connections 2024-08-03T20:48:05.522 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:05.515+0000 7ff07fe00640 1 --2- 172.21.15.32:0/2190870506 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7ff058077040 0x7ff058079550 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:05.523 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:05.515+0000 7ff07fe00640 1 --2- 172.21.15.32:0/2190870506 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ff078073d80 0x7ff0781f8850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:05.523 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:05.515+0000 7ff07fe00640 1 --2- 172.21.15.32:0/2190870506 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7ff0780c58e0 0x7ff078073840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:05.523 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:05.515+0000 7ff07fe00640 1 --2- 172.21.15.32:0/2190870506 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff060004d40 0x7ff07813c060 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:05.523 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:05.515+0000 7ff07fe00640 1 -- 172.21.15.32:0/2190870506 >> 172.21.15.32:0/2190870506 conn(0x7ff0780b1bd0 msgr2=0x7ff07806e950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:48:05.523 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:05.515+0000 7ff07fe00640 1 -- 172.21.15.32:0/2190870506 shutdown_connections 2024-08-03T20:48:05.523 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:05.515+0000 7ff07fe00640 1 -- 172.21.15.32:0/2190870506 wait complete. 2024-08-03T20:48:05.538 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-08-03T20:48:05.771 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:05.770+0000 7facb0a00640 1 -- 172.21.15.32:0/3292654472 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fac90004d40 msgr2=0x7fac900051f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:48:05.771 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:05.770+0000 7facb0a00640 1 --2- 172.21.15.32:0/3292654472 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fac90004d40 0x7fac900051f0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7facac15fec0 tx=0x7fac90005830 comp rx=0 tx=0).stop 2024-08-03T20:48:05.771 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:05.770+0000 7facaa800640 1 -- 172.21.15.32:0/3292654472 reap_dead start 2024-08-03T20:48:05.771 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:05.770+0000 7facb0a00640 1 -- 172.21.15.32:0/3292654472 shutdown_connections 2024-08-03T20:48:05.771 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:05.770+0000 7facb0a00640 1 -- 172.21.15.32:0/3292654472 >> 172.21.15.32:0/3292654472 conn(0x7facac0b1bd0 msgr2=0x7facac0b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:48:05.771 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:05.770+0000 7facb0a00640 1 -- 172.21.15.32:0/3292654472 shutdown_connections 2024-08-03T20:48:05.771 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:05.770+0000 7facb0a00640 1 -- 172.21.15.32:0/3292654472 wait complete. 2024-08-03T20:48:05.772 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:05.772+0000 7facb0a00640 1 Processor -- start 2024-08-03T20:48:05.772 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:05.772+0000 7facb0a00640 1 -- start start 2024-08-03T20:48:05.775 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:05.774+0000 7facb0a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fac90004d40 0x7facac0cbd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:48:05.776 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:05.774+0000 7facb0a00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7facac149a00 0x7facac157430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:48:05.776 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:05.774+0000 7facb0a00640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7facac157970 0x7facac157d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:48:05.776 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:05.774+0000 7facb0a00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7facac0cd510 con 0x7fac90004d40 2024-08-03T20:48:05.776 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:05.774+0000 7facb0a00640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7facac0cd390 con 0x7facac149a00 2024-08-03T20:48:05.776 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:05.774+0000 7facb0a00640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7facac0ced30 con 0x7facac157970 2024-08-03T20:48:05.776 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:05.774+0000 7faca9e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fac90004d40 0x7facac0cbd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:48:05.776 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:05.774+0000 7faca9e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fac90004d40 0x7facac0cbd70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:55920/0 (socket says 172.21.15.32:55920) 2024-08-03T20:48:05.776 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:05.774+0000 7faca9e00640 1 -- 172.21.15.32:0/3610492038 learned_addr learned my addr 172.21.15.32:0/3610492038 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:48:05.776 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:05.774+0000 7facaa800640 1 --2- 172.21.15.32:0/3610492038 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7facac157970 0x7facac157d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:48:05.776 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:05.774+0000 7faca3e00640 1 --2- 172.21.15.32:0/3610492038 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7facac149a00 0x7facac157430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:48:05.776 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:05.774+0000 7faca9e00640 1 -- 172.21.15.32:0/3610492038 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7facac157970 msgr2=0x7facac157d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:48:05.776 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:05.774+0000 7faca9e00640 1 --2- 172.21.15.32:0/3610492038 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7facac157970 0x7facac157d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:05.776 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:05.774+0000 7faca9e00640 1 -- 172.21.15.32:0/3610492038 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7facac149a00 msgr2=0x7facac157430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:48:05.776 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:05.774+0000 7faca9e00640 1 --2- 172.21.15.32:0/3610492038 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7facac149a00 0x7facac157430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:05.777 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:05.774+0000 7faca9e00640 1 -- 172.21.15.32:0/3610492038 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7facac1582c0 con 0x7fac90004d40 2024-08-03T20:48:05.777 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:05.774+0000 7facaa800640 1 --2- 172.21.15.32:0/3610492038 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7facac157970 0x7facac157d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-03T20:48:05.777 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:05.774+0000 7faca3e00640 1 --2- 172.21.15.32:0/3610492038 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7facac149a00 0x7facac157430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-03T20:48:05.777 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:05.775+0000 7faca9e00640 1 --2- 172.21.15.32:0/3610492038 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fac90004d40 0x7facac0cbd70 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fac9c011ae0 tx=0x7fac9c009e90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:48:05.777 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:05.775+0000 7faca2000640 1 -- 172.21.15.32:0/3610492038 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac9c01e070 con 0x7fac90004d40 2024-08-03T20:48:05.777 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:05.775+0000 7faca2000640 1 -- 172.21.15.32:0/3610492038 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fac9c00fe70 con 0x7fac90004d40 2024-08-03T20:48:05.777 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:05.775+0000 7faca2000640 1 -- 172.21.15.32:0/3610492038 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac9c004290 con 0x7fac90004d40 2024-08-03T20:48:05.777 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:05.775+0000 7facb0a00640 1 -- 172.21.15.32:0/3610492038 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7facac0ca470 con 0x7fac90004d40 2024-08-03T20:48:05.777 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:05.775+0000 7facb0a00640 1 -- 172.21.15.32:0/3610492038 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7facac0caa30 con 0x7fac90004d40 2024-08-03T20:48:05.778 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:05.776+0000 7facb0a00640 1 -- 172.21.15.32:0/3610492038 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7facac0cf2c0 con 0x7fac90004d40 2024-08-03T20:48:05.778 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:05.777+0000 7faca2000640 1 -- 172.21.15.32:0/3610492038 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7fac9c021070 con 0x7fac90004d40 2024-08-03T20:48:05.779 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:05.779+0000 7faca2000640 1 --2- 172.21.15.32:0/3610492038 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fac88077010 0x7fac88079520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:48:05.779 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:05.779+0000 7faca2000640 1 -- 172.21.15.32:0/3610492038 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7fac9c099390 con 0x7fac90004d40 2024-08-03T20:48:05.780 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:05.779+0000 7faca3e00640 1 --2- 172.21.15.32:0/3610492038 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fac88077010 0x7fac88079520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:48:05.780 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:05.780+0000 7faca3e00640 1 --2- 172.21.15.32:0/3610492038 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fac88077010 0x7fac88079520 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fac94001260 tx=0x7fac940067a0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:48:05.784 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:05.783+0000 7faca2000640 1 -- 172.21.15.32:0/3610492038 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7fac9c061f60 con 0x7fac90004d40 2024-08-03T20:48:05.988 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:05.988+0000 7facb0a00640 1 -- 172.21.15.32:0/3610492038 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7facac0b33c0 con 0x7fac90004d40 2024-08-03T20:48:05.989 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:05.988+0000 7faca2000640 1 -- 172.21.15.32:0/3610492038 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v21) ==== 104+0+11 (secure 0 0 0) 0x7fac9c0664b0 con 0x7fac90004d40 2024-08-03T20:48:05.989 INFO:teuthology.orchestra.run.smithi032.stdout:pg_num: 32 2024-08-03T20:48:05.991 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:05.991+0000 7facb0a00640 1 -- 172.21.15.32:0/3610492038 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fac88077010 msgr2=0x7fac88079520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:48:05.991 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:05.991+0000 7facb0a00640 1 --2- 172.21.15.32:0/3610492038 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fac88077010 0x7fac88079520 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fac94001260 tx=0x7fac940067a0 comp rx=0 tx=0).stop 2024-08-03T20:48:05.991 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:05.991+0000 7facb0a00640 1 -- 172.21.15.32:0/3610492038 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fac90004d40 msgr2=0x7facac0cbd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:48:05.991 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:05.991+0000 7facb0a00640 1 --2- 172.21.15.32:0/3610492038 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fac90004d40 0x7facac0cbd70 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fac9c011ae0 tx=0x7fac9c009e90 comp rx=0 tx=0).stop 2024-08-03T20:48:05.992 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:05.991+0000 7facb0a00640 1 -- 172.21.15.32:0/3610492038 shutdown_connections 2024-08-03T20:48:05.992 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:05.991+0000 7facb0a00640 1 --2- 172.21.15.32:0/3610492038 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fac88077010 0x7fac88079520 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:05.992 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:05.991+0000 7facb0a00640 1 --2- 172.21.15.32:0/3610492038 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7facac149a00 0x7facac157430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:05.992 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:05.991+0000 7facb0a00640 1 --2- 172.21.15.32:0/3610492038 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7facac157970 0x7facac157d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:05.992 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:05.991+0000 7facb0a00640 1 --2- 172.21.15.32:0/3610492038 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fac90004d40 0x7facac0cbd70 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:05.992 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:05.991+0000 7facb0a00640 1 -- 172.21.15.32:0/3610492038 >> 172.21.15.32:0/3610492038 conn(0x7facac0b1bd0 msgr2=0x7facac06fd30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:48:05.992 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:05.991+0000 7facb0a00640 1 -- 172.21.15.32:0/3610492038 shutdown_connections 2024-08-03T20:48:05.992 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:05.991+0000 7facb0a00640 1 -- 172.21.15.32:0/3610492038 wait complete. 2024-08-03T20:48:06.016 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-08-03T20:48:06.250 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:06.249+0000 7fa4ca400640 1 -- 172.21.15.32:0/3806558869 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa4ac005350 msgr2=0x7fa4ac00b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:48:06.250 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:06.249+0000 7fa4ca400640 1 --2- 172.21.15.32:0/3806558869 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa4ac005350 0x7fa4ac00b9c0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fa4bc00c630 tx=0x7fa4ac005830 comp rx=0 tx=0).stop 2024-08-03T20:48:06.250 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:06.249+0000 7fa4ca400640 1 -- 172.21.15.32:0/3806558869 shutdown_connections 2024-08-03T20:48:06.250 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:06.249+0000 7fa4ca400640 1 --2- 172.21.15.32:0/3806558869 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa4ac005350 0x7fa4ac00b9c0 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:06.250 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:06.249+0000 7fa4ca400640 1 -- 172.21.15.32:0/3806558869 >> 172.21.15.32:0/3806558869 conn(0x7fa4c40d1eb0 msgr2=0x7fa4c40d22c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:48:06.250 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:06.249+0000 7fa4ca400640 1 -- 172.21.15.32:0/3806558869 shutdown_connections 2024-08-03T20:48:06.250 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:06.249+0000 7fa4ca400640 1 -- 172.21.15.32:0/3806558869 wait complete. 2024-08-03T20:48:06.252 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:06.251+0000 7fa4ca400640 1 Processor -- start 2024-08-03T20:48:06.252 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:06.252+0000 7fa4ca400640 1 -- start start 2024-08-03T20:48:06.254 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:06.253+0000 7fa4ca400640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa4ac005350 0x7fa4c4159570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:48:06.254 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:06.253+0000 7fa4ca400640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fa4c415d210 0x7fa4c4158050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:48:06.254 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:06.253+0000 7fa4ca400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa4c4158590 0x7fa4c41589a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:48:06.254 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:06.253+0000 7fa4ca400640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fa4c40ce6e0 con 0x7fa4c4158590 2024-08-03T20:48:06.254 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:06.253+0000 7fa4ca400640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7fa4c40b13c0 con 0x7fa4ac005350 2024-08-03T20:48:06.254 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:06.253+0000 7fa4ca400640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7fa4c40ce860 con 0x7fa4c415d210 2024-08-03T20:48:06.254 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:06.253+0000 7fa4c9a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa4c4158590 0x7fa4c41589a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:48:06.254 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:06.253+0000 7fa4c9a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa4c4158590 0x7fa4c41589a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:55946/0 (socket says 172.21.15.32:55946) 2024-08-03T20:48:06.254 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:06.253+0000 7fa4c9a00640 1 -- 172.21.15.32:0/3527639164 learned_addr learned my addr 172.21.15.32:0/3527639164 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:48:06.255 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:06.253+0000 7fa4c9000640 1 --2- 172.21.15.32:0/3527639164 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa4ac005350 0x7fa4c4159570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:48:06.255 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:06.254+0000 7fa4c3e00640 1 --2- 172.21.15.32:0/3527639164 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fa4c415d210 0x7fa4c4158050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:48:06.255 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:06.254+0000 7fa4c9a00640 1 -- 172.21.15.32:0/3527639164 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fa4c415d210 msgr2=0x7fa4c4158050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:48:06.256 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:06.254+0000 7fa4c9a00640 1 --2- 172.21.15.32:0/3527639164 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fa4c415d210 0x7fa4c4158050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:06.256 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:06.254+0000 7fa4c9a00640 1 -- 172.21.15.32:0/3527639164 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa4ac005350 msgr2=0x7fa4c4159570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:48:06.256 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:06.254+0000 7fa4c9a00640 1 --2- 172.21.15.32:0/3527639164 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa4ac005350 0x7fa4c4159570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:06.256 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:06.254+0000 7fa4c9a00640 1 -- 172.21.15.32:0/3527639164 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa4c415aa30 con 0x7fa4c4158590 2024-08-03T20:48:06.256 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:06.254+0000 7fa4c9a00640 1 --2- 172.21.15.32:0/3527639164 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa4c4158590 0x7fa4c41589a0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fa4bc02b4e0 tx=0x7fa4bc003ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:48:06.256 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:06.254+0000 7fa4c0c00640 1 -- 172.21.15.32:0/3527639164 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4bc039070 con 0x7fa4c4158590 2024-08-03T20:48:06.256 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:06.254+0000 7fa4c0c00640 1 -- 172.21.15.32:0/3527639164 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fa4bc02bb00 con 0x7fa4c4158590 2024-08-03T20:48:06.256 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:06.255+0000 7fa4ca400640 1 -- 172.21.15.32:0/3527639164 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa4c415ad10 con 0x7fa4c4158590 2024-08-03T20:48:06.256 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:06.255+0000 7fa4c0c00640 1 -- 172.21.15.32:0/3527639164 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4bc002090 con 0x7fa4c4158590 2024-08-03T20:48:06.256 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:06.255+0000 7fa4ca400640 1 -- 172.21.15.32:0/3527639164 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa4c415b2d0 con 0x7fa4c4158590 2024-08-03T20:48:06.257 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:06.256+0000 7fa4ca400640 1 -- 172.21.15.32:0/3527639164 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa4c40cd5d0 con 0x7fa4c4158590 2024-08-03T20:48:06.257 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:06.257+0000 7fa4c0c00640 1 -- 172.21.15.32:0/3527639164 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7fa4bc03c020 con 0x7fa4c4158590 2024-08-03T20:48:06.258 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:06.258+0000 7fa4c0c00640 1 --2- 172.21.15.32:0/3527639164 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fa4a4077010 0x7fa4a4079520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:48:06.258 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:06.258+0000 7fa4c0c00640 1 -- 172.21.15.32:0/3527639164 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7fa4bc0b8630 con 0x7fa4c4158590 2024-08-03T20:48:06.258 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:06.258+0000 7fa4c9000640 1 --2- 172.21.15.32:0/3527639164 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fa4a4077010 0x7fa4a4079520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:48:06.259 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:06.259+0000 7fa4c9000640 1 --2- 172.21.15.32:0/3527639164 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fa4a4077010 0x7fa4a4079520 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fa4b8009bd0 tx=0x7fa4b8005f90 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:48:06.263 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:06.262+0000 7fa4c0c00640 1 -- 172.21.15.32:0/3527639164 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7fa4c40cd5d0 con 0x7fa4c4158590 2024-08-03T20:48:06.324 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-08-03T20:48:06.328 DEBUG:teuthology.orchestra.run.smithi118:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-08-03T20:48:06.474 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:06.473+0000 7fa4ca400640 1 -- 172.21.15.32:0/3527639164 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fa4c415d6b0 con 0x7fa4c4158590 2024-08-03T20:48:06.474 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:06.474+0000 7fa4c0c00640 1 -- 172.21.15.32:0/3527639164 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v21) ==== 100+0+11 (secure 0 0 0) 0x7fa4bc081970 con 0x7fa4c4158590 2024-08-03T20:48:06.475 INFO:teuthology.orchestra.run.smithi032.stdout:pg_num: 64 2024-08-03T20:48:06.477 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:06.477+0000 7fa4ca400640 1 -- 172.21.15.32:0/3527639164 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fa4a4077010 msgr2=0x7fa4a4079520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:48:06.477 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:06.477+0000 7fa4ca400640 1 --2- 172.21.15.32:0/3527639164 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fa4a4077010 0x7fa4a4079520 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fa4b8009bd0 tx=0x7fa4b8005f90 comp rx=0 tx=0).stop 2024-08-03T20:48:06.478 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:06.477+0000 7fa4ca400640 1 -- 172.21.15.32:0/3527639164 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa4c4158590 msgr2=0x7fa4c41589a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:48:06.478 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:06.477+0000 7fa4ca400640 1 --2- 172.21.15.32:0/3527639164 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa4c4158590 0x7fa4c41589a0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fa4bc02b4e0 tx=0x7fa4bc003ba0 comp rx=0 tx=0).stop 2024-08-03T20:48:06.478 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:06.477+0000 7fa4ca400640 1 -- 172.21.15.32:0/3527639164 shutdown_connections 2024-08-03T20:48:06.478 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:06.477+0000 7fa4ca400640 1 --2- 172.21.15.32:0/3527639164 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fa4a4077010 0x7fa4a4079520 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:06.478 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:06.477+0000 7fa4ca400640 1 --2- 172.21.15.32:0/3527639164 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa4c4158590 0x7fa4c41589a0 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:06.478 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:06.477+0000 7fa4ca400640 1 --2- 172.21.15.32:0/3527639164 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fa4c415d210 0x7fa4c4158050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:06.478 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:06.477+0000 7fa4ca400640 1 --2- 172.21.15.32:0/3527639164 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa4ac005350 0x7fa4c4159570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:06.478 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:06.477+0000 7fa4ca400640 1 -- 172.21.15.32:0/3527639164 >> 172.21.15.32:0/3527639164 conn(0x7fa4c40d1eb0 msgr2=0x7fa4c40b0530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:48:06.478 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:06.477+0000 7fa4ca400640 1 -- 172.21.15.32:0/3527639164 shutdown_connections 2024-08-03T20:48:06.478 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:06.477+0000 7fa4ca400640 1 -- 172.21.15.32:0/3527639164 wait complete. 2024-08-03T20:48:06.503 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_damage.TestDamage.test_damaged_dentry' 2024-08-03T20:48:06.737 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:06.736+0000 7f0b45000640 1 -- 172.21.15.32:0/4025187306 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0b40160c10 msgr2=0x7f0b4015ca60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:48:06.737 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:06.736+0000 7f0b45000640 1 --2- 172.21.15.32:0/4025187306 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0b40160c10 0x7f0b4015ca60 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f0b30005fd0 tx=0x7f0b3002a530 comp rx=0 tx=0).stop 2024-08-03T20:48:06.737 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:06.736+0000 7f0b45000640 1 -- 172.21.15.32:0/4025187306 shutdown_connections 2024-08-03T20:48:06.737 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:06.736+0000 7f0b45000640 1 --2- 172.21.15.32:0/4025187306 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f0b400b41a0 0x7f0b400b45b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:06.737 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:06.736+0000 7f0b45000640 1 --2- 172.21.15.32:0/4025187306 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0b40160c10 0x7f0b4015ca60 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:06.737 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:06.736+0000 7f0b45000640 1 -- 172.21.15.32:0/4025187306 >> 172.21.15.32:0/4025187306 conn(0x7f0b400b1c40 msgr2=0x7f0b400b2050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:48:06.738 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:06.736+0000 7f0b45000640 1 -- 172.21.15.32:0/4025187306 shutdown_connections 2024-08-03T20:48:06.738 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:06.736+0000 7f0b45000640 1 -- 172.21.15.32:0/4025187306 wait complete. 2024-08-03T20:48:06.739 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:06.738+0000 7f0b45000640 1 Processor -- start 2024-08-03T20:48:06.739 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:06.739+0000 7f0b45000640 1 -- start start 2024-08-03T20:48:06.742 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:06.740+0000 7f0b45000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0b400b41a0 0x7f0b401424a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:48:06.742 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:06.740+0000 7f0b45000640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f0b400cc960 0x7f0b40073b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:48:06.742 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:06.740+0000 7f0b45000640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f0b40160c10 0x7f0b40074050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:48:06.742 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:06.740+0000 7f0b45000640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f0b400b4c70 con 0x7f0b400b41a0 2024-08-03T20:48:06.742 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:06.740+0000 7f0b45000640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f0b400b4af0 con 0x7f0b40160c10 2024-08-03T20:48:06.742 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:06.740+0000 7f0b45000640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7f0b400b4df0 con 0x7f0b400cc960 2024-08-03T20:48:06.742 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:06.740+0000 7f0b3de00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0b400b41a0 0x7f0b401424a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:48:06.743 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:06.740+0000 7f0b3de00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0b400b41a0 0x7f0b401424a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:55966/0 (socket says 172.21.15.32:55966) 2024-08-03T20:48:06.743 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:06.740+0000 7f0b3de00640 1 -- 172.21.15.32:0/3912745138 learned_addr learned my addr 172.21.15.32:0/3912745138 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:48:06.743 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:06.741+0000 7f0b37e00640 1 --2- 172.21.15.32:0/3912745138 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f0b400cc960 0x7f0b40073b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:48:06.743 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:06.741+0000 7f0b3e800640 1 --2- 172.21.15.32:0/3912745138 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f0b40160c10 0x7f0b40074050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:48:06.743 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:06.741+0000 7f0b3de00640 1 -- 172.21.15.32:0/3912745138 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f0b400cc960 msgr2=0x7f0b40073b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:48:06.743 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:06.741+0000 7f0b3de00640 1 --2- 172.21.15.32:0/3912745138 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f0b400cc960 0x7f0b40073b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:06.743 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:06.741+0000 7f0b3de00640 1 -- 172.21.15.32:0/3912745138 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f0b40160c10 msgr2=0x7f0b40074050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:48:06.743 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:06.741+0000 7f0b3de00640 1 --2- 172.21.15.32:0/3912745138 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f0b40160c10 0x7f0b40074050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:06.743 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:06.741+0000 7f0b3de00640 1 -- 172.21.15.32:0/3912745138 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0b401554a0 con 0x7f0b400b41a0 2024-08-03T20:48:06.743 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:06.741+0000 7f0b3de00640 1 --2- 172.21.15.32:0/3912745138 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0b400b41a0 0x7f0b401424a0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f0b30034040 tx=0x7f0b300039e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:48:06.743 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:06.742+0000 7f0b36000640 1 -- 172.21.15.32:0/3912745138 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b3003e030 con 0x7f0b400b41a0 2024-08-03T20:48:06.743 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:06.742+0000 7f0b36000640 1 -- 172.21.15.32:0/3912745138 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f0b300047a0 con 0x7f0b400b41a0 2024-08-03T20:48:06.744 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:06.742+0000 7f0b36000640 1 -- 172.21.15.32:0/3912745138 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b30035040 con 0x7f0b400b41a0 2024-08-03T20:48:06.744 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:06.742+0000 7f0b45000640 1 -- 172.21.15.32:0/3912745138 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0b401551c0 con 0x7f0b400b41a0 2024-08-03T20:48:06.744 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:06.742+0000 7f0b45000640 1 -- 172.21.15.32:0/3912745138 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0b40073310 con 0x7f0b400b41a0 2024-08-03T20:48:06.744 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:06.743+0000 7f0b45000640 1 -- 172.21.15.32:0/3912745138 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0b400b52c0 con 0x7f0b400b41a0 2024-08-03T20:48:06.745 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:06.744+0000 7f0b36000640 1 -- 172.21.15.32:0/3912745138 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f0b30041030 con 0x7f0b400b41a0 2024-08-03T20:48:06.745 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:06.745+0000 7f0b36000640 1 --2- 172.21.15.32:0/3912745138 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f0b1c077010 0x7f0b1c079520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:48:06.746 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:06.745+0000 7f0b36000640 1 -- 172.21.15.32:0/3912745138 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f0b300b8fc0 con 0x7f0b400b41a0 2024-08-03T20:48:06.746 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:06.746+0000 7f0b37e00640 1 --2- 172.21.15.32:0/3912745138 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f0b1c077010 0x7f0b1c079520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:48:06.747 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:06.746+0000 7f0b37e00640 1 --2- 172.21.15.32:0/3912745138 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f0b1c077010 0x7f0b1c079520 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f0b400af5e0 tx=0x7f0b28009060 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:48:06.750 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:06.749+0000 7f0b36000640 1 -- 172.21.15.32:0/3912745138 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7f0b30081b60 con 0x7f0b400b41a0 2024-08-03T20:48:07.037 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:07.037+0000 7f0b45000640 1 -- 172.21.15.32:0/3912745138 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_damage.TestDamage.test_damaged_dentry"]} v 0) -- 0x7f0b401539c0 con 0x7f0b400b41a0 2024-08-03T20:48:08.031 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:08.031+0000 7f0b36000640 1 -- 172.21.15.32:0/3912745138 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_damage.TestDamage.test_damaged_dentry"]}]=0 v48) ==== 137+0+0 (secure 0 0 0) 0x7f0b3008a3f0 con 0x7f0b400b41a0 2024-08-03T20:48:08.034 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:08.033+0000 7f0b45000640 1 -- 172.21.15.32:0/3912745138 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f0b1c077010 msgr2=0x7f0b1c079520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:48:08.034 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:08.033+0000 7f0b45000640 1 --2- 172.21.15.32:0/3912745138 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f0b1c077010 0x7f0b1c079520 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f0b400af5e0 tx=0x7f0b28009060 comp rx=0 tx=0).stop 2024-08-03T20:48:08.034 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:08.033+0000 7f0b45000640 1 -- 172.21.15.32:0/3912745138 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0b400b41a0 msgr2=0x7f0b401424a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:48:08.034 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:08.033+0000 7f0b45000640 1 --2- 172.21.15.32:0/3912745138 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0b400b41a0 0x7f0b401424a0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f0b30034040 tx=0x7f0b300039e0 comp rx=0 tx=0).stop 2024-08-03T20:48:08.034 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:08.033+0000 7f0b45000640 1 -- 172.21.15.32:0/3912745138 shutdown_connections 2024-08-03T20:48:08.035 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:08.033+0000 7f0b45000640 1 --2- 172.21.15.32:0/3912745138 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f0b1c077010 0x7f0b1c079520 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:08.035 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:08.033+0000 7f0b45000640 1 --2- 172.21.15.32:0/3912745138 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f0b40160c10 0x7f0b40074050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:08.035 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:08.033+0000 7f0b45000640 1 --2- 172.21.15.32:0/3912745138 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f0b400cc960 0x7f0b40073b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:08.035 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:08.033+0000 7f0b45000640 1 --2- 172.21.15.32:0/3912745138 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0b400b41a0 0x7f0b401424a0 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:08.035 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:08.033+0000 7f0b45000640 1 -- 172.21.15.32:0/3912745138 >> 172.21.15.32:0/3912745138 conn(0x7f0b400b1c40 msgr2=0x7f0b400cece0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:48:08.035 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:08.033+0000 7f0b45000640 1 -- 172.21.15.32:0/3912745138 shutdown_connections 2024-08-03T20:48:08.035 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:08.034+0000 7f0b45000640 1 -- 172.21.15.32:0/3912745138 wait complete. 2024-08-03T20:48:08.057 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config dump 2024-08-03T20:48:08.288 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:08.287+0000 7faca1c00640 1 -- 172.21.15.32:0/2976970207 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fac80005250 msgr2=0x7fac8000b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:48:08.288 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:08.287+0000 7faca1c00640 1 --2- 172.21.15.32:0/2976970207 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fac80005250 0x7fac8000b9c0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fac9400c610 tx=0x7fac80005830 comp rx=0 tx=0).stop 2024-08-03T20:48:08.288 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:08.287+0000 7faca1c00640 1 -- 172.21.15.32:0/2976970207 shutdown_connections 2024-08-03T20:48:08.288 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:08.287+0000 7faca1c00640 1 --2- 172.21.15.32:0/2976970207 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fac80005250 0x7fac8000b9c0 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:08.289 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:08.287+0000 7faca1c00640 1 -- 172.21.15.32:0/2976970207 >> 172.21.15.32:0/2976970207 conn(0x7fac9c0b1bd0 msgr2=0x7fac9c0b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:48:08.289 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:08.287+0000 7faca1c00640 1 -- 172.21.15.32:0/2976970207 shutdown_connections 2024-08-03T20:48:08.289 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:08.287+0000 7faca1c00640 1 -- 172.21.15.32:0/2976970207 wait complete. 2024-08-03T20:48:08.290 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:08.290+0000 7faca1c00640 1 Processor -- start 2024-08-03T20:48:08.290 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:08.290+0000 7faca1c00640 1 -- start start 2024-08-03T20:48:08.292 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:08.291+0000 7faca1c00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fac80005250 0x7fac9c0ca880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:48:08.292 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:08.291+0000 7faca1c00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fac9c0c5330 0x7fac9c0c5740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:48:08.292 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:08.291+0000 7faca1c00640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fac9c1f6640 0x7fac9c1f6a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:48:08.293 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:08.291+0000 7faca1c00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fac9c145110 con 0x7fac80005250 2024-08-03T20:48:08.293 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:08.291+0000 7faca1c00640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7fac9c144f90 con 0x7fac9c0c5330 2024-08-03T20:48:08.293 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:08.291+0000 7faca1c00640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7fac9c145290 con 0x7fac9c1f6640 2024-08-03T20:48:08.293 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:08.291+0000 7fac9aa00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fac80005250 0x7fac9c0ca880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:48:08.293 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:08.291+0000 7fac9aa00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fac80005250 0x7fac9c0ca880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:55976/0 (socket says 172.21.15.32:55976) 2024-08-03T20:48:08.293 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:08.291+0000 7fac9aa00640 1 -- 172.21.15.32:0/768768860 learned_addr learned my addr 172.21.15.32:0/768768860 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:48:08.293 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:08.292+0000 7fac9b400640 1 --2- 172.21.15.32:0/768768860 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fac9c1f6640 0x7fac9c1f6a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:48:08.293 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:08.292+0000 7fac93e00640 1 --2- 172.21.15.32:0/768768860 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fac9c0c5330 0x7fac9c0c5740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:48:08.293 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:08.292+0000 7fac9aa00640 1 -- 172.21.15.32:0/768768860 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fac9c1f6640 msgr2=0x7fac9c1f6a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:48:08.293 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:08.292+0000 7fac9aa00640 1 --2- 172.21.15.32:0/768768860 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fac9c1f6640 0x7fac9c1f6a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:08.293 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:08.292+0000 7fac9aa00640 1 -- 172.21.15.32:0/768768860 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fac9c0c5330 msgr2=0x7fac9c0c5740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:48:08.293 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:08.292+0000 7fac9aa00640 1 --2- 172.21.15.32:0/768768860 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fac9c0c5330 0x7fac9c0c5740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:08.293 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:08.292+0000 7fac9aa00640 1 -- 172.21.15.32:0/768768860 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fac9c1f6f90 con 0x7fac80005250 2024-08-03T20:48:08.294 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:08.292+0000 7fac93e00640 1 --2- 172.21.15.32:0/768768860 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fac9c0c5330 0x7fac9c0c5740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-03T20:48:08.294 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:08.292+0000 7fac9aa00640 1 --2- 172.21.15.32:0/768768860 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fac80005250 0x7fac9c0ca880 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fac8c011ab0 tx=0x7fac8c011f70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:48:08.295 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:08.292+0000 7fac92a00640 1 -- 172.21.15.32:0/768768860 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac8c01e070 con 0x7fac80005250 2024-08-03T20:48:08.295 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:08.292+0000 7fac92a00640 1 -- 172.21.15.32:0/768768860 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fac8c00fe70 con 0x7fac80005250 2024-08-03T20:48:08.295 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:08.292+0000 7fac92a00640 1 -- 172.21.15.32:0/768768860 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac8c0042b0 con 0x7fac80005250 2024-08-03T20:48:08.295 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:08.292+0000 7faca1c00640 1 -- 172.21.15.32:0/768768860 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fac9c0c6100 con 0x7fac80005250 2024-08-03T20:48:08.295 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:08.293+0000 7faca1c00640 1 -- 172.21.15.32:0/768768860 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fac9c0c66c0 con 0x7fac80005250 2024-08-03T20:48:08.295 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:08.294+0000 7faca1c00640 1 -- 172.21.15.32:0/768768860 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fac9c074350 con 0x7fac80005250 2024-08-03T20:48:08.295 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:08.295+0000 7fac92a00640 1 -- 172.21.15.32:0/768768860 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7fac8c021070 con 0x7fac80005250 2024-08-03T20:48:08.296 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:08.296+0000 7fac92a00640 1 --2- 172.21.15.32:0/768768860 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fac78077040 0x7fac78079550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:48:08.296 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:08.296+0000 7fac92a00640 1 -- 172.21.15.32:0/768768860 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7fac8c099290 con 0x7fac80005250 2024-08-03T20:48:08.297 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:08.296+0000 7fac93e00640 1 --2- 172.21.15.32:0/768768860 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fac78077040 0x7fac78079550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:48:08.297 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:08.297+0000 7fac93e00640 1 --2- 172.21.15.32:0/768768860 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fac78077040 0x7fac78079550 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fac9c14d4c0 tx=0x7fac84006b70 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:48:08.300 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:08.300+0000 7fac92a00640 1 -- 172.21.15.32:0/768768860 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7fac9c1f6f90 con 0x7fac80005250 2024-08-03T20:48:08.506 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:08.506+0000 7faca1c00640 1 -- 172.21.15.32:0/768768860 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "config dump"} v 0) -- 0x7fac9c06f5d0 con 0x7fac80005250 2024-08-03T20:48:08.507 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:08.506+0000 7fac92a00640 1 -- 172.21.15.32:0/768768860 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "config dump"}]=0 v27) ==== 59+0+1998 (secure 0 0 0) 0x7fac8c061e30 con 0x7fac80005250 2024-08-03T20:48:08.507 INFO:teuthology.orchestra.run.smithi032.stdout:WHO MASK LEVEL OPTION VALUE RO 2024-08-03T20:48:08.507 INFO:teuthology.orchestra.run.smithi032.stdout:mon advanced mon_op_complaint_time 120 2024-08-03T20:48:08.507 INFO:teuthology.orchestra.run.smithi032.stdout:mon dev mon_warn_on_pool_no_app false 2024-08-03T20:48:08.507 INFO:teuthology.orchestra.run.smithi032.stdout:osd advanced osd_op_complaint_time 180.000000 2024-08-03T20:48:08.507 INFO:teuthology.orchestra.run.smithi032.stdout:osd.0 basic osd_mclock_max_capacity_iops_ssd 4313.565729 2024-08-03T20:48:08.507 INFO:teuthology.orchestra.run.smithi032.stdout:osd.1 basic osd_mclock_max_capacity_iops_ssd 4272.517979 2024-08-03T20:48:08.508 INFO:teuthology.orchestra.run.smithi032.stdout:osd.2 basic osd_mclock_max_capacity_iops_ssd 4131.201546 2024-08-03T20:48:08.508 INFO:teuthology.orchestra.run.smithi032.stdout:osd.3 basic osd_mclock_max_capacity_iops_ssd 4155.765914 2024-08-03T20:48:08.508 INFO:teuthology.orchestra.run.smithi032.stdout:osd.4 basic osd_mclock_max_capacity_iops_ssd 5744.667532 2024-08-03T20:48:08.508 INFO:teuthology.orchestra.run.smithi032.stdout:osd.5 basic osd_mclock_max_capacity_iops_ssd 3764.678226 2024-08-03T20:48:08.508 INFO:teuthology.orchestra.run.smithi032.stdout:osd.6 basic osd_mclock_max_capacity_iops_ssd 3927.989098 2024-08-03T20:48:08.508 INFO:teuthology.orchestra.run.smithi032.stdout:osd.7 basic osd_mclock_max_capacity_iops_ssd 3760.968870 2024-08-03T20:48:08.508 INFO:teuthology.orchestra.run.smithi032.stdout:mds advanced debug_mds 20/20 2024-08-03T20:48:08.508 INFO:teuthology.orchestra.run.smithi032.stdout:mds advanced debug_mds_balancer 20/20 2024-08-03T20:48:08.508 INFO:teuthology.orchestra.run.smithi032.stdout:mds advanced debug_ms 2/2 2024-08-03T20:48:08.508 INFO:teuthology.orchestra.run.smithi032.stdout:mds dev mds_debug_frag true 2024-08-03T20:48:08.508 INFO:teuthology.orchestra.run.smithi032.stdout:mds dev mds_debug_scatterstat true 2024-08-03T20:48:08.509 INFO:teuthology.orchestra.run.smithi032.stdout:mds advanced mds_op_complaint_time 180.000000 2024-08-03T20:48:08.509 INFO:teuthology.orchestra.run.smithi032.stdout:mds dev mds_verify_scatter true 2024-08-03T20:48:08.509 INFO:teuthology.orchestra.run.smithi032.stdout:mds advanced osd_op_complaint_time 180.000000 2024-08-03T20:48:08.509 INFO:teuthology.orchestra.run.smithi032.stdout:mds advanced rados_mon_op_timeout 900 2024-08-03T20:48:08.509 INFO:teuthology.orchestra.run.smithi032.stdout:mds advanced rados_osd_op_timeout 900 2024-08-03T20:48:08.509 INFO:teuthology.orchestra.run.smithi032.stdout:client advanced client_mount_timeout 600 2024-08-03T20:48:08.509 INFO:teuthology.orchestra.run.smithi032.stdout:client advanced debug_client 20/20 2024-08-03T20:48:08.509 INFO:teuthology.orchestra.run.smithi032.stdout:client advanced debug_ms 1/1 2024-08-03T20:48:08.509 INFO:teuthology.orchestra.run.smithi032.stdout:client advanced rados_mon_op_timeout 900 2024-08-03T20:48:08.509 INFO:teuthology.orchestra.run.smithi032.stdout:client advanced rados_osd_op_timeout 900 2024-08-03T20:48:08.510 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:08.508+0000 7faca1c00640 1 -- 172.21.15.32:0/768768860 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fac78077040 msgr2=0x7fac78079550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:48:08.510 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:08.508+0000 7faca1c00640 1 --2- 172.21.15.32:0/768768860 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fac78077040 0x7fac78079550 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fac9c14d4c0 tx=0x7fac84006b70 comp rx=0 tx=0).stop 2024-08-03T20:48:08.510 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:08.509+0000 7faca1c00640 1 -- 172.21.15.32:0/768768860 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fac80005250 msgr2=0x7fac9c0ca880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:48:08.510 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:08.509+0000 7faca1c00640 1 --2- 172.21.15.32:0/768768860 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fac80005250 0x7fac9c0ca880 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fac8c011ab0 tx=0x7fac8c011f70 comp rx=0 tx=0).stop 2024-08-03T20:48:08.510 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:08.509+0000 7faca1c00640 1 -- 172.21.15.32:0/768768860 shutdown_connections 2024-08-03T20:48:08.510 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:08.509+0000 7faca1c00640 1 --2- 172.21.15.32:0/768768860 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fac78077040 0x7fac78079550 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:08.511 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:08.509+0000 7faca1c00640 1 --2- 172.21.15.32:0/768768860 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fac9c0c5330 0x7fac9c0c5740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:08.511 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:08.509+0000 7faca1c00640 1 --2- 172.21.15.32:0/768768860 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fac9c1f6640 0x7fac9c1f6a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:08.511 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:08.509+0000 7faca1c00640 1 --2- 172.21.15.32:0/768768860 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fac80005250 0x7fac9c0ca880 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:08.511 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:08.509+0000 7faca1c00640 1 -- 172.21.15.32:0/768768860 >> 172.21.15.32:0/768768860 conn(0x7fac9c0b1bd0 msgr2=0x7fac9c145610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:48:08.511 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:08.509+0000 7faca1c00640 1 -- 172.21.15.32:0/768768860 shutdown_connections 2024-08-03T20:48:08.511 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:08.509+0000 7faca1c00640 1 -- 172.21.15.32:0/768768860 wait complete. 2024-08-03T20:48:08.531 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-08-03T20:48:08.763 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:08.762+0000 7f2f16000640 1 -- 172.21.15.32:0/2271414691 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f2f1009ac30 msgr2=0x7f2f1009b040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:48:08.763 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:08.762+0000 7f2f16000640 1 --2- 172.21.15.32:0/2271414691 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f2f1009ac30 0x7f2f1009b040 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f2f00009830 tx=0x7f2f0002a480 comp rx=0 tx=0).stop 2024-08-03T20:48:08.763 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:08.762+0000 7f2f16000640 1 -- 172.21.15.32:0/2271414691 shutdown_connections 2024-08-03T20:48:08.763 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:08.762+0000 7f2f16000640 1 --2- 172.21.15.32:0/2271414691 >> v2:172.21.15.118:3300/0 conn(0x7f2f100d00f0 0x7f2f10166de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:08.763 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:08.762+0000 7f2f16000640 1 --2- 172.21.15.32:0/2271414691 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f2f1009ac30 0x7f2f1009b040 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:08.763 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:08.762+0000 7f2f16000640 1 -- 172.21.15.32:0/2271414691 >> 172.21.15.32:0/2271414691 conn(0x7f2f100b3c00 msgr2=0x7f2f100b4010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:48:08.763 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:08.762+0000 7f2f16000640 1 -- 172.21.15.32:0/2271414691 shutdown_connections 2024-08-03T20:48:08.763 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:08.762+0000 7f2f16000640 1 -- 172.21.15.32:0/2271414691 wait complete. 2024-08-03T20:48:08.765 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:08.764+0000 7f2f16000640 1 Processor -- start 2024-08-03T20:48:08.765 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:08.764+0000 7f2f16000640 1 -- start start 2024-08-03T20:48:08.768 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:08.766+0000 7f2f16000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2f1009ac30 0x7f2f101e2e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:48:08.768 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:08.766+0000 7f2f16000640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f2f100cfce0 0x7f2f101e82a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:48:08.768 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:08.766+0000 7f2f16000640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f2f100d00f0 0x7f2f101f6df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:48:08.768 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:08.766+0000 7f2f16000640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f2f100b7ba0 con 0x7f2f1009ac30 2024-08-03T20:48:08.768 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:08.766+0000 7f2f16000640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f2f100b7a20 con 0x7f2f100d00f0 2024-08-03T20:48:08.768 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:08.766+0000 7f2f16000640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7f2f100b2ca0 con 0x7f2f100cfce0 2024-08-03T20:48:08.768 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:08.766+0000 7f2f0f400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2f1009ac30 0x7f2f101e2e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:48:08.768 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:08.766+0000 7f2f0f400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2f1009ac30 0x7f2f101e2e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:55988/0 (socket says 172.21.15.32:55988) 2024-08-03T20:48:08.768 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:08.766+0000 7f2f0f400640 1 -- 172.21.15.32:0/3979528438 learned_addr learned my addr 172.21.15.32:0/3979528438 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:48:08.769 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:08.766+0000 7f2f0ea00640 1 --2- 172.21.15.32:0/3979528438 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f2f100cfce0 0x7f2f101e82a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:48:08.769 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:08.767+0000 7f2f0fe00640 1 --2- 172.21.15.32:0/3979528438 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f2f100d00f0 0x7f2f101f6df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:48:08.769 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:08.767+0000 7f2f0f400640 1 -- 172.21.15.32:0/3979528438 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f2f100cfce0 msgr2=0x7f2f101e82a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:48:08.769 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:08.767+0000 7f2f0f400640 1 --2- 172.21.15.32:0/3979528438 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f2f100cfce0 0x7f2f101e82a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:08.769 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:08.767+0000 7f2f0f400640 1 -- 172.21.15.32:0/3979528438 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f2f100d00f0 msgr2=0x7f2f101f6df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:48:08.769 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:08.767+0000 7f2f0f400640 1 --2- 172.21.15.32:0/3979528438 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f2f100d00f0 0x7f2f101f6df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:08.769 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:08.767+0000 7f2f0f400640 1 -- 172.21.15.32:0/3979528438 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2f101e6770 con 0x7f2f1009ac30 2024-08-03T20:48:08.769 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:08.767+0000 7f2f0fe00640 1 --2- 172.21.15.32:0/3979528438 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f2f100d00f0 0x7f2f101f6df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-03T20:48:08.769 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:08.767+0000 7f2f0f400640 1 --2- 172.21.15.32:0/3979528438 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2f1009ac30 0x7f2f101e2e30 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f2f00009950 tx=0x7f2f00033c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:48:08.769 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:08.767+0000 7f2f07400640 1 -- 172.21.15.32:0/3979528438 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f00005c90 con 0x7f2f1009ac30 2024-08-03T20:48:08.769 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:08.767+0000 7f2f07400640 1 -- 172.21.15.32:0/3979528438 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f2f00031e50 con 0x7f2f1009ac30 2024-08-03T20:48:08.769 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:08.767+0000 7f2f07400640 1 -- 172.21.15.32:0/3979528438 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f00005260 con 0x7f2f1009ac30 2024-08-03T20:48:08.770 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:08.767+0000 7f2f16000640 1 -- 172.21.15.32:0/3979528438 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2f101e3680 con 0x7f2f1009ac30 2024-08-03T20:48:08.770 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:08.767+0000 7f2f16000640 1 -- 172.21.15.32:0/3979528438 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2f101e3c40 con 0x7f2f1009ac30 2024-08-03T20:48:08.770 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:08.769+0000 7f2f16000640 1 -- 172.21.15.32:0/3979528438 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2f101e6940 con 0x7f2f1009ac30 2024-08-03T20:48:08.770 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:08.770+0000 7f2f07400640 1 -- 172.21.15.32:0/3979528438 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f2f00005400 con 0x7f2f1009ac30 2024-08-03T20:48:08.771 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:08.770+0000 7f2f07400640 1 --2- 172.21.15.32:0/3979528438 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f2ef0077010 0x7f2ef0079520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:48:08.771 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:08.771+0000 7f2f07400640 1 -- 172.21.15.32:0/3979528438 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f2f000b9890 con 0x7f2f1009ac30 2024-08-03T20:48:08.771 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:08.771+0000 7f2f0ea00640 1 --2- 172.21.15.32:0/3979528438 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f2ef0077010 0x7f2ef0079520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:48:08.772 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:08.772+0000 7f2f0ea00640 1 --2- 172.21.15.32:0/3979528438 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f2ef0077010 0x7f2ef0079520 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f2f101e98b0 tx=0x7f2efc009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:48:08.775 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:08.775+0000 7f2f07400640 1 -- 172.21.15.32:0/3979528438 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7f2f0003f030 con 0x7f2f1009ac30 2024-08-03T20:48:08.983 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:08.982+0000 7f2f16000640 1 -- 172.21.15.32:0/3979528438 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) -- 0x7f2f1009b040 con 0x7f2f1009ac30 2024-08-03T20:48:08.988 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:08.988+0000 7f2f07400640 1 -- 172.21.15.32:0/3979528438 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v28)=0 v28) ==== 122+0+0 (secure 0 0 0) 0x7f2f00082b10 con 0x7f2f1009ac30 2024-08-03T20:48:08.991 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:08.990+0000 7f2f16000640 1 -- 172.21.15.32:0/3979528438 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f2ef0077010 msgr2=0x7f2ef0079520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:48:08.991 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:08.990+0000 7f2f16000640 1 --2- 172.21.15.32:0/3979528438 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f2ef0077010 0x7f2ef0079520 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f2f101e98b0 tx=0x7f2efc009040 comp rx=0 tx=0).stop 2024-08-03T20:48:08.991 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:08.990+0000 7f2f16000640 1 -- 172.21.15.32:0/3979528438 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2f1009ac30 msgr2=0x7f2f101e2e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:48:08.991 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:08.990+0000 7f2f16000640 1 --2- 172.21.15.32:0/3979528438 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2f1009ac30 0x7f2f101e2e30 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f2f00009950 tx=0x7f2f00033c20 comp rx=0 tx=0).stop 2024-08-03T20:48:08.991 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:08.990+0000 7f2f16000640 1 -- 172.21.15.32:0/3979528438 shutdown_connections 2024-08-03T20:48:08.991 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:08.990+0000 7f2f16000640 1 --2- 172.21.15.32:0/3979528438 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f2ef0077010 0x7f2ef0079520 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:08.991 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:08.990+0000 7f2f16000640 1 --2- 172.21.15.32:0/3979528438 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f2f100d00f0 0x7f2f101f6df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:08.991 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:08.990+0000 7f2f16000640 1 --2- 172.21.15.32:0/3979528438 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f2f100cfce0 0x7f2f101e82a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:08.991 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:08.990+0000 7f2f16000640 1 --2- 172.21.15.32:0/3979528438 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2f1009ac30 0x7f2f101e2e30 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:08.991 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:08.990+0000 7f2f16000640 1 -- 172.21.15.32:0/3979528438 >> 172.21.15.32:0/3979528438 conn(0x7f2f100b3c00 msgr2=0x7f2f100ceb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:48:08.991 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:08.990+0000 7f2f16000640 1 -- 172.21.15.32:0/3979528438 shutdown_connections 2024-08-03T20:48:08.992 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:08.990+0000 7f2f16000640 1 -- 172.21.15.32:0/3979528438 wait complete. 2024-08-03T20:48:09.013 INFO:teuthology.nuke:Clearing teuthology firewall rules... 2024-08-03T20:48:09.014 DEBUG:teuthology.orchestra.run.smithi032:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-08-03T20:48:09.103 DEBUG:teuthology.orchestra.run.smithi118:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-08-03T20:48:09.195 INFO:teuthology.nuke:Cleared teuthology firewall rules. 2024-08-03T20:48:09.195 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-08-03T20:48:09.195 DEBUG:teuthology.orchestra.run.smithi032:> dd if=/proc/self/mounts of=/dev/stdout 2024-08-03T20:48:09.231 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-08-03T20:48:09.231 DEBUG:teuthology.orchestra.run.smithi032:> dd if=/proc/self/mounts of=/dev/stdout 2024-08-03T20:48:09.306 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-08-03T20:48:09.306 DEBUG:teuthology.orchestra.run.smithi032:> dd if=/proc/self/mounts of=/dev/stdout 2024-08-03T20:48:09.382 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-08-03T20:48:09.652 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:09.651+0000 7f9367e00640 1 -- 172.21.15.32:0/700905639 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9348003a00 msgr2=0x7f9348003e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:48:09.652 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:09.651+0000 7f9367e00640 1 --2- 172.21.15.32:0/700905639 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9348003a00 0x7f9348003e70 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f9354009d00 tx=0x7f9348005830 comp rx=0 tx=0).stop 2024-08-03T20:48:09.652 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:09.651+0000 7f9365e00640 1 -- 172.21.15.32:0/700905639 reap_dead start 2024-08-03T20:48:09.652 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:09.651+0000 7f9367e00640 1 -- 172.21.15.32:0/700905639 shutdown_connections 2024-08-03T20:48:09.652 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:09.651+0000 7f9367e00640 1 -- 172.21.15.32:0/700905639 >> 172.21.15.32:0/700905639 conn(0x7f93600b1bd0 msgr2=0x7f93600b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:48:09.652 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:09.651+0000 7f9367e00640 1 -- 172.21.15.32:0/700905639 shutdown_connections 2024-08-03T20:48:09.652 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:09.651+0000 7f9367e00640 1 -- 172.21.15.32:0/700905639 wait complete. 2024-08-03T20:48:09.654 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:09.653+0000 7f9367e00640 1 Processor -- start 2024-08-03T20:48:09.654 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:09.654+0000 7f9367e00640 1 -- start start 2024-08-03T20:48:09.656 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:09.655+0000 7f9367e00640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f9348003a00 0x7f9360140600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:48:09.656 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:09.655+0000 7f9367e00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f9360073e10 0x7f936013d250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:48:09.656 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:09.655+0000 7f9367e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f936013dbc0 0x7f936013dfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:48:09.656 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:09.655+0000 7f9367e00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f936015ef00 con 0x7f936013dbc0 2024-08-03T20:48:09.656 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:09.655+0000 7f9367e00640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f93600cf210 con 0x7f9360073e10 2024-08-03T20:48:09.656 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:09.655+0000 7f9367e00640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7f936015f080 con 0x7f9348003a00 2024-08-03T20:48:09.656 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:09.655+0000 7f9365e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f936013dbc0 0x7f936013dfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:48:09.657 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:09.655+0000 7f9365e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f936013dbc0 0x7f936013dfd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:55998/0 (socket says 172.21.15.32:55998) 2024-08-03T20:48:09.657 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:09.655+0000 7f9365e00640 1 -- 172.21.15.32:0/4022778272 learned_addr learned my addr 172.21.15.32:0/4022778272 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:48:09.657 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:09.655+0000 7f9364a00640 1 --2- 172.21.15.32:0/4022778272 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f9360073e10 0x7f936013d250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:48:09.657 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:09.655+0000 7f9365400640 1 --2- 172.21.15.32:0/4022778272 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f9348003a00 0x7f9360140600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:48:09.657 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:09.656+0000 7f9365e00640 1 -- 172.21.15.32:0/4022778272 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f9348003a00 msgr2=0x7f9360140600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:48:09.657 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:09.656+0000 7f9365e00640 1 --2- 172.21.15.32:0/4022778272 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f9348003a00 0x7f9360140600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:09.657 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:09.656+0000 7f9365e00640 1 -- 172.21.15.32:0/4022778272 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f9360073e10 msgr2=0x7f936013d250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:48:09.657 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:09.656+0000 7f9365e00640 1 --2- 172.21.15.32:0/4022778272 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f9360073e10 0x7f936013d250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:09.657 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:09.656+0000 7f9365e00640 1 -- 172.21.15.32:0/4022778272 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f936013e510 con 0x7f936013dbc0 2024-08-03T20:48:09.657 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:09.656+0000 7f9365e00640 1 --2- 172.21.15.32:0/4022778272 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f936013dbc0 0x7f936013dfd0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f9358000f80 tx=0x7f9358006660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:48:09.658 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:09.656+0000 7f935d600640 1 -- 172.21.15.32:0/4022778272 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f935800fa80 con 0x7f936013dbc0 2024-08-03T20:48:09.658 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:09.656+0000 7f935d600640 1 -- 172.21.15.32:0/4022778272 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f9358019070 con 0x7f936013dbc0 2024-08-03T20:48:09.658 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:09.656+0000 7f935d600640 1 -- 172.21.15.32:0/4022778272 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9358011430 con 0x7f936013dbc0 2024-08-03T20:48:09.658 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:09.656+0000 7f9367e00640 1 -- 172.21.15.32:0/4022778272 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9360072d70 con 0x7f936013dbc0 2024-08-03T20:48:09.658 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:09.656+0000 7f9367e00640 1 -- 172.21.15.32:0/4022778272 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9360073330 con 0x7f936013dbc0 2024-08-03T20:48:09.658 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:09.657+0000 7f9367e00640 1 -- 172.21.15.32:0/4022778272 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f93601523f0 con 0x7f936013dbc0 2024-08-03T20:48:09.659 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:09.659+0000 7f935d600640 1 -- 172.21.15.32:0/4022778272 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f93580115d0 con 0x7f936013dbc0 2024-08-03T20:48:09.660 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:09.659+0000 7f935d600640 1 --2- 172.21.15.32:0/4022778272 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f9340077010 0x7f9340079520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:48:09.660 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:09.660+0000 7f935d600640 1 -- 172.21.15.32:0/4022778272 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f9358098c60 con 0x7f936013dbc0 2024-08-03T20:48:09.660 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:09.660+0000 7f9365400640 1 --2- 172.21.15.32:0/4022778272 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f9340077010 0x7f9340079520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:48:09.661 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:09.661+0000 7f9365400640 1 --2- 172.21.15.32:0/4022778272 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f9340077010 0x7f9340079520 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f9354035000 tx=0x7f93540046c0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:48:09.664 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:09.664+0000 7f935d600640 1 -- 172.21.15.32:0/4022778272 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7f936015f850 con 0x7f936013dbc0 2024-08-03T20:48:09.879 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:09.879+0000 7f9367e00640 1 -- 172.21.15.32:0/4022778272 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f9360072b00 con 0x7f936013dbc0 2024-08-03T20:48:09.880 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:09.879+0000 7f935d600640 1 -- 172.21.15.32:0/4022778272 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v21) ==== 80+0+0 (secure 0 0 0) 0x7f9358061dc0 con 0x7f936013dbc0 2024-08-03T20:48:09.880 INFO:teuthology.orchestra.run.smithi032.stderr:listed 0 entries 2024-08-03T20:48:09.883 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:09.882+0000 7f9367e00640 1 -- 172.21.15.32:0/4022778272 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f9340077010 msgr2=0x7f9340079520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:48:09.883 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:09.882+0000 7f9367e00640 1 --2- 172.21.15.32:0/4022778272 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f9340077010 0x7f9340079520 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f9354035000 tx=0x7f93540046c0 comp rx=0 tx=0).stop 2024-08-03T20:48:09.883 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:09.882+0000 7f9367e00640 1 -- 172.21.15.32:0/4022778272 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f936013dbc0 msgr2=0x7f936013dfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:48:09.883 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:09.882+0000 7f9367e00640 1 --2- 172.21.15.32:0/4022778272 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f936013dbc0 0x7f936013dfd0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f9358000f80 tx=0x7f9358006660 comp rx=0 tx=0).stop 2024-08-03T20:48:09.883 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:09.882+0000 7f9367e00640 1 -- 172.21.15.32:0/4022778272 shutdown_connections 2024-08-03T20:48:09.883 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:09.882+0000 7f9367e00640 1 --2- 172.21.15.32:0/4022778272 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f9340077010 0x7f9340079520 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:09.883 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:09.882+0000 7f9367e00640 1 --2- 172.21.15.32:0/4022778272 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f9360073e10 0x7f936013d250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:09.883 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:09.882+0000 7f9367e00640 1 --2- 172.21.15.32:0/4022778272 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f936013dbc0 0x7f936013dfd0 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:09.883 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:09.882+0000 7f9367e00640 1 --2- 172.21.15.32:0/4022778272 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f9348003a00 0x7f9360140600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:09.883 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:09.882+0000 7f9367e00640 1 -- 172.21.15.32:0/4022778272 >> 172.21.15.32:0/4022778272 conn(0x7f93600b1bd0 msgr2=0x7f93600cc260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:48:09.884 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:09.882+0000 7f9367e00640 1 -- 172.21.15.32:0/4022778272 shutdown_connections 2024-08-03T20:48:09.884 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:09.882+0000 7f9367e00640 1 -- 172.21.15.32:0/4022778272 wait complete. 2024-08-03T20:48:09.905 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-08-03T20:48:09.905 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-08-03T20:48:09.905 DEBUG:teuthology.orchestra.run.smithi032:> sudo umount /home/ubuntu/cephtest/mnt.0 -f 2024-08-03T20:48:09.967 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi032.front.sepia.ceph.com 2024-08-03T20:48:09.967 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-08-03T20:48:09.967 DEBUG:teuthology.orchestra.run.smithi032:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-08-03T20:48:10.013 DEBUG:teuthology.orchestra.run.smithi118:> set -ex 2024-08-03T20:48:10.013 DEBUG:teuthology.orchestra.run.smithi118:> dd if=/proc/self/mounts of=/dev/stdout 2024-08-03T20:48:10.056 DEBUG:teuthology.orchestra.run.smithi118:> set -ex 2024-08-03T20:48:10.056 DEBUG:teuthology.orchestra.run.smithi118:> dd if=/proc/self/mounts of=/dev/stdout 2024-08-03T20:48:10.136 DEBUG:teuthology.orchestra.run.smithi118:> set -ex 2024-08-03T20:48:10.136 DEBUG:teuthology.orchestra.run.smithi118:> dd if=/proc/self/mounts of=/dev/stdout 2024-08-03T20:48:10.216 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-08-03T20:48:10.447 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:10.446+0000 7f3ee8e00640 1 -- 172.21.15.32:0/1416766344 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3ec8005270 msgr2=0x7f3ec800b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:48:10.447 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:10.446+0000 7f3ee8e00640 1 --2- 172.21.15.32:0/1416766344 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3ec8005270 0x7f3ec800b9c0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f3edc009ed0 tx=0x7f3ec8005830 comp rx=0 tx=0).stop 2024-08-03T20:48:10.447 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:10.446+0000 7f3ee8e00640 1 -- 172.21.15.32:0/1416766344 shutdown_connections 2024-08-03T20:48:10.447 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:10.446+0000 7f3ee8e00640 1 --2- 172.21.15.32:0/1416766344 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3ec8005270 0x7f3ec800b9c0 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:10.447 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:10.446+0000 7f3ee8e00640 1 -- 172.21.15.32:0/1416766344 >> 172.21.15.32:0/1416766344 conn(0x7f3ee40d40c0 msgr2=0x7f3ee40d44d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:48:10.448 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:10.446+0000 7f3ee8e00640 1 -- 172.21.15.32:0/1416766344 shutdown_connections 2024-08-03T20:48:10.448 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:10.446+0000 7f3ee8e00640 1 -- 172.21.15.32:0/1416766344 wait complete. 2024-08-03T20:48:10.449 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:10.448+0000 7f3ee8e00640 1 Processor -- start 2024-08-03T20:48:10.449 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:10.449+0000 7f3ee8e00640 1 -- start start 2024-08-03T20:48:10.452 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:10.450+0000 7f3ee8e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3ec8005270 0x7f3ee40a1760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:48:10.452 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:10.450+0000 7f3ee8e00640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f3ee41e8ed0 0x7f3ee41e92e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:48:10.452 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:10.450+0000 7f3ee8e00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f3ee41f7280 0x7f3ee41f7690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:48:10.453 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:10.450+0000 7f3ee8e00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f3ee40d0ab0 con 0x7f3ec8005270 2024-08-03T20:48:10.453 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:10.450+0000 7f3ee8e00640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f3ee40b5880 con 0x7f3ee41f7280 2024-08-03T20:48:10.453 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:10.450+0000 7f3ee8e00640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7f3ee40d0c30 con 0x7f3ee41e8ed0 2024-08-03T20:48:10.453 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:10.450+0000 7f3ee3400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3ec8005270 0x7f3ee40a1760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:48:10.453 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:10.450+0000 7f3ee3400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3ec8005270 0x7f3ee40a1760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:53088/0 (socket says 172.21.15.32:53088) 2024-08-03T20:48:10.453 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:10.450+0000 7f3ee3400640 1 -- 172.21.15.32:0/1168149563 learned_addr learned my addr 172.21.15.32:0/1168149563 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:48:10.453 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:10.451+0000 7f3ee3e00640 1 --2- 172.21.15.32:0/1168149563 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f3ee41f7280 0x7f3ee41f7690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:48:10.453 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:10.451+0000 7f3edaa00640 1 --2- 172.21.15.32:0/1168149563 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f3ee41e8ed0 0x7f3ee41e92e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:48:10.453 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:10.451+0000 7f3ee3400640 1 -- 172.21.15.32:0/1168149563 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f3ee41e8ed0 msgr2=0x7f3ee41e92e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:48:10.453 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:10.451+0000 7f3ee3400640 1 --2- 172.21.15.32:0/1168149563 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f3ee41e8ed0 0x7f3ee41e92e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:10.453 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:10.451+0000 7f3ee3400640 1 -- 172.21.15.32:0/1168149563 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f3ee41f7280 msgr2=0x7f3ee41f7690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:48:10.453 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:10.451+0000 7f3ee3400640 1 --2- 172.21.15.32:0/1168149563 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f3ee41f7280 0x7f3ee41f7690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:10.453 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:10.451+0000 7f3ee3400640 1 -- 172.21.15.32:0/1168149563 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3ee409dff0 con 0x7f3ec8005270 2024-08-03T20:48:10.454 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:10.451+0000 7f3ee3400640 1 --2- 172.21.15.32:0/1168149563 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3ec8005270 0x7f3ee40a1760 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f3ed4006b30 tx=0x7f3ed4001780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:48:10.454 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:10.451+0000 7f3edbe00640 1 -- 172.21.15.32:0/1168149563 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ed4015070 con 0x7f3ec8005270 2024-08-03T20:48:10.454 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:10.452+0000 7f3edbe00640 1 -- 172.21.15.32:0/1168149563 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f3ed4011840 con 0x7f3ec8005270 2024-08-03T20:48:10.454 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:10.452+0000 7f3edbe00640 1 -- 172.21.15.32:0/1168149563 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ed4011c90 con 0x7f3ec8005270 2024-08-03T20:48:10.454 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:10.452+0000 7f3ee8e00640 1 -- 172.21.15.32:0/1168149563 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3ee409e2b0 con 0x7f3ec8005270 2024-08-03T20:48:10.454 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:10.452+0000 7f3ee8e00640 1 -- 172.21.15.32:0/1168149563 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3ee409e870 con 0x7f3ec8005270 2024-08-03T20:48:10.455 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:10.453+0000 7f3ee8e00640 1 -- 172.21.15.32:0/1168149563 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3ee40d4a00 con 0x7f3ec8005270 2024-08-03T20:48:10.455 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:10.454+0000 7f3edbe00640 1 -- 172.21.15.32:0/1168149563 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f3ed4018070 con 0x7f3ec8005270 2024-08-03T20:48:10.455 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:10.455+0000 7f3edbe00640 1 --2- 172.21.15.32:0/1168149563 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f3ec0077040 0x7f3ec0079550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:48:10.455 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:10.455+0000 7f3edbe00640 1 -- 172.21.15.32:0/1168149563 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f3ed4099560 con 0x7f3ec8005270 2024-08-03T20:48:10.456 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:10.456+0000 7f3edaa00640 1 --2- 172.21.15.32:0/1168149563 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f3ec0077040 0x7f3ec0079550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:48:10.456 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:10.456+0000 7f3edaa00640 1 --2- 172.21.15.32:0/1168149563 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f3ec0077040 0x7f3ec0079550 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f3ee41f8f50 tx=0x7f3ecc006040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:48:10.459 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:10.459+0000 7f3edbe00640 1 -- 172.21.15.32:0/1168149563 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7f3ee40d4a00 con 0x7f3ec8005270 2024-08-03T20:48:10.677 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:10.676+0000 7f3ee8e00640 1 -- 172.21.15.32:0/1168149563 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f3ee40d1190 con 0x7f3ec8005270 2024-08-03T20:48:10.677 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:10.677+0000 7f3edbe00640 1 -- 172.21.15.32:0/1168149563 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v21) ==== 80+0+0 (secure 0 0 0) 0x7f3ed4062100 con 0x7f3ec8005270 2024-08-03T20:48:10.678 INFO:teuthology.orchestra.run.smithi032.stderr:listed 0 entries 2024-08-03T20:48:10.680 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:10.679+0000 7f3ee8e00640 1 -- 172.21.15.32:0/1168149563 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f3ec0077040 msgr2=0x7f3ec0079550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:48:10.680 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:10.679+0000 7f3ee8e00640 1 --2- 172.21.15.32:0/1168149563 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f3ec0077040 0x7f3ec0079550 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f3ee41f8f50 tx=0x7f3ecc006040 comp rx=0 tx=0).stop 2024-08-03T20:48:10.680 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:10.679+0000 7f3ee8e00640 1 -- 172.21.15.32:0/1168149563 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3ec8005270 msgr2=0x7f3ee40a1760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:48:10.680 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:10.679+0000 7f3ee8e00640 1 --2- 172.21.15.32:0/1168149563 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3ec8005270 0x7f3ee40a1760 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f3ed4006b30 tx=0x7f3ed4001780 comp rx=0 tx=0).stop 2024-08-03T20:48:10.681 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:10.679+0000 7f3ee8e00640 1 -- 172.21.15.32:0/1168149563 shutdown_connections 2024-08-03T20:48:10.681 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:10.680+0000 7f3ee8e00640 1 --2- 172.21.15.32:0/1168149563 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f3ec0077040 0x7f3ec0079550 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:10.681 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:10.680+0000 7f3ee8e00640 1 --2- 172.21.15.32:0/1168149563 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f3ee41f7280 0x7f3ee41f7690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:10.681 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:10.680+0000 7f3ee8e00640 1 --2- 172.21.15.32:0/1168149563 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f3ee41e8ed0 0x7f3ee41e92e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:10.681 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:10.680+0000 7f3ee8e00640 1 --2- 172.21.15.32:0/1168149563 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3ec8005270 0x7f3ee40a1760 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:10.681 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:10.680+0000 7f3ee8e00640 1 -- 172.21.15.32:0/1168149563 >> 172.21.15.32:0/1168149563 conn(0x7f3ee40d40c0 msgr2=0x7f3ee415e620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:48:10.681 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:10.680+0000 7f3ee8e00640 1 -- 172.21.15.32:0/1168149563 shutdown_connections 2024-08-03T20:48:10.681 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:10.680+0000 7f3ee8e00640 1 -- 172.21.15.32:0/1168149563 wait complete. 2024-08-03T20:48:10.703 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.1... 2024-08-03T20:48:10.704 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-08-03T20:48:10.704 DEBUG:teuthology.orchestra.run.smithi118:> sudo umount /home/ubuntu/cephtest/mnt.1 -f 2024-08-03T20:48:10.771 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi118.front.sepia.ceph.com 2024-08-03T20:48:10.771 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-08-03T20:48:10.771 DEBUG:teuthology.orchestra.run.smithi118:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2024-08-03T20:48:10.821 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-08-03T20:48:10.821 DEBUG:teuthology.orchestra.run.smithi032:> dd if=/proc/self/mounts of=/dev/stdout 2024-08-03T20:48:10.857 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-08-03T20:48:10.857 DEBUG:teuthology.orchestra.run.smithi032:> dd if=/proc/self/mounts of=/dev/stdout 2024-08-03T20:48:10.932 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-08-03T20:48:10.933 DEBUG:teuthology.orchestra.run.smithi032:> dd if=/proc/self/mounts of=/dev/stdout 2024-08-03T20:48:11.008 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-08-03T20:48:11.276 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:11.276+0000 7f4176c00640 1 -- 172.21.15.32:0/2061447966 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4170165160 msgr2=0x7f4170167580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:48:11.277 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:11.276+0000 7f4176c00640 1 --2- 172.21.15.32:0/2061447966 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4170165160 0x7f4170167580 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f4168005fd0 tx=0x7f416802a8b0 comp rx=0 tx=0).stop 2024-08-03T20:48:11.277 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:11.276+0000 7f4176c00640 1 -- 172.21.15.32:0/2061447966 shutdown_connections 2024-08-03T20:48:11.277 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:11.276+0000 7f4176c00640 1 --2- 172.21.15.32:0/2061447966 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4170165160 0x7f4170167580 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:11.277 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:11.276+0000 7f4176c00640 1 -- 172.21.15.32:0/2061447966 >> 172.21.15.32:0/2061447966 conn(0x7f41700b1bd0 msgr2=0x7f41700b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:48:11.277 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:11.276+0000 7f4176c00640 1 -- 172.21.15.32:0/2061447966 shutdown_connections 2024-08-03T20:48:11.277 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:11.276+0000 7f4176c00640 1 -- 172.21.15.32:0/2061447966 wait complete. 2024-08-03T20:48:11.278 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:11.278+0000 7f4176c00640 1 Processor -- start 2024-08-03T20:48:11.278 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:11.278+0000 7f4176c00640 1 -- start start 2024-08-03T20:48:11.284 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:11.279+0000 7f4176c00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f417006e930 0x7f4170155900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:48:11.284 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:11.279+0000 7f4176c00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4170144f90 0x7f4170155e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:48:11.284 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:11.279+0000 7f4176c00640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f4170165160 0x7f41701380c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:48:11.284 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:11.279+0000 7f4176c00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f41700cf9c0 con 0x7f4170144f90 2024-08-03T20:48:11.284 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:11.279+0000 7f4176c00640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f41700cf840 con 0x7f417006e930 2024-08-03T20:48:11.284 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:11.279+0000 7f4176c00640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7f4170167ac0 con 0x7f4170165160 2024-08-03T20:48:11.284 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:11.280+0000 7f416f400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4170144f90 0x7f4170155e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:48:11.284 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:11.280+0000 7f416f400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4170144f90 0x7f4170155e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:53110/0 (socket says 172.21.15.32:53110) 2024-08-03T20:48:11.284 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:11.280+0000 7f416f400640 1 -- 172.21.15.32:0/4207460333 learned_addr learned my addr 172.21.15.32:0/4207460333 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:48:11.284 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:11.280+0000 7f416fe00640 1 --2- 172.21.15.32:0/4207460333 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f417006e930 0x7f4170155900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:48:11.285 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:11.280+0000 7f4174c00640 1 --2- 172.21.15.32:0/4207460333 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f4170165160 0x7f41701380c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:48:11.285 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:11.280+0000 7f416f400640 1 -- 172.21.15.32:0/4207460333 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f4170165160 msgr2=0x7f41701380c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:48:11.285 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:11.280+0000 7f416f400640 1 --2- 172.21.15.32:0/4207460333 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f4170165160 0x7f41701380c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:11.285 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:11.280+0000 7f416f400640 1 -- 172.21.15.32:0/4207460333 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f417006e930 msgr2=0x7f4170155900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:48:11.285 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:11.280+0000 7f416f400640 1 --2- 172.21.15.32:0/4207460333 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f417006e930 0x7f4170155900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:11.285 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:11.280+0000 7f416f400640 1 -- 172.21.15.32:0/4207460333 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f417015a930 con 0x7f4170144f90 2024-08-03T20:48:11.285 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:11.280+0000 7f4174c00640 1 --2- 172.21.15.32:0/4207460333 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f4170165160 0x7f41701380c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-03T20:48:11.285 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:11.280+0000 7f416f400640 1 --2- 172.21.15.32:0/4207460333 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4170144f90 0x7f4170155e40 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f415c00c3d0 tx=0x7f415c00c890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:48:11.285 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:11.281+0000 7f4167e00640 1 -- 172.21.15.32:0/4207460333 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f415c011790 con 0x7f4170144f90 2024-08-03T20:48:11.285 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:11.281+0000 7f4167e00640 1 -- 172.21.15.32:0/4207460333 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f415c017070 con 0x7f4170144f90 2024-08-03T20:48:11.285 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:11.281+0000 7f4176c00640 1 -- 172.21.15.32:0/4207460333 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f417015a650 con 0x7f4170144f90 2024-08-03T20:48:11.285 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:11.281+0000 7f4167e00640 1 -- 172.21.15.32:0/4207460333 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f415c011ca0 con 0x7f4170144f90 2024-08-03T20:48:11.285 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:11.281+0000 7f4176c00640 1 -- 172.21.15.32:0/4207460333 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f41701378c0 con 0x7f4170144f90 2024-08-03T20:48:11.286 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:11.282+0000 7f4176c00640 1 -- 172.21.15.32:0/4207460333 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4170073e60 con 0x7f4170144f90 2024-08-03T20:48:11.286 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:11.283+0000 7f4167e00640 1 -- 172.21.15.32:0/4207460333 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f415c01c460 con 0x7f4170144f90 2024-08-03T20:48:11.286 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:11.284+0000 7f4167e00640 1 --2- 172.21.15.32:0/4207460333 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f4150077010 0x7f4150079520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:48:11.287 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:11.284+0000 7f4167e00640 1 -- 172.21.15.32:0/4207460333 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f415c098760 con 0x7f4170144f90 2024-08-03T20:48:11.287 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:11.285+0000 7f416fe00640 1 --2- 172.21.15.32:0/4207460333 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f4150077010 0x7f4150079520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:48:11.287 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:11.285+0000 7f416fe00640 1 --2- 172.21.15.32:0/4207460333 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f4150077010 0x7f4150079520 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f4160001720 tx=0x7f4160009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:48:11.289 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:11.288+0000 7f4167e00640 1 -- 172.21.15.32:0/4207460333 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7f415c0618c0 con 0x7f4170144f90 2024-08-03T20:48:11.504 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:11.504+0000 7f4176c00640 1 -- 172.21.15.32:0/4207460333 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f4170160de0 con 0x7f4170144f90 2024-08-03T20:48:11.505 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:11.504+0000 7f4167e00640 1 -- 172.21.15.32:0/4207460333 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v21) ==== 80+0+0 (secure 0 0 0) 0x7f415c0611c0 con 0x7f4170144f90 2024-08-03T20:48:11.505 INFO:teuthology.orchestra.run.smithi032.stderr:listed 0 entries 2024-08-03T20:48:11.507 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:11.507+0000 7f4176c00640 1 -- 172.21.15.32:0/4207460333 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f4150077010 msgr2=0x7f4150079520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:48:11.508 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:11.507+0000 7f4176c00640 1 --2- 172.21.15.32:0/4207460333 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f4150077010 0x7f4150079520 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f4160001720 tx=0x7f4160009040 comp rx=0 tx=0).stop 2024-08-03T20:48:11.508 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:11.507+0000 7f4176c00640 1 -- 172.21.15.32:0/4207460333 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4170144f90 msgr2=0x7f4170155e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:48:11.508 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:11.507+0000 7f4176c00640 1 --2- 172.21.15.32:0/4207460333 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4170144f90 0x7f4170155e40 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f415c00c3d0 tx=0x7f415c00c890 comp rx=0 tx=0).stop 2024-08-03T20:48:11.508 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:11.507+0000 7f4176c00640 1 -- 172.21.15.32:0/4207460333 shutdown_connections 2024-08-03T20:48:11.508 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:11.507+0000 7f4176c00640 1 --2- 172.21.15.32:0/4207460333 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f4150077010 0x7f4150079520 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:11.508 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:11.507+0000 7f4176c00640 1 --2- 172.21.15.32:0/4207460333 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f4170165160 0x7f41701380c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:11.508 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:11.507+0000 7f4176c00640 1 --2- 172.21.15.32:0/4207460333 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4170144f90 0x7f4170155e40 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:11.508 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:11.507+0000 7f4176c00640 1 --2- 172.21.15.32:0/4207460333 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f417006e930 0x7f4170155900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:11.508 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:11.507+0000 7f4176c00640 1 -- 172.21.15.32:0/4207460333 >> 172.21.15.32:0/4207460333 conn(0x7f41700b1bd0 msgr2=0x7f417013ce30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:48:11.508 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:11.507+0000 7f4176c00640 1 -- 172.21.15.32:0/4207460333 shutdown_connections 2024-08-03T20:48:11.508 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:11.507+0000 7f4176c00640 1 -- 172.21.15.32:0/4207460333 wait complete. 2024-08-03T20:48:11.529 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.2... 2024-08-03T20:48:11.529 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-08-03T20:48:11.530 DEBUG:teuthology.orchestra.run.smithi032:> sudo umount /home/ubuntu/cephtest/mnt.2 -f 2024-08-03T20:48:11.589 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi032.front.sepia.ceph.com 2024-08-03T20:48:11.590 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-08-03T20:48:11.590 DEBUG:teuthology.orchestra.run.smithi032:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.2) 2024-08-03T20:48:11.626 DEBUG:teuthology.orchestra.run.smithi118:> set -ex 2024-08-03T20:48:11.626 DEBUG:teuthology.orchestra.run.smithi118:> dd if=/proc/self/mounts of=/dev/stdout 2024-08-03T20:48:11.666 DEBUG:teuthology.orchestra.run.smithi118:> set -ex 2024-08-03T20:48:11.666 DEBUG:teuthology.orchestra.run.smithi118:> dd if=/proc/self/mounts of=/dev/stdout 2024-08-03T20:48:11.746 DEBUG:teuthology.orchestra.run.smithi118:> set -ex 2024-08-03T20:48:11.746 DEBUG:teuthology.orchestra.run.smithi118:> dd if=/proc/self/mounts of=/dev/stdout 2024-08-03T20:48:11.824 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-08-03T20:48:12.055 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:12.054+0000 7f9991000640 1 -- 172.21.15.32:0/1354207853 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f998c0cf510 msgr2=0x7f998c0cf920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:48:12.056 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:12.054+0000 7f9991000640 1 --2- 172.21.15.32:0/1354207853 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f998c0cf510 0x7f998c0cf920 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f9984005fd0 tx=0x7f998402a940 comp rx=0 tx=0).stop 2024-08-03T20:48:12.056 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:12.055+0000 7f9991000640 1 -- 172.21.15.32:0/1354207853 shutdown_connections 2024-08-03T20:48:12.056 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:12.055+0000 7f9991000640 1 --2- 172.21.15.32:0/1354207853 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f998c0cf510 0x7f998c0cf920 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:12.056 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:12.055+0000 7f9991000640 1 -- 172.21.15.32:0/1354207853 >> 172.21.15.32:0/1354207853 conn(0x7f998c08d7a0 msgr2=0x7f998c09d3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:48:12.056 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:12.055+0000 7f9991000640 1 -- 172.21.15.32:0/1354207853 shutdown_connections 2024-08-03T20:48:12.056 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:12.055+0000 7f9991000640 1 -- 172.21.15.32:0/1354207853 wait complete. 2024-08-03T20:48:12.057 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:12.057+0000 7f9991000640 1 Processor -- start 2024-08-03T20:48:12.057 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:12.057+0000 7f9991000640 1 -- start start 2024-08-03T20:48:12.061 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:12.059+0000 7f9991000640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f998c0cf510 0x7f998c09ee30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:48:12.061 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:12.059+0000 7f9991000640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f998c0cffe0 0x7f998c14bf70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:48:12.061 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:12.059+0000 7f9991000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f998c0d26e0 0x7f998c0b8740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:48:12.061 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:12.059+0000 7f9991000640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f998c0d4310 con 0x7f998c0d26e0 2024-08-03T20:48:12.061 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:12.059+0000 7f9991000640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f998c0d4190 con 0x7f998c0cffe0 2024-08-03T20:48:12.061 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:12.059+0000 7f9991000640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7f998c0d4490 con 0x7f998c0cf510 2024-08-03T20:48:12.061 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:12.059+0000 7f998be00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f998c0d26e0 0x7f998c0b8740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:48:12.061 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:12.059+0000 7f998be00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f998c0d26e0 0x7f998c0b8740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:53116/0 (socket says 172.21.15.32:53116) 2024-08-03T20:48:12.062 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:12.059+0000 7f998be00640 1 -- 172.21.15.32:0/1087989698 learned_addr learned my addr 172.21.15.32:0/1087989698 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:48:12.062 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:12.059+0000 7f998b400640 1 --2- 172.21.15.32:0/1087989698 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f998c0cf510 0x7f998c09ee30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:48:12.062 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:12.059+0000 7f998aa00640 1 --2- 172.21.15.32:0/1087989698 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f998c0cffe0 0x7f998c14bf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:48:12.062 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:12.060+0000 7f998be00640 1 -- 172.21.15.32:0/1087989698 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f998c0cf510 msgr2=0x7f998c09ee30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:48:12.062 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:12.060+0000 7f998be00640 1 --2- 172.21.15.32:0/1087989698 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f998c0cf510 0x7f998c09ee30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:12.062 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:12.060+0000 7f998be00640 1 -- 172.21.15.32:0/1087989698 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f998c0cffe0 msgr2=0x7f998c14bf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:48:12.062 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:12.060+0000 7f998be00640 1 --2- 172.21.15.32:0/1087989698 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f998c0cffe0 0x7f998c14bf70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:12.062 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:12.060+0000 7f998be00640 1 -- 172.21.15.32:0/1087989698 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f998c15b310 con 0x7f998c0d26e0 2024-08-03T20:48:12.062 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:12.060+0000 7f998be00640 1 --2- 172.21.15.32:0/1087989698 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f998c0d26e0 0x7f998c0b8740 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f9984009cd0 tx=0x7f9984031e20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:48:12.062 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:12.060+0000 7f9983400640 1 -- 172.21.15.32:0/1087989698 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f998403e070 con 0x7f998c0d26e0 2024-08-03T20:48:12.062 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:12.060+0000 7f9983400640 1 -- 172.21.15.32:0/1087989698 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f9984004540 con 0x7f998c0d26e0 2024-08-03T20:48:12.063 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:12.060+0000 7f9983400640 1 -- 172.21.15.32:0/1087989698 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9984030430 con 0x7f998c0d26e0 2024-08-03T20:48:12.063 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:12.060+0000 7f9991000640 1 -- 172.21.15.32:0/1087989698 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f998c09f370 con 0x7f998c0d26e0 2024-08-03T20:48:12.063 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:12.060+0000 7f9991000640 1 -- 172.21.15.32:0/1087989698 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f998c14ec80 con 0x7f998c0d26e0 2024-08-03T20:48:12.063 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:12.062+0000 7f9991000640 1 -- 172.21.15.32:0/1087989698 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f998c15b4e0 con 0x7f998c0d26e0 2024-08-03T20:48:12.063 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:12.063+0000 7f9983400640 1 -- 172.21.15.32:0/1087989698 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f9984043460 con 0x7f998c0d26e0 2024-08-03T20:48:12.064 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:12.064+0000 7f9983400640 1 --2- 172.21.15.32:0/1087989698 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f996c077040 0x7f996c079550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:48:12.064 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:12.064+0000 7f9983400640 1 -- 172.21.15.32:0/1087989698 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f99840ba1f0 con 0x7f998c0d26e0 2024-08-03T20:48:12.065 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:12.064+0000 7f998b400640 1 --2- 172.21.15.32:0/1087989698 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f996c077040 0x7f996c079550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:48:12.065 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:12.065+0000 7f998b400640 1 --2- 172.21.15.32:0/1087989698 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f996c077040 0x7f996c079550 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f997c000f60 tx=0x7f997c009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:48:12.068 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:12.068+0000 7f9983400640 1 -- 172.21.15.32:0/1087989698 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7f9984082d90 con 0x7f998c0d26e0 2024-08-03T20:48:12.289 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:12.288+0000 7f9991000640 1 -- 172.21.15.32:0/1087989698 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f998c0d4940 con 0x7f998c0d26e0 2024-08-03T20:48:12.289 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:12.289+0000 7f9983400640 1 -- 172.21.15.32:0/1087989698 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v21) ==== 80+0+0 (secure 0 0 0) 0x7f99840872e0 con 0x7f998c0d26e0 2024-08-03T20:48:12.289 INFO:teuthology.orchestra.run.smithi032.stderr:listed 0 entries 2024-08-03T20:48:12.292 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:12.291+0000 7f9991000640 1 -- 172.21.15.32:0/1087989698 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f996c077040 msgr2=0x7f996c079550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:48:12.292 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:12.291+0000 7f9991000640 1 --2- 172.21.15.32:0/1087989698 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f996c077040 0x7f996c079550 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f997c000f60 tx=0x7f997c009040 comp rx=0 tx=0).stop 2024-08-03T20:48:12.292 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:12.291+0000 7f9991000640 1 -- 172.21.15.32:0/1087989698 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f998c0d26e0 msgr2=0x7f998c0b8740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:48:12.292 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:12.291+0000 7f9991000640 1 --2- 172.21.15.32:0/1087989698 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f998c0d26e0 0x7f998c0b8740 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f9984009cd0 tx=0x7f9984031e20 comp rx=0 tx=0).stop 2024-08-03T20:48:12.292 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:12.291+0000 7f9991000640 1 -- 172.21.15.32:0/1087989698 shutdown_connections 2024-08-03T20:48:12.292 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:12.291+0000 7f9991000640 1 --2- 172.21.15.32:0/1087989698 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f996c077040 0x7f996c079550 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:12.292 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:12.291+0000 7f9991000640 1 --2- 172.21.15.32:0/1087989698 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f998c0cffe0 0x7f998c14bf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:12.293 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:12.291+0000 7f9991000640 1 --2- 172.21.15.32:0/1087989698 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f998c0d26e0 0x7f998c0b8740 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:12.293 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:12.291+0000 7f9991000640 1 --2- 172.21.15.32:0/1087989698 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f998c0cf510 0x7f998c09ee30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:12.293 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:12.291+0000 7f9991000640 1 -- 172.21.15.32:0/1087989698 >> 172.21.15.32:0/1087989698 conn(0x7f998c08d7a0 msgr2=0x7f998c06d9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:48:12.293 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:12.291+0000 7f9991000640 1 -- 172.21.15.32:0/1087989698 shutdown_connections 2024-08-03T20:48:12.293 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:12.291+0000 7f9991000640 1 -- 172.21.15.32:0/1087989698 wait complete. 2024-08-03T20:48:12.313 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.3... 2024-08-03T20:48:12.313 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-08-03T20:48:12.313 DEBUG:teuthology.orchestra.run.smithi118:> sudo umount /home/ubuntu/cephtest/mnt.3 -f 2024-08-03T20:48:12.379 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi118.front.sepia.ceph.com 2024-08-03T20:48:12.379 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-08-03T20:48:12.379 DEBUG:teuthology.orchestra.run.smithi118:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.3) 2024-08-03T20:48:12.417 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-08-03T20:48:12.417 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-08-03T20:48:12.644 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:12.643+0000 7f3f41200640 1 -- 172.21.15.32:0/1790001405 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3f3c06e930 msgr2=0x7f3f3c144f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:48:12.644 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:12.643+0000 7f3f41200640 1 --2- 172.21.15.32:0/1790001405 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3f3c06e930 0x7f3f3c144f90 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f3f3c0afaf0 tx=0x7f3f2c02a4d0 comp rx=0 tx=0).stop 2024-08-03T20:48:12.644 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:12.643+0000 7f3f41200640 1 -- 172.21.15.32:0/1790001405 shutdown_connections 2024-08-03T20:48:12.644 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:12.643+0000 7f3f41200640 1 --2- 172.21.15.32:0/1790001405 >> v2:172.21.15.118:3300/0 conn(0x7f3f3c073e60 0x7f3f3c074270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:12.644 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:12.643+0000 7f3f41200640 1 --2- 172.21.15.32:0/1790001405 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3f3c06e930 0x7f3f3c144f90 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:12.644 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:12.643+0000 7f3f41200640 1 -- 172.21.15.32:0/1790001405 >> 172.21.15.32:0/1790001405 conn(0x7f3f3c0b1bd0 msgr2=0x7f3f3c0b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:48:12.645 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:12.643+0000 7f3f41200640 1 -- 172.21.15.32:0/1790001405 shutdown_connections 2024-08-03T20:48:12.645 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:12.643+0000 7f3f41200640 1 -- 172.21.15.32:0/1790001405 wait complete. 2024-08-03T20:48:12.646 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:12.645+0000 7f3f41200640 1 Processor -- start 2024-08-03T20:48:12.646 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:12.646+0000 7f3f41200640 1 -- start start 2024-08-03T20:48:12.649 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:12.647+0000 7f3f41200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3f3c06e930 0x7f3f3c1f4e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:48:12.649 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:12.647+0000 7f3f41200640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f3f3c073e60 0x7f3f3c0b4570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:48:12.649 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:12.647+0000 7f3f41200640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f3f3c0747b0 0x7f3f3c150cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:48:12.649 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:12.647+0000 7f3f41200640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f3f3c0cf9c0 con 0x7f3f3c06e930 2024-08-03T20:48:12.649 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:12.647+0000 7f3f41200640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f3f3c0cf840 con 0x7f3f3c0747b0 2024-08-03T20:48:12.649 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:12.647+0000 7f3f41200640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7f3f3c0ccff0 con 0x7f3f3c073e60 2024-08-03T20:48:12.649 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:12.647+0000 7f3f39e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3f3c06e930 0x7f3f3c1f4e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:48:12.649 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:12.647+0000 7f3f39e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3f3c06e930 0x7f3f3c1f4e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:53126/0 (socket says 172.21.15.32:53126) 2024-08-03T20:48:12.649 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:12.647+0000 7f3f39e00640 1 -- 172.21.15.32:0/1666689235 learned_addr learned my addr 172.21.15.32:0/1666689235 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:48:12.649 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:12.648+0000 7f3f3a800640 1 --2- 172.21.15.32:0/1666689235 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f3f3c0747b0 0x7f3f3c150cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:48:12.649 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:12.648+0000 7f3f33e00640 1 --2- 172.21.15.32:0/1666689235 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f3f3c073e60 0x7f3f3c0b4570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:48:12.649 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:12.648+0000 7f3f39e00640 1 -- 172.21.15.32:0/1666689235 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f3f3c073e60 msgr2=0x7f3f3c0b4570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:48:12.650 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:12.648+0000 7f3f39e00640 1 --2- 172.21.15.32:0/1666689235 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f3f3c073e60 0x7f3f3c0b4570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:12.650 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:12.648+0000 7f3f39e00640 1 -- 172.21.15.32:0/1666689235 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f3f3c0747b0 msgr2=0x7f3f3c150cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:48:12.650 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:12.648+0000 7f3f39e00640 1 --2- 172.21.15.32:0/1666689235 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f3f3c0747b0 0x7f3f3c150cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:12.650 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:12.648+0000 7f3f39e00640 1 -- 172.21.15.32:0/1666689235 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3f3c13cdc0 con 0x7f3f3c06e930 2024-08-03T20:48:12.650 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:12.648+0000 7f3f39e00640 1 --2- 172.21.15.32:0/1666689235 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3f3c06e930 0x7f3f3c1f4e90 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f3f2c034040 tx=0x7f3f2c003910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:48:12.650 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:12.648+0000 7f3f32000640 1 -- 172.21.15.32:0/1666689235 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f2c03e030 con 0x7f3f3c06e930 2024-08-03T20:48:12.650 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:12.648+0000 7f3f32000640 1 -- 172.21.15.32:0/1666689235 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f3f2c003e00 con 0x7f3f3c06e930 2024-08-03T20:48:12.650 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:12.648+0000 7f3f32000640 1 -- 172.21.15.32:0/1666689235 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f2c035040 con 0x7f3f3c06e930 2024-08-03T20:48:12.650 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:12.648+0000 7f3f41200640 1 -- 172.21.15.32:0/1666689235 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3f3c13d0a0 con 0x7f3f3c06e930 2024-08-03T20:48:12.650 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:12.648+0000 7f3f41200640 1 -- 172.21.15.32:0/1666689235 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3f3c15bf70 con 0x7f3f3c06e930 2024-08-03T20:48:12.651 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:12.649+0000 7f3f41200640 1 -- 172.21.15.32:0/1666689235 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3f3c0cd5c0 con 0x7f3f3c06e930 2024-08-03T20:48:12.651 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:12.651+0000 7f3f32000640 1 -- 172.21.15.32:0/1666689235 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f3f2c041030 con 0x7f3f3c06e930 2024-08-03T20:48:12.652 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:12.651+0000 7f3f32000640 1 --2- 172.21.15.32:0/1666689235 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f3f18077010 0x7f3f18079520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:48:12.652 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:12.652+0000 7f3f32000640 1 -- 172.21.15.32:0/1666689235 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f3f2c0b91d0 con 0x7f3f3c06e930 2024-08-03T20:48:12.652 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:12.652+0000 7f3f33e00640 1 --2- 172.21.15.32:0/1666689235 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f3f18077010 0x7f3f18079520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:48:12.653 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:12.653+0000 7f3f33e00640 1 --2- 172.21.15.32:0/1666689235 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f3f18077010 0x7f3f18079520 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f3f24005fd0 tx=0x7f3f24005850 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:48:12.657 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:12.656+0000 7f3f32000640 1 -- 172.21.15.32:0/1666689235 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7f3f2c082330 con 0x7f3f3c06e930 2024-08-03T20:48:12.898 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:12.898+0000 7f3f41200640 1 -- 172.21.15.32:0/1666689235 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f3f3c0b2180 con 0x7f3f3c06e930 2024-08-03T20:48:12.899 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:12.898+0000 7f3f32000640 1 -- 172.21.15.32:0/1666689235 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) ==== 93+0+5220 (secure 0 0 0) 0x7f3f2c081c30 con 0x7f3f3c06e930 2024-08-03T20:48:12.899 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-03T20:48:12.899 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":8,"btime":"2024-08-03T20:47:38:939479+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4289,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6835/1753256117","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6834","nonce":1753256117},{"type":"v1","addr":"172.21.15.118:6835","nonce":1753256117}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4292,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6837/3055588578","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6836","nonce":3055588578},{"type":"v1","addr":"172.21.15.118:6837","nonce":3055588578}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4338,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6834/3026095815","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":3026095815},{"type":"v1","addr":"172.21.15.32:6834","nonce":3026095815}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-08-03T20:47:35.442482+0000","modified":"2024-08-03T20:47:38.460928+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4341},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4341":{"gid":4341,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.32:6835/246278108","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6833","nonce":246278108},{"type":"v1","addr":"172.21.15.32:6835","nonce":246278108}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4341,"qdb_cluster":[4341]},"id":1}]} 2024-08-03T20:48:12.899 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 8 2024-08-03T20:48:12.901 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:12.901+0000 7f3f41200640 1 -- 172.21.15.32:0/1666689235 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f3f18077010 msgr2=0x7f3f18079520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:48:12.901 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:12.901+0000 7f3f41200640 1 --2- 172.21.15.32:0/1666689235 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f3f18077010 0x7f3f18079520 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f3f24005fd0 tx=0x7f3f24005850 comp rx=0 tx=0).stop 2024-08-03T20:48:12.901 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:12.901+0000 7f3f41200640 1 -- 172.21.15.32:0/1666689235 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3f3c06e930 msgr2=0x7f3f3c1f4e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:48:12.902 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:12.901+0000 7f3f41200640 1 --2- 172.21.15.32:0/1666689235 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3f3c06e930 0x7f3f3c1f4e90 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f3f2c034040 tx=0x7f3f2c003910 comp rx=0 tx=0).stop 2024-08-03T20:48:12.902 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:12.901+0000 7f3f41200640 1 -- 172.21.15.32:0/1666689235 shutdown_connections 2024-08-03T20:48:12.902 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:12.901+0000 7f3f41200640 1 --2- 172.21.15.32:0/1666689235 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f3f18077010 0x7f3f18079520 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:12.902 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:12.901+0000 7f3f41200640 1 --2- 172.21.15.32:0/1666689235 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f3f3c0747b0 0x7f3f3c150cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:12.902 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:12.901+0000 7f3f41200640 1 --2- 172.21.15.32:0/1666689235 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f3f3c073e60 0x7f3f3c0b4570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:12.902 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:12.901+0000 7f3f41200640 1 --2- 172.21.15.32:0/1666689235 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3f3c06e930 0x7f3f3c1f4e90 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:12.902 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:12.901+0000 7f3f41200640 1 -- 172.21.15.32:0/1666689235 >> 172.21.15.32:0/1666689235 conn(0x7f3f3c0b1bd0 msgr2=0x7f3f3c06ef70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:48:12.902 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:12.901+0000 7f3f41200640 1 -- 172.21.15.32:0/1666689235 shutdown_connections 2024-08-03T20:48:12.902 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:12.901+0000 7f3f41200640 1 -- 172.21.15.32:0/1666689235 wait complete. 2024-08-03T20:48:12.924 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-08-03T20:48:13.155 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:13.155+0000 7f638f200640 1 -- 172.21.15.32:0/3556407891 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f63900d26e0 msgr2=0x7f6390161b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:48:13.155 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:13.155+0000 7f638f200640 1 --2- 172.21.15.32:0/3556407891 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f63900d26e0 0x7f6390161b80 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f63900d1b20 tx=0x7f638002a5f0 comp rx=0 tx=0).stop 2024-08-03T20:48:13.156 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:13.155+0000 7f638f200640 1 -- 172.21.15.32:0/3556407891 shutdown_connections 2024-08-03T20:48:13.156 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:13.155+0000 7f638f200640 1 --2- 172.21.15.32:0/3556407891 >> v2:172.21.15.118:3300/0 conn(0x7f63900cf510 0x7f63900cf920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:13.156 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:13.155+0000 7f638f200640 1 --2- 172.21.15.32:0/3556407891 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f63900d26e0 0x7f6390161b80 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:13.156 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:13.155+0000 7f638f200640 1 -- 172.21.15.32:0/3556407891 >> 172.21.15.32:0/3556407891 conn(0x7f639008d7a0 msgr2=0x7f639009d3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:48:13.156 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:13.155+0000 7f638f200640 1 -- 172.21.15.32:0/3556407891 shutdown_connections 2024-08-03T20:48:13.156 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:13.155+0000 7f638f200640 1 -- 172.21.15.32:0/3556407891 wait complete. 2024-08-03T20:48:13.157 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:13.157+0000 7f638f200640 1 Processor -- start 2024-08-03T20:48:13.157 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:13.157+0000 7f638f200640 1 -- start start 2024-08-03T20:48:13.160 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:13.159+0000 7f638f200640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f63900cf510 0x7f63900cdf40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:48:13.160 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:13.159+0000 7f638f200640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f63900cffe0 0x7f63901525d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:48:13.160 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:13.159+0000 7f638f200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f63900d26e0 0x7f63900b8200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:48:13.160 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:13.159+0000 7f638f200640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f63900d4100 con 0x7f63900d26e0 2024-08-03T20:48:13.160 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:13.159+0000 7f638f200640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f63900d3f80 con 0x7f63900cf510 2024-08-03T20:48:13.160 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:13.159+0000 7f638f200640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7f63900d4280 con 0x7f63900cffe0 2024-08-03T20:48:13.160 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:13.159+0000 7f638e800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f63900d26e0 0x7f63900b8200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:48:13.160 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:13.159+0000 7f638e800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f63900d26e0 0x7f63900b8200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:53146/0 (socket says 172.21.15.32:53146) 2024-08-03T20:48:13.160 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:13.159+0000 7f638e800640 1 -- 172.21.15.32:0/2151636404 learned_addr learned my addr 172.21.15.32:0/2151636404 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:48:13.160 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:13.159+0000 7f638de00640 1 --2- 172.21.15.32:0/2151636404 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f63900cf510 0x7f63900cdf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:48:13.161 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:13.159+0000 7f6387e00640 1 --2- 172.21.15.32:0/2151636404 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f63900cffe0 0x7f63901525d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:48:13.161 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:13.159+0000 7f638e800640 1 -- 172.21.15.32:0/2151636404 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f63900cffe0 msgr2=0x7f63901525d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:48:13.161 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:13.159+0000 7f638e800640 1 --2- 172.21.15.32:0/2151636404 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f63900cffe0 0x7f63901525d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:13.161 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:13.159+0000 7f638e800640 1 -- 172.21.15.32:0/2151636404 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f63900cf510 msgr2=0x7f63900cdf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:48:13.161 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:13.159+0000 7f638e800640 1 --2- 172.21.15.32:0/2151636404 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f63900cf510 0x7f63900cdf40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:13.161 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:13.159+0000 7f638e800640 1 -- 172.21.15.32:0/2151636404 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f63900cb1b0 con 0x7f63900d26e0 2024-08-03T20:48:13.161 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:13.159+0000 7f638de00640 1 --2- 172.21.15.32:0/2151636404 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f63900cf510 0x7f63900cdf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-03T20:48:13.161 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:13.160+0000 7f638e800640 1 --2- 172.21.15.32:0/2151636404 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f63900d26e0 0x7f63900b8200 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f638800e960 tx=0x7f638800ee20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:48:13.162 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:13.160+0000 7f6386000640 1 -- 172.21.15.32:0/2151636404 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6388004d20 con 0x7f63900d26e0 2024-08-03T20:48:13.162 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:13.160+0000 7f6386000640 1 -- 172.21.15.32:0/2151636404 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f638800cac0 con 0x7f63900d26e0 2024-08-03T20:48:13.162 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:13.160+0000 7f6386000640 1 -- 172.21.15.32:0/2151636404 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6388005020 con 0x7f63900d26e0 2024-08-03T20:48:13.162 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:13.160+0000 7f638f200640 1 -- 172.21.15.32:0/2151636404 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f63900cb6b0 con 0x7f63900d26e0 2024-08-03T20:48:13.162 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:13.160+0000 7f638f200640 1 -- 172.21.15.32:0/2151636404 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f63901f8ff0 con 0x7f63900d26e0 2024-08-03T20:48:13.162 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:13.161+0000 7f638f200640 1 -- 172.21.15.32:0/2151636404 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f63900cb420 con 0x7f63900d26e0 2024-08-03T20:48:13.163 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:13.162+0000 7f6386000640 1 -- 172.21.15.32:0/2151636404 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f63880051c0 con 0x7f63900d26e0 2024-08-03T20:48:13.164 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:13.164+0000 7f6386000640 1 --2- 172.21.15.32:0/2151636404 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f636c077010 0x7f636c079520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:48:13.164 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:13.164+0000 7f6386000640 1 -- 172.21.15.32:0/2151636404 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f63880991a0 con 0x7f63900d26e0 2024-08-03T20:48:13.165 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:13.165+0000 7f638de00640 1 --2- 172.21.15.32:0/2151636404 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f636c077010 0x7f636c079520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:48:13.166 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:13.166+0000 7f638de00640 1 --2- 172.21.15.32:0/2151636404 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f636c077010 0x7f636c079520 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f6380033000 tx=0x7f6380004730 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:48:13.169 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:13.169+0000 7f6386000640 1 -- 172.21.15.32:0/2151636404 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7f6388016030 con 0x7f63900d26e0 2024-08-03T20:48:13.412 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:13.411+0000 7f638f200640 1 -- 172.21.15.32:0/2151636404 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f63900d4750 con 0x7f63900d26e0 2024-08-03T20:48:13.412 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:13.412+0000 7f6386000640 1 -- 172.21.15.32:0/2151636404 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) ==== 93+0+5220 (secure 0 0 0) 0x7f6388062310 con 0x7f63900d26e0 2024-08-03T20:48:13.412 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-03T20:48:13.413 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":8,"btime":"2024-08-03T20:47:38:939479+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4289,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6835/1753256117","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6834","nonce":1753256117},{"type":"v1","addr":"172.21.15.118:6835","nonce":1753256117}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4292,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6837/3055588578","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6836","nonce":3055588578},{"type":"v1","addr":"172.21.15.118:6837","nonce":3055588578}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4338,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6834/3026095815","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":3026095815},{"type":"v1","addr":"172.21.15.32:6834","nonce":3026095815}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-08-03T20:47:35.442482+0000","modified":"2024-08-03T20:47:38.460928+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4341},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4341":{"gid":4341,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.32:6835/246278108","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6833","nonce":246278108},{"type":"v1","addr":"172.21.15.32:6835","nonce":246278108}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4341,"qdb_cluster":[4341]},"id":1}]} 2024-08-03T20:48:13.413 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 8 2024-08-03T20:48:13.415 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:13.414+0000 7f638f200640 1 -- 172.21.15.32:0/2151636404 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f636c077010 msgr2=0x7f636c079520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:48:13.415 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:13.414+0000 7f638f200640 1 --2- 172.21.15.32:0/2151636404 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f636c077010 0x7f636c079520 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f6380033000 tx=0x7f6380004730 comp rx=0 tx=0).stop 2024-08-03T20:48:13.415 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:13.414+0000 7f638f200640 1 -- 172.21.15.32:0/2151636404 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f63900d26e0 msgr2=0x7f63900b8200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:48:13.415 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:13.414+0000 7f638f200640 1 --2- 172.21.15.32:0/2151636404 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f63900d26e0 0x7f63900b8200 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f638800e960 tx=0x7f638800ee20 comp rx=0 tx=0).stop 2024-08-03T20:48:13.415 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:13.414+0000 7f638f200640 1 -- 172.21.15.32:0/2151636404 shutdown_connections 2024-08-03T20:48:13.415 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:13.414+0000 7f638f200640 1 --2- 172.21.15.32:0/2151636404 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f636c077010 0x7f636c079520 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:13.415 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:13.414+0000 7f638f200640 1 --2- 172.21.15.32:0/2151636404 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f63900d26e0 0x7f63900b8200 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:13.415 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:13.414+0000 7f638f200640 1 --2- 172.21.15.32:0/2151636404 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f63900cffe0 0x7f63901525d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:13.415 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:13.414+0000 7f638f200640 1 --2- 172.21.15.32:0/2151636404 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f63900cf510 0x7f63900cdf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:13.416 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:13.414+0000 7f638f200640 1 -- 172.21.15.32:0/2151636404 >> 172.21.15.32:0/2151636404 conn(0x7f639008d7a0 msgr2=0x7f63900d1370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:48:13.416 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:13.414+0000 7f638f200640 1 -- 172.21.15.32:0/2151636404 shutdown_connections 2024-08-03T20:48:13.416 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:13.414+0000 7f638f200640 1 -- 172.21.15.32:0/2151636404 wait complete. 2024-08-03T20:48:13.440 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-08-03T20:48:13.669 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:13.668+0000 7fd98e800640 1 -- 172.21.15.32:0/2668053220 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fd9880cf510 msgr2=0x7fd9880cf920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:48:13.669 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:13.668+0000 7fd98e800640 1 --2- 172.21.15.32:0/2668053220 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fd9880cf510 0x7fd9880cf920 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fd980009be0 tx=0x7fd98002a920 comp rx=0 tx=0).stop 2024-08-03T20:48:13.669 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:13.668+0000 7fd98e800640 1 -- 172.21.15.32:0/2668053220 shutdown_connections 2024-08-03T20:48:13.669 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:13.668+0000 7fd98e800640 1 --2- 172.21.15.32:0/2668053220 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fd9880cf510 0x7fd9880cf920 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:13.669 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:13.668+0000 7fd98e800640 1 -- 172.21.15.32:0/2668053220 >> 172.21.15.32:0/2668053220 conn(0x7fd98808d7a0 msgr2=0x7fd98809d3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:48:13.669 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:13.668+0000 7fd98e800640 1 -- 172.21.15.32:0/2668053220 shutdown_connections 2024-08-03T20:48:13.669 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:13.668+0000 7fd98e800640 1 -- 172.21.15.32:0/2668053220 wait complete. 2024-08-03T20:48:13.671 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:13.670+0000 7fd98e800640 1 Processor -- start 2024-08-03T20:48:13.671 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:13.671+0000 7fd98e800640 1 -- start start 2024-08-03T20:48:13.673 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:13.672+0000 7fd98e800640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fd9880cf510 0x7fd98815caf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:48:13.673 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:13.672+0000 7fd98e800640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fd9880cffe0 0x7fd98814e590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:48:13.673 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:13.672+0000 7fd98e800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd9880d26e0 0x7fd9881f8810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:48:13.673 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:13.672+0000 7fd98e800640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fd9880d4310 con 0x7fd9880d26e0 2024-08-03T20:48:13.673 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:13.672+0000 7fd98e800640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7fd9880d4190 con 0x7fd9880cffe0 2024-08-03T20:48:13.673 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:13.672+0000 7fd98e800640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7fd9880d4490 con 0x7fd9880cf510 2024-08-03T20:48:13.673 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:13.672+0000 7fd98de00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd9880d26e0 0x7fd9881f8810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:48:13.673 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:13.672+0000 7fd98de00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd9880d26e0 0x7fd9881f8810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:53158/0 (socket says 172.21.15.32:53158) 2024-08-03T20:48:13.673 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:13.672+0000 7fd98de00640 1 -- 172.21.15.32:0/902770408 learned_addr learned my addr 172.21.15.32:0/902770408 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:48:13.674 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:13.673+0000 7fd98ca00640 1 --2- 172.21.15.32:0/902770408 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fd9880cffe0 0x7fd98814e590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:48:13.674 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:13.673+0000 7fd98d400640 1 --2- 172.21.15.32:0/902770408 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fd9880cf510 0x7fd98815caf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:48:13.674 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:13.673+0000 7fd98de00640 1 -- 172.21.15.32:0/902770408 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fd9880cf510 msgr2=0x7fd98815caf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:48:13.675 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:13.673+0000 7fd98de00640 1 --2- 172.21.15.32:0/902770408 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fd9880cf510 0x7fd98815caf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:13.675 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:13.673+0000 7fd98de00640 1 -- 172.21.15.32:0/902770408 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fd9880cffe0 msgr2=0x7fd98814e590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:48:13.675 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:13.673+0000 7fd98de00640 1 --2- 172.21.15.32:0/902770408 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fd9880cffe0 0x7fd98814e590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:13.675 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:13.673+0000 7fd98de00640 1 -- 172.21.15.32:0/902770408 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd98814ead0 con 0x7fd9880d26e0 2024-08-03T20:48:13.675 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:13.673+0000 7fd98de00640 1 --2- 172.21.15.32:0/902770408 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd9880d26e0 0x7fd9881f8810 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fd980009d00 tx=0x7fd980003ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:48:13.675 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:13.673+0000 7fd985600640 1 -- 172.21.15.32:0/902770408 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd980030c50 con 0x7fd9880d26e0 2024-08-03T20:48:13.675 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:13.673+0000 7fd985600640 1 -- 172.21.15.32:0/902770408 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fd98003e070 con 0x7fd9880d26e0 2024-08-03T20:48:13.675 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:13.674+0000 7fd985600640 1 -- 172.21.15.32:0/902770408 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd980032330 con 0x7fd9880d26e0 2024-08-03T20:48:13.675 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:13.674+0000 7fd98e800640 1 -- 172.21.15.32:0/902770408 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd9881f9700 con 0x7fd9880d26e0 2024-08-03T20:48:13.675 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:13.674+0000 7fd98e800640 1 -- 172.21.15.32:0/902770408 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd9880c8fc0 con 0x7fd9880d26e0 2024-08-03T20:48:13.676 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:13.675+0000 7fd98e800640 1 -- 172.21.15.32:0/902770408 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd9880d4940 con 0x7fd9880d26e0 2024-08-03T20:48:13.676 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:13.676+0000 7fd985600640 1 -- 172.21.15.32:0/902770408 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7fd9800324d0 con 0x7fd9880d26e0 2024-08-03T20:48:13.677 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:13.677+0000 7fd985600640 1 --2- 172.21.15.32:0/902770408 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fd968077040 0x7fd968079550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:48:13.677 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:13.677+0000 7fd985600640 1 -- 172.21.15.32:0/902770408 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7fd9800ba0c0 con 0x7fd9880d26e0 2024-08-03T20:48:13.678 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:13.678+0000 7fd98d400640 1 --2- 172.21.15.32:0/902770408 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fd968077040 0x7fd968079550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:48:13.678 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:13.678+0000 7fd98d400640 1 --2- 172.21.15.32:0/902770408 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fd968077040 0x7fd968079550 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fd97c001120 tx=0x7fd97c009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:48:13.682 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:13.681+0000 7fd985600640 1 -- 172.21.15.32:0/902770408 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7fd98002d070 con 0x7fd9880d26e0 2024-08-03T20:48:13.886 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:13.886+0000 7fd98e800640 1 -- 172.21.15.32:0/902770408 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fd98815ee50 con 0x7fd9880d26e0 2024-08-03T20:48:13.887 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:13.887+0000 7fd985600640 1 -- 172.21.15.32:0/902770408 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) ==== 74+0+16545 (secure 0 0 0) 0x7fd980083340 con 0x7fd9880d26e0 2024-08-03T20:48:13.888 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-03T20:48:13.888 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":21,"fsid":"40ab141a-01d7-49b9-ae3c-2edec9876e95","created":"2024-08-03T20:47:14.957362+0000","modified":"2024-08-03T20:47:36.927196+0000","last_up_change":"2024-08-03T20:47:28.874451+0000","last_in_change":"2024-08-03T20:47:20.079442+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-08-03T20:47:28.261300+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-08-03T20:47:33.085381+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-08-03T20:47:34.405408+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5a83e759-9035-438b-889a-0a6af1f0a0d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6816","nonce":3066156116},{"type":"v1","addr":"172.21.15.32:6817","nonce":3066156116}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6818","nonce":3066156116},{"type":"v1","addr":"172.21.15.32:6819","nonce":3066156116}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6822","nonce":3066156116},{"type":"v1","addr":"172.21.15.32:6823","nonce":3066156116}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6820","nonce":3066156116},{"type":"v1","addr":"172.21.15.32:6821","nonce":3066156116}]},"public_addr":"172.21.15.32:6817/3066156116","cluster_addr":"172.21.15.32:6819/3066156116","heartbeat_back_addr":"172.21.15.32:6823/3066156116","heartbeat_front_addr":"172.21.15.32:6821/3066156116","state":["exists","up"]},{"osd":1,"uuid":"d7c5c210-da4f-49f9-b9e7-1dfb02daa4bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6824","nonce":1551241759},{"type":"v1","addr":"172.21.15.32:6825","nonce":1551241759}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6826","nonce":1551241759},{"type":"v1","addr":"172.21.15.32:6827","nonce":1551241759}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6830","nonce":1551241759},{"type":"v1","addr":"172.21.15.32:6831","nonce":1551241759}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6828","nonce":1551241759},{"type":"v1","addr":"172.21.15.32:6829","nonce":1551241759}]},"public_addr":"172.21.15.32:6825/1551241759","cluster_addr":"172.21.15.32:6827/1551241759","heartbeat_back_addr":"172.21.15.32:6831/1551241759","heartbeat_front_addr":"172.21.15.32:6829/1551241759","state":["exists","up"]},{"osd":2,"uuid":"57f2e6de-a201-45c2-8981-8e56a4875ebd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6808","nonce":3806708944},{"type":"v1","addr":"172.21.15.32:6809","nonce":3806708944}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6810","nonce":3806708944},{"type":"v1","addr":"172.21.15.32:6811","nonce":3806708944}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6814","nonce":3806708944},{"type":"v1","addr":"172.21.15.32:6815","nonce":3806708944}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6812","nonce":3806708944},{"type":"v1","addr":"172.21.15.32:6813","nonce":3806708944}]},"public_addr":"172.21.15.32:6809/3806708944","cluster_addr":"172.21.15.32:6811/3806708944","heartbeat_back_addr":"172.21.15.32:6815/3806708944","heartbeat_front_addr":"172.21.15.32:6813/3806708944","state":["exists","up"]},{"osd":3,"uuid":"6439a280-da45-4148-8130-f16323935ff3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6800","nonce":9088303},{"type":"v1","addr":"172.21.15.32:6801","nonce":9088303}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6802","nonce":9088303},{"type":"v1","addr":"172.21.15.32:6803","nonce":9088303}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6806","nonce":9088303},{"type":"v1","addr":"172.21.15.32:6807","nonce":9088303}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6804","nonce":9088303},{"type":"v1","addr":"172.21.15.32:6805","nonce":9088303}]},"public_addr":"172.21.15.32:6801/9088303","cluster_addr":"172.21.15.32:6803/9088303","heartbeat_back_addr":"172.21.15.32:6807/9088303","heartbeat_front_addr":"172.21.15.32:6805/9088303","state":["exists","up"]},{"osd":4,"uuid":"4f902eff-1e18-418d-82c9-2151c69524fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6802","nonce":864752275},{"type":"v1","addr":"172.21.15.118:6803","nonce":864752275}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6804","nonce":864752275},{"type":"v1","addr":"172.21.15.118:6805","nonce":864752275}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6808","nonce":864752275},{"type":"v1","addr":"172.21.15.118:6809","nonce":864752275}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6806","nonce":864752275},{"type":"v1","addr":"172.21.15.118:6807","nonce":864752275}]},"public_addr":"172.21.15.118:6803/864752275","cluster_addr":"172.21.15.118:6805/864752275","heartbeat_back_addr":"172.21.15.118:6809/864752275","heartbeat_front_addr":"172.21.15.118:6807/864752275","state":["exists","up"]},{"osd":5,"uuid":"bfd9feac-dd06-458c-91b9-af7f175f612c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6810","nonce":2724891770},{"type":"v1","addr":"172.21.15.118:6811","nonce":2724891770}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6812","nonce":2724891770},{"type":"v1","addr":"172.21.15.118:6813","nonce":2724891770}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6816","nonce":2724891770},{"type":"v1","addr":"172.21.15.118:6817","nonce":2724891770}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6814","nonce":2724891770},{"type":"v1","addr":"172.21.15.118:6815","nonce":2724891770}]},"public_addr":"172.21.15.118:6811/2724891770","cluster_addr":"172.21.15.118:6813/2724891770","heartbeat_back_addr":"172.21.15.118:6817/2724891770","heartbeat_front_addr":"172.21.15.118:6815/2724891770","state":["exists","up"]},{"osd":6,"uuid":"5ed5a279-9d8e-416d-bbc7-7edd3e6d5705","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6825","nonce":801947386},{"type":"v1","addr":"172.21.15.118:6827","nonce":801947386}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6828","nonce":801947386},{"type":"v1","addr":"172.21.15.118:6829","nonce":801947386}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6832","nonce":801947386},{"type":"v1","addr":"172.21.15.118:6833","nonce":801947386}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6830","nonce":801947386},{"type":"v1","addr":"172.21.15.118:6831","nonce":801947386}]},"public_addr":"172.21.15.118:6827/801947386","cluster_addr":"172.21.15.118:6829/801947386","heartbeat_back_addr":"172.21.15.118:6833/801947386","heartbeat_front_addr":"172.21.15.118:6831/801947386","state":["exists","up"]},{"osd":7,"uuid":"f2086c47-875e-4451-a436-2008d1ea16af","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6818","nonce":653000048},{"type":"v1","addr":"172.21.15.118:6819","nonce":653000048}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6820","nonce":653000048},{"type":"v1","addr":"172.21.15.118:6821","nonce":653000048}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6824","nonce":653000048},{"type":"v1","addr":"172.21.15.118:6826","nonce":653000048}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6822","nonce":653000048},{"type":"v1","addr":"172.21.15.118:6823","nonce":653000048}]},"public_addr":"172.21.15.118:6819/653000048","cluster_addr":"172.21.15.118:6821/653000048","heartbeat_back_addr":"172.21.15.118:6826/653000048","heartbeat_front_addr":"172.21.15.118:6823/653000048","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-03T20:47:25.491994+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-03T20:47:25.587358+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-03T20:47:25.454175+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-03T20:47:25.377734+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-03T20:47:25.412351+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-03T20:47:25.499815+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-03T20:47:25.575044+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-03T20:47:25.552383+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-08-03T20:48:13.890 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:13.889+0000 7fd98e800640 1 -- 172.21.15.32:0/902770408 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fd968077040 msgr2=0x7fd968079550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:48:13.890 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:13.889+0000 7fd98e800640 1 --2- 172.21.15.32:0/902770408 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fd968077040 0x7fd968079550 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fd97c001120 tx=0x7fd97c009040 comp rx=0 tx=0).stop 2024-08-03T20:48:13.890 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:13.889+0000 7fd98e800640 1 -- 172.21.15.32:0/902770408 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd9880d26e0 msgr2=0x7fd9881f8810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:48:13.890 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:13.889+0000 7fd98e800640 1 --2- 172.21.15.32:0/902770408 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd9880d26e0 0x7fd9881f8810 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fd980009d00 tx=0x7fd980003ba0 comp rx=0 tx=0).stop 2024-08-03T20:48:13.890 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:13.890+0000 7fd98e800640 1 -- 172.21.15.32:0/902770408 shutdown_connections 2024-08-03T20:48:13.891 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:13.890+0000 7fd98e800640 1 --2- 172.21.15.32:0/902770408 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fd968077040 0x7fd968079550 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:13.891 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:13.890+0000 7fd98e800640 1 --2- 172.21.15.32:0/902770408 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fd9880cffe0 0x7fd98814e590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:13.891 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:13.890+0000 7fd98e800640 1 --2- 172.21.15.32:0/902770408 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd9880d26e0 0x7fd9881f8810 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:13.891 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:13.890+0000 7fd98e800640 1 --2- 172.21.15.32:0/902770408 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fd9880cf510 0x7fd98815caf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:13.891 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:13.890+0000 7fd98e800640 1 -- 172.21.15.32:0/902770408 >> 172.21.15.32:0/902770408 conn(0x7fd98808d7a0 msgr2=0x7fd98815dad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:48:13.891 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:13.890+0000 7fd98e800640 1 -- 172.21.15.32:0/902770408 shutdown_connections 2024-08-03T20:48:13.891 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:13.890+0000 7fd98e800640 1 -- 172.21.15.32:0/902770408 wait complete. 2024-08-03T20:48:13.912 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-08-03T20:48:13.912 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-08-03T20:48:14.143 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:14.142+0000 7f7989000640 1 -- 172.21.15.32:0/663933327 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7968005270 msgr2=0x7f796800b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:48:14.143 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:14.142+0000 7f7989000640 1 --2- 172.21.15.32:0/663933327 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7968005270 0x7f796800b9c0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f797c0013d0 tx=0x7f7968005830 comp rx=0 tx=0).stop 2024-08-03T20:48:14.143 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:14.142+0000 7f7989000640 1 -- 172.21.15.32:0/663933327 shutdown_connections 2024-08-03T20:48:14.143 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:14.142+0000 7f7989000640 1 --2- 172.21.15.32:0/663933327 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7968005270 0x7f796800b9c0 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:14.144 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:14.142+0000 7f7989000640 1 -- 172.21.15.32:0/663933327 >> 172.21.15.32:0/663933327 conn(0x7f79840b1bd0 msgr2=0x7f79840b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:48:14.144 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:14.142+0000 7f7989000640 1 -- 172.21.15.32:0/663933327 shutdown_connections 2024-08-03T20:48:14.144 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:14.142+0000 7f7989000640 1 -- 172.21.15.32:0/663933327 wait complete. 2024-08-03T20:48:14.145 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:14.145+0000 7f7989000640 1 Processor -- start 2024-08-03T20:48:14.145 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:14.145+0000 7f7989000640 1 -- start start 2024-08-03T20:48:14.151 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:14.146+0000 7f7989000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7968005270 0x7f798415be70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:48:14.151 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:14.146+0000 7f7989000640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f79840a9670 0x7f79840a9a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:48:14.151 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:14.146+0000 7f7989000640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f798415f050 0x7f798415f460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:48:14.151 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:14.146+0000 7f7989000640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f79840cf840 con 0x7f7968005270 2024-08-03T20:48:14.151 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:14.146+0000 7f7989000640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f7984145f30 con 0x7f798415f050 2024-08-03T20:48:14.151 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:14.146+0000 7f7989000640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7f79840cf9c0 con 0x7f79840a9670 2024-08-03T20:48:14.151 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:14.147+0000 7f7981e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7968005270 0x7f798415be70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:48:14.151 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:14.147+0000 7f7981e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7968005270 0x7f798415be70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:53166/0 (socket says 172.21.15.32:53166) 2024-08-03T20:48:14.151 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:14.147+0000 7f7981e00640 1 -- 172.21.15.32:0/4145123242 learned_addr learned my addr 172.21.15.32:0/4145123242 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:48:14.151 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:14.147+0000 7f797be00640 1 --2- 172.21.15.32:0/4145123242 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f79840a9670 0x7f79840a9a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:48:14.151 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:14.147+0000 7f7982800640 1 --2- 172.21.15.32:0/4145123242 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f798415f050 0x7f798415f460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:48:14.152 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:14.147+0000 7f7981e00640 1 -- 172.21.15.32:0/4145123242 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f79840a9670 msgr2=0x7f79840a9a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:48:14.152 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:14.147+0000 7f7981e00640 1 --2- 172.21.15.32:0/4145123242 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f79840a9670 0x7f79840a9a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:14.152 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:14.147+0000 7f7981e00640 1 -- 172.21.15.32:0/4145123242 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f798415f050 msgr2=0x7f798415f460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:48:14.152 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:14.147+0000 7f7981e00640 1 --2- 172.21.15.32:0/4145123242 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f798415f050 0x7f798415f460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:14.152 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:14.147+0000 7f7981e00640 1 -- 172.21.15.32:0/4145123242 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f798415f9a0 con 0x7f7968005270 2024-08-03T20:48:14.152 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:14.147+0000 7f797be00640 1 --2- 172.21.15.32:0/4145123242 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f79840a9670 0x7f79840a9a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-03T20:48:14.152 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:14.147+0000 7f7981e00640 1 --2- 172.21.15.32:0/4145123242 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7968005270 0x7f798415be70 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f7974004f00 tx=0x7f7974001630 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:48:14.152 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:14.148+0000 7f797a000640 1 -- 172.21.15.32:0/4145123242 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7974015020 con 0x7f7968005270 2024-08-03T20:48:14.152 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:14.148+0000 7f797a000640 1 -- 172.21.15.32:0/4145123242 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f7974011960 con 0x7f7968005270 2024-08-03T20:48:14.152 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:14.148+0000 7f797a000640 1 -- 172.21.15.32:0/4145123242 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7974011d90 con 0x7f7968005270 2024-08-03T20:48:14.152 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:14.148+0000 7f7989000640 1 -- 172.21.15.32:0/4145123242 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f798415c3b0 con 0x7f7968005270 2024-08-03T20:48:14.152 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:14.148+0000 7f7989000640 1 -- 172.21.15.32:0/4145123242 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f79841f3d10 con 0x7f7968005270 2024-08-03T20:48:14.152 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:14.149+0000 7f7989000640 1 -- 172.21.15.32:0/4145123242 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f79840740e0 con 0x7f7968005270 2024-08-03T20:48:14.153 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:14.150+0000 7f797a000640 1 -- 172.21.15.32:0/4145123242 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f7974018020 con 0x7f7968005270 2024-08-03T20:48:14.153 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:14.151+0000 7f797a000640 1 --2- 172.21.15.32:0/4145123242 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f7960077010 0x7f7960079520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:48:14.153 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:14.151+0000 7f797a000640 1 -- 172.21.15.32:0/4145123242 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f79740994c0 con 0x7f7968005270 2024-08-03T20:48:14.153 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:14.152+0000 7f797be00640 1 --2- 172.21.15.32:0/4145123242 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f7960077010 0x7f7960079520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:48:14.154 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:14.153+0000 7f797be00640 1 --2- 172.21.15.32:0/4145123242 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f7960077010 0x7f7960079520 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f79841f5ee0 tx=0x7f796c005f90 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:48:14.156 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:14.156+0000 7f797a000640 1 -- 172.21.15.32:0/4145123242 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7f797401f020 con 0x7f7968005270 2024-08-03T20:48:14.397 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:14.396+0000 7f7989000640 1 -- 172.21.15.32:0/4145123242 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f79841f3ac0 con 0x7f7968005270 2024-08-03T20:48:14.397 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:14.397+0000 7f797a000640 1 -- 172.21.15.32:0/4145123242 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) ==== 93+0+5220 (secure 0 0 0) 0x7f798415f9a0 con 0x7f7968005270 2024-08-03T20:48:14.398 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-03T20:48:14.398 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":8,"btime":"2024-08-03T20:47:38:939479+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4289,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6835/1753256117","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6834","nonce":1753256117},{"type":"v1","addr":"172.21.15.118:6835","nonce":1753256117}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4292,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6837/3055588578","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6836","nonce":3055588578},{"type":"v1","addr":"172.21.15.118:6837","nonce":3055588578}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4338,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6834/3026095815","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":3026095815},{"type":"v1","addr":"172.21.15.32:6834","nonce":3026095815}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-08-03T20:47:35.442482+0000","modified":"2024-08-03T20:47:38.460928+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4341},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4341":{"gid":4341,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.32:6835/246278108","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6833","nonce":246278108},{"type":"v1","addr":"172.21.15.32:6835","nonce":246278108}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4341,"qdb_cluster":[4341]},"id":1}]} 2024-08-03T20:48:14.398 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 8 2024-08-03T20:48:14.400 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:14.399+0000 7f7989000640 1 -- 172.21.15.32:0/4145123242 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f7960077010 msgr2=0x7f7960079520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:48:14.400 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:14.399+0000 7f7989000640 1 --2- 172.21.15.32:0/4145123242 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f7960077010 0x7f7960079520 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f79841f5ee0 tx=0x7f796c005f90 comp rx=0 tx=0).stop 2024-08-03T20:48:14.400 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:14.399+0000 7f7989000640 1 -- 172.21.15.32:0/4145123242 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7968005270 msgr2=0x7f798415be70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:48:14.400 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:14.399+0000 7f7989000640 1 --2- 172.21.15.32:0/4145123242 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7968005270 0x7f798415be70 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f7974004f00 tx=0x7f7974001630 comp rx=0 tx=0).stop 2024-08-03T20:48:14.400 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:14.399+0000 7f7989000640 1 -- 172.21.15.32:0/4145123242 shutdown_connections 2024-08-03T20:48:14.400 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:14.399+0000 7f7989000640 1 --2- 172.21.15.32:0/4145123242 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f7960077010 0x7f7960079520 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:14.400 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:14.399+0000 7f7989000640 1 --2- 172.21.15.32:0/4145123242 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f798415f050 0x7f798415f460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:14.400 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:14.399+0000 7f7989000640 1 --2- 172.21.15.32:0/4145123242 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f79840a9670 0x7f79840a9a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:14.401 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:14.399+0000 7f7989000640 1 --2- 172.21.15.32:0/4145123242 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7968005270 0x7f798415be70 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:14.401 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:14.399+0000 7f7989000640 1 -- 172.21.15.32:0/4145123242 >> 172.21.15.32:0/4145123242 conn(0x7f79840b1bd0 msgr2=0x7f7984143a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:48:14.401 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:14.399+0000 7f7989000640 1 -- 172.21.15.32:0/4145123242 shutdown_connections 2024-08-03T20:48:14.401 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:14.399+0000 7f7989000640 1 -- 172.21.15.32:0/4145123242 wait complete. 2024-08-03T20:48:14.421 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-08-03T20:48:14.648 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:14.647+0000 7fa12fa00640 1 -- 172.21.15.32:0/787448444 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa12815fff0 msgr2=0x7fa128160400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:48:14.648 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:14.647+0000 7fa12fa00640 1 --2- 172.21.15.32:0/787448444 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa12815fff0 0x7fa128160400 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fa120005fd0 tx=0x7fa12002a8b0 comp rx=0 tx=0).stop 2024-08-03T20:48:14.648 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:14.647+0000 7fa12fa00640 1 -- 172.21.15.32:0/787448444 shutdown_connections 2024-08-03T20:48:14.648 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:14.647+0000 7fa12fa00640 1 --2- 172.21.15.32:0/787448444 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa12815fff0 0x7fa128160400 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:14.648 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:14.647+0000 7fa12fa00640 1 --2- 172.21.15.32:0/787448444 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fa12806e930 0x7fa128144f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:14.648 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:14.647+0000 7fa12fa00640 1 -- 172.21.15.32:0/787448444 >> 172.21.15.32:0/787448444 conn(0x7fa1280b1bd0 msgr2=0x7fa1280b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:48:14.648 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:14.647+0000 7fa12fa00640 1 -- 172.21.15.32:0/787448444 shutdown_connections 2024-08-03T20:48:14.648 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:14.647+0000 7fa12fa00640 1 -- 172.21.15.32:0/787448444 wait complete. 2024-08-03T20:48:14.649 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:14.649+0000 7fa12fa00640 1 Processor -- start 2024-08-03T20:48:14.650 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:14.649+0000 7fa12fa00640 1 -- start start 2024-08-03T20:48:14.652 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:14.651+0000 7fa12fa00640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fa12806e930 0x7fa12815a310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:48:14.652 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:14.651+0000 7fa12fa00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa128073e60 0x7fa128158060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:48:14.652 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:14.651+0000 7fa12fa00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa12815fff0 0x7fa12813dab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:48:14.652 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:14.651+0000 7fa12fa00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fa1280cf840 con 0x7fa128073e60 2024-08-03T20:48:14.652 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:14.651+0000 7fa12fa00640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7fa128145f30 con 0x7fa12815fff0 2024-08-03T20:48:14.653 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:14.651+0000 7fa12fa00640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7fa1280cf9c0 con 0x7fa12806e930 2024-08-03T20:48:14.653 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:14.651+0000 7fa127e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa128073e60 0x7fa128158060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:48:14.653 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:14.651+0000 7fa127e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa128073e60 0x7fa128158060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:53178/0 (socket says 172.21.15.32:53178) 2024-08-03T20:48:14.653 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:14.651+0000 7fa127e00640 1 -- 172.21.15.32:0/3665680813 learned_addr learned my addr 172.21.15.32:0/3665680813 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:48:14.653 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:14.651+0000 7fa12da00640 1 --2- 172.21.15.32:0/3665680813 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa12815fff0 0x7fa12813dab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:48:14.653 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:14.651+0000 7fa12d000640 1 --2- 172.21.15.32:0/3665680813 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fa12806e930 0x7fa12815a310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:48:14.653 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:14.652+0000 7fa127e00640 1 -- 172.21.15.32:0/3665680813 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fa12806e930 msgr2=0x7fa12815a310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:48:14.653 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:14.652+0000 7fa127e00640 1 --2- 172.21.15.32:0/3665680813 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fa12806e930 0x7fa12815a310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:14.653 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:14.652+0000 7fa127e00640 1 -- 172.21.15.32:0/3665680813 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa12815fff0 msgr2=0x7fa12813dab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:48:14.653 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:14.652+0000 7fa127e00640 1 --2- 172.21.15.32:0/3665680813 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa12815fff0 0x7fa12813dab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:14.653 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:14.652+0000 7fa127e00640 1 -- 172.21.15.32:0/3665680813 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa1281f58f0 con 0x7fa128073e60 2024-08-03T20:48:14.653 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:14.652+0000 7fa12d000640 1 --2- 172.21.15.32:0/3665680813 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fa12806e930 0x7fa12815a310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-03T20:48:14.653 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:14.652+0000 7fa127e00640 1 --2- 172.21.15.32:0/3665680813 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa128073e60 0x7fa128158060 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fa11400c3d0 tx=0x7fa11400c890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:48:14.654 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:14.652+0000 7fa124c00640 1 -- 172.21.15.32:0/3665680813 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa114011790 con 0x7fa128073e60 2024-08-03T20:48:14.654 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:14.652+0000 7fa124c00640 1 -- 172.21.15.32:0/3665680813 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fa114017070 con 0x7fa128073e60 2024-08-03T20:48:14.654 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:14.652+0000 7fa124c00640 1 -- 172.21.15.32:0/3665680813 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa114011cc0 con 0x7fa128073e60 2024-08-03T20:48:14.654 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:14.652+0000 7fa12fa00640 1 -- 172.21.15.32:0/3665680813 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa1281f5bd0 con 0x7fa128073e60 2024-08-03T20:48:14.655 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:14.652+0000 7fa12fa00640 1 -- 172.21.15.32:0/3665680813 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa1281f6190 con 0x7fa128073e60 2024-08-03T20:48:14.655 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:14.653+0000 7fa12fa00640 1 -- 172.21.15.32:0/3665680813 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa128160bc0 con 0x7fa128073e60 2024-08-03T20:48:14.655 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:14.655+0000 7fa124c00640 1 -- 172.21.15.32:0/3665680813 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7fa11401c460 con 0x7fa128073e60 2024-08-03T20:48:14.656 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:14.656+0000 7fa124c00640 1 --2- 172.21.15.32:0/3665680813 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fa108077040 0x7fa108079550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:48:14.656 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:14.656+0000 7fa124c00640 1 -- 172.21.15.32:0/3665680813 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7fa114098650 con 0x7fa128073e60 2024-08-03T20:48:14.656 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:14.656+0000 7fa12d000640 1 --2- 172.21.15.32:0/3665680813 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fa108077040 0x7fa108079550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:48:14.657 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:14.657+0000 7fa12d000640 1 --2- 172.21.15.32:0/3665680813 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fa108077040 0x7fa108079550 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fa11c005fd0 tx=0x7fa11c00f360 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:48:14.660 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:14.660+0000 7fa124c00640 1 -- 172.21.15.32:0/3665680813 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7fa114061840 con 0x7fa128073e60 2024-08-03T20:48:14.903 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:14.903+0000 7fa12fa00640 1 -- 172.21.15.32:0/3665680813 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa1281f5680 con 0x7fa128073e60 2024-08-03T20:48:14.904 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:14.903+0000 7fa124c00640 1 -- 172.21.15.32:0/3665680813 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) ==== 93+0+5220 (secure 0 0 0) 0x7fa114061140 con 0x7fa128073e60 2024-08-03T20:48:14.904 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-03T20:48:14.904 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":8,"btime":"2024-08-03T20:47:38:939479+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4289,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6835/1753256117","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6834","nonce":1753256117},{"type":"v1","addr":"172.21.15.118:6835","nonce":1753256117}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4292,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6837/3055588578","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6836","nonce":3055588578},{"type":"v1","addr":"172.21.15.118:6837","nonce":3055588578}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4338,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6834/3026095815","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":3026095815},{"type":"v1","addr":"172.21.15.32:6834","nonce":3026095815}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-08-03T20:47:35.442482+0000","modified":"2024-08-03T20:47:38.460928+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4341},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4341":{"gid":4341,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.32:6835/246278108","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6833","nonce":246278108},{"type":"v1","addr":"172.21.15.32:6835","nonce":246278108}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4341,"qdb_cluster":[4341]},"id":1}]} 2024-08-03T20:48:14.905 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 8 2024-08-03T20:48:14.907 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:14.906+0000 7fa12fa00640 1 -- 172.21.15.32:0/3665680813 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fa108077040 msgr2=0x7fa108079550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:48:14.907 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:14.906+0000 7fa12fa00640 1 --2- 172.21.15.32:0/3665680813 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fa108077040 0x7fa108079550 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fa11c005fd0 tx=0x7fa11c00f360 comp rx=0 tx=0).stop 2024-08-03T20:48:14.907 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:14.906+0000 7fa12fa00640 1 -- 172.21.15.32:0/3665680813 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa128073e60 msgr2=0x7fa128158060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:48:14.907 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:14.906+0000 7fa12fa00640 1 --2- 172.21.15.32:0/3665680813 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa128073e60 0x7fa128158060 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fa11400c3d0 tx=0x7fa11400c890 comp rx=0 tx=0).stop 2024-08-03T20:48:14.907 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:14.906+0000 7fa12fa00640 1 -- 172.21.15.32:0/3665680813 shutdown_connections 2024-08-03T20:48:14.907 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:14.906+0000 7fa12fa00640 1 --2- 172.21.15.32:0/3665680813 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fa108077040 0x7fa108079550 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:14.907 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:14.906+0000 7fa12fa00640 1 --2- 172.21.15.32:0/3665680813 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa12815fff0 0x7fa12813dab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:14.907 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:14.906+0000 7fa12fa00640 1 --2- 172.21.15.32:0/3665680813 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa128073e60 0x7fa128158060 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:14.907 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:14.906+0000 7fa12fa00640 1 --2- 172.21.15.32:0/3665680813 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fa12806e930 0x7fa12815a310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:14.907 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:14.906+0000 7fa12fa00640 1 -- 172.21.15.32:0/3665680813 >> 172.21.15.32:0/3665680813 conn(0x7fa1280b1bd0 msgr2=0x7fa128074270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:48:14.907 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:14.906+0000 7fa12fa00640 1 -- 172.21.15.32:0/3665680813 shutdown_connections 2024-08-03T20:48:14.907 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:14.906+0000 7fa12fa00640 1 -- 172.21.15.32:0/3665680813 wait complete. 2024-08-03T20:48:14.928 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-08-03T20:48:15.157 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:15.156+0000 7fa835200640 1 -- 172.21.15.32:0/289522969 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa830073e60 msgr2=0x7fa830074270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:48:15.158 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:15.156+0000 7fa835200640 1 --2- 172.21.15.32:0/289522969 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa830073e60 0x7fa830074270 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fa830071700 tx=0x7fa81c02a4a0 comp rx=0 tx=0).stop 2024-08-03T20:48:15.158 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:15.157+0000 7fa835200640 1 -- 172.21.15.32:0/289522969 shutdown_connections 2024-08-03T20:48:15.158 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:15.157+0000 7fa835200640 1 --2- 172.21.15.32:0/289522969 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa830073e60 0x7fa830074270 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:15.158 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:15.157+0000 7fa835200640 1 --2- 172.21.15.32:0/289522969 >> v2:172.21.15.118:3300/0 conn(0x7fa83006e930 0x7fa830144f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:15.158 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:15.157+0000 7fa835200640 1 -- 172.21.15.32:0/289522969 >> 172.21.15.32:0/289522969 conn(0x7fa8300b1bd0 msgr2=0x7fa8300b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:48:15.158 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:15.157+0000 7fa835200640 1 -- 172.21.15.32:0/289522969 shutdown_connections 2024-08-03T20:48:15.158 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:15.157+0000 7fa835200640 1 -- 172.21.15.32:0/289522969 wait complete. 2024-08-03T20:48:15.159 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:15.159+0000 7fa835200640 1 Processor -- start 2024-08-03T20:48:15.159 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:15.159+0000 7fa835200640 1 -- start start 2024-08-03T20:48:15.161 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:15.160+0000 7fa835200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa83006e930 0x7fa8300b2fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:48:15.162 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:15.160+0000 7fa835200640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fa830073e60 0x7fa8300b3e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:48:15.162 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:15.160+0000 7fa835200640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa8300747b0 0x7fa8301363c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:48:15.162 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:15.160+0000 7fa835200640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fa8300cf9c0 con 0x7fa83006e930 2024-08-03T20:48:15.162 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:15.160+0000 7fa835200640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7fa8300cf840 con 0x7fa8300747b0 2024-08-03T20:48:15.162 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:15.160+0000 7fa835200640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7fa8300ccff0 con 0x7fa830073e60 2024-08-03T20:48:15.162 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:15.161+0000 7fa82de00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa83006e930 0x7fa8300b2fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:48:15.162 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:15.161+0000 7fa82de00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa83006e930 0x7fa8300b2fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:53190/0 (socket says 172.21.15.32:53190) 2024-08-03T20:48:15.162 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:15.161+0000 7fa82de00640 1 -- 172.21.15.32:0/3116510809 learned_addr learned my addr 172.21.15.32:0/3116510809 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:48:15.162 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:15.161+0000 7fa82d400640 1 --2- 172.21.15.32:0/3116510809 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fa830073e60 0x7fa8300b3e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:48:15.162 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:15.161+0000 7fa82e800640 1 --2- 172.21.15.32:0/3116510809 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa8300747b0 0x7fa8301363c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:48:15.162 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:15.161+0000 7fa82de00640 1 -- 172.21.15.32:0/3116510809 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fa830073e60 msgr2=0x7fa8300b3e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:48:15.162 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:15.161+0000 7fa82de00640 1 --2- 172.21.15.32:0/3116510809 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fa830073e60 0x7fa8300b3e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:15.163 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:15.161+0000 7fa82de00640 1 -- 172.21.15.32:0/3116510809 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa8300747b0 msgr2=0x7fa8301363c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:48:15.163 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:15.161+0000 7fa82de00640 1 --2- 172.21.15.32:0/3116510809 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa8300747b0 0x7fa8301363c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:15.163 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:15.161+0000 7fa82de00640 1 -- 172.21.15.32:0/3116510809 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa8300aeba0 con 0x7fa83006e930 2024-08-03T20:48:15.163 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:15.161+0000 7fa82e800640 1 --2- 172.21.15.32:0/3116510809 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa8300747b0 0x7fa8301363c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-03T20:48:15.163 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:15.161+0000 7fa82de00640 1 --2- 172.21.15.32:0/3116510809 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa83006e930 0x7fa8300b2fd0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fa82000e790 tx=0x7fa82000ec50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:48:15.163 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:15.162+0000 7fa826000640 1 -- 172.21.15.32:0/3116510809 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa82000eea0 con 0x7fa83006e930 2024-08-03T20:48:15.164 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:15.162+0000 7fa826000640 1 -- 172.21.15.32:0/3116510809 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fa82000cdb0 con 0x7fa83006e930 2024-08-03T20:48:15.164 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:15.162+0000 7fa835200640 1 -- 172.21.15.32:0/3116510809 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa8300aee80 con 0x7fa83006e930 2024-08-03T20:48:15.164 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:15.162+0000 7fa826000640 1 -- 172.21.15.32:0/3116510809 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa820010690 con 0x7fa83006e930 2024-08-03T20:48:15.164 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:15.162+0000 7fa835200640 1 -- 172.21.15.32:0/3116510809 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa8300af440 con 0x7fa83006e930 2024-08-03T20:48:15.164 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:15.163+0000 7fa835200640 1 -- 172.21.15.32:0/3116510809 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa8300cd580 con 0x7fa83006e930 2024-08-03T20:48:15.164 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:15.164+0000 7fa826000640 1 -- 172.21.15.32:0/3116510809 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7fa820010830 con 0x7fa83006e930 2024-08-03T20:48:15.165 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:15.165+0000 7fa826000640 1 --2- 172.21.15.32:0/3116510809 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fa810077040 0x7fa810079550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:48:15.166 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:15.165+0000 7fa826000640 1 -- 172.21.15.32:0/3116510809 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7fa8200992a0 con 0x7fa83006e930 2024-08-03T20:48:15.166 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:15.166+0000 7fa82d400640 1 --2- 172.21.15.32:0/3116510809 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fa810077040 0x7fa810079550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:48:15.167 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:15.167+0000 7fa82d400640 1 --2- 172.21.15.32:0/3116510809 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fa810077040 0x7fa810079550 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fa81c034000 tx=0x7fa81c004730 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:48:15.170 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:15.170+0000 7fa826000640 1 -- 172.21.15.32:0/3116510809 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7fa820062490 con 0x7fa83006e930 2024-08-03T20:48:15.374 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:15.373+0000 7fa835200640 1 -- 172.21.15.32:0/3116510809 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fa8301451a0 con 0x7fa83006e930 2024-08-03T20:48:15.375 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:15.375+0000 7fa826000640 1 -- 172.21.15.32:0/3116510809 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) ==== 74+0+16545 (secure 0 0 0) 0x7fa820061d90 con 0x7fa83006e930 2024-08-03T20:48:15.375 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-03T20:48:15.376 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":21,"fsid":"40ab141a-01d7-49b9-ae3c-2edec9876e95","created":"2024-08-03T20:47:14.957362+0000","modified":"2024-08-03T20:47:36.927196+0000","last_up_change":"2024-08-03T20:47:28.874451+0000","last_in_change":"2024-08-03T20:47:20.079442+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-08-03T20:47:28.261300+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-08-03T20:47:33.085381+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-08-03T20:47:34.405408+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5a83e759-9035-438b-889a-0a6af1f0a0d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6816","nonce":3066156116},{"type":"v1","addr":"172.21.15.32:6817","nonce":3066156116}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6818","nonce":3066156116},{"type":"v1","addr":"172.21.15.32:6819","nonce":3066156116}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6822","nonce":3066156116},{"type":"v1","addr":"172.21.15.32:6823","nonce":3066156116}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6820","nonce":3066156116},{"type":"v1","addr":"172.21.15.32:6821","nonce":3066156116}]},"public_addr":"172.21.15.32:6817/3066156116","cluster_addr":"172.21.15.32:6819/3066156116","heartbeat_back_addr":"172.21.15.32:6823/3066156116","heartbeat_front_addr":"172.21.15.32:6821/3066156116","state":["exists","up"]},{"osd":1,"uuid":"d7c5c210-da4f-49f9-b9e7-1dfb02daa4bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6824","nonce":1551241759},{"type":"v1","addr":"172.21.15.32:6825","nonce":1551241759}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6826","nonce":1551241759},{"type":"v1","addr":"172.21.15.32:6827","nonce":1551241759}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6830","nonce":1551241759},{"type":"v1","addr":"172.21.15.32:6831","nonce":1551241759}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6828","nonce":1551241759},{"type":"v1","addr":"172.21.15.32:6829","nonce":1551241759}]},"public_addr":"172.21.15.32:6825/1551241759","cluster_addr":"172.21.15.32:6827/1551241759","heartbeat_back_addr":"172.21.15.32:6831/1551241759","heartbeat_front_addr":"172.21.15.32:6829/1551241759","state":["exists","up"]},{"osd":2,"uuid":"57f2e6de-a201-45c2-8981-8e56a4875ebd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6808","nonce":3806708944},{"type":"v1","addr":"172.21.15.32:6809","nonce":3806708944}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6810","nonce":3806708944},{"type":"v1","addr":"172.21.15.32:6811","nonce":3806708944}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6814","nonce":3806708944},{"type":"v1","addr":"172.21.15.32:6815","nonce":3806708944}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6812","nonce":3806708944},{"type":"v1","addr":"172.21.15.32:6813","nonce":3806708944}]},"public_addr":"172.21.15.32:6809/3806708944","cluster_addr":"172.21.15.32:6811/3806708944","heartbeat_back_addr":"172.21.15.32:6815/3806708944","heartbeat_front_addr":"172.21.15.32:6813/3806708944","state":["exists","up"]},{"osd":3,"uuid":"6439a280-da45-4148-8130-f16323935ff3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6800","nonce":9088303},{"type":"v1","addr":"172.21.15.32:6801","nonce":9088303}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6802","nonce":9088303},{"type":"v1","addr":"172.21.15.32:6803","nonce":9088303}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6806","nonce":9088303},{"type":"v1","addr":"172.21.15.32:6807","nonce":9088303}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6804","nonce":9088303},{"type":"v1","addr":"172.21.15.32:6805","nonce":9088303}]},"public_addr":"172.21.15.32:6801/9088303","cluster_addr":"172.21.15.32:6803/9088303","heartbeat_back_addr":"172.21.15.32:6807/9088303","heartbeat_front_addr":"172.21.15.32:6805/9088303","state":["exists","up"]},{"osd":4,"uuid":"4f902eff-1e18-418d-82c9-2151c69524fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6802","nonce":864752275},{"type":"v1","addr":"172.21.15.118:6803","nonce":864752275}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6804","nonce":864752275},{"type":"v1","addr":"172.21.15.118:6805","nonce":864752275}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6808","nonce":864752275},{"type":"v1","addr":"172.21.15.118:6809","nonce":864752275}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6806","nonce":864752275},{"type":"v1","addr":"172.21.15.118:6807","nonce":864752275}]},"public_addr":"172.21.15.118:6803/864752275","cluster_addr":"172.21.15.118:6805/864752275","heartbeat_back_addr":"172.21.15.118:6809/864752275","heartbeat_front_addr":"172.21.15.118:6807/864752275","state":["exists","up"]},{"osd":5,"uuid":"bfd9feac-dd06-458c-91b9-af7f175f612c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6810","nonce":2724891770},{"type":"v1","addr":"172.21.15.118:6811","nonce":2724891770}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6812","nonce":2724891770},{"type":"v1","addr":"172.21.15.118:6813","nonce":2724891770}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6816","nonce":2724891770},{"type":"v1","addr":"172.21.15.118:6817","nonce":2724891770}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6814","nonce":2724891770},{"type":"v1","addr":"172.21.15.118:6815","nonce":2724891770}]},"public_addr":"172.21.15.118:6811/2724891770","cluster_addr":"172.21.15.118:6813/2724891770","heartbeat_back_addr":"172.21.15.118:6817/2724891770","heartbeat_front_addr":"172.21.15.118:6815/2724891770","state":["exists","up"]},{"osd":6,"uuid":"5ed5a279-9d8e-416d-bbc7-7edd3e6d5705","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6825","nonce":801947386},{"type":"v1","addr":"172.21.15.118:6827","nonce":801947386}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6828","nonce":801947386},{"type":"v1","addr":"172.21.15.118:6829","nonce":801947386}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6832","nonce":801947386},{"type":"v1","addr":"172.21.15.118:6833","nonce":801947386}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6830","nonce":801947386},{"type":"v1","addr":"172.21.15.118:6831","nonce":801947386}]},"public_addr":"172.21.15.118:6827/801947386","cluster_addr":"172.21.15.118:6829/801947386","heartbeat_back_addr":"172.21.15.118:6833/801947386","heartbeat_front_addr":"172.21.15.118:6831/801947386","state":["exists","up"]},{"osd":7,"uuid":"f2086c47-875e-4451-a436-2008d1ea16af","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6818","nonce":653000048},{"type":"v1","addr":"172.21.15.118:6819","nonce":653000048}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6820","nonce":653000048},{"type":"v1","addr":"172.21.15.118:6821","nonce":653000048}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6824","nonce":653000048},{"type":"v1","addr":"172.21.15.118:6826","nonce":653000048}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6822","nonce":653000048},{"type":"v1","addr":"172.21.15.118:6823","nonce":653000048}]},"public_addr":"172.21.15.118:6819/653000048","cluster_addr":"172.21.15.118:6821/653000048","heartbeat_back_addr":"172.21.15.118:6826/653000048","heartbeat_front_addr":"172.21.15.118:6823/653000048","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-03T20:47:25.491994+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-03T20:47:25.587358+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-03T20:47:25.454175+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-03T20:47:25.377734+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-03T20:47:25.412351+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-03T20:47:25.499815+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-03T20:47:25.575044+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-03T20:47:25.552383+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-08-03T20:48:15.378 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:15.377+0000 7fa835200640 1 -- 172.21.15.32:0/3116510809 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fa810077040 msgr2=0x7fa810079550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:48:15.378 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:15.377+0000 7fa835200640 1 --2- 172.21.15.32:0/3116510809 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fa810077040 0x7fa810079550 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fa81c034000 tx=0x7fa81c004730 comp rx=0 tx=0).stop 2024-08-03T20:48:15.378 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:15.377+0000 7fa835200640 1 -- 172.21.15.32:0/3116510809 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa83006e930 msgr2=0x7fa8300b2fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:48:15.378 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:15.377+0000 7fa835200640 1 --2- 172.21.15.32:0/3116510809 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa83006e930 0x7fa8300b2fd0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fa82000e790 tx=0x7fa82000ec50 comp rx=0 tx=0).stop 2024-08-03T20:48:15.378 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:15.377+0000 7fa835200640 1 -- 172.21.15.32:0/3116510809 shutdown_connections 2024-08-03T20:48:15.378 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:15.377+0000 7fa835200640 1 --2- 172.21.15.32:0/3116510809 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fa810077040 0x7fa810079550 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:15.378 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:15.377+0000 7fa835200640 1 --2- 172.21.15.32:0/3116510809 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa8300747b0 0x7fa8301363c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:15.378 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:15.377+0000 7fa835200640 1 --2- 172.21.15.32:0/3116510809 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fa830073e60 0x7fa8300b3e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:15.378 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:15.377+0000 7fa835200640 1 --2- 172.21.15.32:0/3116510809 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa83006e930 0x7fa8300b2fd0 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:15.378 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:15.377+0000 7fa835200640 1 -- 172.21.15.32:0/3116510809 >> 172.21.15.32:0/3116510809 conn(0x7fa8300b1bd0 msgr2=0x7fa8300b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:48:15.378 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:15.377+0000 7fa835200640 1 -- 172.21.15.32:0/3116510809 shutdown_connections 2024-08-03T20:48:15.378 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:15.377+0000 7fa835200640 1 -- 172.21.15.32:0/3116510809 wait complete. 2024-08-03T20:48:15.401 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-08-03T20:48:15.629 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:15.628+0000 7f4d33400640 1 -- 172.21.15.32:0/3019464991 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f4d2c15fff0 msgr2=0x7f4d2c160400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:48:15.629 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:15.628+0000 7f4d33400640 1 --2- 172.21.15.32:0/3019464991 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f4d2c15fff0 0x7f4d2c160400 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f4d24005fd0 tx=0x7f4d2402a8b0 comp rx=0 tx=0).stop 2024-08-03T20:48:15.629 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:15.628+0000 7f4d33400640 1 -- 172.21.15.32:0/3019464991 shutdown_connections 2024-08-03T20:48:15.629 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:15.628+0000 7f4d33400640 1 --2- 172.21.15.32:0/3019464991 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f4d2c15fff0 0x7f4d2c160400 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:15.629 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:15.628+0000 7f4d33400640 1 --2- 172.21.15.32:0/3019464991 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f4d2c06e930 0x7f4d2c144f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:15.629 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:15.628+0000 7f4d33400640 1 -- 172.21.15.32:0/3019464991 >> 172.21.15.32:0/3019464991 conn(0x7f4d2c0b1bd0 msgr2=0x7f4d2c0b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:48:15.629 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:15.628+0000 7f4d33400640 1 -- 172.21.15.32:0/3019464991 shutdown_connections 2024-08-03T20:48:15.629 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:15.628+0000 7f4d33400640 1 -- 172.21.15.32:0/3019464991 wait complete. 2024-08-03T20:48:15.631 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:15.630+0000 7f4d33400640 1 Processor -- start 2024-08-03T20:48:15.631 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:15.631+0000 7f4d33400640 1 -- start start 2024-08-03T20:48:15.633 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:15.632+0000 7f4d33400640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f4d2c06e930 0x7f4d2c13d830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:48:15.633 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:15.632+0000 7f4d33400640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f4d2c073e60 0x7f4d2c0b4b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:48:15.634 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:15.632+0000 7f4d33400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4d2c15fff0 0x7f4d2c15ab00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:48:15.634 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:15.632+0000 7f4d33400640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f4d2c0cf840 con 0x7f4d2c15fff0 2024-08-03T20:48:15.634 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:15.632+0000 7f4d33400640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f4d2c145f30 con 0x7f4d2c073e60 2024-08-03T20:48:15.634 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:15.632+0000 7f4d33400640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7f4d2c0cf9c0 con 0x7f4d2c06e930 2024-08-03T20:48:15.634 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:15.632+0000 7f4d31400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4d2c15fff0 0x7f4d2c15ab00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:48:15.634 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:15.632+0000 7f4d31400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4d2c15fff0 0x7f4d2c15ab00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:53196/0 (socket says 172.21.15.32:53196) 2024-08-03T20:48:15.634 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:15.632+0000 7f4d31400640 1 -- 172.21.15.32:0/149292119 learned_addr learned my addr 172.21.15.32:0/149292119 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:48:15.634 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:15.633+0000 7f4d2be00640 1 --2- 172.21.15.32:0/149292119 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f4d2c073e60 0x7f4d2c0b4b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:48:15.634 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:15.633+0000 7f4d30a00640 1 --2- 172.21.15.32:0/149292119 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f4d2c06e930 0x7f4d2c13d830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:48:15.634 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:15.633+0000 7f4d31400640 1 -- 172.21.15.32:0/149292119 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f4d2c06e930 msgr2=0x7f4d2c13d830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:48:15.634 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:15.633+0000 7f4d31400640 1 --2- 172.21.15.32:0/149292119 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f4d2c06e930 0x7f4d2c13d830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:15.634 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:15.633+0000 7f4d31400640 1 -- 172.21.15.32:0/149292119 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f4d2c073e60 msgr2=0x7f4d2c0b4b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:48:15.635 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:15.633+0000 7f4d31400640 1 --2- 172.21.15.32:0/149292119 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f4d2c073e60 0x7f4d2c0b4b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:15.635 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:15.633+0000 7f4d31400640 1 -- 172.21.15.32:0/149292119 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4d2c0b3ee0 con 0x7f4d2c15fff0 2024-08-03T20:48:15.635 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:15.633+0000 7f4d31400640 1 --2- 172.21.15.32:0/149292119 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4d2c15fff0 0x7f4d2c15ab00 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f4d24009cd0 tx=0x7f4d240316f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:48:15.635 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:15.633+0000 7f4d28c00640 1 -- 172.21.15.32:0/149292119 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d24031da0 con 0x7f4d2c15fff0 2024-08-03T20:48:15.635 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:15.634+0000 7f4d28c00640 1 -- 172.21.15.32:0/149292119 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f4d24003a60 con 0x7f4d2c15fff0 2024-08-03T20:48:15.636 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:15.634+0000 7f4d28c00640 1 -- 172.21.15.32:0/149292119 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d24033070 con 0x7f4d2c15fff0 2024-08-03T20:48:15.636 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:15.634+0000 7f4d33400640 1 -- 172.21.15.32:0/149292119 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4d2c13ca70 con 0x7f4d2c15fff0 2024-08-03T20:48:15.636 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:15.634+0000 7f4d33400640 1 -- 172.21.15.32:0/149292119 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4d2c13d030 con 0x7f4d2c15fff0 2024-08-03T20:48:15.636 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:15.635+0000 7f4d33400640 1 -- 172.21.15.32:0/149292119 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4d2c160bc0 con 0x7f4d2c15fff0 2024-08-03T20:48:15.636 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:15.636+0000 7f4d28c00640 1 -- 172.21.15.32:0/149292119 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f4d24033210 con 0x7f4d2c15fff0 2024-08-03T20:48:15.637 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:15.637+0000 7f4d28c00640 1 --2- 172.21.15.32:0/149292119 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f4d0c077040 0x7f4d0c079550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:48:15.637 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:15.637+0000 7f4d28c00640 1 -- 172.21.15.32:0/149292119 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f4d240b98d0 con 0x7f4d2c15fff0 2024-08-03T20:48:15.638 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:15.638+0000 7f4d30a00640 1 --2- 172.21.15.32:0/149292119 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f4d0c077040 0x7f4d0c079550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:48:15.639 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:15.638+0000 7f4d30a00640 1 --2- 172.21.15.32:0/149292119 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f4d0c077040 0x7f4d0c079550 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f4d20009b30 tx=0x7f4d20005f90 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:48:15.641 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:15.641+0000 7f4d28c00640 1 -- 172.21.15.32:0/149292119 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7f4d2404a030 con 0x7f4d2c15fff0 2024-08-03T20:48:15.884 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:15.883+0000 7f4d33400640 1 -- 172.21.15.32:0/149292119 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7f4d2c145120 con 0x7f4d2c15fff0 2024-08-03T20:48:15.884 INFO:tasks.ceph.mon.a.smithi032.stderr:2024-08-03T20:48:15.884+0000 7f4e6d600640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-08-03T20:48:15.887 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:15.886+0000 7f4d28c00640 1 -- 172.21.15.32:0/149292119 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v9) ==== 161+0+0 (secure 0 0 0) 0x7f4d2403f030 con 0x7f4d2c15fff0 2024-08-03T20:48:15.887 INFO:teuthology.orchestra.run.smithi032.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-08-03T20:48:15.889 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:15.889+0000 7f4d33400640 1 -- 172.21.15.32:0/149292119 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f4d0c077040 msgr2=0x7f4d0c079550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:48:15.890 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:15.889+0000 7f4d33400640 1 --2- 172.21.15.32:0/149292119 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f4d0c077040 0x7f4d0c079550 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f4d20009b30 tx=0x7f4d20005f90 comp rx=0 tx=0).stop 2024-08-03T20:48:15.890 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:15.889+0000 7f4d33400640 1 -- 172.21.15.32:0/149292119 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4d2c15fff0 msgr2=0x7f4d2c15ab00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:48:15.890 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:15.889+0000 7f4d33400640 1 --2- 172.21.15.32:0/149292119 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4d2c15fff0 0x7f4d2c15ab00 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f4d24009cd0 tx=0x7f4d240316f0 comp rx=0 tx=0).stop 2024-08-03T20:48:15.890 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:15.889+0000 7f4d33400640 1 -- 172.21.15.32:0/149292119 shutdown_connections 2024-08-03T20:48:15.890 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:15.889+0000 7f4d33400640 1 --2- 172.21.15.32:0/149292119 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f4d0c077040 0x7f4d0c079550 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:15.890 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:15.889+0000 7f4d33400640 1 --2- 172.21.15.32:0/149292119 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f4d2c073e60 0x7f4d2c0b4b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:15.890 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:15.889+0000 7f4d33400640 1 --2- 172.21.15.32:0/149292119 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4d2c15fff0 0x7f4d2c15ab00 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:15.890 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:15.889+0000 7f4d33400640 1 --2- 172.21.15.32:0/149292119 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f4d2c06e930 0x7f4d2c13d830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:15.890 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:15.889+0000 7f4d33400640 1 -- 172.21.15.32:0/149292119 >> 172.21.15.32:0/149292119 conn(0x7f4d2c0b1bd0 msgr2=0x7f4d2c15d7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:48:15.891 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:15.890+0000 7f4d33400640 1 -- 172.21.15.32:0/149292119 shutdown_connections 2024-08-03T20:48:15.891 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:15.890+0000 7f4d33400640 1 -- 172.21.15.32:0/149292119 wait complete. 2024-08-03T20:48:15.902 INFO:tasks.ceph.mds.c.smithi032.stderr: -7180> 2024-08-03T20:47:40.320+0000 7f8eee600640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-08-03T20:48:15.913 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-08-03T20:48:16.030 INFO:tasks.ceph.mds.c.smithi032.stdout:starting mds.c at 2024-08-03T20:48:16.150 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:16.149+0000 7fa5b1200640 1 -- 172.21.15.32:0/761944145 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa5ac0b0160 msgr2=0x7fa5ac074480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:48:16.150 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:16.149+0000 7fa5b1200640 1 --2- 172.21.15.32:0/761944145 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa5ac0b0160 0x7fa5ac074480 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fa59c009830 tx=0x7fa59c02a5a0 comp rx=0 tx=0).stop 2024-08-03T20:48:16.150 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:16.149+0000 7fa5b1200640 1 -- 172.21.15.32:0/761944145 shutdown_connections 2024-08-03T20:48:16.150 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:16.149+0000 7fa5b1200640 1 --2- 172.21.15.32:0/761944145 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fa5ac06c9e0 0x7fa5ac06cdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:16.150 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:16.149+0000 7fa5b1200640 1 --2- 172.21.15.32:0/761944145 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa5ac0b0160 0x7fa5ac074480 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:16.150 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:16.149+0000 7fa5b1200640 1 -- 172.21.15.32:0/761944145 >> 172.21.15.32:0/761944145 conn(0x7fa5ac0b32f0 msgr2=0x7fa5ac0d1050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:48:16.150 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:16.149+0000 7fa5b1200640 1 -- 172.21.15.32:0/761944145 shutdown_connections 2024-08-03T20:48:16.150 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:16.149+0000 7fa5b1200640 1 -- 172.21.15.32:0/761944145 wait complete. 2024-08-03T20:48:16.152 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:16.151+0000 7fa5b1200640 1 Processor -- start 2024-08-03T20:48:16.152 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:16.151+0000 7fa5b1200640 1 -- start start 2024-08-03T20:48:16.153 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:16.153+0000 7fa5b1200640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fa5ac06c9e0 0x7fa5ac0d0210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:48:16.153 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:16.153+0000 7fa5b1200640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa5ac0b0160 0x7fa5ac153200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:48:16.154 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:16.153+0000 7fa5b1200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa5ac158c60 0x7fa5ac1f5820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:48:16.154 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:16.153+0000 7fa5b1200640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fa5ac162780 con 0x7fa5ac158c60 2024-08-03T20:48:16.154 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:16.153+0000 7fa5b1200640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7fa5ac162600 con 0x7fa5ac0b0160 2024-08-03T20:48:16.154 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:16.153+0000 7fa5b1200640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7fa5ac162900 con 0x7fa5ac06c9e0 2024-08-03T20:48:16.154 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:16.153+0000 7fa5aa800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa5ac158c60 0x7fa5ac1f5820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:48:16.154 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:16.153+0000 7fa5aa800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa5ac158c60 0x7fa5ac1f5820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:53204/0 (socket says 172.21.15.32:53204) 2024-08-03T20:48:16.154 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:16.153+0000 7fa5aa800640 1 -- 172.21.15.32:0/2908712768 learned_addr learned my addr 172.21.15.32:0/2908712768 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:48:16.155 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:16.153+0000 7fa5a9e00640 1 --2- 172.21.15.32:0/2908712768 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fa5ac06c9e0 0x7fa5ac0d0210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:48:16.155 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:16.153+0000 7fa5a3e00640 1 --2- 172.21.15.32:0/2908712768 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa5ac0b0160 0x7fa5ac153200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:48:16.156 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:16.153+0000 7fa5aa800640 1 -- 172.21.15.32:0/2908712768 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fa5ac06c9e0 msgr2=0x7fa5ac0d0210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:48:16.156 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:16.153+0000 7fa5aa800640 1 --2- 172.21.15.32:0/2908712768 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fa5ac06c9e0 0x7fa5ac0d0210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:16.156 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:16.153+0000 7fa5aa800640 1 -- 172.21.15.32:0/2908712768 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa5ac0b0160 msgr2=0x7fa5ac153200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:48:16.156 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:16.153+0000 7fa5aa800640 1 --2- 172.21.15.32:0/2908712768 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa5ac0b0160 0x7fa5ac153200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:16.156 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:16.154+0000 7fa5aa800640 1 -- 172.21.15.32:0/2908712768 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa5ac15fa80 con 0x7fa5ac158c60 2024-08-03T20:48:16.156 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:16.154+0000 7fa5a3e00640 1 --2- 172.21.15.32:0/2908712768 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa5ac0b0160 0x7fa5ac153200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-03T20:48:16.156 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:16.154+0000 7fa5aa800640 1 --2- 172.21.15.32:0/2908712768 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa5ac158c60 0x7fa5ac1f5820 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fa5a400ea80 tx=0x7fa5a400ef40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:48:16.156 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:16.154+0000 7fa5a2000640 1 -- 172.21.15.32:0/2908712768 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5a400ce40 con 0x7fa5ac158c60 2024-08-03T20:48:16.156 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:16.154+0000 7fa5a2000640 1 -- 172.21.15.32:0/2908712768 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fa5a4004b20 con 0x7fa5ac158c60 2024-08-03T20:48:16.156 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:16.154+0000 7fa5a2000640 1 -- 172.21.15.32:0/2908712768 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5a4005020 con 0x7fa5ac158c60 2024-08-03T20:48:16.156 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:16.154+0000 7fa5b1200640 1 -- 172.21.15.32:0/2908712768 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa5ac0d0750 con 0x7fa5ac158c60 2024-08-03T20:48:16.156 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:16.154+0000 7fa5b1200640 1 -- 172.21.15.32:0/2908712768 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa5ac160000 con 0x7fa5ac158c60 2024-08-03T20:48:16.157 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:16.155+0000 7fa5b1200640 1 -- 172.21.15.32:0/2908712768 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa5ac0b3bb0 con 0x7fa5ac158c60 2024-08-03T20:48:16.157 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:16.156+0000 7fa5a2000640 1 -- 172.21.15.32:0/2908712768 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7fa5a40051c0 con 0x7fa5ac158c60 2024-08-03T20:48:16.159 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:16.158+0000 7fa5a2000640 1 --2- 172.21.15.32:0/2908712768 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fa588077010 0x7fa588079520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:48:16.159 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:16.158+0000 7fa5a2000640 1 -- 172.21.15.32:0/2908712768 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(22..22 src has 1..22) ==== 5795+0+0 (secure 0 0 0) 0x7fa5a4099620 con 0x7fa5ac158c60 2024-08-03T20:48:16.159 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:16.158+0000 7fa5a9e00640 1 --2- 172.21.15.32:0/2908712768 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fa588077010 0x7fa588079520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:48:16.159 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:16.159+0000 7fa5a9e00640 1 --2- 172.21.15.32:0/2908712768 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fa588077010 0x7fa588079520 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fa59c034000 tx=0x7fa59c004730 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:48:16.163 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:16.162+0000 7fa5a2000640 1 -- 172.21.15.32:0/2908712768 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7fa5ac0b3bb0 con 0x7fa5ac158c60 2024-08-03T20:48:16.409 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:16.409+0000 7fa5b1200640 1 -- 172.21.15.32:0/2908712768 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) -- 0x7fa5ac162e20 con 0x7fa5ac158c60 2024-08-03T20:48:16.889 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:16.888+0000 7fa5a2000640 1 -- 172.21.15.32:0/2908712768 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v10) ==== 104+0+0 (secure 0 0 0) 0x7fa5ac15fa80 con 0x7fa5ac158c60 2024-08-03T20:48:16.891 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:16.890+0000 7fa5b1200640 1 -- 172.21.15.32:0/2908712768 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fa588077010 msgr2=0x7fa588079520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:48:16.891 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:16.890+0000 7fa5b1200640 1 --2- 172.21.15.32:0/2908712768 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fa588077010 0x7fa588079520 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fa59c034000 tx=0x7fa59c004730 comp rx=0 tx=0).stop 2024-08-03T20:48:16.891 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:16.890+0000 7fa5b1200640 1 -- 172.21.15.32:0/2908712768 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa5ac158c60 msgr2=0x7fa5ac1f5820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:48:16.892 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:16.890+0000 7fa5b1200640 1 --2- 172.21.15.32:0/2908712768 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa5ac158c60 0x7fa5ac1f5820 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fa5a400ea80 tx=0x7fa5a400ef40 comp rx=0 tx=0).stop 2024-08-03T20:48:16.892 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:16.891+0000 7fa5b1200640 1 -- 172.21.15.32:0/2908712768 shutdown_connections 2024-08-03T20:48:16.892 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:16.891+0000 7fa5b1200640 1 --2- 172.21.15.32:0/2908712768 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fa588077010 0x7fa588079520 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:16.892 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:16.891+0000 7fa5b1200640 1 --2- 172.21.15.32:0/2908712768 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa5ac0b0160 0x7fa5ac153200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:16.892 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:16.891+0000 7fa5b1200640 1 --2- 172.21.15.32:0/2908712768 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa5ac158c60 0x7fa5ac1f5820 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:16.892 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:16.891+0000 7fa5b1200640 1 --2- 172.21.15.32:0/2908712768 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fa5ac06c9e0 0x7fa5ac0d0210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:16.892 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:16.891+0000 7fa5b1200640 1 -- 172.21.15.32:0/2908712768 >> 172.21.15.32:0/2908712768 conn(0x7fa5ac0b32f0 msgr2=0x7fa5ac156610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:48:16.892 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:16.891+0000 7fa5b1200640 1 -- 172.21.15.32:0/2908712768 shutdown_connections 2024-08-03T20:48:16.892 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:16.891+0000 7fa5b1200640 1 -- 172.21.15.32:0/2908712768 wait complete. 2024-08-03T20:48:16.913 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-08-03T20:48:17.147 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:17.146+0000 7f327ce00640 1 -- 172.21.15.32:0/1992034851 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f32780cccf0 msgr2=0x7f32780cd100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:48:17.147 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:17.146+0000 7f327ce00640 1 --2- 172.21.15.32:0/1992034851 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f32780cccf0 0x7f32780cd100 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f32780cca80 tx=0x7f326002a4a0 comp rx=0 tx=0).stop 2024-08-03T20:48:17.147 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:17.146+0000 7f327ce00640 1 -- 172.21.15.32:0/1992034851 shutdown_connections 2024-08-03T20:48:17.147 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:17.146+0000 7f327ce00640 1 --2- 172.21.15.32:0/1992034851 >> v2:172.21.15.118:3300/0 conn(0x7f32780ced30 0x7f32780cf140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:17.147 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:17.146+0000 7f327ce00640 1 --2- 172.21.15.32:0/1992034851 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f32780cccf0 0x7f32780cd100 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:17.147 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:17.146+0000 7f327ce00640 1 -- 172.21.15.32:0/1992034851 >> 172.21.15.32:0/1992034851 conn(0x7f32780b1bd0 msgr2=0x7f32780b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:48:17.148 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:17.146+0000 7f327ce00640 1 -- 172.21.15.32:0/1992034851 shutdown_connections 2024-08-03T20:48:17.148 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:17.146+0000 7f327ce00640 1 -- 172.21.15.32:0/1992034851 wait complete. 2024-08-03T20:48:17.149 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:17.148+0000 7f327ce00640 1 Processor -- start 2024-08-03T20:48:17.149 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:17.149+0000 7f327ce00640 1 -- start start 2024-08-03T20:48:17.151 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:17.150+0000 7f327ce00640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f32780cccf0 0x7f3278159e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:48:17.151 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:17.150+0000 7f327ce00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f32780ced30 0x7f32780d0560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:48:17.151 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:17.150+0000 7f327ce00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3278145c50 0x7f327813e5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:48:17.151 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:17.150+0000 7f327ce00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f32780b4070 con 0x7f3278145c50 2024-08-03T20:48:17.151 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:17.150+0000 7f327ce00640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f32780b3ef0 con 0x7f32780ced30 2024-08-03T20:48:17.151 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:17.150+0000 7f327ce00640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7f32780b41f0 con 0x7f32780cccf0 2024-08-03T20:48:17.151 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:17.150+0000 7f3276800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3278145c50 0x7f327813e5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:48:17.151 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:17.150+0000 7f3276800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3278145c50 0x7f327813e5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:53216/0 (socket says 172.21.15.32:53216) 2024-08-03T20:48:17.152 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:17.150+0000 7f3276800640 1 -- 172.21.15.32:0/428269912 learned_addr learned my addr 172.21.15.32:0/428269912 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:48:17.152 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:17.150+0000 7f3275e00640 1 --2- 172.21.15.32:0/428269912 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f32780cccf0 0x7f3278159e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:48:17.152 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:17.150+0000 7f326d400640 1 --2- 172.21.15.32:0/428269912 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f32780ced30 0x7f32780d0560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:48:17.152 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:17.151+0000 7f3276800640 1 -- 172.21.15.32:0/428269912 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f32780cccf0 msgr2=0x7f3278159e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:48:17.152 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:17.151+0000 7f3276800640 1 --2- 172.21.15.32:0/428269912 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f32780cccf0 0x7f3278159e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:17.152 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:17.151+0000 7f3276800640 1 -- 172.21.15.32:0/428269912 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f32780ced30 msgr2=0x7f32780d0560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:48:17.152 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:17.151+0000 7f3276800640 1 --2- 172.21.15.32:0/428269912 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f32780ced30 0x7f32780d0560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:17.152 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:17.151+0000 7f3276800640 1 -- 172.21.15.32:0/428269912 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f32781f5fc0 con 0x7f3278145c50 2024-08-03T20:48:17.153 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:17.151+0000 7f3276800640 1 --2- 172.21.15.32:0/428269912 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3278145c50 0x7f327813e5a0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f327000dac0 tx=0x7f327000df80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:48:17.153 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:17.151+0000 7f326ea00640 1 -- 172.21.15.32:0/428269912 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3270017070 con 0x7f3278145c50 2024-08-03T20:48:17.153 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:17.151+0000 7f326ea00640 1 -- 172.21.15.32:0/428269912 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f3270003970 con 0x7f3278145c50 2024-08-03T20:48:17.153 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:17.151+0000 7f326ea00640 1 -- 172.21.15.32:0/428269912 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3270003dc0 con 0x7f3278145c50 2024-08-03T20:48:17.153 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:17.151+0000 7f327ce00640 1 -- 172.21.15.32:0/428269912 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f327813cab0 con 0x7f3278145c50 2024-08-03T20:48:17.154 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:17.151+0000 7f327ce00640 1 -- 172.21.15.32:0/428269912 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f327813d070 con 0x7f3278145c50 2024-08-03T20:48:17.154 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:17.153+0000 7f327ce00640 1 -- 172.21.15.32:0/428269912 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f32780b4710 con 0x7f3278145c50 2024-08-03T20:48:17.154 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:17.154+0000 7f326ea00640 1 -- 172.21.15.32:0/428269912 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f3270003f60 con 0x7f3278145c50 2024-08-03T20:48:17.155 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:17.155+0000 7f326ea00640 1 --2- 172.21.15.32:0/428269912 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f3254077040 0x7f3254079550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:48:17.155 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:17.155+0000 7f326ea00640 1 -- 172.21.15.32:0/428269912 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(22..22 src has 1..22) ==== 5795+0+0 (secure 0 0 0) 0x7f3270098cf0 con 0x7f3278145c50 2024-08-03T20:48:17.155 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:17.155+0000 7f3275e00640 1 --2- 172.21.15.32:0/428269912 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f3254077040 0x7f3254079550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:48:17.156 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:17.156+0000 7f3275e00640 1 --2- 172.21.15.32:0/428269912 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f3254077040 0x7f3254079550 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f3268000f60 tx=0x7f3268009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:48:17.159 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:17.159+0000 7f326ea00640 1 -- 172.21.15.32:0/428269912 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7f3270061830 con 0x7f3278145c50 2024-08-03T20:48:17.365 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:17.365+0000 7f327ce00640 1 -- 172.21.15.32:0/428269912 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) -- 0x7f327813c800 con 0x7f3278145c50 2024-08-03T20:48:18.072 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:18.071+0000 7f326ea00640 1 -- 172.21.15.32:0/428269912 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v23) ==== 181+0+0 (secure 0 0 0) 0x7f3270065d80 con 0x7f3278145c50 2024-08-03T20:48:18.072 INFO:teuthology.orchestra.run.smithi032.stderr:pool 'cephfs_metadata' removed 2024-08-03T20:48:18.075 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:18.075+0000 7f327ce00640 1 -- 172.21.15.32:0/428269912 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f3254077040 msgr2=0x7f3254079550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:48:18.075 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:18.075+0000 7f327ce00640 1 --2- 172.21.15.32:0/428269912 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f3254077040 0x7f3254079550 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f3268000f60 tx=0x7f3268009040 comp rx=0 tx=0).stop 2024-08-03T20:48:18.075 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:18.075+0000 7f327ce00640 1 -- 172.21.15.32:0/428269912 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3278145c50 msgr2=0x7f327813e5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:48:18.075 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:18.075+0000 7f327ce00640 1 --2- 172.21.15.32:0/428269912 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3278145c50 0x7f327813e5a0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f327000dac0 tx=0x7f327000df80 comp rx=0 tx=0).stop 2024-08-03T20:48:18.076 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:18.075+0000 7f327ce00640 1 -- 172.21.15.32:0/428269912 shutdown_connections 2024-08-03T20:48:18.076 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:18.075+0000 7f327ce00640 1 --2- 172.21.15.32:0/428269912 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f3254077040 0x7f3254079550 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:18.076 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:18.075+0000 7f327ce00640 1 --2- 172.21.15.32:0/428269912 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f32780ced30 0x7f32780d0560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:18.076 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:18.075+0000 7f327ce00640 1 --2- 172.21.15.32:0/428269912 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3278145c50 0x7f327813e5a0 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:18.076 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:18.075+0000 7f327ce00640 1 --2- 172.21.15.32:0/428269912 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f32780cccf0 0x7f3278159e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:18.076 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:18.075+0000 7f327ce00640 1 -- 172.21.15.32:0/428269912 >> 172.21.15.32:0/428269912 conn(0x7f32780b1bd0 msgr2=0x7f327815b4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:48:18.076 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:18.075+0000 7f327ce00640 1 -- 172.21.15.32:0/428269912 shutdown_connections 2024-08-03T20:48:18.077 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:18.075+0000 7f327ce00640 1 -- 172.21.15.32:0/428269912 wait complete. 2024-08-03T20:48:18.100 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-08-03T20:48:18.336 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:18.335+0000 7f952b200640 1 -- 172.21.15.32:0/272061769 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f95240ced30 msgr2=0x7f95240cf140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:48:18.336 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:18.335+0000 7f952b200640 1 --2- 172.21.15.32:0/272061769 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f95240ced30 0x7f95240cf140 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f951c005fd0 tx=0x7f951c02a8a0 comp rx=0 tx=0).stop 2024-08-03T20:48:18.336 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:18.335+0000 7f952b200640 1 -- 172.21.15.32:0/272061769 shutdown_connections 2024-08-03T20:48:18.336 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:18.335+0000 7f952b200640 1 --2- 172.21.15.32:0/272061769 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f95240ced30 0x7f95240cf140 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:18.336 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:18.335+0000 7f952b200640 1 --2- 172.21.15.32:0/272061769 >> v2:172.21.15.118:3300/0 conn(0x7f95240cccf0 0x7f95240cd100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:18.337 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:18.335+0000 7f952b200640 1 -- 172.21.15.32:0/272061769 >> 172.21.15.32:0/272061769 conn(0x7f95240b1bd0 msgr2=0x7f95240b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:48:18.337 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:18.335+0000 7f952b200640 1 -- 172.21.15.32:0/272061769 shutdown_connections 2024-08-03T20:48:18.337 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:18.335+0000 7f952b200640 1 -- 172.21.15.32:0/272061769 wait complete. 2024-08-03T20:48:18.338 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:18.338+0000 7f952b200640 1 Processor -- start 2024-08-03T20:48:18.338 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:18.338+0000 7f952b200640 1 -- start start 2024-08-03T20:48:18.340 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:18.339+0000 7f952b200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f95240cccf0 0x7f952413da10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:48:18.340 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:18.339+0000 7f952b200640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f95240ced30 0x7f9524140da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:48:18.341 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:18.339+0000 7f952b200640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f9524145c50 0x7f95241412e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:48:18.341 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:18.339+0000 7f952b200640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f95240b4070 con 0x7f95240cccf0 2024-08-03T20:48:18.341 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:18.339+0000 7f952b200640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f95240b3ef0 con 0x7f9524145c50 2024-08-03T20:48:18.341 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:18.339+0000 7f952b200640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7f95240b41f0 con 0x7f95240ced30 2024-08-03T20:48:18.341 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:18.339+0000 7f9523e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f95240cccf0 0x7f952413da10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:48:18.341 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:18.339+0000 7f9523e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f95240cccf0 0x7f952413da10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:53230/0 (socket says 172.21.15.32:53230) 2024-08-03T20:48:18.341 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:18.339+0000 7f9523e00640 1 -- 172.21.15.32:0/4008472334 learned_addr learned my addr 172.21.15.32:0/4008472334 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:48:18.341 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:18.340+0000 7f9529200640 1 --2- 172.21.15.32:0/4008472334 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f9524145c50 0x7f95241412e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:48:18.341 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:18.340+0000 7f9523400640 1 --2- 172.21.15.32:0/4008472334 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f95240ced30 0x7f9524140da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:48:18.341 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:18.340+0000 7f9523e00640 1 -- 172.21.15.32:0/4008472334 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f95240ced30 msgr2=0x7f9524140da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:48:18.341 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:18.340+0000 7f9523e00640 1 --2- 172.21.15.32:0/4008472334 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f95240ced30 0x7f9524140da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:18.341 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:18.340+0000 7f9523e00640 1 -- 172.21.15.32:0/4008472334 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f9524145c50 msgr2=0x7f95241412e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:48:18.342 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:18.340+0000 7f9523e00640 1 --2- 172.21.15.32:0/4008472334 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f9524145c50 0x7f95241412e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:18.342 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:18.340+0000 7f9523e00640 1 -- 172.21.15.32:0/4008472334 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f95241f6730 con 0x7f95240cccf0 2024-08-03T20:48:18.342 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:18.340+0000 7f9523e00640 1 --2- 172.21.15.32:0/4008472334 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f95240cccf0 0x7f952413da10 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f951400c430 tx=0x7f951400c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:48:18.343 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:18.341+0000 7f951be00640 1 -- 172.21.15.32:0/4008472334 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9514011790 con 0x7f95240cccf0 2024-08-03T20:48:18.343 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:18.341+0000 7f951be00640 1 -- 172.21.15.32:0/4008472334 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f9514017070 con 0x7f95240cccf0 2024-08-03T20:48:18.343 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:18.341+0000 7f951be00640 1 -- 172.21.15.32:0/4008472334 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9514011ce0 con 0x7f95240cccf0 2024-08-03T20:48:18.343 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:18.341+0000 7f952b200640 1 -- 172.21.15.32:0/4008472334 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f95241f69a0 con 0x7f95240cccf0 2024-08-03T20:48:18.343 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:18.341+0000 7f952b200640 1 -- 172.21.15.32:0/4008472334 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f95241f6f60 con 0x7f95240cccf0 2024-08-03T20:48:18.343 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:18.342+0000 7f952b200640 1 -- 172.21.15.32:0/4008472334 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f95240b46e0 con 0x7f95240cccf0 2024-08-03T20:48:18.343 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:18.343+0000 7f951be00640 1 -- 172.21.15.32:0/4008472334 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f951401c460 con 0x7f95240cccf0 2024-08-03T20:48:18.344 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:18.344+0000 7f951be00640 1 --2- 172.21.15.32:0/4008472334 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f9504077040 0x7f9504079550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:48:18.344 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:18.344+0000 7f951be00640 1 -- 172.21.15.32:0/4008472334 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5357+0+0 (secure 0 0 0) 0x7f9514098500 con 0x7f95240cccf0 2024-08-03T20:48:18.345 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:18.344+0000 7f9523400640 1 --2- 172.21.15.32:0/4008472334 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f9504077040 0x7f9504079550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:48:18.345 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:18.345+0000 7f9523400640 1 --2- 172.21.15.32:0/4008472334 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f9504077040 0x7f9504079550 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f95240b1ba0 tx=0x7f9510005f90 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:48:18.348 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:18.348+0000 7f951be00640 1 -- 172.21.15.32:0/4008472334 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7f9514061170 con 0x7f95240cccf0 2024-08-03T20:48:18.553 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:18.553+0000 7f952b200640 1 -- 172.21.15.32:0/4008472334 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) -- 0x7f952413fb10 con 0x7f95240cccf0 2024-08-03T20:48:19.078 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:19.078+0000 7f951be00640 1 -- 172.21.15.32:0/4008472334 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v24) ==== 169+0+0 (secure 0 0 0) 0x7f95140656c0 con 0x7f95240cccf0 2024-08-03T20:48:19.079 INFO:teuthology.orchestra.run.smithi032.stderr:pool 'cephfs_data' removed 2024-08-03T20:48:19.082 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:19.082+0000 7f952b200640 1 -- 172.21.15.32:0/4008472334 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f9504077040 msgr2=0x7f9504079550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:48:19.083 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:19.082+0000 7f952b200640 1 --2- 172.21.15.32:0/4008472334 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f9504077040 0x7f9504079550 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f95240b1ba0 tx=0x7f9510005f90 comp rx=0 tx=0).stop 2024-08-03T20:48:19.083 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:19.082+0000 7f952b200640 1 -- 172.21.15.32:0/4008472334 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f95240cccf0 msgr2=0x7f952413da10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:48:19.083 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:19.082+0000 7f952b200640 1 --2- 172.21.15.32:0/4008472334 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f95240cccf0 0x7f952413da10 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f951400c430 tx=0x7f951400c8f0 comp rx=0 tx=0).stop 2024-08-03T20:48:19.083 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:19.082+0000 7f952b200640 1 -- 172.21.15.32:0/4008472334 shutdown_connections 2024-08-03T20:48:19.083 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:19.082+0000 7f952b200640 1 --2- 172.21.15.32:0/4008472334 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f9504077040 0x7f9504079550 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:19.083 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:19.082+0000 7f952b200640 1 --2- 172.21.15.32:0/4008472334 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f9524145c50 0x7f95241412e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:19.083 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:19.082+0000 7f952b200640 1 --2- 172.21.15.32:0/4008472334 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f95240ced30 0x7f9524140da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:19.083 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:19.082+0000 7f952b200640 1 --2- 172.21.15.32:0/4008472334 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f95240cccf0 0x7f952413da10 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:19.083 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:19.082+0000 7f952b200640 1 -- 172.21.15.32:0/4008472334 >> 172.21.15.32:0/4008472334 conn(0x7f95240b1bd0 msgr2=0x7f952415c970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:48:19.083 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:19.082+0000 7f952b200640 1 -- 172.21.15.32:0/4008472334 shutdown_connections 2024-08-03T20:48:19.083 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:19.082+0000 7f952b200640 1 -- 172.21.15.32:0/4008472334 wait complete. 2024-08-03T20:48:19.109 INFO:tasks.ceph.mds.a:Restarting daemon 2024-08-03T20:48:19.109 INFO:tasks.ceph.mds.a:Stopping old one... 2024-08-03T20:48:19.109 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-08-03T20:48:19.109 INFO:teuthology.orchestra.run:waiting for 300 2024-08-03T20:48:19.110 INFO:tasks.ceph.mds.c:Restarting daemon 2024-08-03T20:48:19.110 INFO:tasks.ceph.mds.c:Stopping old one... 2024-08-03T20:48:19.110 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-08-03T20:48:19.110 INFO:teuthology.orchestra.run:waiting for 300 2024-08-03T20:48:19.110 INFO:tasks.ceph.mds.b:Restarting daemon 2024-08-03T20:48:19.110 INFO:tasks.ceph.mds.b:Stopping old one... 2024-08-03T20:48:19.110 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-08-03T20:48:19.111 INFO:teuthology.orchestra.run:waiting for 300 2024-08-03T20:48:19.111 INFO:tasks.ceph.mds.d:Restarting daemon 2024-08-03T20:48:19.111 INFO:tasks.ceph.mds.d:Stopping old one... 2024-08-03T20:48:19.111 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-08-03T20:48:19.111 INFO:teuthology.orchestra.run:waiting for 300 2024-08-03T20:48:19.138 INFO:tasks.ceph.mds.b:Stopped 2024-08-03T20:48:19.138 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-08-03T20:48:19.139 INFO:tasks.ceph.mds.d:Stopped 2024-08-03T20:48:19.139 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-08-03T20:48:19.145 INFO:tasks.ceph.mds.a:Stopped 2024-08-03T20:48:19.145 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-08-03T20:48:19.146 INFO:tasks.ceph.mds.c:Stopped 2024-08-03T20:48:19.147 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-08-03T20:48:19.183 INFO:tasks.ceph.mds.b:Started 2024-08-03T20:48:19.183 DEBUG:teuthology.parallel:result is None 2024-08-03T20:48:19.185 INFO:tasks.ceph.mds.d:Started 2024-08-03T20:48:19.186 DEBUG:teuthology.parallel:result is None 2024-08-03T20:48:19.191 INFO:tasks.ceph.mds.a:Started 2024-08-03T20:48:19.191 DEBUG:teuthology.parallel:result is None 2024-08-03T20:48:19.194 INFO:tasks.ceph.mds.c:Started 2024-08-03T20:48:19.194 DEBUG:teuthology.parallel:result is None 2024-08-03T20:48:19.194 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-08-03T20:48:19.386 INFO:tasks.ceph.mds.b.smithi118.stdout:starting mds.b at 2024-08-03T20:48:19.387 INFO:tasks.ceph.mds.d.smithi118.stdout:starting mds.d at 2024-08-03T20:48:19.389 INFO:tasks.ceph.mds.a.smithi032.stdout:starting mds.a at 2024-08-03T20:48:19.389 INFO:tasks.ceph.mds.c.smithi032.stdout:starting mds.c at 2024-08-03T20:48:19.432 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:19.431+0000 7fdd0cc00640 1 -- 172.21.15.32:0/2563239496 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdd080cffe0 msgr2=0x7fdd080b3e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:48:19.432 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:19.431+0000 7fdd0cc00640 1 --2- 172.21.15.32:0/2563239496 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdd080cffe0 0x7fdd080b3e90 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fdcf0005fd0 tx=0x7fdcf002a5d0 comp rx=0 tx=0).stop 2024-08-03T20:48:19.432 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:19.431+0000 7fdd0cc00640 1 -- 172.21.15.32:0/2563239496 shutdown_connections 2024-08-03T20:48:19.432 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:19.431+0000 7fdd0cc00640 1 --2- 172.21.15.32:0/2563239496 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdd080cffe0 0x7fdd080b3e90 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:19.432 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:19.431+0000 7fdd0cc00640 1 -- 172.21.15.32:0/2563239496 >> 172.21.15.32:0/2563239496 conn(0x7fdd0808d7a0 msgr2=0x7fdd0809d3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:48:19.432 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:19.431+0000 7fdd0cc00640 1 -- 172.21.15.32:0/2563239496 shutdown_connections 2024-08-03T20:48:19.432 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:19.431+0000 7fdd0cc00640 1 -- 172.21.15.32:0/2563239496 wait complete. 2024-08-03T20:48:19.433 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:19.433+0000 7fdd0cc00640 1 Processor -- start 2024-08-03T20:48:19.433 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:19.433+0000 7fdd0cc00640 1 -- start start 2024-08-03T20:48:19.436 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:19.435+0000 7fdd0cc00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fdd080b43d0 0x7fdd0814b240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:48:19.436 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:19.435+0000 7fdd0cc00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdd080cffe0 0x7fdd080cb6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:48:19.436 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:19.435+0000 7fdd0cc00640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fdd080d26e0 0x7fdd081f8880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:48:19.436 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:19.435+0000 7fdd0cc00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fdd080d4310 con 0x7fdd080cffe0 2024-08-03T20:48:19.436 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:19.435+0000 7fdd0cc00640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7fdd080d4190 con 0x7fdd080b43d0 2024-08-03T20:48:19.436 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:19.435+0000 7fdd0cc00640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7fdd080d4490 con 0x7fdd080d26e0 2024-08-03T20:48:19.436 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:19.435+0000 7fdcffe00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdd080cffe0 0x7fdd080cb6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:48:19.436 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:19.435+0000 7fdcffe00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdd080cffe0 0x7fdd080cb6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:53274/0 (socket says 172.21.15.32:53274) 2024-08-03T20:48:19.436 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:19.435+0000 7fdcffe00640 1 -- 172.21.15.32:0/2964193832 learned_addr learned my addr 172.21.15.32:0/2964193832 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:48:19.436 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:19.435+0000 7fdd07400640 1 --2- 172.21.15.32:0/2964193832 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fdd080b43d0 0x7fdd0814b240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:48:19.436 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:19.435+0000 7fdd07e00640 1 --2- 172.21.15.32:0/2964193832 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fdd080d26e0 0x7fdd081f8880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:48:19.437 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:19.435+0000 7fdcffe00640 1 -- 172.21.15.32:0/2964193832 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fdd080d26e0 msgr2=0x7fdd081f8880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:48:19.437 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:19.435+0000 7fdcffe00640 1 --2- 172.21.15.32:0/2964193832 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fdd080d26e0 0x7fdd081f8880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:19.437 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:19.435+0000 7fdcffe00640 1 -- 172.21.15.32:0/2964193832 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fdd080b43d0 msgr2=0x7fdd0814b240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:48:19.437 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:19.435+0000 7fdcffe00640 1 --2- 172.21.15.32:0/2964193832 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fdd080b43d0 0x7fdd0814b240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:19.437 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:19.435+0000 7fdcffe00640 1 -- 172.21.15.32:0/2964193832 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdd081f96f0 con 0x7fdd080cffe0 2024-08-03T20:48:19.438 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:19.436+0000 7fdcffe00640 1 --2- 172.21.15.32:0/2964193832 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdd080cffe0 0x7fdd080cb6c0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fdcf0034040 tx=0x7fdcf00038a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:48:19.438 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:19.436+0000 7fdcff400640 1 -- 172.21.15.32:0/2964193832 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdcf003e030 con 0x7fdd080cffe0 2024-08-03T20:48:19.438 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:19.436+0000 7fdcff400640 1 -- 172.21.15.32:0/2964193832 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fdcf0003ca0 con 0x7fdd080cffe0 2024-08-03T20:48:19.439 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:19.436+0000 7fdd0cc00640 1 -- 172.21.15.32:0/2964193832 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdd080ca750 con 0x7fdd080cffe0 2024-08-03T20:48:19.439 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:19.436+0000 7fdcff400640 1 -- 172.21.15.32:0/2964193832 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdcf0035040 con 0x7fdd080cffe0 2024-08-03T20:48:19.439 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:19.436+0000 7fdd0cc00640 1 -- 172.21.15.32:0/2964193832 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdd080c7cd0 con 0x7fdd080cffe0 2024-08-03T20:48:19.439 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:19.437+0000 7fdd0cc00640 1 -- 172.21.15.32:0/2964193832 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdd080ca4c0 con 0x7fdd080cffe0 2024-08-03T20:48:19.439 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:19.438+0000 7fdcff400640 1 -- 172.21.15.32:0/2964193832 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7fdcf0041030 con 0x7fdd080cffe0 2024-08-03T20:48:19.440 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:19.439+0000 7fdcff400640 1 --2- 172.21.15.32:0/2964193832 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fdce4077010 0x7fdce4079520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:48:19.440 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:19.440+0000 7fdcff400640 1 -- 172.21.15.32:0/2964193832 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 4943+0+0 (secure 0 0 0) 0x7fdcf00b8c60 con 0x7fdd080cffe0 2024-08-03T20:48:19.440 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:19.440+0000 7fdd07400640 1 --2- 172.21.15.32:0/2964193832 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fdce4077010 0x7fdce4079520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:48:19.441 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:19.441+0000 7fdd07400640 1 --2- 172.21.15.32:0/2964193832 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fdce4077010 0x7fdce4079520 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fdcf8001720 tx=0x7fdcf8009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:48:19.445 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:19.444+0000 7fdcff400640 1 -- 172.21.15.32:0/2964193832 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7fdcf0081a70 con 0x7fdd080cffe0 2024-08-03T20:48:19.667 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:19.667+0000 7fdd0cc00640 1 -- 172.21.15.32:0/2964193832 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) -- 0x7fdd080d4960 con 0x7fdd080cffe0 2024-08-03T20:48:20.089 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:20.089+0000 7fdcff400640 1 -- 172.21.15.32:0/2964193832 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v26) ==== 97+0+0 (secure 0 0 0) 0x7fdcf008a390 con 0x7fdd080cffe0 2024-08-03T20:48:20.089 INFO:teuthology.orchestra.run.smithi032.stderr: removed all blocklist entries 2024-08-03T20:48:20.093 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:20.091+0000 7fdd0cc00640 1 -- 172.21.15.32:0/2964193832 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fdce4077010 msgr2=0x7fdce4079520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:48:20.093 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:20.091+0000 7fdd0cc00640 1 --2- 172.21.15.32:0/2964193832 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fdce4077010 0x7fdce4079520 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fdcf8001720 tx=0x7fdcf8009040 comp rx=0 tx=0).stop 2024-08-03T20:48:20.093 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:20.091+0000 7fdd0cc00640 1 -- 172.21.15.32:0/2964193832 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdd080cffe0 msgr2=0x7fdd080cb6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:48:20.093 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:20.091+0000 7fdd0cc00640 1 --2- 172.21.15.32:0/2964193832 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdd080cffe0 0x7fdd080cb6c0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fdcf0034040 tx=0x7fdcf00038a0 comp rx=0 tx=0).stop 2024-08-03T20:48:20.093 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:20.091+0000 7fdd0cc00640 1 -- 172.21.15.32:0/2964193832 shutdown_connections 2024-08-03T20:48:20.093 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:20.091+0000 7fdd0cc00640 1 --2- 172.21.15.32:0/2964193832 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fdce4077010 0x7fdce4079520 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:20.093 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:20.091+0000 7fdd0cc00640 1 --2- 172.21.15.32:0/2964193832 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fdd080d26e0 0x7fdd081f8880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:20.093 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:20.092+0000 7fdd0cc00640 1 --2- 172.21.15.32:0/2964193832 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdd080cffe0 0x7fdd080cb6c0 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:20.093 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:20.092+0000 7fdd0cc00640 1 --2- 172.21.15.32:0/2964193832 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fdd080b43d0 0x7fdd0814b240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:20.093 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:20.092+0000 7fdd0cc00640 1 -- 172.21.15.32:0/2964193832 >> 172.21.15.32:0/2964193832 conn(0x7fdd0808d7a0 msgr2=0x7fdd080d1380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:48:20.093 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:20.092+0000 7fdd0cc00640 1 -- 172.21.15.32:0/2964193832 shutdown_connections 2024-08-03T20:48:20.093 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:20.092+0000 7fdd0cc00640 1 -- 172.21.15.32:0/2964193832 wait complete. 2024-08-03T20:48:20.117 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-08-03T20:48:20.344 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:20.344+0000 7f328c400640 1 -- 172.21.15.32:0/1866463088 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f328406e930 msgr2=0x7f3284144f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:48:20.345 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:20.344+0000 7f328c400640 1 --2- 172.21.15.32:0/1866463088 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f328406e930 0x7f3284144f90 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f3278005fd0 tx=0x7f327802a670 comp rx=0 tx=0).stop 2024-08-03T20:48:20.345 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:20.344+0000 7f328c400640 1 -- 172.21.15.32:0/1866463088 shutdown_connections 2024-08-03T20:48:20.345 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:20.344+0000 7f328c400640 1 --2- 172.21.15.32:0/1866463088 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f328406e930 0x7f3284144f90 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:20.345 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:20.344+0000 7f328c400640 1 -- 172.21.15.32:0/1866463088 >> 172.21.15.32:0/1866463088 conn(0x7f32840b1bd0 msgr2=0x7f32840b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:48:20.345 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:20.344+0000 7f328c400640 1 -- 172.21.15.32:0/1866463088 shutdown_connections 2024-08-03T20:48:20.345 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:20.344+0000 7f328c400640 1 -- 172.21.15.32:0/1866463088 wait complete. 2024-08-03T20:48:20.347 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:20.346+0000 7f328c400640 1 Processor -- start 2024-08-03T20:48:20.347 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:20.346+0000 7f328c400640 1 -- start start 2024-08-03T20:48:20.350 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:20.348+0000 7f328c400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f328406e930 0x7f328415ca00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:48:20.350 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:20.348+0000 7f328c400640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f3284073e60 0x7f328413df50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:48:20.350 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:20.348+0000 7f328c400640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f328415fff0 0x7f328415e0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:48:20.350 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:20.348+0000 7f328c400640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f32840cf840 con 0x7f328406e930 2024-08-03T20:48:20.350 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:20.348+0000 7f328c400640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f3284145f30 con 0x7f328415fff0 2024-08-03T20:48:20.350 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:20.348+0000 7f328c400640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7f32840cf9c0 con 0x7f3284073e60 2024-08-03T20:48:20.351 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:20.348+0000 7f3289a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f328406e930 0x7f328415ca00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:48:20.351 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:20.348+0000 7f3289a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f328406e930 0x7f328415ca00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:48054/0 (socket says 172.21.15.32:48054) 2024-08-03T20:48:20.351 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:20.348+0000 7f3289a00640 1 -- 172.21.15.32:0/1994890058 learned_addr learned my addr 172.21.15.32:0/1994890058 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:48:20.351 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:20.348+0000 7f328a400640 1 --2- 172.21.15.32:0/1994890058 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f328415fff0 0x7f328415e0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:48:20.351 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:20.348+0000 7f3289000640 1 --2- 172.21.15.32:0/1994890058 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f3284073e60 0x7f328413df50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:48:20.351 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:20.348+0000 7f3289a00640 1 -- 172.21.15.32:0/1994890058 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f3284073e60 msgr2=0x7f328413df50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:48:20.351 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:20.348+0000 7f3289a00640 1 --2- 172.21.15.32:0/1994890058 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f3284073e60 0x7f328413df50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:20.351 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:20.349+0000 7f3289a00640 1 -- 172.21.15.32:0/1994890058 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f328415fff0 msgr2=0x7f328415e0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:48:20.351 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:20.349+0000 7f3289a00640 1 --2- 172.21.15.32:0/1994890058 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f328415fff0 0x7f328415e0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:20.351 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:20.349+0000 7f3289a00640 1 -- 172.21.15.32:0/1994890058 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f328415d050 con 0x7f328406e930 2024-08-03T20:48:20.351 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:20.349+0000 7f3289a00640 1 --2- 172.21.15.32:0/1994890058 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f328406e930 0x7f328415ca00 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f3278034040 tx=0x7f32780039e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:48:20.351 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:20.349+0000 7f3281600640 1 -- 172.21.15.32:0/1994890058 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f327803e030 con 0x7f328406e930 2024-08-03T20:48:20.351 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:20.349+0000 7f3281600640 1 -- 172.21.15.32:0/1994890058 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f32780047a0 con 0x7f328406e930 2024-08-03T20:48:20.351 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:20.349+0000 7f328c400640 1 -- 172.21.15.32:0/1994890058 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f328415d330 con 0x7f328406e930 2024-08-03T20:48:20.352 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:20.349+0000 7f3281600640 1 -- 172.21.15.32:0/1994890058 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3278035040 con 0x7f328406e930 2024-08-03T20:48:20.352 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:20.349+0000 7f328c400640 1 -- 172.21.15.32:0/1994890058 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f328415d8f0 con 0x7f328406e930 2024-08-03T20:48:20.352 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:20.350+0000 7f328c400640 1 -- 172.21.15.32:0/1994890058 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3284160db0 con 0x7f328406e930 2024-08-03T20:48:20.352 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:20.352+0000 7f3281600640 1 -- 172.21.15.32:0/1994890058 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f3278041030 con 0x7f328406e930 2024-08-03T20:48:20.353 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:20.352+0000 7f3281600640 1 --2- 172.21.15.32:0/1994890058 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f3264077010 0x7f3264079520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:48:20.353 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:20.353+0000 7f3281600640 1 -- 172.21.15.32:0/1994890058 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(28..28 src has 1..28) ==== 4857+0+0 (secure 0 0 0) 0x7f32780b8e30 con 0x7f328406e930 2024-08-03T20:48:20.353 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:20.353+0000 7f3289000640 1 --2- 172.21.15.32:0/1994890058 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f3264077010 0x7f3264079520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:48:20.354 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:20.354+0000 7f3289000640 1 --2- 172.21.15.32:0/1994890058 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f3264077010 0x7f3264079520 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f3270000f60 tx=0x7f3270009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:48:20.357 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:20.357+0000 7f3281600640 1 -- 172.21.15.32:0/1994890058 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7f3278082370 con 0x7f328406e930 2024-08-03T20:48:20.648 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:20.647+0000 7f328c400640 1 -- 172.21.15.32:0/1994890058 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) -- 0x7f32840afdf0 con 0x7f328406e930 2024-08-03T20:48:20.649 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:20.648+0000 7f3281600640 1 -- 172.21.15.32:0/1994890058 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v4) ==== 80+0+6645 (secure 0 0 0) 0x7f3278081c70 con 0x7f328406e930 2024-08-03T20:48:20.649 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-03T20:48:20.650 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-08-03T20:48:20.650 INFO:teuthology.orchestra.run.smithi032.stdout: "auth_dump": [ 2024-08-03T20:48:20.650 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-03T20:48:20.650 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "mds.a", 2024-08-03T20:48:20.650 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQAvl65mfaKbOhAAUgCH82U0SB7owF5Cds3XYA==", 2024-08-03T20:48:20.650 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2024-08-03T20:48:20.651 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": "allow", 2024-08-03T20:48:20.651 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow *", 2024-08-03T20:48:20.651 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow *", 2024-08-03T20:48:20.651 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2024-08-03T20:48:20.651 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-03T20:48:20.652 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-03T20:48:20.652 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-03T20:48:20.652 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "mds.b", 2024-08-03T20:48:20.652 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQAwl65mNcoVGxAAlRXNjZhxVu8Vjslk9Msrxw==", 2024-08-03T20:48:20.652 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2024-08-03T20:48:20.652 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": "allow", 2024-08-03T20:48:20.652 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow *", 2024-08-03T20:48:20.653 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow *", 2024-08-03T20:48:20.653 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2024-08-03T20:48:20.653 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-03T20:48:20.653 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-03T20:48:20.653 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-03T20:48:20.653 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "mds.c", 2024-08-03T20:48:20.653 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQAwl65mEUAFDRAAGdw1cymfvfKxGeJS/Od6Rw==", 2024-08-03T20:48:20.654 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2024-08-03T20:48:20.654 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": "allow", 2024-08-03T20:48:20.654 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow *", 2024-08-03T20:48:20.654 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow *", 2024-08-03T20:48:20.654 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2024-08-03T20:48:20.654 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-03T20:48:20.654 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-03T20:48:20.654 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-03T20:48:20.654 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "mds.d", 2024-08-03T20:48:20.654 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQAwl65mjUHZKBAAqOCnIopBK8Rhk36hrlNnzw==", 2024-08-03T20:48:20.654 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2024-08-03T20:48:20.654 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": "allow", 2024-08-03T20:48:20.655 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow *", 2024-08-03T20:48:20.655 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow *", 2024-08-03T20:48:20.655 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2024-08-03T20:48:20.655 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-03T20:48:20.655 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-03T20:48:20.655 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-03T20:48:20.656 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "osd.0", 2024-08-03T20:48:20.656 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQAyl65m4kIyLxAAdCh0D2MjJNPPu5b7jNkrbw==", 2024-08-03T20:48:20.656 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2024-08-03T20:48:20.656 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow profile osd", 2024-08-03T20:48:20.656 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow profile osd", 2024-08-03T20:48:20.656 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2024-08-03T20:48:20.656 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-03T20:48:20.656 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-03T20:48:20.656 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-03T20:48:20.656 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "osd.1", 2024-08-03T20:48:20.656 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQA1l65mLzVOOhAA081V1cUyYFPz17L0nny7gA==", 2024-08-03T20:48:20.656 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2024-08-03T20:48:20.656 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow profile osd", 2024-08-03T20:48:20.657 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow profile osd", 2024-08-03T20:48:20.657 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2024-08-03T20:48:20.657 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-03T20:48:20.657 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-03T20:48:20.657 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-03T20:48:20.657 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "osd.2", 2024-08-03T20:48:20.657 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQA5l65mBnXDBxAA9/lZVkwD2ykMTOIysoYODg==", 2024-08-03T20:48:20.657 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2024-08-03T20:48:20.657 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow profile osd", 2024-08-03T20:48:20.657 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow profile osd", 2024-08-03T20:48:20.658 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2024-08-03T20:48:20.658 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-03T20:48:20.658 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-03T20:48:20.658 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-03T20:48:20.658 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "osd.3", 2024-08-03T20:48:20.658 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQA8l65mAriiERAAfzV8An67+PnA3EzeAutIbA==", 2024-08-03T20:48:20.658 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2024-08-03T20:48:20.658 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow profile osd", 2024-08-03T20:48:20.658 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow profile osd", 2024-08-03T20:48:20.658 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2024-08-03T20:48:20.658 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-03T20:48:20.658 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-03T20:48:20.658 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-03T20:48:20.658 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "osd.4", 2024-08-03T20:48:20.659 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQBAl65m5uJSJRAA8S2IjJpxuqHY8YK4tTDMPg==", 2024-08-03T20:48:20.659 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2024-08-03T20:48:20.659 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow profile osd", 2024-08-03T20:48:20.659 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow profile osd", 2024-08-03T20:48:20.659 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2024-08-03T20:48:20.659 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-03T20:48:20.659 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-03T20:48:20.659 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-03T20:48:20.659 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "osd.5", 2024-08-03T20:48:20.659 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQBDl65mBP3JLBAADgK2X3zNyfnjjZADX1PFcA==", 2024-08-03T20:48:20.659 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2024-08-03T20:48:20.659 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow profile osd", 2024-08-03T20:48:20.659 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow profile osd", 2024-08-03T20:48:20.659 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2024-08-03T20:48:20.660 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-03T20:48:20.660 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-03T20:48:20.660 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-03T20:48:20.660 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "osd.6", 2024-08-03T20:48:20.660 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQBGl65m8DBNNRAANeOUfW70S5JfqFlpuMF9KQ==", 2024-08-03T20:48:20.660 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2024-08-03T20:48:20.660 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow profile osd", 2024-08-03T20:48:20.660 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow profile osd", 2024-08-03T20:48:20.660 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2024-08-03T20:48:20.660 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-03T20:48:20.661 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-03T20:48:20.661 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-03T20:48:20.661 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "osd.7", 2024-08-03T20:48:20.661 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQBKl65mUnOcABAA41wylzC1Ec9Fj01Ke0HhTw==", 2024-08-03T20:48:20.661 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2024-08-03T20:48:20.661 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow profile osd", 2024-08-03T20:48:20.661 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow profile osd", 2024-08-03T20:48:20.661 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2024-08-03T20:48:20.661 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-03T20:48:20.661 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-03T20:48:20.661 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-03T20:48:20.662 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "client.0", 2024-08-03T20:48:20.662 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQAwl65msgO3NRAAHG/WuMTK9c4lDXocXvqWnA==", 2024-08-03T20:48:20.662 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2024-08-03T20:48:20.662 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": "allow", 2024-08-03T20:48:20.662 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow r", 2024-08-03T20:48:20.662 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow rw", 2024-08-03T20:48:20.662 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow rwx" 2024-08-03T20:48:20.662 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-03T20:48:20.662 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-03T20:48:20.662 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-03T20:48:20.662 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "client.1", 2024-08-03T20:48:20.662 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQAxl65mbe19DxAA+sepc9ucXk2Bqy7C90mfbA==", 2024-08-03T20:48:20.662 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2024-08-03T20:48:20.663 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": "allow", 2024-08-03T20:48:20.663 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow r", 2024-08-03T20:48:20.663 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow rw", 2024-08-03T20:48:20.663 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow rwx" 2024-08-03T20:48:20.663 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-03T20:48:20.663 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-03T20:48:20.663 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-03T20:48:20.663 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "client.2", 2024-08-03T20:48:20.663 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQAxl65mBnbkBBAAjor60WWLTKXidiBMgtKuEA==", 2024-08-03T20:48:20.663 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2024-08-03T20:48:20.663 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": "allow", 2024-08-03T20:48:20.664 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow r", 2024-08-03T20:48:20.664 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow rw", 2024-08-03T20:48:20.664 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow rwx" 2024-08-03T20:48:20.664 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-03T20:48:20.664 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-03T20:48:20.664 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-03T20:48:20.664 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "client.3", 2024-08-03T20:48:20.664 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQAxl65munz8GRAA/cRl3ghwQE42FOizurcW3g==", 2024-08-03T20:48:20.664 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2024-08-03T20:48:20.664 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": "allow", 2024-08-03T20:48:20.664 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow r", 2024-08-03T20:48:20.664 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow rw", 2024-08-03T20:48:20.664 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow rwx" 2024-08-03T20:48:20.665 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-03T20:48:20.665 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-03T20:48:20.665 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-03T20:48:20.665 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "client.admin", 2024-08-03T20:48:20.665 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQAul65mB8VcOhAA105JQ2FX1vhtaGj3Fer8mg==", 2024-08-03T20:48:20.665 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2024-08-03T20:48:20.665 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": "allow *", 2024-08-03T20:48:20.665 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow *", 2024-08-03T20:48:20.665 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow *", 2024-08-03T20:48:20.665 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2024-08-03T20:48:20.665 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-03T20:48:20.665 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-03T20:48:20.666 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-03T20:48:20.666 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "client.bootstrap-mds", 2024-08-03T20:48:20.666 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQBSl65mRTEnORAAmdoldhj6KZWDgjklLhfw3g==", 2024-08-03T20:48:20.666 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2024-08-03T20:48:20.666 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow profile bootstrap-mds" 2024-08-03T20:48:20.666 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-03T20:48:20.666 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-03T20:48:20.666 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-03T20:48:20.666 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "client.bootstrap-mgr", 2024-08-03T20:48:20.667 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQBSl65m2EQnORAAJAnwFZYMiW5NcB+3PQedYQ==", 2024-08-03T20:48:20.667 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2024-08-03T20:48:20.667 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow profile bootstrap-mgr" 2024-08-03T20:48:20.667 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-03T20:48:20.667 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-03T20:48:20.667 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-03T20:48:20.667 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "client.bootstrap-osd", 2024-08-03T20:48:20.667 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQBSl65moVUnORAAuTnl8imkzb84mPfRPaOqdw==", 2024-08-03T20:48:20.667 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2024-08-03T20:48:20.667 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow profile bootstrap-osd" 2024-08-03T20:48:20.667 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-03T20:48:20.667 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-03T20:48:20.667 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-03T20:48:20.667 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "client.bootstrap-rbd", 2024-08-03T20:48:20.668 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQBSl65mB2cnORAAIP+FxrA8i/ypOl3Y5NhJGA==", 2024-08-03T20:48:20.668 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2024-08-03T20:48:20.668 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow profile bootstrap-rbd" 2024-08-03T20:48:20.668 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-03T20:48:20.668 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-03T20:48:20.668 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-03T20:48:20.668 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "client.bootstrap-rbd-mirror", 2024-08-03T20:48:20.668 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQBSl65mdXsnORAAmEbH2yFCEByWgX0qdWbtLQ==", 2024-08-03T20:48:20.668 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2024-08-03T20:48:20.668 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow profile bootstrap-rbd-mirror" 2024-08-03T20:48:20.668 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-03T20:48:20.668 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-03T20:48:20.669 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-03T20:48:20.669 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "client.bootstrap-rgw", 2024-08-03T20:48:20.669 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQBSl65mlYwnORAA78tRvz7gEWMDgXvUmM7M9w==", 2024-08-03T20:48:20.669 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2024-08-03T20:48:20.669 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow profile bootstrap-rgw" 2024-08-03T20:48:20.669 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-03T20:48:20.669 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-03T20:48:20.669 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-03T20:48:20.669 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "mgr.x", 2024-08-03T20:48:20.669 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQAvl65mUaPKLxAAnll74Gmk8nSuN/HuX6rTxA==", 2024-08-03T20:48:20.669 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2024-08-03T20:48:20.669 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": "allow *", 2024-08-03T20:48:20.669 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow profile mgr", 2024-08-03T20:48:20.669 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2024-08-03T20:48:20.670 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-03T20:48:20.670 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-03T20:48:20.670 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-03T20:48:20.670 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "mgr.y", 2024-08-03T20:48:20.670 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQAvl65m5Gq4JBAA306Ge1zHF3Z0WIzUrset4A==", 2024-08-03T20:48:20.670 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2024-08-03T20:48:20.670 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": "allow *", 2024-08-03T20:48:20.670 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow profile mgr", 2024-08-03T20:48:20.670 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2024-08-03T20:48:20.670 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-03T20:48:20.670 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-03T20:48:20.671 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-03T20:48:20.671 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-08-03T20:48:20.672 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:20.651+0000 7f328c400640 1 -- 172.21.15.32:0/1994890058 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f3264077010 msgr2=0x7f3264079520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:48:20.672 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:20.651+0000 7f328c400640 1 --2- 172.21.15.32:0/1994890058 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f3264077010 0x7f3264079520 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f3270000f60 tx=0x7f3270009040 comp rx=0 tx=0).stop 2024-08-03T20:48:20.672 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:20.651+0000 7f328c400640 1 -- 172.21.15.32:0/1994890058 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f328406e930 msgr2=0x7f328415ca00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:48:20.672 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:20.651+0000 7f328c400640 1 --2- 172.21.15.32:0/1994890058 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f328406e930 0x7f328415ca00 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f3278034040 tx=0x7f32780039e0 comp rx=0 tx=0).stop 2024-08-03T20:48:20.672 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:20.651+0000 7f328c400640 1 -- 172.21.15.32:0/1994890058 shutdown_connections 2024-08-03T20:48:20.672 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:20.651+0000 7f328c400640 1 --2- 172.21.15.32:0/1994890058 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f3264077010 0x7f3264079520 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:20.672 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:20.651+0000 7f328c400640 1 --2- 172.21.15.32:0/1994890058 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f328415fff0 0x7f328415e0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:20.672 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:20.651+0000 7f328c400640 1 --2- 172.21.15.32:0/1994890058 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f3284073e60 0x7f328413df50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:20.673 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:20.651+0000 7f328c400640 1 --2- 172.21.15.32:0/1994890058 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f328406e930 0x7f328415ca00 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:20.673 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:20.651+0000 7f328c400640 1 -- 172.21.15.32:0/1994890058 >> 172.21.15.32:0/1994890058 conn(0x7f32840b1bd0 msgr2=0x7f328406f930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:48:20.673 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:20.651+0000 7f328c400640 1 -- 172.21.15.32:0/1994890058 shutdown_connections 2024-08-03T20:48:20.673 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:20.651+0000 7f328c400640 1 -- 172.21.15.32:0/1994890058 wait complete. 2024-08-03T20:48:20.675 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-mds 2024-08-03T20:48:20.905 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:20.904+0000 7f8dd0e00640 1 -- 172.21.15.32:0/2008373903 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8dcc0b29f0 msgr2=0x7f8dcc0cdba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:48:20.905 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:20.904+0000 7f8dd0e00640 1 --2- 172.21.15.32:0/2008373903 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8dcc0b29f0 0x7f8dcc0cdba0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f8dcc0cd890 tx=0x7f8db402a460 comp rx=0 tx=0).stop 2024-08-03T20:48:20.906 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:20.905+0000 7f8dd0e00640 1 -- 172.21.15.32:0/2008373903 shutdown_connections 2024-08-03T20:48:20.906 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:20.905+0000 7f8dd0e00640 1 --2- 172.21.15.32:0/2008373903 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f8dcc0cfc90 0x7f8dcc0d00a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:20.906 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:20.905+0000 7f8dd0e00640 1 --2- 172.21.15.32:0/2008373903 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8dcc0b29f0 0x7f8dcc0cdba0 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:20.906 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:20.905+0000 7f8dd0e00640 1 -- 172.21.15.32:0/2008373903 >> 172.21.15.32:0/2008373903 conn(0x7f8dcc0b3bb0 msgr2=0x7f8dcc0b3fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:48:20.906 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:20.905+0000 7f8dd0e00640 1 -- 172.21.15.32:0/2008373903 shutdown_connections 2024-08-03T20:48:20.906 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:20.905+0000 7f8dd0e00640 1 -- 172.21.15.32:0/2008373903 wait complete. 2024-08-03T20:48:20.907 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:20.907+0000 7f8dd0e00640 1 Processor -- start 2024-08-03T20:48:20.907 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:20.907+0000 7f8dd0e00640 1 -- start start 2024-08-03T20:48:20.909 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:20.908+0000 7f8dd0e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8dcc0b29f0 0x7f8dcc0c4c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:48:20.909 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:20.908+0000 7f8dd0e00640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f8dcc0b7760 0x7f8dcc0b6310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:48:20.909 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:20.908+0000 7f8dd0e00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8dcc0cfc90 0x7f8dcc163e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:48:20.909 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:20.908+0000 7f8dd0e00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f8dcc0d0970 con 0x7f8dcc0b29f0 2024-08-03T20:48:20.909 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:20.908+0000 7f8dd0e00640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f8dcc0d07f0 con 0x7f8dcc0cfc90 2024-08-03T20:48:20.910 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:20.908+0000 7f8dd0e00640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7f8dcc0b17a0 con 0x7f8dcc0b7760 2024-08-03T20:48:20.910 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:20.909+0000 7f8dc9e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8dcc0b29f0 0x7f8dcc0c4c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:48:20.910 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:20.909+0000 7f8dc9e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8dcc0b29f0 0x7f8dcc0c4c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:48076/0 (socket says 172.21.15.32:48076) 2024-08-03T20:48:20.910 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:20.909+0000 7f8dc9e00640 1 -- 172.21.15.32:0/1139840260 learned_addr learned my addr 172.21.15.32:0/1139840260 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:48:20.910 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:20.909+0000 7f8dca800640 1 --2- 172.21.15.32:0/1139840260 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8dcc0cfc90 0x7f8dcc163e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:48:20.910 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:20.909+0000 7f8dc3e00640 1 --2- 172.21.15.32:0/1139840260 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f8dcc0b7760 0x7f8dcc0b6310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:48:20.911 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:20.909+0000 7f8dc9e00640 1 -- 172.21.15.32:0/1139840260 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f8dcc0b7760 msgr2=0x7f8dcc0b6310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:48:20.911 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:20.909+0000 7f8dc9e00640 1 --2- 172.21.15.32:0/1139840260 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f8dcc0b7760 0x7f8dcc0b6310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:20.911 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:20.909+0000 7f8dc9e00640 1 -- 172.21.15.32:0/1139840260 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8dcc0cfc90 msgr2=0x7f8dcc163e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:48:20.911 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:20.909+0000 7f8dc9e00640 1 --2- 172.21.15.32:0/1139840260 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8dcc0cfc90 0x7f8dcc163e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:20.911 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:20.909+0000 7f8dc9e00640 1 -- 172.21.15.32:0/1139840260 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8dcc06f340 con 0x7f8dcc0b29f0 2024-08-03T20:48:20.911 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:20.909+0000 7f8dc3e00640 1 --2- 172.21.15.32:0/1139840260 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f8dcc0b7760 0x7f8dcc0b6310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-03T20:48:20.911 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:20.910+0000 7f8dc9e00640 1 --2- 172.21.15.32:0/1139840260 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8dcc0b29f0 0x7f8dcc0c4c30 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f8dbc00b410 tx=0x7f8dbc00b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:48:20.911 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:20.910+0000 7f8dc2000640 1 -- 172.21.15.32:0/1139840260 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8dbc015020 con 0x7f8dcc0b29f0 2024-08-03T20:48:20.912 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:20.910+0000 7f8dc2000640 1 -- 172.21.15.32:0/1139840260 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f8dbc011830 con 0x7f8dcc0b29f0 2024-08-03T20:48:20.912 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:20.910+0000 7f8dc2000640 1 -- 172.21.15.32:0/1139840260 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8dbc011c80 con 0x7f8dcc0b29f0 2024-08-03T20:48:20.912 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:20.910+0000 7f8dd0e00640 1 -- 172.21.15.32:0/1139840260 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8dcc06f620 con 0x7f8dcc0b29f0 2024-08-03T20:48:20.912 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:20.910+0000 7f8dd0e00640 1 -- 172.21.15.32:0/1139840260 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8dcc1f5eb0 con 0x7f8dcc0b29f0 2024-08-03T20:48:20.912 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:20.911+0000 7f8dd0e00640 1 -- 172.21.15.32:0/1139840260 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8dcc0cdba0 con 0x7f8dcc0b29f0 2024-08-03T20:48:20.912 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:20.912+0000 7f8dc2000640 1 -- 172.21.15.32:0/1139840260 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f8dbc018020 con 0x7f8dcc0b29f0 2024-08-03T20:48:20.913 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:20.913+0000 7f8dc2000640 1 --2- 172.21.15.32:0/1139840260 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f8da8077010 0x7f8da8079520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:48:20.914 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:20.913+0000 7f8dc2000640 1 -- 172.21.15.32:0/1139840260 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(28..28 src has 1..28) ==== 4857+0+0 (secure 0 0 0) 0x7f8dbc0981d0 con 0x7f8dcc0b29f0 2024-08-03T20:48:20.914 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:20.914+0000 7f8dc3e00640 1 --2- 172.21.15.32:0/1139840260 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f8da8077010 0x7f8da8079520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:48:20.915 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:20.914+0000 7f8dc3e00640 1 --2- 172.21.15.32:0/1139840260 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f8da8077010 0x7f8da8079520 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f8db4033000 tx=0x7f8db4004730 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:48:20.918 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:20.917+0000 7f8dc2000640 1 -- 172.21.15.32:0/1139840260 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7f8dbc0617a0 con 0x7f8dcc0b29f0 2024-08-03T20:48:21.203 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:21.203+0000 7f8dd0e00640 1 -- 172.21.15.32:0/1139840260 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-mds"} v 0) -- 0x7f8dcc0b4760 con 0x7f8dcc0b29f0 2024-08-03T20:48:21.206 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:21.206+0000 7f8dc2000640 1 -- 172.21.15.32:0/1139840260 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-mds"}]=0 v5) ==== 90+0+0 (secure 0 0 0) 0x7f8dbc011e20 con 0x7f8dcc0b29f0 2024-08-03T20:48:21.209 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:21.208+0000 7f8dd0e00640 1 -- 172.21.15.32:0/1139840260 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f8da8077010 msgr2=0x7f8da8079520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:48:21.209 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:21.208+0000 7f8dd0e00640 1 --2- 172.21.15.32:0/1139840260 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f8da8077010 0x7f8da8079520 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f8db4033000 tx=0x7f8db4004730 comp rx=0 tx=0).stop 2024-08-03T20:48:21.209 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:21.208+0000 7f8dd0e00640 1 -- 172.21.15.32:0/1139840260 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8dcc0b29f0 msgr2=0x7f8dcc0c4c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:48:21.209 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:21.209+0000 7f8dd0e00640 1 --2- 172.21.15.32:0/1139840260 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8dcc0b29f0 0x7f8dcc0c4c30 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f8dbc00b410 tx=0x7f8dbc00b8d0 comp rx=0 tx=0).stop 2024-08-03T20:48:21.210 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:21.209+0000 7f8dd0e00640 1 -- 172.21.15.32:0/1139840260 shutdown_connections 2024-08-03T20:48:21.210 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:21.209+0000 7f8dd0e00640 1 --2- 172.21.15.32:0/1139840260 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f8da8077010 0x7f8da8079520 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:21.210 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:21.209+0000 7f8dd0e00640 1 --2- 172.21.15.32:0/1139840260 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8dcc0cfc90 0x7f8dcc163e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:21.210 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:21.209+0000 7f8dd0e00640 1 --2- 172.21.15.32:0/1139840260 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f8dcc0b7760 0x7f8dcc0b6310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:21.210 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:21.209+0000 7f8dd0e00640 1 --2- 172.21.15.32:0/1139840260 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8dcc0b29f0 0x7f8dcc0c4c30 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:21.210 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:21.209+0000 7f8dd0e00640 1 -- 172.21.15.32:0/1139840260 >> 172.21.15.32:0/1139840260 conn(0x7f8dcc0b3bb0 msgr2=0x7f8dcc0cead0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:48:21.210 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:21.209+0000 7f8dd0e00640 1 -- 172.21.15.32:0/1139840260 shutdown_connections 2024-08-03T20:48:21.210 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:21.209+0000 7f8dd0e00640 1 -- 172.21.15.32:0/1139840260 wait complete. 2024-08-03T20:48:21.230 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-mgr 2024-08-03T20:48:21.461 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:21.460+0000 7fe8ce000640 1 -- 172.21.15.32:0/3022591798 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe8c806e930 msgr2=0x7fe8c8144f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:48:21.461 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:21.460+0000 7fe8ce000640 1 --2- 172.21.15.32:0/3022591798 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe8c806e930 0x7fe8c8144f90 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fe8b8009830 tx=0x7fe8b802a510 comp rx=0 tx=0).stop 2024-08-03T20:48:21.462 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:21.460+0000 7fe8ce000640 1 -- 172.21.15.32:0/3022591798 shutdown_connections 2024-08-03T20:48:21.462 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:21.460+0000 7fe8ce000640 1 --2- 172.21.15.32:0/3022591798 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fe8c815fff0 0x7fe8c8160400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:21.462 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:21.460+0000 7fe8ce000640 1 --2- 172.21.15.32:0/3022591798 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe8c806e930 0x7fe8c8144f90 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:21.462 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:21.460+0000 7fe8ce000640 1 -- 172.21.15.32:0/3022591798 >> 172.21.15.32:0/3022591798 conn(0x7fe8c80b1bd0 msgr2=0x7fe8c80b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:48:21.462 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:21.460+0000 7fe8ce000640 1 -- 172.21.15.32:0/3022591798 shutdown_connections 2024-08-03T20:48:21.462 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:21.460+0000 7fe8ce000640 1 -- 172.21.15.32:0/3022591798 wait complete. 2024-08-03T20:48:21.463 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:21.463+0000 7fe8ce000640 1 Processor -- start 2024-08-03T20:48:21.463 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:21.463+0000 7fe8ce000640 1 -- start start 2024-08-03T20:48:21.465 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:21.464+0000 7fe8ce000640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fe8c806e930 0x7fe8c81f9410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:48:21.466 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:21.464+0000 7fe8ce000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe8c8073e60 0x7fe8c8140280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:48:21.466 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:21.464+0000 7fe8ce000640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fe8c815fff0 0x7fe8c8143c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:48:21.466 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:21.464+0000 7fe8ce000640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fe8c80cf840 con 0x7fe8c8073e60 2024-08-03T20:48:21.466 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:21.464+0000 7fe8ce000640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7fe8c8145f30 con 0x7fe8c815fff0 2024-08-03T20:48:21.466 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:21.464+0000 7fe8ce000640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7fe8c80cf9c0 con 0x7fe8c806e930 2024-08-03T20:48:21.466 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:21.465+0000 7fe8bfe00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe8c8073e60 0x7fe8c8140280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:48:21.466 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:21.465+0000 7fe8bfe00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe8c8073e60 0x7fe8c8140280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:48080/0 (socket says 172.21.15.32:48080) 2024-08-03T20:48:21.466 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:21.465+0000 7fe8bfe00640 1 -- 172.21.15.32:0/124070775 learned_addr learned my addr 172.21.15.32:0/124070775 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:48:21.466 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:21.465+0000 7fe8c7e00640 1 --2- 172.21.15.32:0/124070775 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fe8c815fff0 0x7fe8c8143c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:48:21.466 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:21.465+0000 7fe8c7400640 1 --2- 172.21.15.32:0/124070775 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fe8c806e930 0x7fe8c81f9410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:48:21.466 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:21.465+0000 7fe8bfe00640 1 -- 172.21.15.32:0/124070775 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fe8c806e930 msgr2=0x7fe8c81f9410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:48:21.466 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:21.465+0000 7fe8bfe00640 1 --2- 172.21.15.32:0/124070775 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fe8c806e930 0x7fe8c81f9410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:21.466 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:21.465+0000 7fe8bfe00640 1 -- 172.21.15.32:0/124070775 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fe8c815fff0 msgr2=0x7fe8c8143c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:48:21.467 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:21.465+0000 7fe8bfe00640 1 --2- 172.21.15.32:0/124070775 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fe8c815fff0 0x7fe8c8143c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:21.467 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:21.465+0000 7fe8bfe00640 1 -- 172.21.15.32:0/124070775 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe8c80b49b0 con 0x7fe8c8073e60 2024-08-03T20:48:21.467 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:21.465+0000 7fe8c7400640 1 --2- 172.21.15.32:0/124070775 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fe8c806e930 0x7fe8c81f9410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-03T20:48:21.468 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:21.465+0000 7fe8bfe00640 1 --2- 172.21.15.32:0/124070775 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe8c8073e60 0x7fe8c8140280 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fe8b000c340 tx=0x7fe8b000c800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:48:21.468 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:21.466+0000 7fe8bf400640 1 -- 172.21.15.32:0/124070775 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe8b0011790 con 0x7fe8c8073e60 2024-08-03T20:48:21.468 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:21.466+0000 7fe8bf400640 1 -- 172.21.15.32:0/124070775 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fe8b0017070 con 0x7fe8c8073e60 2024-08-03T20:48:21.468 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:21.466+0000 7fe8bf400640 1 -- 172.21.15.32:0/124070775 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe8b0011cc0 con 0x7fe8c8073e60 2024-08-03T20:48:21.468 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:21.466+0000 7fe8ce000640 1 -- 172.21.15.32:0/124070775 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe8c80b4e20 con 0x7fe8c8073e60 2024-08-03T20:48:21.468 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:21.466+0000 7fe8ce000640 1 -- 172.21.15.32:0/124070775 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe8c80c5370 con 0x7fe8c8073e60 2024-08-03T20:48:21.468 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:21.467+0000 7fe8ce000640 1 -- 172.21.15.32:0/124070775 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe8c8160c50 con 0x7fe8c8073e60 2024-08-03T20:48:21.468 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:21.468+0000 7fe8bf400640 1 -- 172.21.15.32:0/124070775 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7fe8b001c460 con 0x7fe8c8073e60 2024-08-03T20:48:21.470 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:21.469+0000 7fe8bf400640 1 --2- 172.21.15.32:0/124070775 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fe8a4077010 0x7fe8a4079520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:48:21.470 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:21.469+0000 7fe8bf400640 1 -- 172.21.15.32:0/124070775 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(28..28 src has 1..28) ==== 4857+0+0 (secure 0 0 0) 0x7fe8b0098310 con 0x7fe8c8073e60 2024-08-03T20:48:21.470 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:21.470+0000 7fe8c7400640 1 --2- 172.21.15.32:0/124070775 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fe8a4077010 0x7fe8a4079520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:48:21.470 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:21.470+0000 7fe8c7400640 1 --2- 172.21.15.32:0/124070775 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fe8a4077010 0x7fe8a4079520 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fe8b8034000 tx=0x7fe8b8004730 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:48:21.474 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:21.474+0000 7fe8bf400640 1 -- 172.21.15.32:0/124070775 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7fe8b0061850 con 0x7fe8c8073e60 2024-08-03T20:48:21.760 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:21.759+0000 7fe8ce000640 1 -- 172.21.15.32:0/124070775 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-mgr"} v 0) -- 0x7fe8c80b4b80 con 0x7fe8c8073e60 2024-08-03T20:48:21.762 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:21.762+0000 7fe8bf400640 1 -- 172.21.15.32:0/124070775 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-mgr"}]=0 v6) ==== 90+0+0 (secure 0 0 0) 0x7fe8b0061150 con 0x7fe8c8073e60 2024-08-03T20:48:21.766 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:21.765+0000 7fe8ce000640 1 -- 172.21.15.32:0/124070775 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fe8a4077010 msgr2=0x7fe8a4079520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:48:21.766 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:21.765+0000 7fe8ce000640 1 --2- 172.21.15.32:0/124070775 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fe8a4077010 0x7fe8a4079520 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fe8b8034000 tx=0x7fe8b8004730 comp rx=0 tx=0).stop 2024-08-03T20:48:21.766 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:21.765+0000 7fe8ce000640 1 -- 172.21.15.32:0/124070775 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe8c8073e60 msgr2=0x7fe8c8140280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:48:21.766 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:21.765+0000 7fe8ce000640 1 --2- 172.21.15.32:0/124070775 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe8c8073e60 0x7fe8c8140280 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fe8b000c340 tx=0x7fe8b000c800 comp rx=0 tx=0).stop 2024-08-03T20:48:21.766 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:21.765+0000 7fe8ce000640 1 -- 172.21.15.32:0/124070775 shutdown_connections 2024-08-03T20:48:21.766 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:21.765+0000 7fe8ce000640 1 --2- 172.21.15.32:0/124070775 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fe8a4077010 0x7fe8a4079520 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:21.766 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:21.765+0000 7fe8ce000640 1 --2- 172.21.15.32:0/124070775 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fe8c815fff0 0x7fe8c8143c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:21.767 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:21.765+0000 7fe8ce000640 1 --2- 172.21.15.32:0/124070775 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe8c8073e60 0x7fe8c8140280 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:21.767 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:21.765+0000 7fe8ce000640 1 --2- 172.21.15.32:0/124070775 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fe8c806e930 0x7fe8c81f9410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:21.767 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:21.765+0000 7fe8ce000640 1 -- 172.21.15.32:0/124070775 >> 172.21.15.32:0/124070775 conn(0x7fe8c80b1bd0 msgr2=0x7fe8c80b04e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:48:21.767 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:21.766+0000 7fe8ce000640 1 -- 172.21.15.32:0/124070775 shutdown_connections 2024-08-03T20:48:21.767 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:21.766+0000 7fe8ce000640 1 -- 172.21.15.32:0/124070775 wait complete. 2024-08-03T20:48:21.792 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-osd 2024-08-03T20:48:22.021 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:22.020+0000 7f1857200640 1 -- 172.21.15.32:0/1839847391 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f18580cf510 msgr2=0x7f18580cf920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:48:22.022 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:22.020+0000 7f1857200640 1 --2- 172.21.15.32:0/1839847391 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f18580cf510 0x7f18580cf920 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f1850005fd0 tx=0x7f185002a8b0 comp rx=0 tx=0).stop 2024-08-03T20:48:22.022 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:22.021+0000 7f1857200640 1 -- 172.21.15.32:0/1839847391 shutdown_connections 2024-08-03T20:48:22.022 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:22.021+0000 7f1857200640 1 --2- 172.21.15.32:0/1839847391 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f18580cf510 0x7f18580cf920 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:22.022 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:22.021+0000 7f1857200640 1 -- 172.21.15.32:0/1839847391 >> 172.21.15.32:0/1839847391 conn(0x7f185808d7a0 msgr2=0x7f185809d3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:48:22.022 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:22.021+0000 7f1857200640 1 -- 172.21.15.32:0/1839847391 shutdown_connections 2024-08-03T20:48:22.022 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:22.021+0000 7f1857200640 1 -- 172.21.15.32:0/1839847391 wait complete. 2024-08-03T20:48:22.023 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:22.023+0000 7f1857200640 1 Processor -- start 2024-08-03T20:48:22.023 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:22.023+0000 7f1857200640 1 -- start start 2024-08-03T20:48:22.025 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:22.025+0000 7f1857200640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f18580cf510 0x7f18580b83e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:48:22.026 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:22.025+0000 7f1857200640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f18580cffe0 0x7f18581fd4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:48:22.026 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:22.025+0000 7f1857200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f18580d26e0 0x7f18581fda20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:48:22.026 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:22.025+0000 7f1857200640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f18580d4310 con 0x7f18580d26e0 2024-08-03T20:48:22.026 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:22.025+0000 7f1857200640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f18580d4190 con 0x7f18580cffe0 2024-08-03T20:48:22.026 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:22.025+0000 7f1857200640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7f18580d4490 con 0x7f18580cf510 2024-08-03T20:48:22.026 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:22.025+0000 7f1856800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f18580d26e0 0x7f18581fda20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:48:22.026 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:22.025+0000 7f1856800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f18580d26e0 0x7f18581fda20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:48102/0 (socket says 172.21.15.32:48102) 2024-08-03T20:48:22.026 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:22.025+0000 7f1856800640 1 -- 172.21.15.32:0/2432270784 learned_addr learned my addr 172.21.15.32:0/2432270784 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:48:22.026 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:22.025+0000 7f1855e00640 1 --2- 172.21.15.32:0/2432270784 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f18580cf510 0x7f18580b83e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:48:22.026 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:22.025+0000 7f1855400640 1 --2- 172.21.15.32:0/2432270784 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f18580cffe0 0x7f18581fd4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:48:22.026 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:22.025+0000 7f1856800640 1 -- 172.21.15.32:0/2432270784 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f18580cf510 msgr2=0x7f18580b83e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:48:22.026 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:22.025+0000 7f1856800640 1 --2- 172.21.15.32:0/2432270784 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f18580cf510 0x7f18580b83e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:22.027 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:22.025+0000 7f1856800640 1 -- 172.21.15.32:0/2432270784 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f18580cffe0 msgr2=0x7f18581fd4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:48:22.027 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:22.025+0000 7f1856800640 1 --2- 172.21.15.32:0/2432270784 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f18580cffe0 0x7f18581fd4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:22.027 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:22.025+0000 7f1856800640 1 -- 172.21.15.32:0/2432270784 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f185815aa40 con 0x7f18580d26e0 2024-08-03T20:48:22.027 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:22.026+0000 7f1856800640 1 --2- 172.21.15.32:0/2432270784 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f18580d26e0 0x7f18581fda20 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f1850006010 tx=0x7f1850003ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:48:22.027 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:22.026+0000 7f184e000640 1 -- 172.21.15.32:0/2432270784 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1850031b90 con 0x7f18580d26e0 2024-08-03T20:48:22.027 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:22.026+0000 7f184e000640 1 -- 172.21.15.32:0/2432270784 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f1850031e20 con 0x7f18580d26e0 2024-08-03T20:48:22.028 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:22.026+0000 7f184e000640 1 -- 172.21.15.32:0/2432270784 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18500332f0 con 0x7f18580d26e0 2024-08-03T20:48:22.028 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:22.026+0000 7f1857200640 1 -- 172.21.15.32:0/2432270784 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f185815a1a0 con 0x7f18580d26e0 2024-08-03T20:48:22.028 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:22.026+0000 7f1857200640 1 -- 172.21.15.32:0/2432270784 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f185815a760 con 0x7f18580d26e0 2024-08-03T20:48:22.028 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:22.027+0000 7f1857200640 1 -- 172.21.15.32:0/2432270784 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f18580d4940 con 0x7f18580d26e0 2024-08-03T20:48:22.029 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:22.028+0000 7f184e000640 1 -- 172.21.15.32:0/2432270784 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f1850031660 con 0x7f18580d26e0 2024-08-03T20:48:22.030 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:22.029+0000 7f184e000640 1 --2- 172.21.15.32:0/2432270784 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f1838077010 0x7f1838079520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:48:22.030 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:22.030+0000 7f184e000640 1 -- 172.21.15.32:0/2432270784 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(28..28 src has 1..28) ==== 4857+0+0 (secure 0 0 0) 0x7f18500b9c30 con 0x7f18580d26e0 2024-08-03T20:48:22.030 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:22.030+0000 7f1855e00640 1 --2- 172.21.15.32:0/2432270784 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f1838077010 0x7f1838079520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:48:22.031 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:22.031+0000 7f1855e00640 1 --2- 172.21.15.32:0/2432270784 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f1838077010 0x7f1838079520 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f18480015e0 tx=0x7f1848009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:48:22.034 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:22.034+0000 7f184e000640 1 -- 172.21.15.32:0/2432270784 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7f185003f030 con 0x7f18580d26e0 2024-08-03T20:48:22.326 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:22.326+0000 7f1857200640 1 -- 172.21.15.32:0/2432270784 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-osd"} v 0) -- 0x7f185815ca50 con 0x7f18580d26e0 2024-08-03T20:48:22.329 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:22.329+0000 7f184e000640 1 -- 172.21.15.32:0/2432270784 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-osd"}]=0 v7) ==== 90+0+0 (secure 0 0 0) 0x7f1850083200 con 0x7f18580d26e0 2024-08-03T20:48:22.332 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:22.331+0000 7f1857200640 1 -- 172.21.15.32:0/2432270784 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f1838077010 msgr2=0x7f1838079520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:48:22.332 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:22.331+0000 7f1857200640 1 --2- 172.21.15.32:0/2432270784 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f1838077010 0x7f1838079520 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f18480015e0 tx=0x7f1848009040 comp rx=0 tx=0).stop 2024-08-03T20:48:22.332 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:22.331+0000 7f1857200640 1 -- 172.21.15.32:0/2432270784 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f18580d26e0 msgr2=0x7f18581fda20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:48:22.332 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:22.331+0000 7f1857200640 1 --2- 172.21.15.32:0/2432270784 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f18580d26e0 0x7f18581fda20 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f1850006010 tx=0x7f1850003ba0 comp rx=0 tx=0).stop 2024-08-03T20:48:22.332 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:22.331+0000 7f1857200640 1 -- 172.21.15.32:0/2432270784 shutdown_connections 2024-08-03T20:48:22.332 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:22.331+0000 7f1857200640 1 --2- 172.21.15.32:0/2432270784 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f1838077010 0x7f1838079520 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:22.332 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:22.331+0000 7f1857200640 1 --2- 172.21.15.32:0/2432270784 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f18580cffe0 0x7f18581fd4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:22.332 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:22.331+0000 7f1857200640 1 --2- 172.21.15.32:0/2432270784 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f18580d26e0 0x7f18581fda20 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:22.332 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:22.331+0000 7f1857200640 1 --2- 172.21.15.32:0/2432270784 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f18580cf510 0x7f18580b83e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:22.332 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:22.331+0000 7f1857200640 1 -- 172.21.15.32:0/2432270784 >> 172.21.15.32:0/2432270784 conn(0x7f185808d7a0 msgr2=0x7f1858160b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:48:22.332 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:22.331+0000 7f1857200640 1 -- 172.21.15.32:0/2432270784 shutdown_connections 2024-08-03T20:48:22.333 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:22.331+0000 7f1857200640 1 -- 172.21.15.32:0/2432270784 wait complete. 2024-08-03T20:48:22.356 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-rbd 2024-08-03T20:48:22.586 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:22.584+0000 7f0ae1400640 1 -- 172.21.15.32:0/1366014574 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f0adc0af020 msgr2=0x7f0adc0742b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:48:22.586 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:22.584+0000 7f0ae1400640 1 --2- 172.21.15.32:0/1366014574 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f0adc0af020 0x7f0adc0742b0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f0acc009860 tx=0x7f0acc02a4a0 comp rx=0 tx=0).stop 2024-08-03T20:48:22.586 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:22.584+0000 7f0ae1400640 1 -- 172.21.15.32:0/1366014574 shutdown_connections 2024-08-03T20:48:22.586 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:22.584+0000 7f0ae1400640 1 --2- 172.21.15.32:0/1366014574 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f0adc0af020 0x7f0adc0742b0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:22.586 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:22.584+0000 7f0ae1400640 1 -- 172.21.15.32:0/1366014574 >> 172.21.15.32:0/1366014574 conn(0x7f0adc0b2120 msgr2=0x7f0adc0cfd60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:48:22.586 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:22.585+0000 7f0ae1400640 1 -- 172.21.15.32:0/1366014574 shutdown_connections 2024-08-03T20:48:22.586 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:22.585+0000 7f0ae1400640 1 -- 172.21.15.32:0/1366014574 wait complete. 2024-08-03T20:48:22.587 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:22.587+0000 7f0ae1400640 1 Processor -- start 2024-08-03T20:48:22.587 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:22.587+0000 7f0ae1400640 1 -- start start 2024-08-03T20:48:22.589 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:22.588+0000 7f0ae1400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0adc06fb50 0x7f0adc0cdb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:48:22.590 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:22.588+0000 7f0ae1400640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f0adc0acd80 0x7f0adc153c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:48:22.590 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:22.588+0000 7f0ae1400640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f0adc0af020 0x7f0adc154190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:48:22.590 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:22.588+0000 7f0ae1400640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f0adc0ad310 con 0x7f0adc06fb50 2024-08-03T20:48:22.590 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:22.589+0000 7f0ae1400640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f0adc0ad190 con 0x7f0adc0acd80 2024-08-03T20:48:22.590 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:22.589+0000 7f0ae1400640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7f0adc0ad490 con 0x7f0adc0af020 2024-08-03T20:48:22.590 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:22.589+0000 7f0adbe00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0adc06fb50 0x7f0adc0cdb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:48:22.590 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:22.589+0000 7f0adbe00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0adc06fb50 0x7f0adc0cdb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:48112/0 (socket says 172.21.15.32:48112) 2024-08-03T20:48:22.590 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:22.589+0000 7f0adbe00640 1 -- 172.21.15.32:0/3767390349 learned_addr learned my addr 172.21.15.32:0/3767390349 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:48:22.590 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:22.589+0000 7f0adbe00640 1 -- 172.21.15.32:0/3767390349 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f0adc0af020 msgr2=0x7f0adc154190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:48:22.590 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:22.589+0000 7f0adbe00640 1 --2- 172.21.15.32:0/3767390349 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f0adc0af020 0x7f0adc154190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:22.590 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:22.589+0000 7f0adbe00640 1 -- 172.21.15.32:0/3767390349 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f0adc0acd80 msgr2=0x7f0adc153c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:48:22.591 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:22.589+0000 7f0adbe00640 1 --2- 172.21.15.32:0/3767390349 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f0adc0acd80 0x7f0adc153c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:22.591 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:22.589+0000 7f0adbe00640 1 -- 172.21.15.32:0/3767390349 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0adc153a40 con 0x7f0adc06fb50 2024-08-03T20:48:22.591 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:22.589+0000 7f0adbe00640 1 --2- 172.21.15.32:0/3767390349 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0adc06fb50 0x7f0adc0cdb40 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f0acc02aa90 tx=0x7f0acc003c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:48:22.591 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:22.590+0000 7f0ad8c00640 1 -- 172.21.15.32:0/3767390349 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0acc005c90 con 0x7f0adc06fb50 2024-08-03T20:48:22.592 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:22.590+0000 7f0ad8c00640 1 -- 172.21.15.32:0/3767390349 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f0acc005020 con 0x7f0adc06fb50 2024-08-03T20:48:22.592 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:22.590+0000 7f0ad8c00640 1 -- 172.21.15.32:0/3767390349 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0acc005410 con 0x7f0adc06fb50 2024-08-03T20:48:22.592 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:22.590+0000 7f0ae1400640 1 -- 172.21.15.32:0/3767390349 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0adc0ce4c0 con 0x7f0adc06fb50 2024-08-03T20:48:22.592 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:22.590+0000 7f0ae1400640 1 -- 172.21.15.32:0/3767390349 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0adc0cea80 con 0x7f0adc06fb50 2024-08-03T20:48:22.592 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:22.591+0000 7f0ae1400640 1 -- 172.21.15.32:0/3767390349 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0adc0742b0 con 0x7f0adc06fb50 2024-08-03T20:48:22.592 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:22.592+0000 7f0ad8c00640 1 -- 172.21.15.32:0/3767390349 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f0acc0317c0 con 0x7f0adc06fb50 2024-08-03T20:48:22.594 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:22.594+0000 7f0ad8c00640 1 --2- 172.21.15.32:0/3767390349 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f0ab8077010 0x7f0ab8079520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:48:22.594 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:22.594+0000 7f0ad8c00640 1 -- 172.21.15.32:0/3767390349 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(28..28 src has 1..28) ==== 4857+0+0 (secure 0 0 0) 0x7f0acc0b9540 con 0x7f0adc06fb50 2024-08-03T20:48:22.594 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:22.594+0000 7f0ad3400640 1 --2- 172.21.15.32:0/3767390349 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f0ab8077010 0x7f0ab8079520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:48:22.595 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:22.595+0000 7f0ad3400640 1 --2- 172.21.15.32:0/3767390349 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f0ab8077010 0x7f0ab8079520 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f0adc15ff30 tx=0x7f0ac4007970 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:48:22.598 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:22.598+0000 7f0ad8c00640 1 -- 172.21.15.32:0/3767390349 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7f0acc03f030 con 0x7f0adc06fb50 2024-08-03T20:48:22.885 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:22.884+0000 7f0ae1400640 1 -- 172.21.15.32:0/3767390349 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-rbd"} v 0) -- 0x7f0adc09d8d0 con 0x7f0adc06fb50 2024-08-03T20:48:22.887 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:22.887+0000 7f0ad8c00640 1 -- 172.21.15.32:0/3767390349 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-rbd"}]=0 v8) ==== 90+0+0 (secure 0 0 0) 0x7f0acc082b10 con 0x7f0adc06fb50 2024-08-03T20:48:22.890 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:22.889+0000 7f0ae1400640 1 -- 172.21.15.32:0/3767390349 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f0ab8077010 msgr2=0x7f0ab8079520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:48:22.890 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:22.889+0000 7f0ae1400640 1 --2- 172.21.15.32:0/3767390349 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f0ab8077010 0x7f0ab8079520 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f0adc15ff30 tx=0x7f0ac4007970 comp rx=0 tx=0).stop 2024-08-03T20:48:22.890 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:22.889+0000 7f0ae1400640 1 -- 172.21.15.32:0/3767390349 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0adc06fb50 msgr2=0x7f0adc0cdb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:48:22.890 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:22.889+0000 7f0ae1400640 1 --2- 172.21.15.32:0/3767390349 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0adc06fb50 0x7f0adc0cdb40 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f0acc02aa90 tx=0x7f0acc003c10 comp rx=0 tx=0).stop 2024-08-03T20:48:22.890 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:22.889+0000 7f0ae1400640 1 -- 172.21.15.32:0/3767390349 shutdown_connections 2024-08-03T20:48:22.891 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:22.889+0000 7f0ae1400640 1 --2- 172.21.15.32:0/3767390349 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f0ab8077010 0x7f0ab8079520 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:22.891 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:22.889+0000 7f0ae1400640 1 --2- 172.21.15.32:0/3767390349 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f0adc0acd80 0x7f0adc153c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:22.891 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:22.889+0000 7f0ae1400640 1 --2- 172.21.15.32:0/3767390349 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f0adc0af020 0x7f0adc154190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:22.891 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:22.889+0000 7f0ae1400640 1 --2- 172.21.15.32:0/3767390349 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0adc06fb50 0x7f0adc0cdb40 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:22.891 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:22.889+0000 7f0ae1400640 1 -- 172.21.15.32:0/3767390349 >> 172.21.15.32:0/3767390349 conn(0x7f0adc0b2120 msgr2=0x7f0adc15cb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:48:22.891 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:22.890+0000 7f0ae1400640 1 -- 172.21.15.32:0/3767390349 shutdown_connections 2024-08-03T20:48:22.891 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:22.890+0000 7f0ae1400640 1 -- 172.21.15.32:0/3767390349 wait complete. 2024-08-03T20:48:22.915 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-rbd-mirror 2024-08-03T20:48:23.148 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:23.145+0000 7f361d000640 1 -- 172.21.15.32:0/1157631551 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f35fc005350 msgr2=0x7f35fc00b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:48:23.148 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:23.145+0000 7f361d000640 1 --2- 172.21.15.32:0/1157631551 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f35fc005350 0x7f35fc00b9c0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f3610009ed0 tx=0x7f35fc005830 comp rx=0 tx=0).stop 2024-08-03T20:48:23.148 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:23.146+0000 7f361d000640 1 -- 172.21.15.32:0/1157631551 shutdown_connections 2024-08-03T20:48:23.148 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:23.146+0000 7f361d000640 1 --2- 172.21.15.32:0/1157631551 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f35fc005350 0x7f35fc00b9c0 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:23.148 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:23.146+0000 7f361d000640 1 -- 172.21.15.32:0/1157631551 >> 172.21.15.32:0/1157631551 conn(0x7f36180b3bb0 msgr2=0x7f36180b3fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:48:23.148 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:23.146+0000 7f361d000640 1 -- 172.21.15.32:0/1157631551 shutdown_connections 2024-08-03T20:48:23.148 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:23.146+0000 7f361d000640 1 -- 172.21.15.32:0/1157631551 wait complete. 2024-08-03T20:48:23.149 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:23.148+0000 7f361d000640 1 Processor -- start 2024-08-03T20:48:23.149 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:23.148+0000 7f361d000640 1 -- start start 2024-08-03T20:48:23.151 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:23.150+0000 7f361d000640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f35fc005350 0x7f36181f7090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:48:23.151 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:23.150+0000 7f361d000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f36181e2d20 0x7f36181e24b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:48:23.151 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:23.150+0000 7f361d000640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f36180d0490 0x7f36180d08a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:48:23.151 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:23.150+0000 7f361d000640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f361809afc0 con 0x7f36181e2d20 2024-08-03T20:48:23.151 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:23.150+0000 7f361d000640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f361809ae40 con 0x7f36180d0490 2024-08-03T20:48:23.152 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:23.150+0000 7f361d000640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7f361809b140 con 0x7f35fc005350 2024-08-03T20:48:23.152 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:23.150+0000 7f360fe00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f36181e2d20 0x7f36181e24b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:48:23.152 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:23.150+0000 7f360fe00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f36181e2d20 0x7f36181e24b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:48128/0 (socket says 172.21.15.32:48128) 2024-08-03T20:48:23.152 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:23.150+0000 7f360fe00640 1 -- 172.21.15.32:0/1165781066 learned_addr learned my addr 172.21.15.32:0/1165781066 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:48:23.153 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:23.150+0000 7f360fe00640 1 -- 172.21.15.32:0/1165781066 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f35fc005350 msgr2=0x7f36181f7090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:48:23.153 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:23.150+0000 7f3615e00640 1 --2- 172.21.15.32:0/1165781066 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f35fc005350 0x7f36181f7090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:48:23.153 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:23.150+0000 7f3616800640 1 --2- 172.21.15.32:0/1165781066 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f36180d0490 0x7f36180d08a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:48:23.153 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:23.150+0000 7f360fe00640 1 --2- 172.21.15.32:0/1165781066 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f35fc005350 0x7f36181f7090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:23.154 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:23.150+0000 7f360fe00640 1 -- 172.21.15.32:0/1165781066 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f36180d0490 msgr2=0x7f36180d08a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:48:23.154 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:23.150+0000 7f360fe00640 1 --2- 172.21.15.32:0/1165781066 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f36180d0490 0x7f36180d08a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:23.154 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:23.150+0000 7f360fe00640 1 -- 172.21.15.32:0/1165781066 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f36180d0de0 con 0x7f36181e2d20 2024-08-03T20:48:23.154 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:23.150+0000 7f3615e00640 1 --2- 172.21.15.32:0/1165781066 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f35fc005350 0x7f36181f7090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-03T20:48:23.154 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:23.150+0000 7f3616800640 1 --2- 172.21.15.32:0/1165781066 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f36180d0490 0x7f36180d08a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-03T20:48:23.154 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:23.151+0000 7f360fe00640 1 --2- 172.21.15.32:0/1165781066 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f36181e2d20 0x7f36181e24b0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f3600004de0 tx=0x7f3600001630 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:48:23.154 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:23.151+0000 7f360e000640 1 -- 172.21.15.32:0/1165781066 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36000107d0 con 0x7f36181e2d20 2024-08-03T20:48:23.154 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:23.151+0000 7f360e000640 1 -- 172.21.15.32:0/1165781066 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f3600016070 con 0x7f36181e2d20 2024-08-03T20:48:23.154 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:23.151+0000 7f360e000640 1 -- 172.21.15.32:0/1165781066 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3600010ce0 con 0x7f36181e2d20 2024-08-03T20:48:23.154 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:23.151+0000 7f361d000640 1 -- 172.21.15.32:0/1165781066 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f36181e29f0 con 0x7f36181e2d20 2024-08-03T20:48:23.154 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:23.151+0000 7f361d000640 1 -- 172.21.15.32:0/1165781066 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f36181e72f0 con 0x7f36181e2d20 2024-08-03T20:48:23.155 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:23.152+0000 7f361d000640 1 -- 172.21.15.32:0/1165781066 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f36180cfce0 con 0x7f36181e2d20 2024-08-03T20:48:23.155 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:23.153+0000 7f360e000640 1 -- 172.21.15.32:0/1165781066 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f360001b460 con 0x7f36181e2d20 2024-08-03T20:48:23.156 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:23.154+0000 7f360e000640 1 --2- 172.21.15.32:0/1165781066 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f35f4077010 0x7f35f4079520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:48:23.156 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:23.154+0000 7f360e000640 1 -- 172.21.15.32:0/1165781066 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(28..28 src has 1..28) ==== 4857+0+0 (secure 0 0 0) 0x7f3600098e80 con 0x7f36181e2d20 2024-08-03T20:48:23.156 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:23.155+0000 7f3615e00640 1 --2- 172.21.15.32:0/1165781066 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f35f4077010 0x7f35f4079520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:48:23.157 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:23.156+0000 7f3615e00640 1 --2- 172.21.15.32:0/1165781066 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f35f4077010 0x7f35f4079520 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f3608001100 tx=0x7f36080067c0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:48:23.160 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:23.159+0000 7f360e000640 1 -- 172.21.15.32:0/1165781066 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7f36180d0de0 con 0x7f36181e2d20 2024-08-03T20:48:23.446 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:23.444+0000 7f361d000640 1 -- 172.21.15.32:0/1165781066 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-rbd-mirror"} v 0) -- 0x7f36181e6da0 con 0x7f36181e2d20 2024-08-03T20:48:23.449 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:23.447+0000 7f360e000640 1 -- 172.21.15.32:0/1165781066 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-rbd-mirror"}]=0 v9) ==== 97+0+0 (secure 0 0 0) 0x7f36000623c0 con 0x7f36181e2d20 2024-08-03T20:48:23.451 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:23.450+0000 7f361d000640 1 -- 172.21.15.32:0/1165781066 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f35f4077010 msgr2=0x7f35f4079520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:48:23.451 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:23.450+0000 7f361d000640 1 --2- 172.21.15.32:0/1165781066 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f35f4077010 0x7f35f4079520 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f3608001100 tx=0x7f36080067c0 comp rx=0 tx=0).stop 2024-08-03T20:48:23.452 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:23.450+0000 7f361d000640 1 -- 172.21.15.32:0/1165781066 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f36181e2d20 msgr2=0x7f36181e24b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:48:23.452 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:23.450+0000 7f361d000640 1 --2- 172.21.15.32:0/1165781066 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f36181e2d20 0x7f36181e24b0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f3600004de0 tx=0x7f3600001630 comp rx=0 tx=0).stop 2024-08-03T20:48:23.452 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:23.450+0000 7f361d000640 1 -- 172.21.15.32:0/1165781066 shutdown_connections 2024-08-03T20:48:23.452 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:23.450+0000 7f361d000640 1 --2- 172.21.15.32:0/1165781066 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f35f4077010 0x7f35f4079520 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:23.452 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:23.450+0000 7f361d000640 1 --2- 172.21.15.32:0/1165781066 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f36180d0490 0x7f36180d08a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:23.452 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:23.450+0000 7f361d000640 1 --2- 172.21.15.32:0/1165781066 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f36181e2d20 0x7f36181e24b0 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:23.452 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:23.450+0000 7f361d000640 1 --2- 172.21.15.32:0/1165781066 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f35fc005350 0x7f36181f7090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:23.452 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:23.450+0000 7f361d000640 1 -- 172.21.15.32:0/1165781066 >> 172.21.15.32:0/1165781066 conn(0x7f36180b3bb0 msgr2=0x7f36181e44f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:48:23.452 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:23.450+0000 7f361d000640 1 -- 172.21.15.32:0/1165781066 shutdown_connections 2024-08-03T20:48:23.452 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:23.450+0000 7f361d000640 1 -- 172.21.15.32:0/1165781066 wait complete. 2024-08-03T20:48:23.473 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-rgw 2024-08-03T20:48:23.702 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:23.700+0000 7f9258c00640 1 -- 172.21.15.32:0/3757913221 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f92540b4ce0 msgr2=0x7f92540cdf70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:48:23.703 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:23.700+0000 7f9258c00640 1 --2- 172.21.15.32:0/3757913221 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f92540b4ce0 0x7f92540cdf70 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f925415c920 tx=0x7f924c02aa30 comp rx=0 tx=0).stop 2024-08-03T20:48:23.703 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:23.701+0000 7f9258c00640 1 -- 172.21.15.32:0/3757913221 shutdown_connections 2024-08-03T20:48:23.703 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:23.701+0000 7f9258c00640 1 --2- 172.21.15.32:0/3757913221 >> v2:172.21.15.118:3300/0 conn(0x7f925415f610 0x7f92541523f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:23.703 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:23.701+0000 7f9258c00640 1 --2- 172.21.15.32:0/3757913221 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f92540b4ce0 0x7f92540cdf70 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:23.703 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:23.701+0000 7f9258c00640 1 -- 172.21.15.32:0/3757913221 >> 172.21.15.32:0/3757913221 conn(0x7f92540b1bd0 msgr2=0x7f92540b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:48:23.703 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:23.701+0000 7f9258c00640 1 -- 172.21.15.32:0/3757913221 shutdown_connections 2024-08-03T20:48:23.703 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:23.701+0000 7f9258c00640 1 -- 172.21.15.32:0/3757913221 wait complete. 2024-08-03T20:48:23.704 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:23.703+0000 7f9258c00640 1 Processor -- start 2024-08-03T20:48:23.705 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:23.703+0000 7f9258c00640 1 -- start start 2024-08-03T20:48:23.707 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:23.704+0000 7f9258c00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f92540b38b0 0x7f9254155fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:48:23.707 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:23.705+0000 7f9258c00640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f92540b4ce0 0x7f9254158d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:48:23.707 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:23.705+0000 7f9258c00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f92541f66d0 0x7f92541f6ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:48:23.707 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:23.705+0000 7f9258c00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f92540cea30 con 0x7f92540b38b0 2024-08-03T20:48:23.707 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:23.705+0000 7f9258c00640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f92540ce8b0 con 0x7f92541f66d0 2024-08-03T20:48:23.707 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:23.705+0000 7f9258c00640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7f92540cebb0 con 0x7f92540b4ce0 2024-08-03T20:48:23.707 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:23.705+0000 7f9251e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f92540b38b0 0x7f9254155fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:48:23.707 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:23.705+0000 7f9251e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f92540b38b0 0x7f9254155fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:48148/0 (socket says 172.21.15.32:48148) 2024-08-03T20:48:23.707 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:23.705+0000 7f9251e00640 1 -- 172.21.15.32:0/3213869331 learned_addr learned my addr 172.21.15.32:0/3213869331 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:48:23.707 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:23.705+0000 7f9251400640 1 --2- 172.21.15.32:0/3213869331 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f92540b4ce0 0x7f9254158d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:48:23.708 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:23.705+0000 7f9252800640 1 --2- 172.21.15.32:0/3213869331 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f92541f66d0 0x7f92541f6ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:48:23.708 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:23.705+0000 7f9251e00640 1 -- 172.21.15.32:0/3213869331 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f92540b4ce0 msgr2=0x7f9254158d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:48:23.708 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:23.705+0000 7f9251e00640 1 --2- 172.21.15.32:0/3213869331 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f92540b4ce0 0x7f9254158d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:23.708 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:23.705+0000 7f9251e00640 1 -- 172.21.15.32:0/3213869331 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f92541f66d0 msgr2=0x7f92541f6ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:48:23.708 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:23.705+0000 7f9251e00640 1 --2- 172.21.15.32:0/3213869331 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f92541f66d0 0x7f92541f6ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:23.708 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:23.705+0000 7f9251e00640 1 -- 172.21.15.32:0/3213869331 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9254156610 con 0x7f92540b38b0 2024-08-03T20:48:23.708 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:23.705+0000 7f9252800640 1 --2- 172.21.15.32:0/3213869331 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f92541f66d0 0x7f92541f6ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-03T20:48:23.708 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:23.705+0000 7f9251400640 1 --2- 172.21.15.32:0/3213869331 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f92540b4ce0 0x7f9254158d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-03T20:48:23.708 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:23.706+0000 7f9251e00640 1 --2- 172.21.15.32:0/3213869331 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f92540b38b0 0x7f9254155fd0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f924400c3d0 tx=0x7f924400c890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:48:23.709 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:23.706+0000 7f924a000640 1 -- 172.21.15.32:0/3213869331 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9244017070 con 0x7f92540b38b0 2024-08-03T20:48:23.709 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:23.706+0000 7f924a000640 1 -- 172.21.15.32:0/3213869331 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f92440053c0 con 0x7f92540b38b0 2024-08-03T20:48:23.709 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:23.706+0000 7f9258c00640 1 -- 172.21.15.32:0/3213869331 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9254157ca0 con 0x7f92540b38b0 2024-08-03T20:48:23.709 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:23.706+0000 7f924a000640 1 -- 172.21.15.32:0/3213869331 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9244003850 con 0x7f92540b38b0 2024-08-03T20:48:23.709 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:23.706+0000 7f9258c00640 1 -- 172.21.15.32:0/3213869331 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9254158220 con 0x7f92540b38b0 2024-08-03T20:48:23.709 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:23.707+0000 7f9258c00640 1 -- 172.21.15.32:0/3213869331 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f92540cf0d0 con 0x7f92540b38b0 2024-08-03T20:48:23.710 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:23.708+0000 7f924a000640 1 -- 172.21.15.32:0/3213869331 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f924401a070 con 0x7f92540b38b0 2024-08-03T20:48:23.711 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:23.709+0000 7f924a000640 1 --2- 172.21.15.32:0/3213869331 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f9234077040 0x7f9234079550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:48:23.711 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:23.709+0000 7f924a000640 1 -- 172.21.15.32:0/3213869331 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(28..28 src has 1..28) ==== 4857+0+0 (secure 0 0 0) 0x7f9244097a30 con 0x7f92540b38b0 2024-08-03T20:48:23.711 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:23.710+0000 7f9251400640 1 --2- 172.21.15.32:0/3213869331 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f9234077040 0x7f9234079550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:48:23.712 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:23.710+0000 7f9251400640 1 --2- 172.21.15.32:0/3213869331 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f9234077040 0x7f9234079550 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f92541e4b60 tx=0x7f9240005f10 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:48:23.715 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:23.714+0000 7f924a000640 1 -- 172.21.15.32:0/3213869331 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7f9244060f70 con 0x7f92540b38b0 2024-08-03T20:48:24.005 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:24.004+0000 7f9258c00640 1 -- 172.21.15.32:0/3213869331 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-rgw"} v 0) -- 0x7f92540d03e0 con 0x7f92540b38b0 2024-08-03T20:48:24.008 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:24.007+0000 7f924a000640 1 -- 172.21.15.32:0/3213869331 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-rgw"}]=0 v10) ==== 90+0+0 (secure 0 0 0) 0x7f92440691b0 con 0x7f92540b38b0 2024-08-03T20:48:24.011 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:24.009+0000 7f9258c00640 1 -- 172.21.15.32:0/3213869331 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f9234077040 msgr2=0x7f9234079550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:48:24.011 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:24.009+0000 7f9258c00640 1 --2- 172.21.15.32:0/3213869331 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f9234077040 0x7f9234079550 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f92541e4b60 tx=0x7f9240005f10 comp rx=0 tx=0).stop 2024-08-03T20:48:24.011 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:24.009+0000 7f9258c00640 1 -- 172.21.15.32:0/3213869331 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f92540b38b0 msgr2=0x7f9254155fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:48:24.012 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:24.009+0000 7f9258c00640 1 --2- 172.21.15.32:0/3213869331 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f92540b38b0 0x7f9254155fd0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f924400c3d0 tx=0x7f924400c890 comp rx=0 tx=0).stop 2024-08-03T20:48:24.012 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:24.010+0000 7f9258c00640 1 -- 172.21.15.32:0/3213869331 shutdown_connections 2024-08-03T20:48:24.012 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:24.010+0000 7f9258c00640 1 --2- 172.21.15.32:0/3213869331 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f9234077040 0x7f9234079550 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:24.012 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:24.010+0000 7f9258c00640 1 --2- 172.21.15.32:0/3213869331 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f92541f66d0 0x7f92541f6ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:24.012 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:24.010+0000 7f9258c00640 1 --2- 172.21.15.32:0/3213869331 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f92540b4ce0 0x7f9254158d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:24.012 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:24.010+0000 7f9258c00640 1 --2- 172.21.15.32:0/3213869331 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f92540b38b0 0x7f9254155fd0 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:24.012 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:24.010+0000 7f9258c00640 1 -- 172.21.15.32:0/3213869331 >> 172.21.15.32:0/3213869331 conn(0x7f92540b1bd0 msgr2=0x7f9254150d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:48:24.012 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:24.010+0000 7f9258c00640 1 -- 172.21.15.32:0/3213869331 shutdown_connections 2024-08-03T20:48:24.012 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:24.010+0000 7f9258c00640 1 -- 172.21.15.32:0/3213869331 wait complete. 2024-08-03T20:48:24.037 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-08-03T20:48:24.266 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:24.264+0000 7fb14a000640 1 -- 172.21.15.32:0/2239841699 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb1440b74c0 msgr2=0x7fb1440ce880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:48:24.267 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:24.264+0000 7fb14a000640 1 --2- 172.21.15.32:0/2239841699 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb1440b74c0 0x7fb1440ce880 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fb1440745f0 tx=0x7fb13802a4d0 comp rx=0 tx=0).stop 2024-08-03T20:48:24.267 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:24.265+0000 7fb14a000640 1 -- 172.21.15.32:0/2239841699 shutdown_connections 2024-08-03T20:48:24.267 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:24.265+0000 7fb14a000640 1 --2- 172.21.15.32:0/2239841699 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fb1440b52a0 0x7fb1440b56b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:24.267 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:24.265+0000 7fb14a000640 1 --2- 172.21.15.32:0/2239841699 >> v2:172.21.15.118:3300/0 conn(0x7fb144161d20 0x7fb1440cedc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:24.267 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:24.265+0000 7fb14a000640 1 --2- 172.21.15.32:0/2239841699 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb1440b74c0 0x7fb1440ce880 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:24.267 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:24.265+0000 7fb14a000640 1 -- 172.21.15.32:0/2239841699 >> 172.21.15.32:0/2239841699 conn(0x7fb1440d1e90 msgr2=0x7fb1440d22a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:48:24.267 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:24.265+0000 7fb14a000640 1 -- 172.21.15.32:0/2239841699 shutdown_connections 2024-08-03T20:48:24.267 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:24.265+0000 7fb14a000640 1 -- 172.21.15.32:0/2239841699 wait complete. 2024-08-03T20:48:24.268 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:24.267+0000 7fb14a000640 1 Processor -- start 2024-08-03T20:48:24.268 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:24.267+0000 7fb14a000640 1 -- start start 2024-08-03T20:48:24.271 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:24.268+0000 7fb14a000640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fb1440b52a0 0x7fb1441e9c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:48:24.272 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:24.268+0000 7fb14a000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb1440b74c0 0x7fb1440b20f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:48:24.272 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:24.268+0000 7fb14a000640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb144161d20 0x7fb1440c4900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:48:24.272 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:24.268+0000 7fb14a000640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fb1440b5f80 con 0x7fb1440b74c0 2024-08-03T20:48:24.272 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:24.268+0000 7fb14a000640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7fb1440b5e00 con 0x7fb144161d20 2024-08-03T20:48:24.272 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:24.268+0000 7fb14a000640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7fb1440b6100 con 0x7fb1440b52a0 2024-08-03T20:48:24.272 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:24.269+0000 7fb143e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb1440b74c0 0x7fb1440b20f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:48:24.272 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:24.269+0000 7fb143e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb1440b74c0 0x7fb1440b20f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:48156/0 (socket says 172.21.15.32:48156) 2024-08-03T20:48:24.273 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:24.269+0000 7fb143e00640 1 -- 172.21.15.32:0/2276957848 learned_addr learned my addr 172.21.15.32:0/2276957848 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:48:24.273 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:24.269+0000 7fb148c00640 1 --2- 172.21.15.32:0/2276957848 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fb1440b52a0 0x7fb1441e9c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:48:24.273 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:24.269+0000 7fb149600640 1 --2- 172.21.15.32:0/2276957848 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb144161d20 0x7fb1440c4900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:48:24.273 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:24.269+0000 7fb143e00640 1 -- 172.21.15.32:0/2276957848 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fb1440b52a0 msgr2=0x7fb1441e9c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:48:24.273 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:24.269+0000 7fb143e00640 1 --2- 172.21.15.32:0/2276957848 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fb1440b52a0 0x7fb1441e9c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:24.273 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:24.269+0000 7fb143e00640 1 -- 172.21.15.32:0/2276957848 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb144161d20 msgr2=0x7fb1440c4900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:48:24.273 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:24.269+0000 7fb143e00640 1 --2- 172.21.15.32:0/2276957848 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb144161d20 0x7fb1440c4900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:24.273 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:24.269+0000 7fb143e00640 1 -- 172.21.15.32:0/2276957848 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb1441ea150 con 0x7fb1440b74c0 2024-08-03T20:48:24.273 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:24.269+0000 7fb148c00640 1 --2- 172.21.15.32:0/2276957848 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fb1440b52a0 0x7fb1441e9c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-03T20:48:24.273 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:24.269+0000 7fb143e00640 1 --2- 172.21.15.32:0/2276957848 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb1440b74c0 0x7fb1440b20f0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fb13000c970 tx=0x7fb13000ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:48:24.273 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:24.270+0000 7fb140c00640 1 -- 172.21.15.32:0/2276957848 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb130007b50 con 0x7fb1440b74c0 2024-08-03T20:48:24.273 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:24.270+0000 7fb140c00640 1 -- 172.21.15.32:0/2276957848 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fb130007de0 con 0x7fb1440b74c0 2024-08-03T20:48:24.273 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:24.270+0000 7fb140c00640 1 -- 172.21.15.32:0/2276957848 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb13000f6b0 con 0x7fb1440b74c0 2024-08-03T20:48:24.274 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:24.270+0000 7fb14a000640 1 -- 172.21.15.32:0/2276957848 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb1441e7b20 con 0x7fb1440b74c0 2024-08-03T20:48:24.274 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:24.270+0000 7fb14a000640 1 -- 172.21.15.32:0/2276957848 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb1441e80e0 con 0x7fb1440b74c0 2024-08-03T20:48:24.274 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:24.271+0000 7fb14a000640 1 -- 172.21.15.32:0/2276957848 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb1440b56b0 con 0x7fb1440b74c0 2024-08-03T20:48:24.274 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:24.272+0000 7fb140c00640 1 -- 172.21.15.32:0/2276957848 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7fb13000f850 con 0x7fb1440b74c0 2024-08-03T20:48:24.275 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:24.273+0000 7fb140c00640 1 --2- 172.21.15.32:0/2276957848 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fb124077010 0x7fb124079520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:48:24.275 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:24.273+0000 7fb140c00640 1 -- 172.21.15.32:0/2276957848 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(28..28 src has 1..28) ==== 4857+0+0 (secure 0 0 0) 0x7fb130098d30 con 0x7fb1440b74c0 2024-08-03T20:48:24.275 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:24.274+0000 7fb148c00640 1 --2- 172.21.15.32:0/2276957848 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fb124077010 0x7fb124079520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:48:24.276 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:24.275+0000 7fb148c00640 1 --2- 172.21.15.32:0/2276957848 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fb124077010 0x7fb124079520 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fb138033000 tx=0x7fb138004730 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:48:24.279 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:24.278+0000 7fb140c00640 1 -- 172.21.15.32:0/2276957848 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7fb130014030 con 0x7fb1440b74c0 2024-08-03T20:48:24.483 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:24.481+0000 7fb14a000640 1 -- 172.21.15.32:0/2276957848 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) -- 0x7fb144071a00 con 0x7fb1440b74c0 2024-08-03T20:48:24.483 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:24.482+0000 7fb140c00640 1 -- 172.21.15.32:0/2276957848 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v28) ==== 84+0+65 (secure 0 0 0) 0x7fb1300621e0 con 0x7fb1440b74c0 2024-08-03T20:48:24.484 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-03T20:48:24.484 INFO:teuthology.orchestra.run.smithi032.stdout:[ 2024-08-03T20:48:24.484 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-03T20:48:24.484 INFO:teuthology.orchestra.run.smithi032.stdout: "poolnum": 1, 2024-08-03T20:48:24.484 INFO:teuthology.orchestra.run.smithi032.stdout: "poolname": ".mgr" 2024-08-03T20:48:24.484 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-03T20:48:24.484 INFO:teuthology.orchestra.run.smithi032.stdout:] 2024-08-03T20:48:24.486 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:24.484+0000 7fb14a000640 1 -- 172.21.15.32:0/2276957848 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fb124077010 msgr2=0x7fb124079520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:48:24.486 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:24.484+0000 7fb14a000640 1 --2- 172.21.15.32:0/2276957848 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fb124077010 0x7fb124079520 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fb138033000 tx=0x7fb138004730 comp rx=0 tx=0).stop 2024-08-03T20:48:24.486 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:24.484+0000 7fb14a000640 1 -- 172.21.15.32:0/2276957848 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb1440b74c0 msgr2=0x7fb1440b20f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:48:24.486 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:24.484+0000 7fb14a000640 1 --2- 172.21.15.32:0/2276957848 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb1440b74c0 0x7fb1440b20f0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fb13000c970 tx=0x7fb13000ce30 comp rx=0 tx=0).stop 2024-08-03T20:48:24.486 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:24.484+0000 7fb14a000640 1 -- 172.21.15.32:0/2276957848 shutdown_connections 2024-08-03T20:48:24.486 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:24.484+0000 7fb14a000640 1 --2- 172.21.15.32:0/2276957848 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fb124077010 0x7fb124079520 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:24.486 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:24.484+0000 7fb14a000640 1 --2- 172.21.15.32:0/2276957848 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb144161d20 0x7fb1440c4900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:24.486 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:24.484+0000 7fb14a000640 1 --2- 172.21.15.32:0/2276957848 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb1440b74c0 0x7fb1440b20f0 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:24.486 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:24.484+0000 7fb14a000640 1 --2- 172.21.15.32:0/2276957848 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fb1440b52a0 0x7fb1441e9c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:24.487 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:24.484+0000 7fb14a000640 1 -- 172.21.15.32:0/2276957848 >> 172.21.15.32:0/2276957848 conn(0x7fb1440d1e90 msgr2=0x7fb144071f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:48:24.487 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:24.484+0000 7fb14a000640 1 -- 172.21.15.32:0/2276957848 shutdown_connections 2024-08-03T20:48:24.487 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:24.484+0000 7fb14a000640 1 -- 172.21.15.32:0/2276957848 wait complete. 2024-08-03T20:48:24.508 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-08-03T20:48:24.508 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-08-03T20:48:24.741 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:24.739+0000 7fcfbb200640 1 -- 172.21.15.32:0/4183400489 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcfa0003a50 msgr2=0x7fcfa0003ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:48:24.742 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:24.739+0000 7fcfbb200640 1 --2- 172.21.15.32:0/4183400489 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcfa0003a50 0x7fcfa0003ec0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7fcfac009b50 tx=0x7fcfa0005830 comp rx=0 tx=0).stop 2024-08-03T20:48:24.742 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:24.740+0000 7fcfbb200640 1 -- 172.21.15.32:0/4183400489 shutdown_connections 2024-08-03T20:48:24.742 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:24.740+0000 7fcfbb200640 1 --2- 172.21.15.32:0/4183400489 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcfa0003a50 0x7fcfa0003ec0 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:24.742 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:24.740+0000 7fcfbb200640 1 -- 172.21.15.32:0/4183400489 >> 172.21.15.32:0/4183400489 conn(0x7fcfbc0cfd80 msgr2=0x7fcfbc0d0190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:48:24.742 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:24.740+0000 7fcfbb200640 1 -- 172.21.15.32:0/4183400489 shutdown_connections 2024-08-03T20:48:24.742 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:24.740+0000 7fcfbb200640 1 -- 172.21.15.32:0/4183400489 wait complete. 2024-08-03T20:48:24.743 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:24.742+0000 7fcfbb200640 1 Processor -- start 2024-08-03T20:48:24.743 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:24.742+0000 7fcfbb200640 1 -- start start 2024-08-03T20:48:24.746 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:24.743+0000 7fcfbb200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcfa0003a50 0x7fcfbc15a560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:48:24.746 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:24.743+0000 7fcfbb200640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fcfbc14f320 0x7fcfbc073560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:48:24.746 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:24.743+0000 7fcfbb200640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fcfbc073aa0 0x7fcfbc158a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:48:24.746 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:24.743+0000 7fcfbb200640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fcfbc15c570 con 0x7fcfa0003a50 2024-08-03T20:48:24.746 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:24.743+0000 7fcfbb200640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7fcfbc15c3f0 con 0x7fcfbc073aa0 2024-08-03T20:48:24.746 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:24.743+0000 7fcfbb200640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7fcfbc0cdbf0 con 0x7fcfbc14f320 2024-08-03T20:48:24.746 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:24.744+0000 7fcfb9e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcfa0003a50 0x7fcfbc15a560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:48:24.746 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:24.744+0000 7fcfb9e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcfa0003a50 0x7fcfbc15a560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:48162/0 (socket says 172.21.15.32:48162) 2024-08-03T20:48:24.746 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:24.744+0000 7fcfb9e00640 1 -- 172.21.15.32:0/728605278 learned_addr learned my addr 172.21.15.32:0/728605278 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:48:24.746 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:24.744+0000 7fcfba800640 1 --2- 172.21.15.32:0/728605278 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fcfbc073aa0 0x7fcfbc158a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:48:24.747 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:24.744+0000 7fcfb9400640 1 --2- 172.21.15.32:0/728605278 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fcfbc14f320 0x7fcfbc073560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:48:24.747 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:24.744+0000 7fcfb9e00640 1 -- 172.21.15.32:0/728605278 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fcfbc14f320 msgr2=0x7fcfbc073560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:48:24.747 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:24.744+0000 7fcfb9e00640 1 --2- 172.21.15.32:0/728605278 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fcfbc14f320 0x7fcfbc073560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:24.747 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:24.744+0000 7fcfb9e00640 1 -- 172.21.15.32:0/728605278 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fcfbc073aa0 msgr2=0x7fcfbc158a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:48:24.747 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:24.744+0000 7fcfb9e00640 1 --2- 172.21.15.32:0/728605278 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fcfbc073aa0 0x7fcfbc158a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:24.747 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:24.744+0000 7fcfb9e00640 1 -- 172.21.15.32:0/728605278 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcfbc073eb0 con 0x7fcfa0003a50 2024-08-03T20:48:24.747 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:24.744+0000 7fcfb9400640 1 --2- 172.21.15.32:0/728605278 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fcfbc14f320 0x7fcfbc073560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-03T20:48:24.747 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:24.744+0000 7fcfb9e00640 1 --2- 172.21.15.32:0/728605278 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcfa0003a50 0x7fcfbc15a560 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7fcfac009a70 tx=0x7fcfac0336f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:48:24.748 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:24.745+0000 7fcfb2000640 1 -- 172.21.15.32:0/728605278 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfac038040 con 0x7fcfa0003a50 2024-08-03T20:48:24.748 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:24.745+0000 7fcfb2000640 1 -- 172.21.15.32:0/728605278 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fcfac0338e0 con 0x7fcfa0003a50 2024-08-03T20:48:24.748 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:24.745+0000 7fcfb2000640 1 -- 172.21.15.32:0/728605278 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfac033d30 con 0x7fcfa0003a50 2024-08-03T20:48:24.748 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:24.745+0000 7fcfbb200640 1 -- 172.21.15.32:0/728605278 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcfbc0b59b0 con 0x7fcfa0003a50 2024-08-03T20:48:24.748 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:24.745+0000 7fcfbb200640 1 -- 172.21.15.32:0/728605278 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcfbc1f8990 con 0x7fcfa0003a50 2024-08-03T20:48:24.748 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:24.746+0000 7fcfbb200640 1 -- 172.21.15.32:0/728605278 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcfbc0ce150 con 0x7fcfa0003a50 2024-08-03T20:48:24.749 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:24.747+0000 7fcfb2000640 1 -- 172.21.15.32:0/728605278 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7fcfac043020 con 0x7fcfa0003a50 2024-08-03T20:48:24.749 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:24.748+0000 7fcfb2000640 1 --2- 172.21.15.32:0/728605278 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fcf9c077010 0x7fcf9c079520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:48:24.750 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:24.748+0000 7fcfb2000640 1 -- 172.21.15.32:0/728605278 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(28..28 src has 1..28) ==== 4857+0+0 (secure 0 0 0) 0x7fcfac0b8800 con 0x7fcfa0003a50 2024-08-03T20:48:24.750 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:24.749+0000 7fcfb9400640 1 --2- 172.21.15.32:0/728605278 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fcf9c077010 0x7fcf9c079520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:48:24.750 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:24.749+0000 7fcfb9400640 1 --2- 172.21.15.32:0/728605278 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fcf9c077010 0x7fcf9c079520 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fcfbc1eb420 tx=0x7fcfa80067d0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:48:24.754 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:24.752+0000 7fcfb2000640 1 -- 172.21.15.32:0/728605278 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7fcfac04a020 con 0x7fcfa0003a50 2024-08-03T20:48:24.957 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:24.956+0000 7fcfbb200640 1 -- 172.21.15.32:0/728605278 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) -- 0x7fcfbc0748d0 con 0x7fcfa0003a50 2024-08-03T20:48:25.103 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:25.102+0000 7fcfb2000640 1 -- 172.21.15.32:0/728605278 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v29) ==== 138+0+0 (secure 0 0 0) 0x7fcfbc073eb0 con 0x7fcfa0003a50 2024-08-03T20:48:25.104 INFO:teuthology.orchestra.run.smithi032.stderr:pool 'cephfs_metadata' created 2024-08-03T20:48:25.107 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:25.105+0000 7fcfbb200640 1 -- 172.21.15.32:0/728605278 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fcf9c077010 msgr2=0x7fcf9c079520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:48:25.107 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:25.105+0000 7fcfbb200640 1 --2- 172.21.15.32:0/728605278 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fcf9c077010 0x7fcf9c079520 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fcfbc1eb420 tx=0x7fcfa80067d0 comp rx=0 tx=0).stop 2024-08-03T20:48:25.107 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:25.105+0000 7fcfbb200640 1 -- 172.21.15.32:0/728605278 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcfa0003a50 msgr2=0x7fcfbc15a560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:48:25.108 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:25.105+0000 7fcfbb200640 1 --2- 172.21.15.32:0/728605278 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcfa0003a50 0x7fcfbc15a560 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7fcfac009a70 tx=0x7fcfac0336f0 comp rx=0 tx=0).stop 2024-08-03T20:48:25.108 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:25.105+0000 7fcfbb200640 1 -- 172.21.15.32:0/728605278 shutdown_connections 2024-08-03T20:48:25.108 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:25.105+0000 7fcfbb200640 1 --2- 172.21.15.32:0/728605278 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fcf9c077010 0x7fcf9c079520 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:25.108 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:25.105+0000 7fcfbb200640 1 --2- 172.21.15.32:0/728605278 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fcfbc073aa0 0x7fcfbc158a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:25.108 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:25.105+0000 7fcfbb200640 1 --2- 172.21.15.32:0/728605278 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fcfbc14f320 0x7fcfbc073560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:25.109 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:25.105+0000 7fcfbb200640 1 --2- 172.21.15.32:0/728605278 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcfa0003a50 0x7fcfbc15a560 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:25.109 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:25.105+0000 7fcfbb200640 1 -- 172.21.15.32:0/728605278 >> 172.21.15.32:0/728605278 conn(0x7fcfbc0cfd80 msgr2=0x7fcfbc0affc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:48:25.109 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:25.105+0000 7fcfbb200640 1 -- 172.21.15.32:0/728605278 shutdown_connections 2024-08-03T20:48:25.109 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:25.105+0000 7fcfbb200640 1 -- 172.21.15.32:0/728605278 wait complete. 2024-08-03T20:48:25.133 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-08-03T20:48:25.369 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:25.367+0000 7f9682800640 1 -- 172.21.15.32:0/1959295117 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f967c0ced30 msgr2=0x7f967c0cf140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:48:25.369 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:25.367+0000 7f9682800640 1 --2- 172.21.15.32:0/1959295117 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f967c0ced30 0x7f967c0cf140 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f9674005fd0 tx=0x7f967402aa90 comp rx=0 tx=0).stop 2024-08-03T20:48:25.369 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:25.367+0000 7f9682800640 1 -- 172.21.15.32:0/1959295117 shutdown_connections 2024-08-03T20:48:25.369 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:25.367+0000 7f9682800640 1 --2- 172.21.15.32:0/1959295117 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f967c0ced30 0x7f967c0cf140 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:25.369 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:25.367+0000 7f9682800640 1 --2- 172.21.15.32:0/1959295117 >> v2:172.21.15.118:3300/0 conn(0x7f967c0cccf0 0x7f967c0cd100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:25.370 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:25.367+0000 7f9682800640 1 -- 172.21.15.32:0/1959295117 >> 172.21.15.32:0/1959295117 conn(0x7f967c0b1bd0 msgr2=0x7f967c0b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:48:25.370 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:25.367+0000 7f9682800640 1 -- 172.21.15.32:0/1959295117 shutdown_connections 2024-08-03T20:48:25.370 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:25.367+0000 7f9682800640 1 -- 172.21.15.32:0/1959295117 wait complete. 2024-08-03T20:48:25.371 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:25.370+0000 7f9682800640 1 Processor -- start 2024-08-03T20:48:25.371 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:25.370+0000 7f9682800640 1 -- start start 2024-08-03T20:48:25.373 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:25.371+0000 7f9682800640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f967c0cccf0 0x7f967c1f5230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:48:25.373 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:25.371+0000 7f9682800640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f967c0ced30 0x7f967c13efc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:48:25.373 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:25.371+0000 7f9682800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f967c145c50 0x7f967c144860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:48:25.374 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:25.371+0000 7f9682800640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f967c0b4070 con 0x7f967c145c50 2024-08-03T20:48:25.374 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:25.371+0000 7f9682800640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f967c0b3ef0 con 0x7f967c0ced30 2024-08-03T20:48:25.374 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:25.371+0000 7f9682800640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7f967c0b41f0 con 0x7f967c0cccf0 2024-08-03T20:48:25.374 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:25.372+0000 7f967be00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f967c145c50 0x7f967c144860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:48:25.374 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:25.372+0000 7f967be00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f967c145c50 0x7f967c144860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:48174/0 (socket says 172.21.15.32:48174) 2024-08-03T20:48:25.374 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:25.372+0000 7f967be00640 1 -- 172.21.15.32:0/3483880747 learned_addr learned my addr 172.21.15.32:0/3483880747 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:48:25.374 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:25.372+0000 7f9673e00640 1 --2- 172.21.15.32:0/3483880747 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f967c0ced30 0x7f967c13efc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:48:25.374 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:25.372+0000 7f967b400640 1 --2- 172.21.15.32:0/3483880747 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f967c0cccf0 0x7f967c1f5230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:48:25.374 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:25.372+0000 7f967be00640 1 -- 172.21.15.32:0/3483880747 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f967c0cccf0 msgr2=0x7f967c1f5230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:48:25.374 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:25.372+0000 7f967be00640 1 --2- 172.21.15.32:0/3483880747 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f967c0cccf0 0x7f967c1f5230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:25.374 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:25.372+0000 7f967be00640 1 -- 172.21.15.32:0/3483880747 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f967c0ced30 msgr2=0x7f967c13efc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:48:25.374 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:25.372+0000 7f967be00640 1 --2- 172.21.15.32:0/3483880747 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f967c0ced30 0x7f967c13efc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:25.374 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:25.372+0000 7f967be00640 1 -- 172.21.15.32:0/3483880747 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f967c150540 con 0x7f967c145c50 2024-08-03T20:48:25.375 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:25.372+0000 7f967be00640 1 --2- 172.21.15.32:0/3483880747 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f967c145c50 0x7f967c144860 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f9674009d90 tx=0x7f9674003ce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:48:25.375 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:25.373+0000 7f9673400640 1 -- 172.21.15.32:0/3483880747 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9674044070 con 0x7f967c145c50 2024-08-03T20:48:25.375 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:25.373+0000 7f9673400640 1 -- 172.21.15.32:0/3483880747 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f9674004040 con 0x7f967c145c50 2024-08-03T20:48:25.376 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:25.373+0000 7f9682800640 1 -- 172.21.15.32:0/3483880747 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f967c150820 con 0x7f967c145c50 2024-08-03T20:48:25.376 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:25.373+0000 7f9673400640 1 -- 172.21.15.32:0/3483880747 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9674004430 con 0x7f967c145c50 2024-08-03T20:48:25.376 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:25.373+0000 7f9682800640 1 -- 172.21.15.32:0/3483880747 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f967c150de0 con 0x7f967c145c50 2024-08-03T20:48:25.376 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:25.374+0000 7f9682800640 1 -- 172.21.15.32:0/3483880747 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f967c0ce610 con 0x7f967c145c50 2024-08-03T20:48:25.376 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:25.375+0000 7f9673400640 1 -- 172.21.15.32:0/3483880747 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f9674047070 con 0x7f967c145c50 2024-08-03T20:48:25.378 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:25.376+0000 7f9673400640 1 --2- 172.21.15.32:0/3483880747 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f9658077040 0x7f9658079550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:48:25.378 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:25.376+0000 7f9673400640 1 -- 172.21.15.32:0/3483880747 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(29..29 src has 1..29) ==== 5227+0+0 (secure 0 0 0) 0x7f96740b94b0 con 0x7f967c145c50 2024-08-03T20:48:25.378 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:25.377+0000 7f967b400640 1 --2- 172.21.15.32:0/3483880747 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f9658077040 0x7f9658079550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:48:25.379 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:25.378+0000 7f967b400640 1 --2- 172.21.15.32:0/3483880747 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f9658077040 0x7f9658079550 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f966c000f60 tx=0x7f966c009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:48:25.383 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:25.382+0000 7f9673400640 1 -- 172.21.15.32:0/3483880747 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7f9674032030 con 0x7f967c145c50 2024-08-03T20:48:25.592 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:25.591+0000 7f9682800640 1 -- 172.21.15.32:0/3483880747 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) -- 0x7f967c0cd190 con 0x7f967c145c50 2024-08-03T20:48:26.112 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:26.110+0000 7f9673400640 1 -- 172.21.15.32:0/3483880747 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v30) ==== 170+0+0 (secure 0 0 0) 0x7f9674082240 con 0x7f967c145c50 2024-08-03T20:48:26.112 INFO:teuthology.orchestra.run.smithi032.stderr:pool 'cephfs_data' created 2024-08-03T20:48:26.115 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:26.113+0000 7f9682800640 1 -- 172.21.15.32:0/3483880747 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f9658077040 msgr2=0x7f9658079550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:48:26.115 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:26.113+0000 7f9682800640 1 --2- 172.21.15.32:0/3483880747 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f9658077040 0x7f9658079550 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f966c000f60 tx=0x7f966c009040 comp rx=0 tx=0).stop 2024-08-03T20:48:26.115 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:26.113+0000 7f9682800640 1 -- 172.21.15.32:0/3483880747 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f967c145c50 msgr2=0x7f967c144860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:48:26.115 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:26.113+0000 7f9682800640 1 --2- 172.21.15.32:0/3483880747 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f967c145c50 0x7f967c144860 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f9674009d90 tx=0x7f9674003ce0 comp rx=0 tx=0).stop 2024-08-03T20:48:26.115 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:26.114+0000 7f9682800640 1 -- 172.21.15.32:0/3483880747 shutdown_connections 2024-08-03T20:48:26.115 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:26.114+0000 7f9682800640 1 --2- 172.21.15.32:0/3483880747 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f9658077040 0x7f9658079550 secure :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f966c000f60 tx=0x7f966c009040 comp rx=0 tx=0).stop 2024-08-03T20:48:26.116 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:26.114+0000 7f9682800640 1 --2- 172.21.15.32:0/3483880747 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f967c0ced30 0x7f967c13efc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:26.116 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:26.114+0000 7f9682800640 1 --2- 172.21.15.32:0/3483880747 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f967c145c50 0x7f967c144860 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:26.116 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:26.114+0000 7f9682800640 1 --2- 172.21.15.32:0/3483880747 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f967c0cccf0 0x7f967c1f5230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:26.116 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:26.114+0000 7f9682800640 1 -- 172.21.15.32:0/3483880747 >> 172.21.15.32:0/3483880747 conn(0x7f967c0b1bd0 msgr2=0x7f967c06f890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:48:26.118 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:26.117+0000 7f9682800640 1 -- 172.21.15.32:0/3483880747 shutdown_connections 2024-08-03T20:48:26.119 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:26.117+0000 7f9682800640 1 -- 172.21.15.32:0/3483880747 wait complete. 2024-08-03T20:48:26.160 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-08-03T20:48:26.395 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:26.393+0000 7f558be00640 1 -- 172.21.15.32:0/1358546765 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5570004800 msgr2=0x7f5570004c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:48:26.395 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:26.393+0000 7f558be00640 1 --2- 172.21.15.32:0/1358546765 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5570004800 0x7f5570004c10 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f557400c290 tx=0x7f5570005830 comp rx=0 tx=0).stop 2024-08-03T20:48:26.396 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:26.393+0000 7f558b400640 1 -- 172.21.15.32:0/1358546765 reap_dead start 2024-08-03T20:48:26.396 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:26.393+0000 7f558be00640 1 -- 172.21.15.32:0/1358546765 shutdown_connections 2024-08-03T20:48:26.396 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:26.393+0000 7f558be00640 1 -- 172.21.15.32:0/1358546765 >> 172.21.15.32:0/1358546765 conn(0x7f558c0cfd80 msgr2=0x7f558c0d0190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:48:26.396 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:26.393+0000 7f558be00640 1 -- 172.21.15.32:0/1358546765 shutdown_connections 2024-08-03T20:48:26.396 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:26.393+0000 7f558be00640 1 -- 172.21.15.32:0/1358546765 wait complete. 2024-08-03T20:48:26.397 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:26.395+0000 7f558be00640 1 Processor -- start 2024-08-03T20:48:26.397 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:26.396+0000 7f558be00640 1 -- start start 2024-08-03T20:48:26.401 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:26.397+0000 7f558be00640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f5570004800 0x7f558c15b790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:48:26.402 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:26.397+0000 7f558be00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f558c0c8520 0x7f558c1f6230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:48:26.402 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:26.397+0000 7f558be00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f558c158130 0x7f558c158540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:48:26.402 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:26.397+0000 7f558be00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f558c0cb740 con 0x7f558c0c8520 2024-08-03T20:48:26.402 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:26.397+0000 7f558be00640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f558c160730 con 0x7f558c158130 2024-08-03T20:48:26.402 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:26.397+0000 7f558be00640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7f558c06e630 con 0x7f5570004800 2024-08-03T20:48:26.403 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:26.397+0000 7f5583e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f558c0c8520 0x7f558c1f6230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:48:26.403 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:26.397+0000 7f5583e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f558c0c8520 0x7f558c1f6230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:48192/0 (socket says 172.21.15.32:48192) 2024-08-03T20:48:26.403 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:26.397+0000 7f5583e00640 1 -- 172.21.15.32:0/1183721254 learned_addr learned my addr 172.21.15.32:0/1183721254 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:48:26.403 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:26.398+0000 7f558aa00640 1 --2- 172.21.15.32:0/1183721254 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f5570004800 0x7f558c15b790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:48:26.403 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:26.398+0000 7f558b400640 1 --2- 172.21.15.32:0/1183721254 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f558c158130 0x7f558c158540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:48:26.403 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:26.398+0000 7f5583e00640 1 -- 172.21.15.32:0/1183721254 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f5570004800 msgr2=0x7f558c15b790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:48:26.403 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:26.398+0000 7f5583e00640 1 --2- 172.21.15.32:0/1183721254 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f5570004800 0x7f558c15b790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:26.403 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:26.398+0000 7f5583e00640 1 -- 172.21.15.32:0/1183721254 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f558c158130 msgr2=0x7f558c158540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:48:26.404 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:26.398+0000 7f5583e00640 1 --2- 172.21.15.32:0/1183721254 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f558c158130 0x7f558c158540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:26.404 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:26.398+0000 7f5583e00640 1 -- 172.21.15.32:0/1183721254 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f558c0c8930 con 0x7f558c0c8520 2024-08-03T20:48:26.404 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:26.398+0000 7f5583e00640 1 --2- 172.21.15.32:0/1183721254 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f558c0c8520 0x7f558c1f6230 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f557400c3b0 tx=0x7f5574001d20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:48:26.404 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:26.398+0000 7f5582a00640 1 -- 172.21.15.32:0/1183721254 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5574031040 con 0x7f558c0c8520 2024-08-03T20:48:26.404 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:26.398+0000 7f5582a00640 1 -- 172.21.15.32:0/1183721254 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f55740035f0 con 0x7f558c0c8520 2024-08-03T20:48:26.404 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:26.398+0000 7f558be00640 1 -- 172.21.15.32:0/1183721254 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f558c1f6960 con 0x7f558c0c8520 2024-08-03T20:48:26.404 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:26.398+0000 7f5582a00640 1 -- 172.21.15.32:0/1183721254 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5574003a40 con 0x7f558c0c8520 2024-08-03T20:48:26.404 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:26.399+0000 7f558be00640 1 -- 172.21.15.32:0/1183721254 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f558c1f6f20 con 0x7f558c0c8520 2024-08-03T20:48:26.404 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:26.400+0000 7f558be00640 1 -- 172.21.15.32:0/1183721254 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f558c1f2c40 con 0x7f558c0c8520 2024-08-03T20:48:26.406 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:26.401+0000 7f5582a00640 1 -- 172.21.15.32:0/1183721254 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f5574034070 con 0x7f558c0c8520 2024-08-03T20:48:26.406 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:26.402+0000 7f5582a00640 1 --2- 172.21.15.32:0/1183721254 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f5568077010 0x7f5568079520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:48:26.406 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:26.402+0000 7f5582a00640 1 -- 172.21.15.32:0/1183721254 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 5609+0+0 (secure 0 0 0) 0x7f55740b8960 con 0x7f558c0c8520 2024-08-03T20:48:26.406 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:26.403+0000 7f558aa00640 1 --2- 172.21.15.32:0/1183721254 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f5568077010 0x7f5568079520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:48:26.406 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:26.403+0000 7f558aa00640 1 --2- 172.21.15.32:0/1183721254 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f5568077010 0x7f5568079520 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f557c009aa0 tx=0x7f557c005f90 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:48:26.408 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:26.407+0000 7f5582a00640 1 -- 172.21.15.32:0/1183721254 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7f5574081b20 con 0x7f558c0c8520 2024-08-03T20:48:26.652 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:26.650+0000 7f558be00640 1 -- 172.21.15.32:0/1183721254 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) -- 0x7f558c0cc050 con 0x7f558c0c8520 2024-08-03T20:48:27.128 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:27.123+0000 7f5582a00640 1 -- 172.21.15.32:0/1183721254 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 4 and data pool 5 v13) ==== 172+0+0 (secure 0 0 0) 0x7f5574081420 con 0x7f558c0c8520 2024-08-03T20:48:27.128 INFO:teuthology.orchestra.run.smithi032.stderr:new fs with metadata pool 4 and data pool 5 2024-08-03T20:48:27.131 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:27.129+0000 7f558be00640 1 -- 172.21.15.32:0/1183721254 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f5568077010 msgr2=0x7f5568079520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:48:27.131 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:27.129+0000 7f558be00640 1 --2- 172.21.15.32:0/1183721254 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f5568077010 0x7f5568079520 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f557c009aa0 tx=0x7f557c005f90 comp rx=0 tx=0).stop 2024-08-03T20:48:27.131 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:27.129+0000 7f558be00640 1 -- 172.21.15.32:0/1183721254 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f558c0c8520 msgr2=0x7f558c1f6230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:48:27.131 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:27.129+0000 7f558be00640 1 --2- 172.21.15.32:0/1183721254 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f558c0c8520 0x7f558c1f6230 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f557400c3b0 tx=0x7f5574001d20 comp rx=0 tx=0).stop 2024-08-03T20:48:27.133 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:27.131+0000 7f558be00640 1 -- 172.21.15.32:0/1183721254 shutdown_connections 2024-08-03T20:48:27.133 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:27.131+0000 7f558be00640 1 --2- 172.21.15.32:0/1183721254 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f5568077010 0x7f5568079520 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:27.133 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:27.132+0000 7f558be00640 1 --2- 172.21.15.32:0/1183721254 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f558c158130 0x7f558c158540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:27.133 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:27.132+0000 7f558be00640 1 --2- 172.21.15.32:0/1183721254 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f558c0c8520 0x7f558c1f6230 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:27.133 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:27.132+0000 7f558be00640 1 --2- 172.21.15.32:0/1183721254 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f5570004800 0x7f558c15b790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:27.133 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:27.132+0000 7f558be00640 1 -- 172.21.15.32:0/1183721254 >> 172.21.15.32:0/1183721254 conn(0x7f558c0cfd80 msgr2=0x7f558c0b5020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:48:27.134 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:27.132+0000 7f558be00640 1 -- 172.21.15.32:0/1183721254 shutdown_connections 2024-08-03T20:48:27.134 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:27.132+0000 7f558be00640 1 -- 172.21.15.32:0/1183721254 wait complete. 2024-08-03T20:48:27.174 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-08-03T20:48:27.402 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:27.400+0000 7fc469e00640 1 -- 172.21.15.32:0/1512391654 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc464073e60 msgr2=0x7fc464074270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:48:27.403 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:27.400+0000 7fc469e00640 1 --2- 172.21.15.32:0/1512391654 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc464073e60 0x7fc464074270 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7fc464071700 tx=0x7fc45002a580 comp rx=0 tx=0).stop 2024-08-03T20:48:27.403 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:27.401+0000 7fc469e00640 1 -- 172.21.15.32:0/1512391654 shutdown_connections 2024-08-03T20:48:27.403 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:27.401+0000 7fc469e00640 1 --2- 172.21.15.32:0/1512391654 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc464073e60 0x7fc464074270 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:27.403 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:27.401+0000 7fc469e00640 1 --2- 172.21.15.32:0/1512391654 >> v2:172.21.15.118:3300/0 conn(0x7fc46406e930 0x7fc464144f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:27.403 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:27.401+0000 7fc469e00640 1 -- 172.21.15.32:0/1512391654 >> 172.21.15.32:0/1512391654 conn(0x7fc4640b1bd0 msgr2=0x7fc4640b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:48:27.403 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:27.401+0000 7fc469e00640 1 -- 172.21.15.32:0/1512391654 shutdown_connections 2024-08-03T20:48:27.403 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:27.401+0000 7fc469e00640 1 -- 172.21.15.32:0/1512391654 wait complete. 2024-08-03T20:48:27.404 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:27.403+0000 7fc469e00640 1 Processor -- start 2024-08-03T20:48:27.404 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:27.403+0000 7fc469e00640 1 -- start start 2024-08-03T20:48:27.407 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:27.404+0000 7fc469e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc46406e930 0x7fc4641f5da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:48:27.407 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:27.404+0000 7fc469e00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc464073e60 0x7fc4640a94b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:48:27.407 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:27.404+0000 7fc469e00640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fc4640747b0 0x7fc46415df50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:48:27.407 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:27.404+0000 7fc469e00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fc4640cf9c0 con 0x7fc46406e930 2024-08-03T20:48:27.407 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:27.404+0000 7fc469e00640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7fc4640cf840 con 0x7fc464073e60 2024-08-03T20:48:27.407 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:27.404+0000 7fc469e00640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7fc4640ccff0 con 0x7fc4640747b0 2024-08-03T20:48:27.408 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:27.405+0000 7fc462a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc46406e930 0x7fc4641f5da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:48:27.408 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:27.405+0000 7fc462a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc46406e930 0x7fc4641f5da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:48220/0 (socket says 172.21.15.32:48220) 2024-08-03T20:48:27.408 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:27.405+0000 7fc462a00640 1 -- 172.21.15.32:0/1013864147 learned_addr learned my addr 172.21.15.32:0/1013864147 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:48:27.408 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:27.405+0000 7fc462000640 1 --2- 172.21.15.32:0/1013864147 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc464073e60 0x7fc4640a94b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:48:27.408 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:27.405+0000 7fc463400640 1 --2- 172.21.15.32:0/1013864147 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fc4640747b0 0x7fc46415df50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:48:27.408 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:27.405+0000 7fc462a00640 1 -- 172.21.15.32:0/1013864147 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fc4640747b0 msgr2=0x7fc46415df50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:48:27.408 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:27.405+0000 7fc462a00640 1 --2- 172.21.15.32:0/1013864147 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fc4640747b0 0x7fc46415df50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:27.408 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:27.405+0000 7fc462a00640 1 -- 172.21.15.32:0/1013864147 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc464073e60 msgr2=0x7fc4640a94b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:48:27.408 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:27.405+0000 7fc462a00640 1 --2- 172.21.15.32:0/1013864147 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc464073e60 0x7fc4640a94b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:27.408 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:27.405+0000 7fc462a00640 1 -- 172.21.15.32:0/1013864147 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc46415ceb0 con 0x7fc46406e930 2024-08-03T20:48:27.408 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:27.405+0000 7fc462000640 1 --2- 172.21.15.32:0/1013864147 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc464073e60 0x7fc4640a94b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-03T20:48:27.408 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:27.405+0000 7fc462a00640 1 --2- 172.21.15.32:0/1013864147 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc46406e930 0x7fc4641f5da0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7fc45400d740 tx=0x7fc45400dc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:48:27.408 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:27.406+0000 7fc45b400640 1 -- 172.21.15.32:0/1013864147 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc454017070 con 0x7fc46406e930 2024-08-03T20:48:27.408 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:27.406+0000 7fc45b400640 1 -- 172.21.15.32:0/1013864147 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fc454009d20 con 0x7fc46406e930 2024-08-03T20:48:27.409 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:27.406+0000 7fc469e00640 1 -- 172.21.15.32:0/1013864147 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc46415d190 con 0x7fc46406e930 2024-08-03T20:48:27.409 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:27.406+0000 7fc45b400640 1 -- 172.21.15.32:0/1013864147 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4540106c0 con 0x7fc46406e930 2024-08-03T20:48:27.409 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:27.406+0000 7fc469e00640 1 -- 172.21.15.32:0/1013864147 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc46415d750 con 0x7fc46406e930 2024-08-03T20:48:27.409 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:27.407+0000 7fc469e00640 1 -- 172.21.15.32:0/1013864147 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc4640cd580 con 0x7fc46406e930 2024-08-03T20:48:27.409 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:27.408+0000 7fc45b400640 1 -- 172.21.15.32:0/1013864147 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7fc45401a070 con 0x7fc46406e930 2024-08-03T20:48:27.411 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:27.410+0000 7fc45b400640 1 --2- 172.21.15.32:0/1013864147 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fc444077040 0x7fc444079550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:48:27.412 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:27.410+0000 7fc45b400640 1 -- 172.21.15.32:0/1013864147 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(31..31 src has 1..31) ==== 5709+0+0 (secure 0 0 0) 0x7fc454098b00 con 0x7fc46406e930 2024-08-03T20:48:27.412 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:27.410+0000 7fc462000640 1 --2- 172.21.15.32:0/1013864147 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fc444077040 0x7fc444079550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:48:27.412 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:27.411+0000 7fc462000640 1 --2- 172.21.15.32:0/1013864147 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fc444077040 0x7fc444079550 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fc450033000 tx=0x7fc450004730 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:48:27.416 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:27.414+0000 7fc45b400640 1 -- 172.21.15.32:0/1013864147 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7fc454061be0 con 0x7fc46406e930 2024-08-03T20:48:27.620 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:27.619+0000 7fc469e00640 1 -- 172.21.15.32:0/1013864147 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fc46406fb40 con 0x7fc46406e930 2024-08-03T20:48:27.622 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:27.620+0000 7fc45b400640 1 -- 172.21.15.32:0/1013864147 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v31) ==== 74+0+16622 (secure 0 0 0) 0x7fc4540614e0 con 0x7fc46406e930 2024-08-03T20:48:27.622 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-03T20:48:27.623 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":31,"fsid":"40ab141a-01d7-49b9-ae3c-2edec9876e95","created":"2024-08-03T20:47:14.957362+0000","modified":"2024-08-03T20:48:26.652562+0000","last_up_change":"2024-08-03T20:47:28.874451+0000","last_in_change":"2024-08-03T20:47:20.079442+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-08-03T20:47:28.261300+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_metadata","create_time":"2024-08-03T20:48:24.969840+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"31","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data","create_time":"2024-08-03T20:48:25.603595+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"31","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5a83e759-9035-438b-889a-0a6af1f0a0d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6816","nonce":3066156116},{"type":"v1","addr":"172.21.15.32:6817","nonce":3066156116}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6818","nonce":3066156116},{"type":"v1","addr":"172.21.15.32:6819","nonce":3066156116}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6822","nonce":3066156116},{"type":"v1","addr":"172.21.15.32:6823","nonce":3066156116}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6820","nonce":3066156116},{"type":"v1","addr":"172.21.15.32:6821","nonce":3066156116}]},"public_addr":"172.21.15.32:6817/3066156116","cluster_addr":"172.21.15.32:6819/3066156116","heartbeat_back_addr":"172.21.15.32:6823/3066156116","heartbeat_front_addr":"172.21.15.32:6821/3066156116","state":["exists","up"]},{"osd":1,"uuid":"d7c5c210-da4f-49f9-b9e7-1dfb02daa4bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6824","nonce":1551241759},{"type":"v1","addr":"172.21.15.32:6825","nonce":1551241759}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6826","nonce":1551241759},{"type":"v1","addr":"172.21.15.32:6827","nonce":1551241759}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6830","nonce":1551241759},{"type":"v1","addr":"172.21.15.32:6831","nonce":1551241759}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6828","nonce":1551241759},{"type":"v1","addr":"172.21.15.32:6829","nonce":1551241759}]},"public_addr":"172.21.15.32:6825/1551241759","cluster_addr":"172.21.15.32:6827/1551241759","heartbeat_back_addr":"172.21.15.32:6831/1551241759","heartbeat_front_addr":"172.21.15.32:6829/1551241759","state":["exists","up"]},{"osd":2,"uuid":"57f2e6de-a201-45c2-8981-8e56a4875ebd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6808","nonce":3806708944},{"type":"v1","addr":"172.21.15.32:6809","nonce":3806708944}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6810","nonce":3806708944},{"type":"v1","addr":"172.21.15.32:6811","nonce":3806708944}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6814","nonce":3806708944},{"type":"v1","addr":"172.21.15.32:6815","nonce":3806708944}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6812","nonce":3806708944},{"type":"v1","addr":"172.21.15.32:6813","nonce":3806708944}]},"public_addr":"172.21.15.32:6809/3806708944","cluster_addr":"172.21.15.32:6811/3806708944","heartbeat_back_addr":"172.21.15.32:6815/3806708944","heartbeat_front_addr":"172.21.15.32:6813/3806708944","state":["exists","up"]},{"osd":3,"uuid":"6439a280-da45-4148-8130-f16323935ff3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6800","nonce":9088303},{"type":"v1","addr":"172.21.15.32:6801","nonce":9088303}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6802","nonce":9088303},{"type":"v1","addr":"172.21.15.32:6803","nonce":9088303}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6806","nonce":9088303},{"type":"v1","addr":"172.21.15.32:6807","nonce":9088303}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6804","nonce":9088303},{"type":"v1","addr":"172.21.15.32:6805","nonce":9088303}]},"public_addr":"172.21.15.32:6801/9088303","cluster_addr":"172.21.15.32:6803/9088303","heartbeat_back_addr":"172.21.15.32:6807/9088303","heartbeat_front_addr":"172.21.15.32:6805/9088303","state":["exists","up"]},{"osd":4,"uuid":"4f902eff-1e18-418d-82c9-2151c69524fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6802","nonce":864752275},{"type":"v1","addr":"172.21.15.118:6803","nonce":864752275}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6804","nonce":864752275},{"type":"v1","addr":"172.21.15.118:6805","nonce":864752275}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6808","nonce":864752275},{"type":"v1","addr":"172.21.15.118:6809","nonce":864752275}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6806","nonce":864752275},{"type":"v1","addr":"172.21.15.118:6807","nonce":864752275}]},"public_addr":"172.21.15.118:6803/864752275","cluster_addr":"172.21.15.118:6805/864752275","heartbeat_back_addr":"172.21.15.118:6809/864752275","heartbeat_front_addr":"172.21.15.118:6807/864752275","state":["exists","up"]},{"osd":5,"uuid":"bfd9feac-dd06-458c-91b9-af7f175f612c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6810","nonce":2724891770},{"type":"v1","addr":"172.21.15.118:6811","nonce":2724891770}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6812","nonce":2724891770},{"type":"v1","addr":"172.21.15.118:6813","nonce":2724891770}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6816","nonce":2724891770},{"type":"v1","addr":"172.21.15.118:6817","nonce":2724891770}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6814","nonce":2724891770},{"type":"v1","addr":"172.21.15.118:6815","nonce":2724891770}]},"public_addr":"172.21.15.118:6811/2724891770","cluster_addr":"172.21.15.118:6813/2724891770","heartbeat_back_addr":"172.21.15.118:6817/2724891770","heartbeat_front_addr":"172.21.15.118:6815/2724891770","state":["exists","up"]},{"osd":6,"uuid":"5ed5a279-9d8e-416d-bbc7-7edd3e6d5705","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6825","nonce":801947386},{"type":"v1","addr":"172.21.15.118:6827","nonce":801947386}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6828","nonce":801947386},{"type":"v1","addr":"172.21.15.118:6829","nonce":801947386}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6832","nonce":801947386},{"type":"v1","addr":"172.21.15.118:6833","nonce":801947386}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6830","nonce":801947386},{"type":"v1","addr":"172.21.15.118:6831","nonce":801947386}]},"public_addr":"172.21.15.118:6827/801947386","cluster_addr":"172.21.15.118:6829/801947386","heartbeat_back_addr":"172.21.15.118:6833/801947386","heartbeat_front_addr":"172.21.15.118:6831/801947386","state":["exists","up"]},{"osd":7,"uuid":"f2086c47-875e-4451-a436-2008d1ea16af","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6818","nonce":653000048},{"type":"v1","addr":"172.21.15.118:6819","nonce":653000048}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6820","nonce":653000048},{"type":"v1","addr":"172.21.15.118:6821","nonce":653000048}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6824","nonce":653000048},{"type":"v1","addr":"172.21.15.118:6826","nonce":653000048}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6822","nonce":653000048},{"type":"v1","addr":"172.21.15.118:6823","nonce":653000048}]},"public_addr":"172.21.15.118:6819/653000048","cluster_addr":"172.21.15.118:6821/653000048","heartbeat_back_addr":"172.21.15.118:6826/653000048","heartbeat_front_addr":"172.21.15.118:6823/653000048","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-03T20:47:25.491994+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-03T20:47:25.587358+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-03T20:47:25.454175+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-03T20:47:25.377734+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-03T20:47:25.412351+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-03T20:47:25.499815+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-03T20:47:25.575044+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-03T20:47:25.552383+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-08-03T20:48:27.624 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:27.623+0000 7fc469e00640 1 -- 172.21.15.32:0/1013864147 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fc444077040 msgr2=0x7fc444079550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:48:27.625 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:27.623+0000 7fc469e00640 1 --2- 172.21.15.32:0/1013864147 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fc444077040 0x7fc444079550 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fc450033000 tx=0x7fc450004730 comp rx=0 tx=0).stop 2024-08-03T20:48:27.625 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:27.623+0000 7fc469e00640 1 -- 172.21.15.32:0/1013864147 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc46406e930 msgr2=0x7fc4641f5da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:48:27.625 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:27.623+0000 7fc469e00640 1 --2- 172.21.15.32:0/1013864147 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc46406e930 0x7fc4641f5da0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7fc45400d740 tx=0x7fc45400dc00 comp rx=0 tx=0).stop 2024-08-03T20:48:27.625 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:27.623+0000 7fc469e00640 1 -- 172.21.15.32:0/1013864147 shutdown_connections 2024-08-03T20:48:27.625 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:27.623+0000 7fc469e00640 1 --2- 172.21.15.32:0/1013864147 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fc444077040 0x7fc444079550 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:27.625 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:27.623+0000 7fc469e00640 1 --2- 172.21.15.32:0/1013864147 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc464073e60 0x7fc4640a94b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:27.625 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:27.623+0000 7fc469e00640 1 --2- 172.21.15.32:0/1013864147 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fc4640747b0 0x7fc46415df50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:27.625 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:27.623+0000 7fc469e00640 1 --2- 172.21.15.32:0/1013864147 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc46406e930 0x7fc4641f5da0 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:27.625 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:27.623+0000 7fc469e00640 1 -- 172.21.15.32:0/1013864147 >> 172.21.15.32:0/1013864147 conn(0x7fc4640b1bd0 msgr2=0x7fc4640b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:48:27.625 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:27.623+0000 7fc469e00640 1 -- 172.21.15.32:0/1013864147 shutdown_connections 2024-08-03T20:48:27.625 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:27.623+0000 7fc469e00640 1 -- 172.21.15.32:0/1013864147 wait complete. 2024-08-03T20:48:27.650 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-08-03T20:48:27.879 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:27.877+0000 7fd4be000640 1 -- 172.21.15.32:0/1767301206 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd4b80ce380 msgr2=0x7fd4b8161fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:48:27.879 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:27.877+0000 7fd4be000640 1 --2- 172.21.15.32:0/1767301206 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd4b80ce380 0x7fd4b8161fb0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7fd4b0005fd0 tx=0x7fd4b002aa00 comp rx=0 tx=0).stop 2024-08-03T20:48:27.879 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:27.877+0000 7fd4be000640 1 -- 172.21.15.32:0/1767301206 shutdown_connections 2024-08-03T20:48:27.879 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:27.877+0000 7fd4be000640 1 --2- 172.21.15.32:0/1767301206 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fd4b80af140 0x7fd4b80cde40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:27.879 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:27.877+0000 7fd4be000640 1 --2- 172.21.15.32:0/1767301206 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd4b80ce380 0x7fd4b8161fb0 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:27.879 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:27.877+0000 7fd4be000640 1 -- 172.21.15.32:0/1767301206 >> 172.21.15.32:0/1767301206 conn(0x7fd4b80b2120 msgr2=0x7fd4b80cfd60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:48:27.880 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:27.877+0000 7fd4be000640 1 -- 172.21.15.32:0/1767301206 shutdown_connections 2024-08-03T20:48:27.880 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:27.877+0000 7fd4be000640 1 -- 172.21.15.32:0/1767301206 wait complete. 2024-08-03T20:48:27.881 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:27.879+0000 7fd4be000640 1 Processor -- start 2024-08-03T20:48:27.881 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:27.880+0000 7fd4be000640 1 -- start start 2024-08-03T20:48:27.884 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:27.881+0000 7fd4be000640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fd4b80af140 0x7fd4b814ae40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:48:27.884 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:27.881+0000 7fd4be000640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fd4b80ca8c0 0x7fd4b8157b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:48:27.884 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:27.881+0000 7fd4be000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd4b80ce380 0x7fd4b8158060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:48:27.884 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:27.881+0000 7fd4be000640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fd4b8162880 con 0x7fd4b80ce380 2024-08-03T20:48:27.884 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:27.881+0000 7fd4be000640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7fd4b8162700 con 0x7fd4b80af140 2024-08-03T20:48:27.884 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:27.881+0000 7fd4be000640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7fd4b8162a00 con 0x7fd4b80ca8c0 2024-08-03T20:48:27.884 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:27.881+0000 7fd4bd600640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd4b80ce380 0x7fd4b8158060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:48:27.884 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:27.881+0000 7fd4bcc00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fd4b80af140 0x7fd4b814ae40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:48:27.885 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:27.881+0000 7fd4bd600640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd4b80ce380 0x7fd4b8158060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:48238/0 (socket says 172.21.15.32:48238) 2024-08-03T20:48:27.885 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:27.881+0000 7fd4bd600640 1 -- 172.21.15.32:0/3430633626 learned_addr learned my addr 172.21.15.32:0/3430633626 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:48:27.885 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:27.881+0000 7fd4b7e00640 1 --2- 172.21.15.32:0/3430633626 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fd4b80ca8c0 0x7fd4b8157b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:48:27.885 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:27.882+0000 7fd4bd600640 1 -- 172.21.15.32:0/3430633626 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fd4b80ca8c0 msgr2=0x7fd4b8157b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:48:27.885 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:27.882+0000 7fd4bd600640 1 --2- 172.21.15.32:0/3430633626 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fd4b80ca8c0 0x7fd4b8157b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:27.885 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:27.882+0000 7fd4bd600640 1 -- 172.21.15.32:0/3430633626 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fd4b80af140 msgr2=0x7fd4b814ae40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:48:27.885 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:27.882+0000 7fd4bd600640 1 --2- 172.21.15.32:0/3430633626 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fd4b80af140 0x7fd4b814ae40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:27.885 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:27.882+0000 7fd4bd600640 1 -- 172.21.15.32:0/3430633626 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd4b814bf00 con 0x7fd4b80ce380 2024-08-03T20:48:27.885 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:27.882+0000 7fd4bd600640 1 --2- 172.21.15.32:0/3430633626 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd4b80ce380 0x7fd4b8158060 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7fd4b0006010 tx=0x7fd4b0003f10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:48:27.885 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:27.882+0000 7fd4b4c00640 1 -- 172.21.15.32:0/3430633626 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd4b0046070 con 0x7fd4b80ce380 2024-08-03T20:48:27.885 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:27.882+0000 7fd4b4c00640 1 -- 172.21.15.32:0/3430633626 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fd4b0004540 con 0x7fd4b80ce380 2024-08-03T20:48:27.885 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:27.883+0000 7fd4be000640 1 -- 172.21.15.32:0/3430633626 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd4b815bb40 con 0x7fd4b80ce380 2024-08-03T20:48:27.885 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:27.883+0000 7fd4b4c00640 1 -- 172.21.15.32:0/3430633626 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd4b0033070 con 0x7fd4b80ce380 2024-08-03T20:48:27.885 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:27.883+0000 7fd4be000640 1 -- 172.21.15.32:0/3430633626 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd4b81494d0 con 0x7fd4b80ce380 2024-08-03T20:48:27.886 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:27.884+0000 7fd4be000640 1 -- 172.21.15.32:0/3430633626 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd4b8149730 con 0x7fd4b80ce380 2024-08-03T20:48:27.886 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:27.885+0000 7fd4b4c00640 1 -- 172.21.15.32:0/3430633626 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7fd4b0049070 con 0x7fd4b80ce380 2024-08-03T20:48:27.887 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:27.886+0000 7fd4b4c00640 1 --2- 172.21.15.32:0/3430633626 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fd498077010 0x7fd498079520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:48:27.887 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:27.886+0000 7fd4b4c00640 1 -- 172.21.15.32:0/3430633626 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(31..31 src has 1..31) ==== 5709+0+0 (secure 0 0 0) 0x7fd4b00b99f0 con 0x7fd4b80ce380 2024-08-03T20:48:27.888 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:27.886+0000 7fd4bcc00640 1 --2- 172.21.15.32:0/3430633626 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fd498077010 0x7fd498079520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:48:27.888 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:27.887+0000 7fd4bcc00640 1 --2- 172.21.15.32:0/3430633626 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fd498077010 0x7fd498079520 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fd4ac006fd0 tx=0x7fd4ac009060 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:48:27.891 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:27.890+0000 7fd4b4c00640 1 -- 172.21.15.32:0/3430633626 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7fd4b814bf00 con 0x7fd4b80ce380 2024-08-03T20:48:28.096 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:28.094+0000 7fd4be000640 1 -- 172.21.15.32:0/3430633626 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fd4b809db10 con 0x7fd4b80ce380 2024-08-03T20:48:28.097 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:28.095+0000 7fd4b4c00640 1 -- 172.21.15.32:0/3430633626 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v31) ==== 74+0+16622 (secure 0 0 0) 0x7fd4b003e070 con 0x7fd4b80ce380 2024-08-03T20:48:28.097 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-03T20:48:28.097 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":31,"fsid":"40ab141a-01d7-49b9-ae3c-2edec9876e95","created":"2024-08-03T20:47:14.957362+0000","modified":"2024-08-03T20:48:26.652562+0000","last_up_change":"2024-08-03T20:47:28.874451+0000","last_in_change":"2024-08-03T20:47:20.079442+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-08-03T20:47:28.261300+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_metadata","create_time":"2024-08-03T20:48:24.969840+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"31","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data","create_time":"2024-08-03T20:48:25.603595+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"31","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5a83e759-9035-438b-889a-0a6af1f0a0d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6816","nonce":3066156116},{"type":"v1","addr":"172.21.15.32:6817","nonce":3066156116}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6818","nonce":3066156116},{"type":"v1","addr":"172.21.15.32:6819","nonce":3066156116}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6822","nonce":3066156116},{"type":"v1","addr":"172.21.15.32:6823","nonce":3066156116}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6820","nonce":3066156116},{"type":"v1","addr":"172.21.15.32:6821","nonce":3066156116}]},"public_addr":"172.21.15.32:6817/3066156116","cluster_addr":"172.21.15.32:6819/3066156116","heartbeat_back_addr":"172.21.15.32:6823/3066156116","heartbeat_front_addr":"172.21.15.32:6821/3066156116","state":["exists","up"]},{"osd":1,"uuid":"d7c5c210-da4f-49f9-b9e7-1dfb02daa4bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6824","nonce":1551241759},{"type":"v1","addr":"172.21.15.32:6825","nonce":1551241759}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6826","nonce":1551241759},{"type":"v1","addr":"172.21.15.32:6827","nonce":1551241759}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6830","nonce":1551241759},{"type":"v1","addr":"172.21.15.32:6831","nonce":1551241759}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6828","nonce":1551241759},{"type":"v1","addr":"172.21.15.32:6829","nonce":1551241759}]},"public_addr":"172.21.15.32:6825/1551241759","cluster_addr":"172.21.15.32:6827/1551241759","heartbeat_back_addr":"172.21.15.32:6831/1551241759","heartbeat_front_addr":"172.21.15.32:6829/1551241759","state":["exists","up"]},{"osd":2,"uuid":"57f2e6de-a201-45c2-8981-8e56a4875ebd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6808","nonce":3806708944},{"type":"v1","addr":"172.21.15.32:6809","nonce":3806708944}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6810","nonce":3806708944},{"type":"v1","addr":"172.21.15.32:6811","nonce":3806708944}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6814","nonce":3806708944},{"type":"v1","addr":"172.21.15.32:6815","nonce":3806708944}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6812","nonce":3806708944},{"type":"v1","addr":"172.21.15.32:6813","nonce":3806708944}]},"public_addr":"172.21.15.32:6809/3806708944","cluster_addr":"172.21.15.32:6811/3806708944","heartbeat_back_addr":"172.21.15.32:6815/3806708944","heartbeat_front_addr":"172.21.15.32:6813/3806708944","state":["exists","up"]},{"osd":3,"uuid":"6439a280-da45-4148-8130-f16323935ff3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6800","nonce":9088303},{"type":"v1","addr":"172.21.15.32:6801","nonce":9088303}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6802","nonce":9088303},{"type":"v1","addr":"172.21.15.32:6803","nonce":9088303}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6806","nonce":9088303},{"type":"v1","addr":"172.21.15.32:6807","nonce":9088303}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6804","nonce":9088303},{"type":"v1","addr":"172.21.15.32:6805","nonce":9088303}]},"public_addr":"172.21.15.32:6801/9088303","cluster_addr":"172.21.15.32:6803/9088303","heartbeat_back_addr":"172.21.15.32:6807/9088303","heartbeat_front_addr":"172.21.15.32:6805/9088303","state":["exists","up"]},{"osd":4,"uuid":"4f902eff-1e18-418d-82c9-2151c69524fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6802","nonce":864752275},{"type":"v1","addr":"172.21.15.118:6803","nonce":864752275}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6804","nonce":864752275},{"type":"v1","addr":"172.21.15.118:6805","nonce":864752275}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6808","nonce":864752275},{"type":"v1","addr":"172.21.15.118:6809","nonce":864752275}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6806","nonce":864752275},{"type":"v1","addr":"172.21.15.118:6807","nonce":864752275}]},"public_addr":"172.21.15.118:6803/864752275","cluster_addr":"172.21.15.118:6805/864752275","heartbeat_back_addr":"172.21.15.118:6809/864752275","heartbeat_front_addr":"172.21.15.118:6807/864752275","state":["exists","up"]},{"osd":5,"uuid":"bfd9feac-dd06-458c-91b9-af7f175f612c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6810","nonce":2724891770},{"type":"v1","addr":"172.21.15.118:6811","nonce":2724891770}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6812","nonce":2724891770},{"type":"v1","addr":"172.21.15.118:6813","nonce":2724891770}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6816","nonce":2724891770},{"type":"v1","addr":"172.21.15.118:6817","nonce":2724891770}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6814","nonce":2724891770},{"type":"v1","addr":"172.21.15.118:6815","nonce":2724891770}]},"public_addr":"172.21.15.118:6811/2724891770","cluster_addr":"172.21.15.118:6813/2724891770","heartbeat_back_addr":"172.21.15.118:6817/2724891770","heartbeat_front_addr":"172.21.15.118:6815/2724891770","state":["exists","up"]},{"osd":6,"uuid":"5ed5a279-9d8e-416d-bbc7-7edd3e6d5705","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6825","nonce":801947386},{"type":"v1","addr":"172.21.15.118:6827","nonce":801947386}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6828","nonce":801947386},{"type":"v1","addr":"172.21.15.118:6829","nonce":801947386}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6832","nonce":801947386},{"type":"v1","addr":"172.21.15.118:6833","nonce":801947386}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6830","nonce":801947386},{"type":"v1","addr":"172.21.15.118:6831","nonce":801947386}]},"public_addr":"172.21.15.118:6827/801947386","cluster_addr":"172.21.15.118:6829/801947386","heartbeat_back_addr":"172.21.15.118:6833/801947386","heartbeat_front_addr":"172.21.15.118:6831/801947386","state":["exists","up"]},{"osd":7,"uuid":"f2086c47-875e-4451-a436-2008d1ea16af","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6818","nonce":653000048},{"type":"v1","addr":"172.21.15.118:6819","nonce":653000048}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6820","nonce":653000048},{"type":"v1","addr":"172.21.15.118:6821","nonce":653000048}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6824","nonce":653000048},{"type":"v1","addr":"172.21.15.118:6826","nonce":653000048}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6822","nonce":653000048},{"type":"v1","addr":"172.21.15.118:6823","nonce":653000048}]},"public_addr":"172.21.15.118:6819/653000048","cluster_addr":"172.21.15.118:6821/653000048","heartbeat_back_addr":"172.21.15.118:6826/653000048","heartbeat_front_addr":"172.21.15.118:6823/653000048","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-03T20:47:25.491994+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-03T20:47:25.587358+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-03T20:47:25.454175+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-03T20:47:25.377734+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-03T20:47:25.412351+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-03T20:47:25.499815+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-03T20:47:25.575044+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-03T20:47:25.552383+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-08-03T20:48:28.100 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:28.098+0000 7fd4be000640 1 -- 172.21.15.32:0/3430633626 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fd498077010 msgr2=0x7fd498079520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:48:28.100 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:28.098+0000 7fd4be000640 1 --2- 172.21.15.32:0/3430633626 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fd498077010 0x7fd498079520 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fd4ac006fd0 tx=0x7fd4ac009060 comp rx=0 tx=0).stop 2024-08-03T20:48:28.100 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:28.098+0000 7fd4be000640 1 -- 172.21.15.32:0/3430633626 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd4b80ce380 msgr2=0x7fd4b8158060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:48:28.100 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:28.098+0000 7fd4be000640 1 --2- 172.21.15.32:0/3430633626 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd4b80ce380 0x7fd4b8158060 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7fd4b0006010 tx=0x7fd4b0003f10 comp rx=0 tx=0).stop 2024-08-03T20:48:28.100 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:28.098+0000 7fd4be000640 1 -- 172.21.15.32:0/3430633626 shutdown_connections 2024-08-03T20:48:28.100 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:28.098+0000 7fd4be000640 1 --2- 172.21.15.32:0/3430633626 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fd498077010 0x7fd498079520 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:28.100 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:28.098+0000 7fd4be000640 1 --2- 172.21.15.32:0/3430633626 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd4b80ce380 0x7fd4b8158060 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:28.101 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:28.098+0000 7fd4be000640 1 --2- 172.21.15.32:0/3430633626 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fd4b80ca8c0 0x7fd4b8157b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:28.101 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:28.098+0000 7fd4be000640 1 --2- 172.21.15.32:0/3430633626 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fd4b80af140 0x7fd4b814ae40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:28.101 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:28.098+0000 7fd4be000640 1 -- 172.21.15.32:0/3430633626 >> 172.21.15.32:0/3430633626 conn(0x7fd4b80b2120 msgr2=0x7fd4b80af8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:48:28.101 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:28.098+0000 7fd4be000640 1 -- 172.21.15.32:0/3430633626 shutdown_connections 2024-08-03T20:48:28.101 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:28.098+0000 7fd4be000640 1 -- 172.21.15.32:0/3430633626 wait complete. 2024-08-03T20:48:28.122 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-08-03T20:48:28.356 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:28.355+0000 7f136e600640 1 -- 172.21.15.32:0/2461710300 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1368145c50 msgr2=0x7f136806e950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:48:28.356 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:28.355+0000 7f136e600640 1 --2- 172.21.15.32:0/2461710300 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1368145c50 0x7f136806e950 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f13680cca80 tx=0x7f135802a4d0 comp rx=0 tx=0).stop 2024-08-03T20:48:28.357 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:28.355+0000 7f136e600640 1 -- 172.21.15.32:0/2461710300 shutdown_connections 2024-08-03T20:48:28.357 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:28.355+0000 7f136e600640 1 --2- 172.21.15.32:0/2461710300 >> v2:172.21.15.118:3300/0 conn(0x7f13680ced30 0x7f13680cf140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:28.357 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:28.355+0000 7f136e600640 1 --2- 172.21.15.32:0/2461710300 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1368145c50 0x7f136806e950 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:28.357 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:28.355+0000 7f136e600640 1 -- 172.21.15.32:0/2461710300 >> 172.21.15.32:0/2461710300 conn(0x7f13680b1bd0 msgr2=0x7f13680b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:48:28.357 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:28.355+0000 7f136e600640 1 -- 172.21.15.32:0/2461710300 shutdown_connections 2024-08-03T20:48:28.357 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:28.355+0000 7f136e600640 1 -- 172.21.15.32:0/2461710300 wait complete. 2024-08-03T20:48:28.359 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:28.357+0000 7f136e600640 1 Processor -- start 2024-08-03T20:48:28.359 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:28.357+0000 7f136e600640 1 -- start start 2024-08-03T20:48:28.361 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:28.359+0000 7f136e600640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f13680cccf0 0x7f1368158df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:48:28.361 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:28.359+0000 7f136e600640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f13680ced30 0x7f136813fdf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:48:28.361 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:28.359+0000 7f136e600640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f1368145c50 0x7f1368140330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:48:28.361 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:28.359+0000 7f136e600640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f13680b40b0 con 0x7f13680ced30 2024-08-03T20:48:28.361 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:28.359+0000 7f136e600640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f13680b3f30 con 0x7f13680cccf0 2024-08-03T20:48:28.361 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:28.359+0000 7f136e600640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7f13680b4230 con 0x7f1368145c50 2024-08-03T20:48:28.361 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:28.359+0000 7f1366a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f13680ced30 0x7f136813fdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:48:28.361 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:28.359+0000 7f1366a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f13680ced30 0x7f136813fdf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:48266/0 (socket says 172.21.15.32:48266) 2024-08-03T20:48:28.361 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:28.359+0000 7f1366a00640 1 -- 172.21.15.32:0/2347402885 learned_addr learned my addr 172.21.15.32:0/2347402885 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:48:28.362 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:28.359+0000 7f1367400640 1 --2- 172.21.15.32:0/2347402885 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f13680cccf0 0x7f1368158df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:48:28.362 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:28.359+0000 7f1367e00640 1 --2- 172.21.15.32:0/2347402885 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f1368145c50 0x7f1368140330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:48:28.362 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:28.360+0000 7f1366a00640 1 -- 172.21.15.32:0/2347402885 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f1368145c50 msgr2=0x7f1368140330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:48:28.362 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:28.360+0000 7f1366a00640 1 --2- 172.21.15.32:0/2347402885 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f1368145c50 0x7f1368140330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:28.362 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:28.360+0000 7f1366a00640 1 -- 172.21.15.32:0/2347402885 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f13680cccf0 msgr2=0x7f1368158df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:48:28.362 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:28.360+0000 7f1366a00640 1 --2- 172.21.15.32:0/2347402885 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f13680cccf0 0x7f1368158df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:28.362 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:28.360+0000 7f1366a00640 1 -- 172.21.15.32:0/2347402885 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f136813d980 con 0x7f13680ced30 2024-08-03T20:48:28.362 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:28.360+0000 7f1366a00640 1 --2- 172.21.15.32:0/2347402885 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f13680ced30 0x7f136813fdf0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f135400b3c0 tx=0x7f135400b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:48:28.363 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:28.360+0000 7f135f400640 1 -- 172.21.15.32:0/2347402885 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1354011830 con 0x7f13680ced30 2024-08-03T20:48:28.363 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:28.360+0000 7f135f400640 1 -- 172.21.15.32:0/2347402885 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f1354016070 con 0x7f13680ced30 2024-08-03T20:48:28.363 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:28.360+0000 7f135f400640 1 -- 172.21.15.32:0/2347402885 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1354011d80 con 0x7f13680ced30 2024-08-03T20:48:28.363 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:28.360+0000 7f136e600640 1 -- 172.21.15.32:0/2347402885 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f136813dc20 con 0x7f13680ced30 2024-08-03T20:48:28.363 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:28.360+0000 7f136e600640 1 -- 172.21.15.32:0/2347402885 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f136813e3e0 con 0x7f13680ced30 2024-08-03T20:48:28.363 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:28.361+0000 7f136e600640 1 -- 172.21.15.32:0/2347402885 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f13680b47b0 con 0x7f13680ced30 2024-08-03T20:48:28.364 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:28.363+0000 7f135f400640 1 -- 172.21.15.32:0/2347402885 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f135401b460 con 0x7f13680ced30 2024-08-03T20:48:28.365 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:28.364+0000 7f135f400640 1 --2- 172.21.15.32:0/2347402885 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f1348077010 0x7f1348079520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:48:28.365 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:28.364+0000 7f135f400640 1 -- 172.21.15.32:0/2347402885 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(32..32 src has 1..32) ==== 5709+0+0 (secure 0 0 0) 0x7f13540985c0 con 0x7f13680ced30 2024-08-03T20:48:28.365 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:28.364+0000 7f1367400640 1 --2- 172.21.15.32:0/2347402885 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f1348077010 0x7f1348079520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:48:28.366 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:28.365+0000 7f1367400640 1 --2- 172.21.15.32:0/2347402885 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f1348077010 0x7f1348079520 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f1358033000 tx=0x7f1358004730 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:48:28.369 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:28.368+0000 7f135f400640 1 -- 172.21.15.32:0/2347402885 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7f1354061160 con 0x7f13680ced30 2024-08-03T20:48:28.615 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:28.614+0000 7f136e600640 1 -- 172.21.15.32:0/2347402885 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) -- 0x7f13680d0a10 con 0x7f13680ced30 2024-08-03T20:48:29.143 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:29.141+0000 7f135f400640 1 -- 172.21.15.32:0/2347402885 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v16) ==== 118+0+0 (secure 0 0 0) 0x7f13540656b0 con 0x7f13680ced30 2024-08-03T20:48:29.146 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:29.144+0000 7f136e600640 1 -- 172.21.15.32:0/2347402885 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f1348077010 msgr2=0x7f1348079520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:48:29.146 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:29.144+0000 7f136e600640 1 --2- 172.21.15.32:0/2347402885 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f1348077010 0x7f1348079520 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f1358033000 tx=0x7f1358004730 comp rx=0 tx=0).stop 2024-08-03T20:48:29.146 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:29.144+0000 7f136e600640 1 -- 172.21.15.32:0/2347402885 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f13680ced30 msgr2=0x7f136813fdf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:48:29.146 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:29.144+0000 7f136e600640 1 --2- 172.21.15.32:0/2347402885 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f13680ced30 0x7f136813fdf0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f135400b3c0 tx=0x7f135400b880 comp rx=0 tx=0).stop 2024-08-03T20:48:29.146 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:29.144+0000 7f136e600640 1 -- 172.21.15.32:0/2347402885 shutdown_connections 2024-08-03T20:48:29.146 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:29.144+0000 7f136e600640 1 --2- 172.21.15.32:0/2347402885 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f1348077010 0x7f1348079520 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:29.146 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:29.144+0000 7f136e600640 1 --2- 172.21.15.32:0/2347402885 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f1368145c50 0x7f1368140330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:29.146 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:29.144+0000 7f136e600640 1 --2- 172.21.15.32:0/2347402885 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f13680ced30 0x7f136813fdf0 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:29.147 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:29.144+0000 7f136e600640 1 --2- 172.21.15.32:0/2347402885 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f13680cccf0 0x7f1368158df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:29.147 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:29.144+0000 7f136e600640 1 -- 172.21.15.32:0/2347402885 >> 172.21.15.32:0/2347402885 conn(0x7f13680b1bd0 msgr2=0x7f13680b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:48:29.147 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:29.144+0000 7f136e600640 1 -- 172.21.15.32:0/2347402885 shutdown_connections 2024-08-03T20:48:29.147 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:29.144+0000 7f136e600640 1 -- 172.21.15.32:0/2347402885 wait complete. 2024-08-03T20:48:29.170 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-08-03T20:48:29.171 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-08-03T20:48:29.405 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:29.403+0000 7f73ebe00640 1 -- 172.21.15.32:0/278647953 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f73cc003a00 msgr2=0x7f73cc003e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:48:29.405 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:29.403+0000 7f73ebe00640 1 --2- 172.21.15.32:0/278647953 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f73cc003a00 0x7f73cc003e70 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f73d8009b50 tx=0x7f73cc005830 comp rx=0 tx=0).stop 2024-08-03T20:48:29.405 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:29.403+0000 7f73e9e00640 1 -- 172.21.15.32:0/278647953 reap_dead start 2024-08-03T20:48:29.405 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:29.403+0000 7f73ebe00640 1 -- 172.21.15.32:0/278647953 shutdown_connections 2024-08-03T20:48:29.406 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:29.403+0000 7f73ebe00640 1 -- 172.21.15.32:0/278647953 >> 172.21.15.32:0/278647953 conn(0x7f73e40b1bd0 msgr2=0x7f73e40b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:48:29.406 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:29.404+0000 7f73ebe00640 1 -- 172.21.15.32:0/278647953 shutdown_connections 2024-08-03T20:48:29.406 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:29.404+0000 7f73ebe00640 1 -- 172.21.15.32:0/278647953 wait complete. 2024-08-03T20:48:29.407 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:29.406+0000 7f73ebe00640 1 Processor -- start 2024-08-03T20:48:29.407 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:29.406+0000 7f73ebe00640 1 -- start start 2024-08-03T20:48:29.410 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:29.407+0000 7f73ebe00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f73cc003a00 0x7f73e415b830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:48:29.410 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:29.407+0000 7f73ebe00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f73e41e5d90 0x7f73e4159210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:48:29.410 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:29.407+0000 7f73ebe00640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f73e41580d0 0x7f73e41f8f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:48:29.410 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:29.408+0000 7f73ebe00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f73e40b5420 con 0x7f73e41e5d90 2024-08-03T20:48:29.410 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:29.408+0000 7f73ebe00640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f73e40b52a0 con 0x7f73cc003a00 2024-08-03T20:48:29.410 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:29.408+0000 7f73ebe00640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7f73e4145630 con 0x7f73e41580d0 2024-08-03T20:48:29.411 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:29.408+0000 7f73e8a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f73e41e5d90 0x7f73e4159210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:48:29.411 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:29.408+0000 7f73e8a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f73e41e5d90 0x7f73e4159210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:48290/0 (socket says 172.21.15.32:48290) 2024-08-03T20:48:29.411 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:29.408+0000 7f73e8a00640 1 -- 172.21.15.32:0/3805168253 learned_addr learned my addr 172.21.15.32:0/3805168253 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:48:29.411 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:29.408+0000 7f73e9400640 1 --2- 172.21.15.32:0/3805168253 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f73cc003a00 0x7f73e415b830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:48:29.411 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:29.408+0000 7f73e9e00640 1 --2- 172.21.15.32:0/3805168253 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f73e41580d0 0x7f73e41f8f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:48:29.411 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:29.408+0000 7f73e8a00640 1 -- 172.21.15.32:0/3805168253 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f73e41580d0 msgr2=0x7f73e41f8f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:48:29.411 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:29.408+0000 7f73e8a00640 1 --2- 172.21.15.32:0/3805168253 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f73e41580d0 0x7f73e41f8f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:29.411 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:29.408+0000 7f73e8a00640 1 -- 172.21.15.32:0/3805168253 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f73cc003a00 msgr2=0x7f73e415b830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:48:29.411 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:29.408+0000 7f73e8a00640 1 --2- 172.21.15.32:0/3805168253 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f73cc003a00 0x7f73e415b830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:29.411 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:29.408+0000 7f73e8a00640 1 -- 172.21.15.32:0/3805168253 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f73e415c080 con 0x7f73e41e5d90 2024-08-03T20:48:29.411 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:29.409+0000 7f73e8a00640 1 --2- 172.21.15.32:0/3805168253 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f73e41e5d90 0x7f73e4159210 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f73d0006630 tx=0x7f73d00039c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:48:29.412 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:29.409+0000 7f73e1600640 1 -- 172.21.15.32:0/3805168253 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73d0015070 con 0x7f73e41e5d90 2024-08-03T20:48:29.412 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:29.409+0000 7f73e1600640 1 -- 172.21.15.32:0/3805168253 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f73d0011900 con 0x7f73e41e5d90 2024-08-03T20:48:29.412 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:29.409+0000 7f73ebe00640 1 -- 172.21.15.32:0/3805168253 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f73e415c360 con 0x7f73e41e5d90 2024-08-03T20:48:29.412 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:29.409+0000 7f73e1600640 1 -- 172.21.15.32:0/3805168253 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73d0011d30 con 0x7f73e41e5d90 2024-08-03T20:48:29.413 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:29.409+0000 7f73ebe00640 1 -- 172.21.15.32:0/3805168253 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f73e41f8710 con 0x7f73e41e5d90 2024-08-03T20:48:29.413 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:29.410+0000 7f73ebe00640 1 -- 172.21.15.32:0/3805168253 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f73e415bdf0 con 0x7f73e41e5d90 2024-08-03T20:48:29.413 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:29.411+0000 7f73e1600640 1 -- 172.21.15.32:0/3805168253 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f73d0018070 con 0x7f73e41e5d90 2024-08-03T20:48:29.414 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:29.412+0000 7f73e1600640 1 --2- 172.21.15.32:0/3805168253 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f73c4077010 0x7f73c4079520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:48:29.414 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:29.412+0000 7f73e1600640 1 -- 172.21.15.32:0/3805168253 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(32..32 src has 1..32) ==== 5709+0+0 (secure 0 0 0) 0x7f73d00995f0 con 0x7f73e41e5d90 2024-08-03T20:48:29.414 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:29.413+0000 7f73e9400640 1 --2- 172.21.15.32:0/3805168253 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f73c4077010 0x7f73c4079520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:48:29.415 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:29.413+0000 7f73e9400640 1 --2- 172.21.15.32:0/3805168253 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f73c4077010 0x7f73c4079520 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f73d8009b50 tx=0x7f73d802d040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:48:29.418 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:29.417+0000 7f73e1600640 1 -- 172.21.15.32:0/3805168253 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7f73d0062110 con 0x7f73e41e5d90 2024-08-03T20:48:29.661 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:29.659+0000 7f73ebe00640 1 -- 172.21.15.32:0/3805168253 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) -- 0x7f73e4145bc0 con 0x7f73e41e5d90 2024-08-03T20:48:30.144 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:30.143+0000 7f73e1600640 1 -- 172.21.15.32:0/3805168253 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v17) ==== 122+0+0 (secure 0 0 0) 0x7f73d0066660 con 0x7f73e41e5d90 2024-08-03T20:48:30.147 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:30.145+0000 7f73ebe00640 1 -- 172.21.15.32:0/3805168253 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f73c4077010 msgr2=0x7f73c4079520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:48:30.147 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:30.145+0000 7f73ebe00640 1 --2- 172.21.15.32:0/3805168253 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f73c4077010 0x7f73c4079520 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f73d8009b50 tx=0x7f73d802d040 comp rx=0 tx=0).stop 2024-08-03T20:48:30.147 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:30.145+0000 7f73ebe00640 1 -- 172.21.15.32:0/3805168253 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f73e41e5d90 msgr2=0x7f73e4159210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:48:30.148 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:30.145+0000 7f73ebe00640 1 --2- 172.21.15.32:0/3805168253 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f73e41e5d90 0x7f73e4159210 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f73d0006630 tx=0x7f73d00039c0 comp rx=0 tx=0).stop 2024-08-03T20:48:30.148 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:30.145+0000 7f73ebe00640 1 -- 172.21.15.32:0/3805168253 shutdown_connections 2024-08-03T20:48:30.148 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:30.146+0000 7f73ebe00640 1 --2- 172.21.15.32:0/3805168253 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f73c4077010 0x7f73c4079520 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:30.148 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:30.146+0000 7f73ebe00640 1 --2- 172.21.15.32:0/3805168253 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f73e41580d0 0x7f73e41f8f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:30.148 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:30.146+0000 7f73ebe00640 1 --2- 172.21.15.32:0/3805168253 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f73e41e5d90 0x7f73e4159210 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:30.148 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:30.146+0000 7f73ebe00640 1 --2- 172.21.15.32:0/3805168253 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f73cc003a00 0x7f73e415b830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:30.148 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:30.146+0000 7f73ebe00640 1 -- 172.21.15.32:0/3805168253 >> 172.21.15.32:0/3805168253 conn(0x7f73e40b1bd0 msgr2=0x7f73e4072940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:48:30.148 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:30.146+0000 7f73ebe00640 1 -- 172.21.15.32:0/3805168253 shutdown_connections 2024-08-03T20:48:30.148 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:30.146+0000 7f73ebe00640 1 -- 172.21.15.32:0/3805168253 wait complete. 2024-08-03T20:48:30.172 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-08-03T20:48:30.173 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-08-03T20:48:30.402 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:30.400+0000 7fc5c7e00640 1 -- 172.21.15.32:0/3010739615 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc5c8161d20 msgr2=0x7fc5c80cedc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:48:30.402 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:30.400+0000 7fc5c7e00640 1 --2- 172.21.15.32:0/3010739615 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc5c8161d20 0x7fc5c80cedc0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7fc5c8073750 tx=0x7fc5b002a4d0 comp rx=0 tx=0).stop 2024-08-03T20:48:30.402 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:30.400+0000 7fc5c7e00640 1 -- 172.21.15.32:0/3010739615 shutdown_connections 2024-08-03T20:48:30.402 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:30.400+0000 7fc5c7e00640 1 --2- 172.21.15.32:0/3010739615 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fc5c80b52a0 0x7fc5c80b56b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:30.402 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:30.400+0000 7fc5c7e00640 1 --2- 172.21.15.32:0/3010739615 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc5c8161d20 0x7fc5c80cedc0 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:30.402 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:30.400+0000 7fc5c7e00640 1 --2- 172.21.15.32:0/3010739615 >> v2:172.21.15.118:3300/0 conn(0x7fc5c80b74c0 0x7fc5c80ce880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:30.402 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:30.400+0000 7fc5c7e00640 1 -- 172.21.15.32:0/3010739615 >> 172.21.15.32:0/3010739615 conn(0x7fc5c80d1e90 msgr2=0x7fc5c80d22a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:48:30.402 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:30.400+0000 7fc5c7e00640 1 -- 172.21.15.32:0/3010739615 shutdown_connections 2024-08-03T20:48:30.402 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:30.400+0000 7fc5c7e00640 1 -- 172.21.15.32:0/3010739615 wait complete. 2024-08-03T20:48:30.403 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:30.402+0000 7fc5c7e00640 1 Processor -- start 2024-08-03T20:48:30.403 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:30.402+0000 7fc5c7e00640 1 -- start start 2024-08-03T20:48:30.407 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:30.404+0000 7fc5c7e00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc5c80b52a0 0x7fc5c80b08b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:48:30.407 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:30.404+0000 7fc5c7e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc5c80b74c0 0x7fc5c80aec60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:48:30.407 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:30.404+0000 7fc5c7e00640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fc5c8161d20 0x7fc5c81f8ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:48:30.407 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:30.404+0000 7fc5c7e00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fc5c80b5f80 con 0x7fc5c80b74c0 2024-08-03T20:48:30.407 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:30.404+0000 7fc5c7e00640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7fc5c80b5e00 con 0x7fc5c80b52a0 2024-08-03T20:48:30.407 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:30.404+0000 7fc5c7e00640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7fc5c80b6100 con 0x7fc5c8161d20 2024-08-03T20:48:30.407 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:30.404+0000 7fc5bfe00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc5c80b74c0 0x7fc5c80aec60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:48:30.407 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:30.404+0000 7fc5bfe00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc5c80b74c0 0x7fc5c80aec60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:46428/0 (socket says 172.21.15.32:46428) 2024-08-03T20:48:30.407 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:30.404+0000 7fc5bfe00640 1 -- 172.21.15.32:0/1129075505 learned_addr learned my addr 172.21.15.32:0/1129075505 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:48:30.407 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:30.404+0000 7fc5c6a00640 1 --2- 172.21.15.32:0/1129075505 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc5c80b52a0 0x7fc5c80b08b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:48:30.407 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:30.404+0000 7fc5c7400640 1 --2- 172.21.15.32:0/1129075505 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fc5c8161d20 0x7fc5c81f8ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:48:30.407 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:30.404+0000 7fc5bfe00640 1 -- 172.21.15.32:0/1129075505 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fc5c8161d20 msgr2=0x7fc5c81f8ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:48:30.408 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:30.404+0000 7fc5bfe00640 1 --2- 172.21.15.32:0/1129075505 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fc5c8161d20 0x7fc5c81f8ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:30.408 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:30.404+0000 7fc5bfe00640 1 -- 172.21.15.32:0/1129075505 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc5c80b52a0 msgr2=0x7fc5c80b08b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:48:30.408 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:30.404+0000 7fc5bfe00640 1 --2- 172.21.15.32:0/1129075505 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc5c80b52a0 0x7fc5c80b08b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:30.408 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:30.404+0000 7fc5bfe00640 1 -- 172.21.15.32:0/1129075505 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc5c80b1290 con 0x7fc5c80b74c0 2024-08-03T20:48:30.408 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:30.405+0000 7fc5bfe00640 1 --2- 172.21.15.32:0/1129075505 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc5c80b74c0 0x7fc5c80aec60 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7fc5b0000c00 tx=0x7fc5b0003910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:48:30.408 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:30.405+0000 7fc5bea00640 1 -- 172.21.15.32:0/1129075505 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5b003e030 con 0x7fc5c80b74c0 2024-08-03T20:48:30.408 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:30.405+0000 7fc5bea00640 1 -- 172.21.15.32:0/1129075505 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fc5b0003e00 con 0x7fc5c80b74c0 2024-08-03T20:48:30.408 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:30.405+0000 7fc5c7e00640 1 -- 172.21.15.32:0/1129075505 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc5c80b1570 con 0x7fc5c80b74c0 2024-08-03T20:48:30.408 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:30.405+0000 7fc5bea00640 1 -- 172.21.15.32:0/1129075505 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5b0035040 con 0x7fc5c80b74c0 2024-08-03T20:48:30.408 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:30.405+0000 7fc5c7e00640 1 -- 172.21.15.32:0/1129075505 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc5c80b1b30 con 0x7fc5c80b74c0 2024-08-03T20:48:30.409 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:30.406+0000 7fc5c7e00640 1 -- 172.21.15.32:0/1129075505 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc5c80b1000 con 0x7fc5c80b74c0 2024-08-03T20:48:30.409 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:30.408+0000 7fc5bea00640 1 -- 172.21.15.32:0/1129075505 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7fc5b0041030 con 0x7fc5c80b74c0 2024-08-03T20:48:30.410 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:30.408+0000 7fc5bea00640 1 --2- 172.21.15.32:0/1129075505 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fc5a4077010 0x7fc5a4079520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:48:30.410 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:30.409+0000 7fc5bea00640 1 -- 172.21.15.32:0/1129075505 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(32..32 src has 1..32) ==== 5709+0+0 (secure 0 0 0) 0x7fc5b00b8ff0 con 0x7fc5c80b74c0 2024-08-03T20:48:30.410 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:30.409+0000 7fc5c6a00640 1 --2- 172.21.15.32:0/1129075505 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fc5a4077010 0x7fc5a4079520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:48:30.411 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:30.410+0000 7fc5c6a00640 1 --2- 172.21.15.32:0/1129075505 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fc5a4077010 0x7fc5a4079520 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fc5b800bb50 tx=0x7fc5b800b2e0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:48:30.415 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:30.413+0000 7fc5bea00640 1 -- 172.21.15.32:0/1129075505 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7fc5b0081b90 con 0x7fc5c80b74c0 2024-08-03T20:48:30.661 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:30.659+0000 7fc5c7e00640 1 -- 172.21.15.32:0/1129075505 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc5c80b56b0 con 0x7fc5c80b74c0 2024-08-03T20:48:30.662 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:30.660+0000 7fc5bea00640 1 -- 172.21.15.32:0/1129075505 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) ==== 94+0+5229 (secure 0 0 0) 0x7fc5b008a420 con 0x7fc5c80b74c0 2024-08-03T20:48:30.662 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-03T20:48:30.662 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":17,"btime":"2024-08-03T20:48:30:142972+0000","default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4643,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6836/4002526832","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6834","nonce":4002526832},{"type":"v1","addr":"172.21.15.118:6836","nonce":4002526832}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":4646,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6837/3036256187","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6835","nonce":3036256187},{"type":"v1","addr":"172.21.15.118:6837","nonce":3036256187}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":4652,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6834/2104418334","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6833","nonce":2104418334},{"type":"v1","addr":"172.21.15.32:6834","nonce":2104418334}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-08-03T20:48:26.652521+0000","modified":"2024-08-03T20:48:29.661524+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4866},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4866":{"gid":4866,"name":"a","rank":0,"incarnation":13,"state":"up:active","state_seq":3,"addr":"172.21.15.32:6835/1186001746","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":1186001746},{"type":"v1","addr":"172.21.15.32:6835","nonce":1186001746}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4866,"qdb_cluster":[4866]},"id":2}]} 2024-08-03T20:48:30.662 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 17 2024-08-03T20:48:30.663 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:30.662+0000 7fc5c7e00640 1 -- 172.21.15.32:0/1129075505 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fc5a4077010 msgr2=0x7fc5a4079520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:48:30.664 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:30.662+0000 7fc5c7e00640 1 --2- 172.21.15.32:0/1129075505 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fc5a4077010 0x7fc5a4079520 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fc5b800bb50 tx=0x7fc5b800b2e0 comp rx=0 tx=0).stop 2024-08-03T20:48:30.664 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:30.662+0000 7fc5c7e00640 1 -- 172.21.15.32:0/1129075505 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc5c80b74c0 msgr2=0x7fc5c80aec60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:48:30.664 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:30.662+0000 7fc5c7e00640 1 --2- 172.21.15.32:0/1129075505 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc5c80b74c0 0x7fc5c80aec60 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7fc5b0000c00 tx=0x7fc5b0003910 comp rx=0 tx=0).stop 2024-08-03T20:48:30.664 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:30.662+0000 7fc5c7e00640 1 -- 172.21.15.32:0/1129075505 shutdown_connections 2024-08-03T20:48:30.664 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:30.662+0000 7fc5c7e00640 1 --2- 172.21.15.32:0/1129075505 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fc5a4077010 0x7fc5a4079520 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:30.664 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:30.662+0000 7fc5c7e00640 1 --2- 172.21.15.32:0/1129075505 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fc5c8161d20 0x7fc5c81f8ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:30.664 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:30.662+0000 7fc5c7e00640 1 --2- 172.21.15.32:0/1129075505 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc5c80b74c0 0x7fc5c80aec60 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:30.664 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:30.662+0000 7fc5c7e00640 1 --2- 172.21.15.32:0/1129075505 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc5c80b52a0 0x7fc5c80b08b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:30.664 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:30.662+0000 7fc5c7e00640 1 -- 172.21.15.32:0/1129075505 >> 172.21.15.32:0/1129075505 conn(0x7fc5c80d1e90 msgr2=0x7fc5c8160190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:48:30.664 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:30.662+0000 7fc5c7e00640 1 -- 172.21.15.32:0/1129075505 shutdown_connections 2024-08-03T20:48:30.664 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:30.662+0000 7fc5c7e00640 1 -- 172.21.15.32:0/1129075505 wait complete. 2024-08-03T20:48:30.686 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-08-03T20:48:30.915 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:30.913+0000 7fc59b800640 1 -- 172.21.15.32:0/2739439667 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc594144f90 msgr2=0x7fc594074570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:48:30.915 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:30.913+0000 7fc59b800640 1 --2- 172.21.15.32:0/2739439667 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc594144f90 0x7fc594074570 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7fc594072d80 tx=0x7fc58002a5f0 comp rx=0 tx=0).stop 2024-08-03T20:48:30.915 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:30.913+0000 7fc59b800640 1 -- 172.21.15.32:0/2739439667 shutdown_connections 2024-08-03T20:48:30.915 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:30.913+0000 7fc59b800640 1 --2- 172.21.15.32:0/2739439667 >> v2:172.21.15.118:3300/0 conn(0x7fc59415fff0 0x7fc594160400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:30.915 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:30.913+0000 7fc59b800640 1 --2- 172.21.15.32:0/2739439667 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc594144f90 0x7fc594074570 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:30.915 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:30.913+0000 7fc59b800640 1 -- 172.21.15.32:0/2739439667 >> 172.21.15.32:0/2739439667 conn(0x7fc5940b1bd0 msgr2=0x7fc5940b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:48:30.915 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:30.913+0000 7fc59b800640 1 -- 172.21.15.32:0/2739439667 shutdown_connections 2024-08-03T20:48:30.915 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:30.913+0000 7fc59b800640 1 -- 172.21.15.32:0/2739439667 wait complete. 2024-08-03T20:48:30.917 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:30.915+0000 7fc59b800640 1 Processor -- start 2024-08-03T20:48:30.917 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:30.916+0000 7fc59b800640 1 -- start start 2024-08-03T20:48:30.919 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:30.917+0000 7fc59b800640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fc59406e930 0x7fc5941e5b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:48:30.920 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:30.917+0000 7fc59b800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc594144f90 0x7fc59415bc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:48:30.920 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:30.917+0000 7fc59b800640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc59415fff0 0x7fc594142e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:48:30.920 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:30.917+0000 7fc59b800640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fc5940cf840 con 0x7fc594144f90 2024-08-03T20:48:30.920 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:30.917+0000 7fc59b800640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7fc594145f30 con 0x7fc59415fff0 2024-08-03T20:48:30.920 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:30.917+0000 7fc59b800640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7fc5940cf9c0 con 0x7fc59406e930 2024-08-03T20:48:30.920 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:30.917+0000 7fc593e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc594144f90 0x7fc59415bc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:48:30.920 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:30.917+0000 7fc593e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc594144f90 0x7fc59415bc60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:46440/0 (socket says 172.21.15.32:46440) 2024-08-03T20:48:30.920 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:30.917+0000 7fc593e00640 1 -- 172.21.15.32:0/1372059568 learned_addr learned my addr 172.21.15.32:0/1372059568 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:48:30.920 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:30.918+0000 7fc598e00640 1 --2- 172.21.15.32:0/1372059568 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fc59406e930 0x7fc5941e5b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:48:30.920 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:30.918+0000 7fc599800640 1 --2- 172.21.15.32:0/1372059568 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc59415fff0 0x7fc594142e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:48:30.920 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:30.918+0000 7fc593e00640 1 -- 172.21.15.32:0/1372059568 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fc59406e930 msgr2=0x7fc5941e5b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:48:30.920 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:30.918+0000 7fc593e00640 1 --2- 172.21.15.32:0/1372059568 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fc59406e930 0x7fc5941e5b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:30.920 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:30.918+0000 7fc593e00640 1 -- 172.21.15.32:0/1372059568 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc59415fff0 msgr2=0x7fc594142e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:48:30.921 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:30.918+0000 7fc593e00640 1 --2- 172.21.15.32:0/1372059568 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc59415fff0 0x7fc594142e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:30.921 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:30.918+0000 7fc593e00640 1 -- 172.21.15.32:0/1372059568 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc5941f5790 con 0x7fc594144f90 2024-08-03T20:48:30.921 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:30.918+0000 7fc593e00640 1 --2- 172.21.15.32:0/1372059568 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc594144f90 0x7fc59415bc60 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7fc580034040 tx=0x7fc580003910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:48:30.921 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:30.918+0000 7fc590c00640 1 -- 172.21.15.32:0/1372059568 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc58003e030 con 0x7fc594144f90 2024-08-03T20:48:30.922 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:30.919+0000 7fc590c00640 1 -- 172.21.15.32:0/1372059568 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fc580003e00 con 0x7fc594144f90 2024-08-03T20:48:30.922 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:30.919+0000 7fc590c00640 1 -- 172.21.15.32:0/1372059568 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc580035040 con 0x7fc594144f90 2024-08-03T20:48:30.922 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:30.919+0000 7fc59b800640 1 -- 172.21.15.32:0/1372059568 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc5941f5a30 con 0x7fc594144f90 2024-08-03T20:48:30.922 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:30.919+0000 7fc59b800640 1 -- 172.21.15.32:0/1372059568 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc5941f5ff0 con 0x7fc594144f90 2024-08-03T20:48:30.922 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:30.920+0000 7fc59b800640 1 -- 172.21.15.32:0/1372059568 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc594160c30 con 0x7fc594144f90 2024-08-03T20:48:30.922 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:30.921+0000 7fc590c00640 1 -- 172.21.15.32:0/1372059568 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7fc580041030 con 0x7fc594144f90 2024-08-03T20:48:30.924 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:30.923+0000 7fc590c00640 1 --2- 172.21.15.32:0/1372059568 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fc574077010 0x7fc574079520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:48:30.924 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:30.923+0000 7fc590c00640 1 -- 172.21.15.32:0/1372059568 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(32..32 src has 1..32) ==== 5709+0+0 (secure 0 0 0) 0x7fc5800b90b0 con 0x7fc594144f90 2024-08-03T20:48:30.924 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:30.923+0000 7fc598e00640 1 --2- 172.21.15.32:0/1372059568 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fc574077010 0x7fc574079520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:48:30.925 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:30.924+0000 7fc598e00640 1 --2- 172.21.15.32:0/1372059568 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fc574077010 0x7fc574079520 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fc588001720 tx=0x7fc588009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:48:30.928 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:30.927+0000 7fc590c00640 1 -- 172.21.15.32:0/1372059568 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7fc580081bc0 con 0x7fc594144f90 2024-08-03T20:48:31.095 INFO:tasks.ceph.mds.a.smithi032.stderr:2024-08-03T20:48:31.093+0000 7fe39a200640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-08-03T20:48:31.131 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:31.130+0000 7fc59b800640 1 -- 172.21.15.32:0/1372059568 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fc5941f5510 con 0x7fc594144f90 2024-08-03T20:48:31.132 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:31.131+0000 7fc590c00640 1 -- 172.21.15.32:0/1372059568 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v32) ==== 74+0+16609 (secure 0 0 0) 0x7fc5941f5510 con 0x7fc594144f90 2024-08-03T20:48:31.133 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-03T20:48:31.133 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":32,"fsid":"40ab141a-01d7-49b9-ae3c-2edec9876e95","created":"2024-08-03T20:47:14.957362+0000","modified":"2024-08-03T20:48:28.122923+0000","last_up_change":"2024-08-03T20:47:28.874451+0000","last_in_change":"2024-08-03T20:47:20.079442+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-08-03T20:47:28.261300+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_metadata","create_time":"2024-08-03T20:48:24.969840+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"31","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data","create_time":"2024-08-03T20:48:25.603595+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"32","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5a83e759-9035-438b-889a-0a6af1f0a0d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6816","nonce":3066156116},{"type":"v1","addr":"172.21.15.32:6817","nonce":3066156116}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6818","nonce":3066156116},{"type":"v1","addr":"172.21.15.32:6819","nonce":3066156116}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6822","nonce":3066156116},{"type":"v1","addr":"172.21.15.32:6823","nonce":3066156116}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6820","nonce":3066156116},{"type":"v1","addr":"172.21.15.32:6821","nonce":3066156116}]},"public_addr":"172.21.15.32:6817/3066156116","cluster_addr":"172.21.15.32:6819/3066156116","heartbeat_back_addr":"172.21.15.32:6823/3066156116","heartbeat_front_addr":"172.21.15.32:6821/3066156116","state":["exists","up"]},{"osd":1,"uuid":"d7c5c210-da4f-49f9-b9e7-1dfb02daa4bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6824","nonce":1551241759},{"type":"v1","addr":"172.21.15.32:6825","nonce":1551241759}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6826","nonce":1551241759},{"type":"v1","addr":"172.21.15.32:6827","nonce":1551241759}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6830","nonce":1551241759},{"type":"v1","addr":"172.21.15.32:6831","nonce":1551241759}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6828","nonce":1551241759},{"type":"v1","addr":"172.21.15.32:6829","nonce":1551241759}]},"public_addr":"172.21.15.32:6825/1551241759","cluster_addr":"172.21.15.32:6827/1551241759","heartbeat_back_addr":"172.21.15.32:6831/1551241759","heartbeat_front_addr":"172.21.15.32:6829/1551241759","state":["exists","up"]},{"osd":2,"uuid":"57f2e6de-a201-45c2-8981-8e56a4875ebd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6808","nonce":3806708944},{"type":"v1","addr":"172.21.15.32:6809","nonce":3806708944}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6810","nonce":3806708944},{"type":"v1","addr":"172.21.15.32:6811","nonce":3806708944}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6814","nonce":3806708944},{"type":"v1","addr":"172.21.15.32:6815","nonce":3806708944}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6812","nonce":3806708944},{"type":"v1","addr":"172.21.15.32:6813","nonce":3806708944}]},"public_addr":"172.21.15.32:6809/3806708944","cluster_addr":"172.21.15.32:6811/3806708944","heartbeat_back_addr":"172.21.15.32:6815/3806708944","heartbeat_front_addr":"172.21.15.32:6813/3806708944","state":["exists","up"]},{"osd":3,"uuid":"6439a280-da45-4148-8130-f16323935ff3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6800","nonce":9088303},{"type":"v1","addr":"172.21.15.32:6801","nonce":9088303}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6802","nonce":9088303},{"type":"v1","addr":"172.21.15.32:6803","nonce":9088303}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6806","nonce":9088303},{"type":"v1","addr":"172.21.15.32:6807","nonce":9088303}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6804","nonce":9088303},{"type":"v1","addr":"172.21.15.32:6805","nonce":9088303}]},"public_addr":"172.21.15.32:6801/9088303","cluster_addr":"172.21.15.32:6803/9088303","heartbeat_back_addr":"172.21.15.32:6807/9088303","heartbeat_front_addr":"172.21.15.32:6805/9088303","state":["exists","up"]},{"osd":4,"uuid":"4f902eff-1e18-418d-82c9-2151c69524fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6802","nonce":864752275},{"type":"v1","addr":"172.21.15.118:6803","nonce":864752275}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6804","nonce":864752275},{"type":"v1","addr":"172.21.15.118:6805","nonce":864752275}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6808","nonce":864752275},{"type":"v1","addr":"172.21.15.118:6809","nonce":864752275}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6806","nonce":864752275},{"type":"v1","addr":"172.21.15.118:6807","nonce":864752275}]},"public_addr":"172.21.15.118:6803/864752275","cluster_addr":"172.21.15.118:6805/864752275","heartbeat_back_addr":"172.21.15.118:6809/864752275","heartbeat_front_addr":"172.21.15.118:6807/864752275","state":["exists","up"]},{"osd":5,"uuid":"bfd9feac-dd06-458c-91b9-af7f175f612c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6810","nonce":2724891770},{"type":"v1","addr":"172.21.15.118:6811","nonce":2724891770}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6812","nonce":2724891770},{"type":"v1","addr":"172.21.15.118:6813","nonce":2724891770}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6816","nonce":2724891770},{"type":"v1","addr":"172.21.15.118:6817","nonce":2724891770}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6814","nonce":2724891770},{"type":"v1","addr":"172.21.15.118:6815","nonce":2724891770}]},"public_addr":"172.21.15.118:6811/2724891770","cluster_addr":"172.21.15.118:6813/2724891770","heartbeat_back_addr":"172.21.15.118:6817/2724891770","heartbeat_front_addr":"172.21.15.118:6815/2724891770","state":["exists","up"]},{"osd":6,"uuid":"5ed5a279-9d8e-416d-bbc7-7edd3e6d5705","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6825","nonce":801947386},{"type":"v1","addr":"172.21.15.118:6827","nonce":801947386}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6828","nonce":801947386},{"type":"v1","addr":"172.21.15.118:6829","nonce":801947386}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6832","nonce":801947386},{"type":"v1","addr":"172.21.15.118:6833","nonce":801947386}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6830","nonce":801947386},{"type":"v1","addr":"172.21.15.118:6831","nonce":801947386}]},"public_addr":"172.21.15.118:6827/801947386","cluster_addr":"172.21.15.118:6829/801947386","heartbeat_back_addr":"172.21.15.118:6833/801947386","heartbeat_front_addr":"172.21.15.118:6831/801947386","state":["exists","up"]},{"osd":7,"uuid":"f2086c47-875e-4451-a436-2008d1ea16af","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6818","nonce":653000048},{"type":"v1","addr":"172.21.15.118:6819","nonce":653000048}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6820","nonce":653000048},{"type":"v1","addr":"172.21.15.118:6821","nonce":653000048}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6824","nonce":653000048},{"type":"v1","addr":"172.21.15.118:6826","nonce":653000048}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6822","nonce":653000048},{"type":"v1","addr":"172.21.15.118:6823","nonce":653000048}]},"public_addr":"172.21.15.118:6819/653000048","cluster_addr":"172.21.15.118:6821/653000048","heartbeat_back_addr":"172.21.15.118:6826/653000048","heartbeat_front_addr":"172.21.15.118:6823/653000048","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-03T20:47:25.491994+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-03T20:47:25.587358+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-03T20:47:25.454175+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-03T20:47:25.377734+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-03T20:47:25.412351+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-03T20:47:25.499815+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-03T20:47:25.575044+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-03T20:47:25.552383+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-08-03T20:48:31.135 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:31.133+0000 7fc59b800640 1 -- 172.21.15.32:0/1372059568 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fc574077010 msgr2=0x7fc574079520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:48:31.135 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:31.133+0000 7fc59b800640 1 --2- 172.21.15.32:0/1372059568 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fc574077010 0x7fc574079520 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fc588001720 tx=0x7fc588009040 comp rx=0 tx=0).stop 2024-08-03T20:48:31.135 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:31.133+0000 7fc59b800640 1 -- 172.21.15.32:0/1372059568 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc594144f90 msgr2=0x7fc59415bc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:48:31.136 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:31.133+0000 7fc59b800640 1 --2- 172.21.15.32:0/1372059568 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc594144f90 0x7fc59415bc60 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7fc580034040 tx=0x7fc580003910 comp rx=0 tx=0).stop 2024-08-03T20:48:31.136 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:31.134+0000 7fc59b800640 1 -- 172.21.15.32:0/1372059568 shutdown_connections 2024-08-03T20:48:31.136 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:31.134+0000 7fc59b800640 1 --2- 172.21.15.32:0/1372059568 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fc574077010 0x7fc574079520 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:31.136 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:31.134+0000 7fc59b800640 1 --2- 172.21.15.32:0/1372059568 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc59415fff0 0x7fc594142e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:31.136 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:31.134+0000 7fc59b800640 1 --2- 172.21.15.32:0/1372059568 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc594144f90 0x7fc59415bc60 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:31.136 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:31.134+0000 7fc59b800640 1 --2- 172.21.15.32:0/1372059568 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fc59406e930 0x7fc5941e5b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:31.136 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:31.134+0000 7fc59b800640 1 -- 172.21.15.32:0/1372059568 >> 172.21.15.32:0/1372059568 conn(0x7fc5940b1bd0 msgr2=0x7fc5940d08e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:48:31.136 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:31.134+0000 7fc59b800640 1 -- 172.21.15.32:0/1372059568 shutdown_connections 2024-08-03T20:48:31.136 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:31.134+0000 7fc59b800640 1 -- 172.21.15.32:0/1372059568 wait complete. 2024-08-03T20:48:31.157 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-08-03T20:48:31.157 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-08-03T20:48:31.386 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:31.384+0000 7f7b21800640 1 -- 172.21.15.32:0/3529732131 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7b1c0d0ff0 msgr2=0x7f7b1c0b73a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:48:31.387 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:31.384+0000 7f7b21800640 1 --2- 172.21.15.32:0/3529732131 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7b1c0d0ff0 0x7f7b1c0b73a0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f7b1c0d4ff0 tx=0x7f7b1402aa30 comp rx=0 tx=0).stop 2024-08-03T20:48:31.387 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:31.385+0000 7f7b21800640 1 -- 172.21.15.32:0/3529732131 shutdown_connections 2024-08-03T20:48:31.387 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:31.385+0000 7f7b21800640 1 --2- 172.21.15.32:0/3529732131 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7b1c0d0ff0 0x7f7b1c0b73a0 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:31.387 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:31.385+0000 7f7b21800640 1 --2- 172.21.15.32:0/3529732131 >> v2:172.21.15.118:3300/0 conn(0x7f7b1c0b94f0 0x7f7b1c0d0ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:31.387 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:31.385+0000 7f7b21800640 1 -- 172.21.15.32:0/3529732131 >> 172.21.15.32:0/3529732131 conn(0x7f7b1c0d40c0 msgr2=0x7f7b1c0d44d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:48:31.387 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:31.385+0000 7f7b21800640 1 -- 172.21.15.32:0/3529732131 shutdown_connections 2024-08-03T20:48:31.387 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:31.385+0000 7f7b21800640 1 -- 172.21.15.32:0/3529732131 wait complete. 2024-08-03T20:48:31.388 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:31.387+0000 7f7b21800640 1 Processor -- start 2024-08-03T20:48:31.388 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:31.387+0000 7f7b21800640 1 -- start start 2024-08-03T20:48:31.392 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:31.388+0000 7f7b21800640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f7b1c0b94f0 0x7f7b1c14f2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:48:31.392 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:31.388+0000 7f7b21800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7b1c0d0ff0 0x7f7b1c1f9020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:48:31.392 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:31.389+0000 7f7b21800640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f7b1c0d4be0 0x7f7b1c151550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:48:31.392 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:31.389+0000 7f7b21800640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f7b1c0b7c70 con 0x7f7b1c0d0ff0 2024-08-03T20:48:31.392 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:31.389+0000 7f7b21800640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f7b1c0b7af0 con 0x7f7b1c0d4be0 2024-08-03T20:48:31.392 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:31.389+0000 7f7b21800640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7f7b1c0b7df0 con 0x7f7b1c0b94f0 2024-08-03T20:48:31.392 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:31.389+0000 7f7b13e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7b1c0d0ff0 0x7f7b1c1f9020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:48:31.392 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:31.389+0000 7f7b13e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7b1c0d0ff0 0x7f7b1c1f9020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:46458/0 (socket says 172.21.15.32:46458) 2024-08-03T20:48:31.392 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:31.389+0000 7f7b13e00640 1 -- 172.21.15.32:0/2327861414 learned_addr learned my addr 172.21.15.32:0/2327861414 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:48:31.392 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:31.389+0000 7f7b1be00640 1 --2- 172.21.15.32:0/2327861414 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f7b1c0b94f0 0x7f7b1c14f2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:48:31.392 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:31.389+0000 7f7b20e00640 1 --2- 172.21.15.32:0/2327861414 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f7b1c0d4be0 0x7f7b1c151550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:48:31.392 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:31.389+0000 7f7b13e00640 1 -- 172.21.15.32:0/2327861414 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f7b1c0b94f0 msgr2=0x7f7b1c14f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:48:31.392 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:31.389+0000 7f7b13e00640 1 --2- 172.21.15.32:0/2327861414 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f7b1c0b94f0 0x7f7b1c14f2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:31.393 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:31.389+0000 7f7b13e00640 1 -- 172.21.15.32:0/2327861414 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f7b1c0d4be0 msgr2=0x7f7b1c151550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:48:31.393 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:31.389+0000 7f7b13e00640 1 --2- 172.21.15.32:0/2327861414 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f7b1c0d4be0 0x7f7b1c151550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:31.393 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:31.389+0000 7f7b13e00640 1 -- 172.21.15.32:0/2327861414 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7b1c0c8120 con 0x7f7b1c0d0ff0 2024-08-03T20:48:31.393 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:31.389+0000 7f7b20e00640 1 --2- 172.21.15.32:0/2327861414 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f7b1c0d4be0 0x7f7b1c151550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-03T20:48:31.393 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:31.390+0000 7f7b13e00640 1 --2- 172.21.15.32:0/2327861414 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7b1c0d0ff0 0x7f7b1c1f9020 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f7b0400d770 tx=0x7f7b0400dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:48:31.393 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:31.390+0000 7f7b18c00640 1 -- 172.21.15.32:0/2327861414 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b04017070 con 0x7f7b1c0d0ff0 2024-08-03T20:48:31.393 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:31.390+0000 7f7b18c00640 1 -- 172.21.15.32:0/2327861414 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f7b04009d20 con 0x7f7b1c0d0ff0 2024-08-03T20:48:31.393 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:31.390+0000 7f7b21800640 1 -- 172.21.15.32:0/2327861414 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7b1c1f8860 con 0x7f7b1c0d0ff0 2024-08-03T20:48:31.393 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:31.390+0000 7f7b18c00640 1 -- 172.21.15.32:0/2327861414 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b040106c0 con 0x7f7b1c0d0ff0 2024-08-03T20:48:31.393 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:31.390+0000 7f7b21800640 1 -- 172.21.15.32:0/2327861414 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7b1c1f8b00 con 0x7f7b1c0d0ff0 2024-08-03T20:48:31.394 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:31.391+0000 7f7b21800640 1 -- 172.21.15.32:0/2327861414 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7b1c14f880 con 0x7f7b1c0d0ff0 2024-08-03T20:48:31.394 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:31.392+0000 7f7b18c00640 1 -- 172.21.15.32:0/2327861414 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f7b0401a070 con 0x7f7b1c0d0ff0 2024-08-03T20:48:31.395 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:31.394+0000 7f7b18c00640 1 --2- 172.21.15.32:0/2327861414 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f7af8077010 0x7f7af8079520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:48:31.395 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:31.394+0000 7f7b18c00640 1 -- 172.21.15.32:0/2327861414 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(32..32 src has 1..32) ==== 5709+0+0 (secure 0 0 0) 0x7f7b04098aa0 con 0x7f7b1c0d0ff0 2024-08-03T20:48:31.396 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:31.395+0000 7f7b1be00640 1 --2- 172.21.15.32:0/2327861414 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f7af8077010 0x7f7af8079520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:48:31.396 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:31.395+0000 7f7b1be00640 1 --2- 172.21.15.32:0/2327861414 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f7af8077010 0x7f7af8079520 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f7b0c001710 tx=0x7f7b0c009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:48:31.400 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:31.399+0000 7f7b18c00640 1 -- 172.21.15.32:0/2327861414 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7f7b1c0c8120 con 0x7f7b1c0d0ff0 2024-08-03T20:48:31.601 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:31.599+0000 7f7b21800640 1 -- 172.21.15.32:0/2327861414 --> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f7b1c0d0ab0 con 0x7f7af8077010 2024-08-03T20:48:31.607 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:31.606+0000 7f7b18c00640 1 -- 172.21.15.32:0/2327861414 <== mgr.4106 v2:172.21.15.118:6800/4205950699 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+251938 (secure 0 0 0) 0x7f7b1c0d0ab0 con 0x7f7af8077010 2024-08-03T20:48:31.611 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-03T20:48:31.612 INFO:teuthology.orchestra.run.smithi032.stderr:dumped all 2024-08-03T20:48:31.615 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:31.610+0000 7f7b21800640 1 -- 172.21.15.32:0/2327861414 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f7af8077010 msgr2=0x7f7af8079520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:48:31.615 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:31.610+0000 7f7b21800640 1 --2- 172.21.15.32:0/2327861414 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f7af8077010 0x7f7af8079520 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f7b0c001710 tx=0x7f7b0c009040 comp rx=0 tx=0).stop 2024-08-03T20:48:31.615 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:31.610+0000 7f7b21800640 1 -- 172.21.15.32:0/2327861414 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7b1c0d0ff0 msgr2=0x7f7b1c1f9020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:48:31.615 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:31.610+0000 7f7b21800640 1 --2- 172.21.15.32:0/2327861414 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7b1c0d0ff0 0x7f7b1c1f9020 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f7b0400d770 tx=0x7f7b0400dc30 comp rx=0 tx=0).stop 2024-08-03T20:48:31.615 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:31.610+0000 7f7b21800640 1 -- 172.21.15.32:0/2327861414 shutdown_connections 2024-08-03T20:48:31.615 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:31.610+0000 7f7b21800640 1 --2- 172.21.15.32:0/2327861414 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f7af8077010 0x7f7af8079520 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:31.615 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:31.610+0000 7f7b21800640 1 --2- 172.21.15.32:0/2327861414 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f7b1c0d4be0 0x7f7b1c151550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:31.615 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:31.610+0000 7f7b21800640 1 --2- 172.21.15.32:0/2327861414 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7b1c0d0ff0 0x7f7b1c1f9020 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:31.615 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:31.610+0000 7f7b21800640 1 --2- 172.21.15.32:0/2327861414 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f7b1c0b94f0 0x7f7b1c14f2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:31.615 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:31.610+0000 7f7b21800640 1 -- 172.21.15.32:0/2327861414 >> 172.21.15.32:0/2327861414 conn(0x7f7b1c0d40c0 msgr2=0x7f7b1c15fcf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:48:31.615 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:31.610+0000 7f7b21800640 1 -- 172.21.15.32:0/2327861414 shutdown_connections 2024-08-03T20:48:31.615 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:31.610+0000 7f7b21800640 1 -- 172.21.15.32:0/2327861414 wait complete. 2024-08-03T20:48:31.632 INFO:teuthology.orchestra.run.smithi032.stdout:{"pg_ready":true,"pg_map":{"version":56,"stamp":"2024-08-03T20:48:30.235728+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":146,"num_write_kb":1642,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":111,"ondisk_log_size":111,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":140,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":253404,"kb_used_data":3952,"kb_used_omap":12,"kb_used_meta":249331,"kb_avail":754721316,"statfs":{"total":773094113280,"available":772834627584,"internally_reserved":0,"allocated":4046848,"data_stored":2258820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":255315505},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":28,"apply_latency_ms":28,"commit_latency_ns":28000000,"apply_latency_ns":28000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.131469"},"pg_stats":[{"pgid":"5.3f","version":"0'0","reported_seq":13,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-08-03T20:48:28.422512+0000","last_change":"2024-08-03T20:48:27.143195+0000","last_active":"2024-08-03T20:48:28.422512+0000","last_peered":"2024-08-03T20:48:28.422512+0000","last_clean":"2024-08-03T20:48:28.422512+0000","last_became_active":"2024-08-03T20:48:27.142841+0000","last_became_peered":"2024-08-03T20:48:27.142841+0000","last_unstale":"2024-08-03T20:48:28.422512+0000","last_undegraded":"2024-08-03T20:48:28.422512+0000","last_fullsized":"2024-08-03T20:48:28.422512+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:48:26.104177+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:48:26.104177+0000","last_clean_scrub_stamp":"2024-08-03T20:48:26.104177+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-04T23:17:51.269394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.3e","version":"0'0","reported_seq":13,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-08-03T20:48:28.422242+0000","last_change":"2024-08-03T20:48:27.147259+0000","last_active":"2024-08-03T20:48:28.422242+0000","last_peered":"2024-08-03T20:48:28.422242+0000","last_clean":"2024-08-03T20:48:28.422242+0000","last_became_active":"2024-08-03T20:48:27.146980+0000","last_became_peered":"2024-08-03T20:48:27.146980+0000","last_unstale":"2024-08-03T20:48:28.422242+0000","last_undegraded":"2024-08-03T20:48:28.422242+0000","last_fullsized":"2024-08-03T20:48:28.422242+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:48:26.104177+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:48:26.104177+0000","last_clean_scrub_stamp":"2024-08-03T20:48:26.104177+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-04T23:29:10.771944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.3d","version":"0'0","reported_seq":13,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-08-03T20:48:28.423560+0000","last_change":"2024-08-03T20:48:27.159194+0000","last_active":"2024-08-03T20:48:28.423560+0000","last_peered":"2024-08-03T20:48:28.423560+0000","last_clean":"2024-08-03T20:48:28.423560+0000","last_became_active":"2024-08-03T20:48:27.158826+0000","last_became_peered":"2024-08-03T20:48:27.158826+0000","last_unstale":"2024-08-03T20:48:28.423560+0000","last_undegraded":"2024-08-03T20:48:28.423560+0000","last_fullsized":"2024-08-03T20:48:28.423560+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:48:26.104177+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:48:26.104177+0000","last_clean_scrub_stamp":"2024-08-03T20:48:26.104177+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-04T20:57:36.715292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3c","version":"0'0","reported_seq":13,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-08-03T20:48:28.145230+0000","last_change":"2024-08-03T20:48:27.138551+0000","last_active":"2024-08-03T20:48:28.145230+0000","last_peered":"2024-08-03T20:48:28.145230+0000","last_clean":"2024-08-03T20:48:28.145230+0000","last_became_active":"2024-08-03T20:48:27.138162+0000","last_became_peered":"2024-08-03T20:48:27.138162+0000","last_unstale":"2024-08-03T20:48:28.145230+0000","last_undegraded":"2024-08-03T20:48:28.145230+0000","last_fullsized":"2024-08-03T20:48:28.145230+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:48:26.104177+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:48:26.104177+0000","last_clean_scrub_stamp":"2024-08-03T20:48:26.104177+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T05:00:43.122455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3b","version":"0'0","reported_seq":13,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-08-03T20:48:28.127999+0000","last_change":"2024-08-03T20:48:27.143928+0000","last_active":"2024-08-03T20:48:28.127999+0000","last_peered":"2024-08-03T20:48:28.127999+0000","last_clean":"2024-08-03T20:48:28.127999+0000","last_became_active":"2024-08-03T20:48:27.143678+0000","last_became_peered":"2024-08-03T20:48:27.143678+0000","last_unstale":"2024-08-03T20:48:28.127999+0000","last_undegraded":"2024-08-03T20:48:28.127999+0000","last_fullsized":"2024-08-03T20:48:28.127999+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:48:26.104177+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:48:26.104177+0000","last_clean_scrub_stamp":"2024-08-03T20:48:26.104177+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T07:04:00.770525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3a","version":"0'0","reported_seq":13,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-08-03T20:48:28.421562+0000","last_change":"2024-08-03T20:48:27.150753+0000","last_active":"2024-08-03T20:48:28.421562+0000","last_peered":"2024-08-03T20:48:28.421562+0000","last_clean":"2024-08-03T20:48:28.421562+0000","last_became_active":"2024-08-03T20:48:27.150231+0000","last_became_peered":"2024-08-03T20:48:27.150231+0000","last_unstale":"2024-08-03T20:48:28.421562+0000","last_undegraded":"2024-08-03T20:48:28.421562+0000","last_fullsized":"2024-08-03T20:48:28.421562+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:48:26.104177+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:48:26.104177+0000","last_clean_scrub_stamp":"2024-08-03T20:48:26.104177+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T05:38:08.118990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.39","version":"0'0","reported_seq":13,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-08-03T20:48:28.424356+0000","last_change":"2024-08-03T20:48:27.143187+0000","last_active":"2024-08-03T20:48:28.424356+0000","last_peered":"2024-08-03T20:48:28.424356+0000","last_clean":"2024-08-03T20:48:28.424356+0000","last_became_active":"2024-08-03T20:48:27.142847+0000","last_became_peered":"2024-08-03T20:48:27.142847+0000","last_unstale":"2024-08-03T20:48:28.424356+0000","last_undegraded":"2024-08-03T20:48:28.424356+0000","last_fullsized":"2024-08-03T20:48:28.424356+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:48:26.104177+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:48:26.104177+0000","last_clean_scrub_stamp":"2024-08-03T20:48:26.104177+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T03:32:32.293255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.38","version":"0'0","reported_seq":13,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-08-03T20:48:28.423300+0000","last_change":"2024-08-03T20:48:27.135374+0000","last_active":"2024-08-03T20:48:28.423300+0000","last_peered":"2024-08-03T20:48:28.423300+0000","last_clean":"2024-08-03T20:48:28.423300+0000","last_became_active":"2024-08-03T20:48:27.133427+0000","last_became_peered":"2024-08-03T20:48:27.133427+0000","last_unstale":"2024-08-03T20:48:28.423300+0000","last_undegraded":"2024-08-03T20:48:28.423300+0000","last_fullsized":"2024-08-03T20:48:28.423300+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:48:26.104177+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:48:26.104177+0000","last_clean_scrub_stamp":"2024-08-03T20:48:26.104177+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T08:15:08.218151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.37","version":"0'0","reported_seq":13,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-08-03T20:48:28.130110+0000","last_change":"2024-08-03T20:48:27.130570+0000","last_active":"2024-08-03T20:48:28.130110+0000","last_peered":"2024-08-03T20:48:28.130110+0000","last_clean":"2024-08-03T20:48:28.130110+0000","last_became_active":"2024-08-03T20:48:27.130232+0000","last_became_peered":"2024-08-03T20:48:27.130232+0000","last_unstale":"2024-08-03T20:48:28.130110+0000","last_undegraded":"2024-08-03T20:48:28.130110+0000","last_fullsized":"2024-08-03T20:48:28.130110+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:48:26.104177+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:48:26.104177+0000","last_clean_scrub_stamp":"2024-08-03T20:48:26.104177+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-04T23:03:19.113492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.36","version":"0'0","reported_seq":13,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-08-03T20:48:28.145487+0000","last_change":"2024-08-03T20:48:27.153541+0000","last_active":"2024-08-03T20:48:28.145487+0000","last_peered":"2024-08-03T20:48:28.145487+0000","last_clean":"2024-08-03T20:48:28.145487+0000","last_became_active":"2024-08-03T20:48:27.153246+0000","last_became_peered":"2024-08-03T20:48:27.153246+0000","last_unstale":"2024-08-03T20:48:28.145487+0000","last_undegraded":"2024-08-03T20:48:28.145487+0000","last_fullsized":"2024-08-03T20:48:28.145487+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:48:26.104177+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:48:26.104177+0000","last_clean_scrub_stamp":"2024-08-03T20:48:26.104177+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T06:04:52.253903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.35","version":"0'0","reported_seq":13,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-08-03T20:48:28.422884+0000","last_change":"2024-08-03T20:48:27.158184+0000","last_active":"2024-08-03T20:48:28.422884+0000","last_peered":"2024-08-03T20:48:28.422884+0000","last_clean":"2024-08-03T20:48:28.422884+0000","last_became_active":"2024-08-03T20:48:27.157907+0000","last_became_peered":"2024-08-03T20:48:27.157907+0000","last_unstale":"2024-08-03T20:48:28.422884+0000","last_undegraded":"2024-08-03T20:48:28.422884+0000","last_fullsized":"2024-08-03T20:48:28.422884+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:48:26.104177+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:48:26.104177+0000","last_clean_scrub_stamp":"2024-08-03T20:48:26.104177+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T06:09:59.767222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.34","version":"0'0","reported_seq":13,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-08-03T20:48:28.423365+0000","last_change":"2024-08-03T20:48:27.157166+0000","last_active":"2024-08-03T20:48:28.423365+0000","last_peered":"2024-08-03T20:48:28.423365+0000","last_clean":"2024-08-03T20:48:28.423365+0000","last_became_active":"2024-08-03T20:48:27.156773+0000","last_became_peered":"2024-08-03T20:48:27.156773+0000","last_unstale":"2024-08-03T20:48:28.423365+0000","last_undegraded":"2024-08-03T20:48:28.423365+0000","last_fullsized":"2024-08-03T20:48:28.423365+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:48:26.104177+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:48:26.104177+0000","last_clean_scrub_stamp":"2024-08-03T20:48:26.104177+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-04T21:00:04.257461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.33","version":"0'0","reported_seq":13,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-08-03T20:48:28.128023+0000","last_change":"2024-08-03T20:48:27.132385+0000","last_active":"2024-08-03T20:48:28.128023+0000","last_peered":"2024-08-03T20:48:28.128023+0000","last_clean":"2024-08-03T20:48:28.128023+0000","last_became_active":"2024-08-03T20:48:27.132059+0000","last_became_peered":"2024-08-03T20:48:27.132059+0000","last_unstale":"2024-08-03T20:48:28.128023+0000","last_undegraded":"2024-08-03T20:48:28.128023+0000","last_fullsized":"2024-08-03T20:48:28.128023+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:48:26.104177+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:48:26.104177+0000","last_clean_scrub_stamp":"2024-08-03T20:48:26.104177+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T01:47:24.684656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.32","version":"0'0","reported_seq":13,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-08-03T20:48:28.421773+0000","last_change":"2024-08-03T20:48:27.150846+0000","last_active":"2024-08-03T20:48:28.421773+0000","last_peered":"2024-08-03T20:48:28.421773+0000","last_clean":"2024-08-03T20:48:28.421773+0000","last_became_active":"2024-08-03T20:48:27.150498+0000","last_became_peered":"2024-08-03T20:48:27.150498+0000","last_unstale":"2024-08-03T20:48:28.421773+0000","last_undegraded":"2024-08-03T20:48:28.421773+0000","last_fullsized":"2024-08-03T20:48:28.421773+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:48:26.104177+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:48:26.104177+0000","last_clean_scrub_stamp":"2024-08-03T20:48:26.104177+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T06:21:28.181030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.31","version":"0'0","reported_seq":13,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-08-03T20:48:28.421970+0000","last_change":"2024-08-03T20:48:27.146235+0000","last_active":"2024-08-03T20:48:28.421970+0000","last_peered":"2024-08-03T20:48:28.421970+0000","last_clean":"2024-08-03T20:48:28.421970+0000","last_became_active":"2024-08-03T20:48:27.145486+0000","last_became_peered":"2024-08-03T20:48:27.145486+0000","last_unstale":"2024-08-03T20:48:28.421970+0000","last_undegraded":"2024-08-03T20:48:28.421970+0000","last_fullsized":"2024-08-03T20:48:28.421970+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:48:26.104177+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:48:26.104177+0000","last_clean_scrub_stamp":"2024-08-03T20:48:26.104177+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T03:46:54.747620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.30","version":"0'0","reported_seq":13,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-08-03T20:48:28.424085+0000","last_change":"2024-08-03T20:48:27.141138+0000","last_active":"2024-08-03T20:48:28.424085+0000","last_peered":"2024-08-03T20:48:28.424085+0000","last_clean":"2024-08-03T20:48:28.424085+0000","last_became_active":"2024-08-03T20:48:27.138995+0000","last_became_peered":"2024-08-03T20:48:27.138995+0000","last_unstale":"2024-08-03T20:48:28.424085+0000","last_undegraded":"2024-08-03T20:48:28.424085+0000","last_fullsized":"2024-08-03T20:48:28.424085+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:48:26.104177+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:48:26.104177+0000","last_clean_scrub_stamp":"2024-08-03T20:48:26.104177+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T04:17:36.047269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2f","version":"0'0","reported_seq":13,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-08-03T20:48:28.423115+0000","last_change":"2024-08-03T20:48:27.145592+0000","last_active":"2024-08-03T20:48:28.423115+0000","last_peered":"2024-08-03T20:48:28.423115+0000","last_clean":"2024-08-03T20:48:28.423115+0000","last_became_active":"2024-08-03T20:48:27.145280+0000","last_became_peered":"2024-08-03T20:48:27.145280+0000","last_unstale":"2024-08-03T20:48:28.423115+0000","last_undegraded":"2024-08-03T20:48:28.423115+0000","last_fullsized":"2024-08-03T20:48:28.423115+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:48:26.104177+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:48:26.104177+0000","last_clean_scrub_stamp":"2024-08-03T20:48:26.104177+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T00:06:25.299214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2e","version":"0'0","reported_seq":13,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-08-03T20:48:28.422310+0000","last_change":"2024-08-03T20:48:27.158905+0000","last_active":"2024-08-03T20:48:28.422310+0000","last_peered":"2024-08-03T20:48:28.422310+0000","last_clean":"2024-08-03T20:48:28.422310+0000","last_became_active":"2024-08-03T20:48:27.158502+0000","last_became_peered":"2024-08-03T20:48:27.158502+0000","last_unstale":"2024-08-03T20:48:28.422310+0000","last_undegraded":"2024-08-03T20:48:28.422310+0000","last_fullsized":"2024-08-03T20:48:28.422310+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:48:26.104177+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:48:26.104177+0000","last_clean_scrub_stamp":"2024-08-03T20:48:26.104177+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T01:33:26.863109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2d","version":"0'0","reported_seq":13,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-08-03T20:48:28.421984+0000","last_change":"2024-08-03T20:48:27.136832+0000","last_active":"2024-08-03T20:48:28.421984+0000","last_peered":"2024-08-03T20:48:28.421984+0000","last_clean":"2024-08-03T20:48:28.421984+0000","last_became_active":"2024-08-03T20:48:27.136381+0000","last_became_peered":"2024-08-03T20:48:27.136381+0000","last_unstale":"2024-08-03T20:48:28.421984+0000","last_undegraded":"2024-08-03T20:48:28.421984+0000","last_fullsized":"2024-08-03T20:48:28.421984+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:48:26.104177+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:48:26.104177+0000","last_clean_scrub_stamp":"2024-08-03T20:48:26.104177+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-04T22:37:06.699449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2c","version":"0'0","reported_seq":13,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-08-03T20:48:28.128153+0000","last_change":"2024-08-03T20:48:27.143611+0000","last_active":"2024-08-03T20:48:28.128153+0000","last_peered":"2024-08-03T20:48:28.128153+0000","last_clean":"2024-08-03T20:48:28.128153+0000","last_became_active":"2024-08-03T20:48:27.143009+0000","last_became_peered":"2024-08-03T20:48:27.143009+0000","last_unstale":"2024-08-03T20:48:28.128153+0000","last_undegraded":"2024-08-03T20:48:28.128153+0000","last_fullsized":"2024-08-03T20:48:28.128153+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:48:26.104177+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:48:26.104177+0000","last_clean_scrub_stamp":"2024-08-03T20:48:26.104177+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T01:06:57.219548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.2b","version":"0'0","reported_seq":13,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-08-03T20:48:28.129456+0000","last_change":"2024-08-03T20:48:27.134346+0000","last_active":"2024-08-03T20:48:28.129456+0000","last_peered":"2024-08-03T20:48:28.129456+0000","last_clean":"2024-08-03T20:48:28.129456+0000","last_became_active":"2024-08-03T20:48:27.133645+0000","last_became_peered":"2024-08-03T20:48:27.133645+0000","last_unstale":"2024-08-03T20:48:28.129456+0000","last_undegraded":"2024-08-03T20:48:28.129456+0000","last_fullsized":"2024-08-03T20:48:28.129456+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:48:26.104177+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:48:26.104177+0000","last_clean_scrub_stamp":"2024-08-03T20:48:26.104177+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T07:41:43.044274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.2a","version":"0'0","reported_seq":13,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-08-03T20:48:28.129427+0000","last_change":"2024-08-03T20:48:27.131250+0000","last_active":"2024-08-03T20:48:28.129427+0000","last_peered":"2024-08-03T20:48:28.129427+0000","last_clean":"2024-08-03T20:48:28.129427+0000","last_became_active":"2024-08-03T20:48:27.129718+0000","last_became_peered":"2024-08-03T20:48:27.129718+0000","last_unstale":"2024-08-03T20:48:28.129427+0000","last_undegraded":"2024-08-03T20:48:28.129427+0000","last_fullsized":"2024-08-03T20:48:28.129427+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:48:26.104177+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:48:26.104177+0000","last_clean_scrub_stamp":"2024-08-03T20:48:26.104177+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-04T21:10:14.020760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.29","version":"0'0","reported_seq":13,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-08-03T20:48:28.142100+0000","last_change":"2024-08-03T20:48:27.132413+0000","last_active":"2024-08-03T20:48:28.142100+0000","last_peered":"2024-08-03T20:48:28.142100+0000","last_clean":"2024-08-03T20:48:28.142100+0000","last_became_active":"2024-08-03T20:48:27.132105+0000","last_became_peered":"2024-08-03T20:48:27.132105+0000","last_unstale":"2024-08-03T20:48:28.142100+0000","last_undegraded":"2024-08-03T20:48:28.142100+0000","last_fullsized":"2024-08-03T20:48:28.142100+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:48:26.104177+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:48:26.104177+0000","last_clean_scrub_stamp":"2024-08-03T20:48:26.104177+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T04:19:29.463190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.28","version":"0'0","reported_seq":13,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-08-03T20:48:28.421971+0000","last_change":"2024-08-03T20:48:27.156281+0000","last_active":"2024-08-03T20:48:28.421971+0000","last_peered":"2024-08-03T20:48:28.421971+0000","last_clean":"2024-08-03T20:48:28.421971+0000","last_became_active":"2024-08-03T20:48:27.156001+0000","last_became_peered":"2024-08-03T20:48:27.156001+0000","last_unstale":"2024-08-03T20:48:28.421971+0000","last_undegraded":"2024-08-03T20:48:28.421971+0000","last_fullsized":"2024-08-03T20:48:28.421971+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:48:26.104177+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:48:26.104177+0000","last_clean_scrub_stamp":"2024-08-03T20:48:26.104177+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T06:09:59.767222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.27","version":"0'0","reported_seq":13,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-08-03T20:48:28.145578+0000","last_change":"2024-08-03T20:48:27.148919+0000","last_active":"2024-08-03T20:48:28.145578+0000","last_peered":"2024-08-03T20:48:28.145578+0000","last_clean":"2024-08-03T20:48:28.145578+0000","last_became_active":"2024-08-03T20:48:27.148637+0000","last_became_peered":"2024-08-03T20:48:27.148637+0000","last_unstale":"2024-08-03T20:48:28.145578+0000","last_undegraded":"2024-08-03T20:48:28.145578+0000","last_fullsized":"2024-08-03T20:48:28.145578+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:48:26.104177+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:48:26.104177+0000","last_clean_scrub_stamp":"2024-08-03T20:48:26.104177+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T05:52:18.600376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.26","version":"0'0","reported_seq":13,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-08-03T20:48:28.422460+0000","last_change":"2024-08-03T20:48:27.129677+0000","last_active":"2024-08-03T20:48:28.422460+0000","last_peered":"2024-08-03T20:48:28.422460+0000","last_clean":"2024-08-03T20:48:28.422460+0000","last_became_active":"2024-08-03T20:48:27.129348+0000","last_became_peered":"2024-08-03T20:48:27.129348+0000","last_unstale":"2024-08-03T20:48:28.422460+0000","last_undegraded":"2024-08-03T20:48:28.422460+0000","last_fullsized":"2024-08-03T20:48:28.422460+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:48:26.104177+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:48:26.104177+0000","last_clean_scrub_stamp":"2024-08-03T20:48:26.104177+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T02:06:10.358093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.25","version":"0'0","reported_seq":13,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-08-03T20:48:28.145548+0000","last_change":"2024-08-03T20:48:27.148779+0000","last_active":"2024-08-03T20:48:28.145548+0000","last_peered":"2024-08-03T20:48:28.145548+0000","last_clean":"2024-08-03T20:48:28.145548+0000","last_became_active":"2024-08-03T20:48:27.148161+0000","last_became_peered":"2024-08-03T20:48:27.148161+0000","last_unstale":"2024-08-03T20:48:28.145548+0000","last_undegraded":"2024-08-03T20:48:28.145548+0000","last_fullsized":"2024-08-03T20:48:28.145548+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:48:26.104177+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:48:26.104177+0000","last_clean_scrub_stamp":"2024-08-03T20:48:26.104177+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T01:04:35.590161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.24","version":"0'0","reported_seq":13,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-08-03T20:48:28.423507+0000","last_change":"2024-08-03T20:48:27.142641+0000","last_active":"2024-08-03T20:48:28.423507+0000","last_peered":"2024-08-03T20:48:28.423507+0000","last_clean":"2024-08-03T20:48:28.423507+0000","last_became_active":"2024-08-03T20:48:27.142331+0000","last_became_peered":"2024-08-03T20:48:27.142331+0000","last_unstale":"2024-08-03T20:48:28.423507+0000","last_undegraded":"2024-08-03T20:48:28.423507+0000","last_fullsized":"2024-08-03T20:48:28.423507+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:48:26.104177+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:48:26.104177+0000","last_clean_scrub_stamp":"2024-08-03T20:48:26.104177+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T04:17:36.047269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.23","version":"0'0","reported_seq":13,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-08-03T20:48:28.129737+0000","last_change":"2024-08-03T20:48:27.135815+0000","last_active":"2024-08-03T20:48:28.129737+0000","last_peered":"2024-08-03T20:48:28.129737+0000","last_clean":"2024-08-03T20:48:28.129737+0000","last_became_active":"2024-08-03T20:48:27.134045+0000","last_became_peered":"2024-08-03T20:48:27.134045+0000","last_unstale":"2024-08-03T20:48:28.129737+0000","last_undegraded":"2024-08-03T20:48:28.129737+0000","last_fullsized":"2024-08-03T20:48:28.129737+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:48:26.104177+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:48:26.104177+0000","last_clean_scrub_stamp":"2024-08-03T20:48:26.104177+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T06:33:48.629840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.22","version":"0'0","reported_seq":13,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-08-03T20:48:28.422729+0000","last_change":"2024-08-03T20:48:27.142274+0000","last_active":"2024-08-03T20:48:28.422729+0000","last_peered":"2024-08-03T20:48:28.422729+0000","last_clean":"2024-08-03T20:48:28.422729+0000","last_became_active":"2024-08-03T20:48:27.141948+0000","last_became_peered":"2024-08-03T20:48:27.141948+0000","last_unstale":"2024-08-03T20:48:28.422729+0000","last_undegraded":"2024-08-03T20:48:28.422729+0000","last_fullsized":"2024-08-03T20:48:28.422729+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:48:26.104177+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:48:26.104177+0000","last_clean_scrub_stamp":"2024-08-03T20:48:26.104177+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T03:46:54.747620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.21","version":"0'0","reported_seq":13,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-08-03T20:48:28.422296+0000","last_change":"2024-08-03T20:48:27.140643+0000","last_active":"2024-08-03T20:48:28.422296+0000","last_peered":"2024-08-03T20:48:28.422296+0000","last_clean":"2024-08-03T20:48:28.422296+0000","last_became_active":"2024-08-03T20:48:27.140194+0000","last_became_peered":"2024-08-03T20:48:27.140194+0000","last_unstale":"2024-08-03T20:48:28.422296+0000","last_undegraded":"2024-08-03T20:48:28.422296+0000","last_fullsized":"2024-08-03T20:48:28.422296+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:48:26.104177+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:48:26.104177+0000","last_clean_scrub_stamp":"2024-08-03T20:48:26.104177+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-04T23:58:08.281672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.20","version":"0'0","reported_seq":13,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-08-03T20:48:28.424357+0000","last_change":"2024-08-03T20:48:27.149507+0000","last_active":"2024-08-03T20:48:28.424357+0000","last_peered":"2024-08-03T20:48:28.424357+0000","last_clean":"2024-08-03T20:48:28.424357+0000","last_became_active":"2024-08-03T20:48:27.139413+0000","last_became_peered":"2024-08-03T20:48:27.139413+0000","last_unstale":"2024-08-03T20:48:28.424357+0000","last_undegraded":"2024-08-03T20:48:28.424357+0000","last_fullsized":"2024-08-03T20:48:28.424357+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:48:26.104177+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:48:26.104177+0000","last_clean_scrub_stamp":"2024-08-03T20:48:26.104177+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T07:27:20.062252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":13,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-08-03T20:48:28.142274+0000","last_change":"2024-08-03T20:48:27.135846+0000","last_active":"2024-08-03T20:48:28.142274+0000","last_peered":"2024-08-03T20:48:28.142274+0000","last_clean":"2024-08-03T20:48:28.142274+0000","last_became_active":"2024-08-03T20:48:27.135522+0000","last_became_peered":"2024-08-03T20:48:27.135522+0000","last_unstale":"2024-08-03T20:48:28.142274+0000","last_undegraded":"2024-08-03T20:48:28.142274+0000","last_fullsized":"2024-08-03T20:48:28.142274+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:48:26.104177+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:48:26.104177+0000","last_clean_scrub_stamp":"2024-08-03T20:48:26.104177+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T01:37:34.683431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1f","version":"31'4","reported_seq":19,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-08-03T20:48:28.422645+0000","last_change":"2024-08-03T20:48:26.141319+0000","last_active":"2024-08-03T20:48:28.422645+0000","last_peered":"2024-08-03T20:48:28.422645+0000","last_clean":"2024-08-03T20:48:28.422645+0000","last_became_active":"2024-08-03T20:48:26.140981+0000","last_became_peered":"2024-08-03T20:48:26.140981+0000","last_unstale":"2024-08-03T20:48:28.422645+0000","last_undegraded":"2024-08-03T20:48:28.422645+0000","last_fullsized":"2024-08-03T20:48:28.422645+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:48:25.097349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:48:25.097349+0000","last_clean_scrub_stamp":"2024-08-03T20:48:25.097349+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T00:41:15.511682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1f","version":"0'0","reported_seq":13,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-08-03T20:48:28.422565+0000","last_change":"2024-08-03T20:48:27.139612+0000","last_active":"2024-08-03T20:48:28.422565+0000","last_peered":"2024-08-03T20:48:28.422565+0000","last_clean":"2024-08-03T20:48:28.422565+0000","last_became_active":"2024-08-03T20:48:27.137924+0000","last_became_peered":"2024-08-03T20:48:27.137924+0000","last_unstale":"2024-08-03T20:48:28.422565+0000","last_undegraded":"2024-08-03T20:48:28.422565+0000","last_fullsized":"2024-08-03T20:48:28.422565+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:48:26.104177+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:48:26.104177+0000","last_clean_scrub_stamp":"2024-08-03T20:48:26.104177+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-04T21:58:23.025892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1e","version":"31'1","reported_seq":16,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-08-03T20:48:28.142369+0000","last_change":"2024-08-03T20:48:26.143339+0000","last_active":"2024-08-03T20:48:28.142369+0000","last_peered":"2024-08-03T20:48:28.142369+0000","last_clean":"2024-08-03T20:48:28.142369+0000","last_became_active":"2024-08-03T20:48:26.143040+0000","last_became_peered":"2024-08-03T20:48:26.143040+0000","last_unstale":"2024-08-03T20:48:28.142369+0000","last_undegraded":"2024-08-03T20:48:28.142369+0000","last_fullsized":"2024-08-03T20:48:28.142369+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:48:25.097349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:48:25.097349+0000","last_clean_scrub_stamp":"2024-08-03T20:48:25.097349+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-04T21:51:24.778117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":13,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-08-03T20:48:28.129636+0000","last_change":"2024-08-03T20:48:27.125335+0000","last_active":"2024-08-03T20:48:28.129636+0000","last_peered":"2024-08-03T20:48:28.129636+0000","last_clean":"2024-08-03T20:48:28.129636+0000","last_became_active":"2024-08-03T20:48:27.124987+0000","last_became_peered":"2024-08-03T20:48:27.124987+0000","last_unstale":"2024-08-03T20:48:28.129636+0000","last_undegraded":"2024-08-03T20:48:28.129636+0000","last_fullsized":"2024-08-03T20:48:28.129636+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:48:26.104177+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:48:26.104177+0000","last_clean_scrub_stamp":"2024-08-03T20:48:26.104177+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T08:40:18.971655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"31'2","reported_seq":17,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-08-03T20:48:28.421961+0000","last_change":"2024-08-03T20:48:26.144273+0000","last_active":"2024-08-03T20:48:28.421961+0000","last_peered":"2024-08-03T20:48:28.421961+0000","last_clean":"2024-08-03T20:48:28.421961+0000","last_became_active":"2024-08-03T20:48:26.143990+0000","last_became_peered":"2024-08-03T20:48:26.143990+0000","last_unstale":"2024-08-03T20:48:28.421961+0000","last_undegraded":"2024-08-03T20:48:28.421961+0000","last_fullsized":"2024-08-03T20:48:28.421961+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:48:25.097349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:48:25.097349+0000","last_clean_scrub_stamp":"2024-08-03T20:48:25.097349+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T08:40:17.964827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":13,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-08-03T20:48:28.145733+0000","last_change":"2024-08-03T20:48:27.150048+0000","last_active":"2024-08-03T20:48:28.145733+0000","last_peered":"2024-08-03T20:48:28.145733+0000","last_clean":"2024-08-03T20:48:28.145733+0000","last_became_active":"2024-08-03T20:48:27.149709+0000","last_became_peered":"2024-08-03T20:48:27.149709+0000","last_unstale":"2024-08-03T20:48:28.145733+0000","last_undegraded":"2024-08-03T20:48:28.145733+0000","last_fullsized":"2024-08-03T20:48:28.145733+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:48:26.104177+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:48:26.104177+0000","last_clean_scrub_stamp":"2024-08-03T20:48:26.104177+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T03:46:54.747620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1c","version":"31'2","reported_seq":17,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-08-03T20:48:28.128145+0000","last_change":"2024-08-03T20:48:26.130713+0000","last_active":"2024-08-03T20:48:28.128145+0000","last_peered":"2024-08-03T20:48:28.128145+0000","last_clean":"2024-08-03T20:48:28.128145+0000","last_became_active":"2024-08-03T20:48:26.130361+0000","last_became_peered":"2024-08-03T20:48:26.130361+0000","last_unstale":"2024-08-03T20:48:28.128145+0000","last_undegraded":"2024-08-03T20:48:28.128145+0000","last_fullsized":"2024-08-03T20:48:28.128145+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:48:25.097349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:48:25.097349+0000","last_clean_scrub_stamp":"2024-08-03T20:48:25.097349+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-04T23:11:25.239186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":13,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-08-03T20:48:28.424290+0000","last_change":"2024-08-03T20:48:27.147471+0000","last_active":"2024-08-03T20:48:28.424290+0000","last_peered":"2024-08-03T20:48:28.424290+0000","last_clean":"2024-08-03T20:48:28.424290+0000","last_became_active":"2024-08-03T20:48:27.137844+0000","last_became_peered":"2024-08-03T20:48:27.137844+0000","last_unstale":"2024-08-03T20:48:28.424290+0000","last_undegraded":"2024-08-03T20:48:28.424290+0000","last_fullsized":"2024-08-03T20:48:28.424290+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:48:26.104177+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:48:26.104177+0000","last_clean_scrub_stamp":"2024-08-03T20:48:26.104177+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T01:33:02.971567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1b","version":"31'1","reported_seq":16,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-08-03T20:48:28.129826+0000","last_change":"2024-08-03T20:48:26.133899+0000","last_active":"2024-08-03T20:48:28.129826+0000","last_peered":"2024-08-03T20:48:28.129826+0000","last_clean":"2024-08-03T20:48:28.129826+0000","last_became_active":"2024-08-03T20:48:26.132184+0000","last_became_peered":"2024-08-03T20:48:26.132184+0000","last_unstale":"2024-08-03T20:48:28.129826+0000","last_undegraded":"2024-08-03T20:48:28.129826+0000","last_fullsized":"2024-08-03T20:48:28.129826+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:48:25.097349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:48:25.097349+0000","last_clean_scrub_stamp":"2024-08-03T20:48:25.097349+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T07:03:59.763697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":13,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-08-03T20:48:28.423236+0000","last_change":"2024-08-03T20:48:27.136698+0000","last_active":"2024-08-03T20:48:28.423236+0000","last_peered":"2024-08-03T20:48:28.423236+0000","last_clean":"2024-08-03T20:48:28.423236+0000","last_became_active":"2024-08-03T20:48:27.136364+0000","last_became_peered":"2024-08-03T20:48:27.136364+0000","last_unstale":"2024-08-03T20:48:28.423236+0000","last_undegraded":"2024-08-03T20:48:28.423236+0000","last_fullsized":"2024-08-03T20:48:28.423236+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:48:26.104177+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:48:26.104177+0000","last_clean_scrub_stamp":"2024-08-03T20:48:26.104177+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T00:06:25.299214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"31'1","reported_seq":16,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-08-03T20:48:28.129558+0000","last_change":"2024-08-03T20:48:26.138799+0000","last_active":"2024-08-03T20:48:28.129558+0000","last_peered":"2024-08-03T20:48:28.129558+0000","last_clean":"2024-08-03T20:48:28.129558+0000","last_became_active":"2024-08-03T20:48:26.138410+0000","last_became_peered":"2024-08-03T20:48:26.138410+0000","last_unstale":"2024-08-03T20:48:28.129558+0000","last_undegraded":"2024-08-03T20:48:28.129558+0000","last_fullsized":"2024-08-03T20:48:28.129558+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:48:25.097349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:48:25.097349+0000","last_clean_scrub_stamp":"2024-08-03T20:48:25.097349+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T04:43:29.058607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":13,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-08-03T20:48:28.129244+0000","last_change":"2024-08-03T20:48:27.139757+0000","last_active":"2024-08-03T20:48:28.129244+0000","last_peered":"2024-08-03T20:48:28.129244+0000","last_clean":"2024-08-03T20:48:28.129244+0000","last_became_active":"2024-08-03T20:48:27.138355+0000","last_became_peered":"2024-08-03T20:48:27.138355+0000","last_unstale":"2024-08-03T20:48:28.129244+0000","last_undegraded":"2024-08-03T20:48:28.129244+0000","last_fullsized":"2024-08-03T20:48:28.129244+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:48:26.104177+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:48:26.104177+0000","last_clean_scrub_stamp":"2024-08-03T20:48:26.104177+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T06:38:56.263014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-08-03T20:48:28.421410+0000","last_change":"2024-08-03T20:48:26.132625+0000","last_active":"2024-08-03T20:48:28.421410+0000","last_peered":"2024-08-03T20:48:28.421410+0000","last_clean":"2024-08-03T20:48:28.421410+0000","last_became_active":"2024-08-03T20:48:26.132320+0000","last_became_peered":"2024-08-03T20:48:26.132320+0000","last_unstale":"2024-08-03T20:48:28.421410+0000","last_undegraded":"2024-08-03T20:48:28.421410+0000","last_fullsized":"2024-08-03T20:48:28.421410+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:48:25.097349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:48:25.097349+0000","last_clean_scrub_stamp":"2024-08-03T20:48:25.097349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-04T21:10:13.013933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":13,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-08-03T20:48:28.144947+0000","last_change":"2024-08-03T20:48:27.149116+0000","last_active":"2024-08-03T20:48:28.144947+0000","last_peered":"2024-08-03T20:48:28.144947+0000","last_clean":"2024-08-03T20:48:28.144947+0000","last_became_active":"2024-08-03T20:48:27.148592+0000","last_became_peered":"2024-08-03T20:48:27.148592+0000","last_unstale":"2024-08-03T20:48:28.144947+0000","last_undegraded":"2024-08-03T20:48:28.144947+0000","last_fullsized":"2024-08-03T20:48:28.144947+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:48:26.104177+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:48:26.104177+0000","last_clean_scrub_stamp":"2024-08-03T20:48:26.104177+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T04:55:17.727620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-08-03T20:48:28.129103+0000","last_change":"2024-08-03T20:48:26.125016+0000","last_active":"2024-08-03T20:48:28.129103+0000","last_peered":"2024-08-03T20:48:28.129103+0000","last_clean":"2024-08-03T20:48:28.129103+0000","last_became_active":"2024-08-03T20:48:26.123236+0000","last_became_peered":"2024-08-03T20:48:26.123236+0000","last_unstale":"2024-08-03T20:48:28.129103+0000","last_undegraded":"2024-08-03T20:48:28.129103+0000","last_fullsized":"2024-08-03T20:48:28.129103+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:48:25.097349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:48:25.097349+0000","last_clean_scrub_stamp":"2024-08-03T20:48:25.097349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T02:09:07.905859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":13,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-08-03T20:48:28.423605+0000","last_change":"2024-08-03T20:48:27.153117+0000","last_active":"2024-08-03T20:48:28.423605+0000","last_peered":"2024-08-03T20:48:28.423605+0000","last_clean":"2024-08-03T20:48:28.423605+0000","last_became_active":"2024-08-03T20:48:27.152747+0000","last_became_peered":"2024-08-03T20:48:27.152747+0000","last_unstale":"2024-08-03T20:48:28.423605+0000","last_undegraded":"2024-08-03T20:48:28.423605+0000","last_fullsized":"2024-08-03T20:48:28.423605+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:48:26.104177+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:48:26.104177+0000","last_clean_scrub_stamp":"2024-08-03T20:48:26.104177+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T01:33:02.971567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-08-03T20:48:28.422366+0000","last_change":"2024-08-03T20:48:26.144117+0000","last_active":"2024-08-03T20:48:28.422366+0000","last_peered":"2024-08-03T20:48:28.422366+0000","last_clean":"2024-08-03T20:48:28.422366+0000","last_became_active":"2024-08-03T20:48:26.143642+0000","last_became_peered":"2024-08-03T20:48:26.143642+0000","last_unstale":"2024-08-03T20:48:28.422366+0000","last_undegraded":"2024-08-03T20:48:28.422366+0000","last_fullsized":"2024-08-03T20:48:28.422366+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:48:25.097349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:48:25.097349+0000","last_clean_scrub_stamp":"2024-08-03T20:48:25.097349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T06:33:47.623013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":13,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-08-03T20:48:28.422310+0000","last_change":"2024-08-03T20:48:27.157632+0000","last_active":"2024-08-03T20:48:28.422310+0000","last_peered":"2024-08-03T20:48:28.422310+0000","last_clean":"2024-08-03T20:48:28.422310+0000","last_became_active":"2024-08-03T20:48:27.157201+0000","last_became_peered":"2024-08-03T20:48:27.157201+0000","last_unstale":"2024-08-03T20:48:28.422310+0000","last_undegraded":"2024-08-03T20:48:28.422310+0000","last_fullsized":"2024-08-03T20:48:28.422310+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:48:26.104177+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:48:26.104177+0000","last_clean_scrub_stamp":"2024-08-03T20:48:26.104177+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T04:54:21.125473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-08-03T20:48:28.142495+0000","last_change":"2024-08-03T20:48:26.140459+0000","last_active":"2024-08-03T20:48:28.142495+0000","last_peered":"2024-08-03T20:48:28.142495+0000","last_clean":"2024-08-03T20:48:28.142495+0000","last_became_active":"2024-08-03T20:48:26.139412+0000","last_became_peered":"2024-08-03T20:48:26.139412+0000","last_unstale":"2024-08-03T20:48:28.142495+0000","last_undegraded":"2024-08-03T20:48:28.142495+0000","last_fullsized":"2024-08-03T20:48:28.142495+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:48:25.097349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:48:25.097349+0000","last_clean_scrub_stamp":"2024-08-03T20:48:25.097349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T00:02:20.619397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"0'0","reported_seq":13,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-08-03T20:48:28.422099+0000","last_change":"2024-08-03T20:48:27.139783+0000","last_active":"2024-08-03T20:48:28.422099+0000","last_peered":"2024-08-03T20:48:28.422099+0000","last_clean":"2024-08-03T20:48:28.422099+0000","last_became_active":"2024-08-03T20:48:27.136109+0000","last_became_peered":"2024-08-03T20:48:27.136109+0000","last_unstale":"2024-08-03T20:48:28.422099+0000","last_undegraded":"2024-08-03T20:48:28.422099+0000","last_fullsized":"2024-08-03T20:48:28.422099+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:48:26.104177+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:48:26.104177+0000","last_clean_scrub_stamp":"2024-08-03T20:48:26.104177+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-04T22:39:59.296381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.15","version":"31'3","reported_seq":18,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-08-03T20:48:28.423681+0000","last_change":"2024-08-03T20:48:26.140631+0000","last_active":"2024-08-03T20:48:28.423681+0000","last_peered":"2024-08-03T20:48:28.423681+0000","last_clean":"2024-08-03T20:48:28.423681+0000","last_became_active":"2024-08-03T20:48:26.139867+0000","last_became_peered":"2024-08-03T20:48:26.139867+0000","last_unstale":"2024-08-03T20:48:28.423681+0000","last_undegraded":"2024-08-03T20:48:28.423681+0000","last_fullsized":"2024-08-03T20:48:28.423681+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:48:25.097349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:48:25.097349+0000","last_clean_scrub_stamp":"2024-08-03T20:48:25.097349+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-04T23:17:50.262566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"0'0","reported_seq":13,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-08-03T20:48:28.423699+0000","last_change":"2024-08-03T20:48:27.159216+0000","last_active":"2024-08-03T20:48:28.423699+0000","last_peered":"2024-08-03T20:48:28.423699+0000","last_clean":"2024-08-03T20:48:28.423699+0000","last_became_active":"2024-08-03T20:48:27.158823+0000","last_became_peered":"2024-08-03T20:48:27.158823+0000","last_unstale":"2024-08-03T20:48:28.423699+0000","last_undegraded":"2024-08-03T20:48:28.423699+0000","last_fullsized":"2024-08-03T20:48:28.423699+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:48:26.104177+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:48:26.104177+0000","last_clean_scrub_stamp":"2024-08-03T20:48:26.104177+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T08:15:08.218151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"31'6","reported_seq":21,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-08-03T20:48:28.421992+0000","last_change":"2024-08-03T20:48:26.141579+0000","last_active":"2024-08-03T20:48:28.421992+0000","last_peered":"2024-08-03T20:48:28.421992+0000","last_clean":"2024-08-03T20:48:28.421992+0000","last_became_active":"2024-08-03T20:48:26.141241+0000","last_became_peered":"2024-08-03T20:48:26.141241+0000","last_unstale":"2024-08-03T20:48:28.421992+0000","last_undegraded":"2024-08-03T20:48:28.421992+0000","last_fullsized":"2024-08-03T20:48:28.421992+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:48:25.097349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:48:25.097349+0000","last_clean_scrub_stamp":"2024-08-03T20:48:25.097349+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T03:27:53.316279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":13,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-08-03T20:48:28.145839+0000","last_change":"2024-08-03T20:48:27.148908+0000","last_active":"2024-08-03T20:48:28.145839+0000","last_peered":"2024-08-03T20:48:28.145839+0000","last_clean":"2024-08-03T20:48:28.145839+0000","last_became_active":"2024-08-03T20:48:27.148502+0000","last_became_peered":"2024-08-03T20:48:27.148502+0000","last_unstale":"2024-08-03T20:48:28.145839+0000","last_undegraded":"2024-08-03T20:48:28.145839+0000","last_fullsized":"2024-08-03T20:48:28.145839+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:48:26.104177+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:48:26.104177+0000","last_clean_scrub_stamp":"2024-08-03T20:48:26.104177+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T06:09:59.767222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"31'2","reported_seq":17,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-08-03T20:48:28.129941+0000","last_change":"2024-08-03T20:48:26.131550+0000","last_active":"2024-08-03T20:48:28.129941+0000","last_peered":"2024-08-03T20:48:28.129941+0000","last_clean":"2024-08-03T20:48:28.129941+0000","last_became_active":"2024-08-03T20:48:26.130860+0000","last_became_peered":"2024-08-03T20:48:26.130860+0000","last_unstale":"2024-08-03T20:48:28.129941+0000","last_undegraded":"2024-08-03T20:48:28.129941+0000","last_fullsized":"2024-08-03T20:48:28.129941+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:48:25.097349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:48:25.097349+0000","last_clean_scrub_stamp":"2024-08-03T20:48:25.097349+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T01:47:23.677829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":13,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-08-03T20:48:28.421980+0000","last_change":"2024-08-03T20:48:27.150930+0000","last_active":"2024-08-03T20:48:28.421980+0000","last_peered":"2024-08-03T20:48:28.421980+0000","last_clean":"2024-08-03T20:48:28.421980+0000","last_became_active":"2024-08-03T20:48:27.150013+0000","last_became_peered":"2024-08-03T20:48:27.150013+0000","last_unstale":"2024-08-03T20:48:28.421980+0000","last_undegraded":"2024-08-03T20:48:28.421980+0000","last_fullsized":"2024-08-03T20:48:28.421980+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:48:26.104177+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:48:26.104177+0000","last_clean_scrub_stamp":"2024-08-03T20:48:26.104177+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T07:08:24.833346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-08-03T20:48:28.144939+0000","last_change":"2024-08-03T20:48:26.128882+0000","last_active":"2024-08-03T20:48:28.144939+0000","last_peered":"2024-08-03T20:48:28.144939+0000","last_clean":"2024-08-03T20:48:28.144939+0000","last_became_active":"2024-08-03T20:48:26.127874+0000","last_became_peered":"2024-08-03T20:48:26.127874+0000","last_unstale":"2024-08-03T20:48:28.144939+0000","last_undegraded":"2024-08-03T20:48:28.144939+0000","last_fullsized":"2024-08-03T20:48:28.144939+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:48:25.097349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:48:25.097349+0000","last_clean_scrub_stamp":"2024-08-03T20:48:25.097349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T00:41:15.511682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":13,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-08-03T20:48:28.423743+0000","last_change":"2024-08-03T20:48:27.144778+0000","last_active":"2024-08-03T20:48:28.423743+0000","last_peered":"2024-08-03T20:48:28.423743+0000","last_clean":"2024-08-03T20:48:28.423743+0000","last_became_active":"2024-08-03T20:48:27.144302+0000","last_became_peered":"2024-08-03T20:48:27.144302+0000","last_unstale":"2024-08-03T20:48:28.423743+0000","last_undegraded":"2024-08-03T20:48:28.423743+0000","last_fullsized":"2024-08-03T20:48:28.423743+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:48:26.104177+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:48:26.104177+0000","last_clean_scrub_stamp":"2024-08-03T20:48:26.104177+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-04T21:00:04.257461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-08-03T20:48:28.421495+0000","last_change":"2024-08-03T20:48:26.142689+0000","last_active":"2024-08-03T20:48:28.421495+0000","last_peered":"2024-08-03T20:48:28.421495+0000","last_clean":"2024-08-03T20:48:28.421495+0000","last_became_active":"2024-08-03T20:48:26.142251+0000","last_became_peered":"2024-08-03T20:48:26.142251+0000","last_unstale":"2024-08-03T20:48:28.421495+0000","last_undegraded":"2024-08-03T20:48:28.421495+0000","last_fullsized":"2024-08-03T20:48:28.421495+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:48:25.097349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:48:25.097349+0000","last_clean_scrub_stamp":"2024-08-03T20:48:25.097349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T07:41:42.037447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":13,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-08-03T20:48:28.422531+0000","last_change":"2024-08-03T20:48:27.138306+0000","last_active":"2024-08-03T20:48:28.422531+0000","last_peered":"2024-08-03T20:48:28.422531+0000","last_clean":"2024-08-03T20:48:28.422531+0000","last_became_active":"2024-08-03T20:48:27.138003+0000","last_became_peered":"2024-08-03T20:48:27.138003+0000","last_unstale":"2024-08-03T20:48:28.422531+0000","last_undegraded":"2024-08-03T20:48:28.422531+0000","last_fullsized":"2024-08-03T20:48:28.422531+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:48:26.104177+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:48:26.104177+0000","last_clean_scrub_stamp":"2024-08-03T20:48:26.104177+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T04:19:29.463190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.10","version":"31'8","reported_seq":23,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-08-03T20:48:28.421461+0000","last_change":"2024-08-03T20:48:26.137963+0000","last_active":"2024-08-03T20:48:28.421461+0000","last_peered":"2024-08-03T20:48:28.421461+0000","last_clean":"2024-08-03T20:48:28.421461+0000","last_became_active":"2024-08-03T20:48:26.137650+0000","last_became_peered":"2024-08-03T20:48:26.137650+0000","last_unstale":"2024-08-03T20:48:28.421461+0000","last_undegraded":"2024-08-03T20:48:28.421461+0000","last_fullsized":"2024-08-03T20:48:28.421461+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:48:25.097349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:48:25.097349+0000","last_clean_scrub_stamp":"2024-08-03T20:48:25.097349+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T07:56:10.908634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"0'0","reported_seq":13,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-08-03T20:48:28.129928+0000","last_change":"2024-08-03T20:48:27.141521+0000","last_active":"2024-08-03T20:48:28.129928+0000","last_peered":"2024-08-03T20:48:28.129928+0000","last_clean":"2024-08-03T20:48:28.129928+0000","last_became_active":"2024-08-03T20:48:27.139338+0000","last_became_peered":"2024-08-03T20:48:27.139338+0000","last_unstale":"2024-08-03T20:48:28.129928+0000","last_undegraded":"2024-08-03T20:48:28.129928+0000","last_fullsized":"2024-08-03T20:48:28.129928+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:48:26.104177+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:48:26.104177+0000","last_clean_scrub_stamp":"2024-08-03T20:48:26.104177+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-04T21:37:44.475861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.f","version":"31'1","reported_seq":16,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-08-03T20:48:28.145685+0000","last_change":"2024-08-03T20:48:26.137755+0000","last_active":"2024-08-03T20:48:28.145685+0000","last_peered":"2024-08-03T20:48:28.145685+0000","last_clean":"2024-08-03T20:48:28.145685+0000","last_became_active":"2024-08-03T20:48:26.137424+0000","last_became_peered":"2024-08-03T20:48:26.137424+0000","last_unstale":"2024-08-03T20:48:28.145685+0000","last_undegraded":"2024-08-03T20:48:28.145685+0000","last_fullsized":"2024-08-03T20:48:28.145685+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:48:25.097349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:48:25.097349+0000","last_clean_scrub_stamp":"2024-08-03T20:48:25.097349+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-04T21:51:24.778117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":13,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-08-03T20:48:28.423752+0000","last_change":"2024-08-03T20:48:27.145018+0000","last_active":"2024-08-03T20:48:28.423752+0000","last_peered":"2024-08-03T20:48:28.423752+0000","last_clean":"2024-08-03T20:48:28.423752+0000","last_became_active":"2024-08-03T20:48:27.144129+0000","last_became_peered":"2024-08-03T20:48:27.144129+0000","last_unstale":"2024-08-03T20:48:28.423752+0000","last_undegraded":"2024-08-03T20:48:28.423752+0000","last_fullsized":"2024-08-03T20:48:28.423752+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:48:26.104177+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:48:26.104177+0000","last_clean_scrub_stamp":"2024-08-03T20:48:26.104177+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T07:27:20.062252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.e","version":"31'2","reported_seq":17,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-08-03T20:48:28.129877+0000","last_change":"2024-08-03T20:48:26.130554+0000","last_active":"2024-08-03T20:48:28.129877+0000","last_peered":"2024-08-03T20:48:28.129877+0000","last_clean":"2024-08-03T20:48:28.129877+0000","last_became_active":"2024-08-03T20:48:26.130139+0000","last_became_peered":"2024-08-03T20:48:26.130139+0000","last_unstale":"2024-08-03T20:48:28.129877+0000","last_undegraded":"2024-08-03T20:48:28.129877+0000","last_fullsized":"2024-08-03T20:48:28.129877+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:48:25.097349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:48:25.097349+0000","last_clean_scrub_stamp":"2024-08-03T20:48:25.097349+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T01:09:11.035968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":13,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-08-03T20:48:28.145271+0000","last_change":"2024-08-03T20:48:27.138372+0000","last_active":"2024-08-03T20:48:28.145271+0000","last_peered":"2024-08-03T20:48:28.145271+0000","last_clean":"2024-08-03T20:48:28.145271+0000","last_became_active":"2024-08-03T20:48:27.138073+0000","last_became_peered":"2024-08-03T20:48:27.138073+0000","last_unstale":"2024-08-03T20:48:28.145271+0000","last_undegraded":"2024-08-03T20:48:28.145271+0000","last_fullsized":"2024-08-03T20:48:28.145271+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:48:26.104177+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:48:26.104177+0000","last_clean_scrub_stamp":"2024-08-03T20:48:26.104177+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-04T20:50:11.082404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.d","version":"31'1","reported_seq":16,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-08-03T20:48:28.129942+0000","last_change":"2024-08-03T20:48:26.131174+0000","last_active":"2024-08-03T20:48:28.129942+0000","last_peered":"2024-08-03T20:48:28.129942+0000","last_clean":"2024-08-03T20:48:28.129942+0000","last_became_active":"2024-08-03T20:48:26.130158+0000","last_became_peered":"2024-08-03T20:48:26.130158+0000","last_unstale":"2024-08-03T20:48:28.129942+0000","last_undegraded":"2024-08-03T20:48:28.129942+0000","last_fullsized":"2024-08-03T20:48:28.129942+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:48:25.097349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:48:25.097349+0000","last_clean_scrub_stamp":"2024-08-03T20:48:25.097349+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T01:06:56.212720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"0'0","reported_seq":13,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-08-03T20:48:28.128311+0000","last_change":"2024-08-03T20:48:27.135223+0000","last_active":"2024-08-03T20:48:28.128311+0000","last_peered":"2024-08-03T20:48:28.128311+0000","last_clean":"2024-08-03T20:48:28.128311+0000","last_became_active":"2024-08-03T20:48:27.134926+0000","last_became_peered":"2024-08-03T20:48:27.134926+0000","last_unstale":"2024-08-03T20:48:28.128311+0000","last_undegraded":"2024-08-03T20:48:28.128311+0000","last_fullsized":"2024-08-03T20:48:28.128311+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:48:26.104177+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:48:26.104177+0000","last_clean_scrub_stamp":"2024-08-03T20:48:26.104177+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-04T23:44:46.267836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-08-03T20:48:28.129601+0000","last_change":"2024-08-03T20:48:26.138873+0000","last_active":"2024-08-03T20:48:28.129601+0000","last_peered":"2024-08-03T20:48:28.129601+0000","last_clean":"2024-08-03T20:48:28.129601+0000","last_became_active":"2024-08-03T20:48:26.138603+0000","last_became_peered":"2024-08-03T20:48:26.138603+0000","last_unstale":"2024-08-03T20:48:28.129601+0000","last_undegraded":"2024-08-03T20:48:28.129601+0000","last_fullsized":"2024-08-03T20:48:28.129601+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:48:25.097349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:48:25.097349+0000","last_clean_scrub_stamp":"2024-08-03T20:48:25.097349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T07:43:33.823181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":13,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-08-03T20:48:28.127819+0000","last_change":"2024-08-03T20:48:27.123831+0000","last_active":"2024-08-03T20:48:28.127819+0000","last_peered":"2024-08-03T20:48:28.127819+0000","last_clean":"2024-08-03T20:48:28.127819+0000","last_became_active":"2024-08-03T20:48:27.123215+0000","last_became_peered":"2024-08-03T20:48:27.123215+0000","last_unstale":"2024-08-03T20:48:28.127819+0000","last_undegraded":"2024-08-03T20:48:28.127819+0000","last_fullsized":"2024-08-03T20:48:28.127819+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:48:26.104177+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:48:26.104177+0000","last_clean_scrub_stamp":"2024-08-03T20:48:26.104177+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T07:43:34.830009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.b","version":"31'2","reported_seq":17,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-08-03T20:48:28.142288+0000","last_change":"2024-08-03T20:48:26.138803+0000","last_active":"2024-08-03T20:48:28.142288+0000","last_peered":"2024-08-03T20:48:28.142288+0000","last_clean":"2024-08-03T20:48:28.142288+0000","last_became_active":"2024-08-03T20:48:26.138491+0000","last_became_peered":"2024-08-03T20:48:26.138491+0000","last_unstale":"2024-08-03T20:48:28.142288+0000","last_undegraded":"2024-08-03T20:48:28.142288+0000","last_fullsized":"2024-08-03T20:48:28.142288+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:48:25.097349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:48:25.097349+0000","last_clean_scrub_stamp":"2024-08-03T20:48:25.097349+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T01:34:49.869344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":13,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-08-03T20:48:28.128283+0000","last_change":"2024-08-03T20:48:27.152223+0000","last_active":"2024-08-03T20:48:28.128283+0000","last_peered":"2024-08-03T20:48:28.128283+0000","last_clean":"2024-08-03T20:48:28.128283+0000","last_became_active":"2024-08-03T20:48:27.151893+0000","last_became_peered":"2024-08-03T20:48:27.151893+0000","last_unstale":"2024-08-03T20:48:28.128283+0000","last_undegraded":"2024-08-03T20:48:28.128283+0000","last_fullsized":"2024-08-03T20:48:28.128283+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:48:26.104177+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:48:26.104177+0000","last_clean_scrub_stamp":"2024-08-03T20:48:26.104177+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T01:09:12.042796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-08-03T20:48:28.422682+0000","last_change":"2024-08-03T20:48:26.140981+0000","last_active":"2024-08-03T20:48:28.422682+0000","last_peered":"2024-08-03T20:48:28.422682+0000","last_clean":"2024-08-03T20:48:28.422682+0000","last_became_active":"2024-08-03T20:48:26.140503+0000","last_became_peered":"2024-08-03T20:48:26.140503+0000","last_unstale":"2024-08-03T20:48:28.422682+0000","last_undegraded":"2024-08-03T20:48:28.422682+0000","last_fullsized":"2024-08-03T20:48:28.422682+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:48:25.097349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:48:25.097349+0000","last_clean_scrub_stamp":"2024-08-03T20:48:25.097349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T05:17:20.685991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":13,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-08-03T20:48:28.127542+0000","last_change":"2024-08-03T20:48:27.148919+0000","last_active":"2024-08-03T20:48:28.127542+0000","last_peered":"2024-08-03T20:48:28.127542+0000","last_clean":"2024-08-03T20:48:28.127542+0000","last_became_active":"2024-08-03T20:48:27.148624+0000","last_became_peered":"2024-08-03T20:48:27.148624+0000","last_unstale":"2024-08-03T20:48:28.127542+0000","last_undegraded":"2024-08-03T20:48:28.127542+0000","last_fullsized":"2024-08-03T20:48:28.127542+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:48:26.104177+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:48:26.104177+0000","last_clean_scrub_stamp":"2024-08-03T20:48:26.104177+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T04:43:30.065435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-08-03T20:48:28.129073+0000","last_change":"2024-08-03T20:48:26.130661+0000","last_active":"2024-08-03T20:48:28.129073+0000","last_peered":"2024-08-03T20:48:28.129073+0000","last_clean":"2024-08-03T20:48:28.129073+0000","last_became_active":"2024-08-03T20:48:26.130265+0000","last_became_peered":"2024-08-03T20:48:26.130265+0000","last_unstale":"2024-08-03T20:48:28.129073+0000","last_undegraded":"2024-08-03T20:48:28.129073+0000","last_fullsized":"2024-08-03T20:48:28.129073+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:48:25.097349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:48:25.097349+0000","last_clean_scrub_stamp":"2024-08-03T20:48:25.097349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-04T21:04:30.787042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":13,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-08-03T20:48:28.424249+0000","last_change":"2024-08-03T20:48:27.143587+0000","last_active":"2024-08-03T20:48:28.424249+0000","last_peered":"2024-08-03T20:48:28.424249+0000","last_clean":"2024-08-03T20:48:28.424249+0000","last_became_active":"2024-08-03T20:48:27.143257+0000","last_became_peered":"2024-08-03T20:48:27.143257+0000","last_unstale":"2024-08-03T20:48:28.424249+0000","last_undegraded":"2024-08-03T20:48:28.424249+0000","last_fullsized":"2024-08-03T20:48:28.424249+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:48:26.104177+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:48:26.104177+0000","last_clean_scrub_stamp":"2024-08-03T20:48:26.104177+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-04T20:57:36.715292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-08-03T20:48:28.422171+0000","last_change":"2024-08-03T20:48:26.138189+0000","last_active":"2024-08-03T20:48:28.422171+0000","last_peered":"2024-08-03T20:48:28.422171+0000","last_clean":"2024-08-03T20:48:28.422171+0000","last_became_active":"2024-08-03T20:48:26.136878+0000","last_became_peered":"2024-08-03T20:48:26.136878+0000","last_unstale":"2024-08-03T20:48:28.422171+0000","last_undegraded":"2024-08-03T20:48:28.422171+0000","last_fullsized":"2024-08-03T20:48:28.422171+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:48:25.097349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:48:25.097349+0000","last_clean_scrub_stamp":"2024-08-03T20:48:25.097349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T08:12:43.894277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":13,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-08-03T20:48:28.128318+0000","last_change":"2024-08-03T20:48:27.143599+0000","last_active":"2024-08-03T20:48:28.128318+0000","last_peered":"2024-08-03T20:48:28.128318+0000","last_clean":"2024-08-03T20:48:28.128318+0000","last_became_active":"2024-08-03T20:48:27.143037+0000","last_became_peered":"2024-08-03T20:48:27.143037+0000","last_unstale":"2024-08-03T20:48:28.128318+0000","last_undegraded":"2024-08-03T20:48:28.128318+0000","last_fullsized":"2024-08-03T20:48:28.128318+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:48:26.104177+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:48:26.104177+0000","last_clean_scrub_stamp":"2024-08-03T20:48:26.104177+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T03:02:47.933268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"31'2","reported_seq":17,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-08-03T20:48:28.145742+0000","last_change":"2024-08-03T20:48:26.140287+0000","last_active":"2024-08-03T20:48:28.145742+0000","last_peered":"2024-08-03T20:48:28.145742+0000","last_clean":"2024-08-03T20:48:28.145742+0000","last_became_active":"2024-08-03T20:48:26.139353+0000","last_became_peered":"2024-08-03T20:48:26.139353+0000","last_unstale":"2024-08-03T20:48:28.145742+0000","last_undegraded":"2024-08-03T20:48:28.145742+0000","last_fullsized":"2024-08-03T20:48:28.145742+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:48:25.097349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:48:25.097349+0000","last_clean_scrub_stamp":"2024-08-03T20:48:25.097349+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T08:48:23.338457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":13,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-08-03T20:48:28.423872+0000","last_change":"2024-08-03T20:48:27.159376+0000","last_active":"2024-08-03T20:48:28.423872+0000","last_peered":"2024-08-03T20:48:28.423872+0000","last_clean":"2024-08-03T20:48:28.423872+0000","last_became_active":"2024-08-03T20:48:27.158924+0000","last_became_peered":"2024-08-03T20:48:27.158924+0000","last_unstale":"2024-08-03T20:48:28.423872+0000","last_undegraded":"2024-08-03T20:48:28.423872+0000","last_fullsized":"2024-08-03T20:48:28.423872+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:48:26.104177+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:48:26.104177+0000","last_clean_scrub_stamp":"2024-08-03T20:48:26.104177+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T03:32:32.293255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.6","version":"32'1","reported_seq":16,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-08-03T20:48:28.144168+0000","last_change":"2024-08-03T20:48:26.140418+0000","last_active":"2024-08-03T20:48:28.144168+0000","last_peered":"2024-08-03T20:48:28.144168+0000","last_clean":"2024-08-03T20:48:28.144168+0000","last_became_active":"2024-08-03T20:48:26.139836+0000","last_became_peered":"2024-08-03T20:48:26.139836+0000","last_unstale":"2024-08-03T20:48:28.144168+0000","last_undegraded":"2024-08-03T20:48:28.144168+0000","last_fullsized":"2024-08-03T20:48:28.144168+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:48:25.097349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:48:25.097349+0000","last_clean_scrub_stamp":"2024-08-03T20:48:25.097349+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T08:48:23.338457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":13,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-08-03T20:48:28.142253+0000","last_change":"2024-08-03T20:48:27.136979+0000","last_active":"2024-08-03T20:48:28.142253+0000","last_peered":"2024-08-03T20:48:28.142253+0000","last_clean":"2024-08-03T20:48:28.142253+0000","last_became_active":"2024-08-03T20:48:27.136666+0000","last_became_peered":"2024-08-03T20:48:27.136666+0000","last_unstale":"2024-08-03T20:48:28.142253+0000","last_undegraded":"2024-08-03T20:48:28.142253+0000","last_fullsized":"2024-08-03T20:48:28.142253+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:48:26.104177+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:48:26.104177+0000","last_clean_scrub_stamp":"2024-08-03T20:48:26.104177+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-04T21:58:23.025892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-08-03T20:48:28.145298+0000","last_change":"2024-08-03T20:48:26.132240+0000","last_active":"2024-08-03T20:48:28.145298+0000","last_peered":"2024-08-03T20:48:28.145298+0000","last_clean":"2024-08-03T20:48:28.145298+0000","last_became_active":"2024-08-03T20:48:26.131663+0000","last_became_peered":"2024-08-03T20:48:26.131663+0000","last_unstale":"2024-08-03T20:48:28.145298+0000","last_undegraded":"2024-08-03T20:48:28.145298+0000","last_fullsized":"2024-08-03T20:48:28.145298+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:48:25.097349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:48:25.097349+0000","last_clean_scrub_stamp":"2024-08-03T20:48:25.097349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T00:15:51.465753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":13,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-08-03T20:48:28.422397+0000","last_change":"2024-08-03T20:48:27.140382+0000","last_active":"2024-08-03T20:48:28.422397+0000","last_peered":"2024-08-03T20:48:28.422397+0000","last_clean":"2024-08-03T20:48:28.422397+0000","last_became_active":"2024-08-03T20:48:27.139818+0000","last_became_peered":"2024-08-03T20:48:27.139818+0000","last_unstale":"2024-08-03T20:48:28.422397+0000","last_undegraded":"2024-08-03T20:48:28.422397+0000","last_fullsized":"2024-08-03T20:48:28.422397+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:48:26.104177+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:48:26.104177+0000","last_clean_scrub_stamp":"2024-08-03T20:48:26.104177+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T08:12:44.901105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-08-03T20:48:28.142414+0000","last_change":"2024-08-03T20:48:26.133848+0000","last_active":"2024-08-03T20:48:28.142414+0000","last_peered":"2024-08-03T20:48:28.142414+0000","last_clean":"2024-08-03T20:48:28.142414+0000","last_became_active":"2024-08-03T20:48:26.133510+0000","last_became_peered":"2024-08-03T20:48:26.133510+0000","last_unstale":"2024-08-03T20:48:28.142414+0000","last_undegraded":"2024-08-03T20:48:28.142414+0000","last_fullsized":"2024-08-03T20:48:28.142414+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:48:25.097349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:48:25.097349+0000","last_clean_scrub_stamp":"2024-08-03T20:48:25.097349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T00:15:51.465753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":13,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-08-03T20:48:28.142352+0000","last_change":"2024-08-03T20:48:27.147280+0000","last_active":"2024-08-03T20:48:28.142352+0000","last_peered":"2024-08-03T20:48:28.142352+0000","last_clean":"2024-08-03T20:48:28.142352+0000","last_became_active":"2024-08-03T20:48:27.143350+0000","last_became_peered":"2024-08-03T20:48:27.143350+0000","last_unstale":"2024-08-03T20:48:28.142352+0000","last_undegraded":"2024-08-03T20:48:28.142352+0000","last_fullsized":"2024-08-03T20:48:28.142352+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:48:26.104177+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:48:26.104177+0000","last_clean_scrub_stamp":"2024-08-03T20:48:26.104177+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T03:20:15.238833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-08-03T20:48:28.145353+0000","last_change":"2024-08-03T20:48:26.133691+0000","last_active":"2024-08-03T20:48:28.145353+0000","last_peered":"2024-08-03T20:48:28.145353+0000","last_clean":"2024-08-03T20:48:28.145353+0000","last_became_active":"2024-08-03T20:48:26.133341+0000","last_became_peered":"2024-08-03T20:48:26.133341+0000","last_unstale":"2024-08-03T20:48:28.145353+0000","last_undegraded":"2024-08-03T20:48:28.145353+0000","last_fullsized":"2024-08-03T20:48:28.145353+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:48:25.097349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:48:25.097349+0000","last_clean_scrub_stamp":"2024-08-03T20:48:25.097349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T01:34:49.869344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":13,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-08-03T20:48:28.421440+0000","last_change":"2024-08-03T20:48:27.137770+0000","last_active":"2024-08-03T20:48:28.421440+0000","last_peered":"2024-08-03T20:48:28.421440+0000","last_clean":"2024-08-03T20:48:28.421440+0000","last_became_active":"2024-08-03T20:48:27.137438+0000","last_became_peered":"2024-08-03T20:48:27.137438+0000","last_unstale":"2024-08-03T20:48:28.421440+0000","last_undegraded":"2024-08-03T20:48:28.421440+0000","last_fullsized":"2024-08-03T20:48:28.421440+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:48:26.104177+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:48:26.104177+0000","last_clean_scrub_stamp":"2024-08-03T20:48:26.104177+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-04T20:52:50.166997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-08-03T20:48:28.129220+0000","last_change":"2024-08-03T20:48:26.135756+0000","last_active":"2024-08-03T20:48:28.129220+0000","last_peered":"2024-08-03T20:48:28.129220+0000","last_clean":"2024-08-03T20:48:28.129220+0000","last_became_active":"2024-08-03T20:48:26.134565+0000","last_became_peered":"2024-08-03T20:48:26.134565+0000","last_unstale":"2024-08-03T20:48:28.129220+0000","last_undegraded":"2024-08-03T20:48:28.129220+0000","last_fullsized":"2024-08-03T20:48:28.129220+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:48:25.097349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:48:25.097349+0000","last_clean_scrub_stamp":"2024-08-03T20:48:25.097349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-04T22:13:46.608900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":13,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-08-03T20:48:28.129094+0000","last_change":"2024-08-03T20:48:27.128616+0000","last_active":"2024-08-03T20:48:28.129094+0000","last_peered":"2024-08-03T20:48:28.129094+0000","last_clean":"2024-08-03T20:48:28.129094+0000","last_became_active":"2024-08-03T20:48:27.128257+0000","last_became_peered":"2024-08-03T20:48:27.128257+0000","last_unstale":"2024-08-03T20:48:28.129094+0000","last_undegraded":"2024-08-03T20:48:28.129094+0000","last_fullsized":"2024-08-03T20:48:28.129094+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:48:26.104177+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:48:26.104177+0000","last_clean_scrub_stamp":"2024-08-03T20:48:26.104177+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T03:27:54.323107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-08-03T20:48:28.421552+0000","last_change":"2024-08-03T20:48:26.143986+0000","last_active":"2024-08-03T20:48:28.421552+0000","last_peered":"2024-08-03T20:48:28.421552+0000","last_clean":"2024-08-03T20:48:28.421552+0000","last_became_active":"2024-08-03T20:48:26.143665+0000","last_became_peered":"2024-08-03T20:48:26.143665+0000","last_unstale":"2024-08-03T20:48:28.421552+0000","last_undegraded":"2024-08-03T20:48:28.421552+0000","last_fullsized":"2024-08-03T20:48:28.421552+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:48:25.097349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:48:25.097349+0000","last_clean_scrub_stamp":"2024-08-03T20:48:25.097349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T06:38:55.256186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":13,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-08-03T20:48:28.424416+0000","last_change":"2024-08-03T20:48:27.145565+0000","last_active":"2024-08-03T20:48:28.424416+0000","last_peered":"2024-08-03T20:48:28.424416+0000","last_clean":"2024-08-03T20:48:28.424416+0000","last_became_active":"2024-08-03T20:48:27.144559+0000","last_became_peered":"2024-08-03T20:48:27.144559+0000","last_unstale":"2024-08-03T20:48:28.424416+0000","last_undegraded":"2024-08-03T20:48:28.424416+0000","last_fullsized":"2024-08-03T20:48:28.424416+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:48:26.104177+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:48:26.104177+0000","last_clean_scrub_stamp":"2024-08-03T20:48:26.104177+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-04T22:01:58.560189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-08-03T20:48:28.422263+0000","last_change":"2024-08-03T20:48:26.142596+0000","last_active":"2024-08-03T20:48:28.422263+0000","last_peered":"2024-08-03T20:48:28.422263+0000","last_clean":"2024-08-03T20:48:28.422263+0000","last_became_active":"2024-08-03T20:48:26.142251+0000","last_became_peered":"2024-08-03T20:48:26.142251+0000","last_unstale":"2024-08-03T20:48:28.422263+0000","last_undegraded":"2024-08-03T20:48:28.422263+0000","last_fullsized":"2024-08-03T20:48:28.422263+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:48:25.097349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:48:25.097349+0000","last_clean_scrub_stamp":"2024-08-03T20:48:25.097349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T03:23:20.675312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"16'72","reported_seq":129,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-08-03T20:48:28.424169+0000","last_change":"2024-08-03T20:47:30.547632+0000","last_active":"2024-08-03T20:48:28.424169+0000","last_peered":"2024-08-03T20:48:28.424169+0000","last_clean":"2024-08-03T20:48:28.424169+0000","last_became_active":"2024-08-03T20:47:30.546255+0000","last_became_peered":"2024-08-03T20:47:30.546255+0000","last_unstale":"2024-08-03T20:48:28.424169+0000","last_undegraded":"2024-08-03T20:48:28.424169+0000","last_fullsized":"2024-08-03T20:48:28.424169+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:47:28.874451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:47:28.874451+0000","last_clean_scrub_stamp":"2024-08-03T20:47:28.874451+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T00:19:40.548812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509455,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32112,"kb_used_data":928,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339728,"statfs":{"total":96636764160,"available":96603881472,"internally_reserved":0,"allocated":950272,"data_stored":726191,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":6,"up_from":15,"seq":64424509456,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31532,"kb_used_data":352,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340308,"statfs":{"total":96636764160,"available":96604475392,"internally_reserved":0,"allocated":360448,"data_stored":136349,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":5,"up_from":15,"seq":64424509456,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31532,"kb_used_data":352,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340308,"statfs":{"total":96636764160,"available":96604475392,"internally_reserved":0,"allocated":360448,"data_stored":136405,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509455,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31540,"kb_used_data":360,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340300,"statfs":{"total":96636764160,"available":96604467200,"internally_reserved":0,"allocated":368640,"data_stored":137367,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509457,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32108,"kb_used_data":924,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339732,"statfs":{"total":96636764160,"available":96603885568,"internally_reserved":0,"allocated":946176,"data_stored":726101,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542160,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31520,"kb_used_data":340,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340320,"statfs":{"total":96636764160,"available":96604487680,"internally_reserved":0,"allocated":348160,"data_stored":129537,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":3,"up_from":14,"seq":60129542158,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31528,"kb_used_data":348,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340312,"statfs":{"total":96636764160,"available":96604479488,"internally_reserved":0,"allocated":356352,"data_stored":131067,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":2,"up_from":14,"seq":60129542159,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31532,"kb_used_data":348,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340308,"statfs":{"total":96636764160,"available":96604475392,"internally_reserved":0,"allocated":356352,"data_stored":135803,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-08-03T20:48:31.638 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-08-03T20:48:31.901 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:31.899+0000 7f0f53c00640 1 -- 172.21.15.32:0/3496402256 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0f4c06e930 msgr2=0x7f0f4c144f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:48:31.901 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:31.899+0000 7f0f53c00640 1 --2- 172.21.15.32:0/3496402256 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0f4c06e930 0x7f0f4c144f90 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f0f4c0afaf0 tx=0x7f0f4002a560 comp rx=0 tx=0).stop 2024-08-03T20:48:31.901 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:31.899+0000 7f0f53c00640 1 -- 172.21.15.32:0/3496402256 shutdown_connections 2024-08-03T20:48:31.901 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:31.899+0000 7f0f53c00640 1 --2- 172.21.15.32:0/3496402256 >> v2:172.21.15.118:3300/0 conn(0x7f0f4c073e60 0x7f0f4c074270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:31.901 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:31.899+0000 7f0f53c00640 1 --2- 172.21.15.32:0/3496402256 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0f4c06e930 0x7f0f4c144f90 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:31.901 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:31.899+0000 7f0f53c00640 1 -- 172.21.15.32:0/3496402256 >> 172.21.15.32:0/3496402256 conn(0x7f0f4c0b1bd0 msgr2=0x7f0f4c0b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:48:31.901 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:31.899+0000 7f0f53c00640 1 -- 172.21.15.32:0/3496402256 shutdown_connections 2024-08-03T20:48:31.901 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:31.899+0000 7f0f53c00640 1 -- 172.21.15.32:0/3496402256 wait complete. 2024-08-03T20:48:31.902 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:31.901+0000 7f0f53c00640 1 Processor -- start 2024-08-03T20:48:31.902 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:31.901+0000 7f0f53c00640 1 -- start start 2024-08-03T20:48:31.905 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:31.903+0000 7f0f53c00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0f4c06e930 0x7f0f4c1f63b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:48:31.905 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:31.903+0000 7f0f53c00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f0f4c073e60 0x7f0f4c15f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:48:31.905 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:31.903+0000 7f0f53c00640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f0f4c0747b0 0x7f0f4c0b2fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:48:31.905 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:31.903+0000 7f0f53c00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f0f4c0cf9c0 con 0x7f0f4c06e930 2024-08-03T20:48:31.905 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:31.903+0000 7f0f53c00640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f0f4c0cf840 con 0x7f0f4c073e60 2024-08-03T20:48:31.905 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:31.903+0000 7f0f53c00640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7f0f4c0ccff0 con 0x7f0f4c0747b0 2024-08-03T20:48:31.905 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:31.903+0000 7f0f51200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0f4c06e930 0x7f0f4c1f63b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:48:31.906 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:31.903+0000 7f0f51200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0f4c06e930 0x7f0f4c1f63b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:46482/0 (socket says 172.21.15.32:46482) 2024-08-03T20:48:31.906 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:31.903+0000 7f0f51200640 1 -- 172.21.15.32:0/313317351 learned_addr learned my addr 172.21.15.32:0/313317351 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:48:31.906 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:31.903+0000 7f0f4be00640 1 --2- 172.21.15.32:0/313317351 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f0f4c073e60 0x7f0f4c15f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:48:31.906 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:31.903+0000 7f0f51c00640 1 --2- 172.21.15.32:0/313317351 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f0f4c0747b0 0x7f0f4c0b2fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:48:31.906 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:31.903+0000 7f0f51200640 1 -- 172.21.15.32:0/313317351 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f0f4c0747b0 msgr2=0x7f0f4c0b2fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:48:31.906 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:31.903+0000 7f0f51200640 1 --2- 172.21.15.32:0/313317351 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f0f4c0747b0 0x7f0f4c0b2fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:31.906 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:31.903+0000 7f0f51200640 1 -- 172.21.15.32:0/313317351 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f0f4c073e60 msgr2=0x7f0f4c15f0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:48:31.906 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:31.903+0000 7f0f51200640 1 --2- 172.21.15.32:0/313317351 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f0f4c073e60 0x7f0f4c15f0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:31.906 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:31.903+0000 7f0f51200640 1 -- 172.21.15.32:0/313317351 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0f4c1f57c0 con 0x7f0f4c06e930 2024-08-03T20:48:31.906 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:31.903+0000 7f0f4be00640 1 --2- 172.21.15.32:0/313317351 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f0f4c073e60 0x7f0f4c15f0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-03T20:48:31.906 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:31.903+0000 7f0f51c00640 1 --2- 172.21.15.32:0/313317351 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f0f4c0747b0 0x7f0f4c0b2fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-03T20:48:31.906 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:31.904+0000 7f0f51200640 1 --2- 172.21.15.32:0/313317351 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0f4c06e930 0x7f0f4c1f63b0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f0f40034040 tx=0x7f0f40003910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:48:31.907 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:31.904+0000 7f0f48c00640 1 -- 172.21.15.32:0/313317351 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f4003e030 con 0x7f0f4c06e930 2024-08-03T20:48:31.907 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:31.904+0000 7f0f48c00640 1 -- 172.21.15.32:0/313317351 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f0f40003e00 con 0x7f0f4c06e930 2024-08-03T20:48:31.907 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:31.904+0000 7f0f48c00640 1 -- 172.21.15.32:0/313317351 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f40035040 con 0x7f0f4c06e930 2024-08-03T20:48:31.907 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:31.904+0000 7f0f53c00640 1 -- 172.21.15.32:0/313317351 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0f4c0ae5f0 con 0x7f0f4c06e930 2024-08-03T20:48:31.907 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:31.904+0000 7f0f53c00640 1 -- 172.21.15.32:0/313317351 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0f4c0aebb0 con 0x7f0f4c06e930 2024-08-03T20:48:31.908 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:31.905+0000 7f0f53c00640 1 -- 172.21.15.32:0/313317351 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0f4c06ab00 con 0x7f0f4c06e930 2024-08-03T20:48:31.908 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:31.906+0000 7f0f48c00640 1 -- 172.21.15.32:0/313317351 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f0f40041030 con 0x7f0f4c06e930 2024-08-03T20:48:31.909 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:31.907+0000 7f0f48c00640 1 --2- 172.21.15.32:0/313317351 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f0f2c077040 0x7f0f2c079550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:48:31.909 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:31.907+0000 7f0f48c00640 1 -- 172.21.15.32:0/313317351 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(32..32 src has 1..32) ==== 5709+0+0 (secure 0 0 0) 0x7f0f400b90a0 con 0x7f0f4c06e930 2024-08-03T20:48:31.909 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:31.908+0000 7f0f4be00640 1 --2- 172.21.15.32:0/313317351 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f0f2c077040 0x7f0f2c079550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:48:31.910 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:31.908+0000 7f0f4be00640 1 --2- 172.21.15.32:0/313317351 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f0f2c077040 0x7f0f2c079550 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f0f38005fd0 tx=0x7f0f3800f360 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:48:31.913 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:31.912+0000 7f0f48c00640 1 -- 172.21.15.32:0/313317351 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7f0f4c1f57c0 con 0x7f0f4c06e930 2024-08-03T20:48:32.115 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:32.114+0000 7f0f53c00640 1 -- 172.21.15.32:0/313317351 --> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f0f4c0cd580 con 0x7f0f2c077040 2024-08-03T20:48:32.121 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:32.120+0000 7f0f48c00640 1 -- 172.21.15.32:0/313317351 <== mgr.4106 v2:172.21.15.118:6800/4205950699 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+251938 (secure 0 0 0) 0x7f0f4c0cd580 con 0x7f0f2c077040 2024-08-03T20:48:32.124 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-03T20:48:32.125 INFO:teuthology.orchestra.run.smithi032.stderr:dumped all 2024-08-03T20:48:32.127 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:32.124+0000 7f0f53c00640 1 -- 172.21.15.32:0/313317351 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f0f2c077040 msgr2=0x7f0f2c079550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:48:32.127 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:32.124+0000 7f0f53c00640 1 --2- 172.21.15.32:0/313317351 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f0f2c077040 0x7f0f2c079550 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f0f38005fd0 tx=0x7f0f3800f360 comp rx=0 tx=0).stop 2024-08-03T20:48:32.127 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:32.124+0000 7f0f53c00640 1 -- 172.21.15.32:0/313317351 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0f4c06e930 msgr2=0x7f0f4c1f63b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:48:32.127 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:32.124+0000 7f0f53c00640 1 --2- 172.21.15.32:0/313317351 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0f4c06e930 0x7f0f4c1f63b0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f0f40034040 tx=0x7f0f40003910 comp rx=0 tx=0).stop 2024-08-03T20:48:32.127 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:32.124+0000 7f0f53c00640 1 -- 172.21.15.32:0/313317351 shutdown_connections 2024-08-03T20:48:32.127 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:32.124+0000 7f0f53c00640 1 --2- 172.21.15.32:0/313317351 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f0f2c077040 0x7f0f2c079550 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:32.127 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:32.124+0000 7f0f53c00640 1 --2- 172.21.15.32:0/313317351 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f0f4c073e60 0x7f0f4c15f0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:32.127 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:32.124+0000 7f0f53c00640 1 --2- 172.21.15.32:0/313317351 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f0f4c0747b0 0x7f0f4c0b2fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:32.127 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:32.124+0000 7f0f53c00640 1 --2- 172.21.15.32:0/313317351 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0f4c06e930 0x7f0f4c1f63b0 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:32.128 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:32.124+0000 7f0f53c00640 1 -- 172.21.15.32:0/313317351 >> 172.21.15.32:0/313317351 conn(0x7f0f4c0b1bd0 msgr2=0x7f0f4c140cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:48:32.128 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:32.124+0000 7f0f53c00640 1 -- 172.21.15.32:0/313317351 shutdown_connections 2024-08-03T20:48:32.128 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:32.124+0000 7f0f53c00640 1 -- 172.21.15.32:0/313317351 wait complete. 2024-08-03T20:48:32.150 INFO:teuthology.orchestra.run.smithi032.stdout:{"pg_ready":true,"pg_map":{"version":56,"stamp":"2024-08-03T20:48:30.235728+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":146,"num_write_kb":1642,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":111,"ondisk_log_size":111,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":140,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":253404,"kb_used_data":3952,"kb_used_omap":12,"kb_used_meta":249331,"kb_avail":754721316,"statfs":{"total":773094113280,"available":772834627584,"internally_reserved":0,"allocated":4046848,"data_stored":2258820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":255315505},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":28,"apply_latency_ms":28,"commit_latency_ns":28000000,"apply_latency_ns":28000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.131469"},"pg_stats":[{"pgid":"5.3f","version":"0'0","reported_seq":13,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-08-03T20:48:28.422512+0000","last_change":"2024-08-03T20:48:27.143195+0000","last_active":"2024-08-03T20:48:28.422512+0000","last_peered":"2024-08-03T20:48:28.422512+0000","last_clean":"2024-08-03T20:48:28.422512+0000","last_became_active":"2024-08-03T20:48:27.142841+0000","last_became_peered":"2024-08-03T20:48:27.142841+0000","last_unstale":"2024-08-03T20:48:28.422512+0000","last_undegraded":"2024-08-03T20:48:28.422512+0000","last_fullsized":"2024-08-03T20:48:28.422512+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:48:26.104177+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:48:26.104177+0000","last_clean_scrub_stamp":"2024-08-03T20:48:26.104177+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-04T23:17:51.269394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.3e","version":"0'0","reported_seq":13,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-08-03T20:48:28.422242+0000","last_change":"2024-08-03T20:48:27.147259+0000","last_active":"2024-08-03T20:48:28.422242+0000","last_peered":"2024-08-03T20:48:28.422242+0000","last_clean":"2024-08-03T20:48:28.422242+0000","last_became_active":"2024-08-03T20:48:27.146980+0000","last_became_peered":"2024-08-03T20:48:27.146980+0000","last_unstale":"2024-08-03T20:48:28.422242+0000","last_undegraded":"2024-08-03T20:48:28.422242+0000","last_fullsized":"2024-08-03T20:48:28.422242+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:48:26.104177+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:48:26.104177+0000","last_clean_scrub_stamp":"2024-08-03T20:48:26.104177+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-04T23:29:10.771944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.3d","version":"0'0","reported_seq":13,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-08-03T20:48:28.423560+0000","last_change":"2024-08-03T20:48:27.159194+0000","last_active":"2024-08-03T20:48:28.423560+0000","last_peered":"2024-08-03T20:48:28.423560+0000","last_clean":"2024-08-03T20:48:28.423560+0000","last_became_active":"2024-08-03T20:48:27.158826+0000","last_became_peered":"2024-08-03T20:48:27.158826+0000","last_unstale":"2024-08-03T20:48:28.423560+0000","last_undegraded":"2024-08-03T20:48:28.423560+0000","last_fullsized":"2024-08-03T20:48:28.423560+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:48:26.104177+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:48:26.104177+0000","last_clean_scrub_stamp":"2024-08-03T20:48:26.104177+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-04T20:57:36.715292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3c","version":"0'0","reported_seq":13,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-08-03T20:48:28.145230+0000","last_change":"2024-08-03T20:48:27.138551+0000","last_active":"2024-08-03T20:48:28.145230+0000","last_peered":"2024-08-03T20:48:28.145230+0000","last_clean":"2024-08-03T20:48:28.145230+0000","last_became_active":"2024-08-03T20:48:27.138162+0000","last_became_peered":"2024-08-03T20:48:27.138162+0000","last_unstale":"2024-08-03T20:48:28.145230+0000","last_undegraded":"2024-08-03T20:48:28.145230+0000","last_fullsized":"2024-08-03T20:48:28.145230+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:48:26.104177+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:48:26.104177+0000","last_clean_scrub_stamp":"2024-08-03T20:48:26.104177+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T05:00:43.122455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3b","version":"0'0","reported_seq":13,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-08-03T20:48:28.127999+0000","last_change":"2024-08-03T20:48:27.143928+0000","last_active":"2024-08-03T20:48:28.127999+0000","last_peered":"2024-08-03T20:48:28.127999+0000","last_clean":"2024-08-03T20:48:28.127999+0000","last_became_active":"2024-08-03T20:48:27.143678+0000","last_became_peered":"2024-08-03T20:48:27.143678+0000","last_unstale":"2024-08-03T20:48:28.127999+0000","last_undegraded":"2024-08-03T20:48:28.127999+0000","last_fullsized":"2024-08-03T20:48:28.127999+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:48:26.104177+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:48:26.104177+0000","last_clean_scrub_stamp":"2024-08-03T20:48:26.104177+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T07:04:00.770525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3a","version":"0'0","reported_seq":13,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-08-03T20:48:28.421562+0000","last_change":"2024-08-03T20:48:27.150753+0000","last_active":"2024-08-03T20:48:28.421562+0000","last_peered":"2024-08-03T20:48:28.421562+0000","last_clean":"2024-08-03T20:48:28.421562+0000","last_became_active":"2024-08-03T20:48:27.150231+0000","last_became_peered":"2024-08-03T20:48:27.150231+0000","last_unstale":"2024-08-03T20:48:28.421562+0000","last_undegraded":"2024-08-03T20:48:28.421562+0000","last_fullsized":"2024-08-03T20:48:28.421562+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:48:26.104177+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:48:26.104177+0000","last_clean_scrub_stamp":"2024-08-03T20:48:26.104177+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T05:38:08.118990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.39","version":"0'0","reported_seq":13,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-08-03T20:48:28.424356+0000","last_change":"2024-08-03T20:48:27.143187+0000","last_active":"2024-08-03T20:48:28.424356+0000","last_peered":"2024-08-03T20:48:28.424356+0000","last_clean":"2024-08-03T20:48:28.424356+0000","last_became_active":"2024-08-03T20:48:27.142847+0000","last_became_peered":"2024-08-03T20:48:27.142847+0000","last_unstale":"2024-08-03T20:48:28.424356+0000","last_undegraded":"2024-08-03T20:48:28.424356+0000","last_fullsized":"2024-08-03T20:48:28.424356+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:48:26.104177+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:48:26.104177+0000","last_clean_scrub_stamp":"2024-08-03T20:48:26.104177+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T03:32:32.293255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.38","version":"0'0","reported_seq":13,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-08-03T20:48:28.423300+0000","last_change":"2024-08-03T20:48:27.135374+0000","last_active":"2024-08-03T20:48:28.423300+0000","last_peered":"2024-08-03T20:48:28.423300+0000","last_clean":"2024-08-03T20:48:28.423300+0000","last_became_active":"2024-08-03T20:48:27.133427+0000","last_became_peered":"2024-08-03T20:48:27.133427+0000","last_unstale":"2024-08-03T20:48:28.423300+0000","last_undegraded":"2024-08-03T20:48:28.423300+0000","last_fullsized":"2024-08-03T20:48:28.423300+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:48:26.104177+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:48:26.104177+0000","last_clean_scrub_stamp":"2024-08-03T20:48:26.104177+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T08:15:08.218151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.37","version":"0'0","reported_seq":13,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-08-03T20:48:28.130110+0000","last_change":"2024-08-03T20:48:27.130570+0000","last_active":"2024-08-03T20:48:28.130110+0000","last_peered":"2024-08-03T20:48:28.130110+0000","last_clean":"2024-08-03T20:48:28.130110+0000","last_became_active":"2024-08-03T20:48:27.130232+0000","last_became_peered":"2024-08-03T20:48:27.130232+0000","last_unstale":"2024-08-03T20:48:28.130110+0000","last_undegraded":"2024-08-03T20:48:28.130110+0000","last_fullsized":"2024-08-03T20:48:28.130110+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:48:26.104177+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:48:26.104177+0000","last_clean_scrub_stamp":"2024-08-03T20:48:26.104177+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-04T23:03:19.113492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.36","version":"0'0","reported_seq":13,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-08-03T20:48:28.145487+0000","last_change":"2024-08-03T20:48:27.153541+0000","last_active":"2024-08-03T20:48:28.145487+0000","last_peered":"2024-08-03T20:48:28.145487+0000","last_clean":"2024-08-03T20:48:28.145487+0000","last_became_active":"2024-08-03T20:48:27.153246+0000","last_became_peered":"2024-08-03T20:48:27.153246+0000","last_unstale":"2024-08-03T20:48:28.145487+0000","last_undegraded":"2024-08-03T20:48:28.145487+0000","last_fullsized":"2024-08-03T20:48:28.145487+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:48:26.104177+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:48:26.104177+0000","last_clean_scrub_stamp":"2024-08-03T20:48:26.104177+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T06:04:52.253903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.35","version":"0'0","reported_seq":13,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-08-03T20:48:28.422884+0000","last_change":"2024-08-03T20:48:27.158184+0000","last_active":"2024-08-03T20:48:28.422884+0000","last_peered":"2024-08-03T20:48:28.422884+0000","last_clean":"2024-08-03T20:48:28.422884+0000","last_became_active":"2024-08-03T20:48:27.157907+0000","last_became_peered":"2024-08-03T20:48:27.157907+0000","last_unstale":"2024-08-03T20:48:28.422884+0000","last_undegraded":"2024-08-03T20:48:28.422884+0000","last_fullsized":"2024-08-03T20:48:28.422884+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:48:26.104177+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:48:26.104177+0000","last_clean_scrub_stamp":"2024-08-03T20:48:26.104177+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T06:09:59.767222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.34","version":"0'0","reported_seq":13,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-08-03T20:48:28.423365+0000","last_change":"2024-08-03T20:48:27.157166+0000","last_active":"2024-08-03T20:48:28.423365+0000","last_peered":"2024-08-03T20:48:28.423365+0000","last_clean":"2024-08-03T20:48:28.423365+0000","last_became_active":"2024-08-03T20:48:27.156773+0000","last_became_peered":"2024-08-03T20:48:27.156773+0000","last_unstale":"2024-08-03T20:48:28.423365+0000","last_undegraded":"2024-08-03T20:48:28.423365+0000","last_fullsized":"2024-08-03T20:48:28.423365+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:48:26.104177+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:48:26.104177+0000","last_clean_scrub_stamp":"2024-08-03T20:48:26.104177+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-04T21:00:04.257461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.33","version":"0'0","reported_seq":13,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-08-03T20:48:28.128023+0000","last_change":"2024-08-03T20:48:27.132385+0000","last_active":"2024-08-03T20:48:28.128023+0000","last_peered":"2024-08-03T20:48:28.128023+0000","last_clean":"2024-08-03T20:48:28.128023+0000","last_became_active":"2024-08-03T20:48:27.132059+0000","last_became_peered":"2024-08-03T20:48:27.132059+0000","last_unstale":"2024-08-03T20:48:28.128023+0000","last_undegraded":"2024-08-03T20:48:28.128023+0000","last_fullsized":"2024-08-03T20:48:28.128023+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:48:26.104177+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:48:26.104177+0000","last_clean_scrub_stamp":"2024-08-03T20:48:26.104177+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T01:47:24.684656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.32","version":"0'0","reported_seq":13,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-08-03T20:48:28.421773+0000","last_change":"2024-08-03T20:48:27.150846+0000","last_active":"2024-08-03T20:48:28.421773+0000","last_peered":"2024-08-03T20:48:28.421773+0000","last_clean":"2024-08-03T20:48:28.421773+0000","last_became_active":"2024-08-03T20:48:27.150498+0000","last_became_peered":"2024-08-03T20:48:27.150498+0000","last_unstale":"2024-08-03T20:48:28.421773+0000","last_undegraded":"2024-08-03T20:48:28.421773+0000","last_fullsized":"2024-08-03T20:48:28.421773+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:48:26.104177+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:48:26.104177+0000","last_clean_scrub_stamp":"2024-08-03T20:48:26.104177+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T06:21:28.181030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.31","version":"0'0","reported_seq":13,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-08-03T20:48:28.421970+0000","last_change":"2024-08-03T20:48:27.146235+0000","last_active":"2024-08-03T20:48:28.421970+0000","last_peered":"2024-08-03T20:48:28.421970+0000","last_clean":"2024-08-03T20:48:28.421970+0000","last_became_active":"2024-08-03T20:48:27.145486+0000","last_became_peered":"2024-08-03T20:48:27.145486+0000","last_unstale":"2024-08-03T20:48:28.421970+0000","last_undegraded":"2024-08-03T20:48:28.421970+0000","last_fullsized":"2024-08-03T20:48:28.421970+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:48:26.104177+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:48:26.104177+0000","last_clean_scrub_stamp":"2024-08-03T20:48:26.104177+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T03:46:54.747620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.30","version":"0'0","reported_seq":13,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-08-03T20:48:28.424085+0000","last_change":"2024-08-03T20:48:27.141138+0000","last_active":"2024-08-03T20:48:28.424085+0000","last_peered":"2024-08-03T20:48:28.424085+0000","last_clean":"2024-08-03T20:48:28.424085+0000","last_became_active":"2024-08-03T20:48:27.138995+0000","last_became_peered":"2024-08-03T20:48:27.138995+0000","last_unstale":"2024-08-03T20:48:28.424085+0000","last_undegraded":"2024-08-03T20:48:28.424085+0000","last_fullsized":"2024-08-03T20:48:28.424085+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:48:26.104177+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:48:26.104177+0000","last_clean_scrub_stamp":"2024-08-03T20:48:26.104177+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T04:17:36.047269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2f","version":"0'0","reported_seq":13,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-08-03T20:48:28.423115+0000","last_change":"2024-08-03T20:48:27.145592+0000","last_active":"2024-08-03T20:48:28.423115+0000","last_peered":"2024-08-03T20:48:28.423115+0000","last_clean":"2024-08-03T20:48:28.423115+0000","last_became_active":"2024-08-03T20:48:27.145280+0000","last_became_peered":"2024-08-03T20:48:27.145280+0000","last_unstale":"2024-08-03T20:48:28.423115+0000","last_undegraded":"2024-08-03T20:48:28.423115+0000","last_fullsized":"2024-08-03T20:48:28.423115+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:48:26.104177+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:48:26.104177+0000","last_clean_scrub_stamp":"2024-08-03T20:48:26.104177+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T00:06:25.299214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2e","version":"0'0","reported_seq":13,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-08-03T20:48:28.422310+0000","last_change":"2024-08-03T20:48:27.158905+0000","last_active":"2024-08-03T20:48:28.422310+0000","last_peered":"2024-08-03T20:48:28.422310+0000","last_clean":"2024-08-03T20:48:28.422310+0000","last_became_active":"2024-08-03T20:48:27.158502+0000","last_became_peered":"2024-08-03T20:48:27.158502+0000","last_unstale":"2024-08-03T20:48:28.422310+0000","last_undegraded":"2024-08-03T20:48:28.422310+0000","last_fullsized":"2024-08-03T20:48:28.422310+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:48:26.104177+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:48:26.104177+0000","last_clean_scrub_stamp":"2024-08-03T20:48:26.104177+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T01:33:26.863109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2d","version":"0'0","reported_seq":13,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-08-03T20:48:28.421984+0000","last_change":"2024-08-03T20:48:27.136832+0000","last_active":"2024-08-03T20:48:28.421984+0000","last_peered":"2024-08-03T20:48:28.421984+0000","last_clean":"2024-08-03T20:48:28.421984+0000","last_became_active":"2024-08-03T20:48:27.136381+0000","last_became_peered":"2024-08-03T20:48:27.136381+0000","last_unstale":"2024-08-03T20:48:28.421984+0000","last_undegraded":"2024-08-03T20:48:28.421984+0000","last_fullsized":"2024-08-03T20:48:28.421984+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:48:26.104177+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:48:26.104177+0000","last_clean_scrub_stamp":"2024-08-03T20:48:26.104177+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-04T22:37:06.699449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2c","version":"0'0","reported_seq":13,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-08-03T20:48:28.128153+0000","last_change":"2024-08-03T20:48:27.143611+0000","last_active":"2024-08-03T20:48:28.128153+0000","last_peered":"2024-08-03T20:48:28.128153+0000","last_clean":"2024-08-03T20:48:28.128153+0000","last_became_active":"2024-08-03T20:48:27.143009+0000","last_became_peered":"2024-08-03T20:48:27.143009+0000","last_unstale":"2024-08-03T20:48:28.128153+0000","last_undegraded":"2024-08-03T20:48:28.128153+0000","last_fullsized":"2024-08-03T20:48:28.128153+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:48:26.104177+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:48:26.104177+0000","last_clean_scrub_stamp":"2024-08-03T20:48:26.104177+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T01:06:57.219548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.2b","version":"0'0","reported_seq":13,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-08-03T20:48:28.129456+0000","last_change":"2024-08-03T20:48:27.134346+0000","last_active":"2024-08-03T20:48:28.129456+0000","last_peered":"2024-08-03T20:48:28.129456+0000","last_clean":"2024-08-03T20:48:28.129456+0000","last_became_active":"2024-08-03T20:48:27.133645+0000","last_became_peered":"2024-08-03T20:48:27.133645+0000","last_unstale":"2024-08-03T20:48:28.129456+0000","last_undegraded":"2024-08-03T20:48:28.129456+0000","last_fullsized":"2024-08-03T20:48:28.129456+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:48:26.104177+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:48:26.104177+0000","last_clean_scrub_stamp":"2024-08-03T20:48:26.104177+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T07:41:43.044274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.2a","version":"0'0","reported_seq":13,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-08-03T20:48:28.129427+0000","last_change":"2024-08-03T20:48:27.131250+0000","last_active":"2024-08-03T20:48:28.129427+0000","last_peered":"2024-08-03T20:48:28.129427+0000","last_clean":"2024-08-03T20:48:28.129427+0000","last_became_active":"2024-08-03T20:48:27.129718+0000","last_became_peered":"2024-08-03T20:48:27.129718+0000","last_unstale":"2024-08-03T20:48:28.129427+0000","last_undegraded":"2024-08-03T20:48:28.129427+0000","last_fullsized":"2024-08-03T20:48:28.129427+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:48:26.104177+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:48:26.104177+0000","last_clean_scrub_stamp":"2024-08-03T20:48:26.104177+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-04T21:10:14.020760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.29","version":"0'0","reported_seq":13,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-08-03T20:48:28.142100+0000","last_change":"2024-08-03T20:48:27.132413+0000","last_active":"2024-08-03T20:48:28.142100+0000","last_peered":"2024-08-03T20:48:28.142100+0000","last_clean":"2024-08-03T20:48:28.142100+0000","last_became_active":"2024-08-03T20:48:27.132105+0000","last_became_peered":"2024-08-03T20:48:27.132105+0000","last_unstale":"2024-08-03T20:48:28.142100+0000","last_undegraded":"2024-08-03T20:48:28.142100+0000","last_fullsized":"2024-08-03T20:48:28.142100+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:48:26.104177+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:48:26.104177+0000","last_clean_scrub_stamp":"2024-08-03T20:48:26.104177+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T04:19:29.463190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.28","version":"0'0","reported_seq":13,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-08-03T20:48:28.421971+0000","last_change":"2024-08-03T20:48:27.156281+0000","last_active":"2024-08-03T20:48:28.421971+0000","last_peered":"2024-08-03T20:48:28.421971+0000","last_clean":"2024-08-03T20:48:28.421971+0000","last_became_active":"2024-08-03T20:48:27.156001+0000","last_became_peered":"2024-08-03T20:48:27.156001+0000","last_unstale":"2024-08-03T20:48:28.421971+0000","last_undegraded":"2024-08-03T20:48:28.421971+0000","last_fullsized":"2024-08-03T20:48:28.421971+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:48:26.104177+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:48:26.104177+0000","last_clean_scrub_stamp":"2024-08-03T20:48:26.104177+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T06:09:59.767222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.27","version":"0'0","reported_seq":13,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-08-03T20:48:28.145578+0000","last_change":"2024-08-03T20:48:27.148919+0000","last_active":"2024-08-03T20:48:28.145578+0000","last_peered":"2024-08-03T20:48:28.145578+0000","last_clean":"2024-08-03T20:48:28.145578+0000","last_became_active":"2024-08-03T20:48:27.148637+0000","last_became_peered":"2024-08-03T20:48:27.148637+0000","last_unstale":"2024-08-03T20:48:28.145578+0000","last_undegraded":"2024-08-03T20:48:28.145578+0000","last_fullsized":"2024-08-03T20:48:28.145578+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:48:26.104177+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:48:26.104177+0000","last_clean_scrub_stamp":"2024-08-03T20:48:26.104177+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T05:52:18.600376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.26","version":"0'0","reported_seq":13,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-08-03T20:48:28.422460+0000","last_change":"2024-08-03T20:48:27.129677+0000","last_active":"2024-08-03T20:48:28.422460+0000","last_peered":"2024-08-03T20:48:28.422460+0000","last_clean":"2024-08-03T20:48:28.422460+0000","last_became_active":"2024-08-03T20:48:27.129348+0000","last_became_peered":"2024-08-03T20:48:27.129348+0000","last_unstale":"2024-08-03T20:48:28.422460+0000","last_undegraded":"2024-08-03T20:48:28.422460+0000","last_fullsized":"2024-08-03T20:48:28.422460+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:48:26.104177+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:48:26.104177+0000","last_clean_scrub_stamp":"2024-08-03T20:48:26.104177+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T02:06:10.358093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.25","version":"0'0","reported_seq":13,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-08-03T20:48:28.145548+0000","last_change":"2024-08-03T20:48:27.148779+0000","last_active":"2024-08-03T20:48:28.145548+0000","last_peered":"2024-08-03T20:48:28.145548+0000","last_clean":"2024-08-03T20:48:28.145548+0000","last_became_active":"2024-08-03T20:48:27.148161+0000","last_became_peered":"2024-08-03T20:48:27.148161+0000","last_unstale":"2024-08-03T20:48:28.145548+0000","last_undegraded":"2024-08-03T20:48:28.145548+0000","last_fullsized":"2024-08-03T20:48:28.145548+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:48:26.104177+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:48:26.104177+0000","last_clean_scrub_stamp":"2024-08-03T20:48:26.104177+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T01:04:35.590161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.24","version":"0'0","reported_seq":13,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-08-03T20:48:28.423507+0000","last_change":"2024-08-03T20:48:27.142641+0000","last_active":"2024-08-03T20:48:28.423507+0000","last_peered":"2024-08-03T20:48:28.423507+0000","last_clean":"2024-08-03T20:48:28.423507+0000","last_became_active":"2024-08-03T20:48:27.142331+0000","last_became_peered":"2024-08-03T20:48:27.142331+0000","last_unstale":"2024-08-03T20:48:28.423507+0000","last_undegraded":"2024-08-03T20:48:28.423507+0000","last_fullsized":"2024-08-03T20:48:28.423507+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:48:26.104177+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:48:26.104177+0000","last_clean_scrub_stamp":"2024-08-03T20:48:26.104177+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T04:17:36.047269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.23","version":"0'0","reported_seq":13,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-08-03T20:48:28.129737+0000","last_change":"2024-08-03T20:48:27.135815+0000","last_active":"2024-08-03T20:48:28.129737+0000","last_peered":"2024-08-03T20:48:28.129737+0000","last_clean":"2024-08-03T20:48:28.129737+0000","last_became_active":"2024-08-03T20:48:27.134045+0000","last_became_peered":"2024-08-03T20:48:27.134045+0000","last_unstale":"2024-08-03T20:48:28.129737+0000","last_undegraded":"2024-08-03T20:48:28.129737+0000","last_fullsized":"2024-08-03T20:48:28.129737+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:48:26.104177+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:48:26.104177+0000","last_clean_scrub_stamp":"2024-08-03T20:48:26.104177+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T06:33:48.629840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.22","version":"0'0","reported_seq":13,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-08-03T20:48:28.422729+0000","last_change":"2024-08-03T20:48:27.142274+0000","last_active":"2024-08-03T20:48:28.422729+0000","last_peered":"2024-08-03T20:48:28.422729+0000","last_clean":"2024-08-03T20:48:28.422729+0000","last_became_active":"2024-08-03T20:48:27.141948+0000","last_became_peered":"2024-08-03T20:48:27.141948+0000","last_unstale":"2024-08-03T20:48:28.422729+0000","last_undegraded":"2024-08-03T20:48:28.422729+0000","last_fullsized":"2024-08-03T20:48:28.422729+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:48:26.104177+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:48:26.104177+0000","last_clean_scrub_stamp":"2024-08-03T20:48:26.104177+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T03:46:54.747620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.21","version":"0'0","reported_seq":13,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-08-03T20:48:28.422296+0000","last_change":"2024-08-03T20:48:27.140643+0000","last_active":"2024-08-03T20:48:28.422296+0000","last_peered":"2024-08-03T20:48:28.422296+0000","last_clean":"2024-08-03T20:48:28.422296+0000","last_became_active":"2024-08-03T20:48:27.140194+0000","last_became_peered":"2024-08-03T20:48:27.140194+0000","last_unstale":"2024-08-03T20:48:28.422296+0000","last_undegraded":"2024-08-03T20:48:28.422296+0000","last_fullsized":"2024-08-03T20:48:28.422296+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:48:26.104177+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:48:26.104177+0000","last_clean_scrub_stamp":"2024-08-03T20:48:26.104177+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-04T23:58:08.281672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.20","version":"0'0","reported_seq":13,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-08-03T20:48:28.424357+0000","last_change":"2024-08-03T20:48:27.149507+0000","last_active":"2024-08-03T20:48:28.424357+0000","last_peered":"2024-08-03T20:48:28.424357+0000","last_clean":"2024-08-03T20:48:28.424357+0000","last_became_active":"2024-08-03T20:48:27.139413+0000","last_became_peered":"2024-08-03T20:48:27.139413+0000","last_unstale":"2024-08-03T20:48:28.424357+0000","last_undegraded":"2024-08-03T20:48:28.424357+0000","last_fullsized":"2024-08-03T20:48:28.424357+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:48:26.104177+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:48:26.104177+0000","last_clean_scrub_stamp":"2024-08-03T20:48:26.104177+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T07:27:20.062252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":13,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-08-03T20:48:28.142274+0000","last_change":"2024-08-03T20:48:27.135846+0000","last_active":"2024-08-03T20:48:28.142274+0000","last_peered":"2024-08-03T20:48:28.142274+0000","last_clean":"2024-08-03T20:48:28.142274+0000","last_became_active":"2024-08-03T20:48:27.135522+0000","last_became_peered":"2024-08-03T20:48:27.135522+0000","last_unstale":"2024-08-03T20:48:28.142274+0000","last_undegraded":"2024-08-03T20:48:28.142274+0000","last_fullsized":"2024-08-03T20:48:28.142274+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:48:26.104177+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:48:26.104177+0000","last_clean_scrub_stamp":"2024-08-03T20:48:26.104177+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T01:37:34.683431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1f","version":"31'4","reported_seq":19,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-08-03T20:48:28.422645+0000","last_change":"2024-08-03T20:48:26.141319+0000","last_active":"2024-08-03T20:48:28.422645+0000","last_peered":"2024-08-03T20:48:28.422645+0000","last_clean":"2024-08-03T20:48:28.422645+0000","last_became_active":"2024-08-03T20:48:26.140981+0000","last_became_peered":"2024-08-03T20:48:26.140981+0000","last_unstale":"2024-08-03T20:48:28.422645+0000","last_undegraded":"2024-08-03T20:48:28.422645+0000","last_fullsized":"2024-08-03T20:48:28.422645+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:48:25.097349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:48:25.097349+0000","last_clean_scrub_stamp":"2024-08-03T20:48:25.097349+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T00:41:15.511682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1f","version":"0'0","reported_seq":13,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-08-03T20:48:28.422565+0000","last_change":"2024-08-03T20:48:27.139612+0000","last_active":"2024-08-03T20:48:28.422565+0000","last_peered":"2024-08-03T20:48:28.422565+0000","last_clean":"2024-08-03T20:48:28.422565+0000","last_became_active":"2024-08-03T20:48:27.137924+0000","last_became_peered":"2024-08-03T20:48:27.137924+0000","last_unstale":"2024-08-03T20:48:28.422565+0000","last_undegraded":"2024-08-03T20:48:28.422565+0000","last_fullsized":"2024-08-03T20:48:28.422565+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:48:26.104177+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:48:26.104177+0000","last_clean_scrub_stamp":"2024-08-03T20:48:26.104177+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-04T21:58:23.025892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1e","version":"31'1","reported_seq":16,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-08-03T20:48:28.142369+0000","last_change":"2024-08-03T20:48:26.143339+0000","last_active":"2024-08-03T20:48:28.142369+0000","last_peered":"2024-08-03T20:48:28.142369+0000","last_clean":"2024-08-03T20:48:28.142369+0000","last_became_active":"2024-08-03T20:48:26.143040+0000","last_became_peered":"2024-08-03T20:48:26.143040+0000","last_unstale":"2024-08-03T20:48:28.142369+0000","last_undegraded":"2024-08-03T20:48:28.142369+0000","last_fullsized":"2024-08-03T20:48:28.142369+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:48:25.097349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:48:25.097349+0000","last_clean_scrub_stamp":"2024-08-03T20:48:25.097349+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-04T21:51:24.778117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":13,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-08-03T20:48:28.129636+0000","last_change":"2024-08-03T20:48:27.125335+0000","last_active":"2024-08-03T20:48:28.129636+0000","last_peered":"2024-08-03T20:48:28.129636+0000","last_clean":"2024-08-03T20:48:28.129636+0000","last_became_active":"2024-08-03T20:48:27.124987+0000","last_became_peered":"2024-08-03T20:48:27.124987+0000","last_unstale":"2024-08-03T20:48:28.129636+0000","last_undegraded":"2024-08-03T20:48:28.129636+0000","last_fullsized":"2024-08-03T20:48:28.129636+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:48:26.104177+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:48:26.104177+0000","last_clean_scrub_stamp":"2024-08-03T20:48:26.104177+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T08:40:18.971655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"31'2","reported_seq":17,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-08-03T20:48:28.421961+0000","last_change":"2024-08-03T20:48:26.144273+0000","last_active":"2024-08-03T20:48:28.421961+0000","last_peered":"2024-08-03T20:48:28.421961+0000","last_clean":"2024-08-03T20:48:28.421961+0000","last_became_active":"2024-08-03T20:48:26.143990+0000","last_became_peered":"2024-08-03T20:48:26.143990+0000","last_unstale":"2024-08-03T20:48:28.421961+0000","last_undegraded":"2024-08-03T20:48:28.421961+0000","last_fullsized":"2024-08-03T20:48:28.421961+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:48:25.097349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:48:25.097349+0000","last_clean_scrub_stamp":"2024-08-03T20:48:25.097349+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T08:40:17.964827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":13,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-08-03T20:48:28.145733+0000","last_change":"2024-08-03T20:48:27.150048+0000","last_active":"2024-08-03T20:48:28.145733+0000","last_peered":"2024-08-03T20:48:28.145733+0000","last_clean":"2024-08-03T20:48:28.145733+0000","last_became_active":"2024-08-03T20:48:27.149709+0000","last_became_peered":"2024-08-03T20:48:27.149709+0000","last_unstale":"2024-08-03T20:48:28.145733+0000","last_undegraded":"2024-08-03T20:48:28.145733+0000","last_fullsized":"2024-08-03T20:48:28.145733+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:48:26.104177+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:48:26.104177+0000","last_clean_scrub_stamp":"2024-08-03T20:48:26.104177+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T03:46:54.747620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1c","version":"31'2","reported_seq":17,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-08-03T20:48:28.128145+0000","last_change":"2024-08-03T20:48:26.130713+0000","last_active":"2024-08-03T20:48:28.128145+0000","last_peered":"2024-08-03T20:48:28.128145+0000","last_clean":"2024-08-03T20:48:28.128145+0000","last_became_active":"2024-08-03T20:48:26.130361+0000","last_became_peered":"2024-08-03T20:48:26.130361+0000","last_unstale":"2024-08-03T20:48:28.128145+0000","last_undegraded":"2024-08-03T20:48:28.128145+0000","last_fullsized":"2024-08-03T20:48:28.128145+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:48:25.097349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:48:25.097349+0000","last_clean_scrub_stamp":"2024-08-03T20:48:25.097349+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-04T23:11:25.239186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":13,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-08-03T20:48:28.424290+0000","last_change":"2024-08-03T20:48:27.147471+0000","last_active":"2024-08-03T20:48:28.424290+0000","last_peered":"2024-08-03T20:48:28.424290+0000","last_clean":"2024-08-03T20:48:28.424290+0000","last_became_active":"2024-08-03T20:48:27.137844+0000","last_became_peered":"2024-08-03T20:48:27.137844+0000","last_unstale":"2024-08-03T20:48:28.424290+0000","last_undegraded":"2024-08-03T20:48:28.424290+0000","last_fullsized":"2024-08-03T20:48:28.424290+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:48:26.104177+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:48:26.104177+0000","last_clean_scrub_stamp":"2024-08-03T20:48:26.104177+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T01:33:02.971567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1b","version":"31'1","reported_seq":16,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-08-03T20:48:28.129826+0000","last_change":"2024-08-03T20:48:26.133899+0000","last_active":"2024-08-03T20:48:28.129826+0000","last_peered":"2024-08-03T20:48:28.129826+0000","last_clean":"2024-08-03T20:48:28.129826+0000","last_became_active":"2024-08-03T20:48:26.132184+0000","last_became_peered":"2024-08-03T20:48:26.132184+0000","last_unstale":"2024-08-03T20:48:28.129826+0000","last_undegraded":"2024-08-03T20:48:28.129826+0000","last_fullsized":"2024-08-03T20:48:28.129826+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:48:25.097349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:48:25.097349+0000","last_clean_scrub_stamp":"2024-08-03T20:48:25.097349+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T07:03:59.763697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":13,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-08-03T20:48:28.423236+0000","last_change":"2024-08-03T20:48:27.136698+0000","last_active":"2024-08-03T20:48:28.423236+0000","last_peered":"2024-08-03T20:48:28.423236+0000","last_clean":"2024-08-03T20:48:28.423236+0000","last_became_active":"2024-08-03T20:48:27.136364+0000","last_became_peered":"2024-08-03T20:48:27.136364+0000","last_unstale":"2024-08-03T20:48:28.423236+0000","last_undegraded":"2024-08-03T20:48:28.423236+0000","last_fullsized":"2024-08-03T20:48:28.423236+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:48:26.104177+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:48:26.104177+0000","last_clean_scrub_stamp":"2024-08-03T20:48:26.104177+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T00:06:25.299214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"31'1","reported_seq":16,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-08-03T20:48:28.129558+0000","last_change":"2024-08-03T20:48:26.138799+0000","last_active":"2024-08-03T20:48:28.129558+0000","last_peered":"2024-08-03T20:48:28.129558+0000","last_clean":"2024-08-03T20:48:28.129558+0000","last_became_active":"2024-08-03T20:48:26.138410+0000","last_became_peered":"2024-08-03T20:48:26.138410+0000","last_unstale":"2024-08-03T20:48:28.129558+0000","last_undegraded":"2024-08-03T20:48:28.129558+0000","last_fullsized":"2024-08-03T20:48:28.129558+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:48:25.097349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:48:25.097349+0000","last_clean_scrub_stamp":"2024-08-03T20:48:25.097349+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T04:43:29.058607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":13,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-08-03T20:48:28.129244+0000","last_change":"2024-08-03T20:48:27.139757+0000","last_active":"2024-08-03T20:48:28.129244+0000","last_peered":"2024-08-03T20:48:28.129244+0000","last_clean":"2024-08-03T20:48:28.129244+0000","last_became_active":"2024-08-03T20:48:27.138355+0000","last_became_peered":"2024-08-03T20:48:27.138355+0000","last_unstale":"2024-08-03T20:48:28.129244+0000","last_undegraded":"2024-08-03T20:48:28.129244+0000","last_fullsized":"2024-08-03T20:48:28.129244+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:48:26.104177+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:48:26.104177+0000","last_clean_scrub_stamp":"2024-08-03T20:48:26.104177+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T06:38:56.263014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-08-03T20:48:28.421410+0000","last_change":"2024-08-03T20:48:26.132625+0000","last_active":"2024-08-03T20:48:28.421410+0000","last_peered":"2024-08-03T20:48:28.421410+0000","last_clean":"2024-08-03T20:48:28.421410+0000","last_became_active":"2024-08-03T20:48:26.132320+0000","last_became_peered":"2024-08-03T20:48:26.132320+0000","last_unstale":"2024-08-03T20:48:28.421410+0000","last_undegraded":"2024-08-03T20:48:28.421410+0000","last_fullsized":"2024-08-03T20:48:28.421410+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:48:25.097349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:48:25.097349+0000","last_clean_scrub_stamp":"2024-08-03T20:48:25.097349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-04T21:10:13.013933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":13,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-08-03T20:48:28.144947+0000","last_change":"2024-08-03T20:48:27.149116+0000","last_active":"2024-08-03T20:48:28.144947+0000","last_peered":"2024-08-03T20:48:28.144947+0000","last_clean":"2024-08-03T20:48:28.144947+0000","last_became_active":"2024-08-03T20:48:27.148592+0000","last_became_peered":"2024-08-03T20:48:27.148592+0000","last_unstale":"2024-08-03T20:48:28.144947+0000","last_undegraded":"2024-08-03T20:48:28.144947+0000","last_fullsized":"2024-08-03T20:48:28.144947+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:48:26.104177+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:48:26.104177+0000","last_clean_scrub_stamp":"2024-08-03T20:48:26.104177+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T04:55:17.727620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-08-03T20:48:28.129103+0000","last_change":"2024-08-03T20:48:26.125016+0000","last_active":"2024-08-03T20:48:28.129103+0000","last_peered":"2024-08-03T20:48:28.129103+0000","last_clean":"2024-08-03T20:48:28.129103+0000","last_became_active":"2024-08-03T20:48:26.123236+0000","last_became_peered":"2024-08-03T20:48:26.123236+0000","last_unstale":"2024-08-03T20:48:28.129103+0000","last_undegraded":"2024-08-03T20:48:28.129103+0000","last_fullsized":"2024-08-03T20:48:28.129103+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:48:25.097349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:48:25.097349+0000","last_clean_scrub_stamp":"2024-08-03T20:48:25.097349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T02:09:07.905859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":13,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-08-03T20:48:28.423605+0000","last_change":"2024-08-03T20:48:27.153117+0000","last_active":"2024-08-03T20:48:28.423605+0000","last_peered":"2024-08-03T20:48:28.423605+0000","last_clean":"2024-08-03T20:48:28.423605+0000","last_became_active":"2024-08-03T20:48:27.152747+0000","last_became_peered":"2024-08-03T20:48:27.152747+0000","last_unstale":"2024-08-03T20:48:28.423605+0000","last_undegraded":"2024-08-03T20:48:28.423605+0000","last_fullsized":"2024-08-03T20:48:28.423605+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:48:26.104177+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:48:26.104177+0000","last_clean_scrub_stamp":"2024-08-03T20:48:26.104177+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T01:33:02.971567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-08-03T20:48:28.422366+0000","last_change":"2024-08-03T20:48:26.144117+0000","last_active":"2024-08-03T20:48:28.422366+0000","last_peered":"2024-08-03T20:48:28.422366+0000","last_clean":"2024-08-03T20:48:28.422366+0000","last_became_active":"2024-08-03T20:48:26.143642+0000","last_became_peered":"2024-08-03T20:48:26.143642+0000","last_unstale":"2024-08-03T20:48:28.422366+0000","last_undegraded":"2024-08-03T20:48:28.422366+0000","last_fullsized":"2024-08-03T20:48:28.422366+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:48:25.097349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:48:25.097349+0000","last_clean_scrub_stamp":"2024-08-03T20:48:25.097349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T06:33:47.623013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":13,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-08-03T20:48:28.422310+0000","last_change":"2024-08-03T20:48:27.157632+0000","last_active":"2024-08-03T20:48:28.422310+0000","last_peered":"2024-08-03T20:48:28.422310+0000","last_clean":"2024-08-03T20:48:28.422310+0000","last_became_active":"2024-08-03T20:48:27.157201+0000","last_became_peered":"2024-08-03T20:48:27.157201+0000","last_unstale":"2024-08-03T20:48:28.422310+0000","last_undegraded":"2024-08-03T20:48:28.422310+0000","last_fullsized":"2024-08-03T20:48:28.422310+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:48:26.104177+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:48:26.104177+0000","last_clean_scrub_stamp":"2024-08-03T20:48:26.104177+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T04:54:21.125473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-08-03T20:48:28.142495+0000","last_change":"2024-08-03T20:48:26.140459+0000","last_active":"2024-08-03T20:48:28.142495+0000","last_peered":"2024-08-03T20:48:28.142495+0000","last_clean":"2024-08-03T20:48:28.142495+0000","last_became_active":"2024-08-03T20:48:26.139412+0000","last_became_peered":"2024-08-03T20:48:26.139412+0000","last_unstale":"2024-08-03T20:48:28.142495+0000","last_undegraded":"2024-08-03T20:48:28.142495+0000","last_fullsized":"2024-08-03T20:48:28.142495+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:48:25.097349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:48:25.097349+0000","last_clean_scrub_stamp":"2024-08-03T20:48:25.097349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T00:02:20.619397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"0'0","reported_seq":13,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-08-03T20:48:28.422099+0000","last_change":"2024-08-03T20:48:27.139783+0000","last_active":"2024-08-03T20:48:28.422099+0000","last_peered":"2024-08-03T20:48:28.422099+0000","last_clean":"2024-08-03T20:48:28.422099+0000","last_became_active":"2024-08-03T20:48:27.136109+0000","last_became_peered":"2024-08-03T20:48:27.136109+0000","last_unstale":"2024-08-03T20:48:28.422099+0000","last_undegraded":"2024-08-03T20:48:28.422099+0000","last_fullsized":"2024-08-03T20:48:28.422099+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:48:26.104177+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:48:26.104177+0000","last_clean_scrub_stamp":"2024-08-03T20:48:26.104177+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-04T22:39:59.296381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.15","version":"31'3","reported_seq":18,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-08-03T20:48:28.423681+0000","last_change":"2024-08-03T20:48:26.140631+0000","last_active":"2024-08-03T20:48:28.423681+0000","last_peered":"2024-08-03T20:48:28.423681+0000","last_clean":"2024-08-03T20:48:28.423681+0000","last_became_active":"2024-08-03T20:48:26.139867+0000","last_became_peered":"2024-08-03T20:48:26.139867+0000","last_unstale":"2024-08-03T20:48:28.423681+0000","last_undegraded":"2024-08-03T20:48:28.423681+0000","last_fullsized":"2024-08-03T20:48:28.423681+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:48:25.097349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:48:25.097349+0000","last_clean_scrub_stamp":"2024-08-03T20:48:25.097349+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-04T23:17:50.262566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"0'0","reported_seq":13,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-08-03T20:48:28.423699+0000","last_change":"2024-08-03T20:48:27.159216+0000","last_active":"2024-08-03T20:48:28.423699+0000","last_peered":"2024-08-03T20:48:28.423699+0000","last_clean":"2024-08-03T20:48:28.423699+0000","last_became_active":"2024-08-03T20:48:27.158823+0000","last_became_peered":"2024-08-03T20:48:27.158823+0000","last_unstale":"2024-08-03T20:48:28.423699+0000","last_undegraded":"2024-08-03T20:48:28.423699+0000","last_fullsized":"2024-08-03T20:48:28.423699+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:48:26.104177+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:48:26.104177+0000","last_clean_scrub_stamp":"2024-08-03T20:48:26.104177+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T08:15:08.218151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"31'6","reported_seq":21,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-08-03T20:48:28.421992+0000","last_change":"2024-08-03T20:48:26.141579+0000","last_active":"2024-08-03T20:48:28.421992+0000","last_peered":"2024-08-03T20:48:28.421992+0000","last_clean":"2024-08-03T20:48:28.421992+0000","last_became_active":"2024-08-03T20:48:26.141241+0000","last_became_peered":"2024-08-03T20:48:26.141241+0000","last_unstale":"2024-08-03T20:48:28.421992+0000","last_undegraded":"2024-08-03T20:48:28.421992+0000","last_fullsized":"2024-08-03T20:48:28.421992+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:48:25.097349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:48:25.097349+0000","last_clean_scrub_stamp":"2024-08-03T20:48:25.097349+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T03:27:53.316279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":13,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-08-03T20:48:28.145839+0000","last_change":"2024-08-03T20:48:27.148908+0000","last_active":"2024-08-03T20:48:28.145839+0000","last_peered":"2024-08-03T20:48:28.145839+0000","last_clean":"2024-08-03T20:48:28.145839+0000","last_became_active":"2024-08-03T20:48:27.148502+0000","last_became_peered":"2024-08-03T20:48:27.148502+0000","last_unstale":"2024-08-03T20:48:28.145839+0000","last_undegraded":"2024-08-03T20:48:28.145839+0000","last_fullsized":"2024-08-03T20:48:28.145839+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:48:26.104177+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:48:26.104177+0000","last_clean_scrub_stamp":"2024-08-03T20:48:26.104177+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T06:09:59.767222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"31'2","reported_seq":17,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-08-03T20:48:28.129941+0000","last_change":"2024-08-03T20:48:26.131550+0000","last_active":"2024-08-03T20:48:28.129941+0000","last_peered":"2024-08-03T20:48:28.129941+0000","last_clean":"2024-08-03T20:48:28.129941+0000","last_became_active":"2024-08-03T20:48:26.130860+0000","last_became_peered":"2024-08-03T20:48:26.130860+0000","last_unstale":"2024-08-03T20:48:28.129941+0000","last_undegraded":"2024-08-03T20:48:28.129941+0000","last_fullsized":"2024-08-03T20:48:28.129941+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:48:25.097349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:48:25.097349+0000","last_clean_scrub_stamp":"2024-08-03T20:48:25.097349+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T01:47:23.677829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":13,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-08-03T20:48:28.421980+0000","last_change":"2024-08-03T20:48:27.150930+0000","last_active":"2024-08-03T20:48:28.421980+0000","last_peered":"2024-08-03T20:48:28.421980+0000","last_clean":"2024-08-03T20:48:28.421980+0000","last_became_active":"2024-08-03T20:48:27.150013+0000","last_became_peered":"2024-08-03T20:48:27.150013+0000","last_unstale":"2024-08-03T20:48:28.421980+0000","last_undegraded":"2024-08-03T20:48:28.421980+0000","last_fullsized":"2024-08-03T20:48:28.421980+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:48:26.104177+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:48:26.104177+0000","last_clean_scrub_stamp":"2024-08-03T20:48:26.104177+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T07:08:24.833346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-08-03T20:48:28.144939+0000","last_change":"2024-08-03T20:48:26.128882+0000","last_active":"2024-08-03T20:48:28.144939+0000","last_peered":"2024-08-03T20:48:28.144939+0000","last_clean":"2024-08-03T20:48:28.144939+0000","last_became_active":"2024-08-03T20:48:26.127874+0000","last_became_peered":"2024-08-03T20:48:26.127874+0000","last_unstale":"2024-08-03T20:48:28.144939+0000","last_undegraded":"2024-08-03T20:48:28.144939+0000","last_fullsized":"2024-08-03T20:48:28.144939+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:48:25.097349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:48:25.097349+0000","last_clean_scrub_stamp":"2024-08-03T20:48:25.097349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T00:41:15.511682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":13,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-08-03T20:48:28.423743+0000","last_change":"2024-08-03T20:48:27.144778+0000","last_active":"2024-08-03T20:48:28.423743+0000","last_peered":"2024-08-03T20:48:28.423743+0000","last_clean":"2024-08-03T20:48:28.423743+0000","last_became_active":"2024-08-03T20:48:27.144302+0000","last_became_peered":"2024-08-03T20:48:27.144302+0000","last_unstale":"2024-08-03T20:48:28.423743+0000","last_undegraded":"2024-08-03T20:48:28.423743+0000","last_fullsized":"2024-08-03T20:48:28.423743+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:48:26.104177+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:48:26.104177+0000","last_clean_scrub_stamp":"2024-08-03T20:48:26.104177+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-04T21:00:04.257461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-08-03T20:48:28.421495+0000","last_change":"2024-08-03T20:48:26.142689+0000","last_active":"2024-08-03T20:48:28.421495+0000","last_peered":"2024-08-03T20:48:28.421495+0000","last_clean":"2024-08-03T20:48:28.421495+0000","last_became_active":"2024-08-03T20:48:26.142251+0000","last_became_peered":"2024-08-03T20:48:26.142251+0000","last_unstale":"2024-08-03T20:48:28.421495+0000","last_undegraded":"2024-08-03T20:48:28.421495+0000","last_fullsized":"2024-08-03T20:48:28.421495+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:48:25.097349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:48:25.097349+0000","last_clean_scrub_stamp":"2024-08-03T20:48:25.097349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T07:41:42.037447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":13,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-08-03T20:48:28.422531+0000","last_change":"2024-08-03T20:48:27.138306+0000","last_active":"2024-08-03T20:48:28.422531+0000","last_peered":"2024-08-03T20:48:28.422531+0000","last_clean":"2024-08-03T20:48:28.422531+0000","last_became_active":"2024-08-03T20:48:27.138003+0000","last_became_peered":"2024-08-03T20:48:27.138003+0000","last_unstale":"2024-08-03T20:48:28.422531+0000","last_undegraded":"2024-08-03T20:48:28.422531+0000","last_fullsized":"2024-08-03T20:48:28.422531+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:48:26.104177+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:48:26.104177+0000","last_clean_scrub_stamp":"2024-08-03T20:48:26.104177+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T04:19:29.463190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.10","version":"31'8","reported_seq":23,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-08-03T20:48:28.421461+0000","last_change":"2024-08-03T20:48:26.137963+0000","last_active":"2024-08-03T20:48:28.421461+0000","last_peered":"2024-08-03T20:48:28.421461+0000","last_clean":"2024-08-03T20:48:28.421461+0000","last_became_active":"2024-08-03T20:48:26.137650+0000","last_became_peered":"2024-08-03T20:48:26.137650+0000","last_unstale":"2024-08-03T20:48:28.421461+0000","last_undegraded":"2024-08-03T20:48:28.421461+0000","last_fullsized":"2024-08-03T20:48:28.421461+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:48:25.097349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:48:25.097349+0000","last_clean_scrub_stamp":"2024-08-03T20:48:25.097349+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T07:56:10.908634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"0'0","reported_seq":13,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-08-03T20:48:28.129928+0000","last_change":"2024-08-03T20:48:27.141521+0000","last_active":"2024-08-03T20:48:28.129928+0000","last_peered":"2024-08-03T20:48:28.129928+0000","last_clean":"2024-08-03T20:48:28.129928+0000","last_became_active":"2024-08-03T20:48:27.139338+0000","last_became_peered":"2024-08-03T20:48:27.139338+0000","last_unstale":"2024-08-03T20:48:28.129928+0000","last_undegraded":"2024-08-03T20:48:28.129928+0000","last_fullsized":"2024-08-03T20:48:28.129928+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:48:26.104177+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:48:26.104177+0000","last_clean_scrub_stamp":"2024-08-03T20:48:26.104177+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-04T21:37:44.475861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.f","version":"31'1","reported_seq":16,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-08-03T20:48:28.145685+0000","last_change":"2024-08-03T20:48:26.137755+0000","last_active":"2024-08-03T20:48:28.145685+0000","last_peered":"2024-08-03T20:48:28.145685+0000","last_clean":"2024-08-03T20:48:28.145685+0000","last_became_active":"2024-08-03T20:48:26.137424+0000","last_became_peered":"2024-08-03T20:48:26.137424+0000","last_unstale":"2024-08-03T20:48:28.145685+0000","last_undegraded":"2024-08-03T20:48:28.145685+0000","last_fullsized":"2024-08-03T20:48:28.145685+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:48:25.097349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:48:25.097349+0000","last_clean_scrub_stamp":"2024-08-03T20:48:25.097349+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-04T21:51:24.778117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":13,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-08-03T20:48:28.423752+0000","last_change":"2024-08-03T20:48:27.145018+0000","last_active":"2024-08-03T20:48:28.423752+0000","last_peered":"2024-08-03T20:48:28.423752+0000","last_clean":"2024-08-03T20:48:28.423752+0000","last_became_active":"2024-08-03T20:48:27.144129+0000","last_became_peered":"2024-08-03T20:48:27.144129+0000","last_unstale":"2024-08-03T20:48:28.423752+0000","last_undegraded":"2024-08-03T20:48:28.423752+0000","last_fullsized":"2024-08-03T20:48:28.423752+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:48:26.104177+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:48:26.104177+0000","last_clean_scrub_stamp":"2024-08-03T20:48:26.104177+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T07:27:20.062252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.e","version":"31'2","reported_seq":17,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-08-03T20:48:28.129877+0000","last_change":"2024-08-03T20:48:26.130554+0000","last_active":"2024-08-03T20:48:28.129877+0000","last_peered":"2024-08-03T20:48:28.129877+0000","last_clean":"2024-08-03T20:48:28.129877+0000","last_became_active":"2024-08-03T20:48:26.130139+0000","last_became_peered":"2024-08-03T20:48:26.130139+0000","last_unstale":"2024-08-03T20:48:28.129877+0000","last_undegraded":"2024-08-03T20:48:28.129877+0000","last_fullsized":"2024-08-03T20:48:28.129877+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:48:25.097349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:48:25.097349+0000","last_clean_scrub_stamp":"2024-08-03T20:48:25.097349+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T01:09:11.035968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":13,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-08-03T20:48:28.145271+0000","last_change":"2024-08-03T20:48:27.138372+0000","last_active":"2024-08-03T20:48:28.145271+0000","last_peered":"2024-08-03T20:48:28.145271+0000","last_clean":"2024-08-03T20:48:28.145271+0000","last_became_active":"2024-08-03T20:48:27.138073+0000","last_became_peered":"2024-08-03T20:48:27.138073+0000","last_unstale":"2024-08-03T20:48:28.145271+0000","last_undegraded":"2024-08-03T20:48:28.145271+0000","last_fullsized":"2024-08-03T20:48:28.145271+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:48:26.104177+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:48:26.104177+0000","last_clean_scrub_stamp":"2024-08-03T20:48:26.104177+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-04T20:50:11.082404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.d","version":"31'1","reported_seq":16,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-08-03T20:48:28.129942+0000","last_change":"2024-08-03T20:48:26.131174+0000","last_active":"2024-08-03T20:48:28.129942+0000","last_peered":"2024-08-03T20:48:28.129942+0000","last_clean":"2024-08-03T20:48:28.129942+0000","last_became_active":"2024-08-03T20:48:26.130158+0000","last_became_peered":"2024-08-03T20:48:26.130158+0000","last_unstale":"2024-08-03T20:48:28.129942+0000","last_undegraded":"2024-08-03T20:48:28.129942+0000","last_fullsized":"2024-08-03T20:48:28.129942+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:48:25.097349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:48:25.097349+0000","last_clean_scrub_stamp":"2024-08-03T20:48:25.097349+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T01:06:56.212720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"0'0","reported_seq":13,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-08-03T20:48:28.128311+0000","last_change":"2024-08-03T20:48:27.135223+0000","last_active":"2024-08-03T20:48:28.128311+0000","last_peered":"2024-08-03T20:48:28.128311+0000","last_clean":"2024-08-03T20:48:28.128311+0000","last_became_active":"2024-08-03T20:48:27.134926+0000","last_became_peered":"2024-08-03T20:48:27.134926+0000","last_unstale":"2024-08-03T20:48:28.128311+0000","last_undegraded":"2024-08-03T20:48:28.128311+0000","last_fullsized":"2024-08-03T20:48:28.128311+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:48:26.104177+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:48:26.104177+0000","last_clean_scrub_stamp":"2024-08-03T20:48:26.104177+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-04T23:44:46.267836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-08-03T20:48:28.129601+0000","last_change":"2024-08-03T20:48:26.138873+0000","last_active":"2024-08-03T20:48:28.129601+0000","last_peered":"2024-08-03T20:48:28.129601+0000","last_clean":"2024-08-03T20:48:28.129601+0000","last_became_active":"2024-08-03T20:48:26.138603+0000","last_became_peered":"2024-08-03T20:48:26.138603+0000","last_unstale":"2024-08-03T20:48:28.129601+0000","last_undegraded":"2024-08-03T20:48:28.129601+0000","last_fullsized":"2024-08-03T20:48:28.129601+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:48:25.097349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:48:25.097349+0000","last_clean_scrub_stamp":"2024-08-03T20:48:25.097349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T07:43:33.823181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":13,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-08-03T20:48:28.127819+0000","last_change":"2024-08-03T20:48:27.123831+0000","last_active":"2024-08-03T20:48:28.127819+0000","last_peered":"2024-08-03T20:48:28.127819+0000","last_clean":"2024-08-03T20:48:28.127819+0000","last_became_active":"2024-08-03T20:48:27.123215+0000","last_became_peered":"2024-08-03T20:48:27.123215+0000","last_unstale":"2024-08-03T20:48:28.127819+0000","last_undegraded":"2024-08-03T20:48:28.127819+0000","last_fullsized":"2024-08-03T20:48:28.127819+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:48:26.104177+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:48:26.104177+0000","last_clean_scrub_stamp":"2024-08-03T20:48:26.104177+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T07:43:34.830009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.b","version":"31'2","reported_seq":17,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-08-03T20:48:28.142288+0000","last_change":"2024-08-03T20:48:26.138803+0000","last_active":"2024-08-03T20:48:28.142288+0000","last_peered":"2024-08-03T20:48:28.142288+0000","last_clean":"2024-08-03T20:48:28.142288+0000","last_became_active":"2024-08-03T20:48:26.138491+0000","last_became_peered":"2024-08-03T20:48:26.138491+0000","last_unstale":"2024-08-03T20:48:28.142288+0000","last_undegraded":"2024-08-03T20:48:28.142288+0000","last_fullsized":"2024-08-03T20:48:28.142288+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:48:25.097349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:48:25.097349+0000","last_clean_scrub_stamp":"2024-08-03T20:48:25.097349+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T01:34:49.869344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":13,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-08-03T20:48:28.128283+0000","last_change":"2024-08-03T20:48:27.152223+0000","last_active":"2024-08-03T20:48:28.128283+0000","last_peered":"2024-08-03T20:48:28.128283+0000","last_clean":"2024-08-03T20:48:28.128283+0000","last_became_active":"2024-08-03T20:48:27.151893+0000","last_became_peered":"2024-08-03T20:48:27.151893+0000","last_unstale":"2024-08-03T20:48:28.128283+0000","last_undegraded":"2024-08-03T20:48:28.128283+0000","last_fullsized":"2024-08-03T20:48:28.128283+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:48:26.104177+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:48:26.104177+0000","last_clean_scrub_stamp":"2024-08-03T20:48:26.104177+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T01:09:12.042796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-08-03T20:48:28.422682+0000","last_change":"2024-08-03T20:48:26.140981+0000","last_active":"2024-08-03T20:48:28.422682+0000","last_peered":"2024-08-03T20:48:28.422682+0000","last_clean":"2024-08-03T20:48:28.422682+0000","last_became_active":"2024-08-03T20:48:26.140503+0000","last_became_peered":"2024-08-03T20:48:26.140503+0000","last_unstale":"2024-08-03T20:48:28.422682+0000","last_undegraded":"2024-08-03T20:48:28.422682+0000","last_fullsized":"2024-08-03T20:48:28.422682+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:48:25.097349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:48:25.097349+0000","last_clean_scrub_stamp":"2024-08-03T20:48:25.097349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T05:17:20.685991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":13,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-08-03T20:48:28.127542+0000","last_change":"2024-08-03T20:48:27.148919+0000","last_active":"2024-08-03T20:48:28.127542+0000","last_peered":"2024-08-03T20:48:28.127542+0000","last_clean":"2024-08-03T20:48:28.127542+0000","last_became_active":"2024-08-03T20:48:27.148624+0000","last_became_peered":"2024-08-03T20:48:27.148624+0000","last_unstale":"2024-08-03T20:48:28.127542+0000","last_undegraded":"2024-08-03T20:48:28.127542+0000","last_fullsized":"2024-08-03T20:48:28.127542+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:48:26.104177+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:48:26.104177+0000","last_clean_scrub_stamp":"2024-08-03T20:48:26.104177+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T04:43:30.065435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-08-03T20:48:28.129073+0000","last_change":"2024-08-03T20:48:26.130661+0000","last_active":"2024-08-03T20:48:28.129073+0000","last_peered":"2024-08-03T20:48:28.129073+0000","last_clean":"2024-08-03T20:48:28.129073+0000","last_became_active":"2024-08-03T20:48:26.130265+0000","last_became_peered":"2024-08-03T20:48:26.130265+0000","last_unstale":"2024-08-03T20:48:28.129073+0000","last_undegraded":"2024-08-03T20:48:28.129073+0000","last_fullsized":"2024-08-03T20:48:28.129073+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:48:25.097349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:48:25.097349+0000","last_clean_scrub_stamp":"2024-08-03T20:48:25.097349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-04T21:04:30.787042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":13,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-08-03T20:48:28.424249+0000","last_change":"2024-08-03T20:48:27.143587+0000","last_active":"2024-08-03T20:48:28.424249+0000","last_peered":"2024-08-03T20:48:28.424249+0000","last_clean":"2024-08-03T20:48:28.424249+0000","last_became_active":"2024-08-03T20:48:27.143257+0000","last_became_peered":"2024-08-03T20:48:27.143257+0000","last_unstale":"2024-08-03T20:48:28.424249+0000","last_undegraded":"2024-08-03T20:48:28.424249+0000","last_fullsized":"2024-08-03T20:48:28.424249+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:48:26.104177+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:48:26.104177+0000","last_clean_scrub_stamp":"2024-08-03T20:48:26.104177+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-04T20:57:36.715292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-08-03T20:48:28.422171+0000","last_change":"2024-08-03T20:48:26.138189+0000","last_active":"2024-08-03T20:48:28.422171+0000","last_peered":"2024-08-03T20:48:28.422171+0000","last_clean":"2024-08-03T20:48:28.422171+0000","last_became_active":"2024-08-03T20:48:26.136878+0000","last_became_peered":"2024-08-03T20:48:26.136878+0000","last_unstale":"2024-08-03T20:48:28.422171+0000","last_undegraded":"2024-08-03T20:48:28.422171+0000","last_fullsized":"2024-08-03T20:48:28.422171+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:48:25.097349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:48:25.097349+0000","last_clean_scrub_stamp":"2024-08-03T20:48:25.097349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T08:12:43.894277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":13,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-08-03T20:48:28.128318+0000","last_change":"2024-08-03T20:48:27.143599+0000","last_active":"2024-08-03T20:48:28.128318+0000","last_peered":"2024-08-03T20:48:28.128318+0000","last_clean":"2024-08-03T20:48:28.128318+0000","last_became_active":"2024-08-03T20:48:27.143037+0000","last_became_peered":"2024-08-03T20:48:27.143037+0000","last_unstale":"2024-08-03T20:48:28.128318+0000","last_undegraded":"2024-08-03T20:48:28.128318+0000","last_fullsized":"2024-08-03T20:48:28.128318+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:48:26.104177+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:48:26.104177+0000","last_clean_scrub_stamp":"2024-08-03T20:48:26.104177+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T03:02:47.933268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"31'2","reported_seq":17,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-08-03T20:48:28.145742+0000","last_change":"2024-08-03T20:48:26.140287+0000","last_active":"2024-08-03T20:48:28.145742+0000","last_peered":"2024-08-03T20:48:28.145742+0000","last_clean":"2024-08-03T20:48:28.145742+0000","last_became_active":"2024-08-03T20:48:26.139353+0000","last_became_peered":"2024-08-03T20:48:26.139353+0000","last_unstale":"2024-08-03T20:48:28.145742+0000","last_undegraded":"2024-08-03T20:48:28.145742+0000","last_fullsized":"2024-08-03T20:48:28.145742+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:48:25.097349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:48:25.097349+0000","last_clean_scrub_stamp":"2024-08-03T20:48:25.097349+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T08:48:23.338457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":13,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-08-03T20:48:28.423872+0000","last_change":"2024-08-03T20:48:27.159376+0000","last_active":"2024-08-03T20:48:28.423872+0000","last_peered":"2024-08-03T20:48:28.423872+0000","last_clean":"2024-08-03T20:48:28.423872+0000","last_became_active":"2024-08-03T20:48:27.158924+0000","last_became_peered":"2024-08-03T20:48:27.158924+0000","last_unstale":"2024-08-03T20:48:28.423872+0000","last_undegraded":"2024-08-03T20:48:28.423872+0000","last_fullsized":"2024-08-03T20:48:28.423872+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:48:26.104177+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:48:26.104177+0000","last_clean_scrub_stamp":"2024-08-03T20:48:26.104177+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T03:32:32.293255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.6","version":"32'1","reported_seq":16,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-08-03T20:48:28.144168+0000","last_change":"2024-08-03T20:48:26.140418+0000","last_active":"2024-08-03T20:48:28.144168+0000","last_peered":"2024-08-03T20:48:28.144168+0000","last_clean":"2024-08-03T20:48:28.144168+0000","last_became_active":"2024-08-03T20:48:26.139836+0000","last_became_peered":"2024-08-03T20:48:26.139836+0000","last_unstale":"2024-08-03T20:48:28.144168+0000","last_undegraded":"2024-08-03T20:48:28.144168+0000","last_fullsized":"2024-08-03T20:48:28.144168+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:48:25.097349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:48:25.097349+0000","last_clean_scrub_stamp":"2024-08-03T20:48:25.097349+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T08:48:23.338457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":13,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-08-03T20:48:28.142253+0000","last_change":"2024-08-03T20:48:27.136979+0000","last_active":"2024-08-03T20:48:28.142253+0000","last_peered":"2024-08-03T20:48:28.142253+0000","last_clean":"2024-08-03T20:48:28.142253+0000","last_became_active":"2024-08-03T20:48:27.136666+0000","last_became_peered":"2024-08-03T20:48:27.136666+0000","last_unstale":"2024-08-03T20:48:28.142253+0000","last_undegraded":"2024-08-03T20:48:28.142253+0000","last_fullsized":"2024-08-03T20:48:28.142253+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:48:26.104177+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:48:26.104177+0000","last_clean_scrub_stamp":"2024-08-03T20:48:26.104177+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-04T21:58:23.025892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-08-03T20:48:28.145298+0000","last_change":"2024-08-03T20:48:26.132240+0000","last_active":"2024-08-03T20:48:28.145298+0000","last_peered":"2024-08-03T20:48:28.145298+0000","last_clean":"2024-08-03T20:48:28.145298+0000","last_became_active":"2024-08-03T20:48:26.131663+0000","last_became_peered":"2024-08-03T20:48:26.131663+0000","last_unstale":"2024-08-03T20:48:28.145298+0000","last_undegraded":"2024-08-03T20:48:28.145298+0000","last_fullsized":"2024-08-03T20:48:28.145298+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:48:25.097349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:48:25.097349+0000","last_clean_scrub_stamp":"2024-08-03T20:48:25.097349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T00:15:51.465753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":13,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-08-03T20:48:28.422397+0000","last_change":"2024-08-03T20:48:27.140382+0000","last_active":"2024-08-03T20:48:28.422397+0000","last_peered":"2024-08-03T20:48:28.422397+0000","last_clean":"2024-08-03T20:48:28.422397+0000","last_became_active":"2024-08-03T20:48:27.139818+0000","last_became_peered":"2024-08-03T20:48:27.139818+0000","last_unstale":"2024-08-03T20:48:28.422397+0000","last_undegraded":"2024-08-03T20:48:28.422397+0000","last_fullsized":"2024-08-03T20:48:28.422397+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:48:26.104177+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:48:26.104177+0000","last_clean_scrub_stamp":"2024-08-03T20:48:26.104177+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T08:12:44.901105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-08-03T20:48:28.142414+0000","last_change":"2024-08-03T20:48:26.133848+0000","last_active":"2024-08-03T20:48:28.142414+0000","last_peered":"2024-08-03T20:48:28.142414+0000","last_clean":"2024-08-03T20:48:28.142414+0000","last_became_active":"2024-08-03T20:48:26.133510+0000","last_became_peered":"2024-08-03T20:48:26.133510+0000","last_unstale":"2024-08-03T20:48:28.142414+0000","last_undegraded":"2024-08-03T20:48:28.142414+0000","last_fullsized":"2024-08-03T20:48:28.142414+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:48:25.097349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:48:25.097349+0000","last_clean_scrub_stamp":"2024-08-03T20:48:25.097349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T00:15:51.465753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":13,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-08-03T20:48:28.142352+0000","last_change":"2024-08-03T20:48:27.147280+0000","last_active":"2024-08-03T20:48:28.142352+0000","last_peered":"2024-08-03T20:48:28.142352+0000","last_clean":"2024-08-03T20:48:28.142352+0000","last_became_active":"2024-08-03T20:48:27.143350+0000","last_became_peered":"2024-08-03T20:48:27.143350+0000","last_unstale":"2024-08-03T20:48:28.142352+0000","last_undegraded":"2024-08-03T20:48:28.142352+0000","last_fullsized":"2024-08-03T20:48:28.142352+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:48:26.104177+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:48:26.104177+0000","last_clean_scrub_stamp":"2024-08-03T20:48:26.104177+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T03:20:15.238833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-08-03T20:48:28.145353+0000","last_change":"2024-08-03T20:48:26.133691+0000","last_active":"2024-08-03T20:48:28.145353+0000","last_peered":"2024-08-03T20:48:28.145353+0000","last_clean":"2024-08-03T20:48:28.145353+0000","last_became_active":"2024-08-03T20:48:26.133341+0000","last_became_peered":"2024-08-03T20:48:26.133341+0000","last_unstale":"2024-08-03T20:48:28.145353+0000","last_undegraded":"2024-08-03T20:48:28.145353+0000","last_fullsized":"2024-08-03T20:48:28.145353+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:48:25.097349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:48:25.097349+0000","last_clean_scrub_stamp":"2024-08-03T20:48:25.097349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T01:34:49.869344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":13,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-08-03T20:48:28.421440+0000","last_change":"2024-08-03T20:48:27.137770+0000","last_active":"2024-08-03T20:48:28.421440+0000","last_peered":"2024-08-03T20:48:28.421440+0000","last_clean":"2024-08-03T20:48:28.421440+0000","last_became_active":"2024-08-03T20:48:27.137438+0000","last_became_peered":"2024-08-03T20:48:27.137438+0000","last_unstale":"2024-08-03T20:48:28.421440+0000","last_undegraded":"2024-08-03T20:48:28.421440+0000","last_fullsized":"2024-08-03T20:48:28.421440+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:48:26.104177+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:48:26.104177+0000","last_clean_scrub_stamp":"2024-08-03T20:48:26.104177+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-04T20:52:50.166997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-08-03T20:48:28.129220+0000","last_change":"2024-08-03T20:48:26.135756+0000","last_active":"2024-08-03T20:48:28.129220+0000","last_peered":"2024-08-03T20:48:28.129220+0000","last_clean":"2024-08-03T20:48:28.129220+0000","last_became_active":"2024-08-03T20:48:26.134565+0000","last_became_peered":"2024-08-03T20:48:26.134565+0000","last_unstale":"2024-08-03T20:48:28.129220+0000","last_undegraded":"2024-08-03T20:48:28.129220+0000","last_fullsized":"2024-08-03T20:48:28.129220+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:48:25.097349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:48:25.097349+0000","last_clean_scrub_stamp":"2024-08-03T20:48:25.097349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-04T22:13:46.608900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":13,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-08-03T20:48:28.129094+0000","last_change":"2024-08-03T20:48:27.128616+0000","last_active":"2024-08-03T20:48:28.129094+0000","last_peered":"2024-08-03T20:48:28.129094+0000","last_clean":"2024-08-03T20:48:28.129094+0000","last_became_active":"2024-08-03T20:48:27.128257+0000","last_became_peered":"2024-08-03T20:48:27.128257+0000","last_unstale":"2024-08-03T20:48:28.129094+0000","last_undegraded":"2024-08-03T20:48:28.129094+0000","last_fullsized":"2024-08-03T20:48:28.129094+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:48:26.104177+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:48:26.104177+0000","last_clean_scrub_stamp":"2024-08-03T20:48:26.104177+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T03:27:54.323107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-08-03T20:48:28.421552+0000","last_change":"2024-08-03T20:48:26.143986+0000","last_active":"2024-08-03T20:48:28.421552+0000","last_peered":"2024-08-03T20:48:28.421552+0000","last_clean":"2024-08-03T20:48:28.421552+0000","last_became_active":"2024-08-03T20:48:26.143665+0000","last_became_peered":"2024-08-03T20:48:26.143665+0000","last_unstale":"2024-08-03T20:48:28.421552+0000","last_undegraded":"2024-08-03T20:48:28.421552+0000","last_fullsized":"2024-08-03T20:48:28.421552+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:48:25.097349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:48:25.097349+0000","last_clean_scrub_stamp":"2024-08-03T20:48:25.097349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T06:38:55.256186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":13,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-08-03T20:48:28.424416+0000","last_change":"2024-08-03T20:48:27.145565+0000","last_active":"2024-08-03T20:48:28.424416+0000","last_peered":"2024-08-03T20:48:28.424416+0000","last_clean":"2024-08-03T20:48:28.424416+0000","last_became_active":"2024-08-03T20:48:27.144559+0000","last_became_peered":"2024-08-03T20:48:27.144559+0000","last_unstale":"2024-08-03T20:48:28.424416+0000","last_undegraded":"2024-08-03T20:48:28.424416+0000","last_fullsized":"2024-08-03T20:48:28.424416+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:48:26.104177+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:48:26.104177+0000","last_clean_scrub_stamp":"2024-08-03T20:48:26.104177+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-04T22:01:58.560189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-08-03T20:48:28.422263+0000","last_change":"2024-08-03T20:48:26.142596+0000","last_active":"2024-08-03T20:48:28.422263+0000","last_peered":"2024-08-03T20:48:28.422263+0000","last_clean":"2024-08-03T20:48:28.422263+0000","last_became_active":"2024-08-03T20:48:26.142251+0000","last_became_peered":"2024-08-03T20:48:26.142251+0000","last_unstale":"2024-08-03T20:48:28.422263+0000","last_undegraded":"2024-08-03T20:48:28.422263+0000","last_fullsized":"2024-08-03T20:48:28.422263+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:48:25.097349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:48:25.097349+0000","last_clean_scrub_stamp":"2024-08-03T20:48:25.097349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T03:23:20.675312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"16'72","reported_seq":129,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-08-03T20:48:28.424169+0000","last_change":"2024-08-03T20:47:30.547632+0000","last_active":"2024-08-03T20:48:28.424169+0000","last_peered":"2024-08-03T20:48:28.424169+0000","last_clean":"2024-08-03T20:48:28.424169+0000","last_became_active":"2024-08-03T20:47:30.546255+0000","last_became_peered":"2024-08-03T20:47:30.546255+0000","last_unstale":"2024-08-03T20:48:28.424169+0000","last_undegraded":"2024-08-03T20:48:28.424169+0000","last_fullsized":"2024-08-03T20:48:28.424169+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:47:28.874451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:47:28.874451+0000","last_clean_scrub_stamp":"2024-08-03T20:47:28.874451+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T00:19:40.548812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509455,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32112,"kb_used_data":928,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339728,"statfs":{"total":96636764160,"available":96603881472,"internally_reserved":0,"allocated":950272,"data_stored":726191,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":6,"up_from":15,"seq":64424509456,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31532,"kb_used_data":352,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340308,"statfs":{"total":96636764160,"available":96604475392,"internally_reserved":0,"allocated":360448,"data_stored":136349,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":5,"up_from":15,"seq":64424509456,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31532,"kb_used_data":352,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340308,"statfs":{"total":96636764160,"available":96604475392,"internally_reserved":0,"allocated":360448,"data_stored":136405,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509455,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31540,"kb_used_data":360,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340300,"statfs":{"total":96636764160,"available":96604467200,"internally_reserved":0,"allocated":368640,"data_stored":137367,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509457,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32108,"kb_used_data":924,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339732,"statfs":{"total":96636764160,"available":96603885568,"internally_reserved":0,"allocated":946176,"data_stored":726101,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542160,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31520,"kb_used_data":340,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340320,"statfs":{"total":96636764160,"available":96604487680,"internally_reserved":0,"allocated":348160,"data_stored":129537,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":3,"up_from":14,"seq":60129542158,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31528,"kb_used_data":348,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340312,"statfs":{"total":96636764160,"available":96604479488,"internally_reserved":0,"allocated":356352,"data_stored":131067,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":2,"up_from":14,"seq":60129542159,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31532,"kb_used_data":348,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340308,"statfs":{"total":96636764160,"available":96604475392,"internally_reserved":0,"allocated":356352,"data_stored":135803,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-08-03T20:48:32.154 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-08-03T20:48:32.154 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-08-03T20:48:32.425 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:32.423+0000 7f4559600640 1 -- 172.21.15.32:0/2264196898 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f45540cfb40 msgr2=0x7f4554162820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:48:32.425 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:32.423+0000 7f4559600640 1 --2- 172.21.15.32:0/2264196898 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f45540cfb40 0x7f4554162820 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f454c005fd0 tx=0x7f454c02aa90 comp rx=0 tx=0).stop 2024-08-03T20:48:32.425 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:32.423+0000 7f4559600640 1 -- 172.21.15.32:0/2264196898 shutdown_connections 2024-08-03T20:48:32.426 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:32.423+0000 7f4559600640 1 --2- 172.21.15.32:0/2264196898 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f45540cfb40 0x7f4554162820 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:32.426 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:32.423+0000 7f4559600640 1 --2- 172.21.15.32:0/2264196898 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f45540d26e0 0x7f45540d4090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:32.426 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:32.423+0000 7f4559600640 1 -- 172.21.15.32:0/2264196898 >> 172.21.15.32:0/2264196898 conn(0x7f455408d7e0 msgr2=0x7f455409d410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:48:32.426 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:32.423+0000 7f4559600640 1 -- 172.21.15.32:0/2264196898 shutdown_connections 2024-08-03T20:48:32.426 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:32.424+0000 7f4559600640 1 -- 172.21.15.32:0/2264196898 wait complete. 2024-08-03T20:48:32.427 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:32.426+0000 7f4559600640 1 Processor -- start 2024-08-03T20:48:32.427 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:32.426+0000 7f4559600640 1 -- start start 2024-08-03T20:48:32.430 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:32.427+0000 7f4559600640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f45540cfb40 0x7f45541e93e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:48:32.430 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:32.427+0000 7f4559600640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f45540d26e0 0x7f455415e490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:48:32.431 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:32.427+0000 7f4559600640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f45540d45d0 0x7f45540b8930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:48:32.431 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:32.427+0000 7f4559600640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f45540b53a0 con 0x7f45540d45d0 2024-08-03T20:48:32.431 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:32.427+0000 7f4559600640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f45540b5220 con 0x7f45540d26e0 2024-08-03T20:48:32.431 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:32.427+0000 7f4559600640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7f4554162d60 con 0x7f45540cfb40 2024-08-03T20:48:32.431 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:32.428+0000 7f4558c00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f45540d45d0 0x7f45540b8930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:48:32.431 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:32.428+0000 7f4558c00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f45540d45d0 0x7f45540b8930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:46492/0 (socket says 172.21.15.32:46492) 2024-08-03T20:48:32.431 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:32.428+0000 7f4558c00640 1 -- 172.21.15.32:0/2819977905 learned_addr learned my addr 172.21.15.32:0/2819977905 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:48:32.431 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:32.428+0000 7f4553e00640 1 --2- 172.21.15.32:0/2819977905 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f45540cfb40 0x7f45541e93e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:48:32.431 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:32.428+0000 7f454be00640 1 --2- 172.21.15.32:0/2819977905 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f45540d26e0 0x7f455415e490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:48:32.431 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:32.428+0000 7f4558c00640 1 -- 172.21.15.32:0/2819977905 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f45540cfb40 msgr2=0x7f45541e93e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:48:32.431 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:32.428+0000 7f4558c00640 1 --2- 172.21.15.32:0/2819977905 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f45540cfb40 0x7f45541e93e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:32.431 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:32.428+0000 7f4558c00640 1 -- 172.21.15.32:0/2819977905 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f45540d26e0 msgr2=0x7f455415e490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:48:32.431 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:32.428+0000 7f4558c00640 1 --2- 172.21.15.32:0/2819977905 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f45540d26e0 0x7f455415e490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:32.431 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:32.428+0000 7f4558c00640 1 -- 172.21.15.32:0/2819977905 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f455415e9d0 con 0x7f45540d45d0 2024-08-03T20:48:32.432 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:32.428+0000 7f4553e00640 1 --2- 172.21.15.32:0/2819977905 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f45540cfb40 0x7f45541e93e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-03T20:48:32.432 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:32.428+0000 7f4558c00640 1 --2- 172.21.15.32:0/2819977905 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f45540d45d0 0x7f45540b8930 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f454c009bf0 tx=0x7f454c003ce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:48:32.432 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:32.429+0000 7f4550c00640 1 -- 172.21.15.32:0/2819977905 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f454c044070 con 0x7f45540d45d0 2024-08-03T20:48:32.432 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:32.429+0000 7f4550c00640 1 -- 172.21.15.32:0/2819977905 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f454c004250 con 0x7f45540d45d0 2024-08-03T20:48:32.432 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:32.429+0000 7f4559600640 1 -- 172.21.15.32:0/2819977905 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f45540cd110 con 0x7f45540d45d0 2024-08-03T20:48:32.432 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:32.429+0000 7f4550c00640 1 -- 172.21.15.32:0/2819977905 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f454c004660 con 0x7f45540d45d0 2024-08-03T20:48:32.432 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:32.429+0000 7f4559600640 1 -- 172.21.15.32:0/2819977905 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f45540cd6d0 con 0x7f45540d45d0 2024-08-03T20:48:32.433 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:32.430+0000 7f4559600640 1 -- 172.21.15.32:0/2819977905 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f45540cce80 con 0x7f45540d45d0 2024-08-03T20:48:32.433 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:32.431+0000 7f4550c00640 1 -- 172.21.15.32:0/2819977905 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f454c047070 con 0x7f45540d45d0 2024-08-03T20:48:32.433 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:32.432+0000 7f4550c00640 1 --2- 172.21.15.32:0/2819977905 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f4530077010 0x7f4530079520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:48:32.434 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:32.432+0000 7f4550c00640 1 -- 172.21.15.32:0/2819977905 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(32..32 src has 1..32) ==== 5709+0+0 (secure 0 0 0) 0x7f454c0b9960 con 0x7f45540d45d0 2024-08-03T20:48:32.434 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:32.433+0000 7f4553e00640 1 --2- 172.21.15.32:0/2819977905 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f4530077010 0x7f4530079520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:48:32.434 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:32.433+0000 7f4553e00640 1 --2- 172.21.15.32:0/2819977905 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f4530077010 0x7f4530079520 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f45540d4f10 tx=0x7f4544005f10 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:48:32.438 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:32.436+0000 7f4550c00640 1 -- 172.21.15.32:0/2819977905 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7f454c032030 con 0x7f45540d45d0 2024-08-03T20:48:32.715 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:32.714+0000 7f4559600640 1 -- 172.21.15.32:0/2819977905 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) -- 0x7f45540d4090 con 0x7f45540d45d0 2024-08-03T20:48:32.719 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:32.717+0000 7f4550c00640 1 -- 172.21.15.32:0/2819977905 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v11) ==== 190+0+147 (secure 0 0 0) 0x7f454c082ad0 con 0x7f45540d45d0 2024-08-03T20:48:32.719 INFO:teuthology.orchestra.run.smithi032.stdout:[client.0] 2024-08-03T20:48:32.719 INFO:teuthology.orchestra.run.smithi032.stdout: key = AQAwl65msgO3NRAAHG/WuMTK9c4lDXocXvqWnA== 2024-08-03T20:48:32.719 INFO:teuthology.orchestra.run.smithi032.stdout: caps mds = "allow" 2024-08-03T20:48:32.719 INFO:teuthology.orchestra.run.smithi032.stdout: caps mon = "allow r" 2024-08-03T20:48:32.719 INFO:teuthology.orchestra.run.smithi032.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2024-08-03T20:48:32.720 INFO:teuthology.orchestra.run.smithi032.stderr:updated caps for client.0 2024-08-03T20:48:32.721 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:32.720+0000 7f4559600640 1 -- 172.21.15.32:0/2819977905 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f4530077010 msgr2=0x7f4530079520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:48:32.721 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:32.720+0000 7f4559600640 1 --2- 172.21.15.32:0/2819977905 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f4530077010 0x7f4530079520 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f45540d4f10 tx=0x7f4544005f10 comp rx=0 tx=0).stop 2024-08-03T20:48:32.722 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:32.720+0000 7f4559600640 1 -- 172.21.15.32:0/2819977905 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f45540d45d0 msgr2=0x7f45540b8930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:48:32.722 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:32.720+0000 7f4559600640 1 --2- 172.21.15.32:0/2819977905 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f45540d45d0 0x7f45540b8930 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f454c009bf0 tx=0x7f454c003ce0 comp rx=0 tx=0).stop 2024-08-03T20:48:32.722 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:32.720+0000 7f4559600640 1 -- 172.21.15.32:0/2819977905 shutdown_connections 2024-08-03T20:48:32.722 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:32.720+0000 7f4559600640 1 --2- 172.21.15.32:0/2819977905 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f4530077010 0x7f4530079520 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:32.722 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:32.720+0000 7f4559600640 1 --2- 172.21.15.32:0/2819977905 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f45540d26e0 0x7f455415e490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:32.722 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:32.720+0000 7f4559600640 1 --2- 172.21.15.32:0/2819977905 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f45540d45d0 0x7f45540b8930 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:32.722 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:32.720+0000 7f4559600640 1 --2- 172.21.15.32:0/2819977905 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f45540cfb40 0x7f45541e93e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:32.722 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:32.720+0000 7f4559600640 1 -- 172.21.15.32:0/2819977905 >> 172.21.15.32:0/2819977905 conn(0x7f455408d7e0 msgr2=0x7f45540b41f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:48:32.722 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:32.720+0000 7f4559600640 1 -- 172.21.15.32:0/2819977905 shutdown_connections 2024-08-03T20:48:32.722 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:32.720+0000 7f4559600640 1 -- 172.21.15.32:0/2819977905 wait complete. 2024-08-03T20:48:32.746 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-08-03T20:48:32.974 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:32.972+0000 7f7307200640 1 -- 172.21.15.32:0/787575126 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f73080cffe0 msgr2=0x7f73080b3e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:48:32.974 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:32.972+0000 7f7307200640 1 --2- 172.21.15.32:0/787575126 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f73080cffe0 0x7f73080b3e90 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f72f4009830 tx=0x7f72f402a540 comp rx=0 tx=0).stop 2024-08-03T20:48:32.974 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:32.972+0000 7f7307200640 1 -- 172.21.15.32:0/787575126 shutdown_connections 2024-08-03T20:48:32.974 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:32.972+0000 7f7307200640 1 --2- 172.21.15.32:0/787575126 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f73080cffe0 0x7f73080b3e90 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:32.975 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:32.972+0000 7f7307200640 1 -- 172.21.15.32:0/787575126 >> 172.21.15.32:0/787575126 conn(0x7f730808d7a0 msgr2=0x7f730809d3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:48:32.975 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:32.972+0000 7f7307200640 1 -- 172.21.15.32:0/787575126 shutdown_connections 2024-08-03T20:48:32.975 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:32.972+0000 7f7307200640 1 -- 172.21.15.32:0/787575126 wait complete. 2024-08-03T20:48:32.976 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:32.975+0000 7f7307200640 1 Processor -- start 2024-08-03T20:48:32.976 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:32.975+0000 7f7307200640 1 -- start start 2024-08-03T20:48:32.980 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:32.976+0000 7f7307200640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f73080b43d0 0x7f73080cc3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:48:32.980 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:32.976+0000 7f7307200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f73080cffe0 0x7f73080caac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:48:32.980 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:32.976+0000 7f7307200640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f73080d26e0 0x7f730815c8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:48:32.980 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:32.976+0000 7f7307200640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f73080d4310 con 0x7f73080cffe0 2024-08-03T20:48:32.980 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:32.976+0000 7f7307200640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f73080d4190 con 0x7f73080b43d0 2024-08-03T20:48:32.980 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:32.976+0000 7f7307200640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7f73080d4490 con 0x7f73080d26e0 2024-08-03T20:48:32.981 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:32.977+0000 7f7305400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f73080cffe0 0x7f73080caac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:48:32.981 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:32.977+0000 7f7305400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f73080cffe0 0x7f73080caac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:46504/0 (socket says 172.21.15.32:46504) 2024-08-03T20:48:32.981 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:32.977+0000 7f7305400640 1 -- 172.21.15.32:0/2744421119 learned_addr learned my addr 172.21.15.32:0/2744421119 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:48:32.981 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:32.977+0000 7f7305e00640 1 --2- 172.21.15.32:0/2744421119 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f73080b43d0 0x7f73080cc3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:48:32.981 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:32.977+0000 7f7306800640 1 --2- 172.21.15.32:0/2744421119 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f73080d26e0 0x7f730815c8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:48:32.981 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:32.977+0000 7f7305400640 1 -- 172.21.15.32:0/2744421119 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f73080d26e0 msgr2=0x7f730815c8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:48:32.981 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:32.977+0000 7f7305400640 1 --2- 172.21.15.32:0/2744421119 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f73080d26e0 0x7f730815c8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:32.981 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:32.977+0000 7f7305400640 1 -- 172.21.15.32:0/2744421119 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f73080b43d0 msgr2=0x7f73080cc3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:48:32.982 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:32.977+0000 7f7305400640 1 --2- 172.21.15.32:0/2744421119 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f73080b43d0 0x7f73080cc3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:32.982 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:32.977+0000 7f7305400640 1 -- 172.21.15.32:0/2744421119 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f73080cdc70 con 0x7f73080cffe0 2024-08-03T20:48:32.982 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:32.977+0000 7f7305e00640 1 --2- 172.21.15.32:0/2744421119 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f73080b43d0 0x7f73080cc3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-03T20:48:32.982 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:32.977+0000 7f7306800640 1 --2- 172.21.15.32:0/2744421119 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f73080d26e0 0x7f730815c8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-03T20:48:32.982 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:32.977+0000 7f7305400640 1 --2- 172.21.15.32:0/2744421119 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f73080cffe0 0x7f73080caac0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f72f4009950 tx=0x7f72f4030650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:48:32.982 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:32.978+0000 7f72fe000640 1 -- 172.21.15.32:0/2744421119 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72f403f070 con 0x7f73080cffe0 2024-08-03T20:48:32.982 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:32.978+0000 7f72fe000640 1 -- 172.21.15.32:0/2744421119 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f72f4030d40 con 0x7f73080cffe0 2024-08-03T20:48:32.982 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:32.978+0000 7f72fe000640 1 -- 172.21.15.32:0/2744421119 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72f4004280 con 0x7f73080cffe0 2024-08-03T20:48:32.982 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:32.978+0000 7f7307200640 1 -- 172.21.15.32:0/2744421119 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f73080cdf50 con 0x7f73080cffe0 2024-08-03T20:48:32.982 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:32.978+0000 7f7307200640 1 -- 172.21.15.32:0/2744421119 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f730815cde0 con 0x7f73080cffe0 2024-08-03T20:48:32.983 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:32.979+0000 7f7307200640 1 -- 172.21.15.32:0/2744421119 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7308161060 con 0x7f73080cffe0 2024-08-03T20:48:32.983 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:32.980+0000 7f72fe000640 1 -- 172.21.15.32:0/2744421119 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f72f40035f0 con 0x7f73080cffe0 2024-08-03T20:48:32.983 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:32.981+0000 7f72fe000640 1 --2- 172.21.15.32:0/2744421119 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f72e8077010 0x7f72e8079520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:48:32.983 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:32.981+0000 7f72fe000640 1 -- 172.21.15.32:0/2744421119 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(32..32 src has 1..32) ==== 5709+0+0 (secure 0 0 0) 0x7f72f40b9390 con 0x7f73080cffe0 2024-08-03T20:48:32.984 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:32.982+0000 7f7305e00640 1 --2- 172.21.15.32:0/2744421119 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f72e8077010 0x7f72e8079520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:48:32.984 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:32.982+0000 7f7305e00640 1 --2- 172.21.15.32:0/2744421119 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f72e8077010 0x7f72e8079520 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f72f8001220 tx=0x7f72f8009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:48:32.987 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:32.985+0000 7f72fe000640 1 -- 172.21.15.32:0/2744421119 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7f7308161060 con 0x7f73080cffe0 2024-08-03T20:48:33.232 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:33.231+0000 7f7307200640 1 -- 172.21.15.32:0/2744421119 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f73080c8890 con 0x7f73080cffe0 2024-08-03T20:48:33.233 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:33.231+0000 7f72fe000640 1 -- 172.21.15.32:0/2744421119 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) ==== 94+0+5229 (secure 0 0 0) 0x7f72f40824f0 con 0x7f73080cffe0 2024-08-03T20:48:33.233 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-03T20:48:33.233 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":17,"btime":"2024-08-03T20:48:30:142972+0000","default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4643,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6836/4002526832","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6834","nonce":4002526832},{"type":"v1","addr":"172.21.15.118:6836","nonce":4002526832}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":4646,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6837/3036256187","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6835","nonce":3036256187},{"type":"v1","addr":"172.21.15.118:6837","nonce":3036256187}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":4652,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6834/2104418334","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6833","nonce":2104418334},{"type":"v1","addr":"172.21.15.32:6834","nonce":2104418334}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-08-03T20:48:26.652521+0000","modified":"2024-08-03T20:48:29.661524+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4866},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4866":{"gid":4866,"name":"a","rank":0,"incarnation":13,"state":"up:active","state_seq":3,"addr":"172.21.15.32:6835/1186001746","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":1186001746},{"type":"v1","addr":"172.21.15.32:6835","nonce":1186001746}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4866,"qdb_cluster":[4866]},"id":2}]} 2024-08-03T20:48:33.234 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 17 2024-08-03T20:48:33.235 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:33.234+0000 7f7307200640 1 -- 172.21.15.32:0/2744421119 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f72e8077010 msgr2=0x7f72e8079520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:48:33.235 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:33.234+0000 7f7307200640 1 --2- 172.21.15.32:0/2744421119 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f72e8077010 0x7f72e8079520 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f72f8001220 tx=0x7f72f8009040 comp rx=0 tx=0).stop 2024-08-03T20:48:33.236 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:33.234+0000 7f7307200640 1 -- 172.21.15.32:0/2744421119 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f73080cffe0 msgr2=0x7f73080caac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:48:33.236 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:33.234+0000 7f7307200640 1 --2- 172.21.15.32:0/2744421119 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f73080cffe0 0x7f73080caac0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f72f4009950 tx=0x7f72f4030650 comp rx=0 tx=0).stop 2024-08-03T20:48:33.236 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:33.234+0000 7f7307200640 1 -- 172.21.15.32:0/2744421119 shutdown_connections 2024-08-03T20:48:33.236 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:33.234+0000 7f7307200640 1 --2- 172.21.15.32:0/2744421119 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f72e8077010 0x7f72e8079520 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:33.236 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:33.234+0000 7f7307200640 1 --2- 172.21.15.32:0/2744421119 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f73080d26e0 0x7f730815c8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:33.236 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:33.234+0000 7f7307200640 1 --2- 172.21.15.32:0/2744421119 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f73080cffe0 0x7f73080caac0 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:33.236 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:33.234+0000 7f7307200640 1 --2- 172.21.15.32:0/2744421119 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f73080b43d0 0x7f73080cc3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:33.236 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:33.234+0000 7f7307200640 1 -- 172.21.15.32:0/2744421119 >> 172.21.15.32:0/2744421119 conn(0x7f730808d7a0 msgr2=0x7f73081602a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:48:33.236 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:33.234+0000 7f7307200640 1 -- 172.21.15.32:0/2744421119 shutdown_connections 2024-08-03T20:48:33.236 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:33.234+0000 7f7307200640 1 -- 172.21.15.32:0/2744421119 wait complete. 2024-08-03T20:48:33.664 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 17, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-08-03T20:48:26.652521+0000', 'modified': '2024-08-03T20:48:29.661524+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 4866}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4866': {'gid': 4866, 'name': 'a', 'rank': 0, 'incarnation': 13, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.32:6835/1186001746', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6832', 'nonce': 1186001746}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 1186001746}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [5], 'metadata_pool': 4, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4866, 'qdb_cluster': [4866]} 2024-08-03T20:48:33.665 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-08-03T20:48:33.665 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2024-08-03T20:48:33.897 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:33.895+0000 7fd30a400640 1 -- 172.21.15.32:0/3787595582 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd3040af020 msgr2=0x7fd3040742b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:48:33.897 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:33.895+0000 7fd30a400640 1 --2- 172.21.15.32:0/3787595582 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd3040af020 0x7fd3040742b0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7fd304160f00 tx=0x7fd2f802a560 comp rx=0 tx=0).stop 2024-08-03T20:48:33.897 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:33.895+0000 7fd30a400640 1 -- 172.21.15.32:0/3787595582 shutdown_connections 2024-08-03T20:48:33.897 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:33.895+0000 7fd30a400640 1 --2- 172.21.15.32:0/3787595582 >> v2:172.21.15.118:3300/0 conn(0x7fd3040acd80 0x7fd3040ad190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:33.897 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:33.895+0000 7fd30a400640 1 --2- 172.21.15.32:0/3787595582 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd3040af020 0x7fd3040742b0 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:33.897 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:33.895+0000 7fd30a400640 1 -- 172.21.15.32:0/3787595582 >> 172.21.15.32:0/3787595582 conn(0x7fd3040b2120 msgr2=0x7fd3040cfd60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:48:33.897 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:33.895+0000 7fd30a400640 1 -- 172.21.15.32:0/3787595582 shutdown_connections 2024-08-03T20:48:33.897 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:33.895+0000 7fd30a400640 1 -- 172.21.15.32:0/3787595582 wait complete. 2024-08-03T20:48:33.898 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:33.897+0000 7fd30a400640 1 Processor -- start 2024-08-03T20:48:33.898 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:33.897+0000 7fd30a400640 1 -- start start 2024-08-03T20:48:33.902 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:33.899+0000 7fd30a400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd30406f440 0x7fd3040cebc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:48:33.902 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:33.899+0000 7fd30a400640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fd3040acd80 0x7fd304155970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:48:33.902 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:33.899+0000 7fd30a400640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fd3040af020 0x7fd30415b350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:48:33.902 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:33.899+0000 7fd30a400640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fd3041601e0 con 0x7fd30406f440 2024-08-03T20:48:33.902 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:33.899+0000 7fd30a400640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7fd304160060 con 0x7fd3040af020 2024-08-03T20:48:33.902 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:33.899+0000 7fd30a400640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7fd304160360 con 0x7fd3040acd80 2024-08-03T20:48:33.902 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:33.899+0000 7fd309000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd30406f440 0x7fd3040cebc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:48:33.903 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:33.899+0000 7fd309000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd30406f440 0x7fd3040cebc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:46526/0 (socket says 172.21.15.32:46526) 2024-08-03T20:48:33.903 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:33.899+0000 7fd309000640 1 -- 172.21.15.32:0/3752758950 learned_addr learned my addr 172.21.15.32:0/3752758950 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:48:33.903 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:33.899+0000 7fd303e00640 1 --2- 172.21.15.32:0/3752758950 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fd3040acd80 0x7fd304155970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:48:33.903 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:33.899+0000 7fd309a00640 1 --2- 172.21.15.32:0/3752758950 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fd3040af020 0x7fd30415b350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:48:33.903 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:33.899+0000 7fd309000640 1 -- 172.21.15.32:0/3752758950 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fd3040acd80 msgr2=0x7fd304155970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:48:33.903 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:33.899+0000 7fd309000640 1 --2- 172.21.15.32:0/3752758950 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fd3040acd80 0x7fd304155970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:33.903 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:33.899+0000 7fd309000640 1 -- 172.21.15.32:0/3752758950 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fd3040af020 msgr2=0x7fd30415b350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:48:33.903 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:33.899+0000 7fd309000640 1 --2- 172.21.15.32:0/3752758950 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fd3040af020 0x7fd30415b350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:33.903 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:33.900+0000 7fd309000640 1 -- 172.21.15.32:0/3752758950 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd3040cf100 con 0x7fd30406f440 2024-08-03T20:48:33.903 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:33.900+0000 7fd309a00640 1 --2- 172.21.15.32:0/3752758950 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fd3040af020 0x7fd30415b350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-03T20:48:33.904 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:33.900+0000 7fd309000640 1 --2- 172.21.15.32:0/3752758950 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd30406f440 0x7fd3040cebc0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7fd2f8034040 tx=0x7fd2f8003820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:48:33.904 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:33.900+0000 7fd300c00640 1 -- 172.21.15.32:0/3752758950 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2f803e030 con 0x7fd30406f440 2024-08-03T20:48:33.904 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:33.900+0000 7fd300c00640 1 -- 172.21.15.32:0/3752758950 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fd2f8003d10 con 0x7fd30406f440 2024-08-03T20:48:33.904 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:33.900+0000 7fd30a400640 1 -- 172.21.15.32:0/3752758950 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd3041f9150 con 0x7fd30406f440 2024-08-03T20:48:33.904 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:33.900+0000 7fd300c00640 1 -- 172.21.15.32:0/3752758950 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2f8035040 con 0x7fd30406f440 2024-08-03T20:48:33.904 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:33.900+0000 7fd30a400640 1 -- 172.21.15.32:0/3752758950 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd3041f9710 con 0x7fd30406f440 2024-08-03T20:48:33.905 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:33.903+0000 7fd300c00640 1 -- 172.21.15.32:0/3752758950 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7fd2f8041030 con 0x7fd30406f440 2024-08-03T20:48:33.905 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:33.904+0000 7fd300c00640 1 --2- 172.21.15.32:0/3752758950 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fd2e4077010 0x7fd2e4079520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:48:33.905 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:33.904+0000 7fd300c00640 1 -- 172.21.15.32:0/3752758950 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(32..32 src has 1..32) ==== 5709+0+0 (secure 0 0 0) 0x7fd2f80b8fb0 con 0x7fd30406f440 2024-08-03T20:48:33.906 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:33.904+0000 7fd303e00640 1 --2- 172.21.15.32:0/3752758950 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fd2e4077010 0x7fd2e4079520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:48:33.906 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:33.904+0000 7fd312be3740 1 Processor -- start 2024-08-03T20:48:33.906 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:33.905+0000 7fd312be3740 1 -- start start 2024-08-03T20:48:33.906 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:33.905+0000 7fd303e00640 1 --2- 172.21.15.32:0/3752758950 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fd2e4077010 0x7fd2e4079520 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fd3040704e0 tx=0x7fd2f0005f90 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:48:33.909 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:33.906+0000 7fd312be3740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b5e908d270 0x55b5e9478080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:48:33.909 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:33.906+0000 7fd312be3740 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55b5e90643b0 0x55b5e944ced0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:48:33.909 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:33.906+0000 7fd312be3740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55b5e906c590 0x55b5e944d410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:48:33.909 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:33.906+0000 7fd312be3740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55b5e93a5a60 con 0x55b5e908d270 2024-08-03T20:48:33.909 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:33.906+0000 7fd312be3740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x55b5e9394f60 con 0x55b5e906c590 2024-08-03T20:48:33.909 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:33.906+0000 7fd312be3740 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x55b5e9398fa0 con 0x55b5e90643b0 2024-08-03T20:48:33.909 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:33.906+0000 7fd309000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b5e908d270 0x55b5e9478080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:48:33.910 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:33.906+0000 7fd309000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b5e908d270 0x55b5e9478080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:46528/0 (socket says 172.21.15.32:46528) 2024-08-03T20:48:33.910 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:33.906+0000 7fd309000640 1 -- 172.21.15.32:0/2206121517 learned_addr learned my addr 172.21.15.32:0/2206121517 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:48:33.910 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:33.907+0000 7fd303e00640 1 --2- 172.21.15.32:0/2206121517 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55b5e90643b0 0x55b5e944ced0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:48:33.910 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:33.907+0000 7fd309a00640 1 --2- 172.21.15.32:0/2206121517 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55b5e906c590 0x55b5e944d410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:48:33.910 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:33.907+0000 7fd309000640 1 -- 172.21.15.32:0/2206121517 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55b5e90643b0 msgr2=0x55b5e944ced0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:48:33.910 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:33.907+0000 7fd309000640 1 --2- 172.21.15.32:0/2206121517 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55b5e90643b0 0x55b5e944ced0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:33.910 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:33.907+0000 7fd309000640 1 -- 172.21.15.32:0/2206121517 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55b5e906c590 msgr2=0x55b5e944d410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:48:33.910 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:33.907+0000 7fd309000640 1 --2- 172.21.15.32:0/2206121517 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55b5e906c590 0x55b5e944d410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:33.910 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:33.907+0000 7fd309000640 1 -- 172.21.15.32:0/2206121517 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b5e90f8850 con 0x55b5e908d270 2024-08-03T20:48:33.910 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:33.907+0000 7fd309000640 1 --2- 172.21.15.32:0/2206121517 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b5e908d270 0x55b5e9478080 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7fd2f8081ae0 tx=0x7fd2f8081560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1c56423654d165f9 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:48:33.910 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:33.907+0000 7fd2f5600640 1 -- 172.21.15.32:0/2206121517 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2f8048030 con 0x55b5e908d270 2024-08-03T20:48:33.910 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:33.907+0000 7fd2f5600640 1 -- 172.21.15.32:0/2206121517 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fd2f8081660 con 0x55b5e908d270 2024-08-03T20:48:33.911 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:33.907+0000 7fd2f5600640 1 -- 172.21.15.32:0/2206121517 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2f808a850 con 0x55b5e908d270 2024-08-03T20:48:33.911 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:33.908+0000 7fd312be3740 1 -- 172.21.15.32:0/2206121517 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b5e908d270 msgr2=0x55b5e9478080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:48:33.912 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:33.908+0000 7fd312be3740 1 --2- 172.21.15.32:0/2206121517 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b5e908d270 0x55b5e9478080 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7fd2f8081ae0 tx=0x7fd2f8081560 comp rx=0 tx=0).stop 2024-08-03T20:48:33.912 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:33.908+0000 7fd312be3740 1 -- 172.21.15.32:0/2206121517 shutdown_connections 2024-08-03T20:48:33.912 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:33.908+0000 7fd312be3740 1 --2- 172.21.15.32:0/2206121517 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55b5e906c590 0x55b5e944d410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:33.912 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:33.908+0000 7fd312be3740 1 --2- 172.21.15.32:0/2206121517 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55b5e90643b0 0x55b5e944ced0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:33.912 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:33.908+0000 7fd312be3740 1 --2- 172.21.15.32:0/2206121517 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b5e908d270 0x55b5e9478080 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:33.912 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:33.908+0000 7fd312be3740 1 -- 172.21.15.32:0/2206121517 >> 172.21.15.32:0/2206121517 conn(0x55b5e90813c0 msgr2=0x55b5e9433370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:48:33.912 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:33.908+0000 7fd312be3740 1 -- 172.21.15.32:0/2206121517 shutdown_connections 2024-08-03T20:48:33.912 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:33.908+0000 7fd312be3740 1 -- 172.21.15.32:0/2206121517 wait complete. 2024-08-03T20:48:33.912 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:33.909+0000 7fd312be3740 1 -- start start 2024-08-03T20:48:33.913 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:33.910+0000 7fd312be3740 1 Processor -- start 2024-08-03T20:48:33.916 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:33.911+0000 7fd312be3740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b5e90643b0 0x55b5e94712c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:48:33.916 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:33.911+0000 7fd312be3740 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55b5e906c590 0x55b5e9486630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:48:33.916 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:33.911+0000 7fd312be3740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55b5e908d270 0x55b5e9486b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:48:33.917 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:33.911+0000 7fd312be3740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55b5e9107620 con 0x55b5e90643b0 2024-08-03T20:48:33.917 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:33.911+0000 7fd312be3740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x55b5e9332160 con 0x55b5e908d270 2024-08-03T20:48:33.917 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:33.911+0000 7fd312be3740 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x55b5e93a5a60 con 0x55b5e906c590 2024-08-03T20:48:33.917 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:33.911+0000 7fd312be3740 10 client.0 _ll_register_callbacks cb 0x55b5e93416b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-03T20:48:33.917 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:33.911+0000 7fd309000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b5e90643b0 0x55b5e94712c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:48:33.917 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:33.911+0000 7fd309000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b5e90643b0 0x55b5e94712c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:46530/0 (socket says 172.21.15.32:46530) 2024-08-03T20:48:33.917 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:33.911+0000 7fd309000640 1 -- 172.21.15.32:0/2358210329 learned_addr learned my addr 172.21.15.32:0/2358210329 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:48:33.917 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:33.912+0000 7fd309000640 1 -- 172.21.15.32:0/2358210329 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55b5e906c590 msgr2=0x55b5e9486630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:48:33.917 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:33.912+0000 7fd309000640 1 --2- 172.21.15.32:0/2358210329 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55b5e906c590 0x55b5e9486630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:33.917 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:33.912+0000 7fd309000640 1 -- 172.21.15.32:0/2358210329 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55b5e908d270 msgr2=0x55b5e9486b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:48:33.917 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:33.912+0000 7fd309000640 1 --2- 172.21.15.32:0/2358210329 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55b5e908d270 0x55b5e9486b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:33.917 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:33.912+0000 7fd309000640 1 -- 172.21.15.32:0/2358210329 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b5e916f6b0 con 0x55b5e90643b0 2024-08-03T20:48:33.917 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:33.912+0000 7fd309000640 1 --2- 172.21.15.32:0/2358210329 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b5e90643b0 0x55b5e94712c0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7fd2f80099c0 tx=0x7fd2f808a8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=af44f3e8f2c17d53 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:48:33.918 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:33.912+0000 7fd2ea000640 10 client.0 ms_handle_connect on v2:172.21.15.32:3300/0 2024-08-03T20:48:33.918 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:33.912+0000 7fd2ea000640 1 -- 172.21.15.32:0/2358210329 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2f8048070 con 0x55b5e90643b0 2024-08-03T20:48:33.918 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:33.912+0000 7fd2ea000640 1 -- 172.21.15.32:0/2358210329 <== mon.0 v2:172.21.15.32:3300/0 2 ==== osd_map(32..32 src has 1..32) ==== 5709+0+0 (secure 0 0 0) 0x7fd2f8080fc0 con 0x55b5e90643b0 2024-08-03T20:48:33.918 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:33.912+0000 7fd312be3740 1 -- 172.21.15.32:0/2358210329 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd2f8081660 con 0x55b5e90643b0 2024-08-03T20:48:33.918 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:33.913+0000 7fd2ea000640 1 -- 172.21.15.32:0/2358210329 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_get_version_reply(handle=1 version=17) ==== 24+0+0 (secure 0 0 0) 0x7fd2f80a2050 con 0x55b5e90643b0 2024-08-03T20:48:33.918 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:33.913+0000 7fd312be3740 10 client.5031 fetch_fsmap learned FSMap version 17 2024-08-03T20:48:33.918 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:33.913+0000 7fd312be3740 1 -- 172.21.15.32:0/2358210329 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,fsmap=17,monmap=2+}) -- 0x55b5e9475d80 con 0x55b5e90643b0 2024-08-03T20:48:33.918 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:33.913+0000 7fd2ea000640 1 -- 172.21.15.32:0/2358210329 <== mon.0 v2:172.21.15.32:3300/0 4 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fd2f8081660 con 0x55b5e90643b0 2024-08-03T20:48:33.918 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:33.913+0000 7fd2ea000640 1 -- 172.21.15.32:0/2358210329 <== mon.0 v2:172.21.15.32:3300/0 5 ==== fsmap(e 17) ==== 3502+0+0 (secure 0 0 0) 0x7fd2f80a3240 con 0x55b5e90643b0 2024-08-03T20:48:33.918 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:33.913+0000 7fd312be3740 10 client.5031 fetch_fsmap finished waiting for FSMap version 17 2024-08-03T20:48:33.918 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:33.913+0000 7fd312be3740 10 client.5031 resolve_mds: resolved name 'a' to daemon mds.a 2024-08-03T20:48:33.918 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:33.913+0000 7fd312be3740 20 client.5031 populate_metadata read hostname 'smithi032' 2024-08-03T20:48:33.918 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:33.913+0000 7fd312be3740 1 --2- 172.21.15.32:0/2358210329 >> [v2:172.21.15.32:6832/1186001746,v1:172.21.15.32:6835/1186001746] conn(0x55b5e904be40 0x55b5e948aac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:48:33.918 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:33.913+0000 7fd312be3740 4 client.5031 mds_command: new command op to 4866 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-08-03T20:48:33.919 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:33.913+0000 7fd312be3740 1 -- 172.21.15.32:0/2358210329 --> [v2:172.21.15.32:6832/1186001746,v1:172.21.15.32:6835/1186001746] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b5e93a79b0 con 0x55b5e904be40 2024-08-03T20:48:33.919 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:33.913+0000 7fd303e00640 1 --2- 172.21.15.32:0/2358210329 >> [v2:172.21.15.32:6832/1186001746,v1:172.21.15.32:6835/1186001746] conn(0x55b5e904be40 0x55b5e948aac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:48:33.919 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:33.914+0000 7fd303e00640 1 --2- 172.21.15.32:0/2358210329 >> [v2:172.21.15.32:6832/1186001746,v1:172.21.15.32:6835/1186001746] conn(0x55b5e904be40 0x55b5e948aac0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7e1741a85953fdca server_cookie=4ba38b318d81e4fe in_seq=0 out_seq=0 2024-08-03T20:48:33.919 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:33.914+0000 7fd2ea000640 10 client.5031 ms_handle_connect on v2:172.21.15.32:6832/1186001746 2024-08-03T20:48:33.919 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:33.915+0000 7fd2ea000640 1 -- 172.21.15.32:0/2358210329 <== mds.0 v2:172.21.15.32:6832/1186001746 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55b5e93a79b0 con 0x55b5e904be40 2024-08-03T20:48:33.919 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:33.915+0000 7fd2ea000640 10 client.5031 handle_command_reply: tid=0 2024-08-03T20:48:33.919 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:33.915+0000 7fd312be3740 1 client.5031 shutdown 2024-08-03T20:48:33.920 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:33.915+0000 7fd312be3740 1 -- 172.21.15.32:0/2358210329 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b5e90643b0 msgr2=0x55b5e94712c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:48:33.920 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:33.915+0000 7fd312be3740 1 --2- 172.21.15.32:0/2358210329 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b5e90643b0 0x55b5e94712c0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7fd2f80099c0 tx=0x7fd2f808a8e0 comp rx=0 tx=0).stop 2024-08-03T20:48:33.920 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:33.915+0000 7fd312be3740 1 -- 172.21.15.32:0/2358210329 shutdown_connections 2024-08-03T20:48:33.920 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:33.915+0000 7fd312be3740 1 --2- 172.21.15.32:0/2358210329 >> [v2:172.21.15.32:6832/1186001746,v1:172.21.15.32:6835/1186001746] conn(0x55b5e904be40 0x55b5e948aac0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:33.920 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:33.915+0000 7fd312be3740 1 --2- 172.21.15.32:0/2358210329 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55b5e908d270 0x55b5e9486b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:33.920 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:33.915+0000 7fd312be3740 1 --2- 172.21.15.32:0/2358210329 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55b5e906c590 0x55b5e9486630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:33.920 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:33.915+0000 7fd312be3740 1 --2- 172.21.15.32:0/2358210329 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b5e90643b0 0x55b5e94712c0 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:33.920 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:33.915+0000 7fd2ea000640 0 client.5031 ms_handle_reset on v2:172.21.15.32:6832/1186001746 2024-08-03T20:48:33.921 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:33.915+0000 7fd312be3740 1 -- 172.21.15.32:0/2358210329 >> 172.21.15.32:0/2358210329 conn(0x55b5e90813c0 msgr2=0x55b5e9433340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:48:33.921 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:33.915+0000 7fd312be3740 1 -- 172.21.15.32:0/2358210329 shutdown_connections 2024-08-03T20:48:33.921 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:33.915+0000 7fd312be3740 1 -- 172.21.15.32:0/2358210329 wait complete. 2024-08-03T20:48:33.921 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:33.916+0000 7fd312be3740 20 client.5031 trim_cache size 0 max 16384 2024-08-03T20:48:33.937 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:33.936+0000 7fd312be3740 1 Processor -- start 2024-08-03T20:48:33.937 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:33.936+0000 7fd312be3740 1 -- start start 2024-08-03T20:48:33.939 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:33.937+0000 7fd312be3740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b5e90643b0 0x55b5e94334f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:48:33.939 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:33.937+0000 7fd312be3740 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55b5e906c590 0x55b5e947a280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:48:33.940 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:33.937+0000 7fd312be3740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55b5e908d270 0x55b5e947a7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:48:33.940 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:33.937+0000 7fd312be3740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fd3041f6560 con 0x55b5e90643b0 2024-08-03T20:48:33.940 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:33.937+0000 7fd312be3740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7fd3041f58c0 con 0x55b5e908d270 2024-08-03T20:48:33.940 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:33.937+0000 7fd312be3740 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7fd3041f5f10 con 0x55b5e906c590 2024-08-03T20:48:33.940 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:33.937+0000 7fd309000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b5e90643b0 0x55b5e94334f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:48:33.940 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:33.937+0000 7fd309000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b5e90643b0 0x55b5e94334f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:46532/0 (socket says 172.21.15.32:46532) 2024-08-03T20:48:33.940 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:33.937+0000 7fd309000640 1 -- 172.21.15.32:0/2391481772 learned_addr learned my addr 172.21.15.32:0/2391481772 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:48:33.940 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:33.938+0000 7fd309000640 1 -- 172.21.15.32:0/2391481772 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55b5e906c590 msgr2=0x55b5e947a280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:48:33.940 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:33.938+0000 7fd309000640 1 --2- 172.21.15.32:0/2391481772 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55b5e906c590 0x55b5e947a280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:33.940 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:33.938+0000 7fd309000640 1 -- 172.21.15.32:0/2391481772 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55b5e908d270 msgr2=0x55b5e947a7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:48:33.940 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:33.938+0000 7fd309000640 1 --2- 172.21.15.32:0/2391481772 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55b5e908d270 0x55b5e947a7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:33.940 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:33.938+0000 7fd309000640 1 -- 172.21.15.32:0/2391481772 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b5e942fed0 con 0x55b5e90643b0 2024-08-03T20:48:33.941 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:33.938+0000 7fd309000640 1 --2- 172.21.15.32:0/2391481772 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b5e90643b0 0x55b5e94334f0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7fd2f8081ae0 tx=0x7fd2f8081560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6f84db4fb8a2ce1d server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:48:33.941 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:33.938+0000 7fd2f4c00640 1 -- 172.21.15.32:0/2391481772 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2f8048070 con 0x55b5e90643b0 2024-08-03T20:48:33.941 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:33.938+0000 7fd2f4c00640 1 -- 172.21.15.32:0/2391481772 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fd2f80b6070 con 0x55b5e90643b0 2024-08-03T20:48:33.941 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:33.938+0000 7fd2f4c00640 1 -- 172.21.15.32:0/2391481772 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2f80860a0 con 0x55b5e90643b0 2024-08-03T20:48:33.942 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:33.939+0000 7fd312be3740 1 -- 172.21.15.32:0/2391481772 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b5e90643b0 msgr2=0x55b5e94334f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:48:33.942 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:33.939+0000 7fd312be3740 1 --2- 172.21.15.32:0/2391481772 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b5e90643b0 0x55b5e94334f0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7fd2f8081ae0 tx=0x7fd2f8081560 comp rx=0 tx=0).stop 2024-08-03T20:48:33.942 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:33.939+0000 7fd312be3740 1 -- 172.21.15.32:0/2391481772 shutdown_connections 2024-08-03T20:48:33.942 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:33.939+0000 7fd312be3740 1 --2- 172.21.15.32:0/2391481772 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55b5e908d270 0x55b5e947a7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:33.942 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:33.939+0000 7fd312be3740 1 --2- 172.21.15.32:0/2391481772 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55b5e906c590 0x55b5e947a280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:33.942 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:33.939+0000 7fd312be3740 1 --2- 172.21.15.32:0/2391481772 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b5e90643b0 0x55b5e94334f0 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:33.942 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:33.939+0000 7fd312be3740 1 -- 172.21.15.32:0/2391481772 >> 172.21.15.32:0/2391481772 conn(0x55b5e90813c0 msgr2=0x55b5e946e560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:48:33.942 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:33.939+0000 7fd312be3740 1 -- 172.21.15.32:0/2391481772 shutdown_connections 2024-08-03T20:48:33.942 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:33.939+0000 7fd312be3740 1 -- 172.21.15.32:0/2391481772 wait complete. 2024-08-03T20:48:33.942 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:33.940+0000 7fd312be3740 1 -- start start 2024-08-03T20:48:33.943 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:33.940+0000 7fd312be3740 1 Processor -- start 2024-08-03T20:48:33.945 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:33.942+0000 7fd312be3740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55b5e90643b0 0x55b5e9470cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:48:33.945 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:33.942+0000 7fd312be3740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b5e906c590 0x55b5e94f4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:48:33.945 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:33.942+0000 7fd312be3740 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55b5e908d270 0x55b5e945d5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:48:33.945 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:33.942+0000 7fd312be3740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55b5e9107620 con 0x55b5e906c590 2024-08-03T20:48:33.946 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:33.942+0000 7fd312be3740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x55b5e9332160 con 0x55b5e90643b0 2024-08-03T20:48:33.946 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:33.942+0000 7fd312be3740 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x55b5e9474770 con 0x55b5e908d270 2024-08-03T20:48:33.946 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:33.942+0000 7fd312be3740 10 client.0 _ll_register_callbacks cb 0x7fd2f809b260 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-03T20:48:33.946 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:33.942+0000 7fd303e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b5e906c590 0x55b5e94f4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:48:33.946 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:33.942+0000 7fd303e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b5e906c590 0x55b5e94f4dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:46538/0 (socket says 172.21.15.32:46538) 2024-08-03T20:48:33.946 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:33.942+0000 7fd303e00640 1 -- 172.21.15.32:0/1168124706 learned_addr learned my addr 172.21.15.32:0/1168124706 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:48:33.946 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:33.942+0000 7fd309000640 1 --2- 172.21.15.32:0/1168124706 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55b5e90643b0 0x55b5e9470cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:48:33.946 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:33.942+0000 7fd309a00640 1 --2- 172.21.15.32:0/1168124706 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55b5e908d270 0x55b5e945d5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:48:33.946 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:33.942+0000 7fd303e00640 1 -- 172.21.15.32:0/1168124706 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55b5e908d270 msgr2=0x55b5e945d5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:48:33.946 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:33.942+0000 7fd303e00640 1 --2- 172.21.15.32:0/1168124706 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55b5e908d270 0x55b5e945d5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:33.946 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:33.942+0000 7fd303e00640 1 -- 172.21.15.32:0/1168124706 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55b5e90643b0 msgr2=0x55b5e9470cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:48:33.947 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:33.942+0000 7fd303e00640 1 --2- 172.21.15.32:0/1168124706 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55b5e90643b0 0x55b5e9470cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:33.947 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:33.943+0000 7fd303e00640 1 -- 172.21.15.32:0/1168124706 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b5e947a320 con 0x55b5e906c590 2024-08-03T20:48:33.947 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:33.943+0000 7fd309000640 1 --2- 172.21.15.32:0/1168124706 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55b5e90643b0 0x55b5e9470cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-03T20:48:33.947 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:33.943+0000 7fd303e00640 1 --2- 172.21.15.32:0/1168124706 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b5e906c590 0x55b5e94f4dd0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7fd2f0011e20 tx=0x7fd2f00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e9b4c28a0dd09bc0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:48:33.947 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:33.943+0000 7fd2ea000640 10 client.0 ms_handle_connect on v2:172.21.15.32:3300/0 2024-08-03T20:48:33.947 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:33.943+0000 7fd2ea000640 1 -- 172.21.15.32:0/1168124706 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2f002e060 con 0x55b5e906c590 2024-08-03T20:48:33.947 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:33.943+0000 7fd2ea000640 1 -- 172.21.15.32:0/1168124706 <== mon.0 v2:172.21.15.32:3300/0 2 ==== osd_map(32..32 src has 1..32) ==== 5709+0+0 (secure 0 0 0) 0x7fd2f0028440 con 0x55b5e906c590 2024-08-03T20:48:33.947 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:33.943+0000 7fd312be3740 1 -- 172.21.15.32:0/1168124706 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd2f80b6070 con 0x55b5e906c590 2024-08-03T20:48:33.947 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:33.943+0000 7fd2ea000640 1 -- 172.21.15.32:0/1168124706 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_get_version_reply(handle=1 version=17) ==== 24+0+0 (secure 0 0 0) 0x7fd2f0034050 con 0x55b5e906c590 2024-08-03T20:48:33.947 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:33.943+0000 7fd312be3740 10 client.5037 fetch_fsmap learned FSMap version 17 2024-08-03T20:48:33.947 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:33.943+0000 7fd312be3740 1 -- 172.21.15.32:0/1168124706 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,fsmap=17,monmap=2+}) -- 0x55b5e94f43f0 con 0x55b5e906c590 2024-08-03T20:48:33.948 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:33.944+0000 7fd2ea000640 1 -- 172.21.15.32:0/1168124706 <== mon.0 v2:172.21.15.32:3300/0 4 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fd2f80b6070 con 0x55b5e906c590 2024-08-03T20:48:33.948 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:33.944+0000 7fd2ea000640 1 -- 172.21.15.32:0/1168124706 <== mon.0 v2:172.21.15.32:3300/0 5 ==== fsmap(e 17) ==== 3502+0+0 (secure 0 0 0) 0x7fd2f0037020 con 0x55b5e906c590 2024-08-03T20:48:33.948 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:33.944+0000 7fd312be3740 10 client.5037 fetch_fsmap finished waiting for FSMap version 17 2024-08-03T20:48:33.948 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:33.944+0000 7fd312be3740 10 client.5037 resolve_mds: resolved name 'a' to daemon mds.a 2024-08-03T20:48:33.948 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:33.944+0000 7fd312be3740 20 client.5037 populate_metadata read hostname 'smithi032' 2024-08-03T20:48:33.948 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:33.944+0000 7fd312be3740 1 --2- 172.21.15.32:0/1168124706 >> [v2:172.21.15.32:6832/1186001746,v1:172.21.15.32:6835/1186001746] conn(0x55b5e945db10 0x55b5e94f79e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:48:33.948 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:33.944+0000 7fd312be3740 4 client.5037 mds_command: new command op to 4866 tid=0 multi_id=0 [{"prefix": "status"}] 2024-08-03T20:48:33.948 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:33.944+0000 7fd312be3740 1 -- 172.21.15.32:0/1168124706 --> [v2:172.21.15.32:6832/1186001746,v1:172.21.15.32:6835/1186001746] -- command(tid 0: {"prefix": "status"}) -- 0x55b5e93a3450 con 0x55b5e945db10 2024-08-03T20:48:33.948 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:33.944+0000 7fd309000640 1 --2- 172.21.15.32:0/1168124706 >> [v2:172.21.15.32:6832/1186001746,v1:172.21.15.32:6835/1186001746] conn(0x55b5e945db10 0x55b5e94f79e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:48:33.949 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:33.944+0000 7fd309000640 1 --2- 172.21.15.32:0/1168124706 >> [v2:172.21.15.32:6832/1186001746,v1:172.21.15.32:6835/1186001746] conn(0x55b5e945db10 0x55b5e94f79e0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a6daab8420fcd87f server_cookie=8a1f4afef06e83ef in_seq=0 out_seq=0 2024-08-03T20:48:33.949 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:33.945+0000 7fd2ea000640 10 client.5037 ms_handle_connect on v2:172.21.15.32:6832/1186001746 2024-08-03T20:48:33.950 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:33.945+0000 7fd2ea000640 1 -- 172.21.15.32:0/1168124706 <== mds.0 v2:172.21.15.32:6832/1186001746 1 ==== command_reply(tid 0: 0 ) ==== 8+0+325 (crc 0 0 0) 0x55b5e93a3450 con 0x55b5e945db10 2024-08-03T20:48:33.950 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:33.945+0000 7fd2ea000640 10 client.5037 handle_command_reply: tid=0 2024-08-03T20:48:33.950 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:33.945+0000 7fd312be3740 1 client.5037 shutdown 2024-08-03T20:48:33.950 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:33.945+0000 7fd312be3740 1 -- 172.21.15.32:0/1168124706 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b5e906c590 msgr2=0x55b5e94f4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:48:33.950 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:33.945+0000 7fd312be3740 1 --2- 172.21.15.32:0/1168124706 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b5e906c590 0x55b5e94f4dd0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7fd2f0011e20 tx=0x7fd2f00046c0 comp rx=0 tx=0).stop 2024-08-03T20:48:33.950 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:33.945+0000 7fd312be3740 1 -- 172.21.15.32:0/1168124706 shutdown_connections 2024-08-03T20:48:33.950 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:33.945+0000 7fd312be3740 1 --2- 172.21.15.32:0/1168124706 >> [v2:172.21.15.32:6832/1186001746,v1:172.21.15.32:6835/1186001746] conn(0x55b5e945db10 0x55b5e94f79e0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:33.950 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:33.945+0000 7fd312be3740 1 --2- 172.21.15.32:0/1168124706 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55b5e908d270 0x55b5e945d5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:33.950 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:33.945+0000 7fd312be3740 1 --2- 172.21.15.32:0/1168124706 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b5e906c590 0x55b5e94f4dd0 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:33.950 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:33.945+0000 7fd2ea000640 0 client.5037 ms_handle_reset on v2:172.21.15.32:6832/1186001746 2024-08-03T20:48:33.950 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:33.945+0000 7fd312be3740 1 --2- 172.21.15.32:0/1168124706 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55b5e90643b0 0x55b5e9470cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:33.950 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:33.945+0000 7fd312be3740 1 -- 172.21.15.32:0/1168124706 >> 172.21.15.32:0/1168124706 conn(0x55b5e90813c0 msgr2=0x55b5e946e560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:48:33.951 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:33.945+0000 7fd312be3740 1 -- 172.21.15.32:0/1168124706 shutdown_connections 2024-08-03T20:48:33.951 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:33.945+0000 7fd312be3740 1 -- 172.21.15.32:0/1168124706 wait complete. 2024-08-03T20:48:33.951 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:33.945+0000 7fd312be3740 20 client.5037 trim_cache size 0 max 16384 2024-08-03T20:48:33.951 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-08-03T20:48:33.951 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_fsid": "40ab141a-01d7-49b9-ae3c-2edec9876e95", 2024-08-03T20:48:33.951 INFO:teuthology.orchestra.run.smithi032.stdout: "whoami": 0, 2024-08-03T20:48:33.951 INFO:teuthology.orchestra.run.smithi032.stdout: "id": 4866, 2024-08-03T20:48:33.951 INFO:teuthology.orchestra.run.smithi032.stdout: "want_state": "up:active", 2024-08-03T20:48:33.951 INFO:teuthology.orchestra.run.smithi032.stdout: "state": "up:active", 2024-08-03T20:48:33.951 INFO:teuthology.orchestra.run.smithi032.stdout: "fs_name": "cephfs", 2024-08-03T20:48:33.952 INFO:teuthology.orchestra.run.smithi032.stdout: "rank_uptime": 6.8211242509999996, 2024-08-03T20:48:33.952 INFO:teuthology.orchestra.run.smithi032.stdout: "mdsmap_epoch": 17, 2024-08-03T20:48:33.952 INFO:teuthology.orchestra.run.smithi032.stdout: "osdmap_epoch": 32, 2024-08-03T20:48:33.952 INFO:teuthology.orchestra.run.smithi032.stdout: "osdmap_epoch_barrier": 32, 2024-08-03T20:48:33.952 INFO:teuthology.orchestra.run.smithi032.stdout: "uptime": 14.552858235 2024-08-03T20:48:33.952 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-08-03T20:48:33.953 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:33.946+0000 7fd30a400640 1 -- 172.21.15.32:0/3752758950 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fd2e4077010 msgr2=0x7fd2e4079520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:48:33.953 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:33.946+0000 7fd30a400640 1 --2- 172.21.15.32:0/3752758950 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fd2e4077010 0x7fd2e4079520 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fd3040704e0 tx=0x7fd2f0005f90 comp rx=0 tx=0).stop 2024-08-03T20:48:33.953 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:33.946+0000 7fd30a400640 1 -- 172.21.15.32:0/3752758950 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd30406f440 msgr2=0x7fd3040cebc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:48:33.953 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:33.946+0000 7fd30a400640 1 --2- 172.21.15.32:0/3752758950 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd30406f440 0x7fd3040cebc0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7fd2f8034040 tx=0x7fd2f8003820 comp rx=0 tx=0).stop 2024-08-03T20:48:33.953 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:33.946+0000 7fd30a400640 1 -- 172.21.15.32:0/3752758950 shutdown_connections 2024-08-03T20:48:33.953 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:33.946+0000 7fd30a400640 1 --2- 172.21.15.32:0/3752758950 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fd2e4077010 0x7fd2e4079520 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:33.953 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:33.946+0000 7fd30a400640 1 --2- 172.21.15.32:0/3752758950 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fd3040af020 0x7fd30415b350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:33.953 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:33.946+0000 7fd30a400640 1 --2- 172.21.15.32:0/3752758950 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fd3040acd80 0x7fd304155970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:33.953 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:33.946+0000 7fd30a400640 1 --2- 172.21.15.32:0/3752758950 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd30406f440 0x7fd3040cebc0 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:33.953 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:33.946+0000 7fd30a400640 1 -- 172.21.15.32:0/3752758950 >> 172.21.15.32:0/3752758950 conn(0x7fd3040b2120 msgr2=0x7fd3040ca390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:48:33.954 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:33.946+0000 7fd30a400640 1 -- 172.21.15.32:0/3752758950 shutdown_connections 2024-08-03T20:48:33.954 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:33.946+0000 7fd30a400640 1 -- 172.21.15.32:0/3752758950 wait complete. 2024-08-03T20:48:33.973 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-08-03T20:48:34.204 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:34.202+0000 7fd096a00640 1 -- 172.21.15.32:0/619295119 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fd09006e930 msgr2=0x7fd090144f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:48:34.204 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:34.202+0000 7fd096a00640 1 --2- 172.21.15.32:0/619295119 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fd09006e930 0x7fd090144f90 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fd080005fd0 tx=0x7fd08002a540 comp rx=0 tx=0).stop 2024-08-03T20:48:34.204 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:34.202+0000 7fd096a00640 1 -- 172.21.15.32:0/619295119 shutdown_connections 2024-08-03T20:48:34.204 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:34.202+0000 7fd096a00640 1 --2- 172.21.15.32:0/619295119 >> v2:172.21.15.118:3300/0 conn(0x7fd090073e60 0x7fd090074270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:34.204 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:34.202+0000 7fd096a00640 1 --2- 172.21.15.32:0/619295119 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fd09006e930 0x7fd090144f90 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:34.204 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:34.202+0000 7fd096a00640 1 -- 172.21.15.32:0/619295119 >> 172.21.15.32:0/619295119 conn(0x7fd0900b1bd0 msgr2=0x7fd0900b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:48:34.204 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:34.202+0000 7fd096a00640 1 -- 172.21.15.32:0/619295119 shutdown_connections 2024-08-03T20:48:34.204 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:34.202+0000 7fd096a00640 1 -- 172.21.15.32:0/619295119 wait complete. 2024-08-03T20:48:34.206 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:34.205+0000 7fd096a00640 1 Processor -- start 2024-08-03T20:48:34.206 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:34.205+0000 7fd096a00640 1 -- start start 2024-08-03T20:48:34.208 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:34.206+0000 7fd096a00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fd09006e930 0x7fd09015bd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:48:34.208 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:34.206+0000 7fd096a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd090073e60 0x7fd0900aede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:48:34.209 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:34.206+0000 7fd096a00640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fd0900747b0 0x7fd0901436b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:48:34.209 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:34.206+0000 7fd096a00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fd0900cf9c0 con 0x7fd090073e60 2024-08-03T20:48:34.209 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:34.206+0000 7fd096a00640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7fd0900cf840 con 0x7fd09006e930 2024-08-03T20:48:34.209 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:34.206+0000 7fd096a00640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7fd0900ccff0 con 0x7fd0900747b0 2024-08-03T20:48:34.209 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:34.207+0000 7fd08f400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd090073e60 0x7fd0900aede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:48:34.209 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:34.207+0000 7fd08f400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd090073e60 0x7fd0900aede0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:46544/0 (socket says 172.21.15.32:46544) 2024-08-03T20:48:34.209 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:34.207+0000 7fd08f400640 1 -- 172.21.15.32:0/3418206875 learned_addr learned my addr 172.21.15.32:0/3418206875 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:48:34.209 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:34.207+0000 7fd08fe00640 1 --2- 172.21.15.32:0/3418206875 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fd09006e930 0x7fd09015bd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:48:34.209 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:34.207+0000 7fd094a00640 1 --2- 172.21.15.32:0/3418206875 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fd0900747b0 0x7fd0901436b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:48:34.209 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:34.207+0000 7fd08f400640 1 -- 172.21.15.32:0/3418206875 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fd0900747b0 msgr2=0x7fd0901436b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:48:34.209 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:34.207+0000 7fd08f400640 1 --2- 172.21.15.32:0/3418206875 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fd0900747b0 0x7fd0901436b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:34.210 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:34.207+0000 7fd08f400640 1 -- 172.21.15.32:0/3418206875 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fd09006e930 msgr2=0x7fd09015bd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:48:34.210 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:34.207+0000 7fd08f400640 1 --2- 172.21.15.32:0/3418206875 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fd09006e930 0x7fd09015bd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:34.210 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:34.207+0000 7fd08f400640 1 -- 172.21.15.32:0/3418206875 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd0901f6760 con 0x7fd090073e60 2024-08-03T20:48:34.211 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:34.207+0000 7fd08f400640 1 --2- 172.21.15.32:0/3418206875 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd090073e60 0x7fd0900aede0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7fd07c00e790 tx=0x7fd07c00ec50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:48:34.211 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:34.208+0000 7fd087e00640 1 -- 172.21.15.32:0/3418206875 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd07c00eea0 con 0x7fd090073e60 2024-08-03T20:48:34.211 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:34.208+0000 7fd087e00640 1 -- 172.21.15.32:0/3418206875 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fd07c00cdb0 con 0x7fd090073e60 2024-08-03T20:48:34.211 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:34.208+0000 7fd087e00640 1 -- 172.21.15.32:0/3418206875 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd07c010690 con 0x7fd090073e60 2024-08-03T20:48:34.211 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:34.208+0000 7fd096a00640 1 -- 172.21.15.32:0/3418206875 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd0901f6a40 con 0x7fd090073e60 2024-08-03T20:48:34.211 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:34.208+0000 7fd096a00640 1 -- 172.21.15.32:0/3418206875 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd0901f7000 con 0x7fd090073e60 2024-08-03T20:48:34.211 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:34.209+0000 7fd096a00640 1 -- 172.21.15.32:0/3418206875 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd090072ac0 con 0x7fd090073e60 2024-08-03T20:48:34.211 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:34.210+0000 7fd087e00640 1 -- 172.21.15.32:0/3418206875 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7fd07c010830 con 0x7fd090073e60 2024-08-03T20:48:34.213 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:34.212+0000 7fd087e00640 1 --2- 172.21.15.32:0/3418206875 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fd070077010 0x7fd070079520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:48:34.213 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:34.212+0000 7fd087e00640 1 -- 172.21.15.32:0/3418206875 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(32..32 src has 1..32) ==== 5709+0+0 (secure 0 0 0) 0x7fd07c0991f0 con 0x7fd090073e60 2024-08-03T20:48:34.213 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:34.212+0000 7fd08fe00640 1 --2- 172.21.15.32:0/3418206875 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fd070077010 0x7fd070079520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:48:34.214 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:34.213+0000 7fd08fe00640 1 --2- 172.21.15.32:0/3418206875 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fd070077010 0x7fd070079520 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fd080033000 tx=0x7fd080004730 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:48:34.218 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:34.216+0000 7fd087e00640 1 -- 172.21.15.32:0/3418206875 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7fd090072ac0 con 0x7fd090073e60 2024-08-03T20:48:34.461 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:34.459+0000 7fd096a00640 1 -- 172.21.15.32:0/3418206875 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7fd0900af830 con 0x7fd090073e60 2024-08-03T20:48:34.461 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:34.460+0000 7fd087e00640 1 -- 172.21.15.32:0/3418206875 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v17) ==== 53+0+73 (secure 0 0 0) 0x7fd07c061d90 con 0x7fd090073e60 2024-08-03T20:48:34.462 INFO:teuthology.orchestra.run.smithi032.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-08-03T20:48:34.464 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:34.462+0000 7fd096a00640 1 -- 172.21.15.32:0/3418206875 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fd070077010 msgr2=0x7fd070079520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:48:34.464 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:34.462+0000 7fd096a00640 1 --2- 172.21.15.32:0/3418206875 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fd070077010 0x7fd070079520 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fd080033000 tx=0x7fd080004730 comp rx=0 tx=0).stop 2024-08-03T20:48:34.464 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:34.462+0000 7fd096a00640 1 -- 172.21.15.32:0/3418206875 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd090073e60 msgr2=0x7fd0900aede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:48:34.464 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:34.462+0000 7fd096a00640 1 --2- 172.21.15.32:0/3418206875 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd090073e60 0x7fd0900aede0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7fd07c00e790 tx=0x7fd07c00ec50 comp rx=0 tx=0).stop 2024-08-03T20:48:34.464 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:34.462+0000 7fd096a00640 1 -- 172.21.15.32:0/3418206875 shutdown_connections 2024-08-03T20:48:34.464 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:34.462+0000 7fd096a00640 1 --2- 172.21.15.32:0/3418206875 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fd070077010 0x7fd070079520 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:34.464 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:34.462+0000 7fd096a00640 1 --2- 172.21.15.32:0/3418206875 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fd0900747b0 0x7fd0901436b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:34.464 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:34.462+0000 7fd096a00640 1 --2- 172.21.15.32:0/3418206875 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd090073e60 0x7fd0900aede0 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:34.464 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:34.462+0000 7fd096a00640 1 --2- 172.21.15.32:0/3418206875 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fd09006e930 0x7fd09015bd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:34.464 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:34.462+0000 7fd096a00640 1 -- 172.21.15.32:0/3418206875 >> 172.21.15.32:0/3418206875 conn(0x7fd0900b1bd0 msgr2=0x7fd0900b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:48:34.464 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:34.462+0000 7fd096a00640 1 -- 172.21.15.32:0/3418206875 shutdown_connections 2024-08-03T20:48:34.465 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:34.462+0000 7fd096a00640 1 -- 172.21.15.32:0/3418206875 wait complete. 2024-08-03T20:48:34.485 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-08-03T20:48:34.485 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-08-03T20:48:34.485 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi032.front.sepia.ceph.com 2024-08-03T20:48:34.485 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-08-03T20:48:34.485 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-08-03T20:48:34.485 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-08-03T20:48:34.485 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2024-08-03T20:48:34.486 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-08-03T20:48:34.486 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-08-03T20:48:34.486 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-08-03T20:48:34.486 DEBUG:teuthology.orchestra.run.smithi032:> ip addr 2024-08-03T20:48:34.522 INFO:teuthology.orchestra.run.smithi032.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-08-03T20:48:34.522 INFO:teuthology.orchestra.run.smithi032.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-08-03T20:48:34.522 INFO:teuthology.orchestra.run.smithi032.stdout: inet 127.0.0.1/8 scope host lo 2024-08-03T20:48:34.522 INFO:teuthology.orchestra.run.smithi032.stdout: valid_lft forever preferred_lft forever 2024-08-03T20:48:34.522 INFO:teuthology.orchestra.run.smithi032.stdout: inet6 ::1/128 scope host 2024-08-03T20:48:34.522 INFO:teuthology.orchestra.run.smithi032.stdout: valid_lft forever preferred_lft forever 2024-08-03T20:48:34.522 INFO:teuthology.orchestra.run.smithi032.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-08-03T20:48:34.523 INFO:teuthology.orchestra.run.smithi032.stdout: link/ether 0c:c4:7a:6b:fd:0c brd ff:ff:ff:ff:ff:ff 2024-08-03T20:48:34.523 INFO:teuthology.orchestra.run.smithi032.stdout: altname enp7s0f0 2024-08-03T20:48:34.523 INFO:teuthology.orchestra.run.smithi032.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-08-03T20:48:34.523 INFO:teuthology.orchestra.run.smithi032.stdout: link/ether 0c:c4:7a:6b:fd:0d brd ff:ff:ff:ff:ff:ff 2024-08-03T20:48:34.523 INFO:teuthology.orchestra.run.smithi032.stdout: altname enp7s0f1 2024-08-03T20:48:34.523 INFO:teuthology.orchestra.run.smithi032.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-08-03T20:48:34.523 INFO:teuthology.orchestra.run.smithi032.stdout: link/ether ac:1f:6b:f8:26:02 brd ff:ff:ff:ff:ff:ff 2024-08-03T20:48:34.523 INFO:teuthology.orchestra.run.smithi032.stdout: altname enp1s0f0 2024-08-03T20:48:34.523 INFO:teuthology.orchestra.run.smithi032.stdout: inet 172.21.15.32/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-08-03T20:48:34.523 INFO:teuthology.orchestra.run.smithi032.stdout: valid_lft 42442sec preferred_lft 42442sec 2024-08-03T20:48:34.523 INFO:teuthology.orchestra.run.smithi032.stdout: inet6 fe80::ae1f:6bff:fef8:2602/64 scope link noprefixroute 2024-08-03T20:48:34.523 INFO:teuthology.orchestra.run.smithi032.stdout: valid_lft forever preferred_lft forever 2024-08-03T20:48:34.523 INFO:teuthology.orchestra.run.smithi032.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-08-03T20:48:34.523 INFO:teuthology.orchestra.run.smithi032.stdout: link/ether ac:1f:6b:f8:26:03 brd ff:ff:ff:ff:ff:ff 2024-08-03T20:48:34.524 INFO:teuthology.orchestra.run.smithi032.stdout: altname enp1s0f1 2024-08-03T20:48:34.524 INFO:teuthology.orchestra.run.smithi032.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-08-03T20:48:34.524 INFO:teuthology.orchestra.run.smithi032.stdout: link/ether 0a:23:2b:ca:2a:93 brd ff:ff:ff:ff:ff:ff 2024-08-03T20:48:34.524 INFO:teuthology.orchestra.run.smithi032.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-08-03T20:48:34.524 INFO:teuthology.orchestra.run.smithi032.stdout: valid_lft forever preferred_lft forever 2024-08-03T20:48:34.524 INFO:teuthology.orchestra.run.smithi032.stdout: inet6 fe80::d804:7cff:fed9:7ba4/64 scope link 2024-08-03T20:48:34.524 INFO:teuthology.orchestra.run.smithi032.stdout: valid_lft forever preferred_lft forever 2024-08-03T20:48:34.524 INFO:teuthology.orchestra.run.smithi032.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-08-03T20:48:34.524 INFO:teuthology.orchestra.run.smithi032.stdout: link/ether 22:86:28:30:4e:62 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-08-03T20:48:34.524 INFO:teuthology.orchestra.run.smithi032.stdout: inet6 fe80::2086:28ff:fe30:4e62/64 scope link 2024-08-03T20:48:34.524 INFO:teuthology.orchestra.run.smithi032.stdout: valid_lft forever preferred_lft forever 2024-08-03T20:48:34.524 INFO:teuthology.orchestra.run.smithi032.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-08-03T20:48:34.524 INFO:teuthology.orchestra.run.smithi032.stdout: link/ether 0a:23:2b:ca:2a:93 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-08-03T20:48:34.525 INFO:teuthology.orchestra.run.smithi032.stdout: inet6 fe80::823:2bff:feca:2a93/64 scope link 2024-08-03T20:48:34.525 INFO:teuthology.orchestra.run.smithi032.stdout: valid_lft forever preferred_lft forever 2024-08-03T20:48:34.525 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-08-03T20:48:34.525 DEBUG:teuthology.orchestra.run.smithi032:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-08-03T20:48:34.617 INFO:teuthology.orchestra.run.smithi032.stdout:1 2024-08-03T20:48:34.620 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-08-03T20:48:34.620 DEBUG:teuthology.orchestra.run.smithi032:> route 2024-08-03T20:48:34.656 INFO:teuthology.orchestra.run.smithi032.stdout:Kernel IP routing table 2024-08-03T20:48:34.657 INFO:teuthology.orchestra.run.smithi032.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-08-03T20:48:34.657 INFO:teuthology.orchestra.run.smithi032.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-08-03T20:48:34.657 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-08-03T20:48:34.657 INFO:teuthology.orchestra.run.smithi032.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-08-03T20:48:34.657 DEBUG:teuthology.orchestra.run.smithi032:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-08-03T20:48:34.657 DEBUG:teuthology.orchestra.run.smithi032:> set -e 2024-08-03T20:48:34.657 DEBUG:teuthology.orchestra.run.smithi032:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-08-03T20:48:34.657 DEBUG:teuthology.orchestra.run.smithi032:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-08-03T20:48:34.657 DEBUG:teuthology.orchestra.run.smithi032:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-08-03T20:48:34.658 DEBUG:teuthology.orchestra.run.smithi032:> ') 2024-08-03T20:48:34.776 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:34 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-08-03T20:48:34.848 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:34 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-08-03T20:48:34.856 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-08-03T20:48:34.856 DEBUG:teuthology.orchestra.run.smithi032:> ip netns list 2024-08-03T20:48:34.892 INFO:teuthology.orchestra.run.smithi032.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-08-03T20:48:34.892 INFO:teuthology.orchestra.run.smithi032.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-08-03T20:48:34.892 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-08-03T20:48:34.892 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-08-03T20:48:34.892 DEBUG:teuthology.orchestra.run.smithi032:> sudo ip link set brx.0 up 2024-08-03T20:48:34.990 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-08-03T20:48:34.990 DEBUG:teuthology.orchestra.run.smithi032:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-08-03T20:48:35.025 INFO:teuthology.orchestra.run.smithi032.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-08-03T20:48:35.025 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-08-03T20:48:35.025 DEBUG:teuthology.orchestra.run.smithi032:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-08-03T20:48:35.100 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2024-08-03T20:48:35.100 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-08-03T20:48:35.100 DEBUG:teuthology.orchestra.run.smithi032:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-08-03T20:48:35.299 INFO:teuthology.orchestra.run.smithi032.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-08-03T20:48:35.299 INFO:teuthology.orchestra.run.smithi032.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-08-03T20:48:35.299 INFO:teuthology.orchestra.run.smithi032.stdout:invalid new device string format 2024-08-03T20:48:35.320 INFO:teuthology.orchestra.run.smithi032.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-08-03T20:48:35.320 INFO:teuthology.orchestra.run.smithi032.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-08-03T20:48:35.320 INFO:teuthology.orchestra.run.smithi032.stdout:invalid new device string format 2024-08-03T20:48:35.320 INFO:teuthology.orchestra.run.smithi032.stdout:mount.ceph: resolved to: "172.21.15.32:3300,172.21.15.118:3300,172.21.15.118:3301" 2024-08-03T20:48:35.320 INFO:teuthology.orchestra.run.smithi032.stdout:mount.ceph: switching to using v1 address with old syntax 2024-08-03T20:48:35.351 INFO:teuthology.orchestra.run.smithi032.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-08-03T20:48:35.351 INFO:teuthology.orchestra.run.smithi032.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-08-03T20:48:35.351 INFO:teuthology.orchestra.run.smithi032.stdout:invalid new device string format 2024-08-03T20:48:35.352 INFO:teuthology.orchestra.run.smithi032.stdout:mount.ceph: resolved to: "172.21.15.32:3300,172.21.15.118:3300,172.21.15.118:3301" 2024-08-03T20:48:35.352 INFO:teuthology.orchestra.run.smithi032.stdout:mount.ceph: switching to using v1 address with old syntax 2024-08-03T20:48:35.352 INFO:teuthology.orchestra.run.smithi032.stdout:mount.ceph: resolved to: "172.21.15.32:6789,172.21.15.118:6789,172.21.15.118:6790" 2024-08-03T20:48:35.352 INFO:teuthology.orchestra.run.smithi032.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.32:6789,172.21.15.118:6789,172.21.15.118:6790:/ 2024-08-03T20:48:35.352 INFO:teuthology.orchestra.run.smithi032.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,key=0,fsid=40ab141a-01d7-49b9-ae3c-2edec9876e95" will pass to kernel 2024-08-03T20:48:35.353 INFO:teuthology.orchestra.run.smithi032.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2024-08-03T20:48:35.353 INFO:teuthology.orchestra.run.smithi032.stdout: You just mounted a file system that supports labels which does not 2024-08-03T20:48:35.354 INFO:teuthology.orchestra.run.smithi032.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-08-03T20:48:35.354 INFO:teuthology.orchestra.run.smithi032.stdout: applications will generate AVC messages and not be allowed access to 2024-08-03T20:48:35.354 INFO:teuthology.orchestra.run.smithi032.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-08-03T20:48:35.356 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-08-03T20:48:35.356 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-08-03T20:48:35.357 DEBUG:teuthology.orchestra.run.smithi032:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-08-03T20:48:35.413 DEBUG:teuthology.orchestra.run.smithi032:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-08-03T20:48:35.495 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:35 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-08-03T20:48:35.504 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:35 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-08-03T20:48:35.512 INFO:teuthology.orchestra.run.smithi032.stdout:client4963 2024-08-03T20:48:35.512 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-08-03T20:48:35.513 DEBUG:teuthology.orchestra.run.smithi032:> dd if=/proc/self/mounts of=/dev/stdout 2024-08-03T20:48:35.548 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-08-03T20:48:35.548 DEBUG:teuthology.orchestra.run.smithi032:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir subdir/) 2024-08-03T20:48:35.632 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-08-03T20:48:35.632 DEBUG:teuthology.orchestra.run.smithi032:> (cd /home/ubuntu/cephtest/mnt.0 && exec touch subdir/file_undamaged) 2024-08-03T20:48:35.710 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-08-03T20:48:35.710 DEBUG:teuthology.orchestra.run.smithi032:> (cd /home/ubuntu/cephtest/mnt.0 && exec touch subdir/file_to_be_damaged) 2024-08-03T20:48:35.787 DEBUG:teuthology.orchestra.run.smithi032:> stdin-killer -- python3 -c ' 2024-08-03T20:48:35.787 DEBUG:teuthology.orchestra.run.smithi032:> import os 2024-08-03T20:48:35.787 DEBUG:teuthology.orchestra.run.smithi032:> import stat 2024-08-03T20:48:35.787 DEBUG:teuthology.orchestra.run.smithi032:> 2024-08-03T20:48:35.787 DEBUG:teuthology.orchestra.run.smithi032:> print(os.stat("/home/ubuntu/cephtest/mnt.0/subdir").st_ino) 2024-08-03T20:48:35.787 DEBUG:teuthology.orchestra.run.smithi032:> ' 2024-08-03T20:48:35.928 INFO:teuthology.orchestra.run.smithi032.stdout:1099511627776 2024-08-03T20:48:35.931 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:35 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-08-03T20:48:35.939 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-08-03T20:48:35.939 DEBUG:teuthology.orchestra.run.smithi032:> dd if=/proc/self/mounts of=/dev/stdout 2024-08-03T20:48:36.015 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-08-03T20:48:36.015 DEBUG:teuthology.orchestra.run.smithi032:> dd if=/proc/self/mounts of=/dev/stdout 2024-08-03T20:48:36.092 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-08-03T20:48:36.362 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:36.360+0000 7f9793600640 1 -- 172.21.15.32:0/871314810 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f978c144f90 msgr2=0x7f978c074570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:48:36.362 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:36.360+0000 7f9793600640 1 --2- 172.21.15.32:0/871314810 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f978c144f90 0x7f978c074570 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f9778009860 tx=0x7f977802a5a0 comp rx=0 tx=0).stop 2024-08-03T20:48:36.362 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:36.360+0000 7f9793600640 1 -- 172.21.15.32:0/871314810 shutdown_connections 2024-08-03T20:48:36.362 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:36.360+0000 7f9793600640 1 --2- 172.21.15.32:0/871314810 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f978c144f90 0x7f978c074570 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:36.362 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:36.360+0000 7f9793600640 1 -- 172.21.15.32:0/871314810 >> 172.21.15.32:0/871314810 conn(0x7f978c0b1bd0 msgr2=0x7f978c0b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:48:36.362 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:36.360+0000 7f9793600640 1 -- 172.21.15.32:0/871314810 shutdown_connections 2024-08-03T20:48:36.362 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:36.360+0000 7f9793600640 1 -- 172.21.15.32:0/871314810 wait complete. 2024-08-03T20:48:36.364 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:36.362+0000 7f9793600640 1 Processor -- start 2024-08-03T20:48:36.364 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:36.362+0000 7f9793600640 1 -- start start 2024-08-03T20:48:36.367 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:36.364+0000 7f9793600640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f978c06e930 0x7f978c13dfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:48:36.367 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:36.364+0000 7f9793600640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f978c144f90 0x7f978c1f6730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:48:36.367 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:36.364+0000 7f9793600640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f978c15fff0 0x7f978c1f6c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:48:36.367 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:36.364+0000 7f9793600640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f978c0cf840 con 0x7f978c144f90 2024-08-03T20:48:36.367 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:36.364+0000 7f9793600640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f978c145f30 con 0x7f978c06e930 2024-08-03T20:48:36.367 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:36.364+0000 7f9793600640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7f978c0cf9c0 con 0x7f978c15fff0 2024-08-03T20:48:36.367 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:36.364+0000 7f978be00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f978c144f90 0x7f978c1f6730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:48:36.367 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:36.364+0000 7f978be00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f978c144f90 0x7f978c1f6730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:46560/0 (socket says 172.21.15.32:46560) 2024-08-03T20:48:36.367 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:36.364+0000 7f978be00640 1 -- 172.21.15.32:0/2597858713 learned_addr learned my addr 172.21.15.32:0/2597858713 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:48:36.367 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:36.364+0000 7f9791600640 1 --2- 172.21.15.32:0/2597858713 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f978c15fff0 0x7f978c1f6c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:48:36.368 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:36.364+0000 7f9790c00640 1 --2- 172.21.15.32:0/2597858713 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f978c06e930 0x7f978c13dfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:48:36.368 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:36.365+0000 7f978be00640 1 -- 172.21.15.32:0/2597858713 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f978c15fff0 msgr2=0x7f978c1f6c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:48:36.368 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:36.365+0000 7f978be00640 1 --2- 172.21.15.32:0/2597858713 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f978c15fff0 0x7f978c1f6c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:36.368 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:36.365+0000 7f978be00640 1 -- 172.21.15.32:0/2597858713 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f978c06e930 msgr2=0x7f978c13dfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:48:36.368 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:36.365+0000 7f978be00640 1 --2- 172.21.15.32:0/2597858713 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f978c06e930 0x7f978c13dfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:36.368 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:36.365+0000 7f978be00640 1 -- 172.21.15.32:0/2597858713 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f978c1e6c00 con 0x7f978c144f90 2024-08-03T20:48:36.368 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:36.365+0000 7f978be00640 1 --2- 172.21.15.32:0/2597858713 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f978c144f90 0x7f978c1f6730 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f977802ab90 tx=0x7f9778033ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:48:36.368 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:36.365+0000 7f9788c00640 1 -- 172.21.15.32:0/2597858713 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9778031b50 con 0x7f978c144f90 2024-08-03T20:48:36.368 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:36.365+0000 7f9788c00640 1 -- 172.21.15.32:0/2597858713 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f9778031de0 con 0x7f978c144f90 2024-08-03T20:48:36.368 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:36.365+0000 7f9788c00640 1 -- 172.21.15.32:0/2597858713 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97780332d0 con 0x7f978c144f90 2024-08-03T20:48:36.368 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:36.365+0000 7f9793600640 1 -- 172.21.15.32:0/2597858713 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f978c1e6ea0 con 0x7f978c144f90 2024-08-03T20:48:36.368 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:36.366+0000 7f9793600640 1 -- 172.21.15.32:0/2597858713 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f978c1f7600 con 0x7f978c144f90 2024-08-03T20:48:36.369 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:36.367+0000 7f9793600640 1 -- 172.21.15.32:0/2597858713 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f978c160400 con 0x7f978c144f90 2024-08-03T20:48:36.369 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:36.368+0000 7f9788c00640 1 -- 172.21.15.32:0/2597858713 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f9778033470 con 0x7f978c144f90 2024-08-03T20:48:36.371 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:36.369+0000 7f9788c00640 1 --2- 172.21.15.32:0/2597858713 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f976c077010 0x7f976c079520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:48:36.371 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:36.370+0000 7f9788c00640 1 -- 172.21.15.32:0/2597858713 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(32..32 src has 1..32) ==== 5709+0+0 (secure 0 0 0) 0x7f97780b9920 con 0x7f978c144f90 2024-08-03T20:48:36.371 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:36.370+0000 7f9790c00640 1 --2- 172.21.15.32:0/2597858713 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f976c077010 0x7f976c079520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:48:36.372 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:36.371+0000 7f9790c00640 1 --2- 172.21.15.32:0/2597858713 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f976c077010 0x7f976c079520 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f9780001120 tx=0x7f9780009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:48:36.375 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:36.374+0000 7f9788c00640 1 -- 172.21.15.32:0/2597858713 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7f977803f030 con 0x7f978c144f90 2024-08-03T20:48:36.424 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-08-03T20:48:36.428 DEBUG:teuthology.orchestra.run.smithi118:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-08-03T20:48:36.593 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:36.591+0000 7f9793600640 1 -- 172.21.15.32:0/2597858713 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f978c0b2280 con 0x7f978c144f90 2024-08-03T20:48:36.593 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:36.592+0000 7f9788c00640 1 -- 172.21.15.32:0/2597858713 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v32) ==== 80+0+0 (secure 0 0 0) 0x7f9778031770 con 0x7f978c144f90 2024-08-03T20:48:36.593 INFO:teuthology.orchestra.run.smithi032.stderr:listed 0 entries 2024-08-03T20:48:36.596 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:36.594+0000 7f9793600640 1 -- 172.21.15.32:0/2597858713 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f976c077010 msgr2=0x7f976c079520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:48:36.596 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:36.594+0000 7f9793600640 1 --2- 172.21.15.32:0/2597858713 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f976c077010 0x7f976c079520 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f9780001120 tx=0x7f9780009040 comp rx=0 tx=0).stop 2024-08-03T20:48:36.596 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:36.594+0000 7f9793600640 1 -- 172.21.15.32:0/2597858713 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f978c144f90 msgr2=0x7f978c1f6730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:48:36.596 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:36.594+0000 7f9793600640 1 --2- 172.21.15.32:0/2597858713 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f978c144f90 0x7f978c1f6730 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f977802ab90 tx=0x7f9778033ba0 comp rx=0 tx=0).stop 2024-08-03T20:48:36.596 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:36.594+0000 7f9793600640 1 -- 172.21.15.32:0/2597858713 shutdown_connections 2024-08-03T20:48:36.596 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:36.594+0000 7f9793600640 1 --2- 172.21.15.32:0/2597858713 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f976c077010 0x7f976c079520 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:36.596 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:36.594+0000 7f9793600640 1 --2- 172.21.15.32:0/2597858713 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f978c15fff0 0x7f978c1f6c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:36.597 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:36.594+0000 7f9793600640 1 --2- 172.21.15.32:0/2597858713 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f978c144f90 0x7f978c1f6730 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:36.597 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:36.594+0000 7f9793600640 1 --2- 172.21.15.32:0/2597858713 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f978c06e930 0x7f978c13dfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:36.597 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:36.594+0000 7f9793600640 1 -- 172.21.15.32:0/2597858713 >> 172.21.15.32:0/2597858713 conn(0x7f978c0b1bd0 msgr2=0x7f978c0d0920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:48:36.597 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:36.594+0000 7f9793600640 1 -- 172.21.15.32:0/2597858713 shutdown_connections 2024-08-03T20:48:36.597 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:36.594+0000 7f9793600640 1 -- 172.21.15.32:0/2597858713 wait complete. 2024-08-03T20:48:36.622 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-08-03T20:48:36.622 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-08-03T20:48:36.622 DEBUG:teuthology.orchestra.run.smithi032:> sudo umount /home/ubuntu/cephtest/mnt.0 2024-08-03T20:48:36.729 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi032.front.sepia.ceph.com 2024-08-03T20:48:36.729 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-08-03T20:48:36.729 DEBUG:teuthology.orchestra.run.smithi032:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-08-03T20:48:36.775 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-08-03T20:48:37.043 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:37.041+0000 7ff09da00640 1 -- 172.21.15.32:0/279749704 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7ff098144f90 msgr2=0x7ff098074570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:48:37.043 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:37.041+0000 7ff09da00640 1 --2- 172.21.15.32:0/279749704 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7ff098144f90 0x7ff098074570 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7ff080009860 tx=0x7ff08002a5a0 comp rx=0 tx=0).stop 2024-08-03T20:48:37.044 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:37.042+0000 7ff09da00640 1 -- 172.21.15.32:0/279749704 shutdown_connections 2024-08-03T20:48:37.044 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:37.042+0000 7ff09da00640 1 --2- 172.21.15.32:0/279749704 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7ff098144f90 0x7ff098074570 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:37.044 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:37.042+0000 7ff09da00640 1 -- 172.21.15.32:0/279749704 >> 172.21.15.32:0/279749704 conn(0x7ff0980b1bd0 msgr2=0x7ff0980b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:48:37.044 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:37.042+0000 7ff09da00640 1 -- 172.21.15.32:0/279749704 shutdown_connections 2024-08-03T20:48:37.044 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:37.042+0000 7ff09da00640 1 -- 172.21.15.32:0/279749704 wait complete. 2024-08-03T20:48:37.045 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:37.044+0000 7ff09da00640 1 Processor -- start 2024-08-03T20:48:37.045 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:37.044+0000 7ff09da00640 1 -- start start 2024-08-03T20:48:37.048 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:37.045+0000 7ff09da00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff09806e930 0x7ff09815bb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:48:37.048 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:37.045+0000 7ff09da00640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7ff098144f90 0x7ff09815dac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:48:37.048 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:37.045+0000 7ff09da00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ff09815fff0 0x7ff09815e000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:48:37.048 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:37.045+0000 7ff09da00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7ff0980cf840 con 0x7ff09806e930 2024-08-03T20:48:37.048 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:37.045+0000 7ff09da00640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7ff098145f30 con 0x7ff09815fff0 2024-08-03T20:48:37.048 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:37.045+0000 7ff09da00640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7ff0980cf9c0 con 0x7ff098144f90 2024-08-03T20:48:37.048 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:37.046+0000 7ff096a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff09806e930 0x7ff09815bb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:48:37.048 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:37.046+0000 7ff096a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff09806e930 0x7ff09815bb90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:46568/0 (socket says 172.21.15.32:46568) 2024-08-03T20:48:37.048 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:37.046+0000 7ff096a00640 1 -- 172.21.15.32:0/863322760 learned_addr learned my addr 172.21.15.32:0/863322760 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:48:37.048 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:37.046+0000 7ff08fe00640 1 --2- 172.21.15.32:0/863322760 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7ff098144f90 0x7ff09815dac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:48:37.048 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:37.046+0000 7ff097400640 1 --2- 172.21.15.32:0/863322760 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ff09815fff0 0x7ff09815e000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:48:37.048 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:37.046+0000 7ff096a00640 1 -- 172.21.15.32:0/863322760 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7ff098144f90 msgr2=0x7ff09815dac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:48:37.049 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:37.046+0000 7ff096a00640 1 --2- 172.21.15.32:0/863322760 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7ff098144f90 0x7ff09815dac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:37.049 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:37.046+0000 7ff096a00640 1 -- 172.21.15.32:0/863322760 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ff09815fff0 msgr2=0x7ff09815e000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:48:37.049 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:37.046+0000 7ff096a00640 1 --2- 172.21.15.32:0/863322760 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ff09815fff0 0x7ff09815e000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:37.049 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:37.046+0000 7ff096a00640 1 -- 172.21.15.32:0/863322760 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff098140540 con 0x7ff09806e930 2024-08-03T20:48:37.049 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:37.046+0000 7ff096a00640 1 --2- 172.21.15.32:0/863322760 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff09806e930 0x7ff09815bb90 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7ff08800c460 tx=0x7ff08800c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:48:37.050 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:37.047+0000 7ff08ea00640 1 -- 172.21.15.32:0/863322760 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff088011850 con 0x7ff09806e930 2024-08-03T20:48:37.050 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:37.047+0000 7ff08ea00640 1 -- 172.21.15.32:0/863322760 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7ff088017070 con 0x7ff09806e930 2024-08-03T20:48:37.050 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:37.047+0000 7ff09da00640 1 -- 172.21.15.32:0/863322760 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff0980cbdf0 con 0x7ff09806e930 2024-08-03T20:48:37.050 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:37.047+0000 7ff08ea00640 1 -- 172.21.15.32:0/863322760 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff088011da0 con 0x7ff09806e930 2024-08-03T20:48:37.050 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:37.047+0000 7ff09da00640 1 -- 172.21.15.32:0/863322760 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff09813d5a0 con 0x7ff09806e930 2024-08-03T20:48:37.050 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:37.048+0000 7ff09da00640 1 -- 172.21.15.32:0/863322760 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff09813d7e0 con 0x7ff09806e930 2024-08-03T20:48:37.051 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:37.049+0000 7ff08ea00640 1 -- 172.21.15.32:0/863322760 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7ff08801c460 con 0x7ff09806e930 2024-08-03T20:48:37.051 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:37.050+0000 7ff08ea00640 1 --2- 172.21.15.32:0/863322760 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7ff074077010 0x7ff074079520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:48:37.051 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:37.050+0000 7ff08ea00640 1 -- 172.21.15.32:0/863322760 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(32..32 src has 1..32) ==== 5709+0+0 (secure 0 0 0) 0x7ff0880985e0 con 0x7ff09806e930 2024-08-03T20:48:37.052 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:37.051+0000 7ff08fe00640 1 --2- 172.21.15.32:0/863322760 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7ff074077010 0x7ff074079520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:48:37.053 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:37.051+0000 7ff08fe00640 1 --2- 172.21.15.32:0/863322760 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7ff074077010 0x7ff074079520 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7ff080034000 tx=0x7ff080004730 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:48:37.056 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:37.055+0000 7ff08ea00640 1 -- 172.21.15.32:0/863322760 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7ff09813d7e0 con 0x7ff09806e930 2024-08-03T20:48:37.301 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:37.300+0000 7ff09da00640 1 -- 172.21.15.32:0/863322760 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff09813feb0 con 0x7ff09806e930 2024-08-03T20:48:37.302 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:37.300+0000 7ff08ea00640 1 -- 172.21.15.32:0/863322760 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) ==== 94+0+5229 (secure 0 0 0) 0x7ff09813feb0 con 0x7ff09806e930 2024-08-03T20:48:37.302 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-03T20:48:37.302 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":17,"btime":"2024-08-03T20:48:30:142972+0000","default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4643,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6836/4002526832","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6834","nonce":4002526832},{"type":"v1","addr":"172.21.15.118:6836","nonce":4002526832}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":4646,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6837/3036256187","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6835","nonce":3036256187},{"type":"v1","addr":"172.21.15.118:6837","nonce":3036256187}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":4652,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6834/2104418334","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6833","nonce":2104418334},{"type":"v1","addr":"172.21.15.32:6834","nonce":2104418334}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-08-03T20:48:26.652521+0000","modified":"2024-08-03T20:48:29.661524+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4866},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4866":{"gid":4866,"name":"a","rank":0,"incarnation":13,"state":"up:active","state_seq":3,"addr":"172.21.15.32:6835/1186001746","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":1186001746},{"type":"v1","addr":"172.21.15.32:6835","nonce":1186001746}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4866,"qdb_cluster":[4866]},"id":2}]} 2024-08-03T20:48:37.303 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 17 2024-08-03T20:48:37.305 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:37.303+0000 7ff09da00640 1 -- 172.21.15.32:0/863322760 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7ff074077010 msgr2=0x7ff074079520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:48:37.305 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:37.303+0000 7ff09da00640 1 --2- 172.21.15.32:0/863322760 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7ff074077010 0x7ff074079520 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7ff080034000 tx=0x7ff080004730 comp rx=0 tx=0).stop 2024-08-03T20:48:37.305 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:37.303+0000 7ff09da00640 1 -- 172.21.15.32:0/863322760 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff09806e930 msgr2=0x7ff09815bb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:48:37.305 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:37.303+0000 7ff09da00640 1 --2- 172.21.15.32:0/863322760 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff09806e930 0x7ff09815bb90 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7ff08800c460 tx=0x7ff08800c920 comp rx=0 tx=0).stop 2024-08-03T20:48:37.305 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:37.303+0000 7ff09da00640 1 -- 172.21.15.32:0/863322760 shutdown_connections 2024-08-03T20:48:37.305 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:37.303+0000 7ff09da00640 1 --2- 172.21.15.32:0/863322760 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7ff074077010 0x7ff074079520 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:37.305 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:37.303+0000 7ff09da00640 1 --2- 172.21.15.32:0/863322760 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ff09815fff0 0x7ff09815e000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:37.305 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:37.303+0000 7ff09da00640 1 --2- 172.21.15.32:0/863322760 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7ff098144f90 0x7ff09815dac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:37.305 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:37.303+0000 7ff09da00640 1 --2- 172.21.15.32:0/863322760 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff09806e930 0x7ff09815bb90 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:37.305 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:37.303+0000 7ff09da00640 1 -- 172.21.15.32:0/863322760 >> 172.21.15.32:0/863322760 conn(0x7ff0980b1bd0 msgr2=0x7ff098074520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:48:37.305 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:37.303+0000 7ff09da00640 1 -- 172.21.15.32:0/863322760 shutdown_connections 2024-08-03T20:48:37.305 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:37.303+0000 7ff09da00640 1 -- 172.21.15.32:0/863322760 wait complete. 2024-08-03T20:48:37.331 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json flush journal 2024-08-03T20:48:37.587 INFO:teuthology.orchestra.run.smithi032.stdout:{"message":"","return_code":0} 2024-08-03T20:48:37.588 DEBUG:tasks.cephfs.filesystem:_json_asok output { "message": "", "return_code": 0 } 2024-08-03T20:48:37.588 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-08-03T20:48:37.817 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:37.815+0000 7fdae7200640 1 -- 172.21.15.32:0/2757435474 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdae80ca3b0 msgr2=0x7fdae80ca7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:48:37.817 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:37.815+0000 7fdae7200640 1 --2- 172.21.15.32:0/2757435474 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdae80ca3b0 0x7fdae80ca7c0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7fdad0005fd0 tx=0x7fdad002b910 comp rx=0 tx=0).stop 2024-08-03T20:48:37.817 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:37.815+0000 7fdae7200640 1 -- 172.21.15.32:0/2757435474 shutdown_connections 2024-08-03T20:48:37.817 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:37.815+0000 7fdae7200640 1 --2- 172.21.15.32:0/2757435474 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdae80ca3b0 0x7fdae80ca7c0 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:37.817 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:37.815+0000 7fdae7200640 1 -- 172.21.15.32:0/2757435474 >> 172.21.15.32:0/2757435474 conn(0x7fdae80b2120 msgr2=0x7fdae80cfd60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:48:37.817 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:37.815+0000 7fdae7200640 1 -- 172.21.15.32:0/2757435474 shutdown_connections 2024-08-03T20:48:37.817 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:37.815+0000 7fdae7200640 1 -- 172.21.15.32:0/2757435474 wait complete. 2024-08-03T20:48:37.819 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:37.817+0000 7fdae7200640 1 Processor -- start 2024-08-03T20:48:37.819 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:37.817+0000 7fdae7200640 1 -- start start 2024-08-03T20:48:37.822 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:37.819+0000 7fdae7200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdae80ca3b0 0x7fdae80c3840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:48:37.822 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:37.819+0000 7fdae7200640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fdae80ce150 0x7fdae8156980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:48:37.822 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:37.819+0000 7fdae7200640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fdae815a310 0x7fdae80b42b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:48:37.822 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:37.819+0000 7fdae7200640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fdae8070860 con 0x7fdae80ca3b0 2024-08-03T20:48:37.822 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:37.819+0000 7fdae7200640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7fdae80706e0 con 0x7fdae815a310 2024-08-03T20:48:37.822 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:37.819+0000 7fdae7200640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7fdae80cec70 con 0x7fdae80ce150 2024-08-03T20:48:37.822 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:37.819+0000 7fdae5e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdae80ca3b0 0x7fdae80c3840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:48:37.822 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:37.819+0000 7fdae5e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdae80ca3b0 0x7fdae80c3840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:46576/0 (socket says 172.21.15.32:46576) 2024-08-03T20:48:37.822 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:37.819+0000 7fdae5e00640 1 -- 172.21.15.32:0/446098821 learned_addr learned my addr 172.21.15.32:0/446098821 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:48:37.822 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:37.820+0000 7fdae5e00640 1 -- 172.21.15.32:0/446098821 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fdae80ce150 msgr2=0x7fdae8156980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:48:37.822 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:37.820+0000 7fdae5e00640 1 --2- 172.21.15.32:0/446098821 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fdae80ce150 0x7fdae8156980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:37.823 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:37.820+0000 7fdae5e00640 1 -- 172.21.15.32:0/446098821 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fdae815a310 msgr2=0x7fdae80b42b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:48:37.823 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:37.820+0000 7fdae5e00640 1 --2- 172.21.15.32:0/446098821 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fdae815a310 0x7fdae80b42b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:37.823 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:37.820+0000 7fdae5e00640 1 -- 172.21.15.32:0/446098821 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdae81f8f00 con 0x7fdae80ca3b0 2024-08-03T20:48:37.823 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:37.820+0000 7fdae5e00640 1 --2- 172.21.15.32:0/446098821 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdae80ca3b0 0x7fdae80c3840 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7fdad800b420 tx=0x7fdad800b8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:48:37.823 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:37.820+0000 7fdade000640 1 -- 172.21.15.32:0/446098821 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdad8011830 con 0x7fdae80ca3b0 2024-08-03T20:48:37.823 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:37.820+0000 7fdade000640 1 -- 172.21.15.32:0/446098821 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fdad8016070 con 0x7fdae80ca3b0 2024-08-03T20:48:37.823 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:37.820+0000 7fdae7200640 1 -- 172.21.15.32:0/446098821 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdae81f91e0 con 0x7fdae80ca3b0 2024-08-03T20:48:37.823 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:37.820+0000 7fdade000640 1 -- 172.21.15.32:0/446098821 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdad8011d60 con 0x7fdae80ca3b0 2024-08-03T20:48:37.823 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:37.820+0000 7fdae7200640 1 -- 172.21.15.32:0/446098821 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdae8156ec0 con 0x7fdae80ca3b0 2024-08-03T20:48:37.824 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:37.821+0000 7fdae7200640 1 -- 172.21.15.32:0/446098821 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdae80cf1d0 con 0x7fdae80ca3b0 2024-08-03T20:48:37.824 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:37.823+0000 7fdade000640 1 -- 172.21.15.32:0/446098821 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7fdad801b460 con 0x7fdae80ca3b0 2024-08-03T20:48:37.825 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:37.823+0000 7fdade000640 1 --2- 172.21.15.32:0/446098821 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fdac4077040 0x7fdac4079550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:48:37.825 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:37.824+0000 7fdade000640 1 -- 172.21.15.32:0/446098821 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(32..32 src has 1..32) ==== 5709+0+0 (secure 0 0 0) 0x7fdad8098540 con 0x7fdae80ca3b0 2024-08-03T20:48:37.825 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:37.824+0000 7fdadfe00640 1 --2- 172.21.15.32:0/446098821 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fdac4077040 0x7fdac4079550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:48:37.826 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:37.825+0000 7fdadfe00640 1 --2- 172.21.15.32:0/446098821 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fdac4077040 0x7fdac4079550 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fdad0035000 tx=0x7fdad00046c0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:48:37.829 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:37.828+0000 7fdade000640 1 -- 172.21.15.32:0/446098821 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7fdad8020020 con 0x7fdae80ca3b0 2024-08-03T20:48:38.076 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:38.074+0000 7fdae7200640 1 -- 172.21.15.32:0/446098821 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7fdae80b0f20 con 0x7fdae80ca3b0 2024-08-03T20:48:38.076 INFO:tasks.ceph.mon.a.smithi032.stderr:2024-08-03T20:48:38.075+0000 7f4e6d600640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-08-03T20:48:38.078 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:38.077+0000 7fdade000640 1 -- 172.21.15.32:0/446098821 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v18) ==== 161+0+0 (secure 0 0 0) 0x7fdad80616a0 con 0x7fdae80ca3b0 2024-08-03T20:48:38.079 INFO:teuthology.orchestra.run.smithi032.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-08-03T20:48:38.082 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:38.081+0000 7fdae7200640 1 -- 172.21.15.32:0/446098821 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fdac4077040 msgr2=0x7fdac4079550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:48:38.082 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:38.081+0000 7fdae7200640 1 --2- 172.21.15.32:0/446098821 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fdac4077040 0x7fdac4079550 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fdad0035000 tx=0x7fdad00046c0 comp rx=0 tx=0).stop 2024-08-03T20:48:38.082 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:38.081+0000 7fdae7200640 1 -- 172.21.15.32:0/446098821 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdae80ca3b0 msgr2=0x7fdae80c3840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:48:38.082 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:38.081+0000 7fdae7200640 1 --2- 172.21.15.32:0/446098821 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdae80ca3b0 0x7fdae80c3840 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7fdad800b420 tx=0x7fdad800b8e0 comp rx=0 tx=0).stop 2024-08-03T20:48:38.083 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:38.081+0000 7fdae7200640 1 -- 172.21.15.32:0/446098821 shutdown_connections 2024-08-03T20:48:38.083 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:38.081+0000 7fdae7200640 1 --2- 172.21.15.32:0/446098821 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fdac4077040 0x7fdac4079550 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:38.083 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:38.081+0000 7fdae7200640 1 --2- 172.21.15.32:0/446098821 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fdae815a310 0x7fdae80b42b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:38.083 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:38.081+0000 7fdae7200640 1 --2- 172.21.15.32:0/446098821 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fdae80ce150 0x7fdae8156980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:38.083 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:38.081+0000 7fdae7200640 1 --2- 172.21.15.32:0/446098821 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdae80ca3b0 0x7fdae80c3840 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:38.083 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:38.081+0000 7fdae7200640 1 -- 172.21.15.32:0/446098821 >> 172.21.15.32:0/446098821 conn(0x7fdae80b2120 msgr2=0x7fdae80ce560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:48:38.083 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:38.081+0000 7fdae7200640 1 -- 172.21.15.32:0/446098821 shutdown_connections 2024-08-03T20:48:38.083 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:38.081+0000 7fdae7200640 1 -- 172.21.15.32:0/446098821 wait complete. 2024-08-03T20:48:38.090 INFO:tasks.ceph.mds.a.smithi032.stderr: -5122> 2024-08-03T20:48:31.093+0000 7fe39a200640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-08-03T20:48:38.107 DEBUG:teuthology.orchestra.run.smithi032:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata setomapval 10000000000.00000000 file_to_be_damaged_head deadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeef 2024-08-03T20:48:38.219 INFO:tasks.ceph.mds.a.smithi032.stdout:starting mds.a at 2024-08-03T20:48:38.253 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:38.251+0000 7fef8238ff40 1 Processor -- start 2024-08-03T20:48:38.253 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:38.252+0000 7fef8238ff40 1 -- start start 2024-08-03T20:48:38.255 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:38.253+0000 7fef8238ff40 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55cea75a4740 0x55cea755b730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:48:38.255 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:38.253+0000 7fef8238ff40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55cea7611df0 0x55cea760d0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:48:38.255 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:38.253+0000 7fef8238ff40 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55cea762f280 0x55cea7630cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:48:38.255 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:38.253+0000 7fef8238ff40 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55cea7507bf0 con 0x55cea75a4740 2024-08-03T20:48:38.255 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:38.253+0000 7fef8238ff40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x55cea75146b0 con 0x55cea7611df0 2024-08-03T20:48:38.255 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:38.253+0000 7fef8238ff40 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x55cea74a9e30 con 0x55cea762f280 2024-08-03T20:48:38.255 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:38.253+0000 7fef7b600640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55cea75a4740 0x55cea755b730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:48:38.255 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:38.253+0000 7fef7b600640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55cea75a4740 0x55cea755b730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:46616/0 (socket says 172.21.15.32:46616) 2024-08-03T20:48:38.255 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:38.253+0000 7fef7b600640 1 -- 172.21.15.32:0/4021440274 learned_addr learned my addr 172.21.15.32:0/4021440274 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:48:38.257 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:38.254+0000 7fef7b600640 1 -- 172.21.15.32:0/4021440274 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55cea762f280 msgr2=0x55cea7630cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:48:38.257 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:38.254+0000 7fef7ac00640 1 --2- 172.21.15.32:0/4021440274 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55cea7611df0 0x55cea760d0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:48:38.257 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:38.254+0000 7fef7c000640 1 --2- 172.21.15.32:0/4021440274 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55cea762f280 0x55cea7630cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:48:38.257 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:38.254+0000 7fef7b600640 1 --2- 172.21.15.32:0/4021440274 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55cea762f280 0x55cea7630cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:38.257 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:38.254+0000 7fef7b600640 1 -- 172.21.15.32:0/4021440274 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55cea7611df0 msgr2=0x55cea760d0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:48:38.257 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:38.254+0000 7fef7b600640 1 --2- 172.21.15.32:0/4021440274 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55cea7611df0 0x55cea760d0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:38.257 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:38.254+0000 7fef7b600640 1 -- 172.21.15.32:0/4021440274 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cea76302a0 con 0x55cea75a4740 2024-08-03T20:48:38.257 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:38.254+0000 7fef7ac00640 1 --2- 172.21.15.32:0/4021440274 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55cea7611df0 0x55cea760d0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-03T20:48:38.258 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:38.254+0000 7fef7c000640 1 --2- 172.21.15.32:0/4021440274 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55cea762f280 0x55cea7630cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-03T20:48:38.258 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:38.254+0000 7fef7b600640 1 --2- 172.21.15.32:0/4021440274 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55cea75a4740 0x55cea755b730 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7fef6c001df0 tx=0x7fef6c00b670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cbeb90d4fd383e3b server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:48:38.258 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:38.254+0000 7fef79800640 1 -- 172.21.15.32:0/4021440274 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef6c012070 con 0x55cea75a4740 2024-08-03T20:48:38.258 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:38.254+0000 7fef79800640 1 -- 172.21.15.32:0/4021440274 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fef6c006050 con 0x55cea75a4740 2024-08-03T20:48:38.258 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:38.254+0000 7fef79800640 1 -- 172.21.15.32:0/4021440274 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef6c00ebe0 con 0x55cea75a4740 2024-08-03T20:48:38.258 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:38.255+0000 7fef8238ff40 1 -- 172.21.15.32:0/4021440274 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55cea75a4740 msgr2=0x55cea755b730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:48:38.258 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:38.255+0000 7fef8238ff40 1 --2- 172.21.15.32:0/4021440274 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55cea75a4740 0x55cea755b730 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7fef6c001df0 tx=0x7fef6c00b670 comp rx=0 tx=0).stop 2024-08-03T20:48:38.258 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:38.255+0000 7fef8238ff40 1 -- 172.21.15.32:0/4021440274 shutdown_connections 2024-08-03T20:48:38.258 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:38.255+0000 7fef8238ff40 1 --2- 172.21.15.32:0/4021440274 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55cea7611df0 0x55cea760d0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:38.258 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:38.255+0000 7fef8238ff40 1 --2- 172.21.15.32:0/4021440274 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55cea762f280 0x55cea7630cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:38.258 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:38.255+0000 7fef8238ff40 1 --2- 172.21.15.32:0/4021440274 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55cea75a4740 0x55cea755b730 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:38.258 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:38.255+0000 7fef8238ff40 1 -- 172.21.15.32:0/4021440274 >> 172.21.15.32:0/4021440274 conn(0x55cea747bac0 msgr2=0x55cea7610e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:48:38.258 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:38.255+0000 7fef8238ff40 1 -- 172.21.15.32:0/4021440274 shutdown_connections 2024-08-03T20:48:38.259 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:38.255+0000 7fef8238ff40 1 -- 172.21.15.32:0/4021440274 wait complete. 2024-08-03T20:48:38.260 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:38.256+0000 7fef8238ff40 1 Processor -- start 2024-08-03T20:48:38.260 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:38.256+0000 7fef8238ff40 1 -- start start 2024-08-03T20:48:38.260 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:38.258+0000 7fef8238ff40 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55cea75a4740 0x55cea7608950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:48:38.260 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:38.258+0000 7fef8238ff40 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55cea7611df0 0x55cea759b940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:48:38.261 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:38.258+0000 7fef8238ff40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55cea762f280 0x55cea7630b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:48:38.261 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:38.258+0000 7fef8238ff40 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55cea7630580 con 0x55cea7611df0 2024-08-03T20:48:38.261 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:38.258+0000 7fef8238ff40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x55cea755bc70 con 0x55cea762f280 2024-08-03T20:48:38.261 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:38.258+0000 7fef8238ff40 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x55cea76b88e0 con 0x55cea75a4740 2024-08-03T20:48:38.261 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:38.258+0000 7fef7ac00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55cea7611df0 0x55cea759b940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:48:38.261 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:38.258+0000 7fef7ac00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55cea7611df0 0x55cea759b940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:46626/0 (socket says 172.21.15.32:46626) 2024-08-03T20:48:38.261 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:38.258+0000 7fef7ac00640 1 -- 172.21.15.32:0/4083292875 learned_addr learned my addr 172.21.15.32:0/4083292875 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:48:38.261 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:38.258+0000 7fef7b600640 1 --2- 172.21.15.32:0/4083292875 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55cea75a4740 0x55cea7608950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:48:38.261 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:38.258+0000 7fef7c000640 1 --2- 172.21.15.32:0/4083292875 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55cea762f280 0x55cea7630b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:48:38.261 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:38.259+0000 7fef7ac00640 1 -- 172.21.15.32:0/4083292875 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55cea75a4740 msgr2=0x55cea7608950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:48:38.261 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:38.259+0000 7fef7ac00640 1 --2- 172.21.15.32:0/4083292875 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55cea75a4740 0x55cea7608950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:38.261 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:38.259+0000 7fef7ac00640 1 -- 172.21.15.32:0/4083292875 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55cea762f280 msgr2=0x55cea7630b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:48:38.261 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:38.259+0000 7fef7ac00640 1 --2- 172.21.15.32:0/4083292875 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55cea762f280 0x55cea7630b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:38.261 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:38.259+0000 7fef7ac00640 1 -- 172.21.15.32:0/4083292875 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cea7608e90 con 0x55cea7611df0 2024-08-03T20:48:38.262 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:38.259+0000 7fef7b600640 1 --2- 172.21.15.32:0/4083292875 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55cea75a4740 0x55cea7608950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-03T20:48:38.262 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:38.259+0000 7fef7ac00640 1 --2- 172.21.15.32:0/4083292875 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55cea7611df0 0x55cea759b940 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7fef68004f40 tx=0x7fef6804d740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:48:38.262 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:38.259+0000 7fef73e00640 1 -- 172.21.15.32:0/4083292875 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef68056670 con 0x55cea7611df0 2024-08-03T20:48:38.262 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:38.259+0000 7fef73e00640 1 -- 172.21.15.32:0/4083292875 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fef68066070 con 0x55cea7611df0 2024-08-03T20:48:38.262 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:38.259+0000 7fef8238ff40 1 -- 172.21.15.32:0/4083292875 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x55cea759c4b0 con 0x55cea7611df0 2024-08-03T20:48:38.263 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:38.259+0000 7fef73e00640 1 -- 172.21.15.32:0/4083292875 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef68056b60 con 0x55cea7611df0 2024-08-03T20:48:38.263 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:38.259+0000 7fef8238ff40 1 -- 172.21.15.32:0/4083292875 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cea759ca70 con 0x55cea7611df0 2024-08-03T20:48:38.263 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:38.262+0000 7fef73e00640 1 -- 172.21.15.32:0/4083292875 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7fef6806a410 con 0x55cea7611df0 2024-08-03T20:48:38.264 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:38.262+0000 7fef73e00640 1 --2- 172.21.15.32:0/4083292875 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fef5c075a20 0x7fef5c077f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:48:38.264 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:38.263+0000 7fef73e00640 1 -- 172.21.15.32:0/4083292875 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(33..33 src has 1..33) ==== 5795+0+0 (secure 0 0 0) 0x7fef680dbbe0 con 0x55cea7611df0 2024-08-03T20:48:38.264 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:38.263+0000 7fef8238ff40 1 --2- 172.21.15.32:0/4083292875 >> [v2:172.21.15.32:6800/9088303,v1:172.21.15.32:6801/9088303] conn(0x55cea76b7ab0 0x55cea76b7ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:48:38.264 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:38.263+0000 7fef8238ff40 1 -- 172.21.15.32:0/4083292875 --> [v2:172.21.15.32:6800/9088303,v1:172.21.15.32:6801/9088303] -- osd_op(unknown.0.0:1 4.10 4:0cf6ad0f:::10000000000.00000000:head [omap-set-vals in=115b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e33) -- 0x55cea76d4710 con 0x55cea76b7ab0 2024-08-03T20:48:38.265 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:38.263+0000 7fef7b600640 1 --2- 172.21.15.32:0/4083292875 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fef5c075a20 0x7fef5c077f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:48:38.265 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:38.263+0000 7fef7c000640 1 --2- 172.21.15.32:0/4083292875 >> [v2:172.21.15.32:6800/9088303,v1:172.21.15.32:6801/9088303] conn(0x55cea76b7ab0 0x55cea76b7ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:48:38.265 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:38.264+0000 7fef7c000640 1 --2- 172.21.15.32:0/4083292875 >> [v2:172.21.15.32:6800/9088303,v1:172.21.15.32:6801/9088303] conn(0x55cea76b7ab0 0x55cea76b7ec0 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:48:38.265 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:38.264+0000 7fef7b600640 1 --2- 172.21.15.32:0/4083292875 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fef5c075a20 0x7fef5c077f30 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fef6c001df0 tx=0x7fef6c01a040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:48:38.267 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:38.265+0000 7fef7c000640 1 -- 172.21.15.32:0/4083292875 <== osd.3 v2:172.21.15.32:6800/9088303 1 ==== osd_op_reply(1 10000000000.00000000 [omap-set-vals] v33'11 uv11 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7fef74002610 con 0x55cea76b7ab0 2024-08-03T20:48:38.267 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:38.266+0000 7fef8238ff40 1 -- 172.21.15.32:0/4083292875 >> [v2:172.21.15.32:6800/9088303,v1:172.21.15.32:6801/9088303] conn(0x55cea76b7ab0 msgr2=0x55cea76b7ec0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:48:38.267 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:38.266+0000 7fef8238ff40 1 --2- 172.21.15.32:0/4083292875 >> [v2:172.21.15.32:6800/9088303,v1:172.21.15.32:6801/9088303] conn(0x55cea76b7ab0 0x55cea76b7ec0 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:38.268 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:38.266+0000 7fef8238ff40 1 -- 172.21.15.32:0/4083292875 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fef5c075a20 msgr2=0x7fef5c077f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:48:38.268 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:38.266+0000 7fef8238ff40 1 --2- 172.21.15.32:0/4083292875 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fef5c075a20 0x7fef5c077f30 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fef6c001df0 tx=0x7fef6c01a040 comp rx=0 tx=0).stop 2024-08-03T20:48:38.268 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:38.266+0000 7fef8238ff40 1 -- 172.21.15.32:0/4083292875 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55cea7611df0 msgr2=0x55cea759b940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:48:38.268 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:38.266+0000 7fef8238ff40 1 --2- 172.21.15.32:0/4083292875 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55cea7611df0 0x55cea759b940 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7fef68004f40 tx=0x7fef6804d740 comp rx=0 tx=0).stop 2024-08-03T20:48:38.268 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:38.266+0000 7fef7c000640 1 -- 172.21.15.32:0/4083292875 reap_dead start 2024-08-03T20:48:38.268 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:38.266+0000 7fef8238ff40 1 -- 172.21.15.32:0/4083292875 shutdown_connections 2024-08-03T20:48:38.268 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:38.266+0000 7fef8238ff40 1 -- 172.21.15.32:0/4083292875 >> 172.21.15.32:0/4083292875 conn(0x55cea747bac0 msgr2=0x55cea76319d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:48:38.268 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:38.266+0000 7fef8238ff40 1 -- 172.21.15.32:0/4083292875 shutdown_connections 2024-08-03T20:48:38.268 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:38.266+0000 7fef8238ff40 1 -- 172.21.15.32:0/4083292875 wait complete. 2024-08-03T20:48:38.279 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs joinable true 2024-08-03T20:48:38.512 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:38.511+0000 7f6bf3200640 1 -- 172.21.15.32:0/3884324823 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6bec0d26e0 msgr2=0x7f6bec0d4090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:48:38.513 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:38.511+0000 7f6bf3200640 1 --2- 172.21.15.32:0/3884324823 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6bec0d26e0 0x7f6bec0d4090 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f6be0009830 tx=0x7f6be002a510 comp rx=0 tx=0).stop 2024-08-03T20:48:38.513 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:38.511+0000 7f6bf3200640 1 -- 172.21.15.32:0/3884324823 shutdown_connections 2024-08-03T20:48:38.513 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:38.511+0000 7f6bf3200640 1 --2- 172.21.15.32:0/3884324823 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f6bec0d45d0 0x7f6bec162110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:38.513 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:38.511+0000 7f6bf3200640 1 --2- 172.21.15.32:0/3884324823 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6bec0d26e0 0x7f6bec0d4090 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:38.513 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:38.511+0000 7f6bf3200640 1 -- 172.21.15.32:0/3884324823 >> 172.21.15.32:0/3884324823 conn(0x7f6bec08d7e0 msgr2=0x7f6bec09d410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:48:38.513 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:38.511+0000 7f6bf3200640 1 -- 172.21.15.32:0/3884324823 shutdown_connections 2024-08-03T20:48:38.513 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:38.511+0000 7f6bf3200640 1 -- 172.21.15.32:0/3884324823 wait complete. 2024-08-03T20:48:38.514 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:38.513+0000 7f6bf3200640 1 Processor -- start 2024-08-03T20:48:38.515 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:38.513+0000 7f6bf3200640 1 -- start start 2024-08-03T20:48:38.517 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:38.515+0000 7f6bf3200640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f6bec0d26e0 0x7f6bec0cb8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:48:38.517 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:38.515+0000 7f6bf3200640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f6bec0d45d0 0x7f6bec157940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:48:38.517 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:38.515+0000 7f6bf3200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6bec162650 0x7f6bec0b8820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:48:38.517 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:38.515+0000 7f6bf3200640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f6bec162be0 con 0x7f6bec162650 2024-08-03T20:48:38.517 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:38.515+0000 7f6bf3200640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f6bec162a60 con 0x7f6bec0d26e0 2024-08-03T20:48:38.517 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:38.515+0000 7f6bf3200640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7f6bec162d60 con 0x7f6bec0d45d0 2024-08-03T20:48:38.517 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:38.515+0000 7f6bf2800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6bec162650 0x7f6bec0b8820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:48:38.518 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:38.515+0000 7f6bf2800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6bec162650 0x7f6bec0b8820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:46640/0 (socket says 172.21.15.32:46640) 2024-08-03T20:48:38.518 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:38.515+0000 7f6bf2800640 1 -- 172.21.15.32:0/2322335791 learned_addr learned my addr 172.21.15.32:0/2322335791 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:48:38.518 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:38.515+0000 7f6bf1400640 1 --2- 172.21.15.32:0/2322335791 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f6bec0d45d0 0x7f6bec157940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:48:38.518 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:38.515+0000 7f6bf1e00640 1 --2- 172.21.15.32:0/2322335791 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f6bec0d26e0 0x7f6bec0cb8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:48:38.518 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:38.516+0000 7f6bf2800640 1 -- 172.21.15.32:0/2322335791 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f6bec0d45d0 msgr2=0x7f6bec157940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:48:38.518 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:38.516+0000 7f6bf2800640 1 --2- 172.21.15.32:0/2322335791 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f6bec0d45d0 0x7f6bec157940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:38.518 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:38.516+0000 7f6bf2800640 1 -- 172.21.15.32:0/2322335791 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f6bec0d26e0 msgr2=0x7f6bec0cb8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:48:38.518 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:38.516+0000 7f6bf2800640 1 --2- 172.21.15.32:0/2322335791 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f6bec0d26e0 0x7f6bec0cb8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:38.518 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:38.516+0000 7f6bf2800640 1 -- 172.21.15.32:0/2322335791 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6bec0cd220 con 0x7f6bec162650 2024-08-03T20:48:38.519 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:38.516+0000 7f6bf2800640 1 --2- 172.21.15.32:0/2322335791 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6bec162650 0x7f6bec0b8820 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f6be800cb10 tx=0x7f6be800a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:48:38.519 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:38.516+0000 7f6be6000640 1 -- 172.21.15.32:0/2322335791 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6be8016020 con 0x7f6bec162650 2024-08-03T20:48:38.520 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:38.516+0000 7f6be6000640 1 -- 172.21.15.32:0/2322335791 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f6be800ad00 con 0x7f6bec162650 2024-08-03T20:48:38.520 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:38.516+0000 7f6bf3200640 1 -- 172.21.15.32:0/2322335791 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6bec1587a0 con 0x7f6bec162650 2024-08-03T20:48:38.520 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:38.516+0000 7f6be6000640 1 -- 172.21.15.32:0/2322335791 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6be8012680 con 0x7f6bec162650 2024-08-03T20:48:38.520 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:38.516+0000 7f6bf3200640 1 -- 172.21.15.32:0/2322335791 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6bec158d60 con 0x7f6bec162650 2024-08-03T20:48:38.520 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:38.517+0000 7f6bf3200640 1 -- 172.21.15.32:0/2322335791 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6bec0d4090 con 0x7f6bec162650 2024-08-03T20:48:38.520 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:38.519+0000 7f6be6000640 1 -- 172.21.15.32:0/2322335791 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f6be8019020 con 0x7f6bec162650 2024-08-03T20:48:38.522 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:38.520+0000 7f6be6000640 1 --2- 172.21.15.32:0/2322335791 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f6bd0077010 0x7f6bd0079520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:48:38.522 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:38.520+0000 7f6be6000640 1 -- 172.21.15.32:0/2322335791 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(33..33 src has 1..33) ==== 5795+0+0 (secure 0 0 0) 0x7f6be8098c80 con 0x7f6bec162650 2024-08-03T20:48:38.522 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:38.520+0000 7f6bf1e00640 1 --2- 172.21.15.32:0/2322335791 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f6bd0077010 0x7f6bd0079520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:48:38.522 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:38.521+0000 7f6bf1e00640 1 --2- 172.21.15.32:0/2322335791 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f6bd0077010 0x7f6bd0079520 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f6be0034000 tx=0x7f6be0004730 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:48:38.525 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:38.524+0000 7f6be6000640 1 -- 172.21.15.32:0/2322335791 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7f6be8020030 con 0x7f6bec162650 2024-08-03T20:48:38.769 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:38.767+0000 7f6bf3200640 1 -- 172.21.15.32:0/2322335791 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "joinable", "val": "true"} v 0) -- 0x7f6bec0d24c0 con 0x7f6bec162650 2024-08-03T20:48:39.086 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:39.085+0000 7f6be6000640 1 -- 172.21.15.32:0/2322335791 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "joinable", "val": "true"}]=0 cephfs marked joinable; MDS may join as newly active. v19) ==== 162+0+0 (secure 0 0 0) 0x7f6bec0d49e0 con 0x7f6bec162650 2024-08-03T20:48:39.086 INFO:teuthology.orchestra.run.smithi032.stderr:cephfs marked joinable; MDS may join as newly active. 2024-08-03T20:48:39.089 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:39.087+0000 7f6bf3200640 1 -- 172.21.15.32:0/2322335791 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f6bd0077010 msgr2=0x7f6bd0079520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:48:39.089 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:39.087+0000 7f6bf3200640 1 --2- 172.21.15.32:0/2322335791 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f6bd0077010 0x7f6bd0079520 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f6be0034000 tx=0x7f6be0004730 comp rx=0 tx=0).stop 2024-08-03T20:48:39.089 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:39.087+0000 7f6bf3200640 1 -- 172.21.15.32:0/2322335791 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6bec162650 msgr2=0x7f6bec0b8820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:48:39.089 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:39.087+0000 7f6bf3200640 1 --2- 172.21.15.32:0/2322335791 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6bec162650 0x7f6bec0b8820 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f6be800cb10 tx=0x7f6be800a060 comp rx=0 tx=0).stop 2024-08-03T20:48:39.089 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:39.087+0000 7f6bf3200640 1 -- 172.21.15.32:0/2322335791 shutdown_connections 2024-08-03T20:48:39.089 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:39.087+0000 7f6bf3200640 1 --2- 172.21.15.32:0/2322335791 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f6bd0077010 0x7f6bd0079520 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:39.090 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:39.087+0000 7f6bf3200640 1 --2- 172.21.15.32:0/2322335791 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6bec162650 0x7f6bec0b8820 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:39.090 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:39.087+0000 7f6bf3200640 1 --2- 172.21.15.32:0/2322335791 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f6bec0d45d0 0x7f6bec157940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:39.090 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:39.087+0000 7f6bf3200640 1 --2- 172.21.15.32:0/2322335791 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f6bec0d26e0 0x7f6bec0cb8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:39.090 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:39.087+0000 7f6bf3200640 1 -- 172.21.15.32:0/2322335791 >> 172.21.15.32:0/2322335791 conn(0x7f6bec08d7e0 msgr2=0x7f6bec0d0700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:48:39.090 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:39.088+0000 7f6bf3200640 1 -- 172.21.15.32:0/2322335791 shutdown_connections 2024-08-03T20:48:39.090 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:39.088+0000 7f6bf3200640 1 -- 172.21.15.32:0/2322335791 wait complete. 2024-08-03T20:48:39.117 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-08-03T20:48:39.346 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:39.344+0000 7f30dde00640 1 -- 172.21.15.32:0/4235095490 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f30d80d26e0 msgr2=0x7f30d8161b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:48:39.346 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:39.344+0000 7f30dde00640 1 --2- 172.21.15.32:0/4235095490 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f30d80d26e0 0x7f30d8161b80 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f30cc005fd0 tx=0x7f30cc02a5d0 comp rx=0 tx=0).stop 2024-08-03T20:48:39.346 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:39.344+0000 7f30dde00640 1 -- 172.21.15.32:0/4235095490 shutdown_connections 2024-08-03T20:48:39.346 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:39.344+0000 7f30dde00640 1 --2- 172.21.15.32:0/4235095490 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f30d80d26e0 0x7f30d8161b80 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:39.346 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:39.344+0000 7f30dde00640 1 -- 172.21.15.32:0/4235095490 >> 172.21.15.32:0/4235095490 conn(0x7f30d808d7a0 msgr2=0x7f30d809d3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:48:39.347 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:39.344+0000 7f30dde00640 1 -- 172.21.15.32:0/4235095490 shutdown_connections 2024-08-03T20:48:39.347 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:39.344+0000 7f30dde00640 1 -- 172.21.15.32:0/4235095490 wait complete. 2024-08-03T20:48:39.348 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:39.347+0000 7f30dde00640 1 Processor -- start 2024-08-03T20:48:39.348 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:39.347+0000 7f30dde00640 1 -- start start 2024-08-03T20:48:39.351 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:39.348+0000 7f30dde00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f30d80cf510 0x7f30d80ccec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:48:39.351 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:39.348+0000 7f30dde00640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f30d80cffe0 0x7f30d80cd400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:48:39.351 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:39.348+0000 7f30dde00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f30d80d26e0 0x7f30d80c7c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:48:39.352 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:39.348+0000 7f30dde00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f30d80cf920 con 0x7f30d80d26e0 2024-08-03T20:48:39.352 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:39.348+0000 7f30dde00640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f30d80d48a0 con 0x7f30d80cf510 2024-08-03T20:48:39.352 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:39.348+0000 7f30dde00640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7f30d80cfaa0 con 0x7f30d80cffe0 2024-08-03T20:48:39.352 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:39.349+0000 7f30dd400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f30d80d26e0 0x7f30d80c7c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:48:39.352 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:39.349+0000 7f30dd400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f30d80d26e0 0x7f30d80c7c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:46662/0 (socket says 172.21.15.32:46662) 2024-08-03T20:48:39.352 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:39.349+0000 7f30dd400640 1 -- 172.21.15.32:0/388518996 learned_addr learned my addr 172.21.15.32:0/388518996 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:48:39.352 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:39.349+0000 7f30dca00640 1 --2- 172.21.15.32:0/388518996 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f30d80cf510 0x7f30d80ccec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:48:39.352 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:39.349+0000 7f30d7e00640 1 --2- 172.21.15.32:0/388518996 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f30d80cffe0 0x7f30d80cd400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:48:39.352 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:39.349+0000 7f30dd400640 1 -- 172.21.15.32:0/388518996 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f30d80cffe0 msgr2=0x7f30d80cd400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:48:39.352 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:39.349+0000 7f30dd400640 1 --2- 172.21.15.32:0/388518996 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f30d80cffe0 0x7f30d80cd400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:39.352 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:39.349+0000 7f30dd400640 1 -- 172.21.15.32:0/388518996 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f30d80cf510 msgr2=0x7f30d80ccec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:48:39.353 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:39.349+0000 7f30dd400640 1 --2- 172.21.15.32:0/388518996 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f30d80cf510 0x7f30d80ccec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:39.353 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:39.349+0000 7f30dd400640 1 -- 172.21.15.32:0/388518996 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f30d80a1840 con 0x7f30d80d26e0 2024-08-03T20:48:39.353 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:39.349+0000 7f30dd400640 1 --2- 172.21.15.32:0/388518996 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f30d80d26e0 0x7f30d80c7c30 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f30d000be90 tx=0x7f30d000bf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:48:39.353 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:39.350+0000 7f30d4c00640 1 -- 172.21.15.32:0/388518996 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30d0019070 con 0x7f30d80d26e0 2024-08-03T20:48:39.353 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:39.350+0000 7f30d4c00640 1 -- 172.21.15.32:0/388518996 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f30d0014440 con 0x7f30d80d26e0 2024-08-03T20:48:39.353 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:39.350+0000 7f30dde00640 1 -- 172.21.15.32:0/388518996 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f30d80cb2e0 con 0x7f30d80d26e0 2024-08-03T20:48:39.353 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:39.350+0000 7f30d4c00640 1 -- 172.21.15.32:0/388518996 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30d0014890 con 0x7f30d80d26e0 2024-08-03T20:48:39.353 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:39.350+0000 7f30dde00640 1 -- 172.21.15.32:0/388518996 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f30d80cb8a0 con 0x7f30d80d26e0 2024-08-03T20:48:39.354 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:39.351+0000 7f30dde00640 1 -- 172.21.15.32:0/388518996 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f30d8161b80 con 0x7f30d80d26e0 2024-08-03T20:48:39.354 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:39.352+0000 7f30d4c00640 1 -- 172.21.15.32:0/388518996 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f30d001c070 con 0x7f30d80d26e0 2024-08-03T20:48:39.355 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:39.354+0000 7f30d4c00640 1 --2- 172.21.15.32:0/388518996 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f30b8077010 0x7f30b8079520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:48:39.355 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:39.354+0000 7f30d4c00640 1 -- 172.21.15.32:0/388518996 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(33..33 src has 1..33) ==== 5795+0+0 (secure 0 0 0) 0x7f30d0098a00 con 0x7f30d80d26e0 2024-08-03T20:48:39.355 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:39.354+0000 7f30dca00640 1 --2- 172.21.15.32:0/388518996 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f30b8077010 0x7f30b8079520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:48:39.356 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:39.355+0000 7f30dca00640 1 --2- 172.21.15.32:0/388518996 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f30b8077010 0x7f30b8079520 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f30cc033000 tx=0x7f30cc004730 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:48:39.362 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:39.358+0000 7f30d4c00640 1 -- 172.21.15.32:0/388518996 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7f30d0061540 con 0x7f30d80d26e0 2024-08-03T20:48:39.603 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:39.602+0000 7f30dde00640 1 -- 172.21.15.32:0/388518996 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f30d815fc90 con 0x7f30d80d26e0 2024-08-03T20:48:39.604 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:39.603+0000 7f30d4c00640 1 -- 172.21.15.32:0/388518996 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 20 v20) ==== 94+0+5220 (secure 0 0 0) 0x7f30d0065a90 con 0x7f30d80d26e0 2024-08-03T20:48:39.604 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-03T20:48:39.604 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":20,"btime":"2024-08-03T20:48:39:088995+0000","default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4643,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6836/4002526832","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6834","nonce":4002526832},{"type":"v1","addr":"172.21.15.118:6836","nonce":4002526832}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":4646,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6837/3036256187","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6835","nonce":3036256187},{"type":"v1","addr":"172.21.15.118:6837","nonce":3036256187}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":4652,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6834/2104418334","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6833","nonce":2104418334},{"type":"v1","addr":"172.21.15.32:6834","nonce":2104418334}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-08-03T20:48:26.652521+0000","modified":"2024-08-03T20:48:39.088972+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5070},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5070":{"gid":5070,"name":"a","rank":0,"incarnation":20,"state":"up:replay","state_seq":1,"addr":"172.21.15.32:6835/211300533","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":211300533},{"type":"v1","addr":"172.21.15.32:6835","nonce":211300533}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":2}]} 2024-08-03T20:48:39.605 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 20 2024-08-03T20:48:39.606 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:39.605+0000 7f30dde00640 1 -- 172.21.15.32:0/388518996 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f30b8077010 msgr2=0x7f30b8079520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:48:39.607 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:39.605+0000 7f30dde00640 1 --2- 172.21.15.32:0/388518996 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f30b8077010 0x7f30b8079520 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f30cc033000 tx=0x7f30cc004730 comp rx=0 tx=0).stop 2024-08-03T20:48:39.607 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:39.605+0000 7f30dde00640 1 -- 172.21.15.32:0/388518996 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f30d80d26e0 msgr2=0x7f30d80c7c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:48:39.607 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:39.605+0000 7f30dde00640 1 --2- 172.21.15.32:0/388518996 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f30d80d26e0 0x7f30d80c7c30 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f30d000be90 tx=0x7f30d000bf80 comp rx=0 tx=0).stop 2024-08-03T20:48:39.607 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:39.605+0000 7f30dde00640 1 -- 172.21.15.32:0/388518996 shutdown_connections 2024-08-03T20:48:39.607 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:39.605+0000 7f30dde00640 1 --2- 172.21.15.32:0/388518996 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f30b8077010 0x7f30b8079520 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:39.607 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:39.605+0000 7f30dde00640 1 --2- 172.21.15.32:0/388518996 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f30d80d26e0 0x7f30d80c7c30 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:39.607 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:39.605+0000 7f30dde00640 1 --2- 172.21.15.32:0/388518996 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f30d80cffe0 0x7f30d80cd400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:39.607 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:39.605+0000 7f30dde00640 1 --2- 172.21.15.32:0/388518996 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f30d80cf510 0x7f30d80ccec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:39.607 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:39.605+0000 7f30dde00640 1 -- 172.21.15.32:0/388518996 >> 172.21.15.32:0/388518996 conn(0x7f30d808d7a0 msgr2=0x7f30d80d0ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:48:39.607 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:39.605+0000 7f30dde00640 1 -- 172.21.15.32:0/388518996 shutdown_connections 2024-08-03T20:48:39.607 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:39.605+0000 7f30dde00640 1 -- 172.21.15.32:0/388518996 wait complete. 2024-08-03T20:48:40.034 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 20, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-08-03T20:48:26.652521+0000', 'modified': '2024-08-03T20:48:39.088972+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 33, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 5070}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_5070': {'gid': 5070, 'name': 'a', 'rank': 0, 'incarnation': 20, 'state': 'up:replay', 'state_seq': 1, 'addr': '172.21.15.32:6835/211300533', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6832', 'nonce': 211300533}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 211300533}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [5], 'metadata_pool': 4, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 0, 'qdb_cluster': []} 2024-08-03T20:48:40.034 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_5070:up:replay 2024-08-03T20:48:41.035 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-08-03T20:48:41.276 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:41.274+0000 7fead4400640 1 -- 172.21.15.32:0/143108040 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7feacc0b38b0 msgr2=0x7feacc1523f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:48:41.276 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:41.274+0000 7fead4400640 1 --2- 172.21.15.32:0/143108040 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7feacc0b38b0 0x7feacc1523f0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7feac0005fd0 tx=0x7feac002a5d0 comp rx=0 tx=0).stop 2024-08-03T20:48:41.276 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:41.274+0000 7fead4400640 1 -- 172.21.15.32:0/143108040 shutdown_connections 2024-08-03T20:48:41.276 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:41.274+0000 7fead4400640 1 --2- 172.21.15.32:0/143108040 >> v2:172.21.15.118:3300/0 conn(0x7feacc0b4ce0 0x7feacc15ef00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:41.276 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:41.274+0000 7fead4400640 1 --2- 172.21.15.32:0/143108040 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7feacc0b38b0 0x7feacc1523f0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:41.277 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:41.274+0000 7fead4400640 1 -- 172.21.15.32:0/143108040 >> 172.21.15.32:0/143108040 conn(0x7feacc0b1bd0 msgr2=0x7feacc0b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:48:41.277 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:41.274+0000 7fead4400640 1 -- 172.21.15.32:0/143108040 shutdown_connections 2024-08-03T20:48:41.277 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:41.274+0000 7fead4400640 1 -- 172.21.15.32:0/143108040 wait complete. 2024-08-03T20:48:41.278 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:41.277+0000 7fead4400640 1 Processor -- start 2024-08-03T20:48:41.278 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:41.277+0000 7fead4400640 1 -- start start 2024-08-03T20:48:41.280 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:41.278+0000 7fead4400640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7feacc0b38b0 0x7feacc158f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:48:41.280 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:41.278+0000 7fead4400640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7feacc0b4ce0 0x7feacc0cb5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:48:41.280 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:41.278+0000 7fead4400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7feacc15f440 0x7feacc15a370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:48:41.280 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:41.278+0000 7fead4400640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7feacc15c020 con 0x7feacc15f440 2024-08-03T20:48:41.280 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:41.278+0000 7fead4400640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7feacc0cf240 con 0x7feacc0b4ce0 2024-08-03T20:48:41.280 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:41.278+0000 7fead4400640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7feacc15c1a0 con 0x7feacc0b38b0 2024-08-03T20:48:41.280 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:41.279+0000 7fead1a00640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7feacc0b38b0 0x7feacc158f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:48:41.280 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:41.279+0000 7fead2400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7feacc15f440 0x7feacc15a370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:48:41.281 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:41.279+0000 7fead1a00640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7feacc0b38b0 0x7feacc158f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3301/0 says I am v2:172.21.15.32:45228/0 (socket says 172.21.15.32:45228) 2024-08-03T20:48:41.281 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:41.279+0000 7fead1a00640 1 -- 172.21.15.32:0/2350260210 learned_addr learned my addr 172.21.15.32:0/2350260210 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:48:41.281 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:41.279+0000 7fead1000640 1 --2- 172.21.15.32:0/2350260210 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7feacc0b4ce0 0x7feacc0cb5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:48:41.281 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:41.279+0000 7fead1a00640 1 -- 172.21.15.32:0/2350260210 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7feacc0b4ce0 msgr2=0x7feacc0cb5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:48:41.282 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:41.279+0000 7fead1a00640 1 --2- 172.21.15.32:0/2350260210 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7feacc0b4ce0 0x7feacc0cb5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:41.282 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:41.279+0000 7fead1a00640 1 -- 172.21.15.32:0/2350260210 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7feacc15f440 msgr2=0x7feacc15a370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:48:41.282 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:41.279+0000 7fead1a00640 1 --2- 172.21.15.32:0/2350260210 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7feacc15f440 0x7feacc15a370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:41.282 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:41.279+0000 7fead1a00640 1 -- 172.21.15.32:0/2350260210 --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feacc155910 con 0x7feacc0b38b0 2024-08-03T20:48:41.282 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:41.279+0000 7fead1a00640 1 --2- 172.21.15.32:0/2350260210 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7feacc0b38b0 0x7feacc158f70 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7feac0006010 tx=0x7feac00035f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:48:41.282 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:41.280+0000 7feac9600640 1 -- 172.21.15.32:0/2350260210 <== mon.2 v2:172.21.15.118:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feac003e030 con 0x7feacc0b38b0 2024-08-03T20:48:41.282 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:41.280+0000 7feac9600640 1 -- 172.21.15.32:0/2350260210 <== mon.2 v2:172.21.15.118:3301/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7feac0003ae0 con 0x7feacc0b38b0 2024-08-03T20:48:41.282 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:41.280+0000 7feac9600640 1 -- 172.21.15.32:0/2350260210 <== mon.2 v2:172.21.15.118:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feac0035040 con 0x7feacc0b38b0 2024-08-03T20:48:41.282 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:41.280+0000 7fead4400640 1 -- 172.21.15.32:0/2350260210 --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7feacc146680 con 0x7feacc0b38b0 2024-08-03T20:48:41.282 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:41.280+0000 7fead4400640 1 -- 172.21.15.32:0/2350260210 --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7feacc1481e0 con 0x7feacc0b38b0 2024-08-03T20:48:41.283 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:41.281+0000 7fead4400640 1 -- 172.21.15.32:0/2350260210 --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7feacc15c650 con 0x7feacc0b38b0 2024-08-03T20:48:41.283 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:41.282+0000 7feac9600640 1 -- 172.21.15.32:0/2350260210 <== mon.2 v2:172.21.15.118:3301/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7feac0041030 con 0x7feacc0b38b0 2024-08-03T20:48:41.284 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:41.283+0000 7feac9600640 1 --2- 172.21.15.32:0/2350260210 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7feaac077010 0x7feaac079520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:48:41.284 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:41.283+0000 7feac9600640 1 -- 172.21.15.32:0/2350260210 <== mon.2 v2:172.21.15.118:3301/0 5 ==== osd_map(33..33 src has 1..33) ==== 5795+0+0 (secure 0 0 0) 0x7feac00b8d80 con 0x7feacc0b38b0 2024-08-03T20:48:41.285 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:41.283+0000 7fead1000640 1 --2- 172.21.15.32:0/2350260210 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7feaac077010 0x7feaac079520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:48:41.286 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:41.284+0000 7fead1000640 1 --2- 172.21.15.32:0/2350260210 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7feaac077010 0x7feaac079520 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7feacc158f00 tx=0x7feab8005f90 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:48:41.290 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:41.289+0000 7feac9600640 1 -- 172.21.15.32:0/2350260210 <== mon.2 v2:172.21.15.118:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7feac00818c0 con 0x7feacc0b38b0 2024-08-03T20:48:41.540 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:41.538+0000 7fead4400640 1 -- 172.21.15.32:0/2350260210 --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7feacc155c70 con 0x7feacc0b38b0 2024-08-03T20:48:41.541 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:41.539+0000 7feac9600640 1 -- 172.21.15.32:0/2350260210 <== mon.2 v2:172.21.15.118:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+5220 (secure 0 0 0) 0x7feac008a150 con 0x7feacc0b38b0 2024-08-03T20:48:41.541 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-03T20:48:41.541 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":22,"btime":"2024-08-03T20:48:41:093330+0000","default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4643,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6836/4002526832","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6834","nonce":4002526832},{"type":"v1","addr":"172.21.15.118:6836","nonce":4002526832}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":4646,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6837/3036256187","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6835","nonce":3036256187},{"type":"v1","addr":"172.21.15.118:6837","nonce":3036256187}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":4652,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6834/2104418334","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6833","nonce":2104418334},{"type":"v1","addr":"172.21.15.32:6834","nonce":2104418334}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-08-03T20:48:26.652521+0000","modified":"2024-08-03T20:48:40.095359+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5070},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5070":{"gid":5070,"name":"a","rank":0,"incarnation":20,"state":"up:rejoin","state_seq":3,"addr":"172.21.15.32:6835/211300533","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":211300533},{"type":"v1","addr":"172.21.15.32:6835","nonce":211300533}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":2}]} 2024-08-03T20:48:41.542 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 22 2024-08-03T20:48:41.544 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:41.542+0000 7fead4400640 1 -- 172.21.15.32:0/2350260210 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7feaac077010 msgr2=0x7feaac079520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:48:41.544 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:41.542+0000 7fead4400640 1 --2- 172.21.15.32:0/2350260210 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7feaac077010 0x7feaac079520 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7feacc158f00 tx=0x7feab8005f90 comp rx=0 tx=0).stop 2024-08-03T20:48:41.544 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:41.542+0000 7fead4400640 1 -- 172.21.15.32:0/2350260210 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7feacc0b38b0 msgr2=0x7feacc158f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:48:41.544 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:41.542+0000 7fead4400640 1 --2- 172.21.15.32:0/2350260210 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7feacc0b38b0 0x7feacc158f70 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7feac0006010 tx=0x7feac00035f0 comp rx=0 tx=0).stop 2024-08-03T20:48:41.544 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:41.542+0000 7fead4400640 1 -- 172.21.15.32:0/2350260210 shutdown_connections 2024-08-03T20:48:41.544 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:41.542+0000 7fead4400640 1 --2- 172.21.15.32:0/2350260210 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7feaac077010 0x7feaac079520 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:41.545 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:41.542+0000 7fead4400640 1 --2- 172.21.15.32:0/2350260210 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7feacc0b4ce0 0x7feacc0cb5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:41.545 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:41.542+0000 7fead4400640 1 --2- 172.21.15.32:0/2350260210 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7feacc15f440 0x7feacc15a370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:41.545 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:41.542+0000 7fead4400640 1 --2- 172.21.15.32:0/2350260210 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7feacc0b38b0 0x7feacc158f70 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:41.545 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:41.542+0000 7fead4400640 1 -- 172.21.15.32:0/2350260210 >> 172.21.15.32:0/2350260210 conn(0x7feacc0b1bd0 msgr2=0x7feacc15b910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:48:41.545 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:41.542+0000 7fead4400640 1 -- 172.21.15.32:0/2350260210 shutdown_connections 2024-08-03T20:48:41.545 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:41.542+0000 7fead4400640 1 -- 172.21.15.32:0/2350260210 wait complete. 2024-08-03T20:48:41.969 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-08-03T20:48:26.652521+0000', 'modified': '2024-08-03T20:48:40.095359+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 33, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 5070}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_5070': {'gid': 5070, 'name': 'a', 'rank': 0, 'incarnation': 20, 'state': 'up:rejoin', 'state_seq': 3, 'addr': '172.21.15.32:6835/211300533', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6832', 'nonce': 211300533}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 211300533}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [5], 'metadata_pool': 4, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 0, 'qdb_cluster': []} 2024-08-03T20:48:41.969 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_5070:up:rejoin 2024-08-03T20:48:42.970 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-08-03T20:48:43.214 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:43.212+0000 7fbc28400640 1 -- 172.21.15.32:0/21713430 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbc20073e60 msgr2=0x7fbc20074270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:48:43.214 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:43.212+0000 7fbc28400640 1 --2- 172.21.15.32:0/21713430 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbc20073e60 0x7fbc20074270 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7fbc0c005fd0 tx=0x7fbc0c02a540 comp rx=0 tx=0).stop 2024-08-03T20:48:43.214 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:43.212+0000 7fbc28400640 1 -- 172.21.15.32:0/21713430 shutdown_connections 2024-08-03T20:48:43.214 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:43.212+0000 7fbc28400640 1 --2- 172.21.15.32:0/21713430 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbc20073e60 0x7fbc20074270 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:43.214 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:43.212+0000 7fbc28400640 1 --2- 172.21.15.32:0/21713430 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fbc2006e930 0x7fbc20144f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:43.214 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:43.212+0000 7fbc28400640 1 -- 172.21.15.32:0/21713430 >> 172.21.15.32:0/21713430 conn(0x7fbc200b1bd0 msgr2=0x7fbc200b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:48:43.215 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:43.212+0000 7fbc28400640 1 -- 172.21.15.32:0/21713430 shutdown_connections 2024-08-03T20:48:43.215 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:43.212+0000 7fbc28400640 1 -- 172.21.15.32:0/21713430 wait complete. 2024-08-03T20:48:43.216 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:43.214+0000 7fbc28400640 1 Processor -- start 2024-08-03T20:48:43.216 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:43.214+0000 7fbc28400640 1 -- start start 2024-08-03T20:48:43.218 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:43.216+0000 7fbc28400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbc2006e930 0x7fbc201e84f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:48:43.218 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:43.216+0000 7fbc28400640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fbc20073e60 0x7fbc2015b960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:48:43.218 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:43.216+0000 7fbc28400640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fbc200747b0 0x7fbc2015d1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:48:43.218 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:43.216+0000 7fbc28400640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fbc200cf9c0 con 0x7fbc2006e930 2024-08-03T20:48:43.218 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:43.216+0000 7fbc28400640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7fbc200cf840 con 0x7fbc20073e60 2024-08-03T20:48:43.218 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:43.216+0000 7fbc28400640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7fbc200ccff0 con 0x7fbc200747b0 2024-08-03T20:48:43.218 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:43.216+0000 7fbc25a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbc2006e930 0x7fbc201e84f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:48:43.218 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:43.216+0000 7fbc25a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbc2006e930 0x7fbc201e84f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:44984/0 (socket says 172.21.15.32:44984) 2024-08-03T20:48:43.218 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:43.216+0000 7fbc25a00640 1 -- 172.21.15.32:0/689207787 learned_addr learned my addr 172.21.15.32:0/689207787 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:48:43.219 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:43.217+0000 7fbc25a00640 1 -- 172.21.15.32:0/689207787 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fbc200747b0 msgr2=0x7fbc2015d1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:48:43.220 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:43.217+0000 7fbc25a00640 1 --2- 172.21.15.32:0/689207787 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fbc200747b0 0x7fbc2015d1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:43.220 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:43.217+0000 7fbc25000640 1 --2- 172.21.15.32:0/689207787 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fbc20073e60 0x7fbc2015b960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:48:43.220 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:43.217+0000 7fbc25a00640 1 -- 172.21.15.32:0/689207787 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fbc20073e60 msgr2=0x7fbc2015b960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:48:43.220 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:43.217+0000 7fbc25a00640 1 --2- 172.21.15.32:0/689207787 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fbc20073e60 0x7fbc2015b960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:43.220 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:43.217+0000 7fbc25a00640 1 -- 172.21.15.32:0/689207787 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbc201e7560 con 0x7fbc2006e930 2024-08-03T20:48:43.220 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:43.217+0000 7fbc25000640 1 --2- 172.21.15.32:0/689207787 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fbc20073e60 0x7fbc2015b960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-03T20:48:43.220 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:43.217+0000 7fbc25a00640 1 --2- 172.21.15.32:0/689207787 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbc2006e930 0x7fbc201e84f0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7fbc1400d700 tx=0x7fbc1400dbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:48:43.220 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:43.217+0000 7fbc1d600640 1 -- 172.21.15.32:0/689207787 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc14017070 con 0x7fbc2006e930 2024-08-03T20:48:43.220 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:43.217+0000 7fbc1d600640 1 -- 172.21.15.32:0/689207787 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fbc14005310 con 0x7fbc2006e930 2024-08-03T20:48:43.220 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:43.217+0000 7fbc1d600640 1 -- 172.21.15.32:0/689207787 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc14009820 con 0x7fbc2006e930 2024-08-03T20:48:43.220 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:43.217+0000 7fbc28400640 1 -- 172.21.15.32:0/689207787 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbc201e7800 con 0x7fbc2006e930 2024-08-03T20:48:43.220 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:43.217+0000 7fbc28400640 1 -- 172.21.15.32:0/689207787 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbc2015cc80 con 0x7fbc2006e930 2024-08-03T20:48:43.221 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:43.219+0000 7fbc28400640 1 -- 172.21.15.32:0/689207787 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbc2015ca70 con 0x7fbc2006e930 2024-08-03T20:48:43.221 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:43.220+0000 7fbc1d600640 1 -- 172.21.15.32:0/689207787 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7fbc140099c0 con 0x7fbc2006e930 2024-08-03T20:48:43.222 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:43.221+0000 7fbc1d600640 1 --2- 172.21.15.32:0/689207787 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fbc00077040 0x7fbc00079550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:48:43.222 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:43.221+0000 7fbc1d600640 1 -- 172.21.15.32:0/689207787 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(33..33 src has 1..33) ==== 5795+0+0 (secure 0 0 0) 0x7fbc14098990 con 0x7fbc2006e930 2024-08-03T20:48:43.223 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:43.221+0000 7fbc25000640 1 --2- 172.21.15.32:0/689207787 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fbc00077040 0x7fbc00079550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:48:43.223 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:43.222+0000 7fbc25000640 1 --2- 172.21.15.32:0/689207787 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fbc00077040 0x7fbc00079550 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fbc0c033000 tx=0x7fbc0c004730 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:48:43.226 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:43.225+0000 7fbc1d600640 1 -- 172.21.15.32:0/689207787 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7fbc14061b30 con 0x7fbc2006e930 2024-08-03T20:48:43.472 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:43.470+0000 7fbc28400640 1 -- 172.21.15.32:0/689207787 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fbc200cd510 con 0x7fbc2006e930 2024-08-03T20:48:43.472 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:43.471+0000 7fbc1d600640 1 -- 172.21.15.32:0/689207787 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) ==== 94+0+5227 (secure 0 0 0) 0x7fbc14061430 con 0x7fbc2006e930 2024-08-03T20:48:43.472 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-03T20:48:43.473 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":23,"btime":"2024-08-03T20:48:42:098849+0000","default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4643,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6836/4002526832","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6834","nonce":4002526832},{"type":"v1","addr":"172.21.15.118:6836","nonce":4002526832}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":4646,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6837/3036256187","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6835","nonce":3036256187},{"type":"v1","addr":"172.21.15.118:6837","nonce":3036256187}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":4652,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6834/2104418334","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6833","nonce":2104418334},{"type":"v1","addr":"172.21.15.32:6834","nonce":2104418334}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":23,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-08-03T20:48:26.652521+0000","modified":"2024-08-03T20:48:42.098834+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5070},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5070":{"gid":5070,"name":"a","rank":0,"incarnation":20,"state":"up:active","state_seq":4,"addr":"172.21.15.32:6835/211300533","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":211300533},{"type":"v1","addr":"172.21.15.32:6835","nonce":211300533}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5070,"qdb_cluster":[5070]},"id":2}]} 2024-08-03T20:48:43.473 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 23 2024-08-03T20:48:43.475 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:43.473+0000 7fbc28400640 1 -- 172.21.15.32:0/689207787 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fbc00077040 msgr2=0x7fbc00079550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:48:43.475 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:43.473+0000 7fbc28400640 1 --2- 172.21.15.32:0/689207787 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fbc00077040 0x7fbc00079550 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fbc0c033000 tx=0x7fbc0c004730 comp rx=0 tx=0).stop 2024-08-03T20:48:43.475 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:43.473+0000 7fbc28400640 1 -- 172.21.15.32:0/689207787 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbc2006e930 msgr2=0x7fbc201e84f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:48:43.475 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:43.473+0000 7fbc28400640 1 --2- 172.21.15.32:0/689207787 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbc2006e930 0x7fbc201e84f0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7fbc1400d700 tx=0x7fbc1400dbc0 comp rx=0 tx=0).stop 2024-08-03T20:48:43.475 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:43.473+0000 7fbc28400640 1 -- 172.21.15.32:0/689207787 shutdown_connections 2024-08-03T20:48:43.475 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:43.473+0000 7fbc28400640 1 --2- 172.21.15.32:0/689207787 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fbc00077040 0x7fbc00079550 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:43.475 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:43.473+0000 7fbc28400640 1 --2- 172.21.15.32:0/689207787 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fbc20073e60 0x7fbc2015b960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:43.475 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:43.473+0000 7fbc28400640 1 --2- 172.21.15.32:0/689207787 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fbc200747b0 0x7fbc2015d1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:43.475 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:43.473+0000 7fbc28400640 1 --2- 172.21.15.32:0/689207787 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbc2006e930 0x7fbc201e84f0 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:43.475 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:43.473+0000 7fbc28400640 1 -- 172.21.15.32:0/689207787 >> 172.21.15.32:0/689207787 conn(0x7fbc200b1bd0 msgr2=0x7fbc200b04b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:48:43.476 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:43.473+0000 7fbc28400640 1 -- 172.21.15.32:0/689207787 shutdown_connections 2024-08-03T20:48:43.476 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:43.473+0000 7fbc28400640 1 -- 172.21.15.32:0/689207787 wait complete. 2024-08-03T20:48:43.900 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 23, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-08-03T20:48:26.652521+0000', 'modified': '2024-08-03T20:48:42.098834+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 33, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 5070}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_5070': {'gid': 5070, 'name': 'a', 'rank': 0, 'incarnation': 20, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.32:6835/211300533', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6832', 'nonce': 211300533}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 211300533}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [5], 'metadata_pool': 4, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 5070, 'qdb_cluster': [5070]} 2024-08-03T20:48:43.900 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-08-03T20:48:43.900 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2024-08-03T20:48:44.098 INFO:tasks.ceph.mds.a.smithi032.stderr:2024-08-03T20:48:44.097+0000 7fb2f6a00640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-08-03T20:48:44.131 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:44.129+0000 7f953ca00640 1 -- 172.21.15.32:0/3754597322 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f953806e930 msgr2=0x7f9538144f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:48:44.131 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:44.129+0000 7f953ca00640 1 --2- 172.21.15.32:0/3754597322 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f953806e930 0x7f9538144f90 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f9528009860 tx=0x7f952802a530 comp rx=0 tx=0).stop 2024-08-03T20:48:44.131 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:44.129+0000 7f953ca00640 1 -- 172.21.15.32:0/3754597322 shutdown_connections 2024-08-03T20:48:44.131 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:44.129+0000 7f953ca00640 1 --2- 172.21.15.32:0/3754597322 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f953806e930 0x7f9538144f90 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:44.131 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:44.129+0000 7f953ca00640 1 -- 172.21.15.32:0/3754597322 >> 172.21.15.32:0/3754597322 conn(0x7f95380b1bd0 msgr2=0x7f95380b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:48:44.131 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:44.130+0000 7f953ca00640 1 -- 172.21.15.32:0/3754597322 shutdown_connections 2024-08-03T20:48:44.132 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:44.130+0000 7f953ca00640 1 -- 172.21.15.32:0/3754597322 wait complete. 2024-08-03T20:48:44.133 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:44.132+0000 7f953ca00640 1 Processor -- start 2024-08-03T20:48:44.133 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:44.132+0000 7f953ca00640 1 -- start start 2024-08-03T20:48:44.136 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:44.133+0000 7f953ca00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f953806e930 0x7f953815b610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:48:44.136 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:44.133+0000 7f953ca00640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f9538073e60 0x7f953815eef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:48:44.136 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:44.133+0000 7f953ca00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f953815fff0 0x7f953815f430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:48:44.136 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:44.133+0000 7f953ca00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f95380cf840 con 0x7f953806e930 2024-08-03T20:48:44.136 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:44.133+0000 7f953ca00640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f9538145f30 con 0x7f953815fff0 2024-08-03T20:48:44.136 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:44.133+0000 7f953ca00640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7f95380cf9c0 con 0x7f9538073e60 2024-08-03T20:48:44.136 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:44.134+0000 7f9535e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f953806e930 0x7f953815b610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:48:44.136 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:44.134+0000 7f9535e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f953806e930 0x7f953815b610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:45000/0 (socket says 172.21.15.32:45000) 2024-08-03T20:48:44.136 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:44.134+0000 7f9535e00640 1 -- 172.21.15.32:0/2437726910 learned_addr learned my addr 172.21.15.32:0/2437726910 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:48:44.136 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:44.134+0000 7f9535400640 1 --2- 172.21.15.32:0/2437726910 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f9538073e60 0x7f953815eef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:48:44.136 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:44.134+0000 7f9536800640 1 --2- 172.21.15.32:0/2437726910 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f953815fff0 0x7f953815f430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:48:44.136 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:44.134+0000 7f9535e00640 1 -- 172.21.15.32:0/2437726910 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f9538073e60 msgr2=0x7f953815eef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:48:44.137 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:44.134+0000 7f9535e00640 1 --2- 172.21.15.32:0/2437726910 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f9538073e60 0x7f953815eef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:44.137 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:44.134+0000 7f9535e00640 1 -- 172.21.15.32:0/2437726910 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f953815fff0 msgr2=0x7f953815f430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:48:44.137 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:44.134+0000 7f9535e00640 1 --2- 172.21.15.32:0/2437726910 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f953815fff0 0x7f953815f430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:44.137 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:44.134+0000 7f9535e00640 1 -- 172.21.15.32:0/2437726910 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f95381f5ce0 con 0x7f953806e930 2024-08-03T20:48:44.137 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:44.134+0000 7f9535400640 1 --2- 172.21.15.32:0/2437726910 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f9538073e60 0x7f953815eef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-03T20:48:44.137 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:44.134+0000 7f9535e00640 1 --2- 172.21.15.32:0/2437726910 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f953806e930 0x7f953815b610 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f9528009980 tx=0x7f9528003c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:48:44.138 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:44.135+0000 7f952e000640 1 -- 172.21.15.32:0/2437726910 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9528005c90 con 0x7f953806e930 2024-08-03T20:48:44.138 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:44.135+0000 7f952e000640 1 -- 172.21.15.32:0/2437726910 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f9528005020 con 0x7f953806e930 2024-08-03T20:48:44.138 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:44.135+0000 7f953ca00640 1 -- 172.21.15.32:0/2437726910 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f95381f5fc0 con 0x7f953806e930 2024-08-03T20:48:44.138 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:44.135+0000 7f952e000640 1 -- 172.21.15.32:0/2437726910 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9528005410 con 0x7f953806e930 2024-08-03T20:48:44.138 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:44.135+0000 7f953ca00640 1 -- 172.21.15.32:0/2437726910 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f953815d930 con 0x7f953806e930 2024-08-03T20:48:44.138 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:44.137+0000 7f952e000640 1 -- 172.21.15.32:0/2437726910 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f95280317c0 con 0x7f953806e930 2024-08-03T20:48:44.140 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:44.138+0000 7f952e000640 1 --2- 172.21.15.32:0/2437726910 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f9518077010 0x7f9518079520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:48:44.140 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:44.138+0000 7f952e000640 1 -- 172.21.15.32:0/2437726910 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(33..33 src has 1..33) ==== 5795+0+0 (secure 0 0 0) 0x7f95280b9930 con 0x7f953806e930 2024-08-03T20:48:44.140 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:44.138+0000 7f9543a33740 1 Processor -- start 2024-08-03T20:48:44.140 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:44.139+0000 7f9535400640 1 --2- 172.21.15.32:0/2437726910 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f9518077010 0x7f9518079520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:48:44.140 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:44.139+0000 7f9543a33740 1 -- start start 2024-08-03T20:48:44.141 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:44.139+0000 7f9535400640 1 --2- 172.21.15.32:0/2437726910 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f9518077010 0x7f9518079520 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f9524007fd0 tx=0x7f9524009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:48:44.142 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:44.140+0000 7f9543a33740 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x56366b327270 0x56366b711e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:48:44.143 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:44.140+0000 7f9543a33740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56366b2fe3b0 0x56366b6e6ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:48:44.143 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:44.140+0000 7f9543a33740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56366b306590 0x56366b6e7220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:48:44.143 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:44.140+0000 7f9543a33740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x56366b63fa60 con 0x56366b306590 2024-08-03T20:48:44.143 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:44.140+0000 7f9543a33740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x56366b62ef60 con 0x56366b2fe3b0 2024-08-03T20:48:44.143 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:44.140+0000 7f9543a33740 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x56366b632fa0 con 0x56366b327270 2024-08-03T20:48:44.143 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:44.140+0000 7f9536800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56366b306590 0x56366b6e7220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:48:44.143 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:44.140+0000 7f9536800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56366b306590 0x56366b6e7220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:45008/0 (socket says 172.21.15.32:45008) 2024-08-03T20:48:44.143 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:44.140+0000 7f9536800640 1 -- 172.21.15.32:0/191011648 learned_addr learned my addr 172.21.15.32:0/191011648 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:48:44.143 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:44.140+0000 7f9535400640 1 --2- 172.21.15.32:0/191011648 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56366b2fe3b0 0x56366b6e6ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:48:44.143 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:44.141+0000 7f9535e00640 1 --2- 172.21.15.32:0/191011648 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x56366b327270 0x56366b711e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:48:44.143 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:44.141+0000 7f9536800640 1 -- 172.21.15.32:0/191011648 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x56366b327270 msgr2=0x56366b711e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:48:44.143 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:44.141+0000 7f9536800640 1 --2- 172.21.15.32:0/191011648 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x56366b327270 0x56366b711e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:44.144 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:44.141+0000 7f9536800640 1 -- 172.21.15.32:0/191011648 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56366b2fe3b0 msgr2=0x56366b6e6ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:48:44.144 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:44.141+0000 7f9536800640 1 --2- 172.21.15.32:0/191011648 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56366b2fe3b0 0x56366b6e6ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:44.144 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:44.141+0000 7f9536800640 1 -- 172.21.15.32:0/191011648 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56366b392850 con 0x56366b306590 2024-08-03T20:48:44.144 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:44.141+0000 7f9535400640 1 --2- 172.21.15.32:0/191011648 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56366b2fe3b0 0x56366b6e6ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-03T20:48:44.144 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:44.141+0000 7f9535e00640 1 --2- 172.21.15.32:0/191011648 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x56366b327270 0x56366b711e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-03T20:48:44.144 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:44.141+0000 7f9536800640 1 --2- 172.21.15.32:0/191011648 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56366b306590 0x56366b6e7220 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f95300020a0 tx=0x7f953000e530 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2d456a5f9b8428a server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:48:44.144 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:44.141+0000 7f9522600640 1 -- 172.21.15.32:0/191011648 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9530014480 con 0x56366b306590 2024-08-03T20:48:44.144 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:44.141+0000 7f9522600640 1 -- 172.21.15.32:0/191011648 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f9530019070 con 0x56366b306590 2024-08-03T20:48:44.144 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:44.141+0000 7f9522600640 1 -- 172.21.15.32:0/191011648 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95300149d0 con 0x56366b306590 2024-08-03T20:48:44.145 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:44.142+0000 7f9543a33740 1 -- 172.21.15.32:0/191011648 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56366b306590 msgr2=0x56366b6e7220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:48:44.145 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:44.142+0000 7f9543a33740 1 --2- 172.21.15.32:0/191011648 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56366b306590 0x56366b6e7220 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f95300020a0 tx=0x7f953000e530 comp rx=0 tx=0).stop 2024-08-03T20:48:44.145 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:44.142+0000 7f9543a33740 1 -- 172.21.15.32:0/191011648 shutdown_connections 2024-08-03T20:48:44.145 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:44.142+0000 7f9543a33740 1 --2- 172.21.15.32:0/191011648 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56366b2fe3b0 0x56366b6e6ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:44.145 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:44.142+0000 7f9543a33740 1 --2- 172.21.15.32:0/191011648 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56366b306590 0x56366b6e7220 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:44.146 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:44.142+0000 7f9543a33740 1 --2- 172.21.15.32:0/191011648 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x56366b327270 0x56366b711e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:44.146 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:44.142+0000 7f9543a33740 1 -- 172.21.15.32:0/191011648 >> 172.21.15.32:0/191011648 conn(0x56366b31b3c0 msgr2=0x56366b6cd180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:48:44.146 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:44.142+0000 7f9543a33740 1 -- 172.21.15.32:0/191011648 shutdown_connections 2024-08-03T20:48:44.146 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:44.142+0000 7f9543a33740 1 -- 172.21.15.32:0/191011648 wait complete. 2024-08-03T20:48:44.146 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:44.143+0000 7f9543a33740 1 -- start start 2024-08-03T20:48:44.146 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:44.143+0000 7f9543a33740 1 Processor -- start 2024-08-03T20:48:44.149 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:44.145+0000 7f9543a33740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56366b2fe3b0 0x56366b725150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:48:44.149 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:44.145+0000 7f9543a33740 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x56366b306590 0x56366b7078f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:48:44.149 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:44.145+0000 7f9543a33740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56366b327270 0x56366b6f4aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:48:44.149 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:44.145+0000 7f9543a33740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x56366b3a1620 con 0x56366b2fe3b0 2024-08-03T20:48:44.149 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:44.145+0000 7f9543a33740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x56366b5cc160 con 0x56366b327270 2024-08-03T20:48:44.149 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:44.145+0000 7f9543a33740 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x56366b70b480 con 0x56366b306590 2024-08-03T20:48:44.149 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:44.145+0000 7f9543a33740 10 client.0 _ll_register_callbacks cb 0x56366b5db6b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-03T20:48:44.149 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:44.145+0000 7f9535e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56366b2fe3b0 0x56366b725150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:48:44.149 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:44.145+0000 7f9535e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56366b2fe3b0 0x56366b725150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:45018/0 (socket says 172.21.15.32:45018) 2024-08-03T20:48:44.150 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:44.145+0000 7f9535e00640 1 -- 172.21.15.32:0/159856437 learned_addr learned my addr 172.21.15.32:0/159856437 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:48:44.150 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:44.145+0000 7f9536800640 1 --2- 172.21.15.32:0/159856437 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56366b327270 0x56366b6f4aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:48:44.150 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:44.145+0000 7f9535400640 1 --2- 172.21.15.32:0/159856437 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x56366b306590 0x56366b7078f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:48:44.150 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:44.146+0000 7f9535e00640 1 -- 172.21.15.32:0/159856437 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x56366b306590 msgr2=0x56366b7078f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:48:44.150 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:44.146+0000 7f9535e00640 1 --2- 172.21.15.32:0/159856437 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x56366b306590 0x56366b7078f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:44.150 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:44.146+0000 7f9535e00640 1 -- 172.21.15.32:0/159856437 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56366b327270 msgr2=0x56366b6f4aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:48:44.150 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:44.146+0000 7f9535e00640 1 --2- 172.21.15.32:0/159856437 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56366b327270 0x56366b6f4aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:44.150 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:44.146+0000 7f9535e00640 1 -- 172.21.15.32:0/159856437 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56366b4096b0 con 0x56366b2fe3b0 2024-08-03T20:48:44.150 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:44.146+0000 7f9535400640 1 --2- 172.21.15.32:0/159856437 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x56366b306590 0x56366b7078f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-03T20:48:44.150 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:44.146+0000 7f9535e00640 1 --2- 172.21.15.32:0/159856437 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56366b2fe3b0 0x56366b725150 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f952807e4e0 tx=0x7f9528031400 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5084d05a3d03a134 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:48:44.150 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:44.146+0000 7f9516a00640 10 client.0 ms_handle_connect on v2:172.21.15.32:3300/0 2024-08-03T20:48:44.150 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:44.146+0000 7f9516a00640 1 -- 172.21.15.32:0/159856437 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95280b5060 con 0x56366b2fe3b0 2024-08-03T20:48:44.151 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:44.146+0000 7f9516a00640 1 -- 172.21.15.32:0/159856437 <== mon.0 v2:172.21.15.32:3300/0 2 ==== osd_map(33..33 src has 1..33) ==== 5795+0+0 (secure 0 0 0) 0x7f952808aff0 con 0x56366b2fe3b0 2024-08-03T20:48:44.151 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:44.146+0000 7f9543a33740 1 -- 172.21.15.32:0/159856437 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9530019070 con 0x56366b2fe3b0 2024-08-03T20:48:44.151 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:44.146+0000 7f9516a00640 1 -- 172.21.15.32:0/159856437 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f952809b070 con 0x56366b2fe3b0 2024-08-03T20:48:44.151 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:44.146+0000 7f9543a33740 10 client.5121 fetch_fsmap learned FSMap version 23 2024-08-03T20:48:44.151 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:44.146+0000 7f9543a33740 1 -- 172.21.15.32:0/159856437 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=2+}) -- 0x56366b707e30 con 0x56366b2fe3b0 2024-08-03T20:48:44.151 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:44.147+0000 7f9516a00640 1 -- 172.21.15.32:0/159856437 <== mon.0 v2:172.21.15.32:3300/0 4 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f9530019070 con 0x56366b2fe3b0 2024-08-03T20:48:44.151 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:44.147+0000 7f9516a00640 1 -- 172.21.15.32:0/159856437 <== mon.0 v2:172.21.15.32:3300/0 5 ==== fsmap(e 23) ==== 3502+0+0 (secure 0 0 0) 0x7f95280816c0 con 0x56366b2fe3b0 2024-08-03T20:48:44.151 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:44.147+0000 7f9543a33740 10 client.5121 fetch_fsmap finished waiting for FSMap version 23 2024-08-03T20:48:44.151 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:44.147+0000 7f9543a33740 10 client.5121 resolve_mds: resolved name 'a' to daemon mds.a 2024-08-03T20:48:44.151 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:44.147+0000 7f9543a33740 20 client.5121 populate_metadata read hostname 'smithi032' 2024-08-03T20:48:44.151 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:44.147+0000 7f9543a33740 1 --2- 172.21.15.32:0/159856437 >> [v2:172.21.15.32:6832/211300533,v1:172.21.15.32:6835/211300533] conn(0x56366b2e5e40 0x56366b721280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:48:44.151 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:44.147+0000 7f9543a33740 4 client.5121 mds_command: new command op to 5070 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-08-03T20:48:44.151 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:44.147+0000 7f9543a33740 1 -- 172.21.15.32:0/159856437 --> [v2:172.21.15.32:6832/211300533,v1:172.21.15.32:6835/211300533] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56366b6419b0 con 0x56366b2e5e40 2024-08-03T20:48:44.151 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:44.147+0000 7f9535400640 1 --2- 172.21.15.32:0/159856437 >> [v2:172.21.15.32:6832/211300533,v1:172.21.15.32:6835/211300533] conn(0x56366b2e5e40 0x56366b721280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:48:44.153 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:44.148+0000 7f9535400640 1 --2- 172.21.15.32:0/159856437 >> [v2:172.21.15.32:6832/211300533,v1:172.21.15.32:6835/211300533] conn(0x56366b2e5e40 0x56366b721280 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=765f74589022acbf server_cookie=94834c489dff8a56 in_seq=0 out_seq=0 2024-08-03T20:48:44.153 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:44.148+0000 7f9516a00640 10 client.5121 ms_handle_connect on v2:172.21.15.32:6832/211300533 2024-08-03T20:48:44.153 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:44.149+0000 7f9516a00640 1 -- 172.21.15.32:0/159856437 <== mds.0 v2:172.21.15.32:6832/211300533 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x56366b6419b0 con 0x56366b2e5e40 2024-08-03T20:48:44.153 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:44.149+0000 7f9516a00640 10 client.5121 handle_command_reply: tid=0 2024-08-03T20:48:44.153 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:44.149+0000 7f9543a33740 1 client.5121 shutdown 2024-08-03T20:48:44.153 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:44.149+0000 7f9543a33740 1 -- 172.21.15.32:0/159856437 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56366b2fe3b0 msgr2=0x56366b725150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:48:44.153 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:44.149+0000 7f9543a33740 1 --2- 172.21.15.32:0/159856437 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56366b2fe3b0 0x56366b725150 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f952807e4e0 tx=0x7f9528031400 comp rx=0 tx=0).stop 2024-08-03T20:48:44.153 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:44.149+0000 7f9543a33740 1 -- 172.21.15.32:0/159856437 shutdown_connections 2024-08-03T20:48:44.153 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:44.149+0000 7f9543a33740 1 --2- 172.21.15.32:0/159856437 >> [v2:172.21.15.32:6832/211300533,v1:172.21.15.32:6835/211300533] conn(0x56366b2e5e40 0x56366b721280 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:44.153 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:44.149+0000 7f9543a33740 1 --2- 172.21.15.32:0/159856437 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56366b327270 0x56366b6f4aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:44.154 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:44.149+0000 7f9543a33740 1 --2- 172.21.15.32:0/159856437 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x56366b306590 0x56366b7078f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:44.154 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:44.149+0000 7f9543a33740 1 --2- 172.21.15.32:0/159856437 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56366b2fe3b0 0x56366b725150 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:44.154 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:44.149+0000 7f9543a33740 1 -- 172.21.15.32:0/159856437 >> 172.21.15.32:0/159856437 conn(0x56366b31b3c0 msgr2=0x56366b6c7ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:48:44.154 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:44.149+0000 7f9516a00640 0 client.5121 ms_handle_reset on v2:172.21.15.32:6832/211300533 2024-08-03T20:48:44.154 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:44.149+0000 7f9543a33740 1 -- 172.21.15.32:0/159856437 shutdown_connections 2024-08-03T20:48:44.154 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:44.149+0000 7f9543a33740 1 -- 172.21.15.32:0/159856437 wait complete. 2024-08-03T20:48:44.154 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:44.150+0000 7f9543a33740 20 client.5121 trim_cache size 0 max 16384 2024-08-03T20:48:44.171 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:44.170+0000 7f9543a33740 1 Processor -- start 2024-08-03T20:48:44.171 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:44.170+0000 7f9543a33740 1 -- start start 2024-08-03T20:48:44.174 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:44.171+0000 7f9543a33740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56366b31b3c0 0x56366b6c6740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:48:44.174 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:44.171+0000 7f9543a33740 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x56366b2fe3b0 0x56366b721280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:48:44.174 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:44.171+0000 7f9543a33740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56366b306590 0x56366b7217c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:48:44.174 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:44.171+0000 7f9543a33740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f9538201070 con 0x56366b31b3c0 2024-08-03T20:48:44.174 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:44.171+0000 7f9543a33740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f95380b4000 con 0x56366b306590 2024-08-03T20:48:44.174 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:44.171+0000 7f9543a33740 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7f9538200b40 con 0x56366b2fe3b0 2024-08-03T20:48:44.174 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:44.171+0000 7f9535e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56366b31b3c0 0x56366b6c6740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:48:44.174 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:44.172+0000 7f9535e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56366b31b3c0 0x56366b6c6740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:45034/0 (socket says 172.21.15.32:45034) 2024-08-03T20:48:44.174 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:44.172+0000 7f9535e00640 1 -- 172.21.15.32:0/3879017805 learned_addr learned my addr 172.21.15.32:0/3879017805 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:48:44.175 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:44.172+0000 7f9536800640 1 --2- 172.21.15.32:0/3879017805 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56366b306590 0x56366b7217c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:48:44.175 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:44.172+0000 7f9535400640 1 --2- 172.21.15.32:0/3879017805 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x56366b2fe3b0 0x56366b721280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:48:44.175 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:44.172+0000 7f9535e00640 1 -- 172.21.15.32:0/3879017805 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x56366b2fe3b0 msgr2=0x56366b721280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:48:44.175 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:44.172+0000 7f9535e00640 1 --2- 172.21.15.32:0/3879017805 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x56366b2fe3b0 0x56366b721280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:44.175 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:44.172+0000 7f9535e00640 1 -- 172.21.15.32:0/3879017805 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56366b306590 msgr2=0x56366b7217c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:48:44.175 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:44.172+0000 7f9535e00640 1 --2- 172.21.15.32:0/3879017805 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56366b306590 0x56366b7217c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:44.175 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:44.172+0000 7f9535e00640 1 -- 172.21.15.32:0/3879017805 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56366b6e6810 con 0x56366b31b3c0 2024-08-03T20:48:44.175 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:44.172+0000 7f9535400640 1 --2- 172.21.15.32:0/3879017805 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x56366b2fe3b0 0x56366b721280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-03T20:48:44.175 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:44.172+0000 7f9535e00640 1 --2- 172.21.15.32:0/3879017805 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56366b31b3c0 0x56366b6c6740 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f952807ea50 tx=0x7f9528005f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=44fa7d4b9e8727ad server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:48:44.175 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:44.172+0000 7f9521c00640 1 -- 172.21.15.32:0/3879017805 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9528090070 con 0x56366b31b3c0 2024-08-03T20:48:44.175 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:44.172+0000 7f9521c00640 1 -- 172.21.15.32:0/3879017805 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f952808b480 con 0x56366b31b3c0 2024-08-03T20:48:44.175 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:44.172+0000 7f9521c00640 1 -- 172.21.15.32:0/3879017805 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9528048040 con 0x56366b31b3c0 2024-08-03T20:48:44.176 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:44.173+0000 7f9543a33740 1 -- 172.21.15.32:0/3879017805 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56366b31b3c0 msgr2=0x56366b6c6740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:48:44.176 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:44.173+0000 7f9543a33740 1 --2- 172.21.15.32:0/3879017805 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56366b31b3c0 0x56366b6c6740 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f952807ea50 tx=0x7f9528005f90 comp rx=0 tx=0).stop 2024-08-03T20:48:44.176 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:44.173+0000 7f9543a33740 1 -- 172.21.15.32:0/3879017805 shutdown_connections 2024-08-03T20:48:44.177 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:44.173+0000 7f9543a33740 1 --2- 172.21.15.32:0/3879017805 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56366b306590 0x56366b7217c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:44.177 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:44.173+0000 7f9543a33740 1 --2- 172.21.15.32:0/3879017805 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x56366b2fe3b0 0x56366b721280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:44.177 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:44.173+0000 7f9543a33740 1 --2- 172.21.15.32:0/3879017805 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56366b31b3c0 0x56366b6c6740 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:44.177 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:44.173+0000 7f9543a33740 1 -- 172.21.15.32:0/3879017805 >> 172.21.15.32:0/3879017805 conn(0x56366b6caf50 msgr2=0x56366b6c7ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:48:44.177 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:44.173+0000 7f9543a33740 1 -- 172.21.15.32:0/3879017805 shutdown_connections 2024-08-03T20:48:44.177 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:44.173+0000 7f9543a33740 1 -- 172.21.15.32:0/3879017805 wait complete. 2024-08-03T20:48:44.177 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:44.174+0000 7f9543a33740 1 -- start start 2024-08-03T20:48:44.177 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:44.175+0000 7f9543a33740 1 Processor -- start 2024-08-03T20:48:44.180 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:44.176+0000 7f9543a33740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56366b2fe3b0 0x56366b7351d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:48:44.180 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:44.176+0000 7f9543a33740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56366b306590 0x56366b78c0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:48:44.180 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:44.176+0000 7f9543a33740 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x56366b31b3c0 0x56366b6f3df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:48:44.180 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:44.176+0000 7f9543a33740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x56366b6f6d50 con 0x56366b2fe3b0 2024-08-03T20:48:44.180 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:44.176+0000 7f9543a33740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x56366b70b1c0 con 0x56366b306590 2024-08-03T20:48:44.180 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:44.176+0000 7f9543a33740 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x56366b70a580 con 0x56366b31b3c0 2024-08-03T20:48:44.180 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:44.176+0000 7f9543a33740 10 client.0 _ll_register_callbacks cb 0x7f9528081db0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-03T20:48:44.180 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:44.176+0000 7f9535e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56366b2fe3b0 0x56366b7351d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:48:44.180 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:44.176+0000 7f9535e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56366b2fe3b0 0x56366b7351d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:45036/0 (socket says 172.21.15.32:45036) 2024-08-03T20:48:44.180 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:44.176+0000 7f9535e00640 1 -- 172.21.15.32:0/601363519 learned_addr learned my addr 172.21.15.32:0/601363519 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:48:44.180 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:44.177+0000 7f9536800640 1 --2- 172.21.15.32:0/601363519 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x56366b31b3c0 0x56366b6f3df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:48:44.180 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:44.177+0000 7f9535400640 1 --2- 172.21.15.32:0/601363519 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56366b306590 0x56366b78c0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:48:44.181 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:44.177+0000 7f9535e00640 1 -- 172.21.15.32:0/601363519 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x56366b31b3c0 msgr2=0x56366b6f3df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:48:44.181 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:44.177+0000 7f9535e00640 1 --2- 172.21.15.32:0/601363519 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x56366b31b3c0 0x56366b6f3df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:44.181 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:44.177+0000 7f9535e00640 1 -- 172.21.15.32:0/601363519 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56366b306590 msgr2=0x56366b78c0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:48:44.181 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:44.177+0000 7f9535e00640 1 --2- 172.21.15.32:0/601363519 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56366b306590 0x56366b78c0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:44.181 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:44.177+0000 7f9535e00640 1 -- 172.21.15.32:0/601363519 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56366b707760 con 0x56366b2fe3b0 2024-08-03T20:48:44.181 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:44.177+0000 7f9535e00640 1 --2- 172.21.15.32:0/601363519 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56366b2fe3b0 0x56366b7351d0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f9528009860 tx=0x7f95280890d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6b51609629f463af server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:48:44.181 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:44.177+0000 7f9516a00640 10 client.0 ms_handle_connect on v2:172.21.15.32:3300/0 2024-08-03T20:48:44.181 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:44.177+0000 7f9516a00640 1 -- 172.21.15.32:0/601363519 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9528090070 con 0x56366b2fe3b0 2024-08-03T20:48:44.181 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:44.177+0000 7f9516a00640 1 -- 172.21.15.32:0/601363519 <== mon.0 v2:172.21.15.32:3300/0 2 ==== osd_map(33..33 src has 1..33) ==== 5795+0+0 (secure 0 0 0) 0x7f952807ddd0 con 0x56366b2fe3b0 2024-08-03T20:48:44.181 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:44.177+0000 7f9543a33740 1 -- 172.21.15.32:0/601363519 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f952808b480 con 0x56366b2fe3b0 2024-08-03T20:48:44.181 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:44.178+0000 7f9516a00640 1 -- 172.21.15.32:0/601363519 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f9528048040 con 0x56366b2fe3b0 2024-08-03T20:48:44.181 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:44.178+0000 7f9543a33740 10 client.5127 fetch_fsmap learned FSMap version 23 2024-08-03T20:48:44.181 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:44.178+0000 7f9543a33740 1 -- 172.21.15.32:0/601363519 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=2+}) -- 0x56366b707760 con 0x56366b2fe3b0 2024-08-03T20:48:44.181 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:44.178+0000 7f9516a00640 1 -- 172.21.15.32:0/601363519 <== mon.0 v2:172.21.15.32:3300/0 4 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f952808b480 con 0x56366b2fe3b0 2024-08-03T20:48:44.182 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:44.178+0000 7f9516a00640 1 -- 172.21.15.32:0/601363519 <== mon.0 v2:172.21.15.32:3300/0 5 ==== fsmap(e 23) ==== 3502+0+0 (secure 0 0 0) 0x7f95280a2030 con 0x56366b2fe3b0 2024-08-03T20:48:44.182 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:44.178+0000 7f9543a33740 10 client.5127 fetch_fsmap finished waiting for FSMap version 23 2024-08-03T20:48:44.182 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:44.178+0000 7f9543a33740 10 client.5127 resolve_mds: resolved name 'a' to daemon mds.a 2024-08-03T20:48:44.182 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:44.178+0000 7f9543a33740 20 client.5127 populate_metadata read hostname 'smithi032' 2024-08-03T20:48:44.182 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:44.178+0000 7f9543a33740 1 --2- 172.21.15.32:0/601363519 >> [v2:172.21.15.32:6832/211300533,v1:172.21.15.32:6835/211300533] conn(0x56366b2bb280 0x56366b7248a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:48:44.182 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:44.178+0000 7f9543a33740 4 client.5127 mds_command: new command op to 5070 tid=0 multi_id=0 [{"prefix": "status"}] 2024-08-03T20:48:44.182 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:44.178+0000 7f9543a33740 1 -- 172.21.15.32:0/601363519 --> [v2:172.21.15.32:6832/211300533,v1:172.21.15.32:6835/211300533] -- command(tid 0: {"prefix": "status"}) -- 0x56366b6f67d0 con 0x56366b2bb280 2024-08-03T20:48:44.182 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:44.178+0000 7f9535400640 1 --2- 172.21.15.32:0/601363519 >> [v2:172.21.15.32:6832/211300533,v1:172.21.15.32:6835/211300533] conn(0x56366b2bb280 0x56366b7248a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:48:44.183 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:44.179+0000 7f9516a00640 10 client.5127 ms_handle_connect on v2:172.21.15.32:6832/211300533 2024-08-03T20:48:44.183 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:44.179+0000 7f9535400640 1 --2- 172.21.15.32:0/601363519 >> [v2:172.21.15.32:6832/211300533,v1:172.21.15.32:6835/211300533] conn(0x56366b2bb280 0x56366b7248a0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bb59d58520412bf server_cookie=d51a51e29185fe0d in_seq=0 out_seq=0 2024-08-03T20:48:44.183 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:44.179+0000 7f9516a00640 1 -- 172.21.15.32:0/601363519 <== mds.0 v2:172.21.15.32:6832/211300533 1 ==== command_reply(tid 0: 0 ) ==== 8+0+324 (crc 0 0 0) 0x56366b6f67d0 con 0x56366b2bb280 2024-08-03T20:48:44.183 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:44.179+0000 7f9516a00640 10 client.5127 handle_command_reply: tid=0 2024-08-03T20:48:44.183 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:44.179+0000 7f9543a33740 1 client.5127 shutdown 2024-08-03T20:48:44.184 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:44.179+0000 7f9543a33740 1 -- 172.21.15.32:0/601363519 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56366b2fe3b0 msgr2=0x56366b7351d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:48:44.184 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:44.179+0000 7f9543a33740 1 --2- 172.21.15.32:0/601363519 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56366b2fe3b0 0x56366b7351d0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f9528009860 tx=0x7f95280890d0 comp rx=0 tx=0).stop 2024-08-03T20:48:44.184 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:44.180+0000 7f9543a33740 1 -- 172.21.15.32:0/601363519 shutdown_connections 2024-08-03T20:48:44.184 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:44.180+0000 7f9543a33740 1 --2- 172.21.15.32:0/601363519 >> [v2:172.21.15.32:6832/211300533,v1:172.21.15.32:6835/211300533] conn(0x56366b2bb280 0x56366b7248a0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:44.184 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:44.180+0000 7f9543a33740 1 --2- 172.21.15.32:0/601363519 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56366b306590 0x56366b78c0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:44.184 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:44.180+0000 7f9543a33740 1 --2- 172.21.15.32:0/601363519 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x56366b31b3c0 0x56366b6f3df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:44.184 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:44.180+0000 7f9516a00640 0 client.5127 ms_handle_reset on v2:172.21.15.32:6832/211300533 2024-08-03T20:48:44.184 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:44.180+0000 7f9543a33740 1 --2- 172.21.15.32:0/601363519 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56366b2fe3b0 0x56366b7351d0 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:44.184 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:44.180+0000 7f9543a33740 1 -- 172.21.15.32:0/601363519 >> 172.21.15.32:0/601363519 conn(0x56366b6caf50 msgr2=0x56366b71fa40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:48:44.184 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:44.180+0000 7f9543a33740 1 -- 172.21.15.32:0/601363519 shutdown_connections 2024-08-03T20:48:44.184 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:44.180+0000 7f9543a33740 1 -- 172.21.15.32:0/601363519 wait complete. 2024-08-03T20:48:44.184 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:44.180+0000 7f9543a33740 20 client.5127 trim_cache size 0 max 16384 2024-08-03T20:48:44.185 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-08-03T20:48:44.185 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_fsid": "40ab141a-01d7-49b9-ae3c-2edec9876e95", 2024-08-03T20:48:44.185 INFO:teuthology.orchestra.run.smithi032.stdout: "whoami": 0, 2024-08-03T20:48:44.185 INFO:teuthology.orchestra.run.smithi032.stdout: "id": 5070, 2024-08-03T20:48:44.185 INFO:teuthology.orchestra.run.smithi032.stdout: "want_state": "up:active", 2024-08-03T20:48:44.185 INFO:teuthology.orchestra.run.smithi032.stdout: "state": "up:active", 2024-08-03T20:48:44.185 INFO:teuthology.orchestra.run.smithi032.stdout: "fs_name": "cephfs", 2024-08-03T20:48:44.185 INFO:teuthology.orchestra.run.smithi032.stdout: "rank_uptime": 5.084973411, 2024-08-03T20:48:44.185 INFO:teuthology.orchestra.run.smithi032.stdout: "mdsmap_epoch": 23, 2024-08-03T20:48:44.185 INFO:teuthology.orchestra.run.smithi032.stdout: "osdmap_epoch": 33, 2024-08-03T20:48:44.185 INFO:teuthology.orchestra.run.smithi032.stdout: "osdmap_epoch_barrier": 33, 2024-08-03T20:48:44.185 INFO:teuthology.orchestra.run.smithi032.stdout: "uptime": 5.9586626889999996 2024-08-03T20:48:44.186 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-08-03T20:48:44.186 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:44.180+0000 7f953ca00640 1 -- 172.21.15.32:0/2437726910 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f9518077010 msgr2=0x7f9518079520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:48:44.186 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:44.180+0000 7f953ca00640 1 --2- 172.21.15.32:0/2437726910 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f9518077010 0x7f9518079520 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f9524007fd0 tx=0x7f9524009040 comp rx=0 tx=0).stop 2024-08-03T20:48:44.187 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:44.180+0000 7f953ca00640 1 -- 172.21.15.32:0/2437726910 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f953806e930 msgr2=0x7f953815b610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:48:44.187 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:44.180+0000 7f953ca00640 1 --2- 172.21.15.32:0/2437726910 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f953806e930 0x7f953815b610 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f9528009980 tx=0x7f9528003c10 comp rx=0 tx=0).stop 2024-08-03T20:48:44.187 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:44.181+0000 7f953ca00640 1 -- 172.21.15.32:0/2437726910 shutdown_connections 2024-08-03T20:48:44.187 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:44.181+0000 7f953ca00640 1 --2- 172.21.15.32:0/2437726910 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f9518077010 0x7f9518079520 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:44.187 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:44.181+0000 7f953ca00640 1 --2- 172.21.15.32:0/2437726910 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f953815fff0 0x7f953815f430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:44.187 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:44.181+0000 7f953ca00640 1 --2- 172.21.15.32:0/2437726910 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f9538073e60 0x7f953815eef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:44.187 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:44.181+0000 7f953ca00640 1 --2- 172.21.15.32:0/2437726910 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f953806e930 0x7f953815b610 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:44.187 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:44.181+0000 7f953ca00640 1 -- 172.21.15.32:0/2437726910 >> 172.21.15.32:0/2437726910 conn(0x7f95380b1bd0 msgr2=0x7f9538074270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:48:44.187 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:44.181+0000 7f953ca00640 1 -- 172.21.15.32:0/2437726910 shutdown_connections 2024-08-03T20:48:44.187 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:44.181+0000 7f953ca00640 1 -- 172.21.15.32:0/2437726910 wait complete. 2024-08-03T20:48:44.205 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-08-03T20:48:44.435 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:44.433+0000 7f54b4800640 1 -- 172.21.15.32:0/780997870 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f54ac15f610 msgr2=0x7f54ac0cdf70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:48:44.435 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:44.433+0000 7f54b4800640 1 --2- 172.21.15.32:0/780997870 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f54ac15f610 0x7f54ac0cdf70 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f54a8005fd0 tx=0x7f54a802a830 comp rx=0 tx=0).stop 2024-08-03T20:48:44.435 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:44.433+0000 7f54b4800640 1 -- 172.21.15.32:0/780997870 shutdown_connections 2024-08-03T20:48:44.435 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:44.433+0000 7f54b4800640 1 --2- 172.21.15.32:0/780997870 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f54ac15f610 0x7f54ac0cdf70 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:44.435 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:44.433+0000 7f54b4800640 1 --2- 172.21.15.32:0/780997870 >> v2:172.21.15.118:3300/0 conn(0x7f54ac0b38b0 0x7f54ac1523f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:44.436 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:44.433+0000 7f54b4800640 1 -- 172.21.15.32:0/780997870 >> 172.21.15.32:0/780997870 conn(0x7f54ac0b1bd0 msgr2=0x7f54ac0b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:48:44.436 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:44.433+0000 7f54b4800640 1 -- 172.21.15.32:0/780997870 shutdown_connections 2024-08-03T20:48:44.436 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:44.433+0000 7f54b4800640 1 -- 172.21.15.32:0/780997870 wait complete. 2024-08-03T20:48:44.437 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:44.436+0000 7f54b4800640 1 Processor -- start 2024-08-03T20:48:44.437 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:44.436+0000 7f54b4800640 1 -- start start 2024-08-03T20:48:44.439 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:44.437+0000 7f54b4800640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f54ac0b38b0 0x7f54ac14dd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:48:44.439 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:44.437+0000 7f54b4800640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f54ac0b4ce0 0x7f54ac153550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:48:44.439 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:44.437+0000 7f54b4800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f54ac1556a0 0x7f54ac153a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:48:44.439 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:44.437+0000 7f54b4800640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f54ac0cea30 con 0x7f54ac1556a0 2024-08-03T20:48:44.439 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:44.437+0000 7f54b4800640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f54ac0ce8b0 con 0x7f54ac0b38b0 2024-08-03T20:48:44.439 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:44.437+0000 7f54b4800640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7f54ac0cebb0 con 0x7f54ac0b4ce0 2024-08-03T20:48:44.439 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:44.437+0000 7f54b2800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f54ac1556a0 0x7f54ac153a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:48:44.439 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:44.438+0000 7f54b2800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f54ac1556a0 0x7f54ac153a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:45046/0 (socket says 172.21.15.32:45046) 2024-08-03T20:48:44.440 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:44.438+0000 7f54b2800640 1 -- 172.21.15.32:0/2994744935 learned_addr learned my addr 172.21.15.32:0/2994744935 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:48:44.440 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:44.438+0000 7f54b1e00640 1 --2- 172.21.15.32:0/2994744935 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f54ac0b38b0 0x7f54ac14dd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:48:44.440 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:44.438+0000 7f54b1400640 1 --2- 172.21.15.32:0/2994744935 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f54ac0b4ce0 0x7f54ac153550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:48:44.441 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:44.438+0000 7f54b2800640 1 -- 172.21.15.32:0/2994744935 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f54ac0b4ce0 msgr2=0x7f54ac153550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:48:44.441 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:44.438+0000 7f54b2800640 1 --2- 172.21.15.32:0/2994744935 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f54ac0b4ce0 0x7f54ac153550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:44.441 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:44.438+0000 7f54b2800640 1 -- 172.21.15.32:0/2994744935 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f54ac0b38b0 msgr2=0x7f54ac14dd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:48:44.441 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:44.438+0000 7f54b2800640 1 --2- 172.21.15.32:0/2994744935 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f54ac0b38b0 0x7f54ac14dd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:44.441 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:44.438+0000 7f54b2800640 1 -- 172.21.15.32:0/2994744935 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f54ac0af680 con 0x7f54ac1556a0 2024-08-03T20:48:44.441 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:44.438+0000 7f54b1e00640 1 --2- 172.21.15.32:0/2994744935 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f54ac0b38b0 0x7f54ac14dd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-03T20:48:44.441 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:44.438+0000 7f54b2800640 1 --2- 172.21.15.32:0/2994744935 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f54ac1556a0 0x7f54ac153a90 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f54a8034040 tx=0x7f54a8032c60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:48:44.441 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:44.439+0000 7f54a6000640 1 -- 172.21.15.32:0/2994744935 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54a8041020 con 0x7f54ac1556a0 2024-08-03T20:48:44.441 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:44.439+0000 7f54a6000640 1 -- 172.21.15.32:0/2994744935 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f54a8032e50 con 0x7f54ac1556a0 2024-08-03T20:48:44.441 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:44.439+0000 7f54a6000640 1 -- 172.21.15.32:0/2994744935 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54a8036040 con 0x7f54ac1556a0 2024-08-03T20:48:44.442 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:44.439+0000 7f54b4800640 1 -- 172.21.15.32:0/2994744935 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f54ac072e60 con 0x7f54ac1556a0 2024-08-03T20:48:44.442 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:44.439+0000 7f54b4800640 1 -- 172.21.15.32:0/2994744935 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f54ac150250 con 0x7f54ac1556a0 2024-08-03T20:48:44.442 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:44.440+0000 7f54b4800640 1 -- 172.21.15.32:0/2994744935 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f54ac14fce0 con 0x7f54ac1556a0 2024-08-03T20:48:44.442 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:44.441+0000 7f54a6000640 1 -- 172.21.15.32:0/2994744935 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f54a8044020 con 0x7f54ac1556a0 2024-08-03T20:48:44.443 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:44.442+0000 7f54a6000640 1 --2- 172.21.15.32:0/2994744935 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f5490077040 0x7f5490079550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:48:44.443 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:44.442+0000 7f54a6000640 1 -- 172.21.15.32:0/2994744935 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(33..33 src has 1..33) ==== 5795+0+0 (secure 0 0 0) 0x7f54a80b9860 con 0x7f54ac1556a0 2024-08-03T20:48:44.444 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:44.443+0000 7f54b1e00640 1 --2- 172.21.15.32:0/2994744935 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f5490077040 0x7f5490079550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:48:44.445 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:44.443+0000 7f54b1e00640 1 --2- 172.21.15.32:0/2994744935 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f5490077040 0x7f5490079550 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f54a00095f0 tx=0x7f54a0005f90 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:48:44.448 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:44.447+0000 7f54a6000640 1 -- 172.21.15.32:0/2994744935 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7f54a804b030 con 0x7f54ac1556a0 2024-08-03T20:48:44.689 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:44.688+0000 7f54b4800640 1 -- 172.21.15.32:0/2994744935 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f54ac0cf110 con 0x7f54ac1556a0 2024-08-03T20:48:44.690 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:44.688+0000 7f54a6000640 1 -- 172.21.15.32:0/2994744935 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v23) ==== 53+0+73 (secure 0 0 0) 0x7f54a8082310 con 0x7f54ac1556a0 2024-08-03T20:48:44.690 INFO:teuthology.orchestra.run.smithi032.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-08-03T20:48:44.692 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:44.690+0000 7f54b4800640 1 -- 172.21.15.32:0/2994744935 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f5490077040 msgr2=0x7f5490079550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:48:44.692 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:44.690+0000 7f54b4800640 1 --2- 172.21.15.32:0/2994744935 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f5490077040 0x7f5490079550 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f54a00095f0 tx=0x7f54a0005f90 comp rx=0 tx=0).stop 2024-08-03T20:48:44.692 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:44.690+0000 7f54b4800640 1 -- 172.21.15.32:0/2994744935 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f54ac1556a0 msgr2=0x7f54ac153a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:48:44.692 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:44.690+0000 7f54b4800640 1 --2- 172.21.15.32:0/2994744935 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f54ac1556a0 0x7f54ac153a90 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f54a8034040 tx=0x7f54a8032c60 comp rx=0 tx=0).stop 2024-08-03T20:48:44.692 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:44.690+0000 7f54b4800640 1 -- 172.21.15.32:0/2994744935 shutdown_connections 2024-08-03T20:48:44.692 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:44.690+0000 7f54b4800640 1 --2- 172.21.15.32:0/2994744935 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f5490077040 0x7f5490079550 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:44.692 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:44.690+0000 7f54b4800640 1 --2- 172.21.15.32:0/2994744935 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f54ac1556a0 0x7f54ac153a90 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:44.693 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:44.690+0000 7f54b4800640 1 --2- 172.21.15.32:0/2994744935 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f54ac0b4ce0 0x7f54ac153550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:44.693 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:44.690+0000 7f54b4800640 1 --2- 172.21.15.32:0/2994744935 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f54ac0b38b0 0x7f54ac14dd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:44.693 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:44.690+0000 7f54b4800640 1 -- 172.21.15.32:0/2994744935 >> 172.21.15.32:0/2994744935 conn(0x7f54ac0b1bd0 msgr2=0x7f54ac06f1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:48:44.693 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:44.690+0000 7f54b4800640 1 -- 172.21.15.32:0/2994744935 shutdown_connections 2024-08-03T20:48:44.693 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:44.690+0000 7f54b4800640 1 -- 172.21.15.32:0/2994744935 wait complete. 2024-08-03T20:48:44.716 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-08-03T20:48:44.716 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-08-03T20:48:44.716 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi032.front.sepia.ceph.com 2024-08-03T20:48:44.716 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-08-03T20:48:44.716 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-08-03T20:48:44.716 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-08-03T20:48:44.716 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2024-08-03T20:48:44.716 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-08-03T20:48:44.716 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-08-03T20:48:44.716 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-08-03T20:48:44.717 DEBUG:teuthology.orchestra.run.smithi032:> ip addr 2024-08-03T20:48:44.752 INFO:teuthology.orchestra.run.smithi032.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-08-03T20:48:44.752 INFO:teuthology.orchestra.run.smithi032.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-08-03T20:48:44.752 INFO:teuthology.orchestra.run.smithi032.stdout: inet 127.0.0.1/8 scope host lo 2024-08-03T20:48:44.752 INFO:teuthology.orchestra.run.smithi032.stdout: valid_lft forever preferred_lft forever 2024-08-03T20:48:44.752 INFO:teuthology.orchestra.run.smithi032.stdout: inet6 ::1/128 scope host 2024-08-03T20:48:44.752 INFO:teuthology.orchestra.run.smithi032.stdout: valid_lft forever preferred_lft forever 2024-08-03T20:48:44.753 INFO:teuthology.orchestra.run.smithi032.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-08-03T20:48:44.753 INFO:teuthology.orchestra.run.smithi032.stdout: link/ether 0c:c4:7a:6b:fd:0c brd ff:ff:ff:ff:ff:ff 2024-08-03T20:48:44.753 INFO:teuthology.orchestra.run.smithi032.stdout: altname enp7s0f0 2024-08-03T20:48:44.753 INFO:teuthology.orchestra.run.smithi032.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-08-03T20:48:44.753 INFO:teuthology.orchestra.run.smithi032.stdout: link/ether 0c:c4:7a:6b:fd:0d brd ff:ff:ff:ff:ff:ff 2024-08-03T20:48:44.753 INFO:teuthology.orchestra.run.smithi032.stdout: altname enp7s0f1 2024-08-03T20:48:44.753 INFO:teuthology.orchestra.run.smithi032.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-08-03T20:48:44.753 INFO:teuthology.orchestra.run.smithi032.stdout: link/ether ac:1f:6b:f8:26:02 brd ff:ff:ff:ff:ff:ff 2024-08-03T20:48:44.753 INFO:teuthology.orchestra.run.smithi032.stdout: altname enp1s0f0 2024-08-03T20:48:44.753 INFO:teuthology.orchestra.run.smithi032.stdout: inet 172.21.15.32/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-08-03T20:48:44.753 INFO:teuthology.orchestra.run.smithi032.stdout: valid_lft 42431sec preferred_lft 42431sec 2024-08-03T20:48:44.753 INFO:teuthology.orchestra.run.smithi032.stdout: inet6 fe80::ae1f:6bff:fef8:2602/64 scope link noprefixroute 2024-08-03T20:48:44.754 INFO:teuthology.orchestra.run.smithi032.stdout: valid_lft forever preferred_lft forever 2024-08-03T20:48:44.754 INFO:teuthology.orchestra.run.smithi032.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-08-03T20:48:44.754 INFO:teuthology.orchestra.run.smithi032.stdout: link/ether ac:1f:6b:f8:26:03 brd ff:ff:ff:ff:ff:ff 2024-08-03T20:48:44.754 INFO:teuthology.orchestra.run.smithi032.stdout: altname enp1s0f1 2024-08-03T20:48:44.754 INFO:teuthology.orchestra.run.smithi032.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-08-03T20:48:44.754 INFO:teuthology.orchestra.run.smithi032.stdout: link/ether 0a:23:2b:ca:2a:93 brd ff:ff:ff:ff:ff:ff 2024-08-03T20:48:44.754 INFO:teuthology.orchestra.run.smithi032.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-08-03T20:48:44.754 INFO:teuthology.orchestra.run.smithi032.stdout: valid_lft forever preferred_lft forever 2024-08-03T20:48:44.754 INFO:teuthology.orchestra.run.smithi032.stdout: inet6 fe80::d804:7cff:fed9:7ba4/64 scope link 2024-08-03T20:48:44.754 INFO:teuthology.orchestra.run.smithi032.stdout: valid_lft forever preferred_lft forever 2024-08-03T20:48:44.754 INFO:teuthology.orchestra.run.smithi032.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-08-03T20:48:44.755 INFO:teuthology.orchestra.run.smithi032.stdout: link/ether 22:86:28:30:4e:62 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-08-03T20:48:44.755 INFO:teuthology.orchestra.run.smithi032.stdout: inet6 fe80::2086:28ff:fe30:4e62/64 scope link 2024-08-03T20:48:44.755 INFO:teuthology.orchestra.run.smithi032.stdout: valid_lft forever preferred_lft forever 2024-08-03T20:48:44.755 INFO:teuthology.orchestra.run.smithi032.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-08-03T20:48:44.755 INFO:teuthology.orchestra.run.smithi032.stdout: link/ether 0a:23:2b:ca:2a:93 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-08-03T20:48:44.755 INFO:teuthology.orchestra.run.smithi032.stdout: inet6 fe80::823:2bff:feca:2a93/64 scope link 2024-08-03T20:48:44.755 INFO:teuthology.orchestra.run.smithi032.stdout: valid_lft forever preferred_lft forever 2024-08-03T20:48:44.755 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-08-03T20:48:44.755 DEBUG:teuthology.orchestra.run.smithi032:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-08-03T20:48:44.846 INFO:teuthology.orchestra.run.smithi032.stdout:1 2024-08-03T20:48:44.849 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-08-03T20:48:44.849 DEBUG:teuthology.orchestra.run.smithi032:> route 2024-08-03T20:48:44.886 INFO:teuthology.orchestra.run.smithi032.stdout:Kernel IP routing table 2024-08-03T20:48:44.886 INFO:teuthology.orchestra.run.smithi032.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-08-03T20:48:44.886 INFO:teuthology.orchestra.run.smithi032.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-08-03T20:48:44.886 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-08-03T20:48:44.886 INFO:teuthology.orchestra.run.smithi032.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-08-03T20:48:44.887 DEBUG:teuthology.orchestra.run.smithi032:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-08-03T20:48:44.887 DEBUG:teuthology.orchestra.run.smithi032:> set -e 2024-08-03T20:48:44.887 DEBUG:teuthology.orchestra.run.smithi032:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-08-03T20:48:44.887 DEBUG:teuthology.orchestra.run.smithi032:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-08-03T20:48:44.887 DEBUG:teuthology.orchestra.run.smithi032:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-08-03T20:48:44.887 DEBUG:teuthology.orchestra.run.smithi032:> ') 2024-08-03T20:48:45.005 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:45 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-08-03T20:48:45.077 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:45 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-08-03T20:48:45.086 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-08-03T20:48:45.086 DEBUG:teuthology.orchestra.run.smithi032:> ip netns list 2024-08-03T20:48:45.121 INFO:teuthology.orchestra.run.smithi032.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-08-03T20:48:45.121 INFO:teuthology.orchestra.run.smithi032.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-08-03T20:48:45.122 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-08-03T20:48:45.122 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-08-03T20:48:45.122 DEBUG:teuthology.orchestra.run.smithi032:> sudo ip link set brx.0 up 2024-08-03T20:48:45.217 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-08-03T20:48:45.217 DEBUG:teuthology.orchestra.run.smithi032:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-08-03T20:48:45.252 INFO:teuthology.orchestra.run.smithi032.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-08-03T20:48:45.252 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-08-03T20:48:45.252 DEBUG:teuthology.orchestra.run.smithi032:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-08-03T20:48:45.327 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2024-08-03T20:48:45.327 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-08-03T20:48:45.328 DEBUG:teuthology.orchestra.run.smithi032:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-08-03T20:48:45.526 INFO:teuthology.orchestra.run.smithi032.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-08-03T20:48:45.526 INFO:teuthology.orchestra.run.smithi032.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-08-03T20:48:45.526 INFO:teuthology.orchestra.run.smithi032.stdout:invalid new device string format 2024-08-03T20:48:45.546 INFO:teuthology.orchestra.run.smithi032.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-08-03T20:48:45.546 INFO:teuthology.orchestra.run.smithi032.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-08-03T20:48:45.546 INFO:teuthology.orchestra.run.smithi032.stdout:invalid new device string format 2024-08-03T20:48:45.547 INFO:teuthology.orchestra.run.smithi032.stdout:mount.ceph: resolved to: "172.21.15.32:3300,172.21.15.118:3300,172.21.15.118:3301" 2024-08-03T20:48:45.547 INFO:teuthology.orchestra.run.smithi032.stdout:mount.ceph: switching to using v1 address with old syntax 2024-08-03T20:48:45.578 INFO:teuthology.orchestra.run.smithi032.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-08-03T20:48:45.579 INFO:teuthology.orchestra.run.smithi032.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-08-03T20:48:45.579 INFO:teuthology.orchestra.run.smithi032.stdout:invalid new device string format 2024-08-03T20:48:45.579 INFO:teuthology.orchestra.run.smithi032.stdout:mount.ceph: resolved to: "172.21.15.32:3300,172.21.15.118:3300,172.21.15.118:3301" 2024-08-03T20:48:45.579 INFO:teuthology.orchestra.run.smithi032.stdout:mount.ceph: switching to using v1 address with old syntax 2024-08-03T20:48:45.579 INFO:teuthology.orchestra.run.smithi032.stdout:mount.ceph: resolved to: "172.21.15.32:6789,172.21.15.118:6789,172.21.15.118:6790" 2024-08-03T20:48:45.579 INFO:teuthology.orchestra.run.smithi032.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.32:6789,172.21.15.118:6789,172.21.15.118:6790:/ 2024-08-03T20:48:45.579 INFO:teuthology.orchestra.run.smithi032.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,key=0,fsid=40ab141a-01d7-49b9-ae3c-2edec9876e95" will pass to kernel 2024-08-03T20:48:45.580 INFO:teuthology.orchestra.run.smithi032.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2024-08-03T20:48:45.580 INFO:teuthology.orchestra.run.smithi032.stdout: You just mounted a file system that supports labels which does not 2024-08-03T20:48:45.580 INFO:teuthology.orchestra.run.smithi032.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-08-03T20:48:45.580 INFO:teuthology.orchestra.run.smithi032.stdout: applications will generate AVC messages and not be allowed access to 2024-08-03T20:48:45.581 INFO:teuthology.orchestra.run.smithi032.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-08-03T20:48:45.583 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-08-03T20:48:45.583 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-08-03T20:48:45.584 DEBUG:teuthology.orchestra.run.smithi032:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-08-03T20:48:45.639 DEBUG:teuthology.orchestra.run.smithi032:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-08-03T20:48:45.719 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:45 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-08-03T20:48:45.722 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:45 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-08-03T20:48:45.730 INFO:teuthology.orchestra.run.smithi032.stdout:client5029 2024-08-03T20:48:45.730 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-08-03T20:48:45.730 DEBUG:teuthology.orchestra.run.smithi032:> dd if=/proc/self/mounts of=/dev/stdout 2024-08-03T20:48:45.765 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-08-03T20:48:45.766 DEBUG:teuthology.orchestra.run.smithi032:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls subdir/) 2024-08-03T20:48:45.843 INFO:teuthology.orchestra.run.smithi032.stdout:file_undamaged 2024-08-03T20:48:45.845 DEBUG:teuthology.orchestra.run.smithi032:> stdin-killer -- python3 -c ' 2024-08-03T20:48:45.845 DEBUG:teuthology.orchestra.run.smithi032:> import os 2024-08-03T20:48:45.845 DEBUG:teuthology.orchestra.run.smithi032:> import stat 2024-08-03T20:48:45.845 DEBUG:teuthology.orchestra.run.smithi032:> import json 2024-08-03T20:48:45.845 DEBUG:teuthology.orchestra.run.smithi032:> import sys 2024-08-03T20:48:45.845 DEBUG:teuthology.orchestra.run.smithi032:> 2024-08-03T20:48:45.845 DEBUG:teuthology.orchestra.run.smithi032:> try: 2024-08-03T20:48:45.845 DEBUG:teuthology.orchestra.run.smithi032:> s = os.stat('"'"'/home/ubuntu/cephtest/mnt.0/subdir/file_to_be_damaged'"'"') 2024-08-03T20:48:45.845 DEBUG:teuthology.orchestra.run.smithi032:> except OSError as e: 2024-08-03T20:48:45.845 DEBUG:teuthology.orchestra.run.smithi032:> sys.exit(e.errno) 2024-08-03T20:48:45.845 DEBUG:teuthology.orchestra.run.smithi032:> 2024-08-03T20:48:45.846 DEBUG:teuthology.orchestra.run.smithi032:> attrs = ["st_mode", "st_ino", "st_dev", "st_nlink", "st_uid", "st_gid", "st_size", "st_atime", "st_mtime", "st_ctime"] 2024-08-03T20:48:45.846 DEBUG:teuthology.orchestra.run.smithi032:> print(json.dumps( 2024-08-03T20:48:45.846 DEBUG:teuthology.orchestra.run.smithi032:> dict([(a, getattr(s, a)) for a in attrs]), 2024-08-03T20:48:45.846 DEBUG:teuthology.orchestra.run.smithi032:> indent=2)) 2024-08-03T20:48:45.846 DEBUG:teuthology.orchestra.run.smithi032:> ' 2024-08-03T20:48:45.992 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:45 stdin-killer INFO: command exited with status 2: exiting normally with same code! 2024-08-03T20:48:45.999 DEBUG:teuthology.orchestra.run:got remote process result: 2 2024-08-03T20:48:46.000 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-08-03T20:48:46.269 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:46.267+0000 7fb643c00640 1 -- 172.21.15.32:0/1507194476 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb63c144f90 msgr2=0x7fb63c074570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:48:46.269 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:46.267+0000 7fb643c00640 1 --2- 172.21.15.32:0/1507194476 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb63c144f90 0x7fb63c074570 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7fb63c072d80 tx=0x7fb62802a5f0 comp rx=0 tx=0).stop 2024-08-03T20:48:46.269 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:46.267+0000 7fb643c00640 1 -- 172.21.15.32:0/1507194476 shutdown_connections 2024-08-03T20:48:46.270 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:46.267+0000 7fb643c00640 1 --2- 172.21.15.32:0/1507194476 >> v2:172.21.15.118:3300/0 conn(0x7fb63c15fff0 0x7fb63c160400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:46.270 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:46.267+0000 7fb643c00640 1 --2- 172.21.15.32:0/1507194476 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb63c144f90 0x7fb63c074570 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:46.270 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:46.267+0000 7fb643c00640 1 -- 172.21.15.32:0/1507194476 >> 172.21.15.32:0/1507194476 conn(0x7fb63c0b1bd0 msgr2=0x7fb63c0b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:48:46.270 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:46.267+0000 7fb643c00640 1 -- 172.21.15.32:0/1507194476 shutdown_connections 2024-08-03T20:48:46.270 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:46.268+0000 7fb643c00640 1 -- 172.21.15.32:0/1507194476 wait complete. 2024-08-03T20:48:46.271 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:46.270+0000 7fb643c00640 1 Processor -- start 2024-08-03T20:48:46.271 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:46.270+0000 7fb643c00640 1 -- start start 2024-08-03T20:48:46.273 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:46.271+0000 7fb643c00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb63c06e930 0x7fb63c0b44a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:48:46.273 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:46.271+0000 7fb643c00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb63c144f90 0x7fb63c1f82a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:48:46.273 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:46.271+0000 7fb643c00640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fb63c15fff0 0x7fb63c142e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:48:46.274 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:46.271+0000 7fb643c00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fb63c0cf840 con 0x7fb63c06e930 2024-08-03T20:48:46.274 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:46.271+0000 7fb643c00640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7fb63c145f30 con 0x7fb63c144f90 2024-08-03T20:48:46.274 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:46.271+0000 7fb643c00640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7fb63c0cf9c0 con 0x7fb63c15fff0 2024-08-03T20:48:46.274 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:46.271+0000 7fb641200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb63c06e930 0x7fb63c0b44a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:48:46.274 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:46.272+0000 7fb641200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb63c06e930 0x7fb63c0b44a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:45064/0 (socket says 172.21.15.32:45064) 2024-08-03T20:48:46.274 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:46.272+0000 7fb641200640 1 -- 172.21.15.32:0/675328732 learned_addr learned my addr 172.21.15.32:0/675328732 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:48:46.274 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:46.272+0000 7fb63be00640 1 --2- 172.21.15.32:0/675328732 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb63c144f90 0x7fb63c1f82a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:48:46.274 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:46.272+0000 7fb641c00640 1 --2- 172.21.15.32:0/675328732 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fb63c15fff0 0x7fb63c142e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:48:46.274 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:46.272+0000 7fb641200640 1 -- 172.21.15.32:0/675328732 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fb63c15fff0 msgr2=0x7fb63c142e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:48:46.274 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:46.272+0000 7fb641200640 1 --2- 172.21.15.32:0/675328732 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fb63c15fff0 0x7fb63c142e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:46.274 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:46.272+0000 7fb641200640 1 -- 172.21.15.32:0/675328732 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb63c144f90 msgr2=0x7fb63c1f82a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:48:46.274 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:46.272+0000 7fb641200640 1 --2- 172.21.15.32:0/675328732 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb63c144f90 0x7fb63c1f82a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:46.274 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:46.272+0000 7fb641200640 1 -- 172.21.15.32:0/675328732 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb63c0b4e80 con 0x7fb63c06e930 2024-08-03T20:48:46.275 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:46.272+0000 7fb63be00640 1 --2- 172.21.15.32:0/675328732 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb63c144f90 0x7fb63c1f82a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-03T20:48:46.275 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:46.272+0000 7fb641200640 1 --2- 172.21.15.32:0/675328732 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb63c06e930 0x7fb63c0b44a0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7fb63000c3a0 tx=0x7fb63000c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:48:46.275 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:46.272+0000 7fb638c00640 1 -- 172.21.15.32:0/675328732 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb630017070 con 0x7fb63c06e930 2024-08-03T20:48:46.275 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:46.272+0000 7fb638c00640 1 -- 172.21.15.32:0/675328732 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fb6300053c0 con 0x7fb63c06e930 2024-08-03T20:48:46.276 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:46.272+0000 7fb638c00640 1 -- 172.21.15.32:0/675328732 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb630003850 con 0x7fb63c06e930 2024-08-03T20:48:46.276 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:46.273+0000 7fb643c00640 1 -- 172.21.15.32:0/675328732 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb63c1f8ae0 con 0x7fb63c06e930 2024-08-03T20:48:46.276 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:46.273+0000 7fb643c00640 1 -- 172.21.15.32:0/675328732 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb63c1f90a0 con 0x7fb63c06e930 2024-08-03T20:48:46.276 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:46.274+0000 7fb643c00640 1 -- 172.21.15.32:0/675328732 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb63c160c30 con 0x7fb63c06e930 2024-08-03T20:48:46.276 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:46.275+0000 7fb638c00640 1 -- 172.21.15.32:0/675328732 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7fb63001a070 con 0x7fb63c06e930 2024-08-03T20:48:46.277 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:46.276+0000 7fb638c00640 1 --2- 172.21.15.32:0/675328732 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fb61c077040 0x7fb61c079550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:48:46.277 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:46.276+0000 7fb638c00640 1 -- 172.21.15.32:0/675328732 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(33..33 src has 1..33) ==== 5795+0+0 (secure 0 0 0) 0x7fb630097d40 con 0x7fb63c06e930 2024-08-03T20:48:46.278 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:46.276+0000 7fb63be00640 1 --2- 172.21.15.32:0/675328732 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fb61c077040 0x7fb61c079550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:48:46.278 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:46.277+0000 7fb63be00640 1 --2- 172.21.15.32:0/675328732 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fb61c077040 0x7fb61c079550 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fb628033000 tx=0x7fb628004730 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:48:46.282 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:46.281+0000 7fb638c00640 1 -- 172.21.15.32:0/675328732 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7fb630060ed0 con 0x7fb63c06e930 2024-08-03T20:48:46.533 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:46.532+0000 7fb643c00640 1 -- 172.21.15.32:0/675328732 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb63c145920 con 0x7fb63c06e930 2024-08-03T20:48:46.534 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:46.533+0000 7fb638c00640 1 -- 172.21.15.32:0/675328732 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) ==== 94+0+5227 (secure 0 0 0) 0x7fb630069110 con 0x7fb63c06e930 2024-08-03T20:48:46.534 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-03T20:48:46.534 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":23,"btime":"2024-08-03T20:48:42:098849+0000","default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4643,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6836/4002526832","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6834","nonce":4002526832},{"type":"v1","addr":"172.21.15.118:6836","nonce":4002526832}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":4646,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6837/3036256187","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6835","nonce":3036256187},{"type":"v1","addr":"172.21.15.118:6837","nonce":3036256187}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":4652,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6834/2104418334","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6833","nonce":2104418334},{"type":"v1","addr":"172.21.15.32:6834","nonce":2104418334}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":23,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-08-03T20:48:26.652521+0000","modified":"2024-08-03T20:48:42.098834+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5070},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5070":{"gid":5070,"name":"a","rank":0,"incarnation":20,"state":"up:active","state_seq":4,"addr":"172.21.15.32:6835/211300533","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":211300533},{"type":"v1","addr":"172.21.15.32:6835","nonce":211300533}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5070,"qdb_cluster":[5070]},"id":2}]} 2024-08-03T20:48:46.535 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 23 2024-08-03T20:48:46.537 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:46.535+0000 7fb643c00640 1 -- 172.21.15.32:0/675328732 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fb61c077040 msgr2=0x7fb61c079550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:48:46.537 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:46.535+0000 7fb643c00640 1 --2- 172.21.15.32:0/675328732 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fb61c077040 0x7fb61c079550 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fb628033000 tx=0x7fb628004730 comp rx=0 tx=0).stop 2024-08-03T20:48:46.537 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:46.535+0000 7fb643c00640 1 -- 172.21.15.32:0/675328732 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb63c06e930 msgr2=0x7fb63c0b44a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:48:46.537 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:46.535+0000 7fb643c00640 1 --2- 172.21.15.32:0/675328732 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb63c06e930 0x7fb63c0b44a0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7fb63000c3a0 tx=0x7fb63000c860 comp rx=0 tx=0).stop 2024-08-03T20:48:46.537 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:46.535+0000 7fb643c00640 1 -- 172.21.15.32:0/675328732 shutdown_connections 2024-08-03T20:48:46.537 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:46.535+0000 7fb643c00640 1 --2- 172.21.15.32:0/675328732 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fb61c077040 0x7fb61c079550 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:46.537 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:46.535+0000 7fb643c00640 1 --2- 172.21.15.32:0/675328732 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb63c144f90 0x7fb63c1f82a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:46.537 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:46.535+0000 7fb643c00640 1 --2- 172.21.15.32:0/675328732 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fb63c15fff0 0x7fb63c142e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:46.538 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:46.535+0000 7fb643c00640 1 --2- 172.21.15.32:0/675328732 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb63c06e930 0x7fb63c0b44a0 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:46.538 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:46.535+0000 7fb643c00640 1 -- 172.21.15.32:0/675328732 >> 172.21.15.32:0/675328732 conn(0x7fb63c0b1bd0 msgr2=0x7fb63c15bec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:48:46.538 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:46.535+0000 7fb643c00640 1 -- 172.21.15.32:0/675328732 shutdown_connections 2024-08-03T20:48:46.538 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:46.535+0000 7fb643c00640 1 -- 172.21.15.32:0/675328732 wait complete. 2024-08-03T20:48:46.560 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a damage ls --format=json-pretty 2024-08-03T20:48:46.790 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:46.788+0000 7f8182a00640 1 -- 172.21.15.32:0/708684840 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f817c145c50 msgr2=0x7f817c06e950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:48:46.790 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:46.788+0000 7f8182a00640 1 --2- 172.21.15.32:0/708684840 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f817c145c50 0x7f817c06e950 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f816c005fd0 tx=0x7f816c02a5f0 comp rx=0 tx=0).stop 2024-08-03T20:48:46.790 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:46.788+0000 7f8182a00640 1 -- 172.21.15.32:0/708684840 shutdown_connections 2024-08-03T20:48:46.790 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:46.788+0000 7f8182a00640 1 --2- 172.21.15.32:0/708684840 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f817c0ced30 0x7f817c0cf140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:46.790 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:46.788+0000 7f8182a00640 1 --2- 172.21.15.32:0/708684840 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f817c145c50 0x7f817c06e950 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:46.790 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:46.788+0000 7f8182a00640 1 -- 172.21.15.32:0/708684840 >> 172.21.15.32:0/708684840 conn(0x7f817c0b1bd0 msgr2=0x7f817c0b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:48:46.790 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:46.788+0000 7f8182a00640 1 -- 172.21.15.32:0/708684840 shutdown_connections 2024-08-03T20:48:46.790 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:46.788+0000 7f8182a00640 1 -- 172.21.15.32:0/708684840 wait complete. 2024-08-03T20:48:46.791 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:46.790+0000 7f8182a00640 1 Processor -- start 2024-08-03T20:48:46.792 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:46.790+0000 7f8182a00640 1 -- start start 2024-08-03T20:48:46.795 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:46.792+0000 7f8182a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f817c0cccf0 0x7f817c13ba40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:48:46.795 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:46.792+0000 7f8182a00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f817c0ced30 0x7f817c1e8080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:48:46.795 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:46.792+0000 7f8182a00640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f817c145c50 0x7f817c1e85c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:48:46.795 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:46.792+0000 7f8182a00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f817c0b4070 con 0x7f817c0cccf0 2024-08-03T20:48:46.795 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:46.792+0000 7f8182a00640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f817c0b3ef0 con 0x7f817c0ced30 2024-08-03T20:48:46.795 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:46.792+0000 7f8182a00640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7f817c0b41f0 con 0x7f817c145c50 2024-08-03T20:48:46.795 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:46.792+0000 7f817be00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f817c0cccf0 0x7f817c13ba40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:48:46.796 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:46.792+0000 7f817be00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f817c0cccf0 0x7f817c13ba40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:45072/0 (socket says 172.21.15.32:45072) 2024-08-03T20:48:46.796 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:46.792+0000 7f817be00640 1 -- 172.21.15.32:0/707174188 learned_addr learned my addr 172.21.15.32:0/707174188 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:48:46.796 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:46.792+0000 7f8180a00640 1 --2- 172.21.15.32:0/707174188 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f817c145c50 0x7f817c1e85c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:48:46.796 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:46.792+0000 7f817b400640 1 --2- 172.21.15.32:0/707174188 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f817c0ced30 0x7f817c1e8080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:48:46.796 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:46.792+0000 7f817be00640 1 -- 172.21.15.32:0/707174188 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f817c145c50 msgr2=0x7f817c1e85c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:48:46.796 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:46.792+0000 7f817be00640 1 --2- 172.21.15.32:0/707174188 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f817c145c50 0x7f817c1e85c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:46.796 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:46.793+0000 7f817be00640 1 -- 172.21.15.32:0/707174188 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f817c0ced30 msgr2=0x7f817c1e8080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:48:46.796 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:46.793+0000 7f817be00640 1 --2- 172.21.15.32:0/707174188 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f817c0ced30 0x7f817c1e8080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:46.796 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:46.793+0000 7f817be00640 1 -- 172.21.15.32:0/707174188 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f817c072ad0 con 0x7f817c0cccf0 2024-08-03T20:48:46.796 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:46.793+0000 7f817b400640 1 --2- 172.21.15.32:0/707174188 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f817c0ced30 0x7f817c1e8080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-03T20:48:46.796 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:46.793+0000 7f817be00640 1 --2- 172.21.15.32:0/707174188 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f817c0cccf0 0x7f817c13ba40 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f816c006010 tx=0x7f816c0039e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:48:46.796 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:46.793+0000 7f8173e00640 1 -- 172.21.15.32:0/707174188 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f816c03e030 con 0x7f817c0cccf0 2024-08-03T20:48:46.796 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:46.793+0000 7f8173e00640 1 -- 172.21.15.32:0/707174188 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f816c0047a0 con 0x7f817c0cccf0 2024-08-03T20:48:46.797 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:46.793+0000 7f8173e00640 1 -- 172.21.15.32:0/707174188 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f816c035040 con 0x7f817c0cccf0 2024-08-03T20:48:46.797 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:46.793+0000 7f8182a00640 1 -- 172.21.15.32:0/707174188 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f817c072db0 con 0x7f817c0cccf0 2024-08-03T20:48:46.797 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:46.793+0000 7f8182a00640 1 -- 172.21.15.32:0/707174188 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f817c073370 con 0x7f817c0cccf0 2024-08-03T20:48:46.798 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:46.796+0000 7f8173e00640 1 -- 172.21.15.32:0/707174188 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f816c041030 con 0x7f817c0cccf0 2024-08-03T20:48:46.799 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:46.797+0000 7f8173e00640 1 --2- 172.21.15.32:0/707174188 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f815c077010 0x7f815c079520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:48:46.799 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:46.797+0000 7f8189a76740 1 Processor -- start 2024-08-03T20:48:46.800 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:46.797+0000 7f8173e00640 1 -- 172.21.15.32:0/707174188 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(33..33 src has 1..33) ==== 5795+0+0 (secure 0 0 0) 0x7f816c0b91b0 con 0x7f817c0cccf0 2024-08-03T20:48:46.800 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:46.797+0000 7f8189a76740 1 -- start start 2024-08-03T20:48:46.800 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:46.798+0000 7f817b400640 1 --2- 172.21.15.32:0/707174188 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f815c077010 0x7f815c079520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:48:46.811 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:46.798+0000 7f817b400640 1 --2- 172.21.15.32:0/707174188 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f815c077010 0x7f815c079520 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f817c159bc0 tx=0x7f8168009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:48:46.812 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:46.799+0000 7f8189a76740 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x558231386350 0x5582317737a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:48:46.812 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:46.799+0000 7f8189a76740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55823135d490 0x558231756470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:48:46.812 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:46.799+0000 7f8189a76740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x558231365670 0x5582317569b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:48:46.812 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:46.799+0000 7f8189a76740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55823169f050 con 0x55823135d490 2024-08-03T20:48:46.812 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:46.799+0000 7f8189a76740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x55823168e550 con 0x558231365670 2024-08-03T20:48:46.813 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:46.799+0000 7f8189a76740 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x558231692590 con 0x558231386350 2024-08-03T20:48:46.813 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:46.799+0000 7f817b400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55823135d490 0x558231756470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:48:46.813 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:46.799+0000 7f817b400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55823135d490 0x558231756470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:45082/0 (socket says 172.21.15.32:45082) 2024-08-03T20:48:46.813 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:46.799+0000 7f817b400640 1 -- 172.21.15.32:0/2746334726 learned_addr learned my addr 172.21.15.32:0/2746334726 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:48:46.814 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:46.799+0000 7f817be00640 1 --2- 172.21.15.32:0/2746334726 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x558231386350 0x5582317737a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:48:46.814 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:46.799+0000 7f8180a00640 1 --2- 172.21.15.32:0/2746334726 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x558231365670 0x5582317569b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:48:46.814 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:46.799+0000 7f817b400640 1 -- 172.21.15.32:0/2746334726 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x558231386350 msgr2=0x5582317737a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:48:46.814 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:46.799+0000 7f817b400640 1 --2- 172.21.15.32:0/2746334726 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x558231386350 0x5582317737a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:46.814 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:46.799+0000 7f817b400640 1 -- 172.21.15.32:0/2746334726 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x558231365670 msgr2=0x5582317569b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:48:46.814 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:46.799+0000 7f817b400640 1 --2- 172.21.15.32:0/2746334726 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x558231365670 0x5582317569b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:46.814 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:46.799+0000 7f817b400640 1 -- 172.21.15.32:0/2746334726 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55823166ea10 con 0x55823135d490 2024-08-03T20:48:46.814 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:46.800+0000 7f817b400640 1 --2- 172.21.15.32:0/2746334726 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55823135d490 0x558231756470 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f816800da40 tx=0x7f8168010000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=203eaf6a65e29703 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:48:46.814 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:46.800+0000 7f8167e00640 1 -- 172.21.15.32:0/2746334726 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8168024070 con 0x55823135d490 2024-08-03T20:48:46.814 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:46.800+0000 7f8167e00640 1 -- 172.21.15.32:0/2746334726 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f8168014ba0 con 0x55823135d490 2024-08-03T20:48:46.814 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:46.800+0000 7f8167e00640 1 -- 172.21.15.32:0/2746334726 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f816801f400 con 0x55823135d490 2024-08-03T20:48:46.815 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:46.801+0000 7f8189a76740 1 -- 172.21.15.32:0/2746334726 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55823135d490 msgr2=0x558231756470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:48:46.815 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:46.801+0000 7f8189a76740 1 --2- 172.21.15.32:0/2746334726 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55823135d490 0x558231756470 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f816800da40 tx=0x7f8168010000 comp rx=0 tx=0).stop 2024-08-03T20:48:46.815 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:46.801+0000 7f8189a76740 1 -- 172.21.15.32:0/2746334726 shutdown_connections 2024-08-03T20:48:46.815 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:46.801+0000 7f8189a76740 1 --2- 172.21.15.32:0/2746334726 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x558231365670 0x5582317569b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:46.815 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:46.801+0000 7f8189a76740 1 --2- 172.21.15.32:0/2746334726 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55823135d490 0x558231756470 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:46.815 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:46.801+0000 7f8189a76740 1 --2- 172.21.15.32:0/2746334726 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x558231386350 0x5582317737a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:46.815 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:46.801+0000 7f8189a76740 1 -- 172.21.15.32:0/2746334726 >> 172.21.15.32:0/2746334726 conn(0x55823137a4a0 msgr2=0x55823172c690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:48:46.815 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:46.801+0000 7f8189a76740 1 -- 172.21.15.32:0/2746334726 shutdown_connections 2024-08-03T20:48:46.815 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:46.801+0000 7f8189a76740 1 -- 172.21.15.32:0/2746334726 wait complete. 2024-08-03T20:48:46.815 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:46.802+0000 7f8189a76740 1 -- start start 2024-08-03T20:48:46.815 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:46.802+0000 7f8189a76740 1 Processor -- start 2024-08-03T20:48:46.815 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:46.804+0000 7f8189a76740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55823135d490 0x558231744b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:48:46.815 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:46.804+0000 7f8189a76740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558231365670 0x558231783f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:48:46.816 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:46.804+0000 7f8189a76740 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x558231386350 0x55823176ce90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:48:46.816 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:46.804+0000 7f8189a76740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x5582315c6da0 con 0x558231365670 2024-08-03T20:48:46.816 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:46.804+0000 7f8189a76740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x558231628cd0 con 0x55823135d490 2024-08-03T20:48:46.816 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:46.804+0000 7f8189a76740 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x558231769070 con 0x558231386350 2024-08-03T20:48:46.816 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:46.804+0000 7f8189a76740 10 client.0 _ll_register_callbacks cb 0x55823139dba0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-03T20:48:46.816 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:46.804+0000 7f817b400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558231365670 0x558231783f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:48:46.816 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:46.804+0000 7f817b400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558231365670 0x558231783f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:45088/0 (socket says 172.21.15.32:45088) 2024-08-03T20:48:46.816 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:46.804+0000 7f817b400640 1 -- 172.21.15.32:0/2492280333 learned_addr learned my addr 172.21.15.32:0/2492280333 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:48:46.816 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:46.804+0000 7f817be00640 1 --2- 172.21.15.32:0/2492280333 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55823135d490 0x558231744b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:48:46.816 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:46.804+0000 7f817b400640 1 -- 172.21.15.32:0/2492280333 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x558231386350 msgr2=0x55823176ce90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:48:46.816 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:46.804+0000 7f817b400640 1 --2- 172.21.15.32:0/2492280333 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x558231386350 0x55823176ce90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:46.816 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:46.804+0000 7f817b400640 1 -- 172.21.15.32:0/2492280333 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55823135d490 msgr2=0x558231744b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:48:46.817 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:46.804+0000 7f817b400640 1 --2- 172.21.15.32:0/2492280333 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55823135d490 0x558231744b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:46.817 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:46.804+0000 7f817b400640 1 -- 172.21.15.32:0/2492280333 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5582313f1930 con 0x558231365670 2024-08-03T20:48:46.817 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:46.804+0000 7f817be00640 1 --2- 172.21.15.32:0/2492280333 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55823135d490 0x558231744b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-03T20:48:46.817 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:46.805+0000 7f817b400640 1 --2- 172.21.15.32:0/2492280333 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558231365670 0x558231783f30 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f8168022040 tx=0x7f816801f630 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e4f79fa9664b5ed9 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:48:46.817 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:46.805+0000 7f8164c00640 10 client.0 ms_handle_connect on v2:172.21.15.32:3300/0 2024-08-03T20:48:46.817 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:46.805+0000 7f8164c00640 1 -- 172.21.15.32:0/2492280333 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f816802c070 con 0x558231365670 2024-08-03T20:48:46.817 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:46.805+0000 7f8189a76740 1 -- 172.21.15.32:0/2492280333 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8168014ba0 con 0x558231365670 2024-08-03T20:48:46.817 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:46.805+0000 7f8164c00640 1 -- 172.21.15.32:0/2492280333 <== mon.0 v2:172.21.15.32:3300/0 2 ==== osd_map(33..33 src has 1..33) ==== 5795+0+0 (secure 0 0 0) 0x7f816801f820 con 0x558231365670 2024-08-03T20:48:46.817 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:46.805+0000 7f8164c00640 1 -- 172.21.15.32:0/2492280333 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f8168037050 con 0x558231365670 2024-08-03T20:48:46.817 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:46.805+0000 7f8189a76740 10 client.5157 fetch_fsmap learned FSMap version 23 2024-08-03T20:48:46.817 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:46.805+0000 7f8189a76740 1 -- 172.21.15.32:0/2492280333 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=2+}) -- 0x55823146a7c0 con 0x558231365670 2024-08-03T20:48:46.817 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:46.806+0000 7f8164c00640 1 -- 172.21.15.32:0/2492280333 <== mon.0 v2:172.21.15.32:3300/0 4 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f8168014ba0 con 0x558231365670 2024-08-03T20:48:46.817 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:46.806+0000 7f8164c00640 1 -- 172.21.15.32:0/2492280333 <== mon.0 v2:172.21.15.32:3300/0 5 ==== fsmap(e 23) ==== 3502+0+0 (secure 0 0 0) 0x7f816803a020 con 0x558231365670 2024-08-03T20:48:46.817 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:46.806+0000 7f8189a76740 10 client.5157 fetch_fsmap finished waiting for FSMap version 23 2024-08-03T20:48:46.818 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:46.806+0000 7f8189a76740 10 client.5157 resolve_mds: resolved name 'a' to daemon mds.a 2024-08-03T20:48:46.818 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:46.806+0000 7f8189a76740 20 client.5157 populate_metadata read hostname 'smithi032' 2024-08-03T20:48:46.818 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:46.806+0000 7f8189a76740 1 --2- 172.21.15.32:0/2492280333 >> [v2:172.21.15.32:6832/211300533,v1:172.21.15.32:6835/211300533] conn(0x558231345130 0x55823176bb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:48:46.818 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:46.806+0000 7f8189a76740 4 client.5157 mds_command: new command op to 5070 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-08-03T20:48:46.818 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:46.806+0000 7f8189a76740 1 -- 172.21.15.32:0/2492280333 --> [v2:172.21.15.32:6832/211300533,v1:172.21.15.32:6835/211300533] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5582316a0fa0 con 0x558231345130 2024-08-03T20:48:46.818 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:46.806+0000 7f817be00640 1 --2- 172.21.15.32:0/2492280333 >> [v2:172.21.15.32:6832/211300533,v1:172.21.15.32:6835/211300533] conn(0x558231345130 0x55823176bb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:48:46.818 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:46.807+0000 7f817be00640 1 --2- 172.21.15.32:0/2492280333 >> [v2:172.21.15.32:6832/211300533,v1:172.21.15.32:6835/211300533] conn(0x558231345130 0x55823176bb10 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5d3d33b145262c0c server_cookie=af45e9be70d4cd8b in_seq=0 out_seq=0 2024-08-03T20:48:46.818 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:46.807+0000 7f8164c00640 10 client.5157 ms_handle_connect on v2:172.21.15.32:6832/211300533 2024-08-03T20:48:46.818 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:46.807+0000 7f8164c00640 1 -- 172.21.15.32:0/2492280333 <== mds.0 v2:172.21.15.32:6832/211300533 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x5582316a0fa0 con 0x558231345130 2024-08-03T20:48:46.818 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:46.807+0000 7f8164c00640 10 client.5157 handle_command_reply: tid=0 2024-08-03T20:48:46.818 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:46.807+0000 7f8189a76740 1 client.5157 shutdown 2024-08-03T20:48:46.818 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:46.808+0000 7f8189a76740 1 -- 172.21.15.32:0/2492280333 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558231365670 msgr2=0x558231783f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:48:46.819 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:46.808+0000 7f8189a76740 1 --2- 172.21.15.32:0/2492280333 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558231365670 0x558231783f30 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f8168022040 tx=0x7f816801f630 comp rx=0 tx=0).stop 2024-08-03T20:48:46.819 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:46.808+0000 7f8189a76740 1 -- 172.21.15.32:0/2492280333 shutdown_connections 2024-08-03T20:48:46.819 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:46.808+0000 7f8189a76740 1 --2- 172.21.15.32:0/2492280333 >> [v2:172.21.15.32:6832/211300533,v1:172.21.15.32:6835/211300533] conn(0x558231345130 0x55823176bb10 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:46.819 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:46.808+0000 7f8189a76740 1 --2- 172.21.15.32:0/2492280333 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x558231386350 0x55823176ce90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:46.819 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:46.808+0000 7f8189a76740 1 --2- 172.21.15.32:0/2492280333 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558231365670 0x558231783f30 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:46.819 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:46.808+0000 7f8189a76740 1 --2- 172.21.15.32:0/2492280333 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55823135d490 0x558231744b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:46.819 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:46.808+0000 7f8189a76740 1 -- 172.21.15.32:0/2492280333 >> 172.21.15.32:0/2492280333 conn(0x55823137a4a0 msgr2=0x558231774800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:48:46.819 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:46.808+0000 7f8164c00640 0 client.5157 ms_handle_reset on v2:172.21.15.32:6832/211300533 2024-08-03T20:48:46.819 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:46.808+0000 7f8189a76740 1 -- 172.21.15.32:0/2492280333 shutdown_connections 2024-08-03T20:48:46.819 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:46.808+0000 7f8189a76740 1 -- 172.21.15.32:0/2492280333 wait complete. 2024-08-03T20:48:46.819 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:46.808+0000 7f8189a76740 20 client.5157 trim_cache size 0 max 16384 2024-08-03T20:48:46.824 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:46.823+0000 7f8189a76740 1 Processor -- start 2024-08-03T20:48:46.824 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:46.823+0000 7f8189a76740 1 -- start start 2024-08-03T20:48:46.827 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:46.824+0000 7f8189a76740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55823135d490 0x558231769200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:48:46.827 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:46.825+0000 7f8189a76740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x558231365670 0x558231769740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:48:46.827 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:46.825+0000 7f8189a76740 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x558231386350 0x558231794380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:48:46.827 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:46.825+0000 7f8189a76740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f817c2010e0 con 0x55823135d490 2024-08-03T20:48:46.827 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:46.825+0000 7f8189a76740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f817c1e6bf0 con 0x558231365670 2024-08-03T20:48:46.827 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:46.825+0000 7f8189a76740 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7f817c1e7240 con 0x558231386350 2024-08-03T20:48:46.827 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:46.825+0000 7f817be00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55823135d490 0x558231769200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:48:46.827 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:46.825+0000 7f817be00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55823135d490 0x558231769200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:45094/0 (socket says 172.21.15.32:45094) 2024-08-03T20:48:46.827 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:46.825+0000 7f817be00640 1 -- 172.21.15.32:0/3661784231 learned_addr learned my addr 172.21.15.32:0/3661784231 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:48:46.827 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:46.825+0000 7f8180a00640 1 --2- 172.21.15.32:0/3661784231 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x558231386350 0x558231794380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:48:46.827 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:46.825+0000 7f817b400640 1 --2- 172.21.15.32:0/3661784231 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x558231365670 0x558231769740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:48:46.828 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:46.825+0000 7f817be00640 1 -- 172.21.15.32:0/3661784231 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x558231386350 msgr2=0x558231794380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:48:46.828 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:46.825+0000 7f817be00640 1 --2- 172.21.15.32:0/3661784231 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x558231386350 0x558231794380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:46.828 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:46.825+0000 7f817be00640 1 -- 172.21.15.32:0/3661784231 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x558231365670 msgr2=0x558231769740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:48:46.828 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:46.825+0000 7f817be00640 1 --2- 172.21.15.32:0/3661784231 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x558231365670 0x558231769740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:46.828 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:46.825+0000 7f817be00640 1 -- 172.21.15.32:0/3661784231 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558231721560 con 0x55823135d490 2024-08-03T20:48:46.828 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:46.826+0000 7f817be00640 1 --2- 172.21.15.32:0/3661784231 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55823135d490 0x558231769200 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f816c07eec0 tx=0x7f816c0b7550 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a276a0c2aa90b38f server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:48:46.829 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:46.826+0000 7f8167400640 1 -- 172.21.15.32:0/3661784231 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f816c0b5070 con 0x55823135d490 2024-08-03T20:48:46.829 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:46.826+0000 7f8167400640 1 -- 172.21.15.32:0/3661784231 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f816c0a1070 con 0x55823135d490 2024-08-03T20:48:46.829 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:46.826+0000 7f8167400640 1 -- 172.21.15.32:0/3661784231 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f816c048040 con 0x55823135d490 2024-08-03T20:48:46.829 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:46.826+0000 7f8189a76740 1 -- 172.21.15.32:0/3661784231 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55823135d490 msgr2=0x558231769200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:48:46.829 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:46.826+0000 7f8189a76740 1 --2- 172.21.15.32:0/3661784231 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55823135d490 0x558231769200 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f816c07eec0 tx=0x7f816c0b7550 comp rx=0 tx=0).stop 2024-08-03T20:48:46.830 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:46.827+0000 7f8189a76740 1 -- 172.21.15.32:0/3661784231 shutdown_connections 2024-08-03T20:48:46.830 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:46.827+0000 7f8189a76740 1 --2- 172.21.15.32:0/3661784231 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x558231365670 0x558231769740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:46.830 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:46.827+0000 7f8189a76740 1 --2- 172.21.15.32:0/3661784231 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x558231386350 0x558231794380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:46.830 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:46.827+0000 7f8189a76740 1 --2- 172.21.15.32:0/3661784231 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55823135d490 0x558231769200 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:46.830 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:46.827+0000 7f8189a76740 1 -- 172.21.15.32:0/3661784231 >> 172.21.15.32:0/3661784231 conn(0x55823137a4a0 msgr2=0x5582317222c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:48:46.830 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:46.827+0000 7f8189a76740 1 -- 172.21.15.32:0/3661784231 shutdown_connections 2024-08-03T20:48:46.830 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:46.827+0000 7f8189a76740 1 -- 172.21.15.32:0/3661784231 wait complete. 2024-08-03T20:48:46.830 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:46.828+0000 7f8189a76740 1 -- start start 2024-08-03T20:48:46.830 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:46.828+0000 7f8189a76740 1 Processor -- start 2024-08-03T20:48:46.831 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:46.829+0000 7f8189a76740 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55823135d490 0x55823174d0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:48:46.831 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:46.829+0000 7f8189a76740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558231365670 0x55823176b860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:48:46.831 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:46.829+0000 7f8189a76740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x558231386350 0x55823176bda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:48:46.831 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:46.829+0000 7f8189a76740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x558231784ca0 con 0x558231365670 2024-08-03T20:48:46.831 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:46.829+0000 7f8189a76740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x558231745110 con 0x558231386350 2024-08-03T20:48:46.831 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:46.829+0000 7f8189a76740 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x558231768380 con 0x55823135d490 2024-08-03T20:48:46.831 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:46.829+0000 7f8189a76740 10 client.0 _ll_register_callbacks cb 0x7f8168022070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-03T20:48:46.831 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:46.830+0000 7f817b400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558231365670 0x55823176b860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:48:46.832 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:46.830+0000 7f817b400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558231365670 0x55823176b860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:45100/0 (socket says 172.21.15.32:45100) 2024-08-03T20:48:46.832 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:46.830+0000 7f817b400640 1 -- 172.21.15.32:0/873233449 learned_addr learned my addr 172.21.15.32:0/873233449 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:48:46.834 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:46.830+0000 7f817be00640 1 --2- 172.21.15.32:0/873233449 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55823135d490 0x55823174d0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:48:46.835 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:46.830+0000 7f8180a00640 1 --2- 172.21.15.32:0/873233449 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x558231386350 0x55823176bda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:48:46.835 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:46.830+0000 7f817b400640 1 -- 172.21.15.32:0/873233449 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55823135d490 msgr2=0x55823174d0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:48:46.835 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:46.830+0000 7f817b400640 1 --2- 172.21.15.32:0/873233449 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55823135d490 0x55823174d0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:46.835 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:46.830+0000 7f817b400640 1 -- 172.21.15.32:0/873233449 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x558231386350 msgr2=0x55823176bda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:48:46.835 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:46.830+0000 7f817b400640 1 --2- 172.21.15.32:0/873233449 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x558231386350 0x55823176bda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:46.835 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:46.830+0000 7f817b400640 1 -- 172.21.15.32:0/873233449 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55823174d610 con 0x558231365670 2024-08-03T20:48:46.835 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:46.830+0000 7f817be00640 1 --2- 172.21.15.32:0/873233449 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55823135d490 0x55823174d0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-03T20:48:46.835 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:46.830+0000 7f8180a00640 1 --2- 172.21.15.32:0/873233449 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x558231386350 0x55823176bda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-03T20:48:46.835 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:46.830+0000 7f817b400640 1 --2- 172.21.15.32:0/873233449 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558231365670 0x55823176b860 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f8168007f40 tx=0x7f8168014a20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c5a8c73300bc08fb server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:48:46.835 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:46.830+0000 7f8164c00640 10 client.0 ms_handle_connect on v2:172.21.15.32:3300/0 2024-08-03T20:48:46.835 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:46.831+0000 7f8164c00640 1 -- 172.21.15.32:0/873233449 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8168040060 con 0x558231365670 2024-08-03T20:48:46.835 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:46.831+0000 7f8164c00640 1 -- 172.21.15.32:0/873233449 <== mon.0 v2:172.21.15.32:3300/0 2 ==== osd_map(33..33 src has 1..33) ==== 5795+0+0 (secure 0 0 0) 0x7f8168007450 con 0x558231365670 2024-08-03T20:48:46.836 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:46.831+0000 7f8189a76740 1 -- 172.21.15.32:0/873233449 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f816c0a1070 con 0x558231365670 2024-08-03T20:48:46.836 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:46.831+0000 7f8164c00640 1 -- 172.21.15.32:0/873233449 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f8168046050 con 0x558231365670 2024-08-03T20:48:46.836 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:46.831+0000 7f8189a76740 10 client.5163 fetch_fsmap learned FSMap version 23 2024-08-03T20:48:46.836 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:46.831+0000 7f8189a76740 1 -- 172.21.15.32:0/873233449 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=2+}) -- 0x55823176af20 con 0x558231365670 2024-08-03T20:48:46.836 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:46.831+0000 7f8164c00640 1 -- 172.21.15.32:0/873233449 <== mon.0 v2:172.21.15.32:3300/0 4 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f816c0a1070 con 0x558231365670 2024-08-03T20:48:46.836 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:46.831+0000 7f8164c00640 1 -- 172.21.15.32:0/873233449 <== mon.0 v2:172.21.15.32:3300/0 5 ==== fsmap(e 23) ==== 3502+0+0 (secure 0 0 0) 0x7f8168049020 con 0x558231365670 2024-08-03T20:48:46.836 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:46.831+0000 7f8189a76740 10 client.5163 fetch_fsmap finished waiting for FSMap version 23 2024-08-03T20:48:46.836 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:46.831+0000 7f8189a76740 10 client.5163 resolve_mds: resolved name 'a' to daemon mds.a 2024-08-03T20:48:46.836 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:46.831+0000 7f8189a76740 20 client.5163 populate_metadata read hostname 'smithi032' 2024-08-03T20:48:46.836 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:46.831+0000 7f8189a76740 1 --2- 172.21.15.32:0/873233449 >> [v2:172.21.15.32:6832/211300533,v1:172.21.15.32:6835/211300533] conn(0x55823172a6e0 0x55823172aaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:48:46.836 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:46.831+0000 7f8189a76740 4 client.5163 mds_command: new command op to 5070 tid=0 multi_id=0 [{"prefix": "damage ls", "format": "json-pretty"}] 2024-08-03T20:48:46.836 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:46.831+0000 7f8189a76740 1 -- 172.21.15.32:0/873233449 --> [v2:172.21.15.32:6832/211300533,v1:172.21.15.32:6835/211300533] -- command(tid 0: {"prefix": "damage ls", "format": "json-pretty"}) -- 0x55823169ca40 con 0x55823172a6e0 2024-08-03T20:48:46.836 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:46.832+0000 7f817be00640 1 --2- 172.21.15.32:0/873233449 >> [v2:172.21.15.32:6832/211300533,v1:172.21.15.32:6835/211300533] conn(0x55823172a6e0 0x55823172aaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:48:46.837 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:46.832+0000 7f817be00640 1 --2- 172.21.15.32:0/873233449 >> [v2:172.21.15.32:6832/211300533,v1:172.21.15.32:6835/211300533] conn(0x55823172a6e0 0x55823172aaf0 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e3b82a0410bfece0 server_cookie=c136767a386203f1 in_seq=0 out_seq=0 2024-08-03T20:48:46.837 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:46.832+0000 7f8164c00640 10 client.5163 ms_handle_connect on v2:172.21.15.32:6832/211300533 2024-08-03T20:48:46.837 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:46.832+0000 7f8164c00640 1 -- 172.21.15.32:0/873233449 <== mds.0 v2:172.21.15.32:6832/211300533 1 ==== command_reply(tid 0: 0 ) ==== 8+0+236 (crc 0 0 0) 0x55823169ca40 con 0x55823172a6e0 2024-08-03T20:48:46.837 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:46.832+0000 7f8164c00640 10 client.5163 handle_command_reply: tid=0 2024-08-03T20:48:46.837 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:46.832+0000 7f8189a76740 1 client.5163 shutdown 2024-08-03T20:48:46.837 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:46.833+0000 7f8189a76740 1 -- 172.21.15.32:0/873233449 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558231365670 msgr2=0x55823176b860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:48:46.837 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:46.833+0000 7f8189a76740 1 --2- 172.21.15.32:0/873233449 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558231365670 0x55823176b860 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f8168007f40 tx=0x7f8168014a20 comp rx=0 tx=0).stop 2024-08-03T20:48:46.837 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:46.833+0000 7f8189a76740 1 -- 172.21.15.32:0/873233449 shutdown_connections 2024-08-03T20:48:46.837 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:46.833+0000 7f8189a76740 1 --2- 172.21.15.32:0/873233449 >> [v2:172.21.15.32:6832/211300533,v1:172.21.15.32:6835/211300533] conn(0x55823172a6e0 0x55823172aaf0 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:46.837 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:46.833+0000 7f8189a76740 1 --2- 172.21.15.32:0/873233449 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x558231386350 0x55823176bda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:46.837 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:46.833+0000 7f8189a76740 1 --2- 172.21.15.32:0/873233449 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558231365670 0x55823176b860 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:46.837 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:46.833+0000 7f8189a76740 1 --2- 172.21.15.32:0/873233449 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55823135d490 0x55823174d0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:46.837 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:46.833+0000 7f8164c00640 0 client.5163 ms_handle_reset on v2:172.21.15.32:6832/211300533 2024-08-03T20:48:46.837 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:46.833+0000 7f8189a76740 1 -- 172.21.15.32:0/873233449 >> 172.21.15.32:0/873233449 conn(0x55823137a4a0 msgr2=0x558231721fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:48:46.838 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:46.833+0000 7f8189a76740 1 -- 172.21.15.32:0/873233449 shutdown_connections 2024-08-03T20:48:46.838 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:46.833+0000 7f8189a76740 1 -- 172.21.15.32:0/873233449 wait complete. 2024-08-03T20:48:46.838 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:46.833+0000 7f8189a76740 20 client.5163 trim_cache size 0 max 16384 2024-08-03T20:48:46.838 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-03T20:48:46.838 INFO:teuthology.orchestra.run.smithi032.stdout:[ 2024-08-03T20:48:46.838 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-03T20:48:46.838 INFO:teuthology.orchestra.run.smithi032.stdout: "damage_type": "dentry", 2024-08-03T20:48:46.838 INFO:teuthology.orchestra.run.smithi032.stdout: "id": 353632146, 2024-08-03T20:48:46.839 INFO:teuthology.orchestra.run.smithi032.stdout: "ino": 1099511627776, 2024-08-03T20:48:46.839 INFO:teuthology.orchestra.run.smithi032.stdout: "frag": "*", 2024-08-03T20:48:46.839 INFO:teuthology.orchestra.run.smithi032.stdout: "dname": "file_to_be_damaged", 2024-08-03T20:48:46.839 INFO:teuthology.orchestra.run.smithi032.stdout: "snap_id": "head", 2024-08-03T20:48:46.839 INFO:teuthology.orchestra.run.smithi032.stdout: "path": "/subdir/file_to_be_damaged" 2024-08-03T20:48:46.839 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-03T20:48:46.839 INFO:teuthology.orchestra.run.smithi032.stdout:] 2024-08-03T20:48:46.840 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:46.834+0000 7f8182a00640 1 -- 172.21.15.32:0/707174188 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f815c077010 msgr2=0x7f815c079520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:48:46.840 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:46.834+0000 7f8182a00640 1 --2- 172.21.15.32:0/707174188 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f815c077010 0x7f815c079520 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f817c159bc0 tx=0x7f8168009040 comp rx=0 tx=0).stop 2024-08-03T20:48:46.840 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:46.834+0000 7f8182a00640 1 -- 172.21.15.32:0/707174188 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f817c0cccf0 msgr2=0x7f817c13ba40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:48:46.840 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:46.834+0000 7f8182a00640 1 --2- 172.21.15.32:0/707174188 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f817c0cccf0 0x7f817c13ba40 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f816c006010 tx=0x7f816c0039e0 comp rx=0 tx=0).stop 2024-08-03T20:48:46.840 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:46.834+0000 7f8182a00640 1 -- 172.21.15.32:0/707174188 shutdown_connections 2024-08-03T20:48:46.840 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:46.834+0000 7f8182a00640 1 --2- 172.21.15.32:0/707174188 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f815c077010 0x7f815c079520 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:46.840 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:46.834+0000 7f8182a00640 1 --2- 172.21.15.32:0/707174188 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f817c0ced30 0x7f817c1e8080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:46.841 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:46.834+0000 7f8182a00640 1 --2- 172.21.15.32:0/707174188 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f817c145c50 0x7f817c1e85c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:46.841 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:46.834+0000 7f8182a00640 1 --2- 172.21.15.32:0/707174188 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f817c0cccf0 0x7f817c13ba40 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:46.841 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:46.834+0000 7f8182a00640 1 -- 172.21.15.32:0/707174188 >> 172.21.15.32:0/707174188 conn(0x7f817c0b1bd0 msgr2=0x7f817c15b890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:48:46.841 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:46.834+0000 7f8182a00640 1 -- 172.21.15.32:0/707174188 shutdown_connections 2024-08-03T20:48:46.841 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:46.834+0000 7f8182a00640 1 -- 172.21.15.32:0/707174188 wait complete. 2024-08-03T20:48:46.857 DEBUG:teuthology.orchestra.run.smithi032:> stdin-killer -- python3 -c ' 2024-08-03T20:48:46.857 DEBUG:teuthology.orchestra.run.smithi032:> import sys 2024-08-03T20:48:46.857 DEBUG:teuthology.orchestra.run.smithi032:> import errno 2024-08-03T20:48:46.857 DEBUG:teuthology.orchestra.run.smithi032:> 2024-08-03T20:48:46.857 DEBUG:teuthology.orchestra.run.smithi032:> try: 2024-08-03T20:48:46.858 DEBUG:teuthology.orchestra.run.smithi032:> f = open("/home/ubuntu/cephtest/mnt.0/subdir/file_to_be_damaged", "w") 2024-08-03T20:48:46.858 DEBUG:teuthology.orchestra.run.smithi032:> f.close() 2024-08-03T20:48:46.858 DEBUG:teuthology.orchestra.run.smithi032:> except IOError as e: 2024-08-03T20:48:46.858 DEBUG:teuthology.orchestra.run.smithi032:> sys.exit(errno.EIO) 2024-08-03T20:48:46.858 DEBUG:teuthology.orchestra.run.smithi032:> ' 2024-08-03T20:48:46.963 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:46 stdin-killer INFO: command exited with status 5: exiting normally with same code! 2024-08-03T20:48:46.971 DEBUG:teuthology.orchestra.run:got remote process result: 5 2024-08-03T20:48:46.972 DEBUG:teuthology.orchestra.run.smithi032:> stdin-killer -- python3 -c ' 2024-08-03T20:48:46.972 DEBUG:teuthology.orchestra.run.smithi032:> import os 2024-08-03T20:48:46.972 DEBUG:teuthology.orchestra.run.smithi032:> import stat 2024-08-03T20:48:46.972 DEBUG:teuthology.orchestra.run.smithi032:> import json 2024-08-03T20:48:46.972 DEBUG:teuthology.orchestra.run.smithi032:> import sys 2024-08-03T20:48:46.972 DEBUG:teuthology.orchestra.run.smithi032:> 2024-08-03T20:48:46.972 DEBUG:teuthology.orchestra.run.smithi032:> try: 2024-08-03T20:48:46.972 DEBUG:teuthology.orchestra.run.smithi032:> s = os.stat('"'"'/home/ubuntu/cephtest/mnt.0/subdir/file_to_be_damaged'"'"') 2024-08-03T20:48:46.973 DEBUG:teuthology.orchestra.run.smithi032:> except OSError as e: 2024-08-03T20:48:46.973 DEBUG:teuthology.orchestra.run.smithi032:> sys.exit(e.errno) 2024-08-03T20:48:46.973 DEBUG:teuthology.orchestra.run.smithi032:> 2024-08-03T20:48:46.973 DEBUG:teuthology.orchestra.run.smithi032:> attrs = ["st_mode", "st_ino", "st_dev", "st_nlink", "st_uid", "st_gid", "st_size", "st_atime", "st_mtime", "st_ctime"] 2024-08-03T20:48:46.973 DEBUG:teuthology.orchestra.run.smithi032:> print(json.dumps( 2024-08-03T20:48:46.973 DEBUG:teuthology.orchestra.run.smithi032:> dict([(a, getattr(s, a)) for a in attrs]), 2024-08-03T20:48:46.973 DEBUG:teuthology.orchestra.run.smithi032:> indent=2)) 2024-08-03T20:48:46.973 DEBUG:teuthology.orchestra.run.smithi032:> ' 2024-08-03T20:48:47.120 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:47 stdin-killer INFO: command exited with status 5: exiting normally with same code! 2024-08-03T20:48:47.128 DEBUG:teuthology.orchestra.run:got remote process result: 5 2024-08-03T20:48:47.129 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-08-03T20:48:47.129 DEBUG:teuthology.orchestra.run.smithi032:> (cd /home/ubuntu/cephtest/mnt.0 && exec getfattr --only-values -n ceph.dir.files ./subdir) 2024-08-03T20:48:47.206 INFO:teuthology.orchestra.run.smithi032.stdout:2 2024-08-03T20:48:47.206 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-08-03T20:48:47.206 DEBUG:teuthology.orchestra.run.smithi032:> dd if=/proc/self/mounts of=/dev/stdout 2024-08-03T20:48:47.281 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-08-03T20:48:47.281 DEBUG:teuthology.orchestra.run.smithi032:> dd if=/proc/self/mounts of=/dev/stdout 2024-08-03T20:48:47.357 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-08-03T20:48:47.628 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:47.626+0000 7fb09d200640 1 -- 172.21.15.32:0/4246296561 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb0980b4de0 msgr2=0x7fb0981515c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:48:47.628 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:47.626+0000 7fb09d200640 1 --2- 172.21.15.32:0/4246296561 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb0980b4de0 0x7fb0981515c0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fb088009830 tx=0x7fb08802a480 comp rx=0 tx=0).stop 2024-08-03T20:48:47.628 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:47.626+0000 7fb09d200640 1 -- 172.21.15.32:0/4246296561 shutdown_connections 2024-08-03T20:48:47.628 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:47.626+0000 7fb09d200640 1 --2- 172.21.15.32:0/4246296561 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fb09815e880 0x7fb098073800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:47.628 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:47.626+0000 7fb09d200640 1 --2- 172.21.15.32:0/4246296561 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb0980b4de0 0x7fb0981515c0 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:47.629 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:47.626+0000 7fb09d200640 1 -- 172.21.15.32:0/4246296561 >> 172.21.15.32:0/4246296561 conn(0x7fb0980b1bd0 msgr2=0x7fb0980b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:48:47.629 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:47.626+0000 7fb09d200640 1 -- 172.21.15.32:0/4246296561 shutdown_connections 2024-08-03T20:48:47.629 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:47.626+0000 7fb09d200640 1 -- 172.21.15.32:0/4246296561 wait complete. 2024-08-03T20:48:47.630 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:47.629+0000 7fb09d200640 1 Processor -- start 2024-08-03T20:48:47.630 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:47.629+0000 7fb09d200640 1 -- start start 2024-08-03T20:48:47.632 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:47.630+0000 7fb09d200640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fb0980733f0 0x7fb098150340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:48:47.632 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:47.630+0000 7fb09d200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb0980b4de0 0x7fb0980ce520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:48:47.632 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:47.630+0000 7fb09d200640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb0980cea60 0x7fb098151bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:48:47.632 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:47.630+0000 7fb09d200640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fb09806e930 con 0x7fb0980b4de0 2024-08-03T20:48:47.632 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:47.630+0000 7fb09d200640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7fb09806eab0 con 0x7fb0980cea60 2024-08-03T20:48:47.632 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:47.630+0000 7fb09d200640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7fb09806ec30 con 0x7fb0980733f0 2024-08-03T20:48:47.632 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:47.631+0000 7fb095400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb0980b4de0 0x7fb0980ce520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:48:47.632 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:47.631+0000 7fb095400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb0980b4de0 0x7fb0980ce520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:45112/0 (socket says 172.21.15.32:45112) 2024-08-03T20:48:47.633 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:47.631+0000 7fb095400640 1 -- 172.21.15.32:0/188216798 learned_addr learned my addr 172.21.15.32:0/188216798 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:48:47.633 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:47.631+0000 7fb096800640 1 --2- 172.21.15.32:0/188216798 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb0980cea60 0x7fb098151bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:48:47.633 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:47.631+0000 7fb095e00640 1 --2- 172.21.15.32:0/188216798 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fb0980733f0 0x7fb098150340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:48:47.634 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:47.631+0000 7fb095400640 1 -- 172.21.15.32:0/188216798 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fb0980733f0 msgr2=0x7fb098150340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:48:47.634 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:47.631+0000 7fb095400640 1 --2- 172.21.15.32:0/188216798 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fb0980733f0 0x7fb098150340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:47.634 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:47.631+0000 7fb095400640 1 -- 172.21.15.32:0/188216798 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb0980cea60 msgr2=0x7fb098151bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:48:47.634 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:47.631+0000 7fb095400640 1 --2- 172.21.15.32:0/188216798 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb0980cea60 0x7fb098151bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:47.634 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:47.631+0000 7fb095400640 1 -- 172.21.15.32:0/188216798 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb0980cee70 con 0x7fb0980b4de0 2024-08-03T20:48:47.634 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:47.631+0000 7fb095400640 1 --2- 172.21.15.32:0/188216798 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb0980b4de0 0x7fb0980ce520 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7fb08400c760 tx=0x7fb08400cb60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:48:47.634 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:47.632+0000 7fb08e000640 1 -- 172.21.15.32:0/188216798 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb084006890 con 0x7fb0980b4de0 2024-08-03T20:48:47.634 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:47.632+0000 7fb08e000640 1 -- 172.21.15.32:0/188216798 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fb084006a30 con 0x7fb0980b4de0 2024-08-03T20:48:47.634 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:47.632+0000 7fb08e000640 1 -- 172.21.15.32:0/188216798 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb084007030 con 0x7fb0980b4de0 2024-08-03T20:48:47.635 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:47.632+0000 7fb09d200640 1 -- 172.21.15.32:0/188216798 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb0980ccfd0 con 0x7fb0980b4de0 2024-08-03T20:48:47.635 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:47.632+0000 7fb09d200640 1 -- 172.21.15.32:0/188216798 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb098149920 con 0x7fb0980b4de0 2024-08-03T20:48:47.635 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:47.633+0000 7fb09d200640 1 -- 172.21.15.32:0/188216798 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb0981515c0 con 0x7fb0980b4de0 2024-08-03T20:48:47.635 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:47.634+0000 7fb08e000640 1 -- 172.21.15.32:0/188216798 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7fb0840071d0 con 0x7fb0980b4de0 2024-08-03T20:48:47.636 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:47.635+0000 7fb08e000640 1 --2- 172.21.15.32:0/188216798 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fb078077010 0x7fb078079520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:48:47.636 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:47.635+0000 7fb08e000640 1 -- 172.21.15.32:0/188216798 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(33..33 src has 1..33) ==== 5795+0+0 (secure 0 0 0) 0x7fb084098330 con 0x7fb0980b4de0 2024-08-03T20:48:47.637 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:47.635+0000 7fb095e00640 1 --2- 172.21.15.32:0/188216798 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fb078077010 0x7fb078079520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:48:47.637 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:47.636+0000 7fb095e00640 1 --2- 172.21.15.32:0/188216798 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fb078077010 0x7fb078079520 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fb088034000 tx=0x7fb088004730 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:48:47.641 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:47.639+0000 7fb08e000640 1 -- 172.21.15.32:0/188216798 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7fb0980ccfd0 con 0x7fb0980b4de0 2024-08-03T20:48:47.858 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:47.857+0000 7fb09d200640 1 -- 172.21.15.32:0/188216798 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7fb09814cbf0 con 0x7fb0980b4de0 2024-08-03T20:48:47.859 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:47.857+0000 7fb08e000640 1 -- 172.21.15.32:0/188216798 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 2 entries v33) ==== 80+0+122 (secure 0 0 0) 0x7fb0980b51f0 con 0x7fb0980b4de0 2024-08-03T20:48:47.859 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6835/1186001746 2024-08-04T20:48:38.076677+0000 2024-08-03T20:48:47.859 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6832/1186001746 2024-08-04T20:48:38.076677+0000 2024-08-03T20:48:47.859 INFO:teuthology.orchestra.run.smithi032.stderr:listed 2 entries 2024-08-03T20:48:47.862 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:47.860+0000 7fb09d200640 1 -- 172.21.15.32:0/188216798 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fb078077010 msgr2=0x7fb078079520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:48:47.862 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:47.860+0000 7fb09d200640 1 --2- 172.21.15.32:0/188216798 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fb078077010 0x7fb078079520 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fb088034000 tx=0x7fb088004730 comp rx=0 tx=0).stop 2024-08-03T20:48:47.862 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:47.860+0000 7fb09d200640 1 -- 172.21.15.32:0/188216798 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb0980b4de0 msgr2=0x7fb0980ce520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:48:47.862 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:47.860+0000 7fb09d200640 1 --2- 172.21.15.32:0/188216798 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb0980b4de0 0x7fb0980ce520 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7fb08400c760 tx=0x7fb08400cb60 comp rx=0 tx=0).stop 2024-08-03T20:48:47.862 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:47.860+0000 7fb09d200640 1 -- 172.21.15.32:0/188216798 shutdown_connections 2024-08-03T20:48:47.862 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:47.860+0000 7fb09d200640 1 --2- 172.21.15.32:0/188216798 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fb078077010 0x7fb078079520 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:47.862 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:47.860+0000 7fb09d200640 1 --2- 172.21.15.32:0/188216798 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb0980cea60 0x7fb098151bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:47.862 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:47.860+0000 7fb09d200640 1 --2- 172.21.15.32:0/188216798 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb0980b4de0 0x7fb0980ce520 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:47.862 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:47.860+0000 7fb09d200640 1 --2- 172.21.15.32:0/188216798 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fb0980733f0 0x7fb098150340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:47.863 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:47.860+0000 7fb09d200640 1 -- 172.21.15.32:0/188216798 >> 172.21.15.32:0/188216798 conn(0x7fb0980b1bd0 msgr2=0x7fb09815bc00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:48:47.863 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:47.860+0000 7fb09d200640 1 -- 172.21.15.32:0/188216798 shutdown_connections 2024-08-03T20:48:47.863 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:47.860+0000 7fb09d200640 1 -- 172.21.15.32:0/188216798 wait complete. 2024-08-03T20:48:47.887 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-08-03T20:48:47.887 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-08-03T20:48:47.887 DEBUG:teuthology.orchestra.run.smithi032:> sudo umount /home/ubuntu/cephtest/mnt.0 2024-08-03T20:48:47.950 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi032.front.sepia.ceph.com 2024-08-03T20:48:47.950 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-08-03T20:48:47.950 DEBUG:teuthology.orchestra.run.smithi032:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-08-03T20:48:47.989 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-08-03T20:48:47.989 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.2:0 scrub start /subdir repair 2024-08-03T20:48:48.263 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:48.260+0000 7fa952200640 1 -- 172.21.15.32:0/3661706548 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fa94c160520 msgr2=0x7fa94c1671a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:48:48.263 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:48.260+0000 7fa952200640 1 --2- 172.21.15.32:0/3661706548 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fa94c160520 0x7fa94c1671a0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fa944009be0 tx=0x7fa94402a8b0 comp rx=0 tx=0).stop 2024-08-03T20:48:48.264 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:48.261+0000 7fa952200640 1 -- 172.21.15.32:0/3661706548 shutdown_connections 2024-08-03T20:48:48.264 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:48.261+0000 7fa952200640 1 --2- 172.21.15.32:0/3661706548 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fa94c160520 0x7fa94c1671a0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:48.264 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:48.261+0000 7fa952200640 1 -- 172.21.15.32:0/3661706548 >> 172.21.15.32:0/3661706548 conn(0x7fa94c0cfd80 msgr2=0x7fa94c0d0190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:48:48.264 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:48.261+0000 7fa952200640 1 -- 172.21.15.32:0/3661706548 shutdown_connections 2024-08-03T20:48:48.264 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:48.261+0000 7fa952200640 1 -- 172.21.15.32:0/3661706548 wait complete. 2024-08-03T20:48:48.265 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:48.263+0000 7fa952200640 1 Processor -- start 2024-08-03T20:48:48.265 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:48.263+0000 7fa952200640 1 -- start start 2024-08-03T20:48:48.269 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:48.264+0000 7fa952200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa94c159ff0 0x7fa94c14e960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:48:48.269 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:48.264+0000 7fa952200640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fa94c1609d0 0x7fa94c0b0f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:48:48.269 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:48.264+0000 7fa952200640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa94c0b46e0 0x7fa94c0b4af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:48:48.270 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:48.264+0000 7fa952200640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fa94c167ac0 con 0x7fa94c159ff0 2024-08-03T20:48:48.270 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:48.264+0000 7fa952200640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7fa94c167940 con 0x7fa94c0b46e0 2024-08-03T20:48:48.270 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:48.265+0000 7fa952200640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7fa94c167c40 con 0x7fa94c1609d0 2024-08-03T20:48:48.270 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:48.265+0000 7fa950e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa94c159ff0 0x7fa94c14e960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:48:48.270 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:48.265+0000 7fa950e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa94c159ff0 0x7fa94c14e960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:45114/0 (socket says 172.21.15.32:45114) 2024-08-03T20:48:48.271 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:48.265+0000 7fa950e00640 1 -- 172.21.15.32:0/3545759791 learned_addr learned my addr 172.21.15.32:0/3545759791 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:48:48.271 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:48.265+0000 7fa951800640 1 --2- 172.21.15.32:0/3545759791 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa94c0b46e0 0x7fa94c0b4af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:48:48.271 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:48.265+0000 7fa94be00640 1 --2- 172.21.15.32:0/3545759791 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fa94c1609d0 0x7fa94c0b0f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:48:48.271 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:48.265+0000 7fa950e00640 1 -- 172.21.15.32:0/3545759791 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fa94c1609d0 msgr2=0x7fa94c0b0f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:48:48.271 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:48.265+0000 7fa950e00640 1 --2- 172.21.15.32:0/3545759791 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fa94c1609d0 0x7fa94c0b0f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:48.272 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:48.265+0000 7fa950e00640 1 -- 172.21.15.32:0/3545759791 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa94c0b46e0 msgr2=0x7fa94c0b4af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:48:48.272 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:48.265+0000 7fa950e00640 1 --2- 172.21.15.32:0/3545759791 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa94c0b46e0 0x7fa94c0b4af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:48.272 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:48.265+0000 7fa950e00640 1 -- 172.21.15.32:0/3545759791 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa94c0b5030 con 0x7fa94c159ff0 2024-08-03T20:48:48.272 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:48.265+0000 7fa94be00640 1 --2- 172.21.15.32:0/3545759791 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fa94c1609d0 0x7fa94c0b0f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-03T20:48:48.272 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:48.266+0000 7fa950e00640 1 --2- 172.21.15.32:0/3545759791 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa94c159ff0 0x7fa94c14e960 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7fa94000c500 tx=0x7fa94000c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:48:48.273 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:48.266+0000 7fa948c00640 1 -- 172.21.15.32:0/3545759791 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa940016020 con 0x7fa94c159ff0 2024-08-03T20:48:48.273 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:48.266+0000 7fa948c00640 1 -- 172.21.15.32:0/3545759791 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fa940012920 con 0x7fa94c159ff0 2024-08-03T20:48:48.273 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:48.266+0000 7fa948c00640 1 -- 172.21.15.32:0/3545759791 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa940012d30 con 0x7fa94c159ff0 2024-08-03T20:48:48.273 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:48.266+0000 7fa952200640 1 -- 172.21.15.32:0/3545759791 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa94c0b1460 con 0x7fa94c159ff0 2024-08-03T20:48:48.273 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:48.266+0000 7fa952200640 1 -- 172.21.15.32:0/3545759791 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa94c153580 con 0x7fa94c159ff0 2024-08-03T20:48:48.281 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:48.269+0000 7fa948c00640 1 -- 172.21.15.32:0/3545759791 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7fa940019020 con 0x7fa94c159ff0 2024-08-03T20:48:48.281 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:48.270+0000 7fa948c00640 1 --2- 172.21.15.32:0/3545759791 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fa92c077040 0x7fa92c079550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:48:48.281 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:48.270+0000 7fa948c00640 1 -- 172.21.15.32:0/3545759791 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(33..33 src has 1..33) ==== 5795+0+0 (secure 0 0 0) 0x7fa940098ac0 con 0x7fa94c159ff0 2024-08-03T20:48:48.281 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:48.270+0000 7fa95a783740 1 Processor -- start 2024-08-03T20:48:48.281 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:48.270+0000 7fa95a783740 1 -- start start 2024-08-03T20:48:48.281 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:48.270+0000 7fa94be00640 1 --2- 172.21.15.32:0/3545759791 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fa92c077040 0x7fa92c079550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:48:48.281 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:48.271+0000 7fa94be00640 1 --2- 172.21.15.32:0/3545759791 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fa92c077040 0x7fa92c079550 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fa938007fd0 tx=0x7fa938009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:48:48.282 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:48.272+0000 7fa95a783740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55dd0342b5b0 0x55dd03816a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:48:48.282 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:48.272+0000 7fa95a783740 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55dd034026f0 0x55dd037eab90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:48:48.282 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:48.272+0000 7fa95a783740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55dd0340a8d0 0x55dd037f7a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:48:48.282 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:48.272+0000 7fa95a783740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55dd037442a0 con 0x55dd0342b5b0 2024-08-03T20:48:48.282 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:48.272+0000 7fa95a783740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x55dd037337a0 con 0x55dd0340a8d0 2024-08-03T20:48:48.282 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:48.272+0000 7fa95a783740 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x55dd037377e0 con 0x55dd034026f0 2024-08-03T20:48:48.282 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:48.272+0000 7fa950e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55dd0342b5b0 0x55dd03816a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:48:48.282 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:48.272+0000 7fa950e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55dd0342b5b0 0x55dd03816a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:45120/0 (socket says 172.21.15.32:45120) 2024-08-03T20:48:48.282 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:48.272+0000 7fa950e00640 1 -- 172.21.15.32:0/73282257 learned_addr learned my addr 172.21.15.32:0/73282257 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:48:48.282 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:48.272+0000 7fa951800640 1 --2- 172.21.15.32:0/73282257 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55dd0340a8d0 0x55dd037f7a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:48:48.282 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:48.272+0000 7fa94be00640 1 --2- 172.21.15.32:0/73282257 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55dd034026f0 0x55dd037eab90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:48:48.283 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:48.272+0000 7fa950e00640 1 -- 172.21.15.32:0/73282257 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55dd034026f0 msgr2=0x55dd037eab90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:48:48.283 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:48.272+0000 7fa950e00640 1 --2- 172.21.15.32:0/73282257 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55dd034026f0 0x55dd037eab90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:48.283 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:48.272+0000 7fa950e00640 1 -- 172.21.15.32:0/73282257 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55dd0340a8d0 msgr2=0x55dd037f7a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:48:48.283 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:48.272+0000 7fa950e00640 1 --2- 172.21.15.32:0/73282257 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55dd0340a8d0 0x55dd037f7a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:48.283 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:48.272+0000 7fa950e00640 1 -- 172.21.15.32:0/73282257 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55dd036fce80 con 0x55dd0342b5b0 2024-08-03T20:48:48.283 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:48.273+0000 7fa950e00640 1 --2- 172.21.15.32:0/73282257 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55dd0342b5b0 0x55dd03816a00 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7fa94005dac0 tx=0x7fa940060fe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=47fbbc001ff9a86f server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:48:48.283 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:48.273+0000 7fa93d600640 1 -- 172.21.15.32:0/73282257 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa940065ad0 con 0x55dd0342b5b0 2024-08-03T20:48:48.283 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:48.273+0000 7fa93d600640 1 -- 172.21.15.32:0/73282257 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fa94006a4f0 con 0x55dd0342b5b0 2024-08-03T20:48:48.283 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:48.273+0000 7fa93d600640 1 -- 172.21.15.32:0/73282257 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa940060990 con 0x55dd0342b5b0 2024-08-03T20:48:48.283 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:48.274+0000 7fa95a783740 1 -- 172.21.15.32:0/73282257 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55dd0342b5b0 msgr2=0x55dd03816a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:48:48.283 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:48.274+0000 7fa95a783740 1 --2- 172.21.15.32:0/73282257 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55dd0342b5b0 0x55dd03816a00 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7fa94005dac0 tx=0x7fa940060fe0 comp rx=0 tx=0).stop 2024-08-03T20:48:48.283 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:48.274+0000 7fa95a783740 1 -- 172.21.15.32:0/73282257 shutdown_connections 2024-08-03T20:48:48.283 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:48.274+0000 7fa95a783740 1 --2- 172.21.15.32:0/73282257 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55dd0340a8d0 0x55dd037f7a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:48.284 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:48.274+0000 7fa95a783740 1 --2- 172.21.15.32:0/73282257 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55dd034026f0 0x55dd037eab90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:48.284 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:48.274+0000 7fa95a783740 1 --2- 172.21.15.32:0/73282257 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55dd0342b5b0 0x55dd03816a00 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:48.284 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:48.274+0000 7fa95a783740 1 -- 172.21.15.32:0/73282257 >> 172.21.15.32:0/73282257 conn(0x55dd0341f700 msgr2=0x55dd037d0c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:48:48.284 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:48.274+0000 7fa95a783740 1 -- 172.21.15.32:0/73282257 shutdown_connections 2024-08-03T20:48:48.284 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:48.274+0000 7fa95a783740 1 -- 172.21.15.32:0/73282257 wait complete. 2024-08-03T20:48:48.284 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:48.275+0000 7fa95a783740 1 -- start start 2024-08-03T20:48:48.284 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:48.275+0000 7fa95a783740 1 Processor -- start 2024-08-03T20:48:48.284 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:48.277+0000 7fa95a783740 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55dd034026f0 0x55dd037e9550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:48:48.284 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:48.277+0000 7fa95a783740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55dd0340a8d0 0x55dd037fa1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:48:48.284 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:48.277+0000 7fa95a783740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55dd0342b5b0 0x55dd038122f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:48:48.284 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:48.277+0000 7fa95a783740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55dd0366c000 con 0x55dd0340a8d0 2024-08-03T20:48:48.284 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:48.277+0000 7fa95a783740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x55dd036cdf10 con 0x55dd0342b5b0 2024-08-03T20:48:48.284 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:48.277+0000 7fa95a783740 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x55dd037cd6f0 con 0x55dd034026f0 2024-08-03T20:48:48.285 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:48.277+0000 7fa95a783740 10 client.0 _ll_register_callbacks cb 0x55dd036b9100 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-03T20:48:48.285 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:48.277+0000 7fa94be00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55dd0340a8d0 0x55dd037fa1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:48:48.285 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:48.277+0000 7fa94be00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55dd0340a8d0 0x55dd037fa1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:45134/0 (socket says 172.21.15.32:45134) 2024-08-03T20:48:48.285 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:48.277+0000 7fa94be00640 1 -- 172.21.15.32:0/3508210073 learned_addr learned my addr 172.21.15.32:0/3508210073 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:48:48.285 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:48.277+0000 7fa950e00640 1 --2- 172.21.15.32:0/3508210073 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55dd034026f0 0x55dd037e9550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:48:48.285 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:48.277+0000 7fa951800640 1 --2- 172.21.15.32:0/3508210073 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55dd0342b5b0 0x55dd038122f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:48:48.285 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:48.277+0000 7fa94be00640 1 -- 172.21.15.32:0/3508210073 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55dd034026f0 msgr2=0x55dd037e9550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:48:48.285 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:48.277+0000 7fa94be00640 1 --2- 172.21.15.32:0/3508210073 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55dd034026f0 0x55dd037e9550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:48.285 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:48.277+0000 7fa94be00640 1 -- 172.21.15.32:0/3508210073 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55dd0342b5b0 msgr2=0x55dd038122f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:48:48.285 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:48.277+0000 7fa94be00640 1 --2- 172.21.15.32:0/3508210073 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55dd0342b5b0 0x55dd038122f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:48.285 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:48.277+0000 7fa94be00640 1 -- 172.21.15.32:0/3508210073 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55dd0350fa20 con 0x55dd0340a8d0 2024-08-03T20:48:48.285 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:48.277+0000 7fa950e00640 1 --2- 172.21.15.32:0/3508210073 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55dd034026f0 0x55dd037e9550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-03T20:48:48.285 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:48.277+0000 7fa951800640 1 --2- 172.21.15.32:0/3508210073 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55dd0342b5b0 0x55dd038122f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-03T20:48:48.285 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:48.278+0000 7fa94be00640 1 --2- 172.21.15.32:0/3508210073 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55dd0340a8d0 0x55dd037fa1d0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7fa93800e9f0 tx=0x7fa938000fe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a650decdf5668956 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:48:48.285 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:48.278+0000 7fa932000640 10 client.0 ms_handle_connect on v2:172.21.15.32:3300/0 2024-08-03T20:48:48.286 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:48.278+0000 7fa932000640 1 -- 172.21.15.32:0/3508210073 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa938027060 con 0x55dd0340a8d0 2024-08-03T20:48:48.286 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:48.278+0000 7fa932000640 1 -- 172.21.15.32:0/3508210073 <== mon.0 v2:172.21.15.32:3300/0 2 ==== osd_map(33..33 src has 1..33) ==== 5795+0+0 (secure 0 0 0) 0x7fa938010040 con 0x55dd0340a8d0 2024-08-03T20:48:48.286 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:48.278+0000 7fa95a783740 1 -- 172.21.15.32:0/3508210073 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa94006a4f0 con 0x55dd0340a8d0 2024-08-03T20:48:48.286 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:48.278+0000 7fa932000640 1 -- 172.21.15.32:0/3508210073 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7fa938005480 con 0x55dd0340a8d0 2024-08-03T20:48:48.286 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:48.278+0000 7fa95a783740 10 client.5181 fetch_fsmap learned FSMap version 23 2024-08-03T20:48:48.286 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:48.278+0000 7fa95a783740 1 -- 172.21.15.32:0/3508210073 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=2+}) -- 0x55dd03496b90 con 0x55dd0340a8d0 2024-08-03T20:48:48.286 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:48.278+0000 7fa932000640 1 -- 172.21.15.32:0/3508210073 <== mon.0 v2:172.21.15.32:3300/0 4 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fa94006a4f0 con 0x55dd0340a8d0 2024-08-03T20:48:48.286 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:48.279+0000 7fa932000640 1 -- 172.21.15.32:0/3508210073 <== mon.0 v2:172.21.15.32:3300/0 5 ==== fsmap(e 23) ==== 3502+0+0 (secure 0 0 0) 0x7fa938013420 con 0x55dd0340a8d0 2024-08-03T20:48:48.286 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:48.279+0000 7fa95a783740 10 client.5181 fetch_fsmap finished waiting for FSMap version 23 2024-08-03T20:48:48.286 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:48.279+0000 7fa95a783740 10 client.5181 resolve_mds: resolved 2:0 to role '2:0' aka daemon mds.a 2024-08-03T20:48:48.286 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:48.279+0000 7fa95a783740 20 client.5181 populate_metadata read hostname 'smithi032' 2024-08-03T20:48:48.287 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:48.279+0000 7fa95a783740 1 --2- 172.21.15.32:0/3508210073 >> [v2:172.21.15.32:6832/211300533,v1:172.21.15.32:6835/211300533] conn(0x55dd033ea180 0x55dd03812d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:48:48.287 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:48.279+0000 7fa95a783740 4 client.5181 mds_command: new command op to 5070 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-08-03T20:48:48.287 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:48.279+0000 7fa95a783740 1 -- 172.21.15.32:0/3508210073 --> [v2:172.21.15.32:6832/211300533,v1:172.21.15.32:6835/211300533] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55dd037461f0 con 0x55dd033ea180 2024-08-03T20:48:48.287 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:48.279+0000 7fa950e00640 1 --2- 172.21.15.32:0/3508210073 >> [v2:172.21.15.32:6832/211300533,v1:172.21.15.32:6835/211300533] conn(0x55dd033ea180 0x55dd03812d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:48:48.287 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:48.279+0000 7fa950e00640 1 --2- 172.21.15.32:0/3508210073 >> [v2:172.21.15.32:6832/211300533,v1:172.21.15.32:6835/211300533] conn(0x55dd033ea180 0x55dd03812d40 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8285cdcab2fe6d76 server_cookie=8fd9000b9fc244eb in_seq=0 out_seq=0 2024-08-03T20:48:48.287 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:48.279+0000 7fa932000640 10 client.5181 ms_handle_connect on v2:172.21.15.32:6832/211300533 2024-08-03T20:48:48.288 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:48.280+0000 7fa932000640 1 -- 172.21.15.32:0/3508210073 <== mds.0 v2:172.21.15.32:6832/211300533 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55dd037461f0 con 0x55dd033ea180 2024-08-03T20:48:48.288 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:48.280+0000 7fa932000640 10 client.5181 handle_command_reply: tid=0 2024-08-03T20:48:48.288 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:48.280+0000 7fa95a783740 1 client.5181 shutdown 2024-08-03T20:48:48.289 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:48.280+0000 7fa95a783740 1 -- 172.21.15.32:0/3508210073 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55dd0340a8d0 msgr2=0x55dd037fa1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:48:48.289 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:48.280+0000 7fa95a783740 1 --2- 172.21.15.32:0/3508210073 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55dd0340a8d0 0x55dd037fa1d0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7fa93800e9f0 tx=0x7fa938000fe0 comp rx=0 tx=0).stop 2024-08-03T20:48:48.289 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:48.280+0000 7fa95a783740 1 -- 172.21.15.32:0/3508210073 shutdown_connections 2024-08-03T20:48:48.289 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:48.280+0000 7fa95a783740 1 --2- 172.21.15.32:0/3508210073 >> [v2:172.21.15.32:6832/211300533,v1:172.21.15.32:6835/211300533] conn(0x55dd033ea180 0x55dd03812d40 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:48.289 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:48.280+0000 7fa95a783740 1 --2- 172.21.15.32:0/3508210073 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55dd0342b5b0 0x55dd038122f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:48.289 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:48.280+0000 7fa95a783740 1 --2- 172.21.15.32:0/3508210073 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55dd0340a8d0 0x55dd037fa1d0 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:48.289 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:48.280+0000 7fa932000640 0 client.5181 ms_handle_reset on v2:172.21.15.32:6832/211300533 2024-08-03T20:48:48.289 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:48.280+0000 7fa95a783740 1 --2- 172.21.15.32:0/3508210073 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55dd034026f0 0x55dd037e9550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:48.289 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:48.280+0000 7fa95a783740 1 -- 172.21.15.32:0/3508210073 >> 172.21.15.32:0/3508210073 conn(0x55dd0341f700 msgr2=0x55dd037cc590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:48:48.289 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:48.281+0000 7fa95a783740 1 -- 172.21.15.32:0/3508210073 shutdown_connections 2024-08-03T20:48:48.289 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:48.281+0000 7fa95a783740 1 -- 172.21.15.32:0/3508210073 wait complete. 2024-08-03T20:48:48.289 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:48.281+0000 7fa95a783740 20 client.5181 trim_cache size 0 max 16384 2024-08-03T20:48:48.302 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:48.300+0000 7fa95a783740 1 Processor -- start 2024-08-03T20:48:48.302 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:48.301+0000 7fa95a783740 1 -- start start 2024-08-03T20:48:48.305 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:48.302+0000 7fa95a783740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55dd034026f0 0x55dd037d0be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:48:48.305 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:48.302+0000 7fa95a783740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55dd0340a8d0 0x55dd037d1120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:48:48.305 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:48.302+0000 7fa95a783740 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55dd0342b5b0 0x55dd03832270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:48:48.305 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:48.302+0000 7fa95a783740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fa94c1fd930 con 0x55dd0340a8d0 2024-08-03T20:48:48.305 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:48.302+0000 7fa95a783740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7fa94c0ceb40 con 0x55dd034026f0 2024-08-03T20:48:48.305 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:48.302+0000 7fa95a783740 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7fa94c0b3980 con 0x55dd0342b5b0 2024-08-03T20:48:48.305 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:48.302+0000 7fa94be00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55dd0340a8d0 0x55dd037d1120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:48:48.305 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:48.302+0000 7fa94be00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55dd0340a8d0 0x55dd037d1120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:45144/0 (socket says 172.21.15.32:45144) 2024-08-03T20:48:48.305 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:48.302+0000 7fa94be00640 1 -- 172.21.15.32:0/1011055946 learned_addr learned my addr 172.21.15.32:0/1011055946 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:48:48.305 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:48.302+0000 7fa94be00640 1 -- 172.21.15.32:0/1011055946 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55dd0342b5b0 msgr2=0x55dd03832270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:48:48.305 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:48.302+0000 7fa950e00640 1 --2- 172.21.15.32:0/1011055946 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55dd034026f0 0x55dd037d0be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:48:48.306 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:48.302+0000 7fa951800640 1 --2- 172.21.15.32:0/1011055946 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55dd0342b5b0 0x55dd03832270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:48:48.306 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:48.303+0000 7fa94be00640 1 --2- 172.21.15.32:0/1011055946 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55dd0342b5b0 0x55dd03832270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:48.306 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:48.303+0000 7fa94be00640 1 -- 172.21.15.32:0/1011055946 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55dd034026f0 msgr2=0x55dd037d0be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:48:48.306 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:48.303+0000 7fa94be00640 1 --2- 172.21.15.32:0/1011055946 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55dd034026f0 0x55dd037d0be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:48.306 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:48.303+0000 7fa94be00640 1 -- 172.21.15.32:0/1011055946 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55dd033dc490 con 0x55dd0340a8d0 2024-08-03T20:48:48.306 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:48.303+0000 7fa950e00640 1 --2- 172.21.15.32:0/1011055946 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55dd034026f0 0x55dd037d0be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-03T20:48:48.306 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:48.303+0000 7fa951800640 1 --2- 172.21.15.32:0/1011055946 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55dd0342b5b0 0x55dd03832270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-03T20:48:48.306 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:48.303+0000 7fa94be00640 1 --2- 172.21.15.32:0/1011055946 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55dd0340a8d0 0x55dd037d1120 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7fa938007b60 tx=0x7fa938001520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=224380dd7274fd53 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:48:48.306 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:48.303+0000 7fa93cc00640 1 -- 172.21.15.32:0/1011055946 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa938003d80 con 0x55dd0340a8d0 2024-08-03T20:48:48.307 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:48.303+0000 7fa93cc00640 1 -- 172.21.15.32:0/1011055946 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fa938013420 con 0x55dd0340a8d0 2024-08-03T20:48:48.307 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:48.303+0000 7fa93cc00640 1 -- 172.21.15.32:0/1011055946 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa938013970 con 0x55dd0340a8d0 2024-08-03T20:48:48.311 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:48.304+0000 7fa95a783740 1 -- 172.21.15.32:0/1011055946 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55dd0340a8d0 msgr2=0x55dd037d1120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:48:48.311 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:48.304+0000 7fa95a783740 1 --2- 172.21.15.32:0/1011055946 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55dd0340a8d0 0x55dd037d1120 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7fa938007b60 tx=0x7fa938001520 comp rx=0 tx=0).stop 2024-08-03T20:48:48.311 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:48.304+0000 7fa95a783740 1 -- 172.21.15.32:0/1011055946 shutdown_connections 2024-08-03T20:48:48.311 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:48.304+0000 7fa95a783740 1 --2- 172.21.15.32:0/1011055946 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55dd0342b5b0 0x55dd03832270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:48.311 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:48.304+0000 7fa95a783740 1 --2- 172.21.15.32:0/1011055946 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55dd0340a8d0 0x55dd037d1120 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:48.311 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:48.304+0000 7fa95a783740 1 --2- 172.21.15.32:0/1011055946 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55dd034026f0 0x55dd037d0be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:48.311 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:48.304+0000 7fa95a783740 1 -- 172.21.15.32:0/1011055946 >> 172.21.15.32:0/1011055946 conn(0x55dd0341f700 msgr2=0x55dd037c6840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:48:48.311 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:48.304+0000 7fa95a783740 1 -- 172.21.15.32:0/1011055946 shutdown_connections 2024-08-03T20:48:48.311 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:48.304+0000 7fa95a783740 1 -- 172.21.15.32:0/1011055946 wait complete. 2024-08-03T20:48:48.312 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:48.305+0000 7fa95a783740 1 -- start start 2024-08-03T20:48:48.312 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:48.305+0000 7fa95a783740 1 Processor -- start 2024-08-03T20:48:48.312 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:48.307+0000 7fa95a783740 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55dd034026f0 0x55dd037cd670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:48:48.312 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:48.307+0000 7fa95a783740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55dd0340a8d0 0x55dd0380c530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:48:48.312 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:48.307+0000 7fa95a783740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55dd0342b5b0 0x55dd03899010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:48:48.312 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:48.307+0000 7fa95a783740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fa94c0c86a0 con 0x55dd0340a8d0 2024-08-03T20:48:48.312 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:48.307+0000 7fa95a783740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x55dd03743fb0 con 0x55dd0342b5b0 2024-08-03T20:48:48.312 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:48.307+0000 7fa95a783740 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x55dd037fa550 con 0x55dd034026f0 2024-08-03T20:48:48.312 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:48.307+0000 7fa95a783740 10 client.0 _ll_register_callbacks cb 0x7fa9380053b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-03T20:48:48.312 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:48.307+0000 7fa94be00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55dd0340a8d0 0x55dd0380c530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:48:48.312 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:48.307+0000 7fa94be00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55dd0340a8d0 0x55dd0380c530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:45148/0 (socket says 172.21.15.32:45148) 2024-08-03T20:48:48.313 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:48.307+0000 7fa94be00640 1 -- 172.21.15.32:0/25050254 learned_addr learned my addr 172.21.15.32:0/25050254 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:48:48.313 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:48.307+0000 7fa950e00640 1 --2- 172.21.15.32:0/25050254 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55dd034026f0 0x55dd037cd670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:48:48.313 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:48.307+0000 7fa951800640 1 --2- 172.21.15.32:0/25050254 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55dd0342b5b0 0x55dd03899010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:48:48.313 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:48.307+0000 7fa94be00640 1 -- 172.21.15.32:0/25050254 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55dd034026f0 msgr2=0x55dd037cd670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:48:48.313 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:48.307+0000 7fa94be00640 1 --2- 172.21.15.32:0/25050254 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55dd034026f0 0x55dd037cd670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:48.313 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:48.307+0000 7fa94be00640 1 -- 172.21.15.32:0/25050254 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55dd0342b5b0 msgr2=0x55dd03899010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:48:48.313 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:48.307+0000 7fa94be00640 1 --2- 172.21.15.32:0/25050254 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55dd0342b5b0 0x55dd03899010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:48.313 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:48.307+0000 7fa94be00640 1 -- 172.21.15.32:0/25050254 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55dd0380cfb0 con 0x55dd0340a8d0 2024-08-03T20:48:48.313 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:48.307+0000 7fa951800640 1 --2- 172.21.15.32:0/25050254 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55dd0342b5b0 0x55dd03899010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-03T20:48:48.313 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:48.308+0000 7fa94be00640 1 --2- 172.21.15.32:0/25050254 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55dd0340a8d0 0x55dd0380c530 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7fa938005150 tx=0x7fa9380055a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=952e5356540bbf12 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:48:48.314 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:48.308+0000 7fa932000640 10 client.0 ms_handle_connect on v2:172.21.15.32:3300/0 2024-08-03T20:48:48.314 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:48.308+0000 7fa932000640 1 -- 172.21.15.32:0/25050254 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa93803b020 con 0x55dd0340a8d0 2024-08-03T20:48:48.314 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:48.308+0000 7fa95a783740 1 -- 172.21.15.32:0/25050254 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa938013420 con 0x55dd0340a8d0 2024-08-03T20:48:48.314 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:48.308+0000 7fa932000640 1 -- 172.21.15.32:0/25050254 <== mon.0 v2:172.21.15.32:3300/0 2 ==== osd_map(33..33 src has 1..33) ==== 5795+0+0 (secure 0 0 0) 0x7fa938014e70 con 0x55dd0340a8d0 2024-08-03T20:48:48.314 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:48.308+0000 7fa932000640 1 -- 172.21.15.32:0/25050254 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7fa938045050 con 0x55dd0340a8d0 2024-08-03T20:48:48.314 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:48.308+0000 7fa95a783740 10 client.5187 fetch_fsmap learned FSMap version 23 2024-08-03T20:48:48.314 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:48.308+0000 7fa95a783740 1 -- 172.21.15.32:0/25050254 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=2+}) -- 0x55dd037fa290 con 0x55dd0340a8d0 2024-08-03T20:48:48.314 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:48.308+0000 7fa932000640 1 -- 172.21.15.32:0/25050254 <== mon.0 v2:172.21.15.32:3300/0 4 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fa938013420 con 0x55dd0340a8d0 2024-08-03T20:48:48.314 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:48.309+0000 7fa932000640 1 -- 172.21.15.32:0/25050254 <== mon.0 v2:172.21.15.32:3300/0 5 ==== fsmap(e 23) ==== 3502+0+0 (secure 0 0 0) 0x7fa938048020 con 0x55dd0340a8d0 2024-08-03T20:48:48.314 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:48.309+0000 7fa95a783740 10 client.5187 fetch_fsmap finished waiting for FSMap version 23 2024-08-03T20:48:48.314 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:48.309+0000 7fa95a783740 10 client.5187 resolve_mds: resolved 2:0 to role '2:0' aka daemon mds.a 2024-08-03T20:48:48.314 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:48.309+0000 7fa95a783740 20 client.5187 populate_metadata read hostname 'smithi032' 2024-08-03T20:48:48.315 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:48.309+0000 7fa95a783740 1 --2- 172.21.15.32:0/25050254 >> [v2:172.21.15.32:6832/211300533,v1:172.21.15.32:6835/211300533] conn(0x55dd037f9750 0x55dd037f9b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:48:48.315 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:48.309+0000 7fa95a783740 4 client.5187 mds_command: new command op to 5070 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/subdir", "scrubops": ["repair"]}] 2024-08-03T20:48:48.315 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:48.309+0000 7fa95a783740 1 -- 172.21.15.32:0/25050254 --> [v2:172.21.15.32:6832/211300533,v1:172.21.15.32:6835/211300533] -- command(tid 0: {"prefix": "scrub start", "path": "/subdir", "scrubops": ["repair"]}) -- 0x55dd03741c90 con 0x55dd037f9750 2024-08-03T20:48:48.315 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:48.309+0000 7fa950e00640 1 --2- 172.21.15.32:0/25050254 >> [v2:172.21.15.32:6832/211300533,v1:172.21.15.32:6835/211300533] conn(0x55dd037f9750 0x55dd037f9b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:48:48.315 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:48.309+0000 7fa950e00640 1 --2- 172.21.15.32:0/25050254 >> [v2:172.21.15.32:6832/211300533,v1:172.21.15.32:6835/211300533] conn(0x55dd037f9750 0x55dd037f9b60 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6132774feba5d76c server_cookie=92654b3c26c9d50d in_seq=0 out_seq=0 2024-08-03T20:48:48.315 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:48.309+0000 7fa932000640 10 client.5187 ms_handle_connect on v2:172.21.15.32:6832/211300533 2024-08-03T20:48:48.316 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:48.310+0000 7fa932000640 1 -- 172.21.15.32:0/25050254 <== mds.0 v2:172.21.15.32:6832/211300533 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55dd03741c90 con 0x55dd037f9750 2024-08-03T20:48:48.316 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:48.310+0000 7fa932000640 10 client.5187 handle_command_reply: tid=0 2024-08-03T20:48:48.316 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:48.310+0000 7fa95a783740 1 client.5187 shutdown 2024-08-03T20:48:48.317 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:48.311+0000 7fa95a783740 1 -- 172.21.15.32:0/25050254 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55dd0340a8d0 msgr2=0x55dd0380c530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:48:48.317 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:48.311+0000 7fa95a783740 1 --2- 172.21.15.32:0/25050254 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55dd0340a8d0 0x55dd0380c530 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7fa938005150 tx=0x7fa9380055a0 comp rx=0 tx=0).stop 2024-08-03T20:48:48.317 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:48.311+0000 7fa95a783740 1 -- 172.21.15.32:0/25050254 shutdown_connections 2024-08-03T20:48:48.317 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:48.311+0000 7fa95a783740 1 --2- 172.21.15.32:0/25050254 >> [v2:172.21.15.32:6832/211300533,v1:172.21.15.32:6835/211300533] conn(0x55dd037f9750 0x55dd037f9b60 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:48.317 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:48.311+0000 7fa95a783740 1 --2- 172.21.15.32:0/25050254 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55dd0342b5b0 0x55dd03899010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:48.317 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:48.311+0000 7fa95a783740 1 --2- 172.21.15.32:0/25050254 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55dd0340a8d0 0x55dd0380c530 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:48.317 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:48.311+0000 7fa95a783740 1 --2- 172.21.15.32:0/25050254 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55dd034026f0 0x55dd037cd670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:48.317 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:48.311+0000 7fa932000640 0 client.5187 ms_handle_reset on v2:172.21.15.32:6832/211300533 2024-08-03T20:48:48.317 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:48.311+0000 7fa95a783740 1 -- 172.21.15.32:0/25050254 >> 172.21.15.32:0/25050254 conn(0x55dd0341f700 msgr2=0x55dd03833310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:48:48.317 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:48.311+0000 7fa95a783740 1 -- 172.21.15.32:0/25050254 shutdown_connections 2024-08-03T20:48:48.317 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:48.311+0000 7fa95a783740 1 -- 172.21.15.32:0/25050254 wait complete. 2024-08-03T20:48:48.317 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:48.311+0000 7fa95a783740 20 client.5187 trim_cache size 0 max 16384 2024-08-03T20:48:48.318 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-08-03T20:48:48.318 INFO:teuthology.orchestra.run.smithi032.stdout: "return_code": 0, 2024-08-03T20:48:48.318 INFO:teuthology.orchestra.run.smithi032.stdout: "scrub_tag": "e08b694b-10df-4b47-88e5-9196c4aa63d3", 2024-08-03T20:48:48.318 INFO:teuthology.orchestra.run.smithi032.stdout: "mode": "asynchronous" 2024-08-03T20:48:48.318 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-08-03T20:48:48.319 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:48.312+0000 7fa952200640 1 -- 172.21.15.32:0/3545759791 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fa92c077040 msgr2=0x7fa92c079550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:48:48.319 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:48.312+0000 7fa952200640 1 --2- 172.21.15.32:0/3545759791 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fa92c077040 0x7fa92c079550 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fa938007fd0 tx=0x7fa938009040 comp rx=0 tx=0).stop 2024-08-03T20:48:48.319 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:48.312+0000 7fa952200640 1 -- 172.21.15.32:0/3545759791 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa94c159ff0 msgr2=0x7fa94c14e960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:48:48.319 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:48.312+0000 7fa952200640 1 --2- 172.21.15.32:0/3545759791 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa94c159ff0 0x7fa94c14e960 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7fa94000c500 tx=0x7fa94000c9c0 comp rx=0 tx=0).stop 2024-08-03T20:48:48.319 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:48.312+0000 7fa952200640 1 -- 172.21.15.32:0/3545759791 shutdown_connections 2024-08-03T20:48:48.319 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:48.312+0000 7fa952200640 1 --2- 172.21.15.32:0/3545759791 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fa92c077040 0x7fa92c079550 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:48.319 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:48.312+0000 7fa952200640 1 --2- 172.21.15.32:0/3545759791 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa94c0b46e0 0x7fa94c0b4af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:48.319 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:48.312+0000 7fa952200640 1 --2- 172.21.15.32:0/3545759791 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fa94c1609d0 0x7fa94c0b0f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:48.320 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:48.312+0000 7fa952200640 1 --2- 172.21.15.32:0/3545759791 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa94c159ff0 0x7fa94c14e960 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:48.320 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:48.312+0000 7fa952200640 1 -- 172.21.15.32:0/3545759791 >> 172.21.15.32:0/3545759791 conn(0x7fa94c0cfd80 msgr2=0x7fa94c1596a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:48:48.320 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:48.312+0000 7fa952200640 1 -- 172.21.15.32:0/3545759791 shutdown_connections 2024-08-03T20:48:48.320 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:48.312+0000 7fa952200640 1 -- 172.21.15.32:0/3545759791 wait complete. 2024-08-03T20:48:48.338 INFO:tasks.cephfs.test_damage:{ "return_code": 0, "scrub_tag": "e08b694b-10df-4b47-88e5-9196c4aa63d3", "mode": "asynchronous" } 2024-08-03T20:48:48.338 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-08-03T20:48:48.339 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.2:0 scrub status 2024-08-03T20:48:48.570 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:48.567+0000 7fbbab600640 1 -- 172.21.15.32:0/1617556521 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbb8c005300 msgr2=0x7fbb8c00b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:48:48.570 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:48.567+0000 7fbbab600640 1 --2- 172.21.15.32:0/1617556521 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbb8c005300 0x7fbb8c00b9c0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7fbb9c009e20 tx=0x7fbb8c005830 comp rx=0 tx=0).stop 2024-08-03T20:48:48.571 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:48.567+0000 7fbbab600640 1 -- 172.21.15.32:0/1617556521 shutdown_connections 2024-08-03T20:48:48.571 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:48.567+0000 7fbbab600640 1 --2- 172.21.15.32:0/1617556521 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbb8c005300 0x7fbb8c00b9c0 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:48.571 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:48.567+0000 7fbbab600640 1 -- 172.21.15.32:0/1617556521 >> 172.21.15.32:0/1617556521 conn(0x7fbba40b1bd0 msgr2=0x7fbba40b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:48:48.571 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:48.567+0000 7fbbab600640 1 -- 172.21.15.32:0/1617556521 shutdown_connections 2024-08-03T20:48:48.571 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:48.567+0000 7fbbab600640 1 -- 172.21.15.32:0/1617556521 wait complete. 2024-08-03T20:48:48.571 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:48.570+0000 7fbbab600640 1 Processor -- start 2024-08-03T20:48:48.572 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:48.570+0000 7fbbab600640 1 -- start start 2024-08-03T20:48:48.574 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:48.571+0000 7fbbab600640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fbb8c005300 0x7fbba40741b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:48:48.574 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:48.571+0000 7fbbab600640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fbba4151610 0x7fbba41e8f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:48:48.574 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:48.571+0000 7fbbab600640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbba41e9450 0x7fbba41f4ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:48:48.574 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:48.571+0000 7fbbab600640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fbba40ceeb0 con 0x7fbba41e9450 2024-08-03T20:48:48.574 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:48.571+0000 7fbbab600640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7fbba40ced30 con 0x7fbba4151610 2024-08-03T20:48:48.574 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:48.571+0000 7fbbab600640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7fbba40cf030 con 0x7fbb8c005300 2024-08-03T20:48:48.574 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:48.572+0000 7fbba3e00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fbba4151610 0x7fbba41e8f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:48:48.574 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:48.572+0000 7fbba9600640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbba41e9450 0x7fbba41f4ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:48:48.575 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:48.572+0000 7fbba8c00640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fbb8c005300 0x7fbba40741b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:48:48.575 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:48.572+0000 7fbba3e00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fbba4151610 0x7fbba41e8f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.32:51186/0 (socket says 172.21.15.32:51186) 2024-08-03T20:48:48.575 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:48.572+0000 7fbba3e00640 1 -- 172.21.15.32:0/1900098264 learned_addr learned my addr 172.21.15.32:0/1900098264 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:48:48.575 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:48.572+0000 7fbba9600640 1 -- 172.21.15.32:0/1900098264 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fbb8c005300 msgr2=0x7fbba40741b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:48:48.575 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:48.572+0000 7fbba9600640 1 --2- 172.21.15.32:0/1900098264 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fbb8c005300 0x7fbba40741b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:48.575 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:48.572+0000 7fbba9600640 1 -- 172.21.15.32:0/1900098264 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fbba4151610 msgr2=0x7fbba41e8f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:48:48.575 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:48.572+0000 7fbba9600640 1 --2- 172.21.15.32:0/1900098264 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fbba4151610 0x7fbba41e8f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:48.575 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:48.572+0000 7fbba9600640 1 -- 172.21.15.32:0/1900098264 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbba41e9860 con 0x7fbba41e9450 2024-08-03T20:48:48.575 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:48.573+0000 7fbba9600640 1 --2- 172.21.15.32:0/1900098264 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbba41e9450 0x7fbba41f4ff0 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7fbba415c970 tx=0x7fbb9c02cee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:48:48.576 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:48.573+0000 7fbba0c00640 1 -- 172.21.15.32:0/1900098264 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb9c009a20 con 0x7fbba41e9450 2024-08-03T20:48:48.576 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:48.573+0000 7fbba0c00640 1 -- 172.21.15.32:0/1900098264 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fbb9c044070 con 0x7fbba41e9450 2024-08-03T20:48:48.576 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:48.573+0000 7fbba0c00640 1 -- 172.21.15.32:0/1900098264 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb9c005580 con 0x7fbba41e9450 2024-08-03T20:48:48.576 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:48.573+0000 7fbbab600640 1 -- 172.21.15.32:0/1900098264 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbba4072890 con 0x7fbba41e9450 2024-08-03T20:48:48.577 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:48.573+0000 7fbbab600640 1 -- 172.21.15.32:0/1900098264 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbba4072e50 con 0x7fbba41e9450 2024-08-03T20:48:48.577 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:48.575+0000 7fbba0c00640 1 -- 172.21.15.32:0/1900098264 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7fbb9c005720 con 0x7fbba41e9450 2024-08-03T20:48:48.578 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:48.576+0000 7fbba0c00640 1 --2- 172.21.15.32:0/1900098264 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fbb84077010 0x7fbb84079520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:48:48.578 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:48.576+0000 7fbba0c00640 1 -- 172.21.15.32:0/1900098264 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(33..33 src has 1..33) ==== 5795+0+0 (secure 0 0 0) 0x7fbb9c0b9e60 con 0x7fbba41e9450 2024-08-03T20:48:48.578 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:48.577+0000 7fbbb266e740 1 Processor -- start 2024-08-03T20:48:48.578 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:48.577+0000 7fbba8c00640 1 --2- 172.21.15.32:0/1900098264 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fbb84077010 0x7fbb84079520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:48:48.579 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:48.577+0000 7fbbb266e740 1 -- start start 2024-08-03T20:48:48.579 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:48.578+0000 7fbba8c00640 1 --2- 172.21.15.32:0/1900098264 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fbb84077010 0x7fbb84079520 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fbb980066b0 tx=0x7fbb98006b70 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:48:48.581 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:48.579+0000 7fbbb266e740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5632f16bf390 0x5632f1a8f900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:48:48.581 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:48.579+0000 7fbbb266e740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5632f16964d0 0x5632f1a8fe40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:48:48.581 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:48.579+0000 7fbbb266e740 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x5632f169e6b0 0x5632f1aacd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:48:48.581 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:48.579+0000 7fbbb266e740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x5632f19d8470 con 0x5632f16bf390 2024-08-03T20:48:48.582 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:48.579+0000 7fbbb266e740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x5632f19c7970 con 0x5632f16964d0 2024-08-03T20:48:48.582 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:48.579+0000 7fbbb266e740 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x5632f19cb9b0 con 0x5632f169e6b0 2024-08-03T20:48:48.582 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:48.579+0000 7fbba8c00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5632f16bf390 0x5632f1a8f900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:48:48.582 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:48.579+0000 7fbba8c00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5632f16bf390 0x5632f1a8f900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:45178/0 (socket says 172.21.15.32:45178) 2024-08-03T20:48:48.582 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:48.579+0000 7fbba8c00640 1 -- 172.21.15.32:0/3952856926 learned_addr learned my addr 172.21.15.32:0/3952856926 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:48:48.582 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:48.579+0000 7fbba3e00640 1 --2- 172.21.15.32:0/3952856926 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5632f16964d0 0x5632f1a8fe40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:48:48.582 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:48.579+0000 7fbba9600640 1 --2- 172.21.15.32:0/3952856926 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x5632f169e6b0 0x5632f1aacd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:48:48.582 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:48.579+0000 7fbba8c00640 1 -- 172.21.15.32:0/3952856926 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x5632f169e6b0 msgr2=0x5632f1aacd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:48:48.582 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:48.579+0000 7fbba8c00640 1 --2- 172.21.15.32:0/3952856926 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x5632f169e6b0 0x5632f1aacd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:48.582 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:48.579+0000 7fbba8c00640 1 -- 172.21.15.32:0/3952856926 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5632f16964d0 msgr2=0x5632f1a8fe40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:48:48.582 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:48.579+0000 7fbba8c00640 1 --2- 172.21.15.32:0/3952856926 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5632f16964d0 0x5632f1a8fe40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:48.582 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:48.579+0000 7fbba8c00640 1 -- 172.21.15.32:0/3952856926 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5632f17a17d0 con 0x5632f16bf390 2024-08-03T20:48:48.583 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:48.580+0000 7fbba8c00640 1 --2- 172.21.15.32:0/3952856926 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5632f16bf390 0x5632f1a8f900 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7fbb98006850 tx=0x7fbb98015000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5efe89b2360b8d23 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:48:48.583 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:48.580+0000 7fbb95200640 1 -- 172.21.15.32:0/3952856926 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb9800f960 con 0x5632f16bf390 2024-08-03T20:48:48.583 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:48.580+0000 7fbb95200640 1 -- 172.21.15.32:0/3952856926 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fbb98024020 con 0x5632f16bf390 2024-08-03T20:48:48.583 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:48.580+0000 7fbb95200640 1 -- 172.21.15.32:0/3952856926 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb9800fd20 con 0x5632f16bf390 2024-08-03T20:48:48.584 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:48.581+0000 7fbbb266e740 1 -- 172.21.15.32:0/3952856926 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5632f16bf390 msgr2=0x5632f1a8f900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:48:48.584 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:48.581+0000 7fbbb266e740 1 --2- 172.21.15.32:0/3952856926 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5632f16bf390 0x5632f1a8f900 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7fbb98006850 tx=0x7fbb98015000 comp rx=0 tx=0).stop 2024-08-03T20:48:48.584 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:48.581+0000 7fbbb266e740 1 -- 172.21.15.32:0/3952856926 shutdown_connections 2024-08-03T20:48:48.584 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:48.581+0000 7fbbb266e740 1 --2- 172.21.15.32:0/3952856926 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5632f16964d0 0x5632f1a8fe40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:48.584 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:48.581+0000 7fbbb266e740 1 --2- 172.21.15.32:0/3952856926 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x5632f169e6b0 0x5632f1aacd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:48.584 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:48.581+0000 7fbbb266e740 1 --2- 172.21.15.32:0/3952856926 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5632f16bf390 0x5632f1a8f900 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:48.584 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:48.581+0000 7fbbb266e740 1 -- 172.21.15.32:0/3952856926 >> 172.21.15.32:0/3952856926 conn(0x5632f16b34e0 msgr2=0x5632f1a65bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:48:48.584 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:48.581+0000 7fbbb266e740 1 -- 172.21.15.32:0/3952856926 shutdown_connections 2024-08-03T20:48:48.584 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:48.581+0000 7fbbb266e740 1 -- 172.21.15.32:0/3952856926 wait complete. 2024-08-03T20:48:48.585 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:48.582+0000 7fbbb266e740 1 -- start start 2024-08-03T20:48:48.585 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:48.582+0000 7fbbb266e740 1 Processor -- start 2024-08-03T20:48:48.588 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:48.584+0000 7fbbb266e740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5632f16964d0 0x5632f1abdb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:48:48.588 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:48.584+0000 7fbbb266e740 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x5632f169e6b0 0x5632f1ab9c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:48:48.588 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:48.584+0000 7fbbb266e740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5632f16bf390 0x5632f1aba150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:48:48.588 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:48.584+0000 7fbbb266e740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x5632f18ffc90 con 0x5632f16964d0 2024-08-03T20:48:48.588 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:48.584+0000 7fbbb266e740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x5632f1961fd0 con 0x5632f16bf390 2024-08-03T20:48:48.588 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:48.584+0000 7fbbb266e740 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x5632f1a90380 con 0x5632f169e6b0 2024-08-03T20:48:48.588 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:48.584+0000 7fbbb266e740 10 client.0 _ll_register_callbacks cb 0x5632f197ce50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-03T20:48:48.588 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:48.584+0000 7fbba8c00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5632f16964d0 0x5632f1abdb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:48:48.588 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:48.584+0000 7fbba8c00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5632f16964d0 0x5632f1abdb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:45180/0 (socket says 172.21.15.32:45180) 2024-08-03T20:48:48.588 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:48.584+0000 7fbba8c00640 1 -- 172.21.15.32:0/3398659393 learned_addr learned my addr 172.21.15.32:0/3398659393 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:48:48.588 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:48.584+0000 7fbba9600640 1 --2- 172.21.15.32:0/3398659393 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5632f16bf390 0x5632f1aba150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:48:48.589 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:48.584+0000 7fbba3e00640 1 --2- 172.21.15.32:0/3398659393 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x5632f169e6b0 0x5632f1ab9c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:48:48.589 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:48.585+0000 7fbba8c00640 1 -- 172.21.15.32:0/3398659393 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x5632f169e6b0 msgr2=0x5632f1ab9c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:48:48.589 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:48.585+0000 7fbba8c00640 1 --2- 172.21.15.32:0/3398659393 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x5632f169e6b0 0x5632f1ab9c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:48.589 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:48.585+0000 7fbba8c00640 1 -- 172.21.15.32:0/3398659393 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5632f16bf390 msgr2=0x5632f1aba150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:48:48.589 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:48.585+0000 7fbba8c00640 1 --2- 172.21.15.32:0/3398659393 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5632f16bf390 0x5632f1aba150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:48.589 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:48.585+0000 7fbba8c00640 1 -- 172.21.15.32:0/3398659393 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5632f172a970 con 0x5632f16964d0 2024-08-03T20:48:48.589 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:48.585+0000 7fbba3e00640 1 --2- 172.21.15.32:0/3398659393 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x5632f169e6b0 0x5632f1ab9c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-03T20:48:48.589 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:48.585+0000 7fbba8c00640 1 --2- 172.21.15.32:0/3398659393 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5632f16964d0 0x5632f1abdb70 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7fbb98006820 tx=0x7fbb9802a6f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c5270e6979ad81b7 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:48:48.589 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:48.585+0000 7fbb89600640 10 client.0 ms_handle_connect on v2:172.21.15.32:3300/0 2024-08-03T20:48:48.589 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:48.585+0000 7fbb89600640 1 -- 172.21.15.32:0/3398659393 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb98037060 con 0x5632f16964d0 2024-08-03T20:48:48.589 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:48.585+0000 7fbb89600640 1 -- 172.21.15.32:0/3398659393 <== mon.0 v2:172.21.15.32:3300/0 2 ==== osd_map(33..33 src has 1..33) ==== 5795+0+0 (secure 0 0 0) 0x7fbb9802acf0 con 0x5632f16964d0 2024-08-03T20:48:48.589 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:48.585+0000 7fbbb266e740 1 -- 172.21.15.32:0/3398659393 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbb98024020 con 0x5632f16964d0 2024-08-03T20:48:48.589 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:48.586+0000 7fbb89600640 1 -- 172.21.15.32:0/3398659393 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7fbb9803d050 con 0x5632f16964d0 2024-08-03T20:48:48.590 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:48.586+0000 7fbbb266e740 10 client.5199 fetch_fsmap learned FSMap version 23 2024-08-03T20:48:48.590 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:48.586+0000 7fbbb266e740 1 -- 172.21.15.32:0/3398659393 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=2+}) -- 0x5632f1670270 con 0x5632f16964d0 2024-08-03T20:48:48.590 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:48.586+0000 7fbb89600640 1 -- 172.21.15.32:0/3398659393 <== mon.0 v2:172.21.15.32:3300/0 4 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fbb98024020 con 0x5632f16964d0 2024-08-03T20:48:48.590 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:48.586+0000 7fbb89600640 1 -- 172.21.15.32:0/3398659393 <== mon.0 v2:172.21.15.32:3300/0 5 ==== fsmap(e 23) ==== 3502+0+0 (secure 0 0 0) 0x7fbb98040020 con 0x5632f16964d0 2024-08-03T20:48:48.590 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:48.586+0000 7fbbb266e740 10 client.5199 fetch_fsmap finished waiting for FSMap version 23 2024-08-03T20:48:48.590 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:48.586+0000 7fbbb266e740 10 client.5199 resolve_mds: resolved 2:0 to role '2:0' aka daemon mds.a 2024-08-03T20:48:48.590 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:48.586+0000 7fbbb266e740 20 client.5199 populate_metadata read hostname 'smithi032' 2024-08-03T20:48:48.590 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:48.586+0000 7fbbb266e740 1 --2- 172.21.15.32:0/3398659393 >> [v2:172.21.15.32:6832/211300533,v1:172.21.15.32:6835/211300533] conn(0x5632f167df60 0x5632f1a5fb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:48:48.590 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:48.586+0000 7fbbb266e740 4 client.5199 mds_command: new command op to 5070 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-08-03T20:48:48.590 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:48.586+0000 7fbbb266e740 1 -- 172.21.15.32:0/3398659393 --> [v2:172.21.15.32:6832/211300533,v1:172.21.15.32:6835/211300533] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5632f19da3c0 con 0x5632f167df60 2024-08-03T20:48:48.590 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:48.586+0000 7fbba3e00640 1 --2- 172.21.15.32:0/3398659393 >> [v2:172.21.15.32:6832/211300533,v1:172.21.15.32:6835/211300533] conn(0x5632f167df60 0x5632f1a5fb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:48:48.592 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:48.587+0000 7fbba3e00640 1 --2- 172.21.15.32:0/3398659393 >> [v2:172.21.15.32:6832/211300533,v1:172.21.15.32:6835/211300533] conn(0x5632f167df60 0x5632f1a5fb60 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=858602ac2c1d1e39 server_cookie=9360c75d20009a99 in_seq=0 out_seq=0 2024-08-03T20:48:48.592 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:48.587+0000 7fbb89600640 10 client.5199 ms_handle_connect on v2:172.21.15.32:6832/211300533 2024-08-03T20:48:48.592 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:48.588+0000 7fbb89600640 1 -- 172.21.15.32:0/3398659393 <== mds.0 v2:172.21.15.32:6832/211300533 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x5632f19da3c0 con 0x5632f167df60 2024-08-03T20:48:48.592 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:48.588+0000 7fbb89600640 10 client.5199 handle_command_reply: tid=0 2024-08-03T20:48:48.592 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:48.588+0000 7fbbb266e740 1 client.5199 shutdown 2024-08-03T20:48:48.592 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:48.588+0000 7fbbb266e740 1 -- 172.21.15.32:0/3398659393 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5632f16964d0 msgr2=0x5632f1abdb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:48:48.592 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:48.588+0000 7fbbb266e740 1 --2- 172.21.15.32:0/3398659393 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5632f16964d0 0x5632f1abdb70 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7fbb98006820 tx=0x7fbb9802a6f0 comp rx=0 tx=0).stop 2024-08-03T20:48:48.592 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:48.588+0000 7fbbb266e740 1 -- 172.21.15.32:0/3398659393 shutdown_connections 2024-08-03T20:48:48.592 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:48.588+0000 7fbbb266e740 1 --2- 172.21.15.32:0/3398659393 >> [v2:172.21.15.32:6832/211300533,v1:172.21.15.32:6835/211300533] conn(0x5632f167df60 0x5632f1a5fb60 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:48.593 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:48.588+0000 7fbbb266e740 1 --2- 172.21.15.32:0/3398659393 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5632f16bf390 0x5632f1aba150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:48.593 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:48.588+0000 7fbbb266e740 1 --2- 172.21.15.32:0/3398659393 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x5632f169e6b0 0x5632f1ab9c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:48.593 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:48.588+0000 7fbb89600640 0 client.5199 ms_handle_reset on v2:172.21.15.32:6832/211300533 2024-08-03T20:48:48.593 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:48.588+0000 7fbbb266e740 1 --2- 172.21.15.32:0/3398659393 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5632f16964d0 0x5632f1abdb70 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:48.593 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:48.588+0000 7fbbb266e740 1 -- 172.21.15.32:0/3398659393 >> 172.21.15.32:0/3398659393 conn(0x5632f16b34e0 msgr2=0x5632f1aac310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:48:48.593 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:48.588+0000 7fbbb266e740 1 -- 172.21.15.32:0/3398659393 shutdown_connections 2024-08-03T20:48:48.593 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:48.588+0000 7fbbb266e740 1 -- 172.21.15.32:0/3398659393 wait complete. 2024-08-03T20:48:48.593 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:48.588+0000 7fbbb266e740 20 client.5199 trim_cache size 0 max 16384 2024-08-03T20:48:48.609 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:48.608+0000 7fbbb266e740 1 Processor -- start 2024-08-03T20:48:48.609 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:48.608+0000 7fbbb266e740 1 -- start start 2024-08-03T20:48:48.611 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:48.609+0000 7fbbb266e740 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x5632f16964d0 0x5632f1ab9ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:48:48.611 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:48.609+0000 7fbbb266e740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5632f169e6b0 0x5632f1aba0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:48:48.611 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:48.609+0000 7fbbb266e740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5632f16bf390 0x5632f1aa2be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:48:48.611 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:48.609+0000 7fbbb266e740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fbba42023c0 con 0x5632f16bf390 2024-08-03T20:48:48.611 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:48.609+0000 7fbbb266e740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7fbba41f9110 con 0x5632f169e6b0 2024-08-03T20:48:48.612 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:48.609+0000 7fbbb266e740 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7fbba4150340 con 0x5632f16964d0 2024-08-03T20:48:48.612 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:48.610+0000 7fbba9600640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5632f16bf390 0x5632f1aa2be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:48:48.612 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:48.610+0000 7fbba9600640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5632f16bf390 0x5632f1aa2be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:45190/0 (socket says 172.21.15.32:45190) 2024-08-03T20:48:48.612 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:48.610+0000 7fbba9600640 1 -- 172.21.15.32:0/2411383274 learned_addr learned my addr 172.21.15.32:0/2411383274 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:48:48.612 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:48.610+0000 7fbba8c00640 1 --2- 172.21.15.32:0/2411383274 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x5632f16964d0 0x5632f1ab9ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:48:48.612 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:48.610+0000 7fbba3e00640 1 --2- 172.21.15.32:0/2411383274 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5632f169e6b0 0x5632f1aba0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:48:48.614 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:48.610+0000 7fbba9600640 1 -- 172.21.15.32:0/2411383274 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x5632f16964d0 msgr2=0x5632f1ab9ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:48:48.614 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:48.610+0000 7fbba9600640 1 --2- 172.21.15.32:0/2411383274 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x5632f16964d0 0x5632f1ab9ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:48.614 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:48.610+0000 7fbba9600640 1 -- 172.21.15.32:0/2411383274 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5632f169e6b0 msgr2=0x5632f1aba0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:48:48.614 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:48.610+0000 7fbba9600640 1 --2- 172.21.15.32:0/2411383274 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5632f169e6b0 0x5632f1aba0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:48.614 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:48.610+0000 7fbba9600640 1 -- 172.21.15.32:0/2411383274 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5632f1aad870 con 0x5632f16bf390 2024-08-03T20:48:48.614 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:48.610+0000 7fbba9600640 1 --2- 172.21.15.32:0/2411383274 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5632f16bf390 0x5632f1aa2be0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7fbb9c07fb80 tx=0x7fbb9c07e900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=83c6de635b4b4454 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:48:48.615 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:48.611+0000 7fbb8be00640 1 -- 172.21.15.32:0/2411383274 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb9c08b250 con 0x5632f16bf390 2024-08-03T20:48:48.615 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:48.611+0000 7fbb8be00640 1 -- 172.21.15.32:0/2411383274 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fbb9c082420 con 0x5632f16bf390 2024-08-03T20:48:48.615 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:48.611+0000 7fbb8be00640 1 -- 172.21.15.32:0/2411383274 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb9c082910 con 0x5632f16bf390 2024-08-03T20:48:48.615 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:48.611+0000 7fbbb266e740 1 -- 172.21.15.32:0/2411383274 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5632f16bf390 msgr2=0x5632f1aa2be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:48:48.615 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:48.611+0000 7fbbb266e740 1 --2- 172.21.15.32:0/2411383274 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5632f16bf390 0x5632f1aa2be0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7fbb9c07fb80 tx=0x7fbb9c07e900 comp rx=0 tx=0).stop 2024-08-03T20:48:48.615 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:48.611+0000 7fbbb266e740 1 -- 172.21.15.32:0/2411383274 shutdown_connections 2024-08-03T20:48:48.615 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:48.611+0000 7fbbb266e740 1 --2- 172.21.15.32:0/2411383274 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5632f169e6b0 0x5632f1aba0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:48.615 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:48.611+0000 7fbbb266e740 1 --2- 172.21.15.32:0/2411383274 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5632f16bf390 0x5632f1aa2be0 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:48.615 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:48.611+0000 7fbbb266e740 1 --2- 172.21.15.32:0/2411383274 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x5632f16964d0 0x5632f1ab9ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:48.615 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:48.611+0000 7fbbb266e740 1 -- 172.21.15.32:0/2411383274 >> 172.21.15.32:0/2411383274 conn(0x5632f16b34e0 msgr2=0x5632f1a8c880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:48:48.615 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:48.611+0000 7fbbb266e740 1 -- 172.21.15.32:0/2411383274 shutdown_connections 2024-08-03T20:48:48.615 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:48.611+0000 7fbbb266e740 1 -- 172.21.15.32:0/2411383274 wait complete. 2024-08-03T20:48:48.615 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:48.612+0000 7fbbb266e740 1 -- start start 2024-08-03T20:48:48.616 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:48.613+0000 7fbbb266e740 1 Processor -- start 2024-08-03T20:48:48.618 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:48.614+0000 7fbbb266e740 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x5632f16964d0 0x5632f1acc5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:48:48.618 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:48.614+0000 7fbbb266e740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5632f169e6b0 0x5632f1b26010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:48:48.618 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:48.614+0000 7fbbb266e740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5632f16bf390 0x5632f1aa22c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:48:48.618 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:48.614+0000 7fbbb266e740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fbba414dc60 con 0x5632f16bf390 2024-08-03T20:48:48.618 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:48.614+0000 7fbbb266e740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x5632f1acd0c0 con 0x5632f169e6b0 2024-08-03T20:48:48.618 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:48.614+0000 7fbbb266e740 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x5632f1b2ad90 con 0x5632f16964d0 2024-08-03T20:48:48.618 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:48.614+0000 7fbbb266e740 10 client.0 _ll_register_callbacks cb 0x7fbb98012940 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-03T20:48:48.618 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:48.614+0000 7fbba9600640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5632f16bf390 0x5632f1aa22c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:48:48.619 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:48.614+0000 7fbba9600640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5632f16bf390 0x5632f1aa22c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:45194/0 (socket says 172.21.15.32:45194) 2024-08-03T20:48:48.619 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:48.614+0000 7fbba9600640 1 -- 172.21.15.32:0/231948810 learned_addr learned my addr 172.21.15.32:0/231948810 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:48:48.619 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:48.615+0000 7fbba3e00640 1 --2- 172.21.15.32:0/231948810 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5632f169e6b0 0x5632f1b26010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:48:48.619 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:48.615+0000 7fbba8c00640 1 --2- 172.21.15.32:0/231948810 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x5632f16964d0 0x5632f1acc5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:48:48.619 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:48.615+0000 7fbba9600640 1 -- 172.21.15.32:0/231948810 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x5632f16964d0 msgr2=0x5632f1acc5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:48:48.619 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:48.615+0000 7fbba9600640 1 --2- 172.21.15.32:0/231948810 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x5632f16964d0 0x5632f1acc5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:48.619 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:48.615+0000 7fbba9600640 1 -- 172.21.15.32:0/231948810 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5632f169e6b0 msgr2=0x5632f1b26010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:48:48.619 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:48.615+0000 7fbba9600640 1 --2- 172.21.15.32:0/231948810 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5632f169e6b0 0x5632f1b26010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:48.619 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:48.615+0000 7fbba9600640 1 -- 172.21.15.32:0/231948810 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5632f1abd280 con 0x5632f16bf390 2024-08-03T20:48:48.619 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:48.615+0000 7fbba9600640 1 --2- 172.21.15.32:0/231948810 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5632f16bf390 0x5632f1aa22c0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7fbb9c0828a0 tx=0x7fbb9c082180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=37482db26a40491a server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:48:48.619 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:48.615+0000 7fbb89600640 10 client.0 ms_handle_connect on v2:172.21.15.32:3300/0 2024-08-03T20:48:48.619 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:48.615+0000 7fbb89600640 1 -- 172.21.15.32:0/231948810 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb9c08b910 con 0x5632f16bf390 2024-08-03T20:48:48.620 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:48.615+0000 7fbb89600640 1 -- 172.21.15.32:0/231948810 <== mon.0 v2:172.21.15.32:3300/0 2 ==== osd_map(33..33 src has 1..33) ==== 5795+0+0 (secure 0 0 0) 0x7fbb9c08b480 con 0x5632f16bf390 2024-08-03T20:48:48.620 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:48.615+0000 7fbbb266e740 1 -- 172.21.15.32:0/231948810 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbb9c082420 con 0x5632f16bf390 2024-08-03T20:48:48.620 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:48.616+0000 7fbb89600640 1 -- 172.21.15.32:0/231948810 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7fbb9c082910 con 0x5632f16bf390 2024-08-03T20:48:48.620 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:48.616+0000 7fbbb266e740 10 client.5205 fetch_fsmap learned FSMap version 23 2024-08-03T20:48:48.620 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:48.616+0000 7fbbb266e740 1 -- 172.21.15.32:0/231948810 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=2+}) -- 0x5632f1b26550 con 0x5632f16bf390 2024-08-03T20:48:48.620 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:48.616+0000 7fbb89600640 1 -- 172.21.15.32:0/231948810 <== mon.0 v2:172.21.15.32:3300/0 4 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fbb9c082420 con 0x5632f16bf390 2024-08-03T20:48:48.620 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:48.616+0000 7fbb89600640 1 -- 172.21.15.32:0/231948810 <== mon.0 v2:172.21.15.32:3300/0 5 ==== fsmap(e 23) ==== 3502+0+0 (secure 0 0 0) 0x7fbb9c00f040 con 0x5632f16bf390 2024-08-03T20:48:48.620 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:48.616+0000 7fbbb266e740 10 client.5205 fetch_fsmap finished waiting for FSMap version 23 2024-08-03T20:48:48.620 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:48.616+0000 7fbbb266e740 10 client.5205 resolve_mds: resolved 2:0 to role '2:0' aka daemon mds.a 2024-08-03T20:48:48.620 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:48.616+0000 7fbbb266e740 20 client.5205 populate_metadata read hostname 'smithi032' 2024-08-03T20:48:48.620 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:48.616+0000 7fbbb266e740 1 --2- 172.21.15.32:0/231948810 >> [v2:172.21.15.32:6832/211300533,v1:172.21.15.32:6835/211300533] conn(0x5632f1ab9ac0 0x5632f1ab9ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:48:48.620 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:48.616+0000 7fbbb266e740 4 client.5205 mds_command: new command op to 5070 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-08-03T20:48:48.620 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:48.616+0000 7fbbb266e740 1 -- 172.21.15.32:0/231948810 --> [v2:172.21.15.32:6832/211300533,v1:172.21.15.32:6835/211300533] -- command(tid 0: {"prefix": "scrub status"}) -- 0x5632f19d5e60 con 0x5632f1ab9ac0 2024-08-03T20:48:48.620 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:48.616+0000 7fbba8c00640 1 --2- 172.21.15.32:0/231948810 >> [v2:172.21.15.32:6832/211300533,v1:172.21.15.32:6835/211300533] conn(0x5632f1ab9ac0 0x5632f1ab9ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:48:48.621 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:48.617+0000 7fbba8c00640 1 --2- 172.21.15.32:0/231948810 >> [v2:172.21.15.32:6832/211300533,v1:172.21.15.32:6835/211300533] conn(0x5632f1ab9ac0 0x5632f1ab9ed0 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1caf5f0f08568669 server_cookie=d85567afbcc27745 in_seq=0 out_seq=0 2024-08-03T20:48:48.621 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:48.617+0000 7fbb89600640 10 client.5205 ms_handle_connect on v2:172.21.15.32:6832/211300533 2024-08-03T20:48:48.622 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:48.617+0000 7fbb89600640 1 -- 172.21.15.32:0/231948810 <== mds.0 v2:172.21.15.32:6832/211300533 1 ==== command_reply(tid 0: 0 ) ==== 8+0+256 (crc 0 0 0) 0x5632f19d5e60 con 0x5632f1ab9ac0 2024-08-03T20:48:48.622 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:48.617+0000 7fbb89600640 10 client.5205 handle_command_reply: tid=0 2024-08-03T20:48:48.622 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:48.617+0000 7fbbb266e740 1 client.5205 shutdown 2024-08-03T20:48:48.622 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:48.617+0000 7fbbb266e740 1 -- 172.21.15.32:0/231948810 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5632f16bf390 msgr2=0x5632f1aa22c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:48:48.622 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:48.617+0000 7fbbb266e740 1 --2- 172.21.15.32:0/231948810 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5632f16bf390 0x5632f1aa22c0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7fbb9c0828a0 tx=0x7fbb9c082180 comp rx=0 tx=0).stop 2024-08-03T20:48:48.622 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:48.617+0000 7fbbb266e740 1 -- 172.21.15.32:0/231948810 shutdown_connections 2024-08-03T20:48:48.623 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:48.617+0000 7fbbb266e740 1 --2- 172.21.15.32:0/231948810 >> [v2:172.21.15.32:6832/211300533,v1:172.21.15.32:6835/211300533] conn(0x5632f1ab9ac0 0x5632f1ab9ed0 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:48.623 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:48.617+0000 7fbbb266e740 1 --2- 172.21.15.32:0/231948810 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5632f169e6b0 0x5632f1b26010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:48.623 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:48.617+0000 7fbbb266e740 1 --2- 172.21.15.32:0/231948810 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5632f16bf390 0x5632f1aa22c0 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:48.623 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:48.617+0000 7fbbb266e740 1 --2- 172.21.15.32:0/231948810 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x5632f16964d0 0x5632f1acc5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:48.623 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:48.617+0000 7fbbb266e740 1 -- 172.21.15.32:0/231948810 >> 172.21.15.32:0/231948810 conn(0x5632f16b34e0 msgr2=0x5632f1a5b840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:48:48.623 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:48.617+0000 7fbb89600640 0 client.5205 ms_handle_reset on v2:172.21.15.32:6832/211300533 2024-08-03T20:48:48.623 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:48.617+0000 7fbbb266e740 1 -- 172.21.15.32:0/231948810 shutdown_connections 2024-08-03T20:48:48.623 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:48.618+0000 7fbbb266e740 1 -- 172.21.15.32:0/231948810 wait complete. 2024-08-03T20:48:48.623 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:48.618+0000 7fbbb266e740 20 client.5205 trim_cache size 0 max 16384 2024-08-03T20:48:48.623 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-08-03T20:48:48.624 INFO:teuthology.orchestra.run.smithi032.stdout: "status": "IDLE (waiting for more scrubs)", 2024-08-03T20:48:48.624 INFO:teuthology.orchestra.run.smithi032.stdout: "scrubs": { 2024-08-03T20:48:48.624 INFO:teuthology.orchestra.run.smithi032.stdout: "e08b694b-10df-4b47-88e5-9196c4aa63d3": { 2024-08-03T20:48:48.624 INFO:teuthology.orchestra.run.smithi032.stdout: "path": "/subdir", 2024-08-03T20:48:48.624 INFO:teuthology.orchestra.run.smithi032.stdout: "tag": "e08b694b-10df-4b47-88e5-9196c4aa63d3", 2024-08-03T20:48:48.624 INFO:teuthology.orchestra.run.smithi032.stdout: "options": "repair" 2024-08-03T20:48:48.624 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-03T20:48:48.624 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-03T20:48:48.624 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-08-03T20:48:48.625 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:48.618+0000 7fbbab600640 1 -- 172.21.15.32:0/1900098264 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fbb84077010 msgr2=0x7fbb84079520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:48:48.625 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:48.618+0000 7fbbab600640 1 --2- 172.21.15.32:0/1900098264 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fbb84077010 0x7fbb84079520 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fbb980066b0 tx=0x7fbb98006b70 comp rx=0 tx=0).stop 2024-08-03T20:48:48.625 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:48.618+0000 7fbbab600640 1 -- 172.21.15.32:0/1900098264 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbba41e9450 msgr2=0x7fbba41f4ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:48:48.625 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:48.618+0000 7fbbab600640 1 --2- 172.21.15.32:0/1900098264 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbba41e9450 0x7fbba41f4ff0 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7fbba415c970 tx=0x7fbb9c02cee0 comp rx=0 tx=0).stop 2024-08-03T20:48:48.625 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:48.618+0000 7fbbab600640 1 -- 172.21.15.32:0/1900098264 shutdown_connections 2024-08-03T20:48:48.625 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:48.618+0000 7fbbab600640 1 --2- 172.21.15.32:0/1900098264 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fbb84077010 0x7fbb84079520 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:48.625 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:48.618+0000 7fbbab600640 1 --2- 172.21.15.32:0/1900098264 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fbba4151610 0x7fbba41e8f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:48.625 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:48.618+0000 7fbbab600640 1 --2- 172.21.15.32:0/1900098264 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbba41e9450 0x7fbba41f4ff0 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:48.625 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:48.618+0000 7fbbab600640 1 --2- 172.21.15.32:0/1900098264 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fbb8c005300 0x7fbba40741b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:48:48.626 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:48.618+0000 7fbbab600640 1 -- 172.21.15.32:0/1900098264 >> 172.21.15.32:0/1900098264 conn(0x7fbba40b1bd0 msgr2=0x7fbba415d9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:48:48.626 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:48.618+0000 7fbbab600640 1 -- 172.21.15.32:0/1900098264 shutdown_connections 2024-08-03T20:48:48.626 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:48:48.618+0000 7fbbab600640 1 -- 172.21.15.32:0/1900098264 wait complete. 2024-08-03T20:48:48.641 INFO:tasks.cephfs.filesystem:scrub status for tag:e08b694b-10df-4b47-88e5-9196c4aa63d3 - {'path': '/subdir', 'tag': 'e08b694b-10df-4b47-88e5-9196c4aa63d3', 'options': 'repair'} 2024-08-03T20:48:49.322 INFO:tasks.ceph.mon.a.smithi032.stderr:2024-08-03T20:48:49.321+0000 7f4e70800640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 MDSs report damaged metadata (MDS_DAMAGE) 2024-08-03T20:49:06.534 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-08-03T20:49:06.540 DEBUG:teuthology.orchestra.run.smithi118:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-08-03T20:49:18.642 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-08-03T20:49:18.643 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.2:0 scrub status 2024-08-03T20:49:18.883 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:18.881+0000 7f00a1a00640 1 -- 172.21.15.32:0/1010029106 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f009c0cb7f0 msgr2=0x7f009c15c1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:49:18.883 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:18.881+0000 7f00a1a00640 1 --2- 172.21.15.32:0/1010029106 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f009c0cb7f0 0x7f009c15c1a0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f0094005fd0 tx=0x7f009402aa00 comp rx=0 tx=0).stop 2024-08-03T20:49:18.883 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:18.881+0000 7f00a1a00640 1 -- 172.21.15.32:0/1010029106 shutdown_connections 2024-08-03T20:49:18.883 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:18.881+0000 7f00a1a00640 1 --2- 172.21.15.32:0/1010029106 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f009c0749b0 0x7f009c0cb2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:49:18.883 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:18.881+0000 7f00a1a00640 1 --2- 172.21.15.32:0/1010029106 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f009c0cb7f0 0x7f009c15c1a0 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:49:18.883 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:18.881+0000 7f00a1a00640 1 -- 172.21.15.32:0/1010029106 >> 172.21.15.32:0/1010029106 conn(0x7f009c0d0120 msgr2=0x7f009c0d0530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:49:18.883 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:18.881+0000 7f00a1a00640 1 -- 172.21.15.32:0/1010029106 shutdown_connections 2024-08-03T20:49:18.883 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:18.881+0000 7f00a1a00640 1 -- 172.21.15.32:0/1010029106 wait complete. 2024-08-03T20:49:18.884 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:18.883+0000 7f00a1a00640 1 Processor -- start 2024-08-03T20:49:18.885 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:18.883+0000 7f00a1a00640 1 -- start start 2024-08-03T20:49:18.888 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:18.885+0000 7f00a1a00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f009c0745a0 0x7f009c0c7ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:49:18.888 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:18.885+0000 7f00a1a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f009c0749b0 0x7f009c1511f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:49:18.888 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:18.885+0000 7f00a1a00640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f009c0cb7f0 0x7f009c0c9f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:49:18.888 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:18.885+0000 7f00a1a00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f009c15c8f0 con 0x7f009c0749b0 2024-08-03T20:49:18.888 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:18.885+0000 7f00a1a00640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f009c0731a0 con 0x7f009c0745a0 2024-08-03T20:49:18.888 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:18.885+0000 7f00a1a00640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7f009c15ca70 con 0x7f009c0cb7f0 2024-08-03T20:49:18.888 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:18.885+0000 7f0093e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f009c0749b0 0x7f009c1511f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:49:18.888 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:18.885+0000 7f009be00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f009c0745a0 0x7f009c0c7ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:49:18.888 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:18.885+0000 7f0093e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f009c0749b0 0x7f009c1511f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:34588/0 (socket says 172.21.15.32:34588) 2024-08-03T20:49:18.888 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:18.885+0000 7f0093e00640 1 -- 172.21.15.32:0/3241493742 learned_addr learned my addr 172.21.15.32:0/3241493742 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:49:18.889 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:18.885+0000 7f00a1000640 1 --2- 172.21.15.32:0/3241493742 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f009c0cb7f0 0x7f009c0c9f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:49:18.889 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:18.886+0000 7f0093e00640 1 -- 172.21.15.32:0/3241493742 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f009c0cb7f0 msgr2=0x7f009c0c9f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:49:18.889 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:18.886+0000 7f0093e00640 1 --2- 172.21.15.32:0/3241493742 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f009c0cb7f0 0x7f009c0c9f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:49:18.889 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:18.886+0000 7f0093e00640 1 -- 172.21.15.32:0/3241493742 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f009c0745a0 msgr2=0x7f009c0c7ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:49:18.889 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:18.886+0000 7f0093e00640 1 --2- 172.21.15.32:0/3241493742 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f009c0745a0 0x7f009c0c7ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:49:18.889 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:18.886+0000 7f0093e00640 1 -- 172.21.15.32:0/3241493742 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f009c1f72d0 con 0x7f009c0749b0 2024-08-03T20:49:18.889 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:18.886+0000 7f0093e00640 1 --2- 172.21.15.32:0/3241493742 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f009c0749b0 0x7f009c1511f0 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f008400c970 tx=0x7f008400ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:49:18.889 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:18.886+0000 7f0098c00640 1 -- 172.21.15.32:0/3241493742 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0084007b50 con 0x7f009c0749b0 2024-08-03T20:49:18.889 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:18.886+0000 7f0098c00640 1 -- 172.21.15.32:0/3241493742 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f0084007de0 con 0x7f009c0749b0 2024-08-03T20:49:18.889 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:18.886+0000 7f0098c00640 1 -- 172.21.15.32:0/3241493742 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f008400f6b0 con 0x7f009c0749b0 2024-08-03T20:49:18.889 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:18.886+0000 7f00a1a00640 1 -- 172.21.15.32:0/3241493742 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f009c1f75b0 con 0x7f009c0749b0 2024-08-03T20:49:18.889 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:18.886+0000 7f00a1a00640 1 -- 172.21.15.32:0/3241493742 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f009c0c9700 con 0x7f009c0749b0 2024-08-03T20:49:18.890 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:18.889+0000 7f0098c00640 1 -- 172.21.15.32:0/3241493742 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f008400f850 con 0x7f009c0749b0 2024-08-03T20:49:18.891 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:18.890+0000 7f0098c00640 1 --2- 172.21.15.32:0/3241493742 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f0078077010 0x7f0078079520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:49:18.891 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:18.890+0000 7f0098c00640 1 -- 172.21.15.32:0/3241493742 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(33..33 src has 1..33) ==== 5795+0+0 (secure 0 0 0) 0x7f0084098f40 con 0x7f009c0749b0 2024-08-03T20:49:18.892 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:18.890+0000 7f009be00640 1 --2- 172.21.15.32:0/3241493742 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f0078077010 0x7f0078079520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:49:18.892 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:18.890+0000 7f00aa19f740 1 Processor -- start 2024-08-03T20:49:18.892 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:18.890+0000 7f00aa19f740 1 -- start start 2024-08-03T20:49:18.892 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:18.891+0000 7f009be00640 1 --2- 172.21.15.32:0/3241493742 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f0078077010 0x7f0078079520 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f008c006fd0 tx=0x7f008c009060 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:49:18.894 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:18.892+0000 7f00aa19f740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55ee3544e390 0x55ee3580d4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:49:18.894 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:18.892+0000 7f00aa19f740 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55ee354254d0 0x55ee3581b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:49:18.894 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:18.892+0000 7f00aa19f740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ee3542d6b0 0x55ee3581bee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:49:18.895 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:18.892+0000 7f00aa19f740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55ee35767470 con 0x55ee3542d6b0 2024-08-03T20:49:18.895 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:18.892+0000 7f00aa19f740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x55ee35756970 con 0x55ee3544e390 2024-08-03T20:49:18.895 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:18.892+0000 7f00aa19f740 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x55ee3575a9b0 con 0x55ee354254d0 2024-08-03T20:49:18.895 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:18.892+0000 7f00a1000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ee3542d6b0 0x55ee3581bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:49:18.895 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:18.892+0000 7f00a1000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ee3542d6b0 0x55ee3581bee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:34594/0 (socket says 172.21.15.32:34594) 2024-08-03T20:49:18.895 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:18.892+0000 7f00a1000640 1 -- 172.21.15.32:0/2833298742 learned_addr learned my addr 172.21.15.32:0/2833298742 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:49:18.895 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:18.892+0000 7f009be00640 1 --2- 172.21.15.32:0/2833298742 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55ee3544e390 0x55ee3580d4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:49:18.895 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:18.892+0000 7f0093e00640 1 --2- 172.21.15.32:0/2833298742 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55ee354254d0 0x55ee3581b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:49:18.895 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:18.893+0000 7f00a1000640 1 -- 172.21.15.32:0/2833298742 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55ee354254d0 msgr2=0x55ee3581b9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:49:18.895 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:18.893+0000 7f00a1000640 1 --2- 172.21.15.32:0/2833298742 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55ee354254d0 0x55ee3581b9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:49:18.895 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:18.893+0000 7f00a1000640 1 -- 172.21.15.32:0/2833298742 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55ee3544e390 msgr2=0x55ee3580d4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:49:18.895 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:18.893+0000 7f00a1000640 1 --2- 172.21.15.32:0/2833298742 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55ee3544e390 0x55ee3580d4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:49:18.895 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:18.893+0000 7f00a1000640 1 -- 172.21.15.32:0/2833298742 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ee355307d0 con 0x55ee3542d6b0 2024-08-03T20:49:18.896 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:18.893+0000 7f00a1000640 1 --2- 172.21.15.32:0/2833298742 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ee3542d6b0 0x55ee3581bee0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f0094009d60 tx=0x7f0094004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3f9b74386f694660 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:49:18.897 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:18.893+0000 7f0090c00640 1 -- 172.21.15.32:0/2833298742 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0094033030 con 0x55ee3542d6b0 2024-08-03T20:49:18.897 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:18.893+0000 7f0090c00640 1 -- 172.21.15.32:0/2833298742 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f0094003db0 con 0x55ee3542d6b0 2024-08-03T20:49:18.897 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:18.893+0000 7f0090c00640 1 -- 172.21.15.32:0/2833298742 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f009402fbb0 con 0x55ee3542d6b0 2024-08-03T20:49:18.897 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:18.894+0000 7f00aa19f740 1 -- 172.21.15.32:0/2833298742 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ee3542d6b0 msgr2=0x55ee3581bee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:49:18.897 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:18.894+0000 7f00aa19f740 1 --2- 172.21.15.32:0/2833298742 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ee3542d6b0 0x55ee3581bee0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f0094009d60 tx=0x7f0094004ab0 comp rx=0 tx=0).stop 2024-08-03T20:49:18.897 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:18.894+0000 7f00aa19f740 1 -- 172.21.15.32:0/2833298742 shutdown_connections 2024-08-03T20:49:18.898 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:18.894+0000 7f00aa19f740 1 --2- 172.21.15.32:0/2833298742 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ee3542d6b0 0x55ee3581bee0 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:49:18.898 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:18.894+0000 7f00aa19f740 1 --2- 172.21.15.32:0/2833298742 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55ee354254d0 0x55ee3581b9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:49:18.898 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:18.894+0000 7f00aa19f740 1 --2- 172.21.15.32:0/2833298742 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55ee3544e390 0x55ee3580d4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:49:18.898 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:18.894+0000 7f00aa19f740 1 -- 172.21.15.32:0/2833298742 >> 172.21.15.32:0/2833298742 conn(0x55ee354424e0 msgr2=0x55ee357f4d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:49:18.898 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:18.894+0000 7f00aa19f740 1 -- 172.21.15.32:0/2833298742 shutdown_connections 2024-08-03T20:49:18.898 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:18.894+0000 7f00aa19f740 1 -- 172.21.15.32:0/2833298742 wait complete. 2024-08-03T20:49:18.898 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:18.895+0000 7f00aa19f740 1 -- start start 2024-08-03T20:49:18.898 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:18.896+0000 7f00aa19f740 1 Processor -- start 2024-08-03T20:49:18.901 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:18.897+0000 7f00aa19f740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55ee354254d0 0x55ee358510c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:49:18.901 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:18.897+0000 7f00aa19f740 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55ee3542d6b0 0x55ee3583b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:49:18.902 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:18.897+0000 7f00aa19f740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ee3544e390 0x55ee3582f2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:49:18.902 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:18.897+0000 7f00aa19f740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55ee3568ec90 con 0x55ee3544e390 2024-08-03T20:49:18.902 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:18.897+0000 7f00aa19f740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x55ee356f0fd0 con 0x55ee354254d0 2024-08-03T20:49:18.902 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:18.897+0000 7f00aa19f740 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x55ee358326e0 con 0x55ee3542d6b0 2024-08-03T20:49:18.902 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:18.897+0000 7f00aa19f740 10 client.0 _ll_register_callbacks cb 0x55ee3570be50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-03T20:49:18.902 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:18.897+0000 7f00a1000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ee3544e390 0x55ee3582f2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:49:18.902 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:18.897+0000 7f00a1000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ee3544e390 0x55ee3582f2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:34608/0 (socket says 172.21.15.32:34608) 2024-08-03T20:49:18.903 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:18.897+0000 7f00a1000640 1 -- 172.21.15.32:0/1175115579 learned_addr learned my addr 172.21.15.32:0/1175115579 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:49:18.903 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:18.898+0000 7f0093e00640 1 --2- 172.21.15.32:0/1175115579 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55ee3542d6b0 0x55ee3583b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:49:18.903 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:18.898+0000 7f009be00640 1 --2- 172.21.15.32:0/1175115579 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55ee354254d0 0x55ee358510c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:49:18.903 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:18.898+0000 7f00a1000640 1 -- 172.21.15.32:0/1175115579 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55ee3542d6b0 msgr2=0x55ee3583b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:49:18.903 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:18.898+0000 7f00a1000640 1 --2- 172.21.15.32:0/1175115579 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55ee3542d6b0 0x55ee3583b990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:49:18.903 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:18.898+0000 7f00a1000640 1 -- 172.21.15.32:0/1175115579 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55ee354254d0 msgr2=0x55ee358510c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:49:18.903 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:18.898+0000 7f00a1000640 1 --2- 172.21.15.32:0/1175115579 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55ee354254d0 0x55ee358510c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:49:18.903 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:18.898+0000 7f00a1000640 1 -- 172.21.15.32:0/1175115579 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ee354b9970 con 0x55ee3544e390 2024-08-03T20:49:18.903 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:18.898+0000 7f009be00640 1 --2- 172.21.15.32:0/1175115579 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55ee354254d0 0x55ee358510c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-03T20:49:18.903 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:18.898+0000 7f00a1000640 1 --2- 172.21.15.32:0/1175115579 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ee3544e390 0x55ee3582f2d0 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f0094006010 tx=0x7f009402fb80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=63b22d56feb64ea8 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:49:18.903 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:18.898+0000 7f0089600640 10 client.0 ms_handle_connect on v2:172.21.15.32:3300/0 2024-08-03T20:49:18.904 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:18.898+0000 7f0089600640 1 -- 172.21.15.32:0/1175115579 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0094048040 con 0x55ee3544e390 2024-08-03T20:49:18.904 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:18.898+0000 7f0089600640 1 -- 172.21.15.32:0/1175115579 <== mon.0 v2:172.21.15.32:3300/0 2 ==== osd_map(33..33 src has 1..33) ==== 5795+0+0 (secure 0 0 0) 0x7f009402f660 con 0x55ee3544e390 2024-08-03T20:49:18.904 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:18.898+0000 7f00aa19f740 1 -- 172.21.15.32:0/1175115579 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0094003db0 con 0x55ee3544e390 2024-08-03T20:49:18.904 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:18.899+0000 7f0089600640 1 -- 172.21.15.32:0/1175115579 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f0094003f60 con 0x55ee3544e390 2024-08-03T20:49:18.904 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:18.899+0000 7f00aa19f740 10 client.5217 fetch_fsmap learned FSMap version 25 2024-08-03T20:49:18.904 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:18.899+0000 7f00aa19f740 1 -- 172.21.15.32:0/1175115579 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=2+}) -- 0x55ee353ff270 con 0x55ee3544e390 2024-08-03T20:49:18.905 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:18.899+0000 7f0089600640 1 -- 172.21.15.32:0/1175115579 <== mon.0 v2:172.21.15.32:3300/0 4 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f0094003db0 con 0x55ee3544e390 2024-08-03T20:49:18.905 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:18.899+0000 7f0089600640 1 -- 172.21.15.32:0/1175115579 <== mon.0 v2:172.21.15.32:3300/0 5 ==== fsmap(e 25) ==== 3502+0+0 (secure 0 0 0) 0x7f009404b030 con 0x55ee3544e390 2024-08-03T20:49:18.905 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:18.899+0000 7f00aa19f740 10 client.5217 fetch_fsmap finished waiting for FSMap version 25 2024-08-03T20:49:18.905 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:18.899+0000 7f00aa19f740 10 client.5217 resolve_mds: resolved 2:0 to role '2:0' aka daemon mds.a 2024-08-03T20:49:18.905 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:18.899+0000 7f00aa19f740 20 client.5217 populate_metadata read hostname 'smithi032' 2024-08-03T20:49:18.905 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:18.899+0000 7f00aa19f740 1 --2- 172.21.15.32:0/1175115579 >> [v2:172.21.15.32:6832/211300533,v1:172.21.15.32:6835/211300533] conn(0x55ee3540cf60 0x55ee357f3dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:49:18.906 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:18.899+0000 7f00aa19f740 4 client.5217 mds_command: new command op to 5070 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-08-03T20:49:18.906 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:18.899+0000 7f00aa19f740 1 -- 172.21.15.32:0/1175115579 --> [v2:172.21.15.32:6832/211300533,v1:172.21.15.32:6835/211300533] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ee357693c0 con 0x55ee3540cf60 2024-08-03T20:49:18.906 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:18.899+0000 7f009be00640 1 --2- 172.21.15.32:0/1175115579 >> [v2:172.21.15.32:6832/211300533,v1:172.21.15.32:6835/211300533] conn(0x55ee3540cf60 0x55ee357f3dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:49:18.906 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:18.900+0000 7f0089600640 10 client.5217 ms_handle_connect on v2:172.21.15.32:6832/211300533 2024-08-03T20:49:18.906 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:18.900+0000 7f009be00640 1 --2- 172.21.15.32:0/1175115579 >> [v2:172.21.15.32:6832/211300533,v1:172.21.15.32:6835/211300533] conn(0x55ee3540cf60 0x55ee357f3dd0 crc :-1 s=READY pgs=21 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=def4b65fac14d6e server_cookie=a7ec006cce725e4c in_seq=0 out_seq=0 2024-08-03T20:49:18.908 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:18.901+0000 7f0089600640 1 -- 172.21.15.32:0/1175115579 <== mds.0 v2:172.21.15.32:6832/211300533 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55ee357693c0 con 0x55ee3540cf60 2024-08-03T20:49:18.908 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:18.901+0000 7f0089600640 10 client.5217 handle_command_reply: tid=0 2024-08-03T20:49:18.908 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:18.901+0000 7f00aa19f740 1 client.5217 shutdown 2024-08-03T20:49:18.908 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:18.901+0000 7f00aa19f740 1 -- 172.21.15.32:0/1175115579 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ee3544e390 msgr2=0x55ee3582f2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:49:18.908 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:18.901+0000 7f00aa19f740 1 --2- 172.21.15.32:0/1175115579 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ee3544e390 0x55ee3582f2d0 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f0094006010 tx=0x7f009402fb80 comp rx=0 tx=0).stop 2024-08-03T20:49:18.908 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:18.901+0000 7f00aa19f740 1 -- 172.21.15.32:0/1175115579 shutdown_connections 2024-08-03T20:49:18.908 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:18.901+0000 7f00aa19f740 1 --2- 172.21.15.32:0/1175115579 >> [v2:172.21.15.32:6832/211300533,v1:172.21.15.32:6835/211300533] conn(0x55ee3540cf60 0x55ee357f3dd0 crc :-1 s=READY pgs=21 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:49:18.908 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:18.901+0000 7f00aa19f740 1 --2- 172.21.15.32:0/1175115579 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ee3544e390 0x55ee3582f2d0 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:49:18.908 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:18.901+0000 7f00aa19f740 1 --2- 172.21.15.32:0/1175115579 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55ee3542d6b0 0x55ee3583b990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:49:18.908 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:18.901+0000 7f00aa19f740 1 --2- 172.21.15.32:0/1175115579 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55ee354254d0 0x55ee358510c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:49:18.908 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:18.901+0000 7f0089600640 0 client.5217 ms_handle_reset on v2:172.21.15.32:6832/211300533 2024-08-03T20:49:18.909 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:18.901+0000 7f00aa19f740 1 -- 172.21.15.32:0/1175115579 >> 172.21.15.32:0/1175115579 conn(0x55ee354424e0 msgr2=0x55ee3580eb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:49:18.909 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:18.901+0000 7f00aa19f740 1 -- 172.21.15.32:0/1175115579 shutdown_connections 2024-08-03T20:49:18.909 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:18.901+0000 7f00aa19f740 1 -- 172.21.15.32:0/1175115579 wait complete. 2024-08-03T20:49:18.909 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:18.902+0000 7f00aa19f740 20 client.5217 trim_cache size 0 max 16384 2024-08-03T20:49:18.923 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:18.921+0000 7f00aa19f740 1 Processor -- start 2024-08-03T20:49:18.923 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:18.921+0000 7f00aa19f740 1 -- start start 2024-08-03T20:49:18.924 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:18.922+0000 7f00aa19f740 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55ee354254d0 0x55ee3580c8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:49:18.924 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:18.922+0000 7f00aa19f740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ee3542d6b0 0x55ee35815570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:49:18.925 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:18.922+0000 7f00aa19f740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55ee3544e390 0x55ee35815ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:49:18.925 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:18.922+0000 7f00aa19f740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f009c1fd9a0 con 0x55ee3542d6b0 2024-08-03T20:49:18.925 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:18.922+0000 7f00aa19f740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f009c0b5290 con 0x55ee3544e390 2024-08-03T20:49:18.925 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:18.922+0000 7f00aa19f740 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7f009c0cf340 con 0x55ee354254d0 2024-08-03T20:49:18.925 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:18.923+0000 7f0093e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ee3542d6b0 0x55ee35815570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:49:18.925 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:18.923+0000 7f0093e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ee3542d6b0 0x55ee35815570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:34624/0 (socket says 172.21.15.32:34624) 2024-08-03T20:49:18.925 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:18.923+0000 7f0093e00640 1 -- 172.21.15.32:0/232408234 learned_addr learned my addr 172.21.15.32:0/232408234 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:49:18.925 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:18.923+0000 7f009be00640 1 --2- 172.21.15.32:0/232408234 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55ee354254d0 0x55ee3580c8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:49:18.925 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:18.923+0000 7f00a1000640 1 --2- 172.21.15.32:0/232408234 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55ee3544e390 0x55ee35815ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:49:18.925 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:18.923+0000 7f0093e00640 1 -- 172.21.15.32:0/232408234 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55ee354254d0 msgr2=0x55ee3580c8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:49:18.925 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:18.923+0000 7f0093e00640 1 --2- 172.21.15.32:0/232408234 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55ee354254d0 0x55ee3580c8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:49:18.925 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:18.923+0000 7f0093e00640 1 -- 172.21.15.32:0/232408234 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55ee3544e390 msgr2=0x55ee35815ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:49:18.925 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:18.923+0000 7f0093e00640 1 --2- 172.21.15.32:0/232408234 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55ee3544e390 0x55ee35815ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:49:18.925 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:18.923+0000 7f0093e00640 1 -- 172.21.15.32:0/232408234 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ee35834860 con 0x55ee3542d6b0 2024-08-03T20:49:18.926 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:18.923+0000 7f0093e00640 1 --2- 172.21.15.32:0/232408234 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ee3542d6b0 0x55ee35815570 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f0084097370 tx=0x7f0084007f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=46c689d68dedeab3 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:49:18.927 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:18.923+0000 7f008be00640 1 -- 172.21.15.32:0/232408234 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f008401d040 con 0x55ee3542d6b0 2024-08-03T20:49:18.927 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:18.924+0000 7f008be00640 1 -- 172.21.15.32:0/232408234 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f0084065f50 con 0x55ee3542d6b0 2024-08-03T20:49:18.927 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:18.924+0000 7f008be00640 1 -- 172.21.15.32:0/232408234 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0084083310 con 0x55ee3542d6b0 2024-08-03T20:49:18.927 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:18.924+0000 7f00aa19f740 1 -- 172.21.15.32:0/232408234 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ee3542d6b0 msgr2=0x55ee35815570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:49:18.927 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:18.924+0000 7f00aa19f740 1 --2- 172.21.15.32:0/232408234 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ee3542d6b0 0x55ee35815570 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f0084097370 tx=0x7f0084007f90 comp rx=0 tx=0).stop 2024-08-03T20:49:18.927 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:18.924+0000 7f00aa19f740 1 -- 172.21.15.32:0/232408234 shutdown_connections 2024-08-03T20:49:18.927 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:18.924+0000 7f00aa19f740 1 --2- 172.21.15.32:0/232408234 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55ee3544e390 0x55ee35815ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:49:18.927 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:18.924+0000 7f00aa19f740 1 --2- 172.21.15.32:0/232408234 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ee3542d6b0 0x55ee35815570 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:49:18.927 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:18.924+0000 7f00aa19f740 1 --2- 172.21.15.32:0/232408234 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55ee354254d0 0x55ee3580c8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:49:18.928 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:18.924+0000 7f00aa19f740 1 -- 172.21.15.32:0/232408234 >> 172.21.15.32:0/232408234 conn(0x55ee354424e0 msgr2=0x55ee357f5140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:49:18.928 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:18.924+0000 7f00aa19f740 1 -- 172.21.15.32:0/232408234 shutdown_connections 2024-08-03T20:49:18.928 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:18.924+0000 7f00aa19f740 1 -- 172.21.15.32:0/232408234 wait complete. 2024-08-03T20:49:18.928 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:18.925+0000 7f00aa19f740 1 -- start start 2024-08-03T20:49:18.928 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:18.925+0000 7f00aa19f740 1 Processor -- start 2024-08-03T20:49:18.930 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:18.927+0000 7f00aa19f740 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55ee354254d0 0x55ee358b49a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:49:18.930 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:18.927+0000 7f00aa19f740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ee3542d6b0 0x55ee357ee280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:49:18.931 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:18.927+0000 7f00aa19f740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55ee3544e390 0x55ee357ee7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:49:18.931 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:18.927+0000 7f00aa19f740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f009c0cf990 con 0x55ee3542d6b0 2024-08-03T20:49:18.931 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:18.927+0000 7f00aa19f740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x55ee358165a0 con 0x55ee3544e390 2024-08-03T20:49:18.931 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:18.927+0000 7f00aa19f740 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x55ee35860010 con 0x55ee354254d0 2024-08-03T20:49:18.931 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:18.927+0000 7f00aa19f740 10 client.0 _ll_register_callbacks cb 0x7f008c0092d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-03T20:49:18.932 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:18.927+0000 7f0093e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ee3542d6b0 0x55ee357ee280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:49:18.932 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:18.927+0000 7f0093e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ee3542d6b0 0x55ee357ee280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:34640/0 (socket says 172.21.15.32:34640) 2024-08-03T20:49:18.932 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:18.927+0000 7f0093e00640 1 -- 172.21.15.32:0/2837426456 learned_addr learned my addr 172.21.15.32:0/2837426456 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:49:18.932 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:18.927+0000 7f009be00640 1 --2- 172.21.15.32:0/2837426456 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55ee354254d0 0x55ee358b49a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:49:18.932 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:18.927+0000 7f00a1000640 1 --2- 172.21.15.32:0/2837426456 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55ee3544e390 0x55ee357ee7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:49:18.932 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:18.928+0000 7f0093e00640 1 -- 172.21.15.32:0/2837426456 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55ee354254d0 msgr2=0x55ee358b49a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:49:18.933 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:18.928+0000 7f0093e00640 1 --2- 172.21.15.32:0/2837426456 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55ee354254d0 0x55ee358b49a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:49:18.933 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:18.928+0000 7f0093e00640 1 -- 172.21.15.32:0/2837426456 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55ee3544e390 msgr2=0x55ee357ee7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:49:18.933 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:18.928+0000 7f0093e00640 1 --2- 172.21.15.32:0/2837426456 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55ee3544e390 0x55ee357ee7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:49:18.933 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:18.928+0000 7f0093e00640 1 -- 172.21.15.32:0/2837426456 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ee357ef070 con 0x55ee3542d6b0 2024-08-03T20:49:18.933 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:18.928+0000 7f0093e00640 1 --2- 172.21.15.32:0/2837426456 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ee3542d6b0 0x55ee357ee280 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f008405da80 tx=0x7f00840832f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d510468bead17382 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:49:18.934 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:18.928+0000 7f0089600640 10 client.0 ms_handle_connect on v2:172.21.15.32:3300/0 2024-08-03T20:49:18.934 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:18.928+0000 7f0089600640 1 -- 172.21.15.32:0/2837426456 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f008401d040 con 0x55ee3542d6b0 2024-08-03T20:49:18.934 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:18.928+0000 7f0089600640 1 -- 172.21.15.32:0/2837426456 <== mon.0 v2:172.21.15.32:3300/0 2 ==== osd_map(33..33 src has 1..33) ==== 5795+0+0 (secure 0 0 0) 0x7f008400fd50 con 0x55ee3542d6b0 2024-08-03T20:49:18.934 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:18.928+0000 7f00aa19f740 1 -- 172.21.15.32:0/2837426456 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0084065f50 con 0x55ee3542d6b0 2024-08-03T20:49:18.934 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:18.928+0000 7f0089600640 1 -- 172.21.15.32:0/2837426456 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f0084070050 con 0x55ee3542d6b0 2024-08-03T20:49:18.934 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:18.928+0000 7f00aa19f740 10 client.5223 fetch_fsmap learned FSMap version 25 2024-08-03T20:49:18.935 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:18.928+0000 7f00aa19f740 1 -- 172.21.15.32:0/2837426456 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=2+}) -- 0x55ee358bb920 con 0x55ee3542d6b0 2024-08-03T20:49:18.935 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:18.929+0000 7f0089600640 1 -- 172.21.15.32:0/2837426456 <== mon.0 v2:172.21.15.32:3300/0 4 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f0084065f50 con 0x55ee3542d6b0 2024-08-03T20:49:18.937 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:18.929+0000 7f0089600640 1 -- 172.21.15.32:0/2837426456 <== mon.0 v2:172.21.15.32:3300/0 5 ==== fsmap(e 25) ==== 3502+0+0 (secure 0 0 0) 0x7f0084060c60 con 0x55ee3542d6b0 2024-08-03T20:49:18.937 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:18.929+0000 7f00aa19f740 10 client.5223 fetch_fsmap finished waiting for FSMap version 25 2024-08-03T20:49:18.937 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:18.929+0000 7f00aa19f740 10 client.5223 resolve_mds: resolved 2:0 to role '2:0' aka daemon mds.a 2024-08-03T20:49:18.937 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:18.929+0000 7f00aa19f740 20 client.5223 populate_metadata read hostname 'smithi032' 2024-08-03T20:49:18.937 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:18.929+0000 7f00aa19f740 1 --2- 172.21.15.32:0/2837426456 >> [v2:172.21.15.32:6832/211300533,v1:172.21.15.32:6835/211300533] conn(0x55ee35856d30 0x55ee35815570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:49:18.937 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:18.929+0000 7f00aa19f740 4 client.5223 mds_command: new command op to 5070 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-08-03T20:49:18.937 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:18.929+0000 7f00aa19f740 1 -- 172.21.15.32:0/2837426456 --> [v2:172.21.15.32:6832/211300533,v1:172.21.15.32:6835/211300533] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55ee35764e60 con 0x55ee35856d30 2024-08-03T20:49:18.937 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:18.929+0000 7f009be00640 1 --2- 172.21.15.32:0/2837426456 >> [v2:172.21.15.32:6832/211300533,v1:172.21.15.32:6835/211300533] conn(0x55ee35856d30 0x55ee35815570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:49:18.937 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:18.930+0000 7f009be00640 1 --2- 172.21.15.32:0/2837426456 >> [v2:172.21.15.32:6832/211300533,v1:172.21.15.32:6835/211300533] conn(0x55ee35856d30 0x55ee35815570 crc :-1 s=READY pgs=22 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=df3b4d63dfc20060 server_cookie=26a56bdb5202e93e in_seq=0 out_seq=0 2024-08-03T20:49:18.937 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:18.930+0000 7f0089600640 10 client.5223 ms_handle_connect on v2:172.21.15.32:6832/211300533 2024-08-03T20:49:18.937 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:18.930+0000 7f0089600640 1 -- 172.21.15.32:0/2837426456 <== mds.0 v2:172.21.15.32:6832/211300533 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55ee35764e60 con 0x55ee35856d30 2024-08-03T20:49:18.937 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:18.930+0000 7f0089600640 10 client.5223 handle_command_reply: tid=0 2024-08-03T20:49:18.938 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:18.930+0000 7f00aa19f740 1 client.5223 shutdown 2024-08-03T20:49:18.938 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:18.930+0000 7f00aa19f740 1 -- 172.21.15.32:0/2837426456 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ee3542d6b0 msgr2=0x55ee357ee280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:49:18.938 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:18.930+0000 7f00aa19f740 1 --2- 172.21.15.32:0/2837426456 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ee3542d6b0 0x55ee357ee280 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f008405da80 tx=0x7f00840832f0 comp rx=0 tx=0).stop 2024-08-03T20:49:18.938 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:18.930+0000 7f00aa19f740 1 -- 172.21.15.32:0/2837426456 shutdown_connections 2024-08-03T20:49:18.938 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:18.930+0000 7f00aa19f740 1 --2- 172.21.15.32:0/2837426456 >> [v2:172.21.15.32:6832/211300533,v1:172.21.15.32:6835/211300533] conn(0x55ee35856d30 0x55ee35815570 crc :-1 s=READY pgs=22 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:49:18.938 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:18.930+0000 7f00aa19f740 1 --2- 172.21.15.32:0/2837426456 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55ee3544e390 0x55ee357ee7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:49:18.938 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:18.930+0000 7f00aa19f740 1 --2- 172.21.15.32:0/2837426456 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ee3542d6b0 0x55ee357ee280 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:49:18.938 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:18.930+0000 7f00aa19f740 1 --2- 172.21.15.32:0/2837426456 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55ee354254d0 0x55ee358b49a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:49:18.938 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:18.930+0000 7f00aa19f740 1 -- 172.21.15.32:0/2837426456 >> 172.21.15.32:0/2837426456 conn(0x55ee354424e0 msgr2=0x55ee3580c8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:49:18.938 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:18.930+0000 7f0089600640 0 client.5223 ms_handle_reset on v2:172.21.15.32:6832/211300533 2024-08-03T20:49:18.939 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:18.930+0000 7f00aa19f740 1 -- 172.21.15.32:0/2837426456 shutdown_connections 2024-08-03T20:49:18.939 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:18.931+0000 7f00aa19f740 1 -- 172.21.15.32:0/2837426456 wait complete. 2024-08-03T20:49:18.939 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:18.931+0000 7f00aa19f740 20 client.5223 trim_cache size 0 max 16384 2024-08-03T20:49:18.939 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-08-03T20:49:18.940 INFO:teuthology.orchestra.run.smithi032.stdout: "status": "no active scrubs running", 2024-08-03T20:49:18.940 INFO:teuthology.orchestra.run.smithi032.stdout: "scrubs": {} 2024-08-03T20:49:18.940 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-08-03T20:49:18.941 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:18.931+0000 7f00a1a00640 1 -- 172.21.15.32:0/3241493742 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f0078077010 msgr2=0x7f0078079520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:49:18.941 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:18.931+0000 7f00a1a00640 1 --2- 172.21.15.32:0/3241493742 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f0078077010 0x7f0078079520 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f008c006fd0 tx=0x7f008c009060 comp rx=0 tx=0).stop 2024-08-03T20:49:18.942 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:18.931+0000 7f00a1a00640 1 -- 172.21.15.32:0/3241493742 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f009c0749b0 msgr2=0x7f009c1511f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:49:18.942 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:18.931+0000 7f00a1a00640 1 --2- 172.21.15.32:0/3241493742 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f009c0749b0 0x7f009c1511f0 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f008400c970 tx=0x7f008400ce30 comp rx=0 tx=0).stop 2024-08-03T20:49:18.942 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:18.931+0000 7f00a1a00640 1 -- 172.21.15.32:0/3241493742 shutdown_connections 2024-08-03T20:49:18.942 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:18.931+0000 7f00a1a00640 1 --2- 172.21.15.32:0/3241493742 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f0078077010 0x7f0078079520 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:49:18.942 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:18.931+0000 7f00a1a00640 1 --2- 172.21.15.32:0/3241493742 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f009c0cb7f0 0x7f009c0c9f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:49:18.942 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:18.931+0000 7f00a1a00640 1 --2- 172.21.15.32:0/3241493742 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f009c0749b0 0x7f009c1511f0 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:49:18.943 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:18.931+0000 7f00a1a00640 1 --2- 172.21.15.32:0/3241493742 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f009c0745a0 0x7f009c0c7ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:49:18.943 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:18.931+0000 7f00a1a00640 1 -- 172.21.15.32:0/3241493742 >> 172.21.15.32:0/3241493742 conn(0x7f009c0d0120 msgr2=0x7f009c0aef40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:49:18.943 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:18.931+0000 7f00a1a00640 1 -- 172.21.15.32:0/3241493742 shutdown_connections 2024-08-03T20:49:18.943 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:18.931+0000 7f00a1a00640 1 -- 172.21.15.32:0/3241493742 wait complete. 2024-08-03T20:49:18.958 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-08-03T20:49:18.958 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-08-03T20:49:19.188 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:19.186+0000 7f5622c00640 1 -- 172.21.15.32:0/4015100259 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f561c15fff0 msgr2=0x7f561c160400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:49:19.188 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:19.186+0000 7f5622c00640 1 --2- 172.21.15.32:0/4015100259 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f561c15fff0 0x7f561c160400 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f561c071700 tx=0x7f561402aac0 comp rx=0 tx=0).stop 2024-08-03T20:49:19.188 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:19.186+0000 7f5622c00640 1 -- 172.21.15.32:0/4015100259 shutdown_connections 2024-08-03T20:49:19.188 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:19.186+0000 7f5622c00640 1 --2- 172.21.15.32:0/4015100259 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f561c15fff0 0x7f561c160400 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:49:19.188 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:19.186+0000 7f5622c00640 1 --2- 172.21.15.32:0/4015100259 >> v2:172.21.15.118:3300/0 conn(0x7f561c06e930 0x7f561c144f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:49:19.188 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:19.186+0000 7f5622c00640 1 -- 172.21.15.32:0/4015100259 >> 172.21.15.32:0/4015100259 conn(0x7f561c0b1bd0 msgr2=0x7f561c0b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:49:19.188 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:19.186+0000 7f5622c00640 1 -- 172.21.15.32:0/4015100259 shutdown_connections 2024-08-03T20:49:19.188 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:19.186+0000 7f5622c00640 1 -- 172.21.15.32:0/4015100259 wait complete. 2024-08-03T20:49:19.189 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:19.188+0000 7f5622c00640 1 Processor -- start 2024-08-03T20:49:19.189 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:19.188+0000 7f5622c00640 1 -- start start 2024-08-03T20:49:19.192 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:19.190+0000 7f5622c00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f561c06e930 0x7f561c142ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:49:19.192 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:19.190+0000 7f5622c00640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f561c073e60 0x7f561c15be70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:49:19.192 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:19.190+0000 7f5622c00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f561c15fff0 0x7f561c15ca40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:49:19.192 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:19.190+0000 7f5622c00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f561c0cf840 con 0x7f561c15fff0 2024-08-03T20:49:19.192 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:19.190+0000 7f5622c00640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f561c145f30 con 0x7f561c06e930 2024-08-03T20:49:19.192 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:19.190+0000 7f5622c00640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7f561c0cf9c0 con 0x7f561c073e60 2024-08-03T20:49:19.192 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:19.190+0000 7f5620c00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f561c15fff0 0x7f561c15ca40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:49:19.192 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:19.190+0000 7f5620c00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f561c15fff0 0x7f561c15ca40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:34652/0 (socket says 172.21.15.32:34652) 2024-08-03T20:49:19.193 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:19.190+0000 7f5620c00640 1 -- 172.21.15.32:0/3393540408 learned_addr learned my addr 172.21.15.32:0/3393540408 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:49:19.193 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:19.190+0000 7f561be00640 1 --2- 172.21.15.32:0/3393540408 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f561c06e930 0x7f561c142ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:49:19.193 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:19.190+0000 7f561b400640 1 --2- 172.21.15.32:0/3393540408 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f561c073e60 0x7f561c15be70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:49:19.193 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:19.190+0000 7f5620c00640 1 -- 172.21.15.32:0/3393540408 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f561c073e60 msgr2=0x7f561c15be70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:49:19.193 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:19.190+0000 7f5620c00640 1 --2- 172.21.15.32:0/3393540408 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f561c073e60 0x7f561c15be70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:49:19.193 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:19.190+0000 7f5620c00640 1 -- 172.21.15.32:0/3393540408 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f561c06e930 msgr2=0x7f561c142ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:49:19.193 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:19.190+0000 7f5620c00640 1 --2- 172.21.15.32:0/3393540408 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f561c06e930 0x7f561c142ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:49:19.193 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:19.190+0000 7f5620c00640 1 -- 172.21.15.32:0/3393540408 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f561c15c3b0 con 0x7f561c15fff0 2024-08-03T20:49:19.193 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:19.191+0000 7f5620c00640 1 --2- 172.21.15.32:0/3393540408 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f561c15fff0 0x7f561c15ca40 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f5614009d60 tx=0x7f5614003ce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:49:19.194 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:19.191+0000 7f5613e00640 1 -- 172.21.15.32:0/3393540408 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5614044070 con 0x7f561c15fff0 2024-08-03T20:49:19.194 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:19.191+0000 7f5613e00640 1 -- 172.21.15.32:0/3393540408 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f5614004250 con 0x7f561c15fff0 2024-08-03T20:49:19.194 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:19.191+0000 7f5622c00640 1 -- 172.21.15.32:0/3393540408 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f561c0cba80 con 0x7f561c15fff0 2024-08-03T20:49:19.194 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:19.191+0000 7f5613e00640 1 -- 172.21.15.32:0/3393540408 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56140046a0 con 0x7f561c15fff0 2024-08-03T20:49:19.194 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:19.191+0000 7f5622c00640 1 -- 172.21.15.32:0/3393540408 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f561c143fe0 con 0x7f561c15fff0 2024-08-03T20:49:19.194 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:19.192+0000 7f5622c00640 1 -- 172.21.15.32:0/3393540408 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f561c160be0 con 0x7f561c15fff0 2024-08-03T20:49:19.195 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:19.193+0000 7f5613e00640 1 -- 172.21.15.32:0/3393540408 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f5614036400 con 0x7f561c15fff0 2024-08-03T20:49:19.196 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:19.194+0000 7f5613e00640 1 --2- 172.21.15.32:0/3393540408 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f55fc077010 0x7f55fc079520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:49:19.196 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:19.194+0000 7f5613e00640 1 -- 172.21.15.32:0/3393540408 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(33..33 src has 1..33) ==== 5795+0+0 (secure 0 0 0) 0x7f56140b90c0 con 0x7f561c15fff0 2024-08-03T20:49:19.196 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:19.195+0000 7f561be00640 1 --2- 172.21.15.32:0/3393540408 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f55fc077010 0x7f55fc079520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:49:19.197 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:19.195+0000 7f561be00640 1 --2- 172.21.15.32:0/3393540408 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f55fc077010 0x7f55fc079520 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f560c00bb50 tx=0x7f560c00b2e0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:49:19.200 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:19.199+0000 7f5613e00640 1 -- 172.21.15.32:0/3393540408 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7f5614032030 con 0x7f561c15fff0 2024-08-03T20:49:19.450 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:19.448+0000 7f5622c00640 1 -- 172.21.15.32:0/3393540408 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f561c0b2160 con 0x7f561c15fff0 2024-08-03T20:49:19.450 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:19.449+0000 7f5613e00640 1 -- 172.21.15.32:0/3393540408 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v25) ==== 53+0+73 (secure 0 0 0) 0x7f5614081c10 con 0x7f561c15fff0 2024-08-03T20:49:19.451 INFO:teuthology.orchestra.run.smithi032.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-08-03T20:49:19.453 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:19.451+0000 7f5622c00640 1 -- 172.21.15.32:0/3393540408 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f55fc077010 msgr2=0x7f55fc079520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:49:19.453 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:19.451+0000 7f5622c00640 1 --2- 172.21.15.32:0/3393540408 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f55fc077010 0x7f55fc079520 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f560c00bb50 tx=0x7f560c00b2e0 comp rx=0 tx=0).stop 2024-08-03T20:49:19.453 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:19.451+0000 7f5622c00640 1 -- 172.21.15.32:0/3393540408 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f561c15fff0 msgr2=0x7f561c15ca40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:49:19.453 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:19.451+0000 7f5622c00640 1 --2- 172.21.15.32:0/3393540408 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f561c15fff0 0x7f561c15ca40 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f5614009d60 tx=0x7f5614003ce0 comp rx=0 tx=0).stop 2024-08-03T20:49:19.453 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:19.451+0000 7f5622c00640 1 -- 172.21.15.32:0/3393540408 shutdown_connections 2024-08-03T20:49:19.453 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:19.451+0000 7f5622c00640 1 --2- 172.21.15.32:0/3393540408 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f55fc077010 0x7f55fc079520 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:49:19.454 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:19.451+0000 7f5622c00640 1 --2- 172.21.15.32:0/3393540408 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f561c15fff0 0x7f561c15ca40 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:49:19.454 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:19.451+0000 7f5622c00640 1 --2- 172.21.15.32:0/3393540408 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f561c073e60 0x7f561c15be70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:49:19.454 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:19.451+0000 7f5622c00640 1 --2- 172.21.15.32:0/3393540408 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f561c06e930 0x7f561c142ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:49:19.454 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:19.451+0000 7f5622c00640 1 -- 172.21.15.32:0/3393540408 >> 172.21.15.32:0/3393540408 conn(0x7f561c0b1bd0 msgr2=0x7f561c140cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:49:19.454 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:19.451+0000 7f5622c00640 1 -- 172.21.15.32:0/3393540408 shutdown_connections 2024-08-03T20:49:19.454 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:19.451+0000 7f5622c00640 1 -- 172.21.15.32:0/3393540408 wait complete. 2024-08-03T20:49:19.479 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-08-03T20:49:19.479 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-08-03T20:49:19.479 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi032.front.sepia.ceph.com 2024-08-03T20:49:19.479 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-08-03T20:49:19.479 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-08-03T20:49:19.479 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-08-03T20:49:19.479 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2024-08-03T20:49:19.479 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-08-03T20:49:19.479 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-08-03T20:49:19.479 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-08-03T20:49:19.480 DEBUG:teuthology.orchestra.run.smithi032:> ip addr 2024-08-03T20:49:19.516 INFO:teuthology.orchestra.run.smithi032.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-08-03T20:49:19.516 INFO:teuthology.orchestra.run.smithi032.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-08-03T20:49:19.516 INFO:teuthology.orchestra.run.smithi032.stdout: inet 127.0.0.1/8 scope host lo 2024-08-03T20:49:19.516 INFO:teuthology.orchestra.run.smithi032.stdout: valid_lft forever preferred_lft forever 2024-08-03T20:49:19.516 INFO:teuthology.orchestra.run.smithi032.stdout: inet6 ::1/128 scope host 2024-08-03T20:49:19.516 INFO:teuthology.orchestra.run.smithi032.stdout: valid_lft forever preferred_lft forever 2024-08-03T20:49:19.516 INFO:teuthology.orchestra.run.smithi032.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-08-03T20:49:19.516 INFO:teuthology.orchestra.run.smithi032.stdout: link/ether 0c:c4:7a:6b:fd:0c brd ff:ff:ff:ff:ff:ff 2024-08-03T20:49:19.516 INFO:teuthology.orchestra.run.smithi032.stdout: altname enp7s0f0 2024-08-03T20:49:19.516 INFO:teuthology.orchestra.run.smithi032.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-08-03T20:49:19.517 INFO:teuthology.orchestra.run.smithi032.stdout: link/ether 0c:c4:7a:6b:fd:0d brd ff:ff:ff:ff:ff:ff 2024-08-03T20:49:19.517 INFO:teuthology.orchestra.run.smithi032.stdout: altname enp7s0f1 2024-08-03T20:49:19.517 INFO:teuthology.orchestra.run.smithi032.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-08-03T20:49:19.517 INFO:teuthology.orchestra.run.smithi032.stdout: link/ether ac:1f:6b:f8:26:02 brd ff:ff:ff:ff:ff:ff 2024-08-03T20:49:19.517 INFO:teuthology.orchestra.run.smithi032.stdout: altname enp1s0f0 2024-08-03T20:49:19.517 INFO:teuthology.orchestra.run.smithi032.stdout: inet 172.21.15.32/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-08-03T20:49:19.517 INFO:teuthology.orchestra.run.smithi032.stdout: valid_lft 42397sec preferred_lft 42397sec 2024-08-03T20:49:19.517 INFO:teuthology.orchestra.run.smithi032.stdout: inet6 fe80::ae1f:6bff:fef8:2602/64 scope link noprefixroute 2024-08-03T20:49:19.517 INFO:teuthology.orchestra.run.smithi032.stdout: valid_lft forever preferred_lft forever 2024-08-03T20:49:19.517 INFO:teuthology.orchestra.run.smithi032.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-08-03T20:49:19.517 INFO:teuthology.orchestra.run.smithi032.stdout: link/ether ac:1f:6b:f8:26:03 brd ff:ff:ff:ff:ff:ff 2024-08-03T20:49:19.518 INFO:teuthology.orchestra.run.smithi032.stdout: altname enp1s0f1 2024-08-03T20:49:19.518 INFO:teuthology.orchestra.run.smithi032.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-08-03T20:49:19.518 INFO:teuthology.orchestra.run.smithi032.stdout: link/ether 0a:23:2b:ca:2a:93 brd ff:ff:ff:ff:ff:ff 2024-08-03T20:49:19.518 INFO:teuthology.orchestra.run.smithi032.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-08-03T20:49:19.518 INFO:teuthology.orchestra.run.smithi032.stdout: valid_lft forever preferred_lft forever 2024-08-03T20:49:19.518 INFO:teuthology.orchestra.run.smithi032.stdout: inet6 fe80::d804:7cff:fed9:7ba4/64 scope link 2024-08-03T20:49:19.518 INFO:teuthology.orchestra.run.smithi032.stdout: valid_lft forever preferred_lft forever 2024-08-03T20:49:19.518 INFO:teuthology.orchestra.run.smithi032.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-08-03T20:49:19.518 INFO:teuthology.orchestra.run.smithi032.stdout: link/ether 22:86:28:30:4e:62 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-08-03T20:49:19.519 INFO:teuthology.orchestra.run.smithi032.stdout: inet6 fe80::2086:28ff:fe30:4e62/64 scope link 2024-08-03T20:49:19.519 INFO:teuthology.orchestra.run.smithi032.stdout: valid_lft forever preferred_lft forever 2024-08-03T20:49:19.519 INFO:teuthology.orchestra.run.smithi032.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-08-03T20:49:19.519 INFO:teuthology.orchestra.run.smithi032.stdout: link/ether 0a:23:2b:ca:2a:93 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-08-03T20:49:19.519 INFO:teuthology.orchestra.run.smithi032.stdout: inet6 fe80::823:2bff:feca:2a93/64 scope link 2024-08-03T20:49:19.519 INFO:teuthology.orchestra.run.smithi032.stdout: valid_lft forever preferred_lft forever 2024-08-03T20:49:19.519 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-08-03T20:49:19.520 DEBUG:teuthology.orchestra.run.smithi032:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-08-03T20:49:19.610 INFO:teuthology.orchestra.run.smithi032.stdout:1 2024-08-03T20:49:19.613 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-08-03T20:49:19.613 DEBUG:teuthology.orchestra.run.smithi032:> route 2024-08-03T20:49:19.649 INFO:teuthology.orchestra.run.smithi032.stdout:Kernel IP routing table 2024-08-03T20:49:19.649 INFO:teuthology.orchestra.run.smithi032.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-08-03T20:49:19.649 INFO:teuthology.orchestra.run.smithi032.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-08-03T20:49:19.649 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-08-03T20:49:19.649 INFO:teuthology.orchestra.run.smithi032.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-08-03T20:49:19.650 DEBUG:teuthology.orchestra.run.smithi032:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-08-03T20:49:19.650 DEBUG:teuthology.orchestra.run.smithi032:> set -e 2024-08-03T20:49:19.650 DEBUG:teuthology.orchestra.run.smithi032:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-08-03T20:49:19.650 DEBUG:teuthology.orchestra.run.smithi032:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-08-03T20:49:19.650 DEBUG:teuthology.orchestra.run.smithi032:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-08-03T20:49:19.650 DEBUG:teuthology.orchestra.run.smithi032:> ') 2024-08-03T20:49:19.769 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:19 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-08-03T20:49:19.840 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:19 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-08-03T20:49:19.848 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-08-03T20:49:19.848 DEBUG:teuthology.orchestra.run.smithi032:> ip netns list 2024-08-03T20:49:19.883 INFO:teuthology.orchestra.run.smithi032.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-08-03T20:49:19.884 INFO:teuthology.orchestra.run.smithi032.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-08-03T20:49:19.884 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-08-03T20:49:19.884 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-08-03T20:49:19.884 DEBUG:teuthology.orchestra.run.smithi032:> sudo ip link set brx.0 up 2024-08-03T20:49:19.981 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-08-03T20:49:19.981 DEBUG:teuthology.orchestra.run.smithi032:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-08-03T20:49:20.016 INFO:teuthology.orchestra.run.smithi032.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-08-03T20:49:20.017 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-08-03T20:49:20.017 DEBUG:teuthology.orchestra.run.smithi032:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-08-03T20:49:20.091 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2024-08-03T20:49:20.091 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-08-03T20:49:20.091 DEBUG:teuthology.orchestra.run.smithi032:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-08-03T20:49:20.289 INFO:teuthology.orchestra.run.smithi032.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-08-03T20:49:20.289 INFO:teuthology.orchestra.run.smithi032.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-08-03T20:49:20.289 INFO:teuthology.orchestra.run.smithi032.stdout:invalid new device string format 2024-08-03T20:49:20.309 INFO:teuthology.orchestra.run.smithi032.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-08-03T20:49:20.310 INFO:teuthology.orchestra.run.smithi032.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-08-03T20:49:20.310 INFO:teuthology.orchestra.run.smithi032.stdout:invalid new device string format 2024-08-03T20:49:20.310 INFO:teuthology.orchestra.run.smithi032.stdout:mount.ceph: resolved to: "172.21.15.32:3300,172.21.15.118:3300,172.21.15.118:3301" 2024-08-03T20:49:20.310 INFO:teuthology.orchestra.run.smithi032.stdout:mount.ceph: switching to using v1 address with old syntax 2024-08-03T20:49:20.341 INFO:teuthology.orchestra.run.smithi032.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-08-03T20:49:20.341 INFO:teuthology.orchestra.run.smithi032.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-08-03T20:49:20.341 INFO:teuthology.orchestra.run.smithi032.stdout:invalid new device string format 2024-08-03T20:49:20.341 INFO:teuthology.orchestra.run.smithi032.stdout:mount.ceph: resolved to: "172.21.15.32:3300,172.21.15.118:3300,172.21.15.118:3301" 2024-08-03T20:49:20.341 INFO:teuthology.orchestra.run.smithi032.stdout:mount.ceph: switching to using v1 address with old syntax 2024-08-03T20:49:20.341 INFO:teuthology.orchestra.run.smithi032.stdout:mount.ceph: resolved to: "172.21.15.32:6789,172.21.15.118:6789,172.21.15.118:6790" 2024-08-03T20:49:20.341 INFO:teuthology.orchestra.run.smithi032.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.32:6789,172.21.15.118:6789,172.21.15.118:6790:/ 2024-08-03T20:49:20.341 INFO:teuthology.orchestra.run.smithi032.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,key=0,fsid=40ab141a-01d7-49b9-ae3c-2edec9876e95" will pass to kernel 2024-08-03T20:49:20.342 INFO:teuthology.orchestra.run.smithi032.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2024-08-03T20:49:20.342 INFO:teuthology.orchestra.run.smithi032.stdout: You just mounted a file system that supports labels which does not 2024-08-03T20:49:20.342 INFO:teuthology.orchestra.run.smithi032.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-08-03T20:49:20.342 INFO:teuthology.orchestra.run.smithi032.stdout: applications will generate AVC messages and not be allowed access to 2024-08-03T20:49:20.342 INFO:teuthology.orchestra.run.smithi032.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-08-03T20:49:20.345 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-08-03T20:49:20.345 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-08-03T20:49:20.345 DEBUG:teuthology.orchestra.run.smithi032:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-08-03T20:49:20.401 DEBUG:teuthology.orchestra.run.smithi032:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-08-03T20:49:20.482 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:20 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-08-03T20:49:20.485 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:20 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-08-03T20:49:20.493 INFO:teuthology.orchestra.run.smithi032.stdout:client4868 2024-08-03T20:49:20.493 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-08-03T20:49:20.493 DEBUG:teuthology.orchestra.run.smithi032:> dd if=/proc/self/mounts of=/dev/stdout 2024-08-03T20:49:20.529 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-08-03T20:49:20.529 DEBUG:teuthology.orchestra.run.smithi032:> (cd /home/ubuntu/cephtest/mnt.0 && exec getfattr --only-values -n ceph.dir.files ./subdir) 2024-08-03T20:49:20.606 INFO:teuthology.orchestra.run.smithi032.stdout:1 2024-08-03T20:49:20.606 DEBUG:teuthology.orchestra.run.smithi032:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata setomapval 10000000000.00000000 file_to_be_damaged_head deadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeef 2024-08-03T20:49:20.794 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:20.792+0000 7f3ce3b21f40 1 Processor -- start 2024-08-03T20:49:20.794 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:20.792+0000 7f3ce3b21f40 1 -- start start 2024-08-03T20:49:20.796 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:20.793+0000 7f3ce3b21f40 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55eddf7713f0 0x55eddf786260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:49:20.796 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:20.793+0000 7f3ce3b21f40 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55eddf78cd90 0x55eddf6b52a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:49:20.797 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:20.793+0000 7f3ce3b21f40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55eddf818610 0x55eddf81e8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:49:20.797 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:20.793+0000 7f3ce3b21f40 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55eddf604e30 con 0x55eddf7713f0 2024-08-03T20:49:20.797 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:20.793+0000 7f3ce3b21f40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x55eddf662bf0 con 0x55eddf818610 2024-08-03T20:49:20.797 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:20.793+0000 7f3ce3b21f40 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x55eddf772200 con 0x55eddf78cd90 2024-08-03T20:49:20.797 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:20.794+0000 7f3cdc200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55eddf7713f0 0x55eddf786260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:49:20.797 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:20.794+0000 7f3cdc200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55eddf7713f0 0x55eddf786260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:59910/0 (socket says 172.21.15.32:59910) 2024-08-03T20:49:20.797 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:20.794+0000 7f3cdc200640 1 -- 172.21.15.32:0/903598275 learned_addr learned my addr 172.21.15.32:0/903598275 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:49:20.797 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:20.794+0000 7f3cdcc00640 1 --2- 172.21.15.32:0/903598275 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55eddf818610 0x55eddf81e8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:49:20.797 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:20.794+0000 7f3cdb800640 1 --2- 172.21.15.32:0/903598275 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55eddf78cd90 0x55eddf6b52a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:49:20.797 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:20.794+0000 7f3cdc200640 1 -- 172.21.15.32:0/903598275 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55eddf78cd90 msgr2=0x55eddf6b52a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:49:20.797 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:20.794+0000 7f3cdc200640 1 --2- 172.21.15.32:0/903598275 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55eddf78cd90 0x55eddf6b52a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:49:20.797 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:20.794+0000 7f3cdc200640 1 -- 172.21.15.32:0/903598275 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55eddf818610 msgr2=0x55eddf81e8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:49:20.797 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:20.794+0000 7f3cdc200640 1 --2- 172.21.15.32:0/903598275 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55eddf818610 0x55eddf81e8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:49:20.798 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:20.794+0000 7f3cdc200640 1 -- 172.21.15.32:0/903598275 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55eddf6b57e0 con 0x55eddf7713f0 2024-08-03T20:49:20.798 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:20.794+0000 7f3cdcc00640 1 --2- 172.21.15.32:0/903598275 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55eddf818610 0x55eddf81e8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-03T20:49:20.798 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:20.795+0000 7f3cdc200640 1 --2- 172.21.15.32:0/903598275 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55eddf7713f0 0x55eddf786260 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f3ccc025e40 tx=0x7f3ccc002fb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9bd72bf727b4c771 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:49:20.798 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:20.795+0000 7f3cda400640 1 -- 172.21.15.32:0/903598275 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ccc066070 con 0x55eddf7713f0 2024-08-03T20:49:20.798 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:20.795+0000 7f3cda400640 1 -- 172.21.15.32:0/903598275 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f3ccc04dc20 con 0x55eddf7713f0 2024-08-03T20:49:20.798 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:20.795+0000 7f3cda400640 1 -- 172.21.15.32:0/903598275 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ccc056470 con 0x55eddf7713f0 2024-08-03T20:49:20.799 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:20.796+0000 7f3ce3b21f40 1 -- 172.21.15.32:0/903598275 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55eddf7713f0 msgr2=0x55eddf786260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:49:20.799 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:20.796+0000 7f3ce3b21f40 1 --2- 172.21.15.32:0/903598275 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55eddf7713f0 0x55eddf786260 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f3ccc025e40 tx=0x7f3ccc002fb0 comp rx=0 tx=0).stop 2024-08-03T20:49:20.799 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:20.796+0000 7f3ce3b21f40 1 -- 172.21.15.32:0/903598275 shutdown_connections 2024-08-03T20:49:20.799 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:20.796+0000 7f3ce3b21f40 1 --2- 172.21.15.32:0/903598275 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55eddf818610 0x55eddf81e8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:49:20.799 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:20.796+0000 7f3ce3b21f40 1 --2- 172.21.15.32:0/903598275 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55eddf78cd90 0x55eddf6b52a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:49:20.799 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:20.796+0000 7f3ce3b21f40 1 --2- 172.21.15.32:0/903598275 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55eddf7713f0 0x55eddf786260 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:49:20.799 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:20.796+0000 7f3ce3b21f40 1 -- 172.21.15.32:0/903598275 >> 172.21.15.32:0/903598275 conn(0x55eddf5d6ac0 msgr2=0x55eddf7833a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:49:20.800 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:20.796+0000 7f3ce3b21f40 1 -- 172.21.15.32:0/903598275 shutdown_connections 2024-08-03T20:49:20.800 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:20.796+0000 7f3ce3b21f40 1 -- 172.21.15.32:0/903598275 wait complete. 2024-08-03T20:49:20.800 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:20.797+0000 7f3ce3b21f40 1 Processor -- start 2024-08-03T20:49:20.800 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:20.797+0000 7f3ce3b21f40 1 -- start start 2024-08-03T20:49:20.801 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:20.798+0000 7f3ce3b21f40 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55eddf7713f0 0x55eddf822f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:49:20.801 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:20.798+0000 7f3ce3b21f40 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55eddf78cd90 0x55eddf7849f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:49:20.801 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:20.798+0000 7f3ce3b21f40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55eddf818610 0x55eddf784f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:49:20.801 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:20.798+0000 7f3ce3b21f40 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55eddf6b50a0 con 0x55eddf78cd90 2024-08-03T20:49:20.802 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:20.798+0000 7f3ce3b21f40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x55eddf782a60 con 0x55eddf818610 2024-08-03T20:49:20.802 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:20.798+0000 7f3ce3b21f40 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x55eddf8237f0 con 0x55eddf7713f0 2024-08-03T20:49:20.802 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:20.799+0000 7f3cdb800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55eddf78cd90 0x55eddf7849f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:49:20.802 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:20.799+0000 7f3cdb800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55eddf78cd90 0x55eddf7849f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:59920/0 (socket says 172.21.15.32:59920) 2024-08-03T20:49:20.802 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:20.799+0000 7f3cdb800640 1 -- 172.21.15.32:0/3233140033 learned_addr learned my addr 172.21.15.32:0/3233140033 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:49:20.802 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:20.799+0000 7f3cdc200640 1 --2- 172.21.15.32:0/3233140033 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55eddf7713f0 0x55eddf822f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:49:20.802 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:20.799+0000 7f3cdcc00640 1 --2- 172.21.15.32:0/3233140033 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55eddf818610 0x55eddf784f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:49:20.802 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:20.799+0000 7f3cdb800640 1 -- 172.21.15.32:0/3233140033 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55eddf7713f0 msgr2=0x55eddf822f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:49:20.802 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:20.799+0000 7f3cdb800640 1 --2- 172.21.15.32:0/3233140033 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55eddf7713f0 0x55eddf822f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:49:20.802 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:20.799+0000 7f3cdb800640 1 -- 172.21.15.32:0/3233140033 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55eddf818610 msgr2=0x55eddf784f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:49:20.803 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:20.799+0000 7f3cdb800640 1 --2- 172.21.15.32:0/3233140033 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55eddf818610 0x55eddf784f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:49:20.803 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:20.799+0000 7f3cdb800640 1 -- 172.21.15.32:0/3233140033 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55eddf6b9c50 con 0x55eddf78cd90 2024-08-03T20:49:20.803 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:20.799+0000 7f3cdb800640 1 --2- 172.21.15.32:0/3233140033 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55eddf78cd90 0x55eddf7849f0 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f3cc8002eb0 tx=0x7f3cc800c7a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:49:20.803 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:20.799+0000 7f3cd9000640 1 -- 172.21.15.32:0/3233140033 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3cc800f810 con 0x55eddf78cd90 2024-08-03T20:49:20.803 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:20.800+0000 7f3cd9000640 1 -- 172.21.15.32:0/3233140033 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f3cc8013070 con 0x55eddf78cd90 2024-08-03T20:49:20.803 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:20.800+0000 7f3cd9000640 1 -- 172.21.15.32:0/3233140033 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3cc800fd60 con 0x55eddf78cd90 2024-08-03T20:49:20.804 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:20.800+0000 7f3ce3b21f40 1 -- 172.21.15.32:0/3233140033 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x55eddf785d00 con 0x55eddf78cd90 2024-08-03T20:49:20.804 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:20.800+0000 7f3ce3b21f40 1 -- 172.21.15.32:0/3233140033 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55eddf7693b0 con 0x55eddf78cd90 2024-08-03T20:49:20.806 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:20.802+0000 7f3cd9000640 1 -- 172.21.15.32:0/3233140033 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f3cc801f340 con 0x55eddf78cd90 2024-08-03T20:49:20.806 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:20.803+0000 7f3cd9000640 1 --2- 172.21.15.32:0/3233140033 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f3cbc0759f0 0x7f3cbc077f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:49:20.807 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:20.803+0000 7f3cd9000640 1 -- 172.21.15.32:0/3233140033 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(33..33 src has 1..33) ==== 5795+0+0 (secure 0 0 0) 0x7f3cc8097800 con 0x55eddf78cd90 2024-08-03T20:49:20.807 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:20.803+0000 7f3ce3b21f40 1 --2- 172.21.15.32:0/3233140033 >> [v2:172.21.15.32:6800/9088303,v1:172.21.15.32:6801/9088303] conn(0x55eddf814640 0x55eddf82f4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:49:20.807 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:20.803+0000 7f3ce3b21f40 1 -- 172.21.15.32:0/3233140033 --> [v2:172.21.15.32:6800/9088303,v1:172.21.15.32:6801/9088303] -- osd_op(unknown.0.0:1 4.10 4:0cf6ad0f:::10000000000.00000000:head [omap-set-vals in=115b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e33) -- 0x55eddf76e260 con 0x55eddf814640 2024-08-03T20:49:20.807 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:20.803+0000 7f3cdc200640 1 --2- 172.21.15.32:0/3233140033 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f3cbc0759f0 0x7f3cbc077f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:49:20.807 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:20.803+0000 7f3cdcc00640 1 --2- 172.21.15.32:0/3233140033 >> [v2:172.21.15.32:6800/9088303,v1:172.21.15.32:6801/9088303] conn(0x55eddf814640 0x55eddf82f4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:49:20.807 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:20.804+0000 7f3cdcc00640 1 --2- 172.21.15.32:0/3233140033 >> [v2:172.21.15.32:6800/9088303,v1:172.21.15.32:6801/9088303] conn(0x55eddf814640 0x55eddf82f4b0 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:49:20.807 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:20.804+0000 7f3cdc200640 1 --2- 172.21.15.32:0/3233140033 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f3cbc0759f0 0x7f3cbc077f00 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f3ccc063000 tx=0x7f3ccc004880 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:49:20.809 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:20.806+0000 7f3cdcc00640 1 -- 172.21.15.32:0/3233140033 <== osd.3 v2:172.21.15.32:6800/9088303 1 ==== osd_op_reply(1 10000000000.00000000 [omap-set-vals] v33'12 uv12 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f3cd400b040 con 0x55eddf814640 2024-08-03T20:49:20.809 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:20.806+0000 7f3ce3b21f40 1 -- 172.21.15.32:0/3233140033 >> [v2:172.21.15.32:6800/9088303,v1:172.21.15.32:6801/9088303] conn(0x55eddf814640 msgr2=0x55eddf82f4b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:49:20.809 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:20.806+0000 7f3ce3b21f40 1 --2- 172.21.15.32:0/3233140033 >> [v2:172.21.15.32:6800/9088303,v1:172.21.15.32:6801/9088303] conn(0x55eddf814640 0x55eddf82f4b0 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:49:20.809 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:20.806+0000 7f3ce3b21f40 1 -- 172.21.15.32:0/3233140033 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f3cbc0759f0 msgr2=0x7f3cbc077f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:49:20.809 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:20.806+0000 7f3ce3b21f40 1 --2- 172.21.15.32:0/3233140033 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f3cbc0759f0 0x7f3cbc077f00 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f3ccc063000 tx=0x7f3ccc004880 comp rx=0 tx=0).stop 2024-08-03T20:49:20.809 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:20.806+0000 7f3ce3b21f40 1 -- 172.21.15.32:0/3233140033 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55eddf78cd90 msgr2=0x55eddf7849f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:49:20.809 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:20.806+0000 7f3ce3b21f40 1 --2- 172.21.15.32:0/3233140033 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55eddf78cd90 0x55eddf7849f0 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f3cc8002eb0 tx=0x7f3cc800c7a0 comp rx=0 tx=0).stop 2024-08-03T20:49:20.809 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:20.806+0000 7f3cdcc00640 1 -- 172.21.15.32:0/3233140033 reap_dead start 2024-08-03T20:49:20.809 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:20.806+0000 7f3ce3b21f40 1 -- 172.21.15.32:0/3233140033 shutdown_connections 2024-08-03T20:49:20.810 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:20.806+0000 7f3ce3b21f40 1 -- 172.21.15.32:0/3233140033 >> 172.21.15.32:0/3233140033 conn(0x55eddf5d6ac0 msgr2=0x55eddf7833a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:49:20.810 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:20.806+0000 7f3ce3b21f40 1 -- 172.21.15.32:0/3233140033 shutdown_connections 2024-08-03T20:49:20.810 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:20.806+0000 7f3ce3b21f40 1 -- 172.21.15.32:0/3233140033 wait complete. 2024-08-03T20:49:20.821 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-08-03T20:49:21.051 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:21.049+0000 7fdee9400640 1 -- 172.21.15.32:0/3956371946 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fdee40b4de0 msgr2=0x7fdee41515c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:49:21.051 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:21.049+0000 7fdee9400640 1 --2- 172.21.15.32:0/3956371946 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fdee40b4de0 0x7fdee41515c0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fded400a970 tx=0x7fded402b7d0 comp rx=0 tx=0).stop 2024-08-03T20:49:21.051 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:21.049+0000 7fdee9400640 1 -- 172.21.15.32:0/3956371946 shutdown_connections 2024-08-03T20:49:21.051 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:21.049+0000 7fdee9400640 1 --2- 172.21.15.32:0/3956371946 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fdee40b4de0 0x7fdee41515c0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:49:21.051 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:21.049+0000 7fdee9400640 1 -- 172.21.15.32:0/3956371946 >> 172.21.15.32:0/3956371946 conn(0x7fdee40b1bd0 msgr2=0x7fdee40b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:49:21.051 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:21.049+0000 7fdee9400640 1 -- 172.21.15.32:0/3956371946 shutdown_connections 2024-08-03T20:49:21.051 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:21.049+0000 7fdee9400640 1 -- 172.21.15.32:0/3956371946 wait complete. 2024-08-03T20:49:21.052 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:21.051+0000 7fdee9400640 1 Processor -- start 2024-08-03T20:49:21.053 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:21.051+0000 7fdee9400640 1 -- start start 2024-08-03T20:49:21.054 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:21.053+0000 7fdee9400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdee40733f0 0x7fdee41e4020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:49:21.054 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:21.053+0000 7fdee9400640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fdee40b4de0 0x7fdee4154650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:49:21.054 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:21.053+0000 7fdee9400640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fdee41e6ae0 0x7fdee41e82e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:49:21.054 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:21.053+0000 7fdee9400640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fdee4073980 con 0x7fdee40733f0 2024-08-03T20:49:21.054 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:21.053+0000 7fdee9400640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7fdee4073800 con 0x7fdee41e6ae0 2024-08-03T20:49:21.055 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:21.053+0000 7fdee9400640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7fdee4073b00 con 0x7fdee40b4de0 2024-08-03T20:49:21.055 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:21.053+0000 7fdee2800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdee40733f0 0x7fdee41e4020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:49:21.055 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:21.053+0000 7fdee1e00640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fdee40b4de0 0x7fdee4154650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:49:21.055 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:21.053+0000 7fdee1e00640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fdee40b4de0 0x7fdee4154650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3301/0 says I am v2:172.21.15.32:56590/0 (socket says 172.21.15.32:56590) 2024-08-03T20:49:21.056 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:21.053+0000 7fdee3200640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fdee41e6ae0 0x7fdee41e82e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:49:21.056 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:21.053+0000 7fdee1e00640 1 -- 172.21.15.32:0/3584283483 learned_addr learned my addr 172.21.15.32:0/3584283483 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:49:21.056 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:21.054+0000 7fdee2800640 1 -- 172.21.15.32:0/3584283483 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fdee40b4de0 msgr2=0x7fdee4154650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:49:21.056 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:21.054+0000 7fdee2800640 1 --2- 172.21.15.32:0/3584283483 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fdee40b4de0 0x7fdee4154650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:49:21.056 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:21.054+0000 7fdee2800640 1 -- 172.21.15.32:0/3584283483 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fdee41e6ae0 msgr2=0x7fdee41e82e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:49:21.056 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:21.054+0000 7fdee2800640 1 --2- 172.21.15.32:0/3584283483 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fdee41e6ae0 0x7fdee41e82e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:49:21.056 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:21.054+0000 7fdee2800640 1 -- 172.21.15.32:0/3584283483 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdee41f5c80 con 0x7fdee40733f0 2024-08-03T20:49:21.056 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:21.054+0000 7fdee2800640 1 --2- 172.21.15.32:0/3584283483 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdee40733f0 0x7fdee41e4020 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7fded4032000 tx=0x7fded4030ce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:49:21.057 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:21.054+0000 7fdedaa00640 1 -- 172.21.15.32:0/3584283483 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fded4040070 con 0x7fdee40733f0 2024-08-03T20:49:21.057 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:21.055+0000 7fdedaa00640 1 -- 172.21.15.32:0/3584283483 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fded4005330 con 0x7fdee40733f0 2024-08-03T20:49:21.057 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:21.055+0000 7fdee9400640 1 -- 172.21.15.32:0/3584283483 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdee41f5f60 con 0x7fdee40733f0 2024-08-03T20:49:21.057 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:21.055+0000 7fdedaa00640 1 -- 172.21.15.32:0/3584283483 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fded4005c90 con 0x7fdee40733f0 2024-08-03T20:49:21.057 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:21.055+0000 7fdee9400640 1 -- 172.21.15.32:0/3584283483 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdee41f6520 con 0x7fdee40733f0 2024-08-03T20:49:21.057 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:21.056+0000 7fdee9400640 1 -- 172.21.15.32:0/3584283483 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdee41515c0 con 0x7fdee40733f0 2024-08-03T20:49:21.058 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:21.057+0000 7fdedaa00640 1 -- 172.21.15.32:0/3584283483 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7fded403a040 con 0x7fdee40733f0 2024-08-03T20:49:21.059 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:21.058+0000 7fdedaa00640 1 --2- 172.21.15.32:0/3584283483 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fdec4077040 0x7fdec4079550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:49:21.059 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:21.058+0000 7fdedaa00640 1 -- 172.21.15.32:0/3584283483 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(33..33 src has 1..33) ==== 5795+0+0 (secure 0 0 0) 0x7fded40b9690 con 0x7fdee40733f0 2024-08-03T20:49:21.060 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:21.058+0000 7fdee1e00640 1 --2- 172.21.15.32:0/3584283483 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fdec4077040 0x7fdec4079550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:49:21.060 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:21.059+0000 7fdee1e00640 1 --2- 172.21.15.32:0/3584283483 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fdec4077040 0x7fdec4079550 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fdee40d1060 tx=0x7fded0009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:49:21.064 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:21.063+0000 7fdedaa00640 1 -- 172.21.15.32:0/3584283483 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7fded4082140 con 0x7fdee40733f0 2024-08-03T20:49:21.307 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:21.306+0000 7fdee9400640 1 -- 172.21.15.32:0/3584283483 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fdee40b19b0 con 0x7fdee40733f0 2024-08-03T20:49:21.308 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:21.306+0000 7fdedaa00640 1 -- 172.21.15.32:0/3584283483 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+5227 (secure 0 0 0) 0x7fded4086690 con 0x7fdee40733f0 2024-08-03T20:49:21.308 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-03T20:49:21.308 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":25,"btime":"2024-08-03T20:48:57:346845+0000","default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4643,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6836/4002526832","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6834","nonce":4002526832},{"type":"v1","addr":"172.21.15.118:6836","nonce":4002526832}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":4646,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6837/3036256187","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6835","nonce":3036256187},{"type":"v1","addr":"172.21.15.118:6837","nonce":3036256187}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":4652,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6834/2104418334","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6833","nonce":2104418334},{"type":"v1","addr":"172.21.15.32:6834","nonce":2104418334}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-08-03T20:48:26.652521+0000","modified":"2024-08-03T20:48:57.101761+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5070},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5070":{"gid":5070,"name":"a","rank":0,"incarnation":20,"state":"up:active","state_seq":8,"addr":"172.21.15.32:6835/211300533","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":211300533},{"type":"v1","addr":"172.21.15.32:6835","nonce":211300533}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5070,"qdb_cluster":[5070]},"id":2}]} 2024-08-03T20:49:21.308 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 25 2024-08-03T20:49:21.310 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:21.309+0000 7fdee9400640 1 -- 172.21.15.32:0/3584283483 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fdec4077040 msgr2=0x7fdec4079550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:49:21.311 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:21.309+0000 7fdee9400640 1 --2- 172.21.15.32:0/3584283483 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fdec4077040 0x7fdec4079550 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fdee40d1060 tx=0x7fded0009040 comp rx=0 tx=0).stop 2024-08-03T20:49:21.311 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:21.309+0000 7fdee9400640 1 -- 172.21.15.32:0/3584283483 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdee40733f0 msgr2=0x7fdee41e4020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:49:21.311 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:21.309+0000 7fdee9400640 1 --2- 172.21.15.32:0/3584283483 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdee40733f0 0x7fdee41e4020 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7fded4032000 tx=0x7fded4030ce0 comp rx=0 tx=0).stop 2024-08-03T20:49:21.311 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:21.309+0000 7fdee9400640 1 -- 172.21.15.32:0/3584283483 shutdown_connections 2024-08-03T20:49:21.311 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:21.309+0000 7fdee9400640 1 --2- 172.21.15.32:0/3584283483 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fdec4077040 0x7fdec4079550 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:49:21.311 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:21.309+0000 7fdee9400640 1 --2- 172.21.15.32:0/3584283483 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fdee41e6ae0 0x7fdee41e82e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:49:21.311 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:21.309+0000 7fdee9400640 1 --2- 172.21.15.32:0/3584283483 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fdee40b4de0 0x7fdee4154650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:49:21.311 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:21.309+0000 7fdee9400640 1 --2- 172.21.15.32:0/3584283483 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdee40733f0 0x7fdee41e4020 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:49:21.311 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:21.309+0000 7fdee9400640 1 -- 172.21.15.32:0/3584283483 >> 172.21.15.32:0/3584283483 conn(0x7fdee40b1bd0 msgr2=0x7fdee415e100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:49:21.312 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:21.309+0000 7fdee9400640 1 -- 172.21.15.32:0/3584283483 shutdown_connections 2024-08-03T20:49:21.312 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:21.309+0000 7fdee9400640 1 -- 172.21.15.32:0/3584283483 wait complete. 2024-08-03T20:49:21.334 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a damage rm 353632146 2024-08-03T20:49:21.567 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:21.565+0000 7fa0fb800640 1 -- 172.21.15.32:0/1255288328 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa0f40cecc0 msgr2=0x7fa0f40b4f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:49:21.568 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:21.565+0000 7fa0fb800640 1 --2- 172.21.15.32:0/1255288328 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa0f40cecc0 0x7fa0f40b4f10 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7fa0e8005fd0 tx=0x7fa0e802a5d0 comp rx=0 tx=0).stop 2024-08-03T20:49:21.568 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:21.566+0000 7fa0fb800640 1 -- 172.21.15.32:0/1255288328 shutdown_connections 2024-08-03T20:49:21.568 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:21.566+0000 7fa0fb800640 1 --2- 172.21.15.32:0/1255288328 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fa0f41606f0 0x7fa0f41450f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:49:21.568 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:21.566+0000 7fa0fb800640 1 --2- 172.21.15.32:0/1255288328 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa0f40cecc0 0x7fa0f40b4f10 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:49:21.568 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:21.566+0000 7fa0fb800640 1 -- 172.21.15.32:0/1255288328 >> 172.21.15.32:0/1255288328 conn(0x7fa0f40b1bd0 msgr2=0x7fa0f40b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:49:21.568 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:21.566+0000 7fa0fb800640 1 -- 172.21.15.32:0/1255288328 shutdown_connections 2024-08-03T20:49:21.568 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:21.566+0000 7fa0fb800640 1 -- 172.21.15.32:0/1255288328 wait complete. 2024-08-03T20:49:21.569 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:21.568+0000 7fa0fb800640 1 Processor -- start 2024-08-03T20:49:21.569 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:21.568+0000 7fa0fb800640 1 -- start start 2024-08-03T20:49:21.571 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:21.570+0000 7fa0fb800640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fa0f40cd210 0x7fa0f41564c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:49:21.571 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:21.570+0000 7fa0fb800640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa0f40cecc0 0x7fa0f40733a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:49:21.572 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:21.570+0000 7fa0fb800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa0f40738e0 0x7fa0f40ca1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:49:21.572 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:21.570+0000 7fa0fb800640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fa0f4145ad0 con 0x7fa0f40738e0 2024-08-03T20:49:21.572 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:21.570+0000 7fa0fb800640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7fa0f4145950 con 0x7fa0f40cecc0 2024-08-03T20:49:21.572 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:21.570+0000 7fa0fb800640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7fa0f4145c50 con 0x7fa0f40cd210 2024-08-03T20:49:21.572 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:21.570+0000 7fa0f9800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa0f40738e0 0x7fa0f40ca1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:49:21.572 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:21.570+0000 7fa0f8e00640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fa0f40cd210 0x7fa0f41564c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:49:21.572 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:21.570+0000 7fa0f9800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa0f40738e0 0x7fa0f40ca1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:59950/0 (socket says 172.21.15.32:59950) 2024-08-03T20:49:21.572 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:21.570+0000 7fa0f9800640 1 -- 172.21.15.32:0/2035020027 learned_addr learned my addr 172.21.15.32:0/2035020027 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:49:21.572 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:21.570+0000 7fa0f3e00640 1 --2- 172.21.15.32:0/2035020027 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa0f40cecc0 0x7fa0f40733a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:49:21.573 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:21.570+0000 7fa0f9800640 1 -- 172.21.15.32:0/2035020027 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fa0f40cd210 msgr2=0x7fa0f41564c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:49:21.573 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:21.570+0000 7fa0f9800640 1 --2- 172.21.15.32:0/2035020027 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fa0f40cd210 0x7fa0f41564c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:49:21.573 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:21.570+0000 7fa0f9800640 1 -- 172.21.15.32:0/2035020027 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa0f40cecc0 msgr2=0x7fa0f40733a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:49:21.573 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:21.570+0000 7fa0f9800640 1 --2- 172.21.15.32:0/2035020027 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa0f40cecc0 0x7fa0f40733a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:49:21.574 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:21.570+0000 7fa0f9800640 1 -- 172.21.15.32:0/2035020027 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa0f4073cf0 con 0x7fa0f40738e0 2024-08-03T20:49:21.574 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:21.570+0000 7fa0f8e00640 1 --2- 172.21.15.32:0/2035020027 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fa0f40cd210 0x7fa0f41564c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-03T20:49:21.574 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:21.571+0000 7fa0f9800640 1 --2- 172.21.15.32:0/2035020027 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa0f40738e0 0x7fa0f40ca1b0 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7fa0ec00e990 tx=0x7fa0ec00ee50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:49:21.574 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:21.571+0000 7fa0f0c00640 1 -- 172.21.15.32:0/2035020027 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0ec004d20 con 0x7fa0f40738e0 2024-08-03T20:49:21.574 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:21.571+0000 7fa0f0c00640 1 -- 172.21.15.32:0/2035020027 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fa0ec00caa0 con 0x7fa0f40738e0 2024-08-03T20:49:21.574 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:21.571+0000 7fa0fb800640 1 -- 172.21.15.32:0/2035020027 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa0f4156cd0 con 0x7fa0f40738e0 2024-08-03T20:49:21.574 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:21.571+0000 7fa0f0c00640 1 -- 172.21.15.32:0/2035020027 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0ec005020 con 0x7fa0f40738e0 2024-08-03T20:49:21.574 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:21.571+0000 7fa0fb800640 1 -- 172.21.15.32:0/2035020027 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa0f40c96d0 con 0x7fa0f40738e0 2024-08-03T20:49:21.576 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:21.575+0000 7fa0f0c00640 1 -- 172.21.15.32:0/2035020027 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7fa0ec0051c0 con 0x7fa0f40738e0 2024-08-03T20:49:21.577 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:21.576+0000 7fa0f0c00640 1 --2- 172.21.15.32:0/2035020027 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fa0d4077010 0x7fa0d4079520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:49:21.577 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:21.576+0000 7fa0f0c00640 1 -- 172.21.15.32:0/2035020027 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(33..33 src has 1..33) ==== 5795+0+0 (secure 0 0 0) 0x7fa0ec0990b0 con 0x7fa0f40738e0 2024-08-03T20:49:21.577 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:21.576+0000 7fa10276d740 1 Processor -- start 2024-08-03T20:49:21.577 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:21.576+0000 7fa10276d740 1 -- start start 2024-08-03T20:49:21.578 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:21.576+0000 7fa0f8e00640 1 --2- 172.21.15.32:0/2035020027 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fa0d4077010 0x7fa0d4079520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:49:21.580 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:21.577+0000 7fa0f8e00640 1 --2- 172.21.15.32:0/2035020027 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fa0d4077010 0x7fa0d4079520 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fa0e8033000 tx=0x7fa0e8004730 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:49:21.580 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:21.577+0000 7fa10276d740 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55b19ccbb490 0x55b19d08b1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:49:21.580 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:21.577+0000 7fa10276d740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55b19cc925d0 0x55b19d07a960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:49:21.580 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:21.577+0000 7fa10276d740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b19cc9a7b0 0x55b19d07aea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:49:21.580 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:21.577+0000 7fa10276d740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55b19cfd4180 con 0x55b19cc9a7b0 2024-08-03T20:49:21.581 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:21.577+0000 7fa10276d740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x55b19cfc3680 con 0x55b19cc925d0 2024-08-03T20:49:21.581 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:21.577+0000 7fa10276d740 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x55b19cfc76c0 con 0x55b19ccbb490 2024-08-03T20:49:21.581 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:21.578+0000 7fa0f9800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b19cc9a7b0 0x55b19d07aea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:49:21.581 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:21.578+0000 7fa0f9800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b19cc9a7b0 0x55b19d07aea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:59960/0 (socket says 172.21.15.32:59960) 2024-08-03T20:49:21.581 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:21.578+0000 7fa0f9800640 1 -- 172.21.15.32:0/1639735698 learned_addr learned my addr 172.21.15.32:0/1639735698 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:49:21.581 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:21.578+0000 7fa0f8e00640 1 --2- 172.21.15.32:0/1639735698 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55b19ccbb490 0x55b19d08b1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:49:21.581 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:21.578+0000 7fa0f3e00640 1 --2- 172.21.15.32:0/1639735698 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55b19cc925d0 0x55b19d07a960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:49:21.581 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:21.578+0000 7fa0f9800640 1 -- 172.21.15.32:0/1639735698 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55b19ccbb490 msgr2=0x55b19d08b1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:49:21.581 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:21.578+0000 7fa0f9800640 1 --2- 172.21.15.32:0/1639735698 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55b19ccbb490 0x55b19d08b1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:49:21.581 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:21.578+0000 7fa0f9800640 1 -- 172.21.15.32:0/1639735698 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55b19cc925d0 msgr2=0x55b19d07a960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:49:21.581 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:21.578+0000 7fa0f9800640 1 --2- 172.21.15.32:0/1639735698 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55b19cc925d0 0x55b19d07a960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:49:21.581 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:21.578+0000 7fa0f9800640 1 -- 172.21.15.32:0/1639735698 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b19cd9f900 con 0x55b19cc9a7b0 2024-08-03T20:49:21.582 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:21.578+0000 7fa0f8e00640 1 --2- 172.21.15.32:0/1639735698 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55b19ccbb490 0x55b19d08b1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-03T20:49:21.582 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:21.578+0000 7fa0f9800640 1 --2- 172.21.15.32:0/1639735698 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b19cc9a7b0 0x55b19d07aea0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7fa0ec061980 tx=0x7fa0ec061790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9ece747a8d4b0aff server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:49:21.582 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:21.579+0000 7fa0e5600640 1 -- 172.21.15.32:0/1639735698 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0ec01f040 con 0x55b19cc9a7b0 2024-08-03T20:49:21.582 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:21.579+0000 7fa0e5600640 1 -- 172.21.15.32:0/1639735698 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fa0ec0661d0 con 0x55b19cc9a7b0 2024-08-03T20:49:21.582 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:21.579+0000 7fa0e5600640 1 -- 172.21.15.32:0/1639735698 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0ec06abf0 con 0x55b19cc9a7b0 2024-08-03T20:49:21.583 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:21.579+0000 7fa10276d740 1 -- 172.21.15.32:0/1639735698 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b19cc9a7b0 msgr2=0x55b19d07aea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:49:21.583 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:21.579+0000 7fa10276d740 1 --2- 172.21.15.32:0/1639735698 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b19cc9a7b0 0x55b19d07aea0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7fa0ec061980 tx=0x7fa0ec061790 comp rx=0 tx=0).stop 2024-08-03T20:49:21.583 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:21.580+0000 7fa10276d740 1 -- 172.21.15.32:0/1639735698 shutdown_connections 2024-08-03T20:49:21.583 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:21.580+0000 7fa10276d740 1 --2- 172.21.15.32:0/1639735698 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55b19cc925d0 0x55b19d07a960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:49:21.583 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:21.580+0000 7fa10276d740 1 --2- 172.21.15.32:0/1639735698 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b19cc9a7b0 0x55b19d07aea0 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:49:21.583 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:21.580+0000 7fa10276d740 1 --2- 172.21.15.32:0/1639735698 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55b19ccbb490 0x55b19d08b1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:49:21.583 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:21.580+0000 7fa10276d740 1 -- 172.21.15.32:0/1639735698 >> 172.21.15.32:0/1639735698 conn(0x55b19ccaf5e0 msgr2=0x55b19d061280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:49:21.584 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:21.580+0000 7fa10276d740 1 -- 172.21.15.32:0/1639735698 shutdown_connections 2024-08-03T20:49:21.584 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:21.580+0000 7fa10276d740 1 -- 172.21.15.32:0/1639735698 wait complete. 2024-08-03T20:49:21.584 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:21.581+0000 7fa10276d740 1 -- start start 2024-08-03T20:49:21.584 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:21.581+0000 7fa10276d740 1 Processor -- start 2024-08-03T20:49:21.588 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:21.583+0000 7fa10276d740 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55b19cc925d0 0x55b19d0b9200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:49:21.589 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:21.583+0000 7fa10276d740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55b19cc9a7b0 0x55b19d0793d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:49:21.589 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:21.583+0000 7fa10276d740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b19ccbb490 0x55b19d060330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:49:21.589 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:21.583+0000 7fa10276d740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55b19cefbee0 con 0x55b19ccbb490 2024-08-03T20:49:21.589 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:21.583+0000 7fa10276d740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x55b19cf5df00 con 0x55b19cc9a7b0 2024-08-03T20:49:21.589 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:21.583+0000 7fa10276d740 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x55b19d0a2e30 con 0x55b19cc925d0 2024-08-03T20:49:21.589 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:21.583+0000 7fa10276d740 10 client.0 _ll_register_callbacks cb 0x55b19cf6f8a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-03T20:49:21.589 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:21.583+0000 7fa0f9800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b19ccbb490 0x55b19d060330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:49:21.589 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:21.583+0000 7fa0f9800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b19ccbb490 0x55b19d060330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:59972/0 (socket says 172.21.15.32:59972) 2024-08-03T20:49:21.589 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:21.583+0000 7fa0f9800640 1 -- 172.21.15.32:0/2341464045 learned_addr learned my addr 172.21.15.32:0/2341464045 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:49:21.589 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:21.583+0000 7fa0f3e00640 1 --2- 172.21.15.32:0/2341464045 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55b19cc9a7b0 0x55b19d0793d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:49:21.589 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:21.583+0000 7fa0f8e00640 1 --2- 172.21.15.32:0/2341464045 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55b19cc925d0 0x55b19d0b9200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:49:21.590 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:21.583+0000 7fa0f9800640 1 -- 172.21.15.32:0/2341464045 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55b19cc925d0 msgr2=0x55b19d0b9200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:49:21.590 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:21.583+0000 7fa0f9800640 1 --2- 172.21.15.32:0/2341464045 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55b19cc925d0 0x55b19d0b9200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:49:21.590 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:21.583+0000 7fa0f9800640 1 -- 172.21.15.32:0/2341464045 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55b19cc9a7b0 msgr2=0x55b19d0793d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:49:21.590 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:21.583+0000 7fa0f9800640 1 --2- 172.21.15.32:0/2341464045 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55b19cc9a7b0 0x55b19d0793d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:49:21.590 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:21.583+0000 7fa0f9800640 1 -- 172.21.15.32:0/2341464045 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b19cd26a70 con 0x55b19ccbb490 2024-08-03T20:49:21.590 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:21.583+0000 7fa0f8e00640 1 --2- 172.21.15.32:0/2341464045 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55b19cc925d0 0x55b19d0b9200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-03T20:49:21.590 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:21.584+0000 7fa0f9800640 1 --2- 172.21.15.32:0/2341464045 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b19ccbb490 0x55b19d060330 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7fa0ec061980 tx=0x7fa0ec0661d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=32e76809a121a6c2 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:49:21.590 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:21.584+0000 7fa0da000640 10 client.0 ms_handle_connect on v2:172.21.15.32:3300/0 2024-08-03T20:49:21.590 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:21.584+0000 7fa0da000640 1 -- 172.21.15.32:0/2341464045 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0ec094070 con 0x55b19ccbb490 2024-08-03T20:49:21.590 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:21.584+0000 7fa0da000640 1 -- 172.21.15.32:0/2341464045 <== mon.0 v2:172.21.15.32:3300/0 2 ==== osd_map(33..33 src has 1..33) ==== 5795+0+0 (secure 0 0 0) 0x7fa0ec0683e0 con 0x55b19ccbb490 2024-08-03T20:49:21.590 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:21.584+0000 7fa10276d740 1 -- 172.21.15.32:0/2341464045 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55b19cf71ea0 con 0x55b19ccbb490 2024-08-03T20:49:21.590 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:21.584+0000 7fa0da000640 1 -- 172.21.15.32:0/2341464045 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7fa0ec083050 con 0x55b19ccbb490 2024-08-03T20:49:21.591 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:21.584+0000 7fa10276d740 10 client.5253 fetch_fsmap learned FSMap version 25 2024-08-03T20:49:21.591 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:21.584+0000 7fa10276d740 1 -- 172.21.15.32:0/2341464045 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=2+}) -- 0x55b19cc6c370 con 0x55b19ccbb490 2024-08-03T20:49:21.591 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:21.584+0000 7fa0da000640 1 -- 172.21.15.32:0/2341464045 <== mon.0 v2:172.21.15.32:3300/0 4 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x55b19cf71ea0 con 0x55b19ccbb490 2024-08-03T20:49:21.591 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:21.584+0000 7fa0da000640 1 -- 172.21.15.32:0/2341464045 <== mon.0 v2:172.21.15.32:3300/0 5 ==== fsmap(e 25) ==== 3502+0+0 (secure 0 0 0) 0x7fa0ec00f040 con 0x55b19ccbb490 2024-08-03T20:49:21.591 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:21.584+0000 7fa10276d740 10 client.5253 fetch_fsmap finished waiting for FSMap version 25 2024-08-03T20:49:21.591 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:21.584+0000 7fa10276d740 10 client.5253 resolve_mds: resolved name 'a' to daemon mds.a 2024-08-03T20:49:21.591 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:21.584+0000 7fa10276d740 20 client.5253 populate_metadata read hostname 'smithi032' 2024-08-03T20:49:21.591 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:21.584+0000 7fa10276d740 1 --2- 172.21.15.32:0/2341464045 >> [v2:172.21.15.32:6832/211300533,v1:172.21.15.32:6835/211300533] conn(0x55b19cc7a060 0x55b19d0a1bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:49:21.591 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:21.584+0000 7fa10276d740 4 client.5253 mds_command: new command op to 5070 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-08-03T20:49:21.591 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:21.584+0000 7fa10276d740 1 -- 172.21.15.32:0/2341464045 --> [v2:172.21.15.32:6832/211300533,v1:172.21.15.32:6835/211300533] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b19cfd60d0 con 0x55b19cc7a060 2024-08-03T20:49:21.591 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:21.585+0000 7fa0f8e00640 1 --2- 172.21.15.32:0/2341464045 >> [v2:172.21.15.32:6832/211300533,v1:172.21.15.32:6835/211300533] conn(0x55b19cc7a060 0x55b19d0a1bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:49:21.591 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:21.585+0000 7fa0f8e00640 1 --2- 172.21.15.32:0/2341464045 >> [v2:172.21.15.32:6832/211300533,v1:172.21.15.32:6835/211300533] conn(0x55b19cc7a060 0x55b19d0a1bc0 crc :-1 s=READY pgs=24 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=381b30b1a1c7e64e server_cookie=458edf4b37913aa0 in_seq=0 out_seq=0 2024-08-03T20:49:21.591 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:21.585+0000 7fa0da000640 10 client.5253 ms_handle_connect on v2:172.21.15.32:6832/211300533 2024-08-03T20:49:21.591 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:21.586+0000 7fa0da000640 1 -- 172.21.15.32:0/2341464045 <== mds.0 v2:172.21.15.32:6832/211300533 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55b19cfd60d0 con 0x55b19cc7a060 2024-08-03T20:49:21.592 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:21.586+0000 7fa0da000640 10 client.5253 handle_command_reply: tid=0 2024-08-03T20:49:21.592 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:21.586+0000 7fa10276d740 1 client.5253 shutdown 2024-08-03T20:49:21.592 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:21.586+0000 7fa10276d740 1 -- 172.21.15.32:0/2341464045 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b19ccbb490 msgr2=0x55b19d060330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:49:21.592 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:21.586+0000 7fa10276d740 1 --2- 172.21.15.32:0/2341464045 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b19ccbb490 0x55b19d060330 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7fa0ec061980 tx=0x7fa0ec0661d0 comp rx=0 tx=0).stop 2024-08-03T20:49:21.592 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:21.586+0000 7fa10276d740 1 -- 172.21.15.32:0/2341464045 shutdown_connections 2024-08-03T20:49:21.592 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:21.586+0000 7fa10276d740 1 --2- 172.21.15.32:0/2341464045 >> [v2:172.21.15.32:6832/211300533,v1:172.21.15.32:6835/211300533] conn(0x55b19cc7a060 0x55b19d0a1bc0 crc :-1 s=READY pgs=24 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:49:21.592 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:21.586+0000 7fa10276d740 1 --2- 172.21.15.32:0/2341464045 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55b19cc9a7b0 0x55b19d0793d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:49:21.592 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:21.586+0000 7fa10276d740 1 --2- 172.21.15.32:0/2341464045 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b19ccbb490 0x55b19d060330 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:49:21.592 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:21.586+0000 7fa10276d740 1 --2- 172.21.15.32:0/2341464045 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55b19cc925d0 0x55b19d0b9200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:49:21.592 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:21.586+0000 7fa0da000640 0 client.5253 ms_handle_reset on v2:172.21.15.32:6832/211300533 2024-08-03T20:49:21.592 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:21.586+0000 7fa10276d740 1 -- 172.21.15.32:0/2341464045 >> 172.21.15.32:0/2341464045 conn(0x55b19ccaf5e0 msgr2=0x55b19d0a6b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:49:21.592 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:21.587+0000 7fa10276d740 1 -- 172.21.15.32:0/2341464045 shutdown_connections 2024-08-03T20:49:21.592 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:21.587+0000 7fa10276d740 1 -- 172.21.15.32:0/2341464045 wait complete. 2024-08-03T20:49:21.593 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:21.587+0000 7fa10276d740 20 client.5253 trim_cache size 0 max 16384 2024-08-03T20:49:21.603 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:21.602+0000 7fa10276d740 1 Processor -- start 2024-08-03T20:49:21.603 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:21.602+0000 7fa10276d740 1 -- start start 2024-08-03T20:49:21.606 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:21.603+0000 7fa10276d740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55b19cc925d0 0x55b19d05a980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:49:21.606 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:21.603+0000 7fa10276d740 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55b19cc9a7b0 0x55b19d05aec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:49:21.606 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:21.603+0000 7fa10276d740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b19ccbb490 0x55b19d0c8ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:49:21.606 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:21.603+0000 7fa10276d740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fa0f4201030 con 0x55b19ccbb490 2024-08-03T20:49:21.606 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:21.603+0000 7fa10276d740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7fa0f415ed10 con 0x55b19cc925d0 2024-08-03T20:49:21.607 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:21.603+0000 7fa10276d740 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7fa0f415f360 con 0x55b19cc9a7b0 2024-08-03T20:49:21.607 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:21.603+0000 7fa0f9800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b19ccbb490 0x55b19d0c8ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:49:21.607 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:21.603+0000 7fa0f9800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b19ccbb490 0x55b19d0c8ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:59974/0 (socket says 172.21.15.32:59974) 2024-08-03T20:49:21.607 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:21.603+0000 7fa0f9800640 1 -- 172.21.15.32:0/3905525949 learned_addr learned my addr 172.21.15.32:0/3905525949 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:49:21.607 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:21.604+0000 7fa0f8e00640 1 --2- 172.21.15.32:0/3905525949 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55b19cc925d0 0x55b19d05a980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:49:21.607 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:21.604+0000 7fa0f3e00640 1 --2- 172.21.15.32:0/3905525949 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55b19cc9a7b0 0x55b19d05aec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:49:21.607 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:21.604+0000 7fa0f9800640 1 -- 172.21.15.32:0/3905525949 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55b19cc9a7b0 msgr2=0x55b19d05aec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:49:21.607 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:21.604+0000 7fa0f9800640 1 --2- 172.21.15.32:0/3905525949 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55b19cc9a7b0 0x55b19d05aec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:49:21.607 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:21.604+0000 7fa0f9800640 1 -- 172.21.15.32:0/3905525949 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55b19cc925d0 msgr2=0x55b19d05a980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:49:21.607 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:21.604+0000 7fa0f9800640 1 --2- 172.21.15.32:0/3905525949 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55b19cc925d0 0x55b19d05a980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:49:21.607 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:21.604+0000 7fa0f9800640 1 -- 172.21.15.32:0/3905525949 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b19d09f530 con 0x55b19ccbb490 2024-08-03T20:49:21.607 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:21.604+0000 7fa0f3e00640 1 --2- 172.21.15.32:0/3905525949 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55b19cc9a7b0 0x55b19d05aec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-03T20:49:21.607 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:21.604+0000 7fa0f9800640 1 --2- 172.21.15.32:0/3905525949 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b19ccbb490 0x55b19d0c8ce0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7fa0ec05e220 tx=0x7fa0ec004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=74c691cd5e040dee server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:49:21.607 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:21.604+0000 7fa0e4c00640 1 -- 172.21.15.32:0/3905525949 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0ec094070 con 0x55b19ccbb490 2024-08-03T20:49:21.608 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:21.604+0000 7fa0e4c00640 1 -- 172.21.15.32:0/3905525949 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fa0ec0610f0 con 0x55b19ccbb490 2024-08-03T20:49:21.608 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:21.604+0000 7fa0e4c00640 1 -- 172.21.15.32:0/3905525949 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0ec01f040 con 0x55b19ccbb490 2024-08-03T20:49:21.609 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:21.605+0000 7fa10276d740 1 -- 172.21.15.32:0/3905525949 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b19ccbb490 msgr2=0x55b19d0c8ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:49:21.609 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:21.605+0000 7fa10276d740 1 --2- 172.21.15.32:0/3905525949 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b19ccbb490 0x55b19d0c8ce0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7fa0ec05e220 tx=0x7fa0ec004a40 comp rx=0 tx=0).stop 2024-08-03T20:49:21.609 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:21.605+0000 7fa10276d740 1 -- 172.21.15.32:0/3905525949 shutdown_connections 2024-08-03T20:49:21.609 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:21.605+0000 7fa10276d740 1 --2- 172.21.15.32:0/3905525949 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b19ccbb490 0x55b19d0c8ce0 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:49:21.609 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:21.605+0000 7fa10276d740 1 --2- 172.21.15.32:0/3905525949 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55b19cc9a7b0 0x55b19d05aec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:49:21.609 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:21.605+0000 7fa10276d740 1 --2- 172.21.15.32:0/3905525949 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55b19cc925d0 0x55b19d05a980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:49:21.609 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:21.605+0000 7fa10276d740 1 -- 172.21.15.32:0/3905525949 >> 172.21.15.32:0/3905525949 conn(0x55b19ccaf5e0 msgr2=0x55b19d0b7c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:49:21.609 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:21.605+0000 7fa10276d740 1 -- 172.21.15.32:0/3905525949 shutdown_connections 2024-08-03T20:49:21.610 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:21.605+0000 7fa10276d740 1 -- 172.21.15.32:0/3905525949 wait complete. 2024-08-03T20:49:21.610 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:21.606+0000 7fa10276d740 1 -- start start 2024-08-03T20:49:21.610 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:21.606+0000 7fa10276d740 1 Processor -- start 2024-08-03T20:49:21.614 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:21.608+0000 7fa10276d740 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55b19cc925d0 0x55b19d1188b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:49:21.615 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:21.608+0000 7fa10276d740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b19cc9a7b0 0x55b19d0a9d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:49:21.615 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:21.608+0000 7fa10276d740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55b19ccbb490 0x55b19d0c7cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:49:21.615 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:21.608+0000 7fa10276d740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55b19d0a2b70 con 0x55b19cc9a7b0 2024-08-03T20:49:21.615 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:21.608+0000 7fa10276d740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x55b19d09de60 con 0x55b19ccbb490 2024-08-03T20:49:21.615 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:21.608+0000 7fa10276d740 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x55b19d0a0a40 con 0x55b19cc925d0 2024-08-03T20:49:21.615 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:21.608+0000 7fa10276d740 10 client.0 _ll_register_callbacks cb 0x7fa0ec07b420 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-03T20:49:21.615 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:21.608+0000 7fa0f3e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b19cc9a7b0 0x55b19d0a9d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:49:21.615 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:21.608+0000 7fa0f3e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b19cc9a7b0 0x55b19d0a9d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:59986/0 (socket says 172.21.15.32:59986) 2024-08-03T20:49:21.615 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:21.608+0000 7fa0f3e00640 1 -- 172.21.15.32:0/3653144729 learned_addr learned my addr 172.21.15.32:0/3653144729 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:49:21.616 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:21.608+0000 7fa0f8e00640 1 --2- 172.21.15.32:0/3653144729 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55b19cc925d0 0x55b19d1188b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:49:21.616 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:21.609+0000 7fa0f9800640 1 --2- 172.21.15.32:0/3653144729 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55b19ccbb490 0x55b19d0c7cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:49:21.616 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:21.609+0000 7fa0f3e00640 1 -- 172.21.15.32:0/3653144729 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55b19cc925d0 msgr2=0x55b19d1188b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:49:21.616 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:21.609+0000 7fa0f3e00640 1 --2- 172.21.15.32:0/3653144729 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55b19cc925d0 0x55b19d1188b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:49:21.616 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:21.609+0000 7fa0f3e00640 1 -- 172.21.15.32:0/3653144729 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55b19ccbb490 msgr2=0x55b19d0c7cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:49:21.616 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:21.609+0000 7fa0f3e00640 1 --2- 172.21.15.32:0/3653144729 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55b19ccbb490 0x55b19d0c7cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:49:21.616 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:21.609+0000 7fa0f3e00640 1 -- 172.21.15.32:0/3653144729 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b19d05c1d0 con 0x55b19cc9a7b0 2024-08-03T20:49:21.616 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:21.609+0000 7fa0f9800640 1 --2- 172.21.15.32:0/3653144729 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55b19ccbb490 0x55b19d0c7cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-03T20:49:21.617 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:21.609+0000 7fa0f3e00640 1 --2- 172.21.15.32:0/3653144729 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b19cc9a7b0 0x55b19d0a9d60 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7fa0e0004dc0 tx=0x7fa0e0016790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=646b58a7f56747d0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:49:21.617 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:21.609+0000 7fa0da000640 10 client.0 ms_handle_connect on v2:172.21.15.32:3300/0 2024-08-03T20:49:21.617 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:21.609+0000 7fa0da000640 1 -- 172.21.15.32:0/3653144729 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0e0007050 con 0x55b19cc9a7b0 2024-08-03T20:49:21.617 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:21.609+0000 7fa0da000640 1 -- 172.21.15.32:0/3653144729 <== mon.0 v2:172.21.15.32:3300/0 2 ==== osd_map(33..33 src has 1..33) ==== 5795+0+0 (secure 0 0 0) 0x7fa0e00071f0 con 0x55b19cc9a7b0 2024-08-03T20:49:21.617 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:21.609+0000 7fa10276d740 1 -- 172.21.15.32:0/3653144729 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa0ec0610f0 con 0x55b19cc9a7b0 2024-08-03T20:49:21.617 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:21.610+0000 7fa0da000640 1 -- 172.21.15.32:0/3653144729 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7fa0e0026050 con 0x55b19cc9a7b0 2024-08-03T20:49:21.617 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:21.610+0000 7fa10276d740 10 client.5259 fetch_fsmap learned FSMap version 25 2024-08-03T20:49:21.617 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:21.610+0000 7fa10276d740 1 -- 172.21.15.32:0/3653144729 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=2+}) -- 0x55b19d0c9540 con 0x55b19cc9a7b0 2024-08-03T20:49:21.617 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:21.610+0000 7fa0da000640 1 -- 172.21.15.32:0/3653144729 <== mon.0 v2:172.21.15.32:3300/0 4 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fa0ec0610f0 con 0x55b19cc9a7b0 2024-08-03T20:49:21.617 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:21.610+0000 7fa0da000640 1 -- 172.21.15.32:0/3653144729 <== mon.0 v2:172.21.15.32:3300/0 5 ==== fsmap(e 25) ==== 3502+0+0 (secure 0 0 0) 0x7fa0e0029020 con 0x55b19cc9a7b0 2024-08-03T20:49:21.617 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:21.610+0000 7fa10276d740 10 client.5259 fetch_fsmap finished waiting for FSMap version 25 2024-08-03T20:49:21.617 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:21.610+0000 7fa10276d740 10 client.5259 resolve_mds: resolved name 'a' to daemon mds.a 2024-08-03T20:49:21.617 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:21.610+0000 7fa10276d740 20 client.5259 populate_metadata read hostname 'smithi032' 2024-08-03T20:49:21.617 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:21.610+0000 7fa10276d740 1 --2- 172.21.15.32:0/3653144729 >> [v2:172.21.15.32:6832/211300533,v1:172.21.15.32:6835/211300533] conn(0x55b19d05ed50 0x55b19d05f160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:49:21.618 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:21.610+0000 7fa10276d740 4 client.5259 mds_command: new command op to 5070 tid=0 multi_id=0 [{"prefix": "damage rm", "damage_id": 353632146}] 2024-08-03T20:49:21.618 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:21.610+0000 7fa10276d740 1 -- 172.21.15.32:0/3653144729 --> [v2:172.21.15.32:6832/211300533,v1:172.21.15.32:6835/211300533] -- command(tid 0: {"prefix": "damage rm", "damage_id": 353632146}) -- 0x55b19cfd1b70 con 0x55b19d05ed50 2024-08-03T20:49:21.618 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:21.610+0000 7fa0f8e00640 1 --2- 172.21.15.32:0/3653144729 >> [v2:172.21.15.32:6832/211300533,v1:172.21.15.32:6835/211300533] conn(0x55b19d05ed50 0x55b19d05f160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:49:21.618 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:21.611+0000 7fa0f8e00640 1 --2- 172.21.15.32:0/3653144729 >> [v2:172.21.15.32:6832/211300533,v1:172.21.15.32:6835/211300533] conn(0x55b19d05ed50 0x55b19d05f160 crc :-1 s=READY pgs=25 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fcf5827440bd28c9 server_cookie=a8bcfbeafb8ccd34 in_seq=0 out_seq=0 2024-08-03T20:49:21.618 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:21.611+0000 7fa0da000640 10 client.5259 ms_handle_connect on v2:172.21.15.32:6832/211300533 2024-08-03T20:49:21.618 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:21.611+0000 7fa0da000640 1 -- 172.21.15.32:0/3653144729 <== mds.0 v2:172.21.15.32:6832/211300533 1 ==== command_reply(tid 0: 0 ) ==== 8+0+0 (crc 0 0 0) 0x55b19cfd1b70 con 0x55b19d05ed50 2024-08-03T20:49:21.618 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:21.611+0000 7fa0da000640 10 client.5259 handle_command_reply: tid=0 2024-08-03T20:49:21.618 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:21.611+0000 7fa10276d740 1 client.5259 shutdown 2024-08-03T20:49:21.618 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:21.611+0000 7fa10276d740 1 -- 172.21.15.32:0/3653144729 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b19cc9a7b0 msgr2=0x55b19d0a9d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:49:21.618 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:21.611+0000 7fa10276d740 1 --2- 172.21.15.32:0/3653144729 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b19cc9a7b0 0x55b19d0a9d60 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7fa0e0004dc0 tx=0x7fa0e0016790 comp rx=0 tx=0).stop 2024-08-03T20:49:21.618 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:21.611+0000 7fa10276d740 1 -- 172.21.15.32:0/3653144729 shutdown_connections 2024-08-03T20:49:21.618 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:21.611+0000 7fa10276d740 1 --2- 172.21.15.32:0/3653144729 >> [v2:172.21.15.32:6832/211300533,v1:172.21.15.32:6835/211300533] conn(0x55b19d05ed50 0x55b19d05f160 crc :-1 s=READY pgs=25 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:49:21.619 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:21.611+0000 7fa10276d740 1 --2- 172.21.15.32:0/3653144729 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55b19ccbb490 0x55b19d0c7cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:49:21.619 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:21.611+0000 7fa10276d740 1 --2- 172.21.15.32:0/3653144729 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b19cc9a7b0 0x55b19d0a9d60 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:49:21.619 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:21.611+0000 7fa10276d740 1 --2- 172.21.15.32:0/3653144729 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55b19cc925d0 0x55b19d1188b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:49:21.619 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:21.611+0000 7fa0da000640 0 client.5259 ms_handle_reset on v2:172.21.15.32:6832/211300533 2024-08-03T20:49:21.619 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:21.611+0000 7fa10276d740 1 -- 172.21.15.32:0/3653144729 >> 172.21.15.32:0/3653144729 conn(0x55b19ccaf5e0 msgr2=0x55b19d0a6b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:49:21.619 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:21.612+0000 7fa10276d740 1 -- 172.21.15.32:0/3653144729 shutdown_connections 2024-08-03T20:49:21.619 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:21.612+0000 7fa10276d740 1 -- 172.21.15.32:0/3653144729 wait complete. 2024-08-03T20:49:21.619 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:21.612+0000 7fa10276d740 20 client.5259 trim_cache size 0 max 16384 2024-08-03T20:49:21.619 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:21.612+0000 7fa0fb800640 1 -- 172.21.15.32:0/2035020027 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fa0d4077010 msgr2=0x7fa0d4079520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:49:21.619 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:21.612+0000 7fa0fb800640 1 --2- 172.21.15.32:0/2035020027 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fa0d4077010 0x7fa0d4079520 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fa0e8033000 tx=0x7fa0e8004730 comp rx=0 tx=0).stop 2024-08-03T20:49:21.619 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:21.612+0000 7fa0fb800640 1 -- 172.21.15.32:0/2035020027 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa0f40738e0 msgr2=0x7fa0f40ca1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:49:21.619 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:21.612+0000 7fa0fb800640 1 --2- 172.21.15.32:0/2035020027 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa0f40738e0 0x7fa0f40ca1b0 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7fa0ec00e990 tx=0x7fa0ec00ee50 comp rx=0 tx=0).stop 2024-08-03T20:49:21.619 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:21.612+0000 7fa0fb800640 1 -- 172.21.15.32:0/2035020027 shutdown_connections 2024-08-03T20:49:21.620 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:21.612+0000 7fa0fb800640 1 --2- 172.21.15.32:0/2035020027 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fa0d4077010 0x7fa0d4079520 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:49:21.620 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:21.612+0000 7fa0fb800640 1 --2- 172.21.15.32:0/2035020027 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa0f40cecc0 0x7fa0f40733a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:49:21.620 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:21.612+0000 7fa0fb800640 1 --2- 172.21.15.32:0/2035020027 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa0f40738e0 0x7fa0f40ca1b0 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:49:21.620 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:21.612+0000 7fa0fb800640 1 --2- 172.21.15.32:0/2035020027 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fa0f40cd210 0x7fa0f41564c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:49:21.620 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:21.612+0000 7fa0fb800640 1 -- 172.21.15.32:0/2035020027 >> 172.21.15.32:0/2035020027 conn(0x7fa0f40b1bd0 msgr2=0x7fa0f4143ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:49:21.620 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:21.613+0000 7fa0fb800640 1 -- 172.21.15.32:0/2035020027 shutdown_connections 2024-08-03T20:49:21.620 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:21.613+0000 7fa0fb800640 1 -- 172.21.15.32:0/2035020027 wait complete. 2024-08-03T20:49:21.638 DEBUG:teuthology.orchestra.run.smithi032:> stdin-killer -- python3 -c ' 2024-08-03T20:49:21.638 DEBUG:teuthology.orchestra.run.smithi032:> import sys 2024-08-03T20:49:21.638 DEBUG:teuthology.orchestra.run.smithi032:> import errno 2024-08-03T20:49:21.638 DEBUG:teuthology.orchestra.run.smithi032:> 2024-08-03T20:49:21.638 DEBUG:teuthology.orchestra.run.smithi032:> try: 2024-08-03T20:49:21.638 DEBUG:teuthology.orchestra.run.smithi032:> f = open("/home/ubuntu/cephtest/mnt.0/subdir/file_to_be_damaged", "w") 2024-08-03T20:49:21.638 DEBUG:teuthology.orchestra.run.smithi032:> f.close() 2024-08-03T20:49:21.638 DEBUG:teuthology.orchestra.run.smithi032:> except IOError as e: 2024-08-03T20:49:21.638 DEBUG:teuthology.orchestra.run.smithi032:> sys.exit(errno.EIO) 2024-08-03T20:49:21.639 DEBUG:teuthology.orchestra.run.smithi032:> ' 2024-08-03T20:49:21.747 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:21 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-08-03T20:49:21.755 INFO:teuthology.nuke:Clearing teuthology firewall rules... 2024-08-03T20:49:21.755 DEBUG:teuthology.orchestra.run.smithi032:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-08-03T20:49:21.883 DEBUG:teuthology.orchestra.run.smithi118:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-08-03T20:49:21.986 INFO:teuthology.nuke:Cleared teuthology firewall rules. 2024-08-03T20:49:21.986 INFO:tasks.cephfs.mount:Terminating background process 2024-08-03T20:49:21.987 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-08-03T20:49:21.987 DEBUG:teuthology.orchestra.run.smithi032:> dd if=/proc/self/mounts of=/dev/stdout 2024-08-03T20:49:22.022 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-08-03T20:49:22.022 DEBUG:teuthology.orchestra.run.smithi032:> dd if=/proc/self/mounts of=/dev/stdout 2024-08-03T20:49:22.098 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-08-03T20:49:22.368 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:22.366+0000 7fbeea000640 1 -- 172.21.15.32:0/2377420099 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbee415a310 msgr2=0x7fbee40aefb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:49:22.369 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:22.366+0000 7fbeea000640 1 --2- 172.21.15.32:0/2377420099 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbee415a310 0x7fbee40aefb0 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7fbee40cdfe0 tx=0x7fbed802a530 comp rx=0 tx=0).stop 2024-08-03T20:49:22.369 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:22.367+0000 7fbeea000640 1 -- 172.21.15.32:0/2377420099 shutdown_connections 2024-08-03T20:49:22.369 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:22.367+0000 7fbeea000640 1 --2- 172.21.15.32:0/2377420099 >> v2:172.21.15.118:3300/0 conn(0x7fbee40ce150 0x7fbee40ce560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:49:22.369 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:22.367+0000 7fbeea000640 1 --2- 172.21.15.32:0/2377420099 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbee415a310 0x7fbee40aefb0 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:49:22.369 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:22.367+0000 7fbeea000640 1 -- 172.21.15.32:0/2377420099 >> 172.21.15.32:0/2377420099 conn(0x7fbee40b2120 msgr2=0x7fbee40cfd60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:49:22.369 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:22.367+0000 7fbeea000640 1 -- 172.21.15.32:0/2377420099 shutdown_connections 2024-08-03T20:49:22.369 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:22.367+0000 7fbeea000640 1 -- 172.21.15.32:0/2377420099 wait complete. 2024-08-03T20:49:22.370 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:22.369+0000 7fbeea000640 1 Processor -- start 2024-08-03T20:49:22.371 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:22.369+0000 7fbeea000640 1 -- start start 2024-08-03T20:49:22.373 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:22.371+0000 7fbeea000640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fbee40ca3b0 0x7fbee40b4d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:49:22.374 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:22.371+0000 7fbeea000640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fbee40ce150 0x7fbee40c6280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:49:22.374 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:22.371+0000 7fbeea000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbee415a310 0x7fbee409d710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:49:22.374 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:22.371+0000 7fbeea000640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fbee4070860 con 0x7fbee415a310 2024-08-03T20:49:22.374 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:22.371+0000 7fbeea000640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7fbee40706e0 con 0x7fbee40ce150 2024-08-03T20:49:22.374 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:22.371+0000 7fbeea000640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7fbee40ceaa0 con 0x7fbee40ca3b0 2024-08-03T20:49:22.374 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:22.371+0000 7fbee9600640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbee415a310 0x7fbee409d710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:49:22.374 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:22.371+0000 7fbee9600640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbee415a310 0x7fbee409d710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:60000/0 (socket says 172.21.15.32:60000) 2024-08-03T20:49:22.374 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:22.371+0000 7fbee9600640 1 -- 172.21.15.32:0/1355119796 learned_addr learned my addr 172.21.15.32:0/1355119796 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:49:22.374 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:22.371+0000 7fbee8c00640 1 --2- 172.21.15.32:0/1355119796 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fbee40ca3b0 0x7fbee40b4d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:49:22.374 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:22.371+0000 7fbee3e00640 1 --2- 172.21.15.32:0/1355119796 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fbee40ce150 0x7fbee40c6280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:49:22.374 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:22.371+0000 7fbee9600640 1 -- 172.21.15.32:0/1355119796 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fbee40ca3b0 msgr2=0x7fbee40b4d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:49:22.374 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:22.371+0000 7fbee9600640 1 --2- 172.21.15.32:0/1355119796 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fbee40ca3b0 0x7fbee40b4d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:49:22.375 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:22.371+0000 7fbee9600640 1 -- 172.21.15.32:0/1355119796 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fbee40ce150 msgr2=0x7fbee40c6280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:49:22.375 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:22.371+0000 7fbee9600640 1 --2- 172.21.15.32:0/1355119796 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fbee40ce150 0x7fbee40c6280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:49:22.375 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:22.371+0000 7fbee9600640 1 -- 172.21.15.32:0/1355119796 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbee40c40c0 con 0x7fbee415a310 2024-08-03T20:49:22.375 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:22.371+0000 7fbee3e00640 1 --2- 172.21.15.32:0/1355119796 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fbee40ce150 0x7fbee40c6280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-03T20:49:22.375 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:22.372+0000 7fbee9600640 1 --2- 172.21.15.32:0/1355119796 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbee415a310 0x7fbee409d710 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7fbedc00e990 tx=0x7fbedc00ee50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:49:22.375 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:22.372+0000 7fbee0c00640 1 -- 172.21.15.32:0/1355119796 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbedc004d20 con 0x7fbee415a310 2024-08-03T20:49:22.375 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:22.372+0000 7fbee0c00640 1 -- 172.21.15.32:0/1355119796 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fbedc00caa0 con 0x7fbee415a310 2024-08-03T20:49:22.375 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:22.372+0000 7fbeea000640 1 -- 172.21.15.32:0/1355119796 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbee40c4380 con 0x7fbee415a310 2024-08-03T20:49:22.375 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:22.372+0000 7fbee0c00640 1 -- 172.21.15.32:0/1355119796 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbedc005020 con 0x7fbee415a310 2024-08-03T20:49:22.375 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:22.372+0000 7fbeea000640 1 -- 172.21.15.32:0/1355119796 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbee40c4940 con 0x7fbee415a310 2024-08-03T20:49:22.376 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:22.373+0000 7fbeea000640 1 -- 172.21.15.32:0/1355119796 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbee40ce560 con 0x7fbee415a310 2024-08-03T20:49:22.376 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:22.375+0000 7fbee0c00640 1 -- 172.21.15.32:0/1355119796 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7fbedc0051c0 con 0x7fbee415a310 2024-08-03T20:49:22.377 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:22.376+0000 7fbee0c00640 1 --2- 172.21.15.32:0/1355119796 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fbec4077040 0x7fbec4079550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:49:22.378 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:22.376+0000 7fbee0c00640 1 -- 172.21.15.32:0/1355119796 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(33..33 src has 1..33) ==== 5795+0+0 (secure 0 0 0) 0x7fbedc0990d0 con 0x7fbee415a310 2024-08-03T20:49:22.378 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:22.377+0000 7fbee8c00640 1 --2- 172.21.15.32:0/1355119796 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fbec4077040 0x7fbec4079550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:49:22.379 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:22.377+0000 7fbee8c00640 1 --2- 172.21.15.32:0/1355119796 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fbec4077040 0x7fbec4079550 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fbed8034000 tx=0x7fbed8004730 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:49:22.383 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:22.381+0000 7fbee0c00640 1 -- 172.21.15.32:0/1355119796 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7fbedc016030 con 0x7fbee415a310 2024-08-03T20:49:22.602 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:22.599+0000 7fbeea000640 1 -- 172.21.15.32:0/1355119796 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7fbee40ae420 con 0x7fbee415a310 2024-08-03T20:49:22.603 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:22.600+0000 7fbee0c00640 1 -- 172.21.15.32:0/1355119796 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 2 entries v33) ==== 80+0+122 (secure 0 0 0) 0x7fbedc061c10 con 0x7fbee415a310 2024-08-03T20:49:22.604 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6835/1186001746 2024-08-04T20:48:38.076677+0000 2024-08-03T20:49:22.604 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6832/1186001746 2024-08-04T20:48:38.076677+0000 2024-08-03T20:49:22.606 INFO:teuthology.orchestra.run.smithi032.stderr:listed 2 entries 2024-08-03T20:49:22.606 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:22.602+0000 7fbeea000640 1 -- 172.21.15.32:0/1355119796 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fbec4077040 msgr2=0x7fbec4079550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:49:22.606 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:22.602+0000 7fbeea000640 1 --2- 172.21.15.32:0/1355119796 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fbec4077040 0x7fbec4079550 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fbed8034000 tx=0x7fbed8004730 comp rx=0 tx=0).stop 2024-08-03T20:49:22.607 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:22.602+0000 7fbeea000640 1 -- 172.21.15.32:0/1355119796 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbee415a310 msgr2=0x7fbee409d710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:49:22.607 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:22.602+0000 7fbeea000640 1 --2- 172.21.15.32:0/1355119796 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbee415a310 0x7fbee409d710 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7fbedc00e990 tx=0x7fbedc00ee50 comp rx=0 tx=0).stop 2024-08-03T20:49:22.607 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:22.602+0000 7fbeea000640 1 -- 172.21.15.32:0/1355119796 shutdown_connections 2024-08-03T20:49:22.607 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:22.602+0000 7fbeea000640 1 --2- 172.21.15.32:0/1355119796 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fbec4077040 0x7fbec4079550 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:49:22.607 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:22.602+0000 7fbeea000640 1 --2- 172.21.15.32:0/1355119796 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fbee40ce150 0x7fbee40c6280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:49:22.608 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:22.602+0000 7fbeea000640 1 --2- 172.21.15.32:0/1355119796 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbee415a310 0x7fbee409d710 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:49:22.608 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:22.602+0000 7fbeea000640 1 --2- 172.21.15.32:0/1355119796 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fbee40ca3b0 0x7fbee40b4d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:49:22.608 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:22.602+0000 7fbeea000640 1 -- 172.21.15.32:0/1355119796 >> 172.21.15.32:0/1355119796 conn(0x7fbee40b2120 msgr2=0x7fbee40b0d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:49:22.608 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:22.603+0000 7fbeea000640 1 -- 172.21.15.32:0/1355119796 shutdown_connections 2024-08-03T20:49:22.608 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:22.603+0000 7fbeea000640 1 -- 172.21.15.32:0/1355119796 wait complete. 2024-08-03T20:49:22.627 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-08-03T20:49:22.627 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-08-03T20:49:22.628 DEBUG:teuthology.orchestra.run.smithi032:> sudo umount /home/ubuntu/cephtest/mnt.0 2024-08-03T20:49:22.691 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi032.front.sepia.ceph.com 2024-08-03T20:49:22.691 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-08-03T20:49:22.691 DEBUG:teuthology.orchestra.run.smithi032:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-08-03T20:49:22.727 INFO:tasks.cephfs.mount:Terminating background process 2024-08-03T20:49:22.727 DEBUG:teuthology.orchestra.run.smithi118:> set -ex 2024-08-03T20:49:22.727 DEBUG:teuthology.orchestra.run.smithi118:> dd if=/proc/self/mounts of=/dev/stdout 2024-08-03T20:49:22.767 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052663,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497944k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248972k,nr_inodes=812243,mode=700,uid=1000,gid=1316 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497944k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-08-03T20:49:22.767 INFO:tasks.cephfs.mount:Terminating background process 2024-08-03T20:49:22.767 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-08-03T20:49:22.768 DEBUG:teuthology.orchestra.run.smithi032:> dd if=/proc/self/mounts of=/dev/stdout 2024-08-03T20:49:22.803 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053453,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6499208k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 none /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249604k,nr_inodes=812401,mode=700,uid=1000,gid=1316 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6499208k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-08-03T20:49:22.803 INFO:tasks.cephfs.mount:Terminating background process 2024-08-03T20:49:22.803 DEBUG:teuthology.orchestra.run.smithi118:> set -ex 2024-08-03T20:49:22.803 DEBUG:teuthology.orchestra.run.smithi118:> dd if=/proc/self/mounts of=/dev/stdout 2024-08-03T20:49:22.845 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052663,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497944k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248972k,nr_inodes=812243,mode=700,uid=1000,gid=1316 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497944k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-08-03T20:49:22.845 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-08-03T20:49:23.072 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:23.070+0000 7fcfa9600640 1 -- 172.21.15.32:0/16232026 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcfa4070c90 msgr2=0x7fcfa415c970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:49:23.073 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:23.070+0000 7fcfa9600640 1 --2- 172.21.15.32:0/16232026 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcfa4070c90 0x7fcfa415c970 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7fcfa415a560 tx=0x7fcf9c02a4b0 comp rx=0 tx=0).stop 2024-08-03T20:49:23.073 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:23.071+0000 7fcfa9600640 1 -- 172.21.15.32:0/16232026 shutdown_connections 2024-08-03T20:49:23.073 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:23.071+0000 7fcfa9600640 1 --2- 172.21.15.32:0/16232026 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcfa4070c90 0x7fcfa415c970 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:49:23.073 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:23.071+0000 7fcfa9600640 1 --2- 172.21.15.32:0/16232026 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fcfa41604a0 0x7fcfa4070750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:49:23.073 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:23.071+0000 7fcfa9600640 1 --2- 172.21.15.32:0/16232026 >> v2:172.21.15.118:3300/0 conn(0x7fcfa4159f70 0x7fcfa415ff60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:49:23.073 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:23.071+0000 7fcfa9600640 1 -- 172.21.15.32:0/16232026 >> 172.21.15.32:0/16232026 conn(0x7fcfa40d00f0 msgr2=0x7fcfa40d0500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:49:23.073 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:23.071+0000 7fcfa9600640 1 -- 172.21.15.32:0/16232026 shutdown_connections 2024-08-03T20:49:23.073 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:23.071+0000 7fcfa9600640 1 -- 172.21.15.32:0/16232026 wait complete. 2024-08-03T20:49:23.075 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:23.073+0000 7fcfa9600640 1 Processor -- start 2024-08-03T20:49:23.075 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:23.073+0000 7fcfa9600640 1 -- start start 2024-08-03T20:49:23.077 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:23.075+0000 7fcfa9600640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcfa4070c90 0x7fcfa40b5600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:49:23.077 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:23.075+0000 7fcfa9600640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fcfa4159f70 0x7fcfa40b4690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:49:23.077 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:23.075+0000 7fcfa9600640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fcfa41604a0 0x7fcfa40b4bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:49:23.077 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:23.075+0000 7fcfa9600640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fcfa415d240 con 0x7fcfa4070c90 2024-08-03T20:49:23.077 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:23.075+0000 7fcfa9600640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7fcfa415d0c0 con 0x7fcfa41604a0 2024-08-03T20:49:23.077 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:23.075+0000 7fcfa9600640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7fcfa415d3c0 con 0x7fcfa4159f70 2024-08-03T20:49:23.077 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:23.075+0000 7fcfa3e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcfa4070c90 0x7fcfa40b5600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:49:23.077 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:23.075+0000 7fcfa3e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcfa4070c90 0x7fcfa40b5600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:60014/0 (socket says 172.21.15.32:60014) 2024-08-03T20:49:23.078 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:23.075+0000 7fcfa3e00640 1 -- 172.21.15.32:0/3081983196 learned_addr learned my addr 172.21.15.32:0/3081983196 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:49:23.078 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:23.075+0000 7fcfa3400640 1 --2- 172.21.15.32:0/3081983196 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fcfa4159f70 0x7fcfa40b4690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:49:23.078 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:23.075+0000 7fcfa8c00640 1 --2- 172.21.15.32:0/3081983196 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fcfa41604a0 0x7fcfa40b4bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:49:23.078 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:23.075+0000 7fcfa3e00640 1 -- 172.21.15.32:0/3081983196 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fcfa4159f70 msgr2=0x7fcfa40b4690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:49:23.078 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:23.075+0000 7fcfa3e00640 1 --2- 172.21.15.32:0/3081983196 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fcfa4159f70 0x7fcfa40b4690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:49:23.078 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:23.075+0000 7fcfa3e00640 1 -- 172.21.15.32:0/3081983196 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fcfa41604a0 msgr2=0x7fcfa40b4bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:49:23.078 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:23.075+0000 7fcfa3e00640 1 --2- 172.21.15.32:0/3081983196 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fcfa41604a0 0x7fcfa40b4bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:49:23.078 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:23.075+0000 7fcfa3e00640 1 -- 172.21.15.32:0/3081983196 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcfa40a01b0 con 0x7fcfa4070c90 2024-08-03T20:49:23.078 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:23.076+0000 7fcfa3e00640 1 --2- 172.21.15.32:0/3081983196 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcfa4070c90 0x7fcfa40b5600 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7fcf9400e9c0 tx=0x7fcf9400ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:49:23.079 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:23.076+0000 7fcf9be00640 1 -- 172.21.15.32:0/3081983196 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf9400cd80 con 0x7fcfa4070c90 2024-08-03T20:49:23.079 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:23.076+0000 7fcf9be00640 1 -- 172.21.15.32:0/3081983196 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fcf94011470 con 0x7fcfa4070c90 2024-08-03T20:49:23.079 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:23.076+0000 7fcfa9600640 1 -- 172.21.15.32:0/3081983196 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcfa40cbfe0 con 0x7fcfa4070c90 2024-08-03T20:49:23.079 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:23.076+0000 7fcf9be00640 1 -- 172.21.15.32:0/3081983196 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf940118c0 con 0x7fcfa4070c90 2024-08-03T20:49:23.079 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:23.076+0000 7fcfa9600640 1 -- 172.21.15.32:0/3081983196 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcfa40cc2c0 con 0x7fcfa4070c90 2024-08-03T20:49:23.080 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:23.077+0000 7fcfa9600640 1 -- 172.21.15.32:0/3081983196 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcfa415ff60 con 0x7fcfa4070c90 2024-08-03T20:49:23.080 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:23.078+0000 7fcf9be00640 1 -- 172.21.15.32:0/3081983196 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7fcf94011a60 con 0x7fcfa4070c90 2024-08-03T20:49:23.080 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:23.079+0000 7fcf9be00640 1 --2- 172.21.15.32:0/3081983196 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fcf84077040 0x7fcf84079550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:49:23.081 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:23.079+0000 7fcf9be00640 1 -- 172.21.15.32:0/3081983196 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(33..33 src has 1..33) ==== 5795+0+0 (secure 0 0 0) 0x7fcf94099520 con 0x7fcfa4070c90 2024-08-03T20:49:23.081 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:23.080+0000 7fcfa3400640 1 --2- 172.21.15.32:0/3081983196 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fcf84077040 0x7fcf84079550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:49:23.082 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:23.080+0000 7fcfa3400640 1 --2- 172.21.15.32:0/3081983196 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fcf84077040 0x7fcf84079550 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fcf90005fd0 tx=0x7fcf90005850 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:49:23.085 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:23.084+0000 7fcf9be00640 1 -- 172.21.15.32:0/3081983196 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7fcf94016030 con 0x7fcfa4070c90 2024-08-03T20:49:23.333 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:23.332+0000 7fcfa9600640 1 -- 172.21.15.32:0/3081983196 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fcfa41538e0 con 0x7fcfa4070c90 2024-08-03T20:49:23.334 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:23.333+0000 7fcf9be00640 1 -- 172.21.15.32:0/3081983196 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+5227 (secure 0 0 0) 0x7fcf94062620 con 0x7fcfa4070c90 2024-08-03T20:49:23.334 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-03T20:49:23.334 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":25,"btime":"2024-08-03T20:48:57:346845+0000","default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4643,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6836/4002526832","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6834","nonce":4002526832},{"type":"v1","addr":"172.21.15.118:6836","nonce":4002526832}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":4646,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6837/3036256187","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6835","nonce":3036256187},{"type":"v1","addr":"172.21.15.118:6837","nonce":3036256187}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":4652,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6834/2104418334","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6833","nonce":2104418334},{"type":"v1","addr":"172.21.15.32:6834","nonce":2104418334}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-08-03T20:48:26.652521+0000","modified":"2024-08-03T20:48:57.101761+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5070},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5070":{"gid":5070,"name":"a","rank":0,"incarnation":20,"state":"up:active","state_seq":8,"addr":"172.21.15.32:6835/211300533","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":211300533},{"type":"v1","addr":"172.21.15.32:6835","nonce":211300533}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5070,"qdb_cluster":[5070]},"id":2}]} 2024-08-03T20:49:23.335 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 25 2024-08-03T20:49:23.337 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:23.335+0000 7fcfa9600640 1 -- 172.21.15.32:0/3081983196 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fcf84077040 msgr2=0x7fcf84079550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:49:23.337 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:23.335+0000 7fcfa9600640 1 --2- 172.21.15.32:0/3081983196 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fcf84077040 0x7fcf84079550 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fcf90005fd0 tx=0x7fcf90005850 comp rx=0 tx=0).stop 2024-08-03T20:49:23.338 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:23.335+0000 7fcfa9600640 1 -- 172.21.15.32:0/3081983196 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcfa4070c90 msgr2=0x7fcfa40b5600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:49:23.338 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:23.335+0000 7fcfa9600640 1 --2- 172.21.15.32:0/3081983196 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcfa4070c90 0x7fcfa40b5600 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7fcf9400e9c0 tx=0x7fcf9400ee80 comp rx=0 tx=0).stop 2024-08-03T20:49:23.338 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:23.335+0000 7fcfa9600640 1 -- 172.21.15.32:0/3081983196 shutdown_connections 2024-08-03T20:49:23.338 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:23.335+0000 7fcfa9600640 1 --2- 172.21.15.32:0/3081983196 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fcf84077040 0x7fcf84079550 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:49:23.338 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:23.335+0000 7fcfa9600640 1 --2- 172.21.15.32:0/3081983196 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fcfa41604a0 0x7fcfa40b4bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:49:23.338 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:23.335+0000 7fcfa9600640 1 --2- 172.21.15.32:0/3081983196 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fcfa4159f70 0x7fcfa40b4690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:49:23.338 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:23.335+0000 7fcfa9600640 1 --2- 172.21.15.32:0/3081983196 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcfa4070c90 0x7fcfa40b5600 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:49:23.339 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:23.335+0000 7fcfa9600640 1 -- 172.21.15.32:0/3081983196 >> 172.21.15.32:0/3081983196 conn(0x7fcfa40d00f0 msgr2=0x7fcfa415bbf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:49:23.339 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:23.335+0000 7fcfa9600640 1 -- 172.21.15.32:0/3081983196 shutdown_connections 2024-08-03T20:49:23.339 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:23.335+0000 7fcfa9600640 1 -- 172.21.15.32:0/3081983196 wait complete. 2024-08-03T20:49:23.363 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-08-03T20:49:23.593 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:23.590+0000 7fc948a00640 1 -- 172.21.15.32:0/2549803150 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc9440d4bb0 msgr2=0x7fc9440d1c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:49:23.593 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:23.590+0000 7fc948a00640 1 --2- 172.21.15.32:0/2549803150 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc9440d4bb0 0x7fc9440d1c60 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7fc9440d5190 tx=0x7fc93402a4d0 comp rx=0 tx=0).stop 2024-08-03T20:49:23.593 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:23.591+0000 7fc948a00640 1 -- 172.21.15.32:0/2549803150 shutdown_connections 2024-08-03T20:49:23.593 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:23.591+0000 7fc948a00640 1 --2- 172.21.15.32:0/2549803150 >> v2:172.21.15.118:3300/0 conn(0x7fc9440d2290 0x7fc9440d26a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:49:23.593 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:23.591+0000 7fc948a00640 1 --2- 172.21.15.32:0/2549803150 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc9440d4bb0 0x7fc9440d1c60 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:49:23.593 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:23.591+0000 7fc948a00640 1 -- 172.21.15.32:0/2549803150 >> 172.21.15.32:0/2549803150 conn(0x7fc9440d4090 msgr2=0x7fc9440d44a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:49:23.593 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:23.591+0000 7fc948a00640 1 -- 172.21.15.32:0/2549803150 shutdown_connections 2024-08-03T20:49:23.593 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:23.591+0000 7fc948a00640 1 -- 172.21.15.32:0/2549803150 wait complete. 2024-08-03T20:49:23.594 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:23.593+0000 7fc948a00640 1 Processor -- start 2024-08-03T20:49:23.594 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:23.593+0000 7fc948a00640 1 -- start start 2024-08-03T20:49:23.597 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:23.594+0000 7fc948a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc9440d0aa0 0x7fc94415d040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:49:23.597 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:23.594+0000 7fc948a00640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fc9440d2290 0x7fc9441595f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:49:23.597 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:23.594+0000 7fc948a00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc9440d4bb0 0x7fc944163640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:49:23.597 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:23.594+0000 7fc948a00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fc9440d15c0 con 0x7fc9440d0aa0 2024-08-03T20:49:23.597 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:23.594+0000 7fc948a00640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7fc9440cfe50 con 0x7fc9440d4bb0 2024-08-03T20:49:23.597 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:23.594+0000 7fc948a00640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7fc9440d1740 con 0x7fc9440d2290 2024-08-03T20:49:23.597 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:23.595+0000 7fc942800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc9440d0aa0 0x7fc94415d040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:49:23.597 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:23.595+0000 7fc942800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc9440d0aa0 0x7fc94415d040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:60026/0 (socket says 172.21.15.32:60026) 2024-08-03T20:49:23.597 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:23.595+0000 7fc942800640 1 -- 172.21.15.32:0/84348821 learned_addr learned my addr 172.21.15.32:0/84348821 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:49:23.597 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:23.595+0000 7fc93be00640 1 --2- 172.21.15.32:0/84348821 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fc9440d2290 0x7fc9441595f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:49:23.597 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:23.595+0000 7fc943200640 1 --2- 172.21.15.32:0/84348821 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc9440d4bb0 0x7fc944163640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:49:23.597 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:23.595+0000 7fc942800640 1 -- 172.21.15.32:0/84348821 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fc9440d2290 msgr2=0x7fc9441595f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:49:23.597 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:23.595+0000 7fc942800640 1 --2- 172.21.15.32:0/84348821 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fc9440d2290 0x7fc9441595f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:49:23.598 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:23.595+0000 7fc942800640 1 -- 172.21.15.32:0/84348821 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc9440d4bb0 msgr2=0x7fc944163640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:49:23.598 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:23.595+0000 7fc942800640 1 --2- 172.21.15.32:0/84348821 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc9440d4bb0 0x7fc944163640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:49:23.598 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:23.595+0000 7fc942800640 1 -- 172.21.15.32:0/84348821 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc944159b30 con 0x7fc9440d0aa0 2024-08-03T20:49:23.598 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:23.595+0000 7fc942800640 1 --2- 172.21.15.32:0/84348821 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc9440d0aa0 0x7fc94415d040 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7fc934034040 tx=0x7fc934003910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:49:23.599 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:23.596+0000 7fc93aa00640 1 -- 172.21.15.32:0/84348821 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc93403e030 con 0x7fc9440d0aa0 2024-08-03T20:49:23.599 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:23.596+0000 7fc93aa00640 1 -- 172.21.15.32:0/84348821 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fc934003e00 con 0x7fc9440d0aa0 2024-08-03T20:49:23.599 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:23.596+0000 7fc93aa00640 1 -- 172.21.15.32:0/84348821 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc934035040 con 0x7fc9440d0aa0 2024-08-03T20:49:23.599 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:23.596+0000 7fc948a00640 1 -- 172.21.15.32:0/84348821 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc944159e60 con 0x7fc9440d0aa0 2024-08-03T20:49:23.599 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:23.596+0000 7fc948a00640 1 -- 172.21.15.32:0/84348821 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc944164140 con 0x7fc9440d0aa0 2024-08-03T20:49:23.599 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:23.597+0000 7fc948a00640 1 -- 172.21.15.32:0/84348821 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc9441643a0 con 0x7fc9440d0aa0 2024-08-03T20:49:23.599 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:23.598+0000 7fc93aa00640 1 -- 172.21.15.32:0/84348821 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7fc934041030 con 0x7fc9440d0aa0 2024-08-03T20:49:23.600 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:23.599+0000 7fc93aa00640 1 --2- 172.21.15.32:0/84348821 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fc920077010 0x7fc920079520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:49:23.601 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:23.599+0000 7fc93aa00640 1 -- 172.21.15.32:0/84348821 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(33..33 src has 1..33) ==== 5795+0+0 (secure 0 0 0) 0x7fc9340b9060 con 0x7fc9440d0aa0 2024-08-03T20:49:23.601 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:23.600+0000 7fc93be00640 1 --2- 172.21.15.32:0/84348821 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fc920077010 0x7fc920079520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:49:23.602 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:23.600+0000 7fc93be00640 1 --2- 172.21.15.32:0/84348821 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fc920077010 0x7fc920079520 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fc92c005fd0 tx=0x7fc92c005850 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:49:23.605 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:23.603+0000 7fc93aa00640 1 -- 172.21.15.32:0/84348821 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7fc944159b30 con 0x7fc9440d0aa0 2024-08-03T20:49:23.846 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:23.844+0000 7fc948a00640 1 -- 172.21.15.32:0/84348821 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc9440d26a0 con 0x7fc9440d0aa0 2024-08-03T20:49:23.846 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:23.845+0000 7fc93aa00640 1 -- 172.21.15.32:0/84348821 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+5227 (secure 0 0 0) 0x7fc9440d2be0 con 0x7fc9440d0aa0 2024-08-03T20:49:23.847 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-03T20:49:23.847 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":25,"btime":"2024-08-03T20:48:57:346845+0000","default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4643,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6836/4002526832","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6834","nonce":4002526832},{"type":"v1","addr":"172.21.15.118:6836","nonce":4002526832}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":4646,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6837/3036256187","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6835","nonce":3036256187},{"type":"v1","addr":"172.21.15.118:6837","nonce":3036256187}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":4652,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6834/2104418334","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6833","nonce":2104418334},{"type":"v1","addr":"172.21.15.32:6834","nonce":2104418334}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-08-03T20:48:26.652521+0000","modified":"2024-08-03T20:48:57.101761+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5070},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5070":{"gid":5070,"name":"a","rank":0,"incarnation":20,"state":"up:active","state_seq":8,"addr":"172.21.15.32:6835/211300533","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":211300533},{"type":"v1","addr":"172.21.15.32:6835","nonce":211300533}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5070,"qdb_cluster":[5070]},"id":2}]} 2024-08-03T20:49:23.847 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 25 2024-08-03T20:49:23.851 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:23.847+0000 7fc948a00640 1 -- 172.21.15.32:0/84348821 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fc920077010 msgr2=0x7fc920079520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:49:23.851 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:23.847+0000 7fc948a00640 1 --2- 172.21.15.32:0/84348821 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fc920077010 0x7fc920079520 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fc92c005fd0 tx=0x7fc92c005850 comp rx=0 tx=0).stop 2024-08-03T20:49:23.851 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:23.847+0000 7fc948a00640 1 -- 172.21.15.32:0/84348821 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc9440d0aa0 msgr2=0x7fc94415d040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:49:23.851 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:23.847+0000 7fc948a00640 1 --2- 172.21.15.32:0/84348821 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc9440d0aa0 0x7fc94415d040 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7fc934034040 tx=0x7fc934003910 comp rx=0 tx=0).stop 2024-08-03T20:49:23.851 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:23.847+0000 7fc948a00640 1 -- 172.21.15.32:0/84348821 shutdown_connections 2024-08-03T20:49:23.851 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:23.847+0000 7fc948a00640 1 --2- 172.21.15.32:0/84348821 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fc920077010 0x7fc920079520 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:49:23.851 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:23.847+0000 7fc948a00640 1 --2- 172.21.15.32:0/84348821 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc9440d4bb0 0x7fc944163640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:49:23.851 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:23.847+0000 7fc948a00640 1 --2- 172.21.15.32:0/84348821 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fc9440d2290 0x7fc9441595f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:49:23.851 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:23.847+0000 7fc948a00640 1 --2- 172.21.15.32:0/84348821 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc9440d0aa0 0x7fc94415d040 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:49:23.851 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:23.847+0000 7fc948a00640 1 -- 172.21.15.32:0/84348821 >> 172.21.15.32:0/84348821 conn(0x7fc9440d4090 msgr2=0x7fc94415f790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:49:23.852 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:23.847+0000 7fc948a00640 1 -- 172.21.15.32:0/84348821 shutdown_connections 2024-08-03T20:49:23.852 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:23.848+0000 7fc948a00640 1 -- 172.21.15.32:0/84348821 wait complete. 2024-08-03T20:49:23.871 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-08-03T20:49:24.101 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:24.099+0000 7f6367e00640 1 -- 172.21.15.32:0/4179881960 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f63600ade80 msgr2=0x7f63600ae290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:49:24.102 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:24.099+0000 7f6367e00640 1 --2- 172.21.15.32:0/4179881960 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f63600ade80 0x7f63600ae290 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f6358005fd0 tx=0x7f635802aac0 comp rx=0 tx=0).stop 2024-08-03T20:49:24.102 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:24.099+0000 7f6367e00640 1 -- 172.21.15.32:0/4179881960 shutdown_connections 2024-08-03T20:49:24.102 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:24.099+0000 7f6367e00640 1 --2- 172.21.15.32:0/4179881960 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f63600ade80 0x7f63600ae290 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:49:24.102 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:24.099+0000 7f6367e00640 1 --2- 172.21.15.32:0/4179881960 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f636006f720 0x7f636006fb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:49:24.102 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:24.099+0000 7f6367e00640 1 -- 172.21.15.32:0/4179881960 >> 172.21.15.32:0/4179881960 conn(0x7f63600b1bd0 msgr2=0x7f63600b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:49:24.102 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:24.099+0000 7f6367e00640 1 -- 172.21.15.32:0/4179881960 shutdown_connections 2024-08-03T20:49:24.102 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:24.099+0000 7f6367e00640 1 -- 172.21.15.32:0/4179881960 wait complete. 2024-08-03T20:49:24.103 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:24.102+0000 7f6367e00640 1 Processor -- start 2024-08-03T20:49:24.103 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:24.102+0000 7f6367e00640 1 -- start start 2024-08-03T20:49:24.106 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:24.103+0000 7f6367e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f636006f720 0x7f6360148d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:49:24.106 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:24.103+0000 7f6367e00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f63600ade80 0x7f6360159610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:49:24.107 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:24.103+0000 7f6367e00640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f63600b39a0 0x7f63601479e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:49:24.107 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:24.103+0000 7f6367e00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f63600aebe0 con 0x7f636006f720 2024-08-03T20:49:24.107 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:24.103+0000 7f6367e00640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f63600aea60 con 0x7f63600ade80 2024-08-03T20:49:24.107 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:24.103+0000 7f6367e00640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7f63600aed60 con 0x7f63600b39a0 2024-08-03T20:49:24.107 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:24.104+0000 7f6365400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f636006f720 0x7f6360148d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:49:24.107 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:24.104+0000 7f6365400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f636006f720 0x7f6360148d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:60036/0 (socket says 172.21.15.32:60036) 2024-08-03T20:49:24.107 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:24.104+0000 7f6365400640 1 -- 172.21.15.32:0/2363489801 learned_addr learned my addr 172.21.15.32:0/2363489801 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:49:24.107 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:24.104+0000 7f6365e00640 1 --2- 172.21.15.32:0/2363489801 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f63600b39a0 0x7f63601479e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:49:24.107 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:24.104+0000 7f6364a00640 1 --2- 172.21.15.32:0/2363489801 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f63600ade80 0x7f6360159610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:49:24.107 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:24.104+0000 7f6365400640 1 -- 172.21.15.32:0/2363489801 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f63600b39a0 msgr2=0x7f63601479e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:49:24.107 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:24.104+0000 7f6365400640 1 --2- 172.21.15.32:0/2363489801 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f63600b39a0 0x7f63601479e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:49:24.107 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:24.104+0000 7f6365400640 1 -- 172.21.15.32:0/2363489801 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f63600ade80 msgr2=0x7f6360159610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:49:24.107 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:24.104+0000 7f6365400640 1 --2- 172.21.15.32:0/2363489801 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f63600ade80 0x7f6360159610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:49:24.108 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:24.104+0000 7f6365400640 1 -- 172.21.15.32:0/2363489801 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6360160e50 con 0x7f636006f720 2024-08-03T20:49:24.108 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:24.104+0000 7f6365400640 1 --2- 172.21.15.32:0/2363489801 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f636006f720 0x7f6360148d60 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f635400d750 tx=0x7f635400dc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:49:24.108 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:24.105+0000 7f635d600640 1 -- 172.21.15.32:0/2363489801 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6354017070 con 0x7f636006f720 2024-08-03T20:49:24.108 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:24.105+0000 7f635d600640 1 -- 172.21.15.32:0/2363489801 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f6354004310 con 0x7f636006f720 2024-08-03T20:49:24.108 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:24.105+0000 7f635d600640 1 -- 172.21.15.32:0/2363489801 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f635400de60 con 0x7f636006f720 2024-08-03T20:49:24.108 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:24.105+0000 7f6367e00640 1 -- 172.21.15.32:0/2363489801 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f63601482c0 con 0x7f636006f720 2024-08-03T20:49:24.108 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:24.105+0000 7f6367e00640 1 -- 172.21.15.32:0/2363489801 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6360148560 con 0x7f636006f720 2024-08-03T20:49:24.108 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:24.106+0000 7f6367e00640 1 -- 172.21.15.32:0/2363489801 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f636006fb30 con 0x7f636006f720 2024-08-03T20:49:24.109 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:24.107+0000 7f635d600640 1 -- 172.21.15.32:0/2363489801 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f6354009c30 con 0x7f636006f720 2024-08-03T20:49:24.109 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:24.108+0000 7f635d600640 1 --2- 172.21.15.32:0/2363489801 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f6340077010 0x7f6340079520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:49:24.110 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:24.108+0000 7f635d600640 1 -- 172.21.15.32:0/2363489801 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(33..33 src has 1..33) ==== 5795+0+0 (secure 0 0 0) 0x7f6354098c10 con 0x7f636006f720 2024-08-03T20:49:24.110 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:24.109+0000 7f6364a00640 1 --2- 172.21.15.32:0/2363489801 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f6340077010 0x7f6340079520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:49:24.111 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:24.109+0000 7f6364a00640 1 --2- 172.21.15.32:0/2363489801 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f6340077010 0x7f6340079520 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f636015c710 tx=0x7f634c009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:49:24.114 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:24.112+0000 7f635d600640 1 -- 172.21.15.32:0/2363489801 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7f6354020020 con 0x7f636006f720 2024-08-03T20:49:24.322 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:24.321+0000 7f6367e00640 1 -- 172.21.15.32:0/2363489801 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f6360074a90 con 0x7f636006f720 2024-08-03T20:49:24.323 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:24.322+0000 7f635d600640 1 -- 172.21.15.32:0/2363489801 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v33) ==== 74+0+16738 (secure 0 0 0) 0x7f6354061750 con 0x7f636006f720 2024-08-03T20:49:24.324 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-03T20:49:24.324 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":33,"fsid":"40ab141a-01d7-49b9-ae3c-2edec9876e95","created":"2024-08-03T20:47:14.957362+0000","modified":"2024-08-03T20:48:38.076713+0000","last_up_change":"2024-08-03T20:47:28.874451+0000","last_in_change":"2024-08-03T20:47:20.079442+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-08-03T20:47:28.261300+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_metadata","create_time":"2024-08-03T20:48:24.969840+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"31","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data","create_time":"2024-08-03T20:48:25.603595+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"32","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5a83e759-9035-438b-889a-0a6af1f0a0d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6816","nonce":3066156116},{"type":"v1","addr":"172.21.15.32:6817","nonce":3066156116}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6818","nonce":3066156116},{"type":"v1","addr":"172.21.15.32:6819","nonce":3066156116}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6822","nonce":3066156116},{"type":"v1","addr":"172.21.15.32:6823","nonce":3066156116}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6820","nonce":3066156116},{"type":"v1","addr":"172.21.15.32:6821","nonce":3066156116}]},"public_addr":"172.21.15.32:6817/3066156116","cluster_addr":"172.21.15.32:6819/3066156116","heartbeat_back_addr":"172.21.15.32:6823/3066156116","heartbeat_front_addr":"172.21.15.32:6821/3066156116","state":["exists","up"]},{"osd":1,"uuid":"d7c5c210-da4f-49f9-b9e7-1dfb02daa4bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6824","nonce":1551241759},{"type":"v1","addr":"172.21.15.32:6825","nonce":1551241759}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6826","nonce":1551241759},{"type":"v1","addr":"172.21.15.32:6827","nonce":1551241759}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6830","nonce":1551241759},{"type":"v1","addr":"172.21.15.32:6831","nonce":1551241759}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6828","nonce":1551241759},{"type":"v1","addr":"172.21.15.32:6829","nonce":1551241759}]},"public_addr":"172.21.15.32:6825/1551241759","cluster_addr":"172.21.15.32:6827/1551241759","heartbeat_back_addr":"172.21.15.32:6831/1551241759","heartbeat_front_addr":"172.21.15.32:6829/1551241759","state":["exists","up"]},{"osd":2,"uuid":"57f2e6de-a201-45c2-8981-8e56a4875ebd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6808","nonce":3806708944},{"type":"v1","addr":"172.21.15.32:6809","nonce":3806708944}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6810","nonce":3806708944},{"type":"v1","addr":"172.21.15.32:6811","nonce":3806708944}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6814","nonce":3806708944},{"type":"v1","addr":"172.21.15.32:6815","nonce":3806708944}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6812","nonce":3806708944},{"type":"v1","addr":"172.21.15.32:6813","nonce":3806708944}]},"public_addr":"172.21.15.32:6809/3806708944","cluster_addr":"172.21.15.32:6811/3806708944","heartbeat_back_addr":"172.21.15.32:6815/3806708944","heartbeat_front_addr":"172.21.15.32:6813/3806708944","state":["exists","up"]},{"osd":3,"uuid":"6439a280-da45-4148-8130-f16323935ff3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6800","nonce":9088303},{"type":"v1","addr":"172.21.15.32:6801","nonce":9088303}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6802","nonce":9088303},{"type":"v1","addr":"172.21.15.32:6803","nonce":9088303}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6806","nonce":9088303},{"type":"v1","addr":"172.21.15.32:6807","nonce":9088303}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6804","nonce":9088303},{"type":"v1","addr":"172.21.15.32:6805","nonce":9088303}]},"public_addr":"172.21.15.32:6801/9088303","cluster_addr":"172.21.15.32:6803/9088303","heartbeat_back_addr":"172.21.15.32:6807/9088303","heartbeat_front_addr":"172.21.15.32:6805/9088303","state":["exists","up"]},{"osd":4,"uuid":"4f902eff-1e18-418d-82c9-2151c69524fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6802","nonce":864752275},{"type":"v1","addr":"172.21.15.118:6803","nonce":864752275}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6804","nonce":864752275},{"type":"v1","addr":"172.21.15.118:6805","nonce":864752275}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6808","nonce":864752275},{"type":"v1","addr":"172.21.15.118:6809","nonce":864752275}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6806","nonce":864752275},{"type":"v1","addr":"172.21.15.118:6807","nonce":864752275}]},"public_addr":"172.21.15.118:6803/864752275","cluster_addr":"172.21.15.118:6805/864752275","heartbeat_back_addr":"172.21.15.118:6809/864752275","heartbeat_front_addr":"172.21.15.118:6807/864752275","state":["exists","up"]},{"osd":5,"uuid":"bfd9feac-dd06-458c-91b9-af7f175f612c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6810","nonce":2724891770},{"type":"v1","addr":"172.21.15.118:6811","nonce":2724891770}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6812","nonce":2724891770},{"type":"v1","addr":"172.21.15.118:6813","nonce":2724891770}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6816","nonce":2724891770},{"type":"v1","addr":"172.21.15.118:6817","nonce":2724891770}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6814","nonce":2724891770},{"type":"v1","addr":"172.21.15.118:6815","nonce":2724891770}]},"public_addr":"172.21.15.118:6811/2724891770","cluster_addr":"172.21.15.118:6813/2724891770","heartbeat_back_addr":"172.21.15.118:6817/2724891770","heartbeat_front_addr":"172.21.15.118:6815/2724891770","state":["exists","up"]},{"osd":6,"uuid":"5ed5a279-9d8e-416d-bbc7-7edd3e6d5705","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6825","nonce":801947386},{"type":"v1","addr":"172.21.15.118:6827","nonce":801947386}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6828","nonce":801947386},{"type":"v1","addr":"172.21.15.118:6829","nonce":801947386}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6832","nonce":801947386},{"type":"v1","addr":"172.21.15.118:6833","nonce":801947386}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6830","nonce":801947386},{"type":"v1","addr":"172.21.15.118:6831","nonce":801947386}]},"public_addr":"172.21.15.118:6827/801947386","cluster_addr":"172.21.15.118:6829/801947386","heartbeat_back_addr":"172.21.15.118:6833/801947386","heartbeat_front_addr":"172.21.15.118:6831/801947386","state":["exists","up"]},{"osd":7,"uuid":"f2086c47-875e-4451-a436-2008d1ea16af","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6818","nonce":653000048},{"type":"v1","addr":"172.21.15.118:6819","nonce":653000048}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6820","nonce":653000048},{"type":"v1","addr":"172.21.15.118:6821","nonce":653000048}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6824","nonce":653000048},{"type":"v1","addr":"172.21.15.118:6826","nonce":653000048}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6822","nonce":653000048},{"type":"v1","addr":"172.21.15.118:6823","nonce":653000048}]},"public_addr":"172.21.15.118:6819/653000048","cluster_addr":"172.21.15.118:6821/653000048","heartbeat_back_addr":"172.21.15.118:6826/653000048","heartbeat_front_addr":"172.21.15.118:6823/653000048","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-03T20:47:25.491994+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-03T20:47:25.587358+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-03T20:47:25.454175+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-03T20:47:25.377734+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-03T20:47:25.412351+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-03T20:47:25.499815+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-03T20:47:25.575044+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-03T20:47:25.552383+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.32:6835/1186001746":"2024-08-04T20:48:38.076677+0000","172.21.15.32:6832/1186001746":"2024-08-04T20:48:38.076677+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-08-03T20:49:24.326 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:24.325+0000 7f6367e00640 1 -- 172.21.15.32:0/2363489801 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f6340077010 msgr2=0x7f6340079520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:49:24.327 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:24.325+0000 7f6367e00640 1 --2- 172.21.15.32:0/2363489801 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f6340077010 0x7f6340079520 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f636015c710 tx=0x7f634c009040 comp rx=0 tx=0).stop 2024-08-03T20:49:24.327 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:24.325+0000 7f6367e00640 1 -- 172.21.15.32:0/2363489801 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f636006f720 msgr2=0x7f6360148d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:49:24.327 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:24.325+0000 7f6367e00640 1 --2- 172.21.15.32:0/2363489801 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f636006f720 0x7f6360148d60 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f635400d750 tx=0x7f635400dc10 comp rx=0 tx=0).stop 2024-08-03T20:49:24.327 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:24.325+0000 7f6367e00640 1 -- 172.21.15.32:0/2363489801 shutdown_connections 2024-08-03T20:49:24.327 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:24.325+0000 7f6367e00640 1 --2- 172.21.15.32:0/2363489801 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f6340077010 0x7f6340079520 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:49:24.327 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:24.325+0000 7f6367e00640 1 --2- 172.21.15.32:0/2363489801 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f63600ade80 0x7f6360159610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:49:24.327 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:24.325+0000 7f6367e00640 1 --2- 172.21.15.32:0/2363489801 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f63600b39a0 0x7f63601479e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:49:24.327 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:24.325+0000 7f6367e00640 1 --2- 172.21.15.32:0/2363489801 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f636006f720 0x7f6360148d60 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:49:24.327 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:24.325+0000 7f6367e00640 1 -- 172.21.15.32:0/2363489801 >> 172.21.15.32:0/2363489801 conn(0x7f63600b1bd0 msgr2=0x7f63600b3db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:49:24.327 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:24.325+0000 7f6367e00640 1 -- 172.21.15.32:0/2363489801 shutdown_connections 2024-08-03T20:49:24.327 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:24.325+0000 7f6367e00640 1 -- 172.21.15.32:0/2363489801 wait complete. 2024-08-03T20:49:24.348 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-08-03T20:49:24.349 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-08-03T20:49:24.578 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:24.576+0000 7fdc41a00640 1 -- 172.21.15.32:0/2958773970 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdc3c0b38b0 msgr2=0x7fdc3c1523f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:49:24.578 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:24.576+0000 7fdc41a00640 1 --2- 172.21.15.32:0/2958773970 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdc3c0b38b0 0x7fdc3c1523f0 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7fdc2c005fd0 tx=0x7fdc2c02a540 comp rx=0 tx=0).stop 2024-08-03T20:49:24.578 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:24.576+0000 7fdc41a00640 1 -- 172.21.15.32:0/2958773970 shutdown_connections 2024-08-03T20:49:24.578 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:24.576+0000 7fdc41a00640 1 --2- 172.21.15.32:0/2958773970 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdc3c0b38b0 0x7fdc3c1523f0 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:49:24.578 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:24.576+0000 7fdc41a00640 1 -- 172.21.15.32:0/2958773970 >> 172.21.15.32:0/2958773970 conn(0x7fdc3c0b1bd0 msgr2=0x7fdc3c0b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:49:24.578 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:24.576+0000 7fdc41a00640 1 -- 172.21.15.32:0/2958773970 shutdown_connections 2024-08-03T20:49:24.578 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:24.576+0000 7fdc41a00640 1 -- 172.21.15.32:0/2958773970 wait complete. 2024-08-03T20:49:24.580 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:24.578+0000 7fdc41a00640 1 Processor -- start 2024-08-03T20:49:24.580 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:24.579+0000 7fdc41a00640 1 -- start start 2024-08-03T20:49:24.582 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:24.580+0000 7fdc41a00640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fdc3c0b38b0 0x7fdc3c1f2630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:49:24.583 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:24.580+0000 7fdc41a00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fdc3c0b4ce0 0x7fdc3c15db00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:49:24.583 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:24.580+0000 7fdc41a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdc3c15e040 0x7fdc3c158750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:49:24.583 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:24.580+0000 7fdc41a00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fdc3c0cec60 con 0x7fdc3c15e040 2024-08-03T20:49:24.583 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:24.580+0000 7fdc41a00640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7fdc3c0ceae0 con 0x7fdc3c0b4ce0 2024-08-03T20:49:24.583 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:24.580+0000 7fdc41a00640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7fdc3c0cede0 con 0x7fdc3c0b38b0 2024-08-03T20:49:24.583 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:24.580+0000 7fdc3b400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdc3c15e040 0x7fdc3c158750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:49:24.583 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:24.581+0000 7fdc3b400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdc3c15e040 0x7fdc3c158750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:60050/0 (socket says 172.21.15.32:60050) 2024-08-03T20:49:24.583 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:24.581+0000 7fdc3b400640 1 -- 172.21.15.32:0/1709964098 learned_addr learned my addr 172.21.15.32:0/1709964098 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:49:24.583 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:24.581+0000 7fdc33e00640 1 --2- 172.21.15.32:0/1709964098 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fdc3c0b4ce0 0x7fdc3c15db00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:49:24.583 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:24.581+0000 7fdc3aa00640 1 --2- 172.21.15.32:0/1709964098 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fdc3c0b38b0 0x7fdc3c1f2630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:49:24.583 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:24.581+0000 7fdc3b400640 1 -- 172.21.15.32:0/1709964098 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fdc3c0b38b0 msgr2=0x7fdc3c1f2630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:49:24.583 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:24.581+0000 7fdc3b400640 1 --2- 172.21.15.32:0/1709964098 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fdc3c0b38b0 0x7fdc3c1f2630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:49:24.583 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:24.581+0000 7fdc3b400640 1 -- 172.21.15.32:0/1709964098 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fdc3c0b4ce0 msgr2=0x7fdc3c15db00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:49:24.584 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:24.581+0000 7fdc3b400640 1 --2- 172.21.15.32:0/1709964098 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fdc3c0b4ce0 0x7fdc3c15db00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:49:24.584 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:24.581+0000 7fdc3b400640 1 -- 172.21.15.32:0/1709964098 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdc3c15e450 con 0x7fdc3c15e040 2024-08-03T20:49:24.584 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:24.581+0000 7fdc3b400640 1 --2- 172.21.15.32:0/1709964098 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdc3c15e040 0x7fdc3c158750 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7fdc3400c6e0 tx=0x7fdc3400cba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:49:24.585 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:24.582+0000 7fdc32a00640 1 -- 172.21.15.32:0/1709964098 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc34003b70 con 0x7fdc3c15e040 2024-08-03T20:49:24.585 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:24.582+0000 7fdc32a00640 1 -- 172.21.15.32:0/1709964098 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fdc34003d30 con 0x7fdc3c15e040 2024-08-03T20:49:24.585 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:24.582+0000 7fdc32a00640 1 -- 172.21.15.32:0/1709964098 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc34005330 con 0x7fdc3c15e040 2024-08-03T20:49:24.585 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:24.582+0000 7fdc41a00640 1 -- 172.21.15.32:0/1709964098 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdc3c156fe0 con 0x7fdc3c15e040 2024-08-03T20:49:24.585 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:24.582+0000 7fdc41a00640 1 -- 172.21.15.32:0/1709964098 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdc3c1f8b00 con 0x7fdc3c15e040 2024-08-03T20:49:24.586 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:24.583+0000 7fdc41a00640 1 -- 172.21.15.32:0/1709964098 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdc3c1571b0 con 0x7fdc3c15e040 2024-08-03T20:49:24.586 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:24.584+0000 7fdc32a00640 1 -- 172.21.15.32:0/1709964098 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7fdc340054d0 con 0x7fdc3c15e040 2024-08-03T20:49:24.587 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:24.585+0000 7fdc32a00640 1 --2- 172.21.15.32:0/1709964098 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fdc18077010 0x7fdc18079520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:49:24.587 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:24.586+0000 7fdc32a00640 1 -- 172.21.15.32:0/1709964098 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(33..33 src has 1..33) ==== 5795+0+0 (secure 0 0 0) 0x7fdc340987c0 con 0x7fdc3c15e040 2024-08-03T20:49:24.587 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:24.586+0000 7fdc3aa00640 1 --2- 172.21.15.32:0/1709964098 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fdc18077010 0x7fdc18079520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:49:24.588 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:24.587+0000 7fdc3aa00640 1 --2- 172.21.15.32:0/1709964098 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fdc18077010 0x7fdc18079520 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fdc2c033000 tx=0x7fdc2c004730 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:49:24.591 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:24.590+0000 7fdc32a00640 1 -- 172.21.15.32:0/1709964098 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7fdc3c15e450 con 0x7fdc3c15e040 2024-08-03T20:49:24.834 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:24.833+0000 7fdc41a00640 1 -- 172.21.15.32:0/1709964098 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fdc3c1523f0 con 0x7fdc3c15e040 2024-08-03T20:49:24.835 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:24.833+0000 7fdc32a00640 1 -- 172.21.15.32:0/1709964098 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+5227 (secure 0 0 0) 0x7fdc34061270 con 0x7fdc3c15e040 2024-08-03T20:49:24.835 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-03T20:49:24.835 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":25,"btime":"2024-08-03T20:48:57:346845+0000","default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4643,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6836/4002526832","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6834","nonce":4002526832},{"type":"v1","addr":"172.21.15.118:6836","nonce":4002526832}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":4646,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6837/3036256187","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6835","nonce":3036256187},{"type":"v1","addr":"172.21.15.118:6837","nonce":3036256187}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":4652,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6834/2104418334","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6833","nonce":2104418334},{"type":"v1","addr":"172.21.15.32:6834","nonce":2104418334}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-08-03T20:48:26.652521+0000","modified":"2024-08-03T20:48:57.101761+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5070},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5070":{"gid":5070,"name":"a","rank":0,"incarnation":20,"state":"up:active","state_seq":8,"addr":"172.21.15.32:6835/211300533","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":211300533},{"type":"v1","addr":"172.21.15.32:6835","nonce":211300533}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5070,"qdb_cluster":[5070]},"id":2}]} 2024-08-03T20:49:24.835 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 25 2024-08-03T20:49:24.837 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:24.836+0000 7fdc41a00640 1 -- 172.21.15.32:0/1709964098 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fdc18077010 msgr2=0x7fdc18079520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:49:24.837 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:24.836+0000 7fdc41a00640 1 --2- 172.21.15.32:0/1709964098 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fdc18077010 0x7fdc18079520 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fdc2c033000 tx=0x7fdc2c004730 comp rx=0 tx=0).stop 2024-08-03T20:49:24.838 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:24.836+0000 7fdc41a00640 1 -- 172.21.15.32:0/1709964098 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdc3c15e040 msgr2=0x7fdc3c158750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:49:24.838 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:24.836+0000 7fdc41a00640 1 --2- 172.21.15.32:0/1709964098 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdc3c15e040 0x7fdc3c158750 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7fdc3400c6e0 tx=0x7fdc3400cba0 comp rx=0 tx=0).stop 2024-08-03T20:49:24.838 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:24.836+0000 7fdc41a00640 1 -- 172.21.15.32:0/1709964098 shutdown_connections 2024-08-03T20:49:24.838 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:24.836+0000 7fdc41a00640 1 --2- 172.21.15.32:0/1709964098 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fdc18077010 0x7fdc18079520 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:49:24.838 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:24.836+0000 7fdc41a00640 1 --2- 172.21.15.32:0/1709964098 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fdc3c0b4ce0 0x7fdc3c15db00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:49:24.838 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:24.836+0000 7fdc41a00640 1 --2- 172.21.15.32:0/1709964098 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdc3c15e040 0x7fdc3c158750 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:49:24.838 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:24.836+0000 7fdc41a00640 1 --2- 172.21.15.32:0/1709964098 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fdc3c0b38b0 0x7fdc3c1f2630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:49:24.838 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:24.836+0000 7fdc41a00640 1 -- 172.21.15.32:0/1709964098 >> 172.21.15.32:0/1709964098 conn(0x7fdc3c0b1bd0 msgr2=0x7fdc3c068940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:49:24.838 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:24.836+0000 7fdc41a00640 1 -- 172.21.15.32:0/1709964098 shutdown_connections 2024-08-03T20:49:24.838 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:24.836+0000 7fdc41a00640 1 -- 172.21.15.32:0/1709964098 wait complete. 2024-08-03T20:49:24.859 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-08-03T20:49:25.088 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:25.087+0000 7f4179a00640 1 -- 172.21.15.32:0/1756364265 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f41740ca3b0 msgr2=0x7f41740ca7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:49:25.089 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:25.087+0000 7f4179a00640 1 --2- 172.21.15.32:0/1756364265 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f41740ca3b0 0x7f41740ca7c0 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f415c009830 tx=0x7f415c02a510 comp rx=0 tx=0).stop 2024-08-03T20:49:25.089 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:25.087+0000 7f4179a00640 1 -- 172.21.15.32:0/1756364265 shutdown_connections 2024-08-03T20:49:25.089 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:25.087+0000 7f4179a00640 1 --2- 172.21.15.32:0/1756364265 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f41740ce150 0x7f41740ce560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:49:25.089 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:25.087+0000 7f4179a00640 1 --2- 172.21.15.32:0/1756364265 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f41740ca3b0 0x7f41740ca7c0 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:49:25.089 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:25.087+0000 7f4179a00640 1 -- 172.21.15.32:0/1756364265 >> 172.21.15.32:0/1756364265 conn(0x7f41740b2120 msgr2=0x7f41740cfd60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:49:25.089 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:25.087+0000 7f4179a00640 1 -- 172.21.15.32:0/1756364265 shutdown_connections 2024-08-03T20:49:25.089 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:25.087+0000 7f4179a00640 1 -- 172.21.15.32:0/1756364265 wait complete. 2024-08-03T20:49:25.090 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:25.089+0000 7f4179a00640 1 Processor -- start 2024-08-03T20:49:25.090 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:25.089+0000 7f4179a00640 1 -- start start 2024-08-03T20:49:25.092 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:25.091+0000 7f4179a00640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f41740ca3b0 0x7f417414fd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:49:25.093 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:25.091+0000 7f4179a00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f41740ce150 0x7f41740c8f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:49:25.093 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:25.091+0000 7f4179a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f417415a310 0x7f41740c9460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:49:25.093 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:25.091+0000 7f4179a00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f4174070860 con 0x7f417415a310 2024-08-03T20:49:25.093 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:25.091+0000 7f4179a00640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f41740706e0 con 0x7f41740ce150 2024-08-03T20:49:25.093 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:25.091+0000 7f4179a00640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7f41740ceaa0 con 0x7f41740ca3b0 2024-08-03T20:49:25.093 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:25.091+0000 7f4179000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f417415a310 0x7f41740c9460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:49:25.093 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:25.091+0000 7f4179000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f417415a310 0x7f41740c9460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:60068/0 (socket says 172.21.15.32:60068) 2024-08-03T20:49:25.093 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:25.091+0000 7f4179000640 1 -- 172.21.15.32:0/2320037740 learned_addr learned my addr 172.21.15.32:0/2320037740 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:49:25.093 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:25.091+0000 7f4173e00640 1 --2- 172.21.15.32:0/2320037740 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f41740ca3b0 0x7f417414fd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:49:25.093 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:25.091+0000 7f416be00640 1 --2- 172.21.15.32:0/2320037740 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f41740ce150 0x7f41740c8f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:49:25.094 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:25.092+0000 7f4179000640 1 -- 172.21.15.32:0/2320037740 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f41740ca3b0 msgr2=0x7f417414fd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:49:25.094 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:25.092+0000 7f4179000640 1 --2- 172.21.15.32:0/2320037740 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f41740ca3b0 0x7f417414fd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:49:25.094 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:25.092+0000 7f4179000640 1 -- 172.21.15.32:0/2320037740 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f41740ce150 msgr2=0x7f41740c8f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:49:25.094 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:25.092+0000 7f4179000640 1 --2- 172.21.15.32:0/2320037740 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f41740ce150 0x7f41740c8f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:49:25.094 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:25.092+0000 7f4179000640 1 -- 172.21.15.32:0/2320037740 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f41740c64b0 con 0x7f417415a310 2024-08-03T20:49:25.095 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:25.092+0000 7f4179000640 1 --2- 172.21.15.32:0/2320037740 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f417415a310 0x7f41740c9460 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f416c00cad0 tx=0x7f416c00cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:49:25.095 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:25.092+0000 7f4170c00640 1 -- 172.21.15.32:0/2320037740 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f416c00ae20 con 0x7f417415a310 2024-08-03T20:49:25.095 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:25.092+0000 7f4170c00640 1 -- 172.21.15.32:0/2320037740 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f416c00f540 con 0x7f417415a310 2024-08-03T20:49:25.095 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:25.092+0000 7f4170c00640 1 -- 172.21.15.32:0/2320037740 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f416c00f990 con 0x7f417415a310 2024-08-03T20:49:25.095 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:25.092+0000 7f4179a00640 1 -- 172.21.15.32:0/2320037740 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f41740c4440 con 0x7f417415a310 2024-08-03T20:49:25.095 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:25.092+0000 7f4179a00640 1 -- 172.21.15.32:0/2320037740 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f41740c4a00 con 0x7f417415a310 2024-08-03T20:49:25.095 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:25.093+0000 7f4179a00640 1 -- 172.21.15.32:0/2320037740 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f41740ca7c0 con 0x7f417415a310 2024-08-03T20:49:25.096 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:25.095+0000 7f4170c00640 1 -- 172.21.15.32:0/2320037740 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f416c0213f0 con 0x7f417415a310 2024-08-03T20:49:25.097 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:25.096+0000 7f4170c00640 1 --2- 172.21.15.32:0/2320037740 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f4150077090 0x7f41500795a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:49:25.098 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:25.096+0000 7f4170c00640 1 -- 172.21.15.32:0/2320037740 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(33..33 src has 1..33) ==== 5795+0+0 (secure 0 0 0) 0x7f416c099230 con 0x7f417415a310 2024-08-03T20:49:25.098 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:25.097+0000 7f4173e00640 1 --2- 172.21.15.32:0/2320037740 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f4150077090 0x7f41500795a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:49:25.099 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:25.098+0000 7f4173e00640 1 --2- 172.21.15.32:0/2320037740 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f4150077090 0x7f41500795a0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f4164000f60 tx=0x7f4164009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:49:25.102 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:25.101+0000 7f4170c00640 1 -- 172.21.15.32:0/2320037740 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7f416c014030 con 0x7f417415a310 2024-08-03T20:49:25.345 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:25.343+0000 7f4179a00640 1 -- 172.21.15.32:0/2320037740 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f41740b0680 con 0x7f417415a310 2024-08-03T20:49:25.345 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:25.344+0000 7f4170c00640 1 -- 172.21.15.32:0/2320037740 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+5227 (secure 0 0 0) 0x7f416c061df0 con 0x7f417415a310 2024-08-03T20:49:25.345 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-03T20:49:25.346 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":25,"btime":"2024-08-03T20:48:57:346845+0000","default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4643,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6836/4002526832","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6834","nonce":4002526832},{"type":"v1","addr":"172.21.15.118:6836","nonce":4002526832}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":4646,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6837/3036256187","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6835","nonce":3036256187},{"type":"v1","addr":"172.21.15.118:6837","nonce":3036256187}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":4652,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6834/2104418334","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6833","nonce":2104418334},{"type":"v1","addr":"172.21.15.32:6834","nonce":2104418334}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-08-03T20:48:26.652521+0000","modified":"2024-08-03T20:48:57.101761+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5070},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5070":{"gid":5070,"name":"a","rank":0,"incarnation":20,"state":"up:active","state_seq":8,"addr":"172.21.15.32:6835/211300533","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":211300533},{"type":"v1","addr":"172.21.15.32:6835","nonce":211300533}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5070,"qdb_cluster":[5070]},"id":2}]} 2024-08-03T20:49:25.346 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 25 2024-08-03T20:49:25.348 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:25.346+0000 7f4179a00640 1 -- 172.21.15.32:0/2320037740 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f4150077090 msgr2=0x7f41500795a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:49:25.348 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:25.346+0000 7f4179a00640 1 --2- 172.21.15.32:0/2320037740 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f4150077090 0x7f41500795a0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f4164000f60 tx=0x7f4164009040 comp rx=0 tx=0).stop 2024-08-03T20:49:25.348 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:25.346+0000 7f4179a00640 1 -- 172.21.15.32:0/2320037740 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f417415a310 msgr2=0x7f41740c9460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:49:25.348 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:25.346+0000 7f4179a00640 1 --2- 172.21.15.32:0/2320037740 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f417415a310 0x7f41740c9460 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f416c00cad0 tx=0x7f416c00cf90 comp rx=0 tx=0).stop 2024-08-03T20:49:25.348 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:25.346+0000 7f4179a00640 1 -- 172.21.15.32:0/2320037740 shutdown_connections 2024-08-03T20:49:25.348 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:25.346+0000 7f4179a00640 1 --2- 172.21.15.32:0/2320037740 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f4150077090 0x7f41500795a0 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:49:25.348 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:25.346+0000 7f4179a00640 1 --2- 172.21.15.32:0/2320037740 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f41740ce150 0x7f41740c8f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:49:25.348 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:25.346+0000 7f4179a00640 1 --2- 172.21.15.32:0/2320037740 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f417415a310 0x7f41740c9460 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:49:25.348 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:25.346+0000 7f4179a00640 1 --2- 172.21.15.32:0/2320037740 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f41740ca3b0 0x7f417414fd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:49:25.348 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:25.346+0000 7f4179a00640 1 -- 172.21.15.32:0/2320037740 >> 172.21.15.32:0/2320037740 conn(0x7f41740b2120 msgr2=0x7f41740b0ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:49:25.349 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:25.346+0000 7f4179a00640 1 -- 172.21.15.32:0/2320037740 shutdown_connections 2024-08-03T20:49:25.349 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:25.346+0000 7f4179a00640 1 -- 172.21.15.32:0/2320037740 wait complete. 2024-08-03T20:49:25.373 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-08-03T20:49:25.605 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:25.603+0000 7ff420200640 1 -- 172.21.15.32:0/3324003362 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff400005350 msgr2=0x7ff40000b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:49:25.605 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:25.603+0000 7ff420200640 1 --2- 172.21.15.32:0/3324003362 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff400005350 0x7ff40000b9c0 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7ff410005fd0 tx=0x7ff400005830 comp rx=0 tx=0).stop 2024-08-03T20:49:25.605 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:25.603+0000 7ff420200640 1 -- 172.21.15.32:0/3324003362 shutdown_connections 2024-08-03T20:49:25.605 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:25.603+0000 7ff420200640 1 --2- 172.21.15.32:0/3324003362 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff400005350 0x7ff40000b9c0 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:49:25.605 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:25.603+0000 7ff420200640 1 -- 172.21.15.32:0/3324003362 >> 172.21.15.32:0/3324003362 conn(0x7ff4180b1bd0 msgr2=0x7ff4180b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:49:25.605 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:25.603+0000 7ff420200640 1 -- 172.21.15.32:0/3324003362 shutdown_connections 2024-08-03T20:49:25.605 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:25.603+0000 7ff420200640 1 -- 172.21.15.32:0/3324003362 wait complete. 2024-08-03T20:49:25.607 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:25.605+0000 7ff420200640 1 Processor -- start 2024-08-03T20:49:25.607 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:25.606+0000 7ff420200640 1 -- start start 2024-08-03T20:49:25.609 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:25.607+0000 7ff420200640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ff400005350 0x7ff4181442c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:49:25.609 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:25.607+0000 7ff420200640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7ff4181f5ae0 0x7ff41815b940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:49:25.609 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:25.607+0000 7ff420200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff4181434f0 0x7ff418143900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:49:25.609 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:25.607+0000 7ff420200640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7ff4180cf9c0 con 0x7ff4181434f0 2024-08-03T20:49:25.610 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:25.607+0000 7ff420200640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7ff4180cf840 con 0x7ff400005350 2024-08-03T20:49:25.610 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:25.607+0000 7ff420200640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7ff418073e60 con 0x7ff4181f5ae0 2024-08-03T20:49:25.610 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:25.607+0000 7ff41e200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff4181434f0 0x7ff418143900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:49:25.610 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:25.607+0000 7ff41e200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff4181434f0 0x7ff418143900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:60078/0 (socket says 172.21.15.32:60078) 2024-08-03T20:49:25.610 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:25.607+0000 7ff41e200640 1 -- 172.21.15.32:0/2120218845 learned_addr learned my addr 172.21.15.32:0/2120218845 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:49:25.610 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:25.607+0000 7ff41d800640 1 --2- 172.21.15.32:0/2120218845 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ff400005350 0x7ff4181442c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:49:25.610 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:25.607+0000 7ff41ce00640 1 --2- 172.21.15.32:0/2120218845 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7ff4181f5ae0 0x7ff41815b940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:49:25.610 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:25.608+0000 7ff41e200640 1 -- 172.21.15.32:0/2120218845 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7ff4181f5ae0 msgr2=0x7ff41815b940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:49:25.610 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:25.608+0000 7ff41e200640 1 --2- 172.21.15.32:0/2120218845 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7ff4181f5ae0 0x7ff41815b940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:49:25.610 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:25.608+0000 7ff41e200640 1 -- 172.21.15.32:0/2120218845 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ff400005350 msgr2=0x7ff4181442c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:49:25.610 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:25.608+0000 7ff41e200640 1 --2- 172.21.15.32:0/2120218845 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ff400005350 0x7ff4181442c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:49:25.610 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:25.608+0000 7ff41e200640 1 -- 172.21.15.32:0/2120218845 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff418143e40 con 0x7ff4181434f0 2024-08-03T20:49:25.610 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:25.608+0000 7ff41e200640 1 --2- 172.21.15.32:0/2120218845 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff4181434f0 0x7ff418143900 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7ff41002aec0 tx=0x7ff4100042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:49:25.611 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:25.608+0000 7ff415600640 1 -- 172.21.15.32:0/2120218845 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff410009200 con 0x7ff4181434f0 2024-08-03T20:49:25.611 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:25.608+0000 7ff415600640 1 -- 172.21.15.32:0/2120218845 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7ff41002bc30 con 0x7ff4181434f0 2024-08-03T20:49:25.611 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:25.608+0000 7ff420200640 1 -- 172.21.15.32:0/2120218845 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff418163790 con 0x7ff4181434f0 2024-08-03T20:49:25.612 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:25.608+0000 7ff415600640 1 -- 172.21.15.32:0/2120218845 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff410009ad0 con 0x7ff4181434f0 2024-08-03T20:49:25.612 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:25.608+0000 7ff420200640 1 -- 172.21.15.32:0/2120218845 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff418163d50 con 0x7ff4181434f0 2024-08-03T20:49:25.612 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:25.609+0000 7ff420200640 1 -- 172.21.15.32:0/2120218845 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff4181448f0 con 0x7ff4181434f0 2024-08-03T20:49:25.612 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:25.611+0000 7ff415600640 1 -- 172.21.15.32:0/2120218845 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7ff410033440 con 0x7ff4181434f0 2024-08-03T20:49:25.614 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:25.612+0000 7ff415600640 1 --2- 172.21.15.32:0/2120218845 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7ff3f8077040 0x7ff3f8079550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:49:25.614 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:25.612+0000 7ff415600640 1 -- 172.21.15.32:0/2120218845 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(33..33 src has 1..33) ==== 5795+0+0 (secure 0 0 0) 0x7ff4100b9700 con 0x7ff4181434f0 2024-08-03T20:49:25.614 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:25.612+0000 7ff41d800640 1 --2- 172.21.15.32:0/2120218845 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7ff3f8077040 0x7ff3f8079550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:49:25.614 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:25.613+0000 7ff41d800640 1 --2- 172.21.15.32:0/2120218845 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7ff3f8077040 0x7ff3f8079550 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7ff40c006bf0 tx=0x7ff40c0067a0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:49:25.617 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:25.615+0000 7ff415600640 1 -- 172.21.15.32:0/2120218845 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7ff418143e40 con 0x7ff4181434f0 2024-08-03T20:49:25.821 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:25.819+0000 7ff420200640 1 -- 172.21.15.32:0/2120218845 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7ff4180743c0 con 0x7ff4181434f0 2024-08-03T20:49:25.822 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:25.821+0000 7ff415600640 1 -- 172.21.15.32:0/2120218845 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v33) ==== 74+0+16738 (secure 0 0 0) 0x7ff410082330 con 0x7ff4181434f0 2024-08-03T20:49:25.822 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-03T20:49:25.823 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":33,"fsid":"40ab141a-01d7-49b9-ae3c-2edec9876e95","created":"2024-08-03T20:47:14.957362+0000","modified":"2024-08-03T20:48:38.076713+0000","last_up_change":"2024-08-03T20:47:28.874451+0000","last_in_change":"2024-08-03T20:47:20.079442+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-08-03T20:47:28.261300+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_metadata","create_time":"2024-08-03T20:48:24.969840+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"31","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data","create_time":"2024-08-03T20:48:25.603595+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"32","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5a83e759-9035-438b-889a-0a6af1f0a0d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6816","nonce":3066156116},{"type":"v1","addr":"172.21.15.32:6817","nonce":3066156116}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6818","nonce":3066156116},{"type":"v1","addr":"172.21.15.32:6819","nonce":3066156116}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6822","nonce":3066156116},{"type":"v1","addr":"172.21.15.32:6823","nonce":3066156116}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6820","nonce":3066156116},{"type":"v1","addr":"172.21.15.32:6821","nonce":3066156116}]},"public_addr":"172.21.15.32:6817/3066156116","cluster_addr":"172.21.15.32:6819/3066156116","heartbeat_back_addr":"172.21.15.32:6823/3066156116","heartbeat_front_addr":"172.21.15.32:6821/3066156116","state":["exists","up"]},{"osd":1,"uuid":"d7c5c210-da4f-49f9-b9e7-1dfb02daa4bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6824","nonce":1551241759},{"type":"v1","addr":"172.21.15.32:6825","nonce":1551241759}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6826","nonce":1551241759},{"type":"v1","addr":"172.21.15.32:6827","nonce":1551241759}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6830","nonce":1551241759},{"type":"v1","addr":"172.21.15.32:6831","nonce":1551241759}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6828","nonce":1551241759},{"type":"v1","addr":"172.21.15.32:6829","nonce":1551241759}]},"public_addr":"172.21.15.32:6825/1551241759","cluster_addr":"172.21.15.32:6827/1551241759","heartbeat_back_addr":"172.21.15.32:6831/1551241759","heartbeat_front_addr":"172.21.15.32:6829/1551241759","state":["exists","up"]},{"osd":2,"uuid":"57f2e6de-a201-45c2-8981-8e56a4875ebd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6808","nonce":3806708944},{"type":"v1","addr":"172.21.15.32:6809","nonce":3806708944}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6810","nonce":3806708944},{"type":"v1","addr":"172.21.15.32:6811","nonce":3806708944}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6814","nonce":3806708944},{"type":"v1","addr":"172.21.15.32:6815","nonce":3806708944}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6812","nonce":3806708944},{"type":"v1","addr":"172.21.15.32:6813","nonce":3806708944}]},"public_addr":"172.21.15.32:6809/3806708944","cluster_addr":"172.21.15.32:6811/3806708944","heartbeat_back_addr":"172.21.15.32:6815/3806708944","heartbeat_front_addr":"172.21.15.32:6813/3806708944","state":["exists","up"]},{"osd":3,"uuid":"6439a280-da45-4148-8130-f16323935ff3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6800","nonce":9088303},{"type":"v1","addr":"172.21.15.32:6801","nonce":9088303}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6802","nonce":9088303},{"type":"v1","addr":"172.21.15.32:6803","nonce":9088303}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6806","nonce":9088303},{"type":"v1","addr":"172.21.15.32:6807","nonce":9088303}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6804","nonce":9088303},{"type":"v1","addr":"172.21.15.32:6805","nonce":9088303}]},"public_addr":"172.21.15.32:6801/9088303","cluster_addr":"172.21.15.32:6803/9088303","heartbeat_back_addr":"172.21.15.32:6807/9088303","heartbeat_front_addr":"172.21.15.32:6805/9088303","state":["exists","up"]},{"osd":4,"uuid":"4f902eff-1e18-418d-82c9-2151c69524fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6802","nonce":864752275},{"type":"v1","addr":"172.21.15.118:6803","nonce":864752275}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6804","nonce":864752275},{"type":"v1","addr":"172.21.15.118:6805","nonce":864752275}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6808","nonce":864752275},{"type":"v1","addr":"172.21.15.118:6809","nonce":864752275}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6806","nonce":864752275},{"type":"v1","addr":"172.21.15.118:6807","nonce":864752275}]},"public_addr":"172.21.15.118:6803/864752275","cluster_addr":"172.21.15.118:6805/864752275","heartbeat_back_addr":"172.21.15.118:6809/864752275","heartbeat_front_addr":"172.21.15.118:6807/864752275","state":["exists","up"]},{"osd":5,"uuid":"bfd9feac-dd06-458c-91b9-af7f175f612c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6810","nonce":2724891770},{"type":"v1","addr":"172.21.15.118:6811","nonce":2724891770}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6812","nonce":2724891770},{"type":"v1","addr":"172.21.15.118:6813","nonce":2724891770}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6816","nonce":2724891770},{"type":"v1","addr":"172.21.15.118:6817","nonce":2724891770}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6814","nonce":2724891770},{"type":"v1","addr":"172.21.15.118:6815","nonce":2724891770}]},"public_addr":"172.21.15.118:6811/2724891770","cluster_addr":"172.21.15.118:6813/2724891770","heartbeat_back_addr":"172.21.15.118:6817/2724891770","heartbeat_front_addr":"172.21.15.118:6815/2724891770","state":["exists","up"]},{"osd":6,"uuid":"5ed5a279-9d8e-416d-bbc7-7edd3e6d5705","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6825","nonce":801947386},{"type":"v1","addr":"172.21.15.118:6827","nonce":801947386}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6828","nonce":801947386},{"type":"v1","addr":"172.21.15.118:6829","nonce":801947386}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6832","nonce":801947386},{"type":"v1","addr":"172.21.15.118:6833","nonce":801947386}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6830","nonce":801947386},{"type":"v1","addr":"172.21.15.118:6831","nonce":801947386}]},"public_addr":"172.21.15.118:6827/801947386","cluster_addr":"172.21.15.118:6829/801947386","heartbeat_back_addr":"172.21.15.118:6833/801947386","heartbeat_front_addr":"172.21.15.118:6831/801947386","state":["exists","up"]},{"osd":7,"uuid":"f2086c47-875e-4451-a436-2008d1ea16af","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6818","nonce":653000048},{"type":"v1","addr":"172.21.15.118:6819","nonce":653000048}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6820","nonce":653000048},{"type":"v1","addr":"172.21.15.118:6821","nonce":653000048}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6824","nonce":653000048},{"type":"v1","addr":"172.21.15.118:6826","nonce":653000048}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6822","nonce":653000048},{"type":"v1","addr":"172.21.15.118:6823","nonce":653000048}]},"public_addr":"172.21.15.118:6819/653000048","cluster_addr":"172.21.15.118:6821/653000048","heartbeat_back_addr":"172.21.15.118:6826/653000048","heartbeat_front_addr":"172.21.15.118:6823/653000048","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-03T20:47:25.491994+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-03T20:47:25.587358+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-03T20:47:25.454175+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-03T20:47:25.377734+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-03T20:47:25.412351+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-03T20:47:25.499815+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-03T20:47:25.575044+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-03T20:47:25.552383+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.32:6835/1186001746":"2024-08-04T20:48:38.076677+0000","172.21.15.32:6832/1186001746":"2024-08-04T20:48:38.076677+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-08-03T20:49:25.825 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:25.823+0000 7ff420200640 1 -- 172.21.15.32:0/2120218845 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7ff3f8077040 msgr2=0x7ff3f8079550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:49:25.825 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:25.823+0000 7ff420200640 1 --2- 172.21.15.32:0/2120218845 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7ff3f8077040 0x7ff3f8079550 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7ff40c006bf0 tx=0x7ff40c0067a0 comp rx=0 tx=0).stop 2024-08-03T20:49:25.825 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:25.823+0000 7ff420200640 1 -- 172.21.15.32:0/2120218845 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff4181434f0 msgr2=0x7ff418143900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:49:25.825 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:25.823+0000 7ff420200640 1 --2- 172.21.15.32:0/2120218845 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff4181434f0 0x7ff418143900 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7ff41002aec0 tx=0x7ff4100042f0 comp rx=0 tx=0).stop 2024-08-03T20:49:25.825 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:25.823+0000 7ff420200640 1 -- 172.21.15.32:0/2120218845 shutdown_connections 2024-08-03T20:49:25.825 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:25.823+0000 7ff420200640 1 --2- 172.21.15.32:0/2120218845 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7ff3f8077040 0x7ff3f8079550 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:49:25.825 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:25.823+0000 7ff420200640 1 --2- 172.21.15.32:0/2120218845 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff4181434f0 0x7ff418143900 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:49:25.825 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:25.823+0000 7ff420200640 1 --2- 172.21.15.32:0/2120218845 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7ff4181f5ae0 0x7ff41815b940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:49:25.825 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:25.823+0000 7ff420200640 1 --2- 172.21.15.32:0/2120218845 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ff400005350 0x7ff4181442c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:49:25.825 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:25.823+0000 7ff420200640 1 -- 172.21.15.32:0/2120218845 >> 172.21.15.32:0/2120218845 conn(0x7ff4180b1bd0 msgr2=0x7ff41806ef30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:49:25.826 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:25.823+0000 7ff420200640 1 -- 172.21.15.32:0/2120218845 shutdown_connections 2024-08-03T20:49:25.826 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:25.823+0000 7ff420200640 1 -- 172.21.15.32:0/2120218845 wait complete. 2024-08-03T20:49:25.849 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-08-03T20:49:26.077 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:26.075+0000 7f548a200640 1 -- 172.21.15.32:0/1336434120 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5484144f90 msgr2=0x7f5484074570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:49:26.077 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:26.075+0000 7f548a200640 1 --2- 172.21.15.32:0/1336434120 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5484144f90 0x7f5484074570 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f5484073b50 tx=0x7f546c02a5c0 comp rx=0 tx=0).stop 2024-08-03T20:49:26.077 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:26.076+0000 7f548a200640 1 -- 172.21.15.32:0/1336434120 shutdown_connections 2024-08-03T20:49:26.078 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:26.076+0000 7f548a200640 1 --2- 172.21.15.32:0/1336434120 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5484144f90 0x7f5484074570 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:49:26.078 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:26.076+0000 7f548a200640 1 -- 172.21.15.32:0/1336434120 >> 172.21.15.32:0/1336434120 conn(0x7f54840b1bd0 msgr2=0x7f54840b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:49:26.078 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:26.076+0000 7f548a200640 1 -- 172.21.15.32:0/1336434120 shutdown_connections 2024-08-03T20:49:26.078 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:26.076+0000 7f548a200640 1 -- 172.21.15.32:0/1336434120 wait complete. 2024-08-03T20:49:26.079 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:26.078+0000 7f548a200640 1 Processor -- start 2024-08-03T20:49:26.079 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:26.078+0000 7f548a200640 1 -- start start 2024-08-03T20:49:26.081 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:26.079+0000 7f548a200640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f548406e930 0x7f5484141ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:49:26.082 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:26.079+0000 7f548a200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5484144f90 0x7f548415ebf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:49:26.082 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:26.079+0000 7f548a200640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f548415fff0 0x7f548415f130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:49:26.082 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:26.079+0000 7f548a200640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f54840cf840 con 0x7f5484144f90 2024-08-03T20:49:26.082 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:26.079+0000 7f548a200640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f5484145f30 con 0x7f548415fff0 2024-08-03T20:49:26.082 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:26.079+0000 7f548a200640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7f54840cf9c0 con 0x7f548406e930 2024-08-03T20:49:26.082 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:26.080+0000 7f547be00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5484144f90 0x7f548415ebf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:49:26.082 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:26.080+0000 7f547be00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5484144f90 0x7f548415ebf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:60098/0 (socket says 172.21.15.32:60098) 2024-08-03T20:49:26.082 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:26.080+0000 7f547be00640 1 -- 172.21.15.32:0/2746593969 learned_addr learned my addr 172.21.15.32:0/2746593969 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:49:26.082 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:26.080+0000 7f5483e00640 1 --2- 172.21.15.32:0/2746593969 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f548415fff0 0x7f548415f130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:49:26.082 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:26.080+0000 7f5483400640 1 --2- 172.21.15.32:0/2746593969 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f548406e930 0x7f5484141ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:49:26.082 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:26.080+0000 7f547be00640 1 -- 172.21.15.32:0/2746593969 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f548406e930 msgr2=0x7f5484141ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:49:26.082 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:26.080+0000 7f547be00640 1 --2- 172.21.15.32:0/2746593969 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f548406e930 0x7f5484141ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:49:26.083 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:26.080+0000 7f547be00640 1 -- 172.21.15.32:0/2746593969 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f548415fff0 msgr2=0x7f548415f130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:49:26.083 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:26.080+0000 7f547be00640 1 --2- 172.21.15.32:0/2746593969 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f548415fff0 0x7f548415f130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:49:26.083 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:26.080+0000 7f547be00640 1 -- 172.21.15.32:0/2746593969 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f54841f6970 con 0x7f5484144f90 2024-08-03T20:49:26.084 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:26.081+0000 7f547be00640 1 --2- 172.21.15.32:0/2746593969 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5484144f90 0x7f548415ebf0 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f546c0098f0 tx=0x7f546c033c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:49:26.084 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:26.081+0000 7f547b400640 1 -- 172.21.15.32:0/2746593969 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f546c031d60 con 0x7f5484144f90 2024-08-03T20:49:26.084 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:26.081+0000 7f547b400640 1 -- 172.21.15.32:0/2746593969 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f546c005020 con 0x7f5484144f90 2024-08-03T20:49:26.084 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:26.081+0000 7f547b400640 1 -- 172.21.15.32:0/2746593969 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f546c005430 con 0x7f5484144f90 2024-08-03T20:49:26.084 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:26.081+0000 7f548a200640 1 -- 172.21.15.32:0/2746593969 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f54841f6c50 con 0x7f5484144f90 2024-08-03T20:49:26.084 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:26.081+0000 7f548a200640 1 -- 172.21.15.32:0/2746593969 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f54841f7210 con 0x7f5484144f90 2024-08-03T20:49:26.084 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:26.082+0000 7f548a200640 1 -- 172.21.15.32:0/2746593969 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5484160400 con 0x7f5484144f90 2024-08-03T20:49:26.084 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:26.083+0000 7f547b400640 1 -- 172.21.15.32:0/2746593969 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f546c0047a0 con 0x7f5484144f90 2024-08-03T20:49:26.085 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:26.084+0000 7f547b400640 1 --2- 172.21.15.32:0/2746593969 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f5460077040 0x7f5460079550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:49:26.086 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:26.084+0000 7f547b400640 1 -- 172.21.15.32:0/2746593969 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(33..33 src has 1..33) ==== 5795+0+0 (secure 0 0 0) 0x7f546c0b9b00 con 0x7f5484144f90 2024-08-03T20:49:26.086 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:26.085+0000 7f5483400640 1 --2- 172.21.15.32:0/2746593969 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f5460077040 0x7f5460079550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:49:26.087 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:26.085+0000 7f5483400640 1 --2- 172.21.15.32:0/2746593969 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f5460077040 0x7f5460079550 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f5474000f60 tx=0x7f5474009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:49:26.090 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:26.089+0000 7f547b400640 1 -- 172.21.15.32:0/2746593969 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7f546c03f030 con 0x7f5484144f90 2024-08-03T20:49:26.332 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:26.331+0000 7f548a200640 1 -- 172.21.15.32:0/2746593969 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7f548415a910 con 0x7f5484144f90 2024-08-03T20:49:26.333 INFO:tasks.ceph.mon.a.smithi032.stderr:2024-08-03T20:49:26.332+0000 7f4e6d600640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-08-03T20:49:26.335 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:26.334+0000 7f547b400640 1 -- 172.21.15.32:0/2746593969 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v26) ==== 161+0+0 (secure 0 0 0) 0x7f546c031770 con 0x7f5484144f90 2024-08-03T20:49:26.335 INFO:teuthology.orchestra.run.smithi032.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-08-03T20:49:26.338 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:26.336+0000 7f548a200640 1 -- 172.21.15.32:0/2746593969 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f5460077040 msgr2=0x7f5460079550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:49:26.338 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:26.336+0000 7f548a200640 1 --2- 172.21.15.32:0/2746593969 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f5460077040 0x7f5460079550 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f5474000f60 tx=0x7f5474009040 comp rx=0 tx=0).stop 2024-08-03T20:49:26.338 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:26.336+0000 7f548a200640 1 -- 172.21.15.32:0/2746593969 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5484144f90 msgr2=0x7f548415ebf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:49:26.338 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:26.336+0000 7f548a200640 1 --2- 172.21.15.32:0/2746593969 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5484144f90 0x7f548415ebf0 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f546c0098f0 tx=0x7f546c033c20 comp rx=0 tx=0).stop 2024-08-03T20:49:26.338 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:26.336+0000 7f548a200640 1 -- 172.21.15.32:0/2746593969 shutdown_connections 2024-08-03T20:49:26.338 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:26.336+0000 7f548a200640 1 --2- 172.21.15.32:0/2746593969 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f5460077040 0x7f5460079550 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:49:26.338 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:26.336+0000 7f548a200640 1 --2- 172.21.15.32:0/2746593969 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f548415fff0 0x7f548415f130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:49:26.338 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:26.336+0000 7f548a200640 1 --2- 172.21.15.32:0/2746593969 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5484144f90 0x7f548415ebf0 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:49:26.339 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:26.336+0000 7f548a200640 1 --2- 172.21.15.32:0/2746593969 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f548406e930 0x7f5484141ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:49:26.339 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:26.336+0000 7f548a200640 1 -- 172.21.15.32:0/2746593969 >> 172.21.15.32:0/2746593969 conn(0x7f54840b1bd0 msgr2=0x7f54840b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:49:26.339 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:26.337+0000 7f548a200640 1 -- 172.21.15.32:0/2746593969 shutdown_connections 2024-08-03T20:49:26.339 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:26.337+0000 7f548a200640 1 -- 172.21.15.32:0/2746593969 wait complete. 2024-08-03T20:49:26.346 INFO:tasks.ceph.mds.a.smithi032.stderr: -3694> 2024-08-03T20:48:44.097+0000 7fb2f6a00640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-08-03T20:49:26.365 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-08-03T20:49:26.474 INFO:tasks.ceph.mds.a.smithi032.stdout:starting mds.a at 2024-08-03T20:49:26.601 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:26.599+0000 7fac4b400640 1 -- 172.21.15.32:0/3211372653 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fac2c003a50 msgr2=0x7fac2c003ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:49:26.601 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:26.599+0000 7fac4b400640 1 --2- 172.21.15.32:0/3211372653 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fac2c003a50 0x7fac2c003ec0 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7fac38009b50 tx=0x7fac2c005830 comp rx=0 tx=0).stop 2024-08-03T20:49:26.601 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:26.599+0000 7fac49400640 1 -- 172.21.15.32:0/3211372653 reap_dead start 2024-08-03T20:49:26.601 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:26.599+0000 7fac4b400640 1 -- 172.21.15.32:0/3211372653 shutdown_connections 2024-08-03T20:49:26.601 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:26.599+0000 7fac4b400640 1 -- 172.21.15.32:0/3211372653 >> 172.21.15.32:0/3211372653 conn(0x7fac440b1bd0 msgr2=0x7fac440b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:49:26.601 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:26.599+0000 7fac4b400640 1 -- 172.21.15.32:0/3211372653 shutdown_connections 2024-08-03T20:49:26.602 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:26.599+0000 7fac4b400640 1 -- 172.21.15.32:0/3211372653 wait complete. 2024-08-03T20:49:26.603 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:26.602+0000 7fac4b400640 1 Processor -- start 2024-08-03T20:49:26.603 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:26.602+0000 7fac4b400640 1 -- start start 2024-08-03T20:49:26.606 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:26.603+0000 7fac4b400640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fac2c003a50 0x7fac440a9c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:49:26.606 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:26.603+0000 7fac4b400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fac440aa150 0x7fac441f8c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:49:26.606 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:26.603+0000 7fac4b400640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fac441f91a0 0x7fac441445f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:49:26.606 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:26.603+0000 7fac4b400640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fac440cd490 con 0x7fac440aa150 2024-08-03T20:49:26.606 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:26.603+0000 7fac4b400640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7fac440cd310 con 0x7fac441f91a0 2024-08-03T20:49:26.606 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:26.603+0000 7fac4b400640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7fac440b3ce0 con 0x7fac2c003a50 2024-08-03T20:49:26.606 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:26.603+0000 7fac43e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fac440aa150 0x7fac441f8c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:49:26.607 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:26.604+0000 7fac43e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fac440aa150 0x7fac441f8c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:60132/0 (socket says 172.21.15.32:60132) 2024-08-03T20:49:26.607 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:26.604+0000 7fac43e00640 1 -- 172.21.15.32:0/2439460584 learned_addr learned my addr 172.21.15.32:0/2439460584 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:49:26.607 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:26.604+0000 7fac48a00640 1 --2- 172.21.15.32:0/2439460584 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fac2c003a50 0x7fac440a9c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:49:26.607 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:26.604+0000 7fac49400640 1 --2- 172.21.15.32:0/2439460584 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fac441f91a0 0x7fac441445f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:49:26.607 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:26.604+0000 7fac43e00640 1 -- 172.21.15.32:0/2439460584 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fac2c003a50 msgr2=0x7fac440a9c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:49:26.607 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:26.604+0000 7fac43e00640 1 --2- 172.21.15.32:0/2439460584 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fac2c003a50 0x7fac440a9c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:49:26.607 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:26.604+0000 7fac43e00640 1 -- 172.21.15.32:0/2439460584 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fac441f91a0 msgr2=0x7fac441445f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:49:26.607 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:26.604+0000 7fac43e00640 1 --2- 172.21.15.32:0/2439460584 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fac441f91a0 0x7fac441445f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:49:26.607 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:26.604+0000 7fac43e00640 1 -- 172.21.15.32:0/2439460584 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fac441f50b0 con 0x7fac440aa150 2024-08-03T20:49:26.607 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:26.604+0000 7fac49400640 1 --2- 172.21.15.32:0/2439460584 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fac441f91a0 0x7fac441445f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-03T20:49:26.607 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:26.604+0000 7fac43e00640 1 --2- 172.21.15.32:0/2439460584 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fac440aa150 0x7fac441f8c60 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7fac30011ab0 tx=0x7fac30011f70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:49:26.607 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:26.604+0000 7fac40c00640 1 -- 172.21.15.32:0/2439460584 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac3001e070 con 0x7fac440aa150 2024-08-03T20:49:26.607 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:26.605+0000 7fac40c00640 1 -- 172.21.15.32:0/2439460584 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fac3000fe70 con 0x7fac440aa150 2024-08-03T20:49:26.607 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:26.605+0000 7fac40c00640 1 -- 172.21.15.32:0/2439460584 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac300042b0 con 0x7fac440aa150 2024-08-03T20:49:26.608 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:26.605+0000 7fac4b400640 1 -- 172.21.15.32:0/2439460584 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fac441f5390 con 0x7fac440aa150 2024-08-03T20:49:26.608 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:26.605+0000 7fac4b400640 1 -- 172.21.15.32:0/2439460584 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fac441f5950 con 0x7fac440aa150 2024-08-03T20:49:26.608 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:26.606+0000 7fac4b400640 1 -- 172.21.15.32:0/2439460584 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fac4406e950 con 0x7fac440aa150 2024-08-03T20:49:26.608 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:26.607+0000 7fac40c00640 1 -- 172.21.15.32:0/2439460584 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7fac30021070 con 0x7fac440aa150 2024-08-03T20:49:26.610 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:26.609+0000 7fac40c00640 1 --2- 172.21.15.32:0/2439460584 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fac24077010 0x7fac24079520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:49:26.610 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:26.609+0000 7fac40c00640 1 -- 172.21.15.32:0/2439460584 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(34..34 src has 1..34) ==== 5881+0+0 (secure 0 0 0) 0x7fac30099380 con 0x7fac440aa150 2024-08-03T20:49:26.610 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:26.609+0000 7fac48a00640 1 --2- 172.21.15.32:0/2439460584 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fac24077010 0x7fac24079520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:49:26.611 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:26.610+0000 7fac48a00640 1 --2- 172.21.15.32:0/2439460584 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fac24077010 0x7fac24079520 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fac38035000 tx=0x7fac3802d040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:49:26.615 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:26.614+0000 7fac40c00640 1 -- 172.21.15.32:0/2439460584 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7fac30061e70 con 0x7fac440aa150 2024-08-03T20:49:26.863 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:26.862+0000 7fac4b400640 1 -- 172.21.15.32:0/2439460584 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) -- 0x7fac440aa590 con 0x7fac440aa150 2024-08-03T20:49:27.428 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:27.426+0000 7fac40c00640 1 -- 172.21.15.32:0/2439460584 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v27) ==== 104+0+0 (secure 0 0 0) 0x7fac300663c0 con 0x7fac440aa150 2024-08-03T20:49:27.431 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:27.429+0000 7fac4b400640 1 -- 172.21.15.32:0/2439460584 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fac24077010 msgr2=0x7fac24079520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:49:27.431 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:27.429+0000 7fac4b400640 1 --2- 172.21.15.32:0/2439460584 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fac24077010 0x7fac24079520 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fac38035000 tx=0x7fac3802d040 comp rx=0 tx=0).stop 2024-08-03T20:49:27.431 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:27.429+0000 7fac4b400640 1 -- 172.21.15.32:0/2439460584 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fac440aa150 msgr2=0x7fac441f8c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:49:27.432 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:27.429+0000 7fac4b400640 1 --2- 172.21.15.32:0/2439460584 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fac440aa150 0x7fac441f8c60 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7fac30011ab0 tx=0x7fac30011f70 comp rx=0 tx=0).stop 2024-08-03T20:49:27.432 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:27.430+0000 7fac4b400640 1 -- 172.21.15.32:0/2439460584 shutdown_connections 2024-08-03T20:49:27.432 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:27.430+0000 7fac4b400640 1 --2- 172.21.15.32:0/2439460584 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fac24077010 0x7fac24079520 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:49:27.432 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:27.430+0000 7fac4b400640 1 --2- 172.21.15.32:0/2439460584 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fac441f91a0 0x7fac441445f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:49:27.432 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:27.430+0000 7fac4b400640 1 --2- 172.21.15.32:0/2439460584 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fac440aa150 0x7fac441f8c60 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:49:27.432 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:27.430+0000 7fac4b400640 1 --2- 172.21.15.32:0/2439460584 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fac2c003a50 0x7fac440a9c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:49:27.432 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:27.430+0000 7fac4b400640 1 -- 172.21.15.32:0/2439460584 >> 172.21.15.32:0/2439460584 conn(0x7fac440b1bd0 msgr2=0x7fac4415db10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:49:27.432 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:27.430+0000 7fac4b400640 1 -- 172.21.15.32:0/2439460584 shutdown_connections 2024-08-03T20:49:27.432 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:27.430+0000 7fac4b400640 1 -- 172.21.15.32:0/2439460584 wait complete. 2024-08-03T20:49:27.457 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-08-03T20:49:27.692 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:27.690+0000 7f91b4400640 1 -- 172.21.15.32:0/2178014526 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f91ac145c50 msgr2=0x7f91ac06e950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:49:27.692 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:27.690+0000 7f91b4400640 1 --2- 172.21.15.32:0/2178014526 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f91ac145c50 0x7f91ac06e950 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f91ac0cff30 tx=0x7f91a002a580 comp rx=0 tx=0).stop 2024-08-03T20:49:27.692 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:27.690+0000 7f91b4400640 1 -- 172.21.15.32:0/2178014526 shutdown_connections 2024-08-03T20:49:27.692 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:27.690+0000 7f91b4400640 1 --2- 172.21.15.32:0/2178014526 >> v2:172.21.15.118:3300/0 conn(0x7f91ac0cccf0 0x7f91ac0cd100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:49:27.692 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:27.690+0000 7f91b4400640 1 --2- 172.21.15.32:0/2178014526 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f91ac145c50 0x7f91ac06e950 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:49:27.692 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:27.690+0000 7f91b4400640 1 -- 172.21.15.32:0/2178014526 >> 172.21.15.32:0/2178014526 conn(0x7f91ac0b1bd0 msgr2=0x7f91ac0b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:49:27.692 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:27.690+0000 7f91b4400640 1 -- 172.21.15.32:0/2178014526 shutdown_connections 2024-08-03T20:49:27.692 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:27.690+0000 7f91b4400640 1 -- 172.21.15.32:0/2178014526 wait complete. 2024-08-03T20:49:27.693 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:27.692+0000 7f91b4400640 1 Processor -- start 2024-08-03T20:49:27.694 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:27.692+0000 7f91b4400640 1 -- start start 2024-08-03T20:49:27.696 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:27.694+0000 7f91b4400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f91ac0cccf0 0x7f91ac1461e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:49:27.696 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:27.694+0000 7f91b4400640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f91ac0ced30 0x7f91ac13c060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:49:27.696 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:27.694+0000 7f91b4400640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f91ac145c50 0x7f91ac144320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:49:27.696 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:27.694+0000 7f91b4400640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f91ac0b40b0 con 0x7f91ac0cccf0 2024-08-03T20:49:27.696 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:27.694+0000 7f91b4400640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f91ac0b3f30 con 0x7f91ac0ced30 2024-08-03T20:49:27.696 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:27.694+0000 7f91b4400640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7f91ac0b4230 con 0x7f91ac145c50 2024-08-03T20:49:27.697 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:27.694+0000 7f91b1a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f91ac0cccf0 0x7f91ac1461e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:49:27.697 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:27.694+0000 7f91b1a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f91ac0cccf0 0x7f91ac1461e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:60136/0 (socket says 172.21.15.32:60136) 2024-08-03T20:49:27.697 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:27.694+0000 7f91b1a00640 1 -- 172.21.15.32:0/707837977 learned_addr learned my addr 172.21.15.32:0/707837977 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:49:27.697 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:27.694+0000 7f91b2400640 1 --2- 172.21.15.32:0/707837977 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f91ac145c50 0x7f91ac144320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:49:27.697 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:27.694+0000 7f91b1000640 1 --2- 172.21.15.32:0/707837977 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f91ac0ced30 0x7f91ac13c060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:49:27.697 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:27.695+0000 7f91b1a00640 1 -- 172.21.15.32:0/707837977 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f91ac145c50 msgr2=0x7f91ac144320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:49:27.697 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:27.695+0000 7f91b1a00640 1 --2- 172.21.15.32:0/707837977 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f91ac145c50 0x7f91ac144320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:49:27.697 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:27.695+0000 7f91b1a00640 1 -- 172.21.15.32:0/707837977 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f91ac0ced30 msgr2=0x7f91ac13c060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:49:27.697 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:27.695+0000 7f91b1a00640 1 --2- 172.21.15.32:0/707837977 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f91ac0ced30 0x7f91ac13c060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:49:27.697 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:27.695+0000 7f91b1a00640 1 -- 172.21.15.32:0/707837977 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f91ac13c5a0 con 0x7f91ac0cccf0 2024-08-03T20:49:27.697 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:27.695+0000 7f91b2400640 1 --2- 172.21.15.32:0/707837977 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f91ac145c50 0x7f91ac144320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-03T20:49:27.697 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:27.695+0000 7f91b1000640 1 --2- 172.21.15.32:0/707837977 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f91ac0ced30 0x7f91ac13c060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-03T20:49:27.697 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:27.695+0000 7f91b1a00640 1 --2- 172.21.15.32:0/707837977 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f91ac0cccf0 0x7f91ac1461e0 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f91a0034040 tx=0x7f91a00039e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:49:27.698 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:27.695+0000 7f91a9600640 1 -- 172.21.15.32:0/707837977 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91a003e030 con 0x7f91ac0cccf0 2024-08-03T20:49:27.698 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:27.695+0000 7f91a9600640 1 -- 172.21.15.32:0/707837977 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f91a00047a0 con 0x7f91ac0cccf0 2024-08-03T20:49:27.698 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:27.695+0000 7f91b4400640 1 -- 172.21.15.32:0/707837977 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f91ac146a10 con 0x7f91ac0cccf0 2024-08-03T20:49:27.699 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:27.695+0000 7f91a9600640 1 -- 172.21.15.32:0/707837977 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91a0035040 con 0x7f91ac0cccf0 2024-08-03T20:49:27.699 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:27.695+0000 7f91b4400640 1 -- 172.21.15.32:0/707837977 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f91ac146fd0 con 0x7f91ac0cccf0 2024-08-03T20:49:27.699 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:27.697+0000 7f91b4400640 1 -- 172.21.15.32:0/707837977 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f91ac0b4750 con 0x7f91ac0cccf0 2024-08-03T20:49:27.699 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:27.697+0000 7f91a9600640 1 -- 172.21.15.32:0/707837977 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f91a0041030 con 0x7f91ac0cccf0 2024-08-03T20:49:27.700 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:27.698+0000 7f91a9600640 1 --2- 172.21.15.32:0/707837977 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f918c077040 0x7f918c079550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:49:27.700 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:27.699+0000 7f91a9600640 1 -- 172.21.15.32:0/707837977 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(34..34 src has 1..34) ==== 5881+0+0 (secure 0 0 0) 0x7f91a00b9100 con 0x7f91ac0cccf0 2024-08-03T20:49:27.700 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:27.699+0000 7f91b1000640 1 --2- 172.21.15.32:0/707837977 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f918c077040 0x7f918c079550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:49:27.701 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:27.700+0000 7f91b1000640 1 --2- 172.21.15.32:0/707837977 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f918c077040 0x7f918c079550 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f91ac06ac30 tx=0x7f9198005dd0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:49:27.704 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:27.703+0000 7f91a9600640 1 -- 172.21.15.32:0/707837977 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7f91a00821c0 con 0x7f91ac0cccf0 2024-08-03T20:49:27.910 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:27.909+0000 7f91b4400640 1 -- 172.21.15.32:0/707837977 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) -- 0x7f91ac147520 con 0x7f91ac0cccf0 2024-08-03T20:49:28.440 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:28.438+0000 7f91a9600640 1 -- 172.21.15.32:0/707837977 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v35) ==== 181+0+0 (secure 0 0 0) 0x7f91a0081ac0 con 0x7f91ac0cccf0 2024-08-03T20:49:28.440 INFO:teuthology.orchestra.run.smithi032.stderr:pool 'cephfs_metadata' removed 2024-08-03T20:49:28.443 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:28.441+0000 7f91b4400640 1 -- 172.21.15.32:0/707837977 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f918c077040 msgr2=0x7f918c079550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:49:28.443 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:28.441+0000 7f91b4400640 1 --2- 172.21.15.32:0/707837977 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f918c077040 0x7f918c079550 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f91ac06ac30 tx=0x7f9198005dd0 comp rx=0 tx=0).stop 2024-08-03T20:49:28.443 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:28.441+0000 7f91b4400640 1 -- 172.21.15.32:0/707837977 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f91ac0cccf0 msgr2=0x7f91ac1461e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:49:28.443 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:28.441+0000 7f91b4400640 1 --2- 172.21.15.32:0/707837977 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f91ac0cccf0 0x7f91ac1461e0 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f91a0034040 tx=0x7f91a00039e0 comp rx=0 tx=0).stop 2024-08-03T20:49:28.443 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:28.441+0000 7f91b4400640 1 -- 172.21.15.32:0/707837977 shutdown_connections 2024-08-03T20:49:28.443 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:28.441+0000 7f91b4400640 1 --2- 172.21.15.32:0/707837977 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f918c077040 0x7f918c079550 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:49:28.443 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:28.441+0000 7f91b4400640 1 --2- 172.21.15.32:0/707837977 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f91ac0ced30 0x7f91ac13c060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:49:28.443 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:28.441+0000 7f91b4400640 1 --2- 172.21.15.32:0/707837977 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f91ac145c50 0x7f91ac144320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:49:28.443 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:28.441+0000 7f91b4400640 1 --2- 172.21.15.32:0/707837977 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f91ac0cccf0 0x7f91ac1461e0 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:49:28.443 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:28.441+0000 7f91b4400640 1 -- 172.21.15.32:0/707837977 >> 172.21.15.32:0/707837977 conn(0x7f91ac0b1bd0 msgr2=0x7f91ac0b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:49:28.443 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:28.441+0000 7f91b4400640 1 -- 172.21.15.32:0/707837977 shutdown_connections 2024-08-03T20:49:28.444 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:28.441+0000 7f91b4400640 1 -- 172.21.15.32:0/707837977 wait complete. 2024-08-03T20:49:28.470 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-08-03T20:49:28.716 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:28.714+0000 7f11dda00640 1 -- 172.21.15.32:0/2631879074 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f11bc004800 msgr2=0x7f11bc004c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:49:28.716 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:28.714+0000 7f11dda00640 1 --2- 172.21.15.32:0/2631879074 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f11bc004800 0x7f11bc004c10 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f11c0009b50 tx=0x7f11bc005830 comp rx=0 tx=0).stop 2024-08-03T20:49:28.716 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:28.714+0000 7f11dd000640 1 -- 172.21.15.32:0/2631879074 reap_dead start 2024-08-03T20:49:28.716 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:28.714+0000 7f11dda00640 1 -- 172.21.15.32:0/2631879074 shutdown_connections 2024-08-03T20:49:28.716 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:28.714+0000 7f11dda00640 1 -- 172.21.15.32:0/2631879074 >> 172.21.15.32:0/2631879074 conn(0x7f11d80cfd80 msgr2=0x7f11d80d0190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:49:28.716 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:28.714+0000 7f11dda00640 1 -- 172.21.15.32:0/2631879074 shutdown_connections 2024-08-03T20:49:28.716 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:28.714+0000 7f11dda00640 1 -- 172.21.15.32:0/2631879074 wait complete. 2024-08-03T20:49:28.718 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:28.716+0000 7f11dda00640 1 Processor -- start 2024-08-03T20:49:28.718 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:28.717+0000 7f11dda00640 1 -- start start 2024-08-03T20:49:28.720 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:28.718+0000 7f11dda00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f11bc004800 0x7f11d815a7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:49:28.720 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:28.718+0000 7f11dda00640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f11d80cf460 0x7f11d8152e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:49:28.720 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:28.718+0000 7f11dda00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f11d80aea10 0x7f11d80aee20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:49:28.720 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:28.718+0000 7f11dda00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f11d81601a0 con 0x7f11bc004800 2024-08-03T20:49:28.721 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:28.718+0000 7f11dda00640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f11d8160020 con 0x7f11d80aea10 2024-08-03T20:49:28.721 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:28.718+0000 7f11dda00640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7f11d806eb70 con 0x7f11d80cf460 2024-08-03T20:49:28.721 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:28.718+0000 7f11d7e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f11bc004800 0x7f11d815a7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:49:28.721 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:28.718+0000 7f11d7e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f11bc004800 0x7f11d815a7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:60156/0 (socket says 172.21.15.32:60156) 2024-08-03T20:49:28.721 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:28.718+0000 7f11d7e00640 1 -- 172.21.15.32:0/3004111543 learned_addr learned my addr 172.21.15.32:0/3004111543 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:49:28.721 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:28.719+0000 7f11dd000640 1 --2- 172.21.15.32:0/3004111543 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f11d80aea10 0x7f11d80aee20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:49:28.721 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:28.719+0000 7f11cfe00640 1 --2- 172.21.15.32:0/3004111543 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f11d80cf460 0x7f11d8152e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:49:28.721 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:28.719+0000 7f11d7e00640 1 -- 172.21.15.32:0/3004111543 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f11d80cf460 msgr2=0x7f11d8152e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:49:28.721 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:28.719+0000 7f11d7e00640 1 --2- 172.21.15.32:0/3004111543 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f11d80cf460 0x7f11d8152e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:49:28.721 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:28.719+0000 7f11d7e00640 1 -- 172.21.15.32:0/3004111543 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f11d80aea10 msgr2=0x7f11d80aee20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:49:28.721 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:28.719+0000 7f11d7e00640 1 --2- 172.21.15.32:0/3004111543 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f11d80aea10 0x7f11d80aee20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:49:28.721 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:28.719+0000 7f11d7e00640 1 -- 172.21.15.32:0/3004111543 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f11d80ad970 con 0x7f11bc004800 2024-08-03T20:49:28.721 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:28.719+0000 7f11d7e00640 1 --2- 172.21.15.32:0/3004111543 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f11bc004800 0x7f11d815a7f0 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f11c8006600 tx=0x7f11c80039c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:49:28.722 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:28.719+0000 7f11d4c00640 1 -- 172.21.15.32:0/3004111543 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11c8015070 con 0x7f11bc004800 2024-08-03T20:49:28.722 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:28.719+0000 7f11d4c00640 1 -- 172.21.15.32:0/3004111543 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f11c8011920 con 0x7f11bc004800 2024-08-03T20:49:28.722 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:28.719+0000 7f11dda00640 1 -- 172.21.15.32:0/3004111543 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f11d80adc50 con 0x7f11bc004800 2024-08-03T20:49:28.723 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:28.719+0000 7f11d4c00640 1 -- 172.21.15.32:0/3004111543 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11c8011d70 con 0x7f11bc004800 2024-08-03T20:49:28.723 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:28.719+0000 7f11dda00640 1 -- 172.21.15.32:0/3004111543 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f11d80ae210 con 0x7f11bc004800 2024-08-03T20:49:28.723 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:28.721+0000 7f11dda00640 1 -- 172.21.15.32:0/3004111543 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f11d806e630 con 0x7f11bc004800 2024-08-03T20:49:28.723 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:28.722+0000 7f11d4c00640 1 -- 172.21.15.32:0/3004111543 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f11c8018070 con 0x7f11bc004800 2024-08-03T20:49:28.724 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:28.723+0000 7f11d4c00640 1 --2- 172.21.15.32:0/3004111543 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f11b4077010 0x7f11b4079520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:49:28.724 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:28.723+0000 7f11d4c00640 1 -- 172.21.15.32:0/3004111543 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 5443+0+0 (secure 0 0 0) 0x7f11c8099530 con 0x7f11bc004800 2024-08-03T20:49:28.724 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:28.723+0000 7f11cfe00640 1 --2- 172.21.15.32:0/3004111543 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f11b4077010 0x7f11b4079520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:49:28.725 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:28.724+0000 7f11cfe00640 1 --2- 172.21.15.32:0/3004111543 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f11b4077010 0x7f11b4079520 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f11c0035000 tx=0x7f11c002d040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:49:28.728 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:28.727+0000 7f11d4c00640 1 -- 172.21.15.32:0/3004111543 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7f11d8160760 con 0x7f11bc004800 2024-08-03T20:49:28.934 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:28.933+0000 7f11dda00640 1 -- 172.21.15.32:0/3004111543 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) -- 0x7f11d80d09e0 con 0x7f11bc004800 2024-08-03T20:49:29.442 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:29.440+0000 7f11d4c00640 1 -- 172.21.15.32:0/3004111543 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v36) ==== 169+0+0 (secure 0 0 0) 0x7f11c80628d0 con 0x7f11bc004800 2024-08-03T20:49:29.442 INFO:teuthology.orchestra.run.smithi032.stderr:pool 'cephfs_data' removed 2024-08-03T20:49:29.446 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:29.443+0000 7f11dda00640 1 -- 172.21.15.32:0/3004111543 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f11b4077010 msgr2=0x7f11b4079520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:49:29.446 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:29.443+0000 7f11dda00640 1 --2- 172.21.15.32:0/3004111543 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f11b4077010 0x7f11b4079520 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f11c0035000 tx=0x7f11c002d040 comp rx=0 tx=0).stop 2024-08-03T20:49:29.446 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:29.443+0000 7f11dda00640 1 -- 172.21.15.32:0/3004111543 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f11bc004800 msgr2=0x7f11d815a7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:49:29.446 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:29.443+0000 7f11dda00640 1 --2- 172.21.15.32:0/3004111543 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f11bc004800 0x7f11d815a7f0 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f11c8006600 tx=0x7f11c80039c0 comp rx=0 tx=0).stop 2024-08-03T20:49:29.447 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:29.443+0000 7f11dda00640 1 -- 172.21.15.32:0/3004111543 shutdown_connections 2024-08-03T20:49:29.447 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:29.443+0000 7f11dda00640 1 --2- 172.21.15.32:0/3004111543 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f11b4077010 0x7f11b4079520 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:49:29.447 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:29.443+0000 7f11dda00640 1 --2- 172.21.15.32:0/3004111543 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f11d80aea10 0x7f11d80aee20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:49:29.447 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:29.443+0000 7f11dda00640 1 --2- 172.21.15.32:0/3004111543 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f11d80cf460 0x7f11d8152e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:49:29.447 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:29.443+0000 7f11dda00640 1 --2- 172.21.15.32:0/3004111543 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f11bc004800 0x7f11d815a7f0 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:49:29.448 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:29.443+0000 7f11dda00640 1 -- 172.21.15.32:0/3004111543 >> 172.21.15.32:0/3004111543 conn(0x7f11d80cfd80 msgr2=0x7f11d8072830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:49:29.448 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:29.444+0000 7f11dda00640 1 -- 172.21.15.32:0/3004111543 shutdown_connections 2024-08-03T20:49:29.448 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:29.444+0000 7f11dda00640 1 -- 172.21.15.32:0/3004111543 wait complete. 2024-08-03T20:49:29.475 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config reset 27 2024-08-03T20:49:29.704 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:29.702+0000 7f5a7fe00640 1 -- 172.21.15.32:0/1167302550 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5a7806e930 msgr2=0x7f5a78144f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:49:29.705 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:29.702+0000 7f5a7fe00640 1 --2- 172.21.15.32:0/1167302550 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5a7806e930 0x7f5a78144f90 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f5a780afaf0 tx=0x7f5a6c02a560 comp rx=0 tx=0).stop 2024-08-03T20:49:29.705 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:29.703+0000 7f5a7fe00640 1 -- 172.21.15.32:0/1167302550 shutdown_connections 2024-08-03T20:49:29.705 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:29.703+0000 7f5a7fe00640 1 --2- 172.21.15.32:0/1167302550 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f5a780747b0 0x7f5a780cc8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:49:29.705 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:29.703+0000 7f5a7fe00640 1 --2- 172.21.15.32:0/1167302550 >> v2:172.21.15.118:3300/0 conn(0x7f5a78073e60 0x7f5a78074270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:49:29.705 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:29.703+0000 7f5a7fe00640 1 --2- 172.21.15.32:0/1167302550 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5a7806e930 0x7f5a78144f90 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:49:29.705 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:29.703+0000 7f5a7fe00640 1 -- 172.21.15.32:0/1167302550 >> 172.21.15.32:0/1167302550 conn(0x7f5a780b1bd0 msgr2=0x7f5a780b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:49:29.705 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:29.703+0000 7f5a7fe00640 1 -- 172.21.15.32:0/1167302550 shutdown_connections 2024-08-03T20:49:29.705 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:29.703+0000 7f5a7fe00640 1 -- 172.21.15.32:0/1167302550 wait complete. 2024-08-03T20:49:29.706 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:29.705+0000 7f5a7fe00640 1 Processor -- start 2024-08-03T20:49:29.706 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:29.705+0000 7f5a7fe00640 1 -- start start 2024-08-03T20:49:29.708 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:29.707+0000 7f5a7fe00640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f5a7806e930 0x7f5a7813de20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:49:29.708 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:29.707+0000 7f5a7fe00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5a78073e60 0x7f5a780cb630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:49:29.709 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:29.707+0000 7f5a7fe00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f5a780747b0 0x7f5a780cff60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:49:29.709 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:29.707+0000 7f5a7fe00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f5a780cf9c0 con 0x7f5a78073e60 2024-08-03T20:49:29.709 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:29.707+0000 7f5a7fe00640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f5a780cf840 con 0x7f5a780747b0 2024-08-03T20:49:29.709 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:29.707+0000 7f5a7fe00640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7f5a780cce20 con 0x7f5a7806e930 2024-08-03T20:49:29.709 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:29.707+0000 7f5a7ca00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5a78073e60 0x7f5a780cb630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:49:29.709 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:29.707+0000 7f5a7ca00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5a78073e60 0x7f5a780cb630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:60178/0 (socket says 172.21.15.32:60178) 2024-08-03T20:49:29.709 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:29.707+0000 7f5a7ca00640 1 -- 172.21.15.32:0/3942308961 learned_addr learned my addr 172.21.15.32:0/3942308961 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:49:29.709 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:29.707+0000 7f5a7de00640 1 --2- 172.21.15.32:0/3942308961 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f5a780747b0 0x7f5a780cff60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:49:29.709 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:29.707+0000 7f5a7d400640 1 --2- 172.21.15.32:0/3942308961 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f5a7806e930 0x7f5a7813de20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:49:29.710 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:29.707+0000 7f5a7ca00640 1 -- 172.21.15.32:0/3942308961 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f5a7806e930 msgr2=0x7f5a7813de20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:49:29.710 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:29.708+0000 7f5a7ca00640 1 --2- 172.21.15.32:0/3942308961 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f5a7806e930 0x7f5a7813de20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:49:29.710 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:29.708+0000 7f5a7ca00640 1 -- 172.21.15.32:0/3942308961 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f5a780747b0 msgr2=0x7f5a780cff60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:49:29.711 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:29.708+0000 7f5a7ca00640 1 --2- 172.21.15.32:0/3942308961 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f5a780747b0 0x7f5a780cff60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:49:29.711 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:29.708+0000 7f5a7ca00640 1 -- 172.21.15.32:0/3942308961 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5a7813e360 con 0x7f5a78073e60 2024-08-03T20:49:29.711 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:29.708+0000 7f5a7ca00640 1 --2- 172.21.15.32:0/3942308961 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5a78073e60 0x7f5a780cb630 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f5a6400c970 tx=0x7f5a6400ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:49:29.711 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:29.708+0000 7f5a75600640 1 -- 172.21.15.32:0/3942308961 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a64007b50 con 0x7f5a78073e60 2024-08-03T20:49:29.711 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:29.708+0000 7f5a75600640 1 -- 172.21.15.32:0/3942308961 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f5a64007de0 con 0x7f5a78073e60 2024-08-03T20:49:29.711 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:29.708+0000 7f5a75600640 1 -- 172.21.15.32:0/3942308961 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a6400f6b0 con 0x7f5a78073e60 2024-08-03T20:49:29.711 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:29.708+0000 7f5a7fe00640 1 -- 172.21.15.32:0/3942308961 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5a781f8c50 con 0x7f5a78073e60 2024-08-03T20:49:29.711 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:29.708+0000 7f5a7fe00640 1 -- 172.21.15.32:0/3942308961 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5a781f9210 con 0x7f5a78073e60 2024-08-03T20:49:29.711 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:29.709+0000 7f5a7fe00640 1 -- 172.21.15.32:0/3942308961 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5a78074270 con 0x7f5a78073e60 2024-08-03T20:49:29.712 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:29.711+0000 7f5a75600640 1 -- 172.21.15.32:0/3942308961 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f5a6400f850 con 0x7f5a78073e60 2024-08-03T20:49:29.713 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:29.711+0000 7f5a75600640 1 --2- 172.21.15.32:0/3942308961 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f5a58077010 0x7f5a58079520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:49:29.713 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:29.712+0000 7f5a75600640 1 -- 172.21.15.32:0/3942308961 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(36..36 src has 1..36) ==== 5029+0+0 (secure 0 0 0) 0x7f5a64098b60 con 0x7f5a78073e60 2024-08-03T20:49:29.713 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:29.712+0000 7f5a7d400640 1 --2- 172.21.15.32:0/3942308961 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f5a58077010 0x7f5a58079520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:49:29.714 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:29.713+0000 7f5a7d400640 1 --2- 172.21.15.32:0/3942308961 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f5a58077010 0x7f5a58079520 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f5a6c033000 tx=0x7f5a6c004730 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:49:29.717 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:29.716+0000 7f5a75600640 1 -- 172.21.15.32:0/3942308961 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7f5a7813e360 con 0x7f5a78073e60 2024-08-03T20:49:29.923 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:29.922+0000 7f5a7fe00640 1 -- 172.21.15.32:0/3942308961 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "config reset", "num": 27} v 0) -- 0x7f5a78145760 con 0x7f5a78073e60 2024-08-03T20:49:29.928 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:29.926+0000 7f5a75600640 1 -- 172.21.15.32:0/3942308961 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "config reset", "num": 27}]=0 v29) ==== 71+0+0 (secure 0 0 0) 0x7f5a64014030 con 0x7f5a78073e60 2024-08-03T20:49:29.931 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:29.929+0000 7f5a7fe00640 1 -- 172.21.15.32:0/3942308961 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f5a58077010 msgr2=0x7f5a58079520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:49:29.931 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:29.929+0000 7f5a7fe00640 1 --2- 172.21.15.32:0/3942308961 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f5a58077010 0x7f5a58079520 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f5a6c033000 tx=0x7f5a6c004730 comp rx=0 tx=0).stop 2024-08-03T20:49:29.931 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:29.929+0000 7f5a7fe00640 1 -- 172.21.15.32:0/3942308961 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5a78073e60 msgr2=0x7f5a780cb630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:49:29.931 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:29.929+0000 7f5a7fe00640 1 --2- 172.21.15.32:0/3942308961 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5a78073e60 0x7f5a780cb630 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f5a6400c970 tx=0x7f5a6400ce30 comp rx=0 tx=0).stop 2024-08-03T20:49:29.931 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:29.929+0000 7f5a7fe00640 1 -- 172.21.15.32:0/3942308961 shutdown_connections 2024-08-03T20:49:29.931 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:29.929+0000 7f5a7fe00640 1 --2- 172.21.15.32:0/3942308961 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f5a58077010 0x7f5a58079520 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:49:29.931 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:29.929+0000 7f5a7fe00640 1 --2- 172.21.15.32:0/3942308961 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f5a780747b0 0x7f5a780cff60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:49:29.932 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:29.929+0000 7f5a7fe00640 1 --2- 172.21.15.32:0/3942308961 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5a78073e60 0x7f5a780cb630 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:49:29.932 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:29.929+0000 7f5a7fe00640 1 --2- 172.21.15.32:0/3942308961 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f5a7806e930 0x7f5a7813de20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:49:29.932 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:29.929+0000 7f5a7fe00640 1 -- 172.21.15.32:0/3942308961 >> 172.21.15.32:0/3942308961 conn(0x7f5a780b1bd0 msgr2=0x7f5a78140cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:49:29.932 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:29.929+0000 7f5a7fe00640 1 -- 172.21.15.32:0/3942308961 shutdown_connections 2024-08-03T20:49:29.932 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:29.929+0000 7f5a7fe00640 1 -- 172.21.15.32:0/3942308961 wait complete. 2024-08-03T20:49:29.953 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_damage.TestDamage.test_damaged_dentry' 2024-08-03T20:49:30.180 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:30.179+0000 7fc96d400640 1 -- 172.21.15.32:0/2903273396 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc968160ac0 msgr2=0x7fc9680ce2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:49:30.181 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:30.179+0000 7fc96d400640 1 --2- 172.21.15.32:0/2903273396 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc968160ac0 0x7fc9680ce2e0 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7fc958005fd0 tx=0x7fc95802a680 comp rx=0 tx=0).stop 2024-08-03T20:49:30.181 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:30.179+0000 7fc96d400640 1 -- 172.21.15.32:0/2903273396 shutdown_connections 2024-08-03T20:49:30.181 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:30.179+0000 7fc96d400640 1 --2- 172.21.15.32:0/2903273396 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc968160ac0 0x7fc9680ce2e0 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:49:30.181 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:30.179+0000 7fc96d400640 1 -- 172.21.15.32:0/2903273396 >> 172.21.15.32:0/2903273396 conn(0x7fc96806fd20 msgr2=0x7fc968070130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:49:30.181 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:30.179+0000 7fc96d400640 1 -- 172.21.15.32:0/2903273396 shutdown_connections 2024-08-03T20:49:30.181 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:30.179+0000 7fc96d400640 1 -- 172.21.15.32:0/2903273396 wait complete. 2024-08-03T20:49:30.182 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:30.181+0000 7fc96d400640 1 Processor -- start 2024-08-03T20:49:30.183 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:30.181+0000 7fc96d400640 1 -- start start 2024-08-03T20:49:30.185 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:30.183+0000 7fc96d400640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc9680cff00 0x7fc9681e8ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:49:30.185 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:30.183+0000 7fc96d400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc9680d1360 0x7fc9680aff50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:49:30.185 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:30.183+0000 7fc96d400640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fc968160ac0 0x7fc9681f8ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:49:30.185 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:30.183+0000 7fc96d400640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fc9680d0a20 con 0x7fc9680d1360 2024-08-03T20:49:30.186 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:30.183+0000 7fc96d400640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7fc9680cdb80 con 0x7fc9680cff00 2024-08-03T20:49:30.186 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:30.183+0000 7fc96d400640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7fc9680d0ba0 con 0x7fc968160ac0 2024-08-03T20:49:30.186 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:30.183+0000 7fc95fe00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc9680d1360 0x7fc9680aff50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:49:30.186 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:30.183+0000 7fc95fe00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc9680d1360 0x7fc9680aff50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:54232/0 (socket says 172.21.15.32:54232) 2024-08-03T20:49:30.186 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:30.183+0000 7fc95fe00640 1 -- 172.21.15.32:0/387460737 learned_addr learned my addr 172.21.15.32:0/387460737 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:49:30.186 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:30.183+0000 7fc966800640 1 --2- 172.21.15.32:0/387460737 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc9680cff00 0x7fc9681e8ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:49:30.186 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:30.183+0000 7fc967200640 1 --2- 172.21.15.32:0/387460737 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fc968160ac0 0x7fc9681f8ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:49:30.186 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:30.184+0000 7fc95fe00640 1 -- 172.21.15.32:0/387460737 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fc968160ac0 msgr2=0x7fc9681f8ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:49:30.186 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:30.184+0000 7fc95fe00640 1 --2- 172.21.15.32:0/387460737 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fc968160ac0 0x7fc9681f8ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:49:30.186 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:30.184+0000 7fc95fe00640 1 -- 172.21.15.32:0/387460737 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc9680cff00 msgr2=0x7fc9681e8ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:49:30.186 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:30.184+0000 7fc95fe00640 1 --2- 172.21.15.32:0/387460737 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc9680cff00 0x7fc9681e8ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:49:30.186 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:30.184+0000 7fc95fe00640 1 -- 172.21.15.32:0/387460737 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc9681f66a0 con 0x7fc9680d1360 2024-08-03T20:49:30.186 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:30.184+0000 7fc95fe00640 1 --2- 172.21.15.32:0/387460737 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc9680d1360 0x7fc9680aff50 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7fc95000c400 tx=0x7fc95000c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:49:30.187 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:30.184+0000 7fc95ea00640 1 -- 172.21.15.32:0/387460737 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc950017070 con 0x7fc9680d1360 2024-08-03T20:49:30.187 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:30.184+0000 7fc95ea00640 1 -- 172.21.15.32:0/387460737 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fc9500053c0 con 0x7fc9680d1360 2024-08-03T20:49:30.188 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:30.184+0000 7fc95ea00640 1 -- 172.21.15.32:0/387460737 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc950003850 con 0x7fc9680d1360 2024-08-03T20:49:30.188 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:30.184+0000 7fc96d400640 1 -- 172.21.15.32:0/387460737 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc9681e7050 con 0x7fc9680d1360 2024-08-03T20:49:30.188 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:30.185+0000 7fc96d400640 1 -- 172.21.15.32:0/387460737 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc9681429d0 con 0x7fc9680d1360 2024-08-03T20:49:30.188 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:30.186+0000 7fc96d400640 1 -- 172.21.15.32:0/387460737 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc968070620 con 0x7fc9680d1360 2024-08-03T20:49:30.188 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:30.187+0000 7fc95ea00640 1 -- 172.21.15.32:0/387460737 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7fc95001a070 con 0x7fc9680d1360 2024-08-03T20:49:30.189 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:30.188+0000 7fc95ea00640 1 --2- 172.21.15.32:0/387460737 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fc944077010 0x7fc944079520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:49:30.189 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:30.188+0000 7fc95ea00640 1 -- 172.21.15.32:0/387460737 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(36..36 src has 1..36) ==== 5029+0+0 (secure 0 0 0) 0x7fc9500979b0 con 0x7fc9680d1360 2024-08-03T20:49:30.189 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:30.188+0000 7fc966800640 1 --2- 172.21.15.32:0/387460737 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fc944077010 0x7fc944079520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:49:30.190 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:30.189+0000 7fc966800640 1 --2- 172.21.15.32:0/387460737 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fc944077010 0x7fc944079520 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fc958033000 tx=0x7fc958004730 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:49:30.193 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:30.192+0000 7fc95ea00640 1 -- 172.21.15.32:0/387460737 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7fc950060880 con 0x7fc9680d1360 2024-08-03T20:49:30.482 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:30.480+0000 7fc96d400640 1 -- 172.21.15.32:0/387460737 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_damage.TestDamage.test_damaged_dentry"]} v 0) -- 0x7fc968074630 con 0x7fc9680d1360 2024-08-03T20:49:31.452 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:31.450+0000 7fc95ea00640 1 -- 172.21.15.32:0/387460737 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_damage.TestDamage.test_damaged_dentry"]}]=0 v116) ==== 134+0+0 (secure 0 0 0) 0x7fc950069110 con 0x7fc9680d1360 2024-08-03T20:49:31.456 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:31.455+0000 7fc96d400640 1 -- 172.21.15.32:0/387460737 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fc944077010 msgr2=0x7fc944079520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:49:31.457 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:31.455+0000 7fc96d400640 1 --2- 172.21.15.32:0/387460737 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fc944077010 0x7fc944079520 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fc958033000 tx=0x7fc958004730 comp rx=0 tx=0).stop 2024-08-03T20:49:31.457 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:31.455+0000 7fc96d400640 1 -- 172.21.15.32:0/387460737 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc9680d1360 msgr2=0x7fc9680aff50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:49:31.457 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:31.455+0000 7fc96d400640 1 --2- 172.21.15.32:0/387460737 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc9680d1360 0x7fc9680aff50 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7fc95000c400 tx=0x7fc95000c8c0 comp rx=0 tx=0).stop 2024-08-03T20:49:31.457 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:31.455+0000 7fc96d400640 1 -- 172.21.15.32:0/387460737 shutdown_connections 2024-08-03T20:49:31.457 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:31.455+0000 7fc96d400640 1 --2- 172.21.15.32:0/387460737 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fc944077010 0x7fc944079520 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:49:31.457 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:31.455+0000 7fc96d400640 1 --2- 172.21.15.32:0/387460737 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fc968160ac0 0x7fc9681f8ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:49:31.457 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:31.455+0000 7fc96d400640 1 --2- 172.21.15.32:0/387460737 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc9680d1360 0x7fc9680aff50 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:49:31.457 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:31.455+0000 7fc96d400640 1 --2- 172.21.15.32:0/387460737 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc9680cff00 0x7fc9681e8ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:49:31.457 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:31.455+0000 7fc96d400640 1 -- 172.21.15.32:0/387460737 >> 172.21.15.32:0/387460737 conn(0x7fc96806fd20 msgr2=0x7fc9680b0bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:49:31.457 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:31.455+0000 7fc96d400640 1 -- 172.21.15.32:0/387460737 shutdown_connections 2024-08-03T20:49:31.457 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:31.455+0000 7fc96d400640 1 -- 172.21.15.32:0/387460737 wait complete. 2024-08-03T20:49:31.483 INFO:tasks.cephfs_test_runner:test_damaged_dentry (tasks.cephfs.test_damage.TestDamage) ... ok 2024-08-03T20:49:31.483 INFO:tasks.cephfs_test_runner:Starting test: test_dentry_first_existing (tasks.cephfs.test_damage.TestDamage) That the MDS won't abort when the dentry is already known to be damaged. 2024-08-03T20:49:31.484 INFO:tasks.cephfs_test_runner:test_dentry_first_existing (tasks.cephfs.test_damage.TestDamage) 2024-08-03T20:49:31.484 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-08-03T20:49:31.715 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:31.713+0000 7fc212800640 1 -- 172.21.15.32:0/450536484 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc20c073e60 msgr2=0x7fc20c074270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:49:31.715 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:31.713+0000 7fc212800640 1 --2- 172.21.15.32:0/450536484 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc20c073e60 0x7fc20c074270 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7fc20c071700 tx=0x7fc1f802a560 comp rx=0 tx=0).stop 2024-08-03T20:49:31.715 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:31.714+0000 7fc212800640 1 -- 172.21.15.32:0/450536484 shutdown_connections 2024-08-03T20:49:31.716 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:31.714+0000 7fc212800640 1 --2- 172.21.15.32:0/450536484 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc20c073e60 0x7fc20c074270 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:49:31.716 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:31.714+0000 7fc212800640 1 --2- 172.21.15.32:0/450536484 >> v2:172.21.15.118:3300/0 conn(0x7fc20c06e930 0x7fc20c144f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:49:31.716 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:31.714+0000 7fc212800640 1 -- 172.21.15.32:0/450536484 >> 172.21.15.32:0/450536484 conn(0x7fc20c0b1bd0 msgr2=0x7fc20c0b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:49:31.716 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:31.714+0000 7fc212800640 1 -- 172.21.15.32:0/450536484 shutdown_connections 2024-08-03T20:49:31.716 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:31.714+0000 7fc212800640 1 -- 172.21.15.32:0/450536484 wait complete. 2024-08-03T20:49:31.717 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:31.716+0000 7fc212800640 1 Processor -- start 2024-08-03T20:49:31.717 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:31.716+0000 7fc212800640 1 -- start start 2024-08-03T20:49:31.720 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:31.717+0000 7fc212800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc20c06e930 0x7fc20c0b2fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:49:31.720 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:31.717+0000 7fc212800640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc20c073e60 0x7fc20c15f590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:49:31.720 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:31.717+0000 7fc212800640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fc20c0747b0 0x7fc20c0af100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:49:31.720 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:31.717+0000 7fc212800640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fc20c0cf9c0 con 0x7fc20c06e930 2024-08-03T20:49:31.720 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:31.717+0000 7fc212800640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7fc20c0cf840 con 0x7fc20c073e60 2024-08-03T20:49:31.720 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:31.717+0000 7fc212800640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7fc20c0ccff0 con 0x7fc20c0747b0 2024-08-03T20:49:31.720 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:31.718+0000 7fc20b400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc20c06e930 0x7fc20c0b2fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:49:31.720 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:31.718+0000 7fc20b400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc20c06e930 0x7fc20c0b2fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:54252/0 (socket says 172.21.15.32:54252) 2024-08-03T20:49:31.720 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:31.718+0000 7fc20b400640 1 -- 172.21.15.32:0/3009279021 learned_addr learned my addr 172.21.15.32:0/3009279021 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:49:31.720 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:31.718+0000 7fc20aa00640 1 --2- 172.21.15.32:0/3009279021 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc20c073e60 0x7fc20c15f590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:49:31.720 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:31.718+0000 7fc20be00640 1 --2- 172.21.15.32:0/3009279021 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fc20c0747b0 0x7fc20c0af100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:49:31.720 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:31.718+0000 7fc20b400640 1 -- 172.21.15.32:0/3009279021 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fc20c0747b0 msgr2=0x7fc20c0af100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:49:31.721 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:31.718+0000 7fc20b400640 1 --2- 172.21.15.32:0/3009279021 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fc20c0747b0 0x7fc20c0af100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:49:31.721 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:31.718+0000 7fc20b400640 1 -- 172.21.15.32:0/3009279021 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc20c073e60 msgr2=0x7fc20c15f590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:49:31.721 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:31.718+0000 7fc20b400640 1 --2- 172.21.15.32:0/3009279021 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc20c073e60 0x7fc20c15f590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:49:31.721 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:31.718+0000 7fc20b400640 1 -- 172.21.15.32:0/3009279021 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc20c15fad0 con 0x7fc20c06e930 2024-08-03T20:49:31.721 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:31.718+0000 7fc20be00640 1 --2- 172.21.15.32:0/3009279021 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fc20c0747b0 0x7fc20c0af100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-03T20:49:31.721 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:31.719+0000 7fc20b400640 1 --2- 172.21.15.32:0/3009279021 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc20c06e930 0x7fc20c0b2fd0 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7fc1fc00d740 tx=0x7fc1fc00dc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:49:31.722 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:31.719+0000 7fc203400640 1 -- 172.21.15.32:0/3009279021 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc1fc017070 con 0x7fc20c06e930 2024-08-03T20:49:31.722 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:31.719+0000 7fc203400640 1 -- 172.21.15.32:0/3009279021 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fc1fc009d20 con 0x7fc20c06e930 2024-08-03T20:49:31.722 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:31.719+0000 7fc203400640 1 -- 172.21.15.32:0/3009279021 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc1fc0106c0 con 0x7fc20c06e930 2024-08-03T20:49:31.722 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:31.719+0000 7fc212800640 1 -- 172.21.15.32:0/3009279021 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc20c0ae060 con 0x7fc20c06e930 2024-08-03T20:49:31.722 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:31.719+0000 7fc212800640 1 -- 172.21.15.32:0/3009279021 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc20c0ae620 con 0x7fc20c06e930 2024-08-03T20:49:31.722 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:31.720+0000 7fc212800640 1 -- 172.21.15.32:0/3009279021 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc20c072ac0 con 0x7fc20c06e930 2024-08-03T20:49:31.722 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:31.721+0000 7fc203400640 1 -- 172.21.15.32:0/3009279021 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7fc1fc01a070 con 0x7fc20c06e930 2024-08-03T20:49:31.723 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:31.722+0000 7fc203400640 1 --2- 172.21.15.32:0/3009279021 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fc1ec077040 0x7fc1ec079550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:49:31.724 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:31.722+0000 7fc203400640 1 -- 172.21.15.32:0/3009279021 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(36..36 src has 1..36) ==== 5029+0+0 (secure 0 0 0) 0x7fc1fc098720 con 0x7fc20c06e930 2024-08-03T20:49:31.724 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:31.723+0000 7fc20aa00640 1 --2- 172.21.15.32:0/3009279021 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fc1ec077040 0x7fc1ec079550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:49:31.725 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:31.723+0000 7fc20aa00640 1 --2- 172.21.15.32:0/3009279021 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fc1ec077040 0x7fc1ec079550 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fc1f8033000 tx=0x7fc1f8004730 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:49:31.728 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:31.727+0000 7fc203400640 1 -- 172.21.15.32:0/3009279021 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7fc20c072ac0 con 0x7fc20c06e930 2024-08-03T20:49:31.937 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:31.936+0000 7fc212800640 1 -- 172.21.15.32:0/3009279021 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fc20c0cd580 con 0x7fc20c06e930 2024-08-03T20:49:31.938 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:31.937+0000 7fc203400640 1 -- 172.21.15.32:0/3009279021 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v36) ==== 74+0+13088 (secure 0 0 0) 0x7fc1fc0614d0 con 0x7fc20c06e930 2024-08-03T20:49:31.938 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-03T20:49:31.938 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":36,"fsid":"40ab141a-01d7-49b9-ae3c-2edec9876e95","created":"2024-08-03T20:47:14.957362+0000","modified":"2024-08-03T20:49:29.439803+0000","last_up_change":"2024-08-03T20:47:28.874451+0000","last_in_change":"2024-08-03T20:47:20.079442+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-08-03T20:47:28.261300+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5a83e759-9035-438b-889a-0a6af1f0a0d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6816","nonce":3066156116},{"type":"v1","addr":"172.21.15.32:6817","nonce":3066156116}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6818","nonce":3066156116},{"type":"v1","addr":"172.21.15.32:6819","nonce":3066156116}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6822","nonce":3066156116},{"type":"v1","addr":"172.21.15.32:6823","nonce":3066156116}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6820","nonce":3066156116},{"type":"v1","addr":"172.21.15.32:6821","nonce":3066156116}]},"public_addr":"172.21.15.32:6817/3066156116","cluster_addr":"172.21.15.32:6819/3066156116","heartbeat_back_addr":"172.21.15.32:6823/3066156116","heartbeat_front_addr":"172.21.15.32:6821/3066156116","state":["exists","up"]},{"osd":1,"uuid":"d7c5c210-da4f-49f9-b9e7-1dfb02daa4bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6824","nonce":1551241759},{"type":"v1","addr":"172.21.15.32:6825","nonce":1551241759}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6826","nonce":1551241759},{"type":"v1","addr":"172.21.15.32:6827","nonce":1551241759}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6830","nonce":1551241759},{"type":"v1","addr":"172.21.15.32:6831","nonce":1551241759}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6828","nonce":1551241759},{"type":"v1","addr":"172.21.15.32:6829","nonce":1551241759}]},"public_addr":"172.21.15.32:6825/1551241759","cluster_addr":"172.21.15.32:6827/1551241759","heartbeat_back_addr":"172.21.15.32:6831/1551241759","heartbeat_front_addr":"172.21.15.32:6829/1551241759","state":["exists","up"]},{"osd":2,"uuid":"57f2e6de-a201-45c2-8981-8e56a4875ebd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6808","nonce":3806708944},{"type":"v1","addr":"172.21.15.32:6809","nonce":3806708944}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6810","nonce":3806708944},{"type":"v1","addr":"172.21.15.32:6811","nonce":3806708944}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6814","nonce":3806708944},{"type":"v1","addr":"172.21.15.32:6815","nonce":3806708944}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6812","nonce":3806708944},{"type":"v1","addr":"172.21.15.32:6813","nonce":3806708944}]},"public_addr":"172.21.15.32:6809/3806708944","cluster_addr":"172.21.15.32:6811/3806708944","heartbeat_back_addr":"172.21.15.32:6815/3806708944","heartbeat_front_addr":"172.21.15.32:6813/3806708944","state":["exists","up"]},{"osd":3,"uuid":"6439a280-da45-4148-8130-f16323935ff3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6800","nonce":9088303},{"type":"v1","addr":"172.21.15.32:6801","nonce":9088303}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6802","nonce":9088303},{"type":"v1","addr":"172.21.15.32:6803","nonce":9088303}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6806","nonce":9088303},{"type":"v1","addr":"172.21.15.32:6807","nonce":9088303}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6804","nonce":9088303},{"type":"v1","addr":"172.21.15.32:6805","nonce":9088303}]},"public_addr":"172.21.15.32:6801/9088303","cluster_addr":"172.21.15.32:6803/9088303","heartbeat_back_addr":"172.21.15.32:6807/9088303","heartbeat_front_addr":"172.21.15.32:6805/9088303","state":["exists","up"]},{"osd":4,"uuid":"4f902eff-1e18-418d-82c9-2151c69524fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6802","nonce":864752275},{"type":"v1","addr":"172.21.15.118:6803","nonce":864752275}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6804","nonce":864752275},{"type":"v1","addr":"172.21.15.118:6805","nonce":864752275}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6808","nonce":864752275},{"type":"v1","addr":"172.21.15.118:6809","nonce":864752275}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6806","nonce":864752275},{"type":"v1","addr":"172.21.15.118:6807","nonce":864752275}]},"public_addr":"172.21.15.118:6803/864752275","cluster_addr":"172.21.15.118:6805/864752275","heartbeat_back_addr":"172.21.15.118:6809/864752275","heartbeat_front_addr":"172.21.15.118:6807/864752275","state":["exists","up"]},{"osd":5,"uuid":"bfd9feac-dd06-458c-91b9-af7f175f612c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6810","nonce":2724891770},{"type":"v1","addr":"172.21.15.118:6811","nonce":2724891770}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6812","nonce":2724891770},{"type":"v1","addr":"172.21.15.118:6813","nonce":2724891770}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6816","nonce":2724891770},{"type":"v1","addr":"172.21.15.118:6817","nonce":2724891770}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6814","nonce":2724891770},{"type":"v1","addr":"172.21.15.118:6815","nonce":2724891770}]},"public_addr":"172.21.15.118:6811/2724891770","cluster_addr":"172.21.15.118:6813/2724891770","heartbeat_back_addr":"172.21.15.118:6817/2724891770","heartbeat_front_addr":"172.21.15.118:6815/2724891770","state":["exists","up"]},{"osd":6,"uuid":"5ed5a279-9d8e-416d-bbc7-7edd3e6d5705","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6825","nonce":801947386},{"type":"v1","addr":"172.21.15.118:6827","nonce":801947386}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6828","nonce":801947386},{"type":"v1","addr":"172.21.15.118:6829","nonce":801947386}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6832","nonce":801947386},{"type":"v1","addr":"172.21.15.118:6833","nonce":801947386}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6830","nonce":801947386},{"type":"v1","addr":"172.21.15.118:6831","nonce":801947386}]},"public_addr":"172.21.15.118:6827/801947386","cluster_addr":"172.21.15.118:6829/801947386","heartbeat_back_addr":"172.21.15.118:6833/801947386","heartbeat_front_addr":"172.21.15.118:6831/801947386","state":["exists","up"]},{"osd":7,"uuid":"f2086c47-875e-4451-a436-2008d1ea16af","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6818","nonce":653000048},{"type":"v1","addr":"172.21.15.118:6819","nonce":653000048}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6820","nonce":653000048},{"type":"v1","addr":"172.21.15.118:6821","nonce":653000048}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6824","nonce":653000048},{"type":"v1","addr":"172.21.15.118:6826","nonce":653000048}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6822","nonce":653000048},{"type":"v1","addr":"172.21.15.118:6823","nonce":653000048}]},"public_addr":"172.21.15.118:6819/653000048","cluster_addr":"172.21.15.118:6821/653000048","heartbeat_back_addr":"172.21.15.118:6826/653000048","heartbeat_front_addr":"172.21.15.118:6823/653000048","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-03T20:47:25.491994+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-03T20:47:25.587358+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-03T20:47:25.454175+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-03T20:47:25.377734+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-03T20:47:25.412351+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-03T20:47:25.499815+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-03T20:47:25.575044+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-03T20:47:25.552383+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.32:6835/211300533":"2024-08-04T20:49:26.333158+0000","172.21.15.32:6835/1186001746":"2024-08-04T20:48:38.076677+0000","172.21.15.32:6832/211300533":"2024-08-04T20:49:26.333158+0000","172.21.15.32:6832/1186001746":"2024-08-04T20:48:38.076677+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-08-03T20:49:31.941 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:31.939+0000 7fc212800640 1 -- 172.21.15.32:0/3009279021 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fc1ec077040 msgr2=0x7fc1ec079550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:49:31.941 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:31.939+0000 7fc212800640 1 --2- 172.21.15.32:0/3009279021 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fc1ec077040 0x7fc1ec079550 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fc1f8033000 tx=0x7fc1f8004730 comp rx=0 tx=0).stop 2024-08-03T20:49:31.941 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:31.939+0000 7fc212800640 1 -- 172.21.15.32:0/3009279021 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc20c06e930 msgr2=0x7fc20c0b2fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:49:31.941 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:31.939+0000 7fc212800640 1 --2- 172.21.15.32:0/3009279021 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc20c06e930 0x7fc20c0b2fd0 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7fc1fc00d740 tx=0x7fc1fc00dc00 comp rx=0 tx=0).stop 2024-08-03T20:49:31.941 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:31.939+0000 7fc212800640 1 -- 172.21.15.32:0/3009279021 shutdown_connections 2024-08-03T20:49:31.941 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:31.939+0000 7fc212800640 1 --2- 172.21.15.32:0/3009279021 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fc1ec077040 0x7fc1ec079550 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:49:31.941 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:31.939+0000 7fc212800640 1 --2- 172.21.15.32:0/3009279021 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc20c073e60 0x7fc20c15f590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:49:31.941 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:31.939+0000 7fc212800640 1 --2- 172.21.15.32:0/3009279021 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fc20c0747b0 0x7fc20c0af100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:49:31.941 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:31.939+0000 7fc212800640 1 --2- 172.21.15.32:0/3009279021 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc20c06e930 0x7fc20c0b2fd0 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:49:31.941 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:31.939+0000 7fc212800640 1 -- 172.21.15.32:0/3009279021 >> 172.21.15.32:0/3009279021 conn(0x7fc20c0b1bd0 msgr2=0x7fc20c0b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:49:31.942 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:31.939+0000 7fc212800640 1 -- 172.21.15.32:0/3009279021 shutdown_connections 2024-08-03T20:49:31.942 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:31.939+0000 7fc212800640 1 -- 172.21.15.32:0/3009279021 wait complete. 2024-08-03T20:49:31.968 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-08-03T20:47:28.261300+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-08-03T20:49:31.968 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-08-03T20:49:32.201 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:32.199+0000 7fb864800640 1 -- 172.21.15.32:0/2939733639 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb844005250 msgr2=0x7fb84400b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:49:32.201 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:32.199+0000 7fb864800640 1 --2- 172.21.15.32:0/2939733639 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb844005250 0x7fb84400b9c0 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7fb858001dc0 tx=0x7fb844005830 comp rx=0 tx=0).stop 2024-08-03T20:49:32.202 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:32.199+0000 7fb864800640 1 -- 172.21.15.32:0/2939733639 shutdown_connections 2024-08-03T20:49:32.202 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:32.199+0000 7fb864800640 1 --2- 172.21.15.32:0/2939733639 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb844005250 0x7fb84400b9c0 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:49:32.202 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:32.199+0000 7fb864800640 1 -- 172.21.15.32:0/2939733639 >> 172.21.15.32:0/2939733639 conn(0x7fb85c0b1bd0 msgr2=0x7fb85c0b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:49:32.202 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:32.199+0000 7fb864800640 1 -- 172.21.15.32:0/2939733639 shutdown_connections 2024-08-03T20:49:32.202 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:32.199+0000 7fb864800640 1 -- 172.21.15.32:0/2939733639 wait complete. 2024-08-03T20:49:32.203 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:32.202+0000 7fb864800640 1 Processor -- start 2024-08-03T20:49:32.203 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:32.202+0000 7fb864800640 1 -- start start 2024-08-03T20:49:32.205 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:32.203+0000 7fb864800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb844005250 0x7fb85c0c9250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:49:32.206 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:32.203+0000 7fb864800640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fb85c155ef0 0x7fb85c147970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:49:32.206 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:32.203+0000 7fb864800640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb85c153ed0 0x7fb85c1542e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:49:32.206 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:32.203+0000 7fb864800640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fb85c06f0f0 con 0x7fb844005250 2024-08-03T20:49:32.206 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:32.203+0000 7fb864800640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7fb85c06ef70 con 0x7fb85c153ed0 2024-08-03T20:49:32.206 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:32.203+0000 7fb864800640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7fb85c0b4290 con 0x7fb85c155ef0 2024-08-03T20:49:32.206 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:32.204+0000 7fb861e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb844005250 0x7fb85c0c9250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:49:32.206 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:32.204+0000 7fb861e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb844005250 0x7fb85c0c9250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:54272/0 (socket says 172.21.15.32:54272) 2024-08-03T20:49:32.206 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:32.204+0000 7fb861e00640 1 -- 172.21.15.32:0/852354802 learned_addr learned my addr 172.21.15.32:0/852354802 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:49:32.206 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:32.204+0000 7fb861400640 1 --2- 172.21.15.32:0/852354802 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fb85c155ef0 0x7fb85c147970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:49:32.206 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:32.204+0000 7fb862800640 1 --2- 172.21.15.32:0/852354802 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb85c153ed0 0x7fb85c1542e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:49:32.206 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:32.204+0000 7fb861e00640 1 -- 172.21.15.32:0/852354802 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fb85c155ef0 msgr2=0x7fb85c147970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:49:32.207 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:32.204+0000 7fb861e00640 1 --2- 172.21.15.32:0/852354802 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fb85c155ef0 0x7fb85c147970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:49:32.207 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:32.204+0000 7fb861e00640 1 -- 172.21.15.32:0/852354802 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb85c153ed0 msgr2=0x7fb85c1542e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:49:32.207 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:32.204+0000 7fb861e00640 1 --2- 172.21.15.32:0/852354802 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb85c153ed0 0x7fb85c1542e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:49:32.207 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:32.204+0000 7fb861e00640 1 -- 172.21.15.32:0/852354802 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb85c154820 con 0x7fb844005250 2024-08-03T20:49:32.207 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:32.204+0000 7fb862800640 1 --2- 172.21.15.32:0/852354802 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb85c153ed0 0x7fb85c1542e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-03T20:49:32.207 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:32.204+0000 7fb861e00640 1 --2- 172.21.15.32:0/852354802 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb844005250 0x7fb85c0c9250 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7fb850006680 tx=0x7fb8500039c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:49:32.208 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:32.205+0000 7fb856000640 1 -- 172.21.15.32:0/852354802 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb850015070 con 0x7fb844005250 2024-08-03T20:49:32.208 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:32.205+0000 7fb856000640 1 -- 172.21.15.32:0/852354802 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fb850011900 con 0x7fb844005250 2024-08-03T20:49:32.208 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:32.205+0000 7fb856000640 1 -- 172.21.15.32:0/852354802 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb850011cf0 con 0x7fb844005250 2024-08-03T20:49:32.208 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:32.205+0000 7fb864800640 1 -- 172.21.15.32:0/852354802 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb85c151070 con 0x7fb844005250 2024-08-03T20:49:32.208 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:32.205+0000 7fb864800640 1 -- 172.21.15.32:0/852354802 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb85c1f9520 con 0x7fb844005250 2024-08-03T20:49:32.208 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:32.206+0000 7fb864800640 1 -- 172.21.15.32:0/852354802 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb85c0ced30 con 0x7fb844005250 2024-08-03T20:49:32.209 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:32.207+0000 7fb856000640 1 -- 172.21.15.32:0/852354802 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7fb850018070 con 0x7fb844005250 2024-08-03T20:49:32.210 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:32.208+0000 7fb856000640 1 --2- 172.21.15.32:0/852354802 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fb840077010 0x7fb840079520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:49:32.210 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:32.208+0000 7fb856000640 1 -- 172.21.15.32:0/852354802 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(36..36 src has 1..36) ==== 5029+0+0 (secure 0 0 0) 0x7fb850099340 con 0x7fb844005250 2024-08-03T20:49:32.210 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:32.209+0000 7fb861400640 1 --2- 172.21.15.32:0/852354802 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fb840077010 0x7fb840079520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:49:32.211 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:32.210+0000 7fb861400640 1 --2- 172.21.15.32:0/852354802 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fb840077010 0x7fb840079520 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fb85c15f830 tx=0x7fb84c005f90 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:49:32.214 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:32.213+0000 7fb856000640 1 -- 172.21.15.32:0/852354802 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7fb85c154820 con 0x7fb844005250 2024-08-03T20:49:32.416 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:32.415+0000 7fb864800640 1 -- 172.21.15.32:0/852354802 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fb85c0b33c0 con 0x7fb844005250 2024-08-03T20:49:32.417 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:32.416+0000 7fb856000640 1 -- 172.21.15.32:0/852354802 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v36) ==== 93+0+10 (secure 0 0 0) 0x7fb8500627e0 con 0x7fb844005250 2024-08-03T20:49:32.417 INFO:teuthology.orchestra.run.smithi032.stdout:pg_num: 1 2024-08-03T20:49:32.420 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:32.418+0000 7fb864800640 1 -- 172.21.15.32:0/852354802 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fb840077010 msgr2=0x7fb840079520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:49:32.420 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:32.418+0000 7fb864800640 1 --2- 172.21.15.32:0/852354802 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fb840077010 0x7fb840079520 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fb85c15f830 tx=0x7fb84c005f90 comp rx=0 tx=0).stop 2024-08-03T20:49:32.420 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:32.418+0000 7fb864800640 1 -- 172.21.15.32:0/852354802 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb844005250 msgr2=0x7fb85c0c9250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:49:32.420 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:32.418+0000 7fb864800640 1 --2- 172.21.15.32:0/852354802 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb844005250 0x7fb85c0c9250 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7fb850006680 tx=0x7fb8500039c0 comp rx=0 tx=0).stop 2024-08-03T20:49:32.420 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:32.418+0000 7fb864800640 1 -- 172.21.15.32:0/852354802 shutdown_connections 2024-08-03T20:49:32.420 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:32.418+0000 7fb864800640 1 --2- 172.21.15.32:0/852354802 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fb840077010 0x7fb840079520 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:49:32.420 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:32.418+0000 7fb864800640 1 --2- 172.21.15.32:0/852354802 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb85c153ed0 0x7fb85c1542e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:49:32.420 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:32.418+0000 7fb864800640 1 --2- 172.21.15.32:0/852354802 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fb85c155ef0 0x7fb85c147970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:49:32.420 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:32.418+0000 7fb864800640 1 --2- 172.21.15.32:0/852354802 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb844005250 0x7fb85c0c9250 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:49:32.420 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:32.418+0000 7fb864800640 1 -- 172.21.15.32:0/852354802 >> 172.21.15.32:0/852354802 conn(0x7fb85c0b1bd0 msgr2=0x7fb85c15dca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:49:32.420 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:32.418+0000 7fb864800640 1 -- 172.21.15.32:0/852354802 shutdown_connections 2024-08-03T20:49:32.421 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:32.418+0000 7fb864800640 1 -- 172.21.15.32:0/852354802 wait complete. 2024-08-03T20:49:32.443 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_damage.TestDamage.test_dentry_first_existing' 2024-08-03T20:49:32.674 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:32.672+0000 7f780b000640 1 -- 172.21.15.32:0/1000007728 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f78040b49d0 msgr2=0x7f78040b4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:49:32.674 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:32.672+0000 7f780b000640 1 --2- 172.21.15.32:0/1000007728 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f78040b49d0 0x7f78040b4de0 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f78040d2150 tx=0x7f77fc02a7a0 comp rx=0 tx=0).stop 2024-08-03T20:49:32.675 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:32.672+0000 7f780b000640 1 -- 172.21.15.32:0/1000007728 shutdown_connections 2024-08-03T20:49:32.675 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:32.673+0000 7f780b000640 1 --2- 172.21.15.32:0/1000007728 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f78040b49d0 0x7f78040b4de0 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:49:32.675 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:32.673+0000 7f780b000640 1 --2- 172.21.15.32:0/1000007728 >> v2:172.21.15.118:3300/0 conn(0x7f7804070f90 0x7f78040713a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:49:32.675 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:32.673+0000 7f780b000640 1 -- 172.21.15.32:0/1000007728 >> 172.21.15.32:0/1000007728 conn(0x7f78040b34b0 msgr2=0x7f78040d1050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:49:32.675 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:32.673+0000 7f780b000640 1 -- 172.21.15.32:0/1000007728 shutdown_connections 2024-08-03T20:49:32.675 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:32.673+0000 7f780b000640 1 -- 172.21.15.32:0/1000007728 wait complete. 2024-08-03T20:49:32.676 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:32.675+0000 7f780b000640 1 Processor -- start 2024-08-03T20:49:32.676 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:32.675+0000 7f780b000640 1 -- start start 2024-08-03T20:49:32.680 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:32.676+0000 7f780b000640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f7804070f90 0x7f78040afbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:49:32.681 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:32.677+0000 7f780b000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f78040b49d0 0x7f7804143320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:49:32.681 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:32.677+0000 7f780b000640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f78041555c0 0x7f78040adaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:49:32.681 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:32.677+0000 7f780b000640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f780406cb60 con 0x7f78040b49d0 2024-08-03T20:49:32.681 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:32.677+0000 7f780b000640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f780406c9e0 con 0x7f7804070f90 2024-08-03T20:49:32.681 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:32.677+0000 7f780b000640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7f780406cce0 con 0x7f78041555c0 2024-08-03T20:49:32.681 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:32.677+0000 7f7803400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f78040b49d0 0x7f7804143320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:49:32.681 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:32.677+0000 7f7803400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f78040b49d0 0x7f7804143320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:54290/0 (socket says 172.21.15.32:54290) 2024-08-03T20:49:32.681 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:32.677+0000 7f7803400640 1 -- 172.21.15.32:0/4250581436 learned_addr learned my addr 172.21.15.32:0/4250581436 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:49:32.681 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:32.677+0000 7f7803e00640 1 --2- 172.21.15.32:0/4250581436 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f7804070f90 0x7f78040afbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:49:32.681 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:32.677+0000 7f7809000640 1 --2- 172.21.15.32:0/4250581436 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f78041555c0 0x7f78040adaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:49:32.681 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:32.677+0000 7f7803400640 1 -- 172.21.15.32:0/4250581436 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f78041555c0 msgr2=0x7f78040adaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:49:32.681 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:32.677+0000 7f7803400640 1 --2- 172.21.15.32:0/4250581436 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f78041555c0 0x7f78040adaa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:49:32.682 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:32.677+0000 7f7803400640 1 -- 172.21.15.32:0/4250581436 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f7804070f90 msgr2=0x7f78040afbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:49:32.682 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:32.677+0000 7f7803400640 1 --2- 172.21.15.32:0/4250581436 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f7804070f90 0x7f78040afbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:49:32.682 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:32.677+0000 7f7803400640 1 -- 172.21.15.32:0/4250581436 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7804142280 con 0x7f78040b49d0 2024-08-03T20:49:32.682 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:32.677+0000 7f7803e00640 1 --2- 172.21.15.32:0/4250581436 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f7804070f90 0x7f78040afbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-03T20:49:32.682 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:32.677+0000 7f7809000640 1 --2- 172.21.15.32:0/4250581436 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f78041555c0 0x7f78040adaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-03T20:49:32.682 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:32.678+0000 7f7803400640 1 --2- 172.21.15.32:0/4250581436 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f78040b49d0 0x7f7804143320 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f77f000d6d0 tx=0x7f77f000db90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:49:32.682 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:32.678+0000 7f77fbe00640 1 -- 172.21.15.32:0/4250581436 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77f0017070 con 0x7f78040b49d0 2024-08-03T20:49:32.682 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:32.678+0000 7f77fbe00640 1 -- 172.21.15.32:0/4250581436 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f77f0004310 con 0x7f78040b49d0 2024-08-03T20:49:32.682 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:32.678+0000 7f77fbe00640 1 -- 172.21.15.32:0/4250581436 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77f0009a80 con 0x7f78040b49d0 2024-08-03T20:49:32.682 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:32.678+0000 7f780b000640 1 -- 172.21.15.32:0/4250581436 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7804142560 con 0x7f78040b49d0 2024-08-03T20:49:32.682 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:32.678+0000 7f780b000640 1 -- 172.21.15.32:0/4250581436 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7804142b20 con 0x7f78040b49d0 2024-08-03T20:49:32.682 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:32.679+0000 7f780b000640 1 -- 172.21.15.32:0/4250581436 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7804141ff0 con 0x7f78040b49d0 2024-08-03T20:49:32.683 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:32.680+0000 7f77fbe00640 1 -- 172.21.15.32:0/4250581436 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f77f0009c20 con 0x7f78040b49d0 2024-08-03T20:49:32.683 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:32.681+0000 7f77fbe00640 1 --2- 172.21.15.32:0/4250581436 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f77e4077010 0x7f77e4079520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:49:32.683 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:32.681+0000 7f77fbe00640 1 -- 172.21.15.32:0/4250581436 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(36..36 src has 1..36) ==== 5029+0+0 (secure 0 0 0) 0x7f77f0098710 con 0x7f78040b49d0 2024-08-03T20:49:32.683 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:32.682+0000 7f7803e00640 1 --2- 172.21.15.32:0/4250581436 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f77e4077010 0x7f77e4079520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:49:32.684 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:32.683+0000 7f7803e00640 1 --2- 172.21.15.32:0/4250581436 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f77e4077010 0x7f77e4079520 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f77f40016a0 tx=0x7f77f4009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:49:32.687 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:32.685+0000 7f77fbe00640 1 -- 172.21.15.32:0/4250581436 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7f77f0061b10 con 0x7f78040b49d0 2024-08-03T20:49:32.969 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:32.968+0000 7f780b000640 1 -- 172.21.15.32:0/4250581436 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_damage.TestDamage.test_dentry_first_existing"]} v 0) -- 0x7f780406d200 con 0x7f78040b49d0 2024-08-03T20:49:33.460 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:33.459+0000 7f77fbe00640 1 -- 172.21.15.32:0/4250581436 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_damage.TestDamage.test_dentry_first_existing"]}]=0 v118) ==== 144+0+0 (secure 0 0 0) 0x7f77f0061410 con 0x7f78040b49d0 2024-08-03T20:49:33.464 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:33.461+0000 7f780b000640 1 -- 172.21.15.32:0/4250581436 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f77e4077010 msgr2=0x7f77e4079520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:49:33.464 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:33.461+0000 7f780b000640 1 --2- 172.21.15.32:0/4250581436 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f77e4077010 0x7f77e4079520 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f77f40016a0 tx=0x7f77f4009040 comp rx=0 tx=0).stop 2024-08-03T20:49:33.465 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:33.461+0000 7f780b000640 1 -- 172.21.15.32:0/4250581436 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f78040b49d0 msgr2=0x7f7804143320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:49:33.465 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:33.461+0000 7f780b000640 1 --2- 172.21.15.32:0/4250581436 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f78040b49d0 0x7f7804143320 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f77f000d6d0 tx=0x7f77f000db90 comp rx=0 tx=0).stop 2024-08-03T20:49:33.465 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:33.462+0000 7f780b000640 1 -- 172.21.15.32:0/4250581436 shutdown_connections 2024-08-03T20:49:33.465 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:33.462+0000 7f780b000640 1 --2- 172.21.15.32:0/4250581436 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f77e4077010 0x7f77e4079520 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:49:33.465 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:33.462+0000 7f780b000640 1 --2- 172.21.15.32:0/4250581436 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f78041555c0 0x7f78040adaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:49:33.466 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:33.462+0000 7f780b000640 1 --2- 172.21.15.32:0/4250581436 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f78040b49d0 0x7f7804143320 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:49:33.466 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:33.462+0000 7f780b000640 1 --2- 172.21.15.32:0/4250581436 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f7804070f90 0x7f78040afbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:49:33.466 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:33.462+0000 7f780b000640 1 -- 172.21.15.32:0/4250581436 >> 172.21.15.32:0/4250581436 conn(0x7f78040b34b0 msgr2=0x7f78041528b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:49:33.466 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:33.462+0000 7f780b000640 1 -- 172.21.15.32:0/4250581436 shutdown_connections 2024-08-03T20:49:33.466 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:33.462+0000 7f780b000640 1 -- 172.21.15.32:0/4250581436 wait complete. 2024-08-03T20:49:33.489 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config dump 2024-08-03T20:49:33.720 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:33.718+0000 7f0363000640 1 -- 172.21.15.32:0/520000419 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0340003c10 msgr2=0x7f0340004060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:49:33.720 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:33.718+0000 7f0363000640 1 --2- 172.21.15.32:0/520000419 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0340003c10 0x7f0340004060 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f034c006fa0 tx=0x7f0340005830 comp rx=0 tx=0).stop 2024-08-03T20:49:33.721 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:33.718+0000 7f0361000640 1 -- 172.21.15.32:0/520000419 reap_dead start 2024-08-03T20:49:33.721 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:33.719+0000 7f0363000640 1 -- 172.21.15.32:0/520000419 shutdown_connections 2024-08-03T20:49:33.721 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:33.719+0000 7f0363000640 1 -- 172.21.15.32:0/520000419 >> 172.21.15.32:0/520000419 conn(0x7f035c0b1bd0 msgr2=0x7f035c0b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:49:33.721 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:33.719+0000 7f0363000640 1 -- 172.21.15.32:0/520000419 shutdown_connections 2024-08-03T20:49:33.721 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:33.719+0000 7f0363000640 1 -- 172.21.15.32:0/520000419 wait complete. 2024-08-03T20:49:33.722 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:33.721+0000 7f0363000640 1 Processor -- start 2024-08-03T20:49:33.722 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:33.721+0000 7f0363000640 1 -- start start 2024-08-03T20:49:33.725 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:33.722+0000 7f0363000640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f0340003c10 0x7f035c0b4c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:49:33.725 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:33.722+0000 7f0363000640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f035c0b2fd0 0x7f035c0af310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:49:33.725 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:33.722+0000 7f0363000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f035c15df40 0x7f035c15e350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:49:33.726 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:33.722+0000 7f0363000640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f035c0cf840 con 0x7f035c15df40 2024-08-03T20:49:33.726 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:33.722+0000 7f0363000640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f035c145f30 con 0x7f0340003c10 2024-08-03T20:49:33.726 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:33.722+0000 7f0363000640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7f035c0cf9c0 con 0x7f035c0b2fd0 2024-08-03T20:49:33.726 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:33.723+0000 7f0361000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f035c15df40 0x7f035c15e350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:49:33.726 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:33.723+0000 7f0361000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f035c15df40 0x7f035c15e350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:54298/0 (socket says 172.21.15.32:54298) 2024-08-03T20:49:33.726 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:33.723+0000 7f0361000640 1 -- 172.21.15.32:0/729284124 learned_addr learned my addr 172.21.15.32:0/729284124 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:49:33.726 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:33.723+0000 7f035be00640 1 --2- 172.21.15.32:0/729284124 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f0340003c10 0x7f035c0b4c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:49:33.726 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:33.723+0000 7f035b400640 1 --2- 172.21.15.32:0/729284124 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f035c0b2fd0 0x7f035c0af310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:49:33.726 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:33.723+0000 7f0361000640 1 -- 172.21.15.32:0/729284124 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f035c0b2fd0 msgr2=0x7f035c0af310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:49:33.726 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:33.723+0000 7f0361000640 1 --2- 172.21.15.32:0/729284124 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f035c0b2fd0 0x7f035c0af310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:49:33.726 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:33.723+0000 7f0361000640 1 -- 172.21.15.32:0/729284124 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f0340003c10 msgr2=0x7f035c0b4c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:49:33.726 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:33.723+0000 7f0361000640 1 --2- 172.21.15.32:0/729284124 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f0340003c10 0x7f035c0b4c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:49:33.727 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:33.723+0000 7f0361000640 1 -- 172.21.15.32:0/729284124 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f035c0af850 con 0x7f035c15df40 2024-08-03T20:49:33.727 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:33.723+0000 7f035b400640 1 --2- 172.21.15.32:0/729284124 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f035c0b2fd0 0x7f035c0af310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-03T20:49:33.727 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:33.723+0000 7f0361000640 1 --2- 172.21.15.32:0/729284124 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f035c15df40 0x7f035c15e350 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f035c072db0 tx=0x7f0354001b80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:49:33.727 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:33.724+0000 7f0353e00640 1 -- 172.21.15.32:0/729284124 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0354017070 con 0x7f035c15df40 2024-08-03T20:49:33.727 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:33.724+0000 7f0353e00640 1 -- 172.21.15.32:0/729284124 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f03540109a0 con 0x7f035c15df40 2024-08-03T20:49:33.727 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:33.724+0000 7f0353e00640 1 -- 172.21.15.32:0/729284124 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0354010df0 con 0x7f035c15df40 2024-08-03T20:49:33.727 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:33.724+0000 7f0363000640 1 -- 172.21.15.32:0/729284124 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f035c15e890 con 0x7f035c15df40 2024-08-03T20:49:33.727 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:33.724+0000 7f0363000640 1 -- 172.21.15.32:0/729284124 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f035c1f92f0 con 0x7f035c15df40 2024-08-03T20:49:33.728 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:33.725+0000 7f0363000640 1 -- 172.21.15.32:0/729284124 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f035c0ce910 con 0x7f035c15df40 2024-08-03T20:49:33.728 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:33.727+0000 7f0353e00640 1 -- 172.21.15.32:0/729284124 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f035401a070 con 0x7f035c15df40 2024-08-03T20:49:33.729 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:33.727+0000 7f0353e00640 1 --2- 172.21.15.32:0/729284124 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f033c077010 0x7f033c079520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:49:33.729 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:33.728+0000 7f0353e00640 1 -- 172.21.15.32:0/729284124 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(36..36 src has 1..36) ==== 5029+0+0 (secure 0 0 0) 0x7f03540984f0 con 0x7f035c15df40 2024-08-03T20:49:33.730 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:33.728+0000 7f035be00640 1 --2- 172.21.15.32:0/729284124 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f033c077010 0x7f033c079520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:49:33.730 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:33.729+0000 7f035be00640 1 --2- 172.21.15.32:0/729284124 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f033c077010 0x7f033c079520 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f034c033000 tx=0x7f034c02d000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:49:33.734 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:33.732+0000 7f0353e00640 1 -- 172.21.15.32:0/729284124 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7f035c0ce910 con 0x7f035c15df40 2024-08-03T20:49:33.941 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:33.939+0000 7f0363000640 1 -- 172.21.15.32:0/729284124 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "config dump"} v 0) -- 0x7f035c0748f0 con 0x7f035c15df40 2024-08-03T20:49:33.941 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:33.940+0000 7f0353e00640 1 -- 172.21.15.32:0/729284124 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "config dump"}]=0 v29) ==== 59+0+1998 (secure 0 0 0) 0x7f03540612a0 con 0x7f035c15df40 2024-08-03T20:49:33.942 INFO:teuthology.orchestra.run.smithi032.stdout:WHO MASK LEVEL OPTION VALUE RO 2024-08-03T20:49:33.942 INFO:teuthology.orchestra.run.smithi032.stdout:mon advanced mon_op_complaint_time 120 2024-08-03T20:49:33.942 INFO:teuthology.orchestra.run.smithi032.stdout:mon dev mon_warn_on_pool_no_app false 2024-08-03T20:49:33.942 INFO:teuthology.orchestra.run.smithi032.stdout:osd advanced osd_op_complaint_time 180.000000 2024-08-03T20:49:33.942 INFO:teuthology.orchestra.run.smithi032.stdout:osd.0 basic osd_mclock_max_capacity_iops_ssd 4313.565729 2024-08-03T20:49:33.942 INFO:teuthology.orchestra.run.smithi032.stdout:osd.1 basic osd_mclock_max_capacity_iops_ssd 4272.517979 2024-08-03T20:49:33.942 INFO:teuthology.orchestra.run.smithi032.stdout:osd.2 basic osd_mclock_max_capacity_iops_ssd 4131.201546 2024-08-03T20:49:33.942 INFO:teuthology.orchestra.run.smithi032.stdout:osd.3 basic osd_mclock_max_capacity_iops_ssd 4155.765914 2024-08-03T20:49:33.942 INFO:teuthology.orchestra.run.smithi032.stdout:osd.4 basic osd_mclock_max_capacity_iops_ssd 5744.667532 2024-08-03T20:49:33.942 INFO:teuthology.orchestra.run.smithi032.stdout:osd.5 basic osd_mclock_max_capacity_iops_ssd 3764.678226 2024-08-03T20:49:33.942 INFO:teuthology.orchestra.run.smithi032.stdout:osd.6 basic osd_mclock_max_capacity_iops_ssd 3927.989098 2024-08-03T20:49:33.943 INFO:teuthology.orchestra.run.smithi032.stdout:osd.7 basic osd_mclock_max_capacity_iops_ssd 3760.968870 2024-08-03T20:49:33.943 INFO:teuthology.orchestra.run.smithi032.stdout:mds advanced debug_mds 20/20 2024-08-03T20:49:33.943 INFO:teuthology.orchestra.run.smithi032.stdout:mds advanced debug_mds_balancer 20/20 2024-08-03T20:49:33.943 INFO:teuthology.orchestra.run.smithi032.stdout:mds advanced debug_ms 2/2 2024-08-03T20:49:33.943 INFO:teuthology.orchestra.run.smithi032.stdout:mds dev mds_debug_frag true 2024-08-03T20:49:33.943 INFO:teuthology.orchestra.run.smithi032.stdout:mds dev mds_debug_scatterstat true 2024-08-03T20:49:33.943 INFO:teuthology.orchestra.run.smithi032.stdout:mds advanced mds_op_complaint_time 180.000000 2024-08-03T20:49:33.943 INFO:teuthology.orchestra.run.smithi032.stdout:mds dev mds_verify_scatter true 2024-08-03T20:49:33.943 INFO:teuthology.orchestra.run.smithi032.stdout:mds advanced osd_op_complaint_time 180.000000 2024-08-03T20:49:33.943 INFO:teuthology.orchestra.run.smithi032.stdout:mds advanced rados_mon_op_timeout 900 2024-08-03T20:49:33.943 INFO:teuthology.orchestra.run.smithi032.stdout:mds advanced rados_osd_op_timeout 900 2024-08-03T20:49:33.943 INFO:teuthology.orchestra.run.smithi032.stdout:client advanced client_mount_timeout 600 2024-08-03T20:49:33.943 INFO:teuthology.orchestra.run.smithi032.stdout:client advanced debug_client 20/20 2024-08-03T20:49:33.944 INFO:teuthology.orchestra.run.smithi032.stdout:client advanced debug_ms 1/1 2024-08-03T20:49:33.944 INFO:teuthology.orchestra.run.smithi032.stdout:client advanced rados_mon_op_timeout 900 2024-08-03T20:49:33.944 INFO:teuthology.orchestra.run.smithi032.stdout:client advanced rados_osd_op_timeout 900 2024-08-03T20:49:33.945 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:33.942+0000 7f0363000640 1 -- 172.21.15.32:0/729284124 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f033c077010 msgr2=0x7f033c079520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:49:33.945 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:33.942+0000 7f0363000640 1 --2- 172.21.15.32:0/729284124 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f033c077010 0x7f033c079520 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f034c033000 tx=0x7f034c02d000 comp rx=0 tx=0).stop 2024-08-03T20:49:33.945 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:33.942+0000 7f0363000640 1 -- 172.21.15.32:0/729284124 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f035c15df40 msgr2=0x7f035c15e350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:49:33.945 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:33.942+0000 7f0363000640 1 --2- 172.21.15.32:0/729284124 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f035c15df40 0x7f035c15e350 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f035c072db0 tx=0x7f0354001b80 comp rx=0 tx=0).stop 2024-08-03T20:49:33.945 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:33.942+0000 7f0363000640 1 -- 172.21.15.32:0/729284124 shutdown_connections 2024-08-03T20:49:33.945 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:33.942+0000 7f0363000640 1 --2- 172.21.15.32:0/729284124 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f033c077010 0x7f033c079520 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:49:33.945 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:33.942+0000 7f0363000640 1 --2- 172.21.15.32:0/729284124 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f035c15df40 0x7f035c15e350 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:49:33.945 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:33.942+0000 7f0363000640 1 --2- 172.21.15.32:0/729284124 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f035c0b2fd0 0x7f035c0af310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:49:33.945 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:33.942+0000 7f0363000640 1 --2- 172.21.15.32:0/729284124 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f0340003c10 0x7f035c0b4c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:49:33.945 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:33.942+0000 7f0363000640 1 -- 172.21.15.32:0/729284124 >> 172.21.15.32:0/729284124 conn(0x7f035c0b1bd0 msgr2=0x7f035c140a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:49:33.945 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:33.943+0000 7f0363000640 1 -- 172.21.15.32:0/729284124 shutdown_connections 2024-08-03T20:49:33.946 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:33.943+0000 7f0363000640 1 -- 172.21.15.32:0/729284124 wait complete. 2024-08-03T20:49:33.967 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-08-03T20:49:34.198 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:34.196+0000 7fecb5400640 1 -- 172.21.15.32:0/3778256348 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fecb00ce700 msgr2=0x7fecb00ceb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:49:34.198 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:34.196+0000 7fecb5400640 1 --2- 172.21.15.32:0/3778256348 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fecb00ce700 0x7fecb00ceb10 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7fecb0073d70 tx=0x7feca802aac0 comp rx=0 tx=0).stop 2024-08-03T20:49:34.198 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:34.196+0000 7fecb5400640 1 -- 172.21.15.32:0/3778256348 shutdown_connections 2024-08-03T20:49:34.198 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:34.196+0000 7fecb5400640 1 --2- 172.21.15.32:0/3778256348 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fecb00ce700 0x7fecb00ceb10 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:49:34.198 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:34.196+0000 7fecb5400640 1 --2- 172.21.15.32:0/3778256348 >> v2:172.21.15.118:3300/0 conn(0x7fecb00cd5f0 0x7fecb00d29a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:49:34.198 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:34.196+0000 7fecb5400640 1 -- 172.21.15.32:0/3778256348 >> 172.21.15.32:0/3778256348 conn(0x7fecb00d1e80 msgr2=0x7fecb00d2290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:49:34.198 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:34.196+0000 7fecb5400640 1 -- 172.21.15.32:0/3778256348 shutdown_connections 2024-08-03T20:49:34.199 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:34.196+0000 7fecb5400640 1 -- 172.21.15.32:0/3778256348 wait complete. 2024-08-03T20:49:34.200 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:34.198+0000 7fecb5400640 1 Processor -- start 2024-08-03T20:49:34.200 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:34.199+0000 7fecb5400640 1 -- start start 2024-08-03T20:49:34.202 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:34.200+0000 7fecb5400640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fecb00b7420 0x7fecb015ea10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:49:34.202 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:34.200+0000 7fecb5400640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fecb00cd5f0 0x7fecb015d920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:49:34.202 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:34.200+0000 7fecb5400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fecb00ce700 0x7fecb006eb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:49:34.202 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:34.200+0000 7fecb5400640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fecb00d09f0 con 0x7fecb00ce700 2024-08-03T20:49:34.202 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:34.200+0000 7fecb5400640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7fecb00d0870 con 0x7fecb00cd5f0 2024-08-03T20:49:34.202 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:34.200+0000 7fecb5400640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7fecb00d0b70 con 0x7fecb00b7420 2024-08-03T20:49:34.202 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:34.200+0000 7fecb4a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fecb00ce700 0x7fecb006eb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:49:34.202 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:34.200+0000 7fecb4a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fecb00ce700 0x7fecb006eb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:54312/0 (socket says 172.21.15.32:54312) 2024-08-03T20:49:34.202 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:34.200+0000 7fecb4a00640 1 -- 172.21.15.32:0/4077023139 learned_addr learned my addr 172.21.15.32:0/4077023139 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:49:34.202 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:34.201+0000 7fecaf400640 1 --2- 172.21.15.32:0/4077023139 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fecb00cd5f0 0x7fecb015d920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:49:34.203 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:34.201+0000 7fecafe00640 1 --2- 172.21.15.32:0/4077023139 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fecb00b7420 0x7fecb015ea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:49:34.204 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:34.201+0000 7fecb4a00640 1 -- 172.21.15.32:0/4077023139 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fecb00b7420 msgr2=0x7fecb015ea10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:49:34.204 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:34.201+0000 7fecb4a00640 1 --2- 172.21.15.32:0/4077023139 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fecb00b7420 0x7fecb015ea10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:49:34.204 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:34.201+0000 7fecb4a00640 1 -- 172.21.15.32:0/4077023139 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fecb00cd5f0 msgr2=0x7fecb015d920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:49:34.204 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:34.201+0000 7fecb4a00640 1 --2- 172.21.15.32:0/4077023139 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fecb00cd5f0 0x7fecb015d920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:49:34.204 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:34.201+0000 7fecb4a00640 1 -- 172.21.15.32:0/4077023139 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fecb015ef50 con 0x7fecb00ce700 2024-08-03T20:49:34.204 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:34.201+0000 7fecb4a00640 1 --2- 172.21.15.32:0/4077023139 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fecb00ce700 0x7fecb006eb40 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7feca8009d60 tx=0x7feca8003ce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:49:34.204 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:34.202+0000 7feca7e00640 1 -- 172.21.15.32:0/4077023139 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feca8044070 con 0x7fecb00ce700 2024-08-03T20:49:34.204 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:34.202+0000 7feca7e00640 1 -- 172.21.15.32:0/4077023139 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7feca8004250 con 0x7fecb00ce700 2024-08-03T20:49:34.204 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:34.202+0000 7fecb5400640 1 -- 172.21.15.32:0/4077023139 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fecb01f5610 con 0x7fecb00ce700 2024-08-03T20:49:34.204 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:34.202+0000 7feca7e00640 1 -- 172.21.15.32:0/4077023139 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feca80046a0 con 0x7fecb00ce700 2024-08-03T20:49:34.205 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:34.202+0000 7fecb5400640 1 -- 172.21.15.32:0/4077023139 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fecb01f5bd0 con 0x7fecb00ce700 2024-08-03T20:49:34.205 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:34.203+0000 7fecb5400640 1 -- 172.21.15.32:0/4077023139 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fecb00d1100 con 0x7fecb00ce700 2024-08-03T20:49:34.205 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:34.204+0000 7feca7e00640 1 -- 172.21.15.32:0/4077023139 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7feca803f410 con 0x7fecb00ce700 2024-08-03T20:49:34.206 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:34.205+0000 7feca7e00640 1 --2- 172.21.15.32:0/4077023139 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fec90077010 0x7fec90079520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:49:34.206 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:34.205+0000 7feca7e00640 1 -- 172.21.15.32:0/4077023139 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(36..36 src has 1..36) ==== 5029+0+0 (secure 0 0 0) 0x7feca80b8ca0 con 0x7fecb00ce700 2024-08-03T20:49:34.207 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:34.205+0000 7fecafe00640 1 --2- 172.21.15.32:0/4077023139 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fec90077010 0x7fec90079520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:49:34.207 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:34.206+0000 7fecafe00640 1 --2- 172.21.15.32:0/4077023139 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fec90077010 0x7fec90079520 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7feca0004630 tx=0x7feca000b2e0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:49:34.210 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:34.209+0000 7feca7e00640 1 -- 172.21.15.32:0/4077023139 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7feca8032030 con 0x7fecb00ce700 2024-08-03T20:49:34.421 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:34.420+0000 7fecb5400640 1 -- 172.21.15.32:0/4077023139 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) -- 0x7fecb00cf550 con 0x7fecb00ce700 2024-08-03T20:49:34.425 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:34.424+0000 7feca7e00640 1 -- 172.21.15.32:0/4077023139 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v30)=0 v30) ==== 122+0+0 (secure 0 0 0) 0x7feca8081b70 con 0x7fecb00ce700 2024-08-03T20:49:34.428 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:34.427+0000 7fecb5400640 1 -- 172.21.15.32:0/4077023139 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fec90077010 msgr2=0x7fec90079520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:49:34.429 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:34.427+0000 7fecb5400640 1 --2- 172.21.15.32:0/4077023139 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fec90077010 0x7fec90079520 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7feca0004630 tx=0x7feca000b2e0 comp rx=0 tx=0).stop 2024-08-03T20:49:34.429 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:34.427+0000 7fecb5400640 1 -- 172.21.15.32:0/4077023139 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fecb00ce700 msgr2=0x7fecb006eb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:49:34.429 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:34.427+0000 7fecb5400640 1 --2- 172.21.15.32:0/4077023139 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fecb00ce700 0x7fecb006eb40 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7feca8009d60 tx=0x7feca8003ce0 comp rx=0 tx=0).stop 2024-08-03T20:49:34.429 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:34.427+0000 7fecb5400640 1 -- 172.21.15.32:0/4077023139 shutdown_connections 2024-08-03T20:49:34.429 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:34.427+0000 7fecb5400640 1 --2- 172.21.15.32:0/4077023139 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fec90077010 0x7fec90079520 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:49:34.429 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:34.427+0000 7fecb5400640 1 --2- 172.21.15.32:0/4077023139 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fecb00cd5f0 0x7fecb015d920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:49:34.429 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:34.427+0000 7fecb5400640 1 --2- 172.21.15.32:0/4077023139 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fecb00ce700 0x7fecb006eb40 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:49:34.429 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:34.427+0000 7fecb5400640 1 --2- 172.21.15.32:0/4077023139 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fecb00b7420 0x7fecb015ea10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:49:34.429 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:34.427+0000 7fecb5400640 1 -- 172.21.15.32:0/4077023139 >> 172.21.15.32:0/4077023139 conn(0x7fecb00d1e80 msgr2=0x7fecb00b0390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:49:34.429 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:34.427+0000 7fecb5400640 1 -- 172.21.15.32:0/4077023139 shutdown_connections 2024-08-03T20:49:34.429 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:34.427+0000 7fecb5400640 1 -- 172.21.15.32:0/4077023139 wait complete. 2024-08-03T20:49:34.454 INFO:teuthology.nuke:Clearing teuthology firewall rules... 2024-08-03T20:49:34.454 DEBUG:teuthology.orchestra.run.smithi032:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-08-03T20:49:34.537 DEBUG:teuthology.orchestra.run.smithi118:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-08-03T20:49:34.636 INFO:teuthology.nuke:Cleared teuthology firewall rules. 2024-08-03T20:49:34.637 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-08-03T20:49:34.637 DEBUG:teuthology.orchestra.run.smithi032:> dd if=/proc/self/mounts of=/dev/stdout 2024-08-03T20:49:34.672 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053453,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6499208k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 none /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249604k,nr_inodes=812401,mode=700,uid=1000,gid=1316 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6499208k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-08-03T20:49:34.672 DEBUG:teuthology.orchestra.run.smithi118:> set -ex 2024-08-03T20:49:34.672 DEBUG:teuthology.orchestra.run.smithi118:> dd if=/proc/self/mounts of=/dev/stdout 2024-08-03T20:49:34.709 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052663,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497944k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248972k,nr_inodes=812243,mode=700,uid=1000,gid=1316 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497944k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-08-03T20:49:34.709 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-08-03T20:49:34.709 DEBUG:teuthology.orchestra.run.smithi032:> dd if=/proc/self/mounts of=/dev/stdout 2024-08-03T20:49:34.748 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053453,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6499208k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 none /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249604k,nr_inodes=812401,mode=700,uid=1000,gid=1316 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6499208k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-08-03T20:49:34.748 DEBUG:teuthology.orchestra.run.smithi118:> set -ex 2024-08-03T20:49:34.749 DEBUG:teuthology.orchestra.run.smithi118:> dd if=/proc/self/mounts of=/dev/stdout 2024-08-03T20:49:34.784 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052663,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497944k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248972k,nr_inodes=812243,mode=700,uid=1000,gid=1316 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497944k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-08-03T20:49:34.784 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-08-03T20:49:34.785 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-08-03T20:49:35.018 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:35.016+0000 7f0124400640 1 -- 172.21.15.32:0/3315631900 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f011c15fff0 msgr2=0x7f011c160400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:49:35.018 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:35.016+0000 7f0124400640 1 --2- 172.21.15.32:0/3315631900 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f011c15fff0 0x7f011c160400 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f0114005fd0 tx=0x7f011402aa90 comp rx=0 tx=0).stop 2024-08-03T20:49:35.019 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:35.016+0000 7f0124400640 1 -- 172.21.15.32:0/3315631900 shutdown_connections 2024-08-03T20:49:35.019 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:35.016+0000 7f0124400640 1 --2- 172.21.15.32:0/3315631900 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f011c15fff0 0x7f011c160400 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:49:35.019 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:35.016+0000 7f0124400640 1 --2- 172.21.15.32:0/3315631900 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f011c06e930 0x7f011c144f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:49:35.019 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:35.016+0000 7f0124400640 1 -- 172.21.15.32:0/3315631900 >> 172.21.15.32:0/3315631900 conn(0x7f011c0b1bd0 msgr2=0x7f011c0b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:49:35.019 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:35.016+0000 7f0124400640 1 -- 172.21.15.32:0/3315631900 shutdown_connections 2024-08-03T20:49:35.019 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:35.016+0000 7f0124400640 1 -- 172.21.15.32:0/3315631900 wait complete. 2024-08-03T20:49:35.020 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:35.019+0000 7f0124400640 1 Processor -- start 2024-08-03T20:49:35.020 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:35.019+0000 7f0124400640 1 -- start start 2024-08-03T20:49:35.022 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:35.020+0000 7f0124400640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f011c06e930 0x7f011c1347b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:49:35.022 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:35.020+0000 7f0124400640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f011c073e60 0x7f011c1f8880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:49:35.022 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:35.020+0000 7f0124400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f011c15fff0 0x7f011c1f8dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:49:35.022 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:35.020+0000 7f0124400640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f011c0cf840 con 0x7f011c15fff0 2024-08-03T20:49:35.023 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:35.020+0000 7f0124400640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f011c145f30 con 0x7f011c06e930 2024-08-03T20:49:35.023 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:35.020+0000 7f0124400640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7f011c0cf9c0 con 0x7f011c073e60 2024-08-03T20:49:35.023 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:35.020+0000 7f0122400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f011c15fff0 0x7f011c1f8dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:49:35.023 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:35.020+0000 7f0121a00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f011c06e930 0x7f011c1347b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:49:35.023 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:35.020+0000 7f0121000640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f011c073e60 0x7f011c1f8880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:49:35.023 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:35.020+0000 7f0122400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f011c15fff0 0x7f011c1f8dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:54334/0 (socket says 172.21.15.32:54334) 2024-08-03T20:49:35.023 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:35.020+0000 7f0122400640 1 -- 172.21.15.32:0/1745505262 learned_addr learned my addr 172.21.15.32:0/1745505262 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:49:35.023 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:35.021+0000 7f0122400640 1 -- 172.21.15.32:0/1745505262 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f011c073e60 msgr2=0x7f011c1f8880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:49:35.023 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:35.021+0000 7f0122400640 1 --2- 172.21.15.32:0/1745505262 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f011c073e60 0x7f011c1f8880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:49:35.023 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:35.021+0000 7f0122400640 1 -- 172.21.15.32:0/1745505262 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f011c06e930 msgr2=0x7f011c1347b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:49:35.023 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:35.021+0000 7f0122400640 1 --2- 172.21.15.32:0/1745505262 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f011c06e930 0x7f011c1347b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:49:35.023 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:35.021+0000 7f0122400640 1 -- 172.21.15.32:0/1745505262 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f011c1e7bf0 con 0x7f011c15fff0 2024-08-03T20:49:35.024 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:35.021+0000 7f0122400640 1 --2- 172.21.15.32:0/1745505262 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f011c15fff0 0x7f011c1f8dc0 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f0114006010 tx=0x7f0114003ce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:49:35.024 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:35.021+0000 7f0119600640 1 -- 172.21.15.32:0/1745505262 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0114044070 con 0x7f011c15fff0 2024-08-03T20:49:35.024 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:35.021+0000 7f0119600640 1 -- 172.21.15.32:0/1745505262 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f0114004040 con 0x7f011c15fff0 2024-08-03T20:49:35.025 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:35.021+0000 7f0124400640 1 -- 172.21.15.32:0/1745505262 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f011c134cf0 con 0x7f011c15fff0 2024-08-03T20:49:35.025 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:35.022+0000 7f0119600640 1 -- 172.21.15.32:0/1745505262 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0114004430 con 0x7f011c15fff0 2024-08-03T20:49:35.025 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:35.022+0000 7f0124400640 1 -- 172.21.15.32:0/1745505262 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f011c1e8170 con 0x7f011c15fff0 2024-08-03T20:49:35.025 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:35.023+0000 7f0124400640 1 -- 172.21.15.32:0/1745505262 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f011c160c50 con 0x7f011c15fff0 2024-08-03T20:49:35.025 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:35.024+0000 7f0119600640 1 -- 172.21.15.32:0/1745505262 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f0114047070 con 0x7f011c15fff0 2024-08-03T20:49:35.026 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:35.025+0000 7f0119600640 1 --2- 172.21.15.32:0/1745505262 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f00fc077010 0x7f00fc079520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:49:35.026 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:35.025+0000 7f0119600640 1 -- 172.21.15.32:0/1745505262 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(36..36 src has 1..36) ==== 5029+0+0 (secure 0 0 0) 0x7f01140b94b0 con 0x7f011c15fff0 2024-08-03T20:49:35.027 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:35.025+0000 7f0121a00640 1 --2- 172.21.15.32:0/1745505262 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f00fc077010 0x7f00fc079520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:49:35.027 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:35.026+0000 7f0121a00640 1 --2- 172.21.15.32:0/1745505262 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f00fc077010 0x7f00fc079520 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f0110009b30 tx=0x7f0110005f90 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:49:35.031 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:35.030+0000 7f0119600640 1 -- 172.21.15.32:0/1745505262 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7f0114032030 con 0x7f011c15fff0 2024-08-03T20:49:35.275 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:35.274+0000 7f0124400640 1 -- 172.21.15.32:0/1745505262 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f011c145720 con 0x7f011c15fff0 2024-08-03T20:49:35.276 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:35.274+0000 7f0119600640 1 -- 172.21.15.32:0/1745505262 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) ==== 94+0+3917 (secure 0 0 0) 0x7f01140822f0 con 0x7f011c15fff0 2024-08-03T20:49:35.276 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-03T20:49:35.276 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":27,"btime":"2024-08-03T20:49:27:425834+0000","default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4643,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6836/4002526832","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6834","nonce":4002526832},{"type":"v1","addr":"172.21.15.118:6836","nonce":4002526832}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":4646,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6837/3036256187","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6835","nonce":3036256187},{"type":"v1","addr":"172.21.15.118:6837","nonce":3036256187}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":4652,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6834/2104418334","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6833","nonce":2104418334},{"type":"v1","addr":"172.21.15.32:6834","nonce":2104418334}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":5328,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6835/2723619078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":2723619078},{"type":"v1","addr":"172.21.15.32:6835","nonce":2723619078}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":27}],"filesystems":[]} 2024-08-03T20:49:35.277 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 27 2024-08-03T20:49:35.278 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:35.277+0000 7f0124400640 1 -- 172.21.15.32:0/1745505262 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f00fc077010 msgr2=0x7f00fc079520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:49:35.279 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:35.277+0000 7f0124400640 1 --2- 172.21.15.32:0/1745505262 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f00fc077010 0x7f00fc079520 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f0110009b30 tx=0x7f0110005f90 comp rx=0 tx=0).stop 2024-08-03T20:49:35.279 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:35.277+0000 7f0124400640 1 -- 172.21.15.32:0/1745505262 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f011c15fff0 msgr2=0x7f011c1f8dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:49:35.279 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:35.277+0000 7f0124400640 1 --2- 172.21.15.32:0/1745505262 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f011c15fff0 0x7f011c1f8dc0 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f0114006010 tx=0x7f0114003ce0 comp rx=0 tx=0).stop 2024-08-03T20:49:35.279 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:35.277+0000 7f0124400640 1 -- 172.21.15.32:0/1745505262 shutdown_connections 2024-08-03T20:49:35.279 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:35.277+0000 7f0124400640 1 --2- 172.21.15.32:0/1745505262 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f00fc077010 0x7f00fc079520 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:49:35.279 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:35.277+0000 7f0124400640 1 --2- 172.21.15.32:0/1745505262 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f011c15fff0 0x7f011c1f8dc0 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:49:35.279 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:35.277+0000 7f0124400640 1 --2- 172.21.15.32:0/1745505262 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f011c073e60 0x7f011c1f8880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:49:35.279 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:35.277+0000 7f0124400640 1 --2- 172.21.15.32:0/1745505262 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f011c06e930 0x7f011c1347b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:49:35.279 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:35.277+0000 7f0124400640 1 -- 172.21.15.32:0/1745505262 >> 172.21.15.32:0/1745505262 conn(0x7f011c0b1bd0 msgr2=0x7f011c074270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:49:35.279 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:35.277+0000 7f0124400640 1 -- 172.21.15.32:0/1745505262 shutdown_connections 2024-08-03T20:49:35.279 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:35.277+0000 7f0124400640 1 -- 172.21.15.32:0/1745505262 wait complete. 2024-08-03T20:49:35.300 INFO:tasks.ceph.mds.a:Restarting daemon 2024-08-03T20:49:35.301 INFO:tasks.ceph.mds.a:Stopping old one... 2024-08-03T20:49:35.301 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-08-03T20:49:35.301 INFO:teuthology.orchestra.run:waiting for 300 2024-08-03T20:49:35.301 INFO:tasks.ceph.mds.c:Restarting daemon 2024-08-03T20:49:35.301 INFO:tasks.ceph.mds.c:Stopping old one... 2024-08-03T20:49:35.301 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-08-03T20:49:35.301 INFO:teuthology.orchestra.run:waiting for 300 2024-08-03T20:49:35.302 INFO:tasks.ceph.mds.b:Restarting daemon 2024-08-03T20:49:35.302 INFO:tasks.ceph.mds.b:Stopping old one... 2024-08-03T20:49:35.302 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-08-03T20:49:35.302 INFO:teuthology.orchestra.run:waiting for 300 2024-08-03T20:49:35.302 INFO:tasks.ceph.mds.d:Restarting daemon 2024-08-03T20:49:35.302 INFO:tasks.ceph.mds.d:Stopping old one... 2024-08-03T20:49:35.303 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-08-03T20:49:35.303 INFO:teuthology.orchestra.run:waiting for 300 2024-08-03T20:49:35.323 INFO:tasks.ceph.mds.a:Stopped 2024-08-03T20:49:35.323 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-08-03T20:49:35.325 INFO:tasks.ceph.mds.d:Stopped 2024-08-03T20:49:35.325 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-08-03T20:49:35.326 INFO:tasks.ceph.mds.c:Stopped 2024-08-03T20:49:35.326 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-08-03T20:49:35.369 INFO:tasks.ceph.mds.d:Started 2024-08-03T20:49:35.369 DEBUG:teuthology.parallel:result is None 2024-08-03T20:49:35.371 INFO:tasks.ceph.mds.a:Started 2024-08-03T20:49:35.371 DEBUG:teuthology.parallel:result is None 2024-08-03T20:49:35.374 INFO:tasks.ceph.mds.c:Started 2024-08-03T20:49:35.374 DEBUG:teuthology.parallel:result is None 2024-08-03T20:49:35.558 INFO:tasks.ceph.mds.d.smithi118.stdout:starting mds.d at 2024-08-03T20:49:35.565 INFO:tasks.ceph.mds.a.smithi032.stdout:starting mds.a at 2024-08-03T20:49:35.569 INFO:tasks.ceph.mds.c.smithi032.stdout:starting mds.c at 2024-08-03T20:49:36.641 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-08-03T20:49:36.646 DEBUG:teuthology.orchestra.run.smithi118:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-08-03T20:49:36.690 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~0s 2024-08-03T20:49:41.325 INFO:tasks.ceph.mds.b:Stopped 2024-08-03T20:49:41.325 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-08-03T20:49:41.331 INFO:tasks.ceph.mds.b:Started 2024-08-03T20:49:41.331 DEBUG:teuthology.parallel:result is None 2024-08-03T20:49:41.331 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-08-03T20:49:41.528 INFO:tasks.ceph.mds.b.smithi118.stdout:starting mds.b at 2024-08-03T20:49:41.573 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:41.571+0000 7ffb77e00640 1 -- 172.21.15.32:0/711107096 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ffb7806fb50 msgr2=0x7ffb780ad190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:49:41.573 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:41.571+0000 7ffb77e00640 1 --2- 172.21.15.32:0/711107096 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ffb7806fb50 0x7ffb780ad190 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7ffb780720a0 tx=0x7ffb7002a820 comp rx=0 tx=0).stop 2024-08-03T20:49:41.573 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:41.571+0000 7ffb77e00640 1 -- 172.21.15.32:0/711107096 shutdown_connections 2024-08-03T20:49:41.573 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:41.571+0000 7ffb77e00640 1 --2- 172.21.15.32:0/711107096 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ffb7806fb50 0x7ffb780ad190 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:49:41.573 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:41.571+0000 7ffb77e00640 1 --2- 172.21.15.32:0/711107096 >> v2:172.21.15.118:3300/0 conn(0x7ffb780af020 0x7ffb780742b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:49:41.573 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:41.571+0000 7ffb77e00640 1 -- 172.21.15.32:0/711107096 >> 172.21.15.32:0/711107096 conn(0x7ffb780b2120 msgr2=0x7ffb780cfd60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:49:41.573 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:41.571+0000 7ffb77e00640 1 -- 172.21.15.32:0/711107096 shutdown_connections 2024-08-03T20:49:41.573 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:41.571+0000 7ffb77e00640 1 -- 172.21.15.32:0/711107096 wait complete. 2024-08-03T20:49:41.574 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:41.573+0000 7ffb77e00640 1 Processor -- start 2024-08-03T20:49:41.575 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:41.573+0000 7ffb77e00640 1 -- start start 2024-08-03T20:49:41.577 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:41.575+0000 7ffb77e00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ffb7806fb50 0x7ffb7814d1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:49:41.577 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:41.575+0000 7ffb77e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ffb780acd80 0x7ffb780c5030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:49:41.577 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:41.575+0000 7ffb77e00640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7ffb780af020 0x7ffb780c5570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:49:41.577 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:41.575+0000 7ffb77e00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7ffb781601e0 con 0x7ffb780acd80 2024-08-03T20:49:41.577 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:41.575+0000 7ffb77e00640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7ffb78160060 con 0x7ffb7806fb50 2024-08-03T20:49:41.578 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:41.575+0000 7ffb77e00640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7ffb78160360 con 0x7ffb780af020 2024-08-03T20:49:41.578 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:41.575+0000 7ffb6e000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ffb780acd80 0x7ffb780c5030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:49:41.578 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:41.575+0000 7ffb76a00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ffb7806fb50 0x7ffb7814d1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:49:41.578 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:41.575+0000 7ffb6e000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ffb780acd80 0x7ffb780c5030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:35218/0 (socket says 172.21.15.32:35218) 2024-08-03T20:49:41.578 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:41.575+0000 7ffb6e000640 1 -- 172.21.15.32:0/385839673 learned_addr learned my addr 172.21.15.32:0/385839673 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:49:41.578 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:41.575+0000 7ffb77400640 1 --2- 172.21.15.32:0/385839673 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7ffb780af020 0x7ffb780c5570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:49:41.578 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:41.576+0000 7ffb6e000640 1 -- 172.21.15.32:0/385839673 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7ffb780af020 msgr2=0x7ffb780c5570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:49:41.578 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:41.576+0000 7ffb6e000640 1 --2- 172.21.15.32:0/385839673 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7ffb780af020 0x7ffb780c5570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:49:41.578 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:41.576+0000 7ffb6e000640 1 -- 172.21.15.32:0/385839673 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ffb7806fb50 msgr2=0x7ffb7814d1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:49:41.578 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:41.576+0000 7ffb6e000640 1 --2- 172.21.15.32:0/385839673 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ffb7806fb50 0x7ffb7814d1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:49:41.578 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:41.576+0000 7ffb6e000640 1 -- 172.21.15.32:0/385839673 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffb7815bee0 con 0x7ffb780acd80 2024-08-03T20:49:41.578 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:41.576+0000 7ffb6e000640 1 --2- 172.21.15.32:0/385839673 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ffb780acd80 0x7ffb780c5030 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7ffb6000c430 tx=0x7ffb6000c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:49:41.579 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:41.576+0000 7ffb6fe00640 1 -- 172.21.15.32:0/385839673 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb60011790 con 0x7ffb780acd80 2024-08-03T20:49:41.579 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:41.576+0000 7ffb6fe00640 1 -- 172.21.15.32:0/385839673 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7ffb60017070 con 0x7ffb780acd80 2024-08-03T20:49:41.579 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:41.576+0000 7ffb77e00640 1 -- 172.21.15.32:0/385839673 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffb7814d6f0 con 0x7ffb780acd80 2024-08-03T20:49:41.579 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:41.576+0000 7ffb6fe00640 1 -- 172.21.15.32:0/385839673 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb60011ce0 con 0x7ffb780acd80 2024-08-03T20:49:41.580 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:41.576+0000 7ffb77e00640 1 -- 172.21.15.32:0/385839673 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffb781f8ae0 con 0x7ffb780acd80 2024-08-03T20:49:41.580 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:41.577+0000 7ffb77e00640 1 -- 172.21.15.32:0/385839673 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ffb780cf5c0 con 0x7ffb780acd80 2024-08-03T20:49:41.580 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:41.579+0000 7ffb6fe00640 1 -- 172.21.15.32:0/385839673 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7ffb6001c460 con 0x7ffb780acd80 2024-08-03T20:49:41.582 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:41.580+0000 7ffb6fe00640 1 --2- 172.21.15.32:0/385839673 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7ffb54077010 0x7ffb54079520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:49:41.582 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:41.580+0000 7ffb6fe00640 1 -- 172.21.15.32:0/385839673 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(39..39 src has 1..39) ==== 5029+0+0 (secure 0 0 0) 0x7ffb600984b0 con 0x7ffb780acd80 2024-08-03T20:49:41.582 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:41.581+0000 7ffb76a00640 1 --2- 172.21.15.32:0/385839673 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7ffb54077010 0x7ffb54079520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:49:41.583 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:41.581+0000 7ffb76a00640 1 --2- 172.21.15.32:0/385839673 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7ffb54077010 0x7ffb54079520 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7ffb68009b30 tx=0x7ffb68005f90 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:49:41.586 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:41.585+0000 7ffb6fe00640 1 -- 172.21.15.32:0/385839673 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7ffb7815bee0 con 0x7ffb780acd80 2024-08-03T20:49:41.806 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:41.805+0000 7ffb77e00640 1 -- 172.21.15.32:0/385839673 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) -- 0x7ffb780d0930 con 0x7ffb780acd80 2024-08-03T20:49:43.403 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b has been restored 2024-08-03T20:49:43.497 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:43.496+0000 7ffb6fe00640 1 -- 172.21.15.32:0/385839673 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v41) ==== 97+0+0 (secure 0 0 0) 0x7ffb60061260 con 0x7ffb780acd80 2024-08-03T20:49:43.497 INFO:teuthology.orchestra.run.smithi032.stderr: removed all blocklist entries 2024-08-03T20:49:43.500 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:43.499+0000 7ffb77e00640 1 -- 172.21.15.32:0/385839673 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7ffb54077010 msgr2=0x7ffb54079520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:49:43.500 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:43.499+0000 7ffb77e00640 1 --2- 172.21.15.32:0/385839673 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7ffb54077010 0x7ffb54079520 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7ffb68009b30 tx=0x7ffb68005f90 comp rx=0 tx=0).stop 2024-08-03T20:49:43.500 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:43.499+0000 7ffb77e00640 1 -- 172.21.15.32:0/385839673 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ffb780acd80 msgr2=0x7ffb780c5030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:49:43.500 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:43.499+0000 7ffb77e00640 1 --2- 172.21.15.32:0/385839673 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ffb780acd80 0x7ffb780c5030 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7ffb6000c430 tx=0x7ffb6000c8f0 comp rx=0 tx=0).stop 2024-08-03T20:49:43.500 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:43.499+0000 7ffb77e00640 1 -- 172.21.15.32:0/385839673 shutdown_connections 2024-08-03T20:49:43.500 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:43.499+0000 7ffb77e00640 1 --2- 172.21.15.32:0/385839673 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7ffb54077010 0x7ffb54079520 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:49:43.500 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:43.499+0000 7ffb77e00640 1 --2- 172.21.15.32:0/385839673 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7ffb780af020 0x7ffb780c5570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:49:43.501 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:43.499+0000 7ffb77e00640 1 --2- 172.21.15.32:0/385839673 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ffb780acd80 0x7ffb780c5030 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:49:43.501 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:43.499+0000 7ffb77e00640 1 --2- 172.21.15.32:0/385839673 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ffb7806fb50 0x7ffb7814d1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:49:43.501 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:43.499+0000 7ffb77e00640 1 -- 172.21.15.32:0/385839673 >> 172.21.15.32:0/385839673 conn(0x7ffb780b2120 msgr2=0x7ffb780cbc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:49:43.501 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:43.500+0000 7ffb77e00640 1 -- 172.21.15.32:0/385839673 shutdown_connections 2024-08-03T20:49:43.501 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:43.500+0000 7ffb77e00640 1 -- 172.21.15.32:0/385839673 wait complete. 2024-08-03T20:49:43.530 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-08-03T20:49:43.764 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:43.763+0000 7f5552e00640 1 -- 172.21.15.32:0/1303778184 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f554c0b39a0 msgr2=0x7f554c0b3db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:49:43.764 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:43.763+0000 7f5552e00640 1 --2- 172.21.15.32:0/1303778184 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f554c0b39a0 0x7f554c0b3db0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f5534005fd0 tx=0x7f553402a5d0 comp rx=0 tx=0).stop 2024-08-03T20:49:43.765 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:43.763+0000 7f5552e00640 1 -- 172.21.15.32:0/1303778184 shutdown_connections 2024-08-03T20:49:43.765 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:43.763+0000 7f5552e00640 1 --2- 172.21.15.32:0/1303778184 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f554c0b39a0 0x7f554c0b3db0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:49:43.765 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:43.763+0000 7f5552e00640 1 --2- 172.21.15.32:0/1303778184 >> v2:172.21.15.118:3300/0 conn(0x7f554c06f720 0x7f554c06fb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:49:43.765 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:43.763+0000 7f5552e00640 1 -- 172.21.15.32:0/1303778184 >> 172.21.15.32:0/1303778184 conn(0x7f554c0b1bd0 msgr2=0x7f554c0b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:49:43.765 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:43.763+0000 7f5552e00640 1 -- 172.21.15.32:0/1303778184 shutdown_connections 2024-08-03T20:49:43.765 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:43.763+0000 7f5552e00640 1 -- 172.21.15.32:0/1303778184 wait complete. 2024-08-03T20:49:43.765 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:43.765+0000 7f5552e00640 1 Processor -- start 2024-08-03T20:49:43.765 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:43.765+0000 7f5552e00640 1 -- start start 2024-08-03T20:49:43.769 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:43.767+0000 7f5552e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f554c06f720 0x7f554c14f630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:49:43.769 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:43.767+0000 7f5552e00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f554c0b39a0 0x7f554c1f8c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:49:43.769 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:43.767+0000 7f5552e00640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f554c162210 0x7f554c15a950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:49:43.769 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:43.767+0000 7f5552e00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f554c162d30 con 0x7f554c06f720 2024-08-03T20:49:43.769 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:43.767+0000 7f5552e00640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f554c0af0a0 con 0x7f554c0b39a0 2024-08-03T20:49:43.769 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:43.767+0000 7f5552e00640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7f554c162eb0 con 0x7f554c162210 2024-08-03T20:49:43.769 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:43.767+0000 7f554be00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f554c06f720 0x7f554c14f630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:49:43.769 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:43.767+0000 7f554be00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f554c06f720 0x7f554c14f630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:35228/0 (socket says 172.21.15.32:35228) 2024-08-03T20:49:43.769 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:43.767+0000 7f554be00640 1 -- 172.21.15.32:0/2226017118 learned_addr learned my addr 172.21.15.32:0/2226017118 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:49:43.769 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:43.767+0000 7f5550e00640 1 --2- 172.21.15.32:0/2226017118 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f554c162210 0x7f554c15a950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:49:43.770 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:43.767+0000 7f5543400640 1 --2- 172.21.15.32:0/2226017118 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f554c0b39a0 0x7f554c1f8c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:49:43.770 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:43.768+0000 7f554be00640 1 -- 172.21.15.32:0/2226017118 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f554c162210 msgr2=0x7f554c15a950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:49:43.770 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:43.768+0000 7f554be00640 1 --2- 172.21.15.32:0/2226017118 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f554c162210 0x7f554c15a950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:49:43.770 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:43.768+0000 7f554be00640 1 -- 172.21.15.32:0/2226017118 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f554c0b39a0 msgr2=0x7f554c1f8c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:49:43.770 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:43.768+0000 7f554be00640 1 --2- 172.21.15.32:0/2226017118 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f554c0b39a0 0x7f554c1f8c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:49:43.770 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:43.768+0000 7f554be00640 1 -- 172.21.15.32:0/2226017118 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f554c14fb70 con 0x7f554c06f720 2024-08-03T20:49:43.770 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:43.768+0000 7f554be00640 1 --2- 172.21.15.32:0/2226017118 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f554c06f720 0x7f554c14f630 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f553c00e9f0 tx=0x7f553c00eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:49:43.770 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:43.768+0000 7f5548c00640 1 -- 172.21.15.32:0/2226017118 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f553c00cd80 con 0x7f554c06f720 2024-08-03T20:49:43.770 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:43.768+0000 7f5548c00640 1 -- 172.21.15.32:0/2226017118 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f553c004020 con 0x7f554c06f720 2024-08-03T20:49:43.770 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:43.768+0000 7f5548c00640 1 -- 172.21.15.32:0/2226017118 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f553c004470 con 0x7f554c06f720 2024-08-03T20:49:43.770 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:43.768+0000 7f5552e00640 1 -- 172.21.15.32:0/2226017118 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f554c1f9a00 con 0x7f554c06f720 2024-08-03T20:49:43.770 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:43.768+0000 7f5552e00640 1 -- 172.21.15.32:0/2226017118 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f554c159530 con 0x7f554c06f720 2024-08-03T20:49:43.771 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:43.769+0000 7f5552e00640 1 -- 172.21.15.32:0/2226017118 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f554c06fb30 con 0x7f554c06f720 2024-08-03T20:49:43.771 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:43.770+0000 7f5548c00640 1 -- 172.21.15.32:0/2226017118 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f553c0035f0 con 0x7f554c06f720 2024-08-03T20:49:43.772 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:43.771+0000 7f5548c00640 1 --2- 172.21.15.32:0/2226017118 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f5528077010 0x7f5528079520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:49:43.772 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:43.772+0000 7f5548c00640 1 -- 172.21.15.32:0/2226017118 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(41..41 src has 1..41) ==== 4857+0+0 (secure 0 0 0) 0x7f553c098ec0 con 0x7f554c06f720 2024-08-03T20:49:43.772 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:43.772+0000 7f5543400640 1 --2- 172.21.15.32:0/2226017118 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f5528077010 0x7f5528079520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:49:43.773 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:43.773+0000 7f5543400640 1 --2- 172.21.15.32:0/2226017118 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f5528077010 0x7f5528079520 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f5534033000 tx=0x7f5534004730 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:49:43.776 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:43.776+0000 7f5548c00640 1 -- 172.21.15.32:0/2226017118 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7f553c062400 con 0x7f554c06f720 2024-08-03T20:49:44.054 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:44.053+0000 7f5552e00640 1 -- 172.21.15.32:0/2226017118 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) -- 0x7f554c074a90 con 0x7f554c06f720 2024-08-03T20:49:44.054 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:44.054+0000 7f5548c00640 1 -- 172.21.15.32:0/2226017118 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v11) ==== 80+0+5304 (secure 0 0 0) 0x7f553c061d00 con 0x7f554c06f720 2024-08-03T20:49:44.055 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-03T20:49:44.055 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-08-03T20:49:44.055 INFO:teuthology.orchestra.run.smithi032.stdout: "auth_dump": [ 2024-08-03T20:49:44.055 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-03T20:49:44.055 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "mds.a", 2024-08-03T20:49:44.055 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQAvl65mfaKbOhAAUgCH82U0SB7owF5Cds3XYA==", 2024-08-03T20:49:44.055 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2024-08-03T20:49:44.055 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": "allow", 2024-08-03T20:49:44.055 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow *", 2024-08-03T20:49:44.055 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow *", 2024-08-03T20:49:44.055 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2024-08-03T20:49:44.056 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-03T20:49:44.056 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-03T20:49:44.056 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-03T20:49:44.056 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "mds.b", 2024-08-03T20:49:44.056 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQAwl65mNcoVGxAAlRXNjZhxVu8Vjslk9Msrxw==", 2024-08-03T20:49:44.056 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2024-08-03T20:49:44.056 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": "allow", 2024-08-03T20:49:44.056 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow *", 2024-08-03T20:49:44.056 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow *", 2024-08-03T20:49:44.056 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2024-08-03T20:49:44.057 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-03T20:49:44.057 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-03T20:49:44.057 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-03T20:49:44.057 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "mds.c", 2024-08-03T20:49:44.057 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQAwl65mEUAFDRAAGdw1cymfvfKxGeJS/Od6Rw==", 2024-08-03T20:49:44.057 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2024-08-03T20:49:44.057 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": "allow", 2024-08-03T20:49:44.057 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow *", 2024-08-03T20:49:44.057 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow *", 2024-08-03T20:49:44.057 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2024-08-03T20:49:44.057 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-03T20:49:44.057 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-03T20:49:44.058 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-03T20:49:44.058 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "mds.d", 2024-08-03T20:49:44.058 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQAwl65mjUHZKBAAqOCnIopBK8Rhk36hrlNnzw==", 2024-08-03T20:49:44.058 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2024-08-03T20:49:44.058 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": "allow", 2024-08-03T20:49:44.058 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow *", 2024-08-03T20:49:44.058 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow *", 2024-08-03T20:49:44.058 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2024-08-03T20:49:44.058 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-03T20:49:44.058 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-03T20:49:44.058 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-03T20:49:44.058 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "osd.0", 2024-08-03T20:49:44.058 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQAyl65m4kIyLxAAdCh0D2MjJNPPu5b7jNkrbw==", 2024-08-03T20:49:44.059 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2024-08-03T20:49:44.059 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow profile osd", 2024-08-03T20:49:44.059 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow profile osd", 2024-08-03T20:49:44.059 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2024-08-03T20:49:44.061 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-03T20:49:44.061 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-03T20:49:44.061 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-03T20:49:44.061 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "osd.1", 2024-08-03T20:49:44.062 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQA1l65mLzVOOhAA081V1cUyYFPz17L0nny7gA==", 2024-08-03T20:49:44.062 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2024-08-03T20:49:44.062 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow profile osd", 2024-08-03T20:49:44.062 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow profile osd", 2024-08-03T20:49:44.062 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2024-08-03T20:49:44.062 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-03T20:49:44.062 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-03T20:49:44.062 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-03T20:49:44.062 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "osd.2", 2024-08-03T20:49:44.062 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQA5l65mBnXDBxAA9/lZVkwD2ykMTOIysoYODg==", 2024-08-03T20:49:44.062 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2024-08-03T20:49:44.062 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow profile osd", 2024-08-03T20:49:44.062 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow profile osd", 2024-08-03T20:49:44.063 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2024-08-03T20:49:44.063 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-03T20:49:44.063 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-03T20:49:44.063 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-03T20:49:44.063 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "osd.3", 2024-08-03T20:49:44.063 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQA8l65mAriiERAAfzV8An67+PnA3EzeAutIbA==", 2024-08-03T20:49:44.063 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2024-08-03T20:49:44.063 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow profile osd", 2024-08-03T20:49:44.063 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow profile osd", 2024-08-03T20:49:44.063 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2024-08-03T20:49:44.063 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-03T20:49:44.063 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-03T20:49:44.063 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-03T20:49:44.063 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "osd.4", 2024-08-03T20:49:44.064 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQBAl65m5uJSJRAA8S2IjJpxuqHY8YK4tTDMPg==", 2024-08-03T20:49:44.064 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2024-08-03T20:49:44.064 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow profile osd", 2024-08-03T20:49:44.064 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow profile osd", 2024-08-03T20:49:44.064 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2024-08-03T20:49:44.064 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-03T20:49:44.064 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-03T20:49:44.064 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-03T20:49:44.064 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "osd.5", 2024-08-03T20:49:44.064 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQBDl65mBP3JLBAADgK2X3zNyfnjjZADX1PFcA==", 2024-08-03T20:49:44.064 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2024-08-03T20:49:44.065 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow profile osd", 2024-08-03T20:49:44.065 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow profile osd", 2024-08-03T20:49:44.065 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2024-08-03T20:49:44.065 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-03T20:49:44.065 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-03T20:49:44.065 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-03T20:49:44.065 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "osd.6", 2024-08-03T20:49:44.065 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQBGl65m8DBNNRAANeOUfW70S5JfqFlpuMF9KQ==", 2024-08-03T20:49:44.065 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2024-08-03T20:49:44.065 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow profile osd", 2024-08-03T20:49:44.065 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow profile osd", 2024-08-03T20:49:44.065 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2024-08-03T20:49:44.065 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-03T20:49:44.066 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-03T20:49:44.066 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-03T20:49:44.066 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "osd.7", 2024-08-03T20:49:44.066 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQBKl65mUnOcABAA41wylzC1Ec9Fj01Ke0HhTw==", 2024-08-03T20:49:44.066 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2024-08-03T20:49:44.066 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow profile osd", 2024-08-03T20:49:44.066 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow profile osd", 2024-08-03T20:49:44.066 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2024-08-03T20:49:44.066 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-03T20:49:44.066 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-03T20:49:44.066 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-03T20:49:44.066 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "client.0", 2024-08-03T20:49:44.066 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQAwl65msgO3NRAAHG/WuMTK9c4lDXocXvqWnA==", 2024-08-03T20:49:44.067 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2024-08-03T20:49:44.067 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": "allow", 2024-08-03T20:49:44.067 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow r", 2024-08-03T20:49:44.067 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-08-03T20:49:44.067 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-03T20:49:44.067 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-03T20:49:44.067 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-03T20:49:44.067 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "client.1", 2024-08-03T20:49:44.067 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQAxl65mbe19DxAA+sepc9ucXk2Bqy7C90mfbA==", 2024-08-03T20:49:44.067 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2024-08-03T20:49:44.067 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": "allow", 2024-08-03T20:49:44.067 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow r", 2024-08-03T20:49:44.067 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow rw", 2024-08-03T20:49:44.068 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow rwx" 2024-08-03T20:49:44.068 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-03T20:49:44.068 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-03T20:49:44.068 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-03T20:49:44.068 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "client.2", 2024-08-03T20:49:44.068 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQAxl65mBnbkBBAAjor60WWLTKXidiBMgtKuEA==", 2024-08-03T20:49:44.068 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2024-08-03T20:49:44.068 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": "allow", 2024-08-03T20:49:44.068 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow r", 2024-08-03T20:49:44.068 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow rw", 2024-08-03T20:49:44.068 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow rwx" 2024-08-03T20:49:44.068 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-03T20:49:44.068 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-03T20:49:44.068 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-03T20:49:44.069 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "client.3", 2024-08-03T20:49:44.069 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQAxl65munz8GRAA/cRl3ghwQE42FOizurcW3g==", 2024-08-03T20:49:44.069 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2024-08-03T20:49:44.069 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": "allow", 2024-08-03T20:49:44.069 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow r", 2024-08-03T20:49:44.069 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow rw", 2024-08-03T20:49:44.069 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow rwx" 2024-08-03T20:49:44.069 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-03T20:49:44.069 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-03T20:49:44.069 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-03T20:49:44.069 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "client.admin", 2024-08-03T20:49:44.069 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQAul65mB8VcOhAA105JQ2FX1vhtaGj3Fer8mg==", 2024-08-03T20:49:44.069 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2024-08-03T20:49:44.070 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": "allow *", 2024-08-03T20:49:44.070 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow *", 2024-08-03T20:49:44.070 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow *", 2024-08-03T20:49:44.070 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2024-08-03T20:49:44.070 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-03T20:49:44.070 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-03T20:49:44.070 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-03T20:49:44.070 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "mgr.x", 2024-08-03T20:49:44.070 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQAvl65mUaPKLxAAnll74Gmk8nSuN/HuX6rTxA==", 2024-08-03T20:49:44.070 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2024-08-03T20:49:44.070 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": "allow *", 2024-08-03T20:49:44.070 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow profile mgr", 2024-08-03T20:49:44.070 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2024-08-03T20:49:44.071 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-03T20:49:44.071 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-03T20:49:44.071 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-03T20:49:44.071 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "mgr.y", 2024-08-03T20:49:44.071 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQAvl65m5Gq4JBAA306Ge1zHF3Z0WIzUrset4A==", 2024-08-03T20:49:44.071 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2024-08-03T20:49:44.071 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": "allow *", 2024-08-03T20:49:44.071 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow profile mgr", 2024-08-03T20:49:44.071 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2024-08-03T20:49:44.071 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-03T20:49:44.071 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-03T20:49:44.071 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-03T20:49:44.071 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-08-03T20:49:44.072 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:44.056+0000 7f5552e00640 1 -- 172.21.15.32:0/2226017118 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f5528077010 msgr2=0x7f5528079520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:49:44.073 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:44.056+0000 7f5552e00640 1 --2- 172.21.15.32:0/2226017118 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f5528077010 0x7f5528079520 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f5534033000 tx=0x7f5534004730 comp rx=0 tx=0).stop 2024-08-03T20:49:44.073 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:44.056+0000 7f5552e00640 1 -- 172.21.15.32:0/2226017118 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f554c06f720 msgr2=0x7f554c14f630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:49:44.073 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:44.056+0000 7f5552e00640 1 --2- 172.21.15.32:0/2226017118 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f554c06f720 0x7f554c14f630 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f553c00e9f0 tx=0x7f553c00eeb0 comp rx=0 tx=0).stop 2024-08-03T20:49:44.073 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:44.056+0000 7f5552e00640 1 -- 172.21.15.32:0/2226017118 shutdown_connections 2024-08-03T20:49:44.073 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:44.056+0000 7f5552e00640 1 --2- 172.21.15.32:0/2226017118 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f5528077010 0x7f5528079520 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:49:44.073 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:44.056+0000 7f5552e00640 1 --2- 172.21.15.32:0/2226017118 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f554c0b39a0 0x7f554c1f8c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:49:44.073 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:44.056+0000 7f5552e00640 1 --2- 172.21.15.32:0/2226017118 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f554c162210 0x7f554c15a950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:49:44.073 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:44.056+0000 7f5552e00640 1 --2- 172.21.15.32:0/2226017118 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f554c06f720 0x7f554c14f630 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:49:44.073 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:44.056+0000 7f5552e00640 1 -- 172.21.15.32:0/2226017118 >> 172.21.15.32:0/2226017118 conn(0x7f554c0b1bd0 msgr2=0x7f554c15d5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:49:44.073 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:44.057+0000 7f5552e00640 1 -- 172.21.15.32:0/2226017118 shutdown_connections 2024-08-03T20:49:44.073 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:44.057+0000 7f5552e00640 1 -- 172.21.15.32:0/2226017118 wait complete. 2024-08-03T20:49:44.083 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-08-03T20:49:44.311 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:44.310+0000 7fc1f3a00640 1 -- 172.21.15.32:0/3123879780 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc1ec0b4ce0 msgr2=0x7fc1ec15ef00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:49:44.311 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:44.310+0000 7fc1f3a00640 1 --2- 172.21.15.32:0/3123879780 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc1ec0b4ce0 0x7fc1ec15ef00 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7fc1ec15ee40 tx=0x7fc1d802a560 comp rx=0 tx=0).stop 2024-08-03T20:49:44.312 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:44.310+0000 7fc1f3a00640 1 -- 172.21.15.32:0/3123879780 shutdown_connections 2024-08-03T20:49:44.312 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:44.310+0000 7fc1f3a00640 1 --2- 172.21.15.32:0/3123879780 >> v2:172.21.15.118:3300/0 conn(0x7fc1ec15f440 0x7fc1ec15b910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:49:44.312 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:44.310+0000 7fc1f3a00640 1 --2- 172.21.15.32:0/3123879780 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc1ec0b4ce0 0x7fc1ec15ef00 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:49:44.312 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:44.310+0000 7fc1f3a00640 1 --2- 172.21.15.32:0/3123879780 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fc1ec0b38b0 0x7fc1ec1523f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:49:44.312 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:44.310+0000 7fc1f3a00640 1 -- 172.21.15.32:0/3123879780 >> 172.21.15.32:0/3123879780 conn(0x7fc1ec0b1bd0 msgr2=0x7fc1ec0b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:49:44.312 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:44.311+0000 7fc1f3a00640 1 -- 172.21.15.32:0/3123879780 shutdown_connections 2024-08-03T20:49:44.312 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:44.311+0000 7fc1f3a00640 1 -- 172.21.15.32:0/3123879780 wait complete. 2024-08-03T20:49:44.313 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:44.313+0000 7fc1f3a00640 1 Processor -- start 2024-08-03T20:49:44.313 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:44.313+0000 7fc1f3a00640 1 -- start start 2024-08-03T20:49:44.316 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:44.314+0000 7fc1f3a00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc1ec0b38b0 0x7fc1ec14aef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:49:44.316 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:44.314+0000 7fc1f3a00640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fc1ec0b4ce0 0x7fc1ec072e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:49:44.316 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:44.314+0000 7fc1f3a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc1ec15f440 0x7fc1ec0733d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:49:44.316 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:44.314+0000 7fc1f3a00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fc1ec15c1e0 con 0x7fc1ec15f440 2024-08-03T20:49:44.316 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:44.314+0000 7fc1f3a00640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7fc1ec15c060 con 0x7fc1ec0b38b0 2024-08-03T20:49:44.316 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:44.314+0000 7fc1f3a00640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7fc1ec15c360 con 0x7fc1ec0b4ce0 2024-08-03T20:49:44.317 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:44.315+0000 7fc1f1a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc1ec15f440 0x7fc1ec0733d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:49:44.317 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:44.315+0000 7fc1f1a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc1ec15f440 0x7fc1ec0733d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:35246/0 (socket says 172.21.15.32:35246) 2024-08-03T20:49:44.317 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:44.315+0000 7fc1f1a00640 1 -- 172.21.15.32:0/856059036 learned_addr learned my addr 172.21.15.32:0/856059036 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:49:44.317 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:44.315+0000 7fc1f1000640 1 --2- 172.21.15.32:0/856059036 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc1ec0b38b0 0x7fc1ec14aef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:49:44.317 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:44.315+0000 7fc1ebe00640 1 --2- 172.21.15.32:0/856059036 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fc1ec0b4ce0 0x7fc1ec072e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:49:44.317 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:44.315+0000 7fc1f1a00640 1 -- 172.21.15.32:0/856059036 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fc1ec0b4ce0 msgr2=0x7fc1ec072e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:49:44.317 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:44.315+0000 7fc1f1a00640 1 --2- 172.21.15.32:0/856059036 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fc1ec0b4ce0 0x7fc1ec072e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:49:44.317 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:44.315+0000 7fc1f1a00640 1 -- 172.21.15.32:0/856059036 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc1ec0b38b0 msgr2=0x7fc1ec14aef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:49:44.317 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:44.315+0000 7fc1f1a00640 1 --2- 172.21.15.32:0/856059036 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc1ec0b38b0 0x7fc1ec14aef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:49:44.317 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:44.315+0000 7fc1f1a00640 1 -- 172.21.15.32:0/856059036 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc1ec14c0b0 con 0x7fc1ec15f440 2024-08-03T20:49:44.317 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:44.315+0000 7fc1ebe00640 1 --2- 172.21.15.32:0/856059036 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fc1ec0b4ce0 0x7fc1ec072e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-03T20:49:44.317 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:44.315+0000 7fc1f1a00640 1 --2- 172.21.15.32:0/856059036 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc1ec15f440 0x7fc1ec0733d0 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7fc1e400d9e0 tx=0x7fc1e400dea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:49:44.318 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:44.316+0000 7fc1e8c00640 1 -- 172.21.15.32:0/856059036 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc1e4017070 con 0x7fc1ec15f440 2024-08-03T20:49:44.318 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:44.316+0000 7fc1e8c00640 1 -- 172.21.15.32:0/856059036 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fc1e400be30 con 0x7fc1ec15f440 2024-08-03T20:49:44.318 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:44.316+0000 7fc1f3a00640 1 -- 172.21.15.32:0/856059036 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc1ec159080 con 0x7fc1ec15f440 2024-08-03T20:49:44.318 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:44.316+0000 7fc1e8c00640 1 -- 172.21.15.32:0/856059036 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc1e4003bb0 con 0x7fc1ec15f440 2024-08-03T20:49:44.318 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:44.316+0000 7fc1f3a00640 1 -- 172.21.15.32:0/856059036 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc1ec154210 con 0x7fc1ec15f440 2024-08-03T20:49:44.318 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:44.317+0000 7fc1f3a00640 1 -- 172.21.15.32:0/856059036 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc1ec14c280 con 0x7fc1ec15f440 2024-08-03T20:49:44.318 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:44.318+0000 7fc1e8c00640 1 -- 172.21.15.32:0/856059036 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7fc1e4010460 con 0x7fc1ec15f440 2024-08-03T20:49:44.320 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:44.320+0000 7fc1e8c00640 1 --2- 172.21.15.32:0/856059036 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fc1cc077010 0x7fc1cc079520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:49:44.320 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:44.320+0000 7fc1e8c00640 1 -- 172.21.15.32:0/856059036 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(41..41 src has 1..41) ==== 4857+0+0 (secure 0 0 0) 0x7fc1e4098910 con 0x7fc1ec15f440 2024-08-03T20:49:44.320 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:44.320+0000 7fc1f1000640 1 --2- 172.21.15.32:0/856059036 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fc1cc077010 0x7fc1cc079520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:49:44.321 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:44.321+0000 7fc1f1000640 1 --2- 172.21.15.32:0/856059036 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fc1cc077010 0x7fc1cc079520 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fc1e0005fd0 tx=0x7fc1e000b260 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:49:44.324 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:44.324+0000 7fc1e8c00640 1 -- 172.21.15.32:0/856059036 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7fc1e4061800 con 0x7fc1ec15f440 2024-08-03T20:49:44.532 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:44.531+0000 7fc1f3a00640 1 -- 172.21.15.32:0/856059036 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) -- 0x7fc1ec158df0 con 0x7fc1ec15f440 2024-08-03T20:49:44.532 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:44.532+0000 7fc1e8c00640 1 -- 172.21.15.32:0/856059036 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v41) ==== 84+0+65 (secure 0 0 0) 0x7fc1e4065d50 con 0x7fc1ec15f440 2024-08-03T20:49:44.532 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-03T20:49:44.533 INFO:teuthology.orchestra.run.smithi032.stdout:[ 2024-08-03T20:49:44.533 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-03T20:49:44.533 INFO:teuthology.orchestra.run.smithi032.stdout: "poolnum": 1, 2024-08-03T20:49:44.533 INFO:teuthology.orchestra.run.smithi032.stdout: "poolname": ".mgr" 2024-08-03T20:49:44.533 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-03T20:49:44.533 INFO:teuthology.orchestra.run.smithi032.stdout:] 2024-08-03T20:49:44.535 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:44.534+0000 7fc1f3a00640 1 -- 172.21.15.32:0/856059036 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fc1cc077010 msgr2=0x7fc1cc079520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:49:44.535 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:44.534+0000 7fc1f3a00640 1 --2- 172.21.15.32:0/856059036 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fc1cc077010 0x7fc1cc079520 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fc1e0005fd0 tx=0x7fc1e000b260 comp rx=0 tx=0).stop 2024-08-03T20:49:44.535 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:44.534+0000 7fc1f3a00640 1 -- 172.21.15.32:0/856059036 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc1ec15f440 msgr2=0x7fc1ec0733d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:49:44.535 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:44.534+0000 7fc1f3a00640 1 --2- 172.21.15.32:0/856059036 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc1ec15f440 0x7fc1ec0733d0 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7fc1e400d9e0 tx=0x7fc1e400dea0 comp rx=0 tx=0).stop 2024-08-03T20:49:44.535 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:44.534+0000 7fc1f3a00640 1 -- 172.21.15.32:0/856059036 shutdown_connections 2024-08-03T20:49:44.535 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:44.534+0000 7fc1f3a00640 1 --2- 172.21.15.32:0/856059036 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fc1cc077010 0x7fc1cc079520 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:49:44.535 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:44.534+0000 7fc1f3a00640 1 --2- 172.21.15.32:0/856059036 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc1ec15f440 0x7fc1ec0733d0 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:49:44.535 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:44.534+0000 7fc1f3a00640 1 --2- 172.21.15.32:0/856059036 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fc1ec0b4ce0 0x7fc1ec072e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:49:44.536 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:44.534+0000 7fc1f3a00640 1 --2- 172.21.15.32:0/856059036 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc1ec0b38b0 0x7fc1ec14aef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:49:44.536 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:44.534+0000 7fc1f3a00640 1 -- 172.21.15.32:0/856059036 >> 172.21.15.32:0/856059036 conn(0x7fc1ec0b1bd0 msgr2=0x7fc1ec15a3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:49:44.536 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:44.534+0000 7fc1f3a00640 1 -- 172.21.15.32:0/856059036 shutdown_connections 2024-08-03T20:49:44.536 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:44.534+0000 7fc1f3a00640 1 -- 172.21.15.32:0/856059036 wait complete. 2024-08-03T20:49:44.556 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-08-03T20:49:44.557 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-08-03T20:49:44.789 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:44.788+0000 7f8c24000640 1 -- 172.21.15.32:0/4165792173 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8c1c06ed60 msgr2=0x7f8c1c0ccc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:49:44.789 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:44.788+0000 7f8c24000640 1 --2- 172.21.15.32:0/4165792173 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8c1c06ed60 0x7f8c1c0ccc80 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f8c14005fd0 tx=0x7f8c1402aa90 comp rx=0 tx=0).stop 2024-08-03T20:49:44.790 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:44.789+0000 7f8c24000640 1 -- 172.21.15.32:0/4165792173 shutdown_connections 2024-08-03T20:49:44.790 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:44.789+0000 7f8c24000640 1 --2- 172.21.15.32:0/4165792173 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8c1c06ed60 0x7f8c1c0ccc80 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:49:44.790 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:44.789+0000 7f8c24000640 1 --2- 172.21.15.32:0/4165792173 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f8c1c15d2b0 0x7f8c1c0ced30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:49:44.790 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:44.789+0000 7f8c24000640 1 -- 172.21.15.32:0/4165792173 >> 172.21.15.32:0/4165792173 conn(0x7f8c1c0b1bd0 msgr2=0x7f8c1c0b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:49:44.790 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:44.789+0000 7f8c24000640 1 -- 172.21.15.32:0/4165792173 shutdown_connections 2024-08-03T20:49:44.790 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:44.789+0000 7f8c24000640 1 -- 172.21.15.32:0/4165792173 wait complete. 2024-08-03T20:49:44.791 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:44.791+0000 7f8c24000640 1 Processor -- start 2024-08-03T20:49:44.791 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:44.791+0000 7f8c24000640 1 -- start start 2024-08-03T20:49:44.794 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:44.792+0000 7f8c24000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8c1c06e950 0x7f8c1c0c8340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:49:44.794 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:44.792+0000 7f8c24000640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8c1c06ed60 0x7f8c1c1446a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:49:44.794 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:44.792+0000 7f8c24000640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f8c1c15d2b0 0x7f8c1c144f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:49:44.794 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:44.792+0000 7f8c24000640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f8c1c0b4730 con 0x7f8c1c06e950 2024-08-03T20:49:44.794 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:44.792+0000 7f8c24000640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f8c1c0cf480 con 0x7f8c1c06ed60 2024-08-03T20:49:44.794 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:44.792+0000 7f8c24000640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7f8c1c0cd1c0 con 0x7f8c1c15d2b0 2024-08-03T20:49:44.794 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:44.793+0000 7f8c21600640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8c1c06e950 0x7f8c1c0c8340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:49:44.794 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:44.793+0000 7f8c21600640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8c1c06e950 0x7f8c1c0c8340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:35266/0 (socket says 172.21.15.32:35266) 2024-08-03T20:49:44.794 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:44.793+0000 7f8c21600640 1 -- 172.21.15.32:0/2120699040 learned_addr learned my addr 172.21.15.32:0/2120699040 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:49:44.795 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:44.793+0000 7f8c22000640 1 --2- 172.21.15.32:0/2120699040 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f8c1c15d2b0 0x7f8c1c144f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:49:44.795 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:44.793+0000 7f8c20c00640 1 --2- 172.21.15.32:0/2120699040 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8c1c06ed60 0x7f8c1c1446a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:49:44.795 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:44.793+0000 7f8c21600640 1 -- 172.21.15.32:0/2120699040 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f8c1c15d2b0 msgr2=0x7f8c1c144f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:49:44.795 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:44.793+0000 7f8c21600640 1 --2- 172.21.15.32:0/2120699040 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f8c1c15d2b0 0x7f8c1c144f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:49:44.795 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:44.793+0000 7f8c21600640 1 -- 172.21.15.32:0/2120699040 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8c1c06ed60 msgr2=0x7f8c1c1446a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:49:44.795 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:44.793+0000 7f8c21600640 1 --2- 172.21.15.32:0/2120699040 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8c1c06ed60 0x7f8c1c1446a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:49:44.795 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:44.793+0000 7f8c21600640 1 -- 172.21.15.32:0/2120699040 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8c1c0c72a0 con 0x7f8c1c06e950 2024-08-03T20:49:44.795 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:44.793+0000 7f8c20c00640 1 --2- 172.21.15.32:0/2120699040 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8c1c06ed60 0x7f8c1c1446a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-03T20:49:44.795 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:44.794+0000 7f8c21600640 1 --2- 172.21.15.32:0/2120699040 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8c1c06e950 0x7f8c1c0c8340 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f8c1000d740 tx=0x7f8c1000dc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:49:44.797 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:44.794+0000 7f8c19600640 1 -- 172.21.15.32:0/2120699040 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c10017070 con 0x7f8c1c06e950 2024-08-03T20:49:44.797 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:44.794+0000 7f8c19600640 1 -- 172.21.15.32:0/2120699040 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f8c10009d20 con 0x7f8c1c06e950 2024-08-03T20:49:44.797 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:44.794+0000 7f8c24000640 1 -- 172.21.15.32:0/2120699040 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8c1c0c7580 con 0x7f8c1c06e950 2024-08-03T20:49:44.797 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:44.794+0000 7f8c19600640 1 -- 172.21.15.32:0/2120699040 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c100106c0 con 0x7f8c1c06e950 2024-08-03T20:49:44.797 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:44.794+0000 7f8c24000640 1 -- 172.21.15.32:0/2120699040 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8c1c0c7b40 con 0x7f8c1c06e950 2024-08-03T20:49:44.797 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:44.795+0000 7f8c24000640 1 -- 172.21.15.32:0/2120699040 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8c1c0ccc80 con 0x7f8c1c06e950 2024-08-03T20:49:44.798 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:44.796+0000 7f8c19600640 1 -- 172.21.15.32:0/2120699040 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f8c1001a070 con 0x7f8c1c06e950 2024-08-03T20:49:44.798 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:44.798+0000 7f8c19600640 1 --2- 172.21.15.32:0/2120699040 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f8bfc077070 0x7f8bfc079580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:49:44.798 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:44.798+0000 7f8c19600640 1 -- 172.21.15.32:0/2120699040 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(41..41 src has 1..41) ==== 4857+0+0 (secure 0 0 0) 0x7f8c10098680 con 0x7f8c1c06e950 2024-08-03T20:49:44.799 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:44.798+0000 7f8c20c00640 1 --2- 172.21.15.32:0/2120699040 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f8bfc077070 0x7f8bfc079580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:49:44.799 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:44.799+0000 7f8c20c00640 1 --2- 172.21.15.32:0/2120699040 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f8bfc077070 0x7f8bfc079580 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f8c1c1549a0 tx=0x7f8c08009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:49:44.802 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:44.802+0000 7f8c19600640 1 -- 172.21.15.32:0/2120699040 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7f8c10061bc0 con 0x7f8c1c06e950 2024-08-03T20:49:45.009 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:45.008+0000 7f8c24000640 1 -- 172.21.15.32:0/2120699040 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) -- 0x7f8c1c0b33c0 con 0x7f8c1c06e950 2024-08-03T20:49:45.510 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:45.510+0000 7f8c19600640 1 -- 172.21.15.32:0/2120699040 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v42) ==== 138+0+0 (secure 0 0 0) 0x7f8c100614c0 con 0x7f8c1c06e950 2024-08-03T20:49:45.510 INFO:teuthology.orchestra.run.smithi032.stderr:pool 'cephfs_metadata' created 2024-08-03T20:49:45.514 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:45.513+0000 7f8c24000640 1 -- 172.21.15.32:0/2120699040 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f8bfc077070 msgr2=0x7f8bfc079580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:49:45.514 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:45.513+0000 7f8c24000640 1 --2- 172.21.15.32:0/2120699040 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f8bfc077070 0x7f8bfc079580 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f8c1c1549a0 tx=0x7f8c08009040 comp rx=0 tx=0).stop 2024-08-03T20:49:45.514 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:45.513+0000 7f8c24000640 1 -- 172.21.15.32:0/2120699040 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8c1c06e950 msgr2=0x7f8c1c0c8340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:49:45.514 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:45.513+0000 7f8c24000640 1 --2- 172.21.15.32:0/2120699040 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8c1c06e950 0x7f8c1c0c8340 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f8c1000d740 tx=0x7f8c1000dc00 comp rx=0 tx=0).stop 2024-08-03T20:49:45.514 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:45.513+0000 7f8c24000640 1 -- 172.21.15.32:0/2120699040 shutdown_connections 2024-08-03T20:49:45.514 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:45.513+0000 7f8c24000640 1 --2- 172.21.15.32:0/2120699040 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f8bfc077070 0x7f8bfc079580 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:49:45.514 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:45.513+0000 7f8c24000640 1 --2- 172.21.15.32:0/2120699040 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8c1c06ed60 0x7f8c1c1446a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:49:45.514 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:45.513+0000 7f8c24000640 1 --2- 172.21.15.32:0/2120699040 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f8c1c15d2b0 0x7f8c1c144f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:49:45.514 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:45.513+0000 7f8c24000640 1 --2- 172.21.15.32:0/2120699040 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8c1c06e950 0x7f8c1c0c8340 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:49:45.514 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:45.513+0000 7f8c24000640 1 -- 172.21.15.32:0/2120699040 >> 172.21.15.32:0/2120699040 conn(0x7f8c1c0b1bd0 msgr2=0x7f8c1c15da10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:49:45.514 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:45.513+0000 7f8c24000640 1 -- 172.21.15.32:0/2120699040 shutdown_connections 2024-08-03T20:49:45.514 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:45.513+0000 7f8c24000640 1 -- 172.21.15.32:0/2120699040 wait complete. 2024-08-03T20:49:45.543 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-08-03T20:49:45.780 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:45.779+0000 7f01ef600640 1 -- 172.21.15.32:0/1982387886 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f01e80cf8d0 msgr2=0x7f01e80cc850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:49:45.780 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:45.779+0000 7f01ef600640 1 --2- 172.21.15.32:0/1982387886 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f01e80cf8d0 0x7f01e80cc850 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f01e80ce5c0 tx=0x7f01dc02a410 comp rx=0 tx=0).stop 2024-08-03T20:49:45.780 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:45.779+0000 7f01ef600640 1 -- 172.21.15.32:0/1982387886 shutdown_connections 2024-08-03T20:49:45.780 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:45.779+0000 7f01ef600640 1 --2- 172.21.15.32:0/1982387886 >> v2:172.21.15.118:3300/0 conn(0x7f01e81606f0 0x7f01e81450f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:49:45.780 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:45.779+0000 7f01ef600640 1 --2- 172.21.15.32:0/1982387886 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f01e80cf8d0 0x7f01e80cc850 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:49:45.780 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:45.779+0000 7f01ef600640 1 -- 172.21.15.32:0/1982387886 >> 172.21.15.32:0/1982387886 conn(0x7f01e80b1bd0 msgr2=0x7f01e80b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:49:45.780 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:45.779+0000 7f01ef600640 1 -- 172.21.15.32:0/1982387886 shutdown_connections 2024-08-03T20:49:45.780 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:45.779+0000 7f01ef600640 1 -- 172.21.15.32:0/1982387886 wait complete. 2024-08-03T20:49:45.782 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:45.781+0000 7f01ef600640 1 Processor -- start 2024-08-03T20:49:45.782 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:45.781+0000 7f01ef600640 1 -- start start 2024-08-03T20:49:45.784 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:45.783+0000 7f01ef600640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f01e80b4de0 0x7f01e80c6910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:49:45.784 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:45.783+0000 7f01ef600640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f01e80cf8d0 0x7f01e815b9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:49:45.784 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:45.783+0000 7f01ef600640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f01e815bf20 0x7f01e81f65a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:49:45.784 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:45.783+0000 7f01ef600640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f01e81459f0 con 0x7f01e80cf8d0 2024-08-03T20:49:45.784 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:45.783+0000 7f01ef600640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f01e8145870 con 0x7f01e815bf20 2024-08-03T20:49:45.784 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:45.783+0000 7f01ef600640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7f01e8145b70 con 0x7f01e80b4de0 2024-08-03T20:49:45.784 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:45.783+0000 7f01e7e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f01e80cf8d0 0x7f01e815b9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:49:45.784 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:45.783+0000 7f01e7e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f01e80cf8d0 0x7f01e815b9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:35288/0 (socket says 172.21.15.32:35288) 2024-08-03T20:49:45.784 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:45.783+0000 7f01e7e00640 1 -- 172.21.15.32:0/3780661236 learned_addr learned my addr 172.21.15.32:0/3780661236 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:49:45.785 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:45.783+0000 7f01ecc00640 1 --2- 172.21.15.32:0/3780661236 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f01e80b4de0 0x7f01e80c6910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:49:45.785 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:45.783+0000 7f01ed600640 1 --2- 172.21.15.32:0/3780661236 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f01e815bf20 0x7f01e81f65a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:49:45.785 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:45.784+0000 7f01e7e00640 1 -- 172.21.15.32:0/3780661236 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f01e80b4de0 msgr2=0x7f01e80c6910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:49:45.785 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:45.784+0000 7f01e7e00640 1 --2- 172.21.15.32:0/3780661236 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f01e80b4de0 0x7f01e80c6910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:49:45.785 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:45.784+0000 7f01e7e00640 1 -- 172.21.15.32:0/3780661236 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f01e815bf20 msgr2=0x7f01e81f65a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:49:45.785 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:45.784+0000 7f01e7e00640 1 --2- 172.21.15.32:0/3780661236 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f01e815bf20 0x7f01e81f65a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:49:45.785 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:45.784+0000 7f01e7e00640 1 -- 172.21.15.32:0/3780661236 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f01e81f2de0 con 0x7f01e80cf8d0 2024-08-03T20:49:45.786 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:45.784+0000 7f01e7e00640 1 --2- 172.21.15.32:0/3780661236 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f01e80cf8d0 0x7f01e815b9e0 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f01d400c400 tx=0x7f01d400c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:49:45.786 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:45.784+0000 7f01e4c00640 1 -- 172.21.15.32:0/3780661236 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01d4011790 con 0x7f01e80cf8d0 2024-08-03T20:49:45.786 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:45.784+0000 7f01ef600640 1 -- 172.21.15.32:0/3780661236 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f01e815c330 con 0x7f01e80cf8d0 2024-08-03T20:49:45.786 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:45.784+0000 7f01e4c00640 1 -- 172.21.15.32:0/3780661236 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f01d4017070 con 0x7f01e80cf8d0 2024-08-03T20:49:45.786 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:45.784+0000 7f01e4c00640 1 -- 172.21.15.32:0/3780661236 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01d4011cc0 con 0x7f01e80cf8d0 2024-08-03T20:49:45.786 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:45.784+0000 7f01ef600640 1 -- 172.21.15.32:0/3780661236 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f01e81f3330 con 0x7f01e80cf8d0 2024-08-03T20:49:45.786 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:45.786+0000 7f01ef600640 1 -- 172.21.15.32:0/3780661236 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f01e80cc850 con 0x7f01e80cf8d0 2024-08-03T20:49:45.788 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:45.788+0000 7f01e4c00640 1 -- 172.21.15.32:0/3780661236 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f01d4005020 con 0x7f01e80cf8d0 2024-08-03T20:49:45.789 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:45.789+0000 7f01e4c00640 1 --2- 172.21.15.32:0/3780661236 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f01c8077010 0x7f01c8079520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:49:45.789 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:45.789+0000 7f01e4c00640 1 -- 172.21.15.32:0/3780661236 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(42..42 src has 1..42) ==== 5227+0+0 (secure 0 0 0) 0x7f01d40990d0 con 0x7f01e80cf8d0 2024-08-03T20:49:45.790 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:45.790+0000 7f01ecc00640 1 --2- 172.21.15.32:0/3780661236 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f01c8077010 0x7f01c8079520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:49:45.790 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:45.790+0000 7f01ecc00640 1 --2- 172.21.15.32:0/3780661236 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f01c8077010 0x7f01c8079520 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f01dc034000 tx=0x7f01dc004730 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:49:45.793 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:45.793+0000 7f01e4c00640 1 -- 172.21.15.32:0/3780661236 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7f01e815c330 con 0x7f01e80cf8d0 2024-08-03T20:49:46.000 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:46.000+0000 7f01ef600640 1 -- 172.21.15.32:0/3780661236 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) -- 0x7f01e80c6e60 con 0x7f01e80cf8d0 2024-08-03T20:49:46.518 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:46.517+0000 7f01e4c00640 1 -- 172.21.15.32:0/3780661236 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v43) ==== 170+0+0 (secure 0 0 0) 0x7f01d40625f0 con 0x7f01e80cf8d0 2024-08-03T20:49:46.518 INFO:teuthology.orchestra.run.smithi032.stderr:pool 'cephfs_data' created 2024-08-03T20:49:46.521 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:46.520+0000 7f01ef600640 1 -- 172.21.15.32:0/3780661236 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f01c8077010 msgr2=0x7f01c8079520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:49:46.521 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:46.520+0000 7f01ef600640 1 --2- 172.21.15.32:0/3780661236 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f01c8077010 0x7f01c8079520 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f01dc034000 tx=0x7f01dc004730 comp rx=0 tx=0).stop 2024-08-03T20:49:46.521 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:46.520+0000 7f01ef600640 1 -- 172.21.15.32:0/3780661236 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f01e80cf8d0 msgr2=0x7f01e815b9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:49:46.521 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:46.520+0000 7f01ef600640 1 --2- 172.21.15.32:0/3780661236 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f01e80cf8d0 0x7f01e815b9e0 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f01d400c400 tx=0x7f01d400c8c0 comp rx=0 tx=0).stop 2024-08-03T20:49:46.522 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:46.520+0000 7f01ef600640 1 -- 172.21.15.32:0/3780661236 shutdown_connections 2024-08-03T20:49:46.522 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:46.520+0000 7f01ef600640 1 --2- 172.21.15.32:0/3780661236 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f01c8077010 0x7f01c8079520 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:49:46.522 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:46.520+0000 7f01ef600640 1 --2- 172.21.15.32:0/3780661236 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f01e815bf20 0x7f01e81f65a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:49:46.522 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:46.520+0000 7f01ef600640 1 --2- 172.21.15.32:0/3780661236 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f01e80cf8d0 0x7f01e815b9e0 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:49:46.522 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:46.521+0000 7f01ef600640 1 --2- 172.21.15.32:0/3780661236 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f01e80b4de0 0x7f01e80c6910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:49:46.522 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:46.521+0000 7f01ef600640 1 -- 172.21.15.32:0/3780661236 >> 172.21.15.32:0/3780661236 conn(0x7f01e80b1bd0 msgr2=0x7f01e806f320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:49:46.522 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:46.521+0000 7f01ef600640 1 -- 172.21.15.32:0/3780661236 shutdown_connections 2024-08-03T20:49:46.522 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:46.521+0000 7f01ef600640 1 -- 172.21.15.32:0/3780661236 wait complete. 2024-08-03T20:49:46.568 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-08-03T20:49:46.801 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:46.800+0000 7f3a9e600640 1 -- 172.21.15.32:0/3424058213 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3a98159ff0 msgr2=0x7f3a9815fe10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:49:46.801 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:46.800+0000 7f3a9e600640 1 --2- 172.21.15.32:0/3424058213 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3a98159ff0 0x7f3a9815fe10 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f3a84009830 tx=0x7f3a8402a510 comp rx=0 tx=0).stop 2024-08-03T20:49:46.801 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:46.800+0000 7f3a9e600640 1 -- 172.21.15.32:0/3424058213 shutdown_connections 2024-08-03T20:49:46.801 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:46.800+0000 7f3a9e600640 1 --2- 172.21.15.32:0/3424058213 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f3a98160350 0x7f3a9806e630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:49:46.801 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:46.800+0000 7f3a9e600640 1 --2- 172.21.15.32:0/3424058213 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3a98159ff0 0x7f3a9815fe10 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:49:46.802 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:46.800+0000 7f3a9e600640 1 -- 172.21.15.32:0/3424058213 >> 172.21.15.32:0/3424058213 conn(0x7f3a980cfd80 msgr2=0x7f3a980d0190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:49:46.802 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:46.800+0000 7f3a9e600640 1 -- 172.21.15.32:0/3424058213 shutdown_connections 2024-08-03T20:49:46.802 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:46.800+0000 7f3a9e600640 1 -- 172.21.15.32:0/3424058213 wait complete. 2024-08-03T20:49:46.803 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:46.803+0000 7f3a9e600640 1 Processor -- start 2024-08-03T20:49:46.803 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:46.803+0000 7f3a9e600640 1 -- start start 2024-08-03T20:49:46.805 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:46.804+0000 7f3a9e600640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f3a98159ff0 0x7f3a98156470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:49:46.806 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:46.804+0000 7f3a9e600640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3a98160350 0x7f3a9815bd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:49:46.806 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:46.804+0000 7f3a9e600640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f3a981609d0 0x7f3a980c5e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:49:46.806 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:46.804+0000 7f3a9e600640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f3a9806ef00 con 0x7f3a98160350 2024-08-03T20:49:46.806 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:46.804+0000 7f3a9e600640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f3a9806ed80 con 0x7f3a981609d0 2024-08-03T20:49:46.806 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:46.804+0000 7f3a9e600640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7f3a9806f080 con 0x7f3a98159ff0 2024-08-03T20:49:46.806 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:46.804+0000 7f3a97e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3a98160350 0x7f3a9815bd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:49:46.806 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:46.804+0000 7f3a97e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3a98160350 0x7f3a9815bd70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:35298/0 (socket says 172.21.15.32:35298) 2024-08-03T20:49:46.806 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:46.805+0000 7f3a97e00640 1 -- 172.21.15.32:0/4255450911 learned_addr learned my addr 172.21.15.32:0/4255450911 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:49:46.806 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:46.805+0000 7f3a9d200640 1 --2- 172.21.15.32:0/4255450911 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f3a98159ff0 0x7f3a98156470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:49:46.806 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:46.805+0000 7f3a9dc00640 1 --2- 172.21.15.32:0/4255450911 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f3a981609d0 0x7f3a980c5e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:49:46.806 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:46.805+0000 7f3a97e00640 1 -- 172.21.15.32:0/4255450911 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f3a98159ff0 msgr2=0x7f3a98156470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:49:46.806 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:46.805+0000 7f3a97e00640 1 --2- 172.21.15.32:0/4255450911 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f3a98159ff0 0x7f3a98156470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:49:46.806 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:46.805+0000 7f3a97e00640 1 -- 172.21.15.32:0/4255450911 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f3a981609d0 msgr2=0x7f3a980c5e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:49:46.807 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:46.805+0000 7f3a97e00640 1 --2- 172.21.15.32:0/4255450911 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f3a981609d0 0x7f3a980c5e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:49:46.807 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:46.805+0000 7f3a97e00640 1 -- 172.21.15.32:0/4255450911 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3a9815a440 con 0x7f3a98160350 2024-08-03T20:49:46.807 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:46.805+0000 7f3a97e00640 1 --2- 172.21.15.32:0/4255450911 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3a98160350 0x7f3a9815bd70 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f3a8402ab00 tx=0x7f3a84033c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:49:46.808 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:46.806+0000 7f3a94c00640 1 -- 172.21.15.32:0/4255450911 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a84031d60 con 0x7f3a98160350 2024-08-03T20:49:46.808 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:46.806+0000 7f3a94c00640 1 -- 172.21.15.32:0/4255450911 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f3a84033e10 con 0x7f3a98160350 2024-08-03T20:49:46.808 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:46.806+0000 7f3a94c00640 1 -- 172.21.15.32:0/4255450911 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a84005260 con 0x7f3a98160350 2024-08-03T20:49:46.808 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:46.806+0000 7f3a9e600640 1 -- 172.21.15.32:0/4255450911 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3a9815a940 con 0x7f3a98160350 2024-08-03T20:49:46.808 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:46.806+0000 7f3a9e600640 1 -- 172.21.15.32:0/4255450911 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3a9815af00 con 0x7f3a98160350 2024-08-03T20:49:46.808 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:46.807+0000 7f3a9e600640 1 -- 172.21.15.32:0/4255450911 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3a9806f5a0 con 0x7f3a98160350 2024-08-03T20:49:46.809 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:46.808+0000 7f3a94c00640 1 -- 172.21.15.32:0/4255450911 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f3a84005400 con 0x7f3a98160350 2024-08-03T20:49:46.810 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:46.809+0000 7f3a94c00640 1 --2- 172.21.15.32:0/4255450911 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f3a78077040 0x7f3a78079550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:49:46.810 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:46.809+0000 7f3a94c00640 1 -- 172.21.15.32:0/4255450911 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(43..43 src has 1..43) ==== 5609+0+0 (secure 0 0 0) 0x7f3a840b98f0 con 0x7f3a98160350 2024-08-03T20:49:46.810 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:46.810+0000 7f3a9d200640 1 --2- 172.21.15.32:0/4255450911 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f3a78077040 0x7f3a78079550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:49:46.811 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:46.811+0000 7f3a9d200640 1 --2- 172.21.15.32:0/4255450911 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f3a78077040 0x7f3a78079550 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f3a8c001720 tx=0x7f3a8c007e90 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:49:46.815 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:46.815+0000 7f3a94c00640 1 -- 172.21.15.32:0/4255450911 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7f3a8403f030 con 0x7f3a98160350 2024-08-03T20:49:47.064 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:47.063+0000 7f3a9e600640 1 -- 172.21.15.32:0/4255450911 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) -- 0x7f3a980d09e0 con 0x7f3a98160350 2024-08-03T20:49:47.520 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:47.520+0000 7f3a94c00640 1 -- 172.21.15.32:0/4255450911 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 6 and data pool 7 v31) ==== 172+0+0 (secure 0 0 0) 0x7f3a84082460 con 0x7f3a98160350 2024-08-03T20:49:47.520 INFO:teuthology.orchestra.run.smithi032.stderr:new fs with metadata pool 6 and data pool 7 2024-08-03T20:49:47.524 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:47.523+0000 7f3a9e600640 1 -- 172.21.15.32:0/4255450911 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f3a78077040 msgr2=0x7f3a78079550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:49:47.524 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:47.523+0000 7f3a9e600640 1 --2- 172.21.15.32:0/4255450911 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f3a78077040 0x7f3a78079550 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f3a8c001720 tx=0x7f3a8c007e90 comp rx=0 tx=0).stop 2024-08-03T20:49:47.524 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:47.523+0000 7f3a9e600640 1 -- 172.21.15.32:0/4255450911 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3a98160350 msgr2=0x7f3a9815bd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:49:47.524 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:47.523+0000 7f3a9e600640 1 --2- 172.21.15.32:0/4255450911 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3a98160350 0x7f3a9815bd70 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f3a8402ab00 tx=0x7f3a84033c20 comp rx=0 tx=0).stop 2024-08-03T20:49:47.524 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:47.523+0000 7f3a9e600640 1 -- 172.21.15.32:0/4255450911 shutdown_connections 2024-08-03T20:49:47.524 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:47.523+0000 7f3a9e600640 1 --2- 172.21.15.32:0/4255450911 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f3a78077040 0x7f3a78079550 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:49:47.524 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:47.523+0000 7f3a9e600640 1 --2- 172.21.15.32:0/4255450911 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f3a981609d0 0x7f3a980c5e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:49:47.524 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:47.523+0000 7f3a9e600640 1 --2- 172.21.15.32:0/4255450911 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3a98160350 0x7f3a9815bd70 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:49:47.524 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:47.523+0000 7f3a9e600640 1 --2- 172.21.15.32:0/4255450911 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f3a98159ff0 0x7f3a98156470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:49:47.524 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:47.523+0000 7f3a9e600640 1 -- 172.21.15.32:0/4255450911 >> 172.21.15.32:0/4255450911 conn(0x7f3a980cfd80 msgr2=0x7f3a980af800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:49:47.525 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:47.524+0000 7f3a9e600640 1 -- 172.21.15.32:0/4255450911 shutdown_connections 2024-08-03T20:49:47.525 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:47.524+0000 7f3a9e600640 1 -- 172.21.15.32:0/4255450911 wait complete. 2024-08-03T20:49:47.575 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-08-03T20:49:47.807 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:47.806+0000 7f3c5b200640 1 -- 172.21.15.32:0/2099725002 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f3c54144f90 msgr2=0x7f3c54074570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:49:47.807 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:47.806+0000 7f3c5b200640 1 --2- 172.21.15.32:0/2099725002 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f3c54144f90 0x7f3c54074570 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f3c54073b50 tx=0x7f3c4002a5a0 comp rx=0 tx=0).stop 2024-08-03T20:49:47.807 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:47.806+0000 7f3c5b200640 1 -- 172.21.15.32:0/2099725002 shutdown_connections 2024-08-03T20:49:47.807 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:47.806+0000 7f3c5b200640 1 --2- 172.21.15.32:0/2099725002 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f3c54144f90 0x7f3c54074570 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:49:47.807 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:47.806+0000 7f3c5b200640 1 -- 172.21.15.32:0/2099725002 >> 172.21.15.32:0/2099725002 conn(0x7f3c540b1bd0 msgr2=0x7f3c540b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:49:47.808 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:47.806+0000 7f3c5b200640 1 -- 172.21.15.32:0/2099725002 shutdown_connections 2024-08-03T20:49:47.808 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:47.806+0000 7f3c5b200640 1 -- 172.21.15.32:0/2099725002 wait complete. 2024-08-03T20:49:47.809 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:47.809+0000 7f3c5b200640 1 Processor -- start 2024-08-03T20:49:47.809 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:47.809+0000 7f3c5b200640 1 -- start start 2024-08-03T20:49:47.812 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:47.811+0000 7f3c5b200640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f3c5406e930 0x7f3c5415f160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:49:47.812 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:47.811+0000 7f3c5b200640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f3c54144f90 0x7f3c541f4b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:49:47.812 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:47.811+0000 7f3c5b200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3c5415fff0 0x7f3c5415e010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:49:47.812 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:47.811+0000 7f3c5b200640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f3c540cf840 con 0x7f3c5415fff0 2024-08-03T20:49:47.812 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:47.811+0000 7f3c5b200640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f3c54145f30 con 0x7f3c54144f90 2024-08-03T20:49:47.812 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:47.811+0000 7f3c5b200640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7f3c540cf9c0 con 0x7f3c5406e930 2024-08-03T20:49:47.812 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:47.811+0000 7f3c59200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3c5415fff0 0x7f3c5415e010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:49:47.812 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:47.811+0000 7f3c59200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3c5415fff0 0x7f3c5415e010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:35302/0 (socket says 172.21.15.32:35302) 2024-08-03T20:49:47.813 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:47.811+0000 7f3c59200640 1 -- 172.21.15.32:0/953908100 learned_addr learned my addr 172.21.15.32:0/953908100 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:49:47.813 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:47.811+0000 7f3c53400640 1 --2- 172.21.15.32:0/953908100 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f3c54144f90 0x7f3c541f4b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:49:47.813 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:47.811+0000 7f3c53e00640 1 --2- 172.21.15.32:0/953908100 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f3c5406e930 0x7f3c5415f160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:49:47.813 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:47.811+0000 7f3c59200640 1 -- 172.21.15.32:0/953908100 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f3c5406e930 msgr2=0x7f3c5415f160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:49:47.813 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:47.811+0000 7f3c59200640 1 --2- 172.21.15.32:0/953908100 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f3c5406e930 0x7f3c5415f160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:49:47.813 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:47.811+0000 7f3c59200640 1 -- 172.21.15.32:0/953908100 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f3c54144f90 msgr2=0x7f3c541f4b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:49:47.813 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:47.811+0000 7f3c59200640 1 --2- 172.21.15.32:0/953908100 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f3c54144f90 0x7f3c541f4b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:49:47.813 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:47.811+0000 7f3c59200640 1 -- 172.21.15.32:0/953908100 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3c54141f90 con 0x7f3c5415fff0 2024-08-03T20:49:47.813 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:47.811+0000 7f3c53400640 1 --2- 172.21.15.32:0/953908100 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f3c54144f90 0x7f3c541f4b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-03T20:49:47.814 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:47.812+0000 7f3c59200640 1 --2- 172.21.15.32:0/953908100 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3c5415fff0 0x7f3c5415e010 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f3c4c00caa0 tx=0x7f3c4c00cf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:49:47.814 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:47.812+0000 7f3c4be00640 1 -- 172.21.15.32:0/953908100 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c4c016020 con 0x7f3c5415fff0 2024-08-03T20:49:47.814 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:47.812+0000 7f3c4be00640 1 -- 172.21.15.32:0/953908100 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f3c4c00acf0 con 0x7f3c5415fff0 2024-08-03T20:49:47.814 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:47.812+0000 7f3c4be00640 1 -- 172.21.15.32:0/953908100 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c4c012680 con 0x7f3c5415fff0 2024-08-03T20:49:47.814 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:47.812+0000 7f3c5b200640 1 -- 172.21.15.32:0/953908100 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3c54142200 con 0x7f3c5415fff0 2024-08-03T20:49:47.814 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:47.812+0000 7f3c5b200640 1 -- 172.21.15.32:0/953908100 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3c541427c0 con 0x7f3c5415fff0 2024-08-03T20:49:47.815 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:47.813+0000 7f3c5b200640 1 -- 172.21.15.32:0/953908100 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3c5415f7d0 con 0x7f3c5415fff0 2024-08-03T20:49:47.815 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:47.814+0000 7f3c4be00640 1 -- 172.21.15.32:0/953908100 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f3c4c019020 con 0x7f3c5415fff0 2024-08-03T20:49:47.816 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:47.815+0000 7f3c4be00640 1 --2- 172.21.15.32:0/953908100 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f3c34077010 0x7f3c34079520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:49:47.816 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:47.815+0000 7f3c4be00640 1 -- 172.21.15.32:0/953908100 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(44..44 src has 1..44) ==== 5709+0+0 (secure 0 0 0) 0x7f3c4c098d50 con 0x7f3c5415fff0 2024-08-03T20:49:47.816 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:47.816+0000 7f3c53e00640 1 --2- 172.21.15.32:0/953908100 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f3c34077010 0x7f3c34079520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:49:47.817 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:47.816+0000 7f3c53e00640 1 --2- 172.21.15.32:0/953908100 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f3c34077010 0x7f3c34079520 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f3c44000c00 tx=0x7f3c44009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:49:47.821 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:47.820+0000 7f3c4be00640 1 -- 172.21.15.32:0/953908100 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7f3c4c020030 con 0x7f3c5415fff0 2024-08-03T20:49:48.028 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:48.028+0000 7f3c5b200640 1 -- 172.21.15.32:0/953908100 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f3c540d0ed0 con 0x7f3c5415fff0 2024-08-03T20:49:48.029 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:48.029+0000 7f3c4be00640 1 -- 172.21.15.32:0/953908100 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v44) ==== 74+0+16566 (secure 0 0 0) 0x7f3c540d0ed0 con 0x7f3c5415fff0 2024-08-03T20:49:48.030 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-03T20:49:48.030 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":44,"fsid":"40ab141a-01d7-49b9-ae3c-2edec9876e95","created":"2024-08-03T20:47:14.957362+0000","modified":"2024-08-03T20:49:47.064701+0000","last_up_change":"2024-08-03T20:47:28.874451+0000","last_in_change":"2024-08-03T20:47:20.079442+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-08-03T20:47:28.261300+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_metadata","create_time":"2024-08-03T20:49:45.020976+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"44","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_data","create_time":"2024-08-03T20:49:46.012432+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"44","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5a83e759-9035-438b-889a-0a6af1f0a0d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6816","nonce":3066156116},{"type":"v1","addr":"172.21.15.32:6817","nonce":3066156116}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6818","nonce":3066156116},{"type":"v1","addr":"172.21.15.32:6819","nonce":3066156116}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6822","nonce":3066156116},{"type":"v1","addr":"172.21.15.32:6823","nonce":3066156116}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6820","nonce":3066156116},{"type":"v1","addr":"172.21.15.32:6821","nonce":3066156116}]},"public_addr":"172.21.15.32:6817/3066156116","cluster_addr":"172.21.15.32:6819/3066156116","heartbeat_back_addr":"172.21.15.32:6823/3066156116","heartbeat_front_addr":"172.21.15.32:6821/3066156116","state":["exists","up"]},{"osd":1,"uuid":"d7c5c210-da4f-49f9-b9e7-1dfb02daa4bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6824","nonce":1551241759},{"type":"v1","addr":"172.21.15.32:6825","nonce":1551241759}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6826","nonce":1551241759},{"type":"v1","addr":"172.21.15.32:6827","nonce":1551241759}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6830","nonce":1551241759},{"type":"v1","addr":"172.21.15.32:6831","nonce":1551241759}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6828","nonce":1551241759},{"type":"v1","addr":"172.21.15.32:6829","nonce":1551241759}]},"public_addr":"172.21.15.32:6825/1551241759","cluster_addr":"172.21.15.32:6827/1551241759","heartbeat_back_addr":"172.21.15.32:6831/1551241759","heartbeat_front_addr":"172.21.15.32:6829/1551241759","state":["exists","up"]},{"osd":2,"uuid":"57f2e6de-a201-45c2-8981-8e56a4875ebd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6808","nonce":3806708944},{"type":"v1","addr":"172.21.15.32:6809","nonce":3806708944}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6810","nonce":3806708944},{"type":"v1","addr":"172.21.15.32:6811","nonce":3806708944}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6814","nonce":3806708944},{"type":"v1","addr":"172.21.15.32:6815","nonce":3806708944}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6812","nonce":3806708944},{"type":"v1","addr":"172.21.15.32:6813","nonce":3806708944}]},"public_addr":"172.21.15.32:6809/3806708944","cluster_addr":"172.21.15.32:6811/3806708944","heartbeat_back_addr":"172.21.15.32:6815/3806708944","heartbeat_front_addr":"172.21.15.32:6813/3806708944","state":["exists","up"]},{"osd":3,"uuid":"6439a280-da45-4148-8130-f16323935ff3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6800","nonce":9088303},{"type":"v1","addr":"172.21.15.32:6801","nonce":9088303}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6802","nonce":9088303},{"type":"v1","addr":"172.21.15.32:6803","nonce":9088303}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6806","nonce":9088303},{"type":"v1","addr":"172.21.15.32:6807","nonce":9088303}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6804","nonce":9088303},{"type":"v1","addr":"172.21.15.32:6805","nonce":9088303}]},"public_addr":"172.21.15.32:6801/9088303","cluster_addr":"172.21.15.32:6803/9088303","heartbeat_back_addr":"172.21.15.32:6807/9088303","heartbeat_front_addr":"172.21.15.32:6805/9088303","state":["exists","up"]},{"osd":4,"uuid":"4f902eff-1e18-418d-82c9-2151c69524fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6802","nonce":864752275},{"type":"v1","addr":"172.21.15.118:6803","nonce":864752275}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6804","nonce":864752275},{"type":"v1","addr":"172.21.15.118:6805","nonce":864752275}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6808","nonce":864752275},{"type":"v1","addr":"172.21.15.118:6809","nonce":864752275}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6806","nonce":864752275},{"type":"v1","addr":"172.21.15.118:6807","nonce":864752275}]},"public_addr":"172.21.15.118:6803/864752275","cluster_addr":"172.21.15.118:6805/864752275","heartbeat_back_addr":"172.21.15.118:6809/864752275","heartbeat_front_addr":"172.21.15.118:6807/864752275","state":["exists","up"]},{"osd":5,"uuid":"bfd9feac-dd06-458c-91b9-af7f175f612c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6810","nonce":2724891770},{"type":"v1","addr":"172.21.15.118:6811","nonce":2724891770}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6812","nonce":2724891770},{"type":"v1","addr":"172.21.15.118:6813","nonce":2724891770}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6816","nonce":2724891770},{"type":"v1","addr":"172.21.15.118:6817","nonce":2724891770}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6814","nonce":2724891770},{"type":"v1","addr":"172.21.15.118:6815","nonce":2724891770}]},"public_addr":"172.21.15.118:6811/2724891770","cluster_addr":"172.21.15.118:6813/2724891770","heartbeat_back_addr":"172.21.15.118:6817/2724891770","heartbeat_front_addr":"172.21.15.118:6815/2724891770","state":["exists","up"]},{"osd":6,"uuid":"5ed5a279-9d8e-416d-bbc7-7edd3e6d5705","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6825","nonce":801947386},{"type":"v1","addr":"172.21.15.118:6827","nonce":801947386}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6828","nonce":801947386},{"type":"v1","addr":"172.21.15.118:6829","nonce":801947386}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6832","nonce":801947386},{"type":"v1","addr":"172.21.15.118:6833","nonce":801947386}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6830","nonce":801947386},{"type":"v1","addr":"172.21.15.118:6831","nonce":801947386}]},"public_addr":"172.21.15.118:6827/801947386","cluster_addr":"172.21.15.118:6829/801947386","heartbeat_back_addr":"172.21.15.118:6833/801947386","heartbeat_front_addr":"172.21.15.118:6831/801947386","state":["exists","up"]},{"osd":7,"uuid":"f2086c47-875e-4451-a436-2008d1ea16af","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6818","nonce":653000048},{"type":"v1","addr":"172.21.15.118:6819","nonce":653000048}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6820","nonce":653000048},{"type":"v1","addr":"172.21.15.118:6821","nonce":653000048}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6824","nonce":653000048},{"type":"v1","addr":"172.21.15.118:6826","nonce":653000048}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6822","nonce":653000048},{"type":"v1","addr":"172.21.15.118:6823","nonce":653000048}]},"public_addr":"172.21.15.118:6819/653000048","cluster_addr":"172.21.15.118:6821/653000048","heartbeat_back_addr":"172.21.15.118:6826/653000048","heartbeat_front_addr":"172.21.15.118:6823/653000048","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-03T20:47:25.491994+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-03T20:47:25.587358+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-03T20:47:25.454175+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-03T20:47:25.377734+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-03T20:47:25.412351+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-03T20:47:25.499815+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-03T20:47:25.575044+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-03T20:47:25.552383+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-08-03T20:49:48.032 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:48.031+0000 7f3c5b200640 1 -- 172.21.15.32:0/953908100 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f3c34077010 msgr2=0x7f3c34079520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:49:48.032 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:48.031+0000 7f3c5b200640 1 --2- 172.21.15.32:0/953908100 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f3c34077010 0x7f3c34079520 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f3c44000c00 tx=0x7f3c44009040 comp rx=0 tx=0).stop 2024-08-03T20:49:48.032 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:48.031+0000 7f3c5b200640 1 -- 172.21.15.32:0/953908100 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3c5415fff0 msgr2=0x7f3c5415e010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:49:48.033 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:48.031+0000 7f3c5b200640 1 --2- 172.21.15.32:0/953908100 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3c5415fff0 0x7f3c5415e010 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f3c4c00caa0 tx=0x7f3c4c00cf60 comp rx=0 tx=0).stop 2024-08-03T20:49:48.033 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:48.032+0000 7f3c5b200640 1 -- 172.21.15.32:0/953908100 shutdown_connections 2024-08-03T20:49:48.033 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:48.032+0000 7f3c5b200640 1 --2- 172.21.15.32:0/953908100 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f3c34077010 0x7f3c34079520 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:49:48.033 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:48.032+0000 7f3c5b200640 1 --2- 172.21.15.32:0/953908100 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f3c54144f90 0x7f3c541f4b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:49:48.033 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:48.032+0000 7f3c5b200640 1 --2- 172.21.15.32:0/953908100 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3c5415fff0 0x7f3c5415e010 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:49:48.033 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:48.032+0000 7f3c5b200640 1 --2- 172.21.15.32:0/953908100 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f3c5406e930 0x7f3c5415f160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:49:48.033 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:48.032+0000 7f3c5b200640 1 -- 172.21.15.32:0/953908100 >> 172.21.15.32:0/953908100 conn(0x7f3c540b1bd0 msgr2=0x7f3c540b09e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:49:48.033 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:48.032+0000 7f3c5b200640 1 -- 172.21.15.32:0/953908100 shutdown_connections 2024-08-03T20:49:48.033 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:48.032+0000 7f3c5b200640 1 -- 172.21.15.32:0/953908100 wait complete. 2024-08-03T20:49:48.054 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-08-03T20:49:48.282 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:48.281+0000 7f6778e00640 1 -- 172.21.15.32:0/2923485109 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f677415fff0 msgr2=0x7f6774160400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:49:48.282 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:48.281+0000 7f6778e00640 1 --2- 172.21.15.32:0/2923485109 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f677415fff0 0x7f6774160400 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f676c005fd0 tx=0x7f676c02a840 comp rx=0 tx=0).stop 2024-08-03T20:49:48.283 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:48.282+0000 7f6778e00640 1 -- 172.21.15.32:0/2923485109 shutdown_connections 2024-08-03T20:49:48.283 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:48.282+0000 7f6778e00640 1 --2- 172.21.15.32:0/2923485109 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f677415fff0 0x7f6774160400 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:49:48.283 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:48.282+0000 7f6778e00640 1 --2- 172.21.15.32:0/2923485109 >> v2:172.21.15.118:3300/0 conn(0x7f6774144f90 0x7f6774074570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:49:48.283 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:48.282+0000 7f6778e00640 1 -- 172.21.15.32:0/2923485109 >> 172.21.15.32:0/2923485109 conn(0x7f67740b1bd0 msgr2=0x7f67740b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:49:48.283 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:48.282+0000 7f6778e00640 1 -- 172.21.15.32:0/2923485109 shutdown_connections 2024-08-03T20:49:48.283 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:48.282+0000 7f6778e00640 1 -- 172.21.15.32:0/2923485109 wait complete. 2024-08-03T20:49:48.284 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:48.284+0000 7f6778e00640 1 Processor -- start 2024-08-03T20:49:48.284 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:48.284+0000 7f6778e00640 1 -- start start 2024-08-03T20:49:48.286 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:48.285+0000 7f6778e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f677406e930 0x7f6774143da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:49:48.286 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:48.285+0000 7f6778e00640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f6774144f90 0x7f67740aa4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:49:48.286 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:48.285+0000 7f6778e00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f677415fff0 0x7f677415bd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:49:48.286 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:48.285+0000 7f6778e00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f67740cf840 con 0x7f677406e930 2024-08-03T20:49:48.287 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:48.285+0000 7f6778e00640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f6774145f30 con 0x7f677415fff0 2024-08-03T20:49:48.287 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:48.285+0000 7f6778e00640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7f67740cf9c0 con 0x7f6774144f90 2024-08-03T20:49:48.287 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:48.286+0000 7f6771e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f677406e930 0x7f6774143da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:49:48.287 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:48.286+0000 7f6771e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f677406e930 0x7f6774143da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:35312/0 (socket says 172.21.15.32:35312) 2024-08-03T20:49:48.287 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:48.286+0000 7f6771e00640 1 -- 172.21.15.32:0/3808544381 learned_addr learned my addr 172.21.15.32:0/3808544381 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:49:48.287 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:48.286+0000 7f676be00640 1 --2- 172.21.15.32:0/3808544381 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f6774144f90 0x7f67740aa4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:49:48.287 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:48.286+0000 7f6772800640 1 --2- 172.21.15.32:0/3808544381 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f677415fff0 0x7f677415bd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:49:48.288 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:48.286+0000 7f6771e00640 1 -- 172.21.15.32:0/3808544381 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f6774144f90 msgr2=0x7f67740aa4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:49:48.288 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:48.286+0000 7f6771e00640 1 --2- 172.21.15.32:0/3808544381 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f6774144f90 0x7f67740aa4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:49:48.288 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:48.286+0000 7f6771e00640 1 -- 172.21.15.32:0/3808544381 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f677415fff0 msgr2=0x7f677415bd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:49:48.288 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:48.286+0000 7f6771e00640 1 --2- 172.21.15.32:0/3808544381 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f677415fff0 0x7f677415bd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:49:48.288 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:48.286+0000 7f6771e00640 1 -- 172.21.15.32:0/3808544381 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f67740a9700 con 0x7f677406e930 2024-08-03T20:49:48.288 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:48.287+0000 7f6771e00640 1 --2- 172.21.15.32:0/3808544381 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f677406e930 0x7f6774143da0 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f676400c530 tx=0x7f676400c9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:49:48.289 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:48.287+0000 7f676a000640 1 -- 172.21.15.32:0/3808544381 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6764016020 con 0x7f677406e930 2024-08-03T20:49:48.289 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:48.287+0000 7f676a000640 1 -- 172.21.15.32:0/3808544381 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f6764012940 con 0x7f677406e930 2024-08-03T20:49:48.289 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:48.287+0000 7f6778e00640 1 -- 172.21.15.32:0/3808544381 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f67740a99e0 con 0x7f677406e930 2024-08-03T20:49:48.289 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:48.287+0000 7f676a000640 1 -- 172.21.15.32:0/3808544381 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6764012d90 con 0x7f677406e930 2024-08-03T20:49:48.289 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:48.287+0000 7f6778e00640 1 -- 172.21.15.32:0/3808544381 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f67740a9fa0 con 0x7f677406e930 2024-08-03T20:49:48.289 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:48.288+0000 7f6778e00640 1 -- 172.21.15.32:0/3808544381 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6774160cc0 con 0x7f677406e930 2024-08-03T20:49:48.290 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:48.290+0000 7f676a000640 1 -- 172.21.15.32:0/3808544381 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f6764019020 con 0x7f677406e930 2024-08-03T20:49:48.291 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:48.291+0000 7f676a000640 1 --2- 172.21.15.32:0/3808544381 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f6750077010 0x7f6750079520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:49:48.292 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:48.291+0000 7f676a000640 1 -- 172.21.15.32:0/3808544381 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(44..44 src has 1..44) ==== 5709+0+0 (secure 0 0 0) 0x7f6764098960 con 0x7f677406e930 2024-08-03T20:49:48.292 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:48.291+0000 7f676be00640 1 --2- 172.21.15.32:0/3808544381 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f6750077010 0x7f6750079520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:49:48.292 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:48.292+0000 7f676be00640 1 --2- 172.21.15.32:0/3808544381 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f6750077010 0x7f6750079520 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f67741446e0 tx=0x7f675c005f90 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:49:48.296 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:48.296+0000 7f676a000640 1 -- 172.21.15.32:0/3808544381 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7f6764020020 con 0x7f677406e930 2024-08-03T20:49:48.502 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:48.501+0000 7f6778e00640 1 -- 172.21.15.32:0/3808544381 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f67740b4e70 con 0x7f677406e930 2024-08-03T20:49:48.503 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:48.503+0000 7f676a000640 1 -- 172.21.15.32:0/3808544381 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v44) ==== 74+0+16566 (secure 0 0 0) 0x7f6764061500 con 0x7f677406e930 2024-08-03T20:49:48.504 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-03T20:49:48.504 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":44,"fsid":"40ab141a-01d7-49b9-ae3c-2edec9876e95","created":"2024-08-03T20:47:14.957362+0000","modified":"2024-08-03T20:49:47.064701+0000","last_up_change":"2024-08-03T20:47:28.874451+0000","last_in_change":"2024-08-03T20:47:20.079442+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-08-03T20:47:28.261300+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_metadata","create_time":"2024-08-03T20:49:45.020976+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"44","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_data","create_time":"2024-08-03T20:49:46.012432+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"44","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5a83e759-9035-438b-889a-0a6af1f0a0d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6816","nonce":3066156116},{"type":"v1","addr":"172.21.15.32:6817","nonce":3066156116}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6818","nonce":3066156116},{"type":"v1","addr":"172.21.15.32:6819","nonce":3066156116}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6822","nonce":3066156116},{"type":"v1","addr":"172.21.15.32:6823","nonce":3066156116}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6820","nonce":3066156116},{"type":"v1","addr":"172.21.15.32:6821","nonce":3066156116}]},"public_addr":"172.21.15.32:6817/3066156116","cluster_addr":"172.21.15.32:6819/3066156116","heartbeat_back_addr":"172.21.15.32:6823/3066156116","heartbeat_front_addr":"172.21.15.32:6821/3066156116","state":["exists","up"]},{"osd":1,"uuid":"d7c5c210-da4f-49f9-b9e7-1dfb02daa4bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6824","nonce":1551241759},{"type":"v1","addr":"172.21.15.32:6825","nonce":1551241759}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6826","nonce":1551241759},{"type":"v1","addr":"172.21.15.32:6827","nonce":1551241759}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6830","nonce":1551241759},{"type":"v1","addr":"172.21.15.32:6831","nonce":1551241759}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6828","nonce":1551241759},{"type":"v1","addr":"172.21.15.32:6829","nonce":1551241759}]},"public_addr":"172.21.15.32:6825/1551241759","cluster_addr":"172.21.15.32:6827/1551241759","heartbeat_back_addr":"172.21.15.32:6831/1551241759","heartbeat_front_addr":"172.21.15.32:6829/1551241759","state":["exists","up"]},{"osd":2,"uuid":"57f2e6de-a201-45c2-8981-8e56a4875ebd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6808","nonce":3806708944},{"type":"v1","addr":"172.21.15.32:6809","nonce":3806708944}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6810","nonce":3806708944},{"type":"v1","addr":"172.21.15.32:6811","nonce":3806708944}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6814","nonce":3806708944},{"type":"v1","addr":"172.21.15.32:6815","nonce":3806708944}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6812","nonce":3806708944},{"type":"v1","addr":"172.21.15.32:6813","nonce":3806708944}]},"public_addr":"172.21.15.32:6809/3806708944","cluster_addr":"172.21.15.32:6811/3806708944","heartbeat_back_addr":"172.21.15.32:6815/3806708944","heartbeat_front_addr":"172.21.15.32:6813/3806708944","state":["exists","up"]},{"osd":3,"uuid":"6439a280-da45-4148-8130-f16323935ff3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6800","nonce":9088303},{"type":"v1","addr":"172.21.15.32:6801","nonce":9088303}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6802","nonce":9088303},{"type":"v1","addr":"172.21.15.32:6803","nonce":9088303}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6806","nonce":9088303},{"type":"v1","addr":"172.21.15.32:6807","nonce":9088303}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6804","nonce":9088303},{"type":"v1","addr":"172.21.15.32:6805","nonce":9088303}]},"public_addr":"172.21.15.32:6801/9088303","cluster_addr":"172.21.15.32:6803/9088303","heartbeat_back_addr":"172.21.15.32:6807/9088303","heartbeat_front_addr":"172.21.15.32:6805/9088303","state":["exists","up"]},{"osd":4,"uuid":"4f902eff-1e18-418d-82c9-2151c69524fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6802","nonce":864752275},{"type":"v1","addr":"172.21.15.118:6803","nonce":864752275}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6804","nonce":864752275},{"type":"v1","addr":"172.21.15.118:6805","nonce":864752275}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6808","nonce":864752275},{"type":"v1","addr":"172.21.15.118:6809","nonce":864752275}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6806","nonce":864752275},{"type":"v1","addr":"172.21.15.118:6807","nonce":864752275}]},"public_addr":"172.21.15.118:6803/864752275","cluster_addr":"172.21.15.118:6805/864752275","heartbeat_back_addr":"172.21.15.118:6809/864752275","heartbeat_front_addr":"172.21.15.118:6807/864752275","state":["exists","up"]},{"osd":5,"uuid":"bfd9feac-dd06-458c-91b9-af7f175f612c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6810","nonce":2724891770},{"type":"v1","addr":"172.21.15.118:6811","nonce":2724891770}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6812","nonce":2724891770},{"type":"v1","addr":"172.21.15.118:6813","nonce":2724891770}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6816","nonce":2724891770},{"type":"v1","addr":"172.21.15.118:6817","nonce":2724891770}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6814","nonce":2724891770},{"type":"v1","addr":"172.21.15.118:6815","nonce":2724891770}]},"public_addr":"172.21.15.118:6811/2724891770","cluster_addr":"172.21.15.118:6813/2724891770","heartbeat_back_addr":"172.21.15.118:6817/2724891770","heartbeat_front_addr":"172.21.15.118:6815/2724891770","state":["exists","up"]},{"osd":6,"uuid":"5ed5a279-9d8e-416d-bbc7-7edd3e6d5705","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6825","nonce":801947386},{"type":"v1","addr":"172.21.15.118:6827","nonce":801947386}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6828","nonce":801947386},{"type":"v1","addr":"172.21.15.118:6829","nonce":801947386}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6832","nonce":801947386},{"type":"v1","addr":"172.21.15.118:6833","nonce":801947386}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6830","nonce":801947386},{"type":"v1","addr":"172.21.15.118:6831","nonce":801947386}]},"public_addr":"172.21.15.118:6827/801947386","cluster_addr":"172.21.15.118:6829/801947386","heartbeat_back_addr":"172.21.15.118:6833/801947386","heartbeat_front_addr":"172.21.15.118:6831/801947386","state":["exists","up"]},{"osd":7,"uuid":"f2086c47-875e-4451-a436-2008d1ea16af","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6818","nonce":653000048},{"type":"v1","addr":"172.21.15.118:6819","nonce":653000048}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6820","nonce":653000048},{"type":"v1","addr":"172.21.15.118:6821","nonce":653000048}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6824","nonce":653000048},{"type":"v1","addr":"172.21.15.118:6826","nonce":653000048}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6822","nonce":653000048},{"type":"v1","addr":"172.21.15.118:6823","nonce":653000048}]},"public_addr":"172.21.15.118:6819/653000048","cluster_addr":"172.21.15.118:6821/653000048","heartbeat_back_addr":"172.21.15.118:6826/653000048","heartbeat_front_addr":"172.21.15.118:6823/653000048","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-03T20:47:25.491994+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-03T20:47:25.587358+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-03T20:47:25.454175+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-03T20:47:25.377734+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-03T20:47:25.412351+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-03T20:47:25.499815+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-03T20:47:25.575044+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-03T20:47:25.552383+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-08-03T20:49:48.506 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:48.505+0000 7f6778e00640 1 -- 172.21.15.32:0/3808544381 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f6750077010 msgr2=0x7f6750079520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:49:48.506 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:48.505+0000 7f6778e00640 1 --2- 172.21.15.32:0/3808544381 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f6750077010 0x7f6750079520 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f67741446e0 tx=0x7f675c005f90 comp rx=0 tx=0).stop 2024-08-03T20:49:48.506 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:48.505+0000 7f6778e00640 1 -- 172.21.15.32:0/3808544381 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f677406e930 msgr2=0x7f6774143da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:49:48.506 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:48.505+0000 7f6778e00640 1 --2- 172.21.15.32:0/3808544381 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f677406e930 0x7f6774143da0 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f676400c530 tx=0x7f676400c9f0 comp rx=0 tx=0).stop 2024-08-03T20:49:48.506 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:48.505+0000 7f6778e00640 1 -- 172.21.15.32:0/3808544381 shutdown_connections 2024-08-03T20:49:48.506 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:48.505+0000 7f6778e00640 1 --2- 172.21.15.32:0/3808544381 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f6750077010 0x7f6750079520 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:49:48.506 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:48.505+0000 7f6778e00640 1 --2- 172.21.15.32:0/3808544381 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f677415fff0 0x7f677415bd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:49:48.506 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:48.505+0000 7f6778e00640 1 --2- 172.21.15.32:0/3808544381 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f6774144f90 0x7f67740aa4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:49:48.506 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:48.505+0000 7f6778e00640 1 --2- 172.21.15.32:0/3808544381 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f677406e930 0x7f6774143da0 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:49:48.507 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:48.505+0000 7f6778e00640 1 -- 172.21.15.32:0/3808544381 >> 172.21.15.32:0/3808544381 conn(0x7f67740b1bd0 msgr2=0x7f67740cc540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:49:48.507 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:48.505+0000 7f6778e00640 1 -- 172.21.15.32:0/3808544381 shutdown_connections 2024-08-03T20:49:48.507 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:48.505+0000 7f6778e00640 1 -- 172.21.15.32:0/3808544381 wait complete. 2024-08-03T20:49:48.528 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-08-03T20:49:48.765 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:48.764+0000 7f81d8400640 1 -- 172.21.15.32:0/4115796472 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f81d00cccf0 msgr2=0x7f81d00cd100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:49:48.765 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:48.764+0000 7f81d8400640 1 --2- 172.21.15.32:0/4115796472 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f81d00cccf0 0x7f81d00cd100 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f81bc009830 tx=0x7f81bc02a530 comp rx=0 tx=0).stop 2024-08-03T20:49:48.765 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:48.764+0000 7f81d8400640 1 -- 172.21.15.32:0/4115796472 shutdown_connections 2024-08-03T20:49:48.765 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:48.764+0000 7f81d8400640 1 --2- 172.21.15.32:0/4115796472 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f81d00cccf0 0x7f81d00cd100 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:49:48.765 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:48.764+0000 7f81d8400640 1 -- 172.21.15.32:0/4115796472 >> 172.21.15.32:0/4115796472 conn(0x7f81d00b1bd0 msgr2=0x7f81d00b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:49:48.765 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:48.764+0000 7f81d8400640 1 -- 172.21.15.32:0/4115796472 shutdown_connections 2024-08-03T20:49:48.765 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:48.764+0000 7f81d8400640 1 -- 172.21.15.32:0/4115796472 wait complete. 2024-08-03T20:49:48.766 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:48.766+0000 7f81d8400640 1 Processor -- start 2024-08-03T20:49:48.766 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:48.766+0000 7f81d8400640 1 -- start start 2024-08-03T20:49:48.769 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:48.768+0000 7f81d8400640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f81d00cccf0 0x7f81d0159a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:49:48.769 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:48.768+0000 7f81d8400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f81d00ced30 0x7f81d0144560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:49:48.769 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:48.768+0000 7f81d8400640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f81d0145c50 0x7f81d0144aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:49:48.769 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:48.768+0000 7f81d8400640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f81d00b41e0 con 0x7f81d00ced30 2024-08-03T20:49:48.769 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:48.768+0000 7f81d8400640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f81d00b4060 con 0x7f81d0145c50 2024-08-03T20:49:48.769 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:48.768+0000 7f81d8400640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7f81d00b4360 con 0x7f81d00cccf0 2024-08-03T20:49:48.769 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:48.768+0000 7f81d5000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f81d00ced30 0x7f81d0144560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:49:48.769 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:48.768+0000 7f81d5000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f81d00ced30 0x7f81d0144560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:35318/0 (socket says 172.21.15.32:35318) 2024-08-03T20:49:48.770 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:48.768+0000 7f81d5000640 1 -- 172.21.15.32:0/627751922 learned_addr learned my addr 172.21.15.32:0/627751922 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:49:48.770 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:48.768+0000 7f81d6400640 1 --2- 172.21.15.32:0/627751922 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f81d0145c50 0x7f81d0144aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:49:48.770 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:48.768+0000 7f81d5a00640 1 --2- 172.21.15.32:0/627751922 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f81d00cccf0 0x7f81d0159a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:49:48.770 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:48.768+0000 7f81d5000640 1 -- 172.21.15.32:0/627751922 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f81d00cccf0 msgr2=0x7f81d0159a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:49:48.770 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:48.768+0000 7f81d5000640 1 --2- 172.21.15.32:0/627751922 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f81d00cccf0 0x7f81d0159a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:49:48.770 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:48.768+0000 7f81d5000640 1 -- 172.21.15.32:0/627751922 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f81d0145c50 msgr2=0x7f81d0144aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:49:48.770 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:48.768+0000 7f81d5000640 1 --2- 172.21.15.32:0/627751922 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f81d0145c50 0x7f81d0144aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:49:48.770 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:48.768+0000 7f81d5000640 1 -- 172.21.15.32:0/627751922 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f81d01394f0 con 0x7f81d00ced30 2024-08-03T20:49:48.770 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:48.769+0000 7f81d5000640 1 --2- 172.21.15.32:0/627751922 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f81d00ced30 0x7f81d0144560 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f81bc009950 tx=0x7f81bc033ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:49:48.771 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:48.769+0000 7f81cd600640 1 -- 172.21.15.32:0/627751922 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f81bc031b50 con 0x7f81d00ced30 2024-08-03T20:49:48.771 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:48.769+0000 7f81cd600640 1 -- 172.21.15.32:0/627751922 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f81bc031de0 con 0x7f81d00ced30 2024-08-03T20:49:48.771 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:48.769+0000 7f81cd600640 1 -- 172.21.15.32:0/627751922 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f81bc005280 con 0x7f81d00ced30 2024-08-03T20:49:48.771 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:48.769+0000 7f81d8400640 1 -- 172.21.15.32:0/627751922 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f81d0159f80 con 0x7f81d00ced30 2024-08-03T20:49:48.771 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:48.769+0000 7f81d8400640 1 -- 172.21.15.32:0/627751922 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f81d01f8cb0 con 0x7f81d00ced30 2024-08-03T20:49:48.771 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:48.770+0000 7f81d8400640 1 -- 172.21.15.32:0/627751922 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f81d013a530 con 0x7f81d00ced30 2024-08-03T20:49:48.772 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:48.772+0000 7f81cd600640 1 -- 172.21.15.32:0/627751922 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f81bc005420 con 0x7f81d00ced30 2024-08-03T20:49:48.773 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:48.772+0000 7f81cd600640 1 --2- 172.21.15.32:0/627751922 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f81b0077040 0x7f81b0079550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:49:48.773 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:48.773+0000 7f81cd600640 1 -- 172.21.15.32:0/627751922 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(45..45 src has 1..45) ==== 5709+0+0 (secure 0 0 0) 0x7f81bc0b9aa0 con 0x7f81d00ced30 2024-08-03T20:49:48.773 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:48.773+0000 7f81d5a00640 1 --2- 172.21.15.32:0/627751922 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f81b0077040 0x7f81b0079550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:49:48.774 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:48.774+0000 7f81d5a00640 1 --2- 172.21.15.32:0/627751922 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f81b0077040 0x7f81b0079550 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f81c4000f60 tx=0x7f81c4009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:49:48.777 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:48.777+0000 7f81cd600640 1 -- 172.21.15.32:0/627751922 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7f81bc03f030 con 0x7f81d00ced30 2024-08-03T20:49:49.025 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:49.024+0000 7f81d8400640 1 -- 172.21.15.32:0/627751922 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) -- 0x7f81d00cd100 con 0x7f81d00ced30 2024-08-03T20:49:49.544 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:49.543+0000 7f81cd600640 1 -- 172.21.15.32:0/627751922 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v34) ==== 118+0+0 (secure 0 0 0) 0x7f81bc031770 con 0x7f81d00ced30 2024-08-03T20:49:49.546 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:49.546+0000 7f81d8400640 1 -- 172.21.15.32:0/627751922 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f81b0077040 msgr2=0x7f81b0079550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:49:49.547 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:49.546+0000 7f81d8400640 1 --2- 172.21.15.32:0/627751922 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f81b0077040 0x7f81b0079550 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f81c4000f60 tx=0x7f81c4009040 comp rx=0 tx=0).stop 2024-08-03T20:49:49.547 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:49.546+0000 7f81d8400640 1 -- 172.21.15.32:0/627751922 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f81d00ced30 msgr2=0x7f81d0144560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:49:49.547 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:49.546+0000 7f81d8400640 1 --2- 172.21.15.32:0/627751922 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f81d00ced30 0x7f81d0144560 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f81bc009950 tx=0x7f81bc033ba0 comp rx=0 tx=0).stop 2024-08-03T20:49:49.547 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:49.546+0000 7f81d8400640 1 -- 172.21.15.32:0/627751922 shutdown_connections 2024-08-03T20:49:49.547 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:49.546+0000 7f81d8400640 1 --2- 172.21.15.32:0/627751922 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f81b0077040 0x7f81b0079550 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:49:49.547 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:49.546+0000 7f81d8400640 1 --2- 172.21.15.32:0/627751922 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f81d0145c50 0x7f81d0144aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:49:49.547 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:49.546+0000 7f81d8400640 1 --2- 172.21.15.32:0/627751922 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f81d00ced30 0x7f81d0144560 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:49:49.547 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:49.546+0000 7f81d8400640 1 --2- 172.21.15.32:0/627751922 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f81d00cccf0 0x7f81d0159a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:49:49.547 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:49.546+0000 7f81d8400640 1 -- 172.21.15.32:0/627751922 >> 172.21.15.32:0/627751922 conn(0x7f81d00b1bd0 msgr2=0x7f81d015b970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:49:49.547 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:49.546+0000 7f81d8400640 1 -- 172.21.15.32:0/627751922 shutdown_connections 2024-08-03T20:49:49.547 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:49.546+0000 7f81d8400640 1 -- 172.21.15.32:0/627751922 wait complete. 2024-08-03T20:49:49.569 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-08-03T20:49:49.569 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-08-03T20:49:49.802 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:49.801+0000 7f440b200640 1 -- 172.21.15.32:0/895296587 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f43f0003f90 msgr2=0x7f43f0004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:49:49.802 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:49.801+0000 7f440b200640 1 --2- 172.21.15.32:0/895296587 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f43f0003f90 0x7f43f0004800 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f43f400c2e0 tx=0x7f43f0005830 comp rx=0 tx=0).stop 2024-08-03T20:49:49.802 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:49.801+0000 7f440a800640 1 -- 172.21.15.32:0/895296587 reap_dead start 2024-08-03T20:49:49.803 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:49.801+0000 7f440b200640 1 -- 172.21.15.32:0/895296587 shutdown_connections 2024-08-03T20:49:49.803 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:49.801+0000 7f440b200640 1 -- 172.21.15.32:0/895296587 >> 172.21.15.32:0/895296587 conn(0x7f440c0cfd80 msgr2=0x7f440c0d0190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:49:49.803 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:49.801+0000 7f440b200640 1 -- 172.21.15.32:0/895296587 shutdown_connections 2024-08-03T20:49:49.803 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:49.802+0000 7f440b200640 1 -- 172.21.15.32:0/895296587 wait complete. 2024-08-03T20:49:49.804 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:49.804+0000 7f440b200640 1 Processor -- start 2024-08-03T20:49:49.804 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:49.804+0000 7f440b200640 1 -- start start 2024-08-03T20:49:49.806 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:49.805+0000 7f440b200640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f43f0003f90 0x7f440c0b2bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:49:49.806 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:49.805+0000 7f440b200640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f440c1f8420 0x7f440c1f8830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:49:49.806 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:49.805+0000 7f440b200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f440c0ce280 0x7f440c0ce690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:49:49.806 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:49.805+0000 7f440b200640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f440c0cc5a0 con 0x7f440c0ce280 2024-08-03T20:49:49.807 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:49.805+0000 7f440b200640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f440c0cc420 con 0x7f43f0003f90 2024-08-03T20:49:49.807 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:49.805+0000 7f440b200640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7f440c06ed40 con 0x7f440c1f8420 2024-08-03T20:49:49.807 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:49.806+0000 7f440a800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f440c0ce280 0x7f440c0ce690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:49:49.807 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:49.806+0000 7f440a800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f440c0ce280 0x7f440c0ce690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:38014/0 (socket says 172.21.15.32:38014) 2024-08-03T20:49:49.807 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:49.806+0000 7f440a800640 1 -- 172.21.15.32:0/2844649784 learned_addr learned my addr 172.21.15.32:0/2844649784 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:49:49.807 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:49.806+0000 7f4403e00640 1 --2- 172.21.15.32:0/2844649784 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f440c1f8420 0x7f440c1f8830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:49:49.807 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:49.806+0000 7f4409e00640 1 --2- 172.21.15.32:0/2844649784 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f43f0003f90 0x7f440c0b2bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:49:49.808 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:49.806+0000 7f440a800640 1 -- 172.21.15.32:0/2844649784 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f440c1f8420 msgr2=0x7f440c1f8830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:49:49.808 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:49.806+0000 7f440a800640 1 --2- 172.21.15.32:0/2844649784 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f440c1f8420 0x7f440c1f8830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:49:49.808 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:49.806+0000 7f440a800640 1 -- 172.21.15.32:0/2844649784 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f43f0003f90 msgr2=0x7f440c0b2bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:49:49.808 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:49.806+0000 7f440a800640 1 --2- 172.21.15.32:0/2844649784 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f43f0003f90 0x7f440c0b2bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:49:49.808 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:49.806+0000 7f440a800640 1 -- 172.21.15.32:0/2844649784 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f440c0b30f0 con 0x7f440c0ce280 2024-08-03T20:49:49.808 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:49.807+0000 7f440a800640 1 --2- 172.21.15.32:0/2844649784 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f440c0ce280 0x7f440c0ce690 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f4404001520 tx=0x7f4404006430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:49:49.808 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:49.807+0000 7f4402000640 1 -- 172.21.15.32:0/2844649784 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4404011900 con 0x7f440c0ce280 2024-08-03T20:49:49.808 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:49.807+0000 7f4402000640 1 -- 172.21.15.32:0/2844649784 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f4404018070 con 0x7f440c0ce280 2024-08-03T20:49:49.808 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:49.807+0000 7f4402000640 1 -- 172.21.15.32:0/2844649784 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4404011d40 con 0x7f440c0ce280 2024-08-03T20:49:49.809 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:49.807+0000 7f440b200640 1 -- 172.21.15.32:0/2844649784 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f440c0b5460 con 0x7f440c0ce280 2024-08-03T20:49:49.809 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:49.807+0000 7f440b200640 1 -- 172.21.15.32:0/2844649784 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f440c0cda80 con 0x7f440c0ce280 2024-08-03T20:49:49.809 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:49.808+0000 7f440b200640 1 -- 172.21.15.32:0/2844649784 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f440c151db0 con 0x7f440c0ce280 2024-08-03T20:49:49.809 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:49.809+0000 7f4402000640 1 -- 172.21.15.32:0/2844649784 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f440401d460 con 0x7f440c0ce280 2024-08-03T20:49:49.810 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:49.810+0000 7f4402000640 1 --2- 172.21.15.32:0/2844649784 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f43e8077040 0x7f43e8079550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:49:49.811 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:49.810+0000 7f4402000640 1 -- 172.21.15.32:0/2844649784 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(45..45 src has 1..45) ==== 5709+0+0 (secure 0 0 0) 0x7f4404099100 con 0x7f440c0ce280 2024-08-03T20:49:49.811 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:49.811+0000 7f4409e00640 1 --2- 172.21.15.32:0/2844649784 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f43e8077040 0x7f43e8079550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:49:49.812 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:49.811+0000 7f4409e00640 1 --2- 172.21.15.32:0/2844649784 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f43e8077040 0x7f43e8079550 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f43fc009c50 tx=0x7f43fc005f90 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:49:49.815 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:49.815+0000 7f4402000640 1 -- 172.21.15.32:0/2844649784 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7f440c151db0 con 0x7f440c0ce280 2024-08-03T20:49:50.067 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:50.066+0000 7f440b200640 1 -- 172.21.15.32:0/2844649784 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) -- 0x7f440c06f2a0 con 0x7f440c0ce280 2024-08-03T20:49:50.550 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:50.549+0000 7f4402000640 1 -- 172.21.15.32:0/2844649784 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v35) ==== 122+0+0 (secure 0 0 0) 0x7f44040622f0 con 0x7f440c0ce280 2024-08-03T20:49:50.552 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:50.552+0000 7f440b200640 1 -- 172.21.15.32:0/2844649784 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f43e8077040 msgr2=0x7f43e8079550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:49:50.553 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:50.552+0000 7f440b200640 1 --2- 172.21.15.32:0/2844649784 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f43e8077040 0x7f43e8079550 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f43fc009c50 tx=0x7f43fc005f90 comp rx=0 tx=0).stop 2024-08-03T20:49:50.553 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:50.552+0000 7f440b200640 1 -- 172.21.15.32:0/2844649784 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f440c0ce280 msgr2=0x7f440c0ce690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:49:50.553 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:50.552+0000 7f440b200640 1 --2- 172.21.15.32:0/2844649784 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f440c0ce280 0x7f440c0ce690 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f4404001520 tx=0x7f4404006430 comp rx=0 tx=0).stop 2024-08-03T20:49:50.553 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:50.552+0000 7f440b200640 1 -- 172.21.15.32:0/2844649784 shutdown_connections 2024-08-03T20:49:50.553 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:50.552+0000 7f440b200640 1 --2- 172.21.15.32:0/2844649784 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f43e8077040 0x7f43e8079550 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:49:50.553 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:50.552+0000 7f440b200640 1 --2- 172.21.15.32:0/2844649784 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f440c0ce280 0x7f440c0ce690 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:49:50.553 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:50.552+0000 7f440b200640 1 --2- 172.21.15.32:0/2844649784 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f440c1f8420 0x7f440c1f8830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:49:50.554 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:50.552+0000 7f440b200640 1 --2- 172.21.15.32:0/2844649784 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f43f0003f90 0x7f440c0b2bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:49:50.554 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:50.552+0000 7f440b200640 1 -- 172.21.15.32:0/2844649784 >> 172.21.15.32:0/2844649784 conn(0x7f440c0cfd80 msgr2=0x7f440c15e610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:49:50.554 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:50.552+0000 7f440b200640 1 -- 172.21.15.32:0/2844649784 shutdown_connections 2024-08-03T20:49:50.554 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:50.552+0000 7f440b200640 1 -- 172.21.15.32:0/2844649784 wait complete. 2024-08-03T20:49:50.579 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-08-03T20:49:50.579 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-08-03T20:49:50.807 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:50.806+0000 7fc17de00640 1 -- 172.21.15.32:0/2509703308 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc1780d1180 msgr2=0x7fc1780b7d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:49:50.807 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:50.806+0000 7fc17de00640 1 --2- 172.21.15.32:0/2509703308 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc1780d1180 0x7fc1780b7d00 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fc170005fd0 tx=0x7fc17002a940 comp rx=0 tx=0).stop 2024-08-03T20:49:50.807 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:50.806+0000 7fc17de00640 1 -- 172.21.15.32:0/2509703308 shutdown_connections 2024-08-03T20:49:50.807 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:50.806+0000 7fc17de00640 1 --2- 172.21.15.32:0/2509703308 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc1780d1180 0x7fc1780b7d00 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:49:50.807 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:50.806+0000 7fc17de00640 1 --2- 172.21.15.32:0/2509703308 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fc1780b59f0 0x7fc1780d4cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:49:50.807 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:50.806+0000 7fc17de00640 1 -- 172.21.15.32:0/2509703308 >> 172.21.15.32:0/2509703308 conn(0x7fc17809d660 msgr2=0x7fc1780d4330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:49:50.807 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:50.806+0000 7fc17de00640 1 -- 172.21.15.32:0/2509703308 shutdown_connections 2024-08-03T20:49:50.808 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:50.806+0000 7fc17de00640 1 -- 172.21.15.32:0/2509703308 wait complete. 2024-08-03T20:49:50.809 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:50.808+0000 7fc17de00640 1 Processor -- start 2024-08-03T20:49:50.809 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:50.809+0000 7fc17de00640 1 -- start start 2024-08-03T20:49:50.815 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:50.810+0000 7fc17de00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc1780b59f0 0x7fc1781e7050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:49:50.815 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:50.810+0000 7fc17de00640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fc1780cf9c0 0x7fc17814c730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:49:50.815 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:50.810+0000 7fc17de00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc178157d30 0x7fc17815e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:49:50.815 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:50.810+0000 7fc17de00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fc1780b88d0 con 0x7fc1780b59f0 2024-08-03T20:49:50.815 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:50.810+0000 7fc17de00640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7fc1780b8750 con 0x7fc178157d30 2024-08-03T20:49:50.815 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:50.810+0000 7fc17de00640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7fc1780b8a50 con 0x7fc1780cf9c0 2024-08-03T20:49:50.815 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:50.810+0000 7fc176a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc1780b59f0 0x7fc1781e7050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:49:50.816 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:50.810+0000 7fc176a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc1780b59f0 0x7fc1781e7050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:38030/0 (socket says 172.21.15.32:38030) 2024-08-03T20:49:50.816 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:50.810+0000 7fc176a00640 1 -- 172.21.15.32:0/2981705341 learned_addr learned my addr 172.21.15.32:0/2981705341 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:49:50.816 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:50.811+0000 7fc16e000640 1 --2- 172.21.15.32:0/2981705341 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fc1780cf9c0 0x7fc17814c730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:49:50.816 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:50.811+0000 7fc177400640 1 --2- 172.21.15.32:0/2981705341 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc178157d30 0x7fc17815e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:49:50.816 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:50.811+0000 7fc176a00640 1 -- 172.21.15.32:0/2981705341 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fc1780cf9c0 msgr2=0x7fc17814c730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:49:50.816 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:50.811+0000 7fc176a00640 1 --2- 172.21.15.32:0/2981705341 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fc1780cf9c0 0x7fc17814c730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:49:50.816 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:50.811+0000 7fc176a00640 1 -- 172.21.15.32:0/2981705341 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc178157d30 msgr2=0x7fc17815e9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:49:50.816 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:50.811+0000 7fc176a00640 1 --2- 172.21.15.32:0/2981705341 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc178157d30 0x7fc17815e9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:49:50.816 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:50.811+0000 7fc176a00640 1 -- 172.21.15.32:0/2981705341 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc17814cc70 con 0x7fc1780b59f0 2024-08-03T20:49:50.816 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:50.811+0000 7fc176a00640 1 --2- 172.21.15.32:0/2981705341 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc1780b59f0 0x7fc1781e7050 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7fc16800e790 tx=0x7fc16800ec50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:49:50.816 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:50.811+0000 7fc16fe00640 1 -- 172.21.15.32:0/2981705341 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc16800eea0 con 0x7fc1780b59f0 2024-08-03T20:49:50.816 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:50.812+0000 7fc16fe00640 1 -- 172.21.15.32:0/2981705341 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fc16800ccc0 con 0x7fc1780b59f0 2024-08-03T20:49:50.816 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:50.812+0000 7fc16fe00640 1 -- 172.21.15.32:0/2981705341 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc168005280 con 0x7fc1780b59f0 2024-08-03T20:49:50.817 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:50.812+0000 7fc17de00640 1 -- 172.21.15.32:0/2981705341 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc17815c4b0 con 0x7fc1780b59f0 2024-08-03T20:49:50.817 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:50.812+0000 7fc17de00640 1 -- 172.21.15.32:0/2981705341 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc1781e7ca0 con 0x7fc1780b59f0 2024-08-03T20:49:50.817 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:50.813+0000 7fc17de00640 1 -- 172.21.15.32:0/2981705341 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc1780b8f70 con 0x7fc1780b59f0 2024-08-03T20:49:50.817 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:50.814+0000 7fc16fe00640 1 -- 172.21.15.32:0/2981705341 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7fc168005420 con 0x7fc1780b59f0 2024-08-03T20:49:50.817 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:50.816+0000 7fc16fe00640 1 --2- 172.21.15.32:0/2981705341 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fc154077010 0x7fc154079520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:49:50.818 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:50.816+0000 7fc16fe00640 1 -- 172.21.15.32:0/2981705341 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(45..45 src has 1..45) ==== 5709+0+0 (secure 0 0 0) 0x7fc1680990b0 con 0x7fc1780b59f0 2024-08-03T20:49:50.818 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:50.816+0000 7fc16e000640 1 --2- 172.21.15.32:0/2981705341 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fc154077010 0x7fc154079520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:49:50.818 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:50.817+0000 7fc16e000640 1 --2- 172.21.15.32:0/2981705341 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fc154077010 0x7fc154079520 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fc178157c30 tx=0x7fc160009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:49:50.820 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:50.820+0000 7fc16fe00640 1 -- 172.21.15.32:0/2981705341 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7fc168061c50 con 0x7fc1780b59f0 2024-08-03T20:49:51.066 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:51.066+0000 7fc17de00640 1 -- 172.21.15.32:0/2981705341 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc1780ce310 con 0x7fc1780b59f0 2024-08-03T20:49:51.067 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:51.066+0000 7fc16fe00640 1 -- 172.21.15.32:0/2981705341 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 35 v35) ==== 94+0+5226 (secure 0 0 0) 0x7fc1680661a0 con 0x7fc1780b59f0 2024-08-03T20:49:51.067 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-03T20:49:51.068 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":35,"btime":"2024-08-03T20:49:50:544317+0000","default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4997,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6837/2688500839","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6836","nonce":2688500839},{"type":"v1","addr":"172.21.15.118:6837","nonce":2688500839}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":30},{"gid":5266,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6835/2381075275","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6834","nonce":2381075275},{"type":"v1","addr":"172.21.15.118:6835","nonce":2381075275}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":28},{"gid":5421,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6833/1058332640","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":1058332640},{"type":"v1","addr":"172.21.15.32:6833","nonce":1058332640}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":29}],"filesystems":[{"mdsmap":{"epoch":35,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-08-03T20:49:47.064654+0000","modified":"2024-08-03T20:49:50.067811+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5424},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5424":{"gid":5424,"name":"c","rank":0,"incarnation":31,"state":"up:active","state_seq":4,"addr":"172.21.15.32:6835/678469885","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":678469885},{"type":"v1","addr":"172.21.15.32:6835","nonce":678469885}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5424,"qdb_cluster":[5424]},"id":3}]} 2024-08-03T20:49:51.068 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 35 2024-08-03T20:49:51.070 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:51.069+0000 7fc17de00640 1 -- 172.21.15.32:0/2981705341 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fc154077010 msgr2=0x7fc154079520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:49:51.070 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:51.069+0000 7fc17de00640 1 --2- 172.21.15.32:0/2981705341 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fc154077010 0x7fc154079520 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fc178157c30 tx=0x7fc160009040 comp rx=0 tx=0).stop 2024-08-03T20:49:51.071 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:51.069+0000 7fc17de00640 1 -- 172.21.15.32:0/2981705341 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc1780b59f0 msgr2=0x7fc1781e7050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:49:51.071 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:51.069+0000 7fc17de00640 1 --2- 172.21.15.32:0/2981705341 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc1780b59f0 0x7fc1781e7050 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7fc16800e790 tx=0x7fc16800ec50 comp rx=0 tx=0).stop 2024-08-03T20:49:51.071 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:51.069+0000 7fc17de00640 1 -- 172.21.15.32:0/2981705341 shutdown_connections 2024-08-03T20:49:51.071 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:51.069+0000 7fc17de00640 1 --2- 172.21.15.32:0/2981705341 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fc154077010 0x7fc154079520 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:49:51.071 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:51.069+0000 7fc17de00640 1 --2- 172.21.15.32:0/2981705341 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc178157d30 0x7fc17815e9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:49:51.071 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:51.069+0000 7fc17de00640 1 --2- 172.21.15.32:0/2981705341 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fc1780cf9c0 0x7fc17814c730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:49:51.071 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:51.069+0000 7fc17de00640 1 --2- 172.21.15.32:0/2981705341 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc1780b59f0 0x7fc1781e7050 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:49:51.071 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:51.069+0000 7fc17de00640 1 -- 172.21.15.32:0/2981705341 >> 172.21.15.32:0/2981705341 conn(0x7fc17809d660 msgr2=0x7fc17806d680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:49:51.071 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:51.069+0000 7fc17de00640 1 -- 172.21.15.32:0/2981705341 shutdown_connections 2024-08-03T20:49:51.071 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:51.069+0000 7fc17de00640 1 -- 172.21.15.32:0/2981705341 wait complete. 2024-08-03T20:49:51.093 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-08-03T20:49:51.323 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:51.322+0000 7fc628a00640 1 -- 172.21.15.32:0/318354209 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc6240cffe0 msgr2=0x7fc6240b3e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:49:51.323 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:51.322+0000 7fc628a00640 1 --2- 172.21.15.32:0/318354209 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc6240cffe0 0x7fc6240b3e90 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fc60c009830 tx=0x7fc60c02a4a0 comp rx=0 tx=0).stop 2024-08-03T20:49:51.323 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:51.322+0000 7fc628a00640 1 -- 172.21.15.32:0/318354209 shutdown_connections 2024-08-03T20:49:51.323 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:51.322+0000 7fc628a00640 1 --2- 172.21.15.32:0/318354209 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fc6240b43d0 0x7fc6240cf510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:49:51.323 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:51.322+0000 7fc628a00640 1 --2- 172.21.15.32:0/318354209 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc6240cffe0 0x7fc6240b3e90 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:49:51.324 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:51.322+0000 7fc628a00640 1 -- 172.21.15.32:0/318354209 >> 172.21.15.32:0/318354209 conn(0x7fc62408d7a0 msgr2=0x7fc62409d3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:49:51.324 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:51.322+0000 7fc628a00640 1 -- 172.21.15.32:0/318354209 shutdown_connections 2024-08-03T20:49:51.324 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:51.323+0000 7fc628a00640 1 -- 172.21.15.32:0/318354209 wait complete. 2024-08-03T20:49:51.325 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:51.325+0000 7fc628a00640 1 Processor -- start 2024-08-03T20:49:51.325 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:51.325+0000 7fc628a00640 1 -- start start 2024-08-03T20:49:51.327 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:51.326+0000 7fc628a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc6240b43d0 0x7fc62415e370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:49:51.327 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:51.326+0000 7fc628a00640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fc6240cffe0 0x7fc6240c9690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:49:51.327 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:51.326+0000 7fc628a00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc6240d26e0 0x7fc62406e7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:49:51.327 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:51.326+0000 7fc628a00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fc6240d4100 con 0x7fc6240b43d0 2024-08-03T20:49:51.327 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:51.326+0000 7fc628a00640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7fc6240d3f80 con 0x7fc6240d26e0 2024-08-03T20:49:51.327 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:51.326+0000 7fc628a00640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7fc6240d4280 con 0x7fc6240cffe0 2024-08-03T20:49:51.327 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:51.326+0000 7fc623400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc6240b43d0 0x7fc62415e370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:49:51.327 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:51.327+0000 7fc61aa00640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fc6240cffe0 0x7fc6240c9690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:49:51.327 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:51.327+0000 7fc623400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc6240b43d0 0x7fc62415e370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:38038/0 (socket says 172.21.15.32:38038) 2024-08-03T20:49:51.327 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:51.327+0000 7fc623400640 1 -- 172.21.15.32:0/1901732784 learned_addr learned my addr 172.21.15.32:0/1901732784 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:49:51.327 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:51.327+0000 7fc623e00640 1 --2- 172.21.15.32:0/1901732784 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc6240d26e0 0x7fc62406e7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:49:51.329 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:51.327+0000 7fc623400640 1 -- 172.21.15.32:0/1901732784 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fc6240cffe0 msgr2=0x7fc6240c9690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:49:51.329 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:51.327+0000 7fc623400640 1 --2- 172.21.15.32:0/1901732784 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fc6240cffe0 0x7fc6240c9690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:49:51.329 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:51.327+0000 7fc623400640 1 -- 172.21.15.32:0/1901732784 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc6240d26e0 msgr2=0x7fc62406e7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:49:51.329 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:51.327+0000 7fc623400640 1 --2- 172.21.15.32:0/1901732784 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc6240d26e0 0x7fc62406e7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:49:51.329 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:51.327+0000 7fc623400640 1 -- 172.21.15.32:0/1901732784 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc6240cc480 con 0x7fc6240b43d0 2024-08-03T20:49:51.329 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:51.327+0000 7fc623e00640 1 --2- 172.21.15.32:0/1901732784 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc6240d26e0 0x7fc62406e7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-03T20:49:51.329 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:51.327+0000 7fc623400640 1 --2- 172.21.15.32:0/1901732784 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc6240b43d0 0x7fc62415e370 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7fc61400c530 tx=0x7fc61400c9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:49:51.329 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:51.328+0000 7fc61be00640 1 -- 172.21.15.32:0/1901732784 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc614016020 con 0x7fc6240b43d0 2024-08-03T20:49:51.329 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:51.328+0000 7fc61be00640 1 -- 172.21.15.32:0/1901732784 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fc614012940 con 0x7fc6240b43d0 2024-08-03T20:49:51.329 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:51.328+0000 7fc61be00640 1 -- 172.21.15.32:0/1901732784 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc614012d90 con 0x7fc6240b43d0 2024-08-03T20:49:51.329 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:51.328+0000 7fc628a00640 1 -- 172.21.15.32:0/1901732784 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc6240cb3e0 con 0x7fc6240b43d0 2024-08-03T20:49:51.330 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:51.328+0000 7fc628a00640 1 -- 172.21.15.32:0/1901732784 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc6240cb9a0 con 0x7fc6240b43d0 2024-08-03T20:49:51.330 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:51.329+0000 7fc628a00640 1 -- 172.21.15.32:0/1901732784 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc6240d46e0 con 0x7fc6240b43d0 2024-08-03T20:49:51.332 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:51.330+0000 7fc61be00640 1 -- 172.21.15.32:0/1901732784 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7fc614019020 con 0x7fc6240b43d0 2024-08-03T20:49:51.332 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:51.332+0000 7fc61be00640 1 --2- 172.21.15.32:0/1901732784 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fc600077040 0x7fc600079550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:49:51.332 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:51.332+0000 7fc61be00640 1 -- 172.21.15.32:0/1901732784 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(45..45 src has 1..45) ==== 5709+0+0 (secure 0 0 0) 0x7fc614098300 con 0x7fc6240b43d0 2024-08-03T20:49:51.333 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:51.332+0000 7fc61aa00640 1 --2- 172.21.15.32:0/1901732784 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fc600077040 0x7fc600079550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:49:51.333 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:51.333+0000 7fc61aa00640 1 --2- 172.21.15.32:0/1901732784 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fc600077040 0x7fc600079550 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fc60c034000 tx=0x7fc60c004730 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:49:51.337 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:51.337+0000 7fc61be00640 1 -- 172.21.15.32:0/1901732784 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7fc614020070 con 0x7fc6240b43d0 2024-08-03T20:49:51.542 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:51.541+0000 7fc628a00640 1 -- 172.21.15.32:0/1901732784 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fc6240c9bd0 con 0x7fc6240b43d0 2024-08-03T20:49:51.543 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:51.542+0000 7fc61be00640 1 -- 172.21.15.32:0/1901732784 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v45) ==== 74+0+16553 (secure 0 0 0) 0x7fc614060ea0 con 0x7fc6240b43d0 2024-08-03T20:49:51.543 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-03T20:49:51.544 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":45,"fsid":"40ab141a-01d7-49b9-ae3c-2edec9876e95","created":"2024-08-03T20:47:14.957362+0000","modified":"2024-08-03T20:49:48.531661+0000","last_up_change":"2024-08-03T20:47:28.874451+0000","last_in_change":"2024-08-03T20:47:20.079442+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-08-03T20:47:28.261300+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_metadata","create_time":"2024-08-03T20:49:45.020976+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"44","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_data","create_time":"2024-08-03T20:49:46.012432+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"45","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5a83e759-9035-438b-889a-0a6af1f0a0d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6816","nonce":3066156116},{"type":"v1","addr":"172.21.15.32:6817","nonce":3066156116}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6818","nonce":3066156116},{"type":"v1","addr":"172.21.15.32:6819","nonce":3066156116}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6822","nonce":3066156116},{"type":"v1","addr":"172.21.15.32:6823","nonce":3066156116}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6820","nonce":3066156116},{"type":"v1","addr":"172.21.15.32:6821","nonce":3066156116}]},"public_addr":"172.21.15.32:6817/3066156116","cluster_addr":"172.21.15.32:6819/3066156116","heartbeat_back_addr":"172.21.15.32:6823/3066156116","heartbeat_front_addr":"172.21.15.32:6821/3066156116","state":["exists","up"]},{"osd":1,"uuid":"d7c5c210-da4f-49f9-b9e7-1dfb02daa4bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6824","nonce":1551241759},{"type":"v1","addr":"172.21.15.32:6825","nonce":1551241759}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6826","nonce":1551241759},{"type":"v1","addr":"172.21.15.32:6827","nonce":1551241759}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6830","nonce":1551241759},{"type":"v1","addr":"172.21.15.32:6831","nonce":1551241759}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6828","nonce":1551241759},{"type":"v1","addr":"172.21.15.32:6829","nonce":1551241759}]},"public_addr":"172.21.15.32:6825/1551241759","cluster_addr":"172.21.15.32:6827/1551241759","heartbeat_back_addr":"172.21.15.32:6831/1551241759","heartbeat_front_addr":"172.21.15.32:6829/1551241759","state":["exists","up"]},{"osd":2,"uuid":"57f2e6de-a201-45c2-8981-8e56a4875ebd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6808","nonce":3806708944},{"type":"v1","addr":"172.21.15.32:6809","nonce":3806708944}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6810","nonce":3806708944},{"type":"v1","addr":"172.21.15.32:6811","nonce":3806708944}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6814","nonce":3806708944},{"type":"v1","addr":"172.21.15.32:6815","nonce":3806708944}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6812","nonce":3806708944},{"type":"v1","addr":"172.21.15.32:6813","nonce":3806708944}]},"public_addr":"172.21.15.32:6809/3806708944","cluster_addr":"172.21.15.32:6811/3806708944","heartbeat_back_addr":"172.21.15.32:6815/3806708944","heartbeat_front_addr":"172.21.15.32:6813/3806708944","state":["exists","up"]},{"osd":3,"uuid":"6439a280-da45-4148-8130-f16323935ff3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6800","nonce":9088303},{"type":"v1","addr":"172.21.15.32:6801","nonce":9088303}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6802","nonce":9088303},{"type":"v1","addr":"172.21.15.32:6803","nonce":9088303}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6806","nonce":9088303},{"type":"v1","addr":"172.21.15.32:6807","nonce":9088303}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6804","nonce":9088303},{"type":"v1","addr":"172.21.15.32:6805","nonce":9088303}]},"public_addr":"172.21.15.32:6801/9088303","cluster_addr":"172.21.15.32:6803/9088303","heartbeat_back_addr":"172.21.15.32:6807/9088303","heartbeat_front_addr":"172.21.15.32:6805/9088303","state":["exists","up"]},{"osd":4,"uuid":"4f902eff-1e18-418d-82c9-2151c69524fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6802","nonce":864752275},{"type":"v1","addr":"172.21.15.118:6803","nonce":864752275}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6804","nonce":864752275},{"type":"v1","addr":"172.21.15.118:6805","nonce":864752275}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6808","nonce":864752275},{"type":"v1","addr":"172.21.15.118:6809","nonce":864752275}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6806","nonce":864752275},{"type":"v1","addr":"172.21.15.118:6807","nonce":864752275}]},"public_addr":"172.21.15.118:6803/864752275","cluster_addr":"172.21.15.118:6805/864752275","heartbeat_back_addr":"172.21.15.118:6809/864752275","heartbeat_front_addr":"172.21.15.118:6807/864752275","state":["exists","up"]},{"osd":5,"uuid":"bfd9feac-dd06-458c-91b9-af7f175f612c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6810","nonce":2724891770},{"type":"v1","addr":"172.21.15.118:6811","nonce":2724891770}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6812","nonce":2724891770},{"type":"v1","addr":"172.21.15.118:6813","nonce":2724891770}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6816","nonce":2724891770},{"type":"v1","addr":"172.21.15.118:6817","nonce":2724891770}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6814","nonce":2724891770},{"type":"v1","addr":"172.21.15.118:6815","nonce":2724891770}]},"public_addr":"172.21.15.118:6811/2724891770","cluster_addr":"172.21.15.118:6813/2724891770","heartbeat_back_addr":"172.21.15.118:6817/2724891770","heartbeat_front_addr":"172.21.15.118:6815/2724891770","state":["exists","up"]},{"osd":6,"uuid":"5ed5a279-9d8e-416d-bbc7-7edd3e6d5705","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6825","nonce":801947386},{"type":"v1","addr":"172.21.15.118:6827","nonce":801947386}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6828","nonce":801947386},{"type":"v1","addr":"172.21.15.118:6829","nonce":801947386}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6832","nonce":801947386},{"type":"v1","addr":"172.21.15.118:6833","nonce":801947386}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6830","nonce":801947386},{"type":"v1","addr":"172.21.15.118:6831","nonce":801947386}]},"public_addr":"172.21.15.118:6827/801947386","cluster_addr":"172.21.15.118:6829/801947386","heartbeat_back_addr":"172.21.15.118:6833/801947386","heartbeat_front_addr":"172.21.15.118:6831/801947386","state":["exists","up"]},{"osd":7,"uuid":"f2086c47-875e-4451-a436-2008d1ea16af","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6818","nonce":653000048},{"type":"v1","addr":"172.21.15.118:6819","nonce":653000048}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6820","nonce":653000048},{"type":"v1","addr":"172.21.15.118:6821","nonce":653000048}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6824","nonce":653000048},{"type":"v1","addr":"172.21.15.118:6826","nonce":653000048}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6822","nonce":653000048},{"type":"v1","addr":"172.21.15.118:6823","nonce":653000048}]},"public_addr":"172.21.15.118:6819/653000048","cluster_addr":"172.21.15.118:6821/653000048","heartbeat_back_addr":"172.21.15.118:6826/653000048","heartbeat_front_addr":"172.21.15.118:6823/653000048","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-03T20:47:25.491994+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-03T20:47:25.587358+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-03T20:47:25.454175+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-03T20:47:25.377734+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-03T20:47:25.412351+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-03T20:47:25.499815+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-03T20:47:25.575044+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-03T20:47:25.552383+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-08-03T20:49:51.545 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:51.545+0000 7fc628a00640 1 -- 172.21.15.32:0/1901732784 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fc600077040 msgr2=0x7fc600079550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:49:51.546 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:51.545+0000 7fc628a00640 1 --2- 172.21.15.32:0/1901732784 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fc600077040 0x7fc600079550 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fc60c034000 tx=0x7fc60c004730 comp rx=0 tx=0).stop 2024-08-03T20:49:51.546 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:51.545+0000 7fc628a00640 1 -- 172.21.15.32:0/1901732784 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc6240b43d0 msgr2=0x7fc62415e370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:49:51.546 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:51.545+0000 7fc628a00640 1 --2- 172.21.15.32:0/1901732784 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc6240b43d0 0x7fc62415e370 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7fc61400c530 tx=0x7fc61400c9f0 comp rx=0 tx=0).stop 2024-08-03T20:49:51.546 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:51.545+0000 7fc628a00640 1 -- 172.21.15.32:0/1901732784 shutdown_connections 2024-08-03T20:49:51.546 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:51.545+0000 7fc628a00640 1 --2- 172.21.15.32:0/1901732784 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fc600077040 0x7fc600079550 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:49:51.546 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:51.545+0000 7fc628a00640 1 --2- 172.21.15.32:0/1901732784 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc6240d26e0 0x7fc62406e7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:49:51.546 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:51.545+0000 7fc628a00640 1 --2- 172.21.15.32:0/1901732784 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fc6240cffe0 0x7fc6240c9690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:49:51.546 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:51.545+0000 7fc628a00640 1 --2- 172.21.15.32:0/1901732784 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc6240b43d0 0x7fc62415e370 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:49:51.546 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:51.545+0000 7fc628a00640 1 -- 172.21.15.32:0/1901732784 >> 172.21.15.32:0/1901732784 conn(0x7fc62408d7a0 msgr2=0x7fc62406d8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:49:51.546 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:51.545+0000 7fc628a00640 1 -- 172.21.15.32:0/1901732784 shutdown_connections 2024-08-03T20:49:51.546 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:51.545+0000 7fc628a00640 1 -- 172.21.15.32:0/1901732784 wait complete. 2024-08-03T20:49:51.569 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-08-03T20:49:51.569 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-08-03T20:49:51.797 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:51.796+0000 7f37d6200640 1 -- 172.21.15.32:0/3935548218 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f37d015f610 msgr2=0x7f37d01523f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:49:51.797 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:51.796+0000 7f37d6200640 1 --2- 172.21.15.32:0/3935548218 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f37d015f610 0x7f37d01523f0 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f37d00cd3f0 tx=0x7f37b802a4a0 comp rx=0 tx=0).stop 2024-08-03T20:49:51.798 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:51.796+0000 7f37d6200640 1 -- 172.21.15.32:0/3935548218 shutdown_connections 2024-08-03T20:49:51.798 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:51.796+0000 7f37d6200640 1 --2- 172.21.15.32:0/3935548218 >> v2:172.21.15.118:3300/0 conn(0x7f37d00b4ce0 0x7f37d00cdf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:49:51.798 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:51.796+0000 7f37d6200640 1 --2- 172.21.15.32:0/3935548218 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f37d015f610 0x7f37d01523f0 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:49:51.798 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:51.796+0000 7f37d6200640 1 -- 172.21.15.32:0/3935548218 >> 172.21.15.32:0/3935548218 conn(0x7f37d00b1bd0 msgr2=0x7f37d00b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:49:51.798 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:51.797+0000 7f37d6200640 1 -- 172.21.15.32:0/3935548218 shutdown_connections 2024-08-03T20:49:51.798 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:51.797+0000 7f37d6200640 1 -- 172.21.15.32:0/3935548218 wait complete. 2024-08-03T20:49:51.799 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:51.799+0000 7f37d6200640 1 Processor -- start 2024-08-03T20:49:51.799 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:51.799+0000 7f37d6200640 1 -- start start 2024-08-03T20:49:51.802 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:51.800+0000 7f37d6200640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f37d00b38b0 0x7f37d00aec40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:49:51.802 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:51.800+0000 7f37d6200640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f37d00b4ce0 0x7f37d014ed00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:49:51.802 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:51.800+0000 7f37d6200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f37d01569c0 0x7f37d0156dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:49:51.802 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:51.800+0000 7f37d6200640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f37d00cea30 con 0x7f37d01569c0 2024-08-03T20:49:51.802 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:51.801+0000 7f37d6200640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f37d00ce8b0 con 0x7f37d00b38b0 2024-08-03T20:49:51.802 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:51.801+0000 7f37d6200640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7f37d00cebb0 con 0x7f37d00b4ce0 2024-08-03T20:49:51.802 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:51.801+0000 7f37cfe00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f37d01569c0 0x7f37d0156dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:49:51.802 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:51.801+0000 7f37cfe00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f37d01569c0 0x7f37d0156dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:38046/0 (socket says 172.21.15.32:38046) 2024-08-03T20:49:51.802 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:51.801+0000 7f37cfe00640 1 -- 172.21.15.32:0/746259566 learned_addr learned my addr 172.21.15.32:0/746259566 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:49:51.803 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:51.801+0000 7f37cf400640 1 --2- 172.21.15.32:0/746259566 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f37d00b38b0 0x7f37d00aec40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:49:51.803 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:51.801+0000 7f37c6a00640 1 --2- 172.21.15.32:0/746259566 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f37d00b4ce0 0x7f37d014ed00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:49:51.803 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:51.801+0000 7f37cfe00640 1 -- 172.21.15.32:0/746259566 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f37d00b4ce0 msgr2=0x7f37d014ed00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:49:51.803 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:51.801+0000 7f37cfe00640 1 --2- 172.21.15.32:0/746259566 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f37d00b4ce0 0x7f37d014ed00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:49:51.803 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:51.801+0000 7f37cfe00640 1 -- 172.21.15.32:0/746259566 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f37d00b38b0 msgr2=0x7f37d00aec40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:49:51.803 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:51.801+0000 7f37cfe00640 1 --2- 172.21.15.32:0/746259566 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f37d00b38b0 0x7f37d00aec40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:49:51.803 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:51.801+0000 7f37cfe00640 1 -- 172.21.15.32:0/746259566 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f37d00adba0 con 0x7f37d01569c0 2024-08-03T20:49:51.803 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:51.802+0000 7f37cfe00640 1 --2- 172.21.15.32:0/746259566 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f37d01569c0 0x7f37d0156dd0 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7f37c800d9e0 tx=0x7f37c800dea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:49:51.804 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:51.802+0000 7f37c7e00640 1 -- 172.21.15.32:0/746259566 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37c8017070 con 0x7f37d01569c0 2024-08-03T20:49:51.804 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:51.802+0000 7f37c7e00640 1 -- 172.21.15.32:0/746259566 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f37c800be30 con 0x7f37d01569c0 2024-08-03T20:49:51.804 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:51.802+0000 7f37c7e00640 1 -- 172.21.15.32:0/746259566 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37c8003bb0 con 0x7f37d01569c0 2024-08-03T20:49:51.804 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:51.802+0000 7f37d6200640 1 -- 172.21.15.32:0/746259566 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f37d00ade80 con 0x7f37d01569c0 2024-08-03T20:49:51.804 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:51.802+0000 7f37d6200640 1 -- 172.21.15.32:0/746259566 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f37d00ae440 con 0x7f37d01569c0 2024-08-03T20:49:51.804 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:51.803+0000 7f37d6200640 1 -- 172.21.15.32:0/746259566 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f37d00cf0d0 con 0x7f37d01569c0 2024-08-03T20:49:51.805 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:51.804+0000 7f37c7e00640 1 -- 172.21.15.32:0/746259566 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f37c8010460 con 0x7f37d01569c0 2024-08-03T20:49:51.806 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:51.806+0000 7f37c7e00640 1 --2- 172.21.15.32:0/746259566 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f37ac077010 0x7f37ac079520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:49:51.806 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:51.806+0000 7f37c7e00640 1 -- 172.21.15.32:0/746259566 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(45..45 src has 1..45) ==== 5709+0+0 (secure 0 0 0) 0x7f37c8098d70 con 0x7f37d01569c0 2024-08-03T20:49:51.807 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:51.806+0000 7f37cf400640 1 --2- 172.21.15.32:0/746259566 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f37ac077010 0x7f37ac079520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:49:51.807 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:51.807+0000 7f37cf400640 1 --2- 172.21.15.32:0/746259566 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f37ac077010 0x7f37ac079520 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f37c0001610 tx=0x7f37c0009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:49:51.811 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:51.811+0000 7f37c7e00640 1 -- 172.21.15.32:0/746259566 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7f37c8061ed0 con 0x7f37d01569c0 2024-08-03T20:49:52.011 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:52.011+0000 7f37d6200640 1 -- 172.21.15.32:0/746259566 --> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f37d00ad930 con 0x7f37ac077010 2024-08-03T20:49:52.017 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:52.016+0000 7f37c7e00640 1 -- 172.21.15.32:0/746259566 <== mgr.4106 v2:172.21.15.118:6800/4205950699 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+251943 (secure 0 0 0) 0x7f37d00ad930 con 0x7f37ac077010 2024-08-03T20:49:52.020 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-03T20:49:52.021 INFO:teuthology.orchestra.run.smithi032.stderr:dumped all 2024-08-03T20:49:52.023 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:52.020+0000 7f37d6200640 1 -- 172.21.15.32:0/746259566 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f37ac077010 msgr2=0x7f37ac079520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:49:52.023 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:52.020+0000 7f37d6200640 1 --2- 172.21.15.32:0/746259566 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f37ac077010 0x7f37ac079520 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f37c0001610 tx=0x7f37c0009040 comp rx=0 tx=0).stop 2024-08-03T20:49:52.023 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:52.020+0000 7f37d6200640 1 -- 172.21.15.32:0/746259566 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f37d01569c0 msgr2=0x7f37d0156dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:49:52.023 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:52.020+0000 7f37d6200640 1 --2- 172.21.15.32:0/746259566 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f37d01569c0 0x7f37d0156dd0 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7f37c800d9e0 tx=0x7f37c800dea0 comp rx=0 tx=0).stop 2024-08-03T20:49:52.023 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:52.020+0000 7f37d6200640 1 -- 172.21.15.32:0/746259566 shutdown_connections 2024-08-03T20:49:52.023 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:52.020+0000 7f37d6200640 1 --2- 172.21.15.32:0/746259566 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f37ac077010 0x7f37ac079520 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:49:52.023 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:52.020+0000 7f37d6200640 1 --2- 172.21.15.32:0/746259566 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f37d01569c0 0x7f37d0156dd0 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:49:52.023 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:52.020+0000 7f37d6200640 1 --2- 172.21.15.32:0/746259566 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f37d00b4ce0 0x7f37d014ed00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:49:52.023 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:52.020+0000 7f37d6200640 1 --2- 172.21.15.32:0/746259566 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f37d00b38b0 0x7f37d00aec40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:49:52.024 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:52.020+0000 7f37d6200640 1 -- 172.21.15.32:0/746259566 >> 172.21.15.32:0/746259566 conn(0x7f37d00b1bd0 msgr2=0x7f37d0150cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:49:52.024 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:52.020+0000 7f37d6200640 1 -- 172.21.15.32:0/746259566 shutdown_connections 2024-08-03T20:49:52.024 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:52.020+0000 7f37d6200640 1 -- 172.21.15.32:0/746259566 wait complete. 2024-08-03T20:49:52.046 INFO:teuthology.orchestra.run.smithi032.stdout:{"pg_ready":true,"pg_map":{"version":109,"stamp":"2024-08-03T20:49:50.257626+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":145,"num_write_kb":1642,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":110,"ondisk_log_size":110,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":99,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":254736,"kb_used_data":5240,"kb_used_omap":12,"kb_used_meta":249331,"kb_avail":754719984,"statfs":{"total":773094113280,"available":772833263616,"internally_reserved":0,"allocated":5365760,"data_stored":2848300,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":255315505},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":30,"apply_latency_ms":30,"commit_latency_ns":30000000,"apply_latency_ns":30000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.746612"},"pg_stats":[{"pgid":"7.3f","version":"0'0","reported_seq":13,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-08-03T20:49:48.541258+0000","last_change":"2024-08-03T20:49:47.548778+0000","last_active":"2024-08-03T20:49:48.541258+0000","last_peered":"2024-08-03T20:49:48.541258+0000","last_clean":"2024-08-03T20:49:48.541258+0000","last_became_active":"2024-08-03T20:49:47.547674+0000","last_became_peered":"2024-08-03T20:49:47.547674+0000","last_unstale":"2024-08-03T20:49:48.541258+0000","last_undegraded":"2024-08-03T20:49:48.541258+0000","last_fullsized":"2024-08-03T20:49:48.541258+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:49:46.511193+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:49:46.511193+0000","last_clean_scrub_stamp":"2024-08-03T20:49:46.511193+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T03:39:25.547249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.3e","version":"0'0","reported_seq":13,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-08-03T20:49:48.676693+0000","last_change":"2024-08-03T20:49:47.568416+0000","last_active":"2024-08-03T20:49:48.676693+0000","last_peered":"2024-08-03T20:49:48.676693+0000","last_clean":"2024-08-03T20:49:48.676693+0000","last_became_active":"2024-08-03T20:49:47.568102+0000","last_became_peered":"2024-08-03T20:49:47.568102+0000","last_unstale":"2024-08-03T20:49:48.676693+0000","last_undegraded":"2024-08-03T20:49:48.676693+0000","last_fullsized":"2024-08-03T20:49:48.676693+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:49:46.511193+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:49:46.511193+0000","last_clean_scrub_stamp":"2024-08-03T20:49:46.511193+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-04T23:53:08.188600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.3d","version":"0'0","reported_seq":10,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-08-03T20:49:47.554505+0000","last_change":"2024-08-03T20:49:47.552482+0000","last_active":"2024-08-03T20:49:47.554505+0000","last_peered":"2024-08-03T20:49:47.554505+0000","last_clean":"2024-08-03T20:49:47.554505+0000","last_became_active":"2024-08-03T20:49:47.552169+0000","last_became_peered":"2024-08-03T20:49:47.552169+0000","last_unstale":"2024-08-03T20:49:47.554505+0000","last_undegraded":"2024-08-03T20:49:47.554505+0000","last_fullsized":"2024-08-03T20:49:47.554505+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:49:46.511193+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:49:46.511193+0000","last_clean_scrub_stamp":"2024-08-03T20:49:46.511193+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T00:48:27.200673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.3c","version":"0'0","reported_seq":10,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-08-03T20:49:47.570176+0000","last_change":"2024-08-03T20:49:47.543020+0000","last_active":"2024-08-03T20:49:47.570176+0000","last_peered":"2024-08-03T20:49:47.570176+0000","last_clean":"2024-08-03T20:49:47.570176+0000","last_became_active":"2024-08-03T20:49:47.542702+0000","last_became_peered":"2024-08-03T20:49:47.542702+0000","last_unstale":"2024-08-03T20:49:47.570176+0000","last_undegraded":"2024-08-03T20:49:47.570176+0000","last_fullsized":"2024-08-03T20:49:47.570176+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:49:46.511193+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:49:46.511193+0000","last_clean_scrub_stamp":"2024-08-03T20:49:46.511193+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-04T22:39:10.883457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.3b","version":"0'0","reported_seq":13,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-08-03T20:49:48.676439+0000","last_change":"2024-08-03T20:49:47.548807+0000","last_active":"2024-08-03T20:49:48.676439+0000","last_peered":"2024-08-03T20:49:48.676439+0000","last_clean":"2024-08-03T20:49:48.676439+0000","last_became_active":"2024-08-03T20:49:47.548490+0000","last_became_peered":"2024-08-03T20:49:47.548490+0000","last_unstale":"2024-08-03T20:49:48.676439+0000","last_undegraded":"2024-08-03T20:49:48.676439+0000","last_fullsized":"2024-08-03T20:49:48.676439+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:49:46.511193+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:49:46.511193+0000","last_clean_scrub_stamp":"2024-08-03T20:49:46.511193+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T01:41:53.449951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.3a","version":"0'0","reported_seq":10,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-08-03T20:49:47.566263+0000","last_change":"2024-08-03T20:49:47.560128+0000","last_active":"2024-08-03T20:49:47.566263+0000","last_peered":"2024-08-03T20:49:47.566263+0000","last_clean":"2024-08-03T20:49:47.566263+0000","last_became_active":"2024-08-03T20:49:47.559228+0000","last_became_peered":"2024-08-03T20:49:47.559228+0000","last_unstale":"2024-08-03T20:49:47.566263+0000","last_undegraded":"2024-08-03T20:49:47.566263+0000","last_fullsized":"2024-08-03T20:49:47.566263+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:49:46.511193+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:49:46.511193+0000","last_clean_scrub_stamp":"2024-08-03T20:49:46.511193+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T00:21:17.057936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.39","version":"0'0","reported_seq":13,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-08-03T20:49:48.541904+0000","last_change":"2024-08-03T20:49:47.548447+0000","last_active":"2024-08-03T20:49:48.541904+0000","last_peered":"2024-08-03T20:49:48.541904+0000","last_clean":"2024-08-03T20:49:48.541904+0000","last_became_active":"2024-08-03T20:49:47.548139+0000","last_became_peered":"2024-08-03T20:49:47.548139+0000","last_unstale":"2024-08-03T20:49:48.541904+0000","last_undegraded":"2024-08-03T20:49:48.541904+0000","last_fullsized":"2024-08-03T20:49:48.541904+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:49:46.511193+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:49:46.511193+0000","last_clean_scrub_stamp":"2024-08-03T20:49:46.511193+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T01:03:17.929276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.38","version":"0'0","reported_seq":10,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-08-03T20:49:47.570821+0000","last_change":"2024-08-03T20:49:47.566477+0000","last_active":"2024-08-03T20:49:47.570821+0000","last_peered":"2024-08-03T20:49:47.570821+0000","last_clean":"2024-08-03T20:49:47.570821+0000","last_became_active":"2024-08-03T20:49:47.550738+0000","last_became_peered":"2024-08-03T20:49:47.550738+0000","last_unstale":"2024-08-03T20:49:47.570821+0000","last_undegraded":"2024-08-03T20:49:47.570821+0000","last_fullsized":"2024-08-03T20:49:47.570821+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:49:46.511193+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:49:46.511193+0000","last_clean_scrub_stamp":"2024-08-03T20:49:46.511193+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T05:07:25.106847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.37","version":"0'0","reported_seq":10,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-08-03T20:49:47.574604+0000","last_change":"2024-08-03T20:49:47.563222+0000","last_active":"2024-08-03T20:49:47.574604+0000","last_peered":"2024-08-03T20:49:47.574604+0000","last_clean":"2024-08-03T20:49:47.574604+0000","last_became_active":"2024-08-03T20:49:47.562912+0000","last_became_peered":"2024-08-03T20:49:47.562912+0000","last_unstale":"2024-08-03T20:49:47.574604+0000","last_undegraded":"2024-08-03T20:49:47.574604+0000","last_fullsized":"2024-08-03T20:49:47.574604+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:49:46.511193+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:49:46.511193+0000","last_clean_scrub_stamp":"2024-08-03T20:49:46.511193+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T01:56:27.053963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.36","version":"0'0","reported_seq":10,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-08-03T20:49:47.575649+0000","last_change":"2024-08-03T20:49:47.573922+0000","last_active":"2024-08-03T20:49:47.575649+0000","last_peered":"2024-08-03T20:49:47.575649+0000","last_clean":"2024-08-03T20:49:47.575649+0000","last_became_active":"2024-08-03T20:49:47.573483+0000","last_became_peered":"2024-08-03T20:49:47.573483+0000","last_unstale":"2024-08-03T20:49:47.575649+0000","last_undegraded":"2024-08-03T20:49:47.575649+0000","last_fullsized":"2024-08-03T20:49:47.575649+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:49:46.511193+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:49:46.511193+0000","last_clean_scrub_stamp":"2024-08-03T20:49:46.511193+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T00:56:59.567156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.35","version":"0'0","reported_seq":10,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-08-03T20:49:47.570741+0000","last_change":"2024-08-03T20:49:47.562396+0000","last_active":"2024-08-03T20:49:47.570741+0000","last_peered":"2024-08-03T20:49:47.570741+0000","last_clean":"2024-08-03T20:49:47.570741+0000","last_became_active":"2024-08-03T20:49:47.561104+0000","last_became_peered":"2024-08-03T20:49:47.561104+0000","last_unstale":"2024-08-03T20:49:47.570741+0000","last_undegraded":"2024-08-03T20:49:47.570741+0000","last_fullsized":"2024-08-03T20:49:47.570741+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:49:46.511193+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:49:46.511193+0000","last_clean_scrub_stamp":"2024-08-03T20:49:46.511193+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T08:00:44.546502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.34","version":"0'0","reported_seq":13,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-08-03T20:49:48.676574+0000","last_change":"2024-08-03T20:49:47.558674+0000","last_active":"2024-08-03T20:49:48.676574+0000","last_peered":"2024-08-03T20:49:48.676574+0000","last_clean":"2024-08-03T20:49:48.676574+0000","last_became_active":"2024-08-03T20:49:47.558001+0000","last_became_peered":"2024-08-03T20:49:47.558001+0000","last_unstale":"2024-08-03T20:49:48.676574+0000","last_undegraded":"2024-08-03T20:49:48.676574+0000","last_fullsized":"2024-08-03T20:49:48.676574+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:49:46.511193+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:49:46.511193+0000","last_clean_scrub_stamp":"2024-08-03T20:49:46.511193+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-04T22:05:38.337422+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.33","version":"0'0","reported_seq":10,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-08-03T20:49:47.564715+0000","last_change":"2024-08-03T20:49:47.560780+0000","last_active":"2024-08-03T20:49:47.564715+0000","last_peered":"2024-08-03T20:49:47.564715+0000","last_clean":"2024-08-03T20:49:47.564715+0000","last_became_active":"2024-08-03T20:49:47.560476+0000","last_became_peered":"2024-08-03T20:49:47.560476+0000","last_unstale":"2024-08-03T20:49:47.564715+0000","last_undegraded":"2024-08-03T20:49:47.564715+0000","last_fullsized":"2024-08-03T20:49:47.564715+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:49:46.511193+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:49:46.511193+0000","last_clean_scrub_stamp":"2024-08-03T20:49:46.511193+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T04:55:41.532489+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.32","version":"0'0","reported_seq":10,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-08-03T20:49:47.570108+0000","last_change":"2024-08-03T20:49:47.560570+0000","last_active":"2024-08-03T20:49:47.570108+0000","last_peered":"2024-08-03T20:49:47.570108+0000","last_clean":"2024-08-03T20:49:47.570108+0000","last_became_active":"2024-08-03T20:49:47.551713+0000","last_became_peered":"2024-08-03T20:49:47.551713+0000","last_unstale":"2024-08-03T20:49:47.570108+0000","last_undegraded":"2024-08-03T20:49:47.570108+0000","last_fullsized":"2024-08-03T20:49:47.570108+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:49:46.511193+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:49:46.511193+0000","last_clean_scrub_stamp":"2024-08-03T20:49:46.511193+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T05:40:42.992873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.31","version":"0'0","reported_seq":10,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-08-03T20:49:47.570643+0000","last_change":"2024-08-03T20:49:47.566873+0000","last_active":"2024-08-03T20:49:47.570643+0000","last_peered":"2024-08-03T20:49:47.570643+0000","last_clean":"2024-08-03T20:49:47.570643+0000","last_became_active":"2024-08-03T20:49:47.559897+0000","last_became_peered":"2024-08-03T20:49:47.559897+0000","last_unstale":"2024-08-03T20:49:47.570643+0000","last_undegraded":"2024-08-03T20:49:47.570643+0000","last_fullsized":"2024-08-03T20:49:47.570643+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:49:46.511193+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:49:46.511193+0000","last_clean_scrub_stamp":"2024-08-03T20:49:46.511193+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T00:23:24.876293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.30","version":"0'0","reported_seq":13,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-08-03T20:49:48.776798+0000","last_change":"2024-08-03T20:49:47.572014+0000","last_active":"2024-08-03T20:49:48.776798+0000","last_peered":"2024-08-03T20:49:48.776798+0000","last_clean":"2024-08-03T20:49:48.776798+0000","last_became_active":"2024-08-03T20:49:47.571372+0000","last_became_peered":"2024-08-03T20:49:47.571372+0000","last_unstale":"2024-08-03T20:49:48.776798+0000","last_undegraded":"2024-08-03T20:49:48.776798+0000","last_fullsized":"2024-08-03T20:49:48.776798+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:49:46.511193+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:49:46.511193+0000","last_clean_scrub_stamp":"2024-08-03T20:49:46.511193+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T01:34:35.362747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.2f","version":"0'0","reported_seq":10,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-08-03T20:49:47.546471+0000","last_change":"2024-08-03T20:49:47.536872+0000","last_active":"2024-08-03T20:49:47.546471+0000","last_peered":"2024-08-03T20:49:47.546471+0000","last_clean":"2024-08-03T20:49:47.546471+0000","last_became_active":"2024-08-03T20:49:47.536480+0000","last_became_peered":"2024-08-03T20:49:47.536480+0000","last_unstale":"2024-08-03T20:49:47.546471+0000","last_undegraded":"2024-08-03T20:49:47.546471+0000","last_fullsized":"2024-08-03T20:49:47.546471+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:49:46.511193+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:49:46.511193+0000","last_clean_scrub_stamp":"2024-08-03T20:49:46.511193+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T06:15:04.475315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.2e","version":"0'0","reported_seq":10,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-08-03T20:49:47.550568+0000","last_change":"2024-08-03T20:49:47.547642+0000","last_active":"2024-08-03T20:49:47.550568+0000","last_peered":"2024-08-03T20:49:47.550568+0000","last_clean":"2024-08-03T20:49:47.550568+0000","last_became_active":"2024-08-03T20:49:47.543698+0000","last_became_peered":"2024-08-03T20:49:47.543698+0000","last_unstale":"2024-08-03T20:49:47.550568+0000","last_undegraded":"2024-08-03T20:49:47.550568+0000","last_fullsized":"2024-08-03T20:49:47.550568+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:49:46.511193+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:49:46.511193+0000","last_clean_scrub_stamp":"2024-08-03T20:49:46.511193+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T06:20:54.091363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.2d","version":"0'0","reported_seq":13,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-08-03T20:49:48.777135+0000","last_change":"2024-08-03T20:49:47.571629+0000","last_active":"2024-08-03T20:49:48.777135+0000","last_peered":"2024-08-03T20:49:48.777135+0000","last_clean":"2024-08-03T20:49:48.777135+0000","last_became_active":"2024-08-03T20:49:47.571254+0000","last_became_peered":"2024-08-03T20:49:47.571254+0000","last_unstale":"2024-08-03T20:49:48.777135+0000","last_undegraded":"2024-08-03T20:49:48.777135+0000","last_fullsized":"2024-08-03T20:49:48.777135+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:49:46.511193+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:49:46.511193+0000","last_clean_scrub_stamp":"2024-08-03T20:49:46.511193+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T05:07:25.106847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.2c","version":"0'0","reported_seq":10,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-08-03T20:49:47.573315+0000","last_change":"2024-08-03T20:49:47.571657+0000","last_active":"2024-08-03T20:49:47.573315+0000","last_peered":"2024-08-03T20:49:47.573315+0000","last_clean":"2024-08-03T20:49:47.573315+0000","last_became_active":"2024-08-03T20:49:47.571352+0000","last_became_peered":"2024-08-03T20:49:47.571352+0000","last_unstale":"2024-08-03T20:49:47.573315+0000","last_undegraded":"2024-08-03T20:49:47.573315+0000","last_fullsized":"2024-08-03T20:49:47.573315+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:49:46.511193+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:49:46.511193+0000","last_clean_scrub_stamp":"2024-08-03T20:49:46.511193+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T07:04:32.609067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.2b","version":"0'0","reported_seq":10,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-08-03T20:49:47.569044+0000","last_change":"2024-08-03T20:49:47.563458+0000","last_active":"2024-08-03T20:49:47.569044+0000","last_peered":"2024-08-03T20:49:47.569044+0000","last_clean":"2024-08-03T20:49:47.569044+0000","last_became_active":"2024-08-03T20:49:47.563174+0000","last_became_peered":"2024-08-03T20:49:47.563174+0000","last_unstale":"2024-08-03T20:49:47.569044+0000","last_undegraded":"2024-08-03T20:49:47.569044+0000","last_fullsized":"2024-08-03T20:49:47.569044+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:49:46.511193+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:49:46.511193+0000","last_clean_scrub_stamp":"2024-08-03T20:49:46.511193+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T00:30:12.811578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.2a","version":"0'0","reported_seq":10,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-08-03T20:49:47.573962+0000","last_change":"2024-08-03T20:49:47.571634+0000","last_active":"2024-08-03T20:49:47.573962+0000","last_peered":"2024-08-03T20:49:47.573962+0000","last_clean":"2024-08-03T20:49:47.573962+0000","last_became_active":"2024-08-03T20:49:47.571270+0000","last_became_peered":"2024-08-03T20:49:47.571270+0000","last_unstale":"2024-08-03T20:49:47.573962+0000","last_undegraded":"2024-08-03T20:49:47.573962+0000","last_fullsized":"2024-08-03T20:49:47.573962+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:49:46.511193+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:49:46.511193+0000","last_clean_scrub_stamp":"2024-08-03T20:49:46.511193+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T05:16:35.717962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.29","version":"0'0","reported_seq":13,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-08-03T20:49:48.776933+0000","last_change":"2024-08-03T20:49:47.544241+0000","last_active":"2024-08-03T20:49:48.776933+0000","last_peered":"2024-08-03T20:49:48.776933+0000","last_clean":"2024-08-03T20:49:48.776933+0000","last_became_active":"2024-08-03T20:49:47.543885+0000","last_became_peered":"2024-08-03T20:49:47.543885+0000","last_unstale":"2024-08-03T20:49:48.776933+0000","last_undegraded":"2024-08-03T20:49:48.776933+0000","last_fullsized":"2024-08-03T20:49:48.776933+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:49:46.511193+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:49:46.511193+0000","last_clean_scrub_stamp":"2024-08-03T20:49:46.511193+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-04T22:38:27.858926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.28","version":"0'0","reported_seq":13,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-08-03T20:49:48.776843+0000","last_change":"2024-08-03T20:49:47.540376+0000","last_active":"2024-08-03T20:49:48.776843+0000","last_peered":"2024-08-03T20:49:48.776843+0000","last_clean":"2024-08-03T20:49:48.776843+0000","last_became_active":"2024-08-03T20:49:47.540075+0000","last_became_peered":"2024-08-03T20:49:47.540075+0000","last_unstale":"2024-08-03T20:49:48.776843+0000","last_undegraded":"2024-08-03T20:49:48.776843+0000","last_fullsized":"2024-08-03T20:49:48.776843+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:49:46.511193+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:49:46.511193+0000","last_clean_scrub_stamp":"2024-08-03T20:49:46.511193+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-04T22:39:10.883457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.27","version":"0'0","reported_seq":13,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-08-03T20:49:48.777115+0000","last_change":"2024-08-03T20:49:47.549654+0000","last_active":"2024-08-03T20:49:48.777115+0000","last_peered":"2024-08-03T20:49:48.777115+0000","last_clean":"2024-08-03T20:49:48.777115+0000","last_became_active":"2024-08-03T20:49:47.549359+0000","last_became_peered":"2024-08-03T20:49:47.549359+0000","last_unstale":"2024-08-03T20:49:48.777115+0000","last_undegraded":"2024-08-03T20:49:48.777115+0000","last_fullsized":"2024-08-03T20:49:48.777115+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:49:46.511193+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:49:46.511193+0000","last_clean_scrub_stamp":"2024-08-03T20:49:46.511193+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T01:56:27.053963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.26","version":"0'0","reported_seq":10,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-08-03T20:49:47.568027+0000","last_change":"2024-08-03T20:49:47.566895+0000","last_active":"2024-08-03T20:49:47.568027+0000","last_peered":"2024-08-03T20:49:47.568027+0000","last_clean":"2024-08-03T20:49:47.568027+0000","last_became_active":"2024-08-03T20:49:47.566511+0000","last_became_peered":"2024-08-03T20:49:47.566511+0000","last_unstale":"2024-08-03T20:49:47.568027+0000","last_undegraded":"2024-08-03T20:49:47.568027+0000","last_fullsized":"2024-08-03T20:49:47.568027+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:49:46.511193+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:49:46.511193+0000","last_clean_scrub_stamp":"2024-08-03T20:49:46.511193+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T02:07:30.765109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.25","version":"0'0","reported_seq":10,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-08-03T20:49:47.546952+0000","last_change":"2024-08-03T20:49:47.538651+0000","last_active":"2024-08-03T20:49:47.546952+0000","last_peered":"2024-08-03T20:49:47.546952+0000","last_clean":"2024-08-03T20:49:47.546952+0000","last_became_active":"2024-08-03T20:49:47.538347+0000","last_became_peered":"2024-08-03T20:49:47.538347+0000","last_unstale":"2024-08-03T20:49:47.546952+0000","last_undegraded":"2024-08-03T20:49:47.546952+0000","last_fullsized":"2024-08-03T20:49:47.546952+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:49:46.511193+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:49:46.511193+0000","last_clean_scrub_stamp":"2024-08-03T20:49:46.511193+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T02:10:33.849652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.24","version":"0'0","reported_seq":13,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-08-03T20:49:48.676651+0000","last_change":"2024-08-03T20:49:47.558978+0000","last_active":"2024-08-03T20:49:48.676651+0000","last_peered":"2024-08-03T20:49:48.676651+0000","last_clean":"2024-08-03T20:49:48.676651+0000","last_became_active":"2024-08-03T20:49:47.557136+0000","last_became_peered":"2024-08-03T20:49:47.557136+0000","last_unstale":"2024-08-03T20:49:48.676651+0000","last_undegraded":"2024-08-03T20:49:48.676651+0000","last_fullsized":"2024-08-03T20:49:48.676651+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:49:46.511193+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:49:46.511193+0000","last_clean_scrub_stamp":"2024-08-03T20:49:46.511193+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T04:45:12.792950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.23","version":"0'0","reported_seq":13,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-08-03T20:49:48.777235+0000","last_change":"2024-08-03T20:49:47.547275+0000","last_active":"2024-08-03T20:49:48.777235+0000","last_peered":"2024-08-03T20:49:48.777235+0000","last_clean":"2024-08-03T20:49:48.777235+0000","last_became_active":"2024-08-03T20:49:47.546941+0000","last_became_peered":"2024-08-03T20:49:47.546941+0000","last_unstale":"2024-08-03T20:49:48.777235+0000","last_undegraded":"2024-08-03T20:49:48.777235+0000","last_fullsized":"2024-08-03T20:49:48.777235+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:49:46.511193+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:49:46.511193+0000","last_clean_scrub_stamp":"2024-08-03T20:49:46.511193+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T05:16:35.717962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.22","version":"0'0","reported_seq":10,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-08-03T20:49:47.546243+0000","last_change":"2024-08-03T20:49:47.539492+0000","last_active":"2024-08-03T20:49:47.546243+0000","last_peered":"2024-08-03T20:49:47.546243+0000","last_clean":"2024-08-03T20:49:47.546243+0000","last_became_active":"2024-08-03T20:49:47.539177+0000","last_became_peered":"2024-08-03T20:49:47.539177+0000","last_unstale":"2024-08-03T20:49:47.546243+0000","last_undegraded":"2024-08-03T20:49:47.546243+0000","last_fullsized":"2024-08-03T20:49:47.546243+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:49:46.511193+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:49:46.511193+0000","last_clean_scrub_stamp":"2024-08-03T20:49:46.511193+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T00:58:45.729626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.21","version":"0'0","reported_seq":10,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-08-03T20:49:47.567109+0000","last_change":"2024-08-03T20:49:47.562467+0000","last_active":"2024-08-03T20:49:47.567109+0000","last_peered":"2024-08-03T20:49:47.567109+0000","last_clean":"2024-08-03T20:49:47.567109+0000","last_became_active":"2024-08-03T20:49:47.560186+0000","last_became_peered":"2024-08-03T20:49:47.560186+0000","last_unstale":"2024-08-03T20:49:47.567109+0000","last_undegraded":"2024-08-03T20:49:47.567109+0000","last_fullsized":"2024-08-03T20:49:47.567109+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:49:46.511193+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:49:46.511193+0000","last_clean_scrub_stamp":"2024-08-03T20:49:46.511193+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T08:29:32.556925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.20","version":"0'0","reported_seq":10,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-08-03T20:49:47.553756+0000","last_change":"2024-08-03T20:49:47.546628+0000","last_active":"2024-08-03T20:49:47.553756+0000","last_peered":"2024-08-03T20:49:47.553756+0000","last_clean":"2024-08-03T20:49:47.553756+0000","last_became_active":"2024-08-03T20:49:47.546313+0000","last_became_peered":"2024-08-03T20:49:47.546313+0000","last_unstale":"2024-08-03T20:49:47.553756+0000","last_undegraded":"2024-08-03T20:49:47.553756+0000","last_fullsized":"2024-08-03T20:49:47.553756+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:49:46.511193+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:49:46.511193+0000","last_clean_scrub_stamp":"2024-08-03T20:49:46.511193+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-04T22:41:19.703398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.1e","version":"0'0","reported_seq":13,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-08-03T20:49:48.698883+0000","last_change":"2024-08-03T20:49:47.565194+0000","last_active":"2024-08-03T20:49:48.698883+0000","last_peered":"2024-08-03T20:49:48.698883+0000","last_clean":"2024-08-03T20:49:48.698883+0000","last_became_active":"2024-08-03T20:49:47.564861+0000","last_became_peered":"2024-08-03T20:49:47.564861+0000","last_unstale":"2024-08-03T20:49:48.698883+0000","last_undegraded":"2024-08-03T20:49:48.698883+0000","last_fullsized":"2024-08-03T20:49:48.698883+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:49:46.511193+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:49:46.511193+0000","last_clean_scrub_stamp":"2024-08-03T20:49:46.511193+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T08:12:22.699786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"44'4","reported_seq":17,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-08-03T20:49:47.569217+0000","last_change":"2024-08-03T20:49:46.545203+0000","last_active":"2024-08-03T20:49:47.569217+0000","last_peered":"2024-08-03T20:49:47.569217+0000","last_clean":"2024-08-03T20:49:47.569217+0000","last_became_active":"2024-08-03T20:49:46.544757+0000","last_became_peered":"2024-08-03T20:49:46.544757+0000","last_unstale":"2024-08-03T20:49:47.569217+0000","last_undegraded":"2024-08-03T20:49:47.569217+0000","last_fullsized":"2024-08-03T20:49:47.569217+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:49:45.503385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:49:45.503385+0000","last_clean_scrub_stamp":"2024-08-03T20:49:45.503385+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-04T21:15:34.746119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1f","version":"0'0","reported_seq":13,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-08-03T20:49:48.777218+0000","last_change":"2024-08-03T20:49:47.536235+0000","last_active":"2024-08-03T20:49:48.777218+0000","last_peered":"2024-08-03T20:49:48.777218+0000","last_clean":"2024-08-03T20:49:48.777218+0000","last_became_active":"2024-08-03T20:49:47.535885+0000","last_became_peered":"2024-08-03T20:49:47.535885+0000","last_unstale":"2024-08-03T20:49:48.777218+0000","last_undegraded":"2024-08-03T20:49:48.777218+0000","last_fullsized":"2024-08-03T20:49:48.777218+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:49:46.511193+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:49:46.511193+0000","last_clean_scrub_stamp":"2024-08-03T20:49:46.511193+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-04T21:25:36.997036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1e","version":"44'1","reported_seq":14,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-08-03T20:49:47.555705+0000","last_change":"2024-08-03T20:49:46.535877+0000","last_active":"2024-08-03T20:49:47.555705+0000","last_peered":"2024-08-03T20:49:47.555705+0000","last_clean":"2024-08-03T20:49:47.555705+0000","last_became_active":"2024-08-03T20:49:46.535339+0000","last_became_peered":"2024-08-03T20:49:46.535339+0000","last_unstale":"2024-08-03T20:49:47.555705+0000","last_undegraded":"2024-08-03T20:49:47.555705+0000","last_fullsized":"2024-08-03T20:49:47.555705+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:49:45.503385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:49:45.503385+0000","last_clean_scrub_stamp":"2024-08-03T20:49:45.503385+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T04:47:42.666138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.1c","version":"0'0","reported_seq":13,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-08-03T20:49:48.676753+0000","last_change":"2024-08-03T20:49:47.564973+0000","last_active":"2024-08-03T20:49:48.676753+0000","last_peered":"2024-08-03T20:49:48.676753+0000","last_clean":"2024-08-03T20:49:48.676753+0000","last_became_active":"2024-08-03T20:49:47.564637+0000","last_became_peered":"2024-08-03T20:49:47.564637+0000","last_unstale":"2024-08-03T20:49:48.676753+0000","last_undegraded":"2024-08-03T20:49:48.676753+0000","last_fullsized":"2024-08-03T20:49:48.676753+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:49:46.511193+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:49:46.511193+0000","last_clean_scrub_stamp":"2024-08-03T20:49:46.511193+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T06:34:48.402529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1d","version":"44'2","reported_seq":17,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-08-03T20:49:48.698893+0000","last_change":"2024-08-03T20:49:46.543320+0000","last_active":"2024-08-03T20:49:48.698893+0000","last_peered":"2024-08-03T20:49:48.698893+0000","last_clean":"2024-08-03T20:49:48.698893+0000","last_became_active":"2024-08-03T20:49:46.543004+0000","last_became_peered":"2024-08-03T20:49:46.543004+0000","last_unstale":"2024-08-03T20:49:48.698893+0000","last_undegraded":"2024-08-03T20:49:48.698893+0000","last_fullsized":"2024-08-03T20:49:48.698893+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:49:45.503385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:49:45.503385+0000","last_clean_scrub_stamp":"2024-08-03T20:49:45.503385+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-04T22:39:09.875648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.1d","version":"0'0","reported_seq":13,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-08-03T20:49:48.698928+0000","last_change":"2024-08-03T20:49:47.543428+0000","last_active":"2024-08-03T20:49:48.698928+0000","last_peered":"2024-08-03T20:49:48.698928+0000","last_clean":"2024-08-03T20:49:48.698928+0000","last_became_active":"2024-08-03T20:49:47.542839+0000","last_became_peered":"2024-08-03T20:49:47.542839+0000","last_unstale":"2024-08-03T20:49:48.698928+0000","last_undegraded":"2024-08-03T20:49:48.698928+0000","last_fullsized":"2024-08-03T20:49:48.698928+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:49:46.511193+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:49:46.511193+0000","last_clean_scrub_stamp":"2024-08-03T20:49:46.511193+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-04T21:25:36.997036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1c","version":"44'2","reported_seq":17,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-08-03T20:49:48.540875+0000","last_change":"2024-08-03T20:49:46.547041+0000","last_active":"2024-08-03T20:49:48.540875+0000","last_peered":"2024-08-03T20:49:48.540875+0000","last_clean":"2024-08-03T20:49:48.540875+0000","last_became_active":"2024-08-03T20:49:46.545518+0000","last_became_peered":"2024-08-03T20:49:46.545518+0000","last_unstale":"2024-08-03T20:49:48.540875+0000","last_undegraded":"2024-08-03T20:49:48.540875+0000","last_fullsized":"2024-08-03T20:49:48.540875+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:49:45.503385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:49:45.503385+0000","last_clean_scrub_stamp":"2024-08-03T20:49:45.503385+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T02:46:24.186241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1a","version":"0'0","reported_seq":10,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-08-03T20:49:47.542945+0000","last_change":"2024-08-03T20:49:47.537376+0000","last_active":"2024-08-03T20:49:47.542945+0000","last_peered":"2024-08-03T20:49:47.542945+0000","last_clean":"2024-08-03T20:49:47.542945+0000","last_became_active":"2024-08-03T20:49:47.537104+0000","last_became_peered":"2024-08-03T20:49:47.537104+0000","last_unstale":"2024-08-03T20:49:47.542945+0000","last_undegraded":"2024-08-03T20:49:47.542945+0000","last_fullsized":"2024-08-03T20:49:47.542945+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:49:46.511193+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:49:46.511193+0000","last_clean_scrub_stamp":"2024-08-03T20:49:46.511193+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T05:26:16.037808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"44'1","reported_seq":14,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-08-03T20:49:47.563445+0000","last_change":"2024-08-03T20:49:46.540951+0000","last_active":"2024-08-03T20:49:47.563445+0000","last_peered":"2024-08-03T20:49:47.563445+0000","last_clean":"2024-08-03T20:49:47.563445+0000","last_became_active":"2024-08-03T20:49:46.540594+0000","last_became_peered":"2024-08-03T20:49:46.540594+0000","last_unstale":"2024-08-03T20:49:47.563445+0000","last_undegraded":"2024-08-03T20:49:47.563445+0000","last_fullsized":"2024-08-03T20:49:47.563445+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:49:45.503385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:49:45.503385+0000","last_clean_scrub_stamp":"2024-08-03T20:49:45.503385+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T01:13:46.007414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1b","version":"0'0","reported_seq":10,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-08-03T20:49:47.572449+0000","last_change":"2024-08-03T20:49:47.568859+0000","last_active":"2024-08-03T20:49:47.572449+0000","last_peered":"2024-08-03T20:49:47.572449+0000","last_clean":"2024-08-03T20:49:47.572449+0000","last_became_active":"2024-08-03T20:49:47.568458+0000","last_became_peered":"2024-08-03T20:49:47.568458+0000","last_unstale":"2024-08-03T20:49:47.572449+0000","last_undegraded":"2024-08-03T20:49:47.572449+0000","last_fullsized":"2024-08-03T20:49:47.572449+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:49:46.511193+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:49:46.511193+0000","last_clean_scrub_stamp":"2024-08-03T20:49:46.511193+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T08:12:22.699786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1a","version":"44'1","reported_seq":14,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-08-03T20:49:47.553385+0000","last_change":"2024-08-03T20:49:46.539598+0000","last_active":"2024-08-03T20:49:47.553385+0000","last_peered":"2024-08-03T20:49:47.553385+0000","last_clean":"2024-08-03T20:49:47.553385+0000","last_became_active":"2024-08-03T20:49:46.539244+0000","last_became_peered":"2024-08-03T20:49:46.539244+0000","last_unstale":"2024-08-03T20:49:47.553385+0000","last_undegraded":"2024-08-03T20:49:47.553385+0000","last_fullsized":"2024-08-03T20:49:47.553385+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:49:45.503385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:49:45.503385+0000","last_clean_scrub_stamp":"2024-08-03T20:49:45.503385+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T05:40:57.056135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.18","version":"0'0","reported_seq":10,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-08-03T20:49:47.568183+0000","last_change":"2024-08-03T20:49:47.542493+0000","last_active":"2024-08-03T20:49:47.568183+0000","last_peered":"2024-08-03T20:49:47.568183+0000","last_clean":"2024-08-03T20:49:47.568183+0000","last_became_active":"2024-08-03T20:49:47.542128+0000","last_became_peered":"2024-08-03T20:49:47.542128+0000","last_unstale":"2024-08-03T20:49:47.568183+0000","last_undegraded":"2024-08-03T20:49:47.568183+0000","last_fullsized":"2024-08-03T20:49:47.568183+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:49:46.511193+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:49:46.511193+0000","last_clean_scrub_stamp":"2024-08-03T20:49:46.511193+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-04T22:42:14.349430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":15,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-08-03T20:49:48.676204+0000","last_change":"2024-08-03T20:49:46.550316+0000","last_active":"2024-08-03T20:49:48.676204+0000","last_peered":"2024-08-03T20:49:48.676204+0000","last_clean":"2024-08-03T20:49:48.676204+0000","last_became_active":"2024-08-03T20:49:46.549958+0000","last_became_peered":"2024-08-03T20:49:46.549958+0000","last_unstale":"2024-08-03T20:49:48.676204+0000","last_undegraded":"2024-08-03T20:49:48.676204+0000","last_fullsized":"2024-08-03T20:49:48.676204+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:49:45.503385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:49:45.503385+0000","last_clean_scrub_stamp":"2024-08-03T20:49:45.503385+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T05:16:34.710154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.19","version":"0'0","reported_seq":13,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-08-03T20:49:48.776813+0000","last_change":"2024-08-03T20:49:47.541945+0000","last_active":"2024-08-03T20:49:48.776813+0000","last_peered":"2024-08-03T20:49:48.776813+0000","last_clean":"2024-08-03T20:49:48.776813+0000","last_became_active":"2024-08-03T20:49:47.541629+0000","last_became_peered":"2024-08-03T20:49:47.541629+0000","last_unstale":"2024-08-03T20:49:48.776813+0000","last_undegraded":"2024-08-03T20:49:48.776813+0000","last_fullsized":"2024-08-03T20:49:48.776813+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:49:46.511193+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:49:46.511193+0000","last_clean_scrub_stamp":"2024-08-03T20:49:46.511193+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-04T22:42:14.349430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":13,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-08-03T20:49:47.525192+0000","last_change":"2024-08-03T20:49:46.548247+0000","last_active":"2024-08-03T20:49:47.525192+0000","last_peered":"2024-08-03T20:49:47.525192+0000","last_clean":"2024-08-03T20:49:47.525192+0000","last_became_active":"2024-08-03T20:49:46.547175+0000","last_became_peered":"2024-08-03T20:49:46.547175+0000","last_unstale":"2024-08-03T20:49:47.525192+0000","last_undegraded":"2024-08-03T20:49:47.525192+0000","last_fullsized":"2024-08-03T20:49:47.525192+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:49:45.503385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:49:45.503385+0000","last_clean_scrub_stamp":"2024-08-03T20:49:45.503385+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T05:18:55.568493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.16","version":"0'0","reported_seq":10,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-08-03T20:49:47.573139+0000","last_change":"2024-08-03T20:49:47.572228+0000","last_active":"2024-08-03T20:49:47.573139+0000","last_peered":"2024-08-03T20:49:47.573139+0000","last_clean":"2024-08-03T20:49:47.573139+0000","last_became_active":"2024-08-03T20:49:47.571868+0000","last_became_peered":"2024-08-03T20:49:47.571868+0000","last_unstale":"2024-08-03T20:49:47.573139+0000","last_undegraded":"2024-08-03T20:49:47.573139+0000","last_fullsized":"2024-08-03T20:49:47.573139+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:49:46.511193+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:49:46.511193+0000","last_clean_scrub_stamp":"2024-08-03T20:49:46.511193+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T03:10:36.106156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":13,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-08-03T20:49:47.525770+0000","last_change":"2024-08-03T20:49:46.537616+0000","last_active":"2024-08-03T20:49:47.525770+0000","last_peered":"2024-08-03T20:49:47.525770+0000","last_clean":"2024-08-03T20:49:47.525770+0000","last_became_active":"2024-08-03T20:49:46.536514+0000","last_became_peered":"2024-08-03T20:49:46.536514+0000","last_unstale":"2024-08-03T20:49:47.525770+0000","last_undegraded":"2024-08-03T20:49:47.525770+0000","last_fullsized":"2024-08-03T20:49:47.525770+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:49:45.503385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:49:45.503385+0000","last_clean_scrub_stamp":"2024-08-03T20:49:45.503385+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T06:59:07.982736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.17","version":"0'0","reported_seq":13,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-08-03T20:49:48.541141+0000","last_change":"2024-08-03T20:49:47.545330+0000","last_active":"2024-08-03T20:49:48.541141+0000","last_peered":"2024-08-03T20:49:48.541141+0000","last_clean":"2024-08-03T20:49:48.541141+0000","last_became_active":"2024-08-03T20:49:47.544609+0000","last_became_peered":"2024-08-03T20:49:47.544609+0000","last_unstale":"2024-08-03T20:49:48.541141+0000","last_undegraded":"2024-08-03T20:49:48.541141+0000","last_fullsized":"2024-08-03T20:49:48.541141+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:49:46.511193+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:49:46.511193+0000","last_clean_scrub_stamp":"2024-08-03T20:49:46.511193+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-04T23:10:50.091040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":13,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-08-03T20:49:47.527977+0000","last_change":"2024-08-03T20:49:46.550770+0000","last_active":"2024-08-03T20:49:47.527977+0000","last_peered":"2024-08-03T20:49:47.527977+0000","last_clean":"2024-08-03T20:49:47.527977+0000","last_became_active":"2024-08-03T20:49:46.550435+0000","last_became_peered":"2024-08-03T20:49:46.550435+0000","last_unstale":"2024-08-03T20:49:47.527977+0000","last_undegraded":"2024-08-03T20:49:47.527977+0000","last_fullsized":"2024-08-03T20:49:47.527977+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:49:45.503385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:49:45.503385+0000","last_clean_scrub_stamp":"2024-08-03T20:49:45.503385+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T06:23:00.389684+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.14","version":"0'0","reported_seq":10,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-08-03T20:49:47.566402+0000","last_change":"2024-08-03T20:49:47.561607+0000","last_active":"2024-08-03T20:49:47.566402+0000","last_peered":"2024-08-03T20:49:47.566402+0000","last_clean":"2024-08-03T20:49:47.566402+0000","last_became_active":"2024-08-03T20:49:47.558386+0000","last_became_peered":"2024-08-03T20:49:47.558386+0000","last_unstale":"2024-08-03T20:49:47.566402+0000","last_undegraded":"2024-08-03T20:49:47.566402+0000","last_fullsized":"2024-08-03T20:49:47.566402+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:49:46.511193+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:49:46.511193+0000","last_clean_scrub_stamp":"2024-08-03T20:49:46.511193+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T03:04:54.847476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"44'3","reported_seq":18,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-08-03T20:49:48.541038+0000","last_change":"2024-08-03T20:49:46.544026+0000","last_active":"2024-08-03T20:49:48.541038+0000","last_peered":"2024-08-03T20:49:48.541038+0000","last_clean":"2024-08-03T20:49:48.541038+0000","last_became_active":"2024-08-03T20:49:46.543652+0000","last_became_peered":"2024-08-03T20:49:46.543652+0000","last_unstale":"2024-08-03T20:49:48.541038+0000","last_undegraded":"2024-08-03T20:49:48.541038+0000","last_fullsized":"2024-08-03T20:49:48.541038+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:49:45.503385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:49:45.503385+0000","last_clean_scrub_stamp":"2024-08-03T20:49:45.503385+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T02:10:58.372826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.15","version":"0'0","reported_seq":13,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-08-03T20:49:48.540907+0000","last_change":"2024-08-03T20:49:47.545128+0000","last_active":"2024-08-03T20:49:48.540907+0000","last_peered":"2024-08-03T20:49:48.540907+0000","last_clean":"2024-08-03T20:49:48.540907+0000","last_became_active":"2024-08-03T20:49:47.544775+0000","last_became_peered":"2024-08-03T20:49:47.544775+0000","last_unstale":"2024-08-03T20:49:48.540907+0000","last_undegraded":"2024-08-03T20:49:48.540907+0000","last_fullsized":"2024-08-03T20:49:48.540907+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:49:46.511193+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:49:46.511193+0000","last_clean_scrub_stamp":"2024-08-03T20:49:46.511193+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T06:06:10.660971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.14","version":"44'6","reported_seq":19,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-08-03T20:49:47.570220+0000","last_change":"2024-08-03T20:49:46.536098+0000","last_active":"2024-08-03T20:49:47.570220+0000","last_peered":"2024-08-03T20:49:47.570220+0000","last_clean":"2024-08-03T20:49:47.570220+0000","last_became_active":"2024-08-03T20:49:46.535713+0000","last_became_peered":"2024-08-03T20:49:46.535713+0000","last_unstale":"2024-08-03T20:49:47.570220+0000","last_undegraded":"2024-08-03T20:49:47.570220+0000","last_fullsized":"2024-08-03T20:49:47.570220+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:49:45.503385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:49:45.503385+0000","last_clean_scrub_stamp":"2024-08-03T20:49:45.503385+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-04T22:38:26.098658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.12","version":"0'0","reported_seq":13,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-08-03T20:49:48.777153+0000","last_change":"2024-08-03T20:49:47.571616+0000","last_active":"2024-08-03T20:49:48.777153+0000","last_peered":"2024-08-03T20:49:48.777153+0000","last_clean":"2024-08-03T20:49:48.777153+0000","last_became_active":"2024-08-03T20:49:47.571180+0000","last_became_peered":"2024-08-03T20:49:47.571180+0000","last_unstale":"2024-08-03T20:49:48.777153+0000","last_undegraded":"2024-08-03T20:49:48.777153+0000","last_fullsized":"2024-08-03T20:49:48.777153+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:49:46.511193+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:49:46.511193+0000","last_clean_scrub_stamp":"2024-08-03T20:49:46.511193+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T08:12:22.699786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.13","version":"44'2","reported_seq":15,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-08-03T20:49:47.573622+0000","last_change":"2024-08-03T20:49:46.545991+0000","last_active":"2024-08-03T20:49:47.573622+0000","last_peered":"2024-08-03T20:49:47.573622+0000","last_clean":"2024-08-03T20:49:47.573622+0000","last_became_active":"2024-08-03T20:49:46.545520+0000","last_became_peered":"2024-08-03T20:49:46.545520+0000","last_unstale":"2024-08-03T20:49:47.573622+0000","last_undegraded":"2024-08-03T20:49:47.573622+0000","last_fullsized":"2024-08-03T20:49:47.573622+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:49:45.503385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:49:45.503385+0000","last_clean_scrub_stamp":"2024-08-03T20:49:45.503385+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-04T23:37:23.743502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.13","version":"0'0","reported_seq":10,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-08-03T20:49:47.572981+0000","last_change":"2024-08-03T20:49:47.570446+0000","last_active":"2024-08-03T20:49:47.572981+0000","last_peered":"2024-08-03T20:49:47.572981+0000","last_clean":"2024-08-03T20:49:47.572981+0000","last_became_active":"2024-08-03T20:49:47.568679+0000","last_became_peered":"2024-08-03T20:49:47.568679+0000","last_unstale":"2024-08-03T20:49:47.572981+0000","last_undegraded":"2024-08-03T20:49:47.572981+0000","last_fullsized":"2024-08-03T20:49:47.572981+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:49:46.511193+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:49:46.511193+0000","last_clean_scrub_stamp":"2024-08-03T20:49:46.511193+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-04T21:25:36.997036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":15,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-08-03T20:49:48.540793+0000","last_change":"2024-08-03T20:49:46.548199+0000","last_active":"2024-08-03T20:49:48.540793+0000","last_peered":"2024-08-03T20:49:48.540793+0000","last_clean":"2024-08-03T20:49:48.540793+0000","last_became_active":"2024-08-03T20:49:46.546021+0000","last_became_peered":"2024-08-03T20:49:46.546021+0000","last_unstale":"2024-08-03T20:49:48.540793+0000","last_undegraded":"2024-08-03T20:49:48.540793+0000","last_fullsized":"2024-08-03T20:49:48.540793+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:49:45.503385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:49:45.503385+0000","last_clean_scrub_stamp":"2024-08-03T20:49:45.503385+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-04T23:31:11.860305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.10","version":"0'0","reported_seq":10,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-08-03T20:49:47.565149+0000","last_change":"2024-08-03T20:49:47.555578+0000","last_active":"2024-08-03T20:49:47.565149+0000","last_peered":"2024-08-03T20:49:47.565149+0000","last_clean":"2024-08-03T20:49:47.565149+0000","last_became_active":"2024-08-03T20:49:47.555238+0000","last_became_peered":"2024-08-03T20:49:47.555238+0000","last_unstale":"2024-08-03T20:49:47.565149+0000","last_undegraded":"2024-08-03T20:49:47.565149+0000","last_fullsized":"2024-08-03T20:49:47.565149+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:49:46.511193+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:49:46.511193+0000","last_clean_scrub_stamp":"2024-08-03T20:49:46.511193+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T00:22:52.850737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":13,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-08-03T20:49:47.541450+0000","last_change":"2024-08-03T20:49:46.546071+0000","last_active":"2024-08-03T20:49:47.541450+0000","last_peered":"2024-08-03T20:49:47.541450+0000","last_clean":"2024-08-03T20:49:47.541450+0000","last_became_active":"2024-08-03T20:49:46.540634+0000","last_became_peered":"2024-08-03T20:49:46.540634+0000","last_unstale":"2024-08-03T20:49:47.541450+0000","last_undegraded":"2024-08-03T20:49:47.541450+0000","last_fullsized":"2024-08-03T20:49:47.541450+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:49:45.503385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:49:45.503385+0000","last_clean_scrub_stamp":"2024-08-03T20:49:45.503385+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T00:57:57.111997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.11","version":"0'0","reported_seq":13,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-08-03T20:49:48.541095+0000","last_change":"2024-08-03T20:49:47.546294+0000","last_active":"2024-08-03T20:49:48.541095+0000","last_peered":"2024-08-03T20:49:48.541095+0000","last_clean":"2024-08-03T20:49:48.541095+0000","last_became_active":"2024-08-03T20:49:47.545526+0000","last_became_peered":"2024-08-03T20:49:47.545526+0000","last_unstale":"2024-08-03T20:49:48.541095+0000","last_undegraded":"2024-08-03T20:49:48.541095+0000","last_fullsized":"2024-08-03T20:49:48.541095+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:49:46.511193+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:49:46.511193+0000","last_clean_scrub_stamp":"2024-08-03T20:49:46.511193+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T04:36:12.061926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.10","version":"44'8","reported_seq":21,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-08-03T20:49:47.573969+0000","last_change":"2024-08-03T20:49:46.536489+0000","last_active":"2024-08-03T20:49:47.573969+0000","last_peered":"2024-08-03T20:49:47.573969+0000","last_clean":"2024-08-03T20:49:47.573969+0000","last_became_active":"2024-08-03T20:49:46.536149+0000","last_became_peered":"2024-08-03T20:49:46.536149+0000","last_unstale":"2024-08-03T20:49:47.573969+0000","last_undegraded":"2024-08-03T20:49:47.573969+0000","last_fullsized":"2024-08-03T20:49:47.573969+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:49:45.503385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:49:45.503385+0000","last_clean_scrub_stamp":"2024-08-03T20:49:45.503385+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-04T22:10:43.371385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.e","version":"0'0","reported_seq":10,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-08-03T20:49:47.555877+0000","last_change":"2024-08-03T20:49:47.552334+0000","last_active":"2024-08-03T20:49:47.555877+0000","last_peered":"2024-08-03T20:49:47.555877+0000","last_clean":"2024-08-03T20:49:47.555877+0000","last_became_active":"2024-08-03T20:49:47.551481+0000","last_became_peered":"2024-08-03T20:49:47.551481+0000","last_unstale":"2024-08-03T20:49:47.555877+0000","last_undegraded":"2024-08-03T20:49:47.555877+0000","last_fullsized":"2024-08-03T20:49:47.555877+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:49:46.511193+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:49:46.511193+0000","last_clean_scrub_stamp":"2024-08-03T20:49:46.511193+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T01:34:47.270126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.f","version":"44'1","reported_seq":14,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-08-03T20:49:47.562672+0000","last_change":"2024-08-03T20:49:46.544181+0000","last_active":"2024-08-03T20:49:47.562672+0000","last_peered":"2024-08-03T20:49:47.562672+0000","last_clean":"2024-08-03T20:49:47.562672+0000","last_became_active":"2024-08-03T20:49:46.543855+0000","last_became_peered":"2024-08-03T20:49:46.543855+0000","last_unstale":"2024-08-03T20:49:47.562672+0000","last_undegraded":"2024-08-03T20:49:47.562672+0000","last_fullsized":"2024-08-03T20:49:47.562672+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:49:45.503385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:49:45.503385+0000","last_clean_scrub_stamp":"2024-08-03T20:49:45.503385+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T07:09:44.232554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.f","version":"0'0","reported_seq":10,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-08-03T20:49:47.576497+0000","last_change":"2024-08-03T20:49:47.574712+0000","last_active":"2024-08-03T20:49:47.576497+0000","last_peered":"2024-08-03T20:49:47.576497+0000","last_clean":"2024-08-03T20:49:47.576497+0000","last_became_active":"2024-08-03T20:49:47.574405+0000","last_became_peered":"2024-08-03T20:49:47.574405+0000","last_unstale":"2024-08-03T20:49:47.576497+0000","last_undegraded":"2024-08-03T20:49:47.576497+0000","last_fullsized":"2024-08-03T20:49:47.576497+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:49:46.511193+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:49:46.511193+0000","last_clean_scrub_stamp":"2024-08-03T20:49:46.511193+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T00:00:41.907040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.e","version":"44'2","reported_seq":15,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-08-03T20:49:47.558817+0000","last_change":"2024-08-03T20:49:46.545658+0000","last_active":"2024-08-03T20:49:47.558817+0000","last_peered":"2024-08-03T20:49:47.558817+0000","last_clean":"2024-08-03T20:49:47.558817+0000","last_became_active":"2024-08-03T20:49:46.545308+0000","last_became_peered":"2024-08-03T20:49:46.545308+0000","last_unstale":"2024-08-03T20:49:47.558817+0000","last_undegraded":"2024-08-03T20:49:47.558817+0000","last_fullsized":"2024-08-03T20:49:47.558817+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:49:45.503385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:49:45.503385+0000","last_clean_scrub_stamp":"2024-08-03T20:49:45.503385+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T04:58:01.002374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.c","version":"0'0","reported_seq":13,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-08-03T20:49:48.776972+0000","last_change":"2024-08-03T20:49:47.564300+0000","last_active":"2024-08-03T20:49:48.776972+0000","last_peered":"2024-08-03T20:49:48.776972+0000","last_clean":"2024-08-03T20:49:48.776972+0000","last_became_active":"2024-08-03T20:49:47.563994+0000","last_became_peered":"2024-08-03T20:49:47.563994+0000","last_unstale":"2024-08-03T20:49:48.776972+0000","last_undegraded":"2024-08-03T20:49:48.776972+0000","last_fullsized":"2024-08-03T20:49:48.776972+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:49:46.511193+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:49:46.511193+0000","last_clean_scrub_stamp":"2024-08-03T20:49:46.511193+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T05:40:42.992873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.d","version":"44'1","reported_seq":16,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-08-03T20:49:48.676597+0000","last_change":"2024-08-03T20:49:46.549197+0000","last_active":"2024-08-03T20:49:48.676597+0000","last_peered":"2024-08-03T20:49:48.676597+0000","last_clean":"2024-08-03T20:49:48.676597+0000","last_became_active":"2024-08-03T20:49:46.548326+0000","last_became_peered":"2024-08-03T20:49:46.548326+0000","last_unstale":"2024-08-03T20:49:48.676597+0000","last_undegraded":"2024-08-03T20:49:48.676597+0000","last_fullsized":"2024-08-03T20:49:48.676597+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:49:45.503385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:49:45.503385+0000","last_clean_scrub_stamp":"2024-08-03T20:49:45.503385+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T08:04:29.947302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.d","version":"0'0","reported_seq":13,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-08-03T20:49:48.777040+0000","last_change":"2024-08-03T20:49:47.547432+0000","last_active":"2024-08-03T20:49:48.777040+0000","last_peered":"2024-08-03T20:49:48.777040+0000","last_clean":"2024-08-03T20:49:48.777040+0000","last_became_active":"2024-08-03T20:49:47.545994+0000","last_became_peered":"2024-08-03T20:49:47.545994+0000","last_unstale":"2024-08-03T20:49:48.777040+0000","last_undegraded":"2024-08-03T20:49:48.777040+0000","last_fullsized":"2024-08-03T20:49:48.777040+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:49:46.511193+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:49:46.511193+0000","last_clean_scrub_stamp":"2024-08-03T20:49:46.511193+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T08:00:44.546502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":13,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-08-03T20:49:47.541771+0000","last_change":"2024-08-03T20:49:46.545204+0000","last_active":"2024-08-03T20:49:47.541771+0000","last_peered":"2024-08-03T20:49:47.541771+0000","last_clean":"2024-08-03T20:49:47.541771+0000","last_became_active":"2024-08-03T20:49:46.544838+0000","last_became_peered":"2024-08-03T20:49:46.544838+0000","last_unstale":"2024-08-03T20:49:47.541771+0000","last_undegraded":"2024-08-03T20:49:47.541771+0000","last_fullsized":"2024-08-03T20:49:47.541771+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:49:45.503385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:49:45.503385+0000","last_clean_scrub_stamp":"2024-08-03T20:49:45.503385+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T04:52:58.067545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.a","version":"0'0","reported_seq":10,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-08-03T20:49:47.566237+0000","last_change":"2024-08-03T20:49:47.559214+0000","last_active":"2024-08-03T20:49:47.566237+0000","last_peered":"2024-08-03T20:49:47.566237+0000","last_clean":"2024-08-03T20:49:47.566237+0000","last_became_active":"2024-08-03T20:49:47.558851+0000","last_became_peered":"2024-08-03T20:49:47.558851+0000","last_unstale":"2024-08-03T20:49:47.566237+0000","last_undegraded":"2024-08-03T20:49:47.566237+0000","last_fullsized":"2024-08-03T20:49:47.566237+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:49:46.511193+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:49:46.511193+0000","last_clean_scrub_stamp":"2024-08-03T20:49:46.511193+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T07:48:58.186945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.b","version":"44'2","reported_seq":15,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-08-03T20:49:47.563524+0000","last_change":"2024-08-03T20:49:46.541257+0000","last_active":"2024-08-03T20:49:47.563524+0000","last_peered":"2024-08-03T20:49:47.563524+0000","last_clean":"2024-08-03T20:49:47.563524+0000","last_became_active":"2024-08-03T20:49:46.540935+0000","last_became_peered":"2024-08-03T20:49:46.540935+0000","last_unstale":"2024-08-03T20:49:47.563524+0000","last_undegraded":"2024-08-03T20:49:47.563524+0000","last_fullsized":"2024-08-03T20:49:47.563524+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:49:45.503385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:49:45.503385+0000","last_clean_scrub_stamp":"2024-08-03T20:49:45.503385+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-04T21:39:34.699786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.b","version":"0'0","reported_seq":13,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-08-03T20:49:48.698708+0000","last_change":"2024-08-03T20:49:47.547916+0000","last_active":"2024-08-03T20:49:48.698708+0000","last_peered":"2024-08-03T20:49:48.698708+0000","last_clean":"2024-08-03T20:49:48.698708+0000","last_became_active":"2024-08-03T20:49:47.547342+0000","last_became_peered":"2024-08-03T20:49:47.547342+0000","last_unstale":"2024-08-03T20:49:48.698708+0000","last_undegraded":"2024-08-03T20:49:48.698708+0000","last_fullsized":"2024-08-03T20:49:48.698708+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:49:46.511193+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:49:46.511193+0000","last_clean_scrub_stamp":"2024-08-03T20:49:46.511193+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T05:07:25.106847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":15,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-08-03T20:49:48.676223+0000","last_change":"2024-08-03T20:49:46.542345+0000","last_active":"2024-08-03T20:49:48.676223+0000","last_peered":"2024-08-03T20:49:48.676223+0000","last_clean":"2024-08-03T20:49:48.676223+0000","last_became_active":"2024-08-03T20:49:46.542016+0000","last_became_peered":"2024-08-03T20:49:46.542016+0000","last_unstale":"2024-08-03T20:49:48.676223+0000","last_undegraded":"2024-08-03T20:49:48.676223+0000","last_fullsized":"2024-08-03T20:49:48.676223+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:49:45.503385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:49:45.503385+0000","last_clean_scrub_stamp":"2024-08-03T20:49:45.503385+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T00:56:58.559348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.8","version":"0'0","reported_seq":10,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-08-03T20:49:47.553782+0000","last_change":"2024-08-03T20:49:47.549436+0000","last_active":"2024-08-03T20:49:47.553782+0000","last_peered":"2024-08-03T20:49:47.553782+0000","last_clean":"2024-08-03T20:49:47.553782+0000","last_became_active":"2024-08-03T20:49:47.548669+0000","last_became_peered":"2024-08-03T20:49:47.548669+0000","last_unstale":"2024-08-03T20:49:47.553782+0000","last_undegraded":"2024-08-03T20:49:47.553782+0000","last_fullsized":"2024-08-03T20:49:47.553782+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:49:46.511193+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:49:46.511193+0000","last_clean_scrub_stamp":"2024-08-03T20:49:46.511193+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-04T23:30:31.178961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":13,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-08-03T20:49:47.525630+0000","last_change":"2024-08-03T20:49:46.551559+0000","last_active":"2024-08-03T20:49:47.525630+0000","last_peered":"2024-08-03T20:49:47.525630+0000","last_clean":"2024-08-03T20:49:47.525630+0000","last_became_active":"2024-08-03T20:49:46.551160+0000","last_became_peered":"2024-08-03T20:49:46.551160+0000","last_unstale":"2024-08-03T20:49:47.525630+0000","last_undegraded":"2024-08-03T20:49:47.525630+0000","last_fullsized":"2024-08-03T20:49:47.525630+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:49:45.503385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:49:45.503385+0000","last_clean_scrub_stamp":"2024-08-03T20:49:45.503385+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T01:25:47.191916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.9","version":"0'0","reported_seq":10,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-08-03T20:49:47.567965+0000","last_change":"2024-08-03T20:49:47.551290+0000","last_active":"2024-08-03T20:49:47.567965+0000","last_peered":"2024-08-03T20:49:47.567965+0000","last_clean":"2024-08-03T20:49:47.567965+0000","last_became_active":"2024-08-03T20:49:47.550992+0000","last_became_peered":"2024-08-03T20:49:47.550992+0000","last_unstale":"2024-08-03T20:49:47.567965+0000","last_undegraded":"2024-08-03T20:49:47.567965+0000","last_fullsized":"2024-08-03T20:49:47.567965+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:49:46.511193+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:49:46.511193+0000","last_clean_scrub_stamp":"2024-08-03T20:49:46.511193+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-04T22:38:27.858926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":15,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-08-03T20:49:48.540649+0000","last_change":"2024-08-03T20:49:46.543863+0000","last_active":"2024-08-03T20:49:48.540649+0000","last_peered":"2024-08-03T20:49:48.540649+0000","last_clean":"2024-08-03T20:49:48.540649+0000","last_became_active":"2024-08-03T20:49:46.543554+0000","last_became_peered":"2024-08-03T20:49:46.543554+0000","last_unstale":"2024-08-03T20:49:48.540649+0000","last_undegraded":"2024-08-03T20:49:48.540649+0000","last_fullsized":"2024-08-03T20:49:48.540649+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:49:45.503385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:49:45.503385+0000","last_clean_scrub_stamp":"2024-08-03T20:49:45.503385+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-04T22:36:14.319670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.7","version":"0'0","reported_seq":13,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-08-03T20:49:48.777325+0000","last_change":"2024-08-03T20:49:47.571609+0000","last_active":"2024-08-03T20:49:48.777325+0000","last_peered":"2024-08-03T20:49:48.777325+0000","last_clean":"2024-08-03T20:49:48.777325+0000","last_became_active":"2024-08-03T20:49:47.571187+0000","last_became_peered":"2024-08-03T20:49:47.571187+0000","last_unstale":"2024-08-03T20:49:48.777325+0000","last_undegraded":"2024-08-03T20:49:48.777325+0000","last_fullsized":"2024-08-03T20:49:48.777325+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:49:46.511193+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:49:46.511193+0000","last_clean_scrub_stamp":"2024-08-03T20:49:46.511193+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T00:56:59.567156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":13,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-08-03T20:49:47.536276+0000","last_change":"2024-08-03T20:49:46.548110+0000","last_active":"2024-08-03T20:49:47.536276+0000","last_peered":"2024-08-03T20:49:47.536276+0000","last_clean":"2024-08-03T20:49:47.536276+0000","last_became_active":"2024-08-03T20:49:46.546463+0000","last_became_peered":"2024-08-03T20:49:46.546463+0000","last_unstale":"2024-08-03T20:49:47.536276+0000","last_undegraded":"2024-08-03T20:49:47.536276+0000","last_fullsized":"2024-08-03T20:49:47.536276+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:49:45.503385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:49:45.503385+0000","last_clean_scrub_stamp":"2024-08-03T20:49:45.503385+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-04T21:13:22.709036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.4","version":"0'0","reported_seq":10,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-08-03T20:49:47.561664+0000","last_change":"2024-08-03T20:49:47.537514+0000","last_active":"2024-08-03T20:49:47.561664+0000","last_peered":"2024-08-03T20:49:47.561664+0000","last_clean":"2024-08-03T20:49:47.561664+0000","last_became_active":"2024-08-03T20:49:47.536874+0000","last_became_peered":"2024-08-03T20:49:47.536874+0000","last_unstale":"2024-08-03T20:49:47.561664+0000","last_undegraded":"2024-08-03T20:49:47.561664+0000","last_fullsized":"2024-08-03T20:49:47.561664+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:49:46.511193+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:49:46.511193+0000","last_clean_scrub_stamp":"2024-08-03T20:49:46.511193+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T01:34:35.362747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":15,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-08-03T20:49:48.541166+0000","last_change":"2024-08-03T20:49:46.545183+0000","last_active":"2024-08-03T20:49:48.541166+0000","last_peered":"2024-08-03T20:49:48.541166+0000","last_clean":"2024-08-03T20:49:48.541166+0000","last_became_active":"2024-08-03T20:49:46.544805+0000","last_became_peered":"2024-08-03T20:49:46.544805+0000","last_unstale":"2024-08-03T20:49:48.541166+0000","last_undegraded":"2024-08-03T20:49:48.541166+0000","last_fullsized":"2024-08-03T20:49:48.541166+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:49:45.503385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:49:45.503385+0000","last_clean_scrub_stamp":"2024-08-03T20:49:45.503385+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-04T23:01:41.062466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.5","version":"0'0","reported_seq":13,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-08-03T20:49:48.541446+0000","last_change":"2024-08-03T20:49:47.561794+0000","last_active":"2024-08-03T20:49:48.541446+0000","last_peered":"2024-08-03T20:49:48.541446+0000","last_clean":"2024-08-03T20:49:48.541446+0000","last_became_active":"2024-08-03T20:49:47.561373+0000","last_became_peered":"2024-08-03T20:49:47.561373+0000","last_unstale":"2024-08-03T20:49:48.541446+0000","last_undegraded":"2024-08-03T20:49:48.541446+0000","last_fullsized":"2024-08-03T20:49:48.541446+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:49:46.511193+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:49:46.511193+0000","last_clean_scrub_stamp":"2024-08-03T20:49:46.511193+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-04T21:21:00.913460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":15,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-08-03T20:49:48.698593+0000","last_change":"2024-08-03T20:49:46.543003+0000","last_active":"2024-08-03T20:49:48.698593+0000","last_peered":"2024-08-03T20:49:48.698593+0000","last_clean":"2024-08-03T20:49:48.698593+0000","last_became_active":"2024-08-03T20:49:46.542688+0000","last_became_peered":"2024-08-03T20:49:46.542688+0000","last_unstale":"2024-08-03T20:49:48.698593+0000","last_undegraded":"2024-08-03T20:49:48.698593+0000","last_fullsized":"2024-08-03T20:49:48.698593+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:49:45.503385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:49:45.503385+0000","last_clean_scrub_stamp":"2024-08-03T20:49:45.503385+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-04T22:42:13.341621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.2","version":"0'0","reported_seq":10,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-08-03T20:49:47.566207+0000","last_change":"2024-08-03T20:49:47.557901+0000","last_active":"2024-08-03T20:49:47.566207+0000","last_peered":"2024-08-03T20:49:47.566207+0000","last_clean":"2024-08-03T20:49:47.566207+0000","last_became_active":"2024-08-03T20:49:47.556167+0000","last_became_peered":"2024-08-03T20:49:47.556167+0000","last_unstale":"2024-08-03T20:49:47.566207+0000","last_undegraded":"2024-08-03T20:49:47.566207+0000","last_fullsized":"2024-08-03T20:49:47.566207+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:49:46.511193+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:49:46.511193+0000","last_clean_scrub_stamp":"2024-08-03T20:49:46.511193+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T06:13:43.294746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":15,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-08-03T20:49:48.541075+0000","last_change":"2024-08-03T20:49:46.547432+0000","last_active":"2024-08-03T20:49:48.541075+0000","last_peered":"2024-08-03T20:49:48.541075+0000","last_clean":"2024-08-03T20:49:48.541075+0000","last_became_active":"2024-08-03T20:49:46.546968+0000","last_became_peered":"2024-08-03T20:49:46.546968+0000","last_unstale":"2024-08-03T20:49:48.541075+0000","last_undegraded":"2024-08-03T20:49:48.541075+0000","last_fullsized":"2024-08-03T20:49:48.541075+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:49:45.503385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:49:45.503385+0000","last_clean_scrub_stamp":"2024-08-03T20:49:45.503385+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T06:51:29.839797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.3","version":"0'0","reported_seq":10,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-08-03T20:49:47.569019+0000","last_change":"2024-08-03T20:49:47.561766+0000","last_active":"2024-08-03T20:49:47.569019+0000","last_peered":"2024-08-03T20:49:47.569019+0000","last_clean":"2024-08-03T20:49:47.569019+0000","last_became_active":"2024-08-03T20:49:47.559616+0000","last_became_peered":"2024-08-03T20:49:47.559616+0000","last_unstale":"2024-08-03T20:49:47.569019+0000","last_undegraded":"2024-08-03T20:49:47.569019+0000","last_fullsized":"2024-08-03T20:49:47.569019+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:49:46.511193+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:49:46.511193+0000","last_clean_scrub_stamp":"2024-08-03T20:49:46.511193+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T03:46:55.349698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":13,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-08-03T20:49:47.525907+0000","last_change":"2024-08-03T20:49:46.532586+0000","last_active":"2024-08-03T20:49:47.525907+0000","last_peered":"2024-08-03T20:49:47.525907+0000","last_clean":"2024-08-03T20:49:47.525907+0000","last_became_active":"2024-08-03T20:49:46.529863+0000","last_became_peered":"2024-08-03T20:49:46.529863+0000","last_unstale":"2024-08-03T20:49:47.525907+0000","last_undegraded":"2024-08-03T20:49:47.525907+0000","last_fullsized":"2024-08-03T20:49:47.525907+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:49:45.503385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:49:45.503385+0000","last_clean_scrub_stamp":"2024-08-03T20:49:45.503385+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T08:48:09.710603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.0","version":"0'0","reported_seq":13,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-08-03T20:49:48.676252+0000","last_change":"2024-08-03T20:49:47.554238+0000","last_active":"2024-08-03T20:49:48.676252+0000","last_peered":"2024-08-03T20:49:48.676252+0000","last_clean":"2024-08-03T20:49:48.676252+0000","last_became_active":"2024-08-03T20:49:47.553888+0000","last_became_peered":"2024-08-03T20:49:47.553888+0000","last_unstale":"2024-08-03T20:49:48.676252+0000","last_undegraded":"2024-08-03T20:49:48.676252+0000","last_fullsized":"2024-08-03T20:49:48.676252+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:49:46.511193+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:49:46.511193+0000","last_clean_scrub_stamp":"2024-08-03T20:49:46.511193+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T08:13:23.504204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":15,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-08-03T20:49:48.698605+0000","last_change":"2024-08-03T20:49:46.542820+0000","last_active":"2024-08-03T20:49:48.698605+0000","last_peered":"2024-08-03T20:49:48.698605+0000","last_clean":"2024-08-03T20:49:48.698605+0000","last_became_active":"2024-08-03T20:49:46.542496+0000","last_became_peered":"2024-08-03T20:49:46.542496+0000","last_unstale":"2024-08-03T20:49:48.698605+0000","last_undegraded":"2024-08-03T20:49:48.698605+0000","last_fullsized":"2024-08-03T20:49:48.698605+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:49:45.503385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:49:45.503385+0000","last_clean_scrub_stamp":"2024-08-03T20:49:45.503385+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T01:34:34.354939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.1","version":"0'0","reported_seq":10,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-08-03T20:49:47.563604+0000","last_change":"2024-08-03T20:49:47.562671+0000","last_active":"2024-08-03T20:49:47.563604+0000","last_peered":"2024-08-03T20:49:47.563604+0000","last_clean":"2024-08-03T20:49:47.563604+0000","last_became_active":"2024-08-03T20:49:47.562220+0000","last_became_peered":"2024-08-03T20:49:47.562220+0000","last_unstale":"2024-08-03T20:49:47.563604+0000","last_undegraded":"2024-08-03T20:49:47.563604+0000","last_fullsized":"2024-08-03T20:49:47.563604+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:49:46.511193+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:49:46.511193+0000","last_clean_scrub_stamp":"2024-08-03T20:49:46.511193+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T01:03:09.792447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":13,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-08-03T20:49:47.525556+0000","last_change":"2024-08-03T20:49:46.536206+0000","last_active":"2024-08-03T20:49:47.525556+0000","last_peered":"2024-08-03T20:49:47.525556+0000","last_clean":"2024-08-03T20:49:47.525556+0000","last_became_active":"2024-08-03T20:49:46.535893+0000","last_became_peered":"2024-08-03T20:49:46.535893+0000","last_unstale":"2024-08-03T20:49:47.525556+0000","last_undegraded":"2024-08-03T20:49:47.525556+0000","last_fullsized":"2024-08-03T20:49:47.525556+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:49:45.503385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:49:45.503385+0000","last_clean_scrub_stamp":"2024-08-03T20:49:45.503385+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T01:32:49.376691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.6","version":"0'0","reported_seq":10,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-08-03T20:49:47.575675+0000","last_change":"2024-08-03T20:49:47.574004+0000","last_active":"2024-08-03T20:49:47.575675+0000","last_peered":"2024-08-03T20:49:47.575675+0000","last_clean":"2024-08-03T20:49:47.575675+0000","last_became_active":"2024-08-03T20:49:47.573184+0000","last_became_peered":"2024-08-03T20:49:47.573184+0000","last_unstale":"2024-08-03T20:49:47.575675+0000","last_undegraded":"2024-08-03T20:49:47.575675+0000","last_fullsized":"2024-08-03T20:49:47.575675+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:49:46.511193+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:49:46.511193+0000","last_clean_scrub_stamp":"2024-08-03T20:49:46.511193+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T08:04:30.955111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.7","version":"44'2","reported_seq":17,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-08-03T20:49:48.676672+0000","last_change":"2024-08-03T20:49:46.544225+0000","last_active":"2024-08-03T20:49:48.676672+0000","last_peered":"2024-08-03T20:49:48.676672+0000","last_clean":"2024-08-03T20:49:48.676672+0000","last_became_active":"2024-08-03T20:49:46.543900+0000","last_became_peered":"2024-08-03T20:49:46.543900+0000","last_unstale":"2024-08-03T20:49:48.676672+0000","last_undegraded":"2024-08-03T20:49:48.676672+0000","last_fullsized":"2024-08-03T20:49:48.676672+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:49:45.503385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:49:45.503385+0000","last_clean_scrub_stamp":"2024-08-03T20:49:45.503385+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T00:00:40.899232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"16'72","reported_seq":154,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-08-03T20:49:48.541343+0000","last_change":"2024-08-03T20:47:30.547632+0000","last_active":"2024-08-03T20:49:48.541343+0000","last_peered":"2024-08-03T20:49:48.541343+0000","last_clean":"2024-08-03T20:49:48.541343+0000","last_became_active":"2024-08-03T20:47:30.546255+0000","last_became_peered":"2024-08-03T20:47:30.546255+0000","last_unstale":"2024-08-03T20:49:48.541343+0000","last_undegraded":"2024-08-03T20:49:48.541343+0000","last_fullsized":"2024-08-03T20:49:48.541343+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:47:28.874451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:47:28.874451+0000","last_clean_scrub_stamp":"2024-08-03T20:47:28.874451+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T00:19:40.548812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509471,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32272,"kb_used_data":1092,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339568,"statfs":{"total":96636764160,"available":96603717632,"internally_reserved":0,"allocated":1118208,"data_stored":801451,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":6,"up_from":15,"seq":64424509472,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31704,"kb_used_data":516,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340136,"statfs":{"total":96636764160,"available":96604299264,"internally_reserved":0,"allocated":528384,"data_stored":211665,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":5,"up_from":15,"seq":64424509472,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31700,"kb_used_data":512,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340140,"statfs":{"total":96636764160,"available":96604303360,"internally_reserved":0,"allocated":524288,"data_stored":211063,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509471,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31700,"kb_used_data":512,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340140,"statfs":{"total":96636764160,"available":96604303360,"internally_reserved":0,"allocated":524288,"data_stored":211063,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509473,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32268,"kb_used_data":1080,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339572,"statfs":{"total":96636764160,"available":96603721728,"internally_reserved":0,"allocated":1105920,"data_stored":795095,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542176,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31696,"kb_used_data":508,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340144,"statfs":{"total":96636764160,"available":96604307456,"internally_reserved":0,"allocated":520192,"data_stored":206305,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":3,"up_from":14,"seq":60129542174,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31696,"kb_used_data":508,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340144,"statfs":{"total":96636764160,"available":96604307456,"internally_reserved":0,"allocated":520192,"data_stored":205331,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":14,"seq":60129542175,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31700,"kb_used_data":512,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340140,"statfs":{"total":96636764160,"available":96604303360,"internally_reserved":0,"allocated":524288,"data_stored":206327,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-08-03T20:49:52.051 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-08-03T20:49:52.313 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:52.312+0000 7faa20000640 1 -- 172.21.15.32:0/789309818 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7faa1806f720 msgr2=0x7faa1806fb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:49:52.313 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:52.312+0000 7faa20000640 1 --2- 172.21.15.32:0/789309818 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7faa1806f720 0x7faa1806fb30 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7faa0c009860 tx=0x7faa0c02a530 comp rx=0 tx=0).stop 2024-08-03T20:49:52.313 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:52.312+0000 7faa20000640 1 -- 172.21.15.32:0/789309818 shutdown_connections 2024-08-03T20:49:52.313 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:52.312+0000 7faa20000640 1 --2- 172.21.15.32:0/789309818 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7faa1806f720 0x7faa1806fb30 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:49:52.313 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:52.312+0000 7faa20000640 1 -- 172.21.15.32:0/789309818 >> 172.21.15.32:0/789309818 conn(0x7faa180b1bd0 msgr2=0x7faa180b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:49:52.313 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:52.312+0000 7faa20000640 1 -- 172.21.15.32:0/789309818 shutdown_connections 2024-08-03T20:49:52.314 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:52.312+0000 7faa20000640 1 -- 172.21.15.32:0/789309818 wait complete. 2024-08-03T20:49:52.315 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:52.315+0000 7faa20000640 1 Processor -- start 2024-08-03T20:49:52.315 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:52.315+0000 7faa20000640 1 -- start start 2024-08-03T20:49:52.317 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:52.316+0000 7faa20000640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7faa1806f720 0x7faa181f62d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:49:52.317 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:52.316+0000 7faa20000640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7faa180ade80 0x7faa1815ec50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:49:52.317 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:52.316+0000 7faa20000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faa180b39a0 0x7faa1814c1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:49:52.317 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:52.316+0000 7faa20000640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7faa180aee70 con 0x7faa180b39a0 2024-08-03T20:49:52.317 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:52.316+0000 7faa20000640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7faa180aecf0 con 0x7faa1806f720 2024-08-03T20:49:52.317 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:52.316+0000 7faa20000640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7faa180aeff0 con 0x7faa180ade80 2024-08-03T20:49:52.317 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:52.316+0000 7faa1e000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faa180b39a0 0x7faa1814c1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:49:52.318 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:52.317+0000 7faa1cc00640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7faa180ade80 0x7faa1815ec50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:49:52.318 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:52.317+0000 7faa1d600640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7faa1806f720 0x7faa181f62d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:49:52.318 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:52.317+0000 7faa1e000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faa180b39a0 0x7faa1814c1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:38050/0 (socket says 172.21.15.32:38050) 2024-08-03T20:49:52.318 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:52.317+0000 7faa1e000640 1 -- 172.21.15.32:0/1210249686 learned_addr learned my addr 172.21.15.32:0/1210249686 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:49:52.319 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:52.317+0000 7faa1e000640 1 -- 172.21.15.32:0/1210249686 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7faa180ade80 msgr2=0x7faa1815ec50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:49:52.319 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:52.317+0000 7faa1e000640 1 --2- 172.21.15.32:0/1210249686 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7faa180ade80 0x7faa1815ec50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:49:52.320 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:52.317+0000 7faa1e000640 1 -- 172.21.15.32:0/1210249686 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7faa1806f720 msgr2=0x7faa181f62d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:49:52.320 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:52.317+0000 7faa1e000640 1 --2- 172.21.15.32:0/1210249686 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7faa1806f720 0x7faa181f62d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:49:52.320 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:52.317+0000 7faa1e000640 1 -- 172.21.15.32:0/1210249686 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faa180cc970 con 0x7faa180b39a0 2024-08-03T20:49:52.320 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:52.317+0000 7faa1cc00640 1 --2- 172.21.15.32:0/1210249686 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7faa180ade80 0x7faa1815ec50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-03T20:49:52.320 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:52.317+0000 7faa1e000640 1 --2- 172.21.15.32:0/1210249686 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faa180b39a0 0x7faa1814c1b0 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7faa1000c700 tx=0x7faa1000cbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:49:52.320 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:52.318+0000 7faa15600640 1 -- 172.21.15.32:0/1210249686 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa10003b70 con 0x7faa180b39a0 2024-08-03T20:49:52.320 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:52.318+0000 7faa15600640 1 -- 172.21.15.32:0/1210249686 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7faa10003d10 con 0x7faa180b39a0 2024-08-03T20:49:52.320 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:52.318+0000 7faa20000640 1 -- 172.21.15.32:0/1210249686 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faa1815f6f0 con 0x7faa180b39a0 2024-08-03T20:49:52.320 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:52.318+0000 7faa15600640 1 -- 172.21.15.32:0/1210249686 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa1000aab0 con 0x7faa180b39a0 2024-08-03T20:49:52.320 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:52.318+0000 7faa20000640 1 -- 172.21.15.32:0/1210249686 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faa181606a0 con 0x7faa180b39a0 2024-08-03T20:49:52.320 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:52.319+0000 7faa20000640 1 -- 172.21.15.32:0/1210249686 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faa1806fb30 con 0x7faa180b39a0 2024-08-03T20:49:52.321 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:52.320+0000 7faa15600640 1 -- 172.21.15.32:0/1210249686 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7faa1000ac50 con 0x7faa180b39a0 2024-08-03T20:49:52.322 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:52.322+0000 7faa15600640 1 --2- 172.21.15.32:0/1210249686 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fa9f8077010 0x7fa9f8079520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:49:52.322 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:52.322+0000 7faa15600640 1 -- 172.21.15.32:0/1210249686 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(45..45 src has 1..45) ==== 5709+0+0 (secure 0 0 0) 0x7faa100985f0 con 0x7faa180b39a0 2024-08-03T20:49:52.322 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:52.322+0000 7faa1d600640 1 --2- 172.21.15.32:0/1210249686 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fa9f8077010 0x7fa9f8079520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:49:52.323 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:52.323+0000 7faa1d600640 1 --2- 172.21.15.32:0/1210249686 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fa9f8077010 0x7fa9f8079520 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7faa0c034000 tx=0x7faa0c004730 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:49:52.326 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:52.326+0000 7faa15600640 1 -- 172.21.15.32:0/1210249686 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7faa10061210 con 0x7faa180b39a0 2024-08-03T20:49:52.486 INFO:tasks.ceph.mds.c.smithi032.stderr:2024-08-03T20:49:52.486+0000 7f7390400640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-08-03T20:49:52.527 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:52.527+0000 7faa20000640 1 -- 172.21.15.32:0/1210249686 --> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7faa18074a90 con 0x7fa9f8077010 2024-08-03T20:49:52.533 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:52.532+0000 7faa15600640 1 -- 172.21.15.32:0/1210249686 <== mgr.4106 v2:172.21.15.118:6800/4205950699 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+251943 (secure 0 0 0) 0x7faa18074a90 con 0x7fa9f8077010 2024-08-03T20:49:52.536 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-03T20:49:52.537 INFO:teuthology.orchestra.run.smithi032.stderr:dumped all 2024-08-03T20:49:52.538 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:52.537+0000 7faa20000640 1 -- 172.21.15.32:0/1210249686 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fa9f8077010 msgr2=0x7fa9f8079520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:49:52.538 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:52.537+0000 7faa20000640 1 --2- 172.21.15.32:0/1210249686 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fa9f8077010 0x7fa9f8079520 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7faa0c034000 tx=0x7faa0c004730 comp rx=0 tx=0).stop 2024-08-03T20:49:52.539 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:52.537+0000 7faa20000640 1 -- 172.21.15.32:0/1210249686 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faa180b39a0 msgr2=0x7faa1814c1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:49:52.539 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:52.537+0000 7faa20000640 1 --2- 172.21.15.32:0/1210249686 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faa180b39a0 0x7faa1814c1b0 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7faa1000c700 tx=0x7faa1000cbc0 comp rx=0 tx=0).stop 2024-08-03T20:49:52.539 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:52.537+0000 7faa20000640 1 -- 172.21.15.32:0/1210249686 shutdown_connections 2024-08-03T20:49:52.539 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:52.537+0000 7faa20000640 1 --2- 172.21.15.32:0/1210249686 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fa9f8077010 0x7fa9f8079520 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:49:52.539 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:52.537+0000 7faa20000640 1 --2- 172.21.15.32:0/1210249686 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faa180b39a0 0x7faa1814c1b0 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:49:52.539 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:52.537+0000 7faa20000640 1 --2- 172.21.15.32:0/1210249686 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7faa180ade80 0x7faa1815ec50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:49:52.539 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:52.537+0000 7faa20000640 1 --2- 172.21.15.32:0/1210249686 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7faa1806f720 0x7faa181f62d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:49:52.539 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:52.537+0000 7faa20000640 1 -- 172.21.15.32:0/1210249686 >> 172.21.15.32:0/1210249686 conn(0x7faa180b1bd0 msgr2=0x7faa180b3db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:49:52.539 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:52.537+0000 7faa20000640 1 -- 172.21.15.32:0/1210249686 shutdown_connections 2024-08-03T20:49:52.539 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:52.537+0000 7faa20000640 1 -- 172.21.15.32:0/1210249686 wait complete. 2024-08-03T20:49:52.562 INFO:teuthology.orchestra.run.smithi032.stdout:{"pg_ready":true,"pg_map":{"version":110,"stamp":"2024-08-03T20:49:52.258185+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":145,"num_write_kb":1642,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":110,"ondisk_log_size":110,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":99,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":254736,"kb_used_data":5240,"kb_used_omap":12,"kb_used_meta":249331,"kb_avail":754719984,"statfs":{"total":773094113280,"available":772833263616,"internally_reserved":0,"allocated":5365760,"data_stored":2848300,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":255315505},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":30,"apply_latency_ms":30,"commit_latency_ns":30000000,"apply_latency_ns":30000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001322"},"pg_stats":[{"pgid":"7.3f","version":"0'0","reported_seq":13,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-08-03T20:49:48.541258+0000","last_change":"2024-08-03T20:49:47.548778+0000","last_active":"2024-08-03T20:49:48.541258+0000","last_peered":"2024-08-03T20:49:48.541258+0000","last_clean":"2024-08-03T20:49:48.541258+0000","last_became_active":"2024-08-03T20:49:47.547674+0000","last_became_peered":"2024-08-03T20:49:47.547674+0000","last_unstale":"2024-08-03T20:49:48.541258+0000","last_undegraded":"2024-08-03T20:49:48.541258+0000","last_fullsized":"2024-08-03T20:49:48.541258+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:49:46.511193+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:49:46.511193+0000","last_clean_scrub_stamp":"2024-08-03T20:49:46.511193+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T03:39:25.547249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.3e","version":"0'0","reported_seq":13,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-08-03T20:49:48.676693+0000","last_change":"2024-08-03T20:49:47.568416+0000","last_active":"2024-08-03T20:49:48.676693+0000","last_peered":"2024-08-03T20:49:48.676693+0000","last_clean":"2024-08-03T20:49:48.676693+0000","last_became_active":"2024-08-03T20:49:47.568102+0000","last_became_peered":"2024-08-03T20:49:47.568102+0000","last_unstale":"2024-08-03T20:49:48.676693+0000","last_undegraded":"2024-08-03T20:49:48.676693+0000","last_fullsized":"2024-08-03T20:49:48.676693+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:49:46.511193+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:49:46.511193+0000","last_clean_scrub_stamp":"2024-08-03T20:49:46.511193+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-04T23:53:08.188600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.3d","version":"0'0","reported_seq":10,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-08-03T20:49:47.554505+0000","last_change":"2024-08-03T20:49:47.552482+0000","last_active":"2024-08-03T20:49:47.554505+0000","last_peered":"2024-08-03T20:49:47.554505+0000","last_clean":"2024-08-03T20:49:47.554505+0000","last_became_active":"2024-08-03T20:49:47.552169+0000","last_became_peered":"2024-08-03T20:49:47.552169+0000","last_unstale":"2024-08-03T20:49:47.554505+0000","last_undegraded":"2024-08-03T20:49:47.554505+0000","last_fullsized":"2024-08-03T20:49:47.554505+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:49:46.511193+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:49:46.511193+0000","last_clean_scrub_stamp":"2024-08-03T20:49:46.511193+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T00:48:27.200673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.3c","version":"0'0","reported_seq":10,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-08-03T20:49:47.570176+0000","last_change":"2024-08-03T20:49:47.543020+0000","last_active":"2024-08-03T20:49:47.570176+0000","last_peered":"2024-08-03T20:49:47.570176+0000","last_clean":"2024-08-03T20:49:47.570176+0000","last_became_active":"2024-08-03T20:49:47.542702+0000","last_became_peered":"2024-08-03T20:49:47.542702+0000","last_unstale":"2024-08-03T20:49:47.570176+0000","last_undegraded":"2024-08-03T20:49:47.570176+0000","last_fullsized":"2024-08-03T20:49:47.570176+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:49:46.511193+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:49:46.511193+0000","last_clean_scrub_stamp":"2024-08-03T20:49:46.511193+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-04T22:39:10.883457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.3b","version":"0'0","reported_seq":13,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-08-03T20:49:48.676439+0000","last_change":"2024-08-03T20:49:47.548807+0000","last_active":"2024-08-03T20:49:48.676439+0000","last_peered":"2024-08-03T20:49:48.676439+0000","last_clean":"2024-08-03T20:49:48.676439+0000","last_became_active":"2024-08-03T20:49:47.548490+0000","last_became_peered":"2024-08-03T20:49:47.548490+0000","last_unstale":"2024-08-03T20:49:48.676439+0000","last_undegraded":"2024-08-03T20:49:48.676439+0000","last_fullsized":"2024-08-03T20:49:48.676439+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:49:46.511193+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:49:46.511193+0000","last_clean_scrub_stamp":"2024-08-03T20:49:46.511193+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T01:41:53.449951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.3a","version":"0'0","reported_seq":10,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-08-03T20:49:47.566263+0000","last_change":"2024-08-03T20:49:47.560128+0000","last_active":"2024-08-03T20:49:47.566263+0000","last_peered":"2024-08-03T20:49:47.566263+0000","last_clean":"2024-08-03T20:49:47.566263+0000","last_became_active":"2024-08-03T20:49:47.559228+0000","last_became_peered":"2024-08-03T20:49:47.559228+0000","last_unstale":"2024-08-03T20:49:47.566263+0000","last_undegraded":"2024-08-03T20:49:47.566263+0000","last_fullsized":"2024-08-03T20:49:47.566263+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:49:46.511193+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:49:46.511193+0000","last_clean_scrub_stamp":"2024-08-03T20:49:46.511193+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T00:21:17.057936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.39","version":"0'0","reported_seq":13,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-08-03T20:49:48.541904+0000","last_change":"2024-08-03T20:49:47.548447+0000","last_active":"2024-08-03T20:49:48.541904+0000","last_peered":"2024-08-03T20:49:48.541904+0000","last_clean":"2024-08-03T20:49:48.541904+0000","last_became_active":"2024-08-03T20:49:47.548139+0000","last_became_peered":"2024-08-03T20:49:47.548139+0000","last_unstale":"2024-08-03T20:49:48.541904+0000","last_undegraded":"2024-08-03T20:49:48.541904+0000","last_fullsized":"2024-08-03T20:49:48.541904+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:49:46.511193+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:49:46.511193+0000","last_clean_scrub_stamp":"2024-08-03T20:49:46.511193+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T01:03:17.929276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.38","version":"0'0","reported_seq":10,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-08-03T20:49:47.570821+0000","last_change":"2024-08-03T20:49:47.566477+0000","last_active":"2024-08-03T20:49:47.570821+0000","last_peered":"2024-08-03T20:49:47.570821+0000","last_clean":"2024-08-03T20:49:47.570821+0000","last_became_active":"2024-08-03T20:49:47.550738+0000","last_became_peered":"2024-08-03T20:49:47.550738+0000","last_unstale":"2024-08-03T20:49:47.570821+0000","last_undegraded":"2024-08-03T20:49:47.570821+0000","last_fullsized":"2024-08-03T20:49:47.570821+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:49:46.511193+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:49:46.511193+0000","last_clean_scrub_stamp":"2024-08-03T20:49:46.511193+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T05:07:25.106847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.37","version":"0'0","reported_seq":10,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-08-03T20:49:47.574604+0000","last_change":"2024-08-03T20:49:47.563222+0000","last_active":"2024-08-03T20:49:47.574604+0000","last_peered":"2024-08-03T20:49:47.574604+0000","last_clean":"2024-08-03T20:49:47.574604+0000","last_became_active":"2024-08-03T20:49:47.562912+0000","last_became_peered":"2024-08-03T20:49:47.562912+0000","last_unstale":"2024-08-03T20:49:47.574604+0000","last_undegraded":"2024-08-03T20:49:47.574604+0000","last_fullsized":"2024-08-03T20:49:47.574604+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:49:46.511193+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:49:46.511193+0000","last_clean_scrub_stamp":"2024-08-03T20:49:46.511193+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T01:56:27.053963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.36","version":"0'0","reported_seq":10,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-08-03T20:49:47.575649+0000","last_change":"2024-08-03T20:49:47.573922+0000","last_active":"2024-08-03T20:49:47.575649+0000","last_peered":"2024-08-03T20:49:47.575649+0000","last_clean":"2024-08-03T20:49:47.575649+0000","last_became_active":"2024-08-03T20:49:47.573483+0000","last_became_peered":"2024-08-03T20:49:47.573483+0000","last_unstale":"2024-08-03T20:49:47.575649+0000","last_undegraded":"2024-08-03T20:49:47.575649+0000","last_fullsized":"2024-08-03T20:49:47.575649+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:49:46.511193+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:49:46.511193+0000","last_clean_scrub_stamp":"2024-08-03T20:49:46.511193+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T00:56:59.567156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.35","version":"0'0","reported_seq":10,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-08-03T20:49:47.570741+0000","last_change":"2024-08-03T20:49:47.562396+0000","last_active":"2024-08-03T20:49:47.570741+0000","last_peered":"2024-08-03T20:49:47.570741+0000","last_clean":"2024-08-03T20:49:47.570741+0000","last_became_active":"2024-08-03T20:49:47.561104+0000","last_became_peered":"2024-08-03T20:49:47.561104+0000","last_unstale":"2024-08-03T20:49:47.570741+0000","last_undegraded":"2024-08-03T20:49:47.570741+0000","last_fullsized":"2024-08-03T20:49:47.570741+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:49:46.511193+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:49:46.511193+0000","last_clean_scrub_stamp":"2024-08-03T20:49:46.511193+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T08:00:44.546502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.34","version":"0'0","reported_seq":13,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-08-03T20:49:48.676574+0000","last_change":"2024-08-03T20:49:47.558674+0000","last_active":"2024-08-03T20:49:48.676574+0000","last_peered":"2024-08-03T20:49:48.676574+0000","last_clean":"2024-08-03T20:49:48.676574+0000","last_became_active":"2024-08-03T20:49:47.558001+0000","last_became_peered":"2024-08-03T20:49:47.558001+0000","last_unstale":"2024-08-03T20:49:48.676574+0000","last_undegraded":"2024-08-03T20:49:48.676574+0000","last_fullsized":"2024-08-03T20:49:48.676574+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:49:46.511193+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:49:46.511193+0000","last_clean_scrub_stamp":"2024-08-03T20:49:46.511193+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-04T22:05:38.337422+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.33","version":"0'0","reported_seq":10,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-08-03T20:49:47.564715+0000","last_change":"2024-08-03T20:49:47.560780+0000","last_active":"2024-08-03T20:49:47.564715+0000","last_peered":"2024-08-03T20:49:47.564715+0000","last_clean":"2024-08-03T20:49:47.564715+0000","last_became_active":"2024-08-03T20:49:47.560476+0000","last_became_peered":"2024-08-03T20:49:47.560476+0000","last_unstale":"2024-08-03T20:49:47.564715+0000","last_undegraded":"2024-08-03T20:49:47.564715+0000","last_fullsized":"2024-08-03T20:49:47.564715+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:49:46.511193+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:49:46.511193+0000","last_clean_scrub_stamp":"2024-08-03T20:49:46.511193+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T04:55:41.532489+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.32","version":"0'0","reported_seq":10,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-08-03T20:49:47.570108+0000","last_change":"2024-08-03T20:49:47.560570+0000","last_active":"2024-08-03T20:49:47.570108+0000","last_peered":"2024-08-03T20:49:47.570108+0000","last_clean":"2024-08-03T20:49:47.570108+0000","last_became_active":"2024-08-03T20:49:47.551713+0000","last_became_peered":"2024-08-03T20:49:47.551713+0000","last_unstale":"2024-08-03T20:49:47.570108+0000","last_undegraded":"2024-08-03T20:49:47.570108+0000","last_fullsized":"2024-08-03T20:49:47.570108+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:49:46.511193+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:49:46.511193+0000","last_clean_scrub_stamp":"2024-08-03T20:49:46.511193+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T05:40:42.992873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.31","version":"0'0","reported_seq":10,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-08-03T20:49:47.570643+0000","last_change":"2024-08-03T20:49:47.566873+0000","last_active":"2024-08-03T20:49:47.570643+0000","last_peered":"2024-08-03T20:49:47.570643+0000","last_clean":"2024-08-03T20:49:47.570643+0000","last_became_active":"2024-08-03T20:49:47.559897+0000","last_became_peered":"2024-08-03T20:49:47.559897+0000","last_unstale":"2024-08-03T20:49:47.570643+0000","last_undegraded":"2024-08-03T20:49:47.570643+0000","last_fullsized":"2024-08-03T20:49:47.570643+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:49:46.511193+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:49:46.511193+0000","last_clean_scrub_stamp":"2024-08-03T20:49:46.511193+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T00:23:24.876293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.30","version":"0'0","reported_seq":13,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-08-03T20:49:48.776798+0000","last_change":"2024-08-03T20:49:47.572014+0000","last_active":"2024-08-03T20:49:48.776798+0000","last_peered":"2024-08-03T20:49:48.776798+0000","last_clean":"2024-08-03T20:49:48.776798+0000","last_became_active":"2024-08-03T20:49:47.571372+0000","last_became_peered":"2024-08-03T20:49:47.571372+0000","last_unstale":"2024-08-03T20:49:48.776798+0000","last_undegraded":"2024-08-03T20:49:48.776798+0000","last_fullsized":"2024-08-03T20:49:48.776798+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:49:46.511193+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:49:46.511193+0000","last_clean_scrub_stamp":"2024-08-03T20:49:46.511193+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T01:34:35.362747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.2f","version":"0'0","reported_seq":10,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-08-03T20:49:47.546471+0000","last_change":"2024-08-03T20:49:47.536872+0000","last_active":"2024-08-03T20:49:47.546471+0000","last_peered":"2024-08-03T20:49:47.546471+0000","last_clean":"2024-08-03T20:49:47.546471+0000","last_became_active":"2024-08-03T20:49:47.536480+0000","last_became_peered":"2024-08-03T20:49:47.536480+0000","last_unstale":"2024-08-03T20:49:47.546471+0000","last_undegraded":"2024-08-03T20:49:47.546471+0000","last_fullsized":"2024-08-03T20:49:47.546471+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:49:46.511193+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:49:46.511193+0000","last_clean_scrub_stamp":"2024-08-03T20:49:46.511193+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T06:15:04.475315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.2e","version":"0'0","reported_seq":10,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-08-03T20:49:47.550568+0000","last_change":"2024-08-03T20:49:47.547642+0000","last_active":"2024-08-03T20:49:47.550568+0000","last_peered":"2024-08-03T20:49:47.550568+0000","last_clean":"2024-08-03T20:49:47.550568+0000","last_became_active":"2024-08-03T20:49:47.543698+0000","last_became_peered":"2024-08-03T20:49:47.543698+0000","last_unstale":"2024-08-03T20:49:47.550568+0000","last_undegraded":"2024-08-03T20:49:47.550568+0000","last_fullsized":"2024-08-03T20:49:47.550568+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:49:46.511193+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:49:46.511193+0000","last_clean_scrub_stamp":"2024-08-03T20:49:46.511193+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T06:20:54.091363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.2d","version":"0'0","reported_seq":13,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-08-03T20:49:48.777135+0000","last_change":"2024-08-03T20:49:47.571629+0000","last_active":"2024-08-03T20:49:48.777135+0000","last_peered":"2024-08-03T20:49:48.777135+0000","last_clean":"2024-08-03T20:49:48.777135+0000","last_became_active":"2024-08-03T20:49:47.571254+0000","last_became_peered":"2024-08-03T20:49:47.571254+0000","last_unstale":"2024-08-03T20:49:48.777135+0000","last_undegraded":"2024-08-03T20:49:48.777135+0000","last_fullsized":"2024-08-03T20:49:48.777135+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:49:46.511193+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:49:46.511193+0000","last_clean_scrub_stamp":"2024-08-03T20:49:46.511193+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T05:07:25.106847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.2c","version":"0'0","reported_seq":10,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-08-03T20:49:47.573315+0000","last_change":"2024-08-03T20:49:47.571657+0000","last_active":"2024-08-03T20:49:47.573315+0000","last_peered":"2024-08-03T20:49:47.573315+0000","last_clean":"2024-08-03T20:49:47.573315+0000","last_became_active":"2024-08-03T20:49:47.571352+0000","last_became_peered":"2024-08-03T20:49:47.571352+0000","last_unstale":"2024-08-03T20:49:47.573315+0000","last_undegraded":"2024-08-03T20:49:47.573315+0000","last_fullsized":"2024-08-03T20:49:47.573315+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:49:46.511193+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:49:46.511193+0000","last_clean_scrub_stamp":"2024-08-03T20:49:46.511193+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T07:04:32.609067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.2b","version":"0'0","reported_seq":10,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-08-03T20:49:47.569044+0000","last_change":"2024-08-03T20:49:47.563458+0000","last_active":"2024-08-03T20:49:47.569044+0000","last_peered":"2024-08-03T20:49:47.569044+0000","last_clean":"2024-08-03T20:49:47.569044+0000","last_became_active":"2024-08-03T20:49:47.563174+0000","last_became_peered":"2024-08-03T20:49:47.563174+0000","last_unstale":"2024-08-03T20:49:47.569044+0000","last_undegraded":"2024-08-03T20:49:47.569044+0000","last_fullsized":"2024-08-03T20:49:47.569044+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:49:46.511193+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:49:46.511193+0000","last_clean_scrub_stamp":"2024-08-03T20:49:46.511193+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T00:30:12.811578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.2a","version":"0'0","reported_seq":10,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-08-03T20:49:47.573962+0000","last_change":"2024-08-03T20:49:47.571634+0000","last_active":"2024-08-03T20:49:47.573962+0000","last_peered":"2024-08-03T20:49:47.573962+0000","last_clean":"2024-08-03T20:49:47.573962+0000","last_became_active":"2024-08-03T20:49:47.571270+0000","last_became_peered":"2024-08-03T20:49:47.571270+0000","last_unstale":"2024-08-03T20:49:47.573962+0000","last_undegraded":"2024-08-03T20:49:47.573962+0000","last_fullsized":"2024-08-03T20:49:47.573962+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:49:46.511193+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:49:46.511193+0000","last_clean_scrub_stamp":"2024-08-03T20:49:46.511193+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T05:16:35.717962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.29","version":"0'0","reported_seq":13,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-08-03T20:49:48.776933+0000","last_change":"2024-08-03T20:49:47.544241+0000","last_active":"2024-08-03T20:49:48.776933+0000","last_peered":"2024-08-03T20:49:48.776933+0000","last_clean":"2024-08-03T20:49:48.776933+0000","last_became_active":"2024-08-03T20:49:47.543885+0000","last_became_peered":"2024-08-03T20:49:47.543885+0000","last_unstale":"2024-08-03T20:49:48.776933+0000","last_undegraded":"2024-08-03T20:49:48.776933+0000","last_fullsized":"2024-08-03T20:49:48.776933+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:49:46.511193+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:49:46.511193+0000","last_clean_scrub_stamp":"2024-08-03T20:49:46.511193+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-04T22:38:27.858926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.28","version":"0'0","reported_seq":13,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-08-03T20:49:48.776843+0000","last_change":"2024-08-03T20:49:47.540376+0000","last_active":"2024-08-03T20:49:48.776843+0000","last_peered":"2024-08-03T20:49:48.776843+0000","last_clean":"2024-08-03T20:49:48.776843+0000","last_became_active":"2024-08-03T20:49:47.540075+0000","last_became_peered":"2024-08-03T20:49:47.540075+0000","last_unstale":"2024-08-03T20:49:48.776843+0000","last_undegraded":"2024-08-03T20:49:48.776843+0000","last_fullsized":"2024-08-03T20:49:48.776843+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:49:46.511193+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:49:46.511193+0000","last_clean_scrub_stamp":"2024-08-03T20:49:46.511193+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-04T22:39:10.883457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.27","version":"0'0","reported_seq":13,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-08-03T20:49:48.777115+0000","last_change":"2024-08-03T20:49:47.549654+0000","last_active":"2024-08-03T20:49:48.777115+0000","last_peered":"2024-08-03T20:49:48.777115+0000","last_clean":"2024-08-03T20:49:48.777115+0000","last_became_active":"2024-08-03T20:49:47.549359+0000","last_became_peered":"2024-08-03T20:49:47.549359+0000","last_unstale":"2024-08-03T20:49:48.777115+0000","last_undegraded":"2024-08-03T20:49:48.777115+0000","last_fullsized":"2024-08-03T20:49:48.777115+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:49:46.511193+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:49:46.511193+0000","last_clean_scrub_stamp":"2024-08-03T20:49:46.511193+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T01:56:27.053963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.26","version":"0'0","reported_seq":10,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-08-03T20:49:47.568027+0000","last_change":"2024-08-03T20:49:47.566895+0000","last_active":"2024-08-03T20:49:47.568027+0000","last_peered":"2024-08-03T20:49:47.568027+0000","last_clean":"2024-08-03T20:49:47.568027+0000","last_became_active":"2024-08-03T20:49:47.566511+0000","last_became_peered":"2024-08-03T20:49:47.566511+0000","last_unstale":"2024-08-03T20:49:47.568027+0000","last_undegraded":"2024-08-03T20:49:47.568027+0000","last_fullsized":"2024-08-03T20:49:47.568027+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:49:46.511193+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:49:46.511193+0000","last_clean_scrub_stamp":"2024-08-03T20:49:46.511193+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T02:07:30.765109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.25","version":"0'0","reported_seq":10,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-08-03T20:49:47.546952+0000","last_change":"2024-08-03T20:49:47.538651+0000","last_active":"2024-08-03T20:49:47.546952+0000","last_peered":"2024-08-03T20:49:47.546952+0000","last_clean":"2024-08-03T20:49:47.546952+0000","last_became_active":"2024-08-03T20:49:47.538347+0000","last_became_peered":"2024-08-03T20:49:47.538347+0000","last_unstale":"2024-08-03T20:49:47.546952+0000","last_undegraded":"2024-08-03T20:49:47.546952+0000","last_fullsized":"2024-08-03T20:49:47.546952+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:49:46.511193+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:49:46.511193+0000","last_clean_scrub_stamp":"2024-08-03T20:49:46.511193+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T02:10:33.849652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.24","version":"0'0","reported_seq":13,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-08-03T20:49:48.676651+0000","last_change":"2024-08-03T20:49:47.558978+0000","last_active":"2024-08-03T20:49:48.676651+0000","last_peered":"2024-08-03T20:49:48.676651+0000","last_clean":"2024-08-03T20:49:48.676651+0000","last_became_active":"2024-08-03T20:49:47.557136+0000","last_became_peered":"2024-08-03T20:49:47.557136+0000","last_unstale":"2024-08-03T20:49:48.676651+0000","last_undegraded":"2024-08-03T20:49:48.676651+0000","last_fullsized":"2024-08-03T20:49:48.676651+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:49:46.511193+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:49:46.511193+0000","last_clean_scrub_stamp":"2024-08-03T20:49:46.511193+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T04:45:12.792950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.23","version":"0'0","reported_seq":13,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-08-03T20:49:48.777235+0000","last_change":"2024-08-03T20:49:47.547275+0000","last_active":"2024-08-03T20:49:48.777235+0000","last_peered":"2024-08-03T20:49:48.777235+0000","last_clean":"2024-08-03T20:49:48.777235+0000","last_became_active":"2024-08-03T20:49:47.546941+0000","last_became_peered":"2024-08-03T20:49:47.546941+0000","last_unstale":"2024-08-03T20:49:48.777235+0000","last_undegraded":"2024-08-03T20:49:48.777235+0000","last_fullsized":"2024-08-03T20:49:48.777235+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:49:46.511193+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:49:46.511193+0000","last_clean_scrub_stamp":"2024-08-03T20:49:46.511193+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T05:16:35.717962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.22","version":"0'0","reported_seq":10,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-08-03T20:49:47.546243+0000","last_change":"2024-08-03T20:49:47.539492+0000","last_active":"2024-08-03T20:49:47.546243+0000","last_peered":"2024-08-03T20:49:47.546243+0000","last_clean":"2024-08-03T20:49:47.546243+0000","last_became_active":"2024-08-03T20:49:47.539177+0000","last_became_peered":"2024-08-03T20:49:47.539177+0000","last_unstale":"2024-08-03T20:49:47.546243+0000","last_undegraded":"2024-08-03T20:49:47.546243+0000","last_fullsized":"2024-08-03T20:49:47.546243+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:49:46.511193+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:49:46.511193+0000","last_clean_scrub_stamp":"2024-08-03T20:49:46.511193+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T00:58:45.729626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.21","version":"0'0","reported_seq":10,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-08-03T20:49:47.567109+0000","last_change":"2024-08-03T20:49:47.562467+0000","last_active":"2024-08-03T20:49:47.567109+0000","last_peered":"2024-08-03T20:49:47.567109+0000","last_clean":"2024-08-03T20:49:47.567109+0000","last_became_active":"2024-08-03T20:49:47.560186+0000","last_became_peered":"2024-08-03T20:49:47.560186+0000","last_unstale":"2024-08-03T20:49:47.567109+0000","last_undegraded":"2024-08-03T20:49:47.567109+0000","last_fullsized":"2024-08-03T20:49:47.567109+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:49:46.511193+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:49:46.511193+0000","last_clean_scrub_stamp":"2024-08-03T20:49:46.511193+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T08:29:32.556925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.20","version":"0'0","reported_seq":10,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-08-03T20:49:47.553756+0000","last_change":"2024-08-03T20:49:47.546628+0000","last_active":"2024-08-03T20:49:47.553756+0000","last_peered":"2024-08-03T20:49:47.553756+0000","last_clean":"2024-08-03T20:49:47.553756+0000","last_became_active":"2024-08-03T20:49:47.546313+0000","last_became_peered":"2024-08-03T20:49:47.546313+0000","last_unstale":"2024-08-03T20:49:47.553756+0000","last_undegraded":"2024-08-03T20:49:47.553756+0000","last_fullsized":"2024-08-03T20:49:47.553756+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:49:46.511193+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:49:46.511193+0000","last_clean_scrub_stamp":"2024-08-03T20:49:46.511193+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-04T22:41:19.703398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.1e","version":"0'0","reported_seq":13,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-08-03T20:49:48.698883+0000","last_change":"2024-08-03T20:49:47.565194+0000","last_active":"2024-08-03T20:49:48.698883+0000","last_peered":"2024-08-03T20:49:48.698883+0000","last_clean":"2024-08-03T20:49:48.698883+0000","last_became_active":"2024-08-03T20:49:47.564861+0000","last_became_peered":"2024-08-03T20:49:47.564861+0000","last_unstale":"2024-08-03T20:49:48.698883+0000","last_undegraded":"2024-08-03T20:49:48.698883+0000","last_fullsized":"2024-08-03T20:49:48.698883+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:49:46.511193+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:49:46.511193+0000","last_clean_scrub_stamp":"2024-08-03T20:49:46.511193+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T08:12:22.699786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"44'4","reported_seq":17,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-08-03T20:49:47.569217+0000","last_change":"2024-08-03T20:49:46.545203+0000","last_active":"2024-08-03T20:49:47.569217+0000","last_peered":"2024-08-03T20:49:47.569217+0000","last_clean":"2024-08-03T20:49:47.569217+0000","last_became_active":"2024-08-03T20:49:46.544757+0000","last_became_peered":"2024-08-03T20:49:46.544757+0000","last_unstale":"2024-08-03T20:49:47.569217+0000","last_undegraded":"2024-08-03T20:49:47.569217+0000","last_fullsized":"2024-08-03T20:49:47.569217+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:49:45.503385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:49:45.503385+0000","last_clean_scrub_stamp":"2024-08-03T20:49:45.503385+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-04T21:15:34.746119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1f","version":"0'0","reported_seq":13,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-08-03T20:49:48.777218+0000","last_change":"2024-08-03T20:49:47.536235+0000","last_active":"2024-08-03T20:49:48.777218+0000","last_peered":"2024-08-03T20:49:48.777218+0000","last_clean":"2024-08-03T20:49:48.777218+0000","last_became_active":"2024-08-03T20:49:47.535885+0000","last_became_peered":"2024-08-03T20:49:47.535885+0000","last_unstale":"2024-08-03T20:49:48.777218+0000","last_undegraded":"2024-08-03T20:49:48.777218+0000","last_fullsized":"2024-08-03T20:49:48.777218+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:49:46.511193+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:49:46.511193+0000","last_clean_scrub_stamp":"2024-08-03T20:49:46.511193+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-04T21:25:36.997036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1e","version":"44'1","reported_seq":14,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-08-03T20:49:47.555705+0000","last_change":"2024-08-03T20:49:46.535877+0000","last_active":"2024-08-03T20:49:47.555705+0000","last_peered":"2024-08-03T20:49:47.555705+0000","last_clean":"2024-08-03T20:49:47.555705+0000","last_became_active":"2024-08-03T20:49:46.535339+0000","last_became_peered":"2024-08-03T20:49:46.535339+0000","last_unstale":"2024-08-03T20:49:47.555705+0000","last_undegraded":"2024-08-03T20:49:47.555705+0000","last_fullsized":"2024-08-03T20:49:47.555705+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:49:45.503385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:49:45.503385+0000","last_clean_scrub_stamp":"2024-08-03T20:49:45.503385+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T04:47:42.666138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.1c","version":"0'0","reported_seq":13,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-08-03T20:49:48.676753+0000","last_change":"2024-08-03T20:49:47.564973+0000","last_active":"2024-08-03T20:49:48.676753+0000","last_peered":"2024-08-03T20:49:48.676753+0000","last_clean":"2024-08-03T20:49:48.676753+0000","last_became_active":"2024-08-03T20:49:47.564637+0000","last_became_peered":"2024-08-03T20:49:47.564637+0000","last_unstale":"2024-08-03T20:49:48.676753+0000","last_undegraded":"2024-08-03T20:49:48.676753+0000","last_fullsized":"2024-08-03T20:49:48.676753+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:49:46.511193+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:49:46.511193+0000","last_clean_scrub_stamp":"2024-08-03T20:49:46.511193+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T06:34:48.402529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1d","version":"44'2","reported_seq":17,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-08-03T20:49:48.698893+0000","last_change":"2024-08-03T20:49:46.543320+0000","last_active":"2024-08-03T20:49:48.698893+0000","last_peered":"2024-08-03T20:49:48.698893+0000","last_clean":"2024-08-03T20:49:48.698893+0000","last_became_active":"2024-08-03T20:49:46.543004+0000","last_became_peered":"2024-08-03T20:49:46.543004+0000","last_unstale":"2024-08-03T20:49:48.698893+0000","last_undegraded":"2024-08-03T20:49:48.698893+0000","last_fullsized":"2024-08-03T20:49:48.698893+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:49:45.503385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:49:45.503385+0000","last_clean_scrub_stamp":"2024-08-03T20:49:45.503385+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-04T22:39:09.875648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.1d","version":"0'0","reported_seq":13,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-08-03T20:49:48.698928+0000","last_change":"2024-08-03T20:49:47.543428+0000","last_active":"2024-08-03T20:49:48.698928+0000","last_peered":"2024-08-03T20:49:48.698928+0000","last_clean":"2024-08-03T20:49:48.698928+0000","last_became_active":"2024-08-03T20:49:47.542839+0000","last_became_peered":"2024-08-03T20:49:47.542839+0000","last_unstale":"2024-08-03T20:49:48.698928+0000","last_undegraded":"2024-08-03T20:49:48.698928+0000","last_fullsized":"2024-08-03T20:49:48.698928+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:49:46.511193+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:49:46.511193+0000","last_clean_scrub_stamp":"2024-08-03T20:49:46.511193+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-04T21:25:36.997036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1c","version":"44'2","reported_seq":17,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-08-03T20:49:48.540875+0000","last_change":"2024-08-03T20:49:46.547041+0000","last_active":"2024-08-03T20:49:48.540875+0000","last_peered":"2024-08-03T20:49:48.540875+0000","last_clean":"2024-08-03T20:49:48.540875+0000","last_became_active":"2024-08-03T20:49:46.545518+0000","last_became_peered":"2024-08-03T20:49:46.545518+0000","last_unstale":"2024-08-03T20:49:48.540875+0000","last_undegraded":"2024-08-03T20:49:48.540875+0000","last_fullsized":"2024-08-03T20:49:48.540875+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:49:45.503385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:49:45.503385+0000","last_clean_scrub_stamp":"2024-08-03T20:49:45.503385+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T02:46:24.186241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1a","version":"0'0","reported_seq":10,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-08-03T20:49:47.542945+0000","last_change":"2024-08-03T20:49:47.537376+0000","last_active":"2024-08-03T20:49:47.542945+0000","last_peered":"2024-08-03T20:49:47.542945+0000","last_clean":"2024-08-03T20:49:47.542945+0000","last_became_active":"2024-08-03T20:49:47.537104+0000","last_became_peered":"2024-08-03T20:49:47.537104+0000","last_unstale":"2024-08-03T20:49:47.542945+0000","last_undegraded":"2024-08-03T20:49:47.542945+0000","last_fullsized":"2024-08-03T20:49:47.542945+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:49:46.511193+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:49:46.511193+0000","last_clean_scrub_stamp":"2024-08-03T20:49:46.511193+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T05:26:16.037808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"44'1","reported_seq":14,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-08-03T20:49:47.563445+0000","last_change":"2024-08-03T20:49:46.540951+0000","last_active":"2024-08-03T20:49:47.563445+0000","last_peered":"2024-08-03T20:49:47.563445+0000","last_clean":"2024-08-03T20:49:47.563445+0000","last_became_active":"2024-08-03T20:49:46.540594+0000","last_became_peered":"2024-08-03T20:49:46.540594+0000","last_unstale":"2024-08-03T20:49:47.563445+0000","last_undegraded":"2024-08-03T20:49:47.563445+0000","last_fullsized":"2024-08-03T20:49:47.563445+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:49:45.503385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:49:45.503385+0000","last_clean_scrub_stamp":"2024-08-03T20:49:45.503385+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T01:13:46.007414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1b","version":"0'0","reported_seq":10,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-08-03T20:49:47.572449+0000","last_change":"2024-08-03T20:49:47.568859+0000","last_active":"2024-08-03T20:49:47.572449+0000","last_peered":"2024-08-03T20:49:47.572449+0000","last_clean":"2024-08-03T20:49:47.572449+0000","last_became_active":"2024-08-03T20:49:47.568458+0000","last_became_peered":"2024-08-03T20:49:47.568458+0000","last_unstale":"2024-08-03T20:49:47.572449+0000","last_undegraded":"2024-08-03T20:49:47.572449+0000","last_fullsized":"2024-08-03T20:49:47.572449+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:49:46.511193+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:49:46.511193+0000","last_clean_scrub_stamp":"2024-08-03T20:49:46.511193+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T08:12:22.699786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1a","version":"44'1","reported_seq":14,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-08-03T20:49:47.553385+0000","last_change":"2024-08-03T20:49:46.539598+0000","last_active":"2024-08-03T20:49:47.553385+0000","last_peered":"2024-08-03T20:49:47.553385+0000","last_clean":"2024-08-03T20:49:47.553385+0000","last_became_active":"2024-08-03T20:49:46.539244+0000","last_became_peered":"2024-08-03T20:49:46.539244+0000","last_unstale":"2024-08-03T20:49:47.553385+0000","last_undegraded":"2024-08-03T20:49:47.553385+0000","last_fullsized":"2024-08-03T20:49:47.553385+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:49:45.503385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:49:45.503385+0000","last_clean_scrub_stamp":"2024-08-03T20:49:45.503385+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T05:40:57.056135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.18","version":"0'0","reported_seq":10,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-08-03T20:49:47.568183+0000","last_change":"2024-08-03T20:49:47.542493+0000","last_active":"2024-08-03T20:49:47.568183+0000","last_peered":"2024-08-03T20:49:47.568183+0000","last_clean":"2024-08-03T20:49:47.568183+0000","last_became_active":"2024-08-03T20:49:47.542128+0000","last_became_peered":"2024-08-03T20:49:47.542128+0000","last_unstale":"2024-08-03T20:49:47.568183+0000","last_undegraded":"2024-08-03T20:49:47.568183+0000","last_fullsized":"2024-08-03T20:49:47.568183+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:49:46.511193+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:49:46.511193+0000","last_clean_scrub_stamp":"2024-08-03T20:49:46.511193+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-04T22:42:14.349430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":15,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-08-03T20:49:48.676204+0000","last_change":"2024-08-03T20:49:46.550316+0000","last_active":"2024-08-03T20:49:48.676204+0000","last_peered":"2024-08-03T20:49:48.676204+0000","last_clean":"2024-08-03T20:49:48.676204+0000","last_became_active":"2024-08-03T20:49:46.549958+0000","last_became_peered":"2024-08-03T20:49:46.549958+0000","last_unstale":"2024-08-03T20:49:48.676204+0000","last_undegraded":"2024-08-03T20:49:48.676204+0000","last_fullsized":"2024-08-03T20:49:48.676204+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:49:45.503385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:49:45.503385+0000","last_clean_scrub_stamp":"2024-08-03T20:49:45.503385+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T05:16:34.710154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.19","version":"0'0","reported_seq":13,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-08-03T20:49:48.776813+0000","last_change":"2024-08-03T20:49:47.541945+0000","last_active":"2024-08-03T20:49:48.776813+0000","last_peered":"2024-08-03T20:49:48.776813+0000","last_clean":"2024-08-03T20:49:48.776813+0000","last_became_active":"2024-08-03T20:49:47.541629+0000","last_became_peered":"2024-08-03T20:49:47.541629+0000","last_unstale":"2024-08-03T20:49:48.776813+0000","last_undegraded":"2024-08-03T20:49:48.776813+0000","last_fullsized":"2024-08-03T20:49:48.776813+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:49:46.511193+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:49:46.511193+0000","last_clean_scrub_stamp":"2024-08-03T20:49:46.511193+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-04T22:42:14.349430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":13,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-08-03T20:49:47.525192+0000","last_change":"2024-08-03T20:49:46.548247+0000","last_active":"2024-08-03T20:49:47.525192+0000","last_peered":"2024-08-03T20:49:47.525192+0000","last_clean":"2024-08-03T20:49:47.525192+0000","last_became_active":"2024-08-03T20:49:46.547175+0000","last_became_peered":"2024-08-03T20:49:46.547175+0000","last_unstale":"2024-08-03T20:49:47.525192+0000","last_undegraded":"2024-08-03T20:49:47.525192+0000","last_fullsized":"2024-08-03T20:49:47.525192+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:49:45.503385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:49:45.503385+0000","last_clean_scrub_stamp":"2024-08-03T20:49:45.503385+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T05:18:55.568493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.16","version":"0'0","reported_seq":10,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-08-03T20:49:47.573139+0000","last_change":"2024-08-03T20:49:47.572228+0000","last_active":"2024-08-03T20:49:47.573139+0000","last_peered":"2024-08-03T20:49:47.573139+0000","last_clean":"2024-08-03T20:49:47.573139+0000","last_became_active":"2024-08-03T20:49:47.571868+0000","last_became_peered":"2024-08-03T20:49:47.571868+0000","last_unstale":"2024-08-03T20:49:47.573139+0000","last_undegraded":"2024-08-03T20:49:47.573139+0000","last_fullsized":"2024-08-03T20:49:47.573139+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:49:46.511193+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:49:46.511193+0000","last_clean_scrub_stamp":"2024-08-03T20:49:46.511193+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T03:10:36.106156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":13,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-08-03T20:49:47.525770+0000","last_change":"2024-08-03T20:49:46.537616+0000","last_active":"2024-08-03T20:49:47.525770+0000","last_peered":"2024-08-03T20:49:47.525770+0000","last_clean":"2024-08-03T20:49:47.525770+0000","last_became_active":"2024-08-03T20:49:46.536514+0000","last_became_peered":"2024-08-03T20:49:46.536514+0000","last_unstale":"2024-08-03T20:49:47.525770+0000","last_undegraded":"2024-08-03T20:49:47.525770+0000","last_fullsized":"2024-08-03T20:49:47.525770+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:49:45.503385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:49:45.503385+0000","last_clean_scrub_stamp":"2024-08-03T20:49:45.503385+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T06:59:07.982736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.17","version":"0'0","reported_seq":13,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-08-03T20:49:48.541141+0000","last_change":"2024-08-03T20:49:47.545330+0000","last_active":"2024-08-03T20:49:48.541141+0000","last_peered":"2024-08-03T20:49:48.541141+0000","last_clean":"2024-08-03T20:49:48.541141+0000","last_became_active":"2024-08-03T20:49:47.544609+0000","last_became_peered":"2024-08-03T20:49:47.544609+0000","last_unstale":"2024-08-03T20:49:48.541141+0000","last_undegraded":"2024-08-03T20:49:48.541141+0000","last_fullsized":"2024-08-03T20:49:48.541141+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:49:46.511193+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:49:46.511193+0000","last_clean_scrub_stamp":"2024-08-03T20:49:46.511193+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-04T23:10:50.091040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":13,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-08-03T20:49:47.527977+0000","last_change":"2024-08-03T20:49:46.550770+0000","last_active":"2024-08-03T20:49:47.527977+0000","last_peered":"2024-08-03T20:49:47.527977+0000","last_clean":"2024-08-03T20:49:47.527977+0000","last_became_active":"2024-08-03T20:49:46.550435+0000","last_became_peered":"2024-08-03T20:49:46.550435+0000","last_unstale":"2024-08-03T20:49:47.527977+0000","last_undegraded":"2024-08-03T20:49:47.527977+0000","last_fullsized":"2024-08-03T20:49:47.527977+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:49:45.503385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:49:45.503385+0000","last_clean_scrub_stamp":"2024-08-03T20:49:45.503385+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T06:23:00.389684+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.14","version":"0'0","reported_seq":10,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-08-03T20:49:47.566402+0000","last_change":"2024-08-03T20:49:47.561607+0000","last_active":"2024-08-03T20:49:47.566402+0000","last_peered":"2024-08-03T20:49:47.566402+0000","last_clean":"2024-08-03T20:49:47.566402+0000","last_became_active":"2024-08-03T20:49:47.558386+0000","last_became_peered":"2024-08-03T20:49:47.558386+0000","last_unstale":"2024-08-03T20:49:47.566402+0000","last_undegraded":"2024-08-03T20:49:47.566402+0000","last_fullsized":"2024-08-03T20:49:47.566402+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:49:46.511193+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:49:46.511193+0000","last_clean_scrub_stamp":"2024-08-03T20:49:46.511193+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T03:04:54.847476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"44'3","reported_seq":18,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-08-03T20:49:48.541038+0000","last_change":"2024-08-03T20:49:46.544026+0000","last_active":"2024-08-03T20:49:48.541038+0000","last_peered":"2024-08-03T20:49:48.541038+0000","last_clean":"2024-08-03T20:49:48.541038+0000","last_became_active":"2024-08-03T20:49:46.543652+0000","last_became_peered":"2024-08-03T20:49:46.543652+0000","last_unstale":"2024-08-03T20:49:48.541038+0000","last_undegraded":"2024-08-03T20:49:48.541038+0000","last_fullsized":"2024-08-03T20:49:48.541038+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:49:45.503385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:49:45.503385+0000","last_clean_scrub_stamp":"2024-08-03T20:49:45.503385+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T02:10:58.372826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.15","version":"0'0","reported_seq":13,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-08-03T20:49:48.540907+0000","last_change":"2024-08-03T20:49:47.545128+0000","last_active":"2024-08-03T20:49:48.540907+0000","last_peered":"2024-08-03T20:49:48.540907+0000","last_clean":"2024-08-03T20:49:48.540907+0000","last_became_active":"2024-08-03T20:49:47.544775+0000","last_became_peered":"2024-08-03T20:49:47.544775+0000","last_unstale":"2024-08-03T20:49:48.540907+0000","last_undegraded":"2024-08-03T20:49:48.540907+0000","last_fullsized":"2024-08-03T20:49:48.540907+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:49:46.511193+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:49:46.511193+0000","last_clean_scrub_stamp":"2024-08-03T20:49:46.511193+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T06:06:10.660971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.14","version":"44'6","reported_seq":19,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-08-03T20:49:47.570220+0000","last_change":"2024-08-03T20:49:46.536098+0000","last_active":"2024-08-03T20:49:47.570220+0000","last_peered":"2024-08-03T20:49:47.570220+0000","last_clean":"2024-08-03T20:49:47.570220+0000","last_became_active":"2024-08-03T20:49:46.535713+0000","last_became_peered":"2024-08-03T20:49:46.535713+0000","last_unstale":"2024-08-03T20:49:47.570220+0000","last_undegraded":"2024-08-03T20:49:47.570220+0000","last_fullsized":"2024-08-03T20:49:47.570220+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:49:45.503385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:49:45.503385+0000","last_clean_scrub_stamp":"2024-08-03T20:49:45.503385+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-04T22:38:26.098658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.12","version":"0'0","reported_seq":13,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-08-03T20:49:48.777153+0000","last_change":"2024-08-03T20:49:47.571616+0000","last_active":"2024-08-03T20:49:48.777153+0000","last_peered":"2024-08-03T20:49:48.777153+0000","last_clean":"2024-08-03T20:49:48.777153+0000","last_became_active":"2024-08-03T20:49:47.571180+0000","last_became_peered":"2024-08-03T20:49:47.571180+0000","last_unstale":"2024-08-03T20:49:48.777153+0000","last_undegraded":"2024-08-03T20:49:48.777153+0000","last_fullsized":"2024-08-03T20:49:48.777153+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:49:46.511193+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:49:46.511193+0000","last_clean_scrub_stamp":"2024-08-03T20:49:46.511193+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T08:12:22.699786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.13","version":"44'2","reported_seq":15,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-08-03T20:49:47.573622+0000","last_change":"2024-08-03T20:49:46.545991+0000","last_active":"2024-08-03T20:49:47.573622+0000","last_peered":"2024-08-03T20:49:47.573622+0000","last_clean":"2024-08-03T20:49:47.573622+0000","last_became_active":"2024-08-03T20:49:46.545520+0000","last_became_peered":"2024-08-03T20:49:46.545520+0000","last_unstale":"2024-08-03T20:49:47.573622+0000","last_undegraded":"2024-08-03T20:49:47.573622+0000","last_fullsized":"2024-08-03T20:49:47.573622+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:49:45.503385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:49:45.503385+0000","last_clean_scrub_stamp":"2024-08-03T20:49:45.503385+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-04T23:37:23.743502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.13","version":"0'0","reported_seq":10,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-08-03T20:49:47.572981+0000","last_change":"2024-08-03T20:49:47.570446+0000","last_active":"2024-08-03T20:49:47.572981+0000","last_peered":"2024-08-03T20:49:47.572981+0000","last_clean":"2024-08-03T20:49:47.572981+0000","last_became_active":"2024-08-03T20:49:47.568679+0000","last_became_peered":"2024-08-03T20:49:47.568679+0000","last_unstale":"2024-08-03T20:49:47.572981+0000","last_undegraded":"2024-08-03T20:49:47.572981+0000","last_fullsized":"2024-08-03T20:49:47.572981+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:49:46.511193+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:49:46.511193+0000","last_clean_scrub_stamp":"2024-08-03T20:49:46.511193+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-04T21:25:36.997036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":15,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-08-03T20:49:48.540793+0000","last_change":"2024-08-03T20:49:46.548199+0000","last_active":"2024-08-03T20:49:48.540793+0000","last_peered":"2024-08-03T20:49:48.540793+0000","last_clean":"2024-08-03T20:49:48.540793+0000","last_became_active":"2024-08-03T20:49:46.546021+0000","last_became_peered":"2024-08-03T20:49:46.546021+0000","last_unstale":"2024-08-03T20:49:48.540793+0000","last_undegraded":"2024-08-03T20:49:48.540793+0000","last_fullsized":"2024-08-03T20:49:48.540793+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:49:45.503385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:49:45.503385+0000","last_clean_scrub_stamp":"2024-08-03T20:49:45.503385+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-04T23:31:11.860305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.10","version":"0'0","reported_seq":10,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-08-03T20:49:47.565149+0000","last_change":"2024-08-03T20:49:47.555578+0000","last_active":"2024-08-03T20:49:47.565149+0000","last_peered":"2024-08-03T20:49:47.565149+0000","last_clean":"2024-08-03T20:49:47.565149+0000","last_became_active":"2024-08-03T20:49:47.555238+0000","last_became_peered":"2024-08-03T20:49:47.555238+0000","last_unstale":"2024-08-03T20:49:47.565149+0000","last_undegraded":"2024-08-03T20:49:47.565149+0000","last_fullsized":"2024-08-03T20:49:47.565149+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:49:46.511193+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:49:46.511193+0000","last_clean_scrub_stamp":"2024-08-03T20:49:46.511193+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T00:22:52.850737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":13,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-08-03T20:49:47.541450+0000","last_change":"2024-08-03T20:49:46.546071+0000","last_active":"2024-08-03T20:49:47.541450+0000","last_peered":"2024-08-03T20:49:47.541450+0000","last_clean":"2024-08-03T20:49:47.541450+0000","last_became_active":"2024-08-03T20:49:46.540634+0000","last_became_peered":"2024-08-03T20:49:46.540634+0000","last_unstale":"2024-08-03T20:49:47.541450+0000","last_undegraded":"2024-08-03T20:49:47.541450+0000","last_fullsized":"2024-08-03T20:49:47.541450+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:49:45.503385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:49:45.503385+0000","last_clean_scrub_stamp":"2024-08-03T20:49:45.503385+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T00:57:57.111997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.11","version":"0'0","reported_seq":13,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-08-03T20:49:48.541095+0000","last_change":"2024-08-03T20:49:47.546294+0000","last_active":"2024-08-03T20:49:48.541095+0000","last_peered":"2024-08-03T20:49:48.541095+0000","last_clean":"2024-08-03T20:49:48.541095+0000","last_became_active":"2024-08-03T20:49:47.545526+0000","last_became_peered":"2024-08-03T20:49:47.545526+0000","last_unstale":"2024-08-03T20:49:48.541095+0000","last_undegraded":"2024-08-03T20:49:48.541095+0000","last_fullsized":"2024-08-03T20:49:48.541095+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:49:46.511193+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:49:46.511193+0000","last_clean_scrub_stamp":"2024-08-03T20:49:46.511193+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T04:36:12.061926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.10","version":"44'8","reported_seq":21,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-08-03T20:49:47.573969+0000","last_change":"2024-08-03T20:49:46.536489+0000","last_active":"2024-08-03T20:49:47.573969+0000","last_peered":"2024-08-03T20:49:47.573969+0000","last_clean":"2024-08-03T20:49:47.573969+0000","last_became_active":"2024-08-03T20:49:46.536149+0000","last_became_peered":"2024-08-03T20:49:46.536149+0000","last_unstale":"2024-08-03T20:49:47.573969+0000","last_undegraded":"2024-08-03T20:49:47.573969+0000","last_fullsized":"2024-08-03T20:49:47.573969+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:49:45.503385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:49:45.503385+0000","last_clean_scrub_stamp":"2024-08-03T20:49:45.503385+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-04T22:10:43.371385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.e","version":"0'0","reported_seq":10,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-08-03T20:49:47.555877+0000","last_change":"2024-08-03T20:49:47.552334+0000","last_active":"2024-08-03T20:49:47.555877+0000","last_peered":"2024-08-03T20:49:47.555877+0000","last_clean":"2024-08-03T20:49:47.555877+0000","last_became_active":"2024-08-03T20:49:47.551481+0000","last_became_peered":"2024-08-03T20:49:47.551481+0000","last_unstale":"2024-08-03T20:49:47.555877+0000","last_undegraded":"2024-08-03T20:49:47.555877+0000","last_fullsized":"2024-08-03T20:49:47.555877+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:49:46.511193+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:49:46.511193+0000","last_clean_scrub_stamp":"2024-08-03T20:49:46.511193+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T01:34:47.270126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.f","version":"44'1","reported_seq":14,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-08-03T20:49:47.562672+0000","last_change":"2024-08-03T20:49:46.544181+0000","last_active":"2024-08-03T20:49:47.562672+0000","last_peered":"2024-08-03T20:49:47.562672+0000","last_clean":"2024-08-03T20:49:47.562672+0000","last_became_active":"2024-08-03T20:49:46.543855+0000","last_became_peered":"2024-08-03T20:49:46.543855+0000","last_unstale":"2024-08-03T20:49:47.562672+0000","last_undegraded":"2024-08-03T20:49:47.562672+0000","last_fullsized":"2024-08-03T20:49:47.562672+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:49:45.503385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:49:45.503385+0000","last_clean_scrub_stamp":"2024-08-03T20:49:45.503385+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T07:09:44.232554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.f","version":"0'0","reported_seq":10,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-08-03T20:49:47.576497+0000","last_change":"2024-08-03T20:49:47.574712+0000","last_active":"2024-08-03T20:49:47.576497+0000","last_peered":"2024-08-03T20:49:47.576497+0000","last_clean":"2024-08-03T20:49:47.576497+0000","last_became_active":"2024-08-03T20:49:47.574405+0000","last_became_peered":"2024-08-03T20:49:47.574405+0000","last_unstale":"2024-08-03T20:49:47.576497+0000","last_undegraded":"2024-08-03T20:49:47.576497+0000","last_fullsized":"2024-08-03T20:49:47.576497+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:49:46.511193+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:49:46.511193+0000","last_clean_scrub_stamp":"2024-08-03T20:49:46.511193+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T00:00:41.907040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.e","version":"44'2","reported_seq":15,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-08-03T20:49:47.558817+0000","last_change":"2024-08-03T20:49:46.545658+0000","last_active":"2024-08-03T20:49:47.558817+0000","last_peered":"2024-08-03T20:49:47.558817+0000","last_clean":"2024-08-03T20:49:47.558817+0000","last_became_active":"2024-08-03T20:49:46.545308+0000","last_became_peered":"2024-08-03T20:49:46.545308+0000","last_unstale":"2024-08-03T20:49:47.558817+0000","last_undegraded":"2024-08-03T20:49:47.558817+0000","last_fullsized":"2024-08-03T20:49:47.558817+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:49:45.503385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:49:45.503385+0000","last_clean_scrub_stamp":"2024-08-03T20:49:45.503385+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T04:58:01.002374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.c","version":"0'0","reported_seq":13,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-08-03T20:49:48.776972+0000","last_change":"2024-08-03T20:49:47.564300+0000","last_active":"2024-08-03T20:49:48.776972+0000","last_peered":"2024-08-03T20:49:48.776972+0000","last_clean":"2024-08-03T20:49:48.776972+0000","last_became_active":"2024-08-03T20:49:47.563994+0000","last_became_peered":"2024-08-03T20:49:47.563994+0000","last_unstale":"2024-08-03T20:49:48.776972+0000","last_undegraded":"2024-08-03T20:49:48.776972+0000","last_fullsized":"2024-08-03T20:49:48.776972+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:49:46.511193+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:49:46.511193+0000","last_clean_scrub_stamp":"2024-08-03T20:49:46.511193+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T05:40:42.992873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.d","version":"44'1","reported_seq":16,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-08-03T20:49:48.676597+0000","last_change":"2024-08-03T20:49:46.549197+0000","last_active":"2024-08-03T20:49:48.676597+0000","last_peered":"2024-08-03T20:49:48.676597+0000","last_clean":"2024-08-03T20:49:48.676597+0000","last_became_active":"2024-08-03T20:49:46.548326+0000","last_became_peered":"2024-08-03T20:49:46.548326+0000","last_unstale":"2024-08-03T20:49:48.676597+0000","last_undegraded":"2024-08-03T20:49:48.676597+0000","last_fullsized":"2024-08-03T20:49:48.676597+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:49:45.503385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:49:45.503385+0000","last_clean_scrub_stamp":"2024-08-03T20:49:45.503385+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T08:04:29.947302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.d","version":"0'0","reported_seq":13,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-08-03T20:49:48.777040+0000","last_change":"2024-08-03T20:49:47.547432+0000","last_active":"2024-08-03T20:49:48.777040+0000","last_peered":"2024-08-03T20:49:48.777040+0000","last_clean":"2024-08-03T20:49:48.777040+0000","last_became_active":"2024-08-03T20:49:47.545994+0000","last_became_peered":"2024-08-03T20:49:47.545994+0000","last_unstale":"2024-08-03T20:49:48.777040+0000","last_undegraded":"2024-08-03T20:49:48.777040+0000","last_fullsized":"2024-08-03T20:49:48.777040+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:49:46.511193+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:49:46.511193+0000","last_clean_scrub_stamp":"2024-08-03T20:49:46.511193+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T08:00:44.546502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":13,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-08-03T20:49:47.541771+0000","last_change":"2024-08-03T20:49:46.545204+0000","last_active":"2024-08-03T20:49:47.541771+0000","last_peered":"2024-08-03T20:49:47.541771+0000","last_clean":"2024-08-03T20:49:47.541771+0000","last_became_active":"2024-08-03T20:49:46.544838+0000","last_became_peered":"2024-08-03T20:49:46.544838+0000","last_unstale":"2024-08-03T20:49:47.541771+0000","last_undegraded":"2024-08-03T20:49:47.541771+0000","last_fullsized":"2024-08-03T20:49:47.541771+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:49:45.503385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:49:45.503385+0000","last_clean_scrub_stamp":"2024-08-03T20:49:45.503385+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T04:52:58.067545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.a","version":"0'0","reported_seq":10,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-08-03T20:49:47.566237+0000","last_change":"2024-08-03T20:49:47.559214+0000","last_active":"2024-08-03T20:49:47.566237+0000","last_peered":"2024-08-03T20:49:47.566237+0000","last_clean":"2024-08-03T20:49:47.566237+0000","last_became_active":"2024-08-03T20:49:47.558851+0000","last_became_peered":"2024-08-03T20:49:47.558851+0000","last_unstale":"2024-08-03T20:49:47.566237+0000","last_undegraded":"2024-08-03T20:49:47.566237+0000","last_fullsized":"2024-08-03T20:49:47.566237+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:49:46.511193+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:49:46.511193+0000","last_clean_scrub_stamp":"2024-08-03T20:49:46.511193+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T07:48:58.186945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.b","version":"44'2","reported_seq":15,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-08-03T20:49:47.563524+0000","last_change":"2024-08-03T20:49:46.541257+0000","last_active":"2024-08-03T20:49:47.563524+0000","last_peered":"2024-08-03T20:49:47.563524+0000","last_clean":"2024-08-03T20:49:47.563524+0000","last_became_active":"2024-08-03T20:49:46.540935+0000","last_became_peered":"2024-08-03T20:49:46.540935+0000","last_unstale":"2024-08-03T20:49:47.563524+0000","last_undegraded":"2024-08-03T20:49:47.563524+0000","last_fullsized":"2024-08-03T20:49:47.563524+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:49:45.503385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:49:45.503385+0000","last_clean_scrub_stamp":"2024-08-03T20:49:45.503385+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-04T21:39:34.699786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.b","version":"0'0","reported_seq":13,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-08-03T20:49:48.698708+0000","last_change":"2024-08-03T20:49:47.547916+0000","last_active":"2024-08-03T20:49:48.698708+0000","last_peered":"2024-08-03T20:49:48.698708+0000","last_clean":"2024-08-03T20:49:48.698708+0000","last_became_active":"2024-08-03T20:49:47.547342+0000","last_became_peered":"2024-08-03T20:49:47.547342+0000","last_unstale":"2024-08-03T20:49:48.698708+0000","last_undegraded":"2024-08-03T20:49:48.698708+0000","last_fullsized":"2024-08-03T20:49:48.698708+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:49:46.511193+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:49:46.511193+0000","last_clean_scrub_stamp":"2024-08-03T20:49:46.511193+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T05:07:25.106847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":15,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-08-03T20:49:48.676223+0000","last_change":"2024-08-03T20:49:46.542345+0000","last_active":"2024-08-03T20:49:48.676223+0000","last_peered":"2024-08-03T20:49:48.676223+0000","last_clean":"2024-08-03T20:49:48.676223+0000","last_became_active":"2024-08-03T20:49:46.542016+0000","last_became_peered":"2024-08-03T20:49:46.542016+0000","last_unstale":"2024-08-03T20:49:48.676223+0000","last_undegraded":"2024-08-03T20:49:48.676223+0000","last_fullsized":"2024-08-03T20:49:48.676223+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:49:45.503385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:49:45.503385+0000","last_clean_scrub_stamp":"2024-08-03T20:49:45.503385+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T00:56:58.559348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.8","version":"0'0","reported_seq":10,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-08-03T20:49:47.553782+0000","last_change":"2024-08-03T20:49:47.549436+0000","last_active":"2024-08-03T20:49:47.553782+0000","last_peered":"2024-08-03T20:49:47.553782+0000","last_clean":"2024-08-03T20:49:47.553782+0000","last_became_active":"2024-08-03T20:49:47.548669+0000","last_became_peered":"2024-08-03T20:49:47.548669+0000","last_unstale":"2024-08-03T20:49:47.553782+0000","last_undegraded":"2024-08-03T20:49:47.553782+0000","last_fullsized":"2024-08-03T20:49:47.553782+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:49:46.511193+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:49:46.511193+0000","last_clean_scrub_stamp":"2024-08-03T20:49:46.511193+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-04T23:30:31.178961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":13,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-08-03T20:49:47.525630+0000","last_change":"2024-08-03T20:49:46.551559+0000","last_active":"2024-08-03T20:49:47.525630+0000","last_peered":"2024-08-03T20:49:47.525630+0000","last_clean":"2024-08-03T20:49:47.525630+0000","last_became_active":"2024-08-03T20:49:46.551160+0000","last_became_peered":"2024-08-03T20:49:46.551160+0000","last_unstale":"2024-08-03T20:49:47.525630+0000","last_undegraded":"2024-08-03T20:49:47.525630+0000","last_fullsized":"2024-08-03T20:49:47.525630+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:49:45.503385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:49:45.503385+0000","last_clean_scrub_stamp":"2024-08-03T20:49:45.503385+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T01:25:47.191916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.9","version":"0'0","reported_seq":10,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-08-03T20:49:47.567965+0000","last_change":"2024-08-03T20:49:47.551290+0000","last_active":"2024-08-03T20:49:47.567965+0000","last_peered":"2024-08-03T20:49:47.567965+0000","last_clean":"2024-08-03T20:49:47.567965+0000","last_became_active":"2024-08-03T20:49:47.550992+0000","last_became_peered":"2024-08-03T20:49:47.550992+0000","last_unstale":"2024-08-03T20:49:47.567965+0000","last_undegraded":"2024-08-03T20:49:47.567965+0000","last_fullsized":"2024-08-03T20:49:47.567965+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:49:46.511193+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:49:46.511193+0000","last_clean_scrub_stamp":"2024-08-03T20:49:46.511193+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-04T22:38:27.858926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":15,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-08-03T20:49:48.540649+0000","last_change":"2024-08-03T20:49:46.543863+0000","last_active":"2024-08-03T20:49:48.540649+0000","last_peered":"2024-08-03T20:49:48.540649+0000","last_clean":"2024-08-03T20:49:48.540649+0000","last_became_active":"2024-08-03T20:49:46.543554+0000","last_became_peered":"2024-08-03T20:49:46.543554+0000","last_unstale":"2024-08-03T20:49:48.540649+0000","last_undegraded":"2024-08-03T20:49:48.540649+0000","last_fullsized":"2024-08-03T20:49:48.540649+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:49:45.503385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:49:45.503385+0000","last_clean_scrub_stamp":"2024-08-03T20:49:45.503385+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-04T22:36:14.319670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.7","version":"0'0","reported_seq":13,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-08-03T20:49:48.777325+0000","last_change":"2024-08-03T20:49:47.571609+0000","last_active":"2024-08-03T20:49:48.777325+0000","last_peered":"2024-08-03T20:49:48.777325+0000","last_clean":"2024-08-03T20:49:48.777325+0000","last_became_active":"2024-08-03T20:49:47.571187+0000","last_became_peered":"2024-08-03T20:49:47.571187+0000","last_unstale":"2024-08-03T20:49:48.777325+0000","last_undegraded":"2024-08-03T20:49:48.777325+0000","last_fullsized":"2024-08-03T20:49:48.777325+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:49:46.511193+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:49:46.511193+0000","last_clean_scrub_stamp":"2024-08-03T20:49:46.511193+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T00:56:59.567156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":13,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-08-03T20:49:47.536276+0000","last_change":"2024-08-03T20:49:46.548110+0000","last_active":"2024-08-03T20:49:47.536276+0000","last_peered":"2024-08-03T20:49:47.536276+0000","last_clean":"2024-08-03T20:49:47.536276+0000","last_became_active":"2024-08-03T20:49:46.546463+0000","last_became_peered":"2024-08-03T20:49:46.546463+0000","last_unstale":"2024-08-03T20:49:47.536276+0000","last_undegraded":"2024-08-03T20:49:47.536276+0000","last_fullsized":"2024-08-03T20:49:47.536276+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:49:45.503385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:49:45.503385+0000","last_clean_scrub_stamp":"2024-08-03T20:49:45.503385+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-04T21:13:22.709036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.4","version":"0'0","reported_seq":10,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-08-03T20:49:47.561664+0000","last_change":"2024-08-03T20:49:47.537514+0000","last_active":"2024-08-03T20:49:47.561664+0000","last_peered":"2024-08-03T20:49:47.561664+0000","last_clean":"2024-08-03T20:49:47.561664+0000","last_became_active":"2024-08-03T20:49:47.536874+0000","last_became_peered":"2024-08-03T20:49:47.536874+0000","last_unstale":"2024-08-03T20:49:47.561664+0000","last_undegraded":"2024-08-03T20:49:47.561664+0000","last_fullsized":"2024-08-03T20:49:47.561664+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:49:46.511193+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:49:46.511193+0000","last_clean_scrub_stamp":"2024-08-03T20:49:46.511193+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T01:34:35.362747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":15,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-08-03T20:49:48.541166+0000","last_change":"2024-08-03T20:49:46.545183+0000","last_active":"2024-08-03T20:49:48.541166+0000","last_peered":"2024-08-03T20:49:48.541166+0000","last_clean":"2024-08-03T20:49:48.541166+0000","last_became_active":"2024-08-03T20:49:46.544805+0000","last_became_peered":"2024-08-03T20:49:46.544805+0000","last_unstale":"2024-08-03T20:49:48.541166+0000","last_undegraded":"2024-08-03T20:49:48.541166+0000","last_fullsized":"2024-08-03T20:49:48.541166+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:49:45.503385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:49:45.503385+0000","last_clean_scrub_stamp":"2024-08-03T20:49:45.503385+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-04T23:01:41.062466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.5","version":"0'0","reported_seq":13,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-08-03T20:49:48.541446+0000","last_change":"2024-08-03T20:49:47.561794+0000","last_active":"2024-08-03T20:49:48.541446+0000","last_peered":"2024-08-03T20:49:48.541446+0000","last_clean":"2024-08-03T20:49:48.541446+0000","last_became_active":"2024-08-03T20:49:47.561373+0000","last_became_peered":"2024-08-03T20:49:47.561373+0000","last_unstale":"2024-08-03T20:49:48.541446+0000","last_undegraded":"2024-08-03T20:49:48.541446+0000","last_fullsized":"2024-08-03T20:49:48.541446+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:49:46.511193+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:49:46.511193+0000","last_clean_scrub_stamp":"2024-08-03T20:49:46.511193+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-04T21:21:00.913460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":15,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-08-03T20:49:48.698593+0000","last_change":"2024-08-03T20:49:46.543003+0000","last_active":"2024-08-03T20:49:48.698593+0000","last_peered":"2024-08-03T20:49:48.698593+0000","last_clean":"2024-08-03T20:49:48.698593+0000","last_became_active":"2024-08-03T20:49:46.542688+0000","last_became_peered":"2024-08-03T20:49:46.542688+0000","last_unstale":"2024-08-03T20:49:48.698593+0000","last_undegraded":"2024-08-03T20:49:48.698593+0000","last_fullsized":"2024-08-03T20:49:48.698593+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:49:45.503385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:49:45.503385+0000","last_clean_scrub_stamp":"2024-08-03T20:49:45.503385+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-04T22:42:13.341621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.2","version":"0'0","reported_seq":10,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-08-03T20:49:47.566207+0000","last_change":"2024-08-03T20:49:47.557901+0000","last_active":"2024-08-03T20:49:47.566207+0000","last_peered":"2024-08-03T20:49:47.566207+0000","last_clean":"2024-08-03T20:49:47.566207+0000","last_became_active":"2024-08-03T20:49:47.556167+0000","last_became_peered":"2024-08-03T20:49:47.556167+0000","last_unstale":"2024-08-03T20:49:47.566207+0000","last_undegraded":"2024-08-03T20:49:47.566207+0000","last_fullsized":"2024-08-03T20:49:47.566207+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:49:46.511193+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:49:46.511193+0000","last_clean_scrub_stamp":"2024-08-03T20:49:46.511193+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T06:13:43.294746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":15,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-08-03T20:49:48.541075+0000","last_change":"2024-08-03T20:49:46.547432+0000","last_active":"2024-08-03T20:49:48.541075+0000","last_peered":"2024-08-03T20:49:48.541075+0000","last_clean":"2024-08-03T20:49:48.541075+0000","last_became_active":"2024-08-03T20:49:46.546968+0000","last_became_peered":"2024-08-03T20:49:46.546968+0000","last_unstale":"2024-08-03T20:49:48.541075+0000","last_undegraded":"2024-08-03T20:49:48.541075+0000","last_fullsized":"2024-08-03T20:49:48.541075+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:49:45.503385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:49:45.503385+0000","last_clean_scrub_stamp":"2024-08-03T20:49:45.503385+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T06:51:29.839797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.3","version":"0'0","reported_seq":10,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-08-03T20:49:47.569019+0000","last_change":"2024-08-03T20:49:47.561766+0000","last_active":"2024-08-03T20:49:47.569019+0000","last_peered":"2024-08-03T20:49:47.569019+0000","last_clean":"2024-08-03T20:49:47.569019+0000","last_became_active":"2024-08-03T20:49:47.559616+0000","last_became_peered":"2024-08-03T20:49:47.559616+0000","last_unstale":"2024-08-03T20:49:47.569019+0000","last_undegraded":"2024-08-03T20:49:47.569019+0000","last_fullsized":"2024-08-03T20:49:47.569019+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:49:46.511193+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:49:46.511193+0000","last_clean_scrub_stamp":"2024-08-03T20:49:46.511193+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T03:46:55.349698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":13,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-08-03T20:49:47.525907+0000","last_change":"2024-08-03T20:49:46.532586+0000","last_active":"2024-08-03T20:49:47.525907+0000","last_peered":"2024-08-03T20:49:47.525907+0000","last_clean":"2024-08-03T20:49:47.525907+0000","last_became_active":"2024-08-03T20:49:46.529863+0000","last_became_peered":"2024-08-03T20:49:46.529863+0000","last_unstale":"2024-08-03T20:49:47.525907+0000","last_undegraded":"2024-08-03T20:49:47.525907+0000","last_fullsized":"2024-08-03T20:49:47.525907+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:49:45.503385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:49:45.503385+0000","last_clean_scrub_stamp":"2024-08-03T20:49:45.503385+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T08:48:09.710603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.0","version":"0'0","reported_seq":13,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-08-03T20:49:48.676252+0000","last_change":"2024-08-03T20:49:47.554238+0000","last_active":"2024-08-03T20:49:48.676252+0000","last_peered":"2024-08-03T20:49:48.676252+0000","last_clean":"2024-08-03T20:49:48.676252+0000","last_became_active":"2024-08-03T20:49:47.553888+0000","last_became_peered":"2024-08-03T20:49:47.553888+0000","last_unstale":"2024-08-03T20:49:48.676252+0000","last_undegraded":"2024-08-03T20:49:48.676252+0000","last_fullsized":"2024-08-03T20:49:48.676252+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:49:46.511193+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:49:46.511193+0000","last_clean_scrub_stamp":"2024-08-03T20:49:46.511193+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T08:13:23.504204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":15,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-08-03T20:49:48.698605+0000","last_change":"2024-08-03T20:49:46.542820+0000","last_active":"2024-08-03T20:49:48.698605+0000","last_peered":"2024-08-03T20:49:48.698605+0000","last_clean":"2024-08-03T20:49:48.698605+0000","last_became_active":"2024-08-03T20:49:46.542496+0000","last_became_peered":"2024-08-03T20:49:46.542496+0000","last_unstale":"2024-08-03T20:49:48.698605+0000","last_undegraded":"2024-08-03T20:49:48.698605+0000","last_fullsized":"2024-08-03T20:49:48.698605+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:49:45.503385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:49:45.503385+0000","last_clean_scrub_stamp":"2024-08-03T20:49:45.503385+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T01:34:34.354939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.1","version":"0'0","reported_seq":10,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-08-03T20:49:47.563604+0000","last_change":"2024-08-03T20:49:47.562671+0000","last_active":"2024-08-03T20:49:47.563604+0000","last_peered":"2024-08-03T20:49:47.563604+0000","last_clean":"2024-08-03T20:49:47.563604+0000","last_became_active":"2024-08-03T20:49:47.562220+0000","last_became_peered":"2024-08-03T20:49:47.562220+0000","last_unstale":"2024-08-03T20:49:47.563604+0000","last_undegraded":"2024-08-03T20:49:47.563604+0000","last_fullsized":"2024-08-03T20:49:47.563604+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:49:46.511193+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:49:46.511193+0000","last_clean_scrub_stamp":"2024-08-03T20:49:46.511193+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T01:03:09.792447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":13,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-08-03T20:49:47.525556+0000","last_change":"2024-08-03T20:49:46.536206+0000","last_active":"2024-08-03T20:49:47.525556+0000","last_peered":"2024-08-03T20:49:47.525556+0000","last_clean":"2024-08-03T20:49:47.525556+0000","last_became_active":"2024-08-03T20:49:46.535893+0000","last_became_peered":"2024-08-03T20:49:46.535893+0000","last_unstale":"2024-08-03T20:49:47.525556+0000","last_undegraded":"2024-08-03T20:49:47.525556+0000","last_fullsized":"2024-08-03T20:49:47.525556+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:49:45.503385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:49:45.503385+0000","last_clean_scrub_stamp":"2024-08-03T20:49:45.503385+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T01:32:49.376691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.6","version":"0'0","reported_seq":10,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-08-03T20:49:47.575675+0000","last_change":"2024-08-03T20:49:47.574004+0000","last_active":"2024-08-03T20:49:47.575675+0000","last_peered":"2024-08-03T20:49:47.575675+0000","last_clean":"2024-08-03T20:49:47.575675+0000","last_became_active":"2024-08-03T20:49:47.573184+0000","last_became_peered":"2024-08-03T20:49:47.573184+0000","last_unstale":"2024-08-03T20:49:47.575675+0000","last_undegraded":"2024-08-03T20:49:47.575675+0000","last_fullsized":"2024-08-03T20:49:47.575675+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:49:46.511193+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:49:46.511193+0000","last_clean_scrub_stamp":"2024-08-03T20:49:46.511193+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T08:04:30.955111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.7","version":"44'2","reported_seq":17,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-08-03T20:49:48.676672+0000","last_change":"2024-08-03T20:49:46.544225+0000","last_active":"2024-08-03T20:49:48.676672+0000","last_peered":"2024-08-03T20:49:48.676672+0000","last_clean":"2024-08-03T20:49:48.676672+0000","last_became_active":"2024-08-03T20:49:46.543900+0000","last_became_peered":"2024-08-03T20:49:46.543900+0000","last_unstale":"2024-08-03T20:49:48.676672+0000","last_undegraded":"2024-08-03T20:49:48.676672+0000","last_fullsized":"2024-08-03T20:49:48.676672+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:49:45.503385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:49:45.503385+0000","last_clean_scrub_stamp":"2024-08-03T20:49:45.503385+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T00:00:40.899232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"16'72","reported_seq":154,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-08-03T20:49:48.541343+0000","last_change":"2024-08-03T20:47:30.547632+0000","last_active":"2024-08-03T20:49:48.541343+0000","last_peered":"2024-08-03T20:49:48.541343+0000","last_clean":"2024-08-03T20:49:48.541343+0000","last_became_active":"2024-08-03T20:47:30.546255+0000","last_became_peered":"2024-08-03T20:47:30.546255+0000","last_unstale":"2024-08-03T20:49:48.541343+0000","last_undegraded":"2024-08-03T20:49:48.541343+0000","last_fullsized":"2024-08-03T20:49:48.541343+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:47:28.874451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:47:28.874451+0000","last_clean_scrub_stamp":"2024-08-03T20:47:28.874451+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T00:19:40.548812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509471,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32272,"kb_used_data":1092,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339568,"statfs":{"total":96636764160,"available":96603717632,"internally_reserved":0,"allocated":1118208,"data_stored":801451,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":6,"up_from":15,"seq":64424509472,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31704,"kb_used_data":516,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340136,"statfs":{"total":96636764160,"available":96604299264,"internally_reserved":0,"allocated":528384,"data_stored":211665,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":5,"up_from":15,"seq":64424509472,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31700,"kb_used_data":512,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340140,"statfs":{"total":96636764160,"available":96604303360,"internally_reserved":0,"allocated":524288,"data_stored":211063,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509471,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31700,"kb_used_data":512,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340140,"statfs":{"total":96636764160,"available":96604303360,"internally_reserved":0,"allocated":524288,"data_stored":211063,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509473,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32268,"kb_used_data":1080,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339572,"statfs":{"total":96636764160,"available":96603721728,"internally_reserved":0,"allocated":1105920,"data_stored":795095,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542176,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31696,"kb_used_data":508,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340144,"statfs":{"total":96636764160,"available":96604307456,"internally_reserved":0,"allocated":520192,"data_stored":206305,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":3,"up_from":14,"seq":60129542174,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31696,"kb_used_data":508,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340144,"statfs":{"total":96636764160,"available":96604307456,"internally_reserved":0,"allocated":520192,"data_stored":205331,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":14,"seq":60129542175,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31700,"kb_used_data":512,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340140,"statfs":{"total":96636764160,"available":96604303360,"internally_reserved":0,"allocated":524288,"data_stored":206327,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-08-03T20:49:52.566 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-08-03T20:49:52.566 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-08-03T20:49:52.839 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:52.838+0000 7feca5200640 1 -- 172.21.15.32:0/973580634 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fec84004d40 msgr2=0x7fec84005150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:49:52.840 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:52.838+0000 7feca5200640 1 --2- 172.21.15.32:0/973580634 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fec84004d40 0x7fec84005150 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7fec98009ed0 tx=0x7fec84005830 comp rx=0 tx=0).stop 2024-08-03T20:49:52.840 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:52.838+0000 7fec9e800640 1 -- 172.21.15.32:0/973580634 reap_dead start 2024-08-03T20:49:52.840 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:52.839+0000 7feca5200640 1 -- 172.21.15.32:0/973580634 shutdown_connections 2024-08-03T20:49:52.840 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:52.839+0000 7feca5200640 1 -- 172.21.15.32:0/973580634 >> 172.21.15.32:0/973580634 conn(0x7feca00b1bd0 msgr2=0x7feca00b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:49:52.840 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:52.839+0000 7feca5200640 1 -- 172.21.15.32:0/973580634 shutdown_connections 2024-08-03T20:49:52.840 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:52.839+0000 7feca5200640 1 -- 172.21.15.32:0/973580634 wait complete. 2024-08-03T20:49:52.841 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:52.841+0000 7feca5200640 1 Processor -- start 2024-08-03T20:49:52.841 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:52.841+0000 7feca5200640 1 -- start start 2024-08-03T20:49:52.844 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:52.843+0000 7feca5200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fec84004d40 0x7feca00a94b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:49:52.844 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:52.843+0000 7feca5200640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7feca013dc90 0x7feca013a6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:49:52.844 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:52.843+0000 7feca5200640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7feca0137010 0x7feca0137420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:49:52.845 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:52.843+0000 7feca5200640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7feca0145480 con 0x7fec84004d40 2024-08-03T20:49:52.845 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:52.843+0000 7feca5200640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7feca0145300 con 0x7feca013dc90 2024-08-03T20:49:52.845 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:52.843+0000 7feca5200640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7feca0145600 con 0x7feca0137010 2024-08-03T20:49:52.845 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:52.843+0000 7fec9de00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fec84004d40 0x7feca00a94b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:49:52.845 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:52.843+0000 7fec9de00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fec84004d40 0x7feca00a94b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:38064/0 (socket says 172.21.15.32:38064) 2024-08-03T20:49:52.845 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:52.843+0000 7fec9de00640 1 -- 172.21.15.32:0/2941190934 learned_addr learned my addr 172.21.15.32:0/2941190934 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:49:52.845 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:52.843+0000 7fec9e800640 1 --2- 172.21.15.32:0/2941190934 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7feca0137010 0x7feca0137420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:49:52.845 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:52.843+0000 7fec9d400640 1 --2- 172.21.15.32:0/2941190934 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7feca013dc90 0x7feca013a6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:49:52.845 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:52.843+0000 7fec9de00640 1 -- 172.21.15.32:0/2941190934 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7feca0137010 msgr2=0x7feca0137420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:49:52.845 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:52.843+0000 7fec9de00640 1 --2- 172.21.15.32:0/2941190934 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7feca0137010 0x7feca0137420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:49:52.845 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:52.843+0000 7fec9de00640 1 -- 172.21.15.32:0/2941190934 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7feca013dc90 msgr2=0x7feca013a6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:49:52.845 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:52.843+0000 7fec9de00640 1 --2- 172.21.15.32:0/2941190934 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7feca013dc90 0x7feca013a6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:49:52.845 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:52.843+0000 7fec9de00640 1 -- 172.21.15.32:0/2941190934 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feca01f5d60 con 0x7fec84004d40 2024-08-03T20:49:52.845 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:52.844+0000 7fec9de00640 1 --2- 172.21.15.32:0/2941190934 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fec84004d40 0x7feca00a94b0 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7fec900066a0 tx=0x7fec90006790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:49:52.846 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:52.844+0000 7fec96000640 1 -- 172.21.15.32:0/2941190934 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec90015020 con 0x7fec84004d40 2024-08-03T20:49:52.846 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:52.844+0000 7fec96000640 1 -- 172.21.15.32:0/2941190934 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fec90010980 con 0x7fec84004d40 2024-08-03T20:49:52.846 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:52.844+0000 7fec96000640 1 -- 172.21.15.32:0/2941190934 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec90010dd0 con 0x7fec84004d40 2024-08-03T20:49:52.846 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:52.844+0000 7feca5200640 1 -- 172.21.15.32:0/2941190934 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7feca01f6040 con 0x7fec84004d40 2024-08-03T20:49:52.846 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:52.844+0000 7feca5200640 1 -- 172.21.15.32:0/2941190934 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7feca013a1b0 con 0x7fec84004d40 2024-08-03T20:49:52.846 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:52.845+0000 7feca5200640 1 -- 172.21.15.32:0/2941190934 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7feca0145b90 con 0x7fec84004d40 2024-08-03T20:49:52.847 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:52.846+0000 7fec96000640 1 -- 172.21.15.32:0/2941190934 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7fec90018020 con 0x7fec84004d40 2024-08-03T20:49:52.848 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:52.847+0000 7fec96000640 1 --2- 172.21.15.32:0/2941190934 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fec80077040 0x7fec80079550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:49:52.848 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:52.847+0000 7fec96000640 1 -- 172.21.15.32:0/2941190934 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(45..45 src has 1..45) ==== 5709+0+0 (secure 0 0 0) 0x7fec900995a0 con 0x7fec84004d40 2024-08-03T20:49:52.848 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:52.848+0000 7fec9d400640 1 --2- 172.21.15.32:0/2941190934 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fec80077040 0x7fec80079550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:49:52.849 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:52.849+0000 7fec9d400640 1 --2- 172.21.15.32:0/2941190934 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fec80077040 0x7fec80079550 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7feca00cb9c0 tx=0x7fec8c005f90 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:49:52.852 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:52.852+0000 7fec96000640 1 -- 172.21.15.32:0/2941190934 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7fec9001f020 con 0x7fec84004d40 2024-08-03T20:49:53.140 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:53.139+0000 7feca5200640 1 -- 172.21.15.32:0/2941190934 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) -- 0x7feca006f120 con 0x7fec84004d40 2024-08-03T20:49:53.143 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:53.142+0000 7fec96000640 1 -- 172.21.15.32:0/2941190934 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v12) ==== 190+0+147 (secure 0 0 0) 0x7feca0160930 con 0x7fec84004d40 2024-08-03T20:49:53.143 INFO:teuthology.orchestra.run.smithi032.stdout:[client.0] 2024-08-03T20:49:53.143 INFO:teuthology.orchestra.run.smithi032.stdout: key = AQAwl65msgO3NRAAHG/WuMTK9c4lDXocXvqWnA== 2024-08-03T20:49:53.143 INFO:teuthology.orchestra.run.smithi032.stdout: caps mds = "allow" 2024-08-03T20:49:53.143 INFO:teuthology.orchestra.run.smithi032.stdout: caps mon = "allow r" 2024-08-03T20:49:53.143 INFO:teuthology.orchestra.run.smithi032.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2024-08-03T20:49:53.143 INFO:teuthology.orchestra.run.smithi032.stderr:updated caps for client.0 2024-08-03T20:49:53.146 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:53.145+0000 7feca5200640 1 -- 172.21.15.32:0/2941190934 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fec80077040 msgr2=0x7fec80079550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:49:53.146 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:53.145+0000 7feca5200640 1 --2- 172.21.15.32:0/2941190934 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fec80077040 0x7fec80079550 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7feca00cb9c0 tx=0x7fec8c005f90 comp rx=0 tx=0).stop 2024-08-03T20:49:53.146 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:53.145+0000 7feca5200640 1 -- 172.21.15.32:0/2941190934 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fec84004d40 msgr2=0x7feca00a94b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:49:53.146 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:53.145+0000 7feca5200640 1 --2- 172.21.15.32:0/2941190934 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fec84004d40 0x7feca00a94b0 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7fec900066a0 tx=0x7fec90006790 comp rx=0 tx=0).stop 2024-08-03T20:49:53.147 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:53.145+0000 7feca5200640 1 -- 172.21.15.32:0/2941190934 shutdown_connections 2024-08-03T20:49:53.147 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:53.145+0000 7feca5200640 1 --2- 172.21.15.32:0/2941190934 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fec80077040 0x7fec80079550 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:49:53.147 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:53.145+0000 7feca5200640 1 --2- 172.21.15.32:0/2941190934 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7feca013dc90 0x7feca013a6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:49:53.147 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:53.145+0000 7feca5200640 1 --2- 172.21.15.32:0/2941190934 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7feca0137010 0x7feca0137420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:49:53.147 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:53.145+0000 7feca5200640 1 --2- 172.21.15.32:0/2941190934 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fec84004d40 0x7feca00a94b0 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:49:53.147 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:53.145+0000 7feca5200640 1 -- 172.21.15.32:0/2941190934 >> 172.21.15.32:0/2941190934 conn(0x7feca00b1bd0 msgr2=0x7feca015bef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:49:53.147 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:53.145+0000 7feca5200640 1 -- 172.21.15.32:0/2941190934 shutdown_connections 2024-08-03T20:49:53.147 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:53.145+0000 7feca5200640 1 -- 172.21.15.32:0/2941190934 wait complete. 2024-08-03T20:49:53.171 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-08-03T20:49:53.405 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:53.404+0000 7fd981400640 1 -- 172.21.15.32:0/3074508787 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd97c06fb90 msgr2=0x7fd97c0ca3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:49:53.405 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:53.404+0000 7fd981400640 1 --2- 172.21.15.32:0/3074508787 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd97c06fb90 0x7fd97c0ca3b0 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7fd974005fd0 tx=0x7fd97402a940 comp rx=0 tx=0).stop 2024-08-03T20:49:53.406 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:53.404+0000 7fd981400640 1 -- 172.21.15.32:0/3074508787 shutdown_connections 2024-08-03T20:49:53.406 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:53.404+0000 7fd981400640 1 --2- 172.21.15.32:0/3074508787 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd97c06fb90 0x7fd97c0ca3b0 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:49:53.406 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:53.404+0000 7fd981400640 1 -- 172.21.15.32:0/3074508787 >> 172.21.15.32:0/3074508787 conn(0x7fd97c0b2120 msgr2=0x7fd97c0cfd60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:49:53.406 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:53.405+0000 7fd981400640 1 -- 172.21.15.32:0/3074508787 shutdown_connections 2024-08-03T20:49:53.406 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:53.405+0000 7fd981400640 1 -- 172.21.15.32:0/3074508787 wait complete. 2024-08-03T20:49:53.407 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:53.407+0000 7fd981400640 1 Processor -- start 2024-08-03T20:49:53.407 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:53.407+0000 7fd981400640 1 -- start start 2024-08-03T20:49:53.409 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:53.408+0000 7fd981400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd97c0af040 0x7fd97c15c750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:49:53.409 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:53.408+0000 7fd981400640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fd97c0cee10 0x7fd97c0c8d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:49:53.409 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:53.408+0000 7fd981400640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fd97c1f9270 0x7fd97c1f86d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:49:53.409 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:53.408+0000 7fd981400640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fd97c0caa70 con 0x7fd97c0af040 2024-08-03T20:49:53.410 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:53.408+0000 7fd981400640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7fd97c0ca8f0 con 0x7fd97c0cee10 2024-08-03T20:49:53.410 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:53.408+0000 7fd981400640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7fd97c0cabf0 con 0x7fd97c1f9270 2024-08-03T20:49:53.411 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:53.409+0000 7fd97be00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd97c0af040 0x7fd97c15c750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:49:53.411 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:53.409+0000 7fd97be00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd97c0af040 0x7fd97c15c750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:38082/0 (socket says 172.21.15.32:38082) 2024-08-03T20:49:53.411 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:53.409+0000 7fd97be00640 1 -- 172.21.15.32:0/3275625358 learned_addr learned my addr 172.21.15.32:0/3275625358 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:49:53.411 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:53.409+0000 7fd980a00640 1 --2- 172.21.15.32:0/3275625358 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fd97c1f9270 0x7fd97c1f86d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:49:53.411 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:53.409+0000 7fd97be00640 1 -- 172.21.15.32:0/3275625358 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fd97c1f9270 msgr2=0x7fd97c1f86d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:49:53.411 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:53.409+0000 7fd97be00640 1 --2- 172.21.15.32:0/3275625358 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fd97c1f9270 0x7fd97c1f86d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:49:53.411 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:53.409+0000 7fd97b400640 1 --2- 172.21.15.32:0/3275625358 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fd97c0cee10 0x7fd97c0c8d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:49:53.411 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:53.409+0000 7fd97be00640 1 -- 172.21.15.32:0/3275625358 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fd97c0cee10 msgr2=0x7fd97c0c8d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:49:53.411 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:53.409+0000 7fd97be00640 1 --2- 172.21.15.32:0/3275625358 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fd97c0cee10 0x7fd97c0c8d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:49:53.411 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:53.409+0000 7fd97be00640 1 -- 172.21.15.32:0/3275625358 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd97c0c92a0 con 0x7fd97c0af040 2024-08-03T20:49:53.411 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:53.409+0000 7fd97b400640 1 --2- 172.21.15.32:0/3275625358 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fd97c0cee10 0x7fd97c0c8d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-03T20:49:53.411 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:53.410+0000 7fd97be00640 1 --2- 172.21.15.32:0/3275625358 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd97c0af040 0x7fd97c15c750 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7fd96c00c400 tx=0x7fd96c00c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:49:53.411 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:53.410+0000 7fd973e00640 1 -- 172.21.15.32:0/3275625358 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd96c017070 con 0x7fd97c0af040 2024-08-03T20:49:53.412 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:53.410+0000 7fd973e00640 1 -- 172.21.15.32:0/3275625358 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fd96c0053c0 con 0x7fd97c0af040 2024-08-03T20:49:53.412 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:53.410+0000 7fd981400640 1 -- 172.21.15.32:0/3275625358 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd97c15d360 con 0x7fd97c0af040 2024-08-03T20:49:53.412 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:53.410+0000 7fd973e00640 1 -- 172.21.15.32:0/3275625358 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd96c003850 con 0x7fd97c0af040 2024-08-03T20:49:53.412 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:53.410+0000 7fd981400640 1 -- 172.21.15.32:0/3275625358 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd97c15d920 con 0x7fd97c0af040 2024-08-03T20:49:53.412 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:53.411+0000 7fd981400640 1 -- 172.21.15.32:0/3275625358 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd97c15ac00 con 0x7fd97c0af040 2024-08-03T20:49:53.416 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:53.412+0000 7fd973e00640 1 -- 172.21.15.32:0/3275625358 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7fd96c01a070 con 0x7fd97c0af040 2024-08-03T20:49:53.416 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:53.413+0000 7fd973e00640 1 --2- 172.21.15.32:0/3275625358 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fd95c077010 0x7fd95c079520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:49:53.416 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:53.414+0000 7fd973e00640 1 -- 172.21.15.32:0/3275625358 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(45..45 src has 1..45) ==== 5709+0+0 (secure 0 0 0) 0x7fd96c097ea0 con 0x7fd97c0af040 2024-08-03T20:49:53.416 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:53.414+0000 7fd97b400640 1 --2- 172.21.15.32:0/3275625358 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fd95c077010 0x7fd95c079520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:49:53.416 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:53.415+0000 7fd97b400640 1 --2- 172.21.15.32:0/3275625358 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fd95c077010 0x7fd95c079520 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fd97c0c8310 tx=0x7fd968009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:49:53.419 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:53.418+0000 7fd973e00640 1 -- 172.21.15.32:0/3275625358 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7fd97c0c92a0 con 0x7fd97c0af040 2024-08-03T20:49:53.663 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:53.663+0000 7fd981400640 1 -- 172.21.15.32:0/3275625358 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd97c0ce270 con 0x7fd97c0af040 2024-08-03T20:49:53.664 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:53.664+0000 7fd973e00640 1 -- 172.21.15.32:0/3275625358 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 35 v35) ==== 94+0+5226 (secure 0 0 0) 0x7fd96c060f80 con 0x7fd97c0af040 2024-08-03T20:49:53.664 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-03T20:49:53.665 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":35,"btime":"2024-08-03T20:49:50:544317+0000","default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4997,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6837/2688500839","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6836","nonce":2688500839},{"type":"v1","addr":"172.21.15.118:6837","nonce":2688500839}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":30},{"gid":5266,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6835/2381075275","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6834","nonce":2381075275},{"type":"v1","addr":"172.21.15.118:6835","nonce":2381075275}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":28},{"gid":5421,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6833/1058332640","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":1058332640},{"type":"v1","addr":"172.21.15.32:6833","nonce":1058332640}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":29}],"filesystems":[{"mdsmap":{"epoch":35,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-08-03T20:49:47.064654+0000","modified":"2024-08-03T20:49:50.067811+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5424},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5424":{"gid":5424,"name":"c","rank":0,"incarnation":31,"state":"up:active","state_seq":4,"addr":"172.21.15.32:6835/678469885","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":678469885},{"type":"v1","addr":"172.21.15.32:6835","nonce":678469885}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5424,"qdb_cluster":[5424]},"id":3}]} 2024-08-03T20:49:53.665 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 35 2024-08-03T20:49:53.667 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:53.666+0000 7fd981400640 1 -- 172.21.15.32:0/3275625358 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fd95c077010 msgr2=0x7fd95c079520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:49:53.668 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:53.666+0000 7fd981400640 1 --2- 172.21.15.32:0/3275625358 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fd95c077010 0x7fd95c079520 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fd97c0c8310 tx=0x7fd968009040 comp rx=0 tx=0).stop 2024-08-03T20:49:53.668 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:53.666+0000 7fd981400640 1 -- 172.21.15.32:0/3275625358 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd97c0af040 msgr2=0x7fd97c15c750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:49:53.668 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:53.666+0000 7fd981400640 1 --2- 172.21.15.32:0/3275625358 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd97c0af040 0x7fd97c15c750 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7fd96c00c400 tx=0x7fd96c00c8c0 comp rx=0 tx=0).stop 2024-08-03T20:49:53.668 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:53.666+0000 7fd981400640 1 -- 172.21.15.32:0/3275625358 shutdown_connections 2024-08-03T20:49:53.668 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:53.666+0000 7fd981400640 1 --2- 172.21.15.32:0/3275625358 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fd95c077010 0x7fd95c079520 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:49:53.668 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:53.666+0000 7fd981400640 1 --2- 172.21.15.32:0/3275625358 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fd97c0cee10 0x7fd97c0c8d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:49:53.668 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:53.666+0000 7fd981400640 1 --2- 172.21.15.32:0/3275625358 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fd97c1f9270 0x7fd97c1f86d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:49:53.668 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:53.666+0000 7fd981400640 1 --2- 172.21.15.32:0/3275625358 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd97c0af040 0x7fd97c15c750 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:49:53.669 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:53.666+0000 7fd981400640 1 -- 172.21.15.32:0/3275625358 >> 172.21.15.32:0/3275625358 conn(0x7fd97c0b2120 msgr2=0x7fd97c15eaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:49:53.669 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:53.666+0000 7fd981400640 1 -- 172.21.15.32:0/3275625358 shutdown_connections 2024-08-03T20:49:53.669 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:53.666+0000 7fd981400640 1 -- 172.21.15.32:0/3275625358 wait complete. 2024-08-03T20:49:54.098 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 35, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-08-03T20:49:47.064654+0000', 'modified': '2024-08-03T20:49:50.067811+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 5424}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_5424': {'gid': 5424, 'name': 'c', 'rank': 0, 'incarnation': 31, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.32:6835/678469885', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6834', 'nonce': 678469885}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 678469885}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [7], 'metadata_pool': 6, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 5424, 'qdb_cluster': [5424]} 2024-08-03T20:49:54.099 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-08-03T20:49:54.099 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-08-03T20:49:54.334 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:54.333+0000 7fc777c00640 1 -- 172.21.15.32:0/3687596178 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fc77006e930 msgr2=0x7fc770144f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:49:54.335 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:54.333+0000 7fc777c00640 1 --2- 172.21.15.32:0/3687596178 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fc77006e930 0x7fc770144f90 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fc764009860 tx=0x7fc76402a5a0 comp rx=0 tx=0).stop 2024-08-03T20:49:54.335 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:54.333+0000 7fc777c00640 1 -- 172.21.15.32:0/3687596178 shutdown_connections 2024-08-03T20:49:54.335 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:54.333+0000 7fc777c00640 1 --2- 172.21.15.32:0/3687596178 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fc77006e930 0x7fc770144f90 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:49:54.335 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:54.333+0000 7fc777c00640 1 -- 172.21.15.32:0/3687596178 >> 172.21.15.32:0/3687596178 conn(0x7fc7700b1bd0 msgr2=0x7fc7700b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:49:54.335 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:54.333+0000 7fc777c00640 1 -- 172.21.15.32:0/3687596178 shutdown_connections 2024-08-03T20:49:54.335 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:54.333+0000 7fc777c00640 1 -- 172.21.15.32:0/3687596178 wait complete. 2024-08-03T20:49:54.336 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:54.336+0000 7fc777c00640 1 Processor -- start 2024-08-03T20:49:54.336 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:54.336+0000 7fc777c00640 1 -- start start 2024-08-03T20:49:54.339 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:54.337+0000 7fc777c00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc77006e930 0x7fc77015ca70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:49:54.340 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:54.337+0000 7fc777c00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc770073e60 0x7fc7700b02b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:49:54.340 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:54.337+0000 7fc777c00640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fc77015fff0 0x7fc7700b07f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:49:54.340 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:54.337+0000 7fc777c00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fc7700cf840 con 0x7fc77006e930 2024-08-03T20:49:54.340 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:54.337+0000 7fc777c00640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7fc770145f30 con 0x7fc770073e60 2024-08-03T20:49:54.340 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:54.337+0000 7fc777c00640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7fc7700cf9c0 con 0x7fc77015fff0 2024-08-03T20:49:54.340 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:54.337+0000 7fc775200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc77006e930 0x7fc77015ca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:49:54.340 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:54.337+0000 7fc775200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc77006e930 0x7fc77015ca70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:38096/0 (socket says 172.21.15.32:38096) 2024-08-03T20:49:54.340 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:54.337+0000 7fc775200640 1 -- 172.21.15.32:0/1865697863 learned_addr learned my addr 172.21.15.32:0/1865697863 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:49:54.341 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:54.338+0000 7fc76fe00640 1 --2- 172.21.15.32:0/1865697863 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc770073e60 0x7fc7700b02b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:49:54.341 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:54.338+0000 7fc775c00640 1 --2- 172.21.15.32:0/1865697863 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fc77015fff0 0x7fc7700b07f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:49:54.341 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:54.338+0000 7fc775200640 1 -- 172.21.15.32:0/1865697863 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fc77015fff0 msgr2=0x7fc7700b07f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:49:54.341 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:54.338+0000 7fc775200640 1 --2- 172.21.15.32:0/1865697863 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fc77015fff0 0x7fc7700b07f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:49:54.341 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:54.338+0000 7fc775200640 1 -- 172.21.15.32:0/1865697863 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc770073e60 msgr2=0x7fc7700b02b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:49:54.341 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:54.338+0000 7fc775200640 1 --2- 172.21.15.32:0/1865697863 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc770073e60 0x7fc7700b02b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:49:54.341 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:54.338+0000 7fc775200640 1 -- 172.21.15.32:0/1865697863 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc77013f460 con 0x7fc77006e930 2024-08-03T20:49:54.341 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:54.338+0000 7fc76fe00640 1 --2- 172.21.15.32:0/1865697863 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc770073e60 0x7fc7700b02b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-03T20:49:54.341 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:54.338+0000 7fc775200640 1 --2- 172.21.15.32:0/1865697863 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc77006e930 0x7fc77015ca70 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7fc76402ab90 tx=0x7fc764033ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:49:54.342 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:54.338+0000 7fc76cc00640 1 -- 172.21.15.32:0/1865697863 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc764031b50 con 0x7fc77006e930 2024-08-03T20:49:54.342 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:54.339+0000 7fc777c00640 1 -- 172.21.15.32:0/1865697863 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc7701f5780 con 0x7fc77006e930 2024-08-03T20:49:54.342 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:54.339+0000 7fc76cc00640 1 -- 172.21.15.32:0/1865697863 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fc764031de0 con 0x7fc77006e930 2024-08-03T20:49:54.342 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:54.339+0000 7fc76cc00640 1 -- 172.21.15.32:0/1865697863 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc764033330 con 0x7fc77006e930 2024-08-03T20:49:54.342 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:54.339+0000 7fc777c00640 1 -- 172.21.15.32:0/1865697863 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc7701f5d40 con 0x7fc77006e930 2024-08-03T20:49:54.343 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:54.341+0000 7fc76cc00640 1 -- 172.21.15.32:0/1865697863 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7fc7640334d0 con 0x7fc77006e930 2024-08-03T20:49:54.343 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:54.342+0000 7fc76cc00640 1 --2- 172.21.15.32:0/1865697863 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fc750077010 0x7fc750079520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:49:54.344 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:54.342+0000 7fc76cc00640 1 -- 172.21.15.32:0/1865697863 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(45..45 src has 1..45) ==== 5709+0+0 (secure 0 0 0) 0x7fc7640b9c20 con 0x7fc77006e930 2024-08-03T20:49:54.344 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:54.342+0000 7fc76fe00640 1 --2- 172.21.15.32:0/1865697863 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fc750077010 0x7fc750079520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:49:54.344 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:54.343+0000 7fc77ec61740 1 Processor -- start 2024-08-03T20:49:54.344 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:54.343+0000 7fc77ec61740 1 -- start start 2024-08-03T20:49:54.344 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:54.343+0000 7fc76fe00640 1 --2- 172.21.15.32:0/1865697863 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fc750077010 0x7fc750079520 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fc7701f9bb0 tx=0x7fc75c009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:49:54.346 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:54.344+0000 7fc77ec61740 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55da0b004270 0x55da0b3f10d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:49:54.346 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:54.344+0000 7fc77ec61740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55da0afdb3b0 0x55da0b3d3da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:49:54.346 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:54.344+0000 7fc77ec61740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55da0afe3590 0x55da0b3d42e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:49:54.346 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:54.344+0000 7fc77ec61740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55da0b31ca60 con 0x55da0afe3590 2024-08-03T20:49:54.347 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:54.344+0000 7fc77ec61740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x55da0b30bf60 con 0x55da0afdb3b0 2024-08-03T20:49:54.347 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:54.344+0000 7fc77ec61740 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x55da0b30ffa0 con 0x55da0b004270 2024-08-03T20:49:54.347 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:54.344+0000 7fc775c00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55da0afe3590 0x55da0b3d42e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:49:54.347 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:54.344+0000 7fc775c00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55da0afe3590 0x55da0b3d42e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:38104/0 (socket says 172.21.15.32:38104) 2024-08-03T20:49:54.347 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:54.344+0000 7fc775c00640 1 -- 172.21.15.32:0/537834298 learned_addr learned my addr 172.21.15.32:0/537834298 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:49:54.347 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:54.345+0000 7fc775200640 1 --2- 172.21.15.32:0/537834298 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55da0b004270 0x55da0b3f10d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:49:54.347 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:54.345+0000 7fc76fe00640 1 --2- 172.21.15.32:0/537834298 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55da0afdb3b0 0x55da0b3d3da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:49:54.347 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:54.345+0000 7fc775c00640 1 -- 172.21.15.32:0/537834298 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55da0b004270 msgr2=0x55da0b3f10d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:49:54.347 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:54.345+0000 7fc775c00640 1 --2- 172.21.15.32:0/537834298 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55da0b004270 0x55da0b3f10d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:49:54.347 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:54.345+0000 7fc775c00640 1 -- 172.21.15.32:0/537834298 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55da0afdb3b0 msgr2=0x55da0b3d3da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:49:54.347 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:54.345+0000 7fc775c00640 1 --2- 172.21.15.32:0/537834298 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55da0afdb3b0 0x55da0b3d3da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:49:54.347 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:54.345+0000 7fc775c00640 1 -- 172.21.15.32:0/537834298 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55da0b06f850 con 0x55da0afe3590 2024-08-03T20:49:54.348 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:54.345+0000 7fc775c00640 1 --2- 172.21.15.32:0/537834298 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55da0afe3590 0x55da0b3d42e0 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7fc768002010 tx=0x7fc76800e530 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fa4fdbe4389f5072 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:49:54.348 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:54.345+0000 7fc761600640 1 -- 172.21.15.32:0/537834298 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc76800ee50 con 0x55da0afe3590 2024-08-03T20:49:54.348 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:54.345+0000 7fc761600640 1 -- 172.21.15.32:0/537834298 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fc768004a40 con 0x55da0afe3590 2024-08-03T20:49:54.348 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:54.345+0000 7fc761600640 1 -- 172.21.15.32:0/537834298 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc768004c90 con 0x55da0afe3590 2024-08-03T20:49:54.349 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:54.346+0000 7fc77ec61740 1 -- 172.21.15.32:0/537834298 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55da0afe3590 msgr2=0x55da0b3d42e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:49:54.349 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:54.346+0000 7fc77ec61740 1 --2- 172.21.15.32:0/537834298 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55da0afe3590 0x55da0b3d42e0 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7fc768002010 tx=0x7fc76800e530 comp rx=0 tx=0).stop 2024-08-03T20:49:54.352 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:54.346+0000 7fc77ec61740 1 -- 172.21.15.32:0/537834298 shutdown_connections 2024-08-03T20:49:54.353 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:54.346+0000 7fc77ec61740 1 --2- 172.21.15.32:0/537834298 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55da0afdb3b0 0x55da0b3d3da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:49:54.353 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:54.346+0000 7fc77ec61740 1 --2- 172.21.15.32:0/537834298 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55da0afe3590 0x55da0b3d42e0 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:49:54.353 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:54.346+0000 7fc77ec61740 1 --2- 172.21.15.32:0/537834298 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55da0b004270 0x55da0b3f10d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:49:54.353 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:54.346+0000 7fc77ec61740 1 -- 172.21.15.32:0/537834298 >> 172.21.15.32:0/537834298 conn(0x55da0aff83c0 msgr2=0x55da0b3a9fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:49:54.353 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:54.346+0000 7fc77ec61740 1 -- 172.21.15.32:0/537834298 shutdown_connections 2024-08-03T20:49:54.353 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:54.346+0000 7fc77ec61740 1 -- 172.21.15.32:0/537834298 wait complete. 2024-08-03T20:49:54.353 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:54.347+0000 7fc77ec61740 1 -- start start 2024-08-03T20:49:54.354 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:54.348+0000 7fc77ec61740 1 Processor -- start 2024-08-03T20:49:54.354 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:54.349+0000 7fc77ec61740 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55da0afdb3b0 0x55da0b3ea960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:49:54.354 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:54.349+0000 7fc77ec61740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55da0afe3590 0x55da0b3d0de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:49:54.354 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:54.349+0000 7fc77ec61740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55da0b004270 0x55da0b3c2650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:49:54.354 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:54.349+0000 7fc77ec61740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55da0b07e620 con 0x55da0b004270 2024-08-03T20:49:54.354 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:54.349+0000 7fc77ec61740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x55da0b2a9160 con 0x55da0afe3590 2024-08-03T20:49:54.354 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:54.349+0000 7fc77ec61740 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x55da0b3ed840 con 0x55da0afdb3b0 2024-08-03T20:49:54.354 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:54.349+0000 7fc77ec61740 10 client.0 _ll_register_callbacks cb 0x55da0b2b86b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-03T20:49:54.354 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:54.349+0000 7fc775c00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55da0b004270 0x55da0b3c2650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:49:54.354 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:54.349+0000 7fc775c00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55da0b004270 0x55da0b3c2650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:38120/0 (socket says 172.21.15.32:38120) 2024-08-03T20:49:54.354 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:54.349+0000 7fc775c00640 1 -- 172.21.15.32:0/440148963 learned_addr learned my addr 172.21.15.32:0/440148963 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:49:54.354 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:54.349+0000 7fc775200640 1 --2- 172.21.15.32:0/440148963 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55da0afdb3b0 0x55da0b3ea960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:49:54.354 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:54.350+0000 7fc76fe00640 1 --2- 172.21.15.32:0/440148963 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55da0afe3590 0x55da0b3d0de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:49:54.355 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:54.350+0000 7fc775c00640 1 -- 172.21.15.32:0/440148963 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55da0afdb3b0 msgr2=0x55da0b3ea960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:49:54.355 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:54.350+0000 7fc775c00640 1 --2- 172.21.15.32:0/440148963 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55da0afdb3b0 0x55da0b3ea960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:49:54.355 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:54.350+0000 7fc775c00640 1 -- 172.21.15.32:0/440148963 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55da0afe3590 msgr2=0x55da0b3d0de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:49:54.355 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:54.350+0000 7fc775c00640 1 --2- 172.21.15.32:0/440148963 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55da0afe3590 0x55da0b3d0de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:49:54.355 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:54.350+0000 7fc775c00640 1 -- 172.21.15.32:0/440148963 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55da0b0e66b0 con 0x55da0b004270 2024-08-03T20:49:54.355 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:54.350+0000 7fc76fe00640 1 --2- 172.21.15.32:0/440148963 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55da0afe3590 0x55da0b3d0de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-03T20:49:54.355 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:54.350+0000 7fc775c00640 1 --2- 172.21.15.32:0/440148963 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55da0b004270 0x55da0b3c2650 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7fc768016040 tx=0x7fc768005020 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6836fa5541ff8eb6 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:49:54.355 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:54.350+0000 7fc756000640 10 client.0 ms_handle_connect on v2:172.21.15.32:3300/0 2024-08-03T20:49:54.355 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:54.350+0000 7fc756000640 1 -- 172.21.15.32:0/440148963 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc768005420 con 0x55da0b004270 2024-08-03T20:49:54.355 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:54.350+0000 7fc77ec61740 1 -- 172.21.15.32:0/440148963 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc768004a40 con 0x55da0b004270 2024-08-03T20:49:54.355 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:54.351+0000 7fc756000640 1 -- 172.21.15.32:0/440148963 <== mon.0 v2:172.21.15.32:3300/0 2 ==== osd_map(45..45 src has 1..45) ==== 5709+0+0 (secure 0 0 0) 0x7fc7680055c0 con 0x55da0b004270 2024-08-03T20:49:54.355 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:54.351+0000 7fc756000640 1 -- 172.21.15.32:0/440148963 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7fc76802f050 con 0x55da0b004270 2024-08-03T20:49:54.355 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:54.351+0000 7fc77ec61740 10 client.5547 fetch_fsmap learned FSMap version 35 2024-08-03T20:49:54.356 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:54.351+0000 7fc77ec61740 1 -- 172.21.15.32:0/440148963 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=2+}) -- 0x55da0b3ed5e0 con 0x55da0b004270 2024-08-03T20:49:54.356 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:54.351+0000 7fc756000640 1 -- 172.21.15.32:0/440148963 <== mon.0 v2:172.21.15.32:3300/0 4 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fc768004a40 con 0x55da0b004270 2024-08-03T20:49:54.356 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:54.351+0000 7fc756000640 1 -- 172.21.15.32:0/440148963 <== mon.0 v2:172.21.15.32:3300/0 5 ==== fsmap(e 35) ==== 3502+0+0 (secure 0 0 0) 0x7fc768032020 con 0x55da0b004270 2024-08-03T20:49:54.356 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:54.351+0000 7fc77ec61740 10 client.5547 fetch_fsmap finished waiting for FSMap version 35 2024-08-03T20:49:54.356 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:54.351+0000 7fc77ec61740 10 client.5547 resolve_mds: resolved name 'c' to daemon mds.c 2024-08-03T20:49:54.356 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:54.351+0000 7fc77ec61740 20 client.5547 populate_metadata read hostname 'smithi032' 2024-08-03T20:49:54.356 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:54.351+0000 7fc77ec61740 1 --2- 172.21.15.32:0/440148963 >> [v2:172.21.15.32:6834/678469885,v1:172.21.15.32:6835/678469885] conn(0x55da0afc2e40 0x55da0b3e59b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:49:54.356 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:54.351+0000 7fc77ec61740 4 client.5547 mds_command: new command op to 5424 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-08-03T20:49:54.356 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:54.351+0000 7fc77ec61740 1 -- 172.21.15.32:0/440148963 --> [v2:172.21.15.32:6834/678469885,v1:172.21.15.32:6835/678469885] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55da0b31e9b0 con 0x55da0afc2e40 2024-08-03T20:49:54.356 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:54.351+0000 7fc775200640 1 --2- 172.21.15.32:0/440148963 >> [v2:172.21.15.32:6834/678469885,v1:172.21.15.32:6835/678469885] conn(0x55da0afc2e40 0x55da0b3e59b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:49:54.356 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:54.352+0000 7fc775200640 1 --2- 172.21.15.32:0/440148963 >> [v2:172.21.15.32:6834/678469885,v1:172.21.15.32:6835/678469885] conn(0x55da0afc2e40 0x55da0b3e59b0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2e32ac4948d01c6f server_cookie=9f1dad7b2e5cc6f5 in_seq=0 out_seq=0 2024-08-03T20:49:54.356 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:54.352+0000 7fc756000640 10 client.5547 ms_handle_connect on v2:172.21.15.32:6834/678469885 2024-08-03T20:49:54.358 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:54.353+0000 7fc756000640 1 -- 172.21.15.32:0/440148963 <== mds.0 v2:172.21.15.32:6834/678469885 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55da0b31e9b0 con 0x55da0afc2e40 2024-08-03T20:49:54.358 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:54.353+0000 7fc756000640 10 client.5547 handle_command_reply: tid=0 2024-08-03T20:49:54.358 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:54.353+0000 7fc77ec61740 1 client.5547 shutdown 2024-08-03T20:49:54.358 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:54.353+0000 7fc77ec61740 1 -- 172.21.15.32:0/440148963 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55da0b004270 msgr2=0x55da0b3c2650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:49:54.358 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:54.353+0000 7fc77ec61740 1 --2- 172.21.15.32:0/440148963 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55da0b004270 0x55da0b3c2650 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7fc768016040 tx=0x7fc768005020 comp rx=0 tx=0).stop 2024-08-03T20:49:54.358 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:54.353+0000 7fc77ec61740 1 -- 172.21.15.32:0/440148963 shutdown_connections 2024-08-03T20:49:54.358 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:54.353+0000 7fc77ec61740 1 --2- 172.21.15.32:0/440148963 >> [v2:172.21.15.32:6834/678469885,v1:172.21.15.32:6835/678469885] conn(0x55da0afc2e40 0x55da0b3e59b0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:49:54.358 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:54.353+0000 7fc77ec61740 1 --2- 172.21.15.32:0/440148963 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55da0afe3590 0x55da0b3d0de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:49:54.358 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:54.353+0000 7fc77ec61740 1 --2- 172.21.15.32:0/440148963 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55da0b004270 0x55da0b3c2650 unknown :-1 s=CLOSED pgs=449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:49:54.359 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:54.353+0000 7fc77ec61740 1 --2- 172.21.15.32:0/440148963 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55da0afdb3b0 0x55da0b3ea960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:49:54.359 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:54.353+0000 7fc77ec61740 1 -- 172.21.15.32:0/440148963 >> 172.21.15.32:0/440148963 conn(0x55da0aff83c0 msgr2=0x55da0b3c4000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:49:54.359 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:54.353+0000 7fc756000640 0 client.5547 ms_handle_reset on v2:172.21.15.32:6834/678469885 2024-08-03T20:49:54.359 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:54.353+0000 7fc77ec61740 1 -- 172.21.15.32:0/440148963 shutdown_connections 2024-08-03T20:49:54.359 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:54.353+0000 7fc77ec61740 1 -- 172.21.15.32:0/440148963 wait complete. 2024-08-03T20:49:54.359 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:54.354+0000 7fc77ec61740 20 client.5547 trim_cache size 0 max 16384 2024-08-03T20:49:54.374 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:54.374+0000 7fc77ec61740 1 Processor -- start 2024-08-03T20:49:54.374 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:54.374+0000 7fc77ec61740 1 -- start start 2024-08-03T20:49:54.376 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:54.375+0000 7fc77ec61740 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55da0afdb3b0 0x55da0b3a93f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:49:54.376 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:54.375+0000 7fc77ec61740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55da0afe3590 0x55da0b410800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:49:54.376 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:54.375+0000 7fc77ec61740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55da0b004270 0x55da0b410d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:49:54.376 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:54.375+0000 7fc77ec61740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fc770200e40 con 0x55da0b004270 2024-08-03T20:49:54.376 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:54.375+0000 7fc77ec61740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7fc770150f00 con 0x55da0afe3590 2024-08-03T20:49:54.376 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:54.375+0000 7fc77ec61740 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7fc770151550 con 0x55da0afdb3b0 2024-08-03T20:49:54.376 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:54.375+0000 7fc775c00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55da0b004270 0x55da0b410d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:49:54.376 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:54.375+0000 7fc775c00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55da0b004270 0x55da0b410d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:38132/0 (socket says 172.21.15.32:38132) 2024-08-03T20:49:54.376 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:54.375+0000 7fc775c00640 1 -- 172.21.15.32:0/908278743 learned_addr learned my addr 172.21.15.32:0/908278743 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:49:54.378 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:54.376+0000 7fc775c00640 1 -- 172.21.15.32:0/908278743 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55da0afdb3b0 msgr2=0x55da0b3a93f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:49:54.378 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:54.376+0000 7fc775c00640 1 --2- 172.21.15.32:0/908278743 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55da0afdb3b0 0x55da0b3a93f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:49:54.378 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:54.376+0000 7fc775c00640 1 -- 172.21.15.32:0/908278743 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55da0afe3590 msgr2=0x55da0b410800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:49:54.378 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:54.376+0000 7fc775c00640 1 --2- 172.21.15.32:0/908278743 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55da0afe3590 0x55da0b410800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:49:54.378 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:54.376+0000 7fc775c00640 1 -- 172.21.15.32:0/908278743 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55da0b3c1cb0 con 0x55da0b004270 2024-08-03T20:49:54.378 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:54.376+0000 7fc775c00640 1 --2- 172.21.15.32:0/908278743 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55da0b004270 0x55da0b410d40 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7fc76800ea30 tx=0x7fc768005980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4300b2d139b5eebd server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:49:54.378 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:54.376+0000 7fc760c00640 1 -- 172.21.15.32:0/908278743 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7680011c0 con 0x55da0b004270 2024-08-03T20:49:54.378 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:54.376+0000 7fc760c00640 1 -- 172.21.15.32:0/908278743 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fc76800ee50 con 0x55da0b004270 2024-08-03T20:49:54.378 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:54.376+0000 7fc760c00640 1 -- 172.21.15.32:0/908278743 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc768005420 con 0x55da0b004270 2024-08-03T20:49:54.378 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:54.377+0000 7fc77ec61740 1 -- 172.21.15.32:0/908278743 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55da0b004270 msgr2=0x55da0b410d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:49:54.378 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:54.377+0000 7fc77ec61740 1 --2- 172.21.15.32:0/908278743 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55da0b004270 0x55da0b410d40 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7fc76800ea30 tx=0x7fc768005980 comp rx=0 tx=0).stop 2024-08-03T20:49:54.378 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:54.377+0000 7fc77ec61740 1 -- 172.21.15.32:0/908278743 shutdown_connections 2024-08-03T20:49:54.379 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:54.377+0000 7fc77ec61740 1 --2- 172.21.15.32:0/908278743 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55da0afe3590 0x55da0b410800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:49:54.379 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:54.377+0000 7fc77ec61740 1 --2- 172.21.15.32:0/908278743 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55da0b004270 0x55da0b410d40 unknown :-1 s=CLOSED pgs=450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:49:54.379 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:54.377+0000 7fc77ec61740 1 --2- 172.21.15.32:0/908278743 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55da0afdb3b0 0x55da0b3a93f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:49:54.379 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:54.377+0000 7fc77ec61740 1 -- 172.21.15.32:0/908278743 >> 172.21.15.32:0/908278743 conn(0x55da0aff83c0 msgr2=0x55da0b3f2660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:49:54.379 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:54.377+0000 7fc77ec61740 1 -- 172.21.15.32:0/908278743 shutdown_connections 2024-08-03T20:49:54.379 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:54.377+0000 7fc77ec61740 1 -- 172.21.15.32:0/908278743 wait complete. 2024-08-03T20:49:54.379 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:54.378+0000 7fc77ec61740 1 -- start start 2024-08-03T20:49:54.379 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:54.378+0000 7fc77ec61740 1 Processor -- start 2024-08-03T20:49:54.381 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:54.380+0000 7fc77ec61740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55da0afdb3b0 0x55da0b469450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:49:54.381 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:54.380+0000 7fc77ec61740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55da0afe3590 0x55da0b3ff6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:49:54.381 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:54.380+0000 7fc77ec61740 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55da0b004270 0x55da0b3ffc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:49:54.381 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:54.380+0000 7fc77ec61740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55da0b3eb920 con 0x55da0afe3590 2024-08-03T20:49:54.381 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:54.380+0000 7fc77ec61740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x55da0b3eb420 con 0x55da0afdb3b0 2024-08-03T20:49:54.381 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:54.380+0000 7fc77ec61740 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x55da0b3ecac0 con 0x55da0b004270 2024-08-03T20:49:54.382 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:54.380+0000 7fc77ec61740 10 client.0 _ll_register_callbacks cb 0x7fc768004c30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-03T20:49:54.382 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:54.380+0000 7fc76fe00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55da0afe3590 0x55da0b3ff6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:49:54.382 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:54.380+0000 7fc76fe00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55da0afe3590 0x55da0b3ff6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:38148/0 (socket says 172.21.15.32:38148) 2024-08-03T20:49:54.382 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:54.380+0000 7fc76fe00640 1 -- 172.21.15.32:0/1580986356 learned_addr learned my addr 172.21.15.32:0/1580986356 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:49:54.382 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:54.380+0000 7fc775c00640 1 --2- 172.21.15.32:0/1580986356 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55da0b004270 0x55da0b3ffc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:49:54.382 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:54.380+0000 7fc775200640 1 --2- 172.21.15.32:0/1580986356 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55da0afdb3b0 0x55da0b469450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:49:54.382 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:54.380+0000 7fc76fe00640 1 -- 172.21.15.32:0/1580986356 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55da0b004270 msgr2=0x55da0b3ffc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:49:54.382 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:54.380+0000 7fc76fe00640 1 --2- 172.21.15.32:0/1580986356 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55da0b004270 0x55da0b3ffc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:49:54.382 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:54.381+0000 7fc76fe00640 1 -- 172.21.15.32:0/1580986356 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55da0afdb3b0 msgr2=0x55da0b469450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:49:54.382 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:54.381+0000 7fc76fe00640 1 --2- 172.21.15.32:0/1580986356 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55da0afdb3b0 0x55da0b469450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:49:54.382 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:54.381+0000 7fc76fe00640 1 -- 172.21.15.32:0/1580986356 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55da0b3a6b20 con 0x55da0afe3590 2024-08-03T20:49:54.382 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:54.381+0000 7fc775200640 1 --2- 172.21.15.32:0/1580986356 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55da0afdb3b0 0x55da0b469450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-03T20:49:54.385 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:54.381+0000 7fc76fe00640 1 --2- 172.21.15.32:0/1580986356 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55da0afe3590 0x55da0b3ff6c0 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7fc75c007fd0 tx=0x7fc75c001080 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5bd3e533798b002d server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:49:54.385 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:54.381+0000 7fc756000640 10 client.0 ms_handle_connect on v2:172.21.15.32:3300/0 2024-08-03T20:49:54.385 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:54.381+0000 7fc756000640 1 -- 172.21.15.32:0/1580986356 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc75c029060 con 0x55da0afe3590 2024-08-03T20:49:54.385 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:54.381+0000 7fc756000640 1 -- 172.21.15.32:0/1580986356 <== mon.0 v2:172.21.15.32:3300/0 2 ==== osd_map(45..45 src has 1..45) ==== 5709+0+0 (secure 0 0 0) 0x7fc75c010040 con 0x55da0afe3590 2024-08-03T20:49:54.385 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:54.381+0000 7fc77ec61740 1 -- 172.21.15.32:0/1580986356 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc76800ee50 con 0x55da0afe3590 2024-08-03T20:49:54.385 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:54.381+0000 7fc756000640 1 -- 172.21.15.32:0/1580986356 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7fc75c003980 con 0x55da0afe3590 2024-08-03T20:49:54.385 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:54.381+0000 7fc77ec61740 10 client.5553 fetch_fsmap learned FSMap version 35 2024-08-03T20:49:54.385 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:54.381+0000 7fc77ec61740 1 -- 172.21.15.32:0/1580986356 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=2+}) -- 0x55da0b3ab150 con 0x55da0afe3590 2024-08-03T20:49:54.385 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:54.382+0000 7fc756000640 1 -- 172.21.15.32:0/1580986356 <== mon.0 v2:172.21.15.32:3300/0 4 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fc76800ee50 con 0x55da0afe3590 2024-08-03T20:49:54.385 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:54.382+0000 7fc756000640 1 -- 172.21.15.32:0/1580986356 <== mon.0 v2:172.21.15.32:3300/0 5 ==== fsmap(e 35) ==== 3502+0+0 (secure 0 0 0) 0x7fc75c003b20 con 0x55da0afe3590 2024-08-03T20:49:54.385 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:54.382+0000 7fc77ec61740 10 client.5553 fetch_fsmap finished waiting for FSMap version 35 2024-08-03T20:49:54.386 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:54.382+0000 7fc77ec61740 10 client.5553 resolve_mds: resolved name 'c' to daemon mds.c 2024-08-03T20:49:54.386 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:54.382+0000 7fc77ec61740 20 client.5553 populate_metadata read hostname 'smithi032' 2024-08-03T20:49:54.386 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:54.382+0000 7fc77ec61740 1 --2- 172.21.15.32:0/1580986356 >> [v2:172.21.15.32:6834/678469885,v1:172.21.15.32:6835/678469885] conn(0x55da0b46bfb0 0x55da0b46c3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:49:54.386 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:54.382+0000 7fc77ec61740 4 client.5553 mds_command: new command op to 5424 tid=0 multi_id=0 [{"prefix": "status"}] 2024-08-03T20:49:54.386 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:54.382+0000 7fc77ec61740 1 -- 172.21.15.32:0/1580986356 --> [v2:172.21.15.32:6834/678469885,v1:172.21.15.32:6835/678469885] -- command(tid 0: {"prefix": "status"}) -- 0x55da0b31a450 con 0x55da0b46bfb0 2024-08-03T20:49:54.386 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:54.382+0000 7fc775200640 1 --2- 172.21.15.32:0/1580986356 >> [v2:172.21.15.32:6834/678469885,v1:172.21.15.32:6835/678469885] conn(0x55da0b46bfb0 0x55da0b46c3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:49:54.386 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:54.382+0000 7fc775200640 1 --2- 172.21.15.32:0/1580986356 >> [v2:172.21.15.32:6834/678469885,v1:172.21.15.32:6835/678469885] conn(0x55da0b46bfb0 0x55da0b46c3c0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f0dfe52c33018bee server_cookie=6ebcfa0fb697f0d0 in_seq=0 out_seq=0 2024-08-03T20:49:54.386 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:54.382+0000 7fc756000640 10 client.5553 ms_handle_connect on v2:172.21.15.32:6834/678469885 2024-08-03T20:49:54.386 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:54.383+0000 7fc756000640 1 -- 172.21.15.32:0/1580986356 <== mds.0 v2:172.21.15.32:6834/678469885 1 ==== command_reply(tid 0: 0 ) ==== 8+0+331 (crc 0 0 0) 0x55da0b31a450 con 0x55da0b46bfb0 2024-08-03T20:49:54.386 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:54.383+0000 7fc756000640 10 client.5553 handle_command_reply: tid=0 2024-08-03T20:49:54.386 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:54.383+0000 7fc77ec61740 1 client.5553 shutdown 2024-08-03T20:49:54.386 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:54.383+0000 7fc77ec61740 1 -- 172.21.15.32:0/1580986356 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55da0afe3590 msgr2=0x55da0b3ff6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:49:54.387 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:54.383+0000 7fc77ec61740 1 --2- 172.21.15.32:0/1580986356 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55da0afe3590 0x55da0b3ff6c0 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7fc75c007fd0 tx=0x7fc75c001080 comp rx=0 tx=0).stop 2024-08-03T20:49:54.387 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:54.383+0000 7fc77ec61740 1 -- 172.21.15.32:0/1580986356 shutdown_connections 2024-08-03T20:49:54.387 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:54.383+0000 7fc77ec61740 1 --2- 172.21.15.32:0/1580986356 >> [v2:172.21.15.32:6834/678469885,v1:172.21.15.32:6835/678469885] conn(0x55da0b46bfb0 0x55da0b46c3c0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:49:54.387 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:54.383+0000 7fc77ec61740 1 --2- 172.21.15.32:0/1580986356 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55da0b004270 0x55da0b3ffc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:49:54.387 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:54.383+0000 7fc77ec61740 1 --2- 172.21.15.32:0/1580986356 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55da0afe3590 0x55da0b3ff6c0 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:49:54.387 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:54.383+0000 7fc77ec61740 1 --2- 172.21.15.32:0/1580986356 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55da0afdb3b0 0x55da0b469450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:49:54.387 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:54.383+0000 7fc756000640 0 client.5553 ms_handle_reset on v2:172.21.15.32:6834/678469885 2024-08-03T20:49:54.388 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:54.383+0000 7fc77ec61740 1 -- 172.21.15.32:0/1580986356 >> 172.21.15.32:0/1580986356 conn(0x55da0aff83c0 msgr2=0x55da0b3d3da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:49:54.388 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:54.383+0000 7fc77ec61740 1 -- 172.21.15.32:0/1580986356 shutdown_connections 2024-08-03T20:49:54.388 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:54.383+0000 7fc77ec61740 1 -- 172.21.15.32:0/1580986356 wait complete. 2024-08-03T20:49:54.388 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:54.383+0000 7fc77ec61740 20 client.5553 trim_cache size 0 max 16384 2024-08-03T20:49:54.388 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-08-03T20:49:54.388 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_fsid": "40ab141a-01d7-49b9-ae3c-2edec9876e95", 2024-08-03T20:49:54.388 INFO:teuthology.orchestra.run.smithi032.stdout: "whoami": 0, 2024-08-03T20:49:54.388 INFO:teuthology.orchestra.run.smithi032.stdout: "id": 5424, 2024-08-03T20:49:54.388 INFO:teuthology.orchestra.run.smithi032.stdout: "want_state": "up:active", 2024-08-03T20:49:54.388 INFO:teuthology.orchestra.run.smithi032.stdout: "state": "up:active", 2024-08-03T20:49:54.389 INFO:teuthology.orchestra.run.smithi032.stdout: "fs_name": "cephfs", 2024-08-03T20:49:54.389 INFO:teuthology.orchestra.run.smithi032.stdout: "rank_uptime": 6.8621723210000001, 2024-08-03T20:49:54.389 INFO:teuthology.orchestra.run.smithi032.stdout: "mdsmap_epoch": 35, 2024-08-03T20:49:54.389 INFO:teuthology.orchestra.run.smithi032.stdout: "osdmap_epoch": 45, 2024-08-03T20:49:54.389 INFO:teuthology.orchestra.run.smithi032.stdout: "osdmap_epoch_barrier": 45, 2024-08-03T20:49:54.389 INFO:teuthology.orchestra.run.smithi032.stdout: "uptime": 18.809889352999999 2024-08-03T20:49:54.389 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-08-03T20:49:54.390 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:54.384+0000 7fc777c00640 1 -- 172.21.15.32:0/1865697863 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fc750077010 msgr2=0x7fc750079520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:49:54.390 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:54.384+0000 7fc777c00640 1 --2- 172.21.15.32:0/1865697863 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fc750077010 0x7fc750079520 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fc7701f9bb0 tx=0x7fc75c009040 comp rx=0 tx=0).stop 2024-08-03T20:49:54.390 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:54.384+0000 7fc777c00640 1 -- 172.21.15.32:0/1865697863 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc77006e930 msgr2=0x7fc77015ca70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:49:54.390 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:54.384+0000 7fc777c00640 1 --2- 172.21.15.32:0/1865697863 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc77006e930 0x7fc77015ca70 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7fc76402ab90 tx=0x7fc764033ba0 comp rx=0 tx=0).stop 2024-08-03T20:49:54.390 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:54.384+0000 7fc777c00640 1 -- 172.21.15.32:0/1865697863 shutdown_connections 2024-08-03T20:49:54.390 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:54.384+0000 7fc777c00640 1 --2- 172.21.15.32:0/1865697863 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fc750077010 0x7fc750079520 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:49:54.390 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:54.384+0000 7fc777c00640 1 --2- 172.21.15.32:0/1865697863 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc770073e60 0x7fc7700b02b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:49:54.390 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:54.384+0000 7fc777c00640 1 --2- 172.21.15.32:0/1865697863 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fc77015fff0 0x7fc7700b07f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:49:54.390 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:54.384+0000 7fc777c00640 1 --2- 172.21.15.32:0/1865697863 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc77006e930 0x7fc77015ca70 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:49:54.390 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:54.384+0000 7fc777c00640 1 -- 172.21.15.32:0/1865697863 >> 172.21.15.32:0/1865697863 conn(0x7fc7700b1bd0 msgr2=0x7fc770074270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:49:54.391 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:54.384+0000 7fc777c00640 1 -- 172.21.15.32:0/1865697863 shutdown_connections 2024-08-03T20:49:54.391 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:54.384+0000 7fc777c00640 1 -- 172.21.15.32:0/1865697863 wait complete. 2024-08-03T20:49:54.411 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-08-03T20:49:54.640 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:54.639+0000 7f850aa00640 1 -- 172.21.15.32:0/250239110 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f850406e930 msgr2=0x7f8504144f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:49:54.641 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:54.639+0000 7f850aa00640 1 --2- 172.21.15.32:0/250239110 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f850406e930 0x7f8504144f90 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f85040afaf0 tx=0x7f84f402a4d0 comp rx=0 tx=0).stop 2024-08-03T20:49:54.641 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:54.639+0000 7f850aa00640 1 -- 172.21.15.32:0/250239110 shutdown_connections 2024-08-03T20:49:54.641 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:54.639+0000 7f850aa00640 1 --2- 172.21.15.32:0/250239110 >> v2:172.21.15.118:3300/0 conn(0x7f8504073e60 0x7f8504074270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:49:54.641 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:54.639+0000 7f850aa00640 1 --2- 172.21.15.32:0/250239110 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f850406e930 0x7f8504144f90 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:49:54.641 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:54.639+0000 7f850aa00640 1 -- 172.21.15.32:0/250239110 >> 172.21.15.32:0/250239110 conn(0x7f85040b1bd0 msgr2=0x7f85040b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:49:54.641 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:54.640+0000 7f850aa00640 1 -- 172.21.15.32:0/250239110 shutdown_connections 2024-08-03T20:49:54.641 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:54.640+0000 7f850aa00640 1 -- 172.21.15.32:0/250239110 wait complete. 2024-08-03T20:49:54.642 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:54.642+0000 7f850aa00640 1 Processor -- start 2024-08-03T20:49:54.642 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:54.642+0000 7f850aa00640 1 -- start start 2024-08-03T20:49:54.644 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:54.643+0000 7f850aa00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f850406e930 0x7f8504144200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:49:54.645 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:54.643+0000 7f850aa00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8504073e60 0x7f85041e7c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:49:54.645 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:54.643+0000 7f850aa00640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f85040747b0 0x7f850413c5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:49:54.645 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:54.644+0000 7f850aa00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f85040cf9c0 con 0x7f850406e930 2024-08-03T20:49:54.645 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:54.644+0000 7f850aa00640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f85040cf840 con 0x7f8504073e60 2024-08-03T20:49:54.645 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:54.644+0000 7f850aa00640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7f85040ccff0 con 0x7f85040747b0 2024-08-03T20:49:54.645 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:54.644+0000 7f8503e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f850406e930 0x7f8504144200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:49:54.645 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:54.644+0000 7f8503e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f850406e930 0x7f8504144200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:38160/0 (socket says 172.21.15.32:38160) 2024-08-03T20:49:54.645 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:54.644+0000 7f8503e00640 1 -- 172.21.15.32:0/1774117248 learned_addr learned my addr 172.21.15.32:0/1774117248 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:49:54.645 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:54.644+0000 7f8503400640 1 --2- 172.21.15.32:0/1774117248 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8504073e60 0x7f85041e7c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:49:54.645 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:54.644+0000 7f8508a00640 1 --2- 172.21.15.32:0/1774117248 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f85040747b0 0x7f850413c5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:49:54.645 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:54.644+0000 7f8503e00640 1 -- 172.21.15.32:0/1774117248 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f85040747b0 msgr2=0x7f850413c5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:49:54.645 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:54.644+0000 7f8503e00640 1 --2- 172.21.15.32:0/1774117248 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f85040747b0 0x7f850413c5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:49:54.645 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:54.644+0000 7f8503e00640 1 -- 172.21.15.32:0/1774117248 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8504073e60 msgr2=0x7f85041e7c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:49:54.646 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:54.644+0000 7f8503e00640 1 --2- 172.21.15.32:0/1774117248 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8504073e60 0x7f85041e7c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:49:54.646 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:54.644+0000 7f8503e00640 1 -- 172.21.15.32:0/1774117248 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f85041e6850 con 0x7f850406e930 2024-08-03T20:49:54.646 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:54.644+0000 7f8508a00640 1 --2- 172.21.15.32:0/1774117248 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f85040747b0 0x7f850413c5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-03T20:49:54.647 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:54.645+0000 7f8503e00640 1 --2- 172.21.15.32:0/1774117248 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f850406e930 0x7f8504144200 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f84f4034040 tx=0x7f84f4003910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:49:54.647 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:54.645+0000 7f84fbe00640 1 -- 172.21.15.32:0/1774117248 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84f403e030 con 0x7f850406e930 2024-08-03T20:49:54.647 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:54.645+0000 7f84fbe00640 1 -- 172.21.15.32:0/1774117248 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f84f4003e00 con 0x7f850406e930 2024-08-03T20:49:54.647 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:54.645+0000 7f84fbe00640 1 -- 172.21.15.32:0/1774117248 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84f4035040 con 0x7f850406e930 2024-08-03T20:49:54.647 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:54.645+0000 7f850aa00640 1 -- 172.21.15.32:0/1774117248 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f85041e6b30 con 0x7f850406e930 2024-08-03T20:49:54.647 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:54.645+0000 7f850aa00640 1 -- 172.21.15.32:0/1774117248 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f85041e70f0 con 0x7f850406e930 2024-08-03T20:49:54.647 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:54.646+0000 7f850aa00640 1 -- 172.21.15.32:0/1774117248 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f85040cd5e0 con 0x7f850406e930 2024-08-03T20:49:54.648 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:54.647+0000 7f84fbe00640 1 -- 172.21.15.32:0/1774117248 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f84f4041030 con 0x7f850406e930 2024-08-03T20:49:54.648 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:54.648+0000 7f84fbe00640 1 --2- 172.21.15.32:0/1774117248 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f84e4077010 0x7f84e4079520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:49:54.649 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:54.648+0000 7f84fbe00640 1 -- 172.21.15.32:0/1774117248 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(45..45 src has 1..45) ==== 5709+0+0 (secure 0 0 0) 0x7f84f40b8fe0 con 0x7f850406e930 2024-08-03T20:49:54.649 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:54.649+0000 7f8503400640 1 --2- 172.21.15.32:0/1774117248 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f84e4077010 0x7f84e4079520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:49:54.649 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:54.649+0000 7f8503400640 1 --2- 172.21.15.32:0/1774117248 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f84e4077010 0x7f84e4079520 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f85040703d0 tx=0x7f84f000b2e0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:49:54.653 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:54.653+0000 7f84fbe00640 1 -- 172.21.15.32:0/1774117248 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7f84f4081c30 con 0x7f850406e930 2024-08-03T20:49:54.901 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:54.900+0000 7f850aa00640 1 -- 172.21.15.32:0/1774117248 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f850406d9f0 con 0x7f850406e930 2024-08-03T20:49:54.901 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:54.901+0000 7f84fbe00640 1 -- 172.21.15.32:0/1774117248 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v35) ==== 53+0+73 (secure 0 0 0) 0x7f84f4086180 con 0x7f850406e930 2024-08-03T20:49:54.902 INFO:teuthology.orchestra.run.smithi032.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-08-03T20:49:54.904 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:54.904+0000 7f850aa00640 1 -- 172.21.15.32:0/1774117248 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f84e4077010 msgr2=0x7f84e4079520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:49:54.904 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:54.904+0000 7f850aa00640 1 --2- 172.21.15.32:0/1774117248 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f84e4077010 0x7f84e4079520 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f85040703d0 tx=0x7f84f000b2e0 comp rx=0 tx=0).stop 2024-08-03T20:49:54.905 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:54.904+0000 7f850aa00640 1 -- 172.21.15.32:0/1774117248 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f850406e930 msgr2=0x7f8504144200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:49:54.905 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:54.904+0000 7f850aa00640 1 --2- 172.21.15.32:0/1774117248 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f850406e930 0x7f8504144200 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f84f4034040 tx=0x7f84f4003910 comp rx=0 tx=0).stop 2024-08-03T20:49:54.905 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:54.904+0000 7f850aa00640 1 -- 172.21.15.32:0/1774117248 shutdown_connections 2024-08-03T20:49:54.905 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:54.904+0000 7f850aa00640 1 --2- 172.21.15.32:0/1774117248 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f84e4077010 0x7f84e4079520 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:49:54.905 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:54.904+0000 7f850aa00640 1 --2- 172.21.15.32:0/1774117248 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8504073e60 0x7f85041e7c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:49:54.905 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:54.904+0000 7f850aa00640 1 --2- 172.21.15.32:0/1774117248 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f85040747b0 0x7f850413c5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:49:54.905 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:54.904+0000 7f850aa00640 1 --2- 172.21.15.32:0/1774117248 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f850406e930 0x7f8504144200 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:49:54.905 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:54.904+0000 7f850aa00640 1 -- 172.21.15.32:0/1774117248 >> 172.21.15.32:0/1774117248 conn(0x7f85040b1bd0 msgr2=0x7f85040b3be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:49:54.905 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:54.904+0000 7f850aa00640 1 -- 172.21.15.32:0/1774117248 shutdown_connections 2024-08-03T20:49:54.905 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:54.904+0000 7f850aa00640 1 -- 172.21.15.32:0/1774117248 wait complete. 2024-08-03T20:49:54.928 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-08-03T20:49:54.928 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-08-03T20:49:54.928 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi032.front.sepia.ceph.com 2024-08-03T20:49:54.928 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-08-03T20:49:54.928 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-08-03T20:49:54.928 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-08-03T20:49:54.928 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2024-08-03T20:49:54.928 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-08-03T20:49:54.928 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-08-03T20:49:54.929 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-08-03T20:49:54.929 DEBUG:teuthology.orchestra.run.smithi032:> ip addr 2024-08-03T20:49:54.965 INFO:teuthology.orchestra.run.smithi032.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-08-03T20:49:54.965 INFO:teuthology.orchestra.run.smithi032.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-08-03T20:49:54.965 INFO:teuthology.orchestra.run.smithi032.stdout: inet 127.0.0.1/8 scope host lo 2024-08-03T20:49:54.965 INFO:teuthology.orchestra.run.smithi032.stdout: valid_lft forever preferred_lft forever 2024-08-03T20:49:54.965 INFO:teuthology.orchestra.run.smithi032.stdout: inet6 ::1/128 scope host 2024-08-03T20:49:54.965 INFO:teuthology.orchestra.run.smithi032.stdout: valid_lft forever preferred_lft forever 2024-08-03T20:49:54.966 INFO:teuthology.orchestra.run.smithi032.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-08-03T20:49:54.966 INFO:teuthology.orchestra.run.smithi032.stdout: link/ether 0c:c4:7a:6b:fd:0c brd ff:ff:ff:ff:ff:ff 2024-08-03T20:49:54.966 INFO:teuthology.orchestra.run.smithi032.stdout: altname enp7s0f0 2024-08-03T20:49:54.966 INFO:teuthology.orchestra.run.smithi032.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-08-03T20:49:54.966 INFO:teuthology.orchestra.run.smithi032.stdout: link/ether 0c:c4:7a:6b:fd:0d brd ff:ff:ff:ff:ff:ff 2024-08-03T20:49:54.966 INFO:teuthology.orchestra.run.smithi032.stdout: altname enp7s0f1 2024-08-03T20:49:54.966 INFO:teuthology.orchestra.run.smithi032.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-08-03T20:49:54.966 INFO:teuthology.orchestra.run.smithi032.stdout: link/ether ac:1f:6b:f8:26:02 brd ff:ff:ff:ff:ff:ff 2024-08-03T20:49:54.966 INFO:teuthology.orchestra.run.smithi032.stdout: altname enp1s0f0 2024-08-03T20:49:54.966 INFO:teuthology.orchestra.run.smithi032.stdout: inet 172.21.15.32/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-08-03T20:49:54.966 INFO:teuthology.orchestra.run.smithi032.stdout: valid_lft 42361sec preferred_lft 42361sec 2024-08-03T20:49:54.966 INFO:teuthology.orchestra.run.smithi032.stdout: inet6 fe80::ae1f:6bff:fef8:2602/64 scope link noprefixroute 2024-08-03T20:49:54.966 INFO:teuthology.orchestra.run.smithi032.stdout: valid_lft forever preferred_lft forever 2024-08-03T20:49:54.967 INFO:teuthology.orchestra.run.smithi032.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-08-03T20:49:54.967 INFO:teuthology.orchestra.run.smithi032.stdout: link/ether ac:1f:6b:f8:26:03 brd ff:ff:ff:ff:ff:ff 2024-08-03T20:49:54.967 INFO:teuthology.orchestra.run.smithi032.stdout: altname enp1s0f1 2024-08-03T20:49:54.967 INFO:teuthology.orchestra.run.smithi032.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-08-03T20:49:54.967 INFO:teuthology.orchestra.run.smithi032.stdout: link/ether 0a:23:2b:ca:2a:93 brd ff:ff:ff:ff:ff:ff 2024-08-03T20:49:54.967 INFO:teuthology.orchestra.run.smithi032.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-08-03T20:49:54.967 INFO:teuthology.orchestra.run.smithi032.stdout: valid_lft forever preferred_lft forever 2024-08-03T20:49:54.967 INFO:teuthology.orchestra.run.smithi032.stdout: inet6 fe80::d804:7cff:fed9:7ba4/64 scope link 2024-08-03T20:49:54.967 INFO:teuthology.orchestra.run.smithi032.stdout: valid_lft forever preferred_lft forever 2024-08-03T20:49:54.967 INFO:teuthology.orchestra.run.smithi032.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-08-03T20:49:54.967 INFO:teuthology.orchestra.run.smithi032.stdout: link/ether 22:86:28:30:4e:62 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-08-03T20:49:54.967 INFO:teuthology.orchestra.run.smithi032.stdout: inet6 fe80::2086:28ff:fe30:4e62/64 scope link 2024-08-03T20:49:54.967 INFO:teuthology.orchestra.run.smithi032.stdout: valid_lft forever preferred_lft forever 2024-08-03T20:49:54.967 INFO:teuthology.orchestra.run.smithi032.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-08-03T20:49:54.968 INFO:teuthology.orchestra.run.smithi032.stdout: link/ether 0a:23:2b:ca:2a:93 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-08-03T20:49:54.968 INFO:teuthology.orchestra.run.smithi032.stdout: inet6 fe80::823:2bff:feca:2a93/64 scope link 2024-08-03T20:49:54.968 INFO:teuthology.orchestra.run.smithi032.stdout: valid_lft forever preferred_lft forever 2024-08-03T20:49:54.968 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-08-03T20:49:54.968 DEBUG:teuthology.orchestra.run.smithi032:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-08-03T20:49:55.059 INFO:teuthology.orchestra.run.smithi032.stdout:1 2024-08-03T20:49:55.062 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-08-03T20:49:55.063 DEBUG:teuthology.orchestra.run.smithi032:> route 2024-08-03T20:49:55.102 INFO:teuthology.orchestra.run.smithi032.stdout:Kernel IP routing table 2024-08-03T20:49:55.102 INFO:teuthology.orchestra.run.smithi032.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-08-03T20:49:55.102 INFO:teuthology.orchestra.run.smithi032.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-08-03T20:49:55.102 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-08-03T20:49:55.102 INFO:teuthology.orchestra.run.smithi032.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-08-03T20:49:55.103 DEBUG:teuthology.orchestra.run.smithi032:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-08-03T20:49:55.103 DEBUG:teuthology.orchestra.run.smithi032:> set -e 2024-08-03T20:49:55.103 DEBUG:teuthology.orchestra.run.smithi032:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-08-03T20:49:55.103 DEBUG:teuthology.orchestra.run.smithi032:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-08-03T20:49:55.103 DEBUG:teuthology.orchestra.run.smithi032:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-08-03T20:49:55.103 DEBUG:teuthology.orchestra.run.smithi032:> ') 2024-08-03T20:49:55.221 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:55 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-08-03T20:49:55.292 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:55 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-08-03T20:49:55.303 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-08-03T20:49:55.303 DEBUG:teuthology.orchestra.run.smithi032:> ip netns list 2024-08-03T20:49:55.339 INFO:teuthology.orchestra.run.smithi032.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-08-03T20:49:55.339 INFO:teuthology.orchestra.run.smithi032.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-08-03T20:49:55.339 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-08-03T20:49:55.339 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-08-03T20:49:55.339 DEBUG:teuthology.orchestra.run.smithi032:> sudo ip link set brx.0 up 2024-08-03T20:49:55.435 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-08-03T20:49:55.435 DEBUG:teuthology.orchestra.run.smithi032:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-08-03T20:49:55.471 INFO:teuthology.orchestra.run.smithi032.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-08-03T20:49:55.471 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-08-03T20:49:55.471 DEBUG:teuthology.orchestra.run.smithi032:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-08-03T20:49:55.546 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2024-08-03T20:49:55.546 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-08-03T20:49:55.546 DEBUG:teuthology.orchestra.run.smithi032:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-08-03T20:49:55.745 INFO:teuthology.orchestra.run.smithi032.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-08-03T20:49:55.745 INFO:teuthology.orchestra.run.smithi032.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-08-03T20:49:55.745 INFO:teuthology.orchestra.run.smithi032.stdout:invalid new device string format 2024-08-03T20:49:55.765 INFO:teuthology.orchestra.run.smithi032.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-08-03T20:49:55.765 INFO:teuthology.orchestra.run.smithi032.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-08-03T20:49:55.766 INFO:teuthology.orchestra.run.smithi032.stdout:invalid new device string format 2024-08-03T20:49:55.766 INFO:teuthology.orchestra.run.smithi032.stdout:mount.ceph: resolved to: "172.21.15.32:3300,172.21.15.118:3300,172.21.15.118:3301" 2024-08-03T20:49:55.766 INFO:teuthology.orchestra.run.smithi032.stdout:mount.ceph: switching to using v1 address with old syntax 2024-08-03T20:49:55.796 INFO:teuthology.orchestra.run.smithi032.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-08-03T20:49:55.796 INFO:teuthology.orchestra.run.smithi032.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-08-03T20:49:55.796 INFO:teuthology.orchestra.run.smithi032.stdout:invalid new device string format 2024-08-03T20:49:55.796 INFO:teuthology.orchestra.run.smithi032.stdout:mount.ceph: resolved to: "172.21.15.32:3300,172.21.15.118:3300,172.21.15.118:3301" 2024-08-03T20:49:55.797 INFO:teuthology.orchestra.run.smithi032.stdout:mount.ceph: switching to using v1 address with old syntax 2024-08-03T20:49:55.797 INFO:teuthology.orchestra.run.smithi032.stdout:mount.ceph: resolved to: "172.21.15.32:6789,172.21.15.118:6789,172.21.15.118:6790" 2024-08-03T20:49:55.797 INFO:teuthology.orchestra.run.smithi032.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.32:6789,172.21.15.118:6789,172.21.15.118:6790:/ 2024-08-03T20:49:55.797 INFO:teuthology.orchestra.run.smithi032.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,key=0,fsid=40ab141a-01d7-49b9-ae3c-2edec9876e95" will pass to kernel 2024-08-03T20:49:55.798 INFO:teuthology.orchestra.run.smithi032.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2024-08-03T20:49:55.798 INFO:teuthology.orchestra.run.smithi032.stdout: You just mounted a file system that supports labels which does not 2024-08-03T20:49:55.798 INFO:teuthology.orchestra.run.smithi032.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-08-03T20:49:55.798 INFO:teuthology.orchestra.run.smithi032.stdout: applications will generate AVC messages and not be allowed access to 2024-08-03T20:49:55.798 INFO:teuthology.orchestra.run.smithi032.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-08-03T20:49:55.801 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-08-03T20:49:55.801 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-08-03T20:49:55.801 DEBUG:teuthology.orchestra.run.smithi032:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-08-03T20:49:55.857 DEBUG:teuthology.orchestra.run.smithi032:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-08-03T20:49:55.938 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:55 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-08-03T20:49:55.947 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:55 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-08-03T20:49:55.955 INFO:teuthology.orchestra.run.smithi032.stdout:client5404 2024-08-03T20:49:55.956 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-08-03T20:49:55.956 DEBUG:teuthology.orchestra.run.smithi032:> dd if=/proc/self/mounts of=/dev/stdout 2024-08-03T20:49:55.993 DEBUG:teuthology.orchestra.run.smithi032:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'mkdir -p a/b') 2024-08-03T20:49:56.113 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:56 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-08-03T20:49:56.124 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:56 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-08-03T20:49:56.133 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.3:0 flush journal 2024-08-03T20:49:56.364 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:56.363+0000 7ff4e4c00640 1 -- 172.21.15.32:0/3024422303 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff4e009ac30 msgr2=0x7ff4e009b040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:49:56.364 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:56.363+0000 7ff4e4c00640 1 --2- 172.21.15.32:0/3024422303 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff4e009ac30 0x7ff4e009b040 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7ff4e00b4860 tx=0x7ff4d002a4a0 comp rx=0 tx=0).stop 2024-08-03T20:49:56.364 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:56.363+0000 7ff4e4c00640 1 -- 172.21.15.32:0/3024422303 shutdown_connections 2024-08-03T20:49:56.364 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:56.363+0000 7ff4e4c00640 1 --2- 172.21.15.32:0/3024422303 >> v2:172.21.15.118:3300/0 conn(0x7ff4e00d00f0 0x7ff4e0166de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:49:56.364 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:56.363+0000 7ff4e4c00640 1 --2- 172.21.15.32:0/3024422303 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff4e009ac30 0x7ff4e009b040 unknown :-1 s=CLOSED pgs=455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:49:56.364 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:56.363+0000 7ff4e4c00640 1 -- 172.21.15.32:0/3024422303 >> 172.21.15.32:0/3024422303 conn(0x7ff4e00b3c00 msgr2=0x7ff4e00b4010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:49:56.364 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:56.363+0000 7ff4e4c00640 1 -- 172.21.15.32:0/3024422303 shutdown_connections 2024-08-03T20:49:56.364 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:56.363+0000 7ff4e4c00640 1 -- 172.21.15.32:0/3024422303 wait complete. 2024-08-03T20:49:56.366 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:56.365+0000 7ff4e4c00640 1 Processor -- start 2024-08-03T20:49:56.366 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:56.365+0000 7ff4e4c00640 1 -- start start 2024-08-03T20:49:56.369 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:56.367+0000 7ff4e4c00640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7ff4e009ac30 0x7ff4e01e5270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:49:56.369 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:56.367+0000 7ff4e4c00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ff4e00cfce0 0x7ff4e01e7090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:49:56.369 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:56.367+0000 7ff4e4c00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff4e00d00f0 0x7ff4e01f3080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:49:56.369 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:56.367+0000 7ff4e4c00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7ff4e00b7ba0 con 0x7ff4e00d00f0 2024-08-03T20:49:56.369 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:56.367+0000 7ff4e4c00640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7ff4e00b7a20 con 0x7ff4e00cfce0 2024-08-03T20:49:56.369 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:56.367+0000 7ff4e4c00640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7ff4e00b2ca0 con 0x7ff4e009ac30 2024-08-03T20:49:56.369 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:56.367+0000 7ff4de800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff4e00d00f0 0x7ff4e01f3080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:49:56.369 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:56.367+0000 7ff4de800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff4e00d00f0 0x7ff4e01f3080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:38182/0 (socket says 172.21.15.32:38182) 2024-08-03T20:49:56.369 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:56.367+0000 7ff4de800640 1 -- 172.21.15.32:0/2579017838 learned_addr learned my addr 172.21.15.32:0/2579017838 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:49:56.370 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:56.368+0000 7ff4dde00640 1 --2- 172.21.15.32:0/2579017838 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7ff4e009ac30 0x7ff4e01e5270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:49:56.370 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:56.368+0000 7ff4d7e00640 1 --2- 172.21.15.32:0/2579017838 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ff4e00cfce0 0x7ff4e01e7090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:49:56.370 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:56.368+0000 7ff4de800640 1 -- 172.21.15.32:0/2579017838 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7ff4e009ac30 msgr2=0x7ff4e01e5270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:49:56.370 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:56.368+0000 7ff4de800640 1 --2- 172.21.15.32:0/2579017838 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7ff4e009ac30 0x7ff4e01e5270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:49:56.370 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:56.368+0000 7ff4de800640 1 -- 172.21.15.32:0/2579017838 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ff4e00cfce0 msgr2=0x7ff4e01e7090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:49:56.370 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:56.368+0000 7ff4de800640 1 --2- 172.21.15.32:0/2579017838 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ff4e00cfce0 0x7ff4e01e7090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:49:56.370 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:56.368+0000 7ff4de800640 1 -- 172.21.15.32:0/2579017838 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff4e01f3b60 con 0x7ff4e00d00f0 2024-08-03T20:49:56.370 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:56.368+0000 7ff4d7e00640 1 --2- 172.21.15.32:0/2579017838 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ff4e00cfce0 0x7ff4e01e7090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-03T20:49:56.370 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:56.368+0000 7ff4de800640 1 --2- 172.21.15.32:0/2579017838 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff4e00d00f0 0x7ff4e01f3080 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7ff4d800eab0 tx=0x7ff4d800ef70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:49:56.370 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:56.368+0000 7ff4d6000640 1 -- 172.21.15.32:0/2579017838 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4d800cea0 con 0x7ff4e00d00f0 2024-08-03T20:49:56.370 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:56.368+0000 7ff4d6000640 1 -- 172.21.15.32:0/2579017838 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7ff4d8004b20 con 0x7ff4e00d00f0 2024-08-03T20:49:56.371 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:56.368+0000 7ff4e4c00640 1 -- 172.21.15.32:0/2579017838 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff4e01f3e40 con 0x7ff4e00d00f0 2024-08-03T20:49:56.371 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:56.368+0000 7ff4d6000640 1 -- 172.21.15.32:0/2579017838 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4d8005020 con 0x7ff4e00d00f0 2024-08-03T20:49:56.371 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:56.368+0000 7ff4e4c00640 1 -- 172.21.15.32:0/2579017838 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff4e01f4400 con 0x7ff4e00d00f0 2024-08-03T20:49:56.372 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:56.372+0000 7ff4d6000640 1 -- 172.21.15.32:0/2579017838 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7ff4d80051c0 con 0x7ff4e00d00f0 2024-08-03T20:49:56.374 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:56.373+0000 7ff4d6000640 1 --2- 172.21.15.32:0/2579017838 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7ff4bc077010 0x7ff4bc079520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:49:56.374 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:56.373+0000 7ff4d6000640 1 -- 172.21.15.32:0/2579017838 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(45..45 src has 1..45) ==== 5709+0+0 (secure 0 0 0) 0x7ff4d80995e0 con 0x7ff4e00d00f0 2024-08-03T20:49:56.374 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:56.373+0000 7ff4ebe02740 1 Processor -- start 2024-08-03T20:49:56.374 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:56.373+0000 7ff4ebe02740 1 -- start start 2024-08-03T20:49:56.374 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:56.373+0000 7ff4dde00640 1 --2- 172.21.15.32:0/2579017838 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7ff4bc077010 0x7ff4bc079520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:49:56.376 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:56.374+0000 7ff4dde00640 1 --2- 172.21.15.32:0/2579017838 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7ff4bc077010 0x7ff4bc079520 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7ff4d0034000 tx=0x7ff4d0004730 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:49:56.376 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:56.375+0000 7ff4ebe02740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x559a73aee390 0x559a73ebe900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:49:56.376 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:56.375+0000 7ff4ebe02740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x559a73ac54d0 0x559a73ebee40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:49:56.376 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:56.375+0000 7ff4ebe02740 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x559a73acd6b0 0x559a73edbd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:49:56.376 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:56.375+0000 7ff4ebe02740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x559a73e07470 con 0x559a73aee390 2024-08-03T20:49:56.376 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:56.375+0000 7ff4ebe02740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x559a73df6970 con 0x559a73ac54d0 2024-08-03T20:49:56.377 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:56.375+0000 7ff4ebe02740 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x559a73dfa9b0 con 0x559a73acd6b0 2024-08-03T20:49:56.377 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:56.375+0000 7ff4dde00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x559a73aee390 0x559a73ebe900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:49:56.377 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:56.375+0000 7ff4dde00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x559a73aee390 0x559a73ebe900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:38194/0 (socket says 172.21.15.32:38194) 2024-08-03T20:49:56.377 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:56.375+0000 7ff4dde00640 1 -- 172.21.15.32:0/1794404018 learned_addr learned my addr 172.21.15.32:0/1794404018 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:49:56.377 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:56.375+0000 7ff4d7e00640 1 --2- 172.21.15.32:0/1794404018 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x559a73ac54d0 0x559a73ebee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:49:56.377 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:56.375+0000 7ff4de800640 1 --2- 172.21.15.32:0/1794404018 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x559a73acd6b0 0x559a73edbd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:49:56.377 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:56.375+0000 7ff4dde00640 1 -- 172.21.15.32:0/1794404018 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x559a73acd6b0 msgr2=0x559a73edbd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:49:56.377 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:56.375+0000 7ff4dde00640 1 --2- 172.21.15.32:0/1794404018 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x559a73acd6b0 0x559a73edbd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:49:56.377 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:56.375+0000 7ff4dde00640 1 -- 172.21.15.32:0/1794404018 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x559a73ac54d0 msgr2=0x559a73ebee40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:49:56.377 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:56.375+0000 7ff4dde00640 1 --2- 172.21.15.32:0/1794404018 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x559a73ac54d0 0x559a73ebee40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:49:56.377 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:56.375+0000 7ff4dde00640 1 -- 172.21.15.32:0/1794404018 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559a73bd07d0 con 0x559a73aee390 2024-08-03T20:49:56.377 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:56.376+0000 7ff4dde00640 1 --2- 172.21.15.32:0/1794404018 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x559a73aee390 0x559a73ebe900 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7ff4d002aa90 tx=0x7ff4d003e070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=95abd391b0f14346 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:49:56.379 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:56.376+0000 7ff4ce600640 1 -- 172.21.15.32:0/1794404018 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4d0003d40 con 0x559a73aee390 2024-08-03T20:49:56.379 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:56.376+0000 7ff4ce600640 1 -- 172.21.15.32:0/1794404018 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7ff4d003c420 con 0x559a73aee390 2024-08-03T20:49:56.379 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:56.376+0000 7ff4ce600640 1 -- 172.21.15.32:0/1794404018 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4d003c850 con 0x559a73aee390 2024-08-03T20:49:56.379 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:56.377+0000 7ff4ebe02740 1 -- 172.21.15.32:0/1794404018 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x559a73aee390 msgr2=0x559a73ebe900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:49:56.379 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:56.377+0000 7ff4ebe02740 1 --2- 172.21.15.32:0/1794404018 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x559a73aee390 0x559a73ebe900 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7ff4d002aa90 tx=0x7ff4d003e070 comp rx=0 tx=0).stop 2024-08-03T20:49:56.379 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:56.377+0000 7ff4ebe02740 1 -- 172.21.15.32:0/1794404018 shutdown_connections 2024-08-03T20:49:56.379 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:56.377+0000 7ff4ebe02740 1 --2- 172.21.15.32:0/1794404018 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x559a73ac54d0 0x559a73ebee40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:49:56.379 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:56.377+0000 7ff4ebe02740 1 --2- 172.21.15.32:0/1794404018 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x559a73acd6b0 0x559a73edbd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:49:56.379 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:56.377+0000 7ff4ebe02740 1 --2- 172.21.15.32:0/1794404018 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x559a73aee390 0x559a73ebe900 unknown :-1 s=CLOSED pgs=457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:49:56.379 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:56.377+0000 7ff4ebe02740 1 -- 172.21.15.32:0/1794404018 >> 172.21.15.32:0/1794404018 conn(0x559a73ae24e0 msgr2=0x559a73e94bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:49:56.379 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:56.377+0000 7ff4ebe02740 1 -- 172.21.15.32:0/1794404018 shutdown_connections 2024-08-03T20:49:56.379 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:56.377+0000 7ff4ebe02740 1 -- 172.21.15.32:0/1794404018 wait complete. 2024-08-03T20:49:56.380 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:56.378+0000 7ff4ebe02740 1 -- start start 2024-08-03T20:49:56.380 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:56.378+0000 7ff4ebe02740 1 Processor -- start 2024-08-03T20:49:56.383 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:56.380+0000 7ff4ebe02740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x559a73ac54d0 0x559a73ea01b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:49:56.383 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:56.380+0000 7ff4ebe02740 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x559a73acd6b0 0x559a73ea2c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:49:56.383 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:56.380+0000 7ff4ebe02740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x559a73aee390 0x559a73ed29c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:49:56.383 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:56.380+0000 7ff4ebe02740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x559a73d2ec90 con 0x559a73aee390 2024-08-03T20:49:56.383 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:56.380+0000 7ff4ebe02740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x559a73d90fd0 con 0x559a73ac54d0 2024-08-03T20:49:56.383 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:56.380+0000 7ff4ebe02740 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x559a73ebf380 con 0x559a73acd6b0 2024-08-03T20:49:56.383 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:56.380+0000 7ff4ebe02740 10 client.0 _ll_register_callbacks cb 0x559a73dabe50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-03T20:49:56.383 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:56.380+0000 7ff4de800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x559a73aee390 0x559a73ed29c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:49:56.383 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:56.380+0000 7ff4de800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x559a73aee390 0x559a73ed29c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:38208/0 (socket says 172.21.15.32:38208) 2024-08-03T20:49:56.383 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:56.380+0000 7ff4de800640 1 -- 172.21.15.32:0/4084806469 learned_addr learned my addr 172.21.15.32:0/4084806469 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:49:56.384 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:56.380+0000 7ff4dde00640 1 --2- 172.21.15.32:0/4084806469 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x559a73ac54d0 0x559a73ea01b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:49:56.384 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:56.380+0000 7ff4d7e00640 1 --2- 172.21.15.32:0/4084806469 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x559a73acd6b0 0x559a73ea2c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:49:56.384 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:56.380+0000 7ff4de800640 1 -- 172.21.15.32:0/4084806469 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x559a73acd6b0 msgr2=0x559a73ea2c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:49:56.384 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:56.380+0000 7ff4de800640 1 --2- 172.21.15.32:0/4084806469 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x559a73acd6b0 0x559a73ea2c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:49:56.384 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:56.380+0000 7ff4de800640 1 -- 172.21.15.32:0/4084806469 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x559a73ac54d0 msgr2=0x559a73ea01b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:49:56.384 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:56.380+0000 7ff4de800640 1 --2- 172.21.15.32:0/4084806469 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x559a73ac54d0 0x559a73ea01b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:49:56.384 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:56.380+0000 7ff4de800640 1 -- 172.21.15.32:0/4084806469 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559a73b59970 con 0x559a73aee390 2024-08-03T20:49:56.384 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:56.381+0000 7ff4de800640 1 --2- 172.21.15.32:0/4084806469 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x559a73aee390 0x559a73ed29c0 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7ff4d805e0c0 tx=0x7ff4d8062590 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eabc18d09b15a0af server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:49:56.384 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:56.381+0000 7ff4c2a00640 10 client.0 ms_handle_connect on v2:172.21.15.32:3300/0 2024-08-03T20:49:56.384 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:56.381+0000 7ff4c2a00640 1 -- 172.21.15.32:0/4084806469 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4d8066640 con 0x559a73aee390 2024-08-03T20:49:56.384 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:56.381+0000 7ff4c2a00640 1 -- 172.21.15.32:0/4084806469 <== mon.0 v2:172.21.15.32:3300/0 2 ==== osd_map(45..45 src has 1..45) ==== 5709+0+0 (secure 0 0 0) 0x7ff4d80990c0 con 0x559a73aee390 2024-08-03T20:49:56.384 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:56.381+0000 7ff4ebe02740 1 -- 172.21.15.32:0/4084806469 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff4d003c420 con 0x559a73aee390 2024-08-03T20:49:56.384 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:56.381+0000 7ff4c2a00640 1 -- 172.21.15.32:0/4084806469 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7ff4d806b0f0 con 0x559a73aee390 2024-08-03T20:49:56.384 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:56.381+0000 7ff4ebe02740 10 client.5574 fetch_fsmap learned FSMap version 35 2024-08-03T20:49:56.385 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:56.381+0000 7ff4ebe02740 1 -- 172.21.15.32:0/4084806469 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=2+}) -- 0x559a73a9f270 con 0x559a73aee390 2024-08-03T20:49:56.385 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:56.381+0000 7ff4c2a00640 1 -- 172.21.15.32:0/4084806469 <== mon.0 v2:172.21.15.32:3300/0 4 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7ff4d003c420 con 0x559a73aee390 2024-08-03T20:49:56.385 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:56.382+0000 7ff4c2a00640 1 -- 172.21.15.32:0/4084806469 <== mon.0 v2:172.21.15.32:3300/0 5 ==== fsmap(e 35) ==== 3502+0+0 (secure 0 0 0) 0x7ff4d805d9b0 con 0x559a73aee390 2024-08-03T20:49:56.385 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:56.382+0000 7ff4ebe02740 10 client.5574 fetch_fsmap finished waiting for FSMap version 35 2024-08-03T20:49:56.385 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:56.382+0000 7ff4ebe02740 10 client.5574 resolve_mds: resolved 3:0 to role '3:0' aka daemon mds.c 2024-08-03T20:49:56.385 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:56.382+0000 7ff4ebe02740 20 client.5574 populate_metadata read hostname 'smithi032' 2024-08-03T20:49:56.385 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:56.382+0000 7ff4ebe02740 1 --2- 172.21.15.32:0/4084806469 >> [v2:172.21.15.32:6834/678469885,v1:172.21.15.32:6835/678469885] conn(0x559a73aacf60 0x559a73eee0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:49:56.385 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:56.382+0000 7ff4ebe02740 4 client.5574 mds_command: new command op to 5424 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-08-03T20:49:56.385 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:56.382+0000 7ff4ebe02740 1 -- 172.21.15.32:0/4084806469 --> [v2:172.21.15.32:6834/678469885,v1:172.21.15.32:6835/678469885] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x559a73e093c0 con 0x559a73aacf60 2024-08-03T20:49:56.385 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:56.382+0000 7ff4dde00640 1 --2- 172.21.15.32:0/4084806469 >> [v2:172.21.15.32:6834/678469885,v1:172.21.15.32:6835/678469885] conn(0x559a73aacf60 0x559a73eee0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:49:56.385 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:56.383+0000 7ff4dde00640 1 --2- 172.21.15.32:0/4084806469 >> [v2:172.21.15.32:6834/678469885,v1:172.21.15.32:6835/678469885] conn(0x559a73aacf60 0x559a73eee0a0 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b0b135efc1d629cf server_cookie=71db0aba7fd04671 in_seq=0 out_seq=0 2024-08-03T20:49:56.385 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:56.383+0000 7ff4c2a00640 10 client.5574 ms_handle_connect on v2:172.21.15.32:6834/678469885 2024-08-03T20:49:56.387 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:56.383+0000 7ff4c2a00640 1 -- 172.21.15.32:0/4084806469 <== mds.0 v2:172.21.15.32:6834/678469885 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x559a73e093c0 con 0x559a73aacf60 2024-08-03T20:49:56.387 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:56.383+0000 7ff4c2a00640 10 client.5574 handle_command_reply: tid=0 2024-08-03T20:49:56.387 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:56.383+0000 7ff4ebe02740 1 client.5574 shutdown 2024-08-03T20:49:56.387 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:56.384+0000 7ff4ebe02740 1 -- 172.21.15.32:0/4084806469 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x559a73aee390 msgr2=0x559a73ed29c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:49:56.387 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:56.384+0000 7ff4ebe02740 1 --2- 172.21.15.32:0/4084806469 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x559a73aee390 0x559a73ed29c0 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7ff4d805e0c0 tx=0x7ff4d8062590 comp rx=0 tx=0).stop 2024-08-03T20:49:56.387 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:56.384+0000 7ff4ebe02740 1 -- 172.21.15.32:0/4084806469 shutdown_connections 2024-08-03T20:49:56.387 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:56.384+0000 7ff4ebe02740 1 --2- 172.21.15.32:0/4084806469 >> [v2:172.21.15.32:6834/678469885,v1:172.21.15.32:6835/678469885] conn(0x559a73aacf60 0x559a73eee0a0 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:49:56.387 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:56.384+0000 7ff4ebe02740 1 --2- 172.21.15.32:0/4084806469 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x559a73aee390 0x559a73ed29c0 unknown :-1 s=CLOSED pgs=458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:49:56.387 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:56.384+0000 7ff4ebe02740 1 --2- 172.21.15.32:0/4084806469 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x559a73acd6b0 0x559a73ea2c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:49:56.387 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:56.384+0000 7ff4ebe02740 1 --2- 172.21.15.32:0/4084806469 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x559a73ac54d0 0x559a73ea01b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:49:56.387 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:56.384+0000 7ff4c2a00640 0 client.5574 ms_handle_reset on v2:172.21.15.32:6834/678469885 2024-08-03T20:49:56.388 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:56.384+0000 7ff4ebe02740 1 -- 172.21.15.32:0/4084806469 >> 172.21.15.32:0/4084806469 conn(0x559a73ae24e0 msgr2=0x559a73edb310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:49:56.388 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:56.384+0000 7ff4ebe02740 1 -- 172.21.15.32:0/4084806469 shutdown_connections 2024-08-03T20:49:56.388 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:56.384+0000 7ff4ebe02740 1 -- 172.21.15.32:0/4084806469 wait complete. 2024-08-03T20:49:56.388 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:56.384+0000 7ff4ebe02740 20 client.5574 trim_cache size 0 max 16384 2024-08-03T20:49:56.401 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:56.401+0000 7ff4ebe02740 1 Processor -- start 2024-08-03T20:49:56.401 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:56.401+0000 7ff4ebe02740 1 -- start start 2024-08-03T20:49:56.404 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:56.402+0000 7ff4ebe02740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x559a73ac54d0 0x559a73ebab30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:49:56.404 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:56.402+0000 7ff4ebe02740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x559a73acd6b0 0x559a73ebc590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:49:56.404 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:56.402+0000 7ff4ebe02740 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x559a73aee390 0x559a73ebcad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:49:56.404 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:56.402+0000 7ff4ebe02740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7ff4e01e0a00 con 0x559a73ac54d0 2024-08-03T20:49:56.404 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:56.402+0000 7ff4ebe02740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7ff4e006f9a0 con 0x559a73acd6b0 2024-08-03T20:49:56.404 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:56.402+0000 7ff4ebe02740 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7ff4e006ed00 con 0x559a73aee390 2024-08-03T20:49:56.404 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:56.402+0000 7ff4dde00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x559a73ac54d0 0x559a73ebab30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:49:56.404 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:56.402+0000 7ff4dde00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x559a73ac54d0 0x559a73ebab30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:38214/0 (socket says 172.21.15.32:38214) 2024-08-03T20:49:56.404 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:56.402+0000 7ff4dde00640 1 -- 172.21.15.32:0/1267766448 learned_addr learned my addr 172.21.15.32:0/1267766448 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:49:56.405 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:56.403+0000 7ff4de800640 1 --2- 172.21.15.32:0/1267766448 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x559a73aee390 0x559a73ebcad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:49:56.405 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:56.403+0000 7ff4dde00640 1 -- 172.21.15.32:0/1267766448 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x559a73aee390 msgr2=0x559a73ebcad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:49:56.405 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:56.403+0000 7ff4d7e00640 1 --2- 172.21.15.32:0/1267766448 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x559a73acd6b0 0x559a73ebc590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:49:56.405 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:56.403+0000 7ff4dde00640 1 --2- 172.21.15.32:0/1267766448 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x559a73aee390 0x559a73ebcad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:49:56.405 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:56.403+0000 7ff4dde00640 1 -- 172.21.15.32:0/1267766448 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x559a73acd6b0 msgr2=0x559a73ebc590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:49:56.405 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:56.403+0000 7ff4dde00640 1 --2- 172.21.15.32:0/1267766448 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x559a73acd6b0 0x559a73ebc590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:49:56.405 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:56.403+0000 7ff4dde00640 1 -- 172.21.15.32:0/1267766448 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559a73ef0d70 con 0x559a73ac54d0 2024-08-03T20:49:56.405 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:56.403+0000 7ff4de800640 1 --2- 172.21.15.32:0/1267766448 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x559a73aee390 0x559a73ebcad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-03T20:49:56.405 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:56.403+0000 7ff4d7e00640 1 --2- 172.21.15.32:0/1267766448 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x559a73acd6b0 0x559a73ebc590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-03T20:49:56.405 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:56.403+0000 7ff4dde00640 1 --2- 172.21.15.32:0/1267766448 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x559a73ac54d0 0x559a73ebab30 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7ff4d00095f0 tx=0x7ff4d0005f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cccc28b7abdd0e1a server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:49:56.405 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:56.403+0000 7ff4cdc00640 1 -- 172.21.15.32:0/1267766448 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4d005e420 con 0x559a73ac54d0 2024-08-03T20:49:56.405 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:56.403+0000 7ff4cdc00640 1 -- 172.21.15.32:0/1267766448 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7ff4d005e5c0 con 0x559a73ac54d0 2024-08-03T20:49:56.405 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:56.403+0000 7ff4cdc00640 1 -- 172.21.15.32:0/1267766448 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4d005e9f0 con 0x559a73ac54d0 2024-08-03T20:49:56.406 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:56.404+0000 7ff4ebe02740 1 -- 172.21.15.32:0/1267766448 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x559a73ac54d0 msgr2=0x559a73ebab30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:49:56.406 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:56.404+0000 7ff4ebe02740 1 --2- 172.21.15.32:0/1267766448 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x559a73ac54d0 0x559a73ebab30 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7ff4d00095f0 tx=0x7ff4d0005f90 comp rx=0 tx=0).stop 2024-08-03T20:49:56.407 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:56.404+0000 7ff4ebe02740 1 -- 172.21.15.32:0/1267766448 shutdown_connections 2024-08-03T20:49:56.407 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:56.404+0000 7ff4ebe02740 1 --2- 172.21.15.32:0/1267766448 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x559a73acd6b0 0x559a73ebc590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:49:56.407 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:56.404+0000 7ff4ebe02740 1 --2- 172.21.15.32:0/1267766448 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x559a73aee390 0x559a73ebcad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:49:56.407 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:56.404+0000 7ff4ebe02740 1 --2- 172.21.15.32:0/1267766448 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x559a73ac54d0 0x559a73ebab30 unknown :-1 s=CLOSED pgs=459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:49:56.407 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:56.404+0000 7ff4ebe02740 1 -- 172.21.15.32:0/1267766448 >> 172.21.15.32:0/1267766448 conn(0x559a73ae24e0 msgr2=0x559a73e95d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:49:56.407 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:56.404+0000 7ff4ebe02740 1 -- 172.21.15.32:0/1267766448 shutdown_connections 2024-08-03T20:49:56.407 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:56.404+0000 7ff4ebe02740 1 -- 172.21.15.32:0/1267766448 wait complete. 2024-08-03T20:49:56.407 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:56.405+0000 7ff4ebe02740 1 -- start start 2024-08-03T20:49:56.407 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:56.405+0000 7ff4ebe02740 1 Processor -- start 2024-08-03T20:49:56.410 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:56.407+0000 7ff4ebe02740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x559a73ac54d0 0x559a73edbc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:49:56.410 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:56.407+0000 7ff4ebe02740 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x559a73acd6b0 0x559a73f59a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:49:56.410 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:56.407+0000 7ff4ebe02740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x559a73aee390 0x559a73e9e800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:49:56.410 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:56.407+0000 7ff4ebe02740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x559a73ebb070 con 0x559a73aee390 2024-08-03T20:49:56.410 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:56.407+0000 7ff4ebe02740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7ff4e006f350 con 0x559a73ac54d0 2024-08-03T20:49:56.410 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:56.407+0000 7ff4ebe02740 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x559a73ebab50 con 0x559a73acd6b0 2024-08-03T20:49:56.410 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:56.407+0000 7ff4ebe02740 10 client.0 _ll_register_callbacks cb 0x7ff4d8061e80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-03T20:49:56.410 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:56.407+0000 7ff4de800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x559a73aee390 0x559a73e9e800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:49:56.410 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:56.407+0000 7ff4de800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x559a73aee390 0x559a73e9e800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:38230/0 (socket says 172.21.15.32:38230) 2024-08-03T20:49:56.411 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:56.407+0000 7ff4de800640 1 -- 172.21.15.32:0/2984773250 learned_addr learned my addr 172.21.15.32:0/2984773250 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:49:56.411 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:56.407+0000 7ff4dde00640 1 --2- 172.21.15.32:0/2984773250 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x559a73ac54d0 0x559a73edbc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:49:56.411 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:56.407+0000 7ff4d7e00640 1 --2- 172.21.15.32:0/2984773250 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x559a73acd6b0 0x559a73f59a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:49:56.411 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:56.408+0000 7ff4de800640 1 -- 172.21.15.32:0/2984773250 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x559a73acd6b0 msgr2=0x559a73f59a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:49:56.411 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:56.408+0000 7ff4de800640 1 --2- 172.21.15.32:0/2984773250 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x559a73acd6b0 0x559a73f59a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:49:56.411 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:56.408+0000 7ff4de800640 1 -- 172.21.15.32:0/2984773250 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x559a73ac54d0 msgr2=0x559a73edbc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:49:56.411 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:56.408+0000 7ff4de800640 1 --2- 172.21.15.32:0/2984773250 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x559a73ac54d0 0x559a73edbc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:49:56.411 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:56.408+0000 7ff4de800640 1 -- 172.21.15.32:0/2984773250 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559a73ebc630 con 0x559a73aee390 2024-08-03T20:49:56.411 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:56.408+0000 7ff4de800640 1 --2- 172.21.15.32:0/2984773250 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x559a73aee390 0x559a73e9e800 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7ff4d8004af0 tx=0x7ff4d80627c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fe4891d4a39dfd8a server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:49:56.411 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:56.408+0000 7ff4c2a00640 10 client.0 ms_handle_connect on v2:172.21.15.32:3300/0 2024-08-03T20:49:56.411 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:56.408+0000 7ff4c2a00640 1 -- 172.21.15.32:0/2984773250 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4d8073060 con 0x559a73aee390 2024-08-03T20:49:56.411 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:56.408+0000 7ff4c2a00640 1 -- 172.21.15.32:0/2984773250 <== mon.0 v2:172.21.15.32:3300/0 2 ==== osd_map(45..45 src has 1..45) ==== 5709+0+0 (secure 0 0 0) 0x7ff4d80992b0 con 0x559a73aee390 2024-08-03T20:49:56.411 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:56.408+0000 7ff4ebe02740 1 -- 172.21.15.32:0/2984773250 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff4d005e5c0 con 0x559a73aee390 2024-08-03T20:49:56.411 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:56.408+0000 7ff4c2a00640 1 -- 172.21.15.32:0/2984773250 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7ff4d8077120 con 0x559a73aee390 2024-08-03T20:49:56.412 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:56.408+0000 7ff4ebe02740 10 client.5580 fetch_fsmap learned FSMap version 35 2024-08-03T20:49:56.412 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:56.408+0000 7ff4ebe02740 1 -- 172.21.15.32:0/2984773250 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=2+}) -- 0x559a73ebd760 con 0x559a73aee390 2024-08-03T20:49:56.412 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:56.409+0000 7ff4c2a00640 1 -- 172.21.15.32:0/2984773250 <== mon.0 v2:172.21.15.32:3300/0 4 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7ff4d005e5c0 con 0x559a73aee390 2024-08-03T20:49:56.412 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:56.409+0000 7ff4c2a00640 1 -- 172.21.15.32:0/2984773250 <== mon.0 v2:172.21.15.32:3300/0 5 ==== fsmap(e 35) ==== 3502+0+0 (secure 0 0 0) 0x7ff4d80772c0 con 0x559a73aee390 2024-08-03T20:49:56.412 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:56.409+0000 7ff4ebe02740 10 client.5580 fetch_fsmap finished waiting for FSMap version 35 2024-08-03T20:49:56.412 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:56.409+0000 7ff4ebe02740 10 client.5580 resolve_mds: resolved 3:0 to role '3:0' aka daemon mds.c 2024-08-03T20:49:56.412 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:56.409+0000 7ff4ebe02740 20 client.5580 populate_metadata read hostname 'smithi032' 2024-08-03T20:49:56.412 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:56.409+0000 7ff4ebe02740 1 --2- 172.21.15.32:0/2984773250 >> [v2:172.21.15.32:6834/678469885,v1:172.21.15.32:6835/678469885] conn(0x559a73ebe990 0x559a73e9ed40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:49:56.412 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:56.409+0000 7ff4ebe02740 4 client.5580 mds_command: new command op to 5424 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-08-03T20:49:56.412 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:56.409+0000 7ff4ebe02740 1 -- 172.21.15.32:0/2984773250 --> [v2:172.21.15.32:6834/678469885,v1:172.21.15.32:6835/678469885] -- command(tid 0: {"prefix": "flush journal"}) -- 0x559a73e04e60 con 0x559a73ebe990 2024-08-03T20:49:56.412 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:56.409+0000 7ff4dde00640 1 --2- 172.21.15.32:0/2984773250 >> [v2:172.21.15.32:6834/678469885,v1:172.21.15.32:6835/678469885] conn(0x559a73ebe990 0x559a73e9ed40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:49:56.412 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:56.410+0000 7ff4dde00640 1 --2- 172.21.15.32:0/2984773250 >> [v2:172.21.15.32:6834/678469885,v1:172.21.15.32:6835/678469885] conn(0x559a73ebe990 0x559a73e9ed40 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3bfbf09fc2f62f14 server_cookie=d105bf255c7cd83c in_seq=0 out_seq=0 2024-08-03T20:49:56.412 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:56.410+0000 7ff4c2a00640 10 client.5580 ms_handle_connect on v2:172.21.15.32:6834/678469885 2024-08-03T20:49:56.421 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:56.420+0000 7ff4c2a00640 1 -- 172.21.15.32:0/2984773250 <== mds.0 v2:172.21.15.32:6834/678469885 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x559a73e04e60 con 0x559a73ebe990 2024-08-03T20:49:56.421 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:56.420+0000 7ff4c2a00640 10 client.5580 handle_command_reply: tid=0 2024-08-03T20:49:56.421 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:56.420+0000 7ff4ebe02740 1 client.5580 shutdown 2024-08-03T20:49:56.421 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:56.420+0000 7ff4ebe02740 1 -- 172.21.15.32:0/2984773250 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x559a73aee390 msgr2=0x559a73e9e800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:49:56.421 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:56.420+0000 7ff4ebe02740 1 --2- 172.21.15.32:0/2984773250 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x559a73aee390 0x559a73e9e800 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7ff4d8004af0 tx=0x7ff4d80627c0 comp rx=0 tx=0).stop 2024-08-03T20:49:56.421 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:56.420+0000 7ff4ebe02740 1 -- 172.21.15.32:0/2984773250 shutdown_connections 2024-08-03T20:49:56.421 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:56.420+0000 7ff4ebe02740 1 --2- 172.21.15.32:0/2984773250 >> [v2:172.21.15.32:6834/678469885,v1:172.21.15.32:6835/678469885] conn(0x559a73ebe990 0x559a73e9ed40 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:49:56.422 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:56.420+0000 7ff4ebe02740 1 --2- 172.21.15.32:0/2984773250 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x559a73aee390 0x559a73e9e800 unknown :-1 s=CLOSED pgs=460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:49:56.422 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:56.420+0000 7ff4ebe02740 1 --2- 172.21.15.32:0/2984773250 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x559a73acd6b0 0x559a73f59a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:49:56.422 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:56.420+0000 7ff4ebe02740 1 --2- 172.21.15.32:0/2984773250 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x559a73ac54d0 0x559a73edbc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:49:56.422 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:56.420+0000 7ff4ebe02740 1 -- 172.21.15.32:0/2984773250 >> 172.21.15.32:0/2984773250 conn(0x559a73ae24e0 msgr2=0x559a73ecf7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:49:56.422 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:56.420+0000 7ff4c2a00640 0 client.5580 ms_handle_reset on v2:172.21.15.32:6834/678469885 2024-08-03T20:49:56.422 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:56.420+0000 7ff4ebe02740 1 -- 172.21.15.32:0/2984773250 shutdown_connections 2024-08-03T20:49:56.422 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:56.420+0000 7ff4ebe02740 1 -- 172.21.15.32:0/2984773250 wait complete. 2024-08-03T20:49:56.422 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:56.421+0000 7ff4ebe02740 20 client.5580 trim_cache size 0 max 16384 2024-08-03T20:49:56.422 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-08-03T20:49:56.423 INFO:teuthology.orchestra.run.smithi032.stdout: "message": "", 2024-08-03T20:49:56.423 INFO:teuthology.orchestra.run.smithi032.stdout: "return_code": 0 2024-08-03T20:49:56.423 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-08-03T20:49:56.423 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:56.421+0000 7ff4e4c00640 1 -- 172.21.15.32:0/2579017838 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7ff4bc077010 msgr2=0x7ff4bc079520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:49:56.424 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:56.421+0000 7ff4e4c00640 1 --2- 172.21.15.32:0/2579017838 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7ff4bc077010 0x7ff4bc079520 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7ff4d0034000 tx=0x7ff4d0004730 comp rx=0 tx=0).stop 2024-08-03T20:49:56.424 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:56.421+0000 7ff4e4c00640 1 -- 172.21.15.32:0/2579017838 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff4e00d00f0 msgr2=0x7ff4e01f3080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:49:56.424 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:56.421+0000 7ff4e4c00640 1 --2- 172.21.15.32:0/2579017838 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff4e00d00f0 0x7ff4e01f3080 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7ff4d800eab0 tx=0x7ff4d800ef70 comp rx=0 tx=0).stop 2024-08-03T20:49:56.424 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:56.421+0000 7ff4e4c00640 1 -- 172.21.15.32:0/2579017838 shutdown_connections 2024-08-03T20:49:56.424 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:56.421+0000 7ff4e4c00640 1 --2- 172.21.15.32:0/2579017838 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7ff4bc077010 0x7ff4bc079520 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:49:56.424 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:56.421+0000 7ff4e4c00640 1 --2- 172.21.15.32:0/2579017838 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ff4e00cfce0 0x7ff4e01e7090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:49:56.424 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:56.421+0000 7ff4e4c00640 1 --2- 172.21.15.32:0/2579017838 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff4e00d00f0 0x7ff4e01f3080 unknown :-1 s=CLOSED pgs=456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:49:56.424 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:56.421+0000 7ff4e4c00640 1 --2- 172.21.15.32:0/2579017838 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7ff4e009ac30 0x7ff4e01e5270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:49:56.424 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:56.421+0000 7ff4e4c00640 1 -- 172.21.15.32:0/2579017838 >> 172.21.15.32:0/2579017838 conn(0x7ff4e00b3c00 msgr2=0x7ff4e00d0a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:49:56.424 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:56.422+0000 7ff4e4c00640 1 -- 172.21.15.32:0/2579017838 shutdown_connections 2024-08-03T20:49:56.424 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:56.422+0000 7ff4e4c00640 1 -- 172.21.15.32:0/2579017838 wait complete. 2024-08-03T20:49:56.446 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mds mds_abort_on_newly_corrupt_dentry False 2024-08-03T20:49:56.677 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:56.676+0000 7f6107200640 1 -- 172.21.15.32:0/1139984878 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6100160520 msgr2=0x7f61001450f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:49:56.677 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:56.676+0000 7f6107200640 1 --2- 172.21.15.32:0/1139984878 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6100160520 0x7f61001450f0 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7f60f8005fd0 tx=0x7f60f802a8b0 comp rx=0 tx=0).stop 2024-08-03T20:49:56.677 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:56.676+0000 7f6107200640 1 -- 172.21.15.32:0/1139984878 shutdown_connections 2024-08-03T20:49:56.677 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:56.676+0000 7f6107200640 1 --2- 172.21.15.32:0/1139984878 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6100160520 0x7f61001450f0 unknown :-1 s=CLOSED pgs=462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:49:56.677 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:56.676+0000 7f6107200640 1 --2- 172.21.15.32:0/1139984878 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f61000b4f10 0x7f610015ffe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:49:56.678 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:56.676+0000 7f6107200640 1 -- 172.21.15.32:0/1139984878 >> 172.21.15.32:0/1139984878 conn(0x7f61000b1bd0 msgr2=0x7f61000b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:49:56.678 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:56.676+0000 7f6107200640 1 -- 172.21.15.32:0/1139984878 shutdown_connections 2024-08-03T20:49:56.678 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:56.676+0000 7f6107200640 1 -- 172.21.15.32:0/1139984878 wait complete. 2024-08-03T20:49:56.679 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:56.679+0000 7f6107200640 1 Processor -- start 2024-08-03T20:49:56.679 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:56.679+0000 7f6107200640 1 -- start start 2024-08-03T20:49:56.682 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:56.680+0000 7f6107200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f61000b4f10 0x7f61001444f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:49:56.682 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:56.680+0000 7f6107200640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f61000cecc0 0x7f61001e9030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:49:56.682 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:56.680+0000 7f6107200640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f6100160520 0x7f610015dbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:49:56.683 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:56.680+0000 7f6107200640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f6100145a80 con 0x7f61000b4f10 2024-08-03T20:49:56.683 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:56.680+0000 7f6107200640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f6100145900 con 0x7f61000cecc0 2024-08-03T20:49:56.683 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:56.680+0000 7f6107200640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7f6100145c00 con 0x7f6100160520 2024-08-03T20:49:56.683 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:56.680+0000 7f60ffe00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f61000b4f10 0x7f61001444f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:49:56.683 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:56.680+0000 7f60ffe00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f61000b4f10 0x7f61001444f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:38248/0 (socket says 172.21.15.32:38248) 2024-08-03T20:49:56.683 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:56.681+0000 7f60ffe00640 1 -- 172.21.15.32:0/3280163172 learned_addr learned my addr 172.21.15.32:0/3280163172 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:49:56.683 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:56.681+0000 7f60f7e00640 1 --2- 172.21.15.32:0/3280163172 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f61000cecc0 0x7f61001e9030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:49:56.683 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:56.681+0000 7f6105200640 1 --2- 172.21.15.32:0/3280163172 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f6100160520 0x7f610015dbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:49:56.683 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:56.681+0000 7f60ffe00640 1 -- 172.21.15.32:0/3280163172 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f6100160520 msgr2=0x7f610015dbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:49:56.683 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:56.681+0000 7f60ffe00640 1 --2- 172.21.15.32:0/3280163172 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f6100160520 0x7f610015dbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:49:56.683 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:56.681+0000 7f60ffe00640 1 -- 172.21.15.32:0/3280163172 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f61000cecc0 msgr2=0x7f61001e9030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:49:56.684 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:56.681+0000 7f60ffe00640 1 --2- 172.21.15.32:0/3280163172 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f61000cecc0 0x7f61001e9030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:49:56.684 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:56.681+0000 7f60ffe00640 1 -- 172.21.15.32:0/3280163172 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6100157040 con 0x7f61000b4f10 2024-08-03T20:49:56.684 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:56.681+0000 7f60f7e00640 1 --2- 172.21.15.32:0/3280163172 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f61000cecc0 0x7f61001e9030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-03T20:49:56.684 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:56.681+0000 7f6105200640 1 --2- 172.21.15.32:0/3280163172 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f6100160520 0x7f610015dbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-03T20:49:56.684 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:56.681+0000 7f60ffe00640 1 --2- 172.21.15.32:0/3280163172 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f61000b4f10 0x7f61001444f0 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7f60f000c3d0 tx=0x7f60f000c890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:49:56.684 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:56.681+0000 7f60fcc00640 1 -- 172.21.15.32:0/3280163172 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60f0011790 con 0x7f61000b4f10 2024-08-03T20:49:56.684 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:56.681+0000 7f60fcc00640 1 -- 172.21.15.32:0/3280163172 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f60f0017070 con 0x7f61000b4f10 2024-08-03T20:49:56.684 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:56.681+0000 7f6107200640 1 -- 172.21.15.32:0/3280163172 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6100157320 con 0x7f61000b4f10 2024-08-03T20:49:56.684 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:56.682+0000 7f60fcc00640 1 -- 172.21.15.32:0/3280163172 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60f0011cc0 con 0x7f61000b4f10 2024-08-03T20:49:56.684 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:56.682+0000 7f6107200640 1 -- 172.21.15.32:0/3280163172 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f61001578e0 con 0x7f61000b4f10 2024-08-03T20:49:56.685 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:56.683+0000 7f6107200640 1 -- 172.21.15.32:0/3280163172 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f610015ffe0 con 0x7f61000b4f10 2024-08-03T20:49:56.685 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:56.684+0000 7f60fcc00640 1 -- 172.21.15.32:0/3280163172 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f60f001c460 con 0x7f61000b4f10 2024-08-03T20:49:56.685 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:56.685+0000 7f60fcc00640 1 --2- 172.21.15.32:0/3280163172 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f60dc077010 0x7f60dc079520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:49:56.685 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:56.685+0000 7f60fcc00640 1 -- 172.21.15.32:0/3280163172 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(45..45 src has 1..45) ==== 5709+0+0 (secure 0 0 0) 0x7f60f0098740 con 0x7f61000b4f10 2024-08-03T20:49:56.686 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:56.685+0000 7f60f7e00640 1 --2- 172.21.15.32:0/3280163172 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f60dc077010 0x7f60dc079520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:49:56.686 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:56.686+0000 7f60f7e00640 1 --2- 172.21.15.32:0/3280163172 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f60dc077010 0x7f60dc079520 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f610015e770 tx=0x7f60e8005f90 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:49:56.689 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:56.689+0000 7f60fcc00640 1 -- 172.21.15.32:0/3280163172 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7f6100160930 con 0x7f61000b4f10 2024-08-03T20:49:56.894 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:56.893+0000 7f6107200640 1 -- 172.21.15.32:0/3280163172 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command([{prefix=config set, name=mds_abort_on_newly_corrupt_dentry}] v 0) -- 0x7f6100145410 con 0x7f61000b4f10 2024-08-03T20:49:56.898 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:56.897+0000 7f60fcc00640 1 -- 172.21.15.32:0/3280163172 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mds_abort_on_newly_corrupt_dentry}]=0 v31)=0 v31) ==== 135+0+0 (secure 0 0 0) 0x7f60f0061930 con 0x7f61000b4f10 2024-08-03T20:49:56.901 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:56.900+0000 7f6107200640 1 -- 172.21.15.32:0/3280163172 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f60dc077010 msgr2=0x7f60dc079520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:49:56.901 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:56.900+0000 7f6107200640 1 --2- 172.21.15.32:0/3280163172 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f60dc077010 0x7f60dc079520 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f610015e770 tx=0x7f60e8005f90 comp rx=0 tx=0).stop 2024-08-03T20:49:56.901 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:56.900+0000 7f6107200640 1 -- 172.21.15.32:0/3280163172 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f61000b4f10 msgr2=0x7f61001444f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:49:56.901 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:56.900+0000 7f6107200640 1 --2- 172.21.15.32:0/3280163172 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f61000b4f10 0x7f61001444f0 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7f60f000c3d0 tx=0x7f60f000c890 comp rx=0 tx=0).stop 2024-08-03T20:49:56.901 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:56.900+0000 7f6107200640 1 -- 172.21.15.32:0/3280163172 shutdown_connections 2024-08-03T20:49:56.901 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:56.900+0000 7f6107200640 1 --2- 172.21.15.32:0/3280163172 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f60dc077010 0x7f60dc079520 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:49:56.901 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:56.900+0000 7f6107200640 1 --2- 172.21.15.32:0/3280163172 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f61000cecc0 0x7f61001e9030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:49:56.901 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:56.900+0000 7f6107200640 1 --2- 172.21.15.32:0/3280163172 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f6100160520 0x7f610015dbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:49:56.901 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:56.900+0000 7f6107200640 1 --2- 172.21.15.32:0/3280163172 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f61000b4f10 0x7f61001444f0 unknown :-1 s=CLOSED pgs=463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:49:56.901 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:56.900+0000 7f6107200640 1 -- 172.21.15.32:0/3280163172 >> 172.21.15.32:0/3280163172 conn(0x7f61000b1bd0 msgr2=0x7f610015ab60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:49:56.901 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:56.900+0000 7f6107200640 1 -- 172.21.15.32:0/3280163172 shutdown_connections 2024-08-03T20:49:56.901 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:56.900+0000 7f6107200640 1 -- 172.21.15.32:0/3280163172 wait complete. 2024-08-03T20:49:56.924 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mds mds_inject_rename_corrupt_dentry_first 1.0 2024-08-03T20:49:57.156 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:57.154+0000 7fc960800640 1 -- 172.21.15.32:0/446448550 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc9580cccf0 msgr2=0x7fc9580cd100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:49:57.156 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:57.154+0000 7fc960800640 1 --2- 172.21.15.32:0/446448550 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc9580cccf0 0x7fc9580cd100 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7fc948009830 tx=0x7fc94802a500 comp rx=0 tx=0).stop 2024-08-03T20:49:57.156 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:57.155+0000 7fc960800640 1 -- 172.21.15.32:0/446448550 shutdown_connections 2024-08-03T20:49:57.156 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:57.155+0000 7fc960800640 1 --2- 172.21.15.32:0/446448550 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fc9580ced30 0x7fc9580cf140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:49:57.156 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:57.155+0000 7fc960800640 1 --2- 172.21.15.32:0/446448550 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc9580cccf0 0x7fc9580cd100 unknown :-1 s=CLOSED pgs=465 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:49:57.156 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:57.155+0000 7fc960800640 1 -- 172.21.15.32:0/446448550 >> 172.21.15.32:0/446448550 conn(0x7fc9580b1bd0 msgr2=0x7fc9580b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:49:57.156 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:57.155+0000 7fc960800640 1 -- 172.21.15.32:0/446448550 shutdown_connections 2024-08-03T20:49:57.156 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:57.155+0000 7fc960800640 1 -- 172.21.15.32:0/446448550 wait complete. 2024-08-03T20:49:57.157 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:57.157+0000 7fc960800640 1 Processor -- start 2024-08-03T20:49:57.157 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:57.157+0000 7fc960800640 1 -- start start 2024-08-03T20:49:57.160 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:57.158+0000 7fc960800640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc9580cccf0 0x7fc958159940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:49:57.160 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:57.158+0000 7fc960800640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fc9580ced30 0x7fc958073de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:49:57.160 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:57.158+0000 7fc960800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc958145c50 0x7fc958074320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:49:57.160 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:57.158+0000 7fc960800640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fc9580b4070 con 0x7fc958145c50 2024-08-03T20:49:57.161 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:57.158+0000 7fc960800640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7fc9580b3ef0 con 0x7fc9580cccf0 2024-08-03T20:49:57.161 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:57.158+0000 7fc960800640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7fc9580b41f0 con 0x7fc9580ced30 2024-08-03T20:49:57.161 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:57.159+0000 7fc95e800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc958145c50 0x7fc958074320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:49:57.161 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:57.159+0000 7fc95e800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc958145c50 0x7fc958074320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:38254/0 (socket says 172.21.15.32:38254) 2024-08-03T20:49:57.161 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:57.159+0000 7fc95e800640 1 -- 172.21.15.32:0/2002793656 learned_addr learned my addr 172.21.15.32:0/2002793656 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:49:57.161 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:57.159+0000 7fc95d400640 1 --2- 172.21.15.32:0/2002793656 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fc9580ced30 0x7fc958073de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:49:57.161 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:57.159+0000 7fc95de00640 1 --2- 172.21.15.32:0/2002793656 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc9580cccf0 0x7fc958159940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:49:57.161 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:57.159+0000 7fc95e800640 1 -- 172.21.15.32:0/2002793656 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fc9580ced30 msgr2=0x7fc958073de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:49:57.161 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:57.159+0000 7fc95e800640 1 --2- 172.21.15.32:0/2002793656 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fc9580ced30 0x7fc958073de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:49:57.161 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:57.159+0000 7fc95e800640 1 -- 172.21.15.32:0/2002793656 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc9580cccf0 msgr2=0x7fc958159940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:49:57.161 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:57.159+0000 7fc95e800640 1 --2- 172.21.15.32:0/2002793656 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc9580cccf0 0x7fc958159940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:49:57.162 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:57.159+0000 7fc95e800640 1 -- 172.21.15.32:0/2002793656 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc9581f6690 con 0x7fc958145c50 2024-08-03T20:49:57.162 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:57.159+0000 7fc95e800640 1 --2- 172.21.15.32:0/2002793656 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc958145c50 0x7fc958074320 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7fc95400e960 tx=0x7fc95400ee20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:49:57.162 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:57.160+0000 7fc952000640 1 -- 172.21.15.32:0/2002793656 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc954004d20 con 0x7fc958145c50 2024-08-03T20:49:57.162 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:57.160+0000 7fc952000640 1 -- 172.21.15.32:0/2002793656 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fc95400cac0 con 0x7fc958145c50 2024-08-03T20:49:57.162 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:57.160+0000 7fc952000640 1 -- 172.21.15.32:0/2002793656 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc954005020 con 0x7fc958145c50 2024-08-03T20:49:57.162 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:57.160+0000 7fc960800640 1 -- 172.21.15.32:0/2002793656 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc9581f5b10 con 0x7fc958145c50 2024-08-03T20:49:57.162 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:57.160+0000 7fc960800640 1 -- 172.21.15.32:0/2002793656 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc9581f60d0 con 0x7fc958145c50 2024-08-03T20:49:57.163 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:57.161+0000 7fc960800640 1 -- 172.21.15.32:0/2002793656 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc9580b46e0 con 0x7fc958145c50 2024-08-03T20:49:57.163 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:57.162+0000 7fc952000640 1 -- 172.21.15.32:0/2002793656 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7fc9540051c0 con 0x7fc958145c50 2024-08-03T20:49:57.163 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:57.163+0000 7fc952000640 1 --2- 172.21.15.32:0/2002793656 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fc93c077010 0x7fc93c079520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:49:57.163 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:57.163+0000 7fc952000640 1 -- 172.21.15.32:0/2002793656 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(45..45 src has 1..45) ==== 5709+0+0 (secure 0 0 0) 0x7fc9540990f0 con 0x7fc958145c50 2024-08-03T20:49:57.164 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:57.164+0000 7fc95de00640 1 --2- 172.21.15.32:0/2002793656 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fc93c077010 0x7fc93c079520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:49:57.165 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:57.164+0000 7fc95de00640 1 --2- 172.21.15.32:0/2002793656 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fc93c077010 0x7fc93c079520 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fc94c001610 tx=0x7fc94c009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:49:57.168 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:57.167+0000 7fc952000640 1 -- 172.21.15.32:0/2002793656 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7fc954016030 con 0x7fc958145c50 2024-08-03T20:49:57.369 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:57.368+0000 7fc960800640 1 -- 172.21.15.32:0/2002793656 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command([{prefix=config set, name=mds_inject_rename_corrupt_dentry_first}] v 0) -- 0x7fc958143070 con 0x7fc958145c50 2024-08-03T20:49:57.373 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:57.373+0000 7fc952000640 1 -- 172.21.15.32:0/2002793656 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mds_inject_rename_corrupt_dentry_first}]=0 v32)=0 v32) ==== 138+0+0 (secure 0 0 0) 0x7fc954062370 con 0x7fc958145c50 2024-08-03T20:49:57.376 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:57.375+0000 7fc960800640 1 -- 172.21.15.32:0/2002793656 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fc93c077010 msgr2=0x7fc93c079520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:49:57.376 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:57.375+0000 7fc960800640 1 --2- 172.21.15.32:0/2002793656 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fc93c077010 0x7fc93c079520 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fc94c001610 tx=0x7fc94c009040 comp rx=0 tx=0).stop 2024-08-03T20:49:57.377 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:57.375+0000 7fc960800640 1 -- 172.21.15.32:0/2002793656 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc958145c50 msgr2=0x7fc958074320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:49:57.377 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:57.375+0000 7fc960800640 1 --2- 172.21.15.32:0/2002793656 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc958145c50 0x7fc958074320 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7fc95400e960 tx=0x7fc95400ee20 comp rx=0 tx=0).stop 2024-08-03T20:49:57.377 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:57.376+0000 7fc960800640 1 -- 172.21.15.32:0/2002793656 shutdown_connections 2024-08-03T20:49:57.377 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:57.376+0000 7fc960800640 1 --2- 172.21.15.32:0/2002793656 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fc93c077010 0x7fc93c079520 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:49:57.377 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:57.376+0000 7fc960800640 1 --2- 172.21.15.32:0/2002793656 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc958145c50 0x7fc958074320 unknown :-1 s=CLOSED pgs=466 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:49:57.377 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:57.376+0000 7fc960800640 1 --2- 172.21.15.32:0/2002793656 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fc9580ced30 0x7fc958073de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:49:57.377 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:57.376+0000 7fc960800640 1 --2- 172.21.15.32:0/2002793656 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc9580cccf0 0x7fc958159940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:49:57.377 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:57.376+0000 7fc960800640 1 -- 172.21.15.32:0/2002793656 >> 172.21.15.32:0/2002793656 conn(0x7fc9580b1bd0 msgr2=0x7fc9580cff10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:49:57.377 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:57.376+0000 7fc960800640 1 -- 172.21.15.32:0/2002793656 shutdown_connections 2024-08-03T20:49:57.377 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:49:57.376+0000 7fc960800640 1 -- 172.21.15.32:0/2002793656 wait complete. 2024-08-03T20:50:02.401 DEBUG:teuthology.orchestra.run.smithi032:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'mv a/b a/c; sync .') 2024-08-03T20:50:02.491 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:02 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-08-03T20:50:02.495 INFO:tasks.ceph.mds.c.smithi032.stderr:2024-08-03T20:50:02.494+0000 7f7393600640 -1 mds.0.cache.den(0x10000000000 c) newly corrupt dentry to be committed: [dentry #0x1/a/c [fffffffffffffff6,head] auth NULL (dn xlock x=1 by request(client.5404:44 nref=7 cr=0x558800833180)) (dversion lock w=1 last_client=5404) pv=4 v=3 ap=2 ino=(nil) state=1073741824 | request=1 lock=2 authpin=1 0x5588008b1680] 2024-08-03T20:50:02.498 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:02 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-08-03T20:50:02.507 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-08-03T20:50:02.507 DEBUG:teuthology.orchestra.run.smithi032:> dd if=/proc/self/mounts of=/dev/stdout 2024-08-03T20:50:02.544 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-08-03T20:50:02.818 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:02.816+0000 7fdb01e00640 1 -- 172.21.15.32:0/4017593428 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fdafc06f720 msgr2=0x7fdafc06fb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:50:02.818 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:02.816+0000 7fdb01e00640 1 --2- 172.21.15.32:0/4017593428 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fdafc06f720 0x7fdafc06fb30 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fdaec009830 tx=0x7fdaec02a4a0 comp rx=0 tx=0).stop 2024-08-03T20:50:02.818 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:02.816+0000 7fdb01e00640 1 -- 172.21.15.32:0/4017593428 shutdown_connections 2024-08-03T20:50:02.818 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:02.816+0000 7fdb01e00640 1 --2- 172.21.15.32:0/4017593428 >> v2:172.21.15.118:3300/0 conn(0x7fdafc0b39a0 0x7fdafc0b3db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:02.818 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:02.816+0000 7fdb01e00640 1 --2- 172.21.15.32:0/4017593428 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fdafc06f720 0x7fdafc06fb30 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:02.818 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:02.816+0000 7fdb01e00640 1 -- 172.21.15.32:0/4017593428 >> 172.21.15.32:0/4017593428 conn(0x7fdafc0b1bd0 msgr2=0x7fdafc0b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:50:02.818 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:02.816+0000 7fdb01e00640 1 -- 172.21.15.32:0/4017593428 shutdown_connections 2024-08-03T20:50:02.818 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:02.816+0000 7fdb01e00640 1 -- 172.21.15.32:0/4017593428 wait complete. 2024-08-03T20:50:02.820 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:02.818+0000 7fdb01e00640 1 Processor -- start 2024-08-03T20:50:02.820 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:02.819+0000 7fdb01e00640 1 -- start start 2024-08-03T20:50:02.823 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:02.820+0000 7fdb01e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdafc06f720 0x7fdafc0cc970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:50:02.823 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:02.820+0000 7fdb01e00640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fdafc0b39a0 0x7fdafc1529b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:50:02.823 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:02.820+0000 7fdb01e00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fdafc162210 0x7fdafc1f6080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:50:02.823 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:02.820+0000 7fdb01e00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fdafc162d30 con 0x7fdafc06f720 2024-08-03T20:50:02.823 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:02.820+0000 7fdb01e00640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7fdafc0af0a0 con 0x7fdafc162210 2024-08-03T20:50:02.823 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:02.820+0000 7fdb01e00640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7fdafc162eb0 con 0x7fdafc0b39a0 2024-08-03T20:50:02.823 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:02.820+0000 7fdafaa00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdafc06f720 0x7fdafc0cc970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:50:02.824 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:02.820+0000 7fdafaa00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdafc06f720 0x7fdafc0cc970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:49940/0 (socket says 172.21.15.32:49940) 2024-08-03T20:50:02.824 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:02.820+0000 7fdafaa00640 1 -- 172.21.15.32:0/1267373435 learned_addr learned my addr 172.21.15.32:0/1267373435 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:50:02.824 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:02.820+0000 7fdaf3e00640 1 --2- 172.21.15.32:0/1267373435 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fdafc0b39a0 0x7fdafc1529b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:50:02.824 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:02.821+0000 7fdafb400640 1 --2- 172.21.15.32:0/1267373435 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fdafc162210 0x7fdafc1f6080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:50:02.824 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:02.821+0000 7fdafaa00640 1 -- 172.21.15.32:0/1267373435 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fdafc0b39a0 msgr2=0x7fdafc1529b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:50:02.824 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:02.821+0000 7fdafaa00640 1 --2- 172.21.15.32:0/1267373435 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fdafc0b39a0 0x7fdafc1529b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:02.824 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:02.821+0000 7fdafaa00640 1 -- 172.21.15.32:0/1267373435 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fdafc162210 msgr2=0x7fdafc1f6080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:50:02.824 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:02.821+0000 7fdafaa00640 1 --2- 172.21.15.32:0/1267373435 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fdafc162210 0x7fdafc1f6080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:02.824 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:02.821+0000 7fdafaa00640 1 -- 172.21.15.32:0/1267373435 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdafc0cf290 con 0x7fdafc06f720 2024-08-03T20:50:02.824 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:02.821+0000 7fdafaa00640 1 --2- 172.21.15.32:0/1267373435 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdafc06f720 0x7fdafc0cc970 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7fdaec02aa90 tx=0x7fdaec003c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:50:02.824 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:02.821+0000 7fdaf3400640 1 -- 172.21.15.32:0/1267373435 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdaec005c90 con 0x7fdafc06f720 2024-08-03T20:50:02.824 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:02.822+0000 7fdaf3400640 1 -- 172.21.15.32:0/1267373435 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fdaec005020 con 0x7fdafc06f720 2024-08-03T20:50:02.825 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:02.822+0000 7fdb01e00640 1 -- 172.21.15.32:0/1267373435 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdafc0cf570 con 0x7fdafc06f720 2024-08-03T20:50:02.825 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:02.822+0000 7fdaf3400640 1 -- 172.21.15.32:0/1267373435 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdaec005450 con 0x7fdafc06f720 2024-08-03T20:50:02.825 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:02.822+0000 7fdb01e00640 1 -- 172.21.15.32:0/1267373435 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdafc0ccfb0 con 0x7fdafc06f720 2024-08-03T20:50:02.825 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:02.823+0000 7fdb01e00640 1 -- 172.21.15.32:0/1267373435 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdafc0cd210 con 0x7fdafc06f720 2024-08-03T20:50:02.825 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:02.824+0000 7fdaf3400640 1 -- 172.21.15.32:0/1267373435 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7fdaec0317c0 con 0x7fdafc06f720 2024-08-03T20:50:02.826 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:02.825+0000 7fdaf3400640 1 --2- 172.21.15.32:0/1267373435 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fdad8077010 0x7fdad8079520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:50:02.827 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:02.825+0000 7fdaf3400640 1 -- 172.21.15.32:0/1267373435 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(45..45 src has 1..45) ==== 5709+0+0 (secure 0 0 0) 0x7fdaec0b98f0 con 0x7fdafc06f720 2024-08-03T20:50:02.827 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:02.826+0000 7fdaf3e00640 1 --2- 172.21.15.32:0/1267373435 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fdad8077010 0x7fdad8079520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:50:02.828 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:02.826+0000 7fdaf3e00640 1 --2- 172.21.15.32:0/1267373435 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fdad8077010 0x7fdad8079520 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fdafc1f8f10 tx=0x7fdae4005f90 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:50:02.831 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:02.830+0000 7fdaf3400640 1 -- 172.21.15.32:0/1267373435 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7fdaec03f030 con 0x7fdafc06f720 2024-08-03T20:50:03.051 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:03.049+0000 7fdb01e00640 1 -- 172.21.15.32:0/1267373435 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7fdafc06fb30 con 0x7fdafc06f720 2024-08-03T20:50:03.051 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:03.049+0000 7fdaf3400640 1 -- 172.21.15.32:0/1267373435 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v45) ==== 80+0+0 (secure 0 0 0) 0x7fdaec082410 con 0x7fdafc06f720 2024-08-03T20:50:03.051 INFO:teuthology.orchestra.run.smithi032.stderr:listed 0 entries 2024-08-03T20:50:03.055 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:03.052+0000 7fdb01e00640 1 -- 172.21.15.32:0/1267373435 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fdad8077010 msgr2=0x7fdad8079520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:50:03.055 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:03.052+0000 7fdb01e00640 1 --2- 172.21.15.32:0/1267373435 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fdad8077010 0x7fdad8079520 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fdafc1f8f10 tx=0x7fdae4005f90 comp rx=0 tx=0).stop 2024-08-03T20:50:03.055 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:03.052+0000 7fdb01e00640 1 -- 172.21.15.32:0/1267373435 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdafc06f720 msgr2=0x7fdafc0cc970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:50:03.055 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:03.052+0000 7fdb01e00640 1 --2- 172.21.15.32:0/1267373435 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdafc06f720 0x7fdafc0cc970 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7fdaec02aa90 tx=0x7fdaec003c10 comp rx=0 tx=0).stop 2024-08-03T20:50:03.056 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:03.052+0000 7fdb01e00640 1 -- 172.21.15.32:0/1267373435 shutdown_connections 2024-08-03T20:50:03.056 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:03.052+0000 7fdb01e00640 1 --2- 172.21.15.32:0/1267373435 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fdad8077010 0x7fdad8079520 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:03.056 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:03.052+0000 7fdb01e00640 1 --2- 172.21.15.32:0/1267373435 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fdafc162210 0x7fdafc1f6080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:03.056 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:03.052+0000 7fdb01e00640 1 --2- 172.21.15.32:0/1267373435 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fdafc0b39a0 0x7fdafc1529b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:03.056 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:03.052+0000 7fdb01e00640 1 --2- 172.21.15.32:0/1267373435 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdafc06f720 0x7fdafc0cc970 unknown :-1 s=CLOSED pgs=468 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:03.056 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:03.052+0000 7fdb01e00640 1 -- 172.21.15.32:0/1267373435 >> 172.21.15.32:0/1267373435 conn(0x7fdafc0b1bd0 msgr2=0x7fdafc15db20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:50:03.057 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:03.052+0000 7fdb01e00640 1 -- 172.21.15.32:0/1267373435 shutdown_connections 2024-08-03T20:50:03.057 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:03.052+0000 7fdb01e00640 1 -- 172.21.15.32:0/1267373435 wait complete. 2024-08-03T20:50:03.077 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-08-03T20:50:03.077 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-08-03T20:50:03.077 DEBUG:teuthology.orchestra.run.smithi032:> sudo umount /home/ubuntu/cephtest/mnt.0 2024-08-03T20:50:03.183 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi032.front.sepia.ceph.com 2024-08-03T20:50:03.183 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-08-03T20:50:03.183 DEBUG:teuthology.orchestra.run.smithi032:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-08-03T20:50:03.219 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-08-03T20:50:03.488 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:03.486+0000 7f41f9600640 1 -- 172.21.15.32:0/2632126006 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f41f40af020 msgr2=0x7f41f40742b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:50:03.488 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:03.486+0000 7f41f9600640 1 --2- 172.21.15.32:0/2632126006 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f41f40af020 0x7f41f40742b0 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7f41f40723c0 tx=0x7f41e402a560 comp rx=0 tx=0).stop 2024-08-03T20:50:03.488 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:03.486+0000 7f41f9600640 1 -- 172.21.15.32:0/2632126006 shutdown_connections 2024-08-03T20:50:03.488 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:03.486+0000 7f41f9600640 1 --2- 172.21.15.32:0/2632126006 >> v2:172.21.15.118:3300/0 conn(0x7f41f406fb50 0x7f41f40ad190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:03.488 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:03.486+0000 7f41f9600640 1 --2- 172.21.15.32:0/2632126006 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f41f40af020 0x7f41f40742b0 unknown :-1 s=CLOSED pgs=469 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:03.488 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:03.486+0000 7f41f9600640 1 -- 172.21.15.32:0/2632126006 >> 172.21.15.32:0/2632126006 conn(0x7f41f40b2120 msgr2=0x7f41f40cfd60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:50:03.488 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:03.486+0000 7f41f9600640 1 -- 172.21.15.32:0/2632126006 shutdown_connections 2024-08-03T20:50:03.488 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:03.486+0000 7f41f9600640 1 -- 172.21.15.32:0/2632126006 wait complete. 2024-08-03T20:50:03.489 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:03.488+0000 7f41f9600640 1 Processor -- start 2024-08-03T20:50:03.489 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:03.488+0000 7f41f9600640 1 -- start start 2024-08-03T20:50:03.492 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:03.490+0000 7f41f9600640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f41f406fb50 0x7f41f4150f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:50:03.492 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:03.490+0000 7f41f9600640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f41f40acd80 0x7f41f409c7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:50:03.492 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:03.490+0000 7f41f9600640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f41f40af020 0x7f41f40b3130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:50:03.492 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:03.490+0000 7f41f9600640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f41f41601e0 con 0x7f41f40af020 2024-08-03T20:50:03.492 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:03.490+0000 7f41f9600640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f41f4160060 con 0x7f41f40acd80 2024-08-03T20:50:03.492 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:03.490+0000 7f41f9600640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7f41f4160360 con 0x7f41f406fb50 2024-08-03T20:50:03.492 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:03.490+0000 7f41f8c00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f41f40af020 0x7f41f40b3130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:50:03.492 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:03.490+0000 7f41f8c00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f41f40af020 0x7f41f40b3130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:49956/0 (socket says 172.21.15.32:49956) 2024-08-03T20:50:03.492 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:03.490+0000 7f41f3400640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f41f40acd80 0x7f41f409c7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:50:03.492 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:03.490+0000 7f41f8c00640 1 -- 172.21.15.32:0/3242480023 learned_addr learned my addr 172.21.15.32:0/3242480023 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:50:03.493 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:03.490+0000 7f41f3e00640 1 --2- 172.21.15.32:0/3242480023 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f41f406fb50 0x7f41f4150f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:50:03.494 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:03.491+0000 7f41f3e00640 1 -- 172.21.15.32:0/3242480023 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f41f40acd80 msgr2=0x7f41f409c7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:50:03.494 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:03.491+0000 7f41f3e00640 1 --2- 172.21.15.32:0/3242480023 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f41f40acd80 0x7f41f409c7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:03.494 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:03.491+0000 7f41f3e00640 1 -- 172.21.15.32:0/3242480023 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f41f40af020 msgr2=0x7f41f40b3130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:50:03.494 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:03.491+0000 7f41f3e00640 1 --2- 172.21.15.32:0/3242480023 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f41f40af020 0x7f41f40b3130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:03.495 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:03.491+0000 7f41f3e00640 1 -- 172.21.15.32:0/3242480023 --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f41f4151810 con 0x7f41f406fb50 2024-08-03T20:50:03.495 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:03.491+0000 7f41f8c00640 1 --2- 172.21.15.32:0/3242480023 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f41f40af020 0x7f41f40b3130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-03T20:50:03.495 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:03.491+0000 7f41f3e00640 1 --2- 172.21.15.32:0/3242480023 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f41f406fb50 0x7f41f4150f50 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f41f40b5ef0 tx=0x7f41e40038a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:50:03.498 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:03.491+0000 7f41ebe00640 1 -- 172.21.15.32:0/3242480023 <== mon.2 v2:172.21.15.118:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41e403e030 con 0x7f41f406fb50 2024-08-03T20:50:03.498 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:03.491+0000 7f41ebe00640 1 -- 172.21.15.32:0/3242480023 <== mon.2 v2:172.21.15.118:3301/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f41e4003d90 con 0x7f41f406fb50 2024-08-03T20:50:03.498 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:03.491+0000 7f41ebe00640 1 -- 172.21.15.32:0/3242480023 <== mon.2 v2:172.21.15.118:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41e4035040 con 0x7f41f406fb50 2024-08-03T20:50:03.498 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:03.492+0000 7f41f9600640 1 -- 172.21.15.32:0/3242480023 --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f41f40c6eb0 con 0x7f41f406fb50 2024-08-03T20:50:03.499 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:03.492+0000 7f41f9600640 1 -- 172.21.15.32:0/3242480023 --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f41f40c7150 con 0x7f41f406fb50 2024-08-03T20:50:03.499 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:03.493+0000 7f41f9600640 1 -- 172.21.15.32:0/3242480023 --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f41f40ad190 con 0x7f41f406fb50 2024-08-03T20:50:03.499 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:03.494+0000 7f41ebe00640 1 -- 172.21.15.32:0/3242480023 <== mon.2 v2:172.21.15.118:3301/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f41e4041030 con 0x7f41f406fb50 2024-08-03T20:50:03.500 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:03.495+0000 7f41ebe00640 1 --2- 172.21.15.32:0/3242480023 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f41d4077010 0x7f41d4079520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:50:03.500 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:03.495+0000 7f41ebe00640 1 -- 172.21.15.32:0/3242480023 <== mon.2 v2:172.21.15.118:3301/0 5 ==== osd_map(45..45 src has 1..45) ==== 5709+0+0 (secure 0 0 0) 0x7f41e40b9630 con 0x7f41f406fb50 2024-08-03T20:50:03.500 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:03.495+0000 7f41f3400640 1 --2- 172.21.15.32:0/3242480023 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f41d4077010 0x7f41d4079520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:50:03.500 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:03.496+0000 7f41f3400640 1 --2- 172.21.15.32:0/3242480023 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f41d4077010 0x7f41d4079520 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f41f40c9ea0 tx=0x7f41e0009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:50:03.500 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:03.499+0000 7f41ebe00640 1 -- 172.21.15.32:0/3242480023 <== mon.2 v2:172.21.15.118:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7f41f4151810 con 0x7f41f406fb50 2024-08-03T20:50:03.750 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:03.749+0000 7f41f9600640 1 -- 172.21.15.32:0/3242480023 --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f41f40c9ed0 con 0x7f41f406fb50 2024-08-03T20:50:03.751 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:03.750+0000 7f41ebe00640 1 -- 172.21.15.32:0/3242480023 <== mon.2 v2:172.21.15.118:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 35 v35) ==== 94+0+5226 (secure 0 0 0) 0x7f41e40821c0 con 0x7f41f406fb50 2024-08-03T20:50:03.752 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-03T20:50:03.752 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":35,"btime":"2024-08-03T20:49:50:544317+0000","default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4997,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6837/2688500839","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6836","nonce":2688500839},{"type":"v1","addr":"172.21.15.118:6837","nonce":2688500839}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":30},{"gid":5266,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6835/2381075275","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6834","nonce":2381075275},{"type":"v1","addr":"172.21.15.118:6835","nonce":2381075275}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":28},{"gid":5421,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6833/1058332640","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":1058332640},{"type":"v1","addr":"172.21.15.32:6833","nonce":1058332640}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":29}],"filesystems":[{"mdsmap":{"epoch":35,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-08-03T20:49:47.064654+0000","modified":"2024-08-03T20:49:50.067811+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5424},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5424":{"gid":5424,"name":"c","rank":0,"incarnation":31,"state":"up:active","state_seq":4,"addr":"172.21.15.32:6835/678469885","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":678469885},{"type":"v1","addr":"172.21.15.32:6835","nonce":678469885}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5424,"qdb_cluster":[5424]},"id":3}]} 2024-08-03T20:50:03.752 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 35 2024-08-03T20:50:03.754 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:03.752+0000 7f41f9600640 1 -- 172.21.15.32:0/3242480023 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f41d4077010 msgr2=0x7f41d4079520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:50:03.754 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:03.752+0000 7f41f9600640 1 --2- 172.21.15.32:0/3242480023 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f41d4077010 0x7f41d4079520 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f41f40c9ea0 tx=0x7f41e0009040 comp rx=0 tx=0).stop 2024-08-03T20:50:03.754 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:03.752+0000 7f41f9600640 1 -- 172.21.15.32:0/3242480023 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f41f406fb50 msgr2=0x7f41f4150f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:50:03.754 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:03.752+0000 7f41f9600640 1 --2- 172.21.15.32:0/3242480023 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f41f406fb50 0x7f41f4150f50 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f41f40b5ef0 tx=0x7f41e40038a0 comp rx=0 tx=0).stop 2024-08-03T20:50:03.754 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:03.752+0000 7f41f9600640 1 -- 172.21.15.32:0/3242480023 shutdown_connections 2024-08-03T20:50:03.754 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:03.752+0000 7f41f9600640 1 --2- 172.21.15.32:0/3242480023 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f41d4077010 0x7f41d4079520 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:03.754 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:03.752+0000 7f41f9600640 1 --2- 172.21.15.32:0/3242480023 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f41f40acd80 0x7f41f409c7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:03.754 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:03.752+0000 7f41f9600640 1 --2- 172.21.15.32:0/3242480023 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f41f40af020 0x7f41f40b3130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:03.755 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:03.752+0000 7f41f9600640 1 --2- 172.21.15.32:0/3242480023 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f41f406fb50 0x7f41f4150f50 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:03.755 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:03.752+0000 7f41f9600640 1 -- 172.21.15.32:0/3242480023 >> 172.21.15.32:0/3242480023 conn(0x7f41f40b2120 msgr2=0x7f41f40afcf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:50:03.755 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:03.752+0000 7f41f9600640 1 -- 172.21.15.32:0/3242480023 shutdown_connections 2024-08-03T20:50:03.755 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:03.752+0000 7f41f9600640 1 -- 172.21.15.32:0/3242480023 wait complete. 2024-08-03T20:50:03.779 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-08-03T20:50:04.007 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:04.005+0000 7f24ca800640 1 -- 172.21.15.32:0/1842602238 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f24c406e930 msgr2=0x7f24c4144f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:50:04.007 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:04.005+0000 7f24ca800640 1 --2- 172.21.15.32:0/1842602238 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f24c406e930 0x7f24c4144f90 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7f24c4072d80 tx=0x7f24b402a560 comp rx=0 tx=0).stop 2024-08-03T20:50:04.007 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:04.005+0000 7f24ca800640 1 -- 172.21.15.32:0/1842602238 shutdown_connections 2024-08-03T20:50:04.008 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:04.005+0000 7f24ca800640 1 --2- 172.21.15.32:0/1842602238 >> v2:172.21.15.118:3300/0 conn(0x7f24c415fff0 0x7f24c4160400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:04.008 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:04.005+0000 7f24ca800640 1 --2- 172.21.15.32:0/1842602238 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f24c406e930 0x7f24c4144f90 unknown :-1 s=CLOSED pgs=470 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:04.008 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:04.005+0000 7f24ca800640 1 -- 172.21.15.32:0/1842602238 >> 172.21.15.32:0/1842602238 conn(0x7f24c40b1bd0 msgr2=0x7f24c40b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:50:04.008 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:04.006+0000 7f24ca800640 1 -- 172.21.15.32:0/1842602238 shutdown_connections 2024-08-03T20:50:04.008 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:04.006+0000 7f24ca800640 1 -- 172.21.15.32:0/1842602238 wait complete. 2024-08-03T20:50:04.009 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:04.008+0000 7f24ca800640 1 Processor -- start 2024-08-03T20:50:04.009 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:04.008+0000 7f24ca800640 1 -- start start 2024-08-03T20:50:04.011 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:04.009+0000 7f24ca800640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f24c406e930 0x7f24c4143c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:50:04.012 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:04.009+0000 7f24ca800640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f24c4073e60 0x7f24c415b940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:50:04.012 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:04.009+0000 7f24ca800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f24c415fff0 0x7f24c415be80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:50:04.012 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:04.009+0000 7f24ca800640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f24c40cf840 con 0x7f24c415fff0 2024-08-03T20:50:04.012 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:04.009+0000 7f24ca800640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f24c4145f30 con 0x7f24c4073e60 2024-08-03T20:50:04.012 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:04.009+0000 7f24ca800640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7f24c40cf9c0 con 0x7f24c406e930 2024-08-03T20:50:04.012 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:04.010+0000 7f24c3e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f24c415fff0 0x7f24c415be80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:50:04.012 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:04.010+0000 7f24c3e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f24c415fff0 0x7f24c415be80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:49978/0 (socket says 172.21.15.32:49978) 2024-08-03T20:50:04.012 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:04.010+0000 7f24c3e00640 1 -- 172.21.15.32:0/2550453307 learned_addr learned my addr 172.21.15.32:0/2550453307 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:50:04.012 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:04.010+0000 7f24c3400640 1 --2- 172.21.15.32:0/2550453307 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f24c406e930 0x7f24c4143c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:50:04.012 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:04.010+0000 7f24c2a00640 1 --2- 172.21.15.32:0/2550453307 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f24c4073e60 0x7f24c415b940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:50:04.012 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:04.010+0000 7f24c3e00640 1 -- 172.21.15.32:0/2550453307 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f24c406e930 msgr2=0x7f24c4143c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:50:04.012 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:04.010+0000 7f24c3e00640 1 --2- 172.21.15.32:0/2550453307 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f24c406e930 0x7f24c4143c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:04.013 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:04.010+0000 7f24c3e00640 1 -- 172.21.15.32:0/2550453307 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f24c4073e60 msgr2=0x7f24c415b940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:50:04.013 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:04.010+0000 7f24c3e00640 1 --2- 172.21.15.32:0/2550453307 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f24c4073e60 0x7f24c415b940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:04.013 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:04.010+0000 7f24c3e00640 1 -- 172.21.15.32:0/2550453307 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f24c41f5c50 con 0x7f24c415fff0 2024-08-03T20:50:04.014 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:04.011+0000 7f24c3e00640 1 --2- 172.21.15.32:0/2550453307 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f24c415fff0 0x7f24c415be80 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7f24bc00e990 tx=0x7f24bc00ee50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:50:04.014 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:04.011+0000 7f24bb400640 1 -- 172.21.15.32:0/2550453307 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24bc004d20 con 0x7f24c415fff0 2024-08-03T20:50:04.014 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:04.011+0000 7f24bb400640 1 -- 172.21.15.32:0/2550453307 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f24bc00cb00 con 0x7f24c415fff0 2024-08-03T20:50:04.014 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:04.011+0000 7f24bb400640 1 -- 172.21.15.32:0/2550453307 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24bc005020 con 0x7f24c415fff0 2024-08-03T20:50:04.014 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:04.011+0000 7f24ca800640 1 -- 172.21.15.32:0/2550453307 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f24c41f5f30 con 0x7f24c415fff0 2024-08-03T20:50:04.015 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:04.011+0000 7f24ca800640 1 -- 172.21.15.32:0/2550453307 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f24c41f64f0 con 0x7f24c415fff0 2024-08-03T20:50:04.015 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:04.012+0000 7f24ca800640 1 -- 172.21.15.32:0/2550453307 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f24c4069090 con 0x7f24c415fff0 2024-08-03T20:50:04.015 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:04.013+0000 7f24bb400640 1 -- 172.21.15.32:0/2550453307 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f24bc0051c0 con 0x7f24c415fff0 2024-08-03T20:50:04.016 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:04.015+0000 7f24bb400640 1 --2- 172.21.15.32:0/2550453307 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f24a4077010 0x7f24a4079520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:50:04.016 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:04.015+0000 7f24bb400640 1 -- 172.21.15.32:0/2550453307 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(45..45 src has 1..45) ==== 5709+0+0 (secure 0 0 0) 0x7f24bc0991f0 con 0x7f24c415fff0 2024-08-03T20:50:04.017 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:04.015+0000 7f24c3400640 1 --2- 172.21.15.32:0/2550453307 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f24a4077010 0x7f24a4079520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:50:04.017 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:04.016+0000 7f24c3400640 1 --2- 172.21.15.32:0/2550453307 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f24a4077010 0x7f24a4079520 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f24b4033000 tx=0x7f24b4004730 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:50:04.020 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:04.019+0000 7f24bb400640 1 -- 172.21.15.32:0/2550453307 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7f24c4069090 con 0x7f24c415fff0 2024-08-03T20:50:04.265 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:04.264+0000 7f24ca800640 1 -- 172.21.15.32:0/2550453307 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f24c41f59e0 con 0x7f24c415fff0 2024-08-03T20:50:04.266 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:04.264+0000 7f24bb400640 1 -- 172.21.15.32:0/2550453307 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 35 v35) ==== 94+0+5226 (secure 0 0 0) 0x7f24bc016030 con 0x7f24c415fff0 2024-08-03T20:50:04.267 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-03T20:50:04.267 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":35,"btime":"2024-08-03T20:49:50:544317+0000","default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4997,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6837/2688500839","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6836","nonce":2688500839},{"type":"v1","addr":"172.21.15.118:6837","nonce":2688500839}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":30},{"gid":5266,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6835/2381075275","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6834","nonce":2381075275},{"type":"v1","addr":"172.21.15.118:6835","nonce":2381075275}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":28},{"gid":5421,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6833/1058332640","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":1058332640},{"type":"v1","addr":"172.21.15.32:6833","nonce":1058332640}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":29}],"filesystems":[{"mdsmap":{"epoch":35,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-08-03T20:49:47.064654+0000","modified":"2024-08-03T20:49:50.067811+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5424},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5424":{"gid":5424,"name":"c","rank":0,"incarnation":31,"state":"up:active","state_seq":4,"addr":"172.21.15.32:6835/678469885","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":678469885},{"type":"v1","addr":"172.21.15.32:6835","nonce":678469885}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5424,"qdb_cluster":[5424]},"id":3}]} 2024-08-03T20:50:04.268 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 35 2024-08-03T20:50:04.269 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:04.267+0000 7f24ca800640 1 -- 172.21.15.32:0/2550453307 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f24a4077010 msgr2=0x7f24a4079520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:50:04.269 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:04.267+0000 7f24ca800640 1 --2- 172.21.15.32:0/2550453307 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f24a4077010 0x7f24a4079520 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f24b4033000 tx=0x7f24b4004730 comp rx=0 tx=0).stop 2024-08-03T20:50:04.269 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:04.267+0000 7f24ca800640 1 -- 172.21.15.32:0/2550453307 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f24c415fff0 msgr2=0x7f24c415be80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:50:04.269 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:04.267+0000 7f24ca800640 1 --2- 172.21.15.32:0/2550453307 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f24c415fff0 0x7f24c415be80 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7f24bc00e990 tx=0x7f24bc00ee50 comp rx=0 tx=0).stop 2024-08-03T20:50:04.269 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:04.267+0000 7f24ca800640 1 -- 172.21.15.32:0/2550453307 shutdown_connections 2024-08-03T20:50:04.270 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:04.267+0000 7f24ca800640 1 --2- 172.21.15.32:0/2550453307 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f24a4077010 0x7f24a4079520 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:04.270 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:04.267+0000 7f24ca800640 1 --2- 172.21.15.32:0/2550453307 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f24c4073e60 0x7f24c415b940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:04.270 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:04.267+0000 7f24ca800640 1 --2- 172.21.15.32:0/2550453307 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f24c415fff0 0x7f24c415be80 unknown :-1 s=CLOSED pgs=471 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:04.270 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:04.267+0000 7f24ca800640 1 --2- 172.21.15.32:0/2550453307 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f24c406e930 0x7f24c4143c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:04.270 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:04.267+0000 7f24ca800640 1 -- 172.21.15.32:0/2550453307 >> 172.21.15.32:0/2550453307 conn(0x7f24c40b1bd0 msgr2=0x7f24c40d08e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:50:04.270 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:04.267+0000 7f24ca800640 1 -- 172.21.15.32:0/2550453307 shutdown_connections 2024-08-03T20:50:04.270 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:04.267+0000 7f24ca800640 1 -- 172.21.15.32:0/2550453307 wait complete. 2024-08-03T20:50:04.292 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json dump tree /a 0 2024-08-03T20:50:04.539 INFO:teuthology.orchestra.run.smithi032.stdout:[{"ino":1099511627776,"rdev":0,"ctime":"2024-08-03T20:50:02.493997+0000","btime":"2024-08-03T20:49:56.119125+0000","mode":16877,"uid":1000,"gid":1316,"nlink":1,"dir_layout":{"dir_hash":2,"unused1":0,"unused2":0,"unused3":0},"layout":{"stripe_unit":0,"stripe_count":0,"object_size":0,"pool_id":-1,"pool_ns":""},"old_pools":[],"size":0,"truncate_seq":1,"truncate_size":18446744073709551615,"truncate_from":0,"truncate_pending":0,"mtime":"2024-08-03T20:50:02.493997+0000","atime":"2024-08-03T20:49:56.119125+0000","time_warp_seq":0,"change_attr":3,"export_pin":-1,"export_ephemeral_random_pin":0,"export_ephemeral_distributed_pin":false,"quiesce_block":false,"client_ranges":[],"dirstat":{"version":0,"mtime":"2024-08-03T20:50:02.493997+0000","num_files":0,"num_subdirs":1,"change_attr":3},"rstat":{"version":0,"rbytes":0,"rfiles":0,"rsubdirs":2,"rsnaps":0,"rctime":"2024-08-03T20:50:02.493997+0000"},"accounted_rstat":{"version":0,"rbytes":0,"rfiles":0,"rsubdirs":2,"rsnaps":0,"rctime":"2024-08-03T20:50:02.493997+0000"},"version":9,"file_data_version":0,"xattr_version":1,"backtrace_version":2,"stray_prior_path":"","max_size_ever":0,"quota":{"max_bytes":0,"max_files":0},"last_scrub_stamp":"0.000000","last_scrub_version":0,"symlink":"","xattrs":[{"key":"security.selinux","val":"unconfined_u:object_r:unlabeled_t:s0\u0000"}],"dirfragtree":{"splits":[]},"old_inodes":[],"oldest_snap":18446744073709551614,"damage_flags":0,"is_auth":true,"auth_state":{"replicas":{}},"replica_state":{"authority":[0,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"request":0,"lock":0,"dirfrag":1,"caps":0,"dirtyparent":0,"dirty":1,"authpin":0},"nref":2,"versionlock":{"gather_set":[],"state":"lock","type":"iversion","is_leased":false,"num_rdlocks":0,"num_wrlocks":0,"num_xlocks":0,"xlock_by":{}},"authlock":{},"linklock":{},"dirfragtreelock":{},"filelock":{},"xattrlock":{},"snaplock":{},"nestlock":{"gather_set":[],"state":"lock","type":"inest","is_leased":false,"num_rdlocks":0,"num_wrlocks":0,"num_xlocks":0,"xlock_by":{}},"flocklock":{},"policylock":{},"quiescelock":{"gather_set":[],"state":"lock","type":"iquiesce","is_leased":false,"num_rdlocks":0,"num_wrlocks":0,"num_xlocks":0,"xlock_by":{}},"states":["auth","dirty"],"client_caps":[],"loner":-1,"want_loner":-1,"mds_caps_wanted":[],"dirfrags":[{"path":"/a","dirfrag":"0x10000000000","snapid_first":2,"projected_version":"7","version":"7","committing_version":"3","committed_version":"3","is_rep":false,"dir_auth":"","states":["auth","dirty","complete"],"is_auth":true,"auth_state":{"replicas":{}},"replica_state":{"authority":[0,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"dirty":1,"authpin":0},"nref":2,"dentries":[{"path":"a/c","alternate_name":"","path_ino":1,"snap_first":18446744073709551606,"snap_last":18446744073709551614,"is_primary":true,"is_remote":false,"is_null":false,"is_new":false,"inode":1099511627777,"remote_type":"","version":4,"projected_version":4,"auth_pins":0,"is_auth":true,"auth_state":{"replicas":{}},"replica_state":{"authority":[0,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"request":0,"lock":0,"inodepin":1,"dirty":1,"authpin":0},"nref":2,"lock":{},"versionlock":{"gather_set":[],"state":"lock","type":"dversion","is_leased":false,"num_rdlocks":0,"num_wrlocks":0,"num_xlocks":0,"xlock_by":{}},"states":["auth","dirty"]},{"path":"a/b","alternate_name":"","path_ino":1,"snap_first":2,"snap_last":18446744073709551614,"is_primary":false,"is_remote":false,"is_null":true,"is_new":false,"inode":0,"remote_type":"","version":5,"projected_version":5,"auth_pins":0,"is_auth":true,"auth_state":{"replicas":{}},"replica_state":{"authority":[0,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"request":0,"lock":0,"inodepin":0,"dirty":1,"authpin":0},"nref":1,"lock":{},"versionlock":{"gather_set":[],"state":"lock","type":"dversion","is_leased":false,"num_rdlocks":0,"num_wrlocks":0,"num_xlocks":0,"xlock_by":{}},"states":["auth","dirty"]}]}],"is_subvolume":false}] 2024-08-03T20:50:04.540 DEBUG:tasks.cephfs.filesystem:_json_asok output [ { "accounted_rstat": { "rbytes": 0, "rctime": "2024-08-03T20:50:02.493997+0000", "rfiles": 0, "rsnaps": 0, "rsubdirs": 2, "version": 0 }, "atime": "2024-08-03T20:49:56.119125+0000", "auth_pins": 0, "auth_state": { "replicas": {} }, "authlock": {}, "backtrace_version": 2, "btime": "2024-08-03T20:49:56.119125+0000", "change_attr": 3, "client_caps": [], "client_ranges": [], "ctime": "2024-08-03T20:50:02.493997+0000", "damage_flags": 0, "dir_layout": { "dir_hash": 2, "unused1": 0, "unused2": 0, "unused3": 0 }, "dirfrags": [ { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "3", "committing_version": "3", "dentries": [ { "alternate_name": "", "auth_pins": 0, "auth_state": { "replicas": {} }, "inode": 1099511627777, "is_auth": true, "is_freezing": false, "is_frozen": false, "is_new": false, "is_null": false, "is_primary": true, "is_remote": false, "lock": {}, "nref": 2, "path": "a/c", "path_ino": 1, "pins": { "authpin": 0, "dirty": 1, "inodepin": 1, "lock": 0, "request": 0 }, "projected_version": 4, "remote_type": "", "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 0 }, "snap_first": 18446744073709551606, "snap_last": 18446744073709551614, "states": [ "auth", "dirty" ], "version": 4, "versionlock": { "gather_set": [], "is_leased": false, "num_rdlocks": 0, "num_wrlocks": 0, "num_xlocks": 0, "state": "lock", "type": "dversion", "xlock_by": {} } }, { "alternate_name": "", "auth_pins": 0, "auth_state": { "replicas": {} }, "inode": 0, "is_auth": true, "is_freezing": false, "is_frozen": false, "is_new": false, "is_null": true, "is_primary": false, "is_remote": false, "lock": {}, "nref": 1, "path": "a/b", "path_ino": 1, "pins": { "authpin": 0, "dirty": 1, "inodepin": 0, "lock": 0, "request": 0 }, "projected_version": 5, "remote_type": "", "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 0 }, "snap_first": 2, "snap_last": 18446744073709551614, "states": [ "auth", "dirty" ], "version": 5, "versionlock": { "gather_set": [], "is_leased": false, "num_rdlocks": 0, "num_wrlocks": 0, "num_xlocks": 0, "state": "lock", "type": "dversion", "xlock_by": {} } } ], "dir_auth": "", "dirfrag": "0x10000000000", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 2, "path": "/a", "pins": { "authpin": 0, "child": 1, "dirty": 1 }, "projected_version": "7", "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "dirty", "complete" ], "version": "7" } ], "dirfragtree": { "splits": [] }, "dirfragtreelock": {}, "dirstat": { "change_attr": 3, "mtime": "2024-08-03T20:50:02.493997+0000", "num_files": 0, "num_subdirs": 1, "version": 0 }, "export_ephemeral_distributed_pin": false, "export_ephemeral_random_pin": 0, "export_pin": -1, "file_data_version": 0, "filelock": {}, "flocklock": {}, "gid": 1316, "ino": 1099511627776, "is_auth": true, "is_freezing": false, "is_frozen": false, "is_subvolume": false, "last_scrub_stamp": "0.000000", "last_scrub_version": 0, "layout": { "object_size": 0, "pool_id": -1, "pool_ns": "", "stripe_count": 0, "stripe_unit": 0 }, "linklock": {}, "loner": -1, "max_size_ever": 0, "mds_caps_wanted": [], "mode": 16877, "mtime": "2024-08-03T20:50:02.493997+0000", "nestlock": { "gather_set": [], "is_leased": false, "num_rdlocks": 0, "num_wrlocks": 0, "num_xlocks": 0, "state": "lock", "type": "inest", "xlock_by": {} }, "nlink": 1, "nref": 2, "old_inodes": [], "old_pools": [], "oldest_snap": 18446744073709551614, "pins": { "authpin": 0, "caps": 0, "dirfrag": 1, "dirty": 1, "dirtyparent": 0, "lock": 0, "request": 0 }, "policylock": {}, "quiesce_block": false, "quiescelock": { "gather_set": [], "is_leased": false, "num_rdlocks": 0, "num_wrlocks": 0, "num_xlocks": 0, "state": "lock", "type": "iquiesce", "xlock_by": {} }, "quota": { "max_bytes": 0, "max_files": 0 }, "rdev": 0, "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 0 }, "rstat": { "rbytes": 0, "rctime": "2024-08-03T20:50:02.493997+0000", "rfiles": 0, "rsnaps": 0, "rsubdirs": 2, "version": 0 }, "size": 0, "snaplock": {}, "states": [ "auth", "dirty" ], "stray_prior_path": "", "symlink": "", "time_warp_seq": 0, "truncate_from": 0, "truncate_pending": 0, "truncate_seq": 1, "truncate_size": 18446744073709551615, "uid": 1000, "version": 9, "versionlock": { "gather_set": [], "is_leased": false, "num_rdlocks": 0, "num_wrlocks": 0, "num_xlocks": 0, "state": "lock", "type": "iversion", "xlock_by": {} }, "want_loner": -1, "xattr_version": 1, "xattrlock": {}, "xattrs": [ { "key": "security.selinux", "val": "unconfined_u:object_r:unlabeled_t:s0\u0000" } ] } ] 2024-08-03T20:50:04.541 DEBUG:tasks.cephfs.test_damage:[{'ino': 1099511627776, 'rdev': 0, 'ctime': '2024-08-03T20:50:02.493997+0000', 'btime': '2024-08-03T20:49:56.119125+0000', 'mode': 16877, 'uid': 1000, 'gid': 1316, 'nlink': 1, 'dir_layout': {'dir_hash': 2, 'unused1': 0, 'unused2': 0, 'unused3': 0}, 'layout': {'stripe_unit': 0, 'stripe_count': 0, 'object_size': 0, 'pool_id': -1, 'pool_ns': ''}, 'old_pools': [], 'size': 0, 'truncate_seq': 1, 'truncate_size': 18446744073709551615, 'truncate_from': 0, 'truncate_pending': 0, 'mtime': '2024-08-03T20:50:02.493997+0000', 'atime': '2024-08-03T20:49:56.119125+0000', 'time_warp_seq': 0, 'change_attr': 3, 'export_pin': -1, 'export_ephemeral_random_pin': 0, 'export_ephemeral_distributed_pin': False, 'quiesce_block': False, 'client_ranges': [], 'dirstat': {'version': 0, 'mtime': '2024-08-03T20:50:02.493997+0000', 'num_files': 0, 'num_subdirs': 1, 'change_attr': 3}, 'rstat': {'version': 0, 'rbytes': 0, 'rfiles': 0, 'rsubdirs': 2, 'rsnaps': 0, 'rctime': '2024-08-03T20:50:02.493997+0000'}, 'accounted_rstat': {'version': 0, 'rbytes': 0, 'rfiles': 0, 'rsubdirs': 2, 'rsnaps': 0, 'rctime': '2024-08-03T20:50:02.493997+0000'}, 'version': 9, 'file_data_version': 0, 'xattr_version': 1, 'backtrace_version': 2, 'stray_prior_path': '', 'max_size_ever': 0, 'quota': {'max_bytes': 0, 'max_files': 0}, 'last_scrub_stamp': '0.000000', 'last_scrub_version': 0, 'symlink': '', 'xattrs': [{'key': 'security.selinux', 'val': 'unconfined_u:object_r:unlabeled_t:s0\x00'}], 'dirfragtree': {'splits': []}, 'old_inodes': [], 'oldest_snap': 18446744073709551614, 'damage_flags': 0, 'is_auth': True, 'auth_state': {'replicas': {}}, 'replica_state': {'authority': [0, -2], 'replica_nonce': 0}, 'auth_pins': 0, 'is_frozen': False, 'is_freezing': False, 'pins': {'request': 0, 'lock': 0, 'dirfrag': 1, 'caps': 0, 'dirtyparent': 0, 'dirty': 1, 'authpin': 0}, 'nref': 2, 'versionlock': {'gather_set': [], 'state': 'lock', 'type': 'iversion', 'is_leased': False, 'num_rdlocks': 0, 'num_wrlocks': 0, 'num_xlocks': 0, 'xlock_by': {}}, 'authlock': {}, 'linklock': {}, 'dirfragtreelock': {}, 'filelock': {}, 'xattrlock': {}, 'snaplock': {}, 'nestlock': {'gather_set': [], 'state': 'lock', 'type': 'inest', 'is_leased': False, 'num_rdlocks': 0, 'num_wrlocks': 0, 'num_xlocks': 0, 'xlock_by': {}}, 'flocklock': {}, 'policylock': {}, 'quiescelock': {'gather_set': [], 'state': 'lock', 'type': 'iquiesce', 'is_leased': False, 'num_rdlocks': 0, 'num_wrlocks': 0, 'num_xlocks': 0, 'xlock_by': {}}, 'states': ['auth', 'dirty'], 'client_caps': [], 'loner': -1, 'want_loner': -1, 'mds_caps_wanted': [], 'dirfrags': [{'path': '/a', 'dirfrag': '0x10000000000', 'snapid_first': 2, 'projected_version': '7', 'version': '7', 'committing_version': '3', 'committed_version': '3', 'is_rep': False, 'dir_auth': '', 'states': ['auth', 'dirty', 'complete'], 'is_auth': True, 'auth_state': {'replicas': {}}, 'replica_state': {'authority': [0, -2], 'replica_nonce': 0}, 'auth_pins': 0, 'is_frozen': False, 'is_freezing': False, 'pins': {'child': 1, 'dirty': 1, 'authpin': 0}, 'nref': 2, 'dentries': [{'path': 'a/c', 'alternate_name': '', 'path_ino': 1, 'snap_first': 18446744073709551606, 'snap_last': 18446744073709551614, 'is_primary': True, 'is_remote': False, 'is_null': False, 'is_new': False, 'inode': 1099511627777, 'remote_type': '', 'version': 4, 'projected_version': 4, 'auth_pins': 0, 'is_auth': True, 'auth_state': {'replicas': {}}, 'replica_state': {'authority': [0, -2], 'replica_nonce': 0}, 'is_frozen': False, 'is_freezing': False, 'pins': {'request': 0, 'lock': 0, 'inodepin': 1, 'dirty': 1, 'authpin': 0}, 'nref': 2, 'lock': {}, 'versionlock': {'gather_set': [], 'state': 'lock', 'type': 'dversion', 'is_leased': False, 'num_rdlocks': 0, 'num_wrlocks': 0, 'num_xlocks': 0, 'xlock_by': {}}, 'states': ['auth', 'dirty']}, {'path': 'a/b', 'alternate_name': '', 'path_ino': 1, 'snap_first': 2, 'snap_last': 18446744073709551614, 'is_primary': False, 'is_remote': False, 'is_null': True, 'is_new': False, 'inode': 0, 'remote_type': '', 'version': 5, 'projected_version': 5, 'auth_pins': 0, 'is_auth': True, 'auth_state': {'replicas': {}}, 'replica_state': {'authority': [0, -2], 'replica_nonce': 0}, 'is_frozen': False, 'is_freezing': False, 'pins': {'request': 0, 'lock': 0, 'inodepin': 0, 'dirty': 1, 'authpin': 0}, 'nref': 1, 'lock': {}, 'versionlock': {'gather_set': [], 'state': 'lock', 'type': 'dversion', 'is_leased': False, 'num_rdlocks': 0, 'num_wrlocks': 0, 'num_xlocks': 0, 'xlock_by': {}}, 'states': ['auth', 'dirty']}]}], 'is_subvolume': False}] 2024-08-03T20:50:04.541 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-08-03T20:50:04.770 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:04.768+0000 7f22da800640 1 -- 172.21.15.32:0/2727571914 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f22d406e930 msgr2=0x7f22d4144f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:50:04.770 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:04.768+0000 7f22da800640 1 --2- 172.21.15.32:0/2727571914 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f22d406e930 0x7f22d4144f90 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7f22c4005fd0 tx=0x7f22c402a4a0 comp rx=0 tx=0).stop 2024-08-03T20:50:04.770 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:04.768+0000 7f22da800640 1 -- 172.21.15.32:0/2727571914 shutdown_connections 2024-08-03T20:50:04.770 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:04.768+0000 7f22da800640 1 --2- 172.21.15.32:0/2727571914 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f22d4073e60 0x7f22d4074270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:04.771 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:04.768+0000 7f22da800640 1 --2- 172.21.15.32:0/2727571914 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f22d406e930 0x7f22d4144f90 unknown :-1 s=CLOSED pgs=472 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:04.771 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:04.768+0000 7f22da800640 1 -- 172.21.15.32:0/2727571914 >> 172.21.15.32:0/2727571914 conn(0x7f22d40b1bd0 msgr2=0x7f22d40b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:50:04.771 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:04.768+0000 7f22da800640 1 -- 172.21.15.32:0/2727571914 shutdown_connections 2024-08-03T20:50:04.771 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:04.768+0000 7f22da800640 1 -- 172.21.15.32:0/2727571914 wait complete. 2024-08-03T20:50:04.772 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:04.771+0000 7f22da800640 1 Processor -- start 2024-08-03T20:50:04.772 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:04.771+0000 7f22da800640 1 -- start start 2024-08-03T20:50:04.775 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:04.772+0000 7f22da800640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f22d406e930 0x7f22d41f6130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:50:04.775 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:04.772+0000 7f22da800640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f22d4073e60 0x7f22d41f8e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:50:04.775 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:04.772+0000 7f22da800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f22d40747b0 0x7f22d413f110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:50:04.776 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:04.772+0000 7f22da800640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f22d40cf9c0 con 0x7f22d40747b0 2024-08-03T20:50:04.776 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:04.772+0000 7f22da800640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f22d40cf840 con 0x7f22d4073e60 2024-08-03T20:50:04.776 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:04.772+0000 7f22da800640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7f22d40ccff0 con 0x7f22d406e930 2024-08-03T20:50:04.776 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:04.773+0000 7f22d3e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f22d40747b0 0x7f22d413f110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:50:04.776 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:04.773+0000 7f22d3400640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f22d406e930 0x7f22d41f6130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:50:04.777 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:04.773+0000 7f22d3e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f22d40747b0 0x7f22d413f110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:49994/0 (socket says 172.21.15.32:49994) 2024-08-03T20:50:04.777 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:04.773+0000 7f22d3e00640 1 -- 172.21.15.32:0/1189735216 learned_addr learned my addr 172.21.15.32:0/1189735216 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:50:04.777 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:04.773+0000 7f22cbe00640 1 --2- 172.21.15.32:0/1189735216 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f22d4073e60 0x7f22d41f8e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:50:04.777 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:04.773+0000 7f22d3e00640 1 -- 172.21.15.32:0/1189735216 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f22d406e930 msgr2=0x7f22d41f6130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:50:04.777 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:04.773+0000 7f22d3e00640 1 --2- 172.21.15.32:0/1189735216 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f22d406e930 0x7f22d41f6130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:04.777 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:04.773+0000 7f22d3e00640 1 -- 172.21.15.32:0/1189735216 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f22d4073e60 msgr2=0x7f22d41f8e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:50:04.778 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:04.773+0000 7f22d3e00640 1 --2- 172.21.15.32:0/1189735216 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f22d4073e60 0x7f22d41f8e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:04.778 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:04.773+0000 7f22d3e00640 1 -- 172.21.15.32:0/1189735216 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f22d41f9b00 con 0x7f22d40747b0 2024-08-03T20:50:04.778 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:04.773+0000 7f22d3e00640 1 --2- 172.21.15.32:0/1189735216 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f22d40747b0 0x7f22d413f110 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7f22cc00ba10 tx=0x7f22cc00bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:50:04.778 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:04.774+0000 7f22cb400640 1 -- 172.21.15.32:0/1189735216 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22cc019070 con 0x7f22d40747b0 2024-08-03T20:50:04.778 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:04.774+0000 7f22cb400640 1 -- 172.21.15.32:0/1189735216 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f22cc00cc30 con 0x7f22d40747b0 2024-08-03T20:50:04.778 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:04.774+0000 7f22cb400640 1 -- 172.21.15.32:0/1189735216 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22cc00ce60 con 0x7f22d40747b0 2024-08-03T20:50:04.778 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:04.774+0000 7f22da800640 1 -- 172.21.15.32:0/1189735216 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f22d41f6670 con 0x7f22d40747b0 2024-08-03T20:50:04.778 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:04.774+0000 7f22da800640 1 -- 172.21.15.32:0/1189735216 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f22d41425d0 con 0x7f22d40747b0 2024-08-03T20:50:04.779 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:04.775+0000 7f22da800640 1 -- 172.21.15.32:0/1189735216 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f22d40cd570 con 0x7f22d40747b0 2024-08-03T20:50:04.779 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:04.776+0000 7f22cb400640 1 -- 172.21.15.32:0/1189735216 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f22cc01c070 con 0x7f22d40747b0 2024-08-03T20:50:04.779 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:04.777+0000 7f22cb400640 1 --2- 172.21.15.32:0/1189735216 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f22b0077010 0x7f22b0079520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:50:04.779 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:04.777+0000 7f22cb400640 1 -- 172.21.15.32:0/1189735216 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(45..45 src has 1..45) ==== 5709+0+0 (secure 0 0 0) 0x7f22cc098690 con 0x7f22d40747b0 2024-08-03T20:50:04.779 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:04.777+0000 7f22d3400640 1 --2- 172.21.15.32:0/1189735216 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f22b0077010 0x7f22b0079520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:50:04.780 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:04.778+0000 7f22d3400640 1 --2- 172.21.15.32:0/1189735216 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f22b0077010 0x7f22b0079520 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f22c4033000 tx=0x7f22c4004730 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:50:04.782 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:04.781+0000 7f22cb400640 1 -- 172.21.15.32:0/1189735216 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7f22d41f6670 con 0x7f22d40747b0 2024-08-03T20:50:05.029 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:05.028+0000 7f22da800640 1 -- 172.21.15.32:0/1189735216 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7f22d40af8d0 con 0x7f22d40747b0 2024-08-03T20:50:05.030 INFO:tasks.ceph.mon.a.smithi032.stderr:2024-08-03T20:50:05.029+0000 7f4e6d600640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-08-03T20:50:05.032 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:05.031+0000 7f22cb400640 1 -- 172.21.15.32:0/1189735216 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v36) ==== 161+0+0 (secure 0 0 0) 0x7f22d41f9b00 con 0x7f22d40747b0 2024-08-03T20:50:05.032 INFO:teuthology.orchestra.run.smithi032.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-08-03T20:50:05.035 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:05.033+0000 7f22da800640 1 -- 172.21.15.32:0/1189735216 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f22b0077010 msgr2=0x7f22b0079520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:50:05.035 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:05.033+0000 7f22da800640 1 --2- 172.21.15.32:0/1189735216 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f22b0077010 0x7f22b0079520 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f22c4033000 tx=0x7f22c4004730 comp rx=0 tx=0).stop 2024-08-03T20:50:05.035 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:05.033+0000 7f22da800640 1 -- 172.21.15.32:0/1189735216 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f22d40747b0 msgr2=0x7f22d413f110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:50:05.035 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:05.033+0000 7f22da800640 1 --2- 172.21.15.32:0/1189735216 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f22d40747b0 0x7f22d413f110 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7f22cc00ba10 tx=0x7f22cc00bed0 comp rx=0 tx=0).stop 2024-08-03T20:50:05.035 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:05.033+0000 7f22da800640 1 -- 172.21.15.32:0/1189735216 shutdown_connections 2024-08-03T20:50:05.035 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:05.033+0000 7f22da800640 1 --2- 172.21.15.32:0/1189735216 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f22b0077010 0x7f22b0079520 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:05.035 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:05.033+0000 7f22da800640 1 --2- 172.21.15.32:0/1189735216 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f22d4073e60 0x7f22d41f8e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:05.035 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:05.033+0000 7f22da800640 1 --2- 172.21.15.32:0/1189735216 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f22d40747b0 0x7f22d413f110 unknown :-1 s=CLOSED pgs=473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:05.035 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:05.033+0000 7f22da800640 1 --2- 172.21.15.32:0/1189735216 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f22d406e930 0x7f22d41f6130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:05.036 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:05.033+0000 7f22da800640 1 -- 172.21.15.32:0/1189735216 >> 172.21.15.32:0/1189735216 conn(0x7f22d40b1bd0 msgr2=0x7f22d40b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:50:05.036 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:05.033+0000 7f22da800640 1 -- 172.21.15.32:0/1189735216 shutdown_connections 2024-08-03T20:50:05.036 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:05.033+0000 7f22da800640 1 -- 172.21.15.32:0/1189735216 wait complete. 2024-08-03T20:50:05.043 INFO:tasks.ceph.mds.c.smithi032.stderr: -4163> 2024-08-03T20:49:52.486+0000 7f7390400640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-08-03T20:50:05.050 INFO:tasks.ceph.mds.c.smithi032.stderr: -384> 2024-08-03T20:50:02.494+0000 7f7393600640 -1 mds.0.cache.den(0x10000000000 c) newly corrupt dentry to be committed: [dentry #0x1/a/c [fffffffffffffff6,head] auth NULL (dn xlock x=1 by request(client.5404:44 nref=7 cr=0x558800833180)) (dversion lock w=1 last_client=5404) pv=4 v=3 ap=2 ino=(nil) state=1073741824 | request=1 lock=2 authpin=1 0x5588008b1680] 2024-08-03T20:50:05.060 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mds mds_inject_rename_corrupt_dentry_first 2024-08-03T20:50:05.170 INFO:tasks.ceph.mds.c.smithi032.stdout:starting mds.c at 2024-08-03T20:50:05.294 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:05.292+0000 7f1071000640 1 -- 172.21.15.32:0/2339854873 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f106c145c50 msgr2=0x7f106c06e950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:50:05.294 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:05.292+0000 7f1071000640 1 --2- 172.21.15.32:0/2339854873 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f106c145c50 0x7f106c06e950 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f105c009830 tx=0x7f105c02a510 comp rx=0 tx=0).stop 2024-08-03T20:50:05.294 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:05.292+0000 7f1071000640 1 -- 172.21.15.32:0/2339854873 shutdown_connections 2024-08-03T20:50:05.294 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:05.292+0000 7f1071000640 1 --2- 172.21.15.32:0/2339854873 >> v2:172.21.15.118:3300/0 conn(0x7f106c0ced30 0x7f106c0cf140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:05.294 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:05.292+0000 7f1071000640 1 --2- 172.21.15.32:0/2339854873 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f106c145c50 0x7f106c06e950 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:05.294 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:05.292+0000 7f1071000640 1 -- 172.21.15.32:0/2339854873 >> 172.21.15.32:0/2339854873 conn(0x7f106c0b1bd0 msgr2=0x7f106c0b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:50:05.295 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:05.292+0000 7f1071000640 1 -- 172.21.15.32:0/2339854873 shutdown_connections 2024-08-03T20:50:05.295 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:05.292+0000 7f1071000640 1 -- 172.21.15.32:0/2339854873 wait complete. 2024-08-03T20:50:05.296 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:05.294+0000 7f1071000640 1 Processor -- start 2024-08-03T20:50:05.296 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:05.295+0000 7f1071000640 1 -- start start 2024-08-03T20:50:05.299 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:05.296+0000 7f1071000640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f106c0cccf0 0x7f106c0c5020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:50:05.299 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:05.296+0000 7f1071000640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f106c0ced30 0x7f106c0c5560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:50:05.299 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:05.296+0000 7f1071000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f106c145c50 0x7f106c0c5aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:50:05.299 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:05.296+0000 7f1071000640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f106c0b4070 con 0x7f106c145c50 2024-08-03T20:50:05.299 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:05.296+0000 7f1071000640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f106c0b3ef0 con 0x7f106c0cccf0 2024-08-03T20:50:05.299 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:05.296+0000 7f1071000640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7f106c0b41f0 con 0x7f106c0ced30 2024-08-03T20:50:05.299 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:05.296+0000 7f106a800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f106c145c50 0x7f106c0c5aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:50:05.300 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:05.296+0000 7f106a800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f106c145c50 0x7f106c0c5aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:50016/0 (socket says 172.21.15.32:50016) 2024-08-03T20:50:05.300 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:05.296+0000 7f106a800640 1 -- 172.21.15.32:0/1340693462 learned_addr learned my addr 172.21.15.32:0/1340693462 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:50:05.300 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:05.296+0000 7f1069e00640 1 --2- 172.21.15.32:0/1340693462 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f106c0cccf0 0x7f106c0c5020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:50:05.300 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:05.296+0000 7f1069400640 1 --2- 172.21.15.32:0/1340693462 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f106c0ced30 0x7f106c0c5560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:50:05.300 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:05.297+0000 7f106a800640 1 -- 172.21.15.32:0/1340693462 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f106c0ced30 msgr2=0x7f106c0c5560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:50:05.300 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:05.297+0000 7f106a800640 1 --2- 172.21.15.32:0/1340693462 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f106c0ced30 0x7f106c0c5560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:05.301 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:05.297+0000 7f106a800640 1 -- 172.21.15.32:0/1340693462 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f106c0cccf0 msgr2=0x7f106c0c5020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:50:05.301 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:05.297+0000 7f106a800640 1 --2- 172.21.15.32:0/1340693462 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f106c0cccf0 0x7f106c0c5020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:05.301 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:05.297+0000 7f106a800640 1 -- 172.21.15.32:0/1340693462 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f106c0cbec0 con 0x7f106c145c50 2024-08-03T20:50:05.301 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:05.297+0000 7f1069e00640 1 --2- 172.21.15.32:0/1340693462 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f106c0cccf0 0x7f106c0c5020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-03T20:50:05.301 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:05.297+0000 7f106a800640 1 --2- 172.21.15.32:0/1340693462 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f106c145c50 0x7f106c0c5aa0 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f106400ea40 tx=0x7f106400ef00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:50:05.301 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:05.297+0000 7f1062000640 1 -- 172.21.15.32:0/1340693462 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f106400ce40 con 0x7f106c145c50 2024-08-03T20:50:05.301 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:05.297+0000 7f1062000640 1 -- 172.21.15.32:0/1340693462 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f1064004c10 con 0x7f106c145c50 2024-08-03T20:50:05.301 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:05.297+0000 7f1062000640 1 -- 172.21.15.32:0/1340693462 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1064005020 con 0x7f106c145c50 2024-08-03T20:50:05.301 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:05.297+0000 7f1071000640 1 -- 172.21.15.32:0/1340693462 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f106c0c6110 con 0x7f106c145c50 2024-08-03T20:50:05.302 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:05.298+0000 7f1071000640 1 -- 172.21.15.32:0/1340693462 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f106c0cb800 con 0x7f106c145c50 2024-08-03T20:50:05.304 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:05.299+0000 7f1071000640 1 -- 172.21.15.32:0/1340693462 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f106c0cbc30 con 0x7f106c145c50 2024-08-03T20:50:05.305 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:05.300+0000 7f1062000640 1 -- 172.21.15.32:0/1340693462 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f10640051c0 con 0x7f106c145c50 2024-08-03T20:50:05.305 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:05.301+0000 7f1062000640 1 --2- 172.21.15.32:0/1340693462 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f104c077040 0x7f104c079550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:50:05.305 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:05.301+0000 7f1062000640 1 -- 172.21.15.32:0/1340693462 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(46..46 src has 1..46) ==== 5795+0+0 (secure 0 0 0) 0x7f10640996b0 con 0x7f106c145c50 2024-08-03T20:50:05.305 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:05.301+0000 7f1069e00640 1 --2- 172.21.15.32:0/1340693462 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f104c077040 0x7f104c079550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:50:05.305 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:05.302+0000 7f1069e00640 1 --2- 172.21.15.32:0/1340693462 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f104c077040 0x7f104c079550 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f105c034000 tx=0x7f105c004730 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:50:05.307 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:05.305+0000 7f1062000640 1 -- 172.21.15.32:0/1340693462 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7f1064062840 con 0x7f106c145c50 2024-08-03T20:50:05.510 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:05.509+0000 7f1071000640 1 -- 172.21.15.32:0/1340693462 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "config rm", "who": "mds", "name": "mds_inject_rename_corrupt_dentry_first"} v 0) -- 0x7f106c070150 con 0x7f106c145c50 2024-08-03T20:50:05.514 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:05.513+0000 7f1062000640 1 -- 172.21.15.32:0/1340693462 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mds", "name": "mds_inject_rename_corrupt_dentry_first"}]=0 v33) ==== 121+0+0 (secure 0 0 0) 0x7f106c070150 con 0x7f106c145c50 2024-08-03T20:50:05.517 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:05.515+0000 7f1071000640 1 -- 172.21.15.32:0/1340693462 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f104c077040 msgr2=0x7f104c079550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:50:05.517 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:05.515+0000 7f1071000640 1 --2- 172.21.15.32:0/1340693462 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f104c077040 0x7f104c079550 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f105c034000 tx=0x7f105c004730 comp rx=0 tx=0).stop 2024-08-03T20:50:05.517 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:05.515+0000 7f1071000640 1 -- 172.21.15.32:0/1340693462 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f106c145c50 msgr2=0x7f106c0c5aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:50:05.518 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:05.515+0000 7f1071000640 1 --2- 172.21.15.32:0/1340693462 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f106c145c50 0x7f106c0c5aa0 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f106400ea40 tx=0x7f106400ef00 comp rx=0 tx=0).stop 2024-08-03T20:50:05.518 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:05.516+0000 7f1071000640 1 -- 172.21.15.32:0/1340693462 shutdown_connections 2024-08-03T20:50:05.518 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:05.516+0000 7f1071000640 1 --2- 172.21.15.32:0/1340693462 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f104c077040 0x7f104c079550 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:05.518 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:05.516+0000 7f1071000640 1 --2- 172.21.15.32:0/1340693462 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f106c145c50 0x7f106c0c5aa0 unknown :-1 s=CLOSED pgs=477 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:05.518 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:05.516+0000 7f1071000640 1 --2- 172.21.15.32:0/1340693462 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f106c0ced30 0x7f106c0c5560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:05.518 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:05.516+0000 7f1071000640 1 --2- 172.21.15.32:0/1340693462 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f106c0cccf0 0x7f106c0c5020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:05.518 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:05.516+0000 7f1071000640 1 -- 172.21.15.32:0/1340693462 >> 172.21.15.32:0/1340693462 conn(0x7f106c0b1bd0 msgr2=0x7f106c0b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:50:05.518 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:05.516+0000 7f1071000640 1 -- 172.21.15.32:0/1340693462 shutdown_connections 2024-08-03T20:50:05.518 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:05.516+0000 7f1071000640 1 -- 172.21.15.32:0/1340693462 wait complete. 2024-08-03T20:50:05.543 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mds mds_abort_on_newly_corrupt_dentry False 2024-08-03T20:50:05.778 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:05.776+0000 7f58b7200640 1 -- 172.21.15.32:0/2951157730 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f58b80d29a0 msgr2=0x7f58b80b7420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:50:05.779 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:05.776+0000 7f58b7200640 1 --2- 172.21.15.32:0/2951157730 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f58b80d29a0 0x7f58b80b7420 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f58a0009830 tx=0x7f58a002a420 comp rx=0 tx=0).stop 2024-08-03T20:50:05.779 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:05.776+0000 7f58b7200640 1 -- 172.21.15.32:0/2951157730 shutdown_connections 2024-08-03T20:50:05.779 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:05.776+0000 7f58b7200640 1 --2- 172.21.15.32:0/2951157730 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f58b80ce700 0x7f58b80ceb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:05.779 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:05.776+0000 7f58b7200640 1 --2- 172.21.15.32:0/2951157730 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f58b80d29a0 0x7f58b80b7420 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:05.779 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:05.776+0000 7f58b7200640 1 -- 172.21.15.32:0/2951157730 >> 172.21.15.32:0/2951157730 conn(0x7f58b80d1e80 msgr2=0x7f58b80d2290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:50:05.779 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:05.776+0000 7f58b7200640 1 -- 172.21.15.32:0/2951157730 shutdown_connections 2024-08-03T20:50:05.779 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:05.777+0000 7f58b7200640 1 -- 172.21.15.32:0/2951157730 wait complete. 2024-08-03T20:50:05.781 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:05.779+0000 7f58b7200640 1 Processor -- start 2024-08-03T20:50:05.781 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:05.779+0000 7f58b7200640 1 -- start start 2024-08-03T20:50:05.783 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:05.780+0000 7f58b7200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f58b80cd5f0 0x7f58b8151f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:50:05.783 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:05.780+0000 7f58b7200640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f58b80ce700 0x7f58b814fb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:50:05.783 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:05.780+0000 7f58b7200640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f58b80d29a0 0x7f58b815e360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:50:05.783 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:05.780+0000 7f58b7200640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f58b80d09f0 con 0x7f58b80cd5f0 2024-08-03T20:50:05.783 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:05.780+0000 7f58b7200640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f58b80d0870 con 0x7f58b80ce700 2024-08-03T20:50:05.783 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:05.780+0000 7f58b7200640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7f58b80d0b70 con 0x7f58b80d29a0 2024-08-03T20:50:05.783 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:05.781+0000 7f58b5e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f58b80cd5f0 0x7f58b8151f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:50:05.784 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:05.781+0000 7f58b5e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f58b80cd5f0 0x7f58b8151f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:50032/0 (socket says 172.21.15.32:50032) 2024-08-03T20:50:05.784 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:05.781+0000 7f58b5e00640 1 -- 172.21.15.32:0/339725281 learned_addr learned my addr 172.21.15.32:0/339725281 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:50:05.784 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:05.781+0000 7f58b6800640 1 --2- 172.21.15.32:0/339725281 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f58b80d29a0 0x7f58b815e360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:50:05.784 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:05.781+0000 7f58afe00640 1 --2- 172.21.15.32:0/339725281 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f58b80ce700 0x7f58b814fb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:50:05.784 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:05.781+0000 7f58b5e00640 1 -- 172.21.15.32:0/339725281 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f58b80d29a0 msgr2=0x7f58b815e360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:50:05.784 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:05.781+0000 7f58b5e00640 1 --2- 172.21.15.32:0/339725281 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f58b80d29a0 0x7f58b815e360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:05.784 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:05.781+0000 7f58b5e00640 1 -- 172.21.15.32:0/339725281 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f58b80ce700 msgr2=0x7f58b814fb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:50:05.784 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:05.781+0000 7f58b5e00640 1 --2- 172.21.15.32:0/339725281 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f58b80ce700 0x7f58b814fb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:05.784 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:05.781+0000 7f58b5e00640 1 -- 172.21.15.32:0/339725281 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f58b815d2c0 con 0x7f58b80cd5f0 2024-08-03T20:50:05.784 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:05.782+0000 7f58b5e00640 1 --2- 172.21.15.32:0/339725281 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f58b80cd5f0 0x7f58b8151f90 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f58a800c760 tx=0x7f58a800cc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:50:05.785 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:05.782+0000 7f58ae000640 1 -- 172.21.15.32:0/339725281 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58a8017070 con 0x7f58b80cd5f0 2024-08-03T20:50:05.785 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:05.782+0000 7f58ae000640 1 -- 172.21.15.32:0/339725281 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f58a80035f0 con 0x7f58b80cd5f0 2024-08-03T20:50:05.785 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:05.782+0000 7f58ae000640 1 -- 172.21.15.32:0/339725281 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58a8003a40 con 0x7f58b80cd5f0 2024-08-03T20:50:05.785 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:05.782+0000 7f58b7200640 1 -- 172.21.15.32:0/339725281 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f58b815d5a0 con 0x7f58b80cd5f0 2024-08-03T20:50:05.786 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:05.782+0000 7f58b7200640 1 -- 172.21.15.32:0/339725281 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f58b815db60 con 0x7f58b80cd5f0 2024-08-03T20:50:05.786 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:05.783+0000 7f58b7200640 1 -- 172.21.15.32:0/339725281 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f58b80d1100 con 0x7f58b80cd5f0 2024-08-03T20:50:05.786 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:05.784+0000 7f58ae000640 1 -- 172.21.15.32:0/339725281 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f58a801a070 con 0x7f58b80cd5f0 2024-08-03T20:50:05.787 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:05.785+0000 7f58ae000640 1 --2- 172.21.15.32:0/339725281 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f5894077010 0x7f5894079520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:50:05.787 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:05.785+0000 7f58ae000640 1 -- 172.21.15.32:0/339725281 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(46..46 src has 1..46) ==== 5795+0+0 (secure 0 0 0) 0x7f58a8098060 con 0x7f58b80cd5f0 2024-08-03T20:50:05.787 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:05.786+0000 7f58afe00640 1 --2- 172.21.15.32:0/339725281 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f5894077010 0x7f5894079520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:50:05.788 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:05.787+0000 7f58afe00640 1 --2- 172.21.15.32:0/339725281 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f5894077010 0x7f5894079520 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f58a0034000 tx=0x7f58a002af90 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:50:05.791 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:05.790+0000 7f58ae000640 1 -- 172.21.15.32:0/339725281 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7f58b80b1100 con 0x7f58b80cd5f0 2024-08-03T20:50:05.996 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:05.995+0000 7f58b7200640 1 -- 172.21.15.32:0/339725281 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command([{prefix=config set, name=mds_abort_on_newly_corrupt_dentry}] v 0) -- 0x7f58b80cf550 con 0x7f58b80cd5f0 2024-08-03T20:50:05.997 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:05.995+0000 7f58ae000640 1 -- 172.21.15.32:0/339725281 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mds_abort_on_newly_corrupt_dentry}]=0 v33)=0 v33) ==== 135+0+0 (secure 0 0 0) 0x7f58a80013e0 con 0x7f58b80cd5f0 2024-08-03T20:50:05.999 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:05.998+0000 7f58b7200640 1 -- 172.21.15.32:0/339725281 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f5894077010 msgr2=0x7f5894079520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:50:06.000 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:05.998+0000 7f58b7200640 1 --2- 172.21.15.32:0/339725281 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f5894077010 0x7f5894079520 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f58a0034000 tx=0x7f58a002af90 comp rx=0 tx=0).stop 2024-08-03T20:50:06.000 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:05.998+0000 7f58b7200640 1 -- 172.21.15.32:0/339725281 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f58b80cd5f0 msgr2=0x7f58b8151f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:50:06.000 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:05.998+0000 7f58b7200640 1 --2- 172.21.15.32:0/339725281 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f58b80cd5f0 0x7f58b8151f90 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f58a800c760 tx=0x7f58a800cc20 comp rx=0 tx=0).stop 2024-08-03T20:50:06.000 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:05.998+0000 7f58b7200640 1 -- 172.21.15.32:0/339725281 shutdown_connections 2024-08-03T20:50:06.000 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:05.998+0000 7f58b7200640 1 --2- 172.21.15.32:0/339725281 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f5894077010 0x7f5894079520 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:06.000 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:05.998+0000 7f58b7200640 1 --2- 172.21.15.32:0/339725281 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f58b80ce700 0x7f58b814fb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:06.000 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:05.998+0000 7f58b7200640 1 --2- 172.21.15.32:0/339725281 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f58b80d29a0 0x7f58b815e360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:06.000 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:05.998+0000 7f58b7200640 1 --2- 172.21.15.32:0/339725281 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f58b80cd5f0 0x7f58b8151f90 unknown :-1 s=CLOSED pgs=478 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:06.000 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:05.998+0000 7f58b7200640 1 -- 172.21.15.32:0/339725281 >> 172.21.15.32:0/339725281 conn(0x7f58b80d1e80 msgr2=0x7f58b8158ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:50:06.000 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:05.998+0000 7f58b7200640 1 -- 172.21.15.32:0/339725281 shutdown_connections 2024-08-03T20:50:06.000 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:05.998+0000 7f58b7200640 1 -- 172.21.15.32:0/339725281 wait complete. 2024-08-03T20:50:06.022 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs joinable true 2024-08-03T20:50:06.256 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:06.254+0000 7f2ce2800640 1 -- 172.21.15.32:0/2350186184 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2cdc0af520 msgr2=0x7f2cdc0af930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:50:06.257 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:06.254+0000 7f2ce2800640 1 --2- 172.21.15.32:0/2350186184 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2cdc0af520 0x7f2cdc0af930 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7f2cc4009830 tx=0x7f2cc402a5b0 comp rx=0 tx=0).stop 2024-08-03T20:50:06.257 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:06.255+0000 7f2ce2800640 1 -- 172.21.15.32:0/2350186184 shutdown_connections 2024-08-03T20:50:06.257 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:06.255+0000 7f2ce2800640 1 --2- 172.21.15.32:0/2350186184 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2cdc0af520 0x7f2cdc0af930 unknown :-1 s=CLOSED pgs=480 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:06.257 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:06.255+0000 7f2ce2800640 1 -- 172.21.15.32:0/2350186184 >> 172.21.15.32:0/2350186184 conn(0x7f2cdc0cd1e0 msgr2=0x7f2cdc0cd5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:50:06.257 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:06.255+0000 7f2ce2800640 1 -- 172.21.15.32:0/2350186184 shutdown_connections 2024-08-03T20:50:06.257 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:06.255+0000 7f2ce2800640 1 -- 172.21.15.32:0/2350186184 wait complete. 2024-08-03T20:50:06.258 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:06.257+0000 7f2ce2800640 1 Processor -- start 2024-08-03T20:50:06.258 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:06.257+0000 7f2ce2800640 1 -- start start 2024-08-03T20:50:06.261 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:06.258+0000 7f2ce2800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2cdc06f600 0x7f2cdc0ca0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:50:06.261 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:06.258+0000 7f2ce2800640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f2cdc075a10 0x7f2cdc0b5010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:50:06.261 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:06.258+0000 7f2ce2800640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f2cdc0af520 0x7f2cdc1f8d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:50:06.261 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:06.258+0000 7f2ce2800640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f2cdc0b0060 con 0x7f2cdc06f600 2024-08-03T20:50:06.261 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:06.258+0000 7f2ce2800640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f2cdc06de40 con 0x7f2cdc0af520 2024-08-03T20:50:06.261 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:06.258+0000 7f2ce2800640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7f2cdc070120 con 0x7f2cdc075a10 2024-08-03T20:50:06.261 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:06.259+0000 7f2cdb400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2cdc06f600 0x7f2cdc0ca0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:50:06.262 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:06.259+0000 7f2cdb400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2cdc06f600 0x7f2cdc0ca0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:50044/0 (socket says 172.21.15.32:50044) 2024-08-03T20:50:06.262 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:06.259+0000 7f2cdb400640 1 -- 172.21.15.32:0/3851487034 learned_addr learned my addr 172.21.15.32:0/3851487034 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:50:06.262 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:06.259+0000 7f2cd3e00640 1 --2- 172.21.15.32:0/3851487034 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f2cdc075a10 0x7f2cdc0b5010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:50:06.262 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:06.259+0000 7f2cdbe00640 1 --2- 172.21.15.32:0/3851487034 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f2cdc0af520 0x7f2cdc1f8d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:50:06.262 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:06.259+0000 7f2cdb400640 1 -- 172.21.15.32:0/3851487034 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f2cdc075a10 msgr2=0x7f2cdc0b5010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:50:06.262 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:06.259+0000 7f2cdb400640 1 --2- 172.21.15.32:0/3851487034 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f2cdc075a10 0x7f2cdc0b5010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:06.262 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:06.259+0000 7f2cdb400640 1 -- 172.21.15.32:0/3851487034 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f2cdc0af520 msgr2=0x7f2cdc1f8d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:50:06.262 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:06.259+0000 7f2cdb400640 1 --2- 172.21.15.32:0/3851487034 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f2cdc0af520 0x7f2cdc1f8d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:06.263 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:06.259+0000 7f2cdb400640 1 -- 172.21.15.32:0/3851487034 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2cdc1f6a70 con 0x7f2cdc06f600 2024-08-03T20:50:06.263 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:06.259+0000 7f2cdb400640 1 --2- 172.21.15.32:0/3851487034 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2cdc06f600 0x7f2cdc0ca0e0 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7f2ccc00c400 tx=0x7f2ccc00c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:50:06.263 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:06.260+0000 7f2cd3400640 1 -- 172.21.15.32:0/3851487034 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ccc011790 con 0x7f2cdc06f600 2024-08-03T20:50:06.263 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:06.260+0000 7f2cd3400640 1 -- 172.21.15.32:0/3851487034 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f2ccc017070 con 0x7f2cdc06f600 2024-08-03T20:50:06.263 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:06.260+0000 7f2ce2800640 1 -- 172.21.15.32:0/3851487034 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2cdc0b5d80 con 0x7f2cdc06f600 2024-08-03T20:50:06.263 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:06.260+0000 7f2cd3400640 1 -- 172.21.15.32:0/3851487034 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ccc011ce0 con 0x7f2cdc06f600 2024-08-03T20:50:06.263 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:06.260+0000 7f2ce2800640 1 -- 172.21.15.32:0/3851487034 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2cdc0ca620 con 0x7f2cdc06f600 2024-08-03T20:50:06.264 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:06.261+0000 7f2ce2800640 1 -- 172.21.15.32:0/3851487034 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2cdc0af930 con 0x7f2cdc06f600 2024-08-03T20:50:06.264 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:06.262+0000 7f2cd3400640 1 -- 172.21.15.32:0/3851487034 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f2ccc01c460 con 0x7f2cdc06f600 2024-08-03T20:50:06.264 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:06.263+0000 7f2cd3400640 1 --2- 172.21.15.32:0/3851487034 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f2cb8077010 0x7f2cb8079520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:50:06.264 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:06.263+0000 7f2cd3400640 1 -- 172.21.15.32:0/3851487034 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(46..46 src has 1..46) ==== 5795+0+0 (secure 0 0 0) 0x7f2ccc0987c0 con 0x7f2cdc06f600 2024-08-03T20:50:06.265 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:06.263+0000 7f2cd3e00640 1 --2- 172.21.15.32:0/3851487034 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f2cb8077010 0x7f2cb8079520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:50:06.265 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:06.264+0000 7f2cd3e00640 1 --2- 172.21.15.32:0/3851487034 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f2cb8077010 0x7f2cb8079520 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f2cc4034000 tx=0x7f2cc4004730 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:50:06.268 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:06.267+0000 7f2cd3400640 1 -- 172.21.15.32:0/3851487034 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7f2ccc0619e0 con 0x7f2cdc06f600 2024-08-03T20:50:06.513 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:06.511+0000 7f2ce2800640 1 -- 172.21.15.32:0/3851487034 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "joinable", "val": "true"} v 0) -- 0x7f2cdc0aeff0 con 0x7f2cdc06f600 2024-08-03T20:50:06.754 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-08-03T20:50:06.759 DEBUG:teuthology.orchestra.run.smithi118:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-08-03T20:50:07.041 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:07.039+0000 7f2cd3400640 1 -- 172.21.15.32:0/3851487034 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "joinable", "val": "true"}]=0 cephfs marked joinable; MDS may join as newly active. v38) ==== 162+0+0 (secure 0 0 0) 0x7f2ccc0612e0 con 0x7f2cdc06f600 2024-08-03T20:50:07.041 INFO:teuthology.orchestra.run.smithi032.stderr:cephfs marked joinable; MDS may join as newly active. 2024-08-03T20:50:07.043 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:07.042+0000 7f2ce2800640 1 -- 172.21.15.32:0/3851487034 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f2cb8077010 msgr2=0x7f2cb8079520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:50:07.043 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:07.042+0000 7f2ce2800640 1 --2- 172.21.15.32:0/3851487034 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f2cb8077010 0x7f2cb8079520 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f2cc4034000 tx=0x7f2cc4004730 comp rx=0 tx=0).stop 2024-08-03T20:50:07.044 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:07.042+0000 7f2ce2800640 1 -- 172.21.15.32:0/3851487034 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2cdc06f600 msgr2=0x7f2cdc0ca0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:50:07.044 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:07.042+0000 7f2ce2800640 1 --2- 172.21.15.32:0/3851487034 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2cdc06f600 0x7f2cdc0ca0e0 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7f2ccc00c400 tx=0x7f2ccc00c8c0 comp rx=0 tx=0).stop 2024-08-03T20:50:07.044 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:07.042+0000 7f2ce2800640 1 -- 172.21.15.32:0/3851487034 shutdown_connections 2024-08-03T20:50:07.044 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:07.042+0000 7f2ce2800640 1 --2- 172.21.15.32:0/3851487034 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f2cb8077010 0x7f2cb8079520 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:07.044 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:07.042+0000 7f2ce2800640 1 --2- 172.21.15.32:0/3851487034 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f2cdc0af520 0x7f2cdc1f8d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:07.044 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:07.042+0000 7f2ce2800640 1 --2- 172.21.15.32:0/3851487034 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f2cdc075a10 0x7f2cdc0b5010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:07.044 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:07.042+0000 7f2ce2800640 1 --2- 172.21.15.32:0/3851487034 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2cdc06f600 0x7f2cdc0ca0e0 unknown :-1 s=CLOSED pgs=481 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:07.044 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:07.042+0000 7f2ce2800640 1 -- 172.21.15.32:0/3851487034 >> 172.21.15.32:0/3851487034 conn(0x7f2cdc0cd1e0 msgr2=0x7f2cdc0ae5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:50:07.044 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:07.042+0000 7f2ce2800640 1 -- 172.21.15.32:0/3851487034 shutdown_connections 2024-08-03T20:50:07.044 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:07.042+0000 7f2ce2800640 1 -- 172.21.15.32:0/3851487034 wait complete. 2024-08-03T20:50:07.074 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-08-03T20:50:07.303 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:07.301+0000 7f839b200640 1 -- 172.21.15.32:0/1755347177 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f839406e930 msgr2=0x7f8394144f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:50:07.303 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:07.301+0000 7f839b200640 1 --2- 172.21.15.32:0/1755347177 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f839406e930 0x7f8394144f90 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f8394071700 tx=0x7f838402a4b0 comp rx=0 tx=0).stop 2024-08-03T20:50:07.303 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:07.301+0000 7f839b200640 1 -- 172.21.15.32:0/1755347177 shutdown_connections 2024-08-03T20:50:07.304 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:07.301+0000 7f839b200640 1 --2- 172.21.15.32:0/1755347177 >> v2:172.21.15.32:3300/0 conn(0x7f8394073e60 0x7f8394074270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:07.304 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:07.301+0000 7f839b200640 1 --2- 172.21.15.32:0/1755347177 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f839406e930 0x7f8394144f90 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:07.304 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:07.301+0000 7f839b200640 1 -- 172.21.15.32:0/1755347177 >> 172.21.15.32:0/1755347177 conn(0x7f83940b1bd0 msgr2=0x7f83940b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:50:07.304 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:07.301+0000 7f839b200640 1 -- 172.21.15.32:0/1755347177 shutdown_connections 2024-08-03T20:50:07.304 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:07.301+0000 7f839b200640 1 -- 172.21.15.32:0/1755347177 wait complete. 2024-08-03T20:50:07.305 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:07.304+0000 7f839b200640 1 Processor -- start 2024-08-03T20:50:07.305 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:07.304+0000 7f839b200640 1 -- start start 2024-08-03T20:50:07.307 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:07.305+0000 7f839b200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f839406e930 0x7f839415f440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:50:07.307 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:07.305+0000 7f839b200640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f8394073e60 0x7f83941f8970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:50:07.307 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:07.305+0000 7f839b200640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f83940747b0 0x7f83941f8eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:50:07.307 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:07.305+0000 7f839b200640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f83940cf9c0 con 0x7f839406e930 2024-08-03T20:50:07.307 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:07.305+0000 7f839b200640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f83940cf840 con 0x7f83940747b0 2024-08-03T20:50:07.307 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:07.305+0000 7f839b200640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7f83940ccff0 con 0x7f8394073e60 2024-08-03T20:50:07.308 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:07.305+0000 7f8393e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f839406e930 0x7f839415f440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:50:07.308 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:07.306+0000 7f8393e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f839406e930 0x7f839415f440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:50062/0 (socket says 172.21.15.32:50062) 2024-08-03T20:50:07.308 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:07.306+0000 7f8393e00640 1 -- 172.21.15.32:0/3667032525 learned_addr learned my addr 172.21.15.32:0/3667032525 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:50:07.308 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:07.306+0000 7f8393400640 1 --2- 172.21.15.32:0/3667032525 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f8394073e60 0x7f83941f8970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:50:07.308 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:07.306+0000 7f8399200640 1 --2- 172.21.15.32:0/3667032525 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f83940747b0 0x7f83941f8eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:50:07.308 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:07.306+0000 7f8393e00640 1 -- 172.21.15.32:0/3667032525 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f8394073e60 msgr2=0x7f83941f8970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:50:07.308 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:07.306+0000 7f8393e00640 1 --2- 172.21.15.32:0/3667032525 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f8394073e60 0x7f83941f8970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:07.309 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:07.306+0000 7f8393e00640 1 -- 172.21.15.32:0/3667032525 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f83940747b0 msgr2=0x7f83941f8eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:50:07.309 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:07.306+0000 7f8393e00640 1 --2- 172.21.15.32:0/3667032525 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f83940747b0 0x7f83941f8eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:07.309 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:07.306+0000 7f8393e00640 1 -- 172.21.15.32:0/3667032525 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8394143cd0 con 0x7f839406e930 2024-08-03T20:50:07.309 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:07.306+0000 7f8393e00640 1 --2- 172.21.15.32:0/3667032525 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f839406e930 0x7f839415f440 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7f8384009780 tx=0x7f8384033c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:50:07.309 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:07.307+0000 7f838be00640 1 -- 172.21.15.32:0/3667032525 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8384031d60 con 0x7f839406e930 2024-08-03T20:50:07.309 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:07.307+0000 7f838be00640 1 -- 172.21.15.32:0/3667032525 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f8384033e10 con 0x7f839406e930 2024-08-03T20:50:07.309 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:07.307+0000 7f838be00640 1 -- 172.21.15.32:0/3667032525 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8384005260 con 0x7f839406e930 2024-08-03T20:50:07.309 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:07.307+0000 7f839b200640 1 -- 172.21.15.32:0/3667032525 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8394143fb0 con 0x7f839406e930 2024-08-03T20:50:07.309 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:07.307+0000 7f839b200640 1 -- 172.21.15.32:0/3667032525 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f839413e890 con 0x7f839406e930 2024-08-03T20:50:07.310 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:07.308+0000 7f839b200640 1 -- 172.21.15.32:0/3667032525 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f839413eaf0 con 0x7f839406e930 2024-08-03T20:50:07.310 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:07.309+0000 7f838be00640 1 -- 172.21.15.32:0/3667032525 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f8384005400 con 0x7f839406e930 2024-08-03T20:50:07.311 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:07.310+0000 7f838be00640 1 --2- 172.21.15.32:0/3667032525 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f8374077010 0x7f8374079520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:50:07.311 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:07.310+0000 7f838be00640 1 -- 172.21.15.32:0/3667032525 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(46..46 src has 1..46) ==== 5795+0+0 (secure 0 0 0) 0x7f83840b9b00 con 0x7f839406e930 2024-08-03T20:50:07.312 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:07.310+0000 7f8393400640 1 --2- 172.21.15.32:0/3667032525 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f8374077010 0x7f8374079520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:50:07.312 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:07.311+0000 7f8393400640 1 --2- 172.21.15.32:0/3667032525 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f8374077010 0x7f8374079520 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f83941411b0 tx=0x7f8380005f90 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:50:07.320 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:07.319+0000 7f838be00640 1 -- 172.21.15.32:0/3667032525 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7f838403f030 con 0x7f839406e930 2024-08-03T20:50:07.567 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:07.565+0000 7f839b200640 1 -- 172.21.15.32:0/3667032525 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f83940cd580 con 0x7f839406e930 2024-08-03T20:50:07.567 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:07.566+0000 7f838be00640 1 -- 172.21.15.32:0/3667032525 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 39 v39) ==== 94+0+5223 (secure 0 0 0) 0x7f8384082530 con 0x7f839406e930 2024-08-03T20:50:07.568 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-03T20:50:07.568 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":39,"btime":"2024-08-03T20:50:07:043541+0000","default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4997,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6837/2688500839","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6836","nonce":2688500839},{"type":"v1","addr":"172.21.15.118:6837","nonce":2688500839}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":30},{"gid":5266,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6835/2381075275","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6834","nonce":2381075275},{"type":"v1","addr":"172.21.15.118:6835","nonce":2381075275}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":28},{"gid":5421,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6833/1058332640","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":1058332640},{"type":"v1","addr":"172.21.15.32:6833","nonce":1058332640}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":29}],"filesystems":[{"mdsmap":{"epoch":39,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-08-03T20:49:47.064654+0000","modified":"2024-08-03T20:50:07.043523+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":46,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5628},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5628":{"gid":5628,"name":"c","rank":0,"incarnation":39,"state":"up:replay","state_seq":1,"addr":"172.21.15.32:6835/1726912144","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1726912144},{"type":"v1","addr":"172.21.15.32:6835","nonce":1726912144}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":3}]} 2024-08-03T20:50:07.568 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 39 2024-08-03T20:50:07.570 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:07.568+0000 7f839b200640 1 -- 172.21.15.32:0/3667032525 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f8374077010 msgr2=0x7f8374079520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:50:07.570 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:07.568+0000 7f839b200640 1 --2- 172.21.15.32:0/3667032525 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f8374077010 0x7f8374079520 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f83941411b0 tx=0x7f8380005f90 comp rx=0 tx=0).stop 2024-08-03T20:50:07.570 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:07.569+0000 7f839b200640 1 -- 172.21.15.32:0/3667032525 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f839406e930 msgr2=0x7f839415f440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:50:07.571 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:07.569+0000 7f839b200640 1 --2- 172.21.15.32:0/3667032525 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f839406e930 0x7f839415f440 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7f8384009780 tx=0x7f8384033c20 comp rx=0 tx=0).stop 2024-08-03T20:50:07.571 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:07.569+0000 7f839b200640 1 -- 172.21.15.32:0/3667032525 shutdown_connections 2024-08-03T20:50:07.571 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:07.569+0000 7f839b200640 1 --2- 172.21.15.32:0/3667032525 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f8374077010 0x7f8374079520 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:07.571 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:07.569+0000 7f839b200640 1 --2- 172.21.15.32:0/3667032525 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f83940747b0 0x7f83941f8eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:07.571 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:07.569+0000 7f839b200640 1 --2- 172.21.15.32:0/3667032525 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f8394073e60 0x7f83941f8970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:07.571 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:07.569+0000 7f839b200640 1 --2- 172.21.15.32:0/3667032525 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f839406e930 0x7f839415f440 unknown :-1 s=CLOSED pgs=483 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:07.571 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:07.569+0000 7f839b200640 1 -- 172.21.15.32:0/3667032525 >> 172.21.15.32:0/3667032525 conn(0x7f83940b1bd0 msgr2=0x7f83940b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:50:07.571 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:07.569+0000 7f839b200640 1 -- 172.21.15.32:0/3667032525 shutdown_connections 2024-08-03T20:50:07.571 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:07.569+0000 7f839b200640 1 -- 172.21.15.32:0/3667032525 wait complete. 2024-08-03T20:50:07.597 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.3:0 flush journal 2024-08-03T20:50:07.833 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:07.831+0000 7f1999200640 1 -- 172.21.15.32:0/4211138611 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f199415c3b0 msgr2=0x7f19940cf750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:50:07.833 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:07.831+0000 7f1999200640 1 --2- 172.21.15.32:0/4211138611 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f199415c3b0 0x7f19940cf750 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f198c005fd0 tx=0x7f198c02a8b0 comp rx=0 tx=0).stop 2024-08-03T20:50:07.833 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:07.831+0000 7f1999200640 1 -- 172.21.15.32:0/4211138611 shutdown_connections 2024-08-03T20:50:07.833 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:07.831+0000 7f1999200640 1 --2- 172.21.15.32:0/4211138611 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f199415c3b0 0x7f19940cf750 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:07.834 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:07.831+0000 7f1999200640 1 -- 172.21.15.32:0/4211138611 >> 172.21.15.32:0/4211138611 conn(0x7f19940b32f0 msgr2=0x7f19940d1050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:50:07.834 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:07.831+0000 7f1999200640 1 -- 172.21.15.32:0/4211138611 shutdown_connections 2024-08-03T20:50:07.834 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:07.831+0000 7f1999200640 1 -- 172.21.15.32:0/4211138611 wait complete. 2024-08-03T20:50:07.835 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:07.834+0000 7f1999200640 1 Processor -- start 2024-08-03T20:50:07.835 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:07.834+0000 7f1999200640 1 -- start start 2024-08-03T20:50:07.838 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:07.835+0000 7f1999200640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f1994152480 0x7f19940cae30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:50:07.838 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:07.835+0000 7f1999200640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f199415c3b0 0x7f199406c9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:50:07.838 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:07.835+0000 7f1999200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f19940b1e40 0x7f199406cf20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:50:07.838 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:07.835+0000 7f1999200640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f19940cfe10 con 0x7f19940b1e40 2024-08-03T20:50:07.838 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:07.835+0000 7f1999200640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f19940cfc90 con 0x7f1994152480 2024-08-03T20:50:07.838 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:07.835+0000 7f1999200640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7f19940cff90 con 0x7f199415c3b0 2024-08-03T20:50:07.838 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:07.836+0000 7f1992800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f19940b1e40 0x7f199406cf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:50:07.838 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:07.836+0000 7f1992800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f19940b1e40 0x7f199406cf20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:50068/0 (socket says 172.21.15.32:50068) 2024-08-03T20:50:07.838 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:07.836+0000 7f1992800640 1 -- 172.21.15.32:0/1515659018 learned_addr learned my addr 172.21.15.32:0/1515659018 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:50:07.838 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:07.836+0000 7f1991e00640 1 --2- 172.21.15.32:0/1515659018 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f1994152480 0x7f19940cae30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:50:07.839 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:07.836+0000 7f198be00640 1 --2- 172.21.15.32:0/1515659018 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f199415c3b0 0x7f199406c9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:50:07.839 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:07.836+0000 7f1992800640 1 -- 172.21.15.32:0/1515659018 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f199415c3b0 msgr2=0x7f199406c9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:50:07.839 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:07.836+0000 7f1992800640 1 --2- 172.21.15.32:0/1515659018 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f199415c3b0 0x7f199406c9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:07.839 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:07.836+0000 7f1992800640 1 -- 172.21.15.32:0/1515659018 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f1994152480 msgr2=0x7f19940cae30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:50:07.839 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:07.836+0000 7f1992800640 1 --2- 172.21.15.32:0/1515659018 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f1994152480 0x7f19940cae30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:07.839 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:07.836+0000 7f1992800640 1 -- 172.21.15.32:0/1515659018 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f19940cb370 con 0x7f19940b1e40 2024-08-03T20:50:07.839 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:07.836+0000 7f1992800640 1 --2- 172.21.15.32:0/1515659018 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f19940b1e40 0x7f199406cf20 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f198c02aea0 tx=0x7f198c031670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:50:07.840 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:07.837+0000 7f198a000640 1 -- 172.21.15.32:0/1515659018 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f198c003e00 con 0x7f19940b1e40 2024-08-03T20:50:07.840 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:07.837+0000 7f198a000640 1 -- 172.21.15.32:0/1515659018 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f198c031a80 con 0x7f19940b1e40 2024-08-03T20:50:07.840 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:07.837+0000 7f198a000640 1 -- 172.21.15.32:0/1515659018 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f198c0330d0 con 0x7f19940b1e40 2024-08-03T20:50:07.840 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:07.837+0000 7f1999200640 1 -- 172.21.15.32:0/1515659018 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f19940cb7f0 con 0x7f19940b1e40 2024-08-03T20:50:07.840 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:07.837+0000 7f1999200640 1 -- 172.21.15.32:0/1515659018 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f19940b1360 con 0x7f19940b1e40 2024-08-03T20:50:07.842 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:07.841+0000 7f198a000640 1 -- 172.21.15.32:0/1515659018 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f198c033270 con 0x7f19940b1e40 2024-08-03T20:50:07.843 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:07.841+0000 7f198a000640 1 --2- 172.21.15.32:0/1515659018 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f1970077040 0x7f1970079550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:50:07.843 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:07.842+0000 7f198a000640 1 -- 172.21.15.32:0/1515659018 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(46..46 src has 1..46) ==== 5795+0+0 (secure 0 0 0) 0x7f198c0b9c50 con 0x7f19940b1e40 2024-08-03T20:50:07.843 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:07.842+0000 7f19a0363740 1 Processor -- start 2024-08-03T20:50:07.843 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:07.842+0000 7f19a0363740 1 -- start start 2024-08-03T20:50:07.843 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:07.842+0000 7f1991e00640 1 --2- 172.21.15.32:0/1515659018 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f1970077040 0x7f1970079550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:50:07.844 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:07.843+0000 7f1991e00640 1 --2- 172.21.15.32:0/1515659018 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f1970077040 0x7f1970079550 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f1984001770 tx=0x7f1984009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:50:07.846 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:07.843+0000 7f19a0363740 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55a651b15390 0x55a651ee56f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:50:07.846 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:07.843+0000 7f19a0363740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55a651aec4d0 0x55a651ee5c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:50:07.846 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:07.843+0000 7f19a0363740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a651af46b0 0x55a651f02b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:50:07.846 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:07.843+0000 7f19a0363740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55a651e2e470 con 0x55a651af46b0 2024-08-03T20:50:07.846 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:07.843+0000 7f19a0363740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x55a651e1d970 con 0x55a651aec4d0 2024-08-03T20:50:07.846 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:07.843+0000 7f19a0363740 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x55a651e219b0 con 0x55a651b15390 2024-08-03T20:50:07.846 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:07.844+0000 7f1992800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a651af46b0 0x55a651f02b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:50:07.846 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:07.844+0000 7f1992800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a651af46b0 0x55a651f02b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:50082/0 (socket says 172.21.15.32:50082) 2024-08-03T20:50:07.846 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:07.844+0000 7f1992800640 1 -- 172.21.15.32:0/3295844745 learned_addr learned my addr 172.21.15.32:0/3295844745 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:50:07.846 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:07.844+0000 7f1991e00640 1 --2- 172.21.15.32:0/3295844745 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55a651b15390 0x55a651ee56f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:50:07.847 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:07.844+0000 7f198be00640 1 --2- 172.21.15.32:0/3295844745 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55a651aec4d0 0x55a651ee5c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:50:07.847 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:07.844+0000 7f1992800640 1 -- 172.21.15.32:0/3295844745 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55a651b15390 msgr2=0x55a651ee56f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:50:07.847 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:07.844+0000 7f1992800640 1 --2- 172.21.15.32:0/3295844745 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55a651b15390 0x55a651ee56f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:07.847 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:07.844+0000 7f1992800640 1 -- 172.21.15.32:0/3295844745 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55a651aec4d0 msgr2=0x55a651ee5c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:50:07.847 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:07.844+0000 7f1992800640 1 --2- 172.21.15.32:0/3295844745 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55a651aec4d0 0x55a651ee5c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:07.847 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:07.844+0000 7f1992800640 1 -- 172.21.15.32:0/3295844745 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a651bf77d0 con 0x55a651af46b0 2024-08-03T20:50:07.847 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:07.844+0000 7f1991e00640 1 --2- 172.21.15.32:0/3295844745 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55a651b15390 0x55a651ee56f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-03T20:50:07.847 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:07.844+0000 7f1992800640 1 --2- 172.21.15.32:0/3295844745 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a651af46b0 0x55a651f02b30 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7f198c082610 tx=0x7f198c082450 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c00e6f77e739b488 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:50:07.847 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:07.844+0000 7f1982a00640 1 -- 172.21.15.32:0/3295844745 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f198c048040 con 0x55a651af46b0 2024-08-03T20:50:07.847 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:07.844+0000 7f1982a00640 1 -- 172.21.15.32:0/3295844745 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f198c082190 con 0x55a651af46b0 2024-08-03T20:50:07.847 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:07.844+0000 7f1982a00640 1 -- 172.21.15.32:0/3295844745 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f198c082b20 con 0x55a651af46b0 2024-08-03T20:50:07.848 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:07.845+0000 7f19a0363740 1 -- 172.21.15.32:0/3295844745 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a651af46b0 msgr2=0x55a651f02b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:50:07.848 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:07.845+0000 7f19a0363740 1 --2- 172.21.15.32:0/3295844745 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a651af46b0 0x55a651f02b30 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7f198c082610 tx=0x7f198c082450 comp rx=0 tx=0).stop 2024-08-03T20:50:07.849 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:07.845+0000 7f19a0363740 1 -- 172.21.15.32:0/3295844745 shutdown_connections 2024-08-03T20:50:07.849 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:07.845+0000 7f19a0363740 1 --2- 172.21.15.32:0/3295844745 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55a651aec4d0 0x55a651ee5c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:07.849 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:07.845+0000 7f19a0363740 1 --2- 172.21.15.32:0/3295844745 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a651af46b0 0x55a651f02b30 unknown :-1 s=CLOSED pgs=486 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:07.849 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:07.845+0000 7f19a0363740 1 --2- 172.21.15.32:0/3295844745 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55a651b15390 0x55a651ee56f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:07.849 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:07.845+0000 7f19a0363740 1 -- 172.21.15.32:0/3295844745 >> 172.21.15.32:0/3295844745 conn(0x55a651b094e0 msgr2=0x55a651ebb9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:50:07.849 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:07.845+0000 7f19a0363740 1 -- 172.21.15.32:0/3295844745 shutdown_connections 2024-08-03T20:50:07.849 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:07.846+0000 7f19a0363740 1 -- 172.21.15.32:0/3295844745 wait complete. 2024-08-03T20:50:07.849 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:07.847+0000 7f19a0363740 1 -- start start 2024-08-03T20:50:07.849 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:07.847+0000 7f19a0363740 1 Processor -- start 2024-08-03T20:50:07.852 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:07.848+0000 7f19a0363740 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55a651aec4d0 0x55a651f00ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:50:07.852 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:07.848+0000 7f19a0363740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55a651af46b0 0x55a651ed1dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:50:07.852 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:07.848+0000 7f19a0363740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a651b15390 0x55a651f12a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:50:07.852 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:07.848+0000 7f19a0363740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55a651d55c90 con 0x55a651b15390 2024-08-03T20:50:07.852 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:07.848+0000 7f19a0363740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x55a651db7fd0 con 0x55a651af46b0 2024-08-03T20:50:07.852 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:07.848+0000 7f19a0363740 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x55a651ef8df0 con 0x55a651aec4d0 2024-08-03T20:50:07.852 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:07.848+0000 7f19a0363740 10 client.0 _ll_register_callbacks cb 0x55a651dd2c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-03T20:50:07.852 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:07.849+0000 7f1992800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a651b15390 0x55a651f12a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:50:07.852 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:07.849+0000 7f1992800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a651b15390 0x55a651f12a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:50094/0 (socket says 172.21.15.32:50094) 2024-08-03T20:50:07.852 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:07.849+0000 7f1992800640 1 -- 172.21.15.32:0/2493861753 learned_addr learned my addr 172.21.15.32:0/2493861753 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:50:07.853 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:07.849+0000 7f1991e00640 1 --2- 172.21.15.32:0/2493861753 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55a651aec4d0 0x55a651f00ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:50:07.853 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:07.849+0000 7f198be00640 1 --2- 172.21.15.32:0/2493861753 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55a651af46b0 0x55a651ed1dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:50:07.853 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:07.849+0000 7f1992800640 1 -- 172.21.15.32:0/2493861753 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55a651aec4d0 msgr2=0x55a651f00ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:50:07.853 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:07.849+0000 7f1992800640 1 --2- 172.21.15.32:0/2493861753 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55a651aec4d0 0x55a651f00ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:07.853 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:07.849+0000 7f1992800640 1 -- 172.21.15.32:0/2493861753 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55a651af46b0 msgr2=0x55a651ed1dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:50:07.853 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:07.849+0000 7f1992800640 1 --2- 172.21.15.32:0/2493861753 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55a651af46b0 0x55a651ed1dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:07.853 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:07.849+0000 7f1992800640 1 -- 172.21.15.32:0/2493861753 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a651b80970 con 0x55a651b15390 2024-08-03T20:50:07.853 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:07.850+0000 7f1992800640 1 --2- 172.21.15.32:0/2493861753 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a651b15390 0x55a651f12a70 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7f198c086040 tx=0x7f198c08b420 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fb2f8f91f23b09fe server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:50:07.854 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:07.850+0000 7f1977400640 10 client.0 ms_handle_connect on v2:172.21.15.32:3300/0 2024-08-03T20:50:07.854 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:07.850+0000 7f1977400640 1 -- 172.21.15.32:0/2493861753 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f198c048040 con 0x55a651b15390 2024-08-03T20:50:07.854 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:07.850+0000 7f1977400640 1 -- 172.21.15.32:0/2493861753 <== mon.0 v2:172.21.15.32:3300/0 2 ==== osd_map(46..46 src has 1..46) ==== 5795+0+0 (secure 0 0 0) 0x7f198c033770 con 0x55a651b15390 2024-08-03T20:50:07.854 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:07.850+0000 7f19a0363740 1 -- 172.21.15.32:0/2493861753 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f198c082190 con 0x55a651b15390 2024-08-03T20:50:07.854 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:07.850+0000 7f1977400640 1 -- 172.21.15.32:0/2493861753 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) ==== 24+0+0 (secure 0 0 0) 0x7f198c082680 con 0x55a651b15390 2024-08-03T20:50:07.854 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:07.850+0000 7f19a0363740 10 client.5667 fetch_fsmap learned FSMap version 39 2024-08-03T20:50:07.854 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:07.850+0000 7f19a0363740 1 -- 172.21.15.32:0/2493861753 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=2+}) -- 0x55a651ac6270 con 0x55a651b15390 2024-08-03T20:50:07.854 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:07.850+0000 7f1977400640 1 -- 172.21.15.32:0/2493861753 <== mon.0 v2:172.21.15.32:3300/0 4 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f198c082190 con 0x55a651b15390 2024-08-03T20:50:07.854 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:07.850+0000 7f1977400640 1 -- 172.21.15.32:0/2493861753 <== mon.0 v2:172.21.15.32:3300/0 5 ==== fsmap(e 39) ==== 3494+0+0 (secure 0 0 0) 0x7f198c004b40 con 0x55a651b15390 2024-08-03T20:50:07.854 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:07.850+0000 7f19a0363740 10 client.5667 fetch_fsmap finished waiting for FSMap version 39 2024-08-03T20:50:07.855 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:07.850+0000 7f19a0363740 10 client.5667 resolve_mds: resolved 3:0 to role '3:0' aka daemon mds.c 2024-08-03T20:50:07.855 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:07.850+0000 7f19a0363740 20 client.5667 populate_metadata read hostname 'smithi032' 2024-08-03T20:50:07.855 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:07.850+0000 7f19a0363740 1 --2- 172.21.15.32:0/2493861753 >> [v2:172.21.15.32:6834/1726912144,v1:172.21.15.32:6835/1726912144] conn(0x55a651ad3f60 0x55a651ef7850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:50:07.855 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:07.850+0000 7f19a0363740 4 client.5667 mds_command: new command op to 5628 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-08-03T20:50:07.855 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:07.850+0000 7f19a0363740 1 -- 172.21.15.32:0/2493861753 --> [v2:172.21.15.32:6834/1726912144,v1:172.21.15.32:6835/1726912144] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a651e303c0 con 0x55a651ad3f60 2024-08-03T20:50:07.856 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:07.851+0000 7f1991e00640 1 --2- 172.21.15.32:0/2493861753 >> [v2:172.21.15.32:6834/1726912144,v1:172.21.15.32:6835/1726912144] conn(0x55a651ad3f60 0x55a651ef7850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:50:07.856 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:07.852+0000 7f1977400640 10 client.5667 ms_handle_connect on v2:172.21.15.32:6834/1726912144 2024-08-03T20:50:07.856 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:07.852+0000 7f1991e00640 1 --2- 172.21.15.32:0/2493861753 >> [v2:172.21.15.32:6834/1726912144,v1:172.21.15.32:6835/1726912144] conn(0x55a651ad3f60 0x55a651ef7850 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=18f7f1aa9856760c server_cookie=fef9514bc8af3ba5 in_seq=0 out_seq=0 2024-08-03T20:50:07.856 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:07.853+0000 7f1977400640 1 -- 172.21.15.32:0/2493861753 <== mds.0 v2:172.21.15.32:6834/1726912144 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55a651e303c0 con 0x55a651ad3f60 2024-08-03T20:50:07.857 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:07.853+0000 7f1977400640 10 client.5667 handle_command_reply: tid=0 2024-08-03T20:50:07.857 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:07.853+0000 7f19a0363740 1 client.5667 shutdown 2024-08-03T20:50:07.857 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:07.853+0000 7f19a0363740 1 -- 172.21.15.32:0/2493861753 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a651b15390 msgr2=0x55a651f12a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:50:07.857 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:07.853+0000 7f19a0363740 1 --2- 172.21.15.32:0/2493861753 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a651b15390 0x55a651f12a70 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7f198c086040 tx=0x7f198c08b420 comp rx=0 tx=0).stop 2024-08-03T20:50:07.857 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:07.853+0000 7f19a0363740 1 -- 172.21.15.32:0/2493861753 shutdown_connections 2024-08-03T20:50:07.857 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:07.853+0000 7f19a0363740 1 --2- 172.21.15.32:0/2493861753 >> [v2:172.21.15.32:6834/1726912144,v1:172.21.15.32:6835/1726912144] conn(0x55a651ad3f60 0x55a651ef7850 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:07.857 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:07.853+0000 7f19a0363740 1 --2- 172.21.15.32:0/2493861753 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55a651af46b0 0x55a651ed1dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:07.857 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:07.853+0000 7f19a0363740 1 --2- 172.21.15.32:0/2493861753 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a651b15390 0x55a651f12a70 unknown :-1 s=CLOSED pgs=487 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:07.857 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:07.853+0000 7f1977400640 0 client.5667 ms_handle_reset on v2:172.21.15.32:6834/1726912144 2024-08-03T20:50:07.857 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:07.853+0000 7f19a0363740 1 --2- 172.21.15.32:0/2493861753 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55a651aec4d0 0x55a651f00ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:07.857 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:07.853+0000 7f19a0363740 1 -- 172.21.15.32:0/2493861753 >> 172.21.15.32:0/2493861753 conn(0x55a651b094e0 msgr2=0x55a651ebb860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:50:07.857 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:07.854+0000 7f19a0363740 1 -- 172.21.15.32:0/2493861753 shutdown_connections 2024-08-03T20:50:07.857 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:07.854+0000 7f19a0363740 1 -- 172.21.15.32:0/2493861753 wait complete. 2024-08-03T20:50:07.857 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:07.854+0000 7f19a0363740 20 client.5667 trim_cache size 0 max 16384 2024-08-03T20:50:07.890 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:07.889+0000 7f19a0363740 1 Processor -- start 2024-08-03T20:50:07.890 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:07.889+0000 7f19a0363740 1 -- start start 2024-08-03T20:50:07.893 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:07.891+0000 7f19a0363740 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55a651aec4d0 0x55a651f00b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:50:07.893 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:07.891+0000 7f19a0363740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55a651af46b0 0x55a651eff350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:50:07.893 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:07.891+0000 7f19a0363740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a651b15390 0x55a651eff890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:50:07.893 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:07.891+0000 7f19a0363740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f1994202060 con 0x55a651b15390 2024-08-03T20:50:07.893 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:07.891+0000 7f19a0363740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f1994201060 con 0x55a651af46b0 2024-08-03T20:50:07.894 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:07.891+0000 7f19a0363740 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7f19941f9780 con 0x55a651aec4d0 2024-08-03T20:50:07.894 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:07.891+0000 7f1992800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a651b15390 0x55a651eff890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:50:07.894 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:07.891+0000 7f1992800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a651b15390 0x55a651eff890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:50102/0 (socket says 172.21.15.32:50102) 2024-08-03T20:50:07.894 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:07.891+0000 7f1992800640 1 -- 172.21.15.32:0/206583888 learned_addr learned my addr 172.21.15.32:0/206583888 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:50:07.894 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:07.892+0000 7f1991e00640 1 --2- 172.21.15.32:0/206583888 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55a651aec4d0 0x55a651f00b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:50:07.894 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:07.892+0000 7f198be00640 1 --2- 172.21.15.32:0/206583888 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55a651af46b0 0x55a651eff350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:50:07.894 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:07.892+0000 7f1992800640 1 -- 172.21.15.32:0/206583888 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55a651aec4d0 msgr2=0x55a651f00b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:50:07.894 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:07.892+0000 7f1992800640 1 --2- 172.21.15.32:0/206583888 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55a651aec4d0 0x55a651f00b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:07.894 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:07.892+0000 7f1992800640 1 -- 172.21.15.32:0/206583888 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55a651af46b0 msgr2=0x55a651eff350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:50:07.894 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:07.892+0000 7f1992800640 1 --2- 172.21.15.32:0/206583888 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55a651af46b0 0x55a651eff350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:07.894 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:07.892+0000 7f1992800640 1 -- 172.21.15.32:0/206583888 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a651ed3e00 con 0x55a651b15390 2024-08-03T20:50:07.895 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:07.892+0000 7f1992800640 1 --2- 172.21.15.32:0/206583888 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a651b15390 0x55a651eff890 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7f198c07e700 tx=0x7f198c082450 comp rx=0 tx=0).ready entity=mon.0 client_cookie=63fc0c52e12b7626 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:50:07.895 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:07.892+0000 7f1982000640 1 -- 172.21.15.32:0/206583888 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f198c048040 con 0x55a651b15390 2024-08-03T20:50:07.896 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:07.892+0000 7f1982000640 1 -- 172.21.15.32:0/206583888 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f198c082190 con 0x55a651b15390 2024-08-03T20:50:07.896 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:07.892+0000 7f1982000640 1 -- 172.21.15.32:0/206583888 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f198c081fd0 con 0x55a651b15390 2024-08-03T20:50:07.896 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:07.893+0000 7f19a0363740 1 -- 172.21.15.32:0/206583888 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a651b15390 msgr2=0x55a651eff890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:50:07.896 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:07.893+0000 7f19a0363740 1 --2- 172.21.15.32:0/206583888 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a651b15390 0x55a651eff890 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7f198c07e700 tx=0x7f198c082450 comp rx=0 tx=0).stop 2024-08-03T20:50:07.896 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:07.893+0000 7f19a0363740 1 -- 172.21.15.32:0/206583888 shutdown_connections 2024-08-03T20:50:07.896 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:07.893+0000 7f19a0363740 1 --2- 172.21.15.32:0/206583888 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55a651af46b0 0x55a651eff350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:07.896 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:07.893+0000 7f19a0363740 1 --2- 172.21.15.32:0/206583888 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a651b15390 0x55a651eff890 unknown :-1 s=CLOSED pgs=488 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:07.896 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:07.893+0000 7f19a0363740 1 --2- 172.21.15.32:0/206583888 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55a651aec4d0 0x55a651f00b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:07.896 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:07.893+0000 7f19a0363740 1 -- 172.21.15.32:0/206583888 >> 172.21.15.32:0/206583888 conn(0x55a651b094e0 msgr2=0x55a651ebb840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:50:07.896 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:07.893+0000 7f19a0363740 1 -- 172.21.15.32:0/206583888 shutdown_connections 2024-08-03T20:50:07.896 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:07.894+0000 7f19a0363740 1 -- 172.21.15.32:0/206583888 wait complete. 2024-08-03T20:50:07.897 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:07.894+0000 7f19a0363740 1 -- start start 2024-08-03T20:50:07.897 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:07.895+0000 7f19a0363740 1 Processor -- start 2024-08-03T20:50:07.898 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:07.896+0000 7f19a0363740 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55a651aec4d0 0x55a651f806a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:50:07.898 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:07.896+0000 7f19a0363740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a651af46b0 0x55a651ef8dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:50:07.898 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:07.896+0000 7f19a0363740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55a651b15390 0x55a651f7f310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:50:07.898 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:07.896+0000 7f19a0363740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55a651ed2d80 con 0x55a651af46b0 2024-08-03T20:50:07.898 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:07.896+0000 7f19a0363740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f199415a570 con 0x55a651b15390 2024-08-03T20:50:07.898 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:07.896+0000 7f19a0363740 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x55a651f007a0 con 0x55a651aec4d0 2024-08-03T20:50:07.898 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:07.896+0000 7f19a0363740 10 client.0 _ll_register_callbacks cb 0x7f197c001230 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-03T20:50:07.898 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:07.897+0000 7f198be00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a651af46b0 0x55a651ef8dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:50:07.899 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:07.897+0000 7f198be00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a651af46b0 0x55a651ef8dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:50106/0 (socket says 172.21.15.32:50106) 2024-08-03T20:50:07.899 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:07.897+0000 7f198be00640 1 -- 172.21.15.32:0/3691332676 learned_addr learned my addr 172.21.15.32:0/3691332676 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:50:07.900 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:07.897+0000 7f1992800640 1 --2- 172.21.15.32:0/3691332676 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55a651b15390 0x55a651f7f310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:50:07.900 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:07.897+0000 7f1991e00640 1 --2- 172.21.15.32:0/3691332676 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55a651aec4d0 0x55a651f806a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:50:07.900 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:07.897+0000 7f198be00640 1 -- 172.21.15.32:0/3691332676 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55a651aec4d0 msgr2=0x55a651f806a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:50:07.901 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:07.897+0000 7f198be00640 1 --2- 172.21.15.32:0/3691332676 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55a651aec4d0 0x55a651f806a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:07.901 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:07.897+0000 7f198be00640 1 -- 172.21.15.32:0/3691332676 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55a651b15390 msgr2=0x55a651f7f310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:50:07.901 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:07.897+0000 7f198be00640 1 --2- 172.21.15.32:0/3691332676 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55a651b15390 0x55a651f7f310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:07.901 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:07.897+0000 7f198be00640 1 -- 172.21.15.32:0/3691332676 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a651ef6840 con 0x55a651af46b0 2024-08-03T20:50:07.901 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:07.898+0000 7f198be00640 1 --2- 172.21.15.32:0/3691332676 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a651af46b0 0x55a651ef8dd0 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7f197c016660 tx=0x7f197c016b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=246cc8b1b2258078 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:50:07.901 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:07.898+0000 7f1977400640 10 client.0 ms_handle_connect on v2:172.21.15.32:3300/0 2024-08-03T20:50:07.901 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:07.898+0000 7f1977400640 1 -- 172.21.15.32:0/3691332676 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f197c005020 con 0x55a651af46b0 2024-08-03T20:50:07.901 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:07.898+0000 7f1977400640 1 -- 172.21.15.32:0/3691332676 <== mon.0 v2:172.21.15.32:3300/0 2 ==== osd_map(46..46 src has 1..46) ==== 5795+0+0 (secure 0 0 0) 0x7f197c0051c0 con 0x55a651af46b0 2024-08-03T20:50:07.901 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:07.898+0000 7f19a0363740 1 -- 172.21.15.32:0/3691332676 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f198c082190 con 0x55a651af46b0 2024-08-03T20:50:07.901 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:07.898+0000 7f1977400640 1 -- 172.21.15.32:0/3691332676 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) ==== 24+0+0 (secure 0 0 0) 0x7f197c029050 con 0x55a651af46b0 2024-08-03T20:50:07.901 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:07.898+0000 7f19a0363740 10 client.5673 fetch_fsmap learned FSMap version 39 2024-08-03T20:50:07.901 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:07.898+0000 7f19a0363740 1 -- 172.21.15.32:0/3691332676 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=2+}) -- 0x55a651f7f850 con 0x55a651af46b0 2024-08-03T20:50:07.901 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:07.898+0000 7f1977400640 1 -- 172.21.15.32:0/3691332676 <== mon.0 v2:172.21.15.32:3300/0 4 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f198c082190 con 0x55a651af46b0 2024-08-03T20:50:07.901 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:07.899+0000 7f1977400640 1 -- 172.21.15.32:0/3691332676 <== mon.0 v2:172.21.15.32:3300/0 5 ==== fsmap(e 39) ==== 3494+0+0 (secure 0 0 0) 0x7f197c006890 con 0x55a651af46b0 2024-08-03T20:50:07.902 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:07.899+0000 7f19a0363740 10 client.5673 fetch_fsmap finished waiting for FSMap version 39 2024-08-03T20:50:07.902 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:07.899+0000 7f19a0363740 10 client.5673 resolve_mds: resolved 3:0 to role '3:0' aka daemon mds.c 2024-08-03T20:50:07.902 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:07.899+0000 7f19a0363740 20 client.5673 populate_metadata read hostname 'smithi032' 2024-08-03T20:50:07.902 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:07.899+0000 7f19a0363740 1 --2- 172.21.15.32:0/3691332676 >> [v2:172.21.15.32:6834/1726912144,v1:172.21.15.32:6835/1726912144] conn(0x55a651f11390 0x55a651f117a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:50:07.902 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:07.899+0000 7f19a0363740 4 client.5673 mds_command: new command op to 5628 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-08-03T20:50:07.902 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:07.899+0000 7f19a0363740 1 -- 172.21.15.32:0/3691332676 --> [v2:172.21.15.32:6834/1726912144,v1:172.21.15.32:6835/1726912144] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55a651e2be60 con 0x55a651f11390 2024-08-03T20:50:07.902 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:07.899+0000 7f1991e00640 1 --2- 172.21.15.32:0/3691332676 >> [v2:172.21.15.32:6834/1726912144,v1:172.21.15.32:6835/1726912144] conn(0x55a651f11390 0x55a651f117a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:50:07.903 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:07.899+0000 7f1991e00640 1 --2- 172.21.15.32:0/3691332676 >> [v2:172.21.15.32:6834/1726912144,v1:172.21.15.32:6835/1726912144] conn(0x55a651f11390 0x55a651f117a0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c8f9febb85a0f357 server_cookie=f22d8a0c4ed217aa in_seq=0 out_seq=0 2024-08-03T20:50:07.903 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:07.899+0000 7f1977400640 10 client.5673 ms_handle_connect on v2:172.21.15.32:6834/1726912144 2024-08-03T20:50:07.903 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:07.900+0000 7f1977400640 1 -- 172.21.15.32:0/3691332676 <== mds.0 v2:172.21.15.32:6834/1726912144 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55a651e2be60 con 0x55a651f11390 2024-08-03T20:50:07.903 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:07.900+0000 7f1977400640 10 client.5673 handle_command_reply: tid=0 2024-08-03T20:50:07.903 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:07.900+0000 7f19a0363740 1 client.5673 shutdown 2024-08-03T20:50:07.904 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:07.900+0000 7f19a0363740 1 -- 172.21.15.32:0/3691332676 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a651af46b0 msgr2=0x55a651ef8dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:50:07.904 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:07.900+0000 7f19a0363740 1 --2- 172.21.15.32:0/3691332676 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a651af46b0 0x55a651ef8dd0 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7f197c016660 tx=0x7f197c016b20 comp rx=0 tx=0).stop 2024-08-03T20:50:07.904 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:07.900+0000 7f19a0363740 1 -- 172.21.15.32:0/3691332676 shutdown_connections 2024-08-03T20:50:07.904 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:07.900+0000 7f19a0363740 1 --2- 172.21.15.32:0/3691332676 >> [v2:172.21.15.32:6834/1726912144,v1:172.21.15.32:6835/1726912144] conn(0x55a651f11390 0x55a651f117a0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:07.904 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:07.900+0000 7f19a0363740 1 --2- 172.21.15.32:0/3691332676 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55a651b15390 0x55a651f7f310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:07.904 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:07.900+0000 7f19a0363740 1 --2- 172.21.15.32:0/3691332676 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a651af46b0 0x55a651ef8dd0 unknown :-1 s=CLOSED pgs=489 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:07.904 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:07.900+0000 7f19a0363740 1 --2- 172.21.15.32:0/3691332676 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55a651aec4d0 0x55a651f806a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:07.904 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:07.900+0000 7f1977400640 0 client.5673 ms_handle_reset on v2:172.21.15.32:6834/1726912144 2024-08-03T20:50:07.904 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:07.900+0000 7f19a0363740 1 -- 172.21.15.32:0/3691332676 >> 172.21.15.32:0/3691332676 conn(0x55a651b094e0 msgr2=0x55a651f00b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:50:07.904 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:07.900+0000 7f19a0363740 1 -- 172.21.15.32:0/3691332676 shutdown_connections 2024-08-03T20:50:07.904 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:07.900+0000 7f19a0363740 1 -- 172.21.15.32:0/3691332676 wait complete. 2024-08-03T20:50:07.904 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:07.901+0000 7f19a0363740 20 client.5673 trim_cache size 0 max 16384 2024-08-03T20:50:07.905 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-08-03T20:50:07.905 INFO:teuthology.orchestra.run.smithi032.stdout: "message": "", 2024-08-03T20:50:07.905 INFO:teuthology.orchestra.run.smithi032.stdout: "return_code": 0 2024-08-03T20:50:07.905 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-08-03T20:50:07.906 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:07.901+0000 7f1999200640 1 -- 172.21.15.32:0/1515659018 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f1970077040 msgr2=0x7f1970079550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:50:07.906 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:07.901+0000 7f1999200640 1 --2- 172.21.15.32:0/1515659018 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f1970077040 0x7f1970079550 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f1984001770 tx=0x7f1984009040 comp rx=0 tx=0).stop 2024-08-03T20:50:07.906 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:07.901+0000 7f1999200640 1 -- 172.21.15.32:0/1515659018 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f19940b1e40 msgr2=0x7f199406cf20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:50:07.906 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:07.901+0000 7f1999200640 1 --2- 172.21.15.32:0/1515659018 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f19940b1e40 0x7f199406cf20 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f198c02aea0 tx=0x7f198c031670 comp rx=0 tx=0).stop 2024-08-03T20:50:07.906 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:07.901+0000 7f1999200640 1 -- 172.21.15.32:0/1515659018 shutdown_connections 2024-08-03T20:50:07.906 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:07.901+0000 7f1999200640 1 --2- 172.21.15.32:0/1515659018 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f1970077040 0x7f1970079550 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:07.906 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:07.901+0000 7f1999200640 1 --2- 172.21.15.32:0/1515659018 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f19940b1e40 0x7f199406cf20 unknown :-1 s=CLOSED pgs=485 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:07.906 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:07.901+0000 7f1999200640 1 --2- 172.21.15.32:0/1515659018 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f199415c3b0 0x7f199406c9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:07.907 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:07.901+0000 7f1999200640 1 --2- 172.21.15.32:0/1515659018 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f1994152480 0x7f19940cae30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:07.907 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:07.901+0000 7f1999200640 1 -- 172.21.15.32:0/1515659018 >> 172.21.15.32:0/1515659018 conn(0x7f19940b32f0 msgr2=0x7f19940b2570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:50:07.907 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:07.901+0000 7f1999200640 1 -- 172.21.15.32:0/1515659018 shutdown_connections 2024-08-03T20:50:07.907 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:07.901+0000 7f1999200640 1 -- 172.21.15.32:0/1515659018 wait complete. 2024-08-03T20:50:07.928 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-08-03T20:50:08.159 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:08.157+0000 7f83b2c00640 1 -- 172.21.15.32:0/3363927017 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f83ac09ac30 msgr2=0x7f83ac09b040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:50:08.159 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:08.157+0000 7f83b2c00640 1 --2- 172.21.15.32:0/3363927017 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f83ac09ac30 0x7f83ac09b040 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f8398009830 tx=0x7f839802a3f0 comp rx=0 tx=0).stop 2024-08-03T20:50:08.160 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:08.158+0000 7f83b2c00640 1 -- 172.21.15.32:0/3363927017 shutdown_connections 2024-08-03T20:50:08.160 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:08.158+0000 7f83b2c00640 1 --2- 172.21.15.32:0/3363927017 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f83ac09ac30 0x7f83ac09b040 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:08.160 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:08.158+0000 7f83b2c00640 1 --2- 172.21.15.32:0/3363927017 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f83ac0b7760 0x7f83ac0cdba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:08.160 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:08.158+0000 7f83b2c00640 1 -- 172.21.15.32:0/3363927017 >> 172.21.15.32:0/3363927017 conn(0x7f83ac0b3bb0 msgr2=0x7f83ac0b3fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:50:08.160 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:08.158+0000 7f83b2c00640 1 -- 172.21.15.32:0/3363927017 shutdown_connections 2024-08-03T20:50:08.160 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:08.158+0000 7f83b2c00640 1 -- 172.21.15.32:0/3363927017 wait complete. 2024-08-03T20:50:08.161 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:08.160+0000 7f83b2c00640 1 Processor -- start 2024-08-03T20:50:08.161 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:08.160+0000 7f83b2c00640 1 -- start start 2024-08-03T20:50:08.163 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:08.161+0000 7f83b2c00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f83ac09ac30 0x7f83ac0cc0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:50:08.164 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:08.161+0000 7f83b2c00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f83ac0b7760 0x7f83ac151700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:50:08.164 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:08.161+0000 7f83b2c00640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f83ac0cfc90 0x7f83ac14ee70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:50:08.164 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:08.161+0000 7f83b2c00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f83ac0b2b70 con 0x7f83ac09ac30 2024-08-03T20:50:08.164 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:08.161+0000 7f83b2c00640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f83ac0b29f0 con 0x7f83ac0b7760 2024-08-03T20:50:08.164 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:08.161+0000 7f83b2c00640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7f83ac166d90 con 0x7f83ac0cfc90 2024-08-03T20:50:08.164 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:08.162+0000 7f83abe00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f83ac09ac30 0x7f83ac0cc0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:50:08.164 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:08.162+0000 7f83abe00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f83ac09ac30 0x7f83ac0cc0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:50108/0 (socket says 172.21.15.32:50108) 2024-08-03T20:50:08.164 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:08.162+0000 7f83abe00640 1 -- 172.21.15.32:0/497410395 learned_addr learned my addr 172.21.15.32:0/497410395 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:50:08.164 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:08.162+0000 7f83ab400640 1 --2- 172.21.15.32:0/497410395 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f83ac0b7760 0x7f83ac151700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:50:08.164 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:08.162+0000 7f83b0c00640 1 --2- 172.21.15.32:0/497410395 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f83ac0cfc90 0x7f83ac14ee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:50:08.164 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:08.162+0000 7f83abe00640 1 -- 172.21.15.32:0/497410395 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f83ac0cfc90 msgr2=0x7f83ac14ee70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:50:08.165 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:08.162+0000 7f83abe00640 1 --2- 172.21.15.32:0/497410395 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f83ac0cfc90 0x7f83ac14ee70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:08.165 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:08.162+0000 7f83abe00640 1 -- 172.21.15.32:0/497410395 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f83ac0b7760 msgr2=0x7f83ac151700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:50:08.165 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:08.162+0000 7f83abe00640 1 --2- 172.21.15.32:0/497410395 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f83ac0b7760 0x7f83ac151700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:08.165 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:08.162+0000 7f83abe00640 1 -- 172.21.15.32:0/497410395 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f83ac152000 con 0x7f83ac09ac30 2024-08-03T20:50:08.165 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:08.162+0000 7f83abe00640 1 --2- 172.21.15.32:0/497410395 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f83ac09ac30 0x7f83ac0cc0b0 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7f839c00ea20 tx=0x7f839c00eee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:50:08.166 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:08.163+0000 7f83a3e00640 1 -- 172.21.15.32:0/497410395 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f839c00ce40 con 0x7f83ac09ac30 2024-08-03T20:50:08.166 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:08.163+0000 7f83a3e00640 1 -- 172.21.15.32:0/497410395 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f839c004020 con 0x7f83ac09ac30 2024-08-03T20:50:08.166 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:08.163+0000 7f83a3e00640 1 -- 172.21.15.32:0/497410395 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f839c004470 con 0x7f83ac09ac30 2024-08-03T20:50:08.166 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:08.163+0000 7f83b2c00640 1 -- 172.21.15.32:0/497410395 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f83ac1522e0 con 0x7f83ac09ac30 2024-08-03T20:50:08.166 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:08.163+0000 7f83b2c00640 1 -- 172.21.15.32:0/497410395 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f83ac163350 con 0x7f83ac09ac30 2024-08-03T20:50:08.166 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:08.164+0000 7f83b2c00640 1 -- 172.21.15.32:0/497410395 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f83ac09b040 con 0x7f83ac09ac30 2024-08-03T20:50:08.166 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:08.165+0000 7f83a3e00640 1 -- 172.21.15.32:0/497410395 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f839c005030 con 0x7f83ac09ac30 2024-08-03T20:50:08.167 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:08.166+0000 7f83a3e00640 1 --2- 172.21.15.32:0/497410395 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f838c077040 0x7f838c079550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:50:08.168 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:08.166+0000 7f83a3e00640 1 -- 172.21.15.32:0/497410395 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(46..46 src has 1..46) ==== 5795+0+0 (secure 0 0 0) 0x7f839c099200 con 0x7f83ac09ac30 2024-08-03T20:50:08.168 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:08.167+0000 7f83ab400640 1 --2- 172.21.15.32:0/497410395 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f838c077040 0x7f838c079550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:50:08.169 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:08.168+0000 7f83ab400640 1 --2- 172.21.15.32:0/497410395 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f838c077040 0x7f838c079550 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f8398034000 tx=0x7f8398004730 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:50:08.172 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:08.171+0000 7f83a3e00640 1 -- 172.21.15.32:0/497410395 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7f839c062420 con 0x7f83ac09ac30 2024-08-03T20:50:08.456 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:08.455+0000 7f83b2c00640 1 -- 172.21.15.32:0/497410395 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f83ac0b4760 con 0x7f83ac09ac30 2024-08-03T20:50:08.457 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:08.455+0000 7f83a3e00640 1 -- 172.21.15.32:0/497410395 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 40 v40) ==== 94+0+5226 (secure 0 0 0) 0x7f839c061d20 con 0x7f83ac09ac30 2024-08-03T20:50:08.457 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-03T20:50:08.457 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":40,"btime":"2024-08-03T20:50:08:046841+0000","default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4997,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6837/2688500839","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6836","nonce":2688500839},{"type":"v1","addr":"172.21.15.118:6837","nonce":2688500839}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":30},{"gid":5266,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6835/2381075275","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6834","nonce":2381075275},{"type":"v1","addr":"172.21.15.118:6835","nonce":2381075275}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":28},{"gid":5421,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6833/1058332640","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":1058332640},{"type":"v1","addr":"172.21.15.32:6833","nonce":1058332640}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":29}],"filesystems":[{"mdsmap":{"epoch":40,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-08-03T20:49:47.064654+0000","modified":"2024-08-03T20:50:07.055408+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":46,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5628},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5628":{"gid":5628,"name":"c","rank":0,"incarnation":39,"state":"up:reconnect","state_seq":2,"addr":"172.21.15.32:6835/1726912144","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1726912144},{"type":"v1","addr":"172.21.15.32:6835","nonce":1726912144}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":3}]} 2024-08-03T20:50:08.458 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 40 2024-08-03T20:50:08.460 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:08.458+0000 7f83b2c00640 1 -- 172.21.15.32:0/497410395 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f838c077040 msgr2=0x7f838c079550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:50:08.460 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:08.458+0000 7f83b2c00640 1 --2- 172.21.15.32:0/497410395 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f838c077040 0x7f838c079550 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f8398034000 tx=0x7f8398004730 comp rx=0 tx=0).stop 2024-08-03T20:50:08.460 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:08.458+0000 7f83b2c00640 1 -- 172.21.15.32:0/497410395 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f83ac09ac30 msgr2=0x7f83ac0cc0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:50:08.461 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:08.458+0000 7f83b2c00640 1 --2- 172.21.15.32:0/497410395 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f83ac09ac30 0x7f83ac0cc0b0 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7f839c00ea20 tx=0x7f839c00eee0 comp rx=0 tx=0).stop 2024-08-03T20:50:08.461 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:08.458+0000 7f83b2c00640 1 -- 172.21.15.32:0/497410395 shutdown_connections 2024-08-03T20:50:08.461 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:08.458+0000 7f83b2c00640 1 --2- 172.21.15.32:0/497410395 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f838c077040 0x7f838c079550 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:08.461 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:08.458+0000 7f83b2c00640 1 --2- 172.21.15.32:0/497410395 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f83ac0b7760 0x7f83ac151700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:08.461 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:08.458+0000 7f83b2c00640 1 --2- 172.21.15.32:0/497410395 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f83ac0cfc90 0x7f83ac14ee70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:08.461 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:08.458+0000 7f83b2c00640 1 --2- 172.21.15.32:0/497410395 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f83ac09ac30 0x7f83ac0cc0b0 unknown :-1 s=CLOSED pgs=491 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:08.461 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:08.458+0000 7f83b2c00640 1 -- 172.21.15.32:0/497410395 >> 172.21.15.32:0/497410395 conn(0x7f83ac0b3bb0 msgr2=0x7f83ac0cf430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:50:08.461 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:08.458+0000 7f83b2c00640 1 -- 172.21.15.32:0/497410395 shutdown_connections 2024-08-03T20:50:08.461 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:08.458+0000 7f83b2c00640 1 -- 172.21.15.32:0/497410395 wait complete. 2024-08-03T20:50:08.484 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-08-03T20:50:08.729 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:08.727+0000 7f2930a00640 1 -- 172.21.15.32:0/90623802 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f292c06e930 msgr2=0x7f292c144f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:50:08.729 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:08.727+0000 7f2930a00640 1 --2- 172.21.15.32:0/90623802 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f292c06e930 0x7f292c144f90 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f291c009830 tx=0x7f291c02a410 comp rx=0 tx=0).stop 2024-08-03T20:50:08.729 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:08.727+0000 7f2930a00640 1 -- 172.21.15.32:0/90623802 shutdown_connections 2024-08-03T20:50:08.729 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:08.727+0000 7f2930a00640 1 --2- 172.21.15.32:0/90623802 >> v2:172.21.15.118:3300/0 conn(0x7f292c073e60 0x7f292c074270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:08.729 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:08.727+0000 7f2930a00640 1 --2- 172.21.15.32:0/90623802 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f292c06e930 0x7f292c144f90 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:08.729 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:08.727+0000 7f2930a00640 1 -- 172.21.15.32:0/90623802 >> 172.21.15.32:0/90623802 conn(0x7f292c0b1bd0 msgr2=0x7f292c0b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:50:08.729 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:08.727+0000 7f2930a00640 1 -- 172.21.15.32:0/90623802 shutdown_connections 2024-08-03T20:50:08.729 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:08.727+0000 7f2930a00640 1 -- 172.21.15.32:0/90623802 wait complete. 2024-08-03T20:50:08.731 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:08.729+0000 7f2930a00640 1 Processor -- start 2024-08-03T20:50:08.731 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:08.730+0000 7f2930a00640 1 -- start start 2024-08-03T20:50:08.734 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:08.731+0000 7f2930a00640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f292c06e930 0x7f292c15ec60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:50:08.734 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:08.731+0000 7f2930a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f292c073e60 0x7f292c0a96e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:50:08.734 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:08.731+0000 7f2930a00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f292c0747b0 0x7f292c0a9c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:50:08.734 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:08.731+0000 7f2930a00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f292c0cf9c0 con 0x7f292c073e60 2024-08-03T20:50:08.734 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:08.731+0000 7f2930a00640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f292c0cf840 con 0x7f292c0747b0 2024-08-03T20:50:08.734 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:08.731+0000 7f2930a00640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7f292c0ccff0 con 0x7f292c06e930 2024-08-03T20:50:08.734 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:08.731+0000 7f2923e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f292c073e60 0x7f292c0a96e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:50:08.734 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:08.731+0000 7f2923e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f292c073e60 0x7f292c0a96e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:50124/0 (socket says 172.21.15.32:50124) 2024-08-03T20:50:08.735 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:08.731+0000 7f2923e00640 1 -- 172.21.15.32:0/2761405657 learned_addr learned my addr 172.21.15.32:0/2761405657 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:50:08.735 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:08.732+0000 7f2929e00640 1 --2- 172.21.15.32:0/2761405657 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f292c06e930 0x7f292c15ec60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:50:08.735 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:08.732+0000 7f2923e00640 1 -- 172.21.15.32:0/2761405657 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f292c06e930 msgr2=0x7f292c15ec60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:50:08.735 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:08.732+0000 7f2923e00640 1 --2- 172.21.15.32:0/2761405657 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f292c06e930 0x7f292c15ec60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:08.735 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:08.732+0000 7f2923e00640 1 -- 172.21.15.32:0/2761405657 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f292c0747b0 msgr2=0x7f292c0a9c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:50:08.735 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:08.732+0000 7f292a800640 1 --2- 172.21.15.32:0/2761405657 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f292c0747b0 0x7f292c0a9c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:50:08.735 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:08.732+0000 7f2923e00640 1 --2- 172.21.15.32:0/2761405657 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f292c0747b0 0x7f292c0a9c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:08.735 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:08.732+0000 7f2923e00640 1 -- 172.21.15.32:0/2761405657 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f292c15fbb0 con 0x7f292c073e60 2024-08-03T20:50:08.735 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:08.732+0000 7f2929e00640 1 --2- 172.21.15.32:0/2761405657 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f292c06e930 0x7f292c15ec60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-03T20:50:08.735 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:08.732+0000 7f292a800640 1 --2- 172.21.15.32:0/2761405657 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f292c0747b0 0x7f292c0a9c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-03T20:50:08.735 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:08.732+0000 7f2923e00640 1 --2- 172.21.15.32:0/2761405657 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f292c073e60 0x7f292c0a96e0 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7f291400e790 tx=0x7f291400ec50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:50:08.735 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:08.732+0000 7f2922000640 1 -- 172.21.15.32:0/2761405657 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f291400eea0 con 0x7f292c073e60 2024-08-03T20:50:08.736 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:08.732+0000 7f2922000640 1 -- 172.21.15.32:0/2761405657 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f291400cdb0 con 0x7f292c073e60 2024-08-03T20:50:08.736 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:08.732+0000 7f2922000640 1 -- 172.21.15.32:0/2761405657 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2914010690 con 0x7f292c073e60 2024-08-03T20:50:08.736 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:08.732+0000 7f2930a00640 1 -- 172.21.15.32:0/2761405657 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f292c131040 con 0x7f292c073e60 2024-08-03T20:50:08.736 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:08.733+0000 7f2930a00640 1 -- 172.21.15.32:0/2761405657 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f292c131600 con 0x7f292c073e60 2024-08-03T20:50:08.736 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:08.734+0000 7f2930a00640 1 -- 172.21.15.32:0/2761405657 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f292c130db0 con 0x7f292c073e60 2024-08-03T20:50:08.736 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:08.735+0000 7f2922000640 1 -- 172.21.15.32:0/2761405657 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f2914010830 con 0x7f292c073e60 2024-08-03T20:50:08.738 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:08.736+0000 7f2922000640 1 --2- 172.21.15.32:0/2761405657 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f2908077010 0x7f2908079520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:50:08.738 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:08.737+0000 7f2922000640 1 -- 172.21.15.32:0/2761405657 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(46..46 src has 1..46) ==== 5795+0+0 (secure 0 0 0) 0x7f2914099290 con 0x7f292c073e60 2024-08-03T20:50:08.738 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:08.737+0000 7f2929e00640 1 --2- 172.21.15.32:0/2761405657 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f2908077010 0x7f2908079520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:50:08.739 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:08.738+0000 7f2929e00640 1 --2- 172.21.15.32:0/2761405657 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f2908077010 0x7f2908079520 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f291c034000 tx=0x7f291c004730 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:50:08.742 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:08.741+0000 7f2922000640 1 -- 172.21.15.32:0/2761405657 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7f29140624b0 con 0x7f292c073e60 2024-08-03T20:50:08.983 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:08.982+0000 7f2930a00640 1 -- 172.21.15.32:0/2761405657 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f292c0cd550 con 0x7f292c073e60 2024-08-03T20:50:08.984 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:08.983+0000 7f2922000640 1 -- 172.21.15.32:0/2761405657 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 40 v40) ==== 94+0+5226 (secure 0 0 0) 0x7f2914061db0 con 0x7f292c073e60 2024-08-03T20:50:08.984 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-03T20:50:08.984 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":40,"btime":"2024-08-03T20:50:08:046841+0000","default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4997,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6837/2688500839","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6836","nonce":2688500839},{"type":"v1","addr":"172.21.15.118:6837","nonce":2688500839}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":30},{"gid":5266,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6835/2381075275","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6834","nonce":2381075275},{"type":"v1","addr":"172.21.15.118:6835","nonce":2381075275}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":28},{"gid":5421,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6833/1058332640","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":1058332640},{"type":"v1","addr":"172.21.15.32:6833","nonce":1058332640}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":29}],"filesystems":[{"mdsmap":{"epoch":40,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-08-03T20:49:47.064654+0000","modified":"2024-08-03T20:50:07.055408+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":46,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5628},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5628":{"gid":5628,"name":"c","rank":0,"incarnation":39,"state":"up:reconnect","state_seq":2,"addr":"172.21.15.32:6835/1726912144","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1726912144},{"type":"v1","addr":"172.21.15.32:6835","nonce":1726912144}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":3}]} 2024-08-03T20:50:08.985 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 40 2024-08-03T20:50:08.987 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:08.985+0000 7f2930a00640 1 -- 172.21.15.32:0/2761405657 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f2908077010 msgr2=0x7f2908079520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:50:08.987 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:08.985+0000 7f2930a00640 1 --2- 172.21.15.32:0/2761405657 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f2908077010 0x7f2908079520 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f291c034000 tx=0x7f291c004730 comp rx=0 tx=0).stop 2024-08-03T20:50:08.987 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:08.985+0000 7f2930a00640 1 -- 172.21.15.32:0/2761405657 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f292c073e60 msgr2=0x7f292c0a96e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:50:08.987 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:08.985+0000 7f2930a00640 1 --2- 172.21.15.32:0/2761405657 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f292c073e60 0x7f292c0a96e0 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7f291400e790 tx=0x7f291400ec50 comp rx=0 tx=0).stop 2024-08-03T20:50:08.987 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:08.985+0000 7f2930a00640 1 -- 172.21.15.32:0/2761405657 shutdown_connections 2024-08-03T20:50:08.987 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:08.985+0000 7f2930a00640 1 --2- 172.21.15.32:0/2761405657 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f2908077010 0x7f2908079520 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:08.987 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:08.985+0000 7f2930a00640 1 --2- 172.21.15.32:0/2761405657 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f292c0747b0 0x7f292c0a9c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:08.987 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:08.985+0000 7f2930a00640 1 --2- 172.21.15.32:0/2761405657 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f292c073e60 0x7f292c0a96e0 unknown :-1 s=CLOSED pgs=493 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:08.987 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:08.985+0000 7f2930a00640 1 --2- 172.21.15.32:0/2761405657 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f292c06e930 0x7f292c15ec60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:08.988 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:08.985+0000 7f2930a00640 1 -- 172.21.15.32:0/2761405657 >> 172.21.15.32:0/2761405657 conn(0x7f292c0b1bd0 msgr2=0x7f292c0b0610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:50:08.988 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:08.985+0000 7f2930a00640 1 -- 172.21.15.32:0/2761405657 shutdown_connections 2024-08-03T20:50:08.988 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:08.985+0000 7f2930a00640 1 -- 172.21.15.32:0/2761405657 wait complete. 2024-08-03T20:50:09.009 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-08-03T20:50:09.239 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:09.237+0000 7ff699600640 1 -- 172.21.15.32:0/1809486309 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ff694165160 msgr2=0x7ff694167580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:50:09.239 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:09.237+0000 7ff699600640 1 --2- 172.21.15.32:0/1809486309 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ff694165160 0x7ff694167580 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7ff68c005fd0 tx=0x7ff68c02a840 comp rx=0 tx=0).stop 2024-08-03T20:50:09.239 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:09.237+0000 7ff699600640 1 -- 172.21.15.32:0/1809486309 shutdown_connections 2024-08-03T20:50:09.239 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:09.237+0000 7ff699600640 1 --2- 172.21.15.32:0/1809486309 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ff694165160 0x7ff694167580 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:09.239 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:09.237+0000 7ff699600640 1 -- 172.21.15.32:0/1809486309 >> 172.21.15.32:0/1809486309 conn(0x7ff6940b1bd0 msgr2=0x7ff6940b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:50:09.239 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:09.237+0000 7ff699600640 1 -- 172.21.15.32:0/1809486309 shutdown_connections 2024-08-03T20:50:09.239 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:09.237+0000 7ff699600640 1 -- 172.21.15.32:0/1809486309 wait complete. 2024-08-03T20:50:09.241 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:09.239+0000 7ff699600640 1 Processor -- start 2024-08-03T20:50:09.241 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:09.239+0000 7ff699600640 1 -- start start 2024-08-03T20:50:09.243 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:09.241+0000 7ff699600640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff69406e930 0x7ff6940b4780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:50:09.243 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:09.241+0000 7ff699600640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ff694144f90 0x7ff694138600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:50:09.243 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:09.241+0000 7ff699600640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7ff694165160 0x7ff694138b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:50:09.243 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:09.241+0000 7ff699600640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7ff6940cf9c0 con 0x7ff69406e930 2024-08-03T20:50:09.243 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:09.241+0000 7ff699600640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7ff6940cf840 con 0x7ff694144f90 2024-08-03T20:50:09.243 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:09.241+0000 7ff699600640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7ff694167ac0 con 0x7ff694165160 2024-08-03T20:50:09.244 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:09.241+0000 7ff692a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff69406e930 0x7ff6940b4780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:50:09.244 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:09.241+0000 7ff692a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff69406e930 0x7ff6940b4780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:50138/0 (socket says 172.21.15.32:50138) 2024-08-03T20:50:09.244 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:09.241+0000 7ff692a00640 1 -- 172.21.15.32:0/494370496 learned_addr learned my addr 172.21.15.32:0/494370496 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:50:09.244 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:09.241+0000 7ff693400640 1 --2- 172.21.15.32:0/494370496 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7ff694165160 0x7ff694138b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:50:09.244 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:09.241+0000 7ff68be00640 1 --2- 172.21.15.32:0/494370496 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ff694144f90 0x7ff694138600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:50:09.244 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:09.242+0000 7ff692a00640 1 -- 172.21.15.32:0/494370496 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7ff694165160 msgr2=0x7ff694138b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:50:09.244 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:09.242+0000 7ff692a00640 1 --2- 172.21.15.32:0/494370496 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7ff694165160 0x7ff694138b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:09.244 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:09.242+0000 7ff692a00640 1 -- 172.21.15.32:0/494370496 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ff694144f90 msgr2=0x7ff694138600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:50:09.244 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:09.242+0000 7ff692a00640 1 --2- 172.21.15.32:0/494370496 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ff694144f90 0x7ff694138600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:09.244 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:09.242+0000 7ff692a00640 1 -- 172.21.15.32:0/494370496 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff694146250 con 0x7ff69406e930 2024-08-03T20:50:09.244 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:09.242+0000 7ff68be00640 1 --2- 172.21.15.32:0/494370496 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ff694144f90 0x7ff694138600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-03T20:50:09.244 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:09.242+0000 7ff692a00640 1 --2- 172.21.15.32:0/494370496 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff69406e930 0x7ff6940b4780 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7ff68400c430 tx=0x7ff68400c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:50:09.245 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:09.242+0000 7ff68aa00640 1 -- 172.21.15.32:0/494370496 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff684017070 con 0x7ff69406e930 2024-08-03T20:50:09.245 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:09.242+0000 7ff68aa00640 1 -- 172.21.15.32:0/494370496 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7ff6840053c0 con 0x7ff69406e930 2024-08-03T20:50:09.245 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:09.242+0000 7ff68aa00640 1 -- 172.21.15.32:0/494370496 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff684003850 con 0x7ff69406e930 2024-08-03T20:50:09.245 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:09.242+0000 7ff699600640 1 -- 172.21.15.32:0/494370496 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff694146530 con 0x7ff69406e930 2024-08-03T20:50:09.246 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:09.242+0000 7ff699600640 1 -- 172.21.15.32:0/494370496 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff694162900 con 0x7ff69406e930 2024-08-03T20:50:09.246 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:09.244+0000 7ff699600640 1 -- 172.21.15.32:0/494370496 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff6941678e0 con 0x7ff69406e930 2024-08-03T20:50:09.246 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:09.245+0000 7ff68aa00640 1 -- 172.21.15.32:0/494370496 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7ff68401a070 con 0x7ff69406e930 2024-08-03T20:50:09.248 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:09.246+0000 7ff68aa00640 1 --2- 172.21.15.32:0/494370496 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7ff670077040 0x7ff670079550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:50:09.248 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:09.246+0000 7ff68aa00640 1 -- 172.21.15.32:0/494370496 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(46..46 src has 1..46) ==== 5795+0+0 (secure 0 0 0) 0x7ff684097cd0 con 0x7ff69406e930 2024-08-03T20:50:09.248 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:09.247+0000 7ff68be00640 1 --2- 172.21.15.32:0/494370496 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7ff670077040 0x7ff670079550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:50:09.249 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:09.248+0000 7ff68be00640 1 --2- 172.21.15.32:0/494370496 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7ff670077040 0x7ff670079550 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7ff67c001220 tx=0x7ff67c009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:50:09.252 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:09.251+0000 7ff68aa00640 1 -- 172.21.15.32:0/494370496 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7ff6941678e0 con 0x7ff69406e930 2024-08-03T20:50:09.493 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:09.492+0000 7ff699600640 1 -- 172.21.15.32:0/494370496 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff694073e60 con 0x7ff69406e930 2024-08-03T20:50:09.494 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:09.492+0000 7ff68aa00640 1 -- 172.21.15.32:0/494370496 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 41 v41) ==== 94+0+5223 (secure 0 0 0) 0x7ff6840690a0 con 0x7ff69406e930 2024-08-03T20:50:09.494 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-03T20:50:09.494 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":41,"btime":"2024-08-03T20:50:09:048560+0000","default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4997,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6837/2688500839","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6836","nonce":2688500839},{"type":"v1","addr":"172.21.15.118:6837","nonce":2688500839}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":30},{"gid":5266,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6835/2381075275","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6834","nonce":2381075275},{"type":"v1","addr":"172.21.15.118:6835","nonce":2381075275}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":28},{"gid":5421,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6833/1058332640","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":1058332640},{"type":"v1","addr":"172.21.15.32:6833","nonce":1058332640}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":29}],"filesystems":[{"mdsmap":{"epoch":41,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-08-03T20:49:47.064654+0000","modified":"2024-08-03T20:50:08.051507+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":46,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5628},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5628":{"gid":5628,"name":"c","rank":0,"incarnation":39,"state":"up:rejoin","state_seq":3,"addr":"172.21.15.32:6835/1726912144","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1726912144},{"type":"v1","addr":"172.21.15.32:6835","nonce":1726912144}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":3}]} 2024-08-03T20:50:09.494 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 41 2024-08-03T20:50:09.497 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:09.495+0000 7ff699600640 1 -- 172.21.15.32:0/494370496 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7ff670077040 msgr2=0x7ff670079550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:50:09.497 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:09.495+0000 7ff699600640 1 --2- 172.21.15.32:0/494370496 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7ff670077040 0x7ff670079550 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7ff67c001220 tx=0x7ff67c009040 comp rx=0 tx=0).stop 2024-08-03T20:50:09.497 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:09.495+0000 7ff699600640 1 -- 172.21.15.32:0/494370496 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff69406e930 msgr2=0x7ff6940b4780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:50:09.497 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:09.495+0000 7ff699600640 1 --2- 172.21.15.32:0/494370496 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff69406e930 0x7ff6940b4780 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7ff68400c430 tx=0x7ff68400c8f0 comp rx=0 tx=0).stop 2024-08-03T20:50:09.497 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:09.495+0000 7ff699600640 1 -- 172.21.15.32:0/494370496 shutdown_connections 2024-08-03T20:50:09.497 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:09.495+0000 7ff699600640 1 --2- 172.21.15.32:0/494370496 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7ff670077040 0x7ff670079550 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:09.497 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:09.495+0000 7ff699600640 1 --2- 172.21.15.32:0/494370496 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ff694144f90 0x7ff694138600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:09.497 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:09.495+0000 7ff699600640 1 --2- 172.21.15.32:0/494370496 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7ff694165160 0x7ff694138b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:09.498 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:09.495+0000 7ff699600640 1 --2- 172.21.15.32:0/494370496 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff69406e930 0x7ff6940b4780 unknown :-1 s=CLOSED pgs=495 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:09.498 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:09.495+0000 7ff699600640 1 -- 172.21.15.32:0/494370496 >> 172.21.15.32:0/494370496 conn(0x7ff6940b1bd0 msgr2=0x7ff6941667a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:50:09.498 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:09.495+0000 7ff699600640 1 -- 172.21.15.32:0/494370496 shutdown_connections 2024-08-03T20:50:09.498 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:09.495+0000 7ff699600640 1 -- 172.21.15.32:0/494370496 wait complete. 2024-08-03T20:50:09.523 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json dump tree /a 0 2024-08-03T20:50:09.770 INFO:teuthology.orchestra.run.smithi032.stdout:[{"ino":1099511627776,"rdev":0,"ctime":"2024-08-03T20:50:02.493997+0000","btime":"2024-08-03T20:49:56.119125+0000","mode":16877,"uid":1000,"gid":1316,"nlink":1,"dir_layout":{"dir_hash":2,"unused1":0,"unused2":0,"unused3":0},"layout":{"stripe_unit":0,"stripe_count":0,"object_size":0,"pool_id":-1,"pool_ns":""},"old_pools":[],"size":0,"truncate_seq":1,"truncate_size":18446744073709551615,"truncate_from":0,"truncate_pending":0,"mtime":"2024-08-03T20:50:02.493997+0000","atime":"2024-08-03T20:49:56.119125+0000","time_warp_seq":0,"change_attr":3,"export_pin":-1,"export_ephemeral_random_pin":0,"export_ephemeral_distributed_pin":false,"quiesce_block":false,"client_ranges":[],"dirstat":{"version":0,"mtime":"2024-08-03T20:50:02.493997+0000","num_files":0,"num_subdirs":1,"change_attr":3},"rstat":{"version":0,"rbytes":0,"rfiles":0,"rsubdirs":2,"rsnaps":0,"rctime":"2024-08-03T20:50:02.493997+0000"},"accounted_rstat":{"version":0,"rbytes":0,"rfiles":0,"rsubdirs":2,"rsnaps":0,"rctime":"2024-08-03T20:50:02.493997+0000"},"version":9,"file_data_version":0,"xattr_version":1,"backtrace_version":2,"stray_prior_path":"","max_size_ever":0,"quota":{"max_bytes":0,"max_files":0},"last_scrub_stamp":"0.000000","last_scrub_version":0,"symlink":"","xattrs":[{"key":"security.selinux","val":"unconfined_u:object_r:unlabeled_t:s0\u0000"}],"dirfragtree":{"splits":[]},"old_inodes":[],"oldest_snap":18446744073709551614,"damage_flags":0,"is_auth":true,"auth_state":{"replicas":{}},"replica_state":{"authority":[0,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"dirtyscattered":1,"dirfrag":1,"dirty":1},"nref":3,"versionlock":{"gather_set":[],"state":"lock","type":"iversion","is_leased":false,"num_rdlocks":0,"num_wrlocks":0,"num_xlocks":0,"xlock_by":{}},"authlock":{},"linklock":{},"dirfragtreelock":{},"filelock":{},"xattrlock":{},"snaplock":{},"nestlock":{"gather_set":[],"state":"lock","type":"inest","is_leased":false,"num_rdlocks":0,"num_wrlocks":0,"num_xlocks":0,"xlock_by":{}},"flocklock":{},"policylock":{},"quiescelock":{"gather_set":[],"state":"lock","type":"iquiesce","is_leased":false,"num_rdlocks":0,"num_wrlocks":0,"num_xlocks":0,"xlock_by":{}},"states":["auth","dirty"],"client_caps":[],"loner":-1,"want_loner":-1,"mds_caps_wanted":[],"dirfrags":[{"path":"/a","dirfrag":"0x10000000000","snapid_first":2,"projected_version":"7","version":"7","committing_version":"0","committed_version":"0","is_rep":false,"dir_auth":"","states":["auth"],"is_auth":true,"auth_state":{"replicas":{}},"replica_state":{"authority":[0,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1},"nref":1,"dentries":[{"path":"a/c","alternate_name":"","path_ino":1,"snap_first":18446744073709551606,"snap_last":18446744073709551614,"is_primary":true,"is_remote":false,"is_null":false,"is_new":false,"inode":1099511627777,"remote_type":"","version":4,"projected_version":4,"auth_pins":0,"is_auth":true,"auth_state":{"replicas":{}},"replica_state":{"authority":[0,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"inodepin":1,"dirty":1},"nref":2,"lock":{},"versionlock":{"gather_set":[],"state":"lock","type":"dversion","is_leased":false,"num_rdlocks":0,"num_wrlocks":0,"num_xlocks":0,"xlock_by":{}},"states":["auth","dirty"]},{"path":"a/b","alternate_name":"","path_ino":1,"snap_first":2,"snap_last":18446744073709551614,"is_primary":false,"is_remote":false,"is_null":true,"is_new":false,"inode":0,"remote_type":"","version":5,"projected_version":5,"auth_pins":0,"is_auth":true,"auth_state":{"replicas":{}},"replica_state":{"authority":[0,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"dirty":1},"nref":1,"lock":{},"versionlock":{"gather_set":[],"state":"lock","type":"dversion","is_leased":false,"num_rdlocks":0,"num_wrlocks":0,"num_xlocks":0,"xlock_by":{}},"states":["auth","dirty"]}]}],"is_subvolume":false}] 2024-08-03T20:50:09.771 DEBUG:tasks.cephfs.filesystem:_json_asok output [ { "accounted_rstat": { "rbytes": 0, "rctime": "2024-08-03T20:50:02.493997+0000", "rfiles": 0, "rsnaps": 0, "rsubdirs": 2, "version": 0 }, "atime": "2024-08-03T20:49:56.119125+0000", "auth_pins": 0, "auth_state": { "replicas": {} }, "authlock": {}, "backtrace_version": 2, "btime": "2024-08-03T20:49:56.119125+0000", "change_attr": 3, "client_caps": [], "client_ranges": [], "ctime": "2024-08-03T20:50:02.493997+0000", "damage_flags": 0, "dir_layout": { "dir_hash": 2, "unused1": 0, "unused2": 0, "unused3": 0 }, "dirfrags": [ { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "0", "committing_version": "0", "dentries": [ { "alternate_name": "", "auth_pins": 0, "auth_state": { "replicas": {} }, "inode": 1099511627777, "is_auth": true, "is_freezing": false, "is_frozen": false, "is_new": false, "is_null": false, "is_primary": true, "is_remote": false, "lock": {}, "nref": 2, "path": "a/c", "path_ino": 1, "pins": { "dirty": 1, "inodepin": 1 }, "projected_version": 4, "remote_type": "", "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 0 }, "snap_first": 18446744073709551606, "snap_last": 18446744073709551614, "states": [ "auth", "dirty" ], "version": 4, "versionlock": { "gather_set": [], "is_leased": false, "num_rdlocks": 0, "num_wrlocks": 0, "num_xlocks": 0, "state": "lock", "type": "dversion", "xlock_by": {} } }, { "alternate_name": "", "auth_pins": 0, "auth_state": { "replicas": {} }, "inode": 0, "is_auth": true, "is_freezing": false, "is_frozen": false, "is_new": false, "is_null": true, "is_primary": false, "is_remote": false, "lock": {}, "nref": 1, "path": "a/b", "path_ino": 1, "pins": { "dirty": 1 }, "projected_version": 5, "remote_type": "", "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 0 }, "snap_first": 2, "snap_last": 18446744073709551614, "states": [ "auth", "dirty" ], "version": 5, "versionlock": { "gather_set": [], "is_leased": false, "num_rdlocks": 0, "num_wrlocks": 0, "num_xlocks": 0, "state": "lock", "type": "dversion", "xlock_by": {} } } ], "dir_auth": "", "dirfrag": "0x10000000000", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 1, "path": "/a", "pins": { "child": 1 }, "projected_version": "7", "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth" ], "version": "7" } ], "dirfragtree": { "splits": [] }, "dirfragtreelock": {}, "dirstat": { "change_attr": 3, "mtime": "2024-08-03T20:50:02.493997+0000", "num_files": 0, "num_subdirs": 1, "version": 0 }, "export_ephemeral_distributed_pin": false, "export_ephemeral_random_pin": 0, "export_pin": -1, "file_data_version": 0, "filelock": {}, "flocklock": {}, "gid": 1316, "ino": 1099511627776, "is_auth": true, "is_freezing": false, "is_frozen": false, "is_subvolume": false, "last_scrub_stamp": "0.000000", "last_scrub_version": 0, "layout": { "object_size": 0, "pool_id": -1, "pool_ns": "", "stripe_count": 0, "stripe_unit": 0 }, "linklock": {}, "loner": -1, "max_size_ever": 0, "mds_caps_wanted": [], "mode": 16877, "mtime": "2024-08-03T20:50:02.493997+0000", "nestlock": { "gather_set": [], "is_leased": false, "num_rdlocks": 0, "num_wrlocks": 0, "num_xlocks": 0, "state": "lock", "type": "inest", "xlock_by": {} }, "nlink": 1, "nref": 3, "old_inodes": [], "old_pools": [], "oldest_snap": 18446744073709551614, "pins": { "dirfrag": 1, "dirty": 1, "dirtyscattered": 1 }, "policylock": {}, "quiesce_block": false, "quiescelock": { "gather_set": [], "is_leased": false, "num_rdlocks": 0, "num_wrlocks": 0, "num_xlocks": 0, "state": "lock", "type": "iquiesce", "xlock_by": {} }, "quota": { "max_bytes": 0, "max_files": 0 }, "rdev": 0, "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 0 }, "rstat": { "rbytes": 0, "rctime": "2024-08-03T20:50:02.493997+0000", "rfiles": 0, "rsnaps": 0, "rsubdirs": 2, "version": 0 }, "size": 0, "snaplock": {}, "states": [ "auth", "dirty" ], "stray_prior_path": "", "symlink": "", "time_warp_seq": 0, "truncate_from": 0, "truncate_pending": 0, "truncate_seq": 1, "truncate_size": 18446744073709551615, "uid": 1000, "version": 9, "versionlock": { "gather_set": [], "is_leased": false, "num_rdlocks": 0, "num_wrlocks": 0, "num_xlocks": 0, "state": "lock", "type": "iversion", "xlock_by": {} }, "want_loner": -1, "xattr_version": 1, "xattrlock": {}, "xattrs": [ { "key": "security.selinux", "val": "unconfined_u:object_r:unlabeled_t:s0\u0000" } ] } ] 2024-08-03T20:50:09.772 DEBUG:tasks.cephfs.test_damage:[{'ino': 1099511627776, 'rdev': 0, 'ctime': '2024-08-03T20:50:02.493997+0000', 'btime': '2024-08-03T20:49:56.119125+0000', 'mode': 16877, 'uid': 1000, 'gid': 1316, 'nlink': 1, 'dir_layout': {'dir_hash': 2, 'unused1': 0, 'unused2': 0, 'unused3': 0}, 'layout': {'stripe_unit': 0, 'stripe_count': 0, 'object_size': 0, 'pool_id': -1, 'pool_ns': ''}, 'old_pools': [], 'size': 0, 'truncate_seq': 1, 'truncate_size': 18446744073709551615, 'truncate_from': 0, 'truncate_pending': 0, 'mtime': '2024-08-03T20:50:02.493997+0000', 'atime': '2024-08-03T20:49:56.119125+0000', 'time_warp_seq': 0, 'change_attr': 3, 'export_pin': -1, 'export_ephemeral_random_pin': 0, 'export_ephemeral_distributed_pin': False, 'quiesce_block': False, 'client_ranges': [], 'dirstat': {'version': 0, 'mtime': '2024-08-03T20:50:02.493997+0000', 'num_files': 0, 'num_subdirs': 1, 'change_attr': 3}, 'rstat': {'version': 0, 'rbytes': 0, 'rfiles': 0, 'rsubdirs': 2, 'rsnaps': 0, 'rctime': '2024-08-03T20:50:02.493997+0000'}, 'accounted_rstat': {'version': 0, 'rbytes': 0, 'rfiles': 0, 'rsubdirs': 2, 'rsnaps': 0, 'rctime': '2024-08-03T20:50:02.493997+0000'}, 'version': 9, 'file_data_version': 0, 'xattr_version': 1, 'backtrace_version': 2, 'stray_prior_path': '', 'max_size_ever': 0, 'quota': {'max_bytes': 0, 'max_files': 0}, 'last_scrub_stamp': '0.000000', 'last_scrub_version': 0, 'symlink': '', 'xattrs': [{'key': 'security.selinux', 'val': 'unconfined_u:object_r:unlabeled_t:s0\x00'}], 'dirfragtree': {'splits': []}, 'old_inodes': [], 'oldest_snap': 18446744073709551614, 'damage_flags': 0, 'is_auth': True, 'auth_state': {'replicas': {}}, 'replica_state': {'authority': [0, -2], 'replica_nonce': 0}, 'auth_pins': 0, 'is_frozen': False, 'is_freezing': False, 'pins': {'dirtyscattered': 1, 'dirfrag': 1, 'dirty': 1}, 'nref': 3, 'versionlock': {'gather_set': [], 'state': 'lock', 'type': 'iversion', 'is_leased': False, 'num_rdlocks': 0, 'num_wrlocks': 0, 'num_xlocks': 0, 'xlock_by': {}}, 'authlock': {}, 'linklock': {}, 'dirfragtreelock': {}, 'filelock': {}, 'xattrlock': {}, 'snaplock': {}, 'nestlock': {'gather_set': [], 'state': 'lock', 'type': 'inest', 'is_leased': False, 'num_rdlocks': 0, 'num_wrlocks': 0, 'num_xlocks': 0, 'xlock_by': {}}, 'flocklock': {}, 'policylock': {}, 'quiescelock': {'gather_set': [], 'state': 'lock', 'type': 'iquiesce', 'is_leased': False, 'num_rdlocks': 0, 'num_wrlocks': 0, 'num_xlocks': 0, 'xlock_by': {}}, 'states': ['auth', 'dirty'], 'client_caps': [], 'loner': -1, 'want_loner': -1, 'mds_caps_wanted': [], 'dirfrags': [{'path': '/a', 'dirfrag': '0x10000000000', 'snapid_first': 2, 'projected_version': '7', 'version': '7', 'committing_version': '0', 'committed_version': '0', 'is_rep': False, 'dir_auth': '', 'states': ['auth'], 'is_auth': True, 'auth_state': {'replicas': {}}, 'replica_state': {'authority': [0, -2], 'replica_nonce': 0}, 'auth_pins': 0, 'is_frozen': False, 'is_freezing': False, 'pins': {'child': 1}, 'nref': 1, 'dentries': [{'path': 'a/c', 'alternate_name': '', 'path_ino': 1, 'snap_first': 18446744073709551606, 'snap_last': 18446744073709551614, 'is_primary': True, 'is_remote': False, 'is_null': False, 'is_new': False, 'inode': 1099511627777, 'remote_type': '', 'version': 4, 'projected_version': 4, 'auth_pins': 0, 'is_auth': True, 'auth_state': {'replicas': {}}, 'replica_state': {'authority': [0, -2], 'replica_nonce': 0}, 'is_frozen': False, 'is_freezing': False, 'pins': {'inodepin': 1, 'dirty': 1}, 'nref': 2, 'lock': {}, 'versionlock': {'gather_set': [], 'state': 'lock', 'type': 'dversion', 'is_leased': False, 'num_rdlocks': 0, 'num_wrlocks': 0, 'num_xlocks': 0, 'xlock_by': {}}, 'states': ['auth', 'dirty']}, {'path': 'a/b', 'alternate_name': '', 'path_ino': 1, 'snap_first': 2, 'snap_last': 18446744073709551614, 'is_primary': False, 'is_remote': False, 'is_null': True, 'is_new': False, 'inode': 0, 'remote_type': '', 'version': 5, 'projected_version': 5, 'auth_pins': 0, 'is_auth': True, 'auth_state': {'replicas': {}}, 'replica_state': {'authority': [0, -2], 'replica_nonce': 0}, 'is_frozen': False, 'is_freezing': False, 'pins': {'dirty': 1}, 'nref': 1, 'lock': {}, 'versionlock': {'gather_set': [], 'state': 'lock', 'type': 'dversion', 'is_leased': False, 'num_rdlocks': 0, 'num_wrlocks': 0, 'num_xlocks': 0, 'xlock_by': {}}, 'states': ['auth', 'dirty']}]}], 'is_subvolume': False}] 2024-08-03T20:50:09.772 INFO:teuthology.nuke:Clearing teuthology firewall rules... 2024-08-03T20:50:09.772 DEBUG:teuthology.orchestra.run.smithi032:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-08-03T20:50:09.865 DEBUG:teuthology.orchestra.run.smithi118:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-08-03T20:50:09.957 INFO:teuthology.nuke:Cleared teuthology firewall rules. 2024-08-03T20:50:09.957 INFO:tasks.cephfs.mount:Terminating background process 2024-08-03T20:50:09.957 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-08-03T20:50:09.958 DEBUG:teuthology.orchestra.run.smithi032:> dd if=/proc/self/mounts of=/dev/stdout 2024-08-03T20:50:09.993 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053453,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6499208k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 none /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249604k,nr_inodes=812401,mode=700,uid=1000,gid=1316 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6499208k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-08-03T20:50:09.994 INFO:tasks.cephfs.mount:Terminating background process 2024-08-03T20:50:09.994 DEBUG:teuthology.orchestra.run.smithi118:> set -ex 2024-08-03T20:50:09.994 DEBUG:teuthology.orchestra.run.smithi118:> dd if=/proc/self/mounts of=/dev/stdout 2024-08-03T20:50:10.030 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052663,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497944k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248972k,nr_inodes=812243,mode=700,uid=1000,gid=1316 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497944k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-08-03T20:50:10.031 INFO:tasks.cephfs.mount:Terminating background process 2024-08-03T20:50:10.031 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-08-03T20:50:10.031 DEBUG:teuthology.orchestra.run.smithi032:> dd if=/proc/self/mounts of=/dev/stdout 2024-08-03T20:50:10.070 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053453,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6499208k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 none /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249604k,nr_inodes=812401,mode=700,uid=1000,gid=1316 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6499208k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-08-03T20:50:10.070 INFO:tasks.cephfs.mount:Terminating background process 2024-08-03T20:50:10.070 DEBUG:teuthology.orchestra.run.smithi118:> set -ex 2024-08-03T20:50:10.071 DEBUG:teuthology.orchestra.run.smithi118:> dd if=/proc/self/mounts of=/dev/stdout 2024-08-03T20:50:10.108 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052663,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497944k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248972k,nr_inodes=812243,mode=700,uid=1000,gid=1316 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497944k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-08-03T20:50:10.108 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-08-03T20:50:10.340 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:10.338+0000 7fbb8dc00640 1 -- 172.21.15.32:0/2132936912 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbb6c003c10 msgr2=0x7fbb6c004060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:50:10.340 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:10.338+0000 7fbb8dc00640 1 --2- 172.21.15.32:0/2132936912 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbb6c003c10 0x7fbb6c004060 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7fbb78009b50 tx=0x7fbb6c005830 comp rx=0 tx=0).stop 2024-08-03T20:50:10.340 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:10.338+0000 7fbb87400640 1 -- 172.21.15.32:0/2132936912 reap_dead start 2024-08-03T20:50:10.340 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:10.338+0000 7fbb8dc00640 1 -- 172.21.15.32:0/2132936912 shutdown_connections 2024-08-03T20:50:10.340 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:10.338+0000 7fbb8dc00640 1 -- 172.21.15.32:0/2132936912 >> 172.21.15.32:0/2132936912 conn(0x7fbb880b1bd0 msgr2=0x7fbb880b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:50:10.340 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:10.338+0000 7fbb8dc00640 1 -- 172.21.15.32:0/2132936912 shutdown_connections 2024-08-03T20:50:10.341 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:10.338+0000 7fbb8dc00640 1 -- 172.21.15.32:0/2132936912 wait complete. 2024-08-03T20:50:10.342 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:10.340+0000 7fbb8dc00640 1 Processor -- start 2024-08-03T20:50:10.342 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:10.341+0000 7fbb8dc00640 1 -- start start 2024-08-03T20:50:10.343 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:10.342+0000 7fbb8dc00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbb6c003c10 0x7fbb880c9250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:50:10.344 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:10.342+0000 7fbb8dc00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fbb880cb0b0 0x7fbb88074140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:50:10.344 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:10.342+0000 7fbb8dc00640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fbb881581a0 0x7fbb881585b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:50:10.344 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:10.342+0000 7fbb8dc00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fbb880ce800 con 0x7fbb6c003c10 2024-08-03T20:50:10.344 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:10.342+0000 7fbb8dc00640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7fbb880ce680 con 0x7fbb880cb0b0 2024-08-03T20:50:10.344 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:10.342+0000 7fbb8dc00640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7fbb880ce980 con 0x7fbb881581a0 2024-08-03T20:50:10.345 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:10.343+0000 7fbb7fe00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fbb880cb0b0 0x7fbb88074140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:50:10.345 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:10.343+0000 7fbb7fe00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fbb880cb0b0 0x7fbb88074140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.32:53736/0 (socket says 172.21.15.32:53736) 2024-08-03T20:50:10.345 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:10.343+0000 7fbb7fe00640 1 -- 172.21.15.32:0/2367017873 learned_addr learned my addr 172.21.15.32:0/2367017873 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:50:10.345 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:10.343+0000 7fbb87400640 1 --2- 172.21.15.32:0/2367017873 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fbb881581a0 0x7fbb881585b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:50:10.345 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:10.343+0000 7fbb86a00640 1 --2- 172.21.15.32:0/2367017873 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbb6c003c10 0x7fbb880c9250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:50:10.346 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:10.343+0000 7fbb7fe00640 1 -- 172.21.15.32:0/2367017873 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fbb881581a0 msgr2=0x7fbb881585b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:50:10.346 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:10.343+0000 7fbb7fe00640 1 --2- 172.21.15.32:0/2367017873 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fbb881581a0 0x7fbb881585b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:10.346 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:10.343+0000 7fbb7fe00640 1 -- 172.21.15.32:0/2367017873 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbb6c003c10 msgr2=0x7fbb880c9250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:50:10.346 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:10.343+0000 7fbb7fe00640 1 --2- 172.21.15.32:0/2367017873 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbb6c003c10 0x7fbb880c9250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:10.346 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:10.343+0000 7fbb7fe00640 1 -- 172.21.15.32:0/2367017873 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbb88158af0 con 0x7fbb880cb0b0 2024-08-03T20:50:10.346 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:10.343+0000 7fbb7fe00640 1 --2- 172.21.15.32:0/2367017873 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fbb880cb0b0 0x7fbb88074140 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fbb70006600 tx=0x7fbb700039c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:50:10.346 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:10.344+0000 7fbb7ea00640 1 -- 172.21.15.32:0/2367017873 <== mon.1 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb70015070 con 0x7fbb880cb0b0 2024-08-03T20:50:10.346 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:10.344+0000 7fbb7ea00640 1 -- 172.21.15.32:0/2367017873 <== mon.1 v2:172.21.15.118:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fbb70011920 con 0x7fbb880cb0b0 2024-08-03T20:50:10.346 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:10.344+0000 7fbb7ea00640 1 -- 172.21.15.32:0/2367017873 <== mon.1 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb70011d70 con 0x7fbb880cb0b0 2024-08-03T20:50:10.346 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:10.344+0000 7fbb8dc00640 1 -- 172.21.15.32:0/2367017873 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbb881f88a0 con 0x7fbb880cb0b0 2024-08-03T20:50:10.346 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:10.344+0000 7fbb8dc00640 1 -- 172.21.15.32:0/2367017873 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbb881f8e60 con 0x7fbb880cb0b0 2024-08-03T20:50:10.347 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:10.345+0000 7fbb8dc00640 1 -- 172.21.15.32:0/2367017873 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbb881f8610 con 0x7fbb880cb0b0 2024-08-03T20:50:10.347 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:10.346+0000 7fbb7ea00640 1 -- 172.21.15.32:0/2367017873 <== mon.1 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7fbb70018070 con 0x7fbb880cb0b0 2024-08-03T20:50:10.348 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:10.347+0000 7fbb7ea00640 1 --2- 172.21.15.32:0/2367017873 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fbb64077040 0x7fbb64079550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:50:10.348 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:10.347+0000 7fbb7ea00640 1 -- 172.21.15.32:0/2367017873 <== mon.1 v2:172.21.15.118:3300/0 5 ==== osd_map(46..46 src has 1..46) ==== 5795+0+0 (secure 0 0 0) 0x7fbb70099590 con 0x7fbb880cb0b0 2024-08-03T20:50:10.349 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:10.347+0000 7fbb86a00640 1 --2- 172.21.15.32:0/2367017873 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fbb64077040 0x7fbb64079550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:50:10.349 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:10.348+0000 7fbb86a00640 1 --2- 172.21.15.32:0/2367017873 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fbb64077040 0x7fbb64079550 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fbb78035000 tx=0x7fbb7802d040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:50:10.354 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:10.352+0000 7fbb7ea00640 1 -- 172.21.15.32:0/2367017873 <== mon.1 v2:172.21.15.118:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7fbb88158af0 con 0x7fbb880cb0b0 2024-08-03T20:50:10.601 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:10.600+0000 7fbb8dc00640 1 -- 172.21.15.32:0/2367017873 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fbb880cef10 con 0x7fbb880cb0b0 2024-08-03T20:50:10.602 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:10.601+0000 7fbb7ea00640 1 -- 172.21.15.32:0/2367017873 <== mon.1 v2:172.21.15.118:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 42 v42) ==== 94+0+5230 (secure 0 0 0) 0x7fbb70062050 con 0x7fbb880cb0b0 2024-08-03T20:50:10.603 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-03T20:50:10.603 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":42,"btime":"2024-08-03T20:50:10:054829+0000","default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4997,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6837/2688500839","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6836","nonce":2688500839},{"type":"v1","addr":"172.21.15.118:6837","nonce":2688500839}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":30},{"gid":5266,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6835/2381075275","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6834","nonce":2381075275},{"type":"v1","addr":"172.21.15.118:6835","nonce":2381075275}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":28},{"gid":5421,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6833/1058332640","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":1058332640},{"type":"v1","addr":"172.21.15.32:6833","nonce":1058332640}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":29}],"filesystems":[{"mdsmap":{"epoch":42,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-08-03T20:49:47.064654+0000","modified":"2024-08-03T20:50:10.054820+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":46,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5628},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5628":{"gid":5628,"name":"c","rank":0,"incarnation":39,"state":"up:active","state_seq":4,"addr":"172.21.15.32:6835/1726912144","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1726912144},{"type":"v1","addr":"172.21.15.32:6835","nonce":1726912144}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5628,"qdb_cluster":[5628]},"id":3}]} 2024-08-03T20:50:10.603 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 42 2024-08-03T20:50:10.605 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:10.604+0000 7fbb8dc00640 1 -- 172.21.15.32:0/2367017873 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fbb64077040 msgr2=0x7fbb64079550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:50:10.606 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:10.604+0000 7fbb8dc00640 1 --2- 172.21.15.32:0/2367017873 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fbb64077040 0x7fbb64079550 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fbb78035000 tx=0x7fbb7802d040 comp rx=0 tx=0).stop 2024-08-03T20:50:10.606 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:10.604+0000 7fbb8dc00640 1 -- 172.21.15.32:0/2367017873 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fbb880cb0b0 msgr2=0x7fbb88074140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:50:10.606 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:10.604+0000 7fbb8dc00640 1 --2- 172.21.15.32:0/2367017873 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fbb880cb0b0 0x7fbb88074140 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fbb70006600 tx=0x7fbb700039c0 comp rx=0 tx=0).stop 2024-08-03T20:50:10.606 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:10.604+0000 7fbb8dc00640 1 -- 172.21.15.32:0/2367017873 shutdown_connections 2024-08-03T20:50:10.606 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:10.604+0000 7fbb8dc00640 1 --2- 172.21.15.32:0/2367017873 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fbb64077040 0x7fbb64079550 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:10.606 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:10.604+0000 7fbb8dc00640 1 --2- 172.21.15.32:0/2367017873 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fbb880cb0b0 0x7fbb88074140 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:10.606 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:10.604+0000 7fbb8dc00640 1 --2- 172.21.15.32:0/2367017873 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fbb881581a0 0x7fbb881585b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:10.606 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:10.604+0000 7fbb8dc00640 1 --2- 172.21.15.32:0/2367017873 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbb6c003c10 0x7fbb880c9250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:10.606 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:10.604+0000 7fbb8dc00640 1 -- 172.21.15.32:0/2367017873 >> 172.21.15.32:0/2367017873 conn(0x7fbb880b1bd0 msgr2=0x7fbb88150ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:50:10.606 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:10.604+0000 7fbb8dc00640 1 -- 172.21.15.32:0/2367017873 shutdown_connections 2024-08-03T20:50:10.606 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:10.604+0000 7fbb8dc00640 1 -- 172.21.15.32:0/2367017873 wait complete. 2024-08-03T20:50:10.627 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-08-03T20:50:10.856 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:10.854+0000 7f1f02600640 1 -- 172.21.15.32:0/515809704 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1efc0cea60 msgr2=0x7f1efc0cee70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:50:10.856 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:10.854+0000 7f1f02600640 1 --2- 172.21.15.32:0/515809704 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1efc0cea60 0x7f1efc0cee70 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7f1efc0d1550 tx=0x7f1ef402a540 comp rx=0 tx=0).stop 2024-08-03T20:50:10.856 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:10.854+0000 7f1f02600640 1 -- 172.21.15.32:0/515809704 shutdown_connections 2024-08-03T20:50:10.856 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:10.854+0000 7f1f02600640 1 --2- 172.21.15.32:0/515809704 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1efc0cea60 0x7f1efc0cee70 unknown :-1 s=CLOSED pgs=499 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:10.856 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:10.854+0000 7f1f02600640 1 --2- 172.21.15.32:0/515809704 >> v2:172.21.15.118:3300/0 conn(0x7f1efc069300 0x7f1efc0b1300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:10.857 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:10.854+0000 7f1f02600640 1 -- 172.21.15.32:0/515809704 >> 172.21.15.32:0/515809704 conn(0x7f1efc0b1fe0 msgr2=0x7f1efc0b23f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:50:10.857 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:10.854+0000 7f1f02600640 1 -- 172.21.15.32:0/515809704 shutdown_connections 2024-08-03T20:50:10.857 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:10.854+0000 7f1f02600640 1 -- 172.21.15.32:0/515809704 wait complete. 2024-08-03T20:50:10.858 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:10.857+0000 7f1f02600640 1 Processor -- start 2024-08-03T20:50:10.858 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:10.857+0000 7f1f02600640 1 -- start start 2024-08-03T20:50:10.860 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:10.858+0000 7f1f02600640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f1efc0b1840 0x7f1efc15ba40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:50:10.860 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:10.858+0000 7f1f02600640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f1efc0cea60 0x7f1efc0aff50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:50:10.860 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:10.858+0000 7f1f02600640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1efc0c7d80 0x7f1efc0b0490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:50:10.860 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:10.858+0000 7f1f02600640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f1efc0b5390 con 0x7f1efc0c7d80 2024-08-03T20:50:10.860 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:10.858+0000 7f1f02600640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f1efc0b5210 con 0x7f1efc0b1840 2024-08-03T20:50:10.860 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:10.858+0000 7f1f02600640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7f1efc165ab0 con 0x7f1efc0cea60 2024-08-03T20:50:10.861 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:10.858+0000 7f1efbe00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1efc0c7d80 0x7f1efc0b0490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:50:10.861 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:10.858+0000 7f1efbe00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1efc0c7d80 0x7f1efc0b0490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:39912/0 (socket says 172.21.15.32:39912) 2024-08-03T20:50:10.861 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:10.858+0000 7f1efbe00640 1 -- 172.21.15.32:0/505868973 learned_addr learned my addr 172.21.15.32:0/505868973 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:50:10.861 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:10.859+0000 7f1efb400640 1 --2- 172.21.15.32:0/505868973 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f1efc0b1840 0x7f1efc15ba40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:50:10.861 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:10.859+0000 7f1efaa00640 1 --2- 172.21.15.32:0/505868973 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f1efc0cea60 0x7f1efc0aff50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:50:10.861 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:10.859+0000 7f1efbe00640 1 -- 172.21.15.32:0/505868973 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f1efc0cea60 msgr2=0x7f1efc0aff50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:50:10.861 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:10.859+0000 7f1efbe00640 1 --2- 172.21.15.32:0/505868973 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f1efc0cea60 0x7f1efc0aff50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:10.861 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:10.859+0000 7f1efbe00640 1 -- 172.21.15.32:0/505868973 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f1efc0b1840 msgr2=0x7f1efc15ba40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:50:10.861 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:10.859+0000 7f1efbe00640 1 --2- 172.21.15.32:0/505868973 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f1efc0b1840 0x7f1efc15ba40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:10.861 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:10.859+0000 7f1efbe00640 1 -- 172.21.15.32:0/505868973 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1efc15bf80 con 0x7f1efc0c7d80 2024-08-03T20:50:10.861 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:10.859+0000 7f1efaa00640 1 --2- 172.21.15.32:0/505868973 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f1efc0cea60 0x7f1efc0aff50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-03T20:50:10.862 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:10.859+0000 7f1efbe00640 1 --2- 172.21.15.32:0/505868973 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1efc0c7d80 0x7f1efc0b0490 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7f1ef402ab30 tx=0x7f1ef4033ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:50:10.862 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:10.859+0000 7f1ef3400640 1 -- 172.21.15.32:0/505868973 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ef4031b50 con 0x7f1efc0c7d80 2024-08-03T20:50:10.862 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:10.860+0000 7f1ef3400640 1 -- 172.21.15.32:0/505868973 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f1ef4031e00 con 0x7f1efc0c7d80 2024-08-03T20:50:10.863 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:10.860+0000 7f1f02600640 1 -- 172.21.15.32:0/505868973 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1efc1f6920 con 0x7f1efc0c7d80 2024-08-03T20:50:10.863 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:10.860+0000 7f1ef3400640 1 -- 172.21.15.32:0/505868973 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ef4005280 con 0x7f1efc0c7d80 2024-08-03T20:50:10.863 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:10.860+0000 7f1f02600640 1 -- 172.21.15.32:0/505868973 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1efc1f6ee0 con 0x7f1efc0c7d80 2024-08-03T20:50:10.863 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:10.861+0000 7f1f02600640 1 -- 172.21.15.32:0/505868973 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1efc069300 con 0x7f1efc0c7d80 2024-08-03T20:50:10.863 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:10.862+0000 7f1ef3400640 1 -- 172.21.15.32:0/505868973 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f1ef4005420 con 0x7f1efc0c7d80 2024-08-03T20:50:10.864 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:10.863+0000 7f1ef3400640 1 --2- 172.21.15.32:0/505868973 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f1edc077040 0x7f1edc079550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:50:10.864 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:10.863+0000 7f1ef3400640 1 -- 172.21.15.32:0/505868973 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(46..46 src has 1..46) ==== 5795+0+0 (secure 0 0 0) 0x7f1ef40b99b0 con 0x7f1efc0c7d80 2024-08-03T20:50:10.865 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:10.863+0000 7f1efb400640 1 --2- 172.21.15.32:0/505868973 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f1edc077040 0x7f1edc079550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:50:10.865 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:10.864+0000 7f1efb400640 1 --2- 172.21.15.32:0/505868973 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f1edc077040 0x7f1edc079550 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f1eec005fd0 tx=0x7f1eec009500 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:50:10.869 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:10.867+0000 7f1ef3400640 1 -- 172.21.15.32:0/505868973 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7f1ef403f030 con 0x7f1efc0c7d80 2024-08-03T20:50:11.042 INFO:tasks.ceph.mds.c.smithi032.stderr:2024-08-03T20:50:11.041+0000 7f1a5f400640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-08-03T20:50:11.115 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:11.113+0000 7f1f02600640 1 -- 172.21.15.32:0/505868973 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f1efc0afd30 con 0x7f1efc0c7d80 2024-08-03T20:50:11.116 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:11.114+0000 7f1ef3400640 1 -- 172.21.15.32:0/505868973 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 42 v42) ==== 94+0+5230 (secure 0 0 0) 0x7f1ef4082460 con 0x7f1efc0c7d80 2024-08-03T20:50:11.116 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-03T20:50:11.116 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":42,"btime":"2024-08-03T20:50:10:054829+0000","default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4997,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6837/2688500839","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6836","nonce":2688500839},{"type":"v1","addr":"172.21.15.118:6837","nonce":2688500839}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":30},{"gid":5266,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6835/2381075275","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6834","nonce":2381075275},{"type":"v1","addr":"172.21.15.118:6835","nonce":2381075275}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":28},{"gid":5421,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6833/1058332640","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":1058332640},{"type":"v1","addr":"172.21.15.32:6833","nonce":1058332640}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":29}],"filesystems":[{"mdsmap":{"epoch":42,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-08-03T20:49:47.064654+0000","modified":"2024-08-03T20:50:10.054820+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":46,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5628},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5628":{"gid":5628,"name":"c","rank":0,"incarnation":39,"state":"up:active","state_seq":4,"addr":"172.21.15.32:6835/1726912144","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1726912144},{"type":"v1","addr":"172.21.15.32:6835","nonce":1726912144}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5628,"qdb_cluster":[5628]},"id":3}]} 2024-08-03T20:50:11.116 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 42 2024-08-03T20:50:11.118 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:11.116+0000 7f1f02600640 1 -- 172.21.15.32:0/505868973 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f1edc077040 msgr2=0x7f1edc079550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:50:11.118 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:11.116+0000 7f1f02600640 1 --2- 172.21.15.32:0/505868973 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f1edc077040 0x7f1edc079550 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f1eec005fd0 tx=0x7f1eec009500 comp rx=0 tx=0).stop 2024-08-03T20:50:11.118 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:11.116+0000 7f1f02600640 1 -- 172.21.15.32:0/505868973 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1efc0c7d80 msgr2=0x7f1efc0b0490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:50:11.118 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:11.116+0000 7f1f02600640 1 --2- 172.21.15.32:0/505868973 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1efc0c7d80 0x7f1efc0b0490 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7f1ef402ab30 tx=0x7f1ef4033ba0 comp rx=0 tx=0).stop 2024-08-03T20:50:11.118 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:11.116+0000 7f1f02600640 1 -- 172.21.15.32:0/505868973 shutdown_connections 2024-08-03T20:50:11.119 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:11.116+0000 7f1f02600640 1 --2- 172.21.15.32:0/505868973 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f1edc077040 0x7f1edc079550 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:11.119 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:11.116+0000 7f1f02600640 1 --2- 172.21.15.32:0/505868973 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1efc0c7d80 0x7f1efc0b0490 unknown :-1 s=CLOSED pgs=500 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:11.119 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:11.116+0000 7f1f02600640 1 --2- 172.21.15.32:0/505868973 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f1efc0cea60 0x7f1efc0aff50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:11.119 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:11.116+0000 7f1f02600640 1 --2- 172.21.15.32:0/505868973 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f1efc0b1840 0x7f1efc15ba40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:11.119 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:11.116+0000 7f1f02600640 1 -- 172.21.15.32:0/505868973 >> 172.21.15.32:0/505868973 conn(0x7f1efc0b1fe0 msgr2=0x7f1efc161f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:50:11.119 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:11.117+0000 7f1f02600640 1 -- 172.21.15.32:0/505868973 shutdown_connections 2024-08-03T20:50:11.119 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:11.117+0000 7f1f02600640 1 -- 172.21.15.32:0/505868973 wait complete. 2024-08-03T20:50:11.140 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-08-03T20:50:11.367 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:11.366+0000 7fefe7200640 1 -- 172.21.15.32:0/712330494 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fefe815a310 msgr2=0x7fefe80aefb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:50:11.368 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:11.366+0000 7fefe7200640 1 --2- 172.21.15.32:0/712330494 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fefe815a310 0x7fefe80aefb0 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7fefe80725a0 tx=0x7fefd802a560 comp rx=0 tx=0).stop 2024-08-03T20:50:11.368 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:11.366+0000 7fefe7200640 1 -- 172.21.15.32:0/712330494 shutdown_connections 2024-08-03T20:50:11.368 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:11.366+0000 7fefe7200640 1 --2- 172.21.15.32:0/712330494 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fefe815e000 0x7fefe815e410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:11.368 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:11.366+0000 7fefe7200640 1 --2- 172.21.15.32:0/712330494 >> v2:172.21.15.118:3300/0 conn(0x7fefe80ca3b0 0x7fefe80ca7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:11.368 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:11.366+0000 7fefe7200640 1 --2- 172.21.15.32:0/712330494 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fefe815a310 0x7fefe80aefb0 unknown :-1 s=CLOSED pgs=501 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:11.368 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:11.366+0000 7fefe7200640 1 -- 172.21.15.32:0/712330494 >> 172.21.15.32:0/712330494 conn(0x7fefe80b2120 msgr2=0x7fefe80cfd60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:50:11.368 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:11.366+0000 7fefe7200640 1 -- 172.21.15.32:0/712330494 shutdown_connections 2024-08-03T20:50:11.368 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:11.366+0000 7fefe7200640 1 -- 172.21.15.32:0/712330494 wait complete. 2024-08-03T20:50:11.369 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:11.368+0000 7fefe7200640 1 Processor -- start 2024-08-03T20:50:11.369 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:11.368+0000 7fefe7200640 1 -- start start 2024-08-03T20:50:11.372 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:11.370+0000 7fefe7200640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fefe80ca3b0 0x7fefe8152320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:50:11.372 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:11.370+0000 7fefe7200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fefe815a310 0x7fefe80b4970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:50:11.372 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:11.370+0000 7fefe7200640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fefe815e000 0x7fefe80b4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:50:11.372 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:11.370+0000 7fefe7200640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fefe80af670 con 0x7fefe815a310 2024-08-03T20:50:11.372 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:11.370+0000 7fefe7200640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7fefe80af4f0 con 0x7fefe80ca3b0 2024-08-03T20:50:11.372 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:11.370+0000 7fefe7200640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7fefe80706e0 con 0x7fefe815e000 2024-08-03T20:50:11.372 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:11.370+0000 7fefdfe00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fefe815a310 0x7fefe80b4970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:50:11.372 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:11.370+0000 7fefdfe00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fefe815a310 0x7fefe80b4970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:39930/0 (socket says 172.21.15.32:39930) 2024-08-03T20:50:11.372 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:11.370+0000 7fefdfe00640 1 -- 172.21.15.32:0/1872957859 learned_addr learned my addr 172.21.15.32:0/1872957859 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:50:11.373 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:11.370+0000 7fefe5e00640 1 --2- 172.21.15.32:0/1872957859 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fefe80ca3b0 0x7fefe8152320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:50:11.373 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:11.370+0000 7fefe6800640 1 --2- 172.21.15.32:0/1872957859 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fefe815e000 0x7fefe80b4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:50:11.373 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:11.371+0000 7fefdfe00640 1 -- 172.21.15.32:0/1872957859 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fefe815e000 msgr2=0x7fefe80b4eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:50:11.373 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:11.371+0000 7fefdfe00640 1 --2- 172.21.15.32:0/1872957859 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fefe815e000 0x7fefe80b4eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:11.373 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:11.371+0000 7fefdfe00640 1 -- 172.21.15.32:0/1872957859 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fefe80ca3b0 msgr2=0x7fefe8152320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:50:11.373 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:11.371+0000 7fefdfe00640 1 --2- 172.21.15.32:0/1872957859 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fefe80ca3b0 0x7fefe8152320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:11.373 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:11.371+0000 7fefdfe00640 1 -- 172.21.15.32:0/1872957859 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fefe80c95e0 con 0x7fefe815a310 2024-08-03T20:50:11.374 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:11.371+0000 7fefdfe00640 1 --2- 172.21.15.32:0/1872957859 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fefe815a310 0x7fefe80b4970 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7fefd000d770 tx=0x7fefd000dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:50:11.374 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:11.371+0000 7fefde000640 1 -- 172.21.15.32:0/1872957859 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fefd0017070 con 0x7fefe815a310 2024-08-03T20:50:11.374 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:11.371+0000 7fefde000640 1 -- 172.21.15.32:0/1872957859 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fefd0009d20 con 0x7fefe815a310 2024-08-03T20:50:11.374 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:11.371+0000 7fefde000640 1 -- 172.21.15.32:0/1872957859 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fefd00106c0 con 0x7fefe815a310 2024-08-03T20:50:11.374 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:11.371+0000 7fefe7200640 1 -- 172.21.15.32:0/1872957859 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fefe80c98c0 con 0x7fefe815a310 2024-08-03T20:50:11.374 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:11.371+0000 7fefe7200640 1 -- 172.21.15.32:0/1872957859 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fefe80b53f0 con 0x7fefe815a310 2024-08-03T20:50:11.374 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:11.373+0000 7fefe7200640 1 -- 172.21.15.32:0/1872957859 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fefe80aefb0 con 0x7fefe815a310 2024-08-03T20:50:11.375 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:11.374+0000 7fefde000640 1 -- 172.21.15.32:0/1872957859 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7fefd001a070 con 0x7fefe815a310 2024-08-03T20:50:11.376 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:11.375+0000 7fefde000640 1 --2- 172.21.15.32:0/1872957859 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fefc4077040 0x7fefc4079550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:50:11.376 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:11.375+0000 7fefde000640 1 -- 172.21.15.32:0/1872957859 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(46..46 src has 1..46) ==== 5795+0+0 (secure 0 0 0) 0x7fefd0098970 con 0x7fefe815a310 2024-08-03T20:50:11.377 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:11.375+0000 7fefe5e00640 1 --2- 172.21.15.32:0/1872957859 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fefc4077040 0x7fefc4079550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:50:11.377 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:11.376+0000 7fefe5e00640 1 --2- 172.21.15.32:0/1872957859 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fefc4077040 0x7fefc4079550 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7fefd8033000 tx=0x7fefd8004730 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:50:11.380 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:11.379+0000 7fefde000640 1 -- 172.21.15.32:0/1872957859 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7fefd00614b0 con 0x7fefe815a310 2024-08-03T20:50:11.590 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:11.589+0000 7fefe7200640 1 -- 172.21.15.32:0/1872957859 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fefe8158890 con 0x7fefe815a310 2024-08-03T20:50:11.591 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:11.590+0000 7fefde000640 1 -- 172.21.15.32:0/1872957859 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v46) ==== 74+0+16680 (secure 0 0 0) 0x7fefd0065a00 con 0x7fefe815a310 2024-08-03T20:50:11.592 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-03T20:50:11.592 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":46,"fsid":"40ab141a-01d7-49b9-ae3c-2edec9876e95","created":"2024-08-03T20:47:14.957362+0000","modified":"2024-08-03T20:50:05.030282+0000","last_up_change":"2024-08-03T20:47:28.874451+0000","last_in_change":"2024-08-03T20:47:20.079442+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-08-03T20:47:28.261300+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_metadata","create_time":"2024-08-03T20:49:45.020976+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"44","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_data","create_time":"2024-08-03T20:49:46.012432+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"45","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5a83e759-9035-438b-889a-0a6af1f0a0d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6816","nonce":3066156116},{"type":"v1","addr":"172.21.15.32:6817","nonce":3066156116}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6818","nonce":3066156116},{"type":"v1","addr":"172.21.15.32:6819","nonce":3066156116}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6822","nonce":3066156116},{"type":"v1","addr":"172.21.15.32:6823","nonce":3066156116}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6820","nonce":3066156116},{"type":"v1","addr":"172.21.15.32:6821","nonce":3066156116}]},"public_addr":"172.21.15.32:6817/3066156116","cluster_addr":"172.21.15.32:6819/3066156116","heartbeat_back_addr":"172.21.15.32:6823/3066156116","heartbeat_front_addr":"172.21.15.32:6821/3066156116","state":["exists","up"]},{"osd":1,"uuid":"d7c5c210-da4f-49f9-b9e7-1dfb02daa4bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6824","nonce":1551241759},{"type":"v1","addr":"172.21.15.32:6825","nonce":1551241759}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6826","nonce":1551241759},{"type":"v1","addr":"172.21.15.32:6827","nonce":1551241759}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6830","nonce":1551241759},{"type":"v1","addr":"172.21.15.32:6831","nonce":1551241759}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6828","nonce":1551241759},{"type":"v1","addr":"172.21.15.32:6829","nonce":1551241759}]},"public_addr":"172.21.15.32:6825/1551241759","cluster_addr":"172.21.15.32:6827/1551241759","heartbeat_back_addr":"172.21.15.32:6831/1551241759","heartbeat_front_addr":"172.21.15.32:6829/1551241759","state":["exists","up"]},{"osd":2,"uuid":"57f2e6de-a201-45c2-8981-8e56a4875ebd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6808","nonce":3806708944},{"type":"v1","addr":"172.21.15.32:6809","nonce":3806708944}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6810","nonce":3806708944},{"type":"v1","addr":"172.21.15.32:6811","nonce":3806708944}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6814","nonce":3806708944},{"type":"v1","addr":"172.21.15.32:6815","nonce":3806708944}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6812","nonce":3806708944},{"type":"v1","addr":"172.21.15.32:6813","nonce":3806708944}]},"public_addr":"172.21.15.32:6809/3806708944","cluster_addr":"172.21.15.32:6811/3806708944","heartbeat_back_addr":"172.21.15.32:6815/3806708944","heartbeat_front_addr":"172.21.15.32:6813/3806708944","state":["exists","up"]},{"osd":3,"uuid":"6439a280-da45-4148-8130-f16323935ff3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6800","nonce":9088303},{"type":"v1","addr":"172.21.15.32:6801","nonce":9088303}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6802","nonce":9088303},{"type":"v1","addr":"172.21.15.32:6803","nonce":9088303}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6806","nonce":9088303},{"type":"v1","addr":"172.21.15.32:6807","nonce":9088303}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6804","nonce":9088303},{"type":"v1","addr":"172.21.15.32:6805","nonce":9088303}]},"public_addr":"172.21.15.32:6801/9088303","cluster_addr":"172.21.15.32:6803/9088303","heartbeat_back_addr":"172.21.15.32:6807/9088303","heartbeat_front_addr":"172.21.15.32:6805/9088303","state":["exists","up"]},{"osd":4,"uuid":"4f902eff-1e18-418d-82c9-2151c69524fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6802","nonce":864752275},{"type":"v1","addr":"172.21.15.118:6803","nonce":864752275}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6804","nonce":864752275},{"type":"v1","addr":"172.21.15.118:6805","nonce":864752275}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6808","nonce":864752275},{"type":"v1","addr":"172.21.15.118:6809","nonce":864752275}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6806","nonce":864752275},{"type":"v1","addr":"172.21.15.118:6807","nonce":864752275}]},"public_addr":"172.21.15.118:6803/864752275","cluster_addr":"172.21.15.118:6805/864752275","heartbeat_back_addr":"172.21.15.118:6809/864752275","heartbeat_front_addr":"172.21.15.118:6807/864752275","state":["exists","up"]},{"osd":5,"uuid":"bfd9feac-dd06-458c-91b9-af7f175f612c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6810","nonce":2724891770},{"type":"v1","addr":"172.21.15.118:6811","nonce":2724891770}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6812","nonce":2724891770},{"type":"v1","addr":"172.21.15.118:6813","nonce":2724891770}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6816","nonce":2724891770},{"type":"v1","addr":"172.21.15.118:6817","nonce":2724891770}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6814","nonce":2724891770},{"type":"v1","addr":"172.21.15.118:6815","nonce":2724891770}]},"public_addr":"172.21.15.118:6811/2724891770","cluster_addr":"172.21.15.118:6813/2724891770","heartbeat_back_addr":"172.21.15.118:6817/2724891770","heartbeat_front_addr":"172.21.15.118:6815/2724891770","state":["exists","up"]},{"osd":6,"uuid":"5ed5a279-9d8e-416d-bbc7-7edd3e6d5705","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6825","nonce":801947386},{"type":"v1","addr":"172.21.15.118:6827","nonce":801947386}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6828","nonce":801947386},{"type":"v1","addr":"172.21.15.118:6829","nonce":801947386}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6832","nonce":801947386},{"type":"v1","addr":"172.21.15.118:6833","nonce":801947386}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6830","nonce":801947386},{"type":"v1","addr":"172.21.15.118:6831","nonce":801947386}]},"public_addr":"172.21.15.118:6827/801947386","cluster_addr":"172.21.15.118:6829/801947386","heartbeat_back_addr":"172.21.15.118:6833/801947386","heartbeat_front_addr":"172.21.15.118:6831/801947386","state":["exists","up"]},{"osd":7,"uuid":"f2086c47-875e-4451-a436-2008d1ea16af","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6818","nonce":653000048},{"type":"v1","addr":"172.21.15.118:6819","nonce":653000048}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6820","nonce":653000048},{"type":"v1","addr":"172.21.15.118:6821","nonce":653000048}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6824","nonce":653000048},{"type":"v1","addr":"172.21.15.118:6826","nonce":653000048}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6822","nonce":653000048},{"type":"v1","addr":"172.21.15.118:6823","nonce":653000048}]},"public_addr":"172.21.15.118:6819/653000048","cluster_addr":"172.21.15.118:6821/653000048","heartbeat_back_addr":"172.21.15.118:6826/653000048","heartbeat_front_addr":"172.21.15.118:6823/653000048","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-03T20:47:25.491994+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-03T20:47:25.587358+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-03T20:47:25.454175+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-03T20:47:25.377734+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-03T20:47:25.412351+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-03T20:47:25.499815+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-03T20:47:25.575044+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-03T20:47:25.552383+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.32:6835/678469885":"2024-08-04T20:50:05.030253+0000","172.21.15.32:6834/678469885":"2024-08-04T20:50:05.030253+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-08-03T20:50:11.594 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:11.592+0000 7fefe7200640 1 -- 172.21.15.32:0/1872957859 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fefc4077040 msgr2=0x7fefc4079550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:50:11.594 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:11.592+0000 7fefe7200640 1 --2- 172.21.15.32:0/1872957859 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fefc4077040 0x7fefc4079550 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7fefd8033000 tx=0x7fefd8004730 comp rx=0 tx=0).stop 2024-08-03T20:50:11.594 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:11.592+0000 7fefe7200640 1 -- 172.21.15.32:0/1872957859 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fefe815a310 msgr2=0x7fefe80b4970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:50:11.594 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:11.592+0000 7fefe7200640 1 --2- 172.21.15.32:0/1872957859 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fefe815a310 0x7fefe80b4970 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7fefd000d770 tx=0x7fefd000dc30 comp rx=0 tx=0).stop 2024-08-03T20:50:11.594 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:11.592+0000 7fefe7200640 1 -- 172.21.15.32:0/1872957859 shutdown_connections 2024-08-03T20:50:11.594 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:11.592+0000 7fefe7200640 1 --2- 172.21.15.32:0/1872957859 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fefc4077040 0x7fefc4079550 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:11.594 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:11.592+0000 7fefe7200640 1 --2- 172.21.15.32:0/1872957859 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fefe815e000 0x7fefe80b4eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:11.595 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:11.592+0000 7fefe7200640 1 --2- 172.21.15.32:0/1872957859 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fefe815a310 0x7fefe80b4970 unknown :-1 s=CLOSED pgs=502 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:11.595 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:11.592+0000 7fefe7200640 1 --2- 172.21.15.32:0/1872957859 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fefe80ca3b0 0x7fefe8152320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:11.595 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:11.592+0000 7fefe7200640 1 -- 172.21.15.32:0/1872957859 >> 172.21.15.32:0/1872957859 conn(0x7fefe80b2120 msgr2=0x7fefe80b0c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:50:11.595 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:11.592+0000 7fefe7200640 1 -- 172.21.15.32:0/1872957859 shutdown_connections 2024-08-03T20:50:11.595 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:11.592+0000 7fefe7200640 1 -- 172.21.15.32:0/1872957859 wait complete. 2024-08-03T20:50:11.617 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-08-03T20:50:11.617 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-08-03T20:50:11.846 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:11.843+0000 7f248a400640 1 -- 172.21.15.32:0/3345453911 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2484073e60 msgr2=0x7f2484074270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:50:11.846 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:11.843+0000 7f248a400640 1 --2- 172.21.15.32:0/3345453911 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2484073e60 0x7f2484074270 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7f2484071700 tx=0x7f247002a4a0 comp rx=0 tx=0).stop 2024-08-03T20:50:11.846 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:11.844+0000 7f248a400640 1 -- 172.21.15.32:0/3345453911 shutdown_connections 2024-08-03T20:50:11.846 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:11.844+0000 7f248a400640 1 --2- 172.21.15.32:0/3345453911 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2484073e60 0x7f2484074270 unknown :-1 s=CLOSED pgs=504 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:11.846 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:11.844+0000 7f248a400640 1 --2- 172.21.15.32:0/3345453911 >> v2:172.21.15.118:3300/0 conn(0x7f248406e930 0x7f2484144f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:11.846 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:11.844+0000 7f248a400640 1 -- 172.21.15.32:0/3345453911 >> 172.21.15.32:0/3345453911 conn(0x7f24840b1bd0 msgr2=0x7f24840b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:50:11.846 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:11.844+0000 7f248a400640 1 -- 172.21.15.32:0/3345453911 shutdown_connections 2024-08-03T20:50:11.846 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:11.844+0000 7f248a400640 1 -- 172.21.15.32:0/3345453911 wait complete. 2024-08-03T20:50:11.847 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:11.846+0000 7f248a400640 1 Processor -- start 2024-08-03T20:50:11.847 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:11.846+0000 7f248a400640 1 -- start start 2024-08-03T20:50:11.850 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:11.848+0000 7f248a400640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f248406e930 0x7f24840cafd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:50:11.850 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:11.848+0000 7f248a400640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f2484073e60 0x7f24841440c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:50:11.850 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:11.848+0000 7f248a400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f24840747b0 0x7f24840c8150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:50:11.850 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:11.848+0000 7f248a400640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f24840cf9c0 con 0x7f24840747b0 2024-08-03T20:50:11.850 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:11.848+0000 7f248a400640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f24840cf840 con 0x7f2484073e60 2024-08-03T20:50:11.850 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:11.848+0000 7f248a400640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7f24840ccff0 con 0x7f248406e930 2024-08-03T20:50:11.850 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:11.848+0000 7f2483e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f24840747b0 0x7f24840c8150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:50:11.850 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:11.848+0000 7f2483e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f24840747b0 0x7f24840c8150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:39948/0 (socket says 172.21.15.32:39948) 2024-08-03T20:50:11.850 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:11.848+0000 7f2483e00640 1 -- 172.21.15.32:0/1135871196 learned_addr learned my addr 172.21.15.32:0/1135871196 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:50:11.850 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:11.848+0000 7f2482a00640 1 --2- 172.21.15.32:0/1135871196 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f2484073e60 0x7f24841440c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:50:11.851 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:11.848+0000 7f2483400640 1 --2- 172.21.15.32:0/1135871196 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f248406e930 0x7f24840cafd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:50:11.851 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:11.848+0000 7f2483e00640 1 -- 172.21.15.32:0/1135871196 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f248406e930 msgr2=0x7f24840cafd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:50:11.851 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:11.848+0000 7f2483e00640 1 --2- 172.21.15.32:0/1135871196 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f248406e930 0x7f24840cafd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:11.851 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:11.848+0000 7f2483e00640 1 -- 172.21.15.32:0/1135871196 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f2484073e60 msgr2=0x7f24841440c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:50:11.851 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:11.848+0000 7f2483e00640 1 --2- 172.21.15.32:0/1135871196 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f2484073e60 0x7f24841440c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:11.851 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:11.848+0000 7f2483e00640 1 -- 172.21.15.32:0/1135871196 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f24840c9f30 con 0x7f24840747b0 2024-08-03T20:50:11.852 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:11.849+0000 7f2483e00640 1 --2- 172.21.15.32:0/1135871196 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f24840747b0 0x7f24840c8150 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7f247c00c730 tx=0x7f247c00cb30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:50:11.852 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:11.849+0000 7f247b400640 1 -- 172.21.15.32:0/1135871196 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f247c003b70 con 0x7f24840747b0 2024-08-03T20:50:11.852 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:11.849+0000 7f247b400640 1 -- 172.21.15.32:0/1135871196 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f247c003d10 con 0x7f24840747b0 2024-08-03T20:50:11.852 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:11.849+0000 7f247b400640 1 -- 172.21.15.32:0/1135871196 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f247c0052d0 con 0x7f24840747b0 2024-08-03T20:50:11.852 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:11.849+0000 7f248a400640 1 -- 172.21.15.32:0/1135871196 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f24840ca210 con 0x7f24840747b0 2024-08-03T20:50:11.852 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:11.849+0000 7f248a400640 1 -- 172.21.15.32:0/1135871196 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f24840ca7d0 con 0x7f24840747b0 2024-08-03T20:50:11.852 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:11.850+0000 7f248a400640 1 -- 172.21.15.32:0/1135871196 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f24840cd580 con 0x7f24840747b0 2024-08-03T20:50:11.853 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:11.851+0000 7f247b400640 1 -- 172.21.15.32:0/1135871196 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f247c005470 con 0x7f24840747b0 2024-08-03T20:50:11.854 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:11.853+0000 7f247b400640 1 --2- 172.21.15.32:0/1135871196 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f2464077010 0x7f2464079520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:50:11.854 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:11.853+0000 7f247b400640 1 -- 172.21.15.32:0/1135871196 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(46..46 src has 1..46) ==== 5795+0+0 (secure 0 0 0) 0x7f247c098870 con 0x7f24840747b0 2024-08-03T20:50:11.855 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:11.853+0000 7f2483400640 1 --2- 172.21.15.32:0/1135871196 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f2464077010 0x7f2464079520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:50:11.855 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:11.854+0000 7f2483400640 1 --2- 172.21.15.32:0/1135871196 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f2464077010 0x7f2464079520 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f2474005fd0 tx=0x7f2474005850 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:50:11.859 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:11.857+0000 7f247b400640 1 -- 172.21.15.32:0/1135871196 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7f247c061a00 con 0x7f24840747b0 2024-08-03T20:50:12.102 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:12.100+0000 7f248a400640 1 -- 172.21.15.32:0/1135871196 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f248415cf30 con 0x7f24840747b0 2024-08-03T20:50:12.102 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:12.101+0000 7f247b400640 1 -- 172.21.15.32:0/1135871196 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 42 v42) ==== 94+0+5230 (secure 0 0 0) 0x7f247c061300 con 0x7f24840747b0 2024-08-03T20:50:12.103 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-03T20:50:12.103 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":42,"btime":"2024-08-03T20:50:10:054829+0000","default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4997,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6837/2688500839","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6836","nonce":2688500839},{"type":"v1","addr":"172.21.15.118:6837","nonce":2688500839}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":30},{"gid":5266,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6835/2381075275","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6834","nonce":2381075275},{"type":"v1","addr":"172.21.15.118:6835","nonce":2381075275}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":28},{"gid":5421,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6833/1058332640","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":1058332640},{"type":"v1","addr":"172.21.15.32:6833","nonce":1058332640}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":29}],"filesystems":[{"mdsmap":{"epoch":42,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-08-03T20:49:47.064654+0000","modified":"2024-08-03T20:50:10.054820+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":46,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5628},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5628":{"gid":5628,"name":"c","rank":0,"incarnation":39,"state":"up:active","state_seq":4,"addr":"172.21.15.32:6835/1726912144","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1726912144},{"type":"v1","addr":"172.21.15.32:6835","nonce":1726912144}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5628,"qdb_cluster":[5628]},"id":3}]} 2024-08-03T20:50:12.103 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 42 2024-08-03T20:50:12.105 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:12.103+0000 7f248a400640 1 -- 172.21.15.32:0/1135871196 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f2464077010 msgr2=0x7f2464079520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:50:12.105 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:12.103+0000 7f248a400640 1 --2- 172.21.15.32:0/1135871196 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f2464077010 0x7f2464079520 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f2474005fd0 tx=0x7f2474005850 comp rx=0 tx=0).stop 2024-08-03T20:50:12.105 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:12.103+0000 7f248a400640 1 -- 172.21.15.32:0/1135871196 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f24840747b0 msgr2=0x7f24840c8150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:50:12.105 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:12.103+0000 7f248a400640 1 --2- 172.21.15.32:0/1135871196 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f24840747b0 0x7f24840c8150 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7f247c00c730 tx=0x7f247c00cb30 comp rx=0 tx=0).stop 2024-08-03T20:50:12.105 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:12.103+0000 7f248a400640 1 -- 172.21.15.32:0/1135871196 shutdown_connections 2024-08-03T20:50:12.105 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:12.103+0000 7f248a400640 1 --2- 172.21.15.32:0/1135871196 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f2464077010 0x7f2464079520 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:12.105 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:12.103+0000 7f248a400640 1 --2- 172.21.15.32:0/1135871196 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f2484073e60 0x7f24841440c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:12.105 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:12.103+0000 7f248a400640 1 --2- 172.21.15.32:0/1135871196 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f24840747b0 0x7f24840c8150 unknown :-1 s=CLOSED pgs=505 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:12.105 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:12.103+0000 7f248a400640 1 --2- 172.21.15.32:0/1135871196 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f248406e930 0x7f24840cafd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:12.106 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:12.103+0000 7f248a400640 1 -- 172.21.15.32:0/1135871196 >> 172.21.15.32:0/1135871196 conn(0x7f24840b1bd0 msgr2=0x7f248415de30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:50:12.106 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:12.103+0000 7f248a400640 1 -- 172.21.15.32:0/1135871196 shutdown_connections 2024-08-03T20:50:12.106 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:12.104+0000 7f248a400640 1 -- 172.21.15.32:0/1135871196 wait complete. 2024-08-03T20:50:12.129 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-08-03T20:50:12.356 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:12.354+0000 7f359a600640 1 -- 172.21.15.32:0/653752663 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3594073e60 msgr2=0x7f3594074270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:50:12.356 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:12.354+0000 7f359a600640 1 --2- 172.21.15.32:0/653752663 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3594073e60 0x7f3594074270 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7f3594071700 tx=0x7f357c02a560 comp rx=0 tx=0).stop 2024-08-03T20:50:12.356 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:12.354+0000 7f359a600640 1 -- 172.21.15.32:0/653752663 shutdown_connections 2024-08-03T20:50:12.357 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:12.354+0000 7f359a600640 1 --2- 172.21.15.32:0/653752663 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3594073e60 0x7f3594074270 unknown :-1 s=CLOSED pgs=506 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:12.357 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:12.354+0000 7f359a600640 1 --2- 172.21.15.32:0/653752663 >> v2:172.21.15.118:3300/0 conn(0x7f359406e930 0x7f3594144f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:12.357 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:12.354+0000 7f359a600640 1 -- 172.21.15.32:0/653752663 >> 172.21.15.32:0/653752663 conn(0x7f35940b1bd0 msgr2=0x7f35940b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:50:12.357 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:12.355+0000 7f359a600640 1 -- 172.21.15.32:0/653752663 shutdown_connections 2024-08-03T20:50:12.357 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:12.355+0000 7f359a600640 1 -- 172.21.15.32:0/653752663 wait complete. 2024-08-03T20:50:12.358 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:12.357+0000 7f359a600640 1 Processor -- start 2024-08-03T20:50:12.358 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:12.357+0000 7f359a600640 1 -- start start 2024-08-03T20:50:12.361 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:12.358+0000 7f359a600640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f359406e930 0x7f35941f6120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:50:12.361 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:12.358+0000 7f359a600640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f3594073e60 0x7f359413da60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:50:12.361 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:12.358+0000 7f359a600640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f35940747b0 0x7f35941436b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:50:12.361 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:12.358+0000 7f359a600640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f35940cf9c0 con 0x7f359406e930 2024-08-03T20:50:12.362 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:12.359+0000 7f359a600640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f35940cf840 con 0x7f35940747b0 2024-08-03T20:50:12.362 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:12.359+0000 7f359a600640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7f35940ccff0 con 0x7f3594073e60 2024-08-03T20:50:12.362 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:12.359+0000 7f3593400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f359406e930 0x7f35941f6120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:50:12.362 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:12.359+0000 7f3593400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f359406e930 0x7f35941f6120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:39960/0 (socket says 172.21.15.32:39960) 2024-08-03T20:50:12.362 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:12.359+0000 7f3593400640 1 -- 172.21.15.32:0/3831002096 learned_addr learned my addr 172.21.15.32:0/3831002096 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:50:12.362 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:12.359+0000 7f358be00640 1 --2- 172.21.15.32:0/3831002096 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f3594073e60 0x7f359413da60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:50:12.362 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:12.359+0000 7f3593e00640 1 --2- 172.21.15.32:0/3831002096 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f35940747b0 0x7f35941436b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:50:12.362 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:12.359+0000 7f3593400640 1 -- 172.21.15.32:0/3831002096 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f3594073e60 msgr2=0x7f359413da60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:50:12.362 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:12.359+0000 7f3593400640 1 --2- 172.21.15.32:0/3831002096 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f3594073e60 0x7f359413da60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:12.362 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:12.359+0000 7f3593400640 1 -- 172.21.15.32:0/3831002096 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f35940747b0 msgr2=0x7f35941436b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:50:12.362 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:12.359+0000 7f3593400640 1 --2- 172.21.15.32:0/3831002096 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f35940747b0 0x7f35941436b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:12.362 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:12.359+0000 7f3593400640 1 -- 172.21.15.32:0/3831002096 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f35940b4f50 con 0x7f359406e930 2024-08-03T20:50:12.362 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:12.359+0000 7f358be00640 1 --2- 172.21.15.32:0/3831002096 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f3594073e60 0x7f359413da60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-03T20:50:12.363 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:12.359+0000 7f3593e00640 1 --2- 172.21.15.32:0/3831002096 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f35940747b0 0x7f35941436b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-03T20:50:12.363 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:12.360+0000 7f3593400640 1 --2- 172.21.15.32:0/3831002096 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f359406e930 0x7f35941f6120 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7f358400e7c0 tx=0x7f358400ec80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:50:12.363 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:12.360+0000 7f358b400640 1 -- 172.21.15.32:0/3831002096 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f358400ccc0 con 0x7f359406e930 2024-08-03T20:50:12.363 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:12.360+0000 7f358b400640 1 -- 172.21.15.32:0/3831002096 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f3584010430 con 0x7f359406e930 2024-08-03T20:50:12.363 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:12.360+0000 7f359a600640 1 -- 172.21.15.32:0/3831002096 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f35940b3070 con 0x7f359406e930 2024-08-03T20:50:12.363 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:12.360+0000 7f358b400640 1 -- 172.21.15.32:0/3831002096 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3584010860 con 0x7f359406e930 2024-08-03T20:50:12.363 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:12.360+0000 7f359a600640 1 -- 172.21.15.32:0/3831002096 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f35940b3630 con 0x7f359406e930 2024-08-03T20:50:12.363 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:12.361+0000 7f359a600640 1 -- 172.21.15.32:0/3831002096 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f35940ce3b0 con 0x7f359406e930 2024-08-03T20:50:12.364 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:12.362+0000 7f358b400640 1 -- 172.21.15.32:0/3831002096 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f3584010a00 con 0x7f359406e930 2024-08-03T20:50:12.365 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:12.363+0000 7f358b400640 1 --2- 172.21.15.32:0/3831002096 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f3570077040 0x7f3570079550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:50:12.365 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:12.363+0000 7f358b400640 1 -- 172.21.15.32:0/3831002096 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(46..46 src has 1..46) ==== 5795+0+0 (secure 0 0 0) 0x7f3584099320 con 0x7f359406e930 2024-08-03T20:50:12.365 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:12.364+0000 7f358be00640 1 --2- 172.21.15.32:0/3831002096 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f3570077040 0x7f3570079550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:50:12.366 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:12.365+0000 7f358be00640 1 --2- 172.21.15.32:0/3831002096 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f3570077040 0x7f3570079550 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f357c033000 tx=0x7f357c004730 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:50:12.368 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:12.367+0000 7f358b400640 1 -- 172.21.15.32:0/3831002096 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7f35940ce3b0 con 0x7f359406e930 2024-08-03T20:50:12.615 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:12.614+0000 7f359a600640 1 -- 172.21.15.32:0/3831002096 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f35940cd580 con 0x7f359406e930 2024-08-03T20:50:12.616 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:12.614+0000 7f358b400640 1 -- 172.21.15.32:0/3831002096 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 42 v42) ==== 94+0+5230 (secure 0 0 0) 0x7f35840624b0 con 0x7f359406e930 2024-08-03T20:50:12.616 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-03T20:50:12.616 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":42,"btime":"2024-08-03T20:50:10:054829+0000","default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4997,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6837/2688500839","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6836","nonce":2688500839},{"type":"v1","addr":"172.21.15.118:6837","nonce":2688500839}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":30},{"gid":5266,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6835/2381075275","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6834","nonce":2381075275},{"type":"v1","addr":"172.21.15.118:6835","nonce":2381075275}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":28},{"gid":5421,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6833/1058332640","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":1058332640},{"type":"v1","addr":"172.21.15.32:6833","nonce":1058332640}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":29}],"filesystems":[{"mdsmap":{"epoch":42,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-08-03T20:49:47.064654+0000","modified":"2024-08-03T20:50:10.054820+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":46,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5628},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5628":{"gid":5628,"name":"c","rank":0,"incarnation":39,"state":"up:active","state_seq":4,"addr":"172.21.15.32:6835/1726912144","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1726912144},{"type":"v1","addr":"172.21.15.32:6835","nonce":1726912144}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5628,"qdb_cluster":[5628]},"id":3}]} 2024-08-03T20:50:12.616 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 42 2024-08-03T20:50:12.619 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:12.617+0000 7f359a600640 1 -- 172.21.15.32:0/3831002096 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f3570077040 msgr2=0x7f3570079550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:50:12.619 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:12.617+0000 7f359a600640 1 --2- 172.21.15.32:0/3831002096 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f3570077040 0x7f3570079550 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f357c033000 tx=0x7f357c004730 comp rx=0 tx=0).stop 2024-08-03T20:50:12.619 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:12.617+0000 7f359a600640 1 -- 172.21.15.32:0/3831002096 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f359406e930 msgr2=0x7f35941f6120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:50:12.619 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:12.617+0000 7f359a600640 1 --2- 172.21.15.32:0/3831002096 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f359406e930 0x7f35941f6120 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7f358400e7c0 tx=0x7f358400ec80 comp rx=0 tx=0).stop 2024-08-03T20:50:12.619 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:12.617+0000 7f359a600640 1 -- 172.21.15.32:0/3831002096 shutdown_connections 2024-08-03T20:50:12.619 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:12.617+0000 7f359a600640 1 --2- 172.21.15.32:0/3831002096 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f3570077040 0x7f3570079550 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:12.619 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:12.617+0000 7f359a600640 1 --2- 172.21.15.32:0/3831002096 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f35940747b0 0x7f35941436b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:12.619 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:12.617+0000 7f359a600640 1 --2- 172.21.15.32:0/3831002096 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f3594073e60 0x7f359413da60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:12.619 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:12.617+0000 7f359a600640 1 --2- 172.21.15.32:0/3831002096 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f359406e930 0x7f35941f6120 unknown :-1 s=CLOSED pgs=507 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:12.619 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:12.617+0000 7f359a600640 1 -- 172.21.15.32:0/3831002096 >> 172.21.15.32:0/3831002096 conn(0x7f35940b1bd0 msgr2=0x7f35940b0da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:50:12.619 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:12.617+0000 7f359a600640 1 -- 172.21.15.32:0/3831002096 shutdown_connections 2024-08-03T20:50:12.620 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:12.617+0000 7f359a600640 1 -- 172.21.15.32:0/3831002096 wait complete. 2024-08-03T20:50:12.641 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-08-03T20:50:12.868 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:12.866+0000 7f16f7a00640 1 -- 172.21.15.32:0/1980199147 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f16f01523f0 msgr2=0x7f16f00b4ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:50:12.868 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:12.866+0000 7f16f7a00640 1 --2- 172.21.15.32:0/1980199147 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f16f01523f0 0x7f16f00b4ce0 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7f16e8005fd0 tx=0x7f16e802a8b0 comp rx=0 tx=0).stop 2024-08-03T20:50:12.869 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:12.866+0000 7f16f7a00640 1 -- 172.21.15.32:0/1980199147 shutdown_connections 2024-08-03T20:50:12.869 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:12.866+0000 7f16f7a00640 1 --2- 172.21.15.32:0/1980199147 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f16f01523f0 0x7f16f00b4ce0 unknown :-1 s=CLOSED pgs=509 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:12.869 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:12.866+0000 7f16f7a00640 1 -- 172.21.15.32:0/1980199147 >> 172.21.15.32:0/1980199147 conn(0x7f16f00b1bd0 msgr2=0x7f16f00b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:50:12.869 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:12.866+0000 7f16f7a00640 1 -- 172.21.15.32:0/1980199147 shutdown_connections 2024-08-03T20:50:12.869 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:12.867+0000 7f16f7a00640 1 -- 172.21.15.32:0/1980199147 wait complete. 2024-08-03T20:50:12.870 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:12.869+0000 7f16f7a00640 1 Processor -- start 2024-08-03T20:50:12.870 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:12.869+0000 7f16f7a00640 1 -- start start 2024-08-03T20:50:12.873 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:12.870+0000 7f16f7a00640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f16f00b38b0 0x7f16f00741a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:50:12.873 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:12.870+0000 7f16f7a00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f16f01523f0 0x7f16f01e5660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:50:12.873 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:12.870+0000 7f16f7a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f16f01e5ba0 0x7f16f00ad8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:50:12.874 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:12.870+0000 7f16f7a00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f16f00cec60 con 0x7f16f01e5ba0 2024-08-03T20:50:12.874 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:12.870+0000 7f16f7a00640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f16f00ceae0 con 0x7f16f01523f0 2024-08-03T20:50:12.874 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:12.870+0000 7f16f7a00640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7f16f00cede0 con 0x7f16f00b38b0 2024-08-03T20:50:12.874 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:12.871+0000 7f16f5a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f16f01e5ba0 0x7f16f00ad8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:50:12.874 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:12.871+0000 7f16f5a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f16f01e5ba0 0x7f16f00ad8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:39992/0 (socket says 172.21.15.32:39992) 2024-08-03T20:50:12.874 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:12.871+0000 7f16f5a00640 1 -- 172.21.15.32:0/2507415708 learned_addr learned my addr 172.21.15.32:0/2507415708 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:50:12.874 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:12.871+0000 7f16efe00640 1 --2- 172.21.15.32:0/2507415708 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f16f01523f0 0x7f16f01e5660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:50:12.874 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:12.871+0000 7f16f5000640 1 --2- 172.21.15.32:0/2507415708 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f16f00b38b0 0x7f16f00741a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:50:12.874 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:12.871+0000 7f16f5a00640 1 -- 172.21.15.32:0/2507415708 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f16f00b38b0 msgr2=0x7f16f00741a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:50:12.874 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:12.871+0000 7f16f5a00640 1 --2- 172.21.15.32:0/2507415708 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f16f00b38b0 0x7f16f00741a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:12.875 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:12.871+0000 7f16f5a00640 1 -- 172.21.15.32:0/2507415708 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f16f01523f0 msgr2=0x7f16f01e5660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:50:12.875 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:12.871+0000 7f16f5a00640 1 --2- 172.21.15.32:0/2507415708 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f16f01523f0 0x7f16f01e5660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:12.875 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:12.871+0000 7f16f5a00640 1 -- 172.21.15.32:0/2507415708 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f16f00746e0 con 0x7f16f01e5ba0 2024-08-03T20:50:12.875 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:12.871+0000 7f16efe00640 1 --2- 172.21.15.32:0/2507415708 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f16f01523f0 0x7f16f01e5660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-03T20:50:12.875 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:12.872+0000 7f16f5a00640 1 --2- 172.21.15.32:0/2507415708 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f16f01e5ba0 0x7f16f00ad8c0 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7f16e802aea0 tx=0x7f16e8033f70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:50:12.875 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:12.872+0000 7f16ecc00640 1 -- 172.21.15.32:0/2507415708 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16e8031990 con 0x7f16f01e5ba0 2024-08-03T20:50:12.875 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:12.872+0000 7f16ecc00640 1 -- 172.21.15.32:0/2507415708 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f16e8031c20 con 0x7f16f01e5ba0 2024-08-03T20:50:12.875 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:12.872+0000 7f16ecc00640 1 -- 172.21.15.32:0/2507415708 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16e8033070 con 0x7f16f01e5ba0 2024-08-03T20:50:12.875 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:12.872+0000 7f16f7a00640 1 -- 172.21.15.32:0/2507415708 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f16f00735f0 con 0x7f16f01e5ba0 2024-08-03T20:50:12.875 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:12.872+0000 7f16f7a00640 1 -- 172.21.15.32:0/2507415708 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f16f0072d50 con 0x7f16f01e5ba0 2024-08-03T20:50:12.876 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:12.873+0000 7f16f7a00640 1 -- 172.21.15.32:0/2507415708 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f16f00b4ce0 con 0x7f16f01e5ba0 2024-08-03T20:50:12.876 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:12.874+0000 7f16ecc00640 1 -- 172.21.15.32:0/2507415708 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f16e8033210 con 0x7f16f01e5ba0 2024-08-03T20:50:12.877 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:12.876+0000 7f16ecc00640 1 --2- 172.21.15.32:0/2507415708 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f16d0077010 0x7f16d0079520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:50:12.877 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:12.876+0000 7f16ecc00640 1 -- 172.21.15.32:0/2507415708 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(46..46 src has 1..46) ==== 5795+0+0 (secure 0 0 0) 0x7f16e80b9f80 con 0x7f16f01e5ba0 2024-08-03T20:50:12.878 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:12.877+0000 7f16f5000640 1 --2- 172.21.15.32:0/2507415708 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f16d0077010 0x7f16d0079520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:50:12.878 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:12.877+0000 7f16f5000640 1 --2- 172.21.15.32:0/2507415708 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f16d0077010 0x7f16d0079520 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f16e4001610 tx=0x7f16e4009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:50:12.882 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:12.881+0000 7f16ecc00640 1 -- 172.21.15.32:0/2507415708 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7f16e803f030 con 0x7f16f01e5ba0 2024-08-03T20:50:13.088 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:13.087+0000 7f16f7a00640 1 -- 172.21.15.32:0/2507415708 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f16f01f4ea0 con 0x7f16f01e5ba0 2024-08-03T20:50:13.089 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:13.088+0000 7f16ecc00640 1 -- 172.21.15.32:0/2507415708 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v46) ==== 74+0+16680 (secure 0 0 0) 0x7f16e8083080 con 0x7f16f01e5ba0 2024-08-03T20:50:13.090 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-03T20:50:13.090 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":46,"fsid":"40ab141a-01d7-49b9-ae3c-2edec9876e95","created":"2024-08-03T20:47:14.957362+0000","modified":"2024-08-03T20:50:05.030282+0000","last_up_change":"2024-08-03T20:47:28.874451+0000","last_in_change":"2024-08-03T20:47:20.079442+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-08-03T20:47:28.261300+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_metadata","create_time":"2024-08-03T20:49:45.020976+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"44","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_data","create_time":"2024-08-03T20:49:46.012432+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"45","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5a83e759-9035-438b-889a-0a6af1f0a0d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6816","nonce":3066156116},{"type":"v1","addr":"172.21.15.32:6817","nonce":3066156116}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6818","nonce":3066156116},{"type":"v1","addr":"172.21.15.32:6819","nonce":3066156116}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6822","nonce":3066156116},{"type":"v1","addr":"172.21.15.32:6823","nonce":3066156116}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6820","nonce":3066156116},{"type":"v1","addr":"172.21.15.32:6821","nonce":3066156116}]},"public_addr":"172.21.15.32:6817/3066156116","cluster_addr":"172.21.15.32:6819/3066156116","heartbeat_back_addr":"172.21.15.32:6823/3066156116","heartbeat_front_addr":"172.21.15.32:6821/3066156116","state":["exists","up"]},{"osd":1,"uuid":"d7c5c210-da4f-49f9-b9e7-1dfb02daa4bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6824","nonce":1551241759},{"type":"v1","addr":"172.21.15.32:6825","nonce":1551241759}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6826","nonce":1551241759},{"type":"v1","addr":"172.21.15.32:6827","nonce":1551241759}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6830","nonce":1551241759},{"type":"v1","addr":"172.21.15.32:6831","nonce":1551241759}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6828","nonce":1551241759},{"type":"v1","addr":"172.21.15.32:6829","nonce":1551241759}]},"public_addr":"172.21.15.32:6825/1551241759","cluster_addr":"172.21.15.32:6827/1551241759","heartbeat_back_addr":"172.21.15.32:6831/1551241759","heartbeat_front_addr":"172.21.15.32:6829/1551241759","state":["exists","up"]},{"osd":2,"uuid":"57f2e6de-a201-45c2-8981-8e56a4875ebd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6808","nonce":3806708944},{"type":"v1","addr":"172.21.15.32:6809","nonce":3806708944}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6810","nonce":3806708944},{"type":"v1","addr":"172.21.15.32:6811","nonce":3806708944}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6814","nonce":3806708944},{"type":"v1","addr":"172.21.15.32:6815","nonce":3806708944}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6812","nonce":3806708944},{"type":"v1","addr":"172.21.15.32:6813","nonce":3806708944}]},"public_addr":"172.21.15.32:6809/3806708944","cluster_addr":"172.21.15.32:6811/3806708944","heartbeat_back_addr":"172.21.15.32:6815/3806708944","heartbeat_front_addr":"172.21.15.32:6813/3806708944","state":["exists","up"]},{"osd":3,"uuid":"6439a280-da45-4148-8130-f16323935ff3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6800","nonce":9088303},{"type":"v1","addr":"172.21.15.32:6801","nonce":9088303}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6802","nonce":9088303},{"type":"v1","addr":"172.21.15.32:6803","nonce":9088303}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6806","nonce":9088303},{"type":"v1","addr":"172.21.15.32:6807","nonce":9088303}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6804","nonce":9088303},{"type":"v1","addr":"172.21.15.32:6805","nonce":9088303}]},"public_addr":"172.21.15.32:6801/9088303","cluster_addr":"172.21.15.32:6803/9088303","heartbeat_back_addr":"172.21.15.32:6807/9088303","heartbeat_front_addr":"172.21.15.32:6805/9088303","state":["exists","up"]},{"osd":4,"uuid":"4f902eff-1e18-418d-82c9-2151c69524fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6802","nonce":864752275},{"type":"v1","addr":"172.21.15.118:6803","nonce":864752275}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6804","nonce":864752275},{"type":"v1","addr":"172.21.15.118:6805","nonce":864752275}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6808","nonce":864752275},{"type":"v1","addr":"172.21.15.118:6809","nonce":864752275}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6806","nonce":864752275},{"type":"v1","addr":"172.21.15.118:6807","nonce":864752275}]},"public_addr":"172.21.15.118:6803/864752275","cluster_addr":"172.21.15.118:6805/864752275","heartbeat_back_addr":"172.21.15.118:6809/864752275","heartbeat_front_addr":"172.21.15.118:6807/864752275","state":["exists","up"]},{"osd":5,"uuid":"bfd9feac-dd06-458c-91b9-af7f175f612c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6810","nonce":2724891770},{"type":"v1","addr":"172.21.15.118:6811","nonce":2724891770}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6812","nonce":2724891770},{"type":"v1","addr":"172.21.15.118:6813","nonce":2724891770}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6816","nonce":2724891770},{"type":"v1","addr":"172.21.15.118:6817","nonce":2724891770}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6814","nonce":2724891770},{"type":"v1","addr":"172.21.15.118:6815","nonce":2724891770}]},"public_addr":"172.21.15.118:6811/2724891770","cluster_addr":"172.21.15.118:6813/2724891770","heartbeat_back_addr":"172.21.15.118:6817/2724891770","heartbeat_front_addr":"172.21.15.118:6815/2724891770","state":["exists","up"]},{"osd":6,"uuid":"5ed5a279-9d8e-416d-bbc7-7edd3e6d5705","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6825","nonce":801947386},{"type":"v1","addr":"172.21.15.118:6827","nonce":801947386}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6828","nonce":801947386},{"type":"v1","addr":"172.21.15.118:6829","nonce":801947386}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6832","nonce":801947386},{"type":"v1","addr":"172.21.15.118:6833","nonce":801947386}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6830","nonce":801947386},{"type":"v1","addr":"172.21.15.118:6831","nonce":801947386}]},"public_addr":"172.21.15.118:6827/801947386","cluster_addr":"172.21.15.118:6829/801947386","heartbeat_back_addr":"172.21.15.118:6833/801947386","heartbeat_front_addr":"172.21.15.118:6831/801947386","state":["exists","up"]},{"osd":7,"uuid":"f2086c47-875e-4451-a436-2008d1ea16af","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6818","nonce":653000048},{"type":"v1","addr":"172.21.15.118:6819","nonce":653000048}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6820","nonce":653000048},{"type":"v1","addr":"172.21.15.118:6821","nonce":653000048}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6824","nonce":653000048},{"type":"v1","addr":"172.21.15.118:6826","nonce":653000048}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6822","nonce":653000048},{"type":"v1","addr":"172.21.15.118:6823","nonce":653000048}]},"public_addr":"172.21.15.118:6819/653000048","cluster_addr":"172.21.15.118:6821/653000048","heartbeat_back_addr":"172.21.15.118:6826/653000048","heartbeat_front_addr":"172.21.15.118:6823/653000048","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-03T20:47:25.491994+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-03T20:47:25.587358+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-03T20:47:25.454175+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-03T20:47:25.377734+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-03T20:47:25.412351+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-03T20:47:25.499815+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-03T20:47:25.575044+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-03T20:47:25.552383+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.32:6835/678469885":"2024-08-04T20:50:05.030253+0000","172.21.15.32:6834/678469885":"2024-08-04T20:50:05.030253+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-08-03T20:50:13.092 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:13.090+0000 7f16f7a00640 1 -- 172.21.15.32:0/2507415708 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f16d0077010 msgr2=0x7f16d0079520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:50:13.092 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:13.090+0000 7f16f7a00640 1 --2- 172.21.15.32:0/2507415708 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f16d0077010 0x7f16d0079520 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f16e4001610 tx=0x7f16e4009040 comp rx=0 tx=0).stop 2024-08-03T20:50:13.092 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:13.090+0000 7f16f7a00640 1 -- 172.21.15.32:0/2507415708 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f16f01e5ba0 msgr2=0x7f16f00ad8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:50:13.092 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:13.090+0000 7f16f7a00640 1 --2- 172.21.15.32:0/2507415708 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f16f01e5ba0 0x7f16f00ad8c0 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7f16e802aea0 tx=0x7f16e8033f70 comp rx=0 tx=0).stop 2024-08-03T20:50:13.093 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:13.090+0000 7f16f7a00640 1 -- 172.21.15.32:0/2507415708 shutdown_connections 2024-08-03T20:50:13.093 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:13.090+0000 7f16f7a00640 1 --2- 172.21.15.32:0/2507415708 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f16d0077010 0x7f16d0079520 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:13.093 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:13.090+0000 7f16f7a00640 1 --2- 172.21.15.32:0/2507415708 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f16f01523f0 0x7f16f01e5660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:13.093 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:13.090+0000 7f16f7a00640 1 --2- 172.21.15.32:0/2507415708 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f16f01e5ba0 0x7f16f00ad8c0 unknown :-1 s=CLOSED pgs=510 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:13.093 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:13.090+0000 7f16f7a00640 1 --2- 172.21.15.32:0/2507415708 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f16f00b38b0 0x7f16f00741a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:13.093 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:13.090+0000 7f16f7a00640 1 -- 172.21.15.32:0/2507415708 >> 172.21.15.32:0/2507415708 conn(0x7f16f00b1bd0 msgr2=0x7f16f0151c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:50:13.093 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:13.091+0000 7f16f7a00640 1 -- 172.21.15.32:0/2507415708 shutdown_connections 2024-08-03T20:50:13.093 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:13.091+0000 7f16f7a00640 1 -- 172.21.15.32:0/2507415708 wait complete. 2024-08-03T20:50:13.114 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-08-03T20:50:13.342 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:13.340+0000 7fd526200640 1 -- 172.21.15.32:0/3265918101 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd520073e60 msgr2=0x7fd520074270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:50:13.342 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:13.340+0000 7fd526200640 1 --2- 172.21.15.32:0/3265918101 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd520073e60 0x7fd520074270 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7fd508005fd0 tx=0x7fd50802a540 comp rx=0 tx=0).stop 2024-08-03T20:50:13.342 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:13.340+0000 7fd526200640 1 -- 172.21.15.32:0/3265918101 shutdown_connections 2024-08-03T20:50:13.342 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:13.340+0000 7fd526200640 1 --2- 172.21.15.32:0/3265918101 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd520073e60 0x7fd520074270 unknown :-1 s=CLOSED pgs=511 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:13.342 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:13.340+0000 7fd526200640 1 --2- 172.21.15.32:0/3265918101 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fd52006e930 0x7fd520144f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:13.342 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:13.340+0000 7fd526200640 1 -- 172.21.15.32:0/3265918101 >> 172.21.15.32:0/3265918101 conn(0x7fd5200b1bd0 msgr2=0x7fd5200b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:50:13.343 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:13.340+0000 7fd526200640 1 -- 172.21.15.32:0/3265918101 shutdown_connections 2024-08-03T20:50:13.343 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:13.340+0000 7fd526200640 1 -- 172.21.15.32:0/3265918101 wait complete. 2024-08-03T20:50:13.344 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:13.343+0000 7fd526200640 1 Processor -- start 2024-08-03T20:50:13.344 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:13.343+0000 7fd526200640 1 -- start start 2024-08-03T20:50:13.346 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:13.344+0000 7fd526200640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fd52006e930 0x7fd5201f62e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:50:13.346 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:13.344+0000 7fd526200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd520073e60 0x7fd5201f4570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:50:13.346 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:13.344+0000 7fd526200640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fd5200747b0 0x7fd5201f4ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:50:13.346 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:13.344+0000 7fd526200640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fd5200cf9c0 con 0x7fd520073e60 2024-08-03T20:50:13.346 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:13.344+0000 7fd526200640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7fd5200cf840 con 0x7fd52006e930 2024-08-03T20:50:13.346 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:13.344+0000 7fd526200640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7fd5200ccff0 con 0x7fd5200747b0 2024-08-03T20:50:13.346 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:13.344+0000 7fd517e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd520073e60 0x7fd5201f4570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:50:13.346 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:13.345+0000 7fd517e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd520073e60 0x7fd5201f4570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:40016/0 (socket says 172.21.15.32:40016) 2024-08-03T20:50:13.346 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:13.345+0000 7fd51f400640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fd52006e930 0x7fd5201f62e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:50:13.347 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:13.345+0000 7fd517e00640 1 -- 172.21.15.32:0/3558460999 learned_addr learned my addr 172.21.15.32:0/3558460999 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:50:13.347 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:13.345+0000 7fd51fe00640 1 --2- 172.21.15.32:0/3558460999 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fd5200747b0 0x7fd5201f4ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:50:13.348 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:13.345+0000 7fd517e00640 1 -- 172.21.15.32:0/3558460999 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fd5200747b0 msgr2=0x7fd5201f4ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:50:13.348 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:13.345+0000 7fd517e00640 1 --2- 172.21.15.32:0/3558460999 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fd5200747b0 0x7fd5201f4ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:13.348 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:13.345+0000 7fd517e00640 1 -- 172.21.15.32:0/3558460999 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fd52006e930 msgr2=0x7fd5201f62e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:50:13.348 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:13.345+0000 7fd517e00640 1 --2- 172.21.15.32:0/3558460999 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fd52006e930 0x7fd5201f62e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:13.348 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:13.345+0000 7fd517e00640 1 -- 172.21.15.32:0/3558460999 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd5200a9900 con 0x7fd520073e60 2024-08-03T20:50:13.348 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:13.345+0000 7fd517e00640 1 --2- 172.21.15.32:0/3558460999 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd520073e60 0x7fd5201f4570 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7fd508006010 tx=0x7fd5080038a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:50:13.348 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:13.345+0000 7fd517400640 1 -- 172.21.15.32:0/3558460999 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd50803e030 con 0x7fd520073e60 2024-08-03T20:50:13.348 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:13.346+0000 7fd517400640 1 -- 172.21.15.32:0/3558460999 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fd508003d90 con 0x7fd520073e60 2024-08-03T20:50:13.348 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:13.346+0000 7fd526200640 1 -- 172.21.15.32:0/3558460999 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd5200a9be0 con 0x7fd520073e60 2024-08-03T20:50:13.348 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:13.346+0000 7fd517400640 1 -- 172.21.15.32:0/3558460999 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd508035040 con 0x7fd520073e60 2024-08-03T20:50:13.348 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:13.346+0000 7fd526200640 1 -- 172.21.15.32:0/3558460999 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd5200aa1a0 con 0x7fd520073e60 2024-08-03T20:50:13.349 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:13.347+0000 7fd526200640 1 -- 172.21.15.32:0/3558460999 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd5200a9670 con 0x7fd520073e60 2024-08-03T20:50:13.349 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:13.348+0000 7fd517400640 1 -- 172.21.15.32:0/3558460999 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7fd508041030 con 0x7fd520073e60 2024-08-03T20:50:13.350 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:13.349+0000 7fd517400640 1 --2- 172.21.15.32:0/3558460999 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fd4fc077010 0x7fd4fc079520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:50:13.350 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:13.349+0000 7fd517400640 1 -- 172.21.15.32:0/3558460999 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(46..46 src has 1..46) ==== 5795+0+0 (secure 0 0 0) 0x7fd5080b8fb0 con 0x7fd520073e60 2024-08-03T20:50:13.351 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:13.350+0000 7fd51f400640 1 --2- 172.21.15.32:0/3558460999 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fd4fc077010 0x7fd4fc079520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:50:13.352 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:13.350+0000 7fd51f400640 1 --2- 172.21.15.32:0/3558460999 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fd4fc077010 0x7fd4fc079520 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fd510009b30 tx=0x7fd510005f90 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:50:13.354 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:13.353+0000 7fd517400640 1 -- 172.21.15.32:0/3558460999 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7fd508081af0 con 0x7fd520073e60 2024-08-03T20:50:13.596 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:13.595+0000 7fd526200640 1 -- 172.21.15.32:0/3558460999 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7fd5200cd510 con 0x7fd520073e60 2024-08-03T20:50:13.597 INFO:tasks.ceph.mon.a.smithi032.stderr:2024-08-03T20:50:13.596+0000 7f4e6d600640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-08-03T20:50:13.599 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:13.597+0000 7fd517400640 1 -- 172.21.15.32:0/3558460999 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v43) ==== 161+0+0 (secure 0 0 0) 0x7fd50808a380 con 0x7fd520073e60 2024-08-03T20:50:13.599 INFO:teuthology.orchestra.run.smithi032.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-08-03T20:50:13.602 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:13.601+0000 7fd526200640 1 -- 172.21.15.32:0/3558460999 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fd4fc077010 msgr2=0x7fd4fc079520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:50:13.603 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:13.601+0000 7fd526200640 1 --2- 172.21.15.32:0/3558460999 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fd4fc077010 0x7fd4fc079520 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fd510009b30 tx=0x7fd510005f90 comp rx=0 tx=0).stop 2024-08-03T20:50:13.603 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:13.601+0000 7fd526200640 1 -- 172.21.15.32:0/3558460999 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd520073e60 msgr2=0x7fd5201f4570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:50:13.603 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:13.601+0000 7fd526200640 1 --2- 172.21.15.32:0/3558460999 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd520073e60 0x7fd5201f4570 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7fd508006010 tx=0x7fd5080038a0 comp rx=0 tx=0).stop 2024-08-03T20:50:13.603 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:13.601+0000 7fd526200640 1 -- 172.21.15.32:0/3558460999 shutdown_connections 2024-08-03T20:50:13.603 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:13.601+0000 7fd526200640 1 --2- 172.21.15.32:0/3558460999 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fd4fc077010 0x7fd4fc079520 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:13.603 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:13.601+0000 7fd526200640 1 --2- 172.21.15.32:0/3558460999 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fd5200747b0 0x7fd5201f4ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:13.603 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:13.601+0000 7fd526200640 1 --2- 172.21.15.32:0/3558460999 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd520073e60 0x7fd5201f4570 unknown :-1 s=CLOSED pgs=512 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:13.603 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:13.601+0000 7fd526200640 1 --2- 172.21.15.32:0/3558460999 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fd52006e930 0x7fd5201f62e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:13.603 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:13.601+0000 7fd526200640 1 -- 172.21.15.32:0/3558460999 >> 172.21.15.32:0/3558460999 conn(0x7fd5200b1bd0 msgr2=0x7fd52013fdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:50:13.604 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:13.601+0000 7fd526200640 1 -- 172.21.15.32:0/3558460999 shutdown_connections 2024-08-03T20:50:13.604 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:13.602+0000 7fd526200640 1 -- 172.21.15.32:0/3558460999 wait complete. 2024-08-03T20:50:13.607 INFO:tasks.ceph.mds.c.smithi032.stderr: -98> 2024-08-03T20:50:11.041+0000 7f1a5f400640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-08-03T20:50:13.627 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-08-03T20:50:13.722 INFO:tasks.ceph.mds.c.smithi032.stdout:starting mds.c at 2024-08-03T20:50:13.863 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:13.861+0000 7f2278c00640 1 -- 172.21.15.32:0/3327331358 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2258005270 msgr2=0x7f225800b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:50:13.863 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:13.861+0000 7f2278c00640 1 --2- 172.21.15.32:0/3327331358 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2258005270 0x7f225800b9c0 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7f227415d850 tx=0x7f2258005830 comp rx=0 tx=0).stop 2024-08-03T20:50:13.863 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:13.861+0000 7f2273e00640 1 -- 172.21.15.32:0/3327331358 reap_dead start 2024-08-03T20:50:13.863 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:13.861+0000 7f2278c00640 1 -- 172.21.15.32:0/3327331358 shutdown_connections 2024-08-03T20:50:13.863 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:13.861+0000 7f2278c00640 1 -- 172.21.15.32:0/3327331358 >> 172.21.15.32:0/3327331358 conn(0x7f22740b2b50 msgr2=0x7f22740d0500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:50:13.863 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:13.861+0000 7f2278c00640 1 -- 172.21.15.32:0/3327331358 shutdown_connections 2024-08-03T20:50:13.864 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:13.862+0000 7f2278c00640 1 -- 172.21.15.32:0/3327331358 wait complete. 2024-08-03T20:50:13.865 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:13.864+0000 7f2278c00640 1 Processor -- start 2024-08-03T20:50:13.865 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:13.864+0000 7f2278c00640 1 -- start start 2024-08-03T20:50:13.867 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:13.865+0000 7f2278c00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2258005270 0x7f2274152500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:50:13.867 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:13.865+0000 7f2278c00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f2274152a40 0x7f227415b3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:50:13.867 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:13.865+0000 7f2278c00640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f227415b910 0x7f227414c570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:50:13.867 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:13.865+0000 7f2278c00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f22740cf110 con 0x7f2258005270 2024-08-03T20:50:13.867 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:13.865+0000 7f2278c00640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f22740cef90 con 0x7f2274152a40 2024-08-03T20:50:13.867 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:13.865+0000 7f2278c00640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7f227406f970 con 0x7f227415b910 2024-08-03T20:50:13.867 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:13.866+0000 7f2273400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2258005270 0x7f2274152500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:50:13.868 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:13.866+0000 7f2273400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2258005270 0x7f2274152500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:40050/0 (socket says 172.21.15.32:40050) 2024-08-03T20:50:13.868 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:13.866+0000 7f2273400640 1 -- 172.21.15.32:0/2647316785 learned_addr learned my addr 172.21.15.32:0/2647316785 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:50:13.868 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:13.866+0000 7f226be00640 1 --2- 172.21.15.32:0/2647316785 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f2274152a40 0x7f227415b3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:50:13.868 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:13.866+0000 7f2273e00640 1 --2- 172.21.15.32:0/2647316785 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f227415b910 0x7f227414c570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:50:13.869 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:13.866+0000 7f2273400640 1 -- 172.21.15.32:0/2647316785 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f227415b910 msgr2=0x7f227414c570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:50:13.869 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:13.866+0000 7f2273400640 1 --2- 172.21.15.32:0/2647316785 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f227415b910 0x7f227414c570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:13.869 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:13.866+0000 7f2273400640 1 -- 172.21.15.32:0/2647316785 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f2274152a40 msgr2=0x7f227415b3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:50:13.869 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:13.866+0000 7f2273400640 1 --2- 172.21.15.32:0/2647316785 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f2274152a40 0x7f227415b3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:13.869 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:13.866+0000 7f2273400640 1 -- 172.21.15.32:0/2647316785 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f22740b3880 con 0x7f2258005270 2024-08-03T20:50:13.869 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:13.866+0000 7f2273400640 1 --2- 172.21.15.32:0/2647316785 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2258005270 0x7f2274152500 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7f226400e5e0 tx=0x7f226400eaa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:50:13.869 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:13.867+0000 7f226b400640 1 -- 172.21.15.32:0/2647316785 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2264015020 con 0x7f2258005270 2024-08-03T20:50:13.869 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:13.867+0000 7f226b400640 1 -- 172.21.15.32:0/2647316785 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f226400ca00 con 0x7f2258005270 2024-08-03T20:50:13.870 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:13.867+0000 7f226b400640 1 -- 172.21.15.32:0/2647316785 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f226400ce50 con 0x7f2258005270 2024-08-03T20:50:13.870 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:13.867+0000 7f2278c00640 1 -- 172.21.15.32:0/2647316785 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f22740b3b60 con 0x7f2258005270 2024-08-03T20:50:13.870 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:13.867+0000 7f2278c00640 1 -- 172.21.15.32:0/2647316785 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f22740b4120 con 0x7f2258005270 2024-08-03T20:50:13.870 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:13.868+0000 7f2278c00640 1 -- 172.21.15.32:0/2647316785 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f227406fe60 con 0x7f2258005270 2024-08-03T20:50:13.870 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:13.869+0000 7f226b400640 1 -- 172.21.15.32:0/2647316785 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f2264018020 con 0x7f2258005270 2024-08-03T20:50:13.871 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:13.870+0000 7f226b400640 1 --2- 172.21.15.32:0/2647316785 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f2250077010 0x7f2250079520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:50:13.871 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:13.870+0000 7f226b400640 1 -- 172.21.15.32:0/2647316785 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(47..47 src has 1..47) ==== 5881+0+0 (secure 0 0 0) 0x7f2264098e40 con 0x7f2258005270 2024-08-03T20:50:13.872 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:13.870+0000 7f226be00640 1 --2- 172.21.15.32:0/2647316785 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f2250077010 0x7f2250079520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:50:13.872 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:13.871+0000 7f226be00640 1 --2- 172.21.15.32:0/2647316785 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f2250077010 0x7f2250079520 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f227415bdb0 tx=0x7f225c006d90 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:50:13.876 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:13.874+0000 7f226b400640 1 -- 172.21.15.32:0/2647316785 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7f226401f020 con 0x7f2258005270 2024-08-03T20:50:14.120 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:14.119+0000 7f2278c00640 1 -- 172.21.15.32:0/2647316785 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) -- 0x7f227415d250 con 0x7f2258005270 2024-08-03T20:50:14.601 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:14.600+0000 7f226b400640 1 -- 172.21.15.32:0/2647316785 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v44) ==== 104+0+0 (secure 0 0 0) 0x7f22640618a0 con 0x7f2258005270 2024-08-03T20:50:14.604 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:14.602+0000 7f2278c00640 1 -- 172.21.15.32:0/2647316785 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f2250077010 msgr2=0x7f2250079520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:50:14.604 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:14.602+0000 7f2278c00640 1 --2- 172.21.15.32:0/2647316785 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f2250077010 0x7f2250079520 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f227415bdb0 tx=0x7f225c006d90 comp rx=0 tx=0).stop 2024-08-03T20:50:14.604 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:14.602+0000 7f2278c00640 1 -- 172.21.15.32:0/2647316785 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2258005270 msgr2=0x7f2274152500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:50:14.604 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:14.602+0000 7f2278c00640 1 --2- 172.21.15.32:0/2647316785 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2258005270 0x7f2274152500 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7f226400e5e0 tx=0x7f226400eaa0 comp rx=0 tx=0).stop 2024-08-03T20:50:14.604 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:14.602+0000 7f2278c00640 1 -- 172.21.15.32:0/2647316785 shutdown_connections 2024-08-03T20:50:14.604 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:14.602+0000 7f2278c00640 1 --2- 172.21.15.32:0/2647316785 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f2250077010 0x7f2250079520 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:14.605 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:14.602+0000 7f2278c00640 1 --2- 172.21.15.32:0/2647316785 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f2274152a40 0x7f227415b3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:14.605 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:14.602+0000 7f2278c00640 1 --2- 172.21.15.32:0/2647316785 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f227415b910 0x7f227414c570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:14.605 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:14.602+0000 7f2278c00640 1 --2- 172.21.15.32:0/2647316785 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2258005270 0x7f2274152500 unknown :-1 s=CLOSED pgs=517 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:14.605 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:14.602+0000 7f2278c00640 1 -- 172.21.15.32:0/2647316785 >> 172.21.15.32:0/2647316785 conn(0x7f22740b2b50 msgr2=0x7f22740d0440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:50:14.605 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:14.603+0000 7f2278c00640 1 -- 172.21.15.32:0/2647316785 shutdown_connections 2024-08-03T20:50:14.605 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:14.603+0000 7f2278c00640 1 -- 172.21.15.32:0/2647316785 wait complete. 2024-08-03T20:50:14.629 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-08-03T20:50:14.863 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:14.862+0000 7fb534e00640 1 -- 172.21.15.32:0/285255546 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb5300cd5f0 msgr2=0x7fb5300d29a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:50:14.864 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:14.862+0000 7fb534e00640 1 --2- 172.21.15.32:0/285255546 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb5300cd5f0 0x7fb5300d29a0 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7fb520005fd0 tx=0x7fb52002a520 comp rx=0 tx=0).stop 2024-08-03T20:50:14.864 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:14.862+0000 7fb534e00640 1 -- 172.21.15.32:0/285255546 shutdown_connections 2024-08-03T20:50:14.864 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:14.862+0000 7fb534e00640 1 --2- 172.21.15.32:0/285255546 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fb5300ce700 0x7fb5300ceb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:14.864 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:14.862+0000 7fb534e00640 1 --2- 172.21.15.32:0/285255546 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb5300cd5f0 0x7fb5300d29a0 unknown :-1 s=CLOSED pgs=518 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:14.864 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:14.862+0000 7fb534e00640 1 -- 172.21.15.32:0/285255546 >> 172.21.15.32:0/285255546 conn(0x7fb5300d1e80 msgr2=0x7fb5300d2290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:50:14.864 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:14.862+0000 7fb534e00640 1 -- 172.21.15.32:0/285255546 shutdown_connections 2024-08-03T20:50:14.864 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:14.862+0000 7fb534e00640 1 -- 172.21.15.32:0/285255546 wait complete. 2024-08-03T20:50:14.865 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:14.864+0000 7fb534e00640 1 Processor -- start 2024-08-03T20:50:14.865 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:14.864+0000 7fb534e00640 1 -- start start 2024-08-03T20:50:14.867 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:14.866+0000 7fb534e00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb5300b7420 0x7fb530149080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:50:14.867 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:14.866+0000 7fb534e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb5300cd5f0 0x7fb5301f4290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:50:14.867 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:14.866+0000 7fb534e00640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fb5300ce700 0x7fb53015e3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:50:14.867 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:14.866+0000 7fb534e00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fb5300d09f0 con 0x7fb5300cd5f0 2024-08-03T20:50:14.867 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:14.866+0000 7fb534e00640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7fb5300d0870 con 0x7fb5300b7420 2024-08-03T20:50:14.868 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:14.866+0000 7fb534e00640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7fb5300d0b70 con 0x7fb5300ce700 2024-08-03T20:50:14.868 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:14.866+0000 7fb527e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb5300cd5f0 0x7fb5301f4290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:50:14.868 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:14.866+0000 7fb527e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb5300cd5f0 0x7fb5301f4290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:40064/0 (socket says 172.21.15.32:40064) 2024-08-03T20:50:14.868 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:14.866+0000 7fb527e00640 1 -- 172.21.15.32:0/2120951071 learned_addr learned my addr 172.21.15.32:0/2120951071 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:50:14.869 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:14.866+0000 7fb527e00640 1 -- 172.21.15.32:0/2120951071 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fb5300ce700 msgr2=0x7fb53015e3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:50:14.869 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:14.866+0000 7fb527e00640 1 --2- 172.21.15.32:0/2120951071 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fb5300ce700 0x7fb53015e3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:14.869 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:14.866+0000 7fb527e00640 1 -- 172.21.15.32:0/2120951071 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb5300b7420 msgr2=0x7fb530149080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:50:14.869 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:14.866+0000 7fb527e00640 1 --2- 172.21.15.32:0/2120951071 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb5300b7420 0x7fb530149080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:14.869 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:14.866+0000 7fb527e00640 1 -- 172.21.15.32:0/2120951071 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb530148db0 con 0x7fb5300cd5f0 2024-08-03T20:50:14.869 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:14.867+0000 7fb527e00640 1 --2- 172.21.15.32:0/2120951071 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb5300cd5f0 0x7fb5301f4290 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7fb51800b3e0 tx=0x7fb51800b8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:50:14.869 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:14.867+0000 7fb527400640 1 -- 172.21.15.32:0/2120951071 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb518015020 con 0x7fb5300cd5f0 2024-08-03T20:50:14.869 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:14.867+0000 7fb527400640 1 -- 172.21.15.32:0/2120951071 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fb518011940 con 0x7fb5300cd5f0 2024-08-03T20:50:14.869 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:14.867+0000 7fb527400640 1 -- 172.21.15.32:0/2120951071 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb518011d90 con 0x7fb5300cd5f0 2024-08-03T20:50:14.869 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:14.867+0000 7fb534e00640 1 -- 172.21.15.32:0/2120951071 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb5301495c0 con 0x7fb5300cd5f0 2024-08-03T20:50:14.870 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:14.867+0000 7fb534e00640 1 -- 172.21.15.32:0/2120951071 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb53014b8b0 con 0x7fb5300cd5f0 2024-08-03T20:50:14.870 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:14.868+0000 7fb534e00640 1 -- 172.21.15.32:0/2120951071 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb5300d10c0 con 0x7fb5300cd5f0 2024-08-03T20:50:14.870 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:14.869+0000 7fb527400640 1 -- 172.21.15.32:0/2120951071 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7fb518018020 con 0x7fb5300cd5f0 2024-08-03T20:50:14.871 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:14.870+0000 7fb527400640 1 --2- 172.21.15.32:0/2120951071 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fb50c077010 0x7fb50c079520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:50:14.871 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:14.870+0000 7fb527400640 1 -- 172.21.15.32:0/2120951071 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(47..47 src has 1..47) ==== 5881+0+0 (secure 0 0 0) 0x7fb5180989a0 con 0x7fb5300cd5f0 2024-08-03T20:50:14.872 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:14.871+0000 7fb52f400640 1 --2- 172.21.15.32:0/2120951071 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fb50c077010 0x7fb50c079520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:50:14.873 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:14.871+0000 7fb52f400640 1 --2- 172.21.15.32:0/2120951071 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fb50c077010 0x7fb50c079520 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7fb520033000 tx=0x7fb5200046c0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:50:14.876 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:14.875+0000 7fb527400640 1 -- 172.21.15.32:0/2120951071 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7fb51801f020 con 0x7fb5300cd5f0 2024-08-03T20:50:15.082 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:15.081+0000 7fb534e00640 1 -- 172.21.15.32:0/2120951071 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) -- 0x7fb53015f6c0 con 0x7fb5300cd5f0 2024-08-03T20:50:16.080 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:16.079+0000 7fb527400640 1 -- 172.21.15.32:0/2120951071 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v48) ==== 181+0+0 (secure 0 0 0) 0x7fb518061b70 con 0x7fb5300cd5f0 2024-08-03T20:50:16.081 INFO:teuthology.orchestra.run.smithi032.stderr:pool 'cephfs_metadata' removed 2024-08-03T20:50:16.084 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:16.082+0000 7fb534e00640 1 -- 172.21.15.32:0/2120951071 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fb50c077010 msgr2=0x7fb50c079520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:50:16.084 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:16.082+0000 7fb534e00640 1 --2- 172.21.15.32:0/2120951071 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fb50c077010 0x7fb50c079520 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7fb520033000 tx=0x7fb5200046c0 comp rx=0 tx=0).stop 2024-08-03T20:50:16.084 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:16.082+0000 7fb534e00640 1 -- 172.21.15.32:0/2120951071 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb5300cd5f0 msgr2=0x7fb5301f4290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:50:16.084 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:16.082+0000 7fb534e00640 1 --2- 172.21.15.32:0/2120951071 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb5300cd5f0 0x7fb5301f4290 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7fb51800b3e0 tx=0x7fb51800b8a0 comp rx=0 tx=0).stop 2024-08-03T20:50:16.085 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:16.083+0000 7fb534e00640 1 -- 172.21.15.32:0/2120951071 shutdown_connections 2024-08-03T20:50:16.085 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:16.083+0000 7fb534e00640 1 --2- 172.21.15.32:0/2120951071 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fb50c077010 0x7fb50c079520 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:16.085 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:16.083+0000 7fb534e00640 1 --2- 172.21.15.32:0/2120951071 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fb5300ce700 0x7fb53015e3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:16.085 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:16.083+0000 7fb534e00640 1 --2- 172.21.15.32:0/2120951071 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb5300cd5f0 0x7fb5301f4290 unknown :-1 s=CLOSED pgs=519 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:16.085 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:16.083+0000 7fb534e00640 1 --2- 172.21.15.32:0/2120951071 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb5300b7420 0x7fb530149080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:16.085 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:16.083+0000 7fb534e00640 1 -- 172.21.15.32:0/2120951071 >> 172.21.15.32:0/2120951071 conn(0x7fb5300d1e80 msgr2=0x7fb5300b4010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:50:16.085 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:16.083+0000 7fb534e00640 1 -- 172.21.15.32:0/2120951071 shutdown_connections 2024-08-03T20:50:16.085 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:16.083+0000 7fb534e00640 1 -- 172.21.15.32:0/2120951071 wait complete. 2024-08-03T20:50:16.115 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-08-03T20:50:16.351 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:16.349+0000 7eff05200640 1 -- 172.21.15.32:0/1346616391 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7eff000b31f0 msgr2=0x7eff0006f670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:50:16.352 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:16.349+0000 7eff05200640 1 --2- 172.21.15.32:0/1346616391 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7eff000b31f0 0x7eff0006f670 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7efef8005fd0 tx=0x7efef802a840 comp rx=0 tx=0).stop 2024-08-03T20:50:16.352 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:16.349+0000 7eff05200640 1 -- 172.21.15.32:0/1346616391 shutdown_connections 2024-08-03T20:50:16.352 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:16.349+0000 7eff05200640 1 --2- 172.21.15.32:0/1346616391 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7eff000b31f0 0x7eff0006f670 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:16.352 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:16.349+0000 7eff05200640 1 --2- 172.21.15.32:0/1346616391 >> v2:172.21.15.118:3300/0 conn(0x7eff000710e0 0x7eff00073e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:16.352 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:16.350+0000 7eff05200640 1 -- 172.21.15.32:0/1346616391 >> 172.21.15.32:0/1346616391 conn(0x7eff000cdc50 msgr2=0x7eff000ce060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:50:16.352 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:16.350+0000 7eff05200640 1 -- 172.21.15.32:0/1346616391 shutdown_connections 2024-08-03T20:50:16.353 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:16.350+0000 7eff05200640 1 -- 172.21.15.32:0/1346616391 wait complete. 2024-08-03T20:50:16.353 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:16.352+0000 7eff05200640 1 Processor -- start 2024-08-03T20:50:16.353 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:16.352+0000 7eff05200640 1 -- start start 2024-08-03T20:50:16.357 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:16.353+0000 7eff05200640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7eff000710e0 0x7eff001f4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:50:16.357 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:16.353+0000 7eff05200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7eff00160810 0x7eff000ce980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:50:16.357 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:16.353+0000 7eff05200640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7eff000c7170 0x7eff000ca790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:50:16.357 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:16.354+0000 7eff05200640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7eff0006fd30 con 0x7eff00160810 2024-08-03T20:50:16.357 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:16.354+0000 7eff05200640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7eff0006fbb0 con 0x7eff000c7170 2024-08-03T20:50:16.357 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:16.354+0000 7eff05200640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7eff0006feb0 con 0x7eff000710e0 2024-08-03T20:50:16.357 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:16.354+0000 7efef6a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7eff00160810 0x7eff000ce980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:50:16.357 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:16.354+0000 7efef6a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7eff00160810 0x7eff000ce980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:40080/0 (socket says 172.21.15.32:40080) 2024-08-03T20:50:16.358 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:16.354+0000 7efef6a00640 1 -- 172.21.15.32:0/1038216429 learned_addr learned my addr 172.21.15.32:0/1038216429 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:50:16.358 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:16.354+0000 7efeff400640 1 --2- 172.21.15.32:0/1038216429 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7eff000710e0 0x7eff001f4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:50:16.358 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:16.354+0000 7efeffe00640 1 --2- 172.21.15.32:0/1038216429 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7eff000c7170 0x7eff000ca790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:50:16.358 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:16.354+0000 7efef6a00640 1 -- 172.21.15.32:0/1038216429 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7eff000710e0 msgr2=0x7eff001f4d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:50:16.358 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:16.354+0000 7efef6a00640 1 --2- 172.21.15.32:0/1038216429 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7eff000710e0 0x7eff001f4d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:16.358 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:16.354+0000 7efef6a00640 1 -- 172.21.15.32:0/1038216429 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7eff000c7170 msgr2=0x7eff000ca790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:50:16.358 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:16.354+0000 7efef6a00640 1 --2- 172.21.15.32:0/1038216429 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7eff000c7170 0x7eff000ca790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:16.358 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:16.354+0000 7efef6a00640 1 -- 172.21.15.32:0/1038216429 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7eff000c9fd0 con 0x7eff00160810 2024-08-03T20:50:16.358 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:16.355+0000 7efef6a00640 1 --2- 172.21.15.32:0/1038216429 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7eff00160810 0x7eff000ce980 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7efee800c500 tx=0x7efee800c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:50:16.358 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:16.355+0000 7efef7e00640 1 -- 172.21.15.32:0/1038216429 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efee8016020 con 0x7eff00160810 2024-08-03T20:50:16.358 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:16.355+0000 7efef7e00640 1 -- 172.21.15.32:0/1038216429 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7efee8012940 con 0x7eff00160810 2024-08-03T20:50:16.358 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:16.355+0000 7eff05200640 1 -- 172.21.15.32:0/1038216429 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7eff001f5610 con 0x7eff00160810 2024-08-03T20:50:16.359 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:16.355+0000 7efef7e00640 1 -- 172.21.15.32:0/1038216429 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efee8012d90 con 0x7eff00160810 2024-08-03T20:50:16.359 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:16.355+0000 7eff05200640 1 -- 172.21.15.32:0/1038216429 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7eff001f5bd0 con 0x7eff00160810 2024-08-03T20:50:16.359 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:16.356+0000 7eff05200640 1 -- 172.21.15.32:0/1038216429 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7eff001f5380 con 0x7eff00160810 2024-08-03T20:50:16.359 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:16.357+0000 7efef7e00640 1 -- 172.21.15.32:0/1038216429 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7efee8019020 con 0x7eff00160810 2024-08-03T20:50:16.360 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:16.358+0000 7efef7e00640 1 --2- 172.21.15.32:0/1038216429 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7efedc077010 0x7efedc079520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:50:16.360 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:16.358+0000 7efef7e00640 1 -- 172.21.15.32:0/1038216429 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(48..48 src has 1..48) ==== 5443+0+0 (secure 0 0 0) 0x7efee80988d0 con 0x7eff00160810 2024-08-03T20:50:16.360 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:16.359+0000 7efeff400640 1 --2- 172.21.15.32:0/1038216429 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7efedc077010 0x7efedc079520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:50:16.361 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:16.360+0000 7efeff400640 1 --2- 172.21.15.32:0/1038216429 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7efedc077010 0x7efedc079520 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7efef00095f0 tx=0x7efef0005f90 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:50:16.364 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:16.363+0000 7efef7e00640 1 -- 172.21.15.32:0/1038216429 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7efee8020020 con 0x7eff00160810 2024-08-03T20:50:16.570 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:16.569+0000 7eff05200640 1 -- 172.21.15.32:0/1038216429 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) -- 0x7eff000703d0 con 0x7eff00160810 2024-08-03T20:50:17.093 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:17.092+0000 7efef7e00640 1 -- 172.21.15.32:0/1038216429 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v49) ==== 169+0+0 (secure 0 0 0) 0x7efee80614f0 con 0x7eff00160810 2024-08-03T20:50:17.093 INFO:teuthology.orchestra.run.smithi032.stderr:pool 'cephfs_data' removed 2024-08-03T20:50:17.097 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:17.095+0000 7eff05200640 1 -- 172.21.15.32:0/1038216429 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7efedc077010 msgr2=0x7efedc079520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:50:17.097 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:17.095+0000 7eff05200640 1 --2- 172.21.15.32:0/1038216429 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7efedc077010 0x7efedc079520 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7efef00095f0 tx=0x7efef0005f90 comp rx=0 tx=0).stop 2024-08-03T20:50:17.097 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:17.096+0000 7eff05200640 1 -- 172.21.15.32:0/1038216429 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7eff00160810 msgr2=0x7eff000ce980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:50:17.097 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:17.096+0000 7eff05200640 1 --2- 172.21.15.32:0/1038216429 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7eff00160810 0x7eff000ce980 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7efee800c500 tx=0x7efee800c9c0 comp rx=0 tx=0).stop 2024-08-03T20:50:17.098 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:17.096+0000 7eff05200640 1 -- 172.21.15.32:0/1038216429 shutdown_connections 2024-08-03T20:50:17.098 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:17.096+0000 7eff05200640 1 --2- 172.21.15.32:0/1038216429 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7efedc077010 0x7efedc079520 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:17.098 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:17.096+0000 7eff05200640 1 --2- 172.21.15.32:0/1038216429 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7eff000c7170 0x7eff000ca790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:17.098 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:17.096+0000 7eff05200640 1 --2- 172.21.15.32:0/1038216429 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7eff00160810 0x7eff000ce980 unknown :-1 s=CLOSED pgs=521 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:17.098 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:17.096+0000 7eff05200640 1 --2- 172.21.15.32:0/1038216429 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7eff000710e0 0x7eff001f4d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:17.098 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:17.097+0000 7eff05200640 1 -- 172.21.15.32:0/1038216429 >> 172.21.15.32:0/1038216429 conn(0x7eff000cdc50 msgr2=0x7eff0015bdf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:50:17.099 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:17.098+0000 7eff05200640 1 -- 172.21.15.32:0/1038216429 shutdown_connections 2024-08-03T20:50:17.099 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:17.098+0000 7eff05200640 1 -- 172.21.15.32:0/1038216429 wait complete. 2024-08-03T20:50:17.126 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config reset 27 2024-08-03T20:50:17.366 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:17.364+0000 7f9bdf200640 1 -- 172.21.15.32:0/342827505 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9bd815fff0 msgr2=0x7f9bd8160400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:50:17.366 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:17.364+0000 7f9bdf200640 1 --2- 172.21.15.32:0/342827505 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9bd815fff0 0x7f9bd8160400 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7f9bd8071700 tx=0x7f9bd002aa30 comp rx=0 tx=0).stop 2024-08-03T20:50:17.366 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:17.364+0000 7f9bdf200640 1 -- 172.21.15.32:0/342827505 shutdown_connections 2024-08-03T20:50:17.366 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:17.364+0000 7f9bdf200640 1 --2- 172.21.15.32:0/342827505 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9bd815fff0 0x7f9bd8160400 unknown :-1 s=CLOSED pgs=522 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:17.366 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:17.364+0000 7f9bdf200640 1 --2- 172.21.15.32:0/342827505 >> v2:172.21.15.118:3300/0 conn(0x7f9bd806e930 0x7f9bd8144f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:17.367 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:17.364+0000 7f9bdf200640 1 -- 172.21.15.32:0/342827505 >> 172.21.15.32:0/342827505 conn(0x7f9bd80b1bd0 msgr2=0x7f9bd80b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:50:17.367 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:17.364+0000 7f9bdf200640 1 -- 172.21.15.32:0/342827505 shutdown_connections 2024-08-03T20:50:17.367 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:17.364+0000 7f9bdf200640 1 -- 172.21.15.32:0/342827505 wait complete. 2024-08-03T20:50:17.368 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:17.367+0000 7f9bdf200640 1 Processor -- start 2024-08-03T20:50:17.368 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:17.367+0000 7f9bdf200640 1 -- start start 2024-08-03T20:50:17.370 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:17.368+0000 7f9bdf200640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f9bd806e930 0x7f9bd815f280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:50:17.370 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:17.368+0000 7f9bdf200640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f9bd8073e60 0x7f9bd813d8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:50:17.371 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:17.368+0000 7f9bdf200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9bd815fff0 0x7f9bd815d250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:50:17.371 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:17.368+0000 7f9bdf200640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f9bd80cf840 con 0x7f9bd815fff0 2024-08-03T20:50:17.371 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:17.368+0000 7f9bdf200640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f9bd8145f30 con 0x7f9bd8073e60 2024-08-03T20:50:17.371 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:17.368+0000 7f9bdf200640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7f9bd80cf9c0 con 0x7f9bd806e930 2024-08-03T20:50:17.371 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:17.368+0000 7f9bdd200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9bd815fff0 0x7f9bd815d250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:50:17.371 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:17.368+0000 7f9bdd200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9bd815fff0 0x7f9bd815d250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:40090/0 (socket says 172.21.15.32:40090) 2024-08-03T20:50:17.371 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:17.368+0000 7f9bdd200640 1 -- 172.21.15.32:0/1277361413 learned_addr learned my addr 172.21.15.32:0/1277361413 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:50:17.371 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:17.369+0000 7f9bdd200640 1 -- 172.21.15.32:0/1277361413 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f9bd806e930 msgr2=0x7f9bd815f280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:50:17.371 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:17.369+0000 7f9bd7e00640 1 --2- 172.21.15.32:0/1277361413 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f9bd806e930 0x7f9bd815f280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:50:17.371 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:17.369+0000 7f9bd7400640 1 --2- 172.21.15.32:0/1277361413 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f9bd8073e60 0x7f9bd813d8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:50:17.371 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:17.369+0000 7f9bdd200640 1 --2- 172.21.15.32:0/1277361413 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f9bd806e930 0x7f9bd815f280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:17.371 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:17.369+0000 7f9bdd200640 1 -- 172.21.15.32:0/1277361413 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f9bd8073e60 msgr2=0x7f9bd813d8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:50:17.371 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:17.369+0000 7f9bdd200640 1 --2- 172.21.15.32:0/1277361413 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f9bd8073e60 0x7f9bd813d8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:17.372 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:17.369+0000 7f9bdd200640 1 -- 172.21.15.32:0/1277361413 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9bd81f8870 con 0x7f9bd815fff0 2024-08-03T20:50:17.372 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:17.369+0000 7f9bd7400640 1 --2- 172.21.15.32:0/1277361413 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f9bd8073e60 0x7f9bd813d8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-03T20:50:17.372 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:17.369+0000 7f9bdd200640 1 --2- 172.21.15.32:0/1277361413 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9bd815fff0 0x7f9bd815d250 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7f9bd002b020 tx=0x7f9bd0003f10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:50:17.372 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:17.369+0000 7f9bcfe00640 1 -- 172.21.15.32:0/1277361413 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9bd0033050 con 0x7f9bd815fff0 2024-08-03T20:50:17.373 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:17.370+0000 7f9bcfe00640 1 -- 172.21.15.32:0/1277361413 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f9bd0004450 con 0x7f9bd815fff0 2024-08-03T20:50:17.373 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:17.370+0000 7f9bcfe00640 1 -- 172.21.15.32:0/1277361413 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9bd0038450 con 0x7f9bd815fff0 2024-08-03T20:50:17.373 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:17.370+0000 7f9bdf200640 1 -- 172.21.15.32:0/1277361413 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9bd81f8b50 con 0x7f9bd815fff0 2024-08-03T20:50:17.373 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:17.370+0000 7f9bdf200640 1 -- 172.21.15.32:0/1277361413 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9bd81f9110 con 0x7f9bd815fff0 2024-08-03T20:50:17.373 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:17.371+0000 7f9bdf200640 1 -- 172.21.15.32:0/1277361413 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9bd81f9550 con 0x7f9bd815fff0 2024-08-03T20:50:17.373 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:17.372+0000 7f9bcfe00640 1 -- 172.21.15.32:0/1277361413 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f9bd0036070 con 0x7f9bd815fff0 2024-08-03T20:50:17.374 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:17.373+0000 7f9bcfe00640 1 --2- 172.21.15.32:0/1277361413 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f9bb8077040 0x7f9bb8079550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:50:17.374 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:17.373+0000 7f9bcfe00640 1 -- 172.21.15.32:0/1277361413 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(49..49 src has 1..49) ==== 5029+0+0 (secure 0 0 0) 0x7f9bd00b9b60 con 0x7f9bd815fff0 2024-08-03T20:50:17.375 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:17.373+0000 7f9bd7e00640 1 --2- 172.21.15.32:0/1277361413 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f9bb8077040 0x7f9bb8079550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:50:17.376 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:17.375+0000 7f9bd7e00640 1 --2- 172.21.15.32:0/1277361413 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f9bb8077040 0x7f9bb8079550 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f9bc80058c0 tx=0x7f9bc8005850 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:50:17.379 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:17.377+0000 7f9bcfe00640 1 -- 172.21.15.32:0/1277361413 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7f9bd0082f60 con 0x7f9bd815fff0 2024-08-03T20:50:17.584 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:17.582+0000 7f9bdf200640 1 -- 172.21.15.32:0/1277361413 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "config reset", "num": 27} v 0) -- 0x7f9bd8160c50 con 0x7f9bd815fff0 2024-08-03T20:50:17.587 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:17.586+0000 7f9bcfe00640 1 -- 172.21.15.32:0/1277361413 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "config reset", "num": 27}]=0 v34) ==== 71+0+0 (secure 0 0 0) 0x7f9bd0082860 con 0x7f9bd815fff0 2024-08-03T20:50:17.591 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:17.589+0000 7f9bdf200640 1 -- 172.21.15.32:0/1277361413 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f9bb8077040 msgr2=0x7f9bb8079550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:50:17.591 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:17.589+0000 7f9bdf200640 1 --2- 172.21.15.32:0/1277361413 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f9bb8077040 0x7f9bb8079550 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f9bc80058c0 tx=0x7f9bc8005850 comp rx=0 tx=0).stop 2024-08-03T20:50:17.591 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:17.589+0000 7f9bdf200640 1 -- 172.21.15.32:0/1277361413 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9bd815fff0 msgr2=0x7f9bd815d250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:50:17.591 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:17.589+0000 7f9bdf200640 1 --2- 172.21.15.32:0/1277361413 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9bd815fff0 0x7f9bd815d250 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7f9bd002b020 tx=0x7f9bd0003f10 comp rx=0 tx=0).stop 2024-08-03T20:50:17.591 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:17.589+0000 7f9bdf200640 1 -- 172.21.15.32:0/1277361413 shutdown_connections 2024-08-03T20:50:17.591 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:17.589+0000 7f9bdf200640 1 --2- 172.21.15.32:0/1277361413 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f9bb8077040 0x7f9bb8079550 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:17.591 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:17.589+0000 7f9bdf200640 1 --2- 172.21.15.32:0/1277361413 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f9bd8073e60 0x7f9bd813d8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:17.591 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:17.589+0000 7f9bdf200640 1 --2- 172.21.15.32:0/1277361413 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9bd815fff0 0x7f9bd815d250 unknown :-1 s=CLOSED pgs=523 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:17.591 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:17.589+0000 7f9bdf200640 1 --2- 172.21.15.32:0/1277361413 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f9bd806e930 0x7f9bd815f280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:17.591 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:17.589+0000 7f9bdf200640 1 -- 172.21.15.32:0/1277361413 >> 172.21.15.32:0/1277361413 conn(0x7f9bd80b1bd0 msgr2=0x7f9bd80cd2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:50:17.592 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:17.589+0000 7f9bdf200640 1 -- 172.21.15.32:0/1277361413 shutdown_connections 2024-08-03T20:50:17.592 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:17.589+0000 7f9bdf200640 1 -- 172.21.15.32:0/1277361413 wait complete. 2024-08-03T20:50:17.615 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_damage.TestDamage.test_dentry_first_existing' 2024-08-03T20:50:17.844 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:17.842+0000 7f2293c00640 1 -- 172.21.15.32:0/3725628041 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f228c15e880 msgr2=0x7f228c073800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:50:17.845 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:17.842+0000 7f2293c00640 1 --2- 172.21.15.32:0/3725628041 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f228c15e880 0x7f228c073800 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7f228c072280 tx=0x7f228402b490 comp rx=0 tx=0).stop 2024-08-03T20:50:17.845 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:17.843+0000 7f2293c00640 1 -- 172.21.15.32:0/3725628041 shutdown_connections 2024-08-03T20:50:17.845 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:17.843+0000 7f2293c00640 1 --2- 172.21.15.32:0/3725628041 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f228c15e880 0x7f228c073800 unknown :-1 s=CLOSED pgs=525 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:17.845 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:17.843+0000 7f2293c00640 1 --2- 172.21.15.32:0/3725628041 >> v2:172.21.15.118:3300/0 conn(0x7f228c0b4de0 0x7f228c1515c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:17.845 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:17.843+0000 7f2293c00640 1 -- 172.21.15.32:0/3725628041 >> 172.21.15.32:0/3725628041 conn(0x7f228c0b1bd0 msgr2=0x7f228c0b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:50:17.845 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:17.843+0000 7f2293c00640 1 -- 172.21.15.32:0/3725628041 shutdown_connections 2024-08-03T20:50:17.845 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:17.843+0000 7f2293c00640 1 -- 172.21.15.32:0/3725628041 wait complete. 2024-08-03T20:50:17.848 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:17.846+0000 7f2293c00640 1 Processor -- start 2024-08-03T20:50:17.848 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:17.847+0000 7f2293c00640 1 -- start start 2024-08-03T20:50:17.850 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:17.848+0000 7f2293c00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f228c0733f0 0x7f228c13f980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:50:17.850 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:17.848+0000 7f2293c00640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f228c0b4de0 0x7f228c147cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:50:17.850 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:17.848+0000 7f2293c00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f228c146a90 0x7f228c14e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:50:17.850 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:17.848+0000 7f2293c00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f228c06e930 con 0x7f228c146a90 2024-08-03T20:50:17.850 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:17.848+0000 7f2293c00640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f228c070ad0 con 0x7f228c0733f0 2024-08-03T20:50:17.850 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:17.848+0000 7f2293c00640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7f228c06eab0 con 0x7f228c0b4de0 2024-08-03T20:50:17.851 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:17.849+0000 7f2291c00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f228c146a90 0x7f228c14e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:50:17.851 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:17.849+0000 7f2291c00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f228c146a90 0x7f228c14e9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:40114/0 (socket says 172.21.15.32:40114) 2024-08-03T20:50:17.851 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:17.849+0000 7f228be00640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f228c0b4de0 0x7f228c147cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:50:17.851 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:17.849+0000 7f2291c00640 1 -- 172.21.15.32:0/448760328 learned_addr learned my addr 172.21.15.32:0/448760328 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:50:17.851 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:17.849+0000 7f2291200640 1 --2- 172.21.15.32:0/448760328 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f228c0733f0 0x7f228c13f980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:50:17.852 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:17.850+0000 7f2291c00640 1 -- 172.21.15.32:0/448760328 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f228c0b4de0 msgr2=0x7f228c147cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:50:17.852 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:17.850+0000 7f2291c00640 1 --2- 172.21.15.32:0/448760328 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f228c0b4de0 0x7f228c147cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:17.852 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:17.850+0000 7f2291c00640 1 -- 172.21.15.32:0/448760328 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f228c0733f0 msgr2=0x7f228c13f980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:50:17.852 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:17.850+0000 7f2291c00640 1 --2- 172.21.15.32:0/448760328 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f228c0733f0 0x7f228c13f980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:17.852 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:17.850+0000 7f2291c00640 1 -- 172.21.15.32:0/448760328 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f228c0cd190 con 0x7f228c146a90 2024-08-03T20:50:17.853 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:17.850+0000 7f2291c00640 1 --2- 172.21.15.32:0/448760328 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f228c146a90 0x7f228c14e9a0 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7f228402ba80 tx=0x7f2284004660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:50:17.853 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:17.851+0000 7f2288c00640 1 -- 172.21.15.32:0/448760328 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2284037040 con 0x7f228c146a90 2024-08-03T20:50:17.853 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:17.851+0000 7f2293c00640 1 -- 172.21.15.32:0/448760328 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f228c1f3500 con 0x7f228c146a90 2024-08-03T20:50:17.853 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:17.851+0000 7f2288c00640 1 -- 172.21.15.32:0/448760328 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f2284003e60 con 0x7f228c146a90 2024-08-03T20:50:17.853 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:17.851+0000 7f2288c00640 1 -- 172.21.15.32:0/448760328 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2284007c90 con 0x7f228c146a90 2024-08-03T20:50:17.853 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:17.851+0000 7f2293c00640 1 -- 172.21.15.32:0/448760328 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f228c1f37c0 con 0x7f228c146a90 2024-08-03T20:50:17.854 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:17.852+0000 7f2293c00640 1 -- 172.21.15.32:0/448760328 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f228c1515c0 con 0x7f228c146a90 2024-08-03T20:50:17.855 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:17.854+0000 7f2288c00640 1 -- 172.21.15.32:0/448760328 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f2284042030 con 0x7f228c146a90 2024-08-03T20:50:17.856 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:17.855+0000 7f2288c00640 1 --2- 172.21.15.32:0/448760328 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f226c077040 0x7f226c079550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:50:17.856 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:17.855+0000 7f2288c00640 1 -- 172.21.15.32:0/448760328 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(49..49 src has 1..49) ==== 5029+0+0 (secure 0 0 0) 0x7f22840b8130 con 0x7f228c146a90 2024-08-03T20:50:17.856 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:17.855+0000 7f2291200640 1 --2- 172.21.15.32:0/448760328 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f226c077040 0x7f226c079550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:50:17.857 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:17.856+0000 7f2291200640 1 --2- 172.21.15.32:0/448760328 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f226c077040 0x7f226c079550 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f2280005fd0 tx=0x7f2280005850 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:50:17.860 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:17.859+0000 7f2288c00640 1 -- 172.21.15.32:0/448760328 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7f2284081650 con 0x7f228c146a90 2024-08-03T20:50:18.165 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:18.164+0000 7f2293c00640 1 -- 172.21.15.32:0/448760328 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_damage.TestDamage.test_dentry_first_existing"]} v 0) -- 0x7f228c072c80 con 0x7f228c146a90 2024-08-03T20:50:19.102 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:19.100+0000 7f2288c00640 1 -- 172.21.15.32:0/448760328 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_damage.TestDamage.test_dentry_first_existing"]}]=0 v159) ==== 141+0+0 (secure 0 0 0) 0x7f2284080f50 con 0x7f228c146a90 2024-08-03T20:50:19.104 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:19.103+0000 7f2293c00640 1 -- 172.21.15.32:0/448760328 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f226c077040 msgr2=0x7f226c079550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:50:19.105 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:19.103+0000 7f2293c00640 1 --2- 172.21.15.32:0/448760328 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f226c077040 0x7f226c079550 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f2280005fd0 tx=0x7f2280005850 comp rx=0 tx=0).stop 2024-08-03T20:50:19.105 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:19.103+0000 7f2293c00640 1 -- 172.21.15.32:0/448760328 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f228c146a90 msgr2=0x7f228c14e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:50:19.105 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:19.103+0000 7f2293c00640 1 --2- 172.21.15.32:0/448760328 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f228c146a90 0x7f228c14e9a0 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7f228402ba80 tx=0x7f2284004660 comp rx=0 tx=0).stop 2024-08-03T20:50:19.105 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:19.103+0000 7f2293c00640 1 -- 172.21.15.32:0/448760328 shutdown_connections 2024-08-03T20:50:19.105 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:19.103+0000 7f2293c00640 1 --2- 172.21.15.32:0/448760328 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f226c077040 0x7f226c079550 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:19.105 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:19.103+0000 7f2293c00640 1 --2- 172.21.15.32:0/448760328 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f228c146a90 0x7f228c14e9a0 unknown :-1 s=CLOSED pgs=526 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:19.105 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:19.103+0000 7f2293c00640 1 --2- 172.21.15.32:0/448760328 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f228c0b4de0 0x7f228c147cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:19.105 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:19.103+0000 7f2293c00640 1 --2- 172.21.15.32:0/448760328 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f228c0733f0 0x7f228c13f980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:19.105 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:19.103+0000 7f2293c00640 1 -- 172.21.15.32:0/448760328 >> 172.21.15.32:0/448760328 conn(0x7f228c0b1bd0 msgr2=0x7f228c148990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:50:19.105 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:19.103+0000 7f2293c00640 1 -- 172.21.15.32:0/448760328 shutdown_connections 2024-08-03T20:50:19.105 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:19.103+0000 7f2293c00640 1 -- 172.21.15.32:0/448760328 wait complete. 2024-08-03T20:50:19.129 INFO:tasks.cephfs_test_runner:That the MDS won't abort when the dentry is already known to be damaged. ... ok 2024-08-03T20:50:19.129 INFO:tasks.cephfs_test_runner:Starting test: test_dentry_first_precommit (tasks.cephfs.test_damage.TestDamage) That the MDS won't write a dentry with new damage to CDentry::first 2024-08-03T20:50:19.129 INFO:tasks.cephfs_test_runner:test_dentry_first_precommit (tasks.cephfs.test_damage.TestDamage) 2024-08-03T20:50:19.130 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-08-03T20:50:19.359 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:19.357+0000 7f24a8000640 1 -- 172.21.15.32:0/1681682018 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f24a015e170 msgr2=0x7f24a015e580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:50:19.359 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:19.357+0000 7f24a8000640 1 --2- 172.21.15.32:0/1681682018 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f24a015e170 0x7f24a015e580 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7f24a0072280 tx=0x7f249802a980 comp rx=0 tx=0).stop 2024-08-03T20:50:19.359 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:19.357+0000 7f24a8000640 1 -- 172.21.15.32:0/1681682018 shutdown_connections 2024-08-03T20:50:19.359 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:19.357+0000 7f24a8000640 1 --2- 172.21.15.32:0/1681682018 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f24a015e170 0x7f24a015e580 unknown :-1 s=CLOSED pgs=527 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:19.359 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:19.357+0000 7f24a8000640 1 --2- 172.21.15.32:0/1681682018 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f24a00733f0 0x7f24a0073800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:19.360 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:19.357+0000 7f24a8000640 1 --2- 172.21.15.32:0/1681682018 >> v2:172.21.15.118:3300/0 conn(0x7f24a00b4de0 0x7f24a01515c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:19.360 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:19.357+0000 7f24a8000640 1 -- 172.21.15.32:0/1681682018 >> 172.21.15.32:0/1681682018 conn(0x7f24a00b1bd0 msgr2=0x7f24a00b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:50:19.360 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:19.357+0000 7f24a8000640 1 -- 172.21.15.32:0/1681682018 shutdown_connections 2024-08-03T20:50:19.360 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:19.357+0000 7f24a8000640 1 -- 172.21.15.32:0/1681682018 wait complete. 2024-08-03T20:50:19.361 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:19.359+0000 7f24a8000640 1 Processor -- start 2024-08-03T20:50:19.361 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:19.360+0000 7f24a8000640 1 -- start start 2024-08-03T20:50:19.363 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:19.361+0000 7f24a8000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f24a00733f0 0x7f24a01f5690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:50:19.363 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:19.361+0000 7f24a8000640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f24a00b4de0 0x7f24a01468d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:50:19.363 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:19.361+0000 7f24a8000640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f24a015e170 0x7f24a01e5d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:50:19.363 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:19.361+0000 7f24a8000640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f24a006e930 con 0x7f24a00733f0 2024-08-03T20:50:19.363 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:19.361+0000 7f24a8000640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f24a0070ad0 con 0x7f24a015e170 2024-08-03T20:50:19.363 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:19.361+0000 7f24a8000640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7f24a006eab0 con 0x7f24a00b4de0 2024-08-03T20:50:19.364 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:19.361+0000 7f24a5600640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f24a00733f0 0x7f24a01f5690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:50:19.364 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:19.361+0000 7f24a5600640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f24a00733f0 0x7f24a01f5690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:40132/0 (socket says 172.21.15.32:40132) 2024-08-03T20:50:19.364 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:19.361+0000 7f24a5600640 1 -- 172.21.15.32:0/1661438745 learned_addr learned my addr 172.21.15.32:0/1661438745 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:50:19.364 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:19.361+0000 7f24a4c00640 1 --2- 172.21.15.32:0/1661438745 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f24a00b4de0 0x7f24a01468d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:50:19.364 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:19.361+0000 7f24a6000640 1 --2- 172.21.15.32:0/1661438745 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f24a015e170 0x7f24a01e5d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:50:19.364 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:19.362+0000 7f24a5600640 1 -- 172.21.15.32:0/1661438745 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f24a00b4de0 msgr2=0x7f24a01468d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:50:19.364 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:19.362+0000 7f24a5600640 1 --2- 172.21.15.32:0/1661438745 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f24a00b4de0 0x7f24a01468d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:19.364 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:19.362+0000 7f24a5600640 1 -- 172.21.15.32:0/1661438745 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f24a015e170 msgr2=0x7f24a01e5d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:50:19.364 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:19.362+0000 7f24a5600640 1 --2- 172.21.15.32:0/1661438745 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f24a015e170 0x7f24a01e5d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:19.364 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:19.362+0000 7f24a5600640 1 -- 172.21.15.32:0/1661438745 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f24a0146e10 con 0x7f24a00733f0 2024-08-03T20:50:19.365 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:19.362+0000 7f24a5600640 1 --2- 172.21.15.32:0/1661438745 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f24a00733f0 0x7f24a01f5690 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7f249400d770 tx=0x7f249400dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:50:19.365 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:19.362+0000 7f249d600640 1 -- 172.21.15.32:0/1661438745 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2494017070 con 0x7f24a00733f0 2024-08-03T20:50:19.365 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:19.362+0000 7f249d600640 1 -- 172.21.15.32:0/1661438745 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f2494009d20 con 0x7f24a00733f0 2024-08-03T20:50:19.365 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:19.362+0000 7f249d600640 1 -- 172.21.15.32:0/1661438745 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24940106c0 con 0x7f24a00733f0 2024-08-03T20:50:19.365 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:19.362+0000 7f24a8000640 1 -- 172.21.15.32:0/1661438745 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f24a00ccfe0 con 0x7f24a00733f0 2024-08-03T20:50:19.365 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:19.362+0000 7f24a8000640 1 -- 172.21.15.32:0/1661438745 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f24a00cd5a0 con 0x7f24a00733f0 2024-08-03T20:50:19.366 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:19.364+0000 7f24a8000640 1 -- 172.21.15.32:0/1661438745 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f24a01515c0 con 0x7f24a00733f0 2024-08-03T20:50:19.366 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:19.365+0000 7f249d600640 1 -- 172.21.15.32:0/1661438745 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f249401a070 con 0x7f24a00733f0 2024-08-03T20:50:19.367 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:19.366+0000 7f249d600640 1 --2- 172.21.15.32:0/1661438745 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f2480077040 0x7f2480079550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:50:19.367 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:19.366+0000 7f249d600640 1 -- 172.21.15.32:0/1661438745 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(49..49 src has 1..49) ==== 5029+0+0 (secure 0 0 0) 0x7f2494098610 con 0x7f24a00733f0 2024-08-03T20:50:19.368 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:19.366+0000 7f24a4c00640 1 --2- 172.21.15.32:0/1661438745 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f2480077040 0x7f2480079550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:50:19.368 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:19.367+0000 7f24a4c00640 1 --2- 172.21.15.32:0/1661438745 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f2480077040 0x7f2480079550 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f24a0147170 tx=0x7f248c00b3e0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:50:19.371 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:19.370+0000 7f249d600640 1 -- 172.21.15.32:0/1661438745 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7f2494061460 con 0x7f24a00733f0 2024-08-03T20:50:19.576 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:19.575+0000 7f24a8000640 1 -- 172.21.15.32:0/1661438745 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f24a015d070 con 0x7f24a00733f0 2024-08-03T20:50:19.577 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:19.576+0000 7f249d600640 1 -- 172.21.15.32:0/1661438745 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v49) ==== 74+0+13088 (secure 0 0 0) 0x7f24940659b0 con 0x7f24a00733f0 2024-08-03T20:50:19.578 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-03T20:50:19.578 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":49,"fsid":"40ab141a-01d7-49b9-ae3c-2edec9876e95","created":"2024-08-03T20:47:14.957362+0000","modified":"2024-08-03T20:50:17.087334+0000","last_up_change":"2024-08-03T20:47:28.874451+0000","last_in_change":"2024-08-03T20:47:20.079442+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-08-03T20:47:28.261300+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5a83e759-9035-438b-889a-0a6af1f0a0d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6816","nonce":3066156116},{"type":"v1","addr":"172.21.15.32:6817","nonce":3066156116}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6818","nonce":3066156116},{"type":"v1","addr":"172.21.15.32:6819","nonce":3066156116}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6822","nonce":3066156116},{"type":"v1","addr":"172.21.15.32:6823","nonce":3066156116}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6820","nonce":3066156116},{"type":"v1","addr":"172.21.15.32:6821","nonce":3066156116}]},"public_addr":"172.21.15.32:6817/3066156116","cluster_addr":"172.21.15.32:6819/3066156116","heartbeat_back_addr":"172.21.15.32:6823/3066156116","heartbeat_front_addr":"172.21.15.32:6821/3066156116","state":["exists","up"]},{"osd":1,"uuid":"d7c5c210-da4f-49f9-b9e7-1dfb02daa4bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6824","nonce":1551241759},{"type":"v1","addr":"172.21.15.32:6825","nonce":1551241759}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6826","nonce":1551241759},{"type":"v1","addr":"172.21.15.32:6827","nonce":1551241759}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6830","nonce":1551241759},{"type":"v1","addr":"172.21.15.32:6831","nonce":1551241759}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6828","nonce":1551241759},{"type":"v1","addr":"172.21.15.32:6829","nonce":1551241759}]},"public_addr":"172.21.15.32:6825/1551241759","cluster_addr":"172.21.15.32:6827/1551241759","heartbeat_back_addr":"172.21.15.32:6831/1551241759","heartbeat_front_addr":"172.21.15.32:6829/1551241759","state":["exists","up"]},{"osd":2,"uuid":"57f2e6de-a201-45c2-8981-8e56a4875ebd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6808","nonce":3806708944},{"type":"v1","addr":"172.21.15.32:6809","nonce":3806708944}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6810","nonce":3806708944},{"type":"v1","addr":"172.21.15.32:6811","nonce":3806708944}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6814","nonce":3806708944},{"type":"v1","addr":"172.21.15.32:6815","nonce":3806708944}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6812","nonce":3806708944},{"type":"v1","addr":"172.21.15.32:6813","nonce":3806708944}]},"public_addr":"172.21.15.32:6809/3806708944","cluster_addr":"172.21.15.32:6811/3806708944","heartbeat_back_addr":"172.21.15.32:6815/3806708944","heartbeat_front_addr":"172.21.15.32:6813/3806708944","state":["exists","up"]},{"osd":3,"uuid":"6439a280-da45-4148-8130-f16323935ff3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6800","nonce":9088303},{"type":"v1","addr":"172.21.15.32:6801","nonce":9088303}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6802","nonce":9088303},{"type":"v1","addr":"172.21.15.32:6803","nonce":9088303}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6806","nonce":9088303},{"type":"v1","addr":"172.21.15.32:6807","nonce":9088303}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6804","nonce":9088303},{"type":"v1","addr":"172.21.15.32:6805","nonce":9088303}]},"public_addr":"172.21.15.32:6801/9088303","cluster_addr":"172.21.15.32:6803/9088303","heartbeat_back_addr":"172.21.15.32:6807/9088303","heartbeat_front_addr":"172.21.15.32:6805/9088303","state":["exists","up"]},{"osd":4,"uuid":"4f902eff-1e18-418d-82c9-2151c69524fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6802","nonce":864752275},{"type":"v1","addr":"172.21.15.118:6803","nonce":864752275}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6804","nonce":864752275},{"type":"v1","addr":"172.21.15.118:6805","nonce":864752275}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6808","nonce":864752275},{"type":"v1","addr":"172.21.15.118:6809","nonce":864752275}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6806","nonce":864752275},{"type":"v1","addr":"172.21.15.118:6807","nonce":864752275}]},"public_addr":"172.21.15.118:6803/864752275","cluster_addr":"172.21.15.118:6805/864752275","heartbeat_back_addr":"172.21.15.118:6809/864752275","heartbeat_front_addr":"172.21.15.118:6807/864752275","state":["exists","up"]},{"osd":5,"uuid":"bfd9feac-dd06-458c-91b9-af7f175f612c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6810","nonce":2724891770},{"type":"v1","addr":"172.21.15.118:6811","nonce":2724891770}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6812","nonce":2724891770},{"type":"v1","addr":"172.21.15.118:6813","nonce":2724891770}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6816","nonce":2724891770},{"type":"v1","addr":"172.21.15.118:6817","nonce":2724891770}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6814","nonce":2724891770},{"type":"v1","addr":"172.21.15.118:6815","nonce":2724891770}]},"public_addr":"172.21.15.118:6811/2724891770","cluster_addr":"172.21.15.118:6813/2724891770","heartbeat_back_addr":"172.21.15.118:6817/2724891770","heartbeat_front_addr":"172.21.15.118:6815/2724891770","state":["exists","up"]},{"osd":6,"uuid":"5ed5a279-9d8e-416d-bbc7-7edd3e6d5705","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6825","nonce":801947386},{"type":"v1","addr":"172.21.15.118:6827","nonce":801947386}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6828","nonce":801947386},{"type":"v1","addr":"172.21.15.118:6829","nonce":801947386}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6832","nonce":801947386},{"type":"v1","addr":"172.21.15.118:6833","nonce":801947386}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6830","nonce":801947386},{"type":"v1","addr":"172.21.15.118:6831","nonce":801947386}]},"public_addr":"172.21.15.118:6827/801947386","cluster_addr":"172.21.15.118:6829/801947386","heartbeat_back_addr":"172.21.15.118:6833/801947386","heartbeat_front_addr":"172.21.15.118:6831/801947386","state":["exists","up"]},{"osd":7,"uuid":"f2086c47-875e-4451-a436-2008d1ea16af","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6818","nonce":653000048},{"type":"v1","addr":"172.21.15.118:6819","nonce":653000048}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6820","nonce":653000048},{"type":"v1","addr":"172.21.15.118:6821","nonce":653000048}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6824","nonce":653000048},{"type":"v1","addr":"172.21.15.118:6826","nonce":653000048}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6822","nonce":653000048},{"type":"v1","addr":"172.21.15.118:6823","nonce":653000048}]},"public_addr":"172.21.15.118:6819/653000048","cluster_addr":"172.21.15.118:6821/653000048","heartbeat_back_addr":"172.21.15.118:6826/653000048","heartbeat_front_addr":"172.21.15.118:6823/653000048","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-03T20:47:25.491994+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-03T20:47:25.587358+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-03T20:47:25.454175+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-03T20:47:25.377734+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-03T20:47:25.412351+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-03T20:47:25.499815+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-03T20:47:25.575044+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-03T20:47:25.552383+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.32:6835/1726912144":"2024-08-04T20:50:13.597155+0000","172.21.15.32:6834/1726912144":"2024-08-04T20:50:13.597155+0000","172.21.15.32:6835/678469885":"2024-08-04T20:50:05.030253+0000","172.21.15.32:6834/678469885":"2024-08-04T20:50:05.030253+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-08-03T20:50:19.580 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:19.578+0000 7f24a8000640 1 -- 172.21.15.32:0/1661438745 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f2480077040 msgr2=0x7f2480079550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:50:19.580 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:19.578+0000 7f24a8000640 1 --2- 172.21.15.32:0/1661438745 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f2480077040 0x7f2480079550 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f24a0147170 tx=0x7f248c00b3e0 comp rx=0 tx=0).stop 2024-08-03T20:50:19.580 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:19.578+0000 7f24a8000640 1 -- 172.21.15.32:0/1661438745 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f24a00733f0 msgr2=0x7f24a01f5690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:50:19.581 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:19.578+0000 7f24a8000640 1 --2- 172.21.15.32:0/1661438745 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f24a00733f0 0x7f24a01f5690 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7f249400d770 tx=0x7f249400dc30 comp rx=0 tx=0).stop 2024-08-03T20:50:19.581 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:19.578+0000 7f24a8000640 1 -- 172.21.15.32:0/1661438745 shutdown_connections 2024-08-03T20:50:19.581 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:19.578+0000 7f24a8000640 1 --2- 172.21.15.32:0/1661438745 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f2480077040 0x7f2480079550 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:19.581 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:19.578+0000 7f24a8000640 1 --2- 172.21.15.32:0/1661438745 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f24a015e170 0x7f24a01e5d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:19.581 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:19.578+0000 7f24a8000640 1 --2- 172.21.15.32:0/1661438745 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f24a00b4de0 0x7f24a01468d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:19.581 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:19.578+0000 7f24a8000640 1 --2- 172.21.15.32:0/1661438745 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f24a00733f0 0x7f24a01f5690 unknown :-1 s=CLOSED pgs=528 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:19.581 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:19.578+0000 7f24a8000640 1 -- 172.21.15.32:0/1661438745 >> 172.21.15.32:0/1661438745 conn(0x7f24a00b1bd0 msgr2=0x7f24a0150bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:50:19.581 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:19.579+0000 7f24a8000640 1 -- 172.21.15.32:0/1661438745 shutdown_connections 2024-08-03T20:50:19.581 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:19.579+0000 7f24a8000640 1 -- 172.21.15.32:0/1661438745 wait complete. 2024-08-03T20:50:19.607 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-08-03T20:47:28.261300+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-08-03T20:50:19.607 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-08-03T20:50:19.840 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:19.838+0000 7fdc96400640 1 -- 172.21.15.32:0/76792389 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdc74003a50 msgr2=0x7fdc74003ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:50:19.840 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:19.838+0000 7fdc96400640 1 --2- 172.21.15.32:0/76792389 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdc74003a50 0x7fdc74003ec0 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7fdc8000c290 tx=0x7fdc74005830 comp rx=0 tx=0).stop 2024-08-03T20:50:19.841 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:19.838+0000 7fdc8fe00640 1 -- 172.21.15.32:0/76792389 reap_dead start 2024-08-03T20:50:19.841 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:19.838+0000 7fdc96400640 1 -- 172.21.15.32:0/76792389 shutdown_connections 2024-08-03T20:50:19.841 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:19.838+0000 7fdc96400640 1 -- 172.21.15.32:0/76792389 >> 172.21.15.32:0/76792389 conn(0x7fdc900b1bd0 msgr2=0x7fdc900b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:50:19.841 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:19.838+0000 7fdc96400640 1 -- 172.21.15.32:0/76792389 shutdown_connections 2024-08-03T20:50:19.841 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:19.838+0000 7fdc96400640 1 -- 172.21.15.32:0/76792389 wait complete. 2024-08-03T20:50:19.842 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:19.841+0000 7fdc96400640 1 Processor -- start 2024-08-03T20:50:19.842 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:19.841+0000 7fdc96400640 1 -- start start 2024-08-03T20:50:19.845 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:19.842+0000 7fdc96400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdc74003a50 0x7fdc90159370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:50:19.845 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:19.842+0000 7fdc96400640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fdc901ef250 0x7fdc901f88d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:50:19.845 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:19.842+0000 7fdc96400640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fdc901f8e10 0x7fdc901f5330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:50:19.845 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:19.842+0000 7fdc96400640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fdc900cf6f0 con 0x7fdc74003a50 2024-08-03T20:50:19.845 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:19.842+0000 7fdc96400640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7fdc900cf570 con 0x7fdc901f8e10 2024-08-03T20:50:19.845 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:19.842+0000 7fdc96400640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7fdc900b3ce0 con 0x7fdc901ef250 2024-08-03T20:50:19.846 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:19.842+0000 7fdc8f400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdc74003a50 0x7fdc90159370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:50:19.846 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:19.843+0000 7fdc8f400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdc74003a50 0x7fdc90159370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:60468/0 (socket says 172.21.15.32:60468) 2024-08-03T20:50:19.846 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:19.843+0000 7fdc8f400640 1 -- 172.21.15.32:0/165068490 learned_addr learned my addr 172.21.15.32:0/165068490 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:50:19.846 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:19.843+0000 7fdc87e00640 1 --2- 172.21.15.32:0/165068490 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fdc901ef250 0x7fdc901f88d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:50:19.846 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:19.843+0000 7fdc8fe00640 1 --2- 172.21.15.32:0/165068490 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fdc901f8e10 0x7fdc901f5330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:50:19.846 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:19.843+0000 7fdc8f400640 1 -- 172.21.15.32:0/165068490 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fdc901ef250 msgr2=0x7fdc901f88d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:50:19.846 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:19.843+0000 7fdc8f400640 1 --2- 172.21.15.32:0/165068490 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fdc901ef250 0x7fdc901f88d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:19.846 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:19.843+0000 7fdc8f400640 1 -- 172.21.15.32:0/165068490 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fdc901f8e10 msgr2=0x7fdc901f5330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:50:19.846 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:19.843+0000 7fdc8f400640 1 --2- 172.21.15.32:0/165068490 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fdc901f8e10 0x7fdc901f5330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:19.846 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:19.843+0000 7fdc8f400640 1 -- 172.21.15.32:0/165068490 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdc901f9220 con 0x7fdc74003a50 2024-08-03T20:50:19.846 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:19.843+0000 7fdc8fe00640 1 --2- 172.21.15.32:0/165068490 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fdc901f8e10 0x7fdc901f5330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-03T20:50:19.846 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:19.843+0000 7fdc8f400640 1 --2- 172.21.15.32:0/165068490 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdc74003a50 0x7fdc90159370 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7fdc8000c290 tx=0x7fdc80006d70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:50:19.846 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:19.844+0000 7fdc87400640 1 -- 172.21.15.32:0/165068490 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc80031040 con 0x7fdc74003a50 2024-08-03T20:50:19.847 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:19.844+0000 7fdc87400640 1 -- 172.21.15.32:0/165068490 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fdc800035f0 con 0x7fdc74003a50 2024-08-03T20:50:19.847 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:19.844+0000 7fdc87400640 1 -- 172.21.15.32:0/165068490 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc80003a00 con 0x7fdc74003a50 2024-08-03T20:50:19.847 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:19.844+0000 7fdc96400640 1 -- 172.21.15.32:0/165068490 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdc90143d80 con 0x7fdc74003a50 2024-08-03T20:50:19.847 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:19.844+0000 7fdc96400640 1 -- 172.21.15.32:0/165068490 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdc90144340 con 0x7fdc74003a50 2024-08-03T20:50:19.847 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:19.845+0000 7fdc96400640 1 -- 172.21.15.32:0/165068490 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdc900cd100 con 0x7fdc74003a50 2024-08-03T20:50:19.847 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:19.846+0000 7fdc87400640 1 -- 172.21.15.32:0/165068490 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7fdc80034070 con 0x7fdc74003a50 2024-08-03T20:50:19.849 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:19.848+0000 7fdc87400640 1 --2- 172.21.15.32:0/165068490 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fdc6c077010 0x7fdc6c079520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:50:19.849 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:19.848+0000 7fdc87400640 1 -- 172.21.15.32:0/165068490 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(49..49 src has 1..49) ==== 5029+0+0 (secure 0 0 0) 0x7fdc800b86e0 con 0x7fdc74003a50 2024-08-03T20:50:19.849 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:19.848+0000 7fdc87e00640 1 --2- 172.21.15.32:0/165068490 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fdc6c077010 0x7fdc6c079520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:50:19.850 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:19.849+0000 7fdc87e00640 1 --2- 172.21.15.32:0/165068490 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fdc6c077010 0x7fdc6c079520 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7fdc78005eb0 tx=0x7fdc78005f90 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:50:19.854 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:19.852+0000 7fdc87400640 1 -- 172.21.15.32:0/165068490 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7fdc80081b70 con 0x7fdc74003a50 2024-08-03T20:50:20.061 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:20.059+0000 7fdc96400640 1 -- 172.21.15.32:0/165068490 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fdc9006eba0 con 0x7fdc74003a50 2024-08-03T20:50:20.061 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:20.060+0000 7fdc87400640 1 -- 172.21.15.32:0/165068490 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v49) ==== 93+0+10 (secure 0 0 0) 0x7fdc80081470 con 0x7fdc74003a50 2024-08-03T20:50:20.061 INFO:teuthology.orchestra.run.smithi032.stdout:pg_num: 1 2024-08-03T20:50:20.066 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:20.062+0000 7fdc96400640 1 -- 172.21.15.32:0/165068490 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fdc6c077010 msgr2=0x7fdc6c079520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:50:20.066 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:20.062+0000 7fdc96400640 1 --2- 172.21.15.32:0/165068490 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fdc6c077010 0x7fdc6c079520 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7fdc78005eb0 tx=0x7fdc78005f90 comp rx=0 tx=0).stop 2024-08-03T20:50:20.066 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:20.062+0000 7fdc96400640 1 -- 172.21.15.32:0/165068490 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdc74003a50 msgr2=0x7fdc90159370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:50:20.066 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:20.062+0000 7fdc96400640 1 --2- 172.21.15.32:0/165068490 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdc74003a50 0x7fdc90159370 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7fdc8000c290 tx=0x7fdc80006d70 comp rx=0 tx=0).stop 2024-08-03T20:50:20.066 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:20.062+0000 7fdc96400640 1 -- 172.21.15.32:0/165068490 shutdown_connections 2024-08-03T20:50:20.066 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:20.062+0000 7fdc96400640 1 --2- 172.21.15.32:0/165068490 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fdc6c077010 0x7fdc6c079520 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:20.066 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:20.062+0000 7fdc96400640 1 --2- 172.21.15.32:0/165068490 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fdc901f8e10 0x7fdc901f5330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:20.067 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:20.062+0000 7fdc96400640 1 --2- 172.21.15.32:0/165068490 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fdc901ef250 0x7fdc901f88d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:20.067 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:20.062+0000 7fdc96400640 1 --2- 172.21.15.32:0/165068490 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdc74003a50 0x7fdc90159370 unknown :-1 s=CLOSED pgs=531 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:20.067 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:20.062+0000 7fdc96400640 1 -- 172.21.15.32:0/165068490 >> 172.21.15.32:0/165068490 conn(0x7fdc900b1bd0 msgr2=0x7fdc901416c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:50:20.067 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:20.062+0000 7fdc96400640 1 -- 172.21.15.32:0/165068490 shutdown_connections 2024-08-03T20:50:20.067 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:20.062+0000 7fdc96400640 1 -- 172.21.15.32:0/165068490 wait complete. 2024-08-03T20:50:20.086 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_damage.TestDamage.test_dentry_first_precommit' 2024-08-03T20:50:20.315 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:20.313+0000 7fdab0800640 1 -- 172.21.15.32:0/4223783142 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdaa806f110 msgr2=0x7fdaa8153340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:50:20.315 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:20.313+0000 7fdab0800640 1 --2- 172.21.15.32:0/4223783142 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdaa806f110 0x7fdaa8153340 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7fda9c009830 tx=0x7fda9c02a480 comp rx=0 tx=0).stop 2024-08-03T20:50:20.315 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:20.313+0000 7fdab0800640 1 -- 172.21.15.32:0/4223783142 shutdown_connections 2024-08-03T20:50:20.315 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:20.313+0000 7fdab0800640 1 --2- 172.21.15.32:0/4223783142 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fdaa8153880 0x7fdaa80af520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:20.315 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:20.313+0000 7fdab0800640 1 --2- 172.21.15.32:0/4223783142 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdaa806f110 0x7fdaa8153340 unknown :-1 s=CLOSED pgs=533 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:20.315 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:20.313+0000 7fdab0800640 1 -- 172.21.15.32:0/4223783142 >> 172.21.15.32:0/4223783142 conn(0x7fdaa80cd1e0 msgr2=0x7fdaa80cd5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:50:20.315 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:20.313+0000 7fdab0800640 1 -- 172.21.15.32:0/4223783142 shutdown_connections 2024-08-03T20:50:20.315 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:20.313+0000 7fdab0800640 1 -- 172.21.15.32:0/4223783142 wait complete. 2024-08-03T20:50:20.316 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:20.315+0000 7fdab0800640 1 Processor -- start 2024-08-03T20:50:20.317 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:20.315+0000 7fdab0800640 1 -- start start 2024-08-03T20:50:20.319 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:20.317+0000 7fdab0800640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fdaa806f110 0x7fdaa81566b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:50:20.319 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:20.317+0000 7fdab0800640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fdaa80afa60 0x7fdaa81f6f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:50:20.319 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:20.317+0000 7fdab0800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdaa8153880 0x7fdaa81e8150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:50:20.319 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:20.317+0000 7fdab0800640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fdaa8160bc0 con 0x7fdaa8153880 2024-08-03T20:50:20.319 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:20.317+0000 7fdab0800640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7fdaa8160a40 con 0x7fdaa80afa60 2024-08-03T20:50:20.319 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:20.317+0000 7fdab0800640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7fdaa8160d40 con 0x7fdaa806f110 2024-08-03T20:50:20.319 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:20.317+0000 7fdaae800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdaa8153880 0x7fdaa81e8150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:50:20.319 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:20.317+0000 7fdaae800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdaa8153880 0x7fdaa81e8150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:60494/0 (socket says 172.21.15.32:60494) 2024-08-03T20:50:20.319 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:20.317+0000 7fdaae800640 1 -- 172.21.15.32:0/1821103091 learned_addr learned my addr 172.21.15.32:0/1821103091 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:50:20.320 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:20.317+0000 7fdaade00640 1 --2- 172.21.15.32:0/1821103091 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fdaa806f110 0x7fdaa81566b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:50:20.320 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:20.317+0000 7fdaad400640 1 --2- 172.21.15.32:0/1821103091 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fdaa80afa60 0x7fdaa81f6f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:50:20.320 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:20.318+0000 7fdaae800640 1 -- 172.21.15.32:0/1821103091 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fdaa806f110 msgr2=0x7fdaa81566b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:50:20.320 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:20.318+0000 7fdaae800640 1 --2- 172.21.15.32:0/1821103091 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fdaa806f110 0x7fdaa81566b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:20.320 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:20.318+0000 7fdaae800640 1 -- 172.21.15.32:0/1821103091 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fdaa80afa60 msgr2=0x7fdaa81f6f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:50:20.320 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:20.318+0000 7fdaae800640 1 --2- 172.21.15.32:0/1821103091 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fdaa80afa60 0x7fdaa81f6f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:20.320 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:20.318+0000 7fdaae800640 1 -- 172.21.15.32:0/1821103091 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdaa81541d0 con 0x7fdaa8153880 2024-08-03T20:50:20.321 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:20.318+0000 7fdaae800640 1 --2- 172.21.15.32:0/1821103091 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdaa8153880 0x7fdaa81e8150 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7fdaa400cb10 tx=0x7fdaa400a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:50:20.321 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:20.318+0000 7fdaa2000640 1 -- 172.21.15.32:0/1821103091 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdaa4016020 con 0x7fdaa8153880 2024-08-03T20:50:20.321 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:20.318+0000 7fdaa2000640 1 -- 172.21.15.32:0/1821103091 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fdaa400ad80 con 0x7fdaa8153880 2024-08-03T20:50:20.321 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:20.318+0000 7fdab0800640 1 -- 172.21.15.32:0/1821103091 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdaa81544b0 con 0x7fdaa8153880 2024-08-03T20:50:20.321 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:20.318+0000 7fdaa2000640 1 -- 172.21.15.32:0/1821103091 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdaa4012680 con 0x7fdaa8153880 2024-08-03T20:50:20.321 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:20.318+0000 7fdab0800640 1 -- 172.21.15.32:0/1821103091 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdaa81f6a20 con 0x7fdaa8153880 2024-08-03T20:50:20.321 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:20.320+0000 7fdab0800640 1 -- 172.21.15.32:0/1821103091 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdaa8153340 con 0x7fdaa8153880 2024-08-03T20:50:20.322 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:20.321+0000 7fdaa2000640 1 -- 172.21.15.32:0/1821103091 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7fdaa4019020 con 0x7fdaa8153880 2024-08-03T20:50:20.323 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:20.322+0000 7fdaa2000640 1 --2- 172.21.15.32:0/1821103091 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fda8c077010 0x7fda8c079520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:50:20.324 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:20.322+0000 7fdaa2000640 1 -- 172.21.15.32:0/1821103091 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(49..49 src has 1..49) ==== 5029+0+0 (secure 0 0 0) 0x7fdaa4098ca0 con 0x7fdaa8153880 2024-08-03T20:50:20.324 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:20.323+0000 7fdaade00640 1 --2- 172.21.15.32:0/1821103091 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fda8c077010 0x7fda8c079520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:50:20.325 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:20.323+0000 7fdaade00640 1 --2- 172.21.15.32:0/1821103091 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fda8c077010 0x7fda8c079520 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7fda9c034000 tx=0x7fda9c004730 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:50:20.328 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:20.327+0000 7fdaa2000640 1 -- 172.21.15.32:0/1821103091 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7fdaa4020020 con 0x7fdaa8153880 2024-08-03T20:50:20.606 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:20.604+0000 7fdab0800640 1 -- 172.21.15.32:0/1821103091 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_damage.TestDamage.test_dentry_first_precommit"]} v 0) -- 0x7fdaa81522c0 con 0x7fdaa8153880 2024-08-03T20:50:21.109 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:21.108+0000 7fdaa2000640 1 -- 172.21.15.32:0/1821103091 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_damage.TestDamage.test_dentry_first_precommit"]}]=0 v161) ==== 145+0+0 (secure 0 0 0) 0x7fdaa4061a50 con 0x7fdaa8153880 2024-08-03T20:50:21.112 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:21.110+0000 7fdab0800640 1 -- 172.21.15.32:0/1821103091 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fda8c077010 msgr2=0x7fda8c079520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:50:21.112 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:21.110+0000 7fdab0800640 1 --2- 172.21.15.32:0/1821103091 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fda8c077010 0x7fda8c079520 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7fda9c034000 tx=0x7fda9c004730 comp rx=0 tx=0).stop 2024-08-03T20:50:21.112 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:21.110+0000 7fdab0800640 1 -- 172.21.15.32:0/1821103091 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdaa8153880 msgr2=0x7fdaa81e8150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:50:21.113 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:21.110+0000 7fdab0800640 1 --2- 172.21.15.32:0/1821103091 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdaa8153880 0x7fdaa81e8150 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7fdaa400cb10 tx=0x7fdaa400a060 comp rx=0 tx=0).stop 2024-08-03T20:50:21.113 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:21.110+0000 7fdab0800640 1 -- 172.21.15.32:0/1821103091 shutdown_connections 2024-08-03T20:50:21.113 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:21.110+0000 7fdab0800640 1 --2- 172.21.15.32:0/1821103091 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fda8c077010 0x7fda8c079520 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:21.113 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:21.110+0000 7fdab0800640 1 --2- 172.21.15.32:0/1821103091 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fdaa80afa60 0x7fdaa81f6f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:21.113 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:21.110+0000 7fdab0800640 1 --2- 172.21.15.32:0/1821103091 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdaa8153880 0x7fdaa81e8150 unknown :-1 s=CLOSED pgs=534 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:21.113 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:21.110+0000 7fdab0800640 1 --2- 172.21.15.32:0/1821103091 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fdaa806f110 0x7fdaa81566b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:21.113 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:21.110+0000 7fdab0800640 1 -- 172.21.15.32:0/1821103091 >> 172.21.15.32:0/1821103091 conn(0x7fdaa80cd1e0 msgr2=0x7fdaa8152ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:50:21.113 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:21.111+0000 7fdab0800640 1 -- 172.21.15.32:0/1821103091 shutdown_connections 2024-08-03T20:50:21.113 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:21.111+0000 7fdab0800640 1 -- 172.21.15.32:0/1821103091 wait complete. 2024-08-03T20:50:21.135 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config dump 2024-08-03T20:50:21.363 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:21.361+0000 7f6afa800640 1 -- 172.21.15.32:0/2057029599 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6af40cffe0 msgr2=0x7f6af40b3e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:50:21.363 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:21.361+0000 7f6afa800640 1 --2- 172.21.15.32:0/2057029599 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6af40cffe0 0x7f6af40b3e90 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7f6ae0005fd0 tx=0x7f6ae002a5d0 comp rx=0 tx=0).stop 2024-08-03T20:50:21.363 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:21.361+0000 7f6afa800640 1 -- 172.21.15.32:0/2057029599 shutdown_connections 2024-08-03T20:50:21.363 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:21.361+0000 7f6afa800640 1 --2- 172.21.15.32:0/2057029599 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f6af40b43d0 0x7f6af40cf510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:21.363 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:21.361+0000 7f6afa800640 1 --2- 172.21.15.32:0/2057029599 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6af40cffe0 0x7f6af40b3e90 unknown :-1 s=CLOSED pgs=535 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:21.363 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:21.361+0000 7f6afa800640 1 -- 172.21.15.32:0/2057029599 >> 172.21.15.32:0/2057029599 conn(0x7f6af408d7a0 msgr2=0x7f6af409d3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:50:21.363 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:21.361+0000 7f6afa800640 1 -- 172.21.15.32:0/2057029599 shutdown_connections 2024-08-03T20:50:21.363 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:21.361+0000 7f6afa800640 1 -- 172.21.15.32:0/2057029599 wait complete. 2024-08-03T20:50:21.364 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:21.363+0000 7f6afa800640 1 Processor -- start 2024-08-03T20:50:21.364 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:21.363+0000 7f6afa800640 1 -- start start 2024-08-03T20:50:21.366 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:21.365+0000 7f6afa800640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f6af40b43d0 0x7f6af40ca850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:50:21.366 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:21.365+0000 7f6afa800640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f6af40cffe0 0x7f6af415e370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:50:21.366 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:21.365+0000 7f6afa800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6af40d26e0 0x7f6af40c7e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:50:21.367 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:21.365+0000 7f6afa800640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f6af40d4100 con 0x7f6af40d26e0 2024-08-03T20:50:21.367 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:21.365+0000 7f6afa800640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f6af40d3f80 con 0x7f6af40cffe0 2024-08-03T20:50:21.367 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:21.365+0000 7f6afa800640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7f6af40d4280 con 0x7f6af40b43d0 2024-08-03T20:50:21.367 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:21.365+0000 7f6af9e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6af40d26e0 0x7f6af40c7e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:50:21.367 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:21.365+0000 7f6af9e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6af40d26e0 0x7f6af40c7e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:60500/0 (socket says 172.21.15.32:60500) 2024-08-03T20:50:21.367 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:21.365+0000 7f6af9e00640 1 -- 172.21.15.32:0/823523626 learned_addr learned my addr 172.21.15.32:0/823523626 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:50:21.367 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:21.365+0000 7f6af8a00640 1 --2- 172.21.15.32:0/823523626 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f6af40cffe0 0x7f6af415e370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:50:21.367 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:21.365+0000 7f6af9400640 1 --2- 172.21.15.32:0/823523626 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f6af40b43d0 0x7f6af40ca850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:50:21.368 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:21.365+0000 7f6af9e00640 1 -- 172.21.15.32:0/823523626 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f6af40b43d0 msgr2=0x7f6af40ca850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:50:21.368 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:21.365+0000 7f6af9e00640 1 --2- 172.21.15.32:0/823523626 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f6af40b43d0 0x7f6af40ca850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:21.368 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:21.365+0000 7f6af9e00640 1 -- 172.21.15.32:0/823523626 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f6af40cffe0 msgr2=0x7f6af415e370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:50:21.368 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:21.365+0000 7f6af9e00640 1 --2- 172.21.15.32:0/823523626 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f6af40cffe0 0x7f6af415e370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:21.368 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:21.365+0000 7f6af9e00640 1 -- 172.21.15.32:0/823523626 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6af40cb1b0 con 0x7f6af40d26e0 2024-08-03T20:50:21.368 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:21.366+0000 7f6af9e00640 1 --2- 172.21.15.32:0/823523626 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6af40d26e0 0x7f6af40c7e80 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7f6aec00ea70 tx=0x7f6aec00ef30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:50:21.369 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:21.366+0000 7f6af1600640 1 -- 172.21.15.32:0/823523626 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6aec00ce40 con 0x7f6af40d26e0 2024-08-03T20:50:21.369 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:21.366+0000 7f6af1600640 1 -- 172.21.15.32:0/823523626 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f6aec004c10 con 0x7f6af40d26e0 2024-08-03T20:50:21.369 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:21.366+0000 7f6afa800640 1 -- 172.21.15.32:0/823523626 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6af40cda80 con 0x7f6af40d26e0 2024-08-03T20:50:21.369 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:21.366+0000 7f6af1600640 1 -- 172.21.15.32:0/823523626 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6aec005020 con 0x7f6af40d26e0 2024-08-03T20:50:21.369 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:21.366+0000 7f6afa800640 1 -- 172.21.15.32:0/823523626 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6af40cb470 con 0x7f6af40d26e0 2024-08-03T20:50:21.369 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:21.367+0000 7f6afa800640 1 -- 172.21.15.32:0/823523626 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6af40d46e0 con 0x7f6af40d26e0 2024-08-03T20:50:21.370 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:21.369+0000 7f6af1600640 1 -- 172.21.15.32:0/823523626 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f6aec0051c0 con 0x7f6af40d26e0 2024-08-03T20:50:21.371 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:21.369+0000 7f6af1600640 1 --2- 172.21.15.32:0/823523626 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f6ad4077040 0x7f6ad4079550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:50:21.371 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:21.370+0000 7f6af1600640 1 -- 172.21.15.32:0/823523626 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(49..49 src has 1..49) ==== 5029+0+0 (secure 0 0 0) 0x7f6aec0992a0 con 0x7f6af40d26e0 2024-08-03T20:50:21.371 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:21.370+0000 7f6af9400640 1 --2- 172.21.15.32:0/823523626 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f6ad4077040 0x7f6ad4079550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:50:21.372 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:21.371+0000 7f6af9400640 1 --2- 172.21.15.32:0/823523626 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f6ad4077040 0x7f6ad4079550 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f6ae8001720 tx=0x7f6ae8009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:50:21.375 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:21.374+0000 7f6af1600640 1 -- 172.21.15.32:0/823523626 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7f6aec0620e0 con 0x7f6af40d26e0 2024-08-03T20:50:21.578 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:21.576+0000 7f6afa800640 1 -- 172.21.15.32:0/823523626 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "config dump"} v 0) -- 0x7f6af40cd810 con 0x7f6af40d26e0 2024-08-03T20:50:21.578 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:21.577+0000 7f6af1600640 1 -- 172.21.15.32:0/823523626 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "config dump"}]=0 v34) ==== 59+0+1998 (secure 0 0 0) 0x7f6aec066630 con 0x7f6af40d26e0 2024-08-03T20:50:21.578 INFO:teuthology.orchestra.run.smithi032.stdout:WHO MASK LEVEL OPTION VALUE RO 2024-08-03T20:50:21.579 INFO:teuthology.orchestra.run.smithi032.stdout:mon advanced mon_op_complaint_time 120 2024-08-03T20:50:21.579 INFO:teuthology.orchestra.run.smithi032.stdout:mon dev mon_warn_on_pool_no_app false 2024-08-03T20:50:21.579 INFO:teuthology.orchestra.run.smithi032.stdout:osd advanced osd_op_complaint_time 180.000000 2024-08-03T20:50:21.579 INFO:teuthology.orchestra.run.smithi032.stdout:osd.0 basic osd_mclock_max_capacity_iops_ssd 4313.565729 2024-08-03T20:50:21.579 INFO:teuthology.orchestra.run.smithi032.stdout:osd.1 basic osd_mclock_max_capacity_iops_ssd 4272.517979 2024-08-03T20:50:21.579 INFO:teuthology.orchestra.run.smithi032.stdout:osd.2 basic osd_mclock_max_capacity_iops_ssd 4131.201546 2024-08-03T20:50:21.579 INFO:teuthology.orchestra.run.smithi032.stdout:osd.3 basic osd_mclock_max_capacity_iops_ssd 4155.765914 2024-08-03T20:50:21.579 INFO:teuthology.orchestra.run.smithi032.stdout:osd.4 basic osd_mclock_max_capacity_iops_ssd 5744.667532 2024-08-03T20:50:21.579 INFO:teuthology.orchestra.run.smithi032.stdout:osd.5 basic osd_mclock_max_capacity_iops_ssd 3764.678226 2024-08-03T20:50:21.579 INFO:teuthology.orchestra.run.smithi032.stdout:osd.6 basic osd_mclock_max_capacity_iops_ssd 3927.989098 2024-08-03T20:50:21.579 INFO:teuthology.orchestra.run.smithi032.stdout:osd.7 basic osd_mclock_max_capacity_iops_ssd 3760.968870 2024-08-03T20:50:21.579 INFO:teuthology.orchestra.run.smithi032.stdout:mds advanced debug_mds 20/20 2024-08-03T20:50:21.580 INFO:teuthology.orchestra.run.smithi032.stdout:mds advanced debug_mds_balancer 20/20 2024-08-03T20:50:21.580 INFO:teuthology.orchestra.run.smithi032.stdout:mds advanced debug_ms 2/2 2024-08-03T20:50:21.580 INFO:teuthology.orchestra.run.smithi032.stdout:mds dev mds_debug_frag true 2024-08-03T20:50:21.580 INFO:teuthology.orchestra.run.smithi032.stdout:mds dev mds_debug_scatterstat true 2024-08-03T20:50:21.580 INFO:teuthology.orchestra.run.smithi032.stdout:mds advanced mds_op_complaint_time 180.000000 2024-08-03T20:50:21.580 INFO:teuthology.orchestra.run.smithi032.stdout:mds dev mds_verify_scatter true 2024-08-03T20:50:21.580 INFO:teuthology.orchestra.run.smithi032.stdout:mds advanced osd_op_complaint_time 180.000000 2024-08-03T20:50:21.580 INFO:teuthology.orchestra.run.smithi032.stdout:mds advanced rados_mon_op_timeout 900 2024-08-03T20:50:21.580 INFO:teuthology.orchestra.run.smithi032.stdout:mds advanced rados_osd_op_timeout 900 2024-08-03T20:50:21.580 INFO:teuthology.orchestra.run.smithi032.stdout:client advanced client_mount_timeout 600 2024-08-03T20:50:21.580 INFO:teuthology.orchestra.run.smithi032.stdout:client advanced debug_client 20/20 2024-08-03T20:50:21.580 INFO:teuthology.orchestra.run.smithi032.stdout:client advanced debug_ms 1/1 2024-08-03T20:50:21.580 INFO:teuthology.orchestra.run.smithi032.stdout:client advanced rados_mon_op_timeout 900 2024-08-03T20:50:21.580 INFO:teuthology.orchestra.run.smithi032.stdout:client advanced rados_osd_op_timeout 900 2024-08-03T20:50:21.581 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:21.579+0000 7f6afa800640 1 -- 172.21.15.32:0/823523626 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f6ad4077040 msgr2=0x7f6ad4079550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:50:21.581 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:21.579+0000 7f6afa800640 1 --2- 172.21.15.32:0/823523626 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f6ad4077040 0x7f6ad4079550 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f6ae8001720 tx=0x7f6ae8009040 comp rx=0 tx=0).stop 2024-08-03T20:50:21.582 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:21.579+0000 7f6afa800640 1 -- 172.21.15.32:0/823523626 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6af40d26e0 msgr2=0x7f6af40c7e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:50:21.582 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:21.579+0000 7f6afa800640 1 --2- 172.21.15.32:0/823523626 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6af40d26e0 0x7f6af40c7e80 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7f6aec00ea70 tx=0x7f6aec00ef30 comp rx=0 tx=0).stop 2024-08-03T20:50:21.582 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:21.579+0000 7f6afa800640 1 -- 172.21.15.32:0/823523626 shutdown_connections 2024-08-03T20:50:21.582 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:21.579+0000 7f6afa800640 1 --2- 172.21.15.32:0/823523626 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f6ad4077040 0x7f6ad4079550 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:21.582 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:21.579+0000 7f6afa800640 1 --2- 172.21.15.32:0/823523626 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f6af40cffe0 0x7f6af415e370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:21.582 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:21.579+0000 7f6afa800640 1 --2- 172.21.15.32:0/823523626 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6af40d26e0 0x7f6af40c7e80 unknown :-1 s=CLOSED pgs=536 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:21.582 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:21.579+0000 7f6afa800640 1 --2- 172.21.15.32:0/823523626 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f6af40b43d0 0x7f6af40ca850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:21.582 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:21.579+0000 7f6afa800640 1 -- 172.21.15.32:0/823523626 >> 172.21.15.32:0/823523626 conn(0x7f6af408d7a0 msgr2=0x7f6af406ebe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:50:21.582 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:21.579+0000 7f6afa800640 1 -- 172.21.15.32:0/823523626 shutdown_connections 2024-08-03T20:50:21.582 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:21.579+0000 7f6afa800640 1 -- 172.21.15.32:0/823523626 wait complete. 2024-08-03T20:50:21.603 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-08-03T20:50:21.834 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:21.832+0000 7f2807e00640 1 -- 172.21.15.32:0/2211756813 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f28080cfc50 msgr2=0x7f28080d26a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:50:21.835 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:21.832+0000 7f2807e00640 1 --2- 172.21.15.32:0/2211756813 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f28080cfc50 0x7f28080d26a0 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7f28080d5080 tx=0x7f27f002a540 comp rx=0 tx=0).stop 2024-08-03T20:50:21.835 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:21.833+0000 7f2807e00640 1 -- 172.21.15.32:0/2211756813 shutdown_connections 2024-08-03T20:50:21.835 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:21.833+0000 7f2807e00640 1 --2- 172.21.15.32:0/2211756813 >> v2:172.21.15.118:3300/0 conn(0x7f28080d2be0 0x7f28080d0950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:21.835 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:21.833+0000 7f2807e00640 1 --2- 172.21.15.32:0/2211756813 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f28080cfc50 0x7f28080d26a0 unknown :-1 s=CLOSED pgs=538 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:21.835 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:21.833+0000 7f2807e00640 1 -- 172.21.15.32:0/2211756813 >> 172.21.15.32:0/2211756813 conn(0x7f28080d4130 msgr2=0x7f28080d4540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:50:21.835 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:21.833+0000 7f2807e00640 1 -- 172.21.15.32:0/2211756813 shutdown_connections 2024-08-03T20:50:21.835 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:21.833+0000 7f2807e00640 1 -- 172.21.15.32:0/2211756813 wait complete. 2024-08-03T20:50:21.836 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:21.835+0000 7f2807e00640 1 Processor -- start 2024-08-03T20:50:21.836 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:21.835+0000 7f2807e00640 1 -- start start 2024-08-03T20:50:21.839 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:21.836+0000 7f2807e00640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f28080b92b0 0x7f2808166290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:50:21.839 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:21.836+0000 7f2807e00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f28080cfc50 0x7f2808160650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:50:21.839 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:21.836+0000 7f2807e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f28080d2be0 0x7f28080b81f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:50:21.839 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:21.836+0000 7f2807e00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f28080d1010 con 0x7f28080d2be0 2024-08-03T20:50:21.839 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:21.836+0000 7f2807e00640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f28080d0e90 con 0x7f28080cfc50 2024-08-03T20:50:21.840 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:21.836+0000 7f2807e00640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7f28080d1190 con 0x7f28080b92b0 2024-08-03T20:50:21.840 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:21.837+0000 7f2807400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f28080d2be0 0x7f28080b81f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:50:21.840 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:21.837+0000 7f2807400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f28080d2be0 0x7f28080b81f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:60512/0 (socket says 172.21.15.32:60512) 2024-08-03T20:50:21.840 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:21.837+0000 7f2807400640 1 -- 172.21.15.32:0/3705376409 learned_addr learned my addr 172.21.15.32:0/3705376409 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:50:21.840 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:21.837+0000 7f2806a00640 1 --2- 172.21.15.32:0/3705376409 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f28080b92b0 0x7f2808166290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:50:21.840 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:21.837+0000 7f27ffe00640 1 --2- 172.21.15.32:0/3705376409 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f28080cfc50 0x7f2808160650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:50:21.840 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:21.837+0000 7f2807400640 1 -- 172.21.15.32:0/3705376409 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f28080b92b0 msgr2=0x7f2808166290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:50:21.840 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:21.837+0000 7f2807400640 1 --2- 172.21.15.32:0/3705376409 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f28080b92b0 0x7f2808166290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:21.840 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:21.837+0000 7f2807400640 1 -- 172.21.15.32:0/3705376409 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f28080cfc50 msgr2=0x7f2808160650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:50:21.840 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:21.837+0000 7f2807400640 1 --2- 172.21.15.32:0/3705376409 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f28080cfc50 0x7f2808160650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:21.840 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:21.837+0000 7f2807400640 1 -- 172.21.15.32:0/3705376409 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f28080cd710 con 0x7f28080d2be0 2024-08-03T20:50:21.840 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:21.837+0000 7f27ffe00640 1 --2- 172.21.15.32:0/3705376409 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f28080cfc50 0x7f2808160650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-03T20:50:21.840 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:21.837+0000 7f2807400640 1 --2- 172.21.15.32:0/3705376409 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f28080d2be0 0x7f28080b81f0 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7f280000e990 tx=0x7f280000ee50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:50:21.840 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:21.838+0000 7f27fea00640 1 -- 172.21.15.32:0/3705376409 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2800004d20 con 0x7f28080d2be0 2024-08-03T20:50:21.841 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:21.838+0000 7f27fea00640 1 -- 172.21.15.32:0/3705376409 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f280000caa0 con 0x7f28080d2be0 2024-08-03T20:50:21.841 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:21.838+0000 7f27fea00640 1 -- 172.21.15.32:0/3705376409 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2800005020 con 0x7f28080d2be0 2024-08-03T20:50:21.841 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:21.838+0000 7f2807e00640 1 -- 172.21.15.32:0/3705376409 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f28080cd9f0 con 0x7f28080d2be0 2024-08-03T20:50:21.841 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:21.838+0000 7f2807e00640 1 -- 172.21.15.32:0/3705376409 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2808160b90 con 0x7f28080d2be0 2024-08-03T20:50:21.841 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:21.839+0000 7f2807e00640 1 -- 172.21.15.32:0/3705376409 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f28080b56f0 con 0x7f28080d2be0 2024-08-03T20:50:21.842 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:21.840+0000 7f27fea00640 1 -- 172.21.15.32:0/3705376409 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f28000051c0 con 0x7f28080d2be0 2024-08-03T20:50:21.843 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:21.841+0000 7f27fea00640 1 --2- 172.21.15.32:0/3705376409 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f27e4077040 0x7f27e4079550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:50:21.843 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:21.841+0000 7f27fea00640 1 -- 172.21.15.32:0/3705376409 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(49..49 src has 1..49) ==== 5029+0+0 (secure 0 0 0) 0x7f2800098e30 con 0x7f28080d2be0 2024-08-03T20:50:21.843 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:21.842+0000 7f2806a00640 1 --2- 172.21.15.32:0/3705376409 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f27e4077040 0x7f27e4079550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:50:21.843 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:21.842+0000 7f2806a00640 1 --2- 172.21.15.32:0/3705376409 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f27e4077040 0x7f27e4079550 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f27f80015d0 tx=0x7f27f8009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:50:21.847 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:21.845+0000 7f27fea00640 1 -- 172.21.15.32:0/3705376409 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7f28080b56f0 con 0x7f28080d2be0 2024-08-03T20:50:22.054 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:22.053+0000 7f2807e00640 1 -- 172.21.15.32:0/3705376409 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) -- 0x7f28080d3590 con 0x7f28080d2be0 2024-08-03T20:50:22.058 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:22.057+0000 7f27fea00640 1 -- 172.21.15.32:0/3705376409 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v35)=0 v35) ==== 122+0+0 (secure 0 0 0) 0x7f28080d3590 con 0x7f28080d2be0 2024-08-03T20:50:22.061 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:22.059+0000 7f2807e00640 1 -- 172.21.15.32:0/3705376409 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f27e4077040 msgr2=0x7f27e4079550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:50:22.061 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:22.059+0000 7f2807e00640 1 --2- 172.21.15.32:0/3705376409 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f27e4077040 0x7f27e4079550 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f27f80015d0 tx=0x7f27f8009040 comp rx=0 tx=0).stop 2024-08-03T20:50:22.061 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:22.059+0000 7f2807e00640 1 -- 172.21.15.32:0/3705376409 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f28080d2be0 msgr2=0x7f28080b81f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:50:22.061 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:22.059+0000 7f2807e00640 1 --2- 172.21.15.32:0/3705376409 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f28080d2be0 0x7f28080b81f0 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7f280000e990 tx=0x7f280000ee50 comp rx=0 tx=0).stop 2024-08-03T20:50:22.061 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:22.059+0000 7f2807e00640 1 -- 172.21.15.32:0/3705376409 shutdown_connections 2024-08-03T20:50:22.061 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:22.059+0000 7f2807e00640 1 --2- 172.21.15.32:0/3705376409 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f27e4077040 0x7f27e4079550 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:22.061 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:22.059+0000 7f2807e00640 1 --2- 172.21.15.32:0/3705376409 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f28080cfc50 0x7f2808160650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:22.061 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:22.059+0000 7f2807e00640 1 --2- 172.21.15.32:0/3705376409 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f28080d2be0 0x7f28080b81f0 unknown :-1 s=CLOSED pgs=539 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:22.061 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:22.059+0000 7f2807e00640 1 --2- 172.21.15.32:0/3705376409 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f28080b92b0 0x7f2808166290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:22.061 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:22.059+0000 7f2807e00640 1 -- 172.21.15.32:0/3705376409 >> 172.21.15.32:0/3705376409 conn(0x7f28080d4130 msgr2=0x7f28080b2cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:50:22.061 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:22.059+0000 7f2807e00640 1 -- 172.21.15.32:0/3705376409 shutdown_connections 2024-08-03T20:50:22.062 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:22.059+0000 7f2807e00640 1 -- 172.21.15.32:0/3705376409 wait complete. 2024-08-03T20:50:22.083 INFO:teuthology.nuke:Clearing teuthology firewall rules... 2024-08-03T20:50:22.083 DEBUG:teuthology.orchestra.run.smithi032:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-08-03T20:50:22.162 DEBUG:teuthology.orchestra.run.smithi118:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-08-03T20:50:22.258 INFO:teuthology.nuke:Cleared teuthology firewall rules. 2024-08-03T20:50:22.258 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-08-03T20:50:22.258 DEBUG:teuthology.orchestra.run.smithi032:> dd if=/proc/self/mounts of=/dev/stdout 2024-08-03T20:50:22.294 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053453,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6499208k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 none /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249604k,nr_inodes=812401,mode=700,uid=1000,gid=1316 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6499208k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-08-03T20:50:22.294 DEBUG:teuthology.orchestra.run.smithi118:> set -ex 2024-08-03T20:50:22.295 DEBUG:teuthology.orchestra.run.smithi118:> dd if=/proc/self/mounts of=/dev/stdout 2024-08-03T20:50:22.332 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052663,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497944k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248972k,nr_inodes=812243,mode=700,uid=1000,gid=1316 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497944k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-08-03T20:50:22.332 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-08-03T20:50:22.332 DEBUG:teuthology.orchestra.run.smithi032:> dd if=/proc/self/mounts of=/dev/stdout 2024-08-03T20:50:22.369 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053453,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6499208k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 none /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249604k,nr_inodes=812401,mode=700,uid=1000,gid=1316 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6499208k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-08-03T20:50:22.369 DEBUG:teuthology.orchestra.run.smithi118:> set -ex 2024-08-03T20:50:22.370 DEBUG:teuthology.orchestra.run.smithi118:> dd if=/proc/self/mounts of=/dev/stdout 2024-08-03T20:50:22.408 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052663,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497944k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248972k,nr_inodes=812243,mode=700,uid=1000,gid=1316 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497944k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-08-03T20:50:22.408 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-08-03T20:50:22.408 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-08-03T20:50:22.637 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:22.635+0000 7f2431800640 1 -- 172.21.15.32:0/1716986747 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f242c06e930 msgr2=0x7f242c144f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:50:22.638 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:22.635+0000 7f2431800640 1 --2- 172.21.15.32:0/1716986747 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f242c06e930 0x7f242c144f90 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f241c009860 tx=0x7f241c02a5a0 comp rx=0 tx=0).stop 2024-08-03T20:50:22.638 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:22.636+0000 7f2431800640 1 -- 172.21.15.32:0/1716986747 shutdown_connections 2024-08-03T20:50:22.638 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:22.636+0000 7f2431800640 1 --2- 172.21.15.32:0/1716986747 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f242c06e930 0x7f242c144f90 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:22.638 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:22.636+0000 7f2431800640 1 -- 172.21.15.32:0/1716986747 >> 172.21.15.32:0/1716986747 conn(0x7f242c0b1bd0 msgr2=0x7f242c0b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:50:22.638 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:22.636+0000 7f2431800640 1 -- 172.21.15.32:0/1716986747 shutdown_connections 2024-08-03T20:50:22.638 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:22.636+0000 7f2431800640 1 -- 172.21.15.32:0/1716986747 wait complete. 2024-08-03T20:50:22.639 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:22.638+0000 7f2431800640 1 Processor -- start 2024-08-03T20:50:22.640 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:22.638+0000 7f2431800640 1 -- start start 2024-08-03T20:50:22.643 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:22.639+0000 7f2431800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f242c06e930 0x7f242c15ba60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:50:22.643 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:22.639+0000 7f2431800640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f242c073e60 0x7f242c1f6370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:50:22.644 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:22.639+0000 7f2431800640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f242c15fff0 0x7f242c1f68b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:50:22.644 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:22.639+0000 7f2431800640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f242c0cf840 con 0x7f242c06e930 2024-08-03T20:50:22.644 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:22.639+0000 7f2431800640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f242c145f30 con 0x7f242c073e60 2024-08-03T20:50:22.644 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:22.640+0000 7f2431800640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7f242c0cf9c0 con 0x7f242c15fff0 2024-08-03T20:50:22.644 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:22.640+0000 7f242aa00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f242c06e930 0x7f242c15ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:50:22.645 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:22.640+0000 7f242aa00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f242c06e930 0x7f242c15ba60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:60520/0 (socket says 172.21.15.32:60520) 2024-08-03T20:50:22.645 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:22.640+0000 7f242aa00640 1 -- 172.21.15.32:0/1942873640 learned_addr learned my addr 172.21.15.32:0/1942873640 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:50:22.645 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:22.640+0000 7f242aa00640 1 -- 172.21.15.32:0/1942873640 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f242c15fff0 msgr2=0x7f242c1f68b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:50:22.645 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:22.640+0000 7f242aa00640 1 --2- 172.21.15.32:0/1942873640 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f242c15fff0 0x7f242c1f68b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:22.645 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:22.640+0000 7f242aa00640 1 -- 172.21.15.32:0/1942873640 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f242c073e60 msgr2=0x7f242c1f6370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:50:22.646 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:22.640+0000 7f242aa00640 1 --2- 172.21.15.32:0/1942873640 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f242c073e60 0x7f242c1f6370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:22.646 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:22.640+0000 7f242aa00640 1 -- 172.21.15.32:0/1942873640 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f242c142f10 con 0x7f242c06e930 2024-08-03T20:50:22.646 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:22.640+0000 7f242aa00640 1 --2- 172.21.15.32:0/1942873640 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f242c06e930 0x7f242c15ba60 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7f241c02ab90 tx=0x7f241c033ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:50:22.646 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:22.641+0000 7f2422a00640 1 -- 172.21.15.32:0/1942873640 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f241c031b50 con 0x7f242c06e930 2024-08-03T20:50:22.646 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:22.641+0000 7f2422a00640 1 -- 172.21.15.32:0/1942873640 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f241c031de0 con 0x7f242c06e930 2024-08-03T20:50:22.647 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:22.641+0000 7f2422a00640 1 -- 172.21.15.32:0/1942873640 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f241c033330 con 0x7f242c06e930 2024-08-03T20:50:22.647 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:22.641+0000 7f2431800640 1 -- 172.21.15.32:0/1942873640 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f242c1f6df0 con 0x7f242c06e930 2024-08-03T20:50:22.647 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:22.641+0000 7f2431800640 1 -- 172.21.15.32:0/1942873640 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f242c143470 con 0x7f242c06e930 2024-08-03T20:50:22.647 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:22.642+0000 7f2431800640 1 -- 172.21.15.32:0/1942873640 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f242c160db0 con 0x7f242c06e930 2024-08-03T20:50:22.649 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:22.643+0000 7f2422a00640 1 -- 172.21.15.32:0/1942873640 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f241c0334d0 con 0x7f242c06e930 2024-08-03T20:50:22.649 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:22.644+0000 7f2422a00640 1 --2- 172.21.15.32:0/1942873640 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f2408077010 0x7f2408079520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:50:22.650 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:22.644+0000 7f2422a00640 1 -- 172.21.15.32:0/1942873640 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(49..49 src has 1..49) ==== 5029+0+0 (secure 0 0 0) 0x7f241c0b98a0 con 0x7f242c06e930 2024-08-03T20:50:22.650 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:22.645+0000 7f2423e00640 1 --2- 172.21.15.32:0/1942873640 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f2408077010 0x7f2408079520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:50:22.650 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:22.645+0000 7f2423e00640 1 --2- 172.21.15.32:0/1942873640 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f2408077010 0x7f2408079520 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f24140014d0 tx=0x7f2414007970 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:50:22.650 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:22.648+0000 7f2422a00640 1 -- 172.21.15.32:0/1942873640 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7f242c142f10 con 0x7f242c06e930 2024-08-03T20:50:22.892 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:22.890+0000 7f2431800640 1 -- 172.21.15.32:0/1942873640 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f242c145800 con 0x7f242c06e930 2024-08-03T20:50:22.893 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:22.891+0000 7f2422a00640 1 -- 172.21.15.32:0/1942873640 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 44 v44) ==== 94+0+3917 (secure 0 0 0) 0x7f241c03f030 con 0x7f242c06e930 2024-08-03T20:50:22.893 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-03T20:50:22.893 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":44,"btime":"2024-08-03T20:50:14:599376+0000","default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4997,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6837/2688500839","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6836","nonce":2688500839},{"type":"v1","addr":"172.21.15.118:6837","nonce":2688500839}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":30},{"gid":5266,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6835/2381075275","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6834","nonce":2381075275},{"type":"v1","addr":"172.21.15.118:6835","nonce":2381075275}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":28},{"gid":5421,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6833/1058332640","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":1058332640},{"type":"v1","addr":"172.21.15.32:6833","nonce":1058332640}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":29},{"gid":5748,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6835/1736951062","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1736951062},{"type":"v1","addr":"172.21.15.32:6835","nonce":1736951062}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":44}],"filesystems":[]} 2024-08-03T20:50:22.893 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 44 2024-08-03T20:50:22.895 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:22.893+0000 7f2431800640 1 -- 172.21.15.32:0/1942873640 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f2408077010 msgr2=0x7f2408079520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:50:22.895 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:22.893+0000 7f2431800640 1 --2- 172.21.15.32:0/1942873640 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f2408077010 0x7f2408079520 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f24140014d0 tx=0x7f2414007970 comp rx=0 tx=0).stop 2024-08-03T20:50:22.895 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:22.893+0000 7f2431800640 1 -- 172.21.15.32:0/1942873640 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f242c06e930 msgr2=0x7f242c15ba60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:50:22.895 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:22.893+0000 7f2431800640 1 --2- 172.21.15.32:0/1942873640 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f242c06e930 0x7f242c15ba60 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7f241c02ab90 tx=0x7f241c033ba0 comp rx=0 tx=0).stop 2024-08-03T20:50:22.895 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:22.893+0000 7f2431800640 1 -- 172.21.15.32:0/1942873640 shutdown_connections 2024-08-03T20:50:22.895 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:22.893+0000 7f2431800640 1 --2- 172.21.15.32:0/1942873640 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f2408077010 0x7f2408079520 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:22.895 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:22.893+0000 7f2431800640 1 --2- 172.21.15.32:0/1942873640 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f242c073e60 0x7f242c1f6370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:22.896 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:22.893+0000 7f2431800640 1 --2- 172.21.15.32:0/1942873640 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f242c15fff0 0x7f242c1f68b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:22.896 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:22.893+0000 7f2431800640 1 --2- 172.21.15.32:0/1942873640 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f242c06e930 0x7f242c15ba60 unknown :-1 s=CLOSED pgs=541 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:22.896 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:22.893+0000 7f2431800640 1 -- 172.21.15.32:0/1942873640 >> 172.21.15.32:0/1942873640 conn(0x7f242c0b1bd0 msgr2=0x7f242c074270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:50:22.896 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:22.893+0000 7f2431800640 1 -- 172.21.15.32:0/1942873640 shutdown_connections 2024-08-03T20:50:22.896 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:22.893+0000 7f2431800640 1 -- 172.21.15.32:0/1942873640 wait complete. 2024-08-03T20:50:22.916 INFO:tasks.ceph.mds.a:Restarting daemon 2024-08-03T20:50:22.916 INFO:tasks.ceph.mds.a:Stopping old one... 2024-08-03T20:50:22.917 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-08-03T20:50:22.917 INFO:teuthology.orchestra.run:waiting for 300 2024-08-03T20:50:22.917 INFO:tasks.ceph.mds.c:Restarting daemon 2024-08-03T20:50:22.917 INFO:tasks.ceph.mds.c:Stopping old one... 2024-08-03T20:50:22.917 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-08-03T20:50:22.917 INFO:teuthology.orchestra.run:waiting for 300 2024-08-03T20:50:22.918 INFO:tasks.ceph.mds.b:Restarting daemon 2024-08-03T20:50:22.918 INFO:tasks.ceph.mds.b:Stopping old one... 2024-08-03T20:50:22.918 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-08-03T20:50:22.918 INFO:teuthology.orchestra.run:waiting for 300 2024-08-03T20:50:22.918 INFO:tasks.ceph.mds.d:Restarting daemon 2024-08-03T20:50:22.918 INFO:tasks.ceph.mds.d:Stopping old one... 2024-08-03T20:50:22.918 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-08-03T20:50:22.919 INFO:teuthology.orchestra.run:waiting for 300 2024-08-03T20:50:22.939 INFO:tasks.ceph.mds.a:Stopped 2024-08-03T20:50:22.939 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-08-03T20:50:22.940 INFO:tasks.ceph.mds.c:Stopped 2024-08-03T20:50:22.940 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-08-03T20:50:22.953 INFO:tasks.ceph.mds.b:Stopped 2024-08-03T20:50:22.953 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-08-03T20:50:22.954 INFO:tasks.ceph.mds.d:Stopped 2024-08-03T20:50:22.954 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-08-03T20:50:22.984 INFO:tasks.ceph.mds.a:Started 2024-08-03T20:50:22.984 DEBUG:teuthology.parallel:result is None 2024-08-03T20:50:22.986 INFO:tasks.ceph.mds.c:Started 2024-08-03T20:50:22.987 DEBUG:teuthology.parallel:result is None 2024-08-03T20:50:22.997 INFO:tasks.ceph.mds.b:Started 2024-08-03T20:50:22.997 DEBUG:teuthology.parallel:result is None 2024-08-03T20:50:22.999 INFO:tasks.ceph.mds.d:Started 2024-08-03T20:50:22.999 DEBUG:teuthology.parallel:result is None 2024-08-03T20:50:22.999 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-08-03T20:50:23.184 INFO:tasks.ceph.mds.a.smithi032.stdout:starting mds.a at 2024-08-03T20:50:23.186 INFO:tasks.ceph.mds.c.smithi032.stdout:starting mds.c at 2024-08-03T20:50:23.196 INFO:tasks.ceph.mds.d.smithi118.stdout:starting mds.d at 2024-08-03T20:50:23.197 INFO:tasks.ceph.mds.b.smithi118.stdout:starting mds.b at 2024-08-03T20:50:23.241 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:23.239+0000 7fe94a400640 1 -- 172.21.15.32:0/2734126091 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fe9440cfc90 msgr2=0x7fe9440d00a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:50:23.242 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:23.239+0000 7fe94a400640 1 --2- 172.21.15.32:0/2734126091 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fe9440cfc90 0x7fe9440d00a0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fe9440cd890 tx=0x7fe93c02a9f0 comp rx=0 tx=0).stop 2024-08-03T20:50:23.242 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:23.240+0000 7fe94a400640 1 -- 172.21.15.32:0/2734126091 shutdown_connections 2024-08-03T20:50:23.242 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:23.240+0000 7fe94a400640 1 --2- 172.21.15.32:0/2734126091 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fe9440cfc90 0x7fe9440d00a0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:23.242 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:23.240+0000 7fe94a400640 1 --2- 172.21.15.32:0/2734126091 >> v2:172.21.15.118:3300/0 conn(0x7fe9440b29f0 0x7fe9440cdba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:23.242 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:23.240+0000 7fe94a400640 1 -- 172.21.15.32:0/2734126091 >> 172.21.15.32:0/2734126091 conn(0x7fe9440b3bb0 msgr2=0x7fe9440b3fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:50:23.242 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:23.240+0000 7fe94a400640 1 -- 172.21.15.32:0/2734126091 shutdown_connections 2024-08-03T20:50:23.242 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:23.240+0000 7fe94a400640 1 -- 172.21.15.32:0/2734126091 wait complete. 2024-08-03T20:50:23.243 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:23.242+0000 7fe94a400640 1 Processor -- start 2024-08-03T20:50:23.243 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:23.242+0000 7fe94a400640 1 -- start start 2024-08-03T20:50:23.245 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:23.243+0000 7fe94a400640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fe9440b29f0 0x7fe9441613d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:50:23.245 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:23.244+0000 7fe94a400640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fe9440b7760 0x7fe9441460c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:50:23.246 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:23.244+0000 7fe94a400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe9440cfc90 0x7fe94415ab20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:50:23.246 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:23.244+0000 7fe94a400640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fe9440d0970 con 0x7fe9440cfc90 2024-08-03T20:50:23.246 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:23.244+0000 7fe94a400640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7fe9440d07f0 con 0x7fe9440b29f0 2024-08-03T20:50:23.246 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:23.244+0000 7fe94a400640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7fe9440b17a0 con 0x7fe9440b7760 2024-08-03T20:50:23.246 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:23.244+0000 7fe943400640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fe9440b29f0 0x7fe9441613d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:50:23.246 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:23.244+0000 7fe942a00640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fe9440b7760 0x7fe9441460c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:50:23.246 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:23.244+0000 7fe943e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe9440cfc90 0x7fe94415ab20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:50:23.246 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:23.244+0000 7fe942a00640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fe9440b7760 0x7fe9441460c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3301/0 says I am v2:172.21.15.32:42980/0 (socket says 172.21.15.32:42980) 2024-08-03T20:50:23.246 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:23.244+0000 7fe943400640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fe9440b29f0 0x7fe9441613d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.32:50428/0 (socket says 172.21.15.32:50428) 2024-08-03T20:50:23.246 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:23.244+0000 7fe942a00640 1 -- 172.21.15.32:0/1385311145 learned_addr learned my addr 172.21.15.32:0/1385311145 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:50:23.248 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:23.244+0000 7fe943e00640 1 -- 172.21.15.32:0/1385311145 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fe9440b7760 msgr2=0x7fe9441460c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:50:23.248 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:23.244+0000 7fe943e00640 1 --2- 172.21.15.32:0/1385311145 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fe9440b7760 0x7fe9441460c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:23.248 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:23.244+0000 7fe943e00640 1 -- 172.21.15.32:0/1385311145 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fe9440b29f0 msgr2=0x7fe9441613d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:50:23.248 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:23.244+0000 7fe943e00640 1 --2- 172.21.15.32:0/1385311145 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fe9440b29f0 0x7fe9441613d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:23.248 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:23.244+0000 7fe943e00640 1 -- 172.21.15.32:0/1385311145 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe944164950 con 0x7fe9440cfc90 2024-08-03T20:50:23.248 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:23.245+0000 7fe943e00640 1 --2- 172.21.15.32:0/1385311145 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe9440cfc90 0x7fe94415ab20 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7fe93c032000 tx=0x7fe93c003f10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:50:23.248 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:23.245+0000 7fe93b400640 1 -- 172.21.15.32:0/1385311145 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe93c033050 con 0x7fe9440cfc90 2024-08-03T20:50:23.248 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:23.245+0000 7fe93b400640 1 -- 172.21.15.32:0/1385311145 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fe93c004450 con 0x7fe9440cfc90 2024-08-03T20:50:23.248 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:23.245+0000 7fe93b400640 1 -- 172.21.15.32:0/1385311145 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe93c004860 con 0x7fe9440cfc90 2024-08-03T20:50:23.249 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:23.245+0000 7fe94a400640 1 -- 172.21.15.32:0/1385311145 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe944163ba0 con 0x7fe9440cfc90 2024-08-03T20:50:23.249 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:23.245+0000 7fe94a400640 1 -- 172.21.15.32:0/1385311145 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe944164160 con 0x7fe9440cfc90 2024-08-03T20:50:23.249 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:23.246+0000 7fe94a400640 1 -- 172.21.15.32:0/1385311145 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe9440cdba0 con 0x7fe9440cfc90 2024-08-03T20:50:23.249 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:23.248+0000 7fe93b400640 1 -- 172.21.15.32:0/1385311145 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7fe93c036070 con 0x7fe9440cfc90 2024-08-03T20:50:23.250 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:23.249+0000 7fe93b400640 1 --2- 172.21.15.32:0/1385311145 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fe924077040 0x7fe924079550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:50:23.251 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:23.249+0000 7fe93b400640 1 -- 172.21.15.32:0/1385311145 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(49..49 src has 1..49) ==== 5029+0+0 (secure 0 0 0) 0x7fe93c0b9860 con 0x7fe9440cfc90 2024-08-03T20:50:23.251 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:23.249+0000 7fe943400640 1 --2- 172.21.15.32:0/1385311145 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fe924077040 0x7fe924079550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:50:23.251 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:23.250+0000 7fe943400640 1 --2- 172.21.15.32:0/1385311145 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fe924077040 0x7fe924079550 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7fe934007fd0 tx=0x7fe934009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:50:23.255 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:23.254+0000 7fe93b400640 1 -- 172.21.15.32:0/1385311145 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7fe93c0826a0 con 0x7fe9440cfc90 2024-08-03T20:50:23.471 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:23.470+0000 7fe94a400640 1 -- 172.21.15.32:0/1385311145 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) -- 0x7fe9440b4760 con 0x7fe9440cfc90 2024-08-03T20:50:24.130 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:24.128+0000 7fe93b400640 1 -- 172.21.15.32:0/1385311145 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v51) ==== 97+0+0 (secure 0 0 0) 0x7fe93c086bf0 con 0x7fe9440cfc90 2024-08-03T20:50:24.130 INFO:teuthology.orchestra.run.smithi032.stderr: removed all blocklist entries 2024-08-03T20:50:24.134 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:24.132+0000 7fe94a400640 1 -- 172.21.15.32:0/1385311145 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fe924077040 msgr2=0x7fe924079550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:50:24.134 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:24.132+0000 7fe94a400640 1 --2- 172.21.15.32:0/1385311145 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fe924077040 0x7fe924079550 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7fe934007fd0 tx=0x7fe934009040 comp rx=0 tx=0).stop 2024-08-03T20:50:24.134 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:24.132+0000 7fe94a400640 1 -- 172.21.15.32:0/1385311145 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe9440cfc90 msgr2=0x7fe94415ab20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:50:24.134 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:24.132+0000 7fe94a400640 1 --2- 172.21.15.32:0/1385311145 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe9440cfc90 0x7fe94415ab20 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7fe93c032000 tx=0x7fe93c003f10 comp rx=0 tx=0).stop 2024-08-03T20:50:24.134 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:24.132+0000 7fe94a400640 1 -- 172.21.15.32:0/1385311145 shutdown_connections 2024-08-03T20:50:24.134 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:24.132+0000 7fe94a400640 1 --2- 172.21.15.32:0/1385311145 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fe924077040 0x7fe924079550 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:24.134 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:24.132+0000 7fe94a400640 1 --2- 172.21.15.32:0/1385311145 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe9440cfc90 0x7fe94415ab20 unknown :-1 s=CLOSED pgs=547 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:24.134 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:24.132+0000 7fe94a400640 1 --2- 172.21.15.32:0/1385311145 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fe9440b7760 0x7fe9441460c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:24.134 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:24.132+0000 7fe94a400640 1 --2- 172.21.15.32:0/1385311145 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fe9440b29f0 0x7fe9441613d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:24.134 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:24.132+0000 7fe94a400640 1 -- 172.21.15.32:0/1385311145 >> 172.21.15.32:0/1385311145 conn(0x7fe9440b3bb0 msgr2=0x7fe94415d040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:50:24.134 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:24.132+0000 7fe94a400640 1 -- 172.21.15.32:0/1385311145 shutdown_connections 2024-08-03T20:50:24.135 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:24.132+0000 7fe94a400640 1 -- 172.21.15.32:0/1385311145 wait complete. 2024-08-03T20:50:24.161 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-08-03T20:50:24.392 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:24.390+0000 7f15fa000640 1 -- 172.21.15.32:0/3851868558 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f15f415a310 msgr2=0x7f15f40aefb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:50:24.392 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:24.390+0000 7f15fa000640 1 --2- 172.21.15.32:0/3851868558 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f15f415a310 0x7f15f40aefb0 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7f15e800a940 tx=0x7f15e802b8f0 comp rx=0 tx=0).stop 2024-08-03T20:50:24.392 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:24.390+0000 7f15fa000640 1 -- 172.21.15.32:0/3851868558 shutdown_connections 2024-08-03T20:50:24.392 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:24.390+0000 7f15fa000640 1 --2- 172.21.15.32:0/3851868558 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f15f415a310 0x7f15f40aefb0 unknown :-1 s=CLOSED pgs=549 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:24.392 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:24.390+0000 7f15fa000640 1 -- 172.21.15.32:0/3851868558 >> 172.21.15.32:0/3851868558 conn(0x7f15f40b2120 msgr2=0x7f15f40cfd60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:50:24.392 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:24.390+0000 7f15fa000640 1 -- 172.21.15.32:0/3851868558 shutdown_connections 2024-08-03T20:50:24.392 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:24.390+0000 7f15fa000640 1 -- 172.21.15.32:0/3851868558 wait complete. 2024-08-03T20:50:24.394 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:24.392+0000 7f15fa000640 1 Processor -- start 2024-08-03T20:50:24.394 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:24.392+0000 7f15fa000640 1 -- start start 2024-08-03T20:50:24.396 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:24.394+0000 7f15fa000640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f15f40ca3b0 0x7f15f40b5900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:50:24.396 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:24.394+0000 7f15fa000640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f15f40ce150 0x7f15f4150b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:50:24.396 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:24.394+0000 7f15fa000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f15f415a310 0x7f15f40b3920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:50:24.396 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:24.394+0000 7f15fa000640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f15f4070860 con 0x7f15f415a310 2024-08-03T20:50:24.396 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:24.394+0000 7f15fa000640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f15f40706e0 con 0x7f15f40ce150 2024-08-03T20:50:24.396 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:24.394+0000 7f15fa000640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7f15f40cec70 con 0x7f15f40ca3b0 2024-08-03T20:50:24.396 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:24.394+0000 7f15f9600640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f15f415a310 0x7f15f40b3920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:50:24.396 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:24.394+0000 7f15f9600640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f15f415a310 0x7f15f40b3920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:60584/0 (socket says 172.21.15.32:60584) 2024-08-03T20:50:24.396 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:24.394+0000 7f15f9600640 1 -- 172.21.15.32:0/1189044669 learned_addr learned my addr 172.21.15.32:0/1189044669 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:50:24.396 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:24.394+0000 7f15f8c00640 1 --2- 172.21.15.32:0/1189044669 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f15f40ca3b0 0x7f15f40b5900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:50:24.396 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:24.394+0000 7f15f3e00640 1 --2- 172.21.15.32:0/1189044669 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f15f40ce150 0x7f15f4150b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:50:24.397 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:24.395+0000 7f15f9600640 1 -- 172.21.15.32:0/1189044669 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f15f40ca3b0 msgr2=0x7f15f40b5900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:50:24.398 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:24.395+0000 7f15f9600640 1 --2- 172.21.15.32:0/1189044669 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f15f40ca3b0 0x7f15f40b5900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:24.398 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:24.395+0000 7f15f9600640 1 -- 172.21.15.32:0/1189044669 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f15f40ce150 msgr2=0x7f15f4150b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:50:24.398 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:24.395+0000 7f15f9600640 1 --2- 172.21.15.32:0/1189044669 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f15f40ce150 0x7f15f4150b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:24.398 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:24.395+0000 7f15f9600640 1 -- 172.21.15.32:0/1189044669 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f15f40b3e60 con 0x7f15f415a310 2024-08-03T20:50:24.398 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:24.395+0000 7f15f9600640 1 --2- 172.21.15.32:0/1189044669 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f15f415a310 0x7f15f40b3920 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7f15ec00c700 tx=0x7f15ec00cbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:50:24.398 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:24.395+0000 7f15f0c00640 1 -- 172.21.15.32:0/1189044669 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15ec003b70 con 0x7f15f415a310 2024-08-03T20:50:24.398 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:24.395+0000 7f15f0c00640 1 -- 172.21.15.32:0/1189044669 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f15ec003d10 con 0x7f15f415a310 2024-08-03T20:50:24.398 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:24.395+0000 7f15f0c00640 1 -- 172.21.15.32:0/1189044669 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15ec00aab0 con 0x7f15f415a310 2024-08-03T20:50:24.398 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:24.395+0000 7f15fa000640 1 -- 172.21.15.32:0/1189044669 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f15f40c8c40 con 0x7f15f415a310 2024-08-03T20:50:24.398 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:24.396+0000 7f15fa000640 1 -- 172.21.15.32:0/1189044669 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f15f40c8ee0 con 0x7f15f415a310 2024-08-03T20:50:24.399 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:24.397+0000 7f15fa000640 1 -- 172.21.15.32:0/1189044669 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f15f40cf120 con 0x7f15f415a310 2024-08-03T20:50:24.399 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:24.398+0000 7f15f0c00640 1 -- 172.21.15.32:0/1189044669 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f15ec00ac50 con 0x7f15f415a310 2024-08-03T20:50:24.401 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:24.399+0000 7f15f0c00640 1 --2- 172.21.15.32:0/1189044669 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f15d4077040 0x7f15d4079550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:50:24.401 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:24.399+0000 7f15f0c00640 1 -- 172.21.15.32:0/1189044669 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(53..53 src has 1..53) ==== 4857+0+0 (secure 0 0 0) 0x7f15ec098260 con 0x7f15f415a310 2024-08-03T20:50:24.401 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:24.400+0000 7f15f8c00640 1 --2- 172.21.15.32:0/1189044669 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f15d4077040 0x7f15d4079550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:50:24.402 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:24.401+0000 7f15f8c00640 1 --2- 172.21.15.32:0/1189044669 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f15d4077040 0x7f15d4079550 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f15e8036000 tx=0x7f15e8004730 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:50:24.405 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:24.404+0000 7f15f0c00640 1 -- 172.21.15.32:0/1189044669 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7f15ec0611e0 con 0x7f15f415a310 2024-08-03T20:50:24.684 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:24.682+0000 7f15fa000640 1 -- 172.21.15.32:0/1189044669 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) -- 0x7f15f406e780 con 0x7f15f415a310 2024-08-03T20:50:24.684 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:24.683+0000 7f15f0c00640 1 -- 172.21.15.32:0/1189044669 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v12) ==== 80+0+5304 (secure 0 0 0) 0x7f15ec065730 con 0x7f15f415a310 2024-08-03T20:50:24.684 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-03T20:50:24.685 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-08-03T20:50:24.685 INFO:teuthology.orchestra.run.smithi032.stdout: "auth_dump": [ 2024-08-03T20:50:24.685 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-03T20:50:24.685 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "mds.a", 2024-08-03T20:50:24.685 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQAvl65mfaKbOhAAUgCH82U0SB7owF5Cds3XYA==", 2024-08-03T20:50:24.685 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2024-08-03T20:50:24.685 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": "allow", 2024-08-03T20:50:24.685 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow *", 2024-08-03T20:50:24.685 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow *", 2024-08-03T20:50:24.685 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2024-08-03T20:50:24.685 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-03T20:50:24.686 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-03T20:50:24.686 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-03T20:50:24.686 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "mds.b", 2024-08-03T20:50:24.686 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQAwl65mNcoVGxAAlRXNjZhxVu8Vjslk9Msrxw==", 2024-08-03T20:50:24.686 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2024-08-03T20:50:24.686 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": "allow", 2024-08-03T20:50:24.686 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow *", 2024-08-03T20:50:24.686 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow *", 2024-08-03T20:50:24.686 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2024-08-03T20:50:24.686 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-03T20:50:24.686 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-03T20:50:24.686 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-03T20:50:24.686 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "mds.c", 2024-08-03T20:50:24.687 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQAwl65mEUAFDRAAGdw1cymfvfKxGeJS/Od6Rw==", 2024-08-03T20:50:24.687 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2024-08-03T20:50:24.687 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": "allow", 2024-08-03T20:50:24.687 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow *", 2024-08-03T20:50:24.687 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow *", 2024-08-03T20:50:24.687 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2024-08-03T20:50:24.687 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-03T20:50:24.687 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-03T20:50:24.687 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-03T20:50:24.687 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "mds.d", 2024-08-03T20:50:24.687 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQAwl65mjUHZKBAAqOCnIopBK8Rhk36hrlNnzw==", 2024-08-03T20:50:24.687 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2024-08-03T20:50:24.687 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": "allow", 2024-08-03T20:50:24.688 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow *", 2024-08-03T20:50:24.688 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow *", 2024-08-03T20:50:24.688 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2024-08-03T20:50:24.688 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-03T20:50:24.688 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-03T20:50:24.688 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-03T20:50:24.688 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "osd.0", 2024-08-03T20:50:24.688 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQAyl65m4kIyLxAAdCh0D2MjJNPPu5b7jNkrbw==", 2024-08-03T20:50:24.688 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2024-08-03T20:50:24.688 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow profile osd", 2024-08-03T20:50:24.688 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow profile osd", 2024-08-03T20:50:24.688 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2024-08-03T20:50:24.688 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-03T20:50:24.689 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-03T20:50:24.689 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-03T20:50:24.689 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "osd.1", 2024-08-03T20:50:24.689 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQA1l65mLzVOOhAA081V1cUyYFPz17L0nny7gA==", 2024-08-03T20:50:24.689 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2024-08-03T20:50:24.689 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow profile osd", 2024-08-03T20:50:24.689 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow profile osd", 2024-08-03T20:50:24.689 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2024-08-03T20:50:24.689 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-03T20:50:24.689 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-03T20:50:24.689 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-03T20:50:24.689 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "osd.2", 2024-08-03T20:50:24.689 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQA5l65mBnXDBxAA9/lZVkwD2ykMTOIysoYODg==", 2024-08-03T20:50:24.689 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2024-08-03T20:50:24.690 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow profile osd", 2024-08-03T20:50:24.690 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow profile osd", 2024-08-03T20:50:24.690 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2024-08-03T20:50:24.690 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-03T20:50:24.690 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-03T20:50:24.690 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-03T20:50:24.690 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "osd.3", 2024-08-03T20:50:24.690 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQA8l65mAriiERAAfzV8An67+PnA3EzeAutIbA==", 2024-08-03T20:50:24.690 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2024-08-03T20:50:24.690 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow profile osd", 2024-08-03T20:50:24.690 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow profile osd", 2024-08-03T20:50:24.690 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2024-08-03T20:50:24.691 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-03T20:50:24.691 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-03T20:50:24.691 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-03T20:50:24.691 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "osd.4", 2024-08-03T20:50:24.691 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQBAl65m5uJSJRAA8S2IjJpxuqHY8YK4tTDMPg==", 2024-08-03T20:50:24.691 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2024-08-03T20:50:24.691 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow profile osd", 2024-08-03T20:50:24.691 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow profile osd", 2024-08-03T20:50:24.691 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2024-08-03T20:50:24.691 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-03T20:50:24.691 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-03T20:50:24.691 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-03T20:50:24.691 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "osd.5", 2024-08-03T20:50:24.691 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQBDl65mBP3JLBAADgK2X3zNyfnjjZADX1PFcA==", 2024-08-03T20:50:24.692 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2024-08-03T20:50:24.692 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow profile osd", 2024-08-03T20:50:24.692 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow profile osd", 2024-08-03T20:50:24.692 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2024-08-03T20:50:24.692 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-03T20:50:24.692 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-03T20:50:24.692 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-03T20:50:24.692 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "osd.6", 2024-08-03T20:50:24.692 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQBGl65m8DBNNRAANeOUfW70S5JfqFlpuMF9KQ==", 2024-08-03T20:50:24.692 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2024-08-03T20:50:24.692 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow profile osd", 2024-08-03T20:50:24.692 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow profile osd", 2024-08-03T20:50:24.692 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2024-08-03T20:50:24.693 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-03T20:50:24.693 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-03T20:50:24.693 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-03T20:50:24.693 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "osd.7", 2024-08-03T20:50:24.693 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQBKl65mUnOcABAA41wylzC1Ec9Fj01Ke0HhTw==", 2024-08-03T20:50:24.693 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2024-08-03T20:50:24.693 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow profile osd", 2024-08-03T20:50:24.693 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow profile osd", 2024-08-03T20:50:24.693 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2024-08-03T20:50:24.693 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-03T20:50:24.693 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-03T20:50:24.693 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-03T20:50:24.693 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "client.0", 2024-08-03T20:50:24.693 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQAwl65msgO3NRAAHG/WuMTK9c4lDXocXvqWnA==", 2024-08-03T20:50:24.694 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2024-08-03T20:50:24.694 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": "allow", 2024-08-03T20:50:24.694 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow r", 2024-08-03T20:50:24.694 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-08-03T20:50:24.694 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-03T20:50:24.694 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-03T20:50:24.694 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-03T20:50:24.694 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "client.1", 2024-08-03T20:50:24.694 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQAxl65mbe19DxAA+sepc9ucXk2Bqy7C90mfbA==", 2024-08-03T20:50:24.694 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2024-08-03T20:50:24.694 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": "allow", 2024-08-03T20:50:24.694 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow r", 2024-08-03T20:50:24.694 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow rw", 2024-08-03T20:50:24.694 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow rwx" 2024-08-03T20:50:24.695 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-03T20:50:24.695 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-03T20:50:24.695 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-03T20:50:24.695 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "client.2", 2024-08-03T20:50:24.695 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQAxl65mBnbkBBAAjor60WWLTKXidiBMgtKuEA==", 2024-08-03T20:50:24.695 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2024-08-03T20:50:24.695 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": "allow", 2024-08-03T20:50:24.695 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow r", 2024-08-03T20:50:24.695 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow rw", 2024-08-03T20:50:24.695 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow rwx" 2024-08-03T20:50:24.695 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-03T20:50:24.695 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-03T20:50:24.695 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-03T20:50:24.696 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "client.3", 2024-08-03T20:50:24.696 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQAxl65munz8GRAA/cRl3ghwQE42FOizurcW3g==", 2024-08-03T20:50:24.696 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2024-08-03T20:50:24.696 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": "allow", 2024-08-03T20:50:24.696 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow r", 2024-08-03T20:50:24.696 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow rw", 2024-08-03T20:50:24.696 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow rwx" 2024-08-03T20:50:24.696 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-03T20:50:24.696 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-03T20:50:24.696 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-03T20:50:24.696 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "client.admin", 2024-08-03T20:50:24.696 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQAul65mB8VcOhAA105JQ2FX1vhtaGj3Fer8mg==", 2024-08-03T20:50:24.696 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2024-08-03T20:50:24.697 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": "allow *", 2024-08-03T20:50:24.697 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow *", 2024-08-03T20:50:24.697 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow *", 2024-08-03T20:50:24.697 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2024-08-03T20:50:24.697 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-03T20:50:24.697 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-03T20:50:24.697 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-03T20:50:24.697 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "mgr.x", 2024-08-03T20:50:24.697 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQAvl65mUaPKLxAAnll74Gmk8nSuN/HuX6rTxA==", 2024-08-03T20:50:24.697 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2024-08-03T20:50:24.697 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": "allow *", 2024-08-03T20:50:24.697 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow profile mgr", 2024-08-03T20:50:24.697 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2024-08-03T20:50:24.697 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-03T20:50:24.698 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-03T20:50:24.698 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-03T20:50:24.698 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "mgr.y", 2024-08-03T20:50:24.698 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQAvl65m5Gq4JBAA306Ge1zHF3Z0WIzUrset4A==", 2024-08-03T20:50:24.698 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2024-08-03T20:50:24.698 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": "allow *", 2024-08-03T20:50:24.698 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow profile mgr", 2024-08-03T20:50:24.698 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2024-08-03T20:50:24.698 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-03T20:50:24.698 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-03T20:50:24.698 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-03T20:50:24.698 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-08-03T20:50:24.699 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:24.685+0000 7f15fa000640 1 -- 172.21.15.32:0/1189044669 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f15d4077040 msgr2=0x7f15d4079550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:50:24.699 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:24.685+0000 7f15fa000640 1 --2- 172.21.15.32:0/1189044669 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f15d4077040 0x7f15d4079550 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f15e8036000 tx=0x7f15e8004730 comp rx=0 tx=0).stop 2024-08-03T20:50:24.700 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:24.685+0000 7f15fa000640 1 -- 172.21.15.32:0/1189044669 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f15f415a310 msgr2=0x7f15f40b3920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:50:24.700 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:24.685+0000 7f15fa000640 1 --2- 172.21.15.32:0/1189044669 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f15f415a310 0x7f15f40b3920 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7f15ec00c700 tx=0x7f15ec00cbc0 comp rx=0 tx=0).stop 2024-08-03T20:50:24.700 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:24.685+0000 7f15fa000640 1 -- 172.21.15.32:0/1189044669 shutdown_connections 2024-08-03T20:50:24.700 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:24.685+0000 7f15fa000640 1 --2- 172.21.15.32:0/1189044669 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f15d4077040 0x7f15d4079550 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:24.700 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:24.685+0000 7f15fa000640 1 --2- 172.21.15.32:0/1189044669 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f15f40ce150 0x7f15f4150b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:24.700 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:24.685+0000 7f15fa000640 1 --2- 172.21.15.32:0/1189044669 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f15f415a310 0x7f15f40b3920 unknown :-1 s=CLOSED pgs=550 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:24.700 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:24.685+0000 7f15fa000640 1 --2- 172.21.15.32:0/1189044669 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f15f40ca3b0 0x7f15f40b5900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:24.700 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:24.685+0000 7f15fa000640 1 -- 172.21.15.32:0/1189044669 >> 172.21.15.32:0/1189044669 conn(0x7f15f40b2120 msgr2=0x7f15f40b0360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:50:24.700 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:24.686+0000 7f15fa000640 1 -- 172.21.15.32:0/1189044669 shutdown_connections 2024-08-03T20:50:24.700 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:24.686+0000 7f15fa000640 1 -- 172.21.15.32:0/1189044669 wait complete. 2024-08-03T20:50:24.709 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-08-03T20:50:24.938 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:24.936+0000 7fd0daa00640 1 -- 172.21.15.32:0/1247852168 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd0d406f410 msgr2=0x7fd0d406f820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:50:24.938 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:24.936+0000 7fd0daa00640 1 --2- 172.21.15.32:0/1247852168 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd0d406f410 0x7fd0d406f820 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7fd0d4158fd0 tx=0x7fd0c002a4d0 comp rx=0 tx=0).stop 2024-08-03T20:50:24.938 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:24.936+0000 7fd0daa00640 1 -- 172.21.15.32:0/1247852168 shutdown_connections 2024-08-03T20:50:24.938 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:24.936+0000 7fd0daa00640 1 --2- 172.21.15.32:0/1247852168 >> v2:172.21.15.118:3300/0 conn(0x7fd0d40cde40 0x7fd0d40ce250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:24.938 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:24.936+0000 7fd0daa00640 1 --2- 172.21.15.32:0/1247852168 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd0d406f410 0x7fd0d406f820 unknown :-1 s=CLOSED pgs=551 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:24.938 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:24.936+0000 7fd0daa00640 1 --2- 172.21.15.32:0/1247852168 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fd0d40ca8c0 0x7fd0d40af140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:24.938 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:24.936+0000 7fd0daa00640 1 -- 172.21.15.32:0/1247852168 >> 172.21.15.32:0/1247852168 conn(0x7fd0d40b2120 msgr2=0x7fd0d40cfd60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:50:24.938 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:24.936+0000 7fd0daa00640 1 -- 172.21.15.32:0/1247852168 shutdown_connections 2024-08-03T20:50:24.938 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:24.936+0000 7fd0daa00640 1 -- 172.21.15.32:0/1247852168 wait complete. 2024-08-03T20:50:24.939 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:24.938+0000 7fd0daa00640 1 Processor -- start 2024-08-03T20:50:24.940 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:24.938+0000 7fd0daa00640 1 -- start start 2024-08-03T20:50:24.942 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:24.940+0000 7fd0daa00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fd0d406f410 0x7fd0d409cf80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:50:24.942 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:24.940+0000 7fd0daa00640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fd0d40ca8c0 0x7fd0d41f8e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:50:24.942 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:24.940+0000 7fd0daa00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd0d40cde40 0x7fd0d40c91f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:50:24.942 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:24.940+0000 7fd0daa00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fd0d415ef90 con 0x7fd0d40cde40 2024-08-03T20:50:24.942 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:24.940+0000 7fd0daa00640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7fd0d415ee10 con 0x7fd0d406f410 2024-08-03T20:50:24.942 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:24.940+0000 7fd0daa00640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7fd0d415f110 con 0x7fd0d40ca8c0 2024-08-03T20:50:24.942 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:24.940+0000 7fd0da000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd0d40cde40 0x7fd0d40c91f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:50:24.942 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:24.940+0000 7fd0da000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd0d40cde40 0x7fd0d40c91f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:60604/0 (socket says 172.21.15.32:60604) 2024-08-03T20:50:24.942 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:24.940+0000 7fd0da000640 1 -- 172.21.15.32:0/4074544231 learned_addr learned my addr 172.21.15.32:0/4074544231 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:50:24.942 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:24.940+0000 7fd0d9600640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fd0d406f410 0x7fd0d409cf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:50:24.942 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:24.940+0000 7fd0d8c00640 1 --2- 172.21.15.32:0/4074544231 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fd0d40ca8c0 0x7fd0d41f8e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:50:24.943 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:24.941+0000 7fd0da000640 1 -- 172.21.15.32:0/4074544231 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fd0d40ca8c0 msgr2=0x7fd0d41f8e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:50:24.943 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:24.941+0000 7fd0da000640 1 --2- 172.21.15.32:0/4074544231 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fd0d40ca8c0 0x7fd0d41f8e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:24.944 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:24.941+0000 7fd0da000640 1 -- 172.21.15.32:0/4074544231 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fd0d406f410 msgr2=0x7fd0d409cf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:50:24.944 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:24.941+0000 7fd0da000640 1 --2- 172.21.15.32:0/4074544231 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fd0d406f410 0x7fd0d409cf80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:24.944 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:24.941+0000 7fd0da000640 1 -- 172.21.15.32:0/4074544231 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd0d409dd20 con 0x7fd0d40cde40 2024-08-03T20:50:24.944 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:24.941+0000 7fd0da000640 1 --2- 172.21.15.32:0/4074544231 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd0d40cde40 0x7fd0d40c91f0 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7fd0cc00cb00 tx=0x7fd0cc00a250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:50:24.944 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:24.941+0000 7fd0d1600640 1 -- 172.21.15.32:0/4074544231 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0cc00f450 con 0x7fd0d40cde40 2024-08-03T20:50:24.944 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:24.941+0000 7fd0d1600640 1 -- 172.21.15.32:0/4074544231 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fd0cc00f700 con 0x7fd0d40cde40 2024-08-03T20:50:24.944 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:24.941+0000 7fd0d1600640 1 -- 172.21.15.32:0/4074544231 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0cc00fb50 con 0x7fd0d40cde40 2024-08-03T20:50:24.944 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:24.941+0000 7fd0daa00640 1 -- 172.21.15.32:0/4074544231 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd0d4148c40 con 0x7fd0d40cde40 2024-08-03T20:50:24.944 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:24.941+0000 7fd0daa00640 1 -- 172.21.15.32:0/4074544231 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd0d40b3d90 con 0x7fd0d40cde40 2024-08-03T20:50:24.944 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:24.942+0000 7fd0daa00640 1 -- 172.21.15.32:0/4074544231 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd0d4148e10 con 0x7fd0d40cde40 2024-08-03T20:50:24.945 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:24.944+0000 7fd0d1600640 1 -- 172.21.15.32:0/4074544231 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7fd0cc00fcf0 con 0x7fd0d40cde40 2024-08-03T20:50:24.946 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:24.945+0000 7fd0d1600640 1 --2- 172.21.15.32:0/4074544231 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fd0b4077010 0x7fd0b4079520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:50:24.947 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:24.945+0000 7fd0d1600640 1 -- 172.21.15.32:0/4074544231 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(53..53 src has 1..53) ==== 4857+0+0 (secure 0 0 0) 0x7fd0cc098ff0 con 0x7fd0d40cde40 2024-08-03T20:50:24.947 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:24.946+0000 7fd0d9600640 1 --2- 172.21.15.32:0/4074544231 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fd0b4077010 0x7fd0b4079520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:50:24.948 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:24.946+0000 7fd0d9600640 1 --2- 172.21.15.32:0/4074544231 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fd0b4077010 0x7fd0b4079520 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7fd0c8005fd0 tx=0x7fd0c800b260 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:50:24.951 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:24.950+0000 7fd0d1600640 1 -- 172.21.15.32:0/4074544231 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7fd0cc014030 con 0x7fd0d40cde40 2024-08-03T20:50:25.163 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:25.161+0000 7fd0daa00640 1 -- 172.21.15.32:0/4074544231 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) -- 0x7fd0d415f6a0 con 0x7fd0d40cde40 2024-08-03T20:50:25.163 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:25.162+0000 7fd0d1600640 1 -- 172.21.15.32:0/4074544231 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v53) ==== 84+0+65 (secure 0 0 0) 0x7fd0cc061f00 con 0x7fd0d40cde40 2024-08-03T20:50:25.163 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-03T20:50:25.163 INFO:teuthology.orchestra.run.smithi032.stdout:[ 2024-08-03T20:50:25.164 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-03T20:50:25.164 INFO:teuthology.orchestra.run.smithi032.stdout: "poolnum": 1, 2024-08-03T20:50:25.164 INFO:teuthology.orchestra.run.smithi032.stdout: "poolname": ".mgr" 2024-08-03T20:50:25.164 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-03T20:50:25.164 INFO:teuthology.orchestra.run.smithi032.stdout:] 2024-08-03T20:50:25.166 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:25.164+0000 7fd0daa00640 1 -- 172.21.15.32:0/4074544231 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fd0b4077010 msgr2=0x7fd0b4079520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:50:25.166 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:25.164+0000 7fd0daa00640 1 --2- 172.21.15.32:0/4074544231 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fd0b4077010 0x7fd0b4079520 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7fd0c8005fd0 tx=0x7fd0c800b260 comp rx=0 tx=0).stop 2024-08-03T20:50:25.166 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:25.164+0000 7fd0daa00640 1 -- 172.21.15.32:0/4074544231 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd0d40cde40 msgr2=0x7fd0d40c91f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:50:25.166 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:25.164+0000 7fd0daa00640 1 --2- 172.21.15.32:0/4074544231 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd0d40cde40 0x7fd0d40c91f0 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7fd0cc00cb00 tx=0x7fd0cc00a250 comp rx=0 tx=0).stop 2024-08-03T20:50:25.166 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:25.164+0000 7fd0daa00640 1 -- 172.21.15.32:0/4074544231 shutdown_connections 2024-08-03T20:50:25.166 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:25.164+0000 7fd0daa00640 1 --2- 172.21.15.32:0/4074544231 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fd0b4077010 0x7fd0b4079520 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:25.166 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:25.164+0000 7fd0daa00640 1 --2- 172.21.15.32:0/4074544231 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd0d40cde40 0x7fd0d40c91f0 unknown :-1 s=CLOSED pgs=552 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:25.166 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:25.164+0000 7fd0daa00640 1 --2- 172.21.15.32:0/4074544231 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fd0d40ca8c0 0x7fd0d41f8e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:25.166 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:25.164+0000 7fd0daa00640 1 --2- 172.21.15.32:0/4074544231 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fd0d406f410 0x7fd0d409cf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:25.166 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:25.164+0000 7fd0daa00640 1 -- 172.21.15.32:0/4074544231 >> 172.21.15.32:0/4074544231 conn(0x7fd0d40b2120 msgr2=0x7fd0d4154eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:50:25.167 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:25.164+0000 7fd0daa00640 1 -- 172.21.15.32:0/4074544231 shutdown_connections 2024-08-03T20:50:25.167 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:25.164+0000 7fd0daa00640 1 -- 172.21.15.32:0/4074544231 wait complete. 2024-08-03T20:50:25.191 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-08-03T20:50:25.192 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-08-03T20:50:25.424 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:25.422+0000 7f0a3da00640 1 -- 172.21.15.32:0/4086734525 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0a380b3e80 msgr2=0x7f0a380b4290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:50:25.424 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:25.422+0000 7f0a3da00640 1 --2- 172.21.15.32:0/4086734525 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0a380b3e80 0x7f0a380b4290 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7f0a380b2bf0 tx=0x7f0a3002aa30 comp rx=0 tx=0).stop 2024-08-03T20:50:25.424 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:25.423+0000 7f0a3da00640 1 -- 172.21.15.32:0/4086734525 shutdown_connections 2024-08-03T20:50:25.425 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:25.423+0000 7f0a3da00640 1 --2- 172.21.15.32:0/4086734525 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0a380b3e80 0x7f0a380b4290 unknown :-1 s=CLOSED pgs=553 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:25.425 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:25.423+0000 7f0a3da00640 1 --2- 172.21.15.32:0/4086734525 >> v2:172.21.15.118:3300/0 conn(0x7f0a3806e950 0x7f0a3806ed60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:25.425 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:25.423+0000 7f0a3da00640 1 --2- 172.21.15.32:0/4086734525 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f0a3815d2b0 0x7f0a380ced30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:25.425 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:25.423+0000 7f0a3da00640 1 -- 172.21.15.32:0/4086734525 >> 172.21.15.32:0/4086734525 conn(0x7f0a380b1bd0 msgr2=0x7f0a380b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:50:25.425 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:25.423+0000 7f0a3da00640 1 -- 172.21.15.32:0/4086734525 shutdown_connections 2024-08-03T20:50:25.425 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:25.423+0000 7f0a3da00640 1 -- 172.21.15.32:0/4086734525 wait complete. 2024-08-03T20:50:25.426 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:25.425+0000 7f0a3da00640 1 Processor -- start 2024-08-03T20:50:25.426 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:25.425+0000 7f0a3da00640 1 -- start start 2024-08-03T20:50:25.429 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:25.426+0000 7f0a3da00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0a3806e950 0x7f0a381458c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:50:25.429 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:25.426+0000 7f0a3da00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f0a380b3e80 0x7f0a38146cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:50:25.429 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:25.426+0000 7f0a3da00640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f0a3815d2b0 0x7f0a38147230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:50:25.429 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:25.427+0000 7f0a3da00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f0a380cce00 con 0x7f0a3806e950 2024-08-03T20:50:25.429 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:25.427+0000 7f0a3da00640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f0a380ccc80 con 0x7f0a380b3e80 2024-08-03T20:50:25.429 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:25.427+0000 7f0a3da00640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7f0a380ccf80 con 0x7f0a3815d2b0 2024-08-03T20:50:25.429 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:25.427+0000 7f0a36a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0a3806e950 0x7f0a381458c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:50:25.429 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:25.427+0000 7f0a36a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0a3806e950 0x7f0a381458c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:60622/0 (socket says 172.21.15.32:60622) 2024-08-03T20:50:25.429 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:25.427+0000 7f0a36a00640 1 -- 172.21.15.32:0/3099567293 learned_addr learned my addr 172.21.15.32:0/3099567293 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:50:25.429 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:25.427+0000 7f0a37400640 1 --2- 172.21.15.32:0/3099567293 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f0a3815d2b0 0x7f0a38147230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:50:25.429 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:25.427+0000 7f0a36000640 1 --2- 172.21.15.32:0/3099567293 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f0a380b3e80 0x7f0a38146cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:50:25.429 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:25.427+0000 7f0a36a00640 1 -- 172.21.15.32:0/3099567293 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f0a3815d2b0 msgr2=0x7f0a38147230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:50:25.430 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:25.427+0000 7f0a36a00640 1 --2- 172.21.15.32:0/3099567293 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f0a3815d2b0 0x7f0a38147230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:25.430 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:25.427+0000 7f0a36a00640 1 -- 172.21.15.32:0/3099567293 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f0a380b3e80 msgr2=0x7f0a38146cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:50:25.430 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:25.427+0000 7f0a36a00640 1 --2- 172.21.15.32:0/3099567293 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f0a380b3e80 0x7f0a38146cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:25.430 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:25.427+0000 7f0a36a00640 1 -- 172.21.15.32:0/3099567293 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0a380aac30 con 0x7f0a3806e950 2024-08-03T20:50:25.430 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:25.427+0000 7f0a37400640 1 --2- 172.21.15.32:0/3099567293 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f0a3815d2b0 0x7f0a38147230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-03T20:50:25.430 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:25.427+0000 7f0a36a00640 1 --2- 172.21.15.32:0/3099567293 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0a3806e950 0x7f0a381458c0 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7f0a2800d740 tx=0x7f0a2800dc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:50:25.431 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:25.428+0000 7f0a2ea00640 1 -- 172.21.15.32:0/3099567293 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a28017070 con 0x7f0a3806e950 2024-08-03T20:50:25.431 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:25.428+0000 7f0a2ea00640 1 -- 172.21.15.32:0/3099567293 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f0a28009cc0 con 0x7f0a3806e950 2024-08-03T20:50:25.431 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:25.428+0000 7f0a2ea00640 1 -- 172.21.15.32:0/3099567293 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a280106c0 con 0x7f0a3806e950 2024-08-03T20:50:25.431 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:25.428+0000 7f0a3da00640 1 -- 172.21.15.32:0/3099567293 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0a38146210 con 0x7f0a3806e950 2024-08-03T20:50:25.431 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:25.428+0000 7f0a3da00640 1 -- 172.21.15.32:0/3099567293 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0a381467d0 con 0x7f0a3806e950 2024-08-03T20:50:25.431 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:25.429+0000 7f0a3da00640 1 -- 172.21.15.32:0/3099567293 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0a380cd430 con 0x7f0a3806e950 2024-08-03T20:50:25.431 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:25.430+0000 7f0a2ea00640 1 -- 172.21.15.32:0/3099567293 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f0a2801a070 con 0x7f0a3806e950 2024-08-03T20:50:25.433 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:25.431+0000 7f0a2ea00640 1 --2- 172.21.15.32:0/3099567293 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f0a18077040 0x7f0a18079550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:50:25.433 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:25.431+0000 7f0a2ea00640 1 -- 172.21.15.32:0/3099567293 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(53..53 src has 1..53) ==== 4857+0+0 (secure 0 0 0) 0x7f0a28098630 con 0x7f0a3806e950 2024-08-03T20:50:25.433 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:25.432+0000 7f0a36000640 1 --2- 172.21.15.32:0/3099567293 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f0a18077040 0x7f0a18079550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:50:25.434 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:25.432+0000 7f0a36000640 1 --2- 172.21.15.32:0/3099567293 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f0a18077040 0x7f0a18079550 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f0a3814c780 tx=0x7f0a24005c90 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:50:25.437 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:25.436+0000 7f0a2ea00640 1 -- 172.21.15.32:0/3099567293 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7f0a28061b70 con 0x7f0a3806e950 2024-08-03T20:50:25.640 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:25.639+0000 7f0a3da00640 1 -- 172.21.15.32:0/3099567293 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) -- 0x7f0a380732c0 con 0x7f0a3806e950 2024-08-03T20:50:26.143 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:26.141+0000 7f0a2ea00640 1 -- 172.21.15.32:0/3099567293 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v54) ==== 138+0+0 (secure 0 0 0) 0x7f0a28061470 con 0x7f0a3806e950 2024-08-03T20:50:26.143 INFO:teuthology.orchestra.run.smithi032.stderr:pool 'cephfs_metadata' created 2024-08-03T20:50:26.147 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:26.145+0000 7f0a3da00640 1 -- 172.21.15.32:0/3099567293 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f0a18077040 msgr2=0x7f0a18079550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:50:26.147 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:26.145+0000 7f0a3da00640 1 --2- 172.21.15.32:0/3099567293 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f0a18077040 0x7f0a18079550 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f0a3814c780 tx=0x7f0a24005c90 comp rx=0 tx=0).stop 2024-08-03T20:50:26.147 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:26.145+0000 7f0a3da00640 1 -- 172.21.15.32:0/3099567293 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0a3806e950 msgr2=0x7f0a381458c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:50:26.147 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:26.145+0000 7f0a3da00640 1 --2- 172.21.15.32:0/3099567293 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0a3806e950 0x7f0a381458c0 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7f0a2800d740 tx=0x7f0a2800dc00 comp rx=0 tx=0).stop 2024-08-03T20:50:26.147 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:26.145+0000 7f0a3da00640 1 -- 172.21.15.32:0/3099567293 shutdown_connections 2024-08-03T20:50:26.147 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:26.145+0000 7f0a3da00640 1 --2- 172.21.15.32:0/3099567293 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f0a18077040 0x7f0a18079550 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:26.147 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:26.145+0000 7f0a3da00640 1 --2- 172.21.15.32:0/3099567293 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f0a380b3e80 0x7f0a38146cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:26.147 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:26.145+0000 7f0a3da00640 1 --2- 172.21.15.32:0/3099567293 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f0a3815d2b0 0x7f0a38147230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:26.147 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:26.145+0000 7f0a3da00640 1 --2- 172.21.15.32:0/3099567293 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0a3806e950 0x7f0a381458c0 unknown :-1 s=CLOSED pgs=554 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:26.147 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:26.145+0000 7f0a3da00640 1 -- 172.21.15.32:0/3099567293 >> 172.21.15.32:0/3099567293 conn(0x7f0a380b1bd0 msgr2=0x7f0a38072ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:50:26.147 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:26.145+0000 7f0a3da00640 1 -- 172.21.15.32:0/3099567293 shutdown_connections 2024-08-03T20:50:26.147 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:26.145+0000 7f0a3da00640 1 -- 172.21.15.32:0/3099567293 wait complete. 2024-08-03T20:50:26.177 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-08-03T20:50:26.415 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:26.413+0000 7f38cb200640 1 -- 172.21.15.32:0/2887857202 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f38c4155920 msgr2=0x7f38c40ceb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:50:26.415 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:26.413+0000 7f38cb200640 1 --2- 172.21.15.32:0/2887857202 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f38c4155920 0x7f38c40ceb10 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7f38b4005fd0 tx=0x7f38b402a5d0 comp rx=0 tx=0).stop 2024-08-03T20:50:26.415 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:26.413+0000 7f38cb200640 1 -- 172.21.15.32:0/2887857202 shutdown_connections 2024-08-03T20:50:26.415 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:26.413+0000 7f38cb200640 1 --2- 172.21.15.32:0/2887857202 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f38c40cf3b0 0x7f38c40cf7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:26.415 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:26.413+0000 7f38cb200640 1 --2- 172.21.15.32:0/2887857202 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f38c4155920 0x7f38c40ceb10 unknown :-1 s=CLOSED pgs=555 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:26.415 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:26.413+0000 7f38cb200640 1 -- 172.21.15.32:0/2887857202 >> 172.21.15.32:0/2887857202 conn(0x7f38c40b1bd0 msgr2=0x7f38c40b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:50:26.415 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:26.413+0000 7f38cb200640 1 -- 172.21.15.32:0/2887857202 shutdown_connections 2024-08-03T20:50:26.416 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:26.413+0000 7f38cb200640 1 -- 172.21.15.32:0/2887857202 wait complete. 2024-08-03T20:50:26.418 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:26.415+0000 7f38cb200640 1 Processor -- start 2024-08-03T20:50:26.418 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:26.416+0000 7f38cb200640 1 -- start start 2024-08-03T20:50:26.419 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:26.417+0000 7f38cb200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f38c40cf3b0 0x7f38c40b41d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:50:26.419 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:26.417+0000 7f38cb200640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f38c4155920 0x7f38c41f8930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:50:26.419 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:26.417+0000 7f38cb200640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f38c415ba50 0x7f38c41f8e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:50:26.420 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:26.417+0000 7f38cb200640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f38c40b3260 con 0x7f38c40cf3b0 2024-08-03T20:50:26.420 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:26.417+0000 7f38cb200640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f38c40b30e0 con 0x7f38c415ba50 2024-08-03T20:50:26.420 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:26.417+0000 7f38cb200640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7f38c40b33e0 con 0x7f38c4155920 2024-08-03T20:50:26.420 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:26.417+0000 7f38c3e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f38c40cf3b0 0x7f38c40b41d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:50:26.420 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:26.417+0000 7f38c3e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f38c40cf3b0 0x7f38c40b41d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:60650/0 (socket says 172.21.15.32:60650) 2024-08-03T20:50:26.420 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:26.417+0000 7f38c3e00640 1 -- 172.21.15.32:0/196785534 learned_addr learned my addr 172.21.15.32:0/196785534 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:50:26.420 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:26.418+0000 7f38bb400640 1 --2- 172.21.15.32:0/196785534 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f38c4155920 0x7f38c41f8930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:50:26.420 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:26.418+0000 7f38c9200640 1 --2- 172.21.15.32:0/196785534 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f38c415ba50 0x7f38c41f8e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:50:26.420 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:26.418+0000 7f38c3e00640 1 -- 172.21.15.32:0/196785534 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f38c4155920 msgr2=0x7f38c41f8930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:50:26.420 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:26.418+0000 7f38c3e00640 1 --2- 172.21.15.32:0/196785534 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f38c4155920 0x7f38c41f8930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:26.420 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:26.418+0000 7f38c3e00640 1 -- 172.21.15.32:0/196785534 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f38c415ba50 msgr2=0x7f38c41f8e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:50:26.420 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:26.418+0000 7f38c3e00640 1 --2- 172.21.15.32:0/196785534 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f38c415ba50 0x7f38c41f8e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:26.420 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:26.418+0000 7f38c3e00640 1 -- 172.21.15.32:0/196785534 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f38c41f9680 con 0x7f38c40cf3b0 2024-08-03T20:50:26.421 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:26.418+0000 7f38c9200640 1 --2- 172.21.15.32:0/196785534 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f38c415ba50 0x7f38c41f8e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-03T20:50:26.421 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:26.418+0000 7f38c3e00640 1 --2- 172.21.15.32:0/196785534 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f38c40cf3b0 0x7f38c40b41d0 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7f38b4034040 tx=0x7f38b40035f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:50:26.421 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:26.418+0000 7f38c0c00640 1 -- 172.21.15.32:0/196785534 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38b403e030 con 0x7f38c40cf3b0 2024-08-03T20:50:26.422 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:26.418+0000 7f38c0c00640 1 -- 172.21.15.32:0/196785534 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f38b4003ae0 con 0x7f38c40cf3b0 2024-08-03T20:50:26.422 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:26.418+0000 7f38c0c00640 1 -- 172.21.15.32:0/196785534 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38b4035040 con 0x7f38c40cf3b0 2024-08-03T20:50:26.422 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:26.418+0000 7f38cb200640 1 -- 172.21.15.32:0/196785534 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f38c41561b0 con 0x7f38c40cf3b0 2024-08-03T20:50:26.422 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:26.419+0000 7f38cb200640 1 -- 172.21.15.32:0/196785534 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f38c4156770 con 0x7f38c40cf3b0 2024-08-03T20:50:26.422 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:26.420+0000 7f38cb200640 1 -- 172.21.15.32:0/196785534 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f38c4070620 con 0x7f38c40cf3b0 2024-08-03T20:50:26.422 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:26.421+0000 7f38c0c00640 1 -- 172.21.15.32:0/196785534 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f38b4041030 con 0x7f38c40cf3b0 2024-08-03T20:50:26.423 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:26.422+0000 7f38c0c00640 1 --2- 172.21.15.32:0/196785534 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f38a0077010 0x7f38a0079520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:50:26.423 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:26.422+0000 7f38c0c00640 1 -- 172.21.15.32:0/196785534 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(54..54 src has 1..54) ==== 5227+0+0 (secure 0 0 0) 0x7f38b40b8b90 con 0x7f38c40cf3b0 2024-08-03T20:50:26.424 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:26.422+0000 7f38bb400640 1 --2- 172.21.15.32:0/196785534 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f38a0077010 0x7f38a0079520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:50:26.424 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:26.423+0000 7f38bb400640 1 --2- 172.21.15.32:0/196785534 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f38a0077010 0x7f38a0079520 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f38c4157400 tx=0x7f38ac005f90 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:50:26.429 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:26.427+0000 7f38c0c00640 1 -- 172.21.15.32:0/196785534 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7f38c4070620 con 0x7f38c40cf3b0 2024-08-03T20:50:26.634 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:26.633+0000 7f38cb200640 1 -- 172.21.15.32:0/196785534 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) -- 0x7f38c40b3890 con 0x7f38c40cf3b0 2024-08-03T20:50:27.151 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:27.149+0000 7f38c0c00640 1 -- 172.21.15.32:0/196785534 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v55) ==== 170+0+0 (secure 0 0 0) 0x7f38b4081ff0 con 0x7f38c40cf3b0 2024-08-03T20:50:27.151 INFO:teuthology.orchestra.run.smithi032.stderr:pool 'cephfs_data' created 2024-08-03T20:50:27.154 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:27.152+0000 7f38cb200640 1 -- 172.21.15.32:0/196785534 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f38a0077010 msgr2=0x7f38a0079520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:50:27.154 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:27.152+0000 7f38cb200640 1 --2- 172.21.15.32:0/196785534 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f38a0077010 0x7f38a0079520 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f38c4157400 tx=0x7f38ac005f90 comp rx=0 tx=0).stop 2024-08-03T20:50:27.154 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:27.152+0000 7f38cb200640 1 -- 172.21.15.32:0/196785534 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f38c40cf3b0 msgr2=0x7f38c40b41d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:50:27.154 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:27.152+0000 7f38cb200640 1 --2- 172.21.15.32:0/196785534 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f38c40cf3b0 0x7f38c40b41d0 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7f38b4034040 tx=0x7f38b40035f0 comp rx=0 tx=0).stop 2024-08-03T20:50:27.154 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:27.152+0000 7f38cb200640 1 -- 172.21.15.32:0/196785534 shutdown_connections 2024-08-03T20:50:27.154 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:27.152+0000 7f38cb200640 1 --2- 172.21.15.32:0/196785534 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f38a0077010 0x7f38a0079520 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:27.154 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:27.152+0000 7f38cb200640 1 --2- 172.21.15.32:0/196785534 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f38c415ba50 0x7f38c41f8e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:27.154 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:27.152+0000 7f38cb200640 1 --2- 172.21.15.32:0/196785534 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f38c4155920 0x7f38c41f8930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:27.155 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:27.152+0000 7f38cb200640 1 --2- 172.21.15.32:0/196785534 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f38c40cf3b0 0x7f38c40b41d0 unknown :-1 s=CLOSED pgs=556 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:27.155 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:27.152+0000 7f38cb200640 1 -- 172.21.15.32:0/196785534 >> 172.21.15.32:0/196785534 conn(0x7f38c40b1bd0 msgr2=0x7f38c40cc140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:50:27.155 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:27.152+0000 7f38cb200640 1 -- 172.21.15.32:0/196785534 shutdown_connections 2024-08-03T20:50:27.155 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:27.152+0000 7f38cb200640 1 -- 172.21.15.32:0/196785534 wait complete. 2024-08-03T20:50:27.196 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-08-03T20:50:27.429 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:27.427+0000 7fb329600640 1 -- 172.21.15.32:0/610542346 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb3240cf070 msgr2=0x7fb3240cf480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:50:27.429 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:27.427+0000 7fb329600640 1 --2- 172.21.15.32:0/610542346 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb3240cf070 0x7fb3240cf480 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7fb31c005fd0 tx=0x7fb31c02aa90 comp rx=0 tx=0).stop 2024-08-03T20:50:27.429 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:27.427+0000 7fb329600640 1 -- 172.21.15.32:0/610542346 shutdown_connections 2024-08-03T20:50:27.429 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:27.427+0000 7fb329600640 1 --2- 172.21.15.32:0/610542346 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb3240cf070 0x7fb3240cf480 unknown :-1 s=CLOSED pgs=558 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:27.429 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:27.427+0000 7fb329600640 1 --2- 172.21.15.32:0/610542346 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fb3240d47a0 0x7fb3240ceb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:27.429 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:27.427+0000 7fb329600640 1 -- 172.21.15.32:0/610542346 >> 172.21.15.32:0/610542346 conn(0x7fb32408d7e0 msgr2=0x7fb32409d410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:50:27.429 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:27.427+0000 7fb329600640 1 -- 172.21.15.32:0/610542346 shutdown_connections 2024-08-03T20:50:27.429 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:27.427+0000 7fb329600640 1 -- 172.21.15.32:0/610542346 wait complete. 2024-08-03T20:50:27.431 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:27.429+0000 7fb329600640 1 Processor -- start 2024-08-03T20:50:27.431 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:27.429+0000 7fb329600640 1 -- start start 2024-08-03T20:50:27.434 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:27.431+0000 7fb329600640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb3240cf070 0x7fb3240ca520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:50:27.434 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:27.431+0000 7fb329600640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb3240d26e0 0x7fb324153600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:50:27.434 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:27.431+0000 7fb329600640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fb324153b40 0x7fb324153f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:50:27.434 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:27.431+0000 7fb329600640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fb3240cfb40 con 0x7fb3240cf070 2024-08-03T20:50:27.434 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:27.431+0000 7fb329600640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7fb3240cf9c0 con 0x7fb3240d26e0 2024-08-03T20:50:27.434 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:27.431+0000 7fb329600640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7fb3240cfcc0 con 0x7fb324153b40 2024-08-03T20:50:27.434 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:27.431+0000 7fb323e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb3240cf070 0x7fb3240ca520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:50:27.434 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:27.431+0000 7fb323e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb3240cf070 0x7fb3240ca520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:60670/0 (socket says 172.21.15.32:60670) 2024-08-03T20:50:27.434 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:27.431+0000 7fb323e00640 1 -- 172.21.15.32:0/660214072 learned_addr learned my addr 172.21.15.32:0/660214072 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:50:27.434 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:27.431+0000 7fb328c00640 1 --2- 172.21.15.32:0/660214072 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fb324153b40 0x7fb324153f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:50:27.434 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:27.431+0000 7fb31be00640 1 --2- 172.21.15.32:0/660214072 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb3240d26e0 0x7fb324153600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:50:27.434 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:27.432+0000 7fb323e00640 1 -- 172.21.15.32:0/660214072 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fb324153b40 msgr2=0x7fb324153f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:50:27.435 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:27.432+0000 7fb323e00640 1 --2- 172.21.15.32:0/660214072 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fb324153b40 0x7fb324153f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:27.435 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:27.432+0000 7fb323e00640 1 -- 172.21.15.32:0/660214072 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb3240d26e0 msgr2=0x7fb324153600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:50:27.435 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:27.432+0000 7fb323e00640 1 --2- 172.21.15.32:0/660214072 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb3240d26e0 0x7fb324153600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:27.435 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:27.432+0000 7fb323e00640 1 -- 172.21.15.32:0/660214072 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb3241f8980 con 0x7fb3240cf070 2024-08-03T20:50:27.435 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:27.432+0000 7fb323e00640 1 --2- 172.21.15.32:0/660214072 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb3240cf070 0x7fb3240ca520 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7fb31400c3f0 tx=0x7fb31400c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:50:27.435 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:27.432+0000 7fb320c00640 1 -- 172.21.15.32:0/660214072 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb314017070 con 0x7fb3240cf070 2024-08-03T20:50:27.435 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:27.432+0000 7fb320c00640 1 -- 172.21.15.32:0/660214072 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fb3140053c0 con 0x7fb3240cf070 2024-08-03T20:50:27.435 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:27.432+0000 7fb320c00640 1 -- 172.21.15.32:0/660214072 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb314003850 con 0x7fb3240cf070 2024-08-03T20:50:27.435 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:27.432+0000 7fb329600640 1 -- 172.21.15.32:0/660214072 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb3240caa60 con 0x7fb3240cf070 2024-08-03T20:50:27.435 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:27.432+0000 7fb329600640 1 -- 172.21.15.32:0/660214072 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb3241f9160 con 0x7fb3240cf070 2024-08-03T20:50:27.438 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:27.434+0000 7fb329600640 1 -- 172.21.15.32:0/660214072 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb3240cf480 con 0x7fb3240cf070 2024-08-03T20:50:27.438 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:27.435+0000 7fb320c00640 1 -- 172.21.15.32:0/660214072 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7fb31401a070 con 0x7fb3240cf070 2024-08-03T20:50:27.438 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:27.436+0000 7fb320c00640 1 --2- 172.21.15.32:0/660214072 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fb300077010 0x7fb300079520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:50:27.439 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:27.436+0000 7fb320c00640 1 -- 172.21.15.32:0/660214072 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(55..55 src has 1..55) ==== 5609+0+0 (secure 0 0 0) 0x7fb314097dc0 con 0x7fb3240cf070 2024-08-03T20:50:27.439 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:27.436+0000 7fb31be00640 1 --2- 172.21.15.32:0/660214072 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fb300077010 0x7fb300079520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:50:27.439 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:27.437+0000 7fb31be00640 1 --2- 172.21.15.32:0/660214072 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fb300077010 0x7fb300079520 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7fb324162920 tx=0x7fb30c005f90 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:50:27.443 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:27.441+0000 7fb320c00640 1 -- 172.21.15.32:0/660214072 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7fb314060f80 con 0x7fb3240cf070 2024-08-03T20:50:27.688 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:27.686+0000 7fb329600640 1 -- 172.21.15.32:0/660214072 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) -- 0x7fb3240b2ae0 con 0x7fb3240cf070 2024-08-03T20:50:28.153 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:28.152+0000 7fb320c00640 1 -- 172.21.15.32:0/660214072 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 8 and data pool 9 v47) ==== 172+0+0 (secure 0 0 0) 0x7fb314069250 con 0x7fb3240cf070 2024-08-03T20:50:28.153 INFO:teuthology.orchestra.run.smithi032.stderr:new fs with metadata pool 8 and data pool 9 2024-08-03T20:50:28.157 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:28.155+0000 7fb329600640 1 -- 172.21.15.32:0/660214072 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fb300077010 msgr2=0x7fb300079520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:50:28.157 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:28.155+0000 7fb329600640 1 --2- 172.21.15.32:0/660214072 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fb300077010 0x7fb300079520 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7fb324162920 tx=0x7fb30c005f90 comp rx=0 tx=0).stop 2024-08-03T20:50:28.157 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:28.155+0000 7fb329600640 1 -- 172.21.15.32:0/660214072 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb3240cf070 msgr2=0x7fb3240ca520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:50:28.157 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:28.155+0000 7fb329600640 1 --2- 172.21.15.32:0/660214072 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb3240cf070 0x7fb3240ca520 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7fb31400c3f0 tx=0x7fb31400c8b0 comp rx=0 tx=0).stop 2024-08-03T20:50:28.158 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:28.156+0000 7fb329600640 1 -- 172.21.15.32:0/660214072 shutdown_connections 2024-08-03T20:50:28.158 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:28.156+0000 7fb329600640 1 --2- 172.21.15.32:0/660214072 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fb300077010 0x7fb300079520 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:28.158 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:28.156+0000 7fb329600640 1 --2- 172.21.15.32:0/660214072 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb3240d26e0 0x7fb324153600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:28.158 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:28.156+0000 7fb329600640 1 --2- 172.21.15.32:0/660214072 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fb324153b40 0x7fb324153f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:28.158 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:28.156+0000 7fb329600640 1 --2- 172.21.15.32:0/660214072 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb3240cf070 0x7fb3240ca520 unknown :-1 s=CLOSED pgs=559 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:28.158 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:28.156+0000 7fb329600640 1 -- 172.21.15.32:0/660214072 >> 172.21.15.32:0/660214072 conn(0x7fb32408d7e0 msgr2=0x7fb324162a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:50:28.159 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:28.158+0000 7fb329600640 1 -- 172.21.15.32:0/660214072 shutdown_connections 2024-08-03T20:50:28.161 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:28.160+0000 7fb329600640 1 -- 172.21.15.32:0/660214072 wait complete. 2024-08-03T20:50:28.212 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-08-03T20:50:28.443 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:28.441+0000 7f38a4200640 1 -- 172.21.15.32:0/101152838 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f389c15f610 msgr2=0x7f389c1523f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:50:28.443 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:28.441+0000 7f38a4200640 1 --2- 172.21.15.32:0/101152838 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f389c15f610 0x7f389c1523f0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f3888009830 tx=0x7f388802a530 comp rx=0 tx=0).stop 2024-08-03T20:50:28.443 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:28.441+0000 7f38a4200640 1 -- 172.21.15.32:0/101152838 shutdown_connections 2024-08-03T20:50:28.444 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:28.441+0000 7f38a4200640 1 --2- 172.21.15.32:0/101152838 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f389c15f610 0x7f389c1523f0 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:28.444 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:28.441+0000 7f38a4200640 1 -- 172.21.15.32:0/101152838 >> 172.21.15.32:0/101152838 conn(0x7f389c0b1bd0 msgr2=0x7f389c0b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:50:28.444 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:28.441+0000 7f38a4200640 1 -- 172.21.15.32:0/101152838 shutdown_connections 2024-08-03T20:50:28.444 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:28.441+0000 7f38a4200640 1 -- 172.21.15.32:0/101152838 wait complete. 2024-08-03T20:50:28.445 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:28.444+0000 7f38a4200640 1 Processor -- start 2024-08-03T20:50:28.445 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:28.444+0000 7f38a4200640 1 -- start start 2024-08-03T20:50:28.448 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:28.445+0000 7f38a4200640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f389c0b38b0 0x7f389c1f9500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:50:28.448 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:28.445+0000 7f38a4200640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f389c0b4ce0 0x7f389c14f370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:50:28.448 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:28.445+0000 7f38a4200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f389c15d2a0 0x7f389c15d6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:50:28.449 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:28.445+0000 7f38a4200640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f389c0cec50 con 0x7f389c15d2a0 2024-08-03T20:50:28.449 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:28.445+0000 7f38a4200640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f389c0cead0 con 0x7f389c0b4ce0 2024-08-03T20:50:28.449 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:28.445+0000 7f38a4200640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7f389c0cedd0 con 0x7f389c0b38b0 2024-08-03T20:50:28.449 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:28.445+0000 7f38a2200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f389c15d2a0 0x7f389c15d6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:50:28.449 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:28.446+0000 7f38a2200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f389c15d2a0 0x7f389c15d6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:60674/0 (socket says 172.21.15.32:60674) 2024-08-03T20:50:28.449 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:28.446+0000 7f38a2200640 1 -- 172.21.15.32:0/4182190270 learned_addr learned my addr 172.21.15.32:0/4182190270 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:50:28.449 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:28.446+0000 7f38a0e00640 1 --2- 172.21.15.32:0/4182190270 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f389c0b4ce0 0x7f389c14f370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:50:28.449 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:28.446+0000 7f38a1800640 1 --2- 172.21.15.32:0/4182190270 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f389c0b38b0 0x7f389c1f9500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:50:28.449 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:28.446+0000 7f38a2200640 1 -- 172.21.15.32:0/4182190270 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f389c0b38b0 msgr2=0x7f389c1f9500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:50:28.449 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:28.446+0000 7f38a2200640 1 --2- 172.21.15.32:0/4182190270 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f389c0b38b0 0x7f389c1f9500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:28.449 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:28.446+0000 7f38a2200640 1 -- 172.21.15.32:0/4182190270 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f389c0b4ce0 msgr2=0x7f389c14f370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:50:28.450 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:28.446+0000 7f38a2200640 1 --2- 172.21.15.32:0/4182190270 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f389c0b4ce0 0x7f389c14f370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:28.450 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:28.446+0000 7f38a2200640 1 -- 172.21.15.32:0/4182190270 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f389c15dbf0 con 0x7f389c15d2a0 2024-08-03T20:50:28.450 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:28.446+0000 7f38a2200640 1 --2- 172.21.15.32:0/4182190270 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f389c15d2a0 0x7f389c15d6b0 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7f389400caf0 tx=0x7f389400a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:50:28.450 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:28.447+0000 7f3899600640 1 -- 172.21.15.32:0/4182190270 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3894016020 con 0x7f389c15d2a0 2024-08-03T20:50:28.450 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:28.447+0000 7f3899600640 1 -- 172.21.15.32:0/4182190270 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f389400adb0 con 0x7f389c15d2a0 2024-08-03T20:50:28.450 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:28.447+0000 7f3899600640 1 -- 172.21.15.32:0/4182190270 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3894012680 con 0x7f389c15d2a0 2024-08-03T20:50:28.450 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:28.447+0000 7f38a4200640 1 -- 172.21.15.32:0/4182190270 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f389c1f6430 con 0x7f389c15d2a0 2024-08-03T20:50:28.450 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:28.447+0000 7f38a4200640 1 -- 172.21.15.32:0/4182190270 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f389c1f5950 con 0x7f389c15d2a0 2024-08-03T20:50:28.452 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:28.448+0000 7f38a4200640 1 -- 172.21.15.32:0/4182190270 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f389c0b1640 con 0x7f389c15d2a0 2024-08-03T20:50:28.452 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:28.449+0000 7f3899600640 1 -- 172.21.15.32:0/4182190270 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f3894019020 con 0x7f389c15d2a0 2024-08-03T20:50:28.452 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:28.450+0000 7f3899600640 1 --2- 172.21.15.32:0/4182190270 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f387c077010 0x7f387c079520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:50:28.452 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:28.450+0000 7f3899600640 1 -- 172.21.15.32:0/4182190270 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(56..56 src has 1..56) ==== 5709+0+0 (secure 0 0 0) 0x7f3894098fa0 con 0x7f389c15d2a0 2024-08-03T20:50:28.452 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:28.451+0000 7f38a1800640 1 --2- 172.21.15.32:0/4182190270 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f387c077010 0x7f387c079520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:50:28.453 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:28.451+0000 7f38a1800640 1 --2- 172.21.15.32:0/4182190270 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f387c077010 0x7f387c079520 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f3890001120 tx=0x7f3890009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:50:28.456 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:28.455+0000 7f3899600640 1 -- 172.21.15.32:0/4182190270 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7f3894020020 con 0x7f389c15d2a0 2024-08-03T20:50:28.666 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:28.665+0000 7f38a4200640 1 -- 172.21.15.32:0/4182190270 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f389c0adfd0 con 0x7f389c15d2a0 2024-08-03T20:50:28.668 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:28.666+0000 7f3899600640 1 -- 172.21.15.32:0/4182190270 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v56) ==== 74+0+16618 (secure 0 0 0) 0x7f389c0adfd0 con 0x7f389c15d2a0 2024-08-03T20:50:28.669 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-03T20:50:28.669 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":56,"fsid":"40ab141a-01d7-49b9-ae3c-2edec9876e95","created":"2024-08-03T20:47:14.957362+0000","modified":"2024-08-03T20:50:27.688743+0000","last_up_change":"2024-08-03T20:47:28.874451+0000","last_in_change":"2024-08-03T20:47:20.079442+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-08-03T20:47:28.261300+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_metadata","create_time":"2024-08-03T20:50:25.652248+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"56","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_data","create_time":"2024-08-03T20:50:26.646008+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"56","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5a83e759-9035-438b-889a-0a6af1f0a0d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6816","nonce":3066156116},{"type":"v1","addr":"172.21.15.32:6817","nonce":3066156116}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6818","nonce":3066156116},{"type":"v1","addr":"172.21.15.32:6819","nonce":3066156116}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6822","nonce":3066156116},{"type":"v1","addr":"172.21.15.32:6823","nonce":3066156116}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6820","nonce":3066156116},{"type":"v1","addr":"172.21.15.32:6821","nonce":3066156116}]},"public_addr":"172.21.15.32:6817/3066156116","cluster_addr":"172.21.15.32:6819/3066156116","heartbeat_back_addr":"172.21.15.32:6823/3066156116","heartbeat_front_addr":"172.21.15.32:6821/3066156116","state":["exists","up"]},{"osd":1,"uuid":"d7c5c210-da4f-49f9-b9e7-1dfb02daa4bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6824","nonce":1551241759},{"type":"v1","addr":"172.21.15.32:6825","nonce":1551241759}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6826","nonce":1551241759},{"type":"v1","addr":"172.21.15.32:6827","nonce":1551241759}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6830","nonce":1551241759},{"type":"v1","addr":"172.21.15.32:6831","nonce":1551241759}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6828","nonce":1551241759},{"type":"v1","addr":"172.21.15.32:6829","nonce":1551241759}]},"public_addr":"172.21.15.32:6825/1551241759","cluster_addr":"172.21.15.32:6827/1551241759","heartbeat_back_addr":"172.21.15.32:6831/1551241759","heartbeat_front_addr":"172.21.15.32:6829/1551241759","state":["exists","up"]},{"osd":2,"uuid":"57f2e6de-a201-45c2-8981-8e56a4875ebd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6808","nonce":3806708944},{"type":"v1","addr":"172.21.15.32:6809","nonce":3806708944}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6810","nonce":3806708944},{"type":"v1","addr":"172.21.15.32:6811","nonce":3806708944}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6814","nonce":3806708944},{"type":"v1","addr":"172.21.15.32:6815","nonce":3806708944}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6812","nonce":3806708944},{"type":"v1","addr":"172.21.15.32:6813","nonce":3806708944}]},"public_addr":"172.21.15.32:6809/3806708944","cluster_addr":"172.21.15.32:6811/3806708944","heartbeat_back_addr":"172.21.15.32:6815/3806708944","heartbeat_front_addr":"172.21.15.32:6813/3806708944","state":["exists","up"]},{"osd":3,"uuid":"6439a280-da45-4148-8130-f16323935ff3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6800","nonce":9088303},{"type":"v1","addr":"172.21.15.32:6801","nonce":9088303}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6802","nonce":9088303},{"type":"v1","addr":"172.21.15.32:6803","nonce":9088303}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6806","nonce":9088303},{"type":"v1","addr":"172.21.15.32:6807","nonce":9088303}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6804","nonce":9088303},{"type":"v1","addr":"172.21.15.32:6805","nonce":9088303}]},"public_addr":"172.21.15.32:6801/9088303","cluster_addr":"172.21.15.32:6803/9088303","heartbeat_back_addr":"172.21.15.32:6807/9088303","heartbeat_front_addr":"172.21.15.32:6805/9088303","state":["exists","up"]},{"osd":4,"uuid":"4f902eff-1e18-418d-82c9-2151c69524fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6802","nonce":864752275},{"type":"v1","addr":"172.21.15.118:6803","nonce":864752275}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6804","nonce":864752275},{"type":"v1","addr":"172.21.15.118:6805","nonce":864752275}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6808","nonce":864752275},{"type":"v1","addr":"172.21.15.118:6809","nonce":864752275}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6806","nonce":864752275},{"type":"v1","addr":"172.21.15.118:6807","nonce":864752275}]},"public_addr":"172.21.15.118:6803/864752275","cluster_addr":"172.21.15.118:6805/864752275","heartbeat_back_addr":"172.21.15.118:6809/864752275","heartbeat_front_addr":"172.21.15.118:6807/864752275","state":["exists","up"]},{"osd":5,"uuid":"bfd9feac-dd06-458c-91b9-af7f175f612c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6810","nonce":2724891770},{"type":"v1","addr":"172.21.15.118:6811","nonce":2724891770}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6812","nonce":2724891770},{"type":"v1","addr":"172.21.15.118:6813","nonce":2724891770}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6816","nonce":2724891770},{"type":"v1","addr":"172.21.15.118:6817","nonce":2724891770}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6814","nonce":2724891770},{"type":"v1","addr":"172.21.15.118:6815","nonce":2724891770}]},"public_addr":"172.21.15.118:6811/2724891770","cluster_addr":"172.21.15.118:6813/2724891770","heartbeat_back_addr":"172.21.15.118:6817/2724891770","heartbeat_front_addr":"172.21.15.118:6815/2724891770","state":["exists","up"]},{"osd":6,"uuid":"5ed5a279-9d8e-416d-bbc7-7edd3e6d5705","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6825","nonce":801947386},{"type":"v1","addr":"172.21.15.118:6827","nonce":801947386}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6828","nonce":801947386},{"type":"v1","addr":"172.21.15.118:6829","nonce":801947386}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6832","nonce":801947386},{"type":"v1","addr":"172.21.15.118:6833","nonce":801947386}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6830","nonce":801947386},{"type":"v1","addr":"172.21.15.118:6831","nonce":801947386}]},"public_addr":"172.21.15.118:6827/801947386","cluster_addr":"172.21.15.118:6829/801947386","heartbeat_back_addr":"172.21.15.118:6833/801947386","heartbeat_front_addr":"172.21.15.118:6831/801947386","state":["exists","up"]},{"osd":7,"uuid":"f2086c47-875e-4451-a436-2008d1ea16af","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6818","nonce":653000048},{"type":"v1","addr":"172.21.15.118:6819","nonce":653000048}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6820","nonce":653000048},{"type":"v1","addr":"172.21.15.118:6821","nonce":653000048}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6824","nonce":653000048},{"type":"v1","addr":"172.21.15.118:6826","nonce":653000048}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6822","nonce":653000048},{"type":"v1","addr":"172.21.15.118:6823","nonce":653000048}]},"public_addr":"172.21.15.118:6819/653000048","cluster_addr":"172.21.15.118:6821/653000048","heartbeat_back_addr":"172.21.15.118:6826/653000048","heartbeat_front_addr":"172.21.15.118:6823/653000048","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-03T20:47:25.491994+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-03T20:47:25.587358+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-03T20:47:25.454175+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-03T20:47:25.377734+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-03T20:47:25.412351+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-03T20:47:25.499815+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-03T20:47:25.575044+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-03T20:47:25.552383+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-08-03T20:50:28.671 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:28.669+0000 7f38a4200640 1 -- 172.21.15.32:0/4182190270 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f387c077010 msgr2=0x7f387c079520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:50:28.671 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:28.669+0000 7f38a4200640 1 --2- 172.21.15.32:0/4182190270 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f387c077010 0x7f387c079520 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f3890001120 tx=0x7f3890009040 comp rx=0 tx=0).stop 2024-08-03T20:50:28.671 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:28.669+0000 7f38a4200640 1 -- 172.21.15.32:0/4182190270 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f389c15d2a0 msgr2=0x7f389c15d6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:50:28.671 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:28.669+0000 7f38a4200640 1 --2- 172.21.15.32:0/4182190270 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f389c15d2a0 0x7f389c15d6b0 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7f389400caf0 tx=0x7f389400a060 comp rx=0 tx=0).stop 2024-08-03T20:50:28.671 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:28.669+0000 7f38a4200640 1 -- 172.21.15.32:0/4182190270 shutdown_connections 2024-08-03T20:50:28.671 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:28.669+0000 7f38a4200640 1 --2- 172.21.15.32:0/4182190270 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f387c077010 0x7f387c079520 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:28.671 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:28.669+0000 7f38a4200640 1 --2- 172.21.15.32:0/4182190270 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f389c0b4ce0 0x7f389c14f370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:28.672 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:28.669+0000 7f38a4200640 1 --2- 172.21.15.32:0/4182190270 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f389c15d2a0 0x7f389c15d6b0 unknown :-1 s=CLOSED pgs=561 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:28.672 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:28.669+0000 7f38a4200640 1 --2- 172.21.15.32:0/4182190270 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f389c0b38b0 0x7f389c1f9500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:28.672 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:28.669+0000 7f38a4200640 1 -- 172.21.15.32:0/4182190270 >> 172.21.15.32:0/4182190270 conn(0x7f389c0b1bd0 msgr2=0x7f389c0ad8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:50:28.672 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:28.669+0000 7f38a4200640 1 -- 172.21.15.32:0/4182190270 shutdown_connections 2024-08-03T20:50:28.672 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:28.669+0000 7f38a4200640 1 -- 172.21.15.32:0/4182190270 wait complete. 2024-08-03T20:50:28.696 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-08-03T20:50:28.924 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:28.922+0000 7f859e200640 1 -- 172.21.15.32:0/2752715930 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f859806e930 msgr2=0x7f8598144f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:50:28.924 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:28.922+0000 7f859e200640 1 --2- 172.21.15.32:0/2752715930 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f859806e930 0x7f8598144f90 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7f8588005fd0 tx=0x7f858802a530 comp rx=0 tx=0).stop 2024-08-03T20:50:28.925 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:28.923+0000 7f859e200640 1 -- 172.21.15.32:0/2752715930 shutdown_connections 2024-08-03T20:50:28.925 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:28.923+0000 7f859e200640 1 --2- 172.21.15.32:0/2752715930 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f859815fff0 0x7f8598160400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:28.925 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:28.923+0000 7f859e200640 1 --2- 172.21.15.32:0/2752715930 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f859806e930 0x7f8598144f90 unknown :-1 s=CLOSED pgs=562 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:28.925 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:28.923+0000 7f859e200640 1 -- 172.21.15.32:0/2752715930 >> 172.21.15.32:0/2752715930 conn(0x7f85980b1bd0 msgr2=0x7f85980b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:50:28.925 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:28.923+0000 7f859e200640 1 -- 172.21.15.32:0/2752715930 shutdown_connections 2024-08-03T20:50:28.925 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:28.923+0000 7f859e200640 1 -- 172.21.15.32:0/2752715930 wait complete. 2024-08-03T20:50:28.926 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:28.925+0000 7f859e200640 1 Processor -- start 2024-08-03T20:50:28.926 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:28.925+0000 7f859e200640 1 -- start start 2024-08-03T20:50:28.928 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:28.926+0000 7f859e200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f859806e930 0x7f8598144610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:50:28.929 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:28.926+0000 7f859e200640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f8598073e60 0x7f859815dc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:50:28.929 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:28.926+0000 7f859e200640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f859815fff0 0x7f859815e170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:50:28.929 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:28.926+0000 7f859e200640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f85980cf840 con 0x7f859806e930 2024-08-03T20:50:28.929 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:28.926+0000 7f859e200640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f8598145f30 con 0x7f859815fff0 2024-08-03T20:50:28.929 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:28.926+0000 7f859e200640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7f85980cf9c0 con 0x7f8598073e60 2024-08-03T20:50:28.929 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:28.927+0000 7f8597400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f859806e930 0x7f8598144610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:50:28.929 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:28.927+0000 7f8597400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f859806e930 0x7f8598144610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:60694/0 (socket says 172.21.15.32:60694) 2024-08-03T20:50:28.929 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:28.927+0000 7f8597400640 1 -- 172.21.15.32:0/2430493358 learned_addr learned my addr 172.21.15.32:0/2430493358 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:50:28.929 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:28.927+0000 7f8596a00640 1 --2- 172.21.15.32:0/2430493358 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f8598073e60 0x7f859815dc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:50:28.929 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:28.927+0000 7f8597e00640 1 --2- 172.21.15.32:0/2430493358 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f859815fff0 0x7f859815e170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:50:28.929 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:28.927+0000 7f8597400640 1 -- 172.21.15.32:0/2430493358 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f8598073e60 msgr2=0x7f859815dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:50:28.929 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:28.927+0000 7f8597400640 1 --2- 172.21.15.32:0/2430493358 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f8598073e60 0x7f859815dc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:28.930 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:28.927+0000 7f8597400640 1 -- 172.21.15.32:0/2430493358 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f859815fff0 msgr2=0x7f859815e170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:50:28.930 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:28.927+0000 7f8597400640 1 --2- 172.21.15.32:0/2430493358 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f859815fff0 0x7f859815e170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:28.930 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:28.927+0000 7f8597400640 1 -- 172.21.15.32:0/2430493358 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8598144b50 con 0x7f859806e930 2024-08-03T20:50:28.930 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:28.927+0000 7f8597400640 1 --2- 172.21.15.32:0/2430493358 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f859806e930 0x7f8598144610 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7f8588034040 tx=0x7f8588003970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:50:28.930 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:28.928+0000 7f858f400640 1 -- 172.21.15.32:0/2430493358 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f858803e030 con 0x7f859806e930 2024-08-03T20:50:28.931 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:28.928+0000 7f858f400640 1 -- 172.21.15.32:0/2430493358 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f85880047a0 con 0x7f859806e930 2024-08-03T20:50:28.931 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:28.928+0000 7f858f400640 1 -- 172.21.15.32:0/2430493358 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8588035040 con 0x7f859806e930 2024-08-03T20:50:28.931 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:28.928+0000 7f859e200640 1 -- 172.21.15.32:0/2430493358 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f859815ea40 con 0x7f859806e930 2024-08-03T20:50:28.931 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:28.928+0000 7f859e200640 1 -- 172.21.15.32:0/2430493358 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f85980b33c0 con 0x7f859806e930 2024-08-03T20:50:28.931 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:28.929+0000 7f859e200640 1 -- 172.21.15.32:0/2430493358 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8598160d10 con 0x7f859806e930 2024-08-03T20:50:28.931 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:28.930+0000 7f858f400640 1 -- 172.21.15.32:0/2430493358 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f8588041030 con 0x7f859806e930 2024-08-03T20:50:28.932 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:28.931+0000 7f858f400640 1 --2- 172.21.15.32:0/2430493358 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f8578077040 0x7f8578079550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:50:28.933 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:28.931+0000 7f858f400640 1 -- 172.21.15.32:0/2430493358 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(56..56 src has 1..56) ==== 5709+0+0 (secure 0 0 0) 0x7f85880b9250 con 0x7f859806e930 2024-08-03T20:50:28.933 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:28.932+0000 7f8596a00640 1 --2- 172.21.15.32:0/2430493358 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f8578077040 0x7f8578079550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:50:28.934 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:28.932+0000 7f8596a00640 1 --2- 172.21.15.32:0/2430493358 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f8578077040 0x7f8578079550 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f8584006fd0 tx=0x7f8584008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:50:28.937 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:28.936+0000 7f858f400640 1 -- 172.21.15.32:0/2430493358 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7f85880823b0 con 0x7f859806e930 2024-08-03T20:50:29.141 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:29.140+0000 7f859e200640 1 -- 172.21.15.32:0/2430493358 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f8598074270 con 0x7f859806e930 2024-08-03T20:50:29.142 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:29.141+0000 7f858f400640 1 -- 172.21.15.32:0/2430493358 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v56) ==== 74+0+16618 (secure 0 0 0) 0x7f8588081cb0 con 0x7f859806e930 2024-08-03T20:50:29.143 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-03T20:50:29.143 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":56,"fsid":"40ab141a-01d7-49b9-ae3c-2edec9876e95","created":"2024-08-03T20:47:14.957362+0000","modified":"2024-08-03T20:50:27.688743+0000","last_up_change":"2024-08-03T20:47:28.874451+0000","last_in_change":"2024-08-03T20:47:20.079442+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-08-03T20:47:28.261300+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_metadata","create_time":"2024-08-03T20:50:25.652248+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"56","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_data","create_time":"2024-08-03T20:50:26.646008+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"56","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5a83e759-9035-438b-889a-0a6af1f0a0d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6816","nonce":3066156116},{"type":"v1","addr":"172.21.15.32:6817","nonce":3066156116}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6818","nonce":3066156116},{"type":"v1","addr":"172.21.15.32:6819","nonce":3066156116}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6822","nonce":3066156116},{"type":"v1","addr":"172.21.15.32:6823","nonce":3066156116}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6820","nonce":3066156116},{"type":"v1","addr":"172.21.15.32:6821","nonce":3066156116}]},"public_addr":"172.21.15.32:6817/3066156116","cluster_addr":"172.21.15.32:6819/3066156116","heartbeat_back_addr":"172.21.15.32:6823/3066156116","heartbeat_front_addr":"172.21.15.32:6821/3066156116","state":["exists","up"]},{"osd":1,"uuid":"d7c5c210-da4f-49f9-b9e7-1dfb02daa4bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6824","nonce":1551241759},{"type":"v1","addr":"172.21.15.32:6825","nonce":1551241759}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6826","nonce":1551241759},{"type":"v1","addr":"172.21.15.32:6827","nonce":1551241759}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6830","nonce":1551241759},{"type":"v1","addr":"172.21.15.32:6831","nonce":1551241759}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6828","nonce":1551241759},{"type":"v1","addr":"172.21.15.32:6829","nonce":1551241759}]},"public_addr":"172.21.15.32:6825/1551241759","cluster_addr":"172.21.15.32:6827/1551241759","heartbeat_back_addr":"172.21.15.32:6831/1551241759","heartbeat_front_addr":"172.21.15.32:6829/1551241759","state":["exists","up"]},{"osd":2,"uuid":"57f2e6de-a201-45c2-8981-8e56a4875ebd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6808","nonce":3806708944},{"type":"v1","addr":"172.21.15.32:6809","nonce":3806708944}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6810","nonce":3806708944},{"type":"v1","addr":"172.21.15.32:6811","nonce":3806708944}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6814","nonce":3806708944},{"type":"v1","addr":"172.21.15.32:6815","nonce":3806708944}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6812","nonce":3806708944},{"type":"v1","addr":"172.21.15.32:6813","nonce":3806708944}]},"public_addr":"172.21.15.32:6809/3806708944","cluster_addr":"172.21.15.32:6811/3806708944","heartbeat_back_addr":"172.21.15.32:6815/3806708944","heartbeat_front_addr":"172.21.15.32:6813/3806708944","state":["exists","up"]},{"osd":3,"uuid":"6439a280-da45-4148-8130-f16323935ff3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6800","nonce":9088303},{"type":"v1","addr":"172.21.15.32:6801","nonce":9088303}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6802","nonce":9088303},{"type":"v1","addr":"172.21.15.32:6803","nonce":9088303}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6806","nonce":9088303},{"type":"v1","addr":"172.21.15.32:6807","nonce":9088303}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6804","nonce":9088303},{"type":"v1","addr":"172.21.15.32:6805","nonce":9088303}]},"public_addr":"172.21.15.32:6801/9088303","cluster_addr":"172.21.15.32:6803/9088303","heartbeat_back_addr":"172.21.15.32:6807/9088303","heartbeat_front_addr":"172.21.15.32:6805/9088303","state":["exists","up"]},{"osd":4,"uuid":"4f902eff-1e18-418d-82c9-2151c69524fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6802","nonce":864752275},{"type":"v1","addr":"172.21.15.118:6803","nonce":864752275}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6804","nonce":864752275},{"type":"v1","addr":"172.21.15.118:6805","nonce":864752275}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6808","nonce":864752275},{"type":"v1","addr":"172.21.15.118:6809","nonce":864752275}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6806","nonce":864752275},{"type":"v1","addr":"172.21.15.118:6807","nonce":864752275}]},"public_addr":"172.21.15.118:6803/864752275","cluster_addr":"172.21.15.118:6805/864752275","heartbeat_back_addr":"172.21.15.118:6809/864752275","heartbeat_front_addr":"172.21.15.118:6807/864752275","state":["exists","up"]},{"osd":5,"uuid":"bfd9feac-dd06-458c-91b9-af7f175f612c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6810","nonce":2724891770},{"type":"v1","addr":"172.21.15.118:6811","nonce":2724891770}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6812","nonce":2724891770},{"type":"v1","addr":"172.21.15.118:6813","nonce":2724891770}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6816","nonce":2724891770},{"type":"v1","addr":"172.21.15.118:6817","nonce":2724891770}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6814","nonce":2724891770},{"type":"v1","addr":"172.21.15.118:6815","nonce":2724891770}]},"public_addr":"172.21.15.118:6811/2724891770","cluster_addr":"172.21.15.118:6813/2724891770","heartbeat_back_addr":"172.21.15.118:6817/2724891770","heartbeat_front_addr":"172.21.15.118:6815/2724891770","state":["exists","up"]},{"osd":6,"uuid":"5ed5a279-9d8e-416d-bbc7-7edd3e6d5705","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6825","nonce":801947386},{"type":"v1","addr":"172.21.15.118:6827","nonce":801947386}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6828","nonce":801947386},{"type":"v1","addr":"172.21.15.118:6829","nonce":801947386}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6832","nonce":801947386},{"type":"v1","addr":"172.21.15.118:6833","nonce":801947386}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6830","nonce":801947386},{"type":"v1","addr":"172.21.15.118:6831","nonce":801947386}]},"public_addr":"172.21.15.118:6827/801947386","cluster_addr":"172.21.15.118:6829/801947386","heartbeat_back_addr":"172.21.15.118:6833/801947386","heartbeat_front_addr":"172.21.15.118:6831/801947386","state":["exists","up"]},{"osd":7,"uuid":"f2086c47-875e-4451-a436-2008d1ea16af","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6818","nonce":653000048},{"type":"v1","addr":"172.21.15.118:6819","nonce":653000048}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6820","nonce":653000048},{"type":"v1","addr":"172.21.15.118:6821","nonce":653000048}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6824","nonce":653000048},{"type":"v1","addr":"172.21.15.118:6826","nonce":653000048}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6822","nonce":653000048},{"type":"v1","addr":"172.21.15.118:6823","nonce":653000048}]},"public_addr":"172.21.15.118:6819/653000048","cluster_addr":"172.21.15.118:6821/653000048","heartbeat_back_addr":"172.21.15.118:6826/653000048","heartbeat_front_addr":"172.21.15.118:6823/653000048","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-03T20:47:25.491994+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-03T20:47:25.587358+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-03T20:47:25.454175+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-03T20:47:25.377734+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-03T20:47:25.412351+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-03T20:47:25.499815+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-03T20:47:25.575044+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-03T20:47:25.552383+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-08-03T20:50:29.145 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:29.143+0000 7f859e200640 1 -- 172.21.15.32:0/2430493358 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f8578077040 msgr2=0x7f8578079550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:50:29.145 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:29.143+0000 7f859e200640 1 --2- 172.21.15.32:0/2430493358 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f8578077040 0x7f8578079550 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f8584006fd0 tx=0x7f8584008040 comp rx=0 tx=0).stop 2024-08-03T20:50:29.145 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:29.143+0000 7f859e200640 1 -- 172.21.15.32:0/2430493358 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f859806e930 msgr2=0x7f8598144610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:50:29.145 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:29.143+0000 7f859e200640 1 --2- 172.21.15.32:0/2430493358 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f859806e930 0x7f8598144610 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7f8588034040 tx=0x7f8588003970 comp rx=0 tx=0).stop 2024-08-03T20:50:29.145 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:29.143+0000 7f859e200640 1 -- 172.21.15.32:0/2430493358 shutdown_connections 2024-08-03T20:50:29.145 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:29.143+0000 7f859e200640 1 --2- 172.21.15.32:0/2430493358 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f8578077040 0x7f8578079550 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:29.145 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:29.143+0000 7f859e200640 1 --2- 172.21.15.32:0/2430493358 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f859815fff0 0x7f859815e170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:29.145 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:29.143+0000 7f859e200640 1 --2- 172.21.15.32:0/2430493358 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f8598073e60 0x7f859815dc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:29.145 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:29.143+0000 7f859e200640 1 --2- 172.21.15.32:0/2430493358 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f859806e930 0x7f8598144610 unknown :-1 s=CLOSED pgs=563 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:29.146 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:29.143+0000 7f859e200640 1 -- 172.21.15.32:0/2430493358 >> 172.21.15.32:0/2430493358 conn(0x7f85980b1bd0 msgr2=0x7f85980b0d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:50:29.146 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:29.143+0000 7f859e200640 1 -- 172.21.15.32:0/2430493358 shutdown_connections 2024-08-03T20:50:29.146 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:29.143+0000 7f859e200640 1 -- 172.21.15.32:0/2430493358 wait complete. 2024-08-03T20:50:29.172 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-08-03T20:50:29.403 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:29.401+0000 7f1125000640 1 -- 172.21.15.32:0/1231269061 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1120159ff0 msgr2=0x7f112015fe10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:50:29.403 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:29.401+0000 7f1125000640 1 --2- 172.21.15.32:0/1231269061 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1120159ff0 0x7f112015fe10 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7f1108009830 tx=0x7f110802a510 comp rx=0 tx=0).stop 2024-08-03T20:50:29.403 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:29.401+0000 7f1125000640 1 -- 172.21.15.32:0/1231269061 shutdown_connections 2024-08-03T20:50:29.403 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:29.401+0000 7f1125000640 1 --2- 172.21.15.32:0/1231269061 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1120159ff0 0x7f112015fe10 unknown :-1 s=CLOSED pgs=565 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:29.404 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:29.401+0000 7f1125000640 1 --2- 172.21.15.32:0/1231269061 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f11201609d0 0x7f11200cc050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:29.404 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:29.401+0000 7f1125000640 1 -- 172.21.15.32:0/1231269061 >> 172.21.15.32:0/1231269061 conn(0x7f11200cfd80 msgr2=0x7f11200d0190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:50:29.404 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:29.401+0000 7f1125000640 1 -- 172.21.15.32:0/1231269061 shutdown_connections 2024-08-03T20:50:29.404 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:29.401+0000 7f1125000640 1 -- 172.21.15.32:0/1231269061 wait complete. 2024-08-03T20:50:29.405 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:29.404+0000 7f1125000640 1 Processor -- start 2024-08-03T20:50:29.405 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:29.404+0000 7f1125000640 1 -- start start 2024-08-03T20:50:29.408 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:29.405+0000 7f1125000640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f1120159ff0 0x7f11201594f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:50:29.408 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:29.405+0000 7f1125000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1120160350 0x7f112014dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:50:29.408 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:29.405+0000 7f1125000640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f11201609d0 0x7f112015b450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:50:29.408 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:29.405+0000 7f1125000640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f112015d0b0 con 0x7f1120160350 2024-08-03T20:50:29.408 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:29.405+0000 7f1125000640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f112015cf30 con 0x7f11201609d0 2024-08-03T20:50:29.408 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:29.405+0000 7f1125000640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7f112015d230 con 0x7f1120159ff0 2024-08-03T20:50:29.408 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:29.406+0000 7f1117e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1120160350 0x7f112014dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:50:29.408 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:29.406+0000 7f1117e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1120160350 0x7f112014dd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:60704/0 (socket says 172.21.15.32:60704) 2024-08-03T20:50:29.409 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:29.406+0000 7f1117e00640 1 -- 172.21.15.32:0/2173358872 learned_addr learned my addr 172.21.15.32:0/2173358872 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:50:29.409 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:29.406+0000 7f111f400640 1 --2- 172.21.15.32:0/2173358872 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f1120159ff0 0x7f11201594f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:50:29.409 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:29.406+0000 7f111fe00640 1 --2- 172.21.15.32:0/2173358872 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f11201609d0 0x7f112015b450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:50:29.409 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:29.406+0000 7f1117e00640 1 -- 172.21.15.32:0/2173358872 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f1120159ff0 msgr2=0x7f11201594f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:50:29.409 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:29.406+0000 7f1117e00640 1 --2- 172.21.15.32:0/2173358872 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f1120159ff0 0x7f11201594f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:29.409 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:29.406+0000 7f1117e00640 1 -- 172.21.15.32:0/2173358872 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f11201609d0 msgr2=0x7f112015b450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:50:29.409 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:29.406+0000 7f1117e00640 1 --2- 172.21.15.32:0/2173358872 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f11201609d0 0x7f112015b450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:29.409 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:29.406+0000 7f1117e00640 1 -- 172.21.15.32:0/2173358872 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f11200b3090 con 0x7f1120160350 2024-08-03T20:50:29.409 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:29.407+0000 7f1117e00640 1 --2- 172.21.15.32:0/2173358872 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1120160350 0x7f112014dd00 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7f1108000c00 tx=0x7f1108033c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:50:29.410 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:29.407+0000 7f1117400640 1 -- 172.21.15.32:0/2173358872 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1108031d60 con 0x7f1120160350 2024-08-03T20:50:29.410 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:29.407+0000 7f1117400640 1 -- 172.21.15.32:0/2173358872 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f1108033e10 con 0x7f1120160350 2024-08-03T20:50:29.410 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:29.407+0000 7f1117400640 1 -- 172.21.15.32:0/2173358872 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11080332b0 con 0x7f1120160350 2024-08-03T20:50:29.410 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:29.407+0000 7f1125000640 1 -- 172.21.15.32:0/2173358872 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f11200b3370 con 0x7f1120160350 2024-08-03T20:50:29.410 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:29.407+0000 7f1125000640 1 -- 172.21.15.32:0/2173358872 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f11200b3930 con 0x7f1120160350 2024-08-03T20:50:29.410 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:29.408+0000 7f1125000640 1 -- 172.21.15.32:0/2173358872 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f112015fe10 con 0x7f1120160350 2024-08-03T20:50:29.411 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:29.409+0000 7f1117400640 1 -- 172.21.15.32:0/2173358872 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f1108033450 con 0x7f1120160350 2024-08-03T20:50:29.411 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:29.410+0000 7f1117400640 1 --2- 172.21.15.32:0/2173358872 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f10fc077010 0x7f10fc079520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:50:29.412 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:29.410+0000 7f1117400640 1 -- 172.21.15.32:0/2173358872 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(57..57 src has 1..57) ==== 5709+0+0 (secure 0 0 0) 0x7f11080b9ba0 con 0x7f1120160350 2024-08-03T20:50:29.412 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:29.411+0000 7f111f400640 1 --2- 172.21.15.32:0/2173358872 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f10fc077010 0x7f10fc079520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:50:29.413 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:29.411+0000 7f111f400640 1 --2- 172.21.15.32:0/2173358872 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f10fc077010 0x7f10fc079520 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f1110009b30 tx=0x7f1110005f90 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:50:29.416 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:29.415+0000 7f1117400640 1 -- 172.21.15.32:0/2173358872 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7f110803f030 con 0x7f1120160350 2024-08-03T20:50:29.667 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:29.665+0000 7f1125000640 1 -- 172.21.15.32:0/2173358872 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) -- 0x7f11200d09e0 con 0x7f1120160350 2024-08-03T20:50:30.135 INFO:tasks.ceph.mds.c.smithi032.stderr:2024-08-03T20:50:30.134+0000 7fb3a4200640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-08-03T20:50:30.184 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:30.182+0000 7f1117400640 1 -- 172.21.15.32:0/2173358872 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v50) ==== 118+0+0 (secure 0 0 0) 0x7f1108087e20 con 0x7f1120160350 2024-08-03T20:50:30.187 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:30.185+0000 7f1125000640 1 -- 172.21.15.32:0/2173358872 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f10fc077010 msgr2=0x7f10fc079520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:50:30.187 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:30.185+0000 7f1125000640 1 --2- 172.21.15.32:0/2173358872 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f10fc077010 0x7f10fc079520 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f1110009b30 tx=0x7f1110005f90 comp rx=0 tx=0).stop 2024-08-03T20:50:30.187 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:30.185+0000 7f1125000640 1 -- 172.21.15.32:0/2173358872 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1120160350 msgr2=0x7f112014dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:50:30.187 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:30.185+0000 7f1125000640 1 --2- 172.21.15.32:0/2173358872 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1120160350 0x7f112014dd00 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7f1108000c00 tx=0x7f1108033c20 comp rx=0 tx=0).stop 2024-08-03T20:50:30.187 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:30.185+0000 7f1125000640 1 -- 172.21.15.32:0/2173358872 shutdown_connections 2024-08-03T20:50:30.187 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:30.185+0000 7f1125000640 1 --2- 172.21.15.32:0/2173358872 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f10fc077010 0x7f10fc079520 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:30.188 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:30.185+0000 7f1125000640 1 --2- 172.21.15.32:0/2173358872 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f11201609d0 0x7f112015b450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:30.188 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:30.185+0000 7f1125000640 1 --2- 172.21.15.32:0/2173358872 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1120160350 0x7f112014dd00 unknown :-1 s=CLOSED pgs=566 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:30.188 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:30.185+0000 7f1125000640 1 --2- 172.21.15.32:0/2173358872 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f1120159ff0 0x7f11201594f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:30.188 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:30.185+0000 7f1125000640 1 -- 172.21.15.32:0/2173358872 >> 172.21.15.32:0/2173358872 conn(0x7f11200cfd80 msgr2=0x7f11200728c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:50:30.188 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:30.186+0000 7f1125000640 1 -- 172.21.15.32:0/2173358872 shutdown_connections 2024-08-03T20:50:30.188 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:30.186+0000 7f1125000640 1 -- 172.21.15.32:0/2173358872 wait complete. 2024-08-03T20:50:30.214 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-08-03T20:50:30.214 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-08-03T20:50:30.448 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:30.446+0000 7fb6e9a00640 1 -- 172.21.15.32:0/3686804086 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fb6e4160350 msgr2=0x7fb6e406e630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:50:30.448 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:30.446+0000 7fb6e9a00640 1 --2- 172.21.15.32:0/3686804086 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fb6e4160350 0x7fb6e406e630 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fb6dc005fd0 tx=0x7fb6dc02aa90 comp rx=0 tx=0).stop 2024-08-03T20:50:30.448 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:30.446+0000 7fb6e9a00640 1 -- 172.21.15.32:0/3686804086 shutdown_connections 2024-08-03T20:50:30.448 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:30.446+0000 7fb6e9a00640 1 --2- 172.21.15.32:0/3686804086 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fb6e4160350 0x7fb6e406e630 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:30.448 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:30.446+0000 7fb6e9a00640 1 --2- 172.21.15.32:0/3686804086 >> v2:172.21.15.118:3300/0 conn(0x7fb6e4159ff0 0x7fb6e415fe10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:30.448 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:30.446+0000 7fb6e9a00640 1 -- 172.21.15.32:0/3686804086 >> 172.21.15.32:0/3686804086 conn(0x7fb6e40cfd80 msgr2=0x7fb6e40d0190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:50:30.448 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:30.446+0000 7fb6e9a00640 1 -- 172.21.15.32:0/3686804086 shutdown_connections 2024-08-03T20:50:30.448 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:30.446+0000 7fb6e9a00640 1 -- 172.21.15.32:0/3686804086 wait complete. 2024-08-03T20:50:30.449 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:30.448+0000 7fb6e9a00640 1 Processor -- start 2024-08-03T20:50:30.450 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:30.448+0000 7fb6e9a00640 1 -- start start 2024-08-03T20:50:30.452 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:30.450+0000 7fb6e9a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb6e4159ff0 0x7fb6e415e820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:50:30.452 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:30.450+0000 7fb6e9a00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb6e4160350 0x7fb6e41f3620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:50:30.452 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:30.450+0000 7fb6e9a00640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fb6e41609d0 0x7fb6e41f3b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:50:30.452 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:30.450+0000 7fb6e9a00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fb6e406ef00 con 0x7fb6e4159ff0 2024-08-03T20:50:30.452 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:30.450+0000 7fb6e9a00640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7fb6e406ed80 con 0x7fb6e4160350 2024-08-03T20:50:30.452 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:30.450+0000 7fb6e9a00640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7fb6e406f080 con 0x7fb6e41609d0 2024-08-03T20:50:30.452 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:30.450+0000 7fb6e3e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb6e4159ff0 0x7fb6e415e820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:50:30.453 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:30.450+0000 7fb6e3e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb6e4159ff0 0x7fb6e415e820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:45036/0 (socket says 172.21.15.32:45036) 2024-08-03T20:50:30.453 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:30.450+0000 7fb6e3e00640 1 -- 172.21.15.32:0/11121884 learned_addr learned my addr 172.21.15.32:0/11121884 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:50:30.453 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:30.450+0000 7fb6db400640 1 --2- 172.21.15.32:0/11121884 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb6e4160350 0x7fb6e41f3620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:50:30.453 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:30.450+0000 7fb6e9000640 1 --2- 172.21.15.32:0/11121884 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fb6e41609d0 0x7fb6e41f3b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:50:30.453 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:30.451+0000 7fb6e3e00640 1 -- 172.21.15.32:0/11121884 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fb6e41609d0 msgr2=0x7fb6e41f3b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:50:30.453 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:30.451+0000 7fb6e3e00640 1 --2- 172.21.15.32:0/11121884 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fb6e41609d0 0x7fb6e41f3b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:30.453 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:30.451+0000 7fb6e3e00640 1 -- 172.21.15.32:0/11121884 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb6e4160350 msgr2=0x7fb6e41f3620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:50:30.453 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:30.451+0000 7fb6e3e00640 1 --2- 172.21.15.32:0/11121884 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb6e4160350 0x7fb6e41f3620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:30.453 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:30.451+0000 7fb6e3e00640 1 -- 172.21.15.32:0/11121884 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb6e40c6f80 con 0x7fb6e4159ff0 2024-08-03T20:50:30.453 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:30.451+0000 7fb6e3e00640 1 --2- 172.21.15.32:0/11121884 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb6e4159ff0 0x7fb6e415e820 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7fb6d400c3d0 tx=0x7fb6d400c890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:50:30.455 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:30.451+0000 7fb6e0c00640 1 -- 172.21.15.32:0/11121884 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6d4017070 con 0x7fb6e4159ff0 2024-08-03T20:50:30.455 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:30.451+0000 7fb6e0c00640 1 -- 172.21.15.32:0/11121884 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fb6d40053c0 con 0x7fb6e4159ff0 2024-08-03T20:50:30.455 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:30.451+0000 7fb6e0c00640 1 -- 172.21.15.32:0/11121884 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6d4003850 con 0x7fb6e4159ff0 2024-08-03T20:50:30.455 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:30.451+0000 7fb6e9a00640 1 -- 172.21.15.32:0/11121884 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb6e40c7260 con 0x7fb6e4159ff0 2024-08-03T20:50:30.455 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:30.451+0000 7fb6e9a00640 1 -- 172.21.15.32:0/11121884 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb6e40c7820 con 0x7fb6e4159ff0 2024-08-03T20:50:30.455 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:30.453+0000 7fb6e9a00640 1 -- 172.21.15.32:0/11121884 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb6e406f530 con 0x7fb6e4159ff0 2024-08-03T20:50:30.455 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:30.454+0000 7fb6e0c00640 1 -- 172.21.15.32:0/11121884 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7fb6d401a070 con 0x7fb6e4159ff0 2024-08-03T20:50:30.456 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:30.455+0000 7fb6e0c00640 1 --2- 172.21.15.32:0/11121884 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fb6c0077040 0x7fb6c0079550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:50:30.456 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:30.455+0000 7fb6e0c00640 1 -- 172.21.15.32:0/11121884 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(57..57 src has 1..57) ==== 5709+0+0 (secure 0 0 0) 0x7fb6d4097cc0 con 0x7fb6e4159ff0 2024-08-03T20:50:30.457 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:30.455+0000 7fb6db400640 1 --2- 172.21.15.32:0/11121884 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fb6c0077040 0x7fb6c0079550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:50:30.457 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:30.456+0000 7fb6db400640 1 --2- 172.21.15.32:0/11121884 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fb6c0077040 0x7fb6c0079550 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7fb6e415fee0 tx=0x7fb6cc005f90 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:50:30.460 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:30.459+0000 7fb6e0c00640 1 -- 172.21.15.32:0/11121884 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7fb6d409e080 con 0x7fb6e4159ff0 2024-08-03T20:50:30.704 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:30.702+0000 7fb6e9a00640 1 -- 172.21.15.32:0/11121884 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) -- 0x7fb6e40d09e0 con 0x7fb6e4159ff0 2024-08-03T20:50:31.191 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:31.190+0000 7fb6e0c00640 1 -- 172.21.15.32:0/11121884 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v51) ==== 122+0+0 (secure 0 0 0) 0x7fb6e4160760 con 0x7fb6e4159ff0 2024-08-03T20:50:31.194 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:31.192+0000 7fb6e9a00640 1 -- 172.21.15.32:0/11121884 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fb6c0077040 msgr2=0x7fb6c0079550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:50:31.195 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:31.192+0000 7fb6e9a00640 1 --2- 172.21.15.32:0/11121884 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fb6c0077040 0x7fb6c0079550 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7fb6e415fee0 tx=0x7fb6cc005f90 comp rx=0 tx=0).stop 2024-08-03T20:50:31.195 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:31.193+0000 7fb6e9a00640 1 -- 172.21.15.32:0/11121884 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb6e4159ff0 msgr2=0x7fb6e415e820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:50:31.195 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:31.193+0000 7fb6e9a00640 1 --2- 172.21.15.32:0/11121884 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb6e4159ff0 0x7fb6e415e820 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7fb6d400c3d0 tx=0x7fb6d400c890 comp rx=0 tx=0).stop 2024-08-03T20:50:31.195 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:31.193+0000 7fb6e9a00640 1 -- 172.21.15.32:0/11121884 shutdown_connections 2024-08-03T20:50:31.195 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:31.193+0000 7fb6e9a00640 1 --2- 172.21.15.32:0/11121884 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fb6c0077040 0x7fb6c0079550 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:31.195 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:31.193+0000 7fb6e9a00640 1 --2- 172.21.15.32:0/11121884 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb6e4160350 0x7fb6e41f3620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:31.195 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:31.193+0000 7fb6e9a00640 1 --2- 172.21.15.32:0/11121884 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fb6e41609d0 0x7fb6e41f3b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:31.195 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:31.193+0000 7fb6e9a00640 1 --2- 172.21.15.32:0/11121884 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb6e4159ff0 0x7fb6e415e820 unknown :-1 s=CLOSED pgs=567 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:31.195 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:31.193+0000 7fb6e9a00640 1 -- 172.21.15.32:0/11121884 >> 172.21.15.32:0/11121884 conn(0x7fb6e40cfd80 msgr2=0x7fb6e40cb880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:50:31.195 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:31.193+0000 7fb6e9a00640 1 -- 172.21.15.32:0/11121884 shutdown_connections 2024-08-03T20:50:31.195 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:31.193+0000 7fb6e9a00640 1 -- 172.21.15.32:0/11121884 wait complete. 2024-08-03T20:50:31.222 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-08-03T20:50:31.223 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-08-03T20:50:31.451 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:31.449+0000 7f2f55600640 1 -- 172.21.15.32:0/374112779 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2f50073e60 msgr2=0x7f2f50074270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:50:31.451 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:31.449+0000 7f2f55600640 1 --2- 172.21.15.32:0/374112779 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2f50073e60 0x7f2f50074270 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7f2f50071700 tx=0x7f2f3802a560 comp rx=0 tx=0).stop 2024-08-03T20:50:31.451 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:31.449+0000 7f2f55600640 1 -- 172.21.15.32:0/374112779 shutdown_connections 2024-08-03T20:50:31.451 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:31.449+0000 7f2f55600640 1 --2- 172.21.15.32:0/374112779 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2f50073e60 0x7f2f50074270 unknown :-1 s=CLOSED pgs=568 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:31.451 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:31.449+0000 7f2f55600640 1 --2- 172.21.15.32:0/374112779 >> v2:172.21.15.118:3300/0 conn(0x7f2f5006e930 0x7f2f50144f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:31.451 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:31.449+0000 7f2f55600640 1 -- 172.21.15.32:0/374112779 >> 172.21.15.32:0/374112779 conn(0x7f2f500b1bd0 msgr2=0x7f2f500b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:50:31.452 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:31.449+0000 7f2f55600640 1 -- 172.21.15.32:0/374112779 shutdown_connections 2024-08-03T20:50:31.452 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:31.449+0000 7f2f55600640 1 -- 172.21.15.32:0/374112779 wait complete. 2024-08-03T20:50:31.453 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:31.452+0000 7f2f55600640 1 Processor -- start 2024-08-03T20:50:31.453 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:31.452+0000 7f2f55600640 1 -- start start 2024-08-03T20:50:31.455 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:31.453+0000 7f2f55600640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f2f5006e930 0x7f2f501f3400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:50:31.455 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:31.453+0000 7f2f55600640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2f50073e60 0x7f2f500af380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:50:31.455 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:31.453+0000 7f2f55600640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f2f500747b0 0x7f2f501436b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:50:31.456 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:31.453+0000 7f2f55600640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f2f500cf9c0 con 0x7f2f50073e60 2024-08-03T20:50:31.456 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:31.453+0000 7f2f55600640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f2f500cf840 con 0x7f2f5006e930 2024-08-03T20:50:31.456 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:31.453+0000 7f2f55600640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7f2f500ccff0 con 0x7f2f500747b0 2024-08-03T20:50:31.456 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:31.454+0000 7f2f47e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2f50073e60 0x7f2f500af380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:50:31.456 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:31.454+0000 7f2f47e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2f50073e60 0x7f2f500af380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:45066/0 (socket says 172.21.15.32:45066) 2024-08-03T20:50:31.456 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:31.454+0000 7f2f47e00640 1 -- 172.21.15.32:0/2769562576 learned_addr learned my addr 172.21.15.32:0/2769562576 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:50:31.456 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:31.454+0000 7f2f4ea00640 1 --2- 172.21.15.32:0/2769562576 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f2f5006e930 0x7f2f501f3400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:50:31.456 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:31.454+0000 7f2f4f400640 1 --2- 172.21.15.32:0/2769562576 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f2f500747b0 0x7f2f501436b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:50:31.456 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:31.454+0000 7f2f47e00640 1 -- 172.21.15.32:0/2769562576 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f2f500747b0 msgr2=0x7f2f501436b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:50:31.456 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:31.454+0000 7f2f47e00640 1 --2- 172.21.15.32:0/2769562576 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f2f500747b0 0x7f2f501436b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:31.456 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:31.454+0000 7f2f47e00640 1 -- 172.21.15.32:0/2769562576 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f2f5006e930 msgr2=0x7f2f501f3400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:50:31.456 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:31.454+0000 7f2f47e00640 1 --2- 172.21.15.32:0/2769562576 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f2f5006e930 0x7f2f501f3400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:31.456 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:31.454+0000 7f2f47e00640 1 -- 172.21.15.32:0/2769562576 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2f501f3c50 con 0x7f2f50073e60 2024-08-03T20:50:31.457 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:31.454+0000 7f2f47e00640 1 --2- 172.21.15.32:0/2769562576 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2f50073e60 0x7f2f500af380 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7f2f38034040 tx=0x7f2f38003820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:50:31.458 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:31.455+0000 7f2f46a00640 1 -- 172.21.15.32:0/2769562576 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f3803e030 con 0x7f2f50073e60 2024-08-03T20:50:31.458 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:31.455+0000 7f2f46a00640 1 -- 172.21.15.32:0/2769562576 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f2f38003d10 con 0x7f2f50073e60 2024-08-03T20:50:31.458 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:31.455+0000 7f2f46a00640 1 -- 172.21.15.32:0/2769562576 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f38035040 con 0x7f2f50073e60 2024-08-03T20:50:31.458 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:31.455+0000 7f2f55600640 1 -- 172.21.15.32:0/2769562576 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2f501f3f30 con 0x7f2f50073e60 2024-08-03T20:50:31.458 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:31.455+0000 7f2f55600640 1 -- 172.21.15.32:0/2769562576 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2f501f44f0 con 0x7f2f50073e60 2024-08-03T20:50:31.458 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:31.456+0000 7f2f55600640 1 -- 172.21.15.32:0/2769562576 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2f501f39c0 con 0x7f2f50073e60 2024-08-03T20:50:31.459 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:31.457+0000 7f2f46a00640 1 -- 172.21.15.32:0/2769562576 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f2f38041030 con 0x7f2f50073e60 2024-08-03T20:50:31.460 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:31.458+0000 7f2f46a00640 1 --2- 172.21.15.32:0/2769562576 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f2f2c077040 0x7f2f2c079550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:50:31.460 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:31.458+0000 7f2f46a00640 1 -- 172.21.15.32:0/2769562576 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(57..57 src has 1..57) ==== 5709+0+0 (secure 0 0 0) 0x7f2f380b8f90 con 0x7f2f50073e60 2024-08-03T20:50:31.460 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:31.459+0000 7f2f4ea00640 1 --2- 172.21.15.32:0/2769562576 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f2f2c077040 0x7f2f2c079550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:50:31.461 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:31.460+0000 7f2f4ea00640 1 --2- 172.21.15.32:0/2769562576 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f2f2c077040 0x7f2f2c079550 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f2f40009b30 tx=0x7f2f40005f90 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:50:31.464 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:31.463+0000 7f2f46a00640 1 -- 172.21.15.32:0/2769562576 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7f2f38081b30 con 0x7f2f50073e60 2024-08-03T20:50:31.708 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:31.706+0000 7f2f55600640 1 -- 172.21.15.32:0/2769562576 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f2f500ccde0 con 0x7f2f50073e60 2024-08-03T20:50:31.708 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:31.707+0000 7f2f46a00640 1 -- 172.21.15.32:0/2769562576 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 51 v51) ==== 94+0+5229 (secure 0 0 0) 0x7f2f500ccde0 con 0x7f2f50073e60 2024-08-03T20:50:31.709 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-03T20:50:31.709 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":51,"btime":"2024-08-03T20:50:31:184656+0000","default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5261,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6833/2498840646","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":2498840646},{"type":"v1","addr":"172.21.15.32:6833","nonce":2498840646}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":45},{"gid":5276,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6837/3142148286","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6835","nonce":3142148286},{"type":"v1","addr":"172.21.15.118:6837","nonce":3142148286}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":46},{"gid":5632,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6836/3720010453","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6834","nonce":3720010453},{"type":"v1","addr":"172.21.15.118:6836","nonce":3720010453}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":46}],"filesystems":[{"mdsmap":{"epoch":51,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-08-03T20:50:27.688698+0000","modified":"2024-08-03T20:50:30.704840+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5841},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5841":{"gid":5841,"name":"c","rank":0,"incarnation":47,"state":"up:active","state_seq":3,"addr":"172.21.15.32:6835/3848976369","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":3848976369},{"type":"v1","addr":"172.21.15.32:6835","nonce":3848976369}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[9],"metadata_pool":8,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5841,"qdb_cluster":[5841]},"id":4}]} 2024-08-03T20:50:31.709 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 51 2024-08-03T20:50:31.711 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:31.709+0000 7f2f55600640 1 -- 172.21.15.32:0/2769562576 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f2f2c077040 msgr2=0x7f2f2c079550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:50:31.711 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:31.709+0000 7f2f55600640 1 --2- 172.21.15.32:0/2769562576 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f2f2c077040 0x7f2f2c079550 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f2f40009b30 tx=0x7f2f40005f90 comp rx=0 tx=0).stop 2024-08-03T20:50:31.711 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:31.709+0000 7f2f55600640 1 -- 172.21.15.32:0/2769562576 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2f50073e60 msgr2=0x7f2f500af380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:50:31.711 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:31.709+0000 7f2f55600640 1 --2- 172.21.15.32:0/2769562576 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2f50073e60 0x7f2f500af380 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7f2f38034040 tx=0x7f2f38003820 comp rx=0 tx=0).stop 2024-08-03T20:50:31.711 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:31.709+0000 7f2f55600640 1 -- 172.21.15.32:0/2769562576 shutdown_connections 2024-08-03T20:50:31.711 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:31.709+0000 7f2f55600640 1 --2- 172.21.15.32:0/2769562576 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f2f2c077040 0x7f2f2c079550 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:31.711 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:31.709+0000 7f2f55600640 1 --2- 172.21.15.32:0/2769562576 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f2f500747b0 0x7f2f501436b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:31.712 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:31.709+0000 7f2f55600640 1 --2- 172.21.15.32:0/2769562576 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2f50073e60 0x7f2f500af380 unknown :-1 s=CLOSED pgs=569 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:31.712 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:31.709+0000 7f2f55600640 1 --2- 172.21.15.32:0/2769562576 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f2f5006e930 0x7f2f501f3400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:31.712 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:31.709+0000 7f2f55600640 1 -- 172.21.15.32:0/2769562576 >> 172.21.15.32:0/2769562576 conn(0x7f2f500b1bd0 msgr2=0x7f2f5006ef70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:50:31.712 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:31.709+0000 7f2f55600640 1 -- 172.21.15.32:0/2769562576 shutdown_connections 2024-08-03T20:50:31.712 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:31.709+0000 7f2f55600640 1 -- 172.21.15.32:0/2769562576 wait complete. 2024-08-03T20:50:31.735 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-08-03T20:50:31.965 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:31.963+0000 7faf4e200640 1 -- 172.21.15.32:0/2804844858 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faf48073e60 msgr2=0x7faf48074270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:50:31.965 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:31.963+0000 7faf4e200640 1 --2- 172.21.15.32:0/2804844858 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faf48073e60 0x7faf48074270 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7faf48071700 tx=0x7faf3002a4a0 comp rx=0 tx=0).stop 2024-08-03T20:50:31.965 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:31.963+0000 7faf4e200640 1 -- 172.21.15.32:0/2804844858 shutdown_connections 2024-08-03T20:50:31.965 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:31.963+0000 7faf4e200640 1 --2- 172.21.15.32:0/2804844858 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faf48073e60 0x7faf48074270 unknown :-1 s=CLOSED pgs=571 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:31.965 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:31.963+0000 7faf4e200640 1 --2- 172.21.15.32:0/2804844858 >> v2:172.21.15.118:3300/0 conn(0x7faf4806e930 0x7faf48144f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:31.965 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:31.963+0000 7faf4e200640 1 -- 172.21.15.32:0/2804844858 >> 172.21.15.32:0/2804844858 conn(0x7faf480b1bd0 msgr2=0x7faf480b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:50:31.965 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:31.963+0000 7faf4e200640 1 -- 172.21.15.32:0/2804844858 shutdown_connections 2024-08-03T20:50:31.966 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:31.963+0000 7faf4e200640 1 -- 172.21.15.32:0/2804844858 wait complete. 2024-08-03T20:50:31.967 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:31.966+0000 7faf4e200640 1 Processor -- start 2024-08-03T20:50:31.967 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:31.966+0000 7faf4e200640 1 -- start start 2024-08-03T20:50:31.970 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:31.967+0000 7faf4e200640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7faf4806e930 0x7faf481414e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:50:31.970 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:31.967+0000 7faf4e200640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7faf48073e60 0x7faf48141a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:50:31.970 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:31.967+0000 7faf4e200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faf480747b0 0x7faf480b3ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:50:31.970 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:31.967+0000 7faf4e200640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7faf480cf9c0 con 0x7faf480747b0 2024-08-03T20:50:31.970 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:31.967+0000 7faf4e200640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7faf480cf840 con 0x7faf4806e930 2024-08-03T20:50:31.970 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:31.967+0000 7faf4e200640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7faf480ccff0 con 0x7faf48073e60 2024-08-03T20:50:31.970 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:31.967+0000 7faf47e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faf480747b0 0x7faf480b3ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:50:31.971 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:31.968+0000 7faf47e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faf480747b0 0x7faf480b3ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:45086/0 (socket says 172.21.15.32:45086) 2024-08-03T20:50:31.971 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:31.968+0000 7faf47e00640 1 -- 172.21.15.32:0/2844371462 learned_addr learned my addr 172.21.15.32:0/2844371462 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:50:31.971 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:31.968+0000 7faf47400640 1 --2- 172.21.15.32:0/2844371462 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7faf4806e930 0x7faf481414e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:50:31.971 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:31.968+0000 7faf3ea00640 1 --2- 172.21.15.32:0/2844371462 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7faf48073e60 0x7faf48141a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:50:31.971 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:31.968+0000 7faf47e00640 1 -- 172.21.15.32:0/2844371462 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7faf48073e60 msgr2=0x7faf48141a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:50:31.971 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:31.968+0000 7faf47e00640 1 --2- 172.21.15.32:0/2844371462 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7faf48073e60 0x7faf48141a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:31.971 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:31.968+0000 7faf47e00640 1 -- 172.21.15.32:0/2844371462 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7faf4806e930 msgr2=0x7faf481414e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:50:31.971 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:31.968+0000 7faf47e00640 1 --2- 172.21.15.32:0/2844371462 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7faf4806e930 0x7faf481414e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:31.971 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:31.968+0000 7faf47e00640 1 -- 172.21.15.32:0/2844371462 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faf480b3100 con 0x7faf480747b0 2024-08-03T20:50:31.971 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:31.968+0000 7faf3ea00640 1 --2- 172.21.15.32:0/2844371462 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7faf48073e60 0x7faf48141a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-03T20:50:31.971 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:31.968+0000 7faf47e00640 1 --2- 172.21.15.32:0/2844371462 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faf480747b0 0x7faf480b3ec0 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7faf4000c6c0 tx=0x7faf4000cb80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:50:31.971 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:31.968+0000 7faf3fe00640 1 -- 172.21.15.32:0/2844371462 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf40017070 con 0x7faf480747b0 2024-08-03T20:50:31.971 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:31.969+0000 7faf3fe00640 1 -- 172.21.15.32:0/2844371462 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7faf40003c60 con 0x7faf480747b0 2024-08-03T20:50:31.972 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:31.969+0000 7faf3fe00640 1 -- 172.21.15.32:0/2844371462 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf400052d0 con 0x7faf480747b0 2024-08-03T20:50:31.972 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:31.969+0000 7faf4e200640 1 -- 172.21.15.32:0/2844371462 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faf480b33e0 con 0x7faf480747b0 2024-08-03T20:50:31.972 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:31.969+0000 7faf4e200640 1 -- 172.21.15.32:0/2844371462 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faf48130c10 con 0x7faf480747b0 2024-08-03T20:50:31.972 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:31.970+0000 7faf4e200640 1 -- 172.21.15.32:0/2844371462 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faf48130e70 con 0x7faf480747b0 2024-08-03T20:50:31.972 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:31.972+0000 7faf3fe00640 1 -- 172.21.15.32:0/2844371462 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7faf4001a070 con 0x7faf480747b0 2024-08-03T20:50:31.973 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:31.972+0000 7faf3fe00640 1 --2- 172.21.15.32:0/2844371462 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7faf24077010 0x7faf24079520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:50:31.974 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:31.972+0000 7faf3fe00640 1 -- 172.21.15.32:0/2844371462 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(57..57 src has 1..57) ==== 5709+0+0 (secure 0 0 0) 0x7faf40097ed0 con 0x7faf480747b0 2024-08-03T20:50:31.974 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:31.973+0000 7faf47400640 1 --2- 172.21.15.32:0/2844371462 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7faf24077010 0x7faf24079520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:50:31.975 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:31.973+0000 7faf47400640 1 --2- 172.21.15.32:0/2844371462 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7faf24077010 0x7faf24079520 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7faf3800bb50 tx=0x7faf3800b2e0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:50:31.978 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:31.977+0000 7faf3fe00640 1 -- 172.21.15.32:0/2844371462 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7faf40060b00 con 0x7faf480747b0 2024-08-03T20:50:32.184 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:32.183+0000 7faf4e200640 1 -- 172.21.15.32:0/2844371462 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7faf481445f0 con 0x7faf480747b0 2024-08-03T20:50:32.185 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:32.184+0000 7faf3fe00640 1 -- 172.21.15.32:0/2844371462 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v57) ==== 74+0+16605 (secure 0 0 0) 0x7faf40069390 con 0x7faf480747b0 2024-08-03T20:50:32.186 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-03T20:50:32.186 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":57,"fsid":"40ab141a-01d7-49b9-ae3c-2edec9876e95","created":"2024-08-03T20:47:14.957362+0000","modified":"2024-08-03T20:50:29.154011+0000","last_up_change":"2024-08-03T20:47:28.874451+0000","last_in_change":"2024-08-03T20:47:20.079442+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-08-03T20:47:28.261300+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_metadata","create_time":"2024-08-03T20:50:25.652248+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"56","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_data","create_time":"2024-08-03T20:50:26.646008+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"57","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5a83e759-9035-438b-889a-0a6af1f0a0d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6816","nonce":3066156116},{"type":"v1","addr":"172.21.15.32:6817","nonce":3066156116}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6818","nonce":3066156116},{"type":"v1","addr":"172.21.15.32:6819","nonce":3066156116}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6822","nonce":3066156116},{"type":"v1","addr":"172.21.15.32:6823","nonce":3066156116}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6820","nonce":3066156116},{"type":"v1","addr":"172.21.15.32:6821","nonce":3066156116}]},"public_addr":"172.21.15.32:6817/3066156116","cluster_addr":"172.21.15.32:6819/3066156116","heartbeat_back_addr":"172.21.15.32:6823/3066156116","heartbeat_front_addr":"172.21.15.32:6821/3066156116","state":["exists","up"]},{"osd":1,"uuid":"d7c5c210-da4f-49f9-b9e7-1dfb02daa4bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6824","nonce":1551241759},{"type":"v1","addr":"172.21.15.32:6825","nonce":1551241759}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6826","nonce":1551241759},{"type":"v1","addr":"172.21.15.32:6827","nonce":1551241759}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6830","nonce":1551241759},{"type":"v1","addr":"172.21.15.32:6831","nonce":1551241759}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6828","nonce":1551241759},{"type":"v1","addr":"172.21.15.32:6829","nonce":1551241759}]},"public_addr":"172.21.15.32:6825/1551241759","cluster_addr":"172.21.15.32:6827/1551241759","heartbeat_back_addr":"172.21.15.32:6831/1551241759","heartbeat_front_addr":"172.21.15.32:6829/1551241759","state":["exists","up"]},{"osd":2,"uuid":"57f2e6de-a201-45c2-8981-8e56a4875ebd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6808","nonce":3806708944},{"type":"v1","addr":"172.21.15.32:6809","nonce":3806708944}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6810","nonce":3806708944},{"type":"v1","addr":"172.21.15.32:6811","nonce":3806708944}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6814","nonce":3806708944},{"type":"v1","addr":"172.21.15.32:6815","nonce":3806708944}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6812","nonce":3806708944},{"type":"v1","addr":"172.21.15.32:6813","nonce":3806708944}]},"public_addr":"172.21.15.32:6809/3806708944","cluster_addr":"172.21.15.32:6811/3806708944","heartbeat_back_addr":"172.21.15.32:6815/3806708944","heartbeat_front_addr":"172.21.15.32:6813/3806708944","state":["exists","up"]},{"osd":3,"uuid":"6439a280-da45-4148-8130-f16323935ff3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6800","nonce":9088303},{"type":"v1","addr":"172.21.15.32:6801","nonce":9088303}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6802","nonce":9088303},{"type":"v1","addr":"172.21.15.32:6803","nonce":9088303}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6806","nonce":9088303},{"type":"v1","addr":"172.21.15.32:6807","nonce":9088303}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6804","nonce":9088303},{"type":"v1","addr":"172.21.15.32:6805","nonce":9088303}]},"public_addr":"172.21.15.32:6801/9088303","cluster_addr":"172.21.15.32:6803/9088303","heartbeat_back_addr":"172.21.15.32:6807/9088303","heartbeat_front_addr":"172.21.15.32:6805/9088303","state":["exists","up"]},{"osd":4,"uuid":"4f902eff-1e18-418d-82c9-2151c69524fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6802","nonce":864752275},{"type":"v1","addr":"172.21.15.118:6803","nonce":864752275}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6804","nonce":864752275},{"type":"v1","addr":"172.21.15.118:6805","nonce":864752275}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6808","nonce":864752275},{"type":"v1","addr":"172.21.15.118:6809","nonce":864752275}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6806","nonce":864752275},{"type":"v1","addr":"172.21.15.118:6807","nonce":864752275}]},"public_addr":"172.21.15.118:6803/864752275","cluster_addr":"172.21.15.118:6805/864752275","heartbeat_back_addr":"172.21.15.118:6809/864752275","heartbeat_front_addr":"172.21.15.118:6807/864752275","state":["exists","up"]},{"osd":5,"uuid":"bfd9feac-dd06-458c-91b9-af7f175f612c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6810","nonce":2724891770},{"type":"v1","addr":"172.21.15.118:6811","nonce":2724891770}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6812","nonce":2724891770},{"type":"v1","addr":"172.21.15.118:6813","nonce":2724891770}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6816","nonce":2724891770},{"type":"v1","addr":"172.21.15.118:6817","nonce":2724891770}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6814","nonce":2724891770},{"type":"v1","addr":"172.21.15.118:6815","nonce":2724891770}]},"public_addr":"172.21.15.118:6811/2724891770","cluster_addr":"172.21.15.118:6813/2724891770","heartbeat_back_addr":"172.21.15.118:6817/2724891770","heartbeat_front_addr":"172.21.15.118:6815/2724891770","state":["exists","up"]},{"osd":6,"uuid":"5ed5a279-9d8e-416d-bbc7-7edd3e6d5705","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6825","nonce":801947386},{"type":"v1","addr":"172.21.15.118:6827","nonce":801947386}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6828","nonce":801947386},{"type":"v1","addr":"172.21.15.118:6829","nonce":801947386}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6832","nonce":801947386},{"type":"v1","addr":"172.21.15.118:6833","nonce":801947386}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6830","nonce":801947386},{"type":"v1","addr":"172.21.15.118:6831","nonce":801947386}]},"public_addr":"172.21.15.118:6827/801947386","cluster_addr":"172.21.15.118:6829/801947386","heartbeat_back_addr":"172.21.15.118:6833/801947386","heartbeat_front_addr":"172.21.15.118:6831/801947386","state":["exists","up"]},{"osd":7,"uuid":"f2086c47-875e-4451-a436-2008d1ea16af","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6818","nonce":653000048},{"type":"v1","addr":"172.21.15.118:6819","nonce":653000048}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6820","nonce":653000048},{"type":"v1","addr":"172.21.15.118:6821","nonce":653000048}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6824","nonce":653000048},{"type":"v1","addr":"172.21.15.118:6826","nonce":653000048}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6822","nonce":653000048},{"type":"v1","addr":"172.21.15.118:6823","nonce":653000048}]},"public_addr":"172.21.15.118:6819/653000048","cluster_addr":"172.21.15.118:6821/653000048","heartbeat_back_addr":"172.21.15.118:6826/653000048","heartbeat_front_addr":"172.21.15.118:6823/653000048","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-03T20:47:25.491994+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-03T20:47:25.587358+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-03T20:47:25.454175+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-03T20:47:25.377734+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-03T20:47:25.412351+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-03T20:47:25.499815+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-03T20:47:25.575044+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-03T20:47:25.552383+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-08-03T20:50:32.188 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:32.186+0000 7faf4e200640 1 -- 172.21.15.32:0/2844371462 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7faf24077010 msgr2=0x7faf24079520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:50:32.188 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:32.186+0000 7faf4e200640 1 --2- 172.21.15.32:0/2844371462 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7faf24077010 0x7faf24079520 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7faf3800bb50 tx=0x7faf3800b2e0 comp rx=0 tx=0).stop 2024-08-03T20:50:32.188 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:32.186+0000 7faf4e200640 1 -- 172.21.15.32:0/2844371462 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faf480747b0 msgr2=0x7faf480b3ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:50:32.188 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:32.186+0000 7faf4e200640 1 --2- 172.21.15.32:0/2844371462 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faf480747b0 0x7faf480b3ec0 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7faf4000c6c0 tx=0x7faf4000cb80 comp rx=0 tx=0).stop 2024-08-03T20:50:32.188 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:32.186+0000 7faf4e200640 1 -- 172.21.15.32:0/2844371462 shutdown_connections 2024-08-03T20:50:32.188 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:32.186+0000 7faf4e200640 1 --2- 172.21.15.32:0/2844371462 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7faf24077010 0x7faf24079520 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:32.189 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:32.186+0000 7faf4e200640 1 --2- 172.21.15.32:0/2844371462 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faf480747b0 0x7faf480b3ec0 unknown :-1 s=CLOSED pgs=572 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:32.189 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:32.186+0000 7faf4e200640 1 --2- 172.21.15.32:0/2844371462 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7faf48073e60 0x7faf48141a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:32.189 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:32.186+0000 7faf4e200640 1 --2- 172.21.15.32:0/2844371462 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7faf4806e930 0x7faf481414e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:32.189 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:32.186+0000 7faf4e200640 1 -- 172.21.15.32:0/2844371462 >> 172.21.15.32:0/2844371462 conn(0x7faf480b1bd0 msgr2=0x7faf48140d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:50:32.189 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:32.186+0000 7faf4e200640 1 -- 172.21.15.32:0/2844371462 shutdown_connections 2024-08-03T20:50:32.190 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:32.186+0000 7faf4e200640 1 -- 172.21.15.32:0/2844371462 wait complete. 2024-08-03T20:50:32.211 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-08-03T20:50:32.212 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-08-03T20:50:32.441 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:32.439+0000 7f0f04800640 1 -- 172.21.15.32:0/3744415499 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f0efc15e880 msgr2=0x7f0efc06e930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:50:32.441 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:32.439+0000 7f0f04800640 1 --2- 172.21.15.32:0/3744415499 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f0efc15e880 0x7f0efc06e930 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f0ef800acf0 tx=0x7f0ef802bb50 comp rx=0 tx=0).stop 2024-08-03T20:50:32.441 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:32.439+0000 7f0f04800640 1 -- 172.21.15.32:0/3744415499 shutdown_connections 2024-08-03T20:50:32.441 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:32.439+0000 7f0f04800640 1 --2- 172.21.15.32:0/3744415499 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f0efc15e880 0x7f0efc06e930 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:32.441 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:32.439+0000 7f0f04800640 1 -- 172.21.15.32:0/3744415499 >> 172.21.15.32:0/3744415499 conn(0x7f0efc0b1bd0 msgr2=0x7f0efc0b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:50:32.441 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:32.439+0000 7f0f04800640 1 -- 172.21.15.32:0/3744415499 shutdown_connections 2024-08-03T20:50:32.441 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:32.439+0000 7f0f04800640 1 -- 172.21.15.32:0/3744415499 wait complete. 2024-08-03T20:50:32.443 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:32.441+0000 7f0f04800640 1 Processor -- start 2024-08-03T20:50:32.443 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:32.442+0000 7f0f04800640 1 -- start start 2024-08-03T20:50:32.446 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:32.443+0000 7f0f04800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0efc0b4de0 0x7f0efc1f5360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:50:32.446 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:32.443+0000 7f0f04800640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f0efc1515c0 0x7f0efc1f8090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:50:32.446 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:32.443+0000 7f0f04800640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f0efc1f89c0 0x7f0efc1595f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:50:32.446 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:32.443+0000 7f0f04800640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f0efc070a40 con 0x7f0efc0b4de0 2024-08-03T20:50:32.446 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:32.443+0000 7f0f04800640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f0efc0708c0 con 0x7f0efc1515c0 2024-08-03T20:50:32.446 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:32.443+0000 7f0f04800640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7f0efc06ee70 con 0x7f0efc1f89c0 2024-08-03T20:50:32.446 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:32.443+0000 7f0f01e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0efc0b4de0 0x7f0efc1f5360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:50:32.446 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:32.443+0000 7f0f01e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0efc0b4de0 0x7f0efc1f5360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:45090/0 (socket says 172.21.15.32:45090) 2024-08-03T20:50:32.447 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:32.443+0000 7f0f01e00640 1 -- 172.21.15.32:0/520362288 learned_addr learned my addr 172.21.15.32:0/520362288 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:50:32.447 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:32.444+0000 7f0f01400640 1 --2- 172.21.15.32:0/520362288 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f0efc1515c0 0x7f0efc1f8090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:50:32.447 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:32.444+0000 7f0f02800640 1 --2- 172.21.15.32:0/520362288 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f0efc1f89c0 0x7f0efc1595f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:50:32.447 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:32.444+0000 7f0f01e00640 1 -- 172.21.15.32:0/520362288 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f0efc1f89c0 msgr2=0x7f0efc1595f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:50:32.447 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:32.444+0000 7f0f01e00640 1 --2- 172.21.15.32:0/520362288 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f0efc1f89c0 0x7f0efc1595f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:32.447 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:32.444+0000 7f0f01e00640 1 -- 172.21.15.32:0/520362288 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f0efc1515c0 msgr2=0x7f0efc1f8090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:50:32.447 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:32.444+0000 7f0f01e00640 1 --2- 172.21.15.32:0/520362288 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f0efc1515c0 0x7f0efc1f8090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:32.447 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:32.444+0000 7f0f01e00640 1 -- 172.21.15.32:0/520362288 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0efc1f5cd0 con 0x7f0efc0b4de0 2024-08-03T20:50:32.447 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:32.444+0000 7f0f02800640 1 --2- 172.21.15.32:0/520362288 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f0efc1f89c0 0x7f0efc1595f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-03T20:50:32.447 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:32.444+0000 7f0f01e00640 1 --2- 172.21.15.32:0/520362288 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0efc0b4de0 0x7f0efc1f5360 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7f0ef000c6b0 tx=0x7f0ef000cb70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:50:32.447 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:32.444+0000 7f0ef6000640 1 -- 172.21.15.32:0/520362288 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ef0016020 con 0x7f0efc0b4de0 2024-08-03T20:50:32.447 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:32.444+0000 7f0ef6000640 1 -- 172.21.15.32:0/520362288 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f0ef000a8d0 con 0x7f0efc0b4de0 2024-08-03T20:50:32.447 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:32.444+0000 7f0f04800640 1 -- 172.21.15.32:0/520362288 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0efc1f5fb0 con 0x7f0efc0b4de0 2024-08-03T20:50:32.447 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:32.444+0000 7f0ef6000640 1 -- 172.21.15.32:0/520362288 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ef000ace0 con 0x7f0efc0b4de0 2024-08-03T20:50:32.448 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:32.445+0000 7f0f04800640 1 -- 172.21.15.32:0/520362288 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0efc147ba0 con 0x7f0efc0b4de0 2024-08-03T20:50:32.448 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:32.446+0000 7f0f04800640 1 -- 172.21.15.32:0/520362288 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0efc06f3d0 con 0x7f0efc0b4de0 2024-08-03T20:50:32.448 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:32.447+0000 7f0ef6000640 1 -- 172.21.15.32:0/520362288 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f0ef0019020 con 0x7f0efc0b4de0 2024-08-03T20:50:32.449 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:32.448+0000 7f0ef6000640 1 --2- 172.21.15.32:0/520362288 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f0ee0077010 0x7f0ee0079520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:50:32.449 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:32.448+0000 7f0ef6000640 1 -- 172.21.15.32:0/520362288 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(57..57 src has 1..57) ==== 5709+0+0 (secure 0 0 0) 0x7f0ef0098cf0 con 0x7f0efc0b4de0 2024-08-03T20:50:32.450 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:32.448+0000 7f0f01400640 1 --2- 172.21.15.32:0/520362288 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f0ee0077010 0x7f0ee0079520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:50:32.450 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:32.449+0000 7f0f01400640 1 --2- 172.21.15.32:0/520362288 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f0ee0077010 0x7f0ee0079520 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f0efc1530a0 tx=0x7f0eec009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:50:32.454 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:32.453+0000 7f0ef6000640 1 -- 172.21.15.32:0/520362288 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7f0ef0020020 con 0x7f0efc0b4de0 2024-08-03T20:50:32.654 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:32.653+0000 7f0f04800640 1 -- 172.21.15.32:0/520362288 --> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f0efc1533b0 con 0x7f0ee0077010 2024-08-03T20:50:32.659 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:32.658+0000 7f0ef6000640 1 -- 172.21.15.32:0/520362288 <== mgr.4106 v2:172.21.15.118:6800/4205950699 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+251962 (secure 0 0 0) 0x7f0efc1533b0 con 0x7f0ee0077010 2024-08-03T20:50:32.662 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-03T20:50:32.664 INFO:teuthology.orchestra.run.smithi032.stderr:dumped all 2024-08-03T20:50:32.666 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:32.663+0000 7f0f04800640 1 -- 172.21.15.32:0/520362288 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f0ee0077010 msgr2=0x7f0ee0079520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:50:32.666 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:32.663+0000 7f0f04800640 1 --2- 172.21.15.32:0/520362288 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f0ee0077010 0x7f0ee0079520 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f0efc1530a0 tx=0x7f0eec009040 comp rx=0 tx=0).stop 2024-08-03T20:50:32.666 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:32.663+0000 7f0f04800640 1 -- 172.21.15.32:0/520362288 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0efc0b4de0 msgr2=0x7f0efc1f5360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:50:32.666 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:32.663+0000 7f0f04800640 1 --2- 172.21.15.32:0/520362288 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0efc0b4de0 0x7f0efc1f5360 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7f0ef000c6b0 tx=0x7f0ef000cb70 comp rx=0 tx=0).stop 2024-08-03T20:50:32.666 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:32.663+0000 7f0f04800640 1 -- 172.21.15.32:0/520362288 shutdown_connections 2024-08-03T20:50:32.666 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:32.663+0000 7f0f04800640 1 --2- 172.21.15.32:0/520362288 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f0ee0077010 0x7f0ee0079520 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:32.666 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:32.663+0000 7f0f04800640 1 --2- 172.21.15.32:0/520362288 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f0efc1515c0 0x7f0efc1f8090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:32.666 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:32.663+0000 7f0f04800640 1 --2- 172.21.15.32:0/520362288 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f0efc1f89c0 0x7f0efc1595f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:32.666 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:32.663+0000 7f0f04800640 1 --2- 172.21.15.32:0/520362288 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0efc0b4de0 0x7f0efc1f5360 unknown :-1 s=CLOSED pgs=574 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:32.666 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:32.663+0000 7f0f04800640 1 -- 172.21.15.32:0/520362288 >> 172.21.15.32:0/520362288 conn(0x7f0efc0b1bd0 msgr2=0x7f0efc150c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:50:32.666 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:32.663+0000 7f0f04800640 1 -- 172.21.15.32:0/520362288 shutdown_connections 2024-08-03T20:50:32.667 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:32.663+0000 7f0f04800640 1 -- 172.21.15.32:0/520362288 wait complete. 2024-08-03T20:50:32.688 INFO:teuthology.orchestra.run.smithi032.stdout:{"pg_ready":true,"pg_map":{"version":142,"stamp":"2024-08-03T20:50:32.269157+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":145,"num_write_kb":1642,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":110,"ondisk_log_size":110,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":156,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":255820,"kb_used_data":6368,"kb_used_omap":12,"kb_used_meta":249331,"kb_avail":754718900,"statfs":{"total":773094113280,"available":772832153600,"internally_reserved":0,"allocated":6520832,"data_stored":3366934,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":255315505},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,4,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":29,"apply_latency_ms":29,"commit_latency_ns":29000000,"apply_latency_ns":29000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001644"},"pg_stats":[{"pgid":"9.3f","version":"0'0","reported_seq":10,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-08-03T20:50:28.199771+0000","last_change":"2024-08-03T20:50:28.185044+0000","last_active":"2024-08-03T20:50:28.199771+0000","last_peered":"2024-08-03T20:50:28.199771+0000","last_clean":"2024-08-03T20:50:28.199771+0000","last_became_active":"2024-08-03T20:50:28.183954+0000","last_became_peered":"2024-08-03T20:50:28.183954+0000","last_unstale":"2024-08-03T20:50:28.199771+0000","last_undegraded":"2024-08-03T20:50:28.199771+0000","last_fullsized":"2024-08-03T20:50:28.199771+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:50:27.143838+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:50:27.143838+0000","last_clean_scrub_stamp":"2024-08-03T20:50:27.143838+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-04T23:23:49.154260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.3e","version":"0'0","reported_seq":10,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-08-03T20:50:28.187753+0000","last_change":"2024-08-03T20:50:28.183719+0000","last_active":"2024-08-03T20:50:28.187753+0000","last_peered":"2024-08-03T20:50:28.187753+0000","last_clean":"2024-08-03T20:50:28.187753+0000","last_became_active":"2024-08-03T20:50:28.183445+0000","last_became_peered":"2024-08-03T20:50:28.183445+0000","last_unstale":"2024-08-03T20:50:28.187753+0000","last_undegraded":"2024-08-03T20:50:28.187753+0000","last_fullsized":"2024-08-03T20:50:28.187753+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:50:27.143838+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:50:27.143838+0000","last_clean_scrub_stamp":"2024-08-03T20:50:27.143838+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T00:58:25.712972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.3d","version":"0'0","reported_seq":10,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-08-03T20:50:28.176553+0000","last_change":"2024-08-03T20:50:28.169553+0000","last_active":"2024-08-03T20:50:28.176553+0000","last_peered":"2024-08-03T20:50:28.176553+0000","last_clean":"2024-08-03T20:50:28.176553+0000","last_became_active":"2024-08-03T20:50:28.169046+0000","last_became_peered":"2024-08-03T20:50:28.169046+0000","last_unstale":"2024-08-03T20:50:28.176553+0000","last_undegraded":"2024-08-03T20:50:28.176553+0000","last_fullsized":"2024-08-03T20:50:28.176553+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:50:27.143838+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:50:27.143838+0000","last_clean_scrub_stamp":"2024-08-03T20:50:27.143838+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T05:29:26.363244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.3c","version":"0'0","reported_seq":10,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-08-03T20:50:28.188157+0000","last_change":"2024-08-03T20:50:28.186439+0000","last_active":"2024-08-03T20:50:28.188157+0000","last_peered":"2024-08-03T20:50:28.188157+0000","last_clean":"2024-08-03T20:50:28.188157+0000","last_became_active":"2024-08-03T20:50:28.186099+0000","last_became_peered":"2024-08-03T20:50:28.186099+0000","last_unstale":"2024-08-03T20:50:28.188157+0000","last_undegraded":"2024-08-03T20:50:28.188157+0000","last_fullsized":"2024-08-03T20:50:28.188157+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:50:27.143838+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:50:27.143838+0000","last_clean_scrub_stamp":"2024-08-03T20:50:27.143838+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T06:39:08.774014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.3b","version":"0'0","reported_seq":10,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-08-03T20:50:28.187637+0000","last_change":"2024-08-03T20:50:28.178146+0000","last_active":"2024-08-03T20:50:28.187637+0000","last_peered":"2024-08-03T20:50:28.187637+0000","last_clean":"2024-08-03T20:50:28.187637+0000","last_became_active":"2024-08-03T20:50:28.177802+0000","last_became_peered":"2024-08-03T20:50:28.177802+0000","last_unstale":"2024-08-03T20:50:28.187637+0000","last_undegraded":"2024-08-03T20:50:28.187637+0000","last_fullsized":"2024-08-03T20:50:28.187637+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:50:27.143838+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:50:27.143838+0000","last_clean_scrub_stamp":"2024-08-03T20:50:27.143838+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T03:11:16.738801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.3a","version":"0'0","reported_seq":10,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-08-03T20:50:28.201309+0000","last_change":"2024-08-03T20:50:28.199934+0000","last_active":"2024-08-03T20:50:28.201309+0000","last_peered":"2024-08-03T20:50:28.201309+0000","last_clean":"2024-08-03T20:50:28.201309+0000","last_became_active":"2024-08-03T20:50:28.199581+0000","last_became_peered":"2024-08-03T20:50:28.199581+0000","last_unstale":"2024-08-03T20:50:28.201309+0000","last_undegraded":"2024-08-03T20:50:28.201309+0000","last_fullsized":"2024-08-03T20:50:28.201309+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:50:27.143838+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:50:27.143838+0000","last_clean_scrub_stamp":"2024-08-03T20:50:27.143838+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T06:51:23.660654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.39","version":"0'0","reported_seq":10,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-08-03T20:50:28.177802+0000","last_change":"2024-08-03T20:50:28.172144+0000","last_active":"2024-08-03T20:50:28.177802+0000","last_peered":"2024-08-03T20:50:28.177802+0000","last_clean":"2024-08-03T20:50:28.177802+0000","last_became_active":"2024-08-03T20:50:28.171813+0000","last_became_peered":"2024-08-03T20:50:28.171813+0000","last_unstale":"2024-08-03T20:50:28.177802+0000","last_undegraded":"2024-08-03T20:50:28.177802+0000","last_fullsized":"2024-08-03T20:50:28.177802+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:50:27.143838+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:50:27.143838+0000","last_clean_scrub_stamp":"2024-08-03T20:50:27.143838+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T04:39:52.662655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.38","version":"0'0","reported_seq":10,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-08-03T20:50:28.187592+0000","last_change":"2024-08-03T20:50:28.182772+0000","last_active":"2024-08-03T20:50:28.187592+0000","last_peered":"2024-08-03T20:50:28.187592+0000","last_clean":"2024-08-03T20:50:28.187592+0000","last_became_active":"2024-08-03T20:50:28.182438+0000","last_became_peered":"2024-08-03T20:50:28.182438+0000","last_unstale":"2024-08-03T20:50:28.187592+0000","last_undegraded":"2024-08-03T20:50:28.187592+0000","last_fullsized":"2024-08-03T20:50:28.187592+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:50:27.143838+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:50:27.143838+0000","last_clean_scrub_stamp":"2024-08-03T20:50:27.143838+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-04T21:28:32.662281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.37","version":"0'0","reported_seq":10,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-08-03T20:50:28.178509+0000","last_change":"2024-08-03T20:50:28.175194+0000","last_active":"2024-08-03T20:50:28.178509+0000","last_peered":"2024-08-03T20:50:28.178509+0000","last_clean":"2024-08-03T20:50:28.178509+0000","last_became_active":"2024-08-03T20:50:28.174743+0000","last_became_peered":"2024-08-03T20:50:28.174743+0000","last_unstale":"2024-08-03T20:50:28.178509+0000","last_undegraded":"2024-08-03T20:50:28.178509+0000","last_fullsized":"2024-08-03T20:50:28.178509+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:50:27.143838+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:50:27.143838+0000","last_clean_scrub_stamp":"2024-08-03T20:50:27.143838+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-04T21:55:13.089107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.36","version":"0'0","reported_seq":10,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-08-03T20:50:28.179536+0000","last_change":"2024-08-03T20:50:28.176847+0000","last_active":"2024-08-03T20:50:28.179536+0000","last_peered":"2024-08-03T20:50:28.179536+0000","last_clean":"2024-08-03T20:50:28.179536+0000","last_became_active":"2024-08-03T20:50:28.176513+0000","last_became_peered":"2024-08-03T20:50:28.176513+0000","last_unstale":"2024-08-03T20:50:28.179536+0000","last_undegraded":"2024-08-03T20:50:28.179536+0000","last_fullsized":"2024-08-03T20:50:28.179536+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:50:27.143838+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:50:27.143838+0000","last_clean_scrub_stamp":"2024-08-03T20:50:27.143838+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-04T21:21:41.546104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.35","version":"0'0","reported_seq":10,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-08-03T20:50:28.185292+0000","last_change":"2024-08-03T20:50:28.179703+0000","last_active":"2024-08-03T20:50:28.185292+0000","last_peered":"2024-08-03T20:50:28.185292+0000","last_clean":"2024-08-03T20:50:28.185292+0000","last_became_active":"2024-08-03T20:50:28.179351+0000","last_became_peered":"2024-08-03T20:50:28.179351+0000","last_unstale":"2024-08-03T20:50:28.185292+0000","last_undegraded":"2024-08-03T20:50:28.185292+0000","last_fullsized":"2024-08-03T20:50:28.185292+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:50:27.143838+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:50:27.143838+0000","last_clean_scrub_stamp":"2024-08-03T20:50:27.143838+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T05:29:30.552157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.34","version":"0'0","reported_seq":10,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-08-03T20:50:28.183335+0000","last_change":"2024-08-03T20:50:28.173786+0000","last_active":"2024-08-03T20:50:28.183335+0000","last_peered":"2024-08-03T20:50:28.183335+0000","last_clean":"2024-08-03T20:50:28.183335+0000","last_became_active":"2024-08-03T20:50:28.173395+0000","last_became_peered":"2024-08-03T20:50:28.173395+0000","last_unstale":"2024-08-03T20:50:28.183335+0000","last_undegraded":"2024-08-03T20:50:28.183335+0000","last_fullsized":"2024-08-03T20:50:28.183335+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:50:27.143838+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:50:27.143838+0000","last_clean_scrub_stamp":"2024-08-03T20:50:27.143838+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T05:29:26.363244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.33","version":"0'0","reported_seq":13,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-08-03T20:50:29.159433+0000","last_change":"2024-08-03T20:50:28.186309+0000","last_active":"2024-08-03T20:50:29.159433+0000","last_peered":"2024-08-03T20:50:29.159433+0000","last_clean":"2024-08-03T20:50:29.159433+0000","last_became_active":"2024-08-03T20:50:28.185957+0000","last_became_peered":"2024-08-03T20:50:28.185957+0000","last_unstale":"2024-08-03T20:50:29.159433+0000","last_undegraded":"2024-08-03T20:50:29.159433+0000","last_fullsized":"2024-08-03T20:50:29.159433+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:50:27.143838+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:50:27.143838+0000","last_clean_scrub_stamp":"2024-08-03T20:50:27.143838+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-04T21:54:55.043600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.32","version":"0'0","reported_seq":10,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-08-03T20:50:28.194607+0000","last_change":"2024-08-03T20:50:28.192505+0000","last_active":"2024-08-03T20:50:28.194607+0000","last_peered":"2024-08-03T20:50:28.194607+0000","last_clean":"2024-08-03T20:50:28.194607+0000","last_became_active":"2024-08-03T20:50:28.191454+0000","last_became_peered":"2024-08-03T20:50:28.191454+0000","last_unstale":"2024-08-03T20:50:28.194607+0000","last_undegraded":"2024-08-03T20:50:28.194607+0000","last_fullsized":"2024-08-03T20:50:28.194607+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:50:27.143838+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:50:27.143838+0000","last_clean_scrub_stamp":"2024-08-03T20:50:27.143838+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T07:21:23.983841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.31","version":"0'0","reported_seq":10,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-08-03T20:50:28.183868+0000","last_change":"2024-08-03T20:50:28.182696+0000","last_active":"2024-08-03T20:50:28.183868+0000","last_peered":"2024-08-03T20:50:28.183868+0000","last_clean":"2024-08-03T20:50:28.183868+0000","last_became_active":"2024-08-03T20:50:28.182359+0000","last_became_peered":"2024-08-03T20:50:28.182359+0000","last_unstale":"2024-08-03T20:50:28.183868+0000","last_undegraded":"2024-08-03T20:50:28.183868+0000","last_fullsized":"2024-08-03T20:50:28.183868+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:50:27.143838+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:50:27.143838+0000","last_clean_scrub_stamp":"2024-08-03T20:50:27.143838+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T04:16:02.684070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.30","version":"0'0","reported_seq":10,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-08-03T20:50:28.184465+0000","last_change":"2024-08-03T20:50:28.176177+0000","last_active":"2024-08-03T20:50:28.184465+0000","last_peered":"2024-08-03T20:50:28.184465+0000","last_clean":"2024-08-03T20:50:28.184465+0000","last_became_active":"2024-08-03T20:50:28.175558+0000","last_became_peered":"2024-08-03T20:50:28.175558+0000","last_unstale":"2024-08-03T20:50:28.184465+0000","last_undegraded":"2024-08-03T20:50:28.184465+0000","last_fullsized":"2024-08-03T20:50:28.184465+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:50:27.143838+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:50:27.143838+0000","last_clean_scrub_stamp":"2024-08-03T20:50:27.143838+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T06:45:18.189472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.2f","version":"0'0","reported_seq":10,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-08-03T20:50:28.187712+0000","last_change":"2024-08-03T20:50:28.182623+0000","last_active":"2024-08-03T20:50:28.187712+0000","last_peered":"2024-08-03T20:50:28.187712+0000","last_clean":"2024-08-03T20:50:28.187712+0000","last_became_active":"2024-08-03T20:50:28.182123+0000","last_became_peered":"2024-08-03T20:50:28.182123+0000","last_unstale":"2024-08-03T20:50:28.187712+0000","last_undegraded":"2024-08-03T20:50:28.187712+0000","last_fullsized":"2024-08-03T20:50:28.187712+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:50:27.143838+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:50:27.143838+0000","last_clean_scrub_stamp":"2024-08-03T20:50:27.143838+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T05:08:08.224310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.2e","version":"0'0","reported_seq":10,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-08-03T20:50:28.178896+0000","last_change":"2024-08-03T20:50:28.170884+0000","last_active":"2024-08-03T20:50:28.178896+0000","last_peered":"2024-08-03T20:50:28.178896+0000","last_clean":"2024-08-03T20:50:28.178896+0000","last_became_active":"2024-08-03T20:50:28.170550+0000","last_became_peered":"2024-08-03T20:50:28.170550+0000","last_unstale":"2024-08-03T20:50:28.178896+0000","last_undegraded":"2024-08-03T20:50:28.178896+0000","last_fullsized":"2024-08-03T20:50:28.178896+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:50:27.143838+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:50:27.143838+0000","last_clean_scrub_stamp":"2024-08-03T20:50:27.143838+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-04T22:38:57.045083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.2d","version":"0'0","reported_seq":10,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-08-03T20:50:28.190266+0000","last_change":"2024-08-03T20:50:28.183183+0000","last_active":"2024-08-03T20:50:28.190266+0000","last_peered":"2024-08-03T20:50:28.190266+0000","last_clean":"2024-08-03T20:50:28.190266+0000","last_became_active":"2024-08-03T20:50:28.182837+0000","last_became_peered":"2024-08-03T20:50:28.182837+0000","last_unstale":"2024-08-03T20:50:28.190266+0000","last_undegraded":"2024-08-03T20:50:28.190266+0000","last_fullsized":"2024-08-03T20:50:28.190266+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:50:27.143838+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:50:27.143838+0000","last_clean_scrub_stamp":"2024-08-03T20:50:27.143838+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-04T22:22:09.910171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.2c","version":"0'0","reported_seq":10,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-08-03T20:50:28.189885+0000","last_change":"2024-08-03T20:50:28.187374+0000","last_active":"2024-08-03T20:50:28.189885+0000","last_peered":"2024-08-03T20:50:28.189885+0000","last_clean":"2024-08-03T20:50:28.189885+0000","last_became_active":"2024-08-03T20:50:28.186190+0000","last_became_peered":"2024-08-03T20:50:28.186190+0000","last_unstale":"2024-08-03T20:50:28.189885+0000","last_undegraded":"2024-08-03T20:50:28.189885+0000","last_fullsized":"2024-08-03T20:50:28.189885+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:50:27.143838+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:50:27.143838+0000","last_clean_scrub_stamp":"2024-08-03T20:50:27.143838+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T05:37:03.068592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.2b","version":"0'0","reported_seq":10,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-08-03T20:50:28.182717+0000","last_change":"2024-08-03T20:50:28.175277+0000","last_active":"2024-08-03T20:50:28.182717+0000","last_peered":"2024-08-03T20:50:28.182717+0000","last_clean":"2024-08-03T20:50:28.182717+0000","last_became_active":"2024-08-03T20:50:28.172599+0000","last_became_peered":"2024-08-03T20:50:28.172599+0000","last_unstale":"2024-08-03T20:50:28.182717+0000","last_undegraded":"2024-08-03T20:50:28.182717+0000","last_fullsized":"2024-08-03T20:50:28.182717+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:50:27.143838+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:50:27.143838+0000","last_clean_scrub_stamp":"2024-08-03T20:50:27.143838+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T06:34:23.326860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.2a","version":"0'0","reported_seq":10,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-08-03T20:50:28.204428+0000","last_change":"2024-08-03T20:50:28.201340+0000","last_active":"2024-08-03T20:50:28.204428+0000","last_peered":"2024-08-03T20:50:28.204428+0000","last_clean":"2024-08-03T20:50:28.204428+0000","last_became_active":"2024-08-03T20:50:28.200126+0000","last_became_peered":"2024-08-03T20:50:28.200126+0000","last_unstale":"2024-08-03T20:50:28.204428+0000","last_undegraded":"2024-08-03T20:50:28.204428+0000","last_fullsized":"2024-08-03T20:50:28.204428+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:50:27.143838+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:50:27.143838+0000","last_clean_scrub_stamp":"2024-08-03T20:50:27.143838+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T00:04:32.467985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.29","version":"0'0","reported_seq":10,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-08-03T20:50:28.183171+0000","last_change":"2024-08-03T20:50:28.177428+0000","last_active":"2024-08-03T20:50:28.183171+0000","last_peered":"2024-08-03T20:50:28.183171+0000","last_clean":"2024-08-03T20:50:28.183171+0000","last_became_active":"2024-08-03T20:50:28.176981+0000","last_became_peered":"2024-08-03T20:50:28.176981+0000","last_unstale":"2024-08-03T20:50:28.183171+0000","last_undegraded":"2024-08-03T20:50:28.183171+0000","last_fullsized":"2024-08-03T20:50:28.183171+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:50:27.143838+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:50:27.143838+0000","last_clean_scrub_stamp":"2024-08-03T20:50:27.143838+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T03:50:47.476617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.28","version":"0'0","reported_seq":13,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-08-03T20:50:29.159809+0000","last_change":"2024-08-03T20:50:28.187029+0000","last_active":"2024-08-03T20:50:29.159809+0000","last_peered":"2024-08-03T20:50:29.159809+0000","last_clean":"2024-08-03T20:50:29.159809+0000","last_became_active":"2024-08-03T20:50:28.186671+0000","last_became_peered":"2024-08-03T20:50:28.186671+0000","last_unstale":"2024-08-03T20:50:29.159809+0000","last_undegraded":"2024-08-03T20:50:29.159809+0000","last_fullsized":"2024-08-03T20:50:29.159809+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:50:27.143838+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:50:27.143838+0000","last_clean_scrub_stamp":"2024-08-03T20:50:27.143838+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T04:40:28.587465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.27","version":"0'0","reported_seq":10,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-08-03T20:50:28.199799+0000","last_change":"2024-08-03T20:50:28.193139+0000","last_active":"2024-08-03T20:50:28.199799+0000","last_peered":"2024-08-03T20:50:28.199799+0000","last_clean":"2024-08-03T20:50:28.199799+0000","last_became_active":"2024-08-03T20:50:28.192458+0000","last_became_peered":"2024-08-03T20:50:28.192458+0000","last_unstale":"2024-08-03T20:50:28.199799+0000","last_undegraded":"2024-08-03T20:50:28.199799+0000","last_fullsized":"2024-08-03T20:50:28.199799+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:50:27.143838+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:50:27.143838+0000","last_clean_scrub_stamp":"2024-08-03T20:50:27.143838+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T00:13:43.102375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.26","version":"0'0","reported_seq":10,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-08-03T20:50:28.178100+0000","last_change":"2024-08-03T20:50:28.172136+0000","last_active":"2024-08-03T20:50:28.178100+0000","last_peered":"2024-08-03T20:50:28.178100+0000","last_clean":"2024-08-03T20:50:28.178100+0000","last_became_active":"2024-08-03T20:50:28.171742+0000","last_became_peered":"2024-08-03T20:50:28.171742+0000","last_unstale":"2024-08-03T20:50:28.178100+0000","last_undegraded":"2024-08-03T20:50:28.178100+0000","last_fullsized":"2024-08-03T20:50:28.178100+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:50:27.143838+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:50:27.143838+0000","last_clean_scrub_stamp":"2024-08-03T20:50:27.143838+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-04T22:43:40.157850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.25","version":"0'0","reported_seq":10,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-08-03T20:50:28.184302+0000","last_change":"2024-08-03T20:50:28.176849+0000","last_active":"2024-08-03T20:50:28.184302+0000","last_peered":"2024-08-03T20:50:28.184302+0000","last_clean":"2024-08-03T20:50:28.184302+0000","last_became_active":"2024-08-03T20:50:28.174658+0000","last_became_peered":"2024-08-03T20:50:28.174658+0000","last_unstale":"2024-08-03T20:50:28.184302+0000","last_undegraded":"2024-08-03T20:50:28.184302+0000","last_fullsized":"2024-08-03T20:50:28.184302+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:50:27.143838+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:50:27.143838+0000","last_clean_scrub_stamp":"2024-08-03T20:50:27.143838+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T06:39:08.774014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.24","version":"0'0","reported_seq":10,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-08-03T20:50:28.186767+0000","last_change":"2024-08-03T20:50:28.173281+0000","last_active":"2024-08-03T20:50:28.186767+0000","last_peered":"2024-08-03T20:50:28.186767+0000","last_clean":"2024-08-03T20:50:28.186767+0000","last_became_active":"2024-08-03T20:50:28.172968+0000","last_became_peered":"2024-08-03T20:50:28.172968+0000","last_unstale":"2024-08-03T20:50:28.186767+0000","last_undegraded":"2024-08-03T20:50:28.186767+0000","last_fullsized":"2024-08-03T20:50:28.186767+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:50:27.143838+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:50:27.143838+0000","last_clean_scrub_stamp":"2024-08-03T20:50:27.143838+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T01:03:50.425091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.23","version":"0'0","reported_seq":10,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-08-03T20:50:28.187507+0000","last_change":"2024-08-03T20:50:28.179835+0000","last_active":"2024-08-03T20:50:28.187507+0000","last_peered":"2024-08-03T20:50:28.187507+0000","last_clean":"2024-08-03T20:50:28.187507+0000","last_became_active":"2024-08-03T20:50:28.179537+0000","last_became_peered":"2024-08-03T20:50:28.179537+0000","last_unstale":"2024-08-03T20:50:28.187507+0000","last_undegraded":"2024-08-03T20:50:28.187507+0000","last_fullsized":"2024-08-03T20:50:28.187507+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:50:27.143838+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:50:27.143838+0000","last_clean_scrub_stamp":"2024-08-03T20:50:27.143838+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T05:53:26.931834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.22","version":"0'0","reported_seq":10,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-08-03T20:50:28.178875+0000","last_change":"2024-08-03T20:50:28.171372+0000","last_active":"2024-08-03T20:50:28.178875+0000","last_peered":"2024-08-03T20:50:28.178875+0000","last_clean":"2024-08-03T20:50:28.178875+0000","last_became_active":"2024-08-03T20:50:28.171028+0000","last_became_peered":"2024-08-03T20:50:28.171028+0000","last_unstale":"2024-08-03T20:50:28.178875+0000","last_undegraded":"2024-08-03T20:50:28.178875+0000","last_fullsized":"2024-08-03T20:50:28.178875+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:50:27.143838+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:50:27.143838+0000","last_clean_scrub_stamp":"2024-08-03T20:50:27.143838+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T03:35:47.096608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.21","version":"0'0","reported_seq":10,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-08-03T20:50:28.199922+0000","last_change":"2024-08-03T20:50:28.191047+0000","last_active":"2024-08-03T20:50:28.199922+0000","last_peered":"2024-08-03T20:50:28.199922+0000","last_clean":"2024-08-03T20:50:28.199922+0000","last_became_active":"2024-08-03T20:50:28.190769+0000","last_became_peered":"2024-08-03T20:50:28.190769+0000","last_unstale":"2024-08-03T20:50:28.199922+0000","last_undegraded":"2024-08-03T20:50:28.199922+0000","last_fullsized":"2024-08-03T20:50:28.199922+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:50:27.143838+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:50:27.143838+0000","last_clean_scrub_stamp":"2024-08-03T20:50:27.143838+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T04:59:57.334073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.20","version":"0'0","reported_seq":10,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-08-03T20:50:28.184444+0000","last_change":"2024-08-03T20:50:28.175873+0000","last_active":"2024-08-03T20:50:28.184444+0000","last_peered":"2024-08-03T20:50:28.184444+0000","last_clean":"2024-08-03T20:50:28.184444+0000","last_became_active":"2024-08-03T20:50:28.175018+0000","last_became_peered":"2024-08-03T20:50:28.175018+0000","last_unstale":"2024-08-03T20:50:28.184444+0000","last_undegraded":"2024-08-03T20:50:28.184444+0000","last_fullsized":"2024-08-03T20:50:28.184444+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:50:27.143838+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:50:27.143838+0000","last_clean_scrub_stamp":"2024-08-03T20:50:27.143838+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T03:35:47.096608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1e","version":"0'0","reported_seq":10,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-08-03T20:50:28.185189+0000","last_change":"2024-08-03T20:50:28.176616+0000","last_active":"2024-08-03T20:50:28.185189+0000","last_peered":"2024-08-03T20:50:28.185189+0000","last_clean":"2024-08-03T20:50:28.185189+0000","last_became_active":"2024-08-03T20:50:28.176274+0000","last_became_peered":"2024-08-03T20:50:28.176274+0000","last_unstale":"2024-08-03T20:50:28.185189+0000","last_undegraded":"2024-08-03T20:50:28.185189+0000","last_fullsized":"2024-08-03T20:50:28.185189+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:50:27.143838+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:50:27.143838+0000","last_clean_scrub_stamp":"2024-08-03T20:50:27.143838+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T06:34:23.326860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1f","version":"56'4","reported_seq":17,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-08-03T20:50:28.198928+0000","last_change":"2024-08-03T20:50:27.168467+0000","last_active":"2024-08-03T20:50:28.198928+0000","last_peered":"2024-08-03T20:50:28.198928+0000","last_clean":"2024-08-03T20:50:28.198928+0000","last_became_active":"2024-08-03T20:50:27.168114+0000","last_became_peered":"2024-08-03T20:50:27.168114+0000","last_unstale":"2024-08-03T20:50:28.198928+0000","last_undegraded":"2024-08-03T20:50:28.198928+0000","last_fullsized":"2024-08-03T20:50:28.198928+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:50:26.140950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:50:26.140950+0000","last_clean_scrub_stamp":"2024-08-03T20:50:26.140950+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T01:31:02.341088+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1f","version":"0'0","reported_seq":10,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-08-03T20:50:28.182602+0000","last_change":"2024-08-03T20:50:28.180222+0000","last_active":"2024-08-03T20:50:28.182602+0000","last_peered":"2024-08-03T20:50:28.182602+0000","last_clean":"2024-08-03T20:50:28.182602+0000","last_became_active":"2024-08-03T20:50:28.179359+0000","last_became_peered":"2024-08-03T20:50:28.179359+0000","last_unstale":"2024-08-03T20:50:28.182602+0000","last_undegraded":"2024-08-03T20:50:28.182602+0000","last_fullsized":"2024-08-03T20:50:28.182602+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:50:27.143838+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:50:27.143838+0000","last_clean_scrub_stamp":"2024-08-03T20:50:27.143838+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-04T23:10:22.893837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.1e","version":"56'1","reported_seq":14,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-08-03T20:50:28.198287+0000","last_change":"2024-08-03T20:50:27.176454+0000","last_active":"2024-08-03T20:50:28.198287+0000","last_peered":"2024-08-03T20:50:28.198287+0000","last_clean":"2024-08-03T20:50:28.198287+0000","last_became_active":"2024-08-03T20:50:27.176084+0000","last_became_peered":"2024-08-03T20:50:27.176084+0000","last_unstale":"2024-08-03T20:50:28.198287+0000","last_undegraded":"2024-08-03T20:50:28.198287+0000","last_fullsized":"2024-08-03T20:50:28.198287+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:50:26.140950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:50:26.140950+0000","last_clean_scrub_stamp":"2024-08-03T20:50:26.140950+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T02:29:07.610879+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.1c","version":"0'0","reported_seq":10,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-08-03T20:50:28.190213+0000","last_change":"2024-08-03T20:50:28.180444+0000","last_active":"2024-08-03T20:50:28.190213+0000","last_peered":"2024-08-03T20:50:28.190213+0000","last_clean":"2024-08-03T20:50:28.190213+0000","last_became_active":"2024-08-03T20:50:28.180126+0000","last_became_peered":"2024-08-03T20:50:28.180126+0000","last_unstale":"2024-08-03T20:50:28.190213+0000","last_undegraded":"2024-08-03T20:50:28.190213+0000","last_fullsized":"2024-08-03T20:50:28.190213+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:50:27.143838+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:50:27.143838+0000","last_clean_scrub_stamp":"2024-08-03T20:50:27.143838+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T03:50:47.476617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1d","version":"56'2","reported_seq":15,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-08-03T20:50:28.205773+0000","last_change":"2024-08-03T20:50:27.183728+0000","last_active":"2024-08-03T20:50:28.205773+0000","last_peered":"2024-08-03T20:50:28.205773+0000","last_clean":"2024-08-03T20:50:28.205773+0000","last_became_active":"2024-08-03T20:50:27.181749+0000","last_became_peered":"2024-08-03T20:50:27.181749+0000","last_unstale":"2024-08-03T20:50:28.205773+0000","last_undegraded":"2024-08-03T20:50:28.205773+0000","last_fullsized":"2024-08-03T20:50:28.205773+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:50:26.140950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:50:26.140950+0000","last_clean_scrub_stamp":"2024-08-03T20:50:26.140950+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-04T22:43:39.154962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1d","version":"0'0","reported_seq":10,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-08-03T20:50:28.184345+0000","last_change":"2024-08-03T20:50:28.175292+0000","last_active":"2024-08-03T20:50:28.184345+0000","last_peered":"2024-08-03T20:50:28.184345+0000","last_clean":"2024-08-03T20:50:28.184345+0000","last_became_active":"2024-08-03T20:50:28.174201+0000","last_became_peered":"2024-08-03T20:50:28.174201+0000","last_unstale":"2024-08-03T20:50:28.184345+0000","last_undegraded":"2024-08-03T20:50:28.184345+0000","last_fullsized":"2024-08-03T20:50:28.184345+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:50:27.143838+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:50:27.143838+0000","last_clean_scrub_stamp":"2024-08-03T20:50:27.143838+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-04T22:38:57.045083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1c","version":"56'2","reported_seq":15,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-08-03T20:50:28.198347+0000","last_change":"2024-08-03T20:50:27.171915+0000","last_active":"2024-08-03T20:50:28.198347+0000","last_peered":"2024-08-03T20:50:28.198347+0000","last_clean":"2024-08-03T20:50:28.198347+0000","last_became_active":"2024-08-03T20:50:27.171579+0000","last_became_peered":"2024-08-03T20:50:27.171579+0000","last_unstale":"2024-08-03T20:50:28.198347+0000","last_undegraded":"2024-08-03T20:50:28.198347+0000","last_fullsized":"2024-08-03T20:50:28.198347+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:50:26.140950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:50:26.140950+0000","last_clean_scrub_stamp":"2024-08-03T20:50:26.140950+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T08:31:43.595881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.1a","version":"0'0","reported_seq":10,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-08-03T20:50:28.204449+0000","last_change":"2024-08-03T20:50:28.201626+0000","last_active":"2024-08-03T20:50:28.204449+0000","last_peered":"2024-08-03T20:50:28.204449+0000","last_clean":"2024-08-03T20:50:28.204449+0000","last_became_active":"2024-08-03T20:50:28.200412+0000","last_became_peered":"2024-08-03T20:50:28.200412+0000","last_unstale":"2024-08-03T20:50:28.204449+0000","last_undegraded":"2024-08-03T20:50:28.204449+0000","last_fullsized":"2024-08-03T20:50:28.204449+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:50:27.143838+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:50:27.143838+0000","last_clean_scrub_stamp":"2024-08-03T20:50:27.143838+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T05:50:54.163139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.1b","version":"56'1","reported_seq":14,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-08-03T20:50:28.201003+0000","last_change":"2024-08-03T20:50:27.179699+0000","last_active":"2024-08-03T20:50:28.201003+0000","last_peered":"2024-08-03T20:50:28.201003+0000","last_clean":"2024-08-03T20:50:28.201003+0000","last_became_active":"2024-08-03T20:50:27.179343+0000","last_became_peered":"2024-08-03T20:50:27.179343+0000","last_unstale":"2024-08-03T20:50:28.201003+0000","last_undegraded":"2024-08-03T20:50:28.201003+0000","last_fullsized":"2024-08-03T20:50:28.201003+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:50:26.140950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:50:26.140950+0000","last_clean_scrub_stamp":"2024-08-03T20:50:26.140950+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T08:30:12.186681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.1b","version":"0'0","reported_seq":10,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-08-03T20:50:28.189035+0000","last_change":"2024-08-03T20:50:28.185354+0000","last_active":"2024-08-03T20:50:28.189035+0000","last_peered":"2024-08-03T20:50:28.189035+0000","last_clean":"2024-08-03T20:50:28.189035+0000","last_became_active":"2024-08-03T20:50:28.185048+0000","last_became_peered":"2024-08-03T20:50:28.185048+0000","last_unstale":"2024-08-03T20:50:28.189035+0000","last_undegraded":"2024-08-03T20:50:28.189035+0000","last_fullsized":"2024-08-03T20:50:28.189035+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:50:27.143838+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:50:27.143838+0000","last_clean_scrub_stamp":"2024-08-03T20:50:27.143838+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-04T22:17:31.483960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.1a","version":"56'1","reported_seq":14,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-08-03T20:50:28.194366+0000","last_change":"2024-08-03T20:50:27.170687+0000","last_active":"2024-08-03T20:50:28.194366+0000","last_peered":"2024-08-03T20:50:28.194366+0000","last_clean":"2024-08-03T20:50:28.194366+0000","last_became_active":"2024-08-03T20:50:27.170379+0000","last_became_peered":"2024-08-03T20:50:27.170379+0000","last_unstale":"2024-08-03T20:50:28.194366+0000","last_undegraded":"2024-08-03T20:50:28.194366+0000","last_fullsized":"2024-08-03T20:50:28.194366+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:50:26.140950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:50:26.140950+0000","last_clean_scrub_stamp":"2024-08-03T20:50:26.140950+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-04T23:32:46.677480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.18","version":"0'0","reported_seq":13,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-08-03T20:50:29.159231+0000","last_change":"2024-08-03T20:50:28.180017+0000","last_active":"2024-08-03T20:50:29.159231+0000","last_peered":"2024-08-03T20:50:29.159231+0000","last_clean":"2024-08-03T20:50:29.159231+0000","last_became_active":"2024-08-03T20:50:28.179756+0000","last_became_peered":"2024-08-03T20:50:28.179756+0000","last_unstale":"2024-08-03T20:50:29.159231+0000","last_undegraded":"2024-08-03T20:50:29.159231+0000","last_fullsized":"2024-08-03T20:50:29.159231+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:50:27.143838+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:50:27.143838+0000","last_clean_scrub_stamp":"2024-08-03T20:50:27.143838+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-04T23:10:22.893837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.19","version":"0'0","reported_seq":13,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-08-03T20:50:28.165588+0000","last_change":"2024-08-03T20:50:27.178443+0000","last_active":"2024-08-03T20:50:28.165588+0000","last_peered":"2024-08-03T20:50:28.165588+0000","last_clean":"2024-08-03T20:50:28.165588+0000","last_became_active":"2024-08-03T20:50:27.178092+0000","last_became_peered":"2024-08-03T20:50:27.178092+0000","last_unstale":"2024-08-03T20:50:28.165588+0000","last_undegraded":"2024-08-03T20:50:28.165588+0000","last_fullsized":"2024-08-03T20:50:28.165588+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:50:26.140950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:50:26.140950+0000","last_clean_scrub_stamp":"2024-08-03T20:50:26.140950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T01:31:02.341088+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.19","version":"0'0","reported_seq":10,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-08-03T20:50:28.187540+0000","last_change":"2024-08-03T20:50:28.182908+0000","last_active":"2024-08-03T20:50:28.187540+0000","last_peered":"2024-08-03T20:50:28.187540+0000","last_clean":"2024-08-03T20:50:28.187540+0000","last_became_active":"2024-08-03T20:50:28.182547+0000","last_became_peered":"2024-08-03T20:50:28.182547+0000","last_unstale":"2024-08-03T20:50:28.187540+0000","last_undegraded":"2024-08-03T20:50:28.187540+0000","last_fullsized":"2024-08-03T20:50:28.187540+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:50:27.143838+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:50:27.143838+0000","last_clean_scrub_stamp":"2024-08-03T20:50:27.143838+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-04T20:54:07.705607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.18","version":"0'0","reported_seq":13,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-08-03T20:50:28.165357+0000","last_change":"2024-08-03T20:50:27.178185+0000","last_active":"2024-08-03T20:50:28.165357+0000","last_peered":"2024-08-03T20:50:28.165357+0000","last_clean":"2024-08-03T20:50:28.165357+0000","last_became_active":"2024-08-03T20:50:27.177859+0000","last_became_peered":"2024-08-03T20:50:27.177859+0000","last_unstale":"2024-08-03T20:50:28.165357+0000","last_undegraded":"2024-08-03T20:50:28.165357+0000","last_fullsized":"2024-08-03T20:50:28.165357+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:50:26.140950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:50:26.140950+0000","last_clean_scrub_stamp":"2024-08-03T20:50:26.140950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T00:24:04.506049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.16","version":"0'0","reported_seq":10,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-08-03T20:50:28.183797+0000","last_change":"2024-08-03T20:50:28.182599+0000","last_active":"2024-08-03T20:50:28.183797+0000","last_peered":"2024-08-03T20:50:28.183797+0000","last_clean":"2024-08-03T20:50:28.183797+0000","last_became_active":"2024-08-03T20:50:28.182066+0000","last_became_peered":"2024-08-03T20:50:28.182066+0000","last_unstale":"2024-08-03T20:50:28.183797+0000","last_undegraded":"2024-08-03T20:50:28.183797+0000","last_fullsized":"2024-08-03T20:50:28.183797+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:50:27.143838+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:50:27.143838+0000","last_clean_scrub_stamp":"2024-08-03T20:50:27.143838+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-04T20:53:15.943888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.17","version":"0'0","reported_seq":13,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-08-03T20:50:28.158965+0000","last_change":"2024-08-03T20:50:27.185990+0000","last_active":"2024-08-03T20:50:28.158965+0000","last_peered":"2024-08-03T20:50:28.158965+0000","last_clean":"2024-08-03T20:50:28.158965+0000","last_became_active":"2024-08-03T20:50:27.185658+0000","last_became_peered":"2024-08-03T20:50:27.185658+0000","last_unstale":"2024-08-03T20:50:28.158965+0000","last_undegraded":"2024-08-03T20:50:28.158965+0000","last_fullsized":"2024-08-03T20:50:28.158965+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:50:26.140950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:50:26.140950+0000","last_clean_scrub_stamp":"2024-08-03T20:50:26.140950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-04T23:15:19.049204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.17","version":"0'0","reported_seq":10,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-08-03T20:50:28.184085+0000","last_change":"2024-08-03T20:50:28.181105+0000","last_active":"2024-08-03T20:50:28.184085+0000","last_peered":"2024-08-03T20:50:28.184085+0000","last_clean":"2024-08-03T20:50:28.184085+0000","last_became_active":"2024-08-03T20:50:28.180745+0000","last_became_peered":"2024-08-03T20:50:28.180745+0000","last_unstale":"2024-08-03T20:50:28.184085+0000","last_undegraded":"2024-08-03T20:50:28.184085+0000","last_fullsized":"2024-08-03T20:50:28.184085+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:50:27.143838+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:50:27.143838+0000","last_clean_scrub_stamp":"2024-08-03T20:50:27.143838+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T02:56:48.726993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.16","version":"0'0","reported_seq":13,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-08-03T20:50:28.174724+0000","last_change":"2024-08-03T20:50:27.180129+0000","last_active":"2024-08-03T20:50:28.174724+0000","last_peered":"2024-08-03T20:50:28.174724+0000","last_clean":"2024-08-03T20:50:28.174724+0000","last_became_active":"2024-08-03T20:50:27.179815+0000","last_became_peered":"2024-08-03T20:50:27.179815+0000","last_unstale":"2024-08-03T20:50:28.174724+0000","last_undegraded":"2024-08-03T20:50:28.174724+0000","last_fullsized":"2024-08-03T20:50:28.174724+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:50:26.140950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:50:26.140950+0000","last_clean_scrub_stamp":"2024-08-03T20:50:26.140950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-04T22:27:50.388714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.14","version":"0'0","reported_seq":10,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-08-03T20:50:28.202040+0000","last_change":"2024-08-03T20:50:28.192110+0000","last_active":"2024-08-03T20:50:28.202040+0000","last_peered":"2024-08-03T20:50:28.202040+0000","last_clean":"2024-08-03T20:50:28.202040+0000","last_became_active":"2024-08-03T20:50:28.183686+0000","last_became_peered":"2024-08-03T20:50:28.183686+0000","last_unstale":"2024-08-03T20:50:28.202040+0000","last_undegraded":"2024-08-03T20:50:28.202040+0000","last_fullsized":"2024-08-03T20:50:28.202040+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:50:27.143838+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:50:27.143838+0000","last_clean_scrub_stamp":"2024-08-03T20:50:27.143838+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-04T21:54:55.043600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.15","version":"56'3","reported_seq":16,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-08-03T20:50:28.205573+0000","last_change":"2024-08-03T20:50:27.176805+0000","last_active":"2024-08-03T20:50:28.205573+0000","last_peered":"2024-08-03T20:50:28.205573+0000","last_clean":"2024-08-03T20:50:28.205573+0000","last_became_active":"2024-08-03T20:50:27.176513+0000","last_became_peered":"2024-08-03T20:50:27.176513+0000","last_unstale":"2024-08-03T20:50:28.205573+0000","last_undegraded":"2024-08-03T20:50:28.205573+0000","last_fullsized":"2024-08-03T20:50:28.205573+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:50:26.140950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:50:26.140950+0000","last_clean_scrub_stamp":"2024-08-03T20:50:26.140950+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-04T22:43:39.154962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.15","version":"0'0","reported_seq":10,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-08-03T20:50:28.183971+0000","last_change":"2024-08-03T20:50:28.180318+0000","last_active":"2024-08-03T20:50:28.183971+0000","last_peered":"2024-08-03T20:50:28.183971+0000","last_clean":"2024-08-03T20:50:28.183971+0000","last_became_active":"2024-08-03T20:50:28.179795+0000","last_became_peered":"2024-08-03T20:50:28.179795+0000","last_unstale":"2024-08-03T20:50:28.183971+0000","last_undegraded":"2024-08-03T20:50:28.183971+0000","last_fullsized":"2024-08-03T20:50:28.183971+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:50:27.143838+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:50:27.143838+0000","last_clean_scrub_stamp":"2024-08-03T20:50:27.143838+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-04T22:40:26.512652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.14","version":"56'6","reported_seq":19,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-08-03T20:50:28.206722+0000","last_change":"2024-08-03T20:50:27.181537+0000","last_active":"2024-08-03T20:50:28.206722+0000","last_peered":"2024-08-03T20:50:28.206722+0000","last_clean":"2024-08-03T20:50:28.206722+0000","last_became_active":"2024-08-03T20:50:27.181214+0000","last_became_peered":"2024-08-03T20:50:27.181214+0000","last_unstale":"2024-08-03T20:50:28.206722+0000","last_undegraded":"2024-08-03T20:50:28.206722+0000","last_fullsized":"2024-08-03T20:50:28.206722+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:50:26.140950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:50:26.140950+0000","last_clean_scrub_stamp":"2024-08-03T20:50:26.140950+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T00:30:52.441334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.12","version":"0'0","reported_seq":10,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-08-03T20:50:28.188204+0000","last_change":"2024-08-03T20:50:28.186741+0000","last_active":"2024-08-03T20:50:28.188204+0000","last_peered":"2024-08-03T20:50:28.188204+0000","last_clean":"2024-08-03T20:50:28.188204+0000","last_became_active":"2024-08-03T20:50:28.185916+0000","last_became_peered":"2024-08-03T20:50:28.185916+0000","last_unstale":"2024-08-03T20:50:28.188204+0000","last_undegraded":"2024-08-03T20:50:28.188204+0000","last_fullsized":"2024-08-03T20:50:28.188204+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:50:27.143838+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:50:27.143838+0000","last_clean_scrub_stamp":"2024-08-03T20:50:27.143838+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T04:30:20.080808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.13","version":"56'2","reported_seq":15,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-08-03T20:50:28.205115+0000","last_change":"2024-08-03T20:50:27.180288+0000","last_active":"2024-08-03T20:50:28.205115+0000","last_peered":"2024-08-03T20:50:28.205115+0000","last_clean":"2024-08-03T20:50:28.205115+0000","last_became_active":"2024-08-03T20:50:27.179946+0000","last_became_peered":"2024-08-03T20:50:27.179946+0000","last_unstale":"2024-08-03T20:50:28.205115+0000","last_undegraded":"2024-08-03T20:50:28.205115+0000","last_fullsized":"2024-08-03T20:50:28.205115+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:50:26.140950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:50:26.140950+0000","last_clean_scrub_stamp":"2024-08-03T20:50:26.140950+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T06:39:42.092083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.13","version":"0'0","reported_seq":10,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-08-03T20:50:28.183118+0000","last_change":"2024-08-03T20:50:28.176762+0000","last_active":"2024-08-03T20:50:28.183118+0000","last_peered":"2024-08-03T20:50:28.183118+0000","last_clean":"2024-08-03T20:50:28.183118+0000","last_became_active":"2024-08-03T20:50:28.176375+0000","last_became_peered":"2024-08-03T20:50:28.176375+0000","last_unstale":"2024-08-03T20:50:28.183118+0000","last_undegraded":"2024-08-03T20:50:28.183118+0000","last_fullsized":"2024-08-03T20:50:28.183118+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:50:27.143838+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:50:27.143838+0000","last_clean_scrub_stamp":"2024-08-03T20:50:27.143838+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T04:30:20.080808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.12","version":"0'0","reported_seq":15,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-08-03T20:50:29.159389+0000","last_change":"2024-08-03T20:50:27.169706+0000","last_active":"2024-08-03T20:50:29.159389+0000","last_peered":"2024-08-03T20:50:29.159389+0000","last_clean":"2024-08-03T20:50:29.159389+0000","last_became_active":"2024-08-03T20:50:27.169312+0000","last_became_peered":"2024-08-03T20:50:27.169312+0000","last_unstale":"2024-08-03T20:50:29.159389+0000","last_undegraded":"2024-08-03T20:50:29.159389+0000","last_fullsized":"2024-08-03T20:50:29.159389+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:50:26.140950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:50:26.140950+0000","last_clean_scrub_stamp":"2024-08-03T20:50:26.140950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T04:36:51.691682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.10","version":"0'0","reported_seq":10,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-08-03T20:50:28.181146+0000","last_change":"2024-08-03T20:50:28.178880+0000","last_active":"2024-08-03T20:50:28.181146+0000","last_peered":"2024-08-03T20:50:28.181146+0000","last_clean":"2024-08-03T20:50:28.181146+0000","last_became_active":"2024-08-03T20:50:28.171479+0000","last_became_peered":"2024-08-03T20:50:28.171479+0000","last_unstale":"2024-08-03T20:50:28.181146+0000","last_undegraded":"2024-08-03T20:50:28.181146+0000","last_fullsized":"2024-08-03T20:50:28.181146+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:50:27.143838+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:50:27.143838+0000","last_clean_scrub_stamp":"2024-08-03T20:50:27.143838+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T05:49:41.217508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.11","version":"0'0","reported_seq":13,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-08-03T20:50:28.167750+0000","last_change":"2024-08-03T20:50:27.168903+0000","last_active":"2024-08-03T20:50:28.167750+0000","last_peered":"2024-08-03T20:50:28.167750+0000","last_clean":"2024-08-03T20:50:28.167750+0000","last_became_active":"2024-08-03T20:50:27.167849+0000","last_became_peered":"2024-08-03T20:50:27.167849+0000","last_unstale":"2024-08-03T20:50:28.167750+0000","last_undegraded":"2024-08-03T20:50:28.167750+0000","last_fullsized":"2024-08-03T20:50:28.167750+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:50:26.140950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:50:26.140950+0000","last_clean_scrub_stamp":"2024-08-03T20:50:26.140950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T05:54:06.179784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.11","version":"0'0","reported_seq":10,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-08-03T20:50:28.182699+0000","last_change":"2024-08-03T20:50:28.176478+0000","last_active":"2024-08-03T20:50:28.182699+0000","last_peered":"2024-08-03T20:50:28.182699+0000","last_clean":"2024-08-03T20:50:28.182699+0000","last_became_active":"2024-08-03T20:50:28.176146+0000","last_became_peered":"2024-08-03T20:50:28.176146+0000","last_unstale":"2024-08-03T20:50:28.182699+0000","last_undegraded":"2024-08-03T20:50:28.182699+0000","last_fullsized":"2024-08-03T20:50:28.182699+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:50:27.143838+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:50:27.143838+0000","last_clean_scrub_stamp":"2024-08-03T20:50:27.143838+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T06:39:08.774014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.10","version":"56'8","reported_seq":21,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-08-03T20:50:28.196649+0000","last_change":"2024-08-03T20:50:27.173089+0000","last_active":"2024-08-03T20:50:28.196649+0000","last_peered":"2024-08-03T20:50:28.196649+0000","last_clean":"2024-08-03T20:50:28.196649+0000","last_became_active":"2024-08-03T20:50:27.172165+0000","last_became_peered":"2024-08-03T20:50:27.172165+0000","last_unstale":"2024-08-03T20:50:28.196649+0000","last_undegraded":"2024-08-03T20:50:28.196649+0000","last_fullsized":"2024-08-03T20:50:28.196649+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:50:26.140950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:50:26.140950+0000","last_clean_scrub_stamp":"2024-08-03T20:50:26.140950+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-04T23:32:46.677480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.e","version":"0'0","reported_seq":10,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-08-03T20:50:28.182041+0000","last_change":"2024-08-03T20:50:28.172981+0000","last_active":"2024-08-03T20:50:28.182041+0000","last_peered":"2024-08-03T20:50:28.182041+0000","last_clean":"2024-08-03T20:50:28.182041+0000","last_became_active":"2024-08-03T20:50:28.172642+0000","last_became_peered":"2024-08-03T20:50:28.172642+0000","last_unstale":"2024-08-03T20:50:28.182041+0000","last_undegraded":"2024-08-03T20:50:28.182041+0000","last_fullsized":"2024-08-03T20:50:28.182041+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:50:27.143838+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:50:27.143838+0000","last_clean_scrub_stamp":"2024-08-03T20:50:27.143838+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T06:45:18.189472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.f","version":"56'1","reported_seq":14,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-08-03T20:50:28.205205+0000","last_change":"2024-08-03T20:50:27.182958+0000","last_active":"2024-08-03T20:50:28.205205+0000","last_peered":"2024-08-03T20:50:28.205205+0000","last_clean":"2024-08-03T20:50:28.205205+0000","last_became_active":"2024-08-03T20:50:27.181672+0000","last_became_peered":"2024-08-03T20:50:27.181672+0000","last_unstale":"2024-08-03T20:50:28.205205+0000","last_undegraded":"2024-08-03T20:50:28.205205+0000","last_fullsized":"2024-08-03T20:50:28.205205+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:50:26.140950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:50:26.140950+0000","last_clean_scrub_stamp":"2024-08-03T20:50:26.140950+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T08:28:57.064216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.f","version":"0'0","reported_seq":10,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-08-03T20:50:28.187584+0000","last_change":"2024-08-03T20:50:28.185915+0000","last_active":"2024-08-03T20:50:28.187584+0000","last_peered":"2024-08-03T20:50:28.187584+0000","last_clean":"2024-08-03T20:50:28.187584+0000","last_became_active":"2024-08-03T20:50:28.185636+0000","last_became_peered":"2024-08-03T20:50:28.185636+0000","last_unstale":"2024-08-03T20:50:28.187584+0000","last_undegraded":"2024-08-03T20:50:28.187584+0000","last_fullsized":"2024-08-03T20:50:28.187584+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:50:27.143838+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:50:27.143838+0000","last_clean_scrub_stamp":"2024-08-03T20:50:27.143838+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T06:45:18.189472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.e","version":"56'2","reported_seq":15,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-08-03T20:50:28.201380+0000","last_change":"2024-08-03T20:50:27.185691+0000","last_active":"2024-08-03T20:50:28.201380+0000","last_peered":"2024-08-03T20:50:28.201380+0000","last_clean":"2024-08-03T20:50:28.201380+0000","last_became_active":"2024-08-03T20:50:27.185368+0000","last_became_peered":"2024-08-03T20:50:27.185368+0000","last_unstale":"2024-08-03T20:50:28.201380+0000","last_undegraded":"2024-08-03T20:50:28.201380+0000","last_fullsized":"2024-08-03T20:50:28.201380+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:50:26.140950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:50:26.140950+0000","last_clean_scrub_stamp":"2024-08-03T20:50:26.140950+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T08:31:43.595881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.c","version":"0'0","reported_seq":10,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-08-03T20:50:28.188101+0000","last_change":"2024-08-03T20:50:28.181270+0000","last_active":"2024-08-03T20:50:28.188101+0000","last_peered":"2024-08-03T20:50:28.188101+0000","last_clean":"2024-08-03T20:50:28.188101+0000","last_became_active":"2024-08-03T20:50:28.180955+0000","last_became_peered":"2024-08-03T20:50:28.180955+0000","last_unstale":"2024-08-03T20:50:28.188101+0000","last_undegraded":"2024-08-03T20:50:28.188101+0000","last_fullsized":"2024-08-03T20:50:28.188101+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:50:27.143838+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:50:27.143838+0000","last_clean_scrub_stamp":"2024-08-03T20:50:27.143838+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T06:59:54.907087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.d","version":"56'1","reported_seq":14,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-08-03T20:50:28.198354+0000","last_change":"2024-08-03T20:50:27.169583+0000","last_active":"2024-08-03T20:50:28.198354+0000","last_peered":"2024-08-03T20:50:28.198354+0000","last_clean":"2024-08-03T20:50:28.198354+0000","last_became_active":"2024-08-03T20:50:27.168862+0000","last_became_peered":"2024-08-03T20:50:27.168862+0000","last_unstale":"2024-08-03T20:50:28.198354+0000","last_undegraded":"2024-08-03T20:50:28.198354+0000","last_fullsized":"2024-08-03T20:50:28.198354+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:50:26.140950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:50:26.140950+0000","last_clean_scrub_stamp":"2024-08-03T20:50:26.140950+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-04T23:15:19.049204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.d","version":"0'0","reported_seq":10,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-08-03T20:50:28.183845+0000","last_change":"2024-08-03T20:50:28.180087+0000","last_active":"2024-08-03T20:50:28.183845+0000","last_peered":"2024-08-03T20:50:28.183845+0000","last_clean":"2024-08-03T20:50:28.183845+0000","last_became_active":"2024-08-03T20:50:28.179373+0000","last_became_peered":"2024-08-03T20:50:28.179373+0000","last_unstale":"2024-08-03T20:50:28.183845+0000","last_undegraded":"2024-08-03T20:50:28.183845+0000","last_fullsized":"2024-08-03T20:50:28.183845+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:50:27.143838+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:50:27.143838+0000","last_clean_scrub_stamp":"2024-08-03T20:50:27.143838+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T06:51:23.660654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.c","version":"0'0","reported_seq":13,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-08-03T20:50:28.159661+0000","last_change":"2024-08-03T20:50:27.178147+0000","last_active":"2024-08-03T20:50:28.159661+0000","last_peered":"2024-08-03T20:50:28.159661+0000","last_clean":"2024-08-03T20:50:28.159661+0000","last_became_active":"2024-08-03T20:50:27.177746+0000","last_became_peered":"2024-08-03T20:50:27.177746+0000","last_unstale":"2024-08-03T20:50:28.159661+0000","last_undegraded":"2024-08-03T20:50:28.159661+0000","last_fullsized":"2024-08-03T20:50:28.159661+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:50:26.140950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:50:26.140950+0000","last_clean_scrub_stamp":"2024-08-03T20:50:26.140950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T05:54:06.179784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.a","version":"0'0","reported_seq":10,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-08-03T20:50:28.188187+0000","last_change":"2024-08-03T20:50:28.186221+0000","last_active":"2024-08-03T20:50:28.188187+0000","last_peered":"2024-08-03T20:50:28.188187+0000","last_clean":"2024-08-03T20:50:28.188187+0000","last_became_active":"2024-08-03T20:50:28.185555+0000","last_became_peered":"2024-08-03T20:50:28.185555+0000","last_unstale":"2024-08-03T20:50:28.188187+0000","last_undegraded":"2024-08-03T20:50:28.188187+0000","last_fullsized":"2024-08-03T20:50:28.188187+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:50:27.143838+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:50:27.143838+0000","last_clean_scrub_stamp":"2024-08-03T20:50:27.143838+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T06:34:23.326860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.b","version":"56'2","reported_seq":15,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-08-03T20:50:28.203426+0000","last_change":"2024-08-03T20:50:27.177434+0000","last_active":"2024-08-03T20:50:28.203426+0000","last_peered":"2024-08-03T20:50:28.203426+0000","last_clean":"2024-08-03T20:50:28.203426+0000","last_became_active":"2024-08-03T20:50:27.177093+0000","last_became_peered":"2024-08-03T20:50:27.177093+0000","last_unstale":"2024-08-03T20:50:28.203426+0000","last_undegraded":"2024-08-03T20:50:28.203426+0000","last_fullsized":"2024-08-03T20:50:28.203426+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:50:26.140950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:50:26.140950+0000","last_clean_scrub_stamp":"2024-08-03T20:50:26.140950+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-04T21:23:04.443445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.b","version":"0'0","reported_seq":10,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-08-03T20:50:28.190276+0000","last_change":"2024-08-03T20:50:28.181544+0000","last_active":"2024-08-03T20:50:28.190276+0000","last_peered":"2024-08-03T20:50:28.190276+0000","last_clean":"2024-08-03T20:50:28.190276+0000","last_became_active":"2024-08-03T20:50:28.181116+0000","last_became_peered":"2024-08-03T20:50:28.181116+0000","last_unstale":"2024-08-03T20:50:28.190276+0000","last_undegraded":"2024-08-03T20:50:28.190276+0000","last_fullsized":"2024-08-03T20:50:28.190276+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:50:27.143838+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:50:27.143838+0000","last_clean_scrub_stamp":"2024-08-03T20:50:27.143838+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T02:25:34.324726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.a","version":"0'0","reported_seq":13,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-08-03T20:50:28.163649+0000","last_change":"2024-08-03T20:50:27.182689+0000","last_active":"2024-08-03T20:50:28.163649+0000","last_peered":"2024-08-03T20:50:28.163649+0000","last_clean":"2024-08-03T20:50:28.163649+0000","last_became_active":"2024-08-03T20:50:27.179597+0000","last_became_peered":"2024-08-03T20:50:27.179597+0000","last_unstale":"2024-08-03T20:50:28.163649+0000","last_undegraded":"2024-08-03T20:50:28.163649+0000","last_fullsized":"2024-08-03T20:50:28.163649+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:50:26.140950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:50:26.140950+0000","last_clean_scrub_stamp":"2024-08-03T20:50:26.140950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T04:36:51.691682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.9","version":"0'0","reported_seq":10,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-08-03T20:50:28.184160+0000","last_change":"2024-08-03T20:50:28.170476+0000","last_active":"2024-08-03T20:50:28.184160+0000","last_peered":"2024-08-03T20:50:28.184160+0000","last_clean":"2024-08-03T20:50:28.184160+0000","last_became_active":"2024-08-03T20:50:28.170162+0000","last_became_peered":"2024-08-03T20:50:28.170162+0000","last_unstale":"2024-08-03T20:50:28.184160+0000","last_undegraded":"2024-08-03T20:50:28.184160+0000","last_fullsized":"2024-08-03T20:50:28.184160+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:50:27.143838+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:50:27.143838+0000","last_clean_scrub_stamp":"2024-08-03T20:50:27.143838+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T05:29:26.363244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.8","version":"0'0","reported_seq":13,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-08-03T20:50:28.163532+0000","last_change":"2024-08-03T20:50:27.170004+0000","last_active":"2024-08-03T20:50:28.163532+0000","last_peered":"2024-08-03T20:50:28.163532+0000","last_clean":"2024-08-03T20:50:28.163532+0000","last_became_active":"2024-08-03T20:50:27.169688+0000","last_became_peered":"2024-08-03T20:50:27.169688+0000","last_unstale":"2024-08-03T20:50:28.163532+0000","last_undegraded":"2024-08-03T20:50:28.163532+0000","last_fullsized":"2024-08-03T20:50:28.163532+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:50:26.140950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:50:26.140950+0000","last_clean_scrub_stamp":"2024-08-03T20:50:26.140950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T05:50:17.238961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.6","version":"0'0","reported_seq":10,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-08-03T20:50:28.185200+0000","last_change":"2024-08-03T20:50:28.184395+0000","last_active":"2024-08-03T20:50:28.185200+0000","last_peered":"2024-08-03T20:50:28.185200+0000","last_clean":"2024-08-03T20:50:28.185200+0000","last_became_active":"2024-08-03T20:50:28.184049+0000","last_became_peered":"2024-08-03T20:50:28.184049+0000","last_unstale":"2024-08-03T20:50:28.185200+0000","last_undegraded":"2024-08-03T20:50:28.185200+0000","last_fullsized":"2024-08-03T20:50:28.185200+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:50:27.143838+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:50:27.143838+0000","last_clean_scrub_stamp":"2024-08-03T20:50:27.143838+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T03:35:47.096608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.7","version":"56'2","reported_seq":15,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-08-03T20:50:28.205413+0000","last_change":"2024-08-03T20:50:27.177097+0000","last_active":"2024-08-03T20:50:28.205413+0000","last_peered":"2024-08-03T20:50:28.205413+0000","last_clean":"2024-08-03T20:50:28.205413+0000","last_became_active":"2024-08-03T20:50:27.176767+0000","last_became_peered":"2024-08-03T20:50:27.176767+0000","last_unstale":"2024-08-03T20:50:28.205413+0000","last_undegraded":"2024-08-03T20:50:28.205413+0000","last_fullsized":"2024-08-03T20:50:28.205413+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:50:26.140950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:50:26.140950+0000","last_clean_scrub_stamp":"2024-08-03T20:50:26.140950+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-04T23:11:29.720796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.7","version":"0'0","reported_seq":10,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-08-03T20:50:28.183643+0000","last_change":"2024-08-03T20:50:28.174143+0000","last_active":"2024-08-03T20:50:28.183643+0000","last_peered":"2024-08-03T20:50:28.183643+0000","last_clean":"2024-08-03T20:50:28.183643+0000","last_became_active":"2024-08-03T20:50:28.173270+0000","last_became_peered":"2024-08-03T20:50:28.173270+0000","last_unstale":"2024-08-03T20:50:28.183643+0000","last_undegraded":"2024-08-03T20:50:28.183643+0000","last_fullsized":"2024-08-03T20:50:28.183643+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:50:27.143838+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:50:27.143838+0000","last_clean_scrub_stamp":"2024-08-03T20:50:27.143838+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-04T22:38:57.045083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.6","version":"0'0","reported_seq":13,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-08-03T20:50:28.161128+0000","last_change":"2024-08-03T20:50:27.185137+0000","last_active":"2024-08-03T20:50:28.161128+0000","last_peered":"2024-08-03T20:50:28.161128+0000","last_clean":"2024-08-03T20:50:28.161128+0000","last_became_active":"2024-08-03T20:50:27.184453+0000","last_became_peered":"2024-08-03T20:50:27.184453+0000","last_unstale":"2024-08-03T20:50:28.161128+0000","last_undegraded":"2024-08-03T20:50:28.161128+0000","last_fullsized":"2024-08-03T20:50:28.161128+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:50:26.140950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:50:26.140950+0000","last_clean_scrub_stamp":"2024-08-03T20:50:26.140950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-04T23:15:19.049204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.4","version":"0'0","reported_seq":10,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-08-03T20:50:28.194848+0000","last_change":"2024-08-03T20:50:28.176191+0000","last_active":"2024-08-03T20:50:28.194848+0000","last_peered":"2024-08-03T20:50:28.194848+0000","last_clean":"2024-08-03T20:50:28.194848+0000","last_became_active":"2024-08-03T20:50:28.175386+0000","last_became_peered":"2024-08-03T20:50:28.175386+0000","last_unstale":"2024-08-03T20:50:28.194848+0000","last_undegraded":"2024-08-03T20:50:28.194848+0000","last_fullsized":"2024-08-03T20:50:28.194848+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:50:27.143838+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:50:27.143838+0000","last_clean_scrub_stamp":"2024-08-03T20:50:27.143838+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T03:40:06.179894+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.5","version":"0'0","reported_seq":15,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-08-03T20:50:29.159456+0000","last_change":"2024-08-03T20:50:27.177260+0000","last_active":"2024-08-03T20:50:29.159456+0000","last_peered":"2024-08-03T20:50:29.159456+0000","last_clean":"2024-08-03T20:50:29.159456+0000","last_became_active":"2024-08-03T20:50:27.176895+0000","last_became_peered":"2024-08-03T20:50:27.176895+0000","last_unstale":"2024-08-03T20:50:29.159456+0000","last_undegraded":"2024-08-03T20:50:29.159456+0000","last_fullsized":"2024-08-03T20:50:29.159456+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:50:26.140950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:50:26.140950+0000","last_clean_scrub_stamp":"2024-08-03T20:50:26.140950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-04T23:11:29.720796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.5","version":"0'0","reported_seq":10,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-08-03T20:50:28.184326+0000","last_change":"2024-08-03T20:50:28.177128+0000","last_active":"2024-08-03T20:50:28.184326+0000","last_peered":"2024-08-03T20:50:28.184326+0000","last_clean":"2024-08-03T20:50:28.184326+0000","last_became_active":"2024-08-03T20:50:28.175490+0000","last_became_peered":"2024-08-03T20:50:28.175490+0000","last_unstale":"2024-08-03T20:50:28.184326+0000","last_undegraded":"2024-08-03T20:50:28.184326+0000","last_fullsized":"2024-08-03T20:50:28.184326+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:50:27.143838+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:50:27.143838+0000","last_clean_scrub_stamp":"2024-08-03T20:50:27.143838+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T04:30:20.080808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.4","version":"0'0","reported_seq":13,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-08-03T20:50:28.168531+0000","last_change":"2024-08-03T20:50:27.176793+0000","last_active":"2024-08-03T20:50:28.168531+0000","last_peered":"2024-08-03T20:50:28.168531+0000","last_clean":"2024-08-03T20:50:28.168531+0000","last_became_active":"2024-08-03T20:50:27.176490+0000","last_became_peered":"2024-08-03T20:50:27.176490+0000","last_unstale":"2024-08-03T20:50:28.168531+0000","last_undegraded":"2024-08-03T20:50:28.168531+0000","last_fullsized":"2024-08-03T20:50:28.168531+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:50:26.140950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:50:26.140950+0000","last_clean_scrub_stamp":"2024-08-03T20:50:26.140950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-04T23:32:46.677480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.2","version":"0'0","reported_seq":10,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-08-03T20:50:28.189255+0000","last_change":"2024-08-03T20:50:28.184522+0000","last_active":"2024-08-03T20:50:28.189255+0000","last_peered":"2024-08-03T20:50:28.189255+0000","last_clean":"2024-08-03T20:50:28.189255+0000","last_became_active":"2024-08-03T20:50:28.184144+0000","last_became_peered":"2024-08-03T20:50:28.184144+0000","last_unstale":"2024-08-03T20:50:28.189255+0000","last_undegraded":"2024-08-03T20:50:28.189255+0000","last_fullsized":"2024-08-03T20:50:28.189255+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:50:27.143838+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:50:27.143838+0000","last_clean_scrub_stamp":"2024-08-03T20:50:27.143838+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T01:58:09.341916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.3","version":"0'0","reported_seq":13,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-08-03T20:50:28.163601+0000","last_change":"2024-08-03T20:50:27.167297+0000","last_active":"2024-08-03T20:50:28.163601+0000","last_peered":"2024-08-03T20:50:28.163601+0000","last_clean":"2024-08-03T20:50:28.163601+0000","last_became_active":"2024-08-03T20:50:27.166782+0000","last_became_peered":"2024-08-03T20:50:27.166782+0000","last_unstale":"2024-08-03T20:50:28.163601+0000","last_undegraded":"2024-08-03T20:50:28.163601+0000","last_fullsized":"2024-08-03T20:50:28.163601+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:50:26.140950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:50:26.140950+0000","last_clean_scrub_stamp":"2024-08-03T20:50:26.140950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T06:06:50.290727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.3","version":"0'0","reported_seq":10,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-08-03T20:50:28.183141+0000","last_change":"2024-08-03T20:50:28.177494+0000","last_active":"2024-08-03T20:50:28.183141+0000","last_peered":"2024-08-03T20:50:28.183141+0000","last_clean":"2024-08-03T20:50:28.183141+0000","last_became_active":"2024-08-03T20:50:28.177186+0000","last_became_peered":"2024-08-03T20:50:28.177186+0000","last_unstale":"2024-08-03T20:50:28.183141+0000","last_undegraded":"2024-08-03T20:50:28.183141+0000","last_fullsized":"2024-08-03T20:50:28.183141+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:50:27.143838+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:50:27.143838+0000","last_clean_scrub_stamp":"2024-08-03T20:50:27.143838+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T02:25:34.324726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.2","version":"0'0","reported_seq":13,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-08-03T20:50:28.165657+0000","last_change":"2024-08-03T20:50:27.180189+0000","last_active":"2024-08-03T20:50:28.165657+0000","last_peered":"2024-08-03T20:50:28.165657+0000","last_clean":"2024-08-03T20:50:28.165657+0000","last_became_active":"2024-08-03T20:50:27.178485+0000","last_became_peered":"2024-08-03T20:50:27.178485+0000","last_unstale":"2024-08-03T20:50:28.165657+0000","last_undegraded":"2024-08-03T20:50:28.165657+0000","last_fullsized":"2024-08-03T20:50:28.165657+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:50:26.140950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:50:26.140950+0000","last_clean_scrub_stamp":"2024-08-03T20:50:26.140950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T01:50:59.525120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.0","version":"0'0","reported_seq":10,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-08-03T20:50:28.180478+0000","last_change":"2024-08-03T20:50:28.176927+0000","last_active":"2024-08-03T20:50:28.180478+0000","last_peered":"2024-08-03T20:50:28.180478+0000","last_clean":"2024-08-03T20:50:28.180478+0000","last_became_active":"2024-08-03T20:50:28.176606+0000","last_became_peered":"2024-08-03T20:50:28.176606+0000","last_unstale":"2024-08-03T20:50:28.180478+0000","last_undegraded":"2024-08-03T20:50:28.180478+0000","last_fullsized":"2024-08-03T20:50:28.180478+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:50:27.143838+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:50:27.143838+0000","last_clean_scrub_stamp":"2024-08-03T20:50:27.143838+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T01:03:58.561921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.1","version":"0'0","reported_seq":13,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-08-03T20:50:28.161679+0000","last_change":"2024-08-03T20:50:27.181804+0000","last_active":"2024-08-03T20:50:28.161679+0000","last_peered":"2024-08-03T20:50:28.161679+0000","last_clean":"2024-08-03T20:50:28.161679+0000","last_became_active":"2024-08-03T20:50:27.179785+0000","last_became_peered":"2024-08-03T20:50:27.179785+0000","last_unstale":"2024-08-03T20:50:28.161679+0000","last_undegraded":"2024-08-03T20:50:28.161679+0000","last_fullsized":"2024-08-03T20:50:28.161679+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:50:26.140950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:50:26.140950+0000","last_clean_scrub_stamp":"2024-08-03T20:50:26.140950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T06:35:28.032286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.1","version":"0'0","reported_seq":10,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-08-03T20:50:28.187460+0000","last_change":"2024-08-03T20:50:28.178706+0000","last_active":"2024-08-03T20:50:28.187460+0000","last_peered":"2024-08-03T20:50:28.187460+0000","last_clean":"2024-08-03T20:50:28.187460+0000","last_became_active":"2024-08-03T20:50:28.177015+0000","last_became_peered":"2024-08-03T20:50:28.177015+0000","last_unstale":"2024-08-03T20:50:28.187460+0000","last_undegraded":"2024-08-03T20:50:28.187460+0000","last_fullsized":"2024-08-03T20:50:28.187460+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:50:27.143838+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:50:27.143838+0000","last_clean_scrub_stamp":"2024-08-03T20:50:27.143838+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T02:31:57.413818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.0","version":"0'0","reported_seq":13,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-08-03T20:50:28.173820+0000","last_change":"2024-08-03T20:50:27.171865+0000","last_active":"2024-08-03T20:50:28.173820+0000","last_peered":"2024-08-03T20:50:28.173820+0000","last_clean":"2024-08-03T20:50:28.173820+0000","last_became_active":"2024-08-03T20:50:27.170850+0000","last_became_peered":"2024-08-03T20:50:27.170850+0000","last_unstale":"2024-08-03T20:50:28.173820+0000","last_undegraded":"2024-08-03T20:50:28.173820+0000","last_fullsized":"2024-08-03T20:50:28.173820+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:50:26.140950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:50:26.140950+0000","last_clean_scrub_stamp":"2024-08-03T20:50:26.140950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-04T23:29:42.990546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.8","version":"0'0","reported_seq":10,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-08-03T20:50:28.188928+0000","last_change":"2024-08-03T20:50:28.186122+0000","last_active":"2024-08-03T20:50:28.188928+0000","last_peered":"2024-08-03T20:50:28.188928+0000","last_clean":"2024-08-03T20:50:28.188928+0000","last_became_active":"2024-08-03T20:50:28.185821+0000","last_became_peered":"2024-08-03T20:50:28.185821+0000","last_unstale":"2024-08-03T20:50:28.188928+0000","last_undegraded":"2024-08-03T20:50:28.188928+0000","last_fullsized":"2024-08-03T20:50:28.188928+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:50:27.143838+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:50:27.143838+0000","last_clean_scrub_stamp":"2024-08-03T20:50:27.143838+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T07:20:34.133759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.9","version":"0'0","reported_seq":15,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-08-03T20:50:29.159153+0000","last_change":"2024-08-03T20:50:27.165639+0000","last_active":"2024-08-03T20:50:29.159153+0000","last_peered":"2024-08-03T20:50:29.159153+0000","last_clean":"2024-08-03T20:50:29.159153+0000","last_became_active":"2024-08-03T20:50:27.165179+0000","last_became_peered":"2024-08-03T20:50:27.165179+0000","last_unstale":"2024-08-03T20:50:29.159153+0000","last_undegraded":"2024-08-03T20:50:29.159153+0000","last_fullsized":"2024-08-03T20:50:29.159153+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:50:26.140950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:50:26.140950+0000","last_clean_scrub_stamp":"2024-08-03T20:50:26.140950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T06:06:50.290727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"16'72","reported_seq":173,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-08-03T20:50:28.157654+0000","last_change":"2024-08-03T20:47:30.547632+0000","last_active":"2024-08-03T20:50:28.157654+0000","last_peered":"2024-08-03T20:50:28.157654+0000","last_clean":"2024-08-03T20:50:28.157654+0000","last_became_active":"2024-08-03T20:47:30.546255+0000","last_became_peered":"2024-08-03T20:47:30.546255+0000","last_unstale":"2024-08-03T20:50:28.157654+0000","last_undegraded":"2024-08-03T20:50:28.157654+0000","last_fullsized":"2024-08-03T20:50:28.157654+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:47:28.874451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:47:28.874451+0000","last_clean_scrub_stamp":"2024-08-03T20:47:28.874451+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T00:19:40.548812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":9,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":8,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509479,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32420,"kb_used_data":1236,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339420,"statfs":{"total":96636764160,"available":96603566080,"internally_reserved":0,"allocated":1265664,"data_stored":864908,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":6,"up_from":15,"seq":64424509480,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31828,"kb_used_data":648,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340012,"statfs":{"total":96636764160,"available":96604172288,"internally_reserved":0,"allocated":663552,"data_stored":274380,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":5,"up_from":15,"seq":64424509480,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31836,"kb_used_data":656,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340004,"statfs":{"total":96636764160,"available":96604164096,"internally_reserved":0,"allocated":671744,"data_stored":275032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509479,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31832,"kb_used_data":652,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340008,"statfs":{"total":96636764160,"available":96604168192,"internally_reserved":0,"allocated":667648,"data_stored":274470,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,4,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509481,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32428,"kb_used_data":1244,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339412,"statfs":{"total":96636764160,"available":96603557888,"internally_reserved":0,"allocated":1273856,"data_stored":866942,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542184,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31844,"kb_used_data":660,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339996,"statfs":{"total":96636764160,"available":96604155904,"internally_reserved":0,"allocated":675840,"data_stored":275992,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":3,"up_from":14,"seq":60129542182,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31816,"kb_used_data":636,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340024,"statfs":{"total":96636764160,"available":96604184576,"internally_reserved":0,"allocated":651264,"data_stored":267605,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":2,"up_from":14,"seq":60129542183,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31816,"kb_used_data":636,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340024,"statfs":{"total":96636764160,"available":96604184576,"internally_reserved":0,"allocated":651264,"data_stored":267605,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2194,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-08-03T20:50:32.693 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-08-03T20:50:32.958 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:32.957+0000 7fda04000640 1 -- 172.21.15.32:0/2715656750 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd9e4004800 msgr2=0x7fd9e4004c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:50:32.959 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:32.957+0000 7fda04000640 1 --2- 172.21.15.32:0/2715656750 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd9e4004800 0x7fd9e4004c10 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7fd9e800c310 tx=0x7fd9e4005830 comp rx=0 tx=0).stop 2024-08-03T20:50:32.959 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:32.957+0000 7fda04000640 1 -- 172.21.15.32:0/2715656750 shutdown_connections 2024-08-03T20:50:32.959 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:32.957+0000 7fda04000640 1 --2- 172.21.15.32:0/2715656750 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd9e4004800 0x7fd9e4004c10 unknown :-1 s=CLOSED pgs=576 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:32.959 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:32.957+0000 7fda04000640 1 -- 172.21.15.32:0/2715656750 >> 172.21.15.32:0/2715656750 conn(0x7fd9fc0b1bd0 msgr2=0x7fd9fc0b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:50:32.959 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:32.957+0000 7fda04000640 1 -- 172.21.15.32:0/2715656750 shutdown_connections 2024-08-03T20:50:32.959 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:32.957+0000 7fda04000640 1 -- 172.21.15.32:0/2715656750 wait complete. 2024-08-03T20:50:32.960 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:32.960+0000 7fda04000640 1 Processor -- start 2024-08-03T20:50:32.960 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:32.960+0000 7fda04000640 1 -- start start 2024-08-03T20:50:32.963 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:32.961+0000 7fda04000640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fd9e4004800 0x7fd9fc0cc970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:50:32.963 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:32.961+0000 7fda04000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd9fc152ac0 0x7fd9fc14a930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:50:32.963 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:32.961+0000 7fda04000640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fd9fc0af9a0 0x7fd9fc0afdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:50:32.963 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:32.961+0000 7fda04000640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fd9fc0b3d30 con 0x7fd9fc152ac0 2024-08-03T20:50:32.963 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:32.961+0000 7fda04000640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7fd9fc0b3bb0 con 0x7fd9fc0af9a0 2024-08-03T20:50:32.963 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:32.961+0000 7fda04000640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7fd9fc0b3eb0 con 0x7fd9e4004800 2024-08-03T20:50:32.963 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:32.961+0000 7fda00c00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd9fc152ac0 0x7fd9fc14a930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:50:32.963 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:32.962+0000 7fda00c00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd9fc152ac0 0x7fd9fc14a930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:45110/0 (socket says 172.21.15.32:45110) 2024-08-03T20:50:32.963 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:32.962+0000 7fda00c00640 1 -- 172.21.15.32:0/2621620051 learned_addr learned my addr 172.21.15.32:0/2621620051 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:50:32.963 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:32.962+0000 7fda01600640 1 --2- 172.21.15.32:0/2621620051 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fd9e4004800 0x7fd9fc0cc970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:50:32.963 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:32.962+0000 7fda02000640 1 --2- 172.21.15.32:0/2621620051 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fd9fc0af9a0 0x7fd9fc0afdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:50:32.963 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:32.962+0000 7fda00c00640 1 -- 172.21.15.32:0/2621620051 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fd9e4004800 msgr2=0x7fd9fc0cc970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:50:32.964 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:32.962+0000 7fda00c00640 1 --2- 172.21.15.32:0/2621620051 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fd9e4004800 0x7fd9fc0cc970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:32.964 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:32.962+0000 7fda00c00640 1 -- 172.21.15.32:0/2621620051 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fd9fc0af9a0 msgr2=0x7fd9fc0afdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:50:32.964 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:32.962+0000 7fda00c00640 1 --2- 172.21.15.32:0/2621620051 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fd9fc0af9a0 0x7fd9fc0afdb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:32.964 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:32.962+0000 7fda00c00640 1 -- 172.21.15.32:0/2621620051 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd9fc0cd020 con 0x7fd9fc152ac0 2024-08-03T20:50:32.964 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:32.962+0000 7fda00c00640 1 --2- 172.21.15.32:0/2621620051 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd9fc152ac0 0x7fd9fc14a930 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7fd9e802b130 tx=0x7fd9e8001d80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:50:32.965 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:32.963+0000 7fd9f9600640 1 -- 172.21.15.32:0/2621620051 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9e8031040 con 0x7fd9fc152ac0 2024-08-03T20:50:32.965 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:32.963+0000 7fd9f9600640 1 -- 172.21.15.32:0/2621620051 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fd9e80035f0 con 0x7fd9fc152ac0 2024-08-03T20:50:32.965 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:32.963+0000 7fd9f9600640 1 -- 172.21.15.32:0/2621620051 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9e8003a20 con 0x7fd9fc152ac0 2024-08-03T20:50:32.965 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:32.963+0000 7fda04000640 1 -- 172.21.15.32:0/2621620051 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd9fc0b02f0 con 0x7fd9fc152ac0 2024-08-03T20:50:32.965 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:32.963+0000 7fda04000640 1 -- 172.21.15.32:0/2621620051 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd9fc0b0ed0 con 0x7fd9fc152ac0 2024-08-03T20:50:32.965 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:32.964+0000 7fda04000640 1 -- 172.21.15.32:0/2621620051 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd9fc162420 con 0x7fd9fc152ac0 2024-08-03T20:50:32.965 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:32.965+0000 7fd9f9600640 1 -- 172.21.15.32:0/2621620051 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7fd9e803e030 con 0x7fd9fc152ac0 2024-08-03T20:50:32.967 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:32.966+0000 7fd9f9600640 1 --2- 172.21.15.32:0/2621620051 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fd9dc077010 0x7fd9dc079520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:50:32.967 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:32.966+0000 7fd9f9600640 1 -- 172.21.15.32:0/2621620051 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(57..57 src has 1..57) ==== 5709+0+0 (secure 0 0 0) 0x7fd9e80b8410 con 0x7fd9fc152ac0 2024-08-03T20:50:32.967 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:32.967+0000 7fda01600640 1 --2- 172.21.15.32:0/2621620051 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fd9dc077010 0x7fd9dc079520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:50:32.967 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:32.967+0000 7fda01600640 1 --2- 172.21.15.32:0/2621620051 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fd9dc077010 0x7fd9dc079520 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7fd9f0006680 tx=0x7fd9f0006b70 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:50:32.971 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:32.971+0000 7fd9f9600640 1 -- 172.21.15.32:0/2621620051 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7fd9fc0b02f0 con 0x7fd9fc152ac0 2024-08-03T20:50:33.176 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:33.176+0000 7fda04000640 1 -- 172.21.15.32:0/2621620051 --> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fd9fc074a90 con 0x7fd9dc077010 2024-08-03T20:50:33.182 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:33.181+0000 7fd9f9600640 1 -- 172.21.15.32:0/2621620051 <== mgr.4106 v2:172.21.15.118:6800/4205950699 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+251962 (secure 0 0 0) 0x7fd9fc074a90 con 0x7fd9dc077010 2024-08-03T20:50:33.185 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-03T20:50:33.186 INFO:teuthology.orchestra.run.smithi032.stderr:dumped all 2024-08-03T20:50:33.188 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:33.185+0000 7fda04000640 1 -- 172.21.15.32:0/2621620051 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fd9dc077010 msgr2=0x7fd9dc079520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:50:33.188 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:33.185+0000 7fda04000640 1 --2- 172.21.15.32:0/2621620051 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fd9dc077010 0x7fd9dc079520 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7fd9f0006680 tx=0x7fd9f0006b70 comp rx=0 tx=0).stop 2024-08-03T20:50:33.188 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:33.185+0000 7fda04000640 1 -- 172.21.15.32:0/2621620051 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd9fc152ac0 msgr2=0x7fd9fc14a930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:50:33.188 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:33.185+0000 7fda04000640 1 --2- 172.21.15.32:0/2621620051 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd9fc152ac0 0x7fd9fc14a930 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7fd9e802b130 tx=0x7fd9e8001d80 comp rx=0 tx=0).stop 2024-08-03T20:50:33.188 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:33.185+0000 7fda04000640 1 -- 172.21.15.32:0/2621620051 shutdown_connections 2024-08-03T20:50:33.188 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:33.185+0000 7fda04000640 1 --2- 172.21.15.32:0/2621620051 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fd9dc077010 0x7fd9dc079520 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:33.189 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:33.185+0000 7fda04000640 1 --2- 172.21.15.32:0/2621620051 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fd9fc0af9a0 0x7fd9fc0afdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:33.189 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:33.185+0000 7fda04000640 1 --2- 172.21.15.32:0/2621620051 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd9fc152ac0 0x7fd9fc14a930 unknown :-1 s=CLOSED pgs=577 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:33.189 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:33.185+0000 7fda04000640 1 --2- 172.21.15.32:0/2621620051 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fd9e4004800 0x7fd9fc0cc970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:33.189 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:33.185+0000 7fda04000640 1 -- 172.21.15.32:0/2621620051 >> 172.21.15.32:0/2621620051 conn(0x7fd9fc0b1bd0 msgr2=0x7fd9fc15d430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:50:33.189 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:33.185+0000 7fda04000640 1 -- 172.21.15.32:0/2621620051 shutdown_connections 2024-08-03T20:50:33.189 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:33.186+0000 7fda04000640 1 -- 172.21.15.32:0/2621620051 wait complete. 2024-08-03T20:50:33.212 INFO:teuthology.orchestra.run.smithi032.stdout:{"pg_ready":true,"pg_map":{"version":142,"stamp":"2024-08-03T20:50:32.269157+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":145,"num_write_kb":1642,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":110,"ondisk_log_size":110,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":156,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":255820,"kb_used_data":6368,"kb_used_omap":12,"kb_used_meta":249331,"kb_avail":754718900,"statfs":{"total":773094113280,"available":772832153600,"internally_reserved":0,"allocated":6520832,"data_stored":3366934,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":255315505},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,4,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":29,"apply_latency_ms":29,"commit_latency_ns":29000000,"apply_latency_ns":29000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001644"},"pg_stats":[{"pgid":"9.3f","version":"0'0","reported_seq":10,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-08-03T20:50:28.199771+0000","last_change":"2024-08-03T20:50:28.185044+0000","last_active":"2024-08-03T20:50:28.199771+0000","last_peered":"2024-08-03T20:50:28.199771+0000","last_clean":"2024-08-03T20:50:28.199771+0000","last_became_active":"2024-08-03T20:50:28.183954+0000","last_became_peered":"2024-08-03T20:50:28.183954+0000","last_unstale":"2024-08-03T20:50:28.199771+0000","last_undegraded":"2024-08-03T20:50:28.199771+0000","last_fullsized":"2024-08-03T20:50:28.199771+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:50:27.143838+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:50:27.143838+0000","last_clean_scrub_stamp":"2024-08-03T20:50:27.143838+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-04T23:23:49.154260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.3e","version":"0'0","reported_seq":10,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-08-03T20:50:28.187753+0000","last_change":"2024-08-03T20:50:28.183719+0000","last_active":"2024-08-03T20:50:28.187753+0000","last_peered":"2024-08-03T20:50:28.187753+0000","last_clean":"2024-08-03T20:50:28.187753+0000","last_became_active":"2024-08-03T20:50:28.183445+0000","last_became_peered":"2024-08-03T20:50:28.183445+0000","last_unstale":"2024-08-03T20:50:28.187753+0000","last_undegraded":"2024-08-03T20:50:28.187753+0000","last_fullsized":"2024-08-03T20:50:28.187753+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:50:27.143838+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:50:27.143838+0000","last_clean_scrub_stamp":"2024-08-03T20:50:27.143838+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T00:58:25.712972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.3d","version":"0'0","reported_seq":10,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-08-03T20:50:28.176553+0000","last_change":"2024-08-03T20:50:28.169553+0000","last_active":"2024-08-03T20:50:28.176553+0000","last_peered":"2024-08-03T20:50:28.176553+0000","last_clean":"2024-08-03T20:50:28.176553+0000","last_became_active":"2024-08-03T20:50:28.169046+0000","last_became_peered":"2024-08-03T20:50:28.169046+0000","last_unstale":"2024-08-03T20:50:28.176553+0000","last_undegraded":"2024-08-03T20:50:28.176553+0000","last_fullsized":"2024-08-03T20:50:28.176553+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:50:27.143838+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:50:27.143838+0000","last_clean_scrub_stamp":"2024-08-03T20:50:27.143838+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T05:29:26.363244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.3c","version":"0'0","reported_seq":10,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-08-03T20:50:28.188157+0000","last_change":"2024-08-03T20:50:28.186439+0000","last_active":"2024-08-03T20:50:28.188157+0000","last_peered":"2024-08-03T20:50:28.188157+0000","last_clean":"2024-08-03T20:50:28.188157+0000","last_became_active":"2024-08-03T20:50:28.186099+0000","last_became_peered":"2024-08-03T20:50:28.186099+0000","last_unstale":"2024-08-03T20:50:28.188157+0000","last_undegraded":"2024-08-03T20:50:28.188157+0000","last_fullsized":"2024-08-03T20:50:28.188157+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:50:27.143838+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:50:27.143838+0000","last_clean_scrub_stamp":"2024-08-03T20:50:27.143838+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T06:39:08.774014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.3b","version":"0'0","reported_seq":10,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-08-03T20:50:28.187637+0000","last_change":"2024-08-03T20:50:28.178146+0000","last_active":"2024-08-03T20:50:28.187637+0000","last_peered":"2024-08-03T20:50:28.187637+0000","last_clean":"2024-08-03T20:50:28.187637+0000","last_became_active":"2024-08-03T20:50:28.177802+0000","last_became_peered":"2024-08-03T20:50:28.177802+0000","last_unstale":"2024-08-03T20:50:28.187637+0000","last_undegraded":"2024-08-03T20:50:28.187637+0000","last_fullsized":"2024-08-03T20:50:28.187637+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:50:27.143838+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:50:27.143838+0000","last_clean_scrub_stamp":"2024-08-03T20:50:27.143838+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T03:11:16.738801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.3a","version":"0'0","reported_seq":10,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-08-03T20:50:28.201309+0000","last_change":"2024-08-03T20:50:28.199934+0000","last_active":"2024-08-03T20:50:28.201309+0000","last_peered":"2024-08-03T20:50:28.201309+0000","last_clean":"2024-08-03T20:50:28.201309+0000","last_became_active":"2024-08-03T20:50:28.199581+0000","last_became_peered":"2024-08-03T20:50:28.199581+0000","last_unstale":"2024-08-03T20:50:28.201309+0000","last_undegraded":"2024-08-03T20:50:28.201309+0000","last_fullsized":"2024-08-03T20:50:28.201309+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:50:27.143838+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:50:27.143838+0000","last_clean_scrub_stamp":"2024-08-03T20:50:27.143838+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T06:51:23.660654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.39","version":"0'0","reported_seq":10,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-08-03T20:50:28.177802+0000","last_change":"2024-08-03T20:50:28.172144+0000","last_active":"2024-08-03T20:50:28.177802+0000","last_peered":"2024-08-03T20:50:28.177802+0000","last_clean":"2024-08-03T20:50:28.177802+0000","last_became_active":"2024-08-03T20:50:28.171813+0000","last_became_peered":"2024-08-03T20:50:28.171813+0000","last_unstale":"2024-08-03T20:50:28.177802+0000","last_undegraded":"2024-08-03T20:50:28.177802+0000","last_fullsized":"2024-08-03T20:50:28.177802+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:50:27.143838+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:50:27.143838+0000","last_clean_scrub_stamp":"2024-08-03T20:50:27.143838+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T04:39:52.662655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.38","version":"0'0","reported_seq":10,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-08-03T20:50:28.187592+0000","last_change":"2024-08-03T20:50:28.182772+0000","last_active":"2024-08-03T20:50:28.187592+0000","last_peered":"2024-08-03T20:50:28.187592+0000","last_clean":"2024-08-03T20:50:28.187592+0000","last_became_active":"2024-08-03T20:50:28.182438+0000","last_became_peered":"2024-08-03T20:50:28.182438+0000","last_unstale":"2024-08-03T20:50:28.187592+0000","last_undegraded":"2024-08-03T20:50:28.187592+0000","last_fullsized":"2024-08-03T20:50:28.187592+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:50:27.143838+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:50:27.143838+0000","last_clean_scrub_stamp":"2024-08-03T20:50:27.143838+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-04T21:28:32.662281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.37","version":"0'0","reported_seq":10,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-08-03T20:50:28.178509+0000","last_change":"2024-08-03T20:50:28.175194+0000","last_active":"2024-08-03T20:50:28.178509+0000","last_peered":"2024-08-03T20:50:28.178509+0000","last_clean":"2024-08-03T20:50:28.178509+0000","last_became_active":"2024-08-03T20:50:28.174743+0000","last_became_peered":"2024-08-03T20:50:28.174743+0000","last_unstale":"2024-08-03T20:50:28.178509+0000","last_undegraded":"2024-08-03T20:50:28.178509+0000","last_fullsized":"2024-08-03T20:50:28.178509+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:50:27.143838+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:50:27.143838+0000","last_clean_scrub_stamp":"2024-08-03T20:50:27.143838+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-04T21:55:13.089107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.36","version":"0'0","reported_seq":10,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-08-03T20:50:28.179536+0000","last_change":"2024-08-03T20:50:28.176847+0000","last_active":"2024-08-03T20:50:28.179536+0000","last_peered":"2024-08-03T20:50:28.179536+0000","last_clean":"2024-08-03T20:50:28.179536+0000","last_became_active":"2024-08-03T20:50:28.176513+0000","last_became_peered":"2024-08-03T20:50:28.176513+0000","last_unstale":"2024-08-03T20:50:28.179536+0000","last_undegraded":"2024-08-03T20:50:28.179536+0000","last_fullsized":"2024-08-03T20:50:28.179536+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:50:27.143838+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:50:27.143838+0000","last_clean_scrub_stamp":"2024-08-03T20:50:27.143838+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-04T21:21:41.546104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.35","version":"0'0","reported_seq":10,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-08-03T20:50:28.185292+0000","last_change":"2024-08-03T20:50:28.179703+0000","last_active":"2024-08-03T20:50:28.185292+0000","last_peered":"2024-08-03T20:50:28.185292+0000","last_clean":"2024-08-03T20:50:28.185292+0000","last_became_active":"2024-08-03T20:50:28.179351+0000","last_became_peered":"2024-08-03T20:50:28.179351+0000","last_unstale":"2024-08-03T20:50:28.185292+0000","last_undegraded":"2024-08-03T20:50:28.185292+0000","last_fullsized":"2024-08-03T20:50:28.185292+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:50:27.143838+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:50:27.143838+0000","last_clean_scrub_stamp":"2024-08-03T20:50:27.143838+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T05:29:30.552157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.34","version":"0'0","reported_seq":10,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-08-03T20:50:28.183335+0000","last_change":"2024-08-03T20:50:28.173786+0000","last_active":"2024-08-03T20:50:28.183335+0000","last_peered":"2024-08-03T20:50:28.183335+0000","last_clean":"2024-08-03T20:50:28.183335+0000","last_became_active":"2024-08-03T20:50:28.173395+0000","last_became_peered":"2024-08-03T20:50:28.173395+0000","last_unstale":"2024-08-03T20:50:28.183335+0000","last_undegraded":"2024-08-03T20:50:28.183335+0000","last_fullsized":"2024-08-03T20:50:28.183335+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:50:27.143838+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:50:27.143838+0000","last_clean_scrub_stamp":"2024-08-03T20:50:27.143838+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T05:29:26.363244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.33","version":"0'0","reported_seq":13,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-08-03T20:50:29.159433+0000","last_change":"2024-08-03T20:50:28.186309+0000","last_active":"2024-08-03T20:50:29.159433+0000","last_peered":"2024-08-03T20:50:29.159433+0000","last_clean":"2024-08-03T20:50:29.159433+0000","last_became_active":"2024-08-03T20:50:28.185957+0000","last_became_peered":"2024-08-03T20:50:28.185957+0000","last_unstale":"2024-08-03T20:50:29.159433+0000","last_undegraded":"2024-08-03T20:50:29.159433+0000","last_fullsized":"2024-08-03T20:50:29.159433+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:50:27.143838+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:50:27.143838+0000","last_clean_scrub_stamp":"2024-08-03T20:50:27.143838+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-04T21:54:55.043600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.32","version":"0'0","reported_seq":10,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-08-03T20:50:28.194607+0000","last_change":"2024-08-03T20:50:28.192505+0000","last_active":"2024-08-03T20:50:28.194607+0000","last_peered":"2024-08-03T20:50:28.194607+0000","last_clean":"2024-08-03T20:50:28.194607+0000","last_became_active":"2024-08-03T20:50:28.191454+0000","last_became_peered":"2024-08-03T20:50:28.191454+0000","last_unstale":"2024-08-03T20:50:28.194607+0000","last_undegraded":"2024-08-03T20:50:28.194607+0000","last_fullsized":"2024-08-03T20:50:28.194607+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:50:27.143838+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:50:27.143838+0000","last_clean_scrub_stamp":"2024-08-03T20:50:27.143838+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T07:21:23.983841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.31","version":"0'0","reported_seq":10,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-08-03T20:50:28.183868+0000","last_change":"2024-08-03T20:50:28.182696+0000","last_active":"2024-08-03T20:50:28.183868+0000","last_peered":"2024-08-03T20:50:28.183868+0000","last_clean":"2024-08-03T20:50:28.183868+0000","last_became_active":"2024-08-03T20:50:28.182359+0000","last_became_peered":"2024-08-03T20:50:28.182359+0000","last_unstale":"2024-08-03T20:50:28.183868+0000","last_undegraded":"2024-08-03T20:50:28.183868+0000","last_fullsized":"2024-08-03T20:50:28.183868+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:50:27.143838+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:50:27.143838+0000","last_clean_scrub_stamp":"2024-08-03T20:50:27.143838+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T04:16:02.684070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.30","version":"0'0","reported_seq":10,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-08-03T20:50:28.184465+0000","last_change":"2024-08-03T20:50:28.176177+0000","last_active":"2024-08-03T20:50:28.184465+0000","last_peered":"2024-08-03T20:50:28.184465+0000","last_clean":"2024-08-03T20:50:28.184465+0000","last_became_active":"2024-08-03T20:50:28.175558+0000","last_became_peered":"2024-08-03T20:50:28.175558+0000","last_unstale":"2024-08-03T20:50:28.184465+0000","last_undegraded":"2024-08-03T20:50:28.184465+0000","last_fullsized":"2024-08-03T20:50:28.184465+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:50:27.143838+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:50:27.143838+0000","last_clean_scrub_stamp":"2024-08-03T20:50:27.143838+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T06:45:18.189472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.2f","version":"0'0","reported_seq":10,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-08-03T20:50:28.187712+0000","last_change":"2024-08-03T20:50:28.182623+0000","last_active":"2024-08-03T20:50:28.187712+0000","last_peered":"2024-08-03T20:50:28.187712+0000","last_clean":"2024-08-03T20:50:28.187712+0000","last_became_active":"2024-08-03T20:50:28.182123+0000","last_became_peered":"2024-08-03T20:50:28.182123+0000","last_unstale":"2024-08-03T20:50:28.187712+0000","last_undegraded":"2024-08-03T20:50:28.187712+0000","last_fullsized":"2024-08-03T20:50:28.187712+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:50:27.143838+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:50:27.143838+0000","last_clean_scrub_stamp":"2024-08-03T20:50:27.143838+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T05:08:08.224310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.2e","version":"0'0","reported_seq":10,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-08-03T20:50:28.178896+0000","last_change":"2024-08-03T20:50:28.170884+0000","last_active":"2024-08-03T20:50:28.178896+0000","last_peered":"2024-08-03T20:50:28.178896+0000","last_clean":"2024-08-03T20:50:28.178896+0000","last_became_active":"2024-08-03T20:50:28.170550+0000","last_became_peered":"2024-08-03T20:50:28.170550+0000","last_unstale":"2024-08-03T20:50:28.178896+0000","last_undegraded":"2024-08-03T20:50:28.178896+0000","last_fullsized":"2024-08-03T20:50:28.178896+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:50:27.143838+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:50:27.143838+0000","last_clean_scrub_stamp":"2024-08-03T20:50:27.143838+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-04T22:38:57.045083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.2d","version":"0'0","reported_seq":10,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-08-03T20:50:28.190266+0000","last_change":"2024-08-03T20:50:28.183183+0000","last_active":"2024-08-03T20:50:28.190266+0000","last_peered":"2024-08-03T20:50:28.190266+0000","last_clean":"2024-08-03T20:50:28.190266+0000","last_became_active":"2024-08-03T20:50:28.182837+0000","last_became_peered":"2024-08-03T20:50:28.182837+0000","last_unstale":"2024-08-03T20:50:28.190266+0000","last_undegraded":"2024-08-03T20:50:28.190266+0000","last_fullsized":"2024-08-03T20:50:28.190266+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:50:27.143838+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:50:27.143838+0000","last_clean_scrub_stamp":"2024-08-03T20:50:27.143838+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-04T22:22:09.910171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.2c","version":"0'0","reported_seq":10,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-08-03T20:50:28.189885+0000","last_change":"2024-08-03T20:50:28.187374+0000","last_active":"2024-08-03T20:50:28.189885+0000","last_peered":"2024-08-03T20:50:28.189885+0000","last_clean":"2024-08-03T20:50:28.189885+0000","last_became_active":"2024-08-03T20:50:28.186190+0000","last_became_peered":"2024-08-03T20:50:28.186190+0000","last_unstale":"2024-08-03T20:50:28.189885+0000","last_undegraded":"2024-08-03T20:50:28.189885+0000","last_fullsized":"2024-08-03T20:50:28.189885+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:50:27.143838+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:50:27.143838+0000","last_clean_scrub_stamp":"2024-08-03T20:50:27.143838+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T05:37:03.068592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.2b","version":"0'0","reported_seq":10,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-08-03T20:50:28.182717+0000","last_change":"2024-08-03T20:50:28.175277+0000","last_active":"2024-08-03T20:50:28.182717+0000","last_peered":"2024-08-03T20:50:28.182717+0000","last_clean":"2024-08-03T20:50:28.182717+0000","last_became_active":"2024-08-03T20:50:28.172599+0000","last_became_peered":"2024-08-03T20:50:28.172599+0000","last_unstale":"2024-08-03T20:50:28.182717+0000","last_undegraded":"2024-08-03T20:50:28.182717+0000","last_fullsized":"2024-08-03T20:50:28.182717+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:50:27.143838+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:50:27.143838+0000","last_clean_scrub_stamp":"2024-08-03T20:50:27.143838+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T06:34:23.326860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.2a","version":"0'0","reported_seq":10,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-08-03T20:50:28.204428+0000","last_change":"2024-08-03T20:50:28.201340+0000","last_active":"2024-08-03T20:50:28.204428+0000","last_peered":"2024-08-03T20:50:28.204428+0000","last_clean":"2024-08-03T20:50:28.204428+0000","last_became_active":"2024-08-03T20:50:28.200126+0000","last_became_peered":"2024-08-03T20:50:28.200126+0000","last_unstale":"2024-08-03T20:50:28.204428+0000","last_undegraded":"2024-08-03T20:50:28.204428+0000","last_fullsized":"2024-08-03T20:50:28.204428+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:50:27.143838+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:50:27.143838+0000","last_clean_scrub_stamp":"2024-08-03T20:50:27.143838+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T00:04:32.467985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.29","version":"0'0","reported_seq":10,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-08-03T20:50:28.183171+0000","last_change":"2024-08-03T20:50:28.177428+0000","last_active":"2024-08-03T20:50:28.183171+0000","last_peered":"2024-08-03T20:50:28.183171+0000","last_clean":"2024-08-03T20:50:28.183171+0000","last_became_active":"2024-08-03T20:50:28.176981+0000","last_became_peered":"2024-08-03T20:50:28.176981+0000","last_unstale":"2024-08-03T20:50:28.183171+0000","last_undegraded":"2024-08-03T20:50:28.183171+0000","last_fullsized":"2024-08-03T20:50:28.183171+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:50:27.143838+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:50:27.143838+0000","last_clean_scrub_stamp":"2024-08-03T20:50:27.143838+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T03:50:47.476617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.28","version":"0'0","reported_seq":13,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-08-03T20:50:29.159809+0000","last_change":"2024-08-03T20:50:28.187029+0000","last_active":"2024-08-03T20:50:29.159809+0000","last_peered":"2024-08-03T20:50:29.159809+0000","last_clean":"2024-08-03T20:50:29.159809+0000","last_became_active":"2024-08-03T20:50:28.186671+0000","last_became_peered":"2024-08-03T20:50:28.186671+0000","last_unstale":"2024-08-03T20:50:29.159809+0000","last_undegraded":"2024-08-03T20:50:29.159809+0000","last_fullsized":"2024-08-03T20:50:29.159809+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:50:27.143838+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:50:27.143838+0000","last_clean_scrub_stamp":"2024-08-03T20:50:27.143838+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T04:40:28.587465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.27","version":"0'0","reported_seq":10,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-08-03T20:50:28.199799+0000","last_change":"2024-08-03T20:50:28.193139+0000","last_active":"2024-08-03T20:50:28.199799+0000","last_peered":"2024-08-03T20:50:28.199799+0000","last_clean":"2024-08-03T20:50:28.199799+0000","last_became_active":"2024-08-03T20:50:28.192458+0000","last_became_peered":"2024-08-03T20:50:28.192458+0000","last_unstale":"2024-08-03T20:50:28.199799+0000","last_undegraded":"2024-08-03T20:50:28.199799+0000","last_fullsized":"2024-08-03T20:50:28.199799+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:50:27.143838+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:50:27.143838+0000","last_clean_scrub_stamp":"2024-08-03T20:50:27.143838+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T00:13:43.102375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.26","version":"0'0","reported_seq":10,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-08-03T20:50:28.178100+0000","last_change":"2024-08-03T20:50:28.172136+0000","last_active":"2024-08-03T20:50:28.178100+0000","last_peered":"2024-08-03T20:50:28.178100+0000","last_clean":"2024-08-03T20:50:28.178100+0000","last_became_active":"2024-08-03T20:50:28.171742+0000","last_became_peered":"2024-08-03T20:50:28.171742+0000","last_unstale":"2024-08-03T20:50:28.178100+0000","last_undegraded":"2024-08-03T20:50:28.178100+0000","last_fullsized":"2024-08-03T20:50:28.178100+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:50:27.143838+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:50:27.143838+0000","last_clean_scrub_stamp":"2024-08-03T20:50:27.143838+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-04T22:43:40.157850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.25","version":"0'0","reported_seq":10,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-08-03T20:50:28.184302+0000","last_change":"2024-08-03T20:50:28.176849+0000","last_active":"2024-08-03T20:50:28.184302+0000","last_peered":"2024-08-03T20:50:28.184302+0000","last_clean":"2024-08-03T20:50:28.184302+0000","last_became_active":"2024-08-03T20:50:28.174658+0000","last_became_peered":"2024-08-03T20:50:28.174658+0000","last_unstale":"2024-08-03T20:50:28.184302+0000","last_undegraded":"2024-08-03T20:50:28.184302+0000","last_fullsized":"2024-08-03T20:50:28.184302+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:50:27.143838+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:50:27.143838+0000","last_clean_scrub_stamp":"2024-08-03T20:50:27.143838+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T06:39:08.774014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.24","version":"0'0","reported_seq":10,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-08-03T20:50:28.186767+0000","last_change":"2024-08-03T20:50:28.173281+0000","last_active":"2024-08-03T20:50:28.186767+0000","last_peered":"2024-08-03T20:50:28.186767+0000","last_clean":"2024-08-03T20:50:28.186767+0000","last_became_active":"2024-08-03T20:50:28.172968+0000","last_became_peered":"2024-08-03T20:50:28.172968+0000","last_unstale":"2024-08-03T20:50:28.186767+0000","last_undegraded":"2024-08-03T20:50:28.186767+0000","last_fullsized":"2024-08-03T20:50:28.186767+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:50:27.143838+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:50:27.143838+0000","last_clean_scrub_stamp":"2024-08-03T20:50:27.143838+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T01:03:50.425091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.23","version":"0'0","reported_seq":10,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-08-03T20:50:28.187507+0000","last_change":"2024-08-03T20:50:28.179835+0000","last_active":"2024-08-03T20:50:28.187507+0000","last_peered":"2024-08-03T20:50:28.187507+0000","last_clean":"2024-08-03T20:50:28.187507+0000","last_became_active":"2024-08-03T20:50:28.179537+0000","last_became_peered":"2024-08-03T20:50:28.179537+0000","last_unstale":"2024-08-03T20:50:28.187507+0000","last_undegraded":"2024-08-03T20:50:28.187507+0000","last_fullsized":"2024-08-03T20:50:28.187507+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:50:27.143838+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:50:27.143838+0000","last_clean_scrub_stamp":"2024-08-03T20:50:27.143838+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T05:53:26.931834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.22","version":"0'0","reported_seq":10,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-08-03T20:50:28.178875+0000","last_change":"2024-08-03T20:50:28.171372+0000","last_active":"2024-08-03T20:50:28.178875+0000","last_peered":"2024-08-03T20:50:28.178875+0000","last_clean":"2024-08-03T20:50:28.178875+0000","last_became_active":"2024-08-03T20:50:28.171028+0000","last_became_peered":"2024-08-03T20:50:28.171028+0000","last_unstale":"2024-08-03T20:50:28.178875+0000","last_undegraded":"2024-08-03T20:50:28.178875+0000","last_fullsized":"2024-08-03T20:50:28.178875+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:50:27.143838+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:50:27.143838+0000","last_clean_scrub_stamp":"2024-08-03T20:50:27.143838+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T03:35:47.096608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.21","version":"0'0","reported_seq":10,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-08-03T20:50:28.199922+0000","last_change":"2024-08-03T20:50:28.191047+0000","last_active":"2024-08-03T20:50:28.199922+0000","last_peered":"2024-08-03T20:50:28.199922+0000","last_clean":"2024-08-03T20:50:28.199922+0000","last_became_active":"2024-08-03T20:50:28.190769+0000","last_became_peered":"2024-08-03T20:50:28.190769+0000","last_unstale":"2024-08-03T20:50:28.199922+0000","last_undegraded":"2024-08-03T20:50:28.199922+0000","last_fullsized":"2024-08-03T20:50:28.199922+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:50:27.143838+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:50:27.143838+0000","last_clean_scrub_stamp":"2024-08-03T20:50:27.143838+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T04:59:57.334073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.20","version":"0'0","reported_seq":10,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-08-03T20:50:28.184444+0000","last_change":"2024-08-03T20:50:28.175873+0000","last_active":"2024-08-03T20:50:28.184444+0000","last_peered":"2024-08-03T20:50:28.184444+0000","last_clean":"2024-08-03T20:50:28.184444+0000","last_became_active":"2024-08-03T20:50:28.175018+0000","last_became_peered":"2024-08-03T20:50:28.175018+0000","last_unstale":"2024-08-03T20:50:28.184444+0000","last_undegraded":"2024-08-03T20:50:28.184444+0000","last_fullsized":"2024-08-03T20:50:28.184444+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:50:27.143838+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:50:27.143838+0000","last_clean_scrub_stamp":"2024-08-03T20:50:27.143838+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T03:35:47.096608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1e","version":"0'0","reported_seq":10,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-08-03T20:50:28.185189+0000","last_change":"2024-08-03T20:50:28.176616+0000","last_active":"2024-08-03T20:50:28.185189+0000","last_peered":"2024-08-03T20:50:28.185189+0000","last_clean":"2024-08-03T20:50:28.185189+0000","last_became_active":"2024-08-03T20:50:28.176274+0000","last_became_peered":"2024-08-03T20:50:28.176274+0000","last_unstale":"2024-08-03T20:50:28.185189+0000","last_undegraded":"2024-08-03T20:50:28.185189+0000","last_fullsized":"2024-08-03T20:50:28.185189+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:50:27.143838+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:50:27.143838+0000","last_clean_scrub_stamp":"2024-08-03T20:50:27.143838+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T06:34:23.326860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1f","version":"56'4","reported_seq":17,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-08-03T20:50:28.198928+0000","last_change":"2024-08-03T20:50:27.168467+0000","last_active":"2024-08-03T20:50:28.198928+0000","last_peered":"2024-08-03T20:50:28.198928+0000","last_clean":"2024-08-03T20:50:28.198928+0000","last_became_active":"2024-08-03T20:50:27.168114+0000","last_became_peered":"2024-08-03T20:50:27.168114+0000","last_unstale":"2024-08-03T20:50:28.198928+0000","last_undegraded":"2024-08-03T20:50:28.198928+0000","last_fullsized":"2024-08-03T20:50:28.198928+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:50:26.140950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:50:26.140950+0000","last_clean_scrub_stamp":"2024-08-03T20:50:26.140950+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T01:31:02.341088+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1f","version":"0'0","reported_seq":10,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-08-03T20:50:28.182602+0000","last_change":"2024-08-03T20:50:28.180222+0000","last_active":"2024-08-03T20:50:28.182602+0000","last_peered":"2024-08-03T20:50:28.182602+0000","last_clean":"2024-08-03T20:50:28.182602+0000","last_became_active":"2024-08-03T20:50:28.179359+0000","last_became_peered":"2024-08-03T20:50:28.179359+0000","last_unstale":"2024-08-03T20:50:28.182602+0000","last_undegraded":"2024-08-03T20:50:28.182602+0000","last_fullsized":"2024-08-03T20:50:28.182602+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:50:27.143838+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:50:27.143838+0000","last_clean_scrub_stamp":"2024-08-03T20:50:27.143838+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-04T23:10:22.893837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.1e","version":"56'1","reported_seq":14,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-08-03T20:50:28.198287+0000","last_change":"2024-08-03T20:50:27.176454+0000","last_active":"2024-08-03T20:50:28.198287+0000","last_peered":"2024-08-03T20:50:28.198287+0000","last_clean":"2024-08-03T20:50:28.198287+0000","last_became_active":"2024-08-03T20:50:27.176084+0000","last_became_peered":"2024-08-03T20:50:27.176084+0000","last_unstale":"2024-08-03T20:50:28.198287+0000","last_undegraded":"2024-08-03T20:50:28.198287+0000","last_fullsized":"2024-08-03T20:50:28.198287+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:50:26.140950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:50:26.140950+0000","last_clean_scrub_stamp":"2024-08-03T20:50:26.140950+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T02:29:07.610879+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.1c","version":"0'0","reported_seq":10,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-08-03T20:50:28.190213+0000","last_change":"2024-08-03T20:50:28.180444+0000","last_active":"2024-08-03T20:50:28.190213+0000","last_peered":"2024-08-03T20:50:28.190213+0000","last_clean":"2024-08-03T20:50:28.190213+0000","last_became_active":"2024-08-03T20:50:28.180126+0000","last_became_peered":"2024-08-03T20:50:28.180126+0000","last_unstale":"2024-08-03T20:50:28.190213+0000","last_undegraded":"2024-08-03T20:50:28.190213+0000","last_fullsized":"2024-08-03T20:50:28.190213+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:50:27.143838+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:50:27.143838+0000","last_clean_scrub_stamp":"2024-08-03T20:50:27.143838+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T03:50:47.476617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1d","version":"56'2","reported_seq":15,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-08-03T20:50:28.205773+0000","last_change":"2024-08-03T20:50:27.183728+0000","last_active":"2024-08-03T20:50:28.205773+0000","last_peered":"2024-08-03T20:50:28.205773+0000","last_clean":"2024-08-03T20:50:28.205773+0000","last_became_active":"2024-08-03T20:50:27.181749+0000","last_became_peered":"2024-08-03T20:50:27.181749+0000","last_unstale":"2024-08-03T20:50:28.205773+0000","last_undegraded":"2024-08-03T20:50:28.205773+0000","last_fullsized":"2024-08-03T20:50:28.205773+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:50:26.140950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:50:26.140950+0000","last_clean_scrub_stamp":"2024-08-03T20:50:26.140950+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-04T22:43:39.154962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1d","version":"0'0","reported_seq":10,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-08-03T20:50:28.184345+0000","last_change":"2024-08-03T20:50:28.175292+0000","last_active":"2024-08-03T20:50:28.184345+0000","last_peered":"2024-08-03T20:50:28.184345+0000","last_clean":"2024-08-03T20:50:28.184345+0000","last_became_active":"2024-08-03T20:50:28.174201+0000","last_became_peered":"2024-08-03T20:50:28.174201+0000","last_unstale":"2024-08-03T20:50:28.184345+0000","last_undegraded":"2024-08-03T20:50:28.184345+0000","last_fullsized":"2024-08-03T20:50:28.184345+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:50:27.143838+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:50:27.143838+0000","last_clean_scrub_stamp":"2024-08-03T20:50:27.143838+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-04T22:38:57.045083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1c","version":"56'2","reported_seq":15,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-08-03T20:50:28.198347+0000","last_change":"2024-08-03T20:50:27.171915+0000","last_active":"2024-08-03T20:50:28.198347+0000","last_peered":"2024-08-03T20:50:28.198347+0000","last_clean":"2024-08-03T20:50:28.198347+0000","last_became_active":"2024-08-03T20:50:27.171579+0000","last_became_peered":"2024-08-03T20:50:27.171579+0000","last_unstale":"2024-08-03T20:50:28.198347+0000","last_undegraded":"2024-08-03T20:50:28.198347+0000","last_fullsized":"2024-08-03T20:50:28.198347+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:50:26.140950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:50:26.140950+0000","last_clean_scrub_stamp":"2024-08-03T20:50:26.140950+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T08:31:43.595881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.1a","version":"0'0","reported_seq":10,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-08-03T20:50:28.204449+0000","last_change":"2024-08-03T20:50:28.201626+0000","last_active":"2024-08-03T20:50:28.204449+0000","last_peered":"2024-08-03T20:50:28.204449+0000","last_clean":"2024-08-03T20:50:28.204449+0000","last_became_active":"2024-08-03T20:50:28.200412+0000","last_became_peered":"2024-08-03T20:50:28.200412+0000","last_unstale":"2024-08-03T20:50:28.204449+0000","last_undegraded":"2024-08-03T20:50:28.204449+0000","last_fullsized":"2024-08-03T20:50:28.204449+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:50:27.143838+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:50:27.143838+0000","last_clean_scrub_stamp":"2024-08-03T20:50:27.143838+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T05:50:54.163139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.1b","version":"56'1","reported_seq":14,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-08-03T20:50:28.201003+0000","last_change":"2024-08-03T20:50:27.179699+0000","last_active":"2024-08-03T20:50:28.201003+0000","last_peered":"2024-08-03T20:50:28.201003+0000","last_clean":"2024-08-03T20:50:28.201003+0000","last_became_active":"2024-08-03T20:50:27.179343+0000","last_became_peered":"2024-08-03T20:50:27.179343+0000","last_unstale":"2024-08-03T20:50:28.201003+0000","last_undegraded":"2024-08-03T20:50:28.201003+0000","last_fullsized":"2024-08-03T20:50:28.201003+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:50:26.140950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:50:26.140950+0000","last_clean_scrub_stamp":"2024-08-03T20:50:26.140950+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T08:30:12.186681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.1b","version":"0'0","reported_seq":10,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-08-03T20:50:28.189035+0000","last_change":"2024-08-03T20:50:28.185354+0000","last_active":"2024-08-03T20:50:28.189035+0000","last_peered":"2024-08-03T20:50:28.189035+0000","last_clean":"2024-08-03T20:50:28.189035+0000","last_became_active":"2024-08-03T20:50:28.185048+0000","last_became_peered":"2024-08-03T20:50:28.185048+0000","last_unstale":"2024-08-03T20:50:28.189035+0000","last_undegraded":"2024-08-03T20:50:28.189035+0000","last_fullsized":"2024-08-03T20:50:28.189035+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:50:27.143838+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:50:27.143838+0000","last_clean_scrub_stamp":"2024-08-03T20:50:27.143838+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-04T22:17:31.483960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.1a","version":"56'1","reported_seq":14,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-08-03T20:50:28.194366+0000","last_change":"2024-08-03T20:50:27.170687+0000","last_active":"2024-08-03T20:50:28.194366+0000","last_peered":"2024-08-03T20:50:28.194366+0000","last_clean":"2024-08-03T20:50:28.194366+0000","last_became_active":"2024-08-03T20:50:27.170379+0000","last_became_peered":"2024-08-03T20:50:27.170379+0000","last_unstale":"2024-08-03T20:50:28.194366+0000","last_undegraded":"2024-08-03T20:50:28.194366+0000","last_fullsized":"2024-08-03T20:50:28.194366+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:50:26.140950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:50:26.140950+0000","last_clean_scrub_stamp":"2024-08-03T20:50:26.140950+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-04T23:32:46.677480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.18","version":"0'0","reported_seq":13,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-08-03T20:50:29.159231+0000","last_change":"2024-08-03T20:50:28.180017+0000","last_active":"2024-08-03T20:50:29.159231+0000","last_peered":"2024-08-03T20:50:29.159231+0000","last_clean":"2024-08-03T20:50:29.159231+0000","last_became_active":"2024-08-03T20:50:28.179756+0000","last_became_peered":"2024-08-03T20:50:28.179756+0000","last_unstale":"2024-08-03T20:50:29.159231+0000","last_undegraded":"2024-08-03T20:50:29.159231+0000","last_fullsized":"2024-08-03T20:50:29.159231+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:50:27.143838+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:50:27.143838+0000","last_clean_scrub_stamp":"2024-08-03T20:50:27.143838+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-04T23:10:22.893837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.19","version":"0'0","reported_seq":13,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-08-03T20:50:28.165588+0000","last_change":"2024-08-03T20:50:27.178443+0000","last_active":"2024-08-03T20:50:28.165588+0000","last_peered":"2024-08-03T20:50:28.165588+0000","last_clean":"2024-08-03T20:50:28.165588+0000","last_became_active":"2024-08-03T20:50:27.178092+0000","last_became_peered":"2024-08-03T20:50:27.178092+0000","last_unstale":"2024-08-03T20:50:28.165588+0000","last_undegraded":"2024-08-03T20:50:28.165588+0000","last_fullsized":"2024-08-03T20:50:28.165588+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:50:26.140950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:50:26.140950+0000","last_clean_scrub_stamp":"2024-08-03T20:50:26.140950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T01:31:02.341088+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.19","version":"0'0","reported_seq":10,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-08-03T20:50:28.187540+0000","last_change":"2024-08-03T20:50:28.182908+0000","last_active":"2024-08-03T20:50:28.187540+0000","last_peered":"2024-08-03T20:50:28.187540+0000","last_clean":"2024-08-03T20:50:28.187540+0000","last_became_active":"2024-08-03T20:50:28.182547+0000","last_became_peered":"2024-08-03T20:50:28.182547+0000","last_unstale":"2024-08-03T20:50:28.187540+0000","last_undegraded":"2024-08-03T20:50:28.187540+0000","last_fullsized":"2024-08-03T20:50:28.187540+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:50:27.143838+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:50:27.143838+0000","last_clean_scrub_stamp":"2024-08-03T20:50:27.143838+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-04T20:54:07.705607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.18","version":"0'0","reported_seq":13,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-08-03T20:50:28.165357+0000","last_change":"2024-08-03T20:50:27.178185+0000","last_active":"2024-08-03T20:50:28.165357+0000","last_peered":"2024-08-03T20:50:28.165357+0000","last_clean":"2024-08-03T20:50:28.165357+0000","last_became_active":"2024-08-03T20:50:27.177859+0000","last_became_peered":"2024-08-03T20:50:27.177859+0000","last_unstale":"2024-08-03T20:50:28.165357+0000","last_undegraded":"2024-08-03T20:50:28.165357+0000","last_fullsized":"2024-08-03T20:50:28.165357+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:50:26.140950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:50:26.140950+0000","last_clean_scrub_stamp":"2024-08-03T20:50:26.140950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T00:24:04.506049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.16","version":"0'0","reported_seq":10,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-08-03T20:50:28.183797+0000","last_change":"2024-08-03T20:50:28.182599+0000","last_active":"2024-08-03T20:50:28.183797+0000","last_peered":"2024-08-03T20:50:28.183797+0000","last_clean":"2024-08-03T20:50:28.183797+0000","last_became_active":"2024-08-03T20:50:28.182066+0000","last_became_peered":"2024-08-03T20:50:28.182066+0000","last_unstale":"2024-08-03T20:50:28.183797+0000","last_undegraded":"2024-08-03T20:50:28.183797+0000","last_fullsized":"2024-08-03T20:50:28.183797+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:50:27.143838+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:50:27.143838+0000","last_clean_scrub_stamp":"2024-08-03T20:50:27.143838+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-04T20:53:15.943888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.17","version":"0'0","reported_seq":13,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-08-03T20:50:28.158965+0000","last_change":"2024-08-03T20:50:27.185990+0000","last_active":"2024-08-03T20:50:28.158965+0000","last_peered":"2024-08-03T20:50:28.158965+0000","last_clean":"2024-08-03T20:50:28.158965+0000","last_became_active":"2024-08-03T20:50:27.185658+0000","last_became_peered":"2024-08-03T20:50:27.185658+0000","last_unstale":"2024-08-03T20:50:28.158965+0000","last_undegraded":"2024-08-03T20:50:28.158965+0000","last_fullsized":"2024-08-03T20:50:28.158965+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:50:26.140950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:50:26.140950+0000","last_clean_scrub_stamp":"2024-08-03T20:50:26.140950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-04T23:15:19.049204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.17","version":"0'0","reported_seq":10,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-08-03T20:50:28.184085+0000","last_change":"2024-08-03T20:50:28.181105+0000","last_active":"2024-08-03T20:50:28.184085+0000","last_peered":"2024-08-03T20:50:28.184085+0000","last_clean":"2024-08-03T20:50:28.184085+0000","last_became_active":"2024-08-03T20:50:28.180745+0000","last_became_peered":"2024-08-03T20:50:28.180745+0000","last_unstale":"2024-08-03T20:50:28.184085+0000","last_undegraded":"2024-08-03T20:50:28.184085+0000","last_fullsized":"2024-08-03T20:50:28.184085+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:50:27.143838+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:50:27.143838+0000","last_clean_scrub_stamp":"2024-08-03T20:50:27.143838+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T02:56:48.726993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.16","version":"0'0","reported_seq":13,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-08-03T20:50:28.174724+0000","last_change":"2024-08-03T20:50:27.180129+0000","last_active":"2024-08-03T20:50:28.174724+0000","last_peered":"2024-08-03T20:50:28.174724+0000","last_clean":"2024-08-03T20:50:28.174724+0000","last_became_active":"2024-08-03T20:50:27.179815+0000","last_became_peered":"2024-08-03T20:50:27.179815+0000","last_unstale":"2024-08-03T20:50:28.174724+0000","last_undegraded":"2024-08-03T20:50:28.174724+0000","last_fullsized":"2024-08-03T20:50:28.174724+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:50:26.140950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:50:26.140950+0000","last_clean_scrub_stamp":"2024-08-03T20:50:26.140950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-04T22:27:50.388714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.14","version":"0'0","reported_seq":10,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-08-03T20:50:28.202040+0000","last_change":"2024-08-03T20:50:28.192110+0000","last_active":"2024-08-03T20:50:28.202040+0000","last_peered":"2024-08-03T20:50:28.202040+0000","last_clean":"2024-08-03T20:50:28.202040+0000","last_became_active":"2024-08-03T20:50:28.183686+0000","last_became_peered":"2024-08-03T20:50:28.183686+0000","last_unstale":"2024-08-03T20:50:28.202040+0000","last_undegraded":"2024-08-03T20:50:28.202040+0000","last_fullsized":"2024-08-03T20:50:28.202040+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:50:27.143838+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:50:27.143838+0000","last_clean_scrub_stamp":"2024-08-03T20:50:27.143838+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-04T21:54:55.043600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.15","version":"56'3","reported_seq":16,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-08-03T20:50:28.205573+0000","last_change":"2024-08-03T20:50:27.176805+0000","last_active":"2024-08-03T20:50:28.205573+0000","last_peered":"2024-08-03T20:50:28.205573+0000","last_clean":"2024-08-03T20:50:28.205573+0000","last_became_active":"2024-08-03T20:50:27.176513+0000","last_became_peered":"2024-08-03T20:50:27.176513+0000","last_unstale":"2024-08-03T20:50:28.205573+0000","last_undegraded":"2024-08-03T20:50:28.205573+0000","last_fullsized":"2024-08-03T20:50:28.205573+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:50:26.140950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:50:26.140950+0000","last_clean_scrub_stamp":"2024-08-03T20:50:26.140950+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-04T22:43:39.154962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.15","version":"0'0","reported_seq":10,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-08-03T20:50:28.183971+0000","last_change":"2024-08-03T20:50:28.180318+0000","last_active":"2024-08-03T20:50:28.183971+0000","last_peered":"2024-08-03T20:50:28.183971+0000","last_clean":"2024-08-03T20:50:28.183971+0000","last_became_active":"2024-08-03T20:50:28.179795+0000","last_became_peered":"2024-08-03T20:50:28.179795+0000","last_unstale":"2024-08-03T20:50:28.183971+0000","last_undegraded":"2024-08-03T20:50:28.183971+0000","last_fullsized":"2024-08-03T20:50:28.183971+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:50:27.143838+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:50:27.143838+0000","last_clean_scrub_stamp":"2024-08-03T20:50:27.143838+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-04T22:40:26.512652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.14","version":"56'6","reported_seq":19,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-08-03T20:50:28.206722+0000","last_change":"2024-08-03T20:50:27.181537+0000","last_active":"2024-08-03T20:50:28.206722+0000","last_peered":"2024-08-03T20:50:28.206722+0000","last_clean":"2024-08-03T20:50:28.206722+0000","last_became_active":"2024-08-03T20:50:27.181214+0000","last_became_peered":"2024-08-03T20:50:27.181214+0000","last_unstale":"2024-08-03T20:50:28.206722+0000","last_undegraded":"2024-08-03T20:50:28.206722+0000","last_fullsized":"2024-08-03T20:50:28.206722+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:50:26.140950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:50:26.140950+0000","last_clean_scrub_stamp":"2024-08-03T20:50:26.140950+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T00:30:52.441334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.12","version":"0'0","reported_seq":10,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-08-03T20:50:28.188204+0000","last_change":"2024-08-03T20:50:28.186741+0000","last_active":"2024-08-03T20:50:28.188204+0000","last_peered":"2024-08-03T20:50:28.188204+0000","last_clean":"2024-08-03T20:50:28.188204+0000","last_became_active":"2024-08-03T20:50:28.185916+0000","last_became_peered":"2024-08-03T20:50:28.185916+0000","last_unstale":"2024-08-03T20:50:28.188204+0000","last_undegraded":"2024-08-03T20:50:28.188204+0000","last_fullsized":"2024-08-03T20:50:28.188204+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:50:27.143838+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:50:27.143838+0000","last_clean_scrub_stamp":"2024-08-03T20:50:27.143838+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T04:30:20.080808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.13","version":"56'2","reported_seq":15,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-08-03T20:50:28.205115+0000","last_change":"2024-08-03T20:50:27.180288+0000","last_active":"2024-08-03T20:50:28.205115+0000","last_peered":"2024-08-03T20:50:28.205115+0000","last_clean":"2024-08-03T20:50:28.205115+0000","last_became_active":"2024-08-03T20:50:27.179946+0000","last_became_peered":"2024-08-03T20:50:27.179946+0000","last_unstale":"2024-08-03T20:50:28.205115+0000","last_undegraded":"2024-08-03T20:50:28.205115+0000","last_fullsized":"2024-08-03T20:50:28.205115+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:50:26.140950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:50:26.140950+0000","last_clean_scrub_stamp":"2024-08-03T20:50:26.140950+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T06:39:42.092083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.13","version":"0'0","reported_seq":10,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-08-03T20:50:28.183118+0000","last_change":"2024-08-03T20:50:28.176762+0000","last_active":"2024-08-03T20:50:28.183118+0000","last_peered":"2024-08-03T20:50:28.183118+0000","last_clean":"2024-08-03T20:50:28.183118+0000","last_became_active":"2024-08-03T20:50:28.176375+0000","last_became_peered":"2024-08-03T20:50:28.176375+0000","last_unstale":"2024-08-03T20:50:28.183118+0000","last_undegraded":"2024-08-03T20:50:28.183118+0000","last_fullsized":"2024-08-03T20:50:28.183118+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:50:27.143838+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:50:27.143838+0000","last_clean_scrub_stamp":"2024-08-03T20:50:27.143838+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T04:30:20.080808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.12","version":"0'0","reported_seq":15,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-08-03T20:50:29.159389+0000","last_change":"2024-08-03T20:50:27.169706+0000","last_active":"2024-08-03T20:50:29.159389+0000","last_peered":"2024-08-03T20:50:29.159389+0000","last_clean":"2024-08-03T20:50:29.159389+0000","last_became_active":"2024-08-03T20:50:27.169312+0000","last_became_peered":"2024-08-03T20:50:27.169312+0000","last_unstale":"2024-08-03T20:50:29.159389+0000","last_undegraded":"2024-08-03T20:50:29.159389+0000","last_fullsized":"2024-08-03T20:50:29.159389+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:50:26.140950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:50:26.140950+0000","last_clean_scrub_stamp":"2024-08-03T20:50:26.140950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T04:36:51.691682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.10","version":"0'0","reported_seq":10,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-08-03T20:50:28.181146+0000","last_change":"2024-08-03T20:50:28.178880+0000","last_active":"2024-08-03T20:50:28.181146+0000","last_peered":"2024-08-03T20:50:28.181146+0000","last_clean":"2024-08-03T20:50:28.181146+0000","last_became_active":"2024-08-03T20:50:28.171479+0000","last_became_peered":"2024-08-03T20:50:28.171479+0000","last_unstale":"2024-08-03T20:50:28.181146+0000","last_undegraded":"2024-08-03T20:50:28.181146+0000","last_fullsized":"2024-08-03T20:50:28.181146+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:50:27.143838+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:50:27.143838+0000","last_clean_scrub_stamp":"2024-08-03T20:50:27.143838+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T05:49:41.217508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.11","version":"0'0","reported_seq":13,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-08-03T20:50:28.167750+0000","last_change":"2024-08-03T20:50:27.168903+0000","last_active":"2024-08-03T20:50:28.167750+0000","last_peered":"2024-08-03T20:50:28.167750+0000","last_clean":"2024-08-03T20:50:28.167750+0000","last_became_active":"2024-08-03T20:50:27.167849+0000","last_became_peered":"2024-08-03T20:50:27.167849+0000","last_unstale":"2024-08-03T20:50:28.167750+0000","last_undegraded":"2024-08-03T20:50:28.167750+0000","last_fullsized":"2024-08-03T20:50:28.167750+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:50:26.140950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:50:26.140950+0000","last_clean_scrub_stamp":"2024-08-03T20:50:26.140950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T05:54:06.179784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.11","version":"0'0","reported_seq":10,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-08-03T20:50:28.182699+0000","last_change":"2024-08-03T20:50:28.176478+0000","last_active":"2024-08-03T20:50:28.182699+0000","last_peered":"2024-08-03T20:50:28.182699+0000","last_clean":"2024-08-03T20:50:28.182699+0000","last_became_active":"2024-08-03T20:50:28.176146+0000","last_became_peered":"2024-08-03T20:50:28.176146+0000","last_unstale":"2024-08-03T20:50:28.182699+0000","last_undegraded":"2024-08-03T20:50:28.182699+0000","last_fullsized":"2024-08-03T20:50:28.182699+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:50:27.143838+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:50:27.143838+0000","last_clean_scrub_stamp":"2024-08-03T20:50:27.143838+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T06:39:08.774014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.10","version":"56'8","reported_seq":21,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-08-03T20:50:28.196649+0000","last_change":"2024-08-03T20:50:27.173089+0000","last_active":"2024-08-03T20:50:28.196649+0000","last_peered":"2024-08-03T20:50:28.196649+0000","last_clean":"2024-08-03T20:50:28.196649+0000","last_became_active":"2024-08-03T20:50:27.172165+0000","last_became_peered":"2024-08-03T20:50:27.172165+0000","last_unstale":"2024-08-03T20:50:28.196649+0000","last_undegraded":"2024-08-03T20:50:28.196649+0000","last_fullsized":"2024-08-03T20:50:28.196649+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:50:26.140950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:50:26.140950+0000","last_clean_scrub_stamp":"2024-08-03T20:50:26.140950+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-04T23:32:46.677480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.e","version":"0'0","reported_seq":10,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-08-03T20:50:28.182041+0000","last_change":"2024-08-03T20:50:28.172981+0000","last_active":"2024-08-03T20:50:28.182041+0000","last_peered":"2024-08-03T20:50:28.182041+0000","last_clean":"2024-08-03T20:50:28.182041+0000","last_became_active":"2024-08-03T20:50:28.172642+0000","last_became_peered":"2024-08-03T20:50:28.172642+0000","last_unstale":"2024-08-03T20:50:28.182041+0000","last_undegraded":"2024-08-03T20:50:28.182041+0000","last_fullsized":"2024-08-03T20:50:28.182041+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:50:27.143838+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:50:27.143838+0000","last_clean_scrub_stamp":"2024-08-03T20:50:27.143838+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T06:45:18.189472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.f","version":"56'1","reported_seq":14,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-08-03T20:50:28.205205+0000","last_change":"2024-08-03T20:50:27.182958+0000","last_active":"2024-08-03T20:50:28.205205+0000","last_peered":"2024-08-03T20:50:28.205205+0000","last_clean":"2024-08-03T20:50:28.205205+0000","last_became_active":"2024-08-03T20:50:27.181672+0000","last_became_peered":"2024-08-03T20:50:27.181672+0000","last_unstale":"2024-08-03T20:50:28.205205+0000","last_undegraded":"2024-08-03T20:50:28.205205+0000","last_fullsized":"2024-08-03T20:50:28.205205+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:50:26.140950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:50:26.140950+0000","last_clean_scrub_stamp":"2024-08-03T20:50:26.140950+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T08:28:57.064216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.f","version":"0'0","reported_seq":10,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-08-03T20:50:28.187584+0000","last_change":"2024-08-03T20:50:28.185915+0000","last_active":"2024-08-03T20:50:28.187584+0000","last_peered":"2024-08-03T20:50:28.187584+0000","last_clean":"2024-08-03T20:50:28.187584+0000","last_became_active":"2024-08-03T20:50:28.185636+0000","last_became_peered":"2024-08-03T20:50:28.185636+0000","last_unstale":"2024-08-03T20:50:28.187584+0000","last_undegraded":"2024-08-03T20:50:28.187584+0000","last_fullsized":"2024-08-03T20:50:28.187584+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:50:27.143838+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:50:27.143838+0000","last_clean_scrub_stamp":"2024-08-03T20:50:27.143838+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T06:45:18.189472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.e","version":"56'2","reported_seq":15,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-08-03T20:50:28.201380+0000","last_change":"2024-08-03T20:50:27.185691+0000","last_active":"2024-08-03T20:50:28.201380+0000","last_peered":"2024-08-03T20:50:28.201380+0000","last_clean":"2024-08-03T20:50:28.201380+0000","last_became_active":"2024-08-03T20:50:27.185368+0000","last_became_peered":"2024-08-03T20:50:27.185368+0000","last_unstale":"2024-08-03T20:50:28.201380+0000","last_undegraded":"2024-08-03T20:50:28.201380+0000","last_fullsized":"2024-08-03T20:50:28.201380+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:50:26.140950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:50:26.140950+0000","last_clean_scrub_stamp":"2024-08-03T20:50:26.140950+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T08:31:43.595881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.c","version":"0'0","reported_seq":10,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-08-03T20:50:28.188101+0000","last_change":"2024-08-03T20:50:28.181270+0000","last_active":"2024-08-03T20:50:28.188101+0000","last_peered":"2024-08-03T20:50:28.188101+0000","last_clean":"2024-08-03T20:50:28.188101+0000","last_became_active":"2024-08-03T20:50:28.180955+0000","last_became_peered":"2024-08-03T20:50:28.180955+0000","last_unstale":"2024-08-03T20:50:28.188101+0000","last_undegraded":"2024-08-03T20:50:28.188101+0000","last_fullsized":"2024-08-03T20:50:28.188101+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:50:27.143838+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:50:27.143838+0000","last_clean_scrub_stamp":"2024-08-03T20:50:27.143838+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T06:59:54.907087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.d","version":"56'1","reported_seq":14,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-08-03T20:50:28.198354+0000","last_change":"2024-08-03T20:50:27.169583+0000","last_active":"2024-08-03T20:50:28.198354+0000","last_peered":"2024-08-03T20:50:28.198354+0000","last_clean":"2024-08-03T20:50:28.198354+0000","last_became_active":"2024-08-03T20:50:27.168862+0000","last_became_peered":"2024-08-03T20:50:27.168862+0000","last_unstale":"2024-08-03T20:50:28.198354+0000","last_undegraded":"2024-08-03T20:50:28.198354+0000","last_fullsized":"2024-08-03T20:50:28.198354+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:50:26.140950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:50:26.140950+0000","last_clean_scrub_stamp":"2024-08-03T20:50:26.140950+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-04T23:15:19.049204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.d","version":"0'0","reported_seq":10,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-08-03T20:50:28.183845+0000","last_change":"2024-08-03T20:50:28.180087+0000","last_active":"2024-08-03T20:50:28.183845+0000","last_peered":"2024-08-03T20:50:28.183845+0000","last_clean":"2024-08-03T20:50:28.183845+0000","last_became_active":"2024-08-03T20:50:28.179373+0000","last_became_peered":"2024-08-03T20:50:28.179373+0000","last_unstale":"2024-08-03T20:50:28.183845+0000","last_undegraded":"2024-08-03T20:50:28.183845+0000","last_fullsized":"2024-08-03T20:50:28.183845+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:50:27.143838+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:50:27.143838+0000","last_clean_scrub_stamp":"2024-08-03T20:50:27.143838+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T06:51:23.660654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.c","version":"0'0","reported_seq":13,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-08-03T20:50:28.159661+0000","last_change":"2024-08-03T20:50:27.178147+0000","last_active":"2024-08-03T20:50:28.159661+0000","last_peered":"2024-08-03T20:50:28.159661+0000","last_clean":"2024-08-03T20:50:28.159661+0000","last_became_active":"2024-08-03T20:50:27.177746+0000","last_became_peered":"2024-08-03T20:50:27.177746+0000","last_unstale":"2024-08-03T20:50:28.159661+0000","last_undegraded":"2024-08-03T20:50:28.159661+0000","last_fullsized":"2024-08-03T20:50:28.159661+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:50:26.140950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:50:26.140950+0000","last_clean_scrub_stamp":"2024-08-03T20:50:26.140950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T05:54:06.179784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.a","version":"0'0","reported_seq":10,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-08-03T20:50:28.188187+0000","last_change":"2024-08-03T20:50:28.186221+0000","last_active":"2024-08-03T20:50:28.188187+0000","last_peered":"2024-08-03T20:50:28.188187+0000","last_clean":"2024-08-03T20:50:28.188187+0000","last_became_active":"2024-08-03T20:50:28.185555+0000","last_became_peered":"2024-08-03T20:50:28.185555+0000","last_unstale":"2024-08-03T20:50:28.188187+0000","last_undegraded":"2024-08-03T20:50:28.188187+0000","last_fullsized":"2024-08-03T20:50:28.188187+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:50:27.143838+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:50:27.143838+0000","last_clean_scrub_stamp":"2024-08-03T20:50:27.143838+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T06:34:23.326860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.b","version":"56'2","reported_seq":15,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-08-03T20:50:28.203426+0000","last_change":"2024-08-03T20:50:27.177434+0000","last_active":"2024-08-03T20:50:28.203426+0000","last_peered":"2024-08-03T20:50:28.203426+0000","last_clean":"2024-08-03T20:50:28.203426+0000","last_became_active":"2024-08-03T20:50:27.177093+0000","last_became_peered":"2024-08-03T20:50:27.177093+0000","last_unstale":"2024-08-03T20:50:28.203426+0000","last_undegraded":"2024-08-03T20:50:28.203426+0000","last_fullsized":"2024-08-03T20:50:28.203426+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:50:26.140950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:50:26.140950+0000","last_clean_scrub_stamp":"2024-08-03T20:50:26.140950+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-04T21:23:04.443445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.b","version":"0'0","reported_seq":10,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-08-03T20:50:28.190276+0000","last_change":"2024-08-03T20:50:28.181544+0000","last_active":"2024-08-03T20:50:28.190276+0000","last_peered":"2024-08-03T20:50:28.190276+0000","last_clean":"2024-08-03T20:50:28.190276+0000","last_became_active":"2024-08-03T20:50:28.181116+0000","last_became_peered":"2024-08-03T20:50:28.181116+0000","last_unstale":"2024-08-03T20:50:28.190276+0000","last_undegraded":"2024-08-03T20:50:28.190276+0000","last_fullsized":"2024-08-03T20:50:28.190276+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:50:27.143838+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:50:27.143838+0000","last_clean_scrub_stamp":"2024-08-03T20:50:27.143838+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T02:25:34.324726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.a","version":"0'0","reported_seq":13,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-08-03T20:50:28.163649+0000","last_change":"2024-08-03T20:50:27.182689+0000","last_active":"2024-08-03T20:50:28.163649+0000","last_peered":"2024-08-03T20:50:28.163649+0000","last_clean":"2024-08-03T20:50:28.163649+0000","last_became_active":"2024-08-03T20:50:27.179597+0000","last_became_peered":"2024-08-03T20:50:27.179597+0000","last_unstale":"2024-08-03T20:50:28.163649+0000","last_undegraded":"2024-08-03T20:50:28.163649+0000","last_fullsized":"2024-08-03T20:50:28.163649+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:50:26.140950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:50:26.140950+0000","last_clean_scrub_stamp":"2024-08-03T20:50:26.140950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T04:36:51.691682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.9","version":"0'0","reported_seq":10,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-08-03T20:50:28.184160+0000","last_change":"2024-08-03T20:50:28.170476+0000","last_active":"2024-08-03T20:50:28.184160+0000","last_peered":"2024-08-03T20:50:28.184160+0000","last_clean":"2024-08-03T20:50:28.184160+0000","last_became_active":"2024-08-03T20:50:28.170162+0000","last_became_peered":"2024-08-03T20:50:28.170162+0000","last_unstale":"2024-08-03T20:50:28.184160+0000","last_undegraded":"2024-08-03T20:50:28.184160+0000","last_fullsized":"2024-08-03T20:50:28.184160+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:50:27.143838+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:50:27.143838+0000","last_clean_scrub_stamp":"2024-08-03T20:50:27.143838+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T05:29:26.363244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.8","version":"0'0","reported_seq":13,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-08-03T20:50:28.163532+0000","last_change":"2024-08-03T20:50:27.170004+0000","last_active":"2024-08-03T20:50:28.163532+0000","last_peered":"2024-08-03T20:50:28.163532+0000","last_clean":"2024-08-03T20:50:28.163532+0000","last_became_active":"2024-08-03T20:50:27.169688+0000","last_became_peered":"2024-08-03T20:50:27.169688+0000","last_unstale":"2024-08-03T20:50:28.163532+0000","last_undegraded":"2024-08-03T20:50:28.163532+0000","last_fullsized":"2024-08-03T20:50:28.163532+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:50:26.140950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:50:26.140950+0000","last_clean_scrub_stamp":"2024-08-03T20:50:26.140950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T05:50:17.238961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.6","version":"0'0","reported_seq":10,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-08-03T20:50:28.185200+0000","last_change":"2024-08-03T20:50:28.184395+0000","last_active":"2024-08-03T20:50:28.185200+0000","last_peered":"2024-08-03T20:50:28.185200+0000","last_clean":"2024-08-03T20:50:28.185200+0000","last_became_active":"2024-08-03T20:50:28.184049+0000","last_became_peered":"2024-08-03T20:50:28.184049+0000","last_unstale":"2024-08-03T20:50:28.185200+0000","last_undegraded":"2024-08-03T20:50:28.185200+0000","last_fullsized":"2024-08-03T20:50:28.185200+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:50:27.143838+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:50:27.143838+0000","last_clean_scrub_stamp":"2024-08-03T20:50:27.143838+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T03:35:47.096608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.7","version":"56'2","reported_seq":15,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-08-03T20:50:28.205413+0000","last_change":"2024-08-03T20:50:27.177097+0000","last_active":"2024-08-03T20:50:28.205413+0000","last_peered":"2024-08-03T20:50:28.205413+0000","last_clean":"2024-08-03T20:50:28.205413+0000","last_became_active":"2024-08-03T20:50:27.176767+0000","last_became_peered":"2024-08-03T20:50:27.176767+0000","last_unstale":"2024-08-03T20:50:28.205413+0000","last_undegraded":"2024-08-03T20:50:28.205413+0000","last_fullsized":"2024-08-03T20:50:28.205413+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:50:26.140950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:50:26.140950+0000","last_clean_scrub_stamp":"2024-08-03T20:50:26.140950+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-04T23:11:29.720796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.7","version":"0'0","reported_seq":10,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-08-03T20:50:28.183643+0000","last_change":"2024-08-03T20:50:28.174143+0000","last_active":"2024-08-03T20:50:28.183643+0000","last_peered":"2024-08-03T20:50:28.183643+0000","last_clean":"2024-08-03T20:50:28.183643+0000","last_became_active":"2024-08-03T20:50:28.173270+0000","last_became_peered":"2024-08-03T20:50:28.173270+0000","last_unstale":"2024-08-03T20:50:28.183643+0000","last_undegraded":"2024-08-03T20:50:28.183643+0000","last_fullsized":"2024-08-03T20:50:28.183643+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:50:27.143838+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:50:27.143838+0000","last_clean_scrub_stamp":"2024-08-03T20:50:27.143838+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-04T22:38:57.045083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.6","version":"0'0","reported_seq":13,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-08-03T20:50:28.161128+0000","last_change":"2024-08-03T20:50:27.185137+0000","last_active":"2024-08-03T20:50:28.161128+0000","last_peered":"2024-08-03T20:50:28.161128+0000","last_clean":"2024-08-03T20:50:28.161128+0000","last_became_active":"2024-08-03T20:50:27.184453+0000","last_became_peered":"2024-08-03T20:50:27.184453+0000","last_unstale":"2024-08-03T20:50:28.161128+0000","last_undegraded":"2024-08-03T20:50:28.161128+0000","last_fullsized":"2024-08-03T20:50:28.161128+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:50:26.140950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:50:26.140950+0000","last_clean_scrub_stamp":"2024-08-03T20:50:26.140950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-04T23:15:19.049204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.4","version":"0'0","reported_seq":10,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-08-03T20:50:28.194848+0000","last_change":"2024-08-03T20:50:28.176191+0000","last_active":"2024-08-03T20:50:28.194848+0000","last_peered":"2024-08-03T20:50:28.194848+0000","last_clean":"2024-08-03T20:50:28.194848+0000","last_became_active":"2024-08-03T20:50:28.175386+0000","last_became_peered":"2024-08-03T20:50:28.175386+0000","last_unstale":"2024-08-03T20:50:28.194848+0000","last_undegraded":"2024-08-03T20:50:28.194848+0000","last_fullsized":"2024-08-03T20:50:28.194848+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:50:27.143838+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:50:27.143838+0000","last_clean_scrub_stamp":"2024-08-03T20:50:27.143838+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T03:40:06.179894+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.5","version":"0'0","reported_seq":15,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-08-03T20:50:29.159456+0000","last_change":"2024-08-03T20:50:27.177260+0000","last_active":"2024-08-03T20:50:29.159456+0000","last_peered":"2024-08-03T20:50:29.159456+0000","last_clean":"2024-08-03T20:50:29.159456+0000","last_became_active":"2024-08-03T20:50:27.176895+0000","last_became_peered":"2024-08-03T20:50:27.176895+0000","last_unstale":"2024-08-03T20:50:29.159456+0000","last_undegraded":"2024-08-03T20:50:29.159456+0000","last_fullsized":"2024-08-03T20:50:29.159456+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:50:26.140950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:50:26.140950+0000","last_clean_scrub_stamp":"2024-08-03T20:50:26.140950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-04T23:11:29.720796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.5","version":"0'0","reported_seq":10,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-08-03T20:50:28.184326+0000","last_change":"2024-08-03T20:50:28.177128+0000","last_active":"2024-08-03T20:50:28.184326+0000","last_peered":"2024-08-03T20:50:28.184326+0000","last_clean":"2024-08-03T20:50:28.184326+0000","last_became_active":"2024-08-03T20:50:28.175490+0000","last_became_peered":"2024-08-03T20:50:28.175490+0000","last_unstale":"2024-08-03T20:50:28.184326+0000","last_undegraded":"2024-08-03T20:50:28.184326+0000","last_fullsized":"2024-08-03T20:50:28.184326+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:50:27.143838+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:50:27.143838+0000","last_clean_scrub_stamp":"2024-08-03T20:50:27.143838+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T04:30:20.080808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.4","version":"0'0","reported_seq":13,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-08-03T20:50:28.168531+0000","last_change":"2024-08-03T20:50:27.176793+0000","last_active":"2024-08-03T20:50:28.168531+0000","last_peered":"2024-08-03T20:50:28.168531+0000","last_clean":"2024-08-03T20:50:28.168531+0000","last_became_active":"2024-08-03T20:50:27.176490+0000","last_became_peered":"2024-08-03T20:50:27.176490+0000","last_unstale":"2024-08-03T20:50:28.168531+0000","last_undegraded":"2024-08-03T20:50:28.168531+0000","last_fullsized":"2024-08-03T20:50:28.168531+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:50:26.140950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:50:26.140950+0000","last_clean_scrub_stamp":"2024-08-03T20:50:26.140950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-04T23:32:46.677480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.2","version":"0'0","reported_seq":10,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-08-03T20:50:28.189255+0000","last_change":"2024-08-03T20:50:28.184522+0000","last_active":"2024-08-03T20:50:28.189255+0000","last_peered":"2024-08-03T20:50:28.189255+0000","last_clean":"2024-08-03T20:50:28.189255+0000","last_became_active":"2024-08-03T20:50:28.184144+0000","last_became_peered":"2024-08-03T20:50:28.184144+0000","last_unstale":"2024-08-03T20:50:28.189255+0000","last_undegraded":"2024-08-03T20:50:28.189255+0000","last_fullsized":"2024-08-03T20:50:28.189255+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:50:27.143838+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:50:27.143838+0000","last_clean_scrub_stamp":"2024-08-03T20:50:27.143838+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T01:58:09.341916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.3","version":"0'0","reported_seq":13,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-08-03T20:50:28.163601+0000","last_change":"2024-08-03T20:50:27.167297+0000","last_active":"2024-08-03T20:50:28.163601+0000","last_peered":"2024-08-03T20:50:28.163601+0000","last_clean":"2024-08-03T20:50:28.163601+0000","last_became_active":"2024-08-03T20:50:27.166782+0000","last_became_peered":"2024-08-03T20:50:27.166782+0000","last_unstale":"2024-08-03T20:50:28.163601+0000","last_undegraded":"2024-08-03T20:50:28.163601+0000","last_fullsized":"2024-08-03T20:50:28.163601+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:50:26.140950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:50:26.140950+0000","last_clean_scrub_stamp":"2024-08-03T20:50:26.140950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T06:06:50.290727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.3","version":"0'0","reported_seq":10,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-08-03T20:50:28.183141+0000","last_change":"2024-08-03T20:50:28.177494+0000","last_active":"2024-08-03T20:50:28.183141+0000","last_peered":"2024-08-03T20:50:28.183141+0000","last_clean":"2024-08-03T20:50:28.183141+0000","last_became_active":"2024-08-03T20:50:28.177186+0000","last_became_peered":"2024-08-03T20:50:28.177186+0000","last_unstale":"2024-08-03T20:50:28.183141+0000","last_undegraded":"2024-08-03T20:50:28.183141+0000","last_fullsized":"2024-08-03T20:50:28.183141+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:50:27.143838+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:50:27.143838+0000","last_clean_scrub_stamp":"2024-08-03T20:50:27.143838+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T02:25:34.324726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.2","version":"0'0","reported_seq":13,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-08-03T20:50:28.165657+0000","last_change":"2024-08-03T20:50:27.180189+0000","last_active":"2024-08-03T20:50:28.165657+0000","last_peered":"2024-08-03T20:50:28.165657+0000","last_clean":"2024-08-03T20:50:28.165657+0000","last_became_active":"2024-08-03T20:50:27.178485+0000","last_became_peered":"2024-08-03T20:50:27.178485+0000","last_unstale":"2024-08-03T20:50:28.165657+0000","last_undegraded":"2024-08-03T20:50:28.165657+0000","last_fullsized":"2024-08-03T20:50:28.165657+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:50:26.140950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:50:26.140950+0000","last_clean_scrub_stamp":"2024-08-03T20:50:26.140950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T01:50:59.525120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.0","version":"0'0","reported_seq":10,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-08-03T20:50:28.180478+0000","last_change":"2024-08-03T20:50:28.176927+0000","last_active":"2024-08-03T20:50:28.180478+0000","last_peered":"2024-08-03T20:50:28.180478+0000","last_clean":"2024-08-03T20:50:28.180478+0000","last_became_active":"2024-08-03T20:50:28.176606+0000","last_became_peered":"2024-08-03T20:50:28.176606+0000","last_unstale":"2024-08-03T20:50:28.180478+0000","last_undegraded":"2024-08-03T20:50:28.180478+0000","last_fullsized":"2024-08-03T20:50:28.180478+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:50:27.143838+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:50:27.143838+0000","last_clean_scrub_stamp":"2024-08-03T20:50:27.143838+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T01:03:58.561921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.1","version":"0'0","reported_seq":13,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-08-03T20:50:28.161679+0000","last_change":"2024-08-03T20:50:27.181804+0000","last_active":"2024-08-03T20:50:28.161679+0000","last_peered":"2024-08-03T20:50:28.161679+0000","last_clean":"2024-08-03T20:50:28.161679+0000","last_became_active":"2024-08-03T20:50:27.179785+0000","last_became_peered":"2024-08-03T20:50:27.179785+0000","last_unstale":"2024-08-03T20:50:28.161679+0000","last_undegraded":"2024-08-03T20:50:28.161679+0000","last_fullsized":"2024-08-03T20:50:28.161679+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:50:26.140950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:50:26.140950+0000","last_clean_scrub_stamp":"2024-08-03T20:50:26.140950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T06:35:28.032286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.1","version":"0'0","reported_seq":10,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-08-03T20:50:28.187460+0000","last_change":"2024-08-03T20:50:28.178706+0000","last_active":"2024-08-03T20:50:28.187460+0000","last_peered":"2024-08-03T20:50:28.187460+0000","last_clean":"2024-08-03T20:50:28.187460+0000","last_became_active":"2024-08-03T20:50:28.177015+0000","last_became_peered":"2024-08-03T20:50:28.177015+0000","last_unstale":"2024-08-03T20:50:28.187460+0000","last_undegraded":"2024-08-03T20:50:28.187460+0000","last_fullsized":"2024-08-03T20:50:28.187460+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:50:27.143838+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:50:27.143838+0000","last_clean_scrub_stamp":"2024-08-03T20:50:27.143838+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T02:31:57.413818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.0","version":"0'0","reported_seq":13,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-08-03T20:50:28.173820+0000","last_change":"2024-08-03T20:50:27.171865+0000","last_active":"2024-08-03T20:50:28.173820+0000","last_peered":"2024-08-03T20:50:28.173820+0000","last_clean":"2024-08-03T20:50:28.173820+0000","last_became_active":"2024-08-03T20:50:27.170850+0000","last_became_peered":"2024-08-03T20:50:27.170850+0000","last_unstale":"2024-08-03T20:50:28.173820+0000","last_undegraded":"2024-08-03T20:50:28.173820+0000","last_fullsized":"2024-08-03T20:50:28.173820+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:50:26.140950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:50:26.140950+0000","last_clean_scrub_stamp":"2024-08-03T20:50:26.140950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-04T23:29:42.990546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.8","version":"0'0","reported_seq":10,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-08-03T20:50:28.188928+0000","last_change":"2024-08-03T20:50:28.186122+0000","last_active":"2024-08-03T20:50:28.188928+0000","last_peered":"2024-08-03T20:50:28.188928+0000","last_clean":"2024-08-03T20:50:28.188928+0000","last_became_active":"2024-08-03T20:50:28.185821+0000","last_became_peered":"2024-08-03T20:50:28.185821+0000","last_unstale":"2024-08-03T20:50:28.188928+0000","last_undegraded":"2024-08-03T20:50:28.188928+0000","last_fullsized":"2024-08-03T20:50:28.188928+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:50:27.143838+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:50:27.143838+0000","last_clean_scrub_stamp":"2024-08-03T20:50:27.143838+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T07:20:34.133759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.9","version":"0'0","reported_seq":15,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-08-03T20:50:29.159153+0000","last_change":"2024-08-03T20:50:27.165639+0000","last_active":"2024-08-03T20:50:29.159153+0000","last_peered":"2024-08-03T20:50:29.159153+0000","last_clean":"2024-08-03T20:50:29.159153+0000","last_became_active":"2024-08-03T20:50:27.165179+0000","last_became_peered":"2024-08-03T20:50:27.165179+0000","last_unstale":"2024-08-03T20:50:29.159153+0000","last_undegraded":"2024-08-03T20:50:29.159153+0000","last_fullsized":"2024-08-03T20:50:29.159153+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:50:26.140950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:50:26.140950+0000","last_clean_scrub_stamp":"2024-08-03T20:50:26.140950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T06:06:50.290727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"16'72","reported_seq":173,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-08-03T20:50:28.157654+0000","last_change":"2024-08-03T20:47:30.547632+0000","last_active":"2024-08-03T20:50:28.157654+0000","last_peered":"2024-08-03T20:50:28.157654+0000","last_clean":"2024-08-03T20:50:28.157654+0000","last_became_active":"2024-08-03T20:47:30.546255+0000","last_became_peered":"2024-08-03T20:47:30.546255+0000","last_unstale":"2024-08-03T20:50:28.157654+0000","last_undegraded":"2024-08-03T20:50:28.157654+0000","last_fullsized":"2024-08-03T20:50:28.157654+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:47:28.874451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:47:28.874451+0000","last_clean_scrub_stamp":"2024-08-03T20:47:28.874451+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T00:19:40.548812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":9,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":8,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509479,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32420,"kb_used_data":1236,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339420,"statfs":{"total":96636764160,"available":96603566080,"internally_reserved":0,"allocated":1265664,"data_stored":864908,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":6,"up_from":15,"seq":64424509480,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31828,"kb_used_data":648,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340012,"statfs":{"total":96636764160,"available":96604172288,"internally_reserved":0,"allocated":663552,"data_stored":274380,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":5,"up_from":15,"seq":64424509480,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31836,"kb_used_data":656,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340004,"statfs":{"total":96636764160,"available":96604164096,"internally_reserved":0,"allocated":671744,"data_stored":275032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509479,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31832,"kb_used_data":652,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340008,"statfs":{"total":96636764160,"available":96604168192,"internally_reserved":0,"allocated":667648,"data_stored":274470,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,4,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509481,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32428,"kb_used_data":1244,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339412,"statfs":{"total":96636764160,"available":96603557888,"internally_reserved":0,"allocated":1273856,"data_stored":866942,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542184,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31844,"kb_used_data":660,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339996,"statfs":{"total":96636764160,"available":96604155904,"internally_reserved":0,"allocated":675840,"data_stored":275992,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":3,"up_from":14,"seq":60129542182,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31816,"kb_used_data":636,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340024,"statfs":{"total":96636764160,"available":96604184576,"internally_reserved":0,"allocated":651264,"data_stored":267605,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":2,"up_from":14,"seq":60129542183,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31816,"kb_used_data":636,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340024,"statfs":{"total":96636764160,"available":96604184576,"internally_reserved":0,"allocated":651264,"data_stored":267605,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2194,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-08-03T20:50:33.216 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-08-03T20:50:33.216 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-08-03T20:50:33.487 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:33.486+0000 7f904f400640 1 -- 172.21.15.32:0/676916855 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f90480ced30 msgr2=0x7f90480cf140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:50:33.487 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:33.486+0000 7f904f400640 1 --2- 172.21.15.32:0/676916855 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f90480ced30 0x7f90480cf140 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7f9048074a30 tx=0x7f904002aa50 comp rx=0 tx=0).stop 2024-08-03T20:50:33.487 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:33.486+0000 7f904f400640 1 -- 172.21.15.32:0/676916855 shutdown_connections 2024-08-03T20:50:33.487 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:33.486+0000 7f904f400640 1 --2- 172.21.15.32:0/676916855 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f90480ced30 0x7f90480cf140 unknown :-1 s=CLOSED pgs=578 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:33.487 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:33.486+0000 7f904f400640 1 --2- 172.21.15.32:0/676916855 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f90480cccf0 0x7f90480cd100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:33.488 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:33.486+0000 7f904f400640 1 --2- 172.21.15.32:0/676916855 >> v2:172.21.15.118:3300/0 conn(0x7f9048145c50 0x7f904806e950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:33.488 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:33.486+0000 7f904f400640 1 -- 172.21.15.32:0/676916855 >> 172.21.15.32:0/676916855 conn(0x7f90480b1bd0 msgr2=0x7f90480b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:50:33.488 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:33.486+0000 7f904f400640 1 -- 172.21.15.32:0/676916855 shutdown_connections 2024-08-03T20:50:33.488 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:33.486+0000 7f904f400640 1 -- 172.21.15.32:0/676916855 wait complete. 2024-08-03T20:50:33.489 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:33.488+0000 7f904f400640 1 Processor -- start 2024-08-03T20:50:33.489 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:33.489+0000 7f904f400640 1 -- start start 2024-08-03T20:50:33.491 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:33.490+0000 7f904f400640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f90480cccf0 0x7f904813d660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:50:33.491 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:33.490+0000 7f904f400640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f90480ced30 0x7f904815b960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:50:33.491 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:33.490+0000 7f904f400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9048145c50 0x7f90480c5920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:50:33.491 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:33.490+0000 7f904f400640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f90480b3e60 con 0x7f9048145c50 2024-08-03T20:50:33.491 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:33.490+0000 7f904f400640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f90480b3ce0 con 0x7f90480cccf0 2024-08-03T20:50:33.491 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:33.490+0000 7f904f400640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7f90480b3fe0 con 0x7f90480ced30 2024-08-03T20:50:33.491 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:33.490+0000 7f904d400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9048145c50 0x7f90480c5920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:50:33.491 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:33.490+0000 7f904d400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9048145c50 0x7f90480c5920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:45120/0 (socket says 172.21.15.32:45120) 2024-08-03T20:50:33.491 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:33.490+0000 7f904d400640 1 -- 172.21.15.32:0/3839468101 learned_addr learned my addr 172.21.15.32:0/3839468101 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:50:33.491 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:33.491+0000 7f9047e00640 1 --2- 172.21.15.32:0/3839468101 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f90480ced30 0x7f904815b960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:50:33.492 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:33.491+0000 7f904ca00640 1 --2- 172.21.15.32:0/3839468101 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f90480cccf0 0x7f904813d660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:50:33.493 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:33.491+0000 7f904d400640 1 -- 172.21.15.32:0/3839468101 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f90480ced30 msgr2=0x7f904815b960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:50:33.493 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:33.491+0000 7f904d400640 1 --2- 172.21.15.32:0/3839468101 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f90480ced30 0x7f904815b960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:33.493 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:33.491+0000 7f904d400640 1 -- 172.21.15.32:0/3839468101 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f90480cccf0 msgr2=0x7f904813d660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:50:33.493 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:33.491+0000 7f904d400640 1 --2- 172.21.15.32:0/3839468101 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f90480cccf0 0x7f904813d660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:33.493 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:33.491+0000 7f904d400640 1 -- 172.21.15.32:0/3839468101 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f904815bea0 con 0x7f9048145c50 2024-08-03T20:50:33.493 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:33.491+0000 7f904d400640 1 --2- 172.21.15.32:0/3839468101 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9048145c50 0x7f90480c5920 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7f9040009d90 tx=0x7f9040003f10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:50:33.493 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:33.491+0000 7f9044c00640 1 -- 172.21.15.32:0/3839468101 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9040033050 con 0x7f9048145c50 2024-08-03T20:50:33.493 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:33.492+0000 7f9044c00640 1 -- 172.21.15.32:0/3839468101 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f9040004250 con 0x7f9048145c50 2024-08-03T20:50:33.493 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:33.492+0000 7f904f400640 1 -- 172.21.15.32:0/3839468101 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f90480c7610 con 0x7f9048145c50 2024-08-03T20:50:33.494 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:33.492+0000 7f9044c00640 1 -- 172.21.15.32:0/3839468101 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9040004660 con 0x7f9048145c50 2024-08-03T20:50:33.494 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:33.492+0000 7f904f400640 1 -- 172.21.15.32:0/3839468101 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f90480c4e40 con 0x7f9048145c50 2024-08-03T20:50:33.494 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:33.493+0000 7f904f400640 1 -- 172.21.15.32:0/3839468101 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f904815f220 con 0x7f9048145c50 2024-08-03T20:50:33.494 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:33.494+0000 7f9044c00640 1 -- 172.21.15.32:0/3839468101 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f9040036070 con 0x7f9048145c50 2024-08-03T20:50:33.495 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:33.495+0000 7f9044c00640 1 --2- 172.21.15.32:0/3839468101 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f9028077040 0x7f9028079550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:50:33.495 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:33.495+0000 7f9044c00640 1 -- 172.21.15.32:0/3839468101 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(57..57 src has 1..57) ==== 5709+0+0 (secure 0 0 0) 0x7f90400b9b50 con 0x7f9048145c50 2024-08-03T20:50:33.496 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:33.495+0000 7f904ca00640 1 --2- 172.21.15.32:0/3839468101 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f9028077040 0x7f9028079550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:50:33.496 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:33.496+0000 7f904ca00640 1 --2- 172.21.15.32:0/3839468101 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f9028077040 0x7f9028079550 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f903c009ab0 tx=0x7f903c005f90 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:50:33.500 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:33.499+0000 7f9044c00640 1 -- 172.21.15.32:0/3839468101 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7f90400825e0 con 0x7f9048145c50 2024-08-03T20:50:33.783 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:33.782+0000 7f904f400640 1 -- 172.21.15.32:0/3839468101 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) -- 0x7f90480c73a0 con 0x7f9048145c50 2024-08-03T20:50:33.786 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:33.786+0000 7f9044c00640 1 -- 172.21.15.32:0/3839468101 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v13) ==== 190+0+147 (secure 0 0 0) 0x7f9040086b30 con 0x7f9048145c50 2024-08-03T20:50:33.786 INFO:teuthology.orchestra.run.smithi032.stdout:[client.0] 2024-08-03T20:50:33.787 INFO:teuthology.orchestra.run.smithi032.stdout: key = AQAwl65msgO3NRAAHG/WuMTK9c4lDXocXvqWnA== 2024-08-03T20:50:33.787 INFO:teuthology.orchestra.run.smithi032.stdout: caps mds = "allow" 2024-08-03T20:50:33.787 INFO:teuthology.orchestra.run.smithi032.stdout: caps mon = "allow r" 2024-08-03T20:50:33.787 INFO:teuthology.orchestra.run.smithi032.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2024-08-03T20:50:33.787 INFO:teuthology.orchestra.run.smithi032.stderr:updated caps for client.0 2024-08-03T20:50:33.790 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:33.789+0000 7f904f400640 1 -- 172.21.15.32:0/3839468101 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f9028077040 msgr2=0x7f9028079550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:50:33.790 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:33.789+0000 7f904f400640 1 --2- 172.21.15.32:0/3839468101 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f9028077040 0x7f9028079550 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f903c009ab0 tx=0x7f903c005f90 comp rx=0 tx=0).stop 2024-08-03T20:50:33.790 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:33.789+0000 7f904f400640 1 -- 172.21.15.32:0/3839468101 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9048145c50 msgr2=0x7f90480c5920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:50:33.790 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:33.789+0000 7f904f400640 1 --2- 172.21.15.32:0/3839468101 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9048145c50 0x7f90480c5920 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7f9040009d90 tx=0x7f9040003f10 comp rx=0 tx=0).stop 2024-08-03T20:50:33.791 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:33.789+0000 7f904f400640 1 -- 172.21.15.32:0/3839468101 shutdown_connections 2024-08-03T20:50:33.791 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:33.789+0000 7f904f400640 1 --2- 172.21.15.32:0/3839468101 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f9028077040 0x7f9028079550 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:33.791 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:33.789+0000 7f904f400640 1 --2- 172.21.15.32:0/3839468101 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9048145c50 0x7f90480c5920 unknown :-1 s=CLOSED pgs=579 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:33.791 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:33.789+0000 7f904f400640 1 --2- 172.21.15.32:0/3839468101 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f90480ced30 0x7f904815b960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:33.791 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:33.789+0000 7f904f400640 1 --2- 172.21.15.32:0/3839468101 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f90480cccf0 0x7f904813d660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:33.791 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:33.790+0000 7f904f400640 1 -- 172.21.15.32:0/3839468101 >> 172.21.15.32:0/3839468101 conn(0x7f90480b1bd0 msgr2=0x7f9048070150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:50:33.791 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:33.790+0000 7f904f400640 1 -- 172.21.15.32:0/3839468101 shutdown_connections 2024-08-03T20:50:33.791 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:33.790+0000 7f904f400640 1 -- 172.21.15.32:0/3839468101 wait complete. 2024-08-03T20:50:33.815 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-08-03T20:50:34.043 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:34.042+0000 7fba88c00640 1 -- 172.21.15.32:0/2164341103 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fba8406e930 msgr2=0x7fba84144f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:50:34.043 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:34.042+0000 7fba88c00640 1 --2- 172.21.15.32:0/2164341103 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fba8406e930 0x7fba84144f90 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7fba84072d80 tx=0x7fba7402a530 comp rx=0 tx=0).stop 2024-08-03T20:50:34.044 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:34.042+0000 7fba88c00640 1 -- 172.21.15.32:0/2164341103 shutdown_connections 2024-08-03T20:50:34.044 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:34.042+0000 7fba88c00640 1 --2- 172.21.15.32:0/2164341103 >> v2:172.21.15.118:3300/0 conn(0x7fba8415fff0 0x7fba84160400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:34.044 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:34.042+0000 7fba88c00640 1 --2- 172.21.15.32:0/2164341103 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fba8406e930 0x7fba84144f90 unknown :-1 s=CLOSED pgs=581 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:34.044 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:34.042+0000 7fba88c00640 1 -- 172.21.15.32:0/2164341103 >> 172.21.15.32:0/2164341103 conn(0x7fba840b1bd0 msgr2=0x7fba840b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:50:34.044 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:34.042+0000 7fba88c00640 1 -- 172.21.15.32:0/2164341103 shutdown_connections 2024-08-03T20:50:34.044 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:34.043+0000 7fba88c00640 1 -- 172.21.15.32:0/2164341103 wait complete. 2024-08-03T20:50:34.045 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:34.045+0000 7fba88c00640 1 Processor -- start 2024-08-03T20:50:34.045 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:34.045+0000 7fba88c00640 1 -- start start 2024-08-03T20:50:34.048 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:34.046+0000 7fba88c00640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fba8406e930 0x7fba8415f2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:50:34.048 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:34.046+0000 7fba88c00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fba84073e60 0x7fba84159f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:50:34.049 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:34.046+0000 7fba88c00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fba8415fff0 0x7fba840cc540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:50:34.049 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:34.046+0000 7fba88c00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fba840cf840 con 0x7fba8415fff0 2024-08-03T20:50:34.049 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:34.046+0000 7fba88c00640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7fba84145f30 con 0x7fba84073e60 2024-08-03T20:50:34.049 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:34.046+0000 7fba88c00640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7fba840cf9c0 con 0x7fba8406e930 2024-08-03T20:50:34.049 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:34.047+0000 7fba82800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fba8415fff0 0x7fba840cc540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:50:34.049 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:34.047+0000 7fba82800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fba8415fff0 0x7fba840cc540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:45130/0 (socket says 172.21.15.32:45130) 2024-08-03T20:50:34.049 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:34.047+0000 7fba82800640 1 -- 172.21.15.32:0/2444241759 learned_addr learned my addr 172.21.15.32:0/2444241759 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:50:34.049 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:34.047+0000 7fba81e00640 1 --2- 172.21.15.32:0/2444241759 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fba8406e930 0x7fba8415f2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:50:34.049 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:34.047+0000 7fba81400640 1 --2- 172.21.15.32:0/2444241759 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fba84073e60 0x7fba84159f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:50:34.049 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:34.047+0000 7fba82800640 1 -- 172.21.15.32:0/2444241759 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fba8406e930 msgr2=0x7fba8415f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:50:34.049 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:34.047+0000 7fba82800640 1 --2- 172.21.15.32:0/2444241759 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fba8406e930 0x7fba8415f2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:34.049 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:34.047+0000 7fba82800640 1 -- 172.21.15.32:0/2444241759 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fba84073e60 msgr2=0x7fba84159f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:50:34.049 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:34.047+0000 7fba82800640 1 --2- 172.21.15.32:0/2444241759 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fba84073e60 0x7fba84159f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:34.050 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:34.047+0000 7fba82800640 1 -- 172.21.15.32:0/2444241759 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fba8415f820 con 0x7fba8415fff0 2024-08-03T20:50:34.050 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:34.047+0000 7fba82800640 1 --2- 172.21.15.32:0/2444241759 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fba8415fff0 0x7fba840cc540 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7fba7c00da90 tx=0x7fba7c00df50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:50:34.050 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:34.048+0000 7fba7a000640 1 -- 172.21.15.32:0/2444241759 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba7c017070 con 0x7fba8415fff0 2024-08-03T20:50:34.050 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:34.048+0000 7fba7a000640 1 -- 172.21.15.32:0/2444241759 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fba7c003970 con 0x7fba8415fff0 2024-08-03T20:50:34.050 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:34.048+0000 7fba7a000640 1 -- 172.21.15.32:0/2444241759 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba7c003dc0 con 0x7fba8415fff0 2024-08-03T20:50:34.050 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:34.048+0000 7fba88c00640 1 -- 172.21.15.32:0/2444241759 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fba84141910 con 0x7fba8415fff0 2024-08-03T20:50:34.050 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:34.048+0000 7fba88c00640 1 -- 172.21.15.32:0/2444241759 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fba84141ed0 con 0x7fba8415fff0 2024-08-03T20:50:34.051 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:34.049+0000 7fba88c00640 1 -- 172.21.15.32:0/2444241759 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fba84160c30 con 0x7fba8415fff0 2024-08-03T20:50:34.051 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:34.050+0000 7fba7a000640 1 -- 172.21.15.32:0/2444241759 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7fba7c003f60 con 0x7fba8415fff0 2024-08-03T20:50:34.052 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:34.051+0000 7fba7a000640 1 --2- 172.21.15.32:0/2444241759 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fba64077010 0x7fba64079520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:50:34.052 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:34.051+0000 7fba7a000640 1 -- 172.21.15.32:0/2444241759 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(57..57 src has 1..57) ==== 5709+0+0 (secure 0 0 0) 0x7fba7c098cb0 con 0x7fba8415fff0 2024-08-03T20:50:34.052 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:34.051+0000 7fba81e00640 1 --2- 172.21.15.32:0/2444241759 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fba64077010 0x7fba64079520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:50:34.052 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:34.052+0000 7fba81e00640 1 --2- 172.21.15.32:0/2444241759 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fba64077010 0x7fba64079520 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7fba74034000 tx=0x7fba74004730 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:50:34.056 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:34.056+0000 7fba7a000640 1 -- 172.21.15.32:0/2444241759 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7fba7c0617c0 con 0x7fba8415fff0 2024-08-03T20:50:34.305 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:34.304+0000 7fba88c00640 1 -- 172.21.15.32:0/2444241759 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fba840cd110 con 0x7fba8415fff0 2024-08-03T20:50:34.306 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:34.305+0000 7fba7a000640 1 -- 172.21.15.32:0/2444241759 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 51 v51) ==== 94+0+5229 (secure 0 0 0) 0x7fba7c065d10 con 0x7fba8415fff0 2024-08-03T20:50:34.306 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-03T20:50:34.306 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":51,"btime":"2024-08-03T20:50:31:184656+0000","default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5261,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6833/2498840646","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":2498840646},{"type":"v1","addr":"172.21.15.32:6833","nonce":2498840646}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":45},{"gid":5276,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6837/3142148286","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6835","nonce":3142148286},{"type":"v1","addr":"172.21.15.118:6837","nonce":3142148286}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":46},{"gid":5632,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6836/3720010453","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6834","nonce":3720010453},{"type":"v1","addr":"172.21.15.118:6836","nonce":3720010453}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":46}],"filesystems":[{"mdsmap":{"epoch":51,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-08-03T20:50:27.688698+0000","modified":"2024-08-03T20:50:30.704840+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5841},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5841":{"gid":5841,"name":"c","rank":0,"incarnation":47,"state":"up:active","state_seq":3,"addr":"172.21.15.32:6835/3848976369","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":3848976369},{"type":"v1","addr":"172.21.15.32:6835","nonce":3848976369}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[9],"metadata_pool":8,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5841,"qdb_cluster":[5841]},"id":4}]} 2024-08-03T20:50:34.307 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 51 2024-08-03T20:50:34.308 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:34.307+0000 7fba88c00640 1 -- 172.21.15.32:0/2444241759 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fba64077010 msgr2=0x7fba64079520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:50:34.308 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:34.307+0000 7fba88c00640 1 --2- 172.21.15.32:0/2444241759 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fba64077010 0x7fba64079520 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7fba74034000 tx=0x7fba74004730 comp rx=0 tx=0).stop 2024-08-03T20:50:34.309 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:34.307+0000 7fba88c00640 1 -- 172.21.15.32:0/2444241759 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fba8415fff0 msgr2=0x7fba840cc540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:50:34.309 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:34.307+0000 7fba88c00640 1 --2- 172.21.15.32:0/2444241759 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fba8415fff0 0x7fba840cc540 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7fba7c00da90 tx=0x7fba7c00df50 comp rx=0 tx=0).stop 2024-08-03T20:50:34.309 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:34.307+0000 7fba88c00640 1 -- 172.21.15.32:0/2444241759 shutdown_connections 2024-08-03T20:50:34.309 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:34.307+0000 7fba88c00640 1 --2- 172.21.15.32:0/2444241759 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fba64077010 0x7fba64079520 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:34.309 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:34.307+0000 7fba88c00640 1 --2- 172.21.15.32:0/2444241759 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fba84073e60 0x7fba84159f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:34.309 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:34.307+0000 7fba88c00640 1 --2- 172.21.15.32:0/2444241759 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fba8415fff0 0x7fba840cc540 unknown :-1 s=CLOSED pgs=582 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:34.309 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:34.307+0000 7fba88c00640 1 --2- 172.21.15.32:0/2444241759 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fba8406e930 0x7fba8415f2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:34.309 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:34.307+0000 7fba88c00640 1 -- 172.21.15.32:0/2444241759 >> 172.21.15.32:0/2444241759 conn(0x7fba840b1bd0 msgr2=0x7fba84140cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:50:34.309 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:34.307+0000 7fba88c00640 1 -- 172.21.15.32:0/2444241759 shutdown_connections 2024-08-03T20:50:34.309 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:34.308+0000 7fba88c00640 1 -- 172.21.15.32:0/2444241759 wait complete. 2024-08-03T20:50:34.738 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 51, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-08-03T20:50:27.688698+0000', 'modified': '2024-08-03T20:50:30.704840+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 5841}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_5841': {'gid': 5841, 'name': 'c', 'rank': 0, 'incarnation': 47, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.32:6835/3848976369', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6834', 'nonce': 3848976369}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 3848976369}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [9], 'metadata_pool': 8, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 5841, 'qdb_cluster': [5841]} 2024-08-03T20:50:34.738 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-08-03T20:50:34.738 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-08-03T20:50:34.972 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:34.971+0000 7feb2ca00640 1 -- 172.21.15.32:0/1950773597 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7feb0c0039e0 msgr2=0x7feb0c003e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:50:34.972 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:34.971+0000 7feb2ca00640 1 --2- 172.21.15.32:0/1950773597 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7feb0c0039e0 0x7feb0c003e10 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7feb18001740 tx=0x7feb0c005830 comp rx=0 tx=0).stop 2024-08-03T20:50:34.972 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:34.971+0000 7feb26800640 1 -- 172.21.15.32:0/1950773597 reap_dead start 2024-08-03T20:50:34.973 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:34.972+0000 7feb2ca00640 1 -- 172.21.15.32:0/1950773597 shutdown_connections 2024-08-03T20:50:34.973 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:34.972+0000 7feb2ca00640 1 -- 172.21.15.32:0/1950773597 >> 172.21.15.32:0/1950773597 conn(0x7feb280b1bd0 msgr2=0x7feb280b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:50:34.973 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:34.972+0000 7feb2ca00640 1 -- 172.21.15.32:0/1950773597 shutdown_connections 2024-08-03T20:50:34.973 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:34.972+0000 7feb2ca00640 1 -- 172.21.15.32:0/1950773597 wait complete. 2024-08-03T20:50:34.974 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:34.974+0000 7feb2ca00640 1 Processor -- start 2024-08-03T20:50:34.974 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:34.974+0000 7feb2ca00640 1 -- start start 2024-08-03T20:50:34.976 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:34.975+0000 7feb2ca00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7feb0c0039e0 0x7feb28154200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:50:34.976 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:34.976+0000 7feb2ca00640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7feb281f4f00 0x7feb28150c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:50:34.976 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:34.976+0000 7feb2ca00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7feb2815aba0 0x7feb281599c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:50:34.977 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:34.976+0000 7feb2ca00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7feb280ae920 con 0x7feb2815aba0 2024-08-03T20:50:34.977 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:34.976+0000 7feb2ca00640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7feb280ae7a0 con 0x7feb0c0039e0 2024-08-03T20:50:34.977 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:34.976+0000 7feb2ca00640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7feb280aeaa0 con 0x7feb281f4f00 2024-08-03T20:50:34.977 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:34.976+0000 7feb26800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7feb2815aba0 0x7feb281599c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:50:34.977 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:34.976+0000 7feb26800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7feb2815aba0 0x7feb281599c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:45152/0 (socket says 172.21.15.32:45152) 2024-08-03T20:50:34.977 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:34.976+0000 7feb26800640 1 -- 172.21.15.32:0/1637709586 learned_addr learned my addr 172.21.15.32:0/1637709586 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:50:34.977 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:34.976+0000 7feb25400640 1 --2- 172.21.15.32:0/1637709586 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7feb281f4f00 0x7feb28150c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:50:34.977 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:34.976+0000 7feb25e00640 1 --2- 172.21.15.32:0/1637709586 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7feb0c0039e0 0x7feb28154200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:50:34.978 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:34.976+0000 7feb26800640 1 -- 172.21.15.32:0/1637709586 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7feb281f4f00 msgr2=0x7feb28150c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:50:34.978 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:34.976+0000 7feb26800640 1 --2- 172.21.15.32:0/1637709586 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7feb281f4f00 0x7feb28150c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:34.978 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:34.976+0000 7feb26800640 1 -- 172.21.15.32:0/1637709586 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7feb0c0039e0 msgr2=0x7feb28154200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:50:34.978 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:34.976+0000 7feb26800640 1 --2- 172.21.15.32:0/1637709586 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7feb0c0039e0 0x7feb28154200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:34.978 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:34.976+0000 7feb26800640 1 -- 172.21.15.32:0/1637709586 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feb281f56e0 con 0x7feb2815aba0 2024-08-03T20:50:34.979 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:34.976+0000 7feb25400640 1 --2- 172.21.15.32:0/1637709586 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7feb281f4f00 0x7feb28150c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-03T20:50:34.979 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:34.977+0000 7feb26800640 1 --2- 172.21.15.32:0/1637709586 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7feb2815aba0 0x7feb281599c0 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7feb2815c780 tx=0x7feb20010f40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:50:34.979 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:34.977+0000 7feb1e000640 1 -- 172.21.15.32:0/1637709586 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb20003970 con 0x7feb2815aba0 2024-08-03T20:50:34.979 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:34.977+0000 7feb1e000640 1 -- 172.21.15.32:0/1637709586 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7feb20015070 con 0x7feb2815aba0 2024-08-03T20:50:34.979 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:34.977+0000 7feb1e000640 1 -- 172.21.15.32:0/1637709586 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb20004030 con 0x7feb2815aba0 2024-08-03T20:50:34.979 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:34.977+0000 7feb2ca00640 1 -- 172.21.15.32:0/1637709586 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7feb281f59c0 con 0x7feb2815aba0 2024-08-03T20:50:34.979 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:34.977+0000 7feb2ca00640 1 -- 172.21.15.32:0/1637709586 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7feb281594a0 con 0x7feb2815aba0 2024-08-03T20:50:34.980 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:34.980+0000 7feb1e000640 1 -- 172.21.15.32:0/1637709586 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7feb200041d0 con 0x7feb2815aba0 2024-08-03T20:50:34.981 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:34.981+0000 7feb1e000640 1 --2- 172.21.15.32:0/1637709586 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7feb08077010 0x7feb08079520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:50:34.981 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:34.981+0000 7feb1e000640 1 -- 172.21.15.32:0/1637709586 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(57..57 src has 1..57) ==== 5709+0+0 (secure 0 0 0) 0x7feb20098fd0 con 0x7feb2815aba0 2024-08-03T20:50:34.981 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:34.981+0000 7feb25e00640 1 --2- 172.21.15.32:0/1637709586 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7feb08077010 0x7feb08079520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:50:34.981 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:34.981+0000 7feb33bd6740 1 Processor -- start 2024-08-03T20:50:34.982 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:34.982+0000 7feb33bd6740 1 -- start start 2024-08-03T20:50:34.982 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:34.982+0000 7feb25e00640 1 --2- 172.21.15.32:0/1637709586 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7feb08077010 0x7feb08079520 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7feb18001710 tx=0x7feb18001270 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:50:34.984 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:34.983+0000 7feb33bd6740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55d1c6ef8270 0x55d1c72e2e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:50:34.984 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:34.983+0000 7feb33bd6740 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55d1c6ecf3b0 0x55d1c72b7ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:50:34.985 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:34.983+0000 7feb33bd6740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55d1c6ed7590 0x55d1c72b8220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:50:34.985 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:34.983+0000 7feb33bd6740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55d1c7210a60 con 0x55d1c6ed7590 2024-08-03T20:50:34.985 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:34.983+0000 7feb33bd6740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x55d1c71fff60 con 0x55d1c6ef8270 2024-08-03T20:50:34.985 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:34.983+0000 7feb33bd6740 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x55d1c7203fa0 con 0x55d1c6ecf3b0 2024-08-03T20:50:34.985 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:34.983+0000 7feb26800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55d1c6ed7590 0x55d1c72b8220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:50:34.985 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:34.983+0000 7feb26800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55d1c6ed7590 0x55d1c72b8220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:45156/0 (socket says 172.21.15.32:45156) 2024-08-03T20:50:34.985 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:34.983+0000 7feb26800640 1 -- 172.21.15.32:0/1240785241 learned_addr learned my addr 172.21.15.32:0/1240785241 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:50:34.985 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:34.983+0000 7feb25e00640 1 --2- 172.21.15.32:0/1240785241 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55d1c6ef8270 0x55d1c72e2e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:50:34.986 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:34.983+0000 7feb25400640 1 --2- 172.21.15.32:0/1240785241 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55d1c6ecf3b0 0x55d1c72b7ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:50:34.986 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:34.984+0000 7feb26800640 1 -- 172.21.15.32:0/1240785241 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55d1c6ecf3b0 msgr2=0x55d1c72b7ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:50:34.986 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:34.984+0000 7feb26800640 1 --2- 172.21.15.32:0/1240785241 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55d1c6ecf3b0 0x55d1c72b7ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:34.986 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:34.984+0000 7feb26800640 1 -- 172.21.15.32:0/1240785241 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55d1c6ef8270 msgr2=0x55d1c72e2e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:50:34.986 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:34.984+0000 7feb26800640 1 --2- 172.21.15.32:0/1240785241 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55d1c6ef8270 0x55d1c72e2e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:34.986 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:34.984+0000 7feb26800640 1 -- 172.21.15.32:0/1240785241 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d1c6f63850 con 0x55d1c6ed7590 2024-08-03T20:50:34.986 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:34.984+0000 7feb26800640 1 --2- 172.21.15.32:0/1240785241 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55d1c6ed7590 0x55d1c72b8220 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7feb20097520 tx=0x7feb20061660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cad974b6e998c276 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:50:34.986 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:34.984+0000 7feb12600640 1 -- 172.21.15.32:0/1240785241 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb20078070 con 0x55d1c6ed7590 2024-08-03T20:50:34.986 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:34.984+0000 7feb12600640 1 -- 172.21.15.32:0/1240785241 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7feb20066160 con 0x55d1c6ed7590 2024-08-03T20:50:34.986 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:34.984+0000 7feb12600640 1 -- 172.21.15.32:0/1240785241 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb20021070 con 0x55d1c6ed7590 2024-08-03T20:50:34.987 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:34.985+0000 7feb33bd6740 1 -- 172.21.15.32:0/1240785241 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55d1c6ed7590 msgr2=0x55d1c72b8220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:50:34.987 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:34.985+0000 7feb33bd6740 1 --2- 172.21.15.32:0/1240785241 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55d1c6ed7590 0x55d1c72b8220 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7feb20097520 tx=0x7feb20061660 comp rx=0 tx=0).stop 2024-08-03T20:50:34.987 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:34.985+0000 7feb33bd6740 1 -- 172.21.15.32:0/1240785241 shutdown_connections 2024-08-03T20:50:34.988 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:34.985+0000 7feb33bd6740 1 --2- 172.21.15.32:0/1240785241 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55d1c6ed7590 0x55d1c72b8220 unknown :-1 s=CLOSED pgs=586 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:34.988 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:34.985+0000 7feb33bd6740 1 --2- 172.21.15.32:0/1240785241 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55d1c6ecf3b0 0x55d1c72b7ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:34.988 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:34.985+0000 7feb33bd6740 1 --2- 172.21.15.32:0/1240785241 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55d1c6ef8270 0x55d1c72e2e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:34.988 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:34.985+0000 7feb33bd6740 1 -- 172.21.15.32:0/1240785241 >> 172.21.15.32:0/1240785241 conn(0x55d1c6eec3c0 msgr2=0x55d1c729e180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:50:34.988 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:34.985+0000 7feb33bd6740 1 -- 172.21.15.32:0/1240785241 shutdown_connections 2024-08-03T20:50:34.988 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:34.985+0000 7feb33bd6740 1 -- 172.21.15.32:0/1240785241 wait complete. 2024-08-03T20:50:34.988 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:34.986+0000 7feb33bd6740 1 -- start start 2024-08-03T20:50:34.988 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:34.986+0000 7feb33bd6740 1 Processor -- start 2024-08-03T20:50:34.991 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:34.988+0000 7feb33bd6740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55d1c6ecf3b0 0x55d1c72d8860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:50:34.991 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:34.988+0000 7feb33bd6740 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55d1c6ed7590 0x55d1c72f5770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:50:34.991 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:34.988+0000 7feb33bd6740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55d1c6ef8270 0x55d1c72f5cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:50:34.991 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:34.988+0000 7feb33bd6740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55d1c6f72620 con 0x55d1c6ef8270 2024-08-03T20:50:34.991 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:34.988+0000 7feb33bd6740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x55d1c719d160 con 0x55d1c6ecf3b0 2024-08-03T20:50:34.991 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:34.988+0000 7feb33bd6740 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x55d1c72df350 con 0x55d1c6ed7590 2024-08-03T20:50:34.991 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:34.988+0000 7feb33bd6740 10 client.0 _ll_register_callbacks cb 0x55d1c71ac6b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-03T20:50:34.991 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:34.988+0000 7feb26800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55d1c6ef8270 0x55d1c72f5cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:50:34.991 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:34.988+0000 7feb26800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55d1c6ef8270 0x55d1c72f5cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:45166/0 (socket says 172.21.15.32:45166) 2024-08-03T20:50:34.991 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:34.988+0000 7feb26800640 1 -- 172.21.15.32:0/2099761772 learned_addr learned my addr 172.21.15.32:0/2099761772 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:50:34.992 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:34.989+0000 7feb25400640 1 --2- 172.21.15.32:0/2099761772 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55d1c6ed7590 0x55d1c72f5770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:50:34.992 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:34.989+0000 7feb25e00640 1 --2- 172.21.15.32:0/2099761772 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55d1c6ecf3b0 0x55d1c72d8860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:50:34.992 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:34.989+0000 7feb26800640 1 -- 172.21.15.32:0/2099761772 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55d1c6ed7590 msgr2=0x55d1c72f5770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:50:34.992 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:34.989+0000 7feb26800640 1 --2- 172.21.15.32:0/2099761772 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55d1c6ed7590 0x55d1c72f5770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:34.992 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:34.989+0000 7feb26800640 1 -- 172.21.15.32:0/2099761772 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55d1c6ecf3b0 msgr2=0x55d1c72d8860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:50:34.992 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:34.989+0000 7feb26800640 1 --2- 172.21.15.32:0/2099761772 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55d1c6ecf3b0 0x55d1c72d8860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:34.992 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:34.989+0000 7feb26800640 1 -- 172.21.15.32:0/2099761772 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d1c6fda6b0 con 0x55d1c6ef8270 2024-08-03T20:50:34.993 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:34.989+0000 7feb25e00640 1 --2- 172.21.15.32:0/2099761772 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55d1c6ecf3b0 0x55d1c72d8860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-03T20:50:34.993 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:34.989+0000 7feb26800640 1 --2- 172.21.15.32:0/2099761772 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55d1c6ef8270 0x55d1c72f5cb0 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7feb2005e1b0 tx=0x7feb20097470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e01db94b7e6ca8b3 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:50:34.993 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:34.989+0000 7feb06a00640 10 client.0 ms_handle_connect on v2:172.21.15.32:3300/0 2024-08-03T20:50:34.993 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:34.989+0000 7feb06a00640 1 -- 172.21.15.32:0/2099761772 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb20021070 con 0x55d1c6ef8270 2024-08-03T20:50:34.993 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:34.989+0000 7feb06a00640 1 -- 172.21.15.32:0/2099761772 <== mon.0 v2:172.21.15.32:3300/0 2 ==== osd_map(57..57 src has 1..57) ==== 5709+0+0 (secure 0 0 0) 0x7feb2005dea0 con 0x55d1c6ef8270 2024-08-03T20:50:34.993 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:34.989+0000 7feb33bd6740 1 -- 172.21.15.32:0/2099761772 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7feb20066160 con 0x55d1c6ef8270 2024-08-03T20:50:34.993 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:34.990+0000 7feb06a00640 1 -- 172.21.15.32:0/2099761772 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_get_version_reply(handle=1 version=51) ==== 24+0+0 (secure 0 0 0) 0x7feb20078040 con 0x55d1c6ef8270 2024-08-03T20:50:34.993 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:34.990+0000 7feb33bd6740 10 client.5967 fetch_fsmap learned FSMap version 51 2024-08-03T20:50:34.993 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:34.990+0000 7feb33bd6740 1 -- 172.21.15.32:0/2099761772 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,fsmap=51,monmap=2+}) -- 0x55d1c72b6760 con 0x55d1c6ef8270 2024-08-03T20:50:34.993 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:34.990+0000 7feb06a00640 1 -- 172.21.15.32:0/2099761772 <== mon.0 v2:172.21.15.32:3300/0 4 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7feb20066160 con 0x55d1c6ef8270 2024-08-03T20:50:34.993 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:34.990+0000 7feb06a00640 1 -- 172.21.15.32:0/2099761772 <== mon.0 v2:172.21.15.32:3300/0 5 ==== fsmap(e 51) ==== 3502+0+0 (secure 0 0 0) 0x7feb20068030 con 0x55d1c6ef8270 2024-08-03T20:50:34.994 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:34.990+0000 7feb33bd6740 10 client.5967 fetch_fsmap finished waiting for FSMap version 51 2024-08-03T20:50:34.994 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:34.990+0000 7feb33bd6740 10 client.5967 resolve_mds: resolved name 'c' to daemon mds.c 2024-08-03T20:50:34.994 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:34.990+0000 7feb33bd6740 20 client.5967 populate_metadata read hostname 'smithi032' 2024-08-03T20:50:34.994 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:34.990+0000 7feb33bd6740 1 --2- 172.21.15.32:0/2099761772 >> [v2:172.21.15.32:6834/3848976369,v1:172.21.15.32:6835/3848976369] conn(0x55d1c6eb6e40 0x55d1c72ffbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:50:34.994 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:34.990+0000 7feb33bd6740 4 client.5967 mds_command: new command op to 5841 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-08-03T20:50:34.994 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:34.990+0000 7feb33bd6740 1 -- 172.21.15.32:0/2099761772 --> [v2:172.21.15.32:6834/3848976369,v1:172.21.15.32:6835/3848976369] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d1c72129b0 con 0x55d1c6eb6e40 2024-08-03T20:50:34.994 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:34.990+0000 7feb25e00640 1 --2- 172.21.15.32:0/2099761772 >> [v2:172.21.15.32:6834/3848976369,v1:172.21.15.32:6835/3848976369] conn(0x55d1c6eb6e40 0x55d1c72ffbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:50:34.995 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:34.991+0000 7feb25e00640 1 --2- 172.21.15.32:0/2099761772 >> [v2:172.21.15.32:6834/3848976369,v1:172.21.15.32:6835/3848976369] conn(0x55d1c6eb6e40 0x55d1c72ffbf0 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3b603a95e7cc7190 server_cookie=2d7d672cd20c8bf8 in_seq=0 out_seq=0 2024-08-03T20:50:34.995 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:34.991+0000 7feb06a00640 10 client.5967 ms_handle_connect on v2:172.21.15.32:6834/3848976369 2024-08-03T20:50:34.995 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:34.992+0000 7feb06a00640 1 -- 172.21.15.32:0/2099761772 <== mds.0 v2:172.21.15.32:6834/3848976369 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55d1c72129b0 con 0x55d1c6eb6e40 2024-08-03T20:50:34.995 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:34.992+0000 7feb06a00640 10 client.5967 handle_command_reply: tid=0 2024-08-03T20:50:34.995 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:34.992+0000 7feb33bd6740 1 client.5967 shutdown 2024-08-03T20:50:34.996 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:34.992+0000 7feb33bd6740 1 -- 172.21.15.32:0/2099761772 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55d1c6ef8270 msgr2=0x55d1c72f5cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:50:34.996 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:34.992+0000 7feb33bd6740 1 --2- 172.21.15.32:0/2099761772 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55d1c6ef8270 0x55d1c72f5cb0 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7feb2005e1b0 tx=0x7feb20097470 comp rx=0 tx=0).stop 2024-08-03T20:50:34.996 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:34.992+0000 7feb33bd6740 1 -- 172.21.15.32:0/2099761772 shutdown_connections 2024-08-03T20:50:34.996 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:34.992+0000 7feb33bd6740 1 --2- 172.21.15.32:0/2099761772 >> [v2:172.21.15.32:6834/3848976369,v1:172.21.15.32:6835/3848976369] conn(0x55d1c6eb6e40 0x55d1c72ffbf0 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:34.996 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:34.992+0000 7feb33bd6740 1 --2- 172.21.15.32:0/2099761772 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55d1c6ef8270 0x55d1c72f5cb0 unknown :-1 s=CLOSED pgs=587 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:34.996 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:34.992+0000 7feb33bd6740 1 --2- 172.21.15.32:0/2099761772 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55d1c6ed7590 0x55d1c72f5770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:34.996 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:34.992+0000 7feb33bd6740 1 --2- 172.21.15.32:0/2099761772 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55d1c6ecf3b0 0x55d1c72d8860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:34.996 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:34.992+0000 7feb06a00640 0 client.5967 ms_handle_reset on v2:172.21.15.32:6834/3848976369 2024-08-03T20:50:34.996 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:34.992+0000 7feb33bd6740 1 -- 172.21.15.32:0/2099761772 >> 172.21.15.32:0/2099761772 conn(0x55d1c6eec3c0 msgr2=0x55d1c72e5b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:50:34.996 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:34.992+0000 7feb33bd6740 1 -- 172.21.15.32:0/2099761772 shutdown_connections 2024-08-03T20:50:34.996 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:34.992+0000 7feb33bd6740 1 -- 172.21.15.32:0/2099761772 wait complete. 2024-08-03T20:50:34.996 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:34.993+0000 7feb33bd6740 20 client.5967 trim_cache size 0 max 16384 2024-08-03T20:50:35.013 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:35.013+0000 7feb33bd6740 1 Processor -- start 2024-08-03T20:50:35.013 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:35.013+0000 7feb33bd6740 1 -- start start 2024-08-03T20:50:35.016 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:35.014+0000 7feb33bd6740 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55d1c6ecf3b0 0x55d1c72e27f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:50:35.016 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:35.014+0000 7feb33bd6740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55d1c6ed7590 0x55d1c729dde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:50:35.016 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:35.014+0000 7feb33bd6740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55d1c6ef8270 0x55d1c729e320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:50:35.016 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:35.014+0000 7feb33bd6740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7feb282021e0 con 0x55d1c6ef8270 2024-08-03T20:50:35.016 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:35.014+0000 7feb33bd6740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7feb28201240 con 0x55d1c6ed7590 2024-08-03T20:50:35.016 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:35.014+0000 7feb33bd6740 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7feb281f9280 con 0x55d1c6ecf3b0 2024-08-03T20:50:35.016 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:35.014+0000 7feb26800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55d1c6ef8270 0x55d1c729e320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:50:35.016 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:35.014+0000 7feb26800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55d1c6ef8270 0x55d1c729e320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:45174/0 (socket says 172.21.15.32:45174) 2024-08-03T20:50:35.016 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:35.014+0000 7feb26800640 1 -- 172.21.15.32:0/4146348693 learned_addr learned my addr 172.21.15.32:0/4146348693 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:50:35.016 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:35.015+0000 7feb25e00640 1 --2- 172.21.15.32:0/4146348693 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55d1c6ecf3b0 0x55d1c72e27f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:50:35.016 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:35.015+0000 7feb25400640 1 --2- 172.21.15.32:0/4146348693 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55d1c6ed7590 0x55d1c729dde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:50:35.016 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:35.015+0000 7feb26800640 1 -- 172.21.15.32:0/4146348693 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55d1c6ecf3b0 msgr2=0x55d1c72e27f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:50:35.017 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:35.015+0000 7feb26800640 1 --2- 172.21.15.32:0/4146348693 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55d1c6ecf3b0 0x55d1c72e27f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:35.017 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:35.015+0000 7feb26800640 1 -- 172.21.15.32:0/4146348693 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55d1c6ed7590 msgr2=0x55d1c729dde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:50:35.017 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:35.015+0000 7feb26800640 1 --2- 172.21.15.32:0/4146348693 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55d1c6ed7590 0x55d1c729dde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:35.017 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:35.015+0000 7feb26800640 1 -- 172.21.15.32:0/4146348693 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d1c729b910 con 0x55d1c6ef8270 2024-08-03T20:50:35.017 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:35.015+0000 7feb26800640 1 --2- 172.21.15.32:0/4146348693 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55d1c6ef8270 0x55d1c729e320 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7feb2005ee80 tx=0x7feb20003f20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bdfcd4ddcc05f23e server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:50:35.017 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:35.015+0000 7feb11c00640 1 -- 172.21.15.32:0/4146348693 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb20021040 con 0x55d1c6ef8270 2024-08-03T20:50:35.017 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:35.015+0000 7feb11c00640 1 -- 172.21.15.32:0/4146348693 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7feb200972b0 con 0x55d1c6ef8270 2024-08-03T20:50:35.017 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:35.015+0000 7feb11c00640 1 -- 172.21.15.32:0/4146348693 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb200611a0 con 0x55d1c6ef8270 2024-08-03T20:50:35.018 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:35.016+0000 7feb33bd6740 1 -- 172.21.15.32:0/4146348693 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55d1c6ef8270 msgr2=0x55d1c729e320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:50:35.018 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:35.016+0000 7feb33bd6740 1 --2- 172.21.15.32:0/4146348693 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55d1c6ef8270 0x55d1c729e320 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7feb2005ee80 tx=0x7feb20003f20 comp rx=0 tx=0).stop 2024-08-03T20:50:35.018 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:35.016+0000 7feb33bd6740 1 -- 172.21.15.32:0/4146348693 shutdown_connections 2024-08-03T20:50:35.018 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:35.016+0000 7feb33bd6740 1 --2- 172.21.15.32:0/4146348693 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55d1c6ed7590 0x55d1c729dde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:35.018 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:35.016+0000 7feb33bd6740 1 --2- 172.21.15.32:0/4146348693 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55d1c6ef8270 0x55d1c729e320 unknown :-1 s=CLOSED pgs=588 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:35.018 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:35.016+0000 7feb33bd6740 1 --2- 172.21.15.32:0/4146348693 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55d1c6ecf3b0 0x55d1c72e27f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:35.018 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:35.016+0000 7feb33bd6740 1 -- 172.21.15.32:0/4146348693 >> 172.21.15.32:0/4146348693 conn(0x55d1c6eec3c0 msgr2=0x55d1c72b8210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:50:35.019 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:35.016+0000 7feb33bd6740 1 -- 172.21.15.32:0/4146348693 shutdown_connections 2024-08-03T20:50:35.019 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:35.016+0000 7feb33bd6740 1 -- 172.21.15.32:0/4146348693 wait complete. 2024-08-03T20:50:35.019 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:35.017+0000 7feb33bd6740 1 -- start start 2024-08-03T20:50:35.019 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:35.017+0000 7feb33bd6740 1 Processor -- start 2024-08-03T20:50:35.023 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:35.019+0000 7feb33bd6740 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55d1c6ecf3b0 0x55d1c7355f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:50:35.023 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:35.019+0000 7feb33bd6740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55d1c6ed7590 0x55d1c72d8440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:50:35.023 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:35.019+0000 7feb33bd6740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55d1c6ef8270 0x55d1c72d8980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:50:35.023 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:35.019+0000 7feb33bd6740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55d1c6f72620 con 0x55d1c6ef8270 2024-08-03T20:50:35.023 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:35.019+0000 7feb33bd6740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7feb28200bf0 con 0x55d1c6ed7590 2024-08-03T20:50:35.023 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:35.019+0000 7feb33bd6740 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x55d1c72e0370 con 0x55d1c6ecf3b0 2024-08-03T20:50:35.023 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:35.019+0000 7feb33bd6740 10 client.0 _ll_register_callbacks cb 0x7feb20066050 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-03T20:50:35.023 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:35.019+0000 7feb26800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55d1c6ef8270 0x55d1c72d8980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:50:35.023 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:35.019+0000 7feb26800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55d1c6ef8270 0x55d1c72d8980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:45176/0 (socket says 172.21.15.32:45176) 2024-08-03T20:50:35.023 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:35.019+0000 7feb26800640 1 -- 172.21.15.32:0/3855938680 learned_addr learned my addr 172.21.15.32:0/3855938680 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:50:35.023 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:35.019+0000 7feb26800640 1 -- 172.21.15.32:0/3855938680 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55d1c6ecf3b0 msgr2=0x55d1c7355f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:50:35.024 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:35.019+0000 7feb26800640 1 --2- 172.21.15.32:0/3855938680 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55d1c6ecf3b0 0x55d1c7355f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:35.024 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:35.019+0000 7feb26800640 1 -- 172.21.15.32:0/3855938680 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55d1c6ed7590 msgr2=0x55d1c72d8440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:50:35.024 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:35.020+0000 7feb25400640 1 --2- 172.21.15.32:0/3855938680 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55d1c6ed7590 0x55d1c72d8440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:50:35.024 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:35.020+0000 7feb26800640 1 --2- 172.21.15.32:0/3855938680 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55d1c6ed7590 0x55d1c72d8440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:35.024 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:35.020+0000 7feb26800640 1 -- 172.21.15.32:0/3855938680 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d1c72f57b0 con 0x55d1c6ef8270 2024-08-03T20:50:35.024 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:35.020+0000 7feb25400640 1 --2- 172.21.15.32:0/3855938680 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55d1c6ed7590 0x55d1c72d8440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-03T20:50:35.024 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:35.020+0000 7feb26800640 1 --2- 172.21.15.32:0/3855938680 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55d1c6ef8270 0x55d1c72d8980 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7feb20000f80 tx=0x7feb200685a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=99b43fb22ab52a4e server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:50:35.024 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:35.020+0000 7feb06a00640 10 client.0 ms_handle_connect on v2:172.21.15.32:3300/0 2024-08-03T20:50:35.024 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:35.020+0000 7feb06a00640 1 -- 172.21.15.32:0/3855938680 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb20021070 con 0x55d1c6ef8270 2024-08-03T20:50:35.024 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:35.020+0000 7feb06a00640 1 -- 172.21.15.32:0/3855938680 <== mon.0 v2:172.21.15.32:3300/0 2 ==== osd_map(57..57 src has 1..57) ==== 5709+0+0 (secure 0 0 0) 0x7feb2005dea0 con 0x55d1c6ef8270 2024-08-03T20:50:35.024 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:35.020+0000 7feb33bd6740 1 -- 172.21.15.32:0/3855938680 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55d1c720e450 con 0x55d1c6ef8270 2024-08-03T20:50:35.024 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:35.020+0000 7feb06a00640 1 -- 172.21.15.32:0/3855938680 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_get_version_reply(handle=1 version=51) ==== 24+0+0 (secure 0 0 0) 0x7feb200611a0 con 0x55d1c6ef8270 2024-08-03T20:50:35.024 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:35.020+0000 7feb33bd6740 10 client.5973 fetch_fsmap learned FSMap version 51 2024-08-03T20:50:35.025 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:35.020+0000 7feb33bd6740 1 -- 172.21.15.32:0/3855938680 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,fsmap=51,monmap=2+}) -- 0x55d1c72f57b0 con 0x55d1c6ef8270 2024-08-03T20:50:35.025 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:35.021+0000 7feb06a00640 1 -- 172.21.15.32:0/3855938680 <== mon.0 v2:172.21.15.32:3300/0 4 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x55d1c720e450 con 0x55d1c6ef8270 2024-08-03T20:50:35.025 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:35.021+0000 7feb06a00640 1 -- 172.21.15.32:0/3855938680 <== mon.0 v2:172.21.15.32:3300/0 5 ==== fsmap(e 51) ==== 3502+0+0 (secure 0 0 0) 0x7feb20068050 con 0x55d1c6ef8270 2024-08-03T20:50:35.025 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:35.021+0000 7feb33bd6740 10 client.5973 fetch_fsmap finished waiting for FSMap version 51 2024-08-03T20:50:35.025 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:35.021+0000 7feb33bd6740 10 client.5973 resolve_mds: resolved name 'c' to daemon mds.c 2024-08-03T20:50:35.025 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:35.021+0000 7feb33bd6740 20 client.5973 populate_metadata read hostname 'smithi032' 2024-08-03T20:50:35.025 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:35.021+0000 7feb33bd6740 1 --2- 172.21.15.32:0/3855938680 >> [v2:172.21.15.32:6834/3848976369,v1:172.21.15.32:6835/3848976369] conn(0x55d1c72b43d0 0x55d1c72b47e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:50:35.025 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:35.021+0000 7feb33bd6740 4 client.5973 mds_command: new command op to 5841 tid=0 multi_id=0 [{"prefix": "status"}] 2024-08-03T20:50:35.025 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:35.021+0000 7feb33bd6740 1 -- 172.21.15.32:0/3855938680 --> [v2:172.21.15.32:6834/3848976369,v1:172.21.15.32:6835/3848976369] -- command(tid 0: {"prefix": "status"}) -- 0x55d1c720bb90 con 0x55d1c72b43d0 2024-08-03T20:50:35.025 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:35.021+0000 7feb25e00640 1 --2- 172.21.15.32:0/3855938680 >> [v2:172.21.15.32:6834/3848976369,v1:172.21.15.32:6835/3848976369] conn(0x55d1c72b43d0 0x55d1c72b47e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:50:35.025 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:35.021+0000 7feb25e00640 1 --2- 172.21.15.32:0/3855938680 >> [v2:172.21.15.32:6834/3848976369,v1:172.21.15.32:6835/3848976369] conn(0x55d1c72b43d0 0x55d1c72b47e0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=859c96c4406c3540 server_cookie=b4e0b2124415b793 in_seq=0 out_seq=0 2024-08-03T20:50:35.025 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:35.021+0000 7feb06a00640 10 client.5973 ms_handle_connect on v2:172.21.15.32:6834/3848976369 2024-08-03T20:50:35.025 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:35.022+0000 7feb06a00640 1 -- 172.21.15.32:0/3855938680 <== mds.0 v2:172.21.15.32:6834/3848976369 1 ==== command_reply(tid 0: 0 ) ==== 8+0+325 (crc 0 0 0) 0x55d1c720bb90 con 0x55d1c72b43d0 2024-08-03T20:50:35.025 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:35.022+0000 7feb06a00640 10 client.5973 handle_command_reply: tid=0 2024-08-03T20:50:35.026 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:35.022+0000 7feb33bd6740 1 client.5973 shutdown 2024-08-03T20:50:35.026 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:35.022+0000 7feb33bd6740 1 -- 172.21.15.32:0/3855938680 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55d1c6ef8270 msgr2=0x55d1c72d8980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:50:35.026 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:35.022+0000 7feb33bd6740 1 --2- 172.21.15.32:0/3855938680 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55d1c6ef8270 0x55d1c72d8980 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7feb20000f80 tx=0x7feb200685a0 comp rx=0 tx=0).stop 2024-08-03T20:50:35.026 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:35.022+0000 7feb33bd6740 1 -- 172.21.15.32:0/3855938680 shutdown_connections 2024-08-03T20:50:35.026 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:35.022+0000 7feb33bd6740 1 --2- 172.21.15.32:0/3855938680 >> [v2:172.21.15.32:6834/3848976369,v1:172.21.15.32:6835/3848976369] conn(0x55d1c72b43d0 0x55d1c72b47e0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:35.026 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:35.022+0000 7feb33bd6740 1 --2- 172.21.15.32:0/3855938680 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55d1c6ed7590 0x55d1c72d8440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:35.026 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:35.022+0000 7feb33bd6740 1 --2- 172.21.15.32:0/3855938680 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55d1c6ef8270 0x55d1c72d8980 unknown :-1 s=CLOSED pgs=589 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:35.026 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:35.022+0000 7feb33bd6740 1 --2- 172.21.15.32:0/3855938680 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55d1c6ecf3b0 0x55d1c7355f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:35.026 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:35.022+0000 7feb06a00640 0 client.5973 ms_handle_reset on v2:172.21.15.32:6834/3848976369 2024-08-03T20:50:35.026 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:35.022+0000 7feb33bd6740 1 -- 172.21.15.32:0/3855938680 >> 172.21.15.32:0/3855938680 conn(0x55d1c6eec3c0 msgr2=0x55d1c72e27f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:50:35.026 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:35.022+0000 7feb33bd6740 1 -- 172.21.15.32:0/3855938680 shutdown_connections 2024-08-03T20:50:35.026 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:35.022+0000 7feb33bd6740 1 -- 172.21.15.32:0/3855938680 wait complete. 2024-08-03T20:50:35.026 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:35.022+0000 7feb33bd6740 20 client.5973 trim_cache size 0 max 16384 2024-08-03T20:50:35.027 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-08-03T20:50:35.027 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_fsid": "40ab141a-01d7-49b9-ae3c-2edec9876e95", 2024-08-03T20:50:35.027 INFO:teuthology.orchestra.run.smithi032.stdout: "whoami": 0, 2024-08-03T20:50:35.027 INFO:teuthology.orchestra.run.smithi032.stdout: "id": 5841, 2024-08-03T20:50:35.027 INFO:teuthology.orchestra.run.smithi032.stdout: "want_state": "up:active", 2024-08-03T20:50:35.027 INFO:teuthology.orchestra.run.smithi032.stdout: "state": "up:active", 2024-08-03T20:50:35.027 INFO:teuthology.orchestra.run.smithi032.stdout: "fs_name": "cephfs", 2024-08-03T20:50:35.027 INFO:teuthology.orchestra.run.smithi032.stdout: "rank_uptime": 6.8679713549999999, 2024-08-03T20:50:35.027 INFO:teuthology.orchestra.run.smithi032.stdout: "mdsmap_epoch": 51, 2024-08-03T20:50:35.028 INFO:teuthology.orchestra.run.smithi032.stdout: "osdmap_epoch": 57, 2024-08-03T20:50:35.028 INFO:teuthology.orchestra.run.smithi032.stdout: "osdmap_epoch_barrier": 57, 2024-08-03T20:50:35.028 INFO:teuthology.orchestra.run.smithi032.stdout: "uptime": 11.830831404 2024-08-03T20:50:35.028 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-08-03T20:50:35.028 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:35.023+0000 7feb2ca00640 1 -- 172.21.15.32:0/1637709586 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7feb08077010 msgr2=0x7feb08079520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:50:35.029 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:35.023+0000 7feb2ca00640 1 --2- 172.21.15.32:0/1637709586 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7feb08077010 0x7feb08079520 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7feb18001710 tx=0x7feb18001270 comp rx=0 tx=0).stop 2024-08-03T20:50:35.029 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:35.023+0000 7feb2ca00640 1 -- 172.21.15.32:0/1637709586 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7feb2815aba0 msgr2=0x7feb281599c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:50:35.029 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:35.023+0000 7feb2ca00640 1 --2- 172.21.15.32:0/1637709586 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7feb2815aba0 0x7feb281599c0 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7feb2815c780 tx=0x7feb20010f40 comp rx=0 tx=0).stop 2024-08-03T20:50:35.029 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:35.023+0000 7feb2ca00640 1 -- 172.21.15.32:0/1637709586 shutdown_connections 2024-08-03T20:50:35.029 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:35.023+0000 7feb2ca00640 1 --2- 172.21.15.32:0/1637709586 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7feb08077010 0x7feb08079520 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:35.029 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:35.023+0000 7feb2ca00640 1 --2- 172.21.15.32:0/1637709586 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7feb2815aba0 0x7feb281599c0 unknown :-1 s=CLOSED pgs=585 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:35.029 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:35.023+0000 7feb2ca00640 1 --2- 172.21.15.32:0/1637709586 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7feb281f4f00 0x7feb28150c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:35.029 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:35.023+0000 7feb2ca00640 1 --2- 172.21.15.32:0/1637709586 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7feb0c0039e0 0x7feb28154200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:35.029 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:35.023+0000 7feb2ca00640 1 -- 172.21.15.32:0/1637709586 >> 172.21.15.32:0/1637709586 conn(0x7feb280b1bd0 msgr2=0x7feb280b12d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:50:35.029 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:35.023+0000 7feb2ca00640 1 -- 172.21.15.32:0/1637709586 shutdown_connections 2024-08-03T20:50:35.029 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:35.023+0000 7feb2ca00640 1 -- 172.21.15.32:0/1637709586 wait complete. 2024-08-03T20:50:35.045 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-08-03T20:50:35.276 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:35.274+0000 7f1d82c00640 1 -- 172.21.15.32:0/755971607 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1d7c0d4be0 msgr2=0x7f1d7c0b5460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:50:35.276 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:35.274+0000 7f1d82c00640 1 --2- 172.21.15.32:0/755971607 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1d7c0d4be0 0x7f1d7c0b5460 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7f1d7c0cb120 tx=0x7f1d7002a4b0 comp rx=0 tx=0).stop 2024-08-03T20:50:35.276 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:35.275+0000 7f1d82c00640 1 -- 172.21.15.32:0/755971607 shutdown_connections 2024-08-03T20:50:35.276 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:35.275+0000 7f1d82c00640 1 --2- 172.21.15.32:0/755971607 >> v2:172.21.15.118:3300/0 conn(0x7f1d7c0d11c0 0x7f1d7c0b73a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:35.276 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:35.275+0000 7f1d82c00640 1 --2- 172.21.15.32:0/755971607 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1d7c0d4be0 0x7f1d7c0b5460 unknown :-1 s=CLOSED pgs=591 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:35.276 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:35.275+0000 7f1d82c00640 1 -- 172.21.15.32:0/755971607 >> 172.21.15.32:0/755971607 conn(0x7f1d7c0d40c0 msgr2=0x7f1d7c0d44d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:50:35.276 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:35.275+0000 7f1d82c00640 1 -- 172.21.15.32:0/755971607 shutdown_connections 2024-08-03T20:50:35.276 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:35.275+0000 7f1d82c00640 1 -- 172.21.15.32:0/755971607 wait complete. 2024-08-03T20:50:35.277 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:35.277+0000 7f1d82c00640 1 Processor -- start 2024-08-03T20:50:35.277 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:35.277+0000 7f1d82c00640 1 -- start start 2024-08-03T20:50:35.279 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:35.278+0000 7f1d82c00640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f1d7c0b94f0 0x7f1d7c1e8740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:50:35.279 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:35.278+0000 7f1d82c00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f1d7c0d4be0 0x7f1d7c0cd5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:50:35.279 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:35.278+0000 7f1d82c00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1d7c1609b0 0x7f1d7c160dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:50:35.279 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:35.278+0000 7f1d82c00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f1d7c0b7c70 con 0x7f1d7c1609b0 2024-08-03T20:50:35.279 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:35.278+0000 7f1d82c00640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f1d7c0b7af0 con 0x7f1d7c0d4be0 2024-08-03T20:50:35.279 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:35.278+0000 7f1d82c00640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7f1d7c0b7df0 con 0x7f1d7c0b94f0 2024-08-03T20:50:35.280 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:35.279+0000 7f1d82200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1d7c1609b0 0x7f1d7c160dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:50:35.280 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:35.279+0000 7f1d82200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1d7c1609b0 0x7f1d7c160dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:45194/0 (socket says 172.21.15.32:45194) 2024-08-03T20:50:35.280 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:35.279+0000 7f1d82200640 1 -- 172.21.15.32:0/1979822552 learned_addr learned my addr 172.21.15.32:0/1979822552 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:50:35.280 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:35.279+0000 7f1d81800640 1 --2- 172.21.15.32:0/1979822552 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f1d7c0b94f0 0x7f1d7c1e8740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:50:35.280 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:35.279+0000 7f1d80e00640 1 --2- 172.21.15.32:0/1979822552 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f1d7c0d4be0 0x7f1d7c0cd5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:50:35.281 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:35.279+0000 7f1d82200640 1 -- 172.21.15.32:0/1979822552 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f1d7c0b94f0 msgr2=0x7f1d7c1e8740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:50:35.281 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:35.279+0000 7f1d82200640 1 --2- 172.21.15.32:0/1979822552 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f1d7c0b94f0 0x7f1d7c1e8740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:35.281 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:35.279+0000 7f1d82200640 1 -- 172.21.15.32:0/1979822552 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f1d7c0d4be0 msgr2=0x7f1d7c0cd5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:50:35.281 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:35.279+0000 7f1d82200640 1 --2- 172.21.15.32:0/1979822552 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f1d7c0d4be0 0x7f1d7c0cd5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:35.281 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:35.279+0000 7f1d82200640 1 -- 172.21.15.32:0/1979822552 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1d7c161300 con 0x7f1d7c1609b0 2024-08-03T20:50:35.281 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:35.280+0000 7f1d82200640 1 --2- 172.21.15.32:0/1979822552 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1d7c1609b0 0x7f1d7c160dc0 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7f1d7400ba10 tx=0x7f1d7400bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:50:35.281 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:35.280+0000 7f1d79600640 1 -- 172.21.15.32:0/1979822552 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d74019070 con 0x7f1d7c1609b0 2024-08-03T20:50:35.282 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:35.280+0000 7f1d79600640 1 -- 172.21.15.32:0/1979822552 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f1d7400cc10 con 0x7f1d7c1609b0 2024-08-03T20:50:35.282 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:35.280+0000 7f1d82c00640 1 -- 172.21.15.32:0/1979822552 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1d7c0cdae0 con 0x7f1d7c1609b0 2024-08-03T20:50:35.282 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:35.280+0000 7f1d79600640 1 -- 172.21.15.32:0/1979822552 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d74014440 con 0x7f1d7c1609b0 2024-08-03T20:50:35.282 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:35.280+0000 7f1d82c00640 1 -- 172.21.15.32:0/1979822552 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1d7c0b21a0 con 0x7f1d7c1609b0 2024-08-03T20:50:35.282 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:35.281+0000 7f1d82c00640 1 -- 172.21.15.32:0/1979822552 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1d7c0b73a0 con 0x7f1d7c1609b0 2024-08-03T20:50:35.283 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:35.282+0000 7f1d79600640 1 -- 172.21.15.32:0/1979822552 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f1d7401c070 con 0x7f1d7c1609b0 2024-08-03T20:50:35.284 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:35.283+0000 7f1d79600640 1 --2- 172.21.15.32:0/1979822552 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f1d5c077040 0x7f1d5c079550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:50:35.284 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:35.283+0000 7f1d79600640 1 -- 172.21.15.32:0/1979822552 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(57..57 src has 1..57) ==== 5709+0+0 (secure 0 0 0) 0x7f1d740985e0 con 0x7f1d7c1609b0 2024-08-03T20:50:35.284 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:35.284+0000 7f1d81800640 1 --2- 172.21.15.32:0/1979822552 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f1d5c077040 0x7f1d5c079550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:50:35.285 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:35.285+0000 7f1d81800640 1 --2- 172.21.15.32:0/1979822552 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f1d5c077040 0x7f1d5c079550 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f1d70034000 tx=0x7f1d70004730 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:50:35.288 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:35.288+0000 7f1d79600640 1 -- 172.21.15.32:0/1979822552 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7f1d7c161300 con 0x7f1d7c1609b0 2024-08-03T20:50:35.531 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:35.531+0000 7f1d82c00640 1 -- 172.21.15.32:0/1979822552 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f1d7c161ba0 con 0x7f1d7c1609b0 2024-08-03T20:50:35.532 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:35.531+0000 7f1d79600640 1 -- 172.21.15.32:0/1979822552 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v51) ==== 53+0+73 (secure 0 0 0) 0x7f1d74061180 con 0x7f1d7c1609b0 2024-08-03T20:50:35.532 INFO:teuthology.orchestra.run.smithi032.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-08-03T20:50:35.534 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:35.534+0000 7f1d82c00640 1 -- 172.21.15.32:0/1979822552 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f1d5c077040 msgr2=0x7f1d5c079550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:50:35.534 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:35.534+0000 7f1d82c00640 1 --2- 172.21.15.32:0/1979822552 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f1d5c077040 0x7f1d5c079550 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f1d70034000 tx=0x7f1d70004730 comp rx=0 tx=0).stop 2024-08-03T20:50:35.535 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:35.534+0000 7f1d82c00640 1 -- 172.21.15.32:0/1979822552 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1d7c1609b0 msgr2=0x7f1d7c160dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:50:35.535 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:35.534+0000 7f1d82c00640 1 --2- 172.21.15.32:0/1979822552 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1d7c1609b0 0x7f1d7c160dc0 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7f1d7400ba10 tx=0x7f1d7400bed0 comp rx=0 tx=0).stop 2024-08-03T20:50:35.535 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:35.534+0000 7f1d82c00640 1 -- 172.21.15.32:0/1979822552 shutdown_connections 2024-08-03T20:50:35.535 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:35.534+0000 7f1d82c00640 1 --2- 172.21.15.32:0/1979822552 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f1d5c077040 0x7f1d5c079550 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:35.535 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:35.534+0000 7f1d82c00640 1 --2- 172.21.15.32:0/1979822552 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f1d7c0d4be0 0x7f1d7c0cd5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:35.535 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:35.534+0000 7f1d82c00640 1 --2- 172.21.15.32:0/1979822552 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1d7c1609b0 0x7f1d7c160dc0 unknown :-1 s=CLOSED pgs=592 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:35.535 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:35.534+0000 7f1d82c00640 1 --2- 172.21.15.32:0/1979822552 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f1d7c0b94f0 0x7f1d7c1e8740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:35.535 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:35.534+0000 7f1d82c00640 1 -- 172.21.15.32:0/1979822552 >> 172.21.15.32:0/1979822552 conn(0x7f1d7c0d40c0 msgr2=0x7f1d7c1eaa80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:50:35.535 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:35.534+0000 7f1d82c00640 1 -- 172.21.15.32:0/1979822552 shutdown_connections 2024-08-03T20:50:35.535 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:35.534+0000 7f1d82c00640 1 -- 172.21.15.32:0/1979822552 wait complete. 2024-08-03T20:50:35.558 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-08-03T20:50:35.558 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-08-03T20:50:35.558 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi032.front.sepia.ceph.com 2024-08-03T20:50:35.558 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-08-03T20:50:35.558 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-08-03T20:50:35.558 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-08-03T20:50:35.558 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2024-08-03T20:50:35.558 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-08-03T20:50:35.558 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-08-03T20:50:35.559 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-08-03T20:50:35.559 DEBUG:teuthology.orchestra.run.smithi032:> ip addr 2024-08-03T20:50:35.595 INFO:teuthology.orchestra.run.smithi032.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-08-03T20:50:35.595 INFO:teuthology.orchestra.run.smithi032.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-08-03T20:50:35.595 INFO:teuthology.orchestra.run.smithi032.stdout: inet 127.0.0.1/8 scope host lo 2024-08-03T20:50:35.596 INFO:teuthology.orchestra.run.smithi032.stdout: valid_lft forever preferred_lft forever 2024-08-03T20:50:35.596 INFO:teuthology.orchestra.run.smithi032.stdout: inet6 ::1/128 scope host 2024-08-03T20:50:35.596 INFO:teuthology.orchestra.run.smithi032.stdout: valid_lft forever preferred_lft forever 2024-08-03T20:50:35.596 INFO:teuthology.orchestra.run.smithi032.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-08-03T20:50:35.596 INFO:teuthology.orchestra.run.smithi032.stdout: link/ether 0c:c4:7a:6b:fd:0c brd ff:ff:ff:ff:ff:ff 2024-08-03T20:50:35.596 INFO:teuthology.orchestra.run.smithi032.stdout: altname enp7s0f0 2024-08-03T20:50:35.596 INFO:teuthology.orchestra.run.smithi032.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-08-03T20:50:35.596 INFO:teuthology.orchestra.run.smithi032.stdout: link/ether 0c:c4:7a:6b:fd:0d brd ff:ff:ff:ff:ff:ff 2024-08-03T20:50:35.596 INFO:teuthology.orchestra.run.smithi032.stdout: altname enp7s0f1 2024-08-03T20:50:35.596 INFO:teuthology.orchestra.run.smithi032.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-08-03T20:50:35.596 INFO:teuthology.orchestra.run.smithi032.stdout: link/ether ac:1f:6b:f8:26:02 brd ff:ff:ff:ff:ff:ff 2024-08-03T20:50:35.596 INFO:teuthology.orchestra.run.smithi032.stdout: altname enp1s0f0 2024-08-03T20:50:35.596 INFO:teuthology.orchestra.run.smithi032.stdout: inet 172.21.15.32/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-08-03T20:50:35.597 INFO:teuthology.orchestra.run.smithi032.stdout: valid_lft 42320sec preferred_lft 42320sec 2024-08-03T20:50:35.597 INFO:teuthology.orchestra.run.smithi032.stdout: inet6 fe80::ae1f:6bff:fef8:2602/64 scope link noprefixroute 2024-08-03T20:50:35.597 INFO:teuthology.orchestra.run.smithi032.stdout: valid_lft forever preferred_lft forever 2024-08-03T20:50:35.597 INFO:teuthology.orchestra.run.smithi032.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-08-03T20:50:35.597 INFO:teuthology.orchestra.run.smithi032.stdout: link/ether ac:1f:6b:f8:26:03 brd ff:ff:ff:ff:ff:ff 2024-08-03T20:50:35.597 INFO:teuthology.orchestra.run.smithi032.stdout: altname enp1s0f1 2024-08-03T20:50:35.597 INFO:teuthology.orchestra.run.smithi032.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-08-03T20:50:35.597 INFO:teuthology.orchestra.run.smithi032.stdout: link/ether 0a:23:2b:ca:2a:93 brd ff:ff:ff:ff:ff:ff 2024-08-03T20:50:35.597 INFO:teuthology.orchestra.run.smithi032.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-08-03T20:50:35.597 INFO:teuthology.orchestra.run.smithi032.stdout: valid_lft forever preferred_lft forever 2024-08-03T20:50:35.597 INFO:teuthology.orchestra.run.smithi032.stdout: inet6 fe80::d804:7cff:fed9:7ba4/64 scope link 2024-08-03T20:50:35.597 INFO:teuthology.orchestra.run.smithi032.stdout: valid_lft forever preferred_lft forever 2024-08-03T20:50:35.597 INFO:teuthology.orchestra.run.smithi032.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-08-03T20:50:35.597 INFO:teuthology.orchestra.run.smithi032.stdout: link/ether 22:86:28:30:4e:62 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-08-03T20:50:35.598 INFO:teuthology.orchestra.run.smithi032.stdout: inet6 fe80::2086:28ff:fe30:4e62/64 scope link 2024-08-03T20:50:35.598 INFO:teuthology.orchestra.run.smithi032.stdout: valid_lft forever preferred_lft forever 2024-08-03T20:50:35.598 INFO:teuthology.orchestra.run.smithi032.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-08-03T20:50:35.598 INFO:teuthology.orchestra.run.smithi032.stdout: link/ether 0a:23:2b:ca:2a:93 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-08-03T20:50:35.598 INFO:teuthology.orchestra.run.smithi032.stdout: inet6 fe80::823:2bff:feca:2a93/64 scope link 2024-08-03T20:50:35.598 INFO:teuthology.orchestra.run.smithi032.stdout: valid_lft forever preferred_lft forever 2024-08-03T20:50:35.598 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-08-03T20:50:35.598 DEBUG:teuthology.orchestra.run.smithi032:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-08-03T20:50:35.689 INFO:teuthology.orchestra.run.smithi032.stdout:1 2024-08-03T20:50:35.692 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-08-03T20:50:35.692 DEBUG:teuthology.orchestra.run.smithi032:> route 2024-08-03T20:50:35.729 INFO:teuthology.orchestra.run.smithi032.stdout:Kernel IP routing table 2024-08-03T20:50:35.729 INFO:teuthology.orchestra.run.smithi032.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-08-03T20:50:35.729 INFO:teuthology.orchestra.run.smithi032.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-08-03T20:50:35.729 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-08-03T20:50:35.729 INFO:teuthology.orchestra.run.smithi032.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-08-03T20:50:35.729 DEBUG:teuthology.orchestra.run.smithi032:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-08-03T20:50:35.730 DEBUG:teuthology.orchestra.run.smithi032:> set -e 2024-08-03T20:50:35.730 DEBUG:teuthology.orchestra.run.smithi032:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-08-03T20:50:35.730 DEBUG:teuthology.orchestra.run.smithi032:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-08-03T20:50:35.730 DEBUG:teuthology.orchestra.run.smithi032:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-08-03T20:50:35.730 DEBUG:teuthology.orchestra.run.smithi032:> ') 2024-08-03T20:50:35.848 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:35 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-08-03T20:50:35.919 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:35 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-08-03T20:50:35.928 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-08-03T20:50:35.928 DEBUG:teuthology.orchestra.run.smithi032:> ip netns list 2024-08-03T20:50:35.963 INFO:teuthology.orchestra.run.smithi032.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-08-03T20:50:35.963 INFO:teuthology.orchestra.run.smithi032.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-08-03T20:50:35.964 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-08-03T20:50:35.964 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-08-03T20:50:35.964 DEBUG:teuthology.orchestra.run.smithi032:> sudo ip link set brx.0 up 2024-08-03T20:50:36.059 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-08-03T20:50:36.060 DEBUG:teuthology.orchestra.run.smithi032:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-08-03T20:50:36.095 INFO:teuthology.orchestra.run.smithi032.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-08-03T20:50:36.095 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-08-03T20:50:36.095 DEBUG:teuthology.orchestra.run.smithi032:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-08-03T20:50:36.170 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2024-08-03T20:50:36.170 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-08-03T20:50:36.170 DEBUG:teuthology.orchestra.run.smithi032:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-08-03T20:50:36.368 INFO:teuthology.orchestra.run.smithi032.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-08-03T20:50:36.369 INFO:teuthology.orchestra.run.smithi032.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-08-03T20:50:36.369 INFO:teuthology.orchestra.run.smithi032.stdout:invalid new device string format 2024-08-03T20:50:36.389 INFO:teuthology.orchestra.run.smithi032.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-08-03T20:50:36.389 INFO:teuthology.orchestra.run.smithi032.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-08-03T20:50:36.389 INFO:teuthology.orchestra.run.smithi032.stdout:invalid new device string format 2024-08-03T20:50:36.389 INFO:teuthology.orchestra.run.smithi032.stdout:mount.ceph: resolved to: "172.21.15.32:3300,172.21.15.118:3300,172.21.15.118:3301" 2024-08-03T20:50:36.389 INFO:teuthology.orchestra.run.smithi032.stdout:mount.ceph: switching to using v1 address with old syntax 2024-08-03T20:50:36.419 INFO:teuthology.orchestra.run.smithi032.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-08-03T20:50:36.419 INFO:teuthology.orchestra.run.smithi032.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-08-03T20:50:36.419 INFO:teuthology.orchestra.run.smithi032.stdout:invalid new device string format 2024-08-03T20:50:36.420 INFO:teuthology.orchestra.run.smithi032.stdout:mount.ceph: resolved to: "172.21.15.32:3300,172.21.15.118:3300,172.21.15.118:3301" 2024-08-03T20:50:36.420 INFO:teuthology.orchestra.run.smithi032.stdout:mount.ceph: switching to using v1 address with old syntax 2024-08-03T20:50:36.420 INFO:teuthology.orchestra.run.smithi032.stdout:mount.ceph: resolved to: "172.21.15.32:6789,172.21.15.118:6789,172.21.15.118:6790" 2024-08-03T20:50:36.420 INFO:teuthology.orchestra.run.smithi032.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.32:6789,172.21.15.118:6789,172.21.15.118:6790:/ 2024-08-03T20:50:36.420 INFO:teuthology.orchestra.run.smithi032.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,key=0,fsid=40ab141a-01d7-49b9-ae3c-2edec9876e95" will pass to kernel 2024-08-03T20:50:36.421 INFO:teuthology.orchestra.run.smithi032.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2024-08-03T20:50:36.421 INFO:teuthology.orchestra.run.smithi032.stdout: You just mounted a file system that supports labels which does not 2024-08-03T20:50:36.421 INFO:teuthology.orchestra.run.smithi032.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-08-03T20:50:36.421 INFO:teuthology.orchestra.run.smithi032.stdout: applications will generate AVC messages and not be allowed access to 2024-08-03T20:50:36.421 INFO:teuthology.orchestra.run.smithi032.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-08-03T20:50:36.424 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-08-03T20:50:36.424 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-08-03T20:50:36.424 DEBUG:teuthology.orchestra.run.smithi032:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-08-03T20:50:36.481 DEBUG:teuthology.orchestra.run.smithi032:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-08-03T20:50:36.562 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:36 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-08-03T20:50:36.569 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:36 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-08-03T20:50:36.577 INFO:teuthology.orchestra.run.smithi032.stdout:client5366 2024-08-03T20:50:36.578 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-08-03T20:50:36.578 DEBUG:teuthology.orchestra.run.smithi032:> dd if=/proc/self/mounts of=/dev/stdout 2024-08-03T20:50:36.613 DEBUG:teuthology.orchestra.run.smithi032:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'mkdir -p a/{b,c}/d; sync .') 2024-08-03T20:50:36.735 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:36 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-08-03T20:50:36.750 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:36 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-08-03T20:50:36.759 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-08-03T20:50:36.759 DEBUG:teuthology.orchestra.run.smithi032:> dd if=/proc/self/mounts of=/dev/stdout 2024-08-03T20:50:36.795 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-08-03T20:50:36.878 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-08-03T20:50:36.883 DEBUG:teuthology.orchestra.run.smithi118:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-08-03T20:50:37.071 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:37.070+0000 7f489ea00640 1 -- 172.21.15.32:0/3945291972 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f489806e930 msgr2=0x7f4898144f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:50:37.071 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:37.070+0000 7f489ea00640 1 --2- 172.21.15.32:0/3945291972 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f489806e930 0x7f4898144f90 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7f4888009830 tx=0x7f488802a480 comp rx=0 tx=0).stop 2024-08-03T20:50:37.071 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:37.070+0000 7f489ea00640 1 -- 172.21.15.32:0/3945291972 shutdown_connections 2024-08-03T20:50:37.071 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:37.070+0000 7f489ea00640 1 --2- 172.21.15.32:0/3945291972 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f4898073e60 0x7f4898074270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:37.071 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:37.070+0000 7f489ea00640 1 --2- 172.21.15.32:0/3945291972 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f489806e930 0x7f4898144f90 unknown :-1 s=CLOSED pgs=594 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:37.072 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:37.070+0000 7f489ea00640 1 -- 172.21.15.32:0/3945291972 >> 172.21.15.32:0/3945291972 conn(0x7f48980b1bd0 msgr2=0x7f48980b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:50:37.072 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:37.070+0000 7f489ea00640 1 -- 172.21.15.32:0/3945291972 shutdown_connections 2024-08-03T20:50:37.072 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:37.070+0000 7f489ea00640 1 -- 172.21.15.32:0/3945291972 wait complete. 2024-08-03T20:50:37.073 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:37.072+0000 7f489ea00640 1 Processor -- start 2024-08-03T20:50:37.073 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:37.073+0000 7f489ea00640 1 -- start start 2024-08-03T20:50:37.075 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:37.074+0000 7f489ea00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f489806e930 0x7f48980b2fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:50:37.075 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:37.074+0000 7f489ea00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4898073e60 0x7f48980adff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:50:37.075 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:37.074+0000 7f489ea00640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f48980747b0 0x7f489806ef70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:50:37.076 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:37.074+0000 7f489ea00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f48980cf9c0 con 0x7f4898073e60 2024-08-03T20:50:37.076 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:37.074+0000 7f489ea00640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f48980cf840 con 0x7f489806e930 2024-08-03T20:50:37.076 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:37.074+0000 7f489ea00640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7f48980ccff0 con 0x7f48980747b0 2024-08-03T20:50:37.076 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:37.074+0000 7f488fe00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4898073e60 0x7f48980adff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:50:37.076 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:37.074+0000 7f488fe00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4898073e60 0x7f48980adff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:45216/0 (socket says 172.21.15.32:45216) 2024-08-03T20:50:37.076 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:37.074+0000 7f488fe00640 1 -- 172.21.15.32:0/2936256026 learned_addr learned my addr 172.21.15.32:0/2936256026 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:50:37.076 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:37.075+0000 7f489ca00640 1 --2- 172.21.15.32:0/2936256026 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f48980747b0 0x7f489806ef70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:50:37.076 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:37.075+0000 7f4897e00640 1 --2- 172.21.15.32:0/2936256026 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f489806e930 0x7f48980b2fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:50:37.076 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:37.075+0000 7f488fe00640 1 -- 172.21.15.32:0/2936256026 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f48980747b0 msgr2=0x7f489806ef70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:50:37.076 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:37.075+0000 7f488fe00640 1 --2- 172.21.15.32:0/2936256026 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f48980747b0 0x7f489806ef70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:37.076 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:37.075+0000 7f488fe00640 1 -- 172.21.15.32:0/2936256026 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f489806e930 msgr2=0x7f48980b2fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:50:37.076 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:37.075+0000 7f488fe00640 1 --2- 172.21.15.32:0/2936256026 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f489806e930 0x7f48980b2fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:37.077 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:37.075+0000 7f488fe00640 1 -- 172.21.15.32:0/2936256026 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f48981f8af0 con 0x7f4898073e60 2024-08-03T20:50:37.077 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:37.075+0000 7f4897e00640 1 --2- 172.21.15.32:0/2936256026 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f489806e930 0x7f48980b2fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-03T20:50:37.078 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:37.075+0000 7f488fe00640 1 --2- 172.21.15.32:0/2936256026 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4898073e60 0x7f48980adff0 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7f488000d770 tx=0x7f488000dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:50:37.078 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:37.076+0000 7f4894c00640 1 -- 172.21.15.32:0/2936256026 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4880017070 con 0x7f4898073e60 2024-08-03T20:50:37.078 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:37.076+0000 7f4894c00640 1 -- 172.21.15.32:0/2936256026 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f4880009d20 con 0x7f4898073e60 2024-08-03T20:50:37.078 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:37.076+0000 7f489ea00640 1 -- 172.21.15.32:0/2936256026 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f48981f8dd0 con 0x7f4898073e60 2024-08-03T20:50:37.078 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:37.076+0000 7f4894c00640 1 -- 172.21.15.32:0/2936256026 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48800106c0 con 0x7f4898073e60 2024-08-03T20:50:37.078 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:37.076+0000 7f489ea00640 1 -- 172.21.15.32:0/2936256026 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f48980adad0 con 0x7f4898073e60 2024-08-03T20:50:37.078 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:37.077+0000 7f489ea00640 1 -- 172.21.15.32:0/2936256026 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f48980ad8c0 con 0x7f4898073e60 2024-08-03T20:50:37.078 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:37.078+0000 7f4894c00640 1 -- 172.21.15.32:0/2936256026 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f488001a070 con 0x7f4898073e60 2024-08-03T20:50:37.079 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:37.079+0000 7f4894c00640 1 --2- 172.21.15.32:0/2936256026 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f4874077040 0x7f4874079550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:50:37.079 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:37.079+0000 7f4894c00640 1 -- 172.21.15.32:0/2936256026 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(57..57 src has 1..57) ==== 5709+0+0 (secure 0 0 0) 0x7f4880098a50 con 0x7f4898073e60 2024-08-03T20:50:37.080 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:37.079+0000 7f4897e00640 1 --2- 172.21.15.32:0/2936256026 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f4874077040 0x7f4874079550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:50:37.080 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:37.080+0000 7f4897e00640 1 --2- 172.21.15.32:0/2936256026 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f4874077040 0x7f4874079550 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f4888034000 tx=0x7f4888004730 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:50:37.084 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:37.083+0000 7f4894c00640 1 -- 172.21.15.32:0/2936256026 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7f4880061bc0 con 0x7f4898073e60 2024-08-03T20:50:37.301 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:37.301+0000 7f489ea00640 1 -- 172.21.15.32:0/2936256026 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f48981f8880 con 0x7f4898073e60 2024-08-03T20:50:37.302 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:37.301+0000 7f4894c00640 1 -- 172.21.15.32:0/2936256026 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v57) ==== 80+0+0 (secure 0 0 0) 0x7f48800614c0 con 0x7f4898073e60 2024-08-03T20:50:37.302 INFO:teuthology.orchestra.run.smithi032.stderr:listed 0 entries 2024-08-03T20:50:37.304 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:37.303+0000 7f489ea00640 1 -- 172.21.15.32:0/2936256026 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f4874077040 msgr2=0x7f4874079550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:50:37.304 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:37.303+0000 7f489ea00640 1 --2- 172.21.15.32:0/2936256026 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f4874077040 0x7f4874079550 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f4888034000 tx=0x7f4888004730 comp rx=0 tx=0).stop 2024-08-03T20:50:37.304 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:37.303+0000 7f489ea00640 1 -- 172.21.15.32:0/2936256026 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4898073e60 msgr2=0x7f48980adff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:50:37.304 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:37.303+0000 7f489ea00640 1 --2- 172.21.15.32:0/2936256026 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4898073e60 0x7f48980adff0 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7f488000d770 tx=0x7f488000dc30 comp rx=0 tx=0).stop 2024-08-03T20:50:37.305 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:37.303+0000 7f489ea00640 1 -- 172.21.15.32:0/2936256026 shutdown_connections 2024-08-03T20:50:37.305 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:37.303+0000 7f489ea00640 1 --2- 172.21.15.32:0/2936256026 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f4874077040 0x7f4874079550 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:37.305 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:37.303+0000 7f489ea00640 1 --2- 172.21.15.32:0/2936256026 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f48980747b0 0x7f489806ef70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:37.305 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:37.303+0000 7f489ea00640 1 --2- 172.21.15.32:0/2936256026 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4898073e60 0x7f48980adff0 unknown :-1 s=CLOSED pgs=595 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:37.305 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:37.303+0000 7f489ea00640 1 --2- 172.21.15.32:0/2936256026 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f489806e930 0x7f48980b2fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:37.305 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:37.303+0000 7f489ea00640 1 -- 172.21.15.32:0/2936256026 >> 172.21.15.32:0/2936256026 conn(0x7f48980b1bd0 msgr2=0x7f48980b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:50:37.305 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:37.304+0000 7f489ea00640 1 -- 172.21.15.32:0/2936256026 shutdown_connections 2024-08-03T20:50:37.305 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:37.304+0000 7f489ea00640 1 -- 172.21.15.32:0/2936256026 wait complete. 2024-08-03T20:50:37.328 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-08-03T20:50:37.328 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-08-03T20:50:37.328 DEBUG:teuthology.orchestra.run.smithi032:> sudo umount /home/ubuntu/cephtest/mnt.0 2024-08-03T20:50:37.394 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi032.front.sepia.ceph.com 2024-08-03T20:50:37.394 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-08-03T20:50:37.395 DEBUG:teuthology.orchestra.run.smithi032:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-08-03T20:50:37.431 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.4:0 flush journal 2024-08-03T20:50:37.703 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:37.702+0000 7f32b6600640 1 -- 172.21.15.32:0/2518627692 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f32b00cccf0 msgr2=0x7f32b00cd100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:50:37.703 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:37.702+0000 7f32b6600640 1 --2- 172.21.15.32:0/2518627692 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f32b00cccf0 0x7f32b00cd100 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7f329c005fd0 tx=0x7f329c02a5c0 comp rx=0 tx=0).stop 2024-08-03T20:50:37.703 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:37.702+0000 7f32b6600640 1 -- 172.21.15.32:0/2518627692 shutdown_connections 2024-08-03T20:50:37.703 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:37.702+0000 7f32b6600640 1 --2- 172.21.15.32:0/2518627692 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f32b00cccf0 0x7f32b00cd100 unknown :-1 s=CLOSED pgs=597 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:37.703 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:37.702+0000 7f32b6600640 1 --2- 172.21.15.32:0/2518627692 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f32b0145c50 0x7f32b006e950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:37.703 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:37.702+0000 7f32b6600640 1 -- 172.21.15.32:0/2518627692 >> 172.21.15.32:0/2518627692 conn(0x7f32b00b1bd0 msgr2=0x7f32b00b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:50:37.703 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:37.702+0000 7f32b6600640 1 -- 172.21.15.32:0/2518627692 shutdown_connections 2024-08-03T20:50:37.704 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:37.702+0000 7f32b6600640 1 -- 172.21.15.32:0/2518627692 wait complete. 2024-08-03T20:50:37.704 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:37.704+0000 7f32b6600640 1 Processor -- start 2024-08-03T20:50:37.704 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:37.704+0000 7f32b6600640 1 -- start start 2024-08-03T20:50:37.707 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:37.706+0000 7f32b6600640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f32b00cccf0 0x7f32b01f8710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:50:37.707 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:37.706+0000 7f32b6600640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f32b00ced30 0x7f32b0158c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:50:37.708 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:37.706+0000 7f32b6600640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f32b0145c50 0x7f32b0072ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:50:37.708 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:37.706+0000 7f32b6600640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f32b00b4070 con 0x7f32b00cccf0 2024-08-03T20:50:37.708 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:37.706+0000 7f32b6600640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f32b00b3ef0 con 0x7f32b00ced30 2024-08-03T20:50:37.708 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:37.706+0000 7f32b6600640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7f32b00b41f0 con 0x7f32b0145c50 2024-08-03T20:50:37.708 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:37.706+0000 7f32af400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f32b00cccf0 0x7f32b01f8710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:50:37.708 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:37.706+0000 7f32af400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f32b00cccf0 0x7f32b01f8710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:45234/0 (socket says 172.21.15.32:45234) 2024-08-03T20:50:37.708 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:37.706+0000 7f32af400640 1 -- 172.21.15.32:0/315550581 learned_addr learned my addr 172.21.15.32:0/315550581 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:50:37.708 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:37.706+0000 7f32afe00640 1 --2- 172.21.15.32:0/315550581 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f32b0145c50 0x7f32b0072ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:50:37.708 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:37.706+0000 7f32af400640 1 -- 172.21.15.32:0/315550581 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f32b0145c50 msgr2=0x7f32b0072ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:50:37.708 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:37.706+0000 7f32aea00640 1 --2- 172.21.15.32:0/315550581 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f32b00ced30 0x7f32b0158c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:50:37.708 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:37.706+0000 7f32af400640 1 --2- 172.21.15.32:0/315550581 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f32b0145c50 0x7f32b0072ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:37.709 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:37.706+0000 7f32af400640 1 -- 172.21.15.32:0/315550581 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f32b00ced30 msgr2=0x7f32b0158c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:50:37.709 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:37.706+0000 7f32af400640 1 --2- 172.21.15.32:0/315550581 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f32b00ced30 0x7f32b0158c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:37.709 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:37.706+0000 7f32af400640 1 -- 172.21.15.32:0/315550581 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f32b01f8c50 con 0x7f32b00cccf0 2024-08-03T20:50:37.709 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:37.706+0000 7f32afe00640 1 --2- 172.21.15.32:0/315550581 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f32b0145c50 0x7f32b0072ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-03T20:50:37.709 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:37.706+0000 7f32aea00640 1 --2- 172.21.15.32:0/315550581 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f32b00ced30 0x7f32b0158c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-03T20:50:37.709 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:37.707+0000 7f32af400640 1 --2- 172.21.15.32:0/315550581 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f32b00cccf0 0x7f32b01f8710 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7f32a000d700 tx=0x7f32a000dbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:50:37.709 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:37.707+0000 7f32a7400640 1 -- 172.21.15.32:0/315550581 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32a0017070 con 0x7f32b00cccf0 2024-08-03T20:50:37.709 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:37.707+0000 7f32a7400640 1 -- 172.21.15.32:0/315550581 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f32a0005310 con 0x7f32b00cccf0 2024-08-03T20:50:37.709 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:37.707+0000 7f32b6600640 1 -- 172.21.15.32:0/315550581 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f32b0159430 con 0x7f32b00cccf0 2024-08-03T20:50:37.709 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:37.707+0000 7f32a7400640 1 -- 172.21.15.32:0/315550581 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32a0009820 con 0x7f32b00cccf0 2024-08-03T20:50:37.709 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:37.707+0000 7f32b6600640 1 -- 172.21.15.32:0/315550581 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f32b0073c70 con 0x7f32b00cccf0 2024-08-03T20:50:37.710 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:37.709+0000 7f32a7400640 1 -- 172.21.15.32:0/315550581 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f32a00099c0 con 0x7f32b00cccf0 2024-08-03T20:50:37.711 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:37.711+0000 7f32a7400640 1 --2- 172.21.15.32:0/315550581 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f3290077010 0x7f3290079520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:50:37.711 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:37.711+0000 7f32a7400640 1 -- 172.21.15.32:0/315550581 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(57..57 src has 1..57) ==== 5709+0+0 (secure 0 0 0) 0x7f32a0098a70 con 0x7f32b00cccf0 2024-08-03T20:50:37.712 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:37.712+0000 7f32bd630740 1 Processor -- start 2024-08-03T20:50:37.712 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:37.712+0000 7f32aea00640 1 --2- 172.21.15.32:0/315550581 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f3290077010 0x7f3290079520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:50:37.712 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:37.712+0000 7f32bd630740 1 -- start start 2024-08-03T20:50:37.713 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:37.713+0000 7f32aea00640 1 --2- 172.21.15.32:0/315550581 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f3290077010 0x7f3290079520 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f329c033000 tx=0x7f329c004730 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:50:37.715 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:37.713+0000 7f32bd630740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x559ef8d55390 0x559ef9125900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:50:37.715 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:37.713+0000 7f32bd630740 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x559ef8d2c4d0 0x559ef9125e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:50:37.715 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:37.713+0000 7f32bd630740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x559ef8d346b0 0x559ef9142d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:50:37.715 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:37.713+0000 7f32bd630740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x559ef906e470 con 0x559ef8d346b0 2024-08-03T20:50:37.715 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:37.713+0000 7f32bd630740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x559ef905d970 con 0x559ef8d55390 2024-08-03T20:50:37.715 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:37.713+0000 7f32bd630740 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x559ef90619b0 con 0x559ef8d2c4d0 2024-08-03T20:50:37.715 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:37.713+0000 7f32afe00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x559ef8d346b0 0x559ef9142d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:50:37.715 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:37.714+0000 7f32afe00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x559ef8d346b0 0x559ef9142d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:45236/0 (socket says 172.21.15.32:45236) 2024-08-03T20:50:37.715 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:37.714+0000 7f32afe00640 1 -- 172.21.15.32:0/1640544698 learned_addr learned my addr 172.21.15.32:0/1640544698 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:50:37.715 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:37.714+0000 7f32aea00640 1 --2- 172.21.15.32:0/1640544698 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x559ef8d2c4d0 0x559ef9125e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:50:37.715 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:37.714+0000 7f32af400640 1 --2- 172.21.15.32:0/1640544698 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x559ef8d55390 0x559ef9125900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:50:37.716 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:37.714+0000 7f32afe00640 1 -- 172.21.15.32:0/1640544698 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x559ef8d2c4d0 msgr2=0x559ef9125e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:50:37.716 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:37.714+0000 7f32afe00640 1 --2- 172.21.15.32:0/1640544698 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x559ef8d2c4d0 0x559ef9125e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:37.716 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:37.714+0000 7f32afe00640 1 -- 172.21.15.32:0/1640544698 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x559ef8d55390 msgr2=0x559ef9125900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:50:37.716 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:37.714+0000 7f32afe00640 1 --2- 172.21.15.32:0/1640544698 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x559ef8d55390 0x559ef9125900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:37.716 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:37.714+0000 7f32afe00640 1 -- 172.21.15.32:0/1640544698 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559ef8e377d0 con 0x559ef8d346b0 2024-08-03T20:50:37.716 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:37.714+0000 7f32afe00640 1 --2- 172.21.15.32:0/1640544698 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x559ef8d346b0 0x559ef9142d40 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7f32a8001db0 tx=0x7f32a800d520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7c0bdd1ded960c28 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:50:37.716 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:37.714+0000 7f329be00640 1 -- 172.21.15.32:0/1640544698 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32a8018070 con 0x559ef8d346b0 2024-08-03T20:50:37.716 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:37.714+0000 7f329be00640 1 -- 172.21.15.32:0/1640544698 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f32a8003990 con 0x559ef8d346b0 2024-08-03T20:50:37.716 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:37.715+0000 7f329be00640 1 -- 172.21.15.32:0/1640544698 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32a8003d60 con 0x559ef8d346b0 2024-08-03T20:50:37.717 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:37.715+0000 7f32bd630740 1 -- 172.21.15.32:0/1640544698 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x559ef8d346b0 msgr2=0x559ef9142d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:50:37.717 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:37.715+0000 7f32bd630740 1 --2- 172.21.15.32:0/1640544698 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x559ef8d346b0 0x559ef9142d40 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7f32a8001db0 tx=0x7f32a800d520 comp rx=0 tx=0).stop 2024-08-03T20:50:37.717 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:37.715+0000 7f32bd630740 1 -- 172.21.15.32:0/1640544698 shutdown_connections 2024-08-03T20:50:37.717 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:37.715+0000 7f32bd630740 1 --2- 172.21.15.32:0/1640544698 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x559ef8d346b0 0x559ef9142d40 unknown :-1 s=CLOSED pgs=599 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:37.718 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:37.715+0000 7f32bd630740 1 --2- 172.21.15.32:0/1640544698 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x559ef8d2c4d0 0x559ef9125e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:37.718 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:37.715+0000 7f32bd630740 1 --2- 172.21.15.32:0/1640544698 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x559ef8d55390 0x559ef9125900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:37.718 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:37.715+0000 7f32bd630740 1 -- 172.21.15.32:0/1640544698 >> 172.21.15.32:0/1640544698 conn(0x559ef8d494e0 msgr2=0x559ef90fbbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:50:37.718 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:37.715+0000 7f32bd630740 1 -- 172.21.15.32:0/1640544698 shutdown_connections 2024-08-03T20:50:37.718 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:37.715+0000 7f32bd630740 1 -- 172.21.15.32:0/1640544698 wait complete. 2024-08-03T20:50:37.718 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:37.716+0000 7f32bd630740 1 -- start start 2024-08-03T20:50:37.718 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:37.717+0000 7f32bd630740 1 Processor -- start 2024-08-03T20:50:37.721 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:37.718+0000 7f32bd630740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x559ef8d2c4d0 0x559ef91509c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:50:37.721 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:37.718+0000 7f32bd630740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x559ef8d346b0 0x559ef914d3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:50:37.721 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:37.718+0000 7f32bd630740 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x559ef8d55390 0x559ef913ce10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:50:37.721 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:37.718+0000 7f32bd630740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x559ef8f95c90 con 0x559ef8d2c4d0 2024-08-03T20:50:37.721 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:37.718+0000 7f32bd630740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x559ef8ff7fd0 con 0x559ef8d346b0 2024-08-03T20:50:37.721 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:37.718+0000 7f32bd630740 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x559ef9126380 con 0x559ef8d55390 2024-08-03T20:50:37.721 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:37.718+0000 7f32bd630740 10 client.0 _ll_register_callbacks cb 0x559ef9012e50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-03T20:50:37.722 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:37.718+0000 7f32af400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x559ef8d2c4d0 0x559ef91509c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:50:37.722 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:37.719+0000 7f32af400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x559ef8d2c4d0 0x559ef91509c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:45238/0 (socket says 172.21.15.32:45238) 2024-08-03T20:50:37.722 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:37.719+0000 7f32af400640 1 -- 172.21.15.32:0/3332014728 learned_addr learned my addr 172.21.15.32:0/3332014728 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:50:37.722 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:37.719+0000 7f32aea00640 1 --2- 172.21.15.32:0/3332014728 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x559ef8d346b0 0x559ef914d3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:50:37.722 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:37.719+0000 7f32afe00640 1 --2- 172.21.15.32:0/3332014728 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x559ef8d55390 0x559ef913ce10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:50:37.722 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:37.719+0000 7f32af400640 1 -- 172.21.15.32:0/3332014728 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x559ef8d55390 msgr2=0x559ef913ce10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:50:37.722 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:37.719+0000 7f32af400640 1 --2- 172.21.15.32:0/3332014728 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x559ef8d55390 0x559ef913ce10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:37.722 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:37.719+0000 7f32af400640 1 -- 172.21.15.32:0/3332014728 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x559ef8d346b0 msgr2=0x559ef914d3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:50:37.722 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:37.719+0000 7f32af400640 1 --2- 172.21.15.32:0/3332014728 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x559ef8d346b0 0x559ef914d3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:37.722 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:37.719+0000 7f32af400640 1 -- 172.21.15.32:0/3332014728 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559ef8dc0970 con 0x559ef8d2c4d0 2024-08-03T20:50:37.722 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:37.719+0000 7f32af400640 1 --2- 172.21.15.32:0/3332014728 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x559ef8d2c4d0 0x559ef91509c0 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7f32a005e7d0 tx=0x7f32a0068140 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4f4f9e792b1d11ed server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:50:37.723 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:37.719+0000 7f3298c00640 10 client.0 ms_handle_connect on v2:172.21.15.32:3300/0 2024-08-03T20:50:37.723 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:37.719+0000 7f3298c00640 1 -- 172.21.15.32:0/3332014728 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32a007a070 con 0x559ef8d2c4d0 2024-08-03T20:50:37.723 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:37.719+0000 7f3298c00640 1 -- 172.21.15.32:0/3332014728 <== mon.0 v2:172.21.15.32:3300/0 2 ==== osd_map(57..57 src has 1..57) ==== 5709+0+0 (secure 0 0 0) 0x7f32a0096980 con 0x559ef8d2c4d0 2024-08-03T20:50:37.723 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:37.719+0000 7f32bd630740 1 -- 172.21.15.32:0/3332014728 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f32a8003990 con 0x559ef8d2c4d0 2024-08-03T20:50:37.723 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:37.720+0000 7f3298c00640 1 -- 172.21.15.32:0/3332014728 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_get_version_reply(handle=1 version=51) ==== 24+0+0 (secure 0 0 0) 0x7f32a007f050 con 0x559ef8d2c4d0 2024-08-03T20:50:37.723 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:37.720+0000 7f32bd630740 10 client.6006 fetch_fsmap learned FSMap version 51 2024-08-03T20:50:37.723 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:37.720+0000 7f32bd630740 1 -- 172.21.15.32:0/3332014728 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,fsmap=51,monmap=2+}) -- 0x559ef8d06270 con 0x559ef8d2c4d0 2024-08-03T20:50:37.723 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:37.720+0000 7f3298c00640 1 -- 172.21.15.32:0/3332014728 <== mon.0 v2:172.21.15.32:3300/0 4 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f32a8003990 con 0x559ef8d2c4d0 2024-08-03T20:50:37.723 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:37.720+0000 7f3298c00640 1 -- 172.21.15.32:0/3332014728 <== mon.0 v2:172.21.15.32:3300/0 5 ==== fsmap(e 51) ==== 3502+0+0 (secure 0 0 0) 0x7f32a0082020 con 0x559ef8d2c4d0 2024-08-03T20:50:37.723 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:37.720+0000 7f32bd630740 10 client.6006 fetch_fsmap finished waiting for FSMap version 51 2024-08-03T20:50:37.723 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:37.720+0000 7f32bd630740 10 client.6006 resolve_mds: resolved 4:0 to role '4:0' aka daemon mds.c 2024-08-03T20:50:37.723 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:37.720+0000 7f32bd630740 20 client.6006 populate_metadata read hostname 'smithi032' 2024-08-03T20:50:37.723 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:37.720+0000 7f32bd630740 1 --2- 172.21.15.32:0/3332014728 >> [v2:172.21.15.32:6834/3848976369,v1:172.21.15.32:6835/3848976369] conn(0x559ef8d13f60 0x559ef9137a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:50:37.723 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:37.720+0000 7f32bd630740 4 client.6006 mds_command: new command op to 5841 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-08-03T20:50:37.724 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:37.720+0000 7f32bd630740 1 -- 172.21.15.32:0/3332014728 --> [v2:172.21.15.32:6834/3848976369,v1:172.21.15.32:6835/3848976369] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x559ef90703c0 con 0x559ef8d13f60 2024-08-03T20:50:37.724 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:37.720+0000 7f32aea00640 1 --2- 172.21.15.32:0/3332014728 >> [v2:172.21.15.32:6834/3848976369,v1:172.21.15.32:6835/3848976369] conn(0x559ef8d13f60 0x559ef9137a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:50:37.725 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:37.721+0000 7f32aea00640 1 --2- 172.21.15.32:0/3332014728 >> [v2:172.21.15.32:6834/3848976369,v1:172.21.15.32:6835/3848976369] conn(0x559ef8d13f60 0x559ef9137a30 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=49735815c16bc8a7 server_cookie=50dfa620d2aa4218 in_seq=0 out_seq=0 2024-08-03T20:50:37.725 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:37.721+0000 7f3298c00640 10 client.6006 ms_handle_connect on v2:172.21.15.32:6834/3848976369 2024-08-03T20:50:37.725 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:37.722+0000 7f3298c00640 1 -- 172.21.15.32:0/3332014728 <== mds.0 v2:172.21.15.32:6834/3848976369 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x559ef90703c0 con 0x559ef8d13f60 2024-08-03T20:50:37.725 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:37.722+0000 7f3298c00640 10 client.6006 handle_command_reply: tid=0 2024-08-03T20:50:37.725 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:37.722+0000 7f32bd630740 1 client.6006 shutdown 2024-08-03T20:50:37.725 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:37.722+0000 7f32bd630740 1 -- 172.21.15.32:0/3332014728 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x559ef8d2c4d0 msgr2=0x559ef91509c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:50:37.725 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:37.722+0000 7f32bd630740 1 --2- 172.21.15.32:0/3332014728 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x559ef8d2c4d0 0x559ef91509c0 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7f32a005e7d0 tx=0x7f32a0068140 comp rx=0 tx=0).stop 2024-08-03T20:50:37.725 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:37.722+0000 7f32bd630740 1 -- 172.21.15.32:0/3332014728 shutdown_connections 2024-08-03T20:50:37.725 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:37.722+0000 7f32bd630740 1 --2- 172.21.15.32:0/3332014728 >> [v2:172.21.15.32:6834/3848976369,v1:172.21.15.32:6835/3848976369] conn(0x559ef8d13f60 0x559ef9137a30 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:37.725 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:37.722+0000 7f32bd630740 1 --2- 172.21.15.32:0/3332014728 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x559ef8d346b0 0x559ef914d3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:37.725 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:37.722+0000 7f32bd630740 1 --2- 172.21.15.32:0/3332014728 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x559ef8d55390 0x559ef913ce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:37.726 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:37.722+0000 7f3298c00640 0 client.6006 ms_handle_reset on v2:172.21.15.32:6834/3848976369 2024-08-03T20:50:37.726 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:37.722+0000 7f32bd630740 1 --2- 172.21.15.32:0/3332014728 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x559ef8d2c4d0 0x559ef91509c0 unknown :-1 s=CLOSED pgs=600 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:37.726 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:37.722+0000 7f32bd630740 1 -- 172.21.15.32:0/3332014728 >> 172.21.15.32:0/3332014728 conn(0x559ef8d494e0 msgr2=0x559ef90f64e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:50:37.726 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:37.722+0000 7f32bd630740 1 -- 172.21.15.32:0/3332014728 shutdown_connections 2024-08-03T20:50:37.726 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:37.723+0000 7f32bd630740 1 -- 172.21.15.32:0/3332014728 wait complete. 2024-08-03T20:50:37.726 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:37.723+0000 7f32bd630740 20 client.6006 trim_cache size 0 max 16384 2024-08-03T20:50:37.739 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:37.739+0000 7f32bd630740 1 Processor -- start 2024-08-03T20:50:37.739 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:37.739+0000 7f32bd630740 1 -- start start 2024-08-03T20:50:37.742 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:37.740+0000 7f32bd630740 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x559ef8d2c4d0 0x559ef9114910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:50:37.742 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:37.740+0000 7f32bd630740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x559ef8d346b0 0x559ef9136b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:50:37.742 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:37.740+0000 7f32bd630740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x559ef8d55390 0x559ef9125900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:50:37.742 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:37.740+0000 7f32bd630740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f32b0201e30 con 0x559ef8d55390 2024-08-03T20:50:37.742 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:37.740+0000 7f32bd630740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f32b0200e10 con 0x559ef8d346b0 2024-08-03T20:50:37.742 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:37.740+0000 7f32bd630740 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7f32b013d590 con 0x559ef8d2c4d0 2024-08-03T20:50:37.742 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:37.741+0000 7f32afe00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x559ef8d55390 0x559ef9125900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:50:37.742 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:37.741+0000 7f32afe00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x559ef8d55390 0x559ef9125900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:45246/0 (socket says 172.21.15.32:45246) 2024-08-03T20:50:37.742 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:37.741+0000 7f32afe00640 1 -- 172.21.15.32:0/2189039283 learned_addr learned my addr 172.21.15.32:0/2189039283 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:50:37.742 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:37.741+0000 7f32af400640 1 --2- 172.21.15.32:0/2189039283 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x559ef8d2c4d0 0x559ef9114910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:50:37.742 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:37.741+0000 7f32aea00640 1 --2- 172.21.15.32:0/2189039283 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x559ef8d346b0 0x559ef9136b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:50:37.743 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:37.741+0000 7f32afe00640 1 -- 172.21.15.32:0/2189039283 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x559ef8d2c4d0 msgr2=0x559ef9114910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:50:37.743 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:37.741+0000 7f32afe00640 1 --2- 172.21.15.32:0/2189039283 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x559ef8d2c4d0 0x559ef9114910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:37.743 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:37.741+0000 7f32afe00640 1 -- 172.21.15.32:0/2189039283 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x559ef8d346b0 msgr2=0x559ef9136b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:50:37.743 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:37.741+0000 7f32afe00640 1 --2- 172.21.15.32:0/2189039283 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x559ef8d346b0 0x559ef9136b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:37.743 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:37.741+0000 7f32afe00640 1 -- 172.21.15.32:0/2189039283 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559ef91231b0 con 0x559ef8d55390 2024-08-03T20:50:37.743 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:37.741+0000 7f32af400640 1 --2- 172.21.15.32:0/2189039283 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x559ef8d2c4d0 0x559ef9114910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-03T20:50:37.743 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:37.741+0000 7f32afe00640 1 --2- 172.21.15.32:0/2189039283 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x559ef8d55390 0x559ef9125900 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7f32a8012000 tx=0x7f32a8001400 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9d0fbe6c7500fd9 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:50:37.744 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:37.742+0000 7f329b400640 1 -- 172.21.15.32:0/2189039283 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32a8022070 con 0x559ef8d55390 2024-08-03T20:50:37.744 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:37.742+0000 7f329b400640 1 -- 172.21.15.32:0/2189039283 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f32a8003a60 con 0x559ef8d55390 2024-08-03T20:50:37.745 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:37.742+0000 7f329b400640 1 -- 172.21.15.32:0/2189039283 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32a8018040 con 0x559ef8d55390 2024-08-03T20:50:37.745 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:37.742+0000 7f32bd630740 1 -- 172.21.15.32:0/2189039283 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x559ef8d55390 msgr2=0x559ef9125900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:50:37.745 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:37.742+0000 7f32bd630740 1 --2- 172.21.15.32:0/2189039283 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x559ef8d55390 0x559ef9125900 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7f32a8012000 tx=0x7f32a8001400 comp rx=0 tx=0).stop 2024-08-03T20:50:37.745 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:37.742+0000 7f32bd630740 1 -- 172.21.15.32:0/2189039283 shutdown_connections 2024-08-03T20:50:37.745 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:37.742+0000 7f32bd630740 1 --2- 172.21.15.32:0/2189039283 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x559ef8d346b0 0x559ef9136b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:37.745 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:37.742+0000 7f32bd630740 1 --2- 172.21.15.32:0/2189039283 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x559ef8d55390 0x559ef9125900 unknown :-1 s=CLOSED pgs=601 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:37.745 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:37.742+0000 7f32bd630740 1 --2- 172.21.15.32:0/2189039283 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x559ef8d2c4d0 0x559ef9114910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:37.745 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:37.742+0000 7f32bd630740 1 -- 172.21.15.32:0/2189039283 >> 172.21.15.32:0/2189039283 conn(0x559ef8d494e0 msgr2=0x559ef9140de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:50:37.745 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:37.743+0000 7f32bd630740 1 -- 172.21.15.32:0/2189039283 shutdown_connections 2024-08-03T20:50:37.745 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:37.743+0000 7f32bd630740 1 -- 172.21.15.32:0/2189039283 wait complete. 2024-08-03T20:50:37.745 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:37.744+0000 7f32bd630740 1 -- start start 2024-08-03T20:50:37.745 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:37.744+0000 7f32bd630740 1 Processor -- start 2024-08-03T20:50:37.748 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:37.745+0000 7f32bd630740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x559ef8d2c4d0 0x559ef91374f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:50:37.748 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:37.745+0000 7f32bd630740 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x559ef8d346b0 0x559ef915a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:50:37.748 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:37.745+0000 7f32bd630740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x559ef8d55390 0x559ef91bc4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:50:37.748 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:37.745+0000 7f32bd630740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f32b01e9850 con 0x559ef8d55390 2024-08-03T20:50:37.748 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:37.745+0000 7f32bd630740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x559ef91154a0 con 0x559ef8d2c4d0 2024-08-03T20:50:37.749 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:37.745+0000 7f32bd630740 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x559ef91156f0 con 0x559ef8d346b0 2024-08-03T20:50:37.749 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:37.745+0000 7f32bd630740 10 client.0 _ll_register_callbacks cb 0x7f32a0009da0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-03T20:50:37.749 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:37.745+0000 7f32afe00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x559ef8d55390 0x559ef91bc4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:50:37.749 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:37.746+0000 7f32afe00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x559ef8d55390 0x559ef91bc4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:45256/0 (socket says 172.21.15.32:45256) 2024-08-03T20:50:37.749 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:37.746+0000 7f32afe00640 1 -- 172.21.15.32:0/135191222 learned_addr learned my addr 172.21.15.32:0/135191222 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:50:37.749 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:37.746+0000 7f32aea00640 1 --2- 172.21.15.32:0/135191222 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x559ef8d346b0 0x559ef915a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:50:37.749 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:37.746+0000 7f32af400640 1 --2- 172.21.15.32:0/135191222 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x559ef8d2c4d0 0x559ef91374f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:50:37.749 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:37.746+0000 7f32afe00640 1 -- 172.21.15.32:0/135191222 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x559ef8d346b0 msgr2=0x559ef915a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:50:37.749 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:37.746+0000 7f32afe00640 1 --2- 172.21.15.32:0/135191222 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x559ef8d346b0 0x559ef915a090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:37.749 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:37.746+0000 7f32afe00640 1 -- 172.21.15.32:0/135191222 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x559ef8d2c4d0 msgr2=0x559ef91374f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:50:37.749 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:37.746+0000 7f32afe00640 1 --2- 172.21.15.32:0/135191222 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x559ef8d2c4d0 0x559ef91374f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:37.750 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:37.746+0000 7f32afe00640 1 -- 172.21.15.32:0/135191222 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559ef9122500 con 0x559ef8d55390 2024-08-03T20:50:37.750 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:37.746+0000 7f32afe00640 1 --2- 172.21.15.32:0/135191222 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x559ef8d55390 0x559ef91bc4e0 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7f32a80012e0 tx=0x7f32a8001d00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c820817a7baabc6 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:50:37.750 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:37.746+0000 7f3298c00640 10 client.0 ms_handle_connect on v2:172.21.15.32:3300/0 2024-08-03T20:50:37.750 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:37.746+0000 7f3298c00640 1 -- 172.21.15.32:0/135191222 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32a8018050 con 0x559ef8d55390 2024-08-03T20:50:37.750 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:37.746+0000 7f3298c00640 1 -- 172.21.15.32:0/135191222 <== mon.0 v2:172.21.15.32:3300/0 2 ==== osd_map(57..57 src has 1..57) ==== 5709+0+0 (secure 0 0 0) 0x7f32a80050b0 con 0x559ef8d55390 2024-08-03T20:50:37.750 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:37.746+0000 7f32bd630740 1 -- 172.21.15.32:0/135191222 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f32a8003a60 con 0x559ef8d55390 2024-08-03T20:50:37.750 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:37.747+0000 7f3298c00640 1 -- 172.21.15.32:0/135191222 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_get_version_reply(handle=1 version=51) ==== 24+0+0 (secure 0 0 0) 0x7f32a8022040 con 0x559ef8d55390 2024-08-03T20:50:37.750 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:37.747+0000 7f32bd630740 10 client.6012 fetch_fsmap learned FSMap version 51 2024-08-03T20:50:37.750 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:37.747+0000 7f32bd630740 1 -- 172.21.15.32:0/135191222 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,fsmap=51,monmap=2+}) -- 0x559ef9137f00 con 0x559ef8d55390 2024-08-03T20:50:37.750 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:37.747+0000 7f3298c00640 1 -- 172.21.15.32:0/135191222 <== mon.0 v2:172.21.15.32:3300/0 4 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f32a8003a60 con 0x559ef8d55390 2024-08-03T20:50:37.750 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:37.747+0000 7f3298c00640 1 -- 172.21.15.32:0/135191222 <== mon.0 v2:172.21.15.32:3300/0 5 ==== fsmap(e 51) ==== 3502+0+0 (secure 0 0 0) 0x7f32a80053c0 con 0x559ef8d55390 2024-08-03T20:50:37.750 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:37.747+0000 7f32bd630740 10 client.6012 fetch_fsmap finished waiting for FSMap version 51 2024-08-03T20:50:37.751 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:37.747+0000 7f32bd630740 10 client.6012 resolve_mds: resolved 4:0 to role '4:0' aka daemon mds.c 2024-08-03T20:50:37.751 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:37.747+0000 7f32bd630740 20 client.6012 populate_metadata read hostname 'smithi032' 2024-08-03T20:50:37.751 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:37.747+0000 7f32bd630740 1 --2- 172.21.15.32:0/135191222 >> [v2:172.21.15.32:6834/3848976369,v1:172.21.15.32:6835/3848976369] conn(0x559ef9140f90 0x559ef9125900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:50:37.751 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:37.747+0000 7f32bd630740 4 client.6012 mds_command: new command op to 5841 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-08-03T20:50:37.751 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:37.747+0000 7f32bd630740 1 -- 172.21.15.32:0/135191222 --> [v2:172.21.15.32:6834/3848976369,v1:172.21.15.32:6835/3848976369] -- command(tid 0: {"prefix": "flush journal"}) -- 0x559ef906be60 con 0x559ef9140f90 2024-08-03T20:50:37.751 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:37.747+0000 7f32af400640 1 --2- 172.21.15.32:0/135191222 >> [v2:172.21.15.32:6834/3848976369,v1:172.21.15.32:6835/3848976369] conn(0x559ef9140f90 0x559ef9125900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:50:37.751 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:37.748+0000 7f32af400640 1 --2- 172.21.15.32:0/135191222 >> [v2:172.21.15.32:6834/3848976369,v1:172.21.15.32:6835/3848976369] conn(0x559ef9140f90 0x559ef9125900 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ded28006a0600c13 server_cookie=f15421334f52bd11 in_seq=0 out_seq=0 2024-08-03T20:50:37.751 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:37.748+0000 7f3298c00640 10 client.6012 ms_handle_connect on v2:172.21.15.32:6834/3848976369 2024-08-03T20:50:37.762 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:37.761+0000 7f3298c00640 1 -- 172.21.15.32:0/135191222 <== mds.0 v2:172.21.15.32:6834/3848976369 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x559ef906be60 con 0x559ef9140f90 2024-08-03T20:50:37.762 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:37.761+0000 7f3298c00640 10 client.6012 handle_command_reply: tid=0 2024-08-03T20:50:37.762 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:37.761+0000 7f32bd630740 1 client.6012 shutdown 2024-08-03T20:50:37.762 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:37.761+0000 7f32bd630740 1 -- 172.21.15.32:0/135191222 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x559ef8d55390 msgr2=0x559ef91bc4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:50:37.762 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:37.761+0000 7f32bd630740 1 --2- 172.21.15.32:0/135191222 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x559ef8d55390 0x559ef91bc4e0 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7f32a80012e0 tx=0x7f32a8001d00 comp rx=0 tx=0).stop 2024-08-03T20:50:37.762 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:37.761+0000 7f32bd630740 1 -- 172.21.15.32:0/135191222 shutdown_connections 2024-08-03T20:50:37.763 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:37.761+0000 7f32bd630740 1 --2- 172.21.15.32:0/135191222 >> [v2:172.21.15.32:6834/3848976369,v1:172.21.15.32:6835/3848976369] conn(0x559ef9140f90 0x559ef9125900 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:37.763 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:37.761+0000 7f32bd630740 1 --2- 172.21.15.32:0/135191222 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x559ef8d55390 0x559ef91bc4e0 unknown :-1 s=CLOSED pgs=602 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:37.763 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:37.761+0000 7f32bd630740 1 --2- 172.21.15.32:0/135191222 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x559ef8d346b0 0x559ef915a090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:37.763 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:37.761+0000 7f32bd630740 1 --2- 172.21.15.32:0/135191222 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x559ef8d2c4d0 0x559ef91374f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:37.763 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:37.761+0000 7f3298c00640 0 client.6012 ms_handle_reset on v2:172.21.15.32:6834/3848976369 2024-08-03T20:50:37.763 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:37.761+0000 7f32bd630740 1 -- 172.21.15.32:0/135191222 >> 172.21.15.32:0/135191222 conn(0x559ef8d494e0 msgr2=0x559ef9138cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:50:37.763 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:37.762+0000 7f32bd630740 1 -- 172.21.15.32:0/135191222 shutdown_connections 2024-08-03T20:50:37.763 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:37.762+0000 7f32bd630740 1 -- 172.21.15.32:0/135191222 wait complete. 2024-08-03T20:50:37.763 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:37.762+0000 7f32bd630740 20 client.6012 trim_cache size 0 max 16384 2024-08-03T20:50:37.763 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-08-03T20:50:37.764 INFO:teuthology.orchestra.run.smithi032.stdout: "message": "", 2024-08-03T20:50:37.764 INFO:teuthology.orchestra.run.smithi032.stdout: "return_code": 0 2024-08-03T20:50:37.764 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-08-03T20:50:37.764 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:37.762+0000 7f32b6600640 1 -- 172.21.15.32:0/315550581 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f3290077010 msgr2=0x7f3290079520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:50:37.765 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:37.762+0000 7f32b6600640 1 --2- 172.21.15.32:0/315550581 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f3290077010 0x7f3290079520 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f329c033000 tx=0x7f329c004730 comp rx=0 tx=0).stop 2024-08-03T20:50:37.765 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:37.762+0000 7f32b6600640 1 -- 172.21.15.32:0/315550581 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f32b00cccf0 msgr2=0x7f32b01f8710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:50:37.765 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:37.762+0000 7f32b6600640 1 --2- 172.21.15.32:0/315550581 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f32b00cccf0 0x7f32b01f8710 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7f32a000d700 tx=0x7f32a000dbc0 comp rx=0 tx=0).stop 2024-08-03T20:50:37.765 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:37.763+0000 7f32b6600640 1 -- 172.21.15.32:0/315550581 shutdown_connections 2024-08-03T20:50:37.765 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:37.763+0000 7f32b6600640 1 --2- 172.21.15.32:0/315550581 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f3290077010 0x7f3290079520 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:37.765 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:37.763+0000 7f32b6600640 1 --2- 172.21.15.32:0/315550581 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f32b00ced30 0x7f32b0158c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:37.765 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:37.763+0000 7f32b6600640 1 --2- 172.21.15.32:0/315550581 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f32b0145c50 0x7f32b0072ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:37.765 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:37.763+0000 7f32b6600640 1 --2- 172.21.15.32:0/315550581 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f32b00cccf0 0x7f32b01f8710 unknown :-1 s=CLOSED pgs=598 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:37.765 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:37.763+0000 7f32b6600640 1 -- 172.21.15.32:0/315550581 >> 172.21.15.32:0/315550581 conn(0x7f32b00b1bd0 msgr2=0x7f32b00cff10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:50:37.765 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:37.763+0000 7f32b6600640 1 -- 172.21.15.32:0/315550581 shutdown_connections 2024-08-03T20:50:37.765 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:37.763+0000 7f32b6600640 1 -- 172.21.15.32:0/315550581 wait complete. 2024-08-03T20:50:37.789 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-08-03T20:50:38.018 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:38.017+0000 7f9045200640 1 -- 172.21.15.32:0/1991267530 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f90400d29a0 msgr2=0x7f90400d1c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:50:38.018 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:38.017+0000 7f9045200640 1 --2- 172.21.15.32:0/1991267530 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f90400d29a0 0x7f90400d1c60 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7f90400ba100 tx=0x7f902802a4b0 comp rx=0 tx=0).stop 2024-08-03T20:50:38.018 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:38.017+0000 7f9045200640 1 -- 172.21.15.32:0/1991267530 shutdown_connections 2024-08-03T20:50:38.018 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:38.017+0000 7f9045200640 1 --2- 172.21.15.32:0/1991267530 >> v2:172.21.15.118:3300/0 conn(0x7f90400cec30 0x7f90400cf040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:38.018 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:38.017+0000 7f9045200640 1 --2- 172.21.15.32:0/1991267530 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f90400d29a0 0x7f90400d1c60 unknown :-1 s=CLOSED pgs=604 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:38.018 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:38.017+0000 7f9045200640 1 -- 172.21.15.32:0/1991267530 >> 172.21.15.32:0/1991267530 conn(0x7f90400d4090 msgr2=0x7f90400d44a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:50:38.019 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:38.017+0000 7f9045200640 1 -- 172.21.15.32:0/1991267530 shutdown_connections 2024-08-03T20:50:38.019 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:38.017+0000 7f9045200640 1 -- 172.21.15.32:0/1991267530 wait complete. 2024-08-03T20:50:38.020 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:38.019+0000 7f9045200640 1 Processor -- start 2024-08-03T20:50:38.020 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:38.020+0000 7f9045200640 1 -- start start 2024-08-03T20:50:38.022 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:38.021+0000 7f9045200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f90400cec30 0x7f90400a1740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:50:38.022 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:38.021+0000 7f9045200640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f90400d4bb0 0x7f9040167180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:50:38.022 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:38.021+0000 7f9045200640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f90400cc710 0x7f904015d5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:50:38.022 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:38.021+0000 7f9045200640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f90400cfa80 con 0x7f90400cec30 2024-08-03T20:50:38.022 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:38.021+0000 7f9045200640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f90400cf900 con 0x7f90400cc710 2024-08-03T20:50:38.023 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:38.021+0000 7f9045200640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7f90400cfc00 con 0x7f90400d4bb0 2024-08-03T20:50:38.023 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:38.021+0000 7f903f400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f90400cec30 0x7f90400a1740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:50:38.023 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:38.021+0000 7f903f400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f90400cec30 0x7f90400a1740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:45262/0 (socket says 172.21.15.32:45262) 2024-08-03T20:50:38.023 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:38.021+0000 7f903f400640 1 -- 172.21.15.32:0/7910219 learned_addr learned my addr 172.21.15.32:0/7910219 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:50:38.023 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:38.021+0000 7f9037e00640 1 --2- 172.21.15.32:0/7910219 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f90400d4bb0 0x7f9040167180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:50:38.023 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:38.021+0000 7f903fe00640 1 --2- 172.21.15.32:0/7910219 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f90400cc710 0x7f904015d5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:50:38.023 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:38.022+0000 7f903f400640 1 -- 172.21.15.32:0/7910219 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f90400d4bb0 msgr2=0x7f9040167180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:50:38.023 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:38.022+0000 7f903f400640 1 --2- 172.21.15.32:0/7910219 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f90400d4bb0 0x7f9040167180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:38.023 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:38.022+0000 7f903f400640 1 -- 172.21.15.32:0/7910219 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f90400cc710 msgr2=0x7f904015d5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:50:38.023 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:38.022+0000 7f903f400640 1 --2- 172.21.15.32:0/7910219 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f90400cc710 0x7f904015d5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:38.023 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:38.022+0000 7f903f400640 1 -- 172.21.15.32:0/7910219 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f90400cce40 con 0x7f90400cec30 2024-08-03T20:50:38.024 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:38.022+0000 7f903f400640 1 --2- 172.21.15.32:0/7910219 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f90400cec30 0x7f90400a1740 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7f903000c3d0 tx=0x7f903000c890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:50:38.024 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:38.022+0000 7f9037400640 1 -- 172.21.15.32:0/7910219 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9030011790 con 0x7f90400cec30 2024-08-03T20:50:38.024 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:38.022+0000 7f9037400640 1 -- 172.21.15.32:0/7910219 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f9030017070 con 0x7f90400cec30 2024-08-03T20:50:38.024 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:38.022+0000 7f9037400640 1 -- 172.21.15.32:0/7910219 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9030011cc0 con 0x7f90400cec30 2024-08-03T20:50:38.024 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:38.023+0000 7f9045200640 1 -- 172.21.15.32:0/7910219 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f90400cd120 con 0x7f90400cec30 2024-08-03T20:50:38.024 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:38.023+0000 7f9045200640 1 -- 172.21.15.32:0/7910219 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9040166c60 con 0x7f90400cec30 2024-08-03T20:50:38.024 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:38.024+0000 7f9045200640 1 -- 172.21.15.32:0/7910219 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f90400d49d0 con 0x7f90400cec30 2024-08-03T20:50:38.025 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:38.025+0000 7f9037400640 1 -- 172.21.15.32:0/7910219 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f903001c460 con 0x7f90400cec30 2024-08-03T20:50:38.026 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:38.026+0000 7f9037400640 1 --2- 172.21.15.32:0/7910219 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f901c077040 0x7f901c079550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:50:38.026 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:38.026+0000 7f9037400640 1 -- 172.21.15.32:0/7910219 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(57..57 src has 1..57) ==== 5709+0+0 (secure 0 0 0) 0x7f9030098650 con 0x7f90400cec30 2024-08-03T20:50:38.027 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:38.026+0000 7f9037e00640 1 --2- 172.21.15.32:0/7910219 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f901c077040 0x7f901c079550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:50:38.027 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:38.027+0000 7f9037e00640 1 --2- 172.21.15.32:0/7910219 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f901c077040 0x7f901c079550 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f9028034000 tx=0x7f9028004730 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:50:38.030 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:38.030+0000 7f9037400640 1 -- 172.21.15.32:0/7910219 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7f90400d49d0 con 0x7f90400cec30 2024-08-03T20:50:38.274 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:38.273+0000 7f9045200640 1 -- 172.21.15.32:0/7910219 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f90400cf040 con 0x7f90400cec30 2024-08-03T20:50:38.274 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:38.274+0000 7f9037400640 1 -- 172.21.15.32:0/7910219 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v51) ==== 53+0+73 (secure 0 0 0) 0x7f90300611f0 con 0x7f90400cec30 2024-08-03T20:50:38.275 INFO:teuthology.orchestra.run.smithi032.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-08-03T20:50:38.277 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:38.276+0000 7f9045200640 1 -- 172.21.15.32:0/7910219 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f901c077040 msgr2=0x7f901c079550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:50:38.277 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:38.276+0000 7f9045200640 1 --2- 172.21.15.32:0/7910219 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f901c077040 0x7f901c079550 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f9028034000 tx=0x7f9028004730 comp rx=0 tx=0).stop 2024-08-03T20:50:38.277 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:38.276+0000 7f9045200640 1 -- 172.21.15.32:0/7910219 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f90400cec30 msgr2=0x7f90400a1740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:50:38.277 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:38.276+0000 7f9045200640 1 --2- 172.21.15.32:0/7910219 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f90400cec30 0x7f90400a1740 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7f903000c3d0 tx=0x7f903000c890 comp rx=0 tx=0).stop 2024-08-03T20:50:38.277 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:38.276+0000 7f9045200640 1 -- 172.21.15.32:0/7910219 shutdown_connections 2024-08-03T20:50:38.277 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:38.276+0000 7f9045200640 1 --2- 172.21.15.32:0/7910219 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f901c077040 0x7f901c079550 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:38.278 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:38.276+0000 7f9045200640 1 --2- 172.21.15.32:0/7910219 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f90400cc710 0x7f904015d5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:38.278 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:38.276+0000 7f9045200640 1 --2- 172.21.15.32:0/7910219 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f90400d4bb0 0x7f9040167180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:38.278 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:38.276+0000 7f9045200640 1 --2- 172.21.15.32:0/7910219 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f90400cec30 0x7f90400a1740 unknown :-1 s=CLOSED pgs=605 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:38.278 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:38.276+0000 7f9045200640 1 -- 172.21.15.32:0/7910219 >> 172.21.15.32:0/7910219 conn(0x7f90400d4090 msgr2=0x7f90400b64d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:50:38.278 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:38.277+0000 7f9045200640 1 -- 172.21.15.32:0/7910219 shutdown_connections 2024-08-03T20:50:38.278 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:38.277+0000 7f9045200640 1 -- 172.21.15.32:0/7910219 wait complete. 2024-08-03T20:50:38.299 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-08-03T20:50:38.299 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-08-03T20:50:38.299 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi032.front.sepia.ceph.com 2024-08-03T20:50:38.299 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-08-03T20:50:38.299 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-08-03T20:50:38.299 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-08-03T20:50:38.299 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2024-08-03T20:50:38.299 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-08-03T20:50:38.299 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-08-03T20:50:38.300 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-08-03T20:50:38.300 DEBUG:teuthology.orchestra.run.smithi032:> ip addr 2024-08-03T20:50:38.335 INFO:teuthology.orchestra.run.smithi032.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-08-03T20:50:38.336 INFO:teuthology.orchestra.run.smithi032.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-08-03T20:50:38.336 INFO:teuthology.orchestra.run.smithi032.stdout: inet 127.0.0.1/8 scope host lo 2024-08-03T20:50:38.336 INFO:teuthology.orchestra.run.smithi032.stdout: valid_lft forever preferred_lft forever 2024-08-03T20:50:38.336 INFO:teuthology.orchestra.run.smithi032.stdout: inet6 ::1/128 scope host 2024-08-03T20:50:38.336 INFO:teuthology.orchestra.run.smithi032.stdout: valid_lft forever preferred_lft forever 2024-08-03T20:50:38.336 INFO:teuthology.orchestra.run.smithi032.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-08-03T20:50:38.336 INFO:teuthology.orchestra.run.smithi032.stdout: link/ether 0c:c4:7a:6b:fd:0c brd ff:ff:ff:ff:ff:ff 2024-08-03T20:50:38.336 INFO:teuthology.orchestra.run.smithi032.stdout: altname enp7s0f0 2024-08-03T20:50:38.336 INFO:teuthology.orchestra.run.smithi032.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-08-03T20:50:38.336 INFO:teuthology.orchestra.run.smithi032.stdout: link/ether 0c:c4:7a:6b:fd:0d brd ff:ff:ff:ff:ff:ff 2024-08-03T20:50:38.336 INFO:teuthology.orchestra.run.smithi032.stdout: altname enp7s0f1 2024-08-03T20:50:38.336 INFO:teuthology.orchestra.run.smithi032.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-08-03T20:50:38.337 INFO:teuthology.orchestra.run.smithi032.stdout: link/ether ac:1f:6b:f8:26:02 brd ff:ff:ff:ff:ff:ff 2024-08-03T20:50:38.337 INFO:teuthology.orchestra.run.smithi032.stdout: altname enp1s0f0 2024-08-03T20:50:38.337 INFO:teuthology.orchestra.run.smithi032.stdout: inet 172.21.15.32/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-08-03T20:50:38.337 INFO:teuthology.orchestra.run.smithi032.stdout: valid_lft 42318sec preferred_lft 42318sec 2024-08-03T20:50:38.337 INFO:teuthology.orchestra.run.smithi032.stdout: inet6 fe80::ae1f:6bff:fef8:2602/64 scope link noprefixroute 2024-08-03T20:50:38.337 INFO:teuthology.orchestra.run.smithi032.stdout: valid_lft forever preferred_lft forever 2024-08-03T20:50:38.337 INFO:teuthology.orchestra.run.smithi032.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-08-03T20:50:38.337 INFO:teuthology.orchestra.run.smithi032.stdout: link/ether ac:1f:6b:f8:26:03 brd ff:ff:ff:ff:ff:ff 2024-08-03T20:50:38.337 INFO:teuthology.orchestra.run.smithi032.stdout: altname enp1s0f1 2024-08-03T20:50:38.337 INFO:teuthology.orchestra.run.smithi032.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-08-03T20:50:38.337 INFO:teuthology.orchestra.run.smithi032.stdout: link/ether 0a:23:2b:ca:2a:93 brd ff:ff:ff:ff:ff:ff 2024-08-03T20:50:38.337 INFO:teuthology.orchestra.run.smithi032.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-08-03T20:50:38.337 INFO:teuthology.orchestra.run.smithi032.stdout: valid_lft forever preferred_lft forever 2024-08-03T20:50:38.338 INFO:teuthology.orchestra.run.smithi032.stdout: inet6 fe80::d804:7cff:fed9:7ba4/64 scope link 2024-08-03T20:50:38.338 INFO:teuthology.orchestra.run.smithi032.stdout: valid_lft forever preferred_lft forever 2024-08-03T20:50:38.338 INFO:teuthology.orchestra.run.smithi032.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-08-03T20:50:38.338 INFO:teuthology.orchestra.run.smithi032.stdout: link/ether 22:86:28:30:4e:62 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-08-03T20:50:38.338 INFO:teuthology.orchestra.run.smithi032.stdout: inet6 fe80::2086:28ff:fe30:4e62/64 scope link 2024-08-03T20:50:38.338 INFO:teuthology.orchestra.run.smithi032.stdout: valid_lft forever preferred_lft forever 2024-08-03T20:50:38.338 INFO:teuthology.orchestra.run.smithi032.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-08-03T20:50:38.338 INFO:teuthology.orchestra.run.smithi032.stdout: link/ether 0a:23:2b:ca:2a:93 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-08-03T20:50:38.338 INFO:teuthology.orchestra.run.smithi032.stdout: inet6 fe80::823:2bff:feca:2a93/64 scope link 2024-08-03T20:50:38.338 INFO:teuthology.orchestra.run.smithi032.stdout: valid_lft forever preferred_lft forever 2024-08-03T20:50:38.339 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-08-03T20:50:38.339 DEBUG:teuthology.orchestra.run.smithi032:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-08-03T20:50:38.431 INFO:teuthology.orchestra.run.smithi032.stdout:1 2024-08-03T20:50:38.434 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-08-03T20:50:38.434 DEBUG:teuthology.orchestra.run.smithi032:> route 2024-08-03T20:50:38.470 INFO:teuthology.orchestra.run.smithi032.stdout:Kernel IP routing table 2024-08-03T20:50:38.470 INFO:teuthology.orchestra.run.smithi032.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-08-03T20:50:38.471 INFO:teuthology.orchestra.run.smithi032.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-08-03T20:50:38.471 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-08-03T20:50:38.471 INFO:teuthology.orchestra.run.smithi032.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-08-03T20:50:38.471 DEBUG:teuthology.orchestra.run.smithi032:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-08-03T20:50:38.471 DEBUG:teuthology.orchestra.run.smithi032:> set -e 2024-08-03T20:50:38.471 DEBUG:teuthology.orchestra.run.smithi032:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-08-03T20:50:38.471 DEBUG:teuthology.orchestra.run.smithi032:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-08-03T20:50:38.471 DEBUG:teuthology.orchestra.run.smithi032:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-08-03T20:50:38.472 DEBUG:teuthology.orchestra.run.smithi032:> ') 2024-08-03T20:50:38.589 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:38 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-08-03T20:50:38.661 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:38 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-08-03T20:50:38.669 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-08-03T20:50:38.670 DEBUG:teuthology.orchestra.run.smithi032:> ip netns list 2024-08-03T20:50:38.705 INFO:teuthology.orchestra.run.smithi032.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-08-03T20:50:38.705 INFO:teuthology.orchestra.run.smithi032.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-08-03T20:50:38.706 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-08-03T20:50:38.706 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-08-03T20:50:38.706 DEBUG:teuthology.orchestra.run.smithi032:> sudo ip link set brx.0 up 2024-08-03T20:50:38.802 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-08-03T20:50:38.802 DEBUG:teuthology.orchestra.run.smithi032:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-08-03T20:50:38.838 INFO:teuthology.orchestra.run.smithi032.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-08-03T20:50:38.838 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-08-03T20:50:38.838 DEBUG:teuthology.orchestra.run.smithi032:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-08-03T20:50:38.913 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2024-08-03T20:50:38.913 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-08-03T20:50:38.913 DEBUG:teuthology.orchestra.run.smithi032:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-08-03T20:50:39.111 INFO:teuthology.orchestra.run.smithi032.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-08-03T20:50:39.111 INFO:teuthology.orchestra.run.smithi032.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-08-03T20:50:39.111 INFO:teuthology.orchestra.run.smithi032.stdout:invalid new device string format 2024-08-03T20:50:39.132 INFO:teuthology.orchestra.run.smithi032.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-08-03T20:50:39.132 INFO:teuthology.orchestra.run.smithi032.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-08-03T20:50:39.132 INFO:teuthology.orchestra.run.smithi032.stdout:invalid new device string format 2024-08-03T20:50:39.132 INFO:teuthology.orchestra.run.smithi032.stdout:mount.ceph: resolved to: "172.21.15.32:3300,172.21.15.118:3300,172.21.15.118:3301" 2024-08-03T20:50:39.132 INFO:teuthology.orchestra.run.smithi032.stdout:mount.ceph: switching to using v1 address with old syntax 2024-08-03T20:50:39.161 INFO:teuthology.orchestra.run.smithi032.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-08-03T20:50:39.161 INFO:teuthology.orchestra.run.smithi032.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-08-03T20:50:39.161 INFO:teuthology.orchestra.run.smithi032.stdout:invalid new device string format 2024-08-03T20:50:39.161 INFO:teuthology.orchestra.run.smithi032.stdout:mount.ceph: resolved to: "172.21.15.32:3300,172.21.15.118:3300,172.21.15.118:3301" 2024-08-03T20:50:39.161 INFO:teuthology.orchestra.run.smithi032.stdout:mount.ceph: switching to using v1 address with old syntax 2024-08-03T20:50:39.161 INFO:teuthology.orchestra.run.smithi032.stdout:mount.ceph: resolved to: "172.21.15.32:6789,172.21.15.118:6789,172.21.15.118:6790" 2024-08-03T20:50:39.161 INFO:teuthology.orchestra.run.smithi032.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.32:6789,172.21.15.118:6789,172.21.15.118:6790:/ 2024-08-03T20:50:39.162 INFO:teuthology.orchestra.run.smithi032.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,key=0,fsid=40ab141a-01d7-49b9-ae3c-2edec9876e95" will pass to kernel 2024-08-03T20:50:39.162 INFO:teuthology.orchestra.run.smithi032.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2024-08-03T20:50:39.162 INFO:teuthology.orchestra.run.smithi032.stdout: You just mounted a file system that supports labels which does not 2024-08-03T20:50:39.162 INFO:teuthology.orchestra.run.smithi032.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-08-03T20:50:39.163 INFO:teuthology.orchestra.run.smithi032.stdout: applications will generate AVC messages and not be allowed access to 2024-08-03T20:50:39.163 INFO:teuthology.orchestra.run.smithi032.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-08-03T20:50:39.165 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-08-03T20:50:39.166 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-08-03T20:50:39.166 DEBUG:teuthology.orchestra.run.smithi032:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-08-03T20:50:39.223 DEBUG:teuthology.orchestra.run.smithi032:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-08-03T20:50:39.303 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:39 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-08-03T20:50:39.305 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:39 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-08-03T20:50:39.314 INFO:teuthology.orchestra.run.smithi032.stdout:client6024 2024-08-03T20:50:39.314 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-08-03T20:50:39.314 DEBUG:teuthology.orchestra.run.smithi032:> dd if=/proc/self/mounts of=/dev/stdout 2024-08-03T20:50:39.350 DEBUG:teuthology.orchestra.run.smithi032:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'chmod 711 a/b/d; sync .') 2024-08-03T20:50:39.469 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:39 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-08-03T20:50:39.476 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:39 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-08-03T20:50:39.484 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-08-03T20:50:39.484 DEBUG:teuthology.orchestra.run.smithi032:> dd if=/proc/self/mounts of=/dev/stdout 2024-08-03T20:50:39.520 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-08-03T20:50:39.791 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:39.790+0000 7efc1c400640 1 -- 172.21.15.32:0/1991546363 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efc14144f90 msgr2=0x7efc14074570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:50:39.792 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:39.790+0000 7efc1c400640 1 --2- 172.21.15.32:0/1991546363 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efc14144f90 0x7efc14074570 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7efc14073b50 tx=0x7efc0002a510 comp rx=0 tx=0).stop 2024-08-03T20:50:39.792 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:39.791+0000 7efc1c400640 1 -- 172.21.15.32:0/1991546363 shutdown_connections 2024-08-03T20:50:39.792 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:39.791+0000 7efc1c400640 1 --2- 172.21.15.32:0/1991546363 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efc14144f90 0x7efc14074570 unknown :-1 s=CLOSED pgs=608 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:39.792 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:39.791+0000 7efc1c400640 1 -- 172.21.15.32:0/1991546363 >> 172.21.15.32:0/1991546363 conn(0x7efc140b1bd0 msgr2=0x7efc140b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:50:39.792 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:39.791+0000 7efc1c400640 1 -- 172.21.15.32:0/1991546363 shutdown_connections 2024-08-03T20:50:39.792 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:39.791+0000 7efc1c400640 1 -- 172.21.15.32:0/1991546363 wait complete. 2024-08-03T20:50:39.793 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:39.793+0000 7efc1c400640 1 Processor -- start 2024-08-03T20:50:39.793 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:39.793+0000 7efc1c400640 1 -- start start 2024-08-03T20:50:39.795 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:39.794+0000 7efc1c400640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7efc1406e930 0x7efc14153bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:50:39.795 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:39.795+0000 7efc1c400640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7efc14144f90 0x7efc1415da90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:50:39.795 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:39.795+0000 7efc1c400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efc1415fff0 0x7efc1415dfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:50:39.795 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:39.795+0000 7efc1c400640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7efc140cf840 con 0x7efc1415fff0 2024-08-03T20:50:39.796 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:39.795+0000 7efc1c400640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7efc14145f30 con 0x7efc1406e930 2024-08-03T20:50:39.796 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:39.795+0000 7efc1c400640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7efc140cf9c0 con 0x7efc14144f90 2024-08-03T20:50:39.796 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:39.795+0000 7efc1a400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efc1415fff0 0x7efc1415dfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:50:39.796 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:39.795+0000 7efc1a400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efc1415fff0 0x7efc1415dfd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:46314/0 (socket says 172.21.15.32:46314) 2024-08-03T20:50:39.796 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:39.795+0000 7efc1a400640 1 -- 172.21.15.32:0/4277195464 learned_addr learned my addr 172.21.15.32:0/4277195464 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:50:39.796 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:39.795+0000 7efc19000640 1 --2- 172.21.15.32:0/4277195464 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7efc14144f90 0x7efc1415da90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:50:39.796 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:39.795+0000 7efc19a00640 1 --2- 172.21.15.32:0/4277195464 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7efc1406e930 0x7efc14153bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:50:39.798 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:39.795+0000 7efc1a400640 1 -- 172.21.15.32:0/4277195464 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7efc14144f90 msgr2=0x7efc1415da90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:50:39.798 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:39.795+0000 7efc1a400640 1 --2- 172.21.15.32:0/4277195464 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7efc14144f90 0x7efc1415da90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:39.798 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:39.795+0000 7efc1a400640 1 -- 172.21.15.32:0/4277195464 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7efc1406e930 msgr2=0x7efc14153bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:50:39.798 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:39.795+0000 7efc1a400640 1 --2- 172.21.15.32:0/4277195464 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7efc1406e930 0x7efc14153bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:39.798 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:39.795+0000 7efc1a400640 1 -- 172.21.15.32:0/4277195464 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efc140af950 con 0x7efc1415fff0 2024-08-03T20:50:39.798 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:39.795+0000 7efc19000640 1 --2- 172.21.15.32:0/4277195464 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7efc14144f90 0x7efc1415da90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-03T20:50:39.798 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:39.795+0000 7efc19a00640 1 --2- 172.21.15.32:0/4277195464 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7efc1406e930 0x7efc14153bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-03T20:50:39.798 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:39.796+0000 7efc1a400640 1 --2- 172.21.15.32:0/4277195464 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efc1415fff0 0x7efc1415dfd0 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7efc0c00b970 tx=0x7efc0c00be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:50:39.798 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:39.796+0000 7efc11600640 1 -- 172.21.15.32:0/4277195464 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc0c019070 con 0x7efc1415fff0 2024-08-03T20:50:39.799 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:39.796+0000 7efc11600640 1 -- 172.21.15.32:0/4277195464 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7efc0c00cc30 con 0x7efc1415fff0 2024-08-03T20:50:39.799 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:39.796+0000 7efc1c400640 1 -- 172.21.15.32:0/4277195464 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efc140af670 con 0x7efc1415fff0 2024-08-03T20:50:39.799 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:39.796+0000 7efc11600640 1 -- 172.21.15.32:0/4277195464 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc0c0146a0 con 0x7efc1415fff0 2024-08-03T20:50:39.799 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:39.796+0000 7efc1c400640 1 -- 172.21.15.32:0/4277195464 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efc140ae6c0 con 0x7efc1415fff0 2024-08-03T20:50:39.799 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:39.797+0000 7efc1c400640 1 -- 172.21.15.32:0/4277195464 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efc140af3e0 con 0x7efc1415fff0 2024-08-03T20:50:39.800 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:39.798+0000 7efc11600640 1 -- 172.21.15.32:0/4277195464 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7efc0c01c070 con 0x7efc1415fff0 2024-08-03T20:50:39.800 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:39.799+0000 7efc11600640 1 --2- 172.21.15.32:0/4277195464 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7efbf4077010 0x7efbf4079520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:50:39.800 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:39.800+0000 7efc11600640 1 -- 172.21.15.32:0/4277195464 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(57..57 src has 1..57) ==== 5709+0+0 (secure 0 0 0) 0x7efc0c098640 con 0x7efc1415fff0 2024-08-03T20:50:39.800 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:39.800+0000 7efc19a00640 1 --2- 172.21.15.32:0/4277195464 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7efbf4077010 0x7efbf4079520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:50:39.802 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:39.801+0000 7efc19a00640 1 --2- 172.21.15.32:0/4277195464 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7efbf4077010 0x7efbf4079520 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7efc08000f60 tx=0x7efc08009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:50:39.804 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:39.804+0000 7efc11600640 1 -- 172.21.15.32:0/4277195464 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7efc0c0617b0 con 0x7efc1415fff0 2024-08-03T20:50:40.021 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:40.021+0000 7efc1c400640 1 -- 172.21.15.32:0/4277195464 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7efc14160400 con 0x7efc1415fff0 2024-08-03T20:50:40.022 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:40.021+0000 7efc11600640 1 -- 172.21.15.32:0/4277195464 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v57) ==== 80+0+0 (secure 0 0 0) 0x7efc0c0610b0 con 0x7efc1415fff0 2024-08-03T20:50:40.022 INFO:teuthology.orchestra.run.smithi032.stderr:listed 0 entries 2024-08-03T20:50:40.025 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:40.024+0000 7efc1c400640 1 -- 172.21.15.32:0/4277195464 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7efbf4077010 msgr2=0x7efbf4079520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:50:40.025 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:40.024+0000 7efc1c400640 1 --2- 172.21.15.32:0/4277195464 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7efbf4077010 0x7efbf4079520 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7efc08000f60 tx=0x7efc08009040 comp rx=0 tx=0).stop 2024-08-03T20:50:40.025 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:40.024+0000 7efc1c400640 1 -- 172.21.15.32:0/4277195464 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efc1415fff0 msgr2=0x7efc1415dfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:50:40.025 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:40.024+0000 7efc1c400640 1 --2- 172.21.15.32:0/4277195464 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efc1415fff0 0x7efc1415dfd0 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7efc0c00b970 tx=0x7efc0c00be30 comp rx=0 tx=0).stop 2024-08-03T20:50:40.025 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:40.024+0000 7efc1c400640 1 -- 172.21.15.32:0/4277195464 shutdown_connections 2024-08-03T20:50:40.025 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:40.024+0000 7efc1c400640 1 --2- 172.21.15.32:0/4277195464 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7efbf4077010 0x7efbf4079520 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:40.025 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:40.024+0000 7efc1c400640 1 --2- 172.21.15.32:0/4277195464 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efc1415fff0 0x7efc1415dfd0 unknown :-1 s=CLOSED pgs=609 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:40.025 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:40.024+0000 7efc1c400640 1 --2- 172.21.15.32:0/4277195464 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7efc14144f90 0x7efc1415da90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:40.025 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:40.024+0000 7efc1c400640 1 --2- 172.21.15.32:0/4277195464 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7efc1406e930 0x7efc14153bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:40.026 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:40.024+0000 7efc1c400640 1 -- 172.21.15.32:0/4277195464 >> 172.21.15.32:0/4277195464 conn(0x7efc140b1bd0 msgr2=0x7efc140d0920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:50:40.026 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:40.024+0000 7efc1c400640 1 -- 172.21.15.32:0/4277195464 shutdown_connections 2024-08-03T20:50:40.026 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:40.024+0000 7efc1c400640 1 -- 172.21.15.32:0/4277195464 wait complete. 2024-08-03T20:50:40.050 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-08-03T20:50:40.050 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-08-03T20:50:40.050 DEBUG:teuthology.orchestra.run.smithi032:> sudo umount /home/ubuntu/cephtest/mnt.0 2024-08-03T20:50:40.116 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi032.front.sepia.ceph.com 2024-08-03T20:50:40.116 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-08-03T20:50:40.116 DEBUG:teuthology.orchestra.run.smithi032:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-08-03T20:50:40.152 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-08-03T20:50:40.421 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:40.421+0000 7f76cbe00640 1 -- 172.21.15.32:0/3872048470 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f76cc0cffe0 msgr2=0x7f76cc0b3e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:50:40.422 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:40.421+0000 7f76cbe00640 1 --2- 172.21.15.32:0/3872048470 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f76cc0cffe0 0x7f76cc0b3e90 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7f76b4005fd0 tx=0x7f76b402a540 comp rx=0 tx=0).stop 2024-08-03T20:50:40.422 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:40.421+0000 7f76cbe00640 1 -- 172.21.15.32:0/3872048470 shutdown_connections 2024-08-03T20:50:40.422 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:40.421+0000 7f76cbe00640 1 --2- 172.21.15.32:0/3872048470 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f76cc0b43d0 0x7f76cc0cf510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:40.422 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:40.421+0000 7f76cbe00640 1 --2- 172.21.15.32:0/3872048470 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f76cc0cffe0 0x7f76cc0b3e90 unknown :-1 s=CLOSED pgs=610 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:40.422 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:40.421+0000 7f76cbe00640 1 -- 172.21.15.32:0/3872048470 >> 172.21.15.32:0/3872048470 conn(0x7f76cc08d7a0 msgr2=0x7f76cc09d3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:50:40.422 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:40.421+0000 7f76cbe00640 1 -- 172.21.15.32:0/3872048470 shutdown_connections 2024-08-03T20:50:40.422 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:40.421+0000 7f76cbe00640 1 -- 172.21.15.32:0/3872048470 wait complete. 2024-08-03T20:50:40.423 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:40.423+0000 7f76cbe00640 1 Processor -- start 2024-08-03T20:50:40.423 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:40.423+0000 7f76cbe00640 1 -- start start 2024-08-03T20:50:40.426 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:40.425+0000 7f76cbe00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f76cc0b43d0 0x7f76cc0ca4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:50:40.426 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:40.425+0000 7f76cbe00640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f76cc0cffe0 0x7f76cc0cc4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:50:40.426 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:40.425+0000 7f76cbe00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f76cc0d26e0 0x7f76cc1603f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:50:40.426 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:40.425+0000 7f76cbe00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f76cc0d4100 con 0x7f76cc0d26e0 2024-08-03T20:50:40.426 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:40.425+0000 7f76cbe00640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f76cc0d3f80 con 0x7f76cc0b43d0 2024-08-03T20:50:40.426 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:40.425+0000 7f76cbe00640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7f76cc0d4280 con 0x7f76cc0cffe0 2024-08-03T20:50:40.426 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:40.425+0000 7f76cb400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f76cc0d26e0 0x7f76cc1603f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:50:40.427 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:40.425+0000 7f76cb400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f76cc0d26e0 0x7f76cc1603f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:46326/0 (socket says 172.21.15.32:46326) 2024-08-03T20:50:40.427 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:40.425+0000 7f76cb400640 1 -- 172.21.15.32:0/3044008480 learned_addr learned my addr 172.21.15.32:0/3044008480 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:50:40.427 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:40.425+0000 7f76c3e00640 1 --2- 172.21.15.32:0/3044008480 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f76cc0cffe0 0x7f76cc0cc4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:50:40.427 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:40.425+0000 7f76caa00640 1 --2- 172.21.15.32:0/3044008480 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f76cc0b43d0 0x7f76cc0ca4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:50:40.427 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:40.425+0000 7f76cb400640 1 -- 172.21.15.32:0/3044008480 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f76cc0cffe0 msgr2=0x7f76cc0cc4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:50:40.427 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:40.425+0000 7f76cb400640 1 --2- 172.21.15.32:0/3044008480 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f76cc0cffe0 0x7f76cc0cc4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:40.427 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:40.425+0000 7f76cb400640 1 -- 172.21.15.32:0/3044008480 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f76cc0b43d0 msgr2=0x7f76cc0ca4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:50:40.427 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:40.425+0000 7f76cb400640 1 --2- 172.21.15.32:0/3044008480 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f76cc0b43d0 0x7f76cc0ca4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:40.427 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:40.425+0000 7f76cb400640 1 -- 172.21.15.32:0/3044008480 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f76cc0ccc50 con 0x7f76cc0d26e0 2024-08-03T20:50:40.428 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:40.426+0000 7f76cb400640 1 --2- 172.21.15.32:0/3044008480 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f76cc0d26e0 0x7f76cc1603f0 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7f76c400cad0 tx=0x7f76c400cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:50:40.428 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:40.426+0000 7f76c2a00640 1 -- 172.21.15.32:0/3044008480 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76c400ae20 con 0x7f76cc0d26e0 2024-08-03T20:50:40.428 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:40.426+0000 7f76c2a00640 1 -- 172.21.15.32:0/3044008480 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f76c400f540 con 0x7f76cc0d26e0 2024-08-03T20:50:40.428 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:40.426+0000 7f76c2a00640 1 -- 172.21.15.32:0/3044008480 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76c400f990 con 0x7f76cc0d26e0 2024-08-03T20:50:40.428 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:40.426+0000 7f76cbe00640 1 -- 172.21.15.32:0/3044008480 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f76cc1f5380 con 0x7f76cc0d26e0 2024-08-03T20:50:40.429 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:40.426+0000 7f76cbe00640 1 -- 172.21.15.32:0/3044008480 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f76cc1f5940 con 0x7f76cc0d26e0 2024-08-03T20:50:40.429 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:40.427+0000 7f76cbe00640 1 -- 172.21.15.32:0/3044008480 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f76cc1f5170 con 0x7f76cc0d26e0 2024-08-03T20:50:40.429 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:40.428+0000 7f76c2a00640 1 -- 172.21.15.32:0/3044008480 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f76c400fb30 con 0x7f76cc0d26e0 2024-08-03T20:50:40.430 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:40.429+0000 7f76c2a00640 1 --2- 172.21.15.32:0/3044008480 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f76a8077010 0x7f76a8079520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:50:40.430 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:40.429+0000 7f76c2a00640 1 -- 172.21.15.32:0/3044008480 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(57..57 src has 1..57) ==== 5709+0+0 (secure 0 0 0) 0x7f76c40992a0 con 0x7f76cc0d26e0 2024-08-03T20:50:40.430 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:40.430+0000 7f76caa00640 1 --2- 172.21.15.32:0/3044008480 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f76a8077010 0x7f76a8079520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:50:40.431 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:40.431+0000 7f76caa00640 1 --2- 172.21.15.32:0/3044008480 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f76a8077010 0x7f76a8079520 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f76bc006fd0 tx=0x7f76bc009060 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:50:40.434 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:40.434+0000 7f76c2a00640 1 -- 172.21.15.32:0/3044008480 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7f76c4014030 con 0x7f76cc0d26e0 2024-08-03T20:50:40.678 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:40.678+0000 7f76cbe00640 1 -- 172.21.15.32:0/3044008480 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7f76cc0d46e0 con 0x7f76cc0d26e0 2024-08-03T20:50:40.678 INFO:tasks.ceph.mon.a.smithi032.stderr:2024-08-03T20:50:40.678+0000 7f4e6d600640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-08-03T20:50:40.681 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:40.680+0000 7f76c2a00640 1 -- 172.21.15.32:0/3044008480 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v52) ==== 161+0+0 (secure 0 0 0) 0x7f76c400fe20 con 0x7f76cc0d26e0 2024-08-03T20:50:40.681 INFO:teuthology.orchestra.run.smithi032.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-08-03T20:50:40.683 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:40.683+0000 7f76cbe00640 1 -- 172.21.15.32:0/3044008480 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f76a8077010 msgr2=0x7f76a8079520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:50:40.684 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:40.683+0000 7f76cbe00640 1 --2- 172.21.15.32:0/3044008480 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f76a8077010 0x7f76a8079520 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f76bc006fd0 tx=0x7f76bc009060 comp rx=0 tx=0).stop 2024-08-03T20:50:40.684 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:40.683+0000 7f76cbe00640 1 -- 172.21.15.32:0/3044008480 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f76cc0d26e0 msgr2=0x7f76cc1603f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:50:40.684 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:40.683+0000 7f76cbe00640 1 --2- 172.21.15.32:0/3044008480 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f76cc0d26e0 0x7f76cc1603f0 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7f76c400cad0 tx=0x7f76c400cf90 comp rx=0 tx=0).stop 2024-08-03T20:50:40.684 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:40.684+0000 7f76cbe00640 1 -- 172.21.15.32:0/3044008480 shutdown_connections 2024-08-03T20:50:40.684 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:40.684+0000 7f76cbe00640 1 --2- 172.21.15.32:0/3044008480 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f76a8077010 0x7f76a8079520 secure :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f76bc006fd0 tx=0x7f76bc009060 comp rx=0 tx=0).stop 2024-08-03T20:50:40.684 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:40.684+0000 7f76cbe00640 1 --2- 172.21.15.32:0/3044008480 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f76cc0d26e0 0x7f76cc1603f0 unknown :-1 s=CLOSED pgs=611 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:40.684 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:40.684+0000 7f76cbe00640 1 --2- 172.21.15.32:0/3044008480 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f76cc0cffe0 0x7f76cc0cc4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:40.685 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:40.684+0000 7f76cbe00640 1 --2- 172.21.15.32:0/3044008480 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f76cc0b43d0 0x7f76cc0ca4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:40.685 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:40.684+0000 7f76cbe00640 1 -- 172.21.15.32:0/3044008480 >> 172.21.15.32:0/3044008480 conn(0x7f76cc08d7a0 msgr2=0x7f76cc0b8f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:50:40.685 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:40.684+0000 7f76cbe00640 1 -- 172.21.15.32:0/3044008480 shutdown_connections 2024-08-03T20:50:40.685 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:40.684+0000 7f76cbe00640 1 -- 172.21.15.32:0/3044008480 wait complete. 2024-08-03T20:50:40.693 INFO:tasks.ceph.mds.c.smithi032.stderr: -5466> 2024-08-03T20:50:30.134+0000 7fb3a4200640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-08-03T20:50:40.709 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mds mds_inject_journal_corrupt_dentry_first 1.0 2024-08-03T20:50:40.822 INFO:tasks.ceph.mds.c.smithi032.stdout:starting mds.c at 2024-08-03T20:50:40.946 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:40.945+0000 7fd647e00640 1 -- 172.21.15.32:0/3522533399 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd62c004800 msgr2=0x7fd62c004c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:50:40.946 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:40.945+0000 7fd647e00640 1 --2- 172.21.15.32:0/3522533399 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd62c004800 0x7fd62c004c10 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7fd630009b90 tx=0x7fd62c005830 comp rx=0 tx=0).stop 2024-08-03T20:50:40.947 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:40.945+0000 7fd647400640 1 -- 172.21.15.32:0/3522533399 reap_dead start 2024-08-03T20:50:40.947 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:40.945+0000 7fd647e00640 1 -- 172.21.15.32:0/3522533399 shutdown_connections 2024-08-03T20:50:40.947 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:40.945+0000 7fd647e00640 1 -- 172.21.15.32:0/3522533399 >> 172.21.15.32:0/3522533399 conn(0x7fd6480cfd80 msgr2=0x7fd6480d0190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:50:40.947 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:40.945+0000 7fd647e00640 1 -- 172.21.15.32:0/3522533399 shutdown_connections 2024-08-03T20:50:40.947 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:40.946+0000 7fd647e00640 1 -- 172.21.15.32:0/3522533399 wait complete. 2024-08-03T20:50:40.948 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:40.948+0000 7fd647e00640 1 Processor -- start 2024-08-03T20:50:40.948 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:40.948+0000 7fd647e00640 1 -- start start 2024-08-03T20:50:40.951 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:40.949+0000 7fd647e00640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fd62c004800 0x7fd6481f5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:50:40.951 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:40.950+0000 7fd647e00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fd6480c9bc0 0x7fd6481579b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:50:40.951 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:40.950+0000 7fd647e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd648157ef0 0x7fd64815a7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:50:40.951 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:40.950+0000 7fd647e00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fd6481601a0 con 0x7fd648157ef0 2024-08-03T20:50:40.951 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:40.950+0000 7fd647e00640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7fd648160020 con 0x7fd6480c9bc0 2024-08-03T20:50:40.951 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:40.950+0000 7fd647e00640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7fd64806eb70 con 0x7fd62c004800 2024-08-03T20:50:40.951 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:40.950+0000 7fd647400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd648157ef0 0x7fd64815a7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:50:40.951 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:40.950+0000 7fd646a00640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fd62c004800 0x7fd6481f5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:50:40.951 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:40.950+0000 7fd647400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd648157ef0 0x7fd64815a7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:46352/0 (socket says 172.21.15.32:46352) 2024-08-03T20:50:40.952 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:40.950+0000 7fd647400640 1 -- 172.21.15.32:0/774229282 learned_addr learned my addr 172.21.15.32:0/774229282 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:50:40.952 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:40.950+0000 7fd63fe00640 1 --2- 172.21.15.32:0/774229282 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fd6480c9bc0 0x7fd6481579b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:50:40.952 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:40.950+0000 7fd646a00640 1 -- 172.21.15.32:0/774229282 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fd6480c9bc0 msgr2=0x7fd6481579b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:50:40.952 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:40.950+0000 7fd646a00640 1 --2- 172.21.15.32:0/774229282 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fd6480c9bc0 0x7fd6481579b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:40.952 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:40.950+0000 7fd646a00640 1 -- 172.21.15.32:0/774229282 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd648157ef0 msgr2=0x7fd64815a7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:50:40.952 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:40.950+0000 7fd646a00640 1 --2- 172.21.15.32:0/774229282 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd648157ef0 0x7fd64815a7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:40.952 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:40.950+0000 7fd646a00640 1 -- 172.21.15.32:0/774229282 --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd6480b2bb0 con 0x7fd62c004800 2024-08-03T20:50:40.952 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:40.950+0000 7fd647400640 1 --2- 172.21.15.32:0/774229282 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd648157ef0 0x7fd64815a7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-03T20:50:40.952 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:40.951+0000 7fd646a00640 1 --2- 172.21.15.32:0/774229282 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fd62c004800 0x7fd6481f5170 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fd638006630 tx=0x7fd6380039c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:50:40.953 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:40.951+0000 7fd63ea00640 1 -- 172.21.15.32:0/774229282 <== mon.2 v2:172.21.15.118:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd638015070 con 0x7fd62c004800 2024-08-03T20:50:40.953 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:40.951+0000 7fd63ea00640 1 -- 172.21.15.32:0/774229282 <== mon.2 v2:172.21.15.118:3301/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fd638011900 con 0x7fd62c004800 2024-08-03T20:50:40.953 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:40.951+0000 7fd647e00640 1 -- 172.21.15.32:0/774229282 --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd64815c4d0 con 0x7fd62c004800 2024-08-03T20:50:40.953 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:40.951+0000 7fd63ea00640 1 -- 172.21.15.32:0/774229282 <== mon.2 v2:172.21.15.118:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd638011d10 con 0x7fd62c004800 2024-08-03T20:50:40.953 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:40.951+0000 7fd647e00640 1 -- 172.21.15.32:0/774229282 --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fd64815ca90 con 0x7fd62c004800 2024-08-03T20:50:40.953 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:40.952+0000 7fd647e00640 1 -- 172.21.15.32:0/774229282 --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd64806e630 con 0x7fd62c004800 2024-08-03T20:50:40.954 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:40.954+0000 7fd63ea00640 1 -- 172.21.15.32:0/774229282 <== mon.2 v2:172.21.15.118:3301/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7fd638018070 con 0x7fd62c004800 2024-08-03T20:50:40.955 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:40.955+0000 7fd63ea00640 1 --2- 172.21.15.32:0/774229282 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fd624077010 0x7fd624079520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:50:40.955 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:40.955+0000 7fd63ea00640 1 -- 172.21.15.32:0/774229282 <== mon.2 v2:172.21.15.118:3301/0 5 ==== osd_map(58..58 src has 1..58) ==== 5795+0+0 (secure 0 0 0) 0x7fd638099850 con 0x7fd62c004800 2024-08-03T20:50:40.956 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:40.955+0000 7fd63fe00640 1 --2- 172.21.15.32:0/774229282 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fd624077010 0x7fd624079520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:50:40.956 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:40.956+0000 7fd63fe00640 1 --2- 172.21.15.32:0/774229282 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fd624077010 0x7fd624079520 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7fd630035000 tx=0x7fd6300046c0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:50:40.959 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:40.959+0000 7fd63ea00640 1 -- 172.21.15.32:0/774229282 <== mon.2 v2:172.21.15.118:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7fd648160760 con 0x7fd62c004800 2024-08-03T20:50:41.165 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:41.165+0000 7fd647e00640 1 -- 172.21.15.32:0/774229282 --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_command([{prefix=config set, name=mds_inject_journal_corrupt_dentry_first}] v 0) -- 0x7fd6480d09e0 con 0x7fd62c004800 2024-08-03T20:50:41.173 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:41.173+0000 7fd63ea00640 1 -- 172.21.15.32:0/774229282 <== mon.2 v2:172.21.15.118:3301/0 7 ==== mon_command_ack([{prefix=config set, name=mds_inject_journal_corrupt_dentry_first}]=0 v36)=0 v36) ==== 139+0+0 (secure 0 0 0) 0x7fd6380623a0 con 0x7fd62c004800 2024-08-03T20:50:41.176 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:41.175+0000 7fd647e00640 1 -- 172.21.15.32:0/774229282 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fd624077010 msgr2=0x7fd624079520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:50:41.176 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:41.175+0000 7fd647e00640 1 --2- 172.21.15.32:0/774229282 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fd624077010 0x7fd624079520 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7fd630035000 tx=0x7fd6300046c0 comp rx=0 tx=0).stop 2024-08-03T20:50:41.176 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:41.175+0000 7fd647e00640 1 -- 172.21.15.32:0/774229282 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fd62c004800 msgr2=0x7fd6481f5170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:50:41.176 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:41.175+0000 7fd647e00640 1 --2- 172.21.15.32:0/774229282 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fd62c004800 0x7fd6481f5170 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fd638006630 tx=0x7fd6380039c0 comp rx=0 tx=0).stop 2024-08-03T20:50:41.176 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:41.175+0000 7fd647e00640 1 -- 172.21.15.32:0/774229282 shutdown_connections 2024-08-03T20:50:41.176 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:41.175+0000 7fd647e00640 1 --2- 172.21.15.32:0/774229282 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fd624077010 0x7fd624079520 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:41.176 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:41.175+0000 7fd647e00640 1 --2- 172.21.15.32:0/774229282 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fd6480c9bc0 0x7fd6481579b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:41.177 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:41.175+0000 7fd647e00640 1 --2- 172.21.15.32:0/774229282 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd648157ef0 0x7fd64815a7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:41.177 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:41.175+0000 7fd647e00640 1 --2- 172.21.15.32:0/774229282 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fd62c004800 0x7fd6481f5170 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:41.177 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:41.175+0000 7fd647e00640 1 -- 172.21.15.32:0/774229282 >> 172.21.15.32:0/774229282 conn(0x7fd6480cfd80 msgr2=0x7fd6480cc800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:50:41.177 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:41.175+0000 7fd647e00640 1 -- 172.21.15.32:0/774229282 shutdown_connections 2024-08-03T20:50:41.177 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:41.175+0000 7fd647e00640 1 -- 172.21.15.32:0/774229282 wait complete. 2024-08-03T20:50:46.198 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs joinable true 2024-08-03T20:50:46.440 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:46.439+0000 7f5a87200640 1 -- 172.21.15.32:0/1625759728 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5a801621e0 msgr2=0x7f5a801625f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:50:46.440 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:46.439+0000 7f5a87200640 1 --2- 172.21.15.32:0/1625759728 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5a801621e0 0x7f5a801625f0 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7f5a6c005fd0 tx=0x7f5a6c02a670 comp rx=0 tx=0).stop 2024-08-03T20:50:46.440 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:46.439+0000 7f5a87200640 1 -- 172.21.15.32:0/1625759728 shutdown_connections 2024-08-03T20:50:46.440 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:46.439+0000 7f5a87200640 1 --2- 172.21.15.32:0/1625759728 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5a801621e0 0x7f5a801625f0 unknown :-1 s=CLOSED pgs=617 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:46.440 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:46.439+0000 7f5a87200640 1 -- 172.21.15.32:0/1625759728 >> 172.21.15.32:0/1625759728 conn(0x7f5a800b1bd0 msgr2=0x7f5a800b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:50:46.440 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:46.439+0000 7f5a87200640 1 -- 172.21.15.32:0/1625759728 shutdown_connections 2024-08-03T20:50:46.440 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:46.439+0000 7f5a87200640 1 -- 172.21.15.32:0/1625759728 wait complete. 2024-08-03T20:50:46.441 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:46.441+0000 7f5a87200640 1 Processor -- start 2024-08-03T20:50:46.441 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:46.441+0000 7f5a87200640 1 -- start start 2024-08-03T20:50:46.443 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:46.443+0000 7f5a87200640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f5a800b4640 0x7f5a80159280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:50:46.443 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:46.443+0000 7f5a87200640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f5a80155720 0x7f5a801539a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:50:46.443 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:46.443+0000 7f5a87200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5a801621e0 0x7f5a800cd360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:50:46.444 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:46.443+0000 7f5a87200640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f5a800af970 con 0x7f5a801621e0 2024-08-03T20:50:46.444 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:46.443+0000 7f5a87200640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f5a800af7f0 con 0x7f5a800b4640 2024-08-03T20:50:46.444 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:46.443+0000 7f5a87200640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7f5a800b5160 con 0x7f5a80155720 2024-08-03T20:50:46.444 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:46.443+0000 7f5a85200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5a801621e0 0x7f5a800cd360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:50:46.444 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:46.443+0000 7f5a85200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5a801621e0 0x7f5a800cd360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:46370/0 (socket says 172.21.15.32:46370) 2024-08-03T20:50:46.444 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:46.443+0000 7f5a85200640 1 -- 172.21.15.32:0/3239855052 learned_addr learned my addr 172.21.15.32:0/3239855052 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:50:46.444 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:46.443+0000 7f5a7f400640 1 --2- 172.21.15.32:0/3239855052 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f5a80155720 0x7f5a801539a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:50:46.444 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:46.443+0000 7f5a7fe00640 1 --2- 172.21.15.32:0/3239855052 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f5a800b4640 0x7f5a80159280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:50:46.445 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:46.443+0000 7f5a85200640 1 -- 172.21.15.32:0/3239855052 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f5a80155720 msgr2=0x7f5a801539a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:50:46.445 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:46.443+0000 7f5a85200640 1 --2- 172.21.15.32:0/3239855052 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f5a80155720 0x7f5a801539a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:46.445 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:46.443+0000 7f5a85200640 1 -- 172.21.15.32:0/3239855052 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f5a800b4640 msgr2=0x7f5a80159280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:50:46.445 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:46.443+0000 7f5a85200640 1 --2- 172.21.15.32:0/3239855052 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f5a800b4640 0x7f5a80159280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:46.445 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:46.443+0000 7f5a85200640 1 -- 172.21.15.32:0/3239855052 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5a801597c0 con 0x7f5a801621e0 2024-08-03T20:50:46.446 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:46.443+0000 7f5a7f400640 1 --2- 172.21.15.32:0/3239855052 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f5a80155720 0x7f5a801539a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-03T20:50:46.446 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:46.443+0000 7f5a7fe00640 1 --2- 172.21.15.32:0/3239855052 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f5a800b4640 0x7f5a80159280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-03T20:50:46.446 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:46.444+0000 7f5a85200640 1 --2- 172.21.15.32:0/3239855052 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5a801621e0 0x7f5a800cd360 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7f5a7800c760 tx=0x7f5a7800cb60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:50:46.446 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:46.444+0000 7f5a77e00640 1 -- 172.21.15.32:0/3239855052 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a78003b70 con 0x7f5a801621e0 2024-08-03T20:50:46.446 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:46.444+0000 7f5a77e00640 1 -- 172.21.15.32:0/3239855052 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f5a78003d10 con 0x7f5a801621e0 2024-08-03T20:50:46.446 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:46.444+0000 7f5a77e00640 1 -- 172.21.15.32:0/3239855052 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a78005330 con 0x7f5a801621e0 2024-08-03T20:50:46.446 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:46.444+0000 7f5a87200640 1 -- 172.21.15.32:0/3239855052 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5a800cea70 con 0x7f5a801621e0 2024-08-03T20:50:46.446 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:46.444+0000 7f5a87200640 1 -- 172.21.15.32:0/3239855052 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5a800cf030 con 0x7f5a801621e0 2024-08-03T20:50:46.447 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:46.445+0000 7f5a87200640 1 -- 172.21.15.32:0/3239855052 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5a801625f0 con 0x7f5a801621e0 2024-08-03T20:50:46.447 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:46.447+0000 7f5a77e00640 1 -- 172.21.15.32:0/3239855052 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f5a780054d0 con 0x7f5a801621e0 2024-08-03T20:50:46.449 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:46.448+0000 7f5a77e00640 1 --2- 172.21.15.32:0/3239855052 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f5a60077010 0x7f5a60079520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:50:46.449 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:46.448+0000 7f5a77e00640 1 -- 172.21.15.32:0/3239855052 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(58..58 src has 1..58) ==== 5795+0+0 (secure 0 0 0) 0x7f5a78098750 con 0x7f5a801621e0 2024-08-03T20:50:46.449 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:46.448+0000 7f5a7fe00640 1 --2- 172.21.15.32:0/3239855052 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f5a60077010 0x7f5a60079520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:50:46.449 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:46.449+0000 7f5a7fe00640 1 --2- 172.21.15.32:0/3239855052 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f5a60077010 0x7f5a60079520 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f5a70007fd0 tx=0x7f5a70009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:50:46.453 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:46.453+0000 7f5a77e00640 1 -- 172.21.15.32:0/3239855052 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7f5a78061980 con 0x7f5a801621e0 2024-08-03T20:50:46.698 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:46.698+0000 7f5a87200640 1 -- 172.21.15.32:0/3239855052 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "joinable", "val": "true"} v 0) -- 0x7f5a800d0120 con 0x7f5a801621e0 2024-08-03T20:50:47.704 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:47.703+0000 7f5a77e00640 1 -- 172.21.15.32:0/3239855052 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "joinable", "val": "true"}]=0 cephfs marked joinable; MDS may join as newly active. v54) ==== 162+0+0 (secure 0 0 0) 0x7f5a78061280 con 0x7f5a801621e0 2024-08-03T20:50:47.704 INFO:teuthology.orchestra.run.smithi032.stderr:cephfs marked joinable; MDS may join as newly active. 2024-08-03T20:50:47.707 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:47.706+0000 7f5a87200640 1 -- 172.21.15.32:0/3239855052 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f5a60077010 msgr2=0x7f5a60079520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:50:47.707 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:47.706+0000 7f5a87200640 1 --2- 172.21.15.32:0/3239855052 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f5a60077010 0x7f5a60079520 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f5a70007fd0 tx=0x7f5a70009040 comp rx=0 tx=0).stop 2024-08-03T20:50:47.707 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:47.706+0000 7f5a87200640 1 -- 172.21.15.32:0/3239855052 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5a801621e0 msgr2=0x7f5a800cd360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:50:47.707 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:47.706+0000 7f5a87200640 1 --2- 172.21.15.32:0/3239855052 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5a801621e0 0x7f5a800cd360 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7f5a7800c760 tx=0x7f5a7800cb60 comp rx=0 tx=0).stop 2024-08-03T20:50:47.708 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:47.706+0000 7f5a87200640 1 -- 172.21.15.32:0/3239855052 shutdown_connections 2024-08-03T20:50:47.708 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:47.707+0000 7f5a87200640 1 --2- 172.21.15.32:0/3239855052 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f5a60077010 0x7f5a60079520 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:47.708 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:47.707+0000 7f5a87200640 1 --2- 172.21.15.32:0/3239855052 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5a801621e0 0x7f5a800cd360 unknown :-1 s=CLOSED pgs=618 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:47.708 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:47.707+0000 7f5a87200640 1 --2- 172.21.15.32:0/3239855052 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f5a80155720 0x7f5a801539a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:47.708 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:47.707+0000 7f5a87200640 1 --2- 172.21.15.32:0/3239855052 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f5a800b4640 0x7f5a80159280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:47.708 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:47.707+0000 7f5a87200640 1 -- 172.21.15.32:0/3239855052 >> 172.21.15.32:0/3239855052 conn(0x7f5a800b1bd0 msgr2=0x7f5a800b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:50:47.708 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:47.707+0000 7f5a87200640 1 -- 172.21.15.32:0/3239855052 shutdown_connections 2024-08-03T20:50:47.708 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:47.707+0000 7f5a87200640 1 -- 172.21.15.32:0/3239855052 wait complete. 2024-08-03T20:50:47.740 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-08-03T20:50:47.974 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:47.974+0000 7f670a000640 1 -- 172.21.15.32:0/3416878630 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f66e8004d40 msgr2=0x7f66e8005150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:50:47.975 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:47.974+0000 7f670a000640 1 --2- 172.21.15.32:0/3416878630 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f66e8004d40 0x7f66e8005150 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7f66fc009ed0 tx=0x7f66e8005830 comp rx=0 tx=0).stop 2024-08-03T20:50:47.975 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:47.974+0000 7f6703e00640 1 -- 172.21.15.32:0/3416878630 reap_dead start 2024-08-03T20:50:47.975 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:47.974+0000 7f670a000640 1 -- 172.21.15.32:0/3416878630 shutdown_connections 2024-08-03T20:50:47.975 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:47.974+0000 7f670a000640 1 -- 172.21.15.32:0/3416878630 >> 172.21.15.32:0/3416878630 conn(0x7f67040b34b0 msgr2=0x7f67040d1050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:50:47.975 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:47.974+0000 7f670a000640 1 -- 172.21.15.32:0/3416878630 shutdown_connections 2024-08-03T20:50:47.975 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:47.974+0000 7f670a000640 1 -- 172.21.15.32:0/3416878630 wait complete. 2024-08-03T20:50:47.976 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:47.976+0000 7f670a000640 1 Processor -- start 2024-08-03T20:50:47.977 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:47.976+0000 7f670a000640 1 -- start start 2024-08-03T20:50:47.979 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:47.978+0000 7f670a000640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f66e8004d40 0x7f67041f87f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:50:47.979 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:47.978+0000 7f670a000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f67041f8d30 0x7f67040ae7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:50:47.979 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:47.978+0000 7f670a000640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f67040aecf0 0x7f67041f5200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:50:47.979 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:47.978+0000 7f670a000640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f67040d04d0 con 0x7f67041f8d30 2024-08-03T20:50:47.979 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:47.978+0000 7f670a000640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f67040d0350 con 0x7f67040aecf0 2024-08-03T20:50:47.979 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:47.978+0000 7f670a000640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7f670406c9e0 con 0x7f66e8004d40 2024-08-03T20:50:47.979 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:47.978+0000 7f6702a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f67041f8d30 0x7f67040ae7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:50:47.979 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:47.978+0000 7f6702a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f67041f8d30 0x7f67040ae7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:46390/0 (socket says 172.21.15.32:46390) 2024-08-03T20:50:47.980 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:47.978+0000 7f6702a00640 1 -- 172.21.15.32:0/1618132614 learned_addr learned my addr 172.21.15.32:0/1618132614 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:50:47.980 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:47.978+0000 7f6702a00640 1 -- 172.21.15.32:0/1618132614 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f66e8004d40 msgr2=0x7f67041f87f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:50:47.980 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:47.978+0000 7f6703400640 1 --2- 172.21.15.32:0/1618132614 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f66e8004d40 0x7f67041f87f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:50:47.980 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:47.978+0000 7f6703e00640 1 --2- 172.21.15.32:0/1618132614 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f67040aecf0 0x7f67041f5200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:50:47.980 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:47.978+0000 7f6702a00640 1 --2- 172.21.15.32:0/1618132614 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f66e8004d40 0x7f67041f87f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:47.980 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:47.978+0000 7f6702a00640 1 -- 172.21.15.32:0/1618132614 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f67040aecf0 msgr2=0x7f67041f5200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:50:47.980 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:47.978+0000 7f6702a00640 1 --2- 172.21.15.32:0/1618132614 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f67040aecf0 0x7f67041f5200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:47.980 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:47.978+0000 7f6702a00640 1 -- 172.21.15.32:0/1618132614 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f67041f9140 con 0x7f67041f8d30 2024-08-03T20:50:47.980 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:47.978+0000 7f6703e00640 1 --2- 172.21.15.32:0/1618132614 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f67040aecf0 0x7f67041f5200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-03T20:50:47.980 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:47.979+0000 7f6702a00640 1 --2- 172.21.15.32:0/1618132614 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f67041f8d30 0x7f67040ae7b0 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7f66f0011b40 tx=0x7f66f0009e90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:50:47.981 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:47.979+0000 7f66fb400640 1 -- 172.21.15.32:0/1618132614 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66f001e070 con 0x7f67041f8d30 2024-08-03T20:50:47.982 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:47.979+0000 7f66fb400640 1 -- 172.21.15.32:0/1618132614 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f66f0004050 con 0x7f67041f8d30 2024-08-03T20:50:47.982 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:47.979+0000 7f670a000640 1 -- 172.21.15.32:0/1618132614 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f67040ad9f0 con 0x7f67041f8d30 2024-08-03T20:50:47.982 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:47.979+0000 7f66fb400640 1 -- 172.21.15.32:0/1618132614 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66f00044a0 con 0x7f67041f8d30 2024-08-03T20:50:47.982 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:47.979+0000 7f670a000640 1 -- 172.21.15.32:0/1618132614 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f67040adfb0 con 0x7f67041f8d30 2024-08-03T20:50:47.982 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:47.980+0000 7f670a000640 1 -- 172.21.15.32:0/1618132614 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f67040742b0 con 0x7f67041f8d30 2024-08-03T20:50:47.982 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:47.981+0000 7f66fb400640 1 -- 172.21.15.32:0/1618132614 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f66f0021070 con 0x7f67041f8d30 2024-08-03T20:50:47.983 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:47.982+0000 7f66fb400640 1 --2- 172.21.15.32:0/1618132614 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f66e4077040 0x7f66e4079550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:50:47.983 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:47.982+0000 7f66fb400640 1 -- 172.21.15.32:0/1618132614 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(58..58 src has 1..58) ==== 5795+0+0 (secure 0 0 0) 0x7f66f00992f0 con 0x7f67041f8d30 2024-08-03T20:50:47.983 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:47.983+0000 7f6703400640 1 --2- 172.21.15.32:0/1618132614 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f66e4077040 0x7f66e4079550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:50:47.984 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:47.984+0000 7f6703400640 1 --2- 172.21.15.32:0/1618132614 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f66e4077040 0x7f66e4079550 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f66f4000fb0 tx=0x7f66f4000f40 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:50:47.987 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:47.987+0000 7f66fb400640 1 -- 172.21.15.32:0/1618132614 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7f67041f9140 con 0x7f67041f8d30 2024-08-03T20:50:48.230 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:48.229+0000 7f670a000640 1 -- 172.21.15.32:0/1618132614 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f67040b2830 con 0x7f67041f8d30 2024-08-03T20:50:48.230 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:48.230+0000 7f66fb400640 1 -- 172.21.15.32:0/1618132614 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 55 v55) ==== 94+0+5223 (secure 0 0 0) 0x7f66f0061ef0 con 0x7f67041f8d30 2024-08-03T20:50:48.230 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-03T20:50:48.231 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":55,"btime":"2024-08-03T20:50:47:707170+0000","default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5261,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6833/2498840646","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":2498840646},{"type":"v1","addr":"172.21.15.32:6833","nonce":2498840646}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":45},{"gid":5276,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6837/3142148286","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6835","nonce":3142148286},{"type":"v1","addr":"172.21.15.118:6837","nonce":3142148286}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":46},{"gid":5632,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6836/3720010453","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6834","nonce":3720010453},{"type":"v1","addr":"172.21.15.118:6836","nonce":3720010453}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":46}],"filesystems":[{"mdsmap":{"epoch":55,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-08-03T20:50:27.688698+0000","modified":"2024-08-03T20:50:47.707142+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":58,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6045},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6045":{"gid":6045,"name":"c","rank":0,"incarnation":55,"state":"up:replay","state_seq":1,"addr":"172.21.15.32:6835/2963295201","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":2963295201},{"type":"v1","addr":"172.21.15.32:6835","nonce":2963295201}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[9],"metadata_pool":8,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":4}]} 2024-08-03T20:50:48.231 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 55 2024-08-03T20:50:48.233 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:48.232+0000 7f670a000640 1 -- 172.21.15.32:0/1618132614 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f66e4077040 msgr2=0x7f66e4079550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:50:48.233 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:48.232+0000 7f670a000640 1 --2- 172.21.15.32:0/1618132614 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f66e4077040 0x7f66e4079550 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f66f4000fb0 tx=0x7f66f4000f40 comp rx=0 tx=0).stop 2024-08-03T20:50:48.233 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:48.232+0000 7f670a000640 1 -- 172.21.15.32:0/1618132614 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f67041f8d30 msgr2=0x7f67040ae7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:50:48.233 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:48.232+0000 7f670a000640 1 --2- 172.21.15.32:0/1618132614 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f67041f8d30 0x7f67040ae7b0 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7f66f0011b40 tx=0x7f66f0009e90 comp rx=0 tx=0).stop 2024-08-03T20:50:48.233 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:48.232+0000 7f670a000640 1 -- 172.21.15.32:0/1618132614 shutdown_connections 2024-08-03T20:50:48.233 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:48.232+0000 7f670a000640 1 --2- 172.21.15.32:0/1618132614 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f66e4077040 0x7f66e4079550 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:48.233 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:48.232+0000 7f670a000640 1 --2- 172.21.15.32:0/1618132614 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f67040aecf0 0x7f67041f5200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:48.233 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:48.232+0000 7f670a000640 1 --2- 172.21.15.32:0/1618132614 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f67041f8d30 0x7f67040ae7b0 unknown :-1 s=CLOSED pgs=621 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:48.233 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:48.232+0000 7f670a000640 1 --2- 172.21.15.32:0/1618132614 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f66e8004d40 0x7f67041f87f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:48.233 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:48.232+0000 7f670a000640 1 -- 172.21.15.32:0/1618132614 >> 172.21.15.32:0/1618132614 conn(0x7f67040b34b0 msgr2=0x7f67040b19a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:50:48.234 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:48.232+0000 7f670a000640 1 -- 172.21.15.32:0/1618132614 shutdown_connections 2024-08-03T20:50:48.234 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:48.232+0000 7f670a000640 1 -- 172.21.15.32:0/1618132614 wait complete. 2024-08-03T20:50:48.661 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 55, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-08-03T20:50:27.688698+0000', 'modified': '2024-08-03T20:50:47.707142+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 58, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 6045}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6045': {'gid': 6045, 'name': 'c', 'rank': 0, 'incarnation': 55, 'state': 'up:replay', 'state_seq': 1, 'addr': '172.21.15.32:6835/2963295201', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6834', 'nonce': 2963295201}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 2963295201}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [9], 'metadata_pool': 8, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 0, 'qdb_cluster': []} 2024-08-03T20:50:48.662 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6045:up:replay 2024-08-03T20:50:49.663 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-08-03T20:50:49.905 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:49.904+0000 7f2426e00640 1 -- 172.21.15.32:0/2005099536 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f24200b93d0 msgr2=0x7f24200cfa00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:50:49.905 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:49.904+0000 7f2426e00640 1 --2- 172.21.15.32:0/2005099536 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f24200b93d0 0x7f24200cfa00 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f24140098e0 tx=0x7f241402a540 comp rx=0 tx=0).stop 2024-08-03T20:50:49.905 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:49.904+0000 7f2426e00640 1 -- 172.21.15.32:0/2005099536 shutdown_connections 2024-08-03T20:50:49.905 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:49.904+0000 7f2426e00640 1 --2- 172.21.15.32:0/2005099536 >> v2:172.21.15.118:3300/0 conn(0x7f24200d0af0 0x7f24200d0f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:49.905 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:49.904+0000 7f2426e00640 1 --2- 172.21.15.32:0/2005099536 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f24200b93d0 0x7f24200cfa00 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:49.905 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:49.904+0000 7f2426e00640 1 -- 172.21.15.32:0/2005099536 >> 172.21.15.32:0/2005099536 conn(0x7f24200d4120 msgr2=0x7f24200d4530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:50:49.905 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:49.904+0000 7f2426e00640 1 -- 172.21.15.32:0/2005099536 shutdown_connections 2024-08-03T20:50:49.906 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:49.904+0000 7f2426e00640 1 -- 172.21.15.32:0/2005099536 wait complete. 2024-08-03T20:50:49.907 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:49.906+0000 7f2426e00640 1 Processor -- start 2024-08-03T20:50:49.907 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:49.907+0000 7f2426e00640 1 -- start start 2024-08-03T20:50:49.909 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:49.908+0000 7f2426e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f24200b93d0 0x7f2420152710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:50:49.909 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:49.908+0000 7f2426e00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f24200d0af0 0x7f24200b6f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:50:49.909 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:49.908+0000 7f2426e00640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f24200d1cf0 0x7f24200b7490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:50:49.910 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:49.908+0000 7f2426e00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f24200b4cb0 con 0x7f24200b93d0 2024-08-03T20:50:49.910 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:49.908+0000 7f2426e00640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f24200d1550 con 0x7f24200d0af0 2024-08-03T20:50:49.910 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:49.908+0000 7f2426e00640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7f24200b4e30 con 0x7f24200d1cf0 2024-08-03T20:50:49.910 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:49.908+0000 7f2425a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f24200b93d0 0x7f2420152710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:50:49.910 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:49.908+0000 7f2425a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f24200b93d0 0x7f2420152710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:35000/0 (socket says 172.21.15.32:35000) 2024-08-03T20:50:49.910 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:49.908+0000 7f2425a00640 1 -- 172.21.15.32:0/3528647154 learned_addr learned my addr 172.21.15.32:0/3528647154 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:50:49.910 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:49.908+0000 7f2426400640 1 --2- 172.21.15.32:0/3528647154 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f24200d1cf0 0x7f24200b7490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:50:49.910 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:49.908+0000 7f2425000640 1 --2- 172.21.15.32:0/3528647154 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f24200d0af0 0x7f24200b6f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:50:49.910 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:49.909+0000 7f2425a00640 1 -- 172.21.15.32:0/3528647154 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f24200d1cf0 msgr2=0x7f24200b7490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:50:49.910 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:49.909+0000 7f2425a00640 1 --2- 172.21.15.32:0/3528647154 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f24200d1cf0 0x7f24200b7490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:49.910 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:49.909+0000 7f2425a00640 1 -- 172.21.15.32:0/3528647154 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f24200d0af0 msgr2=0x7f24200b6f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:50:49.910 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:49.909+0000 7f2425a00640 1 --2- 172.21.15.32:0/3528647154 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f24200d0af0 0x7f24200b6f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:49.911 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:49.909+0000 7f2425a00640 1 -- 172.21.15.32:0/3528647154 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2420160b70 con 0x7f24200b93d0 2024-08-03T20:50:49.911 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:49.909+0000 7f2425a00640 1 --2- 172.21.15.32:0/3528647154 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f24200b93d0 0x7f2420152710 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7f2414009a00 tx=0x7f24140052f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:50:49.911 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:49.909+0000 7f241d600640 1 -- 172.21.15.32:0/3528647154 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2414005cf0 con 0x7f24200b93d0 2024-08-03T20:50:49.912 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:49.909+0000 7f241d600640 1 -- 172.21.15.32:0/3528647154 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f2414031c40 con 0x7f24200b93d0 2024-08-03T20:50:49.912 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:49.909+0000 7f241d600640 1 -- 172.21.15.32:0/3528647154 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2414004a00 con 0x7f24200b93d0 2024-08-03T20:50:49.912 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:49.909+0000 7f2426e00640 1 -- 172.21.15.32:0/3528647154 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2420160e50 con 0x7f24200b93d0 2024-08-03T20:50:49.912 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:49.910+0000 7f2426e00640 1 -- 172.21.15.32:0/3528647154 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2420161410 con 0x7f24200b93d0 2024-08-03T20:50:49.912 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:49.911+0000 7f2426e00640 1 -- 172.21.15.32:0/3528647154 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f24200cfa00 con 0x7f24200b93d0 2024-08-03T20:50:49.912 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:49.912+0000 7f241d600640 1 -- 172.21.15.32:0/3528647154 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f2414004ba0 con 0x7f24200b93d0 2024-08-03T20:50:49.913 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:49.913+0000 7f241d600640 1 --2- 172.21.15.32:0/3528647154 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f2400077010 0x7f2400079520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:50:49.913 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:49.913+0000 7f241d600640 1 -- 172.21.15.32:0/3528647154 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(58..58 src has 1..58) ==== 5795+0+0 (secure 0 0 0) 0x7f24140b9ba0 con 0x7f24200b93d0 2024-08-03T20:50:49.914 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:49.913+0000 7f2425000640 1 --2- 172.21.15.32:0/3528647154 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f2400077010 0x7f2400079520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:50:49.914 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:49.914+0000 7f2425000640 1 --2- 172.21.15.32:0/3528647154 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f2400077010 0x7f2400079520 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f242009e530 tx=0x7f240c005f90 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:50:49.917 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:49.917+0000 7f241d600640 1 -- 172.21.15.32:0/3528647154 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7f241403f030 con 0x7f24200b93d0 2024-08-03T20:50:50.165 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:50.165+0000 7f2426e00640 1 -- 172.21.15.32:0/3528647154 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f24200d0580 con 0x7f24200b93d0 2024-08-03T20:50:50.166 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:50.166+0000 7f241d600640 1 -- 172.21.15.32:0/3528647154 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 57 v57) ==== 94+0+5223 (secure 0 0 0) 0x7f2414031770 con 0x7f24200b93d0 2024-08-03T20:50:50.166 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-03T20:50:50.166 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":57,"btime":"2024-08-03T20:50:49:714792+0000","default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5261,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6833/2498840646","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":2498840646},{"type":"v1","addr":"172.21.15.32:6833","nonce":2498840646}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":45},{"gid":5276,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6837/3142148286","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6835","nonce":3142148286},{"type":"v1","addr":"172.21.15.118:6837","nonce":3142148286}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":46},{"gid":5632,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6836/3720010453","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6834","nonce":3720010453},{"type":"v1","addr":"172.21.15.118:6836","nonce":3720010453}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":46}],"filesystems":[{"mdsmap":{"epoch":57,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-08-03T20:50:27.688698+0000","modified":"2024-08-03T20:50:48.716209+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":58,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6045},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6045":{"gid":6045,"name":"c","rank":0,"incarnation":55,"state":"up:rejoin","state_seq":4,"addr":"172.21.15.32:6835/2963295201","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":2963295201},{"type":"v1","addr":"172.21.15.32:6835","nonce":2963295201}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[9],"metadata_pool":8,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":4}]} 2024-08-03T20:50:50.167 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 57 2024-08-03T20:50:50.169 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:50.168+0000 7f2426e00640 1 -- 172.21.15.32:0/3528647154 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f2400077010 msgr2=0x7f2400079520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:50:50.169 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:50.168+0000 7f2426e00640 1 --2- 172.21.15.32:0/3528647154 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f2400077010 0x7f2400079520 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f242009e530 tx=0x7f240c005f90 comp rx=0 tx=0).stop 2024-08-03T20:50:50.169 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:50.168+0000 7f2426e00640 1 -- 172.21.15.32:0/3528647154 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f24200b93d0 msgr2=0x7f2420152710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:50:50.169 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:50.168+0000 7f2426e00640 1 --2- 172.21.15.32:0/3528647154 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f24200b93d0 0x7f2420152710 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7f2414009a00 tx=0x7f24140052f0 comp rx=0 tx=0).stop 2024-08-03T20:50:50.169 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:50.168+0000 7f2426e00640 1 -- 172.21.15.32:0/3528647154 shutdown_connections 2024-08-03T20:50:50.169 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:50.168+0000 7f2426e00640 1 --2- 172.21.15.32:0/3528647154 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f2400077010 0x7f2400079520 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:50.169 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:50.168+0000 7f2426e00640 1 --2- 172.21.15.32:0/3528647154 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f24200d0af0 0x7f24200b6f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:50.169 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:50.168+0000 7f2426e00640 1 --2- 172.21.15.32:0/3528647154 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f24200d1cf0 0x7f24200b7490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:50.170 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:50.168+0000 7f2426e00640 1 --2- 172.21.15.32:0/3528647154 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f24200b93d0 0x7f2420152710 unknown :-1 s=CLOSED pgs=622 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:50.170 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:50.168+0000 7f2426e00640 1 -- 172.21.15.32:0/3528647154 >> 172.21.15.32:0/3528647154 conn(0x7f24200d4120 msgr2=0x7f24200b8ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:50:50.170 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:50.168+0000 7f2426e00640 1 -- 172.21.15.32:0/3528647154 shutdown_connections 2024-08-03T20:50:50.170 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:50.168+0000 7f2426e00640 1 -- 172.21.15.32:0/3528647154 wait complete. 2024-08-03T20:50:50.596 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 57, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-08-03T20:50:27.688698+0000', 'modified': '2024-08-03T20:50:48.716209+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 58, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 6045}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6045': {'gid': 6045, 'name': 'c', 'rank': 0, 'incarnation': 55, 'state': 'up:rejoin', 'state_seq': 4, 'addr': '172.21.15.32:6835/2963295201', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6834', 'nonce': 2963295201}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 2963295201}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [9], 'metadata_pool': 8, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 0, 'qdb_cluster': []} 2024-08-03T20:50:50.596 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6045:up:rejoin 2024-08-03T20:50:51.597 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-08-03T20:50:51.692 INFO:tasks.ceph.mds.c.smithi032.stderr:2024-08-03T20:50:51.691+0000 7fbffd600640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-08-03T20:50:51.837 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:51.836+0000 7fb97fc00640 1 -- 172.21.15.32:0/2798174327 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb9780b38b0 msgr2=0x7fb9781523f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:50:51.837 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:51.836+0000 7fb97fc00640 1 --2- 172.21.15.32:0/2798174327 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb9780b38b0 0x7fb9781523f0 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7fb96c005fd0 tx=0x7fb96c02a5d0 comp rx=0 tx=0).stop 2024-08-03T20:50:51.838 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:51.837+0000 7fb97fc00640 1 -- 172.21.15.32:0/2798174327 shutdown_connections 2024-08-03T20:50:51.838 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:51.837+0000 7fb97fc00640 1 --2- 172.21.15.32:0/2798174327 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fb97815f610 0x7fb9780cdf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:51.838 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:51.837+0000 7fb97fc00640 1 --2- 172.21.15.32:0/2798174327 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb9780b38b0 0x7fb9781523f0 unknown :-1 s=CLOSED pgs=623 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:51.838 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:51.837+0000 7fb97fc00640 1 -- 172.21.15.32:0/2798174327 >> 172.21.15.32:0/2798174327 conn(0x7fb9780b1bd0 msgr2=0x7fb9780b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:50:51.838 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:51.837+0000 7fb97fc00640 1 -- 172.21.15.32:0/2798174327 shutdown_connections 2024-08-03T20:50:51.838 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:51.837+0000 7fb97fc00640 1 -- 172.21.15.32:0/2798174327 wait complete. 2024-08-03T20:50:51.839 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:51.839+0000 7fb97fc00640 1 Processor -- start 2024-08-03T20:50:51.839 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:51.839+0000 7fb97fc00640 1 -- start start 2024-08-03T20:50:51.841 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:51.840+0000 7fb97fc00640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fb9780b38b0 0x7fb9781428b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:50:51.842 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:51.840+0000 7fb97fc00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb9780b4ce0 0x7fb9781f68e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:50:51.842 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:51.840+0000 7fb97fc00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb978150f80 0x7fb9781f6e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:50:51.842 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:51.840+0000 7fb97fc00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fb9780cea90 con 0x7fb9780b4ce0 2024-08-03T20:50:51.842 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:51.840+0000 7fb97fc00640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7fb9780ce910 con 0x7fb978150f80 2024-08-03T20:50:51.842 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:51.840+0000 7fb97fc00640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7fb9780cec10 con 0x7fb9780b38b0 2024-08-03T20:50:51.842 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:51.841+0000 7fb977e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb9780b4ce0 0x7fb9781f68e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:50:51.842 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:51.841+0000 7fb977e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb9780b4ce0 0x7fb9781f68e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:35014/0 (socket says 172.21.15.32:35014) 2024-08-03T20:50:51.842 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:51.841+0000 7fb977e00640 1 -- 172.21.15.32:0/2909093109 learned_addr learned my addr 172.21.15.32:0/2909093109 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:50:51.842 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:51.841+0000 7fb97d200640 1 --2- 172.21.15.32:0/2909093109 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fb9780b38b0 0x7fb9781428b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:50:51.842 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:51.841+0000 7fb97dc00640 1 --2- 172.21.15.32:0/2909093109 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb978150f80 0x7fb9781f6e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:50:51.842 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:51.841+0000 7fb977e00640 1 -- 172.21.15.32:0/2909093109 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fb9780b38b0 msgr2=0x7fb9781428b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:50:51.842 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:51.841+0000 7fb977e00640 1 --2- 172.21.15.32:0/2909093109 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fb9780b38b0 0x7fb9781428b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:51.843 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:51.841+0000 7fb977e00640 1 -- 172.21.15.32:0/2909093109 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb978150f80 msgr2=0x7fb9781f6e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:50:51.843 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:51.841+0000 7fb977e00640 1 --2- 172.21.15.32:0/2909093109 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb978150f80 0x7fb9781f6e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:51.843 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:51.841+0000 7fb977e00640 1 -- 172.21.15.32:0/2909093109 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb978156b80 con 0x7fb9780b4ce0 2024-08-03T20:50:51.843 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:51.841+0000 7fb97dc00640 1 --2- 172.21.15.32:0/2909093109 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb978150f80 0x7fb9781f6e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-03T20:50:51.844 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:51.842+0000 7fb977e00640 1 --2- 172.21.15.32:0/2909093109 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb9780b4ce0 0x7fb9781f68e0 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7fb96400c3a0 tx=0x7fb96400c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:50:51.844 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:51.842+0000 7fb974c00640 1 -- 172.21.15.32:0/2909093109 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb964017070 con 0x7fb9780b4ce0 2024-08-03T20:50:51.844 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:51.842+0000 7fb974c00640 1 -- 172.21.15.32:0/2909093109 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fb9640053c0 con 0x7fb9780b4ce0 2024-08-03T20:50:51.844 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:51.842+0000 7fb974c00640 1 -- 172.21.15.32:0/2909093109 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb964003850 con 0x7fb9780b4ce0 2024-08-03T20:50:51.844 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:51.842+0000 7fb97fc00640 1 -- 172.21.15.32:0/2909093109 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb978156e60 con 0x7fb9780b4ce0 2024-08-03T20:50:51.844 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:51.842+0000 7fb97fc00640 1 -- 172.21.15.32:0/2909093109 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb978157420 con 0x7fb9780b4ce0 2024-08-03T20:50:51.844 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:51.843+0000 7fb97fc00640 1 -- 172.21.15.32:0/2909093109 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb9780cf130 con 0x7fb9780b4ce0 2024-08-03T20:50:51.844 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:51.844+0000 7fb974c00640 1 -- 172.21.15.32:0/2909093109 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7fb96401a070 con 0x7fb9780b4ce0 2024-08-03T20:50:51.846 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:51.845+0000 7fb974c00640 1 --2- 172.21.15.32:0/2909093109 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fb958077010 0x7fb958079520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:50:51.846 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:51.845+0000 7fb974c00640 1 -- 172.21.15.32:0/2909093109 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(58..58 src has 1..58) ==== 5795+0+0 (secure 0 0 0) 0x7fb964097d00 con 0x7fb9780b4ce0 2024-08-03T20:50:51.846 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:51.846+0000 7fb97d200640 1 --2- 172.21.15.32:0/2909093109 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fb958077010 0x7fb958079520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:50:51.847 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:51.847+0000 7fb97d200640 1 --2- 172.21.15.32:0/2909093109 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fb958077010 0x7fb958079520 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7fb96c033000 tx=0x7fb96c004730 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:50:51.850 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:51.850+0000 7fb974c00640 1 -- 172.21.15.32:0/2909093109 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7fb964060840 con 0x7fb9780b4ce0 2024-08-03T20:50:52.094 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:52.093+0000 7fb97fc00640 1 -- 172.21.15.32:0/2909093109 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb97815a2c0 con 0x7fb9780b4ce0 2024-08-03T20:50:52.095 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:52.094+0000 7fb974c00640 1 -- 172.21.15.32:0/2909093109 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 58 v58) ==== 94+0+5230 (secure 0 0 0) 0x7fb9640690d0 con 0x7fb9780b4ce0 2024-08-03T20:50:52.095 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-03T20:50:52.095 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":58,"btime":"2024-08-03T20:50:50:722022+0000","default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5261,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6833/2498840646","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":2498840646},{"type":"v1","addr":"172.21.15.32:6833","nonce":2498840646}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":45},{"gid":5276,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6837/3142148286","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6835","nonce":3142148286},{"type":"v1","addr":"172.21.15.118:6837","nonce":3142148286}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":46},{"gid":5632,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6836/3720010453","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6834","nonce":3720010453},{"type":"v1","addr":"172.21.15.118:6836","nonce":3720010453}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":46}],"filesystems":[{"mdsmap":{"epoch":58,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-08-03T20:50:27.688698+0000","modified":"2024-08-03T20:50:50.722012+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":58,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6045},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6045":{"gid":6045,"name":"c","rank":0,"incarnation":55,"state":"up:active","state_seq":5,"addr":"172.21.15.32:6835/2963295201","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":2963295201},{"type":"v1","addr":"172.21.15.32:6835","nonce":2963295201}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[9],"metadata_pool":8,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6045,"qdb_cluster":[6045]},"id":4}]} 2024-08-03T20:50:52.095 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 58 2024-08-03T20:50:52.097 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:52.096+0000 7fb97fc00640 1 -- 172.21.15.32:0/2909093109 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fb958077010 msgr2=0x7fb958079520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:50:52.097 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:52.096+0000 7fb97fc00640 1 --2- 172.21.15.32:0/2909093109 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fb958077010 0x7fb958079520 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7fb96c033000 tx=0x7fb96c004730 comp rx=0 tx=0).stop 2024-08-03T20:50:52.097 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:52.096+0000 7fb97fc00640 1 -- 172.21.15.32:0/2909093109 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb9780b4ce0 msgr2=0x7fb9781f68e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:50:52.097 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:52.096+0000 7fb97fc00640 1 --2- 172.21.15.32:0/2909093109 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb9780b4ce0 0x7fb9781f68e0 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7fb96400c3a0 tx=0x7fb96400c860 comp rx=0 tx=0).stop 2024-08-03T20:50:52.097 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:52.097+0000 7fb97fc00640 1 -- 172.21.15.32:0/2909093109 shutdown_connections 2024-08-03T20:50:52.097 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:52.097+0000 7fb97fc00640 1 --2- 172.21.15.32:0/2909093109 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fb958077010 0x7fb958079520 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:52.098 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:52.097+0000 7fb97fc00640 1 --2- 172.21.15.32:0/2909093109 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb978150f80 0x7fb9781f6e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:52.098 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:52.097+0000 7fb97fc00640 1 --2- 172.21.15.32:0/2909093109 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb9780b4ce0 0x7fb9781f68e0 unknown :-1 s=CLOSED pgs=624 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:52.098 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:52.097+0000 7fb97fc00640 1 --2- 172.21.15.32:0/2909093109 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fb9780b38b0 0x7fb9781428b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:52.098 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:52.097+0000 7fb97fc00640 1 -- 172.21.15.32:0/2909093109 >> 172.21.15.32:0/2909093109 conn(0x7fb9780b1bd0 msgr2=0x7fb97815aac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:50:52.098 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:52.097+0000 7fb97fc00640 1 -- 172.21.15.32:0/2909093109 shutdown_connections 2024-08-03T20:50:52.098 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:52.097+0000 7fb97fc00640 1 -- 172.21.15.32:0/2909093109 wait complete. 2024-08-03T20:50:52.522 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 58, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-08-03T20:50:27.688698+0000', 'modified': '2024-08-03T20:50:50.722012+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 58, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 6045}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6045': {'gid': 6045, 'name': 'c', 'rank': 0, 'incarnation': 55, 'state': 'up:active', 'state_seq': 5, 'addr': '172.21.15.32:6835/2963295201', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6834', 'nonce': 2963295201}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 2963295201}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [9], 'metadata_pool': 8, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 6045, 'qdb_cluster': [6045]} 2024-08-03T20:50:52.522 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-08-03T20:50:52.522 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-08-03T20:50:52.754 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:52.752+0000 7f9a56c00640 1 -- 172.21.15.32:0/2118199247 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9a5006e930 msgr2=0x7f9a50144f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:50:52.754 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:52.752+0000 7f9a56c00640 1 --2- 172.21.15.32:0/2118199247 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9a5006e930 0x7f9a50144f90 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7f9a40005fd0 tx=0x7f9a4002a540 comp rx=0 tx=0).stop 2024-08-03T20:50:52.754 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:52.752+0000 7f9a56c00640 1 -- 172.21.15.32:0/2118199247 shutdown_connections 2024-08-03T20:50:52.754 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:52.752+0000 7f9a56c00640 1 --2- 172.21.15.32:0/2118199247 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f9a50073e60 0x7f9a50074270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:52.754 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:52.752+0000 7f9a56c00640 1 --2- 172.21.15.32:0/2118199247 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9a5006e930 0x7f9a50144f90 unknown :-1 s=CLOSED pgs=626 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:52.754 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:52.752+0000 7f9a56c00640 1 -- 172.21.15.32:0/2118199247 >> 172.21.15.32:0/2118199247 conn(0x7f9a500b1bd0 msgr2=0x7f9a500b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:50:52.754 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:52.752+0000 7f9a56c00640 1 -- 172.21.15.32:0/2118199247 shutdown_connections 2024-08-03T20:50:52.754 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:52.752+0000 7f9a56c00640 1 -- 172.21.15.32:0/2118199247 wait complete. 2024-08-03T20:50:52.755 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:52.754+0000 7f9a56c00640 1 Processor -- start 2024-08-03T20:50:52.755 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:52.754+0000 7f9a56c00640 1 -- start start 2024-08-03T20:50:52.757 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:52.756+0000 7f9a56c00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f9a5006e930 0x7f9a500ae310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:50:52.757 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:52.756+0000 7f9a56c00640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f9a50073e60 0x7f9a500b39e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:50:52.757 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:52.756+0000 7f9a56c00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9a500747b0 0x7f9a5013ee30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:50:52.757 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:52.756+0000 7f9a56c00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f9a500cf9c0 con 0x7f9a500747b0 2024-08-03T20:50:52.758 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:52.756+0000 7f9a56c00640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f9a500cf840 con 0x7f9a5006e930 2024-08-03T20:50:52.758 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:52.756+0000 7f9a56c00640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7f9a500ccff0 con 0x7f9a50073e60 2024-08-03T20:50:52.758 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:52.756+0000 7f9a54c00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9a500747b0 0x7f9a5013ee30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:50:52.758 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:52.756+0000 7f9a54c00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9a500747b0 0x7f9a5013ee30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:35032/0 (socket says 172.21.15.32:35032) 2024-08-03T20:50:52.758 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:52.756+0000 7f9a54c00640 1 -- 172.21.15.32:0/2557767334 learned_addr learned my addr 172.21.15.32:0/2557767334 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:50:52.759 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:52.756+0000 7f9a4f400640 1 --2- 172.21.15.32:0/2557767334 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f9a50073e60 0x7f9a500b39e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:50:52.759 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:52.756+0000 7f9a4fe00640 1 --2- 172.21.15.32:0/2557767334 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f9a5006e930 0x7f9a500ae310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:50:52.759 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:52.756+0000 7f9a54c00640 1 -- 172.21.15.32:0/2557767334 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f9a50073e60 msgr2=0x7f9a500b39e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:50:52.760 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:52.756+0000 7f9a54c00640 1 --2- 172.21.15.32:0/2557767334 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f9a50073e60 0x7f9a500b39e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:52.760 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:52.756+0000 7f9a54c00640 1 -- 172.21.15.32:0/2557767334 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f9a5006e930 msgr2=0x7f9a500ae310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:50:52.760 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:52.756+0000 7f9a54c00640 1 --2- 172.21.15.32:0/2557767334 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f9a5006e930 0x7f9a500ae310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:52.760 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:52.756+0000 7f9a54c00640 1 -- 172.21.15.32:0/2557767334 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9a500b3f20 con 0x7f9a500747b0 2024-08-03T20:50:52.760 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:52.757+0000 7f9a54c00640 1 --2- 172.21.15.32:0/2557767334 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9a500747b0 0x7f9a5013ee30 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7f9a4800c730 tx=0x7f9a4800cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:50:52.760 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:52.757+0000 7f9a47e00640 1 -- 172.21.15.32:0/2557767334 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a48003b70 con 0x7f9a500747b0 2024-08-03T20:50:52.760 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:52.757+0000 7f9a47e00640 1 -- 172.21.15.32:0/2557767334 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f9a48003d30 con 0x7f9a500747b0 2024-08-03T20:50:52.760 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:52.757+0000 7f9a56c00640 1 -- 172.21.15.32:0/2557767334 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9a500af4b0 con 0x7f9a500747b0 2024-08-03T20:50:52.760 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:52.757+0000 7f9a47e00640 1 -- 172.21.15.32:0/2557767334 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a480052d0 con 0x7f9a500747b0 2024-08-03T20:50:52.760 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:52.757+0000 7f9a56c00640 1 -- 172.21.15.32:0/2557767334 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9a500adb10 con 0x7f9a500747b0 2024-08-03T20:50:52.761 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:52.760+0000 7f9a47e00640 1 -- 172.21.15.32:0/2557767334 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f9a48005680 con 0x7f9a500747b0 2024-08-03T20:50:52.763 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:52.761+0000 7f9a47e00640 1 --2- 172.21.15.32:0/2557767334 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f9a30077010 0x7f9a30079520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:50:52.763 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:52.762+0000 7f9a47e00640 1 -- 172.21.15.32:0/2557767334 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(58..58 src has 1..58) ==== 5795+0+0 (secure 0 0 0) 0x7f9a48098a70 con 0x7f9a500747b0 2024-08-03T20:50:52.763 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:52.762+0000 7f9a4fe00640 1 --2- 172.21.15.32:0/2557767334 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f9a30077010 0x7f9a30079520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:50:52.763 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:52.762+0000 7f9a5dbad740 1 Processor -- start 2024-08-03T20:50:52.764 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:52.762+0000 7f9a5dbad740 1 -- start start 2024-08-03T20:50:52.764 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:52.763+0000 7f9a4fe00640 1 --2- 172.21.15.32:0/2557767334 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f9a30077010 0x7f9a30079520 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f9a40033000 tx=0x7f9a40004730 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:50:52.766 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:52.764+0000 7f9a5dbad740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5621a44c1270 0x5621a48ae0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:50:52.766 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:52.764+0000 7f9a5dbad740 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x5621a44983b0 0x5621a4890da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:50:52.767 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:52.764+0000 7f9a5dbad740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5621a44a0590 0x5621a48912e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:50:52.767 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:52.764+0000 7f9a5dbad740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x5621a47d9a60 con 0x5621a44a0590 2024-08-03T20:50:52.767 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:52.764+0000 7f9a5dbad740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x5621a47c8f60 con 0x5621a44c1270 2024-08-03T20:50:52.767 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:52.764+0000 7f9a5dbad740 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x5621a47ccfa0 con 0x5621a44983b0 2024-08-03T20:50:52.767 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:52.764+0000 7f9a54c00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5621a44a0590 0x5621a48912e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:50:52.767 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:52.764+0000 7f9a54c00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5621a44a0590 0x5621a48912e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:35042/0 (socket says 172.21.15.32:35042) 2024-08-03T20:50:52.767 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:52.764+0000 7f9a54c00640 1 -- 172.21.15.32:0/817551974 learned_addr learned my addr 172.21.15.32:0/817551974 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:50:52.767 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:52.764+0000 7f9a4fe00640 1 --2- 172.21.15.32:0/817551974 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5621a44c1270 0x5621a48ae0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:50:52.767 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:52.764+0000 7f9a4f400640 1 --2- 172.21.15.32:0/817551974 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x5621a44983b0 0x5621a4890da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:50:52.767 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:52.764+0000 7f9a54c00640 1 -- 172.21.15.32:0/817551974 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x5621a44983b0 msgr2=0x5621a4890da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:50:52.767 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:52.764+0000 7f9a54c00640 1 --2- 172.21.15.32:0/817551974 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x5621a44983b0 0x5621a4890da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:52.767 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:52.764+0000 7f9a54c00640 1 -- 172.21.15.32:0/817551974 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5621a44c1270 msgr2=0x5621a48ae0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:50:52.767 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:52.764+0000 7f9a54c00640 1 --2- 172.21.15.32:0/817551974 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5621a44c1270 0x5621a48ae0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:52.768 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:52.764+0000 7f9a54c00640 1 -- 172.21.15.32:0/817551974 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5621a452c850 con 0x5621a44a0590 2024-08-03T20:50:52.768 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:52.764+0000 7f9a4fe00640 1 --2- 172.21.15.32:0/817551974 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5621a44c1270 0x5621a48ae0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-03T20:50:52.768 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:52.765+0000 7f9a54c00640 1 --2- 172.21.15.32:0/817551974 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5621a44a0590 0x5621a48912e0 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7f9a48096e10 tx=0x7f9a480021f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f2b5a99f1e53300e server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:50:52.768 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:52.765+0000 7f9a3be00640 1 -- 172.21.15.32:0/817551974 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a48095070 con 0x5621a44a0590 2024-08-03T20:50:52.768 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:52.765+0000 7f9a3be00640 1 -- 172.21.15.32:0/817551974 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f9a48061470 con 0x5621a44a0590 2024-08-03T20:50:52.768 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:52.765+0000 7f9a3be00640 1 -- 172.21.15.32:0/817551974 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a48065a70 con 0x5621a44a0590 2024-08-03T20:50:52.769 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:52.766+0000 7f9a5dbad740 1 -- 172.21.15.32:0/817551974 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5621a44a0590 msgr2=0x5621a48912e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:50:52.769 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:52.766+0000 7f9a5dbad740 1 --2- 172.21.15.32:0/817551974 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5621a44a0590 0x5621a48912e0 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7f9a48096e10 tx=0x7f9a480021f0 comp rx=0 tx=0).stop 2024-08-03T20:50:52.769 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:52.766+0000 7f9a5dbad740 1 -- 172.21.15.32:0/817551974 shutdown_connections 2024-08-03T20:50:52.769 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:52.766+0000 7f9a5dbad740 1 --2- 172.21.15.32:0/817551974 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5621a44a0590 0x5621a48912e0 unknown :-1 s=CLOSED pgs=628 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:52.769 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:52.766+0000 7f9a5dbad740 1 --2- 172.21.15.32:0/817551974 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x5621a44983b0 0x5621a4890da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:52.769 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:52.766+0000 7f9a5dbad740 1 --2- 172.21.15.32:0/817551974 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5621a44c1270 0x5621a48ae0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:52.769 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:52.766+0000 7f9a5dbad740 1 -- 172.21.15.32:0/817551974 >> 172.21.15.32:0/817551974 conn(0x5621a44b53c0 msgr2=0x5621a4866fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:50:52.769 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:52.766+0000 7f9a5dbad740 1 -- 172.21.15.32:0/817551974 shutdown_connections 2024-08-03T20:50:52.769 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:52.766+0000 7f9a5dbad740 1 -- 172.21.15.32:0/817551974 wait complete. 2024-08-03T20:50:52.770 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:52.767+0000 7f9a5dbad740 1 -- start start 2024-08-03T20:50:52.770 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:52.767+0000 7f9a5dbad740 1 Processor -- start 2024-08-03T20:50:52.773 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:52.768+0000 7f9a5dbad740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5621a44983b0 0x5621a48a4c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:50:52.773 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:52.769+0000 7f9a5dbad740 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x5621a44a0590 0x5621a48bc940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:50:52.773 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:52.769+0000 7f9a5dbad740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5621a44c1270 0x5621a48a75d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:50:52.773 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:52.769+0000 7f9a5dbad740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x5621a453b620 con 0x5621a44c1270 2024-08-03T20:50:52.773 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:52.769+0000 7f9a5dbad740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x5621a4766160 con 0x5621a44983b0 2024-08-03T20:50:52.773 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:52.769+0000 7f9a5dbad740 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x5621a48bd3c0 con 0x5621a44a0590 2024-08-03T20:50:52.773 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:52.769+0000 7f9a5dbad740 10 client.0 _ll_register_callbacks cb 0x5621a47756b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-03T20:50:52.773 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:52.769+0000 7f9a54c00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5621a44c1270 0x5621a48a75d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:50:52.773 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:52.769+0000 7f9a54c00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5621a44c1270 0x5621a48a75d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:35052/0 (socket says 172.21.15.32:35052) 2024-08-03T20:50:52.773 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:52.769+0000 7f9a54c00640 1 -- 172.21.15.32:0/1938289195 learned_addr learned my addr 172.21.15.32:0/1938289195 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:50:52.774 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:52.769+0000 7f9a4f400640 1 --2- 172.21.15.32:0/1938289195 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x5621a44a0590 0x5621a48bc940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:50:52.774 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:52.769+0000 7f9a4fe00640 1 --2- 172.21.15.32:0/1938289195 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5621a44983b0 0x5621a48a4c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:50:52.774 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:52.769+0000 7f9a54c00640 1 -- 172.21.15.32:0/1938289195 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x5621a44a0590 msgr2=0x5621a48bc940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:50:52.774 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:52.769+0000 7f9a54c00640 1 --2- 172.21.15.32:0/1938289195 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x5621a44a0590 0x5621a48bc940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:52.774 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:52.769+0000 7f9a54c00640 1 -- 172.21.15.32:0/1938289195 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5621a44983b0 msgr2=0x5621a48a4c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:50:52.774 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:52.769+0000 7f9a54c00640 1 --2- 172.21.15.32:0/1938289195 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5621a44983b0 0x5621a48a4c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:52.774 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:52.769+0000 7f9a54c00640 1 -- 172.21.15.32:0/1938289195 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5621a45a36b0 con 0x5621a44c1270 2024-08-03T20:50:52.774 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:52.770+0000 7f9a54c00640 1 --2- 172.21.15.32:0/1938289195 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5621a44c1270 0x5621a48a75d0 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7f9a48001870 tx=0x7f9a4805d3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3d6259fb127a7777 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:50:52.774 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:52.770+0000 7f9a38c00640 10 client.0 ms_handle_connect on v2:172.21.15.32:3300/0 2024-08-03T20:50:52.774 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:52.770+0000 7f9a38c00640 1 -- 172.21.15.32:0/1938289195 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a48095040 con 0x5621a44c1270 2024-08-03T20:50:52.774 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:52.770+0000 7f9a38c00640 1 -- 172.21.15.32:0/1938289195 <== mon.0 v2:172.21.15.32:3300/0 2 ==== osd_map(58..58 src has 1..58) ==== 5795+0+0 (secure 0 0 0) 0x7f9a4806a290 con 0x5621a44c1270 2024-08-03T20:50:52.774 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:52.770+0000 7f9a5dbad740 1 -- 172.21.15.32:0/1938289195 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9a48061470 con 0x5621a44c1270 2024-08-03T20:50:52.774 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:52.770+0000 7f9a38c00640 1 -- 172.21.15.32:0/1938289195 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_get_version_reply(handle=1 version=58) ==== 24+0+0 (secure 0 0 0) 0x7f9a48065a70 con 0x5621a44c1270 2024-08-03T20:50:52.775 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:52.770+0000 7f9a5dbad740 10 client.6090 fetch_fsmap learned FSMap version 58 2024-08-03T20:50:52.775 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:52.770+0000 7f9a5dbad740 1 -- 172.21.15.32:0/1938289195 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,fsmap=58,monmap=2+}) -- 0x5621a48bce80 con 0x5621a44c1270 2024-08-03T20:50:52.775 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:52.770+0000 7f9a38c00640 1 -- 172.21.15.32:0/1938289195 <== mon.0 v2:172.21.15.32:3300/0 4 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f9a48061470 con 0x5621a44c1270 2024-08-03T20:50:52.775 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:52.770+0000 7f9a38c00640 1 -- 172.21.15.32:0/1938289195 <== mon.0 v2:172.21.15.32:3300/0 5 ==== fsmap(e 58) ==== 3502+0+0 (secure 0 0 0) 0x7f9a48006c20 con 0x5621a44c1270 2024-08-03T20:50:52.775 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:52.770+0000 7f9a5dbad740 10 client.6090 fetch_fsmap finished waiting for FSMap version 58 2024-08-03T20:50:52.775 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:52.770+0000 7f9a5dbad740 10 client.6090 resolve_mds: resolved name 'c' to daemon mds.c 2024-08-03T20:50:52.775 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:52.771+0000 7f9a5dbad740 20 client.6090 populate_metadata read hostname 'smithi032' 2024-08-03T20:50:52.775 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:52.771+0000 7f9a5dbad740 1 --2- 172.21.15.32:0/1938289195 >> [v2:172.21.15.32:6834/2963295201,v1:172.21.15.32:6835/2963295201] conn(0x5621a447fe40 0x5621a48bb7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:50:52.775 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:52.771+0000 7f9a5dbad740 4 client.6090 mds_command: new command op to 6045 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-08-03T20:50:52.775 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:52.771+0000 7f9a5dbad740 1 -- 172.21.15.32:0/1938289195 --> [v2:172.21.15.32:6834/2963295201,v1:172.21.15.32:6835/2963295201] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5621a47db9b0 con 0x5621a447fe40 2024-08-03T20:50:52.775 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:52.771+0000 7f9a4fe00640 1 --2- 172.21.15.32:0/1938289195 >> [v2:172.21.15.32:6834/2963295201,v1:172.21.15.32:6835/2963295201] conn(0x5621a447fe40 0x5621a48bb7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:50:52.775 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:52.772+0000 7f9a4fe00640 1 --2- 172.21.15.32:0/1938289195 >> [v2:172.21.15.32:6834/2963295201,v1:172.21.15.32:6835/2963295201] conn(0x5621a447fe40 0x5621a48bb7c0 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d51acefa51adb0b server_cookie=e1bc6e222cd6f583 in_seq=0 out_seq=0 2024-08-03T20:50:52.775 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:52.772+0000 7f9a38c00640 10 client.6090 ms_handle_connect on v2:172.21.15.32:6834/2963295201 2024-08-03T20:50:52.776 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:52.772+0000 7f9a38c00640 1 -- 172.21.15.32:0/1938289195 <== mds.0 v2:172.21.15.32:6834/2963295201 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x5621a47db9b0 con 0x5621a447fe40 2024-08-03T20:50:52.777 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:52.772+0000 7f9a38c00640 10 client.6090 handle_command_reply: tid=0 2024-08-03T20:50:52.777 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:52.772+0000 7f9a5dbad740 1 client.6090 shutdown 2024-08-03T20:50:52.777 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:52.773+0000 7f9a5dbad740 1 -- 172.21.15.32:0/1938289195 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5621a44c1270 msgr2=0x5621a48a75d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:50:52.777 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:52.773+0000 7f9a5dbad740 1 --2- 172.21.15.32:0/1938289195 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5621a44c1270 0x5621a48a75d0 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7f9a48001870 tx=0x7f9a4805d3c0 comp rx=0 tx=0).stop 2024-08-03T20:50:52.777 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:52.773+0000 7f9a5dbad740 1 -- 172.21.15.32:0/1938289195 shutdown_connections 2024-08-03T20:50:52.777 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:52.773+0000 7f9a5dbad740 1 --2- 172.21.15.32:0/1938289195 >> [v2:172.21.15.32:6834/2963295201,v1:172.21.15.32:6835/2963295201] conn(0x5621a447fe40 0x5621a48bb7c0 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:52.777 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:52.773+0000 7f9a5dbad740 1 --2- 172.21.15.32:0/1938289195 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5621a44c1270 0x5621a48a75d0 unknown :-1 s=CLOSED pgs=629 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:52.777 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:52.773+0000 7f9a5dbad740 1 --2- 172.21.15.32:0/1938289195 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x5621a44a0590 0x5621a48bc940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:52.777 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:52.773+0000 7f9a5dbad740 1 --2- 172.21.15.32:0/1938289195 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5621a44983b0 0x5621a48a4c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:52.777 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:52.773+0000 7f9a5dbad740 1 -- 172.21.15.32:0/1938289195 >> 172.21.15.32:0/1938289195 conn(0x5621a44b53c0 msgr2=0x5621a4860390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:50:52.777 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:52.773+0000 7f9a38c00640 0 client.6090 ms_handle_reset on v2:172.21.15.32:6834/2963295201 2024-08-03T20:50:52.777 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:52.773+0000 7f9a5dbad740 1 -- 172.21.15.32:0/1938289195 shutdown_connections 2024-08-03T20:50:52.777 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:52.773+0000 7f9a5dbad740 1 -- 172.21.15.32:0/1938289195 wait complete. 2024-08-03T20:50:52.778 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:52.773+0000 7f9a5dbad740 20 client.6090 trim_cache size 0 max 16384 2024-08-03T20:50:52.795 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:52.794+0000 7f9a5dbad740 1 Processor -- start 2024-08-03T20:50:52.795 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:52.794+0000 7f9a5dbad740 1 -- start start 2024-08-03T20:50:52.797 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:52.795+0000 7f9a5dbad740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5621a44983b0 0x5621a48912f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:50:52.797 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:52.795+0000 7f9a5dbad740 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x5621a44a0590 0x5621a48809a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:50:52.797 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:52.795+0000 7f9a5dbad740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5621a44c1270 0x5621a4880ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:50:52.797 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:52.795+0000 7f9a5dbad740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f9a50200e20 con 0x5621a44983b0 2024-08-03T20:50:52.797 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:52.795+0000 7f9a5dbad740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f9a500b4de0 con 0x5621a44c1270 2024-08-03T20:50:52.797 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:52.795+0000 7f9a5dbad740 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7f9a501f9210 con 0x5621a44a0590 2024-08-03T20:50:52.797 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:52.795+0000 7f9a4fe00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5621a44983b0 0x5621a48912f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:50:52.797 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:52.795+0000 7f9a4fe00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5621a44983b0 0x5621a48912f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:35066/0 (socket says 172.21.15.32:35066) 2024-08-03T20:50:52.797 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:52.795+0000 7f9a4fe00640 1 -- 172.21.15.32:0/2167224532 learned_addr learned my addr 172.21.15.32:0/2167224532 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:50:52.799 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:52.796+0000 7f9a4f400640 1 --2- 172.21.15.32:0/2167224532 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x5621a44a0590 0x5621a48809a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:50:52.799 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:52.796+0000 7f9a4fe00640 1 -- 172.21.15.32:0/2167224532 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x5621a44a0590 msgr2=0x5621a48809a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:50:52.799 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:52.796+0000 7f9a4fe00640 1 --2- 172.21.15.32:0/2167224532 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x5621a44a0590 0x5621a48809a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:52.799 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:52.796+0000 7f9a4fe00640 1 -- 172.21.15.32:0/2167224532 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5621a44c1270 msgr2=0x5621a4880ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:50:52.799 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:52.796+0000 7f9a4fe00640 1 --2- 172.21.15.32:0/2167224532 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5621a44c1270 0x5621a4880ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:52.799 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:52.796+0000 7f9a4fe00640 1 -- 172.21.15.32:0/2167224532 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5621a4868860 con 0x5621a44983b0 2024-08-03T20:50:52.799 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:52.796+0000 7f9a4f400640 1 --2- 172.21.15.32:0/2167224532 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x5621a44a0590 0x5621a48809a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-03T20:50:52.800 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:52.796+0000 7f9a4fe00640 1 --2- 172.21.15.32:0/2167224532 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5621a44983b0 0x5621a48912f0 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7f9a400099f0 tx=0x7f9a4003c000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=59e0fec01d6a5a2 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:50:52.800 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:52.796+0000 7f9a3b400640 1 -- 172.21.15.32:0/2167224532 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a40035040 con 0x5621a44983b0 2024-08-03T20:50:52.800 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:52.796+0000 7f9a3b400640 1 -- 172.21.15.32:0/2167224532 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f9a4004e030 con 0x5621a44983b0 2024-08-03T20:50:52.800 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:52.796+0000 7f9a3b400640 1 -- 172.21.15.32:0/2167224532 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a400049c0 con 0x5621a44983b0 2024-08-03T20:50:52.800 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:52.797+0000 7f9a5dbad740 1 -- 172.21.15.32:0/2167224532 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5621a44983b0 msgr2=0x5621a48912f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:50:52.800 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:52.797+0000 7f9a5dbad740 1 --2- 172.21.15.32:0/2167224532 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5621a44983b0 0x5621a48912f0 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7f9a400099f0 tx=0x7f9a4003c000 comp rx=0 tx=0).stop 2024-08-03T20:50:52.800 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:52.797+0000 7f9a5dbad740 1 -- 172.21.15.32:0/2167224532 shutdown_connections 2024-08-03T20:50:52.800 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:52.797+0000 7f9a5dbad740 1 --2- 172.21.15.32:0/2167224532 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5621a44c1270 0x5621a4880ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:52.801 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:52.797+0000 7f9a5dbad740 1 --2- 172.21.15.32:0/2167224532 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x5621a44a0590 0x5621a48809a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:52.801 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:52.797+0000 7f9a5dbad740 1 --2- 172.21.15.32:0/2167224532 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5621a44983b0 0x5621a48912f0 unknown :-1 s=CLOSED pgs=630 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:52.801 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:52.797+0000 7f9a5dbad740 1 -- 172.21.15.32:0/2167224532 >> 172.21.15.32:0/2167224532 conn(0x5621a44b53c0 msgr2=0x5621a48616d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:50:52.801 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:52.797+0000 7f9a5dbad740 1 -- 172.21.15.32:0/2167224532 shutdown_connections 2024-08-03T20:50:52.801 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:52.797+0000 7f9a5dbad740 1 -- 172.21.15.32:0/2167224532 wait complete. 2024-08-03T20:50:52.805 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:52.798+0000 7f9a5dbad740 1 -- start start 2024-08-03T20:50:52.805 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:52.798+0000 7f9a5dbad740 1 Processor -- start 2024-08-03T20:50:52.805 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:52.800+0000 7f9a5dbad740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5621a44983b0 0x5621a48bd910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:50:52.805 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:52.800+0000 7f9a5dbad740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5621a44a0590 0x5621a48bdf60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:50:52.805 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:52.800+0000 7f9a5dbad740 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x5621a44c1270 0x5621a4926550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:50:52.805 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:52.800+0000 7f9a5dbad740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x5621a453b620 con 0x5621a44a0590 2024-08-03T20:50:52.806 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:52.800+0000 7f9a5dbad740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x5621a4766160 con 0x5621a44983b0 2024-08-03T20:50:52.806 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:52.800+0000 7f9a5dbad740 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x5621a487f5e0 con 0x5621a44c1270 2024-08-03T20:50:52.806 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:52.800+0000 7f9a5dbad740 10 client.0 _ll_register_callbacks cb 0x7f9a480612d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-03T20:50:52.806 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:52.800+0000 7f9a4f400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5621a44a0590 0x5621a48bdf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:50:52.806 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:52.800+0000 7f9a4f400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5621a44a0590 0x5621a48bdf60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:35070/0 (socket says 172.21.15.32:35070) 2024-08-03T20:50:52.806 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:52.800+0000 7f9a4f400640 1 -- 172.21.15.32:0/2656884311 learned_addr learned my addr 172.21.15.32:0/2656884311 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:50:52.806 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:52.800+0000 7f9a4fe00640 1 --2- 172.21.15.32:0/2656884311 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5621a44983b0 0x5621a48bd910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:50:52.806 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:52.800+0000 7f9a54c00640 1 --2- 172.21.15.32:0/2656884311 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x5621a44c1270 0x5621a4926550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:50:52.806 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:52.800+0000 7f9a4f400640 1 -- 172.21.15.32:0/2656884311 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x5621a44c1270 msgr2=0x5621a4926550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:50:52.806 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:52.800+0000 7f9a4f400640 1 --2- 172.21.15.32:0/2656884311 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x5621a44c1270 0x5621a4926550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:52.806 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:52.800+0000 7f9a4f400640 1 -- 172.21.15.32:0/2656884311 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5621a44983b0 msgr2=0x5621a48bd910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:50:52.806 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:52.800+0000 7f9a4f400640 1 --2- 172.21.15.32:0/2656884311 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5621a44983b0 0x5621a48bd910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:52.806 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:52.800+0000 7f9a4f400640 1 -- 172.21.15.32:0/2656884311 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5621a48bb520 con 0x5621a44a0590 2024-08-03T20:50:52.807 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:52.801+0000 7f9a4f400640 1 --2- 172.21.15.32:0/2656884311 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5621a44a0590 0x5621a48bdf60 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7f9a500b1480 tx=0x7f9a3c00dda0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7a643ea12d0884f9 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:50:52.807 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:52.801+0000 7f9a38c00640 10 client.0 ms_handle_connect on v2:172.21.15.32:3300/0 2024-08-03T20:50:52.807 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:52.801+0000 7f9a38c00640 1 -- 172.21.15.32:0/2656884311 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a3c003ce0 con 0x5621a44a0590 2024-08-03T20:50:52.807 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:52.801+0000 7f9a38c00640 1 -- 172.21.15.32:0/2656884311 <== mon.0 v2:172.21.15.32:3300/0 2 ==== osd_map(58..58 src has 1..58) ==== 5795+0+0 (secure 0 0 0) 0x7f9a3c01eb30 con 0x5621a44a0590 2024-08-03T20:50:52.807 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:52.801+0000 7f9a5dbad740 1 -- 172.21.15.32:0/2656884311 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9a4004e030 con 0x5621a44a0590 2024-08-03T20:50:52.807 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:52.801+0000 7f9a38c00640 1 -- 172.21.15.32:0/2656884311 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_get_version_reply(handle=1 version=58) ==== 24+0+0 (secure 0 0 0) 0x7f9a3c003e80 con 0x5621a44a0590 2024-08-03T20:50:52.807 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:52.801+0000 7f9a5dbad740 10 client.6096 fetch_fsmap learned FSMap version 58 2024-08-03T20:50:52.807 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:52.801+0000 7f9a5dbad740 1 -- 172.21.15.32:0/2656884311 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,fsmap=58,monmap=2+}) -- 0x5621a4929430 con 0x5621a44a0590 2024-08-03T20:50:52.807 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:52.802+0000 7f9a38c00640 1 -- 172.21.15.32:0/2656884311 <== mon.0 v2:172.21.15.32:3300/0 4 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f9a4004e030 con 0x5621a44a0590 2024-08-03T20:50:52.807 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:52.802+0000 7f9a38c00640 1 -- 172.21.15.32:0/2656884311 <== mon.0 v2:172.21.15.32:3300/0 5 ==== fsmap(e 58) ==== 3502+0+0 (secure 0 0 0) 0x7f9a3c025050 con 0x5621a44a0590 2024-08-03T20:50:52.807 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:52.802+0000 7f9a5dbad740 10 client.6096 fetch_fsmap finished waiting for FSMap version 58 2024-08-03T20:50:52.807 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:52.802+0000 7f9a5dbad740 10 client.6096 resolve_mds: resolved name 'c' to daemon mds.c 2024-08-03T20:50:52.807 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:52.802+0000 7f9a5dbad740 20 client.6096 populate_metadata read hostname 'smithi032' 2024-08-03T20:50:52.808 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:52.802+0000 7f9a5dbad740 1 --2- 172.21.15.32:0/2656884311 >> [v2:172.21.15.32:6834/2963295201,v1:172.21.15.32:6835/2963295201] conn(0x5621a48650f0 0x5621a48a47f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:50:52.808 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:52.802+0000 7f9a5dbad740 4 client.6096 mds_command: new command op to 6045 tid=0 multi_id=0 [{"prefix": "status"}] 2024-08-03T20:50:52.808 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:52.802+0000 7f9a5dbad740 1 -- 172.21.15.32:0/2656884311 --> [v2:172.21.15.32:6834/2963295201,v1:172.21.15.32:6835/2963295201] -- command(tid 0: {"prefix": "status"}) -- 0x5621a47d7450 con 0x5621a48650f0 2024-08-03T20:50:52.808 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:52.802+0000 7f9a4fe00640 1 --2- 172.21.15.32:0/2656884311 >> [v2:172.21.15.32:6834/2963295201,v1:172.21.15.32:6835/2963295201] conn(0x5621a48650f0 0x5621a48a47f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:50:52.808 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:52.803+0000 7f9a4fe00640 1 --2- 172.21.15.32:0/2656884311 >> [v2:172.21.15.32:6834/2963295201,v1:172.21.15.32:6835/2963295201] conn(0x5621a48650f0 0x5621a48a47f0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cbfc80f42cfd8442 server_cookie=ab89a84e86ed8f56 in_seq=0 out_seq=0 2024-08-03T20:50:52.808 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:52.803+0000 7f9a38c00640 10 client.6096 ms_handle_connect on v2:172.21.15.32:6834/2963295201 2024-08-03T20:50:52.808 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:52.803+0000 7f9a38c00640 1 -- 172.21.15.32:0/2656884311 <== mds.0 v2:172.21.15.32:6834/2963295201 1 ==== command_reply(tid 0: 0 ) ==== 8+0+325 (crc 0 0 0) 0x5621a47d7450 con 0x5621a48650f0 2024-08-03T20:50:52.808 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:52.803+0000 7f9a38c00640 10 client.6096 handle_command_reply: tid=0 2024-08-03T20:50:52.808 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:52.803+0000 7f9a5dbad740 1 client.6096 shutdown 2024-08-03T20:50:52.808 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:52.803+0000 7f9a5dbad740 1 -- 172.21.15.32:0/2656884311 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5621a44a0590 msgr2=0x5621a48bdf60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:50:52.808 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:52.803+0000 7f9a5dbad740 1 --2- 172.21.15.32:0/2656884311 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5621a44a0590 0x5621a48bdf60 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7f9a500b1480 tx=0x7f9a3c00dda0 comp rx=0 tx=0).stop 2024-08-03T20:50:52.808 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:52.803+0000 7f9a5dbad740 1 -- 172.21.15.32:0/2656884311 shutdown_connections 2024-08-03T20:50:52.808 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:52.803+0000 7f9a5dbad740 1 --2- 172.21.15.32:0/2656884311 >> [v2:172.21.15.32:6834/2963295201,v1:172.21.15.32:6835/2963295201] conn(0x5621a48650f0 0x5621a48a47f0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:52.808 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:52.803+0000 7f9a5dbad740 1 --2- 172.21.15.32:0/2656884311 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x5621a44c1270 0x5621a4926550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:52.809 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:52.803+0000 7f9a5dbad740 1 --2- 172.21.15.32:0/2656884311 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5621a44a0590 0x5621a48bdf60 unknown :-1 s=CLOSED pgs=631 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:52.809 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:52.803+0000 7f9a5dbad740 1 --2- 172.21.15.32:0/2656884311 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5621a44983b0 0x5621a48bd910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:52.809 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:52.803+0000 7f9a38c00640 0 client.6096 ms_handle_reset on v2:172.21.15.32:6834/2963295201 2024-08-03T20:50:52.809 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:52.803+0000 7f9a5dbad740 1 -- 172.21.15.32:0/2656884311 >> 172.21.15.32:0/2656884311 conn(0x5621a44b53c0 msgr2=0x5621a48667d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:50:52.809 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:52.803+0000 7f9a5dbad740 1 -- 172.21.15.32:0/2656884311 shutdown_connections 2024-08-03T20:50:52.809 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:52.803+0000 7f9a5dbad740 1 -- 172.21.15.32:0/2656884311 wait complete. 2024-08-03T20:50:52.809 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:52.803+0000 7f9a5dbad740 20 client.6096 trim_cache size 0 max 16384 2024-08-03T20:50:52.809 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-08-03T20:50:52.809 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_fsid": "40ab141a-01d7-49b9-ae3c-2edec9876e95", 2024-08-03T20:50:52.809 INFO:teuthology.orchestra.run.smithi032.stdout: "whoami": 0, 2024-08-03T20:50:52.810 INFO:teuthology.orchestra.run.smithi032.stdout: "id": 6045, 2024-08-03T20:50:52.810 INFO:teuthology.orchestra.run.smithi032.stdout: "want_state": "up:active", 2024-08-03T20:50:52.810 INFO:teuthology.orchestra.run.smithi032.stdout: "state": "up:active", 2024-08-03T20:50:52.810 INFO:teuthology.orchestra.run.smithi032.stdout: "fs_name": "cephfs", 2024-08-03T20:50:52.810 INFO:teuthology.orchestra.run.smithi032.stdout: "rank_uptime": 5.0947848709999999, 2024-08-03T20:50:52.810 INFO:teuthology.orchestra.run.smithi032.stdout: "mdsmap_epoch": 58, 2024-08-03T20:50:52.810 INFO:teuthology.orchestra.run.smithi032.stdout: "osdmap_epoch": 58, 2024-08-03T20:50:52.810 INFO:teuthology.orchestra.run.smithi032.stdout: "osdmap_epoch_barrier": 58, 2024-08-03T20:50:52.810 INFO:teuthology.orchestra.run.smithi032.stdout: "uptime": 11.978511626 2024-08-03T20:50:52.810 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-08-03T20:50:52.811 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:52.804+0000 7f9a56c00640 1 -- 172.21.15.32:0/2557767334 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f9a30077010 msgr2=0x7f9a30079520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:50:52.811 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:52.804+0000 7f9a56c00640 1 --2- 172.21.15.32:0/2557767334 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f9a30077010 0x7f9a30079520 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f9a40033000 tx=0x7f9a40004730 comp rx=0 tx=0).stop 2024-08-03T20:50:52.811 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:52.804+0000 7f9a56c00640 1 -- 172.21.15.32:0/2557767334 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9a500747b0 msgr2=0x7f9a5013ee30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:50:52.812 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:52.804+0000 7f9a56c00640 1 --2- 172.21.15.32:0/2557767334 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9a500747b0 0x7f9a5013ee30 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7f9a4800c730 tx=0x7f9a4800cbf0 comp rx=0 tx=0).stop 2024-08-03T20:50:52.812 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:52.804+0000 7f9a56c00640 1 -- 172.21.15.32:0/2557767334 shutdown_connections 2024-08-03T20:50:52.812 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:52.804+0000 7f9a56c00640 1 --2- 172.21.15.32:0/2557767334 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f9a30077010 0x7f9a30079520 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:52.812 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:52.804+0000 7f9a56c00640 1 --2- 172.21.15.32:0/2557767334 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9a500747b0 0x7f9a5013ee30 unknown :-1 s=CLOSED pgs=627 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:52.812 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:52.804+0000 7f9a56c00640 1 --2- 172.21.15.32:0/2557767334 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f9a50073e60 0x7f9a500b39e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:52.812 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:52.804+0000 7f9a56c00640 1 --2- 172.21.15.32:0/2557767334 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f9a5006e930 0x7f9a500ae310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:52.812 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:52.804+0000 7f9a56c00640 1 -- 172.21.15.32:0/2557767334 >> 172.21.15.32:0/2557767334 conn(0x7f9a500b1bd0 msgr2=0x7f9a50140cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:50:52.812 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:52.804+0000 7f9a56c00640 1 -- 172.21.15.32:0/2557767334 shutdown_connections 2024-08-03T20:50:52.812 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:52.804+0000 7f9a56c00640 1 -- 172.21.15.32:0/2557767334 wait complete. 2024-08-03T20:50:52.831 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-08-03T20:50:53.060 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:53.058+0000 7fa421e00640 1 -- 172.21.15.32:0/3665476619 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa41c0cfc50 msgr2=0x7fa41c0b5e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:50:53.061 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:53.058+0000 7fa421e00640 1 --2- 172.21.15.32:0/3665476619 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa41c0cfc50 0x7fa41c0b5e00 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7fa408005fd0 tx=0x7fa40802a5d0 comp rx=0 tx=0).stop 2024-08-03T20:50:53.061 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:53.059+0000 7fa421e00640 1 -- 172.21.15.32:0/3665476619 shutdown_connections 2024-08-03T20:50:53.061 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:53.059+0000 7fa421e00640 1 --2- 172.21.15.32:0/3665476619 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa41c0cfc50 0x7fa41c0b5e00 unknown :-1 s=CLOSED pgs=633 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:53.061 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:53.059+0000 7fa421e00640 1 --2- 172.21.15.32:0/3665476619 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fa41c0d3960 0x7fa41c0cf710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:53.061 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:53.059+0000 7fa421e00640 1 -- 172.21.15.32:0/3665476619 >> 172.21.15.32:0/3665476619 conn(0x7fa41c0d0fe0 msgr2=0x7fa41c0d13f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:50:53.061 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:53.059+0000 7fa421e00640 1 -- 172.21.15.32:0/3665476619 shutdown_connections 2024-08-03T20:50:53.061 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:53.059+0000 7fa421e00640 1 -- 172.21.15.32:0/3665476619 wait complete. 2024-08-03T20:50:53.062 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:53.061+0000 7fa421e00640 1 Processor -- start 2024-08-03T20:50:53.062 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:53.061+0000 7fa421e00640 1 -- start start 2024-08-03T20:50:53.065 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:53.062+0000 7fa421e00640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fa41c0b6340 0x7fa41c14c810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:50:53.065 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:53.062+0000 7fa421e00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa41c0cfc50 0x7fa41c0cdd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:50:53.065 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:53.063+0000 7fa421e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa41c0d3960 0x7fa41c0ce990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:50:53.065 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:53.063+0000 7fa421e00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fa41c0b7f60 con 0x7fa41c0d3960 2024-08-03T20:50:53.065 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:53.063+0000 7fa421e00640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7fa41c0d2680 con 0x7fa41c0cfc50 2024-08-03T20:50:53.065 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:53.063+0000 7fa421e00640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7fa41c0b80e0 con 0x7fa41c0b6340 2024-08-03T20:50:53.065 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:53.063+0000 7fa41b400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa41c0d3960 0x7fa41c0ce990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:50:53.065 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:53.063+0000 7fa41b400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa41c0d3960 0x7fa41c0ce990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:35086/0 (socket says 172.21.15.32:35086) 2024-08-03T20:50:53.066 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:53.063+0000 7fa41b400640 1 -- 172.21.15.32:0/304519264 learned_addr learned my addr 172.21.15.32:0/304519264 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:50:53.066 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:53.063+0000 7fa41aa00640 1 --2- 172.21.15.32:0/304519264 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fa41c0b6340 0x7fa41c14c810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:50:53.066 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:53.063+0000 7fa41a000640 1 --2- 172.21.15.32:0/304519264 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa41c0cfc50 0x7fa41c0cdd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:50:53.066 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:53.063+0000 7fa41b400640 1 -- 172.21.15.32:0/304519264 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fa41c0b6340 msgr2=0x7fa41c14c810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:50:53.066 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:53.063+0000 7fa41b400640 1 --2- 172.21.15.32:0/304519264 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fa41c0b6340 0x7fa41c14c810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:53.066 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:53.063+0000 7fa41b400640 1 -- 172.21.15.32:0/304519264 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa41c0cfc50 msgr2=0x7fa41c0cdd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:50:53.066 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:53.063+0000 7fa41b400640 1 --2- 172.21.15.32:0/304519264 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa41c0cfc50 0x7fa41c0cdd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:53.066 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:53.063+0000 7fa41b400640 1 -- 172.21.15.32:0/304519264 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa41c1f9b60 con 0x7fa41c0d3960 2024-08-03T20:50:53.066 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:53.063+0000 7fa41a000640 1 --2- 172.21.15.32:0/304519264 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa41c0cfc50 0x7fa41c0cdd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-03T20:50:53.067 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:53.064+0000 7fa41b400640 1 --2- 172.21.15.32:0/304519264 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa41c0d3960 0x7fa41c0ce990 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7fa41400c6b0 tx=0x7fa41400cb70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:50:53.067 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:53.064+0000 7fa413400640 1 -- 172.21.15.32:0/304519264 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa414003b70 con 0x7fa41c0d3960 2024-08-03T20:50:53.067 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:53.064+0000 7fa413400640 1 -- 172.21.15.32:0/304519264 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fa414003d10 con 0x7fa41c0d3960 2024-08-03T20:50:53.067 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:53.064+0000 7fa421e00640 1 -- 172.21.15.32:0/304519264 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa41c1f92c0 con 0x7fa41c0d3960 2024-08-03T20:50:53.067 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:53.064+0000 7fa413400640 1 -- 172.21.15.32:0/304519264 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa414005330 con 0x7fa41c0d3960 2024-08-03T20:50:53.067 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:53.064+0000 7fa421e00640 1 -- 172.21.15.32:0/304519264 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa41c1f9880 con 0x7fa41c0d3960 2024-08-03T20:50:53.068 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:53.065+0000 7fa421e00640 1 -- 172.21.15.32:0/304519264 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa41c0ce6b0 con 0x7fa41c0d3960 2024-08-03T20:50:53.068 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:53.067+0000 7fa413400640 1 -- 172.21.15.32:0/304519264 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7fa4140054d0 con 0x7fa41c0d3960 2024-08-03T20:50:53.069 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:53.067+0000 7fa413400640 1 --2- 172.21.15.32:0/304519264 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fa3fc077010 0x7fa3fc079520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:50:53.069 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:53.068+0000 7fa413400640 1 -- 172.21.15.32:0/304519264 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(58..58 src has 1..58) ==== 5795+0+0 (secure 0 0 0) 0x7fa414098790 con 0x7fa41c0d3960 2024-08-03T20:50:53.069 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:53.068+0000 7fa41aa00640 1 --2- 172.21.15.32:0/304519264 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fa3fc077010 0x7fa3fc079520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:50:53.070 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:53.069+0000 7fa41aa00640 1 --2- 172.21.15.32:0/304519264 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fa3fc077010 0x7fa3fc079520 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7fa40c009b30 tx=0x7fa40c005f90 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:50:53.073 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:53.072+0000 7fa413400640 1 -- 172.21.15.32:0/304519264 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7fa41c0ce6b0 con 0x7fa41c0d3960 2024-08-03T20:50:53.319 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:53.317+0000 7fa421e00640 1 -- 172.21.15.32:0/304519264 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa41c0b5e00 con 0x7fa41c0d3960 2024-08-03T20:50:53.319 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:53.318+0000 7fa413400640 1 -- 172.21.15.32:0/304519264 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 58 v58) ==== 94+0+5230 (secure 0 0 0) 0x7fa41c0b6750 con 0x7fa41c0d3960 2024-08-03T20:50:53.320 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-03T20:50:53.320 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":58,"btime":"2024-08-03T20:50:50:722022+0000","default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5261,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6833/2498840646","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":2498840646},{"type":"v1","addr":"172.21.15.32:6833","nonce":2498840646}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":45},{"gid":5276,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6837/3142148286","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6835","nonce":3142148286},{"type":"v1","addr":"172.21.15.118:6837","nonce":3142148286}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":46},{"gid":5632,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6836/3720010453","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6834","nonce":3720010453},{"type":"v1","addr":"172.21.15.118:6836","nonce":3720010453}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":46}],"filesystems":[{"mdsmap":{"epoch":58,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-08-03T20:50:27.688698+0000","modified":"2024-08-03T20:50:50.722012+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":58,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6045},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6045":{"gid":6045,"name":"c","rank":0,"incarnation":55,"state":"up:active","state_seq":5,"addr":"172.21.15.32:6835/2963295201","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":2963295201},{"type":"v1","addr":"172.21.15.32:6835","nonce":2963295201}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[9],"metadata_pool":8,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6045,"qdb_cluster":[6045]},"id":4}]} 2024-08-03T20:50:53.320 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 58 2024-08-03T20:50:53.322 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:53.320+0000 7fa421e00640 1 -- 172.21.15.32:0/304519264 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fa3fc077010 msgr2=0x7fa3fc079520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:50:53.322 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:53.320+0000 7fa421e00640 1 --2- 172.21.15.32:0/304519264 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fa3fc077010 0x7fa3fc079520 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7fa40c009b30 tx=0x7fa40c005f90 comp rx=0 tx=0).stop 2024-08-03T20:50:53.322 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:53.320+0000 7fa421e00640 1 -- 172.21.15.32:0/304519264 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa41c0d3960 msgr2=0x7fa41c0ce990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:50:53.322 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:53.320+0000 7fa421e00640 1 --2- 172.21.15.32:0/304519264 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa41c0d3960 0x7fa41c0ce990 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7fa41400c6b0 tx=0x7fa41400cb70 comp rx=0 tx=0).stop 2024-08-03T20:50:53.322 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:53.320+0000 7fa421e00640 1 -- 172.21.15.32:0/304519264 shutdown_connections 2024-08-03T20:50:53.322 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:53.320+0000 7fa421e00640 1 --2- 172.21.15.32:0/304519264 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fa3fc077010 0x7fa3fc079520 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:53.322 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:53.320+0000 7fa421e00640 1 --2- 172.21.15.32:0/304519264 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa41c0cfc50 0x7fa41c0cdd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:53.322 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:53.320+0000 7fa421e00640 1 --2- 172.21.15.32:0/304519264 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa41c0d3960 0x7fa41c0ce990 unknown :-1 s=CLOSED pgs=634 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:53.322 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:53.320+0000 7fa421e00640 1 --2- 172.21.15.32:0/304519264 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fa41c0b6340 0x7fa41c14c810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:53.323 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:53.320+0000 7fa421e00640 1 -- 172.21.15.32:0/304519264 >> 172.21.15.32:0/304519264 conn(0x7fa41c0d0fe0 msgr2=0x7fa41c06d9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:50:53.323 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:53.320+0000 7fa421e00640 1 -- 172.21.15.32:0/304519264 shutdown_connections 2024-08-03T20:50:53.323 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:53.320+0000 7fa421e00640 1 -- 172.21.15.32:0/304519264 wait complete. 2024-08-03T20:50:53.348 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mds freeze 4:0 true 2024-08-03T20:50:53.578 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:53.577+0000 7fe8dac00640 1 -- 172.21.15.32:0/2481823630 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe8d4155920 msgr2=0x7fe8d40ceb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:50:53.579 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:53.577+0000 7fe8dac00640 1 --2- 172.21.15.32:0/2481823630 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe8d4155920 0x7fe8d40ceb10 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7fe8d4074c20 tx=0x7fe8c402a530 comp rx=0 tx=0).stop 2024-08-03T20:50:53.579 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:53.577+0000 7fe8dac00640 1 -- 172.21.15.32:0/2481823630 shutdown_connections 2024-08-03T20:50:53.579 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:53.577+0000 7fe8dac00640 1 --2- 172.21.15.32:0/2481823630 >> v2:172.21.15.118:3300/0 conn(0x7fe8d40cf3b0 0x7fe8d40cf7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:53.579 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:53.577+0000 7fe8dac00640 1 --2- 172.21.15.32:0/2481823630 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe8d4155920 0x7fe8d40ceb10 unknown :-1 s=CLOSED pgs=636 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:53.579 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:53.577+0000 7fe8dac00640 1 -- 172.21.15.32:0/2481823630 >> 172.21.15.32:0/2481823630 conn(0x7fe8d40b1bd0 msgr2=0x7fe8d40b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:50:53.579 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:53.577+0000 7fe8dac00640 1 -- 172.21.15.32:0/2481823630 shutdown_connections 2024-08-03T20:50:53.579 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:53.577+0000 7fe8dac00640 1 -- 172.21.15.32:0/2481823630 wait complete. 2024-08-03T20:50:53.580 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:53.579+0000 7fe8dac00640 1 Processor -- start 2024-08-03T20:50:53.580 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:53.579+0000 7fe8dac00640 1 -- start start 2024-08-03T20:50:53.583 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:53.580+0000 7fe8dac00640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fe8d40cf3b0 0x7fe8d41422f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:50:53.583 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:53.581+0000 7fe8dac00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe8d4155920 0x7fe8d4144b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:50:53.583 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:53.581+0000 7fe8dac00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fe8d415ba50 0x7fe8d415a300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:50:53.583 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:53.581+0000 7fe8dac00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fe8d40b3260 con 0x7fe8d4155920 2024-08-03T20:50:53.583 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:53.581+0000 7fe8dac00640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7fe8d40b30e0 con 0x7fe8d415ba50 2024-08-03T20:50:53.583 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:53.581+0000 7fe8dac00640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7fe8d40b33e0 con 0x7fe8d40cf3b0 2024-08-03T20:50:53.583 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:53.581+0000 7fe8d3400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe8d4155920 0x7fe8d4144b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:50:53.583 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:53.581+0000 7fe8d3400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe8d4155920 0x7fe8d4144b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:35104/0 (socket says 172.21.15.32:35104) 2024-08-03T20:50:53.584 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:53.581+0000 7fe8d3400640 1 -- 172.21.15.32:0/1657147100 learned_addr learned my addr 172.21.15.32:0/1657147100 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:50:53.584 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:53.581+0000 7fe8d3400640 1 -- 172.21.15.32:0/1657147100 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fe8d40cf3b0 msgr2=0x7fe8d41422f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:50:53.584 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:53.581+0000 7fe8d3400640 1 --2- 172.21.15.32:0/1657147100 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fe8d40cf3b0 0x7fe8d41422f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:53.584 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:53.581+0000 7fe8d3400640 1 -- 172.21.15.32:0/1657147100 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fe8d415ba50 msgr2=0x7fe8d415a300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:50:53.584 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:53.581+0000 7fe8d8c00640 1 --2- 172.21.15.32:0/1657147100 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fe8d415ba50 0x7fe8d415a300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:50:53.584 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:53.581+0000 7fe8d3400640 1 --2- 172.21.15.32:0/1657147100 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fe8d415ba50 0x7fe8d415a300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:53.584 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:53.581+0000 7fe8d3400640 1 -- 172.21.15.32:0/1657147100 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe8d4145ae0 con 0x7fe8d4155920 2024-08-03T20:50:53.584 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:53.581+0000 7fe8d8c00640 1 --2- 172.21.15.32:0/1657147100 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fe8d415ba50 0x7fe8d415a300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-03T20:50:53.584 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:53.582+0000 7fe8d3400640 1 --2- 172.21.15.32:0/1657147100 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe8d4155920 0x7fe8d4144b80 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7fe8c000c9a0 tx=0x7fe8c000ce60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:50:53.585 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:53.582+0000 7fe8cbe00640 1 -- 172.21.15.32:0/1657147100 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe8c0007bb0 con 0x7fe8d4155920 2024-08-03T20:50:53.585 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:53.582+0000 7fe8cbe00640 1 -- 172.21.15.32:0/1657147100 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fe8c0007e40 con 0x7fe8d4155920 2024-08-03T20:50:53.585 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:53.582+0000 7fe8dac00640 1 -- 172.21.15.32:0/1657147100 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe8d4145800 con 0x7fe8d4155920 2024-08-03T20:50:53.585 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:53.582+0000 7fe8cbe00640 1 -- 172.21.15.32:0/1657147100 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe8c000f690 con 0x7fe8d4155920 2024-08-03T20:50:53.585 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:53.582+0000 7fe8dac00640 1 -- 172.21.15.32:0/1657147100 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe8d4142ee0 con 0x7fe8d4155920 2024-08-03T20:50:53.585 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:53.583+0000 7fe8dac00640 1 -- 172.21.15.32:0/1657147100 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe8d40b3900 con 0x7fe8d4155920 2024-08-03T20:50:53.586 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:53.584+0000 7fe8cbe00640 1 -- 172.21.15.32:0/1657147100 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7fe8c000f830 con 0x7fe8d4155920 2024-08-03T20:50:53.587 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:53.585+0000 7fe8cbe00640 1 --2- 172.21.15.32:0/1657147100 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fe8b4077040 0x7fe8b4079550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:50:53.587 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:53.585+0000 7fe8cbe00640 1 -- 172.21.15.32:0/1657147100 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(58..58 src has 1..58) ==== 5795+0+0 (secure 0 0 0) 0x7fe8c00990f0 con 0x7fe8d4155920 2024-08-03T20:50:53.587 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:53.586+0000 7fe8d3e00640 1 --2- 172.21.15.32:0/1657147100 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fe8b4077040 0x7fe8b4079550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:50:53.588 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:53.586+0000 7fe8d3e00640 1 --2- 172.21.15.32:0/1657147100 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fe8b4077040 0x7fe8b4079550 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7fe8c4034000 tx=0x7fe8c40046c0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:50:53.591 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:53.590+0000 7fe8cbe00640 1 -- 172.21.15.32:0/1657147100 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7fe8c0014030 con 0x7fe8d4155920 2024-08-03T20:50:53.873 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:53.871+0000 7fe8dac00640 1 -- 172.21.15.32:0/1657147100 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "mds freeze", "role_or_gid": "4:0", "val": "true"} v 0) -- 0x7fe8d4145590 con 0x7fe8d4155920 2024-08-03T20:50:54.741 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:54.739+0000 7fe8cbe00640 1 -- 172.21.15.32:0/1657147100 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "mds freeze", "role_or_gid": "4:0", "val": "true"}]=0 freezing mds.6045 v59) ==== 112+0+0 (secure 0 0 0) 0x7fe8c0062280 con 0x7fe8d4155920 2024-08-03T20:50:54.741 INFO:teuthology.orchestra.run.smithi032.stderr:freezing mds.6045 2024-08-03T20:50:54.744 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:54.742+0000 7fe8dac00640 1 -- 172.21.15.32:0/1657147100 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fe8b4077040 msgr2=0x7fe8b4079550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:50:54.744 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:54.742+0000 7fe8dac00640 1 --2- 172.21.15.32:0/1657147100 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fe8b4077040 0x7fe8b4079550 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7fe8c4034000 tx=0x7fe8c40046c0 comp rx=0 tx=0).stop 2024-08-03T20:50:54.744 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:54.742+0000 7fe8dac00640 1 -- 172.21.15.32:0/1657147100 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe8d4155920 msgr2=0x7fe8d4144b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:50:54.745 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:54.742+0000 7fe8dac00640 1 --2- 172.21.15.32:0/1657147100 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe8d4155920 0x7fe8d4144b80 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7fe8c000c9a0 tx=0x7fe8c000ce60 comp rx=0 tx=0).stop 2024-08-03T20:50:54.745 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:54.743+0000 7fe8dac00640 1 -- 172.21.15.32:0/1657147100 shutdown_connections 2024-08-03T20:50:54.745 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:54.743+0000 7fe8dac00640 1 --2- 172.21.15.32:0/1657147100 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fe8b4077040 0x7fe8b4079550 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:54.745 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:54.743+0000 7fe8dac00640 1 --2- 172.21.15.32:0/1657147100 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fe8d415ba50 0x7fe8d415a300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:54.745 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:54.743+0000 7fe8dac00640 1 --2- 172.21.15.32:0/1657147100 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe8d4155920 0x7fe8d4144b80 unknown :-1 s=CLOSED pgs=637 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:54.745 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:54.743+0000 7fe8dac00640 1 --2- 172.21.15.32:0/1657147100 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fe8d40cf3b0 0x7fe8d41422f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:54.745 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:54.743+0000 7fe8dac00640 1 -- 172.21.15.32:0/1657147100 >> 172.21.15.32:0/1657147100 conn(0x7fe8d40b1bd0 msgr2=0x7fe8d415caa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:50:54.745 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:54.743+0000 7fe8dac00640 1 -- 172.21.15.32:0/1657147100 shutdown_connections 2024-08-03T20:50:54.745 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:54.743+0000 7fe8dac00640 1 -- 172.21.15.32:0/1657147100 wait complete. 2024-08-03T20:50:54.771 DEBUG:teuthology.orchestra.run.smithi032:> sudo daemon-helper kill ceph --cluster ceph -w 2024-08-03T20:50:54.775 DEBUG:teuthology.orchestra.run.smithi032:> timeout 30 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --connect-timeout=60 tell mds.4:0 flush journal 2024-08-03T20:50:54.779 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mon.a.asok --format=json config get mds_beacon_grace 2024-08-03T20:50:55.017 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:55.015+0000 7f3ab6000640 1 -- 172.21.15.32:0/530945819 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3ab00b0160 msgr2=0x7f3ab0074480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:50:55.017 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:55.015+0000 7f3ab6000640 1 --2- 172.21.15.32:0/530945819 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3ab00b0160 0x7f3ab0074480 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7f3ab00b4a20 tx=0x7f3aa002a680 comp rx=0 tx=0).stop 2024-08-03T20:50:55.017 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:55.015+0000 7f3ab6000640 1 -- 172.21.15.32:0/530945819 shutdown_connections 2024-08-03T20:50:55.017 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:55.015+0000 7f3ab6000640 1 --2- 172.21.15.32:0/530945819 >> v2:172.21.15.118:3300/0 conn(0x7f3ab006c9e0 0x7f3ab006cdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:55.017 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:55.015+0000 7f3ab6000640 1 --2- 172.21.15.32:0/530945819 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3ab00b0160 0x7f3ab0074480 unknown :-1 s=CLOSED pgs=638 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:55.017 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:55.015+0000 7f3ab6000640 1 -- 172.21.15.32:0/530945819 >> 172.21.15.32:0/530945819 conn(0x7f3ab00b32f0 msgr2=0x7f3ab00d1050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:50:55.017 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:55.015+0000 7f3ab6000640 1 -- 172.21.15.32:0/530945819 shutdown_connections 2024-08-03T20:50:55.017 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:55.015+0000 7f3ab6000640 1 -- 172.21.15.32:0/530945819 wait complete. 2024-08-03T20:50:55.019 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:55.017+0000 7f3ab6000640 1 Processor -- start 2024-08-03T20:50:55.019 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:55.017+0000 7f3ab6000640 1 -- start start 2024-08-03T20:50:55.021 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:55.019+0000 7f3ab6000640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f3ab006c9e0 0x7f3ab014ebe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:50:55.021 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:55.019+0000 7f3ab6000640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f3ab00b0160 0x7f3ab014be40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:50:55.021 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:55.019+0000 7f3ab6000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3ab0158c60 0x7f3ab01f9260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:50:55.021 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:55.019+0000 7f3ab6000640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f3ab0162750 con 0x7f3ab0158c60 2024-08-03T20:50:55.021 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:55.019+0000 7f3ab6000640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f3ab01625d0 con 0x7f3ab006c9e0 2024-08-03T20:50:55.021 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:55.019+0000 7f3ab6000640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7f3ab01628d0 con 0x7f3ab00b0160 2024-08-03T20:50:55.021 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:55.019+0000 7f3aafe00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3ab0158c60 0x7f3ab01f9260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:50:55.021 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:55.019+0000 7f3aafe00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3ab0158c60 0x7f3ab01f9260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:35126/0 (socket says 172.21.15.32:35126) 2024-08-03T20:50:55.021 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:55.019+0000 7f3aafe00640 1 -- 172.21.15.32:0/4032964580 learned_addr learned my addr 172.21.15.32:0/4032964580 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:50:55.021 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:55.019+0000 7f3aaea00640 1 --2- 172.21.15.32:0/4032964580 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f3ab00b0160 0x7f3ab014be40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:50:55.021 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:55.019+0000 7f3aaf400640 1 --2- 172.21.15.32:0/4032964580 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f3ab006c9e0 0x7f3ab014ebe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:50:55.022 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:55.020+0000 7f3aafe00640 1 -- 172.21.15.32:0/4032964580 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f3ab00b0160 msgr2=0x7f3ab014be40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:50:55.023 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:55.020+0000 7f3aafe00640 1 --2- 172.21.15.32:0/4032964580 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f3ab00b0160 0x7f3ab014be40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:55.023 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:55.020+0000 7f3aafe00640 1 -- 172.21.15.32:0/4032964580 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f3ab006c9e0 msgr2=0x7f3ab014ebe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:50:55.023 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:55.020+0000 7f3aafe00640 1 --2- 172.21.15.32:0/4032964580 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f3ab006c9e0 0x7f3ab014ebe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:55.023 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:55.020+0000 7f3aafe00640 1 -- 172.21.15.32:0/4032964580 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3ab014de20 con 0x7f3ab0158c60 2024-08-03T20:50:55.023 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:55.020+0000 7f3aafe00640 1 --2- 172.21.15.32:0/4032964580 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3ab0158c60 0x7f3ab01f9260 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7f3aa800e990 tx=0x7f3aa800ee50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:50:55.023 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:55.021+0000 7f3aa7400640 1 -- 172.21.15.32:0/4032964580 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3aa8004d20 con 0x7f3ab0158c60 2024-08-03T20:50:55.023 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:55.021+0000 7f3aa7400640 1 -- 172.21.15.32:0/4032964580 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f3aa800caa0 con 0x7f3ab0158c60 2024-08-03T20:50:55.023 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:55.021+0000 7f3ab6000640 1 -- 172.21.15.32:0/4032964580 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3ab014e100 con 0x7f3ab0158c60 2024-08-03T20:50:55.023 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:55.021+0000 7f3aa7400640 1 -- 172.21.15.32:0/4032964580 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3aa8005020 con 0x7f3ab0158c60 2024-08-03T20:50:55.023 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:55.021+0000 7f3ab6000640 1 -- 172.21.15.32:0/4032964580 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3ab014e6c0 con 0x7f3ab0158c60 2024-08-03T20:50:55.025 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:55.023+0000 7f3aa7400640 1 -- 172.21.15.32:0/4032964580 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f3aa80051c0 con 0x7f3ab0158c60 2024-08-03T20:50:55.025 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:55.024+0000 7f3aa7400640 1 --2- 172.21.15.32:0/4032964580 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f3a90077010 0x7f3a90079520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:50:55.026 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:55.025+0000 7f3aa7400640 1 -- 172.21.15.32:0/4032964580 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(58..58 src has 1..58) ==== 5795+0+0 (secure 0 0 0) 0x7f3aa8099280 con 0x7f3ab0158c60 2024-08-03T20:50:55.026 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:55.025+0000 7f3abd19c740 1 Processor -- start 2024-08-03T20:50:55.026 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:55.025+0000 7f3aaf400640 1 --2- 172.21.15.32:0/4032964580 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f3a90077010 0x7f3a90079520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:50:55.027 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:55.025+0000 7f3abd19c740 1 -- start start 2024-08-03T20:50:55.027 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:55.026+0000 7f3aaf400640 1 --2- 172.21.15.32:0/4032964580 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f3a90077010 0x7f3a90079520 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f3aa0033000 tx=0x7f3aa0004730 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:50:55.028 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:55.026+0000 7f3abd19c740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ca2c2695c0 0x55ca2c639900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:50:55.028 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:55.026+0000 7f3abd19c740 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55ca2c240700 0x55ca2c639e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:50:55.028 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:55.026+0000 7f3abd19c740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55ca2c2488e0 0x55ca2c656d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:50:55.028 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:55.026+0000 7f3abd19c740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55ca2c582410 con 0x55ca2c2695c0 2024-08-03T20:50:55.028 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:55.026+0000 7f3abd19c740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x55ca2c571910 con 0x55ca2c2488e0 2024-08-03T20:50:55.028 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:55.026+0000 7f3abd19c740 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x55ca2c575950 con 0x55ca2c240700 2024-08-03T20:50:55.029 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:55.027+0000 7f3aaf400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ca2c2695c0 0x55ca2c639900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:50:55.029 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:55.027+0000 7f3aaf400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ca2c2695c0 0x55ca2c639900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:35134/0 (socket says 172.21.15.32:35134) 2024-08-03T20:50:55.029 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:55.027+0000 7f3aaf400640 1 -- 172.21.15.32:0/3658537367 learned_addr learned my addr 172.21.15.32:0/3658537367 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:50:55.029 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:55.027+0000 7f3aaea00640 1 --2- 172.21.15.32:0/3658537367 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55ca2c240700 0x55ca2c639e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:50:55.030 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:55.027+0000 7f3aafe00640 1 --2- 172.21.15.32:0/3658537367 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55ca2c2488e0 0x55ca2c656d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:50:55.030 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:55.027+0000 7f3aaf400640 1 -- 172.21.15.32:0/3658537367 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55ca2c240700 msgr2=0x55ca2c639e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:50:55.030 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:55.027+0000 7f3aaf400640 1 --2- 172.21.15.32:0/3658537367 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55ca2c240700 0x55ca2c639e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:55.030 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:55.027+0000 7f3aaf400640 1 -- 172.21.15.32:0/3658537367 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55ca2c2488e0 msgr2=0x55ca2c656d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:50:55.030 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:55.027+0000 7f3aaf400640 1 --2- 172.21.15.32:0/3658537367 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55ca2c2488e0 0x55ca2c656d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:55.030 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:55.027+0000 7f3aaf400640 1 -- 172.21.15.32:0/3658537367 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ca2c34da30 con 0x55ca2c2695c0 2024-08-03T20:50:55.030 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:55.028+0000 7f3aaf400640 1 --2- 172.21.15.32:0/3658537367 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ca2c2695c0 0x55ca2c639900 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7f3aa0000c00 tx=0x7f3aa003d000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=26a38a455b07f1e0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:50:55.031 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:55.028+0000 7f3a9b400640 1 -- 172.21.15.32:0/3658537367 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3aa0035040 con 0x55ca2c2695c0 2024-08-03T20:50:55.031 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:55.028+0000 7f3a9b400640 1 -- 172.21.15.32:0/3658537367 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f3aa004e070 con 0x55ca2c2695c0 2024-08-03T20:50:55.031 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:55.028+0000 7f3a9b400640 1 -- 172.21.15.32:0/3658537367 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3aa0003d60 con 0x55ca2c2695c0 2024-08-03T20:50:55.031 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:55.029+0000 7f3abd19c740 1 -- 172.21.15.32:0/3658537367 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ca2c2695c0 msgr2=0x55ca2c639900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:50:55.031 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:55.029+0000 7f3abd19c740 1 --2- 172.21.15.32:0/3658537367 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ca2c2695c0 0x55ca2c639900 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7f3aa0000c00 tx=0x7f3aa003d000 comp rx=0 tx=0).stop 2024-08-03T20:50:55.031 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:55.030+0000 7f3abd19c740 1 -- 172.21.15.32:0/3658537367 shutdown_connections 2024-08-03T20:50:55.032 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:55.030+0000 7f3abd19c740 1 --2- 172.21.15.32:0/3658537367 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55ca2c2488e0 0x55ca2c656d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:55.032 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:55.030+0000 7f3abd19c740 1 --2- 172.21.15.32:0/3658537367 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55ca2c240700 0x55ca2c639e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:55.032 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:55.030+0000 7f3abd19c740 1 --2- 172.21.15.32:0/3658537367 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ca2c2695c0 0x55ca2c639900 unknown :-1 s=CLOSED pgs=640 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:55.032 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:55.030+0000 7f3abd19c740 1 -- 172.21.15.32:0/3658537367 >> 172.21.15.32:0/3658537367 conn(0x55ca2c25d710 msgr2=0x55ca2c60fbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:50:55.032 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:55.030+0000 7f3abd19c740 1 -- 172.21.15.32:0/3658537367 shutdown_connections 2024-08-03T20:50:55.032 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:55.030+0000 7f3abd19c740 1 -- 172.21.15.32:0/3658537367 wait complete. 2024-08-03T20:50:55.032 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:55.031+0000 7f3abd19c740 1 -- start start 2024-08-03T20:50:55.032 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:55.031+0000 7f3abd19c740 1 Processor -- start 2024-08-03T20:50:55.034 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:55.033+0000 7f3abd19c740 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55ca2c240700 0x55ca2c663940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:50:55.035 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:55.033+0000 7f3abd19c740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ca2c2488e0 0x55ca2c64d280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:50:55.035 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:55.033+0000 7f3abd19c740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55ca2c2695c0 0x55ca2c628100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:50:55.035 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:55.033+0000 7f3abd19c740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55ca2c4aa010 con 0x55ca2c2488e0 2024-08-03T20:50:55.035 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:55.033+0000 7f3abd19c740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x55ca2c50bf60 con 0x55ca2c2695c0 2024-08-03T20:50:55.035 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:55.033+0000 7f3abd19c740 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x55ca2c63a380 con 0x55ca2c240700 2024-08-03T20:50:55.035 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:55.033+0000 7f3abd19c740 10 client.0 _ll_register_callbacks cb 0x55ca2c280e10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-03T20:50:55.035 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:55.033+0000 7f3aaea00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ca2c2488e0 0x55ca2c64d280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:50:55.035 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:55.033+0000 7f3aaea00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ca2c2488e0 0x55ca2c64d280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:35136/0 (socket says 172.21.15.32:35136) 2024-08-03T20:50:55.035 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:55.033+0000 7f3aaea00640 1 -- 172.21.15.32:0/3576346838 learned_addr learned my addr 172.21.15.32:0/3576346838 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:50:55.035 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:55.033+0000 7f3aaf400640 1 --2- 172.21.15.32:0/3576346838 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55ca2c240700 0x55ca2c663940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:50:55.035 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:55.033+0000 7f3aafe00640 1 --2- 172.21.15.32:0/3576346838 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55ca2c2695c0 0x55ca2c628100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:50:55.035 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:55.033+0000 7f3aaea00640 1 -- 172.21.15.32:0/3576346838 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55ca2c240700 msgr2=0x55ca2c663940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:50:55.035 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:55.033+0000 7f3aaea00640 1 --2- 172.21.15.32:0/3576346838 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55ca2c240700 0x55ca2c663940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:55.038 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:55.033+0000 7f3aaea00640 1 -- 172.21.15.32:0/3576346838 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55ca2c2695c0 msgr2=0x55ca2c628100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:50:55.038 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:55.033+0000 7f3aaea00640 1 --2- 172.21.15.32:0/3576346838 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55ca2c2695c0 0x55ca2c628100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:55.038 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:55.033+0000 7f3aaea00640 1 -- 172.21.15.32:0/3576346838 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ca2c2d4ba0 con 0x55ca2c2488e0 2024-08-03T20:50:55.038 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:55.034+0000 7f3aaea00640 1 --2- 172.21.15.32:0/3576346838 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ca2c2488e0 0x55ca2c64d280 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7f3a9c001e20 tx=0x7f3a9c012750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=60052d39e7a529db server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:50:55.038 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:55.034+0000 7f3a8fe00640 10 client.0 ms_handle_connect on v2:172.21.15.32:3300/0 2024-08-03T20:50:55.038 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:55.034+0000 7f3a8fe00640 1 -- 172.21.15.32:0/3576346838 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a9c005320 con 0x55ca2c2488e0 2024-08-03T20:50:55.038 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:55.034+0000 7f3a8fe00640 1 -- 172.21.15.32:0/3576346838 <== mon.0 v2:172.21.15.32:3300/0 2 ==== osd_map(58..58 src has 1..58) ==== 5795+0+0 (secure 0 0 0) 0x7f3a9c007050 con 0x55ca2c2488e0 2024-08-03T20:50:55.038 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:55.034+0000 7f3abd19c740 1 -- 172.21.15.32:0/3576346838 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3aa004e070 con 0x55ca2c2488e0 2024-08-03T20:50:55.038 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:55.034+0000 7f3a8fe00640 1 -- 172.21.15.32:0/3576346838 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_get_version_reply(handle=1 version=59) ==== 24+0+0 (secure 0 0 0) 0x7f3a9c007250 con 0x55ca2c2488e0 2024-08-03T20:50:55.038 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:55.034+0000 7f3abd19c740 10 client.6126 fetch_fsmap learned FSMap version 59 2024-08-03T20:50:55.038 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:55.034+0000 7f3abd19c740 1 -- 172.21.15.32:0/3576346838 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,fsmap=59,monmap=2+}) -- 0x55ca2c21a4a0 con 0x55ca2c2488e0 2024-08-03T20:50:55.038 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:55.035+0000 7f3a8fe00640 1 -- 172.21.15.32:0/3576346838 <== mon.0 v2:172.21.15.32:3300/0 4 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f3aa004e070 con 0x55ca2c2488e0 2024-08-03T20:50:55.039 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:55.035+0000 7f3a8fe00640 1 -- 172.21.15.32:0/3576346838 <== mon.0 v2:172.21.15.32:3300/0 5 ==== fsmap(e 59) ==== 3502+0+0 (secure 0 0 0) 0x7f3a9c0073f0 con 0x55ca2c2488e0 2024-08-03T20:50:55.039 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:55.035+0000 7f3abd19c740 10 client.6126 fetch_fsmap finished waiting for FSMap version 59 2024-08-03T20:50:55.039 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:55.035+0000 7f3abd19c740 10 client.6126 resolve_mds: resolved 4:0 to role '4:0' aka daemon mds.c 2024-08-03T20:50:55.039 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:55.035+0000 7f3abd19c740 20 client.6126 populate_metadata read hostname 'smithi032' 2024-08-03T20:50:55.039 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:55.035+0000 7f3abd19c740 1 --2- 172.21.15.32:0/3576346838 >> [v2:172.21.15.32:6834/2963295201,v1:172.21.15.32:6835/2963295201] conn(0x55ca2c228190 0x55ca2c657f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:50:55.039 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:55.035+0000 7f3abd19c740 4 client.6126 mds_command: new command op to 6045 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-08-03T20:50:55.039 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:55.035+0000 7f3abd19c740 1 -- 172.21.15.32:0/3576346838 --> [v2:172.21.15.32:6834/2963295201,v1:172.21.15.32:6835/2963295201] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ca2c584360 con 0x55ca2c228190 2024-08-03T20:50:55.039 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:55.035+0000 7f3aaf400640 1 --2- 172.21.15.32:0/3576346838 >> [v2:172.21.15.32:6834/2963295201,v1:172.21.15.32:6835/2963295201] conn(0x55ca2c228190 0x55ca2c657f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:50:55.039 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:55.036+0000 7f3aaf400640 1 --2- 172.21.15.32:0/3576346838 >> [v2:172.21.15.32:6834/2963295201,v1:172.21.15.32:6835/2963295201] conn(0x55ca2c228190 0x55ca2c657f20 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e19d7a9dcd47c396 server_cookie=a95e009a8f7df714 in_seq=0 out_seq=0 2024-08-03T20:50:55.039 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:55.036+0000 7f3a8fe00640 10 client.6126 ms_handle_connect on v2:172.21.15.32:6834/2963295201 2024-08-03T20:50:55.040 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:55.037+0000 7f3a8fe00640 1 -- 172.21.15.32:0/3576346838 <== mds.0 v2:172.21.15.32:6834/2963295201 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55ca2c584360 con 0x55ca2c228190 2024-08-03T20:50:55.040 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:55.037+0000 7f3a8fe00640 10 client.6126 handle_command_reply: tid=0 2024-08-03T20:50:55.040 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:55.037+0000 7f3abd19c740 1 client.6126 shutdown 2024-08-03T20:50:55.040 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:55.037+0000 7f3abd19c740 1 -- 172.21.15.32:0/3576346838 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ca2c2488e0 msgr2=0x55ca2c64d280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:50:55.041 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:55.037+0000 7f3abd19c740 1 --2- 172.21.15.32:0/3576346838 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ca2c2488e0 0x55ca2c64d280 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7f3a9c001e20 tx=0x7f3a9c012750 comp rx=0 tx=0).stop 2024-08-03T20:50:55.041 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:55.037+0000 7f3abd19c740 1 -- 172.21.15.32:0/3576346838 shutdown_connections 2024-08-03T20:50:55.041 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:55.037+0000 7f3abd19c740 1 --2- 172.21.15.32:0/3576346838 >> [v2:172.21.15.32:6834/2963295201,v1:172.21.15.32:6835/2963295201] conn(0x55ca2c228190 0x55ca2c657f20 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:55.041 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:55.037+0000 7f3abd19c740 1 --2- 172.21.15.32:0/3576346838 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55ca2c2695c0 0x55ca2c628100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:55.041 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:55.037+0000 7f3abd19c740 1 --2- 172.21.15.32:0/3576346838 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ca2c2488e0 0x55ca2c64d280 unknown :-1 s=CLOSED pgs=641 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:55.041 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:55.037+0000 7f3abd19c740 1 --2- 172.21.15.32:0/3576346838 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55ca2c240700 0x55ca2c663940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:55.041 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:55.037+0000 7f3abd19c740 1 -- 172.21.15.32:0/3576346838 >> 172.21.15.32:0/3576346838 conn(0x55ca2c25d710 msgr2=0x55ca2c656310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:50:55.041 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:55.037+0000 7f3a8fe00640 0 client.6126 ms_handle_reset on v2:172.21.15.32:6834/2963295201 2024-08-03T20:50:55.041 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:55.038+0000 7f3abd19c740 1 -- 172.21.15.32:0/3576346838 shutdown_connections 2024-08-03T20:50:55.041 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:55.038+0000 7f3abd19c740 1 -- 172.21.15.32:0/3576346838 wait complete. 2024-08-03T20:50:55.041 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:55.038+0000 7f3abd19c740 20 client.6126 trim_cache size 0 max 16384 2024-08-03T20:50:55.047 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:55.043+0000 7fb22be00640 1 -- 172.21.15.32:0/4217802162 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb22c161db0 msgr2=0x7fb22c0d2a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:50:55.047 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:55.043+0000 7fb22be00640 1 --2- 172.21.15.32:0/4217802162 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb22c161db0 0x7fb22c0d2a30 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7fb22c0b9d20 tx=0x7fb21c02a580 comp rx=0 tx=0).stop 2024-08-03T20:50:55.047 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:55.043+0000 7fb22be00640 1 -- 172.21.15.32:0/4217802162 shutdown_connections 2024-08-03T20:50:55.047 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:55.043+0000 7fb22be00640 1 --2- 172.21.15.32:0/4217802162 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fb22c0cf770 0x7fb22c0b0d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:55.047 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:55.043+0000 7fb22be00640 1 --2- 172.21.15.32:0/4217802162 >> v2:172.21.15.118:3300/0 conn(0x7fb22c0cee20 0x7fb22c0cf230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:55.047 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:55.043+0000 7fb22be00640 1 --2- 172.21.15.32:0/4217802162 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb22c161db0 0x7fb22c0d2a30 unknown :-1 s=CLOSED pgs=642 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:55.048 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:55.043+0000 7fb22be00640 1 -- 172.21.15.32:0/4217802162 >> 172.21.15.32:0/4217802162 conn(0x7fb22c0d41f0 msgr2=0x7fb22c0d4600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:50:55.048 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:55.043+0000 7fb22be00640 1 -- 172.21.15.32:0/4217802162 shutdown_connections 2024-08-03T20:50:55.048 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:55.043+0000 7fb22be00640 1 -- 172.21.15.32:0/4217802162 wait complete. 2024-08-03T20:50:55.048 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:55.045+0000 7fb22be00640 1 Processor -- start 2024-08-03T20:50:55.048 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:55.046+0000 7fb22be00640 1 -- start start 2024-08-03T20:50:55.049 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:55.047+0000 7fb22be00640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fb22c0cee20 0x7fb22c0b6500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:50:55.049 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:55.047+0000 7fb22be00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb22c0cf770 0x7fb22c0cd510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:50:55.050 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:55.047+0000 7fb22be00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb22c161db0 0x7fb22c0cda50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:50:55.050 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:55.047+0000 7fb22be00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fb22c0b16e0 con 0x7fb22c0cf770 2024-08-03T20:50:55.050 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:55.047+0000 7fb22be00640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7fb22c0b1560 con 0x7fb22c161db0 2024-08-03T20:50:55.050 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:55.047+0000 7fb22be00640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7fb22c0b1860 con 0x7fb22c0cee20 2024-08-03T20:50:55.050 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:55.047+0000 7fb22a000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb22c0cf770 0x7fb22c0cd510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:50:55.050 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:55.048+0000 7fb22a000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb22c0cf770 0x7fb22c0cd510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:35152/0 (socket says 172.21.15.32:35152) 2024-08-03T20:50:55.050 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:55.048+0000 7fb22a000640 1 -- 172.21.15.32:0/2804832187 learned_addr learned my addr 172.21.15.32:0/2804832187 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:50:55.050 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:55.048+0000 7fb22aa00640 1 --2- 172.21.15.32:0/2804832187 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fb22c0cee20 0x7fb22c0b6500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:50:55.050 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:55.048+0000 7fb22b400640 1 --2- 172.21.15.32:0/2804832187 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb22c161db0 0x7fb22c0cda50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:50:55.052 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:55.048+0000 7fb22a000640 1 -- 172.21.15.32:0/2804832187 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fb22c0cee20 msgr2=0x7fb22c0b6500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:50:55.052 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:55.048+0000 7fb22a000640 1 --2- 172.21.15.32:0/2804832187 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fb22c0cee20 0x7fb22c0b6500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:55.053 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:55.048+0000 7fb22a000640 1 -- 172.21.15.32:0/2804832187 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb22c161db0 msgr2=0x7fb22c0cda50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:50:55.053 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:55.048+0000 7fb22a000640 1 --2- 172.21.15.32:0/2804832187 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb22c161db0 0x7fb22c0cda50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:55.053 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:55.048+0000 7fb22a000640 1 -- 172.21.15.32:0/2804832187 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb22c0b8150 con 0x7fb22c0cf770 2024-08-03T20:50:55.053 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:55.048+0000 7fb22b400640 1 --2- 172.21.15.32:0/2804832187 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb22c161db0 0x7fb22c0cda50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-03T20:50:55.053 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:55.049+0000 7fb22a000640 1 --2- 172.21.15.32:0/2804832187 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb22c0cf770 0x7fb22c0cd510 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7fb21800c970 tx=0x7fb21800ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:50:55.053 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:55.049+0000 7fb222a00640 1 -- 172.21.15.32:0/2804832187 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb218007b50 con 0x7fb22c0cf770 2024-08-03T20:50:55.053 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:55.049+0000 7fb222a00640 1 -- 172.21.15.32:0/2804832187 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fb218007de0 con 0x7fb22c0cf770 2024-08-03T20:50:55.053 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:55.049+0000 7fb222a00640 1 -- 172.21.15.32:0/2804832187 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb21800f6b0 con 0x7fb22c0cf770 2024-08-03T20:50:55.053 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:55.049+0000 7fb22be00640 1 -- 172.21.15.32:0/2804832187 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb22c0cca30 con 0x7fb22c0cf770 2024-08-03T20:50:55.053 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:55.049+0000 7fb22be00640 1 -- 172.21.15.32:0/2804832187 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb22c0cc700 con 0x7fb22c0cf770 2024-08-03T20:50:55.053 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:55.050+0000 7fb22be00640 1 -- 172.21.15.32:0/2804832187 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "status"} v 0) -- 0x7fb22c076d80 con 0x7fb22c0cf770 2024-08-03T20:50:55.054 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:55.053+0000 7fb222a00640 1 -- 172.21.15.32:0/2804832187 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7fb21800f850 con 0x7fb22c0cf770 2024-08-03T20:50:55.058 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:55.054+0000 7fb222a00640 1 --2- 172.21.15.32:0/2804832187 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fb20c077010 0x7fb20c079520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:50:55.059 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:55.054+0000 7fb222a00640 1 -- 172.21.15.32:0/2804832187 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(58..58 src has 1..58) ==== 5795+0+0 (secure 0 0 0) 0x7fb218098ed0 con 0x7fb22c0cf770 2024-08-03T20:50:55.059 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:55.054+0000 7fb222a00640 1 -- 172.21.15.32:0/2804832187 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "status"}]=0 v0) ==== 54+0+513 (secure 0 0 0) 0x7fb22c076d80 con 0x7fb22c0cf770 2024-08-03T20:50:55.059 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:55.054+0000 7fb22aa00640 1 --2- 172.21.15.32:0/2804832187 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fb20c077010 0x7fb20c079520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:50:55.059 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:55.055+0000 7fb22be00640 1 -- 172.21.15.32:0/2804832187 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({log-info=0+}) -- 0x7fb22c0b0d20 con 0x7fb22c0cf770 2024-08-03T20:50:55.059 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:55.055+0000 7fb222a00640 1 -- 172.21.15.32:0/2804832187 <== mon.0 v2:172.21.15.32:3300/0 7 ==== log(1 entries from seq 676 at 2024-08-03T20:50:53.873151+0000) ==== 308+0+0 (secure 0 0 0) 0x7fb218097410 con 0x7fb22c0cf770 2024-08-03T20:50:55.059 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:55.055+0000 7fb22aa00640 1 --2- 172.21.15.32:0/2804832187 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fb20c077010 0x7fb20c079520 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7fb21c034000 tx=0x7fb21c004730 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:50:55.060 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:55.058+0000 7f3abd19c740 1 Processor -- start 2024-08-03T20:50:55.060 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:55.058+0000 7f3abd19c740 1 -- start start 2024-08-03T20:50:55.065 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:55.059+0000 7f3abd19c740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55ca2c240700 0x55ca2c60fa50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:50:55.065 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:55.059+0000 7f3abd19c740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ca2c2488e0 0x55ca2c655730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:50:55.065 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:55.059+0000 7f3abd19c740 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55ca2c2695c0 0x55ca2c64a720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:50:55.065 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:55.059+0000 7f3abd19c740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55ca2c631de0 con 0x55ca2c2488e0 2024-08-03T20:50:55.065 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:55.059+0000 7f3abd19c740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f3ab0201f30 con 0x55ca2c240700 2024-08-03T20:50:55.065 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:55.059+0000 7f3abd19c740 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7f3ab0200fb0 con 0x55ca2c2695c0 2024-08-03T20:50:55.065 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:55.060+0000 7f3aaea00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ca2c2488e0 0x55ca2c655730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:50:55.065 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:55.060+0000 7f3aaea00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ca2c2488e0 0x55ca2c655730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:35162/0 (socket says 172.21.15.32:35162) 2024-08-03T20:50:55.066 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:55.060+0000 7f3aaea00640 1 -- 172.21.15.32:0/1988806537 learned_addr learned my addr 172.21.15.32:0/1988806537 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:50:55.066 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:55.060+0000 7f3aafe00640 1 --2- 172.21.15.32:0/1988806537 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55ca2c2695c0 0x55ca2c64a720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:50:55.067 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:55.060+0000 7f3aaf400640 1 --2- 172.21.15.32:0/1988806537 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55ca2c240700 0x55ca2c60fa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:50:55.067 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:55.060+0000 7f3aaea00640 1 -- 172.21.15.32:0/1988806537 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55ca2c2695c0 msgr2=0x55ca2c64a720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:50:55.067 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:55.060+0000 7f3aaea00640 1 --2- 172.21.15.32:0/1988806537 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55ca2c2695c0 0x55ca2c64a720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:55.067 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:55.060+0000 7f3aaea00640 1 -- 172.21.15.32:0/1988806537 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55ca2c240700 msgr2=0x55ca2c60fa50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:50:55.067 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:55.060+0000 7f3aaea00640 1 --2- 172.21.15.32:0/1988806537 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55ca2c240700 0x55ca2c60fa50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:55.067 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:55.060+0000 7f3aaea00640 1 -- 172.21.15.32:0/1988806537 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ca2c64e470 con 0x55ca2c2488e0 2024-08-03T20:50:55.067 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:55.061+0000 7f3aaea00640 1 --2- 172.21.15.32:0/1988806537 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ca2c2488e0 0x55ca2c655730 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7f3a9c012f80 tx=0x7f3a9c012fb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=20feb5216050e3ae server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:50:55.067 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:55.061+0000 7f3a9aa00640 1 -- 172.21.15.32:0/1988806537 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a9c02e070 con 0x55ca2c2488e0 2024-08-03T20:50:55.067 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:55.061+0000 7f3a9aa00640 1 -- 172.21.15.32:0/1988806537 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f3a9c005320 con 0x55ca2c2488e0 2024-08-03T20:50:55.067 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:55.061+0000 7f3a9aa00640 1 -- 172.21.15.32:0/1988806537 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a9c006c10 con 0x55ca2c2488e0 2024-08-03T20:50:55.067 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:55.062+0000 7f3abd19c740 1 -- 172.21.15.32:0/1988806537 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ca2c2488e0 msgr2=0x55ca2c655730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:50:55.067 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:55.062+0000 7f3abd19c740 1 --2- 172.21.15.32:0/1988806537 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ca2c2488e0 0x55ca2c655730 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7f3a9c012f80 tx=0x7f3a9c012fb0 comp rx=0 tx=0).stop 2024-08-03T20:50:55.067 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:55.062+0000 7f3abd19c740 1 -- 172.21.15.32:0/1988806537 shutdown_connections 2024-08-03T20:50:55.068 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:55.062+0000 7f3abd19c740 1 --2- 172.21.15.32:0/1988806537 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55ca2c2695c0 0x55ca2c64a720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:55.068 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:55.062+0000 7f3abd19c740 1 --2- 172.21.15.32:0/1988806537 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ca2c2488e0 0x55ca2c655730 unknown :-1 s=CLOSED pgs=644 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:55.068 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:55.062+0000 7f3abd19c740 1 --2- 172.21.15.32:0/1988806537 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55ca2c240700 0x55ca2c60fa50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:55.068 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:55.062+0000 7f3abd19c740 1 -- 172.21.15.32:0/1988806537 >> 172.21.15.32:0/1988806537 conn(0x55ca2c25d710 msgr2=0x55ca2c638dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:50:55.068 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:55.062+0000 7f3abd19c740 1 -- 172.21.15.32:0/1988806537 shutdown_connections 2024-08-03T20:50:55.068 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:55.062+0000 7f3abd19c740 1 -- 172.21.15.32:0/1988806537 wait complete. 2024-08-03T20:50:55.068 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:55.063+0000 7f3abd19c740 1 -- start start 2024-08-03T20:50:55.068 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:55.063+0000 7f3abd19c740 1 Processor -- start 2024-08-03T20:50:55.071 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:55.064+0000 7f3abd19c740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55ca2c240700 0x55ca2c639ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:50:55.071 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:55.064+0000 7f3abd19c740 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55ca2c2488e0 0x55ca2c639ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:50:55.071 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:55.064+0000 7f3abd19c740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ca2c2695c0 0x55ca2c676590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:50:55.071 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:55.064+0000 7f3abd19c740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f3ab014d460 con 0x55ca2c2695c0 2024-08-03T20:50:55.071 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:55.065+0000 7f3abd19c740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f3ab014ce10 con 0x55ca2c240700 2024-08-03T20:50:55.071 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:55.065+0000 7f3abd19c740 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x55ca2c6110c0 con 0x55ca2c2488e0 2024-08-03T20:50:55.071 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:55.065+0000 7f3abd19c740 10 client.0 _ll_register_callbacks cb 0x7f3a9c0012d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-03T20:50:55.071 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:55.065+0000 7f3aafe00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ca2c2695c0 0x55ca2c676590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:50:55.071 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:55.065+0000 7f3aafe00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ca2c2695c0 0x55ca2c676590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:35164/0 (socket says 172.21.15.32:35164) 2024-08-03T20:50:55.071 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:55.065+0000 7f3aafe00640 1 -- 172.21.15.32:0/294378650 learned_addr learned my addr 172.21.15.32:0/294378650 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:50:55.072 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:55.065+0000 7f3aaea00640 1 --2- 172.21.15.32:0/294378650 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55ca2c2488e0 0x55ca2c639ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:50:55.072 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:55.065+0000 7f3aaf400640 1 --2- 172.21.15.32:0/294378650 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55ca2c240700 0x55ca2c639ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:50:55.072 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:55.065+0000 7f3aafe00640 1 -- 172.21.15.32:0/294378650 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55ca2c2488e0 msgr2=0x55ca2c639ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:50:55.072 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:55.065+0000 7f3aafe00640 1 --2- 172.21.15.32:0/294378650 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55ca2c2488e0 0x55ca2c639ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:55.072 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:55.065+0000 7f3aafe00640 1 -- 172.21.15.32:0/294378650 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55ca2c240700 msgr2=0x55ca2c639ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:50:55.072 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:55.065+0000 7f3aafe00640 1 --2- 172.21.15.32:0/294378650 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55ca2c240700 0x55ca2c639ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:55.072 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:55.065+0000 7f3aafe00640 1 -- 172.21.15.32:0/294378650 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ca2c633880 con 0x55ca2c2695c0 2024-08-03T20:50:55.072 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:55.065+0000 7f3aafe00640 1 --2- 172.21.15.32:0/294378650 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ca2c2695c0 0x55ca2c676590 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7f3aa8004ff0 tx=0x7f3aa80991f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=99e05a58c918121c server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:50:55.072 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:55.065+0000 7f3a8fe00640 10 client.0 ms_handle_connect on v2:172.21.15.32:3300/0 2024-08-03T20:50:55.072 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:55.066+0000 7f3a8fe00640 1 -- 172.21.15.32:0/294378650 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3aa801f040 con 0x55ca2c2695c0 2024-08-03T20:50:55.072 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:55.066+0000 7f3a8fe00640 1 -- 172.21.15.32:0/294378650 <== mon.0 v2:172.21.15.32:3300/0 2 ==== osd_map(58..58 src has 1..58) ==== 5795+0+0 (secure 0 0 0) 0x7f3aa805d8c0 con 0x55ca2c2695c0 2024-08-03T20:50:55.073 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:55.066+0000 7f3abd19c740 1 -- 172.21.15.32:0/294378650 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3a9c005320 con 0x55ca2c2695c0 2024-08-03T20:50:55.073 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:55.066+0000 7f3a8fe00640 1 -- 172.21.15.32:0/294378650 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_get_version_reply(handle=1 version=59) ==== 24+0+0 (secure 0 0 0) 0x7f3aa8095050 con 0x55ca2c2695c0 2024-08-03T20:50:55.073 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:55.066+0000 7f3abd19c740 10 client.6138 fetch_fsmap learned FSMap version 59 2024-08-03T20:50:55.073 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:55.066+0000 7f3abd19c740 1 -- 172.21.15.32:0/294378650 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,fsmap=59,monmap=2+}) -- 0x55ca2c639420 con 0x55ca2c2695c0 2024-08-03T20:50:55.073 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:55.066+0000 7f3a8fe00640 1 -- 172.21.15.32:0/294378650 <== mon.0 v2:172.21.15.32:3300/0 4 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f3a9c005320 con 0x55ca2c2695c0 2024-08-03T20:50:55.073 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:55.066+0000 7f3a8fe00640 1 -- 172.21.15.32:0/294378650 <== mon.0 v2:172.21.15.32:3300/0 5 ==== fsmap(e 59) ==== 3502+0+0 (secure 0 0 0) 0x7f3aa800f040 con 0x55ca2c2695c0 2024-08-03T20:50:55.073 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:55.067+0000 7f3abd19c740 10 client.6138 fetch_fsmap finished waiting for FSMap version 59 2024-08-03T20:50:55.073 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:55.067+0000 7f3abd19c740 10 client.6138 resolve_mds: resolved 4:0 to role '4:0' aka daemon mds.c 2024-08-03T20:50:55.073 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:55.067+0000 7f3abd19c740 20 client.6138 populate_metadata read hostname 'smithi032' 2024-08-03T20:50:55.073 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:55.067+0000 7f3abd19c740 1 --2- 172.21.15.32:0/294378650 >> [v2:172.21.15.32:6834/2963295201,v1:172.21.15.32:6835/2963295201] conn(0x55ca2c6281f0 0x55ca2c676ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:50:55.073 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:55.067+0000 7f3abd19c740 4 client.6138 mds_command: new command op to 6045 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-08-03T20:50:55.073 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:55.067+0000 7f3abd19c740 1 -- 172.21.15.32:0/294378650 --> [v2:172.21.15.32:6834/2963295201,v1:172.21.15.32:6835/2963295201] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55ca2c57fe00 con 0x55ca2c6281f0 2024-08-03T20:50:55.073 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:55.067+0000 7f3aaf400640 1 --2- 172.21.15.32:0/294378650 >> [v2:172.21.15.32:6834/2963295201,v1:172.21.15.32:6835/2963295201] conn(0x55ca2c6281f0 0x55ca2c676ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:50:55.074 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:55.068+0000 7f3aaf400640 1 --2- 172.21.15.32:0/294378650 >> [v2:172.21.15.32:6834/2963295201,v1:172.21.15.32:6835/2963295201] conn(0x55ca2c6281f0 0x55ca2c676ad0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=82af252a66a78dbb server_cookie=1a600debc6e426ed in_seq=0 out_seq=0 2024-08-03T20:50:55.074 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:55.068+0000 7f3a8fe00640 10 client.6138 ms_handle_connect on v2:172.21.15.32:6834/2963295201 2024-08-03T20:50:55.075 INFO:teuthology.orchestra.run.smithi032.stdout:{"mds_beacon_grace":"15.000000"} 2024-08-03T20:50:55.075 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_beacon_grace": "15.000000" } 2024-08-03T20:50:55.076 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-08-03T20:50:55.077 INFO:tasks.ceph.mds.c.smithi032.stderr:2024-08-03T20:50:55.071+0000 7fbff8600640 -1 mds.0.cache.den(0x1 a) newly corrupt dentry to be committed: [dentry #0x1/a [fffffffffffffff6,head] auth (dversion lock) v=13 ino=0x10000000000 state=1610612736 | inodepin=1 dirty=1 0x5559d62e7180] 2024-08-03T20:50:55.077 INFO:tasks.ceph.mds.c.smithi032.stderr:2024-08-03T20:50:55.071+0000 7fbff8600640 -1 log_channel(cluster) log [ERR] : MDS abort because newly corrupt dentry to be committed: [dentry #0x1/a [fffffffffffffff6,head] auth (dversion lock) v=13 ino=0x10000000000 state=1610612736 | inodepin=1 dirty=1 0x5559d62e7180] 2024-08-03T20:50:55.077 INFO:tasks.ceph.mds.c.smithi032.stderr:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-5476-g3792a0a4/rpm/el9/BUILD/ceph-19.0.0-5476-g3792a0a4/src/mds/MDSRank.cc: In function 'void MDSRank::abort(std::string_view)' thread 7fbff8600640 time 2024-08-03T20:50:55.072993+0000 2024-08-03T20:50:55.077 INFO:tasks.ceph.mds.c.smithi032.stderr:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-5476-g3792a0a4/rpm/el9/BUILD/ceph-19.0.0-5476-g3792a0a4/src/mds/MDSRank.cc: 952: ceph_abort_msg("abort() called") 2024-08-03T20:50:55.077 INFO:tasks.ceph.mds.c.smithi032.stderr: ceph version 19.0.0-5476-g3792a0a4 (3792a0a4a40f645dee271188eab388b1c22c10f8) squid (dev) 2024-08-03T20:50:55.077 INFO:tasks.ceph.mds.c.smithi032.stderr: 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xcf) [0x7fc007855480] 2024-08-03T20:50:55.077 INFO:tasks.ceph.mds.c.smithi032.stderr: 2: (MDSRank::abort(std::basic_string_view >)+0x5f) [0x5559b1352297] 2024-08-03T20:50:55.077 INFO:tasks.ceph.mds.c.smithi032.stderr: 3: (CDentry::check_corruption(bool)+0x530) [0x5559b1620a32] 2024-08-03T20:50:55.077 INFO:tasks.ceph.mds.c.smithi032.stderr: 4: ceph-mds(+0x3f4e3d) [0x5559b163be3d] 2024-08-03T20:50:55.077 INFO:tasks.ceph.mds.c.smithi032.stderr: 5: (CDir::_omap_commit(int)+0x7ff) [0x5559b163cbf5] 2024-08-03T20:50:55.078 INFO:tasks.ceph.mds.c.smithi032.stderr: 6: (CDir::_commit(unsigned long, int)+0x5ba) [0x5559b163d998] 2024-08-03T20:50:55.078 INFO:tasks.ceph.mds.c.smithi032.stderr: 7: (CDir::commit(unsigned long, MDSContext*, bool, int)+0x11a) [0x5559b163dc56] 2024-08-03T20:50:55.078 INFO:tasks.ceph.mds.c.smithi032.stderr: 8: (LogSegment::try_to_expire(MDSRank*, C_GatherBuilderBase >&, int)+0x10bf) [0x5559b13bfd17] 2024-08-03T20:50:55.078 INFO:tasks.ceph.mds.c.smithi032.stderr: 9: (MDLog::try_expire(LogSegment*, int)+0x6f) [0x5559b16ea67f] 2024-08-03T20:50:55.078 INFO:tasks.ceph.mds.c.smithi032.stderr: 10: (MDLog::trim_all()+0x85e) [0x5559b16edbb2] 2024-08-03T20:50:55.078 INFO:tasks.ceph.mds.c.smithi032.stderr: 11: ceph-mds(+0x1104a3) [0x5559b13574a3] 2024-08-03T20:50:55.078 INFO:tasks.ceph.mds.c.smithi032.stderr: 12: ceph-mds(+0x110601) [0x5559b1357601] 2024-08-03T20:50:55.078 INFO:tasks.ceph.mds.c.smithi032.stderr: 13: ceph-mds(+0x159629) [0x5559b13a0629] 2024-08-03T20:50:55.078 INFO:tasks.ceph.mds.c.smithi032.stderr: 14: ceph-mds(+0xeb381) [0x5559b1332381] 2024-08-03T20:50:55.078 INFO:tasks.ceph.mds.c.smithi032.stderr: 15: (MDSInternalContextWrapper::finish(int)+0x12) [0x5559b16e511e] 2024-08-03T20:50:55.078 INFO:tasks.ceph.mds.c.smithi032.stderr: 16: (MDSContext::complete(int)+0x7c) [0x5559b16e9ce0] 2024-08-03T20:50:55.078 INFO:tasks.ceph.mds.c.smithi032.stderr: 17: ceph-mds(+0x49e102) [0x5559b16e5102] 2024-08-03T20:50:55.078 INFO:tasks.ceph.mds.c.smithi032.stderr: 18: (MDSContext::complete(int)+0x7c) [0x5559b16e9ce0] 2024-08-03T20:50:55.079 INFO:tasks.ceph.mds.c.smithi032.stderr: 19: (MDSIOContextBase::complete(int)+0x501) [0x5559b16ea35f] 2024-08-03T20:50:55.079 INFO:tasks.ceph.mds.c.smithi032.stderr: 20: (C_IO_Wrapper::complete(int)+0x3c) [0x5559b16ea60e] 2024-08-03T20:50:55.079 INFO:tasks.ceph.mds.c.smithi032.stderr: 21: (Finisher::finisher_thread_entry()+0x495) [0x7fc0077dccb1] 2024-08-03T20:50:55.079 INFO:tasks.ceph.mds.c.smithi032.stderr: 22: ceph-mds(+0x103369) [0x5559b134a369] 2024-08-03T20:50:55.079 INFO:tasks.ceph.mds.c.smithi032.stderr: 23: (Thread::entry_wrapper()+0x43) [0x7fc00782df4f] 2024-08-03T20:50:55.079 INFO:tasks.ceph.mds.c.smithi032.stderr: 24: (Thread::_entry_func(void*)+0xd) [0x7fc00782df6b] 2024-08-03T20:50:55.079 INFO:tasks.ceph.mds.c.smithi032.stderr: 25: /lib64/libc.so.6(+0x89c02) [0x7fc006e89c02] 2024-08-03T20:50:55.079 INFO:tasks.ceph.mds.c.smithi032.stderr: 26: /lib64/libc.so.6(+0x10ec40) [0x7fc006f0ec40] 2024-08-03T20:50:55.079 INFO:tasks.ceph.mds.c.smithi032.stderr:2024-08-03T20:50:55.073+0000 7fbff8600640 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-5476-g3792a0a4/rpm/el9/BUILD/ceph-19.0.0-5476-g3792a0a4/src/mds/MDSRank.cc: In function 'void MDSRank::abort(std::string_view)' thread 7fbff8600640 time 2024-08-03T20:50:55.072993+0000 2024-08-03T20:50:55.079 INFO:tasks.ceph.mds.c.smithi032.stderr:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-5476-g3792a0a4/rpm/el9/BUILD/ceph-19.0.0-5476-g3792a0a4/src/mds/MDSRank.cc: 952: ceph_abort_msg("abort() called") 2024-08-03T20:50:55.080 INFO:tasks.ceph.mds.c.smithi032.stderr: 2024-08-03T20:50:55.080 INFO:tasks.ceph.mds.c.smithi032.stderr: ceph version 19.0.0-5476-g3792a0a4 (3792a0a4a40f645dee271188eab388b1c22c10f8) squid (dev) 2024-08-03T20:50:55.080 INFO:tasks.ceph.mds.c.smithi032.stderr: 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xcf) [0x7fc007855480] 2024-08-03T20:50:55.080 INFO:tasks.ceph.mds.c.smithi032.stderr: 2: (MDSRank::abort(std::basic_string_view >)+0x5f) [0x5559b1352297] 2024-08-03T20:50:55.080 INFO:tasks.ceph.mds.c.smithi032.stderr: 3: (CDentry::check_corruption(bool)+0x530) [0x5559b1620a32] 2024-08-03T20:50:55.080 INFO:tasks.ceph.mds.c.smithi032.stderr: 4: ceph-mds(+0x3f4e3d) [0x5559b163be3d] 2024-08-03T20:50:55.080 INFO:tasks.ceph.mds.c.smithi032.stderr: 5: (CDir::_omap_commit(int)+0x7ff) [0x5559b163cbf5] 2024-08-03T20:50:55.080 INFO:tasks.ceph.mds.c.smithi032.stderr: 6: (CDir::_commit(unsigned long, int)+0x5ba) [0x5559b163d998] 2024-08-03T20:50:55.080 INFO:tasks.ceph.mds.c.smithi032.stderr: 7: (CDir::commit(unsigned long, MDSContext*, bool, int)+0x11a) [0x5559b163dc56] 2024-08-03T20:50:55.080 INFO:tasks.ceph.mds.c.smithi032.stderr: 8: (LogSegment::try_to_expire(MDSRank*, C_GatherBuilderBase >&, int)+0x10bf) [0x5559b13bfd17] 2024-08-03T20:50:55.080 INFO:tasks.ceph.mds.c.smithi032.stderr: 9: (MDLog::try_expire(LogSegment*, int)+0x6f) [0x5559b16ea67f] 2024-08-03T20:50:55.080 INFO:tasks.ceph.mds.c.smithi032.stderr: 10: (MDLog::trim_all()+0x85e) [0x5559b16edbb2] 2024-08-03T20:50:55.080 INFO:tasks.ceph.mds.c.smithi032.stderr: 11: ceph-mds(+0x1104a3) [0x5559b13574a3] 2024-08-03T20:50:55.080 INFO:tasks.ceph.mds.c.smithi032.stderr: 12: ceph-mds(+0x110601) [0x5559b1357601] 2024-08-03T20:50:55.080 INFO:tasks.ceph.mds.c.smithi032.stderr: 13: ceph-mds(+0x159629) [0x5559b13a0629] 2024-08-03T20:50:55.081 INFO:tasks.ceph.mds.c.smithi032.stderr: 14: ceph-mds(+0xeb381) [0x5559b1332381] 2024-08-03T20:50:55.081 INFO:tasks.ceph.mds.c.smithi032.stderr: 15: (MDSInternalContextWrapper::finish(int)+0x12) [0x5559b16e511e] 2024-08-03T20:50:55.081 INFO:tasks.ceph.mds.c.smithi032.stderr: 16: (MDSContext::complete(int)+0x7c) [0x5559b16e9ce0] 2024-08-03T20:50:55.081 INFO:tasks.ceph.mds.c.smithi032.stderr: 17: ceph-mds(+0x49e102) [0x5559b16e5102] 2024-08-03T20:50:55.081 INFO:tasks.ceph.mds.c.smithi032.stderr: 18: (MDSContext::complete(int)+0x7c) [0x5559b16e9ce0] 2024-08-03T20:50:55.081 INFO:tasks.ceph.mds.c.smithi032.stderr: 19: (MDSIOContextBase::complete(int)+0x501) [0x5559b16ea35f] 2024-08-03T20:50:55.081 INFO:tasks.ceph.mds.c.smithi032.stderr: 20: (C_IO_Wrapper::complete(int)+0x3c) [0x5559b16ea60e] 2024-08-03T20:50:55.081 INFO:tasks.ceph.mds.c.smithi032.stderr: 21: (Finisher::finisher_thread_entry()+0x495) [0x7fc0077dccb1] 2024-08-03T20:50:55.081 INFO:tasks.ceph.mds.c.smithi032.stderr: 22: ceph-mds(+0x103369) [0x5559b134a369] 2024-08-03T20:50:55.081 INFO:tasks.ceph.mds.c.smithi032.stderr: 23: (Thread::entry_wrapper()+0x43) [0x7fc00782df4f] 2024-08-03T20:50:55.081 INFO:tasks.ceph.mds.c.smithi032.stderr: 24: (Thread::_entry_func(void*)+0xd) [0x7fc00782df6b] 2024-08-03T20:50:55.081 INFO:tasks.ceph.mds.c.smithi032.stderr: 25: /lib64/libc.so.6(+0x89c02) [0x7fc006e89c02] 2024-08-03T20:50:55.081 INFO:tasks.ceph.mds.c.smithi032.stderr: 26: /lib64/libc.so.6(+0x10ec40) [0x7fc006f0ec40] 2024-08-03T20:50:55.081 INFO:tasks.ceph.mds.c.smithi032.stderr: 2024-08-03T20:50:55.082 INFO:tasks.ceph.mds.c.smithi032.stderr:*** Caught signal (Aborted) ** 2024-08-03T20:50:55.082 INFO:tasks.ceph.mds.c.smithi032.stderr: in thread 7fbff8600640 thread_name:MR_Finisher 2024-08-03T20:50:55.082 INFO:tasks.ceph.mds.c.smithi032.stderr: ceph version 19.0.0-5476-g3792a0a4 (3792a0a4a40f645dee271188eab388b1c22c10f8) squid (dev) 2024-08-03T20:50:55.082 INFO:tasks.ceph.mds.c.smithi032.stderr: 1: /lib64/libc.so.6(+0x3e6f0) [0x7fc006e3e6f0] 2024-08-03T20:50:55.082 INFO:tasks.ceph.mds.c.smithi032.stderr: 2: /lib64/libc.so.6(+0x8b94c) [0x7fc006e8b94c] 2024-08-03T20:50:55.082 INFO:tasks.ceph.mds.c.smithi032.stderr: 3: raise() 2024-08-03T20:50:55.082 INFO:tasks.ceph.mds.c.smithi032.stderr: 4: abort() 2024-08-03T20:50:55.082 INFO:tasks.ceph.mds.c.smithi032.stderr: 5: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0x241) [0x7fc0078555f2] 2024-08-03T20:50:55.083 INFO:tasks.ceph.mds.c.smithi032.stderr: 6: (MDSRank::abort(std::basic_string_view >)+0x5f) [0x5559b1352297] 2024-08-03T20:50:55.083 INFO:tasks.ceph.mds.c.smithi032.stderr: 7: (CDentry::check_corruption(bool)+0x530) [0x5559b1620a32] 2024-08-03T20:50:55.083 INFO:tasks.ceph.mds.c.smithi032.stderr: 8: ceph-mds(+0x3f4e3d) [0x5559b163be3d] 2024-08-03T20:50:55.083 INFO:tasks.ceph.mds.c.smithi032.stderr: 9: (CDir::_omap_commit(int)+0x7ff) [0x5559b163cbf5] 2024-08-03T20:50:55.083 INFO:tasks.ceph.mds.c.smithi032.stderr: 10: (CDir::_commit(unsigned long, int)+0x5ba) [0x5559b163d998] 2024-08-03T20:50:55.083 INFO:tasks.ceph.mds.c.smithi032.stderr: 11: (CDir::commit(unsigned long, MDSContext*, bool, int)+0x11a) [0x5559b163dc56] 2024-08-03T20:50:55.083 INFO:tasks.ceph.mds.c.smithi032.stderr: 12: (LogSegment::try_to_expire(MDSRank*, C_GatherBuilderBase >&, int)+0x10bf) [0x5559b13bfd17] 2024-08-03T20:50:55.083 INFO:tasks.ceph.mds.c.smithi032.stderr: 13: (MDLog::try_expire(LogSegment*, int)+0x6f) [0x5559b16ea67f] 2024-08-03T20:50:55.083 INFO:tasks.ceph.mds.c.smithi032.stderr: 14: (MDLog::trim_all()+0x85e) [0x5559b16edbb2] 2024-08-03T20:50:55.083 INFO:tasks.ceph.mds.c.smithi032.stderr: 15: ceph-mds(+0x1104a3) [0x5559b13574a3] 2024-08-03T20:50:55.083 INFO:tasks.ceph.mds.c.smithi032.stderr: 16: ceph-mds(+0x110601) [0x5559b1357601] 2024-08-03T20:50:55.084 INFO:tasks.ceph.mds.c.smithi032.stderr: 17: ceph-mds(+0x159629) [0x5559b13a0629] 2024-08-03T20:50:55.084 INFO:tasks.ceph.mds.c.smithi032.stderr: 18: ceph-mds(+0xeb381) [0x5559b1332381] 2024-08-03T20:50:55.084 INFO:tasks.ceph.mds.c.smithi032.stderr: 19: (MDSInternalContextWrapper::finish(int)+0x12) [0x5559b16e511e] 2024-08-03T20:50:55.084 INFO:tasks.ceph.mds.c.smithi032.stderr: 20: (MDSContext::complete(int)+0x7c) [0x5559b16e9ce0] 2024-08-03T20:50:55.084 INFO:tasks.ceph.mds.c.smithi032.stderr: 21: ceph-mds(+0x49e102) [0x5559b16e5102] 2024-08-03T20:50:55.084 INFO:tasks.ceph.mds.c.smithi032.stderr: 22: (MDSContext::complete(int)+0x7c) [0x5559b16e9ce0] 2024-08-03T20:50:55.084 INFO:tasks.ceph.mds.c.smithi032.stderr: 23: (MDSIOContextBase::complete(int)+0x501) [0x5559b16ea35f] 2024-08-03T20:50:55.084 INFO:tasks.ceph.mds.c.smithi032.stderr: 24: (C_IO_Wrapper::complete(int)+0x3c) [0x5559b16ea60e] 2024-08-03T20:50:55.084 INFO:tasks.ceph.mds.c.smithi032.stderr: 25: (Finisher::finisher_thread_entry()+0x495) [0x7fc0077dccb1] 2024-08-03T20:50:55.084 INFO:tasks.ceph.mds.c.smithi032.stderr: 26: ceph-mds(+0x103369) [0x5559b134a369] 2024-08-03T20:50:55.084 INFO:tasks.ceph.mds.c.smithi032.stderr: 27: (Thread::entry_wrapper()+0x43) [0x7fc00782df4f] 2024-08-03T20:50:55.085 INFO:tasks.ceph.mds.c.smithi032.stderr: 28: (Thread::_entry_func(void*)+0xd) [0x7fc00782df6b] 2024-08-03T20:50:55.085 INFO:tasks.ceph.mds.c.smithi032.stderr: 29: /lib64/libc.so.6(+0x89c02) [0x7fc006e89c02] 2024-08-03T20:50:55.085 INFO:tasks.ceph.mds.c.smithi032.stderr: 30: /lib64/libc.so.6(+0x10ec40) [0x7fc006f0ec40] 2024-08-03T20:50:55.085 INFO:tasks.ceph.mds.c.smithi032.stderr:2024-08-03T20:50:55.074+0000 7fbff8600640 -1 *** Caught signal (Aborted) ** 2024-08-03T20:50:55.085 INFO:tasks.ceph.mds.c.smithi032.stderr: in thread 7fbff8600640 thread_name:MR_Finisher 2024-08-03T20:50:55.085 INFO:tasks.ceph.mds.c.smithi032.stderr: 2024-08-03T20:50:55.085 INFO:tasks.ceph.mds.c.smithi032.stderr: ceph version 19.0.0-5476-g3792a0a4 (3792a0a4a40f645dee271188eab388b1c22c10f8) squid (dev) 2024-08-03T20:50:55.085 INFO:tasks.ceph.mds.c.smithi032.stderr: 1: /lib64/libc.so.6(+0x3e6f0) [0x7fc006e3e6f0] 2024-08-03T20:50:55.085 INFO:tasks.ceph.mds.c.smithi032.stderr: 2: /lib64/libc.so.6(+0x8b94c) [0x7fc006e8b94c] 2024-08-03T20:50:55.085 INFO:tasks.ceph.mds.c.smithi032.stderr: 3: raise() 2024-08-03T20:50:55.085 INFO:tasks.ceph.mds.c.smithi032.stderr: 4: abort() 2024-08-03T20:50:55.085 INFO:tasks.ceph.mds.c.smithi032.stderr: 5: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0x241) [0x7fc0078555f2] 2024-08-03T20:50:55.085 INFO:tasks.ceph.mds.c.smithi032.stderr: 6: (MDSRank::abort(std::basic_string_view >)+0x5f) [0x5559b1352297] 2024-08-03T20:50:55.085 INFO:tasks.ceph.mds.c.smithi032.stderr: 7: (CDentry::check_corruption(bool)+0x530) [0x5559b1620a32] 2024-08-03T20:50:55.086 INFO:tasks.ceph.mds.c.smithi032.stderr: 8: ceph-mds(+0x3f4e3d) [0x5559b163be3d] 2024-08-03T20:50:55.086 INFO:tasks.ceph.mds.c.smithi032.stderr: 9: (CDir::_omap_commit(int)+0x7ff) [0x5559b163cbf5] 2024-08-03T20:50:55.086 INFO:tasks.ceph.mds.c.smithi032.stderr: 10: (CDir::_commit(unsigned long, int)+0x5ba) [0x5559b163d998] 2024-08-03T20:50:55.086 INFO:tasks.ceph.mds.c.smithi032.stderr: 11: (CDir::commit(unsigned long, MDSContext*, bool, int)+0x11a) [0x5559b163dc56] 2024-08-03T20:50:55.086 INFO:tasks.ceph.mds.c.smithi032.stderr: 12: (LogSegment::try_to_expire(MDSRank*, C_GatherBuilderBase >&, int)+0x10bf) [0x5559b13bfd17] 2024-08-03T20:50:55.086 INFO:tasks.ceph.mds.c.smithi032.stderr: 13: (MDLog::try_expire(LogSegment*, int)+0x6f) [0x5559b16ea67f] 2024-08-03T20:50:55.086 INFO:tasks.ceph.mds.c.smithi032.stderr: 14: (MDLog::trim_all()+0x85e) [0x5559b16edbb2] 2024-08-03T20:50:55.086 INFO:tasks.ceph.mds.c.smithi032.stderr: 15: ceph-mds(+0x1104a3) [0x5559b13574a3] 2024-08-03T20:50:55.086 INFO:tasks.ceph.mds.c.smithi032.stderr: 16: ceph-mds(+0x110601) [0x5559b1357601] 2024-08-03T20:50:55.086 INFO:tasks.ceph.mds.c.smithi032.stderr: 17: ceph-mds(+0x159629) [0x5559b13a0629] 2024-08-03T20:50:55.086 INFO:tasks.ceph.mds.c.smithi032.stderr: 18: ceph-mds(+0xeb381) [0x5559b1332381] 2024-08-03T20:50:55.086 INFO:tasks.ceph.mds.c.smithi032.stderr: 19: (MDSInternalContextWrapper::finish(int)+0x12) [0x5559b16e511e] 2024-08-03T20:50:55.086 INFO:tasks.ceph.mds.c.smithi032.stderr: 20: (MDSContext::complete(int)+0x7c) [0x5559b16e9ce0] 2024-08-03T20:50:55.087 INFO:tasks.ceph.mds.c.smithi032.stderr: 21: ceph-mds(+0x49e102) [0x5559b16e5102] 2024-08-03T20:50:55.087 INFO:tasks.ceph.mds.c.smithi032.stderr: 22: (MDSContext::complete(int)+0x7c) [0x5559b16e9ce0] 2024-08-03T20:50:55.087 INFO:tasks.ceph.mds.c.smithi032.stderr: 23: (MDSIOContextBase::complete(int)+0x501) [0x5559b16ea35f] 2024-08-03T20:50:55.087 INFO:tasks.ceph.mds.c.smithi032.stderr: 24: (C_IO_Wrapper::complete(int)+0x3c) [0x5559b16ea60e] 2024-08-03T20:50:55.087 INFO:tasks.ceph.mds.c.smithi032.stderr: 25: (Finisher::finisher_thread_entry()+0x495) [0x7fc0077dccb1] 2024-08-03T20:50:55.087 INFO:tasks.ceph.mds.c.smithi032.stderr: 26: ceph-mds(+0x103369) [0x5559b134a369] 2024-08-03T20:50:55.087 INFO:tasks.ceph.mds.c.smithi032.stderr: 27: (Thread::entry_wrapper()+0x43) [0x7fc00782df4f] 2024-08-03T20:50:55.087 INFO:tasks.ceph.mds.c.smithi032.stderr: 28: (Thread::_entry_func(void*)+0xd) [0x7fc00782df6b] 2024-08-03T20:50:55.087 INFO:tasks.ceph.mds.c.smithi032.stderr: 29: /lib64/libc.so.6(+0x89c02) [0x7fc006e89c02] 2024-08-03T20:50:55.087 INFO:tasks.ceph.mds.c.smithi032.stderr: 30: /lib64/libc.so.6(+0x10ec40) [0x7fc006f0ec40] 2024-08-03T20:50:55.087 INFO:tasks.ceph.mds.c.smithi032.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-08-03T20:50:55.087 INFO:tasks.ceph.mds.c.smithi032.stderr: 2024-08-03T20:50:55.089 INFO:tasks.ceph.mds.c.smithi032.stderr: -311> 2024-08-03T20:50:51.691+0000 7fbffd600640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-08-03T20:50:55.089 INFO:tasks.ceph.mds.c.smithi032.stderr: -12> 2024-08-03T20:50:55.071+0000 7fbff8600640 -1 mds.0.cache.den(0x1 a) newly corrupt dentry to be committed: [dentry #0x1/a [fffffffffffffff6,head] auth (dversion lock) v=13 ino=0x10000000000 state=1610612736 | inodepin=1 dirty=1 0x5559d62e7180] 2024-08-03T20:50:55.089 INFO:tasks.ceph.mds.c.smithi032.stderr: -10> 2024-08-03T20:50:55.071+0000 7fbff8600640 -1 log_channel(cluster) log [ERR] : MDS abort because newly corrupt dentry to be committed: [dentry #0x1/a [fffffffffffffff6,head] auth (dversion lock) v=13 ino=0x10000000000 state=1610612736 | inodepin=1 dirty=1 0x5559d62e7180] 2024-08-03T20:50:55.089 INFO:tasks.ceph.mds.c.smithi032.stderr: -3> 2024-08-03T20:50:55.073+0000 7fbff8600640 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-5476-g3792a0a4/rpm/el9/BUILD/ceph-19.0.0-5476-g3792a0a4/src/mds/MDSRank.cc: In function 'void MDSRank::abort(std::string_view)' thread 7fbff8600640 time 2024-08-03T20:50:55.072993+0000 2024-08-03T20:50:55.089 INFO:tasks.ceph.mds.c.smithi032.stderr:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-5476-g3792a0a4/rpm/el9/BUILD/ceph-19.0.0-5476-g3792a0a4/src/mds/MDSRank.cc: 952: ceph_abort_msg("abort() called") 2024-08-03T20:50:55.089 INFO:tasks.ceph.mds.c.smithi032.stderr: 2024-08-03T20:50:55.089 INFO:tasks.ceph.mds.c.smithi032.stderr: ceph version 19.0.0-5476-g3792a0a4 (3792a0a4a40f645dee271188eab388b1c22c10f8) squid (dev) 2024-08-03T20:50:55.089 INFO:tasks.ceph.mds.c.smithi032.stderr: 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xcf) [0x7fc007855480] 2024-08-03T20:50:55.089 INFO:tasks.ceph.mds.c.smithi032.stderr: 2: (MDSRank::abort(std::basic_string_view >)+0x5f) [0x5559b1352297] 2024-08-03T20:50:55.089 INFO:tasks.ceph.mds.c.smithi032.stderr: 3: (CDentry::check_corruption(bool)+0x530) [0x5559b1620a32] 2024-08-03T20:50:55.089 INFO:tasks.ceph.mds.c.smithi032.stderr: 4: ceph-mds(+0x3f4e3d) [0x5559b163be3d] 2024-08-03T20:50:55.089 INFO:tasks.ceph.mds.c.smithi032.stderr: 5: (CDir::_omap_commit(int)+0x7ff) [0x5559b163cbf5] 2024-08-03T20:50:55.090 INFO:tasks.ceph.mds.c.smithi032.stderr: 6: (CDir::_commit(unsigned long, int)+0x5ba) [0x5559b163d998] 2024-08-03T20:50:55.090 INFO:tasks.ceph.mds.c.smithi032.stderr: 7: (CDir::commit(unsigned long, MDSContext*, bool, int)+0x11a) [0x5559b163dc56] 2024-08-03T20:50:55.090 INFO:tasks.ceph.mds.c.smithi032.stderr: 8: (LogSegment::try_to_expire(MDSRank*, C_GatherBuilderBase >&, int)+0x10bf) [0x5559b13bfd17] 2024-08-03T20:50:55.090 INFO:tasks.ceph.mds.c.smithi032.stderr: 9: (MDLog::try_expire(LogSegment*, int)+0x6f) [0x5559b16ea67f] 2024-08-03T20:50:55.090 INFO:tasks.ceph.mds.c.smithi032.stderr: 10: (MDLog::trim_all()+0x85e) [0x5559b16edbb2] 2024-08-03T20:50:55.090 INFO:tasks.ceph.mds.c.smithi032.stderr: 11: ceph-mds(+0x1104a3) [0x5559b13574a3] 2024-08-03T20:50:55.090 INFO:tasks.ceph.mds.c.smithi032.stderr: 12: ceph-mds(+0x110601) [0x5559b1357601] 2024-08-03T20:50:55.090 INFO:tasks.ceph.mds.c.smithi032.stderr: 13: ceph-mds(+0x159629) [0x5559b13a0629] 2024-08-03T20:50:55.090 INFO:tasks.ceph.mds.c.smithi032.stderr: 14: ceph-mds(+0xeb381) [0x5559b1332381] 2024-08-03T20:50:55.090 INFO:tasks.ceph.mds.c.smithi032.stderr: 15: (MDSInternalContextWrapper::finish(int)+0x12) [0x5559b16e511e] 2024-08-03T20:50:55.090 INFO:tasks.ceph.mds.c.smithi032.stderr: 16: (MDSContext::complete(int)+0x7c) [0x5559b16e9ce0] 2024-08-03T20:50:55.090 INFO:tasks.ceph.mds.c.smithi032.stderr: 17: ceph-mds(+0x49e102) [0x5559b16e5102] 2024-08-03T20:50:55.090 INFO:tasks.ceph.mds.c.smithi032.stderr: 18: (MDSContext::complete(int)+0x7c) [0x5559b16e9ce0] 2024-08-03T20:50:55.090 INFO:tasks.ceph.mds.c.smithi032.stderr: 19: (MDSIOContextBase::complete(int)+0x501) [0x5559b16ea35f] 2024-08-03T20:50:55.091 INFO:tasks.ceph.mds.c.smithi032.stderr: 20: (C_IO_Wrapper::complete(int)+0x3c) [0x5559b16ea60e] 2024-08-03T20:50:55.091 INFO:tasks.ceph.mds.c.smithi032.stderr: 21: (Finisher::finisher_thread_entry()+0x495) [0x7fc0077dccb1] 2024-08-03T20:50:55.091 INFO:tasks.ceph.mds.c.smithi032.stderr: 22: ceph-mds(+0x103369) [0x5559b134a369] 2024-08-03T20:50:55.091 INFO:tasks.ceph.mds.c.smithi032.stderr: 23: (Thread::entry_wrapper()+0x43) [0x7fc00782df4f] 2024-08-03T20:50:55.091 INFO:tasks.ceph.mds.c.smithi032.stderr: 24: (Thread::_entry_func(void*)+0xd) [0x7fc00782df6b] 2024-08-03T20:50:55.092 INFO:tasks.ceph.mds.c.smithi032.stderr: 25: /lib64/libc.so.6(+0x89c02) [0x7fc006e89c02] 2024-08-03T20:50:55.092 INFO:tasks.ceph.mds.c.smithi032.stderr: 26: /lib64/libc.so.6(+0x10ec40) [0x7fc006f0ec40] 2024-08-03T20:50:55.092 INFO:tasks.ceph.mds.c.smithi032.stderr: 2024-08-03T20:50:55.092 INFO:tasks.ceph.mds.c.smithi032.stderr: 0> 2024-08-03T20:50:55.074+0000 7fbff8600640 -1 *** Caught signal (Aborted) ** 2024-08-03T20:50:55.092 INFO:tasks.ceph.mds.c.smithi032.stderr: in thread 7fbff8600640 thread_name:MR_Finisher 2024-08-03T20:50:55.093 INFO:tasks.ceph.mds.c.smithi032.stderr: 2024-08-03T20:50:55.093 INFO:tasks.ceph.mds.c.smithi032.stderr: ceph version 19.0.0-5476-g3792a0a4 (3792a0a4a40f645dee271188eab388b1c22c10f8) squid (dev) 2024-08-03T20:50:55.093 INFO:tasks.ceph.mds.c.smithi032.stderr: 1: /lib64/libc.so.6(+0x3e6f0) [0x7fc006e3e6f0] 2024-08-03T20:50:55.093 INFO:tasks.ceph.mds.c.smithi032.stderr: 2: /lib64/libc.so.6(+0x8b94c) [0x7fc006e8b94c] 2024-08-03T20:50:55.093 INFO:tasks.ceph.mds.c.smithi032.stderr: 3: raise() 2024-08-03T20:50:55.093 INFO:tasks.ceph.mds.c.smithi032.stderr: 4: abort() 2024-08-03T20:50:55.093 INFO:tasks.ceph.mds.c.smithi032.stderr: 5: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0x241) [0x7fc0078555f2] 2024-08-03T20:50:55.093 INFO:tasks.ceph.mds.c.smithi032.stderr: 6: (MDSRank::abort(std::basic_string_view >)+0x5f) [0x5559b1352297] 2024-08-03T20:50:55.093 INFO:tasks.ceph.mds.c.smithi032.stderr: 7: (CDentry::check_corruption(bool)+0x530) [0x5559b1620a32] 2024-08-03T20:50:55.094 INFO:tasks.ceph.mds.c.smithi032.stderr: 8: ceph-mds(+0x3f4e3d) [0x5559b163be3d] 2024-08-03T20:50:55.094 INFO:tasks.ceph.mds.c.smithi032.stderr: 9: (CDir::_omap_commit(int)+0x7ff) [0x5559b163cbf5] 2024-08-03T20:50:55.094 INFO:tasks.ceph.mds.c.smithi032.stderr: 10: (CDir::_commit(unsigned long, int)+0x5ba) [0x5559b163d998] 2024-08-03T20:50:55.094 INFO:tasks.ceph.mds.c.smithi032.stderr: 11: (CDir::commit(unsigned long, MDSContext*, bool, int)+0x11a) [0x5559b163dc56] 2024-08-03T20:50:55.094 INFO:tasks.ceph.mds.c.smithi032.stderr: 12: (LogSegment::try_to_expire(MDSRank*, C_GatherBuilderBase >&, int)+0x10bf) [0x5559b13bfd17] 2024-08-03T20:50:55.094 INFO:tasks.ceph.mds.c.smithi032.stderr: 13: (MDLog::try_expire(LogSegment*, int)+0x6f) [0x5559b16ea67f] 2024-08-03T20:50:55.094 INFO:tasks.ceph.mds.c.smithi032.stderr: 14: (MDLog::trim_all()+0x85e) [0x5559b16edbb2] 2024-08-03T20:50:55.094 INFO:tasks.ceph.mds.c.smithi032.stderr: 15: ceph-mds(+0x1104a3) [0x5559b13574a3] 2024-08-03T20:50:55.094 INFO:tasks.ceph.mds.c.smithi032.stderr: 16: ceph-mds(+0x110601) [0x5559b1357601] 2024-08-03T20:50:55.095 INFO:tasks.ceph.mds.c.smithi032.stderr: 17: ceph-mds(+0x159629) [0x5559b13a0629] 2024-08-03T20:50:55.095 INFO:tasks.ceph.mds.c.smithi032.stderr: 18: ceph-mds(+0xeb381) [0x5559b1332381] 2024-08-03T20:50:55.095 INFO:tasks.ceph.mds.c.smithi032.stderr: 19: (MDSInternalContextWrapper::finish(int)+0x12) [0x5559b16e511e] 2024-08-03T20:50:55.095 INFO:tasks.ceph.mds.c.smithi032.stderr: 20: (MDSContext::complete(int)+0x7c) [0x5559b16e9ce0] 2024-08-03T20:50:55.095 INFO:tasks.ceph.mds.c.smithi032.stderr: 21: ceph-mds(+0x49e102) [0x5559b16e5102] 2024-08-03T20:50:55.095 INFO:tasks.ceph.mds.c.smithi032.stderr: 22: (MDSContext::complete(int)+0x7c) [0x5559b16e9ce0] 2024-08-03T20:50:55.095 INFO:tasks.ceph.mds.c.smithi032.stderr: 23: (MDSIOContextBase::complete(int)+0x501) [0x5559b16ea35f] 2024-08-03T20:50:55.095 INFO:tasks.ceph.mds.c.smithi032.stderr: 24: (C_IO_Wrapper::complete(int)+0x3c) [0x5559b16ea60e] 2024-08-03T20:50:55.095 INFO:tasks.ceph.mds.c.smithi032.stderr: 25: (Finisher::finisher_thread_entry()+0x495) [0x7fc0077dccb1] 2024-08-03T20:50:55.095 INFO:tasks.ceph.mds.c.smithi032.stderr: 26: ceph-mds(+0x103369) [0x5559b134a369] 2024-08-03T20:50:55.095 INFO:tasks.ceph.mds.c.smithi032.stderr: 27: (Thread::entry_wrapper()+0x43) [0x7fc00782df4f] 2024-08-03T20:50:55.095 INFO:tasks.ceph.mds.c.smithi032.stderr: 28: (Thread::_entry_func(void*)+0xd) [0x7fc00782df6b] 2024-08-03T20:50:55.096 INFO:tasks.ceph.mds.c.smithi032.stderr: 29: /lib64/libc.so.6(+0x89c02) [0x7fc006e89c02] 2024-08-03T20:50:55.096 INFO:tasks.ceph.mds.c.smithi032.stderr: 30: /lib64/libc.so.6(+0x10ec40) [0x7fc006f0ec40] 2024-08-03T20:50:55.096 INFO:tasks.ceph.mds.c.smithi032.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-08-03T20:50:55.096 INFO:tasks.ceph.mds.c.smithi032.stderr: 2024-08-03T20:50:55.096 INFO:tasks.ceph.mds.c.smithi032.stderr: -311> 2024-08-03T20:50:51.691+0000 7fbffd600640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-08-03T20:50:55.096 INFO:tasks.ceph.mds.c.smithi032.stderr: -12> 2024-08-03T20:50:55.071+0000 7fbff8600640 -1 mds.0.cache.den(0x1 a) newly corrupt dentry to be committed: [dentry #0x1/a [fffffffffffffff6,head] auth (dversion lock) v=13 ino=0x10000000000 state=1610612736 | inodepin=1 dirty=1 0x5559d62e7180] 2024-08-03T20:50:55.096 INFO:tasks.ceph.mds.c.smithi032.stderr: -10> 2024-08-03T20:50:55.071+0000 7fbff8600640 -1 log_channel(cluster) log [ERR] : MDS abort because newly corrupt dentry to be committed: [dentry #0x1/a [fffffffffffffff6,head] auth (dversion lock) v=13 ino=0x10000000000 state=1610612736 | inodepin=1 dirty=1 0x5559d62e7180] 2024-08-03T20:50:55.096 INFO:tasks.ceph.mds.c.smithi032.stderr: -3> 2024-08-03T20:50:55.073+0000 7fbff8600640 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-5476-g3792a0a4/rpm/el9/BUILD/ceph-19.0.0-5476-g3792a0a4/src/mds/MDSRank.cc: In function 'void MDSRank::abort(std::string_view)' thread 7fbff8600640 time 2024-08-03T20:50:55.072993+0000 2024-08-03T20:50:55.096 INFO:tasks.ceph.mds.c.smithi032.stderr:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-5476-g3792a0a4/rpm/el9/BUILD/ceph-19.0.0-5476-g3792a0a4/src/mds/MDSRank.cc: 952: ceph_abort_msg("abort() called") 2024-08-03T20:50:55.096 INFO:tasks.ceph.mds.c.smithi032.stderr: 2024-08-03T20:50:55.096 INFO:tasks.ceph.mds.c.smithi032.stderr: ceph version 19.0.0-5476-g3792a0a4 (3792a0a4a40f645dee271188eab388b1c22c10f8) squid (dev) 2024-08-03T20:50:55.097 INFO:tasks.ceph.mds.c.smithi032.stderr: 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xcf) [0x7fc007855480] 2024-08-03T20:50:55.097 INFO:tasks.ceph.mds.c.smithi032.stderr: 2: (MDSRank::abort(std::basic_string_view >)+0x5f) [0x5559b1352297] 2024-08-03T20:50:55.097 INFO:tasks.ceph.mds.c.smithi032.stderr: 3: (CDentry::check_corruption(bool)+0x530) [0x5559b1620a32] 2024-08-03T20:50:55.097 INFO:tasks.ceph.mds.c.smithi032.stderr: 4: ceph-mds(+0x3f4e3d) [0x5559b163be3d] 2024-08-03T20:50:55.097 INFO:tasks.ceph.mds.c.smithi032.stderr: 5: (CDir::_omap_commit(int)+0x7ff) [0x5559b163cbf5] 2024-08-03T20:50:55.097 INFO:tasks.ceph.mds.c.smithi032.stderr: 6: (CDir::_commit(unsigned long, int)+0x5ba) [0x5559b163d998] 2024-08-03T20:50:55.097 INFO:tasks.ceph.mds.c.smithi032.stderr: 7: (CDir::commit(unsigned long, MDSContext*, bool, int)+0x11a) [0x5559b163dc56] 2024-08-03T20:50:55.097 INFO:tasks.ceph.mds.c.smithi032.stderr: 8: (LogSegment::try_to_expire(MDSRank*, C_GatherBuilderBase >&, int)+0x10bf) [0x5559b13bfd17] 2024-08-03T20:50:55.097 INFO:tasks.ceph.mds.c.smithi032.stderr: 9: (MDLog::try_expire(LogSegment*, int)+0x6f) [0x5559b16ea67f] 2024-08-03T20:50:55.097 INFO:tasks.ceph.mds.c.smithi032.stderr: 10: (MDLog::trim_all()+0x85e) [0x5559b16edbb2] 2024-08-03T20:50:55.098 INFO:tasks.ceph.mds.c.smithi032.stderr: 11: ceph-mds(+0x1104a3) [0x5559b13574a3] 2024-08-03T20:50:55.098 INFO:tasks.ceph.mds.c.smithi032.stderr: 12: ceph-mds(+0x110601) [0x5559b1357601] 2024-08-03T20:50:55.098 INFO:tasks.ceph.mds.c.smithi032.stderr: 13: ceph-mds(+0x159629) [0x5559b13a0629] 2024-08-03T20:50:55.098 INFO:tasks.ceph.mds.c.smithi032.stderr: 14: ceph-mds(+0xeb381) [0x5559b1332381] 2024-08-03T20:50:55.098 INFO:tasks.ceph.mds.c.smithi032.stderr: 15: (MDSInternalContextWrapper::finish(int)+0x12) [0x5559b16e511e] 2024-08-03T20:50:55.098 INFO:tasks.ceph.mds.c.smithi032.stderr: 16: (MDSContext::complete(int)+0x7c) [0x5559b16e9ce0] 2024-08-03T20:50:55.098 INFO:tasks.ceph.mds.c.smithi032.stderr: 17: ceph-mds(+0x49e102) [0x5559b16e5102] 2024-08-03T20:50:55.098 INFO:tasks.ceph.mds.c.smithi032.stderr: 18: (MDSContext::complete(int)+0x7c) [0x5559b16e9ce0] 2024-08-03T20:50:55.098 INFO:tasks.ceph.mds.c.smithi032.stderr: 19: (MDSIOContextBase::complete(int)+0x501) [0x5559b16ea35f] 2024-08-03T20:50:55.099 INFO:tasks.ceph.mds.c.smithi032.stderr: 20: (C_IO_Wrapper::complete(int)+0x3c) [0x5559b16ea60e] 2024-08-03T20:50:55.099 INFO:tasks.ceph.mds.c.smithi032.stderr: 21: (Finisher::finisher_thread_entry()+0x495) [0x7fc0077dccb1] 2024-08-03T20:50:55.099 INFO:tasks.ceph.mds.c.smithi032.stderr: 22: ceph-mds(+0x103369) [0x5559b134a369] 2024-08-03T20:50:55.099 INFO:tasks.ceph.mds.c.smithi032.stderr: 23: (Thread::entry_wrapper()+0x43) [0x7fc00782df4f] 2024-08-03T20:50:55.099 INFO:tasks.ceph.mds.c.smithi032.stderr: 24: (Thread::_entry_func(void*)+0xd) [0x7fc00782df6b] 2024-08-03T20:50:55.099 INFO:tasks.ceph.mds.c.smithi032.stderr: 25: /lib64/libc.so.6(+0x89c02) [0x7fc006e89c02] 2024-08-03T20:50:55.099 INFO:tasks.ceph.mds.c.smithi032.stderr: 26: /lib64/libc.so.6(+0x10ec40) [0x7fc006f0ec40] 2024-08-03T20:50:55.099 INFO:tasks.ceph.mds.c.smithi032.stderr: 2024-08-03T20:50:55.099 INFO:tasks.ceph.mds.c.smithi032.stderr: 0> 2024-08-03T20:50:55.074+0000 7fbff8600640 -1 *** Caught signal (Aborted) ** 2024-08-03T20:50:55.100 INFO:tasks.ceph.mds.c.smithi032.stderr: in thread 7fbff8600640 thread_name:MR_Finisher 2024-08-03T20:50:55.100 INFO:tasks.ceph.mds.c.smithi032.stderr: 2024-08-03T20:50:55.100 INFO:tasks.ceph.mds.c.smithi032.stderr: ceph version 19.0.0-5476-g3792a0a4 (3792a0a4a40f645dee271188eab388b1c22c10f8) squid (dev) 2024-08-03T20:50:55.100 INFO:tasks.ceph.mds.c.smithi032.stderr: 1: /lib64/libc.so.6(+0x3e6f0) [0x7fc006e3e6f0] 2024-08-03T20:50:55.100 INFO:tasks.ceph.mds.c.smithi032.stderr: 2: /lib64/libc.so.6(+0x8b94c) [0x7fc006e8b94c] 2024-08-03T20:50:55.100 INFO:tasks.ceph.mds.c.smithi032.stderr: 3: raise() 2024-08-03T20:50:55.100 INFO:tasks.ceph.mds.c.smithi032.stderr: 4: abort() 2024-08-03T20:50:55.100 INFO:tasks.ceph.mds.c.smithi032.stderr: 5: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0x241) [0x7fc0078555f2] 2024-08-03T20:50:55.100 INFO:tasks.ceph.mds.c.smithi032.stderr: 6: (MDSRank::abort(std::basic_string_view >)+0x5f) [0x5559b1352297] 2024-08-03T20:50:55.101 INFO:tasks.ceph.mds.c.smithi032.stderr: 7: (CDentry::check_corruption(bool)+0x530) [0x5559b1620a32] 2024-08-03T20:50:55.101 INFO:tasks.ceph.mds.c.smithi032.stderr: 8: ceph-mds(+0x3f4e3d) [0x5559b163be3d] 2024-08-03T20:50:55.101 INFO:tasks.ceph.mds.c.smithi032.stderr: 9: (CDir::_omap_commit(int)+0x7ff) [0x5559b163cbf5] 2024-08-03T20:50:55.101 INFO:tasks.ceph.mds.c.smithi032.stderr: 10: (CDir::_commit(unsigned long, int)+0x5ba) [0x5559b163d998] 2024-08-03T20:50:55.101 INFO:tasks.ceph.mds.c.smithi032.stderr: 11: (CDir::commit(unsigned long, MDSContext*, bool, int)+0x11a) [0x5559b163dc56] 2024-08-03T20:50:55.101 INFO:tasks.ceph.mds.c.smithi032.stderr: 12: (LogSegment::try_to_expire(MDSRank*, C_GatherBuilderBase >&, int)+0x10bf) [0x5559b13bfd17] 2024-08-03T20:50:55.101 INFO:tasks.ceph.mds.c.smithi032.stderr: 13: (MDLog::try_expire(LogSegment*, int)+0x6f) [0x5559b16ea67f] 2024-08-03T20:50:55.101 INFO:tasks.ceph.mds.c.smithi032.stderr: 14: (MDLog::trim_all()+0x85e) [0x5559b16edbb2] 2024-08-03T20:50:55.101 INFO:tasks.ceph.mds.c.smithi032.stderr: 15: ceph-mds(+0x1104a3) [0x5559b13574a3] 2024-08-03T20:50:55.101 INFO:tasks.ceph.mds.c.smithi032.stderr: 16: ceph-mds(+0x110601) [0x5559b1357601] 2024-08-03T20:50:55.102 INFO:tasks.ceph.mds.c.smithi032.stderr: 17: ceph-mds(+0x159629) [0x5559b13a0629] 2024-08-03T20:50:55.102 INFO:tasks.ceph.mds.c.smithi032.stderr: 18: ceph-mds(+0xeb381) [0x5559b1332381] 2024-08-03T20:50:55.102 INFO:tasks.ceph.mds.c.smithi032.stderr: 19: (MDSInternalContextWrapper::finish(int)+0x12) [0x5559b16e511e] 2024-08-03T20:50:55.102 INFO:tasks.ceph.mds.c.smithi032.stderr: 20: (MDSContext::complete(int)+0x7c) [0x5559b16e9ce0] 2024-08-03T20:50:55.102 INFO:tasks.ceph.mds.c.smithi032.stderr: 21: ceph-mds(+0x49e102) [0x5559b16e5102] 2024-08-03T20:50:55.102 INFO:tasks.ceph.mds.c.smithi032.stderr: 22: (MDSContext::complete(int)+0x7c) [0x5559b16e9ce0] 2024-08-03T20:50:55.102 INFO:tasks.ceph.mds.c.smithi032.stderr: 23: (MDSIOContextBase::complete(int)+0x501) [0x5559b16ea35f] 2024-08-03T20:50:55.102 INFO:tasks.ceph.mds.c.smithi032.stderr: 24: (C_IO_Wrapper::complete(int)+0x3c) [0x5559b16ea60e] 2024-08-03T20:50:55.102 INFO:tasks.ceph.mds.c.smithi032.stderr: 25: (Finisher::finisher_thread_entry()+0x495) [0x7fc0077dccb1] 2024-08-03T20:50:55.102 INFO:tasks.ceph.mds.c.smithi032.stderr: 26: ceph-mds(+0x103369) [0x5559b134a369] 2024-08-03T20:50:55.102 INFO:tasks.ceph.mds.c.smithi032.stderr: 27: (Thread::entry_wrapper()+0x43) [0x7fc00782df4f] 2024-08-03T20:50:55.103 INFO:tasks.ceph.mds.c.smithi032.stderr: 28: (Thread::_entry_func(void*)+0xd) [0x7fc00782df6b] 2024-08-03T20:50:55.103 INFO:tasks.ceph.mds.c.smithi032.stderr: 29: /lib64/libc.so.6(+0x89c02) [0x7fc006e89c02] 2024-08-03T20:50:55.103 INFO:tasks.ceph.mds.c.smithi032.stderr: 30: /lib64/libc.so.6(+0x10ec40) [0x7fc006f0ec40] 2024-08-03T20:50:55.103 INFO:tasks.ceph.mds.c.smithi032.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-08-03T20:50:55.103 INFO:tasks.ceph.mds.c.smithi032.stderr: 2024-08-03T20:50:55.314 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:55.312+0000 7f3aaf400640 1 -- 172.21.15.32:0/294378650 >> [v2:172.21.15.32:6834/2963295201,v1:172.21.15.32:6835/2963295201] conn(0x55ca2c6281f0 msgr2=0x55ca2c676ad0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 17 2024-08-03T20:50:55.315 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:55.312+0000 7f3aaf400640 1 -- 172.21.15.32:0/294378650 >> [v2:172.21.15.32:6834/2963295201,v1:172.21.15.32:6835/2963295201] conn(0x55ca2c6281f0 msgr2=0x55ca2c676ad0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-08-03T20:50:55.315 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:55.312+0000 7f3aaf400640 1 --2- 172.21.15.32:0/294378650 >> [v2:172.21.15.32:6834/2963295201,v1:172.21.15.32:6835/2963295201] conn(0x55ca2c6281f0 0x55ca2c676ad0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-08-03T20:50:55.315 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:55.313+0000 7f3aaf400640 1 --2- 172.21.15.32:0/294378650 >> [v2:172.21.15.32:6834/2963295201,v1:172.21.15.32:6835/2963295201] conn(0x55ca2c6281f0 0x55ca2c676ad0 unknown :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault initiating reconnect 2024-08-03T20:50:55.315 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:55.313+0000 7f3aaf400640 1 -- 172.21.15.32:0/294378650 >> [v2:172.21.15.32:6834/2963295201,v1:172.21.15.32:6835/2963295201] conn(0x55ca2c6281f0 msgr2=0x55ca2c676ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk reading from fd=17 : -104 Unknown error -104 2024-08-03T20:50:55.315 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:55.313+0000 7f3aaf400640 1 -- 172.21.15.32:0/294378650 >> [v2:172.21.15.32:6834/2963295201,v1:172.21.15.32:6835/2963295201] conn(0x55ca2c6281f0 msgr2=0x55ca2c676ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-08-03T20:50:55.315 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:55.313+0000 7f3aaf400640 1 --2- 172.21.15.32:0/294378650 >> [v2:172.21.15.32:6834/2963295201,v1:172.21.15.32:6835/2963295201] conn(0x55ca2c6281f0 0x55ca2c676ad0 unknown :-1 s=BANNER_CONNECTING pgs=14 cs=1 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner read peer banner failed r=-1 ((1) Operation not permitted) 2024-08-03T20:50:55.315 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:55.313+0000 7f3aaf400640 1 --2- 172.21.15.32:0/294378650 >> [v2:172.21.15.32:6834/2963295201,v1:172.21.15.32:6835/2963295201] conn(0x55ca2c6281f0 0x55ca2c676ad0 unknown :-1 s=START_CONNECT pgs=14 cs=2 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-08-03T20:50:55.326 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:55.324+0000 7f86c8000640 1 -- 172.21.15.32:0/1424726153 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f86c00cf9c0 msgr2=0x7f86c00d0a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:50:55.326 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:55.324+0000 7f86c8000640 1 --2- 172.21.15.32:0/1424726153 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f86c00cf9c0 0x7f86c00d0a70 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7f86ac009830 tx=0x7f86ac02a5a0 comp rx=0 tx=0).stop 2024-08-03T20:50:55.326 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:55.324+0000 7f86c8000640 1 -- 172.21.15.32:0/1424726153 shutdown_connections 2024-08-03T20:50:55.326 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:55.324+0000 7f86c8000640 1 --2- 172.21.15.32:0/1424726153 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f86c00cf9c0 0x7f86c00d0a70 unknown :-1 s=CLOSED pgs=647 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:55.326 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:55.324+0000 7f86c8000640 1 -- 172.21.15.32:0/1424726153 >> 172.21.15.32:0/1424726153 conn(0x7f86c009d660 msgr2=0x7f86c00d4330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:50:55.326 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:55.324+0000 7f86c8000640 1 -- 172.21.15.32:0/1424726153 shutdown_connections 2024-08-03T20:50:55.327 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:55.324+0000 7f86c8000640 1 -- 172.21.15.32:0/1424726153 wait complete. 2024-08-03T20:50:55.328 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:55.326+0000 7f86c8000640 1 Processor -- start 2024-08-03T20:50:55.328 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:55.327+0000 7f86c8000640 1 -- start start 2024-08-03T20:50:55.332 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:55.328+0000 7f86c8000640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f86c00b59f0 0x7f86c0158100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:50:55.332 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:55.328+0000 7f86c8000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f86c00cf9c0 0x7f86c015dae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:50:55.332 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:55.328+0000 7f86c8000640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f86c00d0fb0 0x7f86c015e020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:50:55.332 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:55.328+0000 7f86c8000640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f86c00b8a80 con 0x7f86c00cf9c0 2024-08-03T20:50:55.332 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:55.328+0000 7f86c8000640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f86c00b8900 con 0x7f86c00d0fb0 2024-08-03T20:50:55.332 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:55.328+0000 7f86c8000640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7f86c00b8c00 con 0x7f86c00b59f0 2024-08-03T20:50:55.332 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:55.328+0000 7f86c4c00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f86c00cf9c0 0x7f86c015dae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:50:55.332 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:55.328+0000 7f86c4c00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f86c00cf9c0 0x7f86c015dae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:35176/0 (socket says 172.21.15.32:35176) 2024-08-03T20:50:55.332 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:55.328+0000 7f86c4c00640 1 -- 172.21.15.32:0/3362093231 learned_addr learned my addr 172.21.15.32:0/3362093231 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:50:55.333 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:55.328+0000 7f86c6000640 1 --2- 172.21.15.32:0/3362093231 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f86c00d0fb0 0x7f86c015e020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:50:55.333 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:55.328+0000 7f86c5600640 1 --2- 172.21.15.32:0/3362093231 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f86c00b59f0 0x7f86c0158100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:50:55.333 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:55.329+0000 7f86c4c00640 1 -- 172.21.15.32:0/3362093231 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f86c00b59f0 msgr2=0x7f86c0158100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:50:55.333 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:55.329+0000 7f86c4c00640 1 --2- 172.21.15.32:0/3362093231 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f86c00b59f0 0x7f86c0158100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:55.333 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:55.329+0000 7f86c4c00640 1 -- 172.21.15.32:0/3362093231 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f86c00d0fb0 msgr2=0x7f86c015e020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:50:55.333 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:55.329+0000 7f86c4c00640 1 --2- 172.21.15.32:0/3362093231 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f86c00d0fb0 0x7f86c015e020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:55.333 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:55.329+0000 7f86c4c00640 1 -- 172.21.15.32:0/3362093231 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f86c01f62d0 con 0x7f86c00cf9c0 2024-08-03T20:50:55.333 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:55.329+0000 7f86c6000640 1 --2- 172.21.15.32:0/3362093231 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f86c00d0fb0 0x7f86c015e020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-03T20:50:55.333 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:55.329+0000 7f86c5600640 1 --2- 172.21.15.32:0/3362093231 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f86c00b59f0 0x7f86c0158100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-03T20:50:55.333 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:55.329+0000 7f86c4c00640 1 --2- 172.21.15.32:0/3362093231 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f86c00cf9c0 0x7f86c015dae0 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7f86ac02ab90 tx=0x7f86ac033c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:50:55.333 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:55.329+0000 7f86bd600640 1 -- 172.21.15.32:0/3362093231 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f86ac005c90 con 0x7f86c00cf9c0 2024-08-03T20:50:55.333 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:55.329+0000 7f86bd600640 1 -- 172.21.15.32:0/3362093231 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f86ac031b60 con 0x7f86c00cf9c0 2024-08-03T20:50:55.333 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:55.329+0000 7f86bd600640 1 -- 172.21.15.32:0/3362093231 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f86ac005020 con 0x7f86c00cf9c0 2024-08-03T20:50:55.334 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:55.329+0000 7f86c8000640 1 -- 172.21.15.32:0/3362093231 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f86c00b2d30 con 0x7f86c00cf9c0 2024-08-03T20:50:55.334 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:55.330+0000 7f86c8000640 1 -- 172.21.15.32:0/3362093231 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f86c00b32f0 con 0x7f86c00cf9c0 2024-08-03T20:50:55.334 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:55.330+0000 7f86c8000640 1 -- 172.21.15.32:0/3362093231 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f86c01f6040 con 0x7f86c00cf9c0 2024-08-03T20:50:55.334 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:55.332+0000 7f86bd600640 1 -- 172.21.15.32:0/3362093231 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f86ac045020 con 0x7f86c00cf9c0 2024-08-03T20:50:55.334 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:55.333+0000 7f86bd600640 1 --2- 172.21.15.32:0/3362093231 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f86a0077040 0x7f86a0079550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:50:55.334 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:55.333+0000 7f86bd600640 1 -- 172.21.15.32:0/3362093231 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(58..58 src has 1..58) ==== 5795+0+0 (secure 0 0 0) 0x7f86ac0b91b0 con 0x7f86c00cf9c0 2024-08-03T20:50:55.335 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:55.333+0000 7f86c5600640 1 --2- 172.21.15.32:0/3362093231 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f86a0077040 0x7f86a0079550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:50:55.335 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:55.334+0000 7f86c5600640 1 --2- 172.21.15.32:0/3362093231 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f86a0077040 0x7f86a0079550 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f86b40013f0 tx=0x7f86b4009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:50:55.339 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:55.337+0000 7f86bd600640 1 -- 172.21.15.32:0/3362093231 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7f86ac031770 con 0x7f86c00cf9c0 2024-08-03T20:50:55.516 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:55.515+0000 7f3aaf400640 1 -- 172.21.15.32:0/294378650 >> [v2:172.21.15.32:6834/2963295201,v1:172.21.15.32:6835/2963295201] conn(0x55ca2c6281f0 msgr2=0x55ca2c676ad0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.32:6834/2963295201 2024-08-03T20:50:55.516 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:55.515+0000 7f3a8fe00640 1 client.6138 ms_handle_refused on v2:172.21.15.32:6834/2963295201 2024-08-03T20:50:55.516 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:55.515+0000 7f3aaf400640 1 --2- 172.21.15.32:0/294378650 >> [v2:172.21.15.32:6834/2963295201,v1:172.21.15.32:6835/2963295201] conn(0x55ca2c6281f0 0x55ca2c676ad0 unknown :-1 s=START_CONNECT pgs=14 cs=3 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-08-03T20:50:55.585 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:55.584+0000 7f86c8000640 1 -- 172.21.15.32:0/3362093231 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f86c00d0a70 con 0x7f86c00cf9c0 2024-08-03T20:50:55.586 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:55.584+0000 7f86bd600640 1 -- 172.21.15.32:0/3362093231 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 59 v59) ==== 94+0+5230 (secure 0 0 0) 0x7f86ac081c60 con 0x7f86c00cf9c0 2024-08-03T20:50:55.586 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-03T20:50:55.586 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":59,"btime":"2024-08-03T20:50:54:733889+0000","default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5261,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6833/2498840646","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":2498840646},{"type":"v1","addr":"172.21.15.32:6833","nonce":2498840646}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":45},{"gid":5276,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6837/3142148286","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6835","nonce":3142148286},{"type":"v1","addr":"172.21.15.118:6837","nonce":3142148286}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":46},{"gid":5632,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6836/3720010453","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6834","nonce":3720010453},{"type":"v1","addr":"172.21.15.118:6836","nonce":3720010453}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":46}],"filesystems":[{"mdsmap":{"epoch":59,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-08-03T20:50:27.688698+0000","modified":"2024-08-03T20:50:53.873236+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":58,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6045},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6045":{"gid":6045,"name":"c","rank":0,"incarnation":55,"state":"up:active","state_seq":5,"addr":"172.21.15.32:6835/2963295201","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":2963295201},{"type":"v1","addr":"172.21.15.32:6835","nonce":2963295201}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[9],"metadata_pool":8,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6045,"qdb_cluster":[6045]},"id":4}]} 2024-08-03T20:50:55.586 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 59 2024-08-03T20:50:55.589 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:55.587+0000 7f86c8000640 1 -- 172.21.15.32:0/3362093231 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f86a0077040 msgr2=0x7f86a0079550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:50:55.589 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:55.587+0000 7f86c8000640 1 --2- 172.21.15.32:0/3362093231 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f86a0077040 0x7f86a0079550 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f86b40013f0 tx=0x7f86b4009040 comp rx=0 tx=0).stop 2024-08-03T20:50:55.589 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:55.587+0000 7f86c8000640 1 -- 172.21.15.32:0/3362093231 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f86c00cf9c0 msgr2=0x7f86c015dae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:50:55.589 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:55.587+0000 7f86c8000640 1 --2- 172.21.15.32:0/3362093231 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f86c00cf9c0 0x7f86c015dae0 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7f86ac02ab90 tx=0x7f86ac033c20 comp rx=0 tx=0).stop 2024-08-03T20:50:55.589 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:55.587+0000 7f86c8000640 1 -- 172.21.15.32:0/3362093231 shutdown_connections 2024-08-03T20:50:55.589 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:55.587+0000 7f86c8000640 1 --2- 172.21.15.32:0/3362093231 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f86a0077040 0x7f86a0079550 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:55.589 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:55.587+0000 7f86c8000640 1 --2- 172.21.15.32:0/3362093231 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f86c00d0fb0 0x7f86c015e020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:55.589 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:55.587+0000 7f86c8000640 1 --2- 172.21.15.32:0/3362093231 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f86c00cf9c0 0x7f86c015dae0 unknown :-1 s=CLOSED pgs=648 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:55.589 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:55.587+0000 7f86c8000640 1 --2- 172.21.15.32:0/3362093231 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f86c00b59f0 0x7f86c0158100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:50:55.589 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:55.587+0000 7f86c8000640 1 -- 172.21.15.32:0/3362093231 >> 172.21.15.32:0/3362093231 conn(0x7f86c009d660 msgr2=0x7f86c00d4cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:50:55.589 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:55.587+0000 7f86c8000640 1 -- 172.21.15.32:0/3362093231 shutdown_connections 2024-08-03T20:50:55.589 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:55.587+0000 7f86c8000640 1 -- 172.21.15.32:0/3362093231 wait complete. 2024-08-03T20:50:55.611 DEBUG:tasks.ceph_test_case:wait_until_true: waiting (timeout=45.0 retry_count=0)... 2024-08-03T20:50:55.730 INFO:tasks.ceph.mds.c.smithi032.stderr:daemon-helper: command crashed with signal 6 2024-08-03T20:50:55.743 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:55.742+0000 7fb222a00640 1 -- 172.21.15.32:0/2804832187 <== mon.0 v2:172.21.15.32:3300/0 8 ==== log(2 entries from seq 677 at 2024-08-03T20:50:54.740686+0000) ==== 635+0+0 (secure 0 0 0) 0x7fb21805dca0 con 0x7fb22c0cf770 2024-08-03T20:50:55.743 INFO:teuthology.orchestra.run.smithi032.stdout: cluster: 2024-08-03T20:50:55.743 INFO:teuthology.orchestra.run.smithi032.stdout: id: 40ab141a-01d7-49b9-ae3c-2edec9876e95 2024-08-03T20:50:55.743 INFO:teuthology.orchestra.run.smithi032.stdout: health: HEALTH_OK 2024-08-03T20:50:55.743 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-03T20:50:55.743 INFO:teuthology.orchestra.run.smithi032.stdout: services: 2024-08-03T20:50:55.744 INFO:teuthology.orchestra.run.smithi032.stdout: mon: 3 daemons, quorum a,b,c (age 3m) 2024-08-03T20:50:55.744 INFO:teuthology.orchestra.run.smithi032.stdout: mgr: x(active, since 3m), standbys: y 2024-08-03T20:50:55.744 INFO:teuthology.orchestra.run.smithi032.stdout: mds: 1/1 daemons up, 3 standby 2024-08-03T20:50:55.744 INFO:teuthology.orchestra.run.smithi032.stdout: osd: 8 osds: 8 up (since 3m), 8 in (since 3m) 2024-08-03T20:50:55.744 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-03T20:50:55.744 INFO:teuthology.orchestra.run.smithi032.stdout: data: 2024-08-03T20:50:55.744 INFO:teuthology.orchestra.run.smithi032.stdout: volumes: 1/1 healthy 2024-08-03T20:50:55.744 INFO:teuthology.orchestra.run.smithi032.stdout: pools: 3 pools, 97 pgs 2024-08-03T20:50:55.744 INFO:teuthology.orchestra.run.smithi032.stdout: objects: 29 objects, 602 KiB 2024-08-03T20:50:55.744 INFO:teuthology.orchestra.run.smithi032.stdout: usage: 250 MiB used, 720 GiB / 720 GiB avail 2024-08-03T20:50:55.744 INFO:teuthology.orchestra.run.smithi032.stdout: pgs: 97 active+clean 2024-08-03T20:50:55.744 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-03T20:50:55.744 INFO:teuthology.orchestra.run.smithi032.stdout: io: 2024-08-03T20:50:55.744 INFO:teuthology.orchestra.run.smithi032.stdout: client: 2.9 KiB/s rd, 682 B/s wr, 1 op/s rd, 0 op/s wr 2024-08-03T20:50:55.745 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-03T20:50:55.745 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-03T20:50:55.745 INFO:teuthology.orchestra.run.smithi032.stdout:2024-08-03T20:50:55.072966+0000 mds.c [ERR] MDS abort because newly corrupt dentry to be committed: [dentry #0x1/a [fffffffffffffff6,head] auth (dversion lock) v=13 ino=0x10000000000 state=1610612736 | inodepin=1 dirty=1 0x5559d62e7180] 2024-08-03T20:50:55.917 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:55.916+0000 7f3aaf400640 1 -- 172.21.15.32:0/294378650 >> [v2:172.21.15.32:6834/2963295201,v1:172.21.15.32:6835/2963295201] conn(0x55ca2c6281f0 msgr2=0x55ca2c676ad0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.32:6834/2963295201 2024-08-03T20:50:55.917 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:55.916+0000 7f3aaf400640 1 --2- 172.21.15.32:0/294378650 >> [v2:172.21.15.32:6834/2963295201,v1:172.21.15.32:6835/2963295201] conn(0x55ca2c6281f0 0x55ca2c676ad0 unknown :-1 s=START_CONNECT pgs=14 cs=4 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-08-03T20:50:55.917 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:55.916+0000 7f3a8fe00640 1 client.6138 ms_handle_refused on v2:172.21.15.32:6834/2963295201 2024-08-03T20:50:56.718 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:56.717+0000 7f3aaf400640 1 -- 172.21.15.32:0/294378650 >> [v2:172.21.15.32:6834/2963295201,v1:172.21.15.32:6835/2963295201] conn(0x55ca2c6281f0 msgr2=0x55ca2c676ad0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.32:6834/2963295201 2024-08-03T20:50:56.718 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:56.717+0000 7f3a8fe00640 1 client.6138 ms_handle_refused on v2:172.21.15.32:6834/2963295201 2024-08-03T20:50:56.718 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:56.717+0000 7f3aaf400640 1 --2- 172.21.15.32:0/294378650 >> [v2:172.21.15.32:6834/2963295201,v1:172.21.15.32:6835/2963295201] conn(0x55ca2c6281f0 0x55ca2c676ad0 unknown :-1 s=START_CONNECT pgs=14 cs=5 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-08-03T20:50:57.147 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.c is failed for ~0s 2024-08-03T20:50:58.320 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:58.319+0000 7f3aaf400640 1 -- 172.21.15.32:0/294378650 >> [v2:172.21.15.32:6834/2963295201,v1:172.21.15.32:6835/2963295201] conn(0x55ca2c6281f0 msgr2=0x55ca2c676ad0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.32:6834/2963295201 2024-08-03T20:50:58.320 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:58.319+0000 7f3a8fe00640 1 client.6138 ms_handle_refused on v2:172.21.15.32:6834/2963295201 2024-08-03T20:50:58.320 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:50:58.319+0000 7f3aaf400640 1 --2- 172.21.15.32:0/294378650 >> [v2:172.21.15.32:6834/2963295201,v1:172.21.15.32:6835/2963295201] conn(0x55ca2c6281f0 0x55ca2c676ad0 unknown :-1 s=START_CONNECT pgs=14 cs=6 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-08-03T20:51:00.612 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-08-03T20:51:00.853 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:00.851+0000 7fafd9600640 1 -- 172.21.15.32:0/1109571796 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fafd4073ea0 msgr2=0x7fafd40742b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:51:00.853 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:00.851+0000 7fafd9600640 1 --2- 172.21.15.32:0/1109571796 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fafd4073ea0 0x7fafd40742b0 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7fafd40cd480 tx=0x7fafcc02a8b0 comp rx=0 tx=0).stop 2024-08-03T20:51:00.853 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:00.851+0000 7fafd9600640 1 -- 172.21.15.32:0/1109571796 shutdown_connections 2024-08-03T20:51:00.853 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:00.851+0000 7fafd9600640 1 --2- 172.21.15.32:0/1109571796 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fafd4073ea0 0x7fafd40742b0 unknown :-1 s=CLOSED pgs=650 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:51:00.853 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:00.851+0000 7fafd9600640 1 --2- 172.21.15.32:0/1109571796 >> v2:172.21.15.118:3300/0 conn(0x7fafd40cfd30 0x7fafd40d0140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:51:00.853 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:00.851+0000 7fafd9600640 1 -- 172.21.15.32:0/1109571796 >> 172.21.15.32:0/1109571796 conn(0x7fafd40b34b0 msgr2=0x7fafd40d1050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:51:00.854 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:00.851+0000 7fafd9600640 1 -- 172.21.15.32:0/1109571796 shutdown_connections 2024-08-03T20:51:00.854 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:00.851+0000 7fafd9600640 1 -- 172.21.15.32:0/1109571796 wait complete. 2024-08-03T20:51:00.855 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:00.853+0000 7fafd9600640 1 Processor -- start 2024-08-03T20:51:00.855 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:00.853+0000 7fafd9600640 1 -- start start 2024-08-03T20:51:00.858 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:00.855+0000 7fafd9600640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fafd4073ea0 0x7fafd40acc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:51:00.858 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:00.855+0000 7fafd9600640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fafd40cfd30 0x7fafd415edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:51:00.858 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:00.855+0000 7fafd9600640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fafd415ffa0 0x7fafd40afa00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:51:00.858 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:00.855+0000 7fafd9600640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fafd406cd70 con 0x7fafd4073ea0 2024-08-03T20:51:00.858 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:00.855+0000 7fafd9600640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7fafd406cbf0 con 0x7fafd40cfd30 2024-08-03T20:51:00.858 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:00.855+0000 7fafd9600640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7fafd406cef0 con 0x7fafd415ffa0 2024-08-03T20:51:00.858 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:00.855+0000 7fafd2a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fafd4073ea0 0x7fafd40acc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:51:00.859 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:00.855+0000 7fafd2a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fafd4073ea0 0x7fafd40acc70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:39686/0 (socket says 172.21.15.32:39686) 2024-08-03T20:51:00.859 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:00.855+0000 7fafd2a00640 1 -- 172.21.15.32:0/4040425517 learned_addr learned my addr 172.21.15.32:0/4040425517 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:51:00.859 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:00.856+0000 7fafcbe00640 1 --2- 172.21.15.32:0/4040425517 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fafd40cfd30 0x7fafd415edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:51:00.859 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:00.856+0000 7fafd3400640 1 --2- 172.21.15.32:0/4040425517 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fafd415ffa0 0x7fafd40afa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:51:00.859 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:00.856+0000 7fafd2a00640 1 -- 172.21.15.32:0/4040425517 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fafd415ffa0 msgr2=0x7fafd40afa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:51:00.859 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:00.856+0000 7fafd2a00640 1 --2- 172.21.15.32:0/4040425517 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fafd415ffa0 0x7fafd40afa00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:51:00.859 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:00.856+0000 7fafd2a00640 1 -- 172.21.15.32:0/4040425517 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fafd40cfd30 msgr2=0x7fafd415edb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:51:00.859 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:00.856+0000 7fafd2a00640 1 --2- 172.21.15.32:0/4040425517 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fafd40cfd30 0x7fafd415edb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:51:00.859 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:00.856+0000 7fafd2a00640 1 -- 172.21.15.32:0/4040425517 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fafd41f4550 con 0x7fafd4073ea0 2024-08-03T20:51:00.859 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:00.856+0000 7fafd3400640 1 --2- 172.21.15.32:0/4040425517 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fafd415ffa0 0x7fafd40afa00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-03T20:51:00.859 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:00.856+0000 7fafd2a00640 1 --2- 172.21.15.32:0/4040425517 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fafd4073ea0 0x7fafd40acc70 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7fafc400c3a0 tx=0x7fafc400c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:51:00.859 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:00.856+0000 7fafcaa00640 1 -- 172.21.15.32:0/4040425517 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fafc4011790 con 0x7fafd4073ea0 2024-08-03T20:51:00.860 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:00.856+0000 7fafcaa00640 1 -- 172.21.15.32:0/4040425517 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fafc4017070 con 0x7fafd4073ea0 2024-08-03T20:51:00.860 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:00.856+0000 7fafd9600640 1 -- 172.21.15.32:0/4040425517 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fafd40aec40 con 0x7fafd4073ea0 2024-08-03T20:51:00.860 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:00.856+0000 7fafcaa00640 1 -- 172.21.15.32:0/4040425517 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fafc4011c80 con 0x7fafd4073ea0 2024-08-03T20:51:00.860 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:00.856+0000 7fafd9600640 1 -- 172.21.15.32:0/4040425517 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fafd40af200 con 0x7fafd4073ea0 2024-08-03T20:51:00.860 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:00.858+0000 7fafd9600640 1 -- 172.21.15.32:0/4040425517 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fafd40742b0 con 0x7fafd4073ea0 2024-08-03T20:51:00.860 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:00.859+0000 7fafcaa00640 1 -- 172.21.15.32:0/4040425517 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7fafc401c460 con 0x7fafd4073ea0 2024-08-03T20:51:00.861 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:00.860+0000 7fafcaa00640 1 --2- 172.21.15.32:0/4040425517 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fafb0077040 0x7fafb0079550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:51:00.861 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:00.860+0000 7fafcaa00640 1 -- 172.21.15.32:0/4040425517 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(58..58 src has 1..58) ==== 5795+0+0 (secure 0 0 0) 0x7fafc4098710 con 0x7fafd4073ea0 2024-08-03T20:51:00.861 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:00.860+0000 7fafcbe00640 1 --2- 172.21.15.32:0/4040425517 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fafb0077040 0x7fafb0079550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:51:00.862 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:00.861+0000 7fafcbe00640 1 --2- 172.21.15.32:0/4040425517 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fafb0077040 0x7fafb0079550 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7fafbc005fd0 tx=0x7fafbc005850 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:51:00.865 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:00.864+0000 7fafcaa00640 1 -- 172.21.15.32:0/4040425517 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7fafc4061930 con 0x7fafd4073ea0 2024-08-03T20:51:01.109 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:01.108+0000 7fafd9600640 1 -- 172.21.15.32:0/4040425517 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fafd4069090 con 0x7fafd4073ea0 2024-08-03T20:51:01.110 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:01.109+0000 7fafcaa00640 1 -- 172.21.15.32:0/4040425517 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 59 v59) ==== 94+0+5230 (secure 0 0 0) 0x7fafc4061230 con 0x7fafd4073ea0 2024-08-03T20:51:01.110 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-03T20:51:01.111 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":59,"btime":"2024-08-03T20:50:54:733889+0000","default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5261,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6833/2498840646","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":2498840646},{"type":"v1","addr":"172.21.15.32:6833","nonce":2498840646}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":45},{"gid":5276,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6837/3142148286","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6835","nonce":3142148286},{"type":"v1","addr":"172.21.15.118:6837","nonce":3142148286}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":46},{"gid":5632,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6836/3720010453","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6834","nonce":3720010453},{"type":"v1","addr":"172.21.15.118:6836","nonce":3720010453}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":46}],"filesystems":[{"mdsmap":{"epoch":59,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-08-03T20:50:27.688698+0000","modified":"2024-08-03T20:50:53.873236+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":58,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6045},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6045":{"gid":6045,"name":"c","rank":0,"incarnation":55,"state":"up:active","state_seq":5,"addr":"172.21.15.32:6835/2963295201","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":2963295201},{"type":"v1","addr":"172.21.15.32:6835","nonce":2963295201}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[9],"metadata_pool":8,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6045,"qdb_cluster":[6045]},"id":4}]} 2024-08-03T20:51:01.111 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 59 2024-08-03T20:51:01.113 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:01.111+0000 7fafd9600640 1 -- 172.21.15.32:0/4040425517 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fafb0077040 msgr2=0x7fafb0079550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:51:01.113 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:01.111+0000 7fafd9600640 1 --2- 172.21.15.32:0/4040425517 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fafb0077040 0x7fafb0079550 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7fafbc005fd0 tx=0x7fafbc005850 comp rx=0 tx=0).stop 2024-08-03T20:51:01.113 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:01.111+0000 7fafd9600640 1 -- 172.21.15.32:0/4040425517 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fafd4073ea0 msgr2=0x7fafd40acc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:51:01.113 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:01.111+0000 7fafd9600640 1 --2- 172.21.15.32:0/4040425517 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fafd4073ea0 0x7fafd40acc70 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7fafc400c3a0 tx=0x7fafc400c860 comp rx=0 tx=0).stop 2024-08-03T20:51:01.113 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:01.111+0000 7fafd9600640 1 -- 172.21.15.32:0/4040425517 shutdown_connections 2024-08-03T20:51:01.113 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:01.111+0000 7fafd9600640 1 --2- 172.21.15.32:0/4040425517 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fafb0077040 0x7fafb0079550 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:51:01.113 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:01.111+0000 7fafd9600640 1 --2- 172.21.15.32:0/4040425517 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fafd40cfd30 0x7fafd415edb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:51:01.113 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:01.111+0000 7fafd9600640 1 --2- 172.21.15.32:0/4040425517 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fafd415ffa0 0x7fafd40afa00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:51:01.113 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:01.111+0000 7fafd9600640 1 --2- 172.21.15.32:0/4040425517 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fafd4073ea0 0x7fafd40acc70 unknown :-1 s=CLOSED pgs=651 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:51:01.113 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:01.111+0000 7fafd9600640 1 -- 172.21.15.32:0/4040425517 >> 172.21.15.32:0/4040425517 conn(0x7fafd40b34b0 msgr2=0x7fafd41417e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:51:01.113 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:01.111+0000 7fafd9600640 1 -- 172.21.15.32:0/4040425517 shutdown_connections 2024-08-03T20:51:01.114 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:01.111+0000 7fafd9600640 1 -- 172.21.15.32:0/4040425517 wait complete. 2024-08-03T20:51:01.135 DEBUG:tasks.ceph_test_case:wait_until_true: waiting (timeout=45.0 retry_count=0)... 2024-08-03T20:51:01.524 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:01.522+0000 7f3aaf400640 1 -- 172.21.15.32:0/294378650 >> [v2:172.21.15.32:6834/2963295201,v1:172.21.15.32:6835/2963295201] conn(0x55ca2c6281f0 msgr2=0x55ca2c676ad0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.32:6834/2963295201 2024-08-03T20:51:01.524 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:01.522+0000 7f3aaf400640 1 --2- 172.21.15.32:0/294378650 >> [v2:172.21.15.32:6834/2963295201,v1:172.21.15.32:6835/2963295201] conn(0x55ca2c6281f0 0x55ca2c676ad0 unknown :-1 s=START_CONNECT pgs=14 cs=7 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 6.400000 2024-08-03T20:51:01.524 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:01.522+0000 7f3a8fe00640 1 client.6138 ms_handle_refused on v2:172.21.15.32:6834/2963295201 2024-08-03T20:51:03.760 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.c is failed for ~7s 2024-08-03T20:51:06.136 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-08-03T20:51:06.378 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:06.375+0000 7f2727800640 1 -- 172.21.15.32:0/3082835693 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f27200b4ce0 msgr2=0x7f27200cdf70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:51:06.378 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:06.375+0000 7f2727800640 1 --2- 172.21.15.32:0/3082835693 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f27200b4ce0 0x7f27200cdf70 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f2718005fd0 tx=0x7f271802a8b0 comp rx=0 tx=0).stop 2024-08-03T20:51:06.378 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:06.375+0000 7f2727800640 1 -- 172.21.15.32:0/3082835693 shutdown_connections 2024-08-03T20:51:06.378 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:06.375+0000 7f2727800640 1 --2- 172.21.15.32:0/3082835693 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f27200b4ce0 0x7f27200cdf70 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:51:06.378 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:06.375+0000 7f2727800640 1 --2- 172.21.15.32:0/3082835693 >> v2:172.21.15.118:3300/0 conn(0x7f272015f610 0x7f27201523f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:51:06.378 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:06.375+0000 7f2727800640 1 -- 172.21.15.32:0/3082835693 >> 172.21.15.32:0/3082835693 conn(0x7f27200b1bd0 msgr2=0x7f27200b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:51:06.378 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:06.375+0000 7f2727800640 1 -- 172.21.15.32:0/3082835693 shutdown_connections 2024-08-03T20:51:06.378 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:06.376+0000 7f2727800640 1 -- 172.21.15.32:0/3082835693 wait complete. 2024-08-03T20:51:06.379 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:06.378+0000 7f2727800640 1 Processor -- start 2024-08-03T20:51:06.379 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:06.378+0000 7f2727800640 1 -- start start 2024-08-03T20:51:06.383 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:06.380+0000 7f2727800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f27200b38b0 0x7f27201f3480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:51:06.383 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:06.380+0000 7f2727800640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f27200b4ce0 0x7f27200aaaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:51:06.383 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:06.380+0000 7f2727800640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f2720152a30 0x7f2720152e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:51:06.383 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:06.380+0000 7f2727800640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f27200cea90 con 0x7f27200b38b0 2024-08-03T20:51:06.383 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:06.380+0000 7f2727800640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f27200ce910 con 0x7f2720152a30 2024-08-03T20:51:06.384 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:06.380+0000 7f2727800640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7f27200cec10 con 0x7f27200b4ce0 2024-08-03T20:51:06.384 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:06.380+0000 7f2724e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f27200b38b0 0x7f27201f3480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:51:06.384 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:06.380+0000 7f2724e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f27200b38b0 0x7f27201f3480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:39694/0 (socket says 172.21.15.32:39694) 2024-08-03T20:51:06.384 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:06.380+0000 7f2724e00640 1 -- 172.21.15.32:0/1609611394 learned_addr learned my addr 172.21.15.32:0/1609611394 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:51:06.384 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:06.380+0000 7f271fe00640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f27200b4ce0 0x7f27200aaaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:51:06.384 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:06.380+0000 7f2725800640 1 --2- 172.21.15.32:0/1609611394 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f2720152a30 0x7f2720152e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:51:06.384 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:06.380+0000 7f2724e00640 1 -- 172.21.15.32:0/1609611394 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f27200b4ce0 msgr2=0x7f27200aaaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:51:06.384 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:06.380+0000 7f2724e00640 1 --2- 172.21.15.32:0/1609611394 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f27200b4ce0 0x7f27200aaaa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:51:06.385 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:06.380+0000 7f2724e00640 1 -- 172.21.15.32:0/1609611394 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f2720152a30 msgr2=0x7f2720152e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:51:06.385 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:06.380+0000 7f2724e00640 1 --2- 172.21.15.32:0/1609611394 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f2720152a30 0x7f2720152e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:51:06.385 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:06.381+0000 7f2724e00640 1 -- 172.21.15.32:0/1609611394 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f27201f2e90 con 0x7f27200b38b0 2024-08-03T20:51:06.385 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:06.381+0000 7f2724e00640 1 --2- 172.21.15.32:0/1609611394 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f27200b38b0 0x7f27201f3480 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7f271400c3a0 tx=0x7f271400c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:51:06.385 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:06.381+0000 7f271cc00640 1 -- 172.21.15.32:0/1609611394 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2714017070 con 0x7f27200b38b0 2024-08-03T20:51:06.385 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:06.381+0000 7f271cc00640 1 -- 172.21.15.32:0/1609611394 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f27140053c0 con 0x7f27200b38b0 2024-08-03T20:51:06.385 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:06.381+0000 7f2727800640 1 -- 172.21.15.32:0/1609611394 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2720153380 con 0x7f27200b38b0 2024-08-03T20:51:06.386 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:06.381+0000 7f271cc00640 1 -- 172.21.15.32:0/1609611394 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2714003850 con 0x7f27200b38b0 2024-08-03T20:51:06.386 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:06.381+0000 7f2727800640 1 -- 172.21.15.32:0/1609611394 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f27200ab960 con 0x7f27200b38b0 2024-08-03T20:51:06.387 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:06.382+0000 7f2727800640 1 -- 172.21.15.32:0/1609611394 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f27200cf130 con 0x7f27200b38b0 2024-08-03T20:51:06.387 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:06.383+0000 7f271cc00640 1 -- 172.21.15.32:0/1609611394 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f271401a070 con 0x7f27200b38b0 2024-08-03T20:51:06.388 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:06.384+0000 7f271cc00640 1 --2- 172.21.15.32:0/1609611394 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f2700077010 0x7f2700079520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:51:06.388 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:06.384+0000 7f271cc00640 1 -- 172.21.15.32:0/1609611394 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(58..58 src has 1..58) ==== 5795+0+0 (secure 0 0 0) 0x7f2714097d50 con 0x7f27200b38b0 2024-08-03T20:51:06.388 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:06.385+0000 7f271fe00640 1 --2- 172.21.15.32:0/1609611394 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f2700077010 0x7f2700079520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:51:06.388 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:06.386+0000 7f271fe00640 1 --2- 172.21.15.32:0/1609611394 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f2700077010 0x7f2700079520 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f270c0096c0 tx=0x7f270c009260 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:51:06.390 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:06.389+0000 7f271cc00640 1 -- 172.21.15.32:0/1609611394 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7f2720153380 con 0x7f27200b38b0 2024-08-03T20:51:06.632 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:06.631+0000 7f2727800640 1 -- 172.21.15.32:0/1609611394 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f27201f39c0 con 0x7f27200b38b0 2024-08-03T20:51:06.633 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:06.631+0000 7f271cc00640 1 -- 172.21.15.32:0/1609611394 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 59 v59) ==== 94+0+5230 (secure 0 0 0) 0x7f27201f2e90 con 0x7f27200b38b0 2024-08-03T20:51:06.633 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-03T20:51:06.633 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":59,"btime":"2024-08-03T20:50:54:733889+0000","default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5261,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6833/2498840646","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":2498840646},{"type":"v1","addr":"172.21.15.32:6833","nonce":2498840646}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":45},{"gid":5276,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6837/3142148286","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6835","nonce":3142148286},{"type":"v1","addr":"172.21.15.118:6837","nonce":3142148286}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":46},{"gid":5632,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6836/3720010453","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6834","nonce":3720010453},{"type":"v1","addr":"172.21.15.118:6836","nonce":3720010453}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":46}],"filesystems":[{"mdsmap":{"epoch":59,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-08-03T20:50:27.688698+0000","modified":"2024-08-03T20:50:53.873236+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":58,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6045},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6045":{"gid":6045,"name":"c","rank":0,"incarnation":55,"state":"up:active","state_seq":5,"addr":"172.21.15.32:6835/2963295201","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":2963295201},{"type":"v1","addr":"172.21.15.32:6835","nonce":2963295201}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[9],"metadata_pool":8,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6045,"qdb_cluster":[6045]},"id":4}]} 2024-08-03T20:51:06.633 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 59 2024-08-03T20:51:06.636 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:06.634+0000 7f2727800640 1 -- 172.21.15.32:0/1609611394 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f2700077010 msgr2=0x7f2700079520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:51:06.636 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:06.634+0000 7f2727800640 1 --2- 172.21.15.32:0/1609611394 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f2700077010 0x7f2700079520 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f270c0096c0 tx=0x7f270c009260 comp rx=0 tx=0).stop 2024-08-03T20:51:06.636 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:06.634+0000 7f2727800640 1 -- 172.21.15.32:0/1609611394 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f27200b38b0 msgr2=0x7f27201f3480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:51:06.636 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:06.634+0000 7f2727800640 1 --2- 172.21.15.32:0/1609611394 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f27200b38b0 0x7f27201f3480 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7f271400c3a0 tx=0x7f271400c860 comp rx=0 tx=0).stop 2024-08-03T20:51:06.636 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:06.634+0000 7f2727800640 1 -- 172.21.15.32:0/1609611394 shutdown_connections 2024-08-03T20:51:06.636 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:06.634+0000 7f2727800640 1 --2- 172.21.15.32:0/1609611394 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f2700077010 0x7f2700079520 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:51:06.636 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:06.634+0000 7f2727800640 1 --2- 172.21.15.32:0/1609611394 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f2720152a30 0x7f2720152e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:51:06.636 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:06.634+0000 7f2727800640 1 --2- 172.21.15.32:0/1609611394 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f27200b4ce0 0x7f27200aaaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:51:06.636 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:06.634+0000 7f2727800640 1 --2- 172.21.15.32:0/1609611394 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f27200b38b0 0x7f27201f3480 unknown :-1 s=CLOSED pgs=652 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:51:06.636 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:06.634+0000 7f2727800640 1 -- 172.21.15.32:0/1609611394 >> 172.21.15.32:0/1609611394 conn(0x7f27200b1bd0 msgr2=0x7f27201520a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:51:06.636 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:06.634+0000 7f2727800640 1 -- 172.21.15.32:0/1609611394 shutdown_connections 2024-08-03T20:51:06.637 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:06.634+0000 7f2727800640 1 -- 172.21.15.32:0/1609611394 wait complete. 2024-08-03T20:51:06.661 DEBUG:tasks.ceph_test_case:wait_until_true: waiting (timeout=45.0 retry_count=0)... 2024-08-03T20:51:07.011 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-08-03T20:51:07.016 DEBUG:teuthology.orchestra.run.smithi118:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-08-03T20:51:07.930 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:07.929+0000 7f3aaf400640 1 -- 172.21.15.32:0/294378650 >> [v2:172.21.15.32:6834/2963295201,v1:172.21.15.32:6835/2963295201] conn(0x55ca2c6281f0 msgr2=0x55ca2c676ad0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.32:6834/2963295201 2024-08-03T20:51:07.931 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:07.929+0000 7f3aaf400640 1 --2- 172.21.15.32:0/294378650 >> [v2:172.21.15.32:6834/2963295201,v1:172.21.15.32:6835/2963295201] conn(0x55ca2c6281f0 0x55ca2c676ad0 unknown :-1 s=START_CONNECT pgs=14 cs=8 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 12.800000 2024-08-03T20:51:07.931 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:07.929+0000 7f3a8fe00640 1 client.6138 ms_handle_refused on v2:172.21.15.32:6834/2963295201 2024-08-03T20:51:10.372 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.c is failed for ~13s 2024-08-03T20:51:11.663 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-08-03T20:51:11.903 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:11.901+0000 7f1a45400640 1 -- 172.21.15.32:0/3547511875 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1a400d4bb0 msgr2=0x7f1a400d1c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:51:11.903 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:11.901+0000 7f1a45400640 1 --2- 172.21.15.32:0/3547511875 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1a400d4bb0 0x7f1a400d1c60 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7f1a30005fd0 tx=0x7f1a3002a4a0 comp rx=0 tx=0).stop 2024-08-03T20:51:11.903 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:11.901+0000 7f1a45400640 1 -- 172.21.15.32:0/3547511875 shutdown_connections 2024-08-03T20:51:11.903 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:11.901+0000 7f1a45400640 1 --2- 172.21.15.32:0/3547511875 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f1a400d2290 0x7f1a400d26a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:51:11.903 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:11.901+0000 7f1a45400640 1 --2- 172.21.15.32:0/3547511875 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1a400d4bb0 0x7f1a400d1c60 unknown :-1 s=CLOSED pgs=653 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:51:11.903 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:11.901+0000 7f1a45400640 1 -- 172.21.15.32:0/3547511875 >> 172.21.15.32:0/3547511875 conn(0x7f1a400d4090 msgr2=0x7f1a400d44a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:51:11.903 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:11.901+0000 7f1a45400640 1 -- 172.21.15.32:0/3547511875 shutdown_connections 2024-08-03T20:51:11.903 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:11.901+0000 7f1a45400640 1 -- 172.21.15.32:0/3547511875 wait complete. 2024-08-03T20:51:11.904 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:11.903+0000 7f1a45400640 1 Processor -- start 2024-08-03T20:51:11.904 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:11.903+0000 7f1a45400640 1 -- start start 2024-08-03T20:51:11.908 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:11.905+0000 7f1a45400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1a400d0aa0 0x7f1a40159910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:51:11.908 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:11.905+0000 7f1a45400640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f1a400d2290 0x7f1a4015ea50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:51:11.908 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:11.905+0000 7f1a45400640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f1a400d4bb0 0x7f1a4015ef90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:51:11.908 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:11.905+0000 7f1a45400640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f1a400d15c0 con 0x7f1a400d0aa0 2024-08-03T20:51:11.908 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:11.905+0000 7f1a45400640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f1a400cfe50 con 0x7f1a400d4bb0 2024-08-03T20:51:11.908 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:11.905+0000 7f1a45400640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7f1a400d1740 con 0x7f1a400d2290 2024-08-03T20:51:11.908 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:11.905+0000 7f1a3fe00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1a400d0aa0 0x7f1a40159910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:51:11.909 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:11.905+0000 7f1a3fe00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1a400d0aa0 0x7f1a40159910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:54366/0 (socket says 172.21.15.32:54366) 2024-08-03T20:51:11.909 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:11.905+0000 7f1a3fe00640 1 -- 172.21.15.32:0/2716738848 learned_addr learned my addr 172.21.15.32:0/2716738848 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:51:11.909 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:11.905+0000 7f1a37e00640 1 --2- 172.21.15.32:0/2716738848 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f1a400d2290 0x7f1a4015ea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:51:11.909 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:11.905+0000 7f1a44a00640 1 --2- 172.21.15.32:0/2716738848 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f1a400d4bb0 0x7f1a4015ef90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:51:11.909 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:11.905+0000 7f1a3fe00640 1 -- 172.21.15.32:0/2716738848 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f1a400d2290 msgr2=0x7f1a4015ea50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:51:11.909 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:11.905+0000 7f1a3fe00640 1 --2- 172.21.15.32:0/2716738848 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f1a400d2290 0x7f1a4015ea50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:51:11.909 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:11.905+0000 7f1a3fe00640 1 -- 172.21.15.32:0/2716738848 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f1a400d4bb0 msgr2=0x7f1a4015ef90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:51:11.909 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:11.905+0000 7f1a3fe00640 1 --2- 172.21.15.32:0/2716738848 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f1a400d4bb0 0x7f1a4015ef90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:51:11.909 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:11.905+0000 7f1a3fe00640 1 -- 172.21.15.32:0/2716738848 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1a40161600 con 0x7f1a400d0aa0 2024-08-03T20:51:11.909 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:11.906+0000 7f1a3fe00640 1 --2- 172.21.15.32:0/2716738848 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1a400d0aa0 0x7f1a40159910 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7f1a30034040 tx=0x7f1a30003910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:51:11.909 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:11.906+0000 7f1a3cc00640 1 -- 172.21.15.32:0/2716738848 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a3003e030 con 0x7f1a400d0aa0 2024-08-03T20:51:11.909 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:11.906+0000 7f1a3cc00640 1 -- 172.21.15.32:0/2716738848 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f1a30003e00 con 0x7f1a400d0aa0 2024-08-03T20:51:11.909 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:11.906+0000 7f1a3cc00640 1 -- 172.21.15.32:0/2716738848 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a30035040 con 0x7f1a400d0aa0 2024-08-03T20:51:11.910 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:11.906+0000 7f1a45400640 1 -- 172.21.15.32:0/2716738848 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1a400b8a60 con 0x7f1a400d0aa0 2024-08-03T20:51:11.910 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:11.906+0000 7f1a45400640 1 -- 172.21.15.32:0/2716738848 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1a40160fe0 con 0x7f1a400d0aa0 2024-08-03T20:51:11.910 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:11.907+0000 7f1a45400640 1 -- 172.21.15.32:0/2716738848 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1a400b8850 con 0x7f1a400d0aa0 2024-08-03T20:51:11.910 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:11.908+0000 7f1a3cc00640 1 -- 172.21.15.32:0/2716738848 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f1a30041030 con 0x7f1a400d0aa0 2024-08-03T20:51:11.911 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:11.909+0000 7f1a3cc00640 1 --2- 172.21.15.32:0/2716738848 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f1a1c077040 0x7f1a1c079550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:51:11.911 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:11.909+0000 7f1a3cc00640 1 -- 172.21.15.32:0/2716738848 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(58..58 src has 1..58) ==== 5795+0+0 (secure 0 0 0) 0x7f1a300b9090 con 0x7f1a400d0aa0 2024-08-03T20:51:11.911 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:11.910+0000 7f1a37e00640 1 --2- 172.21.15.32:0/2716738848 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f1a1c077040 0x7f1a1c079550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:51:11.912 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:11.911+0000 7f1a37e00640 1 --2- 172.21.15.32:0/2716738848 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f1a1c077040 0x7f1a1c079550 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f1a4015f6d0 tx=0x7f1a2800b2e0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:51:11.915 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:11.914+0000 7f1a3cc00640 1 -- 172.21.15.32:0/2716738848 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7f1a400d2be0 con 0x7f1a400d0aa0 2024-08-03T20:51:12.166 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:12.165+0000 7f1a45400640 1 -- 172.21.15.32:0/2716738848 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f1a401617d0 con 0x7f1a400d0aa0 2024-08-03T20:51:12.167 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:12.165+0000 7f1a3cc00640 1 -- 172.21.15.32:0/2716738848 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 60 v60) ==== 94+0+5278 (secure 0 0 0) 0x7f1a401617d0 con 0x7f1a400d0aa0 2024-08-03T20:51:12.167 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-03T20:51:12.167 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":60,"btime":"2024-08-03T20:51:09:959042+0000","default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5261,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6833/2498840646","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":2498840646},{"type":"v1","addr":"172.21.15.32:6833","nonce":2498840646}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":45},{"gid":5276,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6837/3142148286","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6835","nonce":3142148286},{"type":"v1","addr":"172.21.15.118:6837","nonce":3142148286}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":46},{"gid":5632,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6836/3720010453","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6834","nonce":3720010453},{"type":"v1","addr":"172.21.15.118:6836","nonce":3720010453}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":46}],"filesystems":[{"mdsmap":{"epoch":60,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-08-03T20:50:27.688698+0000","modified":"2024-08-03T20:51:09.958977+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":58,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6045},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6045":{"gid":6045,"name":"c","rank":0,"incarnation":55,"state":"up:active","state_seq":5,"addr":"172.21.15.32:6835/2963295201","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":2963295201},{"type":"v1","addr":"172.21.15.32:6835","nonce":2963295201}]},"join_fscid":-1,"laggy_since":"2024-08-03T20:51:09.958977+0000","export_targets":[],"features":4540701547738038271,"flags":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[9],"metadata_pool":8,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6045,"qdb_cluster":[6045]},"id":4}]} 2024-08-03T20:51:12.167 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 60 2024-08-03T20:51:12.169 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:12.168+0000 7f1a45400640 1 -- 172.21.15.32:0/2716738848 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f1a1c077040 msgr2=0x7f1a1c079550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:51:12.169 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:12.168+0000 7f1a45400640 1 --2- 172.21.15.32:0/2716738848 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f1a1c077040 0x7f1a1c079550 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f1a4015f6d0 tx=0x7f1a2800b2e0 comp rx=0 tx=0).stop 2024-08-03T20:51:12.169 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:12.168+0000 7f1a45400640 1 -- 172.21.15.32:0/2716738848 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1a400d0aa0 msgr2=0x7f1a40159910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:51:12.169 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:12.168+0000 7f1a45400640 1 --2- 172.21.15.32:0/2716738848 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1a400d0aa0 0x7f1a40159910 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7f1a30034040 tx=0x7f1a30003910 comp rx=0 tx=0).stop 2024-08-03T20:51:12.170 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:12.168+0000 7f1a45400640 1 -- 172.21.15.32:0/2716738848 shutdown_connections 2024-08-03T20:51:12.170 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:12.168+0000 7f1a45400640 1 --2- 172.21.15.32:0/2716738848 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f1a1c077040 0x7f1a1c079550 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:51:12.170 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:12.168+0000 7f1a45400640 1 --2- 172.21.15.32:0/2716738848 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f1a400d4bb0 0x7f1a4015ef90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:51:12.170 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:12.168+0000 7f1a45400640 1 --2- 172.21.15.32:0/2716738848 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f1a400d2290 0x7f1a4015ea50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:51:12.170 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:12.168+0000 7f1a45400640 1 --2- 172.21.15.32:0/2716738848 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1a400d0aa0 0x7f1a40159910 unknown :-1 s=CLOSED pgs=654 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:51:12.170 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:12.168+0000 7f1a45400640 1 -- 172.21.15.32:0/2716738848 >> 172.21.15.32:0/2716738848 conn(0x7f1a400d4090 msgr2=0x7f1a401664c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:51:12.170 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:12.168+0000 7f1a45400640 1 -- 172.21.15.32:0/2716738848 shutdown_connections 2024-08-03T20:51:12.170 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:12.168+0000 7f1a45400640 1 -- 172.21.15.32:0/2716738848 wait complete. 2024-08-03T20:51:12.192 DEBUG:tasks.ceph_test_case:wait_until_true: success in 15s and 0 retries 2024-08-03T20:51:12.293 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mds mds_inject_journal_corrupt_dentry_first 2024-08-03T20:51:12.525 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:12.523+0000 7fcbc2200640 1 -- 172.21.15.32:0/128208555 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcba0003c10 msgr2=0x7fcba0004060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:51:12.525 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:12.523+0000 7fcbc2200640 1 --2- 172.21.15.32:0/128208555 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcba0003c10 0x7fcba0004060 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7fcbac006f90 tx=0x7fcba0005830 comp rx=0 tx=0).stop 2024-08-03T20:51:12.525 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:12.523+0000 7fcbbbe00640 1 -- 172.21.15.32:0/128208555 reap_dead start 2024-08-03T20:51:12.525 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:12.523+0000 7fcbc2200640 1 -- 172.21.15.32:0/128208555 shutdown_connections 2024-08-03T20:51:12.526 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:12.523+0000 7fcbc2200640 1 -- 172.21.15.32:0/128208555 >> 172.21.15.32:0/128208555 conn(0x7fcbbc0b1bd0 msgr2=0x7fcbbc0b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:51:12.526 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:12.523+0000 7fcbc2200640 1 -- 172.21.15.32:0/128208555 shutdown_connections 2024-08-03T20:51:12.526 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:12.524+0000 7fcbc2200640 1 -- 172.21.15.32:0/128208555 wait complete. 2024-08-03T20:51:12.527 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:12.526+0000 7fcbc2200640 1 Processor -- start 2024-08-03T20:51:12.527 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:12.526+0000 7fcbc2200640 1 -- start start 2024-08-03T20:51:12.529 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:12.527+0000 7fcbc2200640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fcba0003c10 0x7fcbbc0cbba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:51:12.529 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:12.527+0000 7fcbc2200640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fcbbc0c5ba0 0x7fcbbc0cc0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:51:12.529 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:12.527+0000 7fcbc2200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcbbc143640 0x7fcbbc143a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:51:12.529 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:12.527+0000 7fcbc2200640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fcbbc0b3ce0 con 0x7fcbbc143640 2024-08-03T20:51:12.529 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:12.527+0000 7fcbc2200640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7fcbbc0cf790 con 0x7fcba0003c10 2024-08-03T20:51:12.529 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:12.527+0000 7fcbc2200640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7fcbbc0b3e60 con 0x7fcbbc0c5ba0 2024-08-03T20:51:12.529 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:12.528+0000 7fcbbbe00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcbbc143640 0x7fcbbc143a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:51:12.529 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:12.528+0000 7fcbbaa00640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fcbbc0c5ba0 0x7fcbbc0cc0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:51:12.529 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:12.528+0000 7fcbbb400640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fcba0003c10 0x7fcbbc0cbba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:51:12.530 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:12.528+0000 7fcbbaa00640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fcbbc0c5ba0 0x7fcbbc0cc0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3301/0 says I am v2:172.21.15.32:51530/0 (socket says 172.21.15.32:51530) 2024-08-03T20:51:12.530 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:12.528+0000 7fcbbaa00640 1 -- 172.21.15.32:0/1617762390 learned_addr learned my addr 172.21.15.32:0/1617762390 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:51:12.531 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:12.528+0000 7fcbbbe00640 1 -- 172.21.15.32:0/1617762390 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fcbbc0c5ba0 msgr2=0x7fcbbc0cc0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:51:12.531 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:12.528+0000 7fcbbbe00640 1 --2- 172.21.15.32:0/1617762390 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fcbbc0c5ba0 0x7fcbbc0cc0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:51:12.531 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:12.528+0000 7fcbbbe00640 1 -- 172.21.15.32:0/1617762390 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fcba0003c10 msgr2=0x7fcbbc0cbba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:51:12.531 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:12.528+0000 7fcbbbe00640 1 --2- 172.21.15.32:0/1617762390 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fcba0003c10 0x7fcbbc0cbba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:51:12.531 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:12.528+0000 7fcbbbe00640 1 -- 172.21.15.32:0/1617762390 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcbbc1445c0 con 0x7fcbbc143640 2024-08-03T20:51:12.531 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:12.529+0000 7fcbbbe00640 1 --2- 172.21.15.32:0/1617762390 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcbbc143640 0x7fcbbc143a50 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7fcbbc071720 tx=0x7fcbb400f3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:51:12.531 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:12.529+0000 7fcbb3400640 1 -- 172.21.15.32:0/1617762390 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcbb4004b20 con 0x7fcbbc143640 2024-08-03T20:51:12.531 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:12.529+0000 7fcbb3400640 1 -- 172.21.15.32:0/1617762390 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fcbb4004cc0 con 0x7fcbbc143640 2024-08-03T20:51:12.531 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:12.529+0000 7fcbb3400640 1 -- 172.21.15.32:0/1617762390 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcbb4003bd0 con 0x7fcbbc143640 2024-08-03T20:51:12.531 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:12.529+0000 7fcbc2200640 1 -- 172.21.15.32:0/1617762390 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcbbc1448a0 con 0x7fcbbc143640 2024-08-03T20:51:12.531 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:12.529+0000 7fcbc2200640 1 -- 172.21.15.32:0/1617762390 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcbbc157910 con 0x7fcbbc143640 2024-08-03T20:51:12.532 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:12.530+0000 7fcbc2200640 1 -- 172.21.15.32:0/1617762390 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcbbc0b43f0 con 0x7fcbbc143640 2024-08-03T20:51:12.532 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:12.531+0000 7fcbb3400640 1 -- 172.21.15.32:0/1617762390 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7fcbb4004030 con 0x7fcbbc143640 2024-08-03T20:51:12.533 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:12.532+0000 7fcbb3400640 1 --2- 172.21.15.32:0/1617762390 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fcb9c077010 0x7fcb9c079520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:51:12.534 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:12.532+0000 7fcbb3400640 1 -- 172.21.15.32:0/1617762390 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(58..58 src has 1..58) ==== 5795+0+0 (secure 0 0 0) 0x7fcbb4098db0 con 0x7fcbbc143640 2024-08-03T20:51:12.534 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:12.533+0000 7fcbbb400640 1 --2- 172.21.15.32:0/1617762390 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fcb9c077010 0x7fcb9c079520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:51:12.535 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:12.533+0000 7fcbbb400640 1 --2- 172.21.15.32:0/1617762390 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fcb9c077010 0x7fcb9c079520 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7fcbac033000 tx=0x7fcbac02d000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:51:12.538 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:12.536+0000 7fcbb3400640 1 -- 172.21.15.32:0/1617762390 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7fcbb409d070 con 0x7fcbbc143640 2024-08-03T20:51:12.741 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:12.739+0000 7fcbc2200640 1 -- 172.21.15.32:0/1617762390 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "config rm", "who": "mds", "name": "mds_inject_journal_corrupt_dentry_first"} v 0) -- 0x7fcbbc142d50 con 0x7fcbbc143640 2024-08-03T20:51:12.745 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:12.743+0000 7fcbb3400640 1 -- 172.21.15.32:0/1617762390 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mds", "name": "mds_inject_journal_corrupt_dentry_first"}]=0 v37) ==== 122+0+0 (secure 0 0 0) 0x7fcbb4061860 con 0x7fcbbc143640 2024-08-03T20:51:12.748 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:12.745+0000 7fcbc2200640 1 -- 172.21.15.32:0/1617762390 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fcb9c077010 msgr2=0x7fcb9c079520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:51:12.748 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:12.746+0000 7fcbc2200640 1 --2- 172.21.15.32:0/1617762390 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fcb9c077010 0x7fcb9c079520 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7fcbac033000 tx=0x7fcbac02d000 comp rx=0 tx=0).stop 2024-08-03T20:51:12.748 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:12.746+0000 7fcbc2200640 1 -- 172.21.15.32:0/1617762390 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcbbc143640 msgr2=0x7fcbbc143a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:51:12.748 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:12.746+0000 7fcbc2200640 1 --2- 172.21.15.32:0/1617762390 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcbbc143640 0x7fcbbc143a50 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7fcbbc071720 tx=0x7fcbb400f3c0 comp rx=0 tx=0).stop 2024-08-03T20:51:12.748 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:12.746+0000 7fcbc2200640 1 -- 172.21.15.32:0/1617762390 shutdown_connections 2024-08-03T20:51:12.748 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:12.746+0000 7fcbc2200640 1 --2- 172.21.15.32:0/1617762390 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fcb9c077010 0x7fcb9c079520 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:51:12.748 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:12.746+0000 7fcbc2200640 1 --2- 172.21.15.32:0/1617762390 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcbbc143640 0x7fcbbc143a50 unknown :-1 s=CLOSED pgs=657 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:51:12.748 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:12.746+0000 7fcbc2200640 1 --2- 172.21.15.32:0/1617762390 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fcbbc0c5ba0 0x7fcbbc0cc0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:51:12.748 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:12.746+0000 7fcbc2200640 1 --2- 172.21.15.32:0/1617762390 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fcba0003c10 0x7fcbbc0cbba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:51:12.748 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:12.746+0000 7fcbc2200640 1 -- 172.21.15.32:0/1617762390 >> 172.21.15.32:0/1617762390 conn(0x7fcbbc0b1bd0 msgr2=0x7fcbbc06e950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:51:12.748 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:12.746+0000 7fcbc2200640 1 -- 172.21.15.32:0/1617762390 shutdown_connections 2024-08-03T20:51:12.749 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:12.746+0000 7fcbc2200640 1 -- 172.21.15.32:0/1617762390 wait complete. 2024-08-03T20:51:12.769 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mds freeze 4:0 false 2024-08-03T20:51:13.002 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:13.000+0000 7fc630200640 1 -- 172.21.15.32:0/1968698069 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc62809b040 msgr2=0x7fc6280cd730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:51:13.002 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:13.000+0000 7fc630200640 1 --2- 172.21.15.32:0/1968698069 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc62809b040 0x7fc6280cd730 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7fc614005fd0 tx=0x7fc61402a540 comp rx=0 tx=0).stop 2024-08-03T20:51:13.002 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:13.000+0000 7fc630200640 1 -- 172.21.15.32:0/1968698069 shutdown_connections 2024-08-03T20:51:13.002 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:13.000+0000 7fc630200640 1 --2- 172.21.15.32:0/1968698069 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fc6280cdc70 0x7fc628166de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:51:13.002 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:13.000+0000 7fc630200640 1 --2- 172.21.15.32:0/1968698069 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc62809b040 0x7fc6280cd730 unknown :-1 s=CLOSED pgs=658 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:51:13.002 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:13.000+0000 7fc630200640 1 -- 172.21.15.32:0/1968698069 >> 172.21.15.32:0/1968698069 conn(0x7fc6280b3c00 msgr2=0x7fc6280b4010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:51:13.002 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:13.000+0000 7fc630200640 1 -- 172.21.15.32:0/1968698069 shutdown_connections 2024-08-03T20:51:13.002 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:13.000+0000 7fc630200640 1 -- 172.21.15.32:0/1968698069 wait complete. 2024-08-03T20:51:13.004 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:13.002+0000 7fc630200640 1 Processor -- start 2024-08-03T20:51:13.004 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:13.002+0000 7fc630200640 1 -- start start 2024-08-03T20:51:13.006 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:13.004+0000 7fc630200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc62809ac30 0x7fc6281e2ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:51:13.006 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:13.004+0000 7fc630200640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc62809b040 0x7fc6281f9890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:51:13.006 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:13.004+0000 7fc630200640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fc6280cdc70 0x7fc6280b6690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:51:13.006 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:13.004+0000 7fc630200640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fc6280c9020 con 0x7fc62809ac30 2024-08-03T20:51:13.006 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:13.004+0000 7fc630200640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7fc6280c8ea0 con 0x7fc62809b040 2024-08-03T20:51:13.006 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:13.004+0000 7fc630200640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7fc6280b7a20 con 0x7fc6280cdc70 2024-08-03T20:51:13.007 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:13.004+0000 7fc62d800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc62809ac30 0x7fc6281e2ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:51:13.007 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:13.004+0000 7fc62d800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc62809ac30 0x7fc6281e2ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:54392/0 (socket says 172.21.15.32:54392) 2024-08-03T20:51:13.007 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:13.004+0000 7fc62d800640 1 -- 172.21.15.32:0/3030721128 learned_addr learned my addr 172.21.15.32:0/3030721128 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:51:13.007 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:13.004+0000 7fc62ce00640 1 --2- 172.21.15.32:0/3030721128 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc62809b040 0x7fc6281f9890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:51:13.007 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:13.004+0000 7fc62e200640 1 --2- 172.21.15.32:0/3030721128 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fc6280cdc70 0x7fc6280b6690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:51:13.007 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:13.005+0000 7fc62d800640 1 -- 172.21.15.32:0/3030721128 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fc6280cdc70 msgr2=0x7fc6280b6690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:51:13.007 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:13.005+0000 7fc62d800640 1 --2- 172.21.15.32:0/3030721128 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fc6280cdc70 0x7fc6280b6690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:51:13.007 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:13.005+0000 7fc62d800640 1 -- 172.21.15.32:0/3030721128 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc62809b040 msgr2=0x7fc6281f9890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:51:13.007 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:13.005+0000 7fc62d800640 1 --2- 172.21.15.32:0/3030721128 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc62809b040 0x7fc6281f9890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:51:13.007 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:13.005+0000 7fc62d800640 1 -- 172.21.15.32:0/3030721128 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc6281e33e0 con 0x7fc62809ac30 2024-08-03T20:51:13.007 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:13.005+0000 7fc62e200640 1 --2- 172.21.15.32:0/3030721128 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fc6280cdc70 0x7fc6280b6690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-03T20:51:13.008 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:13.005+0000 7fc62d800640 1 --2- 172.21.15.32:0/3030721128 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc62809ac30 0x7fc6281e2ea0 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7fc61c00c3a0 tx=0x7fc61c00c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:51:13.008 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:13.005+0000 7fc625600640 1 -- 172.21.15.32:0/3030721128 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc61c017070 con 0x7fc62809ac30 2024-08-03T20:51:13.008 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:13.005+0000 7fc625600640 1 -- 172.21.15.32:0/3030721128 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fc61c0053c0 con 0x7fc62809ac30 2024-08-03T20:51:13.008 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:13.005+0000 7fc630200640 1 -- 172.21.15.32:0/3030721128 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc6281f9dd0 con 0x7fc62809ac30 2024-08-03T20:51:13.008 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:13.006+0000 7fc625600640 1 -- 172.21.15.32:0/3030721128 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc61c003850 con 0x7fc62809ac30 2024-08-03T20:51:13.009 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:13.006+0000 7fc630200640 1 -- 172.21.15.32:0/3030721128 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc6281f60d0 con 0x7fc62809ac30 2024-08-03T20:51:13.009 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:13.007+0000 7fc630200640 1 -- 172.21.15.32:0/3030721128 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc6280b2e10 con 0x7fc62809ac30 2024-08-03T20:51:13.009 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:13.008+0000 7fc625600640 1 -- 172.21.15.32:0/3030721128 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7fc61c01a070 con 0x7fc62809ac30 2024-08-03T20:51:13.010 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:13.009+0000 7fc625600640 1 --2- 172.21.15.32:0/3030721128 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fc608077010 0x7fc608079520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:51:13.010 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:13.009+0000 7fc625600640 1 -- 172.21.15.32:0/3030721128 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(58..58 src has 1..58) ==== 5795+0+0 (secure 0 0 0) 0x7fc61c097de0 con 0x7fc62809ac30 2024-08-03T20:51:13.011 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:13.010+0000 7fc62ce00640 1 --2- 172.21.15.32:0/3030721128 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fc608077010 0x7fc608079520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:51:13.012 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:13.010+0000 7fc62ce00640 1 --2- 172.21.15.32:0/3030721128 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fc608077010 0x7fc608079520 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7fc614033000 tx=0x7fc614004730 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:51:13.015 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:13.014+0000 7fc625600640 1 -- 172.21.15.32:0/3030721128 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7fc6281e33e0 con 0x7fc62809ac30 2024-08-03T20:51:13.298 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:13.297+0000 7fc630200640 1 -- 172.21.15.32:0/3030721128 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "mds freeze", "role_or_gid": "4:0", "val": "false"} v 0) -- 0x7fc62806fae0 con 0x7fc62809ac30 2024-08-03T20:51:13.981 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:13.980+0000 7fc625600640 1 -- 172.21.15.32:0/3030721128 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "mds freeze", "role_or_gid": "4:0", "val": "false"}]=0 unfreezing mds.6045 v61) ==== 115+0+0 (secure 0 0 0) 0x7fc61c0691b0 con 0x7fc62809ac30 2024-08-03T20:51:13.982 INFO:teuthology.orchestra.run.smithi032.stderr:unfreezing mds.6045 2024-08-03T20:51:13.986 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:13.984+0000 7fc630200640 1 -- 172.21.15.32:0/3030721128 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fc608077010 msgr2=0x7fc608079520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:51:13.986 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:13.984+0000 7fc630200640 1 --2- 172.21.15.32:0/3030721128 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fc608077010 0x7fc608079520 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7fc614033000 tx=0x7fc614004730 comp rx=0 tx=0).stop 2024-08-03T20:51:13.986 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:13.984+0000 7fc630200640 1 -- 172.21.15.32:0/3030721128 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc62809ac30 msgr2=0x7fc6281e2ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:51:13.986 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:13.984+0000 7fc630200640 1 --2- 172.21.15.32:0/3030721128 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc62809ac30 0x7fc6281e2ea0 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7fc61c00c3a0 tx=0x7fc61c00c860 comp rx=0 tx=0).stop 2024-08-03T20:51:13.986 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:13.984+0000 7fc630200640 1 -- 172.21.15.32:0/3030721128 shutdown_connections 2024-08-03T20:51:13.986 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:13.984+0000 7fc630200640 1 --2- 172.21.15.32:0/3030721128 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fc608077010 0x7fc608079520 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:51:13.986 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:13.984+0000 7fc630200640 1 --2- 172.21.15.32:0/3030721128 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc62809b040 0x7fc6281f9890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:51:13.986 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:13.984+0000 7fc630200640 1 --2- 172.21.15.32:0/3030721128 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fc6280cdc70 0x7fc6280b6690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:51:13.987 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:13.984+0000 7fc630200640 1 --2- 172.21.15.32:0/3030721128 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc62809ac30 0x7fc6281e2ea0 unknown :-1 s=CLOSED pgs=659 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:51:13.987 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:13.984+0000 7fc630200640 1 -- 172.21.15.32:0/3030721128 >> 172.21.15.32:0/3030721128 conn(0x7fc6280b3c00 msgr2=0x7fc6280cfce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:51:13.987 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:13.984+0000 7fc630200640 1 -- 172.21.15.32:0/3030721128 shutdown_connections 2024-08-03T20:51:13.987 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:13.984+0000 7fc630200640 1 -- 172.21.15.32:0/3030721128 wait complete. 2024-08-03T20:51:14.015 DEBUG:teuthology.orchestra.run.smithi032:> sudo sysctl -n kernel.core_pattern 2024-08-03T20:51:14.071 INFO:teuthology.orchestra.run.smithi032.stdout:/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-03T20:51:14.074 INFO:tasks.cephfs.cephfs_test_case:Clearing core from directory: /home/ubuntu/cephtest/archive/coredump 2024-08-03T20:51:14.074 DEBUG:teuthology.orchestra.run.smithi032:> cd /home/ubuntu/cephtest/archive/coredump && sudo ls | sudo xargs file 2024-08-03T20:51:14.137 INFO:teuthology.orchestra.run.smithi032.stdout:1722718255.53795.core: ELF 64-bit LSB core file, x86-64, version 1 (SYSV), SVR4-style, from 'ceph-mds -f --cluster ceph -i c', real uid: 0, effective uid: 0, real gid: 0, effective gid: 0, execfn: '/proc/self/exe', platform: 'x86_64' 2024-08-03T20:51:14.141 INFO:tasks.cephfs.cephfs_test_case:Enumerated cores: ['1722718255.53795.core'] 2024-08-03T20:51:14.141 INFO:tasks.cephfs.cephfs_test_case:Found core file 1722718255.53795.core, deleting it 2024-08-03T20:51:14.142 DEBUG:teuthology.orchestra.run.smithi032:> cd /home/ubuntu/cephtest/archive/coredump && sudo rm -f 1722718255.53795.core 2024-08-03T20:51:14.231 INFO:tasks.ceph.mds.c:Restarting daemon 2024-08-03T20:51:14.232 INFO:tasks.ceph.mds.c:Stopping old one... 2024-08-03T20:51:14.232 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-08-03T20:51:14.232 INFO:teuthology.orchestra.run:waiting for 300 2024-08-03T20:51:14.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-03T20:51:14.232 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_7d7f5fe79c7b6b5080402153d07fb1acc4ce00a2/teuthology/orchestra/daemon/state.py", line 146, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_7d7f5fe79c7b6b5080402153d07fb1acc4ce00a2/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_7d7f5fe79c7b6b5080402153d07fb1acc4ce00a2/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_7d7f5fe79c7b6b5080402153d07fb1acc4ce00a2/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi032 with status 1: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c' 2024-08-03T20:51:14.233 INFO:tasks.ceph.mds.c:Stopped 2024-08-03T20:51:14.233 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-08-03T20:51:14.238 INFO:tasks.ceph.mds.c:Started 2024-08-03T20:51:14.238 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-08-03T20:51:14.434 INFO:tasks.ceph.mds.c.smithi032.stdout:starting mds.c at 2024-08-03T20:51:14.467 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:14.465+0000 7f1202e00640 1 -- 172.21.15.32:0/937272908 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f11e4003c10 msgr2=0x7f11e4004060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:51:14.467 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:14.465+0000 7f1202e00640 1 --2- 172.21.15.32:0/937272908 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f11e4003c10 0x7f11e4004060 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7f11f0006fa0 tx=0x7f11e4005830 comp rx=0 tx=0).stop 2024-08-03T20:51:14.468 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:14.465+0000 7f1202400640 1 -- 172.21.15.32:0/937272908 reap_dead start 2024-08-03T20:51:14.468 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:14.466+0000 7f1202e00640 1 -- 172.21.15.32:0/937272908 shutdown_connections 2024-08-03T20:51:14.468 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:14.466+0000 7f1202e00640 1 -- 172.21.15.32:0/937272908 >> 172.21.15.32:0/937272908 conn(0x7f11fc0b2120 msgr2=0x7f11fc0cfd60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:51:14.468 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:14.466+0000 7f1202e00640 1 -- 172.21.15.32:0/937272908 shutdown_connections 2024-08-03T20:51:14.468 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:14.466+0000 7f1202e00640 1 -- 172.21.15.32:0/937272908 wait complete. 2024-08-03T20:51:14.469 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:14.468+0000 7f1202e00640 1 Processor -- start 2024-08-03T20:51:14.470 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:14.468+0000 7f1202e00640 1 -- start start 2024-08-03T20:51:14.472 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:14.470+0000 7f1202e00640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f11e4003c10 0x7f11fc0c93d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:51:14.473 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:14.470+0000 7f1202e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f11fc0c3f70 0x7f11fc0afdf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:51:14.473 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:14.470+0000 7f1202e00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f11fc0b0330 0x7f11fc1f8d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:51:14.473 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:14.470+0000 7f1202e00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f11fc1600e0 con 0x7f11fc0c3f70 2024-08-03T20:51:14.473 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:14.470+0000 7f1202e00640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f11fc0ad4b0 con 0x7f11fc0b0330 2024-08-03T20:51:14.473 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:14.470+0000 7f1202e00640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7f11fc160260 con 0x7f11e4003c10 2024-08-03T20:51:14.473 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:14.470+0000 7f1201000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f11fc0c3f70 0x7f11fc0afdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:51:14.473 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:14.470+0000 7f1201000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f11fc0c3f70 0x7f11fc0afdf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:54422/0 (socket says 172.21.15.32:54422) 2024-08-03T20:51:14.473 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:14.470+0000 7f1201000640 1 -- 172.21.15.32:0/119273887 learned_addr learned my addr 172.21.15.32:0/119273887 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:51:14.473 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:14.470+0000 7f1201a00640 1 --2- 172.21.15.32:0/119273887 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f11e4003c10 0x7f11fc0c93d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:51:14.473 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:14.470+0000 7f1202400640 1 --2- 172.21.15.32:0/119273887 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f11fc0b0330 0x7f11fc1f8d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:51:14.473 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:14.470+0000 7f1201000640 1 -- 172.21.15.32:0/119273887 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f11e4003c10 msgr2=0x7f11fc0c93d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:51:14.474 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:14.470+0000 7f1201000640 1 --2- 172.21.15.32:0/119273887 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f11e4003c10 0x7f11fc0c93d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:51:14.474 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:14.470+0000 7f1201000640 1 -- 172.21.15.32:0/119273887 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f11fc0b0330 msgr2=0x7f11fc1f8d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:51:14.474 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:14.470+0000 7f1201000640 1 --2- 172.21.15.32:0/119273887 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f11fc0b0330 0x7f11fc1f8d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:51:14.474 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:14.470+0000 7f1201000640 1 -- 172.21.15.32:0/119273887 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f11fc0b0740 con 0x7f11fc0c3f70 2024-08-03T20:51:14.474 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:14.471+0000 7f1201000640 1 --2- 172.21.15.32:0/119273887 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f11fc0c3f70 0x7f11fc0afdf0 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7f11e8011ae0 tx=0x7f11e8009e00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:51:14.474 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:14.471+0000 7f11f9600640 1 -- 172.21.15.32:0/119273887 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11e801e070 con 0x7f11fc0c3f70 2024-08-03T20:51:14.474 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:14.471+0000 7f11f9600640 1 -- 172.21.15.32:0/119273887 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f11e8004460 con 0x7f11fc0c3f70 2024-08-03T20:51:14.474 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:14.471+0000 7f11f9600640 1 -- 172.21.15.32:0/119273887 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11e800fc60 con 0x7f11fc0c3f70 2024-08-03T20:51:14.474 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:14.471+0000 7f1202e00640 1 -- 172.21.15.32:0/119273887 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f11fc0c9910 con 0x7f11fc0c3f70 2024-08-03T20:51:14.475 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:14.471+0000 7f1202e00640 1 -- 172.21.15.32:0/119273887 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f11fc0cefa0 con 0x7f11fc0c3f70 2024-08-03T20:51:14.475 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:14.472+0000 7f1202e00640 1 -- 172.21.15.32:0/119273887 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f11fc160780 con 0x7f11fc0c3f70 2024-08-03T20:51:14.475 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:14.473+0000 7f11f9600640 1 -- 172.21.15.32:0/119273887 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f11e8021070 con 0x7f11fc0c3f70 2024-08-03T20:51:14.476 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:14.474+0000 7f11f9600640 1 --2- 172.21.15.32:0/119273887 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f11dc077010 0x7f11dc079520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:51:14.476 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:14.475+0000 7f11f9600640 1 -- 172.21.15.32:0/119273887 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(59..59 src has 1..59) ==== 5881+0+0 (secure 0 0 0) 0x7f11e80993e0 con 0x7f11fc0c3f70 2024-08-03T20:51:14.476 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:14.475+0000 7f1201a00640 1 --2- 172.21.15.32:0/119273887 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f11dc077010 0x7f11dc079520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:51:14.477 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:14.476+0000 7f1201a00640 1 --2- 172.21.15.32:0/119273887 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f11dc077010 0x7f11dc079520 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f11f0033000 tx=0x7f11f002d000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:51:14.480 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:14.479+0000 7f11f9600640 1 -- 172.21.15.32:0/119273887 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7f11fc0b0740 con 0x7f11fc0c3f70 2024-08-03T20:51:14.726 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:14.725+0000 7f1202e00640 1 -- 172.21.15.32:0/119273887 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f11fc0c44e0 con 0x7f11fc0c3f70 2024-08-03T20:51:14.727 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:14.726+0000 7f11f9600640 1 -- 172.21.15.32:0/119273887 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 62 v62) ==== 94+0+4398 (secure 0 0 0) 0x7f11e8061ed0 con 0x7f11fc0c3f70 2024-08-03T20:51:14.727 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-03T20:51:14.727 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":62,"btime":"2024-08-03T20:51:13:985872+0000","default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5261,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6833/2498840646","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":2498840646},{"type":"v1","addr":"172.21.15.32:6833","nonce":2498840646}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":45},{"gid":5276,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6837/3142148286","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6835","nonce":3142148286},{"type":"v1","addr":"172.21.15.118:6837","nonce":3142148286}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":46}],"filesystems":[{"mdsmap":{"epoch":62,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-08-03T20:50:27.688698+0000","modified":"2024-08-03T20:51:13.985866+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":59,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5632},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5632":{"gid":5632,"name":"d","rank":0,"incarnation":62,"state":"up:replay","state_seq":1,"addr":"172.21.15.118:6836/3720010453","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6834","nonce":3720010453},{"type":"v1","addr":"172.21.15.118:6836","nonce":3720010453}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[9],"metadata_pool":8,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":4}]} 2024-08-03T20:51:14.728 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 62 2024-08-03T20:51:14.730 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:14.728+0000 7f1202e00640 1 -- 172.21.15.32:0/119273887 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f11dc077010 msgr2=0x7f11dc079520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:51:14.730 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:14.728+0000 7f1202e00640 1 --2- 172.21.15.32:0/119273887 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f11dc077010 0x7f11dc079520 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f11f0033000 tx=0x7f11f002d000 comp rx=0 tx=0).stop 2024-08-03T20:51:14.730 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:14.728+0000 7f1202e00640 1 -- 172.21.15.32:0/119273887 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f11fc0c3f70 msgr2=0x7f11fc0afdf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:51:14.730 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:14.728+0000 7f1202e00640 1 --2- 172.21.15.32:0/119273887 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f11fc0c3f70 0x7f11fc0afdf0 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7f11e8011ae0 tx=0x7f11e8009e00 comp rx=0 tx=0).stop 2024-08-03T20:51:14.730 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:14.728+0000 7f1202e00640 1 -- 172.21.15.32:0/119273887 shutdown_connections 2024-08-03T20:51:14.730 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:14.728+0000 7f1202e00640 1 --2- 172.21.15.32:0/119273887 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f11dc077010 0x7f11dc079520 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:51:14.730 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:14.728+0000 7f1202e00640 1 --2- 172.21.15.32:0/119273887 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f11fc0b0330 0x7f11fc1f8d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:51:14.730 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:14.728+0000 7f1202e00640 1 --2- 172.21.15.32:0/119273887 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f11fc0c3f70 0x7f11fc0afdf0 unknown :-1 s=CLOSED pgs=663 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:51:14.730 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:14.728+0000 7f1202e00640 1 --2- 172.21.15.32:0/119273887 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f11e4003c10 0x7f11fc0c93d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:51:14.730 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:14.728+0000 7f1202e00640 1 -- 172.21.15.32:0/119273887 >> 172.21.15.32:0/119273887 conn(0x7f11fc0b2120 msgr2=0x7f11fc0ccf60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:51:14.731 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:14.728+0000 7f1202e00640 1 -- 172.21.15.32:0/119273887 shutdown_connections 2024-08-03T20:51:14.731 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:14.728+0000 7f1202e00640 1 -- 172.21.15.32:0/119273887 wait complete. 2024-08-03T20:51:15.155 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 62, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-08-03T20:50:27.688698+0000', 'modified': '2024-08-03T20:51:13.985866+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 59, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 5632}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_5632': {'gid': 5632, 'name': 'd', 'rank': 0, 'incarnation': 62, 'state': 'up:replay', 'state_seq': 1, 'addr': '172.21.15.118:6836/3720010453', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.118:6834', 'nonce': 3720010453}, {'type': 'v1', 'addr': '172.21.15.118:6836', 'nonce': 3720010453}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [9], 'metadata_pool': 8, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 0, 'qdb_cluster': []} 2024-08-03T20:51:15.155 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_5632:up:replay 2024-08-03T20:51:16.156 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-08-03T20:51:16.398 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:16.397+0000 7f61cc000640 1 -- 172.21.15.32:0/3655304147 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f61c406e930 msgr2=0x7f61c4144f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:51:16.399 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:16.397+0000 7f61cc000640 1 --2- 172.21.15.32:0/3655304147 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f61c406e930 0x7f61c4144f90 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7f61b8009830 tx=0x7f61b802a480 comp rx=0 tx=0).stop 2024-08-03T20:51:16.399 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:16.397+0000 7f61cc000640 1 -- 172.21.15.32:0/3655304147 shutdown_connections 2024-08-03T20:51:16.399 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:16.397+0000 7f61cc000640 1 --2- 172.21.15.32:0/3655304147 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f61c4073e60 0x7f61c4074270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:51:16.399 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:16.397+0000 7f61cc000640 1 --2- 172.21.15.32:0/3655304147 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f61c406e930 0x7f61c4144f90 unknown :-1 s=CLOSED pgs=665 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:51:16.399 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:16.397+0000 7f61cc000640 1 -- 172.21.15.32:0/3655304147 >> 172.21.15.32:0/3655304147 conn(0x7f61c40b1bd0 msgr2=0x7f61c40b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:51:16.399 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:16.397+0000 7f61cc000640 1 -- 172.21.15.32:0/3655304147 shutdown_connections 2024-08-03T20:51:16.399 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:16.397+0000 7f61cc000640 1 -- 172.21.15.32:0/3655304147 wait complete. 2024-08-03T20:51:16.400 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:16.399+0000 7f61cc000640 1 Processor -- start 2024-08-03T20:51:16.400 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:16.399+0000 7f61cc000640 1 -- start start 2024-08-03T20:51:16.404 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:16.401+0000 7f61cc000640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f61c406e930 0x7f61c41f9100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:51:16.404 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:16.401+0000 7f61cc000640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f61c4073e60 0x7f61c40cb970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:51:16.404 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:16.401+0000 7f61cc000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f61c40747b0 0x7f61c4134710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:51:16.404 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:16.401+0000 7f61cc000640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f61c40cf9c0 con 0x7f61c40747b0 2024-08-03T20:51:16.404 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:16.401+0000 7f61cc000640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f61c40cf840 con 0x7f61c4073e60 2024-08-03T20:51:16.404 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:16.401+0000 7f61cc000640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7f61c40ccff0 con 0x7f61c406e930 2024-08-03T20:51:16.404 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:16.401+0000 7f61ca000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f61c40747b0 0x7f61c4134710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:51:16.404 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:16.401+0000 7f61ca000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f61c40747b0 0x7f61c4134710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:54438/0 (socket says 172.21.15.32:54438) 2024-08-03T20:51:16.404 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:16.401+0000 7f61ca000640 1 -- 172.21.15.32:0/2506901010 learned_addr learned my addr 172.21.15.32:0/2506901010 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:51:16.404 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:16.401+0000 7f61c9600640 1 --2- 172.21.15.32:0/2506901010 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f61c406e930 0x7f61c41f9100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:51:16.405 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:16.401+0000 7f61c8c00640 1 --2- 172.21.15.32:0/2506901010 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f61c4073e60 0x7f61c40cb970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:51:16.405 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:16.401+0000 7f61ca000640 1 -- 172.21.15.32:0/2506901010 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f61c406e930 msgr2=0x7f61c41f9100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:51:16.405 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:16.401+0000 7f61ca000640 1 --2- 172.21.15.32:0/2506901010 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f61c406e930 0x7f61c41f9100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:51:16.405 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:16.401+0000 7f61ca000640 1 -- 172.21.15.32:0/2506901010 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f61c4073e60 msgr2=0x7f61c40cb970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:51:16.405 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:16.401+0000 7f61ca000640 1 --2- 172.21.15.32:0/2506901010 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f61c4073e60 0x7f61c40cb970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:51:16.405 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:16.401+0000 7f61ca000640 1 -- 172.21.15.32:0/2506901010 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f61c4144890 con 0x7f61c40747b0 2024-08-03T20:51:16.405 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:16.401+0000 7f61c9600640 1 --2- 172.21.15.32:0/2506901010 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f61c406e930 0x7f61c41f9100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-03T20:51:16.405 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:16.401+0000 7f61c8c00640 1 --2- 172.21.15.32:0/2506901010 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f61c4073e60 0x7f61c40cb970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-03T20:51:16.405 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:16.402+0000 7f61ca000640 1 --2- 172.21.15.32:0/2506901010 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f61c40747b0 0x7f61c4134710 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7f61bc00c760 tx=0x7f61bc00cc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:51:16.405 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:16.402+0000 7f61c1600640 1 -- 172.21.15.32:0/2506901010 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61bc003da0 con 0x7f61c40747b0 2024-08-03T20:51:16.405 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:16.402+0000 7f61c1600640 1 -- 172.21.15.32:0/2506901010 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f61bc00ce10 con 0x7f61c40747b0 2024-08-03T20:51:16.405 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:16.402+0000 7f61c1600640 1 -- 172.21.15.32:0/2506901010 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61bc0052d0 con 0x7f61c40747b0 2024-08-03T20:51:16.405 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:16.402+0000 7f61cc000640 1 -- 172.21.15.32:0/2506901010 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f61c413ff50 con 0x7f61c40747b0 2024-08-03T20:51:16.405 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:16.402+0000 7f61cc000640 1 -- 172.21.15.32:0/2506901010 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f61c41445b0 con 0x7f61c40747b0 2024-08-03T20:51:16.406 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:16.403+0000 7f61cc000640 1 -- 172.21.15.32:0/2506901010 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f61c4141290 con 0x7f61c40747b0 2024-08-03T20:51:16.406 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:16.405+0000 7f61c1600640 1 -- 172.21.15.32:0/2506901010 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f61bc005470 con 0x7f61c40747b0 2024-08-03T20:51:16.407 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:16.406+0000 7f61c1600640 1 --2- 172.21.15.32:0/2506901010 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f61a4077010 0x7f61a4079520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:51:16.407 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:16.406+0000 7f61c1600640 1 -- 172.21.15.32:0/2506901010 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(59..59 src has 1..59) ==== 5881+0+0 (secure 0 0 0) 0x7f61bc0987f0 con 0x7f61c40747b0 2024-08-03T20:51:16.407 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:16.406+0000 7f61c9600640 1 --2- 172.21.15.32:0/2506901010 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f61a4077010 0x7f61a4079520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:51:16.408 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:16.407+0000 7f61c9600640 1 --2- 172.21.15.32:0/2506901010 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f61a4077010 0x7f61a4079520 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f61b8034000 tx=0x7f61b8004730 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:51:16.411 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:16.410+0000 7f61c1600640 1 -- 172.21.15.32:0/2506901010 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7f61c4141290 con 0x7f61c40747b0 2024-08-03T20:51:16.657 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:16.656+0000 7f61cc000640 1 -- 172.21.15.32:0/2506901010 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f61c40cd530 con 0x7f61c40747b0 2024-08-03T20:51:16.658 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:16.656+0000 7f61c1600640 1 -- 172.21.15.32:0/2506901010 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 64 v64) ==== 94+0+5224 (secure 0 0 0) 0x7f61bc0618a0 con 0x7f61c40747b0 2024-08-03T20:51:16.658 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-03T20:51:16.658 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":64,"btime":"2024-08-03T20:51:15:993015+0000","default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5261,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6833/2498840646","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":2498840646},{"type":"v1","addr":"172.21.15.32:6833","nonce":2498840646}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":45},{"gid":5276,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6837/3142148286","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6835","nonce":3142148286},{"type":"v1","addr":"172.21.15.118:6837","nonce":3142148286}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":46},{"gid":6180,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6835/1934242557","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1934242557},{"type":"v1","addr":"172.21.15.32:6835","nonce":1934242557}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":63}],"filesystems":[{"mdsmap":{"epoch":64,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-08-03T20:50:27.688698+0000","modified":"2024-08-03T20:51:14.996411+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":59,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5632},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5632":{"gid":5632,"name":"d","rank":0,"incarnation":62,"state":"up:rejoin","state_seq":15,"addr":"172.21.15.118:6836/3720010453","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6834","nonce":3720010453},{"type":"v1","addr":"172.21.15.118:6836","nonce":3720010453}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[9],"metadata_pool":8,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":4}]} 2024-08-03T20:51:16.658 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 64 2024-08-03T20:51:16.660 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:16.659+0000 7f61cc000640 1 -- 172.21.15.32:0/2506901010 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f61a4077010 msgr2=0x7f61a4079520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:51:16.660 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:16.659+0000 7f61cc000640 1 --2- 172.21.15.32:0/2506901010 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f61a4077010 0x7f61a4079520 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f61b8034000 tx=0x7f61b8004730 comp rx=0 tx=0).stop 2024-08-03T20:51:16.661 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:16.659+0000 7f61cc000640 1 -- 172.21.15.32:0/2506901010 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f61c40747b0 msgr2=0x7f61c4134710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:51:16.661 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:16.659+0000 7f61cc000640 1 --2- 172.21.15.32:0/2506901010 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f61c40747b0 0x7f61c4134710 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7f61bc00c760 tx=0x7f61bc00cc20 comp rx=0 tx=0).stop 2024-08-03T20:51:16.661 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:16.659+0000 7f61cc000640 1 -- 172.21.15.32:0/2506901010 shutdown_connections 2024-08-03T20:51:16.661 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:16.659+0000 7f61cc000640 1 --2- 172.21.15.32:0/2506901010 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f61a4077010 0x7f61a4079520 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:51:16.661 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:16.659+0000 7f61cc000640 1 --2- 172.21.15.32:0/2506901010 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f61c4073e60 0x7f61c40cb970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:51:16.661 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:16.659+0000 7f61cc000640 1 --2- 172.21.15.32:0/2506901010 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f61c40747b0 0x7f61c4134710 unknown :-1 s=CLOSED pgs=666 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:51:16.661 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:16.659+0000 7f61cc000640 1 --2- 172.21.15.32:0/2506901010 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f61c406e930 0x7f61c41f9100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:51:16.661 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:16.659+0000 7f61cc000640 1 -- 172.21.15.32:0/2506901010 >> 172.21.15.32:0/2506901010 conn(0x7f61c40b1bd0 msgr2=0x7f61c40b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:51:16.661 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:16.659+0000 7f61cc000640 1 -- 172.21.15.32:0/2506901010 shutdown_connections 2024-08-03T20:51:16.661 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:16.659+0000 7f61cc000640 1 -- 172.21.15.32:0/2506901010 wait complete. 2024-08-03T20:51:17.086 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.c has been restored 2024-08-03T20:51:17.087 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 64, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-08-03T20:50:27.688698+0000', 'modified': '2024-08-03T20:51:14.996411+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 59, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 5632}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_5632': {'gid': 5632, 'name': 'd', 'rank': 0, 'incarnation': 62, 'state': 'up:rejoin', 'state_seq': 15, 'addr': '172.21.15.118:6836/3720010453', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.118:6834', 'nonce': 3720010453}, {'type': 'v1', 'addr': '172.21.15.118:6836', 'nonce': 3720010453}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [9], 'metadata_pool': 8, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 0, 'qdb_cluster': []} 2024-08-03T20:51:17.088 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_5632:up:rejoin 2024-08-03T20:51:18.089 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-08-03T20:51:18.328 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:18.326+0000 7f5f9a400640 1 -- 172.21.15.32:0/2471343678 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5f94073e60 msgr2=0x7f5f94074270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:51:18.328 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:18.326+0000 7f5f9a400640 1 --2- 172.21.15.32:0/2471343678 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5f94073e60 0x7f5f94074270 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7f5f94071700 tx=0x7f5f7c02a560 comp rx=0 tx=0).stop 2024-08-03T20:51:18.328 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:18.326+0000 7f5f9a400640 1 -- 172.21.15.32:0/2471343678 shutdown_connections 2024-08-03T20:51:18.328 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:18.326+0000 7f5f9a400640 1 --2- 172.21.15.32:0/2471343678 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f5f940747b0 0x7f5f940cc8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:51:18.329 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:18.326+0000 7f5f9a400640 1 --2- 172.21.15.32:0/2471343678 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5f94073e60 0x7f5f94074270 unknown :-1 s=CLOSED pgs=667 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:51:18.329 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:18.326+0000 7f5f9a400640 1 --2- 172.21.15.32:0/2471343678 >> v2:172.21.15.118:3300/0 conn(0x7f5f9406e930 0x7f5f94144f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:51:18.329 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:18.326+0000 7f5f9a400640 1 -- 172.21.15.32:0/2471343678 >> 172.21.15.32:0/2471343678 conn(0x7f5f940b1bd0 msgr2=0x7f5f940b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:51:18.329 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:18.326+0000 7f5f9a400640 1 -- 172.21.15.32:0/2471343678 shutdown_connections 2024-08-03T20:51:18.329 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:18.326+0000 7f5f9a400640 1 -- 172.21.15.32:0/2471343678 wait complete. 2024-08-03T20:51:18.330 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:18.329+0000 7f5f9a400640 1 Processor -- start 2024-08-03T20:51:18.330 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:18.329+0000 7f5f9a400640 1 -- start start 2024-08-03T20:51:18.332 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:18.330+0000 7f5f9a400640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f5f9406e930 0x7f5f94155850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:51:18.333 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:18.330+0000 7f5f9a400640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f5f94073e60 0x7f5f941f9660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:51:18.333 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:18.330+0000 7f5f9a400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5f940747b0 0x7f5f940afb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:51:18.333 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:18.330+0000 7f5f9a400640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f5f940cf9c0 con 0x7f5f940747b0 2024-08-03T20:51:18.333 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:18.330+0000 7f5f9a400640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f5f940cf840 con 0x7f5f9406e930 2024-08-03T20:51:18.333 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:18.330+0000 7f5f9a400640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7f5f940cce20 con 0x7f5f94073e60 2024-08-03T20:51:18.333 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:18.330+0000 7f5f93e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5f940747b0 0x7f5f940afb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:51:18.333 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:18.331+0000 7f5f93e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5f940747b0 0x7f5f940afb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:54460/0 (socket says 172.21.15.32:54460) 2024-08-03T20:51:18.333 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:18.331+0000 7f5f93e00640 1 -- 172.21.15.32:0/1580051080 learned_addr learned my addr 172.21.15.32:0/1580051080 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:51:18.333 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:18.331+0000 7f5f8be00640 1 --2- 172.21.15.32:0/1580051080 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f5f94073e60 0x7f5f941f9660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:51:18.333 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:18.331+0000 7f5f93400640 1 --2- 172.21.15.32:0/1580051080 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f5f9406e930 0x7f5f94155850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:51:18.333 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:18.331+0000 7f5f93e00640 1 -- 172.21.15.32:0/1580051080 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f5f94073e60 msgr2=0x7f5f941f9660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:51:18.333 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:18.331+0000 7f5f93e00640 1 --2- 172.21.15.32:0/1580051080 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f5f94073e60 0x7f5f941f9660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:51:18.333 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:18.331+0000 7f5f93e00640 1 -- 172.21.15.32:0/1580051080 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f5f9406e930 msgr2=0x7f5f94155850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:51:18.334 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:18.331+0000 7f5f93e00640 1 --2- 172.21.15.32:0/1580051080 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f5f9406e930 0x7f5f94155850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:51:18.334 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:18.331+0000 7f5f93e00640 1 -- 172.21.15.32:0/1580051080 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5f941f9ba0 con 0x7f5f940747b0 2024-08-03T20:51:18.334 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:18.331+0000 7f5f93400640 1 --2- 172.21.15.32:0/1580051080 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f5f9406e930 0x7f5f94155850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-03T20:51:18.334 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:18.331+0000 7f5f93e00640 1 --2- 172.21.15.32:0/1580051080 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5f940747b0 0x7f5f940afb20 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7f5f8c00ea80 tx=0x7f5f8c00ef40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:51:18.335 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:18.332+0000 7f5f8b400640 1 -- 172.21.15.32:0/1580051080 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f8c004b20 con 0x7f5f940747b0 2024-08-03T20:51:18.335 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:18.332+0000 7f5f8b400640 1 -- 172.21.15.32:0/1580051080 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f5f8c004ce0 con 0x7f5f940747b0 2024-08-03T20:51:18.335 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:18.332+0000 7f5f8b400640 1 -- 172.21.15.32:0/1580051080 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f8c005280 con 0x7f5f940747b0 2024-08-03T20:51:18.335 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:18.332+0000 7f5f9a400640 1 -- 172.21.15.32:0/1580051080 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5f940ae170 con 0x7f5f940747b0 2024-08-03T20:51:18.335 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:18.332+0000 7f5f9a400640 1 -- 172.21.15.32:0/1580051080 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5f940ae730 con 0x7f5f940747b0 2024-08-03T20:51:18.335 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:18.333+0000 7f5f9a400640 1 -- 172.21.15.32:0/1580051080 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5f94074270 con 0x7f5f940747b0 2024-08-03T20:51:18.335 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:18.334+0000 7f5f8b400640 1 -- 172.21.15.32:0/1580051080 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f5f8c005420 con 0x7f5f940747b0 2024-08-03T20:51:18.337 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:18.336+0000 7f5f8b400640 1 --2- 172.21.15.32:0/1580051080 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f5f70077040 0x7f5f70079550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:51:18.337 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:18.336+0000 7f5f8b400640 1 -- 172.21.15.32:0/1580051080 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(59..59 src has 1..59) ==== 5881+0+0 (secure 0 0 0) 0x7f5f8c099450 con 0x7f5f940747b0 2024-08-03T20:51:18.337 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:18.336+0000 7f5f93400640 1 --2- 172.21.15.32:0/1580051080 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f5f70077040 0x7f5f70079550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:51:18.338 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:18.337+0000 7f5f93400640 1 --2- 172.21.15.32:0/1580051080 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f5f70077040 0x7f5f70079550 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f5f84009b80 tx=0x7f5f84005f90 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:51:18.341 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:18.340+0000 7f5f8b400640 1 -- 172.21.15.32:0/1580051080 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7f5f941f9ba0 con 0x7f5f940747b0 2024-08-03T20:51:18.586 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:18.584+0000 7f5f9a400640 1 -- 172.21.15.32:0/1580051080 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f5f940af8d0 con 0x7f5f940747b0 2024-08-03T20:51:18.586 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:18.585+0000 7f5f8b400640 1 -- 172.21.15.32:0/1580051080 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 65 v65) ==== 94+0+5231 (secure 0 0 0) 0x7f5f8c062590 con 0x7f5f940747b0 2024-08-03T20:51:18.587 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-03T20:51:18.587 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":65,"btime":"2024-08-03T20:51:16:994952+0000","default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5261,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6833/2498840646","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":2498840646},{"type":"v1","addr":"172.21.15.32:6833","nonce":2498840646}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":45},{"gid":5276,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6837/3142148286","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6835","nonce":3142148286},{"type":"v1","addr":"172.21.15.118:6837","nonce":3142148286}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":46},{"gid":6180,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6835/1934242557","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1934242557},{"type":"v1","addr":"172.21.15.32:6835","nonce":1934242557}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":63}],"filesystems":[{"mdsmap":{"epoch":65,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-08-03T20:50:27.688698+0000","modified":"2024-08-03T20:51:16.994940+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":59,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5632},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5632":{"gid":5632,"name":"d","rank":0,"incarnation":62,"state":"up:active","state_seq":16,"addr":"172.21.15.118:6836/3720010453","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6834","nonce":3720010453},{"type":"v1","addr":"172.21.15.118:6836","nonce":3720010453}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[9],"metadata_pool":8,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5632,"qdb_cluster":[5632]},"id":4}]} 2024-08-03T20:51:18.587 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 65 2024-08-03T20:51:18.589 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:18.587+0000 7f5f9a400640 1 -- 172.21.15.32:0/1580051080 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f5f70077040 msgr2=0x7f5f70079550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:51:18.589 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:18.587+0000 7f5f9a400640 1 --2- 172.21.15.32:0/1580051080 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f5f70077040 0x7f5f70079550 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f5f84009b80 tx=0x7f5f84005f90 comp rx=0 tx=0).stop 2024-08-03T20:51:18.589 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:18.587+0000 7f5f9a400640 1 -- 172.21.15.32:0/1580051080 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5f940747b0 msgr2=0x7f5f940afb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:51:18.589 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:18.587+0000 7f5f9a400640 1 --2- 172.21.15.32:0/1580051080 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5f940747b0 0x7f5f940afb20 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7f5f8c00ea80 tx=0x7f5f8c00ef40 comp rx=0 tx=0).stop 2024-08-03T20:51:18.589 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:18.587+0000 7f5f9a400640 1 -- 172.21.15.32:0/1580051080 shutdown_connections 2024-08-03T20:51:18.589 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:18.587+0000 7f5f9a400640 1 --2- 172.21.15.32:0/1580051080 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f5f70077040 0x7f5f70079550 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:51:18.589 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:18.587+0000 7f5f9a400640 1 --2- 172.21.15.32:0/1580051080 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5f940747b0 0x7f5f940afb20 unknown :-1 s=CLOSED pgs=668 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:51:18.589 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:18.587+0000 7f5f9a400640 1 --2- 172.21.15.32:0/1580051080 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f5f94073e60 0x7f5f941f9660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:51:18.589 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:18.587+0000 7f5f9a400640 1 --2- 172.21.15.32:0/1580051080 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f5f9406e930 0x7f5f94155850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:51:18.589 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:18.587+0000 7f5f9a400640 1 -- 172.21.15.32:0/1580051080 >> 172.21.15.32:0/1580051080 conn(0x7f5f940b1bd0 msgr2=0x7f5f940d0920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:51:18.590 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:18.587+0000 7f5f9a400640 1 -- 172.21.15.32:0/1580051080 shutdown_connections 2024-08-03T20:51:18.590 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:18.587+0000 7f5f9a400640 1 -- 172.21.15.32:0/1580051080 wait complete. 2024-08-03T20:51:19.014 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 65, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-08-03T20:50:27.688698+0000', 'modified': '2024-08-03T20:51:16.994940+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 59, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 5632}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_5632': {'gid': 5632, 'name': 'd', 'rank': 0, 'incarnation': 62, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.118:6836/3720010453', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.118:6834', 'nonce': 3720010453}, {'type': 'v1', 'addr': '172.21.15.118:6836', 'nonce': 3720010453}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [9], 'metadata_pool': 8, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 5632, 'qdb_cluster': [5632]} 2024-08-03T20:51:19.014 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-08-03T20:51:19.014 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.d status 2024-08-03T20:51:19.243 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:19.241+0000 7f5d5f200640 1 -- 172.21.15.32:0/1452867880 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f5d580cec30 msgr2=0x7f5d580cf040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:51:19.243 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:19.241+0000 7f5d5f200640 1 --2- 172.21.15.32:0/1452867880 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f5d580cec30 0x7f5d580cf040 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f5d54005fd0 tx=0x7f5d5402a820 comp rx=0 tx=0).stop 2024-08-03T20:51:19.243 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:19.241+0000 7f5d5f200640 1 -- 172.21.15.32:0/1452867880 shutdown_connections 2024-08-03T20:51:19.243 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:19.241+0000 7f5d5f200640 1 --2- 172.21.15.32:0/1452867880 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f5d580cec30 0x7f5d580cf040 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:51:19.244 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:19.241+0000 7f5d5f200640 1 --2- 172.21.15.32:0/1452867880 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f5d580d4bb0 0x7f5d580d1c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:51:19.244 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:19.241+0000 7f5d5f200640 1 -- 172.21.15.32:0/1452867880 >> 172.21.15.32:0/1452867880 conn(0x7f5d580d4090 msgr2=0x7f5d580d44a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:51:19.244 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:19.241+0000 7f5d5f200640 1 -- 172.21.15.32:0/1452867880 shutdown_connections 2024-08-03T20:51:19.244 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:19.241+0000 7f5d5f200640 1 -- 172.21.15.32:0/1452867880 wait complete. 2024-08-03T20:51:19.245 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:19.244+0000 7f5d5f200640 1 Processor -- start 2024-08-03T20:51:19.245 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:19.244+0000 7f5d5f200640 1 -- start start 2024-08-03T20:51:19.248 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:19.245+0000 7f5d5f200640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f5d580cec30 0x7f5d580cccd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:51:19.248 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:19.245+0000 7f5d5f200640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f5d580d2290 0x7f5d581611c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:51:19.248 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:19.245+0000 7f5d5f200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5d580d4bb0 0x7f5d580b78d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:51:19.248 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:19.245+0000 7f5d5f200640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f5d580cfb80 con 0x7f5d580d4bb0 2024-08-03T20:51:19.248 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:19.245+0000 7f5d5f200640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f5d580cfa00 con 0x7f5d580cec30 2024-08-03T20:51:19.248 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:19.245+0000 7f5d5f200640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7f5d580cfd00 con 0x7f5d580d2290 2024-08-03T20:51:19.248 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:19.246+0000 7f5d5e800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5d580d4bb0 0x7f5d580b78d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:51:19.248 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:19.246+0000 7f5d5e800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5d580d4bb0 0x7f5d580b78d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:54464/0 (socket says 172.21.15.32:54464) 2024-08-03T20:51:19.248 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:19.246+0000 7f5d5e800640 1 -- 172.21.15.32:0/3905434282 learned_addr learned my addr 172.21.15.32:0/3905434282 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:51:19.248 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:19.246+0000 7f5d5de00640 1 --2- 172.21.15.32:0/3905434282 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f5d580cec30 0x7f5d580cccd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:51:19.248 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:19.246+0000 7f5d5d400640 1 --2- 172.21.15.32:0/3905434282 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f5d580d2290 0x7f5d581611c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:51:19.248 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:19.246+0000 7f5d5e800640 1 -- 172.21.15.32:0/3905434282 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f5d580d2290 msgr2=0x7f5d581611c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:51:19.249 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:19.246+0000 7f5d5e800640 1 --2- 172.21.15.32:0/3905434282 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f5d580d2290 0x7f5d581611c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:51:19.249 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:19.246+0000 7f5d5e800640 1 -- 172.21.15.32:0/3905434282 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f5d580cec30 msgr2=0x7f5d580cccd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:51:19.249 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:19.246+0000 7f5d5e800640 1 --2- 172.21.15.32:0/3905434282 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f5d580cec30 0x7f5d580cccd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:51:19.249 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:19.246+0000 7f5d5e800640 1 -- 172.21.15.32:0/3905434282 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5d580cd210 con 0x7f5d580d4bb0 2024-08-03T20:51:19.249 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:19.246+0000 7f5d5e800640 1 --2- 172.21.15.32:0/3905434282 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5d580d4bb0 0x7f5d580b78d0 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7f5d54009b30 tx=0x7f5d54032be0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:51:19.249 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:19.247+0000 7f5d52000640 1 -- 172.21.15.32:0/3905434282 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d54037040 con 0x7f5d580d4bb0 2024-08-03T20:51:19.250 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:19.247+0000 7f5d52000640 1 -- 172.21.15.32:0/3905434282 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f5d54042070 con 0x7f5d580d4bb0 2024-08-03T20:51:19.250 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:19.247+0000 7f5d52000640 1 -- 172.21.15.32:0/3905434282 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d54004040 con 0x7f5d580d4bb0 2024-08-03T20:51:19.250 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:19.247+0000 7f5d5f200640 1 -- 172.21.15.32:0/3905434282 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5d5815fa30 con 0x7f5d580d4bb0 2024-08-03T20:51:19.250 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:19.247+0000 7f5d5f200640 1 -- 172.21.15.32:0/3905434282 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5d580b8f40 con 0x7f5d580d4bb0 2024-08-03T20:51:19.250 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:19.249+0000 7f5d52000640 1 -- 172.21.15.32:0/3905434282 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f5d540041e0 con 0x7f5d580d4bb0 2024-08-03T20:51:19.252 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:19.250+0000 7f5d52000640 1 --2- 172.21.15.32:0/3905434282 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f5d3c077010 0x7f5d3c079520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:51:19.252 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:19.250+0000 7f5d52000640 1 -- 172.21.15.32:0/3905434282 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(59..59 src has 1..59) ==== 5881+0+0 (secure 0 0 0) 0x7f5d540b9500 con 0x7f5d580d4bb0 2024-08-03T20:51:19.253 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:19.251+0000 7f5d5de00640 1 --2- 172.21.15.32:0/3905434282 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f5d3c077010 0x7f5d3c079520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:51:19.253 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:19.251+0000 7f5d679af740 1 Processor -- start 2024-08-03T20:51:19.253 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:19.251+0000 7f5d679af740 1 -- start start 2024-08-03T20:51:19.253 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:19.251+0000 7f5d5de00640 1 --2- 172.21.15.32:0/3905434282 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f5d3c077010 0x7f5d3c079520 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f5d4c0095f0 tx=0x7f5d4c005f90 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:51:19.255 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:19.252+0000 7f5d679af740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55fb3446c270 0x55fb348592c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:51:19.255 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:19.253+0000 7f5d679af740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55fb344433b0 0x55fb3483bf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:51:19.255 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:19.253+0000 7f5d679af740 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55fb3444b590 0x55fb3483c4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:51:19.255 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:19.253+0000 7f5d679af740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55fb34784a60 con 0x55fb344433b0 2024-08-03T20:51:19.255 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:19.253+0000 7f5d679af740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x55fb34773f60 con 0x55fb3446c270 2024-08-03T20:51:19.255 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:19.253+0000 7f5d679af740 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x55fb34777fa0 con 0x55fb3444b590 2024-08-03T20:51:19.255 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:19.253+0000 7f5d5d400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55fb344433b0 0x55fb3483bf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:51:19.255 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:19.253+0000 7f5d5d400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55fb344433b0 0x55fb3483bf90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:54474/0 (socket says 172.21.15.32:54474) 2024-08-03T20:51:19.255 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:19.253+0000 7f5d5d400640 1 -- 172.21.15.32:0/4061178265 learned_addr learned my addr 172.21.15.32:0/4061178265 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:51:19.255 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:19.253+0000 7f5d5de00640 1 --2- 172.21.15.32:0/4061178265 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55fb3446c270 0x55fb348592c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:51:19.256 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:19.253+0000 7f5d5e800640 1 --2- 172.21.15.32:0/4061178265 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55fb3444b590 0x55fb3483c4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:51:19.256 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:19.253+0000 7f5d5d400640 1 -- 172.21.15.32:0/4061178265 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55fb3444b590 msgr2=0x55fb3483c4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:51:19.256 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:19.253+0000 7f5d5d400640 1 --2- 172.21.15.32:0/4061178265 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55fb3444b590 0x55fb3483c4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:51:19.256 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:19.253+0000 7f5d5d400640 1 -- 172.21.15.32:0/4061178265 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55fb3446c270 msgr2=0x55fb348592c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:51:19.256 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:19.253+0000 7f5d5d400640 1 --2- 172.21.15.32:0/4061178265 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55fb3446c270 0x55fb348592c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:51:19.256 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:19.253+0000 7f5d5d400640 1 -- 172.21.15.32:0/4061178265 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fb344d7850 con 0x55fb344433b0 2024-08-03T20:51:19.256 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:19.254+0000 7f5d5d400640 1 --2- 172.21.15.32:0/4061178265 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55fb344433b0 0x55fb3483bf90 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7f5d48001780 tx=0x7f5d48001c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bd92dd05f390c561 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:51:19.257 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:19.254+0000 7f5d46600640 1 -- 172.21.15.32:0/4061178265 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d48018020 con 0x55fb344433b0 2024-08-03T20:51:19.257 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:19.254+0000 7f5d46600640 1 -- 172.21.15.32:0/4061178265 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f5d4800fab0 con 0x55fb344433b0 2024-08-03T20:51:19.257 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:19.254+0000 7f5d46600640 1 -- 172.21.15.32:0/4061178265 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d48014470 con 0x55fb344433b0 2024-08-03T20:51:19.258 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:19.255+0000 7f5d679af740 1 -- 172.21.15.32:0/4061178265 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55fb344433b0 msgr2=0x55fb3483bf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:51:19.258 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:19.255+0000 7f5d679af740 1 --2- 172.21.15.32:0/4061178265 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55fb344433b0 0x55fb3483bf90 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7f5d48001780 tx=0x7f5d48001c40 comp rx=0 tx=0).stop 2024-08-03T20:51:19.258 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:19.255+0000 7f5d679af740 1 -- 172.21.15.32:0/4061178265 shutdown_connections 2024-08-03T20:51:19.258 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:19.255+0000 7f5d679af740 1 --2- 172.21.15.32:0/4061178265 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55fb3444b590 0x55fb3483c4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:51:19.258 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:19.255+0000 7f5d679af740 1 --2- 172.21.15.32:0/4061178265 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55fb344433b0 0x55fb3483bf90 unknown :-1 s=CLOSED pgs=670 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:51:19.258 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:19.255+0000 7f5d679af740 1 --2- 172.21.15.32:0/4061178265 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55fb3446c270 0x55fb348592c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:51:19.258 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:19.255+0000 7f5d679af740 1 -- 172.21.15.32:0/4061178265 >> 172.21.15.32:0/4061178265 conn(0x55fb344603c0 msgr2=0x55fb348121b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:51:19.258 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:19.255+0000 7f5d679af740 1 -- 172.21.15.32:0/4061178265 shutdown_connections 2024-08-03T20:51:19.258 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:19.255+0000 7f5d679af740 1 -- 172.21.15.32:0/4061178265 wait complete. 2024-08-03T20:51:19.258 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:19.256+0000 7f5d679af740 1 -- start start 2024-08-03T20:51:19.258 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:19.256+0000 7f5d679af740 1 Processor -- start 2024-08-03T20:51:19.261 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:19.258+0000 7f5d679af740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55fb344433b0 0x55fb3482a720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:51:19.261 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:19.258+0000 7f5d679af740 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55fb3444b590 0x55fb348501d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:51:19.261 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:19.258+0000 7f5d679af740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55fb3446c270 0x55fb34850710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:51:19.262 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:19.258+0000 7f5d679af740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55fb344e6620 con 0x55fb3446c270 2024-08-03T20:51:19.262 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:19.258+0000 7f5d679af740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x55fb34711160 con 0x55fb344433b0 2024-08-03T20:51:19.262 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:19.258+0000 7f5d679af740 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x55fb3484f130 con 0x55fb3444b590 2024-08-03T20:51:19.262 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:19.258+0000 7f5d679af740 10 client.0 _ll_register_callbacks cb 0x55fb347206b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-03T20:51:19.262 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:19.258+0000 7f5d5e800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55fb3446c270 0x55fb34850710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:51:19.262 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:19.258+0000 7f5d5e800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55fb3446c270 0x55fb34850710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:54490/0 (socket says 172.21.15.32:54490) 2024-08-03T20:51:19.262 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:19.258+0000 7f5d5e800640 1 -- 172.21.15.32:0/750603888 learned_addr learned my addr 172.21.15.32:0/750603888 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:51:19.262 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:19.258+0000 7f5d5de00640 1 --2- 172.21.15.32:0/750603888 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55fb344433b0 0x55fb3482a720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:51:19.262 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:19.258+0000 7f5d5d400640 1 --2- 172.21.15.32:0/750603888 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55fb3444b590 0x55fb348501d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:51:19.262 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:19.258+0000 7f5d5e800640 1 -- 172.21.15.32:0/750603888 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55fb3444b590 msgr2=0x55fb348501d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:51:19.262 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:19.258+0000 7f5d5e800640 1 --2- 172.21.15.32:0/750603888 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55fb3444b590 0x55fb348501d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:51:19.262 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:19.258+0000 7f5d5e800640 1 -- 172.21.15.32:0/750603888 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55fb344433b0 msgr2=0x55fb3482a720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:51:19.263 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:19.258+0000 7f5d5e800640 1 --2- 172.21.15.32:0/750603888 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55fb344433b0 0x55fb3482a720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:51:19.263 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:19.258+0000 7f5d5e800640 1 -- 172.21.15.32:0/750603888 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fb3454e6b0 con 0x55fb3446c270 2024-08-03T20:51:19.263 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:19.259+0000 7f5d5e800640 1 --2- 172.21.15.32:0/750603888 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55fb3446c270 0x55fb34850710 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7f5d54081d80 tx=0x7f5d540817f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=afee389dc1d7f953 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:51:19.263 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:19.259+0000 7f5d3aa00640 10 client.0 ms_handle_connect on v2:172.21.15.32:3300/0 2024-08-03T20:51:19.263 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:19.259+0000 7f5d3aa00640 1 -- 172.21.15.32:0/750603888 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d540865d0 con 0x55fb3446c270 2024-08-03T20:51:19.263 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:19.259+0000 7f5d679af740 1 -- 172.21.15.32:0/750603888 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5d4800fab0 con 0x55fb3446c270 2024-08-03T20:51:19.263 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:19.259+0000 7f5d3aa00640 1 -- 172.21.15.32:0/750603888 <== mon.0 v2:172.21.15.32:3300/0 2 ==== osd_map(59..59 src has 1..59) ==== 5881+0+0 (secure 0 0 0) 0x7f5d5408ab60 con 0x55fb3446c270 2024-08-03T20:51:19.263 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:19.259+0000 7f5d3aa00640 1 -- 172.21.15.32:0/750603888 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_get_version_reply(handle=1 version=65) ==== 24+0+0 (secure 0 0 0) 0x7f5d5408aff0 con 0x55fb3446c270 2024-08-03T20:51:19.263 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:19.259+0000 7f5d679af740 10 client.6210 fetch_fsmap learned FSMap version 65 2024-08-03T20:51:19.263 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:19.259+0000 7f5d679af740 1 -- 172.21.15.32:0/750603888 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,fsmap=65,monmap=2+}) -- 0x55fb3484ee70 con 0x55fb3446c270 2024-08-03T20:51:19.263 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:19.259+0000 7f5d3aa00640 1 -- 172.21.15.32:0/750603888 <== mon.0 v2:172.21.15.32:3300/0 4 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f5d4800fab0 con 0x55fb3446c270 2024-08-03T20:51:19.263 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:19.260+0000 7f5d3aa00640 1 -- 172.21.15.32:0/750603888 <== mon.0 v2:172.21.15.32:3300/0 5 ==== fsmap(e 65) ==== 3502+0+0 (secure 0 0 0) 0x7f5d5403a030 con 0x55fb3446c270 2024-08-03T20:51:19.263 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:19.260+0000 7f5d679af740 10 client.6210 fetch_fsmap finished waiting for FSMap version 65 2024-08-03T20:51:19.263 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:19.260+0000 7f5d679af740 10 client.6210 resolve_mds: resolved name 'd' to daemon mds.d 2024-08-03T20:51:19.264 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:19.260+0000 7f5d679af740 20 client.6210 populate_metadata read hostname 'smithi032' 2024-08-03T20:51:19.264 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:19.260+0000 7f5d679af740 1 --2- 172.21.15.32:0/750603888 >> [v2:172.21.15.118:6834/3720010453,v1:172.21.15.118:6836/3720010453] conn(0x55fb3442ae40 0x55fb3484de90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:51:19.264 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:19.260+0000 7f5d679af740 4 client.6210 mds_command: new command op to 5632 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-08-03T20:51:19.264 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:19.260+0000 7f5d679af740 1 -- 172.21.15.32:0/750603888 --> [v2:172.21.15.118:6834/3720010453,v1:172.21.15.118:6836/3720010453] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55fb347869b0 con 0x55fb3442ae40 2024-08-03T20:51:19.264 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:19.260+0000 7f5d5de00640 1 --2- 172.21.15.32:0/750603888 >> [v2:172.21.15.118:6834/3720010453,v1:172.21.15.118:6836/3720010453] conn(0x55fb3442ae40 0x55fb3484de90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:51:19.265 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:19.261+0000 7f5d5de00640 1 --2- 172.21.15.32:0/750603888 >> [v2:172.21.15.118:6834/3720010453,v1:172.21.15.118:6836/3720010453] conn(0x55fb3442ae40 0x55fb3484de90 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ec44a6646d014b0a server_cookie=3af6fad46dbffeb5 in_seq=0 out_seq=0 2024-08-03T20:51:19.265 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:19.261+0000 7f5d3aa00640 10 client.6210 ms_handle_connect on v2:172.21.15.118:6834/3720010453 2024-08-03T20:51:19.265 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:19.262+0000 7f5d3aa00640 1 -- 172.21.15.32:0/750603888 <== mds.0 v2:172.21.15.118:6834/3720010453 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55fb347869b0 con 0x55fb3442ae40 2024-08-03T20:51:19.265 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:19.262+0000 7f5d3aa00640 10 client.6210 handle_command_reply: tid=0 2024-08-03T20:51:19.265 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:19.262+0000 7f5d679af740 1 client.6210 shutdown 2024-08-03T20:51:19.266 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:19.262+0000 7f5d679af740 1 -- 172.21.15.32:0/750603888 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55fb3446c270 msgr2=0x55fb34850710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:51:19.266 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:19.262+0000 7f5d679af740 1 --2- 172.21.15.32:0/750603888 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55fb3446c270 0x55fb34850710 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7f5d54081d80 tx=0x7f5d540817f0 comp rx=0 tx=0).stop 2024-08-03T20:51:19.266 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:19.262+0000 7f5d679af740 1 -- 172.21.15.32:0/750603888 shutdown_connections 2024-08-03T20:51:19.266 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:19.262+0000 7f5d679af740 1 --2- 172.21.15.32:0/750603888 >> [v2:172.21.15.118:6834/3720010453,v1:172.21.15.118:6836/3720010453] conn(0x55fb3442ae40 0x55fb3484de90 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:51:19.266 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:19.262+0000 7f5d679af740 1 --2- 172.21.15.32:0/750603888 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55fb3446c270 0x55fb34850710 unknown :-1 s=CLOSED pgs=671 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:51:19.266 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:19.262+0000 7f5d679af740 1 --2- 172.21.15.32:0/750603888 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55fb3444b590 0x55fb348501d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:51:19.266 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:19.262+0000 7f5d3aa00640 0 client.6210 ms_handle_reset on v2:172.21.15.118:6834/3720010453 2024-08-03T20:51:19.266 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:19.262+0000 7f5d679af740 1 --2- 172.21.15.32:0/750603888 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55fb344433b0 0x55fb3482a720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:51:19.266 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:19.262+0000 7f5d679af740 1 -- 172.21.15.32:0/750603888 >> 172.21.15.32:0/750603888 conn(0x55fb344603c0 msgr2=0x55fb3485a2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:51:19.266 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:19.262+0000 7f5d679af740 1 -- 172.21.15.32:0/750603888 shutdown_connections 2024-08-03T20:51:19.266 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:19.262+0000 7f5d679af740 1 -- 172.21.15.32:0/750603888 wait complete. 2024-08-03T20:51:19.266 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:19.262+0000 7f5d679af740 20 client.6210 trim_cache size 0 max 16384 2024-08-03T20:51:19.284 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:19.282+0000 7f5d679af740 1 Processor -- start 2024-08-03T20:51:19.284 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:19.282+0000 7f5d679af740 1 -- start start 2024-08-03T20:51:19.286 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:19.284+0000 7f5d679af740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55fb344433b0 0x55fb3487a010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:51:19.286 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:19.284+0000 7f5d679af740 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55fb3444b590 0x55fb34811e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:51:19.286 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:19.284+0000 7f5d679af740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55fb3446c270 0x55fb34812390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:51:19.286 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:19.284+0000 7f5d679af740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f5d581f9830 con 0x55fb344433b0 2024-08-03T20:51:19.286 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:19.284+0000 7f5d679af740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f5d58167850 con 0x55fb3446c270 2024-08-03T20:51:19.286 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:19.284+0000 7f5d679af740 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7f5d580d1600 con 0x55fb3444b590 2024-08-03T20:51:19.286 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:19.284+0000 7f5d5de00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55fb344433b0 0x55fb3487a010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:51:19.286 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:19.284+0000 7f5d5de00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55fb344433b0 0x55fb3487a010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:54502/0 (socket says 172.21.15.32:54502) 2024-08-03T20:51:19.286 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:19.284+0000 7f5d5de00640 1 -- 172.21.15.32:0/3887408072 learned_addr learned my addr 172.21.15.32:0/3887408072 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:51:19.286 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:19.284+0000 7f5d5d400640 1 --2- 172.21.15.32:0/3887408072 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55fb3444b590 0x55fb34811e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:51:19.287 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:19.284+0000 7f5d5e800640 1 --2- 172.21.15.32:0/3887408072 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55fb3446c270 0x55fb34812390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:51:19.287 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:19.284+0000 7f5d5de00640 1 -- 172.21.15.32:0/3887408072 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55fb3444b590 msgr2=0x55fb34811e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:51:19.287 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:19.284+0000 7f5d5de00640 1 --2- 172.21.15.32:0/3887408072 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55fb3444b590 0x55fb34811e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:51:19.287 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:19.284+0000 7f5d5de00640 1 -- 172.21.15.32:0/3887408072 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55fb3446c270 msgr2=0x55fb34812390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:51:19.287 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:19.284+0000 7f5d5de00640 1 --2- 172.21.15.32:0/3887408072 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55fb3446c270 0x55fb34812390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:51:19.287 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:19.284+0000 7f5d5de00640 1 -- 172.21.15.32:0/3887408072 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fb34839060 con 0x55fb344433b0 2024-08-03T20:51:19.288 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:19.285+0000 7f5d5de00640 1 --2- 172.21.15.32:0/3887408072 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55fb344433b0 0x55fb3487a010 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7f5d4c004420 tx=0x7f5d4c014000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7d8ff241f495961e server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:51:19.288 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:19.285+0000 7f5d45c00640 1 -- 172.21.15.32:0/3887408072 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d4c003d30 con 0x55fb344433b0 2024-08-03T20:51:19.288 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:19.285+0000 7f5d45c00640 1 -- 172.21.15.32:0/3887408072 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f5d4c032410 con 0x55fb344433b0 2024-08-03T20:51:19.289 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:19.285+0000 7f5d45c00640 1 -- 172.21.15.32:0/3887408072 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d4c032840 con 0x55fb344433b0 2024-08-03T20:51:19.289 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:19.286+0000 7f5d679af740 1 -- 172.21.15.32:0/3887408072 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55fb344433b0 msgr2=0x55fb3487a010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:51:19.289 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:19.286+0000 7f5d679af740 1 --2- 172.21.15.32:0/3887408072 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55fb344433b0 0x55fb3487a010 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7f5d4c004420 tx=0x7f5d4c014000 comp rx=0 tx=0).stop 2024-08-03T20:51:19.289 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:19.286+0000 7f5d679af740 1 -- 172.21.15.32:0/3887408072 shutdown_connections 2024-08-03T20:51:19.289 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:19.286+0000 7f5d679af740 1 --2- 172.21.15.32:0/3887408072 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55fb3446c270 0x55fb34812390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:51:19.289 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:19.286+0000 7f5d679af740 1 --2- 172.21.15.32:0/3887408072 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55fb3444b590 0x55fb34811e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:51:19.289 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:19.286+0000 7f5d679af740 1 --2- 172.21.15.32:0/3887408072 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55fb344433b0 0x55fb3487a010 unknown :-1 s=CLOSED pgs=672 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:51:19.289 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:19.286+0000 7f5d679af740 1 -- 172.21.15.32:0/3887408072 >> 172.21.15.32:0/3887408072 conn(0x55fb344603c0 msgr2=0x55fb3480c470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:51:19.289 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:19.286+0000 7f5d679af740 1 -- 172.21.15.32:0/3887408072 shutdown_connections 2024-08-03T20:51:19.289 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:19.286+0000 7f5d679af740 1 -- 172.21.15.32:0/3887408072 wait complete. 2024-08-03T20:51:19.289 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:19.287+0000 7f5d679af740 1 -- start start 2024-08-03T20:51:19.289 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:19.287+0000 7f5d679af740 1 Processor -- start 2024-08-03T20:51:19.292 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:19.288+0000 7f5d679af740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55fb344433b0 0x55fb348bcbc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:51:19.292 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:19.289+0000 7f5d679af740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55fb3444b590 0x55fb34811c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:51:19.292 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:19.289+0000 7f5d679af740 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55fb3446c270 0x55fb34812160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:51:19.292 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:19.289+0000 7f5d679af740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55fb348697d0 con 0x55fb3444b590 2024-08-03T20:51:19.292 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:19.289+0000 7f5d679af740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x55fb34850c50 con 0x55fb344433b0 2024-08-03T20:51:19.293 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:19.289+0000 7f5d679af740 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x55fb3486a4a0 con 0x55fb3446c270 2024-08-03T20:51:19.293 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:19.289+0000 7f5d679af740 10 client.0 _ll_register_callbacks cb 0x7f5d54081e50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-03T20:51:19.293 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:19.289+0000 7f5d5d400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55fb3444b590 0x55fb34811c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:51:19.293 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:19.289+0000 7f5d5d400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55fb3444b590 0x55fb34811c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:54514/0 (socket says 172.21.15.32:54514) 2024-08-03T20:51:19.293 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:19.289+0000 7f5d5d400640 1 -- 172.21.15.32:0/3703790043 learned_addr learned my addr 172.21.15.32:0/3703790043 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:51:19.293 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:19.289+0000 7f5d5de00640 1 --2- 172.21.15.32:0/3703790043 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55fb344433b0 0x55fb348bcbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:51:19.293 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:19.289+0000 7f5d5e800640 1 --2- 172.21.15.32:0/3703790043 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55fb3446c270 0x55fb34812160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:51:19.293 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:19.289+0000 7f5d5d400640 1 -- 172.21.15.32:0/3703790043 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55fb3446c270 msgr2=0x55fb34812160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:51:19.293 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:19.289+0000 7f5d5d400640 1 --2- 172.21.15.32:0/3703790043 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55fb3446c270 0x55fb34812160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:51:19.293 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:19.289+0000 7f5d5d400640 1 -- 172.21.15.32:0/3703790043 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55fb344433b0 msgr2=0x55fb348bcbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:51:19.293 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:19.289+0000 7f5d5d400640 1 --2- 172.21.15.32:0/3703790043 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55fb344433b0 0x55fb348bcbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:51:19.293 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:19.289+0000 7f5d5d400640 1 -- 172.21.15.32:0/3703790043 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fb34850a40 con 0x55fb3444b590 2024-08-03T20:51:19.293 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:19.289+0000 7f5d5de00640 1 --2- 172.21.15.32:0/3703790043 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55fb344433b0 0x55fb348bcbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-03T20:51:19.294 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:19.290+0000 7f5d5d400640 1 --2- 172.21.15.32:0/3703790043 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55fb3444b590 0x55fb34811c20 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7f5d48001eb0 tx=0x7f5d48004620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6d84814b7678bae8 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:51:19.294 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:19.290+0000 7f5d3aa00640 10 client.0 ms_handle_connect on v2:172.21.15.32:3300/0 2024-08-03T20:51:19.294 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:19.290+0000 7f5d3aa00640 1 -- 172.21.15.32:0/3703790043 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d4802c060 con 0x55fb3444b590 2024-08-03T20:51:19.294 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:19.290+0000 7f5d3aa00640 1 -- 172.21.15.32:0/3703790043 <== mon.0 v2:172.21.15.32:3300/0 2 ==== osd_map(59..59 src has 1..59) ==== 5881+0+0 (secure 0 0 0) 0x7f5d4801e430 con 0x55fb3444b590 2024-08-03T20:51:19.294 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:19.290+0000 7f5d679af740 1 -- 172.21.15.32:0/3703790043 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5d4c032410 con 0x55fb3444b590 2024-08-03T20:51:19.294 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:19.290+0000 7f5d3aa00640 1 -- 172.21.15.32:0/3703790043 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_get_version_reply(handle=1 version=65) ==== 24+0+0 (secure 0 0 0) 0x7f5d48032050 con 0x55fb3444b590 2024-08-03T20:51:19.294 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:19.290+0000 7f5d679af740 10 client.6216 fetch_fsmap learned FSMap version 65 2024-08-03T20:51:19.294 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:19.290+0000 7f5d679af740 1 -- 172.21.15.32:0/3703790043 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,fsmap=65,monmap=2+}) -- 0x55fb3486dd10 con 0x55fb3444b590 2024-08-03T20:51:19.294 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:19.290+0000 7f5d3aa00640 1 -- 172.21.15.32:0/3703790043 <== mon.0 v2:172.21.15.32:3300/0 4 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f5d4c032410 con 0x55fb3444b590 2024-08-03T20:51:19.294 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:19.290+0000 7f5d3aa00640 1 -- 172.21.15.32:0/3703790043 <== mon.0 v2:172.21.15.32:3300/0 5 ==== fsmap(e 65) ==== 3502+0+0 (secure 0 0 0) 0x7f5d48035020 con 0x55fb3444b590 2024-08-03T20:51:19.294 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:19.290+0000 7f5d679af740 10 client.6216 fetch_fsmap finished waiting for FSMap version 65 2024-08-03T20:51:19.294 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:19.290+0000 7f5d679af740 10 client.6216 resolve_mds: resolved name 'd' to daemon mds.d 2024-08-03T20:51:19.294 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:19.290+0000 7f5d679af740 20 client.6216 populate_metadata read hostname 'smithi032' 2024-08-03T20:51:19.295 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:19.290+0000 7f5d679af740 1 --2- 172.21.15.32:0/3703790043 >> [v2:172.21.15.118:6834/3720010453,v1:172.21.15.118:6836/3720010453] conn(0x55fb3482bc30 0x55fb34878c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:51:19.295 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:19.290+0000 7f5d679af740 4 client.6216 mds_command: new command op to 5632 tid=0 multi_id=0 [{"prefix": "status"}] 2024-08-03T20:51:19.295 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:19.290+0000 7f5d679af740 1 -- 172.21.15.32:0/3703790043 --> [v2:172.21.15.118:6834/3720010453,v1:172.21.15.118:6836/3720010453] -- command(tid 0: {"prefix": "status"}) -- 0x55fb34782450 con 0x55fb3482bc30 2024-08-03T20:51:19.295 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:19.291+0000 7f5d5de00640 1 --2- 172.21.15.32:0/3703790043 >> [v2:172.21.15.118:6834/3720010453,v1:172.21.15.118:6836/3720010453] conn(0x55fb3482bc30 0x55fb34878c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:51:19.296 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:19.292+0000 7f5d3aa00640 10 client.6216 ms_handle_connect on v2:172.21.15.118:6834/3720010453 2024-08-03T20:51:19.296 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:19.292+0000 7f5d5de00640 1 --2- 172.21.15.32:0/3703790043 >> [v2:172.21.15.118:6834/3720010453,v1:172.21.15.118:6836/3720010453] conn(0x55fb3482bc30 0x55fb34878c80 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e3a2051835747c91 server_cookie=370e69d2ca524bc4 in_seq=0 out_seq=0 2024-08-03T20:51:19.296 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:19.292+0000 7f5d3aa00640 1 -- 172.21.15.32:0/3703790043 <== mds.0 v2:172.21.15.118:6834/3720010453 1 ==== command_reply(tid 0: 0 ) ==== 8+0+331 (crc 0 0 0) 0x55fb34782450 con 0x55fb3482bc30 2024-08-03T20:51:19.296 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:19.292+0000 7f5d3aa00640 10 client.6216 handle_command_reply: tid=0 2024-08-03T20:51:19.296 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:19.292+0000 7f5d679af740 1 client.6216 shutdown 2024-08-03T20:51:19.296 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:19.292+0000 7f5d679af740 1 -- 172.21.15.32:0/3703790043 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55fb3444b590 msgr2=0x55fb34811c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:51:19.297 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:19.292+0000 7f5d679af740 1 --2- 172.21.15.32:0/3703790043 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55fb3444b590 0x55fb34811c20 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7f5d48001eb0 tx=0x7f5d48004620 comp rx=0 tx=0).stop 2024-08-03T20:51:19.297 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:19.292+0000 7f5d679af740 1 -- 172.21.15.32:0/3703790043 shutdown_connections 2024-08-03T20:51:19.297 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:19.292+0000 7f5d679af740 1 --2- 172.21.15.32:0/3703790043 >> [v2:172.21.15.118:6834/3720010453,v1:172.21.15.118:6836/3720010453] conn(0x55fb3482bc30 0x55fb34878c80 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:51:19.297 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:19.292+0000 7f5d679af740 1 --2- 172.21.15.32:0/3703790043 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55fb3446c270 0x55fb34812160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:51:19.297 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:19.292+0000 7f5d679af740 1 --2- 172.21.15.32:0/3703790043 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55fb3444b590 0x55fb34811c20 unknown :-1 s=CLOSED pgs=673 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:51:19.297 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:19.292+0000 7f5d679af740 1 --2- 172.21.15.32:0/3703790043 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55fb344433b0 0x55fb348bcbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:51:19.297 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:19.292+0000 7f5d3aa00640 0 client.6216 ms_handle_reset on v2:172.21.15.118:6834/3720010453 2024-08-03T20:51:19.297 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:19.292+0000 7f5d679af740 1 -- 172.21.15.32:0/3703790043 >> 172.21.15.32:0/3703790043 conn(0x55fb344603c0 msgr2=0x55fb3485a3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:51:19.297 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:19.292+0000 7f5d679af740 1 -- 172.21.15.32:0/3703790043 shutdown_connections 2024-08-03T20:51:19.297 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:19.292+0000 7f5d679af740 1 -- 172.21.15.32:0/3703790043 wait complete. 2024-08-03T20:51:19.297 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:19.292+0000 7f5d679af740 20 client.6216 trim_cache size 0 max 16384 2024-08-03T20:51:19.298 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-08-03T20:51:19.298 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_fsid": "40ab141a-01d7-49b9-ae3c-2edec9876e95", 2024-08-03T20:51:19.298 INFO:teuthology.orchestra.run.smithi032.stdout: "whoami": 0, 2024-08-03T20:51:19.298 INFO:teuthology.orchestra.run.smithi032.stdout: "id": 5632, 2024-08-03T20:51:19.298 INFO:teuthology.orchestra.run.smithi032.stdout: "want_state": "up:active", 2024-08-03T20:51:19.298 INFO:teuthology.orchestra.run.smithi032.stdout: "state": "up:active", 2024-08-03T20:51:19.298 INFO:teuthology.orchestra.run.smithi032.stdout: "fs_name": "cephfs", 2024-08-03T20:51:19.298 INFO:teuthology.orchestra.run.smithi032.stdout: "rank_uptime": 5.3038617840000004, 2024-08-03T20:51:19.298 INFO:teuthology.orchestra.run.smithi032.stdout: "mdsmap_epoch": 65, 2024-08-03T20:51:19.298 INFO:teuthology.orchestra.run.smithi032.stdout: "osdmap_epoch": 59, 2024-08-03T20:51:19.298 INFO:teuthology.orchestra.run.smithi032.stdout: "osdmap_epoch_barrier": 59, 2024-08-03T20:51:19.298 INFO:teuthology.orchestra.run.smithi032.stdout: "uptime": 56.093678869000001 2024-08-03T20:51:19.299 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-08-03T20:51:19.299 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:19.293+0000 7f5d5f200640 1 -- 172.21.15.32:0/3905434282 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f5d3c077010 msgr2=0x7f5d3c079520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:51:19.299 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:19.293+0000 7f5d5f200640 1 --2- 172.21.15.32:0/3905434282 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f5d3c077010 0x7f5d3c079520 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f5d4c0095f0 tx=0x7f5d4c005f90 comp rx=0 tx=0).stop 2024-08-03T20:51:19.300 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:19.293+0000 7f5d5f200640 1 -- 172.21.15.32:0/3905434282 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5d580d4bb0 msgr2=0x7f5d580b78d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:51:19.300 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:19.293+0000 7f5d5f200640 1 --2- 172.21.15.32:0/3905434282 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5d580d4bb0 0x7f5d580b78d0 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7f5d54009b30 tx=0x7f5d54032be0 comp rx=0 tx=0).stop 2024-08-03T20:51:19.300 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:19.293+0000 7f5d5f200640 1 -- 172.21.15.32:0/3905434282 shutdown_connections 2024-08-03T20:51:19.300 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:19.293+0000 7f5d5f200640 1 --2- 172.21.15.32:0/3905434282 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f5d3c077010 0x7f5d3c079520 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:51:19.300 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:19.293+0000 7f5d5f200640 1 --2- 172.21.15.32:0/3905434282 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5d580d4bb0 0x7f5d580b78d0 unknown :-1 s=CLOSED pgs=669 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:51:19.300 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:19.293+0000 7f5d5f200640 1 --2- 172.21.15.32:0/3905434282 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f5d580d2290 0x7f5d581611c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:51:19.300 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:19.293+0000 7f5d5f200640 1 --2- 172.21.15.32:0/3905434282 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f5d580cec30 0x7f5d580cccd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:51:19.300 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:19.293+0000 7f5d5f200640 1 -- 172.21.15.32:0/3905434282 >> 172.21.15.32:0/3905434282 conn(0x7f5d580d4090 msgr2=0x7f5d58162c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:51:19.300 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:19.293+0000 7f5d5f200640 1 -- 172.21.15.32:0/3905434282 shutdown_connections 2024-08-03T20:51:19.300 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:19.293+0000 7f5d5f200640 1 -- 172.21.15.32:0/3905434282 wait complete. 2024-08-03T20:51:20.138 INFO:tasks.ceph.mds.d.smithi118.stderr:2024-08-03T20:51:20.137+0000 7f66d1a00640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-08-03T20:51:20.734 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:20.733+0000 7f3aaf400640 1 --2- 172.21.15.32:0/294378650 >> [v2:172.21.15.32:6834/2963295201,v1:172.21.15.32:6835/2963295201] conn(0x55ca2c6281f0 0x55ca2c676ad0 unknown :-1 s=BANNER_CONNECTING pgs=14 cs=8 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:51:20.734 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:20.733+0000 7f3aaf400640 1 --2- 172.21.15.32:0/294378650 >> [v2:172.21.15.32:6834/2963295201,v1:172.21.15.32:6835/2963295201] conn(0x55ca2c6281f0 0x55ca2c676ad0 crc :-1 s=SESSION_RECONNECTING pgs=14 cs=8 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_session_reset received session reset full=1 2024-08-03T20:51:20.735 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:20.733+0000 7f3aaf400640 1 --2- 172.21.15.32:0/294378650 >> [v2:172.21.15.32:6834/2963295201,v1:172.21.15.32:6835/2963295201] conn(0x55ca2c6281f0 0x55ca2c676ad0 crc :-1 s=SESSION_RECONNECTING pgs=14 cs=8 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).reset_session 2024-08-03T20:51:20.735 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:20.733+0000 7f3a8fe00640 0 client.6138 ms_handle_remote_reset on v2:172.21.15.32:6834/2963295201 2024-08-03T20:51:20.735 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:20.733+0000 7f3aaf400640 1 -- 172.21.15.32:0/294378650 >> [v2:172.21.15.32:6834/2963295201,v1:172.21.15.32:6835/2963295201] conn(0x55ca2c6281f0 msgr2=0x55ca2c676ad0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 17 2024-08-03T20:51:20.735 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:20.733+0000 7f3aaf400640 1 -- 172.21.15.32:0/294378650 >> [v2:172.21.15.32:6834/2963295201,v1:172.21.15.32:6835/2963295201] conn(0x55ca2c6281f0 msgr2=0x55ca2c676ad0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-08-03T20:51:20.735 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:20.733+0000 7f3aaf400640 1 --2- 172.21.15.32:0/294378650 >> [v2:172.21.15.32:6834/2963295201,v1:172.21.15.32:6835/2963295201] conn(0x55ca2c6281f0 0x55ca2c676ad0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-08-03T20:51:20.735 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:20.733+0000 7f3aaf400640 1 --2- 172.21.15.32:0/294378650 >> [v2:172.21.15.32:6834/2963295201,v1:172.21.15.32:6835/2963295201] conn(0x55ca2c6281f0 0x55ca2c676ad0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-08-03T20:51:35.739 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:35.738+0000 7f3aaf400640 1 --2- 172.21.15.32:0/294378650 >> [v2:172.21.15.32:6834/2963295201,v1:172.21.15.32:6835/2963295201] conn(0x55ca2c6281f0 0x55ca2c676ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:51:35.739 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:35.739+0000 7f3aaf400640 1 -- 172.21.15.32:0/294378650 >> [v2:172.21.15.32:6834/2963295201,v1:172.21.15.32:6835/2963295201] conn(0x55ca2c6281f0 msgr2=0x55ca2c676ad0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 17 2024-08-03T20:51:35.739 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:35.739+0000 7f3aaf400640 1 -- 172.21.15.32:0/294378650 >> [v2:172.21.15.32:6834/2963295201,v1:172.21.15.32:6835/2963295201] conn(0x55ca2c6281f0 msgr2=0x55ca2c676ad0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-08-03T20:51:35.740 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:35.739+0000 7f3aaf400640 1 --2- 172.21.15.32:0/294378650 >> [v2:172.21.15.32:6834/2963295201,v1:172.21.15.32:6835/2963295201] conn(0x55ca2c6281f0 0x55ca2c676ad0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-08-03T20:51:35.740 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:35.739+0000 7f3aaf400640 1 --2- 172.21.15.32:0/294378650 >> [v2:172.21.15.32:6834/2963295201,v1:172.21.15.32:6835/2963295201] conn(0x55ca2c6281f0 0x55ca2c676ad0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-08-03T20:51:37.134 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-08-03T20:51:37.139 DEBUG:teuthology.orchestra.run.smithi118:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-08-03T20:51:50.748 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:50.746+0000 7f3aaf400640 1 --2- 172.21.15.32:0/294378650 >> [v2:172.21.15.32:6834/2963295201,v1:172.21.15.32:6835/2963295201] conn(0x55ca2c6281f0 0x55ca2c676ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:51:50.749 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:50.747+0000 7f3aaf400640 1 -- 172.21.15.32:0/294378650 >> [v2:172.21.15.32:6834/2963295201,v1:172.21.15.32:6835/2963295201] conn(0x55ca2c6281f0 msgr2=0x55ca2c676ad0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 17 2024-08-03T20:51:50.749 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:50.747+0000 7f3aaf400640 1 -- 172.21.15.32:0/294378650 >> [v2:172.21.15.32:6834/2963295201,v1:172.21.15.32:6835/2963295201] conn(0x55ca2c6281f0 msgr2=0x55ca2c676ad0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-08-03T20:51:50.749 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:50.747+0000 7f3aaf400640 1 --2- 172.21.15.32:0/294378650 >> [v2:172.21.15.32:6834/2963295201,v1:172.21.15.32:6835/2963295201] conn(0x55ca2c6281f0 0x55ca2c676ad0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-08-03T20:51:50.749 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:51:50.747+0000 7f3aaf400640 1 --2- 172.21.15.32:0/294378650 >> [v2:172.21.15.32:6834/2963295201,v1:172.21.15.32:6835/2963295201] conn(0x55ca2c6281f0 0x55ca2c676ad0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-08-03T20:52:05.764 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:05.763+0000 7f3aaf400640 1 --2- 172.21.15.32:0/294378650 >> [v2:172.21.15.32:6834/2963295201,v1:172.21.15.32:6835/2963295201] conn(0x55ca2c6281f0 0x55ca2c676ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:52:05.765 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:05.763+0000 7f3aaf400640 1 -- 172.21.15.32:0/294378650 >> [v2:172.21.15.32:6834/2963295201,v1:172.21.15.32:6835/2963295201] conn(0x55ca2c6281f0 msgr2=0x55ca2c676ad0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 17 2024-08-03T20:52:05.765 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:05.763+0000 7f3aaf400640 1 -- 172.21.15.32:0/294378650 >> [v2:172.21.15.32:6834/2963295201,v1:172.21.15.32:6835/2963295201] conn(0x55ca2c6281f0 msgr2=0x55ca2c676ad0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-08-03T20:52:05.765 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:05.763+0000 7f3aaf400640 1 --2- 172.21.15.32:0/294378650 >> [v2:172.21.15.32:6834/2963295201,v1:172.21.15.32:6835/2963295201] conn(0x55ca2c6281f0 0x55ca2c676ad0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-08-03T20:52:05.765 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:05.763+0000 7f3aaf400640 1 --2- 172.21.15.32:0/294378650 >> [v2:172.21.15.32:6834/2963295201,v1:172.21.15.32:6835/2963295201] conn(0x55ca2c6281f0 0x55ca2c676ad0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-08-03T20:52:07.247 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-08-03T20:52:07.253 DEBUG:teuthology.orchestra.run.smithi118:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-08-03T20:52:20.781 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:20.780+0000 7f3aaf400640 1 --2- 172.21.15.32:0/294378650 >> [v2:172.21.15.32:6834/2963295201,v1:172.21.15.32:6835/2963295201] conn(0x55ca2c6281f0 0x55ca2c676ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:52:20.781 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:20.781+0000 7f3aaf400640 1 -- 172.21.15.32:0/294378650 >> [v2:172.21.15.32:6834/2963295201,v1:172.21.15.32:6835/2963295201] conn(0x55ca2c6281f0 msgr2=0x55ca2c676ad0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 17 2024-08-03T20:52:20.782 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:20.781+0000 7f3aaf400640 1 -- 172.21.15.32:0/294378650 >> [v2:172.21.15.32:6834/2963295201,v1:172.21.15.32:6835/2963295201] conn(0x55ca2c6281f0 msgr2=0x55ca2c676ad0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-08-03T20:52:20.782 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:20.781+0000 7f3aaf400640 1 --2- 172.21.15.32:0/294378650 >> [v2:172.21.15.32:6834/2963295201,v1:172.21.15.32:6835/2963295201] conn(0x55ca2c6281f0 0x55ca2c676ad0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-08-03T20:52:20.782 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:20.781+0000 7f3aaf400640 1 --2- 172.21.15.32:0/294378650 >> [v2:172.21.15.32:6834/2963295201,v1:172.21.15.32:6835/2963295201] conn(0x55ca2c6281f0 0x55ca2c676ad0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-08-03T20:52:35.797 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:35.796+0000 7f3aaf400640 1 --2- 172.21.15.32:0/294378650 >> [v2:172.21.15.32:6834/2963295201,v1:172.21.15.32:6835/2963295201] conn(0x55ca2c6281f0 0x55ca2c676ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:52:35.798 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:35.796+0000 7f3aaf400640 1 -- 172.21.15.32:0/294378650 >> [v2:172.21.15.32:6834/2963295201,v1:172.21.15.32:6835/2963295201] conn(0x55ca2c6281f0 msgr2=0x55ca2c676ad0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 17 2024-08-03T20:52:35.798 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:35.796+0000 7f3aaf400640 1 -- 172.21.15.32:0/294378650 >> [v2:172.21.15.32:6834/2963295201,v1:172.21.15.32:6835/2963295201] conn(0x55ca2c6281f0 msgr2=0x55ca2c676ad0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-08-03T20:52:35.798 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:35.796+0000 7f3aaf400640 1 --2- 172.21.15.32:0/294378650 >> [v2:172.21.15.32:6834/2963295201,v1:172.21.15.32:6835/2963295201] conn(0x55ca2c6281f0 0x55ca2c676ad0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-08-03T20:52:35.799 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:35.797+0000 7f3aaf400640 1 --2- 172.21.15.32:0/294378650 >> [v2:172.21.15.32:6834/2963295201,v1:172.21.15.32:6835/2963295201] conn(0x55ca2c6281f0 0x55ca2c676ad0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-08-03T20:52:37.378 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-08-03T20:52:37.384 DEBUG:teuthology.orchestra.run.smithi118:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-08-03T20:52:50.801 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:50.800+0000 7f3aaf400640 1 --2- 172.21.15.32:0/294378650 >> [v2:172.21.15.32:6834/2963295201,v1:172.21.15.32:6835/2963295201] conn(0x55ca2c6281f0 0x55ca2c676ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:52:50.802 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:50.800+0000 7f3aaf400640 1 -- 172.21.15.32:0/294378650 >> [v2:172.21.15.32:6834/2963295201,v1:172.21.15.32:6835/2963295201] conn(0x55ca2c6281f0 msgr2=0x55ca2c676ad0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 17 2024-08-03T20:52:50.802 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:50.800+0000 7f3aaf400640 1 -- 172.21.15.32:0/294378650 >> [v2:172.21.15.32:6834/2963295201,v1:172.21.15.32:6835/2963295201] conn(0x55ca2c6281f0 msgr2=0x55ca2c676ad0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-08-03T20:52:50.802 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:50.800+0000 7f3aaf400640 1 --2- 172.21.15.32:0/294378650 >> [v2:172.21.15.32:6834/2963295201,v1:172.21.15.32:6835/2963295201] conn(0x55ca2c6281f0 0x55ca2c676ad0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-08-03T20:52:50.802 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:50.800+0000 7f3aaf400640 1 --2- 172.21.15.32:0/294378650 >> [v2:172.21.15.32:6834/2963295201,v1:172.21.15.32:6835/2963295201] conn(0x55ca2c6281f0 0x55ca2c676ad0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-08-03T20:52:54.861 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-08-03T20:52:54.862 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-08-03T20:52:55.138 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:55.136+0000 7fbff5400640 1 -- 172.21.15.32:0/3047257142 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbff006e930 msgr2=0x7fbff0144f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:52:55.138 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:55.136+0000 7fbff5400640 1 --2- 172.21.15.32:0/3047257142 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbff006e930 0x7fbff0144f90 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7fbff00afaf0 tx=0x7fbfe002a560 comp rx=0 tx=0).stop 2024-08-03T20:52:55.139 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:55.136+0000 7fbff5400640 1 -- 172.21.15.32:0/3047257142 shutdown_connections 2024-08-03T20:52:55.139 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:55.136+0000 7fbff5400640 1 --2- 172.21.15.32:0/3047257142 >> v2:172.21.15.118:3300/0 conn(0x7fbff0073e60 0x7fbff0074270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:52:55.139 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:55.136+0000 7fbff5400640 1 --2- 172.21.15.32:0/3047257142 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbff006e930 0x7fbff0144f90 unknown :-1 s=CLOSED pgs=674 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:52:55.139 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:55.136+0000 7fbff5400640 1 -- 172.21.15.32:0/3047257142 >> 172.21.15.32:0/3047257142 conn(0x7fbff00b1bd0 msgr2=0x7fbff00b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:52:55.139 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:55.137+0000 7fbff5400640 1 -- 172.21.15.32:0/3047257142 shutdown_connections 2024-08-03T20:52:55.139 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:55.137+0000 7fbff5400640 1 -- 172.21.15.32:0/3047257142 wait complete. 2024-08-03T20:52:55.140 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:55.139+0000 7fbff5400640 1 Processor -- start 2024-08-03T20:52:55.140 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:55.139+0000 7fbff5400640 1 -- start start 2024-08-03T20:52:55.142 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:55.140+0000 7fbff5400640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fbff006e930 0x7fbff015b730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:52:55.142 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:55.140+0000 7fbff5400640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fbff0073e60 0x7fbff0143c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:52:55.142 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:55.141+0000 7fbff5400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbff00747b0 0x7fbff01e85d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:52:55.142 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:55.141+0000 7fbff5400640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fbff00cf9c0 con 0x7fbff00747b0 2024-08-03T20:52:55.143 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:55.141+0000 7fbff5400640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7fbff00cf840 con 0x7fbff006e930 2024-08-03T20:52:55.143 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:55.141+0000 7fbff5400640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7fbff00ccff0 con 0x7fbff0073e60 2024-08-03T20:52:55.143 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:55.141+0000 7fbfef200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbff00747b0 0x7fbff01e85d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:52:55.143 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:55.141+0000 7fbfe7e00640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fbff0073e60 0x7fbff0143c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:52:55.143 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:55.141+0000 7fbfee800640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fbff006e930 0x7fbff015b730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:52:55.143 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:55.141+0000 7fbfef200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbff00747b0 0x7fbff01e85d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:51262/0 (socket says 172.21.15.32:51262) 2024-08-03T20:52:55.143 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:55.141+0000 7fbfef200640 1 -- 172.21.15.32:0/553935437 learned_addr learned my addr 172.21.15.32:0/553935437 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:52:55.144 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:55.141+0000 7fbfef200640 1 -- 172.21.15.32:0/553935437 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fbff0073e60 msgr2=0x7fbff0143c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:52:55.144 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:55.141+0000 7fbfef200640 1 --2- 172.21.15.32:0/553935437 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fbff0073e60 0x7fbff0143c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:52:55.144 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:55.141+0000 7fbfef200640 1 -- 172.21.15.32:0/553935437 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fbff006e930 msgr2=0x7fbff015b730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:52:55.144 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:55.141+0000 7fbfef200640 1 --2- 172.21.15.32:0/553935437 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fbff006e930 0x7fbff015b730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:52:55.145 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:55.141+0000 7fbfef200640 1 -- 172.21.15.32:0/553935437 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbff01f96a0 con 0x7fbff00747b0 2024-08-03T20:52:55.145 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:55.142+0000 7fbfef200640 1 --2- 172.21.15.32:0/553935437 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbff00747b0 0x7fbff01e85d0 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7fbfe800c700 tx=0x7fbfe800cbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:52:55.145 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:55.142+0000 7fbfe6a00640 1 -- 172.21.15.32:0/553935437 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbfe8003b70 con 0x7fbff00747b0 2024-08-03T20:52:55.145 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:55.142+0000 7fbfe6a00640 1 -- 172.21.15.32:0/553935437 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fbfe8003d10 con 0x7fbff00747b0 2024-08-03T20:52:55.145 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:55.142+0000 7fbff5400640 1 -- 172.21.15.32:0/553935437 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbff0142bc0 con 0x7fbff00747b0 2024-08-03T20:52:55.145 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:55.142+0000 7fbfe6a00640 1 -- 172.21.15.32:0/553935437 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbfe800aab0 con 0x7fbff00747b0 2024-08-03T20:52:55.145 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:55.142+0000 7fbff5400640 1 -- 172.21.15.32:0/553935437 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbff0143180 con 0x7fbff00747b0 2024-08-03T20:52:55.145 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:55.143+0000 7fbff5400640 1 -- 172.21.15.32:0/553935437 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbff00cd580 con 0x7fbff00747b0 2024-08-03T20:52:55.146 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:55.144+0000 7fbfe6a00640 1 -- 172.21.15.32:0/553935437 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7fbfe800ac50 con 0x7fbff00747b0 2024-08-03T20:52:55.147 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:55.145+0000 7fbfe6a00640 1 --2- 172.21.15.32:0/553935437 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fbfcc077010 0x7fbfcc079520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:52:55.147 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:55.146+0000 7fbfe6a00640 1 -- 172.21.15.32:0/553935437 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(59..59 src has 1..59) ==== 5881+0+0 (secure 0 0 0) 0x7fbfe8098780 con 0x7fbff00747b0 2024-08-03T20:52:55.147 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:55.146+0000 7fbfee800640 1 --2- 172.21.15.32:0/553935437 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fbfcc077010 0x7fbfcc079520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:52:55.148 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:55.147+0000 7fbfee800640 1 --2- 172.21.15.32:0/553935437 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fbfcc077010 0x7fbfcc079520 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7fbfe0033000 tx=0x7fbfe0004730 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:52:55.151 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:55.150+0000 7fbfe6a00640 1 -- 172.21.15.32:0/553935437 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7fbfe80611e0 con 0x7fbff00747b0 2024-08-03T20:52:55.394 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:55.393+0000 7fbff5400640 1 -- 172.21.15.32:0/553935437 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7fbff00cc790 con 0x7fbff00747b0 2024-08-03T20:52:55.395 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:55.393+0000 7fbfe6a00640 1 -- 172.21.15.32:0/553935437 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v65) ==== 53+0+73 (secure 0 0 0) 0x7fbfe8065730 con 0x7fbff00747b0 2024-08-03T20:52:55.395 INFO:teuthology.orchestra.run.smithi032.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-08-03T20:52:55.398 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:55.396+0000 7fbff5400640 1 -- 172.21.15.32:0/553935437 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fbfcc077010 msgr2=0x7fbfcc079520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:52:55.398 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:55.396+0000 7fbff5400640 1 --2- 172.21.15.32:0/553935437 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fbfcc077010 0x7fbfcc079520 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7fbfe0033000 tx=0x7fbfe0004730 comp rx=0 tx=0).stop 2024-08-03T20:52:55.398 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:55.396+0000 7fbff5400640 1 -- 172.21.15.32:0/553935437 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbff00747b0 msgr2=0x7fbff01e85d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:52:55.398 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:55.396+0000 7fbff5400640 1 --2- 172.21.15.32:0/553935437 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbff00747b0 0x7fbff01e85d0 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7fbfe800c700 tx=0x7fbfe800cbc0 comp rx=0 tx=0).stop 2024-08-03T20:52:55.398 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:55.396+0000 7fbff5400640 1 -- 172.21.15.32:0/553935437 shutdown_connections 2024-08-03T20:52:55.398 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:55.396+0000 7fbff5400640 1 --2- 172.21.15.32:0/553935437 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fbfcc077010 0x7fbfcc079520 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:52:55.398 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:55.396+0000 7fbff5400640 1 --2- 172.21.15.32:0/553935437 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbff00747b0 0x7fbff01e85d0 unknown :-1 s=CLOSED pgs=675 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:52:55.398 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:55.396+0000 7fbff5400640 1 --2- 172.21.15.32:0/553935437 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fbff0073e60 0x7fbff0143c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:52:55.398 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:55.396+0000 7fbff5400640 1 --2- 172.21.15.32:0/553935437 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fbff006e930 0x7fbff015b730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:52:55.398 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:55.396+0000 7fbff5400640 1 -- 172.21.15.32:0/553935437 >> 172.21.15.32:0/553935437 conn(0x7fbff00b1bd0 msgr2=0x7fbff00b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:52:55.398 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:55.396+0000 7fbff5400640 1 -- 172.21.15.32:0/553935437 shutdown_connections 2024-08-03T20:52:55.399 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:55.396+0000 7fbff5400640 1 -- 172.21.15.32:0/553935437 wait complete. 2024-08-03T20:52:55.423 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-08-03T20:52:55.423 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-08-03T20:52:55.423 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi032.front.sepia.ceph.com 2024-08-03T20:52:55.423 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-08-03T20:52:55.423 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-08-03T20:52:55.423 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-08-03T20:52:55.423 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2024-08-03T20:52:55.423 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-08-03T20:52:55.423 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-08-03T20:52:55.424 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-08-03T20:52:55.424 DEBUG:teuthology.orchestra.run.smithi032:> ip addr 2024-08-03T20:52:55.460 INFO:teuthology.orchestra.run.smithi032.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-08-03T20:52:55.460 INFO:teuthology.orchestra.run.smithi032.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-08-03T20:52:55.460 INFO:teuthology.orchestra.run.smithi032.stdout: inet 127.0.0.1/8 scope host lo 2024-08-03T20:52:55.460 INFO:teuthology.orchestra.run.smithi032.stdout: valid_lft forever preferred_lft forever 2024-08-03T20:52:55.460 INFO:teuthology.orchestra.run.smithi032.stdout: inet6 ::1/128 scope host 2024-08-03T20:52:55.460 INFO:teuthology.orchestra.run.smithi032.stdout: valid_lft forever preferred_lft forever 2024-08-03T20:52:55.460 INFO:teuthology.orchestra.run.smithi032.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-08-03T20:52:55.461 INFO:teuthology.orchestra.run.smithi032.stdout: link/ether 0c:c4:7a:6b:fd:0c brd ff:ff:ff:ff:ff:ff 2024-08-03T20:52:55.461 INFO:teuthology.orchestra.run.smithi032.stdout: altname enp7s0f0 2024-08-03T20:52:55.461 INFO:teuthology.orchestra.run.smithi032.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-08-03T20:52:55.461 INFO:teuthology.orchestra.run.smithi032.stdout: link/ether 0c:c4:7a:6b:fd:0d brd ff:ff:ff:ff:ff:ff 2024-08-03T20:52:55.461 INFO:teuthology.orchestra.run.smithi032.stdout: altname enp7s0f1 2024-08-03T20:52:55.461 INFO:teuthology.orchestra.run.smithi032.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-08-03T20:52:55.461 INFO:teuthology.orchestra.run.smithi032.stdout: link/ether ac:1f:6b:f8:26:02 brd ff:ff:ff:ff:ff:ff 2024-08-03T20:52:55.461 INFO:teuthology.orchestra.run.smithi032.stdout: altname enp1s0f0 2024-08-03T20:52:55.461 INFO:teuthology.orchestra.run.smithi032.stdout: inet 172.21.15.32/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-08-03T20:52:55.461 INFO:teuthology.orchestra.run.smithi032.stdout: valid_lft 42181sec preferred_lft 42181sec 2024-08-03T20:52:55.461 INFO:teuthology.orchestra.run.smithi032.stdout: inet6 fe80::ae1f:6bff:fef8:2602/64 scope link noprefixroute 2024-08-03T20:52:55.461 INFO:teuthology.orchestra.run.smithi032.stdout: valid_lft forever preferred_lft forever 2024-08-03T20:52:55.462 INFO:teuthology.orchestra.run.smithi032.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-08-03T20:52:55.462 INFO:teuthology.orchestra.run.smithi032.stdout: link/ether ac:1f:6b:f8:26:03 brd ff:ff:ff:ff:ff:ff 2024-08-03T20:52:55.462 INFO:teuthology.orchestra.run.smithi032.stdout: altname enp1s0f1 2024-08-03T20:52:55.462 INFO:teuthology.orchestra.run.smithi032.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-08-03T20:52:55.462 INFO:teuthology.orchestra.run.smithi032.stdout: link/ether 0a:23:2b:ca:2a:93 brd ff:ff:ff:ff:ff:ff 2024-08-03T20:52:55.462 INFO:teuthology.orchestra.run.smithi032.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-08-03T20:52:55.462 INFO:teuthology.orchestra.run.smithi032.stdout: valid_lft forever preferred_lft forever 2024-08-03T20:52:55.462 INFO:teuthology.orchestra.run.smithi032.stdout: inet6 fe80::d804:7cff:fed9:7ba4/64 scope link 2024-08-03T20:52:55.462 INFO:teuthology.orchestra.run.smithi032.stdout: valid_lft forever preferred_lft forever 2024-08-03T20:52:55.462 INFO:teuthology.orchestra.run.smithi032.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-08-03T20:52:55.462 INFO:teuthology.orchestra.run.smithi032.stdout: link/ether 22:86:28:30:4e:62 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-08-03T20:52:55.462 INFO:teuthology.orchestra.run.smithi032.stdout: inet6 fe80::2086:28ff:fe30:4e62/64 scope link 2024-08-03T20:52:55.463 INFO:teuthology.orchestra.run.smithi032.stdout: valid_lft forever preferred_lft forever 2024-08-03T20:52:55.463 INFO:teuthology.orchestra.run.smithi032.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-08-03T20:52:55.463 INFO:teuthology.orchestra.run.smithi032.stdout: link/ether 0a:23:2b:ca:2a:93 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-08-03T20:52:55.463 INFO:teuthology.orchestra.run.smithi032.stdout: inet6 fe80::823:2bff:feca:2a93/64 scope link 2024-08-03T20:52:55.463 INFO:teuthology.orchestra.run.smithi032.stdout: valid_lft forever preferred_lft forever 2024-08-03T20:52:55.463 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-08-03T20:52:55.463 DEBUG:teuthology.orchestra.run.smithi032:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-08-03T20:52:55.554 INFO:teuthology.orchestra.run.smithi032.stdout:1 2024-08-03T20:52:55.557 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-08-03T20:52:55.557 DEBUG:teuthology.orchestra.run.smithi032:> route 2024-08-03T20:52:55.594 INFO:teuthology.orchestra.run.smithi032.stdout:Kernel IP routing table 2024-08-03T20:52:55.594 INFO:teuthology.orchestra.run.smithi032.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-08-03T20:52:55.594 INFO:teuthology.orchestra.run.smithi032.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-08-03T20:52:55.594 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-08-03T20:52:55.594 INFO:teuthology.orchestra.run.smithi032.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-08-03T20:52:55.595 DEBUG:teuthology.orchestra.run.smithi032:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-08-03T20:52:55.595 DEBUG:teuthology.orchestra.run.smithi032:> set -e 2024-08-03T20:52:55.595 DEBUG:teuthology.orchestra.run.smithi032:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-08-03T20:52:55.595 DEBUG:teuthology.orchestra.run.smithi032:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-08-03T20:52:55.595 DEBUG:teuthology.orchestra.run.smithi032:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-08-03T20:52:55.595 DEBUG:teuthology.orchestra.run.smithi032:> ') 2024-08-03T20:52:55.717 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:55 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-08-03T20:52:55.788 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:55 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-08-03T20:52:55.797 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-08-03T20:52:55.797 DEBUG:teuthology.orchestra.run.smithi032:> ip netns list 2024-08-03T20:52:55.832 INFO:teuthology.orchestra.run.smithi032.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-08-03T20:52:55.833 INFO:teuthology.orchestra.run.smithi032.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-08-03T20:52:55.833 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-08-03T20:52:55.833 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-08-03T20:52:55.833 DEBUG:teuthology.orchestra.run.smithi032:> sudo ip link set brx.0 up 2024-08-03T20:52:55.929 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-08-03T20:52:55.929 DEBUG:teuthology.orchestra.run.smithi032:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-08-03T20:52:55.964 INFO:teuthology.orchestra.run.smithi032.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-08-03T20:52:55.964 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-08-03T20:52:55.964 DEBUG:teuthology.orchestra.run.smithi032:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-08-03T20:52:56.039 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2024-08-03T20:52:56.039 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-08-03T20:52:56.039 DEBUG:teuthology.orchestra.run.smithi032:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-08-03T20:52:56.236 INFO:teuthology.orchestra.run.smithi032.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-08-03T20:52:56.236 INFO:teuthology.orchestra.run.smithi032.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-08-03T20:52:56.236 INFO:teuthology.orchestra.run.smithi032.stdout:invalid new device string format 2024-08-03T20:52:56.256 INFO:teuthology.orchestra.run.smithi032.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-08-03T20:52:56.257 INFO:teuthology.orchestra.run.smithi032.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-08-03T20:52:56.257 INFO:teuthology.orchestra.run.smithi032.stdout:invalid new device string format 2024-08-03T20:52:56.257 INFO:teuthology.orchestra.run.smithi032.stdout:mount.ceph: resolved to: "172.21.15.32:3300,172.21.15.118:3300,172.21.15.118:3301" 2024-08-03T20:52:56.257 INFO:teuthology.orchestra.run.smithi032.stdout:mount.ceph: switching to using v1 address with old syntax 2024-08-03T20:52:56.286 INFO:teuthology.orchestra.run.smithi032.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-08-03T20:52:56.287 INFO:teuthology.orchestra.run.smithi032.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-08-03T20:52:56.287 INFO:teuthology.orchestra.run.smithi032.stdout:invalid new device string format 2024-08-03T20:52:56.287 INFO:teuthology.orchestra.run.smithi032.stdout:mount.ceph: resolved to: "172.21.15.32:3300,172.21.15.118:3300,172.21.15.118:3301" 2024-08-03T20:52:56.287 INFO:teuthology.orchestra.run.smithi032.stdout:mount.ceph: switching to using v1 address with old syntax 2024-08-03T20:52:56.287 INFO:teuthology.orchestra.run.smithi032.stdout:mount.ceph: resolved to: "172.21.15.32:6789,172.21.15.118:6789,172.21.15.118:6790" 2024-08-03T20:52:56.287 INFO:teuthology.orchestra.run.smithi032.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.32:6789,172.21.15.118:6789,172.21.15.118:6790:/ 2024-08-03T20:52:56.287 INFO:teuthology.orchestra.run.smithi032.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,key=0,fsid=40ab141a-01d7-49b9-ae3c-2edec9876e95" will pass to kernel 2024-08-03T20:52:56.288 INFO:teuthology.orchestra.run.smithi032.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2024-08-03T20:52:56.288 INFO:teuthology.orchestra.run.smithi032.stdout: You just mounted a file system that supports labels which does not 2024-08-03T20:52:56.288 INFO:teuthology.orchestra.run.smithi032.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-08-03T20:52:56.288 INFO:teuthology.orchestra.run.smithi032.stdout: applications will generate AVC messages and not be allowed access to 2024-08-03T20:52:56.288 INFO:teuthology.orchestra.run.smithi032.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-08-03T20:52:56.291 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-08-03T20:52:56.292 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-08-03T20:52:56.292 DEBUG:teuthology.orchestra.run.smithi032:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-08-03T20:52:56.348 DEBUG:teuthology.orchestra.run.smithi032:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-08-03T20:52:56.427 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:56 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-08-03T20:52:56.430 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:56 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-08-03T20:52:56.439 INFO:teuthology.orchestra.run.smithi032.stdout:client6225 2024-08-03T20:52:56.439 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-08-03T20:52:56.439 DEBUG:teuthology.orchestra.run.smithi032:> dd if=/proc/self/mounts of=/dev/stdout 2024-08-03T20:52:56.475 DEBUG:teuthology.orchestra.run.smithi032:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'stat a/ && find a/') 2024-08-03T20:52:56.594 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:56 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-08-03T20:52:56.598 INFO:teuthology.orchestra.run.smithi032.stdout: File: a/ 2024-08-03T20:52:56.598 INFO:teuthology.orchestra.run.smithi032.stdout: Size: 2 Blocks: 0 IO Block: 65536 directory 2024-08-03T20:52:56.598 INFO:teuthology.orchestra.run.smithi032.stdout:Device: 29h/41d Inode: 1099511627776 Links: 4 2024-08-03T20:52:56.598 INFO:teuthology.orchestra.run.smithi032.stdout:Access: (0755/drwxr-xr-x) Uid: ( 1000/ ubuntu) Gid: ( 1316/ ubuntu) 2024-08-03T20:52:56.598 INFO:teuthology.orchestra.run.smithi032.stdout:Context: unconfined_u:object_r:unlabeled_t:s0 2024-08-03T20:52:56.598 INFO:teuthology.orchestra.run.smithi032.stdout:Access: 2024-08-03 20:50:36.740309402 +0000 2024-08-03T20:52:56.598 INFO:teuthology.orchestra.run.smithi032.stdout:Modify: 2024-08-03 20:50:36.746309281 +0000 2024-08-03T20:52:56.599 INFO:teuthology.orchestra.run.smithi032.stdout:Change: 2024-08-03 20:50:36.746309281 +0000 2024-08-03T20:52:56.599 INFO:teuthology.orchestra.run.smithi032.stdout: Birth: 2024-08-03 20:50:36.740309402 +0000 2024-08-03T20:52:56.608 INFO:teuthology.orchestra.run.smithi032.stdout:a/ 2024-08-03T20:52:56.608 INFO:teuthology.orchestra.run.smithi032.stdout:a/c 2024-08-03T20:52:56.608 INFO:teuthology.orchestra.run.smithi032.stdout:a/c/d 2024-08-03T20:52:56.608 INFO:teuthology.orchestra.run.smithi032.stdout:a/b 2024-08-03T20:52:56.608 INFO:teuthology.orchestra.run.smithi032.stdout:a/b/d 2024-08-03T20:52:56.609 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:56 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-08-03T20:52:56.617 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.4:0 flush journal 2024-08-03T20:52:56.849 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:56.847+0000 7f4e41a00640 1 -- 172.21.15.32:0/3898654880 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4e3c06e950 msgr2=0x7f4e3c06ed60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:52:56.849 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:56.847+0000 7f4e41a00640 1 --2- 172.21.15.32:0/3898654880 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4e3c06e950 0x7f4e3c06ed60 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7f4e24009830 tx=0x7f4e2402a510 comp rx=0 tx=0).stop 2024-08-03T20:52:56.849 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:56.847+0000 7f4e41a00640 1 -- 172.21.15.32:0/3898654880 shutdown_connections 2024-08-03T20:52:56.849 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:56.847+0000 7f4e41a00640 1 --2- 172.21.15.32:0/3898654880 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4e3c06e950 0x7f4e3c06ed60 unknown :-1 s=CLOSED pgs=678 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:52:56.849 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:56.847+0000 7f4e41a00640 1 --2- 172.21.15.32:0/3898654880 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f4e3c15d2b0 0x7f4e3c0ced30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:52:56.850 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:56.847+0000 7f4e41a00640 1 -- 172.21.15.32:0/3898654880 >> 172.21.15.32:0/3898654880 conn(0x7f4e3c0b1bd0 msgr2=0x7f4e3c0b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:52:56.850 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:56.847+0000 7f4e41a00640 1 -- 172.21.15.32:0/3898654880 shutdown_connections 2024-08-03T20:52:56.850 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:56.847+0000 7f4e41a00640 1 -- 172.21.15.32:0/3898654880 wait complete. 2024-08-03T20:52:56.851 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:56.850+0000 7f4e41a00640 1 Processor -- start 2024-08-03T20:52:56.851 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:56.850+0000 7f4e41a00640 1 -- start start 2024-08-03T20:52:56.853 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:56.851+0000 7f4e41a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4e3c06e950 0x7f4e3c0c5560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:52:56.853 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:56.851+0000 7f4e41a00640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f4e3c0b3e80 0x7f4e3c1f8a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:52:56.853 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:56.851+0000 7f4e41a00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f4e3c15d2b0 0x7f4e3c1f8fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:52:56.853 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:56.851+0000 7f4e41a00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f4e3c0cf490 con 0x7f4e3c06e950 2024-08-03T20:52:56.854 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:56.851+0000 7f4e41a00640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f4e3c0cd5a0 con 0x7f4e3c15d2b0 2024-08-03T20:52:56.854 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:56.851+0000 7f4e41a00640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7f4e3c0b4290 con 0x7f4e3c0b3e80 2024-08-03T20:52:56.854 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:56.851+0000 7f4e3aa00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4e3c06e950 0x7f4e3c0c5560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:52:56.854 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:56.852+0000 7f4e3aa00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4e3c06e950 0x7f4e3c0c5560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:51284/0 (socket says 172.21.15.32:51284) 2024-08-03T20:52:56.854 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:56.852+0000 7f4e3aa00640 1 -- 172.21.15.32:0/945754695 learned_addr learned my addr 172.21.15.32:0/945754695 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:52:56.854 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:56.852+0000 7f4e33e00640 1 --2- 172.21.15.32:0/945754695 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f4e3c0b3e80 0x7f4e3c1f8a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:52:56.854 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:56.852+0000 7f4e3b400640 1 --2- 172.21.15.32:0/945754695 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f4e3c15d2b0 0x7f4e3c1f8fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:52:56.854 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:56.852+0000 7f4e3aa00640 1 -- 172.21.15.32:0/945754695 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f4e3c0b3e80 msgr2=0x7f4e3c1f8a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:52:56.854 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:56.852+0000 7f4e3aa00640 1 --2- 172.21.15.32:0/945754695 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f4e3c0b3e80 0x7f4e3c1f8a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:52:56.854 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:56.852+0000 7f4e3aa00640 1 -- 172.21.15.32:0/945754695 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f4e3c15d2b0 msgr2=0x7f4e3c1f8fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:52:56.854 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:56.852+0000 7f4e3aa00640 1 --2- 172.21.15.32:0/945754695 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f4e3c15d2b0 0x7f4e3c1f8fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:52:56.854 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:56.852+0000 7f4e3aa00640 1 -- 172.21.15.32:0/945754695 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4e3c15f0c0 con 0x7f4e3c06e950 2024-08-03T20:52:56.855 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:56.852+0000 7f4e3aa00640 1 --2- 172.21.15.32:0/945754695 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4e3c06e950 0x7f4e3c0c5560 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7f4e2c00e790 tx=0x7f4e2c00ec50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:52:56.855 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:56.853+0000 7f4e32a00640 1 -- 172.21.15.32:0/945754695 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e2c00eea0 con 0x7f4e3c06e950 2024-08-03T20:52:56.856 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:56.853+0000 7f4e32a00640 1 -- 172.21.15.32:0/945754695 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f4e2c00cdb0 con 0x7f4e3c06e950 2024-08-03T20:52:56.856 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:56.853+0000 7f4e32a00640 1 -- 172.21.15.32:0/945754695 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e2c010690 con 0x7f4e3c06e950 2024-08-03T20:52:56.856 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:56.853+0000 7f4e41a00640 1 -- 172.21.15.32:0/945754695 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4e3c15f3a0 con 0x7f4e3c06e950 2024-08-03T20:52:56.856 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:56.853+0000 7f4e41a00640 1 -- 172.21.15.32:0/945754695 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4e3c15f960 con 0x7f4e3c06e950 2024-08-03T20:52:56.856 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:56.855+0000 7f4e32a00640 1 -- 172.21.15.32:0/945754695 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f4e2c010830 con 0x7f4e3c06e950 2024-08-03T20:52:56.857 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:56.856+0000 7f4e32a00640 1 --2- 172.21.15.32:0/945754695 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f4e18077040 0x7f4e18079550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:52:56.858 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:56.856+0000 7f4e32a00640 1 -- 172.21.15.32:0/945754695 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(59..59 src has 1..59) ==== 5881+0+0 (secure 0 0 0) 0x7f4e2c099330 con 0x7f4e3c06e950 2024-08-03T20:52:56.858 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:56.857+0000 7f4e4896b740 1 Processor -- start 2024-08-03T20:52:56.858 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:56.857+0000 7f4e33e00640 1 --2- 172.21.15.32:0/945754695 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f4e18077040 0x7f4e18079550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:52:56.858 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:56.857+0000 7f4e4896b740 1 -- start start 2024-08-03T20:52:56.859 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:56.857+0000 7f4e33e00640 1 --2- 172.21.15.32:0/945754695 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f4e18077040 0x7f4e18079550 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f4e24034000 tx=0x7f4e24004730 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:52:56.860 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:56.858+0000 7f4e4896b740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55ec0d303390 0x55ec0d6d3900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:52:56.860 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:56.858+0000 7f4e4896b740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ec0d2da4d0 0x55ec0d6d3e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:52:56.860 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:56.858+0000 7f4e4896b740 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55ec0d2e26b0 0x55ec0d6f0d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:52:56.861 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:56.858+0000 7f4e4896b740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55ec0d61c470 con 0x55ec0d2da4d0 2024-08-03T20:52:56.861 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:56.858+0000 7f4e4896b740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x55ec0d60b970 con 0x55ec0d303390 2024-08-03T20:52:56.861 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:56.858+0000 7f4e4896b740 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x55ec0d60f9b0 con 0x55ec0d2e26b0 2024-08-03T20:52:56.861 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:56.858+0000 7f4e33e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ec0d2da4d0 0x55ec0d6d3e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:52:56.861 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:56.858+0000 7f4e33e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ec0d2da4d0 0x55ec0d6d3e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:51292/0 (socket says 172.21.15.32:51292) 2024-08-03T20:52:56.861 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:56.858+0000 7f4e33e00640 1 -- 172.21.15.32:0/2128665217 learned_addr learned my addr 172.21.15.32:0/2128665217 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:52:56.861 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:56.858+0000 7f4e3aa00640 1 --2- 172.21.15.32:0/2128665217 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55ec0d303390 0x55ec0d6d3900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:52:56.861 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:56.859+0000 7f4e3b400640 1 --2- 172.21.15.32:0/2128665217 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55ec0d2e26b0 0x55ec0d6f0d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:52:56.861 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:56.859+0000 7f4e33e00640 1 -- 172.21.15.32:0/2128665217 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55ec0d2e26b0 msgr2=0x55ec0d6f0d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:52:56.861 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:56.859+0000 7f4e33e00640 1 --2- 172.21.15.32:0/2128665217 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55ec0d2e26b0 0x55ec0d6f0d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:52:56.861 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:56.859+0000 7f4e33e00640 1 -- 172.21.15.32:0/2128665217 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55ec0d303390 msgr2=0x55ec0d6d3900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:52:56.861 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:56.859+0000 7f4e33e00640 1 --2- 172.21.15.32:0/2128665217 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55ec0d303390 0x55ec0d6d3900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:52:56.861 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:56.859+0000 7f4e33e00640 1 -- 172.21.15.32:0/2128665217 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ec0d3e57d0 con 0x55ec0d2da4d0 2024-08-03T20:52:56.861 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:56.859+0000 7f4e3aa00640 1 --2- 172.21.15.32:0/2128665217 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55ec0d303390 0x55ec0d6d3900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-03T20:52:56.862 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:56.859+0000 7f4e33e00640 1 --2- 172.21.15.32:0/2128665217 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ec0d2da4d0 0x55ec0d6d3e40 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7f4e24009920 tx=0x7f4e2403e040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a1826a01ebf41bca server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:52:56.863 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:56.859+0000 7f4e2b400640 1 -- 172.21.15.32:0/2128665217 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e240038b0 con 0x55ec0d2da4d0 2024-08-03T20:52:56.863 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:56.859+0000 7f4e2b400640 1 -- 172.21.15.32:0/2128665217 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f4e24003a50 con 0x55ec0d2da4d0 2024-08-03T20:52:56.863 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:56.860+0000 7f4e2b400640 1 -- 172.21.15.32:0/2128665217 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e24045460 con 0x55ec0d2da4d0 2024-08-03T20:52:56.863 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:56.860+0000 7f4e4896b740 1 -- 172.21.15.32:0/2128665217 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ec0d2da4d0 msgr2=0x55ec0d6d3e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:52:56.863 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:56.860+0000 7f4e4896b740 1 --2- 172.21.15.32:0/2128665217 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ec0d2da4d0 0x55ec0d6d3e40 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7f4e24009920 tx=0x7f4e2403e040 comp rx=0 tx=0).stop 2024-08-03T20:52:56.863 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:56.860+0000 7f4e4896b740 1 -- 172.21.15.32:0/2128665217 shutdown_connections 2024-08-03T20:52:56.863 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:56.860+0000 7f4e4896b740 1 --2- 172.21.15.32:0/2128665217 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55ec0d2e26b0 0x55ec0d6f0d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:52:56.864 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:56.860+0000 7f4e4896b740 1 --2- 172.21.15.32:0/2128665217 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ec0d2da4d0 0x55ec0d6d3e40 unknown :-1 s=CLOSED pgs=680 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:52:56.864 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:56.860+0000 7f4e4896b740 1 --2- 172.21.15.32:0/2128665217 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55ec0d303390 0x55ec0d6d3900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:52:56.864 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:56.860+0000 7f4e4896b740 1 -- 172.21.15.32:0/2128665217 >> 172.21.15.32:0/2128665217 conn(0x55ec0d2f74e0 msgr2=0x55ec0d6a9bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:52:56.864 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:56.860+0000 7f4e4896b740 1 -- 172.21.15.32:0/2128665217 shutdown_connections 2024-08-03T20:52:56.864 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:56.860+0000 7f4e4896b740 1 -- 172.21.15.32:0/2128665217 wait complete. 2024-08-03T20:52:56.864 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:56.861+0000 7f4e4896b740 1 -- start start 2024-08-03T20:52:56.864 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:56.862+0000 7f4e4896b740 1 Processor -- start 2024-08-03T20:52:56.866 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:56.863+0000 7f4e4896b740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55ec0d2da4d0 0x55ec0d6e7be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:52:56.866 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:56.863+0000 7f4e4896b740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ec0d2e26b0 0x55ec0d700f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:52:56.866 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:56.863+0000 7f4e4896b740 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55ec0d303390 0x55ec0d6f1d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:52:56.866 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:56.863+0000 7f4e4896b740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55ec0d543c90 con 0x55ec0d2e26b0 2024-08-03T20:52:56.866 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:56.863+0000 7f4e4896b740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x55ec0d5a5fd0 con 0x55ec0d2da4d0 2024-08-03T20:52:56.866 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:56.863+0000 7f4e4896b740 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x55ec0d6d4380 con 0x55ec0d303390 2024-08-03T20:52:56.866 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:56.863+0000 7f4e4896b740 10 client.0 _ll_register_callbacks cb 0x55ec0d5c0e50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-03T20:52:56.866 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:56.863+0000 7f4e33e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ec0d2e26b0 0x55ec0d700f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:52:56.866 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:56.863+0000 7f4e33e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ec0d2e26b0 0x55ec0d700f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:51296/0 (socket says 172.21.15.32:51296) 2024-08-03T20:52:56.866 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:56.863+0000 7f4e33e00640 1 -- 172.21.15.32:0/1723496263 learned_addr learned my addr 172.21.15.32:0/1723496263 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:52:56.866 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:56.864+0000 7f4e3aa00640 1 --2- 172.21.15.32:0/1723496263 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55ec0d2da4d0 0x55ec0d6e7be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:52:56.867 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:56.864+0000 7f4e3b400640 1 --2- 172.21.15.32:0/1723496263 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55ec0d303390 0x55ec0d6f1d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:52:56.867 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:56.864+0000 7f4e33e00640 1 -- 172.21.15.32:0/1723496263 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55ec0d303390 msgr2=0x55ec0d6f1d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:52:56.867 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:56.864+0000 7f4e33e00640 1 --2- 172.21.15.32:0/1723496263 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55ec0d303390 0x55ec0d6f1d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:52:56.867 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:56.864+0000 7f4e33e00640 1 -- 172.21.15.32:0/1723496263 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55ec0d2da4d0 msgr2=0x55ec0d6e7be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:52:56.867 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:56.864+0000 7f4e33e00640 1 --2- 172.21.15.32:0/1723496263 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55ec0d2da4d0 0x55ec0d6e7be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:52:56.867 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:56.864+0000 7f4e33e00640 1 -- 172.21.15.32:0/1723496263 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ec0d36e970 con 0x55ec0d2e26b0 2024-08-03T20:52:56.867 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:56.864+0000 7f4e33e00640 1 --2- 172.21.15.32:0/1723496263 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ec0d2e26b0 0x55ec0d700f20 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 crypto rx=0x7f4e24009830 tx=0x7f4e24003e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=75a9b0a558738147 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:52:56.867 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:56.864+0000 7f4e1fe00640 10 client.0 ms_handle_connect on v2:172.21.15.32:3300/0 2024-08-03T20:52:56.867 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:56.864+0000 7f4e1fe00640 1 -- 172.21.15.32:0/1723496263 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e2405d060 con 0x55ec0d2e26b0 2024-08-03T20:52:56.867 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:56.865+0000 7f4e1fe00640 1 -- 172.21.15.32:0/1723496263 <== mon.0 v2:172.21.15.32:3300/0 2 ==== osd_map(59..59 src has 1..59) ==== 5881+0+0 (secure 0 0 0) 0x7f4e24045870 con 0x55ec0d2e26b0 2024-08-03T20:52:56.867 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:56.865+0000 7f4e4896b740 1 -- 172.21.15.32:0/1723496263 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4e24003a50 con 0x55ec0d2e26b0 2024-08-03T20:52:56.870 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:56.865+0000 7f4e1fe00640 1 -- 172.21.15.32:0/1723496263 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_get_version_reply(handle=1 version=65) ==== 24+0+0 (secure 0 0 0) 0x7f4e24045a70 con 0x55ec0d2e26b0 2024-08-03T20:52:56.870 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:56.865+0000 7f4e4896b740 10 client.6240 fetch_fsmap learned FSMap version 65 2024-08-03T20:52:56.870 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:56.865+0000 7f4e4896b740 1 -- 172.21.15.32:0/1723496263 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,fsmap=65,monmap=2+}) -- 0x55ec0d2b4270 con 0x55ec0d2e26b0 2024-08-03T20:52:56.870 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:56.865+0000 7f4e1fe00640 1 -- 172.21.15.32:0/1723496263 <== mon.0 v2:172.21.15.32:3300/0 4 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f4e24003a50 con 0x55ec0d2e26b0 2024-08-03T20:52:56.870 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:56.865+0000 7f4e1fe00640 1 -- 172.21.15.32:0/1723496263 <== mon.0 v2:172.21.15.32:3300/0 5 ==== fsmap(e 65) ==== 3502+0+0 (secure 0 0 0) 0x7f4e2404e070 con 0x55ec0d2e26b0 2024-08-03T20:52:56.870 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:56.865+0000 7f4e4896b740 10 client.6240 fetch_fsmap finished waiting for FSMap version 65 2024-08-03T20:52:56.870 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:56.865+0000 7f4e4896b740 10 client.6240 resolve_mds: resolved 4:0 to role '4:0' aka daemon mds.d 2024-08-03T20:52:56.870 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:56.865+0000 7f4e4896b740 20 client.6240 populate_metadata read hostname 'smithi032' 2024-08-03T20:52:56.870 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:56.865+0000 7f4e4896b740 1 --2- 172.21.15.32:0/1723496263 >> [v2:172.21.15.118:6834/3720010453,v1:172.21.15.118:6836/3720010453] conn(0x55ec0d2c1f60 0x55ec0d6f2270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:52:56.870 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:56.865+0000 7f4e4896b740 4 client.6240 mds_command: new command op to 5632 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-08-03T20:52:56.870 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:56.865+0000 7f4e4896b740 1 -- 172.21.15.32:0/1723496263 --> [v2:172.21.15.118:6834/3720010453,v1:172.21.15.118:6836/3720010453] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ec0d61e3c0 con 0x55ec0d2c1f60 2024-08-03T20:52:56.870 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:56.866+0000 7f4e3aa00640 1 --2- 172.21.15.32:0/1723496263 >> [v2:172.21.15.118:6834/3720010453,v1:172.21.15.118:6836/3720010453] conn(0x55ec0d2c1f60 0x55ec0d6f2270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:52:56.870 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:56.866+0000 7f4e3aa00640 1 --2- 172.21.15.32:0/1723496263 >> [v2:172.21.15.118:6834/3720010453,v1:172.21.15.118:6836/3720010453] conn(0x55ec0d2c1f60 0x55ec0d6f2270 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d829c3205aef771d server_cookie=4b10aa6fba09a653 in_seq=0 out_seq=0 2024-08-03T20:52:56.871 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:56.866+0000 7f4e1fe00640 10 client.6240 ms_handle_connect on v2:172.21.15.118:6834/3720010453 2024-08-03T20:52:56.871 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:56.867+0000 7f4e1fe00640 1 -- 172.21.15.32:0/1723496263 <== mds.0 v2:172.21.15.118:6834/3720010453 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55ec0d61e3c0 con 0x55ec0d2c1f60 2024-08-03T20:52:56.871 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:56.867+0000 7f4e1fe00640 10 client.6240 handle_command_reply: tid=0 2024-08-03T20:52:56.871 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:56.867+0000 7f4e4896b740 1 client.6240 shutdown 2024-08-03T20:52:56.871 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:56.868+0000 7f4e4896b740 1 -- 172.21.15.32:0/1723496263 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ec0d2e26b0 msgr2=0x55ec0d700f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:52:56.871 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:56.868+0000 7f4e4896b740 1 --2- 172.21.15.32:0/1723496263 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ec0d2e26b0 0x55ec0d700f20 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 crypto rx=0x7f4e24009830 tx=0x7f4e24003e30 comp rx=0 tx=0).stop 2024-08-03T20:52:56.871 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:56.868+0000 7f4e4896b740 1 -- 172.21.15.32:0/1723496263 shutdown_connections 2024-08-03T20:52:56.871 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:56.868+0000 7f4e4896b740 1 --2- 172.21.15.32:0/1723496263 >> [v2:172.21.15.118:6834/3720010453,v1:172.21.15.118:6836/3720010453] conn(0x55ec0d2c1f60 0x55ec0d6f2270 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:52:56.871 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:56.868+0000 7f4e4896b740 1 --2- 172.21.15.32:0/1723496263 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55ec0d303390 0x55ec0d6f1d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:52:56.871 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:56.868+0000 7f4e1fe00640 0 client.6240 ms_handle_reset on v2:172.21.15.118:6834/3720010453 2024-08-03T20:52:56.871 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:56.868+0000 7f4e4896b740 1 --2- 172.21.15.32:0/1723496263 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ec0d2e26b0 0x55ec0d700f20 unknown :-1 s=CLOSED pgs=681 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:52:56.871 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:56.868+0000 7f4e4896b740 1 --2- 172.21.15.32:0/1723496263 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55ec0d2da4d0 0x55ec0d6e7be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:52:56.871 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:56.868+0000 7f4e4896b740 1 -- 172.21.15.32:0/1723496263 >> 172.21.15.32:0/1723496263 conn(0x55ec0d2f74e0 msgr2=0x55ec0d6a9a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:52:56.871 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:56.868+0000 7f4e4896b740 1 -- 172.21.15.32:0/1723496263 shutdown_connections 2024-08-03T20:52:56.872 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:56.868+0000 7f4e4896b740 1 -- 172.21.15.32:0/1723496263 wait complete. 2024-08-03T20:52:56.872 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:56.868+0000 7f4e4896b740 20 client.6240 trim_cache size 0 max 16384 2024-08-03T20:52:56.886 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:56.884+0000 7f4e4896b740 1 Processor -- start 2024-08-03T20:52:56.886 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:56.884+0000 7f4e4896b740 1 -- start start 2024-08-03T20:52:56.887 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:56.886+0000 7f4e4896b740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ec0d2da4d0 0x55ec0d6f0a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:52:56.888 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:56.886+0000 7f4e4896b740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55ec0d2e26b0 0x55ec0d711610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:52:56.888 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:56.886+0000 7f4e4896b740 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55ec0d303390 0x55ec0d711b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:52:56.888 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:56.886+0000 7f4e4896b740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f4e3c201f30 con 0x55ec0d2da4d0 2024-08-03T20:52:56.888 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:56.886+0000 7f4e4896b740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f4e3c200f60 con 0x55ec0d2e26b0 2024-08-03T20:52:56.888 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:56.886+0000 7f4e4896b740 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7f4e3c1f5ce0 con 0x55ec0d303390 2024-08-03T20:52:56.888 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:56.886+0000 7f4e3aa00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ec0d2da4d0 0x55ec0d6f0a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:52:56.888 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:56.886+0000 7f4e3aa00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ec0d2da4d0 0x55ec0d6f0a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:51304/0 (socket says 172.21.15.32:51304) 2024-08-03T20:52:56.888 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:56.886+0000 7f4e3aa00640 1 -- 172.21.15.32:0/1277409351 learned_addr learned my addr 172.21.15.32:0/1277409351 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:52:56.888 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:56.886+0000 7f4e3b400640 1 --2- 172.21.15.32:0/1277409351 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55ec0d303390 0x55ec0d711b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:52:56.888 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:56.886+0000 7f4e33e00640 1 --2- 172.21.15.32:0/1277409351 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55ec0d2e26b0 0x55ec0d711610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:52:56.889 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:56.886+0000 7f4e3aa00640 1 -- 172.21.15.32:0/1277409351 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55ec0d303390 msgr2=0x55ec0d711b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:52:56.889 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:56.886+0000 7f4e3aa00640 1 --2- 172.21.15.32:0/1277409351 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55ec0d303390 0x55ec0d711b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:52:56.889 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:56.886+0000 7f4e3aa00640 1 -- 172.21.15.32:0/1277409351 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55ec0d2e26b0 msgr2=0x55ec0d711610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:52:56.889 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:56.886+0000 7f4e3aa00640 1 --2- 172.21.15.32:0/1277409351 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55ec0d2e26b0 0x55ec0d711610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:52:56.889 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:56.886+0000 7f4e3aa00640 1 -- 172.21.15.32:0/1277409351 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ec0d6a4f20 con 0x55ec0d2da4d0 2024-08-03T20:52:56.889 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:56.886+0000 7f4e33e00640 1 --2- 172.21.15.32:0/1277409351 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55ec0d2e26b0 0x55ec0d711610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-03T20:52:56.891 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:56.887+0000 7f4e3aa00640 1 --2- 172.21.15.32:0/1277409351 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ec0d2da4d0 0x55ec0d6f0a80 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7f4e2c061a90 tx=0x7f4e2c061b60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8d041a65dcb42fad server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:52:56.891 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:56.887+0000 7f4e2aa00640 1 -- 172.21.15.32:0/1277409351 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e2c0990b0 con 0x55ec0d2da4d0 2024-08-03T20:52:56.891 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:56.887+0000 7f4e2aa00640 1 -- 172.21.15.32:0/1277409351 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f4e2c05e020 con 0x55ec0d2da4d0 2024-08-03T20:52:56.891 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:56.887+0000 7f4e2aa00640 1 -- 172.21.15.32:0/1277409351 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e2c097d90 con 0x55ec0d2da4d0 2024-08-03T20:52:56.891 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:56.887+0000 7f4e4896b740 1 -- 172.21.15.32:0/1277409351 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ec0d2da4d0 msgr2=0x55ec0d6f0a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:52:56.891 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:56.887+0000 7f4e4896b740 1 --2- 172.21.15.32:0/1277409351 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ec0d2da4d0 0x55ec0d6f0a80 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7f4e2c061a90 tx=0x7f4e2c061b60 comp rx=0 tx=0).stop 2024-08-03T20:52:56.891 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:56.888+0000 7f4e4896b740 1 -- 172.21.15.32:0/1277409351 shutdown_connections 2024-08-03T20:52:56.891 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:56.888+0000 7f4e4896b740 1 --2- 172.21.15.32:0/1277409351 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55ec0d2e26b0 0x55ec0d711610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:52:56.892 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:56.888+0000 7f4e4896b740 1 --2- 172.21.15.32:0/1277409351 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55ec0d303390 0x55ec0d711b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:52:56.892 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:56.888+0000 7f4e4896b740 1 --2- 172.21.15.32:0/1277409351 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ec0d2da4d0 0x55ec0d6f0a80 unknown :-1 s=CLOSED pgs=682 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:52:56.892 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:56.888+0000 7f4e4896b740 1 -- 172.21.15.32:0/1277409351 >> 172.21.15.32:0/1277409351 conn(0x55ec0d2f74e0 msgr2=0x55ec0d6e77a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:52:56.892 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:56.888+0000 7f4e4896b740 1 -- 172.21.15.32:0/1277409351 shutdown_connections 2024-08-03T20:52:56.892 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:56.888+0000 7f4e4896b740 1 -- 172.21.15.32:0/1277409351 wait complete. 2024-08-03T20:52:56.892 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:56.889+0000 7f4e4896b740 1 -- start start 2024-08-03T20:52:56.892 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:56.889+0000 7f4e4896b740 1 Processor -- start 2024-08-03T20:52:56.895 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:56.890+0000 7f4e4896b740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ec0d2da4d0 0x55ec0d6eea60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:52:56.896 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:56.890+0000 7f4e4896b740 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55ec0d2e26b0 0x55ec0d769c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:52:56.896 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:56.890+0000 7f4e4896b740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55ec0d303390 0x55ec0d6e7740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:52:56.896 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:56.890+0000 7f4e4896b740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f4e3c1f6330 con 0x55ec0d2da4d0 2024-08-03T20:52:56.896 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:56.890+0000 7f4e4896b740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x55ec0d76a1b0 con 0x55ec0d303390 2024-08-03T20:52:56.896 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:56.890+0000 7f4e4896b740 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x55ec0d6ea820 con 0x55ec0d2e26b0 2024-08-03T20:52:56.896 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:56.890+0000 7f4e4896b740 10 client.0 _ll_register_callbacks cb 0x7f4e200036f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-03T20:52:56.896 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:56.890+0000 7f4e3aa00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ec0d2da4d0 0x55ec0d6eea60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:52:56.896 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:56.891+0000 7f4e3aa00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ec0d2da4d0 0x55ec0d6eea60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:51310/0 (socket says 172.21.15.32:51310) 2024-08-03T20:52:56.896 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:56.891+0000 7f4e33e00640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55ec0d2e26b0 0x55ec0d769c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:52:56.897 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:56.891+0000 7f4e3aa00640 1 -- 172.21.15.32:0/4183439965 learned_addr learned my addr 172.21.15.32:0/4183439965 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:52:56.897 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:56.891+0000 7f4e3b400640 1 --2- 172.21.15.32:0/4183439965 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55ec0d303390 0x55ec0d6e7740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:52:56.897 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:56.891+0000 7f4e3aa00640 1 -- 172.21.15.32:0/4183439965 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55ec0d2e26b0 msgr2=0x55ec0d769c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:52:56.897 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:56.891+0000 7f4e3aa00640 1 --2- 172.21.15.32:0/4183439965 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55ec0d2e26b0 0x55ec0d769c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:52:56.897 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:56.891+0000 7f4e3aa00640 1 -- 172.21.15.32:0/4183439965 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55ec0d303390 msgr2=0x55ec0d6e7740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:52:56.897 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:56.891+0000 7f4e3aa00640 1 --2- 172.21.15.32:0/4183439965 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55ec0d303390 0x55ec0d6e7740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:52:56.897 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:56.891+0000 7f4e3aa00640 1 -- 172.21.15.32:0/4183439965 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ec0d701080 con 0x55ec0d2da4d0 2024-08-03T20:52:56.897 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:56.891+0000 7f4e33e00640 1 --2- 172.21.15.32:0/4183439965 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55ec0d2e26b0 0x55ec0d769c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-03T20:52:56.897 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:56.891+0000 7f4e3aa00640 1 --2- 172.21.15.32:0/4183439965 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ec0d2da4d0 0x55ec0d6eea60 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7f4e2c00c6a0 tx=0x7f4e2c061d90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f10bec64476d8bf1 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:52:56.897 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:56.891+0000 7f4e1fe00640 10 client.0 ms_handle_connect on v2:172.21.15.32:3300/0 2024-08-03T20:52:56.897 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:56.892+0000 7f4e1fe00640 1 -- 172.21.15.32:0/4183439965 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e2c0978e0 con 0x55ec0d2da4d0 2024-08-03T20:52:56.897 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:56.892+0000 7f4e1fe00640 1 -- 172.21.15.32:0/4183439965 <== mon.0 v2:172.21.15.32:3300/0 2 ==== osd_map(59..59 src has 1..59) ==== 5881+0+0 (secure 0 0 0) 0x7f4e2c060ff0 con 0x55ec0d2da4d0 2024-08-03T20:52:56.898 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:56.892+0000 7f4e4896b740 1 -- 172.21.15.32:0/4183439965 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4e2c05e020 con 0x55ec0d2da4d0 2024-08-03T20:52:56.898 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:56.892+0000 7f4e1fe00640 1 -- 172.21.15.32:0/4183439965 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_get_version_reply(handle=1 version=65) ==== 24+0+0 (secure 0 0 0) 0x7f4e2c0611f0 con 0x55ec0d2da4d0 2024-08-03T20:52:56.898 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:56.892+0000 7f4e4896b740 10 client.6246 fetch_fsmap learned FSMap version 65 2024-08-03T20:52:56.898 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:56.892+0000 7f4e4896b740 1 -- 172.21.15.32:0/4183439965 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,fsmap=65,monmap=2+}) -- 0x55ec0d6d3e70 con 0x55ec0d2da4d0 2024-08-03T20:52:56.898 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:56.892+0000 7f4e1fe00640 1 -- 172.21.15.32:0/4183439965 <== mon.0 v2:172.21.15.32:3300/0 4 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f4e2c05e020 con 0x55ec0d2da4d0 2024-08-03T20:52:56.898 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:56.892+0000 7f4e1fe00640 1 -- 172.21.15.32:0/4183439965 <== mon.0 v2:172.21.15.32:3300/0 5 ==== fsmap(e 65) ==== 3502+0+0 (secure 0 0 0) 0x7f4e2c010b20 con 0x55ec0d2da4d0 2024-08-03T20:52:56.898 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:56.892+0000 7f4e4896b740 10 client.6246 fetch_fsmap finished waiting for FSMap version 65 2024-08-03T20:52:56.898 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:56.892+0000 7f4e4896b740 10 client.6246 resolve_mds: resolved 4:0 to role '4:0' aka daemon mds.d 2024-08-03T20:52:56.898 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:56.892+0000 7f4e4896b740 20 client.6246 populate_metadata read hostname 'smithi032' 2024-08-03T20:52:56.898 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:56.892+0000 7f4e4896b740 1 --2- 172.21.15.32:0/4183439965 >> [v2:172.21.15.118:6834/3720010453,v1:172.21.15.118:6836/3720010453] conn(0x55ec0d6e56a0 0x55ec0d6e5ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:52:56.899 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:56.892+0000 7f4e4896b740 4 client.6246 mds_command: new command op to 5632 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-08-03T20:52:56.899 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:56.892+0000 7f4e4896b740 1 -- 172.21.15.32:0/4183439965 --> [v2:172.21.15.118:6834/3720010453,v1:172.21.15.118:6836/3720010453] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55ec0d6e4b80 con 0x55ec0d6e56a0 2024-08-03T20:52:56.899 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:56.893+0000 7f4e33e00640 1 --2- 172.21.15.32:0/4183439965 >> [v2:172.21.15.118:6834/3720010453,v1:172.21.15.118:6836/3720010453] conn(0x55ec0d6e56a0 0x55ec0d6e5ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:52:56.899 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:56.893+0000 7f4e33e00640 1 --2- 172.21.15.32:0/4183439965 >> [v2:172.21.15.118:6834/3720010453,v1:172.21.15.118:6836/3720010453] conn(0x55ec0d6e56a0 0x55ec0d6e5ab0 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7bbbbc23558c53ed server_cookie=737f73391acf3831 in_seq=0 out_seq=0 2024-08-03T20:52:56.899 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:56.893+0000 7f4e1fe00640 10 client.6246 ms_handle_connect on v2:172.21.15.118:6834/3720010453 2024-08-03T20:52:56.907 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:56.905+0000 7f4e1fe00640 1 -- 172.21.15.32:0/4183439965 <== mds.0 v2:172.21.15.118:6834/3720010453 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55ec0d6e4b80 con 0x55ec0d6e56a0 2024-08-03T20:52:56.907 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:56.905+0000 7f4e1fe00640 10 client.6246 handle_command_reply: tid=0 2024-08-03T20:52:56.907 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:56.905+0000 7f4e4896b740 1 client.6246 shutdown 2024-08-03T20:52:56.908 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:56.905+0000 7f4e4896b740 1 -- 172.21.15.32:0/4183439965 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ec0d2da4d0 msgr2=0x55ec0d6eea60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:52:56.908 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:56.905+0000 7f4e4896b740 1 --2- 172.21.15.32:0/4183439965 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ec0d2da4d0 0x55ec0d6eea60 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7f4e2c00c6a0 tx=0x7f4e2c061d90 comp rx=0 tx=0).stop 2024-08-03T20:52:56.908 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:56.905+0000 7f4e4896b740 1 -- 172.21.15.32:0/4183439965 shutdown_connections 2024-08-03T20:52:56.908 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:56.905+0000 7f4e4896b740 1 --2- 172.21.15.32:0/4183439965 >> [v2:172.21.15.118:6834/3720010453,v1:172.21.15.118:6836/3720010453] conn(0x55ec0d6e56a0 0x55ec0d6e5ab0 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:52:56.908 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:56.905+0000 7f4e4896b740 1 --2- 172.21.15.32:0/4183439965 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55ec0d303390 0x55ec0d6e7740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:52:56.908 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:56.905+0000 7f4e4896b740 1 --2- 172.21.15.32:0/4183439965 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55ec0d2e26b0 0x55ec0d769c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:52:56.908 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:56.905+0000 7f4e1fe00640 0 client.6246 ms_handle_reset on v2:172.21.15.118:6834/3720010453 2024-08-03T20:52:56.908 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:56.905+0000 7f4e4896b740 1 --2- 172.21.15.32:0/4183439965 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ec0d2da4d0 0x55ec0d6eea60 unknown :-1 s=CLOSED pgs=683 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:52:56.908 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:56.905+0000 7f4e4896b740 1 -- 172.21.15.32:0/4183439965 >> 172.21.15.32:0/4183439965 conn(0x55ec0d2f74e0 msgr2=0x55ec0d6e8e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:52:56.908 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:56.906+0000 7f4e4896b740 1 -- 172.21.15.32:0/4183439965 shutdown_connections 2024-08-03T20:52:56.908 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:56.906+0000 7f4e4896b740 1 -- 172.21.15.32:0/4183439965 wait complete. 2024-08-03T20:52:56.909 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:56.906+0000 7f4e4896b740 20 client.6246 trim_cache size 0 max 16384 2024-08-03T20:52:56.909 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-08-03T20:52:56.909 INFO:teuthology.orchestra.run.smithi032.stdout: "message": "", 2024-08-03T20:52:56.909 INFO:teuthology.orchestra.run.smithi032.stdout: "return_code": 0 2024-08-03T20:52:56.909 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-08-03T20:52:56.910 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:56.906+0000 7f4e41a00640 1 -- 172.21.15.32:0/945754695 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f4e18077040 msgr2=0x7f4e18079550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:52:56.910 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:56.906+0000 7f4e41a00640 1 --2- 172.21.15.32:0/945754695 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f4e18077040 0x7f4e18079550 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f4e24034000 tx=0x7f4e24004730 comp rx=0 tx=0).stop 2024-08-03T20:52:56.910 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:56.906+0000 7f4e41a00640 1 -- 172.21.15.32:0/945754695 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4e3c06e950 msgr2=0x7f4e3c0c5560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:52:56.910 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:56.906+0000 7f4e41a00640 1 --2- 172.21.15.32:0/945754695 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4e3c06e950 0x7f4e3c0c5560 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7f4e2c00e790 tx=0x7f4e2c00ec50 comp rx=0 tx=0).stop 2024-08-03T20:52:56.910 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:56.906+0000 7f4e41a00640 1 -- 172.21.15.32:0/945754695 shutdown_connections 2024-08-03T20:52:56.910 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:56.906+0000 7f4e41a00640 1 --2- 172.21.15.32:0/945754695 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f4e18077040 0x7f4e18079550 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:52:56.911 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:56.906+0000 7f4e41a00640 1 --2- 172.21.15.32:0/945754695 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f4e3c15d2b0 0x7f4e3c1f8fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:52:56.911 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:56.906+0000 7f4e41a00640 1 --2- 172.21.15.32:0/945754695 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f4e3c0b3e80 0x7f4e3c1f8a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:52:56.911 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:56.906+0000 7f4e41a00640 1 --2- 172.21.15.32:0/945754695 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4e3c06e950 0x7f4e3c0c5560 unknown :-1 s=CLOSED pgs=679 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:52:56.911 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:56.906+0000 7f4e41a00640 1 -- 172.21.15.32:0/945754695 >> 172.21.15.32:0/945754695 conn(0x7f4e3c0b1bd0 msgr2=0x7f4e3c0b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:52:56.911 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:56.907+0000 7f4e41a00640 1 -- 172.21.15.32:0/945754695 shutdown_connections 2024-08-03T20:52:56.911 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:56.907+0000 7f4e41a00640 1 -- 172.21.15.32:0/945754695 wait complete. 2024-08-03T20:52:56.934 INFO:teuthology.nuke:Clearing teuthology firewall rules... 2024-08-03T20:52:56.934 DEBUG:teuthology.orchestra.run.smithi032:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-08-03T20:52:57.012 DEBUG:teuthology.orchestra.run.smithi118:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-08-03T20:52:57.105 INFO:teuthology.nuke:Cleared teuthology firewall rules. 2024-08-03T20:52:57.105 INFO:tasks.cephfs.mount:Terminating background process 2024-08-03T20:52:57.105 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-08-03T20:52:57.106 DEBUG:teuthology.orchestra.run.smithi032:> dd if=/proc/self/mounts of=/dev/stdout 2024-08-03T20:52:57.141 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-08-03T20:52:57.141 DEBUG:teuthology.orchestra.run.smithi032:> dd if=/proc/self/mounts of=/dev/stdout 2024-08-03T20:52:57.218 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-08-03T20:52:57.488 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:57.486+0000 7f2338a00640 1 -- 172.21.15.32:0/3906225740 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2318004d40 msgr2=0x7f2318005150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:52:57.488 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:57.486+0000 7f2338a00640 1 --2- 172.21.15.32:0/3906225740 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2318004d40 0x7f2318005150 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7f232c005fd0 tx=0x7f2318005830 comp rx=0 tx=0).stop 2024-08-03T20:52:57.488 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:57.486+0000 7f2332800640 1 -- 172.21.15.32:0/3906225740 reap_dead start 2024-08-03T20:52:57.488 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:57.486+0000 7f2338a00640 1 -- 172.21.15.32:0/3906225740 shutdown_connections 2024-08-03T20:52:57.488 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:57.486+0000 7f2338a00640 1 -- 172.21.15.32:0/3906225740 >> 172.21.15.32:0/3906225740 conn(0x7f23340b1bd0 msgr2=0x7f23340b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:52:57.488 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:57.486+0000 7f2338a00640 1 -- 172.21.15.32:0/3906225740 shutdown_connections 2024-08-03T20:52:57.489 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:57.486+0000 7f2338a00640 1 -- 172.21.15.32:0/3906225740 wait complete. 2024-08-03T20:52:57.490 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:57.488+0000 7f2338a00640 1 Processor -- start 2024-08-03T20:52:57.490 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:57.489+0000 7f2338a00640 1 -- start start 2024-08-03T20:52:57.492 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:57.490+0000 7f2338a00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f2318004d40 0x7f23341f59a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:52:57.492 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:57.490+0000 7f2338a00640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f233413fa10 0x7f2334140710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:52:57.492 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:57.490+0000 7f2338a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2334140c50 0x7f23340ad8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:52:57.492 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:57.490+0000 7f2338a00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f2334074ae0 con 0x7f2334140c50 2024-08-03T20:52:57.492 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:57.490+0000 7f2338a00640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f2334145f30 con 0x7f2318004d40 2024-08-03T20:52:57.493 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:57.490+0000 7f2338a00640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7f23340cf840 con 0x7f233413fa10 2024-08-03T20:52:57.493 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:57.490+0000 7f2332800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2334140c50 0x7f23340ad8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:52:57.493 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:57.490+0000 7f2332800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2334140c50 0x7f23340ad8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:51328/0 (socket says 172.21.15.32:51328) 2024-08-03T20:52:57.493 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:57.490+0000 7f2332800640 1 -- 172.21.15.32:0/3175560815 learned_addr learned my addr 172.21.15.32:0/3175560815 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:52:57.493 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:57.490+0000 7f2331e00640 1 --2- 172.21.15.32:0/3175560815 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f2318004d40 0x7f23341f59a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:52:57.493 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:57.490+0000 7f232be00640 1 --2- 172.21.15.32:0/3175560815 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f233413fa10 0x7f2334140710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:52:57.493 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:57.491+0000 7f2332800640 1 -- 172.21.15.32:0/3175560815 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f233413fa10 msgr2=0x7f2334140710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:52:57.493 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:57.491+0000 7f2332800640 1 --2- 172.21.15.32:0/3175560815 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f233413fa10 0x7f2334140710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:52:57.493 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:57.491+0000 7f2332800640 1 -- 172.21.15.32:0/3175560815 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f2318004d40 msgr2=0x7f23341f59a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:52:57.493 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:57.491+0000 7f2332800640 1 --2- 172.21.15.32:0/3175560815 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f2318004d40 0x7f23341f59a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:52:57.493 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:57.491+0000 7f2332800640 1 -- 172.21.15.32:0/3175560815 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2334141060 con 0x7f2334140c50 2024-08-03T20:52:57.494 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:57.491+0000 7f2332800640 1 --2- 172.21.15.32:0/3175560815 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2334140c50 0x7f23340ad8c0 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7f232c009e20 tx=0x7f232c033c30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:52:57.494 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:57.491+0000 7f232a000640 1 -- 172.21.15.32:0/3175560815 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f232c038040 con 0x7f2334140c50 2024-08-03T20:52:57.494 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:57.491+0000 7f232a000640 1 -- 172.21.15.32:0/3175560815 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f232c02bc30 con 0x7f2334140c50 2024-08-03T20:52:57.494 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:57.491+0000 7f232a000640 1 -- 172.21.15.32:0/3175560815 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f232c009200 con 0x7f2334140c50 2024-08-03T20:52:57.495 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:57.491+0000 7f2338a00640 1 -- 172.21.15.32:0/3175560815 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f23340af5d0 con 0x7f2334140c50 2024-08-03T20:52:57.495 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:57.492+0000 7f2338a00640 1 -- 172.21.15.32:0/3175560815 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f233415f3d0 con 0x7f2334140c50 2024-08-03T20:52:57.495 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:57.492+0000 7f2338a00640 1 -- 172.21.15.32:0/3175560815 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f23340ce910 con 0x7f2334140c50 2024-08-03T20:52:57.495 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:57.494+0000 7f232a000640 1 -- 172.21.15.32:0/3175560815 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f232c044030 con 0x7f2334140c50 2024-08-03T20:52:57.496 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:57.495+0000 7f232a000640 1 --2- 172.21.15.32:0/3175560815 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f2310077010 0x7f2310079520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:52:57.496 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:57.495+0000 7f232a000640 1 -- 172.21.15.32:0/3175560815 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(59..59 src has 1..59) ==== 5881+0+0 (secure 0 0 0) 0x7f232c0b8bc0 con 0x7f2334140c50 2024-08-03T20:52:57.497 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:57.495+0000 7f2331e00640 1 --2- 172.21.15.32:0/3175560815 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f2310077010 0x7f2310079520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:52:57.497 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:57.496+0000 7f2331e00640 1 --2- 172.21.15.32:0/3175560815 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f2310077010 0x7f2310079520 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f2324009c50 tx=0x7f2324005f90 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:52:57.501 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:57.499+0000 7f232a000640 1 -- 172.21.15.32:0/3175560815 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7f23340ce910 con 0x7f2334140c50 2024-08-03T20:52:57.722 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:57.721+0000 7f2338a00640 1 -- 172.21.15.32:0/3175560815 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f2334145100 con 0x7f2334140c50 2024-08-03T20:52:57.723 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:57.721+0000 7f232a000640 1 -- 172.21.15.32:0/3175560815 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 4 entries v59) ==== 80+0+244 (secure 0 0 0) 0x7f232c04b060 con 0x7f2334140c50 2024-08-03T20:52:57.723 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6835/2963295201 2024-08-04T20:51:13.985515+0000 2024-08-03T20:52:57.723 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6834/2963295201 2024-08-04T20:51:13.985515+0000 2024-08-03T20:52:57.723 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6835/3848976369 2024-08-04T20:50:40.678865+0000 2024-08-03T20:52:57.723 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6834/3848976369 2024-08-04T20:50:40.678865+0000 2024-08-03T20:52:57.724 INFO:teuthology.orchestra.run.smithi032.stderr:listed 4 entries 2024-08-03T20:52:57.726 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:57.724+0000 7f2338a00640 1 -- 172.21.15.32:0/3175560815 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f2310077010 msgr2=0x7f2310079520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:52:57.726 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:57.724+0000 7f2338a00640 1 --2- 172.21.15.32:0/3175560815 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f2310077010 0x7f2310079520 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f2324009c50 tx=0x7f2324005f90 comp rx=0 tx=0).stop 2024-08-03T20:52:57.726 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:57.724+0000 7f2338a00640 1 -- 172.21.15.32:0/3175560815 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2334140c50 msgr2=0x7f23340ad8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:52:57.726 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:57.724+0000 7f2338a00640 1 --2- 172.21.15.32:0/3175560815 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2334140c50 0x7f23340ad8c0 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7f232c009e20 tx=0x7f232c033c30 comp rx=0 tx=0).stop 2024-08-03T20:52:57.726 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:57.724+0000 7f2338a00640 1 -- 172.21.15.32:0/3175560815 shutdown_connections 2024-08-03T20:52:57.726 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:57.724+0000 7f2338a00640 1 --2- 172.21.15.32:0/3175560815 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f2310077010 0x7f2310079520 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:52:57.726 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:57.724+0000 7f2338a00640 1 --2- 172.21.15.32:0/3175560815 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2334140c50 0x7f23340ad8c0 unknown :-1 s=CLOSED pgs=686 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:52:57.726 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:57.724+0000 7f2338a00640 1 --2- 172.21.15.32:0/3175560815 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f233413fa10 0x7f2334140710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:52:57.727 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:57.724+0000 7f2338a00640 1 --2- 172.21.15.32:0/3175560815 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f2318004d40 0x7f23341f59a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:52:57.727 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:57.724+0000 7f2338a00640 1 -- 172.21.15.32:0/3175560815 >> 172.21.15.32:0/3175560815 conn(0x7f23340b1bd0 msgr2=0x7f2334143a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:52:57.727 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:57.724+0000 7f2338a00640 1 -- 172.21.15.32:0/3175560815 shutdown_connections 2024-08-03T20:52:57.727 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:57.724+0000 7f2338a00640 1 -- 172.21.15.32:0/3175560815 wait complete. 2024-08-03T20:52:57.748 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-08-03T20:52:57.748 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-08-03T20:52:57.749 DEBUG:teuthology.orchestra.run.smithi032:> sudo umount /home/ubuntu/cephtest/mnt.0 2024-08-03T20:52:57.811 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi032.front.sepia.ceph.com 2024-08-03T20:52:57.812 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-08-03T20:52:57.812 DEBUG:teuthology.orchestra.run.smithi032:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-08-03T20:52:57.848 INFO:tasks.cephfs.mount:Terminating background process 2024-08-03T20:52:57.848 DEBUG:teuthology.orchestra.run.smithi118:> set -ex 2024-08-03T20:52:57.848 DEBUG:teuthology.orchestra.run.smithi118:> dd if=/proc/self/mounts of=/dev/stdout 2024-08-03T20:52:57.888 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052663,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497944k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248972k,nr_inodes=812243,mode=700,uid=1000,gid=1316 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497944k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-08-03T20:52:57.888 INFO:tasks.cephfs.mount:Terminating background process 2024-08-03T20:52:57.888 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-08-03T20:52:57.888 DEBUG:teuthology.orchestra.run.smithi032:> dd if=/proc/self/mounts of=/dev/stdout 2024-08-03T20:52:57.924 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053453,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6499208k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 none /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249604k,nr_inodes=812401,mode=700,uid=1000,gid=1316 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6499208k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-08-03T20:52:57.924 INFO:tasks.cephfs.mount:Terminating background process 2024-08-03T20:52:57.924 DEBUG:teuthology.orchestra.run.smithi118:> set -ex 2024-08-03T20:52:57.924 DEBUG:teuthology.orchestra.run.smithi118:> dd if=/proc/self/mounts of=/dev/stdout 2024-08-03T20:52:57.965 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052663,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497944k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248972k,nr_inodes=812243,mode=700,uid=1000,gid=1316 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497944k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-08-03T20:52:57.965 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-08-03T20:52:58.194 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:58.192+0000 7f0a8f800640 1 -- 172.21.15.32:0/119653337 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0a8815fff0 msgr2=0x7f0a88160400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:52:58.194 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:58.192+0000 7f0a8f800640 1 --2- 172.21.15.32:0/119653337 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0a8815fff0 0x7f0a88160400 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7f0a880afaf0 tx=0x7f0a8002aa30 comp rx=0 tx=0).stop 2024-08-03T20:52:58.194 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:58.192+0000 7f0a8f800640 1 -- 172.21.15.32:0/119653337 shutdown_connections 2024-08-03T20:52:58.195 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:58.192+0000 7f0a8f800640 1 --2- 172.21.15.32:0/119653337 >> v2:172.21.15.118:3300/0 conn(0x7f0a88144f90 0x7f0a88074570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:52:58.195 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:58.192+0000 7f0a8f800640 1 --2- 172.21.15.32:0/119653337 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0a8815fff0 0x7f0a88160400 unknown :-1 s=CLOSED pgs=687 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:52:58.195 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:58.192+0000 7f0a8f800640 1 -- 172.21.15.32:0/119653337 >> 172.21.15.32:0/119653337 conn(0x7f0a880b1bd0 msgr2=0x7f0a880b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:52:58.195 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:58.192+0000 7f0a8f800640 1 -- 172.21.15.32:0/119653337 shutdown_connections 2024-08-03T20:52:58.195 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:58.192+0000 7f0a8f800640 1 -- 172.21.15.32:0/119653337 wait complete. 2024-08-03T20:52:58.196 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:58.195+0000 7f0a8f800640 1 Processor -- start 2024-08-03T20:52:58.196 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:58.195+0000 7f0a8f800640 1 -- start start 2024-08-03T20:52:58.199 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:58.196+0000 7f0a8f800640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f0a8806e930 0x7f0a8815ca40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:52:58.199 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:58.196+0000 7f0a8f800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0a88144f90 0x7f0a8815bd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:52:58.199 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:58.196+0000 7f0a8f800640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f0a8815fff0 0x7f0a881e67f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:52:58.199 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:58.196+0000 7f0a8f800640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f0a880cf840 con 0x7f0a88144f90 2024-08-03T20:52:58.199 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:58.196+0000 7f0a8f800640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f0a88145f30 con 0x7f0a8815fff0 2024-08-03T20:52:58.200 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:58.196+0000 7f0a8f800640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7f0a880cf9c0 con 0x7f0a8806e930 2024-08-03T20:52:58.200 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:58.196+0000 7f0a87e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0a88144f90 0x7f0a8815bd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:52:58.200 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:58.197+0000 7f0a87e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0a88144f90 0x7f0a8815bd70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:51346/0 (socket says 172.21.15.32:51346) 2024-08-03T20:52:58.200 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:58.197+0000 7f0a87e00640 1 -- 172.21.15.32:0/1878324941 learned_addr learned my addr 172.21.15.32:0/1878324941 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:52:58.200 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:58.197+0000 7f0a8ce00640 1 --2- 172.21.15.32:0/1878324941 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f0a8806e930 0x7f0a8815ca40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:52:58.200 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:58.197+0000 7f0a8d800640 1 --2- 172.21.15.32:0/1878324941 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f0a8815fff0 0x7f0a881e67f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:52:58.200 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:58.197+0000 7f0a87e00640 1 -- 172.21.15.32:0/1878324941 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f0a8806e930 msgr2=0x7f0a8815ca40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:52:58.200 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:58.197+0000 7f0a87e00640 1 --2- 172.21.15.32:0/1878324941 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f0a8806e930 0x7f0a8815ca40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:52:58.200 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:58.197+0000 7f0a87e00640 1 -- 172.21.15.32:0/1878324941 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f0a8815fff0 msgr2=0x7f0a881e67f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:52:58.200 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:58.197+0000 7f0a87e00640 1 --2- 172.21.15.32:0/1878324941 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f0a8815fff0 0x7f0a881e67f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:52:58.200 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:58.197+0000 7f0a87e00640 1 -- 172.21.15.32:0/1878324941 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0a881f9960 con 0x7f0a88144f90 2024-08-03T20:52:58.200 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:58.197+0000 7f0a87e00640 1 --2- 172.21.15.32:0/1878324941 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0a88144f90 0x7f0a8815bd70 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7f0a7400d750 tx=0x7f0a7400dc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:52:58.200 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:58.198+0000 7f0a84c00640 1 -- 172.21.15.32:0/1878324941 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a74017070 con 0x7f0a88144f90 2024-08-03T20:52:58.200 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:58.198+0000 7f0a84c00640 1 -- 172.21.15.32:0/1878324941 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f0a74009b10 con 0x7f0a88144f90 2024-08-03T20:52:58.201 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:58.198+0000 7f0a8f800640 1 -- 172.21.15.32:0/1878324941 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0a881e5d10 con 0x7f0a88144f90 2024-08-03T20:52:58.201 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:58.198+0000 7f0a84c00640 1 -- 172.21.15.32:0/1878324941 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a74010460 con 0x7f0a88144f90 2024-08-03T20:52:58.201 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:58.198+0000 7f0a8f800640 1 -- 172.21.15.32:0/1878324941 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0a881e62d0 con 0x7f0a88144f90 2024-08-03T20:52:58.201 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:58.199+0000 7f0a8f800640 1 -- 172.21.15.32:0/1878324941 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0a88160c50 con 0x7f0a88144f90 2024-08-03T20:52:58.201 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:58.200+0000 7f0a84c00640 1 -- 172.21.15.32:0/1878324941 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f0a7401a070 con 0x7f0a88144f90 2024-08-03T20:52:58.203 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:58.201+0000 7f0a84c00640 1 --2- 172.21.15.32:0/1878324941 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f0a68077040 0x7f0a68079550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:52:58.203 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:58.201+0000 7f0a84c00640 1 -- 172.21.15.32:0/1878324941 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(59..59 src has 1..59) ==== 5881+0+0 (secure 0 0 0) 0x7f0a74098d50 con 0x7f0a88144f90 2024-08-03T20:52:58.203 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:58.202+0000 7f0a8ce00640 1 --2- 172.21.15.32:0/1878324941 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f0a68077040 0x7f0a68079550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:52:58.204 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:58.203+0000 7f0a8ce00640 1 --2- 172.21.15.32:0/1878324941 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f0a68077040 0x7f0a68079550 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f0a7c006fd0 tx=0x7f0a7c009060 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:52:58.207 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:58.206+0000 7f0a84c00640 1 -- 172.21.15.32:0/1878324941 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7f0a74061840 con 0x7f0a88144f90 2024-08-03T20:52:58.450 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:58.449+0000 7f0a8f800640 1 -- 172.21.15.32:0/1878324941 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f0a88160530 con 0x7f0a88144f90 2024-08-03T20:52:58.451 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:58.449+0000 7f0a84c00640 1 -- 172.21.15.32:0/1878324941 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 65 v65) ==== 94+0+5231 (secure 0 0 0) 0x7f0a74065d90 con 0x7f0a88144f90 2024-08-03T20:52:58.451 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-03T20:52:58.452 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":65,"btime":"2024-08-03T20:51:16:994952+0000","default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5261,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6833/2498840646","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":2498840646},{"type":"v1","addr":"172.21.15.32:6833","nonce":2498840646}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":45},{"gid":5276,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6837/3142148286","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6835","nonce":3142148286},{"type":"v1","addr":"172.21.15.118:6837","nonce":3142148286}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":46},{"gid":6180,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6835/1934242557","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1934242557},{"type":"v1","addr":"172.21.15.32:6835","nonce":1934242557}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":63}],"filesystems":[{"mdsmap":{"epoch":65,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-08-03T20:50:27.688698+0000","modified":"2024-08-03T20:51:16.994940+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":59,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5632},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5632":{"gid":5632,"name":"d","rank":0,"incarnation":62,"state":"up:active","state_seq":16,"addr":"172.21.15.118:6836/3720010453","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6834","nonce":3720010453},{"type":"v1","addr":"172.21.15.118:6836","nonce":3720010453}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[9],"metadata_pool":8,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5632,"qdb_cluster":[5632]},"id":4}]} 2024-08-03T20:52:58.452 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 65 2024-08-03T20:52:58.453 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:58.452+0000 7f0a8f800640 1 -- 172.21.15.32:0/1878324941 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f0a68077040 msgr2=0x7f0a68079550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:52:58.453 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:58.452+0000 7f0a8f800640 1 --2- 172.21.15.32:0/1878324941 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f0a68077040 0x7f0a68079550 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f0a7c006fd0 tx=0x7f0a7c009060 comp rx=0 tx=0).stop 2024-08-03T20:52:58.454 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:58.452+0000 7f0a8f800640 1 -- 172.21.15.32:0/1878324941 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0a88144f90 msgr2=0x7f0a8815bd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:52:58.454 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:58.452+0000 7f0a8f800640 1 --2- 172.21.15.32:0/1878324941 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0a88144f90 0x7f0a8815bd70 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7f0a7400d750 tx=0x7f0a7400dc10 comp rx=0 tx=0).stop 2024-08-03T20:52:58.454 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:58.452+0000 7f0a8f800640 1 -- 172.21.15.32:0/1878324941 shutdown_connections 2024-08-03T20:52:58.454 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:58.452+0000 7f0a8f800640 1 --2- 172.21.15.32:0/1878324941 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f0a68077040 0x7f0a68079550 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:52:58.454 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:58.452+0000 7f0a8f800640 1 --2- 172.21.15.32:0/1878324941 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f0a8815fff0 0x7f0a881e67f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:52:58.454 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:58.452+0000 7f0a8f800640 1 --2- 172.21.15.32:0/1878324941 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0a88144f90 0x7f0a8815bd70 unknown :-1 s=CLOSED pgs=688 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:52:58.454 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:58.452+0000 7f0a8f800640 1 --2- 172.21.15.32:0/1878324941 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f0a8806e930 0x7f0a8815ca40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:52:58.454 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:58.452+0000 7f0a8f800640 1 -- 172.21.15.32:0/1878324941 >> 172.21.15.32:0/1878324941 conn(0x7f0a880b1bd0 msgr2=0x7f0a8806f930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:52:58.454 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:58.452+0000 7f0a8f800640 1 -- 172.21.15.32:0/1878324941 shutdown_connections 2024-08-03T20:52:58.454 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:58.452+0000 7f0a8f800640 1 -- 172.21.15.32:0/1878324941 wait complete. 2024-08-03T20:52:58.480 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-08-03T20:52:58.710 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:58.708+0000 7f6acd600640 1 -- 172.21.15.32:0/4047669455 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6aac003a00 msgr2=0x7f6aac003e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:52:58.710 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:58.708+0000 7f6acd600640 1 --2- 172.21.15.32:0/4047669455 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6aac003a00 0x7f6aac003e70 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7f6ac8072280 tx=0x7f6aac005830 comp rx=0 tx=0).stop 2024-08-03T20:52:58.711 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:58.708+0000 7f6ac7400640 1 -- 172.21.15.32:0/4047669455 reap_dead start 2024-08-03T20:52:58.711 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:58.709+0000 7f6acd600640 1 -- 172.21.15.32:0/4047669455 shutdown_connections 2024-08-03T20:52:58.711 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:58.709+0000 7f6acd600640 1 -- 172.21.15.32:0/4047669455 >> 172.21.15.32:0/4047669455 conn(0x7f6ac80b1bd0 msgr2=0x7f6ac80b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:52:58.711 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:58.709+0000 7f6acd600640 1 -- 172.21.15.32:0/4047669455 shutdown_connections 2024-08-03T20:52:58.711 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:58.709+0000 7f6acd600640 1 -- 172.21.15.32:0/4047669455 wait complete. 2024-08-03T20:52:58.712 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:58.711+0000 7f6acd600640 1 Processor -- start 2024-08-03T20:52:58.712 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:58.711+0000 7f6acd600640 1 -- start start 2024-08-03T20:52:58.715 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:58.713+0000 7f6acd600640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f6aac003a00 0x7f6ac81f5a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:52:58.716 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:58.713+0000 7f6acd600640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f6ac80af0d0 0x7f6ac81f8770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:52:58.716 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:58.713+0000 7f6acd600640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6ac81f8cb0 0x7f6ac8142530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:52:58.716 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:58.713+0000 7f6acd600640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f6ac80cf9c0 con 0x7f6ac81f8cb0 2024-08-03T20:52:58.716 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:58.713+0000 7f6acd600640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f6ac80cf840 con 0x7f6aac003a00 2024-08-03T20:52:58.716 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:58.713+0000 7f6acd600640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7f6ac8144f90 con 0x7f6ac80af0d0 2024-08-03T20:52:58.716 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:58.713+0000 7f6ac7400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6ac81f8cb0 0x7f6ac8142530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:52:58.716 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:58.713+0000 7f6ac7400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6ac81f8cb0 0x7f6ac8142530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:51362/0 (socket says 172.21.15.32:51362) 2024-08-03T20:52:58.716 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:58.713+0000 7f6ac7400640 1 -- 172.21.15.32:0/3472493269 learned_addr learned my addr 172.21.15.32:0/3472493269 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:52:58.716 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:58.713+0000 7f6ac6a00640 1 --2- 172.21.15.32:0/3472493269 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f6aac003a00 0x7f6ac81f5a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:52:58.716 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:58.713+0000 7f6ac6000640 1 --2- 172.21.15.32:0/3472493269 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f6ac80af0d0 0x7f6ac81f8770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:52:58.716 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:58.713+0000 7f6ac7400640 1 -- 172.21.15.32:0/3472493269 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f6ac80af0d0 msgr2=0x7f6ac81f8770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:52:58.716 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:58.713+0000 7f6ac7400640 1 --2- 172.21.15.32:0/3472493269 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f6ac80af0d0 0x7f6ac81f8770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:52:58.717 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:58.713+0000 7f6ac7400640 1 -- 172.21.15.32:0/3472493269 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f6aac003a00 msgr2=0x7f6ac81f5a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:52:58.717 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:58.713+0000 7f6ac7400640 1 --2- 172.21.15.32:0/3472493269 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f6aac003a00 0x7f6ac81f5a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:52:58.717 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:58.713+0000 7f6ac7400640 1 -- 172.21.15.32:0/3472493269 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6ac81f5f60 con 0x7f6ac81f8cb0 2024-08-03T20:52:58.717 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:58.713+0000 7f6ac6a00640 1 --2- 172.21.15.32:0/3472493269 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f6aac003a00 0x7f6ac81f5a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-03T20:52:58.717 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:58.714+0000 7f6ac7400640 1 --2- 172.21.15.32:0/3472493269 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6ac81f8cb0 0x7f6ac8142530 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7f6ac8072860 tx=0x7f6ac0006660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:52:58.717 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:58.714+0000 7f6abea00640 1 -- 172.21.15.32:0/3472493269 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ac000fb90 con 0x7f6ac81f8cb0 2024-08-03T20:52:58.717 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:58.714+0000 7f6abea00640 1 -- 172.21.15.32:0/3472493269 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f6ac0018070 con 0x7f6ac81f8cb0 2024-08-03T20:52:58.717 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:58.714+0000 7f6abea00640 1 -- 172.21.15.32:0/3472493269 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ac0006970 con 0x7f6ac81f8cb0 2024-08-03T20:52:58.717 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:58.714+0000 7f6acd600640 1 -- 172.21.15.32:0/3472493269 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6ac80ae030 con 0x7f6ac81f8cb0 2024-08-03T20:52:58.717 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:58.714+0000 7f6acd600640 1 -- 172.21.15.32:0/3472493269 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6ac80ae5f0 con 0x7f6ac81f8cb0 2024-08-03T20:52:58.718 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:58.715+0000 7f6acd600640 1 -- 172.21.15.32:0/3472493269 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6ac8074270 con 0x7f6ac81f8cb0 2024-08-03T20:52:58.718 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:58.716+0000 7f6abea00640 1 -- 172.21.15.32:0/3472493269 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f6ac001d460 con 0x7f6ac81f8cb0 2024-08-03T20:52:58.719 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:58.717+0000 7f6abea00640 1 --2- 172.21.15.32:0/3472493269 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f6aa40771b0 0x7f6aa40796c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:52:58.719 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:58.717+0000 7f6abea00640 1 -- 172.21.15.32:0/3472493269 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(59..59 src has 1..59) ==== 5881+0+0 (secure 0 0 0) 0x7f6ac0099210 con 0x7f6ac81f8cb0 2024-08-03T20:52:58.719 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:58.718+0000 7f6ac6a00640 1 --2- 172.21.15.32:0/3472493269 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f6aa40771b0 0x7f6aa40796c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:52:58.720 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:58.719+0000 7f6ac6a00640 1 --2- 172.21.15.32:0/3472493269 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f6aa40771b0 0x7f6aa40796c0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f6ab8035000 tx=0x7f6ab802d040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:52:58.723 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:58.722+0000 7f6abea00640 1 -- 172.21.15.32:0/3472493269 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7f6ac81f5f60 con 0x7f6ac81f8cb0 2024-08-03T20:52:58.965 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:58.964+0000 7f6acd600640 1 -- 172.21.15.32:0/3472493269 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f6ac80701d0 con 0x7f6ac81f8cb0 2024-08-03T20:52:58.966 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:58.964+0000 7f6abea00640 1 -- 172.21.15.32:0/3472493269 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 65 v65) ==== 94+0+5231 (secure 0 0 0) 0x7f6ac00622c0 con 0x7f6ac81f8cb0 2024-08-03T20:52:58.966 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-03T20:52:58.966 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":65,"btime":"2024-08-03T20:51:16:994952+0000","default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5261,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6833/2498840646","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":2498840646},{"type":"v1","addr":"172.21.15.32:6833","nonce":2498840646}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":45},{"gid":5276,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6837/3142148286","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6835","nonce":3142148286},{"type":"v1","addr":"172.21.15.118:6837","nonce":3142148286}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":46},{"gid":6180,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6835/1934242557","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1934242557},{"type":"v1","addr":"172.21.15.32:6835","nonce":1934242557}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":63}],"filesystems":[{"mdsmap":{"epoch":65,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-08-03T20:50:27.688698+0000","modified":"2024-08-03T20:51:16.994940+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":59,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5632},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5632":{"gid":5632,"name":"d","rank":0,"incarnation":62,"state":"up:active","state_seq":16,"addr":"172.21.15.118:6836/3720010453","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6834","nonce":3720010453},{"type":"v1","addr":"172.21.15.118:6836","nonce":3720010453}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[9],"metadata_pool":8,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5632,"qdb_cluster":[5632]},"id":4}]} 2024-08-03T20:52:58.966 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 65 2024-08-03T20:52:58.969 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:58.967+0000 7f6acd600640 1 -- 172.21.15.32:0/3472493269 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f6aa40771b0 msgr2=0x7f6aa40796c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:52:58.969 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:58.967+0000 7f6acd600640 1 --2- 172.21.15.32:0/3472493269 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f6aa40771b0 0x7f6aa40796c0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f6ab8035000 tx=0x7f6ab802d040 comp rx=0 tx=0).stop 2024-08-03T20:52:58.969 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:58.967+0000 7f6acd600640 1 -- 172.21.15.32:0/3472493269 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6ac81f8cb0 msgr2=0x7f6ac8142530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:52:58.969 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:58.967+0000 7f6acd600640 1 --2- 172.21.15.32:0/3472493269 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6ac81f8cb0 0x7f6ac8142530 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7f6ac8072860 tx=0x7f6ac0006660 comp rx=0 tx=0).stop 2024-08-03T20:52:58.969 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:58.967+0000 7f6acd600640 1 -- 172.21.15.32:0/3472493269 shutdown_connections 2024-08-03T20:52:58.969 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:58.967+0000 7f6acd600640 1 --2- 172.21.15.32:0/3472493269 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f6aa40771b0 0x7f6aa40796c0 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:52:58.969 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:58.967+0000 7f6acd600640 1 --2- 172.21.15.32:0/3472493269 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6ac81f8cb0 0x7f6ac8142530 unknown :-1 s=CLOSED pgs=691 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:52:58.969 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:58.967+0000 7f6acd600640 1 --2- 172.21.15.32:0/3472493269 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f6ac80af0d0 0x7f6ac81f8770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:52:58.969 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:58.967+0000 7f6acd600640 1 --2- 172.21.15.32:0/3472493269 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f6aac003a00 0x7f6ac81f5a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:52:58.969 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:58.967+0000 7f6acd600640 1 -- 172.21.15.32:0/3472493269 >> 172.21.15.32:0/3472493269 conn(0x7f6ac80b1bd0 msgr2=0x7f6ac80b3740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:52:58.969 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:58.967+0000 7f6acd600640 1 -- 172.21.15.32:0/3472493269 shutdown_connections 2024-08-03T20:52:58.969 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:58.967+0000 7f6acd600640 1 -- 172.21.15.32:0/3472493269 wait complete. 2024-08-03T20:52:58.991 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-08-03T20:52:59.220 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:59.218+0000 7f097fa00640 1 -- 172.21.15.32:0/3612856689 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f09780b39a0 msgr2=0x7f09780b3db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:52:59.220 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:59.218+0000 7f097fa00640 1 --2- 172.21.15.32:0/3612856689 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f09780b39a0 0x7f09780b3db0 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7f0964005fd0 tx=0x7f096402a5d0 comp rx=0 tx=0).stop 2024-08-03T20:52:59.220 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:59.218+0000 7f097fa00640 1 -- 172.21.15.32:0/3612856689 shutdown_connections 2024-08-03T20:52:59.220 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:59.218+0000 7f097fa00640 1 --2- 172.21.15.32:0/3612856689 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f09780b39a0 0x7f09780b3db0 unknown :-1 s=CLOSED pgs=692 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:52:59.220 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:59.218+0000 7f097fa00640 1 --2- 172.21.15.32:0/3612856689 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f097806f720 0x7f097806fb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:52:59.220 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:59.218+0000 7f097fa00640 1 -- 172.21.15.32:0/3612856689 >> 172.21.15.32:0/3612856689 conn(0x7f09780b1bd0 msgr2=0x7f09780b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:52:59.220 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:59.218+0000 7f097fa00640 1 -- 172.21.15.32:0/3612856689 shutdown_connections 2024-08-03T20:52:59.221 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:59.218+0000 7f097fa00640 1 -- 172.21.15.32:0/3612856689 wait complete. 2024-08-03T20:52:59.222 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:59.220+0000 7f097fa00640 1 Processor -- start 2024-08-03T20:52:59.222 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:59.221+0000 7f097fa00640 1 -- start start 2024-08-03T20:52:59.224 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:59.222+0000 7f097fa00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f097806f720 0x7f09781f5550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:52:59.224 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:59.222+0000 7f097fa00640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f09780b39a0 0x7f09781422a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:52:59.224 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:59.222+0000 7f097fa00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0978162210 0x7f0978155bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:52:59.224 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:59.222+0000 7f097fa00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f0978162d30 con 0x7f0978162210 2024-08-03T20:52:59.224 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:59.222+0000 7f097fa00640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f09780af0a0 con 0x7f097806f720 2024-08-03T20:52:59.224 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:59.222+0000 7f097fa00640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7f0978162eb0 con 0x7f09780b39a0 2024-08-03T20:52:59.224 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:59.222+0000 7f097da00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0978162210 0x7f0978155bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:52:59.224 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:59.222+0000 7f097d000640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f097806f720 0x7f09781f5550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:52:59.225 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:59.222+0000 7f097d000640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f097806f720 0x7f09781f5550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.32:34480/0 (socket says 172.21.15.32:34480) 2024-08-03T20:52:59.225 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:59.222+0000 7f097d000640 1 -- 172.21.15.32:0/1055452531 learned_addr learned my addr 172.21.15.32:0/1055452531 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:52:59.225 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:59.222+0000 7f0977e00640 1 --2- 172.21.15.32:0/1055452531 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f09780b39a0 0x7f09781422a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:52:59.226 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:59.223+0000 7f097da00640 1 -- 172.21.15.32:0/1055452531 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f09780b39a0 msgr2=0x7f09781422a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:52:59.226 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:59.223+0000 7f097da00640 1 --2- 172.21.15.32:0/1055452531 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f09780b39a0 0x7f09781422a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:52:59.226 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:59.223+0000 7f097da00640 1 -- 172.21.15.32:0/1055452531 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f097806f720 msgr2=0x7f09781f5550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:52:59.226 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:59.223+0000 7f097da00640 1 --2- 172.21.15.32:0/1055452531 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f097806f720 0x7f09781f5550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:52:59.226 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:59.223+0000 7f097da00640 1 -- 172.21.15.32:0/1055452531 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f09781427e0 con 0x7f0978162210 2024-08-03T20:52:59.226 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:59.223+0000 7f097da00640 1 --2- 172.21.15.32:0/1055452531 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0978162210 0x7f0978155bd0 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7f097000c700 tx=0x7f097000cbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:52:59.226 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:59.223+0000 7f0974c00640 1 -- 172.21.15.32:0/1055452531 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0970003b70 con 0x7f0978162210 2024-08-03T20:52:59.226 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:59.223+0000 7f0974c00640 1 -- 172.21.15.32:0/1055452531 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f0970003d10 con 0x7f0978162210 2024-08-03T20:52:59.227 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:59.223+0000 7f0974c00640 1 -- 172.21.15.32:0/1055452531 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0970005330 con 0x7f0978162210 2024-08-03T20:52:59.227 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:59.224+0000 7f097fa00640 1 -- 172.21.15.32:0/1055452531 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0978157b60 con 0x7f0978162210 2024-08-03T20:52:59.227 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:59.224+0000 7f097fa00640 1 -- 172.21.15.32:0/1055452531 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f09781f60f0 con 0x7f0978162210 2024-08-03T20:52:59.227 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:59.225+0000 7f097fa00640 1 -- 172.21.15.32:0/1055452531 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f097806fb30 con 0x7f0978162210 2024-08-03T20:52:59.227 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:59.226+0000 7f0974c00640 1 -- 172.21.15.32:0/1055452531 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f09700054d0 con 0x7f0978162210 2024-08-03T20:52:59.228 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:59.227+0000 7f0974c00640 1 --2- 172.21.15.32:0/1055452531 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f0958077010 0x7f0958079520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:52:59.228 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:59.227+0000 7f0974c00640 1 -- 172.21.15.32:0/1055452531 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(59..59 src has 1..59) ==== 5881+0+0 (secure 0 0 0) 0x7f09700986e0 con 0x7f0978162210 2024-08-03T20:52:59.229 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:59.228+0000 7f097d000640 1 --2- 172.21.15.32:0/1055452531 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f0958077010 0x7f0958079520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:52:59.230 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:59.228+0000 7f097d000640 1 --2- 172.21.15.32:0/1055452531 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f0958077010 0x7f0958079520 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f096c009bf0 tx=0x7f096c005f90 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:52:59.233 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:59.232+0000 7f0974c00640 1 -- 172.21.15.32:0/1055452531 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7f0970061140 con 0x7f0978162210 2024-08-03T20:52:59.438 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:59.437+0000 7f097fa00640 1 -- 172.21.15.32:0/1055452531 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f09781f5ba0 con 0x7f0978162210 2024-08-03T20:52:59.440 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:59.438+0000 7f0974c00640 1 -- 172.21.15.32:0/1055452531 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v59) ==== 74+0+16864 (secure 0 0 0) 0x7f0970065690 con 0x7f0978162210 2024-08-03T20:52:59.440 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-03T20:52:59.440 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":59,"fsid":"40ab141a-01d7-49b9-ae3c-2edec9876e95","created":"2024-08-03T20:47:14.957362+0000","modified":"2024-08-03T20:51:13.985593+0000","last_up_change":"2024-08-03T20:47:28.874451+0000","last_in_change":"2024-08-03T20:47:20.079442+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-08-03T20:47:28.261300+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_metadata","create_time":"2024-08-03T20:50:25.652248+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"56","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_data","create_time":"2024-08-03T20:50:26.646008+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"57","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5a83e759-9035-438b-889a-0a6af1f0a0d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6816","nonce":3066156116},{"type":"v1","addr":"172.21.15.32:6817","nonce":3066156116}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6818","nonce":3066156116},{"type":"v1","addr":"172.21.15.32:6819","nonce":3066156116}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6822","nonce":3066156116},{"type":"v1","addr":"172.21.15.32:6823","nonce":3066156116}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6820","nonce":3066156116},{"type":"v1","addr":"172.21.15.32:6821","nonce":3066156116}]},"public_addr":"172.21.15.32:6817/3066156116","cluster_addr":"172.21.15.32:6819/3066156116","heartbeat_back_addr":"172.21.15.32:6823/3066156116","heartbeat_front_addr":"172.21.15.32:6821/3066156116","state":["exists","up"]},{"osd":1,"uuid":"d7c5c210-da4f-49f9-b9e7-1dfb02daa4bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6824","nonce":1551241759},{"type":"v1","addr":"172.21.15.32:6825","nonce":1551241759}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6826","nonce":1551241759},{"type":"v1","addr":"172.21.15.32:6827","nonce":1551241759}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6830","nonce":1551241759},{"type":"v1","addr":"172.21.15.32:6831","nonce":1551241759}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6828","nonce":1551241759},{"type":"v1","addr":"172.21.15.32:6829","nonce":1551241759}]},"public_addr":"172.21.15.32:6825/1551241759","cluster_addr":"172.21.15.32:6827/1551241759","heartbeat_back_addr":"172.21.15.32:6831/1551241759","heartbeat_front_addr":"172.21.15.32:6829/1551241759","state":["exists","up"]},{"osd":2,"uuid":"57f2e6de-a201-45c2-8981-8e56a4875ebd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6808","nonce":3806708944},{"type":"v1","addr":"172.21.15.32:6809","nonce":3806708944}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6810","nonce":3806708944},{"type":"v1","addr":"172.21.15.32:6811","nonce":3806708944}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6814","nonce":3806708944},{"type":"v1","addr":"172.21.15.32:6815","nonce":3806708944}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6812","nonce":3806708944},{"type":"v1","addr":"172.21.15.32:6813","nonce":3806708944}]},"public_addr":"172.21.15.32:6809/3806708944","cluster_addr":"172.21.15.32:6811/3806708944","heartbeat_back_addr":"172.21.15.32:6815/3806708944","heartbeat_front_addr":"172.21.15.32:6813/3806708944","state":["exists","up"]},{"osd":3,"uuid":"6439a280-da45-4148-8130-f16323935ff3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6800","nonce":9088303},{"type":"v1","addr":"172.21.15.32:6801","nonce":9088303}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6802","nonce":9088303},{"type":"v1","addr":"172.21.15.32:6803","nonce":9088303}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6806","nonce":9088303},{"type":"v1","addr":"172.21.15.32:6807","nonce":9088303}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6804","nonce":9088303},{"type":"v1","addr":"172.21.15.32:6805","nonce":9088303}]},"public_addr":"172.21.15.32:6801/9088303","cluster_addr":"172.21.15.32:6803/9088303","heartbeat_back_addr":"172.21.15.32:6807/9088303","heartbeat_front_addr":"172.21.15.32:6805/9088303","state":["exists","up"]},{"osd":4,"uuid":"4f902eff-1e18-418d-82c9-2151c69524fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6802","nonce":864752275},{"type":"v1","addr":"172.21.15.118:6803","nonce":864752275}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6804","nonce":864752275},{"type":"v1","addr":"172.21.15.118:6805","nonce":864752275}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6808","nonce":864752275},{"type":"v1","addr":"172.21.15.118:6809","nonce":864752275}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6806","nonce":864752275},{"type":"v1","addr":"172.21.15.118:6807","nonce":864752275}]},"public_addr":"172.21.15.118:6803/864752275","cluster_addr":"172.21.15.118:6805/864752275","heartbeat_back_addr":"172.21.15.118:6809/864752275","heartbeat_front_addr":"172.21.15.118:6807/864752275","state":["exists","up"]},{"osd":5,"uuid":"bfd9feac-dd06-458c-91b9-af7f175f612c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6810","nonce":2724891770},{"type":"v1","addr":"172.21.15.118:6811","nonce":2724891770}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6812","nonce":2724891770},{"type":"v1","addr":"172.21.15.118:6813","nonce":2724891770}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6816","nonce":2724891770},{"type":"v1","addr":"172.21.15.118:6817","nonce":2724891770}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6814","nonce":2724891770},{"type":"v1","addr":"172.21.15.118:6815","nonce":2724891770}]},"public_addr":"172.21.15.118:6811/2724891770","cluster_addr":"172.21.15.118:6813/2724891770","heartbeat_back_addr":"172.21.15.118:6817/2724891770","heartbeat_front_addr":"172.21.15.118:6815/2724891770","state":["exists","up"]},{"osd":6,"uuid":"5ed5a279-9d8e-416d-bbc7-7edd3e6d5705","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6825","nonce":801947386},{"type":"v1","addr":"172.21.15.118:6827","nonce":801947386}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6828","nonce":801947386},{"type":"v1","addr":"172.21.15.118:6829","nonce":801947386}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6832","nonce":801947386},{"type":"v1","addr":"172.21.15.118:6833","nonce":801947386}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6830","nonce":801947386},{"type":"v1","addr":"172.21.15.118:6831","nonce":801947386}]},"public_addr":"172.21.15.118:6827/801947386","cluster_addr":"172.21.15.118:6829/801947386","heartbeat_back_addr":"172.21.15.118:6833/801947386","heartbeat_front_addr":"172.21.15.118:6831/801947386","state":["exists","up"]},{"osd":7,"uuid":"f2086c47-875e-4451-a436-2008d1ea16af","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6818","nonce":653000048},{"type":"v1","addr":"172.21.15.118:6819","nonce":653000048}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6820","nonce":653000048},{"type":"v1","addr":"172.21.15.118:6821","nonce":653000048}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6824","nonce":653000048},{"type":"v1","addr":"172.21.15.118:6826","nonce":653000048}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6822","nonce":653000048},{"type":"v1","addr":"172.21.15.118:6823","nonce":653000048}]},"public_addr":"172.21.15.118:6819/653000048","cluster_addr":"172.21.15.118:6821/653000048","heartbeat_back_addr":"172.21.15.118:6826/653000048","heartbeat_front_addr":"172.21.15.118:6823/653000048","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-03T20:47:25.491994+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-03T20:47:25.587358+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-03T20:47:25.454175+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-03T20:47:25.377734+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-03T20:47:25.412351+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-03T20:47:25.499815+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-03T20:47:25.575044+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-03T20:47:25.552383+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.32:6835/2963295201":"2024-08-04T20:51:13.985515+0000","172.21.15.32:6834/2963295201":"2024-08-04T20:51:13.985515+0000","172.21.15.32:6835/3848976369":"2024-08-04T20:50:40.678865+0000","172.21.15.32:6834/3848976369":"2024-08-04T20:50:40.678865+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-08-03T20:52:59.443 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:59.441+0000 7f097fa00640 1 -- 172.21.15.32:0/1055452531 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f0958077010 msgr2=0x7f0958079520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:52:59.443 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:59.441+0000 7f097fa00640 1 --2- 172.21.15.32:0/1055452531 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f0958077010 0x7f0958079520 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f096c009bf0 tx=0x7f096c005f90 comp rx=0 tx=0).stop 2024-08-03T20:52:59.443 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:59.441+0000 7f097fa00640 1 -- 172.21.15.32:0/1055452531 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0978162210 msgr2=0x7f0978155bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:52:59.443 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:59.441+0000 7f097fa00640 1 --2- 172.21.15.32:0/1055452531 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0978162210 0x7f0978155bd0 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7f097000c700 tx=0x7f097000cbc0 comp rx=0 tx=0).stop 2024-08-03T20:52:59.443 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:59.441+0000 7f097fa00640 1 -- 172.21.15.32:0/1055452531 shutdown_connections 2024-08-03T20:52:59.443 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:59.441+0000 7f097fa00640 1 --2- 172.21.15.32:0/1055452531 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f0958077010 0x7f0958079520 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:52:59.443 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:59.441+0000 7f097fa00640 1 --2- 172.21.15.32:0/1055452531 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0978162210 0x7f0978155bd0 unknown :-1 s=CLOSED pgs=693 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:52:59.444 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:59.441+0000 7f097fa00640 1 --2- 172.21.15.32:0/1055452531 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f09780b39a0 0x7f09781422a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:52:59.444 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:59.441+0000 7f097fa00640 1 --2- 172.21.15.32:0/1055452531 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f097806f720 0x7f09781f5550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:52:59.444 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:59.441+0000 7f097fa00640 1 -- 172.21.15.32:0/1055452531 >> 172.21.15.32:0/1055452531 conn(0x7f09780b1bd0 msgr2=0x7f09780cfef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:52:59.444 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:59.441+0000 7f097fa00640 1 -- 172.21.15.32:0/1055452531 shutdown_connections 2024-08-03T20:52:59.444 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:59.441+0000 7f097fa00640 1 -- 172.21.15.32:0/1055452531 wait complete. 2024-08-03T20:52:59.466 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-08-03T20:52:59.466 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-08-03T20:52:59.694 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:59.692+0000 7f1772e00640 1 -- 172.21.15.32:0/496449096 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f176c0b29f0 msgr2=0x7f176c0cdba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:52:59.694 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:59.692+0000 7f1772e00640 1 --2- 172.21.15.32:0/496449096 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f176c0b29f0 0x7f176c0cdba0 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7f176c0cd890 tx=0x7f175802a530 comp rx=0 tx=0).stop 2024-08-03T20:52:59.694 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:59.692+0000 7f1772e00640 1 -- 172.21.15.32:0/496449096 shutdown_connections 2024-08-03T20:52:59.694 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:59.692+0000 7f1772e00640 1 --2- 172.21.15.32:0/496449096 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f176c0cfc90 0x7f176c0d00a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:52:59.694 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:59.692+0000 7f1772e00640 1 --2- 172.21.15.32:0/496449096 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f176c0b29f0 0x7f176c0cdba0 unknown :-1 s=CLOSED pgs=694 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:52:59.694 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:59.692+0000 7f1772e00640 1 -- 172.21.15.32:0/496449096 >> 172.21.15.32:0/496449096 conn(0x7f176c0b3bb0 msgr2=0x7f176c0b3fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:52:59.694 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:59.692+0000 7f1772e00640 1 -- 172.21.15.32:0/496449096 shutdown_connections 2024-08-03T20:52:59.694 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:59.692+0000 7f1772e00640 1 -- 172.21.15.32:0/496449096 wait complete. 2024-08-03T20:52:59.695 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:59.694+0000 7f1772e00640 1 Processor -- start 2024-08-03T20:52:59.695 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:59.694+0000 7f1772e00640 1 -- start start 2024-08-03T20:52:59.698 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:59.696+0000 7f1772e00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f176c0b29f0 0x7f176c159520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:52:59.698 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:59.696+0000 7f1772e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f176c0b7760 0x7f176c15ad50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:52:59.698 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:59.696+0000 7f1772e00640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f176c0cfc90 0x7f176c15b290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:52:59.698 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:59.696+0000 7f1772e00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f176c0d0970 con 0x7f176c0b7760 2024-08-03T20:52:59.698 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:59.696+0000 7f1772e00640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f176c0d07f0 con 0x7f176c0b29f0 2024-08-03T20:52:59.698 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:59.696+0000 7f1772e00640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7f176c0b17f0 con 0x7f176c0cfc90 2024-08-03T20:52:59.698 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:59.696+0000 7f176b400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f176c0b7760 0x7f176c15ad50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:52:59.698 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:59.696+0000 7f176b400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f176c0b7760 0x7f176c15ad50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:51384/0 (socket says 172.21.15.32:51384) 2024-08-03T20:52:59.699 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:59.696+0000 7f176b400640 1 -- 172.21.15.32:0/3918845074 learned_addr learned my addr 172.21.15.32:0/3918845074 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:52:59.699 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:59.696+0000 7f176be00640 1 --2- 172.21.15.32:0/3918845074 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f176c0b29f0 0x7f176c159520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:52:59.699 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:59.696+0000 7f1770e00640 1 --2- 172.21.15.32:0/3918845074 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f176c0cfc90 0x7f176c15b290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:52:59.699 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:59.696+0000 7f176b400640 1 -- 172.21.15.32:0/3918845074 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f176c0cfc90 msgr2=0x7f176c15b290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:52:59.699 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:59.696+0000 7f176b400640 1 --2- 172.21.15.32:0/3918845074 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f176c0cfc90 0x7f176c15b290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:52:59.699 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:59.696+0000 7f176b400640 1 -- 172.21.15.32:0/3918845074 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f176c0b29f0 msgr2=0x7f176c159520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:52:59.699 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:59.696+0000 7f176b400640 1 --2- 172.21.15.32:0/3918845074 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f176c0b29f0 0x7f176c159520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:52:59.699 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:59.696+0000 7f176b400640 1 -- 172.21.15.32:0/3918845074 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f176c0cabe0 con 0x7f176c0b7760 2024-08-03T20:52:59.699 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:59.696+0000 7f176be00640 1 --2- 172.21.15.32:0/3918845074 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f176c0b29f0 0x7f176c159520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-03T20:52:59.699 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:59.697+0000 7f176b400640 1 --2- 172.21.15.32:0/3918845074 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f176c0b7760 0x7f176c15ad50 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7f1758000c00 tx=0x7f17580038a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:52:59.700 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:59.697+0000 7f1763e00640 1 -- 172.21.15.32:0/3918845074 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f175803e030 con 0x7f176c0b7760 2024-08-03T20:52:59.700 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:59.697+0000 7f1763e00640 1 -- 172.21.15.32:0/3918845074 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f1758003d90 con 0x7f176c0b7760 2024-08-03T20:52:59.700 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:59.697+0000 7f1772e00640 1 -- 172.21.15.32:0/3918845074 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f176c0c9210 con 0x7f176c0b7760 2024-08-03T20:52:59.700 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:59.697+0000 7f1763e00640 1 -- 172.21.15.32:0/3918845074 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1758035040 con 0x7f176c0b7760 2024-08-03T20:52:59.700 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:59.697+0000 7f1772e00640 1 -- 172.21.15.32:0/3918845074 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f176c15dfd0 con 0x7f176c0b7760 2024-08-03T20:52:59.700 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:59.698+0000 7f1772e00640 1 -- 172.21.15.32:0/3918845074 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f176c0cdba0 con 0x7f176c0b7760 2024-08-03T20:52:59.701 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:59.699+0000 7f1763e00640 1 -- 172.21.15.32:0/3918845074 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f1758041030 con 0x7f176c0b7760 2024-08-03T20:52:59.701 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:59.700+0000 7f1763e00640 1 --2- 172.21.15.32:0/3918845074 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f174c077040 0x7f174c079550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:52:59.702 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:59.700+0000 7f1763e00640 1 -- 172.21.15.32:0/3918845074 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(59..59 src has 1..59) ==== 5881+0+0 (secure 0 0 0) 0x7f17580b91f0 con 0x7f176c0b7760 2024-08-03T20:52:59.702 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:59.701+0000 7f176be00640 1 --2- 172.21.15.32:0/3918845074 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f174c077040 0x7f174c079550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:52:59.703 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:59.702+0000 7f176be00640 1 --2- 172.21.15.32:0/3918845074 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f174c077040 0x7f174c079550 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f176c0c5bc0 tx=0x7f175c009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:52:59.706 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:59.704+0000 7f1763e00640 1 -- 172.21.15.32:0/3918845074 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7f1758082220 con 0x7f176c0b7760 2024-08-03T20:52:59.948 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:59.947+0000 7f1772e00640 1 -- 172.21.15.32:0/3918845074 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f176c0b4760 con 0x7f176c0b7760 2024-08-03T20:52:59.949 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:59.948+0000 7f1763e00640 1 -- 172.21.15.32:0/3918845074 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 65 v65) ==== 94+0+5231 (secure 0 0 0) 0x7f1758081b20 con 0x7f176c0b7760 2024-08-03T20:52:59.949 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-03T20:52:59.950 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":65,"btime":"2024-08-03T20:51:16:994952+0000","default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5261,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6833/2498840646","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":2498840646},{"type":"v1","addr":"172.21.15.32:6833","nonce":2498840646}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":45},{"gid":5276,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6837/3142148286","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6835","nonce":3142148286},{"type":"v1","addr":"172.21.15.118:6837","nonce":3142148286}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":46},{"gid":6180,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6835/1934242557","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1934242557},{"type":"v1","addr":"172.21.15.32:6835","nonce":1934242557}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":63}],"filesystems":[{"mdsmap":{"epoch":65,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-08-03T20:50:27.688698+0000","modified":"2024-08-03T20:51:16.994940+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":59,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5632},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5632":{"gid":5632,"name":"d","rank":0,"incarnation":62,"state":"up:active","state_seq":16,"addr":"172.21.15.118:6836/3720010453","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6834","nonce":3720010453},{"type":"v1","addr":"172.21.15.118:6836","nonce":3720010453}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[9],"metadata_pool":8,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5632,"qdb_cluster":[5632]},"id":4}]} 2024-08-03T20:52:59.950 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 65 2024-08-03T20:52:59.952 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:59.950+0000 7f1772e00640 1 -- 172.21.15.32:0/3918845074 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f174c077040 msgr2=0x7f174c079550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:52:59.952 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:59.950+0000 7f1772e00640 1 --2- 172.21.15.32:0/3918845074 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f174c077040 0x7f174c079550 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f176c0c5bc0 tx=0x7f175c009040 comp rx=0 tx=0).stop 2024-08-03T20:52:59.952 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:59.950+0000 7f1772e00640 1 -- 172.21.15.32:0/3918845074 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f176c0b7760 msgr2=0x7f176c15ad50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:52:59.952 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:59.950+0000 7f1772e00640 1 --2- 172.21.15.32:0/3918845074 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f176c0b7760 0x7f176c15ad50 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7f1758000c00 tx=0x7f17580038a0 comp rx=0 tx=0).stop 2024-08-03T20:52:59.952 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:59.950+0000 7f1772e00640 1 -- 172.21.15.32:0/3918845074 shutdown_connections 2024-08-03T20:52:59.952 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:59.950+0000 7f1772e00640 1 --2- 172.21.15.32:0/3918845074 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f174c077040 0x7f174c079550 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:52:59.952 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:59.950+0000 7f1772e00640 1 --2- 172.21.15.32:0/3918845074 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f176c0cfc90 0x7f176c15b290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:52:59.952 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:59.950+0000 7f1772e00640 1 --2- 172.21.15.32:0/3918845074 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f176c0b7760 0x7f176c15ad50 unknown :-1 s=CLOSED pgs=695 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:52:59.952 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:59.950+0000 7f1772e00640 1 --2- 172.21.15.32:0/3918845074 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f176c0b29f0 0x7f176c159520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:52:59.952 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:59.950+0000 7f1772e00640 1 -- 172.21.15.32:0/3918845074 >> 172.21.15.32:0/3918845074 conn(0x7f176c0b3bb0 msgr2=0x7f176c163f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:52:59.952 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:59.950+0000 7f1772e00640 1 -- 172.21.15.32:0/3918845074 shutdown_connections 2024-08-03T20:52:59.952 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:52:59.950+0000 7f1772e00640 1 -- 172.21.15.32:0/3918845074 wait complete. 2024-08-03T20:52:59.974 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-08-03T20:53:00.207 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:00.205+0000 7f8d18e00640 1 -- 172.21.15.32:0/909645919 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f8d140d0ff0 msgr2=0x7f8cf8005190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:53:00.208 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:00.205+0000 7f8d18e00640 1 --2- 172.21.15.32:0/909645919 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f8d140d0ff0 0x7f8cf8005190 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f8d0c005fd0 tx=0x7f8cf8005830 comp rx=0 tx=0).stop 2024-08-03T20:53:00.208 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:00.205+0000 7f8d13e00640 1 -- 172.21.15.32:0/909645919 reap_dead start 2024-08-03T20:53:00.208 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:00.205+0000 7f8d18e00640 1 -- 172.21.15.32:0/909645919 shutdown_connections 2024-08-03T20:53:00.208 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:00.205+0000 7f8d18e00640 1 -- 172.21.15.32:0/909645919 >> 172.21.15.32:0/909645919 conn(0x7f8d140d40c0 msgr2=0x7f8d140d44d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:53:00.208 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:00.205+0000 7f8d18e00640 1 -- 172.21.15.32:0/909645919 shutdown_connections 2024-08-03T20:53:00.208 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:00.205+0000 7f8d18e00640 1 -- 172.21.15.32:0/909645919 wait complete. 2024-08-03T20:53:00.209 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:00.208+0000 7f8d18e00640 1 Processor -- start 2024-08-03T20:53:00.209 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:00.208+0000 7f8d18e00640 1 -- start start 2024-08-03T20:53:00.212 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:00.209+0000 7f8d18e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8d140d0ff0 0x7f8d1414f460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:53:00.212 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:00.209+0000 7f8d18e00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8d140a18b0 0x7f8d1409f790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:53:00.212 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:00.209+0000 7f8d18e00640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f8d1409fcd0 0x7f8d141f8b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:53:00.212 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:00.209+0000 7f8d18e00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f8d140b7730 con 0x7f8d140d0ff0 2024-08-03T20:53:00.212 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:00.209+0000 7f8d18e00640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f8d140b75b0 con 0x7f8d140a18b0 2024-08-03T20:53:00.212 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:00.209+0000 7f8d18e00640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7f8d140b78b0 con 0x7f8d1409fcd0 2024-08-03T20:53:00.212 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:00.209+0000 7f8d13400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8d140d0ff0 0x7f8d1414f460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:53:00.212 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:00.209+0000 7f8d13400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8d140d0ff0 0x7f8d1414f460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:40002/0 (socket says 172.21.15.32:40002) 2024-08-03T20:53:00.213 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:00.209+0000 7f8d13400640 1 -- 172.21.15.32:0/2953460605 learned_addr learned my addr 172.21.15.32:0/2953460605 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:53:00.213 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:00.209+0000 7f8d12a00640 1 --2- 172.21.15.32:0/2953460605 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8d140a18b0 0x7f8d1409f790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:53:00.213 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:00.210+0000 7f8d13e00640 1 --2- 172.21.15.32:0/2953460605 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f8d1409fcd0 0x7f8d141f8b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:53:00.213 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:00.210+0000 7f8d13400640 1 -- 172.21.15.32:0/2953460605 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f8d1409fcd0 msgr2=0x7f8d141f8b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:53:00.213 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:00.210+0000 7f8d13400640 1 --2- 172.21.15.32:0/2953460605 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f8d1409fcd0 0x7f8d141f8b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:53:00.213 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:00.210+0000 7f8d13400640 1 -- 172.21.15.32:0/2953460605 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8d140a18b0 msgr2=0x7f8d1409f790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:53:00.213 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:00.210+0000 7f8d13400640 1 --2- 172.21.15.32:0/2953460605 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8d140a18b0 0x7f8d1409f790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:53:00.213 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:00.210+0000 7f8d13400640 1 -- 172.21.15.32:0/2953460605 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8d140a00e0 con 0x7f8d140d0ff0 2024-08-03T20:53:00.213 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:00.210+0000 7f8d12a00640 1 --2- 172.21.15.32:0/2953460605 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8d140a18b0 0x7f8d1409f790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-03T20:53:00.213 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:00.210+0000 7f8d13400640 1 --2- 172.21.15.32:0/2953460605 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8d140d0ff0 0x7f8d1414f460 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7f8d04006ed0 tx=0x7f8d040039c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:53:00.213 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:00.210+0000 7f8d0b400640 1 -- 172.21.15.32:0/2953460605 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d04015070 con 0x7f8d140d0ff0 2024-08-03T20:53:00.213 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:00.210+0000 7f8d0b400640 1 -- 172.21.15.32:0/2953460605 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f8d04011900 con 0x7f8d140d0ff0 2024-08-03T20:53:00.213 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:00.210+0000 7f8d0b400640 1 -- 172.21.15.32:0/2953460605 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d04011d10 con 0x7f8d140d0ff0 2024-08-03T20:53:00.213 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:00.210+0000 7f8d18e00640 1 -- 172.21.15.32:0/2953460605 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8d1415d830 con 0x7f8d140d0ff0 2024-08-03T20:53:00.214 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:00.211+0000 7f8d18e00640 1 -- 172.21.15.32:0/2953460605 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8d1409e3c0 con 0x7f8d140d0ff0 2024-08-03T20:53:00.214 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:00.212+0000 7f8d18e00640 1 -- 172.21.15.32:0/2953460605 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8d140d0ab0 con 0x7f8d140d0ff0 2024-08-03T20:53:00.214 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:00.213+0000 7f8d0b400640 1 -- 172.21.15.32:0/2953460605 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f8d0400f430 con 0x7f8d140d0ff0 2024-08-03T20:53:00.215 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:00.214+0000 7f8d0b400640 1 --2- 172.21.15.32:0/2953460605 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f8cf4077010 0x7f8cf4079520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:53:00.216 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:00.215+0000 7f8d0b400640 1 -- 172.21.15.32:0/2953460605 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(59..59 src has 1..59) ==== 5881+0+0 (secure 0 0 0) 0x7f8d040995b0 con 0x7f8d140d0ff0 2024-08-03T20:53:00.216 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:00.215+0000 7f8d12a00640 1 --2- 172.21.15.32:0/2953460605 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f8cf4077010 0x7f8cf4079520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:53:00.217 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:00.216+0000 7f8d12a00640 1 --2- 172.21.15.32:0/2953460605 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f8cf4077010 0x7f8cf4079520 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f8d140b5180 tx=0x7f8d00005f90 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:53:00.221 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:00.219+0000 7f8d0b400640 1 -- 172.21.15.32:0/2953460605 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7f8d140a00e0 con 0x7f8d140d0ff0 2024-08-03T20:53:00.463 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:00.462+0000 7f8d18e00640 1 -- 172.21.15.32:0/2953460605 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f8d140b57e0 con 0x7f8d140d0ff0 2024-08-03T20:53:00.465 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:00.464+0000 7f8d0b400640 1 -- 172.21.15.32:0/2953460605 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 65 v65) ==== 94+0+5231 (secure 0 0 0) 0x7f8d140d1400 con 0x7f8d140d0ff0 2024-08-03T20:53:00.466 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-03T20:53:00.466 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":65,"btime":"2024-08-03T20:51:16:994952+0000","default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5261,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6833/2498840646","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":2498840646},{"type":"v1","addr":"172.21.15.32:6833","nonce":2498840646}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":45},{"gid":5276,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6837/3142148286","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6835","nonce":3142148286},{"type":"v1","addr":"172.21.15.118:6837","nonce":3142148286}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":46},{"gid":6180,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6835/1934242557","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1934242557},{"type":"v1","addr":"172.21.15.32:6835","nonce":1934242557}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":63}],"filesystems":[{"mdsmap":{"epoch":65,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-08-03T20:50:27.688698+0000","modified":"2024-08-03T20:51:16.994940+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":59,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5632},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5632":{"gid":5632,"name":"d","rank":0,"incarnation":62,"state":"up:active","state_seq":16,"addr":"172.21.15.118:6836/3720010453","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6834","nonce":3720010453},{"type":"v1","addr":"172.21.15.118:6836","nonce":3720010453}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[9],"metadata_pool":8,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5632,"qdb_cluster":[5632]},"id":4}]} 2024-08-03T20:53:00.466 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 65 2024-08-03T20:53:00.468 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:00.466+0000 7f8d18e00640 1 -- 172.21.15.32:0/2953460605 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f8cf4077010 msgr2=0x7f8cf4079520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:53:00.468 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:00.466+0000 7f8d18e00640 1 --2- 172.21.15.32:0/2953460605 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f8cf4077010 0x7f8cf4079520 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f8d140b5180 tx=0x7f8d00005f90 comp rx=0 tx=0).stop 2024-08-03T20:53:00.468 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:00.466+0000 7f8d18e00640 1 -- 172.21.15.32:0/2953460605 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8d140d0ff0 msgr2=0x7f8d1414f460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:53:00.468 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:00.466+0000 7f8d18e00640 1 --2- 172.21.15.32:0/2953460605 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8d140d0ff0 0x7f8d1414f460 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7f8d04006ed0 tx=0x7f8d040039c0 comp rx=0 tx=0).stop 2024-08-03T20:53:00.468 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:00.466+0000 7f8d18e00640 1 -- 172.21.15.32:0/2953460605 shutdown_connections 2024-08-03T20:53:00.468 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:00.466+0000 7f8d18e00640 1 --2- 172.21.15.32:0/2953460605 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f8cf4077010 0x7f8cf4079520 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:53:00.469 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:00.466+0000 7f8d18e00640 1 --2- 172.21.15.32:0/2953460605 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8d140a18b0 0x7f8d1409f790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:53:00.469 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:00.466+0000 7f8d18e00640 1 --2- 172.21.15.32:0/2953460605 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f8d1409fcd0 0x7f8d141f8b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:53:00.469 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:00.466+0000 7f8d18e00640 1 --2- 172.21.15.32:0/2953460605 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8d140d0ff0 0x7f8d1414f460 unknown :-1 s=CLOSED pgs=697 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:53:00.469 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:00.466+0000 7f8d18e00640 1 -- 172.21.15.32:0/2953460605 >> 172.21.15.32:0/2953460605 conn(0x7f8d140d40c0 msgr2=0x7f8d1406e030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:53:00.469 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:00.466+0000 7f8d18e00640 1 -- 172.21.15.32:0/2953460605 shutdown_connections 2024-08-03T20:53:00.469 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:00.467+0000 7f8d18e00640 1 -- 172.21.15.32:0/2953460605 wait complete. 2024-08-03T20:53:00.492 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-08-03T20:53:00.720 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:00.718+0000 7f41bd200640 1 -- 172.21.15.32:0/2898902462 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f41b815fff0 msgr2=0x7f41b8160400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:53:00.720 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:00.718+0000 7f41bd200640 1 --2- 172.21.15.32:0/2898902462 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f41b815fff0 0x7f41b8160400 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7f41b8071700 tx=0x7f41b002aac0 comp rx=0 tx=0).stop 2024-08-03T20:53:00.720 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:00.719+0000 7f41bd200640 1 -- 172.21.15.32:0/2898902462 shutdown_connections 2024-08-03T20:53:00.721 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:00.719+0000 7f41bd200640 1 --2- 172.21.15.32:0/2898902462 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f41b815fff0 0x7f41b8160400 unknown :-1 s=CLOSED pgs=699 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:53:00.721 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:00.719+0000 7f41bd200640 1 --2- 172.21.15.32:0/2898902462 >> v2:172.21.15.118:3300/0 conn(0x7f41b806e930 0x7f41b8144f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:53:00.721 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:00.719+0000 7f41bd200640 1 -- 172.21.15.32:0/2898902462 >> 172.21.15.32:0/2898902462 conn(0x7f41b80b1bd0 msgr2=0x7f41b80b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:53:00.721 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:00.719+0000 7f41bd200640 1 -- 172.21.15.32:0/2898902462 shutdown_connections 2024-08-03T20:53:00.721 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:00.719+0000 7f41bd200640 1 -- 172.21.15.32:0/2898902462 wait complete. 2024-08-03T20:53:00.722 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:00.721+0000 7f41bd200640 1 Processor -- start 2024-08-03T20:53:00.722 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:00.721+0000 7f41bd200640 1 -- start start 2024-08-03T20:53:00.724 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:00.722+0000 7f41bd200640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f41b806e930 0x7f41b813cbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:53:00.725 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:00.722+0000 7f41bd200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f41b8073e60 0x7f41b813aa30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:53:00.725 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:00.722+0000 7f41bd200640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f41b815fff0 0x7f41b81437f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:53:00.725 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:00.722+0000 7f41bd200640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f41b80cf840 con 0x7f41b8073e60 2024-08-03T20:53:00.725 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:00.722+0000 7f41bd200640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f41b8145f30 con 0x7f41b815fff0 2024-08-03T20:53:00.725 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:00.722+0000 7f41bd200640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7f41b80cf9c0 con 0x7f41b806e930 2024-08-03T20:53:00.725 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:00.723+0000 7f41b5400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f41b8073e60 0x7f41b813aa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:53:00.725 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:00.723+0000 7f41b5400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f41b8073e60 0x7f41b813aa30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:40012/0 (socket says 172.21.15.32:40012) 2024-08-03T20:53:00.725 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:00.723+0000 7f41b5400640 1 -- 172.21.15.32:0/1694501569 learned_addr learned my addr 172.21.15.32:0/1694501569 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:53:00.725 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:00.723+0000 7f41b6800640 1 --2- 172.21.15.32:0/1694501569 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f41b815fff0 0x7f41b81437f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:53:00.725 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:00.723+0000 7f41b5e00640 1 --2- 172.21.15.32:0/1694501569 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f41b806e930 0x7f41b813cbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:53:00.725 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:00.723+0000 7f41b5400640 1 -- 172.21.15.32:0/1694501569 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f41b806e930 msgr2=0x7f41b813cbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:53:00.725 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:00.723+0000 7f41b5400640 1 --2- 172.21.15.32:0/1694501569 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f41b806e930 0x7f41b813cbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:53:00.725 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:00.723+0000 7f41b5400640 1 -- 172.21.15.32:0/1694501569 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f41b815fff0 msgr2=0x7f41b81437f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:53:00.725 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:00.723+0000 7f41b5400640 1 --2- 172.21.15.32:0/1694501569 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f41b815fff0 0x7f41b81437f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:53:00.726 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:00.723+0000 7f41b5400640 1 -- 172.21.15.32:0/1694501569 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f41b813af70 con 0x7f41b8073e60 2024-08-03T20:53:00.726 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:00.723+0000 7f41b5400640 1 --2- 172.21.15.32:0/1694501569 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f41b8073e60 0x7f41b813aa30 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7f41a400c370 tx=0x7f41a400c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:53:00.727 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:00.724+0000 7f41ae000640 1 -- 172.21.15.32:0/1694501569 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41a4017070 con 0x7f41b8073e60 2024-08-03T20:53:00.727 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:00.724+0000 7f41ae000640 1 -- 172.21.15.32:0/1694501569 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f41a40053c0 con 0x7f41b8073e60 2024-08-03T20:53:00.727 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:00.724+0000 7f41ae000640 1 -- 172.21.15.32:0/1694501569 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41a4003850 con 0x7f41b8073e60 2024-08-03T20:53:00.727 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:00.724+0000 7f41bd200640 1 -- 172.21.15.32:0/1694501569 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f41b813db00 con 0x7f41b8073e60 2024-08-03T20:53:00.727 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:00.724+0000 7f41bd200640 1 -- 172.21.15.32:0/1694501569 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f41b813d820 con 0x7f41b8073e60 2024-08-03T20:53:00.727 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:00.725+0000 7f41bd200640 1 -- 172.21.15.32:0/1694501569 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f41b813d2b0 con 0x7f41b8073e60 2024-08-03T20:53:00.727 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:00.726+0000 7f41ae000640 1 -- 172.21.15.32:0/1694501569 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f41a401a070 con 0x7f41b8073e60 2024-08-03T20:53:00.729 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:00.728+0000 7f41ae000640 1 --2- 172.21.15.32:0/1694501569 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f4198077010 0x7f4198079520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:53:00.729 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:00.728+0000 7f41ae000640 1 -- 172.21.15.32:0/1694501569 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(59..59 src has 1..59) ==== 5881+0+0 (secure 0 0 0) 0x7f41a4097e20 con 0x7f41b8073e60 2024-08-03T20:53:00.730 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:00.728+0000 7f41b5e00640 1 --2- 172.21.15.32:0/1694501569 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f4198077010 0x7f4198079520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:53:00.730 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:00.729+0000 7f41b5e00640 1 --2- 172.21.15.32:0/1694501569 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f4198077010 0x7f4198079520 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f41a8005fd0 tx=0x7f41a8005850 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:53:00.734 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:00.732+0000 7f41ae000640 1 -- 172.21.15.32:0/1694501569 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7f41b813af70 con 0x7f41b8073e60 2024-08-03T20:53:00.939 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:00.937+0000 7f41bd200640 1 -- 172.21.15.32:0/1694501569 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f41b8160be0 con 0x7f41b8073e60 2024-08-03T20:53:00.940 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:00.939+0000 7f41ae000640 1 -- 172.21.15.32:0/1694501569 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v59) ==== 74+0+16864 (secure 0 0 0) 0x7f41a4060f60 con 0x7f41b8073e60 2024-08-03T20:53:00.941 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-03T20:53:00.941 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":59,"fsid":"40ab141a-01d7-49b9-ae3c-2edec9876e95","created":"2024-08-03T20:47:14.957362+0000","modified":"2024-08-03T20:51:13.985593+0000","last_up_change":"2024-08-03T20:47:28.874451+0000","last_in_change":"2024-08-03T20:47:20.079442+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-08-03T20:47:28.261300+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_metadata","create_time":"2024-08-03T20:50:25.652248+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"56","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_data","create_time":"2024-08-03T20:50:26.646008+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"57","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5a83e759-9035-438b-889a-0a6af1f0a0d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6816","nonce":3066156116},{"type":"v1","addr":"172.21.15.32:6817","nonce":3066156116}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6818","nonce":3066156116},{"type":"v1","addr":"172.21.15.32:6819","nonce":3066156116}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6822","nonce":3066156116},{"type":"v1","addr":"172.21.15.32:6823","nonce":3066156116}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6820","nonce":3066156116},{"type":"v1","addr":"172.21.15.32:6821","nonce":3066156116}]},"public_addr":"172.21.15.32:6817/3066156116","cluster_addr":"172.21.15.32:6819/3066156116","heartbeat_back_addr":"172.21.15.32:6823/3066156116","heartbeat_front_addr":"172.21.15.32:6821/3066156116","state":["exists","up"]},{"osd":1,"uuid":"d7c5c210-da4f-49f9-b9e7-1dfb02daa4bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6824","nonce":1551241759},{"type":"v1","addr":"172.21.15.32:6825","nonce":1551241759}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6826","nonce":1551241759},{"type":"v1","addr":"172.21.15.32:6827","nonce":1551241759}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6830","nonce":1551241759},{"type":"v1","addr":"172.21.15.32:6831","nonce":1551241759}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6828","nonce":1551241759},{"type":"v1","addr":"172.21.15.32:6829","nonce":1551241759}]},"public_addr":"172.21.15.32:6825/1551241759","cluster_addr":"172.21.15.32:6827/1551241759","heartbeat_back_addr":"172.21.15.32:6831/1551241759","heartbeat_front_addr":"172.21.15.32:6829/1551241759","state":["exists","up"]},{"osd":2,"uuid":"57f2e6de-a201-45c2-8981-8e56a4875ebd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6808","nonce":3806708944},{"type":"v1","addr":"172.21.15.32:6809","nonce":3806708944}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6810","nonce":3806708944},{"type":"v1","addr":"172.21.15.32:6811","nonce":3806708944}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6814","nonce":3806708944},{"type":"v1","addr":"172.21.15.32:6815","nonce":3806708944}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6812","nonce":3806708944},{"type":"v1","addr":"172.21.15.32:6813","nonce":3806708944}]},"public_addr":"172.21.15.32:6809/3806708944","cluster_addr":"172.21.15.32:6811/3806708944","heartbeat_back_addr":"172.21.15.32:6815/3806708944","heartbeat_front_addr":"172.21.15.32:6813/3806708944","state":["exists","up"]},{"osd":3,"uuid":"6439a280-da45-4148-8130-f16323935ff3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6800","nonce":9088303},{"type":"v1","addr":"172.21.15.32:6801","nonce":9088303}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6802","nonce":9088303},{"type":"v1","addr":"172.21.15.32:6803","nonce":9088303}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6806","nonce":9088303},{"type":"v1","addr":"172.21.15.32:6807","nonce":9088303}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6804","nonce":9088303},{"type":"v1","addr":"172.21.15.32:6805","nonce":9088303}]},"public_addr":"172.21.15.32:6801/9088303","cluster_addr":"172.21.15.32:6803/9088303","heartbeat_back_addr":"172.21.15.32:6807/9088303","heartbeat_front_addr":"172.21.15.32:6805/9088303","state":["exists","up"]},{"osd":4,"uuid":"4f902eff-1e18-418d-82c9-2151c69524fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6802","nonce":864752275},{"type":"v1","addr":"172.21.15.118:6803","nonce":864752275}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6804","nonce":864752275},{"type":"v1","addr":"172.21.15.118:6805","nonce":864752275}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6808","nonce":864752275},{"type":"v1","addr":"172.21.15.118:6809","nonce":864752275}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6806","nonce":864752275},{"type":"v1","addr":"172.21.15.118:6807","nonce":864752275}]},"public_addr":"172.21.15.118:6803/864752275","cluster_addr":"172.21.15.118:6805/864752275","heartbeat_back_addr":"172.21.15.118:6809/864752275","heartbeat_front_addr":"172.21.15.118:6807/864752275","state":["exists","up"]},{"osd":5,"uuid":"bfd9feac-dd06-458c-91b9-af7f175f612c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6810","nonce":2724891770},{"type":"v1","addr":"172.21.15.118:6811","nonce":2724891770}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6812","nonce":2724891770},{"type":"v1","addr":"172.21.15.118:6813","nonce":2724891770}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6816","nonce":2724891770},{"type":"v1","addr":"172.21.15.118:6817","nonce":2724891770}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6814","nonce":2724891770},{"type":"v1","addr":"172.21.15.118:6815","nonce":2724891770}]},"public_addr":"172.21.15.118:6811/2724891770","cluster_addr":"172.21.15.118:6813/2724891770","heartbeat_back_addr":"172.21.15.118:6817/2724891770","heartbeat_front_addr":"172.21.15.118:6815/2724891770","state":["exists","up"]},{"osd":6,"uuid":"5ed5a279-9d8e-416d-bbc7-7edd3e6d5705","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6825","nonce":801947386},{"type":"v1","addr":"172.21.15.118:6827","nonce":801947386}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6828","nonce":801947386},{"type":"v1","addr":"172.21.15.118:6829","nonce":801947386}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6832","nonce":801947386},{"type":"v1","addr":"172.21.15.118:6833","nonce":801947386}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6830","nonce":801947386},{"type":"v1","addr":"172.21.15.118:6831","nonce":801947386}]},"public_addr":"172.21.15.118:6827/801947386","cluster_addr":"172.21.15.118:6829/801947386","heartbeat_back_addr":"172.21.15.118:6833/801947386","heartbeat_front_addr":"172.21.15.118:6831/801947386","state":["exists","up"]},{"osd":7,"uuid":"f2086c47-875e-4451-a436-2008d1ea16af","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6818","nonce":653000048},{"type":"v1","addr":"172.21.15.118:6819","nonce":653000048}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6820","nonce":653000048},{"type":"v1","addr":"172.21.15.118:6821","nonce":653000048}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6824","nonce":653000048},{"type":"v1","addr":"172.21.15.118:6826","nonce":653000048}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6822","nonce":653000048},{"type":"v1","addr":"172.21.15.118:6823","nonce":653000048}]},"public_addr":"172.21.15.118:6819/653000048","cluster_addr":"172.21.15.118:6821/653000048","heartbeat_back_addr":"172.21.15.118:6826/653000048","heartbeat_front_addr":"172.21.15.118:6823/653000048","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-03T20:47:25.491994+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-03T20:47:25.587358+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-03T20:47:25.454175+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-03T20:47:25.377734+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-03T20:47:25.412351+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-03T20:47:25.499815+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-03T20:47:25.575044+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-03T20:47:25.552383+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.32:6835/2963295201":"2024-08-04T20:51:13.985515+0000","172.21.15.32:6834/2963295201":"2024-08-04T20:51:13.985515+0000","172.21.15.32:6835/3848976369":"2024-08-04T20:50:40.678865+0000","172.21.15.32:6834/3848976369":"2024-08-04T20:50:40.678865+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-08-03T20:53:00.943 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:00.941+0000 7f41bd200640 1 -- 172.21.15.32:0/1694501569 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f4198077010 msgr2=0x7f4198079520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:53:00.943 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:00.941+0000 7f41bd200640 1 --2- 172.21.15.32:0/1694501569 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f4198077010 0x7f4198079520 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f41a8005fd0 tx=0x7f41a8005850 comp rx=0 tx=0).stop 2024-08-03T20:53:00.943 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:00.941+0000 7f41bd200640 1 -- 172.21.15.32:0/1694501569 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f41b8073e60 msgr2=0x7f41b813aa30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:53:00.943 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:00.941+0000 7f41bd200640 1 --2- 172.21.15.32:0/1694501569 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f41b8073e60 0x7f41b813aa30 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7f41a400c370 tx=0x7f41a400c830 comp rx=0 tx=0).stop 2024-08-03T20:53:00.943 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:00.941+0000 7f41bd200640 1 -- 172.21.15.32:0/1694501569 shutdown_connections 2024-08-03T20:53:00.943 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:00.941+0000 7f41bd200640 1 --2- 172.21.15.32:0/1694501569 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f4198077010 0x7f4198079520 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:53:00.943 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:00.941+0000 7f41bd200640 1 --2- 172.21.15.32:0/1694501569 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f41b815fff0 0x7f41b81437f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:53:00.944 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:00.941+0000 7f41bd200640 1 --2- 172.21.15.32:0/1694501569 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f41b8073e60 0x7f41b813aa30 unknown :-1 s=CLOSED pgs=700 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:53:00.944 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:00.941+0000 7f41bd200640 1 --2- 172.21.15.32:0/1694501569 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f41b806e930 0x7f41b813cbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:53:00.944 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:00.941+0000 7f41bd200640 1 -- 172.21.15.32:0/1694501569 >> 172.21.15.32:0/1694501569 conn(0x7f41b80b1bd0 msgr2=0x7f41b80b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:53:00.944 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:00.941+0000 7f41bd200640 1 -- 172.21.15.32:0/1694501569 shutdown_connections 2024-08-03T20:53:00.944 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:00.941+0000 7f41bd200640 1 -- 172.21.15.32:0/1694501569 wait complete. 2024-08-03T20:53:00.965 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-08-03T20:53:01.194 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:01.192+0000 7f57b6000640 1 -- 172.21.15.32:0/327755915 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f57b0155970 msgr2=0x7f57b015c2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:53:01.194 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:01.192+0000 7f57b6000640 1 --2- 172.21.15.32:0/327755915 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f57b0155970 0x7f57b015c2c0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f57a0005fd0 tx=0x7f57a002a5d0 comp rx=0 tx=0).stop 2024-08-03T20:53:01.194 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:01.192+0000 7f57b6000640 1 -- 172.21.15.32:0/327755915 shutdown_connections 2024-08-03T20:53:01.194 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:01.192+0000 7f57b6000640 1 --2- 172.21.15.32:0/327755915 >> v2:172.21.15.118:3300/0 conn(0x7f57b0073ff0 0x7f57b015f290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:53:01.194 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:01.192+0000 7f57b6000640 1 --2- 172.21.15.32:0/327755915 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f57b0155970 0x7f57b015c2c0 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:53:01.194 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:01.192+0000 7f57b6000640 1 -- 172.21.15.32:0/327755915 >> 172.21.15.32:0/327755915 conn(0x7f57b00b34b0 msgr2=0x7f57b00d1050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:53:01.194 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:01.192+0000 7f57b6000640 1 -- 172.21.15.32:0/327755915 shutdown_connections 2024-08-03T20:53:01.194 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:01.192+0000 7f57b6000640 1 -- 172.21.15.32:0/327755915 wait complete. 2024-08-03T20:53:01.196 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:01.194+0000 7f57b6000640 1 Processor -- start 2024-08-03T20:53:01.196 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:01.195+0000 7f57b6000640 1 -- start start 2024-08-03T20:53:01.198 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:01.197+0000 7f57b6000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f57b0155970 0x7f57b01409e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:53:01.199 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:01.197+0000 7f57b6000640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f57b015ee80 0x7f57b00cc340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:53:01.199 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:01.197+0000 7f57b6000640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f57b00cc880 0x7f57b015de70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:53:01.199 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:01.197+0000 7f57b6000640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f57b00b4350 con 0x7f57b0155970 2024-08-03T20:53:01.199 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:01.197+0000 7f57b6000640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f57b00b41d0 con 0x7f57b00cc880 2024-08-03T20:53:01.199 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:01.197+0000 7f57b6000640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7f57b00b44d0 con 0x7f57b015ee80 2024-08-03T20:53:01.199 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:01.197+0000 7f57af400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f57b0155970 0x7f57b01409e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:53:01.199 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:01.197+0000 7f57af400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f57b0155970 0x7f57b01409e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:40016/0 (socket says 172.21.15.32:40016) 2024-08-03T20:53:01.199 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:01.197+0000 7f57af400640 1 -- 172.21.15.32:0/4149010856 learned_addr learned my addr 172.21.15.32:0/4149010856 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:53:01.199 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:01.197+0000 7f57aea00640 1 --2- 172.21.15.32:0/4149010856 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f57b015ee80 0x7f57b00cc340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:53:01.199 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:01.197+0000 7f57afe00640 1 --2- 172.21.15.32:0/4149010856 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f57b00cc880 0x7f57b015de70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:53:01.199 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:01.197+0000 7f57af400640 1 -- 172.21.15.32:0/4149010856 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f57b015ee80 msgr2=0x7f57b00cc340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:53:01.199 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:01.197+0000 7f57af400640 1 --2- 172.21.15.32:0/4149010856 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f57b015ee80 0x7f57b00cc340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:53:01.199 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:01.197+0000 7f57af400640 1 -- 172.21.15.32:0/4149010856 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f57b00cc880 msgr2=0x7f57b015de70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:53:01.200 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:01.197+0000 7f57af400640 1 --2- 172.21.15.32:0/4149010856 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f57b00cc880 0x7f57b015de70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:53:01.200 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:01.197+0000 7f57af400640 1 -- 172.21.15.32:0/4149010856 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f57b01f63c0 con 0x7f57b0155970 2024-08-03T20:53:01.200 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:01.197+0000 7f57af400640 1 --2- 172.21.15.32:0/4149010856 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f57b0155970 0x7f57b01409e0 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 crypto rx=0x7f57a0034040 tx=0x7f57a00035f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:53:01.200 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:01.198+0000 7f57a7400640 1 -- 172.21.15.32:0/4149010856 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57a003e030 con 0x7f57b0155970 2024-08-03T20:53:01.200 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:01.198+0000 7f57a7400640 1 -- 172.21.15.32:0/4149010856 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f57a0003b00 con 0x7f57b0155970 2024-08-03T20:53:01.201 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:01.198+0000 7f57b6000640 1 -- 172.21.15.32:0/4149010856 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f57b01f8890 con 0x7f57b0155970 2024-08-03T20:53:01.201 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:01.198+0000 7f57a7400640 1 -- 172.21.15.32:0/4149010856 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57a0035040 con 0x7f57b0155970 2024-08-03T20:53:01.201 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:01.198+0000 7f57b6000640 1 -- 172.21.15.32:0/4149010856 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f57b01f8e10 con 0x7f57b0155970 2024-08-03T20:53:01.201 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:01.199+0000 7f57b6000640 1 -- 172.21.15.32:0/4149010856 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f57b01f6590 con 0x7f57b0155970 2024-08-03T20:53:01.201 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:01.200+0000 7f57a7400640 1 -- 172.21.15.32:0/4149010856 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f57a0041030 con 0x7f57b0155970 2024-08-03T20:53:01.202 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:01.202+0000 7f57a7400640 1 --2- 172.21.15.32:0/4149010856 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f5790077040 0x7f5790079550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:53:01.203 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:01.202+0000 7f57a7400640 1 -- 172.21.15.32:0/4149010856 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(59..59 src has 1..59) ==== 5881+0+0 (secure 0 0 0) 0x7f57a00b8c90 con 0x7f57b0155970 2024-08-03T20:53:01.203 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:01.202+0000 7f57aea00640 1 --2- 172.21.15.32:0/4149010856 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f5790077040 0x7f5790079550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:53:01.203 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:01.202+0000 7f57aea00640 1 --2- 172.21.15.32:0/4149010856 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f5790077040 0x7f5790079550 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f57b015b4a0 tx=0x7f579c009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:53:01.206 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:01.205+0000 7f57a7400640 1 -- 172.21.15.32:0/4149010856 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7f57a0081820 con 0x7f57b0155970 2024-08-03T20:53:01.453 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:01.451+0000 7f57b6000640 1 -- 172.21.15.32:0/4149010856 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7f57b00b1cc0 con 0x7f57b0155970 2024-08-03T20:53:01.453 INFO:tasks.ceph.mon.a.smithi032.stderr:2024-08-03T20:53:01.452+0000 7f4e6d600640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-08-03T20:53:01.456 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:01.454+0000 7f57a7400640 1 -- 172.21.15.32:0/4149010856 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v66) ==== 161+0+0 (secure 0 0 0) 0x7f57a0085d70 con 0x7f57b0155970 2024-08-03T20:53:01.456 INFO:teuthology.orchestra.run.smithi032.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-08-03T20:53:01.459 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:01.457+0000 7f57b6000640 1 -- 172.21.15.32:0/4149010856 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f5790077040 msgr2=0x7f5790079550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:53:01.459 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:01.457+0000 7f57b6000640 1 --2- 172.21.15.32:0/4149010856 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f5790077040 0x7f5790079550 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f57b015b4a0 tx=0x7f579c009040 comp rx=0 tx=0).stop 2024-08-03T20:53:01.459 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:01.457+0000 7f57b6000640 1 -- 172.21.15.32:0/4149010856 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f57b0155970 msgr2=0x7f57b01409e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:53:01.459 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:01.457+0000 7f57b6000640 1 --2- 172.21.15.32:0/4149010856 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f57b0155970 0x7f57b01409e0 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 crypto rx=0x7f57a0034040 tx=0x7f57a00035f0 comp rx=0 tx=0).stop 2024-08-03T20:53:01.459 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:01.457+0000 7f57b6000640 1 -- 172.21.15.32:0/4149010856 shutdown_connections 2024-08-03T20:53:01.459 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:01.457+0000 7f57b6000640 1 --2- 172.21.15.32:0/4149010856 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f5790077040 0x7f5790079550 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:53:01.459 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:01.457+0000 7f57b6000640 1 --2- 172.21.15.32:0/4149010856 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f57b00cc880 0x7f57b015de70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:53:01.459 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:01.457+0000 7f57b6000640 1 --2- 172.21.15.32:0/4149010856 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f57b015ee80 0x7f57b00cc340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:53:01.459 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:01.457+0000 7f57b6000640 1 --2- 172.21.15.32:0/4149010856 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f57b0155970 0x7f57b01409e0 unknown :-1 s=CLOSED pgs=701 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:53:01.459 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:01.457+0000 7f57b6000640 1 -- 172.21.15.32:0/4149010856 >> 172.21.15.32:0/4149010856 conn(0x7f57b00b34b0 msgr2=0x7f57b00b24c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:53:01.460 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:01.458+0000 7f57b6000640 1 -- 172.21.15.32:0/4149010856 shutdown_connections 2024-08-03T20:53:01.460 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:01.458+0000 7f57b6000640 1 -- 172.21.15.32:0/4149010856 wait complete. 2024-08-03T20:53:01.471 INFO:tasks.ceph.mds.d.smithi118.stderr: -4326> 2024-08-03T20:51:20.137+0000 7f66d1a00640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-08-03T20:53:01.482 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-08-03T20:53:01.595 INFO:tasks.ceph.mds.d.smithi118.stdout:starting mds.d at 2024-08-03T20:53:01.713 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:01.711+0000 7fd6aec00640 1 -- 172.21.15.32:0/2606523519 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd6a815fe20 msgr2=0x7fd6a8160230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:53:01.713 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:01.711+0000 7fd6aec00640 1 --2- 172.21.15.32:0/2606523519 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd6a815fe20 0x7fd6a8160230 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0x7fd694005fd0 tx=0x7fd69402a5d0 comp rx=0 tx=0).stop 2024-08-03T20:53:01.713 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:01.711+0000 7fd6aec00640 1 -- 172.21.15.32:0/2606523519 shutdown_connections 2024-08-03T20:53:01.713 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:01.711+0000 7fd6aec00640 1 --2- 172.21.15.32:0/2606523519 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd6a815fe20 0x7fd6a8160230 unknown :-1 s=CLOSED pgs=704 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:53:01.714 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:01.711+0000 7fd6aec00640 1 --2- 172.21.15.32:0/2606523519 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fd6a806fc00 0x7fd6a80cbef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:53:01.714 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:01.711+0000 7fd6aec00640 1 -- 172.21.15.32:0/2606523519 >> 172.21.15.32:0/2606523519 conn(0x7fd6a80cfd40 msgr2=0x7fd6a80d0150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:53:01.714 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:01.711+0000 7fd6aec00640 1 -- 172.21.15.32:0/2606523519 shutdown_connections 2024-08-03T20:53:01.714 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:01.711+0000 7fd6aec00640 1 -- 172.21.15.32:0/2606523519 wait complete. 2024-08-03T20:53:01.715 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:01.714+0000 7fd6aec00640 1 Processor -- start 2024-08-03T20:53:01.715 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:01.714+0000 7fd6aec00640 1 -- start start 2024-08-03T20:53:01.718 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:01.716+0000 7fd6aec00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd6a806fc00 0x7fd6a80b5650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:53:01.718 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:01.716+0000 7fd6aec00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fd6a8152b30 0x7fd6a81442e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:53:01.718 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:01.716+0000 7fd6aec00640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fd6a815fe20 0x7fd6a8155c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:53:01.718 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:01.716+0000 7fd6aec00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fd6a80cb6c0 con 0x7fd6a806fc00 2024-08-03T20:53:01.718 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:01.716+0000 7fd6aec00640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7fd6a80cb540 con 0x7fd6a8152b30 2024-08-03T20:53:01.718 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:01.716+0000 7fd6aec00640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7fd6a80cc430 con 0x7fd6a815fe20 2024-08-03T20:53:01.719 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:01.716+0000 7fd6ad800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd6a806fc00 0x7fd6a80b5650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:53:01.719 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:01.716+0000 7fd6ad800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd6a806fc00 0x7fd6a80b5650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:40040/0 (socket says 172.21.15.32:40040) 2024-08-03T20:53:01.719 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:01.716+0000 7fd6ad800640 1 -- 172.21.15.32:0/104898094 learned_addr learned my addr 172.21.15.32:0/104898094 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:53:01.719 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:01.716+0000 7fd6ace00640 1 --2- 172.21.15.32:0/104898094 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fd6a8152b30 0x7fd6a81442e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:53:01.719 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:01.716+0000 7fd6ad800640 1 -- 172.21.15.32:0/104898094 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fd6a815fe20 msgr2=0x7fd6a8155c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:53:01.719 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:01.716+0000 7fd6ad800640 1 --2- 172.21.15.32:0/104898094 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fd6a815fe20 0x7fd6a8155c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:53:01.719 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:01.716+0000 7fd6ad800640 1 -- 172.21.15.32:0/104898094 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fd6a8152b30 msgr2=0x7fd6a81442e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:53:01.719 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:01.716+0000 7fd6ad800640 1 --2- 172.21.15.32:0/104898094 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fd6a8152b30 0x7fd6a81442e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:53:01.719 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:01.716+0000 7fd6ad800640 1 -- 172.21.15.32:0/104898094 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd6a81f76a0 con 0x7fd6a806fc00 2024-08-03T20:53:01.719 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:01.716+0000 7fd6ace00640 1 --2- 172.21.15.32:0/104898094 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fd6a8152b30 0x7fd6a81442e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-03T20:53:01.719 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:01.716+0000 7fd6ad800640 1 --2- 172.21.15.32:0/104898094 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd6a806fc00 0x7fd6a80b5650 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7fd69c00d770 tx=0x7fd69c00dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:53:01.720 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:01.716+0000 7fd6a5600640 1 -- 172.21.15.32:0/104898094 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd69c017070 con 0x7fd6a806fc00 2024-08-03T20:53:01.720 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:01.716+0000 7fd6a5600640 1 -- 172.21.15.32:0/104898094 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fd69c009d20 con 0x7fd6a806fc00 2024-08-03T20:53:01.720 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:01.716+0000 7fd6a5600640 1 -- 172.21.15.32:0/104898094 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd69c0106c0 con 0x7fd6a806fc00 2024-08-03T20:53:01.720 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:01.716+0000 7fd6aec00640 1 -- 172.21.15.32:0/104898094 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd6a8144820 con 0x7fd6a806fc00 2024-08-03T20:53:01.720 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:01.717+0000 7fd6aec00640 1 -- 172.21.15.32:0/104898094 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd6a81f70e0 con 0x7fd6a806fc00 2024-08-03T20:53:01.721 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:01.717+0000 7fd6aec00640 1 -- 172.21.15.32:0/104898094 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd6a8160bf0 con 0x7fd6a806fc00 2024-08-03T20:53:01.721 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:01.719+0000 7fd6a5600640 1 -- 172.21.15.32:0/104898094 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7fd69c01a070 con 0x7fd6a806fc00 2024-08-03T20:53:01.721 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:01.721+0000 7fd6a5600640 1 --2- 172.21.15.32:0/104898094 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fd688077010 0x7fd688079520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:53:01.722 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:01.721+0000 7fd6a5600640 1 -- 172.21.15.32:0/104898094 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(60..60 src has 1..60) ==== 5967+0+0 (secure 0 0 0) 0x7fd69c098b10 con 0x7fd6a806fc00 2024-08-03T20:53:01.722 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:01.721+0000 7fd6ace00640 1 --2- 172.21.15.32:0/104898094 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fd688077010 0x7fd688079520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:53:01.722 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:01.721+0000 7fd6ace00640 1 --2- 172.21.15.32:0/104898094 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fd688077010 0x7fd688079520 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7fd694033000 tx=0x7fd694004730 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:53:01.725 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:01.724+0000 7fd6a5600640 1 -- 172.21.15.32:0/104898094 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7fd6a8144820 con 0x7fd6a806fc00 2024-08-03T20:53:01.975 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:01.972+0000 7fd6aec00640 1 -- 172.21.15.32:0/104898094 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) -- 0x7fd6a8153070 con 0x7fd6a806fc00 2024-08-03T20:53:02.475 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:02.474+0000 7fd6a5600640 1 -- 172.21.15.32:0/104898094 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v67) ==== 104+0+0 (secure 0 0 0) 0x7fd6a81f76a0 con 0x7fd6a806fc00 2024-08-03T20:53:02.479 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:02.478+0000 7fd6aec00640 1 -- 172.21.15.32:0/104898094 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fd688077010 msgr2=0x7fd688079520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:53:02.479 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:02.478+0000 7fd6aec00640 1 --2- 172.21.15.32:0/104898094 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fd688077010 0x7fd688079520 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7fd694033000 tx=0x7fd694004730 comp rx=0 tx=0).stop 2024-08-03T20:53:02.480 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:02.478+0000 7fd6aec00640 1 -- 172.21.15.32:0/104898094 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd6a806fc00 msgr2=0x7fd6a80b5650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:53:02.480 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:02.478+0000 7fd6aec00640 1 --2- 172.21.15.32:0/104898094 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd6a806fc00 0x7fd6a80b5650 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7fd69c00d770 tx=0x7fd69c00dc30 comp rx=0 tx=0).stop 2024-08-03T20:53:02.480 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:02.479+0000 7fd6aec00640 1 -- 172.21.15.32:0/104898094 shutdown_connections 2024-08-03T20:53:02.480 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:02.479+0000 7fd6aec00640 1 --2- 172.21.15.32:0/104898094 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fd688077010 0x7fd688079520 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:53:02.480 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:02.479+0000 7fd6aec00640 1 --2- 172.21.15.32:0/104898094 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fd6a8152b30 0x7fd6a81442e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:53:02.480 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:02.479+0000 7fd6aec00640 1 --2- 172.21.15.32:0/104898094 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fd6a815fe20 0x7fd6a8155c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:53:02.480 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:02.479+0000 7fd6aec00640 1 --2- 172.21.15.32:0/104898094 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd6a806fc00 0x7fd6a80b5650 unknown :-1 s=CLOSED pgs=705 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:53:02.480 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:02.479+0000 7fd6aec00640 1 -- 172.21.15.32:0/104898094 >> 172.21.15.32:0/104898094 conn(0x7fd6a80cfd40 msgr2=0x7fd6a80c9ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:53:02.480 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:02.479+0000 7fd6aec00640 1 -- 172.21.15.32:0/104898094 shutdown_connections 2024-08-03T20:53:02.480 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:02.479+0000 7fd6aec00640 1 -- 172.21.15.32:0/104898094 wait complete. 2024-08-03T20:53:02.503 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-08-03T20:53:02.736 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:02.735+0000 7f01e0a00640 1 -- 172.21.15.32:0/2878039238 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f01dc0b3e80 msgr2=0x7f01dc0b4290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:53:02.737 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:02.735+0000 7f01e0a00640 1 --2- 172.21.15.32:0/2878039238 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f01dc0b3e80 0x7f01dc0b4290 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x7f01d4005fd0 tx=0x7f01d402aa00 comp rx=0 tx=0).stop 2024-08-03T20:53:02.737 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:02.735+0000 7f01e0a00640 1 -- 172.21.15.32:0/2878039238 shutdown_connections 2024-08-03T20:53:02.737 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:02.735+0000 7f01e0a00640 1 --2- 172.21.15.32:0/2878039238 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f01dc06e950 0x7f01dc06ed60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:53:02.737 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:02.735+0000 7f01e0a00640 1 --2- 172.21.15.32:0/2878039238 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f01dc0b3e80 0x7f01dc0b4290 unknown :-1 s=CLOSED pgs=706 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:53:02.737 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:02.735+0000 7f01e0a00640 1 -- 172.21.15.32:0/2878039238 >> 172.21.15.32:0/2878039238 conn(0x7f01dc0b1bd0 msgr2=0x7f01dc0b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:53:02.737 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:02.736+0000 7f01e0a00640 1 -- 172.21.15.32:0/2878039238 shutdown_connections 2024-08-03T20:53:02.737 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:02.736+0000 7f01e0a00640 1 -- 172.21.15.32:0/2878039238 wait complete. 2024-08-03T20:53:02.738 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:02.738+0000 7f01e0a00640 1 Processor -- start 2024-08-03T20:53:02.738 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:02.738+0000 7f01e0a00640 1 -- start start 2024-08-03T20:53:02.741 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:02.739+0000 7f01e0a00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f01dc06e950 0x7f01dc158550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:53:02.741 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:02.739+0000 7f01e0a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f01dc0b3e80 0x7f01dc15ad10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:53:02.741 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:02.739+0000 7f01e0a00640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f01dc15d2b0 0x7f01dc15b250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:53:02.742 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:02.739+0000 7f01e0a00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f01dc0cd010 con 0x7f01dc0b3e80 2024-08-03T20:53:02.742 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:02.739+0000 7f01e0a00640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f01dc0cce90 con 0x7f01dc06e950 2024-08-03T20:53:02.742 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:02.739+0000 7f01e0a00640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7f01dc0cd190 con 0x7f01dc15d2b0 2024-08-03T20:53:02.742 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:02.740+0000 7f01d9400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f01dc0b3e80 0x7f01dc15ad10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:53:02.742 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:02.740+0000 7f01d9400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f01dc0b3e80 0x7f01dc15ad10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:40068/0 (socket says 172.21.15.32:40068) 2024-08-03T20:53:02.742 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:02.740+0000 7f01d9400640 1 -- 172.21.15.32:0/3022109446 learned_addr learned my addr 172.21.15.32:0/3022109446 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:53:02.742 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:02.740+0000 7f01da800640 1 --2- 172.21.15.32:0/3022109446 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f01dc15d2b0 0x7f01dc15b250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:53:02.742 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:02.740+0000 7f01d9e00640 1 --2- 172.21.15.32:0/3022109446 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f01dc06e950 0x7f01dc158550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:53:02.742 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:02.740+0000 7f01d9400640 1 -- 172.21.15.32:0/3022109446 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f01dc15d2b0 msgr2=0x7f01dc15b250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:53:02.742 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:02.740+0000 7f01d9400640 1 --2- 172.21.15.32:0/3022109446 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f01dc15d2b0 0x7f01dc15b250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:53:02.743 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:02.740+0000 7f01d9400640 1 -- 172.21.15.32:0/3022109446 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f01dc06e950 msgr2=0x7f01dc158550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:53:02.743 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:02.740+0000 7f01d9400640 1 --2- 172.21.15.32:0/3022109446 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f01dc06e950 0x7f01dc158550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:53:02.743 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:02.740+0000 7f01d9400640 1 -- 172.21.15.32:0/3022109446 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f01dc158a90 con 0x7f01dc0b3e80 2024-08-03T20:53:02.743 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:02.740+0000 7f01d9e00640 1 --2- 172.21.15.32:0/3022109446 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f01dc06e950 0x7f01dc158550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-03T20:53:02.743 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:02.740+0000 7f01d9400640 1 --2- 172.21.15.32:0/3022109446 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f01dc0b3e80 0x7f01dc15ad10 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7f01c800d770 tx=0x7f01c800dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:53:02.743 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:02.741+0000 7f01d2000640 1 -- 172.21.15.32:0/3022109446 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01c8017070 con 0x7f01dc0b3e80 2024-08-03T20:53:02.743 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:02.741+0000 7f01d2000640 1 -- 172.21.15.32:0/3022109446 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f01c8009d20 con 0x7f01dc0b3e80 2024-08-03T20:53:02.743 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:02.741+0000 7f01d2000640 1 -- 172.21.15.32:0/3022109446 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01c80106c0 con 0x7f01dc0b3e80 2024-08-03T20:53:02.743 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:02.741+0000 7f01e0a00640 1 -- 172.21.15.32:0/3022109446 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f01dc1549b0 con 0x7f01dc0b3e80 2024-08-03T20:53:02.743 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:02.741+0000 7f01e0a00640 1 -- 172.21.15.32:0/3022109446 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f01dc154f70 con 0x7f01dc0b3e80 2024-08-03T20:53:02.744 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:02.742+0000 7f01e0a00640 1 -- 172.21.15.32:0/3022109446 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f01dc145e30 con 0x7f01dc0b3e80 2024-08-03T20:53:02.744 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:02.743+0000 7f01d2000640 1 -- 172.21.15.32:0/3022109446 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f01c801a070 con 0x7f01dc0b3e80 2024-08-03T20:53:02.744 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:02.744+0000 7f01d2000640 1 --2- 172.21.15.32:0/3022109446 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f01bc077010 0x7f01bc079520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:53:02.745 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:02.744+0000 7f01d2000640 1 -- 172.21.15.32:0/3022109446 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(60..60 src has 1..60) ==== 5967+0+0 (secure 0 0 0) 0x7f01c8098af0 con 0x7f01dc0b3e80 2024-08-03T20:53:02.745 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:02.745+0000 7f01d9e00640 1 --2- 172.21.15.32:0/3022109446 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f01bc077010 0x7f01bc079520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:53:02.746 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:02.745+0000 7f01d9e00640 1 --2- 172.21.15.32:0/3022109446 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f01bc077010 0x7f01bc079520 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f01cc006fd0 tx=0x7f01cc008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:53:02.749 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:02.748+0000 7f01d2000640 1 -- 172.21.15.32:0/3022109446 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7f01c8061500 con 0x7f01dc0b3e80 2024-08-03T20:53:02.952 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:02.952+0000 7f01e0a00640 1 -- 172.21.15.32:0/3022109446 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) -- 0x7f01dc06ed60 con 0x7f01dc0b3e80 2024-08-03T20:53:03.477 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:03.476+0000 7f01d2000640 1 -- 172.21.15.32:0/3022109446 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v61) ==== 181+0+0 (secure 0 0 0) 0x7f01c8065a50 con 0x7f01dc0b3e80 2024-08-03T20:53:03.477 INFO:teuthology.orchestra.run.smithi032.stderr:pool 'cephfs_metadata' removed 2024-08-03T20:53:03.481 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:03.480+0000 7f01e0a00640 1 -- 172.21.15.32:0/3022109446 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f01bc077010 msgr2=0x7f01bc079520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:53:03.481 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:03.480+0000 7f01e0a00640 1 --2- 172.21.15.32:0/3022109446 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f01bc077010 0x7f01bc079520 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f01cc006fd0 tx=0x7f01cc008040 comp rx=0 tx=0).stop 2024-08-03T20:53:03.481 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:03.480+0000 7f01e0a00640 1 -- 172.21.15.32:0/3022109446 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f01dc0b3e80 msgr2=0x7f01dc15ad10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:53:03.482 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:03.480+0000 7f01e0a00640 1 --2- 172.21.15.32:0/3022109446 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f01dc0b3e80 0x7f01dc15ad10 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7f01c800d770 tx=0x7f01c800dc30 comp rx=0 tx=0).stop 2024-08-03T20:53:03.482 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:03.481+0000 7f01e0a00640 1 -- 172.21.15.32:0/3022109446 shutdown_connections 2024-08-03T20:53:03.482 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:03.481+0000 7f01e0a00640 1 --2- 172.21.15.32:0/3022109446 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f01bc077010 0x7f01bc079520 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:53:03.482 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:03.481+0000 7f01e0a00640 1 --2- 172.21.15.32:0/3022109446 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f01dc15d2b0 0x7f01dc15b250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:53:03.483 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:03.481+0000 7f01e0a00640 1 --2- 172.21.15.32:0/3022109446 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f01dc0b3e80 0x7f01dc15ad10 unknown :-1 s=CLOSED pgs=707 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:53:03.483 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:03.481+0000 7f01e0a00640 1 --2- 172.21.15.32:0/3022109446 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f01dc06e950 0x7f01dc158550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:53:03.483 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:03.481+0000 7f01e0a00640 1 -- 172.21.15.32:0/3022109446 >> 172.21.15.32:0/3022109446 conn(0x7f01dc0b1bd0 msgr2=0x7f01dc0b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:53:03.483 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:03.481+0000 7f01e0a00640 1 -- 172.21.15.32:0/3022109446 shutdown_connections 2024-08-03T20:53:03.483 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:03.481+0000 7f01e0a00640 1 -- 172.21.15.32:0/3022109446 wait complete. 2024-08-03T20:53:03.511 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-08-03T20:53:03.746 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:03.745+0000 7fa6ae200640 1 -- 172.21.15.32:0/1375733722 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa6a8160520 msgr2=0x7fa6a81450f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:53:03.746 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:03.745+0000 7fa6ae200640 1 --2- 172.21.15.32:0/1375733722 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa6a8160520 0x7fa6a81450f0 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x7fa6a815da00 tx=0x7fa6a002a790 comp rx=0 tx=0).stop 2024-08-03T20:53:03.746 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:03.745+0000 7fa6ae200640 1 -- 172.21.15.32:0/1375733722 shutdown_connections 2024-08-03T20:53:03.746 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:03.745+0000 7fa6ae200640 1 --2- 172.21.15.32:0/1375733722 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa6a8160520 0x7fa6a81450f0 unknown :-1 s=CLOSED pgs=709 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:53:03.746 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:03.745+0000 7fa6ae200640 1 --2- 172.21.15.32:0/1375733722 >> v2:172.21.15.118:3300/0 conn(0x7fa6a80cc850 0x7fa6a815ffe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:53:03.746 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:03.745+0000 7fa6ae200640 1 -- 172.21.15.32:0/1375733722 >> 172.21.15.32:0/1375733722 conn(0x7fa6a80b1bd0 msgr2=0x7fa6a80b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:53:03.747 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:03.745+0000 7fa6ae200640 1 -- 172.21.15.32:0/1375733722 shutdown_connections 2024-08-03T20:53:03.747 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:03.745+0000 7fa6ae200640 1 -- 172.21.15.32:0/1375733722 wait complete. 2024-08-03T20:53:03.748 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:03.747+0000 7fa6ae200640 1 Processor -- start 2024-08-03T20:53:03.748 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:03.747+0000 7fa6ae200640 1 -- start start 2024-08-03T20:53:03.750 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:03.749+0000 7fa6ae200640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa6a80cc850 0x7fa6a815f500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:53:03.750 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:03.749+0000 7fa6ae200640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fa6a80cf8d0 0x7fa6a81336f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:53:03.750 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:03.749+0000 7fa6ae200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa6a8160520 0x7fa6a8144080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:53:03.750 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:03.749+0000 7fa6ae200640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fa6a8145a70 con 0x7fa6a8160520 2024-08-03T20:53:03.751 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:03.749+0000 7fa6ae200640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7fa6a81458f0 con 0x7fa6a80cc850 2024-08-03T20:53:03.751 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:03.749+0000 7fa6ae200640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7fa6a8145bf0 con 0x7fa6a80cf8d0 2024-08-03T20:53:03.751 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:03.749+0000 7fa6a7e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa6a8160520 0x7fa6a8144080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:53:03.751 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:03.749+0000 7fa6a7e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa6a8160520 0x7fa6a8144080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:40090/0 (socket says 172.21.15.32:40090) 2024-08-03T20:53:03.751 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:03.749+0000 7fa6a7e00640 1 -- 172.21.15.32:0/337216859 learned_addr learned my addr 172.21.15.32:0/337216859 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:53:03.751 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:03.749+0000 7fa6a7400640 1 --2- 172.21.15.32:0/337216859 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa6a80cc850 0x7fa6a815f500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:53:03.751 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:03.749+0000 7fa69fe00640 1 --2- 172.21.15.32:0/337216859 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fa6a80cf8d0 0x7fa6a81336f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:53:03.751 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:03.750+0000 7fa6a7e00640 1 -- 172.21.15.32:0/337216859 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fa6a80cf8d0 msgr2=0x7fa6a81336f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:53:03.751 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:03.750+0000 7fa6a7e00640 1 --2- 172.21.15.32:0/337216859 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fa6a80cf8d0 0x7fa6a81336f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:53:03.751 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:03.750+0000 7fa6a7e00640 1 -- 172.21.15.32:0/337216859 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa6a80cc850 msgr2=0x7fa6a815f500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:53:03.751 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:03.750+0000 7fa6a7e00640 1 --2- 172.21.15.32:0/337216859 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa6a80cc850 0x7fa6a815f500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:53:03.752 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:03.750+0000 7fa6a7e00640 1 -- 172.21.15.32:0/337216859 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa6a81e6110 con 0x7fa6a8160520 2024-08-03T20:53:03.752 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:03.750+0000 7fa6a7e00640 1 --2- 172.21.15.32:0/337216859 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa6a8160520 0x7fa6a8144080 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7fa6a0009ca0 tx=0x7fa6a0032e90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:53:03.752 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:03.750+0000 7fa69f400640 1 -- 172.21.15.32:0/337216859 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6a0041020 con 0x7fa6a8160520 2024-08-03T20:53:03.753 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:03.751+0000 7fa69f400640 1 -- 172.21.15.32:0/337216859 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fa6a0004080 con 0x7fa6a8160520 2024-08-03T20:53:03.753 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:03.751+0000 7fa6ae200640 1 -- 172.21.15.32:0/337216859 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa6a81e63f0 con 0x7fa6a8160520 2024-08-03T20:53:03.753 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:03.751+0000 7fa69f400640 1 -- 172.21.15.32:0/337216859 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6a0036040 con 0x7fa6a8160520 2024-08-03T20:53:03.753 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:03.751+0000 7fa6ae200640 1 -- 172.21.15.32:0/337216859 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa6a815ed00 con 0x7fa6a8160520 2024-08-03T20:53:03.753 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:03.752+0000 7fa6ae200640 1 -- 172.21.15.32:0/337216859 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa6a815ffe0 con 0x7fa6a8160520 2024-08-03T20:53:03.753 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:03.753+0000 7fa69f400640 1 -- 172.21.15.32:0/337216859 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7fa6a0044020 con 0x7fa6a8160520 2024-08-03T20:53:03.754 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:03.754+0000 7fa69f400640 1 --2- 172.21.15.32:0/337216859 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fa684077040 0x7fa684079550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:53:03.754 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:03.754+0000 7fa69f400640 1 -- 172.21.15.32:0/337216859 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(61..61 src has 1..61) ==== 5529+0+0 (secure 0 0 0) 0x7fa6a00b9890 con 0x7fa6a8160520 2024-08-03T20:53:03.755 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:03.754+0000 7fa6a7400640 1 --2- 172.21.15.32:0/337216859 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fa684077040 0x7fa684079550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:53:03.755 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:03.755+0000 7fa6a7400640 1 --2- 172.21.15.32:0/337216859 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fa684077040 0x7fa684079550 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7fa698000f60 tx=0x7fa698009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:53:03.759 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:03.758+0000 7fa69f400640 1 -- 172.21.15.32:0/337216859 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7fa6a004b030 con 0x7fa6a8160520 2024-08-03T20:53:03.973 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:03.973+0000 7fa6ae200640 1 -- 172.21.15.32:0/337216859 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) -- 0x7fa6a806f0b0 con 0x7fa6a8160520 2024-08-03T20:53:04.489 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:04.489+0000 7fa69f400640 1 -- 172.21.15.32:0/337216859 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v62) ==== 169+0+0 (secure 0 0 0) 0x7fa6a00823d0 con 0x7fa6a8160520 2024-08-03T20:53:04.490 INFO:teuthology.orchestra.run.smithi032.stderr:pool 'cephfs_data' removed 2024-08-03T20:53:04.492 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:04.492+0000 7fa6ae200640 1 -- 172.21.15.32:0/337216859 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fa684077040 msgr2=0x7fa684079550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:53:04.493 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:04.492+0000 7fa6ae200640 1 --2- 172.21.15.32:0/337216859 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fa684077040 0x7fa684079550 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7fa698000f60 tx=0x7fa698009040 comp rx=0 tx=0).stop 2024-08-03T20:53:04.493 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:04.492+0000 7fa6ae200640 1 -- 172.21.15.32:0/337216859 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa6a8160520 msgr2=0x7fa6a8144080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:53:04.493 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:04.492+0000 7fa6ae200640 1 --2- 172.21.15.32:0/337216859 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa6a8160520 0x7fa6a8144080 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7fa6a0009ca0 tx=0x7fa6a0032e90 comp rx=0 tx=0).stop 2024-08-03T20:53:04.493 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:04.493+0000 7fa6ae200640 1 -- 172.21.15.32:0/337216859 shutdown_connections 2024-08-03T20:53:04.494 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:04.493+0000 7fa6ae200640 1 --2- 172.21.15.32:0/337216859 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fa684077040 0x7fa684079550 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:53:04.494 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:04.493+0000 7fa6ae200640 1 --2- 172.21.15.32:0/337216859 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa6a8160520 0x7fa6a8144080 unknown :-1 s=CLOSED pgs=710 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:53:04.494 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:04.493+0000 7fa6ae200640 1 --2- 172.21.15.32:0/337216859 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fa6a80cf8d0 0x7fa6a81336f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:53:04.494 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:04.493+0000 7fa6ae200640 1 --2- 172.21.15.32:0/337216859 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa6a80cc850 0x7fa6a815f500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:53:04.494 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:04.493+0000 7fa6ae200640 1 -- 172.21.15.32:0/337216859 >> 172.21.15.32:0/337216859 conn(0x7fa6a80b1bd0 msgr2=0x7fa6a80b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:53:04.495 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:04.494+0000 7fa6ae200640 1 -- 172.21.15.32:0/337216859 shutdown_connections 2024-08-03T20:53:04.495 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:04.494+0000 7fa6ae200640 1 -- 172.21.15.32:0/337216859 wait complete. 2024-08-03T20:53:04.522 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config reset 27 2024-08-03T20:53:04.753 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:04.752+0000 7fdd71a00640 1 -- 172.21.15.32:0/876660247 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdd6c073e60 msgr2=0x7fdd6c074270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:53:04.753 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:04.752+0000 7fdd71a00640 1 --2- 172.21.15.32:0/876660247 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdd6c073e60 0x7fdd6c074270 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 crypto rx=0x7fdd6c071700 tx=0x7fdd5402a4b0 comp rx=0 tx=0).stop 2024-08-03T20:53:04.753 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:04.752+0000 7fdd71a00640 1 -- 172.21.15.32:0/876660247 shutdown_connections 2024-08-03T20:53:04.753 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:04.752+0000 7fdd71a00640 1 --2- 172.21.15.32:0/876660247 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdd6c073e60 0x7fdd6c074270 unknown :-1 s=CLOSED pgs=712 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:53:04.753 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:04.752+0000 7fdd71a00640 1 --2- 172.21.15.32:0/876660247 >> v2:172.21.15.118:3300/0 conn(0x7fdd6c06e930 0x7fdd6c144f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:53:04.753 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:04.752+0000 7fdd71a00640 1 -- 172.21.15.32:0/876660247 >> 172.21.15.32:0/876660247 conn(0x7fdd6c0b1bd0 msgr2=0x7fdd6c0b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:53:04.754 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:04.752+0000 7fdd71a00640 1 -- 172.21.15.32:0/876660247 shutdown_connections 2024-08-03T20:53:04.754 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:04.752+0000 7fdd71a00640 1 -- 172.21.15.32:0/876660247 wait complete. 2024-08-03T20:53:04.755 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:04.755+0000 7fdd71a00640 1 Processor -- start 2024-08-03T20:53:04.755 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:04.755+0000 7fdd71a00640 1 -- start start 2024-08-03T20:53:04.757 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:04.756+0000 7fdd71a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdd6c06e930 0x7fdd6c0cbbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:53:04.757 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:04.756+0000 7fdd71a00640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fdd6c073e60 0x7fdd6c14c410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:53:04.757 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:04.756+0000 7fdd71a00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fdd6c0747b0 0x7fdd6c14c950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:53:04.758 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:04.756+0000 7fdd71a00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fdd6c0cf9c0 con 0x7fdd6c06e930 2024-08-03T20:53:04.758 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:04.756+0000 7fdd71a00640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7fdd6c0cf840 con 0x7fdd6c0747b0 2024-08-03T20:53:04.758 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:04.756+0000 7fdd71a00640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7fdd6c0ccff0 con 0x7fdd6c073e60 2024-08-03T20:53:04.758 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:04.757+0000 7fdd6aa00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdd6c06e930 0x7fdd6c0cbbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:53:04.758 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:04.757+0000 7fdd6aa00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdd6c06e930 0x7fdd6c0cbbe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:40112/0 (socket says 172.21.15.32:40112) 2024-08-03T20:53:04.758 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:04.757+0000 7fdd6aa00640 1 -- 172.21.15.32:0/1362378862 learned_addr learned my addr 172.21.15.32:0/1362378862 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:53:04.758 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:04.757+0000 7fdd6b400640 1 --2- 172.21.15.32:0/1362378862 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fdd6c0747b0 0x7fdd6c14c950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:53:04.758 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:04.757+0000 7fdd63e00640 1 --2- 172.21.15.32:0/1362378862 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fdd6c073e60 0x7fdd6c14c410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:53:04.758 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:04.757+0000 7fdd6aa00640 1 -- 172.21.15.32:0/1362378862 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fdd6c073e60 msgr2=0x7fdd6c14c410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:53:04.758 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:04.757+0000 7fdd6aa00640 1 --2- 172.21.15.32:0/1362378862 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fdd6c073e60 0x7fdd6c14c410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:53:04.758 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:04.757+0000 7fdd6aa00640 1 -- 172.21.15.32:0/1362378862 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fdd6c0747b0 msgr2=0x7fdd6c14c950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:53:04.758 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:04.757+0000 7fdd6aa00640 1 --2- 172.21.15.32:0/1362378862 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fdd6c0747b0 0x7fdd6c14c950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:53:04.758 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:04.757+0000 7fdd6aa00640 1 -- 172.21.15.32:0/1362378862 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdd6c0cab40 con 0x7fdd6c06e930 2024-08-03T20:53:04.759 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:04.757+0000 7fdd6aa00640 1 --2- 172.21.15.32:0/1362378862 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdd6c06e930 0x7fdd6c0cbbe0 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x7fdd5c00d770 tx=0x7fdd5c00dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:53:04.759 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:04.758+0000 7fdd62a00640 1 -- 172.21.15.32:0/1362378862 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd5c017070 con 0x7fdd6c06e930 2024-08-03T20:53:04.760 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:04.758+0000 7fdd62a00640 1 -- 172.21.15.32:0/1362378862 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fdd5c009d20 con 0x7fdd6c06e930 2024-08-03T20:53:04.760 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:04.758+0000 7fdd62a00640 1 -- 172.21.15.32:0/1362378862 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd5c0106c0 con 0x7fdd6c06e930 2024-08-03T20:53:04.760 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:04.758+0000 7fdd71a00640 1 -- 172.21.15.32:0/1362378862 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdd6c0cae20 con 0x7fdd6c06e930 2024-08-03T20:53:04.760 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:04.758+0000 7fdd71a00640 1 -- 172.21.15.32:0/1362378862 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdd6c0cb3e0 con 0x7fdd6c06e930 2024-08-03T20:53:04.760 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:04.759+0000 7fdd71a00640 1 -- 172.21.15.32:0/1362378862 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdd6c0cd580 con 0x7fdd6c06e930 2024-08-03T20:53:04.760 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:04.760+0000 7fdd62a00640 1 -- 172.21.15.32:0/1362378862 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7fdd5c01a070 con 0x7fdd6c06e930 2024-08-03T20:53:04.762 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:04.762+0000 7fdd62a00640 1 --2- 172.21.15.32:0/1362378862 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fdd48077010 0x7fdd48079520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:53:04.762 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:04.762+0000 7fdd62a00640 1 -- 172.21.15.32:0/1362378862 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(62..62 src has 1..62) ==== 5115+0+0 (secure 0 0 0) 0x7fdd5c0987c0 con 0x7fdd6c06e930 2024-08-03T20:53:04.762 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:04.762+0000 7fdd63e00640 1 --2- 172.21.15.32:0/1362378862 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fdd48077010 0x7fdd48079520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:53:04.763 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:04.763+0000 7fdd63e00640 1 --2- 172.21.15.32:0/1362378862 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fdd48077010 0x7fdd48079520 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7fdd54034000 tx=0x7fdd54004730 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:53:04.766 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:04.766+0000 7fdd62a00640 1 -- 172.21.15.32:0/1362378862 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7fdd5c061c00 con 0x7fdd6c06e930 2024-08-03T20:53:04.970 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:04.969+0000 7fdd71a00640 1 -- 172.21.15.32:0/1362378862 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "config reset", "num": 27} v 0) -- 0x7fdd6c06fc40 con 0x7fdd6c06e930 2024-08-03T20:53:04.974 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:04.974+0000 7fdd62a00640 1 -- 172.21.15.32:0/1362378862 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "config reset", "num": 27}]=0 v38) ==== 71+0+0 (secure 0 0 0) 0x7fdd5c061500 con 0x7fdd6c06e930 2024-08-03T20:53:04.977 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:04.976+0000 7fdd71a00640 1 -- 172.21.15.32:0/1362378862 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fdd48077010 msgr2=0x7fdd48079520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:53:04.977 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:04.976+0000 7fdd71a00640 1 --2- 172.21.15.32:0/1362378862 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fdd48077010 0x7fdd48079520 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7fdd54034000 tx=0x7fdd54004730 comp rx=0 tx=0).stop 2024-08-03T20:53:04.977 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:04.976+0000 7fdd71a00640 1 -- 172.21.15.32:0/1362378862 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdd6c06e930 msgr2=0x7fdd6c0cbbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:53:04.977 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:04.976+0000 7fdd71a00640 1 --2- 172.21.15.32:0/1362378862 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdd6c06e930 0x7fdd6c0cbbe0 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x7fdd5c00d770 tx=0x7fdd5c00dc30 comp rx=0 tx=0).stop 2024-08-03T20:53:04.977 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:04.976+0000 7fdd71a00640 1 -- 172.21.15.32:0/1362378862 shutdown_connections 2024-08-03T20:53:04.977 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:04.976+0000 7fdd71a00640 1 --2- 172.21.15.32:0/1362378862 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fdd48077010 0x7fdd48079520 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:53:04.977 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:04.976+0000 7fdd71a00640 1 --2- 172.21.15.32:0/1362378862 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fdd6c0747b0 0x7fdd6c14c950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:53:04.978 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:04.976+0000 7fdd71a00640 1 --2- 172.21.15.32:0/1362378862 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fdd6c073e60 0x7fdd6c14c410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:53:04.978 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:04.976+0000 7fdd71a00640 1 --2- 172.21.15.32:0/1362378862 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdd6c06e930 0x7fdd6c0cbbe0 unknown :-1 s=CLOSED pgs=713 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:53:04.978 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:04.976+0000 7fdd71a00640 1 -- 172.21.15.32:0/1362378862 >> 172.21.15.32:0/1362378862 conn(0x7fdd6c0b1bd0 msgr2=0x7fdd6c0b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:53:04.978 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:04.976+0000 7fdd71a00640 1 -- 172.21.15.32:0/1362378862 shutdown_connections 2024-08-03T20:53:04.978 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:04.976+0000 7fdd71a00640 1 -- 172.21.15.32:0/1362378862 wait complete. 2024-08-03T20:53:05.000 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_damage.TestDamage.test_dentry_first_precommit' 2024-08-03T20:53:05.228 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:05.227+0000 7fa216000640 1 -- 172.21.15.32:0/2595153266 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa21015e880 msgr2=0x7fa210073800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:53:05.229 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:05.227+0000 7fa216000640 1 --2- 172.21.15.32:0/2595153266 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa21015e880 0x7fa210073800 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x7fa210072280 tx=0x7fa20802aa30 comp rx=0 tx=0).stop 2024-08-03T20:53:05.229 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:05.228+0000 7fa216000640 1 -- 172.21.15.32:0/2595153266 shutdown_connections 2024-08-03T20:53:05.229 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:05.228+0000 7fa216000640 1 --2- 172.21.15.32:0/2595153266 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa21015e880 0x7fa210073800 unknown :-1 s=CLOSED pgs=715 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:53:05.229 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:05.228+0000 7fa216000640 1 --2- 172.21.15.32:0/2595153266 >> v2:172.21.15.118:3300/0 conn(0x7fa2100b4de0 0x7fa2101515c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:53:05.229 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:05.228+0000 7fa216000640 1 -- 172.21.15.32:0/2595153266 >> 172.21.15.32:0/2595153266 conn(0x7fa2100b1bd0 msgr2=0x7fa2100b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:53:05.229 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:05.228+0000 7fa216000640 1 -- 172.21.15.32:0/2595153266 shutdown_connections 2024-08-03T20:53:05.229 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:05.228+0000 7fa216000640 1 -- 172.21.15.32:0/2595153266 wait complete. 2024-08-03T20:53:05.230 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:05.230+0000 7fa216000640 1 Processor -- start 2024-08-03T20:53:05.230 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:05.230+0000 7fa216000640 1 -- start start 2024-08-03T20:53:05.232 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:05.231+0000 7fa216000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa2100733f0 0x7fa21013da60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:53:05.233 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:05.231+0000 7fa216000640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fa2100b4de0 0x7fa21014d750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:53:05.233 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:05.231+0000 7fa216000640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa210149cb0 0x7fa21014dc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:53:05.233 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:05.231+0000 7fa216000640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fa21006e930 con 0x7fa2100733f0 2024-08-03T20:53:05.233 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:05.231+0000 7fa216000640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7fa21006eab0 con 0x7fa210149cb0 2024-08-03T20:53:05.233 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:05.231+0000 7fa216000640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7fa21006ec30 con 0x7fa2100b4de0 2024-08-03T20:53:05.233 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:05.232+0000 7fa20f400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa2100733f0 0x7fa21013da60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:53:05.233 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:05.232+0000 7fa20f400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa2100733f0 0x7fa21013da60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:40132/0 (socket says 172.21.15.32:40132) 2024-08-03T20:53:05.233 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:05.232+0000 7fa20f400640 1 -- 172.21.15.32:0/2507811725 learned_addr learned my addr 172.21.15.32:0/2507811725 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:53:05.233 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:05.232+0000 7fa20ea00640 1 --2- 172.21.15.32:0/2507811725 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fa2100b4de0 0x7fa21014d750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:53:05.233 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:05.232+0000 7fa20fe00640 1 --2- 172.21.15.32:0/2507811725 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa210149cb0 0x7fa21014dc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:53:05.233 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:05.232+0000 7fa20f400640 1 -- 172.21.15.32:0/2507811725 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fa2100b4de0 msgr2=0x7fa21014d750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:53:05.233 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:05.232+0000 7fa20f400640 1 --2- 172.21.15.32:0/2507811725 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fa2100b4de0 0x7fa21014d750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:53:05.233 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:05.232+0000 7fa20f400640 1 -- 172.21.15.32:0/2507811725 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa210149cb0 msgr2=0x7fa21014dc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:53:05.234 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:05.232+0000 7fa20f400640 1 --2- 172.21.15.32:0/2507811725 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa210149cb0 0x7fa21014dc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:53:05.234 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:05.232+0000 7fa20f400640 1 -- 172.21.15.32:0/2507811725 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa210147cd0 con 0x7fa2100733f0 2024-08-03T20:53:05.234 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:05.232+0000 7fa20fe00640 1 --2- 172.21.15.32:0/2507811725 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa210149cb0 0x7fa21014dc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-03T20:53:05.234 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:05.232+0000 7fa20f400640 1 --2- 172.21.15.32:0/2507811725 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa2100733f0 0x7fa21013da60 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0x7fa20000d740 tx=0x7fa20000dc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:53:05.235 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:05.233+0000 7fa207400640 1 -- 172.21.15.32:0/2507811725 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa200017070 con 0x7fa2100733f0 2024-08-03T20:53:05.235 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:05.233+0000 7fa207400640 1 -- 172.21.15.32:0/2507811725 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fa200009d20 con 0x7fa2100733f0 2024-08-03T20:53:05.235 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:05.233+0000 7fa207400640 1 -- 172.21.15.32:0/2507811725 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2000106c0 con 0x7fa2100733f0 2024-08-03T20:53:05.235 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:05.233+0000 7fa216000640 1 -- 172.21.15.32:0/2507811725 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa210147fb0 con 0x7fa2100733f0 2024-08-03T20:53:05.235 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:05.233+0000 7fa216000640 1 -- 172.21.15.32:0/2507811725 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa2101505f0 con 0x7fa2100733f0 2024-08-03T20:53:05.235 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:05.234+0000 7fa216000640 1 -- 172.21.15.32:0/2507811725 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa2101515c0 con 0x7fa2100733f0 2024-08-03T20:53:05.236 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:05.235+0000 7fa207400640 1 -- 172.21.15.32:0/2507811725 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7fa20001a070 con 0x7fa2100733f0 2024-08-03T20:53:05.236 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:05.236+0000 7fa207400640 1 --2- 172.21.15.32:0/2507811725 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fa1f0077010 0x7fa1f0079520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:53:05.237 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:05.236+0000 7fa207400640 1 -- 172.21.15.32:0/2507811725 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(62..62 src has 1..62) ==== 5115+0+0 (secure 0 0 0) 0x7fa200098840 con 0x7fa2100733f0 2024-08-03T20:53:05.237 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:05.237+0000 7fa20ea00640 1 --2- 172.21.15.32:0/2507811725 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fa1f0077010 0x7fa1f0079520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:53:05.238 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:05.237+0000 7fa20ea00640 1 --2- 172.21.15.32:0/2507811725 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fa1f0077010 0x7fa1f0079520 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7fa2101494e0 tx=0x7fa1fc009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:53:05.241 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:05.241+0000 7fa207400640 1 -- 172.21.15.32:0/2507811725 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7fa200061bf0 con 0x7fa2100733f0 2024-08-03T20:53:05.528 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:05.527+0000 7fa216000640 1 -- 172.21.15.32:0/2507811725 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_damage.TestDamage.test_dentry_first_precommit"]} v 0) -- 0x7fa2101411d0 con 0x7fa2100733f0 2024-08-03T20:53:06.497 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:06.497+0000 7fa207400640 1 -- 172.21.15.32:0/2507811725 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_damage.TestDamage.test_dentry_first_precommit"]}]=0 v269) ==== 142+0+0 (secure 0 0 0) 0x7fa2000614f0 con 0x7fa2100733f0 2024-08-03T20:53:06.502 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:06.501+0000 7fa216000640 1 -- 172.21.15.32:0/2507811725 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fa1f0077010 msgr2=0x7fa1f0079520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:53:06.502 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:06.501+0000 7fa216000640 1 --2- 172.21.15.32:0/2507811725 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fa1f0077010 0x7fa1f0079520 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7fa2101494e0 tx=0x7fa1fc009040 comp rx=0 tx=0).stop 2024-08-03T20:53:06.502 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:06.501+0000 7fa216000640 1 -- 172.21.15.32:0/2507811725 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa2100733f0 msgr2=0x7fa21013da60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:53:06.502 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:06.501+0000 7fa216000640 1 --2- 172.21.15.32:0/2507811725 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa2100733f0 0x7fa21013da60 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0x7fa20000d740 tx=0x7fa20000dc00 comp rx=0 tx=0).stop 2024-08-03T20:53:06.502 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:06.501+0000 7fa216000640 1 -- 172.21.15.32:0/2507811725 shutdown_connections 2024-08-03T20:53:06.502 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:06.501+0000 7fa216000640 1 --2- 172.21.15.32:0/2507811725 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fa1f0077010 0x7fa1f0079520 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:53:06.502 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:06.501+0000 7fa216000640 1 --2- 172.21.15.32:0/2507811725 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa210149cb0 0x7fa21014dc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:53:06.503 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:06.501+0000 7fa216000640 1 --2- 172.21.15.32:0/2507811725 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fa2100b4de0 0x7fa21014d750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:53:06.503 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:06.501+0000 7fa216000640 1 --2- 172.21.15.32:0/2507811725 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa2100733f0 0x7fa21013da60 unknown :-1 s=CLOSED pgs=716 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:53:06.503 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:06.501+0000 7fa216000640 1 -- 172.21.15.32:0/2507811725 >> 172.21.15.32:0/2507811725 conn(0x7fa2100b1bd0 msgr2=0x7fa21015e100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:53:06.503 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:06.502+0000 7fa216000640 1 -- 172.21.15.32:0/2507811725 shutdown_connections 2024-08-03T20:53:06.503 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:06.502+0000 7fa216000640 1 -- 172.21.15.32:0/2507811725 wait complete. 2024-08-03T20:53:06.530 INFO:tasks.cephfs_test_runner:That the MDS won't write a dentry with new damage to CDentry::first ... ok 2024-08-03T20:53:06.531 INFO:tasks.cephfs_test_runner:Starting test: test_dentry_first_preflush (tasks.cephfs.test_damage.TestDamage) That the MDS won't write a dentry with new damage to CDentry::first 2024-08-03T20:53:06.531 INFO:tasks.cephfs_test_runner:test_dentry_first_preflush (tasks.cephfs.test_damage.TestDamage) 2024-08-03T20:53:06.531 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-08-03T20:53:06.767 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:06.766+0000 7fd3a0200640 1 -- 172.21.15.32:0/2819450301 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd3800041c0 msgr2=0x7fd380004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:53:06.767 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:06.766+0000 7fd3a0200640 1 --2- 172.21.15.32:0/2819450301 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd3800041c0 0x7fd380004800 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 crypto rx=0x7fd384009b50 tx=0x7fd380005830 comp rx=0 tx=0).stop 2024-08-03T20:53:06.767 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:06.766+0000 7fd39e200640 1 -- 172.21.15.32:0/2819450301 reap_dead start 2024-08-03T20:53:06.767 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:06.766+0000 7fd3a0200640 1 -- 172.21.15.32:0/2819450301 shutdown_connections 2024-08-03T20:53:06.768 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:06.766+0000 7fd3a0200640 1 -- 172.21.15.32:0/2819450301 >> 172.21.15.32:0/2819450301 conn(0x7fd3980b1bd0 msgr2=0x7fd3980b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:53:06.768 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:06.766+0000 7fd3a0200640 1 -- 172.21.15.32:0/2819450301 shutdown_connections 2024-08-03T20:53:06.768 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:06.766+0000 7fd3a0200640 1 -- 172.21.15.32:0/2819450301 wait complete. 2024-08-03T20:53:06.769 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:06.769+0000 7fd3a0200640 1 Processor -- start 2024-08-03T20:53:06.769 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:06.769+0000 7fd3a0200640 1 -- start start 2024-08-03T20:53:06.771 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:06.770+0000 7fd3a0200640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fd3800041c0 0x7fd398143730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:53:06.771 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:06.770+0000 7fd3a0200640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fd3981442f0 0x7fd398143c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:53:06.772 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:06.770+0000 7fd3a0200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd3981f5190 0x7fd3981f55a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:53:06.772 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:06.770+0000 7fd3a0200640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fd3980cf840 con 0x7fd3981f5190 2024-08-03T20:53:06.772 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:06.770+0000 7fd3a0200640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7fd398145f30 con 0x7fd3981442f0 2024-08-03T20:53:06.772 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:06.770+0000 7fd3a0200640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7fd3980cf9c0 con 0x7fd3800041c0 2024-08-03T20:53:06.772 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:06.770+0000 7fd39e200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd3981f5190 0x7fd3981f55a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:53:06.772 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:06.771+0000 7fd39e200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd3981f5190 0x7fd3981f55a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:40150/0 (socket says 172.21.15.32:40150) 2024-08-03T20:53:06.772 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:06.771+0000 7fd39e200640 1 -- 172.21.15.32:0/1264390637 learned_addr learned my addr 172.21.15.32:0/1264390637 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:53:06.772 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:06.771+0000 7fd39ce00640 1 --2- 172.21.15.32:0/1264390637 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fd3981442f0 0x7fd398143c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:53:06.772 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:06.771+0000 7fd39d800640 1 --2- 172.21.15.32:0/1264390637 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fd3800041c0 0x7fd398143730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:53:06.772 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:06.771+0000 7fd39e200640 1 -- 172.21.15.32:0/1264390637 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fd3800041c0 msgr2=0x7fd398143730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:53:06.772 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:06.771+0000 7fd39e200640 1 --2- 172.21.15.32:0/1264390637 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fd3800041c0 0x7fd398143730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:53:06.773 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:06.771+0000 7fd39e200640 1 -- 172.21.15.32:0/1264390637 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fd3981442f0 msgr2=0x7fd398143c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:53:06.773 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:06.771+0000 7fd39e200640 1 --2- 172.21.15.32:0/1264390637 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fd3981442f0 0x7fd398143c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:53:06.773 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:06.771+0000 7fd39e200640 1 -- 172.21.15.32:0/1264390637 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd39815f8e0 con 0x7fd3981f5190 2024-08-03T20:53:06.773 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:06.771+0000 7fd39d800640 1 --2- 172.21.15.32:0/1264390637 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fd3800041c0 0x7fd398143730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-03T20:53:06.773 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:06.771+0000 7fd39e200640 1 --2- 172.21.15.32:0/1264390637 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd3981f5190 0x7fd3981f55a0 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 crypto rx=0x7fd3980b26f0 tx=0x7fd390006430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:53:06.774 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:06.771+0000 7fd395600640 1 -- 172.21.15.32:0/1264390637 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd390011900 con 0x7fd3981f5190 2024-08-03T20:53:06.774 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:06.771+0000 7fd395600640 1 -- 172.21.15.32:0/1264390637 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fd390018070 con 0x7fd3981f5190 2024-08-03T20:53:06.774 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:06.771+0000 7fd395600640 1 -- 172.21.15.32:0/1264390637 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd390011d40 con 0x7fd3981f5190 2024-08-03T20:53:06.774 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:06.772+0000 7fd3a0200640 1 -- 172.21.15.32:0/1264390637 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd39813ecd0 con 0x7fd3981f5190 2024-08-03T20:53:06.774 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:06.772+0000 7fd3a0200640 1 -- 172.21.15.32:0/1264390637 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd398141730 con 0x7fd3981f5190 2024-08-03T20:53:06.774 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:06.773+0000 7fd3a0200640 1 -- 172.21.15.32:0/1264390637 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd3980747f0 con 0x7fd3981f5190 2024-08-03T20:53:06.775 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:06.774+0000 7fd395600640 1 -- 172.21.15.32:0/1264390637 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7fd39001d460 con 0x7fd3981f5190 2024-08-03T20:53:06.776 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:06.775+0000 7fd395600640 1 --2- 172.21.15.32:0/1264390637 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fd378077010 0x7fd378079520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:53:06.776 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:06.775+0000 7fd395600640 1 -- 172.21.15.32:0/1264390637 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(62..62 src has 1..62) ==== 5115+0+0 (secure 0 0 0) 0x7fd390098d30 con 0x7fd3981f5190 2024-08-03T20:53:06.776 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:06.776+0000 7fd39d800640 1 --2- 172.21.15.32:0/1264390637 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fd378077010 0x7fd378079520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:53:06.777 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:06.776+0000 7fd39d800640 1 --2- 172.21.15.32:0/1264390637 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fd378077010 0x7fd378079520 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7fd38c001050 tx=0x7fd38c001b80 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:53:06.780 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:06.779+0000 7fd395600640 1 -- 172.21.15.32:0/1264390637 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7fd390062200 con 0x7fd3981f5190 2024-08-03T20:53:06.984 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:06.984+0000 7fd3a0200640 1 -- 172.21.15.32:0/1264390637 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fd3980b0960 con 0x7fd3981f5190 2024-08-03T20:53:06.985 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:06.985+0000 7fd395600640 1 -- 172.21.15.32:0/1264390637 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v62) ==== 74+0+13222 (secure 0 0 0) 0x7fd390061b00 con 0x7fd3981f5190 2024-08-03T20:53:06.986 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-03T20:53:06.986 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":62,"fsid":"40ab141a-01d7-49b9-ae3c-2edec9876e95","created":"2024-08-03T20:47:14.957362+0000","modified":"2024-08-03T20:53:04.483796+0000","last_up_change":"2024-08-03T20:47:28.874451+0000","last_in_change":"2024-08-03T20:47:20.079442+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-08-03T20:47:28.261300+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5a83e759-9035-438b-889a-0a6af1f0a0d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6816","nonce":3066156116},{"type":"v1","addr":"172.21.15.32:6817","nonce":3066156116}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6818","nonce":3066156116},{"type":"v1","addr":"172.21.15.32:6819","nonce":3066156116}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6822","nonce":3066156116},{"type":"v1","addr":"172.21.15.32:6823","nonce":3066156116}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6820","nonce":3066156116},{"type":"v1","addr":"172.21.15.32:6821","nonce":3066156116}]},"public_addr":"172.21.15.32:6817/3066156116","cluster_addr":"172.21.15.32:6819/3066156116","heartbeat_back_addr":"172.21.15.32:6823/3066156116","heartbeat_front_addr":"172.21.15.32:6821/3066156116","state":["exists","up"]},{"osd":1,"uuid":"d7c5c210-da4f-49f9-b9e7-1dfb02daa4bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6824","nonce":1551241759},{"type":"v1","addr":"172.21.15.32:6825","nonce":1551241759}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6826","nonce":1551241759},{"type":"v1","addr":"172.21.15.32:6827","nonce":1551241759}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6830","nonce":1551241759},{"type":"v1","addr":"172.21.15.32:6831","nonce":1551241759}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6828","nonce":1551241759},{"type":"v1","addr":"172.21.15.32:6829","nonce":1551241759}]},"public_addr":"172.21.15.32:6825/1551241759","cluster_addr":"172.21.15.32:6827/1551241759","heartbeat_back_addr":"172.21.15.32:6831/1551241759","heartbeat_front_addr":"172.21.15.32:6829/1551241759","state":["exists","up"]},{"osd":2,"uuid":"57f2e6de-a201-45c2-8981-8e56a4875ebd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6808","nonce":3806708944},{"type":"v1","addr":"172.21.15.32:6809","nonce":3806708944}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6810","nonce":3806708944},{"type":"v1","addr":"172.21.15.32:6811","nonce":3806708944}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6814","nonce":3806708944},{"type":"v1","addr":"172.21.15.32:6815","nonce":3806708944}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6812","nonce":3806708944},{"type":"v1","addr":"172.21.15.32:6813","nonce":3806708944}]},"public_addr":"172.21.15.32:6809/3806708944","cluster_addr":"172.21.15.32:6811/3806708944","heartbeat_back_addr":"172.21.15.32:6815/3806708944","heartbeat_front_addr":"172.21.15.32:6813/3806708944","state":["exists","up"]},{"osd":3,"uuid":"6439a280-da45-4148-8130-f16323935ff3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6800","nonce":9088303},{"type":"v1","addr":"172.21.15.32:6801","nonce":9088303}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6802","nonce":9088303},{"type":"v1","addr":"172.21.15.32:6803","nonce":9088303}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6806","nonce":9088303},{"type":"v1","addr":"172.21.15.32:6807","nonce":9088303}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6804","nonce":9088303},{"type":"v1","addr":"172.21.15.32:6805","nonce":9088303}]},"public_addr":"172.21.15.32:6801/9088303","cluster_addr":"172.21.15.32:6803/9088303","heartbeat_back_addr":"172.21.15.32:6807/9088303","heartbeat_front_addr":"172.21.15.32:6805/9088303","state":["exists","up"]},{"osd":4,"uuid":"4f902eff-1e18-418d-82c9-2151c69524fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6802","nonce":864752275},{"type":"v1","addr":"172.21.15.118:6803","nonce":864752275}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6804","nonce":864752275},{"type":"v1","addr":"172.21.15.118:6805","nonce":864752275}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6808","nonce":864752275},{"type":"v1","addr":"172.21.15.118:6809","nonce":864752275}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6806","nonce":864752275},{"type":"v1","addr":"172.21.15.118:6807","nonce":864752275}]},"public_addr":"172.21.15.118:6803/864752275","cluster_addr":"172.21.15.118:6805/864752275","heartbeat_back_addr":"172.21.15.118:6809/864752275","heartbeat_front_addr":"172.21.15.118:6807/864752275","state":["exists","up"]},{"osd":5,"uuid":"bfd9feac-dd06-458c-91b9-af7f175f612c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6810","nonce":2724891770},{"type":"v1","addr":"172.21.15.118:6811","nonce":2724891770}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6812","nonce":2724891770},{"type":"v1","addr":"172.21.15.118:6813","nonce":2724891770}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6816","nonce":2724891770},{"type":"v1","addr":"172.21.15.118:6817","nonce":2724891770}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6814","nonce":2724891770},{"type":"v1","addr":"172.21.15.118:6815","nonce":2724891770}]},"public_addr":"172.21.15.118:6811/2724891770","cluster_addr":"172.21.15.118:6813/2724891770","heartbeat_back_addr":"172.21.15.118:6817/2724891770","heartbeat_front_addr":"172.21.15.118:6815/2724891770","state":["exists","up"]},{"osd":6,"uuid":"5ed5a279-9d8e-416d-bbc7-7edd3e6d5705","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6825","nonce":801947386},{"type":"v1","addr":"172.21.15.118:6827","nonce":801947386}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6828","nonce":801947386},{"type":"v1","addr":"172.21.15.118:6829","nonce":801947386}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6832","nonce":801947386},{"type":"v1","addr":"172.21.15.118:6833","nonce":801947386}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6830","nonce":801947386},{"type":"v1","addr":"172.21.15.118:6831","nonce":801947386}]},"public_addr":"172.21.15.118:6827/801947386","cluster_addr":"172.21.15.118:6829/801947386","heartbeat_back_addr":"172.21.15.118:6833/801947386","heartbeat_front_addr":"172.21.15.118:6831/801947386","state":["exists","up"]},{"osd":7,"uuid":"f2086c47-875e-4451-a436-2008d1ea16af","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6818","nonce":653000048},{"type":"v1","addr":"172.21.15.118:6819","nonce":653000048}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6820","nonce":653000048},{"type":"v1","addr":"172.21.15.118:6821","nonce":653000048}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6824","nonce":653000048},{"type":"v1","addr":"172.21.15.118:6826","nonce":653000048}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6822","nonce":653000048},{"type":"v1","addr":"172.21.15.118:6823","nonce":653000048}]},"public_addr":"172.21.15.118:6819/653000048","cluster_addr":"172.21.15.118:6821/653000048","heartbeat_back_addr":"172.21.15.118:6826/653000048","heartbeat_front_addr":"172.21.15.118:6823/653000048","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-03T20:47:25.491994+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-03T20:47:25.587358+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-03T20:47:25.454175+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-03T20:47:25.377734+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-03T20:47:25.412351+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-03T20:47:25.499815+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-03T20:47:25.575044+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-03T20:47:25.552383+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.32:6835/2963295201":"2024-08-04T20:51:13.985515+0000","172.21.15.32:6834/2963295201":"2024-08-04T20:51:13.985515+0000","172.21.15.118:6836/3720010453":"2024-08-04T20:53:01.453702+0000","172.21.15.32:6835/3848976369":"2024-08-04T20:50:40.678865+0000","172.21.15.118:6834/3720010453":"2024-08-04T20:53:01.453702+0000","172.21.15.32:6834/3848976369":"2024-08-04T20:50:40.678865+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-08-03T20:53:06.988 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:06.987+0000 7fd3a0200640 1 -- 172.21.15.32:0/1264390637 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fd378077010 msgr2=0x7fd378079520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:53:06.988 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:06.987+0000 7fd3a0200640 1 --2- 172.21.15.32:0/1264390637 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fd378077010 0x7fd378079520 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7fd38c001050 tx=0x7fd38c001b80 comp rx=0 tx=0).stop 2024-08-03T20:53:06.988 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:06.987+0000 7fd3a0200640 1 -- 172.21.15.32:0/1264390637 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd3981f5190 msgr2=0x7fd3981f55a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:53:06.988 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:06.987+0000 7fd3a0200640 1 --2- 172.21.15.32:0/1264390637 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd3981f5190 0x7fd3981f55a0 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 crypto rx=0x7fd3980b26f0 tx=0x7fd390006430 comp rx=0 tx=0).stop 2024-08-03T20:53:06.988 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:06.987+0000 7fd3a0200640 1 -- 172.21.15.32:0/1264390637 shutdown_connections 2024-08-03T20:53:06.988 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:06.987+0000 7fd3a0200640 1 --2- 172.21.15.32:0/1264390637 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fd378077010 0x7fd378079520 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:53:06.988 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:06.987+0000 7fd3a0200640 1 --2- 172.21.15.32:0/1264390637 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fd3981442f0 0x7fd398143c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:53:06.989 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:06.987+0000 7fd3a0200640 1 --2- 172.21.15.32:0/1264390637 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd3981f5190 0x7fd3981f55a0 unknown :-1 s=CLOSED pgs=719 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:53:06.989 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:06.987+0000 7fd3a0200640 1 --2- 172.21.15.32:0/1264390637 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fd3800041c0 0x7fd398143730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:53:06.989 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:06.987+0000 7fd3a0200640 1 -- 172.21.15.32:0/1264390637 >> 172.21.15.32:0/1264390637 conn(0x7fd3980b1bd0 msgr2=0x7fd3980b11b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:53:06.989 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:06.987+0000 7fd3a0200640 1 -- 172.21.15.32:0/1264390637 shutdown_connections 2024-08-03T20:53:06.989 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:06.987+0000 7fd3a0200640 1 -- 172.21.15.32:0/1264390637 wait complete. 2024-08-03T20:53:07.013 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-08-03T20:47:28.261300+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-08-03T20:53:07.013 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-08-03T20:53:07.247 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:07.247+0000 7faba1800640 1 -- 172.21.15.32:0/2924573468 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fab80003a50 msgr2=0x7fab80003ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:53:07.248 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:07.247+0000 7faba1800640 1 --2- 172.21.15.32:0/2924573468 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fab80003a50 0x7fab80003ec0 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 crypto rx=0x7fab8c00c290 tx=0x7fab80005830 comp rx=0 tx=0).stop 2024-08-03T20:53:07.248 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:07.247+0000 7fab9b400640 1 -- 172.21.15.32:0/2924573468 reap_dead start 2024-08-03T20:53:07.248 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:07.247+0000 7faba1800640 1 -- 172.21.15.32:0/2924573468 shutdown_connections 2024-08-03T20:53:07.248 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:07.247+0000 7faba1800640 1 -- 172.21.15.32:0/2924573468 >> 172.21.15.32:0/2924573468 conn(0x7fab9c0b1bd0 msgr2=0x7fab9c0b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:53:07.248 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:07.247+0000 7faba1800640 1 -- 172.21.15.32:0/2924573468 shutdown_connections 2024-08-03T20:53:07.248 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:07.247+0000 7faba1800640 1 -- 172.21.15.32:0/2924573468 wait complete. 2024-08-03T20:53:07.249 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:07.249+0000 7faba1800640 1 Processor -- start 2024-08-03T20:53:07.249 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:07.249+0000 7faba1800640 1 -- start start 2024-08-03T20:53:07.253 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:07.251+0000 7faba1800640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fab80003a50 0x7fab9c13c020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:53:07.253 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:07.251+0000 7faba1800640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fab9c073520 0x7fab9c073930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:53:07.253 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:07.251+0000 7faba1800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fab9c073e70 0x7fab9c1f85f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:53:07.253 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:07.251+0000 7faba1800640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fab9c0cf6f0 con 0x7fab9c073e70 2024-08-03T20:53:07.253 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:07.251+0000 7faba1800640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7fab9c0cf570 con 0x7fab80003a50 2024-08-03T20:53:07.253 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:07.251+0000 7faba1800640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7fab9c0b3ce0 con 0x7fab9c073520 2024-08-03T20:53:07.253 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:07.251+0000 7fab9b400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fab9c073e70 0x7fab9c1f85f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:53:07.253 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:07.251+0000 7fab9b400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fab9c073e70 0x7fab9c1f85f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:40172/0 (socket says 172.21.15.32:40172) 2024-08-03T20:53:07.253 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:07.251+0000 7fab9b400640 1 -- 172.21.15.32:0/2276279107 learned_addr learned my addr 172.21.15.32:0/2276279107 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:53:07.253 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:07.251+0000 7fab9a000640 1 --2- 172.21.15.32:0/2276279107 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fab9c073520 0x7fab9c073930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:53:07.254 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:07.251+0000 7fab9aa00640 1 --2- 172.21.15.32:0/2276279107 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fab80003a50 0x7fab9c13c020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:53:07.254 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:07.251+0000 7fab9b400640 1 -- 172.21.15.32:0/2276279107 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fab9c073520 msgr2=0x7fab9c073930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:53:07.254 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:07.251+0000 7fab9b400640 1 --2- 172.21.15.32:0/2276279107 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fab9c073520 0x7fab9c073930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:53:07.254 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:07.251+0000 7fab9b400640 1 -- 172.21.15.32:0/2276279107 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fab80003a50 msgr2=0x7fab9c13c020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:53:07.254 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:07.251+0000 7fab9b400640 1 --2- 172.21.15.32:0/2276279107 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fab80003a50 0x7fab9c13c020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:53:07.254 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:07.251+0000 7fab9b400640 1 -- 172.21.15.32:0/2276279107 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fab9c0c5810 con 0x7fab9c073e70 2024-08-03T20:53:07.254 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:07.251+0000 7fab9a000640 1 --2- 172.21.15.32:0/2276279107 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fab9c073520 0x7fab9c073930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-03T20:53:07.255 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:07.252+0000 7fab9b400640 1 --2- 172.21.15.32:0/2276279107 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fab9c073e70 0x7fab9c1f85f0 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x7fab9c0ccab0 tx=0x7fab9400f3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:53:07.255 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:07.252+0000 7fab92a00640 1 -- 172.21.15.32:0/2276279107 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab94004b20 con 0x7fab9c073e70 2024-08-03T20:53:07.255 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:07.252+0000 7fab92a00640 1 -- 172.21.15.32:0/2276279107 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fab94004cc0 con 0x7fab9c073e70 2024-08-03T20:53:07.255 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:07.252+0000 7fab92a00640 1 -- 172.21.15.32:0/2276279107 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab94003bd0 con 0x7fab9c073e70 2024-08-03T20:53:07.255 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:07.252+0000 7faba1800640 1 -- 172.21.15.32:0/2276279107 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fab9c13c560 con 0x7fab9c073e70 2024-08-03T20:53:07.255 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:07.252+0000 7faba1800640 1 -- 172.21.15.32:0/2276279107 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fab9c072a40 con 0x7fab9c073e70 2024-08-03T20:53:07.256 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:07.253+0000 7faba1800640 1 -- 172.21.15.32:0/2276279107 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fab9c0cd100 con 0x7fab9c073e70 2024-08-03T20:53:07.256 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:07.254+0000 7fab92a00640 1 -- 172.21.15.32:0/2276279107 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7fab94004030 con 0x7fab9c073e70 2024-08-03T20:53:07.256 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:07.255+0000 7fab92a00640 1 --2- 172.21.15.32:0/2276279107 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fab7c077040 0x7fab7c079550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:53:07.256 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:07.256+0000 7fab92a00640 1 -- 172.21.15.32:0/2276279107 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(62..62 src has 1..62) ==== 5115+0+0 (secure 0 0 0) 0x7fab94098c50 con 0x7fab9c073e70 2024-08-03T20:53:07.257 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:07.256+0000 7fab9aa00640 1 --2- 172.21.15.32:0/2276279107 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fab7c077040 0x7fab7c079550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:53:07.257 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:07.257+0000 7fab9aa00640 1 --2- 172.21.15.32:0/2276279107 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fab7c077040 0x7fab7c079550 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7fab8c033000 tx=0x7fab8c02d000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:53:07.260 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:07.260+0000 7fab92a00640 1 -- 172.21.15.32:0/2276279107 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7fab94062000 con 0x7fab9c073e70 2024-08-03T20:53:07.463 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:07.462+0000 7faba1800640 1 -- 172.21.15.32:0/2276279107 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fab9c0b4ef0 con 0x7fab9c073e70 2024-08-03T20:53:07.463 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:07.463+0000 7fab92a00640 1 -- 172.21.15.32:0/2276279107 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v62) ==== 93+0+10 (secure 0 0 0) 0x7fab94061900 con 0x7fab9c073e70 2024-08-03T20:53:07.463 INFO:teuthology.orchestra.run.smithi032.stdout:pg_num: 1 2024-08-03T20:53:07.466 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:07.465+0000 7faba1800640 1 -- 172.21.15.32:0/2276279107 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fab7c077040 msgr2=0x7fab7c079550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:53:07.466 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:07.465+0000 7faba1800640 1 --2- 172.21.15.32:0/2276279107 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fab7c077040 0x7fab7c079550 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7fab8c033000 tx=0x7fab8c02d000 comp rx=0 tx=0).stop 2024-08-03T20:53:07.466 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:07.465+0000 7faba1800640 1 -- 172.21.15.32:0/2276279107 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fab9c073e70 msgr2=0x7fab9c1f85f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:53:07.466 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:07.465+0000 7faba1800640 1 --2- 172.21.15.32:0/2276279107 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fab9c073e70 0x7fab9c1f85f0 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x7fab9c0ccab0 tx=0x7fab9400f3c0 comp rx=0 tx=0).stop 2024-08-03T20:53:07.466 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:07.465+0000 7faba1800640 1 -- 172.21.15.32:0/2276279107 shutdown_connections 2024-08-03T20:53:07.466 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:07.465+0000 7faba1800640 1 --2- 172.21.15.32:0/2276279107 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fab7c077040 0x7fab7c079550 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:53:07.467 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:07.465+0000 7faba1800640 1 --2- 172.21.15.32:0/2276279107 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fab9c073e70 0x7fab9c1f85f0 unknown :-1 s=CLOSED pgs=722 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:53:07.467 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:07.465+0000 7faba1800640 1 --2- 172.21.15.32:0/2276279107 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fab9c073520 0x7fab9c073930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:53:07.467 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:07.465+0000 7faba1800640 1 --2- 172.21.15.32:0/2276279107 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fab80003a50 0x7fab9c13c020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:53:07.467 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:07.465+0000 7faba1800640 1 -- 172.21.15.32:0/2276279107 >> 172.21.15.32:0/2276279107 conn(0x7fab9c0b1bd0 msgr2=0x7fab9c15b040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:53:07.467 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:07.465+0000 7faba1800640 1 -- 172.21.15.32:0/2276279107 shutdown_connections 2024-08-03T20:53:07.467 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:07.465+0000 7faba1800640 1 -- 172.21.15.32:0/2276279107 wait complete. 2024-08-03T20:53:07.483 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-08-03T20:53:07.487 DEBUG:teuthology.orchestra.run.smithi118:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-08-03T20:53:07.491 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_damage.TestDamage.test_dentry_first_preflush' 2024-08-03T20:53:07.762 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:07.761+0000 7f1f58000640 1 -- 172.21.15.32:0/1052834565 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f1f5015f610 msgr2=0x7f1f500cdf70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:53:07.762 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:07.761+0000 7f1f58000640 1 --2- 172.21.15.32:0/1052834565 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f1f5015f610 0x7f1f500cdf70 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f1f48005fd0 tx=0x7f1f4802a8b0 comp rx=0 tx=0).stop 2024-08-03T20:53:07.762 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:07.761+0000 7f1f58000640 1 -- 172.21.15.32:0/1052834565 shutdown_connections 2024-08-03T20:53:07.762 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:07.761+0000 7f1f58000640 1 --2- 172.21.15.32:0/1052834565 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f1f5015f610 0x7f1f500cdf70 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:53:07.762 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:07.761+0000 7f1f58000640 1 --2- 172.21.15.32:0/1052834565 >> v2:172.21.15.118:3300/0 conn(0x7f1f500b38b0 0x7f1f501523f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:53:07.762 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:07.761+0000 7f1f58000640 1 -- 172.21.15.32:0/1052834565 >> 172.21.15.32:0/1052834565 conn(0x7f1f500b1bd0 msgr2=0x7f1f500b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:53:07.763 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:07.761+0000 7f1f58000640 1 -- 172.21.15.32:0/1052834565 shutdown_connections 2024-08-03T20:53:07.763 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:07.761+0000 7f1f58000640 1 -- 172.21.15.32:0/1052834565 wait complete. 2024-08-03T20:53:07.763 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:07.763+0000 7f1f58000640 1 Processor -- start 2024-08-03T20:53:07.764 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:07.763+0000 7f1f58000640 1 -- start start 2024-08-03T20:53:07.767 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:07.765+0000 7f1f58000640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f1f500b38b0 0x7f1f50157820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:53:07.767 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:07.765+0000 7f1f58000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1f500b4ce0 0x7f1f500aed60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:53:07.767 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:07.765+0000 7f1f58000640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f1f501f5870 0x7f1f50157ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:53:07.767 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:07.765+0000 7f1f58000640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f1f500cea90 con 0x7f1f500b4ce0 2024-08-03T20:53:07.767 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:07.765+0000 7f1f58000640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f1f500ce910 con 0x7f1f501f5870 2024-08-03T20:53:07.767 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:07.765+0000 7f1f58000640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7f1f500cec10 con 0x7f1f500b38b0 2024-08-03T20:53:07.767 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:07.765+0000 7f1f54c00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1f500b4ce0 0x7f1f500aed60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:53:07.767 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:07.765+0000 7f1f54c00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1f500b4ce0 0x7f1f500aed60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:40174/0 (socket says 172.21.15.32:40174) 2024-08-03T20:53:07.767 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:07.765+0000 7f1f54c00640 1 -- 172.21.15.32:0/415928948 learned_addr learned my addr 172.21.15.32:0/415928948 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:53:07.768 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:07.765+0000 7f1f55600640 1 --2- 172.21.15.32:0/415928948 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f1f500b38b0 0x7f1f50157820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:53:07.768 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:07.765+0000 7f1f56000640 1 --2- 172.21.15.32:0/415928948 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f1f501f5870 0x7f1f50157ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:53:07.768 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:07.765+0000 7f1f54c00640 1 -- 172.21.15.32:0/415928948 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f1f500b38b0 msgr2=0x7f1f50157820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:53:07.768 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:07.765+0000 7f1f54c00640 1 --2- 172.21.15.32:0/415928948 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f1f500b38b0 0x7f1f50157820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:53:07.768 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:07.766+0000 7f1f54c00640 1 -- 172.21.15.32:0/415928948 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f1f501f5870 msgr2=0x7f1f50157ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:53:07.768 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:07.766+0000 7f1f54c00640 1 --2- 172.21.15.32:0/415928948 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f1f501f5870 0x7f1f50157ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:53:07.768 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:07.766+0000 7f1f54c00640 1 -- 172.21.15.32:0/415928948 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1f50072e40 con 0x7f1f500b4ce0 2024-08-03T20:53:07.768 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:07.766+0000 7f1f54c00640 1 --2- 172.21.15.32:0/415928948 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1f500b4ce0 0x7f1f500aed60 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 crypto rx=0x7f1f3c00c430 tx=0x7f1f3c00c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:53:07.768 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:07.766+0000 7f1f4d600640 1 -- 172.21.15.32:0/415928948 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f3c011790 con 0x7f1f500b4ce0 2024-08-03T20:53:07.768 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:07.766+0000 7f1f4d600640 1 -- 172.21.15.32:0/415928948 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f1f3c017070 con 0x7f1f500b4ce0 2024-08-03T20:53:07.768 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:07.766+0000 7f1f4d600640 1 -- 172.21.15.32:0/415928948 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f3c011ce0 con 0x7f1f500b4ce0 2024-08-03T20:53:07.769 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:07.766+0000 7f1f58000640 1 -- 172.21.15.32:0/415928948 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1f501f4d90 con 0x7f1f500b4ce0 2024-08-03T20:53:07.769 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:07.766+0000 7f1f58000640 1 -- 172.21.15.32:0/415928948 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1f501f5350 con 0x7f1f500b4ce0 2024-08-03T20:53:07.769 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:07.767+0000 7f1f58000640 1 -- 172.21.15.32:0/415928948 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1f500cf130 con 0x7f1f500b4ce0 2024-08-03T20:53:07.769 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:07.769+0000 7f1f4d600640 1 -- 172.21.15.32:0/415928948 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f1f3c01c460 con 0x7f1f500b4ce0 2024-08-03T20:53:07.771 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:07.770+0000 7f1f4d600640 1 --2- 172.21.15.32:0/415928948 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f1f30077010 0x7f1f30079520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:53:07.771 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:07.770+0000 7f1f4d600640 1 -- 172.21.15.32:0/415928948 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(62..62 src has 1..62) ==== 5115+0+0 (secure 0 0 0) 0x7f1f3c098480 con 0x7f1f500b4ce0 2024-08-03T20:53:07.771 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:07.771+0000 7f1f55600640 1 --2- 172.21.15.32:0/415928948 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f1f30077010 0x7f1f30079520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:53:07.772 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:07.771+0000 7f1f55600640 1 --2- 172.21.15.32:0/415928948 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f1f30077010 0x7f1f30079520 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f1f44009b30 tx=0x7f1f44005f90 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:53:07.775 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:07.775+0000 7f1f4d600640 1 -- 172.21.15.32:0/415928948 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7f1f3c0611e0 con 0x7f1f500b4ce0 2024-08-03T20:53:08.056 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:08.055+0000 7f1f58000640 1 -- 172.21.15.32:0/415928948 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_damage.TestDamage.test_dentry_first_preflush"]} v 0) -- 0x7f1f501f4b30 con 0x7f1f500b4ce0 2024-08-03T20:53:08.505 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:08.504+0000 7f1f4d600640 1 -- 172.21.15.32:0/415928948 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_damage.TestDamage.test_dentry_first_preflush"]}]=0 v271) ==== 144+0+0 (secure 0 0 0) 0x7f1f3c065730 con 0x7f1f500b4ce0 2024-08-03T20:53:08.509 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:08.508+0000 7f1f58000640 1 -- 172.21.15.32:0/415928948 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f1f30077010 msgr2=0x7f1f30079520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:53:08.509 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:08.508+0000 7f1f58000640 1 --2- 172.21.15.32:0/415928948 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f1f30077010 0x7f1f30079520 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f1f44009b30 tx=0x7f1f44005f90 comp rx=0 tx=0).stop 2024-08-03T20:53:08.509 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:08.508+0000 7f1f58000640 1 -- 172.21.15.32:0/415928948 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1f500b4ce0 msgr2=0x7f1f500aed60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:53:08.509 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:08.508+0000 7f1f58000640 1 --2- 172.21.15.32:0/415928948 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1f500b4ce0 0x7f1f500aed60 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 crypto rx=0x7f1f3c00c430 tx=0x7f1f3c00c8f0 comp rx=0 tx=0).stop 2024-08-03T20:53:08.510 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:08.508+0000 7f1f58000640 1 -- 172.21.15.32:0/415928948 shutdown_connections 2024-08-03T20:53:08.510 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:08.508+0000 7f1f58000640 1 --2- 172.21.15.32:0/415928948 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f1f30077010 0x7f1f30079520 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:53:08.510 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:08.508+0000 7f1f58000640 1 --2- 172.21.15.32:0/415928948 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f1f501f5870 0x7f1f50157ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:53:08.510 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:08.508+0000 7f1f58000640 1 --2- 172.21.15.32:0/415928948 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1f500b4ce0 0x7f1f500aed60 unknown :-1 s=CLOSED pgs=724 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:53:08.510 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:08.508+0000 7f1f58000640 1 --2- 172.21.15.32:0/415928948 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f1f500b38b0 0x7f1f50157820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:53:08.510 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:08.508+0000 7f1f58000640 1 -- 172.21.15.32:0/415928948 >> 172.21.15.32:0/415928948 conn(0x7f1f500b1bd0 msgr2=0x7f1f50152210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:53:08.510 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:08.509+0000 7f1f58000640 1 -- 172.21.15.32:0/415928948 shutdown_connections 2024-08-03T20:53:08.510 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:08.509+0000 7f1f58000640 1 -- 172.21.15.32:0/415928948 wait complete. 2024-08-03T20:53:08.533 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config dump 2024-08-03T20:53:08.762 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:08.761+0000 7f4fb0c00640 1 -- 172.21.15.32:0/1007736517 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4fac073e60 msgr2=0x7f4fac074270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:53:08.762 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:08.761+0000 7f4fb0c00640 1 --2- 172.21.15.32:0/1007736517 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4fac073e60 0x7f4fac074270 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7f4f94005fd0 tx=0x7f4f9402a540 comp rx=0 tx=0).stop 2024-08-03T20:53:08.762 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:08.761+0000 7f4fb0c00640 1 -- 172.21.15.32:0/1007736517 shutdown_connections 2024-08-03T20:53:08.762 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:08.761+0000 7f4fb0c00640 1 --2- 172.21.15.32:0/1007736517 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4fac073e60 0x7f4fac074270 unknown :-1 s=CLOSED pgs=726 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:53:08.762 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:08.761+0000 7f4fb0c00640 1 --2- 172.21.15.32:0/1007736517 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f4fac06e930 0x7f4fac144f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:53:08.762 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:08.761+0000 7f4fb0c00640 1 -- 172.21.15.32:0/1007736517 >> 172.21.15.32:0/1007736517 conn(0x7f4fac0b1bd0 msgr2=0x7f4fac0b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:53:08.762 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:08.761+0000 7f4fb0c00640 1 -- 172.21.15.32:0/1007736517 shutdown_connections 2024-08-03T20:53:08.762 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:08.761+0000 7f4fb0c00640 1 -- 172.21.15.32:0/1007736517 wait complete. 2024-08-03T20:53:08.764 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:08.763+0000 7f4fb0c00640 1 Processor -- start 2024-08-03T20:53:08.764 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:08.763+0000 7f4fb0c00640 1 -- start start 2024-08-03T20:53:08.766 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:08.765+0000 7f4fb0c00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f4fac06e930 0x7f4fac1418f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:53:08.766 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:08.765+0000 7f4fb0c00640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f4fac073e60 0x7f4fac143400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:53:08.766 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:08.765+0000 7f4fb0c00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4fac0747b0 0x7f4fac15ec40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:53:08.766 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:08.765+0000 7f4fb0c00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f4fac0cf9c0 con 0x7f4fac0747b0 2024-08-03T20:53:08.766 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:08.765+0000 7f4fb0c00640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f4fac0cf840 con 0x7f4fac06e930 2024-08-03T20:53:08.767 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:08.765+0000 7f4fb0c00640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7f4fac0ccff0 con 0x7f4fac073e60 2024-08-03T20:53:08.767 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:08.765+0000 7f4faa800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4fac0747b0 0x7f4fac15ec40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:53:08.767 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:08.765+0000 7f4faa800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4fac0747b0 0x7f4fac15ec40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:40182/0 (socket says 172.21.15.32:40182) 2024-08-03T20:53:08.767 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:08.765+0000 7f4faa800640 1 -- 172.21.15.32:0/79480677 learned_addr learned my addr 172.21.15.32:0/79480677 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:53:08.767 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:08.765+0000 7f4fa9e00640 1 --2- 172.21.15.32:0/79480677 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f4fac06e930 0x7f4fac1418f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:53:08.767 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:08.765+0000 7f4fa1400640 1 --2- 172.21.15.32:0/79480677 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f4fac073e60 0x7f4fac143400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:53:08.767 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:08.766+0000 7f4faa800640 1 -- 172.21.15.32:0/79480677 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f4fac073e60 msgr2=0x7f4fac143400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:53:08.767 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:08.766+0000 7f4faa800640 1 --2- 172.21.15.32:0/79480677 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f4fac073e60 0x7f4fac143400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:53:08.767 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:08.766+0000 7f4faa800640 1 -- 172.21.15.32:0/79480677 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f4fac06e930 msgr2=0x7f4fac1418f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:53:08.767 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:08.766+0000 7f4faa800640 1 --2- 172.21.15.32:0/79480677 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f4fac06e930 0x7f4fac1418f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:53:08.767 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:08.766+0000 7f4faa800640 1 -- 172.21.15.32:0/79480677 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4fac143ec0 con 0x7f4fac0747b0 2024-08-03T20:53:08.768 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:08.766+0000 7f4faa800640 1 --2- 172.21.15.32:0/79480677 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4fac0747b0 0x7f4fac15ec40 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 crypto rx=0x7f4fa400c700 tx=0x7f4fa400cbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:53:08.768 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:08.766+0000 7f4fa2a00640 1 -- 172.21.15.32:0/79480677 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4fa4003b70 con 0x7f4fac0747b0 2024-08-03T20:53:08.769 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:08.766+0000 7f4fa2a00640 1 -- 172.21.15.32:0/79480677 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f4fa4003d10 con 0x7f4fac0747b0 2024-08-03T20:53:08.769 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:08.766+0000 7f4fb0c00640 1 -- 172.21.15.32:0/79480677 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4fac1441a0 con 0x7f4fac0747b0 2024-08-03T20:53:08.769 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:08.766+0000 7f4fa2a00640 1 -- 172.21.15.32:0/79480677 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4fa400aab0 con 0x7f4fac0747b0 2024-08-03T20:53:08.769 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:08.767+0000 7f4fb0c00640 1 -- 172.21.15.32:0/79480677 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4fac144760 con 0x7f4fac0747b0 2024-08-03T20:53:08.769 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:08.768+0000 7f4fb0c00640 1 -- 172.21.15.32:0/79480677 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4fac143c10 con 0x7f4fac0747b0 2024-08-03T20:53:08.769 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:08.769+0000 7f4fa2a00640 1 -- 172.21.15.32:0/79480677 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f4fa400ac50 con 0x7f4fac0747b0 2024-08-03T20:53:08.770 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:08.770+0000 7f4fa2a00640 1 --2- 172.21.15.32:0/79480677 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f4f88077040 0x7f4f88079550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:53:08.770 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:08.770+0000 7f4fa2a00640 1 -- 172.21.15.32:0/79480677 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(62..62 src has 1..62) ==== 5115+0+0 (secure 0 0 0) 0x7f4fa40983d0 con 0x7f4fac0747b0 2024-08-03T20:53:08.771 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:08.770+0000 7f4fa9e00640 1 --2- 172.21.15.32:0/79480677 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f4f88077040 0x7f4f88079550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:53:08.771 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:08.771+0000 7f4fa9e00640 1 --2- 172.21.15.32:0/79480677 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f4f88077040 0x7f4f88079550 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f4f9c00bb50 tx=0x7f4f9c00b2e0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:53:08.775 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:08.774+0000 7f4fa2a00640 1 -- 172.21.15.32:0/79480677 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7f4fa40611c0 con 0x7f4fac0747b0 2024-08-03T20:53:08.977 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:08.976+0000 7f4fb0c00640 1 -- 172.21.15.32:0/79480677 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "config dump"} v 0) -- 0x7f4fac0ccdb0 con 0x7f4fac0747b0 2024-08-03T20:53:08.977 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:08.977+0000 7f4fa2a00640 1 -- 172.21.15.32:0/79480677 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "config dump"}]=0 v38) ==== 59+0+1998 (secure 0 0 0) 0x7f4fa4065710 con 0x7f4fac0747b0 2024-08-03T20:53:08.978 INFO:teuthology.orchestra.run.smithi032.stdout:WHO MASK LEVEL OPTION VALUE RO 2024-08-03T20:53:08.978 INFO:teuthology.orchestra.run.smithi032.stdout:mon advanced mon_op_complaint_time 120 2024-08-03T20:53:08.978 INFO:teuthology.orchestra.run.smithi032.stdout:mon dev mon_warn_on_pool_no_app false 2024-08-03T20:53:08.978 INFO:teuthology.orchestra.run.smithi032.stdout:osd advanced osd_op_complaint_time 180.000000 2024-08-03T20:53:08.978 INFO:teuthology.orchestra.run.smithi032.stdout:osd.0 basic osd_mclock_max_capacity_iops_ssd 4313.565729 2024-08-03T20:53:08.978 INFO:teuthology.orchestra.run.smithi032.stdout:osd.1 basic osd_mclock_max_capacity_iops_ssd 4272.517979 2024-08-03T20:53:08.978 INFO:teuthology.orchestra.run.smithi032.stdout:osd.2 basic osd_mclock_max_capacity_iops_ssd 4131.201546 2024-08-03T20:53:08.978 INFO:teuthology.orchestra.run.smithi032.stdout:osd.3 basic osd_mclock_max_capacity_iops_ssd 4155.765914 2024-08-03T20:53:08.978 INFO:teuthology.orchestra.run.smithi032.stdout:osd.4 basic osd_mclock_max_capacity_iops_ssd 5744.667532 2024-08-03T20:53:08.979 INFO:teuthology.orchestra.run.smithi032.stdout:osd.5 basic osd_mclock_max_capacity_iops_ssd 3764.678226 2024-08-03T20:53:08.979 INFO:teuthology.orchestra.run.smithi032.stdout:osd.6 basic osd_mclock_max_capacity_iops_ssd 3927.989098 2024-08-03T20:53:08.979 INFO:teuthology.orchestra.run.smithi032.stdout:osd.7 basic osd_mclock_max_capacity_iops_ssd 3760.968870 2024-08-03T20:53:08.979 INFO:teuthology.orchestra.run.smithi032.stdout:mds advanced debug_mds 20/20 2024-08-03T20:53:08.979 INFO:teuthology.orchestra.run.smithi032.stdout:mds advanced debug_mds_balancer 20/20 2024-08-03T20:53:08.979 INFO:teuthology.orchestra.run.smithi032.stdout:mds advanced debug_ms 2/2 2024-08-03T20:53:08.979 INFO:teuthology.orchestra.run.smithi032.stdout:mds dev mds_debug_frag true 2024-08-03T20:53:08.979 INFO:teuthology.orchestra.run.smithi032.stdout:mds dev mds_debug_scatterstat true 2024-08-03T20:53:08.979 INFO:teuthology.orchestra.run.smithi032.stdout:mds advanced mds_op_complaint_time 180.000000 2024-08-03T20:53:08.979 INFO:teuthology.orchestra.run.smithi032.stdout:mds dev mds_verify_scatter true 2024-08-03T20:53:08.979 INFO:teuthology.orchestra.run.smithi032.stdout:mds advanced osd_op_complaint_time 180.000000 2024-08-03T20:53:08.979 INFO:teuthology.orchestra.run.smithi032.stdout:mds advanced rados_mon_op_timeout 900 2024-08-03T20:53:08.979 INFO:teuthology.orchestra.run.smithi032.stdout:mds advanced rados_osd_op_timeout 900 2024-08-03T20:53:08.980 INFO:teuthology.orchestra.run.smithi032.stdout:client advanced client_mount_timeout 600 2024-08-03T20:53:08.980 INFO:teuthology.orchestra.run.smithi032.stdout:client advanced debug_client 20/20 2024-08-03T20:53:08.980 INFO:teuthology.orchestra.run.smithi032.stdout:client advanced debug_ms 1/1 2024-08-03T20:53:08.980 INFO:teuthology.orchestra.run.smithi032.stdout:client advanced rados_mon_op_timeout 900 2024-08-03T20:53:08.980 INFO:teuthology.orchestra.run.smithi032.stdout:client advanced rados_osd_op_timeout 900 2024-08-03T20:53:08.981 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:08.980+0000 7f4fb0c00640 1 -- 172.21.15.32:0/79480677 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f4f88077040 msgr2=0x7f4f88079550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:53:08.981 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:08.980+0000 7f4fb0c00640 1 --2- 172.21.15.32:0/79480677 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f4f88077040 0x7f4f88079550 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f4f9c00bb50 tx=0x7f4f9c00b2e0 comp rx=0 tx=0).stop 2024-08-03T20:53:08.981 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:08.980+0000 7f4fb0c00640 1 -- 172.21.15.32:0/79480677 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4fac0747b0 msgr2=0x7f4fac15ec40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:53:08.981 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:08.980+0000 7f4fb0c00640 1 --2- 172.21.15.32:0/79480677 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4fac0747b0 0x7f4fac15ec40 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 crypto rx=0x7f4fa400c700 tx=0x7f4fa400cbc0 comp rx=0 tx=0).stop 2024-08-03T20:53:08.981 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:08.980+0000 7f4fb0c00640 1 -- 172.21.15.32:0/79480677 shutdown_connections 2024-08-03T20:53:08.981 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:08.980+0000 7f4fb0c00640 1 --2- 172.21.15.32:0/79480677 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f4f88077040 0x7f4f88079550 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:53:08.981 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:08.980+0000 7f4fb0c00640 1 --2- 172.21.15.32:0/79480677 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4fac0747b0 0x7f4fac15ec40 unknown :-1 s=CLOSED pgs=727 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:53:08.981 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:08.980+0000 7f4fb0c00640 1 --2- 172.21.15.32:0/79480677 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f4fac073e60 0x7f4fac143400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:53:08.981 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:08.980+0000 7f4fb0c00640 1 --2- 172.21.15.32:0/79480677 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f4fac06e930 0x7f4fac1418f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:53:08.981 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:08.980+0000 7f4fb0c00640 1 -- 172.21.15.32:0/79480677 >> 172.21.15.32:0/79480677 conn(0x7f4fac0b1bd0 msgr2=0x7f4fac0b07b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:53:08.982 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:08.980+0000 7f4fb0c00640 1 -- 172.21.15.32:0/79480677 shutdown_connections 2024-08-03T20:53:08.982 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:08.980+0000 7f4fb0c00640 1 -- 172.21.15.32:0/79480677 wait complete. 2024-08-03T20:53:09.006 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-08-03T20:53:09.239 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:09.238+0000 7fb3b5e00640 1 -- 172.21.15.32:0/3563622785 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb394003a00 msgr2=0x7fb394003e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:53:09.239 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:09.238+0000 7fb3b5e00640 1 --2- 172.21.15.32:0/3563622785 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb394003a00 0x7fb394003e70 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7fb3a0009b50 tx=0x7fb394005830 comp rx=0 tx=0).stop 2024-08-03T20:53:09.239 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:09.238+0000 7fb3af400640 1 -- 172.21.15.32:0/3563622785 reap_dead start 2024-08-03T20:53:09.239 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:09.238+0000 7fb3b5e00640 1 -- 172.21.15.32:0/3563622785 shutdown_connections 2024-08-03T20:53:09.239 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:09.238+0000 7fb3b5e00640 1 -- 172.21.15.32:0/3563622785 >> 172.21.15.32:0/3563622785 conn(0x7fb3b00b1bd0 msgr2=0x7fb3b00b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:53:09.239 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:09.238+0000 7fb3b5e00640 1 -- 172.21.15.32:0/3563622785 shutdown_connections 2024-08-03T20:53:09.239 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:09.238+0000 7fb3b5e00640 1 -- 172.21.15.32:0/3563622785 wait complete. 2024-08-03T20:53:09.241 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:09.240+0000 7fb3b5e00640 1 Processor -- start 2024-08-03T20:53:09.241 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:09.240+0000 7fb3b5e00640 1 -- start start 2024-08-03T20:53:09.242 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:09.242+0000 7fb3b5e00640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fb394003a00 0x7fb3b01f8480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:53:09.242 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:09.242+0000 7fb3b5e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb3b00727f0 0x7fb3b0072c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:53:09.243 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:09.242+0000 7fb3b5e00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb3b01e4c00 0x7fb3b01e5010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:53:09.243 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:09.242+0000 7fb3b5e00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fb3b00cd340 con 0x7fb3b00727f0 2024-08-03T20:53:09.243 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:09.242+0000 7fb3b5e00640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7fb3b00cd1c0 con 0x7fb3b01e4c00 2024-08-03T20:53:09.243 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:09.242+0000 7fb3b5e00640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7fb3b00cd4c0 con 0x7fb394003a00 2024-08-03T20:53:09.243 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:09.242+0000 7fb3a7e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb3b00727f0 0x7fb3b0072c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:53:09.243 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:09.242+0000 7fb3a7e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb3b00727f0 0x7fb3b0072c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:40194/0 (socket says 172.21.15.32:40194) 2024-08-03T20:53:09.243 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:09.242+0000 7fb3a7e00640 1 -- 172.21.15.32:0/660151067 learned_addr learned my addr 172.21.15.32:0/660151067 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:53:09.244 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:09.242+0000 7fb3af400640 1 --2- 172.21.15.32:0/660151067 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb3b01e4c00 0x7fb3b01e5010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:53:09.244 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:09.242+0000 7fb3a7e00640 1 -- 172.21.15.32:0/660151067 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fb394003a00 msgr2=0x7fb3b01f8480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:53:09.244 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:09.242+0000 7fb3aea00640 1 --2- 172.21.15.32:0/660151067 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fb394003a00 0x7fb3b01f8480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:53:09.245 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:09.242+0000 7fb3a7e00640 1 --2- 172.21.15.32:0/660151067 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fb394003a00 0x7fb3b01f8480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:53:09.245 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:09.242+0000 7fb3a7e00640 1 -- 172.21.15.32:0/660151067 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb3b01e4c00 msgr2=0x7fb3b01e5010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:53:09.245 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:09.242+0000 7fb3a7e00640 1 --2- 172.21.15.32:0/660151067 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb3b01e4c00 0x7fb3b01e5010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:53:09.245 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:09.242+0000 7fb3a7e00640 1 -- 172.21.15.32:0/660151067 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb3b0073520 con 0x7fb3b00727f0 2024-08-03T20:53:09.245 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:09.243+0000 7fb3aea00640 1 --2- 172.21.15.32:0/660151067 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fb394003a00 0x7fb3b01f8480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-03T20:53:09.245 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:09.243+0000 7fb3af400640 1 --2- 172.21.15.32:0/660151067 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb3b01e4c00 0x7fb3b01e5010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-03T20:53:09.245 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:09.243+0000 7fb3a7e00640 1 --2- 172.21.15.32:0/660151067 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb3b00727f0 0x7fb3b0072c00 secure :-1 s=READY pgs=730 cs=0 l=1 rev1=1 crypto rx=0x7fb398006680 tx=0x7fb3980066b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:53:09.245 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:09.243+0000 7fb3a7400640 1 -- 172.21.15.32:0/660151067 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb398015070 con 0x7fb3b00727f0 2024-08-03T20:53:09.245 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:09.243+0000 7fb3a7400640 1 -- 172.21.15.32:0/660151067 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fb398011920 con 0x7fb3b00727f0 2024-08-03T20:53:09.245 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:09.243+0000 7fb3b5e00640 1 -- 172.21.15.32:0/660151067 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb3b0073800 con 0x7fb3b00727f0 2024-08-03T20:53:09.245 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:09.243+0000 7fb3a7400640 1 -- 172.21.15.32:0/660151067 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb398011d70 con 0x7fb3b00727f0 2024-08-03T20:53:09.245 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:09.243+0000 7fb3b5e00640 1 -- 172.21.15.32:0/660151067 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb3b0073dc0 con 0x7fb3b00727f0 2024-08-03T20:53:09.246 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:09.244+0000 7fb3b5e00640 1 -- 172.21.15.32:0/660151067 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb3b00ce630 con 0x7fb3b00727f0 2024-08-03T20:53:09.246 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:09.245+0000 7fb3a7400640 1 -- 172.21.15.32:0/660151067 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7fb398018070 con 0x7fb3b00727f0 2024-08-03T20:53:09.247 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:09.246+0000 7fb3a7400640 1 --2- 172.21.15.32:0/660151067 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fb38c077010 0x7fb38c079520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:53:09.247 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:09.247+0000 7fb3a7400640 1 -- 172.21.15.32:0/660151067 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(62..62 src has 1..62) ==== 5115+0+0 (secure 0 0 0) 0x7fb398099380 con 0x7fb3b00727f0 2024-08-03T20:53:09.247 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:09.247+0000 7fb3aea00640 1 --2- 172.21.15.32:0/660151067 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fb38c077010 0x7fb38c079520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:53:09.248 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:09.248+0000 7fb3aea00640 1 --2- 172.21.15.32:0/660151067 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fb38c077010 0x7fb38c079520 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7fb3a0009b50 tx=0x7fb3a002d040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:53:09.251 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:09.251+0000 7fb3a7400640 1 -- 172.21.15.32:0/660151067 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7fb3b00ce630 con 0x7fb3b00727f0 2024-08-03T20:53:09.455 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:09.454+0000 7fb3b5e00640 1 -- 172.21.15.32:0/660151067 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) -- 0x7fb3b00732b0 con 0x7fb3b00727f0 2024-08-03T20:53:09.459 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:09.459+0000 7fb3a7400640 1 -- 172.21.15.32:0/660151067 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v39)=0 v39) ==== 122+0+0 (secure 0 0 0) 0x7fb3980627c0 con 0x7fb3b00727f0 2024-08-03T20:53:09.462 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:09.461+0000 7fb3b5e00640 1 -- 172.21.15.32:0/660151067 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fb38c077010 msgr2=0x7fb38c079520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:53:09.462 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:09.461+0000 7fb3b5e00640 1 --2- 172.21.15.32:0/660151067 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fb38c077010 0x7fb38c079520 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7fb3a0009b50 tx=0x7fb3a002d040 comp rx=0 tx=0).stop 2024-08-03T20:53:09.462 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:09.461+0000 7fb3b5e00640 1 -- 172.21.15.32:0/660151067 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb3b00727f0 msgr2=0x7fb3b0072c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:53:09.462 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:09.461+0000 7fb3b5e00640 1 --2- 172.21.15.32:0/660151067 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb3b00727f0 0x7fb3b0072c00 secure :-1 s=READY pgs=730 cs=0 l=1 rev1=1 crypto rx=0x7fb398006680 tx=0x7fb3980066b0 comp rx=0 tx=0).stop 2024-08-03T20:53:09.462 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:09.461+0000 7fb3b5e00640 1 -- 172.21.15.32:0/660151067 shutdown_connections 2024-08-03T20:53:09.462 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:09.461+0000 7fb3b5e00640 1 --2- 172.21.15.32:0/660151067 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fb38c077010 0x7fb38c079520 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:53:09.462 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:09.461+0000 7fb3b5e00640 1 --2- 172.21.15.32:0/660151067 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb3b01e4c00 0x7fb3b01e5010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:53:09.462 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:09.461+0000 7fb3b5e00640 1 --2- 172.21.15.32:0/660151067 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb3b00727f0 0x7fb3b0072c00 unknown :-1 s=CLOSED pgs=730 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:53:09.462 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:09.461+0000 7fb3b5e00640 1 --2- 172.21.15.32:0/660151067 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fb394003a00 0x7fb3b01f8480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:53:09.463 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:09.461+0000 7fb3b5e00640 1 -- 172.21.15.32:0/660151067 >> 172.21.15.32:0/660151067 conn(0x7fb3b00b1bd0 msgr2=0x7fb3b015da50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:53:09.463 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:09.461+0000 7fb3b5e00640 1 -- 172.21.15.32:0/660151067 shutdown_connections 2024-08-03T20:53:09.463 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:09.461+0000 7fb3b5e00640 1 -- 172.21.15.32:0/660151067 wait complete. 2024-08-03T20:53:09.484 INFO:teuthology.nuke:Clearing teuthology firewall rules... 2024-08-03T20:53:09.484 DEBUG:teuthology.orchestra.run.smithi032:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-08-03T20:53:09.573 DEBUG:teuthology.orchestra.run.smithi118:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-08-03T20:53:09.671 INFO:teuthology.nuke:Cleared teuthology firewall rules. 2024-08-03T20:53:09.672 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-08-03T20:53:09.672 DEBUG:teuthology.orchestra.run.smithi032:> dd if=/proc/self/mounts of=/dev/stdout 2024-08-03T20:53:09.707 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053453,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6499208k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 none /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249604k,nr_inodes=812401,mode=700,uid=1000,gid=1316 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6499208k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-08-03T20:53:09.707 DEBUG:teuthology.orchestra.run.smithi118:> set -ex 2024-08-03T20:53:09.707 DEBUG:teuthology.orchestra.run.smithi118:> dd if=/proc/self/mounts of=/dev/stdout 2024-08-03T20:53:09.745 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052663,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497944k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248972k,nr_inodes=812243,mode=700,uid=1000,gid=1316 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497944k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-08-03T20:53:09.745 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-08-03T20:53:09.745 DEBUG:teuthology.orchestra.run.smithi032:> dd if=/proc/self/mounts of=/dev/stdout 2024-08-03T20:53:09.782 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053453,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6499208k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 none /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249604k,nr_inodes=812401,mode=700,uid=1000,gid=1316 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6499208k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-08-03T20:53:09.783 DEBUG:teuthology.orchestra.run.smithi118:> set -ex 2024-08-03T20:53:09.783 DEBUG:teuthology.orchestra.run.smithi118:> dd if=/proc/self/mounts of=/dev/stdout 2024-08-03T20:53:09.821 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052663,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497944k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248972k,nr_inodes=812243,mode=700,uid=1000,gid=1316 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497944k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-08-03T20:53:09.821 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-08-03T20:53:09.821 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-08-03T20:53:10.050 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:10.049+0000 7ffb88000640 1 -- 172.21.15.32:0/1445852278 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ffb8006e930 msgr2=0x7ffb80144f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:53:10.051 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:10.049+0000 7ffb88000640 1 --2- 172.21.15.32:0/1445852278 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ffb8006e930 0x7ffb80144f90 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7ffb80072d80 tx=0x7ffb7402a5f0 comp rx=0 tx=0).stop 2024-08-03T20:53:10.051 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:10.049+0000 7ffb88000640 1 -- 172.21.15.32:0/1445852278 shutdown_connections 2024-08-03T20:53:10.051 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:10.049+0000 7ffb88000640 1 --2- 172.21.15.32:0/1445852278 >> v2:172.21.15.118:3300/0 conn(0x7ffb8015fff0 0x7ffb80160400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:53:10.051 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:10.049+0000 7ffb88000640 1 --2- 172.21.15.32:0/1445852278 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ffb8006e930 0x7ffb80144f90 unknown :-1 s=CLOSED pgs=731 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:53:10.051 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:10.049+0000 7ffb88000640 1 -- 172.21.15.32:0/1445852278 >> 172.21.15.32:0/1445852278 conn(0x7ffb800b1bd0 msgr2=0x7ffb800b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:53:10.051 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:10.049+0000 7ffb88000640 1 -- 172.21.15.32:0/1445852278 shutdown_connections 2024-08-03T20:53:10.051 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:10.049+0000 7ffb88000640 1 -- 172.21.15.32:0/1445852278 wait complete. 2024-08-03T20:53:10.052 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:10.052+0000 7ffb88000640 1 Processor -- start 2024-08-03T20:53:10.052 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:10.052+0000 7ffb88000640 1 -- start start 2024-08-03T20:53:10.054 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:10.053+0000 7ffb88000640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7ffb8006e930 0x7ffb801436b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:53:10.055 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:10.053+0000 7ffb88000640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ffb80073e60 0x7ffb801e6970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:53:10.055 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:10.053+0000 7ffb88000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ffb8015fff0 0x7ffb8015ba10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:53:10.055 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:10.053+0000 7ffb88000640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7ffb800cf840 con 0x7ffb8015fff0 2024-08-03T20:53:10.055 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:10.053+0000 7ffb88000640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7ffb80145f30 con 0x7ffb80073e60 2024-08-03T20:53:10.055 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:10.053+0000 7ffb88000640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7ffb800cf9c0 con 0x7ffb8006e930 2024-08-03T20:53:10.055 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:10.054+0000 7ffb86000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ffb8015fff0 0x7ffb8015ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:53:10.055 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:10.054+0000 7ffb86000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ffb8015fff0 0x7ffb8015ba10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:45786/0 (socket says 172.21.15.32:45786) 2024-08-03T20:53:10.055 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:10.054+0000 7ffb86000640 1 -- 172.21.15.32:0/3845321196 learned_addr learned my addr 172.21.15.32:0/3845321196 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:53:10.055 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:10.054+0000 7ffb84c00640 1 --2- 172.21.15.32:0/3845321196 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ffb80073e60 0x7ffb801e6970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:53:10.055 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:10.054+0000 7ffb85600640 1 --2- 172.21.15.32:0/3845321196 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7ffb8006e930 0x7ffb801436b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:53:10.055 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:10.054+0000 7ffb86000640 1 -- 172.21.15.32:0/3845321196 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7ffb8006e930 msgr2=0x7ffb801436b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:53:10.055 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:10.054+0000 7ffb86000640 1 --2- 172.21.15.32:0/3845321196 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7ffb8006e930 0x7ffb801436b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:53:10.055 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:10.054+0000 7ffb86000640 1 -- 172.21.15.32:0/3845321196 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ffb80073e60 msgr2=0x7ffb801e6970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:53:10.056 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:10.054+0000 7ffb86000640 1 --2- 172.21.15.32:0/3845321196 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ffb80073e60 0x7ffb801e6970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:53:10.056 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:10.054+0000 7ffb86000640 1 -- 172.21.15.32:0/3845321196 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffb8015e8a0 con 0x7ffb8015fff0 2024-08-03T20:53:10.056 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:10.054+0000 7ffb86000640 1 --2- 172.21.15.32:0/3845321196 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ffb8015fff0 0x7ffb8015ba10 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7ffb7800e960 tx=0x7ffb7800ee20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:53:10.057 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:10.055+0000 7ffb7d600640 1 -- 172.21.15.32:0/3845321196 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb78004d20 con 0x7ffb8015fff0 2024-08-03T20:53:10.057 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:10.055+0000 7ffb7d600640 1 -- 172.21.15.32:0/3845321196 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7ffb7800cb20 con 0x7ffb8015fff0 2024-08-03T20:53:10.057 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:10.055+0000 7ffb7d600640 1 -- 172.21.15.32:0/3845321196 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb78005020 con 0x7ffb8015fff0 2024-08-03T20:53:10.057 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:10.055+0000 7ffb88000640 1 -- 172.21.15.32:0/3845321196 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffb801f8be0 con 0x7ffb8015fff0 2024-08-03T20:53:10.057 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:10.055+0000 7ffb88000640 1 -- 172.21.15.32:0/3845321196 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffb801f91a0 con 0x7ffb8015fff0 2024-08-03T20:53:10.057 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:10.056+0000 7ffb88000640 1 -- 172.21.15.32:0/3845321196 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ffb8015ea70 con 0x7ffb8015fff0 2024-08-03T20:53:10.057 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:10.057+0000 7ffb7d600640 1 -- 172.21.15.32:0/3845321196 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7ffb780051c0 con 0x7ffb8015fff0 2024-08-03T20:53:10.058 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:10.058+0000 7ffb7d600640 1 --2- 172.21.15.32:0/3845321196 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7ffb60077010 0x7ffb60079520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:53:10.059 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:10.058+0000 7ffb7d600640 1 -- 172.21.15.32:0/3845321196 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(62..62 src has 1..62) ==== 5115+0+0 (secure 0 0 0) 0x7ffb78098e50 con 0x7ffb8015fff0 2024-08-03T20:53:10.059 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:10.059+0000 7ffb85600640 1 --2- 172.21.15.32:0/3845321196 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7ffb60077010 0x7ffb60079520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:53:10.059 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:10.059+0000 7ffb85600640 1 --2- 172.21.15.32:0/3845321196 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7ffb60077010 0x7ffb60079520 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7ffb74033000 tx=0x7ffb74004730 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:53:10.063 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:10.063+0000 7ffb7d600640 1 -- 172.21.15.32:0/3845321196 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7ffb78016030 con 0x7ffb8015fff0 2024-08-03T20:53:10.308 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:10.307+0000 7ffb88000640 1 -- 172.21.15.32:0/3845321196 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ffb80160c30 con 0x7ffb8015fff0 2024-08-03T20:53:10.308 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:10.308+0000 7ffb7d600640 1 -- 172.21.15.32:0/3845321196 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 67 v67) ==== 94+0+3917 (secure 0 0 0) 0x7ffb78061c50 con 0x7ffb8015fff0 2024-08-03T20:53:10.308 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-03T20:53:10.309 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":67,"btime":"2024-08-03T20:53:02:468892+0000","default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5261,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6833/2498840646","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":2498840646},{"type":"v1","addr":"172.21.15.32:6833","nonce":2498840646}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":45},{"gid":5276,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6837/3142148286","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6835","nonce":3142148286},{"type":"v1","addr":"172.21.15.118:6837","nonce":3142148286}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":46},{"gid":5603,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6836/1084110718","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6834","nonce":1084110718},{"type":"v1","addr":"172.21.15.118:6836","nonce":1084110718}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":67},{"gid":6180,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6835/1934242557","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1934242557},{"type":"v1","addr":"172.21.15.32:6835","nonce":1934242557}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":63}],"filesystems":[]} 2024-08-03T20:53:10.309 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 67 2024-08-03T20:53:10.311 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:10.310+0000 7ffb88000640 1 -- 172.21.15.32:0/3845321196 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7ffb60077010 msgr2=0x7ffb60079520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:53:10.311 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:10.310+0000 7ffb88000640 1 --2- 172.21.15.32:0/3845321196 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7ffb60077010 0x7ffb60079520 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7ffb74033000 tx=0x7ffb74004730 comp rx=0 tx=0).stop 2024-08-03T20:53:10.311 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:10.310+0000 7ffb88000640 1 -- 172.21.15.32:0/3845321196 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ffb8015fff0 msgr2=0x7ffb8015ba10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:53:10.311 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:10.310+0000 7ffb88000640 1 --2- 172.21.15.32:0/3845321196 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ffb8015fff0 0x7ffb8015ba10 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7ffb7800e960 tx=0x7ffb7800ee20 comp rx=0 tx=0).stop 2024-08-03T20:53:10.311 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:10.310+0000 7ffb88000640 1 -- 172.21.15.32:0/3845321196 shutdown_connections 2024-08-03T20:53:10.311 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:10.310+0000 7ffb88000640 1 --2- 172.21.15.32:0/3845321196 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7ffb60077010 0x7ffb60079520 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:53:10.311 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:10.310+0000 7ffb88000640 1 --2- 172.21.15.32:0/3845321196 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ffb80073e60 0x7ffb801e6970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:53:10.311 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:10.310+0000 7ffb88000640 1 --2- 172.21.15.32:0/3845321196 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ffb8015fff0 0x7ffb8015ba10 unknown :-1 s=CLOSED pgs=732 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:53:10.311 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:10.310+0000 7ffb88000640 1 --2- 172.21.15.32:0/3845321196 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7ffb8006e930 0x7ffb801436b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:53:10.311 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:10.310+0000 7ffb88000640 1 -- 172.21.15.32:0/3845321196 >> 172.21.15.32:0/3845321196 conn(0x7ffb800b1bd0 msgr2=0x7ffb800cc540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:53:10.311 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:10.310+0000 7ffb88000640 1 -- 172.21.15.32:0/3845321196 shutdown_connections 2024-08-03T20:53:10.312 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:10.310+0000 7ffb88000640 1 -- 172.21.15.32:0/3845321196 wait complete. 2024-08-03T20:53:10.334 INFO:tasks.ceph.mds.a:Restarting daemon 2024-08-03T20:53:10.334 INFO:tasks.ceph.mds.a:Stopping old one... 2024-08-03T20:53:10.334 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-08-03T20:53:10.334 INFO:teuthology.orchestra.run:waiting for 300 2024-08-03T20:53:10.334 INFO:tasks.ceph.mds.c:Restarting daemon 2024-08-03T20:53:10.335 INFO:tasks.ceph.mds.c:Stopping old one... 2024-08-03T20:53:10.335 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-08-03T20:53:10.335 INFO:teuthology.orchestra.run:waiting for 300 2024-08-03T20:53:10.335 INFO:tasks.ceph.mds.b:Restarting daemon 2024-08-03T20:53:10.335 INFO:tasks.ceph.mds.b:Stopping old one... 2024-08-03T20:53:10.335 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-08-03T20:53:10.335 INFO:teuthology.orchestra.run:waiting for 300 2024-08-03T20:53:10.336 INFO:tasks.ceph.mds.d:Restarting daemon 2024-08-03T20:53:10.336 INFO:tasks.ceph.mds.d:Stopping old one... 2024-08-03T20:53:10.336 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-08-03T20:53:10.336 INFO:teuthology.orchestra.run:waiting for 300 2024-08-03T20:53:10.359 INFO:tasks.ceph.mds.b:Stopped 2024-08-03T20:53:10.360 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-08-03T20:53:10.360 INFO:tasks.ceph.mds.d:Stopped 2024-08-03T20:53:10.360 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-08-03T20:53:10.361 INFO:tasks.ceph.mds.a:Stopped 2024-08-03T20:53:10.362 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-08-03T20:53:10.362 INFO:tasks.ceph.mds.c:Stopped 2024-08-03T20:53:10.363 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-08-03T20:53:10.404 INFO:tasks.ceph.mds.b:Started 2024-08-03T20:53:10.404 DEBUG:teuthology.parallel:result is None 2024-08-03T20:53:10.406 INFO:tasks.ceph.mds.a:Started 2024-08-03T20:53:10.407 DEBUG:teuthology.parallel:result is None 2024-08-03T20:53:10.407 INFO:tasks.ceph.mds.d:Started 2024-08-03T20:53:10.407 DEBUG:teuthology.parallel:result is None 2024-08-03T20:53:10.408 INFO:tasks.ceph.mds.c:Started 2024-08-03T20:53:10.409 DEBUG:teuthology.parallel:result is None 2024-08-03T20:53:10.409 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-08-03T20:53:10.603 INFO:tasks.ceph.mds.d.smithi118.stdout:starting mds.d at 2024-08-03T20:53:10.604 INFO:tasks.ceph.mds.b.smithi118.stdout:starting mds.b at 2024-08-03T20:53:10.609 INFO:tasks.ceph.mds.c.smithi032.stdout:starting mds.c at 2024-08-03T20:53:10.609 INFO:tasks.ceph.mds.a.smithi032.stdout:starting mds.a at 2024-08-03T20:53:10.649 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:10.649+0000 7f2717000640 1 -- 172.21.15.32:0/3288532286 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f271015ffa0 msgr2=0x7f2710145290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:53:10.649 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:10.649+0000 7f2717000640 1 --2- 172.21.15.32:0/3288532286 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f271015ffa0 0x7f2710145290 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f2700009860 tx=0x7f270002a5a0 comp rx=0 tx=0).stop 2024-08-03T20:53:10.650 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:10.649+0000 7f2717000640 1 -- 172.21.15.32:0/3288532286 shutdown_connections 2024-08-03T20:53:10.650 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:10.649+0000 7f2717000640 1 --2- 172.21.15.32:0/3288532286 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f271015ffa0 0x7f2710145290 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:53:10.650 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:10.649+0000 7f2717000640 1 -- 172.21.15.32:0/3288532286 >> 172.21.15.32:0/3288532286 conn(0x7f27100b34b0 msgr2=0x7f27100d1050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:53:10.650 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:10.649+0000 7f2717000640 1 -- 172.21.15.32:0/3288532286 shutdown_connections 2024-08-03T20:53:10.650 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:10.649+0000 7f2717000640 1 -- 172.21.15.32:0/3288532286 wait complete. 2024-08-03T20:53:10.651 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:10.651+0000 7f2717000640 1 Processor -- start 2024-08-03T20:53:10.651 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:10.651+0000 7f2717000640 1 -- start start 2024-08-03T20:53:10.653 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:10.652+0000 7f2717000640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f27100cfd30 0x7f2710139520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:53:10.653 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:10.652+0000 7f2717000640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f27100d0140 0x7f271013e6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:53:10.653 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:10.652+0000 7f2717000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f271015ffa0 0x7f271013ec30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:53:10.653 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:10.653+0000 7f2717000640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f2710074730 con 0x7f271015ffa0 2024-08-03T20:53:10.653 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:10.653+0000 7f2717000640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f27100745b0 con 0x7f27100d0140 2024-08-03T20:53:10.654 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:10.653+0000 7f2717000640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7f271006d0f0 con 0x7f27100cfd30 2024-08-03T20:53:10.654 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:10.653+0000 7f2715000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f271015ffa0 0x7f271013ec30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:53:10.654 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:10.653+0000 7f2715000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f271015ffa0 0x7f271013ec30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:45836/0 (socket says 172.21.15.32:45836) 2024-08-03T20:53:10.654 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:10.653+0000 7f2715000640 1 -- 172.21.15.32:0/4060478558 learned_addr learned my addr 172.21.15.32:0/4060478558 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:53:10.655 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:10.653+0000 7f270fe00640 1 --2- 172.21.15.32:0/4060478558 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f27100cfd30 0x7f2710139520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:53:10.655 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:10.653+0000 7f270f400640 1 --2- 172.21.15.32:0/4060478558 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f27100d0140 0x7f271013e6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:53:10.655 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:10.653+0000 7f2715000640 1 -- 172.21.15.32:0/4060478558 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f27100cfd30 msgr2=0x7f2710139520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:53:10.655 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:10.653+0000 7f2715000640 1 --2- 172.21.15.32:0/4060478558 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f27100cfd30 0x7f2710139520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:53:10.656 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:10.653+0000 7f2715000640 1 -- 172.21.15.32:0/4060478558 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f27100d0140 msgr2=0x7f271013e6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:53:10.656 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:10.653+0000 7f2715000640 1 --2- 172.21.15.32:0/4060478558 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f27100d0140 0x7f271013e6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:53:10.656 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:10.653+0000 7f2715000640 1 -- 172.21.15.32:0/4060478558 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f271015ca00 con 0x7f271015ffa0 2024-08-03T20:53:10.656 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:10.653+0000 7f270fe00640 1 --2- 172.21.15.32:0/4060478558 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f27100cfd30 0x7f2710139520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-03T20:53:10.656 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:10.653+0000 7f270f400640 1 --2- 172.21.15.32:0/4060478558 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f27100d0140 0x7f271013e6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-03T20:53:10.656 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:10.654+0000 7f2715000640 1 --2- 172.21.15.32:0/4060478558 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f271015ffa0 0x7f271013ec30 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 crypto rx=0x7f270800cae0 tx=0x7f270800a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:53:10.656 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:10.654+0000 7f2707e00640 1 -- 172.21.15.32:0/4060478558 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2708016020 con 0x7f271015ffa0 2024-08-03T20:53:10.656 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:10.654+0000 7f2717000640 1 -- 172.21.15.32:0/4060478558 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f27101f3940 con 0x7f271015ffa0 2024-08-03T20:53:10.657 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:10.654+0000 7f2707e00640 1 -- 172.21.15.32:0/4060478558 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f270800ad50 con 0x7f271015ffa0 2024-08-03T20:53:10.657 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:10.654+0000 7f2707e00640 1 -- 172.21.15.32:0/4060478558 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2708012680 con 0x7f271015ffa0 2024-08-03T20:53:10.657 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:10.654+0000 7f2717000640 1 -- 172.21.15.32:0/4060478558 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f27101f3f00 con 0x7f271015ffa0 2024-08-03T20:53:10.657 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:10.655+0000 7f2717000640 1 -- 172.21.15.32:0/4060478558 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f27101f4550 con 0x7f271015ffa0 2024-08-03T20:53:10.657 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:10.656+0000 7f2707e00640 1 -- 172.21.15.32:0/4060478558 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f2708019020 con 0x7f271015ffa0 2024-08-03T20:53:10.659 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:10.657+0000 7f2707e00640 1 --2- 172.21.15.32:0/4060478558 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f26f0077010 0x7f26f0079520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:53:10.659 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:10.657+0000 7f2707e00640 1 -- 172.21.15.32:0/4060478558 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(62..62 src has 1..62) ==== 5115+0+0 (secure 0 0 0) 0x7f2708098df0 con 0x7f271015ffa0 2024-08-03T20:53:10.659 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:10.658+0000 7f270fe00640 1 --2- 172.21.15.32:0/4060478558 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f26f0077010 0x7f26f0079520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:53:10.659 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:10.658+0000 7f270fe00640 1 --2- 172.21.15.32:0/4060478558 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f26f0077010 0x7f26f0079520 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f2700034000 tx=0x7f2700004730 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:53:10.662 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:10.662+0000 7f2707e00640 1 -- 172.21.15.32:0/4060478558 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7f2708020020 con 0x7f271015ffa0 2024-08-03T20:53:10.879 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:10.879+0000 7f2717000640 1 -- 172.21.15.32:0/4060478558 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) -- 0x7f27101f36d0 con 0x7f271015ffa0 2024-08-03T20:53:11.524 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:11.524+0000 7f2707e00640 1 -- 172.21.15.32:0/4060478558 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v64) ==== 97+0+0 (secure 0 0 0) 0x7f27080621b0 con 0x7f271015ffa0 2024-08-03T20:53:11.525 INFO:teuthology.orchestra.run.smithi032.stderr: removed all blocklist entries 2024-08-03T20:53:11.528 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:11.527+0000 7f2717000640 1 -- 172.21.15.32:0/4060478558 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f26f0077010 msgr2=0x7f26f0079520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:53:11.528 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:11.527+0000 7f2717000640 1 --2- 172.21.15.32:0/4060478558 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f26f0077010 0x7f26f0079520 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f2700034000 tx=0x7f2700004730 comp rx=0 tx=0).stop 2024-08-03T20:53:11.528 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:11.527+0000 7f2717000640 1 -- 172.21.15.32:0/4060478558 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f271015ffa0 msgr2=0x7f271013ec30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:53:11.528 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:11.527+0000 7f2717000640 1 --2- 172.21.15.32:0/4060478558 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f271015ffa0 0x7f271013ec30 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 crypto rx=0x7f270800cae0 tx=0x7f270800a060 comp rx=0 tx=0).stop 2024-08-03T20:53:11.528 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:11.527+0000 7f2717000640 1 -- 172.21.15.32:0/4060478558 shutdown_connections 2024-08-03T20:53:11.528 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:11.527+0000 7f2717000640 1 --2- 172.21.15.32:0/4060478558 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f26f0077010 0x7f26f0079520 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:53:11.528 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:11.527+0000 7f2717000640 1 --2- 172.21.15.32:0/4060478558 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f27100d0140 0x7f271013e6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:53:11.528 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:11.527+0000 7f2717000640 1 --2- 172.21.15.32:0/4060478558 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f271015ffa0 0x7f271013ec30 unknown :-1 s=CLOSED pgs=738 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:53:11.528 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:11.527+0000 7f2717000640 1 --2- 172.21.15.32:0/4060478558 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f27100cfd30 0x7f2710139520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:53:11.528 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:11.527+0000 7f2717000640 1 -- 172.21.15.32:0/4060478558 >> 172.21.15.32:0/4060478558 conn(0x7f27100b34b0 msgr2=0x7f2710158a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:53:11.528 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:11.527+0000 7f2717000640 1 -- 172.21.15.32:0/4060478558 shutdown_connections 2024-08-03T20:53:11.528 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:11.527+0000 7f2717000640 1 -- 172.21.15.32:0/4060478558 wait complete. 2024-08-03T20:53:11.557 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-08-03T20:53:11.788 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:11.787+0000 7fa62d800640 1 -- 172.21.15.32:0/2054966419 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa62815f610 msgr2=0x7fa6281523f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:53:11.788 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:11.787+0000 7fa62d800640 1 --2- 172.21.15.32:0/2054966419 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa62815f610 0x7fa6281523f0 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 crypto rx=0x7fa6280cd3f0 tx=0x7fa61402a4a0 comp rx=0 tx=0).stop 2024-08-03T20:53:11.789 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:11.788+0000 7fa62d800640 1 -- 172.21.15.32:0/2054966419 shutdown_connections 2024-08-03T20:53:11.789 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:11.788+0000 7fa62d800640 1 --2- 172.21.15.32:0/2054966419 >> v2:172.21.15.118:3300/0 conn(0x7fa6280b4ce0 0x7fa6280cdf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:53:11.789 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:11.788+0000 7fa62d800640 1 --2- 172.21.15.32:0/2054966419 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa62815f610 0x7fa6281523f0 unknown :-1 s=CLOSED pgs=740 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:53:11.789 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:11.788+0000 7fa62d800640 1 -- 172.21.15.32:0/2054966419 >> 172.21.15.32:0/2054966419 conn(0x7fa6280b1bd0 msgr2=0x7fa6280b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:53:11.789 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:11.788+0000 7fa62d800640 1 -- 172.21.15.32:0/2054966419 shutdown_connections 2024-08-03T20:53:11.789 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:11.788+0000 7fa62d800640 1 -- 172.21.15.32:0/2054966419 wait complete. 2024-08-03T20:53:11.790 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:11.790+0000 7fa62d800640 1 Processor -- start 2024-08-03T20:53:11.790 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:11.790+0000 7fa62d800640 1 -- start start 2024-08-03T20:53:11.793 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:11.791+0000 7fa62d800640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fa6280b38b0 0x7fa62814e620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:53:11.793 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:11.791+0000 7fa62d800640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa6280b4ce0 0x7fa628155fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:53:11.793 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:11.792+0000 7fa62d800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa62814f140 0x7fa628156510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:53:11.794 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:11.792+0000 7fa62d800640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fa6280cea30 con 0x7fa62814f140 2024-08-03T20:53:11.794 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:11.792+0000 7fa62d800640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7fa6280ce8b0 con 0x7fa6280b4ce0 2024-08-03T20:53:11.794 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:11.792+0000 7fa62d800640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7fa6280cebb0 con 0x7fa6280b38b0 2024-08-03T20:53:11.794 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:11.792+0000 7fa627400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa62814f140 0x7fa628156510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:53:11.794 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:11.792+0000 7fa627400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa62814f140 0x7fa628156510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:45854/0 (socket says 172.21.15.32:45854) 2024-08-03T20:53:11.794 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:11.792+0000 7fa627400640 1 -- 172.21.15.32:0/1229252786 learned_addr learned my addr 172.21.15.32:0/1229252786 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:53:11.794 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:11.792+0000 7fa626000640 1 --2- 172.21.15.32:0/1229252786 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa6280b4ce0 0x7fa628155fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:53:11.794 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:11.792+0000 7fa626a00640 1 --2- 172.21.15.32:0/1229252786 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fa6280b38b0 0x7fa62814e620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:53:11.794 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:11.792+0000 7fa627400640 1 -- 172.21.15.32:0/1229252786 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fa6280b38b0 msgr2=0x7fa62814e620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:53:11.794 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:11.792+0000 7fa627400640 1 --2- 172.21.15.32:0/1229252786 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fa6280b38b0 0x7fa62814e620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:53:11.794 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:11.792+0000 7fa627400640 1 -- 172.21.15.32:0/1229252786 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa6280b4ce0 msgr2=0x7fa628155fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:53:11.794 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:11.792+0000 7fa627400640 1 --2- 172.21.15.32:0/1229252786 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa6280b4ce0 0x7fa628155fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:53:11.795 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:11.792+0000 7fa627400640 1 -- 172.21.15.32:0/1229252786 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa628152c40 con 0x7fa62814f140 2024-08-03T20:53:11.795 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:11.792+0000 7fa626000640 1 --2- 172.21.15.32:0/1229252786 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa6280b4ce0 0x7fa628155fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-03T20:53:11.795 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:11.793+0000 7fa627400640 1 --2- 172.21.15.32:0/1229252786 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa62814f140 0x7fa628156510 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 crypto rx=0x7fa62000d9e0 tx=0x7fa62000dea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:53:11.795 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:11.793+0000 7fa61ea00640 1 -- 172.21.15.32:0/1229252786 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa620017070 con 0x7fa62814f140 2024-08-03T20:53:11.795 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:11.793+0000 7fa61ea00640 1 -- 172.21.15.32:0/1229252786 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fa62000be30 con 0x7fa62814f140 2024-08-03T20:53:11.795 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:11.793+0000 7fa62d800640 1 -- 172.21.15.32:0/1229252786 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa628152f20 con 0x7fa62814f140 2024-08-03T20:53:11.795 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:11.793+0000 7fa61ea00640 1 -- 172.21.15.32:0/1229252786 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa620003bb0 con 0x7fa62814f140 2024-08-03T20:53:11.795 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:11.793+0000 7fa62d800640 1 -- 172.21.15.32:0/1229252786 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa6281534e0 con 0x7fa62814f140 2024-08-03T20:53:11.796 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:11.794+0000 7fa62d800640 1 -- 172.21.15.32:0/1229252786 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa6280cf0d0 con 0x7fa62814f140 2024-08-03T20:53:11.796 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:11.795+0000 7fa61ea00640 1 -- 172.21.15.32:0/1229252786 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7fa620010460 con 0x7fa62814f140 2024-08-03T20:53:11.797 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:11.796+0000 7fa61ea00640 1 --2- 172.21.15.32:0/1229252786 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fa608077010 0x7fa608079520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:53:11.797 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:11.796+0000 7fa61ea00640 1 -- 172.21.15.32:0/1229252786 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(66..66 src has 1..66) ==== 4857+0+0 (secure 0 0 0) 0x7fa620098a20 con 0x7fa62814f140 2024-08-03T20:53:11.797 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:11.797+0000 7fa626a00640 1 --2- 172.21.15.32:0/1229252786 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fa608077010 0x7fa608079520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:53:11.798 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:11.798+0000 7fa626a00640 1 --2- 172.21.15.32:0/1229252786 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fa608077010 0x7fa608079520 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7fa618001610 tx=0x7fa618009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:53:11.801 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:11.801+0000 7fa61ea00640 1 -- 172.21.15.32:0/1229252786 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7fa620061ed0 con 0x7fa62814f140 2024-08-03T20:53:12.081 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:12.080+0000 7fa62d800640 1 -- 172.21.15.32:0/1229252786 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) -- 0x7fa6280af6b0 con 0x7fa62814f140 2024-08-03T20:53:12.081 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:12.081+0000 7fa61ea00640 1 -- 172.21.15.32:0/1229252786 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v13) ==== 80+0+5304 (secure 0 0 0) 0x7fa6200617d0 con 0x7fa62814f140 2024-08-03T20:53:12.082 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-03T20:53:12.082 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-08-03T20:53:12.082 INFO:teuthology.orchestra.run.smithi032.stdout: "auth_dump": [ 2024-08-03T20:53:12.082 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-03T20:53:12.082 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "mds.a", 2024-08-03T20:53:12.082 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQAvl65mfaKbOhAAUgCH82U0SB7owF5Cds3XYA==", 2024-08-03T20:53:12.082 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2024-08-03T20:53:12.082 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": "allow", 2024-08-03T20:53:12.083 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow *", 2024-08-03T20:53:12.083 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow *", 2024-08-03T20:53:12.083 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2024-08-03T20:53:12.083 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-03T20:53:12.083 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-03T20:53:12.083 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-03T20:53:12.083 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "mds.b", 2024-08-03T20:53:12.083 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQAwl65mNcoVGxAAlRXNjZhxVu8Vjslk9Msrxw==", 2024-08-03T20:53:12.083 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2024-08-03T20:53:12.083 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": "allow", 2024-08-03T20:53:12.083 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow *", 2024-08-03T20:53:12.083 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow *", 2024-08-03T20:53:12.084 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2024-08-03T20:53:12.084 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-03T20:53:12.084 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-03T20:53:12.084 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-03T20:53:12.084 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "mds.c", 2024-08-03T20:53:12.084 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQAwl65mEUAFDRAAGdw1cymfvfKxGeJS/Od6Rw==", 2024-08-03T20:53:12.084 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2024-08-03T20:53:12.084 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": "allow", 2024-08-03T20:53:12.084 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow *", 2024-08-03T20:53:12.084 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow *", 2024-08-03T20:53:12.084 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2024-08-03T20:53:12.085 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-03T20:53:12.085 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-03T20:53:12.085 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-03T20:53:12.085 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "mds.d", 2024-08-03T20:53:12.085 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQAwl65mjUHZKBAAqOCnIopBK8Rhk36hrlNnzw==", 2024-08-03T20:53:12.085 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2024-08-03T20:53:12.085 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": "allow", 2024-08-03T20:53:12.085 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow *", 2024-08-03T20:53:12.085 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow *", 2024-08-03T20:53:12.085 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2024-08-03T20:53:12.085 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-03T20:53:12.085 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-03T20:53:12.085 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-03T20:53:12.085 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "osd.0", 2024-08-03T20:53:12.086 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQAyl65m4kIyLxAAdCh0D2MjJNPPu5b7jNkrbw==", 2024-08-03T20:53:12.086 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2024-08-03T20:53:12.086 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow profile osd", 2024-08-03T20:53:12.086 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow profile osd", 2024-08-03T20:53:12.086 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2024-08-03T20:53:12.086 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-03T20:53:12.086 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-03T20:53:12.086 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-03T20:53:12.086 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "osd.1", 2024-08-03T20:53:12.086 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQA1l65mLzVOOhAA081V1cUyYFPz17L0nny7gA==", 2024-08-03T20:53:12.086 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2024-08-03T20:53:12.086 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow profile osd", 2024-08-03T20:53:12.087 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow profile osd", 2024-08-03T20:53:12.087 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2024-08-03T20:53:12.087 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-03T20:53:12.087 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-03T20:53:12.087 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-03T20:53:12.087 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "osd.2", 2024-08-03T20:53:12.087 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQA5l65mBnXDBxAA9/lZVkwD2ykMTOIysoYODg==", 2024-08-03T20:53:12.087 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2024-08-03T20:53:12.087 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow profile osd", 2024-08-03T20:53:12.087 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow profile osd", 2024-08-03T20:53:12.087 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2024-08-03T20:53:12.087 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-03T20:53:12.087 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-03T20:53:12.088 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-03T20:53:12.088 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "osd.3", 2024-08-03T20:53:12.088 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQA8l65mAriiERAAfzV8An67+PnA3EzeAutIbA==", 2024-08-03T20:53:12.088 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2024-08-03T20:53:12.088 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow profile osd", 2024-08-03T20:53:12.088 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow profile osd", 2024-08-03T20:53:12.088 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2024-08-03T20:53:12.088 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-03T20:53:12.088 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-03T20:53:12.089 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-03T20:53:12.089 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "osd.4", 2024-08-03T20:53:12.089 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQBAl65m5uJSJRAA8S2IjJpxuqHY8YK4tTDMPg==", 2024-08-03T20:53:12.089 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2024-08-03T20:53:12.089 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow profile osd", 2024-08-03T20:53:12.089 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow profile osd", 2024-08-03T20:53:12.089 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2024-08-03T20:53:12.089 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-03T20:53:12.089 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-03T20:53:12.089 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-03T20:53:12.089 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "osd.5", 2024-08-03T20:53:12.089 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQBDl65mBP3JLBAADgK2X3zNyfnjjZADX1PFcA==", 2024-08-03T20:53:12.089 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2024-08-03T20:53:12.090 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow profile osd", 2024-08-03T20:53:12.090 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow profile osd", 2024-08-03T20:53:12.090 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2024-08-03T20:53:12.090 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-03T20:53:12.090 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-03T20:53:12.090 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-03T20:53:12.090 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "osd.6", 2024-08-03T20:53:12.090 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQBGl65m8DBNNRAANeOUfW70S5JfqFlpuMF9KQ==", 2024-08-03T20:53:12.090 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2024-08-03T20:53:12.090 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow profile osd", 2024-08-03T20:53:12.090 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow profile osd", 2024-08-03T20:53:12.090 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2024-08-03T20:53:12.091 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-03T20:53:12.091 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-03T20:53:12.091 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-03T20:53:12.091 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "osd.7", 2024-08-03T20:53:12.091 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQBKl65mUnOcABAA41wylzC1Ec9Fj01Ke0HhTw==", 2024-08-03T20:53:12.091 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2024-08-03T20:53:12.091 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow profile osd", 2024-08-03T20:53:12.091 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow profile osd", 2024-08-03T20:53:12.091 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2024-08-03T20:53:12.091 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-03T20:53:12.091 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-03T20:53:12.091 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-03T20:53:12.091 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "client.0", 2024-08-03T20:53:12.091 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQAwl65msgO3NRAAHG/WuMTK9c4lDXocXvqWnA==", 2024-08-03T20:53:12.092 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2024-08-03T20:53:12.092 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": "allow", 2024-08-03T20:53:12.092 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow r", 2024-08-03T20:53:12.092 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-08-03T20:53:12.092 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-03T20:53:12.092 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-03T20:53:12.092 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-03T20:53:12.092 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "client.1", 2024-08-03T20:53:12.092 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQAxl65mbe19DxAA+sepc9ucXk2Bqy7C90mfbA==", 2024-08-03T20:53:12.092 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2024-08-03T20:53:12.092 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": "allow", 2024-08-03T20:53:12.092 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow r", 2024-08-03T20:53:12.092 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow rw", 2024-08-03T20:53:12.093 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow rwx" 2024-08-03T20:53:12.093 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-03T20:53:12.093 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-03T20:53:12.093 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-03T20:53:12.093 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "client.2", 2024-08-03T20:53:12.093 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQAxl65mBnbkBBAAjor60WWLTKXidiBMgtKuEA==", 2024-08-03T20:53:12.093 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2024-08-03T20:53:12.093 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": "allow", 2024-08-03T20:53:12.093 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow r", 2024-08-03T20:53:12.093 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow rw", 2024-08-03T20:53:12.093 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow rwx" 2024-08-03T20:53:12.093 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-03T20:53:12.093 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-03T20:53:12.094 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-03T20:53:12.094 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "client.3", 2024-08-03T20:53:12.094 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQAxl65munz8GRAA/cRl3ghwQE42FOizurcW3g==", 2024-08-03T20:53:12.094 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2024-08-03T20:53:12.094 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": "allow", 2024-08-03T20:53:12.094 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow r", 2024-08-03T20:53:12.094 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow rw", 2024-08-03T20:53:12.094 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow rwx" 2024-08-03T20:53:12.094 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-03T20:53:12.094 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-03T20:53:12.094 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-03T20:53:12.094 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "client.admin", 2024-08-03T20:53:12.094 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQAul65mB8VcOhAA105JQ2FX1vhtaGj3Fer8mg==", 2024-08-03T20:53:12.095 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2024-08-03T20:53:12.095 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": "allow *", 2024-08-03T20:53:12.095 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow *", 2024-08-03T20:53:12.095 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow *", 2024-08-03T20:53:12.095 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2024-08-03T20:53:12.095 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-03T20:53:12.095 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-03T20:53:12.095 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-03T20:53:12.095 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "mgr.x", 2024-08-03T20:53:12.095 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQAvl65mUaPKLxAAnll74Gmk8nSuN/HuX6rTxA==", 2024-08-03T20:53:12.095 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2024-08-03T20:53:12.095 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": "allow *", 2024-08-03T20:53:12.095 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow profile mgr", 2024-08-03T20:53:12.095 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2024-08-03T20:53:12.096 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-03T20:53:12.096 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-03T20:53:12.096 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-03T20:53:12.096 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "mgr.y", 2024-08-03T20:53:12.096 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQAvl65m5Gq4JBAA306Ge1zHF3Z0WIzUrset4A==", 2024-08-03T20:53:12.096 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2024-08-03T20:53:12.096 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": "allow *", 2024-08-03T20:53:12.096 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow profile mgr", 2024-08-03T20:53:12.096 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2024-08-03T20:53:12.096 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-03T20:53:12.096 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-03T20:53:12.096 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-03T20:53:12.096 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-08-03T20:53:12.097 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:12.083+0000 7fa62d800640 1 -- 172.21.15.32:0/1229252786 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fa608077010 msgr2=0x7fa608079520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:53:12.097 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:12.083+0000 7fa62d800640 1 --2- 172.21.15.32:0/1229252786 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fa608077010 0x7fa608079520 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7fa618001610 tx=0x7fa618009040 comp rx=0 tx=0).stop 2024-08-03T20:53:12.098 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:12.083+0000 7fa62d800640 1 -- 172.21.15.32:0/1229252786 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa62814f140 msgr2=0x7fa628156510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:53:12.098 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:12.083+0000 7fa62d800640 1 --2- 172.21.15.32:0/1229252786 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa62814f140 0x7fa628156510 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 crypto rx=0x7fa62000d9e0 tx=0x7fa62000dea0 comp rx=0 tx=0).stop 2024-08-03T20:53:12.098 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:12.083+0000 7fa62d800640 1 -- 172.21.15.32:0/1229252786 shutdown_connections 2024-08-03T20:53:12.098 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:12.083+0000 7fa62d800640 1 --2- 172.21.15.32:0/1229252786 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fa608077010 0x7fa608079520 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:53:12.098 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:12.083+0000 7fa62d800640 1 --2- 172.21.15.32:0/1229252786 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa6280b4ce0 0x7fa628155fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:53:12.098 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:12.083+0000 7fa62d800640 1 --2- 172.21.15.32:0/1229252786 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa62814f140 0x7fa628156510 unknown :-1 s=CLOSED pgs=741 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:53:12.098 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:12.083+0000 7fa62d800640 1 --2- 172.21.15.32:0/1229252786 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fa6280b38b0 0x7fa62814e620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:53:12.098 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:12.083+0000 7fa62d800640 1 -- 172.21.15.32:0/1229252786 >> 172.21.15.32:0/1229252786 conn(0x7fa6280b1bd0 msgr2=0x7fa628150cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:53:12.098 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:12.083+0000 7fa62d800640 1 -- 172.21.15.32:0/1229252786 shutdown_connections 2024-08-03T20:53:12.098 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:12.083+0000 7fa62d800640 1 -- 172.21.15.32:0/1229252786 wait complete. 2024-08-03T20:53:12.110 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-08-03T20:53:12.338 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:12.337+0000 7f27e8c00640 1 -- 172.21.15.32:0/1701721836 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f27e415ffa0 msgr2=0x7f27e4145290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:53:12.338 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:12.337+0000 7f27e8c00640 1 --2- 172.21.15.32:0/1701721836 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f27e415ffa0 0x7f27e4145290 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 crypto rx=0x7f27e40cd480 tx=0x7f27d402a560 comp rx=0 tx=0).stop 2024-08-03T20:53:12.338 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:12.337+0000 7f27e8c00640 1 -- 172.21.15.32:0/1701721836 shutdown_connections 2024-08-03T20:53:12.338 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:12.337+0000 7f27e8c00640 1 --2- 172.21.15.32:0/1701721836 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f27e4073ea0 0x7f27e40742b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:53:12.338 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:12.337+0000 7f27e8c00640 1 --2- 172.21.15.32:0/1701721836 >> v2:172.21.15.118:3300/0 conn(0x7f27e40cfd30 0x7f27e40d0140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:53:12.338 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:12.337+0000 7f27e8c00640 1 --2- 172.21.15.32:0/1701721836 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f27e415ffa0 0x7f27e4145290 unknown :-1 s=CLOSED pgs=742 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:53:12.338 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:12.337+0000 7f27e8c00640 1 -- 172.21.15.32:0/1701721836 >> 172.21.15.32:0/1701721836 conn(0x7f27e40b34b0 msgr2=0x7f27e40d1050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:53:12.339 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:12.337+0000 7f27e8c00640 1 -- 172.21.15.32:0/1701721836 shutdown_connections 2024-08-03T20:53:12.339 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:12.337+0000 7f27e8c00640 1 -- 172.21.15.32:0/1701721836 wait complete. 2024-08-03T20:53:12.339 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:12.339+0000 7f27e8c00640 1 Processor -- start 2024-08-03T20:53:12.340 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:12.339+0000 7f27e8c00640 1 -- start start 2024-08-03T20:53:12.342 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:12.341+0000 7f27e8c00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f27e4073ea0 0x7f27e40b5670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:53:12.342 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:12.341+0000 7f27e8c00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f27e40cfd30 0x7f27e40b0050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:53:12.342 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:12.341+0000 7f27e8c00640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f27e415ffa0 0x7f27e41f89d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:53:12.342 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:12.341+0000 7f27e8c00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f27e406c9e0 con 0x7f27e4073ea0 2024-08-03T20:53:12.342 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:12.341+0000 7f27e8c00640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f27e41459e0 con 0x7f27e40cfd30 2024-08-03T20:53:12.342 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:12.341+0000 7f27e8c00640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7f27e406cb60 con 0x7f27e415ffa0 2024-08-03T20:53:12.343 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:12.341+0000 7f27e1e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f27e4073ea0 0x7f27e40b5670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:53:12.343 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:12.341+0000 7f27e1e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f27e4073ea0 0x7f27e40b5670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:45876/0 (socket says 172.21.15.32:45876) 2024-08-03T20:53:12.343 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:12.341+0000 7f27e1e00640 1 -- 172.21.15.32:0/3433029609 learned_addr learned my addr 172.21.15.32:0/3433029609 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:53:12.343 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:12.341+0000 7f27e2800640 1 --2- 172.21.15.32:0/3433029609 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f27e415ffa0 0x7f27e41f89d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:53:12.343 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:12.341+0000 7f27e1400640 1 --2- 172.21.15.32:0/3433029609 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f27e40cfd30 0x7f27e40b0050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:53:12.343 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:12.341+0000 7f27e1e00640 1 -- 172.21.15.32:0/3433029609 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f27e415ffa0 msgr2=0x7f27e41f89d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:53:12.343 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:12.341+0000 7f27e1e00640 1 --2- 172.21.15.32:0/3433029609 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f27e415ffa0 0x7f27e41f89d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:53:12.343 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:12.341+0000 7f27e1e00640 1 -- 172.21.15.32:0/3433029609 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f27e40cfd30 msgr2=0x7f27e40b0050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:53:12.343 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:12.341+0000 7f27e1e00640 1 --2- 172.21.15.32:0/3433029609 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f27e40cfd30 0x7f27e40b0050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:53:12.343 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:12.342+0000 7f27e1e00640 1 -- 172.21.15.32:0/3433029609 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f27e413ee10 con 0x7f27e4073ea0 2024-08-03T20:53:12.343 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:12.342+0000 7f27e1400640 1 --2- 172.21.15.32:0/3433029609 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f27e40cfd30 0x7f27e40b0050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-03T20:53:12.343 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:12.342+0000 7f27e1e00640 1 --2- 172.21.15.32:0/3433029609 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f27e4073ea0 0x7f27e40b5670 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 crypto rx=0x7f27d4034040 tx=0x7f27d40038a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:53:12.344 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:12.342+0000 7f27da000640 1 -- 172.21.15.32:0/3433029609 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27d403e030 con 0x7f27e4073ea0 2024-08-03T20:53:12.344 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:12.342+0000 7f27da000640 1 -- 172.21.15.32:0/3433029609 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f27d4003ca0 con 0x7f27e4073ea0 2024-08-03T20:53:12.344 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:12.342+0000 7f27da000640 1 -- 172.21.15.32:0/3433029609 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27d4035040 con 0x7f27e4073ea0 2024-08-03T20:53:12.344 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:12.342+0000 7f27e8c00640 1 -- 172.21.15.32:0/3433029609 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f27e415cb30 con 0x7f27e4073ea0 2024-08-03T20:53:12.344 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:12.342+0000 7f27e8c00640 1 -- 172.21.15.32:0/3433029609 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f27e41f6270 con 0x7f27e4073ea0 2024-08-03T20:53:12.344 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:12.343+0000 7f27e8c00640 1 -- 172.21.15.32:0/3433029609 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f27e41430e0 con 0x7f27e4073ea0 2024-08-03T20:53:12.345 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:12.345+0000 7f27da000640 1 -- 172.21.15.32:0/3433029609 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f27d4041030 con 0x7f27e4073ea0 2024-08-03T20:53:12.346 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:12.346+0000 7f27da000640 1 --2- 172.21.15.32:0/3433029609 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f27c4077010 0x7f27c4079520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:53:12.346 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:12.346+0000 7f27da000640 1 -- 172.21.15.32:0/3433029609 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(66..66 src has 1..66) ==== 4857+0+0 (secure 0 0 0) 0x7f27d40b8b20 con 0x7f27e4073ea0 2024-08-03T20:53:12.346 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:12.346+0000 7f27e1400640 1 --2- 172.21.15.32:0/3433029609 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f27c4077010 0x7f27c4079520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:53:12.347 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:12.347+0000 7f27e1400640 1 --2- 172.21.15.32:0/3433029609 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f27c4077010 0x7f27c4079520 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f27e4142e10 tx=0x7f27d0005f90 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:53:12.350 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:12.350+0000 7f27da000640 1 -- 172.21.15.32:0/3433029609 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7f27d40820f0 con 0x7f27e4073ea0 2024-08-03T20:53:12.569 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:12.568+0000 7f27e8c00640 1 -- 172.21.15.32:0/3433029609 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) -- 0x7f27e406eb00 con 0x7f27e4073ea0 2024-08-03T20:53:12.569 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:12.569+0000 7f27da000640 1 -- 172.21.15.32:0/3433029609 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v66) ==== 84+0+65 (secure 0 0 0) 0x7f27d408a330 con 0x7f27e4073ea0 2024-08-03T20:53:12.569 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-03T20:53:12.569 INFO:teuthology.orchestra.run.smithi032.stdout:[ 2024-08-03T20:53:12.569 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-03T20:53:12.569 INFO:teuthology.orchestra.run.smithi032.stdout: "poolnum": 1, 2024-08-03T20:53:12.570 INFO:teuthology.orchestra.run.smithi032.stdout: "poolname": ".mgr" 2024-08-03T20:53:12.570 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-03T20:53:12.570 INFO:teuthology.orchestra.run.smithi032.stdout:] 2024-08-03T20:53:12.572 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:12.571+0000 7f27e8c00640 1 -- 172.21.15.32:0/3433029609 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f27c4077010 msgr2=0x7f27c4079520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:53:12.572 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:12.571+0000 7f27e8c00640 1 --2- 172.21.15.32:0/3433029609 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f27c4077010 0x7f27c4079520 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f27e4142e10 tx=0x7f27d0005f90 comp rx=0 tx=0).stop 2024-08-03T20:53:12.572 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:12.571+0000 7f27e8c00640 1 -- 172.21.15.32:0/3433029609 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f27e4073ea0 msgr2=0x7f27e40b5670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:53:12.572 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:12.571+0000 7f27e8c00640 1 --2- 172.21.15.32:0/3433029609 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f27e4073ea0 0x7f27e40b5670 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 crypto rx=0x7f27d4034040 tx=0x7f27d40038a0 comp rx=0 tx=0).stop 2024-08-03T20:53:12.572 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:12.571+0000 7f27e8c00640 1 -- 172.21.15.32:0/3433029609 shutdown_connections 2024-08-03T20:53:12.572 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:12.571+0000 7f27e8c00640 1 --2- 172.21.15.32:0/3433029609 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f27c4077010 0x7f27c4079520 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:53:12.572 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:12.571+0000 7f27e8c00640 1 --2- 172.21.15.32:0/3433029609 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f27e40cfd30 0x7f27e40b0050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:53:12.572 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:12.571+0000 7f27e8c00640 1 --2- 172.21.15.32:0/3433029609 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f27e415ffa0 0x7f27e41f89d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:53:12.572 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:12.571+0000 7f27e8c00640 1 --2- 172.21.15.32:0/3433029609 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f27e4073ea0 0x7f27e40b5670 unknown :-1 s=CLOSED pgs=743 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:53:12.573 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:12.571+0000 7f27e8c00640 1 -- 172.21.15.32:0/3433029609 >> 172.21.15.32:0/3433029609 conn(0x7f27e40b34b0 msgr2=0x7f27e40b22e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:53:12.573 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:12.571+0000 7f27e8c00640 1 -- 172.21.15.32:0/3433029609 shutdown_connections 2024-08-03T20:53:12.573 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:12.571+0000 7f27e8c00640 1 -- 172.21.15.32:0/3433029609 wait complete. 2024-08-03T20:53:12.594 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-08-03T20:53:12.594 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-08-03T20:53:12.827 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:12.826+0000 7f4d02400640 1 -- 172.21.15.32:0/1005980247 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f4cfc1609d0 msgr2=0x7f4cfc0cc050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:53:12.827 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:12.826+0000 7f4d02400640 1 --2- 172.21.15.32:0/1005980247 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f4cfc1609d0 0x7f4cfc0cc050 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f4cf0005fd0 tx=0x7f4cf002a5d0 comp rx=0 tx=0).stop 2024-08-03T20:53:12.827 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:12.826+0000 7f4d02400640 1 -- 172.21.15.32:0/1005980247 shutdown_connections 2024-08-03T20:53:12.827 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:12.826+0000 7f4d02400640 1 --2- 172.21.15.32:0/1005980247 >> v2:172.21.15.118:3300/0 conn(0x7f4cfc159ff0 0x7f4cfc15fe10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:53:12.827 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:12.826+0000 7f4d02400640 1 --2- 172.21.15.32:0/1005980247 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f4cfc1609d0 0x7f4cfc0cc050 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:53:12.828 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:12.826+0000 7f4d02400640 1 -- 172.21.15.32:0/1005980247 >> 172.21.15.32:0/1005980247 conn(0x7f4cfc0cfd80 msgr2=0x7f4cfc0d0190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:53:12.828 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:12.826+0000 7f4d02400640 1 -- 172.21.15.32:0/1005980247 shutdown_connections 2024-08-03T20:53:12.828 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:12.826+0000 7f4d02400640 1 -- 172.21.15.32:0/1005980247 wait complete. 2024-08-03T20:53:12.829 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:12.829+0000 7f4d02400640 1 Processor -- start 2024-08-03T20:53:12.829 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:12.829+0000 7f4d02400640 1 -- start start 2024-08-03T20:53:12.831 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:12.830+0000 7f4d02400640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f4cfc159ff0 0x7f4cfc14f720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:53:12.832 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:12.830+0000 7f4d02400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4cfc160350 0x7f4cfc0b37a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:53:12.832 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:12.830+0000 7f4d02400640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f4cfc1609d0 0x7f4cfc0b3ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:53:12.832 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:12.830+0000 7f4d02400640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f4cfc15d0b0 con 0x7f4cfc160350 2024-08-03T20:53:12.832 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:12.830+0000 7f4d02400640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f4cfc15cf30 con 0x7f4cfc159ff0 2024-08-03T20:53:12.832 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:12.830+0000 7f4d02400640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7f4cfc15d230 con 0x7f4cfc1609d0 2024-08-03T20:53:12.832 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:12.830+0000 7f4cfbe00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4cfc160350 0x7f4cfc0b37a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:53:12.832 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:12.830+0000 7f4cfbe00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4cfc160350 0x7f4cfc0b37a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:45880/0 (socket says 172.21.15.32:45880) 2024-08-03T20:53:12.832 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:12.830+0000 7f4cfbe00640 1 -- 172.21.15.32:0/4056163281 learned_addr learned my addr 172.21.15.32:0/4056163281 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:53:12.832 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:12.831+0000 7f4d01000640 1 --2- 172.21.15.32:0/4056163281 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f4cfc159ff0 0x7f4cfc14f720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:53:12.832 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:12.831+0000 7f4d01a00640 1 --2- 172.21.15.32:0/4056163281 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f4cfc1609d0 0x7f4cfc0b3ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:53:12.832 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:12.831+0000 7f4cfbe00640 1 -- 172.21.15.32:0/4056163281 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f4cfc1609d0 msgr2=0x7f4cfc0b3ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:53:12.832 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:12.831+0000 7f4cfbe00640 1 --2- 172.21.15.32:0/4056163281 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f4cfc1609d0 0x7f4cfc0b3ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:53:12.832 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:12.831+0000 7f4cfbe00640 1 -- 172.21.15.32:0/4056163281 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f4cfc159ff0 msgr2=0x7f4cfc14f720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:53:12.832 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:12.831+0000 7f4cfbe00640 1 --2- 172.21.15.32:0/4056163281 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f4cfc159ff0 0x7f4cfc14f720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:53:12.833 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:12.831+0000 7f4cfbe00640 1 -- 172.21.15.32:0/4056163281 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4cfc0ae7b0 con 0x7f4cfc160350 2024-08-03T20:53:12.833 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:12.831+0000 7f4d01000640 1 --2- 172.21.15.32:0/4056163281 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f4cfc159ff0 0x7f4cfc14f720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-03T20:53:12.833 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:12.831+0000 7f4cfbe00640 1 --2- 172.21.15.32:0/4056163281 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4cfc160350 0x7f4cfc0b37a0 secure :-1 s=READY pgs=744 cs=0 l=1 rev1=1 crypto rx=0x7f4ce800ea20 tx=0x7f4ce800eee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:53:12.833 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:12.831+0000 7f4cf8c00640 1 -- 172.21.15.32:0/4056163281 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ce800ce40 con 0x7f4cfc160350 2024-08-03T20:53:12.834 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:12.832+0000 7f4cf8c00640 1 -- 172.21.15.32:0/4056163281 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f4ce8004020 con 0x7f4cfc160350 2024-08-03T20:53:12.834 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:12.832+0000 7f4cf8c00640 1 -- 172.21.15.32:0/4056163281 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ce8004470 con 0x7f4cfc160350 2024-08-03T20:53:12.834 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:12.832+0000 7f4d02400640 1 -- 172.21.15.32:0/4056163281 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4cfc0aea40 con 0x7f4cfc160350 2024-08-03T20:53:12.834 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:12.832+0000 7f4d02400640 1 -- 172.21.15.32:0/4056163281 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4cfc0af000 con 0x7f4cfc160350 2024-08-03T20:53:12.834 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:12.833+0000 7f4d02400640 1 -- 172.21.15.32:0/4056163281 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4cfc15fe10 con 0x7f4cfc160350 2024-08-03T20:53:12.834 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:12.834+0000 7f4cf8c00640 1 -- 172.21.15.32:0/4056163281 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f4ce80035f0 con 0x7f4cfc160350 2024-08-03T20:53:12.836 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:12.836+0000 7f4cf8c00640 1 --2- 172.21.15.32:0/4056163281 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f4cdc077040 0x7f4cdc079550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:53:12.836 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:12.836+0000 7f4cf8c00640 1 -- 172.21.15.32:0/4056163281 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(66..66 src has 1..66) ==== 4857+0+0 (secure 0 0 0) 0x7f4ce8098dd0 con 0x7f4cfc160350 2024-08-03T20:53:12.836 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:12.836+0000 7f4d01000640 1 --2- 172.21.15.32:0/4056163281 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f4cdc077040 0x7f4cdc079550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:53:12.837 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:12.837+0000 7f4d01000640 1 --2- 172.21.15.32:0/4056163281 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f4cdc077040 0x7f4cdc079550 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f4cf0033000 tx=0x7f4cf0004730 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:53:12.840 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:12.840+0000 7f4cf8c00640 1 -- 172.21.15.32:0/4056163281 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7f4ce8061c30 con 0x7f4cfc160350 2024-08-03T20:53:13.042 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:13.042+0000 7f4d02400640 1 -- 172.21.15.32:0/4056163281 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) -- 0x7f4cfc0d09e0 con 0x7f4cfc160350 2024-08-03T20:53:13.539 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:13.538+0000 7f4cf8c00640 1 -- 172.21.15.32:0/4056163281 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v67) ==== 138+0+0 (secure 0 0 0) 0x7f4ce8066180 con 0x7f4cfc160350 2024-08-03T20:53:13.539 INFO:teuthology.orchestra.run.smithi032.stderr:pool 'cephfs_metadata' created 2024-08-03T20:53:13.542 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:13.541+0000 7f4d02400640 1 -- 172.21.15.32:0/4056163281 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f4cdc077040 msgr2=0x7f4cdc079550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:53:13.543 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:13.541+0000 7f4d02400640 1 --2- 172.21.15.32:0/4056163281 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f4cdc077040 0x7f4cdc079550 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f4cf0033000 tx=0x7f4cf0004730 comp rx=0 tx=0).stop 2024-08-03T20:53:13.543 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:13.542+0000 7f4d02400640 1 -- 172.21.15.32:0/4056163281 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4cfc160350 msgr2=0x7f4cfc0b37a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:53:13.543 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:13.542+0000 7f4d02400640 1 --2- 172.21.15.32:0/4056163281 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4cfc160350 0x7f4cfc0b37a0 secure :-1 s=READY pgs=744 cs=0 l=1 rev1=1 crypto rx=0x7f4ce800ea20 tx=0x7f4ce800eee0 comp rx=0 tx=0).stop 2024-08-03T20:53:13.543 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:13.542+0000 7f4d02400640 1 -- 172.21.15.32:0/4056163281 shutdown_connections 2024-08-03T20:53:13.543 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:13.542+0000 7f4d02400640 1 --2- 172.21.15.32:0/4056163281 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f4cdc077040 0x7f4cdc079550 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:53:13.543 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:13.542+0000 7f4d02400640 1 --2- 172.21.15.32:0/4056163281 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f4cfc1609d0 0x7f4cfc0b3ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:53:13.543 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:13.542+0000 7f4d02400640 1 --2- 172.21.15.32:0/4056163281 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4cfc160350 0x7f4cfc0b37a0 unknown :-1 s=CLOSED pgs=744 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:53:13.543 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:13.542+0000 7f4d02400640 1 --2- 172.21.15.32:0/4056163281 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f4cfc159ff0 0x7f4cfc14f720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:53:13.543 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:13.542+0000 7f4d02400640 1 -- 172.21.15.32:0/4056163281 >> 172.21.15.32:0/4056163281 conn(0x7f4cfc0cfd80 msgr2=0x7f4cfc0cb7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:53:13.543 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:13.542+0000 7f4d02400640 1 -- 172.21.15.32:0/4056163281 shutdown_connections 2024-08-03T20:53:13.543 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:13.542+0000 7f4d02400640 1 -- 172.21.15.32:0/4056163281 wait complete. 2024-08-03T20:53:13.574 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-08-03T20:53:13.810 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:13.809+0000 7fb681e00640 1 -- 172.21.15.32:0/2607389046 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb67c0b9530 msgr2=0x7fb67c0b3bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:53:13.810 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:13.809+0000 7fb681e00640 1 --2- 172.21.15.32:0/2607389046 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb67c0b9530 0x7fb67c0b3bc0 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 crypto rx=0x7fb670005fd0 tx=0x7fb67002a4b0 comp rx=0 tx=0).stop 2024-08-03T20:53:13.810 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:13.809+0000 7fb681e00640 1 -- 172.21.15.32:0/2607389046 shutdown_connections 2024-08-03T20:53:13.811 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:13.809+0000 7fb681e00640 1 --2- 172.21.15.32:0/2607389046 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fb67c0b4100 0x7fb67c0b4510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:53:13.811 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:13.809+0000 7fb681e00640 1 --2- 172.21.15.32:0/2607389046 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb67c0b9530 0x7fb67c0b3bc0 unknown :-1 s=CLOSED pgs=745 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:53:13.811 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:13.809+0000 7fb681e00640 1 -- 172.21.15.32:0/2607389046 >> 172.21.15.32:0/2607389046 conn(0x7fb67c0d40c0 msgr2=0x7fb67c0d44d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:53:13.811 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:13.810+0000 7fb681e00640 1 -- 172.21.15.32:0/2607389046 shutdown_connections 2024-08-03T20:53:13.811 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:13.810+0000 7fb681e00640 1 -- 172.21.15.32:0/2607389046 wait complete. 2024-08-03T20:53:13.812 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:13.812+0000 7fb681e00640 1 Processor -- start 2024-08-03T20:53:13.812 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:13.812+0000 7fb681e00640 1 -- start start 2024-08-03T20:53:13.814 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:13.813+0000 7fb681e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb67c0b4100 0x7fb67c0a1210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:53:13.814 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:13.813+0000 7fb681e00640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fb67c0b9530 0x7fb67c0a1750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:53:13.814 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:13.813+0000 7fb681e00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb67c0d1ea0 0x7fb67c09de10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:53:13.814 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:13.813+0000 7fb681e00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fb67c0d2b40 con 0x7fb67c0b4100 2024-08-03T20:53:13.814 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:13.813+0000 7fb681e00640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7fb67c0d29c0 con 0x7fb67c0d1ea0 2024-08-03T20:53:13.814 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:13.813+0000 7fb681e00640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7fb67c0d2cc0 con 0x7fb67c0b9530 2024-08-03T20:53:13.814 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:13.813+0000 7fb680a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb67c0b4100 0x7fb67c0a1210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:53:13.814 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:13.814+0000 7fb680a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb67c0b4100 0x7fb67c0a1210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:45902/0 (socket says 172.21.15.32:45902) 2024-08-03T20:53:13.814 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:13.814+0000 7fb680a00640 1 -- 172.21.15.32:0/2224217116 learned_addr learned my addr 172.21.15.32:0/2224217116 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:53:13.816 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:13.814+0000 7fb681400640 1 --2- 172.21.15.32:0/2224217116 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb67c0d1ea0 0x7fb67c09de10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:53:13.816 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:13.814+0000 7fb67be00640 1 --2- 172.21.15.32:0/2224217116 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fb67c0b9530 0x7fb67c0a1750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:53:13.816 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:13.814+0000 7fb680a00640 1 -- 172.21.15.32:0/2224217116 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fb67c0b9530 msgr2=0x7fb67c0a1750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:53:13.816 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:13.814+0000 7fb680a00640 1 --2- 172.21.15.32:0/2224217116 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fb67c0b9530 0x7fb67c0a1750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:53:13.816 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:13.814+0000 7fb680a00640 1 -- 172.21.15.32:0/2224217116 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb67c0d1ea0 msgr2=0x7fb67c09de10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:53:13.816 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:13.814+0000 7fb680a00640 1 --2- 172.21.15.32:0/2224217116 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb67c0d1ea0 0x7fb67c09de10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:53:13.816 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:13.814+0000 7fb680a00640 1 -- 172.21.15.32:0/2224217116 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb67c09fc50 con 0x7fb67c0b4100 2024-08-03T20:53:13.816 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:13.814+0000 7fb681400640 1 --2- 172.21.15.32:0/2224217116 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb67c0d1ea0 0x7fb67c09de10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-03T20:53:13.816 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:13.814+0000 7fb67be00640 1 --2- 172.21.15.32:0/2224217116 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fb67c0b9530 0x7fb67c0a1750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-03T20:53:13.816 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:13.814+0000 7fb680a00640 1 --2- 172.21.15.32:0/2224217116 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb67c0b4100 0x7fb67c0a1210 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 crypto rx=0x7fb670006010 tx=0x7fb6700039e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:53:13.816 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:13.815+0000 7fb678c00640 1 -- 172.21.15.32:0/2224217116 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb67003e030 con 0x7fb67c0b4100 2024-08-03T20:53:13.816 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:13.815+0000 7fb678c00640 1 -- 172.21.15.32:0/2224217116 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fb6700047a0 con 0x7fb67c0b4100 2024-08-03T20:53:13.816 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:13.815+0000 7fb678c00640 1 -- 172.21.15.32:0/2224217116 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb670035040 con 0x7fb67c0b4100 2024-08-03T20:53:13.817 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:13.815+0000 7fb681e00640 1 -- 172.21.15.32:0/2224217116 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb67c163590 con 0x7fb67c0b4100 2024-08-03T20:53:13.817 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:13.815+0000 7fb681e00640 1 -- 172.21.15.32:0/2224217116 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb67c163b50 con 0x7fb67c0b4100 2024-08-03T20:53:13.817 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:13.816+0000 7fb681e00640 1 -- 172.21.15.32:0/2224217116 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb67c0b4510 con 0x7fb67c0b4100 2024-08-03T20:53:13.818 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:13.818+0000 7fb678c00640 1 -- 172.21.15.32:0/2224217116 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7fb670041030 con 0x7fb67c0b4100 2024-08-03T20:53:13.819 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:13.818+0000 7fb678c00640 1 --2- 172.21.15.32:0/2224217116 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fb65c077010 0x7fb65c079520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:53:13.819 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:13.819+0000 7fb678c00640 1 -- 172.21.15.32:0/2224217116 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(67..67 src has 1..67) ==== 5227+0+0 (secure 0 0 0) 0x7fb6700b8fd0 con 0x7fb67c0b4100 2024-08-03T20:53:13.819 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:13.819+0000 7fb67be00640 1 --2- 172.21.15.32:0/2224217116 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fb65c077010 0x7fb65c079520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:53:13.820 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:13.820+0000 7fb67be00640 1 --2- 172.21.15.32:0/2224217116 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fb65c077010 0x7fb65c079520 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7fb67c1604b0 tx=0x7fb668005d10 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:53:13.824 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:13.824+0000 7fb678c00640 1 -- 172.21.15.32:0/2224217116 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7fb6700823a0 con 0x7fb67c0b4100 2024-08-03T20:53:14.030 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:14.030+0000 7fb681e00640 1 -- 172.21.15.32:0/2224217116 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) -- 0x7fb67c0b2ae0 con 0x7fb67c0b4100 2024-08-03T20:53:14.546 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:14.545+0000 7fb678c00640 1 -- 172.21.15.32:0/2224217116 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v68) ==== 170+0+0 (secure 0 0 0) 0x7fb670081ca0 con 0x7fb67c0b4100 2024-08-03T20:53:14.546 INFO:teuthology.orchestra.run.smithi032.stderr:pool 'cephfs_data' created 2024-08-03T20:53:14.550 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:14.549+0000 7fb681e00640 1 -- 172.21.15.32:0/2224217116 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fb65c077010 msgr2=0x7fb65c079520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:53:14.550 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:14.549+0000 7fb681e00640 1 --2- 172.21.15.32:0/2224217116 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fb65c077010 0x7fb65c079520 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7fb67c1604b0 tx=0x7fb668005d10 comp rx=0 tx=0).stop 2024-08-03T20:53:14.550 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:14.549+0000 7fb681e00640 1 -- 172.21.15.32:0/2224217116 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb67c0b4100 msgr2=0x7fb67c0a1210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:53:14.550 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:14.549+0000 7fb681e00640 1 --2- 172.21.15.32:0/2224217116 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb67c0b4100 0x7fb67c0a1210 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 crypto rx=0x7fb670006010 tx=0x7fb6700039e0 comp rx=0 tx=0).stop 2024-08-03T20:53:14.550 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:14.549+0000 7fb681e00640 1 -- 172.21.15.32:0/2224217116 shutdown_connections 2024-08-03T20:53:14.550 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:14.549+0000 7fb681e00640 1 --2- 172.21.15.32:0/2224217116 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fb65c077010 0x7fb65c079520 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:53:14.551 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:14.549+0000 7fb681e00640 1 --2- 172.21.15.32:0/2224217116 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb67c0d1ea0 0x7fb67c09de10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:53:14.551 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:14.549+0000 7fb681e00640 1 --2- 172.21.15.32:0/2224217116 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fb67c0b9530 0x7fb67c0a1750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:53:14.551 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:14.549+0000 7fb681e00640 1 --2- 172.21.15.32:0/2224217116 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb67c0b4100 0x7fb67c0a1210 unknown :-1 s=CLOSED pgs=746 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:53:14.551 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:14.549+0000 7fb681e00640 1 -- 172.21.15.32:0/2224217116 >> 172.21.15.32:0/2224217116 conn(0x7fb67c0d40c0 msgr2=0x7fb67c168630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:53:14.551 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:14.549+0000 7fb681e00640 1 -- 172.21.15.32:0/2224217116 shutdown_connections 2024-08-03T20:53:14.551 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:14.549+0000 7fb681e00640 1 -- 172.21.15.32:0/2224217116 wait complete. 2024-08-03T20:53:14.588 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-08-03T20:53:14.821 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:14.820+0000 7f3f10800640 1 -- 172.21.15.32:0/1127698939 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f3f08075a10 msgr2=0x7f3f081567e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:53:14.821 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:14.820+0000 7f3f10800640 1 --2- 172.21.15.32:0/1127698939 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f3f08075a10 0x7f3f081567e0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f3efc009830 tx=0x7f3efc02a540 comp rx=0 tx=0).stop 2024-08-03T20:53:14.821 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:14.821+0000 7f3f10800640 1 -- 172.21.15.32:0/1127698939 shutdown_connections 2024-08-03T20:53:14.821 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:14.821+0000 7f3f10800640 1 --2- 172.21.15.32:0/1127698939 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f3f08075a10 0x7f3f081567e0 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:53:14.821 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:14.821+0000 7f3f10800640 1 -- 172.21.15.32:0/1127698939 >> 172.21.15.32:0/1127698939 conn(0x7f3f080cd1e0 msgr2=0x7f3f080cd5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:53:14.822 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:14.821+0000 7f3f10800640 1 -- 172.21.15.32:0/1127698939 shutdown_connections 2024-08-03T20:53:14.822 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:14.821+0000 7f3f10800640 1 -- 172.21.15.32:0/1127698939 wait complete. 2024-08-03T20:53:14.823 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:14.823+0000 7f3f10800640 1 Processor -- start 2024-08-03T20:53:14.823 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:14.823+0000 7f3f10800640 1 -- start start 2024-08-03T20:53:14.826 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:14.824+0000 7f3f10800640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f3f0806f600 0x7f3f0814d4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:53:14.826 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:14.824+0000 7f3f10800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3f08075a10 0x7f3f080ae830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:53:14.826 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:14.824+0000 7f3f10800640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f3f080af520 0x7f3f080aed70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:53:14.826 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:14.824+0000 7f3f10800640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f3f08156ee0 con 0x7f3f08075a10 2024-08-03T20:53:14.826 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:14.824+0000 7f3f10800640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f3f0806de40 con 0x7f3f0806f600 2024-08-03T20:53:14.826 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:14.824+0000 7f3f10800640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7f3f08070120 con 0x7f3f080af520 2024-08-03T20:53:14.826 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:14.825+0000 7f3f0d400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3f08075a10 0x7f3f080ae830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:53:14.826 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:14.825+0000 7f3f0d400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3f08075a10 0x7f3f080ae830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:45906/0 (socket says 172.21.15.32:45906) 2024-08-03T20:53:14.826 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:14.825+0000 7f3f0d400640 1 -- 172.21.15.32:0/1210359612 learned_addr learned my addr 172.21.15.32:0/1210359612 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:53:14.826 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:14.825+0000 7f3f0e800640 1 --2- 172.21.15.32:0/1210359612 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f3f080af520 0x7f3f080aed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:53:14.826 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:14.825+0000 7f3f0de00640 1 --2- 172.21.15.32:0/1210359612 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f3f0806f600 0x7f3f0814d4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:53:14.827 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:14.825+0000 7f3f0d400640 1 -- 172.21.15.32:0/1210359612 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f3f080af520 msgr2=0x7f3f080aed70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:53:14.827 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:14.825+0000 7f3f0d400640 1 --2- 172.21.15.32:0/1210359612 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f3f080af520 0x7f3f080aed70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:53:14.827 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:14.825+0000 7f3f0d400640 1 -- 172.21.15.32:0/1210359612 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f3f0806f600 msgr2=0x7f3f0814d4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:53:14.827 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:14.825+0000 7f3f0d400640 1 --2- 172.21.15.32:0/1210359612 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f3f0806f600 0x7f3f0814d4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:53:14.827 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:14.825+0000 7f3f0d400640 1 -- 172.21.15.32:0/1210359612 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3f080ada70 con 0x7f3f08075a10 2024-08-03T20:53:14.827 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:14.826+0000 7f3f0d400640 1 --2- 172.21.15.32:0/1210359612 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3f08075a10 0x7f3f080ae830 secure :-1 s=READY pgs=748 cs=0 l=1 rev1=1 crypto rx=0x7f3ef800c4d0 tx=0x7f3ef800c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:53:14.828 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:14.826+0000 7f3f02000640 1 -- 172.21.15.32:0/1210359612 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ef8016020 con 0x7f3f08075a10 2024-08-03T20:53:14.828 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:14.826+0000 7f3f02000640 1 -- 172.21.15.32:0/1210359612 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f3ef8012920 con 0x7f3f08075a10 2024-08-03T20:53:14.828 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:14.826+0000 7f3f10800640 1 -- 172.21.15.32:0/1210359612 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3f080add50 con 0x7f3f08075a10 2024-08-03T20:53:14.828 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:14.826+0000 7f3f02000640 1 -- 172.21.15.32:0/1210359612 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ef8012d50 con 0x7f3f08075a10 2024-08-03T20:53:14.828 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:14.826+0000 7f3f10800640 1 -- 172.21.15.32:0/1210359612 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3f0815e8e0 con 0x7f3f08075a10 2024-08-03T20:53:14.828 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:14.827+0000 7f3f10800640 1 -- 172.21.15.32:0/1210359612 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3f081567e0 con 0x7f3f08075a10 2024-08-03T20:53:14.829 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:14.828+0000 7f3f02000640 1 -- 172.21.15.32:0/1210359612 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f3ef8019020 con 0x7f3f08075a10 2024-08-03T20:53:14.830 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:14.829+0000 7f3f02000640 1 --2- 172.21.15.32:0/1210359612 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f3eec077010 0x7f3eec079520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:53:14.830 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:14.830+0000 7f3f02000640 1 -- 172.21.15.32:0/1210359612 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(68..68 src has 1..68) ==== 5609+0+0 (secure 0 0 0) 0x7f3ef8098a20 con 0x7f3f08075a10 2024-08-03T20:53:14.830 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:14.830+0000 7f3f0de00640 1 --2- 172.21.15.32:0/1210359612 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f3eec077010 0x7f3eec079520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:53:14.831 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:14.831+0000 7f3f0de00640 1 --2- 172.21.15.32:0/1210359612 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f3eec077010 0x7f3eec079520 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f3efc034000 tx=0x7f3efc033040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:53:14.835 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:14.834+0000 7f3f02000640 1 -- 172.21.15.32:0/1210359612 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7f3ef8020020 con 0x7f3f08075a10 2024-08-03T20:53:15.080 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:15.079+0000 7f3f10800640 1 -- 172.21.15.32:0/1210359612 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) -- 0x7f3f081f99b0 con 0x7f3f08075a10 2024-08-03T20:53:15.548 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:15.548+0000 7f3f02000640 1 -- 172.21.15.32:0/1210359612 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 10 and data pool 11 v70) ==== 174+0+0 (secure 0 0 0) 0x7f3ef8061be0 con 0x7f3f08075a10 2024-08-03T20:53:15.549 INFO:teuthology.orchestra.run.smithi032.stderr:new fs with metadata pool 10 and data pool 11 2024-08-03T20:53:15.551 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:15.550+0000 7f3f10800640 1 -- 172.21.15.32:0/1210359612 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f3eec077010 msgr2=0x7f3eec079520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:53:15.551 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:15.550+0000 7f3f10800640 1 --2- 172.21.15.32:0/1210359612 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f3eec077010 0x7f3eec079520 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f3efc034000 tx=0x7f3efc033040 comp rx=0 tx=0).stop 2024-08-03T20:53:15.551 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:15.551+0000 7f3f10800640 1 -- 172.21.15.32:0/1210359612 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3f08075a10 msgr2=0x7f3f080ae830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:53:15.551 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:15.551+0000 7f3f10800640 1 --2- 172.21.15.32:0/1210359612 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3f08075a10 0x7f3f080ae830 secure :-1 s=READY pgs=748 cs=0 l=1 rev1=1 crypto rx=0x7f3ef800c4d0 tx=0x7f3ef800c990 comp rx=0 tx=0).stop 2024-08-03T20:53:15.552 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:15.551+0000 7f3f10800640 1 -- 172.21.15.32:0/1210359612 shutdown_connections 2024-08-03T20:53:15.552 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:15.551+0000 7f3f10800640 1 --2- 172.21.15.32:0/1210359612 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f3eec077010 0x7f3eec079520 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:53:15.552 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:15.551+0000 7f3f10800640 1 --2- 172.21.15.32:0/1210359612 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f3f080af520 0x7f3f080aed70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:53:15.552 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:15.551+0000 7f3f10800640 1 --2- 172.21.15.32:0/1210359612 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3f08075a10 0x7f3f080ae830 unknown :-1 s=CLOSED pgs=748 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:53:15.552 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:15.551+0000 7f3f10800640 1 --2- 172.21.15.32:0/1210359612 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f3f0806f600 0x7f3f0814d4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:53:15.552 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:15.551+0000 7f3f10800640 1 -- 172.21.15.32:0/1210359612 >> 172.21.15.32:0/1210359612 conn(0x7f3f080cd1e0 msgr2=0x7f3f080b1b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:53:15.552 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:15.551+0000 7f3f10800640 1 -- 172.21.15.32:0/1210359612 shutdown_connections 2024-08-03T20:53:15.552 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:15.551+0000 7f3f10800640 1 -- 172.21.15.32:0/1210359612 wait complete. 2024-08-03T20:53:15.598 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-08-03T20:53:15.828 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:15.827+0000 7f376fe00640 1 -- 172.21.15.32:0/1529633192 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3768144f90 msgr2=0x7f3768074570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:53:15.828 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:15.827+0000 7f376fe00640 1 --2- 172.21.15.32:0/1529633192 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3768144f90 0x7f3768074570 secure :-1 s=READY pgs=750 cs=0 l=1 rev1=1 crypto rx=0x7f3768073b50 tx=0x7f375402a5c0 comp rx=0 tx=0).stop 2024-08-03T20:53:15.828 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:15.827+0000 7f376fe00640 1 -- 172.21.15.32:0/1529633192 shutdown_connections 2024-08-03T20:53:15.828 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:15.827+0000 7f376fe00640 1 --2- 172.21.15.32:0/1529633192 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3768144f90 0x7f3768074570 unknown :-1 s=CLOSED pgs=750 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:53:15.828 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:15.827+0000 7f376fe00640 1 -- 172.21.15.32:0/1529633192 >> 172.21.15.32:0/1529633192 conn(0x7f37680b1bd0 msgr2=0x7f37680b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:53:15.829 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:15.827+0000 7f376fe00640 1 -- 172.21.15.32:0/1529633192 shutdown_connections 2024-08-03T20:53:15.829 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:15.827+0000 7f376fe00640 1 -- 172.21.15.32:0/1529633192 wait complete. 2024-08-03T20:53:15.829 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:15.829+0000 7f376fe00640 1 Processor -- start 2024-08-03T20:53:15.830 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:15.829+0000 7f376fe00640 1 -- start start 2024-08-03T20:53:15.834 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:15.831+0000 7f376fe00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f376806e930 0x7f37680aeb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:53:15.834 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:15.831+0000 7f376fe00640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f3768144f90 0x7f37680b4a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:53:15.834 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:15.831+0000 7f376fe00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f376815fff0 0x7f376815ebf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:53:15.835 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:15.831+0000 7f376fe00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f37680cf840 con 0x7f376806e930 2024-08-03T20:53:15.835 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:15.831+0000 7f376fe00640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f3768145f30 con 0x7f376815fff0 2024-08-03T20:53:15.835 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:15.831+0000 7f376fe00640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7f37680cf9c0 con 0x7f3768144f90 2024-08-03T20:53:15.835 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:15.831+0000 7f376d400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f376806e930 0x7f37680aeb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:53:15.835 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:15.831+0000 7f376d400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f376806e930 0x7f37680aeb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:45924/0 (socket says 172.21.15.32:45924) 2024-08-03T20:53:15.835 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:15.831+0000 7f376d400640 1 -- 172.21.15.32:0/1348890003 learned_addr learned my addr 172.21.15.32:0/1348890003 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:53:15.836 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:15.831+0000 7f376ca00640 1 --2- 172.21.15.32:0/1348890003 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f3768144f90 0x7f37680b4a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:53:15.836 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:15.831+0000 7f376de00640 1 --2- 172.21.15.32:0/1348890003 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f376815fff0 0x7f376815ebf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:53:15.836 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:15.832+0000 7f376d400640 1 -- 172.21.15.32:0/1348890003 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f3768144f90 msgr2=0x7f37680b4a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:53:15.836 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:15.832+0000 7f376d400640 1 --2- 172.21.15.32:0/1348890003 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f3768144f90 0x7f37680b4a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:53:15.836 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:15.832+0000 7f376d400640 1 -- 172.21.15.32:0/1348890003 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f376815fff0 msgr2=0x7f376815ebf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:53:15.836 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:15.832+0000 7f376d400640 1 --2- 172.21.15.32:0/1348890003 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f376815fff0 0x7f376815ebf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:53:15.836 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:15.832+0000 7f376d400640 1 -- 172.21.15.32:0/1348890003 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f37680ad9f0 con 0x7f376806e930 2024-08-03T20:53:15.836 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:15.832+0000 7f376d400640 1 --2- 172.21.15.32:0/1348890003 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f376806e930 0x7f37680aeb50 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7f375c00c370 tx=0x7f375c00c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:53:15.837 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:15.832+0000 7f3765600640 1 -- 172.21.15.32:0/1348890003 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f375c017070 con 0x7f376806e930 2024-08-03T20:53:15.837 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:15.832+0000 7f3765600640 1 -- 172.21.15.32:0/1348890003 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f375c0035f0 con 0x7f376806e930 2024-08-03T20:53:15.837 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:15.832+0000 7f376fe00640 1 -- 172.21.15.32:0/1348890003 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f37680adcd0 con 0x7f376806e930 2024-08-03T20:53:15.837 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:15.832+0000 7f3765600640 1 -- 172.21.15.32:0/1348890003 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f375c003a40 con 0x7f376806e930 2024-08-03T20:53:15.837 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:15.832+0000 7f376fe00640 1 -- 172.21.15.32:0/1348890003 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f37680ae290 con 0x7f376806e930 2024-08-03T20:53:15.837 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:15.833+0000 7f376fe00640 1 -- 172.21.15.32:0/1348890003 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3768160400 con 0x7f376806e930 2024-08-03T20:53:15.838 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:15.835+0000 7f3765600640 1 -- 172.21.15.32:0/1348890003 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f375c01a070 con 0x7f376806e930 2024-08-03T20:53:15.838 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:15.836+0000 7f3765600640 1 --2- 172.21.15.32:0/1348890003 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f3748077010 0x7f3748079520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:53:15.838 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:15.836+0000 7f3765600640 1 -- 172.21.15.32:0/1348890003 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(69..69 src has 1..69) ==== 5709+0+0 (secure 0 0 0) 0x7f375c097ca0 con 0x7f376806e930 2024-08-03T20:53:15.839 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:15.837+0000 7f376ca00640 1 --2- 172.21.15.32:0/1348890003 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f3748077010 0x7f3748079520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:53:15.839 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:15.837+0000 7f376ca00640 1 --2- 172.21.15.32:0/1348890003 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f3748077010 0x7f3748079520 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f3754034000 tx=0x7f3754004730 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:53:15.841 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:15.841+0000 7f3765600640 1 -- 172.21.15.32:0/1348890003 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7f375c0607b0 con 0x7f376806e930 2024-08-03T20:53:16.045 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:16.044+0000 7f376fe00640 1 -- 172.21.15.32:0/1348890003 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f3768145770 con 0x7f376806e930 2024-08-03T20:53:16.046 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:16.045+0000 7f3765600640 1 -- 172.21.15.32:0/1348890003 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v69) ==== 74+0+16621 (secure 0 0 0) 0x7f375c0690d0 con 0x7f376806e930 2024-08-03T20:53:16.047 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-03T20:53:16.048 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":69,"fsid":"40ab141a-01d7-49b9-ae3c-2edec9876e95","created":"2024-08-03T20:47:14.957362+0000","modified":"2024-08-03T20:53:15.080708+0000","last_up_change":"2024-08-03T20:47:28.874451+0000","last_in_change":"2024-08-03T20:47:20.079442+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-08-03T20:47:28.261300+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs_metadata","create_time":"2024-08-03T20:53:13.054856+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"69","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_data","create_time":"2024-08-03T20:53:14.042324+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"69","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5a83e759-9035-438b-889a-0a6af1f0a0d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6816","nonce":3066156116},{"type":"v1","addr":"172.21.15.32:6817","nonce":3066156116}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6818","nonce":3066156116},{"type":"v1","addr":"172.21.15.32:6819","nonce":3066156116}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6822","nonce":3066156116},{"type":"v1","addr":"172.21.15.32:6823","nonce":3066156116}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6820","nonce":3066156116},{"type":"v1","addr":"172.21.15.32:6821","nonce":3066156116}]},"public_addr":"172.21.15.32:6817/3066156116","cluster_addr":"172.21.15.32:6819/3066156116","heartbeat_back_addr":"172.21.15.32:6823/3066156116","heartbeat_front_addr":"172.21.15.32:6821/3066156116","state":["exists","up"]},{"osd":1,"uuid":"d7c5c210-da4f-49f9-b9e7-1dfb02daa4bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6824","nonce":1551241759},{"type":"v1","addr":"172.21.15.32:6825","nonce":1551241759}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6826","nonce":1551241759},{"type":"v1","addr":"172.21.15.32:6827","nonce":1551241759}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6830","nonce":1551241759},{"type":"v1","addr":"172.21.15.32:6831","nonce":1551241759}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6828","nonce":1551241759},{"type":"v1","addr":"172.21.15.32:6829","nonce":1551241759}]},"public_addr":"172.21.15.32:6825/1551241759","cluster_addr":"172.21.15.32:6827/1551241759","heartbeat_back_addr":"172.21.15.32:6831/1551241759","heartbeat_front_addr":"172.21.15.32:6829/1551241759","state":["exists","up"]},{"osd":2,"uuid":"57f2e6de-a201-45c2-8981-8e56a4875ebd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6808","nonce":3806708944},{"type":"v1","addr":"172.21.15.32:6809","nonce":3806708944}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6810","nonce":3806708944},{"type":"v1","addr":"172.21.15.32:6811","nonce":3806708944}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6814","nonce":3806708944},{"type":"v1","addr":"172.21.15.32:6815","nonce":3806708944}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6812","nonce":3806708944},{"type":"v1","addr":"172.21.15.32:6813","nonce":3806708944}]},"public_addr":"172.21.15.32:6809/3806708944","cluster_addr":"172.21.15.32:6811/3806708944","heartbeat_back_addr":"172.21.15.32:6815/3806708944","heartbeat_front_addr":"172.21.15.32:6813/3806708944","state":["exists","up"]},{"osd":3,"uuid":"6439a280-da45-4148-8130-f16323935ff3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6800","nonce":9088303},{"type":"v1","addr":"172.21.15.32:6801","nonce":9088303}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6802","nonce":9088303},{"type":"v1","addr":"172.21.15.32:6803","nonce":9088303}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6806","nonce":9088303},{"type":"v1","addr":"172.21.15.32:6807","nonce":9088303}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6804","nonce":9088303},{"type":"v1","addr":"172.21.15.32:6805","nonce":9088303}]},"public_addr":"172.21.15.32:6801/9088303","cluster_addr":"172.21.15.32:6803/9088303","heartbeat_back_addr":"172.21.15.32:6807/9088303","heartbeat_front_addr":"172.21.15.32:6805/9088303","state":["exists","up"]},{"osd":4,"uuid":"4f902eff-1e18-418d-82c9-2151c69524fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6802","nonce":864752275},{"type":"v1","addr":"172.21.15.118:6803","nonce":864752275}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6804","nonce":864752275},{"type":"v1","addr":"172.21.15.118:6805","nonce":864752275}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6808","nonce":864752275},{"type":"v1","addr":"172.21.15.118:6809","nonce":864752275}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6806","nonce":864752275},{"type":"v1","addr":"172.21.15.118:6807","nonce":864752275}]},"public_addr":"172.21.15.118:6803/864752275","cluster_addr":"172.21.15.118:6805/864752275","heartbeat_back_addr":"172.21.15.118:6809/864752275","heartbeat_front_addr":"172.21.15.118:6807/864752275","state":["exists","up"]},{"osd":5,"uuid":"bfd9feac-dd06-458c-91b9-af7f175f612c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6810","nonce":2724891770},{"type":"v1","addr":"172.21.15.118:6811","nonce":2724891770}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6812","nonce":2724891770},{"type":"v1","addr":"172.21.15.118:6813","nonce":2724891770}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6816","nonce":2724891770},{"type":"v1","addr":"172.21.15.118:6817","nonce":2724891770}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6814","nonce":2724891770},{"type":"v1","addr":"172.21.15.118:6815","nonce":2724891770}]},"public_addr":"172.21.15.118:6811/2724891770","cluster_addr":"172.21.15.118:6813/2724891770","heartbeat_back_addr":"172.21.15.118:6817/2724891770","heartbeat_front_addr":"172.21.15.118:6815/2724891770","state":["exists","up"]},{"osd":6,"uuid":"5ed5a279-9d8e-416d-bbc7-7edd3e6d5705","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6825","nonce":801947386},{"type":"v1","addr":"172.21.15.118:6827","nonce":801947386}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6828","nonce":801947386},{"type":"v1","addr":"172.21.15.118:6829","nonce":801947386}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6832","nonce":801947386},{"type":"v1","addr":"172.21.15.118:6833","nonce":801947386}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6830","nonce":801947386},{"type":"v1","addr":"172.21.15.118:6831","nonce":801947386}]},"public_addr":"172.21.15.118:6827/801947386","cluster_addr":"172.21.15.118:6829/801947386","heartbeat_back_addr":"172.21.15.118:6833/801947386","heartbeat_front_addr":"172.21.15.118:6831/801947386","state":["exists","up"]},{"osd":7,"uuid":"f2086c47-875e-4451-a436-2008d1ea16af","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6818","nonce":653000048},{"type":"v1","addr":"172.21.15.118:6819","nonce":653000048}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6820","nonce":653000048},{"type":"v1","addr":"172.21.15.118:6821","nonce":653000048}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6824","nonce":653000048},{"type":"v1","addr":"172.21.15.118:6826","nonce":653000048}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6822","nonce":653000048},{"type":"v1","addr":"172.21.15.118:6823","nonce":653000048}]},"public_addr":"172.21.15.118:6819/653000048","cluster_addr":"172.21.15.118:6821/653000048","heartbeat_back_addr":"172.21.15.118:6826/653000048","heartbeat_front_addr":"172.21.15.118:6823/653000048","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-03T20:47:25.491994+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-03T20:47:25.587358+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-03T20:47:25.454175+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-03T20:47:25.377734+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-03T20:47:25.412351+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-03T20:47:25.499815+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-03T20:47:25.575044+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-03T20:47:25.552383+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-08-03T20:53:16.050 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:16.048+0000 7f376fe00640 1 -- 172.21.15.32:0/1348890003 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f3748077010 msgr2=0x7f3748079520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:53:16.050 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:16.048+0000 7f376fe00640 1 --2- 172.21.15.32:0/1348890003 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f3748077010 0x7f3748079520 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f3754034000 tx=0x7f3754004730 comp rx=0 tx=0).stop 2024-08-03T20:53:16.050 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:16.048+0000 7f376fe00640 1 -- 172.21.15.32:0/1348890003 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f376806e930 msgr2=0x7f37680aeb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:53:16.050 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:16.048+0000 7f376fe00640 1 --2- 172.21.15.32:0/1348890003 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f376806e930 0x7f37680aeb50 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7f375c00c370 tx=0x7f375c00c830 comp rx=0 tx=0).stop 2024-08-03T20:53:16.050 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:16.048+0000 7f376fe00640 1 -- 172.21.15.32:0/1348890003 shutdown_connections 2024-08-03T20:53:16.050 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:16.048+0000 7f376fe00640 1 --2- 172.21.15.32:0/1348890003 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f3748077010 0x7f3748079520 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:53:16.051 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:16.048+0000 7f376fe00640 1 --2- 172.21.15.32:0/1348890003 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f376815fff0 0x7f376815ebf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:53:16.051 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:16.048+0000 7f376fe00640 1 --2- 172.21.15.32:0/1348890003 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f3768144f90 0x7f37680b4a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:53:16.051 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:16.048+0000 7f376fe00640 1 --2- 172.21.15.32:0/1348890003 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f376806e930 0x7f37680aeb50 unknown :-1 s=CLOSED pgs=751 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:53:16.051 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:16.048+0000 7f376fe00640 1 -- 172.21.15.32:0/1348890003 >> 172.21.15.32:0/1348890003 conn(0x7f37680b1bd0 msgr2=0x7f37680b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:53:16.051 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:16.048+0000 7f376fe00640 1 -- 172.21.15.32:0/1348890003 shutdown_connections 2024-08-03T20:53:16.051 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:16.048+0000 7f376fe00640 1 -- 172.21.15.32:0/1348890003 wait complete. 2024-08-03T20:53:16.072 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-08-03T20:53:16.300 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:16.300+0000 7fc611200640 1 -- 172.21.15.32:0/4117885668 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc60c0af040 msgr2=0x7fc60c06f480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:53:16.301 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:16.300+0000 7fc611200640 1 --2- 172.21.15.32:0/4117885668 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc60c0af040 0x7fc60c06f480 secure :-1 s=READY pgs=752 cs=0 l=1 rev1=1 crypto rx=0x7fc5f4005fd0 tx=0x7fc5f402a5d0 comp rx=0 tx=0).stop 2024-08-03T20:53:16.301 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:16.300+0000 7fc611200640 1 -- 172.21.15.32:0/4117885668 shutdown_connections 2024-08-03T20:53:16.301 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:16.300+0000 7fc611200640 1 --2- 172.21.15.32:0/4117885668 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc60c0af040 0x7fc60c06f480 unknown :-1 s=CLOSED pgs=752 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:53:16.301 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:16.300+0000 7fc611200640 1 --2- 172.21.15.32:0/4117885668 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fc60c0cee10 0x7fc60c15c0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:53:16.301 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:16.300+0000 7fc611200640 1 -- 172.21.15.32:0/4117885668 >> 172.21.15.32:0/4117885668 conn(0x7fc60c0b2120 msgr2=0x7fc60c0cfd60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:53:16.301 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:16.300+0000 7fc611200640 1 -- 172.21.15.32:0/4117885668 shutdown_connections 2024-08-03T20:53:16.301 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:16.300+0000 7fc611200640 1 -- 172.21.15.32:0/4117885668 wait complete. 2024-08-03T20:53:16.302 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:16.302+0000 7fc611200640 1 Processor -- start 2024-08-03T20:53:16.303 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:16.302+0000 7fc611200640 1 -- start start 2024-08-03T20:53:16.306 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:16.304+0000 7fc611200640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fc60c0af040 0x7fc60c158540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:53:16.306 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:16.304+0000 7fc611200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc60c0af450 0x7fc60c0ce2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:53:16.306 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:16.304+0000 7fc611200640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc60c0cee10 0x7fc60c15a280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:53:16.306 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:16.304+0000 7fc611200640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fc60c06fd50 con 0x7fc60c0af450 2024-08-03T20:53:16.306 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:16.304+0000 7fc611200640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7fc60c06fbd0 con 0x7fc60c0cee10 2024-08-03T20:53:16.306 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:16.304+0000 7fc611200640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7fc60c0caac0 con 0x7fc60c0af040 2024-08-03T20:53:16.307 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:16.304+0000 7fc602a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc60c0af450 0x7fc60c0ce2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:53:16.307 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:16.304+0000 7fc602a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc60c0af450 0x7fc60c0ce2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:45940/0 (socket says 172.21.15.32:45940) 2024-08-03T20:53:16.307 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:16.304+0000 7fc602a00640 1 -- 172.21.15.32:0/2552190031 learned_addr learned my addr 172.21.15.32:0/2552190031 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:53:16.307 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:16.304+0000 7fc60b400640 1 --2- 172.21.15.32:0/2552190031 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fc60c0af040 0x7fc60c158540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:53:16.307 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:16.304+0000 7fc60be00640 1 --2- 172.21.15.32:0/2552190031 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc60c0cee10 0x7fc60c15a280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:53:16.307 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:16.304+0000 7fc602a00640 1 -- 172.21.15.32:0/2552190031 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fc60c0af040 msgr2=0x7fc60c158540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:53:16.307 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:16.304+0000 7fc602a00640 1 --2- 172.21.15.32:0/2552190031 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fc60c0af040 0x7fc60c158540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:53:16.308 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:16.304+0000 7fc602a00640 1 -- 172.21.15.32:0/2552190031 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc60c0cee10 msgr2=0x7fc60c15a280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:53:16.308 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:16.304+0000 7fc602a00640 1 --2- 172.21.15.32:0/2552190031 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc60c0cee10 0x7fc60c15a280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:53:16.308 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:16.304+0000 7fc602a00640 1 -- 172.21.15.32:0/2552190031 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc60c0ca150 con 0x7fc60c0af450 2024-08-03T20:53:16.308 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:16.304+0000 7fc60be00640 1 --2- 172.21.15.32:0/2552190031 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc60c0cee10 0x7fc60c15a280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-03T20:53:16.308 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:16.305+0000 7fc602a00640 1 --2- 172.21.15.32:0/2552190031 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc60c0af450 0x7fc60c0ce2d0 secure :-1 s=READY pgs=753 cs=0 l=1 rev1=1 crypto rx=0x7fc5f4000c00 tx=0x7fc5f40035f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:53:16.308 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:16.305+0000 7fc603e00640 1 -- 172.21.15.32:0/2552190031 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5f403e030 con 0x7fc60c0af450 2024-08-03T20:53:16.308 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:16.305+0000 7fc603e00640 1 -- 172.21.15.32:0/2552190031 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fc5f4003ae0 con 0x7fc60c0af450 2024-08-03T20:53:16.308 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:16.305+0000 7fc603e00640 1 -- 172.21.15.32:0/2552190031 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5f4035040 con 0x7fc60c0af450 2024-08-03T20:53:16.309 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:16.305+0000 7fc611200640 1 -- 172.21.15.32:0/2552190031 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc60c0c8300 con 0x7fc60c0af450 2024-08-03T20:53:16.309 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:16.305+0000 7fc611200640 1 -- 172.21.15.32:0/2552190031 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc60c0c8850 con 0x7fc60c0af450 2024-08-03T20:53:16.310 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:16.306+0000 7fc611200640 1 -- 172.21.15.32:0/2552190031 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc60c06f480 con 0x7fc60c0af450 2024-08-03T20:53:16.310 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:16.307+0000 7fc603e00640 1 -- 172.21.15.32:0/2552190031 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7fc5f4041030 con 0x7fc60c0af450 2024-08-03T20:53:16.310 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:16.309+0000 7fc603e00640 1 --2- 172.21.15.32:0/2552190031 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fc5e8077010 0x7fc5e8079520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:53:16.310 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:16.309+0000 7fc603e00640 1 -- 172.21.15.32:0/2552190031 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(69..69 src has 1..69) ==== 5709+0+0 (secure 0 0 0) 0x7fc5f40b8eb0 con 0x7fc60c0af450 2024-08-03T20:53:16.310 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:16.309+0000 7fc60b400640 1 --2- 172.21.15.32:0/2552190031 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fc5e8077010 0x7fc5e8079520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:53:16.310 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:16.310+0000 7fc60b400640 1 --2- 172.21.15.32:0/2552190031 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fc5e8077010 0x7fc5e8079520 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7fc5fc009ac0 tx=0x7fc5fc005f90 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:53:16.314 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:16.313+0000 7fc603e00640 1 -- 172.21.15.32:0/2552190031 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7fc5f4082010 con 0x7fc60c0af450 2024-08-03T20:53:16.518 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:16.518+0000 7fc611200640 1 -- 172.21.15.32:0/2552190031 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fc60c0ca430 con 0x7fc60c0af450 2024-08-03T20:53:16.520 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:16.519+0000 7fc603e00640 1 -- 172.21.15.32:0/2552190031 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v69) ==== 74+0+16621 (secure 0 0 0) 0x7fc5f4081910 con 0x7fc60c0af450 2024-08-03T20:53:16.520 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-03T20:53:16.520 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":69,"fsid":"40ab141a-01d7-49b9-ae3c-2edec9876e95","created":"2024-08-03T20:47:14.957362+0000","modified":"2024-08-03T20:53:15.080708+0000","last_up_change":"2024-08-03T20:47:28.874451+0000","last_in_change":"2024-08-03T20:47:20.079442+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-08-03T20:47:28.261300+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs_metadata","create_time":"2024-08-03T20:53:13.054856+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"69","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_data","create_time":"2024-08-03T20:53:14.042324+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"69","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5a83e759-9035-438b-889a-0a6af1f0a0d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6816","nonce":3066156116},{"type":"v1","addr":"172.21.15.32:6817","nonce":3066156116}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6818","nonce":3066156116},{"type":"v1","addr":"172.21.15.32:6819","nonce":3066156116}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6822","nonce":3066156116},{"type":"v1","addr":"172.21.15.32:6823","nonce":3066156116}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6820","nonce":3066156116},{"type":"v1","addr":"172.21.15.32:6821","nonce":3066156116}]},"public_addr":"172.21.15.32:6817/3066156116","cluster_addr":"172.21.15.32:6819/3066156116","heartbeat_back_addr":"172.21.15.32:6823/3066156116","heartbeat_front_addr":"172.21.15.32:6821/3066156116","state":["exists","up"]},{"osd":1,"uuid":"d7c5c210-da4f-49f9-b9e7-1dfb02daa4bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6824","nonce":1551241759},{"type":"v1","addr":"172.21.15.32:6825","nonce":1551241759}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6826","nonce":1551241759},{"type":"v1","addr":"172.21.15.32:6827","nonce":1551241759}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6830","nonce":1551241759},{"type":"v1","addr":"172.21.15.32:6831","nonce":1551241759}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6828","nonce":1551241759},{"type":"v1","addr":"172.21.15.32:6829","nonce":1551241759}]},"public_addr":"172.21.15.32:6825/1551241759","cluster_addr":"172.21.15.32:6827/1551241759","heartbeat_back_addr":"172.21.15.32:6831/1551241759","heartbeat_front_addr":"172.21.15.32:6829/1551241759","state":["exists","up"]},{"osd":2,"uuid":"57f2e6de-a201-45c2-8981-8e56a4875ebd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6808","nonce":3806708944},{"type":"v1","addr":"172.21.15.32:6809","nonce":3806708944}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6810","nonce":3806708944},{"type":"v1","addr":"172.21.15.32:6811","nonce":3806708944}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6814","nonce":3806708944},{"type":"v1","addr":"172.21.15.32:6815","nonce":3806708944}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6812","nonce":3806708944},{"type":"v1","addr":"172.21.15.32:6813","nonce":3806708944}]},"public_addr":"172.21.15.32:6809/3806708944","cluster_addr":"172.21.15.32:6811/3806708944","heartbeat_back_addr":"172.21.15.32:6815/3806708944","heartbeat_front_addr":"172.21.15.32:6813/3806708944","state":["exists","up"]},{"osd":3,"uuid":"6439a280-da45-4148-8130-f16323935ff3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6800","nonce":9088303},{"type":"v1","addr":"172.21.15.32:6801","nonce":9088303}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6802","nonce":9088303},{"type":"v1","addr":"172.21.15.32:6803","nonce":9088303}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6806","nonce":9088303},{"type":"v1","addr":"172.21.15.32:6807","nonce":9088303}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6804","nonce":9088303},{"type":"v1","addr":"172.21.15.32:6805","nonce":9088303}]},"public_addr":"172.21.15.32:6801/9088303","cluster_addr":"172.21.15.32:6803/9088303","heartbeat_back_addr":"172.21.15.32:6807/9088303","heartbeat_front_addr":"172.21.15.32:6805/9088303","state":["exists","up"]},{"osd":4,"uuid":"4f902eff-1e18-418d-82c9-2151c69524fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6802","nonce":864752275},{"type":"v1","addr":"172.21.15.118:6803","nonce":864752275}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6804","nonce":864752275},{"type":"v1","addr":"172.21.15.118:6805","nonce":864752275}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6808","nonce":864752275},{"type":"v1","addr":"172.21.15.118:6809","nonce":864752275}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6806","nonce":864752275},{"type":"v1","addr":"172.21.15.118:6807","nonce":864752275}]},"public_addr":"172.21.15.118:6803/864752275","cluster_addr":"172.21.15.118:6805/864752275","heartbeat_back_addr":"172.21.15.118:6809/864752275","heartbeat_front_addr":"172.21.15.118:6807/864752275","state":["exists","up"]},{"osd":5,"uuid":"bfd9feac-dd06-458c-91b9-af7f175f612c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6810","nonce":2724891770},{"type":"v1","addr":"172.21.15.118:6811","nonce":2724891770}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6812","nonce":2724891770},{"type":"v1","addr":"172.21.15.118:6813","nonce":2724891770}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6816","nonce":2724891770},{"type":"v1","addr":"172.21.15.118:6817","nonce":2724891770}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6814","nonce":2724891770},{"type":"v1","addr":"172.21.15.118:6815","nonce":2724891770}]},"public_addr":"172.21.15.118:6811/2724891770","cluster_addr":"172.21.15.118:6813/2724891770","heartbeat_back_addr":"172.21.15.118:6817/2724891770","heartbeat_front_addr":"172.21.15.118:6815/2724891770","state":["exists","up"]},{"osd":6,"uuid":"5ed5a279-9d8e-416d-bbc7-7edd3e6d5705","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6825","nonce":801947386},{"type":"v1","addr":"172.21.15.118:6827","nonce":801947386}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6828","nonce":801947386},{"type":"v1","addr":"172.21.15.118:6829","nonce":801947386}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6832","nonce":801947386},{"type":"v1","addr":"172.21.15.118:6833","nonce":801947386}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6830","nonce":801947386},{"type":"v1","addr":"172.21.15.118:6831","nonce":801947386}]},"public_addr":"172.21.15.118:6827/801947386","cluster_addr":"172.21.15.118:6829/801947386","heartbeat_back_addr":"172.21.15.118:6833/801947386","heartbeat_front_addr":"172.21.15.118:6831/801947386","state":["exists","up"]},{"osd":7,"uuid":"f2086c47-875e-4451-a436-2008d1ea16af","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6818","nonce":653000048},{"type":"v1","addr":"172.21.15.118:6819","nonce":653000048}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6820","nonce":653000048},{"type":"v1","addr":"172.21.15.118:6821","nonce":653000048}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6824","nonce":653000048},{"type":"v1","addr":"172.21.15.118:6826","nonce":653000048}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6822","nonce":653000048},{"type":"v1","addr":"172.21.15.118:6823","nonce":653000048}]},"public_addr":"172.21.15.118:6819/653000048","cluster_addr":"172.21.15.118:6821/653000048","heartbeat_back_addr":"172.21.15.118:6826/653000048","heartbeat_front_addr":"172.21.15.118:6823/653000048","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-03T20:47:25.491994+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-03T20:47:25.587358+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-03T20:47:25.454175+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-03T20:47:25.377734+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-03T20:47:25.412351+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-03T20:47:25.499815+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-03T20:47:25.575044+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-03T20:47:25.552383+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-08-03T20:53:16.522 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:16.521+0000 7fc611200640 1 -- 172.21.15.32:0/2552190031 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fc5e8077010 msgr2=0x7fc5e8079520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:53:16.522 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:16.521+0000 7fc611200640 1 --2- 172.21.15.32:0/2552190031 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fc5e8077010 0x7fc5e8079520 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7fc5fc009ac0 tx=0x7fc5fc005f90 comp rx=0 tx=0).stop 2024-08-03T20:53:16.522 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:16.521+0000 7fc611200640 1 -- 172.21.15.32:0/2552190031 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc60c0af450 msgr2=0x7fc60c0ce2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:53:16.523 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:16.521+0000 7fc611200640 1 --2- 172.21.15.32:0/2552190031 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc60c0af450 0x7fc60c0ce2d0 secure :-1 s=READY pgs=753 cs=0 l=1 rev1=1 crypto rx=0x7fc5f4000c00 tx=0x7fc5f40035f0 comp rx=0 tx=0).stop 2024-08-03T20:53:16.523 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:16.521+0000 7fc611200640 1 -- 172.21.15.32:0/2552190031 shutdown_connections 2024-08-03T20:53:16.523 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:16.521+0000 7fc611200640 1 --2- 172.21.15.32:0/2552190031 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fc5e8077010 0x7fc5e8079520 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:53:16.523 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:16.521+0000 7fc611200640 1 --2- 172.21.15.32:0/2552190031 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc60c0cee10 0x7fc60c15a280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:53:16.523 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:16.521+0000 7fc611200640 1 --2- 172.21.15.32:0/2552190031 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc60c0af450 0x7fc60c0ce2d0 unknown :-1 s=CLOSED pgs=753 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:53:16.523 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:16.521+0000 7fc611200640 1 --2- 172.21.15.32:0/2552190031 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fc60c0af040 0x7fc60c158540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:53:16.523 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:16.521+0000 7fc611200640 1 -- 172.21.15.32:0/2552190031 >> 172.21.15.32:0/2552190031 conn(0x7fc60c0b2120 msgr2=0x7fc60c0cc5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:53:16.523 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:16.522+0000 7fc611200640 1 -- 172.21.15.32:0/2552190031 shutdown_connections 2024-08-03T20:53:16.523 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:16.522+0000 7fc611200640 1 -- 172.21.15.32:0/2552190031 wait complete. 2024-08-03T20:53:16.547 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-08-03T20:53:16.780 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:16.779+0000 7f47a2e00640 1 -- 172.21.15.32:0/3394083056 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f479c1621e0 msgr2=0x7f479c1625f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:53:16.781 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:16.779+0000 7f47a2e00640 1 --2- 172.21.15.32:0/3394083056 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f479c1621e0 0x7f479c1625f0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f4784009830 tx=0x7f478402a510 comp rx=0 tx=0).stop 2024-08-03T20:53:16.781 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:16.779+0000 7f47a2e00640 1 -- 172.21.15.32:0/3394083056 shutdown_connections 2024-08-03T20:53:16.781 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:16.779+0000 7f47a2e00640 1 --2- 172.21.15.32:0/3394083056 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f479c0b4640 0x7f479c0b4a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:53:16.781 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:16.779+0000 7f47a2e00640 1 --2- 172.21.15.32:0/3394083056 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f479c1621e0 0x7f479c1625f0 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:53:16.781 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:16.779+0000 7f47a2e00640 1 -- 172.21.15.32:0/3394083056 >> 172.21.15.32:0/3394083056 conn(0x7f479c0b1bd0 msgr2=0x7f479c0b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:53:16.781 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:16.779+0000 7f47a2e00640 1 -- 172.21.15.32:0/3394083056 shutdown_connections 2024-08-03T20:53:16.782 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:16.779+0000 7f47a2e00640 1 -- 172.21.15.32:0/3394083056 wait complete. 2024-08-03T20:53:16.782 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:16.782+0000 7f47a2e00640 1 Processor -- start 2024-08-03T20:53:16.782 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:16.782+0000 7f47a2e00640 1 -- start start 2024-08-03T20:53:16.786 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:16.783+0000 7f47a2e00640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f479c0b4640 0x7f479c1f93e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:53:16.786 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:16.783+0000 7f47a2e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f479c155720 0x7f479c14eec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:53:16.786 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:16.783+0000 7f47a2e00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f479c1621e0 0x7f479c0afb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:53:16.786 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:16.783+0000 7f47a2e00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f479c0b5110 con 0x7f479c155720 2024-08-03T20:53:16.786 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:16.783+0000 7f47a2e00640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f479c0b4f90 con 0x7f479c1621e0 2024-08-03T20:53:16.786 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:16.783+0000 7f47a2e00640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7f479c0b5290 con 0x7f479c0b4640 2024-08-03T20:53:16.786 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:16.784+0000 7f4793e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f479c155720 0x7f479c14eec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:53:16.787 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:16.784+0000 7f4793e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f479c155720 0x7f479c14eec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:45956/0 (socket says 172.21.15.32:45956) 2024-08-03T20:53:16.787 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:16.784+0000 7f4793e00640 1 -- 172.21.15.32:0/87489400 learned_addr learned my addr 172.21.15.32:0/87489400 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:53:16.787 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:16.784+0000 7f47a0e00640 1 --2- 172.21.15.32:0/87489400 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f479c1621e0 0x7f479c0afb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:53:16.787 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:16.784+0000 7f479be00640 1 --2- 172.21.15.32:0/87489400 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f479c0b4640 0x7f479c1f93e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:53:16.787 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:16.784+0000 7f4793e00640 1 -- 172.21.15.32:0/87489400 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f479c0b4640 msgr2=0x7f479c1f93e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:53:16.787 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:16.784+0000 7f4793e00640 1 --2- 172.21.15.32:0/87489400 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f479c0b4640 0x7f479c1f93e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:53:16.787 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:16.784+0000 7f4793e00640 1 -- 172.21.15.32:0/87489400 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f479c1621e0 msgr2=0x7f479c0afb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:53:16.787 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:16.784+0000 7f4793e00640 1 --2- 172.21.15.32:0/87489400 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f479c1621e0 0x7f479c0afb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:53:16.787 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:16.784+0000 7f4793e00640 1 -- 172.21.15.32:0/87489400 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f479c1f9920 con 0x7f479c155720 2024-08-03T20:53:16.788 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:16.785+0000 7f4793e00640 1 --2- 172.21.15.32:0/87489400 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f479c155720 0x7f479c14eec0 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 crypto rx=0x7f4784009950 tx=0x7f4784033c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:53:16.788 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:16.785+0000 7f4798c00640 1 -- 172.21.15.32:0/87489400 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4784031d60 con 0x7f479c155720 2024-08-03T20:53:16.788 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:16.785+0000 7f4798c00640 1 -- 172.21.15.32:0/87489400 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f4784033e10 con 0x7f479c155720 2024-08-03T20:53:16.788 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:16.785+0000 7f47a2e00640 1 -- 172.21.15.32:0/87489400 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f479c0ccc20 con 0x7f479c155720 2024-08-03T20:53:16.788 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:16.785+0000 7f4798c00640 1 -- 172.21.15.32:0/87489400 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4784033310 con 0x7f479c155720 2024-08-03T20:53:16.788 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:16.785+0000 7f47a2e00640 1 -- 172.21.15.32:0/87489400 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f479c0ce8f0 con 0x7f479c155720 2024-08-03T20:53:16.789 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:16.786+0000 7f47a2e00640 1 -- 172.21.15.32:0/87489400 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f479c0b4a50 con 0x7f479c155720 2024-08-03T20:53:16.789 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:16.788+0000 7f4798c00640 1 -- 172.21.15.32:0/87489400 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f47840334b0 con 0x7f479c155720 2024-08-03T20:53:16.790 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:16.789+0000 7f4798c00640 1 --2- 172.21.15.32:0/87489400 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f4778077010 0x7f4778079520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:53:16.791 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:16.789+0000 7f4798c00640 1 -- 172.21.15.32:0/87489400 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(70..70 src has 1..70) ==== 5709+0+0 (secure 0 0 0) 0x7f47840b9c40 con 0x7f479c155720 2024-08-03T20:53:16.791 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:16.789+0000 7f479be00640 1 --2- 172.21.15.32:0/87489400 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f4778077010 0x7f4778079520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:53:16.791 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:16.790+0000 7f479be00640 1 --2- 172.21.15.32:0/87489400 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f4778077010 0x7f4778079520 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f478c001720 tx=0x7f478c007e90 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:53:16.794 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:16.793+0000 7f4798c00640 1 -- 172.21.15.32:0/87489400 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7f479c1f9920 con 0x7f479c155720 2024-08-03T20:53:17.040 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:17.039+0000 7f47a2e00640 1 -- 172.21.15.32:0/87489400 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) -- 0x7f479c0d0c80 con 0x7f479c155720 2024-08-03T20:53:17.531 INFO:tasks.ceph.mds.a.smithi032.stderr:2024-08-03T20:53:17.531+0000 7f078ca00640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-08-03T20:53:17.573 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:17.573+0000 7f4798c00640 1 -- 172.21.15.32:0/87489400 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v73) ==== 118+0+0 (secure 0 0 0) 0x7f478403f030 con 0x7f479c155720 2024-08-03T20:53:17.576 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:17.576+0000 7f47a2e00640 1 -- 172.21.15.32:0/87489400 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f4778077010 msgr2=0x7f4778079520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:53:17.577 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:17.576+0000 7f47a2e00640 1 --2- 172.21.15.32:0/87489400 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f4778077010 0x7f4778079520 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f478c001720 tx=0x7f478c007e90 comp rx=0 tx=0).stop 2024-08-03T20:53:17.577 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:17.576+0000 7f47a2e00640 1 -- 172.21.15.32:0/87489400 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f479c155720 msgr2=0x7f479c14eec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:53:17.577 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:17.576+0000 7f47a2e00640 1 --2- 172.21.15.32:0/87489400 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f479c155720 0x7f479c14eec0 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 crypto rx=0x7f4784009950 tx=0x7f4784033c20 comp rx=0 tx=0).stop 2024-08-03T20:53:17.577 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:17.576+0000 7f47a2e00640 1 -- 172.21.15.32:0/87489400 shutdown_connections 2024-08-03T20:53:17.577 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:17.576+0000 7f47a2e00640 1 --2- 172.21.15.32:0/87489400 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f4778077010 0x7f4778079520 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:53:17.577 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:17.576+0000 7f47a2e00640 1 --2- 172.21.15.32:0/87489400 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f479c1621e0 0x7f479c0afb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:53:17.577 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:17.576+0000 7f47a2e00640 1 --2- 172.21.15.32:0/87489400 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f479c155720 0x7f479c14eec0 unknown :-1 s=CLOSED pgs=754 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:53:17.577 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:17.576+0000 7f47a2e00640 1 --2- 172.21.15.32:0/87489400 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f479c0b4640 0x7f479c1f93e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:53:17.577 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:17.576+0000 7f47a2e00640 1 -- 172.21.15.32:0/87489400 >> 172.21.15.32:0/87489400 conn(0x7f479c0b1bd0 msgr2=0x7f479c0b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:53:17.577 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:17.576+0000 7f47a2e00640 1 -- 172.21.15.32:0/87489400 shutdown_connections 2024-08-03T20:53:17.577 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:17.576+0000 7f47a2e00640 1 -- 172.21.15.32:0/87489400 wait complete. 2024-08-03T20:53:17.602 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-08-03T20:53:17.602 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-08-03T20:53:17.837 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:17.836+0000 7f0f0a600640 1 -- 172.21.15.32:0/101852165 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f0f040ced30 msgr2=0x7f0f040cf140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:53:17.837 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:17.836+0000 7f0f0a600640 1 --2- 172.21.15.32:0/101852165 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f0f040ced30 0x7f0f040cf140 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f0efc005fd0 tx=0x7f0efc02a8b0 comp rx=0 tx=0).stop 2024-08-03T20:53:17.837 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:17.836+0000 7f0f0a600640 1 -- 172.21.15.32:0/101852165 shutdown_connections 2024-08-03T20:53:17.837 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:17.836+0000 7f0f0a600640 1 --2- 172.21.15.32:0/101852165 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f0f040ced30 0x7f0f040cf140 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:53:17.837 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:17.836+0000 7f0f0a600640 1 --2- 172.21.15.32:0/101852165 >> v2:172.21.15.118:3300/0 conn(0x7f0f04145c50 0x7f0f0406e950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:53:17.837 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:17.836+0000 7f0f0a600640 1 -- 172.21.15.32:0/101852165 >> 172.21.15.32:0/101852165 conn(0x7f0f040b1bd0 msgr2=0x7f0f040b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:53:17.837 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:17.836+0000 7f0f0a600640 1 -- 172.21.15.32:0/101852165 shutdown_connections 2024-08-03T20:53:17.837 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:17.836+0000 7f0f0a600640 1 -- 172.21.15.32:0/101852165 wait complete. 2024-08-03T20:53:17.838 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:17.838+0000 7f0f0a600640 1 Processor -- start 2024-08-03T20:53:17.839 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:17.838+0000 7f0f0a600640 1 -- start start 2024-08-03T20:53:17.841 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:17.840+0000 7f0f0a600640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f0f040cccf0 0x7f0f041f9120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:53:17.841 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:17.840+0000 7f0f0a600640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0f040ced30 0x7f0f04159310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:53:17.841 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:17.840+0000 7f0f0a600640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f0f04145c50 0x7f0f04073190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:53:17.841 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:17.840+0000 7f0f0a600640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f0f040b4070 con 0x7f0f040ced30 2024-08-03T20:53:17.841 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:17.840+0000 7f0f0a600640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f0f040b3ef0 con 0x7f0f04145c50 2024-08-03T20:53:17.841 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:17.840+0000 7f0f0a600640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7f0f040b41f0 con 0x7f0f040cccf0 2024-08-03T20:53:17.841 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:17.840+0000 7f0efaa00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0f040ced30 0x7f0f04159310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:53:17.841 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:17.840+0000 7f0efaa00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0f040ced30 0x7f0f04159310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:45964/0 (socket says 172.21.15.32:45964) 2024-08-03T20:53:17.841 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:17.840+0000 7f0efaa00640 1 -- 172.21.15.32:0/419793035 learned_addr learned my addr 172.21.15.32:0/419793035 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:53:17.842 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:17.840+0000 7f0f03e00640 1 --2- 172.21.15.32:0/419793035 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f0f04145c50 0x7f0f04073190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:53:17.842 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:17.840+0000 7f0f03400640 1 --2- 172.21.15.32:0/419793035 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f0f040cccf0 0x7f0f041f9120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:53:17.842 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:17.841+0000 7f0efaa00640 1 -- 172.21.15.32:0/419793035 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f0f040cccf0 msgr2=0x7f0f041f9120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:53:17.842 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:17.841+0000 7f0efaa00640 1 --2- 172.21.15.32:0/419793035 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f0f040cccf0 0x7f0f041f9120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:53:17.842 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:17.841+0000 7f0efaa00640 1 -- 172.21.15.32:0/419793035 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f0f04145c50 msgr2=0x7f0f04073190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:53:17.842 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:17.841+0000 7f0efaa00640 1 --2- 172.21.15.32:0/419793035 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f0f04145c50 0x7f0f04073190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:53:17.842 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:17.841+0000 7f0efaa00640 1 -- 172.21.15.32:0/419793035 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0f04159850 con 0x7f0f040ced30 2024-08-03T20:53:17.843 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:17.841+0000 7f0efaa00640 1 --2- 172.21.15.32:0/419793035 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0f040ced30 0x7f0f04159310 secure :-1 s=READY pgs=755 cs=0 l=1 rev1=1 crypto rx=0x7f0eec00c430 tx=0x7f0eec00c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:53:17.843 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:17.841+0000 7f0efbe00640 1 -- 172.21.15.32:0/419793035 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0eec011790 con 0x7f0f040ced30 2024-08-03T20:53:17.843 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:17.841+0000 7f0efbe00640 1 -- 172.21.15.32:0/419793035 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f0eec017070 con 0x7f0f040ced30 2024-08-03T20:53:17.843 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:17.841+0000 7f0f0a600640 1 -- 172.21.15.32:0/419793035 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0f04074270 con 0x7f0f040ced30 2024-08-03T20:53:17.843 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:17.841+0000 7f0efbe00640 1 -- 172.21.15.32:0/419793035 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0eec011cc0 con 0x7f0f040ced30 2024-08-03T20:53:17.844 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:17.841+0000 7f0f0a600640 1 -- 172.21.15.32:0/419793035 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0f04072990 con 0x7f0f040ced30 2024-08-03T20:53:17.844 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:17.842+0000 7f0f0a600640 1 -- 172.21.15.32:0/419793035 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0f040b4670 con 0x7f0f040ced30 2024-08-03T20:53:17.844 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:17.844+0000 7f0efbe00640 1 -- 172.21.15.32:0/419793035 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f0eec01c460 con 0x7f0f040ced30 2024-08-03T20:53:17.845 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:17.845+0000 7f0efbe00640 1 --2- 172.21.15.32:0/419793035 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f0ee0077010 0x7f0ee0079520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:53:17.845 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:17.845+0000 7f0efbe00640 1 -- 172.21.15.32:0/419793035 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(70..70 src has 1..70) ==== 5709+0+0 (secure 0 0 0) 0x7f0eec098580 con 0x7f0f040ced30 2024-08-03T20:53:17.846 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:17.845+0000 7f0f03400640 1 --2- 172.21.15.32:0/419793035 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f0ee0077010 0x7f0ee0079520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:53:17.846 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:17.846+0000 7f0f03400640 1 --2- 172.21.15.32:0/419793035 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f0ee0077010 0x7f0ee0079520 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f0ef4009ac0 tx=0x7f0ef4005f90 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:53:17.849 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:17.849+0000 7f0efbe00640 1 -- 172.21.15.32:0/419793035 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7f0eec021020 con 0x7f0f040ced30 2024-08-03T20:53:18.095 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:18.095+0000 7f0f0a600640 1 -- 172.21.15.32:0/419793035 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) -- 0x7f0f04074790 con 0x7f0f040ced30 2024-08-03T20:53:18.579 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:18.579+0000 7f0efbe00640 1 -- 172.21.15.32:0/419793035 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v74) ==== 122+0+0 (secure 0 0 0) 0x7f0eec0617a0 con 0x7f0f040ced30 2024-08-03T20:53:18.582 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:18.581+0000 7f0f0a600640 1 -- 172.21.15.32:0/419793035 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f0ee0077010 msgr2=0x7f0ee0079520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:53:18.582 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:18.581+0000 7f0f0a600640 1 --2- 172.21.15.32:0/419793035 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f0ee0077010 0x7f0ee0079520 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f0ef4009ac0 tx=0x7f0ef4005f90 comp rx=0 tx=0).stop 2024-08-03T20:53:18.582 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:18.582+0000 7f0f0a600640 1 -- 172.21.15.32:0/419793035 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0f040ced30 msgr2=0x7f0f04159310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:53:18.583 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:18.582+0000 7f0f0a600640 1 --2- 172.21.15.32:0/419793035 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0f040ced30 0x7f0f04159310 secure :-1 s=READY pgs=755 cs=0 l=1 rev1=1 crypto rx=0x7f0eec00c430 tx=0x7f0eec00c8f0 comp rx=0 tx=0).stop 2024-08-03T20:53:18.583 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:18.582+0000 7f0f0a600640 1 -- 172.21.15.32:0/419793035 shutdown_connections 2024-08-03T20:53:18.583 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:18.582+0000 7f0f0a600640 1 --2- 172.21.15.32:0/419793035 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f0ee0077010 0x7f0ee0079520 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:53:18.583 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:18.582+0000 7f0f0a600640 1 --2- 172.21.15.32:0/419793035 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f0f04145c50 0x7f0f04073190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:53:18.583 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:18.582+0000 7f0f0a600640 1 --2- 172.21.15.32:0/419793035 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0f040ced30 0x7f0f04159310 unknown :-1 s=CLOSED pgs=755 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:53:18.583 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:18.582+0000 7f0f0a600640 1 --2- 172.21.15.32:0/419793035 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f0f040cccf0 0x7f0f041f9120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:53:18.583 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:18.582+0000 7f0f0a600640 1 -- 172.21.15.32:0/419793035 >> 172.21.15.32:0/419793035 conn(0x7f0f040b1bd0 msgr2=0x7f0f040b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:53:18.583 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:18.582+0000 7f0f0a600640 1 -- 172.21.15.32:0/419793035 shutdown_connections 2024-08-03T20:53:18.583 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:18.582+0000 7f0f0a600640 1 -- 172.21.15.32:0/419793035 wait complete. 2024-08-03T20:53:18.611 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-08-03T20:53:18.611 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-08-03T20:53:18.841 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:18.840+0000 7f92fa800640 1 -- 172.21.15.32:0/237975300 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f92f406d9f0 msgr2=0x7f92f406de00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:53:18.841 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:18.840+0000 7f92fa800640 1 --2- 172.21.15.32:0/237975300 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f92f406d9f0 0x7f92f406de00 secure :-1 s=READY pgs=756 cs=0 l=1 rev1=1 crypto rx=0x7f92ec005fd0 tx=0x7f92ec02aa90 comp rx=0 tx=0).stop 2024-08-03T20:53:18.841 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:18.840+0000 7f92fa800640 1 -- 172.21.15.32:0/237975300 shutdown_connections 2024-08-03T20:53:18.841 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:18.840+0000 7f92fa800640 1 --2- 172.21.15.32:0/237975300 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f92f406d9f0 0x7f92f406de00 unknown :-1 s=CLOSED pgs=756 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:53:18.841 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:18.840+0000 7f92fa800640 1 --2- 172.21.15.32:0/237975300 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f92f4073e30 0x7f92f40ce050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:53:18.842 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:18.840+0000 7f92fa800640 1 -- 172.21.15.32:0/237975300 >> 172.21.15.32:0/237975300 conn(0x7f92f4072e70 msgr2=0x7f92f4073280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:53:18.842 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:18.840+0000 7f92fa800640 1 -- 172.21.15.32:0/237975300 shutdown_connections 2024-08-03T20:53:18.842 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:18.840+0000 7f92fa800640 1 -- 172.21.15.32:0/237975300 wait complete. 2024-08-03T20:53:18.843 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:18.843+0000 7f92fa800640 1 Processor -- start 2024-08-03T20:53:18.843 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:18.843+0000 7f92fa800640 1 -- start start 2024-08-03T20:53:18.846 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:18.844+0000 7f92fa800640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f92f406d9f0 0x7f92f4146170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:53:18.846 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:18.844+0000 7f92fa800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f92f4073e30 0x7f92f4142d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:53:18.846 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:18.844+0000 7f92fa800640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f92f415b1b0 0x7f92f415d750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:53:18.846 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:18.844+0000 7f92fa800640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f92f40b4dc0 con 0x7f92f4073e30 2024-08-03T20:53:18.846 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:18.844+0000 7f92fa800640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f92f40b4c40 con 0x7f92f406d9f0 2024-08-03T20:53:18.846 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:18.844+0000 7f92fa800640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7f92f40b4f40 con 0x7f92f415b1b0 2024-08-03T20:53:18.846 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:18.844+0000 7f92f8a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f92f4073e30 0x7f92f4142d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:53:18.846 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:18.844+0000 7f92f8a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f92f4073e30 0x7f92f4142d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:45984/0 (socket says 172.21.15.32:45984) 2024-08-03T20:53:18.846 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:18.844+0000 7f92f8a00640 1 -- 172.21.15.32:0/2054559402 learned_addr learned my addr 172.21.15.32:0/2054559402 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:53:18.847 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:18.844+0000 7f92f9400640 1 --2- 172.21.15.32:0/2054559402 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f92f406d9f0 0x7f92f4146170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:53:18.847 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:18.845+0000 7f92f9e00640 1 --2- 172.21.15.32:0/2054559402 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f92f415b1b0 0x7f92f415d750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:53:18.847 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:18.845+0000 7f92f8a00640 1 -- 172.21.15.32:0/2054559402 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f92f415b1b0 msgr2=0x7f92f415d750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:53:18.847 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:18.845+0000 7f92f8a00640 1 --2- 172.21.15.32:0/2054559402 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f92f415b1b0 0x7f92f415d750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:53:18.847 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:18.845+0000 7f92f8a00640 1 -- 172.21.15.32:0/2054559402 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f92f406d9f0 msgr2=0x7f92f4146170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:53:18.847 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:18.845+0000 7f92f8a00640 1 --2- 172.21.15.32:0/2054559402 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f92f406d9f0 0x7f92f4146170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:53:18.847 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:18.845+0000 7f92f8a00640 1 -- 172.21.15.32:0/2054559402 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f92f41466b0 con 0x7f92f4073e30 2024-08-03T20:53:18.847 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:18.845+0000 7f92f8a00640 1 --2- 172.21.15.32:0/2054559402 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f92f4073e30 0x7f92f4142d20 secure :-1 s=READY pgs=757 cs=0 l=1 rev1=1 crypto rx=0x7f92e000c3a0 tx=0x7f92e000c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:53:18.847 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:18.845+0000 7f92f1600640 1 -- 172.21.15.32:0/2054559402 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92e0017070 con 0x7f92f4073e30 2024-08-03T20:53:18.847 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:18.845+0000 7f92f1600640 1 -- 172.21.15.32:0/2054559402 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f92e00053c0 con 0x7f92f4073e30 2024-08-03T20:53:18.847 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:18.845+0000 7f92f1600640 1 -- 172.21.15.32:0/2054559402 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92e0003850 con 0x7f92f4073e30 2024-08-03T20:53:18.847 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:18.845+0000 7f92fa800640 1 -- 172.21.15.32:0/2054559402 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f92f41548d0 con 0x7f92f4073e30 2024-08-03T20:53:18.847 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:18.846+0000 7f92fa800640 1 -- 172.21.15.32:0/2054559402 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f92f4154e90 con 0x7f92f4073e30 2024-08-03T20:53:18.848 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:18.846+0000 7f92fa800640 1 -- 172.21.15.32:0/2054559402 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f92f40b5460 con 0x7f92f4073e30 2024-08-03T20:53:18.848 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:18.848+0000 7f92f1600640 1 -- 172.21.15.32:0/2054559402 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f92e001a070 con 0x7f92f4073e30 2024-08-03T20:53:18.849 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:18.849+0000 7f92f1600640 1 --2- 172.21.15.32:0/2054559402 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f92d4077040 0x7f92d4079550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:53:18.849 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:18.849+0000 7f92f1600640 1 -- 172.21.15.32:0/2054559402 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(70..70 src has 1..70) ==== 5709+0+0 (secure 0 0 0) 0x7f92e0097bd0 con 0x7f92f4073e30 2024-08-03T20:53:18.849 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:18.849+0000 7f92f9400640 1 --2- 172.21.15.32:0/2054559402 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f92d4077040 0x7f92d4079550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:53:18.850 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:18.850+0000 7f92f9400640 1 --2- 172.21.15.32:0/2054559402 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f92d4077040 0x7f92d4079550 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f92e8005fd0 tx=0x7f92e800a250 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:53:18.853 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:18.853+0000 7f92f1600640 1 -- 172.21.15.32:0/2054559402 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7f92f406e340 con 0x7f92f4073e30 2024-08-03T20:53:19.098 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:19.098+0000 7f92fa800640 1 -- 172.21.15.32:0/2054559402 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f92f40b1a00 con 0x7f92f4073e30 2024-08-03T20:53:19.099 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:19.098+0000 7f92f1600640 1 -- 172.21.15.32:0/2054559402 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 74 v74) ==== 94+0+5231 (secure 0 0 0) 0x7f92e0069090 con 0x7f92f4073e30 2024-08-03T20:53:19.099 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-03T20:53:19.099 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":74,"btime":"2024-08-03T20:53:18:574147+0000","default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5666,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6837/2261538396","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6835","nonce":2261538396},{"type":"v1","addr":"172.21.15.118:6837","nonce":2261538396}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":69},{"gid":6061,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6836/1133580656","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6834","nonce":1133580656},{"type":"v1","addr":"172.21.15.118:6836","nonce":1133580656}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":68},{"gid":6396,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6834/1798758431","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":1798758431},{"type":"v1","addr":"172.21.15.32:6834","nonce":1798758431}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":69}],"filesystems":[{"mdsmap":{"epoch":74,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-08-03T20:53:15.080669+0000","modified":"2024-08-03T20:53:18.096135+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6399},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6399":{"gid":6399,"name":"a","rank":0,"incarnation":70,"state":"up:active","state_seq":3,"addr":"172.21.15.32:6835/4240363667","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6833","nonce":4240363667},{"type":"v1","addr":"172.21.15.32:6835","nonce":4240363667}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6399,"qdb_cluster":[6399]},"id":5}]} 2024-08-03T20:53:19.099 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 74 2024-08-03T20:53:19.102 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:19.101+0000 7f92fa800640 1 -- 172.21.15.32:0/2054559402 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f92d4077040 msgr2=0x7f92d4079550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:53:19.102 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:19.101+0000 7f92fa800640 1 --2- 172.21.15.32:0/2054559402 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f92d4077040 0x7f92d4079550 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f92e8005fd0 tx=0x7f92e800a250 comp rx=0 tx=0).stop 2024-08-03T20:53:19.102 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:19.101+0000 7f92fa800640 1 -- 172.21.15.32:0/2054559402 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f92f4073e30 msgr2=0x7f92f4142d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:53:19.102 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:19.101+0000 7f92fa800640 1 --2- 172.21.15.32:0/2054559402 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f92f4073e30 0x7f92f4142d20 secure :-1 s=READY pgs=757 cs=0 l=1 rev1=1 crypto rx=0x7f92e000c3a0 tx=0x7f92e000c860 comp rx=0 tx=0).stop 2024-08-03T20:53:19.102 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:19.101+0000 7f92fa800640 1 -- 172.21.15.32:0/2054559402 shutdown_connections 2024-08-03T20:53:19.103 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:19.101+0000 7f92fa800640 1 --2- 172.21.15.32:0/2054559402 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f92d4077040 0x7f92d4079550 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:53:19.103 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:19.101+0000 7f92fa800640 1 --2- 172.21.15.32:0/2054559402 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f92f415b1b0 0x7f92f415d750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:53:19.103 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:19.101+0000 7f92fa800640 1 --2- 172.21.15.32:0/2054559402 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f92f4073e30 0x7f92f4142d20 unknown :-1 s=CLOSED pgs=757 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:53:19.103 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:19.101+0000 7f92fa800640 1 --2- 172.21.15.32:0/2054559402 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f92f406d9f0 0x7f92f4146170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:53:19.103 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:19.101+0000 7f92fa800640 1 -- 172.21.15.32:0/2054559402 >> 172.21.15.32:0/2054559402 conn(0x7f92f4072e70 msgr2=0x7f92f40b00f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:53:19.103 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:19.101+0000 7f92fa800640 1 -- 172.21.15.32:0/2054559402 shutdown_connections 2024-08-03T20:53:19.104 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:19.101+0000 7f92fa800640 1 -- 172.21.15.32:0/2054559402 wait complete. 2024-08-03T20:53:19.123 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-08-03T20:53:19.351 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:19.350+0000 7ff5ff600640 1 -- 172.21.15.32:0/2833890662 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff5f80b39a0 msgr2=0x7ff5f80b3db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:53:19.351 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:19.350+0000 7ff5ff600640 1 --2- 172.21.15.32:0/2833890662 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff5f80b39a0 0x7ff5f80b3db0 secure :-1 s=READY pgs=758 cs=0 l=1 rev1=1 crypto rx=0x7ff5f80b2840 tx=0x7ff5e402a5f0 comp rx=0 tx=0).stop 2024-08-03T20:53:19.352 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:19.350+0000 7ff5ff600640 1 -- 172.21.15.32:0/2833890662 shutdown_connections 2024-08-03T20:53:19.352 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:19.350+0000 7ff5ff600640 1 --2- 172.21.15.32:0/2833890662 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff5f80b39a0 0x7ff5f80b3db0 unknown :-1 s=CLOSED pgs=758 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:53:19.352 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:19.350+0000 7ff5ff600640 1 --2- 172.21.15.32:0/2833890662 >> v2:172.21.15.118:3300/0 conn(0x7ff5f806f720 0x7ff5f806fb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:53:19.352 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:19.350+0000 7ff5ff600640 1 -- 172.21.15.32:0/2833890662 >> 172.21.15.32:0/2833890662 conn(0x7ff5f80b1bd0 msgr2=0x7ff5f80b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:53:19.352 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:19.350+0000 7ff5ff600640 1 -- 172.21.15.32:0/2833890662 shutdown_connections 2024-08-03T20:53:19.352 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:19.350+0000 7ff5ff600640 1 -- 172.21.15.32:0/2833890662 wait complete. 2024-08-03T20:53:19.353 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:19.353+0000 7ff5ff600640 1 Processor -- start 2024-08-03T20:53:19.353 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:19.353+0000 7ff5ff600640 1 -- start start 2024-08-03T20:53:19.357 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:19.354+0000 7ff5ff600640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff5f806f720 0x7ff5f81570d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:53:19.357 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:19.354+0000 7ff5ff600640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ff5f80b39a0 0x7ff5f81605e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:53:19.357 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:19.354+0000 7ff5ff600640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7ff5f8162210 0x7ff5f8160b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:53:19.357 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:19.354+0000 7ff5ff600640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7ff5f80af010 con 0x7ff5f806f720 2024-08-03T20:53:19.357 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:19.354+0000 7ff5ff600640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7ff5f80aee90 con 0x7ff5f80b39a0 2024-08-03T20:53:19.357 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:19.354+0000 7ff5ff600640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7ff5f80af190 con 0x7ff5f8162210 2024-08-03T20:53:19.357 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:19.354+0000 7ff5fcc00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff5f806f720 0x7ff5f81570d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:53:19.357 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:19.354+0000 7ff5fcc00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff5f806f720 0x7ff5f81570d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:46000/0 (socket says 172.21.15.32:46000) 2024-08-03T20:53:19.357 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:19.354+0000 7ff5fcc00640 1 -- 172.21.15.32:0/1366488630 learned_addr learned my addr 172.21.15.32:0/1366488630 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:53:19.357 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:19.355+0000 7ff5f7e00640 1 --2- 172.21.15.32:0/1366488630 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ff5f80b39a0 0x7ff5f81605e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:53:19.358 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:19.355+0000 7ff5fd600640 1 --2- 172.21.15.32:0/1366488630 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7ff5f8162210 0x7ff5f8160b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:53:19.358 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:19.355+0000 7ff5fcc00640 1 -- 172.21.15.32:0/1366488630 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7ff5f8162210 msgr2=0x7ff5f8160b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:53:19.358 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:19.355+0000 7ff5fcc00640 1 --2- 172.21.15.32:0/1366488630 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7ff5f8162210 0x7ff5f8160b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:53:19.358 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:19.355+0000 7ff5fcc00640 1 -- 172.21.15.32:0/1366488630 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ff5f80b39a0 msgr2=0x7ff5f81605e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:53:19.358 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:19.355+0000 7ff5fcc00640 1 --2- 172.21.15.32:0/1366488630 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ff5f80b39a0 0x7ff5f81605e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:53:19.358 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:19.355+0000 7ff5fcc00640 1 -- 172.21.15.32:0/1366488630 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff5f8158bd0 con 0x7ff5f806f720 2024-08-03T20:53:19.358 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:19.355+0000 7ff5fd600640 1 --2- 172.21.15.32:0/1366488630 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7ff5f8162210 0x7ff5f8160b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-03T20:53:19.358 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:19.355+0000 7ff5fcc00640 1 --2- 172.21.15.32:0/1366488630 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff5f806f720 0x7ff5f81570d0 secure :-1 s=READY pgs=759 cs=0 l=1 rev1=1 crypto rx=0x7ff5ec00e7f0 tx=0x7ff5ec00ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:53:19.358 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:19.355+0000 7ff5f4c00640 1 -- 172.21.15.32:0/1366488630 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5ec00ccc0 con 0x7ff5f806f720 2024-08-03T20:53:19.358 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:19.356+0000 7ff5f4c00640 1 -- 172.21.15.32:0/1366488630 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7ff5ec00ce80 con 0x7ff5f806f720 2024-08-03T20:53:19.358 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:19.356+0000 7ff5ff600640 1 -- 172.21.15.32:0/1366488630 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff5f81588f0 con 0x7ff5f806f720 2024-08-03T20:53:19.358 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:19.356+0000 7ff5f4c00640 1 -- 172.21.15.32:0/1366488630 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5ec005280 con 0x7ff5f806f720 2024-08-03T20:53:19.359 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:19.356+0000 7ff5ff600640 1 -- 172.21.15.32:0/1366488630 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff5f815fa20 con 0x7ff5f806f720 2024-08-03T20:53:19.359 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:19.357+0000 7ff5ff600640 1 -- 172.21.15.32:0/1366488630 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff5f806fb30 con 0x7ff5f806f720 2024-08-03T20:53:19.359 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:19.358+0000 7ff5f4c00640 1 -- 172.21.15.32:0/1366488630 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7ff5ec005420 con 0x7ff5f806f720 2024-08-03T20:53:19.360 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:19.359+0000 7ff5f4c00640 1 --2- 172.21.15.32:0/1366488630 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7ff5d8077010 0x7ff5d8079520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:53:19.360 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:19.359+0000 7ff5f4c00640 1 -- 172.21.15.32:0/1366488630 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(70..70 src has 1..70) ==== 5709+0+0 (secure 0 0 0) 0x7ff5ec099180 con 0x7ff5f806f720 2024-08-03T20:53:19.360 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:19.359+0000 7ff5f7e00640 1 --2- 172.21.15.32:0/1366488630 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7ff5d8077010 0x7ff5d8079520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:53:19.361 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:19.360+0000 7ff5f7e00640 1 --2- 172.21.15.32:0/1366488630 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7ff5d8077010 0x7ff5d8079520 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7ff5e4033000 tx=0x7ff5e4004730 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:53:19.363 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:19.363+0000 7ff5f4c00640 1 -- 172.21.15.32:0/1366488630 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7ff5ec062370 con 0x7ff5f806f720 2024-08-03T20:53:19.571 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:19.571+0000 7ff5ff600640 1 -- 172.21.15.32:0/1366488630 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7ff5f8074a90 con 0x7ff5f806f720 2024-08-03T20:53:19.572 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:19.572+0000 7ff5f4c00640 1 -- 172.21.15.32:0/1366488630 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v70) ==== 74+0+16608 (secure 0 0 0) 0x7ff5ec061c70 con 0x7ff5f806f720 2024-08-03T20:53:19.573 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-03T20:53:19.573 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":70,"fsid":"40ab141a-01d7-49b9-ae3c-2edec9876e95","created":"2024-08-03T20:47:14.957362+0000","modified":"2024-08-03T20:53:16.559546+0000","last_up_change":"2024-08-03T20:47:28.874451+0000","last_in_change":"2024-08-03T20:47:20.079442+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-08-03T20:47:28.261300+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs_metadata","create_time":"2024-08-03T20:53:13.054856+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"69","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_data","create_time":"2024-08-03T20:53:14.042324+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"70","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5a83e759-9035-438b-889a-0a6af1f0a0d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6816","nonce":3066156116},{"type":"v1","addr":"172.21.15.32:6817","nonce":3066156116}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6818","nonce":3066156116},{"type":"v1","addr":"172.21.15.32:6819","nonce":3066156116}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6822","nonce":3066156116},{"type":"v1","addr":"172.21.15.32:6823","nonce":3066156116}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6820","nonce":3066156116},{"type":"v1","addr":"172.21.15.32:6821","nonce":3066156116}]},"public_addr":"172.21.15.32:6817/3066156116","cluster_addr":"172.21.15.32:6819/3066156116","heartbeat_back_addr":"172.21.15.32:6823/3066156116","heartbeat_front_addr":"172.21.15.32:6821/3066156116","state":["exists","up"]},{"osd":1,"uuid":"d7c5c210-da4f-49f9-b9e7-1dfb02daa4bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6824","nonce":1551241759},{"type":"v1","addr":"172.21.15.32:6825","nonce":1551241759}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6826","nonce":1551241759},{"type":"v1","addr":"172.21.15.32:6827","nonce":1551241759}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6830","nonce":1551241759},{"type":"v1","addr":"172.21.15.32:6831","nonce":1551241759}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6828","nonce":1551241759},{"type":"v1","addr":"172.21.15.32:6829","nonce":1551241759}]},"public_addr":"172.21.15.32:6825/1551241759","cluster_addr":"172.21.15.32:6827/1551241759","heartbeat_back_addr":"172.21.15.32:6831/1551241759","heartbeat_front_addr":"172.21.15.32:6829/1551241759","state":["exists","up"]},{"osd":2,"uuid":"57f2e6de-a201-45c2-8981-8e56a4875ebd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6808","nonce":3806708944},{"type":"v1","addr":"172.21.15.32:6809","nonce":3806708944}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6810","nonce":3806708944},{"type":"v1","addr":"172.21.15.32:6811","nonce":3806708944}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6814","nonce":3806708944},{"type":"v1","addr":"172.21.15.32:6815","nonce":3806708944}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6812","nonce":3806708944},{"type":"v1","addr":"172.21.15.32:6813","nonce":3806708944}]},"public_addr":"172.21.15.32:6809/3806708944","cluster_addr":"172.21.15.32:6811/3806708944","heartbeat_back_addr":"172.21.15.32:6815/3806708944","heartbeat_front_addr":"172.21.15.32:6813/3806708944","state":["exists","up"]},{"osd":3,"uuid":"6439a280-da45-4148-8130-f16323935ff3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6800","nonce":9088303},{"type":"v1","addr":"172.21.15.32:6801","nonce":9088303}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6802","nonce":9088303},{"type":"v1","addr":"172.21.15.32:6803","nonce":9088303}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6806","nonce":9088303},{"type":"v1","addr":"172.21.15.32:6807","nonce":9088303}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6804","nonce":9088303},{"type":"v1","addr":"172.21.15.32:6805","nonce":9088303}]},"public_addr":"172.21.15.32:6801/9088303","cluster_addr":"172.21.15.32:6803/9088303","heartbeat_back_addr":"172.21.15.32:6807/9088303","heartbeat_front_addr":"172.21.15.32:6805/9088303","state":["exists","up"]},{"osd":4,"uuid":"4f902eff-1e18-418d-82c9-2151c69524fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6802","nonce":864752275},{"type":"v1","addr":"172.21.15.118:6803","nonce":864752275}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6804","nonce":864752275},{"type":"v1","addr":"172.21.15.118:6805","nonce":864752275}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6808","nonce":864752275},{"type":"v1","addr":"172.21.15.118:6809","nonce":864752275}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6806","nonce":864752275},{"type":"v1","addr":"172.21.15.118:6807","nonce":864752275}]},"public_addr":"172.21.15.118:6803/864752275","cluster_addr":"172.21.15.118:6805/864752275","heartbeat_back_addr":"172.21.15.118:6809/864752275","heartbeat_front_addr":"172.21.15.118:6807/864752275","state":["exists","up"]},{"osd":5,"uuid":"bfd9feac-dd06-458c-91b9-af7f175f612c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6810","nonce":2724891770},{"type":"v1","addr":"172.21.15.118:6811","nonce":2724891770}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6812","nonce":2724891770},{"type":"v1","addr":"172.21.15.118:6813","nonce":2724891770}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6816","nonce":2724891770},{"type":"v1","addr":"172.21.15.118:6817","nonce":2724891770}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6814","nonce":2724891770},{"type":"v1","addr":"172.21.15.118:6815","nonce":2724891770}]},"public_addr":"172.21.15.118:6811/2724891770","cluster_addr":"172.21.15.118:6813/2724891770","heartbeat_back_addr":"172.21.15.118:6817/2724891770","heartbeat_front_addr":"172.21.15.118:6815/2724891770","state":["exists","up"]},{"osd":6,"uuid":"5ed5a279-9d8e-416d-bbc7-7edd3e6d5705","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6825","nonce":801947386},{"type":"v1","addr":"172.21.15.118:6827","nonce":801947386}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6828","nonce":801947386},{"type":"v1","addr":"172.21.15.118:6829","nonce":801947386}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6832","nonce":801947386},{"type":"v1","addr":"172.21.15.118:6833","nonce":801947386}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6830","nonce":801947386},{"type":"v1","addr":"172.21.15.118:6831","nonce":801947386}]},"public_addr":"172.21.15.118:6827/801947386","cluster_addr":"172.21.15.118:6829/801947386","heartbeat_back_addr":"172.21.15.118:6833/801947386","heartbeat_front_addr":"172.21.15.118:6831/801947386","state":["exists","up"]},{"osd":7,"uuid":"f2086c47-875e-4451-a436-2008d1ea16af","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6818","nonce":653000048},{"type":"v1","addr":"172.21.15.118:6819","nonce":653000048}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6820","nonce":653000048},{"type":"v1","addr":"172.21.15.118:6821","nonce":653000048}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6824","nonce":653000048},{"type":"v1","addr":"172.21.15.118:6826","nonce":653000048}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6822","nonce":653000048},{"type":"v1","addr":"172.21.15.118:6823","nonce":653000048}]},"public_addr":"172.21.15.118:6819/653000048","cluster_addr":"172.21.15.118:6821/653000048","heartbeat_back_addr":"172.21.15.118:6826/653000048","heartbeat_front_addr":"172.21.15.118:6823/653000048","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-03T20:47:25.491994+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-03T20:47:25.587358+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-03T20:47:25.454175+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-03T20:47:25.377734+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-03T20:47:25.412351+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-03T20:47:25.499815+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-03T20:47:25.575044+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-03T20:47:25.552383+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-08-03T20:53:19.576 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:19.575+0000 7ff5ff600640 1 -- 172.21.15.32:0/1366488630 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7ff5d8077010 msgr2=0x7ff5d8079520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:53:19.576 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:19.575+0000 7ff5ff600640 1 --2- 172.21.15.32:0/1366488630 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7ff5d8077010 0x7ff5d8079520 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7ff5e4033000 tx=0x7ff5e4004730 comp rx=0 tx=0).stop 2024-08-03T20:53:19.576 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:19.575+0000 7ff5ff600640 1 -- 172.21.15.32:0/1366488630 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff5f806f720 msgr2=0x7ff5f81570d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:53:19.576 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:19.575+0000 7ff5ff600640 1 --2- 172.21.15.32:0/1366488630 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff5f806f720 0x7ff5f81570d0 secure :-1 s=READY pgs=759 cs=0 l=1 rev1=1 crypto rx=0x7ff5ec00e7f0 tx=0x7ff5ec00ecb0 comp rx=0 tx=0).stop 2024-08-03T20:53:19.576 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:19.575+0000 7ff5ff600640 1 -- 172.21.15.32:0/1366488630 shutdown_connections 2024-08-03T20:53:19.576 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:19.575+0000 7ff5ff600640 1 --2- 172.21.15.32:0/1366488630 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7ff5d8077010 0x7ff5d8079520 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:53:19.577 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:19.575+0000 7ff5ff600640 1 --2- 172.21.15.32:0/1366488630 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ff5f80b39a0 0x7ff5f81605e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:53:19.577 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:19.575+0000 7ff5ff600640 1 --2- 172.21.15.32:0/1366488630 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7ff5f8162210 0x7ff5f8160b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:53:19.577 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:19.575+0000 7ff5ff600640 1 --2- 172.21.15.32:0/1366488630 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff5f806f720 0x7ff5f81570d0 unknown :-1 s=CLOSED pgs=759 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:53:19.577 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:19.575+0000 7ff5ff600640 1 -- 172.21.15.32:0/1366488630 >> 172.21.15.32:0/1366488630 conn(0x7ff5f80b1bd0 msgr2=0x7ff5f80d08b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:53:19.577 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:19.575+0000 7ff5ff600640 1 -- 172.21.15.32:0/1366488630 shutdown_connections 2024-08-03T20:53:19.577 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:19.575+0000 7ff5ff600640 1 -- 172.21.15.32:0/1366488630 wait complete. 2024-08-03T20:53:19.603 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-08-03T20:53:19.603 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-08-03T20:53:19.832 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:19.831+0000 7fa0c6600640 1 -- 172.21.15.32:0/1528209217 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa0c00d29a0 msgr2=0x7fa0c00d1c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:53:19.832 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:19.831+0000 7fa0c6600640 1 --2- 172.21.15.32:0/1528209217 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa0c00d29a0 0x7fa0c00d1c60 secure :-1 s=READY pgs=761 cs=0 l=1 rev1=1 crypto rx=0x7fa0c00ba100 tx=0x7fa0a802a4b0 comp rx=0 tx=0).stop 2024-08-03T20:53:19.832 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:19.831+0000 7fa0c6600640 1 -- 172.21.15.32:0/1528209217 shutdown_connections 2024-08-03T20:53:19.832 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:19.831+0000 7fa0c6600640 1 --2- 172.21.15.32:0/1528209217 >> v2:172.21.15.118:3300/0 conn(0x7fa0c00cec30 0x7fa0c00cf040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:53:19.832 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:19.831+0000 7fa0c6600640 1 --2- 172.21.15.32:0/1528209217 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa0c00d29a0 0x7fa0c00d1c60 unknown :-1 s=CLOSED pgs=761 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:53:19.832 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:19.831+0000 7fa0c6600640 1 -- 172.21.15.32:0/1528209217 >> 172.21.15.32:0/1528209217 conn(0x7fa0c00d4090 msgr2=0x7fa0c00d44a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:53:19.832 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:19.831+0000 7fa0c6600640 1 -- 172.21.15.32:0/1528209217 shutdown_connections 2024-08-03T20:53:19.832 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:19.831+0000 7fa0c6600640 1 -- 172.21.15.32:0/1528209217 wait complete. 2024-08-03T20:53:19.834 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:19.833+0000 7fa0c6600640 1 Processor -- start 2024-08-03T20:53:19.834 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:19.833+0000 7fa0c6600640 1 -- start start 2024-08-03T20:53:19.836 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:19.835+0000 7fa0c6600640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa0c00cec30 0x7fa0c015f3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:53:19.837 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:19.835+0000 7fa0c6600640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa0c00d4bb0 0x7fa0c00b8c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:53:19.837 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:19.835+0000 7fa0c6600640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fa0c00b7920 0x7fa0c00b7d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:53:19.837 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:19.835+0000 7fa0c6600640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fa0c00cfa80 con 0x7fa0c00cec30 2024-08-03T20:53:19.837 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:19.835+0000 7fa0c6600640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7fa0c00cf900 con 0x7fa0c00d4bb0 2024-08-03T20:53:19.837 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:19.835+0000 7fa0c6600640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7fa0c00cfc00 con 0x7fa0c00b7920 2024-08-03T20:53:19.837 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:19.835+0000 7fa0c5200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa0c00cec30 0x7fa0c015f3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:53:19.837 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:19.835+0000 7fa0c5200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa0c00cec30 0x7fa0c015f3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:45488/0 (socket says 172.21.15.32:45488) 2024-08-03T20:53:19.837 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:19.835+0000 7fa0c5200640 1 -- 172.21.15.32:0/3276820576 learned_addr learned my addr 172.21.15.32:0/3276820576 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:53:19.837 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:19.835+0000 7fa0b7e00640 1 --2- 172.21.15.32:0/3276820576 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa0c00d4bb0 0x7fa0c00b8c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:53:19.837 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:19.835+0000 7fa0c5c00640 1 --2- 172.21.15.32:0/3276820576 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fa0c00b7920 0x7fa0c00b7d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:53:19.837 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:19.835+0000 7fa0c5200640 1 -- 172.21.15.32:0/3276820576 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fa0c00b7920 msgr2=0x7fa0c00b7d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:53:19.837 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:19.835+0000 7fa0c5200640 1 --2- 172.21.15.32:0/3276820576 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fa0c00b7920 0x7fa0c00b7d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:53:19.838 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:19.835+0000 7fa0c5200640 1 -- 172.21.15.32:0/3276820576 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa0c00d4bb0 msgr2=0x7fa0c00b8c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:53:19.838 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:19.835+0000 7fa0c5200640 1 --2- 172.21.15.32:0/3276820576 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa0c00d4bb0 0x7fa0c00b8c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:53:19.838 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:19.836+0000 7fa0c5200640 1 -- 172.21.15.32:0/3276820576 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa0c015e610 con 0x7fa0c00cec30 2024-08-03T20:53:19.838 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:19.836+0000 7fa0b7e00640 1 --2- 172.21.15.32:0/3276820576 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa0c00d4bb0 0x7fa0c00b8c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-03T20:53:19.838 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:19.836+0000 7fa0c5200640 1 --2- 172.21.15.32:0/3276820576 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa0c00cec30 0x7fa0c015f3d0 secure :-1 s=READY pgs=762 cs=0 l=1 rev1=1 crypto rx=0x7fa0b000c3d0 tx=0x7fa0b000c890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:53:19.838 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:19.836+0000 7fa0bd600640 1 -- 172.21.15.32:0/3276820576 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0b0011790 con 0x7fa0c00cec30 2024-08-03T20:53:19.838 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:19.836+0000 7fa0bd600640 1 -- 172.21.15.32:0/3276820576 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fa0b0017070 con 0x7fa0c00cec30 2024-08-03T20:53:19.838 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:19.836+0000 7fa0bd600640 1 -- 172.21.15.32:0/3276820576 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0b0011cc0 con 0x7fa0c00cec30 2024-08-03T20:53:19.838 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:19.836+0000 7fa0c6600640 1 -- 172.21.15.32:0/3276820576 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa0c015e8f0 con 0x7fa0c00cec30 2024-08-03T20:53:19.838 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:19.836+0000 7fa0c6600640 1 -- 172.21.15.32:0/3276820576 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa0c015eeb0 con 0x7fa0c00cec30 2024-08-03T20:53:19.839 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:19.837+0000 7fa0c6600640 1 -- 172.21.15.32:0/3276820576 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa0c00cf040 con 0x7fa0c00cec30 2024-08-03T20:53:19.839 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:19.839+0000 7fa0bd600640 1 -- 172.21.15.32:0/3276820576 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7fa0b001c460 con 0x7fa0c00cec30 2024-08-03T20:53:19.840 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:19.840+0000 7fa0bd600640 1 --2- 172.21.15.32:0/3276820576 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fa09c077010 0x7fa09c079520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:53:19.840 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:19.840+0000 7fa0bd600640 1 -- 172.21.15.32:0/3276820576 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(70..70 src has 1..70) ==== 5709+0+0 (secure 0 0 0) 0x7fa0b00986e0 con 0x7fa0c00cec30 2024-08-03T20:53:19.840 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:19.840+0000 7fa0b7e00640 1 --2- 172.21.15.32:0/3276820576 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fa09c077010 0x7fa09c079520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:53:19.842 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:19.841+0000 7fa0b7e00640 1 --2- 172.21.15.32:0/3276820576 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fa09c077010 0x7fa09c079520 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7fa0a8034000 tx=0x7fa0a8004730 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:53:19.844 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:19.844+0000 7fa0bd600640 1 -- 172.21.15.32:0/3276820576 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7fa0b0061960 con 0x7fa0c00cec30 2024-08-03T20:53:20.045 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:20.045+0000 7fa0c6600640 1 -- 172.21.15.32:0/3276820576 --> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fa0c014d390 con 0x7fa09c077010 2024-08-03T20:53:20.050 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:20.050+0000 7fa0bd600640 1 -- 172.21.15.32:0/3276820576 <== mgr.4106 v2:172.21.15.118:6800/4205950699 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+239163 (secure 0 0 0) 0x7fa0c014d390 con 0x7fa09c077010 2024-08-03T20:53:20.053 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-03T20:53:20.055 INFO:teuthology.orchestra.run.smithi032.stderr:dumped all 2024-08-03T20:53:20.058 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:20.054+0000 7fa0c6600640 1 -- 172.21.15.32:0/3276820576 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fa09c077010 msgr2=0x7fa09c079520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:53:20.058 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:20.054+0000 7fa0c6600640 1 --2- 172.21.15.32:0/3276820576 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fa09c077010 0x7fa09c079520 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7fa0a8034000 tx=0x7fa0a8004730 comp rx=0 tx=0).stop 2024-08-03T20:53:20.058 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:20.054+0000 7fa0c6600640 1 -- 172.21.15.32:0/3276820576 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa0c00cec30 msgr2=0x7fa0c015f3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:53:20.058 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:20.054+0000 7fa0c6600640 1 --2- 172.21.15.32:0/3276820576 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa0c00cec30 0x7fa0c015f3d0 secure :-1 s=READY pgs=762 cs=0 l=1 rev1=1 crypto rx=0x7fa0b000c3d0 tx=0x7fa0b000c890 comp rx=0 tx=0).stop 2024-08-03T20:53:20.058 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:20.054+0000 7fa0c6600640 1 -- 172.21.15.32:0/3276820576 shutdown_connections 2024-08-03T20:53:20.058 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:20.054+0000 7fa0c6600640 1 --2- 172.21.15.32:0/3276820576 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fa09c077010 0x7fa09c079520 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:53:20.058 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:20.054+0000 7fa0c6600640 1 --2- 172.21.15.32:0/3276820576 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa0c00d4bb0 0x7fa0c00b8c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:53:20.058 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:20.054+0000 7fa0c6600640 1 --2- 172.21.15.32:0/3276820576 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fa0c00b7920 0x7fa0c00b7d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:53:20.058 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:20.054+0000 7fa0c6600640 1 --2- 172.21.15.32:0/3276820576 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa0c00cec30 0x7fa0c015f3d0 unknown :-1 s=CLOSED pgs=762 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:53:20.058 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:20.054+0000 7fa0c6600640 1 -- 172.21.15.32:0/3276820576 >> 172.21.15.32:0/3276820576 conn(0x7fa0c00d4090 msgr2=0x7fa0c006df20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:53:20.059 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:20.055+0000 7fa0c6600640 1 -- 172.21.15.32:0/3276820576 shutdown_connections 2024-08-03T20:53:20.059 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:20.055+0000 7fa0c6600640 1 -- 172.21.15.32:0/3276820576 wait complete. 2024-08-03T20:53:20.080 INFO:teuthology.orchestra.run.smithi032.stdout:{"pg_ready":true,"pg_map":{"version":238,"stamp":"2024-08-03T20:53:18.320320+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":46,"acting":46,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":12,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":5,"kb":754974720,"kb_used":261036,"kb_used_data":7404,"kb_used_omap":12,"kb_used_meta":253491,"kb_avail":754713684,"statfs":{"total":773094113280,"available":772826812416,"internally_reserved":0,"allocated":7581696,"data_stored":3854019,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":259575345},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.780853"},"pg_stats":[{"pgid":"11.3f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-08-03T20:53:14.539353+0000","last_change":"2024-08-03T20:53:14.539353+0000","last_active":"2024-08-03T20:53:14.539353+0000","last_peered":"2024-08-03T20:53:14.539353+0000","last_clean":"2024-08-03T20:53:14.539353+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-08-03T20:53:14.539353+0000","last_undegraded":"2024-08-03T20:53:14.539353+0000","last_fullsized":"2024-08-03T20:53:14.539353+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:53:14.539353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:53:14.539353+0000","last_clean_scrub_stamp":"2024-08-03T20:53:14.539353+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.3e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-08-03T20:53:14.539353+0000","last_change":"2024-08-03T20:53:14.539353+0000","last_active":"2024-08-03T20:53:14.539353+0000","last_peered":"2024-08-03T20:53:14.539353+0000","last_clean":"2024-08-03T20:53:14.539353+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-08-03T20:53:14.539353+0000","last_undegraded":"2024-08-03T20:53:14.539353+0000","last_fullsized":"2024-08-03T20:53:14.539353+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:53:14.539353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:53:14.539353+0000","last_clean_scrub_stamp":"2024-08-03T20:53:14.539353+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.3d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-08-03T20:53:14.539353+0000","last_change":"2024-08-03T20:53:14.539353+0000","last_active":"2024-08-03T20:53:14.539353+0000","last_peered":"2024-08-03T20:53:14.539353+0000","last_clean":"2024-08-03T20:53:14.539353+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-08-03T20:53:14.539353+0000","last_undegraded":"2024-08-03T20:53:14.539353+0000","last_fullsized":"2024-08-03T20:53:14.539353+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:53:14.539353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:53:14.539353+0000","last_clean_scrub_stamp":"2024-08-03T20:53:14.539353+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.3c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-08-03T20:53:14.539353+0000","last_change":"2024-08-03T20:53:14.539353+0000","last_active":"2024-08-03T20:53:14.539353+0000","last_peered":"2024-08-03T20:53:14.539353+0000","last_clean":"2024-08-03T20:53:14.539353+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-08-03T20:53:14.539353+0000","last_undegraded":"2024-08-03T20:53:14.539353+0000","last_fullsized":"2024-08-03T20:53:14.539353+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:53:14.539353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:53:14.539353+0000","last_clean_scrub_stamp":"2024-08-03T20:53:14.539353+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.3b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-08-03T20:53:14.539353+0000","last_change":"2024-08-03T20:53:14.539353+0000","last_active":"2024-08-03T20:53:14.539353+0000","last_peered":"2024-08-03T20:53:14.539353+0000","last_clean":"2024-08-03T20:53:14.539353+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-08-03T20:53:14.539353+0000","last_undegraded":"2024-08-03T20:53:14.539353+0000","last_fullsized":"2024-08-03T20:53:14.539353+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:53:14.539353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:53:14.539353+0000","last_clean_scrub_stamp":"2024-08-03T20:53:14.539353+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.3a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-08-03T20:53:14.539353+0000","last_change":"2024-08-03T20:53:14.539353+0000","last_active":"2024-08-03T20:53:14.539353+0000","last_peered":"2024-08-03T20:53:14.539353+0000","last_clean":"2024-08-03T20:53:14.539353+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-08-03T20:53:14.539353+0000","last_undegraded":"2024-08-03T20:53:14.539353+0000","last_fullsized":"2024-08-03T20:53:14.539353+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:53:14.539353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:53:14.539353+0000","last_clean_scrub_stamp":"2024-08-03T20:53:14.539353+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.39","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-08-03T20:53:14.539353+0000","last_change":"2024-08-03T20:53:14.539353+0000","last_active":"2024-08-03T20:53:14.539353+0000","last_peered":"2024-08-03T20:53:14.539353+0000","last_clean":"2024-08-03T20:53:14.539353+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-08-03T20:53:14.539353+0000","last_undegraded":"2024-08-03T20:53:14.539353+0000","last_fullsized":"2024-08-03T20:53:14.539353+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:53:14.539353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:53:14.539353+0000","last_clean_scrub_stamp":"2024-08-03T20:53:14.539353+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.38","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-08-03T20:53:14.539353+0000","last_change":"2024-08-03T20:53:14.539353+0000","last_active":"2024-08-03T20:53:14.539353+0000","last_peered":"2024-08-03T20:53:14.539353+0000","last_clean":"2024-08-03T20:53:14.539353+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-08-03T20:53:14.539353+0000","last_undegraded":"2024-08-03T20:53:14.539353+0000","last_fullsized":"2024-08-03T20:53:14.539353+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:53:14.539353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:53:14.539353+0000","last_clean_scrub_stamp":"2024-08-03T20:53:14.539353+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.37","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-08-03T20:53:14.539353+0000","last_change":"2024-08-03T20:53:14.539353+0000","last_active":"2024-08-03T20:53:14.539353+0000","last_peered":"2024-08-03T20:53:14.539353+0000","last_clean":"2024-08-03T20:53:14.539353+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-08-03T20:53:14.539353+0000","last_undegraded":"2024-08-03T20:53:14.539353+0000","last_fullsized":"2024-08-03T20:53:14.539353+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:53:14.539353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:53:14.539353+0000","last_clean_scrub_stamp":"2024-08-03T20:53:14.539353+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.36","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-08-03T20:53:14.539353+0000","last_change":"2024-08-03T20:53:14.539353+0000","last_active":"2024-08-03T20:53:14.539353+0000","last_peered":"2024-08-03T20:53:14.539353+0000","last_clean":"2024-08-03T20:53:14.539353+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-08-03T20:53:14.539353+0000","last_undegraded":"2024-08-03T20:53:14.539353+0000","last_fullsized":"2024-08-03T20:53:14.539353+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:53:14.539353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:53:14.539353+0000","last_clean_scrub_stamp":"2024-08-03T20:53:14.539353+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.35","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-08-03T20:53:14.539353+0000","last_change":"2024-08-03T20:53:14.539353+0000","last_active":"2024-08-03T20:53:14.539353+0000","last_peered":"2024-08-03T20:53:14.539353+0000","last_clean":"2024-08-03T20:53:14.539353+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-08-03T20:53:14.539353+0000","last_undegraded":"2024-08-03T20:53:14.539353+0000","last_fullsized":"2024-08-03T20:53:14.539353+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:53:14.539353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:53:14.539353+0000","last_clean_scrub_stamp":"2024-08-03T20:53:14.539353+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.34","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-08-03T20:53:14.539353+0000","last_change":"2024-08-03T20:53:14.539353+0000","last_active":"2024-08-03T20:53:14.539353+0000","last_peered":"2024-08-03T20:53:14.539353+0000","last_clean":"2024-08-03T20:53:14.539353+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-08-03T20:53:14.539353+0000","last_undegraded":"2024-08-03T20:53:14.539353+0000","last_fullsized":"2024-08-03T20:53:14.539353+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:53:14.539353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:53:14.539353+0000","last_clean_scrub_stamp":"2024-08-03T20:53:14.539353+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.33","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-08-03T20:53:14.539353+0000","last_change":"2024-08-03T20:53:14.539353+0000","last_active":"2024-08-03T20:53:14.539353+0000","last_peered":"2024-08-03T20:53:14.539353+0000","last_clean":"2024-08-03T20:53:14.539353+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-08-03T20:53:14.539353+0000","last_undegraded":"2024-08-03T20:53:14.539353+0000","last_fullsized":"2024-08-03T20:53:14.539353+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:53:14.539353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:53:14.539353+0000","last_clean_scrub_stamp":"2024-08-03T20:53:14.539353+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.32","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-08-03T20:53:14.539353+0000","last_change":"2024-08-03T20:53:14.539353+0000","last_active":"2024-08-03T20:53:14.539353+0000","last_peered":"2024-08-03T20:53:14.539353+0000","last_clean":"2024-08-03T20:53:14.539353+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-08-03T20:53:14.539353+0000","last_undegraded":"2024-08-03T20:53:14.539353+0000","last_fullsized":"2024-08-03T20:53:14.539353+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:53:14.539353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:53:14.539353+0000","last_clean_scrub_stamp":"2024-08-03T20:53:14.539353+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.31","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-08-03T20:53:14.539353+0000","last_change":"2024-08-03T20:53:14.539353+0000","last_active":"2024-08-03T20:53:14.539353+0000","last_peered":"2024-08-03T20:53:14.539353+0000","last_clean":"2024-08-03T20:53:14.539353+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-08-03T20:53:14.539353+0000","last_undegraded":"2024-08-03T20:53:14.539353+0000","last_fullsized":"2024-08-03T20:53:14.539353+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:53:14.539353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:53:14.539353+0000","last_clean_scrub_stamp":"2024-08-03T20:53:14.539353+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.30","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-08-03T20:53:14.539353+0000","last_change":"2024-08-03T20:53:14.539353+0000","last_active":"2024-08-03T20:53:14.539353+0000","last_peered":"2024-08-03T20:53:14.539353+0000","last_clean":"2024-08-03T20:53:14.539353+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-08-03T20:53:14.539353+0000","last_undegraded":"2024-08-03T20:53:14.539353+0000","last_fullsized":"2024-08-03T20:53:14.539353+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:53:14.539353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:53:14.539353+0000","last_clean_scrub_stamp":"2024-08-03T20:53:14.539353+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.2f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-08-03T20:53:14.539353+0000","last_change":"2024-08-03T20:53:14.539353+0000","last_active":"2024-08-03T20:53:14.539353+0000","last_peered":"2024-08-03T20:53:14.539353+0000","last_clean":"2024-08-03T20:53:14.539353+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-08-03T20:53:14.539353+0000","last_undegraded":"2024-08-03T20:53:14.539353+0000","last_fullsized":"2024-08-03T20:53:14.539353+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:53:14.539353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:53:14.539353+0000","last_clean_scrub_stamp":"2024-08-03T20:53:14.539353+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.2e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-08-03T20:53:14.539353+0000","last_change":"2024-08-03T20:53:14.539353+0000","last_active":"2024-08-03T20:53:14.539353+0000","last_peered":"2024-08-03T20:53:14.539353+0000","last_clean":"2024-08-03T20:53:14.539353+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-08-03T20:53:14.539353+0000","last_undegraded":"2024-08-03T20:53:14.539353+0000","last_fullsized":"2024-08-03T20:53:14.539353+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:53:14.539353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:53:14.539353+0000","last_clean_scrub_stamp":"2024-08-03T20:53:14.539353+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.2d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-08-03T20:53:14.539353+0000","last_change":"2024-08-03T20:53:14.539353+0000","last_active":"2024-08-03T20:53:14.539353+0000","last_peered":"2024-08-03T20:53:14.539353+0000","last_clean":"2024-08-03T20:53:14.539353+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-08-03T20:53:14.539353+0000","last_undegraded":"2024-08-03T20:53:14.539353+0000","last_fullsized":"2024-08-03T20:53:14.539353+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:53:14.539353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:53:14.539353+0000","last_clean_scrub_stamp":"2024-08-03T20:53:14.539353+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.2c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-08-03T20:53:14.539353+0000","last_change":"2024-08-03T20:53:14.539353+0000","last_active":"2024-08-03T20:53:14.539353+0000","last_peered":"2024-08-03T20:53:14.539353+0000","last_clean":"2024-08-03T20:53:14.539353+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-08-03T20:53:14.539353+0000","last_undegraded":"2024-08-03T20:53:14.539353+0000","last_fullsized":"2024-08-03T20:53:14.539353+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:53:14.539353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:53:14.539353+0000","last_clean_scrub_stamp":"2024-08-03T20:53:14.539353+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.2b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-08-03T20:53:14.539353+0000","last_change":"2024-08-03T20:53:14.539353+0000","last_active":"2024-08-03T20:53:14.539353+0000","last_peered":"2024-08-03T20:53:14.539353+0000","last_clean":"2024-08-03T20:53:14.539353+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-08-03T20:53:14.539353+0000","last_undegraded":"2024-08-03T20:53:14.539353+0000","last_fullsized":"2024-08-03T20:53:14.539353+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:53:14.539353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:53:14.539353+0000","last_clean_scrub_stamp":"2024-08-03T20:53:14.539353+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.2a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-08-03T20:53:14.539353+0000","last_change":"2024-08-03T20:53:14.539353+0000","last_active":"2024-08-03T20:53:14.539353+0000","last_peered":"2024-08-03T20:53:14.539353+0000","last_clean":"2024-08-03T20:53:14.539353+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-08-03T20:53:14.539353+0000","last_undegraded":"2024-08-03T20:53:14.539353+0000","last_fullsized":"2024-08-03T20:53:14.539353+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:53:14.539353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:53:14.539353+0000","last_clean_scrub_stamp":"2024-08-03T20:53:14.539353+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.29","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-08-03T20:53:14.539353+0000","last_change":"2024-08-03T20:53:14.539353+0000","last_active":"2024-08-03T20:53:14.539353+0000","last_peered":"2024-08-03T20:53:14.539353+0000","last_clean":"2024-08-03T20:53:14.539353+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-08-03T20:53:14.539353+0000","last_undegraded":"2024-08-03T20:53:14.539353+0000","last_fullsized":"2024-08-03T20:53:14.539353+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:53:14.539353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:53:14.539353+0000","last_clean_scrub_stamp":"2024-08-03T20:53:14.539353+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.28","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-08-03T20:53:14.539353+0000","last_change":"2024-08-03T20:53:14.539353+0000","last_active":"2024-08-03T20:53:14.539353+0000","last_peered":"2024-08-03T20:53:14.539353+0000","last_clean":"2024-08-03T20:53:14.539353+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-08-03T20:53:14.539353+0000","last_undegraded":"2024-08-03T20:53:14.539353+0000","last_fullsized":"2024-08-03T20:53:14.539353+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:53:14.539353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:53:14.539353+0000","last_clean_scrub_stamp":"2024-08-03T20:53:14.539353+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.27","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-08-03T20:53:14.539353+0000","last_change":"2024-08-03T20:53:14.539353+0000","last_active":"2024-08-03T20:53:14.539353+0000","last_peered":"2024-08-03T20:53:14.539353+0000","last_clean":"2024-08-03T20:53:14.539353+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-08-03T20:53:14.539353+0000","last_undegraded":"2024-08-03T20:53:14.539353+0000","last_fullsized":"2024-08-03T20:53:14.539353+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:53:14.539353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:53:14.539353+0000","last_clean_scrub_stamp":"2024-08-03T20:53:14.539353+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.26","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-08-03T20:53:14.539353+0000","last_change":"2024-08-03T20:53:14.539353+0000","last_active":"2024-08-03T20:53:14.539353+0000","last_peered":"2024-08-03T20:53:14.539353+0000","last_clean":"2024-08-03T20:53:14.539353+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-08-03T20:53:14.539353+0000","last_undegraded":"2024-08-03T20:53:14.539353+0000","last_fullsized":"2024-08-03T20:53:14.539353+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:53:14.539353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:53:14.539353+0000","last_clean_scrub_stamp":"2024-08-03T20:53:14.539353+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.25","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-08-03T20:53:14.539353+0000","last_change":"2024-08-03T20:53:14.539353+0000","last_active":"2024-08-03T20:53:14.539353+0000","last_peered":"2024-08-03T20:53:14.539353+0000","last_clean":"2024-08-03T20:53:14.539353+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-08-03T20:53:14.539353+0000","last_undegraded":"2024-08-03T20:53:14.539353+0000","last_fullsized":"2024-08-03T20:53:14.539353+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:53:14.539353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:53:14.539353+0000","last_clean_scrub_stamp":"2024-08-03T20:53:14.539353+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.24","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-08-03T20:53:14.539353+0000","last_change":"2024-08-03T20:53:14.539353+0000","last_active":"2024-08-03T20:53:14.539353+0000","last_peered":"2024-08-03T20:53:14.539353+0000","last_clean":"2024-08-03T20:53:14.539353+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-08-03T20:53:14.539353+0000","last_undegraded":"2024-08-03T20:53:14.539353+0000","last_fullsized":"2024-08-03T20:53:14.539353+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:53:14.539353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:53:14.539353+0000","last_clean_scrub_stamp":"2024-08-03T20:53:14.539353+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.23","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-08-03T20:53:14.539353+0000","last_change":"2024-08-03T20:53:14.539353+0000","last_active":"2024-08-03T20:53:14.539353+0000","last_peered":"2024-08-03T20:53:14.539353+0000","last_clean":"2024-08-03T20:53:14.539353+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-08-03T20:53:14.539353+0000","last_undegraded":"2024-08-03T20:53:14.539353+0000","last_fullsized":"2024-08-03T20:53:14.539353+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:53:14.539353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:53:14.539353+0000","last_clean_scrub_stamp":"2024-08-03T20:53:14.539353+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.22","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-08-03T20:53:14.539353+0000","last_change":"2024-08-03T20:53:14.539353+0000","last_active":"2024-08-03T20:53:14.539353+0000","last_peered":"2024-08-03T20:53:14.539353+0000","last_clean":"2024-08-03T20:53:14.539353+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-08-03T20:53:14.539353+0000","last_undegraded":"2024-08-03T20:53:14.539353+0000","last_fullsized":"2024-08-03T20:53:14.539353+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:53:14.539353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:53:14.539353+0000","last_clean_scrub_stamp":"2024-08-03T20:53:14.539353+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.21","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-08-03T20:53:14.539353+0000","last_change":"2024-08-03T20:53:14.539353+0000","last_active":"2024-08-03T20:53:14.539353+0000","last_peered":"2024-08-03T20:53:14.539353+0000","last_clean":"2024-08-03T20:53:14.539353+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-08-03T20:53:14.539353+0000","last_undegraded":"2024-08-03T20:53:14.539353+0000","last_fullsized":"2024-08-03T20:53:14.539353+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:53:14.539353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:53:14.539353+0000","last_clean_scrub_stamp":"2024-08-03T20:53:14.539353+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.20","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-08-03T20:53:14.539353+0000","last_change":"2024-08-03T20:53:14.539353+0000","last_active":"2024-08-03T20:53:14.539353+0000","last_peered":"2024-08-03T20:53:14.539353+0000","last_clean":"2024-08-03T20:53:14.539353+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-08-03T20:53:14.539353+0000","last_undegraded":"2024-08-03T20:53:14.539353+0000","last_fullsized":"2024-08-03T20:53:14.539353+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:53:14.539353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:53:14.539353+0000","last_clean_scrub_stamp":"2024-08-03T20:53:14.539353+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-08-03T20:53:14.539353+0000","last_change":"2024-08-03T20:53:14.539353+0000","last_active":"2024-08-03T20:53:14.539353+0000","last_peered":"2024-08-03T20:53:14.539353+0000","last_clean":"2024-08-03T20:53:14.539353+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-08-03T20:53:14.539353+0000","last_undegraded":"2024-08-03T20:53:14.539353+0000","last_fullsized":"2024-08-03T20:53:14.539353+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:53:14.539353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:53:14.539353+0000","last_clean_scrub_stamp":"2024-08-03T20:53:14.539353+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.1f","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-08-03T20:53:13.546549+0000","last_change":"2024-08-03T20:53:13.546549+0000","last_active":"2024-08-03T20:53:13.536413+0000","last_peered":"2024-08-03T20:53:13.536413+0000","last_clean":"2024-08-03T20:53:13.536413+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-08-03T20:53:13.546549+0000","last_undegraded":"2024-08-03T20:53:13.546549+0000","last_fullsized":"2024-08-03T20:53:13.546549+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:53:13.536413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:53:13.536413+0000","last_clean_scrub_stamp":"2024-08-03T20:53:13.536413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-08-03T20:53:14.539353+0000","last_change":"2024-08-03T20:53:14.539353+0000","last_active":"2024-08-03T20:53:14.539353+0000","last_peered":"2024-08-03T20:53:14.539353+0000","last_clean":"2024-08-03T20:53:14.539353+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-08-03T20:53:14.539353+0000","last_undegraded":"2024-08-03T20:53:14.539353+0000","last_fullsized":"2024-08-03T20:53:14.539353+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:53:14.539353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:53:14.539353+0000","last_clean_scrub_stamp":"2024-08-03T20:53:14.539353+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.1e","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-08-03T20:53:13.550339+0000","last_change":"2024-08-03T20:53:13.550339+0000","last_active":"2024-08-03T20:53:13.536413+0000","last_peered":"2024-08-03T20:53:13.536413+0000","last_clean":"2024-08-03T20:53:13.536413+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-08-03T20:53:13.550339+0000","last_undegraded":"2024-08-03T20:53:13.550339+0000","last_fullsized":"2024-08-03T20:53:13.550339+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:53:13.536413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:53:13.536413+0000","last_clean_scrub_stamp":"2024-08-03T20:53:13.536413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-08-03T20:53:14.539353+0000","last_change":"2024-08-03T20:53:14.539353+0000","last_active":"2024-08-03T20:53:14.539353+0000","last_peered":"2024-08-03T20:53:14.539353+0000","last_clean":"2024-08-03T20:53:14.539353+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-08-03T20:53:14.539353+0000","last_undegraded":"2024-08-03T20:53:14.539353+0000","last_fullsized":"2024-08-03T20:53:14.539353+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:53:14.539353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:53:14.539353+0000","last_clean_scrub_stamp":"2024-08-03T20:53:14.539353+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-08-03T20:53:13.536413+0000","last_change":"2024-08-03T20:53:13.536413+0000","last_active":"2024-08-03T20:53:13.536413+0000","last_peered":"2024-08-03T20:53:13.536413+0000","last_clean":"2024-08-03T20:53:13.536413+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-08-03T20:53:13.536413+0000","last_undegraded":"2024-08-03T20:53:13.536413+0000","last_fullsized":"2024-08-03T20:53:13.536413+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:53:13.536413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:53:13.536413+0000","last_clean_scrub_stamp":"2024-08-03T20:53:13.536413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-08-03T20:53:14.539353+0000","last_change":"2024-08-03T20:53:14.539353+0000","last_active":"2024-08-03T20:53:14.539353+0000","last_peered":"2024-08-03T20:53:14.539353+0000","last_clean":"2024-08-03T20:53:14.539353+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-08-03T20:53:14.539353+0000","last_undegraded":"2024-08-03T20:53:14.539353+0000","last_fullsized":"2024-08-03T20:53:14.539353+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:53:14.539353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:53:14.539353+0000","last_clean_scrub_stamp":"2024-08-03T20:53:14.539353+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.1c","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-08-03T20:53:13.547327+0000","last_change":"2024-08-03T20:53:13.547327+0000","last_active":"2024-08-03T20:53:13.536413+0000","last_peered":"2024-08-03T20:53:13.536413+0000","last_clean":"2024-08-03T20:53:13.536413+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-08-03T20:53:13.547327+0000","last_undegraded":"2024-08-03T20:53:13.547327+0000","last_fullsized":"2024-08-03T20:53:13.547327+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:53:13.536413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:53:13.536413+0000","last_clean_scrub_stamp":"2024-08-03T20:53:13.536413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-08-03T20:53:14.539353+0000","last_change":"2024-08-03T20:53:14.539353+0000","last_active":"2024-08-03T20:53:14.539353+0000","last_peered":"2024-08-03T20:53:14.539353+0000","last_clean":"2024-08-03T20:53:14.539353+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-08-03T20:53:14.539353+0000","last_undegraded":"2024-08-03T20:53:14.539353+0000","last_fullsized":"2024-08-03T20:53:14.539353+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:53:14.539353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:53:14.539353+0000","last_clean_scrub_stamp":"2024-08-03T20:53:14.539353+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-08-03T20:53:13.536413+0000","last_change":"2024-08-03T20:53:13.536413+0000","last_active":"2024-08-03T20:53:13.536413+0000","last_peered":"2024-08-03T20:53:13.536413+0000","last_clean":"2024-08-03T20:53:13.536413+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-08-03T20:53:13.536413+0000","last_undegraded":"2024-08-03T20:53:13.536413+0000","last_fullsized":"2024-08-03T20:53:13.536413+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:53:13.536413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:53:13.536413+0000","last_clean_scrub_stamp":"2024-08-03T20:53:13.536413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-08-03T20:53:14.539353+0000","last_change":"2024-08-03T20:53:14.539353+0000","last_active":"2024-08-03T20:53:14.539353+0000","last_peered":"2024-08-03T20:53:14.539353+0000","last_clean":"2024-08-03T20:53:14.539353+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-08-03T20:53:14.539353+0000","last_undegraded":"2024-08-03T20:53:14.539353+0000","last_fullsized":"2024-08-03T20:53:14.539353+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:53:14.539353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:53:14.539353+0000","last_clean_scrub_stamp":"2024-08-03T20:53:14.539353+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.1a","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-08-03T20:53:13.548733+0000","last_change":"2024-08-03T20:53:13.548733+0000","last_active":"2024-08-03T20:53:13.536413+0000","last_peered":"2024-08-03T20:53:13.536413+0000","last_clean":"2024-08-03T20:53:13.536413+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-08-03T20:53:13.548733+0000","last_undegraded":"2024-08-03T20:53:13.548733+0000","last_fullsized":"2024-08-03T20:53:13.548733+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:53:13.536413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:53:13.536413+0000","last_clean_scrub_stamp":"2024-08-03T20:53:13.536413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-08-03T20:53:14.539353+0000","last_change":"2024-08-03T20:53:14.539353+0000","last_active":"2024-08-03T20:53:14.539353+0000","last_peered":"2024-08-03T20:53:14.539353+0000","last_clean":"2024-08-03T20:53:14.539353+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-08-03T20:53:14.539353+0000","last_undegraded":"2024-08-03T20:53:14.539353+0000","last_fullsized":"2024-08-03T20:53:14.539353+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:53:14.539353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:53:14.539353+0000","last_clean_scrub_stamp":"2024-08-03T20:53:14.539353+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.19","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-08-03T20:53:13.543650+0000","last_change":"2024-08-03T20:53:13.543650+0000","last_active":"2024-08-03T20:53:13.536413+0000","last_peered":"2024-08-03T20:53:13.536413+0000","last_clean":"2024-08-03T20:53:13.536413+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-08-03T20:53:13.543650+0000","last_undegraded":"2024-08-03T20:53:13.543650+0000","last_fullsized":"2024-08-03T20:53:13.543650+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:53:13.536413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:53:13.536413+0000","last_clean_scrub_stamp":"2024-08-03T20:53:13.536413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-08-03T20:53:14.539353+0000","last_change":"2024-08-03T20:53:14.539353+0000","last_active":"2024-08-03T20:53:14.539353+0000","last_peered":"2024-08-03T20:53:14.539353+0000","last_clean":"2024-08-03T20:53:14.539353+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-08-03T20:53:14.539353+0000","last_undegraded":"2024-08-03T20:53:14.539353+0000","last_fullsized":"2024-08-03T20:53:14.539353+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:53:14.539353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:53:14.539353+0000","last_clean_scrub_stamp":"2024-08-03T20:53:14.539353+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-08-03T20:53:13.536413+0000","last_change":"2024-08-03T20:53:13.536413+0000","last_active":"2024-08-03T20:53:13.536413+0000","last_peered":"2024-08-03T20:53:13.536413+0000","last_clean":"2024-08-03T20:53:13.536413+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-08-03T20:53:13.536413+0000","last_undegraded":"2024-08-03T20:53:13.536413+0000","last_fullsized":"2024-08-03T20:53:13.536413+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:53:13.536413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:53:13.536413+0000","last_clean_scrub_stamp":"2024-08-03T20:53:13.536413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-08-03T20:53:14.539353+0000","last_change":"2024-08-03T20:53:14.539353+0000","last_active":"2024-08-03T20:53:14.539353+0000","last_peered":"2024-08-03T20:53:14.539353+0000","last_clean":"2024-08-03T20:53:14.539353+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-08-03T20:53:14.539353+0000","last_undegraded":"2024-08-03T20:53:14.539353+0000","last_fullsized":"2024-08-03T20:53:14.539353+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:53:14.539353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:53:14.539353+0000","last_clean_scrub_stamp":"2024-08-03T20:53:14.539353+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-08-03T20:53:13.536413+0000","last_change":"2024-08-03T20:53:13.536413+0000","last_active":"2024-08-03T20:53:13.536413+0000","last_peered":"2024-08-03T20:53:13.536413+0000","last_clean":"2024-08-03T20:53:13.536413+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-08-03T20:53:13.536413+0000","last_undegraded":"2024-08-03T20:53:13.536413+0000","last_fullsized":"2024-08-03T20:53:13.536413+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:53:13.536413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:53:13.536413+0000","last_clean_scrub_stamp":"2024-08-03T20:53:13.536413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-08-03T20:53:14.539353+0000","last_change":"2024-08-03T20:53:14.539353+0000","last_active":"2024-08-03T20:53:14.539353+0000","last_peered":"2024-08-03T20:53:14.539353+0000","last_clean":"2024-08-03T20:53:14.539353+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-08-03T20:53:14.539353+0000","last_undegraded":"2024-08-03T20:53:14.539353+0000","last_fullsized":"2024-08-03T20:53:14.539353+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:53:14.539353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:53:14.539353+0000","last_clean_scrub_stamp":"2024-08-03T20:53:14.539353+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.16","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-08-03T20:53:13.548343+0000","last_change":"2024-08-03T20:53:13.548343+0000","last_active":"2024-08-03T20:53:13.536413+0000","last_peered":"2024-08-03T20:53:13.536413+0000","last_clean":"2024-08-03T20:53:13.536413+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-08-03T20:53:13.548343+0000","last_undegraded":"2024-08-03T20:53:13.548343+0000","last_fullsized":"2024-08-03T20:53:13.548343+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:53:13.536413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:53:13.536413+0000","last_clean_scrub_stamp":"2024-08-03T20:53:13.536413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-08-03T20:53:14.539353+0000","last_change":"2024-08-03T20:53:14.539353+0000","last_active":"2024-08-03T20:53:14.539353+0000","last_peered":"2024-08-03T20:53:14.539353+0000","last_clean":"2024-08-03T20:53:14.539353+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-08-03T20:53:14.539353+0000","last_undegraded":"2024-08-03T20:53:14.539353+0000","last_fullsized":"2024-08-03T20:53:14.539353+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:53:14.539353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:53:14.539353+0000","last_clean_scrub_stamp":"2024-08-03T20:53:14.539353+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-08-03T20:53:13.536413+0000","last_change":"2024-08-03T20:53:13.536413+0000","last_active":"2024-08-03T20:53:13.536413+0000","last_peered":"2024-08-03T20:53:13.536413+0000","last_clean":"2024-08-03T20:53:13.536413+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-08-03T20:53:13.536413+0000","last_undegraded":"2024-08-03T20:53:13.536413+0000","last_fullsized":"2024-08-03T20:53:13.536413+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:53:13.536413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:53:13.536413+0000","last_clean_scrub_stamp":"2024-08-03T20:53:13.536413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-08-03T20:53:14.539353+0000","last_change":"2024-08-03T20:53:14.539353+0000","last_active":"2024-08-03T20:53:14.539353+0000","last_peered":"2024-08-03T20:53:14.539353+0000","last_clean":"2024-08-03T20:53:14.539353+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-08-03T20:53:14.539353+0000","last_undegraded":"2024-08-03T20:53:14.539353+0000","last_fullsized":"2024-08-03T20:53:14.539353+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:53:14.539353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:53:14.539353+0000","last_clean_scrub_stamp":"2024-08-03T20:53:14.539353+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.14","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-08-03T20:53:13.544040+0000","last_change":"2024-08-03T20:53:13.544040+0000","last_active":"2024-08-03T20:53:13.536413+0000","last_peered":"2024-08-03T20:53:13.536413+0000","last_clean":"2024-08-03T20:53:13.536413+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-08-03T20:53:13.544040+0000","last_undegraded":"2024-08-03T20:53:13.544040+0000","last_fullsized":"2024-08-03T20:53:13.544040+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:53:13.536413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:53:13.536413+0000","last_clean_scrub_stamp":"2024-08-03T20:53:13.536413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-08-03T20:53:14.539353+0000","last_change":"2024-08-03T20:53:14.539353+0000","last_active":"2024-08-03T20:53:14.539353+0000","last_peered":"2024-08-03T20:53:14.539353+0000","last_clean":"2024-08-03T20:53:14.539353+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-08-03T20:53:14.539353+0000","last_undegraded":"2024-08-03T20:53:14.539353+0000","last_fullsized":"2024-08-03T20:53:14.539353+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:53:14.539353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:53:14.539353+0000","last_clean_scrub_stamp":"2024-08-03T20:53:14.539353+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.13","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-08-03T20:53:13.543660+0000","last_change":"2024-08-03T20:53:13.543660+0000","last_active":"2024-08-03T20:53:13.536413+0000","last_peered":"2024-08-03T20:53:13.536413+0000","last_clean":"2024-08-03T20:53:13.536413+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-08-03T20:53:13.543660+0000","last_undegraded":"2024-08-03T20:53:13.543660+0000","last_fullsized":"2024-08-03T20:53:13.543660+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:53:13.536413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:53:13.536413+0000","last_clean_scrub_stamp":"2024-08-03T20:53:13.536413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-08-03T20:53:14.539353+0000","last_change":"2024-08-03T20:53:14.539353+0000","last_active":"2024-08-03T20:53:14.539353+0000","last_peered":"2024-08-03T20:53:14.539353+0000","last_clean":"2024-08-03T20:53:14.539353+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-08-03T20:53:14.539353+0000","last_undegraded":"2024-08-03T20:53:14.539353+0000","last_fullsized":"2024-08-03T20:53:14.539353+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:53:14.539353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:53:14.539353+0000","last_clean_scrub_stamp":"2024-08-03T20:53:14.539353+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.12","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-08-03T20:53:13.548099+0000","last_change":"2024-08-03T20:53:13.548099+0000","last_active":"2024-08-03T20:53:13.536413+0000","last_peered":"2024-08-03T20:53:13.536413+0000","last_clean":"2024-08-03T20:53:13.536413+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-08-03T20:53:13.548099+0000","last_undegraded":"2024-08-03T20:53:13.548099+0000","last_fullsized":"2024-08-03T20:53:13.548099+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:53:13.536413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:53:13.536413+0000","last_clean_scrub_stamp":"2024-08-03T20:53:13.536413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-08-03T20:53:14.539353+0000","last_change":"2024-08-03T20:53:14.539353+0000","last_active":"2024-08-03T20:53:14.539353+0000","last_peered":"2024-08-03T20:53:14.539353+0000","last_clean":"2024-08-03T20:53:14.539353+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-08-03T20:53:14.539353+0000","last_undegraded":"2024-08-03T20:53:14.539353+0000","last_fullsized":"2024-08-03T20:53:14.539353+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:53:14.539353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:53:14.539353+0000","last_clean_scrub_stamp":"2024-08-03T20:53:14.539353+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.11","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-08-03T20:53:13.549530+0000","last_change":"2024-08-03T20:53:13.549530+0000","last_active":"2024-08-03T20:53:13.536413+0000","last_peered":"2024-08-03T20:53:13.536413+0000","last_clean":"2024-08-03T20:53:13.536413+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-08-03T20:53:13.549530+0000","last_undegraded":"2024-08-03T20:53:13.549530+0000","last_fullsized":"2024-08-03T20:53:13.549530+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:53:13.536413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:53:13.536413+0000","last_clean_scrub_stamp":"2024-08-03T20:53:13.536413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-08-03T20:53:14.539353+0000","last_change":"2024-08-03T20:53:14.539353+0000","last_active":"2024-08-03T20:53:14.539353+0000","last_peered":"2024-08-03T20:53:14.539353+0000","last_clean":"2024-08-03T20:53:14.539353+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-08-03T20:53:14.539353+0000","last_undegraded":"2024-08-03T20:53:14.539353+0000","last_fullsized":"2024-08-03T20:53:14.539353+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:53:14.539353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:53:14.539353+0000","last_clean_scrub_stamp":"2024-08-03T20:53:14.539353+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-08-03T20:53:13.536413+0000","last_change":"2024-08-03T20:53:13.536413+0000","last_active":"2024-08-03T20:53:13.536413+0000","last_peered":"2024-08-03T20:53:13.536413+0000","last_clean":"2024-08-03T20:53:13.536413+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-08-03T20:53:13.536413+0000","last_undegraded":"2024-08-03T20:53:13.536413+0000","last_fullsized":"2024-08-03T20:53:13.536413+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:53:13.536413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:53:13.536413+0000","last_clean_scrub_stamp":"2024-08-03T20:53:13.536413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-08-03T20:53:14.539353+0000","last_change":"2024-08-03T20:53:14.539353+0000","last_active":"2024-08-03T20:53:14.539353+0000","last_peered":"2024-08-03T20:53:14.539353+0000","last_clean":"2024-08-03T20:53:14.539353+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-08-03T20:53:14.539353+0000","last_undegraded":"2024-08-03T20:53:14.539353+0000","last_fullsized":"2024-08-03T20:53:14.539353+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:53:14.539353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:53:14.539353+0000","last_clean_scrub_stamp":"2024-08-03T20:53:14.539353+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.f","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-08-03T20:53:13.548361+0000","last_change":"2024-08-03T20:53:13.548361+0000","last_active":"2024-08-03T20:53:13.536413+0000","last_peered":"2024-08-03T20:53:13.536413+0000","last_clean":"2024-08-03T20:53:13.536413+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-08-03T20:53:13.548361+0000","last_undegraded":"2024-08-03T20:53:13.548361+0000","last_fullsized":"2024-08-03T20:53:13.548361+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:53:13.536413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:53:13.536413+0000","last_clean_scrub_stamp":"2024-08-03T20:53:13.536413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-08-03T20:53:14.539353+0000","last_change":"2024-08-03T20:53:14.539353+0000","last_active":"2024-08-03T20:53:14.539353+0000","last_peered":"2024-08-03T20:53:14.539353+0000","last_clean":"2024-08-03T20:53:14.539353+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-08-03T20:53:14.539353+0000","last_undegraded":"2024-08-03T20:53:14.539353+0000","last_fullsized":"2024-08-03T20:53:14.539353+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:53:14.539353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:53:14.539353+0000","last_clean_scrub_stamp":"2024-08-03T20:53:14.539353+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-08-03T20:53:13.536413+0000","last_change":"2024-08-03T20:53:13.536413+0000","last_active":"2024-08-03T20:53:13.536413+0000","last_peered":"2024-08-03T20:53:13.536413+0000","last_clean":"2024-08-03T20:53:13.536413+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-08-03T20:53:13.536413+0000","last_undegraded":"2024-08-03T20:53:13.536413+0000","last_fullsized":"2024-08-03T20:53:13.536413+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:53:13.536413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:53:13.536413+0000","last_clean_scrub_stamp":"2024-08-03T20:53:13.536413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-08-03T20:53:14.539353+0000","last_change":"2024-08-03T20:53:14.539353+0000","last_active":"2024-08-03T20:53:14.539353+0000","last_peered":"2024-08-03T20:53:14.539353+0000","last_clean":"2024-08-03T20:53:14.539353+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-08-03T20:53:14.539353+0000","last_undegraded":"2024-08-03T20:53:14.539353+0000","last_fullsized":"2024-08-03T20:53:14.539353+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:53:14.539353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:53:14.539353+0000","last_clean_scrub_stamp":"2024-08-03T20:53:14.539353+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.d","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-08-03T20:53:13.546480+0000","last_change":"2024-08-03T20:53:13.546480+0000","last_active":"2024-08-03T20:53:13.536413+0000","last_peered":"2024-08-03T20:53:13.536413+0000","last_clean":"2024-08-03T20:53:13.536413+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-08-03T20:53:13.546480+0000","last_undegraded":"2024-08-03T20:53:13.546480+0000","last_fullsized":"2024-08-03T20:53:13.546480+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:53:13.536413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:53:13.536413+0000","last_clean_scrub_stamp":"2024-08-03T20:53:13.536413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-08-03T20:53:14.539353+0000","last_change":"2024-08-03T20:53:14.539353+0000","last_active":"2024-08-03T20:53:14.539353+0000","last_peered":"2024-08-03T20:53:14.539353+0000","last_clean":"2024-08-03T20:53:14.539353+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-08-03T20:53:14.539353+0000","last_undegraded":"2024-08-03T20:53:14.539353+0000","last_fullsized":"2024-08-03T20:53:14.539353+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:53:14.539353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:53:14.539353+0000","last_clean_scrub_stamp":"2024-08-03T20:53:14.539353+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.c","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-08-03T20:53:13.546098+0000","last_change":"2024-08-03T20:53:13.546098+0000","last_active":"2024-08-03T20:53:13.536413+0000","last_peered":"2024-08-03T20:53:13.536413+0000","last_clean":"2024-08-03T20:53:13.536413+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-08-03T20:53:13.546098+0000","last_undegraded":"2024-08-03T20:53:13.546098+0000","last_fullsized":"2024-08-03T20:53:13.546098+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:53:13.536413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:53:13.536413+0000","last_clean_scrub_stamp":"2024-08-03T20:53:13.536413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-08-03T20:53:14.539353+0000","last_change":"2024-08-03T20:53:14.539353+0000","last_active":"2024-08-03T20:53:14.539353+0000","last_peered":"2024-08-03T20:53:14.539353+0000","last_clean":"2024-08-03T20:53:14.539353+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-08-03T20:53:14.539353+0000","last_undegraded":"2024-08-03T20:53:14.539353+0000","last_fullsized":"2024-08-03T20:53:14.539353+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:53:14.539353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:53:14.539353+0000","last_clean_scrub_stamp":"2024-08-03T20:53:14.539353+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.a","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-08-03T20:53:13.546837+0000","last_change":"2024-08-03T20:53:13.546837+0000","last_active":"2024-08-03T20:53:13.536413+0000","last_peered":"2024-08-03T20:53:13.536413+0000","last_clean":"2024-08-03T20:53:13.536413+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-08-03T20:53:13.546837+0000","last_undegraded":"2024-08-03T20:53:13.546837+0000","last_fullsized":"2024-08-03T20:53:13.546837+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:53:13.536413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:53:13.536413+0000","last_clean_scrub_stamp":"2024-08-03T20:53:13.536413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-08-03T20:53:14.539353+0000","last_change":"2024-08-03T20:53:14.539353+0000","last_active":"2024-08-03T20:53:14.539353+0000","last_peered":"2024-08-03T20:53:14.539353+0000","last_clean":"2024-08-03T20:53:14.539353+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-08-03T20:53:14.539353+0000","last_undegraded":"2024-08-03T20:53:14.539353+0000","last_fullsized":"2024-08-03T20:53:14.539353+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:53:14.539353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:53:14.539353+0000","last_clean_scrub_stamp":"2024-08-03T20:53:14.539353+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.9","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-08-03T20:53:13.548965+0000","last_change":"2024-08-03T20:53:13.548965+0000","last_active":"2024-08-03T20:53:13.536413+0000","last_peered":"2024-08-03T20:53:13.536413+0000","last_clean":"2024-08-03T20:53:13.536413+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-08-03T20:53:13.548965+0000","last_undegraded":"2024-08-03T20:53:13.548965+0000","last_fullsized":"2024-08-03T20:53:13.548965+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:53:13.536413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:53:13.536413+0000","last_clean_scrub_stamp":"2024-08-03T20:53:13.536413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-08-03T20:53:14.539353+0000","last_change":"2024-08-03T20:53:14.539353+0000","last_active":"2024-08-03T20:53:14.539353+0000","last_peered":"2024-08-03T20:53:14.539353+0000","last_clean":"2024-08-03T20:53:14.539353+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-08-03T20:53:14.539353+0000","last_undegraded":"2024-08-03T20:53:14.539353+0000","last_fullsized":"2024-08-03T20:53:14.539353+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:53:14.539353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:53:14.539353+0000","last_clean_scrub_stamp":"2024-08-03T20:53:14.539353+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.8","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-08-03T20:53:13.549654+0000","last_change":"2024-08-03T20:53:13.549654+0000","last_active":"2024-08-03T20:53:13.536413+0000","last_peered":"2024-08-03T20:53:13.536413+0000","last_clean":"2024-08-03T20:53:13.536413+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-08-03T20:53:13.549654+0000","last_undegraded":"2024-08-03T20:53:13.549654+0000","last_fullsized":"2024-08-03T20:53:13.549654+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:53:13.536413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:53:13.536413+0000","last_clean_scrub_stamp":"2024-08-03T20:53:13.536413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-08-03T20:53:14.539353+0000","last_change":"2024-08-03T20:53:14.539353+0000","last_active":"2024-08-03T20:53:14.539353+0000","last_peered":"2024-08-03T20:53:14.539353+0000","last_clean":"2024-08-03T20:53:14.539353+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-08-03T20:53:14.539353+0000","last_undegraded":"2024-08-03T20:53:14.539353+0000","last_fullsized":"2024-08-03T20:53:14.539353+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:53:14.539353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:53:14.539353+0000","last_clean_scrub_stamp":"2024-08-03T20:53:14.539353+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.7","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-08-03T20:53:13.547218+0000","last_change":"2024-08-03T20:53:13.547218+0000","last_active":"2024-08-03T20:53:13.536413+0000","last_peered":"2024-08-03T20:53:13.536413+0000","last_clean":"2024-08-03T20:53:13.536413+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-08-03T20:53:13.547218+0000","last_undegraded":"2024-08-03T20:53:13.547218+0000","last_fullsized":"2024-08-03T20:53:13.547218+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:53:13.536413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:53:13.536413+0000","last_clean_scrub_stamp":"2024-08-03T20:53:13.536413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-08-03T20:53:14.539353+0000","last_change":"2024-08-03T20:53:14.539353+0000","last_active":"2024-08-03T20:53:14.539353+0000","last_peered":"2024-08-03T20:53:14.539353+0000","last_clean":"2024-08-03T20:53:14.539353+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-08-03T20:53:14.539353+0000","last_undegraded":"2024-08-03T20:53:14.539353+0000","last_fullsized":"2024-08-03T20:53:14.539353+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:53:14.539353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:53:14.539353+0000","last_clean_scrub_stamp":"2024-08-03T20:53:14.539353+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.6","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-08-03T20:53:13.548978+0000","last_change":"2024-08-03T20:53:13.548978+0000","last_active":"2024-08-03T20:53:13.536413+0000","last_peered":"2024-08-03T20:53:13.536413+0000","last_clean":"2024-08-03T20:53:13.536413+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-08-03T20:53:13.548978+0000","last_undegraded":"2024-08-03T20:53:13.548978+0000","last_fullsized":"2024-08-03T20:53:13.548978+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:53:13.536413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:53:13.536413+0000","last_clean_scrub_stamp":"2024-08-03T20:53:13.536413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-08-03T20:53:14.539353+0000","last_change":"2024-08-03T20:53:14.539353+0000","last_active":"2024-08-03T20:53:14.539353+0000","last_peered":"2024-08-03T20:53:14.539353+0000","last_clean":"2024-08-03T20:53:14.539353+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-08-03T20:53:14.539353+0000","last_undegraded":"2024-08-03T20:53:14.539353+0000","last_fullsized":"2024-08-03T20:53:14.539353+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:53:14.539353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:53:14.539353+0000","last_clean_scrub_stamp":"2024-08-03T20:53:14.539353+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-08-03T20:53:13.536413+0000","last_change":"2024-08-03T20:53:13.536413+0000","last_active":"2024-08-03T20:53:13.536413+0000","last_peered":"2024-08-03T20:53:13.536413+0000","last_clean":"2024-08-03T20:53:13.536413+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-08-03T20:53:13.536413+0000","last_undegraded":"2024-08-03T20:53:13.536413+0000","last_fullsized":"2024-08-03T20:53:13.536413+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:53:13.536413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:53:13.536413+0000","last_clean_scrub_stamp":"2024-08-03T20:53:13.536413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-08-03T20:53:14.539353+0000","last_change":"2024-08-03T20:53:14.539353+0000","last_active":"2024-08-03T20:53:14.539353+0000","last_peered":"2024-08-03T20:53:14.539353+0000","last_clean":"2024-08-03T20:53:14.539353+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-08-03T20:53:14.539353+0000","last_undegraded":"2024-08-03T20:53:14.539353+0000","last_fullsized":"2024-08-03T20:53:14.539353+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:53:14.539353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:53:14.539353+0000","last_clean_scrub_stamp":"2024-08-03T20:53:14.539353+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.4","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-08-03T20:53:13.550020+0000","last_change":"2024-08-03T20:53:13.550020+0000","last_active":"2024-08-03T20:53:13.536413+0000","last_peered":"2024-08-03T20:53:13.536413+0000","last_clean":"2024-08-03T20:53:13.536413+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-08-03T20:53:13.550020+0000","last_undegraded":"2024-08-03T20:53:13.550020+0000","last_fullsized":"2024-08-03T20:53:13.550020+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:53:13.536413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:53:13.536413+0000","last_clean_scrub_stamp":"2024-08-03T20:53:13.536413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-08-03T20:53:14.539353+0000","last_change":"2024-08-03T20:53:14.539353+0000","last_active":"2024-08-03T20:53:14.539353+0000","last_peered":"2024-08-03T20:53:14.539353+0000","last_clean":"2024-08-03T20:53:14.539353+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-08-03T20:53:14.539353+0000","last_undegraded":"2024-08-03T20:53:14.539353+0000","last_fullsized":"2024-08-03T20:53:14.539353+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:53:14.539353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:53:14.539353+0000","last_clean_scrub_stamp":"2024-08-03T20:53:14.539353+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-08-03T20:53:13.536413+0000","last_change":"2024-08-03T20:53:13.536413+0000","last_active":"2024-08-03T20:53:13.536413+0000","last_peered":"2024-08-03T20:53:13.536413+0000","last_clean":"2024-08-03T20:53:13.536413+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-08-03T20:53:13.536413+0000","last_undegraded":"2024-08-03T20:53:13.536413+0000","last_fullsized":"2024-08-03T20:53:13.536413+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:53:13.536413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:53:13.536413+0000","last_clean_scrub_stamp":"2024-08-03T20:53:13.536413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-08-03T20:53:14.539353+0000","last_change":"2024-08-03T20:53:14.539353+0000","last_active":"2024-08-03T20:53:14.539353+0000","last_peered":"2024-08-03T20:53:14.539353+0000","last_clean":"2024-08-03T20:53:14.539353+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-08-03T20:53:14.539353+0000","last_undegraded":"2024-08-03T20:53:14.539353+0000","last_fullsized":"2024-08-03T20:53:14.539353+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:53:14.539353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:53:14.539353+0000","last_clean_scrub_stamp":"2024-08-03T20:53:14.539353+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.2","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-08-03T20:53:13.543681+0000","last_change":"2024-08-03T20:53:13.543681+0000","last_active":"2024-08-03T20:53:13.536413+0000","last_peered":"2024-08-03T20:53:13.536413+0000","last_clean":"2024-08-03T20:53:13.536413+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-08-03T20:53:13.543681+0000","last_undegraded":"2024-08-03T20:53:13.543681+0000","last_fullsized":"2024-08-03T20:53:13.543681+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:53:13.536413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:53:13.536413+0000","last_clean_scrub_stamp":"2024-08-03T20:53:13.536413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-08-03T20:53:14.539353+0000","last_change":"2024-08-03T20:53:14.539353+0000","last_active":"2024-08-03T20:53:14.539353+0000","last_peered":"2024-08-03T20:53:14.539353+0000","last_clean":"2024-08-03T20:53:14.539353+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-08-03T20:53:14.539353+0000","last_undegraded":"2024-08-03T20:53:14.539353+0000","last_fullsized":"2024-08-03T20:53:14.539353+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:53:14.539353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:53:14.539353+0000","last_clean_scrub_stamp":"2024-08-03T20:53:14.539353+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-08-03T20:53:13.536413+0000","last_change":"2024-08-03T20:53:13.536413+0000","last_active":"2024-08-03T20:53:13.536413+0000","last_peered":"2024-08-03T20:53:13.536413+0000","last_clean":"2024-08-03T20:53:13.536413+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-08-03T20:53:13.536413+0000","last_undegraded":"2024-08-03T20:53:13.536413+0000","last_fullsized":"2024-08-03T20:53:13.536413+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:53:13.536413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:53:13.536413+0000","last_clean_scrub_stamp":"2024-08-03T20:53:13.536413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-08-03T20:53:14.539353+0000","last_change":"2024-08-03T20:53:14.539353+0000","last_active":"2024-08-03T20:53:14.539353+0000","last_peered":"2024-08-03T20:53:14.539353+0000","last_clean":"2024-08-03T20:53:14.539353+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-08-03T20:53:14.539353+0000","last_undegraded":"2024-08-03T20:53:14.539353+0000","last_fullsized":"2024-08-03T20:53:14.539353+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:53:14.539353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:53:14.539353+0000","last_clean_scrub_stamp":"2024-08-03T20:53:14.539353+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.0","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-08-03T20:53:13.546678+0000","last_change":"2024-08-03T20:53:13.546678+0000","last_active":"2024-08-03T20:53:13.536413+0000","last_peered":"2024-08-03T20:53:13.536413+0000","last_clean":"2024-08-03T20:53:13.536413+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-08-03T20:53:13.546678+0000","last_undegraded":"2024-08-03T20:53:13.546678+0000","last_fullsized":"2024-08-03T20:53:13.546678+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:53:13.536413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:53:13.536413+0000","last_clean_scrub_stamp":"2024-08-03T20:53:13.536413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-08-03T20:53:14.539353+0000","last_change":"2024-08-03T20:53:14.539353+0000","last_active":"2024-08-03T20:53:14.539353+0000","last_peered":"2024-08-03T20:53:14.539353+0000","last_clean":"2024-08-03T20:53:14.539353+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-08-03T20:53:14.539353+0000","last_undegraded":"2024-08-03T20:53:14.539353+0000","last_fullsized":"2024-08-03T20:53:14.539353+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:53:14.539353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:53:14.539353+0000","last_clean_scrub_stamp":"2024-08-03T20:53:14.539353+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.b","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-08-03T20:53:13.548204+0000","last_change":"2024-08-03T20:53:13.548204+0000","last_active":"2024-08-03T20:53:13.536413+0000","last_peered":"2024-08-03T20:53:13.536413+0000","last_clean":"2024-08-03T20:53:13.536413+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-08-03T20:53:13.548204+0000","last_undegraded":"2024-08-03T20:53:13.548204+0000","last_fullsized":"2024-08-03T20:53:13.548204+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:53:13.536413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:53:13.536413+0000","last_clean_scrub_stamp":"2024-08-03T20:53:13.536413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"16'72","reported_seq":195,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-08-03T20:53:13.546940+0000","last_change":"2024-08-03T20:47:30.547632+0000","last_active":"2024-08-03T20:53:13.546940+0000","last_peered":"2024-08-03T20:53:13.546940+0000","last_clean":"2024-08-03T20:53:13.546940+0000","last_became_active":"2024-08-03T20:47:30.546255+0000","last_became_peered":"2024-08-03T20:47:30.546255+0000","last_unstale":"2024-08-03T20:53:13.546940+0000","last_undegraded":"2024-08-03T20:53:13.546940+0000","last_fullsized":"2024-08-03T20:53:13.546940+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:47:28.874451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:47:28.874451+0000","last_clean_scrub_stamp":"2024-08-03T20:47:28.874451+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T00:19:40.548812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":11,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0},{"poolid":10,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":44,"acting":44,"num_store_stats":5},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509512,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32548,"kb_used_data":1368,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339292,"statfs":{"total":96636764160,"available":96603435008,"internally_reserved":0,"allocated":1400832,"data_stored":927974,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":6,"up_from":15,"seq":64424509513,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31976,"kb_used_data":788,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339864,"statfs":{"total":96636764160,"available":96604020736,"internally_reserved":0,"allocated":806912,"data_stored":337606,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":5,"up_from":15,"seq":64424509513,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31980,"kb_used_data":788,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339860,"statfs":{"total":96636764160,"available":96604016640,"internally_reserved":0,"allocated":806912,"data_stored":337606,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509512,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31972,"kb_used_data":788,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339868,"statfs":{"total":96636764160,"available":96604024832,"internally_reserved":0,"allocated":806912,"data_stored":337606,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509514,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36712,"kb_used_data":1368,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335128,"statfs":{"total":96636764160,"available":96599171072,"internally_reserved":0,"allocated":1400832,"data_stored":927974,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542217,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":31960,"kb_used_data":776,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339880,"statfs":{"total":96636764160,"available":96604037120,"internally_reserved":0,"allocated":794624,"data_stored":331789,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":14,"seq":60129542215,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":31956,"kb_used_data":776,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339884,"statfs":{"total":96636764160,"available":96604041216,"internally_reserved":0,"allocated":794624,"data_stored":331789,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":14,"seq":60129542216,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":31932,"kb_used_data":752,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339908,"statfs":{"total":96636764160,"available":96604065792,"internally_reserved":0,"allocated":770048,"data_stored":321675,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-08-03T20:53:20.084 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-08-03T20:53:20.350 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:20.349+0000 7f6314c00640 1 -- 172.21.15.32:0/3277708593 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f62f4004800 msgr2=0x7f62f4004c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:53:20.350 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:20.349+0000 7f6314c00640 1 --2- 172.21.15.32:0/3277708593 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f62f4004800 0x7f62f4004c10 secure :-1 s=READY pgs=764 cs=0 l=1 rev1=1 crypto rx=0x7f62fc009b50 tx=0x7f62f4005830 comp rx=0 tx=0).stop 2024-08-03T20:53:20.351 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:20.349+0000 7f630e800640 1 -- 172.21.15.32:0/3277708593 reap_dead start 2024-08-03T20:53:20.351 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:20.349+0000 7f6314c00640 1 -- 172.21.15.32:0/3277708593 shutdown_connections 2024-08-03T20:53:20.351 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:20.349+0000 7f6314c00640 1 -- 172.21.15.32:0/3277708593 >> 172.21.15.32:0/3277708593 conn(0x7f63100cd1e0 msgr2=0x7f63100cd5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:53:20.351 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:20.349+0000 7f6314c00640 1 -- 172.21.15.32:0/3277708593 shutdown_connections 2024-08-03T20:53:20.351 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:20.349+0000 7f6314c00640 1 -- 172.21.15.32:0/3277708593 wait complete. 2024-08-03T20:53:20.352 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:20.352+0000 7f6314c00640 1 Processor -- start 2024-08-03T20:53:20.353 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:20.352+0000 7f6314c00640 1 -- start start 2024-08-03T20:53:20.356 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:20.353+0000 7f6314c00640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f62f4004800 0x7f631015bc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:53:20.356 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:20.353+0000 7f6314c00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f631014f530 0x7f631014f940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:53:20.356 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:20.353+0000 7f6314c00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f631015fc20 0x7f6310160030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:53:20.357 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:20.353+0000 7f6314c00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f631015cfa0 con 0x7f631014f530 2024-08-03T20:53:20.357 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:20.353+0000 7f6314c00640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f63100b0080 con 0x7f631015fc20 2024-08-03T20:53:20.357 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:20.353+0000 7f6314c00640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7f631015d120 con 0x7f62f4004800 2024-08-03T20:53:20.357 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:20.354+0000 7f630d400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f631014f530 0x7f631014f940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:53:20.357 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:20.354+0000 7f630d400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f631014f530 0x7f631014f940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:45498/0 (socket says 172.21.15.32:45498) 2024-08-03T20:53:20.357 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:20.354+0000 7f630d400640 1 -- 172.21.15.32:0/924172723 learned_addr learned my addr 172.21.15.32:0/924172723 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:53:20.358 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:20.354+0000 7f630e800640 1 --2- 172.21.15.32:0/924172723 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f631015fc20 0x7f6310160030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:53:20.358 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:20.354+0000 7f630de00640 1 --2- 172.21.15.32:0/924172723 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f62f4004800 0x7f631015bc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:53:20.358 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:20.354+0000 7f630d400640 1 -- 172.21.15.32:0/924172723 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f62f4004800 msgr2=0x7f631015bc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:53:20.359 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:20.354+0000 7f630d400640 1 --2- 172.21.15.32:0/924172723 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f62f4004800 0x7f631015bc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:53:20.359 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:20.354+0000 7f630d400640 1 -- 172.21.15.32:0/924172723 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f631015fc20 msgr2=0x7f6310160030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:53:20.359 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:20.354+0000 7f630d400640 1 --2- 172.21.15.32:0/924172723 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f631015fc20 0x7f6310160030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:53:20.360 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:20.354+0000 7f630d400640 1 -- 172.21.15.32:0/924172723 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f631015c480 con 0x7f631014f530 2024-08-03T20:53:20.360 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:20.354+0000 7f630de00640 1 --2- 172.21.15.32:0/924172723 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f62f4004800 0x7f631015bc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-03T20:53:20.360 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:20.354+0000 7f630d400640 1 --2- 172.21.15.32:0/924172723 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f631014f530 0x7f631014f940 secure :-1 s=READY pgs=765 cs=0 l=1 rev1=1 crypto rx=0x7f62fc009a70 tx=0x7f62fc009850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:53:20.360 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:20.355+0000 7f6306000640 1 -- 172.21.15.32:0/924172723 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62fc038050 con 0x7f631014f530 2024-08-03T20:53:20.360 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:20.355+0000 7f6306000640 1 -- 172.21.15.32:0/924172723 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f62fc039040 con 0x7f631014f530 2024-08-03T20:53:20.361 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:20.355+0000 7f6306000640 1 -- 172.21.15.32:0/924172723 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62fc033e40 con 0x7f631014f530 2024-08-03T20:53:20.361 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:20.355+0000 7f6314c00640 1 -- 172.21.15.32:0/924172723 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f631015ee60 con 0x7f631014f530 2024-08-03T20:53:20.361 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:20.355+0000 7f6314c00640 1 -- 172.21.15.32:0/924172723 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f631015f420 con 0x7f631014f530 2024-08-03T20:53:20.361 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:20.356+0000 7f6314c00640 1 -- 172.21.15.32:0/924172723 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f63100af930 con 0x7f631014f530 2024-08-03T20:53:20.368 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:20.357+0000 7f6306000640 1 -- 172.21.15.32:0/924172723 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f62fc02b8c0 con 0x7f631014f530 2024-08-03T20:53:20.369 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:20.358+0000 7f6306000640 1 --2- 172.21.15.32:0/924172723 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f62f0077010 0x7f62f0079520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:53:20.369 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:20.358+0000 7f6306000640 1 -- 172.21.15.32:0/924172723 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(70..70 src has 1..70) ==== 5709+0+0 (secure 0 0 0) 0x7f62fc0b9a40 con 0x7f631014f530 2024-08-03T20:53:20.369 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:20.359+0000 7f630de00640 1 --2- 172.21.15.32:0/924172723 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f62f0077010 0x7f62f0079520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:53:20.370 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:20.359+0000 7f630de00640 1 --2- 172.21.15.32:0/924172723 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f62f0077010 0x7f62f0079520 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f6300000fb0 tx=0x7f63000067a0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:53:20.370 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:20.363+0000 7f6306000640 1 -- 172.21.15.32:0/924172723 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7f62fc082710 con 0x7f631014f530 2024-08-03T20:53:20.565 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:20.564+0000 7f6314c00640 1 -- 172.21.15.32:0/924172723 --> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f631015c210 con 0x7f62f0077010 2024-08-03T20:53:20.571 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:20.570+0000 7f6306000640 1 -- 172.21.15.32:0/924172723 <== mgr.4106 v2:172.21.15.118:6800/4205950699 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+252058 (secure 0 0 0) 0x7f631015c210 con 0x7f62f0077010 2024-08-03T20:53:20.571 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-03T20:53:20.576 INFO:teuthology.orchestra.run.smithi032.stderr:dumped all 2024-08-03T20:53:20.577 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:20.573+0000 7f6314c00640 1 -- 172.21.15.32:0/924172723 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f62f0077010 msgr2=0x7f62f0079520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:53:20.578 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:20.573+0000 7f6314c00640 1 --2- 172.21.15.32:0/924172723 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f62f0077010 0x7f62f0079520 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f6300000fb0 tx=0x7f63000067a0 comp rx=0 tx=0).stop 2024-08-03T20:53:20.578 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:20.573+0000 7f6314c00640 1 -- 172.21.15.32:0/924172723 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f631014f530 msgr2=0x7f631014f940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:53:20.578 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:20.573+0000 7f6314c00640 1 --2- 172.21.15.32:0/924172723 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f631014f530 0x7f631014f940 secure :-1 s=READY pgs=765 cs=0 l=1 rev1=1 crypto rx=0x7f62fc009a70 tx=0x7f62fc009850 comp rx=0 tx=0).stop 2024-08-03T20:53:20.578 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:20.573+0000 7f6314c00640 1 -- 172.21.15.32:0/924172723 shutdown_connections 2024-08-03T20:53:20.578 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:20.573+0000 7f6314c00640 1 --2- 172.21.15.32:0/924172723 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f62f0077010 0x7f62f0079520 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:53:20.578 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:20.573+0000 7f6314c00640 1 --2- 172.21.15.32:0/924172723 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f631015fc20 0x7f6310160030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:53:20.578 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:20.573+0000 7f6314c00640 1 --2- 172.21.15.32:0/924172723 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f631014f530 0x7f631014f940 unknown :-1 s=CLOSED pgs=765 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:53:20.578 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:20.573+0000 7f6314c00640 1 --2- 172.21.15.32:0/924172723 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f62f4004800 0x7f631015bc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:53:20.578 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:20.574+0000 7f6314c00640 1 -- 172.21.15.32:0/924172723 >> 172.21.15.32:0/924172723 conn(0x7f63100cd1e0 msgr2=0x7f63100b43c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:53:20.578 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:20.574+0000 7f6314c00640 1 -- 172.21.15.32:0/924172723 shutdown_connections 2024-08-03T20:53:20.578 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:20.574+0000 7f6314c00640 1 -- 172.21.15.32:0/924172723 wait complete. 2024-08-03T20:53:20.598 INFO:teuthology.orchestra.run.smithi032.stdout:{"pg_ready":true,"pg_map":{"version":239,"stamp":"2024-08-03T20:53:20.320848+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":146,"num_write_kb":1642,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":111,"ondisk_log_size":111,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":186,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":261440,"kb_used_data":7816,"kb_used_omap":12,"kb_used_meta":253491,"kb_avail":754713280,"statfs":{"total":773094113280,"available":772826398720,"internally_reserved":0,"allocated":8003584,"data_stored":4041204,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":259575345},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":31,"apply_latency_ms":31,"commit_latency_ns":31000000,"apply_latency_ns":31000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001615"},"pg_stats":[{"pgid":"11.3f","version":"0'0","reported_seq":13,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-08-03T20:53:17.072484+0000","last_change":"2024-08-03T20:53:15.576208+0000","last_active":"2024-08-03T20:53:17.072484+0000","last_peered":"2024-08-03T20:53:17.072484+0000","last_clean":"2024-08-03T20:53:17.072484+0000","last_became_active":"2024-08-03T20:53:15.575756+0000","last_became_peered":"2024-08-03T20:53:15.575756+0000","last_unstale":"2024-08-03T20:53:17.072484+0000","last_undegraded":"2024-08-03T20:53:17.072484+0000","last_fullsized":"2024-08-03T20:53:17.072484+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:53:14.539353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:53:14.539353+0000","last_clean_scrub_stamp":"2024-08-03T20:53:14.539353+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-04T22:40:41.888071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.3e","version":"0'0","reported_seq":13,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-08-03T20:53:16.569627+0000","last_change":"2024-08-03T20:53:15.585136+0000","last_active":"2024-08-03T20:53:16.569627+0000","last_peered":"2024-08-03T20:53:16.569627+0000","last_clean":"2024-08-03T20:53:16.569627+0000","last_became_active":"2024-08-03T20:53:15.584555+0000","last_became_peered":"2024-08-03T20:53:15.584555+0000","last_unstale":"2024-08-03T20:53:16.569627+0000","last_undegraded":"2024-08-03T20:53:16.569627+0000","last_fullsized":"2024-08-03T20:53:16.569627+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:53:14.539353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:53:14.539353+0000","last_clean_scrub_stamp":"2024-08-03T20:53:14.539353+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T04:34:19.841129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.3d","version":"0'0","reported_seq":13,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-08-03T20:53:17.073200+0000","last_change":"2024-08-03T20:53:15.581838+0000","last_active":"2024-08-03T20:53:17.073200+0000","last_peered":"2024-08-03T20:53:17.073200+0000","last_clean":"2024-08-03T20:53:17.073200+0000","last_became_active":"2024-08-03T20:53:15.581505+0000","last_became_peered":"2024-08-03T20:53:15.581505+0000","last_unstale":"2024-08-03T20:53:17.073200+0000","last_undegraded":"2024-08-03T20:53:17.073200+0000","last_fullsized":"2024-08-03T20:53:17.073200+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:53:14.539353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:53:14.539353+0000","last_clean_scrub_stamp":"2024-08-03T20:53:14.539353+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T01:28:11.178891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.3c","version":"0'0","reported_seq":13,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-08-03T20:53:17.072337+0000","last_change":"2024-08-03T20:53:15.585565+0000","last_active":"2024-08-03T20:53:17.072337+0000","last_peered":"2024-08-03T20:53:17.072337+0000","last_clean":"2024-08-03T20:53:17.072337+0000","last_became_active":"2024-08-03T20:53:15.585216+0000","last_became_peered":"2024-08-03T20:53:15.585216+0000","last_unstale":"2024-08-03T20:53:17.072337+0000","last_undegraded":"2024-08-03T20:53:17.072337+0000","last_fullsized":"2024-08-03T20:53:17.072337+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:53:14.539353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:53:14.539353+0000","last_clean_scrub_stamp":"2024-08-03T20:53:14.539353+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-04T21:36:36.797962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.3b","version":"0'0","reported_seq":13,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-08-03T20:53:17.071687+0000","last_change":"2024-08-03T20:53:15.574310+0000","last_active":"2024-08-03T20:53:17.071687+0000","last_peered":"2024-08-03T20:53:17.071687+0000","last_clean":"2024-08-03T20:53:17.071687+0000","last_became_active":"2024-08-03T20:53:15.573890+0000","last_became_peered":"2024-08-03T20:53:15.573890+0000","last_unstale":"2024-08-03T20:53:17.071687+0000","last_undegraded":"2024-08-03T20:53:17.071687+0000","last_fullsized":"2024-08-03T20:53:17.071687+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:53:14.539353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:53:14.539353+0000","last_clean_scrub_stamp":"2024-08-03T20:53:14.539353+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-04T22:53:09.850158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.3a","version":"0'0","reported_seq":13,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-08-03T20:53:16.569528+0000","last_change":"2024-08-03T20:53:15.572962+0000","last_active":"2024-08-03T20:53:16.569528+0000","last_peered":"2024-08-03T20:53:16.569528+0000","last_clean":"2024-08-03T20:53:16.569528+0000","last_became_active":"2024-08-03T20:53:15.569616+0000","last_became_peered":"2024-08-03T20:53:15.569616+0000","last_unstale":"2024-08-03T20:53:16.569528+0000","last_undegraded":"2024-08-03T20:53:16.569528+0000","last_fullsized":"2024-08-03T20:53:16.569528+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:53:14.539353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:53:14.539353+0000","last_clean_scrub_stamp":"2024-08-03T20:53:14.539353+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T03:20:03.716293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.39","version":"0'0","reported_seq":13,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-08-03T20:53:17.072465+0000","last_change":"2024-08-03T20:53:15.576961+0000","last_active":"2024-08-03T20:53:17.072465+0000","last_peered":"2024-08-03T20:53:17.072465+0000","last_clean":"2024-08-03T20:53:17.072465+0000","last_became_active":"2024-08-03T20:53:15.576669+0000","last_became_peered":"2024-08-03T20:53:15.576669+0000","last_unstale":"2024-08-03T20:53:17.072465+0000","last_undegraded":"2024-08-03T20:53:17.072465+0000","last_fullsized":"2024-08-03T20:53:17.072465+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:53:14.539353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:53:14.539353+0000","last_clean_scrub_stamp":"2024-08-03T20:53:14.539353+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T01:00:54.007283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.38","version":"0'0","reported_seq":13,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-08-03T20:53:16.568815+0000","last_change":"2024-08-03T20:53:15.573606+0000","last_active":"2024-08-03T20:53:16.568815+0000","last_peered":"2024-08-03T20:53:16.568815+0000","last_clean":"2024-08-03T20:53:16.568815+0000","last_became_active":"2024-08-03T20:53:15.572445+0000","last_became_peered":"2024-08-03T20:53:15.572445+0000","last_unstale":"2024-08-03T20:53:16.568815+0000","last_undegraded":"2024-08-03T20:53:16.568815+0000","last_fullsized":"2024-08-03T20:53:16.568815+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:53:14.539353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:53:14.539353+0000","last_clean_scrub_stamp":"2024-08-03T20:53:14.539353+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T05:41:42.534210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.37","version":"0'0","reported_seq":13,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-08-03T20:53:17.072372+0000","last_change":"2024-08-03T20:53:15.585677+0000","last_active":"2024-08-03T20:53:17.072372+0000","last_peered":"2024-08-03T20:53:17.072372+0000","last_clean":"2024-08-03T20:53:17.072372+0000","last_became_active":"2024-08-03T20:53:15.584442+0000","last_became_peered":"2024-08-03T20:53:15.584442+0000","last_unstale":"2024-08-03T20:53:17.072372+0000","last_undegraded":"2024-08-03T20:53:17.072372+0000","last_fullsized":"2024-08-03T20:53:17.072372+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:53:14.539353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:53:14.539353+0000","last_clean_scrub_stamp":"2024-08-03T20:53:14.539353+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-04T23:43:28.202164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.36","version":"0'0","reported_seq":13,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-08-03T20:53:16.571114+0000","last_change":"2024-08-03T20:53:15.587816+0000","last_active":"2024-08-03T20:53:16.571114+0000","last_peered":"2024-08-03T20:53:16.571114+0000","last_clean":"2024-08-03T20:53:16.571114+0000","last_became_active":"2024-08-03T20:53:15.582908+0000","last_became_peered":"2024-08-03T20:53:15.582908+0000","last_unstale":"2024-08-03T20:53:16.571114+0000","last_undegraded":"2024-08-03T20:53:16.571114+0000","last_fullsized":"2024-08-03T20:53:16.571114+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:53:14.539353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:53:14.539353+0000","last_clean_scrub_stamp":"2024-08-03T20:53:14.539353+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T00:17:14.895779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.35","version":"0'0","reported_seq":13,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-08-03T20:53:17.071680+0000","last_change":"2024-08-03T20:53:15.568204+0000","last_active":"2024-08-03T20:53:17.071680+0000","last_peered":"2024-08-03T20:53:17.071680+0000","last_clean":"2024-08-03T20:53:17.071680+0000","last_became_active":"2024-08-03T20:53:15.567716+0000","last_became_peered":"2024-08-03T20:53:15.567716+0000","last_unstale":"2024-08-03T20:53:17.071680+0000","last_undegraded":"2024-08-03T20:53:17.071680+0000","last_fullsized":"2024-08-03T20:53:17.071680+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:53:14.539353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:53:14.539353+0000","last_clean_scrub_stamp":"2024-08-03T20:53:14.539353+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T04:31:01.922729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.34","version":"0'0","reported_seq":13,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-08-03T20:53:17.072660+0000","last_change":"2024-08-03T20:53:15.588273+0000","last_active":"2024-08-03T20:53:17.072660+0000","last_peered":"2024-08-03T20:53:17.072660+0000","last_clean":"2024-08-03T20:53:17.072660+0000","last_became_active":"2024-08-03T20:53:15.587917+0000","last_became_peered":"2024-08-03T20:53:15.587917+0000","last_unstale":"2024-08-03T20:53:17.072660+0000","last_undegraded":"2024-08-03T20:53:17.072660+0000","last_fullsized":"2024-08-03T20:53:17.072660+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:53:14.539353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:53:14.539353+0000","last_clean_scrub_stamp":"2024-08-03T20:53:14.539353+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T00:40:35.032482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.33","version":"0'0","reported_seq":13,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-08-03T20:53:17.071989+0000","last_change":"2024-08-03T20:53:15.577686+0000","last_active":"2024-08-03T20:53:17.071989+0000","last_peered":"2024-08-03T20:53:17.071989+0000","last_clean":"2024-08-03T20:53:17.071989+0000","last_became_active":"2024-08-03T20:53:15.577340+0000","last_became_peered":"2024-08-03T20:53:15.577340+0000","last_unstale":"2024-08-03T20:53:17.071989+0000","last_undegraded":"2024-08-03T20:53:17.071989+0000","last_fullsized":"2024-08-03T20:53:17.071989+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:53:14.539353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:53:14.539353+0000","last_clean_scrub_stamp":"2024-08-03T20:53:14.539353+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T00:17:31.748655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.32","version":"0'0","reported_seq":13,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-08-03T20:53:17.071482+0000","last_change":"2024-08-03T20:53:15.577214+0000","last_active":"2024-08-03T20:53:17.071482+0000","last_peered":"2024-08-03T20:53:17.071482+0000","last_clean":"2024-08-03T20:53:17.071482+0000","last_became_active":"2024-08-03T20:53:15.576867+0000","last_became_peered":"2024-08-03T20:53:15.576867+0000","last_unstale":"2024-08-03T20:53:17.071482+0000","last_undegraded":"2024-08-03T20:53:17.071482+0000","last_fullsized":"2024-08-03T20:53:17.071482+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:53:14.539353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:53:14.539353+0000","last_clean_scrub_stamp":"2024-08-03T20:53:14.539353+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T01:14:26.384732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.31","version":"0'0","reported_seq":13,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-08-03T20:53:17.071533+0000","last_change":"2024-08-03T20:53:15.581270+0000","last_active":"2024-08-03T20:53:17.071533+0000","last_peered":"2024-08-03T20:53:17.071533+0000","last_clean":"2024-08-03T20:53:17.071533+0000","last_became_active":"2024-08-03T20:53:15.580676+0000","last_became_peered":"2024-08-03T20:53:15.580676+0000","last_unstale":"2024-08-03T20:53:17.071533+0000","last_undegraded":"2024-08-03T20:53:17.071533+0000","last_fullsized":"2024-08-03T20:53:17.071533+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:53:14.539353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:53:14.539353+0000","last_clean_scrub_stamp":"2024-08-03T20:53:14.539353+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T02:26:39.906813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.30","version":"0'0","reported_seq":13,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-08-03T20:53:16.569978+0000","last_change":"2024-08-03T20:53:15.588935+0000","last_active":"2024-08-03T20:53:16.569978+0000","last_peered":"2024-08-03T20:53:16.569978+0000","last_clean":"2024-08-03T20:53:16.569978+0000","last_became_active":"2024-08-03T20:53:15.588492+0000","last_became_peered":"2024-08-03T20:53:15.588492+0000","last_unstale":"2024-08-03T20:53:16.569978+0000","last_undegraded":"2024-08-03T20:53:16.569978+0000","last_fullsized":"2024-08-03T20:53:16.569978+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:53:14.539353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:53:14.539353+0000","last_clean_scrub_stamp":"2024-08-03T20:53:14.539353+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T05:29:21.263899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.2f","version":"0'0","reported_seq":13,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-08-03T20:53:17.072411+0000","last_change":"2024-08-03T20:53:15.586156+0000","last_active":"2024-08-03T20:53:17.072411+0000","last_peered":"2024-08-03T20:53:17.072411+0000","last_clean":"2024-08-03T20:53:17.072411+0000","last_became_active":"2024-08-03T20:53:15.585034+0000","last_became_peered":"2024-08-03T20:53:15.585034+0000","last_unstale":"2024-08-03T20:53:17.072411+0000","last_undegraded":"2024-08-03T20:53:17.072411+0000","last_fullsized":"2024-08-03T20:53:17.072411+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:53:14.539353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:53:14.539353+0000","last_clean_scrub_stamp":"2024-08-03T20:53:14.539353+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T04:56:18.852181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.2e","version":"0'0","reported_seq":13,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-08-03T20:53:17.073109+0000","last_change":"2024-08-03T20:53:15.588760+0000","last_active":"2024-08-03T20:53:17.073109+0000","last_peered":"2024-08-03T20:53:17.073109+0000","last_clean":"2024-08-03T20:53:17.073109+0000","last_became_active":"2024-08-03T20:53:15.588454+0000","last_became_peered":"2024-08-03T20:53:15.588454+0000","last_unstale":"2024-08-03T20:53:17.073109+0000","last_undegraded":"2024-08-03T20:53:17.073109+0000","last_fullsized":"2024-08-03T20:53:17.073109+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:53:14.539353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:53:14.539353+0000","last_clean_scrub_stamp":"2024-08-03T20:53:14.539353+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T00:47:43.383126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.2d","version":"0'0","reported_seq":13,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-08-03T20:53:16.569660+0000","last_change":"2024-08-03T20:53:15.583481+0000","last_active":"2024-08-03T20:53:16.569660+0000","last_peered":"2024-08-03T20:53:16.569660+0000","last_clean":"2024-08-03T20:53:16.569660+0000","last_became_active":"2024-08-03T20:53:15.579065+0000","last_became_peered":"2024-08-03T20:53:15.579065+0000","last_unstale":"2024-08-03T20:53:16.569660+0000","last_undegraded":"2024-08-03T20:53:16.569660+0000","last_fullsized":"2024-08-03T20:53:16.569660+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:53:14.539353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:53:14.539353+0000","last_clean_scrub_stamp":"2024-08-03T20:53:14.539353+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T06:33:32.079264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.2c","version":"0'0","reported_seq":13,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-08-03T20:53:17.072151+0000","last_change":"2024-08-03T20:53:15.579447+0000","last_active":"2024-08-03T20:53:17.072151+0000","last_peered":"2024-08-03T20:53:17.072151+0000","last_clean":"2024-08-03T20:53:17.072151+0000","last_became_active":"2024-08-03T20:53:15.579122+0000","last_became_peered":"2024-08-03T20:53:15.579122+0000","last_unstale":"2024-08-03T20:53:17.072151+0000","last_undegraded":"2024-08-03T20:53:17.072151+0000","last_fullsized":"2024-08-03T20:53:17.072151+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:53:14.539353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:53:14.539353+0000","last_clean_scrub_stamp":"2024-08-03T20:53:14.539353+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T00:41:51.658701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.2b","version":"0'0","reported_seq":13,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-08-03T20:53:16.570318+0000","last_change":"2024-08-03T20:53:15.585801+0000","last_active":"2024-08-03T20:53:16.570318+0000","last_peered":"2024-08-03T20:53:16.570318+0000","last_clean":"2024-08-03T20:53:16.570318+0000","last_became_active":"2024-08-03T20:53:15.585412+0000","last_became_peered":"2024-08-03T20:53:15.585412+0000","last_unstale":"2024-08-03T20:53:16.570318+0000","last_undegraded":"2024-08-03T20:53:16.570318+0000","last_fullsized":"2024-08-03T20:53:16.570318+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:53:14.539353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:53:14.539353+0000","last_clean_scrub_stamp":"2024-08-03T20:53:14.539353+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T00:03:38.220486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.2a","version":"0'0","reported_seq":13,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-08-03T20:53:16.570391+0000","last_change":"2024-08-03T20:53:15.586624+0000","last_active":"2024-08-03T20:53:16.570391+0000","last_peered":"2024-08-03T20:53:16.570391+0000","last_clean":"2024-08-03T20:53:16.570391+0000","last_became_active":"2024-08-03T20:53:15.586334+0000","last_became_peered":"2024-08-03T20:53:15.586334+0000","last_unstale":"2024-08-03T20:53:16.570391+0000","last_undegraded":"2024-08-03T20:53:16.570391+0000","last_fullsized":"2024-08-03T20:53:16.570391+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:53:14.539353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:53:14.539353+0000","last_clean_scrub_stamp":"2024-08-03T20:53:14.539353+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T07:38:02.735379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.29","version":"0'0","reported_seq":13,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-08-03T20:53:17.071643+0000","last_change":"2024-08-03T20:53:15.578027+0000","last_active":"2024-08-03T20:53:17.071643+0000","last_peered":"2024-08-03T20:53:17.071643+0000","last_clean":"2024-08-03T20:53:17.071643+0000","last_became_active":"2024-08-03T20:53:15.575178+0000","last_became_peered":"2024-08-03T20:53:15.575178+0000","last_unstale":"2024-08-03T20:53:17.071643+0000","last_undegraded":"2024-08-03T20:53:17.071643+0000","last_fullsized":"2024-08-03T20:53:17.071643+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:53:14.539353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:53:14.539353+0000","last_clean_scrub_stamp":"2024-08-03T20:53:14.539353+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T04:55:22.851056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.28","version":"0'0","reported_seq":13,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-08-03T20:53:17.072547+0000","last_change":"2024-08-03T20:53:15.560072+0000","last_active":"2024-08-03T20:53:17.072547+0000","last_peered":"2024-08-03T20:53:17.072547+0000","last_clean":"2024-08-03T20:53:17.072547+0000","last_became_active":"2024-08-03T20:53:15.559666+0000","last_became_peered":"2024-08-03T20:53:15.559666+0000","last_unstale":"2024-08-03T20:53:17.072547+0000","last_undegraded":"2024-08-03T20:53:17.072547+0000","last_fullsized":"2024-08-03T20:53:17.072547+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:53:14.539353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:53:14.539353+0000","last_clean_scrub_stamp":"2024-08-03T20:53:14.539353+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-04T22:42:39.288802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.27","version":"0'0","reported_seq":13,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-08-03T20:53:16.571679+0000","last_change":"2024-08-03T20:53:15.573666+0000","last_active":"2024-08-03T20:53:16.571679+0000","last_peered":"2024-08-03T20:53:16.571679+0000","last_clean":"2024-08-03T20:53:16.571679+0000","last_became_active":"2024-08-03T20:53:15.572202+0000","last_became_peered":"2024-08-03T20:53:15.572202+0000","last_unstale":"2024-08-03T20:53:16.571679+0000","last_undegraded":"2024-08-03T20:53:16.571679+0000","last_fullsized":"2024-08-03T20:53:16.571679+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:53:14.539353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:53:14.539353+0000","last_clean_scrub_stamp":"2024-08-03T20:53:14.539353+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T05:47:11.093905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.26","version":"0'0","reported_seq":13,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-08-03T20:53:17.072304+0000","last_change":"2024-08-03T20:53:15.587461+0000","last_active":"2024-08-03T20:53:17.072304+0000","last_peered":"2024-08-03T20:53:17.072304+0000","last_clean":"2024-08-03T20:53:17.072304+0000","last_became_active":"2024-08-03T20:53:15.587096+0000","last_became_peered":"2024-08-03T20:53:15.587096+0000","last_unstale":"2024-08-03T20:53:17.072304+0000","last_undegraded":"2024-08-03T20:53:17.072304+0000","last_fullsized":"2024-08-03T20:53:17.072304+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:53:14.539353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:53:14.539353+0000","last_clean_scrub_stamp":"2024-08-03T20:53:14.539353+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T00:17:31.748655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.25","version":"0'0","reported_seq":13,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-08-03T20:53:17.073342+0000","last_change":"2024-08-03T20:53:15.575225+0000","last_active":"2024-08-03T20:53:17.073342+0000","last_peered":"2024-08-03T20:53:17.073342+0000","last_clean":"2024-08-03T20:53:17.073342+0000","last_became_active":"2024-08-03T20:53:15.569407+0000","last_became_peered":"2024-08-03T20:53:15.569407+0000","last_unstale":"2024-08-03T20:53:17.073342+0000","last_undegraded":"2024-08-03T20:53:17.073342+0000","last_fullsized":"2024-08-03T20:53:17.073342+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:53:14.539353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:53:14.539353+0000","last_clean_scrub_stamp":"2024-08-03T20:53:14.539353+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T04:16:30.633027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.24","version":"0'0","reported_seq":13,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-08-03T20:53:16.570781+0000","last_change":"2024-08-03T20:53:15.585967+0000","last_active":"2024-08-03T20:53:16.570781+0000","last_peered":"2024-08-03T20:53:16.570781+0000","last_clean":"2024-08-03T20:53:16.570781+0000","last_became_active":"2024-08-03T20:53:15.585667+0000","last_became_peered":"2024-08-03T20:53:15.585667+0000","last_unstale":"2024-08-03T20:53:16.570781+0000","last_undegraded":"2024-08-03T20:53:16.570781+0000","last_fullsized":"2024-08-03T20:53:16.570781+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:53:14.539353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:53:14.539353+0000","last_clean_scrub_stamp":"2024-08-03T20:53:14.539353+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T08:06:42.533333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.23","version":"0'0","reported_seq":13,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-08-03T20:53:17.072829+0000","last_change":"2024-08-03T20:53:15.577169+0000","last_active":"2024-08-03T20:53:17.072829+0000","last_peered":"2024-08-03T20:53:17.072829+0000","last_clean":"2024-08-03T20:53:17.072829+0000","last_became_active":"2024-08-03T20:53:15.576857+0000","last_became_peered":"2024-08-03T20:53:15.576857+0000","last_unstale":"2024-08-03T20:53:17.072829+0000","last_undegraded":"2024-08-03T20:53:17.072829+0000","last_fullsized":"2024-08-03T20:53:17.072829+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:53:14.539353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:53:14.539353+0000","last_clean_scrub_stamp":"2024-08-03T20:53:14.539353+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T08:34:47.192955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.22","version":"0'0","reported_seq":13,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-08-03T20:53:17.071501+0000","last_change":"2024-08-03T20:53:15.587721+0000","last_active":"2024-08-03T20:53:17.071501+0000","last_peered":"2024-08-03T20:53:17.071501+0000","last_clean":"2024-08-03T20:53:17.071501+0000","last_became_active":"2024-08-03T20:53:15.587404+0000","last_became_peered":"2024-08-03T20:53:15.587404+0000","last_unstale":"2024-08-03T20:53:17.071501+0000","last_undegraded":"2024-08-03T20:53:17.071501+0000","last_fullsized":"2024-08-03T20:53:17.071501+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:53:14.539353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:53:14.539353+0000","last_clean_scrub_stamp":"2024-08-03T20:53:14.539353+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T03:03:58.620280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.21","version":"0'0","reported_seq":13,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-08-03T20:53:17.071948+0000","last_change":"2024-08-03T20:53:15.570300+0000","last_active":"2024-08-03T20:53:17.071948+0000","last_peered":"2024-08-03T20:53:17.071948+0000","last_clean":"2024-08-03T20:53:17.071948+0000","last_became_active":"2024-08-03T20:53:15.569978+0000","last_became_peered":"2024-08-03T20:53:15.569978+0000","last_unstale":"2024-08-03T20:53:17.071948+0000","last_undegraded":"2024-08-03T20:53:17.071948+0000","last_fullsized":"2024-08-03T20:53:17.071948+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:53:14.539353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:53:14.539353+0000","last_clean_scrub_stamp":"2024-08-03T20:53:14.539353+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T04:52:51.519610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.20","version":"0'0","reported_seq":13,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-08-03T20:53:16.569016+0000","last_change":"2024-08-03T20:53:15.583470+0000","last_active":"2024-08-03T20:53:16.569016+0000","last_peered":"2024-08-03T20:53:16.569016+0000","last_clean":"2024-08-03T20:53:16.569016+0000","last_became_active":"2024-08-03T20:53:15.582088+0000","last_became_peered":"2024-08-03T20:53:15.582088+0000","last_unstale":"2024-08-03T20:53:16.569016+0000","last_undegraded":"2024-08-03T20:53:16.569016+0000","last_fullsized":"2024-08-03T20:53:16.569016+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:53:14.539353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:53:14.539353+0000","last_clean_scrub_stamp":"2024-08-03T20:53:14.539353+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T04:36:52.350219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.1e","version":"0'0","reported_seq":13,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-08-03T20:53:17.071610+0000","last_change":"2024-08-03T20:53:15.568687+0000","last_active":"2024-08-03T20:53:17.071610+0000","last_peered":"2024-08-03T20:53:17.071610+0000","last_clean":"2024-08-03T20:53:17.071610+0000","last_became_active":"2024-08-03T20:53:15.568367+0000","last_became_peered":"2024-08-03T20:53:15.568367+0000","last_unstale":"2024-08-03T20:53:17.071610+0000","last_undegraded":"2024-08-03T20:53:17.071610+0000","last_fullsized":"2024-08-03T20:53:17.071610+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:53:14.539353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:53:14.539353+0000","last_clean_scrub_stamp":"2024-08-03T20:53:14.539353+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T06:42:49.059064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.1f","version":"69'4","reported_seq":19,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-08-03T20:53:17.072662+0000","last_change":"2024-08-03T20:53:14.572198+0000","last_active":"2024-08-03T20:53:17.072662+0000","last_peered":"2024-08-03T20:53:17.072662+0000","last_clean":"2024-08-03T20:53:17.072662+0000","last_became_active":"2024-08-03T20:53:14.571868+0000","last_became_peered":"2024-08-03T20:53:14.571868+0000","last_unstale":"2024-08-03T20:53:17.072662+0000","last_undegraded":"2024-08-03T20:53:17.072662+0000","last_fullsized":"2024-08-03T20:53:17.072662+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:53:13.536413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:53:13.536413+0000","last_clean_scrub_stamp":"2024-08-03T20:53:13.536413+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-04T22:17:49.755050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.1f","version":"0'0","reported_seq":13,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-08-03T20:53:17.072558+0000","last_change":"2024-08-03T20:53:15.585378+0000","last_active":"2024-08-03T20:53:17.072558+0000","last_peered":"2024-08-03T20:53:17.072558+0000","last_clean":"2024-08-03T20:53:17.072558+0000","last_became_active":"2024-08-03T20:53:15.584025+0000","last_became_peered":"2024-08-03T20:53:15.584025+0000","last_unstale":"2024-08-03T20:53:17.072558+0000","last_undegraded":"2024-08-03T20:53:17.072558+0000","last_fullsized":"2024-08-03T20:53:17.072558+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:53:14.539353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:53:14.539353+0000","last_clean_scrub_stamp":"2024-08-03T20:53:14.539353+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T01:32:26.699754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.1e","version":"69'1","reported_seq":16,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-08-03T20:53:17.072788+0000","last_change":"2024-08-03T20:53:14.574117+0000","last_active":"2024-08-03T20:53:17.072788+0000","last_peered":"2024-08-03T20:53:17.072788+0000","last_clean":"2024-08-03T20:53:17.072788+0000","last_became_active":"2024-08-03T20:53:14.573766+0000","last_became_peered":"2024-08-03T20:53:14.573766+0000","last_unstale":"2024-08-03T20:53:17.072788+0000","last_undegraded":"2024-08-03T20:53:17.072788+0000","last_fullsized":"2024-08-03T20:53:17.072788+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:53:13.536413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:53:13.536413+0000","last_clean_scrub_stamp":"2024-08-03T20:53:13.536413+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T07:38:09.198159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.1c","version":"0'0","reported_seq":13,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-08-03T20:53:16.569595+0000","last_change":"2024-08-03T20:53:15.561346+0000","last_active":"2024-08-03T20:53:16.569595+0000","last_peered":"2024-08-03T20:53:16.569595+0000","last_clean":"2024-08-03T20:53:16.569595+0000","last_became_active":"2024-08-03T20:53:15.558928+0000","last_became_peered":"2024-08-03T20:53:15.558928+0000","last_unstale":"2024-08-03T20:53:16.569595+0000","last_undegraded":"2024-08-03T20:53:16.569595+0000","last_fullsized":"2024-08-03T20:53:16.569595+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:53:14.539353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:53:14.539353+0000","last_clean_scrub_stamp":"2024-08-03T20:53:14.539353+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T03:05:04.829979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.1d","version":"69'2","reported_seq":17,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-08-03T20:53:16.570598+0000","last_change":"2024-08-03T20:53:14.565668+0000","last_active":"2024-08-03T20:53:16.570598+0000","last_peered":"2024-08-03T20:53:16.570598+0000","last_clean":"2024-08-03T20:53:16.570598+0000","last_became_active":"2024-08-03T20:53:14.565334+0000","last_became_peered":"2024-08-03T20:53:14.565334+0000","last_unstale":"2024-08-03T20:53:16.570598+0000","last_undegraded":"2024-08-03T20:53:16.570598+0000","last_fullsized":"2024-08-03T20:53:16.570598+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:53:13.536413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:53:13.536413+0000","last_clean_scrub_stamp":"2024-08-03T20:53:13.536413+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T06:09:05.526276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.1d","version":"0'0","reported_seq":13,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-08-03T20:53:17.071889+0000","last_change":"2024-08-03T20:53:15.572530+0000","last_active":"2024-08-03T20:53:17.071889+0000","last_peered":"2024-08-03T20:53:17.071889+0000","last_clean":"2024-08-03T20:53:17.071889+0000","last_became_active":"2024-08-03T20:53:15.571455+0000","last_became_peered":"2024-08-03T20:53:15.571455+0000","last_unstale":"2024-08-03T20:53:17.071889+0000","last_undegraded":"2024-08-03T20:53:17.071889+0000","last_fullsized":"2024-08-03T20:53:17.071889+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:53:14.539353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:53:14.539353+0000","last_clean_scrub_stamp":"2024-08-03T20:53:14.539353+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-04T22:26:58.235870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.1c","version":"69'2","reported_seq":17,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-08-03T20:53:16.569739+0000","last_change":"2024-08-03T20:53:14.571705+0000","last_active":"2024-08-03T20:53:16.569739+0000","last_peered":"2024-08-03T20:53:16.569739+0000","last_clean":"2024-08-03T20:53:16.569739+0000","last_became_active":"2024-08-03T20:53:14.570618+0000","last_became_peered":"2024-08-03T20:53:14.570618+0000","last_unstale":"2024-08-03T20:53:16.569739+0000","last_undegraded":"2024-08-03T20:53:16.569739+0000","last_fullsized":"2024-08-03T20:53:16.569739+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:53:13.536413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:53:13.536413+0000","last_clean_scrub_stamp":"2024-08-03T20:53:13.536413+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T06:27:34.846150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.1a","version":"0'0","reported_seq":13,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-08-03T20:53:17.071709+0000","last_change":"2024-08-03T20:53:15.593632+0000","last_active":"2024-08-03T20:53:17.071709+0000","last_peered":"2024-08-03T20:53:17.071709+0000","last_clean":"2024-08-03T20:53:17.071709+0000","last_became_active":"2024-08-03T20:53:15.593304+0000","last_became_peered":"2024-08-03T20:53:15.593304+0000","last_unstale":"2024-08-03T20:53:17.071709+0000","last_undegraded":"2024-08-03T20:53:17.071709+0000","last_fullsized":"2024-08-03T20:53:17.071709+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:53:14.539353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:53:14.539353+0000","last_clean_scrub_stamp":"2024-08-03T20:53:14.539353+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T02:26:39.906813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.1b","version":"69'1","reported_seq":16,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-08-03T20:53:17.073065+0000","last_change":"2024-08-03T20:53:14.577044+0000","last_active":"2024-08-03T20:53:17.073065+0000","last_peered":"2024-08-03T20:53:17.073065+0000","last_clean":"2024-08-03T20:53:17.073065+0000","last_became_active":"2024-08-03T20:53:14.576539+0000","last_became_peered":"2024-08-03T20:53:14.576539+0000","last_unstale":"2024-08-03T20:53:17.073065+0000","last_undegraded":"2024-08-03T20:53:17.073065+0000","last_fullsized":"2024-08-03T20:53:17.073065+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:53:13.536413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:53:13.536413+0000","last_clean_scrub_stamp":"2024-08-03T20:53:13.536413+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T04:04:55.518824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.1b","version":"0'0","reported_seq":13,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-08-03T20:53:16.570449+0000","last_change":"2024-08-03T20:53:15.580420+0000","last_active":"2024-08-03T20:53:16.570449+0000","last_peered":"2024-08-03T20:53:16.570449+0000","last_clean":"2024-08-03T20:53:16.570449+0000","last_became_active":"2024-08-03T20:53:15.577124+0000","last_became_peered":"2024-08-03T20:53:15.577124+0000","last_unstale":"2024-08-03T20:53:16.570449+0000","last_undegraded":"2024-08-03T20:53:16.570449+0000","last_fullsized":"2024-08-03T20:53:16.570449+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:53:14.539353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:53:14.539353+0000","last_clean_scrub_stamp":"2024-08-03T20:53:14.539353+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T08:19:34.926710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.1a","version":"69'1","reported_seq":16,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-08-03T20:53:16.568837+0000","last_change":"2024-08-03T20:53:14.573942+0000","last_active":"2024-08-03T20:53:16.568837+0000","last_peered":"2024-08-03T20:53:16.568837+0000","last_clean":"2024-08-03T20:53:16.568837+0000","last_became_active":"2024-08-03T20:53:14.573388+0000","last_became_peered":"2024-08-03T20:53:14.573388+0000","last_unstale":"2024-08-03T20:53:16.568837+0000","last_undegraded":"2024-08-03T20:53:16.568837+0000","last_fullsized":"2024-08-03T20:53:16.568837+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:53:13.536413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:53:13.536413+0000","last_clean_scrub_stamp":"2024-08-03T20:53:13.536413+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-04T22:26:06.673517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.18","version":"0'0","reported_seq":13,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-08-03T20:53:17.070910+0000","last_change":"2024-08-03T20:53:15.587232+0000","last_active":"2024-08-03T20:53:17.070910+0000","last_peered":"2024-08-03T20:53:17.070910+0000","last_clean":"2024-08-03T20:53:17.070910+0000","last_became_active":"2024-08-03T20:53:15.586895+0000","last_became_peered":"2024-08-03T20:53:15.586895+0000","last_unstale":"2024-08-03T20:53:17.070910+0000","last_undegraded":"2024-08-03T20:53:17.070910+0000","last_fullsized":"2024-08-03T20:53:17.070910+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:53:14.539353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:53:14.539353+0000","last_clean_scrub_stamp":"2024-08-03T20:53:14.539353+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T03:15:28.965017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.19","version":"0'0","reported_seq":15,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-08-03T20:53:17.071722+0000","last_change":"2024-08-03T20:53:14.574234+0000","last_active":"2024-08-03T20:53:17.071722+0000","last_peered":"2024-08-03T20:53:17.071722+0000","last_clean":"2024-08-03T20:53:17.071722+0000","last_became_active":"2024-08-03T20:53:14.573922+0000","last_became_peered":"2024-08-03T20:53:14.573922+0000","last_unstale":"2024-08-03T20:53:17.071722+0000","last_undegraded":"2024-08-03T20:53:17.071722+0000","last_fullsized":"2024-08-03T20:53:17.071722+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:53:13.536413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:53:13.536413+0000","last_clean_scrub_stamp":"2024-08-03T20:53:13.536413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T02:39:20.815693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.19","version":"0'0","reported_seq":13,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-08-03T20:53:17.071715+0000","last_change":"2024-08-03T20:53:15.575421+0000","last_active":"2024-08-03T20:53:17.071715+0000","last_peered":"2024-08-03T20:53:17.071715+0000","last_clean":"2024-08-03T20:53:17.071715+0000","last_became_active":"2024-08-03T20:53:15.572013+0000","last_became_peered":"2024-08-03T20:53:15.572013+0000","last_unstale":"2024-08-03T20:53:17.071715+0000","last_undegraded":"2024-08-03T20:53:17.071715+0000","last_fullsized":"2024-08-03T20:53:17.071715+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:53:14.539353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:53:14.539353+0000","last_clean_scrub_stamp":"2024-08-03T20:53:14.539353+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T04:52:51.519610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.18","version":"0'0","reported_seq":15,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-08-03T20:53:17.071028+0000","last_change":"2024-08-03T20:53:14.562869+0000","last_active":"2024-08-03T20:53:17.071028+0000","last_peered":"2024-08-03T20:53:17.071028+0000","last_clean":"2024-08-03T20:53:17.071028+0000","last_became_active":"2024-08-03T20:53:14.562575+0000","last_became_peered":"2024-08-03T20:53:14.562575+0000","last_unstale":"2024-08-03T20:53:17.071028+0000","last_undegraded":"2024-08-03T20:53:17.071028+0000","last_fullsized":"2024-08-03T20:53:17.071028+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:53:13.536413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:53:13.536413+0000","last_clean_scrub_stamp":"2024-08-03T20:53:13.536413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-04T21:31:19.054856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.16","version":"0'0","reported_seq":13,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-08-03T20:53:17.072446+0000","last_change":"2024-08-03T20:53:15.574749+0000","last_active":"2024-08-03T20:53:17.072446+0000","last_peered":"2024-08-03T20:53:17.072446+0000","last_clean":"2024-08-03T20:53:17.072446+0000","last_became_active":"2024-08-03T20:53:15.574420+0000","last_became_peered":"2024-08-03T20:53:15.574420+0000","last_unstale":"2024-08-03T20:53:17.072446+0000","last_undegraded":"2024-08-03T20:53:17.072446+0000","last_fullsized":"2024-08-03T20:53:17.072446+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:53:14.539353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:53:14.539353+0000","last_clean_scrub_stamp":"2024-08-03T20:53:14.539353+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-04T21:39:44.456389+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.17","version":"0'0","reported_seq":15,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-08-03T20:53:16.571882+0000","last_change":"2024-08-03T20:53:14.571511+0000","last_active":"2024-08-03T20:53:16.571882+0000","last_peered":"2024-08-03T20:53:16.571882+0000","last_clean":"2024-08-03T20:53:16.571882+0000","last_became_active":"2024-08-03T20:53:14.571224+0000","last_became_peered":"2024-08-03T20:53:14.571224+0000","last_unstale":"2024-08-03T20:53:16.571882+0000","last_undegraded":"2024-08-03T20:53:16.571882+0000","last_fullsized":"2024-08-03T20:53:16.571882+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:53:13.536413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:53:13.536413+0000","last_clean_scrub_stamp":"2024-08-03T20:53:13.536413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T04:05:14.609367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.17","version":"0'0","reported_seq":13,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-08-03T20:53:17.072615+0000","last_change":"2024-08-03T20:53:15.569893+0000","last_active":"2024-08-03T20:53:17.072615+0000","last_peered":"2024-08-03T20:53:17.072615+0000","last_clean":"2024-08-03T20:53:17.072615+0000","last_became_active":"2024-08-03T20:53:15.569444+0000","last_became_peered":"2024-08-03T20:53:15.569444+0000","last_unstale":"2024-08-03T20:53:17.072615+0000","last_undegraded":"2024-08-03T20:53:17.072615+0000","last_fullsized":"2024-08-03T20:53:17.072615+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:53:14.539353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:53:14.539353+0000","last_clean_scrub_stamp":"2024-08-03T20:53:14.539353+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T05:09:58.001651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.16","version":"0'0","reported_seq":15,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-08-03T20:53:16.569694+0000","last_change":"2024-08-03T20:53:14.578822+0000","last_active":"2024-08-03T20:53:16.569694+0000","last_peered":"2024-08-03T20:53:16.569694+0000","last_clean":"2024-08-03T20:53:16.569694+0000","last_became_active":"2024-08-03T20:53:14.576635+0000","last_became_peered":"2024-08-03T20:53:14.576635+0000","last_unstale":"2024-08-03T20:53:16.569694+0000","last_undegraded":"2024-08-03T20:53:16.569694+0000","last_fullsized":"2024-08-03T20:53:16.569694+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:53:13.536413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:53:13.536413+0000","last_clean_scrub_stamp":"2024-08-03T20:53:13.536413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T06:06:46.664809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.14","version":"0'0","reported_seq":13,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-08-03T20:53:16.569800+0000","last_change":"2024-08-03T20:53:15.578268+0000","last_active":"2024-08-03T20:53:16.569800+0000","last_peered":"2024-08-03T20:53:16.569800+0000","last_clean":"2024-08-03T20:53:16.569800+0000","last_became_active":"2024-08-03T20:53:15.573254+0000","last_became_peered":"2024-08-03T20:53:15.573254+0000","last_unstale":"2024-08-03T20:53:16.569800+0000","last_undegraded":"2024-08-03T20:53:16.569800+0000","last_fullsized":"2024-08-03T20:53:16.569800+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:53:14.539353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:53:14.539353+0000","last_clean_scrub_stamp":"2024-08-03T20:53:14.539353+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-04T23:34:13.826535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.15","version":"69'3","reported_seq":18,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-08-03T20:53:17.073354+0000","last_change":"2024-08-03T20:53:14.574375+0000","last_active":"2024-08-03T20:53:17.073354+0000","last_peered":"2024-08-03T20:53:17.073354+0000","last_clean":"2024-08-03T20:53:17.073354+0000","last_became_active":"2024-08-03T20:53:14.574052+0000","last_became_peered":"2024-08-03T20:53:14.574052+0000","last_unstale":"2024-08-03T20:53:17.073354+0000","last_undegraded":"2024-08-03T20:53:17.073354+0000","last_fullsized":"2024-08-03T20:53:17.073354+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:53:13.536413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:53:13.536413+0000","last_clean_scrub_stamp":"2024-08-03T20:53:13.536413+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T05:17:23.712502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.15","version":"0'0","reported_seq":13,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-08-03T20:53:17.071464+0000","last_change":"2024-08-03T20:53:15.577990+0000","last_active":"2024-08-03T20:53:17.071464+0000","last_peered":"2024-08-03T20:53:17.071464+0000","last_clean":"2024-08-03T20:53:17.071464+0000","last_became_active":"2024-08-03T20:53:15.577703+0000","last_became_peered":"2024-08-03T20:53:15.577703+0000","last_unstale":"2024-08-03T20:53:17.071464+0000","last_undegraded":"2024-08-03T20:53:17.071464+0000","last_fullsized":"2024-08-03T20:53:17.071464+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:53:14.539353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:53:14.539353+0000","last_clean_scrub_stamp":"2024-08-03T20:53:14.539353+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T07:24:40.612310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.14","version":"69'6","reported_seq":21,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-08-03T20:53:17.072732+0000","last_change":"2024-08-03T20:53:14.569303+0000","last_active":"2024-08-03T20:53:17.072732+0000","last_peered":"2024-08-03T20:53:17.072732+0000","last_clean":"2024-08-03T20:53:17.072732+0000","last_became_active":"2024-08-03T20:53:14.568967+0000","last_became_peered":"2024-08-03T20:53:14.568967+0000","last_unstale":"2024-08-03T20:53:17.072732+0000","last_undegraded":"2024-08-03T20:53:17.072732+0000","last_fullsized":"2024-08-03T20:53:17.072732+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:53:13.536413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:53:13.536413+0000","last_clean_scrub_stamp":"2024-08-03T20:53:13.536413+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T06:32:14.441616+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.12","version":"0'0","reported_seq":13,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-08-03T20:53:17.071411+0000","last_change":"2024-08-03T20:53:15.576632+0000","last_active":"2024-08-03T20:53:17.071411+0000","last_peered":"2024-08-03T20:53:17.071411+0000","last_clean":"2024-08-03T20:53:17.071411+0000","last_became_active":"2024-08-03T20:53:15.573567+0000","last_became_peered":"2024-08-03T20:53:15.573567+0000","last_unstale":"2024-08-03T20:53:17.071411+0000","last_undegraded":"2024-08-03T20:53:17.071411+0000","last_fullsized":"2024-08-03T20:53:17.071411+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:53:14.539353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:53:14.539353+0000","last_clean_scrub_stamp":"2024-08-03T20:53:14.539353+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T04:29:12.151830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.13","version":"69'2","reported_seq":17,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-08-03T20:53:17.072060+0000","last_change":"2024-08-03T20:53:14.562362+0000","last_active":"2024-08-03T20:53:17.072060+0000","last_peered":"2024-08-03T20:53:17.072060+0000","last_clean":"2024-08-03T20:53:17.072060+0000","last_became_active":"2024-08-03T20:53:14.562020+0000","last_became_peered":"2024-08-03T20:53:14.562020+0000","last_unstale":"2024-08-03T20:53:17.072060+0000","last_undegraded":"2024-08-03T20:53:17.072060+0000","last_fullsized":"2024-08-03T20:53:17.072060+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:53:13.536413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:53:13.536413+0000","last_clean_scrub_stamp":"2024-08-03T20:53:13.536413+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T04:45:10.894258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.13","version":"0'0","reported_seq":13,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-08-03T20:53:17.071366+0000","last_change":"2024-08-03T20:53:15.579155+0000","last_active":"2024-08-03T20:53:17.071366+0000","last_peered":"2024-08-03T20:53:17.071366+0000","last_clean":"2024-08-03T20:53:17.071366+0000","last_became_active":"2024-08-03T20:53:15.578820+0000","last_became_peered":"2024-08-03T20:53:15.578820+0000","last_unstale":"2024-08-03T20:53:17.071366+0000","last_undegraded":"2024-08-03T20:53:17.071366+0000","last_fullsized":"2024-08-03T20:53:17.071366+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:53:14.539353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:53:14.539353+0000","last_clean_scrub_stamp":"2024-08-03T20:53:14.539353+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T07:18:30.209873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.12","version":"0'0","reported_seq":15,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-08-03T20:53:16.568984+0000","last_change":"2024-08-03T20:53:14.573704+0000","last_active":"2024-08-03T20:53:16.568984+0000","last_peered":"2024-08-03T20:53:16.568984+0000","last_clean":"2024-08-03T20:53:16.568984+0000","last_became_active":"2024-08-03T20:53:14.572963+0000","last_became_peered":"2024-08-03T20:53:14.572963+0000","last_unstale":"2024-08-03T20:53:16.568984+0000","last_undegraded":"2024-08-03T20:53:16.568984+0000","last_fullsized":"2024-08-03T20:53:16.568984+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:53:13.536413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:53:13.536413+0000","last_clean_scrub_stamp":"2024-08-03T20:53:13.536413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T06:16:52.747198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.10","version":"0'0","reported_seq":13,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-08-03T20:53:17.071027+0000","last_change":"2024-08-03T20:53:15.577960+0000","last_active":"2024-08-03T20:53:17.071027+0000","last_peered":"2024-08-03T20:53:17.071027+0000","last_clean":"2024-08-03T20:53:17.071027+0000","last_became_active":"2024-08-03T20:53:15.577619+0000","last_became_peered":"2024-08-03T20:53:15.577619+0000","last_unstale":"2024-08-03T20:53:17.071027+0000","last_undegraded":"2024-08-03T20:53:17.071027+0000","last_fullsized":"2024-08-03T20:53:17.071027+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:53:14.539353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:53:14.539353+0000","last_clean_scrub_stamp":"2024-08-03T20:53:14.539353+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T03:03:58.620280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.11","version":"0'0","reported_seq":15,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-08-03T20:53:17.071704+0000","last_change":"2024-08-03T20:53:14.578344+0000","last_active":"2024-08-03T20:53:17.071704+0000","last_peered":"2024-08-03T20:53:17.071704+0000","last_clean":"2024-08-03T20:53:17.071704+0000","last_became_active":"2024-08-03T20:53:14.577935+0000","last_became_peered":"2024-08-03T20:53:14.577935+0000","last_unstale":"2024-08-03T20:53:17.071704+0000","last_undegraded":"2024-08-03T20:53:17.071704+0000","last_fullsized":"2024-08-03T20:53:17.071704+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:53:13.536413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:53:13.536413+0000","last_clean_scrub_stamp":"2024-08-03T20:53:13.536413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T03:05:03.827039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.11","version":"0'0","reported_seq":13,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-08-03T20:53:17.071168+0000","last_change":"2024-08-03T20:53:15.581028+0000","last_active":"2024-08-03T20:53:17.071168+0000","last_peered":"2024-08-03T20:53:17.071168+0000","last_clean":"2024-08-03T20:53:17.071168+0000","last_became_active":"2024-08-03T20:53:15.580738+0000","last_became_peered":"2024-08-03T20:53:15.580738+0000","last_unstale":"2024-08-03T20:53:17.071168+0000","last_undegraded":"2024-08-03T20:53:17.071168+0000","last_fullsized":"2024-08-03T20:53:17.071168+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:53:14.539353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:53:14.539353+0000","last_clean_scrub_stamp":"2024-08-03T20:53:14.539353+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T07:45:22.600323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.10","version":"69'8","reported_seq":23,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-08-03T20:53:16.570075+0000","last_change":"2024-08-03T20:53:14.568713+0000","last_active":"2024-08-03T20:53:16.570075+0000","last_peered":"2024-08-03T20:53:16.570075+0000","last_clean":"2024-08-03T20:53:16.570075+0000","last_became_active":"2024-08-03T20:53:14.566620+0000","last_became_peered":"2024-08-03T20:53:14.566620+0000","last_unstale":"2024-08-03T20:53:16.570075+0000","last_undegraded":"2024-08-03T20:53:16.570075+0000","last_fullsized":"2024-08-03T20:53:16.570075+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:53:13.536413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:53:13.536413+0000","last_clean_scrub_stamp":"2024-08-03T20:53:13.536413+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-04T21:59:25.578600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.e","version":"0'0","reported_seq":13,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-08-03T20:53:16.571307+0000","last_change":"2024-08-03T20:53:15.581915+0000","last_active":"2024-08-03T20:53:16.571307+0000","last_peered":"2024-08-03T20:53:16.571307+0000","last_clean":"2024-08-03T20:53:16.571307+0000","last_became_active":"2024-08-03T20:53:15.578386+0000","last_became_peered":"2024-08-03T20:53:15.578386+0000","last_unstale":"2024-08-03T20:53:16.571307+0000","last_undegraded":"2024-08-03T20:53:16.571307+0000","last_fullsized":"2024-08-03T20:53:16.571307+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:53:14.539353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:53:14.539353+0000","last_clean_scrub_stamp":"2024-08-03T20:53:14.539353+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T04:48:30.059431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.f","version":"69'1","reported_seq":16,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-08-03T20:53:16.569123+0000","last_change":"2024-08-03T20:53:14.571914+0000","last_active":"2024-08-03T20:53:16.569123+0000","last_peered":"2024-08-03T20:53:16.569123+0000","last_clean":"2024-08-03T20:53:16.569123+0000","last_became_active":"2024-08-03T20:53:14.571430+0000","last_became_peered":"2024-08-03T20:53:14.571430+0000","last_unstale":"2024-08-03T20:53:16.569123+0000","last_undegraded":"2024-08-03T20:53:16.569123+0000","last_fullsized":"2024-08-03T20:53:16.569123+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:53:13.536413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:53:13.536413+0000","last_clean_scrub_stamp":"2024-08-03T20:53:13.536413+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T06:32:14.441616+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.f","version":"0'0","reported_seq":13,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-08-03T20:53:17.072722+0000","last_change":"2024-08-03T20:53:15.582568+0000","last_active":"2024-08-03T20:53:17.072722+0000","last_peered":"2024-08-03T20:53:17.072722+0000","last_clean":"2024-08-03T20:53:17.072722+0000","last_became_active":"2024-08-03T20:53:15.582257+0000","last_became_peered":"2024-08-03T20:53:15.582257+0000","last_unstale":"2024-08-03T20:53:17.072722+0000","last_undegraded":"2024-08-03T20:53:17.072722+0000","last_fullsized":"2024-08-03T20:53:17.072722+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:53:14.539353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:53:14.539353+0000","last_clean_scrub_stamp":"2024-08-03T20:53:14.539353+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T05:07:56.198670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.e","version":"69'2","reported_seq":17,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-08-03T20:53:16.571436+0000","last_change":"2024-08-03T20:53:14.566701+0000","last_active":"2024-08-03T20:53:16.571436+0000","last_peered":"2024-08-03T20:53:16.571436+0000","last_clean":"2024-08-03T20:53:16.571436+0000","last_became_active":"2024-08-03T20:53:14.566353+0000","last_became_peered":"2024-08-03T20:53:14.566353+0000","last_unstale":"2024-08-03T20:53:16.571436+0000","last_undegraded":"2024-08-03T20:53:16.571436+0000","last_fullsized":"2024-08-03T20:53:16.571436+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:53:13.536413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:53:13.536413+0000","last_clean_scrub_stamp":"2024-08-03T20:53:13.536413+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T02:37:47.741827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.c","version":"0'0","reported_seq":13,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-08-03T20:53:17.072672+0000","last_change":"2024-08-03T20:53:15.578129+0000","last_active":"2024-08-03T20:53:17.072672+0000","last_peered":"2024-08-03T20:53:17.072672+0000","last_clean":"2024-08-03T20:53:17.072672+0000","last_became_active":"2024-08-03T20:53:15.577842+0000","last_became_peered":"2024-08-03T20:53:15.577842+0000","last_unstale":"2024-08-03T20:53:17.072672+0000","last_undegraded":"2024-08-03T20:53:17.072672+0000","last_fullsized":"2024-08-03T20:53:17.072672+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:53:14.539353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:53:14.539353+0000","last_clean_scrub_stamp":"2024-08-03T20:53:14.539353+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T05:56:19.142029+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.d","version":"69'1","reported_seq":16,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-08-03T20:53:16.569825+0000","last_change":"2024-08-03T20:53:14.575372+0000","last_active":"2024-08-03T20:53:16.569825+0000","last_peered":"2024-08-03T20:53:16.569825+0000","last_clean":"2024-08-03T20:53:16.569825+0000","last_became_active":"2024-08-03T20:53:14.575017+0000","last_became_peered":"2024-08-03T20:53:14.575017+0000","last_unstale":"2024-08-03T20:53:16.569825+0000","last_undegraded":"2024-08-03T20:53:16.569825+0000","last_fullsized":"2024-08-03T20:53:16.569825+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:53:13.536413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:53:13.536413+0000","last_clean_scrub_stamp":"2024-08-03T20:53:13.536413+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T07:08:25.288301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.d","version":"0'0","reported_seq":13,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-08-03T20:53:17.071650+0000","last_change":"2024-08-03T20:53:15.588724+0000","last_active":"2024-08-03T20:53:17.071650+0000","last_peered":"2024-08-03T20:53:17.071650+0000","last_clean":"2024-08-03T20:53:17.071650+0000","last_became_active":"2024-08-03T20:53:15.587992+0000","last_became_peered":"2024-08-03T20:53:15.587992+0000","last_unstale":"2024-08-03T20:53:17.071650+0000","last_undegraded":"2024-08-03T20:53:17.071650+0000","last_fullsized":"2024-08-03T20:53:17.071650+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:53:14.539353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:53:14.539353+0000","last_clean_scrub_stamp":"2024-08-03T20:53:14.539353+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T01:54:12.251052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.c","version":"0'0","reported_seq":15,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-08-03T20:53:17.072074+0000","last_change":"2024-08-03T20:53:14.576985+0000","last_active":"2024-08-03T20:53:17.072074+0000","last_peered":"2024-08-03T20:53:17.072074+0000","last_clean":"2024-08-03T20:53:17.072074+0000","last_became_active":"2024-08-03T20:53:14.576634+0000","last_became_peered":"2024-08-03T20:53:14.576634+0000","last_unstale":"2024-08-03T20:53:17.072074+0000","last_undegraded":"2024-08-03T20:53:17.072074+0000","last_fullsized":"2024-08-03T20:53:17.072074+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:53:13.536413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:53:13.536413+0000","last_clean_scrub_stamp":"2024-08-03T20:53:13.536413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T03:53:21.022450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.b","version":"0'0","reported_seq":13,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-08-03T20:53:17.073074+0000","last_change":"2024-08-03T20:53:15.575998+0000","last_active":"2024-08-03T20:53:17.073074+0000","last_peered":"2024-08-03T20:53:17.073074+0000","last_clean":"2024-08-03T20:53:17.073074+0000","last_became_active":"2024-08-03T20:53:15.575279+0000","last_became_peered":"2024-08-03T20:53:15.575279+0000","last_unstale":"2024-08-03T20:53:17.073074+0000","last_undegraded":"2024-08-03T20:53:17.073074+0000","last_fullsized":"2024-08-03T20:53:17.073074+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:53:14.539353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:53:14.539353+0000","last_clean_scrub_stamp":"2024-08-03T20:53:14.539353+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-04T23:37:48.166531+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.a","version":"0'0","reported_seq":15,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-08-03T20:53:16.569136+0000","last_change":"2024-08-03T20:53:14.570733+0000","last_active":"2024-08-03T20:53:16.569136+0000","last_peered":"2024-08-03T20:53:16.569136+0000","last_clean":"2024-08-03T20:53:16.569136+0000","last_became_active":"2024-08-03T20:53:14.570411+0000","last_became_peered":"2024-08-03T20:53:14.570411+0000","last_unstale":"2024-08-03T20:53:16.569136+0000","last_undegraded":"2024-08-03T20:53:16.569136+0000","last_fullsized":"2024-08-03T20:53:16.569136+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:53:13.536413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:53:13.536413+0000","last_clean_scrub_stamp":"2024-08-03T20:53:13.536413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T04:45:10.894258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.8","version":"0'0","reported_seq":13,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-08-03T20:53:17.071191+0000","last_change":"2024-08-03T20:53:15.586041+0000","last_active":"2024-08-03T20:53:17.071191+0000","last_peered":"2024-08-03T20:53:17.071191+0000","last_clean":"2024-08-03T20:53:17.071191+0000","last_became_active":"2024-08-03T20:53:15.585540+0000","last_became_peered":"2024-08-03T20:53:15.585540+0000","last_unstale":"2024-08-03T20:53:17.071191+0000","last_undegraded":"2024-08-03T20:53:17.071191+0000","last_fullsized":"2024-08-03T20:53:17.071191+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:53:14.539353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:53:14.539353+0000","last_clean_scrub_stamp":"2024-08-03T20:53:14.539353+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T05:10:46.836530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.9","version":"0'0","reported_seq":15,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-08-03T20:53:17.072020+0000","last_change":"2024-08-03T20:53:14.574283+0000","last_active":"2024-08-03T20:53:17.072020+0000","last_peered":"2024-08-03T20:53:17.072020+0000","last_clean":"2024-08-03T20:53:17.072020+0000","last_became_active":"2024-08-03T20:53:14.572944+0000","last_became_peered":"2024-08-03T20:53:14.572944+0000","last_unstale":"2024-08-03T20:53:17.072020+0000","last_undegraded":"2024-08-03T20:53:17.072020+0000","last_fullsized":"2024-08-03T20:53:17.072020+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:53:13.536413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:53:13.536413+0000","last_clean_scrub_stamp":"2024-08-03T20:53:13.536413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T04:34:18.838189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.9","version":"0'0","reported_seq":13,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-08-03T20:53:17.072177+0000","last_change":"2024-08-03T20:53:15.580077+0000","last_active":"2024-08-03T20:53:17.072177+0000","last_peered":"2024-08-03T20:53:17.072177+0000","last_clean":"2024-08-03T20:53:17.072177+0000","last_became_active":"2024-08-03T20:53:15.579573+0000","last_became_peered":"2024-08-03T20:53:15.579573+0000","last_unstale":"2024-08-03T20:53:17.072177+0000","last_undegraded":"2024-08-03T20:53:17.072177+0000","last_fullsized":"2024-08-03T20:53:17.072177+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:53:14.539353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:53:14.539353+0000","last_clean_scrub_stamp":"2024-08-03T20:53:14.539353+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T03:42:32.646474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.8","version":"0'0","reported_seq":15,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-08-03T20:53:17.070785+0000","last_change":"2024-08-03T20:53:14.569349+0000","last_active":"2024-08-03T20:53:17.070785+0000","last_peered":"2024-08-03T20:53:17.070785+0000","last_clean":"2024-08-03T20:53:17.070785+0000","last_became_active":"2024-08-03T20:53:14.568790+0000","last_became_peered":"2024-08-03T20:53:14.568790+0000","last_unstale":"2024-08-03T20:53:17.070785+0000","last_undegraded":"2024-08-03T20:53:17.070785+0000","last_fullsized":"2024-08-03T20:53:17.070785+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:53:13.536413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:53:13.536413+0000","last_clean_scrub_stamp":"2024-08-03T20:53:13.536413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T06:33:31.076324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.6","version":"0'0","reported_seq":13,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-08-03T20:53:17.072880+0000","last_change":"2024-08-03T20:53:15.578912+0000","last_active":"2024-08-03T20:53:17.072880+0000","last_peered":"2024-08-03T20:53:17.072880+0000","last_clean":"2024-08-03T20:53:17.072880+0000","last_became_active":"2024-08-03T20:53:15.578630+0000","last_became_peered":"2024-08-03T20:53:15.578630+0000","last_unstale":"2024-08-03T20:53:17.072880+0000","last_undegraded":"2024-08-03T20:53:17.072880+0000","last_fullsized":"2024-08-03T20:53:17.072880+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:53:14.539353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:53:14.539353+0000","last_clean_scrub_stamp":"2024-08-03T20:53:14.539353+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-04T21:39:44.456389+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.7","version":"69'2","reported_seq":17,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-08-03T20:53:16.569415+0000","last_change":"2024-08-03T20:53:14.571815+0000","last_active":"2024-08-03T20:53:16.569415+0000","last_peered":"2024-08-03T20:53:16.569415+0000","last_clean":"2024-08-03T20:53:16.569415+0000","last_became_active":"2024-08-03T20:53:14.571428+0000","last_became_peered":"2024-08-03T20:53:14.571428+0000","last_unstale":"2024-08-03T20:53:16.569415+0000","last_undegraded":"2024-08-03T20:53:16.569415+0000","last_fullsized":"2024-08-03T20:53:16.569415+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:53:13.536413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:53:13.536413+0000","last_clean_scrub_stamp":"2024-08-03T20:53:13.536413+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-04T23:36:48.632587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.7","version":"0'0","reported_seq":13,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-08-03T20:53:17.072306+0000","last_change":"2024-08-03T20:53:15.590031+0000","last_active":"2024-08-03T20:53:17.072306+0000","last_peered":"2024-08-03T20:53:17.072306+0000","last_clean":"2024-08-03T20:53:17.072306+0000","last_became_active":"2024-08-03T20:53:15.589584+0000","last_became_peered":"2024-08-03T20:53:15.589584+0000","last_unstale":"2024-08-03T20:53:17.072306+0000","last_undegraded":"2024-08-03T20:53:17.072306+0000","last_fullsized":"2024-08-03T20:53:17.072306+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:53:14.539353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:53:14.539353+0000","last_clean_scrub_stamp":"2024-08-03T20:53:14.539353+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T00:41:51.658701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.6","version":"70'1","reported_seq":16,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-08-03T20:53:17.072428+0000","last_change":"2024-08-03T20:53:14.577438+0000","last_active":"2024-08-03T20:53:17.072428+0000","last_peered":"2024-08-03T20:53:17.072428+0000","last_clean":"2024-08-03T20:53:17.072428+0000","last_became_active":"2024-08-03T20:53:14.576996+0000","last_became_peered":"2024-08-03T20:53:14.576996+0000","last_unstale":"2024-08-03T20:53:17.072428+0000","last_undegraded":"2024-08-03T20:53:17.072428+0000","last_fullsized":"2024-08-03T20:53:17.072428+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:53:13.536413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:53:13.536413+0000","last_clean_scrub_stamp":"2024-08-03T20:53:13.536413+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T06:33:31.076324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.4","version":"0'0","reported_seq":13,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-08-03T20:53:17.071987+0000","last_change":"2024-08-03T20:53:15.560626+0000","last_active":"2024-08-03T20:53:17.071987+0000","last_peered":"2024-08-03T20:53:17.071987+0000","last_clean":"2024-08-03T20:53:17.071987+0000","last_became_active":"2024-08-03T20:53:15.560182+0000","last_became_peered":"2024-08-03T20:53:15.560182+0000","last_unstale":"2024-08-03T20:53:17.071987+0000","last_undegraded":"2024-08-03T20:53:17.071987+0000","last_fullsized":"2024-08-03T20:53:17.071987+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:53:14.539353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:53:14.539353+0000","last_clean_scrub_stamp":"2024-08-03T20:53:14.539353+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T01:54:11.293966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.5","version":"0'0","reported_seq":15,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-08-03T20:53:17.071767+0000","last_change":"2024-08-03T20:53:14.566886+0000","last_active":"2024-08-03T20:53:17.071767+0000","last_peered":"2024-08-03T20:53:17.071767+0000","last_clean":"2024-08-03T20:53:17.071767+0000","last_became_active":"2024-08-03T20:53:14.565890+0000","last_became_peered":"2024-08-03T20:53:14.565890+0000","last_unstale":"2024-08-03T20:53:17.071767+0000","last_undegraded":"2024-08-03T20:53:17.071767+0000","last_fullsized":"2024-08-03T20:53:17.071767+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:53:13.536413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:53:13.536413+0000","last_clean_scrub_stamp":"2024-08-03T20:53:13.536413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-04T20:56:54.098182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.5","version":"0'0","reported_seq":13,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-08-03T20:53:17.072023+0000","last_change":"2024-08-03T20:53:15.572003+0000","last_active":"2024-08-03T20:53:17.072023+0000","last_peered":"2024-08-03T20:53:17.072023+0000","last_clean":"2024-08-03T20:53:17.072023+0000","last_became_active":"2024-08-03T20:53:15.570003+0000","last_became_peered":"2024-08-03T20:53:15.570003+0000","last_unstale":"2024-08-03T20:53:17.072023+0000","last_undegraded":"2024-08-03T20:53:17.072023+0000","last_fullsized":"2024-08-03T20:53:17.072023+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:53:14.539353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:53:14.539353+0000","last_clean_scrub_stamp":"2024-08-03T20:53:14.539353+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-04T21:26:20.190785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.4","version":"0'0","reported_seq":15,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-08-03T20:53:17.072274+0000","last_change":"2024-08-03T20:53:14.575843+0000","last_active":"2024-08-03T20:53:17.072274+0000","last_peered":"2024-08-03T20:53:17.072274+0000","last_clean":"2024-08-03T20:53:17.072274+0000","last_became_active":"2024-08-03T20:53:14.575382+0000","last_became_peered":"2024-08-03T20:53:14.575382+0000","last_unstale":"2024-08-03T20:53:17.072274+0000","last_undegraded":"2024-08-03T20:53:17.072274+0000","last_fullsized":"2024-08-03T20:53:17.072274+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:53:13.536413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:53:13.536413+0000","last_clean_scrub_stamp":"2024-08-03T20:53:13.536413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-04T23:34:12.823595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.2","version":"0'0","reported_seq":13,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-08-03T20:53:17.071679+0000","last_change":"2024-08-03T20:53:15.584342+0000","last_active":"2024-08-03T20:53:17.071679+0000","last_peered":"2024-08-03T20:53:17.071679+0000","last_clean":"2024-08-03T20:53:17.071679+0000","last_became_active":"2024-08-03T20:53:15.584016+0000","last_became_peered":"2024-08-03T20:53:15.584016+0000","last_unstale":"2024-08-03T20:53:17.071679+0000","last_undegraded":"2024-08-03T20:53:17.071679+0000","last_fullsized":"2024-08-03T20:53:17.071679+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:53:14.539353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:53:14.539353+0000","last_clean_scrub_stamp":"2024-08-03T20:53:14.539353+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T04:23:40.549686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.3","version":"0'0","reported_seq":15,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-08-03T20:53:17.071353+0000","last_change":"2024-08-03T20:53:14.573474+0000","last_active":"2024-08-03T20:53:17.071353+0000","last_peered":"2024-08-03T20:53:17.071353+0000","last_clean":"2024-08-03T20:53:17.071353+0000","last_became_active":"2024-08-03T20:53:14.572473+0000","last_became_peered":"2024-08-03T20:53:14.572473+0000","last_unstale":"2024-08-03T20:53:17.071353+0000","last_undegraded":"2024-08-03T20:53:17.071353+0000","last_fullsized":"2024-08-03T20:53:17.071353+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:53:13.536413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:53:13.536413+0000","last_clean_scrub_stamp":"2024-08-03T20:53:13.536413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T01:01:12.105547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.3","version":"0'0","reported_seq":13,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-08-03T20:53:16.569177+0000","last_change":"2024-08-03T20:53:15.559710+0000","last_active":"2024-08-03T20:53:16.569177+0000","last_peered":"2024-08-03T20:53:16.569177+0000","last_clean":"2024-08-03T20:53:16.569177+0000","last_became_active":"2024-08-03T20:53:15.558758+0000","last_became_peered":"2024-08-03T20:53:15.558758+0000","last_unstale":"2024-08-03T20:53:16.569177+0000","last_undegraded":"2024-08-03T20:53:16.569177+0000","last_fullsized":"2024-08-03T20:53:16.569177+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:53:14.539353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:53:14.539353+0000","last_clean_scrub_stamp":"2024-08-03T20:53:14.539353+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T03:20:28.849952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.2","version":"0'0","reported_seq":15,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-08-03T20:53:17.071558+0000","last_change":"2024-08-03T20:53:14.576533+0000","last_active":"2024-08-03T20:53:17.071558+0000","last_peered":"2024-08-03T20:53:17.071558+0000","last_clean":"2024-08-03T20:53:17.071558+0000","last_became_active":"2024-08-03T20:53:14.576201+0000","last_became_peered":"2024-08-03T20:53:14.576201+0000","last_unstale":"2024-08-03T20:53:17.071558+0000","last_undegraded":"2024-08-03T20:53:17.071558+0000","last_fullsized":"2024-08-03T20:53:17.071558+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:53:13.536413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:53:13.536413+0000","last_clean_scrub_stamp":"2024-08-03T20:53:13.536413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-04T22:55:46.302530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.0","version":"0'0","reported_seq":13,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-08-03T20:53:17.071268+0000","last_change":"2024-08-03T20:53:15.585610+0000","last_active":"2024-08-03T20:53:17.071268+0000","last_peered":"2024-08-03T20:53:17.071268+0000","last_clean":"2024-08-03T20:53:17.071268+0000","last_became_active":"2024-08-03T20:53:15.583477+0000","last_became_peered":"2024-08-03T20:53:15.583477+0000","last_unstale":"2024-08-03T20:53:17.071268+0000","last_undegraded":"2024-08-03T20:53:17.071268+0000","last_fullsized":"2024-08-03T20:53:17.071268+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:53:14.539353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:53:14.539353+0000","last_clean_scrub_stamp":"2024-08-03T20:53:14.539353+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T01:40:07.476152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.1","version":"0'0","reported_seq":15,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-08-03T20:53:16.569937+0000","last_change":"2024-08-03T20:53:14.566529+0000","last_active":"2024-08-03T20:53:16.569937+0000","last_peered":"2024-08-03T20:53:16.569937+0000","last_clean":"2024-08-03T20:53:16.569937+0000","last_became_active":"2024-08-03T20:53:14.566184+0000","last_became_peered":"2024-08-03T20:53:14.566184+0000","last_unstale":"2024-08-03T20:53:16.569937+0000","last_undegraded":"2024-08-03T20:53:16.569937+0000","last_fullsized":"2024-08-03T20:53:16.569937+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:53:13.536413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:53:13.536413+0000","last_clean_scrub_stamp":"2024-08-03T20:53:13.536413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T06:59:52.597902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.1","version":"0'0","reported_seq":13,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-08-03T20:53:16.568718+0000","last_change":"2024-08-03T20:53:15.573839+0000","last_active":"2024-08-03T20:53:16.568718+0000","last_peered":"2024-08-03T20:53:16.568718+0000","last_clean":"2024-08-03T20:53:16.568718+0000","last_became_active":"2024-08-03T20:53:15.571756+0000","last_became_peered":"2024-08-03T20:53:15.571756+0000","last_unstale":"2024-08-03T20:53:16.568718+0000","last_undegraded":"2024-08-03T20:53:16.568718+0000","last_fullsized":"2024-08-03T20:53:16.568718+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:53:14.539353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:53:14.539353+0000","last_clean_scrub_stamp":"2024-08-03T20:53:14.539353+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-04T21:06:58.191555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.0","version":"0'0","reported_seq":15,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-08-03T20:53:16.569275+0000","last_change":"2024-08-03T20:53:14.568012+0000","last_active":"2024-08-03T20:53:16.569275+0000","last_peered":"2024-08-03T20:53:16.569275+0000","last_clean":"2024-08-03T20:53:16.569275+0000","last_became_active":"2024-08-03T20:53:14.567694+0000","last_became_peered":"2024-08-03T20:53:14.567694+0000","last_unstale":"2024-08-03T20:53:16.569275+0000","last_undegraded":"2024-08-03T20:53:16.569275+0000","last_fullsized":"2024-08-03T20:53:16.569275+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:53:13.536413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:53:13.536413+0000","last_clean_scrub_stamp":"2024-08-03T20:53:13.536413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-04T22:55:46.302530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.a","version":"0'0","reported_seq":13,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-08-03T20:53:17.072620+0000","last_change":"2024-08-03T20:53:15.580530+0000","last_active":"2024-08-03T20:53:17.072620+0000","last_peered":"2024-08-03T20:53:17.072620+0000","last_clean":"2024-08-03T20:53:17.072620+0000","last_became_active":"2024-08-03T20:53:15.579218+0000","last_became_peered":"2024-08-03T20:53:15.579218+0000","last_unstale":"2024-08-03T20:53:17.072620+0000","last_undegraded":"2024-08-03T20:53:17.072620+0000","last_fullsized":"2024-08-03T20:53:17.072620+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:53:14.539353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:53:14.539353+0000","last_clean_scrub_stamp":"2024-08-03T20:53:14.539353+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T04:34:14.624360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.b","version":"69'2","reported_seq":17,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-08-03T20:53:16.569395+0000","last_change":"2024-08-03T20:53:14.565691+0000","last_active":"2024-08-03T20:53:16.569395+0000","last_peered":"2024-08-03T20:53:16.569395+0000","last_clean":"2024-08-03T20:53:16.569395+0000","last_became_active":"2024-08-03T20:53:14.565364+0000","last_became_peered":"2024-08-03T20:53:14.565364+0000","last_unstale":"2024-08-03T20:53:16.569395+0000","last_undegraded":"2024-08-03T20:53:16.569395+0000","last_fullsized":"2024-08-03T20:53:16.569395+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:53:13.536413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:53:13.536413+0000","last_clean_scrub_stamp":"2024-08-03T20:53:13.536413+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T02:39:20.815693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"16'72","reported_seq":201,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-08-03T20:53:17.072305+0000","last_change":"2024-08-03T20:47:30.547632+0000","last_active":"2024-08-03T20:53:17.072305+0000","last_peered":"2024-08-03T20:53:17.072305+0000","last_clean":"2024-08-03T20:53:17.072305+0000","last_became_active":"2024-08-03T20:47:30.546255+0000","last_became_peered":"2024-08-03T20:47:30.546255+0000","last_unstale":"2024-08-03T20:53:17.072305+0000","last_undegraded":"2024-08-03T20:53:17.072305+0000","last_fullsized":"2024-08-03T20:53:17.072305+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:47:28.874451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:47:28.874451+0000","last_clean_scrub_stamp":"2024-08-03T20:47:28.874451+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T00:19:40.548812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":11,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":10,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509513,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32592,"kb_used_data":1412,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339248,"statfs":{"total":96636764160,"available":96603389952,"internally_reserved":0,"allocated":1445888,"data_stored":947940,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":6,"up_from":15,"seq":64424509514,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32020,"kb_used_data":832,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339820,"statfs":{"total":96636764160,"available":96603975680,"internally_reserved":0,"allocated":851968,"data_stored":357048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":5,"up_from":15,"seq":64424509514,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32008,"kb_used_data":828,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339832,"statfs":{"total":96636764160,"available":96603987968,"internally_reserved":0,"allocated":847872,"data_stored":357046,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509513,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32024,"kb_used_data":844,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339816,"statfs":{"total":96636764160,"available":96603971584,"internally_reserved":0,"allocated":864256,"data_stored":359184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509515,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36748,"kb_used_data":1404,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335092,"statfs":{"total":96636764160,"available":96599134208,"internally_reserved":0,"allocated":1437696,"data_stored":947324,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542218,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32012,"kb_used_data":828,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339828,"statfs":{"total":96636764160,"available":96603983872,"internally_reserved":0,"allocated":847872,"data_stored":357026,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":3,"up_from":14,"seq":60129542216,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32016,"kb_used_data":832,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339824,"statfs":{"total":96636764160,"available":96603979776,"internally_reserved":0,"allocated":851968,"data_stored":357068,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":2,"up_from":14,"seq":60129542217,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32020,"kb_used_data":836,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339820,"statfs":{"total":96636764160,"available":96603975680,"internally_reserved":0,"allocated":856064,"data_stored":358568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2228,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-08-03T20:53:20.603 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-08-03T20:53:20.603 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-08-03T20:53:20.879 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:20.878+0000 7f6d67e00640 1 -- 172.21.15.32:0/2345729228 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6d600cf9c0 msgr2=0x7f6d600d0a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:53:20.879 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:20.878+0000 7f6d67e00640 1 --2- 172.21.15.32:0/2345729228 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6d600cf9c0 0x7f6d600d0a70 secure :-1 s=READY pgs=767 cs=0 l=1 rev1=1 crypto rx=0x7f6d4c009830 tx=0x7f6d4c02a5a0 comp rx=0 tx=0).stop 2024-08-03T20:53:20.879 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:20.878+0000 7f6d67e00640 1 -- 172.21.15.32:0/2345729228 shutdown_connections 2024-08-03T20:53:20.879 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:20.878+0000 7f6d67e00640 1 --2- 172.21.15.32:0/2345729228 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f6d600d0fb0 0x7f6d600b7d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:53:20.879 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:20.878+0000 7f6d67e00640 1 --2- 172.21.15.32:0/2345729228 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6d600cf9c0 0x7f6d600d0a70 unknown :-1 s=CLOSED pgs=767 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:53:20.879 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:20.878+0000 7f6d67e00640 1 -- 172.21.15.32:0/2345729228 >> 172.21.15.32:0/2345729228 conn(0x7f6d6009d660 msgr2=0x7f6d600d4330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:53:20.879 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:20.878+0000 7f6d67e00640 1 -- 172.21.15.32:0/2345729228 shutdown_connections 2024-08-03T20:53:20.879 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:20.878+0000 7f6d67e00640 1 -- 172.21.15.32:0/2345729228 wait complete. 2024-08-03T20:53:20.880 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:20.880+0000 7f6d67e00640 1 Processor -- start 2024-08-03T20:53:20.881 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:20.880+0000 7f6d67e00640 1 -- start start 2024-08-03T20:53:20.883 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:20.882+0000 7f6d67e00640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f6d600b59f0 0x7f6d6015c2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:53:20.883 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:20.882+0000 7f6d67e00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f6d600cf9c0 0x7f6d600b39e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:53:20.883 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:20.882+0000 7f6d67e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6d600d0fb0 0x7f6d601f80d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:53:20.883 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:20.882+0000 7f6d67e00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f6d600b8870 con 0x7f6d600d0fb0 2024-08-03T20:53:20.884 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:20.882+0000 7f6d67e00640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f6d600b86f0 con 0x7f6d600cf9c0 2024-08-03T20:53:20.884 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:20.882+0000 7f6d67e00640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7f6d600b89f0 con 0x7f6d600b59f0 2024-08-03T20:53:20.884 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:20.882+0000 7f6d65e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6d600d0fb0 0x7f6d601f80d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:53:20.884 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:20.882+0000 7f6d65e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6d600d0fb0 0x7f6d601f80d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:45506/0 (socket says 172.21.15.32:45506) 2024-08-03T20:53:20.884 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:20.882+0000 7f6d65e00640 1 -- 172.21.15.32:0/1179383721 learned_addr learned my addr 172.21.15.32:0/1179383721 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:53:20.884 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:20.882+0000 7f6d65400640 1 --2- 172.21.15.32:0/1179383721 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f6d600b59f0 0x7f6d6015c2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:53:20.884 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:20.882+0000 7f6d64a00640 1 --2- 172.21.15.32:0/1179383721 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f6d600cf9c0 0x7f6d600b39e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:53:20.884 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:20.882+0000 7f6d65e00640 1 -- 172.21.15.32:0/1179383721 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f6d600b59f0 msgr2=0x7f6d6015c2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:53:20.884 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:20.882+0000 7f6d65e00640 1 --2- 172.21.15.32:0/1179383721 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f6d600b59f0 0x7f6d6015c2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:53:20.884 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:20.882+0000 7f6d65e00640 1 -- 172.21.15.32:0/1179383721 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f6d600cf9c0 msgr2=0x7f6d600b39e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:53:20.884 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:20.882+0000 7f6d65e00640 1 --2- 172.21.15.32:0/1179383721 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f6d600cf9c0 0x7f6d600b39e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:53:20.885 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:20.882+0000 7f6d65e00640 1 -- 172.21.15.32:0/1179383721 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6d600b13a0 con 0x7f6d600d0fb0 2024-08-03T20:53:20.885 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:20.883+0000 7f6d65e00640 1 --2- 172.21.15.32:0/1179383721 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6d600d0fb0 0x7f6d601f80d0 secure :-1 s=READY pgs=768 cs=0 l=1 rev1=1 crypto rx=0x7f6d5800e990 tx=0x7f6d5800ee50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:53:20.885 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:20.883+0000 7f6d5d600640 1 -- 172.21.15.32:0/1179383721 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d58004d20 con 0x7f6d600d0fb0 2024-08-03T20:53:20.885 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:20.883+0000 7f6d5d600640 1 -- 172.21.15.32:0/1179383721 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f6d5800caa0 con 0x7f6d600d0fb0 2024-08-03T20:53:20.885 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:20.883+0000 7f6d67e00640 1 -- 172.21.15.32:0/1179383721 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6d600b1680 con 0x7f6d600d0fb0 2024-08-03T20:53:20.885 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:20.883+0000 7f6d5d600640 1 -- 172.21.15.32:0/1179383721 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d58005020 con 0x7f6d600d0fb0 2024-08-03T20:53:20.886 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:20.883+0000 7f6d67e00640 1 -- 172.21.15.32:0/1179383721 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6d600b1c40 con 0x7f6d600d0fb0 2024-08-03T20:53:20.886 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:20.884+0000 7f6d67e00640 1 -- 172.21.15.32:0/1179383721 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6d600b8ea0 con 0x7f6d600d0fb0 2024-08-03T20:53:20.886 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:20.885+0000 7f6d5d600640 1 -- 172.21.15.32:0/1179383721 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f6d580051c0 con 0x7f6d600d0fb0 2024-08-03T20:53:20.887 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:20.886+0000 7f6d5d600640 1 --2- 172.21.15.32:0/1179383721 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f6d40077040 0x7f6d40079550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:53:20.887 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:20.886+0000 7f6d5d600640 1 -- 172.21.15.32:0/1179383721 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(70..70 src has 1..70) ==== 5709+0+0 (secure 0 0 0) 0x7f6d580991a0 con 0x7f6d600d0fb0 2024-08-03T20:53:20.887 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:20.887+0000 7f6d65400640 1 --2- 172.21.15.32:0/1179383721 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f6d40077040 0x7f6d40079550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:53:20.888 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:20.888+0000 7f6d65400640 1 --2- 172.21.15.32:0/1179383721 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f6d40077040 0x7f6d40079550 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f6d540015e0 tx=0x7f6d54009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:53:20.891 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:20.891+0000 7f6d5d600640 1 -- 172.21.15.32:0/1179383721 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7f6d58016030 con 0x7f6d600d0fb0 2024-08-03T20:53:21.174 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:21.173+0000 7f6d67e00640 1 -- 172.21.15.32:0/1179383721 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) -- 0x7f6d600ce310 con 0x7f6d600d0fb0 2024-08-03T20:53:21.177 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:21.177+0000 7f6d5d600640 1 -- 172.21.15.32:0/1179383721 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v14) ==== 190+0+147 (secure 0 0 0) 0x7f6d58062300 con 0x7f6d600d0fb0 2024-08-03T20:53:21.177 INFO:teuthology.orchestra.run.smithi032.stdout:[client.0] 2024-08-03T20:53:21.177 INFO:teuthology.orchestra.run.smithi032.stdout: key = AQAwl65msgO3NRAAHG/WuMTK9c4lDXocXvqWnA== 2024-08-03T20:53:21.177 INFO:teuthology.orchestra.run.smithi032.stdout: caps mds = "allow" 2024-08-03T20:53:21.178 INFO:teuthology.orchestra.run.smithi032.stdout: caps mon = "allow r" 2024-08-03T20:53:21.178 INFO:teuthology.orchestra.run.smithi032.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2024-08-03T20:53:21.178 INFO:teuthology.orchestra.run.smithi032.stderr:updated caps for client.0 2024-08-03T20:53:21.180 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:21.179+0000 7f6d67e00640 1 -- 172.21.15.32:0/1179383721 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f6d40077040 msgr2=0x7f6d40079550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:53:21.180 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:21.179+0000 7f6d67e00640 1 --2- 172.21.15.32:0/1179383721 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f6d40077040 0x7f6d40079550 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f6d540015e0 tx=0x7f6d54009040 comp rx=0 tx=0).stop 2024-08-03T20:53:21.180 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:21.179+0000 7f6d67e00640 1 -- 172.21.15.32:0/1179383721 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6d600d0fb0 msgr2=0x7f6d601f80d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:53:21.180 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:21.179+0000 7f6d67e00640 1 --2- 172.21.15.32:0/1179383721 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6d600d0fb0 0x7f6d601f80d0 secure :-1 s=READY pgs=768 cs=0 l=1 rev1=1 crypto rx=0x7f6d5800e990 tx=0x7f6d5800ee50 comp rx=0 tx=0).stop 2024-08-03T20:53:21.180 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:21.179+0000 7f6d67e00640 1 -- 172.21.15.32:0/1179383721 shutdown_connections 2024-08-03T20:53:21.181 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:21.179+0000 7f6d67e00640 1 --2- 172.21.15.32:0/1179383721 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f6d40077040 0x7f6d40079550 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:53:21.181 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:21.179+0000 7f6d67e00640 1 --2- 172.21.15.32:0/1179383721 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f6d600cf9c0 0x7f6d600b39e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:53:21.181 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:21.179+0000 7f6d67e00640 1 --2- 172.21.15.32:0/1179383721 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6d600d0fb0 0x7f6d601f80d0 unknown :-1 s=CLOSED pgs=768 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:53:21.181 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:21.179+0000 7f6d67e00640 1 --2- 172.21.15.32:0/1179383721 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f6d600b59f0 0x7f6d6015c2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:53:21.181 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:21.179+0000 7f6d67e00640 1 -- 172.21.15.32:0/1179383721 >> 172.21.15.32:0/1179383721 conn(0x7f6d6009d660 msgr2=0x7f6d6006d680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:53:21.181 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:21.179+0000 7f6d67e00640 1 -- 172.21.15.32:0/1179383721 shutdown_connections 2024-08-03T20:53:21.181 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:21.180+0000 7f6d67e00640 1 -- 172.21.15.32:0/1179383721 wait complete. 2024-08-03T20:53:21.205 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-08-03T20:53:21.433 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:21.431+0000 7f2317e00640 1 -- 172.21.15.32:0/3334383036 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f23180cf510 msgr2=0x7f23180cf920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:53:21.434 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:21.432+0000 7f2317e00640 1 --2- 172.21.15.32:0/3334383036 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f23180cf510 0x7f23180cf920 secure :-1 s=READY pgs=769 cs=0 l=1 rev1=1 crypto rx=0x7f2310005fd0 tx=0x7f231002a9f0 comp rx=0 tx=0).stop 2024-08-03T20:53:21.434 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:21.432+0000 7f2317e00640 1 -- 172.21.15.32:0/3334383036 shutdown_connections 2024-08-03T20:53:21.434 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:21.432+0000 7f2317e00640 1 --2- 172.21.15.32:0/3334383036 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f23180cf510 0x7f23180cf920 unknown :-1 s=CLOSED pgs=769 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:53:21.434 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:21.432+0000 7f2317e00640 1 --2- 172.21.15.32:0/3334383036 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f23180d26e0 0x7f2318161b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:53:21.434 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:21.432+0000 7f2317e00640 1 -- 172.21.15.32:0/3334383036 >> 172.21.15.32:0/3334383036 conn(0x7f231808d7a0 msgr2=0x7f231809d3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:53:21.434 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:21.432+0000 7f2317e00640 1 -- 172.21.15.32:0/3334383036 shutdown_connections 2024-08-03T20:53:21.434 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:21.432+0000 7f2317e00640 1 -- 172.21.15.32:0/3334383036 wait complete. 2024-08-03T20:53:21.435 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:21.434+0000 7f2317e00640 1 Processor -- start 2024-08-03T20:53:21.435 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:21.434+0000 7f2317e00640 1 -- start start 2024-08-03T20:53:21.438 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:21.435+0000 7f2317e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f23180cf510 0x7f23181f8360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:53:21.438 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:21.435+0000 7f2317e00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f23180cffe0 0x7f23180cba00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:53:21.438 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:21.435+0000 7f2317e00640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f23180d26e0 0x7f23180cbf40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:53:21.438 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:21.435+0000 7f2317e00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f23180d4310 con 0x7f23180cf510 2024-08-03T20:53:21.438 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:21.435+0000 7f2317e00640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f23180d4190 con 0x7f23180cffe0 2024-08-03T20:53:21.438 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:21.435+0000 7f2317e00640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7f23180d4490 con 0x7f23180d26e0 2024-08-03T20:53:21.438 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:21.436+0000 7f2316a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f23180cf510 0x7f23181f8360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:53:21.438 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:21.436+0000 7f2316a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f23180cf510 0x7f23181f8360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:45520/0 (socket says 172.21.15.32:45520) 2024-08-03T20:53:21.438 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:21.436+0000 7f2316a00640 1 -- 172.21.15.32:0/4137716806 learned_addr learned my addr 172.21.15.32:0/4137716806 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:53:21.439 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:21.436+0000 7f2317400640 1 --2- 172.21.15.32:0/4137716806 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f23180d26e0 0x7f23180cbf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:53:21.439 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:21.436+0000 7f230fe00640 1 --2- 172.21.15.32:0/4137716806 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f23180cffe0 0x7f23180cba00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:53:21.439 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:21.436+0000 7f2316a00640 1 -- 172.21.15.32:0/4137716806 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f23180d26e0 msgr2=0x7f23180cbf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:53:21.439 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:21.436+0000 7f2316a00640 1 --2- 172.21.15.32:0/4137716806 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f23180d26e0 0x7f23180cbf40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:53:21.439 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:21.436+0000 7f2316a00640 1 -- 172.21.15.32:0/4137716806 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f23180cffe0 msgr2=0x7f23180cba00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:53:21.439 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:21.436+0000 7f2316a00640 1 --2- 172.21.15.32:0/4137716806 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f23180cffe0 0x7f23180cba00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:53:21.439 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:21.436+0000 7f2316a00640 1 -- 172.21.15.32:0/4137716806 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f23180d0d00 con 0x7f23180cf510 2024-08-03T20:53:21.439 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:21.437+0000 7f2316a00640 1 --2- 172.21.15.32:0/4137716806 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f23180cf510 0x7f23181f8360 secure :-1 s=READY pgs=770 cs=0 l=1 rev1=1 crypto rx=0x7f230800c9a0 tx=0x7f230800ce60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:53:21.440 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:21.437+0000 7f230f400640 1 -- 172.21.15.32:0/4137716806 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2308007b50 con 0x7f23180cf510 2024-08-03T20:53:21.440 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:21.437+0000 7f230f400640 1 -- 172.21.15.32:0/4137716806 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f2308007de0 con 0x7f23180cf510 2024-08-03T20:53:21.440 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:21.437+0000 7f230f400640 1 -- 172.21.15.32:0/4137716806 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f230800f6b0 con 0x7f23180cf510 2024-08-03T20:53:21.440 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:21.437+0000 7f2317e00640 1 -- 172.21.15.32:0/4137716806 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f23180d0fa0 con 0x7f23180cf510 2024-08-03T20:53:21.440 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:21.437+0000 7f2317e00640 1 -- 172.21.15.32:0/4137716806 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f23180cd990 con 0x7f23180cf510 2024-08-03T20:53:21.440 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:21.438+0000 7f2317e00640 1 -- 172.21.15.32:0/4137716806 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f23180d4940 con 0x7f23180cf510 2024-08-03T20:53:21.441 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:21.439+0000 7f230f400640 1 -- 172.21.15.32:0/4137716806 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f230800f850 con 0x7f23180cf510 2024-08-03T20:53:21.441 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:21.440+0000 7f230f400640 1 --2- 172.21.15.32:0/4137716806 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f22f4077010 0x7f22f4079520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:53:21.441 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:21.440+0000 7f230f400640 1 -- 172.21.15.32:0/4137716806 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(70..70 src has 1..70) ==== 5709+0+0 (secure 0 0 0) 0x7f2308098fa0 con 0x7f23180cf510 2024-08-03T20:53:21.442 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:21.441+0000 7f230fe00640 1 --2- 172.21.15.32:0/4137716806 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f22f4077010 0x7f22f4079520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:53:21.443 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:21.441+0000 7f230fe00640 1 --2- 172.21.15.32:0/4137716806 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f22f4077010 0x7f22f4079520 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f2300006fd0 tx=0x7f2300008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:53:21.446 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:21.444+0000 7f230f400640 1 -- 172.21.15.32:0/4137716806 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7f2308014030 con 0x7f23180cf510 2024-08-03T20:53:21.691 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:21.689+0000 7f2317e00640 1 -- 172.21.15.32:0/4137716806 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f23180a1b60 con 0x7f23180cf510 2024-08-03T20:53:21.691 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:21.690+0000 7f230f400640 1 -- 172.21.15.32:0/4137716806 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 74 v74) ==== 94+0+5231 (secure 0 0 0) 0x7f23080621d0 con 0x7f23180cf510 2024-08-03T20:53:21.691 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-03T20:53:21.692 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":74,"btime":"2024-08-03T20:53:18:574147+0000","default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5666,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6837/2261538396","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6835","nonce":2261538396},{"type":"v1","addr":"172.21.15.118:6837","nonce":2261538396}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":69},{"gid":6061,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6836/1133580656","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6834","nonce":1133580656},{"type":"v1","addr":"172.21.15.118:6836","nonce":1133580656}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":68},{"gid":6396,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6834/1798758431","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":1798758431},{"type":"v1","addr":"172.21.15.32:6834","nonce":1798758431}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":69}],"filesystems":[{"mdsmap":{"epoch":74,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-08-03T20:53:15.080669+0000","modified":"2024-08-03T20:53:18.096135+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6399},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6399":{"gid":6399,"name":"a","rank":0,"incarnation":70,"state":"up:active","state_seq":3,"addr":"172.21.15.32:6835/4240363667","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6833","nonce":4240363667},{"type":"v1","addr":"172.21.15.32:6835","nonce":4240363667}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6399,"qdb_cluster":[6399]},"id":5}]} 2024-08-03T20:53:21.692 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 74 2024-08-03T20:53:21.694 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:21.692+0000 7f2317e00640 1 -- 172.21.15.32:0/4137716806 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f22f4077010 msgr2=0x7f22f4079520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:53:21.694 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:21.692+0000 7f2317e00640 1 --2- 172.21.15.32:0/4137716806 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f22f4077010 0x7f22f4079520 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f2300006fd0 tx=0x7f2300008040 comp rx=0 tx=0).stop 2024-08-03T20:53:21.694 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:21.692+0000 7f2317e00640 1 -- 172.21.15.32:0/4137716806 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f23180cf510 msgr2=0x7f23181f8360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:53:21.694 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:21.692+0000 7f2317e00640 1 --2- 172.21.15.32:0/4137716806 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f23180cf510 0x7f23181f8360 secure :-1 s=READY pgs=770 cs=0 l=1 rev1=1 crypto rx=0x7f230800c9a0 tx=0x7f230800ce60 comp rx=0 tx=0).stop 2024-08-03T20:53:21.694 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:21.692+0000 7f2317e00640 1 -- 172.21.15.32:0/4137716806 shutdown_connections 2024-08-03T20:53:21.694 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:21.692+0000 7f2317e00640 1 --2- 172.21.15.32:0/4137716806 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f22f4077010 0x7f22f4079520 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:53:21.694 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:21.692+0000 7f2317e00640 1 --2- 172.21.15.32:0/4137716806 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f23180cffe0 0x7f23180cba00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:53:21.694 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:21.692+0000 7f2317e00640 1 --2- 172.21.15.32:0/4137716806 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f23180d26e0 0x7f23180cbf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:53:21.695 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:21.692+0000 7f2317e00640 1 --2- 172.21.15.32:0/4137716806 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f23180cf510 0x7f23181f8360 unknown :-1 s=CLOSED pgs=770 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:53:21.695 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:21.692+0000 7f2317e00640 1 -- 172.21.15.32:0/4137716806 >> 172.21.15.32:0/4137716806 conn(0x7f231808d7a0 msgr2=0x7f23180b3030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:53:21.695 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:21.692+0000 7f2317e00640 1 -- 172.21.15.32:0/4137716806 shutdown_connections 2024-08-03T20:53:21.695 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:21.692+0000 7f2317e00640 1 -- 172.21.15.32:0/4137716806 wait complete. 2024-08-03T20:53:22.123 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 74, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-08-03T20:53:15.080669+0000', 'modified': '2024-08-03T20:53:18.096135+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 6399}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6399': {'gid': 6399, 'name': 'a', 'rank': 0, 'incarnation': 70, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.32:6835/4240363667', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6833', 'nonce': 4240363667}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 4240363667}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [11], 'metadata_pool': 10, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 6399, 'qdb_cluster': [6399]} 2024-08-03T20:53:22.123 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-08-03T20:53:22.123 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2024-08-03T20:53:22.354 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:22.352+0000 7feb10000640 1 -- 172.21.15.32:0/3431688329 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7feaf0004800 msgr2=0x7feaf0004c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:53:22.354 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:22.352+0000 7feb10000640 1 --2- 172.21.15.32:0/3431688329 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7feaf0004800 0x7feaf0004c10 secure :-1 s=READY pgs=772 cs=0 l=1 rev1=1 crypto rx=0x7feb08072880 tx=0x7feaf0005830 comp rx=0 tx=0).stop 2024-08-03T20:53:22.355 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:22.352+0000 7feb0e000640 1 -- 172.21.15.32:0/3431688329 reap_dead start 2024-08-03T20:53:22.355 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:22.353+0000 7feb10000640 1 -- 172.21.15.32:0/3431688329 shutdown_connections 2024-08-03T20:53:22.355 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:22.353+0000 7feb10000640 1 -- 172.21.15.32:0/3431688329 >> 172.21.15.32:0/3431688329 conn(0x7feb080b1bd0 msgr2=0x7feb080b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:53:22.355 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:22.353+0000 7feb10000640 1 -- 172.21.15.32:0/3431688329 shutdown_connections 2024-08-03T20:53:22.355 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:22.353+0000 7feb10000640 1 -- 172.21.15.32:0/3431688329 wait complete. 2024-08-03T20:53:22.356 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:22.355+0000 7feb10000640 1 Processor -- start 2024-08-03T20:53:22.356 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:22.355+0000 7feb10000640 1 -- start start 2024-08-03T20:53:22.359 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:22.356+0000 7feb10000640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7feaf0004800 0x7feb081431d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:53:22.359 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:22.357+0000 7feb10000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7feb08143710 0x7feb081f8b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:53:22.359 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:22.357+0000 7feb10000640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7feb081f9080 0x7feb08139980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:53:22.359 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:22.357+0000 7feb10000640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7feb080cf840 con 0x7feb08143710 2024-08-03T20:53:22.359 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:22.357+0000 7feb10000640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7feb08145f30 con 0x7feaf0004800 2024-08-03T20:53:22.359 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:22.357+0000 7feb10000640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7feb080cf9c0 con 0x7feb081f9080 2024-08-03T20:53:22.359 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:22.357+0000 7feb0cc00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7feb08143710 0x7feb081f8b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:53:22.359 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:22.357+0000 7feb0cc00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7feb08143710 0x7feb081f8b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:45532/0 (socket says 172.21.15.32:45532) 2024-08-03T20:53:22.359 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:22.357+0000 7feb0cc00640 1 -- 172.21.15.32:0/4189583763 learned_addr learned my addr 172.21.15.32:0/4189583763 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:53:22.359 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:22.357+0000 7feb0d600640 1 --2- 172.21.15.32:0/4189583763 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7feaf0004800 0x7feb081431d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:53:22.359 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:22.357+0000 7feb0e000640 1 --2- 172.21.15.32:0/4189583763 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7feb081f9080 0x7feb08139980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:53:22.360 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:22.357+0000 7feb0cc00640 1 -- 172.21.15.32:0/4189583763 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7feb081f9080 msgr2=0x7feb08139980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:53:22.360 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:22.357+0000 7feb0cc00640 1 --2- 172.21.15.32:0/4189583763 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7feb081f9080 0x7feb08139980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:53:22.360 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:22.357+0000 7feb0cc00640 1 -- 172.21.15.32:0/4189583763 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7feaf0004800 msgr2=0x7feb081431d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:53:22.360 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:22.357+0000 7feb0cc00640 1 --2- 172.21.15.32:0/4189583763 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7feaf0004800 0x7feb081431d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:53:22.360 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:22.357+0000 7feb0cc00640 1 -- 172.21.15.32:0/4189583763 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feb081f98f0 con 0x7feb08143710 2024-08-03T20:53:22.360 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:22.357+0000 7feb0d600640 1 --2- 172.21.15.32:0/4189583763 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7feaf0004800 0x7feb081431d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-03T20:53:22.361 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:22.358+0000 7feb0cc00640 1 --2- 172.21.15.32:0/4189583763 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7feb08143710 0x7feb081f8b40 secure :-1 s=READY pgs=773 cs=0 l=1 rev1=1 crypto rx=0x7feaf400c350 tx=0x7feaf4006420 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:53:22.361 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:22.358+0000 7feb05600640 1 -- 172.21.15.32:0/4189583763 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feaf4032030 con 0x7feb08143710 2024-08-03T20:53:22.361 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:22.358+0000 7feb05600640 1 -- 172.21.15.32:0/4189583763 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7feaf4003af0 con 0x7feb08143710 2024-08-03T20:53:22.361 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:22.358+0000 7feb10000640 1 -- 172.21.15.32:0/4189583763 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7feb081f5520 con 0x7feb08143710 2024-08-03T20:53:22.361 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:22.358+0000 7feb05600640 1 -- 172.21.15.32:0/4189583763 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feaf4042480 con 0x7feb08143710 2024-08-03T20:53:22.361 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:22.358+0000 7feb10000640 1 -- 172.21.15.32:0/4189583763 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7feb081f5ae0 con 0x7feb08143710 2024-08-03T20:53:22.362 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:22.360+0000 7feb05600640 1 -- 172.21.15.32:0/4189583763 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7feaf4035020 con 0x7feb08143710 2024-08-03T20:53:22.363 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:22.361+0000 7feb05600640 1 --2- 172.21.15.32:0/4189583763 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7feae8077010 0x7feae8079520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:53:22.363 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:22.361+0000 7feb05600640 1 -- 172.21.15.32:0/4189583763 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(70..70 src has 1..70) ==== 5709+0+0 (secure 0 0 0) 0x7feaf40b9800 con 0x7feb08143710 2024-08-03T20:53:22.363 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:22.362+0000 7feb0d600640 1 --2- 172.21.15.32:0/4189583763 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7feae8077010 0x7feae8079520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:53:22.363 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:22.362+0000 7feb16f8c740 1 Processor -- start 2024-08-03T20:53:22.364 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:22.362+0000 7feb16f8c740 1 -- start start 2024-08-03T20:53:22.364 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:22.363+0000 7feb0d600640 1 --2- 172.21.15.32:0/4189583763 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7feae8077010 0x7feae8079520 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7feafc009b60 tx=0x7feafc005f90 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:53:22.366 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:22.364+0000 7feb16f8c740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555b11cea270 0x555b120d70d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:53:22.366 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:22.364+0000 7feb16f8c740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x555b11cc13b0 0x555b120b9da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:53:22.366 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:22.364+0000 7feb16f8c740 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x555b11cc9590 0x555b120ba2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:53:22.366 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:22.364+0000 7feb16f8c740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x555b12002a60 con 0x555b11cea270 2024-08-03T20:53:22.366 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:22.364+0000 7feb16f8c740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x555b11ff1f60 con 0x555b11cc13b0 2024-08-03T20:53:22.366 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:22.364+0000 7feb16f8c740 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x555b11ff5fa0 con 0x555b11cc9590 2024-08-03T20:53:22.367 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:22.364+0000 7feb0d600640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555b11cea270 0x555b120d70d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:53:22.367 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:22.364+0000 7feb0d600640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555b11cea270 0x555b120d70d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:45534/0 (socket says 172.21.15.32:45534) 2024-08-03T20:53:22.367 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:22.364+0000 7feb0d600640 1 -- 172.21.15.32:0/960553878 learned_addr learned my addr 172.21.15.32:0/960553878 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:53:22.367 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:22.364+0000 7feb0cc00640 1 --2- 172.21.15.32:0/960553878 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x555b11cc13b0 0x555b120b9da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:53:22.367 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:22.364+0000 7feb0e000640 1 --2- 172.21.15.32:0/960553878 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x555b11cc9590 0x555b120ba2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:53:22.367 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:22.364+0000 7feb0d600640 1 -- 172.21.15.32:0/960553878 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x555b11cc9590 msgr2=0x555b120ba2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:53:22.367 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:22.364+0000 7feb0d600640 1 --2- 172.21.15.32:0/960553878 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x555b11cc9590 0x555b120ba2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:53:22.367 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:22.364+0000 7feb0d600640 1 -- 172.21.15.32:0/960553878 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x555b11cc13b0 msgr2=0x555b120b9da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:53:22.367 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:22.364+0000 7feb0d600640 1 --2- 172.21.15.32:0/960553878 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x555b11cc13b0 0x555b120b9da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:53:22.367 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:22.364+0000 7feb0d600640 1 -- 172.21.15.32:0/960553878 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555b11d55850 con 0x555b11cea270 2024-08-03T20:53:22.367 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:22.365+0000 7feb0d600640 1 --2- 172.21.15.32:0/960553878 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555b11cea270 0x555b120d70d0 secure :-1 s=READY pgs=774 cs=0 l=1 rev1=1 crypto rx=0x7feafc0118c0 tx=0x7feafc016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=50e21bfe95b0aca1 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:53:22.369 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:22.365+0000 7feaf9c00640 1 -- 172.21.15.32:0/960553878 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feafc01eb80 con 0x555b11cea270 2024-08-03T20:53:22.369 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:22.365+0000 7feaf9c00640 1 -- 172.21.15.32:0/960553878 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7feafc01ed20 con 0x555b11cea270 2024-08-03T20:53:22.369 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:22.365+0000 7feaf9c00640 1 -- 172.21.15.32:0/960553878 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feafc027660 con 0x555b11cea270 2024-08-03T20:53:22.369 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:22.366+0000 7feb16f8c740 1 -- 172.21.15.32:0/960553878 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555b11cea270 msgr2=0x555b120d70d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:53:22.369 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:22.366+0000 7feb16f8c740 1 --2- 172.21.15.32:0/960553878 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555b11cea270 0x555b120d70d0 secure :-1 s=READY pgs=774 cs=0 l=1 rev1=1 crypto rx=0x7feafc0118c0 tx=0x7feafc016000 comp rx=0 tx=0).stop 2024-08-03T20:53:22.369 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:22.366+0000 7feb16f8c740 1 -- 172.21.15.32:0/960553878 shutdown_connections 2024-08-03T20:53:22.369 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:22.366+0000 7feb16f8c740 1 --2- 172.21.15.32:0/960553878 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x555b11cc13b0 0x555b120b9da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:53:22.369 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:22.366+0000 7feb16f8c740 1 --2- 172.21.15.32:0/960553878 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x555b11cc9590 0x555b120ba2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:53:22.369 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:22.366+0000 7feb16f8c740 1 --2- 172.21.15.32:0/960553878 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555b11cea270 0x555b120d70d0 unknown :-1 s=CLOSED pgs=774 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:53:22.369 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:22.366+0000 7feb16f8c740 1 -- 172.21.15.32:0/960553878 >> 172.21.15.32:0/960553878 conn(0x555b11cde3c0 msgr2=0x555b1208ffc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:53:22.370 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:22.366+0000 7feb16f8c740 1 -- 172.21.15.32:0/960553878 shutdown_connections 2024-08-03T20:53:22.370 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:22.366+0000 7feb16f8c740 1 -- 172.21.15.32:0/960553878 wait complete. 2024-08-03T20:53:22.370 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:22.367+0000 7feb16f8c740 1 -- start start 2024-08-03T20:53:22.370 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:22.367+0000 7feb16f8c740 1 Processor -- start 2024-08-03T20:53:22.371 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:22.369+0000 7feb16f8c740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555b11cc13b0 0x555b120a79e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:53:22.372 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:22.369+0000 7feb16f8c740 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x555b11cc9590 0x555b120cb970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:53:22.372 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:22.369+0000 7feb16f8c740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x555b11cea270 0x555b120b6b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:53:22.372 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:22.369+0000 7feb16f8c740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x555b11d64620 con 0x555b11cc13b0 2024-08-03T20:53:22.372 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:22.369+0000 7feb16f8c740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x555b11f8f160 con 0x555b11cea270 2024-08-03T20:53:22.372 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:22.369+0000 7feb16f8c740 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x555b120a8840 con 0x555b11cc9590 2024-08-03T20:53:22.372 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:22.369+0000 7feb16f8c740 10 client.0 _ll_register_callbacks cb 0x555b11f9e6b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-03T20:53:22.372 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:22.369+0000 7feb0d600640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555b11cc13b0 0x555b120a79e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:53:22.372 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:22.369+0000 7feb0d600640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555b11cc13b0 0x555b120a79e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:45548/0 (socket says 172.21.15.32:45548) 2024-08-03T20:53:22.372 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:22.369+0000 7feb0d600640 1 -- 172.21.15.32:0/1393737187 learned_addr learned my addr 172.21.15.32:0/1393737187 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:53:22.372 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:22.369+0000 7feb0e000640 1 --2- 172.21.15.32:0/1393737187 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x555b11cea270 0x555b120b6b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:53:22.372 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:22.369+0000 7feb0cc00640 1 --2- 172.21.15.32:0/1393737187 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x555b11cc9590 0x555b120cb970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:53:22.372 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:22.370+0000 7feb0d600640 1 -- 172.21.15.32:0/1393737187 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x555b11cc9590 msgr2=0x555b120cb970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:53:22.373 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:22.370+0000 7feb0d600640 1 --2- 172.21.15.32:0/1393737187 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x555b11cc9590 0x555b120cb970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:53:22.373 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:22.370+0000 7feb0d600640 1 -- 172.21.15.32:0/1393737187 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x555b11cea270 msgr2=0x555b120b6b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:53:22.373 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:22.370+0000 7feb0d600640 1 --2- 172.21.15.32:0/1393737187 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x555b11cea270 0x555b120b6b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:53:22.373 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:22.370+0000 7feb0d600640 1 -- 172.21.15.32:0/1393737187 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555b11dcc6b0 con 0x555b11cc13b0 2024-08-03T20:53:22.373 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:22.370+0000 7feb0e000640 1 --2- 172.21.15.32:0/1393737187 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x555b11cea270 0x555b120b6b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-03T20:53:22.373 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:22.370+0000 7feb0d600640 1 --2- 172.21.15.32:0/1393737187 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555b11cc13b0 0x555b120a79e0 secure :-1 s=READY pgs=775 cs=0 l=1 rev1=1 crypto rx=0x7feafc01efa0 tx=0x7feafc01ec90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=84371858e964a63e server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:53:22.373 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:22.370+0000 7feaee000640 10 client.0 ms_handle_connect on v2:172.21.15.32:3300/0 2024-08-03T20:53:22.373 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:22.370+0000 7feaee000640 1 -- 172.21.15.32:0/1393737187 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feafc034020 con 0x555b11cc13b0 2024-08-03T20:53:22.373 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:22.370+0000 7feaee000640 1 -- 172.21.15.32:0/1393737187 <== mon.0 v2:172.21.15.32:3300/0 2 ==== osd_map(70..70 src has 1..70) ==== 5709+0+0 (secure 0 0 0) 0x7feafc027640 con 0x555b11cc13b0 2024-08-03T20:53:22.373 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:22.370+0000 7feb16f8c740 1 -- 172.21.15.32:0/1393737187 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7feafc01ed20 con 0x555b11cc13b0 2024-08-03T20:53:22.375 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:22.371+0000 7feaee000640 1 -- 172.21.15.32:0/1393737187 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_get_version_reply(handle=1 version=74) ==== 24+0+0 (secure 0 0 0) 0x7feafc03e050 con 0x555b11cc13b0 2024-08-03T20:53:22.375 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:22.371+0000 7feb16f8c740 10 client.6510 fetch_fsmap learned FSMap version 74 2024-08-03T20:53:22.375 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:22.371+0000 7feb16f8c740 1 -- 172.21.15.32:0/1393737187 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,fsmap=74,monmap=2+}) -- 0x555b120b6610 con 0x555b11cc13b0 2024-08-03T20:53:22.377 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:22.371+0000 7feaee000640 1 -- 172.21.15.32:0/1393737187 <== mon.0 v2:172.21.15.32:3300/0 4 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7feafc01ed20 con 0x555b11cc13b0 2024-08-03T20:53:22.377 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:22.371+0000 7feaee000640 1 -- 172.21.15.32:0/1393737187 <== mon.0 v2:172.21.15.32:3300/0 5 ==== fsmap(e 74) ==== 3502+0+0 (secure 0 0 0) 0x7feafc041020 con 0x555b11cc13b0 2024-08-03T20:53:22.377 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:22.371+0000 7feb16f8c740 10 client.6510 fetch_fsmap finished waiting for FSMap version 74 2024-08-03T20:53:22.377 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:22.371+0000 7feb16f8c740 10 client.6510 resolve_mds: resolved name 'a' to daemon mds.a 2024-08-03T20:53:22.377 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:22.371+0000 7feb16f8c740 20 client.6510 populate_metadata read hostname 'smithi032' 2024-08-03T20:53:22.377 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:22.371+0000 7feb16f8c740 1 --2- 172.21.15.32:0/1393737187 >> [v2:172.21.15.32:6833/4240363667,v1:172.21.15.32:6835/4240363667] conn(0x555b11ca8e40 0x555b120b22e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:53:22.377 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:22.371+0000 7feb16f8c740 4 client.6510 mds_command: new command op to 6399 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-08-03T20:53:22.377 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:22.371+0000 7feb16f8c740 1 -- 172.21.15.32:0/1393737187 --> [v2:172.21.15.32:6833/4240363667,v1:172.21.15.32:6835/4240363667] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x555b120049b0 con 0x555b11ca8e40 2024-08-03T20:53:22.377 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:22.371+0000 7feb0cc00640 1 --2- 172.21.15.32:0/1393737187 >> [v2:172.21.15.32:6833/4240363667,v1:172.21.15.32:6835/4240363667] conn(0x555b11ca8e40 0x555b120b22e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:53:22.378 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:22.372+0000 7feaee000640 10 client.6510 ms_handle_connect on v2:172.21.15.32:6833/4240363667 2024-08-03T20:53:22.378 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:22.372+0000 7feb0cc00640 1 --2- 172.21.15.32:0/1393737187 >> [v2:172.21.15.32:6833/4240363667,v1:172.21.15.32:6835/4240363667] conn(0x555b11ca8e40 0x555b120b22e0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c971e5be0380020b server_cookie=bf415fb9bce4ab23 in_seq=0 out_seq=0 2024-08-03T20:53:22.378 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:22.373+0000 7feaee000640 1 -- 172.21.15.32:0/1393737187 <== mds.0 v2:172.21.15.32:6833/4240363667 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x555b120049b0 con 0x555b11ca8e40 2024-08-03T20:53:22.378 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:22.373+0000 7feaee000640 10 client.6510 handle_command_reply: tid=0 2024-08-03T20:53:22.378 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:22.373+0000 7feb16f8c740 1 client.6510 shutdown 2024-08-03T20:53:22.378 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:22.373+0000 7feb16f8c740 1 -- 172.21.15.32:0/1393737187 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555b11cc13b0 msgr2=0x555b120a79e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:53:22.378 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:22.373+0000 7feb16f8c740 1 --2- 172.21.15.32:0/1393737187 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555b11cc13b0 0x555b120a79e0 secure :-1 s=READY pgs=775 cs=0 l=1 rev1=1 crypto rx=0x7feafc01efa0 tx=0x7feafc01ec90 comp rx=0 tx=0).stop 2024-08-03T20:53:22.378 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:22.373+0000 7feb16f8c740 1 -- 172.21.15.32:0/1393737187 shutdown_connections 2024-08-03T20:53:22.378 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:22.373+0000 7feb16f8c740 1 --2- 172.21.15.32:0/1393737187 >> [v2:172.21.15.32:6833/4240363667,v1:172.21.15.32:6835/4240363667] conn(0x555b11ca8e40 0x555b120b22e0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:53:22.378 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:22.373+0000 7feb16f8c740 1 --2- 172.21.15.32:0/1393737187 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x555b11cea270 0x555b120b6b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:53:22.379 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:22.373+0000 7feb16f8c740 1 --2- 172.21.15.32:0/1393737187 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x555b11cc9590 0x555b120cb970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:53:22.379 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:22.373+0000 7feaee000640 0 client.6510 ms_handle_reset on v2:172.21.15.32:6833/4240363667 2024-08-03T20:53:22.379 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:22.373+0000 7feb16f8c740 1 --2- 172.21.15.32:0/1393737187 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555b11cc13b0 0x555b120a79e0 unknown :-1 s=CLOSED pgs=775 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:53:22.379 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:22.373+0000 7feb16f8c740 1 -- 172.21.15.32:0/1393737187 >> 172.21.15.32:0/1393737187 conn(0x555b11cde3c0 msgr2=0x555b120a9560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:53:22.379 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:22.373+0000 7feb16f8c740 1 -- 172.21.15.32:0/1393737187 shutdown_connections 2024-08-03T20:53:22.379 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:22.373+0000 7feb16f8c740 1 -- 172.21.15.32:0/1393737187 wait complete. 2024-08-03T20:53:22.379 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:22.374+0000 7feb16f8c740 20 client.6510 trim_cache size 0 max 16384 2024-08-03T20:53:22.395 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:22.393+0000 7feb16f8c740 1 Processor -- start 2024-08-03T20:53:22.395 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:22.394+0000 7feb16f8c740 1 -- start start 2024-08-03T20:53:22.397 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:22.395+0000 7feb16f8c740 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x555b11cde3c0 0x555b120d8160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:53:22.397 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:22.395+0000 7feb16f8c740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555b11cc13b0 0x555b120d86a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:53:22.397 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:22.395+0000 7feb16f8c740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x555b11cc9590 0x555b120f7d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:53:22.397 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:22.395+0000 7feb16f8c740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7feb08201310 con 0x555b11cc13b0 2024-08-03T20:53:22.397 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:22.395+0000 7feb16f8c740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7feb08152020 con 0x555b11cc9590 2024-08-03T20:53:22.397 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:22.395+0000 7feb16f8c740 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7feb08200d90 con 0x555b11cde3c0 2024-08-03T20:53:22.397 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:22.395+0000 7feb0cc00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555b11cc13b0 0x555b120d86a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:53:22.398 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:22.395+0000 7feb0cc00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555b11cc13b0 0x555b120d86a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:45562/0 (socket says 172.21.15.32:45562) 2024-08-03T20:53:22.398 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:22.395+0000 7feb0cc00640 1 -- 172.21.15.32:0/1513336766 learned_addr learned my addr 172.21.15.32:0/1513336766 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:53:22.398 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:22.395+0000 7feb0e000640 1 --2- 172.21.15.32:0/1513336766 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x555b11cc9590 0x555b120f7d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:53:22.398 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:22.395+0000 7feb0d600640 1 --2- 172.21.15.32:0/1513336766 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x555b11cde3c0 0x555b120d8160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:53:22.398 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:22.396+0000 7feb0cc00640 1 -- 172.21.15.32:0/1513336766 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x555b11cde3c0 msgr2=0x555b120d8160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:53:22.398 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:22.396+0000 7feb0cc00640 1 --2- 172.21.15.32:0/1513336766 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x555b11cde3c0 0x555b120d8160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:53:22.398 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:22.396+0000 7feb0cc00640 1 -- 172.21.15.32:0/1513336766 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x555b11cc9590 msgr2=0x555b120f7d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:53:22.398 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:22.396+0000 7feb0cc00640 1 --2- 172.21.15.32:0/1513336766 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x555b11cc9590 0x555b120f7d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:53:22.398 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:22.396+0000 7feb0cc00640 1 -- 172.21.15.32:0/1513336766 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555b120a7d40 con 0x555b11cc13b0 2024-08-03T20:53:22.398 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:22.396+0000 7feb0e000640 1 --2- 172.21.15.32:0/1513336766 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x555b11cc9590 0x555b120f7d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-03T20:53:22.398 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:22.396+0000 7feb0d600640 1 --2- 172.21.15.32:0/1513336766 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x555b11cde3c0 0x555b120d8160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-03T20:53:22.399 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:22.396+0000 7feb0cc00640 1 --2- 172.21.15.32:0/1513336766 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555b11cc13b0 0x555b120d86a0 secure :-1 s=READY pgs=776 cs=0 l=1 rev1=1 crypto rx=0x7feaf4081ff0 tx=0x7feaf4005040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c151423c76653020 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:53:22.400 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:22.396+0000 7feaf9200640 1 -- 172.21.15.32:0/1513336766 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feaf40822f0 con 0x555b11cc13b0 2024-08-03T20:53:22.400 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:22.396+0000 7feaf9200640 1 -- 172.21.15.32:0/1513336766 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7feaf40b4030 con 0x555b11cc13b0 2024-08-03T20:53:22.400 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:22.396+0000 7feaf9200640 1 -- 172.21.15.32:0/1513336766 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feaf40868f0 con 0x555b11cc13b0 2024-08-03T20:53:22.400 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:22.397+0000 7feb16f8c740 1 -- 172.21.15.32:0/1513336766 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555b11cc13b0 msgr2=0x555b120d86a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:53:22.400 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:22.397+0000 7feb16f8c740 1 --2- 172.21.15.32:0/1513336766 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555b11cc13b0 0x555b120d86a0 secure :-1 s=READY pgs=776 cs=0 l=1 rev1=1 crypto rx=0x7feaf4081ff0 tx=0x7feaf4005040 comp rx=0 tx=0).stop 2024-08-03T20:53:22.400 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:22.397+0000 7feb16f8c740 1 -- 172.21.15.32:0/1513336766 shutdown_connections 2024-08-03T20:53:22.400 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:22.397+0000 7feb16f8c740 1 --2- 172.21.15.32:0/1513336766 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x555b11cc9590 0x555b120f7d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:53:22.400 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:22.397+0000 7feb16f8c740 1 --2- 172.21.15.32:0/1513336766 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555b11cc13b0 0x555b120d86a0 unknown :-1 s=CLOSED pgs=776 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:53:22.400 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:22.397+0000 7feb16f8c740 1 --2- 172.21.15.32:0/1513336766 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x555b11cde3c0 0x555b120d8160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:53:22.400 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:22.397+0000 7feb16f8c740 1 -- 172.21.15.32:0/1513336766 >> 172.21.15.32:0/1513336766 conn(0x555b12090600 msgr2=0x555b120b4a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:53:22.400 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:22.397+0000 7feb16f8c740 1 -- 172.21.15.32:0/1513336766 shutdown_connections 2024-08-03T20:53:22.401 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:22.397+0000 7feb16f8c740 1 -- 172.21.15.32:0/1513336766 wait complete. 2024-08-03T20:53:22.401 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:22.398+0000 7feb16f8c740 1 -- start start 2024-08-03T20:53:22.401 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:22.398+0000 7feb16f8c740 1 Processor -- start 2024-08-03T20:53:22.403 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:22.400+0000 7feb16f8c740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x555b11cc13b0 0x555b1208df60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:53:22.403 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:22.400+0000 7feb16f8c740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555b11cc9590 0x555b120b1270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:53:22.403 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:22.400+0000 7feb16f8c740 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x555b11cde3c0 0x555b120b17b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:53:22.404 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:22.400+0000 7feb16f8c740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x555b120b3f90 con 0x555b11cc9590 2024-08-03T20:53:22.404 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:22.400+0000 7feb16f8c740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x555b120cc170 con 0x555b11cc13b0 2024-08-03T20:53:22.404 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:22.400+0000 7feb16f8c740 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x555b120ce000 con 0x555b11cde3c0 2024-08-03T20:53:22.404 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:22.400+0000 7feb16f8c740 10 client.0 _ll_register_callbacks cb 0x7feafc01eed0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-03T20:53:22.404 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:22.400+0000 7feb0cc00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555b11cc9590 0x555b120b1270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:53:22.404 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:22.400+0000 7feb0cc00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555b11cc9590 0x555b120b1270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:45568/0 (socket says 172.21.15.32:45568) 2024-08-03T20:53:22.404 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:22.400+0000 7feb0cc00640 1 -- 172.21.15.32:0/2377192209 learned_addr learned my addr 172.21.15.32:0/2377192209 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:53:22.404 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:22.400+0000 7feb0d600640 1 --2- 172.21.15.32:0/2377192209 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x555b11cc13b0 0x555b1208df60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:53:22.404 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:22.400+0000 7feb0e000640 1 --2- 172.21.15.32:0/2377192209 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x555b11cde3c0 0x555b120b17b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:53:22.404 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:22.400+0000 7feb0cc00640 1 -- 172.21.15.32:0/2377192209 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x555b11cde3c0 msgr2=0x555b120b17b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:53:22.404 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:22.400+0000 7feb0cc00640 1 --2- 172.21.15.32:0/2377192209 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x555b11cde3c0 0x555b120b17b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:53:22.404 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:22.400+0000 7feb0cc00640 1 -- 172.21.15.32:0/2377192209 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x555b11cc13b0 msgr2=0x555b1208df60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:53:22.404 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:22.400+0000 7feb0cc00640 1 --2- 172.21.15.32:0/2377192209 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x555b11cc13b0 0x555b1208df60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:53:22.405 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:22.400+0000 7feb0cc00640 1 -- 172.21.15.32:0/2377192209 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555b12156200 con 0x555b11cc9590 2024-08-03T20:53:22.405 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:22.401+0000 7feb0cc00640 1 --2- 172.21.15.32:0/2377192209 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555b11cc9590 0x555b120b1270 secure :-1 s=READY pgs=777 cs=0 l=1 rev1=1 crypto rx=0x7feaf4082280 tx=0x7feaf408b310 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8dc13d7d5c555bba server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:53:22.405 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:22.401+0000 7feaee000640 10 client.0 ms_handle_connect on v2:172.21.15.32:3300/0 2024-08-03T20:53:22.405 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:22.401+0000 7feaee000640 1 -- 172.21.15.32:0/2377192209 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feaf40868f0 con 0x555b11cc9590 2024-08-03T20:53:22.405 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:22.401+0000 7feaee000640 1 -- 172.21.15.32:0/2377192209 <== mon.0 v2:172.21.15.32:3300/0 2 ==== osd_map(70..70 src has 1..70) ==== 5709+0+0 (secure 0 0 0) 0x7feaf4082550 con 0x555b11cc9590 2024-08-03T20:53:22.405 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:22.401+0000 7feb16f8c740 1 -- 172.21.15.32:0/2377192209 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7feaf40b4030 con 0x555b11cc9590 2024-08-03T20:53:22.405 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:22.401+0000 7feaee000640 1 -- 172.21.15.32:0/2377192209 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_get_version_reply(handle=1 version=74) ==== 24+0+0 (secure 0 0 0) 0x7feaf40827b0 con 0x555b11cc9590 2024-08-03T20:53:22.405 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:22.401+0000 7feb16f8c740 10 client.6516 fetch_fsmap learned FSMap version 74 2024-08-03T20:53:22.405 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:22.401+0000 7feb16f8c740 1 -- 172.21.15.32:0/2377192209 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,fsmap=74,monmap=2+}) -- 0x555b120b0b60 con 0x555b11cc9590 2024-08-03T20:53:22.405 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:22.401+0000 7feaee000640 1 -- 172.21.15.32:0/2377192209 <== mon.0 v2:172.21.15.32:3300/0 4 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7feaf40b4030 con 0x555b11cc9590 2024-08-03T20:53:22.405 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:22.401+0000 7feaee000640 1 -- 172.21.15.32:0/2377192209 <== mon.0 v2:172.21.15.32:3300/0 5 ==== fsmap(e 74) ==== 3502+0+0 (secure 0 0 0) 0x7feaf404a070 con 0x555b11cc9590 2024-08-03T20:53:22.405 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:22.401+0000 7feb16f8c740 10 client.6516 fetch_fsmap finished waiting for FSMap version 74 2024-08-03T20:53:22.405 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:22.401+0000 7feb16f8c740 10 client.6516 resolve_mds: resolved name 'a' to daemon mds.a 2024-08-03T20:53:22.406 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:22.402+0000 7feb16f8c740 20 client.6516 populate_metadata read hostname 'smithi032' 2024-08-03T20:53:22.406 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:22.402+0000 7feb16f8c740 1 --2- 172.21.15.32:0/2377192209 >> [v2:172.21.15.32:6833/4240363667,v1:172.21.15.32:6835/4240363667] conn(0x555b11c7e280 0x555b12152c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:53:22.406 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:22.402+0000 7feb16f8c740 4 client.6516 mds_command: new command op to 6399 tid=0 multi_id=0 [{"prefix": "status"}] 2024-08-03T20:53:22.406 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:22.402+0000 7feb16f8c740 1 -- 172.21.15.32:0/2377192209 --> [v2:172.21.15.32:6833/4240363667,v1:172.21.15.32:6835/4240363667] -- command(tid 0: {"prefix": "status"}) -- 0x555b12000450 con 0x555b11c7e280 2024-08-03T20:53:22.406 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:22.402+0000 7feb0d600640 1 --2- 172.21.15.32:0/2377192209 >> [v2:172.21.15.32:6833/4240363667,v1:172.21.15.32:6835/4240363667] conn(0x555b11c7e280 0x555b12152c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:53:22.407 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:22.402+0000 7feb0d600640 1 --2- 172.21.15.32:0/2377192209 >> [v2:172.21.15.32:6833/4240363667,v1:172.21.15.32:6835/4240363667] conn(0x555b11c7e280 0x555b12152c80 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f4e481be3875c9e7 server_cookie=8c134993dc3c5c02 in_seq=0 out_seq=0 2024-08-03T20:53:22.407 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:22.402+0000 7feaee000640 10 client.6516 ms_handle_connect on v2:172.21.15.32:6833/4240363667 2024-08-03T20:53:22.407 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:22.403+0000 7feaee000640 1 -- 172.21.15.32:0/2377192209 <== mds.0 v2:172.21.15.32:6833/4240363667 1 ==== command_reply(tid 0: 0 ) ==== 8+0+331 (crc 0 0 0) 0x555b12000450 con 0x555b11c7e280 2024-08-03T20:53:22.408 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:22.403+0000 7feaee000640 10 client.6516 handle_command_reply: tid=0 2024-08-03T20:53:22.408 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:22.403+0000 7feb16f8c740 1 client.6516 shutdown 2024-08-03T20:53:22.408 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:22.403+0000 7feb16f8c740 1 -- 172.21.15.32:0/2377192209 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555b11cc9590 msgr2=0x555b120b1270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:53:22.408 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:22.403+0000 7feb16f8c740 1 --2- 172.21.15.32:0/2377192209 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555b11cc9590 0x555b120b1270 secure :-1 s=READY pgs=777 cs=0 l=1 rev1=1 crypto rx=0x7feaf4082280 tx=0x7feaf408b310 comp rx=0 tx=0).stop 2024-08-03T20:53:22.408 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:22.403+0000 7feb16f8c740 1 -- 172.21.15.32:0/2377192209 shutdown_connections 2024-08-03T20:53:22.408 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:22.403+0000 7feb16f8c740 1 --2- 172.21.15.32:0/2377192209 >> [v2:172.21.15.32:6833/4240363667,v1:172.21.15.32:6835/4240363667] conn(0x555b11c7e280 0x555b12152c80 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:53:22.408 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:22.403+0000 7feb16f8c740 1 --2- 172.21.15.32:0/2377192209 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x555b11cde3c0 0x555b120b17b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:53:22.408 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:22.403+0000 7feb16f8c740 1 --2- 172.21.15.32:0/2377192209 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555b11cc9590 0x555b120b1270 unknown :-1 s=CLOSED pgs=777 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:53:22.408 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:22.403+0000 7feaee000640 0 client.6516 ms_handle_reset on v2:172.21.15.32:6833/4240363667 2024-08-03T20:53:22.408 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:22.403+0000 7feb16f8c740 1 --2- 172.21.15.32:0/2377192209 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x555b11cc13b0 0x555b1208df60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:53:22.408 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:22.403+0000 7feb16f8c740 1 -- 172.21.15.32:0/2377192209 >> 172.21.15.32:0/2377192209 conn(0x555b12090600 msgr2=0x555b120d8160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:53:22.408 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:22.403+0000 7feb16f8c740 1 -- 172.21.15.32:0/2377192209 shutdown_connections 2024-08-03T20:53:22.408 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:22.403+0000 7feb16f8c740 1 -- 172.21.15.32:0/2377192209 wait complete. 2024-08-03T20:53:22.408 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:22.403+0000 7feb16f8c740 20 client.6516 trim_cache size 0 max 16384 2024-08-03T20:53:22.409 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-08-03T20:53:22.409 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_fsid": "40ab141a-01d7-49b9-ae3c-2edec9876e95", 2024-08-03T20:53:22.409 INFO:teuthology.orchestra.run.smithi032.stdout: "whoami": 0, 2024-08-03T20:53:22.409 INFO:teuthology.orchestra.run.smithi032.stdout: "id": 6399, 2024-08-03T20:53:22.409 INFO:teuthology.orchestra.run.smithi032.stdout: "want_state": "up:active", 2024-08-03T20:53:22.409 INFO:teuthology.orchestra.run.smithi032.stdout: "state": "up:active", 2024-08-03T20:53:22.409 INFO:teuthology.orchestra.run.smithi032.stdout: "fs_name": "cephfs", 2024-08-03T20:53:22.409 INFO:teuthology.orchestra.run.smithi032.stdout: "rank_uptime": 6.8548795440000001, 2024-08-03T20:53:22.409 INFO:teuthology.orchestra.run.smithi032.stdout: "mdsmap_epoch": 74, 2024-08-03T20:53:22.409 INFO:teuthology.orchestra.run.smithi032.stdout: "osdmap_epoch": 70, 2024-08-03T20:53:22.410 INFO:teuthology.orchestra.run.smithi032.stdout: "osdmap_epoch_barrier": 70, 2024-08-03T20:53:22.410 INFO:teuthology.orchestra.run.smithi032.stdout: "uptime": 11.788390849000001 2024-08-03T20:53:22.410 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-08-03T20:53:22.410 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:22.404+0000 7feb10000640 1 -- 172.21.15.32:0/4189583763 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7feae8077010 msgr2=0x7feae8079520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:53:22.411 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:22.404+0000 7feb10000640 1 --2- 172.21.15.32:0/4189583763 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7feae8077010 0x7feae8079520 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7feafc009b60 tx=0x7feafc005f90 comp rx=0 tx=0).stop 2024-08-03T20:53:22.411 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:22.404+0000 7feb10000640 1 -- 172.21.15.32:0/4189583763 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7feb08143710 msgr2=0x7feb081f8b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:53:22.411 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:22.404+0000 7feb10000640 1 --2- 172.21.15.32:0/4189583763 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7feb08143710 0x7feb081f8b40 secure :-1 s=READY pgs=773 cs=0 l=1 rev1=1 crypto rx=0x7feaf400c350 tx=0x7feaf4006420 comp rx=0 tx=0).stop 2024-08-03T20:53:22.411 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:22.404+0000 7feb10000640 1 -- 172.21.15.32:0/4189583763 shutdown_connections 2024-08-03T20:53:22.411 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:22.404+0000 7feb10000640 1 --2- 172.21.15.32:0/4189583763 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7feae8077010 0x7feae8079520 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:53:22.411 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:22.404+0000 7feb10000640 1 --2- 172.21.15.32:0/4189583763 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7feb081f9080 0x7feb08139980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:53:22.411 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:22.404+0000 7feb10000640 1 --2- 172.21.15.32:0/4189583763 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7feb08143710 0x7feb081f8b40 unknown :-1 s=CLOSED pgs=773 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:53:22.411 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:22.404+0000 7feb10000640 1 --2- 172.21.15.32:0/4189583763 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7feaf0004800 0x7feb081431d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:53:22.411 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:22.404+0000 7feb10000640 1 -- 172.21.15.32:0/4189583763 >> 172.21.15.32:0/4189583763 conn(0x7feb080b1bd0 msgr2=0x7feb081406f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:53:22.411 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:22.404+0000 7feb10000640 1 -- 172.21.15.32:0/4189583763 shutdown_connections 2024-08-03T20:53:22.411 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:22.404+0000 7feb10000640 1 -- 172.21.15.32:0/4189583763 wait complete. 2024-08-03T20:53:22.427 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-08-03T20:53:22.656 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:22.654+0000 7fb4ace00640 1 -- 172.21.15.32:0/399777862 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb4a806e930 msgr2=0x7fb4a8144f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:53:22.656 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:22.654+0000 7fb4ace00640 1 --2- 172.21.15.32:0/399777862 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb4a806e930 0x7fb4a8144f90 secure :-1 s=READY pgs=778 cs=0 l=1 rev1=1 crypto rx=0x7fb498005fd0 tx=0x7fb49802a540 comp rx=0 tx=0).stop 2024-08-03T20:53:22.656 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:22.654+0000 7fb4ace00640 1 -- 172.21.15.32:0/399777862 shutdown_connections 2024-08-03T20:53:22.656 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:22.654+0000 7fb4ace00640 1 --2- 172.21.15.32:0/399777862 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fb4a8073e60 0x7fb4a8074270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:53:22.656 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:22.654+0000 7fb4ace00640 1 --2- 172.21.15.32:0/399777862 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb4a806e930 0x7fb4a8144f90 unknown :-1 s=CLOSED pgs=778 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:53:22.656 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:22.654+0000 7fb4ace00640 1 -- 172.21.15.32:0/399777862 >> 172.21.15.32:0/399777862 conn(0x7fb4a80b1bd0 msgr2=0x7fb4a80b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:53:22.656 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:22.654+0000 7fb4ace00640 1 -- 172.21.15.32:0/399777862 shutdown_connections 2024-08-03T20:53:22.656 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:22.654+0000 7fb4ace00640 1 -- 172.21.15.32:0/399777862 wait complete. 2024-08-03T20:53:22.657 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:22.656+0000 7fb4ace00640 1 Processor -- start 2024-08-03T20:53:22.657 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:22.656+0000 7fb4ace00640 1 -- start start 2024-08-03T20:53:22.661 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:22.658+0000 7fb4ace00640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fb4a806e930 0x7fb4a81429a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:53:22.661 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:22.658+0000 7fb4ace00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb4a8073e60 0x7fb4a815b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:53:22.661 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:22.658+0000 7fb4ace00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb4a80747b0 0x7fb4a815eef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:53:22.661 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:22.658+0000 7fb4ace00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fb4a80cf9c0 con 0x7fb4a8073e60 2024-08-03T20:53:22.661 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:22.658+0000 7fb4ace00640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7fb4a80cf840 con 0x7fb4a80747b0 2024-08-03T20:53:22.661 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:22.658+0000 7fb4ace00640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7fb4a80ccff0 con 0x7fb4a806e930 2024-08-03T20:53:22.661 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:22.658+0000 7fb49fe00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb4a8073e60 0x7fb4a815b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:53:22.661 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:22.658+0000 7fb49fe00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb4a8073e60 0x7fb4a815b9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:45586/0 (socket says 172.21.15.32:45586) 2024-08-03T20:53:22.661 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:22.658+0000 7fb49fe00640 1 -- 172.21.15.32:0/3102952644 learned_addr learned my addr 172.21.15.32:0/3102952644 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:53:22.661 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:22.658+0000 7fb4a5e00640 1 --2- 172.21.15.32:0/3102952644 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fb4a806e930 0x7fb4a81429a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:53:22.661 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:22.658+0000 7fb4a6800640 1 --2- 172.21.15.32:0/3102952644 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb4a80747b0 0x7fb4a815eef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:53:22.662 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:22.658+0000 7fb49fe00640 1 -- 172.21.15.32:0/3102952644 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fb4a806e930 msgr2=0x7fb4a81429a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:53:22.662 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:22.658+0000 7fb49fe00640 1 --2- 172.21.15.32:0/3102952644 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fb4a806e930 0x7fb4a81429a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:53:22.662 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:22.658+0000 7fb49fe00640 1 -- 172.21.15.32:0/3102952644 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb4a80747b0 msgr2=0x7fb4a815eef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:53:22.662 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:22.658+0000 7fb49fe00640 1 --2- 172.21.15.32:0/3102952644 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb4a80747b0 0x7fb4a815eef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:53:22.662 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:22.658+0000 7fb49fe00640 1 -- 172.21.15.32:0/3102952644 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb4a81431f0 con 0x7fb4a8073e60 2024-08-03T20:53:22.662 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:22.658+0000 7fb4a6800640 1 --2- 172.21.15.32:0/3102952644 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb4a80747b0 0x7fb4a815eef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-03T20:53:22.662 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:22.659+0000 7fb49fe00640 1 --2- 172.21.15.32:0/3102952644 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb4a8073e60 0x7fb4a815b9d0 secure :-1 s=READY pgs=779 cs=0 l=1 rev1=1 crypto rx=0x7fb49000e7c0 tx=0x7fb49000ec80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:53:22.662 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:22.659+0000 7fb49e000640 1 -- 172.21.15.32:0/3102952644 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb49000ccc0 con 0x7fb4a8073e60 2024-08-03T20:53:22.662 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:22.659+0000 7fb49e000640 1 -- 172.21.15.32:0/3102952644 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fb49000ce80 con 0x7fb4a8073e60 2024-08-03T20:53:22.662 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:22.659+0000 7fb49e000640 1 -- 172.21.15.32:0/3102952644 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb490005280 con 0x7fb4a8073e60 2024-08-03T20:53:22.662 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:22.659+0000 7fb4ace00640 1 -- 172.21.15.32:0/3102952644 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb4a81434d0 con 0x7fb4a8073e60 2024-08-03T20:53:22.662 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:22.659+0000 7fb4ace00640 1 -- 172.21.15.32:0/3102952644 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb4a81f6590 con 0x7fb4a8073e60 2024-08-03T20:53:22.663 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:22.660+0000 7fb4ace00640 1 -- 172.21.15.32:0/3102952644 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb4a8142f60 con 0x7fb4a8073e60 2024-08-03T20:53:22.663 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:22.662+0000 7fb49e000640 1 -- 172.21.15.32:0/3102952644 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7fb490005420 con 0x7fb4a8073e60 2024-08-03T20:53:22.664 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:22.662+0000 7fb49e000640 1 --2- 172.21.15.32:0/3102952644 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fb484077040 0x7fb484079550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:53:22.664 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:22.663+0000 7fb49e000640 1 -- 172.21.15.32:0/3102952644 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(70..70 src has 1..70) ==== 5709+0+0 (secure 0 0 0) 0x7fb4900991b0 con 0x7fb4a8073e60 2024-08-03T20:53:22.664 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:22.663+0000 7fb4a5e00640 1 --2- 172.21.15.32:0/3102952644 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fb484077040 0x7fb484079550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:53:22.665 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:22.664+0000 7fb4a5e00640 1 --2- 172.21.15.32:0/3102952644 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fb484077040 0x7fb484079550 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7fb498033000 tx=0x7fb498004730 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:53:22.668 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:22.667+0000 7fb49e000640 1 -- 172.21.15.32:0/3102952644 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7fb4900623a0 con 0x7fb4a8073e60 2024-08-03T20:53:22.917 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:22.915+0000 7fb4ace00640 1 -- 172.21.15.32:0/3102952644 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7fb4a81f63b0 con 0x7fb4a8073e60 2024-08-03T20:53:22.917 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:22.916+0000 7fb49e000640 1 -- 172.21.15.32:0/3102952644 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v74) ==== 53+0+73 (secure 0 0 0) 0x7fb4a81f63b0 con 0x7fb4a8073e60 2024-08-03T20:53:22.918 INFO:teuthology.orchestra.run.smithi032.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-08-03T20:53:22.920 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:22.918+0000 7fb4ace00640 1 -- 172.21.15.32:0/3102952644 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fb484077040 msgr2=0x7fb484079550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:53:22.920 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:22.918+0000 7fb4ace00640 1 --2- 172.21.15.32:0/3102952644 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fb484077040 0x7fb484079550 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7fb498033000 tx=0x7fb498004730 comp rx=0 tx=0).stop 2024-08-03T20:53:22.921 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:22.918+0000 7fb4ace00640 1 -- 172.21.15.32:0/3102952644 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb4a8073e60 msgr2=0x7fb4a815b9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:53:22.921 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:22.918+0000 7fb4ace00640 1 --2- 172.21.15.32:0/3102952644 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb4a8073e60 0x7fb4a815b9d0 secure :-1 s=READY pgs=779 cs=0 l=1 rev1=1 crypto rx=0x7fb49000e7c0 tx=0x7fb49000ec80 comp rx=0 tx=0).stop 2024-08-03T20:53:22.921 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:22.918+0000 7fb4ace00640 1 -- 172.21.15.32:0/3102952644 shutdown_connections 2024-08-03T20:53:22.921 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:22.918+0000 7fb4ace00640 1 --2- 172.21.15.32:0/3102952644 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fb484077040 0x7fb484079550 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:53:22.921 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:22.918+0000 7fb4ace00640 1 --2- 172.21.15.32:0/3102952644 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb4a80747b0 0x7fb4a815eef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:53:22.921 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:22.918+0000 7fb4ace00640 1 --2- 172.21.15.32:0/3102952644 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb4a8073e60 0x7fb4a815b9d0 unknown :-1 s=CLOSED pgs=779 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:53:22.921 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:22.918+0000 7fb4ace00640 1 --2- 172.21.15.32:0/3102952644 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fb4a806e930 0x7fb4a81429a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:53:22.921 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:22.918+0000 7fb4ace00640 1 -- 172.21.15.32:0/3102952644 >> 172.21.15.32:0/3102952644 conn(0x7fb4a80b1bd0 msgr2=0x7fb4a80b14a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:53:22.921 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:22.919+0000 7fb4ace00640 1 -- 172.21.15.32:0/3102952644 shutdown_connections 2024-08-03T20:53:22.921 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:22.919+0000 7fb4ace00640 1 -- 172.21.15.32:0/3102952644 wait complete. 2024-08-03T20:53:22.943 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-08-03T20:53:22.943 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-08-03T20:53:22.943 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi032.front.sepia.ceph.com 2024-08-03T20:53:22.943 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-08-03T20:53:22.943 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-08-03T20:53:22.943 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-08-03T20:53:22.943 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2024-08-03T20:53:22.944 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-08-03T20:53:22.944 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-08-03T20:53:22.944 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-08-03T20:53:22.944 DEBUG:teuthology.orchestra.run.smithi032:> ip addr 2024-08-03T20:53:22.980 INFO:teuthology.orchestra.run.smithi032.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-08-03T20:53:22.980 INFO:teuthology.orchestra.run.smithi032.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-08-03T20:53:22.980 INFO:teuthology.orchestra.run.smithi032.stdout: inet 127.0.0.1/8 scope host lo 2024-08-03T20:53:22.980 INFO:teuthology.orchestra.run.smithi032.stdout: valid_lft forever preferred_lft forever 2024-08-03T20:53:22.980 INFO:teuthology.orchestra.run.smithi032.stdout: inet6 ::1/128 scope host 2024-08-03T20:53:22.980 INFO:teuthology.orchestra.run.smithi032.stdout: valid_lft forever preferred_lft forever 2024-08-03T20:53:22.980 INFO:teuthology.orchestra.run.smithi032.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-08-03T20:53:22.981 INFO:teuthology.orchestra.run.smithi032.stdout: link/ether 0c:c4:7a:6b:fd:0c brd ff:ff:ff:ff:ff:ff 2024-08-03T20:53:22.981 INFO:teuthology.orchestra.run.smithi032.stdout: altname enp7s0f0 2024-08-03T20:53:22.981 INFO:teuthology.orchestra.run.smithi032.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-08-03T20:53:22.981 INFO:teuthology.orchestra.run.smithi032.stdout: link/ether 0c:c4:7a:6b:fd:0d brd ff:ff:ff:ff:ff:ff 2024-08-03T20:53:22.981 INFO:teuthology.orchestra.run.smithi032.stdout: altname enp7s0f1 2024-08-03T20:53:22.981 INFO:teuthology.orchestra.run.smithi032.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-08-03T20:53:22.981 INFO:teuthology.orchestra.run.smithi032.stdout: link/ether ac:1f:6b:f8:26:02 brd ff:ff:ff:ff:ff:ff 2024-08-03T20:53:22.981 INFO:teuthology.orchestra.run.smithi032.stdout: altname enp1s0f0 2024-08-03T20:53:22.981 INFO:teuthology.orchestra.run.smithi032.stdout: inet 172.21.15.32/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-08-03T20:53:22.981 INFO:teuthology.orchestra.run.smithi032.stdout: valid_lft 42153sec preferred_lft 42153sec 2024-08-03T20:53:22.981 INFO:teuthology.orchestra.run.smithi032.stdout: inet6 fe80::ae1f:6bff:fef8:2602/64 scope link noprefixroute 2024-08-03T20:53:22.981 INFO:teuthology.orchestra.run.smithi032.stdout: valid_lft forever preferred_lft forever 2024-08-03T20:53:22.981 INFO:teuthology.orchestra.run.smithi032.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-08-03T20:53:22.982 INFO:teuthology.orchestra.run.smithi032.stdout: link/ether ac:1f:6b:f8:26:03 brd ff:ff:ff:ff:ff:ff 2024-08-03T20:53:22.982 INFO:teuthology.orchestra.run.smithi032.stdout: altname enp1s0f1 2024-08-03T20:53:22.982 INFO:teuthology.orchestra.run.smithi032.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-08-03T20:53:22.982 INFO:teuthology.orchestra.run.smithi032.stdout: link/ether 0a:23:2b:ca:2a:93 brd ff:ff:ff:ff:ff:ff 2024-08-03T20:53:22.982 INFO:teuthology.orchestra.run.smithi032.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-08-03T20:53:22.982 INFO:teuthology.orchestra.run.smithi032.stdout: valid_lft forever preferred_lft forever 2024-08-03T20:53:22.982 INFO:teuthology.orchestra.run.smithi032.stdout: inet6 fe80::d804:7cff:fed9:7ba4/64 scope link 2024-08-03T20:53:22.982 INFO:teuthology.orchestra.run.smithi032.stdout: valid_lft forever preferred_lft forever 2024-08-03T20:53:22.982 INFO:teuthology.orchestra.run.smithi032.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-08-03T20:53:22.982 INFO:teuthology.orchestra.run.smithi032.stdout: link/ether 22:86:28:30:4e:62 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-08-03T20:53:22.982 INFO:teuthology.orchestra.run.smithi032.stdout: inet6 fe80::2086:28ff:fe30:4e62/64 scope link 2024-08-03T20:53:22.982 INFO:teuthology.orchestra.run.smithi032.stdout: valid_lft forever preferred_lft forever 2024-08-03T20:53:22.982 INFO:teuthology.orchestra.run.smithi032.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-08-03T20:53:22.983 INFO:teuthology.orchestra.run.smithi032.stdout: link/ether 0a:23:2b:ca:2a:93 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-08-03T20:53:22.983 INFO:teuthology.orchestra.run.smithi032.stdout: inet6 fe80::823:2bff:feca:2a93/64 scope link 2024-08-03T20:53:22.983 INFO:teuthology.orchestra.run.smithi032.stdout: valid_lft forever preferred_lft forever 2024-08-03T20:53:22.983 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-08-03T20:53:22.983 DEBUG:teuthology.orchestra.run.smithi032:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-08-03T20:53:23.074 INFO:teuthology.orchestra.run.smithi032.stdout:1 2024-08-03T20:53:23.077 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-08-03T20:53:23.077 DEBUG:teuthology.orchestra.run.smithi032:> route 2024-08-03T20:53:23.113 INFO:teuthology.orchestra.run.smithi032.stdout:Kernel IP routing table 2024-08-03T20:53:23.113 INFO:teuthology.orchestra.run.smithi032.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-08-03T20:53:23.113 INFO:teuthology.orchestra.run.smithi032.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-08-03T20:53:23.114 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-08-03T20:53:23.114 INFO:teuthology.orchestra.run.smithi032.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-08-03T20:53:23.114 DEBUG:teuthology.orchestra.run.smithi032:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-08-03T20:53:23.114 DEBUG:teuthology.orchestra.run.smithi032:> set -e 2024-08-03T20:53:23.114 DEBUG:teuthology.orchestra.run.smithi032:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-08-03T20:53:23.114 DEBUG:teuthology.orchestra.run.smithi032:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-08-03T20:53:23.114 DEBUG:teuthology.orchestra.run.smithi032:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-08-03T20:53:23.114 DEBUG:teuthology.orchestra.run.smithi032:> ') 2024-08-03T20:53:23.232 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:23 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-08-03T20:53:23.303 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:23 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-08-03T20:53:23.311 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-08-03T20:53:23.311 DEBUG:teuthology.orchestra.run.smithi032:> ip netns list 2024-08-03T20:53:23.347 INFO:teuthology.orchestra.run.smithi032.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-08-03T20:53:23.347 INFO:teuthology.orchestra.run.smithi032.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-08-03T20:53:23.347 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-08-03T20:53:23.347 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-08-03T20:53:23.348 DEBUG:teuthology.orchestra.run.smithi032:> sudo ip link set brx.0 up 2024-08-03T20:53:23.443 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-08-03T20:53:23.443 DEBUG:teuthology.orchestra.run.smithi032:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-08-03T20:53:23.480 INFO:teuthology.orchestra.run.smithi032.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-08-03T20:53:23.480 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-08-03T20:53:23.480 DEBUG:teuthology.orchestra.run.smithi032:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-08-03T20:53:23.555 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2024-08-03T20:53:23.555 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-08-03T20:53:23.555 DEBUG:teuthology.orchestra.run.smithi032:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-08-03T20:53:23.752 INFO:teuthology.orchestra.run.smithi032.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-08-03T20:53:23.752 INFO:teuthology.orchestra.run.smithi032.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-08-03T20:53:23.752 INFO:teuthology.orchestra.run.smithi032.stdout:invalid new device string format 2024-08-03T20:53:23.772 INFO:teuthology.orchestra.run.smithi032.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-08-03T20:53:23.772 INFO:teuthology.orchestra.run.smithi032.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-08-03T20:53:23.772 INFO:teuthology.orchestra.run.smithi032.stdout:invalid new device string format 2024-08-03T20:53:23.772 INFO:teuthology.orchestra.run.smithi032.stdout:mount.ceph: resolved to: "172.21.15.32:3300,172.21.15.118:3300,172.21.15.118:3301" 2024-08-03T20:53:23.773 INFO:teuthology.orchestra.run.smithi032.stdout:mount.ceph: switching to using v1 address with old syntax 2024-08-03T20:53:23.805 INFO:teuthology.orchestra.run.smithi032.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-08-03T20:53:23.805 INFO:teuthology.orchestra.run.smithi032.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-08-03T20:53:23.805 INFO:teuthology.orchestra.run.smithi032.stdout:invalid new device string format 2024-08-03T20:53:23.805 INFO:teuthology.orchestra.run.smithi032.stdout:mount.ceph: resolved to: "172.21.15.32:3300,172.21.15.118:3300,172.21.15.118:3301" 2024-08-03T20:53:23.805 INFO:teuthology.orchestra.run.smithi032.stdout:mount.ceph: switching to using v1 address with old syntax 2024-08-03T20:53:23.805 INFO:teuthology.orchestra.run.smithi032.stdout:mount.ceph: resolved to: "172.21.15.32:6789,172.21.15.118:6789,172.21.15.118:6790" 2024-08-03T20:53:23.805 INFO:teuthology.orchestra.run.smithi032.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.32:6789,172.21.15.118:6789,172.21.15.118:6790:/ 2024-08-03T20:53:23.805 INFO:teuthology.orchestra.run.smithi032.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,key=0,fsid=40ab141a-01d7-49b9-ae3c-2edec9876e95" will pass to kernel 2024-08-03T20:53:23.807 INFO:teuthology.orchestra.run.smithi032.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2024-08-03T20:53:23.807 INFO:teuthology.orchestra.run.smithi032.stdout: You just mounted a file system that supports labels which does not 2024-08-03T20:53:23.807 INFO:teuthology.orchestra.run.smithi032.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-08-03T20:53:23.807 INFO:teuthology.orchestra.run.smithi032.stdout: applications will generate AVC messages and not be allowed access to 2024-08-03T20:53:23.807 INFO:teuthology.orchestra.run.smithi032.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-08-03T20:53:23.810 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-08-03T20:53:23.810 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-08-03T20:53:23.810 DEBUG:teuthology.orchestra.run.smithi032:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-08-03T20:53:23.867 DEBUG:teuthology.orchestra.run.smithi032:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-08-03T20:53:23.948 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:23 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-08-03T20:53:23.957 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:23 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-08-03T20:53:23.965 INFO:teuthology.orchestra.run.smithi032.stdout:client5762 2024-08-03T20:53:23.965 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-08-03T20:53:23.965 DEBUG:teuthology.orchestra.run.smithi032:> dd if=/proc/self/mounts of=/dev/stdout 2024-08-03T20:53:24.000 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-08-03T20:53:24.268 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:24.266+0000 7fc47a600640 1 -- 172.21.15.32:0/2776652161 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc4740acd80 msgr2=0x7fc4740ad190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:53:24.268 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:24.266+0000 7fc47a600640 1 --2- 172.21.15.32:0/2776652161 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc4740acd80 0x7fc4740ad190 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7fc45c005fd0 tx=0x7fc45c02a540 comp rx=0 tx=0).stop 2024-08-03T20:53:24.268 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:24.266+0000 7fc47a600640 1 -- 172.21.15.32:0/2776652161 shutdown_connections 2024-08-03T20:53:24.268 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:24.266+0000 7fc47a600640 1 --2- 172.21.15.32:0/2776652161 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc4740acd80 0x7fc4740ad190 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:53:24.268 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:24.266+0000 7fc47a600640 1 --2- 172.21.15.32:0/2776652161 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fc4740af020 0x7fc4740742b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:53:24.268 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:24.266+0000 7fc47a600640 1 -- 172.21.15.32:0/2776652161 >> 172.21.15.32:0/2776652161 conn(0x7fc4740b2120 msgr2=0x7fc4740cfd60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:53:24.269 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:24.266+0000 7fc47a600640 1 -- 172.21.15.32:0/2776652161 shutdown_connections 2024-08-03T20:53:24.269 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:24.266+0000 7fc47a600640 1 -- 172.21.15.32:0/2776652161 wait complete. 2024-08-03T20:53:24.270 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:24.268+0000 7fc47a600640 1 Processor -- start 2024-08-03T20:53:24.270 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:24.269+0000 7fc47a600640 1 -- start start 2024-08-03T20:53:24.272 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:24.270+0000 7fc47a600640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc47406f440 0x7fc4741f9620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:53:24.273 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:24.270+0000 7fc47a600640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc4740acd80 0x7fc47415b500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:53:24.273 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:24.270+0000 7fc47a600640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fc4740af020 0x7fc4741f8060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:53:24.273 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:24.270+0000 7fc47a600640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fc474160260 con 0x7fc47406f440 2024-08-03T20:53:24.273 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:24.270+0000 7fc47a600640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7fc4741600e0 con 0x7fc4740acd80 2024-08-03T20:53:24.273 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:24.270+0000 7fc47a600640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7fc4741603e0 con 0x7fc4740af020 2024-08-03T20:53:24.273 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:24.271+0000 7fc479200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc47406f440 0x7fc4741f9620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:53:24.273 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:24.271+0000 7fc479200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc47406f440 0x7fc4741f9620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:45596/0 (socket says 172.21.15.32:45596) 2024-08-03T20:53:24.273 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:24.271+0000 7fc479200640 1 -- 172.21.15.32:0/911138078 learned_addr learned my addr 172.21.15.32:0/911138078 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:53:24.273 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:24.271+0000 7fc46be00640 1 --2- 172.21.15.32:0/911138078 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc4740acd80 0x7fc47415b500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:53:24.273 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:24.271+0000 7fc479c00640 1 --2- 172.21.15.32:0/911138078 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fc4740af020 0x7fc4741f8060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:53:24.273 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:24.271+0000 7fc479200640 1 -- 172.21.15.32:0/911138078 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fc4740af020 msgr2=0x7fc4741f8060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:53:24.273 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:24.271+0000 7fc479200640 1 --2- 172.21.15.32:0/911138078 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fc4740af020 0x7fc4741f8060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:53:24.274 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:24.271+0000 7fc479200640 1 -- 172.21.15.32:0/911138078 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc4740acd80 msgr2=0x7fc47415b500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:53:24.274 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:24.271+0000 7fc479200640 1 --2- 172.21.15.32:0/911138078 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc4740acd80 0x7fc47415b500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:53:24.274 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:24.271+0000 7fc479200640 1 -- 172.21.15.32:0/911138078 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc4740c8610 con 0x7fc47406f440 2024-08-03T20:53:24.274 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:24.271+0000 7fc479200640 1 --2- 172.21.15.32:0/911138078 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc47406f440 0x7fc4741f9620 secure :-1 s=READY pgs=780 cs=0 l=1 rev1=1 crypto rx=0x7fc46400e790 tx=0x7fc46400ec50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:53:24.275 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:24.272+0000 7fc471600640 1 -- 172.21.15.32:0/911138078 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc46400eea0 con 0x7fc47406f440 2024-08-03T20:53:24.275 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:24.272+0000 7fc471600640 1 -- 172.21.15.32:0/911138078 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fc46400cdb0 con 0x7fc47406f440 2024-08-03T20:53:24.275 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:24.272+0000 7fc471600640 1 -- 172.21.15.32:0/911138078 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc464010690 con 0x7fc47406f440 2024-08-03T20:53:24.275 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:24.272+0000 7fc47a600640 1 -- 172.21.15.32:0/911138078 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc4740c88f0 con 0x7fc47406f440 2024-08-03T20:53:24.275 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:24.272+0000 7fc47a600640 1 -- 172.21.15.32:0/911138078 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc4740c7020 con 0x7fc47406f440 2024-08-03T20:53:24.275 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:24.273+0000 7fc47a600640 1 -- 172.21.15.32:0/911138078 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc474073030 con 0x7fc47406f440 2024-08-03T20:53:24.275 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:24.274+0000 7fc471600640 1 -- 172.21.15.32:0/911138078 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7fc464010830 con 0x7fc47406f440 2024-08-03T20:53:24.276 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:24.275+0000 7fc471600640 1 --2- 172.21.15.32:0/911138078 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fc450077010 0x7fc450079520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:53:24.277 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:24.275+0000 7fc471600640 1 -- 172.21.15.32:0/911138078 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(70..70 src has 1..70) ==== 5709+0+0 (secure 0 0 0) 0x7fc464099170 con 0x7fc47406f440 2024-08-03T20:53:24.277 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:24.276+0000 7fc46be00640 1 --2- 172.21.15.32:0/911138078 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fc450077010 0x7fc450079520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:53:24.278 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:24.277+0000 7fc46be00640 1 --2- 172.21.15.32:0/911138078 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fc450077010 0x7fc450079520 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7fc45c033000 tx=0x7fc45c004730 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:53:24.281 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:24.279+0000 7fc471600640 1 -- 172.21.15.32:0/911138078 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7fc474073030 con 0x7fc47406f440 2024-08-03T20:53:24.522 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:24.520+0000 7fc47a600640 1 -- 172.21.15.32:0/911138078 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc4740c8e10 con 0x7fc47406f440 2024-08-03T20:53:24.523 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:24.521+0000 7fc471600640 1 -- 172.21.15.32:0/911138078 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 74 v74) ==== 94+0+5231 (secure 0 0 0) 0x7fc464061db0 con 0x7fc47406f440 2024-08-03T20:53:24.523 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-03T20:53:24.523 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":74,"btime":"2024-08-03T20:53:18:574147+0000","default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5666,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6837/2261538396","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6835","nonce":2261538396},{"type":"v1","addr":"172.21.15.118:6837","nonce":2261538396}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":69},{"gid":6061,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6836/1133580656","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6834","nonce":1133580656},{"type":"v1","addr":"172.21.15.118:6836","nonce":1133580656}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":68},{"gid":6396,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6834/1798758431","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":1798758431},{"type":"v1","addr":"172.21.15.32:6834","nonce":1798758431}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":69}],"filesystems":[{"mdsmap":{"epoch":74,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-08-03T20:53:15.080669+0000","modified":"2024-08-03T20:53:18.096135+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6399},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6399":{"gid":6399,"name":"a","rank":0,"incarnation":70,"state":"up:active","state_seq":3,"addr":"172.21.15.32:6835/4240363667","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6833","nonce":4240363667},{"type":"v1","addr":"172.21.15.32:6835","nonce":4240363667}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6399,"qdb_cluster":[6399]},"id":5}]} 2024-08-03T20:53:24.523 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 74 2024-08-03T20:53:24.526 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:24.524+0000 7fc47a600640 1 -- 172.21.15.32:0/911138078 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fc450077010 msgr2=0x7fc450079520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:53:24.526 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:24.524+0000 7fc47a600640 1 --2- 172.21.15.32:0/911138078 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fc450077010 0x7fc450079520 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7fc45c033000 tx=0x7fc45c004730 comp rx=0 tx=0).stop 2024-08-03T20:53:24.526 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:24.524+0000 7fc47a600640 1 -- 172.21.15.32:0/911138078 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc47406f440 msgr2=0x7fc4741f9620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:53:24.526 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:24.524+0000 7fc47a600640 1 --2- 172.21.15.32:0/911138078 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc47406f440 0x7fc4741f9620 secure :-1 s=READY pgs=780 cs=0 l=1 rev1=1 crypto rx=0x7fc46400e790 tx=0x7fc46400ec50 comp rx=0 tx=0).stop 2024-08-03T20:53:24.526 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:24.524+0000 7fc47a600640 1 -- 172.21.15.32:0/911138078 shutdown_connections 2024-08-03T20:53:24.526 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:24.524+0000 7fc47a600640 1 --2- 172.21.15.32:0/911138078 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fc450077010 0x7fc450079520 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:53:24.526 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:24.524+0000 7fc47a600640 1 --2- 172.21.15.32:0/911138078 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc4740acd80 0x7fc47415b500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:53:24.526 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:24.524+0000 7fc47a600640 1 --2- 172.21.15.32:0/911138078 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fc4740af020 0x7fc4741f8060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:53:24.526 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:24.524+0000 7fc47a600640 1 --2- 172.21.15.32:0/911138078 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc47406f440 0x7fc4741f9620 unknown :-1 s=CLOSED pgs=780 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:53:24.527 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:24.524+0000 7fc47a600640 1 -- 172.21.15.32:0/911138078 >> 172.21.15.32:0/911138078 conn(0x7fc4740b2120 msgr2=0x7fc4740afc80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:53:24.527 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:24.524+0000 7fc47a600640 1 -- 172.21.15.32:0/911138078 shutdown_connections 2024-08-03T20:53:24.527 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:24.524+0000 7fc47a600640 1 -- 172.21.15.32:0/911138078 wait complete. 2024-08-03T20:53:24.548 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mds freeze 5:0 true 2024-08-03T20:53:24.780 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:24.778+0000 7fc05ee00640 1 -- 172.21.15.32:0/1152585299 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc058152b30 msgr2=0x7fc058152f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:53:24.781 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:24.778+0000 7fc05ee00640 1 --2- 172.21.15.32:0/1152585299 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc058152b30 0x7fc058152f40 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7fc050005fd0 tx=0x7fc05002a8b0 comp rx=0 tx=0).stop 2024-08-03T20:53:24.781 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:24.778+0000 7fc05ee00640 1 -- 172.21.15.32:0/1152585299 shutdown_connections 2024-08-03T20:53:24.781 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:24.779+0000 7fc05ee00640 1 --2- 172.21.15.32:0/1152585299 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc058152b30 0x7fc058152f40 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:53:24.781 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:24.779+0000 7fc05ee00640 1 --2- 172.21.15.32:0/1152585299 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fc05815fe20 0x7fc058160230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:53:24.781 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:24.779+0000 7fc05ee00640 1 -- 172.21.15.32:0/1152585299 >> 172.21.15.32:0/1152585299 conn(0x7fc0580cfd40 msgr2=0x7fc0580d0150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:53:24.781 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:24.779+0000 7fc05ee00640 1 -- 172.21.15.32:0/1152585299 shutdown_connections 2024-08-03T20:53:24.781 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:24.779+0000 7fc05ee00640 1 -- 172.21.15.32:0/1152585299 wait complete. 2024-08-03T20:53:24.782 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:24.781+0000 7fc05ee00640 1 Processor -- start 2024-08-03T20:53:24.783 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:24.781+0000 7fc05ee00640 1 -- start start 2024-08-03T20:53:24.785 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:24.783+0000 7fc05ee00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc05806fc00 0x7fc05815c830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:53:24.785 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:24.783+0000 7fc05ee00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc058152b30 0x7fc0581f80d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:53:24.786 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:24.783+0000 7fc05ee00640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fc05815fe20 0x7fc05815e290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:53:24.786 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:24.783+0000 7fc05ee00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fc0580cb6c0 con 0x7fc05806fc00 2024-08-03T20:53:24.786 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:24.783+0000 7fc05ee00640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7fc0580cb540 con 0x7fc058152b30 2024-08-03T20:53:24.786 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:24.783+0000 7fc05ee00640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7fc058160b10 con 0x7fc05815fe20 2024-08-03T20:53:24.786 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:24.783+0000 7fc05da00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc05806fc00 0x7fc05815c830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:53:24.786 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:24.783+0000 7fc05da00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc05806fc00 0x7fc05815c830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:45604/0 (socket says 172.21.15.32:45604) 2024-08-03T20:53:24.786 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:24.783+0000 7fc05da00640 1 -- 172.21.15.32:0/4024543004 learned_addr learned my addr 172.21.15.32:0/4024543004 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:53:24.786 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:24.783+0000 7fc05d000640 1 --2- 172.21.15.32:0/4024543004 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc058152b30 0x7fc0581f80d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:53:24.786 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:24.783+0000 7fc05e400640 1 --2- 172.21.15.32:0/4024543004 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fc05815fe20 0x7fc05815e290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:53:24.786 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:24.783+0000 7fc05da00640 1 -- 172.21.15.32:0/4024543004 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fc05815fe20 msgr2=0x7fc05815e290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:53:24.786 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:24.783+0000 7fc05da00640 1 --2- 172.21.15.32:0/4024543004 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fc05815fe20 0x7fc05815e290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:53:24.786 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:24.783+0000 7fc05da00640 1 -- 172.21.15.32:0/4024543004 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc058152b30 msgr2=0x7fc0581f80d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:53:24.787 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:24.783+0000 7fc05da00640 1 --2- 172.21.15.32:0/4024543004 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc058152b30 0x7fc0581f80d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:53:24.787 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:24.783+0000 7fc05da00640 1 -- 172.21.15.32:0/4024543004 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc0580b47e0 con 0x7fc05806fc00 2024-08-03T20:53:24.787 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:24.783+0000 7fc05d000640 1 --2- 172.21.15.32:0/4024543004 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc058152b30 0x7fc0581f80d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-03T20:53:24.787 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:24.784+0000 7fc05da00640 1 --2- 172.21.15.32:0/4024543004 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc05806fc00 0x7fc05815c830 secure :-1 s=READY pgs=781 cs=0 l=1 rev1=1 crypto rx=0x7fc04c00c730 tx=0x7fc04c00cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:53:24.787 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:24.784+0000 7fc055600640 1 -- 172.21.15.32:0/4024543004 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc04c0052d0 con 0x7fc05806fc00 2024-08-03T20:53:24.787 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:24.784+0000 7fc055600640 1 -- 172.21.15.32:0/4024543004 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fc04c005490 con 0x7fc05806fc00 2024-08-03T20:53:24.787 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:24.784+0000 7fc055600640 1 -- 172.21.15.32:0/4024543004 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc04c00a8a0 con 0x7fc05806fc00 2024-08-03T20:53:24.787 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:24.784+0000 7fc05ee00640 1 -- 172.21.15.32:0/4024543004 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc0580b4aa0 con 0x7fc05806fc00 2024-08-03T20:53:24.787 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:24.784+0000 7fc05ee00640 1 -- 172.21.15.32:0/4024543004 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc0580b5060 con 0x7fc05806fc00 2024-08-03T20:53:24.787 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:24.785+0000 7fc05ee00640 1 -- 172.21.15.32:0/4024543004 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc058153560 con 0x7fc05806fc00 2024-08-03T20:53:24.788 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:24.786+0000 7fc055600640 1 -- 172.21.15.32:0/4024543004 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7fc04c00aa40 con 0x7fc05806fc00 2024-08-03T20:53:24.789 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:24.787+0000 7fc055600640 1 --2- 172.21.15.32:0/4024543004 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fc038077040 0x7fc038079550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:53:24.789 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:24.788+0000 7fc055600640 1 -- 172.21.15.32:0/4024543004 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(70..70 src has 1..70) ==== 5709+0+0 (secure 0 0 0) 0x7fc04c098450 con 0x7fc05806fc00 2024-08-03T20:53:24.789 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:24.788+0000 7fc05d000640 1 --2- 172.21.15.32:0/4024543004 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fc038077040 0x7fc038079550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:53:24.790 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:24.789+0000 7fc05d000640 1 --2- 172.21.15.32:0/4024543004 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fc038077040 0x7fc038079550 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7fc0581599d0 tx=0x7fc044009260 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:53:24.793 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:24.792+0000 7fc055600640 1 -- 172.21.15.32:0/4024543004 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7fc058160770 con 0x7fc05806fc00 2024-08-03T20:53:25.076 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:25.075+0000 7fc05ee00640 1 -- 172.21.15.32:0/4024543004 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "mds freeze", "role_or_gid": "5:0", "val": "true"} v 0) -- 0x7fc0580c9d40 con 0x7fc05806fc00 2024-08-03T20:53:25.609 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:25.607+0000 7fc055600640 1 -- 172.21.15.32:0/4024543004 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "mds freeze", "role_or_gid": "5:0", "val": "true"}]=0 freezing mds.6399 v75) ==== 112+0+0 (secure 0 0 0) 0x7fc04c061640 con 0x7fc05806fc00 2024-08-03T20:53:25.652 INFO:teuthology.orchestra.run.smithi032.stderr:freezing mds.6399 2024-08-03T20:53:25.654 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:25.609+0000 7fc05ee00640 1 -- 172.21.15.32:0/4024543004 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fc038077040 msgr2=0x7fc038079550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:53:25.654 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:25.609+0000 7fc05ee00640 1 --2- 172.21.15.32:0/4024543004 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fc038077040 0x7fc038079550 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7fc0581599d0 tx=0x7fc044009260 comp rx=0 tx=0).stop 2024-08-03T20:53:25.654 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:25.609+0000 7fc05ee00640 1 -- 172.21.15.32:0/4024543004 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc05806fc00 msgr2=0x7fc05815c830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:53:25.654 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:25.609+0000 7fc05ee00640 1 --2- 172.21.15.32:0/4024543004 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc05806fc00 0x7fc05815c830 secure :-1 s=READY pgs=781 cs=0 l=1 rev1=1 crypto rx=0x7fc04c00c730 tx=0x7fc04c00cbf0 comp rx=0 tx=0).stop 2024-08-03T20:53:25.654 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:25.609+0000 7fc05ee00640 1 -- 172.21.15.32:0/4024543004 shutdown_connections 2024-08-03T20:53:25.654 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:25.609+0000 7fc05ee00640 1 --2- 172.21.15.32:0/4024543004 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fc038077040 0x7fc038079550 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:53:25.654 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:25.609+0000 7fc05ee00640 1 --2- 172.21.15.32:0/4024543004 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc058152b30 0x7fc0581f80d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:53:25.654 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:25.609+0000 7fc05ee00640 1 --2- 172.21.15.32:0/4024543004 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fc05815fe20 0x7fc05815e290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:53:25.654 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:25.609+0000 7fc05ee00640 1 --2- 172.21.15.32:0/4024543004 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc05806fc00 0x7fc05815c830 unknown :-1 s=CLOSED pgs=781 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:53:25.654 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:25.609+0000 7fc05ee00640 1 -- 172.21.15.32:0/4024543004 >> 172.21.15.32:0/4024543004 conn(0x7fc0580cfd40 msgr2=0x7fc0580ad8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:53:25.654 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:25.610+0000 7fc05ee00640 1 -- 172.21.15.32:0/4024543004 shutdown_connections 2024-08-03T20:53:25.654 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:25.610+0000 7fc05ee00640 1 -- 172.21.15.32:0/4024543004 wait complete. 2024-08-03T20:53:25.655 DEBUG:teuthology.orchestra.run.smithi032:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'mkdir -p a/{b,c}/d') 2024-08-03T20:53:25.736 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:25 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-08-03T20:53:25.745 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:25 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-08-03T20:53:25.754 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.5:0 flush journal 2024-08-03T20:53:25.984 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:25.982+0000 7fdee0e00640 1 -- 172.21.15.32:0/1826088647 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdedc1621e0 msgr2=0x7fdedc1625f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:53:25.985 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:25.982+0000 7fdee0e00640 1 --2- 172.21.15.32:0/1826088647 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdedc1621e0 0x7fdedc1625f0 secure :-1 s=READY pgs=783 cs=0 l=1 rev1=1 crypto rx=0x7fdedc071cb0 tx=0x7fdec402a530 comp rx=0 tx=0).stop 2024-08-03T20:53:25.985 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:25.983+0000 7fdee0e00640 1 -- 172.21.15.32:0/1826088647 shutdown_connections 2024-08-03T20:53:25.985 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:25.983+0000 7fdee0e00640 1 --2- 172.21.15.32:0/1826088647 >> v2:172.21.15.118:3300/0 conn(0x7fdedc0b4640 0x7fdedc0b4a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:53:25.985 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:25.983+0000 7fdee0e00640 1 --2- 172.21.15.32:0/1826088647 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdedc1621e0 0x7fdedc1625f0 unknown :-1 s=CLOSED pgs=783 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:53:25.985 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:25.983+0000 7fdee0e00640 1 -- 172.21.15.32:0/1826088647 >> 172.21.15.32:0/1826088647 conn(0x7fdedc0b1bd0 msgr2=0x7fdedc0b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:53:25.985 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:25.983+0000 7fdee0e00640 1 -- 172.21.15.32:0/1826088647 shutdown_connections 2024-08-03T20:53:25.985 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:25.983+0000 7fdee0e00640 1 -- 172.21.15.32:0/1826088647 wait complete. 2024-08-03T20:53:25.986 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:25.985+0000 7fdee0e00640 1 Processor -- start 2024-08-03T20:53:25.986 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:25.985+0000 7fdee0e00640 1 -- start start 2024-08-03T20:53:25.989 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:25.986+0000 7fdee0e00640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fdedc0b4640 0x7fdedc1f8b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:53:25.989 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:25.986+0000 7fdee0e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdedc155720 0x7fdedc146a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:53:25.989 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:25.986+0000 7fdee0e00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fdedc1621e0 0x7fdedc0affb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:53:25.989 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:25.986+0000 7fdee0e00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fdedc0b5110 con 0x7fdedc155720 2024-08-03T20:53:25.989 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:25.986+0000 7fdee0e00640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7fdedc0b4f90 con 0x7fdedc1621e0 2024-08-03T20:53:25.989 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:25.986+0000 7fdee0e00640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7fdedc0b5290 con 0x7fdedc0b4640 2024-08-03T20:53:25.989 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:25.987+0000 7fded1400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdedc155720 0x7fdedc146a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:53:25.989 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:25.987+0000 7fded1400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdedc155720 0x7fdedc146a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:45630/0 (socket says 172.21.15.32:45630) 2024-08-03T20:53:25.989 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:25.987+0000 7fded1400640 1 -- 172.21.15.32:0/443050207 learned_addr learned my addr 172.21.15.32:0/443050207 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:53:25.989 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:25.987+0000 7fded9e00640 1 --2- 172.21.15.32:0/443050207 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fdedc0b4640 0x7fdedc1f8b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:53:25.990 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:25.987+0000 7fdeda800640 1 --2- 172.21.15.32:0/443050207 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fdedc1621e0 0x7fdedc0affb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:53:25.990 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:25.987+0000 7fded1400640 1 -- 172.21.15.32:0/443050207 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fdedc0b4640 msgr2=0x7fdedc1f8b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:53:25.990 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:25.987+0000 7fded1400640 1 --2- 172.21.15.32:0/443050207 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fdedc0b4640 0x7fdedc1f8b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:53:25.990 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:25.987+0000 7fded1400640 1 -- 172.21.15.32:0/443050207 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fdedc1621e0 msgr2=0x7fdedc0affb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:53:25.990 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:25.987+0000 7fded1400640 1 --2- 172.21.15.32:0/443050207 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fdedc1621e0 0x7fdedc0affb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:53:25.990 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:25.987+0000 7fded1400640 1 -- 172.21.15.32:0/443050207 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdedc0ce830 con 0x7fdedc155720 2024-08-03T20:53:25.990 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:25.987+0000 7fdeda800640 1 --2- 172.21.15.32:0/443050207 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fdedc1621e0 0x7fdedc0affb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-03T20:53:25.991 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:25.988+0000 7fded1400640 1 --2- 172.21.15.32:0/443050207 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdedc155720 0x7fdedc146a60 secure :-1 s=READY pgs=784 cs=0 l=1 rev1=1 crypto rx=0x7fdec40098f0 tx=0x7fdec4033c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:53:25.991 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:25.988+0000 7fded2a00640 1 -- 172.21.15.32:0/443050207 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdec4005c90 con 0x7fdedc155720 2024-08-03T20:53:25.991 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:25.988+0000 7fded2a00640 1 -- 172.21.15.32:0/443050207 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fdec4031e50 con 0x7fdedc155720 2024-08-03T20:53:25.991 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:25.988+0000 7fdee0e00640 1 -- 172.21.15.32:0/443050207 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdedc1f9330 con 0x7fdedc155720 2024-08-03T20:53:25.991 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:25.988+0000 7fded2a00640 1 -- 172.21.15.32:0/443050207 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdec4005260 con 0x7fdedc155720 2024-08-03T20:53:25.991 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:25.988+0000 7fdee0e00640 1 -- 172.21.15.32:0/443050207 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdedc1f98f0 con 0x7fdedc155720 2024-08-03T20:53:25.992 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:25.991+0000 7fded2a00640 1 -- 172.21.15.32:0/443050207 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7fdec4005400 con 0x7fdedc155720 2024-08-03T20:53:25.993 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:25.991+0000 7fded2a00640 1 --2- 172.21.15.32:0/443050207 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fdeb8077040 0x7fdeb8079550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:53:25.993 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:25.992+0000 7fded2a00640 1 -- 172.21.15.32:0/443050207 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(70..70 src has 1..70) ==== 5709+0+0 (secure 0 0 0) 0x7fdec40b9a40 con 0x7fdedc155720 2024-08-03T20:53:25.993 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:25.992+0000 7fdee7ff7740 1 Processor -- start 2024-08-03T20:53:25.993 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:25.992+0000 7fdee7ff7740 1 -- start start 2024-08-03T20:53:25.994 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:25.992+0000 7fded9e00640 1 --2- 172.21.15.32:0/443050207 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fdeb8077040 0x7fdeb8079550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:53:25.994 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:25.993+0000 7fded9e00640 1 --2- 172.21.15.32:0/443050207 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fdeb8077040 0x7fdeb8079550 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7fdecc007fd0 tx=0x7fdecc001370 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:53:25.995 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:25.993+0000 7fdee7ff7740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56325455d390 0x56325492d900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:53:25.995 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:25.993+0000 7fdee7ff7740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5632545344d0 0x56325492de40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:53:25.995 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:25.993+0000 7fdee7ff7740 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x56325453c6b0 0x56325494ad40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:53:25.996 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:25.993+0000 7fdee7ff7740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x563254876470 con 0x5632545344d0 2024-08-03T20:53:25.996 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:25.993+0000 7fdee7ff7740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x563254865970 con 0x56325455d390 2024-08-03T20:53:25.996 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:25.993+0000 7fdee7ff7740 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x5632548699b0 con 0x56325453c6b0 2024-08-03T20:53:25.996 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:25.994+0000 7fded1400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5632545344d0 0x56325492de40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:53:25.996 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:25.994+0000 7fded1400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5632545344d0 0x56325492de40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:45646/0 (socket says 172.21.15.32:45646) 2024-08-03T20:53:25.996 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:25.994+0000 7fded1400640 1 -- 172.21.15.32:0/3058875049 learned_addr learned my addr 172.21.15.32:0/3058875049 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:53:25.996 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:25.994+0000 7fdeda800640 1 --2- 172.21.15.32:0/3058875049 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x56325453c6b0 0x56325494ad40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:53:25.996 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:25.994+0000 7fded9e00640 1 --2- 172.21.15.32:0/3058875049 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56325455d390 0x56325492d900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:53:25.996 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:25.994+0000 7fded1400640 1 -- 172.21.15.32:0/3058875049 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x56325453c6b0 msgr2=0x56325494ad40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:53:25.996 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:25.994+0000 7fded1400640 1 --2- 172.21.15.32:0/3058875049 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x56325453c6b0 0x56325494ad40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:53:25.996 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:25.994+0000 7fded1400640 1 -- 172.21.15.32:0/3058875049 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56325455d390 msgr2=0x56325492d900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:53:25.996 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:25.994+0000 7fded1400640 1 --2- 172.21.15.32:0/3058875049 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56325455d390 0x56325492d900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:53:25.996 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:25.994+0000 7fded1400640 1 -- 172.21.15.32:0/3058875049 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56325463f7d0 con 0x5632545344d0 2024-08-03T20:53:25.997 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:25.994+0000 7fded9e00640 1 --2- 172.21.15.32:0/3058875049 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56325455d390 0x56325492d900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-03T20:53:25.998 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:25.994+0000 7fded1400640 1 --2- 172.21.15.32:0/3058875049 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5632545344d0 0x56325492de40 secure :-1 s=READY pgs=785 cs=0 l=1 rev1=1 crypto rx=0x7fdec4005eb0 tx=0x7fdec4081d80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=25e4fdfa2229026f server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:53:25.998 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:25.995+0000 7fdecaa00640 1 -- 172.21.15.32:0/3058875049 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdec4048030 con 0x5632545344d0 2024-08-03T20:53:25.998 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:25.995+0000 7fdecaa00640 1 -- 172.21.15.32:0/3058875049 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fdec408b550 con 0x5632545344d0 2024-08-03T20:53:25.998 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:25.995+0000 7fdecaa00640 1 -- 172.21.15.32:0/3058875049 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdec40829b0 con 0x5632545344d0 2024-08-03T20:53:25.998 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:25.995+0000 7fdee7ff7740 1 -- 172.21.15.32:0/3058875049 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5632545344d0 msgr2=0x56325492de40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:53:25.998 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:25.995+0000 7fdee7ff7740 1 --2- 172.21.15.32:0/3058875049 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5632545344d0 0x56325492de40 secure :-1 s=READY pgs=785 cs=0 l=1 rev1=1 crypto rx=0x7fdec4005eb0 tx=0x7fdec4081d80 comp rx=0 tx=0).stop 2024-08-03T20:53:25.998 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:25.996+0000 7fdee7ff7740 1 -- 172.21.15.32:0/3058875049 shutdown_connections 2024-08-03T20:53:25.998 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:25.996+0000 7fdee7ff7740 1 --2- 172.21.15.32:0/3058875049 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x56325453c6b0 0x56325494ad40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:53:25.998 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:25.996+0000 7fdee7ff7740 1 --2- 172.21.15.32:0/3058875049 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5632545344d0 0x56325492de40 unknown :-1 s=CLOSED pgs=785 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:53:25.998 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:25.996+0000 7fdee7ff7740 1 --2- 172.21.15.32:0/3058875049 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56325455d390 0x56325492d900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:53:25.999 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:25.996+0000 7fdee7ff7740 1 -- 172.21.15.32:0/3058875049 >> 172.21.15.32:0/3058875049 conn(0x5632545514e0 msgr2=0x563254903bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:53:25.999 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:25.996+0000 7fdee7ff7740 1 -- 172.21.15.32:0/3058875049 shutdown_connections 2024-08-03T20:53:25.999 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:25.996+0000 7fdee7ff7740 1 -- 172.21.15.32:0/3058875049 wait complete. 2024-08-03T20:53:25.999 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:25.997+0000 7fdee7ff7740 1 -- start start 2024-08-03T20:53:25.999 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:25.997+0000 7fdee7ff7740 1 Processor -- start 2024-08-03T20:53:26.000 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:25.998+0000 7fdee7ff7740 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x5632545344d0 0x563254949130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:53:26.000 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:25.998+0000 7fdee7ff7740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56325453c6b0 0x563254942540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:53:26.000 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:25.998+0000 7fdee7ff7740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56325455d390 0x56325495af40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:53:26.000 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:25.998+0000 7fdee7ff7740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x56325479dc90 con 0x56325455d390 2024-08-03T20:53:26.000 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:25.998+0000 7fdee7ff7740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x5632547fffd0 con 0x56325453c6b0 2024-08-03T20:53:26.000 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:25.998+0000 7fdee7ff7740 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x563254876470 con 0x5632545344d0 2024-08-03T20:53:26.000 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:25.998+0000 7fdee7ff7740 10 client.0 _ll_register_callbacks cb 0x56325481ae50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-03T20:53:26.000 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:25.999+0000 7fdeda800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56325455d390 0x56325495af40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:53:26.000 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:25.999+0000 7fdeda800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56325455d390 0x56325495af40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:45654/0 (socket says 172.21.15.32:45654) 2024-08-03T20:53:26.001 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:25.999+0000 7fdeda800640 1 -- 172.21.15.32:0/1952385775 learned_addr learned my addr 172.21.15.32:0/1952385775 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:53:26.002 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:25.999+0000 7fded1400640 1 --2- 172.21.15.32:0/1952385775 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56325453c6b0 0x563254942540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:53:26.003 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:25.999+0000 7fded9e00640 1 --2- 172.21.15.32:0/1952385775 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x5632545344d0 0x563254949130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:53:26.003 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:25.999+0000 7fdeda800640 1 -- 172.21.15.32:0/1952385775 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x5632545344d0 msgr2=0x563254949130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:53:26.003 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:25.999+0000 7fdeda800640 1 --2- 172.21.15.32:0/1952385775 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x5632545344d0 0x563254949130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:53:26.003 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:25.999+0000 7fdeda800640 1 -- 172.21.15.32:0/1952385775 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56325453c6b0 msgr2=0x563254942540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:53:26.003 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:25.999+0000 7fdeda800640 1 --2- 172.21.15.32:0/1952385775 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56325453c6b0 0x563254942540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:53:26.003 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:25.999+0000 7fdeda800640 1 -- 172.21.15.32:0/1952385775 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5632545c8970 con 0x56325455d390 2024-08-03T20:53:26.003 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:25.999+0000 7fdeda800640 1 --2- 172.21.15.32:0/1952385775 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56325455d390 0x56325495af40 secure :-1 s=READY pgs=786 cs=0 l=1 rev1=1 crypto rx=0x7fded40138e0 tx=0x7fded4013da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f3a89d1737243b5d server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:53:26.003 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:25.999+0000 7fdebf400640 10 client.0 ms_handle_connect on v2:172.21.15.32:3300/0 2024-08-03T20:53:26.003 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:26.000+0000 7fdebf400640 1 -- 172.21.15.32:0/1952385775 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fded4011e30 con 0x56325455d390 2024-08-03T20:53:26.003 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:26.000+0000 7fdebf400640 1 -- 172.21.15.32:0/1952385775 <== mon.0 v2:172.21.15.32:3300/0 2 ==== osd_map(70..70 src has 1..70) ==== 5709+0+0 (secure 0 0 0) 0x7fded4004b20 con 0x56325455d390 2024-08-03T20:53:26.003 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:26.000+0000 7fdee7ff7740 1 -- 172.21.15.32:0/1952385775 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdec408b550 con 0x56325455d390 2024-08-03T20:53:26.003 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:26.000+0000 7fdebf400640 1 -- 172.21.15.32:0/1952385775 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_get_version_reply(handle=1 version=75) ==== 24+0+0 (secure 0 0 0) 0x7fded4004d20 con 0x56325455d390 2024-08-03T20:53:26.004 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:26.000+0000 7fdee7ff7740 10 client.6543 fetch_fsmap learned FSMap version 75 2024-08-03T20:53:26.004 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:26.000+0000 7fdee7ff7740 1 -- 172.21.15.32:0/1952385775 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,fsmap=75,monmap=2+}) -- 0x56325450e270 con 0x56325455d390 2024-08-03T20:53:26.004 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:26.000+0000 7fdebf400640 1 -- 172.21.15.32:0/1952385775 <== mon.0 v2:172.21.15.32:3300/0 4 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fdec408b550 con 0x56325455d390 2024-08-03T20:53:26.004 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:26.000+0000 7fdebf400640 1 -- 172.21.15.32:0/1952385775 <== mon.0 v2:172.21.15.32:3300/0 5 ==== fsmap(e 75) ==== 3502+0+0 (secure 0 0 0) 0x7fded4023050 con 0x56325455d390 2024-08-03T20:53:26.004 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:26.000+0000 7fdee7ff7740 10 client.6543 fetch_fsmap finished waiting for FSMap version 75 2024-08-03T20:53:26.004 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:26.000+0000 7fdee7ff7740 10 client.6543 resolve_mds: resolved 5:0 to role '5:0' aka daemon mds.a 2024-08-03T20:53:26.004 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:26.001+0000 7fdee7ff7740 20 client.6543 populate_metadata read hostname 'smithi032' 2024-08-03T20:53:26.004 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:26.001+0000 7fdee7ff7740 1 --2- 172.21.15.32:0/1952385775 >> [v2:172.21.15.32:6833/4240363667,v1:172.21.15.32:6835/4240363667] conn(0x56325451bf60 0x56325490d0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:53:26.004 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:26.001+0000 7fdee7ff7740 4 client.6543 mds_command: new command op to 6399 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-08-03T20:53:26.004 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:26.001+0000 7fdee7ff7740 1 -- 172.21.15.32:0/1952385775 --> [v2:172.21.15.32:6833/4240363667,v1:172.21.15.32:6835/4240363667] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5632548783c0 con 0x56325451bf60 2024-08-03T20:53:26.004 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:26.001+0000 7fded9e00640 1 --2- 172.21.15.32:0/1952385775 >> [v2:172.21.15.32:6833/4240363667,v1:172.21.15.32:6835/4240363667] conn(0x56325451bf60 0x56325490d0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:53:26.004 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:26.001+0000 7fded9e00640 1 --2- 172.21.15.32:0/1952385775 >> [v2:172.21.15.32:6833/4240363667,v1:172.21.15.32:6835/4240363667] conn(0x56325451bf60 0x56325490d0b0 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d372f96ef4e35078 server_cookie=e84c8f0e53594476 in_seq=0 out_seq=0 2024-08-03T20:53:26.004 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:26.001+0000 7fdebf400640 10 client.6543 ms_handle_connect on v2:172.21.15.32:6833/4240363667 2024-08-03T20:53:26.005 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:26.002+0000 7fdebf400640 1 -- 172.21.15.32:0/1952385775 <== mds.0 v2:172.21.15.32:6833/4240363667 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x5632548783c0 con 0x56325451bf60 2024-08-03T20:53:26.006 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:26.002+0000 7fdebf400640 10 client.6543 handle_command_reply: tid=0 2024-08-03T20:53:26.006 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:26.002+0000 7fdee7ff7740 1 client.6543 shutdown 2024-08-03T20:53:26.006 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:26.002+0000 7fdee7ff7740 1 -- 172.21.15.32:0/1952385775 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56325455d390 msgr2=0x56325495af40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:53:26.006 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:26.002+0000 7fdee7ff7740 1 --2- 172.21.15.32:0/1952385775 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56325455d390 0x56325495af40 secure :-1 s=READY pgs=786 cs=0 l=1 rev1=1 crypto rx=0x7fded40138e0 tx=0x7fded4013da0 comp rx=0 tx=0).stop 2024-08-03T20:53:26.006 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:26.002+0000 7fdee7ff7740 1 -- 172.21.15.32:0/1952385775 shutdown_connections 2024-08-03T20:53:26.006 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:26.002+0000 7fdee7ff7740 1 --2- 172.21.15.32:0/1952385775 >> [v2:172.21.15.32:6833/4240363667,v1:172.21.15.32:6835/4240363667] conn(0x56325451bf60 0x56325490d0b0 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:53:26.006 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:26.002+0000 7fdee7ff7740 1 --2- 172.21.15.32:0/1952385775 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56325453c6b0 0x563254942540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:53:26.006 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:26.002+0000 7fdee7ff7740 1 --2- 172.21.15.32:0/1952385775 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56325455d390 0x56325495af40 unknown :-1 s=CLOSED pgs=786 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:53:26.006 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:26.002+0000 7fdee7ff7740 1 --2- 172.21.15.32:0/1952385775 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x5632545344d0 0x563254949130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:53:26.006 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:26.003+0000 7fdee7ff7740 1 -- 172.21.15.32:0/1952385775 >> 172.21.15.32:0/1952385775 conn(0x5632545514e0 msgr2=0x563254903a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:53:26.006 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:26.003+0000 7fdebf400640 0 client.6543 ms_handle_reset on v2:172.21.15.32:6833/4240363667 2024-08-03T20:53:26.006 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:26.003+0000 7fdee7ff7740 1 -- 172.21.15.32:0/1952385775 shutdown_connections 2024-08-03T20:53:26.007 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:26.003+0000 7fdee7ff7740 1 -- 172.21.15.32:0/1952385775 wait complete. 2024-08-03T20:53:26.007 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:26.003+0000 7fdee7ff7740 20 client.6543 trim_cache size 0 max 16384 2024-08-03T20:53:26.020 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:26.019+0000 7fdee7ff7740 1 Processor -- start 2024-08-03T20:53:26.021 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:26.019+0000 7fdee7ff7740 1 -- start start 2024-08-03T20:53:26.022 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:26.020+0000 7fdee7ff7740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5632545344d0 0x563254903c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:53:26.022 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:26.020+0000 7fdee7ff7740 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x56325453c6b0 0x56325491afa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:53:26.022 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:26.021+0000 7fdee7ff7740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56325455d390 0x56325491b4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:53:26.022 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:26.021+0000 7fdee7ff7740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fdedc201d90 con 0x56325455d390 2024-08-03T20:53:26.022 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:26.021+0000 7fdee7ff7740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7fdedc200e50 con 0x5632545344d0 2024-08-03T20:53:26.023 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:26.021+0000 7fdee7ff7740 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7fdedc0cf3e0 con 0x56325453c6b0 2024-08-03T20:53:26.023 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:26.021+0000 7fdeda800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56325455d390 0x56325491b4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:53:26.023 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:26.021+0000 7fdeda800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56325455d390 0x56325491b4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:45664/0 (socket says 172.21.15.32:45664) 2024-08-03T20:53:26.023 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:26.021+0000 7fdeda800640 1 -- 172.21.15.32:0/121685036 learned_addr learned my addr 172.21.15.32:0/121685036 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:53:26.024 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:26.021+0000 7fdeda800640 1 -- 172.21.15.32:0/121685036 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x56325453c6b0 msgr2=0x56325491afa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:53:26.024 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:26.021+0000 7fdeda800640 1 --2- 172.21.15.32:0/121685036 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x56325453c6b0 0x56325491afa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:53:26.024 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:26.021+0000 7fded9e00640 1 --2- 172.21.15.32:0/121685036 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5632545344d0 0x563254903c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:53:26.024 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:26.021+0000 7fdeda800640 1 -- 172.21.15.32:0/121685036 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5632545344d0 msgr2=0x563254903c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:53:26.025 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:26.021+0000 7fdeda800640 1 --2- 172.21.15.32:0/121685036 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5632545344d0 0x563254903c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:53:26.025 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:26.021+0000 7fdeda800640 1 -- 172.21.15.32:0/121685036 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563254949970 con 0x56325455d390 2024-08-03T20:53:26.025 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:26.021+0000 7fdeda800640 1 --2- 172.21.15.32:0/121685036 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56325455d390 0x56325491b4e0 secure :-1 s=READY pgs=787 cs=0 l=1 rev1=1 crypto rx=0x7fded4011920 tx=0x7fded4004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=18d5f65caaec715 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:53:26.025 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:26.022+0000 7fdeca000640 1 -- 172.21.15.32:0/121685036 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fded40050e0 con 0x56325455d390 2024-08-03T20:53:26.025 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:26.022+0000 7fdeca000640 1 -- 172.21.15.32:0/121685036 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fded402a020 con 0x56325455d390 2024-08-03T20:53:26.025 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:26.022+0000 7fdeca000640 1 -- 172.21.15.32:0/121685036 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fded4005370 con 0x56325455d390 2024-08-03T20:53:26.025 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:26.022+0000 7fdee7ff7740 1 -- 172.21.15.32:0/121685036 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56325455d390 msgr2=0x56325491b4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:53:26.025 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:26.022+0000 7fdee7ff7740 1 --2- 172.21.15.32:0/121685036 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56325455d390 0x56325491b4e0 secure :-1 s=READY pgs=787 cs=0 l=1 rev1=1 crypto rx=0x7fded4011920 tx=0x7fded4004ab0 comp rx=0 tx=0).stop 2024-08-03T20:53:26.025 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:26.022+0000 7fdee7ff7740 1 -- 172.21.15.32:0/121685036 shutdown_connections 2024-08-03T20:53:26.025 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:26.022+0000 7fdee7ff7740 1 --2- 172.21.15.32:0/121685036 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56325455d390 0x56325491b4e0 unknown :-1 s=CLOSED pgs=787 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:53:26.026 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:26.022+0000 7fdee7ff7740 1 --2- 172.21.15.32:0/121685036 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x56325453c6b0 0x56325491afa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:53:26.026 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:26.022+0000 7fdee7ff7740 1 --2- 172.21.15.32:0/121685036 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5632545344d0 0x563254903c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:53:26.026 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:26.023+0000 7fdee7ff7740 1 -- 172.21.15.32:0/121685036 >> 172.21.15.32:0/121685036 conn(0x5632545514e0 msgr2=0x563254904ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:53:26.026 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:26.023+0000 7fdee7ff7740 1 -- 172.21.15.32:0/121685036 shutdown_connections 2024-08-03T20:53:26.026 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:26.023+0000 7fdee7ff7740 1 -- 172.21.15.32:0/121685036 wait complete. 2024-08-03T20:53:26.028 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:26.024+0000 7fdee7ff7740 1 -- start start 2024-08-03T20:53:26.028 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:26.024+0000 7fdee7ff7740 1 Processor -- start 2024-08-03T20:53:26.029 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:26.025+0000 7fdee7ff7740 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x5632545344d0 0x563254954620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:53:26.029 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:26.025+0000 7fdee7ff7740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56325453c6b0 0x5632549c8bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:53:26.029 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:26.025+0000 7fdee7ff7740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56325455d390 0x563254961410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:53:26.029 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:26.025+0000 7fdee7ff7740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x56325493e910 con 0x56325453c6b0 2024-08-03T20:53:26.029 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:26.025+0000 7fdee7ff7740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7fdedc0cfa30 con 0x56325455d390 2024-08-03T20:53:26.029 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:26.025+0000 7fdee7ff7740 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x5632549c58d0 con 0x5632545344d0 2024-08-03T20:53:26.029 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:26.025+0000 7fdee7ff7740 10 client.0 _ll_register_callbacks cb 0x7fded4009aa0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-03T20:53:26.029 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:26.025+0000 7fded1400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56325453c6b0 0x5632549c8bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:53:26.029 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:26.026+0000 7fded1400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56325453c6b0 0x5632549c8bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:45680/0 (socket says 172.21.15.32:45680) 2024-08-03T20:53:26.029 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:26.026+0000 7fded1400640 1 -- 172.21.15.32:0/2694275318 learned_addr learned my addr 172.21.15.32:0/2694275318 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:53:26.029 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:26.026+0000 7fded9e00640 1 --2- 172.21.15.32:0/2694275318 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x5632545344d0 0x563254954620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:53:26.029 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:26.026+0000 7fdeda800640 1 --2- 172.21.15.32:0/2694275318 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56325455d390 0x563254961410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:53:26.029 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:26.026+0000 7fded1400640 1 -- 172.21.15.32:0/2694275318 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x5632545344d0 msgr2=0x563254954620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:53:26.029 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:26.026+0000 7fded1400640 1 --2- 172.21.15.32:0/2694275318 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x5632545344d0 0x563254954620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:53:26.030 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:26.026+0000 7fded1400640 1 -- 172.21.15.32:0/2694275318 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56325455d390 msgr2=0x563254961410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:53:26.030 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:26.026+0000 7fded1400640 1 --2- 172.21.15.32:0/2694275318 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56325455d390 0x563254961410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:53:26.030 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:26.026+0000 7fded1400640 1 -- 172.21.15.32:0/2694275318 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5632549c9100 con 0x56325453c6b0 2024-08-03T20:53:26.030 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:26.026+0000 7fded1400640 1 --2- 172.21.15.32:0/2694275318 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56325453c6b0 0x5632549c8bc0 secure :-1 s=READY pgs=788 cs=0 l=1 rev1=1 crypto rx=0x7fdec407eaf0 tx=0x7fdec40826d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4b5c4925821ca93d server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:53:26.030 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:26.026+0000 7fdebf400640 10 client.0 ms_handle_connect on v2:172.21.15.32:3300/0 2024-08-03T20:53:26.030 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:26.026+0000 7fdebf400640 1 -- 172.21.15.32:0/2694275318 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdec40a0060 con 0x56325453c6b0 2024-08-03T20:53:26.030 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:26.026+0000 7fdebf400640 1 -- 172.21.15.32:0/2694275318 <== mon.0 v2:172.21.15.32:3300/0 2 ==== osd_map(70..70 src has 1..70) ==== 5709+0+0 (secure 0 0 0) 0x7fdec408b2d0 con 0x56325453c6b0 2024-08-03T20:53:26.030 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:26.026+0000 7fdee7ff7740 1 -- 172.21.15.32:0/2694275318 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fded402a020 con 0x56325453c6b0 2024-08-03T20:53:26.030 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:26.027+0000 7fdebf400640 1 -- 172.21.15.32:0/2694275318 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_get_version_reply(handle=1 version=75) ==== 24+0+0 (secure 0 0 0) 0x7fdec4048050 con 0x56325453c6b0 2024-08-03T20:53:26.030 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:26.027+0000 7fdee7ff7740 10 client.6549 fetch_fsmap learned FSMap version 75 2024-08-03T20:53:26.030 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:26.027+0000 7fdee7ff7740 1 -- 172.21.15.32:0/2694275318 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,fsmap=75,monmap=2+}) -- 0x56325491b9b0 con 0x56325453c6b0 2024-08-03T20:53:26.030 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:26.027+0000 7fdebf400640 1 -- 172.21.15.32:0/2694275318 <== mon.0 v2:172.21.15.32:3300/0 4 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fded402a020 con 0x56325453c6b0 2024-08-03T20:53:26.030 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:26.027+0000 7fdebf400640 1 -- 172.21.15.32:0/2694275318 <== mon.0 v2:172.21.15.32:3300/0 5 ==== fsmap(e 75) ==== 3502+0+0 (secure 0 0 0) 0x7fdec4097b40 con 0x56325453c6b0 2024-08-03T20:53:26.031 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:26.027+0000 7fdee7ff7740 10 client.6549 fetch_fsmap finished waiting for FSMap version 75 2024-08-03T20:53:26.031 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:26.027+0000 7fdee7ff7740 10 client.6549 resolve_mds: resolved 5:0 to role '5:0' aka daemon mds.a 2024-08-03T20:53:26.031 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:26.027+0000 7fdee7ff7740 20 client.6549 populate_metadata read hostname 'smithi032' 2024-08-03T20:53:26.031 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:26.027+0000 7fdee7ff7740 1 --2- 172.21.15.32:0/2694275318 >> [v2:172.21.15.32:6833/4240363667,v1:172.21.15.32:6835/4240363667] conn(0x5632549c4c50 0x5632549c5060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:53:26.031 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:26.027+0000 7fdee7ff7740 4 client.6549 mds_command: new command op to 6399 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-08-03T20:53:26.031 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:26.027+0000 7fdee7ff7740 1 -- 172.21.15.32:0/2694275318 --> [v2:172.21.15.32:6833/4240363667,v1:172.21.15.32:6835/4240363667] -- command(tid 0: {"prefix": "flush journal"}) -- 0x563254873e60 con 0x5632549c4c50 2024-08-03T20:53:26.031 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:26.027+0000 7fded9e00640 1 --2- 172.21.15.32:0/2694275318 >> [v2:172.21.15.32:6833/4240363667,v1:172.21.15.32:6835/4240363667] conn(0x5632549c4c50 0x5632549c5060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:53:26.031 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:26.028+0000 7fded9e00640 1 --2- 172.21.15.32:0/2694275318 >> [v2:172.21.15.32:6833/4240363667,v1:172.21.15.32:6835/4240363667] conn(0x5632549c4c50 0x5632549c5060 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=432830e47aa45b09 server_cookie=fda5febac359295f in_seq=0 out_seq=0 2024-08-03T20:53:26.031 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:26.028+0000 7fdebf400640 10 client.6549 ms_handle_connect on v2:172.21.15.32:6833/4240363667 2024-08-03T20:53:26.043 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:26.041+0000 7fdebf400640 1 -- 172.21.15.32:0/2694275318 <== mds.0 v2:172.21.15.32:6833/4240363667 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x563254873e60 con 0x5632549c4c50 2024-08-03T20:53:26.043 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:26.041+0000 7fdebf400640 10 client.6549 handle_command_reply: tid=0 2024-08-03T20:53:26.043 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:26.041+0000 7fdee7ff7740 1 client.6549 shutdown 2024-08-03T20:53:26.043 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:26.041+0000 7fdee7ff7740 1 -- 172.21.15.32:0/2694275318 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56325453c6b0 msgr2=0x5632549c8bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:53:26.043 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:26.041+0000 7fdee7ff7740 1 --2- 172.21.15.32:0/2694275318 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56325453c6b0 0x5632549c8bc0 secure :-1 s=READY pgs=788 cs=0 l=1 rev1=1 crypto rx=0x7fdec407eaf0 tx=0x7fdec40826d0 comp rx=0 tx=0).stop 2024-08-03T20:53:26.043 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:26.041+0000 7fdee7ff7740 1 -- 172.21.15.32:0/2694275318 shutdown_connections 2024-08-03T20:53:26.044 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:26.041+0000 7fdee7ff7740 1 --2- 172.21.15.32:0/2694275318 >> [v2:172.21.15.32:6833/4240363667,v1:172.21.15.32:6835/4240363667] conn(0x5632549c4c50 0x5632549c5060 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:53:26.044 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:26.041+0000 7fdee7ff7740 1 --2- 172.21.15.32:0/2694275318 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56325455d390 0x563254961410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:53:26.044 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:26.041+0000 7fdee7ff7740 1 --2- 172.21.15.32:0/2694275318 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56325453c6b0 0x5632549c8bc0 unknown :-1 s=CLOSED pgs=788 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:53:26.044 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:26.041+0000 7fdee7ff7740 1 --2- 172.21.15.32:0/2694275318 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x5632545344d0 0x563254954620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:53:26.044 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:26.041+0000 7fdebf400640 0 client.6549 ms_handle_reset on v2:172.21.15.32:6833/4240363667 2024-08-03T20:53:26.044 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:26.041+0000 7fdee7ff7740 1 -- 172.21.15.32:0/2694275318 >> 172.21.15.32:0/2694275318 conn(0x5632545514e0 msgr2=0x563254904ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:53:26.044 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:26.042+0000 7fdee7ff7740 1 -- 172.21.15.32:0/2694275318 shutdown_connections 2024-08-03T20:53:26.044 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:26.042+0000 7fdee7ff7740 1 -- 172.21.15.32:0/2694275318 wait complete. 2024-08-03T20:53:26.044 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:26.042+0000 7fdee7ff7740 20 client.6549 trim_cache size 0 max 16384 2024-08-03T20:53:26.044 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-08-03T20:53:26.044 INFO:teuthology.orchestra.run.smithi032.stdout: "message": "", 2024-08-03T20:53:26.045 INFO:teuthology.orchestra.run.smithi032.stdout: "return_code": 0 2024-08-03T20:53:26.045 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-08-03T20:53:26.045 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:26.042+0000 7fdee0e00640 1 -- 172.21.15.32:0/443050207 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fdeb8077040 msgr2=0x7fdeb8079550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:53:26.045 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:26.042+0000 7fdee0e00640 1 --2- 172.21.15.32:0/443050207 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fdeb8077040 0x7fdeb8079550 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7fdecc007fd0 tx=0x7fdecc001370 comp rx=0 tx=0).stop 2024-08-03T20:53:26.045 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:26.042+0000 7fdee0e00640 1 -- 172.21.15.32:0/443050207 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdedc155720 msgr2=0x7fdedc146a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:53:26.046 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:26.042+0000 7fdee0e00640 1 --2- 172.21.15.32:0/443050207 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdedc155720 0x7fdedc146a60 secure :-1 s=READY pgs=784 cs=0 l=1 rev1=1 crypto rx=0x7fdec40098f0 tx=0x7fdec4033c20 comp rx=0 tx=0).stop 2024-08-03T20:53:26.046 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:26.042+0000 7fdee0e00640 1 -- 172.21.15.32:0/443050207 shutdown_connections 2024-08-03T20:53:26.046 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:26.042+0000 7fdee0e00640 1 --2- 172.21.15.32:0/443050207 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fdeb8077040 0x7fdeb8079550 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:53:26.046 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:26.042+0000 7fdee0e00640 1 --2- 172.21.15.32:0/443050207 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fdedc1621e0 0x7fdedc0affb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:53:26.046 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:26.042+0000 7fdee0e00640 1 --2- 172.21.15.32:0/443050207 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdedc155720 0x7fdedc146a60 unknown :-1 s=CLOSED pgs=784 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:53:26.046 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:26.042+0000 7fdee0e00640 1 --2- 172.21.15.32:0/443050207 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fdedc0b4640 0x7fdedc1f8b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:53:26.046 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:26.042+0000 7fdee0e00640 1 -- 172.21.15.32:0/443050207 >> 172.21.15.32:0/443050207 conn(0x7fdedc0b1bd0 msgr2=0x7fdedc0b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:53:26.046 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:26.043+0000 7fdee0e00640 1 -- 172.21.15.32:0/443050207 shutdown_connections 2024-08-03T20:53:26.046 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:26.043+0000 7fdee0e00640 1 -- 172.21.15.32:0/443050207 wait complete. 2024-08-03T20:53:26.066 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mds mds_inject_rename_corrupt_dentry_first 1.0 2024-08-03T20:53:26.299 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:26.297+0000 7f0e91400640 1 -- 172.21.15.32:0/1137540051 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0e70004d40 msgr2=0x7f0e70005190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:53:26.299 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:26.297+0000 7f0e91400640 1 --2- 172.21.15.32:0/1137540051 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0e70004d40 0x7f0e70005190 secure :-1 s=READY pgs=790 cs=0 l=1 rev1=1 crypto rx=0x7f0e84009ed0 tx=0x7f0e70005830 comp rx=0 tx=0).stop 2024-08-03T20:53:26.299 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:26.297+0000 7f0e8b200640 1 -- 172.21.15.32:0/1137540051 reap_dead start 2024-08-03T20:53:26.299 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:26.297+0000 7f0e91400640 1 -- 172.21.15.32:0/1137540051 shutdown_connections 2024-08-03T20:53:26.299 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:26.297+0000 7f0e91400640 1 -- 172.21.15.32:0/1137540051 >> 172.21.15.32:0/1137540051 conn(0x7f0e8c0b1bd0 msgr2=0x7f0e8c0b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:53:26.299 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:26.297+0000 7f0e91400640 1 -- 172.21.15.32:0/1137540051 shutdown_connections 2024-08-03T20:53:26.300 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:26.297+0000 7f0e91400640 1 -- 172.21.15.32:0/1137540051 wait complete. 2024-08-03T20:53:26.300 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:26.299+0000 7f0e91400640 1 Processor -- start 2024-08-03T20:53:26.301 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:26.299+0000 7f0e91400640 1 -- start start 2024-08-03T20:53:26.303 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:26.301+0000 7f0e91400640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f0e70004d40 0x7f0e8c159940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:53:26.304 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:26.301+0000 7f0e91400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0e8c0740d0 0x7f0e8c154a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:53:26.304 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:26.301+0000 7f0e91400640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f0e8c0744e0 0x7f0e8c154f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:53:26.304 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:26.301+0000 7f0e91400640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f0e8c15d850 con 0x7f0e8c0740d0 2024-08-03T20:53:26.304 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:26.301+0000 7f0e91400640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f0e8c0ccc80 con 0x7f0e8c0744e0 2024-08-03T20:53:26.304 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:26.301+0000 7f0e91400640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7f0e8c0cce00 con 0x7f0e70004d40 2024-08-03T20:53:26.304 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:26.301+0000 7f0e89e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0e8c0740d0 0x7f0e8c154a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:53:26.304 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:26.301+0000 7f0e89e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0e8c0740d0 0x7f0e8c154a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:45700/0 (socket says 172.21.15.32:45700) 2024-08-03T20:53:26.304 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:26.301+0000 7f0e89e00640 1 -- 172.21.15.32:0/2602702083 learned_addr learned my addr 172.21.15.32:0/2602702083 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:53:26.304 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:26.301+0000 7f0e89e00640 1 -- 172.21.15.32:0/2602702083 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f0e70004d40 msgr2=0x7f0e8c159940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:53:26.304 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:26.301+0000 7f0e89e00640 1 --2- 172.21.15.32:0/2602702083 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f0e70004d40 0x7f0e8c159940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:53:26.304 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:26.301+0000 7f0e89e00640 1 -- 172.21.15.32:0/2602702083 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f0e8c0744e0 msgr2=0x7f0e8c154f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:53:26.304 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:26.301+0000 7f0e8b200640 1 --2- 172.21.15.32:0/2602702083 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f0e8c0744e0 0x7f0e8c154f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:53:26.304 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:26.301+0000 7f0e89e00640 1 --2- 172.21.15.32:0/2602702083 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f0e8c0744e0 0x7f0e8c154f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:53:26.305 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:26.301+0000 7f0e89e00640 1 -- 172.21.15.32:0/2602702083 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0e8c0c9de0 con 0x7f0e8c0740d0 2024-08-03T20:53:26.305 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:26.301+0000 7f0e8b200640 1 --2- 172.21.15.32:0/2602702083 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f0e8c0744e0 0x7f0e8c154f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-03T20:53:26.305 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:26.302+0000 7f0e89e00640 1 --2- 172.21.15.32:0/2602702083 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0e8c0740d0 0x7f0e8c154a00 secure :-1 s=READY pgs=791 cs=0 l=1 rev1=1 crypto rx=0x7f0e78006630 tx=0x7f0e780039c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:53:26.305 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:26.302+0000 7f0e82a00640 1 -- 172.21.15.32:0/2602702083 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e78015070 con 0x7f0e8c0740d0 2024-08-03T20:53:26.305 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:26.302+0000 7f0e82a00640 1 -- 172.21.15.32:0/2602702083 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f0e78011900 con 0x7f0e8c0740d0 2024-08-03T20:53:26.305 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:26.302+0000 7f0e91400640 1 -- 172.21.15.32:0/2602702083 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0e8c0ca0c0 con 0x7f0e8c0740d0 2024-08-03T20:53:26.305 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:26.302+0000 7f0e82a00640 1 -- 172.21.15.32:0/2602702083 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e78011d50 con 0x7f0e8c0740d0 2024-08-03T20:53:26.305 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:26.302+0000 7f0e91400640 1 -- 172.21.15.32:0/2602702083 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0e8c0ca680 con 0x7f0e8c0740d0 2024-08-03T20:53:26.306 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:26.303+0000 7f0e91400640 1 -- 172.21.15.32:0/2602702083 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0e8c0cd360 con 0x7f0e8c0740d0 2024-08-03T20:53:26.306 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:26.304+0000 7f0e82a00640 1 -- 172.21.15.32:0/2602702083 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f0e78018070 con 0x7f0e8c0740d0 2024-08-03T20:53:26.307 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:26.305+0000 7f0e82a00640 1 --2- 172.21.15.32:0/2602702083 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f0e6c077040 0x7f0e6c079550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:53:26.307 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:26.305+0000 7f0e82a00640 1 -- 172.21.15.32:0/2602702083 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(70..70 src has 1..70) ==== 5709+0+0 (secure 0 0 0) 0x7f0e78099580 con 0x7f0e8c0740d0 2024-08-03T20:53:26.307 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:26.306+0000 7f0e8a800640 1 --2- 172.21.15.32:0/2602702083 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f0e6c077040 0x7f0e6c079550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:53:26.308 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:26.306+0000 7f0e8a800640 1 --2- 172.21.15.32:0/2602702083 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f0e6c077040 0x7f0e6c079550 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f0e7c0064b0 tx=0x7f0e7c006b20 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:53:26.311 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:26.310+0000 7f0e82a00640 1 -- 172.21.15.32:0/2602702083 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7f0e78062780 con 0x7f0e8c0740d0 2024-08-03T20:53:26.519 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:26.517+0000 7f0e91400640 1 -- 172.21.15.32:0/2602702083 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command([{prefix=config set, name=mds_inject_rename_corrupt_dentry_first}] v 0) -- 0x7f0e8c0b33c0 con 0x7f0e8c0740d0 2024-08-03T20:53:26.523 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:26.521+0000 7f0e82a00640 1 -- 172.21.15.32:0/2602702083 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mds_inject_rename_corrupt_dentry_first}]=0 v40)=0 v40) ==== 138+0+0 (secure 0 0 0) 0x7f0e78062080 con 0x7f0e8c0740d0 2024-08-03T20:53:26.526 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:26.524+0000 7f0e91400640 1 -- 172.21.15.32:0/2602702083 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f0e6c077040 msgr2=0x7f0e6c079550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:53:26.526 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:26.524+0000 7f0e91400640 1 --2- 172.21.15.32:0/2602702083 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f0e6c077040 0x7f0e6c079550 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f0e7c0064b0 tx=0x7f0e7c006b20 comp rx=0 tx=0).stop 2024-08-03T20:53:26.526 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:26.524+0000 7f0e91400640 1 -- 172.21.15.32:0/2602702083 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0e8c0740d0 msgr2=0x7f0e8c154a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:53:26.526 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:26.524+0000 7f0e91400640 1 --2- 172.21.15.32:0/2602702083 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0e8c0740d0 0x7f0e8c154a00 secure :-1 s=READY pgs=791 cs=0 l=1 rev1=1 crypto rx=0x7f0e78006630 tx=0x7f0e780039c0 comp rx=0 tx=0).stop 2024-08-03T20:53:26.526 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:26.524+0000 7f0e91400640 1 -- 172.21.15.32:0/2602702083 shutdown_connections 2024-08-03T20:53:26.527 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:26.524+0000 7f0e91400640 1 --2- 172.21.15.32:0/2602702083 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f0e6c077040 0x7f0e6c079550 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:53:26.527 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:26.524+0000 7f0e91400640 1 --2- 172.21.15.32:0/2602702083 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f0e8c0744e0 0x7f0e8c154f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:53:26.527 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:26.524+0000 7f0e91400640 1 --2- 172.21.15.32:0/2602702083 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0e8c0740d0 0x7f0e8c154a00 unknown :-1 s=CLOSED pgs=791 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:53:26.527 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:26.524+0000 7f0e91400640 1 --2- 172.21.15.32:0/2602702083 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f0e70004d40 0x7f0e8c159940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:53:26.527 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:26.524+0000 7f0e91400640 1 -- 172.21.15.32:0/2602702083 >> 172.21.15.32:0/2602702083 conn(0x7f0e8c0b1bd0 msgr2=0x7f0e8c0cc0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:53:26.527 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:26.524+0000 7f0e91400640 1 -- 172.21.15.32:0/2602702083 shutdown_connections 2024-08-03T20:53:26.527 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:26.524+0000 7f0e91400640 1 -- 172.21.15.32:0/2602702083 wait complete. 2024-08-03T20:53:31.549 DEBUG:teuthology.orchestra.run.smithi032:> sudo daemon-helper kill ceph --cluster ceph -w 2024-08-03T20:53:31.554 DEBUG:teuthology.orchestra.run.smithi032:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'timeout 60 mv a/b a/z') 2024-08-03T20:53:31.558 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mon.a.asok --format=json config get mds_beacon_grace 2024-08-03T20:53:31.659 INFO:tasks.ceph.mds.a.smithi032.stderr:2024-08-03T20:53:31.657+0000 7f078fc00640 -1 mds.0.cache.den(0x10000000000 z) newly corrupt dentry to be committed: [dentry #0x1/a/z [fffffffffffffff6,head] auth NULL (dn xlock x=1 by request(client.5762:33 nref=7 cr=0x562158b7f500)) (dversion lock w=1 last_client=5762) pv=11 v=10 ap=2 ino=(nil) state=1073741824 | request=1 lock=2 authpin=1 0x562158c00780] 2024-08-03T20:53:31.659 INFO:tasks.ceph.mds.a.smithi032.stderr:2024-08-03T20:53:31.657+0000 7f078fc00640 -1 log_channel(cluster) log [ERR] : MDS abort because newly corrupt dentry to be committed: [dentry #0x1/a/z [fffffffffffffff6,head] auth NULL (dn xlock x=1 by request(client.5762:33 nref=7 cr=0x562158b7f500)) (dversion lock w=1 last_client=5762) pv=11 v=10 ap=2 ino=(nil) state=1073741824 | request=1 lock=2 authpin=1 0x562158c00780] 2024-08-03T20:53:31.659 INFO:tasks.ceph.mds.a.smithi032.stderr:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-5476-g3792a0a4/rpm/el9/BUILD/ceph-19.0.0-5476-g3792a0a4/src/mds/MDSRank.cc: In function 'void MDSRank::abort(std::string_view)' thread 7f078fc00640 time 2024-08-03T20:53:31.658596+0000 2024-08-03T20:53:31.660 INFO:tasks.ceph.mds.a.smithi032.stderr:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-5476-g3792a0a4/rpm/el9/BUILD/ceph-19.0.0-5476-g3792a0a4/src/mds/MDSRank.cc: 952: ceph_abort_msg("abort() called") 2024-08-03T20:53:31.660 INFO:tasks.ceph.mds.a.smithi032.stderr: ceph version 19.0.0-5476-g3792a0a4 (3792a0a4a40f645dee271188eab388b1c22c10f8) squid (dev) 2024-08-03T20:53:31.660 INFO:tasks.ceph.mds.a.smithi032.stderr: 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xcf) [0x7f0796c55480] 2024-08-03T20:53:31.660 INFO:tasks.ceph.mds.a.smithi032.stderr: 2: (MDSRank::abort(std::basic_string_view >)+0x5f) [0x562125c76297] 2024-08-03T20:53:31.660 INFO:tasks.ceph.mds.a.smithi032.stderr: 3: (CDentry::check_corruption(bool)+0x530) [0x562125f44a32] 2024-08-03T20:53:31.660 INFO:tasks.ceph.mds.a.smithi032.stderr: 4: ceph-mds(+0x173a38) [0x562125cdea38] 2024-08-03T20:53:31.660 INFO:tasks.ceph.mds.a.smithi032.stderr: 5: ceph-mds(+0x173e38) [0x562125cdee38] 2024-08-03T20:53:31.660 INFO:tasks.ceph.mds.a.smithi032.stderr: 6: (Server::_rename_prepare(boost::intrusive_ptr const&, EMetaBlob*, ceph::buffer::v15_2_0::list*, CDentry*, CDentry*, std::basic_string_view >, CDentry*)+0x249b) [0x562125d8264d] 2024-08-03T20:53:31.660 INFO:tasks.ceph.mds.a.smithi032.stderr: 7: (Server::handle_client_rename(boost::intrusive_ptr const&)+0x2ec1) [0x562125d85e0f] 2024-08-03T20:53:31.660 INFO:tasks.ceph.mds.a.smithi032.stderr: 8: (Server::dispatch_client_request(boost::intrusive_ptr const&)+0xd27) [0x562125d4a795] 2024-08-03T20:53:31.660 INFO:tasks.ceph.mds.a.smithi032.stderr: 9: (Server::handle_client_request(boost::intrusive_ptr const&)+0xa2a) [0x562125d4b442] 2024-08-03T20:53:31.661 INFO:tasks.ceph.mds.a.smithi032.stderr: 10: (Server::dispatch(boost::intrusive_ptr const&)+0xc0) [0x562125d31e3c] 2024-08-03T20:53:31.661 INFO:tasks.ceph.mds.a.smithi032.stderr: 11: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x5ac) [0x562125c8073e] 2024-08-03T20:53:31.661 INFO:tasks.ceph.mds.a.smithi032.stderr: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x562125c8458d] 2024-08-03T20:53:31.661 INFO:tasks.ceph.mds.a.smithi032.stderr: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x562125c84eb4] 2024-08-03T20:53:31.661 INFO:tasks.ceph.mds.a.smithi032.stderr: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x562125c6800a] 2024-08-03T20:53:31.661 INFO:tasks.ceph.mds.a.smithi032.stderr: 15: /usr/lib64/ceph/libceph-common.so.2(+0x3a3e77) [0x7f0796da3e77] 2024-08-03T20:53:31.661 INFO:tasks.ceph.mds.a.smithi032.stderr: 16: (DispatchQueue::entry()+0x60f) [0x7f0796da4769] 2024-08-03T20:53:31.661 INFO:tasks.ceph.mds.a.smithi032.stderr: 17: /usr/lib64/ceph/libceph-common.so.2(+0x49091d) [0x7f0796e9091d] 2024-08-03T20:53:31.661 INFO:tasks.ceph.mds.a.smithi032.stderr: 18: (Thread::entry_wrapper()+0x43) [0x7f0796c2df4f] 2024-08-03T20:53:31.661 INFO:tasks.ceph.mds.a.smithi032.stderr: 19: (Thread::_entry_func(void*)+0xd) [0x7f0796c2df6b] 2024-08-03T20:53:31.661 INFO:tasks.ceph.mds.a.smithi032.stderr: 20: /lib64/libc.so.6(+0x89c02) [0x7f0796289c02] 2024-08-03T20:53:31.661 INFO:tasks.ceph.mds.a.smithi032.stderr: 21: /lib64/libc.so.6(+0x10ec40) [0x7f079630ec40] 2024-08-03T20:53:31.661 INFO:tasks.ceph.mds.a.smithi032.stderr:2024-08-03T20:53:31.658+0000 7f078fc00640 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-5476-g3792a0a4/rpm/el9/BUILD/ceph-19.0.0-5476-g3792a0a4/src/mds/MDSRank.cc: In function 'void MDSRank::abort(std::string_view)' thread 7f078fc00640 time 2024-08-03T20:53:31.658596+0000 2024-08-03T20:53:31.662 INFO:tasks.ceph.mds.a.smithi032.stderr:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-5476-g3792a0a4/rpm/el9/BUILD/ceph-19.0.0-5476-g3792a0a4/src/mds/MDSRank.cc: 952: ceph_abort_msg("abort() called") 2024-08-03T20:53:31.662 INFO:tasks.ceph.mds.a.smithi032.stderr: 2024-08-03T20:53:31.662 INFO:tasks.ceph.mds.a.smithi032.stderr: ceph version 19.0.0-5476-g3792a0a4 (3792a0a4a40f645dee271188eab388b1c22c10f8) squid (dev) 2024-08-03T20:53:31.662 INFO:tasks.ceph.mds.a.smithi032.stderr: 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xcf) [0x7f0796c55480] 2024-08-03T20:53:31.662 INFO:tasks.ceph.mds.a.smithi032.stderr: 2: (MDSRank::abort(std::basic_string_view >)+0x5f) [0x562125c76297] 2024-08-03T20:53:31.662 INFO:tasks.ceph.mds.a.smithi032.stderr: 3: (CDentry::check_corruption(bool)+0x530) [0x562125f44a32] 2024-08-03T20:53:31.662 INFO:tasks.ceph.mds.a.smithi032.stderr: 4: ceph-mds(+0x173a38) [0x562125cdea38] 2024-08-03T20:53:31.662 INFO:tasks.ceph.mds.a.smithi032.stderr: 5: ceph-mds(+0x173e38) [0x562125cdee38] 2024-08-03T20:53:31.662 INFO:tasks.ceph.mds.a.smithi032.stderr: 6: (Server::_rename_prepare(boost::intrusive_ptr const&, EMetaBlob*, ceph::buffer::v15_2_0::list*, CDentry*, CDentry*, std::basic_string_view >, CDentry*)+0x249b) [0x562125d8264d] 2024-08-03T20:53:31.662 INFO:tasks.ceph.mds.a.smithi032.stderr: 7: (Server::handle_client_rename(boost::intrusive_ptr const&)+0x2ec1) [0x562125d85e0f] 2024-08-03T20:53:31.662 INFO:tasks.ceph.mds.a.smithi032.stderr: 8: (Server::dispatch_client_request(boost::intrusive_ptr const&)+0xd27) [0x562125d4a795] 2024-08-03T20:53:31.662 INFO:tasks.ceph.mds.a.smithi032.stderr: 9: (Server::handle_client_request(boost::intrusive_ptr const&)+0xa2a) [0x562125d4b442] 2024-08-03T20:53:31.662 INFO:tasks.ceph.mds.a.smithi032.stderr: 10: (Server::dispatch(boost::intrusive_ptr const&)+0xc0) [0x562125d31e3c] 2024-08-03T20:53:31.663 INFO:tasks.ceph.mds.a.smithi032.stderr: 11: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x5ac) [0x562125c8073e] 2024-08-03T20:53:31.663 INFO:tasks.ceph.mds.a.smithi032.stderr: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x562125c8458d] 2024-08-03T20:53:31.663 INFO:tasks.ceph.mds.a.smithi032.stderr: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x562125c84eb4] 2024-08-03T20:53:31.663 INFO:tasks.ceph.mds.a.smithi032.stderr: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x562125c6800a] 2024-08-03T20:53:31.663 INFO:tasks.ceph.mds.a.smithi032.stderr: 15: /usr/lib64/ceph/libceph-common.so.2(+0x3a3e77) [0x7f0796da3e77] 2024-08-03T20:53:31.663 INFO:tasks.ceph.mds.a.smithi032.stderr: 16: (DispatchQueue::entry()+0x60f) [0x7f0796da4769] 2024-08-03T20:53:31.663 INFO:tasks.ceph.mds.a.smithi032.stderr: 17: /usr/lib64/ceph/libceph-common.so.2(+0x49091d) [0x7f0796e9091d] 2024-08-03T20:53:31.663 INFO:tasks.ceph.mds.a.smithi032.stderr: 18: (Thread::entry_wrapper()+0x43) [0x7f0796c2df4f] 2024-08-03T20:53:31.663 INFO:tasks.ceph.mds.a.smithi032.stderr: 19: (Thread::_entry_func(void*)+0xd) [0x7f0796c2df6b] 2024-08-03T20:53:31.664 INFO:tasks.ceph.mds.a.smithi032.stderr: 20: /lib64/libc.so.6(+0x89c02) [0x7f0796289c02] 2024-08-03T20:53:31.664 INFO:tasks.ceph.mds.a.smithi032.stderr: 21: /lib64/libc.so.6(+0x10ec40) [0x7f079630ec40] 2024-08-03T20:53:31.664 INFO:tasks.ceph.mds.a.smithi032.stderr: 2024-08-03T20:53:31.664 INFO:tasks.ceph.mds.a.smithi032.stderr:*** Caught signal (Aborted) ** 2024-08-03T20:53:31.664 INFO:tasks.ceph.mds.a.smithi032.stderr: in thread 7f078fc00640 thread_name:ms_dispatch 2024-08-03T20:53:31.665 INFO:tasks.ceph.mds.a.smithi032.stderr: ceph version 19.0.0-5476-g3792a0a4 (3792a0a4a40f645dee271188eab388b1c22c10f8) squid (dev) 2024-08-03T20:53:31.665 INFO:tasks.ceph.mds.a.smithi032.stderr: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f079623e6f0] 2024-08-03T20:53:31.666 INFO:tasks.ceph.mds.a.smithi032.stderr: 2: /lib64/libc.so.6(+0x8b94c) [0x7f079628b94c] 2024-08-03T20:53:31.666 INFO:tasks.ceph.mds.a.smithi032.stderr: 3: raise() 2024-08-03T20:53:31.666 INFO:tasks.ceph.mds.a.smithi032.stderr: 4: abort() 2024-08-03T20:53:31.666 INFO:tasks.ceph.mds.a.smithi032.stderr: 5: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0x241) [0x7f0796c555f2] 2024-08-03T20:53:31.666 INFO:tasks.ceph.mds.a.smithi032.stderr: 6: (MDSRank::abort(std::basic_string_view >)+0x5f) [0x562125c76297] 2024-08-03T20:53:31.666 INFO:tasks.ceph.mds.a.smithi032.stderr: 7: (CDentry::check_corruption(bool)+0x530) [0x562125f44a32] 2024-08-03T20:53:31.666 INFO:tasks.ceph.mds.a.smithi032.stderr: 8: ceph-mds(+0x173a38) [0x562125cdea38] 2024-08-03T20:53:31.666 INFO:tasks.ceph.mds.a.smithi032.stderr: 9: ceph-mds(+0x173e38) [0x562125cdee38] 2024-08-03T20:53:31.666 INFO:tasks.ceph.mds.a.smithi032.stderr: 10: (Server::_rename_prepare(boost::intrusive_ptr const&, EMetaBlob*, ceph::buffer::v15_2_0::list*, CDentry*, CDentry*, std::basic_string_view >, CDentry*)+0x249b) [0x562125d8264d] 2024-08-03T20:53:31.666 INFO:tasks.ceph.mds.a.smithi032.stderr: 11: (Server::handle_client_rename(boost::intrusive_ptr const&)+0x2ec1) [0x562125d85e0f] 2024-08-03T20:53:31.666 INFO:tasks.ceph.mds.a.smithi032.stderr: 12: (Server::dispatch_client_request(boost::intrusive_ptr const&)+0xd27) [0x562125d4a795] 2024-08-03T20:53:31.666 INFO:tasks.ceph.mds.a.smithi032.stderr: 13: (Server::handle_client_request(boost::intrusive_ptr const&)+0xa2a) [0x562125d4b442] 2024-08-03T20:53:31.666 INFO:tasks.ceph.mds.a.smithi032.stderr: 14: (Server::dispatch(boost::intrusive_ptr const&)+0xc0) [0x562125d31e3c] 2024-08-03T20:53:31.666 INFO:tasks.ceph.mds.a.smithi032.stderr: 15: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x5ac) [0x562125c8073e] 2024-08-03T20:53:31.667 INFO:tasks.ceph.mds.a.smithi032.stderr: 16: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x562125c8458d] 2024-08-03T20:53:31.667 INFO:tasks.ceph.mds.a.smithi032.stderr: 17: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x562125c84eb4] 2024-08-03T20:53:31.667 INFO:tasks.ceph.mds.a.smithi032.stderr: 18: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x562125c6800a] 2024-08-03T20:53:31.667 INFO:tasks.ceph.mds.a.smithi032.stderr: 19: /usr/lib64/ceph/libceph-common.so.2(+0x3a3e77) [0x7f0796da3e77] 2024-08-03T20:53:31.667 INFO:tasks.ceph.mds.a.smithi032.stderr: 20: (DispatchQueue::entry()+0x60f) [0x7f0796da4769] 2024-08-03T20:53:31.667 INFO:tasks.ceph.mds.a.smithi032.stderr: 21: /usr/lib64/ceph/libceph-common.so.2(+0x49091d) [0x7f0796e9091d] 2024-08-03T20:53:31.668 INFO:tasks.ceph.mds.a.smithi032.stderr: 22: (Thread::entry_wrapper()+0x43) [0x7f0796c2df4f] 2024-08-03T20:53:31.668 INFO:tasks.ceph.mds.a.smithi032.stderr: 23: (Thread::_entry_func(void*)+0xd) [0x7f0796c2df6b] 2024-08-03T20:53:31.668 INFO:tasks.ceph.mds.a.smithi032.stderr: 24: /lib64/libc.so.6(+0x89c02) [0x7f0796289c02] 2024-08-03T20:53:31.668 INFO:tasks.ceph.mds.a.smithi032.stderr: 25: /lib64/libc.so.6(+0x10ec40) [0x7f079630ec40] 2024-08-03T20:53:31.668 INFO:tasks.ceph.mds.a.smithi032.stderr:2024-08-03T20:53:31.659+0000 7f078fc00640 -1 *** Caught signal (Aborted) ** 2024-08-03T20:53:31.668 INFO:tasks.ceph.mds.a.smithi032.stderr: in thread 7f078fc00640 thread_name:ms_dispatch 2024-08-03T20:53:31.668 INFO:tasks.ceph.mds.a.smithi032.stderr: 2024-08-03T20:53:31.668 INFO:tasks.ceph.mds.a.smithi032.stderr: ceph version 19.0.0-5476-g3792a0a4 (3792a0a4a40f645dee271188eab388b1c22c10f8) squid (dev) 2024-08-03T20:53:31.668 INFO:tasks.ceph.mds.a.smithi032.stderr: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f079623e6f0] 2024-08-03T20:53:31.668 INFO:tasks.ceph.mds.a.smithi032.stderr: 2: /lib64/libc.so.6(+0x8b94c) [0x7f079628b94c] 2024-08-03T20:53:31.668 INFO:tasks.ceph.mds.a.smithi032.stderr: 3: raise() 2024-08-03T20:53:31.668 INFO:tasks.ceph.mds.a.smithi032.stderr: 4: abort() 2024-08-03T20:53:31.669 INFO:tasks.ceph.mds.a.smithi032.stderr: 5: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0x241) [0x7f0796c555f2] 2024-08-03T20:53:31.669 INFO:tasks.ceph.mds.a.smithi032.stderr: 6: (MDSRank::abort(std::basic_string_view >)+0x5f) [0x562125c76297] 2024-08-03T20:53:31.669 INFO:tasks.ceph.mds.a.smithi032.stderr: 7: (CDentry::check_corruption(bool)+0x530) [0x562125f44a32] 2024-08-03T20:53:31.669 INFO:tasks.ceph.mds.a.smithi032.stderr: 8: ceph-mds(+0x173a38) [0x562125cdea38] 2024-08-03T20:53:31.669 INFO:tasks.ceph.mds.a.smithi032.stderr: 9: ceph-mds(+0x173e38) [0x562125cdee38] 2024-08-03T20:53:31.669 INFO:tasks.ceph.mds.a.smithi032.stderr: 10: (Server::_rename_prepare(boost::intrusive_ptr const&, EMetaBlob*, ceph::buffer::v15_2_0::list*, CDentry*, CDentry*, std::basic_string_view >, CDentry*)+0x249b) [0x562125d8264d] 2024-08-03T20:53:31.669 INFO:tasks.ceph.mds.a.smithi032.stderr: 11: (Server::handle_client_rename(boost::intrusive_ptr const&)+0x2ec1) [0x562125d85e0f] 2024-08-03T20:53:31.669 INFO:tasks.ceph.mds.a.smithi032.stderr: 12: (Server::dispatch_client_request(boost::intrusive_ptr const&)+0xd27) [0x562125d4a795] 2024-08-03T20:53:31.669 INFO:tasks.ceph.mds.a.smithi032.stderr: 13: (Server::handle_client_request(boost::intrusive_ptr const&)+0xa2a) [0x562125d4b442] 2024-08-03T20:53:31.669 INFO:tasks.ceph.mds.a.smithi032.stderr: 14: (Server::dispatch(boost::intrusive_ptr const&)+0xc0) [0x562125d31e3c] 2024-08-03T20:53:31.669 INFO:tasks.ceph.mds.a.smithi032.stderr: 15: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x5ac) [0x562125c8073e] 2024-08-03T20:53:31.669 INFO:tasks.ceph.mds.a.smithi032.stderr: 16: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x562125c8458d] 2024-08-03T20:53:31.669 INFO:tasks.ceph.mds.a.smithi032.stderr: 17: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x562125c84eb4] 2024-08-03T20:53:31.670 INFO:tasks.ceph.mds.a.smithi032.stderr: 18: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x562125c6800a] 2024-08-03T20:53:31.670 INFO:tasks.ceph.mds.a.smithi032.stderr: 19: /usr/lib64/ceph/libceph-common.so.2(+0x3a3e77) [0x7f0796da3e77] 2024-08-03T20:53:31.670 INFO:tasks.ceph.mds.a.smithi032.stderr: 20: (DispatchQueue::entry()+0x60f) [0x7f0796da4769] 2024-08-03T20:53:31.670 INFO:tasks.ceph.mds.a.smithi032.stderr: 21: /usr/lib64/ceph/libceph-common.so.2(+0x49091d) [0x7f0796e9091d] 2024-08-03T20:53:31.670 INFO:tasks.ceph.mds.a.smithi032.stderr: 22: (Thread::entry_wrapper()+0x43) [0x7f0796c2df4f] 2024-08-03T20:53:31.670 INFO:tasks.ceph.mds.a.smithi032.stderr: 23: (Thread::_entry_func(void*)+0xd) [0x7f0796c2df6b] 2024-08-03T20:53:31.670 INFO:tasks.ceph.mds.a.smithi032.stderr: 24: /lib64/libc.so.6(+0x89c02) [0x7f0796289c02] 2024-08-03T20:53:31.670 INFO:tasks.ceph.mds.a.smithi032.stderr: 25: /lib64/libc.so.6(+0x10ec40) [0x7f079630ec40] 2024-08-03T20:53:31.670 INFO:tasks.ceph.mds.a.smithi032.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-08-03T20:53:31.670 INFO:tasks.ceph.mds.a.smithi032.stderr: 2024-08-03T20:53:31.671 INFO:tasks.ceph.mds.a.smithi032.stderr: -4303> 2024-08-03T20:53:17.531+0000 7f078ca00640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-08-03T20:53:31.679 INFO:tasks.ceph.mds.a.smithi032.stderr: -9> 2024-08-03T20:53:31.657+0000 7f078fc00640 -1 mds.0.cache.den(0x10000000000 z) newly corrupt dentry to be committed: [dentry #0x1/a/z [fffffffffffffff6,head] auth NULL (dn xlock x=1 by request(client.5762:33 nref=7 cr=0x562158b7f500)) (dversion lock w=1 last_client=5762) pv=11 v=10 ap=2 ino=(nil) state=1073741824 | request=1 lock=2 authpin=1 0x562158c00780] 2024-08-03T20:53:31.679 INFO:tasks.ceph.mds.a.smithi032.stderr: -7> 2024-08-03T20:53:31.657+0000 7f078fc00640 -1 log_channel(cluster) log [ERR] : MDS abort because newly corrupt dentry to be committed: [dentry #0x1/a/z [fffffffffffffff6,head] auth NULL (dn xlock x=1 by request(client.5762:33 nref=7 cr=0x562158b7f500)) (dversion lock w=1 last_client=5762) pv=11 v=10 ap=2 ino=(nil) state=1073741824 | request=1 lock=2 authpin=1 0x562158c00780] 2024-08-03T20:53:31.679 INFO:tasks.ceph.mds.a.smithi032.stderr: -1> 2024-08-03T20:53:31.658+0000 7f078fc00640 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-5476-g3792a0a4/rpm/el9/BUILD/ceph-19.0.0-5476-g3792a0a4/src/mds/MDSRank.cc: In function 'void MDSRank::abort(std::string_view)' thread 7f078fc00640 time 2024-08-03T20:53:31.658596+0000 2024-08-03T20:53:31.680 INFO:tasks.ceph.mds.a.smithi032.stderr:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-5476-g3792a0a4/rpm/el9/BUILD/ceph-19.0.0-5476-g3792a0a4/src/mds/MDSRank.cc: 952: ceph_abort_msg("abort() called") 2024-08-03T20:53:31.680 INFO:tasks.ceph.mds.a.smithi032.stderr: 2024-08-03T20:53:31.680 INFO:tasks.ceph.mds.a.smithi032.stderr: ceph version 19.0.0-5476-g3792a0a4 (3792a0a4a40f645dee271188eab388b1c22c10f8) squid (dev) 2024-08-03T20:53:31.680 INFO:tasks.ceph.mds.a.smithi032.stderr: 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xcf) [0x7f0796c55480] 2024-08-03T20:53:31.680 INFO:tasks.ceph.mds.a.smithi032.stderr: 2: (MDSRank::abort(std::basic_string_view >)+0x5f) [0x562125c76297] 2024-08-03T20:53:31.681 INFO:tasks.ceph.mds.a.smithi032.stderr: 3: (CDentry::check_corruption(bool)+0x530) [0x562125f44a32] 2024-08-03T20:53:31.681 INFO:tasks.ceph.mds.a.smithi032.stderr: 4: ceph-mds(+0x173a38) [0x562125cdea38] 2024-08-03T20:53:31.681 INFO:tasks.ceph.mds.a.smithi032.stderr: 5: ceph-mds(+0x173e38) [0x562125cdee38] 2024-08-03T20:53:31.682 INFO:tasks.ceph.mds.a.smithi032.stderr: 6: (Server::_rename_prepare(boost::intrusive_ptr const&, EMetaBlob*, ceph::buffer::v15_2_0::list*, CDentry*, CDentry*, std::basic_string_view >, CDentry*)+0x249b) [0x562125d8264d] 2024-08-03T20:53:31.682 INFO:tasks.ceph.mds.a.smithi032.stderr: 7: (Server::handle_client_rename(boost::intrusive_ptr const&)+0x2ec1) [0x562125d85e0f] 2024-08-03T20:53:31.682 INFO:tasks.ceph.mds.a.smithi032.stderr: 8: (Server::dispatch_client_request(boost::intrusive_ptr const&)+0xd27) [0x562125d4a795] 2024-08-03T20:53:31.682 INFO:tasks.ceph.mds.a.smithi032.stderr: 9: (Server::handle_client_request(boost::intrusive_ptr const&)+0xa2a) [0x562125d4b442] 2024-08-03T20:53:31.682 INFO:tasks.ceph.mds.a.smithi032.stderr: 10: (Server::dispatch(boost::intrusive_ptr const&)+0xc0) [0x562125d31e3c] 2024-08-03T20:53:31.682 INFO:tasks.ceph.mds.a.smithi032.stderr: 11: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x5ac) [0x562125c8073e] 2024-08-03T20:53:31.683 INFO:tasks.ceph.mds.a.smithi032.stderr: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x562125c8458d] 2024-08-03T20:53:31.683 INFO:tasks.ceph.mds.a.smithi032.stderr: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x562125c84eb4] 2024-08-03T20:53:31.683 INFO:tasks.ceph.mds.a.smithi032.stderr: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x562125c6800a] 2024-08-03T20:53:31.683 INFO:tasks.ceph.mds.a.smithi032.stderr: 15: /usr/lib64/ceph/libceph-common.so.2(+0x3a3e77) [0x7f0796da3e77] 2024-08-03T20:53:31.683 INFO:tasks.ceph.mds.a.smithi032.stderr: 16: (DispatchQueue::entry()+0x60f) [0x7f0796da4769] 2024-08-03T20:53:31.684 INFO:tasks.ceph.mds.a.smithi032.stderr: 17: /usr/lib64/ceph/libceph-common.so.2(+0x49091d) [0x7f0796e9091d] 2024-08-03T20:53:31.684 INFO:tasks.ceph.mds.a.smithi032.stderr: 18: (Thread::entry_wrapper()+0x43) [0x7f0796c2df4f] 2024-08-03T20:53:31.684 INFO:tasks.ceph.mds.a.smithi032.stderr: 19: (Thread::_entry_func(void*)+0xd) [0x7f0796c2df6b] 2024-08-03T20:53:31.685 INFO:tasks.ceph.mds.a.smithi032.stderr: 20: /lib64/libc.so.6(+0x89c02) [0x7f0796289c02] 2024-08-03T20:53:31.685 INFO:tasks.ceph.mds.a.smithi032.stderr: 21: /lib64/libc.so.6(+0x10ec40) [0x7f079630ec40] 2024-08-03T20:53:31.685 INFO:tasks.ceph.mds.a.smithi032.stderr: 2024-08-03T20:53:31.685 INFO:tasks.ceph.mds.a.smithi032.stderr: 0> 2024-08-03T20:53:31.659+0000 7f078fc00640 -1 *** Caught signal (Aborted) ** 2024-08-03T20:53:31.685 INFO:tasks.ceph.mds.a.smithi032.stderr: in thread 7f078fc00640 thread_name:ms_dispatch 2024-08-03T20:53:31.685 INFO:tasks.ceph.mds.a.smithi032.stderr: 2024-08-03T20:53:31.685 INFO:tasks.ceph.mds.a.smithi032.stderr: ceph version 19.0.0-5476-g3792a0a4 (3792a0a4a40f645dee271188eab388b1c22c10f8) squid (dev) 2024-08-03T20:53:31.685 INFO:tasks.ceph.mds.a.smithi032.stderr: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f079623e6f0] 2024-08-03T20:53:31.685 INFO:tasks.ceph.mds.a.smithi032.stderr: 2: /lib64/libc.so.6(+0x8b94c) [0x7f079628b94c] 2024-08-03T20:53:31.685 INFO:tasks.ceph.mds.a.smithi032.stderr: 3: raise() 2024-08-03T20:53:31.685 INFO:tasks.ceph.mds.a.smithi032.stderr: 4: abort() 2024-08-03T20:53:31.685 INFO:tasks.ceph.mds.a.smithi032.stderr: 5: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0x241) [0x7f0796c555f2] 2024-08-03T20:53:31.685 INFO:tasks.ceph.mds.a.smithi032.stderr: 6: (MDSRank::abort(std::basic_string_view >)+0x5f) [0x562125c76297] 2024-08-03T20:53:31.685 INFO:tasks.ceph.mds.a.smithi032.stderr: 7: (CDentry::check_corruption(bool)+0x530) [0x562125f44a32] 2024-08-03T20:53:31.686 INFO:tasks.ceph.mds.a.smithi032.stderr: 8: ceph-mds(+0x173a38) [0x562125cdea38] 2024-08-03T20:53:31.686 INFO:tasks.ceph.mds.a.smithi032.stderr: 9: ceph-mds(+0x173e38) [0x562125cdee38] 2024-08-03T20:53:31.686 INFO:tasks.ceph.mds.a.smithi032.stderr: 10: (Server::_rename_prepare(boost::intrusive_ptr const&, EMetaBlob*, ceph::buffer::v15_2_0::list*, CDentry*, CDentry*, std::basic_string_view >, CDentry*)+0x249b) [0x562125d8264d] 2024-08-03T20:53:31.686 INFO:tasks.ceph.mds.a.smithi032.stderr: 11: (Server::handle_client_rename(boost::intrusive_ptr const&)+0x2ec1) [0x562125d85e0f] 2024-08-03T20:53:31.686 INFO:tasks.ceph.mds.a.smithi032.stderr: 12: (Server::dispatch_client_request(boost::intrusive_ptr const&)+0xd27) [0x562125d4a795] 2024-08-03T20:53:31.686 INFO:tasks.ceph.mds.a.smithi032.stderr: 13: (Server::handle_client_request(boost::intrusive_ptr const&)+0xa2a) [0x562125d4b442] 2024-08-03T20:53:31.686 INFO:tasks.ceph.mds.a.smithi032.stderr: 14: (Server::dispatch(boost::intrusive_ptr const&)+0xc0) [0x562125d31e3c] 2024-08-03T20:53:31.686 INFO:tasks.ceph.mds.a.smithi032.stderr: 15: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x5ac) [0x562125c8073e] 2024-08-03T20:53:31.686 INFO:tasks.ceph.mds.a.smithi032.stderr: 16: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x562125c8458d] 2024-08-03T20:53:31.686 INFO:tasks.ceph.mds.a.smithi032.stderr: 17: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x562125c84eb4] 2024-08-03T20:53:31.686 INFO:tasks.ceph.mds.a.smithi032.stderr: 18: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x562125c6800a] 2024-08-03T20:53:31.686 INFO:tasks.ceph.mds.a.smithi032.stderr: 19: /usr/lib64/ceph/libceph-common.so.2(+0x3a3e77) [0x7f0796da3e77] 2024-08-03T20:53:31.686 INFO:tasks.ceph.mds.a.smithi032.stderr: 20: (DispatchQueue::entry()+0x60f) [0x7f0796da4769] 2024-08-03T20:53:31.687 INFO:tasks.ceph.mds.a.smithi032.stderr: 21: /usr/lib64/ceph/libceph-common.so.2(+0x49091d) [0x7f0796e9091d] 2024-08-03T20:53:31.687 INFO:tasks.ceph.mds.a.smithi032.stderr: 22: (Thread::entry_wrapper()+0x43) [0x7f0796c2df4f] 2024-08-03T20:53:31.687 INFO:tasks.ceph.mds.a.smithi032.stderr: 23: (Thread::_entry_func(void*)+0xd) [0x7f0796c2df6b] 2024-08-03T20:53:31.687 INFO:tasks.ceph.mds.a.smithi032.stderr: 24: /lib64/libc.so.6(+0x89c02) [0x7f0796289c02] 2024-08-03T20:53:31.687 INFO:tasks.ceph.mds.a.smithi032.stderr: 25: /lib64/libc.so.6(+0x10ec40) [0x7f079630ec40] 2024-08-03T20:53:31.687 INFO:tasks.ceph.mds.a.smithi032.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-08-03T20:53:31.687 INFO:tasks.ceph.mds.a.smithi032.stderr: 2024-08-03T20:53:31.688 INFO:tasks.ceph.mds.a.smithi032.stderr: -4303> 2024-08-03T20:53:17.531+0000 7f078ca00640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-08-03T20:53:31.688 INFO:tasks.ceph.mds.a.smithi032.stderr: -9> 2024-08-03T20:53:31.657+0000 7f078fc00640 -1 mds.0.cache.den(0x10000000000 z) newly corrupt dentry to be committed: [dentry #0x1/a/z [fffffffffffffff6,head] auth NULL (dn xlock x=1 by request(client.5762:33 nref=7 cr=0x562158b7f500)) (dversion lock w=1 last_client=5762) pv=11 v=10 ap=2 ino=(nil) state=1073741824 | request=1 lock=2 authpin=1 0x562158c00780] 2024-08-03T20:53:31.688 INFO:tasks.ceph.mds.a.smithi032.stderr: -7> 2024-08-03T20:53:31.657+0000 7f078fc00640 -1 log_channel(cluster) log [ERR] : MDS abort because newly corrupt dentry to be committed: [dentry #0x1/a/z [fffffffffffffff6,head] auth NULL (dn xlock x=1 by request(client.5762:33 nref=7 cr=0x562158b7f500)) (dversion lock w=1 last_client=5762) pv=11 v=10 ap=2 ino=(nil) state=1073741824 | request=1 lock=2 authpin=1 0x562158c00780] 2024-08-03T20:53:31.688 INFO:tasks.ceph.mds.a.smithi032.stderr: -1> 2024-08-03T20:53:31.658+0000 7f078fc00640 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-5476-g3792a0a4/rpm/el9/BUILD/ceph-19.0.0-5476-g3792a0a4/src/mds/MDSRank.cc: In function 'void MDSRank::abort(std::string_view)' thread 7f078fc00640 time 2024-08-03T20:53:31.658596+0000 2024-08-03T20:53:31.688 INFO:tasks.ceph.mds.a.smithi032.stderr:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-5476-g3792a0a4/rpm/el9/BUILD/ceph-19.0.0-5476-g3792a0a4/src/mds/MDSRank.cc: 952: ceph_abort_msg("abort() called") 2024-08-03T20:53:31.688 INFO:tasks.ceph.mds.a.smithi032.stderr: 2024-08-03T20:53:31.688 INFO:tasks.ceph.mds.a.smithi032.stderr: ceph version 19.0.0-5476-g3792a0a4 (3792a0a4a40f645dee271188eab388b1c22c10f8) squid (dev) 2024-08-03T20:53:31.688 INFO:tasks.ceph.mds.a.smithi032.stderr: 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xcf) [0x7f0796c55480] 2024-08-03T20:53:31.688 INFO:tasks.ceph.mds.a.smithi032.stderr: 2: (MDSRank::abort(std::basic_string_view >)+0x5f) [0x562125c76297] 2024-08-03T20:53:31.688 INFO:tasks.ceph.mds.a.smithi032.stderr: 3: (CDentry::check_corruption(bool)+0x530) [0x562125f44a32] 2024-08-03T20:53:31.688 INFO:tasks.ceph.mds.a.smithi032.stderr: 4: ceph-mds(+0x173a38) [0x562125cdea38] 2024-08-03T20:53:31.689 INFO:tasks.ceph.mds.a.smithi032.stderr: 5: ceph-mds(+0x173e38) [0x562125cdee38] 2024-08-03T20:53:31.689 INFO:tasks.ceph.mds.a.smithi032.stderr: 6: (Server::_rename_prepare(boost::intrusive_ptr const&, EMetaBlob*, ceph::buffer::v15_2_0::list*, CDentry*, CDentry*, std::basic_string_view >, CDentry*)+0x249b) [0x562125d8264d] 2024-08-03T20:53:31.689 INFO:tasks.ceph.mds.a.smithi032.stderr: 7: (Server::handle_client_rename(boost::intrusive_ptr const&)+0x2ec1) [0x562125d85e0f] 2024-08-03T20:53:31.689 INFO:tasks.ceph.mds.a.smithi032.stderr: 8: (Server::dispatch_client_request(boost::intrusive_ptr const&)+0xd27) [0x562125d4a795] 2024-08-03T20:53:31.689 INFO:tasks.ceph.mds.a.smithi032.stderr: 9: (Server::handle_client_request(boost::intrusive_ptr const&)+0xa2a) [0x562125d4b442] 2024-08-03T20:53:31.689 INFO:tasks.ceph.mds.a.smithi032.stderr: 10: (Server::dispatch(boost::intrusive_ptr const&)+0xc0) [0x562125d31e3c] 2024-08-03T20:53:31.689 INFO:tasks.ceph.mds.a.smithi032.stderr: 11: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x5ac) [0x562125c8073e] 2024-08-03T20:53:31.689 INFO:tasks.ceph.mds.a.smithi032.stderr: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x562125c8458d] 2024-08-03T20:53:31.689 INFO:tasks.ceph.mds.a.smithi032.stderr: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x562125c84eb4] 2024-08-03T20:53:31.689 INFO:tasks.ceph.mds.a.smithi032.stderr: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x562125c6800a] 2024-08-03T20:53:31.689 INFO:tasks.ceph.mds.a.smithi032.stderr: 15: /usr/lib64/ceph/libceph-common.so.2(+0x3a3e77) [0x7f0796da3e77] 2024-08-03T20:53:31.689 INFO:tasks.ceph.mds.a.smithi032.stderr: 16: (DispatchQueue::entry()+0x60f) [0x7f0796da4769] 2024-08-03T20:53:31.689 INFO:tasks.ceph.mds.a.smithi032.stderr: 17: /usr/lib64/ceph/libceph-common.so.2(+0x49091d) [0x7f0796e9091d] 2024-08-03T20:53:31.689 INFO:tasks.ceph.mds.a.smithi032.stderr: 18: (Thread::entry_wrapper()+0x43) [0x7f0796c2df4f] 2024-08-03T20:53:31.690 INFO:tasks.ceph.mds.a.smithi032.stderr: 19: (Thread::_entry_func(void*)+0xd) [0x7f0796c2df6b] 2024-08-03T20:53:31.690 INFO:tasks.ceph.mds.a.smithi032.stderr: 20: /lib64/libc.so.6(+0x89c02) [0x7f0796289c02] 2024-08-03T20:53:31.690 INFO:tasks.ceph.mds.a.smithi032.stderr: 21: /lib64/libc.so.6(+0x10ec40) [0x7f079630ec40] 2024-08-03T20:53:31.690 INFO:tasks.ceph.mds.a.smithi032.stderr: 2024-08-03T20:53:31.690 INFO:tasks.ceph.mds.a.smithi032.stderr: 0> 2024-08-03T20:53:31.659+0000 7f078fc00640 -1 *** Caught signal (Aborted) ** 2024-08-03T20:53:31.690 INFO:tasks.ceph.mds.a.smithi032.stderr: in thread 7f078fc00640 thread_name:ms_dispatch 2024-08-03T20:53:31.690 INFO:tasks.ceph.mds.a.smithi032.stderr: 2024-08-03T20:53:31.690 INFO:tasks.ceph.mds.a.smithi032.stderr: ceph version 19.0.0-5476-g3792a0a4 (3792a0a4a40f645dee271188eab388b1c22c10f8) squid (dev) 2024-08-03T20:53:31.690 INFO:tasks.ceph.mds.a.smithi032.stderr: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f079623e6f0] 2024-08-03T20:53:31.690 INFO:tasks.ceph.mds.a.smithi032.stderr: 2: /lib64/libc.so.6(+0x8b94c) [0x7f079628b94c] 2024-08-03T20:53:31.690 INFO:tasks.ceph.mds.a.smithi032.stderr: 3: raise() 2024-08-03T20:53:31.690 INFO:tasks.ceph.mds.a.smithi032.stderr: 4: abort() 2024-08-03T20:53:31.690 INFO:tasks.ceph.mds.a.smithi032.stderr: 5: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0x241) [0x7f0796c555f2] 2024-08-03T20:53:31.691 INFO:tasks.ceph.mds.a.smithi032.stderr: 6: (MDSRank::abort(std::basic_string_view >)+0x5f) [0x562125c76297] 2024-08-03T20:53:31.691 INFO:tasks.ceph.mds.a.smithi032.stderr: 7: (CDentry::check_corruption(bool)+0x530) [0x562125f44a32] 2024-08-03T20:53:31.691 INFO:tasks.ceph.mds.a.smithi032.stderr: 8: ceph-mds(+0x173a38) [0x562125cdea38] 2024-08-03T20:53:31.691 INFO:tasks.ceph.mds.a.smithi032.stderr: 9: ceph-mds(+0x173e38) [0x562125cdee38] 2024-08-03T20:53:31.691 INFO:tasks.ceph.mds.a.smithi032.stderr: 10: (Server::_rename_prepare(boost::intrusive_ptr const&, EMetaBlob*, ceph::buffer::v15_2_0::list*, CDentry*, CDentry*, std::basic_string_view >, CDentry*)+0x249b) [0x562125d8264d] 2024-08-03T20:53:31.691 INFO:tasks.ceph.mds.a.smithi032.stderr: 11: (Server::handle_client_rename(boost::intrusive_ptr const&)+0x2ec1) [0x562125d85e0f] 2024-08-03T20:53:31.691 INFO:tasks.ceph.mds.a.smithi032.stderr: 12: (Server::dispatch_client_request(boost::intrusive_ptr const&)+0xd27) [0x562125d4a795] 2024-08-03T20:53:31.691 INFO:tasks.ceph.mds.a.smithi032.stderr: 13: (Server::handle_client_request(boost::intrusive_ptr const&)+0xa2a) [0x562125d4b442] 2024-08-03T20:53:31.691 INFO:tasks.ceph.mds.a.smithi032.stderr: 14: (Server::dispatch(boost::intrusive_ptr const&)+0xc0) [0x562125d31e3c] 2024-08-03T20:53:31.691 INFO:tasks.ceph.mds.a.smithi032.stderr: 15: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x5ac) [0x562125c8073e] 2024-08-03T20:53:31.691 INFO:tasks.ceph.mds.a.smithi032.stderr: 16: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x562125c8458d] 2024-08-03T20:53:31.691 INFO:tasks.ceph.mds.a.smithi032.stderr: 17: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x562125c84eb4] 2024-08-03T20:53:31.691 INFO:tasks.ceph.mds.a.smithi032.stderr: 18: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x562125c6800a] 2024-08-03T20:53:31.691 INFO:tasks.ceph.mds.a.smithi032.stderr: 19: /usr/lib64/ceph/libceph-common.so.2(+0x3a3e77) [0x7f0796da3e77] 2024-08-03T20:53:31.692 INFO:tasks.ceph.mds.a.smithi032.stderr: 20: (DispatchQueue::entry()+0x60f) [0x7f0796da4769] 2024-08-03T20:53:31.692 INFO:tasks.ceph.mds.a.smithi032.stderr: 21: /usr/lib64/ceph/libceph-common.so.2(+0x49091d) [0x7f0796e9091d] 2024-08-03T20:53:31.692 INFO:tasks.ceph.mds.a.smithi032.stderr: 22: (Thread::entry_wrapper()+0x43) [0x7f0796c2df4f] 2024-08-03T20:53:31.692 INFO:tasks.ceph.mds.a.smithi032.stderr: 23: (Thread::_entry_func(void*)+0xd) [0x7f0796c2df6b] 2024-08-03T20:53:31.692 INFO:tasks.ceph.mds.a.smithi032.stderr: 24: /lib64/libc.so.6(+0x89c02) [0x7f0796289c02] 2024-08-03T20:53:31.692 INFO:tasks.ceph.mds.a.smithi032.stderr: 25: /lib64/libc.so.6(+0x10ec40) [0x7f079630ec40] 2024-08-03T20:53:31.692 INFO:tasks.ceph.mds.a.smithi032.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-08-03T20:53:31.692 INFO:tasks.ceph.mds.a.smithi032.stderr: 2024-08-03T20:53:31.818 INFO:teuthology.orchestra.run.smithi032.stdout:{"mds_beacon_grace":"15.000000"} 2024-08-03T20:53:31.818 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_beacon_grace": "15.000000" } 2024-08-03T20:53:31.818 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-08-03T20:53:31.831 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:31.829+0000 7f2444c00640 1 -- 172.21.15.32:0/253282899 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f2440074660 msgr2=0x7f244006fac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:53:31.831 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:31.829+0000 7f2444c00640 1 --2- 172.21.15.32:0/253282899 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f2440074660 0x7f244006fac0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f24400cfe70 tx=0x7f243002a560 comp rx=0 tx=0).stop 2024-08-03T20:53:31.831 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:31.829+0000 7f2444c00640 1 -- 172.21.15.32:0/253282899 shutdown_connections 2024-08-03T20:53:31.831 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:31.829+0000 7f2444c00640 1 --2- 172.21.15.32:0/253282899 >> v2:172.21.15.32:3300/0 conn(0x7f24400caaa0 0x7f2440160040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:53:31.831 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:31.829+0000 7f2444c00640 1 --2- 172.21.15.32:0/253282899 >> v2:172.21.15.118:3300/0 conn(0x7f2440070000 0x7f24400ca560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:53:31.831 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:31.829+0000 7f2444c00640 1 --2- 172.21.15.32:0/253282899 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f2440074660 0x7f244006fac0 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:53:31.831 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:31.829+0000 7f2444c00640 1 -- 172.21.15.32:0/253282899 >> 172.21.15.32:0/253282899 conn(0x7f24400b1a00 msgr2=0x7f24400b1e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:53:31.831 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:31.829+0000 7f2444c00640 1 -- 172.21.15.32:0/253282899 shutdown_connections 2024-08-03T20:53:31.831 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:31.829+0000 7f2444c00640 1 -- 172.21.15.32:0/253282899 wait complete. 2024-08-03T20:53:31.833 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:31.831+0000 7f2444c00640 1 Processor -- start 2024-08-03T20:53:31.833 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:31.831+0000 7f2444c00640 1 -- start start 2024-08-03T20:53:31.835 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:31.833+0000 7f2444c00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f2440070000 0x7f24401f53d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:53:31.835 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:31.833+0000 7f2444c00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2440074660 0x7f244014d610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:53:31.835 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:31.833+0000 7f2444c00640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f24400caaa0 0x7f24400b4a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:53:31.835 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:31.833+0000 7f2444c00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f2440160580 con 0x7f2440074660 2024-08-03T20:53:31.835 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:31.833+0000 7f2444c00640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f2440073030 con 0x7f2440070000 2024-08-03T20:53:31.835 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:31.833+0000 7f2444c00640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7f2440160700 con 0x7f24400caaa0 2024-08-03T20:53:31.835 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:31.833+0000 7f2437e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2440074660 0x7f244014d610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:53:31.835 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:31.833+0000 7f2437e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2440074660 0x7f244014d610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:45070/0 (socket says 172.21.15.32:45070) 2024-08-03T20:53:31.835 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:31.833+0000 7f2437e00640 1 -- 172.21.15.32:0/1561994681 learned_addr learned my addr 172.21.15.32:0/1561994681 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:53:31.836 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:31.833+0000 7f243e800640 1 --2- 172.21.15.32:0/1561994681 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f24400caaa0 0x7f24400b4a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:53:31.836 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:31.833+0000 7f243de00640 1 --2- 172.21.15.32:0/1561994681 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f2440070000 0x7f24401f53d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:53:31.837 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:31.834+0000 7f2437e00640 1 -- 172.21.15.32:0/1561994681 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f24400caaa0 msgr2=0x7f24400b4a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:53:31.837 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:31.834+0000 7f2437e00640 1 --2- 172.21.15.32:0/1561994681 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f24400caaa0 0x7f24400b4a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:53:31.837 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:31.834+0000 7f2437e00640 1 -- 172.21.15.32:0/1561994681 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f2440070000 msgr2=0x7f24401f53d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:53:31.837 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:31.834+0000 7f2437e00640 1 --2- 172.21.15.32:0/1561994681 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f2440070000 0x7f24401f53d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:53:31.838 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:31.834+0000 7f2437e00640 1 -- 172.21.15.32:0/1561994681 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f24401f5a10 con 0x7f2440074660 2024-08-03T20:53:31.838 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:31.834+0000 7f2437e00640 1 --2- 172.21.15.32:0/1561994681 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2440074660 0x7f244014d610 secure :-1 s=READY pgs=792 cs=0 l=1 rev1=1 crypto rx=0x7f242800e720 tx=0x7f242800ebe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:53:31.838 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:31.834+0000 7f2436000640 1 -- 172.21.15.32:0/1561994681 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f242800cab0 con 0x7f2440074660 2024-08-03T20:53:31.838 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:31.835+0000 7f2436000640 1 -- 172.21.15.32:0/1561994681 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f242800cc50 con 0x7f2440074660 2024-08-03T20:53:31.838 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:31.835+0000 7f2436000640 1 -- 172.21.15.32:0/1561994681 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2428005280 con 0x7f2440074660 2024-08-03T20:53:31.838 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:31.835+0000 7f2444c00640 1 -- 172.21.15.32:0/1561994681 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f24401f5cf0 con 0x7f2440074660 2024-08-03T20:53:31.838 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:31.835+0000 7f2444c00640 1 -- 172.21.15.32:0/1561994681 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f24401f62b0 con 0x7f2440074660 2024-08-03T20:53:31.838 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:31.835+0000 7f2444c00640 1 -- 172.21.15.32:0/1561994681 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "status"} v 0) -- 0x7f2440160bb0 con 0x7f2440074660 2024-08-03T20:53:31.839 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:31.837+0000 7f2436000640 1 -- 172.21.15.32:0/1561994681 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f2428005420 con 0x7f2440074660 2024-08-03T20:53:31.840 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:31.838+0000 7f2436000640 1 --2- 172.21.15.32:0/1561994681 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f241c077040 0x7f241c079550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:53:31.840 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:31.839+0000 7f2436000640 1 -- 172.21.15.32:0/1561994681 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(70..70 src has 1..70) ==== 5709+0+0 (secure 0 0 0) 0x7f24280996d0 con 0x7f2440074660 2024-08-03T20:53:31.840 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:31.839+0000 7f2436000640 1 -- 172.21.15.32:0/1561994681 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "status"}]=0 v0) ==== 54+0+511 (secure 0 0 0) 0x7f2428026020 con 0x7f2440074660 2024-08-03T20:53:31.841 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:31.839+0000 7f243de00640 1 --2- 172.21.15.32:0/1561994681 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f241c077040 0x7f241c079550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:53:31.841 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:31.839+0000 7f2444c00640 1 -- 172.21.15.32:0/1561994681 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({log-info=0+}) -- 0x7f244015ce60 con 0x7f2440074660 2024-08-03T20:53:31.841 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:31.840+0000 7f243de00640 1 --2- 172.21.15.32:0/1561994681 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f241c077040 0x7f241c079550 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f2430033000 tx=0x7f2430004730 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:53:32.050 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:32.048+0000 7f0192600640 1 -- 172.21.15.32:0/1922412245 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f018c0747b0 msgr2=0x7f018c0cc8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:53:32.050 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:32.048+0000 7f0192600640 1 --2- 172.21.15.32:0/1922412245 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f018c0747b0 0x7f018c0cc8e0 secure :-1 s=READY pgs=793 cs=0 l=1 rev1=1 crypto rx=0x7f018c071700 tx=0x7f018402aa10 comp rx=0 tx=0).stop 2024-08-03T20:53:32.050 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:32.048+0000 7f0192600640 1 -- 172.21.15.32:0/1922412245 shutdown_connections 2024-08-03T20:53:32.050 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:32.048+0000 7f0192600640 1 --2- 172.21.15.32:0/1922412245 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f018c0747b0 0x7f018c0cc8e0 unknown :-1 s=CLOSED pgs=793 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:53:32.050 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:32.048+0000 7f0192600640 1 --2- 172.21.15.32:0/1922412245 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f018c073e60 0x7f018c074270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:53:32.051 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:32.048+0000 7f0192600640 1 --2- 172.21.15.32:0/1922412245 >> v2:172.21.15.118:3300/0 conn(0x7f018c06e930 0x7f018c144f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:53:32.051 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:32.048+0000 7f0192600640 1 -- 172.21.15.32:0/1922412245 >> 172.21.15.32:0/1922412245 conn(0x7f018c0b1bd0 msgr2=0x7f018c0b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:53:32.051 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:32.048+0000 7f0192600640 1 -- 172.21.15.32:0/1922412245 shutdown_connections 2024-08-03T20:53:32.051 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:32.048+0000 7f0192600640 1 -- 172.21.15.32:0/1922412245 wait complete. 2024-08-03T20:53:32.052 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:32.051+0000 7f0192600640 1 Processor -- start 2024-08-03T20:53:32.052 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:32.051+0000 7f0192600640 1 -- start start 2024-08-03T20:53:32.054 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:32.052+0000 7f0192600640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f018c06e930 0x7f018c15b140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:53:32.054 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:32.052+0000 7f0192600640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f018c073e60 0x7f018c1f86f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:53:32.054 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:32.052+0000 7f0192600640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f018c0747b0 0x7f018c141b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:53:32.054 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:32.052+0000 7f0192600640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f018c0cf9c0 con 0x7f018c0747b0 2024-08-03T20:53:32.054 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:32.052+0000 7f0192600640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f018c0cf840 con 0x7f018c06e930 2024-08-03T20:53:32.055 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:32.052+0000 7f0192600640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7f018c0cce20 con 0x7f018c073e60 2024-08-03T20:53:32.055 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:32.052+0000 7f018be00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f018c0747b0 0x7f018c141b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:53:32.055 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:32.053+0000 7f018be00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f018c0747b0 0x7f018c141b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:45090/0 (socket says 172.21.15.32:45090) 2024-08-03T20:53:32.055 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:32.053+0000 7f018be00640 1 -- 172.21.15.32:0/2044475232 learned_addr learned my addr 172.21.15.32:0/2044475232 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:53:32.055 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:32.053+0000 7f018b400640 1 --2- 172.21.15.32:0/2044475232 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f018c06e930 0x7f018c15b140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:53:32.055 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:32.053+0000 7f0182a00640 1 --2- 172.21.15.32:0/2044475232 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f018c073e60 0x7f018c1f86f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:53:32.056 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:32.053+0000 7f018be00640 1 -- 172.21.15.32:0/2044475232 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f018c073e60 msgr2=0x7f018c1f86f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:53:32.056 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:32.053+0000 7f018be00640 1 --2- 172.21.15.32:0/2044475232 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f018c073e60 0x7f018c1f86f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:53:32.056 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:32.053+0000 7f018be00640 1 -- 172.21.15.32:0/2044475232 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f018c06e930 msgr2=0x7f018c15b140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:53:32.056 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:32.053+0000 7f018be00640 1 --2- 172.21.15.32:0/2044475232 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f018c06e930 0x7f018c15b140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:53:32.057 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:32.053+0000 7f018be00640 1 -- 172.21.15.32:0/2044475232 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f018c1f8c30 con 0x7f018c0747b0 2024-08-03T20:53:32.057 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:32.053+0000 7f0182a00640 1 --2- 172.21.15.32:0/2044475232 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f018c073e60 0x7f018c1f86f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-03T20:53:32.057 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:32.053+0000 7f018be00640 1 --2- 172.21.15.32:0/2044475232 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f018c0747b0 0x7f018c141b00 secure :-1 s=READY pgs=794 cs=0 l=1 rev1=1 crypto rx=0x7f018402b000 tx=0x7f0184003f10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:53:32.057 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:32.054+0000 7f0183e00640 1 -- 172.21.15.32:0/2044475232 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0184033050 con 0x7f018c0747b0 2024-08-03T20:53:32.057 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:32.054+0000 7f0192600640 1 -- 172.21.15.32:0/2044475232 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f018c1f9140 con 0x7f018c0747b0 2024-08-03T20:53:32.057 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:32.054+0000 7f0183e00640 1 -- 172.21.15.32:0/2044475232 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f0184004450 con 0x7f018c0747b0 2024-08-03T20:53:32.057 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:32.054+0000 7f0183e00640 1 -- 172.21.15.32:0/2044475232 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0184038450 con 0x7f018c0747b0 2024-08-03T20:53:32.057 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:32.054+0000 7f0192600640 1 -- 172.21.15.32:0/2044475232 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f018c0c8040 con 0x7f018c0747b0 2024-08-03T20:53:32.057 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:32.055+0000 7f0192600640 1 -- 172.21.15.32:0/2044475232 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f018c074270 con 0x7f018c0747b0 2024-08-03T20:53:32.057 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:32.056+0000 7f0183e00640 1 -- 172.21.15.32:0/2044475232 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f0184036070 con 0x7f018c0747b0 2024-08-03T20:53:32.059 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:32.057+0000 7f0183e00640 1 --2- 172.21.15.32:0/2044475232 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f0168077010 0x7f0168079520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:53:32.059 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:32.057+0000 7f0183e00640 1 -- 172.21.15.32:0/2044475232 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(70..70 src has 1..70) ==== 5709+0+0 (secure 0 0 0) 0x7f01840b9ce0 con 0x7f018c0747b0 2024-08-03T20:53:32.059 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:32.058+0000 7f018b400640 1 --2- 172.21.15.32:0/2044475232 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f0168077010 0x7f0168079520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:53:32.060 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:32.058+0000 7f018b400640 1 --2- 172.21.15.32:0/2044475232 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f0168077010 0x7f0168079520 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f017c00bb50 tx=0x7f017c00b2e0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:53:32.063 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:32.062+0000 7f0183e00640 1 -- 172.21.15.32:0/2044475232 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7f0184082ee0 con 0x7f018c0747b0 2024-08-03T20:53:32.312 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:32.310+0000 7f0192600640 1 -- 172.21.15.32:0/2044475232 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f018c1f8ed0 con 0x7f018c0747b0 2024-08-03T20:53:32.312 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:32.311+0000 7f0183e00640 1 -- 172.21.15.32:0/2044475232 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 75 v75) ==== 94+0+5231 (secure 0 0 0) 0x7f01840827e0 con 0x7f018c0747b0 2024-08-03T20:53:32.313 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-03T20:53:32.313 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":75,"btime":"2024-08-03T20:53:25:602045+0000","default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5666,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6837/2261538396","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6835","nonce":2261538396},{"type":"v1","addr":"172.21.15.118:6837","nonce":2261538396}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":69},{"gid":6061,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6836/1133580656","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6834","nonce":1133580656},{"type":"v1","addr":"172.21.15.118:6836","nonce":1133580656}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":68},{"gid":6396,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6834/1798758431","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":1798758431},{"type":"v1","addr":"172.21.15.32:6834","nonce":1798758431}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":69}],"filesystems":[{"mdsmap":{"epoch":75,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-08-03T20:53:15.080669+0000","modified":"2024-08-03T20:53:25.076887+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6399},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6399":{"gid":6399,"name":"a","rank":0,"incarnation":70,"state":"up:active","state_seq":3,"addr":"172.21.15.32:6835/4240363667","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6833","nonce":4240363667},{"type":"v1","addr":"172.21.15.32:6835","nonce":4240363667}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6399,"qdb_cluster":[6399]},"id":5}]} 2024-08-03T20:53:32.313 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 75 2024-08-03T20:53:32.315 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:32.313+0000 7f0192600640 1 -- 172.21.15.32:0/2044475232 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f0168077010 msgr2=0x7f0168079520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:53:32.315 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:32.313+0000 7f0192600640 1 --2- 172.21.15.32:0/2044475232 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f0168077010 0x7f0168079520 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f017c00bb50 tx=0x7f017c00b2e0 comp rx=0 tx=0).stop 2024-08-03T20:53:32.316 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:32.313+0000 7f0192600640 1 -- 172.21.15.32:0/2044475232 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f018c0747b0 msgr2=0x7f018c141b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:53:32.316 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:32.313+0000 7f0192600640 1 --2- 172.21.15.32:0/2044475232 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f018c0747b0 0x7f018c141b00 secure :-1 s=READY pgs=794 cs=0 l=1 rev1=1 crypto rx=0x7f018402b000 tx=0x7f0184003f10 comp rx=0 tx=0).stop 2024-08-03T20:53:32.316 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:32.313+0000 7f0192600640 1 -- 172.21.15.32:0/2044475232 shutdown_connections 2024-08-03T20:53:32.316 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:32.313+0000 7f0192600640 1 --2- 172.21.15.32:0/2044475232 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f0168077010 0x7f0168079520 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:53:32.316 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:32.313+0000 7f0192600640 1 --2- 172.21.15.32:0/2044475232 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f018c0747b0 0x7f018c141b00 unknown :-1 s=CLOSED pgs=794 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:53:32.316 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:32.313+0000 7f0192600640 1 --2- 172.21.15.32:0/2044475232 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f018c073e60 0x7f018c1f86f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:53:32.316 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:32.313+0000 7f0192600640 1 --2- 172.21.15.32:0/2044475232 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f018c06e930 0x7f018c15b140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:53:32.316 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:32.313+0000 7f0192600640 1 -- 172.21.15.32:0/2044475232 >> 172.21.15.32:0/2044475232 conn(0x7f018c0b1bd0 msgr2=0x7f018c0cba60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:53:32.316 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:32.314+0000 7f0192600640 1 -- 172.21.15.32:0/2044475232 shutdown_connections 2024-08-03T20:53:32.316 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:32.314+0000 7f0192600640 1 -- 172.21.15.32:0/2044475232 wait complete. 2024-08-03T20:53:32.336 INFO:tasks.ceph.mds.a.smithi032.stderr:daemon-helper: command crashed with signal 6 2024-08-03T20:53:32.337 DEBUG:tasks.ceph_test_case:wait_until_true: waiting (timeout=45.0 retry_count=0)... 2024-08-03T20:53:32.630 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:32.628+0000 7f2436000640 1 -- 172.21.15.32:0/1561994681 <== mon.0 v2:172.21.15.32:3300/0 7 ==== log(1 entries from seq 1 at 2024-08-03T20:53:31.658566+0000) ==== 468+0+0 (secure 0 0 0) 0x7f242805e480 con 0x7f2440074660 2024-08-03T20:53:32.630 INFO:teuthology.orchestra.run.smithi032.stdout: cluster: 2024-08-03T20:53:32.631 INFO:teuthology.orchestra.run.smithi032.stdout: id: 40ab141a-01d7-49b9-ae3c-2edec9876e95 2024-08-03T20:53:32.631 INFO:teuthology.orchestra.run.smithi032.stdout: health: HEALTH_OK 2024-08-03T20:53:32.631 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-03T20:53:32.631 INFO:teuthology.orchestra.run.smithi032.stdout: services: 2024-08-03T20:53:32.631 INFO:teuthology.orchestra.run.smithi032.stdout: mon: 3 daemons, quorum a,b,c (age 6m) 2024-08-03T20:53:32.631 INFO:teuthology.orchestra.run.smithi032.stdout: mgr: x(active, since 6m), standbys: y 2024-08-03T20:53:32.631 INFO:teuthology.orchestra.run.smithi032.stdout: mds: 1/1 daemons up, 3 standby 2024-08-03T20:53:32.631 INFO:teuthology.orchestra.run.smithi032.stdout: osd: 8 osds: 8 up (since 6m), 8 in (since 6m) 2024-08-03T20:53:32.631 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-03T20:53:32.631 INFO:teuthology.orchestra.run.smithi032.stdout: data: 2024-08-03T20:53:32.631 INFO:teuthology.orchestra.run.smithi032.stdout: volumes: 1/1 healthy 2024-08-03T20:53:32.632 INFO:teuthology.orchestra.run.smithi032.stdout: pools: 3 pools, 97 pgs 2024-08-03T20:53:32.632 INFO:teuthology.orchestra.run.smithi032.stdout: objects: 29 objects, 596 KiB 2024-08-03T20:53:32.632 INFO:teuthology.orchestra.run.smithi032.stdout: usage: 255 MiB used, 720 GiB / 720 GiB avail 2024-08-03T20:53:32.632 INFO:teuthology.orchestra.run.smithi032.stdout: pgs: 97 active+clean 2024-08-03T20:53:32.632 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-03T20:53:32.632 INFO:teuthology.orchestra.run.smithi032.stdout: io: 2024-08-03T20:53:32.632 INFO:teuthology.orchestra.run.smithi032.stdout: client: 0 B/s rd, 4.0 KiB/s wr, 0 op/s rd, 6 op/s wr 2024-08-03T20:53:32.632 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-03T20:53:32.632 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-03T20:53:32.632 INFO:teuthology.orchestra.run.smithi032.stdout:2024-08-03T20:53:31.658566+0000 mds.a [ERR] MDS abort because newly corrupt dentry to be committed: [dentry #0x1/a/z [fffffffffffffff6,head] auth NULL (dn xlock x=1 by request(client.5762:33 nref=7 cr=0x562158b7f500)) (dversion lock w=1 last_client=5762) pv=11 v=10 ap=2 ino=(nil) state=1073741824 | request=1 lock=2 authpin=1 0x562158c00780] 2024-08-03T20:53:37.339 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-08-03T20:53:37.578 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:37.576+0000 7f38faa00640 1 -- 172.21.15.32:0/3844610400 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f38d8003c30 msgr2=0x7f38d8004080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:53:37.578 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:37.576+0000 7f38faa00640 1 --2- 172.21.15.32:0/3844610400 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f38d8003c30 0x7f38d8004080 secure :-1 s=READY pgs=796 cs=0 l=1 rev1=1 crypto rx=0x7f38e4006fb0 tx=0x7f38d8005830 comp rx=0 tx=0).stop 2024-08-03T20:53:37.578 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:37.576+0000 7f38faa00640 1 -- 172.21.15.32:0/3844610400 shutdown_connections 2024-08-03T20:53:37.578 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:37.576+0000 7f38faa00640 1 --2- 172.21.15.32:0/3844610400 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f38d8003c30 0x7f38d8004080 unknown :-1 s=CLOSED pgs=796 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:53:37.578 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:37.576+0000 7f38faa00640 1 -- 172.21.15.32:0/3844610400 >> 172.21.15.32:0/3844610400 conn(0x7f38f40b1bd0 msgr2=0x7f38f40b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:53:37.578 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:37.576+0000 7f38faa00640 1 -- 172.21.15.32:0/3844610400 shutdown_connections 2024-08-03T20:53:37.579 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:37.576+0000 7f38faa00640 1 -- 172.21.15.32:0/3844610400 wait complete. 2024-08-03T20:53:37.580 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:37.579+0000 7f38faa00640 1 Processor -- start 2024-08-03T20:53:37.580 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:37.579+0000 7f38faa00640 1 -- start start 2024-08-03T20:53:37.581 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-08-03T20:53:37.583 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:37.580+0000 7f38faa00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f38d8003c30 0x7f38f414d010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:53:37.583 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:37.580+0000 7f38faa00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f38f41434f0 0x7f38f4143900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:53:37.583 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:37.580+0000 7f38faa00640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f38f414a8a0 0x7f38f414acb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:53:37.583 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:37.580+0000 7f38faa00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f38f40cf9c0 con 0x7f38d8003c30 2024-08-03T20:53:37.583 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:37.580+0000 7f38faa00640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f38f40cf840 con 0x7f38f41434f0 2024-08-03T20:53:37.583 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:37.580+0000 7f38faa00640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7f38f4073e60 con 0x7f38f414a8a0 2024-08-03T20:53:37.583 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:37.580+0000 7f38f3e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f38d8003c30 0x7f38f414d010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:53:37.583 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:37.580+0000 7f38f3e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f38d8003c30 0x7f38f414d010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:45108/0 (socket says 172.21.15.32:45108) 2024-08-03T20:53:37.584 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:37.581+0000 7f38f3e00640 1 -- 172.21.15.32:0/1968577692 learned_addr learned my addr 172.21.15.32:0/1968577692 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:53:37.584 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:37.581+0000 7f38f8a00640 1 --2- 172.21.15.32:0/1968577692 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f38f414a8a0 0x7f38f414acb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:53:37.584 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:37.581+0000 7f38eb400640 1 --2- 172.21.15.32:0/1968577692 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f38f41434f0 0x7f38f4143900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:53:37.584 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:37.581+0000 7f38f3e00640 1 -- 172.21.15.32:0/1968577692 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f38f414a8a0 msgr2=0x7f38f414acb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:53:37.584 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:37.581+0000 7f38f3e00640 1 --2- 172.21.15.32:0/1968577692 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f38f414a8a0 0x7f38f414acb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:53:37.584 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:37.581+0000 7f38f3e00640 1 -- 172.21.15.32:0/1968577692 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f38f41434f0 msgr2=0x7f38f4143900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:53:37.584 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:37.581+0000 7f38f3e00640 1 --2- 172.21.15.32:0/1968577692 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f38f41434f0 0x7f38f4143900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:53:37.584 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:37.581+0000 7f38f3e00640 1 -- 172.21.15.32:0/1968577692 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f38f4143e40 con 0x7f38d8003c30 2024-08-03T20:53:37.586 DEBUG:teuthology.orchestra.run.smithi118:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-08-03T20:53:37.587 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:37.581+0000 7f38f3e00640 1 --2- 172.21.15.32:0/1968577692 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f38d8003c30 0x7f38f414d010 secure :-1 s=READY pgs=797 cs=0 l=1 rev1=1 crypto rx=0x7f38e4006fb0 tx=0x7f38e40010a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:53:37.587 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:37.582+0000 7f38f0c00640 1 -- 172.21.15.32:0/1968577692 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38e40037f0 con 0x7f38d8003c30 2024-08-03T20:53:37.587 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:37.582+0000 7f38f0c00640 1 -- 172.21.15.32:0/1968577692 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f38e4003a80 con 0x7f38d8003c30 2024-08-03T20:53:37.587 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:37.582+0000 7f38f0c00640 1 -- 172.21.15.32:0/1968577692 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38e402e480 con 0x7f38d8003c30 2024-08-03T20:53:37.587 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:37.582+0000 7f38faa00640 1 -- 172.21.15.32:0/1968577692 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f38f414d550 con 0x7f38d8003c30 2024-08-03T20:53:37.587 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:37.582+0000 7f38faa00640 1 -- 172.21.15.32:0/1968577692 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f38f41f8da0 con 0x7f38d8003c30 2024-08-03T20:53:37.587 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:37.583+0000 7f38faa00640 1 -- 172.21.15.32:0/1968577692 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f38f40742e0 con 0x7f38d8003c30 2024-08-03T20:53:37.587 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:37.584+0000 7f38f0c00640 1 -- 172.21.15.32:0/1968577692 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f38e4035020 con 0x7f38d8003c30 2024-08-03T20:53:37.588 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:37.585+0000 7f38f0c00640 1 --2- 172.21.15.32:0/1968577692 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f38d0077010 0x7f38d0079520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:53:37.588 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:37.585+0000 7f38f0c00640 1 -- 172.21.15.32:0/1968577692 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(70..70 src has 1..70) ==== 5709+0+0 (secure 0 0 0) 0x7f38e40b93d0 con 0x7f38d8003c30 2024-08-03T20:53:37.588 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:37.586+0000 7f38eb400640 1 --2- 172.21.15.32:0/1968577692 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f38d0077010 0x7f38d0079520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:53:37.588 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:37.586+0000 7f38eb400640 1 --2- 172.21.15.32:0/1968577692 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f38d0077010 0x7f38d0079520 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f38f41f86a0 tx=0x7f38dc006270 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:53:37.590 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:37.589+0000 7f38f0c00640 1 -- 172.21.15.32:0/1968577692 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7f38f4143e40 con 0x7f38d8003c30 2024-08-03T20:53:37.853 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:37.852+0000 7f38faa00640 1 -- 172.21.15.32:0/1968577692 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f38f413ef80 con 0x7f38d8003c30 2024-08-03T20:53:37.854 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:37.853+0000 7f38f0c00640 1 -- 172.21.15.32:0/1968577692 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 75 v75) ==== 94+0+5231 (secure 0 0 0) 0x7f38e4081f60 con 0x7f38d8003c30 2024-08-03T20:53:37.854 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-03T20:53:37.854 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":75,"btime":"2024-08-03T20:53:25:602045+0000","default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5666,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6837/2261538396","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6835","nonce":2261538396},{"type":"v1","addr":"172.21.15.118:6837","nonce":2261538396}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":69},{"gid":6061,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6836/1133580656","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6834","nonce":1133580656},{"type":"v1","addr":"172.21.15.118:6836","nonce":1133580656}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":68},{"gid":6396,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6834/1798758431","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":1798758431},{"type":"v1","addr":"172.21.15.32:6834","nonce":1798758431}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":69}],"filesystems":[{"mdsmap":{"epoch":75,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-08-03T20:53:15.080669+0000","modified":"2024-08-03T20:53:25.076887+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6399},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6399":{"gid":6399,"name":"a","rank":0,"incarnation":70,"state":"up:active","state_seq":3,"addr":"172.21.15.32:6835/4240363667","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6833","nonce":4240363667},{"type":"v1","addr":"172.21.15.32:6835","nonce":4240363667}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6399,"qdb_cluster":[6399]},"id":5}]} 2024-08-03T20:53:37.855 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 75 2024-08-03T20:53:37.856 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:37.855+0000 7f38faa00640 1 -- 172.21.15.32:0/1968577692 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f38d0077010 msgr2=0x7f38d0079520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:53:37.857 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:37.855+0000 7f38faa00640 1 --2- 172.21.15.32:0/1968577692 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f38d0077010 0x7f38d0079520 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f38f41f86a0 tx=0x7f38dc006270 comp rx=0 tx=0).stop 2024-08-03T20:53:37.857 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:37.855+0000 7f38faa00640 1 -- 172.21.15.32:0/1968577692 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f38d8003c30 msgr2=0x7f38f414d010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:53:37.857 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:37.855+0000 7f38faa00640 1 --2- 172.21.15.32:0/1968577692 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f38d8003c30 0x7f38f414d010 secure :-1 s=READY pgs=797 cs=0 l=1 rev1=1 crypto rx=0x7f38e4006fb0 tx=0x7f38e40010a0 comp rx=0 tx=0).stop 2024-08-03T20:53:37.857 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:37.855+0000 7f38faa00640 1 -- 172.21.15.32:0/1968577692 shutdown_connections 2024-08-03T20:53:37.857 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:37.855+0000 7f38faa00640 1 --2- 172.21.15.32:0/1968577692 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f38d0077010 0x7f38d0079520 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:53:37.857 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:37.855+0000 7f38faa00640 1 --2- 172.21.15.32:0/1968577692 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f38f41434f0 0x7f38f4143900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:53:37.857 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:37.855+0000 7f38faa00640 1 --2- 172.21.15.32:0/1968577692 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f38f414a8a0 0x7f38f414acb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:53:37.857 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:37.855+0000 7f38faa00640 1 --2- 172.21.15.32:0/1968577692 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f38d8003c30 0x7f38f414d010 unknown :-1 s=CLOSED pgs=797 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:53:37.858 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:37.855+0000 7f38faa00640 1 -- 172.21.15.32:0/1968577692 >> 172.21.15.32:0/1968577692 conn(0x7f38f40b1bd0 msgr2=0x7f38f40b1110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:53:37.858 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:37.855+0000 7f38faa00640 1 -- 172.21.15.32:0/1968577692 shutdown_connections 2024-08-03T20:53:37.858 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:37.855+0000 7f38faa00640 1 -- 172.21.15.32:0/1968577692 wait complete. 2024-08-03T20:53:37.879 DEBUG:tasks.ceph_test_case:wait_until_true: waiting (timeout=45.0 retry_count=0)... 2024-08-03T20:53:37.956 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~0s 2024-08-03T20:53:42.881 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-08-03T20:53:43.120 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:43.117+0000 7fd3af200640 1 -- 172.21.15.32:0/1911779868 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd3b00ca3b0 msgr2=0x7fd3b00ca7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:53:43.120 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:43.117+0000 7fd3af200640 1 --2- 172.21.15.32:0/1911779868 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd3b00ca3b0 0x7fd3b00ca7c0 secure :-1 s=READY pgs=799 cs=0 l=1 rev1=1 crypto rx=0x7fd39c005fd0 tx=0x7fd39c02b9a0 comp rx=0 tx=0).stop 2024-08-03T20:53:43.120 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:43.117+0000 7fd3af200640 1 -- 172.21.15.32:0/1911779868 shutdown_connections 2024-08-03T20:53:43.120 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:43.117+0000 7fd3af200640 1 --2- 172.21.15.32:0/1911779868 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd3b00ca3b0 0x7fd3b00ca7c0 unknown :-1 s=CLOSED pgs=799 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:53:43.120 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:43.117+0000 7fd3af200640 1 -- 172.21.15.32:0/1911779868 >> 172.21.15.32:0/1911779868 conn(0x7fd3b00b2120 msgr2=0x7fd3b00cfd60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:53:43.121 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:43.118+0000 7fd3af200640 1 -- 172.21.15.32:0/1911779868 shutdown_connections 2024-08-03T20:53:43.121 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:43.118+0000 7fd3af200640 1 -- 172.21.15.32:0/1911779868 wait complete. 2024-08-03T20:53:43.121 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:43.120+0000 7fd3af200640 1 Processor -- start 2024-08-03T20:53:43.122 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:43.120+0000 7fd3af200640 1 -- start start 2024-08-03T20:53:43.125 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:43.121+0000 7fd3af200640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fd3b00ca3b0 0x7fd3b0073080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:53:43.125 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:43.121+0000 7fd3af200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd3b00ce150 0x7fd3b01f60e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:53:43.125 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:43.122+0000 7fd3af200640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fd3b015a310 0x7fd3b014bab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:53:43.125 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:43.122+0000 7fd3af200640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fd3b0070860 con 0x7fd3b00ce150 2024-08-03T20:53:43.125 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:43.122+0000 7fd3af200640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7fd3b00706e0 con 0x7fd3b00ca3b0 2024-08-03T20:53:43.125 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:43.122+0000 7fd3af200640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7fd3b00cec70 con 0x7fd3b015a310 2024-08-03T20:53:43.125 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:43.122+0000 7fd3ad400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd3b00ce150 0x7fd3b01f60e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:53:43.125 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:43.122+0000 7fd3ade00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fd3b00ca3b0 0x7fd3b0073080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:53:43.125 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:43.122+0000 7fd3ad400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd3b00ce150 0x7fd3b01f60e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:43344/0 (socket says 172.21.15.32:43344) 2024-08-03T20:53:43.125 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:43.122+0000 7fd3ad400640 1 -- 172.21.15.32:0/3944671577 learned_addr learned my addr 172.21.15.32:0/3944671577 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:53:43.126 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:43.122+0000 7fd3ae800640 1 --2- 172.21.15.32:0/3944671577 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fd3b015a310 0x7fd3b014bab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:53:43.126 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:43.122+0000 7fd3ad400640 1 -- 172.21.15.32:0/3944671577 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fd3b015a310 msgr2=0x7fd3b014bab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:53:43.126 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:43.122+0000 7fd3ad400640 1 --2- 172.21.15.32:0/3944671577 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fd3b015a310 0x7fd3b014bab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:53:43.126 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:43.122+0000 7fd3ad400640 1 -- 172.21.15.32:0/3944671577 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fd3b00ca3b0 msgr2=0x7fd3b0073080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:53:43.126 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:43.122+0000 7fd3ad400640 1 --2- 172.21.15.32:0/3944671577 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fd3b00ca3b0 0x7fd3b0073080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:53:43.126 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:43.122+0000 7fd3ad400640 1 -- 172.21.15.32:0/3944671577 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd3b00738d0 con 0x7fd3b00ce150 2024-08-03T20:53:43.126 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:43.123+0000 7fd3ad400640 1 --2- 172.21.15.32:0/3944671577 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd3b00ce150 0x7fd3b01f60e0 secure :-1 s=READY pgs=800 cs=0 l=1 rev1=1 crypto rx=0x7fd39c006010 tx=0x7fd39c0309f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:53:43.126 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:43.123+0000 7fd3a6000640 1 -- 172.21.15.32:0/3944671577 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd39c034030 con 0x7fd3b00ce150 2024-08-03T20:53:43.126 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:43.123+0000 7fd3a6000640 1 -- 172.21.15.32:0/3944671577 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fd39c0036e0 con 0x7fd3b00ce150 2024-08-03T20:53:43.126 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:43.123+0000 7fd3a6000640 1 -- 172.21.15.32:0/3944671577 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd39c003ad0 con 0x7fd3b00ce150 2024-08-03T20:53:43.126 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:43.123+0000 7fd3af200640 1 -- 172.21.15.32:0/3944671577 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd3b0073bb0 con 0x7fd3b00ce150 2024-08-03T20:53:43.127 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:43.123+0000 7fd3af200640 1 -- 172.21.15.32:0/3944671577 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd3b0074170 con 0x7fd3b00ce150 2024-08-03T20:53:43.127 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:43.124+0000 7fd3af200640 1 -- 172.21.15.32:0/3944671577 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd3b00cf190 con 0x7fd3b00ce150 2024-08-03T20:53:43.127 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:43.126+0000 7fd3a6000640 1 -- 172.21.15.32:0/3944671577 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7fd39c003c70 con 0x7fd3b00ce150 2024-08-03T20:53:43.128 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:43.126+0000 7fd3a6000640 1 --2- 172.21.15.32:0/3944671577 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fd390077010 0x7fd390079520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:53:43.128 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:43.127+0000 7fd3a6000640 1 -- 172.21.15.32:0/3944671577 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(70..70 src has 1..70) ==== 5709+0+0 (secure 0 0 0) 0x7fd39c0b9800 con 0x7fd3b00ce150 2024-08-03T20:53:43.128 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:43.127+0000 7fd3ade00640 1 --2- 172.21.15.32:0/3944671577 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fd390077010 0x7fd390079520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:53:43.129 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:43.128+0000 7fd3ade00640 1 --2- 172.21.15.32:0/3944671577 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fd390077010 0x7fd390079520 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7fd3a0001720 tx=0x7fd3a0009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:53:43.132 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:43.131+0000 7fd3a6000640 1 -- 172.21.15.32:0/3944671577 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7fd39c03d070 con 0x7fd3b00ce150 2024-08-03T20:53:43.376 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:43.375+0000 7fd3af200640 1 -- 172.21.15.32:0/3944671577 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd3b00b0d60 con 0x7fd3b00ce150 2024-08-03T20:53:43.377 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:43.375+0000 7fd3a6000640 1 -- 172.21.15.32:0/3944671577 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 75 v75) ==== 94+0+5231 (secure 0 0 0) 0x7fd39c082310 con 0x7fd3b00ce150 2024-08-03T20:53:43.377 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-03T20:53:43.377 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":75,"btime":"2024-08-03T20:53:25:602045+0000","default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5666,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6837/2261538396","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6835","nonce":2261538396},{"type":"v1","addr":"172.21.15.118:6837","nonce":2261538396}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":69},{"gid":6061,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6836/1133580656","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6834","nonce":1133580656},{"type":"v1","addr":"172.21.15.118:6836","nonce":1133580656}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":68},{"gid":6396,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6834/1798758431","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":1798758431},{"type":"v1","addr":"172.21.15.32:6834","nonce":1798758431}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":69}],"filesystems":[{"mdsmap":{"epoch":75,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-08-03T20:53:15.080669+0000","modified":"2024-08-03T20:53:25.076887+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6399},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6399":{"gid":6399,"name":"a","rank":0,"incarnation":70,"state":"up:active","state_seq":3,"addr":"172.21.15.32:6835/4240363667","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6833","nonce":4240363667},{"type":"v1","addr":"172.21.15.32:6835","nonce":4240363667}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6399,"qdb_cluster":[6399]},"id":5}]} 2024-08-03T20:53:43.378 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 75 2024-08-03T20:53:43.380 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:43.378+0000 7fd3af200640 1 -- 172.21.15.32:0/3944671577 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fd390077010 msgr2=0x7fd390079520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:53:43.380 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:43.378+0000 7fd3af200640 1 --2- 172.21.15.32:0/3944671577 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fd390077010 0x7fd390079520 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7fd3a0001720 tx=0x7fd3a0009040 comp rx=0 tx=0).stop 2024-08-03T20:53:43.380 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:43.378+0000 7fd3af200640 1 -- 172.21.15.32:0/3944671577 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd3b00ce150 msgr2=0x7fd3b01f60e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:53:43.380 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:43.378+0000 7fd3af200640 1 --2- 172.21.15.32:0/3944671577 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd3b00ce150 0x7fd3b01f60e0 secure :-1 s=READY pgs=800 cs=0 l=1 rev1=1 crypto rx=0x7fd39c006010 tx=0x7fd39c0309f0 comp rx=0 tx=0).stop 2024-08-03T20:53:43.380 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:43.378+0000 7fd3af200640 1 -- 172.21.15.32:0/3944671577 shutdown_connections 2024-08-03T20:53:43.380 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:43.378+0000 7fd3af200640 1 --2- 172.21.15.32:0/3944671577 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fd390077010 0x7fd390079520 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:53:43.380 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:43.378+0000 7fd3af200640 1 --2- 172.21.15.32:0/3944671577 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fd3b015a310 0x7fd3b014bab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:53:43.380 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:43.378+0000 7fd3af200640 1 --2- 172.21.15.32:0/3944671577 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd3b00ce150 0x7fd3b01f60e0 unknown :-1 s=CLOSED pgs=800 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:53:43.380 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:43.378+0000 7fd3af200640 1 --2- 172.21.15.32:0/3944671577 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fd3b00ca3b0 0x7fd3b0073080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:53:43.381 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:43.378+0000 7fd3af200640 1 -- 172.21.15.32:0/3944671577 >> 172.21.15.32:0/3944671577 conn(0x7fd3b00b2120 msgr2=0x7fd3b00c8e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:53:43.381 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:43.378+0000 7fd3af200640 1 -- 172.21.15.32:0/3944671577 shutdown_connections 2024-08-03T20:53:43.381 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:43.378+0000 7fd3af200640 1 -- 172.21.15.32:0/3944671577 wait complete. 2024-08-03T20:53:43.402 DEBUG:tasks.ceph_test_case:wait_until_true: waiting (timeout=45.0 retry_count=0)... 2024-08-03T20:53:44.568 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~7s 2024-08-03T20:53:48.403 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-08-03T20:53:48.642 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:48.640+0000 7f81cb800640 1 -- 172.21.15.32:0/3561506503 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f81c406e930 msgr2=0x7f81c4144f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:53:48.642 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:48.640+0000 7f81cb800640 1 --2- 172.21.15.32:0/3561506503 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f81c406e930 0x7f81c4144f90 secure :-1 s=READY pgs=801 cs=0 l=1 rev1=1 crypto rx=0x7f81c40afaf0 tx=0x7f81b402a4f0 comp rx=0 tx=0).stop 2024-08-03T20:53:48.642 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:48.640+0000 7f81cb800640 1 -- 172.21.15.32:0/3561506503 shutdown_connections 2024-08-03T20:53:48.642 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:48.640+0000 7f81cb800640 1 --2- 172.21.15.32:0/3561506503 >> v2:172.21.15.118:3300/0 conn(0x7f81c4073e60 0x7f81c4074270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:53:48.642 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:48.640+0000 7f81cb800640 1 --2- 172.21.15.32:0/3561506503 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f81c406e930 0x7f81c4144f90 unknown :-1 s=CLOSED pgs=801 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:53:48.642 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:48.640+0000 7f81cb800640 1 -- 172.21.15.32:0/3561506503 >> 172.21.15.32:0/3561506503 conn(0x7f81c40b1bd0 msgr2=0x7f81c40b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:53:48.642 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:48.640+0000 7f81cb800640 1 -- 172.21.15.32:0/3561506503 shutdown_connections 2024-08-03T20:53:48.642 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:48.640+0000 7f81cb800640 1 -- 172.21.15.32:0/3561506503 wait complete. 2024-08-03T20:53:48.643 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:48.642+0000 7f81cb800640 1 Processor -- start 2024-08-03T20:53:48.644 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:48.642+0000 7f81cb800640 1 -- start start 2024-08-03T20:53:48.646 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:48.644+0000 7f81cb800640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f81c406e930 0x7f81c40af0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:53:48.646 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:48.644+0000 7f81cb800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f81c4073e60 0x7f81c40aada0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:53:48.646 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:48.644+0000 7f81cb800640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f81c40747b0 0x7f81c40ab2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:53:48.646 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:48.644+0000 7f81cb800640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f81c40cf9c0 con 0x7f81c4073e60 2024-08-03T20:53:48.646 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:48.644+0000 7f81cb800640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f81c40cf840 con 0x7f81c40747b0 2024-08-03T20:53:48.647 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:48.644+0000 7f81cb800640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7f81c40ccff0 con 0x7f81c406e930 2024-08-03T20:53:48.647 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:48.644+0000 7f81bbe00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f81c4073e60 0x7f81c40aada0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:53:48.647 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:48.644+0000 7f81bbe00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f81c4073e60 0x7f81c40aada0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:43360/0 (socket says 172.21.15.32:43360) 2024-08-03T20:53:48.647 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:48.644+0000 7f81bbe00640 1 -- 172.21.15.32:0/831854698 learned_addr learned my addr 172.21.15.32:0/831854698 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:53:48.647 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:48.644+0000 7f81c9800640 1 --2- 172.21.15.32:0/831854698 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f81c40747b0 0x7f81c40ab2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:53:48.647 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:48.644+0000 7f81c8e00640 1 --2- 172.21.15.32:0/831854698 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f81c406e930 0x7f81c40af0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:53:48.647 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:48.645+0000 7f81bbe00640 1 -- 172.21.15.32:0/831854698 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f81c406e930 msgr2=0x7f81c40af0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:53:48.647 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:48.645+0000 7f81bbe00640 1 --2- 172.21.15.32:0/831854698 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f81c406e930 0x7f81c40af0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:53:48.647 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:48.645+0000 7f81bbe00640 1 -- 172.21.15.32:0/831854698 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f81c40747b0 msgr2=0x7f81c40ab2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:53:48.647 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:48.645+0000 7f81bbe00640 1 --2- 172.21.15.32:0/831854698 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f81c40747b0 0x7f81c40ab2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:53:48.647 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:48.645+0000 7f81bbe00640 1 -- 172.21.15.32:0/831854698 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f81c40abe20 con 0x7f81c4073e60 2024-08-03T20:53:48.647 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:48.645+0000 7f81c9800640 1 --2- 172.21.15.32:0/831854698 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f81c40747b0 0x7f81c40ab2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-03T20:53:48.647 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:48.645+0000 7f81bbe00640 1 --2- 172.21.15.32:0/831854698 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f81c4073e60 0x7f81c40aada0 secure :-1 s=READY pgs=802 cs=0 l=1 rev1=1 crypto rx=0x7f81ac00b440 tx=0x7f81ac009f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:53:48.648 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:48.645+0000 7f81c1600640 1 -- 172.21.15.32:0/831854698 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f81ac015020 con 0x7f81c4073e60 2024-08-03T20:53:48.648 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:48.645+0000 7f81c1600640 1 -- 172.21.15.32:0/831854698 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f81ac011940 con 0x7f81c4073e60 2024-08-03T20:53:48.648 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:48.645+0000 7f81cb800640 1 -- 172.21.15.32:0/831854698 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f81c41e5ff0 con 0x7f81c4073e60 2024-08-03T20:53:48.648 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:48.645+0000 7f81c1600640 1 -- 172.21.15.32:0/831854698 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f81ac011d90 con 0x7f81c4073e60 2024-08-03T20:53:48.648 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:48.645+0000 7f81cb800640 1 -- 172.21.15.32:0/831854698 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f81c41e62d0 con 0x7f81c4073e60 2024-08-03T20:53:48.648 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:48.646+0000 7f81cb800640 1 -- 172.21.15.32:0/831854698 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f81c40cd5a0 con 0x7f81c4073e60 2024-08-03T20:53:48.649 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:48.648+0000 7f81c1600640 1 -- 172.21.15.32:0/831854698 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f81ac018020 con 0x7f81c4073e60 2024-08-03T20:53:48.650 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:48.649+0000 7f81c1600640 1 --2- 172.21.15.32:0/831854698 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f81a0077010 0x7f81a0079520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:53:48.650 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:48.649+0000 7f81c1600640 1 -- 172.21.15.32:0/831854698 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(70..70 src has 1..70) ==== 5709+0+0 (secure 0 0 0) 0x7f81ac098890 con 0x7f81c4073e60 2024-08-03T20:53:48.651 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:48.649+0000 7f81c8e00640 1 --2- 172.21.15.32:0/831854698 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f81a0077010 0x7f81a0079520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:53:48.652 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:48.650+0000 7f81c8e00640 1 --2- 172.21.15.32:0/831854698 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f81a0077010 0x7f81a0079520 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f81b4033000 tx=0x7f81b4004730 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:53:48.655 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:48.654+0000 7f81c1600640 1 -- 172.21.15.32:0/831854698 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7f81ac01f020 con 0x7f81c4073e60 2024-08-03T20:53:48.898 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:48.896+0000 7f81cb800640 1 -- 172.21.15.32:0/831854698 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f81c40aa980 con 0x7f81c4073e60 2024-08-03T20:53:48.898 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:48.897+0000 7f81c1600640 1 -- 172.21.15.32:0/831854698 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 75 v75) ==== 94+0+5231 (secure 0 0 0) 0x7f81ac061a80 con 0x7f81c4073e60 2024-08-03T20:53:48.899 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-03T20:53:48.899 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":75,"btime":"2024-08-03T20:53:25:602045+0000","default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5666,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6837/2261538396","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6835","nonce":2261538396},{"type":"v1","addr":"172.21.15.118:6837","nonce":2261538396}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":69},{"gid":6061,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6836/1133580656","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6834","nonce":1133580656},{"type":"v1","addr":"172.21.15.118:6836","nonce":1133580656}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":68},{"gid":6396,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6834/1798758431","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":1798758431},{"type":"v1","addr":"172.21.15.32:6834","nonce":1798758431}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":69}],"filesystems":[{"mdsmap":{"epoch":75,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-08-03T20:53:15.080669+0000","modified":"2024-08-03T20:53:25.076887+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6399},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6399":{"gid":6399,"name":"a","rank":0,"incarnation":70,"state":"up:active","state_seq":3,"addr":"172.21.15.32:6835/4240363667","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6833","nonce":4240363667},{"type":"v1","addr":"172.21.15.32:6835","nonce":4240363667}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6399,"qdb_cluster":[6399]},"id":5}]} 2024-08-03T20:53:48.899 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 75 2024-08-03T20:53:48.901 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:48.899+0000 7f81cb800640 1 -- 172.21.15.32:0/831854698 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f81a0077010 msgr2=0x7f81a0079520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:53:48.901 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:48.899+0000 7f81cb800640 1 --2- 172.21.15.32:0/831854698 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f81a0077010 0x7f81a0079520 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f81b4033000 tx=0x7f81b4004730 comp rx=0 tx=0).stop 2024-08-03T20:53:48.901 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:48.899+0000 7f81cb800640 1 -- 172.21.15.32:0/831854698 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f81c4073e60 msgr2=0x7f81c40aada0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:53:48.901 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:48.899+0000 7f81cb800640 1 --2- 172.21.15.32:0/831854698 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f81c4073e60 0x7f81c40aada0 secure :-1 s=READY pgs=802 cs=0 l=1 rev1=1 crypto rx=0x7f81ac00b440 tx=0x7f81ac009f90 comp rx=0 tx=0).stop 2024-08-03T20:53:48.901 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:48.899+0000 7f81cb800640 1 -- 172.21.15.32:0/831854698 shutdown_connections 2024-08-03T20:53:48.902 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:48.899+0000 7f81cb800640 1 --2- 172.21.15.32:0/831854698 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f81a0077010 0x7f81a0079520 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:53:48.902 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:48.899+0000 7f81cb800640 1 --2- 172.21.15.32:0/831854698 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f81c40747b0 0x7f81c40ab2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:53:48.902 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:48.899+0000 7f81cb800640 1 --2- 172.21.15.32:0/831854698 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f81c4073e60 0x7f81c40aada0 unknown :-1 s=CLOSED pgs=802 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:53:48.902 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:48.899+0000 7f81cb800640 1 --2- 172.21.15.32:0/831854698 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f81c406e930 0x7f81c40af0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:53:48.902 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:48.899+0000 7f81cb800640 1 -- 172.21.15.32:0/831854698 >> 172.21.15.32:0/831854698 conn(0x7f81c40b1bd0 msgr2=0x7f81c40b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:53:48.902 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:48.900+0000 7f81cb800640 1 -- 172.21.15.32:0/831854698 shutdown_connections 2024-08-03T20:53:48.902 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:48.900+0000 7f81cb800640 1 -- 172.21.15.32:0/831854698 wait complete. 2024-08-03T20:53:48.928 DEBUG:tasks.ceph_test_case:wait_until_true: waiting (timeout=45.0 retry_count=0)... 2024-08-03T20:53:51.181 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~13s 2024-08-03T20:53:53.930 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-08-03T20:53:54.172 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:54.171+0000 7fbb43200640 1 -- 172.21.15.32:0/1229046241 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbb3c165160 msgr2=0x7fbb3c167580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:53:54.172 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:54.171+0000 7fbb43200640 1 --2- 172.21.15.32:0/1229046241 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbb3c165160 0x7fbb3c167580 secure :-1 s=READY pgs=804 cs=0 l=1 rev1=1 crypto rx=0x7fbb34005fd0 tx=0x7fbb3402a8b0 comp rx=0 tx=0).stop 2024-08-03T20:53:54.172 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:54.171+0000 7fbb43200640 1 -- 172.21.15.32:0/1229046241 shutdown_connections 2024-08-03T20:53:54.172 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:54.171+0000 7fbb43200640 1 --2- 172.21.15.32:0/1229046241 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbb3c165160 0x7fbb3c167580 unknown :-1 s=CLOSED pgs=804 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:53:54.172 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:54.171+0000 7fbb43200640 1 -- 172.21.15.32:0/1229046241 >> 172.21.15.32:0/1229046241 conn(0x7fbb3c0b1bd0 msgr2=0x7fbb3c0b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:53:54.172 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:54.171+0000 7fbb43200640 1 -- 172.21.15.32:0/1229046241 shutdown_connections 2024-08-03T20:53:54.173 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:54.171+0000 7fbb43200640 1 -- 172.21.15.32:0/1229046241 wait complete. 2024-08-03T20:53:54.174 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:54.173+0000 7fbb43200640 1 Processor -- start 2024-08-03T20:53:54.174 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:54.173+0000 7fbb43200640 1 -- start start 2024-08-03T20:53:54.175 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:54.175+0000 7fbb43200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbb3c06e930 0x7fbb3c156aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:53:54.175 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:54.175+0000 7fbb43200640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fbb3c144f90 0x7fbb3c0b3160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:53:54.175 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:54.175+0000 7fbb43200640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fbb3c165160 0x7fbb3c1593f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:53:54.176 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:54.175+0000 7fbb43200640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fbb3c0cf9c0 con 0x7fbb3c06e930 2024-08-03T20:53:54.176 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:54.175+0000 7fbb43200640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7fbb3c0cf840 con 0x7fbb3c165160 2024-08-03T20:53:54.176 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:54.175+0000 7fbb43200640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7fbb3c167ac0 con 0x7fbb3c144f90 2024-08-03T20:53:54.177 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:54.175+0000 7fbb3be00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbb3c06e930 0x7fbb3c156aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:53:54.177 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:54.175+0000 7fbb3be00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbb3c06e930 0x7fbb3c156aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:60202/0 (socket says 172.21.15.32:60202) 2024-08-03T20:53:54.177 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:54.175+0000 7fbb3be00640 1 -- 172.21.15.32:0/2451800937 learned_addr learned my addr 172.21.15.32:0/2451800937 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:53:54.177 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:54.176+0000 7fbb41200640 1 --2- 172.21.15.32:0/2451800937 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fbb3c165160 0x7fbb3c1593f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:53:54.177 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:54.176+0000 7fbb33e00640 1 --2- 172.21.15.32:0/2451800937 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fbb3c144f90 0x7fbb3c0b3160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:53:54.177 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:54.176+0000 7fbb3be00640 1 -- 172.21.15.32:0/2451800937 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fbb3c144f90 msgr2=0x7fbb3c0b3160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:53:54.177 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:54.176+0000 7fbb3be00640 1 --2- 172.21.15.32:0/2451800937 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fbb3c144f90 0x7fbb3c0b3160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:53:54.177 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:54.176+0000 7fbb3be00640 1 -- 172.21.15.32:0/2451800937 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fbb3c165160 msgr2=0x7fbb3c1593f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:53:54.178 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:54.176+0000 7fbb3be00640 1 --2- 172.21.15.32:0/2451800937 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fbb3c165160 0x7fbb3c1593f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:53:54.178 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:54.176+0000 7fbb3be00640 1 -- 172.21.15.32:0/2451800937 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbb3c158720 con 0x7fbb3c06e930 2024-08-03T20:53:54.178 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:54.176+0000 7fbb3be00640 1 --2- 172.21.15.32:0/2451800937 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbb3c06e930 0x7fbb3c156aa0 secure :-1 s=READY pgs=805 cs=0 l=1 rev1=1 crypto rx=0x7fbb2c00c370 tx=0x7fbb2c00c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:53:54.178 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:54.177+0000 7fbb38c00640 1 -- 172.21.15.32:0/2451800937 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb2c017070 con 0x7fbb3c06e930 2024-08-03T20:53:54.178 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:54.177+0000 7fbb38c00640 1 -- 172.21.15.32:0/2451800937 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fbb2c0035f0 con 0x7fbb3c06e930 2024-08-03T20:53:54.178 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:54.177+0000 7fbb38c00640 1 -- 172.21.15.32:0/2451800937 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb2c003a40 con 0x7fbb3c06e930 2024-08-03T20:53:54.178 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:54.177+0000 7fbb43200640 1 -- 172.21.15.32:0/2451800937 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbb3c158a00 con 0x7fbb3c06e930 2024-08-03T20:53:54.178 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:54.177+0000 7fbb43200640 1 -- 172.21.15.32:0/2451800937 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbb3c0b3e60 con 0x7fbb3c06e930 2024-08-03T20:53:54.178 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:54.178+0000 7fbb43200640 1 -- 172.21.15.32:0/2451800937 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbb3c073e60 con 0x7fbb3c06e930 2024-08-03T20:53:54.179 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:54.179+0000 7fbb38c00640 1 -- 172.21.15.32:0/2451800937 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7fbb2c01a070 con 0x7fbb3c06e930 2024-08-03T20:53:54.180 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:54.180+0000 7fbb38c00640 1 --2- 172.21.15.32:0/2451800937 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fbb18077010 0x7fbb18079520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:53:54.180 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:54.180+0000 7fbb38c00640 1 -- 172.21.15.32:0/2451800937 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(70..70 src has 1..70) ==== 5709+0+0 (secure 0 0 0) 0x7fbb2c097bb0 con 0x7fbb3c06e930 2024-08-03T20:53:54.181 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:54.180+0000 7fbb33e00640 1 --2- 172.21.15.32:0/2451800937 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fbb18077010 0x7fbb18079520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:53:54.181 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:54.181+0000 7fbb33e00640 1 --2- 172.21.15.32:0/2451800937 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fbb18077010 0x7fbb18079520 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7fbb24000f60 tx=0x7fbb24009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:53:54.185 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:54.185+0000 7fbb38c00640 1 -- 172.21.15.32:0/2451800937 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7fbb2c060750 con 0x7fbb3c06e930 2024-08-03T20:53:54.427 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:54.426+0000 7fbb43200640 1 -- 172.21.15.32:0/2451800937 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fbb3c158bd0 con 0x7fbb3c06e930 2024-08-03T20:53:54.427 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:54.427+0000 7fbb38c00640 1 -- 172.21.15.32:0/2451800937 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 76 v76) ==== 94+0+5279 (secure 0 0 0) 0x7fbb2c064ca0 con 0x7fbb3c06e930 2024-08-03T20:53:54.428 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-03T20:53:54.428 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":76,"btime":"2024-08-03T20:53:49:989325+0000","default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5666,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6837/2261538396","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6835","nonce":2261538396},{"type":"v1","addr":"172.21.15.118:6837","nonce":2261538396}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":69},{"gid":6061,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6836/1133580656","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6834","nonce":1133580656},{"type":"v1","addr":"172.21.15.118:6836","nonce":1133580656}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":68},{"gid":6396,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6834/1798758431","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":1798758431},{"type":"v1","addr":"172.21.15.32:6834","nonce":1798758431}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":69}],"filesystems":[{"mdsmap":{"epoch":76,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-08-03T20:53:15.080669+0000","modified":"2024-08-03T20:53:49.989259+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6399},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6399":{"gid":6399,"name":"a","rank":0,"incarnation":70,"state":"up:active","state_seq":3,"addr":"172.21.15.32:6835/4240363667","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6833","nonce":4240363667},{"type":"v1","addr":"172.21.15.32:6835","nonce":4240363667}]},"join_fscid":-1,"laggy_since":"2024-08-03T20:53:49.989259+0000","export_targets":[],"features":4540701547738038271,"flags":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6399,"qdb_cluster":[6399]},"id":5}]} 2024-08-03T20:53:54.428 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 76 2024-08-03T20:53:54.430 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:54.429+0000 7fbb43200640 1 -- 172.21.15.32:0/2451800937 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fbb18077010 msgr2=0x7fbb18079520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:53:54.430 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:54.429+0000 7fbb43200640 1 --2- 172.21.15.32:0/2451800937 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fbb18077010 0x7fbb18079520 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7fbb24000f60 tx=0x7fbb24009040 comp rx=0 tx=0).stop 2024-08-03T20:53:54.430 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:54.429+0000 7fbb43200640 1 -- 172.21.15.32:0/2451800937 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbb3c06e930 msgr2=0x7fbb3c156aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:53:54.430 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:54.429+0000 7fbb43200640 1 --2- 172.21.15.32:0/2451800937 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbb3c06e930 0x7fbb3c156aa0 secure :-1 s=READY pgs=805 cs=0 l=1 rev1=1 crypto rx=0x7fbb2c00c370 tx=0x7fbb2c00c830 comp rx=0 tx=0).stop 2024-08-03T20:53:54.430 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:54.429+0000 7fbb43200640 1 -- 172.21.15.32:0/2451800937 shutdown_connections 2024-08-03T20:53:54.430 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:54.429+0000 7fbb43200640 1 --2- 172.21.15.32:0/2451800937 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fbb18077010 0x7fbb18079520 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:53:54.431 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:54.429+0000 7fbb43200640 1 --2- 172.21.15.32:0/2451800937 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fbb3c165160 0x7fbb3c1593f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:53:54.431 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:54.429+0000 7fbb43200640 1 --2- 172.21.15.32:0/2451800937 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fbb3c144f90 0x7fbb3c0b3160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:53:54.431 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:54.429+0000 7fbb43200640 1 --2- 172.21.15.32:0/2451800937 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbb3c06e930 0x7fbb3c156aa0 unknown :-1 s=CLOSED pgs=805 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:53:54.431 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:54.429+0000 7fbb43200640 1 -- 172.21.15.32:0/2451800937 >> 172.21.15.32:0/2451800937 conn(0x7fbb3c0b1bd0 msgr2=0x7fbb3c0cc540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:53:54.431 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:54.429+0000 7fbb43200640 1 -- 172.21.15.32:0/2451800937 shutdown_connections 2024-08-03T20:53:54.431 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:54.429+0000 7fbb43200640 1 -- 172.21.15.32:0/2451800937 wait complete. 2024-08-03T20:53:54.456 DEBUG:tasks.ceph_test_case:wait_until_true: success in 20s and 0 retries 2024-08-03T20:53:54.557 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mds mds_inject_rename_corrupt_dentry_first 2024-08-03T20:53:54.788 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:54.787+0000 7f36f9600640 1 -- 172.21.15.32:0/2057495616 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f36f406e730 msgr2=0x7f36f40cdb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:53:54.788 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:54.787+0000 7f36f9600640 1 --2- 172.21.15.32:0/2057495616 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f36f406e730 0x7f36f40cdb20 secure :-1 s=READY pgs=806 cs=0 l=1 rev1=1 crypto rx=0x7f36f40700b0 tx=0x7f36dc02a560 comp rx=0 tx=0).stop 2024-08-03T20:53:54.789 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:54.787+0000 7f36f9600640 1 -- 172.21.15.32:0/2057495616 shutdown_connections 2024-08-03T20:53:54.789 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:54.787+0000 7f36f9600640 1 --2- 172.21.15.32:0/2057495616 >> v2:172.21.15.118:3300/0 conn(0x7f36f40ce060 0x7f36f40ce470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:53:54.789 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:54.787+0000 7f36f9600640 1 --2- 172.21.15.32:0/2057495616 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f36f406e730 0x7f36f40cdb20 unknown :-1 s=CLOSED pgs=806 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:53:54.789 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:54.788+0000 7f36f9600640 1 --2- 172.21.15.32:0/2057495616 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f36f4158d70 0x7f36f415eb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:53:54.789 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:54.788+0000 7f36f9600640 1 -- 172.21.15.32:0/2057495616 >> 172.21.15.32:0/2057495616 conn(0x7f36f40cfd40 msgr2=0x7f36f40d0150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:53:54.789 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:54.788+0000 7f36f9600640 1 -- 172.21.15.32:0/2057495616 shutdown_connections 2024-08-03T20:53:54.789 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:54.788+0000 7f36f9600640 1 -- 172.21.15.32:0/2057495616 wait complete. 2024-08-03T20:53:54.790 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:54.790+0000 7f36f9600640 1 Processor -- start 2024-08-03T20:53:54.790 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:54.790+0000 7f36f9600640 1 -- start start 2024-08-03T20:53:54.792 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:54.791+0000 7f36f9600640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f36f406e730 0x7f36f40b5620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:53:54.793 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:54.791+0000 7f36f9600640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f36f40ce060 0x7f36f4154d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:53:54.793 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:54.791+0000 7f36f9600640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f36f4158d70 0x7f36f40cb530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:53:54.793 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:54.791+0000 7f36f9600640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f36f4162600 con 0x7f36f40ce060 2024-08-03T20:53:54.793 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:54.791+0000 7f36f9600640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f36f4162480 con 0x7f36f406e730 2024-08-03T20:53:54.793 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:54.791+0000 7f36f9600640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7f36f4162780 con 0x7f36f4158d70 2024-08-03T20:53:54.793 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:54.792+0000 7f36ebe00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f36f40ce060 0x7f36f4154d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:53:54.793 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:54.792+0000 7f36ebe00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f36f40ce060 0x7f36f4154d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:60210/0 (socket says 172.21.15.32:60210) 2024-08-03T20:53:54.793 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:54.792+0000 7f36ebe00640 1 -- 172.21.15.32:0/567094282 learned_addr learned my addr 172.21.15.32:0/567094282 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:53:54.793 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:54.792+0000 7f36f8c00640 1 --2- 172.21.15.32:0/567094282 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f36f4158d70 0x7f36f40cb530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:53:54.793 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:54.792+0000 7f36f3e00640 1 --2- 172.21.15.32:0/567094282 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f36f406e730 0x7f36f40b5620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:53:54.793 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:54.792+0000 7f36ebe00640 1 -- 172.21.15.32:0/567094282 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f36f4158d70 msgr2=0x7f36f40cb530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:53:54.793 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:54.792+0000 7f36ebe00640 1 --2- 172.21.15.32:0/567094282 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f36f4158d70 0x7f36f40cb530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:53:54.794 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:54.792+0000 7f36ebe00640 1 -- 172.21.15.32:0/567094282 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f36f406e730 msgr2=0x7f36f40b5620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:53:54.794 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:54.792+0000 7f36ebe00640 1 --2- 172.21.15.32:0/567094282 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f36f406e730 0x7f36f40b5620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:53:54.794 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:54.792+0000 7f36ebe00640 1 -- 172.21.15.32:0/567094282 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f36f41580a0 con 0x7f36f40ce060 2024-08-03T20:53:54.794 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:54.792+0000 7f36ebe00640 1 --2- 172.21.15.32:0/567094282 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f36f40ce060 0x7f36f4154d10 secure :-1 s=READY pgs=807 cs=0 l=1 rev1=1 crypto rx=0x7f36dc034040 tx=0x7f36dc0039e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:53:54.795 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:54.793+0000 7f36f0c00640 1 -- 172.21.15.32:0/567094282 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36dc03e030 con 0x7f36f40ce060 2024-08-03T20:53:54.795 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:54.793+0000 7f36f0c00640 1 -- 172.21.15.32:0/567094282 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f36dc0047a0 con 0x7f36f40ce060 2024-08-03T20:53:54.795 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:54.793+0000 7f36f9600640 1 -- 172.21.15.32:0/567094282 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f36f415d930 con 0x7f36f40ce060 2024-08-03T20:53:54.795 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:54.793+0000 7f36f0c00640 1 -- 172.21.15.32:0/567094282 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36dc035040 con 0x7f36f40ce060 2024-08-03T20:53:54.795 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:54.793+0000 7f36f9600640 1 -- 172.21.15.32:0/567094282 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f36f41578a0 con 0x7f36f40ce060 2024-08-03T20:53:54.795 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:54.794+0000 7f36f9600640 1 -- 172.21.15.32:0/567094282 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f36f4162d10 con 0x7f36f40ce060 2024-08-03T20:53:54.795 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:54.795+0000 7f36f0c00640 1 -- 172.21.15.32:0/567094282 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f36dc041030 con 0x7f36f40ce060 2024-08-03T20:53:54.796 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:54.796+0000 7f36f0c00640 1 --2- 172.21.15.32:0/567094282 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f36d0077040 0x7f36d0079550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:53:54.796 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:54.796+0000 7f36f0c00640 1 -- 172.21.15.32:0/567094282 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(70..70 src has 1..70) ==== 5709+0+0 (secure 0 0 0) 0x7f36dc0b90f0 con 0x7f36f40ce060 2024-08-03T20:53:54.797 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:54.796+0000 7f36f3e00640 1 --2- 172.21.15.32:0/567094282 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f36d0077040 0x7f36d0079550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:53:54.798 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:54.797+0000 7f36f3e00640 1 --2- 172.21.15.32:0/567094282 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f36d0077040 0x7f36d0079550 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f36e4005fd0 tx=0x7f36e4005850 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:53:54.801 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:54.800+0000 7f36f0c00640 1 -- 172.21.15.32:0/567094282 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7f36dc0822e0 con 0x7f36f40ce060 2024-08-03T20:53:55.003 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:55.003+0000 7f36f9600640 1 -- 172.21.15.32:0/567094282 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "config rm", "who": "mds", "name": "mds_inject_rename_corrupt_dentry_first"} v 0) -- 0x7f36f40b5cc0 con 0x7f36f40ce060 2024-08-03T20:53:55.010 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:55.010+0000 7f36f0c00640 1 -- 172.21.15.32:0/567094282 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mds", "name": "mds_inject_rename_corrupt_dentry_first"}]=0 v41) ==== 121+0+0 (secure 0 0 0) 0x7f36dc081be0 con 0x7f36f40ce060 2024-08-03T20:53:55.013 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:55.013+0000 7f36f9600640 1 -- 172.21.15.32:0/567094282 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f36d0077040 msgr2=0x7f36d0079550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:53:55.014 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:55.013+0000 7f36f9600640 1 --2- 172.21.15.32:0/567094282 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f36d0077040 0x7f36d0079550 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f36e4005fd0 tx=0x7f36e4005850 comp rx=0 tx=0).stop 2024-08-03T20:53:55.014 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:55.013+0000 7f36f9600640 1 -- 172.21.15.32:0/567094282 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f36f40ce060 msgr2=0x7f36f4154d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:53:55.014 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:55.013+0000 7f36f9600640 1 --2- 172.21.15.32:0/567094282 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f36f40ce060 0x7f36f4154d10 secure :-1 s=READY pgs=807 cs=0 l=1 rev1=1 crypto rx=0x7f36dc034040 tx=0x7f36dc0039e0 comp rx=0 tx=0).stop 2024-08-03T20:53:55.014 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:55.013+0000 7f36f9600640 1 -- 172.21.15.32:0/567094282 shutdown_connections 2024-08-03T20:53:55.014 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:55.013+0000 7f36f9600640 1 --2- 172.21.15.32:0/567094282 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f36d0077040 0x7f36d0079550 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:53:55.014 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:55.013+0000 7f36f9600640 1 --2- 172.21.15.32:0/567094282 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f36f4158d70 0x7f36f40cb530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:53:55.014 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:55.013+0000 7f36f9600640 1 --2- 172.21.15.32:0/567094282 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f36f40ce060 0x7f36f4154d10 unknown :-1 s=CLOSED pgs=807 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:53:55.014 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:55.013+0000 7f36f9600640 1 --2- 172.21.15.32:0/567094282 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f36f406e730 0x7f36f40b5620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:53:55.014 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:55.013+0000 7f36f9600640 1 -- 172.21.15.32:0/567094282 >> 172.21.15.32:0/567094282 conn(0x7f36f40cfd40 msgr2=0x7f36f4073550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:53:55.014 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:55.013+0000 7f36f9600640 1 -- 172.21.15.32:0/567094282 shutdown_connections 2024-08-03T20:53:55.014 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:55.013+0000 7f36f9600640 1 -- 172.21.15.32:0/567094282 wait complete. 2024-08-03T20:53:55.040 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mds freeze 5:0 false 2024-08-03T20:53:55.270 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:55.269+0000 7faafce00640 1 -- 172.21.15.32:0/3061731040 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faaf815ee60 msgr2=0x7faaf815f270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:53:55.270 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:55.269+0000 7faafce00640 1 --2- 172.21.15.32:0/3061731040 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faaf815ee60 0x7faaf815f270 secure :-1 s=READY pgs=808 cs=0 l=1 rev1=1 crypto rx=0x7faae0005fd0 tx=0x7faae002a4a0 comp rx=0 tx=0).stop 2024-08-03T20:53:55.270 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:55.269+0000 7faafce00640 1 -- 172.21.15.32:0/3061731040 shutdown_connections 2024-08-03T20:53:55.270 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:55.269+0000 7faafce00640 1 --2- 172.21.15.32:0/3061731040 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7faaf8151b20 0x7faaf8151f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:53:55.270 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:55.269+0000 7faafce00640 1 --2- 172.21.15.32:0/3061731040 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faaf815ee60 0x7faaf815f270 unknown :-1 s=CLOSED pgs=808 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:53:55.271 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:55.269+0000 7faafce00640 1 -- 172.21.15.32:0/3061731040 >> 172.21.15.32:0/3061731040 conn(0x7faaf80cfd40 msgr2=0x7faaf80d0150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:53:55.271 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:55.269+0000 7faafce00640 1 -- 172.21.15.32:0/3061731040 shutdown_connections 2024-08-03T20:53:55.271 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:55.269+0000 7faafce00640 1 -- 172.21.15.32:0/3061731040 wait complete. 2024-08-03T20:53:55.271 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:55.271+0000 7faafce00640 1 Processor -- start 2024-08-03T20:53:55.272 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:55.271+0000 7faafce00640 1 -- start start 2024-08-03T20:53:55.275 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:55.273+0000 7faafce00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faaf806e630 0x7faaf80c3870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:53:55.275 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:55.273+0000 7faafce00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7faaf8151b20 0x7faaf80c9670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:53:55.275 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:55.273+0000 7faafce00640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7faaf815ee60 0x7faaf81549b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:53:55.275 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:55.273+0000 7faafce00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7faaf8152470 con 0x7faaf806e630 2024-08-03T20:53:55.275 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:55.273+0000 7faafce00640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7faaf806ec50 con 0x7faaf8151b20 2024-08-03T20:53:55.275 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:55.273+0000 7faafce00640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7faaf81525f0 con 0x7faaf815ee60 2024-08-03T20:53:55.275 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:55.273+0000 7faaf7400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faaf806e630 0x7faaf80c3870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:53:55.275 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:55.273+0000 7faaf7400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faaf806e630 0x7faaf80c3870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:60224/0 (socket says 172.21.15.32:60224) 2024-08-03T20:53:55.275 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:55.273+0000 7faaf7400640 1 -- 172.21.15.32:0/1951829825 learned_addr learned my addr 172.21.15.32:0/1951829825 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:53:55.275 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:55.273+0000 7faaf7e00640 1 --2- 172.21.15.32:0/1951829825 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7faaf815ee60 0x7faaf81549b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:53:55.275 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:55.273+0000 7faaefe00640 1 --2- 172.21.15.32:0/1951829825 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7faaf8151b20 0x7faaf80c9670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:53:55.276 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:55.274+0000 7faaf7400640 1 -- 172.21.15.32:0/1951829825 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7faaf815ee60 msgr2=0x7faaf81549b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:53:55.276 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:55.274+0000 7faaf7400640 1 --2- 172.21.15.32:0/1951829825 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7faaf815ee60 0x7faaf81549b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:53:55.276 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:55.274+0000 7faaf7400640 1 -- 172.21.15.32:0/1951829825 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7faaf8151b20 msgr2=0x7faaf80c9670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:53:55.276 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:55.274+0000 7faaf7400640 1 --2- 172.21.15.32:0/1951829825 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7faaf8151b20 0x7faaf80c9670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:53:55.276 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:55.274+0000 7faaf7400640 1 -- 172.21.15.32:0/1951829825 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faaf81f8a30 con 0x7faaf806e630 2024-08-03T20:53:55.276 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:55.274+0000 7faaefe00640 1 --2- 172.21.15.32:0/1951829825 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7faaf8151b20 0x7faaf80c9670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-03T20:53:55.276 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:55.274+0000 7faaf7400640 1 --2- 172.21.15.32:0/1951829825 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faaf806e630 0x7faaf80c3870 secure :-1 s=READY pgs=809 cs=0 l=1 rev1=1 crypto rx=0x7faae800b3f0 tx=0x7faae800b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:53:55.276 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:55.274+0000 7faaef400640 1 -- 172.21.15.32:0/1951829825 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faae8015020 con 0x7faaf806e630 2024-08-03T20:53:55.276 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:55.274+0000 7faaef400640 1 -- 172.21.15.32:0/1951829825 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7faae8011920 con 0x7faaf806e630 2024-08-03T20:53:55.276 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:55.274+0000 7faaef400640 1 -- 172.21.15.32:0/1951829825 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faae8011d70 con 0x7faaf806e630 2024-08-03T20:53:55.276 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:55.274+0000 7faafce00640 1 -- 172.21.15.32:0/1951829825 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faaf80c9bb0 con 0x7faaf806e630 2024-08-03T20:53:55.276 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:55.274+0000 7faafce00640 1 -- 172.21.15.32:0/1951829825 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faaf81f8ff0 con 0x7faaf806e630 2024-08-03T20:53:55.277 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:55.275+0000 7faafce00640 1 -- 172.21.15.32:0/1951829825 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faaf81f96b0 con 0x7faaf806e630 2024-08-03T20:53:55.277 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:55.277+0000 7faaef400640 1 -- 172.21.15.32:0/1951829825 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7faae8018020 con 0x7faaf806e630 2024-08-03T20:53:55.278 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:55.277+0000 7faaef400640 1 --2- 172.21.15.32:0/1951829825 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7faad4077010 0x7faad4079520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:53:55.278 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:55.278+0000 7faaef400640 1 -- 172.21.15.32:0/1951829825 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(70..70 src has 1..70) ==== 5709+0+0 (secure 0 0 0) 0x7faae8098580 con 0x7faaf806e630 2024-08-03T20:53:55.278 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:55.278+0000 7faaefe00640 1 --2- 172.21.15.32:0/1951829825 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7faad4077010 0x7faad4079520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:53:55.279 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:55.279+0000 7faaefe00640 1 --2- 172.21.15.32:0/1951829825 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7faad4077010 0x7faad4079520 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7faae0033000 tx=0x7faae0004730 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:53:55.283 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:55.282+0000 7faaef400640 1 -- 172.21.15.32:0/1951829825 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7faae801f030 con 0x7faaf806e630 2024-08-03T20:53:55.569 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:55.568+0000 7faafce00640 1 -- 172.21.15.32:0/1951829825 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "mds freeze", "role_or_gid": "5:0", "val": "false"} v 0) -- 0x7faaf8151f30 con 0x7faaf806e630 2024-08-03T20:53:56.016 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:56.016+0000 7faaef400640 1 -- 172.21.15.32:0/1951829825 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "mds freeze", "role_or_gid": "5:0", "val": "false"}]=0 unfreezing mds.6399 v77) ==== 115+0+0 (secure 0 0 0) 0x7faae8061780 con 0x7faaf806e630 2024-08-03T20:53:56.017 INFO:teuthology.orchestra.run.smithi032.stderr:unfreezing mds.6399 2024-08-03T20:53:56.020 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:56.019+0000 7faafce00640 1 -- 172.21.15.32:0/1951829825 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7faad4077010 msgr2=0x7faad4079520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:53:56.020 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:56.019+0000 7faafce00640 1 --2- 172.21.15.32:0/1951829825 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7faad4077010 0x7faad4079520 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7faae0033000 tx=0x7faae0004730 comp rx=0 tx=0).stop 2024-08-03T20:53:56.020 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:56.019+0000 7faafce00640 1 -- 172.21.15.32:0/1951829825 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faaf806e630 msgr2=0x7faaf80c3870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:53:56.020 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:56.019+0000 7faafce00640 1 --2- 172.21.15.32:0/1951829825 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faaf806e630 0x7faaf80c3870 secure :-1 s=READY pgs=809 cs=0 l=1 rev1=1 crypto rx=0x7faae800b3f0 tx=0x7faae800b8b0 comp rx=0 tx=0).stop 2024-08-03T20:53:56.020 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:56.019+0000 7faafce00640 1 -- 172.21.15.32:0/1951829825 shutdown_connections 2024-08-03T20:53:56.020 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:56.019+0000 7faafce00640 1 --2- 172.21.15.32:0/1951829825 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7faad4077010 0x7faad4079520 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:53:56.020 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:56.019+0000 7faafce00640 1 --2- 172.21.15.32:0/1951829825 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7faaf8151b20 0x7faaf80c9670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:53:56.020 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:56.019+0000 7faafce00640 1 --2- 172.21.15.32:0/1951829825 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7faaf815ee60 0x7faaf81549b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:53:56.020 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:56.019+0000 7faafce00640 1 --2- 172.21.15.32:0/1951829825 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faaf806e630 0x7faaf80c3870 unknown :-1 s=CLOSED pgs=809 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:53:56.020 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:56.019+0000 7faafce00640 1 -- 172.21.15.32:0/1951829825 >> 172.21.15.32:0/1951829825 conn(0x7faaf80cfd40 msgr2=0x7faaf815daf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:53:56.020 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:56.019+0000 7faafce00640 1 -- 172.21.15.32:0/1951829825 shutdown_connections 2024-08-03T20:53:56.021 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:56.019+0000 7faafce00640 1 -- 172.21.15.32:0/1951829825 wait complete. 2024-08-03T20:53:56.052 DEBUG:teuthology.orchestra.run.smithi032:> sudo sysctl -n kernel.core_pattern 2024-08-03T20:53:56.106 INFO:teuthology.orchestra.run.smithi032.stdout:/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-03T20:53:56.108 INFO:tasks.cephfs.cephfs_test_case:Clearing core from directory: /home/ubuntu/cephtest/archive/coredump 2024-08-03T20:53:56.109 DEBUG:teuthology.orchestra.run.smithi032:> cd /home/ubuntu/cephtest/archive/coredump && sudo ls | sudo xargs file 2024-08-03T20:53:56.171 INFO:teuthology.orchestra.run.smithi032.stdout:1722718411.57982.core: ELF 64-bit LSB core file, x86-64, version 1 (SYSV), SVR4-style, from 'ceph-mds -f --cluster ceph -i a', real uid: 0, effective uid: 0, real gid: 0, effective gid: 0, execfn: '/bin/ceph-mds', platform: 'x86_64' 2024-08-03T20:53:56.175 INFO:tasks.cephfs.cephfs_test_case:Enumerated cores: ['1722718411.57982.core'] 2024-08-03T20:53:56.175 INFO:tasks.cephfs.cephfs_test_case:Found core file 1722718411.57982.core, deleting it 2024-08-03T20:53:56.175 DEBUG:teuthology.orchestra.run.smithi032:> cd /home/ubuntu/cephtest/archive/coredump && sudo rm -f 1722718411.57982.core 2024-08-03T20:53:56.270 INFO:tasks.ceph.mds.a:Restarting daemon 2024-08-03T20:53:56.270 INFO:tasks.ceph.mds.a:Stopping old one... 2024-08-03T20:53:56.270 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-08-03T20:53:56.270 INFO:teuthology.orchestra.run:waiting for 300 2024-08-03T20:53:56.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-03T20:53:56.271 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_7d7f5fe79c7b6b5080402153d07fb1acc4ce00a2/teuthology/orchestra/daemon/state.py", line 146, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_7d7f5fe79c7b6b5080402153d07fb1acc4ce00a2/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_7d7f5fe79c7b6b5080402153d07fb1acc4ce00a2/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_7d7f5fe79c7b6b5080402153d07fb1acc4ce00a2/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi032 with status 1: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a' 2024-08-03T20:53:56.271 INFO:tasks.ceph.mds.a:Stopped 2024-08-03T20:53:56.271 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-08-03T20:53:56.275 INFO:tasks.ceph.mds.a:Started 2024-08-03T20:53:56.276 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-08-03T20:53:56.469 INFO:tasks.ceph.mds.a.smithi032.stdout:starting mds.a at 2024-08-03T20:53:56.505 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:56.504+0000 7fbc98200640 1 -- 172.21.15.32:0/1618957013 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbc90144f90 msgr2=0x7fbc90074570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:53:56.505 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:56.504+0000 7fbc98200640 1 --2- 172.21.15.32:0/1618957013 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbc90144f90 0x7fbc90074570 secure :-1 s=READY pgs=813 cs=0 l=1 rev1=1 crypto rx=0x7fbc90072d80 tx=0x7fbc7c02a5f0 comp rx=0 tx=0).stop 2024-08-03T20:53:56.505 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:56.504+0000 7fbc98200640 1 -- 172.21.15.32:0/1618957013 shutdown_connections 2024-08-03T20:53:56.505 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:56.504+0000 7fbc98200640 1 --2- 172.21.15.32:0/1618957013 >> v2:172.21.15.118:3300/0 conn(0x7fbc9015fff0 0x7fbc90160400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:53:56.505 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:56.504+0000 7fbc98200640 1 --2- 172.21.15.32:0/1618957013 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbc90144f90 0x7fbc90074570 unknown :-1 s=CLOSED pgs=813 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:53:56.505 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:56.504+0000 7fbc98200640 1 -- 172.21.15.32:0/1618957013 >> 172.21.15.32:0/1618957013 conn(0x7fbc900b1bd0 msgr2=0x7fbc900b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:53:56.506 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:56.504+0000 7fbc98200640 1 -- 172.21.15.32:0/1618957013 shutdown_connections 2024-08-03T20:53:56.506 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:56.504+0000 7fbc98200640 1 -- 172.21.15.32:0/1618957013 wait complete. 2024-08-03T20:53:56.507 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:56.507+0000 7fbc98200640 1 Processor -- start 2024-08-03T20:53:56.507 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:56.507+0000 7fbc98200640 1 -- start start 2024-08-03T20:53:56.510 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:56.508+0000 7fbc98200640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fbc9006e930 0x7fbc900ae2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:53:56.510 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:56.508+0000 7fbc98200640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fbc90144f90 0x7fbc9015be80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:53:56.510 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:56.508+0000 7fbc98200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbc9015fff0 0x7fbc901435e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:53:56.510 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:56.508+0000 7fbc98200640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fbc900cf840 con 0x7fbc9015fff0 2024-08-03T20:53:56.510 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:56.508+0000 7fbc98200640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7fbc90145f30 con 0x7fbc9006e930 2024-08-03T20:53:56.510 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:56.508+0000 7fbc98200640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7fbc900cf9c0 con 0x7fbc90144f90 2024-08-03T20:53:56.510 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:56.509+0000 7fbc96200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbc9015fff0 0x7fbc901435e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:53:56.510 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:56.509+0000 7fbc96200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbc9015fff0 0x7fbc901435e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:60248/0 (socket says 172.21.15.32:60248) 2024-08-03T20:53:56.510 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:56.509+0000 7fbc96200640 1 -- 172.21.15.32:0/3542804686 learned_addr learned my addr 172.21.15.32:0/3542804686 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:53:56.511 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:56.509+0000 7fbc94e00640 1 --2- 172.21.15.32:0/3542804686 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fbc90144f90 0x7fbc9015be80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:53:56.511 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:56.509+0000 7fbc95800640 1 --2- 172.21.15.32:0/3542804686 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fbc9006e930 0x7fbc900ae2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:53:56.511 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:56.509+0000 7fbc96200640 1 -- 172.21.15.32:0/3542804686 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fbc90144f90 msgr2=0x7fbc9015be80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:53:56.511 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:56.509+0000 7fbc96200640 1 --2- 172.21.15.32:0/3542804686 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fbc90144f90 0x7fbc9015be80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:53:56.511 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:56.509+0000 7fbc96200640 1 -- 172.21.15.32:0/3542804686 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fbc9006e930 msgr2=0x7fbc900ae2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:53:56.511 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:56.509+0000 7fbc96200640 1 --2- 172.21.15.32:0/3542804686 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fbc9006e930 0x7fbc900ae2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:53:56.511 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:56.509+0000 7fbc96200640 1 -- 172.21.15.32:0/3542804686 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbc901e5e30 con 0x7fbc9015fff0 2024-08-03T20:53:56.511 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:56.510+0000 7fbc96200640 1 --2- 172.21.15.32:0/3542804686 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbc9015fff0 0x7fbc901435e0 secure :-1 s=READY pgs=814 cs=0 l=1 rev1=1 crypto rx=0x7fbc8800eab0 tx=0x7fbc8800ef70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:53:56.512 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:56.510+0000 7fbc8d600640 1 -- 172.21.15.32:0/3542804686 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc8800cea0 con 0x7fbc9015fff0 2024-08-03T20:53:56.512 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:56.510+0000 7fbc8d600640 1 -- 172.21.15.32:0/3542804686 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fbc88004b20 con 0x7fbc9015fff0 2024-08-03T20:53:56.512 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:56.510+0000 7fbc8d600640 1 -- 172.21.15.32:0/3542804686 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc88005020 con 0x7fbc9015fff0 2024-08-03T20:53:56.512 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:56.510+0000 7fbc98200640 1 -- 172.21.15.32:0/3542804686 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbc901e6110 con 0x7fbc9015fff0 2024-08-03T20:53:56.512 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:56.510+0000 7fbc98200640 1 -- 172.21.15.32:0/3542804686 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbc9015c3c0 con 0x7fbc9015fff0 2024-08-03T20:53:56.512 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:56.511+0000 7fbc98200640 1 -- 172.21.15.32:0/3542804686 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbc90160c30 con 0x7fbc9015fff0 2024-08-03T20:53:56.513 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:56.512+0000 7fbc8d600640 1 -- 172.21.15.32:0/3542804686 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7fbc880051c0 con 0x7fbc9015fff0 2024-08-03T20:53:56.514 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:56.513+0000 7fbc8d600640 1 --2- 172.21.15.32:0/3542804686 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fbc70077040 0x7fbc70079550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:53:56.514 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:56.513+0000 7fbc8d600640 1 -- 172.21.15.32:0/3542804686 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(71..71 src has 1..71) ==== 5795+0+0 (secure 0 0 0) 0x7fbc880993f0 con 0x7fbc9015fff0 2024-08-03T20:53:56.514 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:56.514+0000 7fbc95800640 1 --2- 172.21.15.32:0/3542804686 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fbc70077040 0x7fbc70079550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:53:56.515 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:56.515+0000 7fbc95800640 1 --2- 172.21.15.32:0/3542804686 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fbc70077040 0x7fbc70079550 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7fbc84001720 tx=0x7fbc84009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:53:56.518 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:56.518+0000 7fbc8d600640 1 -- 172.21.15.32:0/3542804686 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7fbc88061fd0 con 0x7fbc9015fff0 2024-08-03T20:53:56.767 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:56.767+0000 7fbc98200640 1 -- 172.21.15.32:0/3542804686 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fbc900b2fd0 con 0x7fbc9015fff0 2024-08-03T20:53:56.768 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:56.767+0000 7fbc8d600640 1 -- 172.21.15.32:0/3542804686 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 78 v78) ==== 94+0+4400 (secure 0 0 0) 0x7fbc88066520 con 0x7fbc9015fff0 2024-08-03T20:53:56.768 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-03T20:53:56.768 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":78,"btime":"2024-08-03T20:53:56:019136+0000","default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5666,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6837/2261538396","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6835","nonce":2261538396},{"type":"v1","addr":"172.21.15.118:6837","nonce":2261538396}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":69},{"gid":6061,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6836/1133580656","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6834","nonce":1133580656},{"type":"v1","addr":"172.21.15.118:6836","nonce":1133580656}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":68}],"filesystems":[{"mdsmap":{"epoch":78,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-08-03T20:53:15.080669+0000","modified":"2024-08-03T20:53:56.019133+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":71,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6396},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6396":{"gid":6396,"name":"c","rank":0,"incarnation":78,"state":"up:replay","state_seq":1,"addr":"172.21.15.32:6834/1798758431","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":1798758431},{"type":"v1","addr":"172.21.15.32:6834","nonce":1798758431}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":5}]} 2024-08-03T20:53:56.768 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 78 2024-08-03T20:53:56.771 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:56.770+0000 7fbc98200640 1 -- 172.21.15.32:0/3542804686 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fbc70077040 msgr2=0x7fbc70079550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:53:56.771 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:56.770+0000 7fbc98200640 1 --2- 172.21.15.32:0/3542804686 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fbc70077040 0x7fbc70079550 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7fbc84001720 tx=0x7fbc84009040 comp rx=0 tx=0).stop 2024-08-03T20:53:56.771 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:56.770+0000 7fbc98200640 1 -- 172.21.15.32:0/3542804686 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbc9015fff0 msgr2=0x7fbc901435e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:53:56.771 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:56.770+0000 7fbc98200640 1 --2- 172.21.15.32:0/3542804686 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbc9015fff0 0x7fbc901435e0 secure :-1 s=READY pgs=814 cs=0 l=1 rev1=1 crypto rx=0x7fbc8800eab0 tx=0x7fbc8800ef70 comp rx=0 tx=0).stop 2024-08-03T20:53:56.771 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:56.770+0000 7fbc98200640 1 -- 172.21.15.32:0/3542804686 shutdown_connections 2024-08-03T20:53:56.771 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:56.770+0000 7fbc98200640 1 --2- 172.21.15.32:0/3542804686 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fbc70077040 0x7fbc70079550 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:53:56.771 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:56.770+0000 7fbc98200640 1 --2- 172.21.15.32:0/3542804686 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbc9015fff0 0x7fbc901435e0 unknown :-1 s=CLOSED pgs=814 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:53:56.771 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:56.770+0000 7fbc98200640 1 --2- 172.21.15.32:0/3542804686 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fbc90144f90 0x7fbc9015be80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:53:56.771 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:56.770+0000 7fbc98200640 1 --2- 172.21.15.32:0/3542804686 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fbc9006e930 0x7fbc900ae2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:53:56.771 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:56.770+0000 7fbc98200640 1 -- 172.21.15.32:0/3542804686 >> 172.21.15.32:0/3542804686 conn(0x7fbc900b1bd0 msgr2=0x7fbc900cc540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:53:56.772 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:56.770+0000 7fbc98200640 1 -- 172.21.15.32:0/3542804686 shutdown_connections 2024-08-03T20:53:56.772 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:56.770+0000 7fbc98200640 1 -- 172.21.15.32:0/3542804686 wait complete. 2024-08-03T20:53:57.196 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 78, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-08-03T20:53:15.080669+0000', 'modified': '2024-08-03T20:53:56.019133+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 71, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 6396}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6396': {'gid': 6396, 'name': 'c', 'rank': 0, 'incarnation': 78, 'state': 'up:replay', 'state_seq': 1, 'addr': '172.21.15.32:6834/1798758431', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6832', 'nonce': 1798758431}, {'type': 'v1', 'addr': '172.21.15.32:6834', 'nonce': 1798758431}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [11], 'metadata_pool': 10, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 0, 'qdb_cluster': []} 2024-08-03T20:53:57.196 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6396:up:replay 2024-08-03T20:53:57.895 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a has been restored 2024-08-03T20:53:58.197 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-08-03T20:53:58.438 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:58.437+0000 7f8ae1e00640 1 -- 172.21.15.32:0/1010154857 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f8adc0b4ce0 msgr2=0x7f8adc15ef00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:53:58.438 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:58.437+0000 7f8ae1e00640 1 --2- 172.21.15.32:0/1010154857 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f8adc0b4ce0 0x7f8adc15ef00 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f8ac8009830 tx=0x7f8ac802a4b0 comp rx=0 tx=0).stop 2024-08-03T20:53:58.439 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:58.438+0000 7f8ae1e00640 1 -- 172.21.15.32:0/1010154857 shutdown_connections 2024-08-03T20:53:58.439 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:58.438+0000 7f8ae1e00640 1 --2- 172.21.15.32:0/1010154857 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f8adc0b4ce0 0x7f8adc15ef00 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:53:58.439 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:58.438+0000 7f8ae1e00640 1 --2- 172.21.15.32:0/1010154857 >> v2:172.21.15.118:3300/0 conn(0x7f8adc0b38b0 0x7f8adc1523f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:53:58.439 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:58.438+0000 7f8ae1e00640 1 -- 172.21.15.32:0/1010154857 >> 172.21.15.32:0/1010154857 conn(0x7f8adc0b1bd0 msgr2=0x7f8adc0b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:53:58.439 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:58.438+0000 7f8ae1e00640 1 -- 172.21.15.32:0/1010154857 shutdown_connections 2024-08-03T20:53:58.439 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:58.438+0000 7f8ae1e00640 1 -- 172.21.15.32:0/1010154857 wait complete. 2024-08-03T20:53:58.440 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:58.440+0000 7f8ae1e00640 1 Processor -- start 2024-08-03T20:53:58.440 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:58.440+0000 7f8ae1e00640 1 -- start start 2024-08-03T20:53:58.443 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:58.441+0000 7f8ae1e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8adc0b38b0 0x7f8adc0a9b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:53:58.443 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:58.442+0000 7f8ae1e00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8adc0b4ce0 0x7f8adc158dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:53:58.443 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:58.442+0000 7f8ae1e00640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f8adc15f440 0x7f8adc154270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:53:58.443 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:58.442+0000 7f8ae1e00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f8adc15c020 con 0x7f8adc0b38b0 2024-08-03T20:53:58.443 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:58.442+0000 7f8ae1e00640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f8adc0cf240 con 0x7f8adc0b4ce0 2024-08-03T20:53:58.444 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:58.442+0000 7f8ae1e00640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7f8adc15c1a0 con 0x7f8adc15f440 2024-08-03T20:53:58.444 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:58.442+0000 7f8adaa00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8adc0b38b0 0x7f8adc0a9b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:53:58.444 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:58.442+0000 7f8adaa00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8adc0b38b0 0x7f8adc0a9b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:60258/0 (socket says 172.21.15.32:60258) 2024-08-03T20:53:58.444 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:58.442+0000 7f8adaa00640 1 -- 172.21.15.32:0/1410318622 learned_addr learned my addr 172.21.15.32:0/1410318622 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:53:58.444 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:58.442+0000 7f8ada000640 1 --2- 172.21.15.32:0/1410318622 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8adc0b4ce0 0x7f8adc158dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:53:58.444 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:58.442+0000 7f8adb400640 1 --2- 172.21.15.32:0/1410318622 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f8adc15f440 0x7f8adc154270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:53:58.444 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:58.442+0000 7f8adaa00640 1 -- 172.21.15.32:0/1410318622 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f8adc15f440 msgr2=0x7f8adc154270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:53:58.444 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:58.442+0000 7f8adaa00640 1 --2- 172.21.15.32:0/1410318622 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f8adc15f440 0x7f8adc154270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:53:58.444 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:58.442+0000 7f8adaa00640 1 -- 172.21.15.32:0/1410318622 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8adc0b4ce0 msgr2=0x7f8adc158dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:53:58.444 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:58.442+0000 7f8adaa00640 1 --2- 172.21.15.32:0/1410318622 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8adc0b4ce0 0x7f8adc158dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:53:58.444 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:58.442+0000 7f8adaa00640 1 -- 172.21.15.32:0/1410318622 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8adc159300 con 0x7f8adc0b38b0 2024-08-03T20:53:58.444 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:58.443+0000 7f8adaa00640 1 --2- 172.21.15.32:0/1410318622 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8adc0b38b0 0x7f8adc0a9b80 secure :-1 s=READY pgs=816 cs=0 l=1 rev1=1 crypto rx=0x7f8acc00ea20 tx=0x7f8acc00eee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:53:58.445 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:58.443+0000 7f8ad3400640 1 -- 172.21.15.32:0/1410318622 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8acc00ce40 con 0x7f8adc0b38b0 2024-08-03T20:53:58.445 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:58.443+0000 7f8ad3400640 1 -- 172.21.15.32:0/1410318622 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f8acc004020 con 0x7f8adc0b38b0 2024-08-03T20:53:58.445 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:58.443+0000 7f8ad3400640 1 -- 172.21.15.32:0/1410318622 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8acc004470 con 0x7f8adc0b38b0 2024-08-03T20:53:58.445 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:58.443+0000 7f8ae1e00640 1 -- 172.21.15.32:0/1410318622 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8adc0aa4f0 con 0x7f8adc0b38b0 2024-08-03T20:53:58.445 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:58.443+0000 7f8ae1e00640 1 -- 172.21.15.32:0/1410318622 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8adc156350 con 0x7f8adc0b38b0 2024-08-03T20:53:58.445 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:58.444+0000 7f8ae1e00640 1 -- 172.21.15.32:0/1410318622 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8adc1521d0 con 0x7f8adc0b38b0 2024-08-03T20:53:58.446 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:58.445+0000 7f8ad3400640 1 -- 172.21.15.32:0/1410318622 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f8acc0035f0 con 0x7f8adc0b38b0 2024-08-03T20:53:58.446 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:58.446+0000 7f8ad3400640 1 --2- 172.21.15.32:0/1410318622 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f8abc077010 0x7f8abc079520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:53:58.447 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:58.446+0000 7f8ad3400640 1 -- 172.21.15.32:0/1410318622 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(71..71 src has 1..71) ==== 5795+0+0 (secure 0 0 0) 0x7f8acc099350 con 0x7f8adc0b38b0 2024-08-03T20:53:58.447 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:58.447+0000 7f8ada000640 1 --2- 172.21.15.32:0/1410318622 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f8abc077010 0x7f8abc079520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:53:58.448 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:58.448+0000 7f8ada000640 1 --2- 172.21.15.32:0/1410318622 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f8abc077010 0x7f8abc079520 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f8ac8034000 tx=0x7f8ac8004730 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:53:58.451 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:58.451+0000 7f8ad3400640 1 -- 172.21.15.32:0/1410318622 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7f8acc062450 con 0x7f8adc0b38b0 2024-08-03T20:53:58.703 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:58.702+0000 7f8ae1e00640 1 -- 172.21.15.32:0/1410318622 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f8adc15c670 con 0x7f8adc0b38b0 2024-08-03T20:53:58.703 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:58.703+0000 7f8ad3400640 1 -- 172.21.15.32:0/1410318622 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 80 v80) ==== 94+0+5226 (secure 0 0 0) 0x7f8acc061d50 con 0x7f8adc0b38b0 2024-08-03T20:53:58.703 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-03T20:53:58.704 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":80,"btime":"2024-08-03T20:53:58:027349+0000","default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5666,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6837/2261538396","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6835","nonce":2261538396},{"type":"v1","addr":"172.21.15.118:6837","nonce":2261538396}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":69},{"gid":6061,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6836/1133580656","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6834","nonce":1133580656},{"type":"v1","addr":"172.21.15.118:6836","nonce":1133580656}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":68},{"gid":6618,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6835/3466281885","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6833","nonce":3466281885},{"type":"v1","addr":"172.21.15.32:6835","nonce":3466281885}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":79}],"filesystems":[{"mdsmap":{"epoch":80,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-08-03T20:53:15.080669+0000","modified":"2024-08-03T20:53:57.037253+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":71,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6396},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6396":{"gid":6396,"name":"c","rank":0,"incarnation":78,"state":"up:rejoin","state_seq":14,"addr":"172.21.15.32:6834/1798758431","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":1798758431},{"type":"v1","addr":"172.21.15.32:6834","nonce":1798758431}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":5}]} 2024-08-03T20:53:58.704 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 80 2024-08-03T20:53:58.706 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:58.705+0000 7f8ae1e00640 1 -- 172.21.15.32:0/1410318622 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f8abc077010 msgr2=0x7f8abc079520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:53:58.706 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:58.705+0000 7f8ae1e00640 1 --2- 172.21.15.32:0/1410318622 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f8abc077010 0x7f8abc079520 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f8ac8034000 tx=0x7f8ac8004730 comp rx=0 tx=0).stop 2024-08-03T20:53:58.706 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:58.705+0000 7f8ae1e00640 1 -- 172.21.15.32:0/1410318622 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8adc0b38b0 msgr2=0x7f8adc0a9b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:53:58.706 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:58.705+0000 7f8ae1e00640 1 --2- 172.21.15.32:0/1410318622 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8adc0b38b0 0x7f8adc0a9b80 secure :-1 s=READY pgs=816 cs=0 l=1 rev1=1 crypto rx=0x7f8acc00ea20 tx=0x7f8acc00eee0 comp rx=0 tx=0).stop 2024-08-03T20:53:58.706 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:58.705+0000 7f8ae1e00640 1 -- 172.21.15.32:0/1410318622 shutdown_connections 2024-08-03T20:53:58.706 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:58.705+0000 7f8ae1e00640 1 --2- 172.21.15.32:0/1410318622 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f8abc077010 0x7f8abc079520 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:53:58.706 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:58.705+0000 7f8ae1e00640 1 --2- 172.21.15.32:0/1410318622 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8adc0b4ce0 0x7f8adc158dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:53:58.707 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:58.705+0000 7f8ae1e00640 1 --2- 172.21.15.32:0/1410318622 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f8adc15f440 0x7f8adc154270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:53:58.707 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:58.705+0000 7f8ae1e00640 1 --2- 172.21.15.32:0/1410318622 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8adc0b38b0 0x7f8adc0a9b80 unknown :-1 s=CLOSED pgs=816 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:53:58.707 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:58.705+0000 7f8ae1e00640 1 -- 172.21.15.32:0/1410318622 >> 172.21.15.32:0/1410318622 conn(0x7f8adc0b1bd0 msgr2=0x7f8adc159530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:53:58.707 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:58.706+0000 7f8ae1e00640 1 -- 172.21.15.32:0/1410318622 shutdown_connections 2024-08-03T20:53:58.707 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:58.706+0000 7f8ae1e00640 1 -- 172.21.15.32:0/1410318622 wait complete. 2024-08-03T20:53:59.060 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:53:59 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-08-03T20:53:59.129 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 80, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-08-03T20:53:15.080669+0000', 'modified': '2024-08-03T20:53:57.037253+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 71, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 6396}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6396': {'gid': 6396, 'name': 'c', 'rank': 0, 'incarnation': 78, 'state': 'up:rejoin', 'state_seq': 14, 'addr': '172.21.15.32:6834/1798758431', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6832', 'nonce': 1798758431}, {'type': 'v1', 'addr': '172.21.15.32:6834', 'nonce': 1798758431}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [11], 'metadata_pool': 10, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 0, 'qdb_cluster': []} 2024-08-03T20:53:59.129 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6396:up:rejoin 2024-08-03T20:54:00.130 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-08-03T20:54:00.371 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:00.370+0000 7f0b11200640 1 -- 172.21.15.32:0/3945027079 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f0b0c15f610 msgr2=0x7f0b0c1523f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:00.371 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:00.370+0000 7f0b11200640 1 --2- 172.21.15.32:0/3945027079 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f0b0c15f610 0x7f0b0c1523f0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f0af4009830 tx=0x7f0af402a530 comp rx=0 tx=0).stop 2024-08-03T20:54:00.371 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:00.370+0000 7f0b11200640 1 -- 172.21.15.32:0/3945027079 shutdown_connections 2024-08-03T20:54:00.371 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:00.370+0000 7f0b11200640 1 --2- 172.21.15.32:0/3945027079 >> v2:172.21.15.118:3300/0 conn(0x7f0b0c0b4ce0 0x7f0b0c0cdf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:00.371 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:00.370+0000 7f0b11200640 1 --2- 172.21.15.32:0/3945027079 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f0b0c15f610 0x7f0b0c1523f0 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:00.372 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:00.370+0000 7f0b11200640 1 -- 172.21.15.32:0/3945027079 >> 172.21.15.32:0/3945027079 conn(0x7f0b0c0b1bd0 msgr2=0x7f0b0c0b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:54:00.372 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:00.370+0000 7f0b11200640 1 -- 172.21.15.32:0/3945027079 shutdown_connections 2024-08-03T20:54:00.372 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:00.370+0000 7f0b11200640 1 -- 172.21.15.32:0/3945027079 wait complete. 2024-08-03T20:54:00.372 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:00.372+0000 7f0b11200640 1 Processor -- start 2024-08-03T20:54:00.373 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:00.372+0000 7f0b11200640 1 -- start start 2024-08-03T20:54:00.376 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:00.374+0000 7f0b11200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0b0c0b38b0 0x7f0b0c148d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:00.376 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:00.374+0000 7f0b11200640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f0b0c0b4ce0 0x7f0b0c0b1160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:00.376 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:00.374+0000 7f0b11200640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f0b0c1505f0 0x7f0b0c150a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:00.376 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:00.374+0000 7f0b11200640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f0b0c0cea90 con 0x7f0b0c0b38b0 2024-08-03T20:54:00.376 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:00.374+0000 7f0b11200640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f0b0c0ce910 con 0x7f0b0c1505f0 2024-08-03T20:54:00.376 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:00.374+0000 7f0b11200640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7f0b0c0cec10 con 0x7f0b0c0b4ce0 2024-08-03T20:54:00.376 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:00.374+0000 7f0b09e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0b0c0b38b0 0x7f0b0c148d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:00.376 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:00.374+0000 7f0b09e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0b0c0b38b0 0x7f0b0c148d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:57798/0 (socket says 172.21.15.32:57798) 2024-08-03T20:54:00.376 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:00.374+0000 7f0b09e00640 1 -- 172.21.15.32:0/2160885345 learned_addr learned my addr 172.21.15.32:0/2160885345 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:54:00.377 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:00.374+0000 7f0b0a800640 1 --2- 172.21.15.32:0/2160885345 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f0b0c1505f0 0x7f0b0c150a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:00.377 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:00.374+0000 7f0b03e00640 1 --2- 172.21.15.32:0/2160885345 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f0b0c0b4ce0 0x7f0b0c0b1160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:00.377 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:00.374+0000 7f0b09e00640 1 -- 172.21.15.32:0/2160885345 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f0b0c0b4ce0 msgr2=0x7f0b0c0b1160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:00.377 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:00.374+0000 7f0b09e00640 1 --2- 172.21.15.32:0/2160885345 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f0b0c0b4ce0 0x7f0b0c0b1160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:00.377 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:00.375+0000 7f0b09e00640 1 -- 172.21.15.32:0/2160885345 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f0b0c1505f0 msgr2=0x7f0b0c150a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:00.377 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:00.375+0000 7f0b09e00640 1 --2- 172.21.15.32:0/2160885345 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f0b0c1505f0 0x7f0b0c150a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:00.377 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:00.375+0000 7f0b09e00640 1 -- 172.21.15.32:0/2160885345 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0b0c150f40 con 0x7f0b0c0b38b0 2024-08-03T20:54:00.377 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:00.375+0000 7f0b09e00640 1 --2- 172.21.15.32:0/2160885345 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0b0c0b38b0 0x7f0b0c148d30 secure :-1 s=READY pgs=818 cs=0 l=1 rev1=1 crypto rx=0x7f0afc00c460 tx=0x7f0afc00c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:54:00.377 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:00.375+0000 7f0b02000640 1 -- 172.21.15.32:0/2160885345 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0afc016020 con 0x7f0b0c0b38b0 2024-08-03T20:54:00.377 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:00.375+0000 7f0b02000640 1 -- 172.21.15.32:0/2160885345 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f0afc012940 con 0x7f0b0c0b38b0 2024-08-03T20:54:00.378 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:00.375+0000 7f0b02000640 1 -- 172.21.15.32:0/2160885345 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0afc012d90 con 0x7f0b0c0b38b0 2024-08-03T20:54:00.378 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:00.375+0000 7f0b11200640 1 -- 172.21.15.32:0/2160885345 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0b0c0b0830 con 0x7f0b0c0b38b0 2024-08-03T20:54:00.378 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:00.375+0000 7f0b11200640 1 -- 172.21.15.32:0/2160885345 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0b0c0afd50 con 0x7f0b0c0b38b0 2024-08-03T20:54:00.378 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:00.377+0000 7f0b11200640 1 -- 172.21.15.32:0/2160885345 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0b0c0cf130 con 0x7f0b0c0b38b0 2024-08-03T20:54:00.379 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:00.378+0000 7f0b02000640 1 -- 172.21.15.32:0/2160885345 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f0afc019020 con 0x7f0b0c0b38b0 2024-08-03T20:54:00.379 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:00.379+0000 7f0b02000640 1 --2- 172.21.15.32:0/2160885345 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f0ae8077010 0x7f0ae8079520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:00.379 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:00.379+0000 7f0b02000640 1 -- 172.21.15.32:0/2160885345 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(71..71 src has 1..71) ==== 5795+0+0 (secure 0 0 0) 0x7f0afc0986a0 con 0x7f0b0c0b38b0 2024-08-03T20:54:00.380 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:00.379+0000 7f0b03e00640 1 --2- 172.21.15.32:0/2160885345 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f0ae8077010 0x7f0ae8079520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:00.380 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:00.380+0000 7f0b03e00640 1 --2- 172.21.15.32:0/2160885345 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f0ae8077010 0x7f0ae8079520 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f0af4034000 tx=0x7f0af4004730 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:54:00.383 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:00.383+0000 7f0b02000640 1 -- 172.21.15.32:0/2160885345 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7f0afc020030 con 0x7f0b0c0b38b0 2024-08-03T20:54:00.629 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:00.628+0000 7f0b11200640 1 -- 172.21.15.32:0/2160885345 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f0b0c1eaf00 con 0x7f0b0c0b38b0 2024-08-03T20:54:00.629 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:00.629+0000 7f0b02000640 1 -- 172.21.15.32:0/2160885345 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 81 v81) ==== 94+0+5233 (secure 0 0 0) 0x7f0b0c150f40 con 0x7f0b0c0b38b0 2024-08-03T20:54:00.630 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-03T20:54:00.630 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":81,"btime":"2024-08-03T20:53:59:036145+0000","default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5666,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6837/2261538396","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6835","nonce":2261538396},{"type":"v1","addr":"172.21.15.118:6837","nonce":2261538396}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":69},{"gid":6061,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6836/1133580656","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6834","nonce":1133580656},{"type":"v1","addr":"172.21.15.118:6836","nonce":1133580656}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":68},{"gid":6618,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6835/3466281885","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6833","nonce":3466281885},{"type":"v1","addr":"172.21.15.32:6835","nonce":3466281885}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":79}],"filesystems":[{"mdsmap":{"epoch":81,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-08-03T20:53:15.080669+0000","modified":"2024-08-03T20:53:59.036137+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":71,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6396},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6396":{"gid":6396,"name":"c","rank":0,"incarnation":78,"state":"up:active","state_seq":15,"addr":"172.21.15.32:6834/1798758431","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":1798758431},{"type":"v1","addr":"172.21.15.32:6834","nonce":1798758431}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6396,"qdb_cluster":[6396]},"id":5}]} 2024-08-03T20:54:00.630 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 81 2024-08-03T20:54:00.633 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:00.631+0000 7f0b11200640 1 -- 172.21.15.32:0/2160885345 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f0ae8077010 msgr2=0x7f0ae8079520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:00.633 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:00.631+0000 7f0b11200640 1 --2- 172.21.15.32:0/2160885345 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f0ae8077010 0x7f0ae8079520 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f0af4034000 tx=0x7f0af4004730 comp rx=0 tx=0).stop 2024-08-03T20:54:00.633 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:00.631+0000 7f0b11200640 1 -- 172.21.15.32:0/2160885345 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0b0c0b38b0 msgr2=0x7f0b0c148d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:00.634 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:00.631+0000 7f0b11200640 1 --2- 172.21.15.32:0/2160885345 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0b0c0b38b0 0x7f0b0c148d30 secure :-1 s=READY pgs=818 cs=0 l=1 rev1=1 crypto rx=0x7f0afc00c460 tx=0x7f0afc00c920 comp rx=0 tx=0).stop 2024-08-03T20:54:00.634 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:00.631+0000 7f0b11200640 1 -- 172.21.15.32:0/2160885345 shutdown_connections 2024-08-03T20:54:00.634 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:00.631+0000 7f0b11200640 1 --2- 172.21.15.32:0/2160885345 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f0ae8077010 0x7f0ae8079520 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:00.634 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:00.631+0000 7f0b11200640 1 --2- 172.21.15.32:0/2160885345 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f0b0c1505f0 0x7f0b0c150a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:00.635 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:00.631+0000 7f0b11200640 1 --2- 172.21.15.32:0/2160885345 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f0b0c0b4ce0 0x7f0b0c0b1160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:00.635 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:00.631+0000 7f0b11200640 1 --2- 172.21.15.32:0/2160885345 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0b0c0b38b0 0x7f0b0c148d30 unknown :-1 s=CLOSED pgs=818 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:00.635 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:00.631+0000 7f0b11200640 1 -- 172.21.15.32:0/2160885345 >> 172.21.15.32:0/2160885345 conn(0x7f0b0c0b1bd0 msgr2=0x7f0b0c15a4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:54:00.635 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:00.631+0000 7f0b11200640 1 -- 172.21.15.32:0/2160885345 shutdown_connections 2024-08-03T20:54:00.635 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:00.631+0000 7f0b11200640 1 -- 172.21.15.32:0/2160885345 wait complete. 2024-08-03T20:54:01.059 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 81, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-08-03T20:53:15.080669+0000', 'modified': '2024-08-03T20:53:59.036137+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 71, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 6396}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6396': {'gid': 6396, 'name': 'c', 'rank': 0, 'incarnation': 78, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.32:6834/1798758431', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6832', 'nonce': 1798758431}, {'type': 'v1', 'addr': '172.21.15.32:6834', 'nonce': 1798758431}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [11], 'metadata_pool': 10, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 6396, 'qdb_cluster': [6396]} 2024-08-03T20:54:01.059 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-08-03T20:54:01.060 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-08-03T20:54:01.293 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:01.292+0000 7fe69f000640 1 -- 172.21.15.32:0/2144677838 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe67c003a30 msgr2=0x7fe67c003ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:01.293 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:01.292+0000 7fe69f000640 1 --2- 172.21.15.32:0/2144677838 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe67c003a30 0x7fe67c003ea0 secure :-1 s=READY pgs=820 cs=0 l=1 rev1=1 crypto rx=0x7fe688009b50 tx=0x7fe67c005830 comp rx=0 tx=0).stop 2024-08-03T20:54:01.293 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:01.292+0000 7fe69d000640 1 -- 172.21.15.32:0/2144677838 reap_dead start 2024-08-03T20:54:01.293 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:01.292+0000 7fe69f000640 1 -- 172.21.15.32:0/2144677838 shutdown_connections 2024-08-03T20:54:01.293 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:01.292+0000 7fe69f000640 1 -- 172.21.15.32:0/2144677838 >> 172.21.15.32:0/2144677838 conn(0x7fe6980b1bd0 msgr2=0x7fe6980b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:54:01.293 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:01.292+0000 7fe69f000640 1 -- 172.21.15.32:0/2144677838 shutdown_connections 2024-08-03T20:54:01.293 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:01.292+0000 7fe69f000640 1 -- 172.21.15.32:0/2144677838 wait complete. 2024-08-03T20:54:01.295 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:01.294+0000 7fe69f000640 1 Processor -- start 2024-08-03T20:54:01.295 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:01.295+0000 7fe69f000640 1 -- start start 2024-08-03T20:54:01.297 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:01.296+0000 7fe69f000640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fe67c003a30 0x7fe69813cb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:01.297 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:01.296+0000 7fe69f000640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fe69813a1a0 0x7fe69813a5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:01.297 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:01.296+0000 7fe69f000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe69815b9b0 0x7fe69815bdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:01.297 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:01.296+0000 7fe69f000640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fe698145eb0 con 0x7fe69815b9b0 2024-08-03T20:54:01.297 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:01.296+0000 7fe69f000640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7fe6980cfa50 con 0x7fe67c003a30 2024-08-03T20:54:01.297 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:01.296+0000 7fe69f000640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7fe698160b10 con 0x7fe69813a1a0 2024-08-03T20:54:01.297 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:01.296+0000 7fe69d000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe69815b9b0 0x7fe69815bdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:01.298 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:01.296+0000 7fe69d000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe69815b9b0 0x7fe69815bdc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:57826/0 (socket says 172.21.15.32:57826) 2024-08-03T20:54:01.298 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:01.296+0000 7fe69d000640 1 -- 172.21.15.32:0/1019794313 learned_addr learned my addr 172.21.15.32:0/1019794313 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:54:01.298 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:01.296+0000 7fe697e00640 1 --2- 172.21.15.32:0/1019794313 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fe67c003a30 0x7fe69813cb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:01.298 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:01.297+0000 7fe68fe00640 1 --2- 172.21.15.32:0/1019794313 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fe69813a1a0 0x7fe69813a5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:01.298 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:01.297+0000 7fe69d000640 1 -- 172.21.15.32:0/1019794313 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fe69813a1a0 msgr2=0x7fe69813a5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:01.298 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:01.297+0000 7fe69d000640 1 --2- 172.21.15.32:0/1019794313 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fe69813a1a0 0x7fe69813a5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:01.298 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:01.297+0000 7fe69d000640 1 -- 172.21.15.32:0/1019794313 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fe67c003a30 msgr2=0x7fe69813cb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:01.298 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:01.297+0000 7fe69d000640 1 --2- 172.21.15.32:0/1019794313 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fe67c003a30 0x7fe69813cb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:01.298 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:01.297+0000 7fe69d000640 1 -- 172.21.15.32:0/1019794313 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe6981f3030 con 0x7fe69815b9b0 2024-08-03T20:54:01.298 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:01.297+0000 7fe68fe00640 1 --2- 172.21.15.32:0/1019794313 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fe69813a1a0 0x7fe69813a5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-03T20:54:01.299 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:01.297+0000 7fe69d000640 1 --2- 172.21.15.32:0/1019794313 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe69815b9b0 0x7fe69815bdc0 secure :-1 s=READY pgs=821 cs=0 l=1 rev1=1 crypto rx=0x7fe698072860 tx=0x7fe690001950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:54:01.299 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:01.297+0000 7fe694c00640 1 -- 172.21.15.32:0/1019794313 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6900107d0 con 0x7fe69815b9b0 2024-08-03T20:54:01.299 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:01.297+0000 7fe694c00640 1 -- 172.21.15.32:0/1019794313 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fe690016070 con 0x7fe69815b9b0 2024-08-03T20:54:01.299 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:01.297+0000 7fe694c00640 1 -- 172.21.15.32:0/1019794313 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe690010d00 con 0x7fe69815b9b0 2024-08-03T20:54:01.299 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:01.297+0000 7fe69f000640 1 -- 172.21.15.32:0/1019794313 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe6981f3310 con 0x7fe69815b9b0 2024-08-03T20:54:01.299 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:01.298+0000 7fe69f000640 1 -- 172.21.15.32:0/1019794313 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe6981f38d0 con 0x7fe69815b9b0 2024-08-03T20:54:01.300 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:01.300+0000 7fe694c00640 1 -- 172.21.15.32:0/1019794313 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7fe69001b460 con 0x7fe69815b9b0 2024-08-03T20:54:01.301 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:01.301+0000 7fe694c00640 1 --2- 172.21.15.32:0/1019794313 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fe674077010 0x7fe674079520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:01.301 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:01.301+0000 7fe694c00640 1 -- 172.21.15.32:0/1019794313 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(71..71 src has 1..71) ==== 5795+0+0 (secure 0 0 0) 0x7fe690099120 con 0x7fe69815b9b0 2024-08-03T20:54:01.302 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:01.301+0000 7fe6a5f8a740 1 Processor -- start 2024-08-03T20:54:01.302 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:01.301+0000 7fe697e00640 1 --2- 172.21.15.32:0/1019794313 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fe674077010 0x7fe674079520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:01.302 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:01.302+0000 7fe6a5f8a740 1 -- start start 2024-08-03T20:54:01.303 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:01.302+0000 7fe697e00640 1 --2- 172.21.15.32:0/1019794313 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fe674077010 0x7fe674079520 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7fe688035000 tx=0x7fe68802d040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:54:01.304 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:01.303+0000 7fe6a5f8a740 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x5614b1fec270 0x5614b23ac0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:01.304 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:01.303+0000 7fe6a5f8a740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5614b1fc33b0 0x5614b23d7c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:01.305 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:01.303+0000 7fe6a5f8a740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5614b1fcb590 0x5614b23b8cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:01.305 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:01.303+0000 7fe6a5f8a740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x5614b2304a60 con 0x5614b1fc33b0 2024-08-03T20:54:01.305 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:01.303+0000 7fe6a5f8a740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x5614b22f3f60 con 0x5614b1fcb590 2024-08-03T20:54:01.305 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:01.303+0000 7fe6a5f8a740 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x5614b22f7fa0 con 0x5614b1fec270 2024-08-03T20:54:01.305 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:01.303+0000 7fe68fe00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5614b1fc33b0 0x5614b23d7c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:01.305 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:01.303+0000 7fe68fe00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5614b1fc33b0 0x5614b23d7c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:57840/0 (socket says 172.21.15.32:57840) 2024-08-03T20:54:01.305 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:01.303+0000 7fe68fe00640 1 -- 172.21.15.32:0/2213959428 learned_addr learned my addr 172.21.15.32:0/2213959428 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:54:01.305 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:01.303+0000 7fe697e00640 1 --2- 172.21.15.32:0/2213959428 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x5614b1fec270 0x5614b23ac0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:01.305 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:01.303+0000 7fe69d000640 1 --2- 172.21.15.32:0/2213959428 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5614b1fcb590 0x5614b23b8cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:01.305 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:01.304+0000 7fe68fe00640 1 -- 172.21.15.32:0/2213959428 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x5614b1fec270 msgr2=0x5614b23ac0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:54:01.305 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:01.304+0000 7fe68fe00640 1 --2- 172.21.15.32:0/2213959428 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x5614b1fec270 0x5614b23ac0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:01.305 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:01.304+0000 7fe68fe00640 1 -- 172.21.15.32:0/2213959428 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5614b1fcb590 msgr2=0x5614b23b8cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:54:01.305 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:01.304+0000 7fe68fe00640 1 --2- 172.21.15.32:0/2213959428 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5614b1fcb590 0x5614b23b8cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:01.306 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:01.304+0000 7fe68fe00640 1 -- 172.21.15.32:0/2213959428 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5614b2057850 con 0x5614b1fc33b0 2024-08-03T20:54:01.306 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:01.304+0000 7fe68fe00640 1 --2- 172.21.15.32:0/2213959428 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5614b1fc33b0 0x5614b23d7c70 secure :-1 s=READY pgs=822 cs=0 l=1 rev1=1 crypto rx=0x7fe680004a70 tx=0x7fe680004f30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2843b64649d9b23 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:54:01.306 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:01.304+0000 7fe68cc00640 1 -- 172.21.15.32:0/2213959428 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe680010b80 con 0x5614b1fc33b0 2024-08-03T20:54:01.306 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:01.304+0000 7fe68cc00640 1 -- 172.21.15.32:0/2213959428 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fe680010e10 con 0x5614b1fc33b0 2024-08-03T20:54:01.306 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:01.304+0000 7fe68cc00640 1 -- 172.21.15.32:0/2213959428 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe680004290 con 0x5614b1fc33b0 2024-08-03T20:54:01.307 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:01.305+0000 7fe6a5f8a740 1 -- 172.21.15.32:0/2213959428 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5614b1fc33b0 msgr2=0x5614b23d7c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:01.307 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:01.305+0000 7fe6a5f8a740 1 --2- 172.21.15.32:0/2213959428 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5614b1fc33b0 0x5614b23d7c70 secure :-1 s=READY pgs=822 cs=0 l=1 rev1=1 crypto rx=0x7fe680004a70 tx=0x7fe680004f30 comp rx=0 tx=0).stop 2024-08-03T20:54:01.307 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:01.305+0000 7fe6a5f8a740 1 -- 172.21.15.32:0/2213959428 shutdown_connections 2024-08-03T20:54:01.307 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:01.305+0000 7fe6a5f8a740 1 --2- 172.21.15.32:0/2213959428 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5614b1fcb590 0x5614b23b8cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:01.307 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:01.305+0000 7fe6a5f8a740 1 --2- 172.21.15.32:0/2213959428 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5614b1fc33b0 0x5614b23d7c70 unknown :-1 s=CLOSED pgs=822 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:01.307 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:01.305+0000 7fe6a5f8a740 1 --2- 172.21.15.32:0/2213959428 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x5614b1fec270 0x5614b23ac0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:01.307 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:01.305+0000 7fe6a5f8a740 1 -- 172.21.15.32:0/2213959428 >> 172.21.15.32:0/2213959428 conn(0x5614b1fe03c0 msgr2=0x5614b2391fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:54:01.307 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:01.305+0000 7fe6a5f8a740 1 -- 172.21.15.32:0/2213959428 shutdown_connections 2024-08-03T20:54:01.307 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:01.305+0000 7fe6a5f8a740 1 -- 172.21.15.32:0/2213959428 wait complete. 2024-08-03T20:54:01.308 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:01.306+0000 7fe6a5f8a740 1 -- start start 2024-08-03T20:54:01.308 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:01.306+0000 7fe6a5f8a740 1 Processor -- start 2024-08-03T20:54:01.309 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:01.308+0000 7fe6a5f8a740 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x5614b1fc33b0 0x5614b23cfcc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:01.309 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:01.308+0000 7fe6a5f8a740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5614b1fcb590 0x5614b23ee280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:01.309 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:01.308+0000 7fe6a5f8a740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5614b1fec270 0x5614b23ee7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:01.309 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:01.308+0000 7fe6a5f8a740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x5614b2066620 con 0x5614b1fcb590 2024-08-03T20:54:01.309 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:01.308+0000 7fe6a5f8a740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x5614b2291160 con 0x5614b1fec270 2024-08-03T20:54:01.309 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:01.308+0000 7fe6a5f8a740 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x5614b23ab380 con 0x5614b1fc33b0 2024-08-03T20:54:01.310 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:01.308+0000 7fe6a5f8a740 10 client.0 _ll_register_callbacks cb 0x5614b22a06b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-03T20:54:01.310 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:01.308+0000 7fe68fe00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5614b1fcb590 0x5614b23ee280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:01.310 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:01.308+0000 7fe68fe00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5614b1fcb590 0x5614b23ee280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:57844/0 (socket says 172.21.15.32:57844) 2024-08-03T20:54:01.310 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:01.308+0000 7fe68fe00640 1 -- 172.21.15.32:0/1307845301 learned_addr learned my addr 172.21.15.32:0/1307845301 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:54:01.310 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:01.308+0000 7fe697e00640 1 --2- 172.21.15.32:0/1307845301 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x5614b1fc33b0 0x5614b23cfcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:01.310 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:01.308+0000 7fe69d000640 1 --2- 172.21.15.32:0/1307845301 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5614b1fec270 0x5614b23ee7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:01.310 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:01.309+0000 7fe68fe00640 1 -- 172.21.15.32:0/1307845301 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x5614b1fc33b0 msgr2=0x5614b23cfcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:54:01.310 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:01.309+0000 7fe68fe00640 1 --2- 172.21.15.32:0/1307845301 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x5614b1fc33b0 0x5614b23cfcc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:01.310 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:01.309+0000 7fe68fe00640 1 -- 172.21.15.32:0/1307845301 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5614b1fec270 msgr2=0x5614b23ee7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:54:01.310 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:01.309+0000 7fe68fe00640 1 --2- 172.21.15.32:0/1307845301 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5614b1fec270 0x5614b23ee7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:01.310 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:01.309+0000 7fe68fe00640 1 -- 172.21.15.32:0/1307845301 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5614b20ce6b0 con 0x5614b1fcb590 2024-08-03T20:54:01.312 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:01.309+0000 7fe68fe00640 1 --2- 172.21.15.32:0/1307845301 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5614b1fcb590 0x5614b23ee280 secure :-1 s=READY pgs=823 cs=0 l=1 rev1=1 crypto rx=0x7fe680005000 tx=0x7fe680010b80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b4ca0afca609291f server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:54:01.312 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:01.309+0000 7fe685600640 10 client.0 ms_handle_connect on v2:172.21.15.32:3300/0 2024-08-03T20:54:01.312 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:01.309+0000 7fe685600640 1 -- 172.21.15.32:0/1307845301 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6800144e0 con 0x5614b1fcb590 2024-08-03T20:54:01.312 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:01.309+0000 7fe685600640 1 -- 172.21.15.32:0/1307845301 <== mon.0 v2:172.21.15.32:3300/0 2 ==== osd_map(71..71 src has 1..71) ==== 5795+0+0 (secure 0 0 0) 0x7fe680014680 con 0x5614b1fcb590 2024-08-03T20:54:01.312 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:01.309+0000 7fe6a5f8a740 1 -- 172.21.15.32:0/1307845301 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe680010e10 con 0x5614b1fcb590 2024-08-03T20:54:01.312 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:01.310+0000 7fe685600640 1 -- 172.21.15.32:0/1307845301 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_get_version_reply(handle=1 version=81) ==== 24+0+0 (secure 0 0 0) 0x7fe68002d050 con 0x5614b1fcb590 2024-08-03T20:54:01.313 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:01.310+0000 7fe6a5f8a740 10 client.6648 fetch_fsmap learned FSMap version 81 2024-08-03T20:54:01.313 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:01.310+0000 7fe6a5f8a740 1 -- 172.21.15.32:0/1307845301 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,fsmap=81,monmap=2+}) -- 0x5614b23cf6f0 con 0x5614b1fcb590 2024-08-03T20:54:01.313 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:01.310+0000 7fe685600640 1 -- 172.21.15.32:0/1307845301 <== mon.0 v2:172.21.15.32:3300/0 4 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fe680010e10 con 0x5614b1fcb590 2024-08-03T20:54:01.313 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:01.310+0000 7fe685600640 1 -- 172.21.15.32:0/1307845301 <== mon.0 v2:172.21.15.32:3300/0 5 ==== fsmap(e 81) ==== 3502+0+0 (secure 0 0 0) 0x7fe680030020 con 0x5614b1fcb590 2024-08-03T20:54:01.313 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:01.310+0000 7fe6a5f8a740 10 client.6648 fetch_fsmap finished waiting for FSMap version 81 2024-08-03T20:54:01.313 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:01.310+0000 7fe6a5f8a740 10 client.6648 resolve_mds: resolved name 'c' to daemon mds.c 2024-08-03T20:54:01.313 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:01.310+0000 7fe6a5f8a740 20 client.6648 populate_metadata read hostname 'smithi032' 2024-08-03T20:54:01.313 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:01.310+0000 7fe6a5f8a740 1 --2- 172.21.15.32:0/1307845301 >> [v2:172.21.15.32:6832/1798758431,v1:172.21.15.32:6834/1798758431] conn(0x5614b1faae40 0x5614b23ead80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:01.313 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:01.310+0000 7fe6a5f8a740 4 client.6648 mds_command: new command op to 6396 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-08-03T20:54:01.313 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:01.310+0000 7fe6a5f8a740 1 -- 172.21.15.32:0/1307845301 --> [v2:172.21.15.32:6832/1798758431,v1:172.21.15.32:6834/1798758431] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5614b23069b0 con 0x5614b1faae40 2024-08-03T20:54:01.313 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:01.310+0000 7fe697e00640 1 --2- 172.21.15.32:0/1307845301 >> [v2:172.21.15.32:6832/1798758431,v1:172.21.15.32:6834/1798758431] conn(0x5614b1faae40 0x5614b23ead80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:01.313 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:01.311+0000 7fe697e00640 1 --2- 172.21.15.32:0/1307845301 >> [v2:172.21.15.32:6832/1798758431,v1:172.21.15.32:6834/1798758431] conn(0x5614b1faae40 0x5614b23ead80 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7108919c8efee8ed server_cookie=130703da970041d4 in_seq=0 out_seq=0 2024-08-03T20:54:01.313 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:01.311+0000 7fe685600640 10 client.6648 ms_handle_connect on v2:172.21.15.32:6832/1798758431 2024-08-03T20:54:01.314 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:01.312+0000 7fe685600640 1 -- 172.21.15.32:0/1307845301 <== mds.0 v2:172.21.15.32:6832/1798758431 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x5614b23069b0 con 0x5614b1faae40 2024-08-03T20:54:01.314 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:01.312+0000 7fe685600640 10 client.6648 handle_command_reply: tid=0 2024-08-03T20:54:01.314 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:01.312+0000 7fe6a5f8a740 1 client.6648 shutdown 2024-08-03T20:54:01.315 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:01.312+0000 7fe6a5f8a740 1 -- 172.21.15.32:0/1307845301 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5614b1fcb590 msgr2=0x5614b23ee280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:01.315 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:01.312+0000 7fe6a5f8a740 1 --2- 172.21.15.32:0/1307845301 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5614b1fcb590 0x5614b23ee280 secure :-1 s=READY pgs=823 cs=0 l=1 rev1=1 crypto rx=0x7fe680005000 tx=0x7fe680010b80 comp rx=0 tx=0).stop 2024-08-03T20:54:01.315 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:01.312+0000 7fe6a5f8a740 1 -- 172.21.15.32:0/1307845301 shutdown_connections 2024-08-03T20:54:01.315 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:01.312+0000 7fe6a5f8a740 1 --2- 172.21.15.32:0/1307845301 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5614b1fec270 0x5614b23ee7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:01.315 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:01.312+0000 7fe6a5f8a740 1 --2- 172.21.15.32:0/1307845301 >> [v2:172.21.15.32:6832/1798758431,v1:172.21.15.32:6834/1798758431] conn(0x5614b1faae40 0x5614b23ead80 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:01.315 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:01.312+0000 7fe6a5f8a740 1 --2- 172.21.15.32:0/1307845301 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5614b1fcb590 0x5614b23ee280 unknown :-1 s=CLOSED pgs=823 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:01.315 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:01.312+0000 7fe6a5f8a740 1 --2- 172.21.15.32:0/1307845301 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x5614b1fc33b0 0x5614b23cfcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:01.315 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:01.312+0000 7fe6a5f8a740 1 -- 172.21.15.32:0/1307845301 >> 172.21.15.32:0/1307845301 conn(0x5614b1fe03c0 msgr2=0x5614b238b390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:54:01.315 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:01.312+0000 7fe685600640 0 client.6648 ms_handle_reset on v2:172.21.15.32:6832/1798758431 2024-08-03T20:54:01.315 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:01.313+0000 7fe6a5f8a740 1 -- 172.21.15.32:0/1307845301 shutdown_connections 2024-08-03T20:54:01.315 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:01.313+0000 7fe6a5f8a740 1 -- 172.21.15.32:0/1307845301 wait complete. 2024-08-03T20:54:01.315 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:01.313+0000 7fe6a5f8a740 20 client.6648 trim_cache size 0 max 16384 2024-08-03T20:54:01.333 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:01.333+0000 7fe6a5f8a740 1 Processor -- start 2024-08-03T20:54:01.333 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:01.333+0000 7fe6a5f8a740 1 -- start start 2024-08-03T20:54:01.335 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:01.334+0000 7fe6a5f8a740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5614b1fc33b0 0x5614b238b940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:01.335 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:01.334+0000 7fe6a5f8a740 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x5614b1fcb590 0x5614b238d640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:01.335 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:01.334+0000 7fe6a5f8a740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5614b1fec270 0x5614b238db80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:01.335 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:01.334+0000 7fe6a5f8a740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fe6980b4d00 con 0x5614b1fec270 2024-08-03T20:54:01.335 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:01.334+0000 7fe6a5f8a740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7fe698138d30 con 0x5614b1fc33b0 2024-08-03T20:54:01.335 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:01.334+0000 7fe6a5f8a740 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7fe6980b46b0 con 0x5614b1fcb590 2024-08-03T20:54:01.336 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:01.334+0000 7fe69d000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5614b1fec270 0x5614b238db80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:01.336 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:01.334+0000 7fe69d000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5614b1fec270 0x5614b238db80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:57856/0 (socket says 172.21.15.32:57856) 2024-08-03T20:54:01.336 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:01.334+0000 7fe69d000640 1 -- 172.21.15.32:0/4233112026 learned_addr learned my addr 172.21.15.32:0/4233112026 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:54:01.336 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:01.335+0000 7fe697e00640 1 --2- 172.21.15.32:0/4233112026 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5614b1fc33b0 0x5614b238b940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:01.336 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:01.335+0000 7fe68fe00640 1 --2- 172.21.15.32:0/4233112026 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x5614b1fcb590 0x5614b238d640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:01.336 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:01.335+0000 7fe69d000640 1 -- 172.21.15.32:0/4233112026 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x5614b1fcb590 msgr2=0x5614b238d640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:54:01.336 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:01.335+0000 7fe69d000640 1 --2- 172.21.15.32:0/4233112026 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x5614b1fcb590 0x5614b238d640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:01.336 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:01.335+0000 7fe69d000640 1 -- 172.21.15.32:0/4233112026 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5614b1fc33b0 msgr2=0x5614b238b940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:54:01.336 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:01.335+0000 7fe69d000640 1 --2- 172.21.15.32:0/4233112026 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5614b1fc33b0 0x5614b238b940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:01.336 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:01.335+0000 7fe69d000640 1 -- 172.21.15.32:0/4233112026 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5614b23a5c90 con 0x5614b1fec270 2024-08-03T20:54:01.337 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:01.335+0000 7fe69d000640 1 --2- 172.21.15.32:0/4233112026 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5614b1fec270 0x5614b238db80 secure :-1 s=READY pgs=824 cs=0 l=1 rev1=1 crypto rx=0x7fe690097550 tx=0x7fe690061470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aac8a8308c142be4 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:54:01.338 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:01.335+0000 7fe687e00640 1 -- 172.21.15.32:0/4233112026 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe690063e60 con 0x5614b1fec270 2024-08-03T20:54:01.338 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:01.335+0000 7fe687e00640 1 -- 172.21.15.32:0/4233112026 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fe690062220 con 0x5614b1fec270 2024-08-03T20:54:01.338 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:01.335+0000 7fe687e00640 1 -- 172.21.15.32:0/4233112026 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe690061b20 con 0x5614b1fec270 2024-08-03T20:54:01.338 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:01.336+0000 7fe6a5f8a740 1 -- 172.21.15.32:0/4233112026 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5614b1fec270 msgr2=0x5614b238db80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:01.338 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:01.336+0000 7fe6a5f8a740 1 --2- 172.21.15.32:0/4233112026 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5614b1fec270 0x5614b238db80 secure :-1 s=READY pgs=824 cs=0 l=1 rev1=1 crypto rx=0x7fe690097550 tx=0x7fe690061470 comp rx=0 tx=0).stop 2024-08-03T20:54:01.338 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:01.336+0000 7fe6a5f8a740 1 -- 172.21.15.32:0/4233112026 shutdown_connections 2024-08-03T20:54:01.338 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:01.336+0000 7fe6a5f8a740 1 --2- 172.21.15.32:0/4233112026 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5614b1fec270 0x5614b238db80 unknown :-1 s=CLOSED pgs=824 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:01.338 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:01.336+0000 7fe6a5f8a740 1 --2- 172.21.15.32:0/4233112026 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x5614b1fcb590 0x5614b238d640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:01.338 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:01.336+0000 7fe6a5f8a740 1 --2- 172.21.15.32:0/4233112026 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5614b1fc33b0 0x5614b238b940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:01.338 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:01.336+0000 7fe6a5f8a740 1 -- 172.21.15.32:0/4233112026 >> 172.21.15.32:0/4233112026 conn(0x5614b1fe03c0 msgr2=0x5614b23abea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:54:01.338 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:01.336+0000 7fe6a5f8a740 1 -- 172.21.15.32:0/4233112026 shutdown_connections 2024-08-03T20:54:01.338 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:01.336+0000 7fe6a5f8a740 1 -- 172.21.15.32:0/4233112026 wait complete. 2024-08-03T20:54:01.338 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:01.337+0000 7fe6a5f8a740 1 -- start start 2024-08-03T20:54:01.338 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:01.337+0000 7fe6a5f8a740 1 Processor -- start 2024-08-03T20:54:01.341 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:01.339+0000 7fe6a5f8a740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5614b1fc33b0 0x5614b24559a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:01.341 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:01.339+0000 7fe6a5f8a740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5614b1fcb590 0x5614b238d640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:01.341 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:01.339+0000 7fe6a5f8a740 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x5614b1fec270 0x5614b238db80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:01.341 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:01.339+0000 7fe6a5f8a740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x5614b23a73c0 con 0x5614b1fc33b0 2024-08-03T20:54:01.342 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:01.339+0000 7fe6a5f8a740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x5614b23d02c0 con 0x5614b1fcb590 2024-08-03T20:54:01.342 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:01.339+0000 7fe6a5f8a740 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x5614b23cf190 con 0x5614b1fec270 2024-08-03T20:54:01.342 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:01.339+0000 7fe6a5f8a740 10 client.0 _ll_register_callbacks cb 0x7fe6800017d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-03T20:54:01.342 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:01.339+0000 7fe697e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5614b1fc33b0 0x5614b24559a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:01.342 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:01.339+0000 7fe697e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5614b1fc33b0 0x5614b24559a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:57870/0 (socket says 172.21.15.32:57870) 2024-08-03T20:54:01.342 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:01.339+0000 7fe697e00640 1 -- 172.21.15.32:0/762450762 learned_addr learned my addr 172.21.15.32:0/762450762 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:54:01.342 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:01.339+0000 7fe69d000640 1 --2- 172.21.15.32:0/762450762 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x5614b1fec270 0x5614b238db80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:01.342 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:01.339+0000 7fe68fe00640 1 --2- 172.21.15.32:0/762450762 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5614b1fcb590 0x5614b238d640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:01.342 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:01.339+0000 7fe697e00640 1 -- 172.21.15.32:0/762450762 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x5614b1fec270 msgr2=0x5614b238db80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:54:01.342 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:01.339+0000 7fe697e00640 1 --2- 172.21.15.32:0/762450762 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x5614b1fec270 0x5614b238db80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:01.342 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:01.339+0000 7fe697e00640 1 -- 172.21.15.32:0/762450762 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5614b1fcb590 msgr2=0x5614b238d640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:54:01.342 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:01.339+0000 7fe697e00640 1 --2- 172.21.15.32:0/762450762 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5614b1fcb590 0x5614b238d640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:01.343 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:01.339+0000 7fe697e00640 1 -- 172.21.15.32:0/762450762 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5614b23d04a0 con 0x5614b1fc33b0 2024-08-03T20:54:01.343 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:01.339+0000 7fe68fe00640 1 --2- 172.21.15.32:0/762450762 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5614b1fcb590 0x5614b238d640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-03T20:54:01.343 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:01.340+0000 7fe697e00640 1 --2- 172.21.15.32:0/762450762 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5614b1fc33b0 0x5614b24559a0 secure :-1 s=READY pgs=825 cs=0 l=1 rev1=1 crypto rx=0x7fe688009b50 tx=0x7fe688005f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=812d17a8ea266040 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:54:01.343 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:01.340+0000 7fe685600640 10 client.0 ms_handle_connect on v2:172.21.15.32:3300/0 2024-08-03T20:54:01.343 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:01.340+0000 7fe685600640 1 -- 172.21.15.32:0/762450762 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe688038040 con 0x5614b1fc33b0 2024-08-03T20:54:01.343 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:01.340+0000 7fe685600640 1 -- 172.21.15.32:0/762450762 <== mon.0 v2:172.21.15.32:3300/0 2 ==== osd_map(71..71 src has 1..71) ==== 5795+0+0 (secure 0 0 0) 0x7fe68804c920 con 0x5614b1fc33b0 2024-08-03T20:54:01.343 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:01.340+0000 7fe6a5f8a740 1 -- 172.21.15.32:0/762450762 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe690062220 con 0x5614b1fc33b0 2024-08-03T20:54:01.343 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:01.340+0000 7fe685600640 1 -- 172.21.15.32:0/762450762 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_get_version_reply(handle=1 version=81) ==== 24+0+0 (secure 0 0 0) 0x7fe68804cb20 con 0x5614b1fc33b0 2024-08-03T20:54:01.343 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:01.340+0000 7fe6a5f8a740 10 client.6654 fetch_fsmap learned FSMap version 81 2024-08-03T20:54:01.343 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:01.340+0000 7fe6a5f8a740 1 -- 172.21.15.32:0/762450762 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,fsmap=81,monmap=2+}) -- 0x5614b2458770 con 0x5614b1fc33b0 2024-08-03T20:54:01.343 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:01.341+0000 7fe685600640 1 -- 172.21.15.32:0/762450762 <== mon.0 v2:172.21.15.32:3300/0 4 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fe690062220 con 0x5614b1fc33b0 2024-08-03T20:54:01.343 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:01.341+0000 7fe685600640 1 -- 172.21.15.32:0/762450762 <== mon.0 v2:172.21.15.32:3300/0 5 ==== fsmap(e 81) ==== 3502+0+0 (secure 0 0 0) 0x7fe688049040 con 0x5614b1fc33b0 2024-08-03T20:54:01.343 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:01.341+0000 7fe6a5f8a740 10 client.6654 fetch_fsmap finished waiting for FSMap version 81 2024-08-03T20:54:01.343 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:01.341+0000 7fe6a5f8a740 10 client.6654 resolve_mds: resolved name 'c' to daemon mds.c 2024-08-03T20:54:01.344 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:01.341+0000 7fe6a5f8a740 20 client.6654 populate_metadata read hostname 'smithi032' 2024-08-03T20:54:01.344 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:01.341+0000 7fe6a5f8a740 1 --2- 172.21.15.32:0/762450762 >> [v2:172.21.15.32:6832/1798758431,v1:172.21.15.32:6834/1798758431] conn(0x5614b23ed960 0x5614b23a9130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:01.344 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:01.341+0000 7fe6a5f8a740 4 client.6654 mds_command: new command op to 6396 tid=0 multi_id=0 [{"prefix": "status"}] 2024-08-03T20:54:01.344 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:01.341+0000 7fe6a5f8a740 1 -- 172.21.15.32:0/762450762 --> [v2:172.21.15.32:6832/1798758431,v1:172.21.15.32:6834/1798758431] -- command(tid 0: {"prefix": "status"}) -- 0x5614b2302450 con 0x5614b23ed960 2024-08-03T20:54:01.344 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:01.341+0000 7fe68fe00640 1 --2- 172.21.15.32:0/762450762 >> [v2:172.21.15.32:6832/1798758431,v1:172.21.15.32:6834/1798758431] conn(0x5614b23ed960 0x5614b23a9130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:01.345 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:01.342+0000 7fe68fe00640 1 --2- 172.21.15.32:0/762450762 >> [v2:172.21.15.32:6832/1798758431,v1:172.21.15.32:6834/1798758431] conn(0x5614b23ed960 0x5614b23a9130 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=149e1eaadf66b53b server_cookie=c776f93a09f015db in_seq=0 out_seq=0 2024-08-03T20:54:01.345 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:01.342+0000 7fe685600640 10 client.6654 ms_handle_connect on v2:172.21.15.32:6832/1798758431 2024-08-03T20:54:01.345 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:01.342+0000 7fe685600640 1 -- 172.21.15.32:0/762450762 <== mds.0 v2:172.21.15.32:6832/1798758431 1 ==== command_reply(tid 0: 0 ) ==== 8+0+331 (crc 0 0 0) 0x5614b2302450 con 0x5614b23ed960 2024-08-03T20:54:01.345 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:01.342+0000 7fe685600640 10 client.6654 handle_command_reply: tid=0 2024-08-03T20:54:01.345 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:01.342+0000 7fe6a5f8a740 1 client.6654 shutdown 2024-08-03T20:54:01.345 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:01.342+0000 7fe6a5f8a740 1 -- 172.21.15.32:0/762450762 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5614b1fc33b0 msgr2=0x5614b24559a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:01.345 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:01.342+0000 7fe6a5f8a740 1 --2- 172.21.15.32:0/762450762 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5614b1fc33b0 0x5614b24559a0 secure :-1 s=READY pgs=825 cs=0 l=1 rev1=1 crypto rx=0x7fe688009b50 tx=0x7fe688005f90 comp rx=0 tx=0).stop 2024-08-03T20:54:01.345 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:01.342+0000 7fe6a5f8a740 1 -- 172.21.15.32:0/762450762 shutdown_connections 2024-08-03T20:54:01.346 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:01.342+0000 7fe6a5f8a740 1 --2- 172.21.15.32:0/762450762 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5614b1fcb590 0x5614b238d640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:01.346 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:01.342+0000 7fe6a5f8a740 1 --2- 172.21.15.32:0/762450762 >> [v2:172.21.15.32:6832/1798758431,v1:172.21.15.32:6834/1798758431] conn(0x5614b23ed960 0x5614b23a9130 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:01.346 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:01.342+0000 7fe6a5f8a740 1 --2- 172.21.15.32:0/762450762 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x5614b1fec270 0x5614b238db80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:01.346 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:01.342+0000 7fe6a5f8a740 1 --2- 172.21.15.32:0/762450762 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5614b1fc33b0 0x5614b24559a0 unknown :-1 s=CLOSED pgs=825 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:01.346 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:01.342+0000 7fe685600640 0 client.6654 ms_handle_reset on v2:172.21.15.32:6832/1798758431 2024-08-03T20:54:01.346 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:01.342+0000 7fe6a5f8a740 1 -- 172.21.15.32:0/762450762 >> 172.21.15.32:0/762450762 conn(0x5614b1fe03c0 msgr2=0x5614b23abe00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:54:01.346 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:01.342+0000 7fe6a5f8a740 1 -- 172.21.15.32:0/762450762 shutdown_connections 2024-08-03T20:54:01.346 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:01.342+0000 7fe6a5f8a740 1 -- 172.21.15.32:0/762450762 wait complete. 2024-08-03T20:54:01.346 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:01.342+0000 7fe6a5f8a740 20 client.6654 trim_cache size 0 max 16384 2024-08-03T20:54:01.346 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-08-03T20:54:01.347 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_fsid": "40ab141a-01d7-49b9-ae3c-2edec9876e95", 2024-08-03T20:54:01.347 INFO:teuthology.orchestra.run.smithi032.stdout: "whoami": 0, 2024-08-03T20:54:01.347 INFO:teuthology.orchestra.run.smithi032.stdout: "id": 6396, 2024-08-03T20:54:01.347 INFO:teuthology.orchestra.run.smithi032.stdout: "want_state": "up:active", 2024-08-03T20:54:01.347 INFO:teuthology.orchestra.run.smithi032.stdout: "state": "up:active", 2024-08-03T20:54:01.347 INFO:teuthology.orchestra.run.smithi032.stdout: "fs_name": "cephfs", 2024-08-03T20:54:01.347 INFO:teuthology.orchestra.run.smithi032.stdout: "rank_uptime": 5.3207956750000003, 2024-08-03T20:54:01.347 INFO:teuthology.orchestra.run.smithi032.stdout: "mdsmap_epoch": 81, 2024-08-03T20:54:01.347 INFO:teuthology.orchestra.run.smithi032.stdout: "osdmap_epoch": 71, 2024-08-03T20:54:01.347 INFO:teuthology.orchestra.run.smithi032.stdout: "osdmap_epoch_barrier": 71, 2024-08-03T20:54:01.347 INFO:teuthology.orchestra.run.smithi032.stdout: "uptime": 50.729034900000002 2024-08-03T20:54:01.347 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-08-03T20:54:01.348 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:01.343+0000 7fe69f000640 1 -- 172.21.15.32:0/1019794313 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fe674077010 msgr2=0x7fe674079520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:01.348 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:01.343+0000 7fe69f000640 1 --2- 172.21.15.32:0/1019794313 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fe674077010 0x7fe674079520 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7fe688035000 tx=0x7fe68802d040 comp rx=0 tx=0).stop 2024-08-03T20:54:01.348 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:01.343+0000 7fe69f000640 1 -- 172.21.15.32:0/1019794313 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe69815b9b0 msgr2=0x7fe69815bdc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:01.348 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:01.343+0000 7fe69f000640 1 --2- 172.21.15.32:0/1019794313 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe69815b9b0 0x7fe69815bdc0 secure :-1 s=READY pgs=821 cs=0 l=1 rev1=1 crypto rx=0x7fe698072860 tx=0x7fe690001950 comp rx=0 tx=0).stop 2024-08-03T20:54:01.349 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:01.343+0000 7fe69f000640 1 -- 172.21.15.32:0/1019794313 shutdown_connections 2024-08-03T20:54:01.349 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:01.343+0000 7fe69f000640 1 --2- 172.21.15.32:0/1019794313 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fe674077010 0x7fe674079520 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:01.349 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:01.343+0000 7fe69f000640 1 --2- 172.21.15.32:0/1019794313 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe69815b9b0 0x7fe69815bdc0 unknown :-1 s=CLOSED pgs=821 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:01.349 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:01.343+0000 7fe69f000640 1 --2- 172.21.15.32:0/1019794313 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fe69813a1a0 0x7fe69813a5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:01.349 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:01.343+0000 7fe69f000640 1 --2- 172.21.15.32:0/1019794313 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fe67c003a30 0x7fe69813cb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:01.349 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:01.343+0000 7fe69f000640 1 -- 172.21.15.32:0/1019794313 >> 172.21.15.32:0/1019794313 conn(0x7fe6980b1bd0 msgr2=0x7fe6980b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:54:01.349 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:01.343+0000 7fe69f000640 1 -- 172.21.15.32:0/1019794313 shutdown_connections 2024-08-03T20:54:01.349 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:01.343+0000 7fe69f000640 1 -- 172.21.15.32:0/1019794313 wait complete. 2024-08-03T20:54:01.365 DEBUG:teuthology.orchestra.run.smithi032:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'stat a/ && find a/') 2024-08-03T20:54:01.444 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:01 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-08-03T20:54:01.448 INFO:teuthology.orchestra.run.smithi032.stdout: File: a/ 2024-08-03T20:54:01.448 INFO:teuthology.orchestra.run.smithi032.stdout: Size: 2 Blocks: 0 IO Block: 65536 directory 2024-08-03T20:54:01.449 INFO:teuthology.orchestra.run.smithi032.stdout:Device: 29h/41d Inode: 1099511627776 Links: 4 2024-08-03T20:54:01.449 INFO:teuthology.orchestra.run.smithi032.stdout:Access: (0755/drwxr-xr-x) Uid: ( 1000/ ubuntu) Gid: ( 1316/ ubuntu) 2024-08-03T20:54:01.449 INFO:teuthology.orchestra.run.smithi032.stdout:Context: unconfined_u:object_r:unlabeled_t:s0 2024-08-03T20:54:01.449 INFO:teuthology.orchestra.run.smithi032.stdout:Access: 2024-08-03 20:53:25.737910674 +0000 2024-08-03T20:54:01.449 INFO:teuthology.orchestra.run.smithi032.stdout:Modify: 2024-08-03 20:53:31.656791471 +0000 2024-08-03T20:54:01.449 INFO:teuthology.orchestra.run.smithi032.stdout:Change: 2024-08-03 20:53:31.656791471 +0000 2024-08-03T20:54:01.449 INFO:teuthology.orchestra.run.smithi032.stdout: Birth: 2024-08-03 20:53:25.737910674 +0000 2024-08-03T20:54:01.455 INFO:teuthology.orchestra.run.smithi032.stdout:a/ 2024-08-03T20:54:01.455 INFO:teuthology.orchestra.run.smithi032.stdout:a/z 2024-08-03T20:54:01.455 INFO:teuthology.orchestra.run.smithi032.stdout:a/z/d 2024-08-03T20:54:01.455 INFO:teuthology.orchestra.run.smithi032.stdout:a/c 2024-08-03T20:54:01.455 INFO:teuthology.orchestra.run.smithi032.stdout:a/c/d 2024-08-03T20:54:01.455 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:01 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-08-03T20:54:01.464 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.5:0 flush journal 2024-08-03T20:54:01.695 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:01.694+0000 7f4f85800640 1 -- 172.21.15.32:0/3545679865 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4f64004e60 msgr2=0x7f4f640052b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:01.695 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:01.694+0000 7f4f85800640 1 --2- 172.21.15.32:0/3545679865 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4f64004e60 0x7f4f640052b0 secure :-1 s=READY pgs=827 cs=0 l=1 rev1=1 crypto rx=0x7f4f78001d90 tx=0x7f4f64005830 comp rx=0 tx=0).stop 2024-08-03T20:54:01.695 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:01.694+0000 7f4f7f400640 1 -- 172.21.15.32:0/3545679865 reap_dead start 2024-08-03T20:54:01.695 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:01.694+0000 7f4f85800640 1 -- 172.21.15.32:0/3545679865 shutdown_connections 2024-08-03T20:54:01.695 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:01.694+0000 7f4f85800640 1 -- 172.21.15.32:0/3545679865 >> 172.21.15.32:0/3545679865 conn(0x7f4f800b1bd0 msgr2=0x7f4f800b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:54:01.695 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:01.694+0000 7f4f85800640 1 -- 172.21.15.32:0/3545679865 shutdown_connections 2024-08-03T20:54:01.695 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:01.694+0000 7f4f85800640 1 -- 172.21.15.32:0/3545679865 wait complete. 2024-08-03T20:54:01.697 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:01.696+0000 7f4f85800640 1 Processor -- start 2024-08-03T20:54:01.697 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:01.696+0000 7f4f85800640 1 -- start start 2024-08-03T20:54:01.700 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:01.698+0000 7f4f85800640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f4f64004e60 0x7f4f8013a770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:01.700 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:01.698+0000 7f4f85800640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f4f8013ddc0 0x7f4f80073400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:01.700 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:01.698+0000 7f4f85800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4f80073940 0x7f4f801f8820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:01.700 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:01.698+0000 7f4f85800640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f4f8015ffe0 con 0x7f4f80073940 2024-08-03T20:54:01.700 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:01.698+0000 7f4f85800640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f4f800700d0 con 0x7f4f64004e60 2024-08-03T20:54:01.700 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:01.698+0000 7f4f85800640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7f4f80160160 con 0x7f4f8013ddc0 2024-08-03T20:54:01.700 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:01.698+0000 7f4f7f400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4f80073940 0x7f4f801f8820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:01.700 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:01.698+0000 7f4f7f400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4f80073940 0x7f4f801f8820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:57902/0 (socket says 172.21.15.32:57902) 2024-08-03T20:54:01.700 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:01.698+0000 7f4f7f400640 1 -- 172.21.15.32:0/3707637078 learned_addr learned my addr 172.21.15.32:0/3707637078 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:54:01.700 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:01.698+0000 7f4f77e00640 1 --2- 172.21.15.32:0/3707637078 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f4f8013ddc0 0x7f4f80073400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:01.701 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:01.698+0000 7f4f7ea00640 1 --2- 172.21.15.32:0/3707637078 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f4f64004e60 0x7f4f8013a770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:01.701 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:01.699+0000 7f4f7f400640 1 -- 172.21.15.32:0/3707637078 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f4f8013ddc0 msgr2=0x7f4f80073400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:01.701 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:01.699+0000 7f4f7f400640 1 --2- 172.21.15.32:0/3707637078 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f4f8013ddc0 0x7f4f80073400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:01.701 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:01.699+0000 7f4f7f400640 1 -- 172.21.15.32:0/3707637078 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f4f64004e60 msgr2=0x7f4f8013a770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:01.701 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:01.699+0000 7f4f7f400640 1 --2- 172.21.15.32:0/3707637078 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f4f64004e60 0x7f4f8013a770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:01.701 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:01.699+0000 7f4f7f400640 1 -- 172.21.15.32:0/3707637078 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4f80073d50 con 0x7f4f80073940 2024-08-03T20:54:01.701 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:01.699+0000 7f4f7ea00640 1 --2- 172.21.15.32:0/3707637078 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f4f64004e60 0x7f4f8013a770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-03T20:54:01.701 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:01.699+0000 7f4f7f400640 1 --2- 172.21.15.32:0/3707637078 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4f80073940 0x7f4f801f8820 secure :-1 s=READY pgs=828 cs=0 l=1 rev1=1 crypto rx=0x7f4f78001eb0 tx=0x7f4f78001b10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:54:01.701 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:01.699+0000 7f4f76a00640 1 -- 172.21.15.32:0/3707637078 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f780062b0 con 0x7f4f80073940 2024-08-03T20:54:01.701 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:01.699+0000 7f4f76a00640 1 -- 172.21.15.32:0/3707637078 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f4f78006450 con 0x7f4f80073940 2024-08-03T20:54:01.702 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:01.699+0000 7f4f76a00640 1 -- 172.21.15.32:0/3707637078 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f78006880 con 0x7f4f80073940 2024-08-03T20:54:01.702 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:01.699+0000 7f4f85800640 1 -- 172.21.15.32:0/3707637078 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4f801f52b0 con 0x7f4f80073940 2024-08-03T20:54:01.702 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:01.699+0000 7f4f85800640 1 -- 172.21.15.32:0/3707637078 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4f800a9b40 con 0x7f4f80073940 2024-08-03T20:54:01.702 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:01.702+0000 7f4f76a00640 1 -- 172.21.15.32:0/3707637078 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f4f78003a30 con 0x7f4f80073940 2024-08-03T20:54:01.703 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:01.703+0000 7f4f76a00640 1 --2- 172.21.15.32:0/3707637078 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f4f5c077010 0x7f4f5c079520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:01.703 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:01.703+0000 7f4f76a00640 1 -- 172.21.15.32:0/3707637078 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(71..71 src has 1..71) ==== 5795+0+0 (secure 0 0 0) 0x7f4f780b85f0 con 0x7f4f80073940 2024-08-03T20:54:01.704 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:01.703+0000 7f4f7ea00640 1 --2- 172.21.15.32:0/3707637078 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f4f5c077010 0x7f4f5c079520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:01.704 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:01.703+0000 7f4f8c7c2740 1 Processor -- start 2024-08-03T20:54:01.704 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:01.704+0000 7f4f8c7c2740 1 -- start start 2024-08-03T20:54:01.704 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:01.704+0000 7f4f7ea00640 1 --2- 172.21.15.32:0/3707637078 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f4f5c077010 0x7f4f5c079520 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f4f70001010 tx=0x7f4f70006970 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:54:01.707 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:01.705+0000 7f4f8c7c2740 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55a6b1d4c390 0x55a6b211c900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:01.707 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:01.705+0000 7f4f8c7c2740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a6b1d234d0 0x55a6b211ce40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:01.707 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:01.705+0000 7f4f8c7c2740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55a6b1d2b6b0 0x55a6b2139d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:01.707 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:01.705+0000 7f4f8c7c2740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55a6b2065470 con 0x55a6b1d234d0 2024-08-03T20:54:01.707 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:01.705+0000 7f4f8c7c2740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x55a6b2054970 con 0x55a6b1d2b6b0 2024-08-03T20:54:01.708 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:01.705+0000 7f4f8c7c2740 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x55a6b20589b0 con 0x55a6b1d4c390 2024-08-03T20:54:01.708 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:01.705+0000 7f4f77e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a6b1d234d0 0x55a6b211ce40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:01.708 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:01.705+0000 7f4f77e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a6b1d234d0 0x55a6b211ce40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:57906/0 (socket says 172.21.15.32:57906) 2024-08-03T20:54:01.708 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:01.705+0000 7f4f77e00640 1 -- 172.21.15.32:0/2594378604 learned_addr learned my addr 172.21.15.32:0/2594378604 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:54:01.708 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:01.705+0000 7f4f7ea00640 1 --2- 172.21.15.32:0/2594378604 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55a6b1d4c390 0x55a6b211c900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:01.708 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:01.705+0000 7f4f7f400640 1 --2- 172.21.15.32:0/2594378604 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55a6b1d2b6b0 0x55a6b2139d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:01.708 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:01.706+0000 7f4f77e00640 1 -- 172.21.15.32:0/2594378604 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55a6b1d4c390 msgr2=0x55a6b211c900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:54:01.708 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:01.706+0000 7f4f77e00640 1 --2- 172.21.15.32:0/2594378604 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55a6b1d4c390 0x55a6b211c900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:01.708 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:01.706+0000 7f4f77e00640 1 -- 172.21.15.32:0/2594378604 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55a6b1d2b6b0 msgr2=0x55a6b2139d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:54:01.709 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:01.706+0000 7f4f77e00640 1 --2- 172.21.15.32:0/2594378604 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55a6b1d2b6b0 0x55a6b2139d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:01.709 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:01.706+0000 7f4f77e00640 1 -- 172.21.15.32:0/2594378604 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a6b1e2e7d0 con 0x55a6b1d234d0 2024-08-03T20:54:01.709 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:01.706+0000 7f4f77e00640 1 --2- 172.21.15.32:0/2594378604 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a6b1d234d0 0x55a6b211ce40 secure :-1 s=READY pgs=829 cs=0 l=1 rev1=1 crypto rx=0x7f4f6800dc20 tx=0x7f4f68012820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=262e7cbfd346b851 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:54:01.709 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:01.706+0000 7f4f6f400640 1 -- 172.21.15.32:0/2594378604 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f6801d450 con 0x55a6b1d234d0 2024-08-03T20:54:01.709 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:01.706+0000 7f4f6f400640 1 -- 172.21.15.32:0/2594378604 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f4f6801d6e0 con 0x55a6b1d234d0 2024-08-03T20:54:01.709 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:01.706+0000 7f4f6f400640 1 -- 172.21.15.32:0/2594378604 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f6801daf0 con 0x55a6b1d234d0 2024-08-03T20:54:01.710 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:01.707+0000 7f4f8c7c2740 1 -- 172.21.15.32:0/2594378604 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a6b1d234d0 msgr2=0x55a6b211ce40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:01.710 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:01.707+0000 7f4f8c7c2740 1 --2- 172.21.15.32:0/2594378604 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a6b1d234d0 0x55a6b211ce40 secure :-1 s=READY pgs=829 cs=0 l=1 rev1=1 crypto rx=0x7f4f6800dc20 tx=0x7f4f68012820 comp rx=0 tx=0).stop 2024-08-03T20:54:01.710 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:01.707+0000 7f4f8c7c2740 1 -- 172.21.15.32:0/2594378604 shutdown_connections 2024-08-03T20:54:01.710 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:01.707+0000 7f4f8c7c2740 1 --2- 172.21.15.32:0/2594378604 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55a6b1d2b6b0 0x55a6b2139d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:01.710 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:01.707+0000 7f4f8c7c2740 1 --2- 172.21.15.32:0/2594378604 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a6b1d234d0 0x55a6b211ce40 unknown :-1 s=CLOSED pgs=829 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:01.710 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:01.707+0000 7f4f8c7c2740 1 --2- 172.21.15.32:0/2594378604 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55a6b1d4c390 0x55a6b211c900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:01.711 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:01.707+0000 7f4f8c7c2740 1 -- 172.21.15.32:0/2594378604 >> 172.21.15.32:0/2594378604 conn(0x55a6b1d404e0 msgr2=0x55a6b20f2bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:54:01.711 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:01.707+0000 7f4f8c7c2740 1 -- 172.21.15.32:0/2594378604 shutdown_connections 2024-08-03T20:54:01.711 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:01.707+0000 7f4f8c7c2740 1 -- 172.21.15.32:0/2594378604 wait complete. 2024-08-03T20:54:01.711 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:01.708+0000 7f4f8c7c2740 1 -- start start 2024-08-03T20:54:01.711 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:01.709+0000 7f4f8c7c2740 1 Processor -- start 2024-08-03T20:54:01.714 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:01.710+0000 7f4f8c7c2740 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55a6b1d234d0 0x55a6b2131640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:01.714 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:01.710+0000 7f4f8c7c2740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a6b1d2b6b0 0x55a6b2147f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:01.714 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:01.710+0000 7f4f8c7c2740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55a6b1d4c390 0x55a6b21475c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:01.714 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:01.710+0000 7f4f8c7c2740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55a6b1f8cc90 con 0x55a6b1d2b6b0 2024-08-03T20:54:01.714 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:01.710+0000 7f4f8c7c2740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x55a6b1feefd0 con 0x55a6b1d4c390 2024-08-03T20:54:01.714 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:01.710+0000 7f4f8c7c2740 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x55a6b211d380 con 0x55a6b1d234d0 2024-08-03T20:54:01.714 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:01.710+0000 7f4f8c7c2740 10 client.0 _ll_register_callbacks cb 0x55a6b2009e50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-03T20:54:01.714 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:01.710+0000 7f4f77e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a6b1d2b6b0 0x55a6b2147f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:01.714 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:01.710+0000 7f4f77e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a6b1d2b6b0 0x55a6b2147f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:57918/0 (socket says 172.21.15.32:57918) 2024-08-03T20:54:01.714 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:01.710+0000 7f4f77e00640 1 -- 172.21.15.32:0/3570860714 learned_addr learned my addr 172.21.15.32:0/3570860714 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:54:01.714 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:01.711+0000 7f4f7f400640 1 --2- 172.21.15.32:0/3570860714 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55a6b1d4c390 0x55a6b21475c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:01.715 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:01.711+0000 7f4f7ea00640 1 --2- 172.21.15.32:0/3570860714 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55a6b1d234d0 0x55a6b2131640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:01.715 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:01.711+0000 7f4f77e00640 1 -- 172.21.15.32:0/3570860714 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55a6b1d234d0 msgr2=0x55a6b2131640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:54:01.715 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:01.711+0000 7f4f77e00640 1 --2- 172.21.15.32:0/3570860714 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55a6b1d234d0 0x55a6b2131640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:01.715 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:01.711+0000 7f4f77e00640 1 -- 172.21.15.32:0/3570860714 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55a6b1d4c390 msgr2=0x55a6b21475c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:54:01.715 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:01.711+0000 7f4f77e00640 1 --2- 172.21.15.32:0/3570860714 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55a6b1d4c390 0x55a6b21475c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:01.715 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:01.711+0000 7f4f77e00640 1 -- 172.21.15.32:0/3570860714 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a6b1db7970 con 0x55a6b1d2b6b0 2024-08-03T20:54:01.715 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:01.711+0000 7f4f77e00640 1 --2- 172.21.15.32:0/3570860714 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a6b1d2b6b0 0x55a6b2147f10 secure :-1 s=READY pgs=830 cs=0 l=1 rev1=1 crypto rx=0x55a6b20f4790 tx=0x7f4f6800d830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b4bf914046b6c4bd server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:54:01.715 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:01.711+0000 7f4f63e00640 10 client.0 ms_handle_connect on v2:172.21.15.32:3300/0 2024-08-03T20:54:01.715 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:01.711+0000 7f4f63e00640 1 -- 172.21.15.32:0/3570860714 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f6802b060 con 0x55a6b1d2b6b0 2024-08-03T20:54:01.715 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:01.711+0000 7f4f63e00640 1 -- 172.21.15.32:0/3570860714 <== mon.0 v2:172.21.15.32:3300/0 2 ==== osd_map(71..71 src has 1..71) ==== 5795+0+0 (secure 0 0 0) 0x7f4f68004390 con 0x55a6b1d2b6b0 2024-08-03T20:54:01.715 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:01.711+0000 7f4f8c7c2740 1 -- 172.21.15.32:0/3570860714 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4f6801d6e0 con 0x55a6b1d2b6b0 2024-08-03T20:54:01.715 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:01.712+0000 7f4f63e00640 1 -- 172.21.15.32:0/3570860714 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_get_version_reply(handle=1 version=81) ==== 24+0+0 (secure 0 0 0) 0x7f4f68031050 con 0x55a6b1d2b6b0 2024-08-03T20:54:01.715 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:01.712+0000 7f4f8c7c2740 10 client.6666 fetch_fsmap learned FSMap version 81 2024-08-03T20:54:01.716 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:01.712+0000 7f4f8c7c2740 1 -- 172.21.15.32:0/3570860714 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,fsmap=81,monmap=2+}) -- 0x55a6b1cfd270 con 0x55a6b1d2b6b0 2024-08-03T20:54:01.716 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:01.712+0000 7f4f63e00640 1 -- 172.21.15.32:0/3570860714 <== mon.0 v2:172.21.15.32:3300/0 4 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f4f6801d6e0 con 0x55a6b1d2b6b0 2024-08-03T20:54:01.716 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:01.712+0000 7f4f63e00640 1 -- 172.21.15.32:0/3570860714 <== mon.0 v2:172.21.15.32:3300/0 5 ==== fsmap(e 81) ==== 3502+0+0 (secure 0 0 0) 0x7f4f68034020 con 0x55a6b1d2b6b0 2024-08-03T20:54:01.716 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:01.712+0000 7f4f8c7c2740 10 client.6666 fetch_fsmap finished waiting for FSMap version 81 2024-08-03T20:54:01.716 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:01.712+0000 7f4f8c7c2740 10 client.6666 resolve_mds: resolved 5:0 to role '5:0' aka daemon mds.c 2024-08-03T20:54:01.716 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:01.712+0000 7f4f8c7c2740 20 client.6666 populate_metadata read hostname 'smithi032' 2024-08-03T20:54:01.716 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:01.712+0000 7f4f8c7c2740 1 --2- 172.21.15.32:0/3570860714 >> [v2:172.21.15.32:6832/1798758431,v1:172.21.15.32:6834/1798758431] conn(0x55a6b1d0af60 0x55a6b2139aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:01.716 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:01.712+0000 7f4f8c7c2740 4 client.6666 mds_command: new command op to 6396 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-08-03T20:54:01.716 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:01.712+0000 7f4f8c7c2740 1 -- 172.21.15.32:0/3570860714 --> [v2:172.21.15.32:6832/1798758431,v1:172.21.15.32:6834/1798758431] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a6b20673c0 con 0x55a6b1d0af60 2024-08-03T20:54:01.716 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:01.713+0000 7f4f7ea00640 1 --2- 172.21.15.32:0/3570860714 >> [v2:172.21.15.32:6832/1798758431,v1:172.21.15.32:6834/1798758431] conn(0x55a6b1d0af60 0x55a6b2139aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:01.716 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:01.713+0000 7f4f7ea00640 1 --2- 172.21.15.32:0/3570860714 >> [v2:172.21.15.32:6832/1798758431,v1:172.21.15.32:6834/1798758431] conn(0x55a6b1d0af60 0x55a6b2139aa0 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e8cf56b8e5537d6b server_cookie=a6100ec4b83589d4 in_seq=0 out_seq=0 2024-08-03T20:54:01.716 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:01.713+0000 7f4f63e00640 10 client.6666 ms_handle_connect on v2:172.21.15.32:6832/1798758431 2024-08-03T20:54:01.718 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:01.714+0000 7f4f63e00640 1 -- 172.21.15.32:0/3570860714 <== mds.0 v2:172.21.15.32:6832/1798758431 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55a6b20673c0 con 0x55a6b1d0af60 2024-08-03T20:54:01.718 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:01.714+0000 7f4f63e00640 10 client.6666 handle_command_reply: tid=0 2024-08-03T20:54:01.718 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:01.714+0000 7f4f8c7c2740 1 client.6666 shutdown 2024-08-03T20:54:01.718 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:01.714+0000 7f4f8c7c2740 1 -- 172.21.15.32:0/3570860714 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a6b1d2b6b0 msgr2=0x55a6b2147f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:01.718 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:01.714+0000 7f4f8c7c2740 1 --2- 172.21.15.32:0/3570860714 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a6b1d2b6b0 0x55a6b2147f10 secure :-1 s=READY pgs=830 cs=0 l=1 rev1=1 crypto rx=0x55a6b20f4790 tx=0x7f4f6800d830 comp rx=0 tx=0).stop 2024-08-03T20:54:01.718 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:01.714+0000 7f4f8c7c2740 1 -- 172.21.15.32:0/3570860714 shutdown_connections 2024-08-03T20:54:01.718 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:01.714+0000 7f4f8c7c2740 1 --2- 172.21.15.32:0/3570860714 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55a6b1d4c390 0x55a6b21475c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:01.718 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:01.714+0000 7f4f8c7c2740 1 --2- 172.21.15.32:0/3570860714 >> [v2:172.21.15.32:6832/1798758431,v1:172.21.15.32:6834/1798758431] conn(0x55a6b1d0af60 0x55a6b2139aa0 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:01.718 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:01.714+0000 7f4f8c7c2740 1 --2- 172.21.15.32:0/3570860714 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a6b1d2b6b0 0x55a6b2147f10 unknown :-1 s=CLOSED pgs=830 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:01.718 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:01.714+0000 7f4f8c7c2740 1 --2- 172.21.15.32:0/3570860714 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55a6b1d234d0 0x55a6b2131640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:01.718 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:01.714+0000 7f4f63e00640 0 client.6666 ms_handle_reset on v2:172.21.15.32:6832/1798758431 2024-08-03T20:54:01.718 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:01.714+0000 7f4f8c7c2740 1 -- 172.21.15.32:0/3570860714 >> 172.21.15.32:0/3570860714 conn(0x55a6b1d404e0 msgr2=0x55a6b210b2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:54:01.719 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:01.715+0000 7f4f8c7c2740 1 -- 172.21.15.32:0/3570860714 shutdown_connections 2024-08-03T20:54:01.719 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:01.715+0000 7f4f8c7c2740 1 -- 172.21.15.32:0/3570860714 wait complete. 2024-08-03T20:54:01.719 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:01.715+0000 7f4f8c7c2740 20 client.6666 trim_cache size 0 max 16384 2024-08-03T20:54:01.731 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:01.731+0000 7f4f8c7c2740 1 Processor -- start 2024-08-03T20:54:01.732 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:01.731+0000 7f4f8c7c2740 1 -- start start 2024-08-03T20:54:01.733 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:01.732+0000 7f4f8c7c2740 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55a6b1d234d0 0x55a6b20e8be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:01.733 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:01.733+0000 7f4f8c7c2740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55a6b1d2b6b0 0x55a6b215ad40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:01.733 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:01.733+0000 7f4f8c7c2740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a6b1d4c390 0x55a6b211c900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:01.733 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:01.733+0000 7f4f8c7c2740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f4f80202780 con 0x55a6b1d4c390 2024-08-03T20:54:01.734 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:01.733+0000 7f4f8c7c2740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f4f80201760 con 0x55a6b1d2b6b0 2024-08-03T20:54:01.734 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:01.733+0000 7f4f8c7c2740 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7f4f80200b70 con 0x55a6b1d234d0 2024-08-03T20:54:01.734 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:01.733+0000 7f4f7f400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a6b1d4c390 0x55a6b211c900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:01.734 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:01.733+0000 7f4f7f400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a6b1d4c390 0x55a6b211c900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:57922/0 (socket says 172.21.15.32:57922) 2024-08-03T20:54:01.734 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:01.733+0000 7f4f7f400640 1 -- 172.21.15.32:0/2728837157 learned_addr learned my addr 172.21.15.32:0/2728837157 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:54:01.736 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:01.733+0000 7f4f77e00640 1 --2- 172.21.15.32:0/2728837157 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55a6b1d2b6b0 0x55a6b215ad40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:01.736 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:01.733+0000 7f4f7ea00640 1 --2- 172.21.15.32:0/2728837157 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55a6b1d234d0 0x55a6b20e8be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:01.736 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:01.733+0000 7f4f7f400640 1 -- 172.21.15.32:0/2728837157 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55a6b1d234d0 msgr2=0x55a6b20e8be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:54:01.736 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:01.733+0000 7f4f7f400640 1 --2- 172.21.15.32:0/2728837157 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55a6b1d234d0 0x55a6b20e8be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:01.736 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:01.733+0000 7f4f7f400640 1 -- 172.21.15.32:0/2728837157 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55a6b1d2b6b0 msgr2=0x55a6b215ad40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:54:01.737 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:01.733+0000 7f4f7f400640 1 --2- 172.21.15.32:0/2728837157 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55a6b1d2b6b0 0x55a6b215ad40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:01.737 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:01.733+0000 7f4f7f400640 1 -- 172.21.15.32:0/2728837157 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a6b20ec740 con 0x55a6b1d4c390 2024-08-03T20:54:01.737 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:01.733+0000 7f4f77e00640 1 --2- 172.21.15.32:0/2728837157 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55a6b1d2b6b0 0x55a6b215ad40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-03T20:54:01.737 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:01.733+0000 7f4f7ea00640 1 --2- 172.21.15.32:0/2728837157 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55a6b1d234d0 0x55a6b20e8be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-03T20:54:01.737 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:01.734+0000 7f4f7f400640 1 --2- 172.21.15.32:0/2728837157 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a6b1d4c390 0x55a6b211c900 secure :-1 s=READY pgs=831 cs=0 l=1 rev1=1 crypto rx=0x7f4f780b6a20 tx=0x7f4f78036040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9bc69c1da94ee601 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:54:01.737 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:01.734+0000 7f4f6ea00640 1 -- 172.21.15.32:0/2728837157 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f78045040 con 0x55a6b1d4c390 2024-08-03T20:54:01.737 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:01.734+0000 7f4f6ea00640 1 -- 172.21.15.32:0/2728837157 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f4f780810a0 con 0x55a6b1d4c390 2024-08-03T20:54:01.737 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:01.734+0000 7f4f6ea00640 1 -- 172.21.15.32:0/2728837157 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f78089e10 con 0x55a6b1d4c390 2024-08-03T20:54:01.737 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:01.734+0000 7f4f8c7c2740 1 -- 172.21.15.32:0/2728837157 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a6b1d4c390 msgr2=0x55a6b211c900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:01.737 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:01.734+0000 7f4f8c7c2740 1 --2- 172.21.15.32:0/2728837157 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a6b1d4c390 0x55a6b211c900 secure :-1 s=READY pgs=831 cs=0 l=1 rev1=1 crypto rx=0x7f4f780b6a20 tx=0x7f4f78036040 comp rx=0 tx=0).stop 2024-08-03T20:54:01.737 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:01.735+0000 7f4f8c7c2740 1 -- 172.21.15.32:0/2728837157 shutdown_connections 2024-08-03T20:54:01.737 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:01.735+0000 7f4f8c7c2740 1 --2- 172.21.15.32:0/2728837157 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55a6b1d2b6b0 0x55a6b215ad40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:01.737 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:01.735+0000 7f4f8c7c2740 1 --2- 172.21.15.32:0/2728837157 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a6b1d4c390 0x55a6b211c900 unknown :-1 s=CLOSED pgs=831 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:01.738 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:01.735+0000 7f4f8c7c2740 1 --2- 172.21.15.32:0/2728837157 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55a6b1d234d0 0x55a6b20e8be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:01.738 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:01.735+0000 7f4f8c7c2740 1 -- 172.21.15.32:0/2728837157 >> 172.21.15.32:0/2728837157 conn(0x55a6b1d404e0 msgr2=0x55a6b211bf20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:54:01.738 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:01.735+0000 7f4f8c7c2740 1 -- 172.21.15.32:0/2728837157 shutdown_connections 2024-08-03T20:54:01.738 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:01.735+0000 7f4f8c7c2740 1 -- 172.21.15.32:0/2728837157 wait complete. 2024-08-03T20:54:01.738 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:01.736+0000 7f4f8c7c2740 1 -- start start 2024-08-03T20:54:01.738 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:01.736+0000 7f4f8c7c2740 1 Processor -- start 2024-08-03T20:54:01.740 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:01.737+0000 7f4f8c7c2740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a6b1d234d0 0x55a6b21b8110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:01.740 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:01.737+0000 7f4f8c7c2740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55a6b1d2b6b0 0x55a6b2131340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:01.740 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:01.737+0000 7f4f8c7c2740 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55a6b1d4c390 0x55a6b21374f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:01.740 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:01.737+0000 7f4f8c7c2740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55a6b210ba80 con 0x55a6b1d234d0 2024-08-03T20:54:01.740 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:01.737+0000 7f4f8c7c2740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f4f80201110 con 0x55a6b1d2b6b0 2024-08-03T20:54:01.741 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:01.737+0000 7f4f8c7c2740 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x55a6b215b3c0 con 0x55a6b1d4c390 2024-08-03T20:54:01.741 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:01.737+0000 7f4f8c7c2740 10 client.0 _ll_register_callbacks cb 0x7f4f78003f50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-03T20:54:01.741 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:01.738+0000 7f4f7ea00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a6b1d234d0 0x55a6b21b8110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:01.741 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:01.738+0000 7f4f7ea00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a6b1d234d0 0x55a6b21b8110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:57928/0 (socket says 172.21.15.32:57928) 2024-08-03T20:54:01.741 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:01.738+0000 7f4f7ea00640 1 -- 172.21.15.32:0/1453224207 learned_addr learned my addr 172.21.15.32:0/1453224207 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:54:01.741 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:01.738+0000 7f4f77e00640 1 --2- 172.21.15.32:0/1453224207 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55a6b1d2b6b0 0x55a6b2131340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:01.741 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:01.738+0000 7f4f7f400640 1 --2- 172.21.15.32:0/1453224207 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55a6b1d4c390 0x55a6b21374f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:01.741 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:01.738+0000 7f4f7ea00640 1 -- 172.21.15.32:0/1453224207 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55a6b1d4c390 msgr2=0x55a6b21374f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:54:01.741 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:01.738+0000 7f4f7ea00640 1 --2- 172.21.15.32:0/1453224207 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55a6b1d4c390 0x55a6b21374f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:01.741 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:01.738+0000 7f4f7ea00640 1 -- 172.21.15.32:0/1453224207 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55a6b1d2b6b0 msgr2=0x55a6b2131340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:54:01.741 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:01.738+0000 7f4f7ea00640 1 --2- 172.21.15.32:0/1453224207 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55a6b1d2b6b0 0x55a6b2131340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:01.741 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:01.738+0000 7f4f7ea00640 1 -- 172.21.15.32:0/1453224207 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a6b21b8650 con 0x55a6b1d234d0 2024-08-03T20:54:01.741 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:01.738+0000 7f4f7ea00640 1 --2- 172.21.15.32:0/1453224207 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a6b1d234d0 0x55a6b21b8110 secure :-1 s=READY pgs=832 cs=0 l=1 rev1=1 crypto rx=0x55a6b2132100 tx=0x7f4f70001bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=834ad06ebc4b1da8 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:54:01.742 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:01.738+0000 7f4f63e00640 10 client.0 ms_handle_connect on v2:172.21.15.32:3300/0 2024-08-03T20:54:01.742 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:01.739+0000 7f4f63e00640 1 -- 172.21.15.32:0/1453224207 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f70012b10 con 0x55a6b1d234d0 2024-08-03T20:54:01.742 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:01.739+0000 7f4f63e00640 1 -- 172.21.15.32:0/1453224207 <== mon.0 v2:172.21.15.32:3300/0 2 ==== osd_map(71..71 src has 1..71) ==== 5795+0+0 (secure 0 0 0) 0x7f4f70012cb0 con 0x55a6b1d234d0 2024-08-03T20:54:01.742 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:01.739+0000 7f4f8c7c2740 1 -- 172.21.15.32:0/1453224207 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4f780810a0 con 0x55a6b1d234d0 2024-08-03T20:54:01.742 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:01.739+0000 7f4f63e00640 1 -- 172.21.15.32:0/1453224207 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_get_version_reply(handle=1 version=81) ==== 24+0+0 (secure 0 0 0) 0x7f4f70037050 con 0x55a6b1d234d0 2024-08-03T20:54:01.742 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:01.739+0000 7f4f8c7c2740 10 client.6672 fetch_fsmap learned FSMap version 81 2024-08-03T20:54:01.742 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:01.739+0000 7f4f8c7c2740 1 -- 172.21.15.32:0/1453224207 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,fsmap=81,monmap=2+}) -- 0x55a6b2131880 con 0x55a6b1d234d0 2024-08-03T20:54:01.742 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:01.739+0000 7f4f63e00640 1 -- 172.21.15.32:0/1453224207 <== mon.0 v2:172.21.15.32:3300/0 4 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f4f780810a0 con 0x55a6b1d234d0 2024-08-03T20:54:01.742 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:01.739+0000 7f4f63e00640 1 -- 172.21.15.32:0/1453224207 <== mon.0 v2:172.21.15.32:3300/0 5 ==== fsmap(e 81) ==== 3502+0+0 (secure 0 0 0) 0x7f4f7003a020 con 0x55a6b1d234d0 2024-08-03T20:54:01.742 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:01.739+0000 7f4f8c7c2740 10 client.6672 fetch_fsmap finished waiting for FSMap version 81 2024-08-03T20:54:01.742 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:01.739+0000 7f4f8c7c2740 10 client.6672 resolve_mds: resolved 5:0 to role '5:0' aka daemon mds.c 2024-08-03T20:54:01.742 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:01.739+0000 7f4f8c7c2740 20 client.6672 populate_metadata read hostname 'smithi032' 2024-08-03T20:54:01.742 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:01.739+0000 7f4f8c7c2740 1 --2- 172.21.15.32:0/1453224207 >> [v2:172.21.15.32:6832/1798758431,v1:172.21.15.32:6834/1798758431] conn(0x55a6b2137a30 0x55a6b21b44b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:01.743 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:01.739+0000 7f4f8c7c2740 4 client.6672 mds_command: new command op to 6396 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-08-03T20:54:01.743 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:01.739+0000 7f4f8c7c2740 1 -- 172.21.15.32:0/1453224207 --> [v2:172.21.15.32:6832/1798758431,v1:172.21.15.32:6834/1798758431] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55a6b2062e60 con 0x55a6b2137a30 2024-08-03T20:54:01.743 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:01.740+0000 7f4f77e00640 1 --2- 172.21.15.32:0/1453224207 >> [v2:172.21.15.32:6832/1798758431,v1:172.21.15.32:6834/1798758431] conn(0x55a6b2137a30 0x55a6b21b44b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:01.743 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:01.740+0000 7f4f63e00640 10 client.6672 ms_handle_connect on v2:172.21.15.32:6832/1798758431 2024-08-03T20:54:01.743 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:01.740+0000 7f4f77e00640 1 --2- 172.21.15.32:0/1453224207 >> [v2:172.21.15.32:6832/1798758431,v1:172.21.15.32:6834/1798758431] conn(0x55a6b2137a30 0x55a6b21b44b0 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bc86b3f4ed093264 server_cookie=cc591439a71ca738 in_seq=0 out_seq=0 2024-08-03T20:54:01.753 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:01.752+0000 7f4f63e00640 1 -- 172.21.15.32:0/1453224207 <== mds.0 v2:172.21.15.32:6832/1798758431 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55a6b2062e60 con 0x55a6b2137a30 2024-08-03T20:54:01.753 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:01.752+0000 7f4f63e00640 10 client.6672 handle_command_reply: tid=0 2024-08-03T20:54:01.753 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:01.752+0000 7f4f8c7c2740 1 client.6672 shutdown 2024-08-03T20:54:01.753 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:01.752+0000 7f4f8c7c2740 1 -- 172.21.15.32:0/1453224207 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a6b1d234d0 msgr2=0x55a6b21b8110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:01.753 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:01.752+0000 7f4f8c7c2740 1 --2- 172.21.15.32:0/1453224207 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a6b1d234d0 0x55a6b21b8110 secure :-1 s=READY pgs=832 cs=0 l=1 rev1=1 crypto rx=0x55a6b2132100 tx=0x7f4f70001bc0 comp rx=0 tx=0).stop 2024-08-03T20:54:01.753 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:01.752+0000 7f4f8c7c2740 1 -- 172.21.15.32:0/1453224207 shutdown_connections 2024-08-03T20:54:01.753 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:01.752+0000 7f4f8c7c2740 1 --2- 172.21.15.32:0/1453224207 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55a6b1d2b6b0 0x55a6b2131340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:01.753 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:01.752+0000 7f4f8c7c2740 1 --2- 172.21.15.32:0/1453224207 >> [v2:172.21.15.32:6832/1798758431,v1:172.21.15.32:6834/1798758431] conn(0x55a6b2137a30 0x55a6b21b44b0 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:01.753 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:01.752+0000 7f4f8c7c2740 1 --2- 172.21.15.32:0/1453224207 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55a6b1d4c390 0x55a6b21374f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:01.754 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:01.752+0000 7f4f8c7c2740 1 --2- 172.21.15.32:0/1453224207 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a6b1d234d0 0x55a6b21b8110 unknown :-1 s=CLOSED pgs=832 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:01.754 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:01.752+0000 7f4f63e00640 0 client.6672 ms_handle_reset on v2:172.21.15.32:6832/1798758431 2024-08-03T20:54:01.754 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:01.752+0000 7f4f8c7c2740 1 -- 172.21.15.32:0/1453224207 >> 172.21.15.32:0/1453224207 conn(0x55a6b1d404e0 msgr2=0x55a6b211cdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:54:01.754 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:01.752+0000 7f4f8c7c2740 1 -- 172.21.15.32:0/1453224207 shutdown_connections 2024-08-03T20:54:01.754 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:01.752+0000 7f4f8c7c2740 1 -- 172.21.15.32:0/1453224207 wait complete. 2024-08-03T20:54:01.754 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:01.753+0000 7f4f8c7c2740 20 client.6672 trim_cache size 0 max 16384 2024-08-03T20:54:01.754 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-08-03T20:54:01.754 INFO:teuthology.orchestra.run.smithi032.stdout: "message": "", 2024-08-03T20:54:01.754 INFO:teuthology.orchestra.run.smithi032.stdout: "return_code": 0 2024-08-03T20:54:01.754 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-08-03T20:54:01.755 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:01.753+0000 7f4f85800640 1 -- 172.21.15.32:0/3707637078 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f4f5c077010 msgr2=0x7f4f5c079520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:01.756 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:01.753+0000 7f4f85800640 1 --2- 172.21.15.32:0/3707637078 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f4f5c077010 0x7f4f5c079520 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f4f70001010 tx=0x7f4f70006970 comp rx=0 tx=0).stop 2024-08-03T20:54:01.756 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:01.753+0000 7f4f85800640 1 -- 172.21.15.32:0/3707637078 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4f80073940 msgr2=0x7f4f801f8820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:01.756 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:01.753+0000 7f4f85800640 1 --2- 172.21.15.32:0/3707637078 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4f80073940 0x7f4f801f8820 secure :-1 s=READY pgs=828 cs=0 l=1 rev1=1 crypto rx=0x7f4f78001eb0 tx=0x7f4f78001b10 comp rx=0 tx=0).stop 2024-08-03T20:54:01.756 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:01.753+0000 7f4f85800640 1 -- 172.21.15.32:0/3707637078 shutdown_connections 2024-08-03T20:54:01.756 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:01.753+0000 7f4f85800640 1 --2- 172.21.15.32:0/3707637078 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f4f5c077010 0x7f4f5c079520 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:01.756 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:01.753+0000 7f4f85800640 1 --2- 172.21.15.32:0/3707637078 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4f80073940 0x7f4f801f8820 unknown :-1 s=CLOSED pgs=828 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:01.756 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:01.753+0000 7f4f85800640 1 --2- 172.21.15.32:0/3707637078 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f4f8013ddc0 0x7f4f80073400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:01.756 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:01.753+0000 7f4f85800640 1 --2- 172.21.15.32:0/3707637078 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f4f64004e60 0x7f4f8013a770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:01.756 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:01.753+0000 7f4f85800640 1 -- 172.21.15.32:0/3707637078 >> 172.21.15.32:0/3707637078 conn(0x7f4f800b1bd0 msgr2=0x7f4f800cb970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:54:01.756 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:01.753+0000 7f4f85800640 1 -- 172.21.15.32:0/3707637078 shutdown_connections 2024-08-03T20:54:01.756 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:01.753+0000 7f4f85800640 1 -- 172.21.15.32:0/3707637078 wait complete. 2024-08-03T20:54:01.779 INFO:teuthology.nuke:Clearing teuthology firewall rules... 2024-08-03T20:54:01.779 DEBUG:teuthology.orchestra.run.smithi032:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-08-03T20:54:01.872 DEBUG:teuthology.orchestra.run.smithi118:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-08-03T20:54:01.974 INFO:teuthology.nuke:Cleared teuthology firewall rules. 2024-08-03T20:54:01.974 INFO:tasks.cephfs.mount:Terminating background process 2024-08-03T20:54:01.974 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-08-03T20:54:01.975 DEBUG:teuthology.orchestra.run.smithi032:> dd if=/proc/self/mounts of=/dev/stdout 2024-08-03T20:54:02.010 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-08-03T20:54:02.010 DEBUG:teuthology.orchestra.run.smithi032:> dd if=/proc/self/mounts of=/dev/stdout 2024-08-03T20:54:02.087 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-08-03T20:54:02.356 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:02.355+0000 7f488be00640 1 -- 172.21.15.32:0/844126905 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f488c0d11c0 msgr2=0x7f488c1692d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:02.356 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:02.355+0000 7f488be00640 1 --2- 172.21.15.32:0/844126905 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f488c0d11c0 0x7f488c1692d0 secure :-1 s=READY pgs=834 cs=0 l=1 rev1=1 crypto rx=0x7f4884005fd0 tx=0x7f488402a8b0 comp rx=0 tx=0).stop 2024-08-03T20:54:02.356 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:02.355+0000 7f488be00640 1 -- 172.21.15.32:0/844126905 shutdown_connections 2024-08-03T20:54:02.356 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:02.355+0000 7f488be00640 1 --2- 172.21.15.32:0/844126905 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f488c0d11c0 0x7f488c1692d0 unknown :-1 s=CLOSED pgs=834 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:02.356 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:02.355+0000 7f488be00640 1 -- 172.21.15.32:0/844126905 >> 172.21.15.32:0/844126905 conn(0x7f488c0d40c0 msgr2=0x7f488c0d44d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:54:02.356 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:02.355+0000 7f488be00640 1 -- 172.21.15.32:0/844126905 shutdown_connections 2024-08-03T20:54:02.356 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:02.355+0000 7f488be00640 1 -- 172.21.15.32:0/844126905 wait complete. 2024-08-03T20:54:02.358 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:02.357+0000 7f488be00640 1 Processor -- start 2024-08-03T20:54:02.358 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:02.357+0000 7f488be00640 1 -- start start 2024-08-03T20:54:02.360 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:02.359+0000 7f488be00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f488c0b94f0 0x7f488c159940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:02.360 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:02.359+0000 7f488be00640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f488c0d4be0 0x7f488c0a1540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:02.360 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:02.359+0000 7f488be00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f488c15d3b0 0x7f488c15d7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:02.360 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:02.359+0000 7f488be00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f488c169990 con 0x7f488c0b94f0 2024-08-03T20:54:02.361 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:02.359+0000 7f488be00640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f488c169810 con 0x7f488c15d3b0 2024-08-03T20:54:02.361 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:02.359+0000 7f488be00640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7f488c169b10 con 0x7f488c0d4be0 2024-08-03T20:54:02.361 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:02.359+0000 7f488aa00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f488c0b94f0 0x7f488c159940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:02.361 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:02.359+0000 7f488aa00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f488c0b94f0 0x7f488c159940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:57958/0 (socket says 172.21.15.32:57958) 2024-08-03T20:54:02.361 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:02.359+0000 7f488aa00640 1 -- 172.21.15.32:0/484414994 learned_addr learned my addr 172.21.15.32:0/484414994 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:54:02.361 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:02.359+0000 7f488b400640 1 --2- 172.21.15.32:0/484414994 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f488c15d3b0 0x7f488c15d7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:02.361 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:02.360+0000 7f4883e00640 1 --2- 172.21.15.32:0/484414994 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f488c0d4be0 0x7f488c0a1540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:02.361 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:02.360+0000 7f488aa00640 1 -- 172.21.15.32:0/484414994 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f488c0d4be0 msgr2=0x7f488c0a1540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:02.361 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:02.360+0000 7f488aa00640 1 --2- 172.21.15.32:0/484414994 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f488c0d4be0 0x7f488c0a1540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:02.362 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:02.360+0000 7f488aa00640 1 -- 172.21.15.32:0/484414994 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f488c15d3b0 msgr2=0x7f488c15d7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:02.362 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:02.360+0000 7f488aa00640 1 --2- 172.21.15.32:0/484414994 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f488c15d3b0 0x7f488c15d7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:02.362 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:02.360+0000 7f488aa00640 1 -- 172.21.15.32:0/484414994 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f488c15dd00 con 0x7f488c0b94f0 2024-08-03T20:54:02.362 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:02.360+0000 7f4883e00640 1 --2- 172.21.15.32:0/484414994 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f488c0d4be0 0x7f488c0a1540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-03T20:54:02.362 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:02.360+0000 7f488aa00640 1 --2- 172.21.15.32:0/484414994 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f488c0b94f0 0x7f488c159940 secure :-1 s=READY pgs=835 cs=0 l=1 rev1=1 crypto rx=0x7f487c00c410 tx=0x7f487c00c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:54:02.363 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:02.360+0000 7f4882a00640 1 -- 172.21.15.32:0/484414994 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f487c017070 con 0x7f488c0b94f0 2024-08-03T20:54:02.363 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:02.360+0000 7f4882a00640 1 -- 172.21.15.32:0/484414994 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f487c0035f0 con 0x7f488c0b94f0 2024-08-03T20:54:02.363 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:02.360+0000 7f488be00640 1 -- 172.21.15.32:0/484414994 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f488c09de70 con 0x7f488c0b94f0 2024-08-03T20:54:02.363 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:02.360+0000 7f4882a00640 1 -- 172.21.15.32:0/484414994 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f487c003a80 con 0x7f488c0b94f0 2024-08-03T20:54:02.363 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:02.360+0000 7f488be00640 1 -- 172.21.15.32:0/484414994 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f488c09e430 con 0x7f488c0b94f0 2024-08-03T20:54:02.364 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:02.361+0000 7f488be00640 1 -- 172.21.15.32:0/484414994 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f488c0b5460 con 0x7f488c0b94f0 2024-08-03T20:54:02.364 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:02.363+0000 7f4882a00640 1 -- 172.21.15.32:0/484414994 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f487c01a070 con 0x7f488c0b94f0 2024-08-03T20:54:02.364 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:02.364+0000 7f4882a00640 1 --2- 172.21.15.32:0/484414994 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f4868077040 0x7f4868079550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:02.364 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:02.364+0000 7f4882a00640 1 -- 172.21.15.32:0/484414994 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(71..71 src has 1..71) ==== 5795+0+0 (secure 0 0 0) 0x7f487c097d10 con 0x7f488c0b94f0 2024-08-03T20:54:02.364 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:02.364+0000 7f4883e00640 1 --2- 172.21.15.32:0/484414994 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f4868077040 0x7f4868079550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:02.365 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:02.365+0000 7f4883e00640 1 --2- 172.21.15.32:0/484414994 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f4868077040 0x7f4868079550 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f488c168ea0 tx=0x7f4874009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:54:02.368 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:02.368+0000 7f4882a00640 1 -- 172.21.15.32:0/484414994 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7f487c09d020 con 0x7f488c0b94f0 2024-08-03T20:54:02.533 INFO:tasks.ceph.mds.c.smithi032.stderr:2024-08-03T20:54:02.532+0000 7faa8fc00640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-08-03T20:54:02.590 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:02.589+0000 7f488be00640 1 -- 172.21.15.32:0/484414994 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f488c15eb00 con 0x7f488c0b94f0 2024-08-03T20:54:02.590 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:02.590+0000 7f4882a00640 1 -- 172.21.15.32:0/484414994 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 2 entries v71) ==== 80+0+122 (secure 0 0 0) 0x7f488c15dd00 con 0x7f488c0b94f0 2024-08-03T20:54:02.591 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6835/4240363667 2024-08-04T20:53:56.018944+0000 2024-08-03T20:54:02.591 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6833/4240363667 2024-08-04T20:53:56.018944+0000 2024-08-03T20:54:02.591 INFO:teuthology.orchestra.run.smithi032.stderr:listed 2 entries 2024-08-03T20:54:02.593 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:02.592+0000 7f488be00640 1 -- 172.21.15.32:0/484414994 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f4868077040 msgr2=0x7f4868079550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:02.594 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:02.592+0000 7f488be00640 1 --2- 172.21.15.32:0/484414994 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f4868077040 0x7f4868079550 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f488c168ea0 tx=0x7f4874009040 comp rx=0 tx=0).stop 2024-08-03T20:54:02.594 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:02.592+0000 7f488be00640 1 -- 172.21.15.32:0/484414994 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f488c0b94f0 msgr2=0x7f488c159940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:02.594 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:02.592+0000 7f488be00640 1 --2- 172.21.15.32:0/484414994 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f488c0b94f0 0x7f488c159940 secure :-1 s=READY pgs=835 cs=0 l=1 rev1=1 crypto rx=0x7f487c00c410 tx=0x7f487c00c8d0 comp rx=0 tx=0).stop 2024-08-03T20:54:02.594 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:02.592+0000 7f488be00640 1 -- 172.21.15.32:0/484414994 shutdown_connections 2024-08-03T20:54:02.594 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:02.592+0000 7f488be00640 1 --2- 172.21.15.32:0/484414994 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f4868077040 0x7f4868079550 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:02.594 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:02.592+0000 7f488be00640 1 --2- 172.21.15.32:0/484414994 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f488c15d3b0 0x7f488c15d7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:02.594 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:02.592+0000 7f488be00640 1 --2- 172.21.15.32:0/484414994 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f488c0d4be0 0x7f488c0a1540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:02.594 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:02.592+0000 7f488be00640 1 --2- 172.21.15.32:0/484414994 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f488c0b94f0 0x7f488c159940 unknown :-1 s=CLOSED pgs=835 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:02.594 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:02.592+0000 7f488be00640 1 -- 172.21.15.32:0/484414994 >> 172.21.15.32:0/484414994 conn(0x7f488c0d40c0 msgr2=0x7f488c0d44d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:54:02.594 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:02.593+0000 7f488be00640 1 -- 172.21.15.32:0/484414994 shutdown_connections 2024-08-03T20:54:02.594 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:02.593+0000 7f488be00640 1 -- 172.21.15.32:0/484414994 wait complete. 2024-08-03T20:54:02.616 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-08-03T20:54:02.616 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-08-03T20:54:02.616 DEBUG:teuthology.orchestra.run.smithi032:> sudo umount /home/ubuntu/cephtest/mnt.0 2024-08-03T20:54:02.678 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi032.front.sepia.ceph.com 2024-08-03T20:54:02.678 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-08-03T20:54:02.678 DEBUG:teuthology.orchestra.run.smithi032:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-08-03T20:54:02.714 INFO:tasks.cephfs.mount:Terminating background process 2024-08-03T20:54:02.714 DEBUG:teuthology.orchestra.run.smithi118:> set -ex 2024-08-03T20:54:02.714 DEBUG:teuthology.orchestra.run.smithi118:> dd if=/proc/self/mounts of=/dev/stdout 2024-08-03T20:54:02.754 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052663,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497944k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248972k,nr_inodes=812243,mode=700,uid=1000,gid=1316 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497944k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-08-03T20:54:02.754 INFO:tasks.cephfs.mount:Terminating background process 2024-08-03T20:54:02.755 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-08-03T20:54:02.755 DEBUG:teuthology.orchestra.run.smithi032:> dd if=/proc/self/mounts of=/dev/stdout 2024-08-03T20:54:02.791 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053453,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6499208k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 none /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249604k,nr_inodes=812401,mode=700,uid=1000,gid=1316 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6499208k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-08-03T20:54:02.791 INFO:tasks.cephfs.mount:Terminating background process 2024-08-03T20:54:02.791 DEBUG:teuthology.orchestra.run.smithi118:> set -ex 2024-08-03T20:54:02.791 DEBUG:teuthology.orchestra.run.smithi118:> dd if=/proc/self/mounts of=/dev/stdout 2024-08-03T20:54:02.834 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052663,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497944k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248972k,nr_inodes=812243,mode=700,uid=1000,gid=1316 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497944k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-08-03T20:54:02.834 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-08-03T20:54:03.065 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:03.064+0000 7f0c55400640 1 -- 172.21.15.32:0/3272433325 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0c500d4bb0 msgr2=0x7f0c500d1c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:03.065 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:03.064+0000 7f0c55400640 1 --2- 172.21.15.32:0/3272433325 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0c500d4bb0 0x7f0c500d1c60 secure :-1 s=READY pgs=837 cs=0 l=1 rev1=1 crypto rx=0x7f0c40009830 tx=0x7f0c4002a480 comp rx=0 tx=0).stop 2024-08-03T20:54:03.065 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:03.064+0000 7f0c55400640 1 -- 172.21.15.32:0/3272433325 shutdown_connections 2024-08-03T20:54:03.065 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:03.064+0000 7f0c55400640 1 --2- 172.21.15.32:0/3272433325 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f0c500cec30 0x7f0c500cf040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:03.066 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:03.064+0000 7f0c55400640 1 --2- 172.21.15.32:0/3272433325 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0c500d4bb0 0x7f0c500d1c60 unknown :-1 s=CLOSED pgs=837 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:03.066 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:03.064+0000 7f0c55400640 1 -- 172.21.15.32:0/3272433325 >> 172.21.15.32:0/3272433325 conn(0x7f0c500d4090 msgr2=0x7f0c500d44a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:54:03.066 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:03.064+0000 7f0c55400640 1 -- 172.21.15.32:0/3272433325 shutdown_connections 2024-08-03T20:54:03.066 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:03.064+0000 7f0c55400640 1 -- 172.21.15.32:0/3272433325 wait complete. 2024-08-03T20:54:03.067 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:03.067+0000 7f0c55400640 1 Processor -- start 2024-08-03T20:54:03.067 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:03.067+0000 7f0c55400640 1 -- start start 2024-08-03T20:54:03.069 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:03.068+0000 7f0c55400640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f0c500cec30 0x7f0c5015c6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:03.069 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:03.068+0000 7f0c55400640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f0c500d2290 0x7f0c50163e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:03.070 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:03.068+0000 7f0c55400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0c500d4bb0 0x7f0c50164340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:03.070 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:03.068+0000 7f0c55400640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f0c500cfad0 con 0x7f0c500d4bb0 2024-08-03T20:54:03.070 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:03.068+0000 7f0c55400640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f0c500cf950 con 0x7f0c500d2290 2024-08-03T20:54:03.070 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:03.068+0000 7f0c55400640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7f0c500cfc50 con 0x7f0c500cec30 2024-08-03T20:54:03.070 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:03.069+0000 7f0c54a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0c500d4bb0 0x7f0c50164340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:03.070 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:03.069+0000 7f0c54a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0c500d4bb0 0x7f0c50164340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:57982/0 (socket says 172.21.15.32:57982) 2024-08-03T20:54:03.070 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:03.069+0000 7f0c54a00640 1 -- 172.21.15.32:0/2543180615 learned_addr learned my addr 172.21.15.32:0/2543180615 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:54:03.070 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:03.069+0000 7f0c4fe00640 1 --2- 172.21.15.32:0/2543180615 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f0c500cec30 0x7f0c5015c6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:03.070 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:03.069+0000 7f0c47e00640 1 --2- 172.21.15.32:0/2543180615 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f0c500d2290 0x7f0c50163e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:03.070 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:03.069+0000 7f0c54a00640 1 -- 172.21.15.32:0/2543180615 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f0c500cec30 msgr2=0x7f0c5015c6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:03.070 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:03.069+0000 7f0c54a00640 1 --2- 172.21.15.32:0/2543180615 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f0c500cec30 0x7f0c5015c6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:03.070 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:03.069+0000 7f0c54a00640 1 -- 172.21.15.32:0/2543180615 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f0c500d2290 msgr2=0x7f0c50163e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:03.071 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:03.069+0000 7f0c54a00640 1 --2- 172.21.15.32:0/2543180615 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f0c500d2290 0x7f0c50163e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:03.071 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:03.069+0000 7f0c54a00640 1 -- 172.21.15.32:0/2543180615 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0c500cb460 con 0x7f0c500d4bb0 2024-08-03T20:54:03.071 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:03.069+0000 7f0c54a00640 1 --2- 172.21.15.32:0/2543180615 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0c500d4bb0 0x7f0c50164340 secure :-1 s=READY pgs=838 cs=0 l=1 rev1=1 crypto rx=0x7f0c4800cb00 tx=0x7f0c4800a250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:54:03.071 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:03.070+0000 7f0c4cc00640 1 -- 172.21.15.32:0/2543180615 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c4800f450 con 0x7f0c500d4bb0 2024-08-03T20:54:03.072 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:03.070+0000 7f0c4cc00640 1 -- 172.21.15.32:0/2543180615 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f0c4800f700 con 0x7f0c500d4bb0 2024-08-03T20:54:03.072 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:03.070+0000 7f0c55400640 1 -- 172.21.15.32:0/2543180615 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0c500cd9f0 con 0x7f0c500d4bb0 2024-08-03T20:54:03.072 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:03.070+0000 7f0c4cc00640 1 -- 172.21.15.32:0/2543180615 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c4800fb50 con 0x7f0c500d4bb0 2024-08-03T20:54:03.072 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:03.070+0000 7f0c55400640 1 -- 172.21.15.32:0/2543180615 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0c500cdfb0 con 0x7f0c500d4bb0 2024-08-03T20:54:03.072 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:03.071+0000 7f0c55400640 1 -- 172.21.15.32:0/2543180615 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0c500cb1c0 con 0x7f0c500d4bb0 2024-08-03T20:54:03.072 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:03.072+0000 7f0c4cc00640 1 -- 172.21.15.32:0/2543180615 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f0c4800fcf0 con 0x7f0c500d4bb0 2024-08-03T20:54:03.073 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:03.073+0000 7f0c4cc00640 1 --2- 172.21.15.32:0/2543180615 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f0c2c077040 0x7f0c2c079550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:03.073 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:03.073+0000 7f0c4cc00640 1 -- 172.21.15.32:0/2543180615 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(71..71 src has 1..71) ==== 5795+0+0 (secure 0 0 0) 0x7f0c48099490 con 0x7f0c500d4bb0 2024-08-03T20:54:03.074 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:03.074+0000 7f0c4fe00640 1 --2- 172.21.15.32:0/2543180615 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f0c2c077040 0x7f0c2c079550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:03.075 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:03.074+0000 7f0c4fe00640 1 --2- 172.21.15.32:0/2543180615 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f0c2c077040 0x7f0c2c079550 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f0c40034000 tx=0x7f0c40004730 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:54:03.078 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:03.077+0000 7f0c4cc00640 1 -- 172.21.15.32:0/2543180615 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7f0c500cb1c0 con 0x7f0c500d4bb0 2024-08-03T20:54:03.319 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:03.319+0000 7f0c55400640 1 -- 172.21.15.32:0/2543180615 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f0c500cf040 con 0x7f0c500d4bb0 2024-08-03T20:54:03.320 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:03.320+0000 7f0c4cc00640 1 -- 172.21.15.32:0/2543180615 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 81 v81) ==== 94+0+5233 (secure 0 0 0) 0x7f0c48014030 con 0x7f0c500d4bb0 2024-08-03T20:54:03.320 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-03T20:54:03.321 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":81,"btime":"2024-08-03T20:53:59:036145+0000","default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5666,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6837/2261538396","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6835","nonce":2261538396},{"type":"v1","addr":"172.21.15.118:6837","nonce":2261538396}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":69},{"gid":6061,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6836/1133580656","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6834","nonce":1133580656},{"type":"v1","addr":"172.21.15.118:6836","nonce":1133580656}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":68},{"gid":6618,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6835/3466281885","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6833","nonce":3466281885},{"type":"v1","addr":"172.21.15.32:6835","nonce":3466281885}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":79}],"filesystems":[{"mdsmap":{"epoch":81,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-08-03T20:53:15.080669+0000","modified":"2024-08-03T20:53:59.036137+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":71,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6396},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6396":{"gid":6396,"name":"c","rank":0,"incarnation":78,"state":"up:active","state_seq":15,"addr":"172.21.15.32:6834/1798758431","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":1798758431},{"type":"v1","addr":"172.21.15.32:6834","nonce":1798758431}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6396,"qdb_cluster":[6396]},"id":5}]} 2024-08-03T20:54:03.321 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 81 2024-08-03T20:54:03.323 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:03.322+0000 7f0c55400640 1 -- 172.21.15.32:0/2543180615 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f0c2c077040 msgr2=0x7f0c2c079550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:03.323 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:03.322+0000 7f0c55400640 1 --2- 172.21.15.32:0/2543180615 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f0c2c077040 0x7f0c2c079550 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f0c40034000 tx=0x7f0c40004730 comp rx=0 tx=0).stop 2024-08-03T20:54:03.323 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:03.322+0000 7f0c55400640 1 -- 172.21.15.32:0/2543180615 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0c500d4bb0 msgr2=0x7f0c50164340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:03.323 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:03.322+0000 7f0c55400640 1 --2- 172.21.15.32:0/2543180615 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0c500d4bb0 0x7f0c50164340 secure :-1 s=READY pgs=838 cs=0 l=1 rev1=1 crypto rx=0x7f0c4800cb00 tx=0x7f0c4800a250 comp rx=0 tx=0).stop 2024-08-03T20:54:03.323 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:03.322+0000 7f0c55400640 1 -- 172.21.15.32:0/2543180615 shutdown_connections 2024-08-03T20:54:03.323 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:03.322+0000 7f0c55400640 1 --2- 172.21.15.32:0/2543180615 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f0c2c077040 0x7f0c2c079550 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:03.323 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:03.322+0000 7f0c55400640 1 --2- 172.21.15.32:0/2543180615 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f0c500d2290 0x7f0c50163e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:03.323 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:03.322+0000 7f0c55400640 1 --2- 172.21.15.32:0/2543180615 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0c500d4bb0 0x7f0c50164340 unknown :-1 s=CLOSED pgs=838 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:03.323 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:03.322+0000 7f0c55400640 1 --2- 172.21.15.32:0/2543180615 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f0c500cec30 0x7f0c5015c6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:03.323 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:03.322+0000 7f0c55400640 1 -- 172.21.15.32:0/2543180615 >> 172.21.15.32:0/2543180615 conn(0x7f0c500d4090 msgr2=0x7f0c500b4e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:54:03.324 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:03.322+0000 7f0c55400640 1 -- 172.21.15.32:0/2543180615 shutdown_connections 2024-08-03T20:54:03.324 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:03.322+0000 7f0c55400640 1 -- 172.21.15.32:0/2543180615 wait complete. 2024-08-03T20:54:03.344 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-08-03T20:54:03.573 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:03.572+0000 7f320a800640 1 -- 172.21.15.32:0/2047522137 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f32040b93d0 msgr2=0x7f32040cfa00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:03.573 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:03.572+0000 7f320a800640 1 --2- 172.21.15.32:0/2047522137 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f32040b93d0 0x7f32040cfa00 secure :-1 s=READY pgs=839 cs=0 l=1 rev1=1 crypto rx=0x7f31f80098e0 tx=0x7f31f802a530 comp rx=0 tx=0).stop 2024-08-03T20:54:03.573 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:03.572+0000 7f320a800640 1 -- 172.21.15.32:0/2047522137 shutdown_connections 2024-08-03T20:54:03.573 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:03.572+0000 7f320a800640 1 --2- 172.21.15.32:0/2047522137 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f32040d1cf0 0x7f32040d2100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:03.573 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:03.572+0000 7f320a800640 1 --2- 172.21.15.32:0/2047522137 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f32040b93d0 0x7f32040cfa00 unknown :-1 s=CLOSED pgs=839 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:03.573 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:03.572+0000 7f320a800640 1 -- 172.21.15.32:0/2047522137 >> 172.21.15.32:0/2047522137 conn(0x7f32040d4120 msgr2=0x7f32040d4530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:54:03.573 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:03.572+0000 7f320a800640 1 -- 172.21.15.32:0/2047522137 shutdown_connections 2024-08-03T20:54:03.574 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:03.572+0000 7f320a800640 1 -- 172.21.15.32:0/2047522137 wait complete. 2024-08-03T20:54:03.575 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:03.574+0000 7f320a800640 1 Processor -- start 2024-08-03T20:54:03.575 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:03.575+0000 7f320a800640 1 -- start start 2024-08-03T20:54:03.577 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:03.576+0000 7f320a800640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f32040b93d0 0x7f32041f94c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:03.577 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:03.576+0000 7f320a800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f32040d0af0 0x7f320409ee70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:03.577 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:03.576+0000 7f320a800640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f32040d1cf0 0x7f320414be30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:03.578 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:03.576+0000 7f320a800640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f32040b4cb0 con 0x7f32040d0af0 2024-08-03T20:54:03.578 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:03.576+0000 7f320a800640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f32040d2780 con 0x7f32040d1cf0 2024-08-03T20:54:03.578 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:03.576+0000 7f320a800640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7f32040b4e30 con 0x7f32040b93d0 2024-08-03T20:54:03.578 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:03.576+0000 7f3209400640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f32040b93d0 0x7f32041f94c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:03.578 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:03.576+0000 7f3208a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f32040d0af0 0x7f320409ee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:03.578 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:03.576+0000 7f3209400640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f32040b93d0 0x7f32041f94c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3301/0 says I am v2:172.21.15.32:37728/0 (socket says 172.21.15.32:37728) 2024-08-03T20:54:03.578 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:03.576+0000 7f3209400640 1 -- 172.21.15.32:0/3736859299 learned_addr learned my addr 172.21.15.32:0/3736859299 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:54:03.578 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:03.577+0000 7f3209e00640 1 --2- 172.21.15.32:0/3736859299 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f32040d1cf0 0x7f320414be30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:03.578 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:03.577+0000 7f3209400640 1 -- 172.21.15.32:0/3736859299 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f32040d1cf0 msgr2=0x7f320414be30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:03.578 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:03.577+0000 7f3209400640 1 --2- 172.21.15.32:0/3736859299 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f32040d1cf0 0x7f320414be30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:03.578 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:03.577+0000 7f3209400640 1 -- 172.21.15.32:0/3736859299 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f32040d0af0 msgr2=0x7f320409ee70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:03.578 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:03.577+0000 7f3209400640 1 --2- 172.21.15.32:0/3736859299 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f32040d0af0 0x7f320409ee70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:03.579 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:03.577+0000 7f3209400640 1 -- 172.21.15.32:0/3736859299 --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f320414c370 con 0x7f32040b93d0 2024-08-03T20:54:03.579 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:03.577+0000 7f3208a00640 1 --2- 172.21.15.32:0/3736859299 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f32040d0af0 0x7f320409ee70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-03T20:54:03.579 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:03.578+0000 7f3209400640 1 --2- 172.21.15.32:0/3736859299 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f32040b93d0 0x7f32041f94c0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f31f802ab20 tx=0x7f31f80052f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:54:03.579 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:03.578+0000 7f3201600640 1 -- 172.21.15.32:0/3736859299 <== mon.2 v2:172.21.15.118:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31f803f070 con 0x7f32040b93d0 2024-08-03T20:54:03.580 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:03.578+0000 7f3201600640 1 -- 172.21.15.32:0/3736859299 <== mon.2 v2:172.21.15.118:3301/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f31f8004230 con 0x7f32040b93d0 2024-08-03T20:54:03.580 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:03.578+0000 7f320a800640 1 -- 172.21.15.32:0/3736859299 --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f32041f5330 con 0x7f32040b93d0 2024-08-03T20:54:03.580 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:03.578+0000 7f320a800640 1 -- 172.21.15.32:0/3736859299 --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f32041f58c0 con 0x7f32040b93d0 2024-08-03T20:54:03.580 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:03.578+0000 7f3201600640 1 -- 172.21.15.32:0/3736859299 <== mon.2 v2:172.21.15.118:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31f8005cf0 con 0x7f32040b93d0 2024-08-03T20:54:03.580 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:03.579+0000 7f320a800640 1 -- 172.21.15.32:0/3736859299 --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f32040cfa00 con 0x7f32040b93d0 2024-08-03T20:54:03.582 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:03.581+0000 7f3201600640 1 -- 172.21.15.32:0/3736859299 <== mon.2 v2:172.21.15.118:3301/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f31f8042070 con 0x7f32040b93d0 2024-08-03T20:54:03.582 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:03.582+0000 7f3201600640 1 --2- 172.21.15.32:0/3736859299 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f31e4077010 0x7f31e4079520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:03.583 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:03.582+0000 7f3201600640 1 -- 172.21.15.32:0/3736859299 <== mon.2 v2:172.21.15.118:3301/0 5 ==== osd_map(71..71 src has 1..71) ==== 5795+0+0 (secure 0 0 0) 0x7f31f80b9af0 con 0x7f32040b93d0 2024-08-03T20:54:03.583 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:03.583+0000 7f3208a00640 1 --2- 172.21.15.32:0/3736859299 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f31e4077010 0x7f31e4079520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:03.584 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:03.584+0000 7f3208a00640 1 --2- 172.21.15.32:0/3736859299 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f31e4077010 0x7f31e4079520 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f3204168a50 tx=0x7f31f0009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:54:03.586 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:03.586+0000 7f3201600640 1 -- 172.21.15.32:0/3736859299 <== mon.2 v2:172.21.15.118:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7f31f8082730 con 0x7f32040b93d0 2024-08-03T20:54:03.830 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:03.830+0000 7f320a800640 1 -- 172.21.15.32:0/3736859299 --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f32040d1240 con 0x7f32040b93d0 2024-08-03T20:54:03.831 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:03.831+0000 7f3201600640 1 -- 172.21.15.32:0/3736859299 <== mon.2 v2:172.21.15.118:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 81 v81) ==== 94+0+5233 (secure 0 0 0) 0x7f31f8086c80 con 0x7f32040b93d0 2024-08-03T20:54:03.832 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-03T20:54:03.832 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":81,"btime":"2024-08-03T20:53:59:036145+0000","default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5666,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6837/2261538396","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6835","nonce":2261538396},{"type":"v1","addr":"172.21.15.118:6837","nonce":2261538396}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":69},{"gid":6061,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6836/1133580656","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6834","nonce":1133580656},{"type":"v1","addr":"172.21.15.118:6836","nonce":1133580656}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":68},{"gid":6618,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6835/3466281885","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6833","nonce":3466281885},{"type":"v1","addr":"172.21.15.32:6835","nonce":3466281885}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":79}],"filesystems":[{"mdsmap":{"epoch":81,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-08-03T20:53:15.080669+0000","modified":"2024-08-03T20:53:59.036137+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":71,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6396},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6396":{"gid":6396,"name":"c","rank":0,"incarnation":78,"state":"up:active","state_seq":15,"addr":"172.21.15.32:6834/1798758431","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":1798758431},{"type":"v1","addr":"172.21.15.32:6834","nonce":1798758431}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6396,"qdb_cluster":[6396]},"id":5}]} 2024-08-03T20:54:03.832 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 81 2024-08-03T20:54:03.834 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:03.833+0000 7f320a800640 1 -- 172.21.15.32:0/3736859299 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f31e4077010 msgr2=0x7f31e4079520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:03.834 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:03.833+0000 7f320a800640 1 --2- 172.21.15.32:0/3736859299 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f31e4077010 0x7f31e4079520 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f3204168a50 tx=0x7f31f0009040 comp rx=0 tx=0).stop 2024-08-03T20:54:03.835 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:03.833+0000 7f320a800640 1 -- 172.21.15.32:0/3736859299 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f32040b93d0 msgr2=0x7f32041f94c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:03.835 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:03.833+0000 7f320a800640 1 --2- 172.21.15.32:0/3736859299 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f32040b93d0 0x7f32041f94c0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f31f802ab20 tx=0x7f31f80052f0 comp rx=0 tx=0).stop 2024-08-03T20:54:03.835 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:03.834+0000 7f320a800640 1 -- 172.21.15.32:0/3736859299 shutdown_connections 2024-08-03T20:54:03.835 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:03.834+0000 7f320a800640 1 --2- 172.21.15.32:0/3736859299 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f31e4077010 0x7f31e4079520 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:03.835 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:03.834+0000 7f320a800640 1 --2- 172.21.15.32:0/3736859299 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f32040d1cf0 0x7f320414be30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:03.835 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:03.834+0000 7f320a800640 1 --2- 172.21.15.32:0/3736859299 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f32040d0af0 0x7f320409ee70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:03.835 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:03.834+0000 7f320a800640 1 --2- 172.21.15.32:0/3736859299 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f32040b93d0 0x7f32041f94c0 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:03.835 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:03.834+0000 7f320a800640 1 -- 172.21.15.32:0/3736859299 >> 172.21.15.32:0/3736859299 conn(0x7f32040d4120 msgr2=0x7f3204068b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:54:03.836 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:03.834+0000 7f320a800640 1 -- 172.21.15.32:0/3736859299 shutdown_connections 2024-08-03T20:54:03.836 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:03.834+0000 7f320a800640 1 -- 172.21.15.32:0/3736859299 wait complete. 2024-08-03T20:54:03.860 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-08-03T20:54:04.090 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:04.089+0000 7f1f77e00640 1 -- 172.21.15.32:0/1125191045 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1f5c004800 msgr2=0x7f1f5c004c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:04.090 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:04.089+0000 7f1f77e00640 1 --2- 172.21.15.32:0/1125191045 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1f5c004800 0x7f1f5c004c10 secure :-1 s=READY pgs=841 cs=0 l=1 rev1=1 crypto rx=0x7f1f780703f0 tx=0x7f1f5c005830 comp rx=0 tx=0).stop 2024-08-03T20:54:04.090 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:04.089+0000 7f1f77400640 1 -- 172.21.15.32:0/1125191045 reap_dead start 2024-08-03T20:54:04.090 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:04.089+0000 7f1f77e00640 1 -- 172.21.15.32:0/1125191045 shutdown_connections 2024-08-03T20:54:04.090 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:04.089+0000 7f1f77e00640 1 -- 172.21.15.32:0/1125191045 >> 172.21.15.32:0/1125191045 conn(0x7f1f780b2120 msgr2=0x7f1f780cfd60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:54:04.091 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:04.089+0000 7f1f77e00640 1 -- 172.21.15.32:0/1125191045 shutdown_connections 2024-08-03T20:54:04.091 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:04.089+0000 7f1f77e00640 1 -- 172.21.15.32:0/1125191045 wait complete. 2024-08-03T20:54:04.092 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:04.091+0000 7f1f77e00640 1 Processor -- start 2024-08-03T20:54:04.092 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:04.092+0000 7f1f77e00640 1 -- start start 2024-08-03T20:54:04.095 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:04.093+0000 7f1f77e00640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f1f5c004800 0x7f1f781f5910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:04.095 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:04.093+0000 7f1f77e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1f781f47a0 0x7f1f780cd930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:04.095 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:04.093+0000 7f1f77e00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f1f780b4870 0x7f1f780b4c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:04.095 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:04.093+0000 7f1f77e00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f1f781600e0 con 0x7f1f781f47a0 2024-08-03T20:54:04.095 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:04.093+0000 7f1f77e00640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f1f780ad4b0 con 0x7f1f780b4870 2024-08-03T20:54:04.095 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:04.093+0000 7f1f77e00640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7f1f78160260 con 0x7f1f5c004800 2024-08-03T20:54:04.095 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:04.093+0000 7f1f6e000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1f781f47a0 0x7f1f780cd930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:04.095 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:04.093+0000 7f1f6e000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1f781f47a0 0x7f1f780cd930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:58014/0 (socket says 172.21.15.32:58014) 2024-08-03T20:54:04.096 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:04.093+0000 7f1f6e000640 1 -- 172.21.15.32:0/666073236 learned_addr learned my addr 172.21.15.32:0/666073236 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:54:04.096 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:04.094+0000 7f1f77400640 1 --2- 172.21.15.32:0/666073236 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f1f780b4870 0x7f1f780b4c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:04.096 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:04.094+0000 7f1f76a00640 1 --2- 172.21.15.32:0/666073236 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f1f5c004800 0x7f1f781f5910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:04.096 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:04.094+0000 7f1f6e000640 1 -- 172.21.15.32:0/666073236 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f1f5c004800 msgr2=0x7f1f781f5910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:04.096 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:04.094+0000 7f1f6e000640 1 --2- 172.21.15.32:0/666073236 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f1f5c004800 0x7f1f781f5910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:04.096 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:04.094+0000 7f1f6e000640 1 -- 172.21.15.32:0/666073236 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f1f780b4870 msgr2=0x7f1f780b4c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:04.096 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:04.094+0000 7f1f6e000640 1 --2- 172.21.15.32:0/666073236 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f1f780b4870 0x7f1f780b4c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:04.096 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:04.094+0000 7f1f6e000640 1 -- 172.21.15.32:0/666073236 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1f781f5e50 con 0x7f1f781f47a0 2024-08-03T20:54:04.096 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:04.094+0000 7f1f77400640 1 --2- 172.21.15.32:0/666073236 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f1f780b4870 0x7f1f780b4c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-03T20:54:04.096 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:04.094+0000 7f1f76a00640 1 --2- 172.21.15.32:0/666073236 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f1f5c004800 0x7f1f781f5910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-03T20:54:04.096 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:04.094+0000 7f1f6e000640 1 --2- 172.21.15.32:0/666073236 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1f781f47a0 0x7f1f780cd930 secure :-1 s=READY pgs=842 cs=0 l=1 rev1=1 crypto rx=0x7f1f60036040 tx=0x7f1f60033670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:54:04.096 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:04.094+0000 7f1f6f400640 1 -- 172.21.15.32:0/666073236 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f60043070 con 0x7f1f781f47a0 2024-08-03T20:54:04.097 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:04.095+0000 7f1f6f400640 1 -- 172.21.15.32:0/666073236 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f1f60004140 con 0x7f1f781f47a0 2024-08-03T20:54:04.097 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:04.095+0000 7f1f6f400640 1 -- 172.21.15.32:0/666073236 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f60038040 con 0x7f1f781f47a0 2024-08-03T20:54:04.097 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:04.095+0000 7f1f77e00640 1 -- 172.21.15.32:0/666073236 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1f780c6e10 con 0x7f1f781f47a0 2024-08-03T20:54:04.097 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:04.095+0000 7f1f77e00640 1 -- 172.21.15.32:0/666073236 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1f780c7620 con 0x7f1f781f47a0 2024-08-03T20:54:04.097 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:04.096+0000 7f1f77e00640 1 -- 172.21.15.32:0/666073236 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1f78160780 con 0x7f1f781f47a0 2024-08-03T20:54:04.097 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:04.097+0000 7f1f6f400640 1 -- 172.21.15.32:0/666073236 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f1f60046070 con 0x7f1f781f47a0 2024-08-03T20:54:04.098 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:04.098+0000 7f1f6f400640 1 --2- 172.21.15.32:0/666073236 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f1f54077010 0x7f1f54079520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:04.098 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:04.098+0000 7f1f6f400640 1 -- 172.21.15.32:0/666073236 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(71..71 src has 1..71) ==== 5795+0+0 (secure 0 0 0) 0x7f1f600b91a0 con 0x7f1f781f47a0 2024-08-03T20:54:04.099 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:04.098+0000 7f1f76a00640 1 --2- 172.21.15.32:0/666073236 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f1f54077010 0x7f1f54079520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:04.099 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:04.099+0000 7f1f76a00640 1 --2- 172.21.15.32:0/666073236 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f1f54077010 0x7f1f54079520 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f1f68001d80 tx=0x7f1f680067a0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:54:04.103 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:04.102+0000 7f1f6f400640 1 -- 172.21.15.32:0/666073236 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7f1f60081c50 con 0x7f1f781f47a0 2024-08-03T20:54:04.311 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:04.310+0000 7f1f77e00640 1 -- 172.21.15.32:0/666073236 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f1f780c70d0 con 0x7f1f781f47a0 2024-08-03T20:54:04.312 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:04.312+0000 7f1f6f400640 1 -- 172.21.15.32:0/666073236 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v71) ==== 74+0+16737 (secure 0 0 0) 0x7f1f600861a0 con 0x7f1f781f47a0 2024-08-03T20:54:04.312 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-03T20:54:04.313 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":71,"fsid":"40ab141a-01d7-49b9-ae3c-2edec9876e95","created":"2024-08-03T20:47:14.957362+0000","modified":"2024-08-03T20:53:56.018994+0000","last_up_change":"2024-08-03T20:47:28.874451+0000","last_in_change":"2024-08-03T20:47:20.079442+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-08-03T20:47:28.261300+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs_metadata","create_time":"2024-08-03T20:53:13.054856+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"69","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_data","create_time":"2024-08-03T20:53:14.042324+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"70","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5a83e759-9035-438b-889a-0a6af1f0a0d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6816","nonce":3066156116},{"type":"v1","addr":"172.21.15.32:6817","nonce":3066156116}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6818","nonce":3066156116},{"type":"v1","addr":"172.21.15.32:6819","nonce":3066156116}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6822","nonce":3066156116},{"type":"v1","addr":"172.21.15.32:6823","nonce":3066156116}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6820","nonce":3066156116},{"type":"v1","addr":"172.21.15.32:6821","nonce":3066156116}]},"public_addr":"172.21.15.32:6817/3066156116","cluster_addr":"172.21.15.32:6819/3066156116","heartbeat_back_addr":"172.21.15.32:6823/3066156116","heartbeat_front_addr":"172.21.15.32:6821/3066156116","state":["exists","up"]},{"osd":1,"uuid":"d7c5c210-da4f-49f9-b9e7-1dfb02daa4bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6824","nonce":1551241759},{"type":"v1","addr":"172.21.15.32:6825","nonce":1551241759}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6826","nonce":1551241759},{"type":"v1","addr":"172.21.15.32:6827","nonce":1551241759}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6830","nonce":1551241759},{"type":"v1","addr":"172.21.15.32:6831","nonce":1551241759}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6828","nonce":1551241759},{"type":"v1","addr":"172.21.15.32:6829","nonce":1551241759}]},"public_addr":"172.21.15.32:6825/1551241759","cluster_addr":"172.21.15.32:6827/1551241759","heartbeat_back_addr":"172.21.15.32:6831/1551241759","heartbeat_front_addr":"172.21.15.32:6829/1551241759","state":["exists","up"]},{"osd":2,"uuid":"57f2e6de-a201-45c2-8981-8e56a4875ebd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6808","nonce":3806708944},{"type":"v1","addr":"172.21.15.32:6809","nonce":3806708944}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6810","nonce":3806708944},{"type":"v1","addr":"172.21.15.32:6811","nonce":3806708944}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6814","nonce":3806708944},{"type":"v1","addr":"172.21.15.32:6815","nonce":3806708944}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6812","nonce":3806708944},{"type":"v1","addr":"172.21.15.32:6813","nonce":3806708944}]},"public_addr":"172.21.15.32:6809/3806708944","cluster_addr":"172.21.15.32:6811/3806708944","heartbeat_back_addr":"172.21.15.32:6815/3806708944","heartbeat_front_addr":"172.21.15.32:6813/3806708944","state":["exists","up"]},{"osd":3,"uuid":"6439a280-da45-4148-8130-f16323935ff3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6800","nonce":9088303},{"type":"v1","addr":"172.21.15.32:6801","nonce":9088303}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6802","nonce":9088303},{"type":"v1","addr":"172.21.15.32:6803","nonce":9088303}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6806","nonce":9088303},{"type":"v1","addr":"172.21.15.32:6807","nonce":9088303}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6804","nonce":9088303},{"type":"v1","addr":"172.21.15.32:6805","nonce":9088303}]},"public_addr":"172.21.15.32:6801/9088303","cluster_addr":"172.21.15.32:6803/9088303","heartbeat_back_addr":"172.21.15.32:6807/9088303","heartbeat_front_addr":"172.21.15.32:6805/9088303","state":["exists","up"]},{"osd":4,"uuid":"4f902eff-1e18-418d-82c9-2151c69524fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6802","nonce":864752275},{"type":"v1","addr":"172.21.15.118:6803","nonce":864752275}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6804","nonce":864752275},{"type":"v1","addr":"172.21.15.118:6805","nonce":864752275}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6808","nonce":864752275},{"type":"v1","addr":"172.21.15.118:6809","nonce":864752275}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6806","nonce":864752275},{"type":"v1","addr":"172.21.15.118:6807","nonce":864752275}]},"public_addr":"172.21.15.118:6803/864752275","cluster_addr":"172.21.15.118:6805/864752275","heartbeat_back_addr":"172.21.15.118:6809/864752275","heartbeat_front_addr":"172.21.15.118:6807/864752275","state":["exists","up"]},{"osd":5,"uuid":"bfd9feac-dd06-458c-91b9-af7f175f612c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6810","nonce":2724891770},{"type":"v1","addr":"172.21.15.118:6811","nonce":2724891770}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6812","nonce":2724891770},{"type":"v1","addr":"172.21.15.118:6813","nonce":2724891770}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6816","nonce":2724891770},{"type":"v1","addr":"172.21.15.118:6817","nonce":2724891770}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6814","nonce":2724891770},{"type":"v1","addr":"172.21.15.118:6815","nonce":2724891770}]},"public_addr":"172.21.15.118:6811/2724891770","cluster_addr":"172.21.15.118:6813/2724891770","heartbeat_back_addr":"172.21.15.118:6817/2724891770","heartbeat_front_addr":"172.21.15.118:6815/2724891770","state":["exists","up"]},{"osd":6,"uuid":"5ed5a279-9d8e-416d-bbc7-7edd3e6d5705","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6825","nonce":801947386},{"type":"v1","addr":"172.21.15.118:6827","nonce":801947386}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6828","nonce":801947386},{"type":"v1","addr":"172.21.15.118:6829","nonce":801947386}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6832","nonce":801947386},{"type":"v1","addr":"172.21.15.118:6833","nonce":801947386}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6830","nonce":801947386},{"type":"v1","addr":"172.21.15.118:6831","nonce":801947386}]},"public_addr":"172.21.15.118:6827/801947386","cluster_addr":"172.21.15.118:6829/801947386","heartbeat_back_addr":"172.21.15.118:6833/801947386","heartbeat_front_addr":"172.21.15.118:6831/801947386","state":["exists","up"]},{"osd":7,"uuid":"f2086c47-875e-4451-a436-2008d1ea16af","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6818","nonce":653000048},{"type":"v1","addr":"172.21.15.118:6819","nonce":653000048}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6820","nonce":653000048},{"type":"v1","addr":"172.21.15.118:6821","nonce":653000048}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6824","nonce":653000048},{"type":"v1","addr":"172.21.15.118:6826","nonce":653000048}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6822","nonce":653000048},{"type":"v1","addr":"172.21.15.118:6823","nonce":653000048}]},"public_addr":"172.21.15.118:6819/653000048","cluster_addr":"172.21.15.118:6821/653000048","heartbeat_back_addr":"172.21.15.118:6826/653000048","heartbeat_front_addr":"172.21.15.118:6823/653000048","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-03T20:47:25.491994+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-03T20:47:25.587358+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-03T20:47:25.454175+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-03T20:47:25.377734+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-03T20:47:25.412351+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-03T20:47:25.499815+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-03T20:47:25.575044+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-03T20:47:25.552383+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.32:6835/4240363667":"2024-08-04T20:53:56.018944+0000","172.21.15.32:6833/4240363667":"2024-08-04T20:53:56.018944+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-08-03T20:54:04.315 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:04.314+0000 7f1f77e00640 1 -- 172.21.15.32:0/666073236 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f1f54077010 msgr2=0x7f1f54079520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:04.315 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:04.314+0000 7f1f77e00640 1 --2- 172.21.15.32:0/666073236 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f1f54077010 0x7f1f54079520 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f1f68001d80 tx=0x7f1f680067a0 comp rx=0 tx=0).stop 2024-08-03T20:54:04.315 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:04.314+0000 7f1f77e00640 1 -- 172.21.15.32:0/666073236 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1f781f47a0 msgr2=0x7f1f780cd930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:04.315 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:04.314+0000 7f1f77e00640 1 --2- 172.21.15.32:0/666073236 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1f781f47a0 0x7f1f780cd930 secure :-1 s=READY pgs=842 cs=0 l=1 rev1=1 crypto rx=0x7f1f60036040 tx=0x7f1f60033670 comp rx=0 tx=0).stop 2024-08-03T20:54:04.315 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:04.314+0000 7f1f77e00640 1 -- 172.21.15.32:0/666073236 shutdown_connections 2024-08-03T20:54:04.315 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:04.314+0000 7f1f77e00640 1 --2- 172.21.15.32:0/666073236 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f1f54077010 0x7f1f54079520 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:04.315 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:04.314+0000 7f1f77e00640 1 --2- 172.21.15.32:0/666073236 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f1f780b4870 0x7f1f780b4c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:04.316 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:04.314+0000 7f1f77e00640 1 --2- 172.21.15.32:0/666073236 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1f781f47a0 0x7f1f780cd930 unknown :-1 s=CLOSED pgs=842 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:04.316 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:04.314+0000 7f1f77e00640 1 --2- 172.21.15.32:0/666073236 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f1f5c004800 0x7f1f781f5910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:04.316 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:04.314+0000 7f1f77e00640 1 -- 172.21.15.32:0/666073236 >> 172.21.15.32:0/666073236 conn(0x7f1f780b2120 msgr2=0x7f1f780ad630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:54:04.316 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:04.314+0000 7f1f77e00640 1 -- 172.21.15.32:0/666073236 shutdown_connections 2024-08-03T20:54:04.316 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:04.314+0000 7f1f77e00640 1 -- 172.21.15.32:0/666073236 wait complete. 2024-08-03T20:54:04.339 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-08-03T20:54:04.339 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-08-03T20:54:04.569 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:04.568+0000 7f3620200640 1 -- 172.21.15.32:0/2679871319 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f36180b40b0 msgr2=0x7f361806fb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:04.569 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:04.568+0000 7f3620200640 1 --2- 172.21.15.32:0/2679871319 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f36180b40b0 0x7f361806fb30 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f3604009830 tx=0x7f360402a510 comp rx=0 tx=0).stop 2024-08-03T20:54:04.569 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:04.568+0000 7f3620200640 1 -- 172.21.15.32:0/2679871319 shutdown_connections 2024-08-03T20:54:04.569 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:04.568+0000 7f3620200640 1 --2- 172.21.15.32:0/2679871319 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f36180ade80 0x7f36180ae290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:04.569 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:04.568+0000 7f3620200640 1 --2- 172.21.15.32:0/2679871319 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f36180b40b0 0x7f361806fb30 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:04.569 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:04.568+0000 7f3620200640 1 -- 172.21.15.32:0/2679871319 >> 172.21.15.32:0/2679871319 conn(0x7f36180b1bd0 msgr2=0x7f36180b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:54:04.569 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:04.568+0000 7f3620200640 1 -- 172.21.15.32:0/2679871319 shutdown_connections 2024-08-03T20:54:04.569 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:04.568+0000 7f3620200640 1 -- 172.21.15.32:0/2679871319 wait complete. 2024-08-03T20:54:04.570 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:04.570+0000 7f3620200640 1 Processor -- start 2024-08-03T20:54:04.570 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:04.570+0000 7f3620200640 1 -- start start 2024-08-03T20:54:04.573 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:04.572+0000 7f3620200640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f361806f720 0x7f361814e590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:04.573 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:04.572+0000 7f3620200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f36180ade80 0x7f3618159250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:04.573 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:04.572+0000 7f3620200640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f36180b40b0 0x7f3618159790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:04.573 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:04.572+0000 7f3620200640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f36180aeb60 con 0x7f36180ade80 2024-08-03T20:54:04.573 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:04.572+0000 7f3620200640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f36180ae9e0 con 0x7f36180b40b0 2024-08-03T20:54:04.573 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:04.572+0000 7f3620200640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7f36180aece0 con 0x7f361806f720 2024-08-03T20:54:04.573 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:04.572+0000 7f361ce00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f36180ade80 0x7f3618159250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:04.574 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:04.572+0000 7f361ce00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f36180ade80 0x7f3618159250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:58030/0 (socket says 172.21.15.32:58030) 2024-08-03T20:54:04.574 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:04.572+0000 7f361ce00640 1 -- 172.21.15.32:0/4081489682 learned_addr learned my addr 172.21.15.32:0/4081489682 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:54:04.574 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:04.572+0000 7f361e200640 1 --2- 172.21.15.32:0/4081489682 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f36180b40b0 0x7f3618159790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:04.574 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:04.572+0000 7f361d800640 1 --2- 172.21.15.32:0/4081489682 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f361806f720 0x7f361814e590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:04.574 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:04.572+0000 7f361ce00640 1 -- 172.21.15.32:0/4081489682 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f361806f720 msgr2=0x7f361814e590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:04.574 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:04.572+0000 7f361ce00640 1 --2- 172.21.15.32:0/4081489682 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f361806f720 0x7f361814e590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:04.574 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:04.572+0000 7f361ce00640 1 -- 172.21.15.32:0/4081489682 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f36180b40b0 msgr2=0x7f3618159790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:04.574 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:04.572+0000 7f361ce00640 1 --2- 172.21.15.32:0/4081489682 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f36180b40b0 0x7f3618159790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:04.574 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:04.572+0000 7f361ce00640 1 -- 172.21.15.32:0/4081489682 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3618157230 con 0x7f36180ade80 2024-08-03T20:54:04.574 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:04.573+0000 7f361ce00640 1 --2- 172.21.15.32:0/4081489682 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f36180ade80 0x7f3618159250 secure :-1 s=READY pgs=843 cs=0 l=1 rev1=1 crypto rx=0x7f360402ab00 tx=0x7f3604033c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:54:04.575 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:04.573+0000 7f3615600640 1 -- 172.21.15.32:0/4081489682 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3604005c90 con 0x7f36180ade80 2024-08-03T20:54:04.575 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:04.573+0000 7f3615600640 1 -- 172.21.15.32:0/4081489682 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f3604031e50 con 0x7f36180ade80 2024-08-03T20:54:04.575 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:04.573+0000 7f3615600640 1 -- 172.21.15.32:0/4081489682 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3604005260 con 0x7f36180ade80 2024-08-03T20:54:04.575 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:04.573+0000 7f3620200640 1 -- 172.21.15.32:0/4081489682 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3618157510 con 0x7f36180ade80 2024-08-03T20:54:04.576 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:04.573+0000 7f3620200640 1 -- 172.21.15.32:0/4081489682 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3618159cd0 con 0x7f36180ade80 2024-08-03T20:54:04.576 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:04.574+0000 7f3620200640 1 -- 172.21.15.32:0/4081489682 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f36180ae290 con 0x7f36180ade80 2024-08-03T20:54:04.576 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:04.575+0000 7f3615600640 1 -- 172.21.15.32:0/4081489682 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f3604005400 con 0x7f36180ade80 2024-08-03T20:54:04.577 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:04.576+0000 7f3615600640 1 --2- 172.21.15.32:0/4081489682 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f35f8077010 0x7f35f8079520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:04.577 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:04.576+0000 7f3615600640 1 -- 172.21.15.32:0/4081489682 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(71..71 src has 1..71) ==== 5795+0+0 (secure 0 0 0) 0x7f36040b9aa0 con 0x7f36180ade80 2024-08-03T20:54:04.577 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:04.577+0000 7f361d800640 1 --2- 172.21.15.32:0/4081489682 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f35f8077010 0x7f35f8079520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:04.578 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:04.578+0000 7f361d800640 1 --2- 172.21.15.32:0/4081489682 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f35f8077010 0x7f35f8079520 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f360c007fd0 tx=0x7f360c009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:54:04.581 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:04.581+0000 7f3615600640 1 -- 172.21.15.32:0/4081489682 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7f360403f030 con 0x7f36180ade80 2024-08-03T20:54:04.824 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:04.823+0000 7f3620200640 1 -- 172.21.15.32:0/4081489682 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f3618156fc0 con 0x7f36180ade80 2024-08-03T20:54:04.824 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:04.824+0000 7f3615600640 1 -- 172.21.15.32:0/4081489682 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 81 v81) ==== 94+0+5233 (secure 0 0 0) 0x7f3604089e20 con 0x7f36180ade80 2024-08-03T20:54:04.825 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-03T20:54:04.825 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":81,"btime":"2024-08-03T20:53:59:036145+0000","default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5666,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6837/2261538396","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6835","nonce":2261538396},{"type":"v1","addr":"172.21.15.118:6837","nonce":2261538396}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":69},{"gid":6061,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6836/1133580656","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6834","nonce":1133580656},{"type":"v1","addr":"172.21.15.118:6836","nonce":1133580656}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":68},{"gid":6618,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6835/3466281885","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6833","nonce":3466281885},{"type":"v1","addr":"172.21.15.32:6835","nonce":3466281885}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":79}],"filesystems":[{"mdsmap":{"epoch":81,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-08-03T20:53:15.080669+0000","modified":"2024-08-03T20:53:59.036137+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":71,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6396},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6396":{"gid":6396,"name":"c","rank":0,"incarnation":78,"state":"up:active","state_seq":15,"addr":"172.21.15.32:6834/1798758431","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":1798758431},{"type":"v1","addr":"172.21.15.32:6834","nonce":1798758431}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6396,"qdb_cluster":[6396]},"id":5}]} 2024-08-03T20:54:04.825 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 81 2024-08-03T20:54:04.827 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:04.826+0000 7f3620200640 1 -- 172.21.15.32:0/4081489682 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f35f8077010 msgr2=0x7f35f8079520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:04.827 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:04.826+0000 7f3620200640 1 --2- 172.21.15.32:0/4081489682 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f35f8077010 0x7f35f8079520 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f360c007fd0 tx=0x7f360c009040 comp rx=0 tx=0).stop 2024-08-03T20:54:04.827 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:04.826+0000 7f3620200640 1 -- 172.21.15.32:0/4081489682 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f36180ade80 msgr2=0x7f3618159250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:04.827 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:04.826+0000 7f3620200640 1 --2- 172.21.15.32:0/4081489682 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f36180ade80 0x7f3618159250 secure :-1 s=READY pgs=843 cs=0 l=1 rev1=1 crypto rx=0x7f360402ab00 tx=0x7f3604033c20 comp rx=0 tx=0).stop 2024-08-03T20:54:04.827 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:04.826+0000 7f3620200640 1 -- 172.21.15.32:0/4081489682 shutdown_connections 2024-08-03T20:54:04.827 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:04.826+0000 7f3620200640 1 --2- 172.21.15.32:0/4081489682 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f35f8077010 0x7f35f8079520 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:04.827 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:04.826+0000 7f3620200640 1 --2- 172.21.15.32:0/4081489682 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f36180b40b0 0x7f3618159790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:04.827 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:04.826+0000 7f3620200640 1 --2- 172.21.15.32:0/4081489682 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f36180ade80 0x7f3618159250 unknown :-1 s=CLOSED pgs=843 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:04.827 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:04.826+0000 7f3620200640 1 --2- 172.21.15.32:0/4081489682 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f361806f720 0x7f361814e590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:04.827 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:04.826+0000 7f3620200640 1 -- 172.21.15.32:0/4081489682 >> 172.21.15.32:0/4081489682 conn(0x7f36180b1bd0 msgr2=0x7f36181533f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:54:04.828 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:04.826+0000 7f3620200640 1 -- 172.21.15.32:0/4081489682 shutdown_connections 2024-08-03T20:54:04.828 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:04.826+0000 7f3620200640 1 -- 172.21.15.32:0/4081489682 wait complete. 2024-08-03T20:54:04.849 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-08-03T20:54:05.079 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:05.078+0000 7fa7ab200640 1 -- 172.21.15.32:0/594277744 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa7ac0d26e0 msgr2=0x7fa7ac161b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:05.079 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:05.078+0000 7fa7ab200640 1 --2- 172.21.15.32:0/594277744 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa7ac0d26e0 0x7fa7ac161b80 secure :-1 s=READY pgs=844 cs=0 l=1 rev1=1 crypto rx=0x7fa79c005fd0 tx=0x7fa79c02a5d0 comp rx=0 tx=0).stop 2024-08-03T20:54:05.079 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:05.078+0000 7fa7ab200640 1 -- 172.21.15.32:0/594277744 shutdown_connections 2024-08-03T20:54:05.079 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:05.078+0000 7fa7ab200640 1 --2- 172.21.15.32:0/594277744 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fa7ac0cffe0 0x7fa7ac0b3e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:05.079 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:05.078+0000 7fa7ab200640 1 --2- 172.21.15.32:0/594277744 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa7ac0d26e0 0x7fa7ac161b80 unknown :-1 s=CLOSED pgs=844 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:05.079 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:05.078+0000 7fa7ab200640 1 -- 172.21.15.32:0/594277744 >> 172.21.15.32:0/594277744 conn(0x7fa7ac08d7a0 msgr2=0x7fa7ac09d3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:54:05.079 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:05.078+0000 7fa7ab200640 1 -- 172.21.15.32:0/594277744 shutdown_connections 2024-08-03T20:54:05.079 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:05.078+0000 7fa7ab200640 1 -- 172.21.15.32:0/594277744 wait complete. 2024-08-03T20:54:05.080 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:05.080+0000 7fa7ab200640 1 Processor -- start 2024-08-03T20:54:05.081 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:05.080+0000 7fa7ab200640 1 -- start start 2024-08-03T20:54:05.083 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:05.082+0000 7fa7ab200640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fa7ac0b43d0 0x7fa7ac156740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:05.083 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:05.082+0000 7fa7ab200640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa7ac0cffe0 0x7fa7ac0ca080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:05.084 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:05.082+0000 7fa7ab200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa7ac0d26e0 0x7fa7ac0b5a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:05.084 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:05.082+0000 7fa7ab200640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fa7ac0d4310 con 0x7fa7ac0d26e0 2024-08-03T20:54:05.084 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:05.082+0000 7fa7ab200640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7fa7ac0d4190 con 0x7fa7ac0cffe0 2024-08-03T20:54:05.084 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:05.082+0000 7fa7ab200640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7fa7ac0d4490 con 0x7fa7ac0b43d0 2024-08-03T20:54:05.084 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:05.082+0000 7fa7aa800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa7ac0d26e0 0x7fa7ac0b5a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:05.084 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:05.082+0000 7fa7aa800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa7ac0d26e0 0x7fa7ac0b5a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:58056/0 (socket says 172.21.15.32:58056) 2024-08-03T20:54:05.084 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:05.082+0000 7fa7aa800640 1 -- 172.21.15.32:0/1848729701 learned_addr learned my addr 172.21.15.32:0/1848729701 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:54:05.084 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:05.082+0000 7fa7a9e00640 1 --2- 172.21.15.32:0/1848729701 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fa7ac0b43d0 0x7fa7ac156740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:05.084 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:05.082+0000 7fa7a3e00640 1 --2- 172.21.15.32:0/1848729701 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa7ac0cffe0 0x7fa7ac0ca080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:05.084 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:05.082+0000 7fa7aa800640 1 -- 172.21.15.32:0/1848729701 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fa7ac0b43d0 msgr2=0x7fa7ac156740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:05.084 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:05.082+0000 7fa7aa800640 1 --2- 172.21.15.32:0/1848729701 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fa7ac0b43d0 0x7fa7ac156740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:05.084 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:05.082+0000 7fa7aa800640 1 -- 172.21.15.32:0/1848729701 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa7ac0cffe0 msgr2=0x7fa7ac0ca080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:05.084 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:05.082+0000 7fa7aa800640 1 --2- 172.21.15.32:0/1848729701 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa7ac0cffe0 0x7fa7ac0ca080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:05.084 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:05.082+0000 7fa7aa800640 1 -- 172.21.15.32:0/1848729701 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa7ac15eb60 con 0x7fa7ac0d26e0 2024-08-03T20:54:05.085 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:05.082+0000 7fa7a9e00640 1 --2- 172.21.15.32:0/1848729701 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fa7ac0b43d0 0x7fa7ac156740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-03T20:54:05.085 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:05.083+0000 7fa7aa800640 1 --2- 172.21.15.32:0/1848729701 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa7ac0d26e0 0x7fa7ac0b5a30 secure :-1 s=READY pgs=845 cs=0 l=1 rev1=1 crypto rx=0x7fa7a400c700 tx=0x7fa7a400cbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:54:05.085 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:05.083+0000 7fa7a2000640 1 -- 172.21.15.32:0/1848729701 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7a4003b70 con 0x7fa7ac0d26e0 2024-08-03T20:54:05.085 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:05.083+0000 7fa7a2000640 1 -- 172.21.15.32:0/1848729701 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fa7a4003d10 con 0x7fa7ac0d26e0 2024-08-03T20:54:05.085 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:05.083+0000 7fa7a2000640 1 -- 172.21.15.32:0/1848729701 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7a400aab0 con 0x7fa7ac0d26e0 2024-08-03T20:54:05.085 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:05.083+0000 7fa7ab200640 1 -- 172.21.15.32:0/1848729701 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa7ac1f5810 con 0x7fa7ac0d26e0 2024-08-03T20:54:05.085 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:05.083+0000 7fa7ab200640 1 -- 172.21.15.32:0/1848729701 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa7ac1f5dd0 con 0x7fa7ac0d26e0 2024-08-03T20:54:05.085 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:05.084+0000 7fa7ab200640 1 -- 172.21.15.32:0/1848729701 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa7ac15ff90 con 0x7fa7ac0d26e0 2024-08-03T20:54:05.086 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:05.085+0000 7fa7a2000640 1 -- 172.21.15.32:0/1848729701 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7fa7a400ac50 con 0x7fa7ac0d26e0 2024-08-03T20:54:05.087 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:05.086+0000 7fa7a2000640 1 --2- 172.21.15.32:0/1848729701 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fa788077010 0x7fa788079520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:05.087 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:05.087+0000 7fa7a2000640 1 -- 172.21.15.32:0/1848729701 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(71..71 src has 1..71) ==== 5795+0+0 (secure 0 0 0) 0x7fa7a40987a0 con 0x7fa7ac0d26e0 2024-08-03T20:54:05.087 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:05.087+0000 7fa7a9e00640 1 --2- 172.21.15.32:0/1848729701 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fa788077010 0x7fa788079520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:05.088 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:05.088+0000 7fa7a9e00640 1 --2- 172.21.15.32:0/1848729701 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fa788077010 0x7fa788079520 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7fa79c033000 tx=0x7fa79c004730 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:54:05.091 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:05.091+0000 7fa7a2000640 1 -- 172.21.15.32:0/1848729701 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7fa7ac15ff90 con 0x7fa7ac0d26e0 2024-08-03T20:54:05.334 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:05.334+0000 7fa7ab200640 1 -- 172.21.15.32:0/1848729701 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa7ac0d4940 con 0x7fa7ac0d26e0 2024-08-03T20:54:05.335 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:05.334+0000 7fa7a2000640 1 -- 172.21.15.32:0/1848729701 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 81 v81) ==== 94+0+5233 (secure 0 0 0) 0x7fa7ac15eb60 con 0x7fa7ac0d26e0 2024-08-03T20:54:05.335 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-03T20:54:05.335 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":81,"btime":"2024-08-03T20:53:59:036145+0000","default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5666,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6837/2261538396","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6835","nonce":2261538396},{"type":"v1","addr":"172.21.15.118:6837","nonce":2261538396}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":69},{"gid":6061,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6836/1133580656","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6834","nonce":1133580656},{"type":"v1","addr":"172.21.15.118:6836","nonce":1133580656}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":68},{"gid":6618,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6835/3466281885","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6833","nonce":3466281885},{"type":"v1","addr":"172.21.15.32:6835","nonce":3466281885}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":79}],"filesystems":[{"mdsmap":{"epoch":81,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-08-03T20:53:15.080669+0000","modified":"2024-08-03T20:53:59.036137+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":71,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6396},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6396":{"gid":6396,"name":"c","rank":0,"incarnation":78,"state":"up:active","state_seq":15,"addr":"172.21.15.32:6834/1798758431","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":1798758431},{"type":"v1","addr":"172.21.15.32:6834","nonce":1798758431}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6396,"qdb_cluster":[6396]},"id":5}]} 2024-08-03T20:54:05.335 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 81 2024-08-03T20:54:05.338 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:05.337+0000 7fa7ab200640 1 -- 172.21.15.32:0/1848729701 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fa788077010 msgr2=0x7fa788079520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:05.338 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:05.337+0000 7fa7ab200640 1 --2- 172.21.15.32:0/1848729701 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fa788077010 0x7fa788079520 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7fa79c033000 tx=0x7fa79c004730 comp rx=0 tx=0).stop 2024-08-03T20:54:05.338 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:05.337+0000 7fa7ab200640 1 -- 172.21.15.32:0/1848729701 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa7ac0d26e0 msgr2=0x7fa7ac0b5a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:05.338 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:05.337+0000 7fa7ab200640 1 --2- 172.21.15.32:0/1848729701 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa7ac0d26e0 0x7fa7ac0b5a30 secure :-1 s=READY pgs=845 cs=0 l=1 rev1=1 crypto rx=0x7fa7a400c700 tx=0x7fa7a400cbc0 comp rx=0 tx=0).stop 2024-08-03T20:54:05.338 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:05.337+0000 7fa7ab200640 1 -- 172.21.15.32:0/1848729701 shutdown_connections 2024-08-03T20:54:05.338 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:05.337+0000 7fa7ab200640 1 --2- 172.21.15.32:0/1848729701 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fa788077010 0x7fa788079520 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:05.338 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:05.337+0000 7fa7ab200640 1 --2- 172.21.15.32:0/1848729701 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa7ac0cffe0 0x7fa7ac0ca080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:05.338 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:05.337+0000 7fa7ab200640 1 --2- 172.21.15.32:0/1848729701 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa7ac0d26e0 0x7fa7ac0b5a30 unknown :-1 s=CLOSED pgs=845 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:05.338 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:05.337+0000 7fa7ab200640 1 --2- 172.21.15.32:0/1848729701 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fa7ac0b43d0 0x7fa7ac156740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:05.339 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:05.337+0000 7fa7ab200640 1 -- 172.21.15.32:0/1848729701 >> 172.21.15.32:0/1848729701 conn(0x7fa7ac08d7a0 msgr2=0x7fa7ac0b8f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:54:05.339 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:05.337+0000 7fa7ab200640 1 -- 172.21.15.32:0/1848729701 shutdown_connections 2024-08-03T20:54:05.339 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:05.337+0000 7fa7ab200640 1 -- 172.21.15.32:0/1848729701 wait complete. 2024-08-03T20:54:05.361 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-08-03T20:54:05.589 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:05.588+0000 7fa5c7600640 1 -- 172.21.15.32:0/1628443216 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa5a4004800 msgr2=0x7fa5a4004c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:05.589 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:05.588+0000 7fa5c7600640 1 --2- 172.21.15.32:0/1628443216 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa5a4004800 0x7fa5a4004c10 secure :-1 s=READY pgs=847 cs=0 l=1 rev1=1 crypto rx=0x7fa5a8001660 tx=0x7fa5a4005830 comp rx=0 tx=0).stop 2024-08-03T20:54:05.590 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:05.588+0000 7fa5c5600640 1 -- 172.21.15.32:0/1628443216 reap_dead start 2024-08-03T20:54:05.590 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:05.589+0000 7fa5c7600640 1 -- 172.21.15.32:0/1628443216 shutdown_connections 2024-08-03T20:54:05.590 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:05.589+0000 7fa5c7600640 1 -- 172.21.15.32:0/1628443216 >> 172.21.15.32:0/1628443216 conn(0x7fa5c00b1bd0 msgr2=0x7fa5c00b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:54:05.590 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:05.589+0000 7fa5c7600640 1 -- 172.21.15.32:0/1628443216 shutdown_connections 2024-08-03T20:54:05.590 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:05.589+0000 7fa5c7600640 1 -- 172.21.15.32:0/1628443216 wait complete. 2024-08-03T20:54:05.591 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:05.591+0000 7fa5c7600640 1 Processor -- start 2024-08-03T20:54:05.591 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:05.591+0000 7fa5c7600640 1 -- start start 2024-08-03T20:54:05.594 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:05.592+0000 7fa5c7600640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa5a4004800 0x7fa5c013b180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:05.595 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:05.592+0000 7fa5c7600640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fa5c00b3ce0 0x7fa5c00b40f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:05.595 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:05.592+0000 7fa5c7600640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa5c00adad0 0x7fa5c00adee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:05.595 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:05.592+0000 7fa5c7600640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fa5c0145f10 con 0x7fa5a4004800 2024-08-03T20:54:05.595 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:05.592+0000 7fa5c7600640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7fa5c00cfa50 con 0x7fa5c00adad0 2024-08-03T20:54:05.595 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:05.593+0000 7fa5c7600640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7fa5c0160b10 con 0x7fa5c00b3ce0 2024-08-03T20:54:05.595 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:05.593+0000 7fa5c4c00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa5a4004800 0x7fa5c013b180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:05.595 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:05.593+0000 7fa5c4c00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa5a4004800 0x7fa5c013b180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:58074/0 (socket says 172.21.15.32:58074) 2024-08-03T20:54:05.595 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:05.593+0000 7fa5c4c00640 1 -- 172.21.15.32:0/3367074532 learned_addr learned my addr 172.21.15.32:0/3367074532 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:54:05.595 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:05.593+0000 7fa5b7e00640 1 --2- 172.21.15.32:0/3367074532 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fa5c00b3ce0 0x7fa5c00b40f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:05.595 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:05.593+0000 7fa5c5600640 1 --2- 172.21.15.32:0/3367074532 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa5c00adad0 0x7fa5c00adee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:05.595 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:05.593+0000 7fa5c4c00640 1 -- 172.21.15.32:0/3367074532 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fa5c00b3ce0 msgr2=0x7fa5c00b40f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:05.595 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:05.593+0000 7fa5c4c00640 1 --2- 172.21.15.32:0/3367074532 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fa5c00b3ce0 0x7fa5c00b40f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:05.595 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:05.593+0000 7fa5c4c00640 1 -- 172.21.15.32:0/3367074532 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa5c00adad0 msgr2=0x7fa5c00adee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:05.596 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:05.593+0000 7fa5c4c00640 1 --2- 172.21.15.32:0/3367074532 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa5c00adad0 0x7fa5c00adee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:05.596 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:05.593+0000 7fa5c4c00640 1 -- 172.21.15.32:0/3367074532 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa5c00b4630 con 0x7fa5a4004800 2024-08-03T20:54:05.596 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:05.593+0000 7fa5b7e00640 1 --2- 172.21.15.32:0/3367074532 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fa5c00b3ce0 0x7fa5c00b40f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-03T20:54:05.596 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:05.593+0000 7fa5c5600640 1 --2- 172.21.15.32:0/3367074532 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa5c00adad0 0x7fa5c00adee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-03T20:54:05.596 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:05.594+0000 7fa5c4c00640 1 --2- 172.21.15.32:0/3367074532 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa5a4004800 0x7fa5c013b180 secure :-1 s=READY pgs=848 cs=0 l=1 rev1=1 crypto rx=0x7fa5b0011ae0 tx=0x7fa5b0009e90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:54:05.596 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:05.594+0000 7fa5bd600640 1 -- 172.21.15.32:0/3367074532 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5b001e070 con 0x7fa5a4004800 2024-08-03T20:54:05.596 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:05.594+0000 7fa5bd600640 1 -- 172.21.15.32:0/3367074532 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fa5b000fe70 con 0x7fa5a4004800 2024-08-03T20:54:05.596 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:05.594+0000 7fa5c7600640 1 -- 172.21.15.32:0/3367074532 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa5c00aeae0 con 0x7fa5a4004800 2024-08-03T20:54:05.596 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:05.594+0000 7fa5bd600640 1 -- 172.21.15.32:0/3367074532 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5b0004290 con 0x7fa5a4004800 2024-08-03T20:54:05.596 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:05.594+0000 7fa5c7600640 1 -- 172.21.15.32:0/3367074532 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa5c00af0a0 con 0x7fa5a4004800 2024-08-03T20:54:05.597 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:05.595+0000 7fa5c7600640 1 -- 172.21.15.32:0/3367074532 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa5c0144f90 con 0x7fa5a4004800 2024-08-03T20:54:05.597 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:05.596+0000 7fa5bd600640 1 -- 172.21.15.32:0/3367074532 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7fa5b0021070 con 0x7fa5a4004800 2024-08-03T20:54:05.597 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:05.597+0000 7fa5bd600640 1 --2- 172.21.15.32:0/3367074532 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fa59c077010 0x7fa59c079520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:05.598 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:05.597+0000 7fa5bd600640 1 -- 172.21.15.32:0/3367074532 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(71..71 src has 1..71) ==== 5795+0+0 (secure 0 0 0) 0x7fa5b00993a0 con 0x7fa5a4004800 2024-08-03T20:54:05.598 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:05.598+0000 7fa5b7e00640 1 --2- 172.21.15.32:0/3367074532 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fa59c077010 0x7fa59c079520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:05.599 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:05.599+0000 7fa5b7e00640 1 --2- 172.21.15.32:0/3367074532 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fa59c077010 0x7fa59c079520 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7fa5a8032000 tx=0x7fa5a8001190 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:54:05.602 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:05.602+0000 7fa5bd600640 1 -- 172.21.15.32:0/3367074532 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7fa5c00b4630 con 0x7fa5a4004800 2024-08-03T20:54:05.809 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:05.809+0000 7fa5c7600640 1 -- 172.21.15.32:0/3367074532 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fa5c0160580 con 0x7fa5a4004800 2024-08-03T20:54:05.810 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:05.810+0000 7fa5bd600640 1 -- 172.21.15.32:0/3367074532 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v71) ==== 74+0+16737 (secure 0 0 0) 0x7fa5b0061fa0 con 0x7fa5a4004800 2024-08-03T20:54:05.811 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-03T20:54:05.811 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":71,"fsid":"40ab141a-01d7-49b9-ae3c-2edec9876e95","created":"2024-08-03T20:47:14.957362+0000","modified":"2024-08-03T20:53:56.018994+0000","last_up_change":"2024-08-03T20:47:28.874451+0000","last_in_change":"2024-08-03T20:47:20.079442+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-08-03T20:47:28.261300+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs_metadata","create_time":"2024-08-03T20:53:13.054856+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"69","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_data","create_time":"2024-08-03T20:53:14.042324+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"70","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5a83e759-9035-438b-889a-0a6af1f0a0d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6816","nonce":3066156116},{"type":"v1","addr":"172.21.15.32:6817","nonce":3066156116}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6818","nonce":3066156116},{"type":"v1","addr":"172.21.15.32:6819","nonce":3066156116}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6822","nonce":3066156116},{"type":"v1","addr":"172.21.15.32:6823","nonce":3066156116}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6820","nonce":3066156116},{"type":"v1","addr":"172.21.15.32:6821","nonce":3066156116}]},"public_addr":"172.21.15.32:6817/3066156116","cluster_addr":"172.21.15.32:6819/3066156116","heartbeat_back_addr":"172.21.15.32:6823/3066156116","heartbeat_front_addr":"172.21.15.32:6821/3066156116","state":["exists","up"]},{"osd":1,"uuid":"d7c5c210-da4f-49f9-b9e7-1dfb02daa4bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6824","nonce":1551241759},{"type":"v1","addr":"172.21.15.32:6825","nonce":1551241759}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6826","nonce":1551241759},{"type":"v1","addr":"172.21.15.32:6827","nonce":1551241759}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6830","nonce":1551241759},{"type":"v1","addr":"172.21.15.32:6831","nonce":1551241759}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6828","nonce":1551241759},{"type":"v1","addr":"172.21.15.32:6829","nonce":1551241759}]},"public_addr":"172.21.15.32:6825/1551241759","cluster_addr":"172.21.15.32:6827/1551241759","heartbeat_back_addr":"172.21.15.32:6831/1551241759","heartbeat_front_addr":"172.21.15.32:6829/1551241759","state":["exists","up"]},{"osd":2,"uuid":"57f2e6de-a201-45c2-8981-8e56a4875ebd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6808","nonce":3806708944},{"type":"v1","addr":"172.21.15.32:6809","nonce":3806708944}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6810","nonce":3806708944},{"type":"v1","addr":"172.21.15.32:6811","nonce":3806708944}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6814","nonce":3806708944},{"type":"v1","addr":"172.21.15.32:6815","nonce":3806708944}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6812","nonce":3806708944},{"type":"v1","addr":"172.21.15.32:6813","nonce":3806708944}]},"public_addr":"172.21.15.32:6809/3806708944","cluster_addr":"172.21.15.32:6811/3806708944","heartbeat_back_addr":"172.21.15.32:6815/3806708944","heartbeat_front_addr":"172.21.15.32:6813/3806708944","state":["exists","up"]},{"osd":3,"uuid":"6439a280-da45-4148-8130-f16323935ff3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6800","nonce":9088303},{"type":"v1","addr":"172.21.15.32:6801","nonce":9088303}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6802","nonce":9088303},{"type":"v1","addr":"172.21.15.32:6803","nonce":9088303}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6806","nonce":9088303},{"type":"v1","addr":"172.21.15.32:6807","nonce":9088303}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6804","nonce":9088303},{"type":"v1","addr":"172.21.15.32:6805","nonce":9088303}]},"public_addr":"172.21.15.32:6801/9088303","cluster_addr":"172.21.15.32:6803/9088303","heartbeat_back_addr":"172.21.15.32:6807/9088303","heartbeat_front_addr":"172.21.15.32:6805/9088303","state":["exists","up"]},{"osd":4,"uuid":"4f902eff-1e18-418d-82c9-2151c69524fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6802","nonce":864752275},{"type":"v1","addr":"172.21.15.118:6803","nonce":864752275}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6804","nonce":864752275},{"type":"v1","addr":"172.21.15.118:6805","nonce":864752275}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6808","nonce":864752275},{"type":"v1","addr":"172.21.15.118:6809","nonce":864752275}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6806","nonce":864752275},{"type":"v1","addr":"172.21.15.118:6807","nonce":864752275}]},"public_addr":"172.21.15.118:6803/864752275","cluster_addr":"172.21.15.118:6805/864752275","heartbeat_back_addr":"172.21.15.118:6809/864752275","heartbeat_front_addr":"172.21.15.118:6807/864752275","state":["exists","up"]},{"osd":5,"uuid":"bfd9feac-dd06-458c-91b9-af7f175f612c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6810","nonce":2724891770},{"type":"v1","addr":"172.21.15.118:6811","nonce":2724891770}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6812","nonce":2724891770},{"type":"v1","addr":"172.21.15.118:6813","nonce":2724891770}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6816","nonce":2724891770},{"type":"v1","addr":"172.21.15.118:6817","nonce":2724891770}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6814","nonce":2724891770},{"type":"v1","addr":"172.21.15.118:6815","nonce":2724891770}]},"public_addr":"172.21.15.118:6811/2724891770","cluster_addr":"172.21.15.118:6813/2724891770","heartbeat_back_addr":"172.21.15.118:6817/2724891770","heartbeat_front_addr":"172.21.15.118:6815/2724891770","state":["exists","up"]},{"osd":6,"uuid":"5ed5a279-9d8e-416d-bbc7-7edd3e6d5705","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6825","nonce":801947386},{"type":"v1","addr":"172.21.15.118:6827","nonce":801947386}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6828","nonce":801947386},{"type":"v1","addr":"172.21.15.118:6829","nonce":801947386}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6832","nonce":801947386},{"type":"v1","addr":"172.21.15.118:6833","nonce":801947386}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6830","nonce":801947386},{"type":"v1","addr":"172.21.15.118:6831","nonce":801947386}]},"public_addr":"172.21.15.118:6827/801947386","cluster_addr":"172.21.15.118:6829/801947386","heartbeat_back_addr":"172.21.15.118:6833/801947386","heartbeat_front_addr":"172.21.15.118:6831/801947386","state":["exists","up"]},{"osd":7,"uuid":"f2086c47-875e-4451-a436-2008d1ea16af","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6818","nonce":653000048},{"type":"v1","addr":"172.21.15.118:6819","nonce":653000048}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6820","nonce":653000048},{"type":"v1","addr":"172.21.15.118:6821","nonce":653000048}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6824","nonce":653000048},{"type":"v1","addr":"172.21.15.118:6826","nonce":653000048}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6822","nonce":653000048},{"type":"v1","addr":"172.21.15.118:6823","nonce":653000048}]},"public_addr":"172.21.15.118:6819/653000048","cluster_addr":"172.21.15.118:6821/653000048","heartbeat_back_addr":"172.21.15.118:6826/653000048","heartbeat_front_addr":"172.21.15.118:6823/653000048","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-03T20:47:25.491994+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-03T20:47:25.587358+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-03T20:47:25.454175+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-03T20:47:25.377734+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-03T20:47:25.412351+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-03T20:47:25.499815+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-03T20:47:25.575044+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-03T20:47:25.552383+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.32:6835/4240363667":"2024-08-04T20:53:56.018944+0000","172.21.15.32:6833/4240363667":"2024-08-04T20:53:56.018944+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-08-03T20:54:05.813 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:05.812+0000 7fa5c7600640 1 -- 172.21.15.32:0/3367074532 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fa59c077010 msgr2=0x7fa59c079520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:05.813 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:05.812+0000 7fa5c7600640 1 --2- 172.21.15.32:0/3367074532 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fa59c077010 0x7fa59c079520 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7fa5a8032000 tx=0x7fa5a8001190 comp rx=0 tx=0).stop 2024-08-03T20:54:05.813 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:05.812+0000 7fa5c7600640 1 -- 172.21.15.32:0/3367074532 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa5a4004800 msgr2=0x7fa5c013b180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:05.813 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:05.812+0000 7fa5c7600640 1 --2- 172.21.15.32:0/3367074532 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa5a4004800 0x7fa5c013b180 secure :-1 s=READY pgs=848 cs=0 l=1 rev1=1 crypto rx=0x7fa5b0011ae0 tx=0x7fa5b0009e90 comp rx=0 tx=0).stop 2024-08-03T20:54:05.814 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:05.813+0000 7fa5c7600640 1 -- 172.21.15.32:0/3367074532 shutdown_connections 2024-08-03T20:54:05.814 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:05.813+0000 7fa5c7600640 1 --2- 172.21.15.32:0/3367074532 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fa59c077010 0x7fa59c079520 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:05.814 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:05.813+0000 7fa5c7600640 1 --2- 172.21.15.32:0/3367074532 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa5c00adad0 0x7fa5c00adee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:05.814 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:05.813+0000 7fa5c7600640 1 --2- 172.21.15.32:0/3367074532 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fa5c00b3ce0 0x7fa5c00b40f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:05.814 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:05.813+0000 7fa5c7600640 1 --2- 172.21.15.32:0/3367074532 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa5a4004800 0x7fa5c013b180 unknown :-1 s=CLOSED pgs=848 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:05.814 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:05.813+0000 7fa5c7600640 1 -- 172.21.15.32:0/3367074532 >> 172.21.15.32:0/3367074532 conn(0x7fa5c00b1bd0 msgr2=0x7fa5c01451a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:54:05.814 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:05.813+0000 7fa5c7600640 1 -- 172.21.15.32:0/3367074532 shutdown_connections 2024-08-03T20:54:05.814 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:05.813+0000 7fa5c7600640 1 -- 172.21.15.32:0/3367074532 wait complete. 2024-08-03T20:54:05.836 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-08-03T20:54:06.065 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:06.064+0000 7f1b09400640 1 -- 172.21.15.32:0/782637151 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1b04144f90 msgr2=0x7f1b04074570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:06.066 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:06.064+0000 7f1b09400640 1 --2- 172.21.15.32:0/782637151 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1b04144f90 0x7f1b04074570 secure :-1 s=READY pgs=849 cs=0 l=1 rev1=1 crypto rx=0x7f1af0005fd0 tx=0x7f1af002a5d0 comp rx=0 tx=0).stop 2024-08-03T20:54:06.066 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:06.065+0000 7f1b09400640 1 -- 172.21.15.32:0/782637151 shutdown_connections 2024-08-03T20:54:06.066 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:06.065+0000 7f1b09400640 1 --2- 172.21.15.32:0/782637151 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f1b0415fff0 0x7f1b04160400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:06.066 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:06.065+0000 7f1b09400640 1 --2- 172.21.15.32:0/782637151 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1b04144f90 0x7f1b04074570 unknown :-1 s=CLOSED pgs=849 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:06.066 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:06.065+0000 7f1b09400640 1 -- 172.21.15.32:0/782637151 >> 172.21.15.32:0/782637151 conn(0x7f1b040b1bd0 msgr2=0x7f1b040b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:54:06.066 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:06.065+0000 7f1b09400640 1 -- 172.21.15.32:0/782637151 shutdown_connections 2024-08-03T20:54:06.066 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:06.065+0000 7f1b09400640 1 -- 172.21.15.32:0/782637151 wait complete. 2024-08-03T20:54:06.067 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:06.067+0000 7f1b09400640 1 Processor -- start 2024-08-03T20:54:06.067 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:06.067+0000 7f1b09400640 1 -- start start 2024-08-03T20:54:06.070 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:06.069+0000 7f1b09400640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f1b0406e930 0x7f1b04157980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:06.070 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:06.069+0000 7f1b09400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1b04144f90 0x7f1b041f6440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:06.070 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:06.069+0000 7f1b09400640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f1b0415fff0 0x7f1b04159250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:06.071 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:06.069+0000 7f1b09400640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f1b040cf840 con 0x7f1b04144f90 2024-08-03T20:54:06.071 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:06.069+0000 7f1b09400640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f1b04145f30 con 0x7f1b0406e930 2024-08-03T20:54:06.071 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:06.069+0000 7f1b09400640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7f1b040cf9c0 con 0x7f1b0415fff0 2024-08-03T20:54:06.071 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:06.069+0000 7f1b01e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1b04144f90 0x7f1b041f6440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:06.071 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:06.069+0000 7f1b01e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1b04144f90 0x7f1b041f6440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:58092/0 (socket says 172.21.15.32:58092) 2024-08-03T20:54:06.071 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:06.069+0000 7f1b01e00640 1 -- 172.21.15.32:0/1651127719 learned_addr learned my addr 172.21.15.32:0/1651127719 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:54:06.071 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:06.069+0000 7f1b03200640 1 --2- 172.21.15.32:0/1651127719 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f1b0415fff0 0x7f1b04159250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:06.071 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:06.069+0000 7f1b02800640 1 --2- 172.21.15.32:0/1651127719 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f1b0406e930 0x7f1b04157980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:06.071 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:06.069+0000 7f1b01e00640 1 -- 172.21.15.32:0/1651127719 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f1b0415fff0 msgr2=0x7f1b04159250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:06.071 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:06.069+0000 7f1b01e00640 1 --2- 172.21.15.32:0/1651127719 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f1b0415fff0 0x7f1b04159250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:06.071 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:06.069+0000 7f1b01e00640 1 -- 172.21.15.32:0/1651127719 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f1b0406e930 msgr2=0x7f1b04157980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:06.071 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:06.069+0000 7f1b01e00640 1 --2- 172.21.15.32:0/1651127719 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f1b0406e930 0x7f1b04157980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:06.072 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:06.069+0000 7f1b01e00640 1 -- 172.21.15.32:0/1651127719 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1b041f6980 con 0x7f1b04144f90 2024-08-03T20:54:06.072 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:06.070+0000 7f1b01e00640 1 --2- 172.21.15.32:0/1651127719 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1b04144f90 0x7f1b041f6440 secure :-1 s=READY pgs=850 cs=0 l=1 rev1=1 crypto rx=0x7f1af0034040 tx=0x7f1af00038a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:54:06.072 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:06.070+0000 7f1afaa00640 1 -- 172.21.15.32:0/1651127719 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1af003e030 con 0x7f1b04144f90 2024-08-03T20:54:06.073 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:06.070+0000 7f1afaa00640 1 -- 172.21.15.32:0/1651127719 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f1af0003ca0 con 0x7f1b04144f90 2024-08-03T20:54:06.073 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:06.070+0000 7f1afaa00640 1 -- 172.21.15.32:0/1651127719 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1af0035040 con 0x7f1b04144f90 2024-08-03T20:54:06.073 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:06.070+0000 7f1b09400640 1 -- 172.21.15.32:0/1651127719 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1b041581b0 con 0x7f1b04144f90 2024-08-03T20:54:06.073 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:06.070+0000 7f1b09400640 1 -- 172.21.15.32:0/1651127719 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1b04158770 con 0x7f1b04144f90 2024-08-03T20:54:06.073 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:06.072+0000 7f1b09400640 1 -- 172.21.15.32:0/1651127719 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1b04160cc0 con 0x7f1b04144f90 2024-08-03T20:54:06.073 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:06.072+0000 7f1afaa00640 1 -- 172.21.15.32:0/1651127719 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f1af0041030 con 0x7f1b04144f90 2024-08-03T20:54:06.074 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:06.073+0000 7f1afaa00640 1 --2- 172.21.15.32:0/1651127719 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f1ae4077010 0x7f1ae4079520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:06.074 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:06.074+0000 7f1afaa00640 1 -- 172.21.15.32:0/1651127719 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(71..71 src has 1..71) ==== 5795+0+0 (secure 0 0 0) 0x7f1af00b8fe0 con 0x7f1b04144f90 2024-08-03T20:54:06.074 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:06.074+0000 7f1b02800640 1 --2- 172.21.15.32:0/1651127719 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f1ae4077010 0x7f1ae4079520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:06.075 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:06.075+0000 7f1b02800640 1 --2- 172.21.15.32:0/1651127719 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f1ae4077010 0x7f1ae4079520 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f1af40015e0 tx=0x7f1af4009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:54:06.078 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:06.078+0000 7f1afaa00640 1 -- 172.21.15.32:0/1651127719 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7f1af00820e0 con 0x7f1b04144f90 2024-08-03T20:54:06.326 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:06.326+0000 7f1b09400640 1 -- 172.21.15.32:0/1651127719 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7f1b040b2080 con 0x7f1b04144f90 2024-08-03T20:54:06.327 INFO:tasks.ceph.mon.a.smithi032.stderr:2024-08-03T20:54:06.327+0000 7f4e6d600640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-08-03T20:54:06.329 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:06.329+0000 7f1afaa00640 1 -- 172.21.15.32:0/1651127719 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v82) ==== 161+0+0 (secure 0 0 0) 0x7f1af008a3b0 con 0x7f1b04144f90 2024-08-03T20:54:06.329 INFO:teuthology.orchestra.run.smithi032.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-08-03T20:54:06.331 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:06.331+0000 7f1b09400640 1 -- 172.21.15.32:0/1651127719 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f1ae4077010 msgr2=0x7f1ae4079520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:06.331 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:06.331+0000 7f1b09400640 1 --2- 172.21.15.32:0/1651127719 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f1ae4077010 0x7f1ae4079520 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f1af40015e0 tx=0x7f1af4009040 comp rx=0 tx=0).stop 2024-08-03T20:54:06.332 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:06.331+0000 7f1b09400640 1 -- 172.21.15.32:0/1651127719 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1b04144f90 msgr2=0x7f1b041f6440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:06.332 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:06.331+0000 7f1b09400640 1 --2- 172.21.15.32:0/1651127719 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1b04144f90 0x7f1b041f6440 secure :-1 s=READY pgs=850 cs=0 l=1 rev1=1 crypto rx=0x7f1af0034040 tx=0x7f1af00038a0 comp rx=0 tx=0).stop 2024-08-03T20:54:06.332 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:06.332+0000 7f1b09400640 1 -- 172.21.15.32:0/1651127719 shutdown_connections 2024-08-03T20:54:06.332 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:06.332+0000 7f1b09400640 1 --2- 172.21.15.32:0/1651127719 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f1ae4077010 0x7f1ae4079520 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:06.333 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:06.332+0000 7f1b09400640 1 --2- 172.21.15.32:0/1651127719 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f1b0415fff0 0x7f1b04159250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:06.333 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:06.332+0000 7f1b09400640 1 --2- 172.21.15.32:0/1651127719 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1b04144f90 0x7f1b041f6440 unknown :-1 s=CLOSED pgs=850 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:06.333 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:06.332+0000 7f1b09400640 1 --2- 172.21.15.32:0/1651127719 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f1b0406e930 0x7f1b04157980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:06.333 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:06.332+0000 7f1b09400640 1 -- 172.21.15.32:0/1651127719 >> 172.21.15.32:0/1651127719 conn(0x7f1b040b1bd0 msgr2=0x7f1b0406f930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:54:06.333 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:06.332+0000 7f1b09400640 1 -- 172.21.15.32:0/1651127719 shutdown_connections 2024-08-03T20:54:06.333 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:06.332+0000 7f1b09400640 1 -- 172.21.15.32:0/1651127719 wait complete. 2024-08-03T20:54:06.343 INFO:tasks.ceph.mds.c.smithi032.stderr: -225> 2024-08-03T20:54:02.532+0000 7faa8fc00640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-08-03T20:54:06.359 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-08-03T20:54:06.460 INFO:tasks.ceph.mds.c.smithi032.stdout:starting mds.c at 2024-08-03T20:54:06.593 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:06.591+0000 7fb86d800640 1 -- 172.21.15.32:0/2947147203 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb84c003a50 msgr2=0x7fb84c003ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:06.593 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:06.591+0000 7fb86d800640 1 --2- 172.21.15.32:0/2947147203 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb84c003a50 0x7fb84c003ec0 secure :-1 s=READY pgs=854 cs=0 l=1 rev1=1 crypto rx=0x7fb85800c310 tx=0x7fb84c005830 comp rx=0 tx=0).stop 2024-08-03T20:54:06.593 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:06.592+0000 7fb86d800640 1 -- 172.21.15.32:0/2947147203 shutdown_connections 2024-08-03T20:54:06.593 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:06.592+0000 7fb86d800640 1 --2- 172.21.15.32:0/2947147203 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb84c003a50 0x7fb84c003ec0 unknown :-1 s=CLOSED pgs=854 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:06.593 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:06.592+0000 7fb86d800640 1 -- 172.21.15.32:0/2947147203 >> 172.21.15.32:0/2947147203 conn(0x7fb8680b1bd0 msgr2=0x7fb8680b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:54:06.593 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:06.592+0000 7fb86d800640 1 -- 172.21.15.32:0/2947147203 shutdown_connections 2024-08-03T20:54:06.593 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:06.592+0000 7fb86d800640 1 -- 172.21.15.32:0/2947147203 wait complete. 2024-08-03T20:54:06.594 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:06.594+0000 7fb86d800640 1 Processor -- start 2024-08-03T20:54:06.594 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:06.594+0000 7fb86d800640 1 -- start start 2024-08-03T20:54:06.596 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:06.595+0000 7fb86d800640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fb84c003a50 0x7fb86813bef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:06.597 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:06.595+0000 7fb86d800640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb8681438e0 0x7fb868159d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:06.597 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:06.595+0000 7fb86d800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb86813ee30 0x7fb86813f240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:06.597 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:06.595+0000 7fb86d800640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fb86806ead0 con 0x7fb86813ee30 2024-08-03T20:54:06.597 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:06.595+0000 7fb86d800640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7fb86806e950 con 0x7fb8681438e0 2024-08-03T20:54:06.597 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:06.595+0000 7fb86d800640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7fb86806ec50 con 0x7fb84c003a50 2024-08-03T20:54:06.597 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:06.596+0000 7fb867400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb86813ee30 0x7fb86813f240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:06.597 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:06.596+0000 7fb867400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb86813ee30 0x7fb86813f240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:58134/0 (socket says 172.21.15.32:58134) 2024-08-03T20:54:06.597 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:06.596+0000 7fb85e000640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb8681438e0 0x7fb868159d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:06.597 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:06.596+0000 7fb867400640 1 -- 172.21.15.32:0/2792870100 learned_addr learned my addr 172.21.15.32:0/2792870100 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:54:06.597 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:06.596+0000 7fb866a00640 1 --2- 172.21.15.32:0/2792870100 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fb84c003a50 0x7fb86813bef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:06.599 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:06.596+0000 7fb867400640 1 -- 172.21.15.32:0/2792870100 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fb84c003a50 msgr2=0x7fb86813bef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:06.599 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:06.596+0000 7fb867400640 1 --2- 172.21.15.32:0/2792870100 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fb84c003a50 0x7fb86813bef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:06.599 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:06.596+0000 7fb867400640 1 -- 172.21.15.32:0/2792870100 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb8681438e0 msgr2=0x7fb868159d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:06.600 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:06.596+0000 7fb867400640 1 --2- 172.21.15.32:0/2792870100 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb8681438e0 0x7fb868159d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:06.600 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:06.596+0000 7fb867400640 1 -- 172.21.15.32:0/2792870100 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb86813f780 con 0x7fb86813ee30 2024-08-03T20:54:06.600 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:06.596+0000 7fb85e000640 1 --2- 172.21.15.32:0/2792870100 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb8681438e0 0x7fb868159d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-03T20:54:06.600 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:06.597+0000 7fb867400640 1 --2- 172.21.15.32:0/2792870100 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb86813ee30 0x7fb86813f240 secure :-1 s=READY pgs=855 cs=0 l=1 rev1=1 crypto rx=0x7fb8680cdd20 tx=0x7fb8600019b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:54:06.600 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:06.597+0000 7fb85f400640 1 -- 172.21.15.32:0/2792870100 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb860015020 con 0x7fb86813ee30 2024-08-03T20:54:06.600 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:06.597+0000 7fb85f400640 1 -- 172.21.15.32:0/2792870100 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fb860011960 con 0x7fb86813ee30 2024-08-03T20:54:06.600 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:06.597+0000 7fb85f400640 1 -- 172.21.15.32:0/2792870100 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb860011d90 con 0x7fb86813ee30 2024-08-03T20:54:06.601 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:06.597+0000 7fb86d800640 1 -- 172.21.15.32:0/2792870100 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb8681f31f0 con 0x7fb86813ee30 2024-08-03T20:54:06.601 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:06.597+0000 7fb86d800640 1 -- 172.21.15.32:0/2792870100 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb8681f37b0 con 0x7fb86813ee30 2024-08-03T20:54:06.601 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:06.598+0000 7fb86d800640 1 -- 172.21.15.32:0/2792870100 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb86813c560 con 0x7fb86813ee30 2024-08-03T20:54:06.602 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:06.599+0000 7fb85f400640 1 -- 172.21.15.32:0/2792870100 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7fb860018020 con 0x7fb86813ee30 2024-08-03T20:54:06.602 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:06.601+0000 7fb85f400640 1 --2- 172.21.15.32:0/2792870100 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fb844077010 0x7fb844079520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:06.602 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:06.601+0000 7fb85f400640 1 -- 172.21.15.32:0/2792870100 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(72..72 src has 1..72) ==== 5881+0+0 (secure 0 0 0) 0x7fb860099150 con 0x7fb86813ee30 2024-08-03T20:54:06.602 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:06.601+0000 7fb866a00640 1 --2- 172.21.15.32:0/2792870100 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fb844077010 0x7fb844079520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:06.603 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:06.602+0000 7fb866a00640 1 --2- 172.21.15.32:0/2792870100 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fb844077010 0x7fb844079520 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7fb858033000 tx=0x7fb858038040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:54:06.606 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:06.606+0000 7fb85f400640 1 -- 172.21.15.32:0/2792870100 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7fb86001f020 con 0x7fb86813ee30 2024-08-03T20:54:06.848 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:06.848+0000 7fb86d800640 1 -- 172.21.15.32:0/2792870100 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) -- 0x7fb8680cf140 con 0x7fb86813ee30 2024-08-03T20:54:07.336 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:07.335+0000 7fb85f400640 1 -- 172.21.15.32:0/2792870100 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v83) ==== 104+0+0 (secure 0 0 0) 0x7fb86813f780 con 0x7fb86813ee30 2024-08-03T20:54:07.339 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:07.338+0000 7fb86d800640 1 -- 172.21.15.32:0/2792870100 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fb844077010 msgr2=0x7fb844079520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:07.339 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:07.338+0000 7fb86d800640 1 --2- 172.21.15.32:0/2792870100 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fb844077010 0x7fb844079520 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7fb858033000 tx=0x7fb858038040 comp rx=0 tx=0).stop 2024-08-03T20:54:07.339 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:07.338+0000 7fb86d800640 1 -- 172.21.15.32:0/2792870100 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb86813ee30 msgr2=0x7fb86813f240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:07.339 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:07.338+0000 7fb86d800640 1 --2- 172.21.15.32:0/2792870100 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb86813ee30 0x7fb86813f240 secure :-1 s=READY pgs=855 cs=0 l=1 rev1=1 crypto rx=0x7fb8680cdd20 tx=0x7fb8600019b0 comp rx=0 tx=0).stop 2024-08-03T20:54:07.339 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:07.338+0000 7fb86d800640 1 -- 172.21.15.32:0/2792870100 shutdown_connections 2024-08-03T20:54:07.339 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:07.338+0000 7fb86d800640 1 --2- 172.21.15.32:0/2792870100 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fb844077010 0x7fb844079520 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:07.339 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:07.338+0000 7fb86d800640 1 --2- 172.21.15.32:0/2792870100 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb8681438e0 0x7fb868159d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:07.339 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:07.338+0000 7fb86d800640 1 --2- 172.21.15.32:0/2792870100 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb86813ee30 0x7fb86813f240 unknown :-1 s=CLOSED pgs=855 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:07.339 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:07.338+0000 7fb86d800640 1 --2- 172.21.15.32:0/2792870100 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fb84c003a50 0x7fb86813bef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:07.339 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:07.338+0000 7fb86d800640 1 -- 172.21.15.32:0/2792870100 >> 172.21.15.32:0/2792870100 conn(0x7fb8680b1bd0 msgr2=0x7fb8680b4730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:54:07.340 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:07.338+0000 7fb86d800640 1 -- 172.21.15.32:0/2792870100 shutdown_connections 2024-08-03T20:54:07.340 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:07.338+0000 7fb86d800640 1 -- 172.21.15.32:0/2792870100 wait complete. 2024-08-03T20:54:07.363 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-08-03T20:54:07.602 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:07.597+0000 7f96e3000640 1 -- 172.21.15.32:0/2847503429 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f96dc15eb60 msgr2=0x7f96dc15c080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:07.602 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:07.597+0000 7f96e3000640 1 --2- 172.21.15.32:0/2847503429 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f96dc15eb60 0x7f96dc15c080 secure :-1 s=READY pgs=856 cs=0 l=1 rev1=1 crypto rx=0x7f96d0005fd0 tx=0x7f96d002a530 comp rx=0 tx=0).stop 2024-08-03T20:54:07.602 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:07.598+0000 7f96e3000640 1 -- 172.21.15.32:0/2847503429 shutdown_connections 2024-08-03T20:54:07.603 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:07.598+0000 7f96e3000640 1 --2- 172.21.15.32:0/2847503429 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f96dc0ce300 0x7f96dc0ad8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:07.604 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:07.598+0000 7f96e3000640 1 --2- 172.21.15.32:0/2847503429 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f96dc15eb60 0x7f96dc15c080 unknown :-1 s=CLOSED pgs=856 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:07.604 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:07.598+0000 7f96e3000640 1 -- 172.21.15.32:0/2847503429 >> 172.21.15.32:0/2847503429 conn(0x7f96dc0cfd80 msgr2=0x7f96dc0d0190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:54:07.604 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:07.598+0000 7f96e3000640 1 -- 172.21.15.32:0/2847503429 shutdown_connections 2024-08-03T20:54:07.604 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:07.598+0000 7f96e3000640 1 -- 172.21.15.32:0/2847503429 wait complete. 2024-08-03T20:54:07.605 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:07.600+0000 7f96e3000640 1 Processor -- start 2024-08-03T20:54:07.606 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:07.600+0000 7f96e3000640 1 -- start start 2024-08-03T20:54:07.609 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:07.601+0000 7f96e3000640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f96dc06e630 0x7f96dc158a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:07.609 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:07.601+0000 7f96e3000640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f96dc15eb60 0x7f96dc155460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:07.610 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:07.601+0000 7f96e3000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f96dc0b4040 0x7f96dc15dca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:07.610 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:07.601+0000 7f96e3000640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f96dc0ae2e0 con 0x7f96dc0b4040 2024-08-03T20:54:07.610 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:07.601+0000 7f96e3000640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f96dc0ae160 con 0x7f96dc06e630 2024-08-03T20:54:07.610 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:07.601+0000 7f96e3000640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7f96dc0ae460 con 0x7f96dc15eb60 2024-08-03T20:54:07.610 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:07.602+0000 7f96e2600640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f96dc0b4040 0x7f96dc15dca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:07.610 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:07.602+0000 7f96e2600640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f96dc0b4040 0x7f96dc15dca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:58156/0 (socket says 172.21.15.32:58156) 2024-08-03T20:54:07.610 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:07.602+0000 7f96e2600640 1 -- 172.21.15.32:0/3407899978 learned_addr learned my addr 172.21.15.32:0/3407899978 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:54:07.610 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:07.602+0000 7f96e1c00640 1 --2- 172.21.15.32:0/3407899978 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f96dc06e630 0x7f96dc158a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:07.610 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:07.602+0000 7f96e1200640 1 --2- 172.21.15.32:0/3407899978 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f96dc15eb60 0x7f96dc155460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:07.610 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:07.602+0000 7f96e2600640 1 -- 172.21.15.32:0/3407899978 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f96dc15eb60 msgr2=0x7f96dc155460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:07.610 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:07.602+0000 7f96e2600640 1 --2- 172.21.15.32:0/3407899978 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f96dc15eb60 0x7f96dc155460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:07.610 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:07.602+0000 7f96e2600640 1 -- 172.21.15.32:0/3407899978 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f96dc06e630 msgr2=0x7f96dc158a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:07.611 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:07.602+0000 7f96e2600640 1 --2- 172.21.15.32:0/3407899978 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f96dc06e630 0x7f96dc158a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:07.611 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:07.602+0000 7f96e2600640 1 -- 172.21.15.32:0/3407899978 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f96dc158fb0 con 0x7f96dc0b4040 2024-08-03T20:54:07.611 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:07.602+0000 7f96e1200640 1 --2- 172.21.15.32:0/3407899978 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f96dc15eb60 0x7f96dc155460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-03T20:54:07.611 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:07.603+0000 7f96e2600640 1 --2- 172.21.15.32:0/3407899978 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f96dc0b4040 0x7f96dc15dca0 secure :-1 s=READY pgs=857 cs=0 l=1 rev1=1 crypto rx=0x7f96d400ba10 tx=0x7f96d400bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:54:07.611 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:07.603+0000 7f96d9600640 1 -- 172.21.15.32:0/3407899978 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96d4019070 con 0x7f96dc0b4040 2024-08-03T20:54:07.611 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:07.603+0000 7f96d9600640 1 -- 172.21.15.32:0/3407899978 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f96d400cc50 con 0x7f96dc0b4040 2024-08-03T20:54:07.611 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:07.603+0000 7f96d9600640 1 -- 172.21.15.32:0/3407899978 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96d40146a0 con 0x7f96dc0b4040 2024-08-03T20:54:07.611 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:07.603+0000 7f96e3000640 1 -- 172.21.15.32:0/3407899978 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f96dc1f5900 con 0x7f96dc0b4040 2024-08-03T20:54:07.611 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:07.603+0000 7f96e3000640 1 -- 172.21.15.32:0/3407899978 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f96dc1f5ec0 con 0x7f96dc0b4040 2024-08-03T20:54:07.611 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:07.604+0000 7f96e3000640 1 -- 172.21.15.32:0/3407899978 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f96dc0ae970 con 0x7f96dc0b4040 2024-08-03T20:54:07.611 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:07.605+0000 7f96d9600640 1 -- 172.21.15.32:0/3407899978 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f96d401c070 con 0x7f96dc0b4040 2024-08-03T20:54:07.611 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:07.606+0000 7f96d9600640 1 --2- 172.21.15.32:0/3407899978 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f96bc077040 0x7f96bc079550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:07.611 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:07.607+0000 7f96d9600640 1 -- 172.21.15.32:0/3407899978 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(72..72 src has 1..72) ==== 5881+0+0 (secure 0 0 0) 0x7f96d4098660 con 0x7f96dc0b4040 2024-08-03T20:54:07.611 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:07.607+0000 7f96e1c00640 1 --2- 172.21.15.32:0/3407899978 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f96bc077040 0x7f96bc079550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:07.612 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:07.608+0000 7f96e1c00640 1 --2- 172.21.15.32:0/3407899978 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f96bc077040 0x7f96bc079550 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f96d0033000 tx=0x7f96d0004730 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:54:07.612 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:07.611+0000 7f96d9600640 1 -- 172.21.15.32:0/3407899978 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7f96d4061150 con 0x7f96dc0b4040 2024-08-03T20:54:07.731 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-08-03T20:54:07.736 DEBUG:teuthology.orchestra.run.smithi118:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-08-03T20:54:07.818 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:07.818+0000 7f96e3000640 1 -- 172.21.15.32:0/3407899978 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) -- 0x7f96dc0b5d00 con 0x7f96dc0b4040 2024-08-03T20:54:08.338 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:08.338+0000 7f96d9600640 1 -- 172.21.15.32:0/3407899978 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v73) ==== 181+0+0 (secure 0 0 0) 0x7f96d40656a0 con 0x7f96dc0b4040 2024-08-03T20:54:08.338 INFO:teuthology.orchestra.run.smithi032.stderr:pool 'cephfs_metadata' removed 2024-08-03T20:54:08.343 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:08.342+0000 7f96e3000640 1 -- 172.21.15.32:0/3407899978 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f96bc077040 msgr2=0x7f96bc079550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:08.343 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:08.342+0000 7f96e3000640 1 --2- 172.21.15.32:0/3407899978 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f96bc077040 0x7f96bc079550 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f96d0033000 tx=0x7f96d0004730 comp rx=0 tx=0).stop 2024-08-03T20:54:08.343 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:08.342+0000 7f96e3000640 1 -- 172.21.15.32:0/3407899978 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f96dc0b4040 msgr2=0x7f96dc15dca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:08.343 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:08.342+0000 7f96e3000640 1 --2- 172.21.15.32:0/3407899978 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f96dc0b4040 0x7f96dc15dca0 secure :-1 s=READY pgs=857 cs=0 l=1 rev1=1 crypto rx=0x7f96d400ba10 tx=0x7f96d400bed0 comp rx=0 tx=0).stop 2024-08-03T20:54:08.344 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:08.343+0000 7f96e3000640 1 -- 172.21.15.32:0/3407899978 shutdown_connections 2024-08-03T20:54:08.344 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:08.343+0000 7f96e3000640 1 --2- 172.21.15.32:0/3407899978 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f96bc077040 0x7f96bc079550 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:08.344 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:08.343+0000 7f96e3000640 1 --2- 172.21.15.32:0/3407899978 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f96dc0b4040 0x7f96dc15dca0 unknown :-1 s=CLOSED pgs=857 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:08.344 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:08.343+0000 7f96e3000640 1 --2- 172.21.15.32:0/3407899978 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f96dc15eb60 0x7f96dc155460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:08.344 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:08.343+0000 7f96e3000640 1 --2- 172.21.15.32:0/3407899978 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f96dc06e630 0x7f96dc158a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:08.344 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:08.343+0000 7f96e3000640 1 -- 172.21.15.32:0/3407899978 >> 172.21.15.32:0/3407899978 conn(0x7f96dc0cfd80 msgr2=0x7f96dc15b690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:54:08.344 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:08.343+0000 7f96e3000640 1 -- 172.21.15.32:0/3407899978 shutdown_connections 2024-08-03T20:54:08.344 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:08.343+0000 7f96e3000640 1 -- 172.21.15.32:0/3407899978 wait complete. 2024-08-03T20:54:08.374 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-08-03T20:54:08.650 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:08.649+0000 7f588b200640 1 -- 172.21.15.32:0/2366674978 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f588415b1b0 msgr2=0x7f588415b5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:08.650 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:08.649+0000 7f588b200640 1 --2- 172.21.15.32:0/2366674978 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f588415b1b0 0x7f588415b5c0 secure :-1 s=READY pgs=858 cs=0 l=1 rev1=1 crypto rx=0x7f5884072ba0 tx=0x7f588002aa10 comp rx=0 tx=0).stop 2024-08-03T20:54:08.650 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:08.649+0000 7f588b200640 1 -- 172.21.15.32:0/2366674978 shutdown_connections 2024-08-03T20:54:08.650 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:08.649+0000 7f588b200640 1 --2- 172.21.15.32:0/2366674978 >> v2:172.21.15.118:3300/0 conn(0x7f58840b4070 0x7f58840b4480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:08.650 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:08.649+0000 7f588b200640 1 --2- 172.21.15.32:0/2366674978 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f588415b1b0 0x7f588415b5c0 unknown :-1 s=CLOSED pgs=858 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:08.650 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:08.649+0000 7f588b200640 1 -- 172.21.15.32:0/2366674978 >> 172.21.15.32:0/2366674978 conn(0x7f58840b2b50 msgr2=0x7f58840d0500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:54:08.651 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:08.649+0000 7f588b200640 1 -- 172.21.15.32:0/2366674978 shutdown_connections 2024-08-03T20:54:08.651 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:08.649+0000 7f588b200640 1 -- 172.21.15.32:0/2366674978 wait complete. 2024-08-03T20:54:08.652 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:08.651+0000 7f588b200640 1 Processor -- start 2024-08-03T20:54:08.652 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:08.652+0000 7f588b200640 1 -- start start 2024-08-03T20:54:08.655 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:08.653+0000 7f588b200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f58840b4070 0x7f5884157700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:08.655 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:08.653+0000 7f588b200640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f58840d0f70 0x7f5884150b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:08.655 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:08.653+0000 7f588b200640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f588415b1b0 0x7f58840c5f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:08.655 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:08.653+0000 7f588b200640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f58840b4aa0 con 0x7f58840b4070 2024-08-03T20:54:08.655 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:08.653+0000 7f588b200640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f58840ccb30 con 0x7f588415b1b0 2024-08-03T20:54:08.655 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:08.653+0000 7f588b200640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7f588415bb00 con 0x7f58840d0f70 2024-08-03T20:54:08.655 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:08.653+0000 7f5889e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f58840b4070 0x7f5884157700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:08.655 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:08.653+0000 7f5889e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f58840b4070 0x7f5884157700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:58164/0 (socket says 172.21.15.32:58164) 2024-08-03T20:54:08.655 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:08.653+0000 7f5889e00640 1 -- 172.21.15.32:0/1608424975 learned_addr learned my addr 172.21.15.32:0/1608424975 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:54:08.655 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:08.653+0000 7f588a800640 1 --2- 172.21.15.32:0/1608424975 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f588415b1b0 0x7f58840c5f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:08.656 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:08.653+0000 7f5889400640 1 --2- 172.21.15.32:0/1608424975 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f58840d0f70 0x7f5884150b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:08.656 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:08.654+0000 7f5889e00640 1 -- 172.21.15.32:0/1608424975 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f58840d0f70 msgr2=0x7f5884150b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:08.656 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:08.654+0000 7f5889e00640 1 --2- 172.21.15.32:0/1608424975 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f58840d0f70 0x7f5884150b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:08.656 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:08.654+0000 7f5889e00640 1 -- 172.21.15.32:0/1608424975 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f588415b1b0 msgr2=0x7f58840c5f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:08.656 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:08.654+0000 7f5889e00640 1 --2- 172.21.15.32:0/1608424975 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f588415b1b0 0x7f58840c5f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:08.656 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:08.654+0000 7f5889e00640 1 -- 172.21.15.32:0/1608424975 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f588415aa10 con 0x7f58840b4070 2024-08-03T20:54:08.656 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:08.654+0000 7f588a800640 1 --2- 172.21.15.32:0/1608424975 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f588415b1b0 0x7f58840c5f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-03T20:54:08.656 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:08.654+0000 7f5889e00640 1 --2- 172.21.15.32:0/1608424975 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f58840b4070 0x7f5884157700 secure :-1 s=READY pgs=859 cs=0 l=1 rev1=1 crypto rx=0x7f587800b420 tx=0x7f587800b8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:54:08.656 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:08.654+0000 7f587e000640 1 -- 172.21.15.32:0/1608424975 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5878011830 con 0x7f58840b4070 2024-08-03T20:54:08.656 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:08.654+0000 7f587e000640 1 -- 172.21.15.32:0/1608424975 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f5878016070 con 0x7f58840b4070 2024-08-03T20:54:08.656 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:08.654+0000 7f587e000640 1 -- 172.21.15.32:0/1608424975 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5878011d60 con 0x7f58840b4070 2024-08-03T20:54:08.656 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:08.654+0000 7f588b200640 1 -- 172.21.15.32:0/1608424975 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f588415a730 con 0x7f58840b4070 2024-08-03T20:54:08.656 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:08.654+0000 7f588b200640 1 -- 172.21.15.32:0/1608424975 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f58840c5700 con 0x7f58840b4070 2024-08-03T20:54:08.657 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:08.656+0000 7f588b200640 1 -- 172.21.15.32:0/1608424975 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f588415bf70 con 0x7f58840b4070 2024-08-03T20:54:08.657 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:08.657+0000 7f587e000640 1 -- 172.21.15.32:0/1608424975 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f587801b460 con 0x7f58840b4070 2024-08-03T20:54:08.658 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:08.658+0000 7f587e000640 1 --2- 172.21.15.32:0/1608424975 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f5868077010 0x7f5868079520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:08.658 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:08.658+0000 7f587e000640 1 -- 172.21.15.32:0/1608424975 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(73..73 src has 1..73) ==== 5443+0+0 (secure 0 0 0) 0x7f5878098390 con 0x7f58840b4070 2024-08-03T20:54:08.659 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:08.659+0000 7f5889400640 1 --2- 172.21.15.32:0/1608424975 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f5868077010 0x7f5868079520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:08.659 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:08.659+0000 7f5889400640 1 --2- 172.21.15.32:0/1608424975 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f5868077010 0x7f5868079520 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f5874005fd0 tx=0x7f5874005cf0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:54:08.663 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:08.663+0000 7f587e000640 1 -- 172.21.15.32:0/1608424975 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7f5878020020 con 0x7f58840b4070 2024-08-03T20:54:08.868 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:08.868+0000 7f588b200640 1 -- 172.21.15.32:0/1608424975 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) -- 0x7f58840d02e0 con 0x7f58840b4070 2024-08-03T20:54:09.345 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:09.344+0000 7f587e000640 1 -- 172.21.15.32:0/1608424975 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v74) ==== 169+0+0 (secure 0 0 0) 0x7f5878061680 con 0x7f58840b4070 2024-08-03T20:54:09.345 INFO:teuthology.orchestra.run.smithi032.stderr:pool 'cephfs_data' removed 2024-08-03T20:54:09.348 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:09.347+0000 7f588b200640 1 -- 172.21.15.32:0/1608424975 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f5868077010 msgr2=0x7f5868079520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:09.348 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:09.347+0000 7f588b200640 1 --2- 172.21.15.32:0/1608424975 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f5868077010 0x7f5868079520 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f5874005fd0 tx=0x7f5874005cf0 comp rx=0 tx=0).stop 2024-08-03T20:54:09.349 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:09.347+0000 7f588b200640 1 -- 172.21.15.32:0/1608424975 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f58840b4070 msgr2=0x7f5884157700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:09.349 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:09.347+0000 7f588b200640 1 --2- 172.21.15.32:0/1608424975 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f58840b4070 0x7f5884157700 secure :-1 s=READY pgs=859 cs=0 l=1 rev1=1 crypto rx=0x7f587800b420 tx=0x7f587800b8e0 comp rx=0 tx=0).stop 2024-08-03T20:54:09.349 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:09.348+0000 7f588b200640 1 -- 172.21.15.32:0/1608424975 shutdown_connections 2024-08-03T20:54:09.349 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:09.348+0000 7f588b200640 1 --2- 172.21.15.32:0/1608424975 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f5868077010 0x7f5868079520 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:09.349 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:09.348+0000 7f588b200640 1 --2- 172.21.15.32:0/1608424975 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f588415b1b0 0x7f58840c5f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:09.349 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:09.348+0000 7f588b200640 1 --2- 172.21.15.32:0/1608424975 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f58840d0f70 0x7f5884150b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:09.349 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:09.348+0000 7f588b200640 1 --2- 172.21.15.32:0/1608424975 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f58840b4070 0x7f5884157700 unknown :-1 s=CLOSED pgs=859 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:09.349 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:09.348+0000 7f588b200640 1 -- 172.21.15.32:0/1608424975 >> 172.21.15.32:0/1608424975 conn(0x7f58840b2b50 msgr2=0x7f588406f680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:54:09.349 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:09.348+0000 7f588b200640 1 -- 172.21.15.32:0/1608424975 shutdown_connections 2024-08-03T20:54:09.349 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:09.348+0000 7f588b200640 1 -- 172.21.15.32:0/1608424975 wait complete. 2024-08-03T20:54:09.373 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config reset 27 2024-08-03T20:54:09.605 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:09.604+0000 7fb837a00640 1 -- 172.21.15.32:0/1977231748 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb818004800 msgr2=0x7fb818004c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:09.606 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:09.604+0000 7fb837a00640 1 --2- 172.21.15.32:0/1977231748 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb818004800 0x7fb818004c10 secure :-1 s=READY pgs=861 cs=0 l=1 rev1=1 crypto rx=0x7fb81c009d00 tx=0x7fb818005830 comp rx=0 tx=0).stop 2024-08-03T20:54:09.606 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:09.604+0000 7fb835a00640 1 -- 172.21.15.32:0/1977231748 reap_dead start 2024-08-03T20:54:09.606 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:09.605+0000 7fb837a00640 1 -- 172.21.15.32:0/1977231748 shutdown_connections 2024-08-03T20:54:09.606 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:09.605+0000 7fb837a00640 1 -- 172.21.15.32:0/1977231748 >> 172.21.15.32:0/1977231748 conn(0x7fb8300b1bd0 msgr2=0x7fb8300b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:54:09.606 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:09.605+0000 7fb837a00640 1 -- 172.21.15.32:0/1977231748 shutdown_connections 2024-08-03T20:54:09.606 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:09.605+0000 7fb837a00640 1 -- 172.21.15.32:0/1977231748 wait complete. 2024-08-03T20:54:09.607 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:09.607+0000 7fb837a00640 1 Processor -- start 2024-08-03T20:54:09.607 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:09.607+0000 7fb837a00640 1 -- start start 2024-08-03T20:54:09.610 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:09.608+0000 7fb837a00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb818004800 0x7fb8300733e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:09.611 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:09.608+0000 7fb837a00640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fb830073b80 0x7fb83014a550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:09.611 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:09.608+0000 7fb837a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb830156da0 0x7fb8301571b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:09.611 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:09.608+0000 7fb837a00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fb8301523f0 con 0x7fb830156da0 2024-08-03T20:54:09.611 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:09.609+0000 7fb837a00640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7fb8300cf2a0 con 0x7fb818004800 2024-08-03T20:54:09.611 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:09.609+0000 7fb837a00640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7fb830152570 con 0x7fb830073b80 2024-08-03T20:54:09.611 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:09.609+0000 7fb835a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb830156da0 0x7fb8301571b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:09.611 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:09.609+0000 7fb835a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb830156da0 0x7fb8301571b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:58184/0 (socket says 172.21.15.32:58184) 2024-08-03T20:54:09.611 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:09.609+0000 7fb835a00640 1 -- 172.21.15.32:0/3104693409 learned_addr learned my addr 172.21.15.32:0/3104693409 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:54:09.611 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:09.609+0000 7fb835000640 1 --2- 172.21.15.32:0/3104693409 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb818004800 0x7fb8300733e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:09.611 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:09.609+0000 7fb82fe00640 1 --2- 172.21.15.32:0/3104693409 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fb830073b80 0x7fb83014a550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:09.611 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:09.609+0000 7fb835a00640 1 -- 172.21.15.32:0/3104693409 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fb830073b80 msgr2=0x7fb83014a550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:09.611 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:09.609+0000 7fb835a00640 1 --2- 172.21.15.32:0/3104693409 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fb830073b80 0x7fb83014a550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:09.612 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:09.609+0000 7fb835a00640 1 -- 172.21.15.32:0/3104693409 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb818004800 msgr2=0x7fb8300733e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:09.612 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:09.609+0000 7fb835a00640 1 --2- 172.21.15.32:0/3104693409 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb818004800 0x7fb8300733e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:09.612 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:09.609+0000 7fb835a00640 1 -- 172.21.15.32:0/3104693409 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb8301576f0 con 0x7fb830156da0 2024-08-03T20:54:09.612 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:09.609+0000 7fb835000640 1 --2- 172.21.15.32:0/3104693409 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb818004800 0x7fb8300733e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-03T20:54:09.612 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:09.609+0000 7fb82fe00640 1 --2- 172.21.15.32:0/3104693409 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fb830073b80 0x7fb83014a550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-03T20:54:09.612 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:09.610+0000 7fb835a00640 1 --2- 172.21.15.32:0/3104693409 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb830156da0 0x7fb8301571b0 secure :-1 s=READY pgs=862 cs=0 l=1 rev1=1 crypto rx=0x7fb8300cd110 tx=0x7fb828006720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:54:09.612 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:09.610+0000 7fb82cc00640 1 -- 172.21.15.32:0/3104693409 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb82800fa80 con 0x7fb830156da0 2024-08-03T20:54:09.612 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:09.610+0000 7fb82cc00640 1 -- 172.21.15.32:0/3104693409 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fb82800fc60 con 0x7fb830156da0 2024-08-03T20:54:09.612 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:09.610+0000 7fb82cc00640 1 -- 172.21.15.32:0/3104693409 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb828011690 con 0x7fb830156da0 2024-08-03T20:54:09.612 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:09.610+0000 7fb837a00640 1 -- 172.21.15.32:0/3104693409 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb830074540 con 0x7fb830156da0 2024-08-03T20:54:09.612 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:09.610+0000 7fb837a00640 1 -- 172.21.15.32:0/3104693409 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb830072be0 con 0x7fb830156da0 2024-08-03T20:54:09.613 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:09.611+0000 7fb837a00640 1 -- 172.21.15.32:0/3104693409 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb83006ec00 con 0x7fb830156da0 2024-08-03T20:54:09.613 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:09.612+0000 7fb82cc00640 1 -- 172.21.15.32:0/3104693409 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7fb828018020 con 0x7fb830156da0 2024-08-03T20:54:09.614 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:09.613+0000 7fb82cc00640 1 --2- 172.21.15.32:0/3104693409 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fb810077010 0x7fb810079520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:09.614 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:09.613+0000 7fb82cc00640 1 -- 172.21.15.32:0/3104693409 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(74..74 src has 1..74) ==== 5029+0+0 (secure 0 0 0) 0x7fb828098d10 con 0x7fb830156da0 2024-08-03T20:54:09.614 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:09.614+0000 7fb835000640 1 --2- 172.21.15.32:0/3104693409 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fb810077010 0x7fb810079520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:09.614 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:09.614+0000 7fb835000640 1 --2- 172.21.15.32:0/3104693409 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fb810077010 0x7fb810079520 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7fb824009a30 tx=0x7fb824005f90 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:54:09.618 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:09.618+0000 7fb82cc00640 1 -- 172.21.15.32:0/3104693409 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7fb82809e020 con 0x7fb830156da0 2024-08-03T20:54:09.840 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:09.839+0000 7fb837a00640 1 -- 172.21.15.32:0/3104693409 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "config reset", "num": 27} v 0) -- 0x7fb83015ef00 con 0x7fb830156da0 2024-08-03T20:54:09.847 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:09.844+0000 7fb82cc00640 1 -- 172.21.15.32:0/3104693409 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "config reset", "num": 27}]=0 v42) ==== 71+0+0 (secure 0 0 0) 0x7fb828015030 con 0x7fb830156da0 2024-08-03T20:54:09.847 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:09.846+0000 7fb837a00640 1 -- 172.21.15.32:0/3104693409 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fb810077010 msgr2=0x7fb810079520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:09.847 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:09.846+0000 7fb837a00640 1 --2- 172.21.15.32:0/3104693409 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fb810077010 0x7fb810079520 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7fb824009a30 tx=0x7fb824005f90 comp rx=0 tx=0).stop 2024-08-03T20:54:09.847 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:09.846+0000 7fb837a00640 1 -- 172.21.15.32:0/3104693409 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb830156da0 msgr2=0x7fb8301571b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:09.847 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:09.846+0000 7fb837a00640 1 --2- 172.21.15.32:0/3104693409 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb830156da0 0x7fb8301571b0 secure :-1 s=READY pgs=862 cs=0 l=1 rev1=1 crypto rx=0x7fb8300cd110 tx=0x7fb828006720 comp rx=0 tx=0).stop 2024-08-03T20:54:09.847 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:09.846+0000 7fb837a00640 1 -- 172.21.15.32:0/3104693409 shutdown_connections 2024-08-03T20:54:09.847 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:09.846+0000 7fb837a00640 1 --2- 172.21.15.32:0/3104693409 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fb810077010 0x7fb810079520 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:09.847 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:09.846+0000 7fb837a00640 1 --2- 172.21.15.32:0/3104693409 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb830156da0 0x7fb8301571b0 unknown :-1 s=CLOSED pgs=862 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:09.847 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:09.846+0000 7fb837a00640 1 --2- 172.21.15.32:0/3104693409 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fb830073b80 0x7fb83014a550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:09.848 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:09.846+0000 7fb837a00640 1 --2- 172.21.15.32:0/3104693409 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb818004800 0x7fb8300733e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:09.848 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:09.846+0000 7fb837a00640 1 -- 172.21.15.32:0/3104693409 >> 172.21.15.32:0/3104693409 conn(0x7fb8300b1bd0 msgr2=0x7fb83015e420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:54:09.848 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:09.846+0000 7fb837a00640 1 -- 172.21.15.32:0/3104693409 shutdown_connections 2024-08-03T20:54:09.848 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:09.846+0000 7fb837a00640 1 -- 172.21.15.32:0/3104693409 wait complete. 2024-08-03T20:54:09.873 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_damage.TestDamage.test_dentry_first_preflush' 2024-08-03T20:54:10.101 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:10.100+0000 7f2e8f000640 1 -- 172.21.15.32:0/2629706015 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f2e8806fb30 msgr2=0x7f2e880ae590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:10.102 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:10.101+0000 7f2e8f000640 1 --2- 172.21.15.32:0/2629706015 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f2e8806fb30 0x7f2e880ae590 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f2e80005fd0 tx=0x7f2e8002a8d0 comp rx=0 tx=0).stop 2024-08-03T20:54:10.102 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:10.101+0000 7f2e8f000640 1 -- 172.21.15.32:0/2629706015 shutdown_connections 2024-08-03T20:54:10.102 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:10.101+0000 7f2e8f000640 1 --2- 172.21.15.32:0/2629706015 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f2e8806fb30 0x7f2e880ae590 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:10.102 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:10.101+0000 7f2e8f000640 1 -- 172.21.15.32:0/2629706015 >> 172.21.15.32:0/2629706015 conn(0x7f2e880b1bd0 msgr2=0x7f2e880b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:54:10.102 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:10.101+0000 7f2e8f000640 1 -- 172.21.15.32:0/2629706015 shutdown_connections 2024-08-03T20:54:10.102 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:10.101+0000 7f2e8f000640 1 -- 172.21.15.32:0/2629706015 wait complete. 2024-08-03T20:54:10.103 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:10.103+0000 7f2e8f000640 1 Processor -- start 2024-08-03T20:54:10.104 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:10.103+0000 7f2e8f000640 1 -- start start 2024-08-03T20:54:10.106 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:10.105+0000 7f2e8f000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2e8806f720 0x7f2e881f9890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:10.106 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:10.105+0000 7f2e8f000640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f2e8806fb30 0x7f2e88152660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:10.106 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:10.105+0000 7f2e8f000640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f2e880b40b0 0x7f2e880cc9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:10.106 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:10.105+0000 7f2e8f000640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f2e880aee60 con 0x7f2e8806f720 2024-08-03T20:54:10.106 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:10.105+0000 7f2e8f000640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f2e880aece0 con 0x7f2e8806fb30 2024-08-03T20:54:10.106 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:10.105+0000 7f2e8f000640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7f2e880aefe0 con 0x7f2e880b40b0 2024-08-03T20:54:10.106 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:10.105+0000 7f2e87e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2e8806f720 0x7f2e881f9890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:10.106 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:10.105+0000 7f2e87e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2e8806f720 0x7f2e881f9890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:43654/0 (socket says 172.21.15.32:43654) 2024-08-03T20:54:10.107 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:10.105+0000 7f2e87e00640 1 -- 172.21.15.32:0/1401462274 learned_addr learned my addr 172.21.15.32:0/1401462274 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:54:10.107 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:10.105+0000 7f2e7fe00640 1 --2- 172.21.15.32:0/1401462274 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f2e8806fb30 0x7f2e88152660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:10.107 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:10.105+0000 7f2e8d000640 1 --2- 172.21.15.32:0/1401462274 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f2e880b40b0 0x7f2e880cc9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:10.107 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:10.105+0000 7f2e87e00640 1 -- 172.21.15.32:0/1401462274 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f2e880b40b0 msgr2=0x7f2e880cc9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:10.107 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:10.105+0000 7f2e87e00640 1 --2- 172.21.15.32:0/1401462274 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f2e880b40b0 0x7f2e880cc9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:10.107 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:10.105+0000 7f2e87e00640 1 -- 172.21.15.32:0/1401462274 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f2e8806fb30 msgr2=0x7f2e88152660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:10.107 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:10.105+0000 7f2e87e00640 1 --2- 172.21.15.32:0/1401462274 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f2e8806fb30 0x7f2e88152660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:10.107 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:10.105+0000 7f2e87e00640 1 -- 172.21.15.32:0/1401462274 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2e880cf810 con 0x7f2e8806f720 2024-08-03T20:54:10.107 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:10.106+0000 7f2e87e00640 1 --2- 172.21.15.32:0/1401462274 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2e8806f720 0x7f2e881f9890 secure :-1 s=READY pgs=864 cs=0 l=1 rev1=1 crypto rx=0x7f2e7800c4b0 tx=0x7f2e7800c970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:54:10.108 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:10.106+0000 7f2e84c00640 1 -- 172.21.15.32:0/1401462274 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e78016020 con 0x7f2e8806f720 2024-08-03T20:54:10.108 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:10.106+0000 7f2e84c00640 1 -- 172.21.15.32:0/1401462274 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f2e78012920 con 0x7f2e8806f720 2024-08-03T20:54:10.108 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:10.106+0000 7f2e84c00640 1 -- 172.21.15.32:0/1401462274 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e78012d70 con 0x7f2e8806f720 2024-08-03T20:54:10.108 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:10.106+0000 7f2e8f000640 1 -- 172.21.15.32:0/1401462274 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2e880cfaf0 con 0x7f2e8806f720 2024-08-03T20:54:10.108 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:10.106+0000 7f2e8f000640 1 -- 172.21.15.32:0/1401462274 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2e881f9090 con 0x7f2e8806f720 2024-08-03T20:54:10.108 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:10.107+0000 7f2e8f000640 1 -- 172.21.15.32:0/1401462274 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2e880b39a0 con 0x7f2e8806f720 2024-08-03T20:54:10.109 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:10.108+0000 7f2e84c00640 1 -- 172.21.15.32:0/1401462274 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f2e78019020 con 0x7f2e8806f720 2024-08-03T20:54:10.110 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:10.109+0000 7f2e84c00640 1 --2- 172.21.15.32:0/1401462274 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f2e64077010 0x7f2e64079520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:10.110 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:10.110+0000 7f2e84c00640 1 -- 172.21.15.32:0/1401462274 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(74..74 src has 1..74) ==== 5029+0+0 (secure 0 0 0) 0x7f2e78098480 con 0x7f2e8806f720 2024-08-03T20:54:10.110 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:10.110+0000 7f2e7fe00640 1 --2- 172.21.15.32:0/1401462274 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f2e64077010 0x7f2e64079520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:10.111 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:10.111+0000 7f2e7fe00640 1 --2- 172.21.15.32:0/1401462274 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f2e64077010 0x7f2e64079520 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f2e88074d60 tx=0x7f2e70009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:54:10.114 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:10.114+0000 7f2e84c00640 1 -- 172.21.15.32:0/1401462274 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7f2e78020030 con 0x7f2e8806f720 2024-08-03T20:54:10.410 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:10.409+0000 7f2e8f000640 1 -- 172.21.15.32:0/1401462274 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_damage.TestDamage.test_dentry_first_preflush"]} v 0) -- 0x7f2e88074a90 con 0x7f2e8806f720 2024-08-03T20:54:11.349 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:11.347+0000 7f2e84c00640 1 -- 172.21.15.32:0/1401462274 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_damage.TestDamage.test_dentry_first_preflush"]}]=0 v323) ==== 141+0+0 (secure 0 0 0) 0x7f2e78061910 con 0x7f2e8806f720 2024-08-03T20:54:11.352 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:11.350+0000 7f2e8f000640 1 -- 172.21.15.32:0/1401462274 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f2e64077010 msgr2=0x7f2e64079520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:11.352 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:11.350+0000 7f2e8f000640 1 --2- 172.21.15.32:0/1401462274 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f2e64077010 0x7f2e64079520 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f2e88074d60 tx=0x7f2e70009040 comp rx=0 tx=0).stop 2024-08-03T20:54:11.353 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:11.350+0000 7f2e8f000640 1 -- 172.21.15.32:0/1401462274 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2e8806f720 msgr2=0x7f2e881f9890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:11.353 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:11.350+0000 7f2e8f000640 1 --2- 172.21.15.32:0/1401462274 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2e8806f720 0x7f2e881f9890 secure :-1 s=READY pgs=864 cs=0 l=1 rev1=1 crypto rx=0x7f2e7800c4b0 tx=0x7f2e7800c970 comp rx=0 tx=0).stop 2024-08-03T20:54:11.353 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:11.351+0000 7f2e8f000640 1 -- 172.21.15.32:0/1401462274 shutdown_connections 2024-08-03T20:54:11.353 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:11.351+0000 7f2e8f000640 1 --2- 172.21.15.32:0/1401462274 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f2e64077010 0x7f2e64079520 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:11.353 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:11.351+0000 7f2e8f000640 1 --2- 172.21.15.32:0/1401462274 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f2e8806fb30 0x7f2e88152660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:11.353 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:11.351+0000 7f2e8f000640 1 --2- 172.21.15.32:0/1401462274 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f2e880b40b0 0x7f2e880cc9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:11.353 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:11.351+0000 7f2e8f000640 1 --2- 172.21.15.32:0/1401462274 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2e8806f720 0x7f2e881f9890 unknown :-1 s=CLOSED pgs=864 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:11.353 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:11.351+0000 7f2e8f000640 1 -- 172.21.15.32:0/1401462274 >> 172.21.15.32:0/1401462274 conn(0x7f2e880b1bd0 msgr2=0x7f2e880b2ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:54:11.353 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:11.351+0000 7f2e8f000640 1 -- 172.21.15.32:0/1401462274 shutdown_connections 2024-08-03T20:54:11.353 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:11.351+0000 7f2e8f000640 1 -- 172.21.15.32:0/1401462274 wait complete. 2024-08-03T20:54:11.378 INFO:tasks.cephfs_test_runner:That the MDS won't write a dentry with new damage to CDentry::first ... ok 2024-08-03T20:54:11.379 INFO:tasks.cephfs_test_runner:Starting test: test_object_deletion (tasks.cephfs.test_damage.TestDamage) That the MDS has a clean 'damaged' response to loss of any single metadata object 2024-08-03T20:54:11.379 INFO:tasks.cephfs_test_runner:test_object_deletion (tasks.cephfs.test_damage.TestDamage) 2024-08-03T20:54:11.379 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-08-03T20:54:11.613 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:11.611+0000 7f879de00640 1 -- 172.21.15.32:0/2239084702 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f877c004800 msgr2=0x7f877c004c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:11.614 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:11.611+0000 7f879de00640 1 --2- 172.21.15.32:0/2239084702 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f877c004800 0x7f877c004c10 secure :-1 s=READY pgs=866 cs=0 l=1 rev1=1 crypto rx=0x7f8798072880 tx=0x7f877c005830 comp rx=0 tx=0).stop 2024-08-03T20:54:11.614 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:11.612+0000 7f879de00640 1 -- 172.21.15.32:0/2239084702 shutdown_connections 2024-08-03T20:54:11.614 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:11.612+0000 7f879de00640 1 --2- 172.21.15.32:0/2239084702 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f877c004800 0x7f877c004c10 unknown :-1 s=CLOSED pgs=866 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:11.614 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:11.612+0000 7f879de00640 1 -- 172.21.15.32:0/2239084702 >> 172.21.15.32:0/2239084702 conn(0x7f87980b1bd0 msgr2=0x7f87980b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:54:11.614 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:11.612+0000 7f879de00640 1 -- 172.21.15.32:0/2239084702 shutdown_connections 2024-08-03T20:54:11.614 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:11.612+0000 7f879de00640 1 -- 172.21.15.32:0/2239084702 wait complete. 2024-08-03T20:54:11.615 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:11.614+0000 7f879de00640 1 Processor -- start 2024-08-03T20:54:11.615 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:11.614+0000 7f879de00640 1 -- start start 2024-08-03T20:54:11.618 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:11.616+0000 7f879de00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f877c004800 0x7f87980aede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:11.618 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:11.616+0000 7f879de00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f879813b930 0x7f8798155520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:11.619 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:11.616+0000 7f879de00640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f87981570f0 0x7f8798157500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:11.619 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:11.616+0000 7f879de00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f87980cf9c0 con 0x7f877c004800 2024-08-03T20:54:11.619 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:11.616+0000 7f879de00640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f87980cf840 con 0x7f879813b930 2024-08-03T20:54:11.619 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:11.616+0000 7f879de00640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7f8798073e60 con 0x7f87981570f0 2024-08-03T20:54:11.619 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:11.616+0000 7f8796a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f877c004800 0x7f87980aede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:11.619 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:11.616+0000 7f8796a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f877c004800 0x7f87980aede0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:43676/0 (socket says 172.21.15.32:43676) 2024-08-03T20:54:11.619 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:11.616+0000 7f8796a00640 1 -- 172.21.15.32:0/2883211831 learned_addr learned my addr 172.21.15.32:0/2883211831 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:54:11.619 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:11.616+0000 7f8796a00640 1 -- 172.21.15.32:0/2883211831 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f87981570f0 msgr2=0x7f8798157500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:11.619 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:11.616+0000 7f8797400640 1 --2- 172.21.15.32:0/2883211831 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f87981570f0 0x7f8798157500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:11.619 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:11.616+0000 7f878e000640 1 --2- 172.21.15.32:0/2883211831 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f879813b930 0x7f8798155520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:11.619 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:11.616+0000 7f8796a00640 1 --2- 172.21.15.32:0/2883211831 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f87981570f0 0x7f8798157500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:11.620 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:11.616+0000 7f8796a00640 1 -- 172.21.15.32:0/2883211831 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f879813b930 msgr2=0x7f8798155520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:11.620 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:11.616+0000 7f8796a00640 1 --2- 172.21.15.32:0/2883211831 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f879813b930 0x7f8798155520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:11.620 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:11.616+0000 7f8796a00640 1 -- 172.21.15.32:0/2883211831 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8798157a40 con 0x7f877c004800 2024-08-03T20:54:11.620 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:11.616+0000 7f8797400640 1 --2- 172.21.15.32:0/2883211831 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f87981570f0 0x7f8798157500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-03T20:54:11.620 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:11.616+0000 7f878e000640 1 --2- 172.21.15.32:0/2883211831 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f879813b930 0x7f8798155520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-03T20:54:11.620 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:11.617+0000 7f8796a00640 1 --2- 172.21.15.32:0/2883211831 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f877c004800 0x7f87980aede0 secure :-1 s=READY pgs=867 cs=0 l=1 rev1=1 crypto rx=0x7f8788003cd0 tx=0x7f8788001800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:54:11.620 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:11.617+0000 7f878fe00640 1 -- 172.21.15.32:0/2883211831 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8788011640 con 0x7f877c004800 2024-08-03T20:54:11.620 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:11.617+0000 7f878fe00640 1 -- 172.21.15.32:0/2883211831 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f8788016020 con 0x7f877c004800 2024-08-03T20:54:11.620 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:11.617+0000 7f878fe00640 1 -- 172.21.15.32:0/2883211831 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8788011a80 con 0x7f877c004800 2024-08-03T20:54:11.620 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:11.617+0000 7f879de00640 1 -- 172.21.15.32:0/2883211831 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f87981f9a30 con 0x7f877c004800 2024-08-03T20:54:11.620 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:11.617+0000 7f879de00640 1 -- 172.21.15.32:0/2883211831 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f879813c5e0 con 0x7f877c004800 2024-08-03T20:54:11.621 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:11.618+0000 7f879de00640 1 -- 172.21.15.32:0/2883211831 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f87980742e0 con 0x7f877c004800 2024-08-03T20:54:11.621 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:11.619+0000 7f878fe00640 1 -- 172.21.15.32:0/2883211831 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f8788011c20 con 0x7f877c004800 2024-08-03T20:54:11.622 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:11.620+0000 7f878fe00640 1 --2- 172.21.15.32:0/2883211831 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f8774077010 0x7f8774079520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:11.622 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:11.621+0000 7f878fe00640 1 -- 172.21.15.32:0/2883211831 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(74..74 src has 1..74) ==== 5029+0+0 (secure 0 0 0) 0x7f87880991a0 con 0x7f877c004800 2024-08-03T20:54:11.622 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:11.621+0000 7f878e000640 1 --2- 172.21.15.32:0/2883211831 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f8774077010 0x7f8774079520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:11.623 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:11.622+0000 7f878e000640 1 --2- 172.21.15.32:0/2883211831 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f8774077010 0x7f8774079520 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f8780033000 tx=0x7f878002d000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:54:11.626 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:11.625+0000 7f878fe00640 1 -- 172.21.15.32:0/2883211831 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7f8788020020 con 0x7f877c004800 2024-08-03T20:54:11.832 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:11.830+0000 7f879de00640 1 -- 172.21.15.32:0/2883211831 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f87980cd7f0 con 0x7f877c004800 2024-08-03T20:54:11.833 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:11.831+0000 7f878fe00640 1 -- 172.21.15.32:0/2883211831 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v74) ==== 74+0+13091 (secure 0 0 0) 0x7f8798157a40 con 0x7f877c004800 2024-08-03T20:54:11.833 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-03T20:54:11.833 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":74,"fsid":"40ab141a-01d7-49b9-ae3c-2edec9876e95","created":"2024-08-03T20:47:14.957362+0000","modified":"2024-08-03T20:54:09.338709+0000","last_up_change":"2024-08-03T20:47:28.874451+0000","last_in_change":"2024-08-03T20:47:20.079442+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-08-03T20:47:28.261300+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5a83e759-9035-438b-889a-0a6af1f0a0d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6816","nonce":3066156116},{"type":"v1","addr":"172.21.15.32:6817","nonce":3066156116}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6818","nonce":3066156116},{"type":"v1","addr":"172.21.15.32:6819","nonce":3066156116}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6822","nonce":3066156116},{"type":"v1","addr":"172.21.15.32:6823","nonce":3066156116}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6820","nonce":3066156116},{"type":"v1","addr":"172.21.15.32:6821","nonce":3066156116}]},"public_addr":"172.21.15.32:6817/3066156116","cluster_addr":"172.21.15.32:6819/3066156116","heartbeat_back_addr":"172.21.15.32:6823/3066156116","heartbeat_front_addr":"172.21.15.32:6821/3066156116","state":["exists","up"]},{"osd":1,"uuid":"d7c5c210-da4f-49f9-b9e7-1dfb02daa4bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6824","nonce":1551241759},{"type":"v1","addr":"172.21.15.32:6825","nonce":1551241759}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6826","nonce":1551241759},{"type":"v1","addr":"172.21.15.32:6827","nonce":1551241759}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6830","nonce":1551241759},{"type":"v1","addr":"172.21.15.32:6831","nonce":1551241759}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6828","nonce":1551241759},{"type":"v1","addr":"172.21.15.32:6829","nonce":1551241759}]},"public_addr":"172.21.15.32:6825/1551241759","cluster_addr":"172.21.15.32:6827/1551241759","heartbeat_back_addr":"172.21.15.32:6831/1551241759","heartbeat_front_addr":"172.21.15.32:6829/1551241759","state":["exists","up"]},{"osd":2,"uuid":"57f2e6de-a201-45c2-8981-8e56a4875ebd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6808","nonce":3806708944},{"type":"v1","addr":"172.21.15.32:6809","nonce":3806708944}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6810","nonce":3806708944},{"type":"v1","addr":"172.21.15.32:6811","nonce":3806708944}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6814","nonce":3806708944},{"type":"v1","addr":"172.21.15.32:6815","nonce":3806708944}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6812","nonce":3806708944},{"type":"v1","addr":"172.21.15.32:6813","nonce":3806708944}]},"public_addr":"172.21.15.32:6809/3806708944","cluster_addr":"172.21.15.32:6811/3806708944","heartbeat_back_addr":"172.21.15.32:6815/3806708944","heartbeat_front_addr":"172.21.15.32:6813/3806708944","state":["exists","up"]},{"osd":3,"uuid":"6439a280-da45-4148-8130-f16323935ff3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6800","nonce":9088303},{"type":"v1","addr":"172.21.15.32:6801","nonce":9088303}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6802","nonce":9088303},{"type":"v1","addr":"172.21.15.32:6803","nonce":9088303}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6806","nonce":9088303},{"type":"v1","addr":"172.21.15.32:6807","nonce":9088303}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6804","nonce":9088303},{"type":"v1","addr":"172.21.15.32:6805","nonce":9088303}]},"public_addr":"172.21.15.32:6801/9088303","cluster_addr":"172.21.15.32:6803/9088303","heartbeat_back_addr":"172.21.15.32:6807/9088303","heartbeat_front_addr":"172.21.15.32:6805/9088303","state":["exists","up"]},{"osd":4,"uuid":"4f902eff-1e18-418d-82c9-2151c69524fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6802","nonce":864752275},{"type":"v1","addr":"172.21.15.118:6803","nonce":864752275}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6804","nonce":864752275},{"type":"v1","addr":"172.21.15.118:6805","nonce":864752275}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6808","nonce":864752275},{"type":"v1","addr":"172.21.15.118:6809","nonce":864752275}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6806","nonce":864752275},{"type":"v1","addr":"172.21.15.118:6807","nonce":864752275}]},"public_addr":"172.21.15.118:6803/864752275","cluster_addr":"172.21.15.118:6805/864752275","heartbeat_back_addr":"172.21.15.118:6809/864752275","heartbeat_front_addr":"172.21.15.118:6807/864752275","state":["exists","up"]},{"osd":5,"uuid":"bfd9feac-dd06-458c-91b9-af7f175f612c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6810","nonce":2724891770},{"type":"v1","addr":"172.21.15.118:6811","nonce":2724891770}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6812","nonce":2724891770},{"type":"v1","addr":"172.21.15.118:6813","nonce":2724891770}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6816","nonce":2724891770},{"type":"v1","addr":"172.21.15.118:6817","nonce":2724891770}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6814","nonce":2724891770},{"type":"v1","addr":"172.21.15.118:6815","nonce":2724891770}]},"public_addr":"172.21.15.118:6811/2724891770","cluster_addr":"172.21.15.118:6813/2724891770","heartbeat_back_addr":"172.21.15.118:6817/2724891770","heartbeat_front_addr":"172.21.15.118:6815/2724891770","state":["exists","up"]},{"osd":6,"uuid":"5ed5a279-9d8e-416d-bbc7-7edd3e6d5705","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6825","nonce":801947386},{"type":"v1","addr":"172.21.15.118:6827","nonce":801947386}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6828","nonce":801947386},{"type":"v1","addr":"172.21.15.118:6829","nonce":801947386}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6832","nonce":801947386},{"type":"v1","addr":"172.21.15.118:6833","nonce":801947386}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6830","nonce":801947386},{"type":"v1","addr":"172.21.15.118:6831","nonce":801947386}]},"public_addr":"172.21.15.118:6827/801947386","cluster_addr":"172.21.15.118:6829/801947386","heartbeat_back_addr":"172.21.15.118:6833/801947386","heartbeat_front_addr":"172.21.15.118:6831/801947386","state":["exists","up"]},{"osd":7,"uuid":"f2086c47-875e-4451-a436-2008d1ea16af","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6818","nonce":653000048},{"type":"v1","addr":"172.21.15.118:6819","nonce":653000048}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6820","nonce":653000048},{"type":"v1","addr":"172.21.15.118:6821","nonce":653000048}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6824","nonce":653000048},{"type":"v1","addr":"172.21.15.118:6826","nonce":653000048}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6822","nonce":653000048},{"type":"v1","addr":"172.21.15.118:6823","nonce":653000048}]},"public_addr":"172.21.15.118:6819/653000048","cluster_addr":"172.21.15.118:6821/653000048","heartbeat_back_addr":"172.21.15.118:6826/653000048","heartbeat_front_addr":"172.21.15.118:6823/653000048","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-03T20:47:25.491994+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-03T20:47:25.587358+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-03T20:47:25.454175+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-03T20:47:25.377734+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-03T20:47:25.412351+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-03T20:47:25.499815+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-03T20:47:25.575044+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-03T20:47:25.552383+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.32:6834/1798758431":"2024-08-04T20:54:06.327185+0000","172.21.15.32:6832/1798758431":"2024-08-04T20:54:06.327185+0000","172.21.15.32:6835/4240363667":"2024-08-04T20:53:56.018944+0000","172.21.15.32:6833/4240363667":"2024-08-04T20:53:56.018944+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-08-03T20:54:11.835 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:11.834+0000 7f879de00640 1 -- 172.21.15.32:0/2883211831 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f8774077010 msgr2=0x7f8774079520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:11.835 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:11.834+0000 7f879de00640 1 --2- 172.21.15.32:0/2883211831 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f8774077010 0x7f8774079520 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f8780033000 tx=0x7f878002d000 comp rx=0 tx=0).stop 2024-08-03T20:54:11.836 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:11.834+0000 7f879de00640 1 -- 172.21.15.32:0/2883211831 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f877c004800 msgr2=0x7f87980aede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:11.836 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:11.834+0000 7f879de00640 1 --2- 172.21.15.32:0/2883211831 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f877c004800 0x7f87980aede0 secure :-1 s=READY pgs=867 cs=0 l=1 rev1=1 crypto rx=0x7f8788003cd0 tx=0x7f8788001800 comp rx=0 tx=0).stop 2024-08-03T20:54:11.836 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:11.834+0000 7f879de00640 1 -- 172.21.15.32:0/2883211831 shutdown_connections 2024-08-03T20:54:11.836 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:11.834+0000 7f879de00640 1 --2- 172.21.15.32:0/2883211831 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f8774077010 0x7f8774079520 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:11.836 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:11.834+0000 7f879de00640 1 --2- 172.21.15.32:0/2883211831 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f879813b930 0x7f8798155520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:11.836 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:11.834+0000 7f879de00640 1 --2- 172.21.15.32:0/2883211831 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f87981570f0 0x7f8798157500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:11.836 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:11.834+0000 7f879de00640 1 --2- 172.21.15.32:0/2883211831 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f877c004800 0x7f87980aede0 unknown :-1 s=CLOSED pgs=867 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:11.836 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:11.834+0000 7f879de00640 1 -- 172.21.15.32:0/2883211831 >> 172.21.15.32:0/2883211831 conn(0x7f87980b1bd0 msgr2=0x7f8798145610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:54:11.836 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:11.834+0000 7f879de00640 1 -- 172.21.15.32:0/2883211831 shutdown_connections 2024-08-03T20:54:11.836 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:11.834+0000 7f879de00640 1 -- 172.21.15.32:0/2883211831 wait complete. 2024-08-03T20:54:11.859 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-08-03T20:47:28.261300+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-08-03T20:54:11.859 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-08-03T20:54:12.089 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:12.087+0000 7f5aa8e00640 1 -- 172.21.15.32:0/2856989544 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f5aa40cccf0 msgr2=0x7f5aa40cd100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:12.090 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:12.088+0000 7f5aa8e00640 1 --2- 172.21.15.32:0/2856989544 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f5aa40cccf0 0x7f5aa40cd100 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f5a8c009830 tx=0x7f5a8c02a520 comp rx=0 tx=0).stop 2024-08-03T20:54:12.090 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:12.088+0000 7f5aa8e00640 1 -- 172.21.15.32:0/2856989544 shutdown_connections 2024-08-03T20:54:12.090 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:12.088+0000 7f5aa8e00640 1 --2- 172.21.15.32:0/2856989544 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f5aa40cccf0 0x7f5aa40cd100 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:12.090 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:12.088+0000 7f5aa8e00640 1 --2- 172.21.15.32:0/2856989544 >> v2:172.21.15.118:3300/0 conn(0x7f5aa4145c50 0x7f5aa406e950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:12.090 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:12.088+0000 7f5aa8e00640 1 -- 172.21.15.32:0/2856989544 >> 172.21.15.32:0/2856989544 conn(0x7f5aa40b1bd0 msgr2=0x7f5aa40b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:54:12.090 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:12.088+0000 7f5aa8e00640 1 -- 172.21.15.32:0/2856989544 shutdown_connections 2024-08-03T20:54:12.090 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:12.088+0000 7f5aa8e00640 1 -- 172.21.15.32:0/2856989544 wait complete. 2024-08-03T20:54:12.091 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:12.090+0000 7f5aa8e00640 1 Processor -- start 2024-08-03T20:54:12.091 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:12.090+0000 7f5aa8e00640 1 -- start start 2024-08-03T20:54:12.094 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:12.092+0000 7f5aa8e00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f5aa40cccf0 0x7f5aa413de40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:12.094 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:12.092+0000 7f5aa8e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5aa40ced30 0x7f5aa4159be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:12.094 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:12.092+0000 7f5aa8e00640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f5aa4145c50 0x7f5aa41e4be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:12.094 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:12.092+0000 7f5aa8e00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f5aa40b4070 con 0x7f5aa40ced30 2024-08-03T20:54:12.094 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:12.092+0000 7f5aa8e00640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f5aa40b3ef0 con 0x7f5aa40cccf0 2024-08-03T20:54:12.094 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:12.092+0000 7f5aa8e00640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7f5aa40b41f0 con 0x7f5aa4145c50 2024-08-03T20:54:12.094 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:12.092+0000 7f5a9be00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5aa40ced30 0x7f5aa4159be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:12.094 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:12.092+0000 7f5a9be00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5aa40ced30 0x7f5aa4159be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:43688/0 (socket says 172.21.15.32:43688) 2024-08-03T20:54:12.094 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:12.092+0000 7f5a9be00640 1 -- 172.21.15.32:0/839130139 learned_addr learned my addr 172.21.15.32:0/839130139 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:54:12.095 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:12.092+0000 7f5aa2800640 1 --2- 172.21.15.32:0/839130139 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f5aa4145c50 0x7f5aa41e4be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:12.095 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:12.092+0000 7f5aa1e00640 1 --2- 172.21.15.32:0/839130139 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f5aa40cccf0 0x7f5aa413de40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:12.095 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:12.092+0000 7f5a9be00640 1 -- 172.21.15.32:0/839130139 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f5aa4145c50 msgr2=0x7f5aa41e4be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:12.095 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:12.092+0000 7f5a9be00640 1 --2- 172.21.15.32:0/839130139 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f5aa4145c50 0x7f5aa41e4be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:12.095 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:12.092+0000 7f5a9be00640 1 -- 172.21.15.32:0/839130139 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f5aa40cccf0 msgr2=0x7f5aa413de40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:12.095 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:12.092+0000 7f5a9be00640 1 --2- 172.21.15.32:0/839130139 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f5aa40cccf0 0x7f5aa413de40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:12.095 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:12.092+0000 7f5a9be00640 1 -- 172.21.15.32:0/839130139 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5aa413f950 con 0x7f5aa40ced30 2024-08-03T20:54:12.095 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:12.092+0000 7f5aa1e00640 1 --2- 172.21.15.32:0/839130139 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f5aa40cccf0 0x7f5aa413de40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-03T20:54:12.096 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:12.093+0000 7f5a9be00640 1 --2- 172.21.15.32:0/839130139 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5aa40ced30 0x7f5aa4159be0 secure :-1 s=READY pgs=869 cs=0 l=1 rev1=1 crypto rx=0x7f5a8c009950 tx=0x7f5a8c0040c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:54:12.096 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:12.093+0000 7f5a9a000640 1 -- 172.21.15.32:0/839130139 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a8c005220 con 0x7f5aa40ced30 2024-08-03T20:54:12.096 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:12.093+0000 7f5aa8e00640 1 -- 172.21.15.32:0/839130139 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5aa413d360 con 0x7f5aa40ced30 2024-08-03T20:54:12.096 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:12.093+0000 7f5a9a000640 1 -- 172.21.15.32:0/839130139 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f5a8c0054b0 con 0x7f5aa40ced30 2024-08-03T20:54:12.096 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:12.093+0000 7f5a9a000640 1 -- 172.21.15.32:0/839130139 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a8c004a00 con 0x7f5aa40ced30 2024-08-03T20:54:12.096 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:12.093+0000 7f5aa8e00640 1 -- 172.21.15.32:0/839130139 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5aa413d920 con 0x7f5aa40ced30 2024-08-03T20:54:12.096 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:12.095+0000 7f5aa8e00640 1 -- 172.21.15.32:0/839130139 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5aa413d190 con 0x7f5aa40ced30 2024-08-03T20:54:12.097 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:12.096+0000 7f5a9a000640 1 -- 172.21.15.32:0/839130139 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f5a8c004ba0 con 0x7f5aa40ced30 2024-08-03T20:54:12.098 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:12.097+0000 7f5a9a000640 1 --2- 172.21.15.32:0/839130139 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f5a80077040 0x7f5a80079550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:12.098 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:12.097+0000 7f5a9a000640 1 -- 172.21.15.32:0/839130139 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(74..74 src has 1..74) ==== 5029+0+0 (secure 0 0 0) 0x7f5a8c0b9940 con 0x7f5aa40ced30 2024-08-03T20:54:12.098 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:12.097+0000 7f5aa1e00640 1 --2- 172.21.15.32:0/839130139 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f5a80077040 0x7f5a80079550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:12.103 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:12.098+0000 7f5aa1e00640 1 --2- 172.21.15.32:0/839130139 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f5a80077040 0x7f5a80079550 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f5a94009b80 tx=0x7f5a94005f90 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:54:12.103 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:12.101+0000 7f5a9a000640 1 -- 172.21.15.32:0/839130139 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7f5a8c03f030 con 0x7f5aa40ced30 2024-08-03T20:54:12.307 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:12.303+0000 7f5aa8e00640 1 -- 172.21.15.32:0/839130139 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f5aa406a3f0 con 0x7f5aa40ced30 2024-08-03T20:54:12.307 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:12.304+0000 7f5a9a000640 1 -- 172.21.15.32:0/839130139 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v74) ==== 93+0+10 (secure 0 0 0) 0x7f5aa406a3f0 con 0x7f5aa40ced30 2024-08-03T20:54:12.307 INFO:teuthology.orchestra.run.smithi032.stdout:pg_num: 1 2024-08-03T20:54:12.308 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:12.306+0000 7f5aa8e00640 1 -- 172.21.15.32:0/839130139 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f5a80077040 msgr2=0x7f5a80079550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:12.308 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:12.306+0000 7f5aa8e00640 1 --2- 172.21.15.32:0/839130139 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f5a80077040 0x7f5a80079550 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f5a94009b80 tx=0x7f5a94005f90 comp rx=0 tx=0).stop 2024-08-03T20:54:12.309 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:12.306+0000 7f5aa8e00640 1 -- 172.21.15.32:0/839130139 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5aa40ced30 msgr2=0x7f5aa4159be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:12.309 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:12.306+0000 7f5aa8e00640 1 --2- 172.21.15.32:0/839130139 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5aa40ced30 0x7f5aa4159be0 secure :-1 s=READY pgs=869 cs=0 l=1 rev1=1 crypto rx=0x7f5a8c009950 tx=0x7f5a8c0040c0 comp rx=0 tx=0).stop 2024-08-03T20:54:12.309 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:12.306+0000 7f5aa8e00640 1 -- 172.21.15.32:0/839130139 shutdown_connections 2024-08-03T20:54:12.309 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:12.306+0000 7f5aa8e00640 1 --2- 172.21.15.32:0/839130139 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f5a80077040 0x7f5a80079550 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:12.309 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:12.306+0000 7f5aa8e00640 1 --2- 172.21.15.32:0/839130139 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f5aa4145c50 0x7f5aa41e4be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:12.309 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:12.306+0000 7f5aa8e00640 1 --2- 172.21.15.32:0/839130139 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5aa40ced30 0x7f5aa4159be0 unknown :-1 s=CLOSED pgs=869 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:12.309 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:12.306+0000 7f5aa8e00640 1 --2- 172.21.15.32:0/839130139 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f5aa40cccf0 0x7f5aa413de40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:12.309 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:12.306+0000 7f5aa8e00640 1 -- 172.21.15.32:0/839130139 >> 172.21.15.32:0/839130139 conn(0x7f5aa40b1bd0 msgr2=0x7f5aa40cf140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:54:12.309 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:12.306+0000 7f5aa8e00640 1 -- 172.21.15.32:0/839130139 shutdown_connections 2024-08-03T20:54:12.309 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:12.306+0000 7f5aa8e00640 1 -- 172.21.15.32:0/839130139 wait complete. 2024-08-03T20:54:12.333 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_damage.TestDamage.test_object_deletion' 2024-08-03T20:54:12.564 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:12.562+0000 7f124c000640 1 -- 172.21.15.32:0/4174875016 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f124406e930 msgr2=0x7f1244144f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:12.564 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:12.562+0000 7f124c000640 1 --2- 172.21.15.32:0/4174875016 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f124406e930 0x7f1244144f90 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f1238009830 tx=0x7f123802a540 comp rx=0 tx=0).stop 2024-08-03T20:54:12.564 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:12.562+0000 7f124c000640 1 -- 172.21.15.32:0/4174875016 shutdown_connections 2024-08-03T20:54:12.564 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:12.562+0000 7f124c000640 1 --2- 172.21.15.32:0/4174875016 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f124406e930 0x7f1244144f90 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:12.564 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:12.562+0000 7f124c000640 1 -- 172.21.15.32:0/4174875016 >> 172.21.15.32:0/4174875016 conn(0x7f12440b1bd0 msgr2=0x7f12440b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:54:12.564 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:12.562+0000 7f124c000640 1 -- 172.21.15.32:0/4174875016 shutdown_connections 2024-08-03T20:54:12.564 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:12.562+0000 7f124c000640 1 -- 172.21.15.32:0/4174875016 wait complete. 2024-08-03T20:54:12.565 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:12.564+0000 7f124c000640 1 Processor -- start 2024-08-03T20:54:12.566 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:12.564+0000 7f124c000640 1 -- start start 2024-08-03T20:54:12.567 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:12.566+0000 7f124c000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f124406e930 0x7f124413ac30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:12.568 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:12.566+0000 7f124c000640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f1244073e60 0x7f124413fc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:12.568 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:12.566+0000 7f124c000640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f124415fff0 0x7f12441401d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:12.568 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:12.566+0000 7f124c000640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f12440cf840 con 0x7f124406e930 2024-08-03T20:54:12.568 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:12.566+0000 7f124c000640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f1244145f30 con 0x7f124415fff0 2024-08-03T20:54:12.568 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:12.566+0000 7f124c000640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7f12440cf9c0 con 0x7f1244073e60 2024-08-03T20:54:12.568 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:12.566+0000 7f1249600640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f124406e930 0x7f124413ac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:12.568 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:12.566+0000 7f1249600640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f124406e930 0x7f124413ac30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:43698/0 (socket says 172.21.15.32:43698) 2024-08-03T20:54:12.568 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:12.566+0000 7f1249600640 1 -- 172.21.15.32:0/2742370888 learned_addr learned my addr 172.21.15.32:0/2742370888 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:54:12.569 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:12.566+0000 7f1249600640 1 -- 172.21.15.32:0/2742370888 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f1244073e60 msgr2=0x7f124413fc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:12.569 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:12.566+0000 7f124a000640 1 --2- 172.21.15.32:0/2742370888 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f124415fff0 0x7f12441401d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:12.570 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:12.566+0000 7f1248c00640 1 --2- 172.21.15.32:0/2742370888 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f1244073e60 0x7f124413fc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:12.570 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:12.567+0000 7f1249600640 1 --2- 172.21.15.32:0/2742370888 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f1244073e60 0x7f124413fc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:12.570 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:12.567+0000 7f1249600640 1 -- 172.21.15.32:0/2742370888 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f124415fff0 msgr2=0x7f12441401d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:12.570 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:12.567+0000 7f1249600640 1 --2- 172.21.15.32:0/2742370888 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f124415fff0 0x7f12441401d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:12.570 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:12.567+0000 7f1249600640 1 -- 172.21.15.32:0/2742370888 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f12441430b0 con 0x7f124406e930 2024-08-03T20:54:12.570 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:12.567+0000 7f124a000640 1 --2- 172.21.15.32:0/2742370888 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f124415fff0 0x7f12441401d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-03T20:54:12.570 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:12.567+0000 7f1248c00640 1 --2- 172.21.15.32:0/2742370888 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f1244073e60 0x7f124413fc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-03T20:54:12.570 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:12.567+0000 7f1249600640 1 --2- 172.21.15.32:0/2742370888 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f124406e930 0x7f124413ac30 secure :-1 s=READY pgs=871 cs=0 l=1 rev1=1 crypto rx=0x7f123802ab30 tx=0x7f1238003c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:54:12.570 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:12.567+0000 7f1241600640 1 -- 172.21.15.32:0/2742370888 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1238005c90 con 0x7f124406e930 2024-08-03T20:54:12.570 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:12.567+0000 7f1241600640 1 -- 172.21.15.32:0/2742370888 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f1238005020 con 0x7f124406e930 2024-08-03T20:54:12.570 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:12.567+0000 7f124c000640 1 -- 172.21.15.32:0/2742370888 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1244143390 con 0x7f124406e930 2024-08-03T20:54:12.570 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:12.567+0000 7f1241600640 1 -- 172.21.15.32:0/2742370888 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1238005450 con 0x7f124406e930 2024-08-03T20:54:12.570 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:12.567+0000 7f124c000640 1 -- 172.21.15.32:0/2742370888 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f124415ce40 con 0x7f124406e930 2024-08-03T20:54:12.571 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:12.568+0000 7f124c000640 1 -- 172.21.15.32:0/2742370888 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f124415d0a0 con 0x7f124406e930 2024-08-03T20:54:12.571 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:12.570+0000 7f1241600640 1 -- 172.21.15.32:0/2742370888 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f12380317c0 con 0x7f124406e930 2024-08-03T20:54:12.572 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:12.570+0000 7f1241600640 1 --2- 172.21.15.32:0/2742370888 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f1224077010 0x7f1224079520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:12.572 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:12.571+0000 7f1241600640 1 -- 172.21.15.32:0/2742370888 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(74..74 src has 1..74) ==== 5029+0+0 (secure 0 0 0) 0x7f12380b9710 con 0x7f124406e930 2024-08-03T20:54:12.572 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:12.571+0000 7f1248c00640 1 --2- 172.21.15.32:0/2742370888 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f1224077010 0x7f1224079520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:12.573 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:12.572+0000 7f1248c00640 1 --2- 172.21.15.32:0/2742370888 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f1224077010 0x7f1224079520 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f12441445e0 tx=0x7f1230009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:54:12.576 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:12.575+0000 7f1241600640 1 -- 172.21.15.32:0/2742370888 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7f123803f030 con 0x7f124406e930 2024-08-03T20:54:12.860 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:12.859+0000 7f124c000640 1 -- 172.21.15.32:0/2742370888 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_damage.TestDamage.test_object_deletion"]} v 0) -- 0x7f12441414e0 con 0x7f124406e930 2024-08-03T20:54:13.356 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:13.355+0000 7f1241600640 1 -- 172.21.15.32:0/2742370888 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_damage.TestDamage.test_object_deletion"]}]=0 v325) ==== 138+0+0 (secure 0 0 0) 0x7f1238082ba0 con 0x7f124406e930 2024-08-03T20:54:13.359 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:13.358+0000 7f124c000640 1 -- 172.21.15.32:0/2742370888 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f1224077010 msgr2=0x7f1224079520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:13.360 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:13.358+0000 7f124c000640 1 --2- 172.21.15.32:0/2742370888 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f1224077010 0x7f1224079520 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f12441445e0 tx=0x7f1230009040 comp rx=0 tx=0).stop 2024-08-03T20:54:13.360 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:13.358+0000 7f124c000640 1 -- 172.21.15.32:0/2742370888 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f124406e930 msgr2=0x7f124413ac30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:13.360 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:13.358+0000 7f124c000640 1 --2- 172.21.15.32:0/2742370888 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f124406e930 0x7f124413ac30 secure :-1 s=READY pgs=871 cs=0 l=1 rev1=1 crypto rx=0x7f123802ab30 tx=0x7f1238003c10 comp rx=0 tx=0).stop 2024-08-03T20:54:13.360 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:13.358+0000 7f124c000640 1 -- 172.21.15.32:0/2742370888 shutdown_connections 2024-08-03T20:54:13.360 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:13.358+0000 7f124c000640 1 --2- 172.21.15.32:0/2742370888 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f1224077010 0x7f1224079520 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:13.360 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:13.358+0000 7f124c000640 1 --2- 172.21.15.32:0/2742370888 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f124415fff0 0x7f12441401d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:13.360 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:13.358+0000 7f124c000640 1 --2- 172.21.15.32:0/2742370888 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f1244073e60 0x7f124413fc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:13.360 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:13.358+0000 7f124c000640 1 --2- 172.21.15.32:0/2742370888 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f124406e930 0x7f124413ac30 unknown :-1 s=CLOSED pgs=871 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:13.360 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:13.358+0000 7f124c000640 1 -- 172.21.15.32:0/2742370888 >> 172.21.15.32:0/2742370888 conn(0x7f12440b1bd0 msgr2=0x7f12440cc540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:54:13.360 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:13.358+0000 7f124c000640 1 -- 172.21.15.32:0/2742370888 shutdown_connections 2024-08-03T20:54:13.360 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:13.358+0000 7f124c000640 1 -- 172.21.15.32:0/2742370888 wait complete. 2024-08-03T20:54:13.386 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config dump 2024-08-03T20:54:13.615 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:13.612+0000 7f7af9c00640 1 -- 172.21.15.32:0/3929301692 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7af415fff0 msgr2=0x7f7af4160400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:13.615 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:13.612+0000 7f7af9c00640 1 --2- 172.21.15.32:0/3929301692 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7af415fff0 0x7f7af4160400 secure :-1 s=READY pgs=873 cs=0 l=1 rev1=1 crypto rx=0x7f7af4071700 tx=0x7f7aec02a8b0 comp rx=0 tx=0).stop 2024-08-03T20:54:13.615 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:13.613+0000 7f7af9c00640 1 -- 172.21.15.32:0/3929301692 shutdown_connections 2024-08-03T20:54:13.615 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:13.613+0000 7f7af9c00640 1 --2- 172.21.15.32:0/3929301692 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7af415fff0 0x7f7af4160400 unknown :-1 s=CLOSED pgs=873 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:13.615 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:13.613+0000 7f7af9c00640 1 --2- 172.21.15.32:0/3929301692 >> v2:172.21.15.118:3300/0 conn(0x7f7af406e930 0x7f7af4144f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:13.616 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:13.613+0000 7f7af9c00640 1 -- 172.21.15.32:0/3929301692 >> 172.21.15.32:0/3929301692 conn(0x7f7af40b1bd0 msgr2=0x7f7af40b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:54:13.616 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:13.613+0000 7f7af9c00640 1 -- 172.21.15.32:0/3929301692 shutdown_connections 2024-08-03T20:54:13.616 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:13.613+0000 7f7af9c00640 1 -- 172.21.15.32:0/3929301692 wait complete. 2024-08-03T20:54:13.616 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:13.615+0000 7f7af9c00640 1 Processor -- start 2024-08-03T20:54:13.616 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:13.615+0000 7f7af9c00640 1 -- start start 2024-08-03T20:54:13.620 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:13.617+0000 7f7af9c00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7af406e930 0x7f7af415b540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:13.620 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:13.617+0000 7f7af9c00640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f7af4073e60 0x7f7af415f2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:13.620 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:13.617+0000 7f7af9c00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f7af415fff0 0x7f7af415f810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:13.620 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:13.617+0000 7f7af9c00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f7af40cf840 con 0x7f7af406e930 2024-08-03T20:54:13.620 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:13.617+0000 7f7af9c00640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f7af4145f30 con 0x7f7af415fff0 2024-08-03T20:54:13.620 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:13.617+0000 7f7af9c00640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7f7af40cf9c0 con 0x7f7af4073e60 2024-08-03T20:54:13.620 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:13.617+0000 7f7af2a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7af406e930 0x7f7af415b540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:13.620 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:13.617+0000 7f7af2a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7af406e930 0x7f7af415b540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:43704/0 (socket says 172.21.15.32:43704) 2024-08-03T20:54:13.620 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:13.617+0000 7f7af2a00640 1 -- 172.21.15.32:0/1171157285 learned_addr learned my addr 172.21.15.32:0/1171157285 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:54:13.620 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:13.617+0000 7f7af2000640 1 --2- 172.21.15.32:0/1171157285 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f7af4073e60 0x7f7af415f2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:13.620 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:13.617+0000 7f7af3400640 1 --2- 172.21.15.32:0/1171157285 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f7af415fff0 0x7f7af415f810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:13.620 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:13.617+0000 7f7af2a00640 1 -- 172.21.15.32:0/1171157285 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f7af4073e60 msgr2=0x7f7af415f2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:13.621 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:13.617+0000 7f7af2a00640 1 --2- 172.21.15.32:0/1171157285 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f7af4073e60 0x7f7af415f2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:13.621 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:13.618+0000 7f7af2a00640 1 -- 172.21.15.32:0/1171157285 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f7af415fff0 msgr2=0x7f7af415f810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:13.621 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:13.618+0000 7f7af2a00640 1 --2- 172.21.15.32:0/1171157285 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f7af415fff0 0x7f7af415f810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:13.621 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:13.618+0000 7f7af2a00640 1 -- 172.21.15.32:0/1171157285 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7af415ba80 con 0x7f7af406e930 2024-08-03T20:54:13.621 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:13.618+0000 7f7af2a00640 1 --2- 172.21.15.32:0/1171157285 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7af406e930 0x7f7af415b540 secure :-1 s=READY pgs=874 cs=0 l=1 rev1=1 crypto rx=0x7f7ae400d770 tx=0x7f7ae400dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:54:13.621 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:13.618+0000 7f7aeaa00640 1 -- 172.21.15.32:0/1171157285 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ae4017070 con 0x7f7af406e930 2024-08-03T20:54:13.621 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:13.618+0000 7f7aeaa00640 1 -- 172.21.15.32:0/1171157285 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f7ae4009d20 con 0x7f7af406e930 2024-08-03T20:54:13.621 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:13.618+0000 7f7af9c00640 1 -- 172.21.15.32:0/1171157285 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7af413b830 con 0x7f7af406e930 2024-08-03T20:54:13.621 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:13.618+0000 7f7aeaa00640 1 -- 172.21.15.32:0/1171157285 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ae4004280 con 0x7f7af406e930 2024-08-03T20:54:13.621 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:13.618+0000 7f7af9c00640 1 -- 172.21.15.32:0/1171157285 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7af413bdf0 con 0x7f7af406e930 2024-08-03T20:54:13.622 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:13.619+0000 7f7af9c00640 1 -- 172.21.15.32:0/1171157285 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7af4160c30 con 0x7f7af406e930 2024-08-03T20:54:13.622 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:13.621+0000 7f7aeaa00640 1 -- 172.21.15.32:0/1171157285 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f7ae4005020 con 0x7f7af406e930 2024-08-03T20:54:13.623 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:13.621+0000 7f7aeaa00640 1 --2- 172.21.15.32:0/1171157285 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f7ad4077010 0x7f7ad4079520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:13.623 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:13.622+0000 7f7aeaa00640 1 -- 172.21.15.32:0/1171157285 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(74..74 src has 1..74) ==== 5029+0+0 (secure 0 0 0) 0x7f7ae4098850 con 0x7f7af406e930 2024-08-03T20:54:13.623 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:13.622+0000 7f7af2000640 1 --2- 172.21.15.32:0/1171157285 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f7ad4077010 0x7f7ad4079520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:13.624 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:13.623+0000 7f7af2000640 1 --2- 172.21.15.32:0/1171157285 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f7ad4077010 0x7f7ad4079520 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f7ae0007fd0 tx=0x7f7ae0001390 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:54:13.627 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:13.626+0000 7f7aeaa00640 1 -- 172.21.15.32:0/1171157285 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7f7ae4020020 con 0x7f7af406e930 2024-08-03T20:54:13.836 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:13.834+0000 7f7af9c00640 1 -- 172.21.15.32:0/1171157285 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "config dump"} v 0) -- 0x7f7af406fad0 con 0x7f7af406e930 2024-08-03T20:54:13.836 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:13.835+0000 7f7aeaa00640 1 -- 172.21.15.32:0/1171157285 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "config dump"}]=0 v42) ==== 59+0+1998 (secure 0 0 0) 0x7f7ae4061720 con 0x7f7af406e930 2024-08-03T20:54:13.836 INFO:teuthology.orchestra.run.smithi032.stdout:WHO MASK LEVEL OPTION VALUE RO 2024-08-03T20:54:13.837 INFO:teuthology.orchestra.run.smithi032.stdout:mon advanced mon_op_complaint_time 120 2024-08-03T20:54:13.837 INFO:teuthology.orchestra.run.smithi032.stdout:mon dev mon_warn_on_pool_no_app false 2024-08-03T20:54:13.837 INFO:teuthology.orchestra.run.smithi032.stdout:osd advanced osd_op_complaint_time 180.000000 2024-08-03T20:54:13.837 INFO:teuthology.orchestra.run.smithi032.stdout:osd.0 basic osd_mclock_max_capacity_iops_ssd 4313.565729 2024-08-03T20:54:13.837 INFO:teuthology.orchestra.run.smithi032.stdout:osd.1 basic osd_mclock_max_capacity_iops_ssd 4272.517979 2024-08-03T20:54:13.837 INFO:teuthology.orchestra.run.smithi032.stdout:osd.2 basic osd_mclock_max_capacity_iops_ssd 4131.201546 2024-08-03T20:54:13.837 INFO:teuthology.orchestra.run.smithi032.stdout:osd.3 basic osd_mclock_max_capacity_iops_ssd 4155.765914 2024-08-03T20:54:13.837 INFO:teuthology.orchestra.run.smithi032.stdout:osd.4 basic osd_mclock_max_capacity_iops_ssd 5744.667532 2024-08-03T20:54:13.837 INFO:teuthology.orchestra.run.smithi032.stdout:osd.5 basic osd_mclock_max_capacity_iops_ssd 3764.678226 2024-08-03T20:54:13.837 INFO:teuthology.orchestra.run.smithi032.stdout:osd.6 basic osd_mclock_max_capacity_iops_ssd 3927.989098 2024-08-03T20:54:13.837 INFO:teuthology.orchestra.run.smithi032.stdout:osd.7 basic osd_mclock_max_capacity_iops_ssd 3760.968870 2024-08-03T20:54:13.838 INFO:teuthology.orchestra.run.smithi032.stdout:mds advanced debug_mds 20/20 2024-08-03T20:54:13.838 INFO:teuthology.orchestra.run.smithi032.stdout:mds advanced debug_mds_balancer 20/20 2024-08-03T20:54:13.838 INFO:teuthology.orchestra.run.smithi032.stdout:mds advanced debug_ms 2/2 2024-08-03T20:54:13.838 INFO:teuthology.orchestra.run.smithi032.stdout:mds dev mds_debug_frag true 2024-08-03T20:54:13.838 INFO:teuthology.orchestra.run.smithi032.stdout:mds dev mds_debug_scatterstat true 2024-08-03T20:54:13.838 INFO:teuthology.orchestra.run.smithi032.stdout:mds advanced mds_op_complaint_time 180.000000 2024-08-03T20:54:13.838 INFO:teuthology.orchestra.run.smithi032.stdout:mds dev mds_verify_scatter true 2024-08-03T20:54:13.838 INFO:teuthology.orchestra.run.smithi032.stdout:mds advanced osd_op_complaint_time 180.000000 2024-08-03T20:54:13.838 INFO:teuthology.orchestra.run.smithi032.stdout:mds advanced rados_mon_op_timeout 900 2024-08-03T20:54:13.838 INFO:teuthology.orchestra.run.smithi032.stdout:mds advanced rados_osd_op_timeout 900 2024-08-03T20:54:13.839 INFO:teuthology.orchestra.run.smithi032.stdout:client advanced client_mount_timeout 600 2024-08-03T20:54:13.839 INFO:teuthology.orchestra.run.smithi032.stdout:client advanced debug_client 20/20 2024-08-03T20:54:13.839 INFO:teuthology.orchestra.run.smithi032.stdout:client advanced debug_ms 1/1 2024-08-03T20:54:13.839 INFO:teuthology.orchestra.run.smithi032.stdout:client advanced rados_mon_op_timeout 900 2024-08-03T20:54:13.839 INFO:teuthology.orchestra.run.smithi032.stdout:client advanced rados_osd_op_timeout 900 2024-08-03T20:54:13.840 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:13.837+0000 7f7af9c00640 1 -- 172.21.15.32:0/1171157285 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f7ad4077010 msgr2=0x7f7ad4079520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:13.840 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:13.837+0000 7f7af9c00640 1 --2- 172.21.15.32:0/1171157285 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f7ad4077010 0x7f7ad4079520 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f7ae0007fd0 tx=0x7f7ae0001390 comp rx=0 tx=0).stop 2024-08-03T20:54:13.840 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:13.837+0000 7f7af9c00640 1 -- 172.21.15.32:0/1171157285 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7af406e930 msgr2=0x7f7af415b540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:13.840 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:13.837+0000 7f7af9c00640 1 --2- 172.21.15.32:0/1171157285 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7af406e930 0x7f7af415b540 secure :-1 s=READY pgs=874 cs=0 l=1 rev1=1 crypto rx=0x7f7ae400d770 tx=0x7f7ae400dc30 comp rx=0 tx=0).stop 2024-08-03T20:54:13.840 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:13.837+0000 7f7af9c00640 1 -- 172.21.15.32:0/1171157285 shutdown_connections 2024-08-03T20:54:13.840 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:13.837+0000 7f7af9c00640 1 --2- 172.21.15.32:0/1171157285 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f7ad4077010 0x7f7ad4079520 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:13.840 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:13.837+0000 7f7af9c00640 1 --2- 172.21.15.32:0/1171157285 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f7af415fff0 0x7f7af415f810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:13.840 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:13.837+0000 7f7af9c00640 1 --2- 172.21.15.32:0/1171157285 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f7af4073e60 0x7f7af415f2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:13.841 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:13.837+0000 7f7af9c00640 1 --2- 172.21.15.32:0/1171157285 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7af406e930 0x7f7af415b540 unknown :-1 s=CLOSED pgs=874 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:13.841 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:13.837+0000 7f7af9c00640 1 -- 172.21.15.32:0/1171157285 >> 172.21.15.32:0/1171157285 conn(0x7f7af40b1bd0 msgr2=0x7f7af40b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:54:13.841 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:13.837+0000 7f7af9c00640 1 -- 172.21.15.32:0/1171157285 shutdown_connections 2024-08-03T20:54:13.841 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:13.838+0000 7f7af9c00640 1 -- 172.21.15.32:0/1171157285 wait complete. 2024-08-03T20:54:13.862 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-08-03T20:54:14.094 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:14.092+0000 7f1bc5200640 1 -- 172.21.15.32:0/57686746 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1bc0160350 msgr2=0x7f1bc015c820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:14.094 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:14.092+0000 7f1bc5200640 1 --2- 172.21.15.32:0/57686746 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1bc0160350 0x7f1bc015c820 secure :-1 s=READY pgs=875 cs=0 l=1 rev1=1 crypto rx=0x7f1bc00cd900 tx=0x7f1bb802aa10 comp rx=0 tx=0).stop 2024-08-03T20:54:14.094 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:14.092+0000 7f1bc5200640 1 -- 172.21.15.32:0/57686746 shutdown_connections 2024-08-03T20:54:14.094 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:14.092+0000 7f1bc5200640 1 --2- 172.21.15.32:0/57686746 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1bc0160350 0x7f1bc015c820 unknown :-1 s=CLOSED pgs=875 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:14.094 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:14.092+0000 7f1bc5200640 1 --2- 172.21.15.32:0/57686746 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f1bc0159ff0 0x7f1bc015fe10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:14.094 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:14.092+0000 7f1bc5200640 1 --2- 172.21.15.32:0/57686746 >> v2:172.21.15.118:3300/0 conn(0x7f1bc01609d0 0x7f1bc00cc050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:14.095 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:14.092+0000 7f1bc5200640 1 -- 172.21.15.32:0/57686746 >> 172.21.15.32:0/57686746 conn(0x7f1bc00cfd80 msgr2=0x7f1bc00d0190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:54:14.095 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:14.092+0000 7f1bc5200640 1 -- 172.21.15.32:0/57686746 shutdown_connections 2024-08-03T20:54:14.095 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:14.092+0000 7f1bc5200640 1 -- 172.21.15.32:0/57686746 wait complete. 2024-08-03T20:54:14.096 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:14.094+0000 7f1bc5200640 1 Processor -- start 2024-08-03T20:54:14.096 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:14.095+0000 7f1bc5200640 1 -- start start 2024-08-03T20:54:14.098 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:14.096+0000 7f1bc5200640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f1bc0159ff0 0x7f1bc00c6e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:14.098 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:14.096+0000 7f1bc5200640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f1bc0160350 0x7f1bc014a800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:14.099 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:14.096+0000 7f1bc5200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1bc01609d0 0x7f1bc01f5540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:14.099 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:14.096+0000 7f1bc5200640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f1bc015cee0 con 0x7f1bc01609d0 2024-08-03T20:54:14.099 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:14.096+0000 7f1bc5200640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f1bc015cd60 con 0x7f1bc0159ff0 2024-08-03T20:54:14.099 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:14.096+0000 7f1bc5200640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7f1bc015d060 con 0x7f1bc0160350 2024-08-03T20:54:14.099 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:14.096+0000 7f1bbfe00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1bc01609d0 0x7f1bc01f5540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:14.099 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:14.096+0000 7f1bbfe00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1bc01609d0 0x7f1bc01f5540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:43718/0 (socket says 172.21.15.32:43718) 2024-08-03T20:54:14.099 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:14.096+0000 7f1bbfe00640 1 -- 172.21.15.32:0/3311651117 learned_addr learned my addr 172.21.15.32:0/3311651117 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:54:14.099 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:14.096+0000 7f1bbf400640 1 --2- 172.21.15.32:0/3311651117 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f1bc0159ff0 0x7f1bc00c6e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:14.099 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:14.096+0000 7f1bb7e00640 1 --2- 172.21.15.32:0/3311651117 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f1bc0160350 0x7f1bc014a800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:14.099 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:14.097+0000 7f1bbfe00640 1 -- 172.21.15.32:0/3311651117 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f1bc0160350 msgr2=0x7f1bc014a800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:14.099 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:14.097+0000 7f1bbfe00640 1 --2- 172.21.15.32:0/3311651117 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f1bc0160350 0x7f1bc014a800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:14.099 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:14.097+0000 7f1bbfe00640 1 -- 172.21.15.32:0/3311651117 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f1bc0159ff0 msgr2=0x7f1bc00c6e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:14.099 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:14.097+0000 7f1bbfe00640 1 --2- 172.21.15.32:0/3311651117 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f1bc0159ff0 0x7f1bc00c6e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:14.100 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:14.097+0000 7f1bbfe00640 1 -- 172.21.15.32:0/3311651117 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1bc00b53b0 con 0x7f1bc01609d0 2024-08-03T20:54:14.100 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:14.097+0000 7f1bbfe00640 1 --2- 172.21.15.32:0/3311651117 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1bc01609d0 0x7f1bc01f5540 secure :-1 s=READY pgs=876 cs=0 l=1 rev1=1 crypto rx=0x7f1bb8009d60 tx=0x7f1bb8003f10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:54:14.100 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:14.097+0000 7f1bb7400640 1 -- 172.21.15.32:0/3311651117 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1bb8033050 con 0x7f1bc01609d0 2024-08-03T20:54:14.100 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:14.098+0000 7f1bb7400640 1 -- 172.21.15.32:0/3311651117 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f1bb80042d0 con 0x7f1bc01609d0 2024-08-03T20:54:14.101 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:14.098+0000 7f1bb7400640 1 -- 172.21.15.32:0/3311651117 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1bb8004720 con 0x7f1bc01609d0 2024-08-03T20:54:14.101 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:14.098+0000 7f1bc5200640 1 -- 172.21.15.32:0/3311651117 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1bc00b5690 con 0x7f1bc01609d0 2024-08-03T20:54:14.101 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:14.098+0000 7f1bc5200640 1 -- 172.21.15.32:0/3311651117 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1bc01f8f00 con 0x7f1bc01609d0 2024-08-03T20:54:14.101 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:14.099+0000 7f1bc5200640 1 -- 172.21.15.32:0/3311651117 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1bc015d580 con 0x7f1bc01609d0 2024-08-03T20:54:14.101 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:14.100+0000 7f1bb7400640 1 -- 172.21.15.32:0/3311651117 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f1bb8036070 con 0x7f1bc01609d0 2024-08-03T20:54:14.102 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:14.101+0000 7f1bb7400640 1 --2- 172.21.15.32:0/3311651117 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f1b9c077010 0x7f1b9c079520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:14.102 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:14.101+0000 7f1bb7400640 1 -- 172.21.15.32:0/3311651117 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(74..74 src has 1..74) ==== 5029+0+0 (secure 0 0 0) 0x7f1bb80b9490 con 0x7f1bc01609d0 2024-08-03T20:54:14.103 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:14.101+0000 7f1bbf400640 1 --2- 172.21.15.32:0/3311651117 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f1b9c077010 0x7f1b9c079520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:14.103 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:14.102+0000 7f1bbf400640 1 --2- 172.21.15.32:0/3311651117 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f1b9c077010 0x7f1b9c079520 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f1bb000bb50 tx=0x7f1bb000b2e0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:54:14.107 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:14.105+0000 7f1bb7400640 1 -- 172.21.15.32:0/3311651117 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7f1bb8049020 con 0x7f1bc01609d0 2024-08-03T20:54:14.312 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:14.311+0000 7f1bc5200640 1 -- 172.21.15.32:0/3311651117 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) -- 0x7f1bc00d09e0 con 0x7f1bc01609d0 2024-08-03T20:54:14.316 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:14.315+0000 7f1bb7400640 1 -- 172.21.15.32:0/3311651117 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v43)=0 v43) ==== 122+0+0 (secure 0 0 0) 0x7f1bb8082240 con 0x7f1bc01609d0 2024-08-03T20:54:14.319 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:14.317+0000 7f1bc5200640 1 -- 172.21.15.32:0/3311651117 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f1b9c077010 msgr2=0x7f1b9c079520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:14.319 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:14.317+0000 7f1bc5200640 1 --2- 172.21.15.32:0/3311651117 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f1b9c077010 0x7f1b9c079520 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f1bb000bb50 tx=0x7f1bb000b2e0 comp rx=0 tx=0).stop 2024-08-03T20:54:14.319 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:14.317+0000 7f1bc5200640 1 -- 172.21.15.32:0/3311651117 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1bc01609d0 msgr2=0x7f1bc01f5540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:14.319 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:14.317+0000 7f1bc5200640 1 --2- 172.21.15.32:0/3311651117 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1bc01609d0 0x7f1bc01f5540 secure :-1 s=READY pgs=876 cs=0 l=1 rev1=1 crypto rx=0x7f1bb8009d60 tx=0x7f1bb8003f10 comp rx=0 tx=0).stop 2024-08-03T20:54:14.320 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:14.317+0000 7f1bc5200640 1 -- 172.21.15.32:0/3311651117 shutdown_connections 2024-08-03T20:54:14.320 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:14.317+0000 7f1bc5200640 1 --2- 172.21.15.32:0/3311651117 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f1b9c077010 0x7f1b9c079520 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:14.320 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:14.317+0000 7f1bc5200640 1 --2- 172.21.15.32:0/3311651117 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1bc01609d0 0x7f1bc01f5540 unknown :-1 s=CLOSED pgs=876 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:14.320 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:14.317+0000 7f1bc5200640 1 --2- 172.21.15.32:0/3311651117 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f1bc0160350 0x7f1bc014a800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:14.320 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:14.317+0000 7f1bc5200640 1 --2- 172.21.15.32:0/3311651117 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f1bc0159ff0 0x7f1bc00c6e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:14.320 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:14.317+0000 7f1bc5200640 1 -- 172.21.15.32:0/3311651117 >> 172.21.15.32:0/3311651117 conn(0x7f1bc00cfd80 msgr2=0x7f1bc00cb7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:54:14.320 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:14.317+0000 7f1bc5200640 1 -- 172.21.15.32:0/3311651117 shutdown_connections 2024-08-03T20:54:14.320 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:14.318+0000 7f1bc5200640 1 -- 172.21.15.32:0/3311651117 wait complete. 2024-08-03T20:54:14.341 INFO:teuthology.nuke:Clearing teuthology firewall rules... 2024-08-03T20:54:14.342 DEBUG:teuthology.orchestra.run.smithi032:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-08-03T20:54:14.419 DEBUG:teuthology.orchestra.run.smithi118:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-08-03T20:54:14.511 INFO:teuthology.nuke:Cleared teuthology firewall rules. 2024-08-03T20:54:14.512 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-08-03T20:54:14.512 DEBUG:teuthology.orchestra.run.smithi032:> dd if=/proc/self/mounts of=/dev/stdout 2024-08-03T20:54:14.548 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053453,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6499208k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 none /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249604k,nr_inodes=812401,mode=700,uid=1000,gid=1316 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6499208k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-08-03T20:54:14.548 DEBUG:teuthology.orchestra.run.smithi118:> set -ex 2024-08-03T20:54:14.548 DEBUG:teuthology.orchestra.run.smithi118:> dd if=/proc/self/mounts of=/dev/stdout 2024-08-03T20:54:14.586 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052663,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497944k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248972k,nr_inodes=812243,mode=700,uid=1000,gid=1316 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497944k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-08-03T20:54:14.587 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-08-03T20:54:14.587 DEBUG:teuthology.orchestra.run.smithi032:> dd if=/proc/self/mounts of=/dev/stdout 2024-08-03T20:54:14.624 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053453,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6499208k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 none /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249604k,nr_inodes=812401,mode=700,uid=1000,gid=1316 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6499208k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-08-03T20:54:14.624 DEBUG:teuthology.orchestra.run.smithi118:> set -ex 2024-08-03T20:54:14.624 DEBUG:teuthology.orchestra.run.smithi118:> dd if=/proc/self/mounts of=/dev/stdout 2024-08-03T20:54:14.662 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052663,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497944k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248972k,nr_inodes=812243,mode=700,uid=1000,gid=1316 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497944k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-08-03T20:54:14.662 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-08-03T20:54:14.662 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-08-03T20:54:14.891 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:14.889+0000 7fe36c200640 1 -- 172.21.15.32:0/1229539681 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe364144f90 msgr2=0x7fe364074570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:14.891 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:14.889+0000 7fe36c200640 1 --2- 172.21.15.32:0/1229539681 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe364144f90 0x7fe364074570 secure :-1 s=READY pgs=877 cs=0 l=1 rev1=1 crypto rx=0x7fe364072d80 tx=0x7fe35002a560 comp rx=0 tx=0).stop 2024-08-03T20:54:14.891 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:14.889+0000 7fe36c200640 1 -- 172.21.15.32:0/1229539681 shutdown_connections 2024-08-03T20:54:14.891 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:14.889+0000 7fe36c200640 1 --2- 172.21.15.32:0/1229539681 >> v2:172.21.15.118:3300/0 conn(0x7fe36415fff0 0x7fe364160400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:14.891 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:14.889+0000 7fe36c200640 1 --2- 172.21.15.32:0/1229539681 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe364144f90 0x7fe364074570 unknown :-1 s=CLOSED pgs=877 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:14.892 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:14.889+0000 7fe36c200640 1 -- 172.21.15.32:0/1229539681 >> 172.21.15.32:0/1229539681 conn(0x7fe3640b1bd0 msgr2=0x7fe3640b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:54:14.892 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:14.889+0000 7fe36c200640 1 -- 172.21.15.32:0/1229539681 shutdown_connections 2024-08-03T20:54:14.892 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:14.889+0000 7fe36c200640 1 -- 172.21.15.32:0/1229539681 wait complete. 2024-08-03T20:54:14.893 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:14.892+0000 7fe36c200640 1 Processor -- start 2024-08-03T20:54:14.893 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:14.892+0000 7fe36c200640 1 -- start start 2024-08-03T20:54:14.895 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:14.893+0000 7fe36c200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe36406e930 0x7fe36415f480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:14.896 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:14.893+0000 7fe36c200640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fe364144f90 0x7fe36413dcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:14.896 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:14.893+0000 7fe36c200640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fe36415fff0 0x7fe364142d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:14.896 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:14.893+0000 7fe36c200640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fe3640cf840 con 0x7fe36406e930 2024-08-03T20:54:14.896 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:14.893+0000 7fe36c200640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7fe364145f30 con 0x7fe364144f90 2024-08-03T20:54:14.896 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:14.893+0000 7fe36c200640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7fe3640cf9c0 con 0x7fe36415fff0 2024-08-03T20:54:14.896 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:14.893+0000 7fe369800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe36406e930 0x7fe36415f480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:14.896 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:14.894+0000 7fe369800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe36406e930 0x7fe36415f480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:43738/0 (socket says 172.21.15.32:43738) 2024-08-03T20:54:14.896 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:14.894+0000 7fe369800640 1 -- 172.21.15.32:0/2652376089 learned_addr learned my addr 172.21.15.32:0/2652376089 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:54:14.896 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:14.894+0000 7fe368e00640 1 --2- 172.21.15.32:0/2652376089 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fe364144f90 0x7fe36413dcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:14.896 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:14.894+0000 7fe36a200640 1 --2- 172.21.15.32:0/2652376089 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fe36415fff0 0x7fe364142d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:14.896 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:14.894+0000 7fe369800640 1 -- 172.21.15.32:0/2652376089 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fe36415fff0 msgr2=0x7fe364142d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:14.896 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:14.894+0000 7fe369800640 1 --2- 172.21.15.32:0/2652376089 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fe36415fff0 0x7fe364142d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:14.896 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:14.894+0000 7fe369800640 1 -- 172.21.15.32:0/2652376089 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fe364144f90 msgr2=0x7fe36413dcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:14.897 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:14.894+0000 7fe369800640 1 --2- 172.21.15.32:0/2652376089 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fe364144f90 0x7fe36413dcf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:14.897 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:14.894+0000 7fe369800640 1 -- 172.21.15.32:0/2652376089 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe3641418a0 con 0x7fe36406e930 2024-08-03T20:54:14.897 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:14.894+0000 7fe369800640 1 --2- 172.21.15.32:0/2652376089 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe36406e930 0x7fe36415f480 secure :-1 s=READY pgs=878 cs=0 l=1 rev1=1 crypto rx=0x7fe35800b3f0 tx=0x7fe35800b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:54:14.897 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:14.895+0000 7fe361600640 1 -- 172.21.15.32:0/2652376089 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe358015020 con 0x7fe36406e930 2024-08-03T20:54:14.898 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:14.895+0000 7fe361600640 1 -- 172.21.15.32:0/2652376089 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fe358011920 con 0x7fe36406e930 2024-08-03T20:54:14.898 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:14.895+0000 7fe361600640 1 -- 172.21.15.32:0/2652376089 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe358011d70 con 0x7fe36406e930 2024-08-03T20:54:14.898 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:14.895+0000 7fe36c200640 1 -- 172.21.15.32:0/2652376089 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe364141b80 con 0x7fe36406e930 2024-08-03T20:54:14.898 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:14.895+0000 7fe36c200640 1 -- 172.21.15.32:0/2652376089 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe364142140 con 0x7fe36406e930 2024-08-03T20:54:14.898 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:14.896+0000 7fe36c200640 1 -- 172.21.15.32:0/2652376089 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe36406a430 con 0x7fe36406e930 2024-08-03T20:54:14.898 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:14.897+0000 7fe361600640 1 -- 172.21.15.32:0/2652376089 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7fe358018020 con 0x7fe36406e930 2024-08-03T20:54:14.899 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:14.898+0000 7fe361600640 1 --2- 172.21.15.32:0/2652376089 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fe344077010 0x7fe344079520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:14.900 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:14.898+0000 7fe361600640 1 -- 172.21.15.32:0/2652376089 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(74..74 src has 1..74) ==== 5029+0+0 (secure 0 0 0) 0x7fe3580982f0 con 0x7fe36406e930 2024-08-03T20:54:14.900 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:14.899+0000 7fe368e00640 1 --2- 172.21.15.32:0/2652376089 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fe344077010 0x7fe344079520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:14.900 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:14.899+0000 7fe368e00640 1 --2- 172.21.15.32:0/2652376089 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fe344077010 0x7fe344079520 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7fe350033000 tx=0x7fe350004730 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:54:14.904 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:14.903+0000 7fe361600640 1 -- 172.21.15.32:0/2652376089 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7fe36406a430 con 0x7fe36406e930 2024-08-03T20:54:15.147 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:15.146+0000 7fe36c200640 1 -- 172.21.15.32:0/2652376089 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe364160c30 con 0x7fe36406e930 2024-08-03T20:54:15.148 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:15.146+0000 7fe361600640 1 -- 172.21.15.32:0/2652376089 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 83 v83) ==== 94+0+3917 (secure 0 0 0) 0x7fe35801f030 con 0x7fe36406e930 2024-08-03T20:54:15.148 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-03T20:54:15.148 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":83,"btime":"2024-08-03T20:54:07:329942+0000","default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5666,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6837/2261538396","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6835","nonce":2261538396},{"type":"v1","addr":"172.21.15.118:6837","nonce":2261538396}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":69},{"gid":6061,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6836/1133580656","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6834","nonce":1133580656},{"type":"v1","addr":"172.21.15.118:6836","nonce":1133580656}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":68},{"gid":6618,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6835/3466281885","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6833","nonce":3466281885},{"type":"v1","addr":"172.21.15.32:6835","nonce":3466281885}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":79},{"gid":6729,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6834/1755103888","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":1755103888},{"type":"v1","addr":"172.21.15.32:6834","nonce":1755103888}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":83}],"filesystems":[]} 2024-08-03T20:54:15.148 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 83 2024-08-03T20:54:15.150 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:15.149+0000 7fe36c200640 1 -- 172.21.15.32:0/2652376089 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fe344077010 msgr2=0x7fe344079520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:15.151 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:15.149+0000 7fe36c200640 1 --2- 172.21.15.32:0/2652376089 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fe344077010 0x7fe344079520 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7fe350033000 tx=0x7fe350004730 comp rx=0 tx=0).stop 2024-08-03T20:54:15.151 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:15.149+0000 7fe36c200640 1 -- 172.21.15.32:0/2652376089 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe36406e930 msgr2=0x7fe36415f480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:15.151 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:15.149+0000 7fe36c200640 1 --2- 172.21.15.32:0/2652376089 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe36406e930 0x7fe36415f480 secure :-1 s=READY pgs=878 cs=0 l=1 rev1=1 crypto rx=0x7fe35800b3f0 tx=0x7fe35800b8b0 comp rx=0 tx=0).stop 2024-08-03T20:54:15.151 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:15.149+0000 7fe36c200640 1 -- 172.21.15.32:0/2652376089 shutdown_connections 2024-08-03T20:54:15.151 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:15.149+0000 7fe36c200640 1 --2- 172.21.15.32:0/2652376089 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fe344077010 0x7fe344079520 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:15.151 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:15.149+0000 7fe36c200640 1 --2- 172.21.15.32:0/2652376089 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fe364144f90 0x7fe36413dcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:15.151 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:15.149+0000 7fe36c200640 1 --2- 172.21.15.32:0/2652376089 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fe36415fff0 0x7fe364142d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:15.151 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:15.149+0000 7fe36c200640 1 --2- 172.21.15.32:0/2652376089 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe36406e930 0x7fe36415f480 unknown :-1 s=CLOSED pgs=878 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:15.151 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:15.149+0000 7fe36c200640 1 -- 172.21.15.32:0/2652376089 >> 172.21.15.32:0/2652376089 conn(0x7fe3640b1bd0 msgr2=0x7fe3640b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:54:15.151 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:15.149+0000 7fe36c200640 1 -- 172.21.15.32:0/2652376089 shutdown_connections 2024-08-03T20:54:15.151 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:15.149+0000 7fe36c200640 1 -- 172.21.15.32:0/2652376089 wait complete. 2024-08-03T20:54:15.172 INFO:tasks.ceph.mds.a:Restarting daemon 2024-08-03T20:54:15.172 INFO:tasks.ceph.mds.a:Stopping old one... 2024-08-03T20:54:15.172 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-08-03T20:54:15.173 INFO:teuthology.orchestra.run:waiting for 300 2024-08-03T20:54:15.173 INFO:tasks.ceph.mds.c:Restarting daemon 2024-08-03T20:54:15.173 INFO:tasks.ceph.mds.c:Stopping old one... 2024-08-03T20:54:15.173 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-08-03T20:54:15.173 INFO:teuthology.orchestra.run:waiting for 300 2024-08-03T20:54:15.173 INFO:tasks.ceph.mds.b:Restarting daemon 2024-08-03T20:54:15.173 INFO:tasks.ceph.mds.b:Stopping old one... 2024-08-03T20:54:15.174 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-08-03T20:54:15.174 INFO:teuthology.orchestra.run:waiting for 300 2024-08-03T20:54:15.174 INFO:tasks.ceph.mds.d:Restarting daemon 2024-08-03T20:54:15.174 INFO:tasks.ceph.mds.d:Stopping old one... 2024-08-03T20:54:15.174 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-08-03T20:54:15.174 INFO:teuthology.orchestra.run:waiting for 300 2024-08-03T20:54:15.196 INFO:tasks.ceph.mds.c:Stopped 2024-08-03T20:54:15.197 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-08-03T20:54:15.198 INFO:tasks.ceph.mds.d:Stopped 2024-08-03T20:54:15.198 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-08-03T20:54:15.199 INFO:tasks.ceph.mds.a:Stopped 2024-08-03T20:54:15.200 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-08-03T20:54:15.243 INFO:tasks.ceph.mds.c:Started 2024-08-03T20:54:15.243 DEBUG:teuthology.parallel:result is None 2024-08-03T20:54:15.243 INFO:tasks.ceph.mds.d:Started 2024-08-03T20:54:15.244 DEBUG:teuthology.parallel:result is None 2024-08-03T20:54:15.246 INFO:tasks.ceph.mds.a:Started 2024-08-03T20:54:15.246 DEBUG:teuthology.parallel:result is None 2024-08-03T20:54:15.434 INFO:tasks.ceph.mds.d.smithi118.stdout:starting mds.d at 2024-08-03T20:54:15.438 INFO:tasks.ceph.mds.c.smithi032.stdout:starting mds.c at 2024-08-03T20:54:15.439 INFO:tasks.ceph.mds.a.smithi032.stdout:starting mds.a at 2024-08-03T20:54:17.933 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~0s 2024-08-03T20:54:21.199 INFO:tasks.ceph.mds.b:Stopped 2024-08-03T20:54:21.199 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-08-03T20:54:21.204 INFO:tasks.ceph.mds.b:Started 2024-08-03T20:54:21.204 DEBUG:teuthology.parallel:result is None 2024-08-03T20:54:21.205 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-08-03T20:54:21.402 INFO:tasks.ceph.mds.b.smithi118.stdout:starting mds.b at 2024-08-03T20:54:21.445 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:21.444+0000 7fa1f1600640 1 -- 172.21.15.32:0/2960354987 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fa1ec144f90 msgr2=0x7fa1ec074570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:21.446 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:21.444+0000 7fa1f1600640 1 --2- 172.21.15.32:0/2960354987 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fa1ec144f90 0x7fa1ec074570 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fa1d4009860 tx=0x7fa1d402a530 comp rx=0 tx=0).stop 2024-08-03T20:54:21.446 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:21.444+0000 7fa1f1600640 1 -- 172.21.15.32:0/2960354987 shutdown_connections 2024-08-03T20:54:21.446 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:21.444+0000 7fa1f1600640 1 --2- 172.21.15.32:0/2960354987 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fa1ec144f90 0x7fa1ec074570 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:21.446 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:21.444+0000 7fa1f1600640 1 -- 172.21.15.32:0/2960354987 >> 172.21.15.32:0/2960354987 conn(0x7fa1ec0b1bd0 msgr2=0x7fa1ec0b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:54:21.446 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:21.444+0000 7fa1f1600640 1 -- 172.21.15.32:0/2960354987 shutdown_connections 2024-08-03T20:54:21.446 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:21.444+0000 7fa1f1600640 1 -- 172.21.15.32:0/2960354987 wait complete. 2024-08-03T20:54:21.447 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:21.446+0000 7fa1f1600640 1 Processor -- start 2024-08-03T20:54:21.448 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:21.446+0000 7fa1f1600640 1 -- start start 2024-08-03T20:54:21.452 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:21.448+0000 7fa1f1600640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa1ec06e930 0x7fa1ec1e7a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:21.452 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:21.448+0000 7fa1f1600640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fa1ec144f90 0x7fa1ec1e8210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:21.452 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:21.448+0000 7fa1f1600640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa1ec15fff0 0x7fa1ec14c5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:21.452 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:21.448+0000 7fa1f1600640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fa1ec0cf840 con 0x7fa1ec06e930 2024-08-03T20:54:21.452 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:21.448+0000 7fa1f1600640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7fa1ec145f30 con 0x7fa1ec15fff0 2024-08-03T20:54:21.452 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:21.448+0000 7fa1f1600640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7fa1ec0cf9c0 con 0x7fa1ec144f90 2024-08-03T20:54:21.452 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:21.448+0000 7fa1eaa00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa1ec06e930 0x7fa1ec1e7a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:21.452 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:21.448+0000 7fa1eaa00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa1ec06e930 0x7fa1ec1e7a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:50236/0 (socket says 172.21.15.32:50236) 2024-08-03T20:54:21.452 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:21.448+0000 7fa1eaa00640 1 -- 172.21.15.32:0/3691451002 learned_addr learned my addr 172.21.15.32:0/3691451002 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:54:21.452 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:21.448+0000 7fa1eb400640 1 --2- 172.21.15.32:0/3691451002 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa1ec15fff0 0x7fa1ec14c5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:21.452 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:21.448+0000 7fa1e3e00640 1 --2- 172.21.15.32:0/3691451002 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fa1ec144f90 0x7fa1ec1e8210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:21.453 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:21.448+0000 7fa1eaa00640 1 -- 172.21.15.32:0/3691451002 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fa1ec144f90 msgr2=0x7fa1ec1e8210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:21.453 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:21.448+0000 7fa1eaa00640 1 --2- 172.21.15.32:0/3691451002 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fa1ec144f90 0x7fa1ec1e8210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:21.453 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:21.448+0000 7fa1eaa00640 1 -- 172.21.15.32:0/3691451002 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa1ec15fff0 msgr2=0x7fa1ec14c5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:21.453 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:21.448+0000 7fa1eaa00640 1 --2- 172.21.15.32:0/3691451002 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa1ec15fff0 0x7fa1ec14c5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:21.453 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:21.448+0000 7fa1eaa00640 1 -- 172.21.15.32:0/3691451002 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa1ec1e69f0 con 0x7fa1ec06e930 2024-08-03T20:54:21.453 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:21.449+0000 7fa1eaa00640 1 --2- 172.21.15.32:0/3691451002 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa1ec06e930 0x7fa1ec1e7a90 secure :-1 s=READY pgs=885 cs=0 l=1 rev1=1 crypto rx=0x7fa1dc00c500 tx=0x7fa1dc00c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:54:21.453 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:21.449+0000 7fa1e2a00640 1 -- 172.21.15.32:0/3691451002 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1dc016020 con 0x7fa1ec06e930 2024-08-03T20:54:21.453 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:21.449+0000 7fa1e2a00640 1 -- 172.21.15.32:0/3691451002 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fa1dc012920 con 0x7fa1ec06e930 2024-08-03T20:54:21.453 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:21.449+0000 7fa1e2a00640 1 -- 172.21.15.32:0/3691451002 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1dc012d30 con 0x7fa1ec06e930 2024-08-03T20:54:21.454 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:21.449+0000 7fa1f1600640 1 -- 172.21.15.32:0/3691451002 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa1ec1e6cd0 con 0x7fa1ec06e930 2024-08-03T20:54:21.454 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:21.449+0000 7fa1f1600640 1 -- 172.21.15.32:0/3691451002 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa1ec1e7290 con 0x7fa1ec06e930 2024-08-03T20:54:21.454 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:21.450+0000 7fa1f1600640 1 -- 172.21.15.32:0/3691451002 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa1ec160400 con 0x7fa1ec06e930 2024-08-03T20:54:21.455 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:21.451+0000 7fa1e2a00640 1 -- 172.21.15.32:0/3691451002 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7fa1dc019020 con 0x7fa1ec06e930 2024-08-03T20:54:21.455 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:21.452+0000 7fa1e2a00640 1 --2- 172.21.15.32:0/3691451002 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fa1c8077010 0x7fa1c8079520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:21.455 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:21.452+0000 7fa1e2a00640 1 -- 172.21.15.32:0/3691451002 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 5029+0+0 (secure 0 0 0) 0x7fa1dc098400 con 0x7fa1ec06e930 2024-08-03T20:54:21.455 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:21.453+0000 7fa1e3e00640 1 --2- 172.21.15.32:0/3691451002 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fa1c8077010 0x7fa1c8079520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:21.455 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:21.453+0000 7fa1e3e00640 1 --2- 172.21.15.32:0/3691451002 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fa1c8077010 0x7fa1c8079520 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7fa1d4034000 tx=0x7fa1d4004730 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:54:21.458 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:21.457+0000 7fa1e2a00640 1 -- 172.21.15.32:0/3691451002 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7fa1dc020030 con 0x7fa1ec06e930 2024-08-03T20:54:21.674 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:21.672+0000 7fa1f1600640 1 -- 172.21.15.32:0/3691451002 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) -- 0x7fa1ec0745c0 con 0x7fa1ec06e930 2024-08-03T20:54:23.391 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:23.390+0000 7fa1e2a00640 1 -- 172.21.15.32:0/3691451002 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v79) ==== 97+0+0 (secure 0 0 0) 0x7fa1dc0611c0 con 0x7fa1ec06e930 2024-08-03T20:54:23.391 INFO:teuthology.orchestra.run.smithi032.stderr: removed all blocklist entries 2024-08-03T20:54:23.395 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:23.393+0000 7fa1f1600640 1 -- 172.21.15.32:0/3691451002 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fa1c8077010 msgr2=0x7fa1c8079520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:23.395 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:23.393+0000 7fa1f1600640 1 --2- 172.21.15.32:0/3691451002 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fa1c8077010 0x7fa1c8079520 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7fa1d4034000 tx=0x7fa1d4004730 comp rx=0 tx=0).stop 2024-08-03T20:54:23.395 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:23.393+0000 7fa1f1600640 1 -- 172.21.15.32:0/3691451002 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa1ec06e930 msgr2=0x7fa1ec1e7a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:23.395 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:23.393+0000 7fa1f1600640 1 --2- 172.21.15.32:0/3691451002 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa1ec06e930 0x7fa1ec1e7a90 secure :-1 s=READY pgs=885 cs=0 l=1 rev1=1 crypto rx=0x7fa1dc00c500 tx=0x7fa1dc00c9c0 comp rx=0 tx=0).stop 2024-08-03T20:54:23.395 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:23.393+0000 7fa1f1600640 1 -- 172.21.15.32:0/3691451002 shutdown_connections 2024-08-03T20:54:23.395 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:23.393+0000 7fa1f1600640 1 --2- 172.21.15.32:0/3691451002 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fa1c8077010 0x7fa1c8079520 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:23.396 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:23.393+0000 7fa1f1600640 1 --2- 172.21.15.32:0/3691451002 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa1ec15fff0 0x7fa1ec14c5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:23.396 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:23.393+0000 7fa1f1600640 1 --2- 172.21.15.32:0/3691451002 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fa1ec144f90 0x7fa1ec1e8210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:23.396 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:23.393+0000 7fa1f1600640 1 --2- 172.21.15.32:0/3691451002 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa1ec06e930 0x7fa1ec1e7a90 unknown :-1 s=CLOSED pgs=885 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:23.396 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:23.393+0000 7fa1f1600640 1 -- 172.21.15.32:0/3691451002 >> 172.21.15.32:0/3691451002 conn(0x7fa1ec0b1bd0 msgr2=0x7fa1ec0b0cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:54:23.396 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:23.394+0000 7fa1f1600640 1 -- 172.21.15.32:0/3691451002 shutdown_connections 2024-08-03T20:54:23.396 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:23.394+0000 7fa1f1600640 1 -- 172.21.15.32:0/3691451002 wait complete. 2024-08-03T20:54:23.422 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-08-03T20:54:23.656 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:23.653+0000 7f6f06a00640 1 -- 172.21.15.32:0/1188973526 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6f00144f90 msgr2=0x7f6f00074570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:23.656 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:23.653+0000 7f6f06a00640 1 --2- 172.21.15.32:0/1188973526 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6f00144f90 0x7f6f00074570 secure :-1 s=READY pgs=886 cs=0 l=1 rev1=1 crypto rx=0x7f6ee8005fd0 tx=0x7f6ee802a530 comp rx=0 tx=0).stop 2024-08-03T20:54:23.656 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:23.653+0000 7f6f06a00640 1 -- 172.21.15.32:0/1188973526 shutdown_connections 2024-08-03T20:54:23.656 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:23.653+0000 7f6f06a00640 1 --2- 172.21.15.32:0/1188973526 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f6f0015fff0 0x7f6f00160400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:23.656 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:23.653+0000 7f6f06a00640 1 --2- 172.21.15.32:0/1188973526 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6f00144f90 0x7f6f00074570 unknown :-1 s=CLOSED pgs=886 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:23.656 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:23.653+0000 7f6f06a00640 1 -- 172.21.15.32:0/1188973526 >> 172.21.15.32:0/1188973526 conn(0x7f6f000b1bd0 msgr2=0x7f6f000b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:54:23.657 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:23.653+0000 7f6f06a00640 1 -- 172.21.15.32:0/1188973526 shutdown_connections 2024-08-03T20:54:23.657 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:23.653+0000 7f6f06a00640 1 -- 172.21.15.32:0/1188973526 wait complete. 2024-08-03T20:54:23.657 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:23.656+0000 7f6f06a00640 1 Processor -- start 2024-08-03T20:54:23.657 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:23.656+0000 7f6f06a00640 1 -- start start 2024-08-03T20:54:23.659 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:23.657+0000 7f6f06a00640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f6f0006e930 0x7f6f0015da80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:23.659 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:23.657+0000 7f6f06a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6f00144f90 0x7f6f00152080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:23.660 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:23.657+0000 7f6f06a00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f6f0015fff0 0x7f6f001f8b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:23.660 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:23.657+0000 7f6f06a00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f6f000cf840 con 0x7f6f00144f90 2024-08-03T20:54:23.660 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:23.657+0000 7f6f06a00640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f6f00145f30 con 0x7f6f0015fff0 2024-08-03T20:54:23.660 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:23.657+0000 7f6f06a00640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7f6f000cf9c0 con 0x7f6f0006e930 2024-08-03T20:54:23.660 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:23.657+0000 7f6ef7400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6f00144f90 0x7f6f00152080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:23.660 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:23.658+0000 7f6ef7400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6f00144f90 0x7f6f00152080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:50252/0 (socket says 172.21.15.32:50252) 2024-08-03T20:54:23.660 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:23.658+0000 7f6ef7400640 1 -- 172.21.15.32:0/4139873347 learned_addr learned my addr 172.21.15.32:0/4139873347 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:54:23.660 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:23.658+0000 7f6effe00640 1 --2- 172.21.15.32:0/4139873347 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f6f0006e930 0x7f6f0015da80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:23.660 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:23.658+0000 7f6f04a00640 1 --2- 172.21.15.32:0/4139873347 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f6f0015fff0 0x7f6f001f8b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:23.660 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:23.658+0000 7f6ef7400640 1 -- 172.21.15.32:0/4139873347 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f6f0006e930 msgr2=0x7f6f0015da80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:23.660 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:23.658+0000 7f6ef7400640 1 --2- 172.21.15.32:0/4139873347 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f6f0006e930 0x7f6f0015da80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:23.661 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:23.658+0000 7f6ef7400640 1 -- 172.21.15.32:0/4139873347 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f6f0015fff0 msgr2=0x7f6f001f8b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:23.661 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:23.658+0000 7f6ef7400640 1 --2- 172.21.15.32:0/4139873347 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f6f0015fff0 0x7f6f001f8b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:23.661 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:23.658+0000 7f6ef7400640 1 -- 172.21.15.32:0/4139873347 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6f0015e3b0 con 0x7f6f00144f90 2024-08-03T20:54:23.661 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:23.658+0000 7f6ef7400640 1 --2- 172.21.15.32:0/4139873347 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6f00144f90 0x7f6f00152080 secure :-1 s=READY pgs=887 cs=0 l=1 rev1=1 crypto rx=0x7f6ee8006010 tx=0x7f6ee80038a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:54:23.662 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:23.659+0000 7f6efcc00640 1 -- 172.21.15.32:0/4139873347 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ee803e030 con 0x7f6f00144f90 2024-08-03T20:54:23.662 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:23.659+0000 7f6efcc00640 1 -- 172.21.15.32:0/4139873347 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f6ee8003d90 con 0x7f6f00144f90 2024-08-03T20:54:23.662 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:23.659+0000 7f6efcc00640 1 -- 172.21.15.32:0/4139873347 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ee8035040 con 0x7f6f00144f90 2024-08-03T20:54:23.662 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:23.659+0000 7f6f06a00640 1 -- 172.21.15.32:0/4139873347 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6f0015e690 con 0x7f6f00144f90 2024-08-03T20:54:23.662 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:23.659+0000 7f6f06a00640 1 -- 172.21.15.32:0/4139873347 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6f0015ec50 con 0x7f6f00144f90 2024-08-03T20:54:23.662 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:23.660+0000 7f6f06a00640 1 -- 172.21.15.32:0/4139873347 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6f0015e120 con 0x7f6f00144f90 2024-08-03T20:54:23.662 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:23.661+0000 7f6efcc00640 1 -- 172.21.15.32:0/4139873347 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f6ee8041030 con 0x7f6f00144f90 2024-08-03T20:54:23.664 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:23.662+0000 7f6efcc00640 1 --2- 172.21.15.32:0/4139873347 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f6edc077010 0x7f6edc079520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:23.664 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:23.662+0000 7f6efcc00640 1 -- 172.21.15.32:0/4139873347 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(79..79 src has 1..79) ==== 4857+0+0 (secure 0 0 0) 0x7f6ee80b8d40 con 0x7f6f00144f90 2024-08-03T20:54:23.664 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:23.663+0000 7f6effe00640 1 --2- 172.21.15.32:0/4139873347 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f6edc077010 0x7f6edc079520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:23.665 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:23.664+0000 7f6effe00640 1 --2- 172.21.15.32:0/4139873347 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f6edc077010 0x7f6edc079520 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f6ef0006fd0 tx=0x7f6ef0009060 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:54:23.668 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:23.667+0000 7f6efcc00640 1 -- 172.21.15.32:0/4139873347 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7f6ee8081ba0 con 0x7f6f00144f90 2024-08-03T20:54:23.945 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:23.943+0000 7f6f06a00640 1 -- 172.21.15.32:0/4139873347 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) -- 0x7f6f00160cc0 con 0x7f6f00144f90 2024-08-03T20:54:23.945 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:23.944+0000 7f6efcc00640 1 -- 172.21.15.32:0/4139873347 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v14) ==== 80+0+5304 (secure 0 0 0) 0x7f6ee80860f0 con 0x7f6f00144f90 2024-08-03T20:54:23.946 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-03T20:54:23.946 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-08-03T20:54:23.946 INFO:teuthology.orchestra.run.smithi032.stdout: "auth_dump": [ 2024-08-03T20:54:23.946 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-03T20:54:23.946 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "mds.a", 2024-08-03T20:54:23.946 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQAvl65mfaKbOhAAUgCH82U0SB7owF5Cds3XYA==", 2024-08-03T20:54:23.946 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2024-08-03T20:54:23.946 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": "allow", 2024-08-03T20:54:23.946 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow *", 2024-08-03T20:54:23.946 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow *", 2024-08-03T20:54:23.946 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2024-08-03T20:54:23.946 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-03T20:54:23.947 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-03T20:54:23.947 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-03T20:54:23.947 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "mds.b", 2024-08-03T20:54:23.947 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQAwl65mNcoVGxAAlRXNjZhxVu8Vjslk9Msrxw==", 2024-08-03T20:54:23.947 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2024-08-03T20:54:23.947 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": "allow", 2024-08-03T20:54:23.947 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow *", 2024-08-03T20:54:23.947 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow *", 2024-08-03T20:54:23.947 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2024-08-03T20:54:23.947 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-03T20:54:23.947 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-03T20:54:23.947 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-03T20:54:23.947 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "mds.c", 2024-08-03T20:54:23.947 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQAwl65mEUAFDRAAGdw1cymfvfKxGeJS/Od6Rw==", 2024-08-03T20:54:23.948 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2024-08-03T20:54:23.948 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": "allow", 2024-08-03T20:54:23.948 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow *", 2024-08-03T20:54:23.948 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow *", 2024-08-03T20:54:23.948 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2024-08-03T20:54:23.948 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-03T20:54:23.948 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-03T20:54:23.948 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-03T20:54:23.948 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "mds.d", 2024-08-03T20:54:23.948 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQAwl65mjUHZKBAAqOCnIopBK8Rhk36hrlNnzw==", 2024-08-03T20:54:23.948 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2024-08-03T20:54:23.948 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": "allow", 2024-08-03T20:54:23.948 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow *", 2024-08-03T20:54:23.949 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow *", 2024-08-03T20:54:23.949 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2024-08-03T20:54:23.949 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-03T20:54:23.949 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-03T20:54:23.949 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-03T20:54:23.949 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "osd.0", 2024-08-03T20:54:23.949 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQAyl65m4kIyLxAAdCh0D2MjJNPPu5b7jNkrbw==", 2024-08-03T20:54:23.949 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2024-08-03T20:54:23.949 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow profile osd", 2024-08-03T20:54:23.949 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow profile osd", 2024-08-03T20:54:23.949 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2024-08-03T20:54:23.949 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-03T20:54:23.949 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-03T20:54:23.949 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-03T20:54:23.950 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "osd.1", 2024-08-03T20:54:23.950 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQA1l65mLzVOOhAA081V1cUyYFPz17L0nny7gA==", 2024-08-03T20:54:23.950 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2024-08-03T20:54:23.950 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow profile osd", 2024-08-03T20:54:23.950 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow profile osd", 2024-08-03T20:54:23.950 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2024-08-03T20:54:23.950 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-03T20:54:23.950 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-03T20:54:23.950 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-03T20:54:23.950 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "osd.2", 2024-08-03T20:54:23.950 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQA5l65mBnXDBxAA9/lZVkwD2ykMTOIysoYODg==", 2024-08-03T20:54:23.950 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2024-08-03T20:54:23.951 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow profile osd", 2024-08-03T20:54:23.951 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow profile osd", 2024-08-03T20:54:23.951 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2024-08-03T20:54:23.951 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-03T20:54:23.951 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-03T20:54:23.951 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-03T20:54:23.951 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "osd.3", 2024-08-03T20:54:23.951 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQA8l65mAriiERAAfzV8An67+PnA3EzeAutIbA==", 2024-08-03T20:54:23.951 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2024-08-03T20:54:23.951 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow profile osd", 2024-08-03T20:54:23.952 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow profile osd", 2024-08-03T20:54:23.952 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2024-08-03T20:54:23.952 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-03T20:54:23.952 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-03T20:54:23.952 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-03T20:54:23.952 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "osd.4", 2024-08-03T20:54:23.952 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQBAl65m5uJSJRAA8S2IjJpxuqHY8YK4tTDMPg==", 2024-08-03T20:54:23.952 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2024-08-03T20:54:23.952 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow profile osd", 2024-08-03T20:54:23.952 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow profile osd", 2024-08-03T20:54:23.952 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2024-08-03T20:54:23.952 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-03T20:54:23.952 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-03T20:54:23.952 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-03T20:54:23.953 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "osd.5", 2024-08-03T20:54:23.953 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQBDl65mBP3JLBAADgK2X3zNyfnjjZADX1PFcA==", 2024-08-03T20:54:23.953 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2024-08-03T20:54:23.953 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow profile osd", 2024-08-03T20:54:23.953 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow profile osd", 2024-08-03T20:54:23.953 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2024-08-03T20:54:23.953 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-03T20:54:23.953 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-03T20:54:23.953 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-03T20:54:23.953 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "osd.6", 2024-08-03T20:54:23.953 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQBGl65m8DBNNRAANeOUfW70S5JfqFlpuMF9KQ==", 2024-08-03T20:54:23.953 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2024-08-03T20:54:23.953 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow profile osd", 2024-08-03T20:54:23.954 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow profile osd", 2024-08-03T20:54:23.954 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2024-08-03T20:54:23.954 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-03T20:54:23.954 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-03T20:54:23.954 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-03T20:54:23.954 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "osd.7", 2024-08-03T20:54:23.954 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQBKl65mUnOcABAA41wylzC1Ec9Fj01Ke0HhTw==", 2024-08-03T20:54:23.954 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2024-08-03T20:54:23.954 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow profile osd", 2024-08-03T20:54:23.954 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow profile osd", 2024-08-03T20:54:23.954 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2024-08-03T20:54:23.954 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-03T20:54:23.954 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-03T20:54:23.954 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-03T20:54:23.955 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "client.0", 2024-08-03T20:54:23.955 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQAwl65msgO3NRAAHG/WuMTK9c4lDXocXvqWnA==", 2024-08-03T20:54:23.955 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2024-08-03T20:54:23.955 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": "allow", 2024-08-03T20:54:23.955 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow r", 2024-08-03T20:54:23.955 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-08-03T20:54:23.955 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-03T20:54:23.955 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-03T20:54:23.955 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-03T20:54:23.955 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "client.1", 2024-08-03T20:54:23.955 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQAxl65mbe19DxAA+sepc9ucXk2Bqy7C90mfbA==", 2024-08-03T20:54:23.955 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2024-08-03T20:54:23.955 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": "allow", 2024-08-03T20:54:23.956 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow r", 2024-08-03T20:54:23.956 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow rw", 2024-08-03T20:54:23.956 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow rwx" 2024-08-03T20:54:23.956 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-03T20:54:23.956 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-03T20:54:23.956 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-03T20:54:23.956 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "client.2", 2024-08-03T20:54:23.957 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQAxl65mBnbkBBAAjor60WWLTKXidiBMgtKuEA==", 2024-08-03T20:54:23.957 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2024-08-03T20:54:23.957 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": "allow", 2024-08-03T20:54:23.957 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow r", 2024-08-03T20:54:23.957 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow rw", 2024-08-03T20:54:23.957 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow rwx" 2024-08-03T20:54:23.957 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-03T20:54:23.957 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-03T20:54:23.957 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-03T20:54:23.957 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "client.3", 2024-08-03T20:54:23.957 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQAxl65munz8GRAA/cRl3ghwQE42FOizurcW3g==", 2024-08-03T20:54:23.957 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2024-08-03T20:54:23.957 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": "allow", 2024-08-03T20:54:23.958 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow r", 2024-08-03T20:54:23.958 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow rw", 2024-08-03T20:54:23.958 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow rwx" 2024-08-03T20:54:23.958 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-03T20:54:23.958 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-03T20:54:23.958 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-03T20:54:23.958 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "client.admin", 2024-08-03T20:54:23.958 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQAul65mB8VcOhAA105JQ2FX1vhtaGj3Fer8mg==", 2024-08-03T20:54:23.958 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2024-08-03T20:54:23.958 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": "allow *", 2024-08-03T20:54:23.958 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow *", 2024-08-03T20:54:23.958 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow *", 2024-08-03T20:54:23.958 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2024-08-03T20:54:23.959 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-03T20:54:23.959 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-03T20:54:23.959 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-03T20:54:23.959 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "mgr.x", 2024-08-03T20:54:23.959 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQAvl65mUaPKLxAAnll74Gmk8nSuN/HuX6rTxA==", 2024-08-03T20:54:23.959 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2024-08-03T20:54:23.959 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": "allow *", 2024-08-03T20:54:23.959 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow profile mgr", 2024-08-03T20:54:23.959 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2024-08-03T20:54:23.959 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-03T20:54:23.959 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-03T20:54:23.959 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-03T20:54:23.959 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "mgr.y", 2024-08-03T20:54:23.960 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQAvl65m5Gq4JBAA306Ge1zHF3Z0WIzUrset4A==", 2024-08-03T20:54:23.960 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2024-08-03T20:54:23.960 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": "allow *", 2024-08-03T20:54:23.960 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow profile mgr", 2024-08-03T20:54:23.960 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2024-08-03T20:54:23.960 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-03T20:54:23.960 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-03T20:54:23.960 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-03T20:54:23.960 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-08-03T20:54:23.961 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:23.946+0000 7f6f06a00640 1 -- 172.21.15.32:0/4139873347 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f6edc077010 msgr2=0x7f6edc079520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:23.961 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:23.946+0000 7f6f06a00640 1 --2- 172.21.15.32:0/4139873347 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f6edc077010 0x7f6edc079520 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f6ef0006fd0 tx=0x7f6ef0009060 comp rx=0 tx=0).stop 2024-08-03T20:54:23.961 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:23.946+0000 7f6f06a00640 1 -- 172.21.15.32:0/4139873347 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6f00144f90 msgr2=0x7f6f00152080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:23.961 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:23.946+0000 7f6f06a00640 1 --2- 172.21.15.32:0/4139873347 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6f00144f90 0x7f6f00152080 secure :-1 s=READY pgs=887 cs=0 l=1 rev1=1 crypto rx=0x7f6ee8006010 tx=0x7f6ee80038a0 comp rx=0 tx=0).stop 2024-08-03T20:54:23.961 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:23.946+0000 7f6f06a00640 1 -- 172.21.15.32:0/4139873347 shutdown_connections 2024-08-03T20:54:23.961 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:23.946+0000 7f6f06a00640 1 --2- 172.21.15.32:0/4139873347 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f6edc077010 0x7f6edc079520 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:23.962 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:23.946+0000 7f6f06a00640 1 --2- 172.21.15.32:0/4139873347 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f6f0015fff0 0x7f6f001f8b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:23.962 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:23.946+0000 7f6f06a00640 1 --2- 172.21.15.32:0/4139873347 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6f00144f90 0x7f6f00152080 unknown :-1 s=CLOSED pgs=887 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:23.962 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:23.946+0000 7f6f06a00640 1 --2- 172.21.15.32:0/4139873347 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f6f0006e930 0x7f6f0015da80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:23.962 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:23.946+0000 7f6f06a00640 1 -- 172.21.15.32:0/4139873347 >> 172.21.15.32:0/4139873347 conn(0x7f6f000b1bd0 msgr2=0x7f6f000b3930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:54:23.962 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:23.946+0000 7f6f06a00640 1 -- 172.21.15.32:0/4139873347 shutdown_connections 2024-08-03T20:54:23.962 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:23.946+0000 7f6f06a00640 1 -- 172.21.15.32:0/4139873347 wait complete. 2024-08-03T20:54:23.974 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-08-03T20:54:24.204 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:24.202+0000 7f6695000640 1 -- 172.21.15.32:0/1228278013 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6690165160 msgr2=0x7f6690167580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:24.204 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:24.202+0000 7f6695000640 1 --2- 172.21.15.32:0/1228278013 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6690165160 0x7f6690167580 secure :-1 s=READY pgs=889 cs=0 l=1 rev1=1 crypto rx=0x7f6688005fd0 tx=0x7f668802a8b0 comp rx=0 tx=0).stop 2024-08-03T20:54:24.204 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:24.203+0000 7f6695000640 1 -- 172.21.15.32:0/1228278013 shutdown_connections 2024-08-03T20:54:24.205 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:24.203+0000 7f6695000640 1 --2- 172.21.15.32:0/1228278013 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6690165160 0x7f6690167580 unknown :-1 s=CLOSED pgs=889 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:24.205 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:24.203+0000 7f6695000640 1 -- 172.21.15.32:0/1228278013 >> 172.21.15.32:0/1228278013 conn(0x7f66900b1bd0 msgr2=0x7f66900b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:54:24.205 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:24.203+0000 7f6695000640 1 -- 172.21.15.32:0/1228278013 shutdown_connections 2024-08-03T20:54:24.205 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:24.203+0000 7f6695000640 1 -- 172.21.15.32:0/1228278013 wait complete. 2024-08-03T20:54:24.206 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:24.205+0000 7f6695000640 1 Processor -- start 2024-08-03T20:54:24.206 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:24.205+0000 7f6695000640 1 -- start start 2024-08-03T20:54:24.209 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:24.206+0000 7f6695000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f669006e930 0x7f66901f8300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:24.209 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:24.207+0000 7f6695000640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f6690144f90 0x7f669013e960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:24.209 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:24.207+0000 7f6695000640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f6690165160 0x7f6690166b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:24.209 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:24.207+0000 7f6695000640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f66900cf9c0 con 0x7f669006e930 2024-08-03T20:54:24.209 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:24.207+0000 7f6695000640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f66900cf840 con 0x7f6690144f90 2024-08-03T20:54:24.209 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:24.207+0000 7f6695000640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7f6690167ac0 con 0x7f6690165160 2024-08-03T20:54:24.209 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:24.207+0000 7f668de00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f669006e930 0x7f66901f8300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:24.209 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:24.207+0000 7f668de00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f669006e930 0x7f66901f8300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:50270/0 (socket says 172.21.15.32:50270) 2024-08-03T20:54:24.210 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:24.207+0000 7f668de00640 1 -- 172.21.15.32:0/3639091435 learned_addr learned my addr 172.21.15.32:0/3639091435 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:54:24.210 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:24.207+0000 7f668e800640 1 --2- 172.21.15.32:0/3639091435 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f6690165160 0x7f6690166b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:24.210 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:24.207+0000 7f668d400640 1 --2- 172.21.15.32:0/3639091435 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f6690144f90 0x7f669013e960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:24.210 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:24.207+0000 7f668de00640 1 -- 172.21.15.32:0/3639091435 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f6690165160 msgr2=0x7f6690166b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:24.210 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:24.207+0000 7f668de00640 1 --2- 172.21.15.32:0/3639091435 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f6690165160 0x7f6690166b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:24.210 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:24.207+0000 7f668de00640 1 -- 172.21.15.32:0/3639091435 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f6690144f90 msgr2=0x7f669013e960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:24.210 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:24.207+0000 7f668de00640 1 --2- 172.21.15.32:0/3639091435 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f6690144f90 0x7f669013e960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:24.210 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:24.207+0000 7f668de00640 1 -- 172.21.15.32:0/3639091435 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f669013eea0 con 0x7f669006e930 2024-08-03T20:54:24.210 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:24.208+0000 7f668de00640 1 --2- 172.21.15.32:0/3639091435 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f669006e930 0x7f66901f8300 secure :-1 s=READY pgs=890 cs=0 l=1 rev1=1 crypto rx=0x7f668000c430 tx=0x7f668000c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:54:24.211 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:24.208+0000 7f6686000640 1 -- 172.21.15.32:0/3639091435 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6680017070 con 0x7f669006e930 2024-08-03T20:54:24.211 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:24.208+0000 7f6686000640 1 -- 172.21.15.32:0/3639091435 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f66800053c0 con 0x7f669006e930 2024-08-03T20:54:24.211 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:24.208+0000 7f6686000640 1 -- 172.21.15.32:0/3639091435 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6680003850 con 0x7f669006e930 2024-08-03T20:54:24.211 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:24.208+0000 7f6695000640 1 -- 172.21.15.32:0/3639091435 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f66901f8c80 con 0x7f669006e930 2024-08-03T20:54:24.211 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:24.208+0000 7f6695000640 1 -- 172.21.15.32:0/3639091435 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f66901f9240 con 0x7f669006e930 2024-08-03T20:54:24.211 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:24.209+0000 7f6695000640 1 -- 172.21.15.32:0/3639091435 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6690073e60 con 0x7f669006e930 2024-08-03T20:54:24.212 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:24.210+0000 7f6686000640 1 -- 172.21.15.32:0/3639091435 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f668001a070 con 0x7f669006e930 2024-08-03T20:54:24.213 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:24.211+0000 7f6686000640 1 --2- 172.21.15.32:0/3639091435 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f6670077040 0x7f6670079550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:24.213 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:24.211+0000 7f6686000640 1 -- 172.21.15.32:0/3639091435 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(79..79 src has 1..79) ==== 4857+0+0 (secure 0 0 0) 0x7f6680097960 con 0x7f669006e930 2024-08-03T20:54:24.213 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:24.212+0000 7f668d400640 1 --2- 172.21.15.32:0/3639091435 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f6670077040 0x7f6670079550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:24.214 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:24.213+0000 7f668d400640 1 --2- 172.21.15.32:0/3639091435 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f6670077040 0x7f6670079550 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f667c0046b0 tx=0x7f667c009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:54:24.217 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:24.216+0000 7f6686000640 1 -- 172.21.15.32:0/3639091435 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7f6680060f30 con 0x7f669006e930 2024-08-03T20:54:24.424 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:24.422+0000 7f6695000640 1 -- 172.21.15.32:0/3639091435 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) -- 0x7f66900b3350 con 0x7f669006e930 2024-08-03T20:54:24.424 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:24.423+0000 7f6686000640 1 -- 172.21.15.32:0/3639091435 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v79) ==== 84+0+65 (secure 0 0 0) 0x7f6680069170 con 0x7f669006e930 2024-08-03T20:54:24.425 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-03T20:54:24.425 INFO:teuthology.orchestra.run.smithi032.stdout:[ 2024-08-03T20:54:24.425 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-03T20:54:24.425 INFO:teuthology.orchestra.run.smithi032.stdout: "poolnum": 1, 2024-08-03T20:54:24.425 INFO:teuthology.orchestra.run.smithi032.stdout: "poolname": ".mgr" 2024-08-03T20:54:24.425 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-03T20:54:24.425 INFO:teuthology.orchestra.run.smithi032.stdout:] 2024-08-03T20:54:24.427 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:24.425+0000 7f6695000640 1 -- 172.21.15.32:0/3639091435 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f6670077040 msgr2=0x7f6670079550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:24.427 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:24.425+0000 7f6695000640 1 --2- 172.21.15.32:0/3639091435 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f6670077040 0x7f6670079550 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f667c0046b0 tx=0x7f667c009040 comp rx=0 tx=0).stop 2024-08-03T20:54:24.427 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:24.425+0000 7f6695000640 1 -- 172.21.15.32:0/3639091435 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f669006e930 msgr2=0x7f66901f8300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:24.427 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:24.425+0000 7f6695000640 1 --2- 172.21.15.32:0/3639091435 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f669006e930 0x7f66901f8300 secure :-1 s=READY pgs=890 cs=0 l=1 rev1=1 crypto rx=0x7f668000c430 tx=0x7f668000c8f0 comp rx=0 tx=0).stop 2024-08-03T20:54:24.427 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:24.425+0000 7f6695000640 1 -- 172.21.15.32:0/3639091435 shutdown_connections 2024-08-03T20:54:24.427 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:24.425+0000 7f6695000640 1 --2- 172.21.15.32:0/3639091435 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f6670077040 0x7f6670079550 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:24.427 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:24.425+0000 7f6695000640 1 --2- 172.21.15.32:0/3639091435 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f6690144f90 0x7f669013e960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:24.427 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:24.425+0000 7f6695000640 1 --2- 172.21.15.32:0/3639091435 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f6690165160 0x7f6690166b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:24.428 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:24.425+0000 7f6695000640 1 --2- 172.21.15.32:0/3639091435 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f669006e930 0x7f66901f8300 unknown :-1 s=CLOSED pgs=890 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:24.428 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:24.425+0000 7f6695000640 1 -- 172.21.15.32:0/3639091435 >> 172.21.15.32:0/3639091435 conn(0x7f66900b1bd0 msgr2=0x7f66900cc500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:54:24.428 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:24.425+0000 7f6695000640 1 -- 172.21.15.32:0/3639091435 shutdown_connections 2024-08-03T20:54:24.428 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:24.425+0000 7f6695000640 1 -- 172.21.15.32:0/3639091435 wait complete. 2024-08-03T20:54:24.449 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-08-03T20:54:24.450 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-08-03T20:54:24.646 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b has been restored 2024-08-03T20:54:24.685 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:24.683+0000 7f54c3c00640 1 -- 172.21.15.32:0/447922684 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f54a0004080 msgr2=0x7f54a0004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:24.685 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:24.683+0000 7f54c3c00640 1 --2- 172.21.15.32:0/447922684 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f54a0004080 0x7f54a0004800 secure :-1 s=READY pgs=892 cs=0 l=1 rev1=1 crypto rx=0x7f54bc070510 tx=0x7f54a0005830 comp rx=0 tx=0).stop 2024-08-03T20:54:24.685 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:24.683+0000 7f54c1c00640 1 -- 172.21.15.32:0/447922684 reap_dead start 2024-08-03T20:54:24.685 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:24.683+0000 7f54c3c00640 1 -- 172.21.15.32:0/447922684 shutdown_connections 2024-08-03T20:54:24.685 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:24.683+0000 7f54c3c00640 1 -- 172.21.15.32:0/447922684 >> 172.21.15.32:0/447922684 conn(0x7f54bc0b1bd0 msgr2=0x7f54bc0b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:54:24.685 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:24.683+0000 7f54c3c00640 1 -- 172.21.15.32:0/447922684 shutdown_connections 2024-08-03T20:54:24.685 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:24.683+0000 7f54c3c00640 1 -- 172.21.15.32:0/447922684 wait complete. 2024-08-03T20:54:24.687 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:24.685+0000 7f54c3c00640 1 Processor -- start 2024-08-03T20:54:24.687 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:24.685+0000 7f54c3c00640 1 -- start start 2024-08-03T20:54:24.690 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:24.687+0000 7f54c3c00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f54a0004080 0x7f54bc13d520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:24.690 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:24.687+0000 7f54c3c00640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f54bc13f790 0x7f54bc14d8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:24.690 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:24.687+0000 7f54c3c00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f54bc14de00 0x7f54bc13e460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:24.690 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:24.687+0000 7f54c3c00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f54bc06ee40 con 0x7f54bc14de00 2024-08-03T20:54:24.690 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:24.687+0000 7f54c3c00640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f54bc06ecc0 con 0x7f54a0004080 2024-08-03T20:54:24.690 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:24.687+0000 7f54c3c00640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7f54bc06efc0 con 0x7f54bc13f790 2024-08-03T20:54:24.691 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:24.687+0000 7f54c1c00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f54bc14de00 0x7f54bc13e460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:24.691 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:24.687+0000 7f54c1c00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f54bc14de00 0x7f54bc13e460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:50290/0 (socket says 172.21.15.32:50290) 2024-08-03T20:54:24.691 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:24.687+0000 7f54c1c00640 1 -- 172.21.15.32:0/920523430 learned_addr learned my addr 172.21.15.32:0/920523430 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:54:24.691 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:24.688+0000 7f54c1200640 1 --2- 172.21.15.32:0/920523430 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f54a0004080 0x7f54bc13d520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:24.691 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:24.688+0000 7f54c1c00640 1 -- 172.21.15.32:0/920523430 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f54bc13f790 msgr2=0x7f54bc14d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:24.691 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:24.688+0000 7f54b3e00640 1 --2- 172.21.15.32:0/920523430 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f54bc13f790 0x7f54bc14d8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:24.691 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:24.688+0000 7f54c1c00640 1 --2- 172.21.15.32:0/920523430 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f54bc13f790 0x7f54bc14d8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:24.691 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:24.688+0000 7f54c1c00640 1 -- 172.21.15.32:0/920523430 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f54a0004080 msgr2=0x7f54bc13d520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:24.691 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:24.688+0000 7f54c1c00640 1 --2- 172.21.15.32:0/920523430 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f54a0004080 0x7f54bc13d520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:24.691 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:24.688+0000 7f54c1c00640 1 -- 172.21.15.32:0/920523430 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f54bc14e820 con 0x7f54bc14de00 2024-08-03T20:54:24.691 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:24.688+0000 7f54b3e00640 1 --2- 172.21.15.32:0/920523430 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f54bc13f790 0x7f54bc14d8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-03T20:54:24.691 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:24.688+0000 7f54c1c00640 1 --2- 172.21.15.32:0/920523430 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f54bc14de00 0x7f54bc13e460 secure :-1 s=READY pgs=893 cs=0 l=1 rev1=1 crypto rx=0x7f54b4005fd0 tx=0x7f54b4010ee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:54:24.692 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:24.688+0000 7f54b9600640 1 -- 172.21.15.32:0/920523430 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54b4003c40 con 0x7f54bc14de00 2024-08-03T20:54:24.692 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:24.688+0000 7f54b9600640 1 -- 172.21.15.32:0/920523430 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f54b4003e00 con 0x7f54bc14de00 2024-08-03T20:54:24.692 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:24.688+0000 7f54b9600640 1 -- 172.21.15.32:0/920523430 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54b4004d80 con 0x7f54bc14de00 2024-08-03T20:54:24.692 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:24.688+0000 7f54c3c00640 1 -- 172.21.15.32:0/920523430 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f54bc14e540 con 0x7f54bc14de00 2024-08-03T20:54:24.692 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:24.688+0000 7f54c3c00640 1 -- 172.21.15.32:0/920523430 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f54bc13cd20 con 0x7f54bc14de00 2024-08-03T20:54:24.692 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:24.689+0000 7f54c3c00640 1 -- 172.21.15.32:0/920523430 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f54bc14e2b0 con 0x7f54bc14de00 2024-08-03T20:54:24.692 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:24.691+0000 7f54b9600640 1 -- 172.21.15.32:0/920523430 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f54b401c020 con 0x7f54bc14de00 2024-08-03T20:54:24.693 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:24.692+0000 7f54b9600640 1 --2- 172.21.15.32:0/920523430 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f5498077010 0x7f5498079520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:24.693 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:24.692+0000 7f54b9600640 1 -- 172.21.15.32:0/920523430 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(79..79 src has 1..79) ==== 4857+0+0 (secure 0 0 0) 0x7f54b4098900 con 0x7f54bc14de00 2024-08-03T20:54:24.693 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:24.692+0000 7f54c1200640 1 --2- 172.21.15.32:0/920523430 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f5498077010 0x7f5498079520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:24.694 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:24.693+0000 7f54c1200640 1 --2- 172.21.15.32:0/920523430 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f5498077010 0x7f5498079520 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f54ac0058c0 tx=0x7f54ac005850 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:54:24.698 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:24.696+0000 7f54b9600640 1 -- 172.21.15.32:0/920523430 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7f54b4061d30 con 0x7f54bc14de00 2024-08-03T20:54:24.899 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:24.898+0000 7f54c3c00640 1 -- 172.21.15.32:0/920523430 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) -- 0x7f54bc0cf140 con 0x7f54bc14de00 2024-08-03T20:54:25.410 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:25.408+0000 7f54b9600640 1 -- 172.21.15.32:0/920523430 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v80) ==== 138+0+0 (secure 0 0 0) 0x7f54b4061630 con 0x7f54bc14de00 2024-08-03T20:54:25.410 INFO:teuthology.orchestra.run.smithi032.stderr:pool 'cephfs_metadata' created 2024-08-03T20:54:25.413 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:25.411+0000 7f54c3c00640 1 -- 172.21.15.32:0/920523430 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f5498077010 msgr2=0x7f5498079520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:25.413 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:25.411+0000 7f54c3c00640 1 --2- 172.21.15.32:0/920523430 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f5498077010 0x7f5498079520 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f54ac0058c0 tx=0x7f54ac005850 comp rx=0 tx=0).stop 2024-08-03T20:54:25.413 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:25.411+0000 7f54c3c00640 1 -- 172.21.15.32:0/920523430 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f54bc14de00 msgr2=0x7f54bc13e460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:25.413 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:25.411+0000 7f54c3c00640 1 --2- 172.21.15.32:0/920523430 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f54bc14de00 0x7f54bc13e460 secure :-1 s=READY pgs=893 cs=0 l=1 rev1=1 crypto rx=0x7f54b4005fd0 tx=0x7f54b4010ee0 comp rx=0 tx=0).stop 2024-08-03T20:54:25.413 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:25.411+0000 7f54c3c00640 1 -- 172.21.15.32:0/920523430 shutdown_connections 2024-08-03T20:54:25.413 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:25.411+0000 7f54c3c00640 1 --2- 172.21.15.32:0/920523430 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f5498077010 0x7f5498079520 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:25.413 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:25.411+0000 7f54c3c00640 1 --2- 172.21.15.32:0/920523430 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f54bc14de00 0x7f54bc13e460 unknown :-1 s=CLOSED pgs=893 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:25.413 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:25.411+0000 7f54c3c00640 1 --2- 172.21.15.32:0/920523430 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f54bc13f790 0x7f54bc14d8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:25.413 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:25.411+0000 7f54c3c00640 1 --2- 172.21.15.32:0/920523430 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f54a0004080 0x7f54bc13d520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:25.413 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:25.411+0000 7f54c3c00640 1 -- 172.21.15.32:0/920523430 >> 172.21.15.32:0/920523430 conn(0x7f54bc0b1bd0 msgr2=0x7f54bc0b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:54:25.414 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:25.411+0000 7f54c3c00640 1 -- 172.21.15.32:0/920523430 shutdown_connections 2024-08-03T20:54:25.414 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:25.411+0000 7f54c3c00640 1 -- 172.21.15.32:0/920523430 wait complete. 2024-08-03T20:54:25.445 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-08-03T20:54:25.686 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:25.684+0000 7fc377e00640 1 -- 172.21.15.32:0/237901931 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc3780b9530 msgr2=0x7fc3780b3bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:25.686 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:25.684+0000 7fc377e00640 1 --2- 172.21.15.32:0/237901931 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc3780b9530 0x7fc3780b3bc0 secure :-1 s=READY pgs=894 cs=0 l=1 rev1=1 crypto rx=0x7fc37808e310 tx=0x7fc36802a560 comp rx=0 tx=0).stop 2024-08-03T20:54:25.686 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:25.684+0000 7fc377e00640 1 -- 172.21.15.32:0/237901931 shutdown_connections 2024-08-03T20:54:25.686 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:25.684+0000 7fc377e00640 1 --2- 172.21.15.32:0/237901931 >> v2:172.21.15.118:3300/0 conn(0x7fc3780b4510 0x7fc3780d25b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:25.687 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:25.684+0000 7fc377e00640 1 --2- 172.21.15.32:0/237901931 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc3780b9530 0x7fc3780b3bc0 unknown :-1 s=CLOSED pgs=894 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:25.687 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:25.684+0000 7fc377e00640 1 -- 172.21.15.32:0/237901931 >> 172.21.15.32:0/237901931 conn(0x7fc3780d40c0 msgr2=0x7fc3780d44d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:54:25.687 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:25.684+0000 7fc377e00640 1 -- 172.21.15.32:0/237901931 shutdown_connections 2024-08-03T20:54:25.687 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:25.684+0000 7fc377e00640 1 -- 172.21.15.32:0/237901931 wait complete. 2024-08-03T20:54:25.688 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:25.686+0000 7fc377e00640 1 Processor -- start 2024-08-03T20:54:25.688 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:25.687+0000 7fc377e00640 1 -- start start 2024-08-03T20:54:25.690 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:25.688+0000 7fc377e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc3780b4100 0x7fc378168330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:25.690 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:25.688+0000 7fc377e00640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fc3780b4510 0x7fc378157140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:25.690 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:25.688+0000 7fc377e00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc3780b9530 0x7fc378160b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:25.691 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:25.688+0000 7fc377e00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fc3780d2af0 con 0x7fc3780b4100 2024-08-03T20:54:25.691 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:25.688+0000 7fc377e00640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7fc3780cfee0 con 0x7fc3780b9530 2024-08-03T20:54:25.691 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:25.688+0000 7fc377e00640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7fc3780d2c70 con 0x7fc3780b4510 2024-08-03T20:54:25.691 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:25.688+0000 7fc376a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc3780b4100 0x7fc378168330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:25.691 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:25.688+0000 7fc376a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc3780b4100 0x7fc378168330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:50312/0 (socket says 172.21.15.32:50312) 2024-08-03T20:54:25.691 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:25.688+0000 7fc376a00640 1 -- 172.21.15.32:0/2633283377 learned_addr learned my addr 172.21.15.32:0/2633283377 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:54:25.691 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:25.689+0000 7fc377400640 1 --2- 172.21.15.32:0/2633283377 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc3780b9530 0x7fc378160b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:25.691 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:25.689+0000 7fc376000640 1 --2- 172.21.15.32:0/2633283377 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fc3780b4510 0x7fc378157140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:25.691 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:25.689+0000 7fc376a00640 1 -- 172.21.15.32:0/2633283377 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fc3780b4510 msgr2=0x7fc378157140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:25.692 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:25.689+0000 7fc376a00640 1 --2- 172.21.15.32:0/2633283377 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fc3780b4510 0x7fc378157140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:25.692 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:25.689+0000 7fc376a00640 1 -- 172.21.15.32:0/2633283377 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc3780b9530 msgr2=0x7fc378160b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:25.692 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:25.689+0000 7fc376a00640 1 --2- 172.21.15.32:0/2633283377 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc3780b9530 0x7fc378160b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:25.692 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:25.689+0000 7fc376a00640 1 -- 172.21.15.32:0/2633283377 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc3781668e0 con 0x7fc3780b4100 2024-08-03T20:54:25.692 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:25.689+0000 7fc377400640 1 --2- 172.21.15.32:0/2633283377 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc3780b9530 0x7fc378160b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-03T20:54:25.692 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:25.689+0000 7fc376a00640 1 --2- 172.21.15.32:0/2633283377 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc3780b4100 0x7fc378168330 secure :-1 s=READY pgs=895 cs=0 l=1 rev1=1 crypto rx=0x7fc368034040 tx=0x7fc368003910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:54:25.693 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:25.689+0000 7fc36f400640 1 -- 172.21.15.32:0/2633283377 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc36803e030 con 0x7fc3780b4100 2024-08-03T20:54:25.693 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:25.689+0000 7fc36f400640 1 -- 172.21.15.32:0/2633283377 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fc368003e00 con 0x7fc3780b4100 2024-08-03T20:54:25.693 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:25.689+0000 7fc377e00640 1 -- 172.21.15.32:0/2633283377 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc378168870 con 0x7fc3780b4100 2024-08-03T20:54:25.693 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:25.690+0000 7fc36f400640 1 -- 172.21.15.32:0/2633283377 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc368035040 con 0x7fc3780b4100 2024-08-03T20:54:25.693 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:25.690+0000 7fc377e00640 1 -- 172.21.15.32:0/2633283377 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc378166e30 con 0x7fc3780b4100 2024-08-03T20:54:25.693 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:25.691+0000 7fc377e00640 1 -- 172.21.15.32:0/2633283377 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc3780b3bc0 con 0x7fc3780b4100 2024-08-03T20:54:25.694 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:25.692+0000 7fc36f400640 1 -- 172.21.15.32:0/2633283377 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7fc368041030 con 0x7fc3780b4100 2024-08-03T20:54:25.694 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:25.693+0000 7fc36f400640 1 --2- 172.21.15.32:0/2633283377 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fc358077010 0x7fc358079520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:25.694 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:25.693+0000 7fc36f400640 1 -- 172.21.15.32:0/2633283377 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 5227+0+0 (secure 0 0 0) 0x7fc3680b93e0 con 0x7fc3780b4100 2024-08-03T20:54:25.695 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:25.694+0000 7fc376000640 1 --2- 172.21.15.32:0/2633283377 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fc358077010 0x7fc358079520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:25.696 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:25.694+0000 7fc376000640 1 --2- 172.21.15.32:0/2633283377 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fc358077010 0x7fc358079520 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7fc3780a21f0 tx=0x7fc364009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:54:25.698 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:25.697+0000 7fc36f400640 1 -- 172.21.15.32:0/2633283377 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7fc3680827a0 con 0x7fc3780b4100 2024-08-03T20:54:25.904 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:25.903+0000 7fc377e00640 1 -- 172.21.15.32:0/2633283377 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) -- 0x7fc3780d0f50 con 0x7fc3780b4100 2024-08-03T20:54:26.417 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:26.415+0000 7fc36f400640 1 -- 172.21.15.32:0/2633283377 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v81) ==== 170+0+0 (secure 0 0 0) 0x7fc3680820a0 con 0x7fc3780b4100 2024-08-03T20:54:26.417 INFO:teuthology.orchestra.run.smithi032.stderr:pool 'cephfs_data' created 2024-08-03T20:54:26.420 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:26.418+0000 7fc377e00640 1 -- 172.21.15.32:0/2633283377 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fc358077010 msgr2=0x7fc358079520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:26.420 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:26.418+0000 7fc377e00640 1 --2- 172.21.15.32:0/2633283377 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fc358077010 0x7fc358079520 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7fc3780a21f0 tx=0x7fc364009040 comp rx=0 tx=0).stop 2024-08-03T20:54:26.420 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:26.418+0000 7fc377e00640 1 -- 172.21.15.32:0/2633283377 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc3780b4100 msgr2=0x7fc378168330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:26.420 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:26.418+0000 7fc377e00640 1 --2- 172.21.15.32:0/2633283377 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc3780b4100 0x7fc378168330 secure :-1 s=READY pgs=895 cs=0 l=1 rev1=1 crypto rx=0x7fc368034040 tx=0x7fc368003910 comp rx=0 tx=0).stop 2024-08-03T20:54:26.420 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:26.418+0000 7fc377e00640 1 -- 172.21.15.32:0/2633283377 shutdown_connections 2024-08-03T20:54:26.420 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:26.418+0000 7fc377e00640 1 --2- 172.21.15.32:0/2633283377 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fc358077010 0x7fc358079520 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:26.420 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:26.418+0000 7fc377e00640 1 --2- 172.21.15.32:0/2633283377 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc3780b9530 0x7fc378160b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:26.420 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:26.418+0000 7fc377e00640 1 --2- 172.21.15.32:0/2633283377 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fc3780b4510 0x7fc378157140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:26.420 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:26.418+0000 7fc377e00640 1 --2- 172.21.15.32:0/2633283377 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc3780b4100 0x7fc378168330 unknown :-1 s=CLOSED pgs=895 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:26.420 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:26.418+0000 7fc377e00640 1 -- 172.21.15.32:0/2633283377 >> 172.21.15.32:0/2633283377 conn(0x7fc3780d40c0 msgr2=0x7fc3780b0d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:54:26.421 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:26.418+0000 7fc377e00640 1 -- 172.21.15.32:0/2633283377 shutdown_connections 2024-08-03T20:54:26.421 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:26.418+0000 7fc377e00640 1 -- 172.21.15.32:0/2633283377 wait complete. 2024-08-03T20:54:26.459 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-08-03T20:54:26.692 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:26.690+0000 7fd68b200640 1 -- 172.21.15.32:0/4294878566 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd68c0caae0 msgr2=0x7fd68c0cd950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:26.692 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:26.690+0000 7fd68b200640 1 --2- 172.21.15.32:0/4294878566 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd68c0caae0 0x7fd68c0cd950 secure :-1 s=READY pgs=896 cs=0 l=1 rev1=1 crypto rx=0x7fd684005fd0 tx=0x7fd68402aa90 comp rx=0 tx=0).stop 2024-08-03T20:54:26.692 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:26.690+0000 7fd68b200640 1 -- 172.21.15.32:0/4294878566 shutdown_connections 2024-08-03T20:54:26.692 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:26.690+0000 7fd68b200640 1 --2- 172.21.15.32:0/4294878566 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd68c0caae0 0x7fd68c0cd950 unknown :-1 s=CLOSED pgs=896 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:26.692 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:26.690+0000 7fd68b200640 1 --2- 172.21.15.32:0/4294878566 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fd68c0cf6a0 0x7fd68c0742a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:26.692 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:26.690+0000 7fd68b200640 1 -- 172.21.15.32:0/4294878566 >> 172.21.15.32:0/4294878566 conn(0x7fd68c0cfd80 msgr2=0x7fd68c0d0190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:54:26.692 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:26.690+0000 7fd68b200640 1 -- 172.21.15.32:0/4294878566 shutdown_connections 2024-08-03T20:54:26.693 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:26.690+0000 7fd68b200640 1 -- 172.21.15.32:0/4294878566 wait complete. 2024-08-03T20:54:26.694 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:26.692+0000 7fd68b200640 1 Processor -- start 2024-08-03T20:54:26.694 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:26.693+0000 7fd68b200640 1 -- start start 2024-08-03T20:54:26.697 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:26.694+0000 7fd68b200640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fd68c0700f0 0x7fd68c14f270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:26.697 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:26.694+0000 7fd68b200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd68c0cf6a0 0x7fd68c0c58c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:26.698 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:26.694+0000 7fd68b200640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fd68c14fa10 0x7fd68c0c5e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:26.698 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:26.694+0000 7fd68b200640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fd68c0cb200 con 0x7fd68c0cf6a0 2024-08-03T20:54:26.698 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:26.694+0000 7fd68b200640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7fd68c15b530 con 0x7fd68c14fa10 2024-08-03T20:54:26.698 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:26.694+0000 7fd68b200640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7fd68c0cde90 con 0x7fd68c0700f0 2024-08-03T20:54:26.698 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:26.694+0000 7fd681400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd68c0cf6a0 0x7fd68c0c58c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:26.698 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:26.694+0000 7fd681400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd68c0cf6a0 0x7fd68c0c58c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:50326/0 (socket says 172.21.15.32:50326) 2024-08-03T20:54:26.698 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:26.694+0000 7fd681400640 1 -- 172.21.15.32:0/2972453062 learned_addr learned my addr 172.21.15.32:0/2972453062 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:54:26.698 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:26.694+0000 7fd68a800640 1 --2- 172.21.15.32:0/2972453062 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fd68c14fa10 0x7fd68c0c5e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:26.698 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:26.694+0000 7fd689e00640 1 --2- 172.21.15.32:0/2972453062 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fd68c0700f0 0x7fd68c14f270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:26.699 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:26.695+0000 7fd681400640 1 -- 172.21.15.32:0/2972453062 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fd68c0700f0 msgr2=0x7fd68c14f270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:26.699 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:26.695+0000 7fd681400640 1 --2- 172.21.15.32:0/2972453062 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fd68c0700f0 0x7fd68c14f270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:26.699 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:26.695+0000 7fd681400640 1 -- 172.21.15.32:0/2972453062 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fd68c14fa10 msgr2=0x7fd68c0c5e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:26.699 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:26.695+0000 7fd681400640 1 --2- 172.21.15.32:0/2972453062 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fd68c14fa10 0x7fd68c0c5e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:26.699 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:26.695+0000 7fd681400640 1 -- 172.21.15.32:0/2972453062 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd68c156a00 con 0x7fd68c0cf6a0 2024-08-03T20:54:26.699 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:26.695+0000 7fd681400640 1 --2- 172.21.15.32:0/2972453062 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd68c0cf6a0 0x7fd68c0c58c0 secure :-1 s=READY pgs=897 cs=0 l=1 rev1=1 crypto rx=0x7fd67400c370 tx=0x7fd67400c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:54:26.699 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:26.695+0000 7fd682a00640 1 -- 172.21.15.32:0/2972453062 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6740052d0 con 0x7fd68c0cf6a0 2024-08-03T20:54:26.699 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:26.695+0000 7fd682a00640 1 -- 172.21.15.32:0/2972453062 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fd674007020 con 0x7fd68c0cf6a0 2024-08-03T20:54:26.699 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:26.695+0000 7fd68b200640 1 -- 172.21.15.32:0/2972453062 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd68c156ce0 con 0x7fd68c0cf6a0 2024-08-03T20:54:26.699 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:26.695+0000 7fd682a00640 1 -- 172.21.15.32:0/2972453062 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd674007470 con 0x7fd68c0cf6a0 2024-08-03T20:54:26.699 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:26.695+0000 7fd68b200640 1 -- 172.21.15.32:0/2972453062 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd68c1572a0 con 0x7fd68c0cf6a0 2024-08-03T20:54:26.700 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:26.696+0000 7fd68b200640 1 -- 172.21.15.32:0/2972453062 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd68c0ce3c0 con 0x7fd68c0cf6a0 2024-08-03T20:54:26.700 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:26.698+0000 7fd682a00640 1 -- 172.21.15.32:0/2972453062 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7fd6740067d0 con 0x7fd68c0cf6a0 2024-08-03T20:54:26.701 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:26.699+0000 7fd682a00640 1 --2- 172.21.15.32:0/2972453062 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fd668077040 0x7fd668079550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:26.701 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:26.700+0000 7fd682a00640 1 -- 172.21.15.32:0/2972453062 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 5609+0+0 (secure 0 0 0) 0x7fd674097df0 con 0x7fd68c0cf6a0 2024-08-03T20:54:26.701 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:26.700+0000 7fd689e00640 1 --2- 172.21.15.32:0/2972453062 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fd668077040 0x7fd668079550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:26.702 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:26.701+0000 7fd689e00640 1 --2- 172.21.15.32:0/2972453062 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fd668077040 0x7fd668079550 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7fd67c009b30 tx=0x7fd67c005f90 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:54:26.706 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:26.705+0000 7fd682a00640 1 -- 172.21.15.32:0/2972453062 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7fd674060960 con 0x7fd68c0cf6a0 2024-08-03T20:54:26.950 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:26.948+0000 7fd68b200640 1 -- 172.21.15.32:0/2972453062 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) -- 0x7fd68c0b5d00 con 0x7fd68c0cf6a0 2024-08-03T20:54:27.419 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:27.417+0000 7fd682a00640 1 -- 172.21.15.32:0/2972453062 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 12 and data pool 13 v87) ==== 174+0+0 (secure 0 0 0) 0x7fd674064eb0 con 0x7fd68c0cf6a0 2024-08-03T20:54:27.419 INFO:teuthology.orchestra.run.smithi032.stderr:new fs with metadata pool 12 and data pool 13 2024-08-03T20:54:27.424 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:27.422+0000 7fd68b200640 1 -- 172.21.15.32:0/2972453062 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fd668077040 msgr2=0x7fd668079550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:27.424 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:27.422+0000 7fd68b200640 1 --2- 172.21.15.32:0/2972453062 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fd668077040 0x7fd668079550 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7fd67c009b30 tx=0x7fd67c005f90 comp rx=0 tx=0).stop 2024-08-03T20:54:27.424 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:27.422+0000 7fd68b200640 1 -- 172.21.15.32:0/2972453062 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd68c0cf6a0 msgr2=0x7fd68c0c58c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:27.424 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:27.422+0000 7fd68b200640 1 --2- 172.21.15.32:0/2972453062 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd68c0cf6a0 0x7fd68c0c58c0 secure :-1 s=READY pgs=897 cs=0 l=1 rev1=1 crypto rx=0x7fd67400c370 tx=0x7fd67400c830 comp rx=0 tx=0).stop 2024-08-03T20:54:27.424 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:27.422+0000 7fd68b200640 1 -- 172.21.15.32:0/2972453062 shutdown_connections 2024-08-03T20:54:27.424 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:27.422+0000 7fd68b200640 1 --2- 172.21.15.32:0/2972453062 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fd668077040 0x7fd668079550 secure :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7fd67c009b30 tx=0x7fd67c005f90 comp rx=0 tx=0).stop 2024-08-03T20:54:27.424 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:27.422+0000 7fd68b200640 1 --2- 172.21.15.32:0/2972453062 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fd68c14fa10 0x7fd68c0c5e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:27.425 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:27.422+0000 7fd68b200640 1 --2- 172.21.15.32:0/2972453062 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd68c0cf6a0 0x7fd68c0c58c0 unknown :-1 s=CLOSED pgs=897 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:27.425 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:27.422+0000 7fd68b200640 1 --2- 172.21.15.32:0/2972453062 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fd68c0700f0 0x7fd68c14f270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:27.425 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:27.422+0000 7fd68b200640 1 -- 172.21.15.32:0/2972453062 >> 172.21.15.32:0/2972453062 conn(0x7fd68c0cfd80 msgr2=0x7fd68c0afc40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:54:27.425 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:27.422+0000 7fd68b200640 1 -- 172.21.15.32:0/2972453062 shutdown_connections 2024-08-03T20:54:27.425 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:27.422+0000 7fd68b200640 1 -- 172.21.15.32:0/2972453062 wait complete. 2024-08-03T20:54:27.475 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-08-03T20:54:27.704 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:27.702+0000 7fa08be00640 1 -- 172.21.15.32:0/1927042389 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fa08406e930 msgr2=0x7fa084144f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:27.704 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:27.702+0000 7fa08be00640 1 --2- 172.21.15.32:0/1927042389 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fa08406e930 0x7fa084144f90 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fa078009830 tx=0x7fa07802a480 comp rx=0 tx=0).stop 2024-08-03T20:54:27.704 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:27.702+0000 7fa08be00640 1 -- 172.21.15.32:0/1927042389 shutdown_connections 2024-08-03T20:54:27.704 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:27.702+0000 7fa08be00640 1 --2- 172.21.15.32:0/1927042389 >> v2:172.21.15.118:3300/0 conn(0x7fa084073e60 0x7fa084074270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:27.704 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:27.702+0000 7fa08be00640 1 --2- 172.21.15.32:0/1927042389 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fa08406e930 0x7fa084144f90 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:27.704 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:27.702+0000 7fa08be00640 1 -- 172.21.15.32:0/1927042389 >> 172.21.15.32:0/1927042389 conn(0x7fa0840b1bd0 msgr2=0x7fa0840b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:54:27.705 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:27.702+0000 7fa08be00640 1 -- 172.21.15.32:0/1927042389 shutdown_connections 2024-08-03T20:54:27.705 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:27.702+0000 7fa08be00640 1 -- 172.21.15.32:0/1927042389 wait complete. 2024-08-03T20:54:27.706 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:27.705+0000 7fa08be00640 1 Processor -- start 2024-08-03T20:54:27.706 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:27.705+0000 7fa08be00640 1 -- start start 2024-08-03T20:54:27.709 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:27.706+0000 7fa08be00640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fa08406e930 0x7fa08415bd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:27.709 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:27.706+0000 7fa08be00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa084073e60 0x7fa0840aede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:27.709 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:27.706+0000 7fa08be00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa0840747b0 0x7fa0841436b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:27.709 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:27.706+0000 7fa08be00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fa0840cf9c0 con 0x7fa0840747b0 2024-08-03T20:54:27.709 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:27.706+0000 7fa08be00640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7fa0840cf840 con 0x7fa084073e60 2024-08-03T20:54:27.709 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:27.706+0000 7fa08be00640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7fa0840ccff0 con 0x7fa08406e930 2024-08-03T20:54:27.709 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:27.706+0000 7fa089e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa0840747b0 0x7fa0841436b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:27.709 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:27.707+0000 7fa089e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa0840747b0 0x7fa0841436b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:50336/0 (socket says 172.21.15.32:50336) 2024-08-03T20:54:27.709 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:27.707+0000 7fa089e00640 1 -- 172.21.15.32:0/4058210930 learned_addr learned my addr 172.21.15.32:0/4058210930 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:54:27.709 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:27.707+0000 7fa089400640 1 --2- 172.21.15.32:0/4058210930 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fa08406e930 0x7fa08415bd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:27.709 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:27.707+0000 7fa088a00640 1 --2- 172.21.15.32:0/4058210930 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa084073e60 0x7fa0840aede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:27.709 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:27.707+0000 7fa089e00640 1 -- 172.21.15.32:0/4058210930 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fa08406e930 msgr2=0x7fa08415bd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:27.710 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:27.707+0000 7fa089e00640 1 --2- 172.21.15.32:0/4058210930 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fa08406e930 0x7fa08415bd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:27.710 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:27.707+0000 7fa089e00640 1 -- 172.21.15.32:0/4058210930 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa084073e60 msgr2=0x7fa0840aede0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:27.710 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:27.707+0000 7fa089e00640 1 --2- 172.21.15.32:0/4058210930 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa084073e60 0x7fa0840aede0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:27.710 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:27.707+0000 7fa089e00640 1 -- 172.21.15.32:0/4058210930 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa0841f6760 con 0x7fa0840747b0 2024-08-03T20:54:27.710 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:27.707+0000 7fa089e00640 1 --2- 172.21.15.32:0/4058210930 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa0840747b0 0x7fa0841436b0 secure :-1 s=READY pgs=899 cs=0 l=1 rev1=1 crypto rx=0x7fa07c00cae0 tx=0x7fa07c00a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:54:27.711 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:27.708+0000 7fa081600640 1 -- 172.21.15.32:0/4058210930 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa07c016020 con 0x7fa0840747b0 2024-08-03T20:54:27.711 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:27.708+0000 7fa081600640 1 -- 172.21.15.32:0/4058210930 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fa07c00ad30 con 0x7fa0840747b0 2024-08-03T20:54:27.711 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:27.708+0000 7fa081600640 1 -- 172.21.15.32:0/4058210930 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa07c012680 con 0x7fa0840747b0 2024-08-03T20:54:27.711 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:27.708+0000 7fa08be00640 1 -- 172.21.15.32:0/4058210930 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa0841f6a40 con 0x7fa0840747b0 2024-08-03T20:54:27.711 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:27.708+0000 7fa08be00640 1 -- 172.21.15.32:0/4058210930 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa0841f7000 con 0x7fa0840747b0 2024-08-03T20:54:27.711 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:27.709+0000 7fa08be00640 1 -- 172.21.15.32:0/4058210930 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa084072ac0 con 0x7fa0840747b0 2024-08-03T20:54:27.712 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:27.710+0000 7fa081600640 1 -- 172.21.15.32:0/4058210930 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7fa07c019020 con 0x7fa0840747b0 2024-08-03T20:54:27.713 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:27.711+0000 7fa081600640 1 --2- 172.21.15.32:0/4058210930 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fa064077010 0x7fa064079520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:27.713 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:27.711+0000 7fa081600640 1 -- 172.21.15.32:0/4058210930 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 5709+0+0 (secure 0 0 0) 0x7fa07c098e50 con 0x7fa0840747b0 2024-08-03T20:54:27.713 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:27.711+0000 7fa089400640 1 --2- 172.21.15.32:0/4058210930 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fa064077010 0x7fa064079520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:27.713 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:27.712+0000 7fa089400640 1 --2- 172.21.15.32:0/4058210930 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fa064077010 0x7fa064079520 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7fa078034000 tx=0x7fa078004730 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:54:27.717 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:27.716+0000 7fa081600640 1 -- 172.21.15.32:0/4058210930 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7fa084072ac0 con 0x7fa0840747b0 2024-08-03T20:54:27.927 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:27.925+0000 7fa08be00640 1 -- 172.21.15.32:0/4058210930 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fa0840af830 con 0x7fa0840747b0 2024-08-03T20:54:27.928 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:27.927+0000 7fa081600640 1 -- 172.21.15.32:0/4058210930 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v82) ==== 74+0+16613 (secure 0 0 0) 0x7fa07c020020 con 0x7fa0840747b0 2024-08-03T20:54:27.928 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-03T20:54:27.929 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":82,"fsid":"40ab141a-01d7-49b9-ae3c-2edec9876e95","created":"2024-08-03T20:47:14.957362+0000","modified":"2024-08-03T20:54:26.950564+0000","last_up_change":"2024-08-03T20:47:28.874451+0000","last_in_change":"2024-08-03T20:47:20.079442+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":13,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-08-03T20:47:28.261300+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_metadata","create_time":"2024-08-03T20:54:24.911555+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"82","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs_data","create_time":"2024-08-03T20:54:25.917207+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"82","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5a83e759-9035-438b-889a-0a6af1f0a0d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6816","nonce":3066156116},{"type":"v1","addr":"172.21.15.32:6817","nonce":3066156116}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6818","nonce":3066156116},{"type":"v1","addr":"172.21.15.32:6819","nonce":3066156116}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6822","nonce":3066156116},{"type":"v1","addr":"172.21.15.32:6823","nonce":3066156116}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6820","nonce":3066156116},{"type":"v1","addr":"172.21.15.32:6821","nonce":3066156116}]},"public_addr":"172.21.15.32:6817/3066156116","cluster_addr":"172.21.15.32:6819/3066156116","heartbeat_back_addr":"172.21.15.32:6823/3066156116","heartbeat_front_addr":"172.21.15.32:6821/3066156116","state":["exists","up"]},{"osd":1,"uuid":"d7c5c210-da4f-49f9-b9e7-1dfb02daa4bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6824","nonce":1551241759},{"type":"v1","addr":"172.21.15.32:6825","nonce":1551241759}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6826","nonce":1551241759},{"type":"v1","addr":"172.21.15.32:6827","nonce":1551241759}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6830","nonce":1551241759},{"type":"v1","addr":"172.21.15.32:6831","nonce":1551241759}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6828","nonce":1551241759},{"type":"v1","addr":"172.21.15.32:6829","nonce":1551241759}]},"public_addr":"172.21.15.32:6825/1551241759","cluster_addr":"172.21.15.32:6827/1551241759","heartbeat_back_addr":"172.21.15.32:6831/1551241759","heartbeat_front_addr":"172.21.15.32:6829/1551241759","state":["exists","up"]},{"osd":2,"uuid":"57f2e6de-a201-45c2-8981-8e56a4875ebd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6808","nonce":3806708944},{"type":"v1","addr":"172.21.15.32:6809","nonce":3806708944}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6810","nonce":3806708944},{"type":"v1","addr":"172.21.15.32:6811","nonce":3806708944}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6814","nonce":3806708944},{"type":"v1","addr":"172.21.15.32:6815","nonce":3806708944}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6812","nonce":3806708944},{"type":"v1","addr":"172.21.15.32:6813","nonce":3806708944}]},"public_addr":"172.21.15.32:6809/3806708944","cluster_addr":"172.21.15.32:6811/3806708944","heartbeat_back_addr":"172.21.15.32:6815/3806708944","heartbeat_front_addr":"172.21.15.32:6813/3806708944","state":["exists","up"]},{"osd":3,"uuid":"6439a280-da45-4148-8130-f16323935ff3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6800","nonce":9088303},{"type":"v1","addr":"172.21.15.32:6801","nonce":9088303}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6802","nonce":9088303},{"type":"v1","addr":"172.21.15.32:6803","nonce":9088303}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6806","nonce":9088303},{"type":"v1","addr":"172.21.15.32:6807","nonce":9088303}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6804","nonce":9088303},{"type":"v1","addr":"172.21.15.32:6805","nonce":9088303}]},"public_addr":"172.21.15.32:6801/9088303","cluster_addr":"172.21.15.32:6803/9088303","heartbeat_back_addr":"172.21.15.32:6807/9088303","heartbeat_front_addr":"172.21.15.32:6805/9088303","state":["exists","up"]},{"osd":4,"uuid":"4f902eff-1e18-418d-82c9-2151c69524fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6802","nonce":864752275},{"type":"v1","addr":"172.21.15.118:6803","nonce":864752275}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6804","nonce":864752275},{"type":"v1","addr":"172.21.15.118:6805","nonce":864752275}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6808","nonce":864752275},{"type":"v1","addr":"172.21.15.118:6809","nonce":864752275}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6806","nonce":864752275},{"type":"v1","addr":"172.21.15.118:6807","nonce":864752275}]},"public_addr":"172.21.15.118:6803/864752275","cluster_addr":"172.21.15.118:6805/864752275","heartbeat_back_addr":"172.21.15.118:6809/864752275","heartbeat_front_addr":"172.21.15.118:6807/864752275","state":["exists","up"]},{"osd":5,"uuid":"bfd9feac-dd06-458c-91b9-af7f175f612c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6810","nonce":2724891770},{"type":"v1","addr":"172.21.15.118:6811","nonce":2724891770}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6812","nonce":2724891770},{"type":"v1","addr":"172.21.15.118:6813","nonce":2724891770}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6816","nonce":2724891770},{"type":"v1","addr":"172.21.15.118:6817","nonce":2724891770}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6814","nonce":2724891770},{"type":"v1","addr":"172.21.15.118:6815","nonce":2724891770}]},"public_addr":"172.21.15.118:6811/2724891770","cluster_addr":"172.21.15.118:6813/2724891770","heartbeat_back_addr":"172.21.15.118:6817/2724891770","heartbeat_front_addr":"172.21.15.118:6815/2724891770","state":["exists","up"]},{"osd":6,"uuid":"5ed5a279-9d8e-416d-bbc7-7edd3e6d5705","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6825","nonce":801947386},{"type":"v1","addr":"172.21.15.118:6827","nonce":801947386}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6828","nonce":801947386},{"type":"v1","addr":"172.21.15.118:6829","nonce":801947386}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6832","nonce":801947386},{"type":"v1","addr":"172.21.15.118:6833","nonce":801947386}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6830","nonce":801947386},{"type":"v1","addr":"172.21.15.118:6831","nonce":801947386}]},"public_addr":"172.21.15.118:6827/801947386","cluster_addr":"172.21.15.118:6829/801947386","heartbeat_back_addr":"172.21.15.118:6833/801947386","heartbeat_front_addr":"172.21.15.118:6831/801947386","state":["exists","up"]},{"osd":7,"uuid":"f2086c47-875e-4451-a436-2008d1ea16af","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6818","nonce":653000048},{"type":"v1","addr":"172.21.15.118:6819","nonce":653000048}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6820","nonce":653000048},{"type":"v1","addr":"172.21.15.118:6821","nonce":653000048}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6824","nonce":653000048},{"type":"v1","addr":"172.21.15.118:6826","nonce":653000048}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6822","nonce":653000048},{"type":"v1","addr":"172.21.15.118:6823","nonce":653000048}]},"public_addr":"172.21.15.118:6819/653000048","cluster_addr":"172.21.15.118:6821/653000048","heartbeat_back_addr":"172.21.15.118:6826/653000048","heartbeat_front_addr":"172.21.15.118:6823/653000048","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-03T20:47:25.491994+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-03T20:47:25.587358+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-03T20:47:25.454175+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-03T20:47:25.377734+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-03T20:47:25.412351+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-03T20:47:25.499815+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-03T20:47:25.575044+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-03T20:47:25.552383+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-08-03T20:54:27.931 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:27.929+0000 7fa08be00640 1 -- 172.21.15.32:0/4058210930 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fa064077010 msgr2=0x7fa064079520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:27.931 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:27.929+0000 7fa08be00640 1 --2- 172.21.15.32:0/4058210930 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fa064077010 0x7fa064079520 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7fa078034000 tx=0x7fa078004730 comp rx=0 tx=0).stop 2024-08-03T20:54:27.931 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:27.929+0000 7fa08be00640 1 -- 172.21.15.32:0/4058210930 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa0840747b0 msgr2=0x7fa0841436b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:27.931 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:27.929+0000 7fa08be00640 1 --2- 172.21.15.32:0/4058210930 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa0840747b0 0x7fa0841436b0 secure :-1 s=READY pgs=899 cs=0 l=1 rev1=1 crypto rx=0x7fa07c00cae0 tx=0x7fa07c00a060 comp rx=0 tx=0).stop 2024-08-03T20:54:27.931 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:27.929+0000 7fa08be00640 1 -- 172.21.15.32:0/4058210930 shutdown_connections 2024-08-03T20:54:27.931 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:27.929+0000 7fa08be00640 1 --2- 172.21.15.32:0/4058210930 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fa064077010 0x7fa064079520 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:27.931 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:27.929+0000 7fa08be00640 1 --2- 172.21.15.32:0/4058210930 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa084073e60 0x7fa0840aede0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:27.931 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:27.929+0000 7fa08be00640 1 --2- 172.21.15.32:0/4058210930 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa0840747b0 0x7fa0841436b0 unknown :-1 s=CLOSED pgs=899 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:27.931 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:27.929+0000 7fa08be00640 1 --2- 172.21.15.32:0/4058210930 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fa08406e930 0x7fa08415bd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:27.931 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:27.929+0000 7fa08be00640 1 -- 172.21.15.32:0/4058210930 >> 172.21.15.32:0/4058210930 conn(0x7fa0840b1bd0 msgr2=0x7fa0840b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:54:27.932 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:27.929+0000 7fa08be00640 1 -- 172.21.15.32:0/4058210930 shutdown_connections 2024-08-03T20:54:27.932 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:27.929+0000 7fa08be00640 1 -- 172.21.15.32:0/4058210930 wait complete. 2024-08-03T20:54:27.954 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-08-03T20:54:28.184 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:28.182+0000 7fecd1000640 1 -- 172.21.15.32:0/4178793666 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7feccc144f90 msgr2=0x7feccc074570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:28.184 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:28.182+0000 7fecd1000640 1 --2- 172.21.15.32:0/4178793666 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7feccc144f90 0x7feccc074570 secure :-1 s=READY pgs=900 cs=0 l=1 rev1=1 crypto rx=0x7feccc072d80 tx=0x7fecb402a5f0 comp rx=0 tx=0).stop 2024-08-03T20:54:28.184 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:28.182+0000 7fecd1000640 1 -- 172.21.15.32:0/4178793666 shutdown_connections 2024-08-03T20:54:28.184 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:28.182+0000 7fecd1000640 1 --2- 172.21.15.32:0/4178793666 >> v2:172.21.15.118:3300/0 conn(0x7feccc15fff0 0x7feccc160400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:28.184 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:28.182+0000 7fecd1000640 1 --2- 172.21.15.32:0/4178793666 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7feccc144f90 0x7feccc074570 unknown :-1 s=CLOSED pgs=900 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:28.184 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:28.182+0000 7fecd1000640 1 -- 172.21.15.32:0/4178793666 >> 172.21.15.32:0/4178793666 conn(0x7feccc0b1bd0 msgr2=0x7feccc0b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:54:28.184 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:28.182+0000 7fecd1000640 1 -- 172.21.15.32:0/4178793666 shutdown_connections 2024-08-03T20:54:28.184 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:28.182+0000 7fecd1000640 1 -- 172.21.15.32:0/4178793666 wait complete. 2024-08-03T20:54:28.186 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:28.184+0000 7fecd1000640 1 Processor -- start 2024-08-03T20:54:28.186 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:28.185+0000 7fecd1000640 1 -- start start 2024-08-03T20:54:28.189 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:28.186+0000 7fecd1000640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7feccc06e930 0x7feccc15e4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:28.189 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:28.186+0000 7fecd1000640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7feccc144f90 0x7feccc15f2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:28.189 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:28.186+0000 7fecd1000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7feccc15fff0 0x7feccc15f820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:28.189 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:28.186+0000 7fecd1000640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7feccc0cf840 con 0x7feccc15fff0 2024-08-03T20:54:28.189 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:28.186+0000 7fecd1000640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7feccc145f30 con 0x7feccc06e930 2024-08-03T20:54:28.189 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:28.186+0000 7fecd1000640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7feccc0cf9c0 con 0x7feccc144f90 2024-08-03T20:54:28.189 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:28.186+0000 7fecca800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7feccc15fff0 0x7feccc15f820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:28.189 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:28.186+0000 7fecca800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7feccc15fff0 0x7feccc15f820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:50344/0 (socket says 172.21.15.32:50344) 2024-08-03T20:54:28.190 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:28.186+0000 7fecca800640 1 -- 172.21.15.32:0/2876879669 learned_addr learned my addr 172.21.15.32:0/2876879669 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:54:28.190 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:28.186+0000 7fecc9e00640 1 --2- 172.21.15.32:0/2876879669 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7feccc06e930 0x7feccc15e4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:28.190 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:28.186+0000 7fecc3e00640 1 --2- 172.21.15.32:0/2876879669 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7feccc144f90 0x7feccc15f2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:28.190 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:28.187+0000 7fecca800640 1 -- 172.21.15.32:0/2876879669 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7feccc144f90 msgr2=0x7feccc15f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:28.190 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:28.187+0000 7fecca800640 1 --2- 172.21.15.32:0/2876879669 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7feccc144f90 0x7feccc15f2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:28.190 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:28.187+0000 7fecca800640 1 -- 172.21.15.32:0/2876879669 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7feccc06e930 msgr2=0x7feccc15e4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:28.190 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:28.187+0000 7fecca800640 1 --2- 172.21.15.32:0/2876879669 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7feccc06e930 0x7feccc15e4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:28.190 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:28.187+0000 7fecca800640 1 -- 172.21.15.32:0/2876879669 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feccc0add80 con 0x7feccc15fff0 2024-08-03T20:54:28.190 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:28.187+0000 7fecca800640 1 --2- 172.21.15.32:0/2876879669 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7feccc15fff0 0x7feccc15f820 secure :-1 s=READY pgs=901 cs=0 l=1 rev1=1 crypto rx=0x7fecc400e990 tx=0x7fecc400ee50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:54:28.190 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:28.187+0000 7fecc2000640 1 -- 172.21.15.32:0/2876879669 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fecc4004d20 con 0x7feccc15fff0 2024-08-03T20:54:28.190 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:28.187+0000 7fecc2000640 1 -- 172.21.15.32:0/2876879669 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fecc400cb00 con 0x7feccc15fff0 2024-08-03T20:54:28.190 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:28.187+0000 7fecc2000640 1 -- 172.21.15.32:0/2876879669 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fecc4005020 con 0x7feccc15fff0 2024-08-03T20:54:28.190 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:28.187+0000 7fecd1000640 1 -- 172.21.15.32:0/2876879669 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7feccc0ae060 con 0x7feccc15fff0 2024-08-03T20:54:28.191 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:28.188+0000 7fecd1000640 1 -- 172.21.15.32:0/2876879669 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7feccc0ae620 con 0x7feccc15fff0 2024-08-03T20:54:28.191 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:28.189+0000 7fecd1000640 1 -- 172.21.15.32:0/2876879669 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7feccc160c30 con 0x7feccc15fff0 2024-08-03T20:54:28.191 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:28.190+0000 7fecc2000640 1 -- 172.21.15.32:0/2876879669 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7fecc40051c0 con 0x7feccc15fff0 2024-08-03T20:54:28.193 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:28.191+0000 7fecc2000640 1 --2- 172.21.15.32:0/2876879669 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7feca8077010 0x7feca8079520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:28.193 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:28.192+0000 7fecc2000640 1 -- 172.21.15.32:0/2876879669 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 5709+0+0 (secure 0 0 0) 0x7fecc40991b0 con 0x7feccc15fff0 2024-08-03T20:54:28.193 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:28.192+0000 7fecc9e00640 1 --2- 172.21.15.32:0/2876879669 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7feca8077010 0x7feca8079520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:28.194 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:28.193+0000 7fecc9e00640 1 --2- 172.21.15.32:0/2876879669 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7feca8077010 0x7feca8079520 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7fecbc001720 tx=0x7fecbc009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:54:28.197 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:28.196+0000 7fecc2000640 1 -- 172.21.15.32:0/2876879669 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7fecc4016030 con 0x7feccc15fff0 2024-08-03T20:54:28.404 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:28.403+0000 7fecd1000640 1 -- 172.21.15.32:0/2876879669 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7feccc0adb10 con 0x7feccc15fff0 2024-08-03T20:54:28.405 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:28.404+0000 7fecc2000640 1 -- 172.21.15.32:0/2876879669 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v82) ==== 74+0+16613 (secure 0 0 0) 0x7fecc4061d50 con 0x7feccc15fff0 2024-08-03T20:54:28.406 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-03T20:54:28.406 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":82,"fsid":"40ab141a-01d7-49b9-ae3c-2edec9876e95","created":"2024-08-03T20:47:14.957362+0000","modified":"2024-08-03T20:54:26.950564+0000","last_up_change":"2024-08-03T20:47:28.874451+0000","last_in_change":"2024-08-03T20:47:20.079442+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":13,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-08-03T20:47:28.261300+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_metadata","create_time":"2024-08-03T20:54:24.911555+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"82","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs_data","create_time":"2024-08-03T20:54:25.917207+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"82","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5a83e759-9035-438b-889a-0a6af1f0a0d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6816","nonce":3066156116},{"type":"v1","addr":"172.21.15.32:6817","nonce":3066156116}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6818","nonce":3066156116},{"type":"v1","addr":"172.21.15.32:6819","nonce":3066156116}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6822","nonce":3066156116},{"type":"v1","addr":"172.21.15.32:6823","nonce":3066156116}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6820","nonce":3066156116},{"type":"v1","addr":"172.21.15.32:6821","nonce":3066156116}]},"public_addr":"172.21.15.32:6817/3066156116","cluster_addr":"172.21.15.32:6819/3066156116","heartbeat_back_addr":"172.21.15.32:6823/3066156116","heartbeat_front_addr":"172.21.15.32:6821/3066156116","state":["exists","up"]},{"osd":1,"uuid":"d7c5c210-da4f-49f9-b9e7-1dfb02daa4bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6824","nonce":1551241759},{"type":"v1","addr":"172.21.15.32:6825","nonce":1551241759}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6826","nonce":1551241759},{"type":"v1","addr":"172.21.15.32:6827","nonce":1551241759}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6830","nonce":1551241759},{"type":"v1","addr":"172.21.15.32:6831","nonce":1551241759}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6828","nonce":1551241759},{"type":"v1","addr":"172.21.15.32:6829","nonce":1551241759}]},"public_addr":"172.21.15.32:6825/1551241759","cluster_addr":"172.21.15.32:6827/1551241759","heartbeat_back_addr":"172.21.15.32:6831/1551241759","heartbeat_front_addr":"172.21.15.32:6829/1551241759","state":["exists","up"]},{"osd":2,"uuid":"57f2e6de-a201-45c2-8981-8e56a4875ebd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6808","nonce":3806708944},{"type":"v1","addr":"172.21.15.32:6809","nonce":3806708944}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6810","nonce":3806708944},{"type":"v1","addr":"172.21.15.32:6811","nonce":3806708944}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6814","nonce":3806708944},{"type":"v1","addr":"172.21.15.32:6815","nonce":3806708944}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6812","nonce":3806708944},{"type":"v1","addr":"172.21.15.32:6813","nonce":3806708944}]},"public_addr":"172.21.15.32:6809/3806708944","cluster_addr":"172.21.15.32:6811/3806708944","heartbeat_back_addr":"172.21.15.32:6815/3806708944","heartbeat_front_addr":"172.21.15.32:6813/3806708944","state":["exists","up"]},{"osd":3,"uuid":"6439a280-da45-4148-8130-f16323935ff3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6800","nonce":9088303},{"type":"v1","addr":"172.21.15.32:6801","nonce":9088303}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6802","nonce":9088303},{"type":"v1","addr":"172.21.15.32:6803","nonce":9088303}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6806","nonce":9088303},{"type":"v1","addr":"172.21.15.32:6807","nonce":9088303}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6804","nonce":9088303},{"type":"v1","addr":"172.21.15.32:6805","nonce":9088303}]},"public_addr":"172.21.15.32:6801/9088303","cluster_addr":"172.21.15.32:6803/9088303","heartbeat_back_addr":"172.21.15.32:6807/9088303","heartbeat_front_addr":"172.21.15.32:6805/9088303","state":["exists","up"]},{"osd":4,"uuid":"4f902eff-1e18-418d-82c9-2151c69524fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6802","nonce":864752275},{"type":"v1","addr":"172.21.15.118:6803","nonce":864752275}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6804","nonce":864752275},{"type":"v1","addr":"172.21.15.118:6805","nonce":864752275}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6808","nonce":864752275},{"type":"v1","addr":"172.21.15.118:6809","nonce":864752275}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6806","nonce":864752275},{"type":"v1","addr":"172.21.15.118:6807","nonce":864752275}]},"public_addr":"172.21.15.118:6803/864752275","cluster_addr":"172.21.15.118:6805/864752275","heartbeat_back_addr":"172.21.15.118:6809/864752275","heartbeat_front_addr":"172.21.15.118:6807/864752275","state":["exists","up"]},{"osd":5,"uuid":"bfd9feac-dd06-458c-91b9-af7f175f612c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6810","nonce":2724891770},{"type":"v1","addr":"172.21.15.118:6811","nonce":2724891770}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6812","nonce":2724891770},{"type":"v1","addr":"172.21.15.118:6813","nonce":2724891770}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6816","nonce":2724891770},{"type":"v1","addr":"172.21.15.118:6817","nonce":2724891770}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6814","nonce":2724891770},{"type":"v1","addr":"172.21.15.118:6815","nonce":2724891770}]},"public_addr":"172.21.15.118:6811/2724891770","cluster_addr":"172.21.15.118:6813/2724891770","heartbeat_back_addr":"172.21.15.118:6817/2724891770","heartbeat_front_addr":"172.21.15.118:6815/2724891770","state":["exists","up"]},{"osd":6,"uuid":"5ed5a279-9d8e-416d-bbc7-7edd3e6d5705","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6825","nonce":801947386},{"type":"v1","addr":"172.21.15.118:6827","nonce":801947386}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6828","nonce":801947386},{"type":"v1","addr":"172.21.15.118:6829","nonce":801947386}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6832","nonce":801947386},{"type":"v1","addr":"172.21.15.118:6833","nonce":801947386}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6830","nonce":801947386},{"type":"v1","addr":"172.21.15.118:6831","nonce":801947386}]},"public_addr":"172.21.15.118:6827/801947386","cluster_addr":"172.21.15.118:6829/801947386","heartbeat_back_addr":"172.21.15.118:6833/801947386","heartbeat_front_addr":"172.21.15.118:6831/801947386","state":["exists","up"]},{"osd":7,"uuid":"f2086c47-875e-4451-a436-2008d1ea16af","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6818","nonce":653000048},{"type":"v1","addr":"172.21.15.118:6819","nonce":653000048}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6820","nonce":653000048},{"type":"v1","addr":"172.21.15.118:6821","nonce":653000048}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6824","nonce":653000048},{"type":"v1","addr":"172.21.15.118:6826","nonce":653000048}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6822","nonce":653000048},{"type":"v1","addr":"172.21.15.118:6823","nonce":653000048}]},"public_addr":"172.21.15.118:6819/653000048","cluster_addr":"172.21.15.118:6821/653000048","heartbeat_back_addr":"172.21.15.118:6826/653000048","heartbeat_front_addr":"172.21.15.118:6823/653000048","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-03T20:47:25.491994+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-03T20:47:25.587358+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-03T20:47:25.454175+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-03T20:47:25.377734+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-03T20:47:25.412351+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-03T20:47:25.499815+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-03T20:47:25.575044+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-03T20:47:25.552383+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-08-03T20:54:28.408 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:28.406+0000 7fecd1000640 1 -- 172.21.15.32:0/2876879669 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7feca8077010 msgr2=0x7feca8079520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:28.408 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:28.406+0000 7fecd1000640 1 --2- 172.21.15.32:0/2876879669 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7feca8077010 0x7feca8079520 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7fecbc001720 tx=0x7fecbc009040 comp rx=0 tx=0).stop 2024-08-03T20:54:28.408 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:28.406+0000 7fecd1000640 1 -- 172.21.15.32:0/2876879669 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7feccc15fff0 msgr2=0x7feccc15f820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:28.408 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:28.406+0000 7fecd1000640 1 --2- 172.21.15.32:0/2876879669 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7feccc15fff0 0x7feccc15f820 secure :-1 s=READY pgs=901 cs=0 l=1 rev1=1 crypto rx=0x7fecc400e990 tx=0x7fecc400ee50 comp rx=0 tx=0).stop 2024-08-03T20:54:28.408 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:28.406+0000 7fecd1000640 1 -- 172.21.15.32:0/2876879669 shutdown_connections 2024-08-03T20:54:28.408 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:28.406+0000 7fecd1000640 1 --2- 172.21.15.32:0/2876879669 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7feca8077010 0x7feca8079520 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:28.408 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:28.406+0000 7fecd1000640 1 --2- 172.21.15.32:0/2876879669 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7feccc15fff0 0x7feccc15f820 unknown :-1 s=CLOSED pgs=901 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:28.408 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:28.406+0000 7fecd1000640 1 --2- 172.21.15.32:0/2876879669 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7feccc144f90 0x7feccc15f2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:28.409 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:28.406+0000 7fecd1000640 1 --2- 172.21.15.32:0/2876879669 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7feccc06e930 0x7feccc15e4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:28.409 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:28.406+0000 7fecd1000640 1 -- 172.21.15.32:0/2876879669 >> 172.21.15.32:0/2876879669 conn(0x7feccc0b1bd0 msgr2=0x7feccc0b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:54:28.409 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:28.406+0000 7fecd1000640 1 -- 172.21.15.32:0/2876879669 shutdown_connections 2024-08-03T20:54:28.409 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:28.406+0000 7fecd1000640 1 -- 172.21.15.32:0/2876879669 wait complete. 2024-08-03T20:54:28.430 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-08-03T20:54:28.661 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:28.659+0000 7f4132400640 1 -- 172.21.15.32:0/1497884905 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f412c0cccf0 msgr2=0x7f412c0cd100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:28.662 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:28.659+0000 7f4132400640 1 --2- 172.21.15.32:0/1497884905 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f412c0cccf0 0x7f412c0cd100 secure :-1 s=READY pgs=903 cs=0 l=1 rev1=1 crypto rx=0x7f4118009830 tx=0x7f411802a500 comp rx=0 tx=0).stop 2024-08-03T20:54:28.662 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:28.660+0000 7f4132400640 1 -- 172.21.15.32:0/1497884905 shutdown_connections 2024-08-03T20:54:28.662 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:28.660+0000 7f4132400640 1 --2- 172.21.15.32:0/1497884905 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f412c0ced30 0x7f412c0cf140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:28.662 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:28.660+0000 7f4132400640 1 --2- 172.21.15.32:0/1497884905 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f412c0cccf0 0x7f412c0cd100 unknown :-1 s=CLOSED pgs=903 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:28.662 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:28.660+0000 7f4132400640 1 -- 172.21.15.32:0/1497884905 >> 172.21.15.32:0/1497884905 conn(0x7f412c0b1bd0 msgr2=0x7f412c0b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:54:28.662 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:28.660+0000 7f4132400640 1 -- 172.21.15.32:0/1497884905 shutdown_connections 2024-08-03T20:54:28.662 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:28.660+0000 7f4132400640 1 -- 172.21.15.32:0/1497884905 wait complete. 2024-08-03T20:54:28.663 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:28.662+0000 7f4132400640 1 Processor -- start 2024-08-03T20:54:28.663 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:28.662+0000 7f4132400640 1 -- start start 2024-08-03T20:54:28.666 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:28.663+0000 7f4132400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f412c0cccf0 0x7f412c138b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:28.666 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:28.663+0000 7f4132400640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f412c0ced30 0x7f412c13a610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:28.666 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:28.663+0000 7f4132400640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f412c145c50 0x7f412c13d3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:28.666 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:28.663+0000 7f4132400640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f412c0b4070 con 0x7f412c0cccf0 2024-08-03T20:54:28.666 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:28.663+0000 7f4132400640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f412c0b3ef0 con 0x7f412c0ced30 2024-08-03T20:54:28.666 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:28.663+0000 7f4132400640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7f412c0b41f0 con 0x7f412c145c50 2024-08-03T20:54:28.666 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:28.664+0000 7f412b400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f412c0cccf0 0x7f412c138b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:28.666 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:28.664+0000 7f412b400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f412c0cccf0 0x7f412c138b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:50372/0 (socket says 172.21.15.32:50372) 2024-08-03T20:54:28.666 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:28.664+0000 7f412b400640 1 -- 172.21.15.32:0/4161393666 learned_addr learned my addr 172.21.15.32:0/4161393666 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:54:28.666 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:28.664+0000 7f412be00640 1 --2- 172.21.15.32:0/4161393666 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f412c145c50 0x7f412c13d3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:28.667 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:28.664+0000 7f412aa00640 1 --2- 172.21.15.32:0/4161393666 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f412c0ced30 0x7f412c13a610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:28.667 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:28.664+0000 7f412b400640 1 -- 172.21.15.32:0/4161393666 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f412c145c50 msgr2=0x7f412c13d3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:28.667 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:28.664+0000 7f412b400640 1 --2- 172.21.15.32:0/4161393666 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f412c145c50 0x7f412c13d3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:28.667 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:28.664+0000 7f412b400640 1 -- 172.21.15.32:0/4161393666 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f412c0ced30 msgr2=0x7f412c13a610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:28.667 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:28.664+0000 7f412b400640 1 --2- 172.21.15.32:0/4161393666 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f412c0ced30 0x7f412c13a610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:28.667 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:28.664+0000 7f412b400640 1 -- 172.21.15.32:0/4161393666 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f412c13ab50 con 0x7f412c0cccf0 2024-08-03T20:54:28.667 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:28.664+0000 7f412b400640 1 --2- 172.21.15.32:0/4161393666 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f412c0cccf0 0x7f412c138b30 secure :-1 s=READY pgs=904 cs=0 l=1 rev1=1 crypto rx=0x7f411c00c3f0 tx=0x7f411c00c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:54:28.668 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:28.665+0000 7f4123400640 1 -- 172.21.15.32:0/4161393666 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f411c017070 con 0x7f412c0cccf0 2024-08-03T20:54:28.668 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:28.665+0000 7f4123400640 1 -- 172.21.15.32:0/4161393666 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f411c0035f0 con 0x7f412c0cccf0 2024-08-03T20:54:28.668 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:28.665+0000 7f4123400640 1 -- 172.21.15.32:0/4161393666 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f411c003a40 con 0x7f412c0cccf0 2024-08-03T20:54:28.668 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:28.665+0000 7f4132400640 1 -- 172.21.15.32:0/4161393666 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f412c1393d0 con 0x7f412c0cccf0 2024-08-03T20:54:28.668 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:28.665+0000 7f4132400640 1 -- 172.21.15.32:0/4161393666 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f412c139990 con 0x7f412c0cccf0 2024-08-03T20:54:28.668 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:28.666+0000 7f4132400640 1 -- 172.21.15.32:0/4161393666 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f412c0b4700 con 0x7f412c0cccf0 2024-08-03T20:54:28.668 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:28.667+0000 7f4123400640 1 -- 172.21.15.32:0/4161393666 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f411c01a070 con 0x7f412c0cccf0 2024-08-03T20:54:28.670 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:28.668+0000 7f4123400640 1 --2- 172.21.15.32:0/4161393666 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f410c077010 0x7f410c079520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:28.670 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:28.668+0000 7f4123400640 1 -- 172.21.15.32:0/4161393666 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 5709+0+0 (secure 0 0 0) 0x7f411c097ca0 con 0x7f412c0cccf0 2024-08-03T20:54:28.670 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:28.669+0000 7f412aa00640 1 --2- 172.21.15.32:0/4161393666 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f410c077010 0x7f410c079520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:28.671 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:28.670+0000 7f412aa00640 1 --2- 172.21.15.32:0/4161393666 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f410c077010 0x7f410c079520 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f4118034000 tx=0x7f4118004730 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:54:28.674 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:28.673+0000 7f4123400640 1 -- 172.21.15.32:0/4161393666 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7f411c060f20 con 0x7f412c0cccf0 2024-08-03T20:54:28.918 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:28.916+0000 7f4132400640 1 -- 172.21.15.32:0/4161393666 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) -- 0x7f412c0d0b50 con 0x7f412c0cccf0 2024-08-03T20:54:29.444 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:29.443+0000 7f4123400640 1 -- 172.21.15.32:0/4161393666 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v90) ==== 118+0+0 (secure 0 0 0) 0x7f411c069160 con 0x7f412c0cccf0 2024-08-03T20:54:29.447 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:29.445+0000 7f4132400640 1 -- 172.21.15.32:0/4161393666 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f410c077010 msgr2=0x7f410c079520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:29.447 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:29.445+0000 7f4132400640 1 --2- 172.21.15.32:0/4161393666 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f410c077010 0x7f410c079520 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f4118034000 tx=0x7f4118004730 comp rx=0 tx=0).stop 2024-08-03T20:54:29.447 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:29.445+0000 7f4132400640 1 -- 172.21.15.32:0/4161393666 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f412c0cccf0 msgr2=0x7f412c138b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:29.447 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:29.445+0000 7f4132400640 1 --2- 172.21.15.32:0/4161393666 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f412c0cccf0 0x7f412c138b30 secure :-1 s=READY pgs=904 cs=0 l=1 rev1=1 crypto rx=0x7f411c00c3f0 tx=0x7f411c00c8b0 comp rx=0 tx=0).stop 2024-08-03T20:54:29.448 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:29.446+0000 7f4132400640 1 -- 172.21.15.32:0/4161393666 shutdown_connections 2024-08-03T20:54:29.448 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:29.446+0000 7f4132400640 1 --2- 172.21.15.32:0/4161393666 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f410c077010 0x7f410c079520 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:29.448 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:29.446+0000 7f4132400640 1 --2- 172.21.15.32:0/4161393666 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f412c0ced30 0x7f412c13a610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:29.448 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:29.446+0000 7f4132400640 1 --2- 172.21.15.32:0/4161393666 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f412c145c50 0x7f412c13d3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:29.448 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:29.446+0000 7f4132400640 1 --2- 172.21.15.32:0/4161393666 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f412c0cccf0 0x7f412c138b30 unknown :-1 s=CLOSED pgs=904 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:29.448 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:29.446+0000 7f4132400640 1 -- 172.21.15.32:0/4161393666 >> 172.21.15.32:0/4161393666 conn(0x7f412c0b1bd0 msgr2=0x7f412c15f4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:54:29.448 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:29.446+0000 7f4132400640 1 -- 172.21.15.32:0/4161393666 shutdown_connections 2024-08-03T20:54:29.448 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:29.446+0000 7f4132400640 1 -- 172.21.15.32:0/4161393666 wait complete. 2024-08-03T20:54:29.479 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-08-03T20:54:29.479 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-08-03T20:54:29.714 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:29.712+0000 7fe1c5e00640 1 -- 172.21.15.32:0/208236094 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe1c015d2b0 msgr2=0x7fe1c00ced30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:29.714 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:29.712+0000 7fe1c5e00640 1 --2- 172.21.15.32:0/208236094 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe1c015d2b0 0x7fe1c00ced30 secure :-1 s=READY pgs=905 cs=0 l=1 rev1=1 crypto rx=0x7fe1b0005fd0 tx=0x7fe1b002a530 comp rx=0 tx=0).stop 2024-08-03T20:54:29.714 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:29.712+0000 7fe1c5e00640 1 -- 172.21.15.32:0/208236094 shutdown_connections 2024-08-03T20:54:29.714 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:29.712+0000 7fe1c5e00640 1 --2- 172.21.15.32:0/208236094 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fe1c006e950 0x7fe1c006ed60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:29.714 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:29.712+0000 7fe1c5e00640 1 --2- 172.21.15.32:0/208236094 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe1c015d2b0 0x7fe1c00ced30 unknown :-1 s=CLOSED pgs=905 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:29.714 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:29.712+0000 7fe1c5e00640 1 -- 172.21.15.32:0/208236094 >> 172.21.15.32:0/208236094 conn(0x7fe1c00b1bd0 msgr2=0x7fe1c00b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:54:29.714 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:29.712+0000 7fe1c5e00640 1 -- 172.21.15.32:0/208236094 shutdown_connections 2024-08-03T20:54:29.714 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:29.712+0000 7fe1c5e00640 1 -- 172.21.15.32:0/208236094 wait complete. 2024-08-03T20:54:29.715 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:29.714+0000 7fe1c5e00640 1 Processor -- start 2024-08-03T20:54:29.715 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:29.714+0000 7fe1c5e00640 1 -- start start 2024-08-03T20:54:29.717 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:29.716+0000 7fe1c5e00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fe1c006e950 0x7fe1c0151c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:29.718 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:29.716+0000 7fe1c5e00640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fe1c00b3e80 0x7fe1c01f47d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:29.718 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:29.716+0000 7fe1c5e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe1c015d2b0 0x7fe1c01f4d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:29.718 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:29.716+0000 7fe1c5e00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fe1c00cd510 con 0x7fe1c015d2b0 2024-08-03T20:54:29.718 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:29.716+0000 7fe1c5e00640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7fe1c00cd390 con 0x7fe1c006e950 2024-08-03T20:54:29.718 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:29.716+0000 7fe1c5e00640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7fe1c00b4290 con 0x7fe1c00b3e80 2024-08-03T20:54:29.718 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:29.716+0000 7fe1bf400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe1c015d2b0 0x7fe1c01f4d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:29.718 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:29.716+0000 7fe1bea00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fe1c006e950 0x7fe1c0151c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:29.718 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:29.716+0000 7fe1bf400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe1c015d2b0 0x7fe1c01f4d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:50388/0 (socket says 172.21.15.32:50388) 2024-08-03T20:54:29.718 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:29.716+0000 7fe1bf400640 1 -- 172.21.15.32:0/2981339917 learned_addr learned my addr 172.21.15.32:0/2981339917 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:54:29.718 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:29.716+0000 7fe1b7e00640 1 --2- 172.21.15.32:0/2981339917 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fe1c00b3e80 0x7fe1c01f47d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:29.719 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:29.717+0000 7fe1bf400640 1 -- 172.21.15.32:0/2981339917 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fe1c00b3e80 msgr2=0x7fe1c01f47d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:29.719 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:29.717+0000 7fe1bf400640 1 --2- 172.21.15.32:0/2981339917 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fe1c00b3e80 0x7fe1c01f47d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:29.719 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:29.717+0000 7fe1bf400640 1 -- 172.21.15.32:0/2981339917 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fe1c006e950 msgr2=0x7fe1c0151c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:29.719 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:29.717+0000 7fe1bf400640 1 --2- 172.21.15.32:0/2981339917 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fe1c006e950 0x7fe1c0151c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:29.720 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:29.717+0000 7fe1bf400640 1 -- 172.21.15.32:0/2981339917 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe1c0152940 con 0x7fe1c015d2b0 2024-08-03T20:54:29.720 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:29.717+0000 7fe1bf400640 1 --2- 172.21.15.32:0/2981339917 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe1c015d2b0 0x7fe1c01f4d10 secure :-1 s=READY pgs=906 cs=0 l=1 rev1=1 crypto rx=0x7fe1b800be90 tx=0x7fe1b800bf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:54:29.720 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:29.717+0000 7fe1b7400640 1 -- 172.21.15.32:0/2981339917 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1b8019070 con 0x7fe1c015d2b0 2024-08-03T20:54:29.720 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:29.717+0000 7fe1b7400640 1 -- 172.21.15.32:0/2981339917 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fe1b8014440 con 0x7fe1c015d2b0 2024-08-03T20:54:29.720 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:29.717+0000 7fe1b7400640 1 -- 172.21.15.32:0/2981339917 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1b80148d0 con 0x7fe1c015d2b0 2024-08-03T20:54:29.721 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:29.717+0000 7fe1c5e00640 1 -- 172.21.15.32:0/2981339917 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe1c0152c20 con 0x7fe1c015d2b0 2024-08-03T20:54:29.721 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:29.717+0000 7fe1c5e00640 1 -- 172.21.15.32:0/2981339917 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe1c01531e0 con 0x7fe1c015d2b0 2024-08-03T20:54:29.722 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:29.718+0000 7fe1c5e00640 1 -- 172.21.15.32:0/2981339917 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe1c006ed60 con 0x7fe1c015d2b0 2024-08-03T20:54:29.722 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:29.720+0000 7fe1b7400640 1 -- 172.21.15.32:0/2981339917 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7fe1b801c070 con 0x7fe1c015d2b0 2024-08-03T20:54:29.722 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:29.721+0000 7fe1b7400640 1 --2- 172.21.15.32:0/2981339917 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fe19c077010 0x7fe19c079520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:29.723 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:29.721+0000 7fe1b7400640 1 -- 172.21.15.32:0/2981339917 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 5709+0+0 (secure 0 0 0) 0x7fe1b80989e0 con 0x7fe1c015d2b0 2024-08-03T20:54:29.723 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:29.721+0000 7fe1bea00640 1 --2- 172.21.15.32:0/2981339917 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fe19c077010 0x7fe19c079520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:29.723 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:29.722+0000 7fe1bea00640 1 --2- 172.21.15.32:0/2981339917 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fe19c077010 0x7fe19c079520 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7fe1b0033000 tx=0x7fe1b0004730 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:54:29.726 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:29.725+0000 7fe1b7400640 1 -- 172.21.15.32:0/2981339917 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7fe1b80614f0 con 0x7fe1c015d2b0 2024-08-03T20:54:29.976 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:29.974+0000 7fe1c5e00640 1 -- 172.21.15.32:0/2981339917 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) -- 0x7fe1c01526d0 con 0x7fe1c015d2b0 2024-08-03T20:54:30.451 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:30.449+0000 7fe1b7400640 1 -- 172.21.15.32:0/2981339917 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v91) ==== 122+0+0 (secure 0 0 0) 0x7fe1b8065a40 con 0x7fe1c015d2b0 2024-08-03T20:54:30.455 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:30.453+0000 7fe1c5e00640 1 -- 172.21.15.32:0/2981339917 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fe19c077010 msgr2=0x7fe19c079520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:30.455 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:30.453+0000 7fe1c5e00640 1 --2- 172.21.15.32:0/2981339917 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fe19c077010 0x7fe19c079520 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7fe1b0033000 tx=0x7fe1b0004730 comp rx=0 tx=0).stop 2024-08-03T20:54:30.455 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:30.453+0000 7fe1c5e00640 1 -- 172.21.15.32:0/2981339917 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe1c015d2b0 msgr2=0x7fe1c01f4d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:30.455 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:30.453+0000 7fe1c5e00640 1 --2- 172.21.15.32:0/2981339917 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe1c015d2b0 0x7fe1c01f4d10 secure :-1 s=READY pgs=906 cs=0 l=1 rev1=1 crypto rx=0x7fe1b800be90 tx=0x7fe1b800bf80 comp rx=0 tx=0).stop 2024-08-03T20:54:30.455 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:30.453+0000 7fe1c5e00640 1 -- 172.21.15.32:0/2981339917 shutdown_connections 2024-08-03T20:54:30.455 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:30.453+0000 7fe1c5e00640 1 --2- 172.21.15.32:0/2981339917 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fe19c077010 0x7fe19c079520 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:30.455 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:30.453+0000 7fe1c5e00640 1 --2- 172.21.15.32:0/2981339917 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe1c015d2b0 0x7fe1c01f4d10 unknown :-1 s=CLOSED pgs=906 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:30.455 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:30.453+0000 7fe1c5e00640 1 --2- 172.21.15.32:0/2981339917 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fe1c00b3e80 0x7fe1c01f47d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:30.455 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:30.453+0000 7fe1c5e00640 1 --2- 172.21.15.32:0/2981339917 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fe1c006e950 0x7fe1c0151c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:30.455 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:30.453+0000 7fe1c5e00640 1 -- 172.21.15.32:0/2981339917 >> 172.21.15.32:0/2981339917 conn(0x7fe1c00b1bd0 msgr2=0x7fe1c00cff60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:54:30.456 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:30.453+0000 7fe1c5e00640 1 -- 172.21.15.32:0/2981339917 shutdown_connections 2024-08-03T20:54:30.456 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:30.453+0000 7fe1c5e00640 1 -- 172.21.15.32:0/2981339917 wait complete. 2024-08-03T20:54:30.480 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-08-03T20:54:30.480 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-08-03T20:54:30.712 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:30.711+0000 7f28b6c00640 1 -- 172.21.15.32:0/67234594 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f28b006e930 msgr2=0x7f28b0144f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:30.713 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:30.711+0000 7f28b6c00640 1 --2- 172.21.15.32:0/67234594 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f28b006e930 0x7f28b0144f90 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f28a0009830 tx=0x7f28a002a480 comp rx=0 tx=0).stop 2024-08-03T20:54:30.713 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:30.711+0000 7f28b6c00640 1 -- 172.21.15.32:0/67234594 shutdown_connections 2024-08-03T20:54:30.713 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:30.711+0000 7f28b6c00640 1 --2- 172.21.15.32:0/67234594 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f28b0073e60 0x7f28b0074270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:30.713 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:30.711+0000 7f28b6c00640 1 --2- 172.21.15.32:0/67234594 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f28b006e930 0x7f28b0144f90 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:30.713 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:30.711+0000 7f28b6c00640 1 -- 172.21.15.32:0/67234594 >> 172.21.15.32:0/67234594 conn(0x7f28b00b1bd0 msgr2=0x7f28b00b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:54:30.713 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:30.711+0000 7f28b6c00640 1 -- 172.21.15.32:0/67234594 shutdown_connections 2024-08-03T20:54:30.713 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:30.711+0000 7f28b6c00640 1 -- 172.21.15.32:0/67234594 wait complete. 2024-08-03T20:54:30.714 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:30.713+0000 7f28b6c00640 1 Processor -- start 2024-08-03T20:54:30.714 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:30.713+0000 7f28b6c00640 1 -- start start 2024-08-03T20:54:30.716 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:30.714+0000 7f28b6c00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f28b006e930 0x7f28b015f360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:30.716 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:30.715+0000 7f28b6c00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f28b0073e60 0x7f28b0141e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:30.716 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:30.715+0000 7f28b6c00640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f28b00747b0 0x7f28b01423d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:30.716 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:30.715+0000 7f28b6c00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f28b00cf9c0 con 0x7f28b006e930 2024-08-03T20:54:30.716 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:30.715+0000 7f28b6c00640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f28b00cf840 con 0x7f28b0073e60 2024-08-03T20:54:30.717 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:30.715+0000 7f28b6c00640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7f28b00ccff0 con 0x7f28b00747b0 2024-08-03T20:54:30.717 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:30.715+0000 7f28afe00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f28b006e930 0x7f28b015f360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:30.717 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:30.715+0000 7f28afe00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f28b006e930 0x7f28b015f360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:39960/0 (socket says 172.21.15.32:39960) 2024-08-03T20:54:30.717 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:30.715+0000 7f28afe00640 1 -- 172.21.15.32:0/2100098826 learned_addr learned my addr 172.21.15.32:0/2100098826 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:54:30.718 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:30.715+0000 7f28b4c00640 1 --2- 172.21.15.32:0/2100098826 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f28b00747b0 0x7f28b01423d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:30.718 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:30.715+0000 7f28af400640 1 --2- 172.21.15.32:0/2100098826 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f28b0073e60 0x7f28b0141e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:30.718 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:30.715+0000 7f28afe00640 1 -- 172.21.15.32:0/2100098826 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f28b00747b0 msgr2=0x7f28b01423d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:30.718 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:30.715+0000 7f28afe00640 1 --2- 172.21.15.32:0/2100098826 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f28b00747b0 0x7f28b01423d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:30.718 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:30.715+0000 7f28afe00640 1 -- 172.21.15.32:0/2100098826 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f28b0073e60 msgr2=0x7f28b0141e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:30.718 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:30.715+0000 7f28afe00640 1 --2- 172.21.15.32:0/2100098826 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f28b0073e60 0x7f28b0141e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:30.718 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:30.715+0000 7f28afe00640 1 -- 172.21.15.32:0/2100098826 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f28b0141700 con 0x7f28b006e930 2024-08-03T20:54:30.718 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:30.716+0000 7f28afe00640 1 --2- 172.21.15.32:0/2100098826 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f28b006e930 0x7f28b015f360 secure :-1 s=READY pgs=907 cs=0 l=1 rev1=1 crypto rx=0x7f28a0009950 tx=0x7f28a0033c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:54:30.718 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:30.716+0000 7f28a7e00640 1 -- 172.21.15.32:0/2100098826 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28a0005c90 con 0x7f28b006e930 2024-08-03T20:54:30.719 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:30.716+0000 7f28a7e00640 1 -- 172.21.15.32:0/2100098826 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f28a0031e50 con 0x7f28b006e930 2024-08-03T20:54:30.719 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:30.716+0000 7f28a7e00640 1 -- 172.21.15.32:0/2100098826 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28a0005260 con 0x7f28b006e930 2024-08-03T20:54:30.719 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:30.716+0000 7f28b6c00640 1 -- 172.21.15.32:0/2100098826 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f28b013c9c0 con 0x7f28b006e930 2024-08-03T20:54:30.719 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:30.716+0000 7f28b6c00640 1 -- 172.21.15.32:0/2100098826 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f28b013cf80 con 0x7f28b006e930 2024-08-03T20:54:30.719 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:30.717+0000 7f28b6c00640 1 -- 172.21.15.32:0/2100098826 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f28b00cd510 con 0x7f28b006e930 2024-08-03T20:54:30.720 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:30.719+0000 7f28a7e00640 1 -- 172.21.15.32:0/2100098826 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f28a0005400 con 0x7f28b006e930 2024-08-03T20:54:30.722 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:30.720+0000 7f28a7e00640 1 --2- 172.21.15.32:0/2100098826 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f2890077010 0x7f2890079520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:30.722 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:30.720+0000 7f28a7e00640 1 -- 172.21.15.32:0/2100098826 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 5709+0+0 (secure 0 0 0) 0x7f28a00b9990 con 0x7f28b006e930 2024-08-03T20:54:30.723 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:30.721+0000 7f28af400640 1 --2- 172.21.15.32:0/2100098826 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f2890077010 0x7f2890079520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:30.723 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:30.722+0000 7f28af400640 1 --2- 172.21.15.32:0/2100098826 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f2890077010 0x7f2890079520 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f28b0144080 tx=0x7f289c005f90 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:54:30.726 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:30.725+0000 7f28a7e00640 1 -- 172.21.15.32:0/2100098826 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7f28a003f030 con 0x7f28b006e930 2024-08-03T20:54:30.969 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:30.968+0000 7f28b6c00640 1 -- 172.21.15.32:0/2100098826 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f28b00b17b0 con 0x7f28b006e930 2024-08-03T20:54:30.970 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:30.969+0000 7f28a7e00640 1 -- 172.21.15.32:0/2100098826 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 91 v91) ==== 94+0+5228 (secure 0 0 0) 0x7f28a0082b80 con 0x7f28b006e930 2024-08-03T20:54:30.970 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-03T20:54:30.970 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":91,"btime":"2024-08-03T20:54:30:445075+0000","default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6439,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6835/531080160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6834","nonce":531080160},{"type":"v1","addr":"172.21.15.118:6835","nonce":531080160}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":84},{"gid":6454,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6837/2796882319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6836","nonce":2796882319},{"type":"v1","addr":"172.21.15.118:6837","nonce":2796882319}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":86},{"gid":6813,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6834/2686933890","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":2686933890},{"type":"v1","addr":"172.21.15.32:6834","nonce":2686933890}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":85}],"filesystems":[{"mdsmap":{"epoch":91,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-08-03T20:54:26.950518+0000","modified":"2024-08-03T20:54:29.976506+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6816},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6816":{"gid":6816,"name":"a","rank":0,"incarnation":87,"state":"up:active","state_seq":5,"addr":"172.21.15.32:6835/1707454554","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6833","nonce":1707454554},{"type":"v1","addr":"172.21.15.32:6835","nonce":1707454554}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6816,"qdb_cluster":[6816]},"id":6}]} 2024-08-03T20:54:30.971 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 91 2024-08-03T20:54:30.973 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:30.971+0000 7f28b6c00640 1 -- 172.21.15.32:0/2100098826 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f2890077010 msgr2=0x7f2890079520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:30.973 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:30.971+0000 7f28b6c00640 1 --2- 172.21.15.32:0/2100098826 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f2890077010 0x7f2890079520 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f28b0144080 tx=0x7f289c005f90 comp rx=0 tx=0).stop 2024-08-03T20:54:30.973 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:30.971+0000 7f28b6c00640 1 -- 172.21.15.32:0/2100098826 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f28b006e930 msgr2=0x7f28b015f360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:30.973 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:30.971+0000 7f28b6c00640 1 --2- 172.21.15.32:0/2100098826 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f28b006e930 0x7f28b015f360 secure :-1 s=READY pgs=907 cs=0 l=1 rev1=1 crypto rx=0x7f28a0009950 tx=0x7f28a0033c20 comp rx=0 tx=0).stop 2024-08-03T20:54:30.973 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:30.971+0000 7f28b6c00640 1 -- 172.21.15.32:0/2100098826 shutdown_connections 2024-08-03T20:54:30.973 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:30.971+0000 7f28b6c00640 1 --2- 172.21.15.32:0/2100098826 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f2890077010 0x7f2890079520 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:30.973 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:30.971+0000 7f28b6c00640 1 --2- 172.21.15.32:0/2100098826 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f28b0073e60 0x7f28b0141e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:30.973 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:30.971+0000 7f28b6c00640 1 --2- 172.21.15.32:0/2100098826 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f28b00747b0 0x7f28b01423d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:30.973 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:30.971+0000 7f28b6c00640 1 --2- 172.21.15.32:0/2100098826 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f28b006e930 0x7f28b015f360 unknown :-1 s=CLOSED pgs=907 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:30.974 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:30.971+0000 7f28b6c00640 1 -- 172.21.15.32:0/2100098826 >> 172.21.15.32:0/2100098826 conn(0x7f28b00b1bd0 msgr2=0x7f28b0140b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:54:30.974 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:30.971+0000 7f28b6c00640 1 -- 172.21.15.32:0/2100098826 shutdown_connections 2024-08-03T20:54:30.974 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:30.971+0000 7f28b6c00640 1 -- 172.21.15.32:0/2100098826 wait complete. 2024-08-03T20:54:30.995 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-08-03T20:54:31.223 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:31.221+0000 7f5d71a00640 1 -- 172.21.15.32:0/2201414202 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f5d6c06e930 msgr2=0x7f5d6c144f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:31.224 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:31.221+0000 7f5d71a00640 1 --2- 172.21.15.32:0/2201414202 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f5d6c06e930 0x7f5d6c144f90 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f5d5c009830 tx=0x7f5d5c02a540 comp rx=0 tx=0).stop 2024-08-03T20:54:31.224 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:31.222+0000 7f5d71a00640 1 -- 172.21.15.32:0/2201414202 shutdown_connections 2024-08-03T20:54:31.224 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:31.222+0000 7f5d71a00640 1 --2- 172.21.15.32:0/2201414202 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f5d6c06e930 0x7f5d6c144f90 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:31.224 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:31.222+0000 7f5d71a00640 1 -- 172.21.15.32:0/2201414202 >> 172.21.15.32:0/2201414202 conn(0x7f5d6c0b1bd0 msgr2=0x7f5d6c0b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:54:31.224 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:31.222+0000 7f5d71a00640 1 -- 172.21.15.32:0/2201414202 shutdown_connections 2024-08-03T20:54:31.224 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:31.222+0000 7f5d71a00640 1 -- 172.21.15.32:0/2201414202 wait complete. 2024-08-03T20:54:31.225 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:31.224+0000 7f5d71a00640 1 Processor -- start 2024-08-03T20:54:31.225 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:31.224+0000 7f5d71a00640 1 -- start start 2024-08-03T20:54:31.229 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:31.225+0000 7f5d71a00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f5d6c06e930 0x7f5d6c13daf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:31.229 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:31.225+0000 7f5d71a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5d6c073e60 0x7f5d6c1436b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:31.229 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:31.225+0000 7f5d71a00640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f5d6c15fff0 0x7f5d6c143bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:31.229 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:31.226+0000 7f5d71a00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f5d6c0cf840 con 0x7f5d6c073e60 2024-08-03T20:54:31.229 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:31.226+0000 7f5d71a00640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f5d6c145f30 con 0x7f5d6c06e930 2024-08-03T20:54:31.230 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:31.226+0000 7f5d71a00640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7f5d6c0cf9c0 con 0x7f5d6c15fff0 2024-08-03T20:54:31.230 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:31.226+0000 7f5d63e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5d6c073e60 0x7f5d6c1436b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:31.230 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:31.226+0000 7f5d63e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5d6c073e60 0x7f5d6c1436b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:39966/0 (socket says 172.21.15.32:39966) 2024-08-03T20:54:31.230 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:31.226+0000 7f5d63e00640 1 -- 172.21.15.32:0/1294614017 learned_addr learned my addr 172.21.15.32:0/1294614017 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:54:31.230 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:31.226+0000 7f5d6aa00640 1 --2- 172.21.15.32:0/1294614017 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f5d6c06e930 0x7f5d6c13daf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:31.230 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:31.226+0000 7f5d6b400640 1 --2- 172.21.15.32:0/1294614017 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f5d6c15fff0 0x7f5d6c143bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:31.231 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:31.226+0000 7f5d63e00640 1 -- 172.21.15.32:0/1294614017 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f5d6c15fff0 msgr2=0x7f5d6c143bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:31.231 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:31.226+0000 7f5d63e00640 1 --2- 172.21.15.32:0/1294614017 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f5d6c15fff0 0x7f5d6c143bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:31.231 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:31.226+0000 7f5d63e00640 1 -- 172.21.15.32:0/1294614017 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f5d6c06e930 msgr2=0x7f5d6c13daf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:31.231 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:31.226+0000 7f5d63e00640 1 --2- 172.21.15.32:0/1294614017 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f5d6c06e930 0x7f5d6c13daf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:31.231 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:31.226+0000 7f5d63e00640 1 -- 172.21.15.32:0/1294614017 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5d6c15e9e0 con 0x7f5d6c073e60 2024-08-03T20:54:31.231 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:31.227+0000 7f5d63e00640 1 --2- 172.21.15.32:0/1294614017 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5d6c073e60 0x7f5d6c1436b0 secure :-1 s=READY pgs=909 cs=0 l=1 rev1=1 crypto rx=0x7f5d5400c680 tx=0x7f5d5400cb40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:54:31.231 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:31.227+0000 7f5d62a00640 1 -- 172.21.15.32:0/1294614017 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d54016020 con 0x7f5d6c073e60 2024-08-03T20:54:31.231 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:31.227+0000 7f5d62a00640 1 -- 172.21.15.32:0/1294614017 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f5d5400a930 con 0x7f5d6c073e60 2024-08-03T20:54:31.231 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:31.227+0000 7f5d71a00640 1 -- 172.21.15.32:0/1294614017 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5d6c15ecc0 con 0x7f5d6c073e60 2024-08-03T20:54:31.232 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:31.227+0000 7f5d62a00640 1 -- 172.21.15.32:0/1294614017 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d5400ad60 con 0x7f5d6c073e60 2024-08-03T20:54:31.232 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:31.227+0000 7f5d71a00640 1 -- 172.21.15.32:0/1294614017 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5d6c0add40 con 0x7f5d6c073e60 2024-08-03T20:54:31.233 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:31.228+0000 7f5d71a00640 1 -- 172.21.15.32:0/1294614017 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5d6c15e750 con 0x7f5d6c073e60 2024-08-03T20:54:31.233 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:31.229+0000 7f5d62a00640 1 -- 172.21.15.32:0/1294614017 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f5d54019020 con 0x7f5d6c073e60 2024-08-03T20:54:31.233 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:31.231+0000 7f5d62a00640 1 --2- 172.21.15.32:0/1294614017 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f5d48077010 0x7f5d48079520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:31.233 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:31.231+0000 7f5d62a00640 1 -- 172.21.15.32:0/1294614017 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 5709+0+0 (secure 0 0 0) 0x7f5d54098a30 con 0x7f5d6c073e60 2024-08-03T20:54:31.233 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:31.231+0000 7f5d6aa00640 1 --2- 172.21.15.32:0/1294614017 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f5d48077010 0x7f5d48079520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:31.233 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:31.232+0000 7f5d6aa00640 1 --2- 172.21.15.32:0/1294614017 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f5d48077010 0x7f5d48079520 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f5d5c034000 tx=0x7f5d5c004730 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:54:31.237 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:31.236+0000 7f5d62a00640 1 -- 172.21.15.32:0/1294614017 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7f5d54020020 con 0x7f5d6c073e60 2024-08-03T20:54:31.441 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:31.439+0000 7f5d71a00640 1 -- 172.21.15.32:0/1294614017 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f5d6c160e20 con 0x7f5d6c073e60 2024-08-03T20:54:31.442 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:31.440+0000 7f5d62a00640 1 -- 172.21.15.32:0/1294614017 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v83) ==== 74+0+16600 (secure 0 0 0) 0x7f5d54061b90 con 0x7f5d6c073e60 2024-08-03T20:54:31.442 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-03T20:54:31.442 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":83,"fsid":"40ab141a-01d7-49b9-ae3c-2edec9876e95","created":"2024-08-03T20:47:14.957362+0000","modified":"2024-08-03T20:54:28.430560+0000","last_up_change":"2024-08-03T20:47:28.874451+0000","last_in_change":"2024-08-03T20:47:20.079442+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":13,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-08-03T20:47:28.261300+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_metadata","create_time":"2024-08-03T20:54:24.911555+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"82","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs_data","create_time":"2024-08-03T20:54:25.917207+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"83","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5a83e759-9035-438b-889a-0a6af1f0a0d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6816","nonce":3066156116},{"type":"v1","addr":"172.21.15.32:6817","nonce":3066156116}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6818","nonce":3066156116},{"type":"v1","addr":"172.21.15.32:6819","nonce":3066156116}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6822","nonce":3066156116},{"type":"v1","addr":"172.21.15.32:6823","nonce":3066156116}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6820","nonce":3066156116},{"type":"v1","addr":"172.21.15.32:6821","nonce":3066156116}]},"public_addr":"172.21.15.32:6817/3066156116","cluster_addr":"172.21.15.32:6819/3066156116","heartbeat_back_addr":"172.21.15.32:6823/3066156116","heartbeat_front_addr":"172.21.15.32:6821/3066156116","state":["exists","up"]},{"osd":1,"uuid":"d7c5c210-da4f-49f9-b9e7-1dfb02daa4bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6824","nonce":1551241759},{"type":"v1","addr":"172.21.15.32:6825","nonce":1551241759}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6826","nonce":1551241759},{"type":"v1","addr":"172.21.15.32:6827","nonce":1551241759}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6830","nonce":1551241759},{"type":"v1","addr":"172.21.15.32:6831","nonce":1551241759}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6828","nonce":1551241759},{"type":"v1","addr":"172.21.15.32:6829","nonce":1551241759}]},"public_addr":"172.21.15.32:6825/1551241759","cluster_addr":"172.21.15.32:6827/1551241759","heartbeat_back_addr":"172.21.15.32:6831/1551241759","heartbeat_front_addr":"172.21.15.32:6829/1551241759","state":["exists","up"]},{"osd":2,"uuid":"57f2e6de-a201-45c2-8981-8e56a4875ebd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6808","nonce":3806708944},{"type":"v1","addr":"172.21.15.32:6809","nonce":3806708944}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6810","nonce":3806708944},{"type":"v1","addr":"172.21.15.32:6811","nonce":3806708944}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6814","nonce":3806708944},{"type":"v1","addr":"172.21.15.32:6815","nonce":3806708944}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6812","nonce":3806708944},{"type":"v1","addr":"172.21.15.32:6813","nonce":3806708944}]},"public_addr":"172.21.15.32:6809/3806708944","cluster_addr":"172.21.15.32:6811/3806708944","heartbeat_back_addr":"172.21.15.32:6815/3806708944","heartbeat_front_addr":"172.21.15.32:6813/3806708944","state":["exists","up"]},{"osd":3,"uuid":"6439a280-da45-4148-8130-f16323935ff3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6800","nonce":9088303},{"type":"v1","addr":"172.21.15.32:6801","nonce":9088303}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6802","nonce":9088303},{"type":"v1","addr":"172.21.15.32:6803","nonce":9088303}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6806","nonce":9088303},{"type":"v1","addr":"172.21.15.32:6807","nonce":9088303}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6804","nonce":9088303},{"type":"v1","addr":"172.21.15.32:6805","nonce":9088303}]},"public_addr":"172.21.15.32:6801/9088303","cluster_addr":"172.21.15.32:6803/9088303","heartbeat_back_addr":"172.21.15.32:6807/9088303","heartbeat_front_addr":"172.21.15.32:6805/9088303","state":["exists","up"]},{"osd":4,"uuid":"4f902eff-1e18-418d-82c9-2151c69524fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6802","nonce":864752275},{"type":"v1","addr":"172.21.15.118:6803","nonce":864752275}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6804","nonce":864752275},{"type":"v1","addr":"172.21.15.118:6805","nonce":864752275}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6808","nonce":864752275},{"type":"v1","addr":"172.21.15.118:6809","nonce":864752275}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6806","nonce":864752275},{"type":"v1","addr":"172.21.15.118:6807","nonce":864752275}]},"public_addr":"172.21.15.118:6803/864752275","cluster_addr":"172.21.15.118:6805/864752275","heartbeat_back_addr":"172.21.15.118:6809/864752275","heartbeat_front_addr":"172.21.15.118:6807/864752275","state":["exists","up"]},{"osd":5,"uuid":"bfd9feac-dd06-458c-91b9-af7f175f612c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6810","nonce":2724891770},{"type":"v1","addr":"172.21.15.118:6811","nonce":2724891770}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6812","nonce":2724891770},{"type":"v1","addr":"172.21.15.118:6813","nonce":2724891770}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6816","nonce":2724891770},{"type":"v1","addr":"172.21.15.118:6817","nonce":2724891770}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6814","nonce":2724891770},{"type":"v1","addr":"172.21.15.118:6815","nonce":2724891770}]},"public_addr":"172.21.15.118:6811/2724891770","cluster_addr":"172.21.15.118:6813/2724891770","heartbeat_back_addr":"172.21.15.118:6817/2724891770","heartbeat_front_addr":"172.21.15.118:6815/2724891770","state":["exists","up"]},{"osd":6,"uuid":"5ed5a279-9d8e-416d-bbc7-7edd3e6d5705","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6825","nonce":801947386},{"type":"v1","addr":"172.21.15.118:6827","nonce":801947386}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6828","nonce":801947386},{"type":"v1","addr":"172.21.15.118:6829","nonce":801947386}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6832","nonce":801947386},{"type":"v1","addr":"172.21.15.118:6833","nonce":801947386}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6830","nonce":801947386},{"type":"v1","addr":"172.21.15.118:6831","nonce":801947386}]},"public_addr":"172.21.15.118:6827/801947386","cluster_addr":"172.21.15.118:6829/801947386","heartbeat_back_addr":"172.21.15.118:6833/801947386","heartbeat_front_addr":"172.21.15.118:6831/801947386","state":["exists","up"]},{"osd":7,"uuid":"f2086c47-875e-4451-a436-2008d1ea16af","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6818","nonce":653000048},{"type":"v1","addr":"172.21.15.118:6819","nonce":653000048}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6820","nonce":653000048},{"type":"v1","addr":"172.21.15.118:6821","nonce":653000048}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6824","nonce":653000048},{"type":"v1","addr":"172.21.15.118:6826","nonce":653000048}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6822","nonce":653000048},{"type":"v1","addr":"172.21.15.118:6823","nonce":653000048}]},"public_addr":"172.21.15.118:6819/653000048","cluster_addr":"172.21.15.118:6821/653000048","heartbeat_back_addr":"172.21.15.118:6826/653000048","heartbeat_front_addr":"172.21.15.118:6823/653000048","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-03T20:47:25.491994+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-03T20:47:25.587358+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-03T20:47:25.454175+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-03T20:47:25.377734+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-03T20:47:25.412351+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-03T20:47:25.499815+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-03T20:47:25.575044+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-03T20:47:25.552383+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-08-03T20:54:31.444 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:31.443+0000 7f5d71a00640 1 -- 172.21.15.32:0/1294614017 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f5d48077010 msgr2=0x7f5d48079520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:31.445 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:31.443+0000 7f5d71a00640 1 --2- 172.21.15.32:0/1294614017 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f5d48077010 0x7f5d48079520 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f5d5c034000 tx=0x7f5d5c004730 comp rx=0 tx=0).stop 2024-08-03T20:54:31.445 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:31.443+0000 7f5d71a00640 1 -- 172.21.15.32:0/1294614017 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5d6c073e60 msgr2=0x7f5d6c1436b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:31.445 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:31.443+0000 7f5d71a00640 1 --2- 172.21.15.32:0/1294614017 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5d6c073e60 0x7f5d6c1436b0 secure :-1 s=READY pgs=909 cs=0 l=1 rev1=1 crypto rx=0x7f5d5400c680 tx=0x7f5d5400cb40 comp rx=0 tx=0).stop 2024-08-03T20:54:31.445 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:31.443+0000 7f5d71a00640 1 -- 172.21.15.32:0/1294614017 shutdown_connections 2024-08-03T20:54:31.445 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:31.443+0000 7f5d71a00640 1 --2- 172.21.15.32:0/1294614017 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f5d48077010 0x7f5d48079520 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:31.445 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:31.443+0000 7f5d71a00640 1 --2- 172.21.15.32:0/1294614017 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f5d6c15fff0 0x7f5d6c143bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:31.445 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:31.443+0000 7f5d71a00640 1 --2- 172.21.15.32:0/1294614017 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5d6c073e60 0x7f5d6c1436b0 unknown :-1 s=CLOSED pgs=909 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:31.445 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:31.443+0000 7f5d71a00640 1 --2- 172.21.15.32:0/1294614017 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f5d6c06e930 0x7f5d6c13daf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:31.445 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:31.443+0000 7f5d71a00640 1 -- 172.21.15.32:0/1294614017 >> 172.21.15.32:0/1294614017 conn(0x7f5d6c0b1bd0 msgr2=0x7f5d6c0b14b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:54:31.445 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:31.443+0000 7f5d71a00640 1 -- 172.21.15.32:0/1294614017 shutdown_connections 2024-08-03T20:54:31.445 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:31.443+0000 7f5d71a00640 1 -- 172.21.15.32:0/1294614017 wait complete. 2024-08-03T20:54:31.466 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-08-03T20:54:31.466 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-08-03T20:54:31.696 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:31.694+0000 7f53c2600640 1 -- 172.21.15.32:0/2813179528 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f53bc06e930 msgr2=0x7f53bc144f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:31.696 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:31.694+0000 7f53c2600640 1 --2- 172.21.15.32:0/2813179528 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f53bc06e930 0x7f53bc144f90 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f53ac009830 tx=0x7f53ac02a4a0 comp rx=0 tx=0).stop 2024-08-03T20:54:31.696 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:31.694+0000 7f53c2600640 1 -- 172.21.15.32:0/2813179528 shutdown_connections 2024-08-03T20:54:31.696 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:31.694+0000 7f53c2600640 1 --2- 172.21.15.32:0/2813179528 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f53bc15fff0 0x7f53bc160400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:31.696 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:31.694+0000 7f53c2600640 1 --2- 172.21.15.32:0/2813179528 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f53bc06e930 0x7f53bc144f90 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:31.697 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:31.694+0000 7f53c2600640 1 -- 172.21.15.32:0/2813179528 >> 172.21.15.32:0/2813179528 conn(0x7f53bc0b1bd0 msgr2=0x7f53bc0b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:54:31.697 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:31.694+0000 7f53c2600640 1 -- 172.21.15.32:0/2813179528 shutdown_connections 2024-08-03T20:54:31.697 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:31.694+0000 7f53c2600640 1 -- 172.21.15.32:0/2813179528 wait complete. 2024-08-03T20:54:31.698 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:31.696+0000 7f53c2600640 1 Processor -- start 2024-08-03T20:54:31.698 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:31.696+0000 7f53c2600640 1 -- start start 2024-08-03T20:54:31.701 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:31.698+0000 7f53c2600640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f53bc06e930 0x7f53bc15e970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:31.701 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:31.698+0000 7f53c2600640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f53bc073e60 0x7f53bc142b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:31.701 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:31.698+0000 7f53c2600640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f53bc15fff0 0x7f53bc143060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:31.701 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:31.698+0000 7f53c2600640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f53bc0cf840 con 0x7f53bc06e930 2024-08-03T20:54:31.701 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:31.698+0000 7f53c2600640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f53bc145f30 con 0x7f53bc073e60 2024-08-03T20:54:31.701 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:31.698+0000 7f53c2600640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7f53bc0cf9c0 con 0x7f53bc15fff0 2024-08-03T20:54:31.701 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:31.698+0000 7f53bb400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f53bc06e930 0x7f53bc15e970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:31.702 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:31.698+0000 7f53bb400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f53bc06e930 0x7f53bc15e970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:39970/0 (socket says 172.21.15.32:39970) 2024-08-03T20:54:31.702 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:31.698+0000 7f53bb400640 1 -- 172.21.15.32:0/2915583245 learned_addr learned my addr 172.21.15.32:0/2915583245 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:54:31.702 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:31.698+0000 7f53bbe00640 1 --2- 172.21.15.32:0/2915583245 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f53bc15fff0 0x7f53bc143060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:31.702 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:31.698+0000 7f53b3e00640 1 --2- 172.21.15.32:0/2915583245 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f53bc073e60 0x7f53bc142b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:31.702 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:31.699+0000 7f53bb400640 1 -- 172.21.15.32:0/2915583245 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f53bc15fff0 msgr2=0x7f53bc143060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:31.702 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:31.699+0000 7f53bb400640 1 --2- 172.21.15.32:0/2915583245 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f53bc15fff0 0x7f53bc143060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:31.702 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:31.699+0000 7f53bb400640 1 -- 172.21.15.32:0/2915583245 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f53bc073e60 msgr2=0x7f53bc142b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:31.702 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:31.699+0000 7f53bb400640 1 --2- 172.21.15.32:0/2915583245 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f53bc073e60 0x7f53bc142b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:31.702 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:31.699+0000 7f53bb400640 1 -- 172.21.15.32:0/2915583245 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f53bc140280 con 0x7f53bc06e930 2024-08-03T20:54:31.702 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:31.699+0000 7f53bb400640 1 --2- 172.21.15.32:0/2915583245 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f53bc06e930 0x7f53bc15e970 secure :-1 s=READY pgs=911 cs=0 l=1 rev1=1 crypto rx=0x7f53ac009950 tx=0x7f53ac003c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:54:31.702 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:31.699+0000 7f53b3400640 1 -- 172.21.15.32:0/2915583245 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53ac005c90 con 0x7f53bc06e930 2024-08-03T20:54:31.702 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:31.699+0000 7f53b3400640 1 -- 172.21.15.32:0/2915583245 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f53ac005020 con 0x7f53bc06e930 2024-08-03T20:54:31.703 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:31.699+0000 7f53b3400640 1 -- 172.21.15.32:0/2915583245 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53ac005450 con 0x7f53bc06e930 2024-08-03T20:54:31.703 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:31.699+0000 7f53c2600640 1 -- 172.21.15.32:0/2915583245 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f53bc15de90 con 0x7f53bc06e930 2024-08-03T20:54:31.703 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:31.700+0000 7f53c2600640 1 -- 172.21.15.32:0/2915583245 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f53bc15e450 con 0x7f53bc06e930 2024-08-03T20:54:31.703 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:31.701+0000 7f53c2600640 1 -- 172.21.15.32:0/2915583245 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f53bc15dcc0 con 0x7f53bc06e930 2024-08-03T20:54:31.703 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:31.702+0000 7f53b3400640 1 -- 172.21.15.32:0/2915583245 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f53ac0317c0 con 0x7f53bc06e930 2024-08-03T20:54:31.704 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:31.703+0000 7f53b3400640 1 --2- 172.21.15.32:0/2915583245 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f5398077040 0x7f5398079550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:31.705 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:31.703+0000 7f53b3400640 1 -- 172.21.15.32:0/2915583245 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 5709+0+0 (secure 0 0 0) 0x7f53ac0b98c0 con 0x7f53bc06e930 2024-08-03T20:54:31.705 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:31.704+0000 7f53b3e00640 1 --2- 172.21.15.32:0/2915583245 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f5398077040 0x7f5398079550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:31.706 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:31.705+0000 7f53b3e00640 1 --2- 172.21.15.32:0/2915583245 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f5398077040 0x7f5398079550 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f53bc155030 tx=0x7f53a4001db0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:54:31.709 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:31.708+0000 7f53b3400640 1 -- 172.21.15.32:0/2915583245 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7f53ac03f030 con 0x7f53bc06e930 2024-08-03T20:54:31.909 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:31.908+0000 7f53c2600640 1 -- 172.21.15.32:0/2915583245 --> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f53bc13fdd0 con 0x7f5398077040 2024-08-03T20:54:31.915 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:31.914+0000 7f53b3400640 1 -- 172.21.15.32:0/2915583245 <== mgr.4106 v2:172.21.15.118:6800/4205950699 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+252067 (secure 0 0 0) 0x7f53bc13fdd0 con 0x7f5398077040 2024-08-03T20:54:31.920 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-03T20:54:31.922 INFO:teuthology.orchestra.run.smithi032.stderr:dumped all 2024-08-03T20:54:31.925 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:31.918+0000 7f53c2600640 1 -- 172.21.15.32:0/2915583245 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f5398077040 msgr2=0x7f5398079550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:31.925 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:31.918+0000 7f53c2600640 1 --2- 172.21.15.32:0/2915583245 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f5398077040 0x7f5398079550 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f53bc155030 tx=0x7f53a4001db0 comp rx=0 tx=0).stop 2024-08-03T20:54:31.925 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:31.918+0000 7f53c2600640 1 -- 172.21.15.32:0/2915583245 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f53bc06e930 msgr2=0x7f53bc15e970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:31.925 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:31.918+0000 7f53c2600640 1 --2- 172.21.15.32:0/2915583245 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f53bc06e930 0x7f53bc15e970 secure :-1 s=READY pgs=911 cs=0 l=1 rev1=1 crypto rx=0x7f53ac009950 tx=0x7f53ac003c10 comp rx=0 tx=0).stop 2024-08-03T20:54:31.925 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:31.918+0000 7f53c2600640 1 -- 172.21.15.32:0/2915583245 shutdown_connections 2024-08-03T20:54:31.926 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:31.918+0000 7f53c2600640 1 --2- 172.21.15.32:0/2915583245 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f5398077040 0x7f5398079550 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:31.926 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:31.918+0000 7f53c2600640 1 --2- 172.21.15.32:0/2915583245 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f53bc073e60 0x7f53bc142b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:31.926 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:31.918+0000 7f53c2600640 1 --2- 172.21.15.32:0/2915583245 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f53bc15fff0 0x7f53bc143060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:31.926 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:31.918+0000 7f53c2600640 1 --2- 172.21.15.32:0/2915583245 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f53bc06e930 0x7f53bc15e970 unknown :-1 s=CLOSED pgs=911 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:31.926 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:31.918+0000 7f53c2600640 1 -- 172.21.15.32:0/2915583245 >> 172.21.15.32:0/2915583245 conn(0x7f53bc0b1bd0 msgr2=0x7f53bc06f930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:54:31.927 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:31.918+0000 7f53c2600640 1 -- 172.21.15.32:0/2915583245 shutdown_connections 2024-08-03T20:54:31.927 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:31.918+0000 7f53c2600640 1 -- 172.21.15.32:0/2915583245 wait complete. 2024-08-03T20:54:31.943 INFO:teuthology.orchestra.run.smithi032.stdout:{"pg_ready":true,"pg_map":{"version":287,"stamp":"2024-08-03T20:54:30.341658+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":146,"num_write_kb":1642,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":111,"ondisk_log_size":111,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":122,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":271024,"kb_used_data":9056,"kb_used_omap":12,"kb_used_meta":261811,"kb_avail":754703696,"statfs":{"total":773094113280,"available":772816584704,"internally_reserved":0,"allocated":9273344,"data_stored":4605548,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":268095025},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":24,"apply_latency_ms":24,"commit_latency_ns":24000000,"apply_latency_ns":24000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.931044"},"pg_stats":[{"pgid":"13.3f","version":"0'0","reported_seq":13,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-08-03T20:54:28.440420+0000","last_change":"2024-08-03T20:54:27.441044+0000","last_active":"2024-08-03T20:54:28.440420+0000","last_peered":"2024-08-03T20:54:28.440420+0000","last_clean":"2024-08-03T20:54:28.440420+0000","last_became_active":"2024-08-03T20:54:27.440740+0000","last_became_peered":"2024-08-03T20:54:27.440740+0000","last_unstale":"2024-08-03T20:54:28.440420+0000","last_undegraded":"2024-08-03T20:54:28.440420+0000","last_fullsized":"2024-08-03T20:54:28.440420+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:54:26.410149+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:54:26.410149+0000","last_clean_scrub_stamp":"2024-08-03T20:54:26.410149+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T07:47:09.153912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.3e","version":"0'0","reported_seq":13,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-08-03T20:54:29.435334+0000","last_change":"2024-08-03T20:54:27.453508+0000","last_active":"2024-08-03T20:54:29.435334+0000","last_peered":"2024-08-03T20:54:29.435334+0000","last_clean":"2024-08-03T20:54:29.435334+0000","last_became_active":"2024-08-03T20:54:27.453015+0000","last_became_peered":"2024-08-03T20:54:27.453015+0000","last_unstale":"2024-08-03T20:54:29.435334+0000","last_undegraded":"2024-08-03T20:54:29.435334+0000","last_fullsized":"2024-08-03T20:54:29.435334+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:54:26.410149+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:54:26.410149+0000","last_clean_scrub_stamp":"2024-08-03T20:54:26.410149+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T01:06:28.734529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.3d","version":"0'0","reported_seq":13,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-08-03T20:54:29.482216+0000","last_change":"2024-08-03T20:54:27.445123+0000","last_active":"2024-08-03T20:54:29.482216+0000","last_peered":"2024-08-03T20:54:29.482216+0000","last_clean":"2024-08-03T20:54:29.482216+0000","last_became_active":"2024-08-03T20:54:27.444760+0000","last_became_peered":"2024-08-03T20:54:27.444760+0000","last_unstale":"2024-08-03T20:54:29.482216+0000","last_undegraded":"2024-08-03T20:54:29.482216+0000","last_fullsized":"2024-08-03T20:54:29.482216+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:54:26.410149+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:54:26.410149+0000","last_clean_scrub_stamp":"2024-08-03T20:54:26.410149+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-04T21:50:39.503703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.3c","version":"0'0","reported_seq":10,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-03T20:54:27.469370+0000","last_change":"2024-08-03T20:54:27.466461+0000","last_active":"2024-08-03T20:54:27.469370+0000","last_peered":"2024-08-03T20:54:27.469370+0000","last_clean":"2024-08-03T20:54:27.469370+0000","last_became_active":"2024-08-03T20:54:27.466141+0000","last_became_peered":"2024-08-03T20:54:27.466141+0000","last_unstale":"2024-08-03T20:54:27.469370+0000","last_undegraded":"2024-08-03T20:54:27.469370+0000","last_fullsized":"2024-08-03T20:54:27.469370+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:54:26.410149+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:54:26.410149+0000","last_clean_scrub_stamp":"2024-08-03T20:54:26.410149+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T00:31:51.313267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.3b","version":"0'0","reported_seq":13,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-08-03T20:54:28.441847+0000","last_change":"2024-08-03T20:54:27.448707+0000","last_active":"2024-08-03T20:54:28.441847+0000","last_peered":"2024-08-03T20:54:28.441847+0000","last_clean":"2024-08-03T20:54:28.441847+0000","last_became_active":"2024-08-03T20:54:27.444927+0000","last_became_peered":"2024-08-03T20:54:27.444927+0000","last_unstale":"2024-08-03T20:54:28.441847+0000","last_undegraded":"2024-08-03T20:54:28.441847+0000","last_fullsized":"2024-08-03T20:54:28.441847+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:54:26.410149+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:54:26.410149+0000","last_clean_scrub_stamp":"2024-08-03T20:54:26.410149+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-04T21:58:31.396236+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.3a","version":"0'0","reported_seq":13,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-08-03T20:54:29.323592+0000","last_change":"2024-08-03T20:54:27.451021+0000","last_active":"2024-08-03T20:54:29.323592+0000","last_peered":"2024-08-03T20:54:29.323592+0000","last_clean":"2024-08-03T20:54:29.323592+0000","last_became_active":"2024-08-03T20:54:27.450613+0000","last_became_peered":"2024-08-03T20:54:27.450613+0000","last_unstale":"2024-08-03T20:54:29.323592+0000","last_undegraded":"2024-08-03T20:54:29.323592+0000","last_fullsized":"2024-08-03T20:54:29.323592+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:54:26.410149+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:54:26.410149+0000","last_clean_scrub_stamp":"2024-08-03T20:54:26.410149+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-04T21:11:37.122543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.39","version":"0'0","reported_seq":10,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-03T20:54:27.464346+0000","last_change":"2024-08-03T20:54:27.453127+0000","last_active":"2024-08-03T20:54:27.464346+0000","last_peered":"2024-08-03T20:54:27.464346+0000","last_clean":"2024-08-03T20:54:27.464346+0000","last_became_active":"2024-08-03T20:54:27.450479+0000","last_became_peered":"2024-08-03T20:54:27.450479+0000","last_unstale":"2024-08-03T20:54:27.464346+0000","last_undegraded":"2024-08-03T20:54:27.464346+0000","last_fullsized":"2024-08-03T20:54:27.464346+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:54:26.410149+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:54:26.410149+0000","last_clean_scrub_stamp":"2024-08-03T20:54:26.410149+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T07:39:14.606174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.38","version":"0'0","reported_seq":13,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-08-03T20:54:29.323693+0000","last_change":"2024-08-03T20:54:27.464523+0000","last_active":"2024-08-03T20:54:29.323693+0000","last_peered":"2024-08-03T20:54:29.323693+0000","last_clean":"2024-08-03T20:54:29.323693+0000","last_became_active":"2024-08-03T20:54:27.464250+0000","last_became_peered":"2024-08-03T20:54:27.464250+0000","last_unstale":"2024-08-03T20:54:29.323693+0000","last_undegraded":"2024-08-03T20:54:29.323693+0000","last_fullsized":"2024-08-03T20:54:29.323693+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:54:26.410149+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:54:26.410149+0000","last_clean_scrub_stamp":"2024-08-03T20:54:26.410149+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-04T23:30:55.227524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.37","version":"0'0","reported_seq":13,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-08-03T20:54:29.323867+0000","last_change":"2024-08-03T20:54:27.448082+0000","last_active":"2024-08-03T20:54:29.323867+0000","last_peered":"2024-08-03T20:54:29.323867+0000","last_clean":"2024-08-03T20:54:29.323867+0000","last_became_active":"2024-08-03T20:54:27.447546+0000","last_became_peered":"2024-08-03T20:54:27.447546+0000","last_unstale":"2024-08-03T20:54:29.323867+0000","last_undegraded":"2024-08-03T20:54:29.323867+0000","last_fullsized":"2024-08-03T20:54:29.323867+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:54:26.410149+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:54:26.410149+0000","last_clean_scrub_stamp":"2024-08-03T20:54:26.410149+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T07:47:09.153912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.36","version":"0'0","reported_seq":13,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-08-03T20:54:28.440301+0000","last_change":"2024-08-03T20:54:27.445637+0000","last_active":"2024-08-03T20:54:28.440301+0000","last_peered":"2024-08-03T20:54:28.440301+0000","last_clean":"2024-08-03T20:54:28.440301+0000","last_became_active":"2024-08-03T20:54:27.445278+0000","last_became_peered":"2024-08-03T20:54:27.445278+0000","last_unstale":"2024-08-03T20:54:28.440301+0000","last_undegraded":"2024-08-03T20:54:28.440301+0000","last_fullsized":"2024-08-03T20:54:28.440301+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:54:26.410149+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:54:26.410149+0000","last_clean_scrub_stamp":"2024-08-03T20:54:26.410149+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T05:28:20.584840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.35","version":"0'0","reported_seq":13,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-08-03T20:54:29.435217+0000","last_change":"2024-08-03T20:54:27.454353+0000","last_active":"2024-08-03T20:54:29.435217+0000","last_peered":"2024-08-03T20:54:29.435217+0000","last_clean":"2024-08-03T20:54:29.435217+0000","last_became_active":"2024-08-03T20:54:27.454049+0000","last_became_peered":"2024-08-03T20:54:27.454049+0000","last_unstale":"2024-08-03T20:54:29.435217+0000","last_undegraded":"2024-08-03T20:54:29.435217+0000","last_fullsized":"2024-08-03T20:54:29.435217+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:54:26.410149+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:54:26.410149+0000","last_clean_scrub_stamp":"2024-08-03T20:54:26.410149+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-04T22:37:20.127782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.34","version":"0'0","reported_seq":13,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-08-03T20:54:28.440323+0000","last_change":"2024-08-03T20:54:27.444784+0000","last_active":"2024-08-03T20:54:28.440323+0000","last_peered":"2024-08-03T20:54:28.440323+0000","last_clean":"2024-08-03T20:54:28.440323+0000","last_became_active":"2024-08-03T20:54:27.444485+0000","last_became_peered":"2024-08-03T20:54:27.444485+0000","last_unstale":"2024-08-03T20:54:28.440323+0000","last_undegraded":"2024-08-03T20:54:28.440323+0000","last_fullsized":"2024-08-03T20:54:28.440323+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:54:26.410149+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:54:26.410149+0000","last_clean_scrub_stamp":"2024-08-03T20:54:26.410149+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T03:14:09.919834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.33","version":"0'0","reported_seq":13,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-08-03T20:54:29.482081+0000","last_change":"2024-08-03T20:54:27.439898+0000","last_active":"2024-08-03T20:54:29.482081+0000","last_peered":"2024-08-03T20:54:29.482081+0000","last_clean":"2024-08-03T20:54:29.482081+0000","last_became_active":"2024-08-03T20:54:27.439239+0000","last_became_peered":"2024-08-03T20:54:27.439239+0000","last_unstale":"2024-08-03T20:54:29.482081+0000","last_undegraded":"2024-08-03T20:54:29.482081+0000","last_fullsized":"2024-08-03T20:54:29.482081+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:54:26.410149+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:54:26.410149+0000","last_clean_scrub_stamp":"2024-08-03T20:54:26.410149+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T07:09:56.880905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.32","version":"0'0","reported_seq":10,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-03T20:54:27.465996+0000","last_change":"2024-08-03T20:54:27.460892+0000","last_active":"2024-08-03T20:54:27.465996+0000","last_peered":"2024-08-03T20:54:27.465996+0000","last_clean":"2024-08-03T20:54:27.465996+0000","last_became_active":"2024-08-03T20:54:27.460176+0000","last_became_peered":"2024-08-03T20:54:27.460176+0000","last_unstale":"2024-08-03T20:54:27.465996+0000","last_undegraded":"2024-08-03T20:54:27.465996+0000","last_fullsized":"2024-08-03T20:54:27.465996+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:54:26.410149+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:54:26.410149+0000","last_clean_scrub_stamp":"2024-08-03T20:54:26.410149+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T04:49:41.930226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.31","version":"0'0","reported_seq":10,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-03T20:54:27.446546+0000","last_change":"2024-08-03T20:54:27.444614+0000","last_active":"2024-08-03T20:54:27.446546+0000","last_peered":"2024-08-03T20:54:27.446546+0000","last_clean":"2024-08-03T20:54:27.446546+0000","last_became_active":"2024-08-03T20:54:27.443574+0000","last_became_peered":"2024-08-03T20:54:27.443574+0000","last_unstale":"2024-08-03T20:54:27.446546+0000","last_undegraded":"2024-08-03T20:54:27.446546+0000","last_fullsized":"2024-08-03T20:54:27.446546+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:54:26.410149+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:54:26.410149+0000","last_clean_scrub_stamp":"2024-08-03T20:54:26.410149+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T03:52:35.096710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.30","version":"0'0","reported_seq":13,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-08-03T20:54:28.441211+0000","last_change":"2024-08-03T20:54:27.435703+0000","last_active":"2024-08-03T20:54:28.441211+0000","last_peered":"2024-08-03T20:54:28.441211+0000","last_clean":"2024-08-03T20:54:28.441211+0000","last_became_active":"2024-08-03T20:54:27.434882+0000","last_became_peered":"2024-08-03T20:54:27.434882+0000","last_unstale":"2024-08-03T20:54:28.441211+0000","last_undegraded":"2024-08-03T20:54:28.441211+0000","last_fullsized":"2024-08-03T20:54:28.441211+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:54:26.410149+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:54:26.410149+0000","last_clean_scrub_stamp":"2024-08-03T20:54:26.410149+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T05:44:25.324689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.2f","version":"0'0","reported_seq":13,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-08-03T20:54:28.442148+0000","last_change":"2024-08-03T20:54:27.452617+0000","last_active":"2024-08-03T20:54:28.442148+0000","last_peered":"2024-08-03T20:54:28.442148+0000","last_clean":"2024-08-03T20:54:28.442148+0000","last_became_active":"2024-08-03T20:54:27.452278+0000","last_became_peered":"2024-08-03T20:54:27.452278+0000","last_unstale":"2024-08-03T20:54:28.442148+0000","last_undegraded":"2024-08-03T20:54:28.442148+0000","last_fullsized":"2024-08-03T20:54:28.442148+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:54:26.410149+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:54:26.410149+0000","last_clean_scrub_stamp":"2024-08-03T20:54:26.410149+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T08:45:22.391494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.2e","version":"0'0","reported_seq":10,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-03T20:54:27.466462+0000","last_change":"2024-08-03T20:54:27.462672+0000","last_active":"2024-08-03T20:54:27.466462+0000","last_peered":"2024-08-03T20:54:27.466462+0000","last_clean":"2024-08-03T20:54:27.466462+0000","last_became_active":"2024-08-03T20:54:27.462357+0000","last_became_peered":"2024-08-03T20:54:27.462357+0000","last_unstale":"2024-08-03T20:54:27.466462+0000","last_undegraded":"2024-08-03T20:54:27.466462+0000","last_fullsized":"2024-08-03T20:54:27.466462+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:54:26.410149+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:54:26.410149+0000","last_clean_scrub_stamp":"2024-08-03T20:54:26.410149+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T05:48:22.964700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.2d","version":"0'0","reported_seq":13,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-08-03T20:54:29.438574+0000","last_change":"2024-08-03T20:54:27.439122+0000","last_active":"2024-08-03T20:54:29.438574+0000","last_peered":"2024-08-03T20:54:29.438574+0000","last_clean":"2024-08-03T20:54:29.438574+0000","last_became_active":"2024-08-03T20:54:27.438765+0000","last_became_peered":"2024-08-03T20:54:27.438765+0000","last_unstale":"2024-08-03T20:54:29.438574+0000","last_undegraded":"2024-08-03T20:54:29.438574+0000","last_fullsized":"2024-08-03T20:54:29.438574+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:54:26.410149+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:54:26.410149+0000","last_clean_scrub_stamp":"2024-08-03T20:54:26.410149+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T07:47:09.153912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.2c","version":"0'0","reported_seq":10,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-03T20:54:27.469346+0000","last_change":"2024-08-03T20:54:27.463774+0000","last_active":"2024-08-03T20:54:27.469346+0000","last_peered":"2024-08-03T20:54:27.469346+0000","last_clean":"2024-08-03T20:54:27.469346+0000","last_became_active":"2024-08-03T20:54:27.458783+0000","last_became_peered":"2024-08-03T20:54:27.458783+0000","last_unstale":"2024-08-03T20:54:27.469346+0000","last_undegraded":"2024-08-03T20:54:27.469346+0000","last_fullsized":"2024-08-03T20:54:27.469346+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:54:26.410149+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:54:26.410149+0000","last_clean_scrub_stamp":"2024-08-03T20:54:26.410149+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T07:00:07.332926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.2b","version":"0'0","reported_seq":10,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-03T20:54:27.468079+0000","last_change":"2024-08-03T20:54:27.464347+0000","last_active":"2024-08-03T20:54:27.468079+0000","last_peered":"2024-08-03T20:54:27.468079+0000","last_clean":"2024-08-03T20:54:27.468079+0000","last_became_active":"2024-08-03T20:54:27.464032+0000","last_became_peered":"2024-08-03T20:54:27.464032+0000","last_unstale":"2024-08-03T20:54:27.468079+0000","last_undegraded":"2024-08-03T20:54:27.468079+0000","last_fullsized":"2024-08-03T20:54:27.468079+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:54:26.410149+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:54:26.410149+0000","last_clean_scrub_stamp":"2024-08-03T20:54:26.410149+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T04:36:09.850241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.2a","version":"0'0","reported_seq":13,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-08-03T20:54:29.434917+0000","last_change":"2024-08-03T20:54:27.455588+0000","last_active":"2024-08-03T20:54:29.434917+0000","last_peered":"2024-08-03T20:54:29.434917+0000","last_clean":"2024-08-03T20:54:29.434917+0000","last_became_active":"2024-08-03T20:54:27.455262+0000","last_became_peered":"2024-08-03T20:54:27.455262+0000","last_unstale":"2024-08-03T20:54:29.434917+0000","last_undegraded":"2024-08-03T20:54:29.434917+0000","last_fullsized":"2024-08-03T20:54:29.434917+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:54:26.410149+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:54:26.410149+0000","last_clean_scrub_stamp":"2024-08-03T20:54:26.410149+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-04T21:51:05.362465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.29","version":"0'0","reported_seq":13,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-08-03T20:54:28.439669+0000","last_change":"2024-08-03T20:54:27.454311+0000","last_active":"2024-08-03T20:54:28.439669+0000","last_peered":"2024-08-03T20:54:28.439669+0000","last_clean":"2024-08-03T20:54:28.439669+0000","last_became_active":"2024-08-03T20:54:27.453739+0000","last_became_peered":"2024-08-03T20:54:27.453739+0000","last_unstale":"2024-08-03T20:54:28.439669+0000","last_undegraded":"2024-08-03T20:54:28.439669+0000","last_fullsized":"2024-08-03T20:54:28.439669+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:54:26.410149+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:54:26.410149+0000","last_clean_scrub_stamp":"2024-08-03T20:54:26.410149+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-04T21:11:37.122543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.28","version":"0'0","reported_seq":10,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-03T20:54:27.468314+0000","last_change":"2024-08-03T20:54:27.455361+0000","last_active":"2024-08-03T20:54:27.468314+0000","last_peered":"2024-08-03T20:54:27.468314+0000","last_clean":"2024-08-03T20:54:27.468314+0000","last_became_active":"2024-08-03T20:54:27.455033+0000","last_became_peered":"2024-08-03T20:54:27.455033+0000","last_unstale":"2024-08-03T20:54:27.468314+0000","last_undegraded":"2024-08-03T20:54:27.468314+0000","last_fullsized":"2024-08-03T20:54:27.468314+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:54:26.410149+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:54:26.410149+0000","last_clean_scrub_stamp":"2024-08-03T20:54:26.410149+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T08:07:54.404129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.27","version":"0'0","reported_seq":13,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-08-03T20:54:29.435298+0000","last_change":"2024-08-03T20:54:27.455147+0000","last_active":"2024-08-03T20:54:29.435298+0000","last_peered":"2024-08-03T20:54:29.435298+0000","last_clean":"2024-08-03T20:54:29.435298+0000","last_became_active":"2024-08-03T20:54:27.448048+0000","last_became_peered":"2024-08-03T20:54:27.448048+0000","last_unstale":"2024-08-03T20:54:29.435298+0000","last_undegraded":"2024-08-03T20:54:29.435298+0000","last_fullsized":"2024-08-03T20:54:29.435298+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:54:26.410149+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:54:26.410149+0000","last_clean_scrub_stamp":"2024-08-03T20:54:26.410149+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T08:02:43.988492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.26","version":"0'0","reported_seq":13,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-08-03T20:54:29.482089+0000","last_change":"2024-08-03T20:54:27.445865+0000","last_active":"2024-08-03T20:54:29.482089+0000","last_peered":"2024-08-03T20:54:29.482089+0000","last_clean":"2024-08-03T20:54:29.482089+0000","last_became_active":"2024-08-03T20:54:27.445574+0000","last_became_peered":"2024-08-03T20:54:27.445574+0000","last_unstale":"2024-08-03T20:54:29.482089+0000","last_undegraded":"2024-08-03T20:54:29.482089+0000","last_fullsized":"2024-08-03T20:54:29.482089+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:54:26.410149+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:54:26.410149+0000","last_clean_scrub_stamp":"2024-08-03T20:54:26.410149+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T01:43:40.620191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.25","version":"0'0","reported_seq":13,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-08-03T20:54:28.441322+0000","last_change":"2024-08-03T20:54:27.436769+0000","last_active":"2024-08-03T20:54:28.441322+0000","last_peered":"2024-08-03T20:54:28.441322+0000","last_clean":"2024-08-03T20:54:28.441322+0000","last_became_active":"2024-08-03T20:54:27.436388+0000","last_became_peered":"2024-08-03T20:54:27.436388+0000","last_unstale":"2024-08-03T20:54:28.441322+0000","last_undegraded":"2024-08-03T20:54:28.441322+0000","last_fullsized":"2024-08-03T20:54:28.441322+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:54:26.410149+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:54:26.410149+0000","last_clean_scrub_stamp":"2024-08-03T20:54:26.410149+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T07:33:46.448288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.24","version":"0'0","reported_seq":13,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-08-03T20:54:28.440017+0000","last_change":"2024-08-03T20:54:27.442047+0000","last_active":"2024-08-03T20:54:28.440017+0000","last_peered":"2024-08-03T20:54:28.440017+0000","last_clean":"2024-08-03T20:54:28.440017+0000","last_became_active":"2024-08-03T20:54:27.441434+0000","last_became_peered":"2024-08-03T20:54:27.441434+0000","last_unstale":"2024-08-03T20:54:28.440017+0000","last_undegraded":"2024-08-03T20:54:28.440017+0000","last_fullsized":"2024-08-03T20:54:28.440017+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:54:26.410149+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:54:26.410149+0000","last_clean_scrub_stamp":"2024-08-03T20:54:26.410149+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T06:59:28.260517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.23","version":"0'0","reported_seq":13,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-08-03T20:54:29.434977+0000","last_change":"2024-08-03T20:54:27.453761+0000","last_active":"2024-08-03T20:54:29.434977+0000","last_peered":"2024-08-03T20:54:29.434977+0000","last_clean":"2024-08-03T20:54:29.434977+0000","last_became_active":"2024-08-03T20:54:27.453459+0000","last_became_peered":"2024-08-03T20:54:27.453459+0000","last_unstale":"2024-08-03T20:54:29.434977+0000","last_undegraded":"2024-08-03T20:54:29.434977+0000","last_fullsized":"2024-08-03T20:54:29.434977+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:54:26.410149+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:54:26.410149+0000","last_clean_scrub_stamp":"2024-08-03T20:54:26.410149+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T03:28:35.668870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.22","version":"0'0","reported_seq":13,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-08-03T20:54:29.435035+0000","last_change":"2024-08-03T20:54:27.457503+0000","last_active":"2024-08-03T20:54:29.435035+0000","last_peered":"2024-08-03T20:54:29.435035+0000","last_clean":"2024-08-03T20:54:29.435035+0000","last_became_active":"2024-08-03T20:54:27.456783+0000","last_became_peered":"2024-08-03T20:54:27.456783+0000","last_unstale":"2024-08-03T20:54:29.435035+0000","last_undegraded":"2024-08-03T20:54:29.435035+0000","last_fullsized":"2024-08-03T20:54:29.435035+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:54:26.410149+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:54:26.410149+0000","last_clean_scrub_stamp":"2024-08-03T20:54:26.410149+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-04T23:52:57.831797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.21","version":"0'0","reported_seq":13,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-08-03T20:54:28.441236+0000","last_change":"2024-08-03T20:54:27.448194+0000","last_active":"2024-08-03T20:54:28.441236+0000","last_peered":"2024-08-03T20:54:28.441236+0000","last_clean":"2024-08-03T20:54:28.441236+0000","last_became_active":"2024-08-03T20:54:27.447664+0000","last_became_peered":"2024-08-03T20:54:27.447664+0000","last_unstale":"2024-08-03T20:54:28.441236+0000","last_undegraded":"2024-08-03T20:54:28.441236+0000","last_fullsized":"2024-08-03T20:54:28.441236+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:54:26.410149+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:54:26.410149+0000","last_clean_scrub_stamp":"2024-08-03T20:54:26.410149+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T00:18:08.383331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.20","version":"0'0","reported_seq":13,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-08-03T20:54:28.439602+0000","last_change":"2024-08-03T20:54:27.466029+0000","last_active":"2024-08-03T20:54:28.439602+0000","last_peered":"2024-08-03T20:54:28.439602+0000","last_clean":"2024-08-03T20:54:28.439602+0000","last_became_active":"2024-08-03T20:54:27.465689+0000","last_became_peered":"2024-08-03T20:54:27.465689+0000","last_unstale":"2024-08-03T20:54:28.439602+0000","last_undegraded":"2024-08-03T20:54:28.439602+0000","last_fullsized":"2024-08-03T20:54:28.439602+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:54:26.410149+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:54:26.410149+0000","last_clean_scrub_stamp":"2024-08-03T20:54:26.410149+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-04T21:41:30.478055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1e","version":"0'0","reported_seq":13,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-08-03T20:54:28.441943+0000","last_change":"2024-08-03T20:54:27.446192+0000","last_active":"2024-08-03T20:54:28.441943+0000","last_peered":"2024-08-03T20:54:28.441943+0000","last_clean":"2024-08-03T20:54:28.441943+0000","last_became_active":"2024-08-03T20:54:27.445882+0000","last_became_peered":"2024-08-03T20:54:27.445882+0000","last_unstale":"2024-08-03T20:54:28.441943+0000","last_undegraded":"2024-08-03T20:54:28.441943+0000","last_fullsized":"2024-08-03T20:54:28.441943+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:54:26.410149+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:54:26.410149+0000","last_clean_scrub_stamp":"2024-08-03T20:54:26.410149+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T05:39:36.328590+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.1f","version":"82'4","reported_seq":19,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-08-03T20:54:28.440761+0000","last_change":"2024-08-03T20:54:26.444466+0000","last_active":"2024-08-03T20:54:28.440761+0000","last_peered":"2024-08-03T20:54:28.440761+0000","last_clean":"2024-08-03T20:54:28.440761+0000","last_became_active":"2024-08-03T20:54:26.444145+0000","last_became_peered":"2024-08-03T20:54:26.444145+0000","last_unstale":"2024-08-03T20:54:28.440761+0000","last_undegraded":"2024-08-03T20:54:28.440761+0000","last_fullsized":"2024-08-03T20:54:28.440761+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:54:25.402282+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:54:25.402282+0000","last_clean_scrub_stamp":"2024-08-03T20:54:25.402282+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-04T22:37:19.119915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1f","version":"0'0","reported_seq":13,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-08-03T20:54:28.440635+0000","last_change":"2024-08-03T20:54:27.455848+0000","last_active":"2024-08-03T20:54:28.440635+0000","last_peered":"2024-08-03T20:54:28.440635+0000","last_clean":"2024-08-03T20:54:28.440635+0000","last_became_active":"2024-08-03T20:54:27.455487+0000","last_became_peered":"2024-08-03T20:54:27.455487+0000","last_unstale":"2024-08-03T20:54:28.440635+0000","last_undegraded":"2024-08-03T20:54:28.440635+0000","last_fullsized":"2024-08-03T20:54:28.440635+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:54:26.410149+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:54:26.410149+0000","last_clean_scrub_stamp":"2024-08-03T20:54:26.410149+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T01:43:40.620191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.1e","version":"82'1","reported_seq":16,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-08-03T20:54:28.441990+0000","last_change":"2024-08-03T20:54:26.439724+0000","last_active":"2024-08-03T20:54:28.441990+0000","last_peered":"2024-08-03T20:54:28.441990+0000","last_clean":"2024-08-03T20:54:28.441990+0000","last_became_active":"2024-08-03T20:54:26.438773+0000","last_became_peered":"2024-08-03T20:54:26.438773+0000","last_unstale":"2024-08-03T20:54:28.441990+0000","last_undegraded":"2024-08-03T20:54:28.441990+0000","last_fullsized":"2024-08-03T20:54:28.441990+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:54:25.402282+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:54:25.402282+0000","last_clean_scrub_stamp":"2024-08-03T20:54:25.402282+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T08:35:58.055884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.1c","version":"0'0","reported_seq":13,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-08-03T20:54:28.440241+0000","last_change":"2024-08-03T20:54:27.444247+0000","last_active":"2024-08-03T20:54:28.440241+0000","last_peered":"2024-08-03T20:54:28.440241+0000","last_clean":"2024-08-03T20:54:28.440241+0000","last_became_active":"2024-08-03T20:54:27.443893+0000","last_became_peered":"2024-08-03T20:54:27.443893+0000","last_unstale":"2024-08-03T20:54:28.440241+0000","last_undegraded":"2024-08-03T20:54:28.440241+0000","last_fullsized":"2024-08-03T20:54:28.440241+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:54:26.410149+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:54:26.410149+0000","last_clean_scrub_stamp":"2024-08-03T20:54:26.410149+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-04T23:30:55.227524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.1d","version":"82'2","reported_seq":17,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-08-03T20:54:29.438574+0000","last_change":"2024-08-03T20:54:26.444870+0000","last_active":"2024-08-03T20:54:29.438574+0000","last_peered":"2024-08-03T20:54:29.438574+0000","last_clean":"2024-08-03T20:54:29.438574+0000","last_became_active":"2024-08-03T20:54:26.442179+0000","last_became_peered":"2024-08-03T20:54:26.442179+0000","last_unstale":"2024-08-03T20:54:29.438574+0000","last_undegraded":"2024-08-03T20:54:29.438574+0000","last_fullsized":"2024-08-03T20:54:29.438574+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:54:25.402282+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:54:25.402282+0000","last_clean_scrub_stamp":"2024-08-03T20:54:25.402282+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T04:40:32.715596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1d","version":"0'0","reported_seq":13,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-08-03T20:54:29.482438+0000","last_change":"2024-08-03T20:54:27.461330+0000","last_active":"2024-08-03T20:54:29.482438+0000","last_peered":"2024-08-03T20:54:29.482438+0000","last_clean":"2024-08-03T20:54:29.482438+0000","last_became_active":"2024-08-03T20:54:27.461060+0000","last_became_peered":"2024-08-03T20:54:27.461060+0000","last_unstale":"2024-08-03T20:54:29.482438+0000","last_undegraded":"2024-08-03T20:54:29.482438+0000","last_fullsized":"2024-08-03T20:54:29.482438+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:54:26.410149+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:54:26.410149+0000","last_clean_scrub_stamp":"2024-08-03T20:54:26.410149+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T05:41:54.880022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.1c","version":"82'2","reported_seq":17,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-08-03T20:54:29.438463+0000","last_change":"2024-08-03T20:54:26.445095+0000","last_active":"2024-08-03T20:54:29.438463+0000","last_peered":"2024-08-03T20:54:29.438463+0000","last_clean":"2024-08-03T20:54:29.438463+0000","last_became_active":"2024-08-03T20:54:26.441622+0000","last_became_peered":"2024-08-03T20:54:26.441622+0000","last_unstale":"2024-08-03T20:54:29.438463+0000","last_undegraded":"2024-08-03T20:54:29.438463+0000","last_fullsized":"2024-08-03T20:54:29.438463+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:54:25.402282+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:54:25.402282+0000","last_clean_scrub_stamp":"2024-08-03T20:54:25.402282+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T01:00:43.079889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1a","version":"0'0","reported_seq":13,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-08-03T20:54:29.435327+0000","last_change":"2024-08-03T20:54:27.457761+0000","last_active":"2024-08-03T20:54:29.435327+0000","last_peered":"2024-08-03T20:54:29.435327+0000","last_clean":"2024-08-03T20:54:29.435327+0000","last_became_active":"2024-08-03T20:54:27.457196+0000","last_became_peered":"2024-08-03T20:54:27.457196+0000","last_unstale":"2024-08-03T20:54:29.435327+0000","last_undegraded":"2024-08-03T20:54:29.435327+0000","last_fullsized":"2024-08-03T20:54:29.435327+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:54:26.410149+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:54:26.410149+0000","last_clean_scrub_stamp":"2024-08-03T20:54:26.410149+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-04T21:02:35.798301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.1b","version":"82'1","reported_seq":16,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-08-03T20:54:28.442165+0000","last_change":"2024-08-03T20:54:26.440177+0000","last_active":"2024-08-03T20:54:28.442165+0000","last_peered":"2024-08-03T20:54:28.442165+0000","last_clean":"2024-08-03T20:54:28.442165+0000","last_became_active":"2024-08-03T20:54:26.439861+0000","last_became_peered":"2024-08-03T20:54:26.439861+0000","last_unstale":"2024-08-03T20:54:28.442165+0000","last_undegraded":"2024-08-03T20:54:28.442165+0000","last_fullsized":"2024-08-03T20:54:28.442165+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:54:25.402282+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:54:25.402282+0000","last_clean_scrub_stamp":"2024-08-03T20:54:25.402282+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-04T23:38:59.029460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.1b","version":"0'0","reported_seq":13,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-08-03T20:54:28.441992+0000","last_change":"2024-08-03T20:54:27.457638+0000","last_active":"2024-08-03T20:54:28.441992+0000","last_peered":"2024-08-03T20:54:28.441992+0000","last_clean":"2024-08-03T20:54:28.441992+0000","last_became_active":"2024-08-03T20:54:27.457210+0000","last_became_peered":"2024-08-03T20:54:27.457210+0000","last_unstale":"2024-08-03T20:54:28.441992+0000","last_undegraded":"2024-08-03T20:54:28.441992+0000","last_fullsized":"2024-08-03T20:54:28.441992+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:54:26.410149+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:54:26.410149+0000","last_clean_scrub_stamp":"2024-08-03T20:54:26.410149+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T01:19:10.278598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.1a","version":"82'1","reported_seq":16,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-08-03T20:54:28.441433+0000","last_change":"2024-08-03T20:54:26.430929+0000","last_active":"2024-08-03T20:54:28.441433+0000","last_peered":"2024-08-03T20:54:28.441433+0000","last_clean":"2024-08-03T20:54:28.441433+0000","last_became_active":"2024-08-03T20:54:26.430532+0000","last_became_peered":"2024-08-03T20:54:26.430532+0000","last_unstale":"2024-08-03T20:54:28.441433+0000","last_undegraded":"2024-08-03T20:54:28.441433+0000","last_fullsized":"2024-08-03T20:54:28.441433+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:54:25.402282+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:54:25.402282+0000","last_clean_scrub_stamp":"2024-08-03T20:54:25.402282+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T03:43:43.509403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.18","version":"0'0","reported_seq":13,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-08-03T20:54:29.482228+0000","last_change":"2024-08-03T20:54:27.454767+0000","last_active":"2024-08-03T20:54:29.482228+0000","last_peered":"2024-08-03T20:54:29.482228+0000","last_clean":"2024-08-03T20:54:29.482228+0000","last_became_active":"2024-08-03T20:54:27.454333+0000","last_became_peered":"2024-08-03T20:54:27.454333+0000","last_unstale":"2024-08-03T20:54:29.482228+0000","last_undegraded":"2024-08-03T20:54:29.482228+0000","last_fullsized":"2024-08-03T20:54:29.482228+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:54:26.410149+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:54:26.410149+0000","last_clean_scrub_stamp":"2024-08-03T20:54:26.410149+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T02:40:19.572527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.19","version":"0'0","reported_seq":15,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-08-03T20:54:29.438118+0000","last_change":"2024-08-03T20:54:26.437810+0000","last_active":"2024-08-03T20:54:29.438118+0000","last_peered":"2024-08-03T20:54:29.438118+0000","last_clean":"2024-08-03T20:54:29.438118+0000","last_became_active":"2024-08-03T20:54:26.437340+0000","last_became_peered":"2024-08-03T20:54:26.437340+0000","last_unstale":"2024-08-03T20:54:29.438118+0000","last_undegraded":"2024-08-03T20:54:29.438118+0000","last_fullsized":"2024-08-03T20:54:29.438118+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:54:25.402282+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:54:25.402282+0000","last_clean_scrub_stamp":"2024-08-03T20:54:25.402282+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T05:48:41.687726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.19","version":"0'0","reported_seq":10,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-03T20:54:27.442546+0000","last_change":"2024-08-03T20:54:27.440130+0000","last_active":"2024-08-03T20:54:27.442546+0000","last_peered":"2024-08-03T20:54:27.442546+0000","last_clean":"2024-08-03T20:54:27.442546+0000","last_became_active":"2024-08-03T20:54:27.439791+0000","last_became_peered":"2024-08-03T20:54:27.439791+0000","last_unstale":"2024-08-03T20:54:27.442546+0000","last_undegraded":"2024-08-03T20:54:27.442546+0000","last_fullsized":"2024-08-03T20:54:27.442546+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:54:26.410149+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:54:26.410149+0000","last_clean_scrub_stamp":"2024-08-03T20:54:26.410149+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T02:14:15.424140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.18","version":"0'0","reported_seq":15,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-08-03T20:54:29.434815+0000","last_change":"2024-08-03T20:54:26.438649+0000","last_active":"2024-08-03T20:54:29.434815+0000","last_peered":"2024-08-03T20:54:29.434815+0000","last_clean":"2024-08-03T20:54:29.434815+0000","last_became_active":"2024-08-03T20:54:26.438312+0000","last_became_peered":"2024-08-03T20:54:26.438312+0000","last_unstale":"2024-08-03T20:54:29.434815+0000","last_undegraded":"2024-08-03T20:54:29.434815+0000","last_fullsized":"2024-08-03T20:54:29.434815+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:54:25.402282+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:54:25.402282+0000","last_clean_scrub_stamp":"2024-08-03T20:54:25.402282+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T04:32:12.785658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.16","version":"0'0","reported_seq":10,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-03T20:54:27.447146+0000","last_change":"2024-08-03T20:54:27.443533+0000","last_active":"2024-08-03T20:54:27.447146+0000","last_peered":"2024-08-03T20:54:27.447146+0000","last_clean":"2024-08-03T20:54:27.447146+0000","last_became_active":"2024-08-03T20:54:27.443050+0000","last_became_peered":"2024-08-03T20:54:27.443050+0000","last_unstale":"2024-08-03T20:54:27.447146+0000","last_undegraded":"2024-08-03T20:54:27.447146+0000","last_fullsized":"2024-08-03T20:54:27.447146+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:54:26.410149+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:54:26.410149+0000","last_clean_scrub_stamp":"2024-08-03T20:54:26.410149+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T02:32:26.395472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.17","version":"0'0","reported_seq":15,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-08-03T20:54:28.440826+0000","last_change":"2024-08-03T20:54:26.445519+0000","last_active":"2024-08-03T20:54:28.440826+0000","last_peered":"2024-08-03T20:54:28.440826+0000","last_clean":"2024-08-03T20:54:28.440826+0000","last_became_active":"2024-08-03T20:54:26.445143+0000","last_became_peered":"2024-08-03T20:54:26.445143+0000","last_unstale":"2024-08-03T20:54:28.440826+0000","last_undegraded":"2024-08-03T20:54:28.440826+0000","last_fullsized":"2024-08-03T20:54:28.440826+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:54:25.402282+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:54:25.402282+0000","last_clean_scrub_stamp":"2024-08-03T20:54:25.402282+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T08:02:42.980625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.17","version":"0'0","reported_seq":13,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-08-03T20:54:29.323861+0000","last_change":"2024-08-03T20:54:27.453807+0000","last_active":"2024-08-03T20:54:29.323861+0000","last_peered":"2024-08-03T20:54:29.323861+0000","last_clean":"2024-08-03T20:54:29.323861+0000","last_became_active":"2024-08-03T20:54:27.453482+0000","last_became_peered":"2024-08-03T20:54:27.453482+0000","last_unstale":"2024-08-03T20:54:29.323861+0000","last_undegraded":"2024-08-03T20:54:29.323861+0000","last_fullsized":"2024-08-03T20:54:29.323861+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:54:26.410149+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:54:26.410149+0000","last_clean_scrub_stamp":"2024-08-03T20:54:26.410149+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T03:14:09.919834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.16","version":"0'0","reported_seq":15,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-08-03T20:54:28.440192+0000","last_change":"2024-08-03T20:54:26.443025+0000","last_active":"2024-08-03T20:54:28.440192+0000","last_peered":"2024-08-03T20:54:28.440192+0000","last_clean":"2024-08-03T20:54:28.440192+0000","last_became_active":"2024-08-03T20:54:26.442717+0000","last_became_peered":"2024-08-03T20:54:26.442717+0000","last_unstale":"2024-08-03T20:54:28.440192+0000","last_undegraded":"2024-08-03T20:54:28.440192+0000","last_fullsized":"2024-08-03T20:54:28.440192+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:54:25.402282+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:54:25.402282+0000","last_clean_scrub_stamp":"2024-08-03T20:54:25.402282+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-04T23:52:56.823930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.14","version":"0'0","reported_seq":13,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-08-03T20:54:29.438462+0000","last_change":"2024-08-03T20:54:27.457277+0000","last_active":"2024-08-03T20:54:29.438462+0000","last_peered":"2024-08-03T20:54:29.438462+0000","last_clean":"2024-08-03T20:54:29.438462+0000","last_became_active":"2024-08-03T20:54:27.456953+0000","last_became_peered":"2024-08-03T20:54:27.456953+0000","last_unstale":"2024-08-03T20:54:29.438462+0000","last_undegraded":"2024-08-03T20:54:29.438462+0000","last_fullsized":"2024-08-03T20:54:29.438462+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:54:26.410149+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:54:26.410149+0000","last_clean_scrub_stamp":"2024-08-03T20:54:26.410149+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T05:28:20.584840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.15","version":"82'3","reported_seq":18,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-08-03T20:54:28.439817+0000","last_change":"2024-08-03T20:54:26.444297+0000","last_active":"2024-08-03T20:54:28.439817+0000","last_peered":"2024-08-03T20:54:28.439817+0000","last_clean":"2024-08-03T20:54:28.439817+0000","last_became_active":"2024-08-03T20:54:26.443688+0000","last_became_peered":"2024-08-03T20:54:26.443688+0000","last_unstale":"2024-08-03T20:54:28.439817+0000","last_undegraded":"2024-08-03T20:54:28.439817+0000","last_fullsized":"2024-08-03T20:54:28.439817+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:54:25.402282+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:54:25.402282+0000","last_clean_scrub_stamp":"2024-08-03T20:54:25.402282+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T06:55:43.645755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.15","version":"0'0","reported_seq":13,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-08-03T20:54:28.441517+0000","last_change":"2024-08-03T20:54:27.450035+0000","last_active":"2024-08-03T20:54:28.441517+0000","last_peered":"2024-08-03T20:54:28.441517+0000","last_clean":"2024-08-03T20:54:28.441517+0000","last_became_active":"2024-08-03T20:54:27.449500+0000","last_became_peered":"2024-08-03T20:54:27.449500+0000","last_unstale":"2024-08-03T20:54:28.441517+0000","last_undegraded":"2024-08-03T20:54:28.441517+0000","last_fullsized":"2024-08-03T20:54:28.441517+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:54:26.410149+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:54:26.410149+0000","last_clean_scrub_stamp":"2024-08-03T20:54:26.410149+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T04:07:08.886579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.14","version":"82'6","reported_seq":21,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-08-03T20:54:29.323668+0000","last_change":"2024-08-03T20:54:26.448489+0000","last_active":"2024-08-03T20:54:29.323668+0000","last_peered":"2024-08-03T20:54:29.323668+0000","last_clean":"2024-08-03T20:54:29.323668+0000","last_became_active":"2024-08-03T20:54:26.448232+0000","last_became_peered":"2024-08-03T20:54:26.448232+0000","last_unstale":"2024-08-03T20:54:29.323668+0000","last_undegraded":"2024-08-03T20:54:29.323668+0000","last_fullsized":"2024-08-03T20:54:29.323668+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:54:25.402282+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:54:25.402282+0000","last_clean_scrub_stamp":"2024-08-03T20:54:25.402282+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T05:02:11.770691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.12","version":"0'0","reported_seq":13,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-08-03T20:54:29.438195+0000","last_change":"2024-08-03T20:54:27.430210+0000","last_active":"2024-08-03T20:54:29.438195+0000","last_peered":"2024-08-03T20:54:29.438195+0000","last_clean":"2024-08-03T20:54:29.438195+0000","last_became_active":"2024-08-03T20:54:27.429901+0000","last_became_peered":"2024-08-03T20:54:27.429901+0000","last_unstale":"2024-08-03T20:54:29.438195+0000","last_undegraded":"2024-08-03T20:54:29.438195+0000","last_fullsized":"2024-08-03T20:54:29.438195+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:54:26.410149+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:54:26.410149+0000","last_clean_scrub_stamp":"2024-08-03T20:54:26.410149+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-04T23:30:55.227524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.13","version":"82'2","reported_seq":17,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-08-03T20:54:29.434900+0000","last_change":"2024-08-03T20:54:26.443771+0000","last_active":"2024-08-03T20:54:29.434900+0000","last_peered":"2024-08-03T20:54:29.434900+0000","last_clean":"2024-08-03T20:54:29.434900+0000","last_became_active":"2024-08-03T20:54:26.443423+0000","last_became_peered":"2024-08-03T20:54:26.443423+0000","last_unstale":"2024-08-03T20:54:29.434900+0000","last_undegraded":"2024-08-03T20:54:29.434900+0000","last_fullsized":"2024-08-03T20:54:29.434900+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:54:25.402282+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:54:25.402282+0000","last_clean_scrub_stamp":"2024-08-03T20:54:25.402282+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T06:43:59.921992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.13","version":"0'0","reported_seq":13,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-08-03T20:54:29.482142+0000","last_change":"2024-08-03T20:54:27.454959+0000","last_active":"2024-08-03T20:54:29.482142+0000","last_peered":"2024-08-03T20:54:29.482142+0000","last_clean":"2024-08-03T20:54:29.482142+0000","last_became_active":"2024-08-03T20:54:27.454627+0000","last_became_peered":"2024-08-03T20:54:27.454627+0000","last_unstale":"2024-08-03T20:54:29.482142+0000","last_undegraded":"2024-08-03T20:54:29.482142+0000","last_fullsized":"2024-08-03T20:54:29.482142+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:54:26.410149+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:54:26.410149+0000","last_clean_scrub_stamp":"2024-08-03T20:54:26.410149+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T01:06:27.377827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.12","version":"0'0","reported_seq":13,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-03T20:54:27.429097+0000","last_change":"2024-08-03T20:54:26.438693+0000","last_active":"2024-08-03T20:54:27.429097+0000","last_peered":"2024-08-03T20:54:27.429097+0000","last_clean":"2024-08-03T20:54:27.429097+0000","last_became_active":"2024-08-03T20:54:26.438415+0000","last_became_peered":"2024-08-03T20:54:26.438415+0000","last_unstale":"2024-08-03T20:54:27.429097+0000","last_undegraded":"2024-08-03T20:54:27.429097+0000","last_fullsized":"2024-08-03T20:54:27.429097+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:54:25.402282+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:54:25.402282+0000","last_clean_scrub_stamp":"2024-08-03T20:54:25.402282+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T07:40:12.546187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.10","version":"0'0","reported_seq":13,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-08-03T20:54:29.434831+0000","last_change":"2024-08-03T20:54:27.469162+0000","last_active":"2024-08-03T20:54:29.434831+0000","last_peered":"2024-08-03T20:54:29.434831+0000","last_clean":"2024-08-03T20:54:29.434831+0000","last_became_active":"2024-08-03T20:54:27.468898+0000","last_became_peered":"2024-08-03T20:54:27.468898+0000","last_unstale":"2024-08-03T20:54:29.434831+0000","last_undegraded":"2024-08-03T20:54:29.434831+0000","last_fullsized":"2024-08-03T20:54:29.434831+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:54:26.410149+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:54:26.410149+0000","last_clean_scrub_stamp":"2024-08-03T20:54:26.410149+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-04T23:35:09.032967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.11","version":"0'0","reported_seq":15,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-08-03T20:54:29.323466+0000","last_change":"2024-08-03T20:54:26.435525+0000","last_active":"2024-08-03T20:54:29.323466+0000","last_peered":"2024-08-03T20:54:29.323466+0000","last_clean":"2024-08-03T20:54:29.323466+0000","last_became_active":"2024-08-03T20:54:26.435186+0000","last_became_peered":"2024-08-03T20:54:26.435186+0000","last_unstale":"2024-08-03T20:54:29.323466+0000","last_undegraded":"2024-08-03T20:54:29.323466+0000","last_fullsized":"2024-08-03T20:54:29.323466+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:54:25.402282+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:54:25.402282+0000","last_clean_scrub_stamp":"2024-08-03T20:54:25.402282+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T04:30:44.935983+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.11","version":"0'0","reported_seq":13,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-08-03T20:54:29.434773+0000","last_change":"2024-08-03T20:54:27.450906+0000","last_active":"2024-08-03T20:54:29.434773+0000","last_peered":"2024-08-03T20:54:29.434773+0000","last_clean":"2024-08-03T20:54:29.434773+0000","last_became_active":"2024-08-03T20:54:27.450609+0000","last_became_peered":"2024-08-03T20:54:27.450609+0000","last_unstale":"2024-08-03T20:54:29.434773+0000","last_undegraded":"2024-08-03T20:54:29.434773+0000","last_fullsized":"2024-08-03T20:54:29.434773+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:54:26.410149+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:54:26.410149+0000","last_clean_scrub_stamp":"2024-08-03T20:54:26.410149+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T02:03:59.450233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.10","version":"82'8","reported_seq":23,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-08-03T20:54:29.438106+0000","last_change":"2024-08-03T20:54:26.437602+0000","last_active":"2024-08-03T20:54:29.438106+0000","last_peered":"2024-08-03T20:54:29.438106+0000","last_clean":"2024-08-03T20:54:29.438106+0000","last_became_active":"2024-08-03T20:54:26.437247+0000","last_became_peered":"2024-08-03T20:54:26.437247+0000","last_unstale":"2024-08-03T20:54:29.438106+0000","last_undegraded":"2024-08-03T20:54:29.438106+0000","last_fullsized":"2024-08-03T20:54:29.438106+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:54:25.402282+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:54:25.402282+0000","last_clean_scrub_stamp":"2024-08-03T20:54:25.402282+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T05:24:45.467432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.e","version":"0'0","reported_seq":13,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-08-03T20:54:29.482289+0000","last_change":"2024-08-03T20:54:27.445155+0000","last_active":"2024-08-03T20:54:29.482289+0000","last_peered":"2024-08-03T20:54:29.482289+0000","last_clean":"2024-08-03T20:54:29.482289+0000","last_became_active":"2024-08-03T20:54:27.443888+0000","last_became_peered":"2024-08-03T20:54:27.443888+0000","last_unstale":"2024-08-03T20:54:29.482289+0000","last_undegraded":"2024-08-03T20:54:29.482289+0000","last_fullsized":"2024-08-03T20:54:29.482289+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:54:26.410149+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:54:26.410149+0000","last_clean_scrub_stamp":"2024-08-03T20:54:26.410149+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T00:49:10.925372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.f","version":"82'1","reported_seq":16,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-08-03T20:54:28.440933+0000","last_change":"2024-08-03T20:54:26.438576+0000","last_active":"2024-08-03T20:54:28.440933+0000","last_peered":"2024-08-03T20:54:28.440933+0000","last_clean":"2024-08-03T20:54:28.440933+0000","last_became_active":"2024-08-03T20:54:26.437929+0000","last_became_peered":"2024-08-03T20:54:26.437929+0000","last_unstale":"2024-08-03T20:54:28.440933+0000","last_undegraded":"2024-08-03T20:54:28.440933+0000","last_fullsized":"2024-08-03T20:54:28.440933+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:54:25.402282+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:54:25.402282+0000","last_clean_scrub_stamp":"2024-08-03T20:54:25.402282+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-04T21:02:34.790434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.f","version":"0'0","reported_seq":13,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-08-03T20:54:28.440713+0000","last_change":"2024-08-03T20:54:27.441892+0000","last_active":"2024-08-03T20:54:28.440713+0000","last_peered":"2024-08-03T20:54:28.440713+0000","last_clean":"2024-08-03T20:54:28.440713+0000","last_became_active":"2024-08-03T20:54:27.440993+0000","last_became_peered":"2024-08-03T20:54:27.440993+0000","last_unstale":"2024-08-03T20:54:28.440713+0000","last_undegraded":"2024-08-03T20:54:28.440713+0000","last_fullsized":"2024-08-03T20:54:28.440713+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:54:26.410149+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:54:26.410149+0000","last_clean_scrub_stamp":"2024-08-03T20:54:26.410149+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T01:06:27.377827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.e","version":"82'2","reported_seq":17,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-08-03T20:54:29.438574+0000","last_change":"2024-08-03T20:54:26.446254+0000","last_active":"2024-08-03T20:54:29.438574+0000","last_peered":"2024-08-03T20:54:29.438574+0000","last_clean":"2024-08-03T20:54:29.438574+0000","last_became_active":"2024-08-03T20:54:26.443273+0000","last_became_peered":"2024-08-03T20:54:26.443273+0000","last_unstale":"2024-08-03T20:54:29.438574+0000","last_undegraded":"2024-08-03T20:54:29.438574+0000","last_fullsized":"2024-08-03T20:54:29.438574+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:54:25.402282+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:54:25.402282+0000","last_clean_scrub_stamp":"2024-08-03T20:54:25.402282+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T08:38:30.463265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.d","version":"0'0","reported_seq":13,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-08-03T20:54:28.441726+0000","last_change":"2024-08-03T20:54:27.446287+0000","last_active":"2024-08-03T20:54:28.441726+0000","last_peered":"2024-08-03T20:54:28.441726+0000","last_clean":"2024-08-03T20:54:28.441726+0000","last_became_active":"2024-08-03T20:54:27.443555+0000","last_became_peered":"2024-08-03T20:54:27.443555+0000","last_unstale":"2024-08-03T20:54:28.441726+0000","last_undegraded":"2024-08-03T20:54:28.441726+0000","last_fullsized":"2024-08-03T20:54:28.441726+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:54:26.410149+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:54:26.410149+0000","last_clean_scrub_stamp":"2024-08-03T20:54:26.410149+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T02:46:49.413845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.c","version":"0'0","reported_seq":15,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-08-03T20:54:29.438657+0000","last_change":"2024-08-03T20:54:26.438540+0000","last_active":"2024-08-03T20:54:29.438657+0000","last_peered":"2024-08-03T20:54:29.438657+0000","last_clean":"2024-08-03T20:54:29.438657+0000","last_became_active":"2024-08-03T20:54:26.438215+0000","last_became_peered":"2024-08-03T20:54:26.438215+0000","last_unstale":"2024-08-03T20:54:29.438657+0000","last_undegraded":"2024-08-03T20:54:29.438657+0000","last_fullsized":"2024-08-03T20:54:29.438657+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:54:25.402282+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:54:25.402282+0000","last_clean_scrub_stamp":"2024-08-03T20:54:25.402282+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T03:38:16.207416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.a","version":"0'0","reported_seq":10,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-03T20:54:27.465460+0000","last_change":"2024-08-03T20:54:27.458690+0000","last_active":"2024-08-03T20:54:27.465460+0000","last_peered":"2024-08-03T20:54:27.465460+0000","last_clean":"2024-08-03T20:54:27.465460+0000","last_became_active":"2024-08-03T20:54:27.458264+0000","last_became_peered":"2024-08-03T20:54:27.458264+0000","last_unstale":"2024-08-03T20:54:27.465460+0000","last_undegraded":"2024-08-03T20:54:27.465460+0000","last_fullsized":"2024-08-03T20:54:27.465460+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:54:26.410149+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:54:26.410149+0000","last_clean_scrub_stamp":"2024-08-03T20:54:26.410149+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T00:18:26.766575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.b","version":"82'2","reported_seq":15,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-03T20:54:27.471748+0000","last_change":"2024-08-03T20:54:26.439824+0000","last_active":"2024-08-03T20:54:27.471748+0000","last_peered":"2024-08-03T20:54:27.471748+0000","last_clean":"2024-08-03T20:54:27.471748+0000","last_became_active":"2024-08-03T20:54:26.439533+0000","last_became_peered":"2024-08-03T20:54:26.439533+0000","last_unstale":"2024-08-03T20:54:27.471748+0000","last_undegraded":"2024-08-03T20:54:27.471748+0000","last_fullsized":"2024-08-03T20:54:27.471748+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:54:25.402282+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:54:25.402282+0000","last_clean_scrub_stamp":"2024-08-03T20:54:25.402282+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T01:42:10.837670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.b","version":"0'0","reported_seq":13,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-08-03T20:54:29.323745+0000","last_change":"2024-08-03T20:54:27.439785+0000","last_active":"2024-08-03T20:54:29.323745+0000","last_peered":"2024-08-03T20:54:29.323745+0000","last_clean":"2024-08-03T20:54:29.323745+0000","last_became_active":"2024-08-03T20:54:27.438739+0000","last_became_peered":"2024-08-03T20:54:27.438739+0000","last_unstale":"2024-08-03T20:54:29.323745+0000","last_undegraded":"2024-08-03T20:54:29.323745+0000","last_fullsized":"2024-08-03T20:54:29.323745+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:54:26.410149+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:54:26.410149+0000","last_clean_scrub_stamp":"2024-08-03T20:54:26.410149+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T05:28:20.584840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.a","version":"0'0","reported_seq":13,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-03T20:54:27.450560+0000","last_change":"2024-08-03T20:54:26.443888+0000","last_active":"2024-08-03T20:54:27.450560+0000","last_peered":"2024-08-03T20:54:27.450560+0000","last_clean":"2024-08-03T20:54:27.450560+0000","last_became_active":"2024-08-03T20:54:26.441806+0000","last_became_peered":"2024-08-03T20:54:26.441806+0000","last_unstale":"2024-08-03T20:54:27.450560+0000","last_undegraded":"2024-08-03T20:54:27.450560+0000","last_fullsized":"2024-08-03T20:54:27.450560+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:54:25.402282+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:54:25.402282+0000","last_clean_scrub_stamp":"2024-08-03T20:54:25.402282+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T05:30:32.126828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.8","version":"0'0","reported_seq":10,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-03T20:54:27.465768+0000","last_change":"2024-08-03T20:54:27.453663+0000","last_active":"2024-08-03T20:54:27.465768+0000","last_peered":"2024-08-03T20:54:27.465768+0000","last_clean":"2024-08-03T20:54:27.465768+0000","last_became_active":"2024-08-03T20:54:27.450915+0000","last_became_peered":"2024-08-03T20:54:27.450915+0000","last_unstale":"2024-08-03T20:54:27.465768+0000","last_undegraded":"2024-08-03T20:54:27.465768+0000","last_fullsized":"2024-08-03T20:54:27.465768+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:54:26.410149+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:54:26.410149+0000","last_clean_scrub_stamp":"2024-08-03T20:54:26.410149+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T08:20:46.797506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.9","version":"0'0","reported_seq":13,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-03T20:54:27.450827+0000","last_change":"2024-08-03T20:54:26.445547+0000","last_active":"2024-08-03T20:54:27.450827+0000","last_peered":"2024-08-03T20:54:27.450827+0000","last_clean":"2024-08-03T20:54:27.450827+0000","last_became_active":"2024-08-03T20:54:26.445196+0000","last_became_peered":"2024-08-03T20:54:26.445196+0000","last_unstale":"2024-08-03T20:54:27.450827+0000","last_undegraded":"2024-08-03T20:54:27.450827+0000","last_fullsized":"2024-08-03T20:54:27.450827+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:54:25.402282+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:54:25.402282+0000","last_clean_scrub_stamp":"2024-08-03T20:54:25.402282+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T00:04:49.083414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.9","version":"0'0","reported_seq":13,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-08-03T20:54:29.323520+0000","last_change":"2024-08-03T20:54:27.447378+0000","last_active":"2024-08-03T20:54:29.323520+0000","last_peered":"2024-08-03T20:54:29.323520+0000","last_clean":"2024-08-03T20:54:29.323520+0000","last_became_active":"2024-08-03T20:54:27.444614+0000","last_became_peered":"2024-08-03T20:54:27.444614+0000","last_unstale":"2024-08-03T20:54:29.323520+0000","last_undegraded":"2024-08-03T20:54:29.323520+0000","last_fullsized":"2024-08-03T20:54:29.323520+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:54:26.410149+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:54:26.410149+0000","last_clean_scrub_stamp":"2024-08-03T20:54:26.410149+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-04T21:41:30.478055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.8","version":"0'0","reported_seq":13,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-03T20:54:27.428451+0000","last_change":"2024-08-03T20:54:26.440569+0000","last_active":"2024-08-03T20:54:27.428451+0000","last_peered":"2024-08-03T20:54:27.428451+0000","last_clean":"2024-08-03T20:54:27.428451+0000","last_became_active":"2024-08-03T20:54:26.440262+0000","last_became_peered":"2024-08-03T20:54:26.440262+0000","last_unstale":"2024-08-03T20:54:27.428451+0000","last_undegraded":"2024-08-03T20:54:27.428451+0000","last_fullsized":"2024-08-03T20:54:27.428451+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:54:25.402282+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:54:25.402282+0000","last_clean_scrub_stamp":"2024-08-03T20:54:25.402282+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T06:50:58.322620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.6","version":"0'0","reported_seq":13,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-08-03T20:54:28.441840+0000","last_change":"2024-08-03T20:54:27.434816+0000","last_active":"2024-08-03T20:54:28.441840+0000","last_peered":"2024-08-03T20:54:28.441840+0000","last_clean":"2024-08-03T20:54:28.441840+0000","last_became_active":"2024-08-03T20:54:27.434505+0000","last_became_peered":"2024-08-03T20:54:27.434505+0000","last_unstale":"2024-08-03T20:54:28.441840+0000","last_undegraded":"2024-08-03T20:54:28.441840+0000","last_fullsized":"2024-08-03T20:54:28.441840+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:54:26.410149+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:54:26.410149+0000","last_clean_scrub_stamp":"2024-08-03T20:54:26.410149+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T00:31:19.062668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.7","version":"82'2","reported_seq":17,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-08-03T20:54:29.435271+0000","last_change":"2024-08-03T20:54:26.445984+0000","last_active":"2024-08-03T20:54:29.435271+0000","last_peered":"2024-08-03T20:54:29.435271+0000","last_clean":"2024-08-03T20:54:29.435271+0000","last_became_active":"2024-08-03T20:54:26.445698+0000","last_became_peered":"2024-08-03T20:54:26.445698+0000","last_unstale":"2024-08-03T20:54:29.435271+0000","last_undegraded":"2024-08-03T20:54:29.435271+0000","last_fullsized":"2024-08-03T20:54:29.435271+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:54:25.402282+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:54:25.402282+0000","last_clean_scrub_stamp":"2024-08-03T20:54:25.402282+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T04:56:33.713984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.7","version":"0'0","reported_seq":13,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-08-03T20:54:29.482258+0000","last_change":"2024-08-03T20:54:27.443424+0000","last_active":"2024-08-03T20:54:29.482258+0000","last_peered":"2024-08-03T20:54:29.482258+0000","last_clean":"2024-08-03T20:54:29.482258+0000","last_became_active":"2024-08-03T20:54:27.443031+0000","last_became_peered":"2024-08-03T20:54:27.443031+0000","last_unstale":"2024-08-03T20:54:29.482258+0000","last_undegraded":"2024-08-03T20:54:29.482258+0000","last_fullsized":"2024-08-03T20:54:29.482258+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:54:26.410149+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:54:26.410149+0000","last_clean_scrub_stamp":"2024-08-03T20:54:26.410149+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T07:40:36.636568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.6","version":"83'1","reported_seq":16,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-08-03T20:54:29.437258+0000","last_change":"2024-08-03T20:54:26.443003+0000","last_active":"2024-08-03T20:54:29.437258+0000","last_peered":"2024-08-03T20:54:29.437258+0000","last_clean":"2024-08-03T20:54:29.437258+0000","last_became_active":"2024-08-03T20:54:26.442653+0000","last_became_peered":"2024-08-03T20:54:26.442653+0000","last_unstale":"2024-08-03T20:54:29.437258+0000","last_undegraded":"2024-08-03T20:54:29.437258+0000","last_fullsized":"2024-08-03T20:54:29.437258+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:54:25.402282+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:54:25.402282+0000","last_clean_scrub_stamp":"2024-08-03T20:54:25.402282+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T04:30:23.014758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.4","version":"0'0","reported_seq":13,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-08-03T20:54:28.440469+0000","last_change":"2024-08-03T20:54:27.457535+0000","last_active":"2024-08-03T20:54:28.440469+0000","last_peered":"2024-08-03T20:54:28.440469+0000","last_clean":"2024-08-03T20:54:28.440469+0000","last_became_active":"2024-08-03T20:54:27.457180+0000","last_became_peered":"2024-08-03T20:54:27.457180+0000","last_unstale":"2024-08-03T20:54:28.440469+0000","last_undegraded":"2024-08-03T20:54:28.440469+0000","last_fullsized":"2024-08-03T20:54:28.440469+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:54:26.410149+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:54:26.410149+0000","last_clean_scrub_stamp":"2024-08-03T20:54:26.410149+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T07:09:56.880905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.5","version":"0'0","reported_seq":15,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-08-03T20:54:28.440118+0000","last_change":"2024-08-03T20:54:26.444463+0000","last_active":"2024-08-03T20:54:28.440118+0000","last_peered":"2024-08-03T20:54:28.440118+0000","last_clean":"2024-08-03T20:54:28.440118+0000","last_became_active":"2024-08-03T20:54:26.444121+0000","last_became_peered":"2024-08-03T20:54:26.444121+0000","last_unstale":"2024-08-03T20:54:28.440118+0000","last_undegraded":"2024-08-03T20:54:28.440118+0000","last_fullsized":"2024-08-03T20:54:28.440118+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:54:25.402282+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:54:25.402282+0000","last_clean_scrub_stamp":"2024-08-03T20:54:25.402282+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T03:28:34.661003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.5","version":"0'0","reported_seq":13,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-08-03T20:54:28.441816+0000","last_change":"2024-08-03T20:54:27.448379+0000","last_active":"2024-08-03T20:54:28.441816+0000","last_peered":"2024-08-03T20:54:28.441816+0000","last_clean":"2024-08-03T20:54:28.441816+0000","last_became_active":"2024-08-03T20:54:27.448078+0000","last_became_peered":"2024-08-03T20:54:27.448078+0000","last_unstale":"2024-08-03T20:54:28.441816+0000","last_undegraded":"2024-08-03T20:54:28.441816+0000","last_fullsized":"2024-08-03T20:54:28.441816+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:54:26.410149+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:54:26.410149+0000","last_clean_scrub_stamp":"2024-08-03T20:54:26.410149+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T08:02:25.589517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.4","version":"0'0","reported_seq":15,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-08-03T20:54:28.440481+0000","last_change":"2024-08-03T20:54:26.440460+0000","last_active":"2024-08-03T20:54:28.440481+0000","last_peered":"2024-08-03T20:54:28.440481+0000","last_clean":"2024-08-03T20:54:28.440481+0000","last_became_active":"2024-08-03T20:54:26.439877+0000","last_became_peered":"2024-08-03T20:54:26.439877+0000","last_unstale":"2024-08-03T20:54:28.440481+0000","last_undegraded":"2024-08-03T20:54:28.440481+0000","last_fullsized":"2024-08-03T20:54:28.440481+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:54:25.402282+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:54:25.402282+0000","last_clean_scrub_stamp":"2024-08-03T20:54:25.402282+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-04T21:51:04.354598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.2","version":"0'0","reported_seq":13,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-08-03T20:54:29.323650+0000","last_change":"2024-08-03T20:54:27.439499+0000","last_active":"2024-08-03T20:54:29.323650+0000","last_peered":"2024-08-03T20:54:29.323650+0000","last_clean":"2024-08-03T20:54:29.323650+0000","last_became_active":"2024-08-03T20:54:27.438801+0000","last_became_peered":"2024-08-03T20:54:27.438801+0000","last_unstale":"2024-08-03T20:54:29.323650+0000","last_undegraded":"2024-08-03T20:54:29.323650+0000","last_fullsized":"2024-08-03T20:54:29.323650+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:54:26.410149+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:54:26.410149+0000","last_clean_scrub_stamp":"2024-08-03T20:54:26.410149+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T06:59:28.260517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.3","version":"0'0","reported_seq":15,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-08-03T20:54:29.438732+0000","last_change":"2024-08-03T20:54:26.445654+0000","last_active":"2024-08-03T20:54:29.438732+0000","last_peered":"2024-08-03T20:54:29.438732+0000","last_clean":"2024-08-03T20:54:29.438732+0000","last_became_active":"2024-08-03T20:54:26.442568+0000","last_became_peered":"2024-08-03T20:54:26.442568+0000","last_unstale":"2024-08-03T20:54:29.438732+0000","last_undegraded":"2024-08-03T20:54:29.438732+0000","last_fullsized":"2024-08-03T20:54:29.438732+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:54:25.402282+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:54:25.402282+0000","last_clean_scrub_stamp":"2024-08-03T20:54:25.402282+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T08:38:24.403913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.3","version":"0'0","reported_seq":13,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-08-03T20:54:29.438541+0000","last_change":"2024-08-03T20:54:27.430945+0000","last_active":"2024-08-03T20:54:29.438541+0000","last_peered":"2024-08-03T20:54:29.438541+0000","last_clean":"2024-08-03T20:54:29.438541+0000","last_became_active":"2024-08-03T20:54:27.430617+0000","last_became_peered":"2024-08-03T20:54:27.430617+0000","last_unstale":"2024-08-03T20:54:29.438541+0000","last_undegraded":"2024-08-03T20:54:29.438541+0000","last_fullsized":"2024-08-03T20:54:29.438541+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:54:26.410149+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:54:26.410149+0000","last_clean_scrub_stamp":"2024-08-03T20:54:26.410149+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T03:14:09.919834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.2","version":"0'0","reported_seq":15,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-08-03T20:54:28.441463+0000","last_change":"2024-08-03T20:54:26.438196+0000","last_active":"2024-08-03T20:54:28.441463+0000","last_peered":"2024-08-03T20:54:28.441463+0000","last_clean":"2024-08-03T20:54:28.441463+0000","last_became_active":"2024-08-03T20:54:26.437831+0000","last_became_peered":"2024-08-03T20:54:26.437831+0000","last_unstale":"2024-08-03T20:54:28.441463+0000","last_undegraded":"2024-08-03T20:54:28.441463+0000","last_fullsized":"2024-08-03T20:54:28.441463+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:54:25.402282+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:54:25.402282+0000","last_clean_scrub_stamp":"2024-08-03T20:54:25.402282+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T04:35:25.487288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.0","version":"0'0","reported_seq":13,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-08-03T20:54:29.482432+0000","last_change":"2024-08-03T20:54:27.453943+0000","last_active":"2024-08-03T20:54:29.482432+0000","last_peered":"2024-08-03T20:54:29.482432+0000","last_clean":"2024-08-03T20:54:29.482432+0000","last_became_active":"2024-08-03T20:54:27.453580+0000","last_became_peered":"2024-08-03T20:54:27.453580+0000","last_unstale":"2024-08-03T20:54:29.482432+0000","last_undegraded":"2024-08-03T20:54:29.482432+0000","last_fullsized":"2024-08-03T20:54:29.482432+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:54:26.410149+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:54:26.410149+0000","last_clean_scrub_stamp":"2024-08-03T20:54:26.410149+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T04:23:13.290160+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.1","version":"0'0","reported_seq":15,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-08-03T20:54:28.439874+0000","last_change":"2024-08-03T20:54:26.439778+0000","last_active":"2024-08-03T20:54:28.439874+0000","last_peered":"2024-08-03T20:54:28.439874+0000","last_clean":"2024-08-03T20:54:28.439874+0000","last_became_active":"2024-08-03T20:54:26.438340+0000","last_became_peered":"2024-08-03T20:54:26.438340+0000","last_unstale":"2024-08-03T20:54:28.439874+0000","last_undegraded":"2024-08-03T20:54:28.439874+0000","last_fullsized":"2024-08-03T20:54:28.439874+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:54:25.402282+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:54:25.402282+0000","last_clean_scrub_stamp":"2024-08-03T20:54:25.402282+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-04T23:35:08.025100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1","version":"0'0","reported_seq":13,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-08-03T20:54:28.440442+0000","last_change":"2024-08-03T20:54:27.457876+0000","last_active":"2024-08-03T20:54:28.440442+0000","last_peered":"2024-08-03T20:54:28.440442+0000","last_clean":"2024-08-03T20:54:28.440442+0000","last_became_active":"2024-08-03T20:54:27.457492+0000","last_became_peered":"2024-08-03T20:54:27.457492+0000","last_unstale":"2024-08-03T20:54:28.440442+0000","last_undegraded":"2024-08-03T20:54:28.440442+0000","last_fullsized":"2024-08-03T20:54:28.440442+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:54:26.410149+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:54:26.410149+0000","last_clean_scrub_stamp":"2024-08-03T20:54:26.410149+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T07:56:54.337763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.0","version":"0'0","reported_seq":15,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-08-03T20:54:28.441279+0000","last_change":"2024-08-03T20:54:26.442870+0000","last_active":"2024-08-03T20:54:28.441279+0000","last_peered":"2024-08-03T20:54:28.441279+0000","last_clean":"2024-08-03T20:54:28.441279+0000","last_became_active":"2024-08-03T20:54:26.442475+0000","last_became_peered":"2024-08-03T20:54:26.442475+0000","last_unstale":"2024-08-03T20:54:28.441279+0000","last_undegraded":"2024-08-03T20:54:28.441279+0000","last_fullsized":"2024-08-03T20:54:28.441279+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:54:25.402282+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:54:25.402282+0000","last_clean_scrub_stamp":"2024-08-03T20:54:25.402282+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T05:11:57.699458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.c","version":"0'0","reported_seq":13,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-08-03T20:54:29.435238+0000","last_change":"2024-08-03T20:54:27.442895+0000","last_active":"2024-08-03T20:54:29.435238+0000","last_peered":"2024-08-03T20:54:29.435238+0000","last_clean":"2024-08-03T20:54:29.435238+0000","last_became_active":"2024-08-03T20:54:27.441598+0000","last_became_peered":"2024-08-03T20:54:27.441598+0000","last_unstale":"2024-08-03T20:54:29.435238+0000","last_undegraded":"2024-08-03T20:54:29.435238+0000","last_fullsized":"2024-08-03T20:54:29.435238+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:54:26.410149+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:54:26.410149+0000","last_clean_scrub_stamp":"2024-08-03T20:54:26.410149+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T06:55:44.653623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.d","version":"82'1","reported_seq":16,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-08-03T20:54:29.482363+0000","last_change":"2024-08-03T20:54:26.443737+0000","last_active":"2024-08-03T20:54:29.482363+0000","last_peered":"2024-08-03T20:54:29.482363+0000","last_clean":"2024-08-03T20:54:29.482363+0000","last_became_active":"2024-08-03T20:54:26.443276+0000","last_became_peered":"2024-08-03T20:54:26.443276+0000","last_unstale":"2024-08-03T20:54:29.482363+0000","last_undegraded":"2024-08-03T20:54:29.482363+0000","last_fullsized":"2024-08-03T20:54:29.482363+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:54:25.402282+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:54:25.402282+0000","last_clean_scrub_stamp":"2024-08-03T20:54:25.402282+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T04:40:32.715596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"16'72","reported_seq":227,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-08-03T20:54:29.482297+0000","last_change":"2024-08-03T20:47:30.547632+0000","last_active":"2024-08-03T20:54:29.482297+0000","last_peered":"2024-08-03T20:54:29.482297+0000","last_clean":"2024-08-03T20:54:29.482297+0000","last_became_active":"2024-08-03T20:47:30.546255+0000","last_became_peered":"2024-08-03T20:47:30.546255+0000","last_unstale":"2024-08-03T20:54:29.482297+0000","last_undegraded":"2024-08-03T20:54:29.482297+0000","last_fullsized":"2024-08-03T20:54:29.482297+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:47:28.874451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:47:28.874451+0000","last_clean_scrub_stamp":"2024-08-03T20:47:28.874451+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T00:19:40.548812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":13,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":12,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509527,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32748,"kb_used_data":1560,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339092,"statfs":{"total":96636764160,"available":96603230208,"internally_reserved":0,"allocated":1597440,"data_stored":1016362,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":6,"up_from":15,"seq":64424509528,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32188,"kb_used_data":1000,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339652,"statfs":{"total":96636764160,"available":96603803648,"internally_reserved":0,"allocated":1024000,"data_stored":432918,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":5,"up_from":15,"seq":64424509528,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32176,"kb_used_data":992,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339664,"statfs":{"total":96636764160,"available":96603815936,"internally_reserved":0,"allocated":1015808,"data_stored":427558,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509527,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36332,"kb_used_data":988,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335508,"statfs":{"total":96636764160,"available":96599560192,"internally_reserved":0,"allocated":1011712,"data_stored":426118,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509529,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36908,"kb_used_data":1560,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334932,"statfs":{"total":96636764160,"available":96598970368,"internally_reserved":0,"allocated":1597440,"data_stored":1016326,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542232,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36328,"kb_used_data":988,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335512,"statfs":{"total":96636764160,"available":96599564288,"internally_reserved":0,"allocated":1011712,"data_stored":432224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":3,"up_from":14,"seq":60129542230,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32176,"kb_used_data":988,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339664,"statfs":{"total":96636764160,"available":96603815936,"internally_reserved":0,"allocated":1011712,"data_stored":427536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":2,"up_from":14,"seq":60129542231,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32168,"kb_used_data":980,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339672,"statfs":{"total":96636764160,"available":96603824128,"internally_reserved":0,"allocated":1003520,"data_stored":426506,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":194,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-08-03T20:54:31.948 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-08-03T20:54:32.211 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:32.209+0000 7f30ec400640 1 -- 172.21.15.32:0/1028143534 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f30e40b4ce0 msgr2=0x7f30e415ef00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:32.211 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:32.209+0000 7f30ec400640 1 --2- 172.21.15.32:0/1028143534 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f30e40b4ce0 0x7f30e415ef00 secure :-1 s=READY pgs=912 cs=0 l=1 rev1=1 crypto rx=0x7f30e40d1030 tx=0x7f30d002a560 comp rx=0 tx=0).stop 2024-08-03T20:54:32.211 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:32.209+0000 7f30ec400640 1 -- 172.21.15.32:0/1028143534 shutdown_connections 2024-08-03T20:54:32.212 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:32.209+0000 7f30ec400640 1 --2- 172.21.15.32:0/1028143534 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f30e415f440 0x7f30e415b910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:32.212 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:32.209+0000 7f30ec400640 1 --2- 172.21.15.32:0/1028143534 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f30e40b4ce0 0x7f30e415ef00 unknown :-1 s=CLOSED pgs=912 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:32.212 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:32.209+0000 7f30ec400640 1 --2- 172.21.15.32:0/1028143534 >> v2:172.21.15.118:3300/0 conn(0x7f30e40b38b0 0x7f30e41523f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:32.212 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:32.209+0000 7f30ec400640 1 -- 172.21.15.32:0/1028143534 >> 172.21.15.32:0/1028143534 conn(0x7f30e40b1bd0 msgr2=0x7f30e40b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:54:32.212 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:32.209+0000 7f30ec400640 1 -- 172.21.15.32:0/1028143534 shutdown_connections 2024-08-03T20:54:32.212 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:32.210+0000 7f30ec400640 1 -- 172.21.15.32:0/1028143534 wait complete. 2024-08-03T20:54:32.213 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:32.212+0000 7f30ec400640 1 Processor -- start 2024-08-03T20:54:32.213 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:32.212+0000 7f30ec400640 1 -- start start 2024-08-03T20:54:32.215 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:32.213+0000 7f30ec400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f30e40b38b0 0x7f30e4138980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:32.215 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:32.213+0000 7f30ec400640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f30e40b4ce0 0x7f30e413bb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:32.216 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:32.213+0000 7f30ec400640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f30e415f440 0x7f30e4073b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:32.216 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:32.213+0000 7f30ec400640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f30e415c1e0 con 0x7f30e40b38b0 2024-08-03T20:54:32.216 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:32.213+0000 7f30ec400640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f30e415c060 con 0x7f30e40b4ce0 2024-08-03T20:54:32.216 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:32.213+0000 7f30ec400640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7f30e415c360 con 0x7f30e415f440 2024-08-03T20:54:32.216 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:32.213+0000 7f30e9a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f30e40b38b0 0x7f30e4138980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:32.216 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:32.213+0000 7f30e9a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f30e40b38b0 0x7f30e4138980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:39984/0 (socket says 172.21.15.32:39984) 2024-08-03T20:54:32.216 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:32.214+0000 7f30e9a00640 1 -- 172.21.15.32:0/1010095134 learned_addr learned my addr 172.21.15.32:0/1010095134 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:54:32.216 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:32.214+0000 7f30ea400640 1 --2- 172.21.15.32:0/1010095134 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f30e415f440 0x7f30e4073b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:32.216 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:32.214+0000 7f30e9000640 1 --2- 172.21.15.32:0/1010095134 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f30e40b4ce0 0x7f30e413bb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:32.216 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:32.214+0000 7f30e9a00640 1 -- 172.21.15.32:0/1010095134 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f30e415f440 msgr2=0x7f30e4073b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:32.216 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:32.214+0000 7f30e9a00640 1 --2- 172.21.15.32:0/1010095134 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f30e415f440 0x7f30e4073b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:32.216 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:32.214+0000 7f30e9a00640 1 -- 172.21.15.32:0/1010095134 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f30e40b4ce0 msgr2=0x7f30e413bb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:32.216 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:32.214+0000 7f30e9a00640 1 --2- 172.21.15.32:0/1010095134 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f30e40b4ce0 0x7f30e413bb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:32.217 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:32.214+0000 7f30e9a00640 1 -- 172.21.15.32:0/1010095134 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f30e413c0d0 con 0x7f30e40b38b0 2024-08-03T20:54:32.217 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:32.214+0000 7f30e9a00640 1 --2- 172.21.15.32:0/1010095134 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f30e40b38b0 0x7f30e4138980 secure :-1 s=READY pgs=913 cs=0 l=1 rev1=1 crypto rx=0x7f30d800d750 tx=0x7f30d800dc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:54:32.217 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:32.215+0000 7f30e1600640 1 -- 172.21.15.32:0/1010095134 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30d8017070 con 0x7f30e40b38b0 2024-08-03T20:54:32.218 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:32.215+0000 7f30e1600640 1 -- 172.21.15.32:0/1010095134 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f30d8009cc0 con 0x7f30e40b38b0 2024-08-03T20:54:32.218 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:32.215+0000 7f30e1600640 1 -- 172.21.15.32:0/1010095134 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30d80106c0 con 0x7f30e40b38b0 2024-08-03T20:54:32.218 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:32.215+0000 7f30ec400640 1 -- 172.21.15.32:0/1010095134 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f30e4155630 con 0x7f30e40b38b0 2024-08-03T20:54:32.218 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:32.215+0000 7f30ec400640 1 -- 172.21.15.32:0/1010095134 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f30e4157b00 con 0x7f30e40b38b0 2024-08-03T20:54:32.218 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:32.216+0000 7f30ec400640 1 -- 172.21.15.32:0/1010095134 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f30e4159c20 con 0x7f30e40b38b0 2024-08-03T20:54:32.218 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:32.217+0000 7f30e1600640 1 -- 172.21.15.32:0/1010095134 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f30d801a070 con 0x7f30e40b38b0 2024-08-03T20:54:32.219 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:32.218+0000 7f30e1600640 1 --2- 172.21.15.32:0/1010095134 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f30c4077010 0x7f30c4079520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:32.219 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:32.218+0000 7f30e1600640 1 -- 172.21.15.32:0/1010095134 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 5709+0+0 (secure 0 0 0) 0x7f30d8098960 con 0x7f30e40b38b0 2024-08-03T20:54:32.220 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:32.218+0000 7f30e9000640 1 --2- 172.21.15.32:0/1010095134 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f30c4077010 0x7f30c4079520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:32.220 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:32.219+0000 7f30e9000640 1 --2- 172.21.15.32:0/1010095134 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f30c4077010 0x7f30c4079520 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f30d0033000 tx=0x7f30d0004730 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:54:32.224 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:32.222+0000 7f30e1600640 1 -- 172.21.15.32:0/1010095134 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7f30e4159c20 con 0x7f30e40b38b0 2024-08-03T20:54:32.385 INFO:tasks.ceph.mds.a.smithi032.stderr:2024-08-03T20:54:32.383+0000 7f7cfba00640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-08-03T20:54:32.425 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:32.423+0000 7f30ec400640 1 -- 172.21.15.32:0/1010095134 --> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f30e4157640 con 0x7f30c4077010 2024-08-03T20:54:32.430 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:32.428+0000 7f30e1600640 1 -- 172.21.15.32:0/1010095134 <== mgr.4106 v2:172.21.15.118:6800/4205950699 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+252067 (secure 0 0 0) 0x7f30e4157640 con 0x7f30c4077010 2024-08-03T20:54:32.434 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-03T20:54:32.435 INFO:teuthology.orchestra.run.smithi032.stderr:dumped all 2024-08-03T20:54:32.437 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:32.432+0000 7f30ec400640 1 -- 172.21.15.32:0/1010095134 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f30c4077010 msgr2=0x7f30c4079520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:32.437 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:32.432+0000 7f30ec400640 1 --2- 172.21.15.32:0/1010095134 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f30c4077010 0x7f30c4079520 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f30d0033000 tx=0x7f30d0004730 comp rx=0 tx=0).stop 2024-08-03T20:54:32.438 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:32.432+0000 7f30ec400640 1 -- 172.21.15.32:0/1010095134 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f30e40b38b0 msgr2=0x7f30e4138980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:32.438 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:32.432+0000 7f30ec400640 1 --2- 172.21.15.32:0/1010095134 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f30e40b38b0 0x7f30e4138980 secure :-1 s=READY pgs=913 cs=0 l=1 rev1=1 crypto rx=0x7f30d800d750 tx=0x7f30d800dc10 comp rx=0 tx=0).stop 2024-08-03T20:54:32.438 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:32.432+0000 7f30ec400640 1 -- 172.21.15.32:0/1010095134 shutdown_connections 2024-08-03T20:54:32.438 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:32.432+0000 7f30ec400640 1 --2- 172.21.15.32:0/1010095134 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f30c4077010 0x7f30c4079520 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:32.438 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:32.433+0000 7f30ec400640 1 --2- 172.21.15.32:0/1010095134 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f30e40b4ce0 0x7f30e413bb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:32.438 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:32.433+0000 7f30ec400640 1 --2- 172.21.15.32:0/1010095134 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f30e415f440 0x7f30e4073b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:32.438 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:32.433+0000 7f30ec400640 1 --2- 172.21.15.32:0/1010095134 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f30e40b38b0 0x7f30e4138980 unknown :-1 s=CLOSED pgs=913 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:32.438 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:32.433+0000 7f30ec400640 1 -- 172.21.15.32:0/1010095134 >> 172.21.15.32:0/1010095134 conn(0x7f30e40b1bd0 msgr2=0x7f30e4159510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:54:32.439 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:32.433+0000 7f30ec400640 1 -- 172.21.15.32:0/1010095134 shutdown_connections 2024-08-03T20:54:32.439 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:32.433+0000 7f30ec400640 1 -- 172.21.15.32:0/1010095134 wait complete. 2024-08-03T20:54:32.460 INFO:teuthology.orchestra.run.smithi032.stdout:{"pg_ready":true,"pg_map":{"version":288,"stamp":"2024-08-03T20:54:32.342375+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":146,"num_write_kb":1642,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":111,"ondisk_log_size":111,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":122,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":271024,"kb_used_data":9056,"kb_used_omap":12,"kb_used_meta":261811,"kb_avail":754703696,"statfs":{"total":773094113280,"available":772816584704,"internally_reserved":0,"allocated":9273344,"data_stored":4605548,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":268095025},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":24,"apply_latency_ms":24,"commit_latency_ns":24000000,"apply_latency_ns":24000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001709"},"pg_stats":[{"pgid":"13.3f","version":"0'0","reported_seq":13,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-08-03T20:54:28.440420+0000","last_change":"2024-08-03T20:54:27.441044+0000","last_active":"2024-08-03T20:54:28.440420+0000","last_peered":"2024-08-03T20:54:28.440420+0000","last_clean":"2024-08-03T20:54:28.440420+0000","last_became_active":"2024-08-03T20:54:27.440740+0000","last_became_peered":"2024-08-03T20:54:27.440740+0000","last_unstale":"2024-08-03T20:54:28.440420+0000","last_undegraded":"2024-08-03T20:54:28.440420+0000","last_fullsized":"2024-08-03T20:54:28.440420+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:54:26.410149+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:54:26.410149+0000","last_clean_scrub_stamp":"2024-08-03T20:54:26.410149+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T07:47:09.153912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.3e","version":"0'0","reported_seq":13,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-08-03T20:54:29.435334+0000","last_change":"2024-08-03T20:54:27.453508+0000","last_active":"2024-08-03T20:54:29.435334+0000","last_peered":"2024-08-03T20:54:29.435334+0000","last_clean":"2024-08-03T20:54:29.435334+0000","last_became_active":"2024-08-03T20:54:27.453015+0000","last_became_peered":"2024-08-03T20:54:27.453015+0000","last_unstale":"2024-08-03T20:54:29.435334+0000","last_undegraded":"2024-08-03T20:54:29.435334+0000","last_fullsized":"2024-08-03T20:54:29.435334+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:54:26.410149+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:54:26.410149+0000","last_clean_scrub_stamp":"2024-08-03T20:54:26.410149+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T01:06:28.734529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.3d","version":"0'0","reported_seq":13,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-08-03T20:54:29.482216+0000","last_change":"2024-08-03T20:54:27.445123+0000","last_active":"2024-08-03T20:54:29.482216+0000","last_peered":"2024-08-03T20:54:29.482216+0000","last_clean":"2024-08-03T20:54:29.482216+0000","last_became_active":"2024-08-03T20:54:27.444760+0000","last_became_peered":"2024-08-03T20:54:27.444760+0000","last_unstale":"2024-08-03T20:54:29.482216+0000","last_undegraded":"2024-08-03T20:54:29.482216+0000","last_fullsized":"2024-08-03T20:54:29.482216+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:54:26.410149+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:54:26.410149+0000","last_clean_scrub_stamp":"2024-08-03T20:54:26.410149+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-04T21:50:39.503703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.3c","version":"0'0","reported_seq":10,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-03T20:54:27.469370+0000","last_change":"2024-08-03T20:54:27.466461+0000","last_active":"2024-08-03T20:54:27.469370+0000","last_peered":"2024-08-03T20:54:27.469370+0000","last_clean":"2024-08-03T20:54:27.469370+0000","last_became_active":"2024-08-03T20:54:27.466141+0000","last_became_peered":"2024-08-03T20:54:27.466141+0000","last_unstale":"2024-08-03T20:54:27.469370+0000","last_undegraded":"2024-08-03T20:54:27.469370+0000","last_fullsized":"2024-08-03T20:54:27.469370+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:54:26.410149+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:54:26.410149+0000","last_clean_scrub_stamp":"2024-08-03T20:54:26.410149+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T00:31:51.313267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.3b","version":"0'0","reported_seq":13,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-08-03T20:54:28.441847+0000","last_change":"2024-08-03T20:54:27.448707+0000","last_active":"2024-08-03T20:54:28.441847+0000","last_peered":"2024-08-03T20:54:28.441847+0000","last_clean":"2024-08-03T20:54:28.441847+0000","last_became_active":"2024-08-03T20:54:27.444927+0000","last_became_peered":"2024-08-03T20:54:27.444927+0000","last_unstale":"2024-08-03T20:54:28.441847+0000","last_undegraded":"2024-08-03T20:54:28.441847+0000","last_fullsized":"2024-08-03T20:54:28.441847+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:54:26.410149+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:54:26.410149+0000","last_clean_scrub_stamp":"2024-08-03T20:54:26.410149+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-04T21:58:31.396236+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.3a","version":"0'0","reported_seq":13,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-08-03T20:54:29.323592+0000","last_change":"2024-08-03T20:54:27.451021+0000","last_active":"2024-08-03T20:54:29.323592+0000","last_peered":"2024-08-03T20:54:29.323592+0000","last_clean":"2024-08-03T20:54:29.323592+0000","last_became_active":"2024-08-03T20:54:27.450613+0000","last_became_peered":"2024-08-03T20:54:27.450613+0000","last_unstale":"2024-08-03T20:54:29.323592+0000","last_undegraded":"2024-08-03T20:54:29.323592+0000","last_fullsized":"2024-08-03T20:54:29.323592+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:54:26.410149+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:54:26.410149+0000","last_clean_scrub_stamp":"2024-08-03T20:54:26.410149+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-04T21:11:37.122543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.39","version":"0'0","reported_seq":10,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-03T20:54:27.464346+0000","last_change":"2024-08-03T20:54:27.453127+0000","last_active":"2024-08-03T20:54:27.464346+0000","last_peered":"2024-08-03T20:54:27.464346+0000","last_clean":"2024-08-03T20:54:27.464346+0000","last_became_active":"2024-08-03T20:54:27.450479+0000","last_became_peered":"2024-08-03T20:54:27.450479+0000","last_unstale":"2024-08-03T20:54:27.464346+0000","last_undegraded":"2024-08-03T20:54:27.464346+0000","last_fullsized":"2024-08-03T20:54:27.464346+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:54:26.410149+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:54:26.410149+0000","last_clean_scrub_stamp":"2024-08-03T20:54:26.410149+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T07:39:14.606174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.38","version":"0'0","reported_seq":13,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-08-03T20:54:29.323693+0000","last_change":"2024-08-03T20:54:27.464523+0000","last_active":"2024-08-03T20:54:29.323693+0000","last_peered":"2024-08-03T20:54:29.323693+0000","last_clean":"2024-08-03T20:54:29.323693+0000","last_became_active":"2024-08-03T20:54:27.464250+0000","last_became_peered":"2024-08-03T20:54:27.464250+0000","last_unstale":"2024-08-03T20:54:29.323693+0000","last_undegraded":"2024-08-03T20:54:29.323693+0000","last_fullsized":"2024-08-03T20:54:29.323693+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:54:26.410149+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:54:26.410149+0000","last_clean_scrub_stamp":"2024-08-03T20:54:26.410149+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-04T23:30:55.227524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.37","version":"0'0","reported_seq":13,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-08-03T20:54:29.323867+0000","last_change":"2024-08-03T20:54:27.448082+0000","last_active":"2024-08-03T20:54:29.323867+0000","last_peered":"2024-08-03T20:54:29.323867+0000","last_clean":"2024-08-03T20:54:29.323867+0000","last_became_active":"2024-08-03T20:54:27.447546+0000","last_became_peered":"2024-08-03T20:54:27.447546+0000","last_unstale":"2024-08-03T20:54:29.323867+0000","last_undegraded":"2024-08-03T20:54:29.323867+0000","last_fullsized":"2024-08-03T20:54:29.323867+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:54:26.410149+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:54:26.410149+0000","last_clean_scrub_stamp":"2024-08-03T20:54:26.410149+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T07:47:09.153912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.36","version":"0'0","reported_seq":13,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-08-03T20:54:28.440301+0000","last_change":"2024-08-03T20:54:27.445637+0000","last_active":"2024-08-03T20:54:28.440301+0000","last_peered":"2024-08-03T20:54:28.440301+0000","last_clean":"2024-08-03T20:54:28.440301+0000","last_became_active":"2024-08-03T20:54:27.445278+0000","last_became_peered":"2024-08-03T20:54:27.445278+0000","last_unstale":"2024-08-03T20:54:28.440301+0000","last_undegraded":"2024-08-03T20:54:28.440301+0000","last_fullsized":"2024-08-03T20:54:28.440301+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:54:26.410149+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:54:26.410149+0000","last_clean_scrub_stamp":"2024-08-03T20:54:26.410149+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T05:28:20.584840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.35","version":"0'0","reported_seq":13,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-08-03T20:54:29.435217+0000","last_change":"2024-08-03T20:54:27.454353+0000","last_active":"2024-08-03T20:54:29.435217+0000","last_peered":"2024-08-03T20:54:29.435217+0000","last_clean":"2024-08-03T20:54:29.435217+0000","last_became_active":"2024-08-03T20:54:27.454049+0000","last_became_peered":"2024-08-03T20:54:27.454049+0000","last_unstale":"2024-08-03T20:54:29.435217+0000","last_undegraded":"2024-08-03T20:54:29.435217+0000","last_fullsized":"2024-08-03T20:54:29.435217+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:54:26.410149+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:54:26.410149+0000","last_clean_scrub_stamp":"2024-08-03T20:54:26.410149+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-04T22:37:20.127782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.34","version":"0'0","reported_seq":13,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-08-03T20:54:28.440323+0000","last_change":"2024-08-03T20:54:27.444784+0000","last_active":"2024-08-03T20:54:28.440323+0000","last_peered":"2024-08-03T20:54:28.440323+0000","last_clean":"2024-08-03T20:54:28.440323+0000","last_became_active":"2024-08-03T20:54:27.444485+0000","last_became_peered":"2024-08-03T20:54:27.444485+0000","last_unstale":"2024-08-03T20:54:28.440323+0000","last_undegraded":"2024-08-03T20:54:28.440323+0000","last_fullsized":"2024-08-03T20:54:28.440323+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:54:26.410149+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:54:26.410149+0000","last_clean_scrub_stamp":"2024-08-03T20:54:26.410149+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T03:14:09.919834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.33","version":"0'0","reported_seq":13,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-08-03T20:54:29.482081+0000","last_change":"2024-08-03T20:54:27.439898+0000","last_active":"2024-08-03T20:54:29.482081+0000","last_peered":"2024-08-03T20:54:29.482081+0000","last_clean":"2024-08-03T20:54:29.482081+0000","last_became_active":"2024-08-03T20:54:27.439239+0000","last_became_peered":"2024-08-03T20:54:27.439239+0000","last_unstale":"2024-08-03T20:54:29.482081+0000","last_undegraded":"2024-08-03T20:54:29.482081+0000","last_fullsized":"2024-08-03T20:54:29.482081+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:54:26.410149+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:54:26.410149+0000","last_clean_scrub_stamp":"2024-08-03T20:54:26.410149+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T07:09:56.880905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.32","version":"0'0","reported_seq":10,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-03T20:54:27.465996+0000","last_change":"2024-08-03T20:54:27.460892+0000","last_active":"2024-08-03T20:54:27.465996+0000","last_peered":"2024-08-03T20:54:27.465996+0000","last_clean":"2024-08-03T20:54:27.465996+0000","last_became_active":"2024-08-03T20:54:27.460176+0000","last_became_peered":"2024-08-03T20:54:27.460176+0000","last_unstale":"2024-08-03T20:54:27.465996+0000","last_undegraded":"2024-08-03T20:54:27.465996+0000","last_fullsized":"2024-08-03T20:54:27.465996+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:54:26.410149+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:54:26.410149+0000","last_clean_scrub_stamp":"2024-08-03T20:54:26.410149+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T04:49:41.930226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.31","version":"0'0","reported_seq":10,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-03T20:54:27.446546+0000","last_change":"2024-08-03T20:54:27.444614+0000","last_active":"2024-08-03T20:54:27.446546+0000","last_peered":"2024-08-03T20:54:27.446546+0000","last_clean":"2024-08-03T20:54:27.446546+0000","last_became_active":"2024-08-03T20:54:27.443574+0000","last_became_peered":"2024-08-03T20:54:27.443574+0000","last_unstale":"2024-08-03T20:54:27.446546+0000","last_undegraded":"2024-08-03T20:54:27.446546+0000","last_fullsized":"2024-08-03T20:54:27.446546+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:54:26.410149+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:54:26.410149+0000","last_clean_scrub_stamp":"2024-08-03T20:54:26.410149+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T03:52:35.096710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.30","version":"0'0","reported_seq":13,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-08-03T20:54:28.441211+0000","last_change":"2024-08-03T20:54:27.435703+0000","last_active":"2024-08-03T20:54:28.441211+0000","last_peered":"2024-08-03T20:54:28.441211+0000","last_clean":"2024-08-03T20:54:28.441211+0000","last_became_active":"2024-08-03T20:54:27.434882+0000","last_became_peered":"2024-08-03T20:54:27.434882+0000","last_unstale":"2024-08-03T20:54:28.441211+0000","last_undegraded":"2024-08-03T20:54:28.441211+0000","last_fullsized":"2024-08-03T20:54:28.441211+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:54:26.410149+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:54:26.410149+0000","last_clean_scrub_stamp":"2024-08-03T20:54:26.410149+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T05:44:25.324689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.2f","version":"0'0","reported_seq":13,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-08-03T20:54:28.442148+0000","last_change":"2024-08-03T20:54:27.452617+0000","last_active":"2024-08-03T20:54:28.442148+0000","last_peered":"2024-08-03T20:54:28.442148+0000","last_clean":"2024-08-03T20:54:28.442148+0000","last_became_active":"2024-08-03T20:54:27.452278+0000","last_became_peered":"2024-08-03T20:54:27.452278+0000","last_unstale":"2024-08-03T20:54:28.442148+0000","last_undegraded":"2024-08-03T20:54:28.442148+0000","last_fullsized":"2024-08-03T20:54:28.442148+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:54:26.410149+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:54:26.410149+0000","last_clean_scrub_stamp":"2024-08-03T20:54:26.410149+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T08:45:22.391494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.2e","version":"0'0","reported_seq":10,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-03T20:54:27.466462+0000","last_change":"2024-08-03T20:54:27.462672+0000","last_active":"2024-08-03T20:54:27.466462+0000","last_peered":"2024-08-03T20:54:27.466462+0000","last_clean":"2024-08-03T20:54:27.466462+0000","last_became_active":"2024-08-03T20:54:27.462357+0000","last_became_peered":"2024-08-03T20:54:27.462357+0000","last_unstale":"2024-08-03T20:54:27.466462+0000","last_undegraded":"2024-08-03T20:54:27.466462+0000","last_fullsized":"2024-08-03T20:54:27.466462+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:54:26.410149+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:54:26.410149+0000","last_clean_scrub_stamp":"2024-08-03T20:54:26.410149+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T05:48:22.964700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.2d","version":"0'0","reported_seq":13,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-08-03T20:54:29.438574+0000","last_change":"2024-08-03T20:54:27.439122+0000","last_active":"2024-08-03T20:54:29.438574+0000","last_peered":"2024-08-03T20:54:29.438574+0000","last_clean":"2024-08-03T20:54:29.438574+0000","last_became_active":"2024-08-03T20:54:27.438765+0000","last_became_peered":"2024-08-03T20:54:27.438765+0000","last_unstale":"2024-08-03T20:54:29.438574+0000","last_undegraded":"2024-08-03T20:54:29.438574+0000","last_fullsized":"2024-08-03T20:54:29.438574+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:54:26.410149+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:54:26.410149+0000","last_clean_scrub_stamp":"2024-08-03T20:54:26.410149+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T07:47:09.153912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.2c","version":"0'0","reported_seq":10,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-03T20:54:27.469346+0000","last_change":"2024-08-03T20:54:27.463774+0000","last_active":"2024-08-03T20:54:27.469346+0000","last_peered":"2024-08-03T20:54:27.469346+0000","last_clean":"2024-08-03T20:54:27.469346+0000","last_became_active":"2024-08-03T20:54:27.458783+0000","last_became_peered":"2024-08-03T20:54:27.458783+0000","last_unstale":"2024-08-03T20:54:27.469346+0000","last_undegraded":"2024-08-03T20:54:27.469346+0000","last_fullsized":"2024-08-03T20:54:27.469346+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:54:26.410149+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:54:26.410149+0000","last_clean_scrub_stamp":"2024-08-03T20:54:26.410149+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T07:00:07.332926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.2b","version":"0'0","reported_seq":10,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-03T20:54:27.468079+0000","last_change":"2024-08-03T20:54:27.464347+0000","last_active":"2024-08-03T20:54:27.468079+0000","last_peered":"2024-08-03T20:54:27.468079+0000","last_clean":"2024-08-03T20:54:27.468079+0000","last_became_active":"2024-08-03T20:54:27.464032+0000","last_became_peered":"2024-08-03T20:54:27.464032+0000","last_unstale":"2024-08-03T20:54:27.468079+0000","last_undegraded":"2024-08-03T20:54:27.468079+0000","last_fullsized":"2024-08-03T20:54:27.468079+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:54:26.410149+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:54:26.410149+0000","last_clean_scrub_stamp":"2024-08-03T20:54:26.410149+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T04:36:09.850241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.2a","version":"0'0","reported_seq":13,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-08-03T20:54:29.434917+0000","last_change":"2024-08-03T20:54:27.455588+0000","last_active":"2024-08-03T20:54:29.434917+0000","last_peered":"2024-08-03T20:54:29.434917+0000","last_clean":"2024-08-03T20:54:29.434917+0000","last_became_active":"2024-08-03T20:54:27.455262+0000","last_became_peered":"2024-08-03T20:54:27.455262+0000","last_unstale":"2024-08-03T20:54:29.434917+0000","last_undegraded":"2024-08-03T20:54:29.434917+0000","last_fullsized":"2024-08-03T20:54:29.434917+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:54:26.410149+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:54:26.410149+0000","last_clean_scrub_stamp":"2024-08-03T20:54:26.410149+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-04T21:51:05.362465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.29","version":"0'0","reported_seq":13,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-08-03T20:54:28.439669+0000","last_change":"2024-08-03T20:54:27.454311+0000","last_active":"2024-08-03T20:54:28.439669+0000","last_peered":"2024-08-03T20:54:28.439669+0000","last_clean":"2024-08-03T20:54:28.439669+0000","last_became_active":"2024-08-03T20:54:27.453739+0000","last_became_peered":"2024-08-03T20:54:27.453739+0000","last_unstale":"2024-08-03T20:54:28.439669+0000","last_undegraded":"2024-08-03T20:54:28.439669+0000","last_fullsized":"2024-08-03T20:54:28.439669+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:54:26.410149+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:54:26.410149+0000","last_clean_scrub_stamp":"2024-08-03T20:54:26.410149+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-04T21:11:37.122543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.28","version":"0'0","reported_seq":10,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-03T20:54:27.468314+0000","last_change":"2024-08-03T20:54:27.455361+0000","last_active":"2024-08-03T20:54:27.468314+0000","last_peered":"2024-08-03T20:54:27.468314+0000","last_clean":"2024-08-03T20:54:27.468314+0000","last_became_active":"2024-08-03T20:54:27.455033+0000","last_became_peered":"2024-08-03T20:54:27.455033+0000","last_unstale":"2024-08-03T20:54:27.468314+0000","last_undegraded":"2024-08-03T20:54:27.468314+0000","last_fullsized":"2024-08-03T20:54:27.468314+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:54:26.410149+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:54:26.410149+0000","last_clean_scrub_stamp":"2024-08-03T20:54:26.410149+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T08:07:54.404129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.27","version":"0'0","reported_seq":13,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-08-03T20:54:29.435298+0000","last_change":"2024-08-03T20:54:27.455147+0000","last_active":"2024-08-03T20:54:29.435298+0000","last_peered":"2024-08-03T20:54:29.435298+0000","last_clean":"2024-08-03T20:54:29.435298+0000","last_became_active":"2024-08-03T20:54:27.448048+0000","last_became_peered":"2024-08-03T20:54:27.448048+0000","last_unstale":"2024-08-03T20:54:29.435298+0000","last_undegraded":"2024-08-03T20:54:29.435298+0000","last_fullsized":"2024-08-03T20:54:29.435298+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:54:26.410149+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:54:26.410149+0000","last_clean_scrub_stamp":"2024-08-03T20:54:26.410149+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T08:02:43.988492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.26","version":"0'0","reported_seq":13,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-08-03T20:54:29.482089+0000","last_change":"2024-08-03T20:54:27.445865+0000","last_active":"2024-08-03T20:54:29.482089+0000","last_peered":"2024-08-03T20:54:29.482089+0000","last_clean":"2024-08-03T20:54:29.482089+0000","last_became_active":"2024-08-03T20:54:27.445574+0000","last_became_peered":"2024-08-03T20:54:27.445574+0000","last_unstale":"2024-08-03T20:54:29.482089+0000","last_undegraded":"2024-08-03T20:54:29.482089+0000","last_fullsized":"2024-08-03T20:54:29.482089+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:54:26.410149+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:54:26.410149+0000","last_clean_scrub_stamp":"2024-08-03T20:54:26.410149+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T01:43:40.620191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.25","version":"0'0","reported_seq":13,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-08-03T20:54:28.441322+0000","last_change":"2024-08-03T20:54:27.436769+0000","last_active":"2024-08-03T20:54:28.441322+0000","last_peered":"2024-08-03T20:54:28.441322+0000","last_clean":"2024-08-03T20:54:28.441322+0000","last_became_active":"2024-08-03T20:54:27.436388+0000","last_became_peered":"2024-08-03T20:54:27.436388+0000","last_unstale":"2024-08-03T20:54:28.441322+0000","last_undegraded":"2024-08-03T20:54:28.441322+0000","last_fullsized":"2024-08-03T20:54:28.441322+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:54:26.410149+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:54:26.410149+0000","last_clean_scrub_stamp":"2024-08-03T20:54:26.410149+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T07:33:46.448288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.24","version":"0'0","reported_seq":13,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-08-03T20:54:28.440017+0000","last_change":"2024-08-03T20:54:27.442047+0000","last_active":"2024-08-03T20:54:28.440017+0000","last_peered":"2024-08-03T20:54:28.440017+0000","last_clean":"2024-08-03T20:54:28.440017+0000","last_became_active":"2024-08-03T20:54:27.441434+0000","last_became_peered":"2024-08-03T20:54:27.441434+0000","last_unstale":"2024-08-03T20:54:28.440017+0000","last_undegraded":"2024-08-03T20:54:28.440017+0000","last_fullsized":"2024-08-03T20:54:28.440017+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:54:26.410149+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:54:26.410149+0000","last_clean_scrub_stamp":"2024-08-03T20:54:26.410149+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T06:59:28.260517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.23","version":"0'0","reported_seq":13,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-08-03T20:54:29.434977+0000","last_change":"2024-08-03T20:54:27.453761+0000","last_active":"2024-08-03T20:54:29.434977+0000","last_peered":"2024-08-03T20:54:29.434977+0000","last_clean":"2024-08-03T20:54:29.434977+0000","last_became_active":"2024-08-03T20:54:27.453459+0000","last_became_peered":"2024-08-03T20:54:27.453459+0000","last_unstale":"2024-08-03T20:54:29.434977+0000","last_undegraded":"2024-08-03T20:54:29.434977+0000","last_fullsized":"2024-08-03T20:54:29.434977+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:54:26.410149+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:54:26.410149+0000","last_clean_scrub_stamp":"2024-08-03T20:54:26.410149+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T03:28:35.668870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.22","version":"0'0","reported_seq":13,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-08-03T20:54:29.435035+0000","last_change":"2024-08-03T20:54:27.457503+0000","last_active":"2024-08-03T20:54:29.435035+0000","last_peered":"2024-08-03T20:54:29.435035+0000","last_clean":"2024-08-03T20:54:29.435035+0000","last_became_active":"2024-08-03T20:54:27.456783+0000","last_became_peered":"2024-08-03T20:54:27.456783+0000","last_unstale":"2024-08-03T20:54:29.435035+0000","last_undegraded":"2024-08-03T20:54:29.435035+0000","last_fullsized":"2024-08-03T20:54:29.435035+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:54:26.410149+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:54:26.410149+0000","last_clean_scrub_stamp":"2024-08-03T20:54:26.410149+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-04T23:52:57.831797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.21","version":"0'0","reported_seq":13,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-08-03T20:54:28.441236+0000","last_change":"2024-08-03T20:54:27.448194+0000","last_active":"2024-08-03T20:54:28.441236+0000","last_peered":"2024-08-03T20:54:28.441236+0000","last_clean":"2024-08-03T20:54:28.441236+0000","last_became_active":"2024-08-03T20:54:27.447664+0000","last_became_peered":"2024-08-03T20:54:27.447664+0000","last_unstale":"2024-08-03T20:54:28.441236+0000","last_undegraded":"2024-08-03T20:54:28.441236+0000","last_fullsized":"2024-08-03T20:54:28.441236+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:54:26.410149+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:54:26.410149+0000","last_clean_scrub_stamp":"2024-08-03T20:54:26.410149+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T00:18:08.383331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.20","version":"0'0","reported_seq":13,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-08-03T20:54:28.439602+0000","last_change":"2024-08-03T20:54:27.466029+0000","last_active":"2024-08-03T20:54:28.439602+0000","last_peered":"2024-08-03T20:54:28.439602+0000","last_clean":"2024-08-03T20:54:28.439602+0000","last_became_active":"2024-08-03T20:54:27.465689+0000","last_became_peered":"2024-08-03T20:54:27.465689+0000","last_unstale":"2024-08-03T20:54:28.439602+0000","last_undegraded":"2024-08-03T20:54:28.439602+0000","last_fullsized":"2024-08-03T20:54:28.439602+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:54:26.410149+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:54:26.410149+0000","last_clean_scrub_stamp":"2024-08-03T20:54:26.410149+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-04T21:41:30.478055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1e","version":"0'0","reported_seq":13,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-08-03T20:54:28.441943+0000","last_change":"2024-08-03T20:54:27.446192+0000","last_active":"2024-08-03T20:54:28.441943+0000","last_peered":"2024-08-03T20:54:28.441943+0000","last_clean":"2024-08-03T20:54:28.441943+0000","last_became_active":"2024-08-03T20:54:27.445882+0000","last_became_peered":"2024-08-03T20:54:27.445882+0000","last_unstale":"2024-08-03T20:54:28.441943+0000","last_undegraded":"2024-08-03T20:54:28.441943+0000","last_fullsized":"2024-08-03T20:54:28.441943+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:54:26.410149+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:54:26.410149+0000","last_clean_scrub_stamp":"2024-08-03T20:54:26.410149+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T05:39:36.328590+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.1f","version":"82'4","reported_seq":19,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-08-03T20:54:28.440761+0000","last_change":"2024-08-03T20:54:26.444466+0000","last_active":"2024-08-03T20:54:28.440761+0000","last_peered":"2024-08-03T20:54:28.440761+0000","last_clean":"2024-08-03T20:54:28.440761+0000","last_became_active":"2024-08-03T20:54:26.444145+0000","last_became_peered":"2024-08-03T20:54:26.444145+0000","last_unstale":"2024-08-03T20:54:28.440761+0000","last_undegraded":"2024-08-03T20:54:28.440761+0000","last_fullsized":"2024-08-03T20:54:28.440761+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:54:25.402282+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:54:25.402282+0000","last_clean_scrub_stamp":"2024-08-03T20:54:25.402282+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-04T22:37:19.119915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1f","version":"0'0","reported_seq":13,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-08-03T20:54:28.440635+0000","last_change":"2024-08-03T20:54:27.455848+0000","last_active":"2024-08-03T20:54:28.440635+0000","last_peered":"2024-08-03T20:54:28.440635+0000","last_clean":"2024-08-03T20:54:28.440635+0000","last_became_active":"2024-08-03T20:54:27.455487+0000","last_became_peered":"2024-08-03T20:54:27.455487+0000","last_unstale":"2024-08-03T20:54:28.440635+0000","last_undegraded":"2024-08-03T20:54:28.440635+0000","last_fullsized":"2024-08-03T20:54:28.440635+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:54:26.410149+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:54:26.410149+0000","last_clean_scrub_stamp":"2024-08-03T20:54:26.410149+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T01:43:40.620191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.1e","version":"82'1","reported_seq":16,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-08-03T20:54:28.441990+0000","last_change":"2024-08-03T20:54:26.439724+0000","last_active":"2024-08-03T20:54:28.441990+0000","last_peered":"2024-08-03T20:54:28.441990+0000","last_clean":"2024-08-03T20:54:28.441990+0000","last_became_active":"2024-08-03T20:54:26.438773+0000","last_became_peered":"2024-08-03T20:54:26.438773+0000","last_unstale":"2024-08-03T20:54:28.441990+0000","last_undegraded":"2024-08-03T20:54:28.441990+0000","last_fullsized":"2024-08-03T20:54:28.441990+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:54:25.402282+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:54:25.402282+0000","last_clean_scrub_stamp":"2024-08-03T20:54:25.402282+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T08:35:58.055884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.1c","version":"0'0","reported_seq":13,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-08-03T20:54:28.440241+0000","last_change":"2024-08-03T20:54:27.444247+0000","last_active":"2024-08-03T20:54:28.440241+0000","last_peered":"2024-08-03T20:54:28.440241+0000","last_clean":"2024-08-03T20:54:28.440241+0000","last_became_active":"2024-08-03T20:54:27.443893+0000","last_became_peered":"2024-08-03T20:54:27.443893+0000","last_unstale":"2024-08-03T20:54:28.440241+0000","last_undegraded":"2024-08-03T20:54:28.440241+0000","last_fullsized":"2024-08-03T20:54:28.440241+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:54:26.410149+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:54:26.410149+0000","last_clean_scrub_stamp":"2024-08-03T20:54:26.410149+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-04T23:30:55.227524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.1d","version":"82'2","reported_seq":17,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-08-03T20:54:29.438574+0000","last_change":"2024-08-03T20:54:26.444870+0000","last_active":"2024-08-03T20:54:29.438574+0000","last_peered":"2024-08-03T20:54:29.438574+0000","last_clean":"2024-08-03T20:54:29.438574+0000","last_became_active":"2024-08-03T20:54:26.442179+0000","last_became_peered":"2024-08-03T20:54:26.442179+0000","last_unstale":"2024-08-03T20:54:29.438574+0000","last_undegraded":"2024-08-03T20:54:29.438574+0000","last_fullsized":"2024-08-03T20:54:29.438574+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:54:25.402282+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:54:25.402282+0000","last_clean_scrub_stamp":"2024-08-03T20:54:25.402282+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T04:40:32.715596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1d","version":"0'0","reported_seq":13,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-08-03T20:54:29.482438+0000","last_change":"2024-08-03T20:54:27.461330+0000","last_active":"2024-08-03T20:54:29.482438+0000","last_peered":"2024-08-03T20:54:29.482438+0000","last_clean":"2024-08-03T20:54:29.482438+0000","last_became_active":"2024-08-03T20:54:27.461060+0000","last_became_peered":"2024-08-03T20:54:27.461060+0000","last_unstale":"2024-08-03T20:54:29.482438+0000","last_undegraded":"2024-08-03T20:54:29.482438+0000","last_fullsized":"2024-08-03T20:54:29.482438+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:54:26.410149+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:54:26.410149+0000","last_clean_scrub_stamp":"2024-08-03T20:54:26.410149+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T05:41:54.880022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.1c","version":"82'2","reported_seq":17,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-08-03T20:54:29.438463+0000","last_change":"2024-08-03T20:54:26.445095+0000","last_active":"2024-08-03T20:54:29.438463+0000","last_peered":"2024-08-03T20:54:29.438463+0000","last_clean":"2024-08-03T20:54:29.438463+0000","last_became_active":"2024-08-03T20:54:26.441622+0000","last_became_peered":"2024-08-03T20:54:26.441622+0000","last_unstale":"2024-08-03T20:54:29.438463+0000","last_undegraded":"2024-08-03T20:54:29.438463+0000","last_fullsized":"2024-08-03T20:54:29.438463+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:54:25.402282+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:54:25.402282+0000","last_clean_scrub_stamp":"2024-08-03T20:54:25.402282+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T01:00:43.079889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1a","version":"0'0","reported_seq":13,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-08-03T20:54:29.435327+0000","last_change":"2024-08-03T20:54:27.457761+0000","last_active":"2024-08-03T20:54:29.435327+0000","last_peered":"2024-08-03T20:54:29.435327+0000","last_clean":"2024-08-03T20:54:29.435327+0000","last_became_active":"2024-08-03T20:54:27.457196+0000","last_became_peered":"2024-08-03T20:54:27.457196+0000","last_unstale":"2024-08-03T20:54:29.435327+0000","last_undegraded":"2024-08-03T20:54:29.435327+0000","last_fullsized":"2024-08-03T20:54:29.435327+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:54:26.410149+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:54:26.410149+0000","last_clean_scrub_stamp":"2024-08-03T20:54:26.410149+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-04T21:02:35.798301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.1b","version":"82'1","reported_seq":16,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-08-03T20:54:28.442165+0000","last_change":"2024-08-03T20:54:26.440177+0000","last_active":"2024-08-03T20:54:28.442165+0000","last_peered":"2024-08-03T20:54:28.442165+0000","last_clean":"2024-08-03T20:54:28.442165+0000","last_became_active":"2024-08-03T20:54:26.439861+0000","last_became_peered":"2024-08-03T20:54:26.439861+0000","last_unstale":"2024-08-03T20:54:28.442165+0000","last_undegraded":"2024-08-03T20:54:28.442165+0000","last_fullsized":"2024-08-03T20:54:28.442165+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:54:25.402282+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:54:25.402282+0000","last_clean_scrub_stamp":"2024-08-03T20:54:25.402282+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-04T23:38:59.029460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.1b","version":"0'0","reported_seq":13,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-08-03T20:54:28.441992+0000","last_change":"2024-08-03T20:54:27.457638+0000","last_active":"2024-08-03T20:54:28.441992+0000","last_peered":"2024-08-03T20:54:28.441992+0000","last_clean":"2024-08-03T20:54:28.441992+0000","last_became_active":"2024-08-03T20:54:27.457210+0000","last_became_peered":"2024-08-03T20:54:27.457210+0000","last_unstale":"2024-08-03T20:54:28.441992+0000","last_undegraded":"2024-08-03T20:54:28.441992+0000","last_fullsized":"2024-08-03T20:54:28.441992+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:54:26.410149+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:54:26.410149+0000","last_clean_scrub_stamp":"2024-08-03T20:54:26.410149+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T01:19:10.278598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.1a","version":"82'1","reported_seq":16,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-08-03T20:54:28.441433+0000","last_change":"2024-08-03T20:54:26.430929+0000","last_active":"2024-08-03T20:54:28.441433+0000","last_peered":"2024-08-03T20:54:28.441433+0000","last_clean":"2024-08-03T20:54:28.441433+0000","last_became_active":"2024-08-03T20:54:26.430532+0000","last_became_peered":"2024-08-03T20:54:26.430532+0000","last_unstale":"2024-08-03T20:54:28.441433+0000","last_undegraded":"2024-08-03T20:54:28.441433+0000","last_fullsized":"2024-08-03T20:54:28.441433+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:54:25.402282+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:54:25.402282+0000","last_clean_scrub_stamp":"2024-08-03T20:54:25.402282+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T03:43:43.509403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.18","version":"0'0","reported_seq":13,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-08-03T20:54:29.482228+0000","last_change":"2024-08-03T20:54:27.454767+0000","last_active":"2024-08-03T20:54:29.482228+0000","last_peered":"2024-08-03T20:54:29.482228+0000","last_clean":"2024-08-03T20:54:29.482228+0000","last_became_active":"2024-08-03T20:54:27.454333+0000","last_became_peered":"2024-08-03T20:54:27.454333+0000","last_unstale":"2024-08-03T20:54:29.482228+0000","last_undegraded":"2024-08-03T20:54:29.482228+0000","last_fullsized":"2024-08-03T20:54:29.482228+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:54:26.410149+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:54:26.410149+0000","last_clean_scrub_stamp":"2024-08-03T20:54:26.410149+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T02:40:19.572527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.19","version":"0'0","reported_seq":15,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-08-03T20:54:29.438118+0000","last_change":"2024-08-03T20:54:26.437810+0000","last_active":"2024-08-03T20:54:29.438118+0000","last_peered":"2024-08-03T20:54:29.438118+0000","last_clean":"2024-08-03T20:54:29.438118+0000","last_became_active":"2024-08-03T20:54:26.437340+0000","last_became_peered":"2024-08-03T20:54:26.437340+0000","last_unstale":"2024-08-03T20:54:29.438118+0000","last_undegraded":"2024-08-03T20:54:29.438118+0000","last_fullsized":"2024-08-03T20:54:29.438118+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:54:25.402282+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:54:25.402282+0000","last_clean_scrub_stamp":"2024-08-03T20:54:25.402282+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T05:48:41.687726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.19","version":"0'0","reported_seq":10,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-03T20:54:27.442546+0000","last_change":"2024-08-03T20:54:27.440130+0000","last_active":"2024-08-03T20:54:27.442546+0000","last_peered":"2024-08-03T20:54:27.442546+0000","last_clean":"2024-08-03T20:54:27.442546+0000","last_became_active":"2024-08-03T20:54:27.439791+0000","last_became_peered":"2024-08-03T20:54:27.439791+0000","last_unstale":"2024-08-03T20:54:27.442546+0000","last_undegraded":"2024-08-03T20:54:27.442546+0000","last_fullsized":"2024-08-03T20:54:27.442546+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:54:26.410149+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:54:26.410149+0000","last_clean_scrub_stamp":"2024-08-03T20:54:26.410149+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T02:14:15.424140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.18","version":"0'0","reported_seq":15,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-08-03T20:54:29.434815+0000","last_change":"2024-08-03T20:54:26.438649+0000","last_active":"2024-08-03T20:54:29.434815+0000","last_peered":"2024-08-03T20:54:29.434815+0000","last_clean":"2024-08-03T20:54:29.434815+0000","last_became_active":"2024-08-03T20:54:26.438312+0000","last_became_peered":"2024-08-03T20:54:26.438312+0000","last_unstale":"2024-08-03T20:54:29.434815+0000","last_undegraded":"2024-08-03T20:54:29.434815+0000","last_fullsized":"2024-08-03T20:54:29.434815+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:54:25.402282+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:54:25.402282+0000","last_clean_scrub_stamp":"2024-08-03T20:54:25.402282+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T04:32:12.785658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.16","version":"0'0","reported_seq":10,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-03T20:54:27.447146+0000","last_change":"2024-08-03T20:54:27.443533+0000","last_active":"2024-08-03T20:54:27.447146+0000","last_peered":"2024-08-03T20:54:27.447146+0000","last_clean":"2024-08-03T20:54:27.447146+0000","last_became_active":"2024-08-03T20:54:27.443050+0000","last_became_peered":"2024-08-03T20:54:27.443050+0000","last_unstale":"2024-08-03T20:54:27.447146+0000","last_undegraded":"2024-08-03T20:54:27.447146+0000","last_fullsized":"2024-08-03T20:54:27.447146+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:54:26.410149+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:54:26.410149+0000","last_clean_scrub_stamp":"2024-08-03T20:54:26.410149+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T02:32:26.395472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.17","version":"0'0","reported_seq":15,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-08-03T20:54:28.440826+0000","last_change":"2024-08-03T20:54:26.445519+0000","last_active":"2024-08-03T20:54:28.440826+0000","last_peered":"2024-08-03T20:54:28.440826+0000","last_clean":"2024-08-03T20:54:28.440826+0000","last_became_active":"2024-08-03T20:54:26.445143+0000","last_became_peered":"2024-08-03T20:54:26.445143+0000","last_unstale":"2024-08-03T20:54:28.440826+0000","last_undegraded":"2024-08-03T20:54:28.440826+0000","last_fullsized":"2024-08-03T20:54:28.440826+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:54:25.402282+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:54:25.402282+0000","last_clean_scrub_stamp":"2024-08-03T20:54:25.402282+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T08:02:42.980625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.17","version":"0'0","reported_seq":13,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-08-03T20:54:29.323861+0000","last_change":"2024-08-03T20:54:27.453807+0000","last_active":"2024-08-03T20:54:29.323861+0000","last_peered":"2024-08-03T20:54:29.323861+0000","last_clean":"2024-08-03T20:54:29.323861+0000","last_became_active":"2024-08-03T20:54:27.453482+0000","last_became_peered":"2024-08-03T20:54:27.453482+0000","last_unstale":"2024-08-03T20:54:29.323861+0000","last_undegraded":"2024-08-03T20:54:29.323861+0000","last_fullsized":"2024-08-03T20:54:29.323861+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:54:26.410149+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:54:26.410149+0000","last_clean_scrub_stamp":"2024-08-03T20:54:26.410149+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T03:14:09.919834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.16","version":"0'0","reported_seq":15,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-08-03T20:54:28.440192+0000","last_change":"2024-08-03T20:54:26.443025+0000","last_active":"2024-08-03T20:54:28.440192+0000","last_peered":"2024-08-03T20:54:28.440192+0000","last_clean":"2024-08-03T20:54:28.440192+0000","last_became_active":"2024-08-03T20:54:26.442717+0000","last_became_peered":"2024-08-03T20:54:26.442717+0000","last_unstale":"2024-08-03T20:54:28.440192+0000","last_undegraded":"2024-08-03T20:54:28.440192+0000","last_fullsized":"2024-08-03T20:54:28.440192+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:54:25.402282+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:54:25.402282+0000","last_clean_scrub_stamp":"2024-08-03T20:54:25.402282+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-04T23:52:56.823930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.14","version":"0'0","reported_seq":13,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-08-03T20:54:29.438462+0000","last_change":"2024-08-03T20:54:27.457277+0000","last_active":"2024-08-03T20:54:29.438462+0000","last_peered":"2024-08-03T20:54:29.438462+0000","last_clean":"2024-08-03T20:54:29.438462+0000","last_became_active":"2024-08-03T20:54:27.456953+0000","last_became_peered":"2024-08-03T20:54:27.456953+0000","last_unstale":"2024-08-03T20:54:29.438462+0000","last_undegraded":"2024-08-03T20:54:29.438462+0000","last_fullsized":"2024-08-03T20:54:29.438462+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:54:26.410149+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:54:26.410149+0000","last_clean_scrub_stamp":"2024-08-03T20:54:26.410149+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T05:28:20.584840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.15","version":"82'3","reported_seq":18,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-08-03T20:54:28.439817+0000","last_change":"2024-08-03T20:54:26.444297+0000","last_active":"2024-08-03T20:54:28.439817+0000","last_peered":"2024-08-03T20:54:28.439817+0000","last_clean":"2024-08-03T20:54:28.439817+0000","last_became_active":"2024-08-03T20:54:26.443688+0000","last_became_peered":"2024-08-03T20:54:26.443688+0000","last_unstale":"2024-08-03T20:54:28.439817+0000","last_undegraded":"2024-08-03T20:54:28.439817+0000","last_fullsized":"2024-08-03T20:54:28.439817+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:54:25.402282+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:54:25.402282+0000","last_clean_scrub_stamp":"2024-08-03T20:54:25.402282+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T06:55:43.645755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.15","version":"0'0","reported_seq":13,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-08-03T20:54:28.441517+0000","last_change":"2024-08-03T20:54:27.450035+0000","last_active":"2024-08-03T20:54:28.441517+0000","last_peered":"2024-08-03T20:54:28.441517+0000","last_clean":"2024-08-03T20:54:28.441517+0000","last_became_active":"2024-08-03T20:54:27.449500+0000","last_became_peered":"2024-08-03T20:54:27.449500+0000","last_unstale":"2024-08-03T20:54:28.441517+0000","last_undegraded":"2024-08-03T20:54:28.441517+0000","last_fullsized":"2024-08-03T20:54:28.441517+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:54:26.410149+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:54:26.410149+0000","last_clean_scrub_stamp":"2024-08-03T20:54:26.410149+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T04:07:08.886579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.14","version":"82'6","reported_seq":21,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-08-03T20:54:29.323668+0000","last_change":"2024-08-03T20:54:26.448489+0000","last_active":"2024-08-03T20:54:29.323668+0000","last_peered":"2024-08-03T20:54:29.323668+0000","last_clean":"2024-08-03T20:54:29.323668+0000","last_became_active":"2024-08-03T20:54:26.448232+0000","last_became_peered":"2024-08-03T20:54:26.448232+0000","last_unstale":"2024-08-03T20:54:29.323668+0000","last_undegraded":"2024-08-03T20:54:29.323668+0000","last_fullsized":"2024-08-03T20:54:29.323668+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:54:25.402282+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:54:25.402282+0000","last_clean_scrub_stamp":"2024-08-03T20:54:25.402282+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T05:02:11.770691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.12","version":"0'0","reported_seq":13,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-08-03T20:54:29.438195+0000","last_change":"2024-08-03T20:54:27.430210+0000","last_active":"2024-08-03T20:54:29.438195+0000","last_peered":"2024-08-03T20:54:29.438195+0000","last_clean":"2024-08-03T20:54:29.438195+0000","last_became_active":"2024-08-03T20:54:27.429901+0000","last_became_peered":"2024-08-03T20:54:27.429901+0000","last_unstale":"2024-08-03T20:54:29.438195+0000","last_undegraded":"2024-08-03T20:54:29.438195+0000","last_fullsized":"2024-08-03T20:54:29.438195+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:54:26.410149+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:54:26.410149+0000","last_clean_scrub_stamp":"2024-08-03T20:54:26.410149+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-04T23:30:55.227524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.13","version":"82'2","reported_seq":17,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-08-03T20:54:29.434900+0000","last_change":"2024-08-03T20:54:26.443771+0000","last_active":"2024-08-03T20:54:29.434900+0000","last_peered":"2024-08-03T20:54:29.434900+0000","last_clean":"2024-08-03T20:54:29.434900+0000","last_became_active":"2024-08-03T20:54:26.443423+0000","last_became_peered":"2024-08-03T20:54:26.443423+0000","last_unstale":"2024-08-03T20:54:29.434900+0000","last_undegraded":"2024-08-03T20:54:29.434900+0000","last_fullsized":"2024-08-03T20:54:29.434900+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:54:25.402282+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:54:25.402282+0000","last_clean_scrub_stamp":"2024-08-03T20:54:25.402282+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T06:43:59.921992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.13","version":"0'0","reported_seq":13,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-08-03T20:54:29.482142+0000","last_change":"2024-08-03T20:54:27.454959+0000","last_active":"2024-08-03T20:54:29.482142+0000","last_peered":"2024-08-03T20:54:29.482142+0000","last_clean":"2024-08-03T20:54:29.482142+0000","last_became_active":"2024-08-03T20:54:27.454627+0000","last_became_peered":"2024-08-03T20:54:27.454627+0000","last_unstale":"2024-08-03T20:54:29.482142+0000","last_undegraded":"2024-08-03T20:54:29.482142+0000","last_fullsized":"2024-08-03T20:54:29.482142+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:54:26.410149+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:54:26.410149+0000","last_clean_scrub_stamp":"2024-08-03T20:54:26.410149+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T01:06:27.377827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.12","version":"0'0","reported_seq":13,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-03T20:54:27.429097+0000","last_change":"2024-08-03T20:54:26.438693+0000","last_active":"2024-08-03T20:54:27.429097+0000","last_peered":"2024-08-03T20:54:27.429097+0000","last_clean":"2024-08-03T20:54:27.429097+0000","last_became_active":"2024-08-03T20:54:26.438415+0000","last_became_peered":"2024-08-03T20:54:26.438415+0000","last_unstale":"2024-08-03T20:54:27.429097+0000","last_undegraded":"2024-08-03T20:54:27.429097+0000","last_fullsized":"2024-08-03T20:54:27.429097+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:54:25.402282+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:54:25.402282+0000","last_clean_scrub_stamp":"2024-08-03T20:54:25.402282+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T07:40:12.546187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.10","version":"0'0","reported_seq":13,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-08-03T20:54:29.434831+0000","last_change":"2024-08-03T20:54:27.469162+0000","last_active":"2024-08-03T20:54:29.434831+0000","last_peered":"2024-08-03T20:54:29.434831+0000","last_clean":"2024-08-03T20:54:29.434831+0000","last_became_active":"2024-08-03T20:54:27.468898+0000","last_became_peered":"2024-08-03T20:54:27.468898+0000","last_unstale":"2024-08-03T20:54:29.434831+0000","last_undegraded":"2024-08-03T20:54:29.434831+0000","last_fullsized":"2024-08-03T20:54:29.434831+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:54:26.410149+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:54:26.410149+0000","last_clean_scrub_stamp":"2024-08-03T20:54:26.410149+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-04T23:35:09.032967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.11","version":"0'0","reported_seq":15,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-08-03T20:54:29.323466+0000","last_change":"2024-08-03T20:54:26.435525+0000","last_active":"2024-08-03T20:54:29.323466+0000","last_peered":"2024-08-03T20:54:29.323466+0000","last_clean":"2024-08-03T20:54:29.323466+0000","last_became_active":"2024-08-03T20:54:26.435186+0000","last_became_peered":"2024-08-03T20:54:26.435186+0000","last_unstale":"2024-08-03T20:54:29.323466+0000","last_undegraded":"2024-08-03T20:54:29.323466+0000","last_fullsized":"2024-08-03T20:54:29.323466+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:54:25.402282+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:54:25.402282+0000","last_clean_scrub_stamp":"2024-08-03T20:54:25.402282+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T04:30:44.935983+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.11","version":"0'0","reported_seq":13,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-08-03T20:54:29.434773+0000","last_change":"2024-08-03T20:54:27.450906+0000","last_active":"2024-08-03T20:54:29.434773+0000","last_peered":"2024-08-03T20:54:29.434773+0000","last_clean":"2024-08-03T20:54:29.434773+0000","last_became_active":"2024-08-03T20:54:27.450609+0000","last_became_peered":"2024-08-03T20:54:27.450609+0000","last_unstale":"2024-08-03T20:54:29.434773+0000","last_undegraded":"2024-08-03T20:54:29.434773+0000","last_fullsized":"2024-08-03T20:54:29.434773+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:54:26.410149+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:54:26.410149+0000","last_clean_scrub_stamp":"2024-08-03T20:54:26.410149+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T02:03:59.450233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.10","version":"82'8","reported_seq":23,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-08-03T20:54:29.438106+0000","last_change":"2024-08-03T20:54:26.437602+0000","last_active":"2024-08-03T20:54:29.438106+0000","last_peered":"2024-08-03T20:54:29.438106+0000","last_clean":"2024-08-03T20:54:29.438106+0000","last_became_active":"2024-08-03T20:54:26.437247+0000","last_became_peered":"2024-08-03T20:54:26.437247+0000","last_unstale":"2024-08-03T20:54:29.438106+0000","last_undegraded":"2024-08-03T20:54:29.438106+0000","last_fullsized":"2024-08-03T20:54:29.438106+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:54:25.402282+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:54:25.402282+0000","last_clean_scrub_stamp":"2024-08-03T20:54:25.402282+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T05:24:45.467432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.e","version":"0'0","reported_seq":13,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-08-03T20:54:29.482289+0000","last_change":"2024-08-03T20:54:27.445155+0000","last_active":"2024-08-03T20:54:29.482289+0000","last_peered":"2024-08-03T20:54:29.482289+0000","last_clean":"2024-08-03T20:54:29.482289+0000","last_became_active":"2024-08-03T20:54:27.443888+0000","last_became_peered":"2024-08-03T20:54:27.443888+0000","last_unstale":"2024-08-03T20:54:29.482289+0000","last_undegraded":"2024-08-03T20:54:29.482289+0000","last_fullsized":"2024-08-03T20:54:29.482289+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:54:26.410149+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:54:26.410149+0000","last_clean_scrub_stamp":"2024-08-03T20:54:26.410149+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T00:49:10.925372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.f","version":"82'1","reported_seq":16,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-08-03T20:54:28.440933+0000","last_change":"2024-08-03T20:54:26.438576+0000","last_active":"2024-08-03T20:54:28.440933+0000","last_peered":"2024-08-03T20:54:28.440933+0000","last_clean":"2024-08-03T20:54:28.440933+0000","last_became_active":"2024-08-03T20:54:26.437929+0000","last_became_peered":"2024-08-03T20:54:26.437929+0000","last_unstale":"2024-08-03T20:54:28.440933+0000","last_undegraded":"2024-08-03T20:54:28.440933+0000","last_fullsized":"2024-08-03T20:54:28.440933+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:54:25.402282+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:54:25.402282+0000","last_clean_scrub_stamp":"2024-08-03T20:54:25.402282+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-04T21:02:34.790434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.f","version":"0'0","reported_seq":13,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-08-03T20:54:28.440713+0000","last_change":"2024-08-03T20:54:27.441892+0000","last_active":"2024-08-03T20:54:28.440713+0000","last_peered":"2024-08-03T20:54:28.440713+0000","last_clean":"2024-08-03T20:54:28.440713+0000","last_became_active":"2024-08-03T20:54:27.440993+0000","last_became_peered":"2024-08-03T20:54:27.440993+0000","last_unstale":"2024-08-03T20:54:28.440713+0000","last_undegraded":"2024-08-03T20:54:28.440713+0000","last_fullsized":"2024-08-03T20:54:28.440713+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:54:26.410149+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:54:26.410149+0000","last_clean_scrub_stamp":"2024-08-03T20:54:26.410149+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T01:06:27.377827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.e","version":"82'2","reported_seq":17,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-08-03T20:54:29.438574+0000","last_change":"2024-08-03T20:54:26.446254+0000","last_active":"2024-08-03T20:54:29.438574+0000","last_peered":"2024-08-03T20:54:29.438574+0000","last_clean":"2024-08-03T20:54:29.438574+0000","last_became_active":"2024-08-03T20:54:26.443273+0000","last_became_peered":"2024-08-03T20:54:26.443273+0000","last_unstale":"2024-08-03T20:54:29.438574+0000","last_undegraded":"2024-08-03T20:54:29.438574+0000","last_fullsized":"2024-08-03T20:54:29.438574+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:54:25.402282+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:54:25.402282+0000","last_clean_scrub_stamp":"2024-08-03T20:54:25.402282+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T08:38:30.463265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.d","version":"0'0","reported_seq":13,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-08-03T20:54:28.441726+0000","last_change":"2024-08-03T20:54:27.446287+0000","last_active":"2024-08-03T20:54:28.441726+0000","last_peered":"2024-08-03T20:54:28.441726+0000","last_clean":"2024-08-03T20:54:28.441726+0000","last_became_active":"2024-08-03T20:54:27.443555+0000","last_became_peered":"2024-08-03T20:54:27.443555+0000","last_unstale":"2024-08-03T20:54:28.441726+0000","last_undegraded":"2024-08-03T20:54:28.441726+0000","last_fullsized":"2024-08-03T20:54:28.441726+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:54:26.410149+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:54:26.410149+0000","last_clean_scrub_stamp":"2024-08-03T20:54:26.410149+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T02:46:49.413845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.c","version":"0'0","reported_seq":15,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-08-03T20:54:29.438657+0000","last_change":"2024-08-03T20:54:26.438540+0000","last_active":"2024-08-03T20:54:29.438657+0000","last_peered":"2024-08-03T20:54:29.438657+0000","last_clean":"2024-08-03T20:54:29.438657+0000","last_became_active":"2024-08-03T20:54:26.438215+0000","last_became_peered":"2024-08-03T20:54:26.438215+0000","last_unstale":"2024-08-03T20:54:29.438657+0000","last_undegraded":"2024-08-03T20:54:29.438657+0000","last_fullsized":"2024-08-03T20:54:29.438657+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:54:25.402282+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:54:25.402282+0000","last_clean_scrub_stamp":"2024-08-03T20:54:25.402282+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T03:38:16.207416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.a","version":"0'0","reported_seq":10,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-03T20:54:27.465460+0000","last_change":"2024-08-03T20:54:27.458690+0000","last_active":"2024-08-03T20:54:27.465460+0000","last_peered":"2024-08-03T20:54:27.465460+0000","last_clean":"2024-08-03T20:54:27.465460+0000","last_became_active":"2024-08-03T20:54:27.458264+0000","last_became_peered":"2024-08-03T20:54:27.458264+0000","last_unstale":"2024-08-03T20:54:27.465460+0000","last_undegraded":"2024-08-03T20:54:27.465460+0000","last_fullsized":"2024-08-03T20:54:27.465460+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:54:26.410149+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:54:26.410149+0000","last_clean_scrub_stamp":"2024-08-03T20:54:26.410149+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T00:18:26.766575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.b","version":"82'2","reported_seq":15,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-03T20:54:27.471748+0000","last_change":"2024-08-03T20:54:26.439824+0000","last_active":"2024-08-03T20:54:27.471748+0000","last_peered":"2024-08-03T20:54:27.471748+0000","last_clean":"2024-08-03T20:54:27.471748+0000","last_became_active":"2024-08-03T20:54:26.439533+0000","last_became_peered":"2024-08-03T20:54:26.439533+0000","last_unstale":"2024-08-03T20:54:27.471748+0000","last_undegraded":"2024-08-03T20:54:27.471748+0000","last_fullsized":"2024-08-03T20:54:27.471748+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:54:25.402282+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:54:25.402282+0000","last_clean_scrub_stamp":"2024-08-03T20:54:25.402282+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T01:42:10.837670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.b","version":"0'0","reported_seq":13,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-08-03T20:54:29.323745+0000","last_change":"2024-08-03T20:54:27.439785+0000","last_active":"2024-08-03T20:54:29.323745+0000","last_peered":"2024-08-03T20:54:29.323745+0000","last_clean":"2024-08-03T20:54:29.323745+0000","last_became_active":"2024-08-03T20:54:27.438739+0000","last_became_peered":"2024-08-03T20:54:27.438739+0000","last_unstale":"2024-08-03T20:54:29.323745+0000","last_undegraded":"2024-08-03T20:54:29.323745+0000","last_fullsized":"2024-08-03T20:54:29.323745+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:54:26.410149+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:54:26.410149+0000","last_clean_scrub_stamp":"2024-08-03T20:54:26.410149+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T05:28:20.584840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.a","version":"0'0","reported_seq":13,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-03T20:54:27.450560+0000","last_change":"2024-08-03T20:54:26.443888+0000","last_active":"2024-08-03T20:54:27.450560+0000","last_peered":"2024-08-03T20:54:27.450560+0000","last_clean":"2024-08-03T20:54:27.450560+0000","last_became_active":"2024-08-03T20:54:26.441806+0000","last_became_peered":"2024-08-03T20:54:26.441806+0000","last_unstale":"2024-08-03T20:54:27.450560+0000","last_undegraded":"2024-08-03T20:54:27.450560+0000","last_fullsized":"2024-08-03T20:54:27.450560+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:54:25.402282+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:54:25.402282+0000","last_clean_scrub_stamp":"2024-08-03T20:54:25.402282+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T05:30:32.126828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.8","version":"0'0","reported_seq":10,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-03T20:54:27.465768+0000","last_change":"2024-08-03T20:54:27.453663+0000","last_active":"2024-08-03T20:54:27.465768+0000","last_peered":"2024-08-03T20:54:27.465768+0000","last_clean":"2024-08-03T20:54:27.465768+0000","last_became_active":"2024-08-03T20:54:27.450915+0000","last_became_peered":"2024-08-03T20:54:27.450915+0000","last_unstale":"2024-08-03T20:54:27.465768+0000","last_undegraded":"2024-08-03T20:54:27.465768+0000","last_fullsized":"2024-08-03T20:54:27.465768+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:54:26.410149+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:54:26.410149+0000","last_clean_scrub_stamp":"2024-08-03T20:54:26.410149+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T08:20:46.797506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.9","version":"0'0","reported_seq":13,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-03T20:54:27.450827+0000","last_change":"2024-08-03T20:54:26.445547+0000","last_active":"2024-08-03T20:54:27.450827+0000","last_peered":"2024-08-03T20:54:27.450827+0000","last_clean":"2024-08-03T20:54:27.450827+0000","last_became_active":"2024-08-03T20:54:26.445196+0000","last_became_peered":"2024-08-03T20:54:26.445196+0000","last_unstale":"2024-08-03T20:54:27.450827+0000","last_undegraded":"2024-08-03T20:54:27.450827+0000","last_fullsized":"2024-08-03T20:54:27.450827+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:54:25.402282+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:54:25.402282+0000","last_clean_scrub_stamp":"2024-08-03T20:54:25.402282+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T00:04:49.083414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.9","version":"0'0","reported_seq":13,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-08-03T20:54:29.323520+0000","last_change":"2024-08-03T20:54:27.447378+0000","last_active":"2024-08-03T20:54:29.323520+0000","last_peered":"2024-08-03T20:54:29.323520+0000","last_clean":"2024-08-03T20:54:29.323520+0000","last_became_active":"2024-08-03T20:54:27.444614+0000","last_became_peered":"2024-08-03T20:54:27.444614+0000","last_unstale":"2024-08-03T20:54:29.323520+0000","last_undegraded":"2024-08-03T20:54:29.323520+0000","last_fullsized":"2024-08-03T20:54:29.323520+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:54:26.410149+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:54:26.410149+0000","last_clean_scrub_stamp":"2024-08-03T20:54:26.410149+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-04T21:41:30.478055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.8","version":"0'0","reported_seq":13,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-08-03T20:54:27.428451+0000","last_change":"2024-08-03T20:54:26.440569+0000","last_active":"2024-08-03T20:54:27.428451+0000","last_peered":"2024-08-03T20:54:27.428451+0000","last_clean":"2024-08-03T20:54:27.428451+0000","last_became_active":"2024-08-03T20:54:26.440262+0000","last_became_peered":"2024-08-03T20:54:26.440262+0000","last_unstale":"2024-08-03T20:54:27.428451+0000","last_undegraded":"2024-08-03T20:54:27.428451+0000","last_fullsized":"2024-08-03T20:54:27.428451+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:54:25.402282+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:54:25.402282+0000","last_clean_scrub_stamp":"2024-08-03T20:54:25.402282+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T06:50:58.322620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.6","version":"0'0","reported_seq":13,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-08-03T20:54:28.441840+0000","last_change":"2024-08-03T20:54:27.434816+0000","last_active":"2024-08-03T20:54:28.441840+0000","last_peered":"2024-08-03T20:54:28.441840+0000","last_clean":"2024-08-03T20:54:28.441840+0000","last_became_active":"2024-08-03T20:54:27.434505+0000","last_became_peered":"2024-08-03T20:54:27.434505+0000","last_unstale":"2024-08-03T20:54:28.441840+0000","last_undegraded":"2024-08-03T20:54:28.441840+0000","last_fullsized":"2024-08-03T20:54:28.441840+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:54:26.410149+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:54:26.410149+0000","last_clean_scrub_stamp":"2024-08-03T20:54:26.410149+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T00:31:19.062668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.7","version":"82'2","reported_seq":17,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-08-03T20:54:29.435271+0000","last_change":"2024-08-03T20:54:26.445984+0000","last_active":"2024-08-03T20:54:29.435271+0000","last_peered":"2024-08-03T20:54:29.435271+0000","last_clean":"2024-08-03T20:54:29.435271+0000","last_became_active":"2024-08-03T20:54:26.445698+0000","last_became_peered":"2024-08-03T20:54:26.445698+0000","last_unstale":"2024-08-03T20:54:29.435271+0000","last_undegraded":"2024-08-03T20:54:29.435271+0000","last_fullsized":"2024-08-03T20:54:29.435271+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:54:25.402282+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:54:25.402282+0000","last_clean_scrub_stamp":"2024-08-03T20:54:25.402282+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T04:56:33.713984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.7","version":"0'0","reported_seq":13,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-08-03T20:54:29.482258+0000","last_change":"2024-08-03T20:54:27.443424+0000","last_active":"2024-08-03T20:54:29.482258+0000","last_peered":"2024-08-03T20:54:29.482258+0000","last_clean":"2024-08-03T20:54:29.482258+0000","last_became_active":"2024-08-03T20:54:27.443031+0000","last_became_peered":"2024-08-03T20:54:27.443031+0000","last_unstale":"2024-08-03T20:54:29.482258+0000","last_undegraded":"2024-08-03T20:54:29.482258+0000","last_fullsized":"2024-08-03T20:54:29.482258+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:54:26.410149+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:54:26.410149+0000","last_clean_scrub_stamp":"2024-08-03T20:54:26.410149+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T07:40:36.636568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.6","version":"83'1","reported_seq":16,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-08-03T20:54:29.437258+0000","last_change":"2024-08-03T20:54:26.443003+0000","last_active":"2024-08-03T20:54:29.437258+0000","last_peered":"2024-08-03T20:54:29.437258+0000","last_clean":"2024-08-03T20:54:29.437258+0000","last_became_active":"2024-08-03T20:54:26.442653+0000","last_became_peered":"2024-08-03T20:54:26.442653+0000","last_unstale":"2024-08-03T20:54:29.437258+0000","last_undegraded":"2024-08-03T20:54:29.437258+0000","last_fullsized":"2024-08-03T20:54:29.437258+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:54:25.402282+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:54:25.402282+0000","last_clean_scrub_stamp":"2024-08-03T20:54:25.402282+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T04:30:23.014758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.4","version":"0'0","reported_seq":13,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-08-03T20:54:28.440469+0000","last_change":"2024-08-03T20:54:27.457535+0000","last_active":"2024-08-03T20:54:28.440469+0000","last_peered":"2024-08-03T20:54:28.440469+0000","last_clean":"2024-08-03T20:54:28.440469+0000","last_became_active":"2024-08-03T20:54:27.457180+0000","last_became_peered":"2024-08-03T20:54:27.457180+0000","last_unstale":"2024-08-03T20:54:28.440469+0000","last_undegraded":"2024-08-03T20:54:28.440469+0000","last_fullsized":"2024-08-03T20:54:28.440469+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:54:26.410149+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:54:26.410149+0000","last_clean_scrub_stamp":"2024-08-03T20:54:26.410149+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T07:09:56.880905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.5","version":"0'0","reported_seq":15,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-08-03T20:54:28.440118+0000","last_change":"2024-08-03T20:54:26.444463+0000","last_active":"2024-08-03T20:54:28.440118+0000","last_peered":"2024-08-03T20:54:28.440118+0000","last_clean":"2024-08-03T20:54:28.440118+0000","last_became_active":"2024-08-03T20:54:26.444121+0000","last_became_peered":"2024-08-03T20:54:26.444121+0000","last_unstale":"2024-08-03T20:54:28.440118+0000","last_undegraded":"2024-08-03T20:54:28.440118+0000","last_fullsized":"2024-08-03T20:54:28.440118+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:54:25.402282+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:54:25.402282+0000","last_clean_scrub_stamp":"2024-08-03T20:54:25.402282+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T03:28:34.661003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.5","version":"0'0","reported_seq":13,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-08-03T20:54:28.441816+0000","last_change":"2024-08-03T20:54:27.448379+0000","last_active":"2024-08-03T20:54:28.441816+0000","last_peered":"2024-08-03T20:54:28.441816+0000","last_clean":"2024-08-03T20:54:28.441816+0000","last_became_active":"2024-08-03T20:54:27.448078+0000","last_became_peered":"2024-08-03T20:54:27.448078+0000","last_unstale":"2024-08-03T20:54:28.441816+0000","last_undegraded":"2024-08-03T20:54:28.441816+0000","last_fullsized":"2024-08-03T20:54:28.441816+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:54:26.410149+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:54:26.410149+0000","last_clean_scrub_stamp":"2024-08-03T20:54:26.410149+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T08:02:25.589517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.4","version":"0'0","reported_seq":15,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-08-03T20:54:28.440481+0000","last_change":"2024-08-03T20:54:26.440460+0000","last_active":"2024-08-03T20:54:28.440481+0000","last_peered":"2024-08-03T20:54:28.440481+0000","last_clean":"2024-08-03T20:54:28.440481+0000","last_became_active":"2024-08-03T20:54:26.439877+0000","last_became_peered":"2024-08-03T20:54:26.439877+0000","last_unstale":"2024-08-03T20:54:28.440481+0000","last_undegraded":"2024-08-03T20:54:28.440481+0000","last_fullsized":"2024-08-03T20:54:28.440481+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:54:25.402282+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:54:25.402282+0000","last_clean_scrub_stamp":"2024-08-03T20:54:25.402282+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-04T21:51:04.354598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.2","version":"0'0","reported_seq":13,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-08-03T20:54:29.323650+0000","last_change":"2024-08-03T20:54:27.439499+0000","last_active":"2024-08-03T20:54:29.323650+0000","last_peered":"2024-08-03T20:54:29.323650+0000","last_clean":"2024-08-03T20:54:29.323650+0000","last_became_active":"2024-08-03T20:54:27.438801+0000","last_became_peered":"2024-08-03T20:54:27.438801+0000","last_unstale":"2024-08-03T20:54:29.323650+0000","last_undegraded":"2024-08-03T20:54:29.323650+0000","last_fullsized":"2024-08-03T20:54:29.323650+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:54:26.410149+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:54:26.410149+0000","last_clean_scrub_stamp":"2024-08-03T20:54:26.410149+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T06:59:28.260517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.3","version":"0'0","reported_seq":15,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-08-03T20:54:29.438732+0000","last_change":"2024-08-03T20:54:26.445654+0000","last_active":"2024-08-03T20:54:29.438732+0000","last_peered":"2024-08-03T20:54:29.438732+0000","last_clean":"2024-08-03T20:54:29.438732+0000","last_became_active":"2024-08-03T20:54:26.442568+0000","last_became_peered":"2024-08-03T20:54:26.442568+0000","last_unstale":"2024-08-03T20:54:29.438732+0000","last_undegraded":"2024-08-03T20:54:29.438732+0000","last_fullsized":"2024-08-03T20:54:29.438732+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:54:25.402282+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:54:25.402282+0000","last_clean_scrub_stamp":"2024-08-03T20:54:25.402282+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T08:38:24.403913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.3","version":"0'0","reported_seq":13,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-08-03T20:54:29.438541+0000","last_change":"2024-08-03T20:54:27.430945+0000","last_active":"2024-08-03T20:54:29.438541+0000","last_peered":"2024-08-03T20:54:29.438541+0000","last_clean":"2024-08-03T20:54:29.438541+0000","last_became_active":"2024-08-03T20:54:27.430617+0000","last_became_peered":"2024-08-03T20:54:27.430617+0000","last_unstale":"2024-08-03T20:54:29.438541+0000","last_undegraded":"2024-08-03T20:54:29.438541+0000","last_fullsized":"2024-08-03T20:54:29.438541+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:54:26.410149+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:54:26.410149+0000","last_clean_scrub_stamp":"2024-08-03T20:54:26.410149+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T03:14:09.919834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.2","version":"0'0","reported_seq":15,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-08-03T20:54:28.441463+0000","last_change":"2024-08-03T20:54:26.438196+0000","last_active":"2024-08-03T20:54:28.441463+0000","last_peered":"2024-08-03T20:54:28.441463+0000","last_clean":"2024-08-03T20:54:28.441463+0000","last_became_active":"2024-08-03T20:54:26.437831+0000","last_became_peered":"2024-08-03T20:54:26.437831+0000","last_unstale":"2024-08-03T20:54:28.441463+0000","last_undegraded":"2024-08-03T20:54:28.441463+0000","last_fullsized":"2024-08-03T20:54:28.441463+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:54:25.402282+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:54:25.402282+0000","last_clean_scrub_stamp":"2024-08-03T20:54:25.402282+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T04:35:25.487288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.0","version":"0'0","reported_seq":13,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-08-03T20:54:29.482432+0000","last_change":"2024-08-03T20:54:27.453943+0000","last_active":"2024-08-03T20:54:29.482432+0000","last_peered":"2024-08-03T20:54:29.482432+0000","last_clean":"2024-08-03T20:54:29.482432+0000","last_became_active":"2024-08-03T20:54:27.453580+0000","last_became_peered":"2024-08-03T20:54:27.453580+0000","last_unstale":"2024-08-03T20:54:29.482432+0000","last_undegraded":"2024-08-03T20:54:29.482432+0000","last_fullsized":"2024-08-03T20:54:29.482432+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:54:26.410149+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:54:26.410149+0000","last_clean_scrub_stamp":"2024-08-03T20:54:26.410149+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T04:23:13.290160+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.1","version":"0'0","reported_seq":15,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-08-03T20:54:28.439874+0000","last_change":"2024-08-03T20:54:26.439778+0000","last_active":"2024-08-03T20:54:28.439874+0000","last_peered":"2024-08-03T20:54:28.439874+0000","last_clean":"2024-08-03T20:54:28.439874+0000","last_became_active":"2024-08-03T20:54:26.438340+0000","last_became_peered":"2024-08-03T20:54:26.438340+0000","last_unstale":"2024-08-03T20:54:28.439874+0000","last_undegraded":"2024-08-03T20:54:28.439874+0000","last_fullsized":"2024-08-03T20:54:28.439874+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:54:25.402282+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:54:25.402282+0000","last_clean_scrub_stamp":"2024-08-03T20:54:25.402282+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-04T23:35:08.025100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1","version":"0'0","reported_seq":13,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-08-03T20:54:28.440442+0000","last_change":"2024-08-03T20:54:27.457876+0000","last_active":"2024-08-03T20:54:28.440442+0000","last_peered":"2024-08-03T20:54:28.440442+0000","last_clean":"2024-08-03T20:54:28.440442+0000","last_became_active":"2024-08-03T20:54:27.457492+0000","last_became_peered":"2024-08-03T20:54:27.457492+0000","last_unstale":"2024-08-03T20:54:28.440442+0000","last_undegraded":"2024-08-03T20:54:28.440442+0000","last_fullsized":"2024-08-03T20:54:28.440442+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:54:26.410149+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:54:26.410149+0000","last_clean_scrub_stamp":"2024-08-03T20:54:26.410149+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T07:56:54.337763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.0","version":"0'0","reported_seq":15,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-08-03T20:54:28.441279+0000","last_change":"2024-08-03T20:54:26.442870+0000","last_active":"2024-08-03T20:54:28.441279+0000","last_peered":"2024-08-03T20:54:28.441279+0000","last_clean":"2024-08-03T20:54:28.441279+0000","last_became_active":"2024-08-03T20:54:26.442475+0000","last_became_peered":"2024-08-03T20:54:26.442475+0000","last_unstale":"2024-08-03T20:54:28.441279+0000","last_undegraded":"2024-08-03T20:54:28.441279+0000","last_fullsized":"2024-08-03T20:54:28.441279+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:54:25.402282+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:54:25.402282+0000","last_clean_scrub_stamp":"2024-08-03T20:54:25.402282+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T05:11:57.699458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.c","version":"0'0","reported_seq":13,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-08-03T20:54:29.435238+0000","last_change":"2024-08-03T20:54:27.442895+0000","last_active":"2024-08-03T20:54:29.435238+0000","last_peered":"2024-08-03T20:54:29.435238+0000","last_clean":"2024-08-03T20:54:29.435238+0000","last_became_active":"2024-08-03T20:54:27.441598+0000","last_became_peered":"2024-08-03T20:54:27.441598+0000","last_unstale":"2024-08-03T20:54:29.435238+0000","last_undegraded":"2024-08-03T20:54:29.435238+0000","last_fullsized":"2024-08-03T20:54:29.435238+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:54:26.410149+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:54:26.410149+0000","last_clean_scrub_stamp":"2024-08-03T20:54:26.410149+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T06:55:44.653623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.d","version":"82'1","reported_seq":16,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-08-03T20:54:29.482363+0000","last_change":"2024-08-03T20:54:26.443737+0000","last_active":"2024-08-03T20:54:29.482363+0000","last_peered":"2024-08-03T20:54:29.482363+0000","last_clean":"2024-08-03T20:54:29.482363+0000","last_became_active":"2024-08-03T20:54:26.443276+0000","last_became_peered":"2024-08-03T20:54:26.443276+0000","last_unstale":"2024-08-03T20:54:29.482363+0000","last_undegraded":"2024-08-03T20:54:29.482363+0000","last_fullsized":"2024-08-03T20:54:29.482363+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:54:25.402282+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:54:25.402282+0000","last_clean_scrub_stamp":"2024-08-03T20:54:25.402282+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T04:40:32.715596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"16'72","reported_seq":227,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-08-03T20:54:29.482297+0000","last_change":"2024-08-03T20:47:30.547632+0000","last_active":"2024-08-03T20:54:29.482297+0000","last_peered":"2024-08-03T20:54:29.482297+0000","last_clean":"2024-08-03T20:54:29.482297+0000","last_became_active":"2024-08-03T20:47:30.546255+0000","last_became_peered":"2024-08-03T20:47:30.546255+0000","last_unstale":"2024-08-03T20:54:29.482297+0000","last_undegraded":"2024-08-03T20:54:29.482297+0000","last_fullsized":"2024-08-03T20:54:29.482297+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:47:28.874451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:47:28.874451+0000","last_clean_scrub_stamp":"2024-08-03T20:47:28.874451+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T00:19:40.548812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":13,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":12,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509527,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32748,"kb_used_data":1560,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339092,"statfs":{"total":96636764160,"available":96603230208,"internally_reserved":0,"allocated":1597440,"data_stored":1016362,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":6,"up_from":15,"seq":64424509528,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32188,"kb_used_data":1000,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339652,"statfs":{"total":96636764160,"available":96603803648,"internally_reserved":0,"allocated":1024000,"data_stored":432918,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":5,"up_from":15,"seq":64424509528,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32176,"kb_used_data":992,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339664,"statfs":{"total":96636764160,"available":96603815936,"internally_reserved":0,"allocated":1015808,"data_stored":427558,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509527,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36332,"kb_used_data":988,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335508,"statfs":{"total":96636764160,"available":96599560192,"internally_reserved":0,"allocated":1011712,"data_stored":426118,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509529,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36908,"kb_used_data":1560,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334932,"statfs":{"total":96636764160,"available":96598970368,"internally_reserved":0,"allocated":1597440,"data_stored":1016326,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542232,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36328,"kb_used_data":988,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335512,"statfs":{"total":96636764160,"available":96599564288,"internally_reserved":0,"allocated":1011712,"data_stored":432224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":3,"up_from":14,"seq":60129542230,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32176,"kb_used_data":988,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339664,"statfs":{"total":96636764160,"available":96603815936,"internally_reserved":0,"allocated":1011712,"data_stored":427536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":2,"up_from":14,"seq":60129542231,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32168,"kb_used_data":980,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339672,"statfs":{"total":96636764160,"available":96603824128,"internally_reserved":0,"allocated":1003520,"data_stored":426506,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":194,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-08-03T20:54:32.464 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-08-03T20:54:32.465 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-08-03T20:54:32.737 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:32.734+0000 7f32d1400640 1 -- 172.21.15.32:0/3807316730 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f32cc0cd930 msgr2=0x7f32cc0b2190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:32.737 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:32.734+0000 7f32d1400640 1 --2- 172.21.15.32:0/3807316730 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f32cc0cd930 0x7f32cc0b2190 secure :-1 s=READY pgs=914 cs=0 l=1 rev1=1 crypto rx=0x7f32c4005fd0 tx=0x7f32c402a980 comp rx=0 tx=0).stop 2024-08-03T20:54:32.737 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:32.734+0000 7f32d1400640 1 -- 172.21.15.32:0/3807316730 shutdown_connections 2024-08-03T20:54:32.737 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:32.734+0000 7f32d1400640 1 --2- 172.21.15.32:0/3807316730 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f32cc0cd930 0x7f32cc0b2190 unknown :-1 s=CLOSED pgs=914 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:32.737 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:32.734+0000 7f32d1400640 1 --2- 172.21.15.32:0/3807316730 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f32cc0b71b0 0x7f32cc0ce6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:32.738 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:32.734+0000 7f32d1400640 1 -- 172.21.15.32:0/3807316730 >> 172.21.15.32:0/3807316730 conn(0x7f32cc0d1eb0 msgr2=0x7f32cc0d22c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:54:32.738 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:32.735+0000 7f32d1400640 1 -- 172.21.15.32:0/3807316730 shutdown_connections 2024-08-03T20:54:32.738 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:32.735+0000 7f32d1400640 1 -- 172.21.15.32:0/3807316730 wait complete. 2024-08-03T20:54:32.739 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:32.737+0000 7f32d1400640 1 Processor -- start 2024-08-03T20:54:32.739 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:32.737+0000 7f32d1400640 1 -- start start 2024-08-03T20:54:32.742 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:32.738+0000 7f32d1400640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f32cc0b71b0 0x7f32cc0cb5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:32.742 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:32.738+0000 7f32d1400640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f32cc0cd930 0x7f32cc159600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:32.742 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:32.739+0000 7f32d1400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f32cc0cec20 0x7f32cc159b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:32.742 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:32.739+0000 7f32d1400640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f32cc0b26d0 con 0x7f32cc0cec20 2024-08-03T20:54:32.742 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:32.739+0000 7f32d1400640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f32cc0b13c0 con 0x7f32cc0cd930 2024-08-03T20:54:32.742 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:32.739+0000 7f32d1400640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7f32cc0b2850 con 0x7f32cc0b71b0 2024-08-03T20:54:32.742 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:32.739+0000 7f32d0a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f32cc0cec20 0x7f32cc159b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:32.742 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:32.739+0000 7f32d0a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f32cc0cec20 0x7f32cc159b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:39996/0 (socket says 172.21.15.32:39996) 2024-08-03T20:54:32.743 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:32.739+0000 7f32d0a00640 1 -- 172.21.15.32:0/865277999 learned_addr learned my addr 172.21.15.32:0/865277999 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:54:32.743 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:32.739+0000 7f32cbe00640 1 --2- 172.21.15.32:0/865277999 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f32cc0b71b0 0x7f32cc0cb5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:32.743 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:32.739+0000 7f32cb400640 1 --2- 172.21.15.32:0/865277999 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f32cc0cd930 0x7f32cc159600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:32.743 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:32.739+0000 7f32d0a00640 1 -- 172.21.15.32:0/865277999 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f32cc0b71b0 msgr2=0x7f32cc0cb5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:32.743 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:32.739+0000 7f32d0a00640 1 --2- 172.21.15.32:0/865277999 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f32cc0b71b0 0x7f32cc0cb5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:32.743 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:32.739+0000 7f32d0a00640 1 -- 172.21.15.32:0/865277999 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f32cc0cd930 msgr2=0x7f32cc159600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:32.743 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:32.739+0000 7f32d0a00640 1 --2- 172.21.15.32:0/865277999 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f32cc0cd930 0x7f32cc159600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:32.743 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:32.739+0000 7f32d0a00640 1 -- 172.21.15.32:0/865277999 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f32cc1f6880 con 0x7f32cc0cec20 2024-08-03T20:54:32.743 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:32.739+0000 7f32cbe00640 1 --2- 172.21.15.32:0/865277999 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f32cc0b71b0 0x7f32cc0cb5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-03T20:54:32.744 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:32.739+0000 7f32cb400640 1 --2- 172.21.15.32:0/865277999 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f32cc0cd930 0x7f32cc159600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-03T20:54:32.744 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:32.740+0000 7f32d0a00640 1 --2- 172.21.15.32:0/865277999 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f32cc0cec20 0x7f32cc159b40 secure :-1 s=READY pgs=915 cs=0 l=1 rev1=1 crypto rx=0x7f32c4006010 tx=0x7f32c4003f10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:54:32.744 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:32.740+0000 7f32c3e00640 1 -- 172.21.15.32:0/865277999 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32c4034040 con 0x7f32cc0cec20 2024-08-03T20:54:32.744 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:32.740+0000 7f32c3e00640 1 -- 172.21.15.32:0/865277999 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f32c4004640 con 0x7f32cc0cec20 2024-08-03T20:54:32.744 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:32.740+0000 7f32c3e00640 1 -- 172.21.15.32:0/865277999 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32c4045480 con 0x7f32cc0cec20 2024-08-03T20:54:32.744 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:32.740+0000 7f32d1400640 1 -- 172.21.15.32:0/865277999 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f32cc15bab0 con 0x7f32cc0cec20 2024-08-03T20:54:32.746 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:32.740+0000 7f32d1400640 1 -- 172.21.15.32:0/865277999 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f32cc15c070 con 0x7f32cc0cec20 2024-08-03T20:54:32.746 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:32.742+0000 7f32d1400640 1 -- 172.21.15.32:0/865277999 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f32cc06a250 con 0x7f32cc0cec20 2024-08-03T20:54:32.746 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:32.743+0000 7f32c3e00640 1 -- 172.21.15.32:0/865277999 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f32c4037020 con 0x7f32cc0cec20 2024-08-03T20:54:32.746 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:32.743+0000 7f32c3e00640 1 --2- 172.21.15.32:0/865277999 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f32ac077010 0x7f32ac079520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:32.746 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:32.744+0000 7f32c3e00640 1 -- 172.21.15.32:0/865277999 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 5709+0+0 (secure 0 0 0) 0x7f32c40ba390 con 0x7f32cc0cec20 2024-08-03T20:54:32.746 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:32.744+0000 7f32cbe00640 1 --2- 172.21.15.32:0/865277999 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f32ac077010 0x7f32ac079520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:32.747 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:32.745+0000 7f32cbe00640 1 --2- 172.21.15.32:0/865277999 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f32ac077010 0x7f32ac079520 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f32bc00ab40 tx=0x7f32bc00a520 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:54:32.749 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:32.748+0000 7f32c3e00640 1 -- 172.21.15.32:0/865277999 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7f32cc0cf030 con 0x7f32cc0cec20 2024-08-03T20:54:33.032 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:33.031+0000 7f32d1400640 1 -- 172.21.15.32:0/865277999 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) -- 0x7f32cc158b20 con 0x7f32cc0cec20 2024-08-03T20:54:33.035 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:33.034+0000 7f32c3e00640 1 -- 172.21.15.32:0/865277999 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v15) ==== 190+0+147 (secure 0 0 0) 0x7f32c4083580 con 0x7f32cc0cec20 2024-08-03T20:54:33.036 INFO:teuthology.orchestra.run.smithi032.stdout:[client.0] 2024-08-03T20:54:33.036 INFO:teuthology.orchestra.run.smithi032.stdout: key = AQAwl65msgO3NRAAHG/WuMTK9c4lDXocXvqWnA== 2024-08-03T20:54:33.036 INFO:teuthology.orchestra.run.smithi032.stdout: caps mds = "allow" 2024-08-03T20:54:33.036 INFO:teuthology.orchestra.run.smithi032.stdout: caps mon = "allow r" 2024-08-03T20:54:33.036 INFO:teuthology.orchestra.run.smithi032.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2024-08-03T20:54:33.036 INFO:teuthology.orchestra.run.smithi032.stderr:updated caps for client.0 2024-08-03T20:54:33.038 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:33.037+0000 7f32d1400640 1 -- 172.21.15.32:0/865277999 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f32ac077010 msgr2=0x7f32ac079520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:33.039 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:33.037+0000 7f32d1400640 1 --2- 172.21.15.32:0/865277999 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f32ac077010 0x7f32ac079520 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f32bc00ab40 tx=0x7f32bc00a520 comp rx=0 tx=0).stop 2024-08-03T20:54:33.039 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:33.037+0000 7f32d1400640 1 -- 172.21.15.32:0/865277999 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f32cc0cec20 msgr2=0x7f32cc159b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:33.039 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:33.037+0000 7f32d1400640 1 --2- 172.21.15.32:0/865277999 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f32cc0cec20 0x7f32cc159b40 secure :-1 s=READY pgs=915 cs=0 l=1 rev1=1 crypto rx=0x7f32c4006010 tx=0x7f32c4003f10 comp rx=0 tx=0).stop 2024-08-03T20:54:33.039 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:33.037+0000 7f32d1400640 1 -- 172.21.15.32:0/865277999 shutdown_connections 2024-08-03T20:54:33.039 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:33.037+0000 7f32d1400640 1 --2- 172.21.15.32:0/865277999 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f32ac077010 0x7f32ac079520 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:33.039 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:33.037+0000 7f32d1400640 1 --2- 172.21.15.32:0/865277999 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f32cc0cd930 0x7f32cc159600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:33.039 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:33.037+0000 7f32d1400640 1 --2- 172.21.15.32:0/865277999 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f32cc0cec20 0x7f32cc159b40 unknown :-1 s=CLOSED pgs=915 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:33.039 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:33.037+0000 7f32d1400640 1 --2- 172.21.15.32:0/865277999 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f32cc0b71b0 0x7f32cc0cb5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:33.039 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:33.037+0000 7f32d1400640 1 -- 172.21.15.32:0/865277999 >> 172.21.15.32:0/865277999 conn(0x7f32cc0d1eb0 msgr2=0x7f32cc06db40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:54:33.039 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:33.037+0000 7f32d1400640 1 -- 172.21.15.32:0/865277999 shutdown_connections 2024-08-03T20:54:33.039 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:33.037+0000 7f32d1400640 1 -- 172.21.15.32:0/865277999 wait complete. 2024-08-03T20:54:33.065 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-08-03T20:54:33.294 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:33.292+0000 7f36bda00640 1 -- 172.21.15.32:0/1070613525 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f36b80af020 msgr2=0x7f36b80742b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:33.294 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:33.292+0000 7f36bda00640 1 --2- 172.21.15.32:0/1070613525 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f36b80af020 0x7f36b80742b0 secure :-1 s=READY pgs=916 cs=0 l=1 rev1=1 crypto rx=0x7f36a8005fd0 tx=0x7f36a802a4a0 comp rx=0 tx=0).stop 2024-08-03T20:54:33.294 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:33.292+0000 7f36bda00640 1 -- 172.21.15.32:0/1070613525 shutdown_connections 2024-08-03T20:54:33.294 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:33.292+0000 7f36bda00640 1 --2- 172.21.15.32:0/1070613525 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f36b806fb50 0x7f36b80ad190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:33.294 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:33.292+0000 7f36bda00640 1 --2- 172.21.15.32:0/1070613525 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f36b80af020 0x7f36b80742b0 unknown :-1 s=CLOSED pgs=916 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:33.294 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:33.292+0000 7f36bda00640 1 -- 172.21.15.32:0/1070613525 >> 172.21.15.32:0/1070613525 conn(0x7f36b80b2120 msgr2=0x7f36b80cfd60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:54:33.294 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:33.292+0000 7f36bda00640 1 -- 172.21.15.32:0/1070613525 shutdown_connections 2024-08-03T20:54:33.294 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:33.292+0000 7f36bda00640 1 -- 172.21.15.32:0/1070613525 wait complete. 2024-08-03T20:54:33.295 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:33.294+0000 7f36bda00640 1 Processor -- start 2024-08-03T20:54:33.295 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:33.294+0000 7f36bda00640 1 -- start start 2024-08-03T20:54:33.298 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:33.296+0000 7f36bda00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f36b806fb50 0x7f36b8149ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:33.298 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:33.296+0000 7f36bda00640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f36b80acd80 0x7f36b815b350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:33.298 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:33.296+0000 7f36bda00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f36b80af020 0x7f36b815b890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:33.298 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:33.296+0000 7f36bda00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f36b81601e0 con 0x7f36b806fb50 2024-08-03T20:54:33.298 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:33.296+0000 7f36bda00640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f36b8160060 con 0x7f36b80af020 2024-08-03T20:54:33.298 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:33.296+0000 7f36bda00640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7f36b8160360 con 0x7f36b80acd80 2024-08-03T20:54:33.298 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:33.296+0000 7f36b7e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f36b806fb50 0x7f36b8149ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:33.298 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:33.296+0000 7f36b7e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f36b806fb50 0x7f36b8149ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:40014/0 (socket says 172.21.15.32:40014) 2024-08-03T20:54:33.299 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:33.296+0000 7f36b7e00640 1 -- 172.21.15.32:0/4183152206 learned_addr learned my addr 172.21.15.32:0/4183152206 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:54:33.299 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:33.296+0000 7f36af400640 1 --2- 172.21.15.32:0/4183152206 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f36b80acd80 0x7f36b815b350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:33.299 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:33.296+0000 7f36bd000640 1 --2- 172.21.15.32:0/4183152206 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f36b80af020 0x7f36b815b890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:33.299 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:33.297+0000 7f36b7e00640 1 -- 172.21.15.32:0/4183152206 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f36b80acd80 msgr2=0x7f36b815b350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:33.299 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:33.297+0000 7f36b7e00640 1 --2- 172.21.15.32:0/4183152206 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f36b80acd80 0x7f36b815b350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:33.299 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:33.297+0000 7f36b7e00640 1 -- 172.21.15.32:0/4183152206 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f36b80af020 msgr2=0x7f36b815b890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:33.299 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:33.297+0000 7f36b7e00640 1 --2- 172.21.15.32:0/4183152206 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f36b80af020 0x7f36b815b890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:33.299 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:33.297+0000 7f36b7e00640 1 -- 172.21.15.32:0/4183152206 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f36b80c5ce0 con 0x7f36b806fb50 2024-08-03T20:54:33.300 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:33.297+0000 7f36b7e00640 1 --2- 172.21.15.32:0/4183152206 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f36b806fb50 0x7f36b8149ed0 secure :-1 s=READY pgs=917 cs=0 l=1 rev1=1 crypto rx=0x7f36a8000c00 tx=0x7f36a8003970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:54:33.300 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:33.297+0000 7f36b4c00640 1 -- 172.21.15.32:0/4183152206 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36a803e030 con 0x7f36b806fb50 2024-08-03T20:54:33.300 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:33.297+0000 7f36b4c00640 1 -- 172.21.15.32:0/4183152206 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f36a8003e60 con 0x7f36b806fb50 2024-08-03T20:54:33.300 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:33.297+0000 7f36b4c00640 1 -- 172.21.15.32:0/4183152206 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36a8035040 con 0x7f36b806fb50 2024-08-03T20:54:33.300 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:33.297+0000 7f36bda00640 1 -- 172.21.15.32:0/4183152206 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f36b80c67a0 con 0x7f36b806fb50 2024-08-03T20:54:33.300 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:33.297+0000 7f36bda00640 1 -- 172.21.15.32:0/4183152206 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f36b80c5fa0 con 0x7f36b806fb50 2024-08-03T20:54:33.300 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:33.298+0000 7f36bda00640 1 -- 172.21.15.32:0/4183152206 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f36b80ad190 con 0x7f36b806fb50 2024-08-03T20:54:33.301 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:33.300+0000 7f36b4c00640 1 -- 172.21.15.32:0/4183152206 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f36a8041030 con 0x7f36b806fb50 2024-08-03T20:54:33.302 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:33.301+0000 7f36b4c00640 1 --2- 172.21.15.32:0/4183152206 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f3694077040 0x7f3694079550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:33.302 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:33.301+0000 7f36b4c00640 1 -- 172.21.15.32:0/4183152206 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 5709+0+0 (secure 0 0 0) 0x7f36a80b90c0 con 0x7f36b806fb50 2024-08-03T20:54:33.302 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:33.301+0000 7f36af400640 1 --2- 172.21.15.32:0/4183152206 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f3694077040 0x7f3694079550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:33.303 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:33.302+0000 7f36af400640 1 --2- 172.21.15.32:0/4183152206 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f3694077040 0x7f3694079550 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f36b80c69f0 tx=0x7f36a0009060 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:54:33.307 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:33.305+0000 7f36b4c00640 1 -- 172.21.15.32:0/4183152206 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7f36a8081c60 con 0x7f36b806fb50 2024-08-03T20:54:33.557 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:33.555+0000 7f36bda00640 1 -- 172.21.15.32:0/4183152206 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f36b806e830 con 0x7f36b806fb50 2024-08-03T20:54:33.557 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:33.556+0000 7f36b4c00640 1 -- 172.21.15.32:0/4183152206 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 91 v91) ==== 94+0+5228 (secure 0 0 0) 0x7f36a80861b0 con 0x7f36b806fb50 2024-08-03T20:54:33.558 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-03T20:54:33.558 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":91,"btime":"2024-08-03T20:54:30:445075+0000","default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6439,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6835/531080160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6834","nonce":531080160},{"type":"v1","addr":"172.21.15.118:6835","nonce":531080160}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":84},{"gid":6454,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6837/2796882319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6836","nonce":2796882319},{"type":"v1","addr":"172.21.15.118:6837","nonce":2796882319}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":86},{"gid":6813,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6834/2686933890","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":2686933890},{"type":"v1","addr":"172.21.15.32:6834","nonce":2686933890}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":85}],"filesystems":[{"mdsmap":{"epoch":91,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-08-03T20:54:26.950518+0000","modified":"2024-08-03T20:54:29.976506+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6816},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6816":{"gid":6816,"name":"a","rank":0,"incarnation":87,"state":"up:active","state_seq":5,"addr":"172.21.15.32:6835/1707454554","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6833","nonce":1707454554},{"type":"v1","addr":"172.21.15.32:6835","nonce":1707454554}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6816,"qdb_cluster":[6816]},"id":6}]} 2024-08-03T20:54:33.558 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 91 2024-08-03T20:54:33.560 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:33.558+0000 7f36bda00640 1 -- 172.21.15.32:0/4183152206 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f3694077040 msgr2=0x7f3694079550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:33.560 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:33.558+0000 7f36bda00640 1 --2- 172.21.15.32:0/4183152206 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f3694077040 0x7f3694079550 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f36b80c69f0 tx=0x7f36a0009060 comp rx=0 tx=0).stop 2024-08-03T20:54:33.560 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:33.558+0000 7f36bda00640 1 -- 172.21.15.32:0/4183152206 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f36b806fb50 msgr2=0x7f36b8149ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:33.560 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:33.558+0000 7f36bda00640 1 --2- 172.21.15.32:0/4183152206 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f36b806fb50 0x7f36b8149ed0 secure :-1 s=READY pgs=917 cs=0 l=1 rev1=1 crypto rx=0x7f36a8000c00 tx=0x7f36a8003970 comp rx=0 tx=0).stop 2024-08-03T20:54:33.560 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:33.558+0000 7f36bda00640 1 -- 172.21.15.32:0/4183152206 shutdown_connections 2024-08-03T20:54:33.560 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:33.558+0000 7f36bda00640 1 --2- 172.21.15.32:0/4183152206 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f3694077040 0x7f3694079550 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:33.560 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:33.558+0000 7f36bda00640 1 --2- 172.21.15.32:0/4183152206 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f36b80af020 0x7f36b815b890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:33.560 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:33.558+0000 7f36bda00640 1 --2- 172.21.15.32:0/4183152206 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f36b80acd80 0x7f36b815b350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:33.560 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:33.558+0000 7f36bda00640 1 --2- 172.21.15.32:0/4183152206 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f36b806fb50 0x7f36b8149ed0 unknown :-1 s=CLOSED pgs=917 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:33.561 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:33.558+0000 7f36bda00640 1 -- 172.21.15.32:0/4183152206 >> 172.21.15.32:0/4183152206 conn(0x7f36b80b2120 msgr2=0x7f36b80cc7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:54:33.561 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:33.558+0000 7f36bda00640 1 -- 172.21.15.32:0/4183152206 shutdown_connections 2024-08-03T20:54:33.561 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:33.558+0000 7f36bda00640 1 -- 172.21.15.32:0/4183152206 wait complete. 2024-08-03T20:54:33.985 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 91, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-08-03T20:54:26.950518+0000', 'modified': '2024-08-03T20:54:29.976506+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 6816}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6816': {'gid': 6816, 'name': 'a', 'rank': 0, 'incarnation': 87, 'state': 'up:active', 'state_seq': 5, 'addr': '172.21.15.32:6835/1707454554', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6833', 'nonce': 1707454554}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 1707454554}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [13], 'metadata_pool': 12, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 6816, 'qdb_cluster': [6816]} 2024-08-03T20:54:33.985 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-08-03T20:54:33.985 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2024-08-03T20:54:34.219 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:34.217+0000 7f962ce00640 1 -- 172.21.15.32:0/2078276061 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f962806e930 msgr2=0x7f9628144f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:34.219 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:34.217+0000 7f962ce00640 1 --2- 172.21.15.32:0/2078276061 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f962806e930 0x7f9628144f90 secure :-1 s=READY pgs=919 cs=0 l=1 rev1=1 crypto rx=0x7f9618009830 tx=0x7f961802a5c0 comp rx=0 tx=0).stop 2024-08-03T20:54:34.219 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:34.217+0000 7f962ce00640 1 -- 172.21.15.32:0/2078276061 shutdown_connections 2024-08-03T20:54:34.220 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:34.217+0000 7f962ce00640 1 --2- 172.21.15.32:0/2078276061 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f962806e930 0x7f9628144f90 unknown :-1 s=CLOSED pgs=919 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:34.220 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:34.217+0000 7f962ce00640 1 -- 172.21.15.32:0/2078276061 >> 172.21.15.32:0/2078276061 conn(0x7f96280b1bd0 msgr2=0x7f96280b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:54:34.220 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:34.218+0000 7f962ce00640 1 -- 172.21.15.32:0/2078276061 shutdown_connections 2024-08-03T20:54:34.220 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:34.218+0000 7f962ce00640 1 -- 172.21.15.32:0/2078276061 wait complete. 2024-08-03T20:54:34.221 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:34.220+0000 7f962ce00640 1 Processor -- start 2024-08-03T20:54:34.221 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:34.220+0000 7f962ce00640 1 -- start start 2024-08-03T20:54:34.223 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:34.221+0000 7f962ce00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f962806e930 0x7f962815ca20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:34.223 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:34.221+0000 7f962ce00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f9628073e60 0x7f962815b2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:34.223 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:34.221+0000 7f962ce00640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f962815fff0 0x7f962815f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:34.223 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:34.221+0000 7f962ce00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f96280cf840 con 0x7f962806e930 2024-08-03T20:54:34.223 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:34.221+0000 7f962ce00640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f9628145f30 con 0x7f9628073e60 2024-08-03T20:54:34.223 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:34.221+0000 7f962ce00640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7f96280cf9c0 con 0x7f962815fff0 2024-08-03T20:54:34.223 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:34.222+0000 7f9625e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f962806e930 0x7f962815ca20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:34.223 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:34.222+0000 7f9625e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f962806e930 0x7f962815ca20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:40040/0 (socket says 172.21.15.32:40040) 2024-08-03T20:54:34.224 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:34.222+0000 7f9625e00640 1 -- 172.21.15.32:0/4014600898 learned_addr learned my addr 172.21.15.32:0/4014600898 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:54:34.225 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:34.222+0000 7f9626800640 1 --2- 172.21.15.32:0/4014600898 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f962815fff0 0x7f962815f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:34.225 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:34.222+0000 7f961d400640 1 --2- 172.21.15.32:0/4014600898 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f9628073e60 0x7f962815b2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:34.225 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:34.222+0000 7f9625e00640 1 -- 172.21.15.32:0/4014600898 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f962815fff0 msgr2=0x7f962815f0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:34.225 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:34.222+0000 7f9625e00640 1 --2- 172.21.15.32:0/4014600898 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f962815fff0 0x7f962815f0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:34.225 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:34.222+0000 7f9625e00640 1 -- 172.21.15.32:0/4014600898 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f9628073e60 msgr2=0x7f962815b2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:34.225 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:34.222+0000 7f9625e00640 1 --2- 172.21.15.32:0/4014600898 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f9628073e60 0x7f962815b2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:34.225 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:34.222+0000 7f9625e00640 1 -- 172.21.15.32:0/4014600898 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f96281f63c0 con 0x7f962806e930 2024-08-03T20:54:34.225 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:34.223+0000 7f9625e00640 1 --2- 172.21.15.32:0/4014600898 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f962806e930 0x7f962815ca20 secure :-1 s=READY pgs=920 cs=0 l=1 rev1=1 crypto rx=0x7f961802abb0 tx=0x7f9618033c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:54:34.225 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:34.223+0000 7f961ea00640 1 -- 172.21.15.32:0/4014600898 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9618031b60 con 0x7f962806e930 2024-08-03T20:54:34.225 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:34.223+0000 7f961ea00640 1 -- 172.21.15.32:0/4014600898 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f9618031df0 con 0x7f962806e930 2024-08-03T20:54:34.225 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:34.223+0000 7f961ea00640 1 -- 172.21.15.32:0/4014600898 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9618005280 con 0x7f962806e930 2024-08-03T20:54:34.226 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:34.223+0000 7f962ce00640 1 -- 172.21.15.32:0/4014600898 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f962815d530 con 0x7f962806e930 2024-08-03T20:54:34.226 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:34.223+0000 7f962ce00640 1 -- 172.21.15.32:0/4014600898 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f962815d7d0 con 0x7f962806e930 2024-08-03T20:54:34.227 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:34.226+0000 7f961ea00640 1 -- 172.21.15.32:0/4014600898 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f9618005420 con 0x7f962806e930 2024-08-03T20:54:34.228 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:34.227+0000 7f961ea00640 1 --2- 172.21.15.32:0/4014600898 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f9604077040 0x7f9604079550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:34.228 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:34.227+0000 7f961ea00640 1 -- 172.21.15.32:0/4014600898 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 5709+0+0 (secure 0 0 0) 0x7f96180b9ab0 con 0x7f962806e930 2024-08-03T20:54:34.228 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:34.227+0000 7f9633fa1740 1 Processor -- start 2024-08-03T20:54:34.229 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:34.227+0000 7f961d400640 1 --2- 172.21.15.32:0/4014600898 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f9604077040 0x7f9604079550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:34.229 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:34.227+0000 7f9633fa1740 1 -- start start 2024-08-03T20:54:34.229 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:34.228+0000 7f961d400640 1 --2- 172.21.15.32:0/4014600898 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f9604077040 0x7f9604079550 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f9610007fd0 tx=0x7f9610009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:54:34.231 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:34.229+0000 7f9633fa1740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5593c09a1270 0x5593c0d8be90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:34.231 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:34.229+0000 7f9633fa1740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5593c09783b0 0x5593c0d60ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:34.231 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:34.229+0000 7f9633fa1740 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x5593c0980590 0x5593c0d61220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:34.231 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:34.229+0000 7f9633fa1740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x5593c0cb9a60 con 0x5593c09783b0 2024-08-03T20:54:34.231 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:34.229+0000 7f9633fa1740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x5593c0ca8f60 con 0x5593c09a1270 2024-08-03T20:54:34.231 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:34.229+0000 7f9633fa1740 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x5593c0cacfa0 con 0x5593c0980590 2024-08-03T20:54:34.231 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:34.229+0000 7f961d400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5593c09783b0 0x5593c0d60ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:34.231 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:34.229+0000 7f961d400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5593c09783b0 0x5593c0d60ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:40046/0 (socket says 172.21.15.32:40046) 2024-08-03T20:54:34.231 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:34.229+0000 7f961d400640 1 -- 172.21.15.32:0/3558226570 learned_addr learned my addr 172.21.15.32:0/3558226570 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:54:34.232 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:34.229+0000 7f961d400640 1 -- 172.21.15.32:0/3558226570 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x5593c0980590 msgr2=0x5593c0d61220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:54:34.232 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:34.229+0000 7f961d400640 1 --2- 172.21.15.32:0/3558226570 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x5593c0980590 0x5593c0d61220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:34.232 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:34.229+0000 7f961d400640 1 -- 172.21.15.32:0/3558226570 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5593c09a1270 msgr2=0x5593c0d8be90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:54:34.232 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:34.229+0000 7f961d400640 1 --2- 172.21.15.32:0/3558226570 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5593c09a1270 0x5593c0d8be90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:34.232 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:34.229+0000 7f961d400640 1 -- 172.21.15.32:0/3558226570 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5593c0a0c850 con 0x5593c09783b0 2024-08-03T20:54:34.232 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:34.230+0000 7f961d400640 1 --2- 172.21.15.32:0/3558226570 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5593c09783b0 0x5593c0d60ce0 secure :-1 s=READY pgs=921 cs=0 l=1 rev1=1 crypto rx=0x7f9610005240 tx=0x7f9610011000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ab562eff20f8cfa6 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:54:34.233 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:34.230+0000 7f9616a00640 1 -- 172.21.15.32:0/3558226570 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f961000e4a0 con 0x5593c09783b0 2024-08-03T20:54:34.233 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:34.230+0000 7f9616a00640 1 -- 172.21.15.32:0/3558226570 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f961000e640 con 0x5593c09783b0 2024-08-03T20:54:34.233 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:34.230+0000 7f9616a00640 1 -- 172.21.15.32:0/3558226570 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f961000e9f0 con 0x5593c09783b0 2024-08-03T20:54:34.233 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:34.231+0000 7f9633fa1740 1 -- 172.21.15.32:0/3558226570 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5593c09783b0 msgr2=0x5593c0d60ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:34.234 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:34.231+0000 7f9633fa1740 1 --2- 172.21.15.32:0/3558226570 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5593c09783b0 0x5593c0d60ce0 secure :-1 s=READY pgs=921 cs=0 l=1 rev1=1 crypto rx=0x7f9610005240 tx=0x7f9610011000 comp rx=0 tx=0).stop 2024-08-03T20:54:34.234 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:34.231+0000 7f9633fa1740 1 -- 172.21.15.32:0/3558226570 shutdown_connections 2024-08-03T20:54:34.234 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:34.231+0000 7f9633fa1740 1 --2- 172.21.15.32:0/3558226570 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x5593c0980590 0x5593c0d61220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:34.234 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:34.231+0000 7f9633fa1740 1 --2- 172.21.15.32:0/3558226570 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5593c09783b0 0x5593c0d60ce0 unknown :-1 s=CLOSED pgs=921 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:34.234 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:34.231+0000 7f9633fa1740 1 --2- 172.21.15.32:0/3558226570 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5593c09a1270 0x5593c0d8be90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:34.234 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:34.231+0000 7f9633fa1740 1 -- 172.21.15.32:0/3558226570 >> 172.21.15.32:0/3558226570 conn(0x5593c09953c0 msgr2=0x5593c0d47180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:54:34.234 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:34.231+0000 7f9633fa1740 1 -- 172.21.15.32:0/3558226570 shutdown_connections 2024-08-03T20:54:34.234 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:34.231+0000 7f9633fa1740 1 -- 172.21.15.32:0/3558226570 wait complete. 2024-08-03T20:54:34.234 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:34.232+0000 7f9633fa1740 1 -- start start 2024-08-03T20:54:34.235 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:34.232+0000 7f9633fa1740 1 Processor -- start 2024-08-03T20:54:34.237 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:34.234+0000 7f9633fa1740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5593c09783b0 0x5593c0d5bbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:34.237 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:34.234+0000 7f9633fa1740 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x5593c0980590 0x5593c0d5a4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:34.238 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:34.234+0000 7f9633fa1740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5593c09a1270 0x5593c0d863d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:34.238 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:34.234+0000 7f9633fa1740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x5593c0a1b620 con 0x5593c09783b0 2024-08-03T20:54:34.238 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:34.234+0000 7f9633fa1740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x5593c0c46160 con 0x5593c09a1270 2024-08-03T20:54:34.238 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:34.234+0000 7f9633fa1740 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x5593c0d5c130 con 0x5593c0980590 2024-08-03T20:54:34.238 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:34.234+0000 7f9633fa1740 10 client.0 _ll_register_callbacks cb 0x5593c0c556b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-03T20:54:34.238 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:34.234+0000 7f9625e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5593c09783b0 0x5593c0d5bbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:34.238 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:34.234+0000 7f9625e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5593c09783b0 0x5593c0d5bbf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:40050/0 (socket says 172.21.15.32:40050) 2024-08-03T20:54:34.238 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:34.234+0000 7f9625e00640 1 -- 172.21.15.32:0/1242406684 learned_addr learned my addr 172.21.15.32:0/1242406684 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:54:34.238 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:34.234+0000 7f9626800640 1 --2- 172.21.15.32:0/1242406684 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5593c09a1270 0x5593c0d863d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:34.238 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:34.234+0000 7f961d400640 1 --2- 172.21.15.32:0/1242406684 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x5593c0980590 0x5593c0d5a4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:34.238 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:34.235+0000 7f9625e00640 1 -- 172.21.15.32:0/1242406684 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x5593c0980590 msgr2=0x5593c0d5a4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:54:34.239 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:34.235+0000 7f9625e00640 1 --2- 172.21.15.32:0/1242406684 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x5593c0980590 0x5593c0d5a4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:34.239 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:34.235+0000 7f9625e00640 1 -- 172.21.15.32:0/1242406684 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5593c09a1270 msgr2=0x5593c0d863d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:54:34.239 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:34.235+0000 7f9625e00640 1 --2- 172.21.15.32:0/1242406684 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5593c09a1270 0x5593c0d863d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:34.239 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:34.235+0000 7f9625e00640 1 -- 172.21.15.32:0/1242406684 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5593c0a836b0 con 0x5593c09783b0 2024-08-03T20:54:34.239 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:34.235+0000 7f9625e00640 1 --2- 172.21.15.32:0/1242406684 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5593c09783b0 0x5593c0d5bbf0 secure :-1 s=READY pgs=922 cs=0 l=1 rev1=1 crypto rx=0x7f961807f8a0 tx=0x7f9618081da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2056bf18bcdd6b0d server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:54:34.239 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:34.235+0000 7f960b400640 10 client.0 ms_handle_connect on v2:172.21.15.32:3300/0 2024-08-03T20:54:34.239 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:34.235+0000 7f960b400640 1 -- 172.21.15.32:0/1242406684 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96180b2030 con 0x5593c09783b0 2024-08-03T20:54:34.239 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:34.235+0000 7f960b400640 1 -- 172.21.15.32:0/1242406684 <== mon.0 v2:172.21.15.32:3300/0 2 ==== osd_map(83..83 src has 1..83) ==== 5709+0+0 (secure 0 0 0) 0x7f9618082890 con 0x5593c09783b0 2024-08-03T20:54:34.239 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:34.235+0000 7f9633fa1740 1 -- 172.21.15.32:0/1242406684 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f961000e640 con 0x5593c09783b0 2024-08-03T20:54:34.239 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:34.235+0000 7f960b400640 1 -- 172.21.15.32:0/1242406684 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_get_version_reply(handle=1 version=91) ==== 24+0+0 (secure 0 0 0) 0x7f961809b020 con 0x5593c09783b0 2024-08-03T20:54:34.239 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:34.235+0000 7f9633fa1740 10 client.6936 fetch_fsmap learned FSMap version 91 2024-08-03T20:54:34.239 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:34.235+0000 7f9633fa1740 1 -- 172.21.15.32:0/1242406684 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,fsmap=91,monmap=2+}) -- 0x5593c0d5c9d0 con 0x5593c09783b0 2024-08-03T20:54:34.239 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:34.236+0000 7f960b400640 1 -- 172.21.15.32:0/1242406684 <== mon.0 v2:172.21.15.32:3300/0 4 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f961000e640 con 0x5593c09783b0 2024-08-03T20:54:34.239 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:34.236+0000 7f960b400640 1 -- 172.21.15.32:0/1242406684 <== mon.0 v2:172.21.15.32:3300/0 5 ==== fsmap(e 91) ==== 3502+0+0 (secure 0 0 0) 0x7f961808b1c0 con 0x5593c09783b0 2024-08-03T20:54:34.240 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:34.236+0000 7f9633fa1740 10 client.6936 fetch_fsmap finished waiting for FSMap version 91 2024-08-03T20:54:34.240 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:34.236+0000 7f9633fa1740 10 client.6936 resolve_mds: resolved name 'a' to daemon mds.a 2024-08-03T20:54:34.240 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:34.236+0000 7f9633fa1740 20 client.6936 populate_metadata read hostname 'smithi032' 2024-08-03T20:54:34.240 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:34.236+0000 7f9633fa1740 1 --2- 172.21.15.32:0/1242406684 >> [v2:172.21.15.32:6833/1707454554,v1:172.21.15.32:6835/1707454554] conn(0x5593c095fe40 0x5593c0d44cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:34.240 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:34.236+0000 7f9633fa1740 4 client.6936 mds_command: new command op to 6816 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-08-03T20:54:34.240 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:34.236+0000 7f9633fa1740 1 -- 172.21.15.32:0/1242406684 --> [v2:172.21.15.32:6833/1707454554,v1:172.21.15.32:6835/1707454554] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5593c0cbb9b0 con 0x5593c095fe40 2024-08-03T20:54:34.240 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:34.236+0000 7f961d400640 1 --2- 172.21.15.32:0/1242406684 >> [v2:172.21.15.32:6833/1707454554,v1:172.21.15.32:6835/1707454554] conn(0x5593c095fe40 0x5593c0d44cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:34.241 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:34.237+0000 7f961d400640 1 --2- 172.21.15.32:0/1242406684 >> [v2:172.21.15.32:6833/1707454554,v1:172.21.15.32:6835/1707454554] conn(0x5593c095fe40 0x5593c0d44cb0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=12f34f84fc066089 server_cookie=2c116bf62748c79c in_seq=0 out_seq=0 2024-08-03T20:54:34.241 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:34.237+0000 7f960b400640 10 client.6936 ms_handle_connect on v2:172.21.15.32:6833/1707454554 2024-08-03T20:54:34.241 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:34.237+0000 7f960b400640 1 -- 172.21.15.32:0/1242406684 <== mds.0 v2:172.21.15.32:6833/1707454554 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x5593c0cbb9b0 con 0x5593c095fe40 2024-08-03T20:54:34.241 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:34.237+0000 7f960b400640 10 client.6936 handle_command_reply: tid=0 2024-08-03T20:54:34.241 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:34.238+0000 7f9633fa1740 1 client.6936 shutdown 2024-08-03T20:54:34.242 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:34.238+0000 7f9633fa1740 1 -- 172.21.15.32:0/1242406684 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5593c09783b0 msgr2=0x5593c0d5bbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:34.242 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:34.238+0000 7f9633fa1740 1 --2- 172.21.15.32:0/1242406684 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5593c09783b0 0x5593c0d5bbf0 secure :-1 s=READY pgs=922 cs=0 l=1 rev1=1 crypto rx=0x7f961807f8a0 tx=0x7f9618081da0 comp rx=0 tx=0).stop 2024-08-03T20:54:34.242 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:34.238+0000 7f9633fa1740 1 -- 172.21.15.32:0/1242406684 shutdown_connections 2024-08-03T20:54:34.242 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:34.238+0000 7f9633fa1740 1 --2- 172.21.15.32:0/1242406684 >> [v2:172.21.15.32:6833/1707454554,v1:172.21.15.32:6835/1707454554] conn(0x5593c095fe40 0x5593c0d44cb0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:34.242 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:34.238+0000 7f9633fa1740 1 --2- 172.21.15.32:0/1242406684 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5593c09a1270 0x5593c0d863d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:34.242 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:34.238+0000 7f9633fa1740 1 --2- 172.21.15.32:0/1242406684 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x5593c0980590 0x5593c0d5a4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:34.242 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:34.238+0000 7f9633fa1740 1 --2- 172.21.15.32:0/1242406684 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5593c09783b0 0x5593c0d5bbf0 unknown :-1 s=CLOSED pgs=922 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:34.242 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:34.238+0000 7f960b400640 0 client.6936 ms_handle_reset on v2:172.21.15.32:6833/1707454554 2024-08-03T20:54:34.242 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:34.238+0000 7f9633fa1740 1 -- 172.21.15.32:0/1242406684 >> 172.21.15.32:0/1242406684 conn(0x5593c09953c0 msgr2=0x5593c0d41ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:54:34.242 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:34.238+0000 7f9633fa1740 1 -- 172.21.15.32:0/1242406684 shutdown_connections 2024-08-03T20:54:34.242 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:34.238+0000 7f9633fa1740 1 -- 172.21.15.32:0/1242406684 wait complete. 2024-08-03T20:54:34.242 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:34.238+0000 7f9633fa1740 20 client.6936 trim_cache size 0 max 16384 2024-08-03T20:54:34.260 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:34.258+0000 7f9633fa1740 1 Processor -- start 2024-08-03T20:54:34.260 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:34.258+0000 7f9633fa1740 1 -- start start 2024-08-03T20:54:34.262 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:34.260+0000 7f9633fa1740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5593c09783b0 0x5593c0d86240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:34.262 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:34.260+0000 7f9633fa1740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5593c0980590 0x5593c0d84da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:34.263 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:34.260+0000 7f9633fa1740 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x5593c09a1270 0x5593c0d852e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:34.263 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:34.260+0000 7f9633fa1740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f9628200d90 con 0x5593c0980590 2024-08-03T20:54:34.263 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:34.260+0000 7f9633fa1740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f96281420f0 con 0x5593c09783b0 2024-08-03T20:54:34.263 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:34.260+0000 7f9633fa1740 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7f9628142740 con 0x5593c09a1270 2024-08-03T20:54:34.263 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:34.260+0000 7f961d400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5593c0980590 0x5593c0d84da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:34.263 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:34.260+0000 7f961d400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5593c0980590 0x5593c0d84da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:40058/0 (socket says 172.21.15.32:40058) 2024-08-03T20:54:34.264 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:34.260+0000 7f961d400640 1 -- 172.21.15.32:0/2672171175 learned_addr learned my addr 172.21.15.32:0/2672171175 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:54:34.264 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:34.260+0000 7f9625e00640 1 --2- 172.21.15.32:0/2672171175 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5593c09783b0 0x5593c0d86240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:34.264 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:34.260+0000 7f9626800640 1 --2- 172.21.15.32:0/2672171175 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x5593c09a1270 0x5593c0d852e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:34.264 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:34.260+0000 7f961d400640 1 -- 172.21.15.32:0/2672171175 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x5593c09a1270 msgr2=0x5593c0d852e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:54:34.264 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:34.260+0000 7f961d400640 1 --2- 172.21.15.32:0/2672171175 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x5593c09a1270 0x5593c0d852e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:34.264 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:34.260+0000 7f961d400640 1 -- 172.21.15.32:0/2672171175 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5593c09783b0 msgr2=0x5593c0d86240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:54:34.264 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:34.260+0000 7f961d400640 1 --2- 172.21.15.32:0/2672171175 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5593c09783b0 0x5593c0d86240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:34.264 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:34.261+0000 7f961d400640 1 -- 172.21.15.32:0/2672171175 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5593c0d8c400 con 0x5593c0980590 2024-08-03T20:54:34.264 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:34.261+0000 7f961d400640 1 --2- 172.21.15.32:0/2672171175 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5593c0980590 0x5593c0d84da0 secure :-1 s=READY pgs=923 cs=0 l=1 rev1=1 crypto rx=0x7f9610005210 tx=0x7f9610000ed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3b150c7b59798e1a server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:54:34.265 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:34.261+0000 7f9616000640 1 -- 172.21.15.32:0/2672171175 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f961000e560 con 0x5593c0980590 2024-08-03T20:54:34.265 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:34.261+0000 7f9616000640 1 -- 172.21.15.32:0/2672171175 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f9610017040 con 0x5593c0980590 2024-08-03T20:54:34.266 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:34.261+0000 7f9616000640 1 -- 172.21.15.32:0/2672171175 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f961000ec90 con 0x5593c0980590 2024-08-03T20:54:34.266 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:34.262+0000 7f9633fa1740 1 -- 172.21.15.32:0/2672171175 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5593c0980590 msgr2=0x5593c0d84da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:34.266 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:34.262+0000 7f9633fa1740 1 --2- 172.21.15.32:0/2672171175 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5593c0980590 0x5593c0d84da0 secure :-1 s=READY pgs=923 cs=0 l=1 rev1=1 crypto rx=0x7f9610005210 tx=0x7f9610000ed0 comp rx=0 tx=0).stop 2024-08-03T20:54:34.266 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:34.262+0000 7f9633fa1740 1 -- 172.21.15.32:0/2672171175 shutdown_connections 2024-08-03T20:54:34.266 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:34.262+0000 7f9633fa1740 1 --2- 172.21.15.32:0/2672171175 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x5593c09a1270 0x5593c0d852e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:34.266 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:34.262+0000 7f9633fa1740 1 --2- 172.21.15.32:0/2672171175 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5593c0980590 0x5593c0d84da0 unknown :-1 s=CLOSED pgs=923 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:34.266 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:34.262+0000 7f9633fa1740 1 --2- 172.21.15.32:0/2672171175 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5593c09783b0 0x5593c0d86240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:34.266 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:34.262+0000 7f9633fa1740 1 -- 172.21.15.32:0/2672171175 >> 172.21.15.32:0/2672171175 conn(0x5593c09953c0 msgr2=0x5593c0d8b1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:54:34.266 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:34.262+0000 7f9633fa1740 1 -- 172.21.15.32:0/2672171175 shutdown_connections 2024-08-03T20:54:34.266 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:34.262+0000 7f9633fa1740 1 -- 172.21.15.32:0/2672171175 wait complete. 2024-08-03T20:54:34.266 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:34.263+0000 7f9633fa1740 1 -- start start 2024-08-03T20:54:34.267 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:34.263+0000 7f9633fa1740 1 Processor -- start 2024-08-03T20:54:34.270 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:34.265+0000 7f9633fa1740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5593c09783b0 0x5593c0e00380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:34.270 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:34.265+0000 7f9633fa1740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5593c0980590 0x5593c0e0d7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:34.270 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:34.265+0000 7f9633fa1740 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x5593c09a1270 0x5593c0d53660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:34.270 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:34.265+0000 7f9633fa1740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x5593c0d5abc0 con 0x5593c09783b0 2024-08-03T20:54:34.270 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:34.265+0000 7f9633fa1740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x5593c0d716b0 con 0x5593c0980590 2024-08-03T20:54:34.270 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:34.265+0000 7f9633fa1740 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x5593c0d8e0e0 con 0x5593c09a1270 2024-08-03T20:54:34.270 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:34.265+0000 7f9633fa1740 10 client.0 _ll_register_callbacks cb 0x7f9618082400 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-03T20:54:34.270 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:34.265+0000 7f9625e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5593c09783b0 0x5593c0e00380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:34.270 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:34.265+0000 7f9625e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5593c09783b0 0x5593c0e00380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:40060/0 (socket says 172.21.15.32:40060) 2024-08-03T20:54:34.270 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:34.265+0000 7f9625e00640 1 -- 172.21.15.32:0/193493220 learned_addr learned my addr 172.21.15.32:0/193493220 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:54:34.270 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:34.265+0000 7f961d400640 1 --2- 172.21.15.32:0/193493220 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5593c0980590 0x5593c0e0d7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:34.271 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:34.265+0000 7f9626800640 1 --2- 172.21.15.32:0/193493220 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x5593c09a1270 0x5593c0d53660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:34.271 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:34.265+0000 7f9625e00640 1 -- 172.21.15.32:0/193493220 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x5593c09a1270 msgr2=0x5593c0d53660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:54:34.271 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:34.265+0000 7f9625e00640 1 --2- 172.21.15.32:0/193493220 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x5593c09a1270 0x5593c0d53660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:34.271 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:34.265+0000 7f9625e00640 1 -- 172.21.15.32:0/193493220 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5593c0980590 msgr2=0x5593c0e0d7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:54:34.271 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:34.265+0000 7f9625e00640 1 --2- 172.21.15.32:0/193493220 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5593c0980590 0x5593c0e0d7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:34.271 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:34.265+0000 7f9625e00640 1 -- 172.21.15.32:0/193493220 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5593c0d53270 con 0x5593c09783b0 2024-08-03T20:54:34.271 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:34.266+0000 7f9625e00640 1 --2- 172.21.15.32:0/193493220 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5593c09783b0 0x5593c0e00380 secure :-1 s=READY pgs=924 cs=0 l=1 rev1=1 crypto rx=0x7f9618031790 tx=0x7f9618082b40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fa23f688b5913152 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:54:34.271 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:34.266+0000 7f960b400640 10 client.0 ms_handle_connect on v2:172.21.15.32:3300/0 2024-08-03T20:54:34.271 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:34.266+0000 7f960b400640 1 -- 172.21.15.32:0/193493220 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96180b2060 con 0x5593c09783b0 2024-08-03T20:54:34.271 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:34.266+0000 7f960b400640 1 -- 172.21.15.32:0/193493220 <== mon.0 v2:172.21.15.32:3300/0 2 ==== osd_map(83..83 src has 1..83) ==== 5709+0+0 (secure 0 0 0) 0x7f961807dfa0 con 0x5593c09783b0 2024-08-03T20:54:34.271 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:34.266+0000 7f9633fa1740 1 -- 172.21.15.32:0/193493220 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9610017040 con 0x5593c09783b0 2024-08-03T20:54:34.271 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:34.266+0000 7f960b400640 1 -- 172.21.15.32:0/193493220 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_get_version_reply(handle=1 version=91) ==== 24+0+0 (secure 0 0 0) 0x7f9618090050 con 0x5593c09783b0 2024-08-03T20:54:34.271 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:34.266+0000 7f9633fa1740 10 client.6942 fetch_fsmap learned FSMap version 91 2024-08-03T20:54:34.271 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:34.266+0000 7f9633fa1740 1 -- 172.21.15.32:0/193493220 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,fsmap=91,monmap=2+}) -- 0x5593c0d53270 con 0x5593c09783b0 2024-08-03T20:54:34.272 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:34.266+0000 7f960b400640 1 -- 172.21.15.32:0/193493220 <== mon.0 v2:172.21.15.32:3300/0 4 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f9610017040 con 0x5593c09783b0 2024-08-03T20:54:34.272 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:34.266+0000 7f960b400640 1 -- 172.21.15.32:0/193493220 <== mon.0 v2:172.21.15.32:3300/0 5 ==== fsmap(e 91) ==== 3502+0+0 (secure 0 0 0) 0x7f9618093020 con 0x5593c09783b0 2024-08-03T20:54:34.272 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:34.266+0000 7f9633fa1740 10 client.6942 fetch_fsmap finished waiting for FSMap version 91 2024-08-03T20:54:34.272 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:34.266+0000 7f9633fa1740 10 client.6942 resolve_mds: resolved name 'a' to daemon mds.a 2024-08-03T20:54:34.272 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:34.266+0000 7f9633fa1740 20 client.6942 populate_metadata read hostname 'smithi032' 2024-08-03T20:54:34.272 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:34.266+0000 7f9633fa1740 1 --2- 172.21.15.32:0/193493220 >> [v2:172.21.15.32:6833/1707454554,v1:172.21.15.32:6835/1707454554] conn(0x5593c0d934d0 0x5593c0da45a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:34.272 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:34.267+0000 7f9633fa1740 4 client.6942 mds_command: new command op to 6816 tid=0 multi_id=0 [{"prefix": "status"}] 2024-08-03T20:54:34.272 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:34.267+0000 7f9633fa1740 1 -- 172.21.15.32:0/193493220 --> [v2:172.21.15.32:6833/1707454554,v1:172.21.15.32:6835/1707454554] -- command(tid 0: {"prefix": "status"}) -- 0x5593c0cb7450 con 0x5593c0d934d0 2024-08-03T20:54:34.272 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:34.267+0000 7f961d400640 1 --2- 172.21.15.32:0/193493220 >> [v2:172.21.15.32:6833/1707454554,v1:172.21.15.32:6835/1707454554] conn(0x5593c0d934d0 0x5593c0da45a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:34.272 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:34.267+0000 7f961d400640 1 --2- 172.21.15.32:0/193493220 >> [v2:172.21.15.32:6833/1707454554,v1:172.21.15.32:6835/1707454554] conn(0x5593c0d934d0 0x5593c0da45a0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=82df4cc218d5c7dd server_cookie=be2bc5bd5ed9f189 in_seq=0 out_seq=0 2024-08-03T20:54:34.272 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:34.267+0000 7f960b400640 10 client.6942 ms_handle_connect on v2:172.21.15.32:6833/1707454554 2024-08-03T20:54:34.272 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:34.268+0000 7f960b400640 1 -- 172.21.15.32:0/193493220 <== mds.0 v2:172.21.15.32:6833/1707454554 1 ==== command_reply(tid 0: 0 ) ==== 8+0+325 (crc 0 0 0) 0x5593c0cb7450 con 0x5593c0d934d0 2024-08-03T20:54:34.272 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:34.268+0000 7f960b400640 10 client.6942 handle_command_reply: tid=0 2024-08-03T20:54:34.273 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:34.268+0000 7f9633fa1740 1 client.6942 shutdown 2024-08-03T20:54:34.273 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:34.268+0000 7f9633fa1740 1 -- 172.21.15.32:0/193493220 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5593c09783b0 msgr2=0x5593c0e00380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:34.273 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:34.268+0000 7f9633fa1740 1 --2- 172.21.15.32:0/193493220 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5593c09783b0 0x5593c0e00380 secure :-1 s=READY pgs=924 cs=0 l=1 rev1=1 crypto rx=0x7f9618031790 tx=0x7f9618082b40 comp rx=0 tx=0).stop 2024-08-03T20:54:34.273 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:34.268+0000 7f9633fa1740 1 -- 172.21.15.32:0/193493220 shutdown_connections 2024-08-03T20:54:34.273 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:34.268+0000 7f9633fa1740 1 --2- 172.21.15.32:0/193493220 >> [v2:172.21.15.32:6833/1707454554,v1:172.21.15.32:6835/1707454554] conn(0x5593c0d934d0 0x5593c0da45a0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:34.273 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:34.268+0000 7f9633fa1740 1 --2- 172.21.15.32:0/193493220 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5593c0980590 0x5593c0e0d7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:34.273 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:34.268+0000 7f9633fa1740 1 --2- 172.21.15.32:0/193493220 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x5593c09a1270 0x5593c0d53660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:34.273 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:34.268+0000 7f960b400640 0 client.6942 ms_handle_reset on v2:172.21.15.32:6833/1707454554 2024-08-03T20:54:34.273 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:34.268+0000 7f9633fa1740 1 --2- 172.21.15.32:0/193493220 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5593c09783b0 0x5593c0e00380 unknown :-1 s=CLOSED pgs=924 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:34.273 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:34.268+0000 7f9633fa1740 1 -- 172.21.15.32:0/193493220 >> 172.21.15.32:0/193493220 conn(0x5593c09953c0 msgr2=0x5593c0d46cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:54:34.273 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:34.268+0000 7f9633fa1740 1 -- 172.21.15.32:0/193493220 shutdown_connections 2024-08-03T20:54:34.273 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:34.268+0000 7f9633fa1740 1 -- 172.21.15.32:0/193493220 wait complete. 2024-08-03T20:54:34.273 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:34.268+0000 7f9633fa1740 20 client.6942 trim_cache size 0 max 16384 2024-08-03T20:54:34.274 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-08-03T20:54:34.274 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_fsid": "40ab141a-01d7-49b9-ae3c-2edec9876e95", 2024-08-03T20:54:34.274 INFO:teuthology.orchestra.run.smithi032.stdout: "whoami": 0, 2024-08-03T20:54:34.274 INFO:teuthology.orchestra.run.smithi032.stdout: "id": 6816, 2024-08-03T20:54:34.274 INFO:teuthology.orchestra.run.smithi032.stdout: "want_state": "up:active", 2024-08-03T20:54:34.274 INFO:teuthology.orchestra.run.smithi032.stdout: "state": "up:active", 2024-08-03T20:54:34.274 INFO:teuthology.orchestra.run.smithi032.stdout: "fs_name": "cephfs", 2024-08-03T20:54:34.274 INFO:teuthology.orchestra.run.smithi032.stdout: "rank_uptime": 6.8489153659999999, 2024-08-03T20:54:34.274 INFO:teuthology.orchestra.run.smithi032.stdout: "mdsmap_epoch": 91, 2024-08-03T20:54:34.275 INFO:teuthology.orchestra.run.smithi032.stdout: "osdmap_epoch": 83, 2024-08-03T20:54:34.275 INFO:teuthology.orchestra.run.smithi032.stdout: "osdmap_epoch_barrier": 83, 2024-08-03T20:54:34.275 INFO:teuthology.orchestra.run.smithi032.stdout: "uptime": 18.825716559 2024-08-03T20:54:34.275 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-08-03T20:54:34.276 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:34.269+0000 7f962ce00640 1 -- 172.21.15.32:0/4014600898 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f9604077040 msgr2=0x7f9604079550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:34.276 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:34.269+0000 7f962ce00640 1 --2- 172.21.15.32:0/4014600898 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f9604077040 0x7f9604079550 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f9610007fd0 tx=0x7f9610009040 comp rx=0 tx=0).stop 2024-08-03T20:54:34.276 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:34.269+0000 7f962ce00640 1 -- 172.21.15.32:0/4014600898 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f962806e930 msgr2=0x7f962815ca20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:34.276 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:34.269+0000 7f962ce00640 1 --2- 172.21.15.32:0/4014600898 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f962806e930 0x7f962815ca20 secure :-1 s=READY pgs=920 cs=0 l=1 rev1=1 crypto rx=0x7f961802abb0 tx=0x7f9618033c10 comp rx=0 tx=0).stop 2024-08-03T20:54:34.276 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:34.269+0000 7f962ce00640 1 -- 172.21.15.32:0/4014600898 shutdown_connections 2024-08-03T20:54:34.276 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:34.269+0000 7f962ce00640 1 --2- 172.21.15.32:0/4014600898 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f9604077040 0x7f9604079550 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:34.276 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:34.269+0000 7f962ce00640 1 --2- 172.21.15.32:0/4014600898 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f9628073e60 0x7f962815b2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:34.276 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:34.269+0000 7f962ce00640 1 --2- 172.21.15.32:0/4014600898 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f962815fff0 0x7f962815f0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:34.276 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:34.269+0000 7f962ce00640 1 --2- 172.21.15.32:0/4014600898 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f962806e930 0x7f962815ca20 unknown :-1 s=CLOSED pgs=920 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:34.276 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:34.269+0000 7f962ce00640 1 -- 172.21.15.32:0/4014600898 >> 172.21.15.32:0/4014600898 conn(0x7f96280b1bd0 msgr2=0x7f9628074270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:54:34.276 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:34.269+0000 7f962ce00640 1 -- 172.21.15.32:0/4014600898 shutdown_connections 2024-08-03T20:54:34.277 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:34.269+0000 7f962ce00640 1 -- 172.21.15.32:0/4014600898 wait complete. 2024-08-03T20:54:34.297 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-08-03T20:54:34.526 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:34.524+0000 7ff8ac600640 1 -- 172.21.15.32:0/3165733350 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff8a415fff0 msgr2=0x7ff8a4160400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:34.526 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:34.524+0000 7ff8ac600640 1 --2- 172.21.15.32:0/3165733350 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff8a415fff0 0x7ff8a4160400 secure :-1 s=READY pgs=926 cs=0 l=1 rev1=1 crypto rx=0x7ff8a40afaf0 tx=0x7ff89c02a8b0 comp rx=0 tx=0).stop 2024-08-03T20:54:34.526 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:34.524+0000 7ff8ac600640 1 -- 172.21.15.32:0/3165733350 shutdown_connections 2024-08-03T20:54:34.526 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:34.524+0000 7ff8ac600640 1 --2- 172.21.15.32:0/3165733350 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff8a415fff0 0x7ff8a4160400 unknown :-1 s=CLOSED pgs=926 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:34.526 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:34.524+0000 7ff8ac600640 1 --2- 172.21.15.32:0/3165733350 >> v2:172.21.15.118:3300/0 conn(0x7ff8a4144f90 0x7ff8a4074570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:34.527 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:34.524+0000 7ff8ac600640 1 -- 172.21.15.32:0/3165733350 >> 172.21.15.32:0/3165733350 conn(0x7ff8a40b1bd0 msgr2=0x7ff8a40b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:54:34.527 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:34.524+0000 7ff8ac600640 1 -- 172.21.15.32:0/3165733350 shutdown_connections 2024-08-03T20:54:34.527 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:34.524+0000 7ff8ac600640 1 -- 172.21.15.32:0/3165733350 wait complete. 2024-08-03T20:54:34.528 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:34.526+0000 7ff8ac600640 1 Processor -- start 2024-08-03T20:54:34.528 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:34.527+0000 7ff8ac600640 1 -- start start 2024-08-03T20:54:34.530 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:34.528+0000 7ff8ac600640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7ff8a406e930 0x7ff8a41f5d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:34.530 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:34.528+0000 7ff8ac600640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ff8a4144f90 0x7ff8a4143ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:34.530 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:34.528+0000 7ff8ac600640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff8a415fff0 0x7ff8a40cc540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:34.530 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:34.528+0000 7ff8ac600640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7ff8a40cf840 con 0x7ff8a415fff0 2024-08-03T20:54:34.530 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:34.528+0000 7ff8ac600640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7ff8a4145f30 con 0x7ff8a4144f90 2024-08-03T20:54:34.530 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:34.528+0000 7ff8ac600640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7ff8a40cf9c0 con 0x7ff8a406e930 2024-08-03T20:54:34.530 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:34.528+0000 7ff8aa600640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff8a415fff0 0x7ff8a40cc540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:34.530 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:34.528+0000 7ff8aa600640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff8a415fff0 0x7ff8a40cc540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:40084/0 (socket says 172.21.15.32:40084) 2024-08-03T20:54:34.531 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:34.528+0000 7ff8aa600640 1 -- 172.21.15.32:0/857063407 learned_addr learned my addr 172.21.15.32:0/857063407 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:54:34.531 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:34.528+0000 7ff8a9c00640 1 --2- 172.21.15.32:0/857063407 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7ff8a406e930 0x7ff8a41f5d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:34.531 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:34.528+0000 7ff8a9200640 1 --2- 172.21.15.32:0/857063407 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ff8a4144f90 0x7ff8a4143ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:34.532 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:34.529+0000 7ff8aa600640 1 -- 172.21.15.32:0/857063407 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7ff8a406e930 msgr2=0x7ff8a41f5d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:34.532 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:34.529+0000 7ff8aa600640 1 --2- 172.21.15.32:0/857063407 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7ff8a406e930 0x7ff8a41f5d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:34.532 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:34.529+0000 7ff8aa600640 1 -- 172.21.15.32:0/857063407 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ff8a4144f90 msgr2=0x7ff8a4143ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:34.532 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:34.529+0000 7ff8aa600640 1 --2- 172.21.15.32:0/857063407 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ff8a4144f90 0x7ff8a4143ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:34.532 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:34.529+0000 7ff8aa600640 1 -- 172.21.15.32:0/857063407 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff8a41f31b0 con 0x7ff8a415fff0 2024-08-03T20:54:34.532 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:34.529+0000 7ff8aa600640 1 --2- 172.21.15.32:0/857063407 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff8a415fff0 0x7ff8a40cc540 secure :-1 s=READY pgs=927 cs=0 l=1 rev1=1 crypto rx=0x7ff89c005fd0 tx=0x7ff89c0316d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:54:34.532 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:34.529+0000 7ff8a1600640 1 -- 172.21.15.32:0/857063407 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff89c031990 con 0x7ff8a415fff0 2024-08-03T20:54:34.532 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:34.529+0000 7ff8a1600640 1 -- 172.21.15.32:0/857063407 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7ff89c031c20 con 0x7ff8a415fff0 2024-08-03T20:54:34.533 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:34.529+0000 7ff8ac600640 1 -- 172.21.15.32:0/857063407 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff8a41f3490 con 0x7ff8a415fff0 2024-08-03T20:54:34.533 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:34.530+0000 7ff8a1600640 1 -- 172.21.15.32:0/857063407 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff89c033070 con 0x7ff8a415fff0 2024-08-03T20:54:34.533 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:34.530+0000 7ff8ac600640 1 -- 172.21.15.32:0/857063407 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff8a41f3a50 con 0x7ff8a415fff0 2024-08-03T20:54:34.533 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:34.531+0000 7ff8ac600640 1 -- 172.21.15.32:0/857063407 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff8a4160c30 con 0x7ff8a415fff0 2024-08-03T20:54:34.533 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:34.532+0000 7ff8a1600640 1 -- 172.21.15.32:0/857063407 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7ff89c033210 con 0x7ff8a415fff0 2024-08-03T20:54:34.534 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:34.533+0000 7ff8a1600640 1 --2- 172.21.15.32:0/857063407 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7ff884077010 0x7ff884079520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:34.534 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:34.533+0000 7ff8a1600640 1 -- 172.21.15.32:0/857063407 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 5709+0+0 (secure 0 0 0) 0x7ff89c0b9c50 con 0x7ff8a415fff0 2024-08-03T20:54:34.535 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:34.533+0000 7ff8a9c00640 1 --2- 172.21.15.32:0/857063407 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7ff884077010 0x7ff884079520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:34.536 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:34.534+0000 7ff8a9c00640 1 --2- 172.21.15.32:0/857063407 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7ff884077010 0x7ff884079520 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7ff898007fd0 tx=0x7ff898001370 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:54:34.539 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:34.538+0000 7ff8a1600640 1 -- 172.21.15.32:0/857063407 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7ff89c03f030 con 0x7ff8a415fff0 2024-08-03T20:54:34.783 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:34.782+0000 7ff8ac600640 1 -- 172.21.15.32:0/857063407 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7ff8a40b2180 con 0x7ff8a415fff0 2024-08-03T20:54:34.784 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:34.783+0000 7ff8a1600640 1 -- 172.21.15.32:0/857063407 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v91) ==== 53+0+73 (secure 0 0 0) 0x7ff89c082760 con 0x7ff8a415fff0 2024-08-03T20:54:34.784 INFO:teuthology.orchestra.run.smithi032.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-08-03T20:54:34.787 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:34.785+0000 7ff8ac600640 1 -- 172.21.15.32:0/857063407 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7ff884077010 msgr2=0x7ff884079520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:34.787 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:34.785+0000 7ff8ac600640 1 --2- 172.21.15.32:0/857063407 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7ff884077010 0x7ff884079520 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7ff898007fd0 tx=0x7ff898001370 comp rx=0 tx=0).stop 2024-08-03T20:54:34.787 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:34.785+0000 7ff8ac600640 1 -- 172.21.15.32:0/857063407 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff8a415fff0 msgr2=0x7ff8a40cc540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:34.787 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:34.785+0000 7ff8ac600640 1 --2- 172.21.15.32:0/857063407 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff8a415fff0 0x7ff8a40cc540 secure :-1 s=READY pgs=927 cs=0 l=1 rev1=1 crypto rx=0x7ff89c005fd0 tx=0x7ff89c0316d0 comp rx=0 tx=0).stop 2024-08-03T20:54:34.787 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:34.785+0000 7ff8ac600640 1 -- 172.21.15.32:0/857063407 shutdown_connections 2024-08-03T20:54:34.787 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:34.785+0000 7ff8ac600640 1 --2- 172.21.15.32:0/857063407 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7ff884077010 0x7ff884079520 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:34.787 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:34.785+0000 7ff8ac600640 1 --2- 172.21.15.32:0/857063407 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ff8a4144f90 0x7ff8a4143ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:34.787 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:34.785+0000 7ff8ac600640 1 --2- 172.21.15.32:0/857063407 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff8a415fff0 0x7ff8a40cc540 unknown :-1 s=CLOSED pgs=927 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:34.787 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:34.785+0000 7ff8ac600640 1 --2- 172.21.15.32:0/857063407 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7ff8a406e930 0x7ff8a41f5d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:34.788 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:34.785+0000 7ff8ac600640 1 -- 172.21.15.32:0/857063407 >> 172.21.15.32:0/857063407 conn(0x7ff8a40b1bd0 msgr2=0x7ff8a4140cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:54:34.788 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:34.785+0000 7ff8ac600640 1 -- 172.21.15.32:0/857063407 shutdown_connections 2024-08-03T20:54:34.788 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:34.785+0000 7ff8ac600640 1 -- 172.21.15.32:0/857063407 wait complete. 2024-08-03T20:54:34.809 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-08-03T20:54:34.809 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-08-03T20:54:34.809 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi032.front.sepia.ceph.com 2024-08-03T20:54:34.809 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-08-03T20:54:34.809 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-08-03T20:54:34.810 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-08-03T20:54:34.810 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2024-08-03T20:54:34.810 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-08-03T20:54:34.810 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-08-03T20:54:34.810 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-08-03T20:54:34.810 DEBUG:teuthology.orchestra.run.smithi032:> ip addr 2024-08-03T20:54:34.846 INFO:teuthology.orchestra.run.smithi032.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-08-03T20:54:34.846 INFO:teuthology.orchestra.run.smithi032.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-08-03T20:54:34.846 INFO:teuthology.orchestra.run.smithi032.stdout: inet 127.0.0.1/8 scope host lo 2024-08-03T20:54:34.846 INFO:teuthology.orchestra.run.smithi032.stdout: valid_lft forever preferred_lft forever 2024-08-03T20:54:34.847 INFO:teuthology.orchestra.run.smithi032.stdout: inet6 ::1/128 scope host 2024-08-03T20:54:34.847 INFO:teuthology.orchestra.run.smithi032.stdout: valid_lft forever preferred_lft forever 2024-08-03T20:54:34.847 INFO:teuthology.orchestra.run.smithi032.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-08-03T20:54:34.847 INFO:teuthology.orchestra.run.smithi032.stdout: link/ether 0c:c4:7a:6b:fd:0c brd ff:ff:ff:ff:ff:ff 2024-08-03T20:54:34.847 INFO:teuthology.orchestra.run.smithi032.stdout: altname enp7s0f0 2024-08-03T20:54:34.847 INFO:teuthology.orchestra.run.smithi032.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-08-03T20:54:34.847 INFO:teuthology.orchestra.run.smithi032.stdout: link/ether 0c:c4:7a:6b:fd:0d brd ff:ff:ff:ff:ff:ff 2024-08-03T20:54:34.847 INFO:teuthology.orchestra.run.smithi032.stdout: altname enp7s0f1 2024-08-03T20:54:34.847 INFO:teuthology.orchestra.run.smithi032.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-08-03T20:54:34.847 INFO:teuthology.orchestra.run.smithi032.stdout: link/ether ac:1f:6b:f8:26:02 brd ff:ff:ff:ff:ff:ff 2024-08-03T20:54:34.847 INFO:teuthology.orchestra.run.smithi032.stdout: altname enp1s0f0 2024-08-03T20:54:34.847 INFO:teuthology.orchestra.run.smithi032.stdout: inet 172.21.15.32/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-08-03T20:54:34.847 INFO:teuthology.orchestra.run.smithi032.stdout: valid_lft 42081sec preferred_lft 42081sec 2024-08-03T20:54:34.848 INFO:teuthology.orchestra.run.smithi032.stdout: inet6 fe80::ae1f:6bff:fef8:2602/64 scope link noprefixroute 2024-08-03T20:54:34.848 INFO:teuthology.orchestra.run.smithi032.stdout: valid_lft forever preferred_lft forever 2024-08-03T20:54:34.848 INFO:teuthology.orchestra.run.smithi032.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-08-03T20:54:34.848 INFO:teuthology.orchestra.run.smithi032.stdout: link/ether ac:1f:6b:f8:26:03 brd ff:ff:ff:ff:ff:ff 2024-08-03T20:54:34.848 INFO:teuthology.orchestra.run.smithi032.stdout: altname enp1s0f1 2024-08-03T20:54:34.848 INFO:teuthology.orchestra.run.smithi032.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-08-03T20:54:34.848 INFO:teuthology.orchestra.run.smithi032.stdout: link/ether 0a:23:2b:ca:2a:93 brd ff:ff:ff:ff:ff:ff 2024-08-03T20:54:34.848 INFO:teuthology.orchestra.run.smithi032.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-08-03T20:54:34.848 INFO:teuthology.orchestra.run.smithi032.stdout: valid_lft forever preferred_lft forever 2024-08-03T20:54:34.848 INFO:teuthology.orchestra.run.smithi032.stdout: inet6 fe80::d804:7cff:fed9:7ba4/64 scope link 2024-08-03T20:54:34.848 INFO:teuthology.orchestra.run.smithi032.stdout: valid_lft forever preferred_lft forever 2024-08-03T20:54:34.848 INFO:teuthology.orchestra.run.smithi032.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-08-03T20:54:34.848 INFO:teuthology.orchestra.run.smithi032.stdout: link/ether 22:86:28:30:4e:62 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-08-03T20:54:34.848 INFO:teuthology.orchestra.run.smithi032.stdout: inet6 fe80::2086:28ff:fe30:4e62/64 scope link 2024-08-03T20:54:34.849 INFO:teuthology.orchestra.run.smithi032.stdout: valid_lft forever preferred_lft forever 2024-08-03T20:54:34.849 INFO:teuthology.orchestra.run.smithi032.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-08-03T20:54:34.849 INFO:teuthology.orchestra.run.smithi032.stdout: link/ether 0a:23:2b:ca:2a:93 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-08-03T20:54:34.849 INFO:teuthology.orchestra.run.smithi032.stdout: inet6 fe80::823:2bff:feca:2a93/64 scope link 2024-08-03T20:54:34.849 INFO:teuthology.orchestra.run.smithi032.stdout: valid_lft forever preferred_lft forever 2024-08-03T20:54:34.849 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-08-03T20:54:34.849 DEBUG:teuthology.orchestra.run.smithi032:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-08-03T20:54:34.939 INFO:teuthology.orchestra.run.smithi032.stdout:1 2024-08-03T20:54:34.942 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-08-03T20:54:34.942 DEBUG:teuthology.orchestra.run.smithi032:> route 2024-08-03T20:54:34.978 INFO:teuthology.orchestra.run.smithi032.stdout:Kernel IP routing table 2024-08-03T20:54:34.978 INFO:teuthology.orchestra.run.smithi032.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-08-03T20:54:34.978 INFO:teuthology.orchestra.run.smithi032.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-08-03T20:54:34.978 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-08-03T20:54:34.978 INFO:teuthology.orchestra.run.smithi032.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-08-03T20:54:34.978 DEBUG:teuthology.orchestra.run.smithi032:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-08-03T20:54:34.979 DEBUG:teuthology.orchestra.run.smithi032:> set -e 2024-08-03T20:54:34.979 DEBUG:teuthology.orchestra.run.smithi032:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-08-03T20:54:34.979 DEBUG:teuthology.orchestra.run.smithi032:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-08-03T20:54:34.979 DEBUG:teuthology.orchestra.run.smithi032:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-08-03T20:54:34.979 DEBUG:teuthology.orchestra.run.smithi032:> ') 2024-08-03T20:54:35.098 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:35 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-08-03T20:54:35.170 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:35 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-08-03T20:54:35.179 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-08-03T20:54:35.179 DEBUG:teuthology.orchestra.run.smithi032:> ip netns list 2024-08-03T20:54:35.214 INFO:teuthology.orchestra.run.smithi032.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-08-03T20:54:35.215 INFO:teuthology.orchestra.run.smithi032.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-08-03T20:54:35.215 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-08-03T20:54:35.215 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-08-03T20:54:35.215 DEBUG:teuthology.orchestra.run.smithi032:> sudo ip link set brx.0 up 2024-08-03T20:54:35.312 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-08-03T20:54:35.312 DEBUG:teuthology.orchestra.run.smithi032:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-08-03T20:54:35.346 INFO:teuthology.orchestra.run.smithi032.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-08-03T20:54:35.347 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-08-03T20:54:35.347 DEBUG:teuthology.orchestra.run.smithi032:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-08-03T20:54:35.422 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2024-08-03T20:54:35.422 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-08-03T20:54:35.422 DEBUG:teuthology.orchestra.run.smithi032:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-08-03T20:54:35.621 INFO:teuthology.orchestra.run.smithi032.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-08-03T20:54:35.621 INFO:teuthology.orchestra.run.smithi032.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-08-03T20:54:35.621 INFO:teuthology.orchestra.run.smithi032.stdout:invalid new device string format 2024-08-03T20:54:35.641 INFO:teuthology.orchestra.run.smithi032.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-08-03T20:54:35.641 INFO:teuthology.orchestra.run.smithi032.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-08-03T20:54:35.641 INFO:teuthology.orchestra.run.smithi032.stdout:invalid new device string format 2024-08-03T20:54:35.641 INFO:teuthology.orchestra.run.smithi032.stdout:mount.ceph: resolved to: "172.21.15.32:3300,172.21.15.118:3300,172.21.15.118:3301" 2024-08-03T20:54:35.641 INFO:teuthology.orchestra.run.smithi032.stdout:mount.ceph: switching to using v1 address with old syntax 2024-08-03T20:54:35.670 INFO:teuthology.orchestra.run.smithi032.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-08-03T20:54:35.671 INFO:teuthology.orchestra.run.smithi032.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-08-03T20:54:35.671 INFO:teuthology.orchestra.run.smithi032.stdout:invalid new device string format 2024-08-03T20:54:35.671 INFO:teuthology.orchestra.run.smithi032.stdout:mount.ceph: resolved to: "172.21.15.32:3300,172.21.15.118:3300,172.21.15.118:3301" 2024-08-03T20:54:35.671 INFO:teuthology.orchestra.run.smithi032.stdout:mount.ceph: switching to using v1 address with old syntax 2024-08-03T20:54:35.671 INFO:teuthology.orchestra.run.smithi032.stdout:mount.ceph: resolved to: "172.21.15.32:6789,172.21.15.118:6789,172.21.15.118:6790" 2024-08-03T20:54:35.671 INFO:teuthology.orchestra.run.smithi032.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.32:6789,172.21.15.118:6789,172.21.15.118:6790:/ 2024-08-03T20:54:35.671 INFO:teuthology.orchestra.run.smithi032.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,key=0,fsid=40ab141a-01d7-49b9-ae3c-2edec9876e95" will pass to kernel 2024-08-03T20:54:35.672 INFO:teuthology.orchestra.run.smithi032.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2024-08-03T20:54:35.672 INFO:teuthology.orchestra.run.smithi032.stdout: You just mounted a file system that supports labels which does not 2024-08-03T20:54:35.672 INFO:teuthology.orchestra.run.smithi032.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-08-03T20:54:35.672 INFO:teuthology.orchestra.run.smithi032.stdout: applications will generate AVC messages and not be allowed access to 2024-08-03T20:54:35.673 INFO:teuthology.orchestra.run.smithi032.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-08-03T20:54:35.675 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-08-03T20:54:35.675 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-08-03T20:54:35.676 DEBUG:teuthology.orchestra.run.smithi032:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-08-03T20:54:35.732 DEBUG:teuthology.orchestra.run.smithi032:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-08-03T20:54:35.814 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:35 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-08-03T20:54:35.821 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:35 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-08-03T20:54:35.829 INFO:teuthology.orchestra.run.smithi032.stdout:client6056 2024-08-03T20:54:35.829 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-08-03T20:54:35.829 DEBUG:teuthology.orchestra.run.smithi032:> dd if=/proc/self/mounts of=/dev/stdout 2024-08-03T20:54:35.864 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-08-03T20:54:35.865 DEBUG:teuthology.orchestra.run.smithi032:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir subdir) 2024-08-03T20:54:35.946 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-08-03T20:54:35.946 DEBUG:teuthology.orchestra.run.smithi032:> dd if=/proc/self/mounts of=/dev/stdout 2024-08-03T20:54:36.021 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-08-03T20:54:36.022 DEBUG:teuthology.orchestra.run.smithi032:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=subdir/sixmegs bs=1M conv=fdatasync count=6 seek=0) 2024-08-03T20:54:36.170 INFO:teuthology.orchestra.run.smithi032.stderr:6+0 records in 2024-08-03T20:54:36.170 INFO:teuthology.orchestra.run.smithi032.stderr:6+0 records out 2024-08-03T20:54:36.170 INFO:teuthology.orchestra.run.smithi032.stderr:6291456 bytes (6.3 MB, 6.0 MiB) copied, 0.0722862 s, 87.0 MB/s 2024-08-03T20:54:36.171 DEBUG:teuthology.orchestra.run.smithi032:> stdin-killer -- python3 -c ' 2024-08-03T20:54:36.171 DEBUG:teuthology.orchestra.run.smithi032:> import os 2024-08-03T20:54:36.171 DEBUG:teuthology.orchestra.run.smithi032:> import stat 2024-08-03T20:54:36.171 DEBUG:teuthology.orchestra.run.smithi032:> import json 2024-08-03T20:54:36.172 DEBUG:teuthology.orchestra.run.smithi032:> import sys 2024-08-03T20:54:36.172 DEBUG:teuthology.orchestra.run.smithi032:> 2024-08-03T20:54:36.172 DEBUG:teuthology.orchestra.run.smithi032:> try: 2024-08-03T20:54:36.172 DEBUG:teuthology.orchestra.run.smithi032:> s = os.stat('"'"'/home/ubuntu/cephtest/mnt.0/subdir/sixmegs'"'"') 2024-08-03T20:54:36.172 DEBUG:teuthology.orchestra.run.smithi032:> except OSError as e: 2024-08-03T20:54:36.172 DEBUG:teuthology.orchestra.run.smithi032:> sys.exit(e.errno) 2024-08-03T20:54:36.172 DEBUG:teuthology.orchestra.run.smithi032:> 2024-08-03T20:54:36.172 DEBUG:teuthology.orchestra.run.smithi032:> attrs = ["st_mode", "st_ino", "st_dev", "st_nlink", "st_uid", "st_gid", "st_size", "st_atime", "st_mtime", "st_ctime"] 2024-08-03T20:54:36.172 DEBUG:teuthology.orchestra.run.smithi032:> print(json.dumps( 2024-08-03T20:54:36.172 DEBUG:teuthology.orchestra.run.smithi032:> dict([(a, getattr(s, a)) for a in attrs]), 2024-08-03T20:54:36.172 DEBUG:teuthology.orchestra.run.smithi032:> indent=2)) 2024-08-03T20:54:36.172 DEBUG:teuthology.orchestra.run.smithi032:> ' 2024-08-03T20:54:36.275 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-08-03T20:54:36.276 INFO:teuthology.orchestra.run.smithi032.stdout: "st_mode": 33188, 2024-08-03T20:54:36.276 INFO:teuthology.orchestra.run.smithi032.stdout: "st_ino": 1099511627777, 2024-08-03T20:54:36.276 INFO:teuthology.orchestra.run.smithi032.stdout: "st_dev": 41, 2024-08-03T20:54:36.276 INFO:teuthology.orchestra.run.smithi032.stdout: "st_nlink": 1, 2024-08-03T20:54:36.276 INFO:teuthology.orchestra.run.smithi032.stdout: "st_uid": 1000, 2024-08-03T20:54:36.276 INFO:teuthology.orchestra.run.smithi032.stdout: "st_gid": 1316, 2024-08-03T20:54:36.276 INFO:teuthology.orchestra.run.smithi032.stdout: "st_size": 6291456, 2024-08-03T20:54:36.276 INFO:teuthology.orchestra.run.smithi032.stdout: "st_atime": 1722718476.0954933, 2024-08-03T20:54:36.276 INFO:teuthology.orchestra.run.smithi032.stdout: "st_mtime": 1722718476.1324925, 2024-08-03T20:54:36.276 INFO:teuthology.orchestra.run.smithi032.stdout: "st_ctime": 1722718476.1324925 2024-08-03T20:54:36.276 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-08-03T20:54:36.279 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:36 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-08-03T20:54:36.287 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-08-03T20:54:36.287 DEBUG:teuthology.orchestra.run.smithi032:> dd if=/proc/self/mounts of=/dev/stdout 2024-08-03T20:54:36.363 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-08-03T20:54:36.364 DEBUG:teuthology.orchestra.run.smithi032:> dd if=/proc/self/mounts of=/dev/stdout 2024-08-03T20:54:36.439 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-08-03T20:54:36.708 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:36.707+0000 7fcc79c00640 1 -- 172.21.15.32:0/371576155 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcc58003a00 msgr2=0x7fcc58003e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:36.709 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:36.707+0000 7fcc79c00640 1 --2- 172.21.15.32:0/371576155 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcc58003a00 0x7fcc58003e70 secure :-1 s=READY pgs=929 cs=0 l=1 rev1=1 crypto rx=0x7fcc6400c2e0 tx=0x7fcc58005830 comp rx=0 tx=0).stop 2024-08-03T20:54:36.709 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:36.707+0000 7fcc79200640 1 -- 172.21.15.32:0/371576155 reap_dead start 2024-08-03T20:54:36.709 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:36.707+0000 7fcc79c00640 1 -- 172.21.15.32:0/371576155 shutdown_connections 2024-08-03T20:54:36.709 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:36.707+0000 7fcc79c00640 1 -- 172.21.15.32:0/371576155 >> 172.21.15.32:0/371576155 conn(0x7fcc740b2120 msgr2=0x7fcc740cfd60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:54:36.709 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:36.707+0000 7fcc79c00640 1 -- 172.21.15.32:0/371576155 shutdown_connections 2024-08-03T20:54:36.709 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:36.707+0000 7fcc79c00640 1 -- 172.21.15.32:0/371576155 wait complete. 2024-08-03T20:54:36.710 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:36.709+0000 7fcc79c00640 1 Processor -- start 2024-08-03T20:54:36.710 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:36.709+0000 7fcc79c00640 1 -- start start 2024-08-03T20:54:36.713 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:36.710+0000 7fcc79c00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fcc58003a00 0x7fcc7414c0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:36.713 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:36.711+0000 7fcc79c00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcc741532d0 0x7fcc74160360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:36.713 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:36.711+0000 7fcc79c00640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fcc741608a0 0x7fcc74154780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:36.713 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:36.711+0000 7fcc79c00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fcc7406fbe0 con 0x7fcc741532d0 2024-08-03T20:54:36.713 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:36.711+0000 7fcc79c00640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7fcc7406fa60 con 0x7fcc58003a00 2024-08-03T20:54:36.713 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:36.711+0000 7fcc79c00640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7fcc7415fe50 con 0x7fcc741608a0 2024-08-03T20:54:36.713 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:36.711+0000 7fcc6be00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcc741532d0 0x7fcc74160360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:36.713 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:36.711+0000 7fcc6be00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcc741532d0 0x7fcc74160360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:40114/0 (socket says 172.21.15.32:40114) 2024-08-03T20:54:36.714 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:36.711+0000 7fcc6be00640 1 -- 172.21.15.32:0/1209379656 learned_addr learned my addr 172.21.15.32:0/1209379656 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:54:36.714 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:36.711+0000 7fcc79200640 1 --2- 172.21.15.32:0/1209379656 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fcc741608a0 0x7fcc74154780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:36.714 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:36.711+0000 7fcc73e00640 1 --2- 172.21.15.32:0/1209379656 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fcc58003a00 0x7fcc7414c0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:36.714 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:36.711+0000 7fcc6be00640 1 -- 172.21.15.32:0/1209379656 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fcc741608a0 msgr2=0x7fcc74154780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:36.714 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:36.711+0000 7fcc6be00640 1 --2- 172.21.15.32:0/1209379656 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fcc741608a0 0x7fcc74154780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:36.714 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:36.711+0000 7fcc6be00640 1 -- 172.21.15.32:0/1209379656 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fcc58003a00 msgr2=0x7fcc7414c0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:36.714 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:36.711+0000 7fcc6be00640 1 --2- 172.21.15.32:0/1209379656 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fcc58003a00 0x7fcc7414c0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:36.714 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:36.711+0000 7fcc6be00640 1 -- 172.21.15.32:0/1209379656 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcc74160cb0 con 0x7fcc741532d0 2024-08-03T20:54:36.714 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:36.711+0000 7fcc79200640 1 --2- 172.21.15.32:0/1209379656 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fcc741608a0 0x7fcc74154780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-03T20:54:36.714 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:36.711+0000 7fcc73e00640 1 --2- 172.21.15.32:0/1209379656 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fcc58003a00 0x7fcc7414c0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-03T20:54:36.714 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:36.712+0000 7fcc6be00640 1 --2- 172.21.15.32:0/1209379656 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcc741532d0 0x7fcc74160360 secure :-1 s=READY pgs=930 cs=0 l=1 rev1=1 crypto rx=0x7fcc5c011a80 tx=0x7fcc5c011f40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:54:36.715 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:36.712+0000 7fcc70c00640 1 -- 172.21.15.32:0/1209379656 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc5c00fd50 con 0x7fcc741532d0 2024-08-03T20:54:36.715 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:36.712+0000 7fcc70c00640 1 -- 172.21.15.32:0/1209379656 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fcc5c016070 con 0x7fcc741532d0 2024-08-03T20:54:36.715 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:36.712+0000 7fcc70c00640 1 -- 172.21.15.32:0/1209379656 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc5c0126b0 con 0x7fcc741532d0 2024-08-03T20:54:36.716 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:36.712+0000 7fcc79c00640 1 -- 172.21.15.32:0/1209379656 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcc740b2c30 con 0x7fcc741532d0 2024-08-03T20:54:36.716 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:36.712+0000 7fcc79c00640 1 -- 172.21.15.32:0/1209379656 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcc741f8ae0 con 0x7fcc741532d0 2024-08-03T20:54:36.716 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:36.713+0000 7fcc79c00640 1 -- 172.21.15.32:0/1209379656 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcc7406e520 con 0x7fcc741532d0 2024-08-03T20:54:36.716 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:36.714+0000 7fcc70c00640 1 -- 172.21.15.32:0/1209379656 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7fcc5c018030 con 0x7fcc741532d0 2024-08-03T20:54:36.717 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:36.716+0000 7fcc70c00640 1 --2- 172.21.15.32:0/1209379656 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fcc50077010 0x7fcc50079520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:36.717 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:36.716+0000 7fcc70c00640 1 -- 172.21.15.32:0/1209379656 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 5709+0+0 (secure 0 0 0) 0x7fcc5c0991f0 con 0x7fcc741532d0 2024-08-03T20:54:36.718 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:36.717+0000 7fcc73e00640 1 --2- 172.21.15.32:0/1209379656 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fcc50077010 0x7fcc50079520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:36.718 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:36.717+0000 7fcc73e00640 1 --2- 172.21.15.32:0/1209379656 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fcc50077010 0x7fcc50079520 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7fcc64033000 tx=0x7fcc6402d000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:54:36.722 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:36.721+0000 7fcc70c00640 1 -- 172.21.15.32:0/1209379656 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7fcc74160cb0 con 0x7fcc741532d0 2024-08-03T20:54:36.940 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:36.939+0000 7fcc79c00640 1 -- 172.21.15.32:0/1209379656 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7fcc740ad190 con 0x7fcc741532d0 2024-08-03T20:54:36.941 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:36.939+0000 7fcc70c00640 1 -- 172.21.15.32:0/1209379656 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v83) ==== 80+0+0 (secure 0 0 0) 0x7fcc5c0623e0 con 0x7fcc741532d0 2024-08-03T20:54:36.941 INFO:teuthology.orchestra.run.smithi032.stderr:listed 0 entries 2024-08-03T20:54:36.943 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:36.941+0000 7fcc79c00640 1 -- 172.21.15.32:0/1209379656 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fcc50077010 msgr2=0x7fcc50079520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:36.943 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:36.941+0000 7fcc79c00640 1 --2- 172.21.15.32:0/1209379656 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fcc50077010 0x7fcc50079520 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7fcc64033000 tx=0x7fcc6402d000 comp rx=0 tx=0).stop 2024-08-03T20:54:36.943 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:36.941+0000 7fcc79c00640 1 -- 172.21.15.32:0/1209379656 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcc741532d0 msgr2=0x7fcc74160360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:36.944 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:36.941+0000 7fcc79c00640 1 --2- 172.21.15.32:0/1209379656 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcc741532d0 0x7fcc74160360 secure :-1 s=READY pgs=930 cs=0 l=1 rev1=1 crypto rx=0x7fcc5c011a80 tx=0x7fcc5c011f40 comp rx=0 tx=0).stop 2024-08-03T20:54:36.944 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:36.941+0000 7fcc79c00640 1 -- 172.21.15.32:0/1209379656 shutdown_connections 2024-08-03T20:54:36.944 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:36.941+0000 7fcc79c00640 1 --2- 172.21.15.32:0/1209379656 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fcc50077010 0x7fcc50079520 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:36.944 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:36.941+0000 7fcc79c00640 1 --2- 172.21.15.32:0/1209379656 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fcc741608a0 0x7fcc74154780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:36.944 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:36.941+0000 7fcc79c00640 1 --2- 172.21.15.32:0/1209379656 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcc741532d0 0x7fcc74160360 unknown :-1 s=CLOSED pgs=930 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:36.944 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:36.941+0000 7fcc79c00640 1 --2- 172.21.15.32:0/1209379656 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fcc58003a00 0x7fcc7414c0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:36.944 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:36.941+0000 7fcc79c00640 1 -- 172.21.15.32:0/1209379656 >> 172.21.15.32:0/1209379656 conn(0x7fcc740b2120 msgr2=0x7fcc740ca7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:54:36.944 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:36.942+0000 7fcc79c00640 1 -- 172.21.15.32:0/1209379656 shutdown_connections 2024-08-03T20:54:36.944 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:36.942+0000 7fcc79c00640 1 -- 172.21.15.32:0/1209379656 wait complete. 2024-08-03T20:54:36.965 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-08-03T20:54:36.965 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-08-03T20:54:36.966 DEBUG:teuthology.orchestra.run.smithi032:> sudo umount /home/ubuntu/cephtest/mnt.0 2024-08-03T20:54:37.048 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi032.front.sepia.ceph.com 2024-08-03T20:54:37.048 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-08-03T20:54:37.048 DEBUG:teuthology.orchestra.run.smithi032:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-08-03T20:54:37.106 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-08-03T20:54:37.335 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:37.333+0000 7f76bc000640 1 -- 172.21.15.32:0/1897990958 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f76b4073e60 msgr2=0x7f76b4074270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:37.335 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:37.333+0000 7f76bc000640 1 --2- 172.21.15.32:0/1897990958 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f76b4073e60 0x7f76b4074270 secure :-1 s=READY pgs=932 cs=0 l=1 rev1=1 crypto rx=0x7f76b4071700 tx=0x7f76a002a4a0 comp rx=0 tx=0).stop 2024-08-03T20:54:37.335 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:37.333+0000 7f76bc000640 1 -- 172.21.15.32:0/1897990958 shutdown_connections 2024-08-03T20:54:37.335 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:37.333+0000 7f76bc000640 1 --2- 172.21.15.32:0/1897990958 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f76b4073e60 0x7f76b4074270 unknown :-1 s=CLOSED pgs=932 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:37.335 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:37.333+0000 7f76bc000640 1 --2- 172.21.15.32:0/1897990958 >> v2:172.21.15.118:3300/0 conn(0x7f76b406e930 0x7f76b4144f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:37.335 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:37.333+0000 7f76bc000640 1 -- 172.21.15.32:0/1897990958 >> 172.21.15.32:0/1897990958 conn(0x7f76b40b1bd0 msgr2=0x7f76b40b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:54:37.335 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:37.333+0000 7f76bc000640 1 -- 172.21.15.32:0/1897990958 shutdown_connections 2024-08-03T20:54:37.335 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:37.333+0000 7f76bc000640 1 -- 172.21.15.32:0/1897990958 wait complete. 2024-08-03T20:54:37.336 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:37.335+0000 7f76bc000640 1 Processor -- start 2024-08-03T20:54:37.336 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:37.335+0000 7f76bc000640 1 -- start start 2024-08-03T20:54:37.339 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:37.336+0000 7f76bc000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f76b406e930 0x7f76b4142530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:37.339 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:37.336+0000 7f76bc000640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f76b4073e60 0x7f76b41f8be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:37.339 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:37.337+0000 7f76bc000640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f76b40747b0 0x7f76b414c5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:37.339 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:37.337+0000 7f76bc000640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f76b40cf9c0 con 0x7f76b406e930 2024-08-03T20:54:37.339 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:37.337+0000 7f76bc000640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f76b40cf840 con 0x7f76b40747b0 2024-08-03T20:54:37.339 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:37.337+0000 7f76bc000640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7f76b40ccff0 con 0x7f76b4073e60 2024-08-03T20:54:37.339 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:37.337+0000 7f76b9600640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f76b406e930 0x7f76b4142530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:37.339 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:37.337+0000 7f76b9600640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f76b406e930 0x7f76b4142530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:40142/0 (socket says 172.21.15.32:40142) 2024-08-03T20:54:37.339 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:37.337+0000 7f76b9600640 1 -- 172.21.15.32:0/1785931454 learned_addr learned my addr 172.21.15.32:0/1785931454 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:54:37.339 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:37.337+0000 7f76b8c00640 1 --2- 172.21.15.32:0/1785931454 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f76b4073e60 0x7f76b41f8be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:37.339 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:37.337+0000 7f76ba000640 1 --2- 172.21.15.32:0/1785931454 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f76b40747b0 0x7f76b414c5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:37.340 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:37.337+0000 7f76b9600640 1 -- 172.21.15.32:0/1785931454 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f76b4073e60 msgr2=0x7f76b41f8be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:37.340 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:37.337+0000 7f76b9600640 1 --2- 172.21.15.32:0/1785931454 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f76b4073e60 0x7f76b41f8be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:37.340 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:37.337+0000 7f76b9600640 1 -- 172.21.15.32:0/1785931454 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f76b40747b0 msgr2=0x7f76b414c5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:37.340 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:37.337+0000 7f76b9600640 1 --2- 172.21.15.32:0/1785931454 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f76b40747b0 0x7f76b414c5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:37.340 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:37.337+0000 7f76b9600640 1 -- 172.21.15.32:0/1785931454 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f76b4142a70 con 0x7f76b406e930 2024-08-03T20:54:37.340 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:37.338+0000 7f76b9600640 1 --2- 172.21.15.32:0/1785931454 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f76b406e930 0x7f76b4142530 secure :-1 s=READY pgs=933 cs=0 l=1 rev1=1 crypto rx=0x7f76a800e7c0 tx=0x7f76a800ec80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:54:37.341 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:37.338+0000 7f76b1600640 1 -- 172.21.15.32:0/1785931454 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76a800ccc0 con 0x7f76b406e930 2024-08-03T20:54:37.341 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:37.338+0000 7f76b1600640 1 -- 172.21.15.32:0/1785931454 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f76a8010430 con 0x7f76b406e930 2024-08-03T20:54:37.341 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:37.338+0000 7f76bc000640 1 -- 172.21.15.32:0/1785931454 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f76b4142d40 con 0x7f76b406e930 2024-08-03T20:54:37.341 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:37.338+0000 7f76b1600640 1 -- 172.21.15.32:0/1785931454 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76a8010860 con 0x7f76b406e930 2024-08-03T20:54:37.341 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:37.338+0000 7f76bc000640 1 -- 172.21.15.32:0/1785931454 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f76b4143300 con 0x7f76b406e930 2024-08-03T20:54:37.341 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:37.339+0000 7f76bc000640 1 -- 172.21.15.32:0/1785931454 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f76b40cd580 con 0x7f76b406e930 2024-08-03T20:54:37.342 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:37.340+0000 7f76b1600640 1 -- 172.21.15.32:0/1785931454 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f76a8010a00 con 0x7f76b406e930 2024-08-03T20:54:37.342 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:37.341+0000 7f76b1600640 1 --2- 172.21.15.32:0/1785931454 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f7694077010 0x7f7694079520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:37.343 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:37.341+0000 7f76b1600640 1 -- 172.21.15.32:0/1785931454 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 5709+0+0 (secure 0 0 0) 0x7f76a8099360 con 0x7f76b406e930 2024-08-03T20:54:37.343 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:37.342+0000 7f76b8c00640 1 --2- 172.21.15.32:0/1785931454 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f7694077010 0x7f7694079520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:37.344 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:37.343+0000 7f76b8c00640 1 --2- 172.21.15.32:0/1785931454 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f7694077010 0x7f7694079520 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f76a0034000 tx=0x7f76a0004730 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:54:37.347 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:37.346+0000 7f76b1600640 1 -- 172.21.15.32:0/1785931454 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7f76a80624c0 con 0x7f76b406e930 2024-08-03T20:54:37.594 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:37.592+0000 7f76bc000640 1 -- 172.21.15.32:0/1785931454 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f76b4074590 con 0x7f76b406e930 2024-08-03T20:54:37.594 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:37.593+0000 7f76b1600640 1 -- 172.21.15.32:0/1785931454 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 91 v91) ==== 94+0+5228 (secure 0 0 0) 0x7f76a8061dc0 con 0x7f76b406e930 2024-08-03T20:54:37.594 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-03T20:54:37.595 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":91,"btime":"2024-08-03T20:54:30:445075+0000","default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6439,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6835/531080160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6834","nonce":531080160},{"type":"v1","addr":"172.21.15.118:6835","nonce":531080160}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":84},{"gid":6454,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6837/2796882319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6836","nonce":2796882319},{"type":"v1","addr":"172.21.15.118:6837","nonce":2796882319}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":86},{"gid":6813,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6834/2686933890","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":2686933890},{"type":"v1","addr":"172.21.15.32:6834","nonce":2686933890}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":85}],"filesystems":[{"mdsmap":{"epoch":91,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-08-03T20:54:26.950518+0000","modified":"2024-08-03T20:54:29.976506+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6816},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6816":{"gid":6816,"name":"a","rank":0,"incarnation":87,"state":"up:active","state_seq":5,"addr":"172.21.15.32:6835/1707454554","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6833","nonce":1707454554},{"type":"v1","addr":"172.21.15.32:6835","nonce":1707454554}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6816,"qdb_cluster":[6816]},"id":6}]} 2024-08-03T20:54:37.595 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 91 2024-08-03T20:54:37.597 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:37.595+0000 7f76bc000640 1 -- 172.21.15.32:0/1785931454 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f7694077010 msgr2=0x7f7694079520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:37.597 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:37.595+0000 7f76bc000640 1 --2- 172.21.15.32:0/1785931454 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f7694077010 0x7f7694079520 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f76a0034000 tx=0x7f76a0004730 comp rx=0 tx=0).stop 2024-08-03T20:54:37.597 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:37.595+0000 7f76bc000640 1 -- 172.21.15.32:0/1785931454 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f76b406e930 msgr2=0x7f76b4142530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:37.597 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:37.595+0000 7f76bc000640 1 --2- 172.21.15.32:0/1785931454 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f76b406e930 0x7f76b4142530 secure :-1 s=READY pgs=933 cs=0 l=1 rev1=1 crypto rx=0x7f76a800e7c0 tx=0x7f76a800ec80 comp rx=0 tx=0).stop 2024-08-03T20:54:37.597 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:37.595+0000 7f76bc000640 1 -- 172.21.15.32:0/1785931454 shutdown_connections 2024-08-03T20:54:37.597 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:37.595+0000 7f76bc000640 1 --2- 172.21.15.32:0/1785931454 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f7694077010 0x7f7694079520 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:37.597 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:37.595+0000 7f76bc000640 1 --2- 172.21.15.32:0/1785931454 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f76b40747b0 0x7f76b414c5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:37.597 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:37.595+0000 7f76bc000640 1 --2- 172.21.15.32:0/1785931454 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f76b4073e60 0x7f76b41f8be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:37.598 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:37.595+0000 7f76bc000640 1 --2- 172.21.15.32:0/1785931454 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f76b406e930 0x7f76b4142530 unknown :-1 s=CLOSED pgs=933 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:37.598 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:37.595+0000 7f76bc000640 1 -- 172.21.15.32:0/1785931454 >> 172.21.15.32:0/1785931454 conn(0x7f76b40b1bd0 msgr2=0x7f76b40b4150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:54:37.598 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:37.595+0000 7f76bc000640 1 -- 172.21.15.32:0/1785931454 shutdown_connections 2024-08-03T20:54:37.598 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:37.595+0000 7f76bc000640 1 -- 172.21.15.32:0/1785931454 wait complete. 2024-08-03T20:54:37.621 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json flush journal 2024-08-03T20:54:37.843 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-08-03T20:54:37.848 DEBUG:teuthology.orchestra.run.smithi118:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-08-03T20:54:37.882 INFO:teuthology.orchestra.run.smithi032.stdout:{"message":"","return_code":0} 2024-08-03T20:54:37.882 DEBUG:tasks.cephfs.filesystem:_json_asok output { "message": "", "return_code": 0 } 2024-08-03T20:54:37.882 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-08-03T20:54:38.153 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.151+0000 7fc344000640 1 -- 172.21.15.32:0/3673059072 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc33c15f610 msgr2=0x7fc33c0cdf70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:38.153 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.151+0000 7fc344000640 1 --2- 172.21.15.32:0/3673059072 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc33c15f610 0x7fc33c0cdf70 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7fc334005fd0 tx=0x7fc33402aa90 comp rx=0 tx=0).stop 2024-08-03T20:54:38.153 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.151+0000 7fc344000640 1 -- 172.21.15.32:0/3673059072 shutdown_connections 2024-08-03T20:54:38.153 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.151+0000 7fc344000640 1 --2- 172.21.15.32:0/3673059072 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc33c15f610 0x7fc33c0cdf70 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:38.153 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.151+0000 7fc344000640 1 --2- 172.21.15.32:0/3673059072 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fc33c0b38b0 0x7fc33c1523f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:38.153 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.151+0000 7fc344000640 1 -- 172.21.15.32:0/3673059072 >> 172.21.15.32:0/3673059072 conn(0x7fc33c0b1bd0 msgr2=0x7fc33c0b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:54:38.153 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.151+0000 7fc344000640 1 -- 172.21.15.32:0/3673059072 shutdown_connections 2024-08-03T20:54:38.153 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.151+0000 7fc344000640 1 -- 172.21.15.32:0/3673059072 wait complete. 2024-08-03T20:54:38.155 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.153+0000 7fc344000640 1 Processor -- start 2024-08-03T20:54:38.155 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.153+0000 7fc344000640 1 -- start start 2024-08-03T20:54:38.157 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.155+0000 7fc344000640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc33c0b38b0 0x7fc33c1580c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:38.157 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.155+0000 7fc344000640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fc33c0b4ce0 0x7fc33c155490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:38.157 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.155+0000 7fc344000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc33c1e6010 0x7fc33c150e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:38.157 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.155+0000 7fc344000640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fc33c0cea30 con 0x7fc33c1e6010 2024-08-03T20:54:38.157 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.155+0000 7fc344000640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7fc33c0ce8b0 con 0x7fc33c0b38b0 2024-08-03T20:54:38.157 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.155+0000 7fc344000640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7fc33c0cebb0 con 0x7fc33c0b4ce0 2024-08-03T20:54:38.158 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.155+0000 7fc342000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc33c1e6010 0x7fc33c150e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:38.158 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.155+0000 7fc342000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc33c1e6010 0x7fc33c150e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:40148/0 (socket says 172.21.15.32:40148) 2024-08-03T20:54:38.158 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.155+0000 7fc342000640 1 -- 172.21.15.32:0/1111383917 learned_addr learned my addr 172.21.15.32:0/1111383917 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:54:38.158 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.155+0000 7fc341600640 1 --2- 172.21.15.32:0/1111383917 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc33c0b38b0 0x7fc33c1580c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:38.158 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.155+0000 7fc340c00640 1 --2- 172.21.15.32:0/1111383917 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fc33c0b4ce0 0x7fc33c155490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:38.158 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.156+0000 7fc342000640 1 -- 172.21.15.32:0/1111383917 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fc33c0b4ce0 msgr2=0x7fc33c155490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:38.158 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.156+0000 7fc342000640 1 --2- 172.21.15.32:0/1111383917 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fc33c0b4ce0 0x7fc33c155490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:38.158 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.156+0000 7fc342000640 1 -- 172.21.15.32:0/1111383917 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc33c0b38b0 msgr2=0x7fc33c1580c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:38.158 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.156+0000 7fc342000640 1 --2- 172.21.15.32:0/1111383917 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc33c0b38b0 0x7fc33c1580c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:38.158 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.156+0000 7fc342000640 1 -- 172.21.15.32:0/1111383917 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc33c156220 con 0x7fc33c1e6010 2024-08-03T20:54:38.158 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.156+0000 7fc340c00640 1 --2- 172.21.15.32:0/1111383917 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fc33c0b4ce0 0x7fc33c155490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-03T20:54:38.158 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.156+0000 7fc342000640 1 --2- 172.21.15.32:0/1111383917 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc33c1e6010 0x7fc33c150e30 secure :-1 s=READY pgs=934 cs=0 l=1 rev1=1 crypto rx=0x7fc334009d90 tx=0x7fc334003c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:54:38.159 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.156+0000 7fc339600640 1 -- 172.21.15.32:0/1111383917 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc334044070 con 0x7fc33c1e6010 2024-08-03T20:54:38.159 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.156+0000 7fc339600640 1 -- 172.21.15.32:0/1111383917 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fc334003e00 con 0x7fc33c1e6010 2024-08-03T20:54:38.159 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.156+0000 7fc339600640 1 -- 172.21.15.32:0/1111383917 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3340044b0 con 0x7fc33c1e6010 2024-08-03T20:54:38.159 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.156+0000 7fc344000640 1 -- 172.21.15.32:0/1111383917 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc33c156500 con 0x7fc33c1e6010 2024-08-03T20:54:38.159 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.156+0000 7fc344000640 1 -- 172.21.15.32:0/1111383917 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc33c156ac0 con 0x7fc33c1e6010 2024-08-03T20:54:38.160 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.158+0000 7fc344000640 1 -- 172.21.15.32:0/1111383917 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc33c0cf0d0 con 0x7fc33c1e6010 2024-08-03T20:54:38.160 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.159+0000 7fc339600640 1 -- 172.21.15.32:0/1111383917 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7fc334047070 con 0x7fc33c1e6010 2024-08-03T20:54:38.161 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.160+0000 7fc339600640 1 --2- 172.21.15.32:0/1111383917 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fc31c077010 0x7fc31c079520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:38.161 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.160+0000 7fc339600640 1 -- 172.21.15.32:0/1111383917 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 5709+0+0 (secure 0 0 0) 0x7fc3340b9700 con 0x7fc33c1e6010 2024-08-03T20:54:38.161 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.160+0000 7fc341600640 1 --2- 172.21.15.32:0/1111383917 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fc31c077010 0x7fc31c079520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:38.162 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.161+0000 7fc341600640 1 --2- 172.21.15.32:0/1111383917 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fc31c077010 0x7fc31c079520 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7fc3300095f0 tx=0x7fc330005f90 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:54:38.165 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.164+0000 7fc339600640 1 -- 172.21.15.32:0/1111383917 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7fc334032030 con 0x7fc33c1e6010 2024-08-03T20:54:38.409 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.407+0000 7fc344000640 1 -- 172.21.15.32:0/1111383917 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7fc33c15b0f0 con 0x7fc33c1e6010 2024-08-03T20:54:38.409 INFO:tasks.ceph.mon.a.smithi032.stderr:2024-08-03T20:54:38.408+0000 7f4e6d600640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-08-03T20:54:38.411 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.410+0000 7fc339600640 1 -- 172.21.15.32:0/1111383917 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v92) ==== 161+0+0 (secure 0 0 0) 0x7fc3340098b0 con 0x7fc33c1e6010 2024-08-03T20:54:38.411 INFO:teuthology.orchestra.run.smithi032.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-08-03T20:54:38.415 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.413+0000 7fc344000640 1 -- 172.21.15.32:0/1111383917 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fc31c077010 msgr2=0x7fc31c079520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:38.415 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.413+0000 7fc344000640 1 --2- 172.21.15.32:0/1111383917 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fc31c077010 0x7fc31c079520 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7fc3300095f0 tx=0x7fc330005f90 comp rx=0 tx=0).stop 2024-08-03T20:54:38.415 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.413+0000 7fc344000640 1 -- 172.21.15.32:0/1111383917 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc33c1e6010 msgr2=0x7fc33c150e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:38.415 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.413+0000 7fc344000640 1 --2- 172.21.15.32:0/1111383917 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc33c1e6010 0x7fc33c150e30 secure :-1 s=READY pgs=934 cs=0 l=1 rev1=1 crypto rx=0x7fc334009d90 tx=0x7fc334003c10 comp rx=0 tx=0).stop 2024-08-03T20:54:38.415 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.413+0000 7fc344000640 1 -- 172.21.15.32:0/1111383917 shutdown_connections 2024-08-03T20:54:38.415 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.413+0000 7fc344000640 1 --2- 172.21.15.32:0/1111383917 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fc31c077010 0x7fc31c079520 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:38.415 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.413+0000 7fc344000640 1 --2- 172.21.15.32:0/1111383917 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc33c1e6010 0x7fc33c150e30 unknown :-1 s=CLOSED pgs=934 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:38.415 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.413+0000 7fc344000640 1 --2- 172.21.15.32:0/1111383917 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fc33c0b4ce0 0x7fc33c155490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:38.415 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.413+0000 7fc344000640 1 --2- 172.21.15.32:0/1111383917 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc33c0b38b0 0x7fc33c1580c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:38.416 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.413+0000 7fc344000640 1 -- 172.21.15.32:0/1111383917 >> 172.21.15.32:0/1111383917 conn(0x7fc33c0b1bd0 msgr2=0x7fc33c159530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:54:38.416 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.413+0000 7fc344000640 1 -- 172.21.15.32:0/1111383917 shutdown_connections 2024-08-03T20:54:38.416 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.414+0000 7fc344000640 1 -- 172.21.15.32:0/1111383917 wait complete. 2024-08-03T20:54:38.423 INFO:tasks.ceph.mds.a.smithi032.stderr: -4875> 2024-08-03T20:54:32.383+0000 7f7cfba00640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-08-03T20:54:38.439 DEBUG:teuthology.orchestra.run.smithi032:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata export - 2024-08-03T20:54:38.552 INFO:tasks.ceph.mds.a.smithi032.stdout:starting mds.a at 2024-08-03T20:54:38.589 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.588+0000 7f9dde35ef40 1 Processor -- start 2024-08-03T20:54:38.589 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.588+0000 7f9dde35ef40 1 -- start start 2024-08-03T20:54:38.592 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.589+0000 7f9dde35ef40 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55dce19c9010 0x55dce1a56600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:38.592 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.589+0000 7f9dde35ef40 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55dce19caf50 0x55dce1981300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:38.592 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.589+0000 7f9dde35ef40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55dce1a53820 0x55dce1981840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:38.592 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.589+0000 7f9dde35ef40 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55dce19c96f0 con 0x55dce19caf50 2024-08-03T20:54:38.592 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.589+0000 7f9dde35ef40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x55dce18cfe30 con 0x55dce1a53820 2024-08-03T20:54:38.592 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.589+0000 7f9dde35ef40 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x55dce19ada10 con 0x55dce19c9010 2024-08-03T20:54:38.592 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.589+0000 7f9dd6a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55dce19caf50 0x55dce1981300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:38.592 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.589+0000 7f9dd6a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55dce19caf50 0x55dce1981300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:40176/0 (socket says 172.21.15.32:40176) 2024-08-03T20:54:38.592 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.589+0000 7f9dd6a00640 1 -- 172.21.15.32:0/590178320 learned_addr learned my addr 172.21.15.32:0/590178320 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:54:38.593 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.590+0000 7f9dd7e00640 1 --2- 172.21.15.32:0/590178320 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55dce1a53820 0x55dce1981840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:38.593 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.590+0000 7f9dd7400640 1 --2- 172.21.15.32:0/590178320 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55dce19c9010 0x55dce1a56600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:38.593 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.590+0000 7f9dd6a00640 1 -- 172.21.15.32:0/590178320 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55dce19c9010 msgr2=0x55dce1a56600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:54:38.593 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.590+0000 7f9dd6a00640 1 --2- 172.21.15.32:0/590178320 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55dce19c9010 0x55dce1a56600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:38.593 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.590+0000 7f9dd6a00640 1 -- 172.21.15.32:0/590178320 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55dce1a53820 msgr2=0x55dce1981840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:54:38.593 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.590+0000 7f9dd6a00640 1 --2- 172.21.15.32:0/590178320 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55dce1a53820 0x55dce1981840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:38.593 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.590+0000 7f9dd6a00640 1 -- 172.21.15.32:0/590178320 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55dce18a8f10 con 0x55dce19caf50 2024-08-03T20:54:38.593 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.590+0000 7f9dd7400640 1 --2- 172.21.15.32:0/590178320 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55dce19c9010 0x55dce1a56600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-03T20:54:38.593 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.590+0000 7f9dd7e00640 1 --2- 172.21.15.32:0/590178320 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55dce1a53820 0x55dce1981840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-03T20:54:38.593 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.590+0000 7f9dd6a00640 1 --2- 172.21.15.32:0/590178320 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55dce19caf50 0x55dce1981300 secure :-1 s=READY pgs=938 cs=0 l=1 rev1=1 crypto rx=0x7f9dc400b630 tx=0x7f9dc400baf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5fc94c8fb2624e5 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:54:38.593 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.590+0000 7f9dd5600640 1 -- 172.21.15.32:0/590178320 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9dc400a940 con 0x55dce19caf50 2024-08-03T20:54:38.593 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.590+0000 7f9dd5600640 1 -- 172.21.15.32:0/590178320 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f9dc400aae0 con 0x55dce19caf50 2024-08-03T20:54:38.593 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.590+0000 7f9dd5600640 1 -- 172.21.15.32:0/590178320 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9dc4008450 con 0x55dce19caf50 2024-08-03T20:54:38.595 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.591+0000 7f9dde35ef40 1 -- 172.21.15.32:0/590178320 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55dce19caf50 msgr2=0x55dce1981300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:38.595 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.591+0000 7f9dde35ef40 1 --2- 172.21.15.32:0/590178320 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55dce19caf50 0x55dce1981300 secure :-1 s=READY pgs=938 cs=0 l=1 rev1=1 crypto rx=0x7f9dc400b630 tx=0x7f9dc400baf0 comp rx=0 tx=0).stop 2024-08-03T20:54:38.595 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.591+0000 7f9dde35ef40 1 -- 172.21.15.32:0/590178320 shutdown_connections 2024-08-03T20:54:38.595 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.591+0000 7f9dde35ef40 1 --2- 172.21.15.32:0/590178320 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55dce1a53820 0x55dce1981840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:38.595 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.591+0000 7f9dde35ef40 1 --2- 172.21.15.32:0/590178320 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55dce19caf50 0x55dce1981300 unknown :-1 s=CLOSED pgs=938 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:38.595 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.591+0000 7f9dde35ef40 1 --2- 172.21.15.32:0/590178320 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55dce19c9010 0x55dce1a56600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:38.595 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.591+0000 7f9dde35ef40 1 -- 172.21.15.32:0/590178320 >> 172.21.15.32:0/590178320 conn(0x55dce18a1ac0 msgr2=0x55dce19c86f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:54:38.595 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.591+0000 7f9dde35ef40 1 -- 172.21.15.32:0/590178320 shutdown_connections 2024-08-03T20:54:38.595 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.591+0000 7f9dde35ef40 1 -- 172.21.15.32:0/590178320 wait complete. 2024-08-03T20:54:38.596 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.592+0000 7f9dde35ef40 1 Processor -- start 2024-08-03T20:54:38.596 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.592+0000 7f9dde35ef40 1 -- start start 2024-08-03T20:54:38.597 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.594+0000 7f9dde35ef40 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55dce19c9010 0x55dce1a4ff40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:38.597 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.594+0000 7f9dde35ef40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55dce1a53820 0x55dce1a48670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:38.598 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.594+0000 7f9dde35ef40 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55dce1a4d7b0 0x55dce19aa110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:38.598 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.594+0000 7f9dde35ef40 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55dce1a4d550 con 0x55dce1a4d7b0 2024-08-03T20:54:38.598 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.594+0000 7f9dde35ef40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x55dce1a49b90 con 0x55dce1a53820 2024-08-03T20:54:38.598 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.594+0000 7f9dde35ef40 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x55dce19aae30 con 0x55dce19c9010 2024-08-03T20:54:38.598 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.594+0000 7f9dd7e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55dce1a4d7b0 0x55dce19aa110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:38.598 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.594+0000 7f9dd7e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55dce1a4d7b0 0x55dce19aa110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:40178/0 (socket says 172.21.15.32:40178) 2024-08-03T20:54:38.598 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.594+0000 7f9dd7e00640 1 -- 172.21.15.32:0/648633679 learned_addr learned my addr 172.21.15.32:0/648633679 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:54:38.598 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.594+0000 7f9dd7400640 1 --2- 172.21.15.32:0/648633679 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55dce19c9010 0x55dce1a4ff40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:38.598 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.594+0000 7f9dd6a00640 1 --2- 172.21.15.32:0/648633679 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55dce1a53820 0x55dce1a48670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:38.598 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.595+0000 7f9dd7e00640 1 -- 172.21.15.32:0/648633679 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55dce19c9010 msgr2=0x55dce1a4ff40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:38.598 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.595+0000 7f9dd7e00640 1 --2- 172.21.15.32:0/648633679 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55dce19c9010 0x55dce1a4ff40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:38.598 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.595+0000 7f9dd7e00640 1 -- 172.21.15.32:0/648633679 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55dce1a53820 msgr2=0x55dce1a48670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:38.599 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.595+0000 7f9dd7e00640 1 --2- 172.21.15.32:0/648633679 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55dce1a53820 0x55dce1a48670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:38.599 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.595+0000 7f9dd7e00640 1 -- 172.21.15.32:0/648633679 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55dce1a490f0 con 0x55dce1a4d7b0 2024-08-03T20:54:38.599 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.595+0000 7f9dd7400640 1 --2- 172.21.15.32:0/648633679 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55dce19c9010 0x55dce1a4ff40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-03T20:54:38.599 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.595+0000 7f9dd7e00640 1 --2- 172.21.15.32:0/648633679 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55dce1a4d7b0 0x55dce19aa110 secure :-1 s=READY pgs=939 cs=0 l=1 rev1=1 crypto rx=0x7f9dd0057040 tx=0x7f9dd00049c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:54:38.599 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.595+0000 7f9dcfe00640 1 -- 172.21.15.32:0/648633679 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9dd0064030 con 0x55dce1a4d7b0 2024-08-03T20:54:38.599 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.595+0000 7f9dcfe00640 1 -- 172.21.15.32:0/648633679 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f9dd004d780 con 0x55dce1a4d7b0 2024-08-03T20:54:38.599 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.595+0000 7f9dcfe00640 1 -- 172.21.15.32:0/648633679 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9dd0060440 con 0x55dce1a4d7b0 2024-08-03T20:54:38.599 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.596+0000 7f9dde35ef40 1 -- 172.21.15.32:0/648633679 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x55dce1a48e10 con 0x55dce1a4d7b0 2024-08-03T20:54:38.599 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.596+0000 7f9dde35ef40 1 -- 172.21.15.32:0/648633679 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55dce1a48bb0 con 0x55dce1a4d7b0 2024-08-03T20:54:38.599 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.598+0000 7f9dcfe00640 1 -- 172.21.15.32:0/648633679 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f9dd0067030 con 0x55dce1a4d7b0 2024-08-03T20:54:38.600 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.599+0000 7f9dcfe00640 1 --2- 172.21.15.32:0/648633679 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f9db8075a20 0x7f9db8077f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:38.600 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.599+0000 7f9dcfe00640 1 -- 172.21.15.32:0/648633679 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 5795+0+0 (secure 0 0 0) 0x7f9dd00dbac0 con 0x55dce1a4d7b0 2024-08-03T20:54:38.601 INFO:teuthology.orchestra.run.smithi032.stdout:���� 2024-08-03T20:54:38.601 INFO:teuthology.orchestra.run.smithi032.stdout: �� 2024-08-03T20:54:38.601 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-03T20:54:38.601 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.599+0000 7f9dde35ef40 1 --2- 172.21.15.32:0/648633679 >> [v2:172.21.15.118:6802/864752275,v1:172.21.15.118:6803/864752275] conn(0x55dce19810c0 0x55dce19814d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:38.602 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.599+0000 7f9dde35ef40 1 -- 172.21.15.32:0/648633679 --> [v2:172.21.15.118:6802/864752275,v1:172.21.15.118:6803/864752275] -- osd_op(unknown.0.0:1 12.0 12:00000000:%01:::head [pgnls start_epoch 84 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e84) -- 0x55dce1afa5f0 con 0x55dce19810c0 2024-08-03T20:54:38.602 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.599+0000 7f9dd7400640 1 --2- 172.21.15.32:0/648633679 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f9db8075a20 0x7f9db8077f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:38.602 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.600+0000 7f9dd6a00640 1 --2- 172.21.15.32:0/648633679 >> [v2:172.21.15.118:6802/864752275,v1:172.21.15.118:6803/864752275] conn(0x55dce19810c0 0x55dce19814d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:38.602 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.600+0000 7f9dd7400640 1 --2- 172.21.15.32:0/648633679 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f9db8075a20 0x7f9db8077f30 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f9dc800cab0 tx=0x7f9dc8004880 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:54:38.602 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.600+0000 7f9dd6a00640 1 --2- 172.21.15.32:0/648633679 >> [v2:172.21.15.118:6802/864752275,v1:172.21.15.118:6803/864752275] conn(0x55dce19810c0 0x55dce19814d0 crc :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:54:38.605 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.603+0000 7f9dd6a00640 1 -- 172.21.15.32:0/648633679 <== osd.4 v2:172.21.15.118:6802/864752275 1 ==== osd_op_reply(1 [pgnls start_epoch 84 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f9dc400aac0 con 0x55dce19810c0 2024-08-03T20:54:38.605 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.603+0000 7f9dd6a00640 1 --2- 172.21.15.32:0/648633679 >> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] conn(0x55dce19caf50 0x7f9dc4008450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:38.605 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.603+0000 7f9dd6a00640 1 -- 172.21.15.32:0/648633679 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:2 12.10 12:08000000:%01:::head [pgnls start_epoch 84 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e84) -- 0x55dce1afa5f0 con 0x55dce19caf50 2024-08-03T20:54:38.605 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.604+0000 7f9dd7400640 1 --2- 172.21.15.32:0/648633679 >> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] conn(0x55dce19caf50 0x7f9dc4008450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:38.606 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.604+0000 7f9dd7400640 1 --2- 172.21.15.32:0/648633679 >> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] conn(0x55dce19caf50 0x7f9dc4008450 crc :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:54:38.609 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.607+0000 7f9dd7400640 1 -- 172.21.15.32:0/648633679 <== osd.0 v2:172.21.15.32:6816/3066156116 1 ==== osd_op_reply(2 [pgnls start_epoch 84 out=177b] v83'10 uv10 ondisk = 1) ==== 144+0+177 (crc 0 0 0) 0x7f9dc800abb0 con 0x55dce19caf50 2024-08-03T20:54:38.609 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.607+0000 7f9dd7400640 1 --2- 172.21.15.32:0/648633679 >> [v2:172.21.15.32:6800/9088303,v1:172.21.15.32:6801/9088303] conn(0x7f9dc8008950 0x7f9dc8016410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:38.609 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.607+0000 7f9dd7400640 1 -- 172.21.15.32:0/648633679 --> [v2:172.21.15.32:6800/9088303,v1:172.21.15.32:6801/9088303] -- osd_op(unknown.0.0:3 12.8 12:10000000:%01:::head [pgnls start_epoch 84 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e84) -- 0x55dce1afa5f0 con 0x7f9dc8008950 2024-08-03T20:54:38.609 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.607+0000 7f9dd6a00640 1 --2- 172.21.15.32:0/648633679 >> [v2:172.21.15.32:6800/9088303,v1:172.21.15.32:6801/9088303] conn(0x7f9dc8008950 0x7f9dc8016410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:38.609 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.608+0000 7f9dd6a00640 1 --2- 172.21.15.32:0/648633679 >> [v2:172.21.15.32:6800/9088303,v1:172.21.15.32:6801/9088303] conn(0x7f9dc8008950 0x7f9dc8016410 crc :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:54:38.610 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.608+0000 7f9dd6a00640 1 -- 172.21.15.32:0/648633679 <== osd.3 v2:172.21.15.32:6800/9088303 1 ==== osd_op_reply(3 [pgnls start_epoch 84 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f9dc400aac0 con 0x7f9dc8008950 2024-08-03T20:54:38.610 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.608+0000 7f9dd6a00640 1 --2- 172.21.15.32:0/648633679 >> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] conn(0x7f9dc4024400 0x7f9dc4024810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:38.610 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.608+0000 7f9dd6a00640 1 -- 172.21.15.32:0/648633679 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:4 12.18 12:18000000:%01:::head [pgnls start_epoch 84 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e84) -- 0x55dce1afa5f0 con 0x7f9dc4024400 2024-08-03T20:54:38.610 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.609+0000 7f9dd7e00640 1 --2- 172.21.15.32:0/648633679 >> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] conn(0x7f9dc4024400 0x7f9dc4024810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:38.611 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.609+0000 7f9dd7e00640 1 --2- 172.21.15.32:0/648633679 >> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] conn(0x7f9dc4024400 0x7f9dc4024810 crc :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:54:38.613 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.611+0000 7f9dd7e00640 1 -- 172.21.15.32:0/648633679 <== osd.1 v2:172.21.15.32:6824/1551241759 1 ==== osd_op_reply(4 [pgnls start_epoch 84 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f9dd00a9d70 con 0x7f9dc4024400 2024-08-03T20:54:38.613 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.612+0000 7f9dd7e00640 1 --2- 172.21.15.32:0/648633679 >> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] conn(0x7f9dd0022d70 0x7f9dd00a3360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:38.613 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.612+0000 7f9dd7e00640 1 -- 172.21.15.32:0/648633679 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:5 12.4 12:20000000:%01:::head [pgnls start_epoch 84 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e84) -- 0x55dce1afa5f0 con 0x7f9dd0022d70 2024-08-03T20:54:38.614 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.612+0000 7f9dd7400640 1 --2- 172.21.15.32:0/648633679 >> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] conn(0x7f9dd0022d70 0x7f9dd00a3360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:38.614 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.613+0000 7f9dd7400640 1 --2- 172.21.15.32:0/648633679 >> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] conn(0x7f9dd0022d70 0x7f9dd00a3360 crc :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:54:38.618 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.616+0000 7f9dd7400640 1 -- 172.21.15.32:0/648633679 <== osd.6 v2:172.21.15.118:6825/801947386 1 ==== osd_op_reply(5 [pgnls start_epoch 84 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f9dc800c0d0 con 0x7f9dd0022d70 2024-08-03T20:54:38.619 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.617+0000 7f9dd7400640 1 --2- 172.21.15.32:0/648633679 >> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] conn(0x7f9dc8028040 0x7f9dc8028960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:38.619 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.617+0000 7f9dd7400640 1 -- 172.21.15.32:0/648633679 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:6 12.14 12:28000000:%01:::head [pgnls start_epoch 84 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e84) -- 0x55dce1afa5f0 con 0x7f9dc8028040 2024-08-03T20:54:38.619 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.617+0000 7f9dd6a00640 1 --2- 172.21.15.32:0/648633679 >> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] conn(0x7f9dc8028040 0x7f9dc8028960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:38.619 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.618+0000 7f9dd6a00640 1 --2- 172.21.15.32:0/648633679 >> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] conn(0x7f9dc8028040 0x7f9dc8028960 crc :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:54:38.622 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.621+0000 7f9dd6a00640 1 -- 172.21.15.32:0/648633679 <== osd.5 v2:172.21.15.118:6810/2724891770 1 ==== osd_op_reply(6 [pgnls start_epoch 84 out=149b] v83'19 uv19 ondisk = 1) ==== 144+0+149 (crc 0 0 0) 0x7f9dc400aac0 con 0x7f9dc8028040 2024-08-03T20:54:38.622 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.621+0000 7f9dd6a00640 1 -- 172.21.15.32:0/648633679 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:7 12.c 12:30000000:%01:::head [pgnls start_epoch 84 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e84) -- 0x55dce1afa5f0 con 0x55dce19caf50 2024-08-03T20:54:38.623 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.621+0000 7f9dd7400640 1 -- 172.21.15.32:0/648633679 <== osd.0 v2:172.21.15.32:6816/3066156116 2 ==== osd_op_reply(7 [pgnls start_epoch 84 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f9dc800c0d0 con 0x55dce19caf50 2024-08-03T20:54:38.623 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.621+0000 7f9dd7400640 1 -- 172.21.15.32:0/648633679 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:8 12.1c 12:38000000:%01:::head [pgnls start_epoch 84 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e84) -- 0x55dce1afa5f0 con 0x55dce19caf50 2024-08-03T20:54:38.623 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.622+0000 7f9dd7400640 1 -- 172.21.15.32:0/648633679 <== osd.0 v2:172.21.15.32:6816/3066156116 3 ==== osd_op_reply(8 [pgnls start_epoch 84 out=73b] v82'2 uv2 ondisk = 1) ==== 144+0+73 (crc 0 0 0) 0x7f9dc800c0d0 con 0x55dce19caf50 2024-08-03T20:54:38.623 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.622+0000 7f9dd7400640 1 -- 172.21.15.32:0/648633679 --> [v2:172.21.15.118:6802/864752275,v1:172.21.15.118:6803/864752275] -- osd_op(unknown.0.0:9 12.2 12:40000000:%01:::head [pgnls start_epoch 84 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e84) -- 0x55dce1afa5f0 con 0x55dce19810c0 2024-08-03T20:54:38.624 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.622+0000 7f9dd6a00640 1 -- 172.21.15.32:0/648633679 <== osd.4 v2:172.21.15.118:6802/864752275 2 ==== osd_op_reply(9 [pgnls start_epoch 84 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f9dc400aac0 con 0x55dce19810c0 2024-08-03T20:54:38.624 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.622+0000 7f9dd6a00640 1 -- 172.21.15.32:0/648633679 --> [v2:172.21.15.32:6800/9088303,v1:172.21.15.32:6801/9088303] -- osd_op(unknown.0.0:10 12.12 12:48000000:%01:::head [pgnls start_epoch 84 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e84) -- 0x55dce1afa5f0 con 0x7f9dc8008950 2024-08-03T20:54:38.624 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.623+0000 7f9dd6a00640 1 -- 172.21.15.32:0/648633679 <== osd.3 v2:172.21.15.32:6800/9088303 2 ==== osd_op_reply(10 [pgnls start_epoch 84 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f9dc403a040 con 0x7f9dc8008950 2024-08-03T20:54:38.625 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.623+0000 7f9dd6a00640 1 --2- 172.21.15.32:0/648633679 >> [v2:172.21.15.32:6808/3806708944,v1:172.21.15.32:6809/3806708944] conn(0x7f9dc40153d0 0x7f9dc40157e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:38.625 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.623+0000 7f9dd6a00640 1 -- 172.21.15.32:0/648633679 --> [v2:172.21.15.32:6808/3806708944,v1:172.21.15.32:6809/3806708944] -- osd_op(unknown.0.0:11 12.a 12:50000000:%01:::head [pgnls start_epoch 84 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e84) -- 0x55dce1afa5f0 con 0x7f9dc40153d0 2024-08-03T20:54:38.625 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.623+0000 7f9dd7e00640 1 --2- 172.21.15.32:0/648633679 >> [v2:172.21.15.32:6808/3806708944,v1:172.21.15.32:6809/3806708944] conn(0x7f9dc40153d0 0x7f9dc40157e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:38.625 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.624+0000 7f9dd7e00640 1 --2- 172.21.15.32:0/648633679 >> [v2:172.21.15.32:6808/3806708944,v1:172.21.15.32:6809/3806708944] conn(0x7f9dc40153d0 0x7f9dc40157e0 crc :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:54:38.627 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.626+0000 7f9dd7e00640 1 -- 172.21.15.32:0/648633679 <== osd.2 v2:172.21.15.32:6808/3806708944 1 ==== osd_op_reply(11 [pgnls start_epoch 84 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f9dd00a9d70 con 0x7f9dc40153d0 2024-08-03T20:54:38.628 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.626+0000 7f9dd7e00640 1 -- 172.21.15.32:0/648633679 --> [v2:172.21.15.118:6802/864752275,v1:172.21.15.118:6803/864752275] -- osd_op(unknown.0.0:12 12.1a 12:58000000:%01:::head [pgnls start_epoch 84 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e84) -- 0x55dce1afa5f0 con 0x55dce19810c0 2024-08-03T20:54:38.628 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.627+0000 7f9dd6a00640 1 -- 172.21.15.32:0/648633679 <== osd.4 v2:172.21.15.118:6802/864752275 3 ==== osd_op_reply(12 [pgnls start_epoch 84 out=49b] v82'1 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f9dc403a040 con 0x55dce19810c0 2024-08-03T20:54:38.628 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.627+0000 7f9dd6a00640 1 -- 172.21.15.32:0/648633679 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:13 12.6 12:60000000:%01:::head [pgnls start_epoch 84 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e84) -- 0x55dce1afa5f0 con 0x7f9dc4024400 2024-08-03T20:54:38.629 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.627+0000 7f9dd7e00640 1 -- 172.21.15.32:0/648633679 <== osd.1 v2:172.21.15.32:6824/1551241759 2 ==== osd_op_reply(13 [pgnls start_epoch 84 out=77b] v83'4 uv4 ondisk = 1) ==== 144+0+77 (crc 0 0 0) 0x7f9dd00a9d70 con 0x7f9dc4024400 2024-08-03T20:54:38.629 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.627+0000 7f9dd7e00640 1 -- 172.21.15.32:0/648633679 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:14 12.16 12:68000000:%01:::head [pgnls start_epoch 84 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e84) -- 0x55dce1afa5f0 con 0x7f9dd0022d70 2024-08-03T20:54:38.629 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.628+0000 7f9dd7400640 1 -- 172.21.15.32:0/648633679 <== osd.6 v2:172.21.15.118:6825/801947386 2 ==== osd_op_reply(14 [pgnls start_epoch 84 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f9dc800c0d0 con 0x7f9dd0022d70 2024-08-03T20:54:38.629 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.628+0000 7f9dd7400640 1 -- 172.21.15.32:0/648633679 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:15 12.e 12:70000000:%01:::head [pgnls start_epoch 84 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e84) -- 0x55dce1afa5f0 con 0x55dce19caf50 2024-08-03T20:54:38.629 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.628+0000 7f9dd7400640 1 -- 172.21.15.32:0/648633679 <== osd.0 v2:172.21.15.32:6816/3066156116 4 ==== osd_op_reply(15 [pgnls start_epoch 84 out=73b] v82'2 uv2 ondisk = 1) ==== 144+0+73 (crc 0 0 0) 0x7f9dc800c0d0 con 0x55dce19caf50 2024-08-03T20:54:38.630 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.628+0000 7f9dd7400640 1 -- 172.21.15.32:0/648633679 --> [v2:172.21.15.118:6802/864752275,v1:172.21.15.118:6803/864752275] -- osd_op(unknown.0.0:16 12.1e 12:78000000:%01:::head [pgnls start_epoch 84 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e84) -- 0x7f9dc8034060 con 0x55dce19810c0 2024-08-03T20:54:38.630 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.629+0000 7f9dd6a00640 1 -- 172.21.15.32:0/648633679 <== osd.4 v2:172.21.15.118:6802/864752275 4 ==== osd_op_reply(16 [pgnls start_epoch 84 out=49b] v82'1 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f9dc403a040 con 0x55dce19810c0 2024-08-03T20:54:38.630 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.629+0000 7f9dd6a00640 1 -- 172.21.15.32:0/648633679 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:17 12.1 12:80000000:%01:::head [pgnls start_epoch 84 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e84) -- 0x7f9dc8034060 con 0x7f9dd0022d70 2024-08-03T20:54:38.631 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.629+0000 7f9dd7400640 1 -- 172.21.15.32:0/648633679 <== osd.6 v2:172.21.15.118:6825/801947386 3 ==== osd_op_reply(17 [pgnls start_epoch 84 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f9dc800c0d0 con 0x7f9dd0022d70 2024-08-03T20:54:38.631 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.629+0000 7f9dd7400640 1 -- 172.21.15.32:0/648633679 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:18 12.11 12:88000000:%01:::head [pgnls start_epoch 84 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e84) -- 0x7f9dc8034060 con 0x7f9dc8028040 2024-08-03T20:54:38.631 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.630+0000 7f9dd6a00640 1 -- 172.21.15.32:0/648633679 <== osd.5 v2:172.21.15.118:6810/2724891770 2 ==== osd_op_reply(18 [pgnls start_epoch 84 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f9dc403a040 con 0x7f9dc8028040 2024-08-03T20:54:38.632 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.630+0000 7f9dd6a00640 1 -- 172.21.15.32:0/648633679 --> [v2:172.21.15.32:6808/3806708944,v1:172.21.15.32:6809/3806708944] -- osd_op(unknown.0.0:19 12.9 12:90000000:%01:::head [pgnls start_epoch 84 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e84) -- 0x7f9dc8034060 con 0x7f9dc40153d0 2024-08-03T20:54:38.632 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.630+0000 7f9dd7e00640 1 -- 172.21.15.32:0/648633679 <== osd.2 v2:172.21.15.32:6808/3806708944 2 ==== osd_op_reply(19 [pgnls start_epoch 84 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f9dd00a9d70 con 0x7f9dc40153d0 2024-08-03T20:54:38.632 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.630+0000 7f9dd7e00640 1 -- 172.21.15.32:0/648633679 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:20 12.19 12:98000000:%01:::head [pgnls start_epoch 84 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e84) -- 0x7f9dc8034060 con 0x55dce19caf50 2024-08-03T20:54:38.632 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.631+0000 7f9dd7400640 1 -- 172.21.15.32:0/648633679 <== osd.0 v2:172.21.15.32:6816/3066156116 5 ==== osd_op_reply(20 [pgnls start_epoch 84 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f9dc800c0d0 con 0x55dce19caf50 2024-08-03T20:54:38.632 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.631+0000 7f9dd7400640 1 -- 172.21.15.32:0/648633679 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:21 12.5 12:a0000000:%01:::head [pgnls start_epoch 84 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e84) -- 0x7f9dc8034060 con 0x7f9dd0022d70 2024-08-03T20:54:38.633 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.631+0000 7f9dd7400640 1 -- 172.21.15.32:0/648633679 <== osd.6 v2:172.21.15.118:6825/801947386 4 ==== osd_op_reply(21 [pgnls start_epoch 84 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f9dc800c0d0 con 0x7f9dd0022d70 2024-08-03T20:54:38.633 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.632+0000 7f9dd7400640 1 -- 172.21.15.32:0/648633679 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:22 12.15 12:a8000000:%01:::head [pgnls start_epoch 84 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e84) -- 0x7f9dc8034060 con 0x7f9dd0022d70 2024-08-03T20:54:38.633 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.632+0000 7f9dd7400640 1 -- 172.21.15.32:0/648633679 <== osd.6 v2:172.21.15.118:6825/801947386 5 ==== osd_op_reply(22 [pgnls start_epoch 84 out=97b] v82'3 uv3 ondisk = 1) ==== 144+0+97 (crc 0 0 0) 0x7f9dc800c0d0 con 0x7f9dd0022d70 2024-08-03T20:54:38.633 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.632+0000 7f9dd7400640 1 --2- 172.21.15.32:0/648633679 >> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] conn(0x7f9dc8034a80 0x7f9dc80093c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:38.634 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.632+0000 7f9dd7400640 1 -- 172.21.15.32:0/648633679 --> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] -- osd_op(unknown.0.0:23 12.d 12:b0000000:%01:::head [pgnls start_epoch 84 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e84) -- 0x7f9dc8034060 con 0x7f9dc8034a80 2024-08-03T20:54:38.634 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.633+0000 7f9dd7400640 1 --2- 172.21.15.32:0/648633679 >> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] conn(0x7f9dc8034a80 0x7f9dc80093c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:38.635 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.633+0000 7f9dd7400640 1 --2- 172.21.15.32:0/648633679 >> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] conn(0x7f9dc8034a80 0x7f9dc80093c0 crc :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:54:38.636 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.634+0000 7f9dd7400640 1 -- 172.21.15.32:0/648633679 <== osd.7 v2:172.21.15.118:6818/653000048 1 ==== osd_op_reply(23 [pgnls start_epoch 84 out=74b] v82'1 uv1 ondisk = 1) ==== 144+0+74 (crc 0 0 0) 0x7f9dc8043040 con 0x7f9dc8034a80 2024-08-03T20:54:38.637 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.634+0000 7f9dd7400640 1 -- 172.21.15.32:0/648633679 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:24 12.1d 12:b8000000:%01:::head [pgnls start_epoch 84 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e84) -- 0x7f9dc8034060 con 0x55dce19caf50 2024-08-03T20:54:38.637 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.634+0000 7f9dd7400640 1 -- 172.21.15.32:0/648633679 <== osd.0 v2:172.21.15.32:6816/3066156116 6 ==== osd_op_reply(24 [pgnls start_epoch 84 out=73b] v82'2 uv2 ondisk = 1) ==== 144+0+73 (crc 0 0 0) 0x7f9dc8043040 con 0x55dce19caf50 2024-08-03T20:54:38.637 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.634+0000 7f9dd7400640 1 -- 172.21.15.32:0/648633679 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:25 12.3 12:c0000000:%01:::head [pgnls start_epoch 84 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e84) -- 0x7f9dc8034060 con 0x55dce19caf50 2024-08-03T20:54:38.637 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.635+0000 7f9dd7400640 1 -- 172.21.15.32:0/648633679 <== osd.0 v2:172.21.15.32:6816/3066156116 7 ==== osd_op_reply(25 [pgnls start_epoch 84 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f9dc8043040 con 0x55dce19caf50 2024-08-03T20:54:38.637 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.635+0000 7f9dd7400640 1 -- 172.21.15.32:0/648633679 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:26 12.13 12:c8000000:%01:::head [pgnls start_epoch 84 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e84) -- 0x7f9dc8034060 con 0x7f9dc4024400 2024-08-03T20:54:38.637 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.635+0000 7f9dd7e00640 1 -- 172.21.15.32:0/648633679 <== osd.1 v2:172.21.15.32:6824/1551241759 3 ==== osd_op_reply(26 [pgnls start_epoch 84 out=98b] v83'3 uv3 ondisk = 1) ==== 144+0+98 (crc 0 0 0) 0x7f9dd00a9d70 con 0x7f9dc4024400 2024-08-03T20:54:38.637 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.635+0000 7f9dd7e00640 1 -- 172.21.15.32:0/648633679 --> [v2:172.21.15.32:6800/9088303,v1:172.21.15.32:6801/9088303] -- osd_op(unknown.0.0:27 12.b 12:d0000000:%01:::head [pgnls start_epoch 84 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e84) -- 0x7f9dc8034060 con 0x7f9dc8008950 2024-08-03T20:54:38.638 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.636+0000 7f9dd6a00640 1 -- 172.21.15.32:0/648633679 <== osd.3 v2:172.21.15.32:6800/9088303 3 ==== osd_op_reply(27 [pgnls start_epoch 84 out=76b] v83'3 uv3 ondisk = 1) ==== 144+0+76 (crc 0 0 0) 0x7f9dc403a040 con 0x7f9dc8008950 2024-08-03T20:54:38.638 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.636+0000 7f9dd6a00640 1 -- 172.21.15.32:0/648633679 --> [v2:172.21.15.118:6802/864752275,v1:172.21.15.118:6803/864752275] -- osd_op(unknown.0.0:28 12.1b 12:d8000000:%01:::head [pgnls start_epoch 84 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e84) -- 0x7f9dc8034060 con 0x55dce19810c0 2024-08-03T20:54:38.638 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.636+0000 7f9dd6a00640 1 -- 172.21.15.32:0/648633679 <== osd.4 v2:172.21.15.118:6802/864752275 5 ==== osd_op_reply(28 [pgnls start_epoch 84 out=49b] v82'1 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f9dc403a040 con 0x55dce19810c0 2024-08-03T20:54:38.638 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.636+0000 7f9dd6a00640 1 -- 172.21.15.32:0/648633679 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:29 12.7 12:e0000000:%01:::head [pgnls start_epoch 84 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e84) -- 0x7f9dc8034060 con 0x7f9dc4024400 2024-08-03T20:54:38.639 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.637+0000 7f9dd7e00640 1 -- 172.21.15.32:0/648633679 <== osd.1 v2:172.21.15.32:6824/1551241759 4 ==== osd_op_reply(29 [pgnls start_epoch 84 out=73b] v82'2 uv2 ondisk = 1) ==== 144+0+73 (crc 0 0 0) 0x7f9dd00a9d70 con 0x7f9dc4024400 2024-08-03T20:54:38.639 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.637+0000 7f9dd7e00640 1 -- 172.21.15.32:0/648633679 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:30 12.17 12:e8000000:%01:::head [pgnls start_epoch 84 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e84) -- 0x7f9dc8034060 con 0x7f9dd0022d70 2024-08-03T20:54:38.639 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.637+0000 7f9dd7400640 1 -- 172.21.15.32:0/648633679 <== osd.6 v2:172.21.15.118:6825/801947386 6 ==== osd_op_reply(30 [pgnls start_epoch 84 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f9dc8043040 con 0x7f9dd0022d70 2024-08-03T20:54:38.639 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.637+0000 7f9dd7400640 1 -- 172.21.15.32:0/648633679 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:31 12.f 12:f0000000:%01:::head [pgnls start_epoch 84 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e84) -- 0x7f9dc8034060 con 0x7f9dd0022d70 2024-08-03T20:54:38.639 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.638+0000 7f9dd7400640 1 -- 172.21.15.32:0/648633679 <== osd.6 v2:172.21.15.118:6825/801947386 7 ==== osd_op_reply(31 [pgnls start_epoch 84 out=73b] v82'1 uv1 ondisk = 1) ==== 144+0+73 (crc 0 0 0) 0x7f9dc8043040 con 0x7f9dd0022d70 2024-08-03T20:54:38.639 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.638+0000 7f9dd7400640 1 -- 172.21.15.32:0/648633679 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:32 12.1f 12:f8000000:%01:::head [pgnls start_epoch 84 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e84) -- 0x7f9dc8034060 con 0x7f9dd0022d70 2024-08-03T20:54:38.640 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.638+0000 7f9dd7400640 1 -- 172.21.15.32:0/648633679 <== osd.6 v2:172.21.15.118:6825/801947386 8 ==== osd_op_reply(32 [pgnls start_epoch 84 out=101b] v83'6 uv6 ondisk = 1) ==== 144+0+101 (crc 0 0 0) 0x7f9dc8043040 con 0x7f9dd0022d70 2024-08-03T20:54:38.641 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.638+0000 7f9dde35ef40 1 -- 172.21.15.32:0/648633679 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:33 12.10 12:09401a81:::601.00000000:head [read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84) -- 0x55dce1af1480 con 0x55dce19caf50 2024-08-03T20:54:38.641 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.639+0000 7f9dd7400640 1 -- 172.21.15.32:0/648633679 <== osd.0 v2:172.21.15.32:6816/3066156116 8 ==== osd_op_reply(33 601.00000000 [read 0~0] v0'0 uv6 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f9dc8043040 con 0x55dce19caf50 2024-08-03T20:54:38.641 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.639+0000 7f9dde35ef40 1 -- 172.21.15.32:0/648633679 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:34 12.10 12:09401a81:::601.00000000:head [getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84) -- 0x55dce1af18a0 con 0x55dce19caf50 2024-08-03T20:54:38.641 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.639+0000 7f9dd7400640 1 -- 172.21.15.32:0/648633679 <== osd.0 v2:172.21.15.32:6816/3066156116 9 ==== osd_op_reply(34 601.00000000 [getxattrs out=124b] v0'0 uv6 ondisk = 0) ==== 156+0+124 (crc 0 0 0) 0x7f9dc8043040 con 0x55dce19caf50 2024-08-03T20:54:38.641 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.639+0000 7f9dde35ef40 1 -- 172.21.15.32:0/648633679 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:35 12.10 12:09401a81:::601.00000000:head [omap-get-header] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84) -- 0x55dce1af1cc0 con 0x55dce19caf50 2024-08-03T20:54:38.642 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.639+0000 7f9dd7400640 1 -- 172.21.15.32:0/648633679 <== osd.0 v2:172.21.15.32:6816/3066156116 10 ==== osd_op_reply(35 601.00000000 [omap-get-header out=274b] v0'0 uv6 ondisk = 0) ==== 156+0+274 (crc 0 0 0) 0x7f9dc804f040 con 0x55dce19caf50 2024-08-03T20:54:38.642 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.639+0000 7f9dde35ef40 1 -- 172.21.15.32:0/648633679 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:36 12.10 12:09401a81:::601.00000000:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84) -- 0x55dce1af2060 con 0x55dce19caf50 2024-08-03T20:54:38.642 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.640+0000 7f9dd7400640 1 -- 172.21.15.32:0/648633679 <== osd.0 v2:172.21.15.32:6816/3066156116 11 ==== osd_op_reply(36 601.00000000 [omap-get-vals out=5b] v0'0 uv6 ondisk = 0) ==== 156+0+5 (crc 0 0 0) 0x7f9dc804f040 con 0x55dce19caf50 2024-08-03T20:54:38.643 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.640+0000 7f9dde35ef40 1 -- 172.21.15.32:0/648633679 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:37 12.10 12:0bd6d154:::602.00000000:head [read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84) -- 0x55dce1afa990 con 0x55dce19caf50 2024-08-03T20:54:38.643 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.640+0000 7f9dd7400640 1 -- 172.21.15.32:0/648633679 <== osd.0 v2:172.21.15.32:6816/3066156116 12 ==== osd_op_reply(37 602.00000000 [read 0~0] v0'0 uv7 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f9dc804f040 con 0x55dce19caf50 2024-08-03T20:54:38.643 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.640+0000 7f9dde35ef40 1 -- 172.21.15.32:0/648633679 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:38 12.10 12:0bd6d154:::602.00000000:head [getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84) -- 0x55dce1afa990 con 0x55dce19caf50 2024-08-03T20:54:38.643 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.640+0000 7f9dd7400640 1 -- 172.21.15.32:0/648633679 <== osd.0 v2:172.21.15.32:6816/3066156116 13 ==== osd_op_reply(38 602.00000000 [getxattrs out=124b] v0'0 uv7 ondisk = 0) ==== 156+0+124 (crc 0 0 0) 0x7f9dc804f040 con 0x55dce19caf50 2024-08-03T20:54:38.644 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.640+0000 7f9dde35ef40 1 -- 172.21.15.32:0/648633679 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:39 12.10 12:0bd6d154:::602.00000000:head [omap-get-header] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84) -- 0x55dce1afa990 con 0x55dce19caf50 2024-08-03T20:54:38.644 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.641+0000 7f9dd7400640 1 -- 172.21.15.32:0/648633679 <== osd.0 v2:172.21.15.32:6816/3066156116 14 ==== osd_op_reply(39 602.00000000 [omap-get-header out=274b] v0'0 uv7 ondisk = 0) ==== 156+0+274 (crc 0 0 0) 0x7f9dc804f040 con 0x55dce19caf50 2024-08-03T20:54:38.645 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.641+0000 7f9dde35ef40 1 -- 172.21.15.32:0/648633679 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:40 12.10 12:0bd6d154:::602.00000000:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84) -- 0x55dce1afb9a0 con 0x55dce19caf50 2024-08-03T20:54:38.645 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.641+0000 7f9dd7400640 1 -- 172.21.15.32:0/648633679 <== osd.0 v2:172.21.15.32:6816/3066156116 15 ==== osd_op_reply(40 602.00000000 [omap-get-vals out=5b] v0'0 uv7 ondisk = 0) ==== 156+0+5 (crc 0 0 0) 0x7f9dc804f040 con 0x55dce19caf50 2024-08-03T20:54:38.646 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.641+0000 7f9dde35ef40 1 -- 172.21.15.32:0/648633679 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:41 12.10 12:0cf6ad0f:::10000000000.00000000:head [read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84) -- 0x55dce1afa990 con 0x55dce19caf50 2024-08-03T20:54:38.646 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.641+0000 7f9dd7400640 1 -- 172.21.15.32:0/648633679 <== osd.0 v2:172.21.15.32:6816/3066156116 16 ==== osd_op_reply(41 10000000000.00000000 [read 0~0] v0'0 uv10 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f9dc804f040 con 0x55dce19caf50 2024-08-03T20:54:38.646 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.641+0000 7f9dde35ef40 1 -- 172.21.15.32:0/648633679 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:42 12.10 12:0cf6ad0f:::10000000000.00000000:head [getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84) -- 0x55dce1afa990 con 0x55dce19caf50 2024-08-03T20:54:38.646 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.642+0000 7f9dd7400640 1 -- 172.21.15.32:0/648633679 <== osd.0 v2:172.21.15.32:6816/3066156116 17 ==== osd_op_reply(42 10000000000.00000000 [getxattrs out=124b] v0'0 uv10 ondisk = 0) ==== 164+0+124 (crc 0 0 0) 0x7f9dc804f040 con 0x55dce19caf50 2024-08-03T20:54:38.646 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.642+0000 7f9dde35ef40 1 -- 172.21.15.32:0/648633679 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:43 12.10 12:0cf6ad0f:::10000000000.00000000:head [omap-get-header] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84) -- 0x55dce1afa990 con 0x55dce19caf50 2024-08-03T20:54:38.646 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.642+0000 7f9dd7400640 1 -- 172.21.15.32:0/648633679 <== osd.0 v2:172.21.15.32:6816/3066156116 18 ==== osd_op_reply(43 10000000000.00000000 [omap-get-header out=274b] v0'0 uv10 ondisk = 0) ==== 164+0+274 (crc 0 0 0) 0x7f9dc804f040 con 0x55dce19caf50 2024-08-03T20:54:38.647 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.642+0000 7f9dde35ef40 1 -- 172.21.15.32:0/648633679 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:44 12.10 12:0cf6ad0f:::10000000000.00000000:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84) -- 0x55dce1afb9a0 con 0x55dce19caf50 2024-08-03T20:54:38.647 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.642+0000 7f9dd7400640 1 -- 172.21.15.32:0/648633679 <== osd.0 v2:172.21.15.32:6816/3066156116 19 ==== osd_op_reply(44 10000000000.00000000 [omap-get-vals out=594b] v0'0 uv10 ondisk = 0) ==== 164+0+594 (crc 0 0 0) 0x7f9dc804f040 con 0x55dce19caf50 2024-08-03T20:54:38.647 INFO:teuthology.orchestra.run.smithi032.stdout: ��*$- 601.00000000�������������������!�������������������������� ���~_layout��������_parent>8stray1 ���� �� ((88���� �� ��*$- 602.00000000�������������������!�������������������������� ���~_layout��������_parent>8stray2 ���� �� ((88���� �� ��2,510000000000.00000000�������������������!�������������������������� ���~_layout��������_parent>8subdir ���� ��  ( ��f�( ��f�8` ��fh��8` ��fh������ ��WQ sixmegs_head9i* � ��fh�����$@@ `�������� ��fh�� ��f�(8` ��fh��8` ��fh�� 2024-08-03T20:54:38.648 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.642+0000 7f9dde35ef40 1 -- 172.21.15.32:0/648633679 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:45 12.10 12:0d82a743:::600.00000000:head [read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84) -- 0x55dce1afa990 con 0x55dce19caf50 2024-08-03T20:54:38.648 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.643+0000 7f9dd7400640 1 -- 172.21.15.32:0/648633679 <== osd.0 v2:172.21.15.32:6816/3066156116 20 ==== osd_op_reply(45 600.00000000 [read 0~0] v0'0 uv5 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f9dc804f040 con 0x55dce19caf50 2024-08-03T20:54:38.648 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.643+0000 7f9dde35ef40 1 -- 172.21.15.32:0/648633679 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:46 12.10 12:0d82a743:::600.00000000:head [getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84) -- 0x55dce1afa990 con 0x55dce19caf50 2024-08-03T20:54:38.648 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.643+0000 7f9dd7400640 1 -- 172.21.15.32:0/648633679 <== osd.0 v2:172.21.15.32:6816/3066156116 21 ==== osd_op_reply(46 600.00000000 [getxattrs out=124b] v0'0 uv5 ondisk = 0) ==== 156+0+124 (crc 0 0 0) 0x7f9dc804f040 con 0x55dce19caf50 2024-08-03T20:54:38.649 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.643+0000 7f9dde35ef40 1 -- 172.21.15.32:0/648633679 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:47 12.10 12:0d82a743:::600.00000000:head [omap-get-header] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84) -- 0x55dce1afa990 con 0x55dce19caf50 2024-08-03T20:54:38.649 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.643+0000 7f9dd7400640 1 -- 172.21.15.32:0/648633679 <== osd.0 v2:172.21.15.32:6816/3066156116 22 ==== osd_op_reply(47 600.00000000 [omap-get-header out=274b] v0'0 uv5 ondisk = 0) ==== 156+0+274 (crc 0 0 0) 0x7f9dc804f040 con 0x55dce19caf50 2024-08-03T20:54:38.650 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.643+0000 7f9dde35ef40 1 -- 172.21.15.32:0/648633679 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:48 12.10 12:0d82a743:::600.00000000:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84) -- 0x55dce1afb9a0 con 0x55dce19caf50 2024-08-03T20:54:38.650 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.644+0000 7f9dd7400640 1 -- 172.21.15.32:0/648633679 <== osd.0 v2:172.21.15.32:6816/3066156116 23 ==== osd_op_reply(48 600.00000000 [omap-get-vals out=5b] v0'0 uv5 ondisk = 0) ==== 156+0+5 (crc 0 0 0) 0x7f9dc804f040 con 0x55dce19caf50 2024-08-03T20:54:38.651 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.644+0000 7f9dde35ef40 1 -- 172.21.15.32:0/648633679 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:49 12.10 12:0d89b25e:::603.00000000:head [read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84) -- 0x55dce1afa990 con 0x55dce19caf50 2024-08-03T20:54:38.651 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.644+0000 7f9dd7400640 1 -- 172.21.15.32:0/648633679 <== osd.0 v2:172.21.15.32:6816/3066156116 24 ==== osd_op_reply(49 603.00000000 [read 0~0] v0'0 uv8 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f9dc804f040 con 0x55dce19caf50 2024-08-03T20:54:38.651 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.644+0000 7f9dde35ef40 1 -- 172.21.15.32:0/648633679 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:50 12.10 12:0d89b25e:::603.00000000:head [getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84) -- 0x55dce1afa990 con 0x55dce19caf50 2024-08-03T20:54:38.651 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.644+0000 7f9dd7400640 1 -- 172.21.15.32:0/648633679 <== osd.0 v2:172.21.15.32:6816/3066156116 25 ==== osd_op_reply(50 603.00000000 [getxattrs out=124b] v0'0 uv8 ondisk = 0) ==== 156+0+124 (crc 0 0 0) 0x7f9dc804f040 con 0x55dce19caf50 2024-08-03T20:54:38.652 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.644+0000 7f9dde35ef40 1 -- 172.21.15.32:0/648633679 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:51 12.10 12:0d89b25e:::603.00000000:head [omap-get-header] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84) -- 0x55dce1afa990 con 0x55dce19caf50 2024-08-03T20:54:38.653 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.645+0000 7f9dd7400640 1 -- 172.21.15.32:0/648633679 <== osd.0 v2:172.21.15.32:6816/3066156116 26 ==== osd_op_reply(51 603.00000000 [omap-get-header out=274b] v0'0 uv8 ondisk = 0) ==== 156+0+274 (crc 0 0 0) 0x7f9dc804f040 con 0x55dce19caf50 2024-08-03T20:54:38.653 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.645+0000 7f9dde35ef40 1 -- 172.21.15.32:0/648633679 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:52 12.10 12:0d89b25e:::603.00000000:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84) -- 0x55dce1afb9a0 con 0x55dce19caf50 2024-08-03T20:54:38.653 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.645+0000 7f9dd7400640 1 -- 172.21.15.32:0/648633679 <== osd.0 v2:172.21.15.32:6816/3066156116 27 ==== osd_op_reply(52 603.00000000 [omap-get-vals out=5b] v0'0 uv8 ondisk = 0) ==== 156+0+5 (crc 0 0 0) 0x7f9dc804f040 con 0x55dce19caf50 2024-08-03T20:54:38.653 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.645+0000 7f9dde35ef40 1 -- 172.21.15.32:0/648633679 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:53 12.14 12:287034c4:::1.00000000.inode:head [read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84) -- 0x55dce1afa990 con 0x7f9dc8028040 2024-08-03T20:54:38.654 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.646+0000 7f9dd6a00640 1 -- 172.21.15.32:0/648633679 <== osd.5 v2:172.21.15.118:6810/2724891770 3 ==== osd_op_reply(53 1.00000000.inode [read 0~582 out=582b] v0'0 uv15 ondisk = 0) ==== 160+0+582 (crc 0 0 0) 0x7f9dc403a040 con 0x7f9dc8028040 2024-08-03T20:54:38.654 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.646+0000 7f9dde35ef40 1 -- 172.21.15.32:0/648633679 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:54 12.14 12:287034c4:::1.00000000.inode:head [getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84) -- 0x55dce1afad30 con 0x7f9dc8028040 2024-08-03T20:54:38.654 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.646+0000 7f9dd6a00640 1 -- 172.21.15.32:0/648633679 <== osd.5 v2:172.21.15.118:6810/2724891770 4 ==== osd_op_reply(54 1.00000000.inode [getxattrs out=4b] v0'0 uv15 ondisk = 0) ==== 160+0+4 (crc 0 0 0) 0x7f9dc403a040 con 0x7f9dc8028040 2024-08-03T20:54:38.655 INFO:teuthology.orchestra.run.smithi032.stdout:�������� ��f�����`security.selinux%unconfined_u:object_r:unlabeled_t:s0������������ �� ��*$- 600.00000000�������������������!�������������������������� ���~_layout��������_parent>8stray0 ���� �� ((88���� �� ��*$- 603.00000000�������������������!�������������������������� ���~_layout��������_parent>8stray3 ���� �� ((88���� �� ��.(11.00000000.inode�������������������!�������������������������� ��`ZFFceph fs volume v011)� ��f��8�C@@ �������� ��f��8( ��f��88` ��fh��8��f��&��f��&����NH��f��&������������ �� 2024-08-03T20:54:38.655 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.646+0000 7f9dde35ef40 1 -- 172.21.15.32:0/648633679 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:55 12.14 12:287034c4:::1.00000000.inode:head [omap-get-header] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84) -- 0x55dce1afb0d0 con 0x7f9dc8028040 2024-08-03T20:54:38.655 INFO:teuthology.orchestra.run.smithi032.stdout:���� �� 2024-08-03T20:54:38.656 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.647+0000 7f9dd6a00640 1 -- 172.21.15.32:0/648633679 <== osd.5 v2:172.21.15.118:6810/2724891770 5 ==== osd_op_reply(55 1.00000000.inode [omap-get-header] v0'0 uv15 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f9dc403a040 con 0x7f9dc8028040 2024-08-03T20:54:38.656 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.647+0000 7f9dde35ef40 1 -- 172.21.15.32:0/648633679 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:56 12.14 12:287034c4:::1.00000000.inode:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84) -- 0x55dce1afc480 con 0x7f9dc8028040 2024-08-03T20:54:38.657 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.647+0000 7f9dd6a00640 1 -- 172.21.15.32:0/648633679 <== osd.5 v2:172.21.15.118:6810/2724891770 6 ==== osd_op_reply(56 1.00000000.inode [omap-get-vals out=5b] v0'0 uv15 ondisk = 0) ==== 160+0+5 (crc 0 0 0) 0x7f9dc403a040 con 0x7f9dc8028040 2024-08-03T20:54:38.657 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.647+0000 7f9dde35ef40 1 -- 172.21.15.32:0/648633679 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:57 12.14 12:292cf221:::200.00000000:head [read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84) -- 0x55dce1afb470 con 0x7f9dc8028040 2024-08-03T20:54:38.657 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.648+0000 7f9dd6a00640 1 -- 172.21.15.32:0/648633679 <== osd.5 v2:172.21.15.118:6810/2724891770 7 ==== osd_op_reply(57 200.00000000 [read 0~90 out=90b] v0'0 uv19 ondisk = 0) ==== 156+0+90 (crc 0 0 0) 0x7f9dc403a040 con 0x7f9dc8028040 2024-08-03T20:54:38.657 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.648+0000 7f9dde35ef40 1 -- 172.21.15.32:0/648633679 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:58 12.14 12:292cf221:::200.00000000:head [getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84) -- 0x55dce1afb860 con 0x7f9dc8028040 2024-08-03T20:54:38.657 INFO:teuthology.orchestra.run.smithi032.stdout:���� �� ��*$- 200.00000000�������������������!�������������������������� ��tnZZTceph fs volume v011@�>@�>@�F@@@ ���� �� 2024-08-03T20:54:38.658 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.648+0000 7f9dd6a00640 1 -- 172.21.15.32:0/648633679 <== osd.5 v2:172.21.15.118:6810/2724891770 8 ==== osd_op_reply(58 200.00000000 [getxattrs out=4b] v0'0 uv19 ondisk = 0) ==== 156+0+4 (crc 0 0 0) 0x7f9dc403a040 con 0x7f9dc8028040 2024-08-03T20:54:38.658 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.648+0000 7f9dde35ef40 1 -- 172.21.15.32:0/648633679 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:59 12.14 12:292cf221:::200.00000000:head [omap-get-header] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84) -- 0x55dce1afbc50 con 0x7f9dc8028040 2024-08-03T20:54:38.658 INFO:teuthology.orchestra.run.smithi032.stdout:���� �� 2024-08-03T20:54:38.659 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.649+0000 7f9dd6a00640 1 -- 172.21.15.32:0/648633679 <== osd.5 v2:172.21.15.118:6810/2724891770 9 ==== osd_op_reply(59 200.00000000 [omap-get-header] v0'0 uv19 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f9dc403a040 con 0x7f9dc8028040 2024-08-03T20:54:38.659 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.649+0000 7f9dde35ef40 1 -- 172.21.15.32:0/648633679 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:60 12.14 12:292cf221:::200.00000000:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84) -- 0x55dce1afc040 con 0x7f9dc8028040 2024-08-03T20:54:38.660 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.649+0000 7f9dd6a00640 1 -- 172.21.15.32:0/648633679 <== osd.5 v2:172.21.15.118:6810/2724891770 10 ==== osd_op_reply(60 200.00000000 [omap-get-vals out=5b] v0'0 uv19 ondisk = 0) ==== 156+0+5 (crc 0 0 0) 0x7f9dc403a040 con 0x7f9dc8028040 2024-08-03T20:54:38.660 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.649+0000 7f9dde35ef40 1 -- 172.21.15.32:0/648633679 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:61 12.14 12:2e2fa760:::200.00000001:head [read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84) -- 0x55dce1afc040 con 0x7f9dc8028040 2024-08-03T20:54:38.660 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.650+0000 7f9dd6a00640 1 -- 172.21.15.32:0/648633679 <== osd.5 v2:172.21.15.118:6810/2724891770 11 ==== osd_op_reply(61 200.00000001 [read 0~18118 out=18118b] v0'0 uv16 ondisk = 0) ==== 156+0+18118 (crc 0 0 0) 0x7f9dc403a040 con 0x7f9dc8028040 2024-08-03T20:54:38.660 INFO:teuthology.orchestra.run.smithi032.stdout:���� �� ��*$- 200.00000001�������������������!�������������������������� ���F�F�F�F��XS&YA1e@��XS&YA1h^T��f���2 ((88 2 ( 2024-08-03T20:54:38.660 INFO:teuthology.orchestra.run.smithi032.stdout:( 2024-08-03T20:54:38.661 INFO:teuthology.orchestra.run.smithi032.stdout:8 2024-08-03T20:54:38.661 INFO:teuthology.orchestra.run.smithi032.stdout:��f���8 2024-08-03T20:54:38.661 INFO:teuthology.orchestra.run.smithi032.stdout:��f��� ��������@0@��XS&YA1�� 2024-08-03T20:54:38.661 INFO:teuthology.orchestra.run.smithi032.stdout:� ��f�K['�,� �� � entity_id0hostname smithi032kernel_version6.11.0-rc1-gef231aaae597root/��7 ����@��XS&YA1��� ��f�x+n���������@��XS&YA1 ��f��{+setattr� C��������� ��f�\+�C@@ ����������f��&(8 ��f�\+8��f��&��f��&����NH��f��&�����������������[@��XS&YA1�� ��f�z)8mkdir�p ( ��f��8( ��f��88 ��f��88 ��f��8J 8subdir��������� ��f��8�A�$���������������� ��f��8 ��f��8(88�������� ��f��8����security.selinux%unconfined_u:object_r:unlabeled_t:s0��������2 ((88  C��������� ��f��8�C@@ �������� ��f��8( ��f��88 ��f��88��f��&��f��&����NH��f��&��������������������@��XS&YA1� ��f4��openc�p ( ��f��8( ��f��88 ��f�8 ��f��8J 8subdir��������� ��f��A�$���������������� ��f� ��f��8( ��f�8 ��f�8 ��f��������� ��f��8����security.selinux%unconfined_u:object_r:unlabeled_t:s0��������� ( ��f�( ��f�8 ��f�8 ��f�m [sixmegs��������� ��f����$@@ �������� ��f� ��f��@(88�������� ��f�����security.selinux%unconfined_u:object_r:unlabeled_t:s0����������!!���������@��XS&YA1 ��f��� 2024-08-03T20:54:38.661 INFO:teuthology.orchestra.run.smithi032.stdout:cap update�p ( ��f��8( ��f��88  ��f�?�8 ��f��8J 8subdir��������� ��f��A�$���������������� ��f� ��f��8( ��f�8  ��f�?�8  ��f�?��������� ��f��8����security.selinux%unconfined_u:object_r:unlabeled_t:s0��������� ( ��f�( ��f�8  ��f�?�8  ��f�?�u csixmegs��������� ��f�?����$@@ �������� ��f�?� ��f���(8  ��f�?�8  ��f�?��������� ��f����� security.selinux%unconfined_u:object_r:unlabeled_t:s0������������������@��XS&YA1 ��fP̹ 2024-08-03T20:54:38.661 INFO:teuthology.orchestra.run.smithi032.stdout:cap update�p  ( ��f��8( ��f��88P ��fa �8 ��f��8J 8subdir��������� ��f��A�$���������������� ��f� ��f��8( ��f�8P ��fa �8P ��fa ��������� ��f��8����security.selinux%unconfined_u:object_r:unlabeled_t:s0��������� ( ��f�( ��f�8P ��fa �8P ��fa �u csixmegs��������� ��fa ����$@@ P�������� ��fa � ��f���(8P ��fa �8P ��fa ��������� ��f�����Psecurity.selinux%unconfined_u:object_r:unlabeled_t:s0������������������ @��XS&YA1e 2024-08-03T20:54:38.661 INFO:teuthology.orchestra.run.smithi032.stdout:[ 2024-08-03T20:54:38.661 INFO:teuthology.orchestra.run.smithi032.stdout:Q 2024-08-03T20:54:38.661 INFO:teuthology.orchestra.run.smithi032.stdout: ��f�G 2024-08-03T20:54:38.661 INFO:teuthology.orchestra.run.smithi032.stdout:cap update 2024-08-03T20:54:38.662 INFO:teuthology.orchestra.run.smithi032.stdout:p  ( ��f��8( ��f��88` ��fh��8` ��fh��J 8subdir�������� 2024-08-03T20:54:38.662 INFO:teuthology.orchestra.run.smithi032.stdout:� ��f��A�$���������������� ��f� ��f��8( ��f�8` ��fh��8` ��fh�� 2024-08-03T20:54:38.662 INFO:teuthology.orchestra.run.smithi032.stdout:�������� ��f��8����security.selinux%unconfined_u:object_r:unlabeled_t:s0���������  ( ��f�( ��f�8` ��fh��8` ��fh��u csixmegs��������� ��fh�����$@@ `�������� ��fh�� ��f��(8` ��fh��8` ��fh���������� ��f�����`security.selinux%unconfined_u:object_r:unlabeled_t:s0�������� C��������� ��f��8�C@@ �������� ��f��8( ��f��88` ��fh��8��f��&��f��&����NH��f��&�����������������()@��XS&YA1�� 2024-08-03T20:54:38.662 INFO:teuthology.orchestra.run.smithi032.stdout:� ��f<���,� �� ����3@��XS&YA1* 2024-08-03T20:54:38.662 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-03T20:54:38.662 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-03T20:54:38.662 INFO:teuthology.orchestra.run.smithi032.stdout: ��f[bRcheck_inode_max_size� p  ( ��f��8( ��f��88` ��fh��8` ��fh��J 8subdir�������� � ��f��A�$���������������� ��f� ��f��8( ��f�8` ��fh��8` ��fh�� �������� ��f��8����security.selinux%unconfined_u:object_r:unlabeled_t:s0���������  ( ��f�( ��f�8` ��fh��8` ��fh��� =sixmegs�������� 2024-08-03T20:54:38.662 INFO:teuthology.orchestra.run.smithi032.stdout:� ��fh�����$@@ `�������� ��fh�� ��f�(8` ��fh��8` ��fh�� 2024-08-03T20:54:38.662 INFO:teuthology.orchestra.run.smithi032.stdout:�������� ��f�����`security.selinux%unconfined_u:object_r:unlabeled_t:s0�������� =sixmegs�������� 2024-08-03T20:54:38.662 INFO:teuthology.orchestra.run.smithi032.stdout:� ��fh�����$@@ `�������� ��fh�� ��f�(8` ��fh��8` ��fh�� 2024-08-03T20:54:38.663 INFO:teuthology.orchestra.run.smithi032.stdout:�������� ��f�����`security.selinux%unconfined_u:object_r:unlabeled_t:s0����������������N4@��XS&YA18.$��>@��XS&YA1h^T ��f���22  ( ��f��8( ��f��88` ��fh��8` ��fh�� 2 ( 2024-08-03T20:54:38.663 INFO:teuthology.orchestra.run.smithi032.stdout:( 2024-08-03T20:54:38.663 INFO:teuthology.orchestra.run.smithi032.stdout:8 2024-08-03T20:54:38.663 INFO:teuthology.orchestra.run.smithi032.stdout:��f���8 2024-08-03T20:54:38.663 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.650+0000 7f9dde35ef40 1 -- 172.21.15.32:0/648633679 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:62 12.14 12:2e2fa760:::200.00000001:head [getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84) -- 0x55dce1afc040 con 0x7f9dc8028040 2024-08-03T20:54:38.663 INFO:teuthology.orchestra.run.smithi032.stdout:��f��� ��������@�>@��XS&YA1^TJ ��fm'�2scatter_writebehind2 ( ��f��8( ��f��88` ��fh��8` ��fh��  C��������� ��f��8�C@@ �������� ��f��8( ��f��88` ��fh��8��f��&��f��&����NH��f��&����������������TB@���� �� 2024-08-03T20:54:38.664 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.650+0000 7f9dd6a00640 1 -- 172.21.15.32:0/648633679 <== osd.5 v2:172.21.15.118:6810/2724891770 12 ==== osd_op_reply(62 200.00000001 [getxattrs out=4b] v0'0 uv16 ondisk = 0) ==== 156+0+4 (crc 0 0 0) 0x7f9dc403a040 con 0x7f9dc8028040 2024-08-03T20:54:38.664 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.650+0000 7f9dde35ef40 1 -- 172.21.15.32:0/648633679 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:63 12.14 12:2e2fa760:::200.00000001:head [omap-get-header] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84) -- 0x55dce1afc040 con 0x7f9dc8028040 2024-08-03T20:54:38.664 INFO:teuthology.orchestra.run.smithi032.stdout:���� �� 2024-08-03T20:54:38.665 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.651+0000 7f9dd6a00640 1 -- 172.21.15.32:0/648633679 <== osd.5 v2:172.21.15.118:6810/2724891770 13 ==== osd_op_reply(63 200.00000001 [omap-get-header] v0'0 uv16 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f9dc403a040 con 0x7f9dc8028040 2024-08-03T20:54:38.665 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.651+0000 7f9dde35ef40 1 -- 172.21.15.32:0/648633679 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:64 12.14 12:2e2fa760:::200.00000001:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84) -- 0x55dce1afbff0 con 0x7f9dc8028040 2024-08-03T20:54:38.665 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.651+0000 7f9dd6a00640 1 -- 172.21.15.32:0/648633679 <== osd.5 v2:172.21.15.118:6810/2724891770 14 ==== osd_op_reply(64 200.00000001 [omap-get-vals out=5b] v0'0 uv16 ondisk = 0) ==== 156+0+5 (crc 0 0 0) 0x7f9dc403a040 con 0x7f9dc8028040 2024-08-03T20:54:38.666 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.651+0000 7f9dde35ef40 1 -- 172.21.15.32:0/648633679 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:65 12.14 12:2f55791f:::606.00000000:head [read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84) -- 0x55dce1afbff0 con 0x7f9dc8028040 2024-08-03T20:54:38.666 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.652+0000 7f9dd6a00640 1 -- 172.21.15.32:0/648633679 <== osd.5 v2:172.21.15.118:6810/2724891770 15 ==== osd_op_reply(65 606.00000000 [read 0~0] v0'0 uv2 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f9dc403a040 con 0x7f9dc8028040 2024-08-03T20:54:38.666 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.652+0000 7f9dde35ef40 1 -- 172.21.15.32:0/648633679 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:66 12.14 12:2f55791f:::606.00000000:head [getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84) -- 0x55dce1afbff0 con 0x7f9dc8028040 2024-08-03T20:54:38.666 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.652+0000 7f9dd6a00640 1 -- 172.21.15.32:0/648633679 <== osd.5 v2:172.21.15.118:6810/2724891770 16 ==== osd_op_reply(66 606.00000000 [getxattrs out=124b] v0'0 uv2 ondisk = 0) ==== 156+0+124 (crc 0 0 0) 0x7f9dc403a040 con 0x7f9dc8028040 2024-08-03T20:54:38.666 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.652+0000 7f9dde35ef40 1 -- 172.21.15.32:0/648633679 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:67 12.14 12:2f55791f:::606.00000000:head [omap-get-header] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84) -- 0x55dce1afbff0 con 0x7f9dc8028040 2024-08-03T20:54:38.667 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.652+0000 7f9dd6a00640 1 -- 172.21.15.32:0/648633679 <== osd.5 v2:172.21.15.118:6810/2724891770 17 ==== osd_op_reply(67 606.00000000 [omap-get-header out=274b] v0'0 uv2 ondisk = 0) ==== 156+0+274 (crc 0 0 0) 0x7f9dc403a040 con 0x7f9dc8028040 2024-08-03T20:54:38.667 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.652+0000 7f9dde35ef40 1 -- 172.21.15.32:0/648633679 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:68 12.14 12:2f55791f:::606.00000000:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84) -- 0x55dce1afbff0 con 0x7f9dc8028040 2024-08-03T20:54:38.668 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.653+0000 7f9dd6a00640 1 -- 172.21.15.32:0/648633679 <== osd.5 v2:172.21.15.118:6810/2724891770 18 ==== osd_op_reply(68 606.00000000 [omap-get-vals out=5b] v0'0 uv2 ondisk = 0) ==== 156+0+5 (crc 0 0 0) 0x7f9dc403a040 con 0x7f9dc8028040 2024-08-03T20:54:38.668 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.653+0000 7f9dde35ef40 1 -- 172.21.15.32:0/648633679 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:69 12.1c 12:3ed09add:::607.00000000:head [read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84) -- 0x55dce1afbff0 con 0x55dce19caf50 2024-08-03T20:54:38.668 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.653+0000 7f9dd7400640 1 -- 172.21.15.32:0/648633679 <== osd.0 v2:172.21.15.32:6816/3066156116 28 ==== osd_op_reply(69 607.00000000 [read 0~0] v0'0 uv2 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f9dc804f040 con 0x55dce19caf50 2024-08-03T20:54:38.668 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.653+0000 7f9dde35ef40 1 -- 172.21.15.32:0/648633679 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:70 12.1c 12:3ed09add:::607.00000000:head [getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84) -- 0x55dce1afbff0 con 0x55dce19caf50 2024-08-03T20:54:38.668 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.654+0000 7f9dd7400640 1 -- 172.21.15.32:0/648633679 <== osd.0 v2:172.21.15.32:6816/3066156116 29 ==== osd_op_reply(70 607.00000000 [getxattrs out=124b] v0'0 uv2 ondisk = 0) ==== 156+0+124 (crc 0 0 0) 0x7f9dc804f040 con 0x55dce19caf50 2024-08-03T20:54:38.669 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.654+0000 7f9dde35ef40 1 -- 172.21.15.32:0/648633679 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:71 12.1c 12:3ed09add:::607.00000000:head [omap-get-header] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84) -- 0x55dce1afbff0 con 0x55dce19caf50 2024-08-03T20:54:38.669 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.654+0000 7f9dd7400640 1 -- 172.21.15.32:0/648633679 <== osd.0 v2:172.21.15.32:6816/3066156116 30 ==== osd_op_reply(71 607.00000000 [omap-get-header out=274b] v0'0 uv2 ondisk = 0) ==== 156+0+274 (crc 0 0 0) 0x7f9dc804f040 con 0x55dce19caf50 2024-08-03T20:54:38.669 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.654+0000 7f9dde35ef40 1 -- 172.21.15.32:0/648633679 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:72 12.1c 12:3ed09add:::607.00000000:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84) -- 0x55dce1afbff0 con 0x55dce19caf50 2024-08-03T20:54:38.670 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.654+0000 7f9dd7400640 1 -- 172.21.15.32:0/648633679 <== osd.0 v2:172.21.15.32:6816/3066156116 31 ==== osd_op_reply(72 607.00000000 [omap-get-vals out=5b] v0'0 uv2 ondisk = 0) ==== 156+0+5 (crc 0 0 0) 0x7f9dc804f040 con 0x55dce19caf50 2024-08-03T20:54:38.670 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.654+0000 7f9dde35ef40 1 -- 172.21.15.32:0/648633679 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:73 12.6 12:654134d2:::mds0_openfiles.0:head [read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84) -- 0x55dce1afbff0 con 0x7f9dc4024400 2024-08-03T20:54:38.670 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.655+0000 7f9dd7e00640 1 -- 172.21.15.32:0/648633679 <== osd.1 v2:172.21.15.32:6824/1551241759 5 ==== osd_op_reply(73 mds0_openfiles.0 [read 0~0] v0'0 uv4 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f9dd00a9d70 con 0x7f9dc4024400 2024-08-03T20:54:38.671 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.655+0000 7f9dde35ef40 1 -- 172.21.15.32:0/648633679 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:74 12.6 12:654134d2:::mds0_openfiles.0:head [getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84) -- 0x55dce1afc820 con 0x7f9dc4024400 2024-08-03T20:54:38.671 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.655+0000 7f9dd7e00640 1 -- 172.21.15.32:0/648633679 <== osd.1 v2:172.21.15.32:6824/1551241759 6 ==== osd_op_reply(74 mds0_openfiles.0 [getxattrs out=4b] v0'0 uv4 ondisk = 0) ==== 160+0+4 (crc 0 0 0) 0x7f9dd00a9d70 con 0x7f9dc4024400 2024-08-03T20:54:38.671 INFO:teuthology.orchestra.run.smithi032.stdout:���� �� ��*$- 606.00000000�������������������!�������������������������� ���~_layout��������_parent>8stray6 ���� �� ((88���� �� ��*$- 607.00000000�������������������!�������������������������� ���~_layout��������_parent>8stray7 ���� �� ((88���� �� ��.(1mds0_openfiles.0�������������������!�������������������������� �� 2024-08-03T20:54:38.672 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.655+0000 7f9dde35ef40 1 -- 172.21.15.32:0/648633679 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:75 12.6 12:654134d2:::mds0_openfiles.0:head [omap-get-header] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84) -- 0x55dce1afcbc0 con 0x7f9dc4024400 2024-08-03T20:54:38.672 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.655+0000 7f9dd7e00640 1 -- 172.21.15.32:0/648633679 <== osd.1 v2:172.21.15.32:6824/1551241759 7 ==== osd_op_reply(75 mds0_openfiles.0 [omap-get-header out=42b] v0'0 uv4 ondisk = 0) ==== 160+0+42 (crc 0 0 0) 0x7f9dd00a9d70 con 0x7f9dc4024400 2024-08-03T20:54:38.672 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.655+0000 7f9dde35ef40 1 -- 172.21.15.32:0/648633679 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:76 12.6 12:654134d2:::mds0_openfiles.0:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84) -- 0x55dce1afdf70 con 0x7f9dc4024400 2024-08-03T20:54:38.672 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.656+0000 7f9dd7e00640 1 -- 172.21.15.32:0/648633679 <== osd.1 v2:172.21.15.32:6824/1551241759 8 ==== osd_op_reply(76 mds0_openfiles.0 [omap-get-vals out=5b] v0'0 uv4 ondisk = 0) ==== 160+0+5 (crc 0 0 0) 0x7f9dd00a9d70 con 0x7f9dc4024400 2024-08-03T20:54:38.673 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.656+0000 7f9dde35ef40 1 -- 172.21.15.32:0/648633679 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:77 12.e 12:717a0223:::608.00000000:head [read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84) -- 0x55dce1afcf60 con 0x55dce19caf50 2024-08-03T20:54:38.673 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.656+0000 7f9dd7400640 1 -- 172.21.15.32:0/648633679 <== osd.0 v2:172.21.15.32:6816/3066156116 32 ==== osd_op_reply(77 608.00000000 [read 0~0] v0'0 uv2 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f9dc804f040 con 0x55dce19caf50 2024-08-03T20:54:38.673 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.656+0000 7f9dde35ef40 1 -- 172.21.15.32:0/648633679 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:78 12.e 12:717a0223:::608.00000000:head [getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84) -- 0x55dce1afcf60 con 0x55dce19caf50 2024-08-03T20:54:38.673 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.656+0000 7f9dd7400640 1 -- 172.21.15.32:0/648633679 <== osd.0 v2:172.21.15.32:6816/3066156116 33 ==== osd_op_reply(78 608.00000000 [getxattrs out=124b] v0'0 uv2 ondisk = 0) ==== 156+0+124 (crc 0 0 0) 0x7f9dc804f040 con 0x55dce19caf50 2024-08-03T20:54:38.674 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.656+0000 7f9dde35ef40 1 -- 172.21.15.32:0/648633679 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:79 12.e 12:717a0223:::608.00000000:head [omap-get-header] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84) -- 0x55dce1afcf60 con 0x55dce19caf50 2024-08-03T20:54:38.674 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.657+0000 7f9dd7400640 1 -- 172.21.15.32:0/648633679 <== osd.0 v2:172.21.15.32:6816/3066156116 34 ==== osd_op_reply(79 608.00000000 [omap-get-header out=274b] v0'0 uv2 ondisk = 0) ==== 156+0+274 (crc 0 0 0) 0x7f9dc804f040 con 0x55dce19caf50 2024-08-03T20:54:38.674 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.657+0000 7f9dde35ef40 1 -- 172.21.15.32:0/648633679 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:80 12.e 12:717a0223:::608.00000000:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84) -- 0x55dce1afe310 con 0x55dce19caf50 2024-08-03T20:54:38.675 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.657+0000 7f9dd7400640 1 -- 172.21.15.32:0/648633679 <== osd.0 v2:172.21.15.32:6816/3066156116 35 ==== osd_op_reply(80 608.00000000 [omap-get-vals out=5b] v0'0 uv2 ondisk = 0) ==== 156+0+5 (crc 0 0 0) 0x7f9dc804f040 con 0x55dce19caf50 2024-08-03T20:54:38.676 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.657+0000 7f9dde35ef40 1 -- 172.21.15.32:0/648633679 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:81 12.15 12:a93a17c2:::604.00000000:head [read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84) -- 0x55dce1afcf60 con 0x7f9dd0022d70 2024-08-03T20:54:38.676 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.658+0000 7f9dd7400640 1 -- 172.21.15.32:0/648633679 <== osd.6 v2:172.21.15.118:6825/801947386 9 ==== osd_op_reply(81 604.00000000 [read 0~0] v0'0 uv2 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f9dc804f040 con 0x7f9dd0022d70 2024-08-03T20:54:38.676 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.658+0000 7f9dde35ef40 1 -- 172.21.15.32:0/648633679 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:82 12.15 12:a93a17c2:::604.00000000:head [getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84) -- 0x55dce1afcf60 con 0x7f9dd0022d70 2024-08-03T20:54:38.676 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.658+0000 7f9dd7400640 1 -- 172.21.15.32:0/648633679 <== osd.6 v2:172.21.15.118:6825/801947386 10 ==== osd_op_reply(82 604.00000000 [getxattrs out=124b] v0'0 uv2 ondisk = 0) ==== 156+0+124 (crc 0 0 0) 0x7f9dc804f040 con 0x7f9dd0022d70 2024-08-03T20:54:38.676 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.658+0000 7f9dde35ef40 1 -- 172.21.15.32:0/648633679 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:83 12.15 12:a93a17c2:::604.00000000:head [omap-get-header] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84) -- 0x55dce1afcf60 con 0x7f9dd0022d70 2024-08-03T20:54:38.677 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.658+0000 7f9dd7400640 1 -- 172.21.15.32:0/648633679 <== osd.6 v2:172.21.15.118:6825/801947386 11 ==== osd_op_reply(83 604.00000000 [omap-get-header out=274b] v0'0 uv2 ondisk = 0) ==== 156+0+274 (crc 0 0 0) 0x7f9dc804f040 con 0x7f9dd0022d70 2024-08-03T20:54:38.677 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.659+0000 7f9dde35ef40 1 -- 172.21.15.32:0/648633679 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:84 12.15 12:a93a17c2:::604.00000000:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84) -- 0x55dce1afe310 con 0x7f9dd0022d70 2024-08-03T20:54:38.678 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.659+0000 7f9dd7400640 1 -- 172.21.15.32:0/648633679 <== osd.6 v2:172.21.15.118:6825/801947386 12 ==== osd_op_reply(84 604.00000000 [omap-get-vals out=5b] v0'0 uv2 ondisk = 0) ==== 156+0+5 (crc 0 0 0) 0x7f9dc804f040 con 0x7f9dd0022d70 2024-08-03T20:54:38.678 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.659+0000 7f9dde35ef40 1 -- 172.21.15.32:0/648633679 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:85 12.15 12:aa448500:::500.00000000:head [read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84) -- 0x55dce1afcf60 con 0x7f9dd0022d70 2024-08-03T20:54:38.678 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.660+0000 7f9dd7400640 1 -- 172.21.15.32:0/648633679 <== osd.6 v2:172.21.15.118:6825/801947386 13 ==== osd_op_reply(85 500.00000000 [read 0~90 out=90b] v0'0 uv3 ondisk = 0) ==== 156+0+90 (crc 0 0 0) 0x7f9dc804f040 con 0x7f9dd0022d70 2024-08-03T20:54:38.678 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.660+0000 7f9dde35ef40 1 -- 172.21.15.32:0/648633679 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:86 12.15 12:aa448500:::500.00000000:head [getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84) -- 0x55dce1afcf60 con 0x7f9dd0022d70 2024-08-03T20:54:38.679 INFO:teuthology.orchestra.run.smithi032.stdout:���� ��4.*ceph fs volume v011 ���� �� ��*$- 608.00000000�������������������!�������������������������� ���~_layout��������_parent>8stray8 ���� �� ((88���� �� ��*$- 604.00000000�������������������!�������������������������� ���~_layout��������_parent>8stray4 ���� �� ((88���� �� ��*$- 500.00000000�������������������!�������������������������� ��tnZZTceph fs volume v011@@@@@@ ���� �� 2024-08-03T20:54:38.679 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.660+0000 7f9dd7400640 1 -- 172.21.15.32:0/648633679 <== osd.6 v2:172.21.15.118:6825/801947386 14 ==== osd_op_reply(86 500.00000000 [getxattrs out=4b] v0'0 uv3 ondisk = 0) ==== 156+0+4 (crc 0 0 0) 0x7f9dc804f040 con 0x7f9dd0022d70 2024-08-03T20:54:38.679 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.660+0000 7f9dde35ef40 1 -- 172.21.15.32:0/648633679 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:87 12.15 12:aa448500:::500.00000000:head [omap-get-header] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84) -- 0x55dce1afcf60 con 0x7f9dd0022d70 2024-08-03T20:54:38.679 INFO:teuthology.orchestra.run.smithi032.stdout:���� �� 2024-08-03T20:54:38.680 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.661+0000 7f9dd7400640 1 -- 172.21.15.32:0/648633679 <== osd.6 v2:172.21.15.118:6825/801947386 15 ==== osd_op_reply(87 500.00000000 [omap-get-header] v0'0 uv3 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f9dc804f040 con 0x7f9dd0022d70 2024-08-03T20:54:38.680 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.661+0000 7f9dde35ef40 1 -- 172.21.15.32:0/648633679 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:88 12.15 12:aa448500:::500.00000000:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84) -- 0x55dce1afcf60 con 0x7f9dd0022d70 2024-08-03T20:54:38.680 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.661+0000 7f9dd7400640 1 -- 172.21.15.32:0/648633679 <== osd.6 v2:172.21.15.118:6825/801947386 16 ==== osd_op_reply(88 500.00000000 [omap-get-vals out=5b] v0'0 uv3 ondisk = 0) ==== 156+0+5 (crc 0 0 0) 0x7f9dc804f040 con 0x7f9dd0022d70 2024-08-03T20:54:38.681 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.661+0000 7f9dde35ef40 1 -- 172.21.15.32:0/648633679 --> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] -- osd_op(unknown.0.0:89 12.d 12:b50e409b:::mds_snaptable:head [read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84) -- 0x55dce1afcf60 con 0x7f9dc8034a80 2024-08-03T20:54:38.681 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.662+0000 7f9dd7400640 1 -- 172.21.15.32:0/648633679 <== osd.7 v2:172.21.15.118:6818/653000048 2 ==== osd_op_reply(89 mds_snaptable [read 0~70 out=70b] v0'0 uv1 ondisk = 0) ==== 157+0+70 (crc 0 0 0) 0x7f9dc804f040 con 0x7f9dc8034a80 2024-08-03T20:54:38.681 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.662+0000 7f9dde35ef40 1 -- 172.21.15.32:0/648633679 --> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] -- osd_op(unknown.0.0:90 12.d 12:b50e409b:::mds_snaptable:head [getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84) -- 0x55dce1afcf60 con 0x7f9dc8034a80 2024-08-03T20:54:38.682 INFO:teuthology.orchestra.run.smithi032.stdout:���� �� ��+%. mds_snaptable�������������������!�������������������������� ��`ZFF4���� �� 2024-08-03T20:54:38.682 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.662+0000 7f9dd7400640 1 -- 172.21.15.32:0/648633679 <== osd.7 v2:172.21.15.118:6818/653000048 3 ==== osd_op_reply(90 mds_snaptable [getxattrs out=4b] v0'0 uv1 ondisk = 0) ==== 157+0+4 (crc 0 0 0) 0x7f9dc804f040 con 0x7f9dc8034a80 2024-08-03T20:54:38.682 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.662+0000 7f9dde35ef40 1 -- 172.21.15.32:0/648633679 --> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] -- osd_op(unknown.0.0:91 12.d 12:b50e409b:::mds_snaptable:head [omap-get-header] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84) -- 0x55dce1afcf60 con 0x7f9dc8034a80 2024-08-03T20:54:38.682 INFO:teuthology.orchestra.run.smithi032.stdout:���� �� 2024-08-03T20:54:38.683 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.663+0000 7f9dd7400640 1 -- 172.21.15.32:0/648633679 <== osd.7 v2:172.21.15.118:6818/653000048 4 ==== osd_op_reply(91 mds_snaptable [omap-get-header] v0'0 uv1 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f9dc804f040 con 0x7f9dc8034a80 2024-08-03T20:54:38.683 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.663+0000 7f9dde35ef40 1 -- 172.21.15.32:0/648633679 --> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] -- osd_op(unknown.0.0:92 12.d 12:b50e409b:::mds_snaptable:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84) -- 0x55dce1afe310 con 0x7f9dc8034a80 2024-08-03T20:54:38.684 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.663+0000 7f9dd7400640 1 -- 172.21.15.32:0/648633679 <== osd.7 v2:172.21.15.118:6818/653000048 5 ==== osd_op_reply(92 mds_snaptable [omap-get-vals out=5b] v0'0 uv1 ondisk = 0) ==== 157+0+5 (crc 0 0 0) 0x7f9dc804f040 con 0x7f9dc8034a80 2024-08-03T20:54:38.684 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.663+0000 7f9dde35ef40 1 -- 172.21.15.32:0/648633679 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:93 12.1d 12:b871830b:::605.00000000:head [read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84) -- 0x55dce1afcf60 con 0x55dce19caf50 2024-08-03T20:54:38.684 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.663+0000 7f9dd7400640 1 -- 172.21.15.32:0/648633679 <== osd.0 v2:172.21.15.32:6816/3066156116 36 ==== osd_op_reply(93 605.00000000 [read 0~0] v0'0 uv2 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f9dc804f040 con 0x55dce19caf50 2024-08-03T20:54:38.684 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.663+0000 7f9dde35ef40 1 -- 172.21.15.32:0/648633679 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:94 12.1d 12:b871830b:::605.00000000:head [getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84) -- 0x55dce1afcf60 con 0x55dce19caf50 2024-08-03T20:54:38.684 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.664+0000 7f9dd7400640 1 -- 172.21.15.32:0/648633679 <== osd.0 v2:172.21.15.32:6816/3066156116 37 ==== osd_op_reply(94 605.00000000 [getxattrs out=124b] v0'0 uv2 ondisk = 0) ==== 156+0+124 (crc 0 0 0) 0x7f9dc804f040 con 0x55dce19caf50 2024-08-03T20:54:38.685 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.664+0000 7f9dde35ef40 1 -- 172.21.15.32:0/648633679 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:95 12.1d 12:b871830b:::605.00000000:head [omap-get-header] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84) -- 0x55dce1afcf60 con 0x55dce19caf50 2024-08-03T20:54:38.685 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.664+0000 7f9dd7400640 1 -- 172.21.15.32:0/648633679 <== osd.0 v2:172.21.15.32:6816/3066156116 38 ==== osd_op_reply(95 605.00000000 [omap-get-header out=274b] v0'0 uv2 ondisk = 0) ==== 156+0+274 (crc 0 0 0) 0x7f9dc804f040 con 0x55dce19caf50 2024-08-03T20:54:38.686 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.664+0000 7f9dde35ef40 1 -- 172.21.15.32:0/648633679 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:96 12.1d 12:b871830b:::605.00000000:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84) -- 0x55dce1afe310 con 0x55dce19caf50 2024-08-03T20:54:38.686 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.664+0000 7f9dd7400640 1 -- 172.21.15.32:0/648633679 <== osd.0 v2:172.21.15.32:6816/3066156116 39 ==== osd_op_reply(96 605.00000000 [omap-get-vals out=5b] v0'0 uv2 ondisk = 0) ==== 156+0+5 (crc 0 0 0) 0x7f9dc804f040 con 0x55dce19caf50 2024-08-03T20:54:38.686 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.664+0000 7f9dde35ef40 1 -- 172.21.15.32:0/648633679 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:97 12.13 12:c91d4a1d:::mds0_inotable:head [read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84) -- 0x55dce1afcf60 con 0x7f9dc4024400 2024-08-03T20:54:38.686 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.665+0000 7f9dd7e00640 1 -- 172.21.15.32:0/648633679 <== osd.1 v2:172.21.15.32:6824/1551241759 9 ==== osd_op_reply(97 mds0_inotable [read 0~34 out=34b] v0'0 uv3 ondisk = 0) ==== 157+0+34 (crc 0 0 0) 0x7f9dd00a9d70 con 0x7f9dc4024400 2024-08-03T20:54:38.687 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.665+0000 7f9dde35ef40 1 -- 172.21.15.32:0/648633679 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:98 12.13 12:c91d4a1d:::mds0_inotable:head [getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84) -- 0x55dce1afd300 con 0x7f9dc4024400 2024-08-03T20:54:38.687 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.665+0000 7f9dd7e00640 1 -- 172.21.15.32:0/648633679 <== osd.1 v2:172.21.15.32:6824/1551241759 10 ==== osd_op_reply(98 mds0_inotable [getxattrs out=4b] v0'0 uv3 ondisk = 0) ==== 157+0+4 (crc 0 0 0) 0x7f9dd00a9d70 con 0x7f9dc4024400 2024-08-03T20:54:38.687 INFO:teuthology.orchestra.run.smithi032.stdout:���� �� ��*$- 605.00000000�������������������!�������������������������� ���~_layout��������_parent>8stray5 ���� �� ((88���� �� ��+%. mds0_inotable�������������������!�������������������������� ��<6""��������� �� 2024-08-03T20:54:38.688 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.665+0000 7f9dde35ef40 1 -- 172.21.15.32:0/648633679 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:99 12.13 12:c91d4a1d:::mds0_inotable:head [omap-get-header] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84) -- 0x55dce1afd6f0 con 0x7f9dc4024400 2024-08-03T20:54:38.688 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.665+0000 7f9dd7e00640 1 -- 172.21.15.32:0/648633679 <== osd.1 v2:172.21.15.32:6824/1551241759 11 ==== osd_op_reply(99 mds0_inotable [omap-get-header] v0'0 uv3 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f9dd00a9d70 con 0x7f9dc4024400 2024-08-03T20:54:38.688 INFO:teuthology.orchestra.run.smithi032.stdout:���� �� 2024-08-03T20:54:38.689 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.665+0000 7f9dde35ef40 1 -- 172.21.15.32:0/648633679 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:100 12.13 12:c91d4a1d:::mds0_inotable:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84) -- 0x55dce1afdae0 con 0x7f9dc4024400 2024-08-03T20:54:38.689 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.666+0000 7f9dd7e00640 1 -- 172.21.15.32:0/648633679 <== osd.1 v2:172.21.15.32:6824/1551241759 12 ==== osd_op_reply(100 mds0_inotable [omap-get-vals out=5b] v0'0 uv3 ondisk = 0) ==== 157+0+5 (crc 0 0 0) 0x7f9dd00a9d70 con 0x7f9dc4024400 2024-08-03T20:54:38.689 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.666+0000 7f9dde35ef40 1 -- 172.21.15.32:0/648633679 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:101 12.13 12:cd5a64a3:::100.00000000:head [read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84) -- 0x55dce1afdae0 con 0x7f9dc4024400 2024-08-03T20:54:38.689 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.666+0000 7f9dd7e00640 1 -- 172.21.15.32:0/648633679 <== osd.1 v2:172.21.15.32:6824/1551241759 13 ==== osd_op_reply(101 100.00000000 [read 0~0] v0'0 uv1 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f9dd00a9d70 con 0x7f9dc4024400 2024-08-03T20:54:38.690 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.666+0000 7f9dde35ef40 1 -- 172.21.15.32:0/648633679 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:102 12.13 12:cd5a64a3:::100.00000000:head [getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84) -- 0x55dce1afdae0 con 0x7f9dc4024400 2024-08-03T20:54:38.690 INFO:teuthology.orchestra.run.smithi032.stdout:���� �� ��*$- 100.00000000�������������������!�������������������������� �� 2024-08-03T20:54:38.690 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.667+0000 7f9dd7e00640 1 -- 172.21.15.32:0/648633679 <== osd.1 v2:172.21.15.32:6824/1551241759 14 ==== osd_op_reply(102 100.00000000 [getxattrs out=4b] v0'0 uv1 ondisk = 0) ==== 156+0+4 (crc 0 0 0) 0x7f9dd00a9d70 con 0x7f9dc4024400 2024-08-03T20:54:38.690 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.667+0000 7f9dde35ef40 1 -- 172.21.15.32:0/648633679 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:103 12.13 12:cd5a64a3:::100.00000000:head [omap-get-header] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84) -- 0x55dce1afdae0 con 0x7f9dc4024400 2024-08-03T20:54:38.691 INFO:teuthology.orchestra.run.smithi032.stdout:���� �� ( 2024-08-03T20:54:38.691 INFO:teuthology.orchestra.run.smithi032.stdout:( 2024-08-03T20:54:38.691 INFO:teuthology.orchestra.run.smithi032.stdout:8 2024-08-03T20:54:38.691 INFO:teuthology.orchestra.run.smithi032.stdout:��f���8 2024-08-03T20:54:38.691 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.667+0000 7f9dd7e00640 1 -- 172.21.15.32:0/648633679 <== osd.1 v2:172.21.15.32:6824/1551241759 15 ==== osd_op_reply(103 100.00000000 [omap-get-header out=274b] v0'0 uv1 ondisk = 0) ==== 156+0+274 (crc 0 0 0) 0x7f9dd00a9d70 con 0x7f9dc4024400 2024-08-03T20:54:38.691 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.667+0000 7f9dde35ef40 1 -- 172.21.15.32:0/648633679 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:104 12.13 12:cd5a64a3:::100.00000000:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84) -- 0x55dce1afdae0 con 0x7f9dc4024400 2024-08-03T20:54:38.692 INFO:teuthology.orchestra.run.smithi032.stdout:��f������� ��PJ 2024-08-03T20:54:38.692 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.667+0000 7f9dd7e00640 1 -- 172.21.15.32:0/648633679 <== osd.1 v2:172.21.15.32:6824/1551241759 16 ==== osd_op_reply(104 100.00000000 [omap-get-vals out=5195b] v0'0 uv1 ondisk = 0) ==== 156+0+5195 (crc 0 0 0) 0x7f9dd00a9d70 con 0x7f9dc4024400 2024-08-03T20:54:38.693 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.668+0000 7f9dde35ef40 1 -- 172.21.15.32:0/648633679 --> [v2:172.21.15.32:6800/9088303,v1:172.21.15.32:6801/9088303] -- osd_op(unknown.0.0:105 12.b 12:d0630e4c:::mds0_sessionmap:head [read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84) -- 0x55dce1afe310 con 0x7f9dc8008950 2024-08-03T20:54:38.693 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.668+0000 7f9dd6a00640 1 -- 172.21.15.32:0/648633679 <== osd.3 v2:172.21.15.32:6800/9088303 4 ==== osd_op_reply(105 mds0_sessionmap [read 0~0] v0'0 uv3 ondisk = 0) ==== 159+0+0 (crc 0 0 0) 0x7f9dc403a040 con 0x7f9dc8008950 2024-08-03T20:54:38.693 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.668+0000 7f9dde35ef40 1 -- 172.21.15.32:0/648633679 --> [v2:172.21.15.32:6800/9088303,v1:172.21.15.32:6801/9088303] -- osd_op(unknown.0.0:106 12.b 12:d0630e4c:::mds0_sessionmap:head [getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84) -- 0x55dce1afe310 con 0x7f9dc8008950 2024-08-03T20:54:38.693 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.668+0000 7f9dd6a00640 1 -- 172.21.15.32:0/648633679 <== osd.3 v2:172.21.15.32:6800/9088303 5 ==== osd_op_reply(106 mds0_sessionmap [getxattrs out=4b] v0'0 uv3 ondisk = 0) ==== 159+0+4 (crc 0 0 0) 0x7f9dc403a040 con 0x7f9dc8008950 2024-08-03T20:54:38.693 INFO:teuthology.orchestra.run.smithi032.stdout: stray0_head�i�����f)/>@A������������������f)/>(8��f)/>8��f)/>��f)/>������������ stray1_head�i�����f�@@A������������������f�@(8��f�@8��f�@��f�@������������ stray2_head�i�����f C@A������������������f C(8��f C8��f C��f C������������ stray3_head�i�����f�F@A������������������f�F(8��f�F8��f�F��f�F������������ stray4_head�i�����f��H@A������������������f��H(8��f��H8��f��H��f��H������������ stray5_head�i�����f�c@A������������������f�c(8��f�c8��f�c��f�c������������ stray6_head�i�����f�-s@A������������������f�-s(8��f�-s8��f�-s��f�-s������������ stray7_head�i�����f35z@A������������������f35z(8��f35z8��f35z��f35z������������ stray8_head�i�����f��@A������������������f��(8��f��8��f����f�������������� stray9_head�i��� ��f���@A������������������f���(8��f���8��f�����f������������������� �� ��-'0mds0_sessionmap�������������������!�������������������������� �� 2024-08-03T20:54:38.694 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.668+0000 7f9dde35ef40 1 -- 172.21.15.32:0/648633679 --> [v2:172.21.15.32:6800/9088303,v1:172.21.15.32:6801/9088303] -- osd_op(unknown.0.0:107 12.b 12:d0630e4c:::mds0_sessionmap:head [omap-get-header] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84) -- 0x55dce1afe310 con 0x7f9dc8008950 2024-08-03T20:54:38.694 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.669+0000 7f9dd6a00640 1 -- 172.21.15.32:0/648633679 <== osd.3 v2:172.21.15.32:6800/9088303 6 ==== osd_op_reply(107 mds0_sessionmap [omap-get-header out=14b] v0'0 uv3 ondisk = 0) ==== 159+0+14 (crc 0 0 0) 0x7f9dc403a040 con 0x7f9dc8008950 2024-08-03T20:54:38.695 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.669+0000 7f9dde35ef40 1 -- 172.21.15.32:0/648633679 --> [v2:172.21.15.32:6800/9088303,v1:172.21.15.32:6801/9088303] -- osd_op(unknown.0.0:108 12.b 12:d0630e4c:::mds0_sessionmap:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84) -- 0x55dce1aff320 con 0x7f9dc8008950 2024-08-03T20:54:38.695 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.669+0000 7f9dd6a00640 1 -- 172.21.15.32:0/648633679 <== osd.3 v2:172.21.15.32:6800/9088303 7 ==== osd_op_reply(108 mds0_sessionmap [omap-get-vals out=5b] v0'0 uv3 ondisk = 0) ==== 159+0+5 (crc 0 0 0) 0x7f9dc403a040 con 0x7f9dc8008950 2024-08-03T20:54:38.695 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.669+0000 7f9dde35ef40 1 -- 172.21.15.32:0/648633679 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:109 12.7 12:e0b41d06:::609.00000000:head [read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84) -- 0x55dce1afdae0 con 0x7f9dc4024400 2024-08-03T20:54:38.696 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.669+0000 7f9dd7e00640 1 -- 172.21.15.32:0/648633679 <== osd.1 v2:172.21.15.32:6824/1551241759 17 ==== osd_op_reply(109 609.00000000 [read 0~0] v0'0 uv2 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f9dd00a9d70 con 0x7f9dc4024400 2024-08-03T20:54:38.696 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.669+0000 7f9dde35ef40 1 -- 172.21.15.32:0/648633679 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:110 12.7 12:e0b41d06:::609.00000000:head [getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84) -- 0x55dce1afdae0 con 0x7f9dc4024400 2024-08-03T20:54:38.696 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.670+0000 7f9dd7e00640 1 -- 172.21.15.32:0/648633679 <== osd.1 v2:172.21.15.32:6824/1551241759 18 ==== osd_op_reply(110 609.00000000 [getxattrs out=124b] v0'0 uv2 ondisk = 0) ==== 156+0+124 (crc 0 0 0) 0x7f9dd00a9d70 con 0x7f9dc4024400 2024-08-03T20:54:38.696 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.670+0000 7f9dde35ef40 1 -- 172.21.15.32:0/648633679 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:111 12.7 12:e0b41d06:::609.00000000:head [omap-get-header] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84) -- 0x55dce1afdae0 con 0x7f9dc4024400 2024-08-03T20:54:38.696 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.670+0000 7f9dd7e00640 1 -- 172.21.15.32:0/648633679 <== osd.1 v2:172.21.15.32:6824/1551241759 19 ==== osd_op_reply(111 609.00000000 [omap-get-header out=274b] v0'0 uv2 ondisk = 0) ==== 156+0+274 (crc 0 0 0) 0x7f9dd00a9d70 con 0x7f9dc4024400 2024-08-03T20:54:38.697 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.670+0000 7f9dde35ef40 1 -- 172.21.15.32:0/648633679 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:112 12.7 12:e0b41d06:::609.00000000:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84) -- 0x55dce1afdae0 con 0x7f9dc4024400 2024-08-03T20:54:38.697 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.670+0000 7f9dd7e00640 1 -- 172.21.15.32:0/648633679 <== osd.1 v2:172.21.15.32:6824/1551241759 20 ==== osd_op_reply(112 609.00000000 [omap-get-vals out=5b] v0'0 uv2 ondisk = 0) ==== 156+0+5 (crc 0 0 0) 0x7f9dd00a9d70 con 0x7f9dc4024400 2024-08-03T20:54:38.698 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.670+0000 7f9dde35ef40 1 -- 172.21.15.32:0/648633679 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:113 12.f 12:f1f69726:::400.00000000:head [read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84) -- 0x55dce1afdae0 con 0x7f9dd0022d70 2024-08-03T20:54:38.698 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.671+0000 7f9dd7400640 1 -- 172.21.15.32:0/648633679 <== osd.6 v2:172.21.15.118:6825/801947386 17 ==== osd_op_reply(113 400.00000000 [read 0~22 out=22b] v0'0 uv1 ondisk = 0) ==== 156+0+22 (crc 0 0 0) 0x7f9dc804f040 con 0x7f9dd0022d70 2024-08-03T20:54:38.698 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.671+0000 7f9dde35ef40 1 -- 172.21.15.32:0/648633679 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:114 12.f 12:f1f69726:::400.00000000:head [getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84) -- 0x55dce1afdae0 con 0x7f9dd0022d70 2024-08-03T20:54:38.699 INFO:teuthology.orchestra.run.smithi032.stdout:���� ������ �� ��*$- 609.00000000�������������������!�������������������������� ���~_layout��������_parent>8 stray9 ���� �� ((88���� �� ��*$- 400.00000000�������������������!�������������������������� ��0*���� �� 2024-08-03T20:54:38.699 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.672+0000 7f9dd7400640 1 -- 172.21.15.32:0/648633679 <== osd.6 v2:172.21.15.118:6825/801947386 18 ==== osd_op_reply(114 400.00000000 [getxattrs out=4b] v0'0 uv1 ondisk = 0) ==== 156+0+4 (crc 0 0 0) 0x7f9dc804f040 con 0x7f9dd0022d70 2024-08-03T20:54:38.699 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.672+0000 7f9dde35ef40 1 -- 172.21.15.32:0/648633679 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:115 12.f 12:f1f69726:::400.00000000:head [omap-get-header] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84) -- 0x55dce1afdae0 con 0x7f9dd0022d70 2024-08-03T20:54:38.699 INFO:teuthology.orchestra.run.smithi032.stdout:���� �� 2024-08-03T20:54:38.700 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.673+0000 7f9dd7400640 1 -- 172.21.15.32:0/648633679 <== osd.6 v2:172.21.15.118:6825/801947386 19 ==== osd_op_reply(115 400.00000000 [omap-get-header] v0'0 uv1 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f9dc804f040 con 0x7f9dd0022d70 2024-08-03T20:54:38.700 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.673+0000 7f9dde35ef40 1 -- 172.21.15.32:0/648633679 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:116 12.f 12:f1f69726:::400.00000000:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84) -- 0x55dce1afdae0 con 0x7f9dd0022d70 2024-08-03T20:54:38.700 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.673+0000 7f9dd7400640 1 -- 172.21.15.32:0/648633679 <== osd.6 v2:172.21.15.118:6825/801947386 20 ==== osd_op_reply(116 400.00000000 [omap-get-vals out=5b] v0'0 uv1 ondisk = 0) ==== 156+0+5 (crc 0 0 0) 0x7f9dc804f040 con 0x7f9dd0022d70 2024-08-03T20:54:38.700 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.674+0000 7f9dde35ef40 1 -- 172.21.15.32:0/648633679 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:117 12.1f 12:fe07bba1:::100.00000000.inode:head [read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84) -- 0x55dce1afdae0 con 0x7f9dd0022d70 2024-08-03T20:54:38.701 INFO:teuthology.orchestra.run.smithi032.stdout:���� �� ��0*3100.00000000.inode�������������������!�������������������������� ��`ZFFceph fs volume v011)���f��=@A������������������f��=( 2024-08-03T20:54:38.701 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.674+0000 7f9dd7400640 1 -- 172.21.15.32:0/648633679 <== osd.6 v2:172.21.15.118:6825/801947386 21 ==== osd_op_reply(117 100.00000000.inode [read 0~582 out=582b] v0'0 uv3 ondisk = 0) ==== 162+0+582 (crc 0 0 0) 0x7f9dc804f040 con 0x7f9dd0022d70 2024-08-03T20:54:38.701 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.674+0000 7f9dde35ef40 1 -- 172.21.15.32:0/648633679 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:118 12.1f 12:fe07bba1:::100.00000000.inode:head [getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84) -- 0x55dce1afdae0 con 0x7f9dd0022d70 2024-08-03T20:54:38.702 INFO:teuthology.orchestra.run.smithi032.stdout:8 ��f���8 ��f�����f��=����NH������������ �� 2024-08-03T20:54:38.702 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.675+0000 7f9dd7400640 1 -- 172.21.15.32:0/648633679 <== osd.6 v2:172.21.15.118:6825/801947386 22 ==== osd_op_reply(118 100.00000000.inode [getxattrs out=4b] v0'0 uv3 ondisk = 0) ==== 162+0+4 (crc 0 0 0) 0x7f9dc804f040 con 0x7f9dd0022d70 2024-08-03T20:54:38.702 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.675+0000 7f9dde35ef40 1 -- 172.21.15.32:0/648633679 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:119 12.1f 12:fe07bba1:::100.00000000.inode:head [omap-get-header] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84) -- 0x55dce1afdae0 con 0x7f9dd0022d70 2024-08-03T20:54:38.702 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.676+0000 7f9dd7400640 1 -- 172.21.15.32:0/648633679 <== osd.6 v2:172.21.15.118:6825/801947386 23 ==== osd_op_reply(119 100.00000000.inode [omap-get-header] v0'0 uv3 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7f9dc804f040 con 0x7f9dd0022d70 2024-08-03T20:54:38.702 INFO:teuthology.orchestra.run.smithi032.stdout:���� �� 2024-08-03T20:54:38.703 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.676+0000 7f9dde35ef40 1 -- 172.21.15.32:0/648633679 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:120 12.1f 12:fe07bba1:::100.00000000.inode:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84) -- 0x55dce1afdae0 con 0x7f9dd0022d70 2024-08-03T20:54:38.703 INFO:teuthology.orchestra.run.smithi032.stdout:���� �� ��("+ 2024-08-03T20:54:38.704 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.676+0000 7f9dd7400640 1 -- 172.21.15.32:0/648633679 <== osd.6 v2:172.21.15.118:6825/801947386 24 ==== osd_op_reply(120 100.00000000.inode [omap-get-vals out=5b] v0'0 uv3 ondisk = 0) ==== 162+0+5 (crc 0 0 0) 0x7f9dc804f040 con 0x7f9dd0022d70 2024-08-03T20:54:38.704 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.676+0000 7f9dde35ef40 1 -- 172.21.15.32:0/648633679 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:121 12.1f 12:ff5b34d6:::1.00000000:head [read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84) -- 0x55dce1afdae0 con 0x7f9dd0022d70 2024-08-03T20:54:38.704 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.677+0000 7f9dd7400640 1 -- 172.21.15.32:0/648633679 <== osd.6 v2:172.21.15.118:6825/801947386 25 ==== osd_op_reply(121 1.00000000 [read 0~0] v0'0 uv6 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7f9dc804f040 con 0x7f9dd0022d70 2024-08-03T20:54:38.704 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.677+0000 7f9dde35ef40 1 -- 172.21.15.32:0/648633679 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:122 12.1f 12:ff5b34d6:::1.00000000:head [getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84) -- 0x55dce1afdae0 con 0x7f9dd0022d70 2024-08-03T20:54:38.704 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.678+0000 7f9dd7400640 1 -- 172.21.15.32:0/648633679 <== osd.6 v2:172.21.15.118:6825/801947386 26 ==== osd_op_reply(122 1.00000000 [getxattrs out=92b] v0'0 uv6 ondisk = 0) ==== 154+0+92 (crc 0 0 0) 0x7f9dc804f040 con 0x7f9dd0022d70 2024-08-03T20:54:38.705 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.678+0000 7f9dde35ef40 1 -- 172.21.15.32:0/648633679 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:123 12.1f 12:ff5b34d6:::1.00000000:head [omap-get-header] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84) -- 0x55dce1afdae0 con 0x7f9dd0022d70 2024-08-03T20:54:38.705 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.679+0000 7f9dd7400640 1 -- 172.21.15.32:0/648633679 <== osd.6 v2:172.21.15.118:6825/801947386 27 ==== osd_op_reply(123 1.00000000 [omap-get-header out=274b] v0'0 uv6 ondisk = 0) ==== 154+0+274 (crc 0 0 0) 0x7f9dc804f040 con 0x7f9dd0022d70 2024-08-03T20:54:38.705 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.679+0000 7f9dde35ef40 1 -- 172.21.15.32:0/648633679 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:124 12.1f 12:ff5b34d6:::1.00000000:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84) -- 0x55dce1afdae0 con 0x7f9dd0022d70 2024-08-03T20:54:38.705 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.680+0000 7f9dd7400640 1 -- 172.21.15.32:0/648633679 <== osd.6 v2:172.21.15.118:6825/801947386 28 ==== osd_op_reply(124 1.00000000 [omap-get-vals out=585b] v0'0 uv6 ondisk = 0) ==== 154+0+585 (crc 0 0 0) 0x7f9dc804f040 con 0x7f9dd0022d70 2024-08-03T20:54:38.708 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.680+0000 7f9dde35ef40 1 -- 172.21.15.32:0/648633679 >> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] conn(0x55dce19caf50 msgr2=0x7f9dc4008450 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:38.708 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.680+0000 7f9dde35ef40 1 --2- 172.21.15.32:0/648633679 >> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] conn(0x55dce19caf50 0x7f9dc4008450 crc :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:38.708 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.680+0000 7f9dde35ef40 1 -- 172.21.15.32:0/648633679 >> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] conn(0x7f9dc4024400 msgr2=0x7f9dc4024810 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:38.708 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.680+0000 7f9dde35ef40 1 --2- 172.21.15.32:0/648633679 >> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] conn(0x7f9dc4024400 0x7f9dc4024810 crc :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:38.708 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.680+0000 7f9dde35ef40 1 -- 172.21.15.32:0/648633679 >> [v2:172.21.15.32:6808/3806708944,v1:172.21.15.32:6809/3806708944] conn(0x7f9dc40153d0 msgr2=0x7f9dc40157e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:38.708 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.680+0000 7f9dde35ef40 1 --2- 172.21.15.32:0/648633679 >> [v2:172.21.15.32:6808/3806708944,v1:172.21.15.32:6809/3806708944] conn(0x7f9dc40153d0 0x7f9dc40157e0 crc :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:38.708 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.680+0000 7f9dde35ef40 1 -- 172.21.15.32:0/648633679 >> [v2:172.21.15.32:6800/9088303,v1:172.21.15.32:6801/9088303] conn(0x7f9dc8008950 msgr2=0x7f9dc8016410 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:38.708 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.680+0000 7f9dde35ef40 1 --2- 172.21.15.32:0/648633679 >> [v2:172.21.15.32:6800/9088303,v1:172.21.15.32:6801/9088303] conn(0x7f9dc8008950 0x7f9dc8016410 crc :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:38.708 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.680+0000 7f9dde35ef40 1 -- 172.21.15.32:0/648633679 >> [v2:172.21.15.118:6802/864752275,v1:172.21.15.118:6803/864752275] conn(0x55dce19810c0 msgr2=0x55dce19814d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:38.708 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.680+0000 7f9dde35ef40 1 --2- 172.21.15.32:0/648633679 >> [v2:172.21.15.118:6802/864752275,v1:172.21.15.118:6803/864752275] conn(0x55dce19810c0 0x55dce19814d0 crc :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:38.708 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.680+0000 7f9dde35ef40 1 -- 172.21.15.32:0/648633679 >> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] conn(0x7f9dc8028040 msgr2=0x7f9dc8028960 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:38.708 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.680+0000 7f9dde35ef40 1 --2- 172.21.15.32:0/648633679 >> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] conn(0x7f9dc8028040 0x7f9dc8028960 crc :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:38.708 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.680+0000 7f9dde35ef40 1 -- 172.21.15.32:0/648633679 >> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] conn(0x7f9dd0022d70 msgr2=0x7f9dd00a3360 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:38.709 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.680+0000 7f9dde35ef40 1 --2- 172.21.15.32:0/648633679 >> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] conn(0x7f9dd0022d70 0x7f9dd00a3360 crc :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:38.709 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.680+0000 7f9dde35ef40 1 -- 172.21.15.32:0/648633679 >> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] conn(0x7f9dc8034a80 msgr2=0x7f9dc80093c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:38.709 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.680+0000 7f9dde35ef40 1 --2- 172.21.15.32:0/648633679 >> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] conn(0x7f9dc8034a80 0x7f9dc80093c0 crc :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:38.709 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.680+0000 7f9dd7e00640 1 -- 172.21.15.32:0/648633679 reap_dead start 2024-08-03T20:54:38.709 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.680+0000 7f9dd7e00640 1 -- 172.21.15.32:0/648633679 reap_dead start 2024-08-03T20:54:38.709 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.680+0000 7f9dd7e00640 1 -- 172.21.15.32:0/648633679 reap_dead start 2024-08-03T20:54:38.709 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.681+0000 7f9dde35ef40 1 -- 172.21.15.32:0/648633679 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f9db8075a20 msgr2=0x7f9db8077f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:38.709 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.681+0000 7f9dde35ef40 1 --2- 172.21.15.32:0/648633679 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f9db8075a20 0x7f9db8077f30 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f9dc800cab0 tx=0x7f9dc8004880 comp rx=0 tx=0).stop 2024-08-03T20:54:38.709 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.681+0000 7f9dde35ef40 1 -- 172.21.15.32:0/648633679 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55dce1a4d7b0 msgr2=0x55dce19aa110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:38.709 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.681+0000 7f9dde35ef40 1 --2- 172.21.15.32:0/648633679 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55dce1a4d7b0 0x55dce19aa110 secure :-1 s=READY pgs=939 cs=0 l=1 rev1=1 crypto rx=0x7f9dd0057040 tx=0x7f9dd00049c0 comp rx=0 tx=0).stop 2024-08-03T20:54:38.709 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.681+0000 7f9dde35ef40 1 -- 172.21.15.32:0/648633679 shutdown_connections 2024-08-03T20:54:38.709 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.681+0000 7f9dde35ef40 1 --2- 172.21.15.32:0/648633679 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f9db8075a20 0x7f9db8077f30 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:38.709 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.681+0000 7f9dde35ef40 1 --2- 172.21.15.32:0/648633679 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55dce1a4d7b0 0x55dce19aa110 unknown :-1 s=CLOSED pgs=939 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:38.709 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.681+0000 7f9dde35ef40 1 -- 172.21.15.32:0/648633679 >> 172.21.15.32:0/648633679 conn(0x55dce18a1ac0 msgr2=0x55dce19c86f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:54:38.710 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.681+0000 7f9dde35ef40 1 -- 172.21.15.32:0/648633679 shutdown_connections 2024-08-03T20:54:38.710 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.681+0000 7f9dde35ef40 1 -- 172.21.15.32:0/648633679 wait complete. 2024-08-03T20:54:38.710 INFO:teuthology.orchestra.run.smithi032.stdout:1.00000000�������������������!�������������������������� ��d^_layout@@ _parent ���� ��  ( ��f��8( ��f��88` ��fh��8` ��fh������ ��NH subdir_head1i"� ��f��A�$���������������� ��f� ��f��8( ��f�8` ��fh��8` ��fh�� �������� ��f��8����security.selinux%unconfined_u:object_r:unlabeled_t:s0������������ �� �� 2024-08-03T20:54:38.710 DEBUG:teuthology.orchestra.run.smithi032:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata ls 2024-08-03T20:54:38.854 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.853+0000 7f5609c77f40 1 Processor -- start 2024-08-03T20:54:38.854 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.853+0000 7f5609c77f40 1 -- start start 2024-08-03T20:54:38.856 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.854+0000 7f5609c77f40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55b99c2f5d80 0x55b99c3cf320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:38.856 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.854+0000 7f5609c77f40 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55b99c33b2d0 0x55b99c3cf860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:38.856 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.854+0000 7f5609c77f40 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b99c3d0e10 0x55b99c45c4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:38.857 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.854+0000 7f5609c77f40 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55b99c2f6df0 con 0x55b99c3d0e10 2024-08-03T20:54:38.857 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.854+0000 7f5609c77f40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x55b99c245e30 con 0x55b99c2f5d80 2024-08-03T20:54:38.857 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.854+0000 7f5609c77f40 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x55b99c2f75f0 con 0x55b99c33b2d0 2024-08-03T20:54:38.857 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.854+0000 7f5603800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b99c3d0e10 0x55b99c45c4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:38.857 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.854+0000 7f5603800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b99c3d0e10 0x55b99c45c4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:40196/0 (socket says 172.21.15.32:40196) 2024-08-03T20:54:38.857 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.854+0000 7f5603800640 1 -- 172.21.15.32:0/2455726143 learned_addr learned my addr 172.21.15.32:0/2455726143 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:54:38.857 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.854+0000 7f5602e00640 1 --2- 172.21.15.32:0/2455726143 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55b99c2f5d80 0x55b99c3cf320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:38.857 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.854+0000 7f5602400640 1 --2- 172.21.15.32:0/2455726143 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55b99c33b2d0 0x55b99c3cf860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:38.857 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.855+0000 7f5603800640 1 -- 172.21.15.32:0/2455726143 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55b99c33b2d0 msgr2=0x55b99c3cf860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:54:38.857 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.855+0000 7f5603800640 1 --2- 172.21.15.32:0/2455726143 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55b99c33b2d0 0x55b99c3cf860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:38.857 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.855+0000 7f5603800640 1 -- 172.21.15.32:0/2455726143 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55b99c2f5d80 msgr2=0x55b99c3cf320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:54:38.857 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.855+0000 7f5603800640 1 --2- 172.21.15.32:0/2455726143 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55b99c2f5d80 0x55b99c3cf320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:38.858 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.855+0000 7f5603800640 1 -- 172.21.15.32:0/2455726143 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b99c21ef10 con 0x55b99c3d0e10 2024-08-03T20:54:38.858 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.855+0000 7f5602400640 1 --2- 172.21.15.32:0/2455726143 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55b99c33b2d0 0x55b99c3cf860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-03T20:54:38.858 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.855+0000 7f5603800640 1 --2- 172.21.15.32:0/2455726143 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b99c3d0e10 0x55b99c45c4f0 secure :-1 s=READY pgs=941 cs=0 l=1 rev1=1 crypto rx=0x7f55fc025d30 tx=0x7f55fc003330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bff89f163749521c server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:54:38.859 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.855+0000 7f5601000640 1 -- 172.21.15.32:0/2455726143 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55fc064030 con 0x55b99c3d0e10 2024-08-03T20:54:38.859 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.855+0000 7f5601000640 1 -- 172.21.15.32:0/2455726143 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f55fc004020 con 0x55b99c3d0e10 2024-08-03T20:54:38.859 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.855+0000 7f5601000640 1 -- 172.21.15.32:0/2455726143 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55fc004270 con 0x55b99c3d0e10 2024-08-03T20:54:38.859 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.856+0000 7f5609c77f40 1 -- 172.21.15.32:0/2455726143 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b99c3d0e10 msgr2=0x55b99c45c4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:38.859 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.856+0000 7f5609c77f40 1 --2- 172.21.15.32:0/2455726143 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b99c3d0e10 0x55b99c45c4f0 secure :-1 s=READY pgs=941 cs=0 l=1 rev1=1 crypto rx=0x7f55fc025d30 tx=0x7f55fc003330 comp rx=0 tx=0).stop 2024-08-03T20:54:38.860 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.856+0000 7f5609c77f40 1 -- 172.21.15.32:0/2455726143 shutdown_connections 2024-08-03T20:54:38.860 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.856+0000 7f5609c77f40 1 --2- 172.21.15.32:0/2455726143 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b99c3d0e10 0x55b99c45c4f0 unknown :-1 s=CLOSED pgs=941 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:38.860 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.856+0000 7f5609c77f40 1 --2- 172.21.15.32:0/2455726143 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55b99c33b2d0 0x55b99c3cf860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:38.860 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.856+0000 7f5609c77f40 1 --2- 172.21.15.32:0/2455726143 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55b99c2f5d80 0x55b99c3cf320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:38.860 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.856+0000 7f5609c77f40 1 -- 172.21.15.32:0/2455726143 >> 172.21.15.32:0/2455726143 conn(0x55b99c217ac0 msgr2=0x55b99c33a0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:54:38.860 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.856+0000 7f5609c77f40 1 -- 172.21.15.32:0/2455726143 shutdown_connections 2024-08-03T20:54:38.860 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.856+0000 7f5609c77f40 1 -- 172.21.15.32:0/2455726143 wait complete. 2024-08-03T20:54:38.860 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.857+0000 7f5609c77f40 1 Processor -- start 2024-08-03T20:54:38.860 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.857+0000 7f5609c77f40 1 -- start start 2024-08-03T20:54:38.862 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.859+0000 7f5609c77f40 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b99c2f5d80 0x55b99c3429d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:38.862 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.859+0000 7f5609c77f40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55b99c33b2d0 0x55b99c3c5540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:38.862 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.859+0000 7f5609c77f40 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55b99c3d0e10 0x55b99c3401f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:38.862 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.859+0000 7f5609c77f40 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55b99c3389b0 con 0x55b99c2f5d80 2024-08-03T20:54:38.862 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.859+0000 7f5609c77f40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x55b99c338f70 con 0x55b99c33b2d0 2024-08-03T20:54:38.862 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.859+0000 7f5609c77f40 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x55b99c3b5d80 con 0x55b99c3d0e10 2024-08-03T20:54:38.862 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.859+0000 7f5602e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b99c2f5d80 0x55b99c3429d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:38.863 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.859+0000 7f5602e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b99c2f5d80 0x55b99c3429d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:40210/0 (socket says 172.21.15.32:40210) 2024-08-03T20:54:38.863 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.859+0000 7f5602e00640 1 -- 172.21.15.32:0/4255060924 learned_addr learned my addr 172.21.15.32:0/4255060924 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:54:38.863 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.859+0000 7f5602e00640 1 -- 172.21.15.32:0/4255060924 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55b99c3d0e10 msgr2=0x55b99c3401f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:38.863 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.859+0000 7f5603800640 1 --2- 172.21.15.32:0/4255060924 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55b99c3d0e10 0x55b99c3401f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:38.863 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.859+0000 7f5602400640 1 --2- 172.21.15.32:0/4255060924 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55b99c33b2d0 0x55b99c3c5540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:38.863 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.859+0000 7f5602e00640 1 --2- 172.21.15.32:0/4255060924 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55b99c3d0e10 0x55b99c3401f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:38.863 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.859+0000 7f5602e00640 1 -- 172.21.15.32:0/4255060924 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55b99c33b2d0 msgr2=0x55b99c3c5540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:38.863 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.859+0000 7f5602e00640 1 --2- 172.21.15.32:0/4255060924 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55b99c33b2d0 0x55b99c3c5540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:38.863 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.859+0000 7f5602e00640 1 -- 172.21.15.32:0/4255060924 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b99c3b4540 con 0x55b99c2f5d80 2024-08-03T20:54:38.863 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.860+0000 7f5603800640 1 --2- 172.21.15.32:0/4255060924 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55b99c3d0e10 0x55b99c3401f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-03T20:54:38.863 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.860+0000 7f5602400640 1 --2- 172.21.15.32:0/4255060924 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55b99c33b2d0 0x55b99c3c5540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-03T20:54:38.863 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.860+0000 7f5602e00640 1 --2- 172.21.15.32:0/4255060924 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b99c2f5d80 0x55b99c3429d0 secure :-1 s=READY pgs=942 cs=0 l=1 rev1=1 crypto rx=0x7f55f40086c0 tx=0x7f55f40087b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:54:38.864 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.860+0000 7f55fb400640 1 -- 172.21.15.32:0/4255060924 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55f400a850 con 0x55b99c2f5d80 2024-08-03T20:54:38.864 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.860+0000 7f55fb400640 1 -- 172.21.15.32:0/4255060924 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f55f400a9f0 con 0x55b99c2f5d80 2024-08-03T20:54:38.864 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.860+0000 7f55fb400640 1 -- 172.21.15.32:0/4255060924 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55f400ae00 con 0x55b99c2f5d80 2024-08-03T20:54:38.864 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.860+0000 7f5609c77f40 1 -- 172.21.15.32:0/4255060924 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x55b99c467d10 con 0x55b99c2f5d80 2024-08-03T20:54:38.864 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.860+0000 7f5609c77f40 1 -- 172.21.15.32:0/4255060924 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b99c4682d0 con 0x55b99c2f5d80 2024-08-03T20:54:38.864 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.862+0000 7f55fb400640 1 -- 172.21.15.32:0/4255060924 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f55f4016430 con 0x55b99c2f5d80 2024-08-03T20:54:38.865 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.863+0000 7f55fb400640 1 --2- 172.21.15.32:0/4255060924 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f55e4075a20 0x7f55e4077f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:38.865 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.863+0000 7f55fb400640 1 -- 172.21.15.32:0/4255060924 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 5795+0+0 (secure 0 0 0) 0x7f55f4097df0 con 0x55b99c2f5d80 2024-08-03T20:54:38.865 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.864+0000 7f5609c77f40 1 --2- 172.21.15.32:0/4255060924 >> [v2:172.21.15.118:6802/864752275,v1:172.21.15.118:6803/864752275] conn(0x55b99c3c5a80 0x55b99c3b4f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:38.865 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.864+0000 7f5609c77f40 1 -- 172.21.15.32:0/4255060924 --> [v2:172.21.15.118:6802/864752275,v1:172.21.15.118:6803/864752275] -- osd_op(unknown.0.0:1 12.0 12:00000000::::head [pgnls start_epoch 84 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e84) -- 0x55b99c3b5490 con 0x55b99c3c5a80 2024-08-03T20:54:38.865 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.864+0000 7f5602400640 1 --2- 172.21.15.32:0/4255060924 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f55e4075a20 0x7f55e4077f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:38.866 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.864+0000 7f5603800640 1 --2- 172.21.15.32:0/4255060924 >> [v2:172.21.15.118:6802/864752275,v1:172.21.15.118:6803/864752275] conn(0x55b99c3c5a80 0x55b99c3b4f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:38.866 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.865+0000 7f5602400640 1 --2- 172.21.15.32:0/4255060924 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f55e4075a20 0x7f55e4077f30 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f55f0004230 tx=0x7f55f0002040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:54:38.866 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.865+0000 7f5603800640 1 --2- 172.21.15.32:0/4255060924 >> [v2:172.21.15.118:6802/864752275,v1:172.21.15.118:6803/864752275] conn(0x55b99c3c5a80 0x55b99c3b4f50 crc :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:54:38.867 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.865+0000 7f5603800640 1 -- 172.21.15.32:0/4255060924 <== osd.4 v2:172.21.15.118:6802/864752275 1 ==== osd_op_reply(1 [pgnls start_epoch 84 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f55fc00b040 con 0x55b99c3c5a80 2024-08-03T20:54:38.867 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.865+0000 7f5603800640 1 --2- 172.21.15.32:0/4255060924 >> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] conn(0x7f55fc06a040 0x7f55fc003410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:38.867 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.865+0000 7f5603800640 1 -- 172.21.15.32:0/4255060924 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:2 12.10 12:08000000::::head [pgnls start_epoch 84 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e84) -- 0x55b99c3b5490 con 0x7f55fc06a040 2024-08-03T20:54:38.867 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.866+0000 7f5602e00640 1 --2- 172.21.15.32:0/4255060924 >> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] conn(0x7f55fc06a040 0x7f55fc003410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:38.867 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.866+0000 7f5602e00640 1 --2- 172.21.15.32:0/4255060924 >> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] conn(0x7f55fc06a040 0x7f55fc003410 crc :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:54:38.868 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.867+0000 7f5602e00640 1 -- 172.21.15.32:0/4255060924 <== osd.0 v2:172.21.15.32:6816/3066156116 1 ==== osd_op_reply(2 [pgnls start_epoch 84 out=177b] v83'10 uv10 ondisk = 1) ==== 144+0+177 (crc 0 0 0) 0x7f55f405cba0 con 0x7f55fc06a040 2024-08-03T20:54:38.868 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.867+0000 7f5602e00640 1 --2- 172.21.15.32:0/4255060924 >> [v2:172.21.15.32:6800/9088303,v1:172.21.15.32:6801/9088303] conn(0x7f55f4016020 0x7f55f405f5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:38.868 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.867+0000 7f5602e00640 1 -- 172.21.15.32:0/4255060924 --> [v2:172.21.15.32:6800/9088303,v1:172.21.15.32:6801/9088303] -- osd_op(unknown.0.0:3 12.8 12:10000000::::head [pgnls start_epoch 84 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e84) -- 0x55b99c3b5490 con 0x7f55f4016020 2024-08-03T20:54:38.868 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.867+0000 7f5602400640 1 --2- 172.21.15.32:0/4255060924 >> [v2:172.21.15.32:6800/9088303,v1:172.21.15.32:6801/9088303] conn(0x7f55f4016020 0x7f55f405f5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:38.869 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.867+0000 7f5602400640 1 --2- 172.21.15.32:0/4255060924 >> [v2:172.21.15.32:6800/9088303,v1:172.21.15.32:6801/9088303] conn(0x7f55f4016020 0x7f55f405f5e0 crc :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:54:38.869 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.868+0000 7f5602400640 1 -- 172.21.15.32:0/4255060924 <== osd.3 v2:172.21.15.32:6800/9088303 1 ==== osd_op_reply(3 [pgnls start_epoch 84 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f55f000a970 con 0x7f55f4016020 2024-08-03T20:54:38.869 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.868+0000 7f5602400640 1 --2- 172.21.15.32:0/4255060924 >> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] conn(0x7f55f0008420 0x7f55f0008830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:38.869 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.868+0000 7f5602400640 1 -- 172.21.15.32:0/4255060924 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:4 12.18 12:18000000::::head [pgnls start_epoch 84 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e84) -- 0x55b99c3b5490 con 0x7f55f0008420 2024-08-03T20:54:38.870 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.868+0000 7f5603800640 1 --2- 172.21.15.32:0/4255060924 >> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] conn(0x7f55f0008420 0x7f55f0008830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:38.870 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.869+0000 7f5603800640 1 --2- 172.21.15.32:0/4255060924 >> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] conn(0x7f55f0008420 0x7f55f0008830 crc :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:54:38.870 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.869+0000 7f5603800640 1 -- 172.21.15.32:0/4255060924 <== osd.1 v2:172.21.15.32:6824/1551241759 1 ==== osd_op_reply(4 [pgnls start_epoch 84 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f55fc00b040 con 0x7f55f0008420 2024-08-03T20:54:38.870 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.869+0000 7f5603800640 1 --2- 172.21.15.32:0/4255060924 >> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] conn(0x7f55fc022cf0 0x7f55fc06c460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:38.871 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.869+0000 7f5603800640 1 -- 172.21.15.32:0/4255060924 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:5 12.4 12:20000000::::head [pgnls start_epoch 84 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e84) -- 0x55b99c3b5490 con 0x7f55fc022cf0 2024-08-03T20:54:38.871 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.869+0000 7f5602e00640 1 --2- 172.21.15.32:0/4255060924 >> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] conn(0x7f55fc022cf0 0x7f55fc06c460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:38.871 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.870+0000 7f5602e00640 1 --2- 172.21.15.32:0/4255060924 >> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] conn(0x7f55fc022cf0 0x7f55fc06c460 crc :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:54:38.872 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.870+0000 7f5602e00640 1 -- 172.21.15.32:0/4255060924 <== osd.6 v2:172.21.15.118:6825/801947386 1 ==== osd_op_reply(5 [pgnls start_epoch 84 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f55f405cba0 con 0x7f55fc022cf0 2024-08-03T20:54:38.872 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.870+0000 7f5602e00640 1 --2- 172.21.15.32:0/4255060924 >> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] conn(0x7f55f4060b70 0x7f55f4016720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:38.872 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.870+0000 7f5602e00640 1 -- 172.21.15.32:0/4255060924 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:6 12.14 12:28000000::::head [pgnls start_epoch 84 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e84) -- 0x55b99c3b5490 con 0x7f55f4060b70 2024-08-03T20:54:38.872 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.871+0000 7f5602400640 1 --2- 172.21.15.32:0/4255060924 >> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] conn(0x7f55f4060b70 0x7f55f4016720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:38.873 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.871+0000 7f5602400640 1 --2- 172.21.15.32:0/4255060924 >> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] conn(0x7f55f4060b70 0x7f55f4016720 crc :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:54:38.873 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.872+0000 7f5602400640 1 -- 172.21.15.32:0/4255060924 <== osd.5 v2:172.21.15.118:6810/2724891770 1 ==== osd_op_reply(6 [pgnls start_epoch 84 out=149b] v83'19 uv19 ondisk = 1) ==== 144+0+149 (crc 0 0 0) 0x7f55f000a970 con 0x7f55f4060b70 2024-08-03T20:54:38.873 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.872+0000 7f5602400640 1 -- 172.21.15.32:0/4255060924 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:7 12.c 12:30000000::::head [pgnls start_epoch 84 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e84) -- 0x55b99c3b5490 con 0x7f55fc06a040 2024-08-03T20:54:38.874 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.872+0000 7f5602e00640 1 -- 172.21.15.32:0/4255060924 <== osd.0 v2:172.21.15.32:6816/3066156116 2 ==== osd_op_reply(7 [pgnls start_epoch 84 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f55f405cba0 con 0x7f55fc06a040 2024-08-03T20:54:38.874 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.872+0000 7f5602e00640 1 -- 172.21.15.32:0/4255060924 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:8 12.1c 12:38000000::::head [pgnls start_epoch 84 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e84) -- 0x55b99c3b5490 con 0x7f55fc06a040 2024-08-03T20:54:38.874 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.873+0000 7f5602e00640 1 -- 172.21.15.32:0/4255060924 <== osd.0 v2:172.21.15.32:6816/3066156116 3 ==== osd_op_reply(8 [pgnls start_epoch 84 out=73b] v82'2 uv2 ondisk = 1) ==== 144+0+73 (crc 0 0 0) 0x7f55f405cba0 con 0x7f55fc06a040 2024-08-03T20:54:38.874 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.873+0000 7f5602e00640 1 -- 172.21.15.32:0/4255060924 --> [v2:172.21.15.118:6802/864752275,v1:172.21.15.118:6803/864752275] -- osd_op(unknown.0.0:9 12.2 12:40000000::::head [pgnls start_epoch 84 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e84) -- 0x55b99c3b5490 con 0x55b99c3c5a80 2024-08-03T20:54:38.875 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.873+0000 7f5603800640 1 -- 172.21.15.32:0/4255060924 <== osd.4 v2:172.21.15.118:6802/864752275 2 ==== osd_op_reply(9 [pgnls start_epoch 84 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f55fc00b040 con 0x55b99c3c5a80 2024-08-03T20:54:38.875 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.873+0000 7f5603800640 1 -- 172.21.15.32:0/4255060924 --> [v2:172.21.15.32:6800/9088303,v1:172.21.15.32:6801/9088303] -- osd_op(unknown.0.0:10 12.12 12:48000000::::head [pgnls start_epoch 84 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e84) -- 0x55b99c3b5490 con 0x7f55f4016020 2024-08-03T20:54:38.875 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.873+0000 7f5602400640 1 -- 172.21.15.32:0/4255060924 <== osd.3 v2:172.21.15.32:6800/9088303 2 ==== osd_op_reply(10 [pgnls start_epoch 84 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f55f000a970 con 0x7f55f4016020 2024-08-03T20:54:38.875 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.873+0000 7f5602400640 1 --2- 172.21.15.32:0/4255060924 >> [v2:172.21.15.32:6808/3806708944,v1:172.21.15.32:6809/3806708944] conn(0x7f55f0028970 0x7f55f0018400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:38.875 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.873+0000 7f5602400640 1 -- 172.21.15.32:0/4255060924 --> [v2:172.21.15.32:6808/3806708944,v1:172.21.15.32:6809/3806708944] -- osd_op(unknown.0.0:11 12.a 12:50000000::::head [pgnls start_epoch 84 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e84) -- 0x55b99c3b5490 con 0x7f55f0028970 2024-08-03T20:54:38.875 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.874+0000 7f5603800640 1 --2- 172.21.15.32:0/4255060924 >> [v2:172.21.15.32:6808/3806708944,v1:172.21.15.32:6809/3806708944] conn(0x7f55f0028970 0x7f55f0018400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:38.877 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.874+0000 7f5603800640 1 --2- 172.21.15.32:0/4255060924 >> [v2:172.21.15.32:6808/3806708944,v1:172.21.15.32:6809/3806708944] conn(0x7f55f0028970 0x7f55f0018400 crc :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:54:38.877 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.875+0000 7f5603800640 1 -- 172.21.15.32:0/4255060924 <== osd.2 v2:172.21.15.32:6808/3806708944 1 ==== osd_op_reply(11 [pgnls start_epoch 84 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f55fc00b040 con 0x7f55f0028970 2024-08-03T20:54:38.877 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.875+0000 7f5603800640 1 -- 172.21.15.32:0/4255060924 --> [v2:172.21.15.118:6802/864752275,v1:172.21.15.118:6803/864752275] -- osd_op(unknown.0.0:12 12.1a 12:58000000::::head [pgnls start_epoch 84 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e84) -- 0x55b99c3b5490 con 0x55b99c3c5a80 2024-08-03T20:54:38.877 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.875+0000 7f5603800640 1 -- 172.21.15.32:0/4255060924 <== osd.4 v2:172.21.15.118:6802/864752275 3 ==== osd_op_reply(12 [pgnls start_epoch 84 out=49b] v82'1 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f55fc00b040 con 0x55b99c3c5a80 2024-08-03T20:54:38.877 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.875+0000 7f5603800640 1 -- 172.21.15.32:0/4255060924 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:13 12.6 12:60000000::::head [pgnls start_epoch 84 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e84) -- 0x55b99c3b5490 con 0x7f55f0008420 2024-08-03T20:54:38.877 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.875+0000 7f5603800640 1 -- 172.21.15.32:0/4255060924 <== osd.1 v2:172.21.15.32:6824/1551241759 2 ==== osd_op_reply(13 [pgnls start_epoch 84 out=77b] v83'4 uv4 ondisk = 1) ==== 144+0+77 (crc 0 0 0) 0x7f55fc00b040 con 0x7f55f0008420 2024-08-03T20:54:38.877 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.875+0000 7f5603800640 1 -- 172.21.15.32:0/4255060924 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:14 12.16 12:68000000::::head [pgnls start_epoch 84 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e84) -- 0x55b99c3b5490 con 0x7f55fc022cf0 2024-08-03T20:54:38.878 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.876+0000 7f5602e00640 1 -- 172.21.15.32:0/4255060924 <== osd.6 v2:172.21.15.118:6825/801947386 2 ==== osd_op_reply(14 [pgnls start_epoch 84 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f55f405cba0 con 0x7f55fc022cf0 2024-08-03T20:54:38.878 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.876+0000 7f5602e00640 1 -- 172.21.15.32:0/4255060924 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:15 12.e 12:70000000::::head [pgnls start_epoch 84 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e84) -- 0x55b99c3b5490 con 0x7f55fc06a040 2024-08-03T20:54:38.878 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.876+0000 7f5602e00640 1 -- 172.21.15.32:0/4255060924 <== osd.0 v2:172.21.15.32:6816/3066156116 4 ==== osd_op_reply(15 [pgnls start_epoch 84 out=73b] v82'2 uv2 ondisk = 1) ==== 144+0+73 (crc 0 0 0) 0x7f55f405cba0 con 0x7f55fc06a040 2024-08-03T20:54:38.878 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.876+0000 7f5602e00640 1 -- 172.21.15.32:0/4255060924 --> [v2:172.21.15.118:6802/864752275,v1:172.21.15.118:6803/864752275] -- osd_op(unknown.0.0:16 12.1e 12:78000000::::head [pgnls start_epoch 84 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e84) -- 0x55b99c3b5490 con 0x55b99c3c5a80 2024-08-03T20:54:38.879 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.877+0000 7f5603800640 1 -- 172.21.15.32:0/4255060924 <== osd.4 v2:172.21.15.118:6802/864752275 4 ==== osd_op_reply(16 [pgnls start_epoch 84 out=49b] v82'1 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f55fc00b040 con 0x55b99c3c5a80 2024-08-03T20:54:38.879 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.877+0000 7f5603800640 1 -- 172.21.15.32:0/4255060924 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:17 12.1 12:80000000::::head [pgnls start_epoch 84 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e84) -- 0x55b99c3b5490 con 0x7f55fc022cf0 2024-08-03T20:54:38.879 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.878+0000 7f5602e00640 1 -- 172.21.15.32:0/4255060924 <== osd.6 v2:172.21.15.118:6825/801947386 3 ==== osd_op_reply(17 [pgnls start_epoch 84 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f55f405cba0 con 0x7f55fc022cf0 2024-08-03T20:54:38.879 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.878+0000 7f5602e00640 1 -- 172.21.15.32:0/4255060924 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:18 12.11 12:88000000::::head [pgnls start_epoch 84 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e84) -- 0x55b99c3b5490 con 0x7f55f4060b70 2024-08-03T20:54:38.880 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.878+0000 7f5602400640 1 -- 172.21.15.32:0/4255060924 <== osd.5 v2:172.21.15.118:6810/2724891770 2 ==== osd_op_reply(18 [pgnls start_epoch 84 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f55f000a970 con 0x7f55f4060b70 2024-08-03T20:54:38.880 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.878+0000 7f5602400640 1 -- 172.21.15.32:0/4255060924 --> [v2:172.21.15.32:6808/3806708944,v1:172.21.15.32:6809/3806708944] -- osd_op(unknown.0.0:19 12.9 12:90000000::::head [pgnls start_epoch 84 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e84) -- 0x55b99c3b5490 con 0x7f55f0028970 2024-08-03T20:54:38.881 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.879+0000 7f5603800640 1 -- 172.21.15.32:0/4255060924 <== osd.2 v2:172.21.15.32:6808/3806708944 2 ==== osd_op_reply(19 [pgnls start_epoch 84 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f55fc00b040 con 0x7f55f0028970 2024-08-03T20:54:38.881 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.879+0000 7f5603800640 1 -- 172.21.15.32:0/4255060924 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:20 12.19 12:98000000::::head [pgnls start_epoch 84 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e84) -- 0x55b99c3b5490 con 0x7f55fc06a040 2024-08-03T20:54:38.881 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.879+0000 7f5602e00640 1 -- 172.21.15.32:0/4255060924 <== osd.0 v2:172.21.15.32:6816/3066156116 5 ==== osd_op_reply(20 [pgnls start_epoch 84 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f55f405cba0 con 0x7f55fc06a040 2024-08-03T20:54:38.881 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.879+0000 7f5602e00640 1 -- 172.21.15.32:0/4255060924 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:21 12.5 12:a0000000::::head [pgnls start_epoch 84 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e84) -- 0x55b99c3b5490 con 0x7f55fc022cf0 2024-08-03T20:54:38.882 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.879+0000 7f5602e00640 1 -- 172.21.15.32:0/4255060924 <== osd.6 v2:172.21.15.118:6825/801947386 4 ==== osd_op_reply(21 [pgnls start_epoch 84 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f55f405cba0 con 0x7f55fc022cf0 2024-08-03T20:54:38.882 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.879+0000 7f5602e00640 1 -- 172.21.15.32:0/4255060924 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:22 12.15 12:a8000000::::head [pgnls start_epoch 84 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e84) -- 0x55b99c3b5490 con 0x7f55fc022cf0 2024-08-03T20:54:38.882 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.880+0000 7f5602e00640 1 -- 172.21.15.32:0/4255060924 <== osd.6 v2:172.21.15.118:6825/801947386 5 ==== osd_op_reply(22 [pgnls start_epoch 84 out=97b] v82'3 uv3 ondisk = 1) ==== 144+0+97 (crc 0 0 0) 0x7f55f405cba0 con 0x7f55fc022cf0 2024-08-03T20:54:38.882 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.880+0000 7f5602e00640 1 --2- 172.21.15.32:0/4255060924 >> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] conn(0x7f55f40694a0 0x7f55f40612a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:38.882 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.880+0000 7f5602e00640 1 -- 172.21.15.32:0/4255060924 --> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] -- osd_op(unknown.0.0:23 12.d 12:b0000000::::head [pgnls start_epoch 84 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e84) -- 0x55b99c3b5490 con 0x7f55f40694a0 2024-08-03T20:54:38.882 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.880+0000 7f5602e00640 1 --2- 172.21.15.32:0/4255060924 >> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] conn(0x7f55f40694a0 0x7f55f40612a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:38.882 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.881+0000 7f5602e00640 1 --2- 172.21.15.32:0/4255060924 >> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] conn(0x7f55f40694a0 0x7f55f40612a0 crc :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:54:38.883 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.881+0000 7f5602e00640 1 -- 172.21.15.32:0/4255060924 <== osd.7 v2:172.21.15.118:6818/653000048 1 ==== osd_op_reply(23 [pgnls start_epoch 84 out=74b] v82'1 uv1 ondisk = 1) ==== 144+0+74 (crc 0 0 0) 0x7f55f405cba0 con 0x7f55f40694a0 2024-08-03T20:54:38.883 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.881+0000 7f5602e00640 1 -- 172.21.15.32:0/4255060924 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:24 12.1d 12:b8000000::::head [pgnls start_epoch 84 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e84) -- 0x55b99c3b5490 con 0x7f55fc06a040 2024-08-03T20:54:38.883 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.882+0000 7f5602e00640 1 -- 172.21.15.32:0/4255060924 <== osd.0 v2:172.21.15.32:6816/3066156116 6 ==== osd_op_reply(24 [pgnls start_epoch 84 out=73b] v82'2 uv2 ondisk = 1) ==== 144+0+73 (crc 0 0 0) 0x7f55f405cba0 con 0x7f55fc06a040 2024-08-03T20:54:38.883 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.882+0000 7f5602e00640 1 -- 172.21.15.32:0/4255060924 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:25 12.3 12:c0000000::::head [pgnls start_epoch 84 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e84) -- 0x55b99c3b5490 con 0x7f55fc06a040 2024-08-03T20:54:38.883 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.882+0000 7f5602e00640 1 -- 172.21.15.32:0/4255060924 <== osd.0 v2:172.21.15.32:6816/3066156116 7 ==== osd_op_reply(25 [pgnls start_epoch 84 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f55f405cba0 con 0x7f55fc06a040 2024-08-03T20:54:38.883 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.882+0000 7f5602e00640 1 -- 172.21.15.32:0/4255060924 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:26 12.13 12:c8000000::::head [pgnls start_epoch 84 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e84) -- 0x55b99c3b5490 con 0x7f55f0008420 2024-08-03T20:54:38.884 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.882+0000 7f5603800640 1 -- 172.21.15.32:0/4255060924 <== osd.1 v2:172.21.15.32:6824/1551241759 3 ==== osd_op_reply(26 [pgnls start_epoch 84 out=98b] v83'3 uv3 ondisk = 1) ==== 144+0+98 (crc 0 0 0) 0x7f55fc00b040 con 0x7f55f0008420 2024-08-03T20:54:38.884 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.882+0000 7f5603800640 1 -- 172.21.15.32:0/4255060924 --> [v2:172.21.15.32:6800/9088303,v1:172.21.15.32:6801/9088303] -- osd_op(unknown.0.0:27 12.b 12:d0000000::::head [pgnls start_epoch 84 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e84) -- 0x55b99c3b5490 con 0x7f55f4016020 2024-08-03T20:54:38.884 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.883+0000 7f5602400640 1 -- 172.21.15.32:0/4255060924 <== osd.3 v2:172.21.15.32:6800/9088303 3 ==== osd_op_reply(27 [pgnls start_epoch 84 out=76b] v83'3 uv3 ondisk = 1) ==== 144+0+76 (crc 0 0 0) 0x7f55f000a970 con 0x7f55f4016020 2024-08-03T20:54:38.884 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.883+0000 7f5602400640 1 -- 172.21.15.32:0/4255060924 --> [v2:172.21.15.118:6802/864752275,v1:172.21.15.118:6803/864752275] -- osd_op(unknown.0.0:28 12.1b 12:d8000000::::head [pgnls start_epoch 84 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e84) -- 0x55b99c3b5490 con 0x55b99c3c5a80 2024-08-03T20:54:38.885 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.883+0000 7f5603800640 1 -- 172.21.15.32:0/4255060924 <== osd.4 v2:172.21.15.118:6802/864752275 5 ==== osd_op_reply(28 [pgnls start_epoch 84 out=49b] v82'1 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f55fc00b040 con 0x55b99c3c5a80 2024-08-03T20:54:38.885 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.883+0000 7f5603800640 1 -- 172.21.15.32:0/4255060924 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:29 12.7 12:e0000000::::head [pgnls start_epoch 84 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e84) -- 0x55b99c3b5490 con 0x7f55f0008420 2024-08-03T20:54:38.885 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.884+0000 7f5603800640 1 -- 172.21.15.32:0/4255060924 <== osd.1 v2:172.21.15.32:6824/1551241759 4 ==== osd_op_reply(29 [pgnls start_epoch 84 out=73b] v82'2 uv2 ondisk = 1) ==== 144+0+73 (crc 0 0 0) 0x7f55fc00b040 con 0x7f55f0008420 2024-08-03T20:54:38.885 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.884+0000 7f5603800640 1 -- 172.21.15.32:0/4255060924 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:30 12.17 12:e8000000::::head [pgnls start_epoch 84 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e84) -- 0x55b99c3b5490 con 0x7f55fc022cf0 2024-08-03T20:54:38.886 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.884+0000 7f5602e00640 1 -- 172.21.15.32:0/4255060924 <== osd.6 v2:172.21.15.118:6825/801947386 6 ==== osd_op_reply(30 [pgnls start_epoch 84 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f55f405cba0 con 0x7f55fc022cf0 2024-08-03T20:54:38.886 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.884+0000 7f5602e00640 1 -- 172.21.15.32:0/4255060924 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:31 12.f 12:f0000000::::head [pgnls start_epoch 84 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e84) -- 0x55b99c3b5490 con 0x7f55fc022cf0 2024-08-03T20:54:38.886 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.885+0000 7f5602e00640 1 -- 172.21.15.32:0/4255060924 <== osd.6 v2:172.21.15.118:6825/801947386 7 ==== osd_op_reply(31 [pgnls start_epoch 84 out=73b] v82'1 uv1 ondisk = 1) ==== 144+0+73 (crc 0 0 0) 0x7f55f405cba0 con 0x7f55fc022cf0 2024-08-03T20:54:38.886 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.885+0000 7f5602e00640 1 -- 172.21.15.32:0/4255060924 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:32 12.1f 12:f8000000::::head [pgnls start_epoch 84 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e84) -- 0x55b99c3b5490 con 0x7f55fc022cf0 2024-08-03T20:54:38.887 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.885+0000 7f5602e00640 1 -- 172.21.15.32:0/4255060924 <== osd.6 v2:172.21.15.118:6825/801947386 8 ==== osd_op_reply(32 [pgnls start_epoch 84 out=101b] v83'6 uv6 ondisk = 1) ==== 144+0+101 (crc 0 0 0) 0x7f55f405cba0 con 0x7f55fc022cf0 2024-08-03T20:54:38.887 INFO:teuthology.orchestra.run.smithi032.stdout:601.00000000 2024-08-03T20:54:38.887 INFO:teuthology.orchestra.run.smithi032.stdout:602.00000000 2024-08-03T20:54:38.887 INFO:teuthology.orchestra.run.smithi032.stdout:10000000000.00000000 2024-08-03T20:54:38.887 INFO:teuthology.orchestra.run.smithi032.stdout:600.00000000 2024-08-03T20:54:38.888 INFO:teuthology.orchestra.run.smithi032.stdout:603.00000000 2024-08-03T20:54:38.888 INFO:teuthology.orchestra.run.smithi032.stdout:1.00000000.inode 2024-08-03T20:54:38.888 INFO:teuthology.orchestra.run.smithi032.stdout:200.00000000 2024-08-03T20:54:38.888 INFO:teuthology.orchestra.run.smithi032.stdout:200.00000001 2024-08-03T20:54:38.888 INFO:teuthology.orchestra.run.smithi032.stdout:606.00000000 2024-08-03T20:54:38.888 INFO:teuthology.orchestra.run.smithi032.stdout:607.00000000 2024-08-03T20:54:38.888 INFO:teuthology.orchestra.run.smithi032.stdout:mds0_openfiles.0 2024-08-03T20:54:38.888 INFO:teuthology.orchestra.run.smithi032.stdout:608.00000000 2024-08-03T20:54:38.888 INFO:teuthology.orchestra.run.smithi032.stdout:604.00000000 2024-08-03T20:54:38.888 INFO:teuthology.orchestra.run.smithi032.stdout:500.00000000 2024-08-03T20:54:38.888 INFO:teuthology.orchestra.run.smithi032.stdout:mds_snaptable 2024-08-03T20:54:38.888 INFO:teuthology.orchestra.run.smithi032.stdout:605.00000000 2024-08-03T20:54:38.889 INFO:teuthology.orchestra.run.smithi032.stdout:mds0_inotable 2024-08-03T20:54:38.889 INFO:teuthology.orchestra.run.smithi032.stdout:100.00000000 2024-08-03T20:54:38.889 INFO:teuthology.orchestra.run.smithi032.stdout:mds0_sessionmap 2024-08-03T20:54:38.889 INFO:teuthology.orchestra.run.smithi032.stdout:609.00000000 2024-08-03T20:54:38.889 INFO:teuthology.orchestra.run.smithi032.stdout:400.00000000 2024-08-03T20:54:38.889 INFO:teuthology.orchestra.run.smithi032.stdout:100.00000000.inode 2024-08-03T20:54:38.889 INFO:teuthology.orchestra.run.smithi032.stdout:1.00000000 2024-08-03T20:54:38.890 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.885+0000 7f5609c77f40 1 -- 172.21.15.32:0/4255060924 >> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] conn(0x7f55fc06a040 msgr2=0x7f55fc003410 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:38.890 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.885+0000 7f5609c77f40 1 --2- 172.21.15.32:0/4255060924 >> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] conn(0x7f55fc06a040 0x7f55fc003410 crc :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:38.890 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.885+0000 7f5609c77f40 1 -- 172.21.15.32:0/4255060924 >> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] conn(0x7f55f0008420 msgr2=0x7f55f0008830 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:38.891 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.885+0000 7f5609c77f40 1 --2- 172.21.15.32:0/4255060924 >> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] conn(0x7f55f0008420 0x7f55f0008830 crc :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:38.891 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.885+0000 7f5609c77f40 1 -- 172.21.15.32:0/4255060924 >> [v2:172.21.15.32:6808/3806708944,v1:172.21.15.32:6809/3806708944] conn(0x7f55f0028970 msgr2=0x7f55f0018400 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:38.891 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.885+0000 7f5609c77f40 1 --2- 172.21.15.32:0/4255060924 >> [v2:172.21.15.32:6808/3806708944,v1:172.21.15.32:6809/3806708944] conn(0x7f55f0028970 0x7f55f0018400 crc :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:38.891 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.885+0000 7f5609c77f40 1 -- 172.21.15.32:0/4255060924 >> [v2:172.21.15.32:6800/9088303,v1:172.21.15.32:6801/9088303] conn(0x7f55f4016020 msgr2=0x7f55f405f5e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:38.891 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.885+0000 7f5609c77f40 1 --2- 172.21.15.32:0/4255060924 >> [v2:172.21.15.32:6800/9088303,v1:172.21.15.32:6801/9088303] conn(0x7f55f4016020 0x7f55f405f5e0 crc :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:38.891 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.885+0000 7f5609c77f40 1 -- 172.21.15.32:0/4255060924 >> [v2:172.21.15.118:6802/864752275,v1:172.21.15.118:6803/864752275] conn(0x55b99c3c5a80 msgr2=0x55b99c3b4f50 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:38.891 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.885+0000 7f5609c77f40 1 --2- 172.21.15.32:0/4255060924 >> [v2:172.21.15.118:6802/864752275,v1:172.21.15.118:6803/864752275] conn(0x55b99c3c5a80 0x55b99c3b4f50 crc :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:38.891 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.885+0000 7f5609c77f40 1 -- 172.21.15.32:0/4255060924 >> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] conn(0x7f55f4060b70 msgr2=0x7f55f4016720 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:38.891 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.885+0000 7f5609c77f40 1 --2- 172.21.15.32:0/4255060924 >> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] conn(0x7f55f4060b70 0x7f55f4016720 crc :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:38.891 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.885+0000 7f5609c77f40 1 -- 172.21.15.32:0/4255060924 >> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] conn(0x7f55fc022cf0 msgr2=0x7f55fc06c460 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:38.891 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.885+0000 7f5609c77f40 1 --2- 172.21.15.32:0/4255060924 >> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] conn(0x7f55fc022cf0 0x7f55fc06c460 crc :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:38.891 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.885+0000 7f5609c77f40 1 -- 172.21.15.32:0/4255060924 >> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] conn(0x7f55f40694a0 msgr2=0x7f55f40612a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:38.891 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.885+0000 7f5603800640 1 -- 172.21.15.32:0/4255060924 reap_dead start 2024-08-03T20:54:38.892 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.885+0000 7f5609c77f40 1 --2- 172.21.15.32:0/4255060924 >> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] conn(0x7f55f40694a0 0x7f55f40612a0 crc :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:38.892 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.885+0000 7f5603800640 1 -- 172.21.15.32:0/4255060924 reap_dead start 2024-08-03T20:54:38.892 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.886+0000 7f5603800640 1 -- 172.21.15.32:0/4255060924 reap_dead start 2024-08-03T20:54:38.892 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.886+0000 7f5603800640 1 -- 172.21.15.32:0/4255060924 reap_dead start 2024-08-03T20:54:38.892 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.886+0000 7f5609c77f40 1 -- 172.21.15.32:0/4255060924 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f55e4075a20 msgr2=0x7f55e4077f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:38.892 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.886+0000 7f5609c77f40 1 --2- 172.21.15.32:0/4255060924 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f55e4075a20 0x7f55e4077f30 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f55f0004230 tx=0x7f55f0002040 comp rx=0 tx=0).stop 2024-08-03T20:54:38.892 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.886+0000 7f5609c77f40 1 -- 172.21.15.32:0/4255060924 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b99c2f5d80 msgr2=0x55b99c3429d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:38.892 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.886+0000 7f5609c77f40 1 --2- 172.21.15.32:0/4255060924 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b99c2f5d80 0x55b99c3429d0 secure :-1 s=READY pgs=942 cs=0 l=1 rev1=1 crypto rx=0x7f55f40086c0 tx=0x7f55f40087b0 comp rx=0 tx=0).stop 2024-08-03T20:54:38.892 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.886+0000 7f5609c77f40 1 -- 172.21.15.32:0/4255060924 shutdown_connections 2024-08-03T20:54:38.892 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.886+0000 7f5609c77f40 1 --2- 172.21.15.32:0/4255060924 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f55e4075a20 0x7f55e4077f30 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:38.892 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.886+0000 7f5609c77f40 1 --2- 172.21.15.32:0/4255060924 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b99c2f5d80 0x55b99c3429d0 unknown :-1 s=CLOSED pgs=942 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:38.892 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.886+0000 7f5609c77f40 1 -- 172.21.15.32:0/4255060924 >> 172.21.15.32:0/4255060924 conn(0x55b99c217ac0 msgr2=0x55b99c33a0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:54:38.892 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.886+0000 7f5609c77f40 1 -- 172.21.15.32:0/4255060924 shutdown_connections 2024-08-03T20:54:38.893 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:38.886+0000 7f5609c77f40 1 -- 172.21.15.32:0/4255060924 wait complete. 2024-08-03T20:54:38.902 DEBUG:teuthology.orchestra.run.smithi032:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata getomapheader 10000000000.00000000 2024-08-03T20:54:39.045 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.044+0000 7f78c8a0ff40 1 Processor -- start 2024-08-03T20:54:39.045 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.044+0000 7f78c8a0ff40 1 -- start start 2024-08-03T20:54:39.048 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.045+0000 7f78c8a0ff40 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x564da21c6b40 0x564da222f120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:39.048 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.045+0000 7f78c8a0ff40 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x564da2239be0 0x564da223aa50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:39.048 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.045+0000 7f78c8a0ff40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x564da223af90 0x564da22e4480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:39.048 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.045+0000 7f78c8a0ff40 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x564da20cfe30 con 0x564da21c6b40 2024-08-03T20:54:39.048 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.045+0000 7f78c8a0ff40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x564da212dbf0 con 0x564da223af90 2024-08-03T20:54:39.048 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.045+0000 7f78c8a0ff40 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x564da223a510 con 0x564da2239be0 2024-08-03T20:54:39.048 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.045+0000 7f78c0e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x564da21c6b40 0x564da222f120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:39.048 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.045+0000 7f78c0e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x564da21c6b40 0x564da222f120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:40232/0 (socket says 172.21.15.32:40232) 2024-08-03T20:54:39.048 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.045+0000 7f78c0e00640 1 -- 172.21.15.32:0/1196555171 learned_addr learned my addr 172.21.15.32:0/1196555171 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:54:39.048 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.045+0000 7f78c1800640 1 --2- 172.21.15.32:0/1196555171 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x564da223af90 0x564da22e4480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:39.048 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.045+0000 7f78bbe00640 1 --2- 172.21.15.32:0/1196555171 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x564da2239be0 0x564da223aa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:39.049 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.046+0000 7f78c0e00640 1 -- 172.21.15.32:0/1196555171 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x564da2239be0 msgr2=0x564da223aa50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:54:39.049 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.046+0000 7f78c0e00640 1 --2- 172.21.15.32:0/1196555171 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x564da2239be0 0x564da223aa50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:39.049 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.046+0000 7f78c0e00640 1 -- 172.21.15.32:0/1196555171 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x564da223af90 msgr2=0x564da22e4480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:54:39.049 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.046+0000 7f78c0e00640 1 --2- 172.21.15.32:0/1196555171 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x564da223af90 0x564da22e4480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:39.049 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.046+0000 7f78c0e00640 1 -- 172.21.15.32:0/1196555171 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564da2257f90 con 0x564da21c6b40 2024-08-03T20:54:39.049 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.046+0000 7f78c0e00640 1 --2- 172.21.15.32:0/1196555171 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x564da21c6b40 0x564da222f120 secure :-1 s=READY pgs=944 cs=0 l=1 rev1=1 crypto rx=0x7f78b400c5d0 tx=0x7f78b400ca90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=85740f07f73fc4bf server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:54:39.049 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.046+0000 7f78baa00640 1 -- 172.21.15.32:0/1196555171 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78b400aae0 con 0x564da21c6b40 2024-08-03T20:54:39.049 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.046+0000 7f78baa00640 1 -- 172.21.15.32:0/1196555171 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f78b400ac80 con 0x564da21c6b40 2024-08-03T20:54:39.049 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.046+0000 7f78baa00640 1 -- 172.21.15.32:0/1196555171 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78b40156a0 con 0x564da21c6b40 2024-08-03T20:54:39.050 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.047+0000 7f78c8a0ff40 1 -- 172.21.15.32:0/1196555171 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x564da21c6b40 msgr2=0x564da222f120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:39.050 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.047+0000 7f78c8a0ff40 1 --2- 172.21.15.32:0/1196555171 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x564da21c6b40 0x564da222f120 secure :-1 s=READY pgs=944 cs=0 l=1 rev1=1 crypto rx=0x7f78b400c5d0 tx=0x7f78b400ca90 comp rx=0 tx=0).stop 2024-08-03T20:54:39.050 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.047+0000 7f78c8a0ff40 1 -- 172.21.15.32:0/1196555171 shutdown_connections 2024-08-03T20:54:39.050 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.047+0000 7f78c8a0ff40 1 --2- 172.21.15.32:0/1196555171 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x564da223af90 0x564da22e4480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:39.051 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.047+0000 7f78c8a0ff40 1 --2- 172.21.15.32:0/1196555171 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x564da2239be0 0x564da223aa50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:39.051 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.047+0000 7f78c8a0ff40 1 --2- 172.21.15.32:0/1196555171 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x564da21c6b40 0x564da222f120 unknown :-1 s=CLOSED pgs=944 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:39.051 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.047+0000 7f78c8a0ff40 1 -- 172.21.15.32:0/1196555171 >> 172.21.15.32:0/1196555171 conn(0x564da20a1ac0 msgr2=0x564da2259cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:54:39.051 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.047+0000 7f78c8a0ff40 1 -- 172.21.15.32:0/1196555171 shutdown_connections 2024-08-03T20:54:39.051 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.047+0000 7f78c8a0ff40 1 -- 172.21.15.32:0/1196555171 wait complete. 2024-08-03T20:54:39.051 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.048+0000 7f78c8a0ff40 1 Processor -- start 2024-08-03T20:54:39.052 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.048+0000 7f78c8a0ff40 1 -- start start 2024-08-03T20:54:39.053 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.050+0000 7f78c8a0ff40 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x564da21c6b40 0x564da2232a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:39.053 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.050+0000 7f78c8a0ff40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x564da2239be0 0x564da22554a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:39.053 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.050+0000 7f78c8a0ff40 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x564da223af90 0x564da2253ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:39.054 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.050+0000 7f78c8a0ff40 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x564da222e3f0 con 0x564da21c6b40 2024-08-03T20:54:39.054 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.050+0000 7f78c8a0ff40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x564da222e640 con 0x564da2239be0 2024-08-03T20:54:39.054 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.050+0000 7f78c8a0ff40 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x564da22ef260 con 0x564da223af90 2024-08-03T20:54:39.054 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.050+0000 7f78c0e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x564da21c6b40 0x564da2232a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:39.054 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.050+0000 7f78c0e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x564da21c6b40 0x564da2232a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:40248/0 (socket says 172.21.15.32:40248) 2024-08-03T20:54:39.054 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.050+0000 7f78c0e00640 1 -- 172.21.15.32:0/2139704252 learned_addr learned my addr 172.21.15.32:0/2139704252 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:54:39.054 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.050+0000 7f78c1800640 1 --2- 172.21.15.32:0/2139704252 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x564da223af90 0x564da2253ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:39.054 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.050+0000 7f78bbe00640 1 --2- 172.21.15.32:0/2139704252 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x564da2239be0 0x564da22554a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:39.054 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.051+0000 7f78c0e00640 1 -- 172.21.15.32:0/2139704252 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x564da223af90 msgr2=0x564da2253ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:39.054 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.051+0000 7f78c0e00640 1 --2- 172.21.15.32:0/2139704252 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x564da223af90 0x564da2253ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:39.054 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.051+0000 7f78c0e00640 1 -- 172.21.15.32:0/2139704252 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x564da2239be0 msgr2=0x564da22554a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:39.055 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.051+0000 7f78c0e00640 1 --2- 172.21.15.32:0/2139704252 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x564da2239be0 0x564da22554a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:39.055 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.051+0000 7f78c0e00640 1 -- 172.21.15.32:0/2139704252 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564da21c9850 con 0x564da21c6b40 2024-08-03T20:54:39.055 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.051+0000 7f78bbe00640 1 --2- 172.21.15.32:0/2139704252 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x564da2239be0 0x564da22554a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-03T20:54:39.055 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.051+0000 7f78c0e00640 1 --2- 172.21.15.32:0/2139704252 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x564da21c6b40 0x564da2232a50 secure :-1 s=READY pgs=945 cs=0 l=1 rev1=1 crypto rx=0x7f78b400cf90 tx=0x7f78b4015440 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:54:39.055 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.051+0000 7f78b9600640 1 -- 172.21.15.32:0/2139704252 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78b4024070 con 0x564da21c6b40 2024-08-03T20:54:39.055 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.051+0000 7f78b9600640 1 -- 172.21.15.32:0/2139704252 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f78b4015ae0 con 0x564da21c6b40 2024-08-03T20:54:39.055 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.051+0000 7f78c8a0ff40 1 -- 172.21.15.32:0/2139704252 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x564da22ef050 con 0x564da21c6b40 2024-08-03T20:54:39.055 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.051+0000 7f78b9600640 1 -- 172.21.15.32:0/2139704252 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78b401f430 con 0x564da21c6b40 2024-08-03T20:54:39.055 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.051+0000 7f78c8a0ff40 1 -- 172.21.15.32:0/2139704252 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564da2259360 con 0x564da21c6b40 2024-08-03T20:54:39.056 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.054+0000 7f78b9600640 1 -- 172.21.15.32:0/2139704252 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f78b4027070 con 0x564da21c6b40 2024-08-03T20:54:39.056 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.054+0000 7f78b9600640 1 --2- 172.21.15.32:0/2139704252 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f78a0075a20 0x7f78a0077f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:39.056 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.055+0000 7f78b9600640 1 -- 172.21.15.32:0/2139704252 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 5795+0+0 (secure 0 0 0) 0x7f78b4098fb0 con 0x564da21c6b40 2024-08-03T20:54:39.057 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.055+0000 7f78c8a0ff40 1 --2- 172.21.15.32:0/2139704252 >> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] conn(0x564da2259c80 0x564da21c8020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:39.057 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.055+0000 7f78c8a0ff40 1 -- 172.21.15.32:0/2139704252 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:1 12.10 12:0cf6ad0f:::10000000000.00000000:head [omap-get-header] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84) -- 0x564da21c8560 con 0x564da2259c80 2024-08-03T20:54:39.057 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.055+0000 7f78bbe00640 1 --2- 172.21.15.32:0/2139704252 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f78a0075a20 0x7f78a0077f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:39.057 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.055+0000 7f78c1800640 1 --2- 172.21.15.32:0/2139704252 >> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] conn(0x564da2259c80 0x564da21c8020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:39.058 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.056+0000 7f78c1800640 1 --2- 172.21.15.32:0/2139704252 >> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] conn(0x564da2259c80 0x564da21c8020 crc :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:54:39.058 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.056+0000 7f78bbe00640 1 --2- 172.21.15.32:0/2139704252 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f78a0075a20 0x7f78a0077f30 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x564da222eeb0 tx=0x7f78ac001730 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:54:39.058 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.056+0000 7f78c1800640 1 -- 172.21.15.32:0/2139704252 <== osd.0 v2:172.21.15.32:6816/3066156116 1 ==== osd_op_reply(1 10000000000.00000000 [omap-get-header out=274b] v0'0 uv10 ondisk = 0) ==== 164+0+274 (crc 0 0 0) 0x7f78bc00b040 con 0x564da2259c80 2024-08-03T20:54:39.058 INFO:teuthology.orchestra.run.smithi032.stdout:header (274 bytes) : 2024-08-03T20:54:39.058 INFO:teuthology.orchestra.run.smithi032.stdout:00000000 04 03 0c 01 00 00 0b 00 00 00 00 00 00 00 00 00 |................| 2024-08-03T20:54:39.058 INFO:teuthology.orchestra.run.smithi032.stdout:00000010 00 00 00 00 00 00 03 02 28 00 00 00 00 00 00 00 |........(.......| 2024-08-03T20:54:39.058 INFO:teuthology.orchestra.run.smithi032.stdout:00000020 00 00 00 00 0c 99 ae 66 13 1d b1 05 01 00 00 00 |.......f........| 2024-08-03T20:54:39.058 INFO:teuthology.orchestra.run.smithi032.stdout:00000030 00 00 00 00 00 00 00 00 00 00 00 00 01 00 00 00 |................| 2024-08-03T20:54:39.059 INFO:teuthology.orchestra.run.smithi032.stdout:00000040 00 00 00 00 03 02 28 00 00 00 00 00 00 00 00 00 |......(.........| 2024-08-03T20:54:39.059 INFO:teuthology.orchestra.run.smithi032.stdout:00000050 00 00 0c 99 ae 66 13 1d b1 05 01 00 00 00 00 00 |.....f..........| 2024-08-03T20:54:39.059 INFO:teuthology.orchestra.run.smithi032.stdout:00000060 00 00 00 00 00 00 00 00 00 00 01 00 00 00 00 00 |................| 2024-08-03T20:54:39.059 INFO:teuthology.orchestra.run.smithi032.stdout:00000070 00 00 03 02 38 00 00 00 00 00 00 00 00 00 00 00 |....8...........| 2024-08-03T20:54:39.059 INFO:teuthology.orchestra.run.smithi032.stdout:00000080 00 00 60 00 00 00 00 00 01 00 00 00 00 00 00 00 |..`.............| 2024-08-03T20:54:39.059 INFO:teuthology.orchestra.run.smithi032.stdout:00000090 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| 2024-08-03T20:54:39.059 INFO:teuthology.orchestra.run.smithi032.stdout:000000a0 00 00 00 00 00 00 00 00 0c 99 ae 66 68 ad e5 07 |...........fh...| 2024-08-03T20:54:39.059 INFO:teuthology.orchestra.run.smithi032.stdout:000000b0 03 02 38 00 00 00 00 00 00 00 00 00 00 00 00 00 |..8.............| 2024-08-03T20:54:39.059 INFO:teuthology.orchestra.run.smithi032.stdout:000000c0 60 00 00 00 00 00 01 00 00 00 00 00 00 00 00 00 |`...............| 2024-08-03T20:54:39.059 INFO:teuthology.orchestra.run.smithi032.stdout:000000d0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| 2024-08-03T20:54:39.059 INFO:teuthology.orchestra.run.smithi032.stdout:000000e0 00 00 00 00 00 00 0c 99 ae 66 68 ad e5 07 00 00 |.........fh.....| 2024-08-03T20:54:39.059 INFO:teuthology.orchestra.run.smithi032.stdout:000000f0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| 2024-08-03T20:54:39.059 INFO:teuthology.orchestra.run.smithi032.stdout:* 2024-08-03T20:54:39.060 INFO:teuthology.orchestra.run.smithi032.stdout:00000110 00 00 |..| 2024-08-03T20:54:39.060 INFO:teuthology.orchestra.run.smithi032.stdout:00000112 2024-08-03T20:54:39.060 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-03T20:54:39.061 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.056+0000 7f78c8a0ff40 1 -- 172.21.15.32:0/2139704252 >> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] conn(0x564da2259c80 msgr2=0x564da21c8020 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:39.061 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.056+0000 7f78c8a0ff40 1 --2- 172.21.15.32:0/2139704252 >> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] conn(0x564da2259c80 0x564da21c8020 crc :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:39.061 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.056+0000 7f78c8a0ff40 1 -- 172.21.15.32:0/2139704252 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f78a0075a20 msgr2=0x7f78a0077f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:39.061 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.056+0000 7f78c8a0ff40 1 --2- 172.21.15.32:0/2139704252 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f78a0075a20 0x7f78a0077f30 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x564da222eeb0 tx=0x7f78ac001730 comp rx=0 tx=0).stop 2024-08-03T20:54:39.061 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.056+0000 7f78c8a0ff40 1 -- 172.21.15.32:0/2139704252 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x564da21c6b40 msgr2=0x564da2232a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:39.061 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.056+0000 7f78c8a0ff40 1 --2- 172.21.15.32:0/2139704252 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x564da21c6b40 0x564da2232a50 secure :-1 s=READY pgs=945 cs=0 l=1 rev1=1 crypto rx=0x7f78b400cf90 tx=0x7f78b4015440 comp rx=0 tx=0).stop 2024-08-03T20:54:39.061 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.056+0000 7f78c1800640 1 -- 172.21.15.32:0/2139704252 reap_dead start 2024-08-03T20:54:39.061 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.057+0000 7f78c8a0ff40 1 -- 172.21.15.32:0/2139704252 shutdown_connections 2024-08-03T20:54:39.061 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.057+0000 7f78c8a0ff40 1 -- 172.21.15.32:0/2139704252 >> 172.21.15.32:0/2139704252 conn(0x564da20a1ac0 msgr2=0x564da222f800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:54:39.061 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.057+0000 7f78c8a0ff40 1 -- 172.21.15.32:0/2139704252 shutdown_connections 2024-08-03T20:54:39.061 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.057+0000 7f78c8a0ff40 1 -- 172.21.15.32:0/2139704252 wait complete. 2024-08-03T20:54:39.072 DEBUG:teuthology.orchestra.run.smithi032:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata getomapheader 600.00000000 2024-08-03T20:54:39.215 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.214+0000 7f8c2c321f40 1 Processor -- start 2024-08-03T20:54:39.215 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.214+0000 7f8c2c321f40 1 -- start start 2024-08-03T20:54:39.218 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.215+0000 7f8c2c321f40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55fa51bffb40 0x55fa51c71ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:39.218 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.215+0000 7f8c2c321f40 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55fa51c72be0 0x55fa51bba170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:39.218 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.215+0000 7f8c2c321f40 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55fa51bb86c0 0x55fa51bba6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:39.218 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.215+0000 7f8c2c321f40 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55fa51b66bf0 con 0x55fa51bb86c0 2024-08-03T20:54:39.218 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.215+0000 7f8c2c321f40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x55fa51b736b0 con 0x55fa51bffb40 2024-08-03T20:54:39.218 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.215+0000 7f8c2c321f40 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x55fa51b08e30 con 0x55fa51c72be0 2024-08-03T20:54:39.218 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.216+0000 7f8c25400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55fa51bb86c0 0x55fa51bba6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:39.218 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.216+0000 7f8c25400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55fa51bb86c0 0x55fa51bba6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:40254/0 (socket says 172.21.15.32:40254) 2024-08-03T20:54:39.218 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.216+0000 7f8c25400640 1 -- 172.21.15.32:0/513888547 learned_addr learned my addr 172.21.15.32:0/513888547 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:54:39.218 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.216+0000 7f8c24a00640 1 --2- 172.21.15.32:0/513888547 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55fa51bffb40 0x55fa51c71ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:39.218 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.216+0000 7f8c1fe00640 1 --2- 172.21.15.32:0/513888547 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55fa51c72be0 0x55fa51bba170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:39.218 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.216+0000 7f8c25400640 1 -- 172.21.15.32:0/513888547 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55fa51c72be0 msgr2=0x55fa51bba170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:54:39.219 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.216+0000 7f8c25400640 1 --2- 172.21.15.32:0/513888547 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55fa51c72be0 0x55fa51bba170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:39.219 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.216+0000 7f8c25400640 1 -- 172.21.15.32:0/513888547 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55fa51bffb40 msgr2=0x55fa51c71ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:54:39.219 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.216+0000 7f8c25400640 1 --2- 172.21.15.32:0/513888547 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55fa51bffb40 0x55fa51c71ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:39.219 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.216+0000 7f8c25400640 1 -- 172.21.15.32:0/513888547 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fa51bb8d90 con 0x55fa51bb86c0 2024-08-03T20:54:39.219 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.216+0000 7f8c1fe00640 1 --2- 172.21.15.32:0/513888547 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55fa51c72be0 0x55fa51bba170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-03T20:54:39.219 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.216+0000 7f8c25400640 1 --2- 172.21.15.32:0/513888547 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55fa51bb86c0 0x55fa51bba6b0 secure :-1 s=READY pgs=947 cs=0 l=1 rev1=1 crypto rx=0x7f8c20006590 tx=0x7f8c20006a50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4927dd888be43498 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:54:39.220 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.217+0000 7f8c1ea00640 1 -- 172.21.15.32:0/513888547 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c20016070 con 0x55fa51bb86c0 2024-08-03T20:54:39.220 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.217+0000 7f8c1ea00640 1 -- 172.21.15.32:0/513888547 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f8c2000f730 con 0x55fa51bb86c0 2024-08-03T20:54:39.220 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.217+0000 7f8c1ea00640 1 -- 172.21.15.32:0/513888547 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c2000fb80 con 0x55fa51bb86c0 2024-08-03T20:54:39.221 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.217+0000 7f8c2c321f40 1 -- 172.21.15.32:0/513888547 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55fa51bb86c0 msgr2=0x55fa51bba6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:39.221 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.217+0000 7f8c2c321f40 1 --2- 172.21.15.32:0/513888547 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55fa51bb86c0 0x55fa51bba6b0 secure :-1 s=READY pgs=947 cs=0 l=1 rev1=1 crypto rx=0x7f8c20006590 tx=0x7f8c20006a50 comp rx=0 tx=0).stop 2024-08-03T20:54:39.221 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.217+0000 7f8c2c321f40 1 -- 172.21.15.32:0/513888547 shutdown_connections 2024-08-03T20:54:39.221 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.217+0000 7f8c2c321f40 1 --2- 172.21.15.32:0/513888547 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55fa51bb86c0 0x55fa51bba6b0 unknown :-1 s=CLOSED pgs=947 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:39.221 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.217+0000 7f8c2c321f40 1 --2- 172.21.15.32:0/513888547 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55fa51c72be0 0x55fa51bba170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:39.221 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.217+0000 7f8c2c321f40 1 --2- 172.21.15.32:0/513888547 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55fa51bffb40 0x55fa51c71ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:39.221 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.217+0000 7f8c2c321f40 1 -- 172.21.15.32:0/513888547 >> 172.21.15.32:0/513888547 conn(0x55fa51adaac0 msgr2=0x55fa51bbb700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:54:39.221 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.217+0000 7f8c2c321f40 1 -- 172.21.15.32:0/513888547 shutdown_connections 2024-08-03T20:54:39.221 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.217+0000 7f8c2c321f40 1 -- 172.21.15.32:0/513888547 wait complete. 2024-08-03T20:54:39.221 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.218+0000 7f8c2c321f40 1 Processor -- start 2024-08-03T20:54:39.221 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.219+0000 7f8c2c321f40 1 -- start start 2024-08-03T20:54:39.224 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.220+0000 7f8c2c321f40 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55fa51bb86c0 0x55fa51c67f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:39.224 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.220+0000 7f8c2c321f40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55fa51bffb40 0x55fa51c70380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:39.224 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.220+0000 7f8c2c321f40 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55fa51c72be0 0x55fa51c6c710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:39.224 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.220+0000 7f8c2c321f40 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55fa51bbad60 con 0x55fa51c72be0 2024-08-03T20:54:39.224 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.220+0000 7f8c2c321f40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x55fa51bff160 con 0x55fa51bffb40 2024-08-03T20:54:39.224 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.220+0000 7f8c2c321f40 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x55fa51c8e100 con 0x55fa51bb86c0 2024-08-03T20:54:39.224 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.220+0000 7f8c25400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55fa51c72be0 0x55fa51c6c710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:39.224 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.220+0000 7f8c25400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55fa51c72be0 0x55fa51c6c710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:40260/0 (socket says 172.21.15.32:40260) 2024-08-03T20:54:39.225 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.220+0000 7f8c25400640 1 -- 172.21.15.32:0/2851481569 learned_addr learned my addr 172.21.15.32:0/2851481569 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:54:39.225 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.220+0000 7f8c24a00640 1 --2- 172.21.15.32:0/2851481569 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55fa51bb86c0 0x55fa51c67f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:39.225 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.220+0000 7f8c1fe00640 1 --2- 172.21.15.32:0/2851481569 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55fa51bffb40 0x55fa51c70380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:39.225 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.221+0000 7f8c25400640 1 -- 172.21.15.32:0/2851481569 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55fa51bb86c0 msgr2=0x55fa51c67f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:39.225 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.221+0000 7f8c25400640 1 --2- 172.21.15.32:0/2851481569 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55fa51bb86c0 0x55fa51c67f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:39.225 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.221+0000 7f8c25400640 1 -- 172.21.15.32:0/2851481569 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55fa51bffb40 msgr2=0x55fa51c70380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:39.225 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.221+0000 7f8c25400640 1 --2- 172.21.15.32:0/2851481569 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55fa51bffb40 0x55fa51c70380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:39.225 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.221+0000 7f8c25400640 1 -- 172.21.15.32:0/2851481569 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fa51c03c80 con 0x55fa51c72be0 2024-08-03T20:54:39.225 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.221+0000 7f8c25400640 1 --2- 172.21.15.32:0/2851481569 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55fa51c72be0 0x55fa51c6c710 secure :-1 s=READY pgs=948 cs=0 l=1 rev1=1 crypto rx=0x55fa51c6c690 tx=0x7f8c2000ff90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:54:39.225 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.221+0000 7f8c1d600640 1 -- 172.21.15.32:0/2851481569 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c20020030 con 0x55fa51c72be0 2024-08-03T20:54:39.225 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.221+0000 7f8c1d600640 1 -- 172.21.15.32:0/2851481569 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f8c20025520 con 0x55fa51c72be0 2024-08-03T20:54:39.225 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.221+0000 7f8c2c321f40 1 -- 172.21.15.32:0/2851481569 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x55fa51c03f60 con 0x55fa51c72be0 2024-08-03T20:54:39.225 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.221+0000 7f8c1d600640 1 -- 172.21.15.32:0/2851481569 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c20016040 con 0x55fa51c72be0 2024-08-03T20:54:39.226 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.222+0000 7f8c2c321f40 1 -- 172.21.15.32:0/2851481569 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fa51c676d0 con 0x55fa51c72be0 2024-08-03T20:54:39.227 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.224+0000 7f8c1d600640 1 -- 172.21.15.32:0/2851481569 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f8c20023030 con 0x55fa51c72be0 2024-08-03T20:54:39.227 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.225+0000 7f8c1d600640 1 --2- 172.21.15.32:0/2851481569 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f8c040759f0 0x7f8c04077f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:39.227 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.225+0000 7f8c1d600640 1 -- 172.21.15.32:0/2851481569 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 5795+0+0 (secure 0 0 0) 0x7f8c20097c80 con 0x55fa51c72be0 2024-08-03T20:54:39.227 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.225+0000 7f8c2c321f40 1 --2- 172.21.15.32:0/2851481569 >> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] conn(0x55fa51c708c0 0x55fa51c65cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:39.227 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.225+0000 7f8c2c321f40 1 -- 172.21.15.32:0/2851481569 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:1 12.10 12:0d82a743:::600.00000000:head [omap-get-header] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84) -- 0x55fa51c031f0 con 0x55fa51c708c0 2024-08-03T20:54:39.227 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.225+0000 7f8c24a00640 1 --2- 172.21.15.32:0/2851481569 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f8c040759f0 0x7f8c04077f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:39.227 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.225+0000 7f8c1fe00640 1 --2- 172.21.15.32:0/2851481569 >> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] conn(0x55fa51c708c0 0x55fa51c65cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:39.228 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.226+0000 7f8c24a00640 1 --2- 172.21.15.32:0/2851481569 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f8c040759f0 0x7f8c04077f00 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f8c18055000 tx=0x7f8c18003020 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:54:39.228 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.226+0000 7f8c1fe00640 1 --2- 172.21.15.32:0/2851481569 >> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] conn(0x55fa51c708c0 0x55fa51c65cf0 crc :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:54:39.228 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.226+0000 7f8c1fe00640 1 -- 172.21.15.32:0/2851481569 <== osd.0 v2:172.21.15.32:6816/3066156116 1 ==== osd_op_reply(1 600.00000000 [omap-get-header out=274b] v0'0 uv5 ondisk = 0) ==== 156+0+274 (crc 0 0 0) 0x7f8c1000f680 con 0x55fa51c708c0 2024-08-03T20:54:39.228 INFO:teuthology.orchestra.run.smithi032.stdout:header (274 bytes) : 2024-08-03T20:54:39.229 INFO:teuthology.orchestra.run.smithi032.stdout:00000000 04 03 0c 01 00 00 01 00 00 00 00 00 00 00 00 00 |................| 2024-08-03T20:54:39.229 INFO:teuthology.orchestra.run.smithi032.stdout:00000010 00 00 00 00 00 00 03 02 28 00 00 00 00 00 00 00 |........(.......| 2024-08-03T20:54:39.229 INFO:teuthology.orchestra.run.smithi032.stdout:00000020 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| 2024-08-03T20:54:39.229 INFO:teuthology.orchestra.run.smithi032.stdout:* 2024-08-03T20:54:39.229 INFO:teuthology.orchestra.run.smithi032.stdout:00000040 00 00 00 00 03 02 28 00 00 00 00 00 00 00 00 00 |......(.........| 2024-08-03T20:54:39.229 INFO:teuthology.orchestra.run.smithi032.stdout:00000050 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| 2024-08-03T20:54:39.229 INFO:teuthology.orchestra.run.smithi032.stdout:* 2024-08-03T20:54:39.229 INFO:teuthology.orchestra.run.smithi032.stdout:00000070 00 00 03 02 38 00 00 00 00 00 00 00 00 00 00 00 |....8...........| 2024-08-03T20:54:39.229 INFO:teuthology.orchestra.run.smithi032.stdout:00000080 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| 2024-08-03T20:54:39.229 INFO:teuthology.orchestra.run.smithi032.stdout:* 2024-08-03T20:54:39.229 INFO:teuthology.orchestra.run.smithi032.stdout:000000b0 03 02 38 00 00 00 00 00 00 00 00 00 00 00 00 00 |..8.............| 2024-08-03T20:54:39.229 INFO:teuthology.orchestra.run.smithi032.stdout:000000c0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| 2024-08-03T20:54:39.229 INFO:teuthology.orchestra.run.smithi032.stdout:* 2024-08-03T20:54:39.230 INFO:teuthology.orchestra.run.smithi032.stdout:00000110 00 00 |..| 2024-08-03T20:54:39.230 INFO:teuthology.orchestra.run.smithi032.stdout:00000112 2024-08-03T20:54:39.230 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-03T20:54:39.231 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.226+0000 7f8c2c321f40 1 -- 172.21.15.32:0/2851481569 >> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] conn(0x55fa51c708c0 msgr2=0x55fa51c65cf0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:39.231 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.226+0000 7f8c2c321f40 1 --2- 172.21.15.32:0/2851481569 >> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] conn(0x55fa51c708c0 0x55fa51c65cf0 crc :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:39.231 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.227+0000 7f8c2c321f40 1 -- 172.21.15.32:0/2851481569 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f8c040759f0 msgr2=0x7f8c04077f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:39.231 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.227+0000 7f8c2c321f40 1 --2- 172.21.15.32:0/2851481569 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f8c040759f0 0x7f8c04077f00 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f8c18055000 tx=0x7f8c18003020 comp rx=0 tx=0).stop 2024-08-03T20:54:39.231 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.227+0000 7f8c2c321f40 1 -- 172.21.15.32:0/2851481569 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55fa51c72be0 msgr2=0x55fa51c6c710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:39.231 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.227+0000 7f8c2c321f40 1 --2- 172.21.15.32:0/2851481569 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55fa51c72be0 0x55fa51c6c710 secure :-1 s=READY pgs=948 cs=0 l=1 rev1=1 crypto rx=0x55fa51c6c690 tx=0x7f8c2000ff90 comp rx=0 tx=0).stop 2024-08-03T20:54:39.231 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.227+0000 7f8c25400640 1 -- 172.21.15.32:0/2851481569 reap_dead start 2024-08-03T20:54:39.231 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.227+0000 7f8c2c321f40 1 -- 172.21.15.32:0/2851481569 shutdown_connections 2024-08-03T20:54:39.231 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.227+0000 7f8c2c321f40 1 -- 172.21.15.32:0/2851481569 >> 172.21.15.32:0/2851481569 conn(0x55fa51adaac0 msgr2=0x55fa51bbb700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:54:39.231 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.227+0000 7f8c2c321f40 1 -- 172.21.15.32:0/2851481569 shutdown_connections 2024-08-03T20:54:39.231 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.227+0000 7f8c2c321f40 1 -- 172.21.15.32:0/2851481569 wait complete. 2024-08-03T20:54:39.242 DEBUG:teuthology.orchestra.run.smithi032:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata getomapheader 1.00000000.inode 2024-08-03T20:54:39.385 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.383+0000 7f5d09152f40 1 Processor -- start 2024-08-03T20:54:39.385 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.383+0000 7f5d09152f40 1 -- start start 2024-08-03T20:54:39.387 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.385+0000 7f5d09152f40 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x559c32f72b40 0x559c32fe3380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:39.388 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.385+0000 7f5d09152f40 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x559c32fe5be0 0x559c33095bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:39.388 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.385+0000 7f5d09152f40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x559c32f2d940 0x559c33096130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:39.388 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.385+0000 7f5d09152f40 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x559c32fe6510 con 0x559c32fe5be0 2024-08-03T20:54:39.388 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.385+0000 7f5d09152f40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x559c32e7be30 con 0x559c32f2d940 2024-08-03T20:54:39.388 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.385+0000 7f5d09152f40 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x559c32fe6690 con 0x559c32f72b40 2024-08-03T20:54:39.388 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.385+0000 7f5d01800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x559c32fe5be0 0x559c33095bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:39.388 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.385+0000 7f5d01800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x559c32fe5be0 0x559c33095bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:40270/0 (socket says 172.21.15.32:40270) 2024-08-03T20:54:39.388 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.385+0000 7f5d01800640 1 -- 172.21.15.32:0/353384995 learned_addr learned my addr 172.21.15.32:0/353384995 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:54:39.388 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.385+0000 7f5d02c00640 1 --2- 172.21.15.32:0/353384995 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x559c32f2d940 0x559c33096130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:39.388 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.385+0000 7f5d01800640 1 -- 172.21.15.32:0/353384995 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x559c32f72b40 msgr2=0x559c32fe3380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:54:39.389 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.385+0000 7f5d01800640 1 --2- 172.21.15.32:0/353384995 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x559c32f72b40 0x559c32fe3380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:39.389 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.385+0000 7f5d01800640 1 -- 172.21.15.32:0/353384995 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x559c32f2d940 msgr2=0x559c33096130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:54:39.389 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.385+0000 7f5d01800640 1 --2- 172.21.15.32:0/353384995 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x559c32f2d940 0x559c33096130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:39.389 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.385+0000 7f5d01800640 1 -- 172.21.15.32:0/353384995 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559c32e54f10 con 0x559c32fe5be0 2024-08-03T20:54:39.389 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.385+0000 7f5d02c00640 1 --2- 172.21.15.32:0/353384995 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x559c32f2d940 0x559c33096130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-03T20:54:39.389 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.386+0000 7f5d01800640 1 --2- 172.21.15.32:0/353384995 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x559c32fe5be0 0x559c33095bf0 secure :-1 s=READY pgs=950 cs=0 l=1 rev1=1 crypto rx=0x7f5cf000c530 tx=0x7f5cf000c9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=599862d9e0a38384 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:54:39.389 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.386+0000 7f5cfbe00640 1 -- 172.21.15.32:0/353384995 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5cf000aa00 con 0x559c32fe5be0 2024-08-03T20:54:39.389 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.386+0000 7f5cfbe00640 1 -- 172.21.15.32:0/353384995 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f5cf000aba0 con 0x559c32fe5be0 2024-08-03T20:54:39.389 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.386+0000 7f5cfbe00640 1 -- 172.21.15.32:0/353384995 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5cf0015440 con 0x559c32fe5be0 2024-08-03T20:54:39.389 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.386+0000 7f5d09152f40 1 -- 172.21.15.32:0/353384995 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x559c32fe5be0 msgr2=0x559c33095bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:39.390 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.386+0000 7f5d09152f40 1 --2- 172.21.15.32:0/353384995 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x559c32fe5be0 0x559c33095bf0 secure :-1 s=READY pgs=950 cs=0 l=1 rev1=1 crypto rx=0x7f5cf000c530 tx=0x7f5cf000c9f0 comp rx=0 tx=0).stop 2024-08-03T20:54:39.393 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.386+0000 7f5d09152f40 1 -- 172.21.15.32:0/353384995 shutdown_connections 2024-08-03T20:54:39.393 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.386+0000 7f5d09152f40 1 --2- 172.21.15.32:0/353384995 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x559c32f2d940 0x559c33096130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:39.393 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.386+0000 7f5d09152f40 1 --2- 172.21.15.32:0/353384995 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x559c32fe5be0 0x559c33095bf0 unknown :-1 s=CLOSED pgs=950 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:39.393 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.386+0000 7f5d09152f40 1 --2- 172.21.15.32:0/353384995 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x559c32f72b40 0x559c32fe3380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:39.393 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.386+0000 7f5d09152f40 1 -- 172.21.15.32:0/353384995 >> 172.21.15.32:0/353384995 conn(0x559c32e4dac0 msgr2=0x559c330026f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:54:39.394 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.387+0000 7f5d09152f40 1 -- 172.21.15.32:0/353384995 shutdown_connections 2024-08-03T20:54:39.394 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.387+0000 7f5d09152f40 1 -- 172.21.15.32:0/353384995 wait complete. 2024-08-03T20:54:39.394 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.388+0000 7f5d09152f40 1 Processor -- start 2024-08-03T20:54:39.394 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.388+0000 7f5d09152f40 1 -- start start 2024-08-03T20:54:39.394 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.389+0000 7f5d09152f40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x559c32f2d940 0x559c32fda0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:39.394 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.389+0000 7f5d09152f40 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x559c32f72b40 0x559c330044a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:39.394 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.389+0000 7f5d09152f40 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x559c32fe5be0 0x559c330049e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:39.394 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.389+0000 7f5d09152f40 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x559c32fe7d40 con 0x559c32fe5be0 2024-08-03T20:54:39.395 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.389+0000 7f5d09152f40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x559c32fe7810 con 0x559c32f2d940 2024-08-03T20:54:39.395 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.389+0000 7f5d09152f40 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x559c3309a5f0 con 0x559c32f72b40 2024-08-03T20:54:39.395 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.389+0000 7f5d02c00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x559c32fe5be0 0x559c330049e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:39.395 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.389+0000 7f5d02c00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x559c32fe5be0 0x559c330049e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:40278/0 (socket says 172.21.15.32:40278) 2024-08-03T20:54:39.395 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.389+0000 7f5d02c00640 1 -- 172.21.15.32:0/1062262589 learned_addr learned my addr 172.21.15.32:0/1062262589 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:54:39.395 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.389+0000 7f5d02200640 1 --2- 172.21.15.32:0/1062262589 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x559c32f2d940 0x559c32fda0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:39.395 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.390+0000 7f5d01800640 1 --2- 172.21.15.32:0/1062262589 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x559c32f72b40 0x559c330044a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:39.395 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.390+0000 7f5d02c00640 1 -- 172.21.15.32:0/1062262589 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x559c32f72b40 msgr2=0x559c330044a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:39.395 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.390+0000 7f5d02c00640 1 --2- 172.21.15.32:0/1062262589 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x559c32f72b40 0x559c330044a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:39.395 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.390+0000 7f5d02c00640 1 -- 172.21.15.32:0/1062262589 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x559c32f2d940 msgr2=0x559c32fda0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:39.395 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.390+0000 7f5d02c00640 1 --2- 172.21.15.32:0/1062262589 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x559c32f2d940 0x559c32fda0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:39.395 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.390+0000 7f5d02c00640 1 -- 172.21.15.32:0/1062262589 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559c3308c600 con 0x559c32fe5be0 2024-08-03T20:54:39.395 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.390+0000 7f5d02c00640 1 --2- 172.21.15.32:0/1062262589 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x559c32fe5be0 0x559c330049e0 secure :-1 s=READY pgs=951 cs=0 l=1 rev1=1 crypto rx=0x7f5cfc00e610 tx=0x7f5cfc00ead0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:54:39.396 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.390+0000 7f5cfaa00640 1 -- 172.21.15.32:0/1062262589 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5cfc004220 con 0x559c32fe5be0 2024-08-03T20:54:39.396 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.390+0000 7f5cfaa00640 1 -- 172.21.15.32:0/1062262589 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f5cfc010710 con 0x559c32fe5be0 2024-08-03T20:54:39.396 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.390+0000 7f5cfaa00640 1 -- 172.21.15.32:0/1062262589 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5cfc010b20 con 0x559c32fe5be0 2024-08-03T20:54:39.396 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.390+0000 7f5d09152f40 1 -- 172.21.15.32:0/1062262589 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x559c3308c320 con 0x559c32fe5be0 2024-08-03T20:54:39.396 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.391+0000 7f5d09152f40 1 -- 172.21.15.32:0/1062262589 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559c32f76860 con 0x559c32fe5be0 2024-08-03T20:54:39.397 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.393+0000 7f5cfaa00640 1 -- 172.21.15.32:0/1062262589 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f5cfc010cc0 con 0x559c32fe5be0 2024-08-03T20:54:39.397 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.394+0000 7f5cfaa00640 1 --2- 172.21.15.32:0/1062262589 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f5ce40759f0 0x7f5ce4077f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:39.397 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.394+0000 7f5cfaa00640 1 -- 172.21.15.32:0/1062262589 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 5795+0+0 (secure 0 0 0) 0x7f5cfc098260 con 0x559c32fe5be0 2024-08-03T20:54:39.397 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.394+0000 7f5d09152f40 1 --2- 172.21.15.32:0/1062262589 >> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] conn(0x559c3308df60 0x559c3308e370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:39.398 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.394+0000 7f5d09152f40 1 -- 172.21.15.32:0/1062262589 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:1 12.14 12:287034c4:::1.00000000.inode:head [omap-get-header] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84) -- 0x559c3309a250 con 0x559c3308df60 2024-08-03T20:54:39.398 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.394+0000 7f5d02200640 1 --2- 172.21.15.32:0/1062262589 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f5ce40759f0 0x7f5ce4077f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:39.398 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.394+0000 7f5d01800640 1 --2- 172.21.15.32:0/1062262589 >> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] conn(0x559c3308df60 0x559c3308e370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:39.398 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.395+0000 7f5d02200640 1 --2- 172.21.15.32:0/1062262589 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f5ce40759f0 0x7f5ce4077f00 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f5cf4055000 tx=0x7f5cf4002fb0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:54:39.398 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.395+0000 7f5d01800640 1 --2- 172.21.15.32:0/1062262589 >> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] conn(0x559c3308df60 0x559c3308e370 crc :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:54:39.398 INFO:teuthology.orchestra.run.smithi032.stdout:header (0 bytes) : 2024-08-03T20:54:39.398 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-03T20:54:39.399 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.396+0000 7f5d01800640 1 -- 172.21.15.32:0/1062262589 <== osd.5 v2:172.21.15.118:6810/2724891770 1 ==== osd_op_reply(1 1.00000000.inode [omap-get-header] v0'0 uv15 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f5cf001d040 con 0x559c3308df60 2024-08-03T20:54:39.399 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.396+0000 7f5d09152f40 1 -- 172.21.15.32:0/1062262589 >> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] conn(0x559c3308df60 msgr2=0x559c3308e370 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:39.400 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.396+0000 7f5d09152f40 1 --2- 172.21.15.32:0/1062262589 >> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] conn(0x559c3308df60 0x559c3308e370 crc :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:39.400 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.396+0000 7f5d09152f40 1 -- 172.21.15.32:0/1062262589 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f5ce40759f0 msgr2=0x7f5ce4077f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:39.400 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.396+0000 7f5d09152f40 1 --2- 172.21.15.32:0/1062262589 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f5ce40759f0 0x7f5ce4077f00 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f5cf4055000 tx=0x7f5cf4002fb0 comp rx=0 tx=0).stop 2024-08-03T20:54:39.400 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.396+0000 7f5d09152f40 1 -- 172.21.15.32:0/1062262589 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x559c32fe5be0 msgr2=0x559c330049e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:39.400 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.396+0000 7f5d09152f40 1 --2- 172.21.15.32:0/1062262589 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x559c32fe5be0 0x559c330049e0 secure :-1 s=READY pgs=951 cs=0 l=1 rev1=1 crypto rx=0x7f5cfc00e610 tx=0x7f5cfc00ead0 comp rx=0 tx=0).stop 2024-08-03T20:54:39.400 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.396+0000 7f5d02c00640 1 -- 172.21.15.32:0/1062262589 reap_dead start 2024-08-03T20:54:39.400 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.396+0000 7f5d09152f40 1 -- 172.21.15.32:0/1062262589 shutdown_connections 2024-08-03T20:54:39.400 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.396+0000 7f5d09152f40 1 -- 172.21.15.32:0/1062262589 >> 172.21.15.32:0/1062262589 conn(0x559c32e4dac0 msgr2=0x559c33002660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:54:39.400 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.396+0000 7f5d09152f40 1 -- 172.21.15.32:0/1062262589 shutdown_connections 2024-08-03T20:54:39.401 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.396+0000 7f5d09152f40 1 -- 172.21.15.32:0/1062262589 wait complete. 2024-08-03T20:54:39.408 DEBUG:teuthology.orchestra.run.smithi032:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata getomapheader 200.00000000 2024-08-03T20:54:39.557 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.551+0000 7fc8cf7c7f40 1 Processor -- start 2024-08-03T20:54:39.557 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.551+0000 7fc8cf7c7f40 1 -- start start 2024-08-03T20:54:39.557 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.552+0000 7fc8cf7c7f40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x561f76feb360 0x561f76f7d700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:39.557 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.552+0000 7fc8cf7c7f40 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561f76fecea0 0x561f77006a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:39.557 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.552+0000 7fc8cf7c7f40 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x561f77008cb0 0x561f7709cdb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:39.557 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.552+0000 7fc8cf7c7f40 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x561f76e82e30 con 0x561f76fecea0 2024-08-03T20:54:39.557 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.552+0000 7fc8cf7c7f40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x561f76ee0bf0 con 0x561f76feb360 2024-08-03T20:54:39.557 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.552+0000 7fc8cf7c7f40 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x561f76fec1b0 con 0x561f77008cb0 2024-08-03T20:54:39.558 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.552+0000 7fc8c3e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561f76fecea0 0x561f77006a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:39.558 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.552+0000 7fc8c3e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561f76fecea0 0x561f77006a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:40294/0 (socket says 172.21.15.32:40294) 2024-08-03T20:54:39.558 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.552+0000 7fc8c3e00640 1 -- 172.21.15.32:0/255498399 learned_addr learned my addr 172.21.15.32:0/255498399 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:54:39.558 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.553+0000 7fc8c8a00640 1 --2- 172.21.15.32:0/255498399 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x561f76feb360 0x561f76f7d700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:39.558 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.553+0000 7fc8c3e00640 1 -- 172.21.15.32:0/255498399 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x561f77008cb0 msgr2=0x561f7709cdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:54:39.558 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.553+0000 7fc8c9400640 1 --2- 172.21.15.32:0/255498399 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x561f77008cb0 0x561f7709cdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:39.558 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.553+0000 7fc8c3e00640 1 --2- 172.21.15.32:0/255498399 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x561f77008cb0 0x561f7709cdb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:39.558 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.553+0000 7fc8c3e00640 1 -- 172.21.15.32:0/255498399 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x561f76feb360 msgr2=0x561f76f7d700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:54:39.558 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.553+0000 7fc8c3e00640 1 --2- 172.21.15.32:0/255498399 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x561f76feb360 0x561f76f7d700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:39.558 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.553+0000 7fc8c3e00640 1 -- 172.21.15.32:0/255498399 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561f77000b40 con 0x561f76fecea0 2024-08-03T20:54:39.558 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.553+0000 7fc8c8a00640 1 --2- 172.21.15.32:0/255498399 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x561f76feb360 0x561f76f7d700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-03T20:54:39.559 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.553+0000 7fc8c9400640 1 --2- 172.21.15.32:0/255498399 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x561f77008cb0 0x561f7709cdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-03T20:54:39.559 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.553+0000 7fc8c3e00640 1 --2- 172.21.15.32:0/255498399 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561f76fecea0 0x561f77006a00 secure :-1 s=READY pgs=954 cs=0 l=1 rev1=1 crypto rx=0x7fc8b400b4f0 tx=0x7fc8b400b9b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fac7b626aff46fa6 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:54:39.559 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.553+0000 7fc8c2a00640 1 -- 172.21.15.32:0/255498399 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc8b40099d0 con 0x561f76fecea0 2024-08-03T20:54:39.559 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.553+0000 7fc8c2a00640 1 -- 172.21.15.32:0/255498399 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fc8b4009b70 con 0x561f76fecea0 2024-08-03T20:54:39.559 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.553+0000 7fc8c2a00640 1 -- 172.21.15.32:0/255498399 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc8b4014440 con 0x561f76fecea0 2024-08-03T20:54:39.559 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.554+0000 7fc8cf7c7f40 1 -- 172.21.15.32:0/255498399 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561f76fecea0 msgr2=0x561f77006a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:39.559 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.554+0000 7fc8cf7c7f40 1 --2- 172.21.15.32:0/255498399 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561f76fecea0 0x561f77006a00 secure :-1 s=READY pgs=954 cs=0 l=1 rev1=1 crypto rx=0x7fc8b400b4f0 tx=0x7fc8b400b9b0 comp rx=0 tx=0).stop 2024-08-03T20:54:39.559 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.554+0000 7fc8cf7c7f40 1 -- 172.21.15.32:0/255498399 shutdown_connections 2024-08-03T20:54:39.559 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.554+0000 7fc8cf7c7f40 1 --2- 172.21.15.32:0/255498399 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x561f77008cb0 0x561f7709cdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:39.559 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.554+0000 7fc8cf7c7f40 1 --2- 172.21.15.32:0/255498399 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561f76fecea0 0x561f77006a00 unknown :-1 s=CLOSED pgs=954 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:39.559 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.554+0000 7fc8cf7c7f40 1 --2- 172.21.15.32:0/255498399 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x561f76feb360 0x561f76f7d700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:39.559 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.554+0000 7fc8cf7c7f40 1 -- 172.21.15.32:0/255498399 >> 172.21.15.32:0/255498399 conn(0x561f76e54ac0 msgr2=0x561f76f7aff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:54:39.559 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.554+0000 7fc8cf7c7f40 1 -- 172.21.15.32:0/255498399 shutdown_connections 2024-08-03T20:54:39.560 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.554+0000 7fc8cf7c7f40 1 -- 172.21.15.32:0/255498399 wait complete. 2024-08-03T20:54:39.560 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.555+0000 7fc8cf7c7f40 1 Processor -- start 2024-08-03T20:54:39.560 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.555+0000 7fc8cf7c7f40 1 -- start start 2024-08-03T20:54:39.562 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.557+0000 7fc8cf7c7f40 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x561f76feb360 0x561f77005050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:39.562 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.557+0000 7fc8cf7c7f40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x561f76fecea0 0x561f76fe8780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:39.562 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.557+0000 7fc8cf7c7f40 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561f77008cb0 0x561f77003340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:39.562 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.557+0000 7fc8cf7c7f40 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x561f77000de0 con 0x561f77008cb0 2024-08-03T20:54:39.563 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.557+0000 7fc8cf7c7f40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x561f77005810 con 0x561f76fecea0 2024-08-03T20:54:39.563 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.557+0000 7fc8cf7c7f40 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x561f76f79560 con 0x561f76feb360 2024-08-03T20:54:39.563 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.557+0000 7fc8c9400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561f77008cb0 0x561f77003340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:39.563 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.557+0000 7fc8c9400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561f77008cb0 0x561f77003340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:40302/0 (socket says 172.21.15.32:40302) 2024-08-03T20:54:39.563 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.557+0000 7fc8c9400640 1 -- 172.21.15.32:0/1429568924 learned_addr learned my addr 172.21.15.32:0/1429568924 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:54:39.563 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.557+0000 7fc8c3e00640 1 --2- 172.21.15.32:0/1429568924 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x561f76fecea0 0x561f76fe8780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:39.563 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.557+0000 7fc8c8a00640 1 --2- 172.21.15.32:0/1429568924 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x561f76feb360 0x561f77005050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:39.563 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.558+0000 7fc8c9400640 1 -- 172.21.15.32:0/1429568924 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x561f76feb360 msgr2=0x561f77005050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:39.563 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.558+0000 7fc8c9400640 1 --2- 172.21.15.32:0/1429568924 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x561f76feb360 0x561f77005050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:39.563 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.558+0000 7fc8c9400640 1 -- 172.21.15.32:0/1429568924 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x561f76fecea0 msgr2=0x561f76fe8780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:39.564 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.558+0000 7fc8c9400640 1 --2- 172.21.15.32:0/1429568924 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x561f76fecea0 0x561f76fe8780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:39.564 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.558+0000 7fc8c9400640 1 -- 172.21.15.32:0/1429568924 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561f77002cb0 con 0x561f77008cb0 2024-08-03T20:54:39.564 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.558+0000 7fc8c8a00640 1 --2- 172.21.15.32:0/1429568924 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x561f76feb360 0x561f77005050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-03T20:54:39.564 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.558+0000 7fc8c9400640 1 --2- 172.21.15.32:0/1429568924 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561f77008cb0 0x561f77003340 secure :-1 s=READY pgs=955 cs=0 l=1 rev1=1 crypto rx=0x7fc8c4006fb0 tx=0x7fc8c400c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:54:39.564 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.558+0000 7fc8c1600640 1 -- 172.21.15.32:0/1429568924 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc8c4017070 con 0x561f77008cb0 2024-08-03T20:54:39.564 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.558+0000 7fc8c1600640 1 -- 172.21.15.32:0/1429568924 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fc8c4001500 con 0x561f77008cb0 2024-08-03T20:54:39.565 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.558+0000 7fc8cf7c7f40 1 -- 172.21.15.32:0/1429568924 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x561f76f7d8f0 con 0x561f77008cb0 2024-08-03T20:54:39.565 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.558+0000 7fc8c1600640 1 -- 172.21.15.32:0/1429568924 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc8c4001950 con 0x561f77008cb0 2024-08-03T20:54:39.565 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.558+0000 7fc8cf7c7f40 1 -- 172.21.15.32:0/1429568924 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561f77006510 con 0x561f77008cb0 2024-08-03T20:54:39.565 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.561+0000 7fc8c1600640 1 -- 172.21.15.32:0/1429568924 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7fc8c401a070 con 0x561f77008cb0 2024-08-03T20:54:39.567 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.561+0000 7fc8c1600640 1 --2- 172.21.15.32:0/1429568924 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fc8a80759f0 0x7fc8a8077f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:39.567 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.562+0000 7fc8c1600640 1 -- 172.21.15.32:0/1429568924 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 5795+0+0 (secure 0 0 0) 0x7fc8c4096ea0 con 0x561f77008cb0 2024-08-03T20:54:39.567 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.562+0000 7fc8cf7c7f40 1 --2- 172.21.15.32:0/1429568924 >> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] conn(0x561f76f76130 0x561f770078d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:39.567 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.562+0000 7fc8cf7c7f40 1 -- 172.21.15.32:0/1429568924 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:1 12.14 12:292cf221:::200.00000000:head [omap-get-header] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84) -- 0x561f770a46c0 con 0x561f76f76130 2024-08-03T20:54:39.567 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.562+0000 7fc8c8a00640 1 --2- 172.21.15.32:0/1429568924 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fc8a80759f0 0x7fc8a8077f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:39.567 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.562+0000 7fc8c3e00640 1 --2- 172.21.15.32:0/1429568924 >> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] conn(0x561f76f76130 0x561f770078d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:39.567 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.563+0000 7fc8c8a00640 1 --2- 172.21.15.32:0/1429568924 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fc8a80759f0 0x7fc8a8077f00 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7fc8bc055000 tx=0x7fc8bc058040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:54:39.567 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.563+0000 7fc8c3e00640 1 --2- 172.21.15.32:0/1429568924 >> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] conn(0x561f76f76130 0x561f770078d0 crc :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:54:39.567 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.564+0000 7fc8c3e00640 1 -- 172.21.15.32:0/1429568924 <== osd.5 v2:172.21.15.118:6810/2724891770 1 ==== osd_op_reply(1 200.00000000 [omap-get-header] v0'0 uv19 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fc8b40099f0 con 0x561f76f76130 2024-08-03T20:54:39.568 INFO:teuthology.orchestra.run.smithi032.stdout:header (0 bytes) : 2024-08-03T20:54:39.568 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-03T20:54:39.569 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.564+0000 7fc8cf7c7f40 1 -- 172.21.15.32:0/1429568924 >> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] conn(0x561f76f76130 msgr2=0x561f770078d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:39.569 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.564+0000 7fc8cf7c7f40 1 --2- 172.21.15.32:0/1429568924 >> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] conn(0x561f76f76130 0x561f770078d0 crc :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:39.569 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.564+0000 7fc8cf7c7f40 1 -- 172.21.15.32:0/1429568924 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fc8a80759f0 msgr2=0x7fc8a8077f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:39.569 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.564+0000 7fc8cf7c7f40 1 --2- 172.21.15.32:0/1429568924 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fc8a80759f0 0x7fc8a8077f00 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7fc8bc055000 tx=0x7fc8bc058040 comp rx=0 tx=0).stop 2024-08-03T20:54:39.570 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.564+0000 7fc8cf7c7f40 1 -- 172.21.15.32:0/1429568924 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561f77008cb0 msgr2=0x561f77003340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:39.570 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.564+0000 7fc8cf7c7f40 1 --2- 172.21.15.32:0/1429568924 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561f77008cb0 0x561f77003340 secure :-1 s=READY pgs=955 cs=0 l=1 rev1=1 crypto rx=0x7fc8c4006fb0 tx=0x7fc8c400c520 comp rx=0 tx=0).stop 2024-08-03T20:54:39.570 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.564+0000 7fc8c9400640 1 -- 172.21.15.32:0/1429568924 reap_dead start 2024-08-03T20:54:39.570 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.564+0000 7fc8cf7c7f40 1 -- 172.21.15.32:0/1429568924 shutdown_connections 2024-08-03T20:54:39.570 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.564+0000 7fc8cf7c7f40 1 -- 172.21.15.32:0/1429568924 >> 172.21.15.32:0/1429568924 conn(0x561f76e54ac0 msgr2=0x561f76feaa40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:54:39.570 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.564+0000 7fc8cf7c7f40 1 -- 172.21.15.32:0/1429568924 shutdown_connections 2024-08-03T20:54:39.570 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.564+0000 7fc8cf7c7f40 1 -- 172.21.15.32:0/1429568924 wait complete. 2024-08-03T20:54:39.576 DEBUG:teuthology.orchestra.run.smithi032:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata getomapheader 200.00000001 2024-08-03T20:54:39.719 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.718+0000 7fc4f5726f40 1 Processor -- start 2024-08-03T20:54:39.720 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.718+0000 7fc4f5726f40 1 -- start start 2024-08-03T20:54:39.723 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.719+0000 7fc4f5726f40 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55f1529ba360 0x55f1529d06b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:39.723 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.719+0000 7fc4f5726f40 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55f1529bbea0 0x55f152a62a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:39.724 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.719+0000 7fc4f5726f40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55f1529d7cb0 0x55f152a65460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:39.724 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.719+0000 7fc4f5726f40 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55f1529d80c0 con 0x55f1529bbea0 2024-08-03T20:54:39.724 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.719+0000 7fc4f5726f40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x55f152851e30 con 0x55f1529d7cb0 2024-08-03T20:54:39.724 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.719+0000 7fc4f5726f40 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x55f15294a840 con 0x55f1529ba360 2024-08-03T20:54:39.724 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.719+0000 7fc4ede00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55f1529bbea0 0x55f152a62a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:39.724 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.719+0000 7fc4ede00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55f1529bbea0 0x55f152a62a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:40312/0 (socket says 172.21.15.32:40312) 2024-08-03T20:54:39.724 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.719+0000 7fc4ede00640 1 -- 172.21.15.32:0/1784585075 learned_addr learned my addr 172.21.15.32:0/1784585075 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:54:39.724 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.720+0000 7fc4ee800640 1 --2- 172.21.15.32:0/1784585075 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55f1529ba360 0x55f1529d06b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:39.725 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.720+0000 7fc4ef200640 1 --2- 172.21.15.32:0/1784585075 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55f1529d7cb0 0x55f152a65460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:39.725 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.720+0000 7fc4ede00640 1 -- 172.21.15.32:0/1784585075 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55f1529ba360 msgr2=0x55f1529d06b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:54:39.725 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.720+0000 7fc4ede00640 1 --2- 172.21.15.32:0/1784585075 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55f1529ba360 0x55f1529d06b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:39.725 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.720+0000 7fc4ede00640 1 -- 172.21.15.32:0/1784585075 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55f1529d7cb0 msgr2=0x55f152a65460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:54:39.725 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.720+0000 7fc4ede00640 1 --2- 172.21.15.32:0/1784585075 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55f1529d7cb0 0x55f152a65460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:39.725 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.720+0000 7fc4ede00640 1 -- 172.21.15.32:0/1784585075 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f15282af10 con 0x55f1529bbea0 2024-08-03T20:54:39.725 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.720+0000 7fc4ee800640 1 --2- 172.21.15.32:0/1784585075 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55f1529ba360 0x55f1529d06b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-03T20:54:39.726 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.720+0000 7fc4ede00640 1 --2- 172.21.15.32:0/1784585075 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55f1529bbea0 0x55f152a62a80 secure :-1 s=READY pgs=957 cs=0 l=1 rev1=1 crypto rx=0x7fc4dc046500 tx=0x7fc4dc002fb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4a9a697d149ec841 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:54:39.726 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.720+0000 7fc4eca00640 1 -- 172.21.15.32:0/1784585075 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4dc04d860 con 0x55f1529bbea0 2024-08-03T20:54:39.726 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.720+0000 7fc4eca00640 1 -- 172.21.15.32:0/1784585075 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fc4dc04dcf0 con 0x55f1529bbea0 2024-08-03T20:54:39.726 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.720+0000 7fc4eca00640 1 -- 172.21.15.32:0/1784585075 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4dc0566d0 con 0x55f1529bbea0 2024-08-03T20:54:39.726 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.721+0000 7fc4f5726f40 1 -- 172.21.15.32:0/1784585075 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55f1529bbea0 msgr2=0x55f152a62a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:39.726 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.721+0000 7fc4f5726f40 1 --2- 172.21.15.32:0/1784585075 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55f1529bbea0 0x55f152a62a80 secure :-1 s=READY pgs=957 cs=0 l=1 rev1=1 crypto rx=0x7fc4dc046500 tx=0x7fc4dc002fb0 comp rx=0 tx=0).stop 2024-08-03T20:54:39.726 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.721+0000 7fc4f5726f40 1 -- 172.21.15.32:0/1784585075 shutdown_connections 2024-08-03T20:54:39.726 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.721+0000 7fc4f5726f40 1 --2- 172.21.15.32:0/1784585075 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55f1529d7cb0 0x55f152a65460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:39.726 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.721+0000 7fc4f5726f40 1 --2- 172.21.15.32:0/1784585075 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55f1529bbea0 0x55f152a62a80 unknown :-1 s=CLOSED pgs=957 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:39.726 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.721+0000 7fc4f5726f40 1 --2- 172.21.15.32:0/1784585075 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55f1529ba360 0x55f1529d06b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:39.726 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.721+0000 7fc4f5726f40 1 -- 172.21.15.32:0/1784585075 >> 172.21.15.32:0/1784585075 conn(0x55f152823ac0 msgr2=0x55f152a64d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:54:39.727 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.721+0000 7fc4f5726f40 1 -- 172.21.15.32:0/1784585075 shutdown_connections 2024-08-03T20:54:39.727 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.721+0000 7fc4f5726f40 1 -- 172.21.15.32:0/1784585075 wait complete. 2024-08-03T20:54:39.729 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.722+0000 7fc4f5726f40 1 Processor -- start 2024-08-03T20:54:39.729 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.722+0000 7fc4f5726f40 1 -- start start 2024-08-03T20:54:39.729 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.724+0000 7fc4f5726f40 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55f1529ba360 0x55f152a6fa80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:39.729 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.724+0000 7fc4f5726f40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55f1529bbea0 0x55f152a737c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:39.729 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.724+0000 7fc4f5726f40 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55f1529d7cb0 0x55f152a73d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:39.729 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.724+0000 7fc4f5726f40 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55f15294ce60 con 0x55f1529ba360 2024-08-03T20:54:39.729 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.724+0000 7fc4f5726f40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x55f1529d0bf0 con 0x55f1529bbea0 2024-08-03T20:54:39.729 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.724+0000 7fc4f5726f40 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x55f15294b490 con 0x55f1529d7cb0 2024-08-03T20:54:39.729 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.724+0000 7fc4ee800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55f1529ba360 0x55f152a6fa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:39.729 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.724+0000 7fc4ee800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55f1529ba360 0x55f152a6fa80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:40326/0 (socket says 172.21.15.32:40326) 2024-08-03T20:54:39.729 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.724+0000 7fc4ee800640 1 -- 172.21.15.32:0/99193575 learned_addr learned my addr 172.21.15.32:0/99193575 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:54:39.730 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.724+0000 7fc4ef200640 1 --2- 172.21.15.32:0/99193575 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55f1529d7cb0 0x55f152a73d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:39.730 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.724+0000 7fc4ede00640 1 --2- 172.21.15.32:0/99193575 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55f1529bbea0 0x55f152a737c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:39.730 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.724+0000 7fc4ee800640 1 -- 172.21.15.32:0/99193575 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55f1529d7cb0 msgr2=0x55f152a73d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:39.730 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.724+0000 7fc4ee800640 1 --2- 172.21.15.32:0/99193575 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55f1529d7cb0 0x55f152a73d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:39.730 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.724+0000 7fc4ee800640 1 -- 172.21.15.32:0/99193575 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55f1529bbea0 msgr2=0x55f152a737c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:39.730 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.724+0000 7fc4ee800640 1 --2- 172.21.15.32:0/99193575 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55f1529bbea0 0x55f152a737c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:39.733 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.725+0000 7fc4ee800640 1 -- 172.21.15.32:0/99193575 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f152a61fa0 con 0x55f1529ba360 2024-08-03T20:54:39.733 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.725+0000 7fc4ede00640 1 --2- 172.21.15.32:0/99193575 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55f1529bbea0 0x55f152a737c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-03T20:54:39.734 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.725+0000 7fc4ee800640 1 --2- 172.21.15.32:0/99193575 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55f1529ba360 0x55f152a6fa80 secure :-1 s=READY pgs=958 cs=0 l=1 rev1=1 crypto rx=0x7fc4e000a650 tx=0x7fc4e000ab10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:54:39.734 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.725+0000 7fc4e7400640 1 -- 172.21.15.32:0/99193575 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4e001a070 con 0x55f1529ba360 2024-08-03T20:54:39.734 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.725+0000 7fc4e7400640 1 -- 172.21.15.32:0/99193575 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fc4e000ae50 con 0x55f1529ba360 2024-08-03T20:54:39.734 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.725+0000 7fc4e7400640 1 -- 172.21.15.32:0/99193575 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4e0008bf0 con 0x55f1529ba360 2024-08-03T20:54:39.734 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.725+0000 7fc4f5726f40 1 -- 172.21.15.32:0/99193575 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x55f152a62240 con 0x55f1529ba360 2024-08-03T20:54:39.734 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.725+0000 7fc4f5726f40 1 -- 172.21.15.32:0/99193575 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f1529490f0 con 0x55f1529ba360 2024-08-03T20:54:39.734 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.727+0000 7fc4e7400640 1 -- 172.21.15.32:0/99193575 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7fc4e001d070 con 0x55f1529ba360 2024-08-03T20:54:39.735 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.728+0000 7fc4e7400640 1 --2- 172.21.15.32:0/99193575 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fc4d00759f0 0x7fc4d0077f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:39.735 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.728+0000 7fc4e7400640 1 -- 172.21.15.32:0/99193575 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 5795+0+0 (secure 0 0 0) 0x7fc4e0096ff0 con 0x55f1529ba360 2024-08-03T20:54:39.735 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.729+0000 7fc4f5726f40 1 --2- 172.21.15.32:0/99193575 >> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] conn(0x55f1529037c0 0x55f1529022b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:39.736 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.729+0000 7fc4f5726f40 1 -- 172.21.15.32:0/99193575 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:1 12.14 12:2e2fa760:::200.00000001:head [omap-get-header] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84) -- 0x55f1529027f0 con 0x55f1529037c0 2024-08-03T20:54:39.736 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.729+0000 7fc4ede00640 1 --2- 172.21.15.32:0/99193575 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fc4d00759f0 0x7fc4d0077f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:39.736 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.729+0000 7fc4ef200640 1 --2- 172.21.15.32:0/99193575 >> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] conn(0x55f1529037c0 0x55f1529022b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:39.736 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.729+0000 7fc4ede00640 1 --2- 172.21.15.32:0/99193575 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fc4d00759f0 0x7fc4d0077f00 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x55f1529b9d20 tx=0x7fc4dc004940 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:54:39.736 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.730+0000 7fc4ef200640 1 --2- 172.21.15.32:0/99193575 >> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] conn(0x55f1529037c0 0x55f1529022b0 crc :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:54:39.736 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.730+0000 7fc4ef200640 1 -- 172.21.15.32:0/99193575 <== osd.5 v2:172.21.15.118:6810/2724891770 1 ==== osd_op_reply(1 200.00000001 [omap-get-header] v0'0 uv16 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fc4e800b040 con 0x55f1529037c0 2024-08-03T20:54:39.736 INFO:teuthology.orchestra.run.smithi032.stdout:header (0 bytes) : 2024-08-03T20:54:39.736 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-03T20:54:39.737 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.730+0000 7fc4f5726f40 1 -- 172.21.15.32:0/99193575 >> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] conn(0x55f1529037c0 msgr2=0x55f1529022b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:39.737 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.730+0000 7fc4f5726f40 1 --2- 172.21.15.32:0/99193575 >> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] conn(0x55f1529037c0 0x55f1529022b0 crc :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:39.737 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.730+0000 7fc4f5726f40 1 -- 172.21.15.32:0/99193575 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fc4d00759f0 msgr2=0x7fc4d0077f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:39.737 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.730+0000 7fc4f5726f40 1 --2- 172.21.15.32:0/99193575 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fc4d00759f0 0x7fc4d0077f00 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x55f1529b9d20 tx=0x7fc4dc004940 comp rx=0 tx=0).stop 2024-08-03T20:54:39.737 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.730+0000 7fc4f5726f40 1 -- 172.21.15.32:0/99193575 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55f1529ba360 msgr2=0x55f152a6fa80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:39.738 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.730+0000 7fc4f5726f40 1 --2- 172.21.15.32:0/99193575 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55f1529ba360 0x55f152a6fa80 secure :-1 s=READY pgs=958 cs=0 l=1 rev1=1 crypto rx=0x7fc4e000a650 tx=0x7fc4e000ab10 comp rx=0 tx=0).stop 2024-08-03T20:54:39.738 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.730+0000 7fc4ef200640 1 -- 172.21.15.32:0/99193575 reap_dead start 2024-08-03T20:54:39.738 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.731+0000 7fc4f5726f40 1 -- 172.21.15.32:0/99193575 shutdown_connections 2024-08-03T20:54:39.738 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.731+0000 7fc4f5726f40 1 -- 172.21.15.32:0/99193575 >> 172.21.15.32:0/99193575 conn(0x55f152823ac0 msgr2=0x55f1529b21f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:54:39.738 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.731+0000 7fc4f5726f40 1 -- 172.21.15.32:0/99193575 shutdown_connections 2024-08-03T20:54:39.738 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.731+0000 7fc4f5726f40 1 -- 172.21.15.32:0/99193575 wait complete. 2024-08-03T20:54:39.743 DEBUG:teuthology.orchestra.run.smithi032:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata getomapheader mds0_openfiles.0 2024-08-03T20:54:39.887 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.886+0000 7fcdb23c0f40 1 Processor -- start 2024-08-03T20:54:39.887 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.886+0000 7fcdb23c0f40 1 -- start start 2024-08-03T20:54:39.889 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.887+0000 7fcdb23c0f40 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcd90002320 0x5589facaba80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:39.889 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.887+0000 7fcdb23c0f40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5589fac3d2a0 0x5589fac3d6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:39.889 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.887+0000 7fcdb23c0f40 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x5589fac3a480 0x5589fac3a890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:39.890 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.887+0000 7fcdb23c0f40 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x5589faba0bf0 con 0x7fcd90002320 2024-08-03T20:54:39.890 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.887+0000 7fcdb23c0f40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x5589fabad6b0 con 0x5589fac3d2a0 2024-08-03T20:54:39.890 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.887+0000 7fcdb23c0f40 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x5589fab42e30 con 0x5589fac3a480 2024-08-03T20:54:39.890 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.887+0000 7fcdab600640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcd90002320 0x5589facaba80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:39.890 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.887+0000 7fcdab600640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcd90002320 0x5589facaba80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:36592/0 (socket says 172.21.15.32:36592) 2024-08-03T20:54:39.890 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.887+0000 7fcdab600640 1 -- 172.21.15.32:0/813546105 learned_addr learned my addr 172.21.15.32:0/813546105 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:54:39.890 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.888+0000 7fcdaac00640 1 --2- 172.21.15.32:0/813546105 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5589fac3d2a0 0x5589fac3d6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:39.890 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.888+0000 7fcdac000640 1 --2- 172.21.15.32:0/813546105 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x5589fac3a480 0x5589fac3a890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:39.890 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.888+0000 7fcdab600640 1 -- 172.21.15.32:0/813546105 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x5589fac3a480 msgr2=0x5589fac3a890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:54:39.890 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.888+0000 7fcdab600640 1 --2- 172.21.15.32:0/813546105 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x5589fac3a480 0x5589fac3a890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:39.890 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.888+0000 7fcdab600640 1 -- 172.21.15.32:0/813546105 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5589fac3d2a0 msgr2=0x5589fac3d6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:54:39.890 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.888+0000 7fcdab600640 1 --2- 172.21.15.32:0/813546105 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5589fac3d2a0 0x5589fac3d6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:39.890 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.888+0000 7fcdab600640 1 -- 172.21.15.32:0/813546105 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5589facaed00 con 0x7fcd90002320 2024-08-03T20:54:39.891 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.888+0000 7fcdaac00640 1 --2- 172.21.15.32:0/813546105 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5589fac3d2a0 0x5589fac3d6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-03T20:54:39.891 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.888+0000 7fcdac000640 1 --2- 172.21.15.32:0/813546105 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x5589fac3a480 0x5589fac3a890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-03T20:54:39.891 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.888+0000 7fcdab600640 1 --2- 172.21.15.32:0/813546105 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcd90002320 0x5589facaba80 secure :-1 s=READY pgs=961 cs=0 l=1 rev1=1 crypto rx=0x7fcd9c025880 tx=0x7fcd9c001120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9219a603db606a90 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:54:39.892 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.888+0000 7fcda9800640 1 -- 172.21.15.32:0/813546105 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd9c063070 con 0x7fcd90002320 2024-08-03T20:54:39.892 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.889+0000 7fcda9800640 1 -- 172.21.15.32:0/813546105 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fcd9c053b10 con 0x7fcd90002320 2024-08-03T20:54:39.892 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.889+0000 7fcda9800640 1 -- 172.21.15.32:0/813546105 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd9c058040 con 0x7fcd90002320 2024-08-03T20:54:39.892 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.889+0000 7fcdb23c0f40 1 -- 172.21.15.32:0/813546105 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcd90002320 msgr2=0x5589facaba80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:39.892 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.889+0000 7fcdb23c0f40 1 --2- 172.21.15.32:0/813546105 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcd90002320 0x5589facaba80 secure :-1 s=READY pgs=961 cs=0 l=1 rev1=1 crypto rx=0x7fcd9c025880 tx=0x7fcd9c001120 comp rx=0 tx=0).stop 2024-08-03T20:54:39.892 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.889+0000 7fcdb23c0f40 1 -- 172.21.15.32:0/813546105 shutdown_connections 2024-08-03T20:54:39.892 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.889+0000 7fcdb23c0f40 1 --2- 172.21.15.32:0/813546105 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5589fac3d2a0 0x5589fac3d6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:39.892 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.889+0000 7fcdb23c0f40 1 --2- 172.21.15.32:0/813546105 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x5589fac3a480 0x5589fac3a890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:39.893 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.889+0000 7fcdb23c0f40 1 --2- 172.21.15.32:0/813546105 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcd90002320 0x5589facaba80 unknown :-1 s=CLOSED pgs=961 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:39.893 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.889+0000 7fcdb23c0f40 1 -- 172.21.15.32:0/813546105 >> 172.21.15.32:0/813546105 conn(0x5589fab14ac0 msgr2=0x5589fac37940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:54:39.893 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.889+0000 7fcdb23c0f40 1 -- 172.21.15.32:0/813546105 shutdown_connections 2024-08-03T20:54:39.893 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.889+0000 7fcdb23c0f40 1 -- 172.21.15.32:0/813546105 wait complete. 2024-08-03T20:54:39.893 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.890+0000 7fcdb23c0f40 1 Processor -- start 2024-08-03T20:54:39.893 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.891+0000 7fcdb23c0f40 1 -- start start 2024-08-03T20:54:39.895 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.892+0000 7fcdb23c0f40 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x5589fac3a480 0x5589fac9f270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:39.895 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.892+0000 7fcdb23c0f40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5589fac3d2a0 0x5589fad62610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:39.895 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.892+0000 7fcdb23c0f40 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcd90002320 0x5589fac3a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:39.895 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.892+0000 7fcdb23c0f40 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x5589fac3dbf0 con 0x7fcd90002320 2024-08-03T20:54:39.895 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.892+0000 7fcdb23c0f40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x5589facae3e0 con 0x5589fac3d2a0 2024-08-03T20:54:39.895 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.892+0000 7fcdb23c0f40 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x5589facae9c0 con 0x5589fac3a480 2024-08-03T20:54:39.895 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.892+0000 7fcdac000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcd90002320 0x5589fac3a890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:39.895 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.892+0000 7fcdac000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcd90002320 0x5589fac3a890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:36596/0 (socket says 172.21.15.32:36596) 2024-08-03T20:54:39.895 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.892+0000 7fcdac000640 1 -- 172.21.15.32:0/977018092 learned_addr learned my addr 172.21.15.32:0/977018092 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:54:39.895 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.892+0000 7fcdaac00640 1 --2- 172.21.15.32:0/977018092 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5589fac3d2a0 0x5589fad62610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:39.896 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.892+0000 7fcdab600640 1 --2- 172.21.15.32:0/977018092 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x5589fac3a480 0x5589fac9f270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:39.896 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.893+0000 7fcdac000640 1 -- 172.21.15.32:0/977018092 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x5589fac3a480 msgr2=0x5589fac9f270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:39.896 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.893+0000 7fcdac000640 1 --2- 172.21.15.32:0/977018092 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x5589fac3a480 0x5589fac9f270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:39.896 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.893+0000 7fcdac000640 1 -- 172.21.15.32:0/977018092 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5589fac3d2a0 msgr2=0x5589fad62610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:39.896 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.893+0000 7fcdac000640 1 --2- 172.21.15.32:0/977018092 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5589fac3d2a0 0x5589fad62610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:39.896 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.893+0000 7fcdac000640 1 -- 172.21.15.32:0/977018092 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5589faca9070 con 0x7fcd90002320 2024-08-03T20:54:39.896 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.893+0000 7fcdac000640 1 --2- 172.21.15.32:0/977018092 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcd90002320 0x5589fac3a890 secure :-1 s=READY pgs=962 cs=0 l=1 rev1=1 crypto rx=0x5589fac410c0 tx=0x7fcda4005ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:54:39.896 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.893+0000 7fcda3e00640 1 -- 172.21.15.32:0/977018092 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcda4019070 con 0x7fcd90002320 2024-08-03T20:54:39.896 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.893+0000 7fcda3e00640 1 -- 172.21.15.32:0/977018092 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fcda40039f0 con 0x7fcd90002320 2024-08-03T20:54:39.896 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.893+0000 7fcda3e00640 1 -- 172.21.15.32:0/977018092 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcda4001a70 con 0x7fcd90002320 2024-08-03T20:54:39.896 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.893+0000 7fcdb23c0f40 1 -- 172.21.15.32:0/977018092 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x5589fac36dc0 con 0x7fcd90002320 2024-08-03T20:54:39.896 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.893+0000 7fcdb23c0f40 1 -- 172.21.15.32:0/977018092 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5589fad630e0 con 0x7fcd90002320 2024-08-03T20:54:39.897 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.896+0000 7fcda3e00640 1 -- 172.21.15.32:0/977018092 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7fcda401c070 con 0x7fcd90002320 2024-08-03T20:54:39.898 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.896+0000 7fcda3e00640 1 --2- 172.21.15.32:0/977018092 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fcd8c075a20 0x7fcd8c077f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:39.898 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.897+0000 7fcda3e00640 1 -- 172.21.15.32:0/977018092 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 5795+0+0 (secure 0 0 0) 0x7fcda4096f60 con 0x7fcd90002320 2024-08-03T20:54:39.899 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.897+0000 7fcdb23c0f40 1 --2- 172.21.15.32:0/977018092 >> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] conn(0x5589faca9d10 0x5589fad651e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:39.899 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.897+0000 7fcdb23c0f40 1 -- 172.21.15.32:0/977018092 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:1 12.6 12:654134d2:::mds0_openfiles.0:head [omap-get-header] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84) -- 0x5589faca3020 con 0x5589faca9d10 2024-08-03T20:54:39.899 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.897+0000 7fcdab600640 1 --2- 172.21.15.32:0/977018092 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fcd8c075a20 0x7fcd8c077f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:39.899 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.897+0000 7fcdaac00640 1 --2- 172.21.15.32:0/977018092 >> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] conn(0x5589faca9d10 0x5589fad651e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:39.899 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.898+0000 7fcdaac00640 1 --2- 172.21.15.32:0/977018092 >> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] conn(0x5589faca9d10 0x5589fad651e0 crc :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:54:39.900 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.898+0000 7fcdab600640 1 --2- 172.21.15.32:0/977018092 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fcd8c075a20 0x7fcd8c077f30 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7fcd9c064000 tx=0x7fcd9c069040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:54:39.900 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.898+0000 7fcdaac00640 1 -- 172.21.15.32:0/977018092 <== osd.1 v2:172.21.15.32:6824/1551241759 1 ==== osd_op_reply(1 mds0_openfiles.0 [omap-get-header out=42b] v0'0 uv4 ondisk = 0) ==== 160+0+42 (crc 0 0 0) 0x7fcd9800c140 con 0x5589faca9d10 2024-08-03T20:54:39.900 INFO:teuthology.orchestra.run.smithi032.stdout:header (42 bytes) : 2024-08-03T20:54:39.900 INFO:teuthology.orchestra.run.smithi032.stdout:00000000 13 00 00 00 63 65 70 68 20 66 73 20 76 6f 6c 75 |....ceph fs volu| 2024-08-03T20:54:39.900 INFO:teuthology.orchestra.run.smithi032.stdout:00000010 6d 65 20 76 30 31 31 01 01 0d 00 00 00 04 00 00 |me v011.........| 2024-08-03T20:54:39.900 INFO:teuthology.orchestra.run.smithi032.stdout:00000020 00 00 00 00 00 01 00 00 00 00 |..........| 2024-08-03T20:54:39.900 INFO:teuthology.orchestra.run.smithi032.stdout:0000002a 2024-08-03T20:54:39.900 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-03T20:54:39.901 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.898+0000 7fcdb23c0f40 1 -- 172.21.15.32:0/977018092 >> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] conn(0x5589faca9d10 msgr2=0x5589fad651e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:39.901 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.898+0000 7fcdb23c0f40 1 --2- 172.21.15.32:0/977018092 >> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] conn(0x5589faca9d10 0x5589fad651e0 crc :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:39.901 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.898+0000 7fcdb23c0f40 1 -- 172.21.15.32:0/977018092 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fcd8c075a20 msgr2=0x7fcd8c077f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:39.902 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.898+0000 7fcdb23c0f40 1 --2- 172.21.15.32:0/977018092 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fcd8c075a20 0x7fcd8c077f30 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7fcd9c064000 tx=0x7fcd9c069040 comp rx=0 tx=0).stop 2024-08-03T20:54:39.902 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.898+0000 7fcdb23c0f40 1 -- 172.21.15.32:0/977018092 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcd90002320 msgr2=0x5589fac3a890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:39.902 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.898+0000 7fcdb23c0f40 1 --2- 172.21.15.32:0/977018092 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcd90002320 0x5589fac3a890 secure :-1 s=READY pgs=962 cs=0 l=1 rev1=1 crypto rx=0x5589fac410c0 tx=0x7fcda4005ba0 comp rx=0 tx=0).stop 2024-08-03T20:54:39.902 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.898+0000 7fcdac000640 1 -- 172.21.15.32:0/977018092 reap_dead start 2024-08-03T20:54:39.902 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.899+0000 7fcdb23c0f40 1 -- 172.21.15.32:0/977018092 shutdown_connections 2024-08-03T20:54:39.902 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.899+0000 7fcdb23c0f40 1 -- 172.21.15.32:0/977018092 >> 172.21.15.32:0/977018092 conn(0x5589fab14ac0 msgr2=0x5589fac37940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:54:39.902 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.899+0000 7fcdb23c0f40 1 -- 172.21.15.32:0/977018092 shutdown_connections 2024-08-03T20:54:39.902 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:39.899+0000 7fcdb23c0f40 1 -- 172.21.15.32:0/977018092 wait complete. 2024-08-03T20:54:39.911 DEBUG:teuthology.orchestra.run.smithi032:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata getomapheader 500.00000000 2024-08-03T20:54:40.055 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.053+0000 7ffa832b1f40 1 Processor -- start 2024-08-03T20:54:40.055 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.054+0000 7ffa832b1f40 1 -- start start 2024-08-03T20:54:40.057 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.055+0000 7ffa832b1f40 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558171677360 0x5581715c0620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:40.057 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.055+0000 7ffa832b1f40 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x558171678ea0 0x558171690b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:40.058 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.055+0000 7ffa832b1f40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x558171694cb0 0x558171691040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:40.058 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.055+0000 7ffa832b1f40 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55817150ee30 con 0x558171677360 2024-08-03T20:54:40.058 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.055+0000 7ffa832b1f40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x5581716950c0 con 0x558171694cb0 2024-08-03T20:54:40.058 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.055+0000 7ffa832b1f40 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x558171692ca0 con 0x558171678ea0 2024-08-03T20:54:40.058 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.055+0000 7ffa7c400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558171677360 0x5581715c0620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:40.058 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.055+0000 7ffa7c400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558171677360 0x5581715c0620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:36610/0 (socket says 172.21.15.32:36610) 2024-08-03T20:54:40.058 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.055+0000 7ffa7c400640 1 -- 172.21.15.32:0/501628672 learned_addr learned my addr 172.21.15.32:0/501628672 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:54:40.058 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.055+0000 7ffa7ce00640 1 --2- 172.21.15.32:0/501628672 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x558171694cb0 0x558171691040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:40.058 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.055+0000 7ffa7ba00640 1 --2- 172.21.15.32:0/501628672 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x558171678ea0 0x558171690b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:40.058 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.056+0000 7ffa7c400640 1 -- 172.21.15.32:0/501628672 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x558171678ea0 msgr2=0x558171690b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:54:40.058 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.056+0000 7ffa7c400640 1 --2- 172.21.15.32:0/501628672 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x558171678ea0 0x558171690b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:40.058 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.056+0000 7ffa7c400640 1 -- 172.21.15.32:0/501628672 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x558171694cb0 msgr2=0x558171691040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:54:40.059 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.056+0000 7ffa7c400640 1 --2- 172.21.15.32:0/501628672 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x558171694cb0 0x558171691040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:40.059 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.056+0000 7ffa7c400640 1 -- 172.21.15.32:0/501628672 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5581714e7f10 con 0x558171677360 2024-08-03T20:54:40.059 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.056+0000 7ffa7c400640 1 --2- 172.21.15.32:0/501628672 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558171677360 0x5581715c0620 secure :-1 s=READY pgs=964 cs=0 l=1 rev1=1 crypto rx=0x7ffa6c00c500 tx=0x7ffa6c00c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a41529b3c2779de2 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:54:40.059 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.056+0000 7ffa7a600640 1 -- 172.21.15.32:0/501628672 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa6c00a9e0 con 0x558171677360 2024-08-03T20:54:40.059 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.056+0000 7ffa7a600640 1 -- 172.21.15.32:0/501628672 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7ffa6c00ab80 con 0x558171677360 2024-08-03T20:54:40.059 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.056+0000 7ffa7a600640 1 -- 172.21.15.32:0/501628672 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa6c015440 con 0x558171677360 2024-08-03T20:54:40.060 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.057+0000 7ffa832b1f40 1 -- 172.21.15.32:0/501628672 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558171677360 msgr2=0x5581715c0620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:40.060 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.057+0000 7ffa832b1f40 1 --2- 172.21.15.32:0/501628672 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558171677360 0x5581715c0620 secure :-1 s=READY pgs=964 cs=0 l=1 rev1=1 crypto rx=0x7ffa6c00c500 tx=0x7ffa6c00c9c0 comp rx=0 tx=0).stop 2024-08-03T20:54:40.060 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.057+0000 7ffa832b1f40 1 -- 172.21.15.32:0/501628672 shutdown_connections 2024-08-03T20:54:40.060 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.057+0000 7ffa832b1f40 1 --2- 172.21.15.32:0/501628672 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x558171694cb0 0x558171691040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:40.060 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.057+0000 7ffa832b1f40 1 --2- 172.21.15.32:0/501628672 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x558171678ea0 0x558171690b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:40.060 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.057+0000 7ffa832b1f40 1 --2- 172.21.15.32:0/501628672 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558171677360 0x5581715c0620 unknown :-1 s=CLOSED pgs=964 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:40.060 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.057+0000 7ffa832b1f40 1 -- 172.21.15.32:0/501628672 >> 172.21.15.32:0/501628672 conn(0x5581714e0ac0 msgr2=0x5581716903f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:54:40.060 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.057+0000 7ffa832b1f40 1 -- 172.21.15.32:0/501628672 shutdown_connections 2024-08-03T20:54:40.060 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.057+0000 7ffa832b1f40 1 -- 172.21.15.32:0/501628672 wait complete. 2024-08-03T20:54:40.061 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.058+0000 7ffa832b1f40 1 Processor -- start 2024-08-03T20:54:40.061 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.058+0000 7ffa832b1f40 1 -- start start 2024-08-03T20:54:40.062 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.060+0000 7ffa832b1f40 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558171677360 0x55817166b060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:40.063 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.060+0000 7ffa832b1f40 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x558171678ea0 0x55817168f070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:40.063 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.060+0000 7ffa832b1f40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x558171694cb0 0x55817168f5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:40.063 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.060+0000 7ffa832b1f40 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55817168fe80 con 0x558171677360 2024-08-03T20:54:40.063 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.060+0000 7ffa832b1f40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x558171690160 con 0x558171694cb0 2024-08-03T20:54:40.063 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.060+0000 7ffa832b1f40 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x558171608970 con 0x558171678ea0 2024-08-03T20:54:40.063 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.060+0000 7ffa7c400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558171677360 0x55817166b060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:40.063 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.060+0000 7ffa7c400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558171677360 0x55817166b060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:36624/0 (socket says 172.21.15.32:36624) 2024-08-03T20:54:40.063 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.060+0000 7ffa7c400640 1 -- 172.21.15.32:0/1551859063 learned_addr learned my addr 172.21.15.32:0/1551859063 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:54:40.063 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.060+0000 7ffa7ba00640 1 --2- 172.21.15.32:0/1551859063 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x558171678ea0 0x55817168f070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:40.063 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.060+0000 7ffa7ce00640 1 --2- 172.21.15.32:0/1551859063 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x558171694cb0 0x55817168f5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:40.063 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.060+0000 7ffa7c400640 1 -- 172.21.15.32:0/1551859063 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x558171678ea0 msgr2=0x55817168f070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:40.063 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.060+0000 7ffa7c400640 1 --2- 172.21.15.32:0/1551859063 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x558171678ea0 0x55817168f070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:40.063 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.060+0000 7ffa7c400640 1 -- 172.21.15.32:0/1551859063 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x558171694cb0 msgr2=0x55817168f5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:40.064 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.060+0000 7ffa7c400640 1 --2- 172.21.15.32:0/1551859063 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x558171694cb0 0x55817168f5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:40.064 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.060+0000 7ffa7c400640 1 -- 172.21.15.32:0/1551859063 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558171672e80 con 0x558171677360 2024-08-03T20:54:40.064 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.061+0000 7ffa7c400640 1 --2- 172.21.15.32:0/1551859063 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558171677360 0x55817166b060 secure :-1 s=READY pgs=965 cs=0 l=1 rev1=1 crypto rx=0x7ffa6c00cec0 tx=0x7ffa6c0156f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:54:40.064 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.061+0000 7ffa79200640 1 -- 172.21.15.32:0/1551859063 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa6c004c80 con 0x558171677360 2024-08-03T20:54:40.064 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.061+0000 7ffa832b1f40 1 -- 172.21.15.32:0/1551859063 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x558171609b90 con 0x558171677360 2024-08-03T20:54:40.064 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.061+0000 7ffa79200640 1 -- 172.21.15.32:0/1551859063 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7ffa6c022070 con 0x558171677360 2024-08-03T20:54:40.064 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.061+0000 7ffa79200640 1 -- 172.21.15.32:0/1551859063 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa6c01d430 con 0x558171677360 2024-08-03T20:54:40.064 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.061+0000 7ffa832b1f40 1 -- 172.21.15.32:0/1551859063 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55817168e870 con 0x558171677360 2024-08-03T20:54:40.064 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.063+0000 7ffa79200640 1 -- 172.21.15.32:0/1551859063 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7ffa6c01d5d0 con 0x558171677360 2024-08-03T20:54:40.066 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.064+0000 7ffa79200640 1 --2- 172.21.15.32:0/1551859063 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7ffa5c075b90 0x7ffa5c0780a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:40.066 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.064+0000 7ffa79200640 1 -- 172.21.15.32:0/1551859063 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 5795+0+0 (secure 0 0 0) 0x7ffa6c099630 con 0x558171677360 2024-08-03T20:54:40.066 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.065+0000 7ffa832b1f40 1 --2- 172.21.15.32:0/1551859063 >> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] conn(0x55817166fe30 0x558171670240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:40.066 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.065+0000 7ffa7ba00640 1 --2- 172.21.15.32:0/1551859063 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7ffa5c075b90 0x7ffa5c0780a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:40.066 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.065+0000 7ffa832b1f40 1 -- 172.21.15.32:0/1551859063 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:1 12.15 12:aa448500:::500.00000000:head [omap-get-header] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84) -- 0x5581716914f0 con 0x55817166fe30 2024-08-03T20:54:40.067 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.065+0000 7ffa7ce00640 1 --2- 172.21.15.32:0/1551859063 >> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] conn(0x55817166fe30 0x558171670240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:40.067 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.065+0000 7ffa7ba00640 1 --2- 172.21.15.32:0/1551859063 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7ffa5c075b90 0x7ffa5c0780a0 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7ffa680041b0 tx=0x7ffa68004140 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:54:40.067 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.066+0000 7ffa7ce00640 1 --2- 172.21.15.32:0/1551859063 >> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] conn(0x55817166fe30 0x558171670240 crc :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:54:40.067 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.066+0000 7ffa7ce00640 1 -- 172.21.15.32:0/1551859063 <== osd.6 v2:172.21.15.118:6825/801947386 1 ==== osd_op_reply(1 500.00000000 [omap-get-header] v0'0 uv3 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7ffa7400b040 con 0x55817166fe30 2024-08-03T20:54:40.068 INFO:teuthology.orchestra.run.smithi032.stdout:header (0 bytes) : 2024-08-03T20:54:40.068 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-03T20:54:40.069 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.066+0000 7ffa832b1f40 1 -- 172.21.15.32:0/1551859063 >> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] conn(0x55817166fe30 msgr2=0x558171670240 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:40.069 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.066+0000 7ffa832b1f40 1 --2- 172.21.15.32:0/1551859063 >> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] conn(0x55817166fe30 0x558171670240 crc :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:40.069 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.066+0000 7ffa832b1f40 1 -- 172.21.15.32:0/1551859063 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7ffa5c075b90 msgr2=0x7ffa5c0780a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:40.069 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.066+0000 7ffa832b1f40 1 --2- 172.21.15.32:0/1551859063 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7ffa5c075b90 0x7ffa5c0780a0 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7ffa680041b0 tx=0x7ffa68004140 comp rx=0 tx=0).stop 2024-08-03T20:54:40.069 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.066+0000 7ffa832b1f40 1 -- 172.21.15.32:0/1551859063 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558171677360 msgr2=0x55817166b060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:40.069 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.066+0000 7ffa832b1f40 1 --2- 172.21.15.32:0/1551859063 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558171677360 0x55817166b060 secure :-1 s=READY pgs=965 cs=0 l=1 rev1=1 crypto rx=0x7ffa6c00cec0 tx=0x7ffa6c0156f0 comp rx=0 tx=0).stop 2024-08-03T20:54:40.069 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.066+0000 7ffa7ce00640 1 -- 172.21.15.32:0/1551859063 reap_dead start 2024-08-03T20:54:40.069 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.066+0000 7ffa832b1f40 1 -- 172.21.15.32:0/1551859063 shutdown_connections 2024-08-03T20:54:40.069 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.066+0000 7ffa832b1f40 1 -- 172.21.15.32:0/1551859063 >> 172.21.15.32:0/1551859063 conn(0x5581714e0ac0 msgr2=0x5581715c0620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:54:40.070 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.066+0000 7ffa832b1f40 1 -- 172.21.15.32:0/1551859063 shutdown_connections 2024-08-03T20:54:40.070 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.067+0000 7ffa832b1f40 1 -- 172.21.15.32:0/1551859063 wait complete. 2024-08-03T20:54:40.078 DEBUG:teuthology.orchestra.run.smithi032:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata getomapheader mds_snaptable 2024-08-03T20:54:40.219 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.218+0000 7f5bae33af40 1 Processor -- start 2024-08-03T20:54:40.219 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.218+0000 7f5bae33af40 1 -- start start 2024-08-03T20:54:40.221 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.219+0000 7f5bae33af40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5615a5037b40 0x5615a50cae00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:40.221 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.219+0000 7f5bae33af40 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x5615a50aabe0 0x5615a50a8380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:40.221 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.219+0000 7f5bae33af40 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5615a50a74e0 0x5615a50c62a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:40.222 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.219+0000 7f5bae33af40 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x5615a50ab510 con 0x5615a50a74e0 2024-08-03T20:54:40.222 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.219+0000 7f5bae33af40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x5615a4f40e30 con 0x5615a5037b40 2024-08-03T20:54:40.222 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.219+0000 7f5bae33af40 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x5615a50ab690 con 0x5615a50aabe0 2024-08-03T20:54:40.222 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.219+0000 7f5ba7e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5615a50a74e0 0x5615a50c62a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:40.222 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.219+0000 7f5ba7e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5615a50a74e0 0x5615a50c62a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:36642/0 (socket says 172.21.15.32:36642) 2024-08-03T20:54:40.222 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.219+0000 7f5ba7e00640 1 -- 172.21.15.32:0/2513043427 learned_addr learned my addr 172.21.15.32:0/2513043427 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:54:40.222 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.220+0000 7f5ba6a00640 1 --2- 172.21.15.32:0/2513043427 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x5615a50aabe0 0x5615a50a8380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:40.222 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.220+0000 7f5ba7400640 1 --2- 172.21.15.32:0/2513043427 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5615a5037b40 0x5615a50cae00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:40.223 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.220+0000 7f5ba7e00640 1 -- 172.21.15.32:0/2513043427 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x5615a50aabe0 msgr2=0x5615a50a8380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:54:40.223 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.220+0000 7f5ba7e00640 1 --2- 172.21.15.32:0/2513043427 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x5615a50aabe0 0x5615a50a8380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:40.223 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.220+0000 7f5ba7e00640 1 -- 172.21.15.32:0/2513043427 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5615a5037b40 msgr2=0x5615a50cae00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:54:40.223 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.220+0000 7f5ba7e00640 1 --2- 172.21.15.32:0/2513043427 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5615a5037b40 0x5615a50cae00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:40.223 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.220+0000 7f5ba7e00640 1 -- 172.21.15.32:0/2513043427 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5615a4f19f10 con 0x5615a50a74e0 2024-08-03T20:54:40.223 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.220+0000 7f5ba7400640 1 --2- 172.21.15.32:0/2513043427 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5615a5037b40 0x5615a50cae00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-03T20:54:40.223 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.220+0000 7f5ba7e00640 1 --2- 172.21.15.32:0/2513043427 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5615a50a74e0 0x5615a50c62a0 secure :-1 s=READY pgs=967 cs=0 l=1 rev1=1 crypto rx=0x7f5ba000c6a0 tx=0x7f5ba000cb60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cdf9b03f4f60d4a0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:54:40.224 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.220+0000 7f5ba5600640 1 -- 172.21.15.32:0/2513043427 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ba0004020 con 0x5615a50a74e0 2024-08-03T20:54:40.225 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.220+0000 7f5ba5600640 1 -- 172.21.15.32:0/2513043427 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f5ba00042b0 con 0x5615a50a74e0 2024-08-03T20:54:40.225 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.220+0000 7f5ba5600640 1 -- 172.21.15.32:0/2513043427 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ba000aa80 con 0x5615a50a74e0 2024-08-03T20:54:40.225 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.221+0000 7f5bae33af40 1 -- 172.21.15.32:0/2513043427 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5615a50a74e0 msgr2=0x5615a50c62a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:40.225 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.221+0000 7f5bae33af40 1 --2- 172.21.15.32:0/2513043427 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5615a50a74e0 0x5615a50c62a0 secure :-1 s=READY pgs=967 cs=0 l=1 rev1=1 crypto rx=0x7f5ba000c6a0 tx=0x7f5ba000cb60 comp rx=0 tx=0).stop 2024-08-03T20:54:40.225 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.221+0000 7f5bae33af40 1 -- 172.21.15.32:0/2513043427 shutdown_connections 2024-08-03T20:54:40.225 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.221+0000 7f5bae33af40 1 --2- 172.21.15.32:0/2513043427 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5615a50a74e0 0x5615a50c62a0 unknown :-1 s=CLOSED pgs=967 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:40.225 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.221+0000 7f5bae33af40 1 --2- 172.21.15.32:0/2513043427 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x5615a50aabe0 0x5615a50a8380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:40.225 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.221+0000 7f5bae33af40 1 --2- 172.21.15.32:0/2513043427 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5615a5037b40 0x5615a50cae00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:40.226 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.221+0000 7f5bae33af40 1 -- 172.21.15.32:0/2513043427 >> 172.21.15.32:0/2513043427 conn(0x5615a4f12ac0 msgr2=0x5615a50c7760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:54:40.226 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.221+0000 7f5bae33af40 1 -- 172.21.15.32:0/2513043427 shutdown_connections 2024-08-03T20:54:40.226 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.221+0000 7f5bae33af40 1 -- 172.21.15.32:0/2513043427 wait complete. 2024-08-03T20:54:40.226 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.222+0000 7f5bae33af40 1 Processor -- start 2024-08-03T20:54:40.226 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.222+0000 7f5bae33af40 1 -- start start 2024-08-03T20:54:40.228 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.224+0000 7f5bae33af40 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x5615a5037b40 0x5615a50a1150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:40.228 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.224+0000 7f5bae33af40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5615a50a74e0 0x5615a50c75b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:40.229 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.224+0000 7f5bae33af40 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5615a50aabe0 0x5615a50c7af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:40.229 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.224+0000 7f5bae33af40 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x5615a50a7b40 con 0x5615a50aabe0 2024-08-03T20:54:40.229 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.224+0000 7f5bae33af40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x5615a50acd30 con 0x5615a50a74e0 2024-08-03T20:54:40.229 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.224+0000 7f5bae33af40 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x5615a503a110 con 0x5615a5037b40 2024-08-03T20:54:40.229 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.224+0000 7f5ba7e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5615a50aabe0 0x5615a50c7af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:40.229 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.224+0000 7f5ba7e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5615a50aabe0 0x5615a50c7af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:36658/0 (socket says 172.21.15.32:36658) 2024-08-03T20:54:40.230 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.224+0000 7f5ba7e00640 1 -- 172.21.15.32:0/1376234289 learned_addr learned my addr 172.21.15.32:0/1376234289 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:54:40.230 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.224+0000 7f5ba7400640 1 --2- 172.21.15.32:0/1376234289 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x5615a5037b40 0x5615a50a1150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:40.230 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.224+0000 7f5ba6a00640 1 --2- 172.21.15.32:0/1376234289 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5615a50a74e0 0x5615a50c75b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:40.230 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.225+0000 7f5ba7e00640 1 -- 172.21.15.32:0/1376234289 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x5615a5037b40 msgr2=0x5615a50a1150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:40.230 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.225+0000 7f5ba7e00640 1 --2- 172.21.15.32:0/1376234289 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x5615a5037b40 0x5615a50a1150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:40.230 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.225+0000 7f5ba7e00640 1 -- 172.21.15.32:0/1376234289 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5615a50a74e0 msgr2=0x5615a50c75b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:40.230 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.225+0000 7f5ba7e00640 1 --2- 172.21.15.32:0/1376234289 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5615a50a74e0 0x5615a50c75b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:40.230 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.225+0000 7f5ba7e00640 1 -- 172.21.15.32:0/1376234289 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5615a5155a80 con 0x5615a50aabe0 2024-08-03T20:54:40.230 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.225+0000 7f5ba7e00640 1 --2- 172.21.15.32:0/1376234289 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5615a50aabe0 0x5615a50c7af0 secure :-1 s=READY pgs=968 cs=0 l=1 rev1=1 crypto rx=0x5615a50ca3d0 tx=0x7f5ba0004560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:54:40.230 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.225+0000 7f5b9fe00640 1 -- 172.21.15.32:0/1376234289 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ba0025070 con 0x5615a50aabe0 2024-08-03T20:54:40.230 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.225+0000 7f5b9fe00640 1 -- 172.21.15.32:0/1376234289 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f5ba00042b0 con 0x5615a50aabe0 2024-08-03T20:54:40.230 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.225+0000 7f5b9fe00640 1 -- 172.21.15.32:0/1376234289 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ba000f8b0 con 0x5615a50aabe0 2024-08-03T20:54:40.230 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.225+0000 7f5bae33af40 1 -- 172.21.15.32:0/1376234289 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x5615a51549e0 con 0x5615a50aabe0 2024-08-03T20:54:40.230 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.226+0000 7f5bae33af40 1 -- 172.21.15.32:0/1376234289 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5615a5154fa0 con 0x5615a50aabe0 2024-08-03T20:54:40.232 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.228+0000 7f5b9fe00640 1 -- 172.21.15.32:0/1376234289 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f5ba0028070 con 0x5615a50aabe0 2024-08-03T20:54:40.232 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.228+0000 7f5b9fe00640 1 --2- 172.21.15.32:0/1376234289 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f5b88075a20 0x7f5b88077f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:40.232 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.229+0000 7f5b9fe00640 1 -- 172.21.15.32:0/1376234289 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 5795+0+0 (secure 0 0 0) 0x7f5ba0098c20 con 0x5615a50aabe0 2024-08-03T20:54:40.232 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.229+0000 7f5bae33af40 1 --2- 172.21.15.32:0/1376234289 >> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] conn(0x5615a50a4f70 0x5615a5153830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:40.232 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.229+0000 7f5bae33af40 1 -- 172.21.15.32:0/1376234289 --> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] -- osd_op(unknown.0.0:1 12.d 12:b50e409b:::mds_snaptable:head [omap-get-header] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84) -- 0x5615a50399c0 con 0x5615a50a4f70 2024-08-03T20:54:40.232 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.229+0000 7f5ba7400640 1 --2- 172.21.15.32:0/1376234289 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f5b88075a20 0x7f5b88077f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:40.232 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.229+0000 7f5ba6a00640 1 --2- 172.21.15.32:0/1376234289 >> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] conn(0x5615a50a4f70 0x5615a5153830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:40.232 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.230+0000 7f5ba7400640 1 --2- 172.21.15.32:0/1376234289 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f5b88075a20 0x7f5b88077f30 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f5b98055000 tx=0x7f5b98003020 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:54:40.232 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.230+0000 7f5ba6a00640 1 --2- 172.21.15.32:0/1376234289 >> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] conn(0x5615a50a4f70 0x5615a5153830 crc :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:54:40.233 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.230+0000 7f5ba6a00640 1 -- 172.21.15.32:0/1376234289 <== osd.7 v2:172.21.15.118:6818/653000048 1 ==== osd_op_reply(1 mds_snaptable [omap-get-header] v0'0 uv1 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f5b9400c9b0 con 0x5615a50a4f70 2024-08-03T20:54:40.233 INFO:teuthology.orchestra.run.smithi032.stdout:header (0 bytes) : 2024-08-03T20:54:40.233 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-03T20:54:40.234 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.231+0000 7f5bae33af40 1 -- 172.21.15.32:0/1376234289 >> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] conn(0x5615a50a4f70 msgr2=0x5615a5153830 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:40.234 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.231+0000 7f5bae33af40 1 --2- 172.21.15.32:0/1376234289 >> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] conn(0x5615a50a4f70 0x5615a5153830 crc :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:40.234 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.231+0000 7f5bae33af40 1 -- 172.21.15.32:0/1376234289 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f5b88075a20 msgr2=0x7f5b88077f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:40.234 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.231+0000 7f5bae33af40 1 --2- 172.21.15.32:0/1376234289 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f5b88075a20 0x7f5b88077f30 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f5b98055000 tx=0x7f5b98003020 comp rx=0 tx=0).stop 2024-08-03T20:54:40.234 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.231+0000 7f5bae33af40 1 -- 172.21.15.32:0/1376234289 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5615a50aabe0 msgr2=0x5615a50c7af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:40.234 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.231+0000 7f5bae33af40 1 --2- 172.21.15.32:0/1376234289 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5615a50aabe0 0x5615a50c7af0 secure :-1 s=READY pgs=968 cs=0 l=1 rev1=1 crypto rx=0x5615a50ca3d0 tx=0x7f5ba0004560 comp rx=0 tx=0).stop 2024-08-03T20:54:40.234 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.231+0000 7f5ba7e00640 1 -- 172.21.15.32:0/1376234289 reap_dead start 2024-08-03T20:54:40.234 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.231+0000 7f5bae33af40 1 -- 172.21.15.32:0/1376234289 shutdown_connections 2024-08-03T20:54:40.234 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.231+0000 7f5bae33af40 1 -- 172.21.15.32:0/1376234289 >> 172.21.15.32:0/1376234289 conn(0x5615a4f12ac0 msgr2=0x5615a4ff4410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:54:40.235 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.231+0000 7f5bae33af40 1 -- 172.21.15.32:0/1376234289 shutdown_connections 2024-08-03T20:54:40.235 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.231+0000 7f5bae33af40 1 -- 172.21.15.32:0/1376234289 wait complete. 2024-08-03T20:54:40.243 DEBUG:teuthology.orchestra.run.smithi032:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata getomapheader mds0_inotable 2024-08-03T20:54:40.385 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.384+0000 7fbd23225f40 1 Processor -- start 2024-08-03T20:54:40.385 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.384+0000 7fbd23225f40 1 -- start start 2024-08-03T20:54:40.388 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.385+0000 7fbd23225f40 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55fd5f4a1360 0x55fd5f4bcf50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:40.388 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.385+0000 7fbd23225f40 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55fd5f4becb0 0x55fd5f3e9be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:40.388 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.385+0000 7fbd23225f40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55fd5f4bf3f0 0x55fd5f3ea120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:40.388 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.385+0000 7fbd23225f40 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55fd5f338e30 con 0x55fd5f4a1360 2024-08-03T20:54:40.388 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.385+0000 7fbd23225f40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x55fd5f396bf0 con 0x55fd5f4bf3f0 2024-08-03T20:54:40.388 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.385+0000 7fbd23225f40 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x55fd5f4a2440 con 0x55fd5f4becb0 2024-08-03T20:54:40.388 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.385+0000 7fbd1c400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55fd5f4a1360 0x55fd5f4bcf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:40.388 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.385+0000 7fbd1c400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55fd5f4a1360 0x55fd5f4bcf50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:36678/0 (socket says 172.21.15.32:36678) 2024-08-03T20:54:40.388 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.385+0000 7fbd1c400640 1 -- 172.21.15.32:0/97274111 learned_addr learned my addr 172.21.15.32:0/97274111 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:54:40.389 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.386+0000 7fbd1ce00640 1 --2- 172.21.15.32:0/97274111 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55fd5f4bf3f0 0x55fd5f3ea120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:40.389 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.386+0000 7fbd1ba00640 1 --2- 172.21.15.32:0/97274111 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55fd5f4becb0 0x55fd5f3e9be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:40.389 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.386+0000 7fbd1c400640 1 -- 172.21.15.32:0/97274111 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55fd5f4becb0 msgr2=0x55fd5f3e9be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:54:40.389 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.386+0000 7fbd1c400640 1 --2- 172.21.15.32:0/97274111 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55fd5f4becb0 0x55fd5f3e9be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:40.389 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.386+0000 7fbd1c400640 1 -- 172.21.15.32:0/97274111 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55fd5f4bf3f0 msgr2=0x55fd5f3ea120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:54:40.389 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.386+0000 7fbd1c400640 1 --2- 172.21.15.32:0/97274111 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55fd5f4bf3f0 0x55fd5f3ea120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:40.389 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.386+0000 7fbd1c400640 1 -- 172.21.15.32:0/97274111 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fd5f4b8480 con 0x55fd5f4a1360 2024-08-03T20:54:40.389 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.386+0000 7fbd1ce00640 1 --2- 172.21.15.32:0/97274111 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55fd5f4bf3f0 0x55fd5f3ea120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-03T20:54:40.389 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.386+0000 7fbd1c400640 1 --2- 172.21.15.32:0/97274111 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55fd5f4a1360 0x55fd5f4bcf50 secure :-1 s=READY pgs=971 cs=0 l=1 rev1=1 crypto rx=0x7fbd0c025de0 tx=0x7fbd0c002fb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bc0c18a78c1fa4ff server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:54:40.389 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.386+0000 7fbd1a600640 1 -- 172.21.15.32:0/97274111 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd0c04dcd0 con 0x55fd5f4a1360 2024-08-03T20:54:40.389 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.386+0000 7fbd1a600640 1 -- 172.21.15.32:0/97274111 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fbd0c056470 con 0x55fd5f4a1360 2024-08-03T20:54:40.389 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.386+0000 7fbd1a600640 1 -- 172.21.15.32:0/97274111 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd0c0568a0 con 0x55fd5f4a1360 2024-08-03T20:54:40.390 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.387+0000 7fbd23225f40 1 -- 172.21.15.32:0/97274111 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55fd5f4a1360 msgr2=0x55fd5f4bcf50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:40.391 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.387+0000 7fbd23225f40 1 --2- 172.21.15.32:0/97274111 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55fd5f4a1360 0x55fd5f4bcf50 secure :-1 s=READY pgs=971 cs=0 l=1 rev1=1 crypto rx=0x7fbd0c025de0 tx=0x7fbd0c002fb0 comp rx=0 tx=0).stop 2024-08-03T20:54:40.391 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.387+0000 7fbd23225f40 1 -- 172.21.15.32:0/97274111 shutdown_connections 2024-08-03T20:54:40.391 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.387+0000 7fbd23225f40 1 --2- 172.21.15.32:0/97274111 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55fd5f4bf3f0 0x55fd5f3ea120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:40.391 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.387+0000 7fbd23225f40 1 --2- 172.21.15.32:0/97274111 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55fd5f4becb0 0x55fd5f3e9be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:40.391 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.387+0000 7fbd23225f40 1 --2- 172.21.15.32:0/97274111 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55fd5f4a1360 0x55fd5f4bcf50 unknown :-1 s=CLOSED pgs=971 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:40.391 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.387+0000 7fbd23225f40 1 -- 172.21.15.32:0/97274111 >> 172.21.15.32:0/97274111 conn(0x55fd5f30aac0 msgr2=0x55fd5f4bdb40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:54:40.391 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.387+0000 7fbd23225f40 1 -- 172.21.15.32:0/97274111 shutdown_connections 2024-08-03T20:54:40.391 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.387+0000 7fbd23225f40 1 -- 172.21.15.32:0/97274111 wait complete. 2024-08-03T20:54:40.391 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.388+0000 7fbd23225f40 1 Processor -- start 2024-08-03T20:54:40.391 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.388+0000 7fbd23225f40 1 -- start start 2024-08-03T20:54:40.393 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.390+0000 7fbd23225f40 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55fd5f4a1360 0x55fd5f49db90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:40.393 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.390+0000 7fbd23225f40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55fd5f4becb0 0x55fd5f429ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:40.393 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.390+0000 7fbd23225f40 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55fd5f4bf3f0 0x55fd5f42a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:40.394 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.390+0000 7fbd23225f40 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55fd5f4a00d0 con 0x55fd5f4a1360 2024-08-03T20:54:40.394 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.390+0000 7fbd23225f40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x55fd5f49fb10 con 0x55fd5f4becb0 2024-08-03T20:54:40.394 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.390+0000 7fbd23225f40 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x55fd5f557250 con 0x55fd5f4bf3f0 2024-08-03T20:54:40.394 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.390+0000 7fbd1c400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55fd5f4a1360 0x55fd5f49db90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:40.394 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.390+0000 7fbd1c400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55fd5f4a1360 0x55fd5f49db90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:36686/0 (socket says 172.21.15.32:36686) 2024-08-03T20:54:40.394 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.390+0000 7fbd1c400640 1 -- 172.21.15.32:0/2540349738 learned_addr learned my addr 172.21.15.32:0/2540349738 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:54:40.394 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.390+0000 7fbd1ce00640 1 --2- 172.21.15.32:0/2540349738 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55fd5f4bf3f0 0x55fd5f42a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:40.394 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.390+0000 7fbd1ba00640 1 --2- 172.21.15.32:0/2540349738 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55fd5f4becb0 0x55fd5f429ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:40.394 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.391+0000 7fbd1c400640 1 -- 172.21.15.32:0/2540349738 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55fd5f4bf3f0 msgr2=0x55fd5f42a430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:40.394 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.391+0000 7fbd1c400640 1 --2- 172.21.15.32:0/2540349738 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55fd5f4bf3f0 0x55fd5f42a430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:40.394 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.391+0000 7fbd1c400640 1 -- 172.21.15.32:0/2540349738 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55fd5f4becb0 msgr2=0x55fd5f429ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:40.394 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.391+0000 7fbd1c400640 1 --2- 172.21.15.32:0/2540349738 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55fd5f4becb0 0x55fd5f429ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:40.394 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.391+0000 7fbd1c400640 1 -- 172.21.15.32:0/2540349738 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fd5f4b99c0 con 0x55fd5f4a1360 2024-08-03T20:54:40.395 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.391+0000 7fbd1c400640 1 --2- 172.21.15.32:0/2540349738 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55fd5f4a1360 0x55fd5f49db90 secure :-1 s=READY pgs=972 cs=0 l=1 rev1=1 crypto rx=0x7fbd0c0464e0 tx=0x7fbd0c0041d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:54:40.395 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.391+0000 7fbd19200640 1 -- 172.21.15.32:0/2540349738 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd0c057580 con 0x55fd5f4a1360 2024-08-03T20:54:40.395 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.391+0000 7fbd19200640 1 -- 172.21.15.32:0/2540349738 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fbd0c057810 con 0x55fd5f4a1360 2024-08-03T20:54:40.395 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.391+0000 7fbd19200640 1 -- 172.21.15.32:0/2540349738 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd0c0579e0 con 0x55fd5f4a1360 2024-08-03T20:54:40.395 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.391+0000 7fbd23225f40 1 -- 172.21.15.32:0/2540349738 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x55fd5f4b9ca0 con 0x55fd5f4a1360 2024-08-03T20:54:40.395 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.391+0000 7fbd23225f40 1 -- 172.21.15.32:0/2540349738 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fd5f4bdd50 con 0x55fd5f4a1360 2024-08-03T20:54:40.395 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.394+0000 7fbd19200640 1 -- 172.21.15.32:0/2540349738 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7fbd0c072020 con 0x55fd5f4a1360 2024-08-03T20:54:40.396 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.394+0000 7fbd19200640 1 --2- 172.21.15.32:0/2540349738 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fbcfc0759f0 0x7fbcfc077f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:40.396 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.395+0000 7fbd19200640 1 -- 172.21.15.32:0/2540349738 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 5795+0+0 (secure 0 0 0) 0x7fbd0c0dc930 con 0x55fd5f4a1360 2024-08-03T20:54:40.396 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.395+0000 7fbd23225f40 1 --2- 172.21.15.32:0/2540349738 >> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] conn(0x55fd5f432120 0x55fd5f55b200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:40.397 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.395+0000 7fbd23225f40 1 -- 172.21.15.32:0/2540349738 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:1 12.13 12:c91d4a1d:::mds0_inotable:head [omap-get-header] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84) -- 0x55fd5f42bea0 con 0x55fd5f432120 2024-08-03T20:54:40.397 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.395+0000 7fbd1ce00640 1 --2- 172.21.15.32:0/2540349738 >> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] conn(0x55fd5f432120 0x55fd5f55b200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:40.397 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.395+0000 7fbd1ba00640 1 --2- 172.21.15.32:0/2540349738 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fbcfc0759f0 0x7fbcfc077f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:40.397 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.396+0000 7fbd1ce00640 1 --2- 172.21.15.32:0/2540349738 >> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] conn(0x55fd5f432120 0x55fd5f55b200 crc :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:54:40.397 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.396+0000 7fbd1ba00640 1 --2- 172.21.15.32:0/2540349738 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fbcfc0759f0 0x7fbcfc077f00 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7fbd08002a60 tx=0x7fbd080029f0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:54:40.398 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.396+0000 7fbd1ce00640 1 -- 172.21.15.32:0/2540349738 <== osd.1 v2:172.21.15.32:6824/1551241759 1 ==== osd_op_reply(1 mds0_inotable [omap-get-header] v0'0 uv3 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7fbd1400b040 con 0x55fd5f432120 2024-08-03T20:54:40.398 INFO:teuthology.orchestra.run.smithi032.stdout:header (0 bytes) : 2024-08-03T20:54:40.398 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-03T20:54:40.399 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.396+0000 7fbd23225f40 1 -- 172.21.15.32:0/2540349738 >> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] conn(0x55fd5f432120 msgr2=0x55fd5f55b200 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:40.399 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.396+0000 7fbd23225f40 1 --2- 172.21.15.32:0/2540349738 >> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] conn(0x55fd5f432120 0x55fd5f55b200 crc :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:40.399 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.396+0000 7fbd23225f40 1 -- 172.21.15.32:0/2540349738 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fbcfc0759f0 msgr2=0x7fbcfc077f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:40.399 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.396+0000 7fbd23225f40 1 --2- 172.21.15.32:0/2540349738 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fbcfc0759f0 0x7fbcfc077f00 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7fbd08002a60 tx=0x7fbd080029f0 comp rx=0 tx=0).stop 2024-08-03T20:54:40.399 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.396+0000 7fbd23225f40 1 -- 172.21.15.32:0/2540349738 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55fd5f4a1360 msgr2=0x55fd5f49db90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:40.399 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.396+0000 7fbd23225f40 1 --2- 172.21.15.32:0/2540349738 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55fd5f4a1360 0x55fd5f49db90 secure :-1 s=READY pgs=972 cs=0 l=1 rev1=1 crypto rx=0x7fbd0c0464e0 tx=0x7fbd0c0041d0 comp rx=0 tx=0).stop 2024-08-03T20:54:40.399 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.396+0000 7fbd1ce00640 1 -- 172.21.15.32:0/2540349738 reap_dead start 2024-08-03T20:54:40.399 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.396+0000 7fbd23225f40 1 -- 172.21.15.32:0/2540349738 shutdown_connections 2024-08-03T20:54:40.399 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.396+0000 7fbd23225f40 1 -- 172.21.15.32:0/2540349738 >> 172.21.15.32:0/2540349738 conn(0x55fd5f30aac0 msgr2=0x55fd5f42ff40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:54:40.399 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.396+0000 7fbd23225f40 1 -- 172.21.15.32:0/2540349738 shutdown_connections 2024-08-03T20:54:40.400 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.396+0000 7fbd23225f40 1 -- 172.21.15.32:0/2540349738 wait complete. 2024-08-03T20:54:40.412 DEBUG:teuthology.orchestra.run.smithi032:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata getomapheader 100.00000000 2024-08-03T20:54:40.555 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.553+0000 7f9e36ffef40 1 Processor -- start 2024-08-03T20:54:40.555 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.553+0000 7f9e36ffef40 1 -- start start 2024-08-03T20:54:40.557 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.556+0000 7f9e36ffef40 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561988d6e360 0x561988d879b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:40.557 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.556+0000 7f9e36ffef40 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x561988d6fea0 0x561988cf8030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:40.557 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.556+0000 7f9e36ffef40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x561988d8bcb0 0x561988cf8570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:40.557 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.556+0000 7f9e36ffef40 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x561988d6f1b0 con 0x561988d6e360 2024-08-03T20:54:40.557 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.556+0000 7f9e36ffef40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x561988c05e30 con 0x561988d8bcb0 2024-08-03T20:54:40.557 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.556+0000 7f9e36ffef40 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x561988d8c0c0 con 0x561988d6fea0 2024-08-03T20:54:40.558 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.556+0000 7f9e30200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561988d6e360 0x561988d879b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:40.558 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.556+0000 7f9e30200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561988d6e360 0x561988d879b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:36716/0 (socket says 172.21.15.32:36716) 2024-08-03T20:54:40.558 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.556+0000 7f9e30200640 1 -- 172.21.15.32:0/2504176119 learned_addr learned my addr 172.21.15.32:0/2504176119 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:54:40.558 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.556+0000 7f9e2f800640 1 --2- 172.21.15.32:0/2504176119 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x561988d6fea0 0x561988cf8030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:40.558 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.556+0000 7f9e30c00640 1 --2- 172.21.15.32:0/2504176119 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x561988d8bcb0 0x561988cf8570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:40.558 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.556+0000 7f9e30200640 1 -- 172.21.15.32:0/2504176119 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x561988d6fea0 msgr2=0x561988cf8030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:54:40.558 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.556+0000 7f9e30200640 1 --2- 172.21.15.32:0/2504176119 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x561988d6fea0 0x561988cf8030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:40.558 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.556+0000 7f9e30200640 1 -- 172.21.15.32:0/2504176119 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x561988d8bcb0 msgr2=0x561988cf8570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:54:40.558 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.556+0000 7f9e30200640 1 --2- 172.21.15.32:0/2504176119 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x561988d8bcb0 0x561988cf8570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:40.558 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.556+0000 7f9e30200640 1 -- 172.21.15.32:0/2504176119 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561988bdef10 con 0x561988d6e360 2024-08-03T20:54:40.558 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.556+0000 7f9e30c00640 1 --2- 172.21.15.32:0/2504176119 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x561988d8bcb0 0x561988cf8570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-03T20:54:40.558 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.556+0000 7f9e2f800640 1 --2- 172.21.15.32:0/2504176119 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x561988d6fea0 0x561988cf8030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-03T20:54:40.559 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.556+0000 7f9e30200640 1 --2- 172.21.15.32:0/2504176119 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561988d6e360 0x561988d879b0 secure :-1 s=READY pgs=974 cs=0 l=1 rev1=1 crypto rx=0x7f9e20025e00 tx=0x7f9e20002fb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dc78b0d2fd6288a1 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:54:40.560 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.556+0000 7f9e2e400640 1 -- 172.21.15.32:0/2504176119 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e2004d860 con 0x561988d6e360 2024-08-03T20:54:40.560 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.556+0000 7f9e2e400640 1 -- 172.21.15.32:0/2504176119 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f9e2004dcf0 con 0x561988d6e360 2024-08-03T20:54:40.560 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.556+0000 7f9e2e400640 1 -- 172.21.15.32:0/2504176119 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e200566d0 con 0x561988d6e360 2024-08-03T20:54:40.560 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.557+0000 7f9e36ffef40 1 -- 172.21.15.32:0/2504176119 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561988d6e360 msgr2=0x561988d879b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:40.560 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.557+0000 7f9e36ffef40 1 --2- 172.21.15.32:0/2504176119 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561988d6e360 0x561988d879b0 secure :-1 s=READY pgs=974 cs=0 l=1 rev1=1 crypto rx=0x7f9e20025e00 tx=0x7f9e20002fb0 comp rx=0 tx=0).stop 2024-08-03T20:54:40.560 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.557+0000 7f9e36ffef40 1 -- 172.21.15.32:0/2504176119 shutdown_connections 2024-08-03T20:54:40.560 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.557+0000 7f9e36ffef40 1 --2- 172.21.15.32:0/2504176119 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x561988d8bcb0 0x561988cf8570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:40.560 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.557+0000 7f9e36ffef40 1 --2- 172.21.15.32:0/2504176119 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x561988d6fea0 0x561988cf8030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:40.560 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.557+0000 7f9e36ffef40 1 --2- 172.21.15.32:0/2504176119 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561988d6e360 0x561988d879b0 unknown :-1 s=CLOSED pgs=974 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:40.561 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.557+0000 7f9e36ffef40 1 -- 172.21.15.32:0/2504176119 >> 172.21.15.32:0/2504176119 conn(0x561988bd7ac0 msgr2=0x561988d8a8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:54:40.561 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.557+0000 7f9e36ffef40 1 -- 172.21.15.32:0/2504176119 shutdown_connections 2024-08-03T20:54:40.561 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.557+0000 7f9e36ffef40 1 -- 172.21.15.32:0/2504176119 wait complete. 2024-08-03T20:54:40.561 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.558+0000 7f9e36ffef40 1 Processor -- start 2024-08-03T20:54:40.561 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.558+0000 7f9e36ffef40 1 -- start start 2024-08-03T20:54:40.563 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.559+0000 7f9e36ffef40 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561988d6e360 0x561988d65590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:40.563 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.559+0000 7f9e36ffef40 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x561988d6fea0 0x561988e24b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:40.563 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.559+0000 7f9e36ffef40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x561988d8bcb0 0x561988cff0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:40.563 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.559+0000 7f9e36ffef40 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x561988d88800 con 0x561988d6e360 2024-08-03T20:54:40.563 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.559+0000 7f9e36ffef40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x561988d00d00 con 0x561988d8bcb0 2024-08-03T20:54:40.563 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.559+0000 7f9e36ffef40 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x561988e255f0 con 0x561988d6fea0 2024-08-03T20:54:40.563 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.560+0000 7f9e30200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561988d6e360 0x561988d65590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:40.563 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.560+0000 7f9e30200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561988d6e360 0x561988d65590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:36720/0 (socket says 172.21.15.32:36720) 2024-08-03T20:54:40.563 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.560+0000 7f9e30200640 1 -- 172.21.15.32:0/397843457 learned_addr learned my addr 172.21.15.32:0/397843457 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:54:40.563 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.560+0000 7f9e30c00640 1 --2- 172.21.15.32:0/397843457 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x561988d8bcb0 0x561988cff0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:40.563 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.560+0000 7f9e2f800640 1 --2- 172.21.15.32:0/397843457 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x561988d6fea0 0x561988e24b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:40.563 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.560+0000 7f9e30200640 1 -- 172.21.15.32:0/397843457 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x561988d6fea0 msgr2=0x561988e24b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:40.564 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.560+0000 7f9e30200640 1 --2- 172.21.15.32:0/397843457 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x561988d6fea0 0x561988e24b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:40.564 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.560+0000 7f9e30200640 1 -- 172.21.15.32:0/397843457 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x561988d8bcb0 msgr2=0x561988cff0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:40.564 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.560+0000 7f9e30200640 1 --2- 172.21.15.32:0/397843457 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x561988d8bcb0 0x561988cff0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:40.564 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.560+0000 7f9e30200640 1 -- 172.21.15.32:0/397843457 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561988d8a530 con 0x561988d6e360 2024-08-03T20:54:40.564 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.560+0000 7f9e2f800640 1 --2- 172.21.15.32:0/397843457 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x561988d6fea0 0x561988e24b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-03T20:54:40.564 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.560+0000 7f9e30c00640 1 --2- 172.21.15.32:0/397843457 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x561988d8bcb0 0x561988cff0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-03T20:54:40.564 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.560+0000 7f9e30200640 1 --2- 172.21.15.32:0/397843457 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561988d6e360 0x561988d65590 secure :-1 s=READY pgs=975 cs=0 l=1 rev1=1 crypto rx=0x7f9e20046500 tx=0x7f9e200575e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:54:40.564 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.561+0000 7f9e2d000640 1 -- 172.21.15.32:0/397843457 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e20057790 con 0x561988d6e360 2024-08-03T20:54:40.564 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.561+0000 7f9e2d000640 1 -- 172.21.15.32:0/397843457 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f9e20070070 con 0x561988d6e360 2024-08-03T20:54:40.564 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.561+0000 7f9e2d000640 1 -- 172.21.15.32:0/397843457 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e20057cc0 con 0x561988d6e360 2024-08-03T20:54:40.564 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.561+0000 7f9e36ffef40 1 -- 172.21.15.32:0/397843457 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x561988e25090 con 0x561988d6e360 2024-08-03T20:54:40.564 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.561+0000 7f9e36ffef40 1 -- 172.21.15.32:0/397843457 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561988d6a620 con 0x561988d6e360 2024-08-03T20:54:40.565 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.564+0000 7f9e2d000640 1 -- 172.21.15.32:0/397843457 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f9e2006a070 con 0x561988d6e360 2024-08-03T20:54:40.565 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.565+0000 7f9e2d000640 1 --2- 172.21.15.32:0/397843457 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f9e10075a20 0x7f9e10077f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:40.565 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.565+0000 7f9e2d000640 1 -- 172.21.15.32:0/397843457 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 5795+0+0 (secure 0 0 0) 0x7f9e200dcf80 con 0x561988d6e360 2024-08-03T20:54:40.566 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.565+0000 7f9e36ffef40 1 --2- 172.21.15.32:0/397843457 >> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] conn(0x561988d6b610 0x561988d6ba20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:40.566 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.565+0000 7f9e36ffef40 1 -- 172.21.15.32:0/397843457 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:1 12.13 12:cd5a64a3:::100.00000000:head [omap-get-header] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84) -- 0x561988e276f0 con 0x561988d6b610 2024-08-03T20:54:40.566 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.565+0000 7f9e2f800640 1 --2- 172.21.15.32:0/397843457 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f9e10075a20 0x7f9e10077f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:40.566 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.565+0000 7f9e30c00640 1 --2- 172.21.15.32:0/397843457 >> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] conn(0x561988d6b610 0x561988d6ba20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:40.567 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.565+0000 7f9e30c00640 1 --2- 172.21.15.32:0/397843457 >> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] conn(0x561988d6b610 0x561988d6ba20 crc :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:54:40.567 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.565+0000 7f9e2f800640 1 --2- 172.21.15.32:0/397843457 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f9e10075a20 0x7f9e10077f30 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f9e1c004230 tx=0x7f9e1c002040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:54:40.567 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.566+0000 7f9e30c00640 1 -- 172.21.15.32:0/397843457 <== osd.1 v2:172.21.15.32:6824/1551241759 1 ==== osd_op_reply(1 100.00000000 [omap-get-header out=274b] v0'0 uv1 ondisk = 0) ==== 156+0+274 (crc 0 0 0) 0x7f9e2800b040 con 0x561988d6b610 2024-08-03T20:54:40.567 INFO:teuthology.orchestra.run.smithi032.stdout:header (274 bytes) : 2024-08-03T20:54:40.567 INFO:teuthology.orchestra.run.smithi032.stdout:00000000 04 03 0c 01 00 00 01 00 00 00 00 00 00 00 00 00 |................| 2024-08-03T20:54:40.568 INFO:teuthology.orchestra.run.smithi032.stdout:00000010 00 00 00 00 00 00 03 02 28 00 00 00 00 00 00 00 |........(.......| 2024-08-03T20:54:40.568 INFO:teuthology.orchestra.run.smithi032.stdout:00000020 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| 2024-08-03T20:54:40.568 INFO:teuthology.orchestra.run.smithi032.stdout:00000030 00 00 00 00 0a 00 00 00 00 00 00 00 00 00 00 00 |................| 2024-08-03T20:54:40.568 INFO:teuthology.orchestra.run.smithi032.stdout:00000040 00 00 00 00 03 02 28 00 00 00 00 00 00 00 00 00 |......(.........| 2024-08-03T20:54:40.568 INFO:teuthology.orchestra.run.smithi032.stdout:00000050 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| 2024-08-03T20:54:40.568 INFO:teuthology.orchestra.run.smithi032.stdout:00000060 00 00 0a 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| 2024-08-03T20:54:40.568 INFO:teuthology.orchestra.run.smithi032.stdout:00000070 00 00 03 02 38 00 00 00 00 00 00 00 00 00 00 00 |....8...........| 2024-08-03T20:54:40.568 INFO:teuthology.orchestra.run.smithi032.stdout:00000080 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| 2024-08-03T20:54:40.568 INFO:teuthology.orchestra.run.smithi032.stdout:00000090 0a 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| 2024-08-03T20:54:40.568 INFO:teuthology.orchestra.run.smithi032.stdout:000000a0 00 00 00 00 00 00 00 00 03 99 ae 66 9e 9b 84 19 |...........f....| 2024-08-03T20:54:40.568 INFO:teuthology.orchestra.run.smithi032.stdout:000000b0 03 02 38 00 00 00 00 00 00 00 00 00 00 00 00 00 |..8.............| 2024-08-03T20:54:40.568 INFO:teuthology.orchestra.run.smithi032.stdout:000000c0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 0a 00 |................| 2024-08-03T20:54:40.568 INFO:teuthology.orchestra.run.smithi032.stdout:000000d0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| 2024-08-03T20:54:40.569 INFO:teuthology.orchestra.run.smithi032.stdout:000000e0 00 00 00 00 00 00 03 99 ae 66 9e 9b 84 19 00 00 |.........f......| 2024-08-03T20:54:40.569 INFO:teuthology.orchestra.run.smithi032.stdout:000000f0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| 2024-08-03T20:54:40.569 INFO:teuthology.orchestra.run.smithi032.stdout:* 2024-08-03T20:54:40.569 INFO:teuthology.orchestra.run.smithi032.stdout:00000110 00 00 |..| 2024-08-03T20:54:40.569 INFO:teuthology.orchestra.run.smithi032.stdout:00000112 2024-08-03T20:54:40.569 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-03T20:54:40.570 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.566+0000 7f9e36ffef40 1 -- 172.21.15.32:0/397843457 >> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] conn(0x561988d6b610 msgr2=0x561988d6ba20 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:40.570 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.566+0000 7f9e36ffef40 1 --2- 172.21.15.32:0/397843457 >> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] conn(0x561988d6b610 0x561988d6ba20 crc :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:40.570 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.566+0000 7f9e36ffef40 1 -- 172.21.15.32:0/397843457 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f9e10075a20 msgr2=0x7f9e10077f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:40.570 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.566+0000 7f9e36ffef40 1 --2- 172.21.15.32:0/397843457 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f9e10075a20 0x7f9e10077f30 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f9e1c004230 tx=0x7f9e1c002040 comp rx=0 tx=0).stop 2024-08-03T20:54:40.570 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.566+0000 7f9e36ffef40 1 -- 172.21.15.32:0/397843457 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561988d6e360 msgr2=0x561988d65590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:40.570 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.566+0000 7f9e36ffef40 1 --2- 172.21.15.32:0/397843457 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561988d6e360 0x561988d65590 secure :-1 s=READY pgs=975 cs=0 l=1 rev1=1 crypto rx=0x7f9e20046500 tx=0x7f9e200575e0 comp rx=0 tx=0).stop 2024-08-03T20:54:40.570 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.566+0000 7f9e30c00640 1 -- 172.21.15.32:0/397843457 reap_dead start 2024-08-03T20:54:40.570 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.566+0000 7f9e36ffef40 1 -- 172.21.15.32:0/397843457 shutdown_connections 2024-08-03T20:54:40.570 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.566+0000 7f9e36ffef40 1 -- 172.21.15.32:0/397843457 >> 172.21.15.32:0/397843457 conn(0x561988bd7ac0 msgr2=0x561988d8a8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:54:40.570 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.566+0000 7f9e36ffef40 1 -- 172.21.15.32:0/397843457 shutdown_connections 2024-08-03T20:54:40.570 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.566+0000 7f9e36ffef40 1 -- 172.21.15.32:0/397843457 wait complete. 2024-08-03T20:54:40.581 DEBUG:teuthology.orchestra.run.smithi032:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata getomapheader mds0_sessionmap 2024-08-03T20:54:40.722 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.721+0000 7f8628c0ef40 1 Processor -- start 2024-08-03T20:54:40.722 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.721+0000 7f8628c0ef40 1 -- start start 2024-08-03T20:54:40.724 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.722+0000 7f8628c0ef40 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55fffc383010 0x55fffc365790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:40.724 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.722+0000 7f8628c0ef40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55fffc384f50 0x55fffc404b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:40.724 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.722+0000 7f8628c0ef40 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55fffc40d820 0x55fffc364580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:40.724 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.722+0000 7f8628c0ef40 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55fffc289e30 con 0x55fffc383010 2024-08-03T20:54:40.725 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.722+0000 7f8628c0ef40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x55fffc2e7bf0 con 0x55fffc384f50 2024-08-03T20:54:40.725 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.722+0000 7f8628c0ef40 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x55fffc3836f0 con 0x55fffc40d820 2024-08-03T20:54:40.725 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.722+0000 7f8621000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55fffc383010 0x55fffc365790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:40.725 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.722+0000 7f8621000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55fffc383010 0x55fffc365790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:36734/0 (socket says 172.21.15.32:36734) 2024-08-03T20:54:40.725 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.722+0000 7f8621000640 1 -- 172.21.15.32:0/3775086005 learned_addr learned my addr 172.21.15.32:0/3775086005 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:54:40.725 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.723+0000 7f861be00640 1 --2- 172.21.15.32:0/3775086005 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55fffc384f50 0x55fffc404b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:40.725 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.723+0000 7f8621a00640 1 --2- 172.21.15.32:0/3775086005 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55fffc40d820 0x55fffc364580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:40.725 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.723+0000 7f8621000640 1 -- 172.21.15.32:0/3775086005 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55fffc40d820 msgr2=0x55fffc364580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:54:40.725 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.723+0000 7f8621000640 1 --2- 172.21.15.32:0/3775086005 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55fffc40d820 0x55fffc364580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:40.725 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.723+0000 7f8621000640 1 -- 172.21.15.32:0/3775086005 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55fffc384f50 msgr2=0x55fffc404b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:54:40.725 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.723+0000 7f8621000640 1 --2- 172.21.15.32:0/3775086005 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55fffc384f50 0x55fffc404b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:40.725 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.723+0000 7f8621000640 1 -- 172.21.15.32:0/3775086005 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fffc409be0 con 0x55fffc383010 2024-08-03T20:54:40.725 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.723+0000 7f861be00640 1 --2- 172.21.15.32:0/3775086005 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55fffc384f50 0x55fffc404b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-03T20:54:40.727 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.723+0000 7f8621000640 1 --2- 172.21.15.32:0/3775086005 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55fffc383010 0x55fffc365790 secure :-1 s=READY pgs=978 cs=0 l=1 rev1=1 crypto rx=0x7f8614025e40 tx=0x7f8614002fb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c7f27fa1636be6c5 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:54:40.727 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.723+0000 7f861aa00640 1 -- 172.21.15.32:0/3775086005 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f861404d8a0 con 0x55fffc383010 2024-08-03T20:54:40.727 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.723+0000 7f861aa00640 1 -- 172.21.15.32:0/3775086005 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f861404dd50 con 0x55fffc383010 2024-08-03T20:54:40.727 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.723+0000 7f861aa00640 1 -- 172.21.15.32:0/3775086005 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f86140566d0 con 0x55fffc383010 2024-08-03T20:54:40.727 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.724+0000 7f8628c0ef40 1 -- 172.21.15.32:0/3775086005 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55fffc383010 msgr2=0x55fffc365790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:40.727 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.724+0000 7f8628c0ef40 1 --2- 172.21.15.32:0/3775086005 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55fffc383010 0x55fffc365790 secure :-1 s=READY pgs=978 cs=0 l=1 rev1=1 crypto rx=0x7f8614025e40 tx=0x7f8614002fb0 comp rx=0 tx=0).stop 2024-08-03T20:54:40.727 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.724+0000 7f8628c0ef40 1 -- 172.21.15.32:0/3775086005 shutdown_connections 2024-08-03T20:54:40.728 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.724+0000 7f8628c0ef40 1 --2- 172.21.15.32:0/3775086005 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55fffc384f50 0x55fffc404b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:40.728 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.724+0000 7f8628c0ef40 1 --2- 172.21.15.32:0/3775086005 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55fffc40d820 0x55fffc364580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:40.728 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.724+0000 7f8628c0ef40 1 --2- 172.21.15.32:0/3775086005 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55fffc383010 0x55fffc365790 unknown :-1 s=CLOSED pgs=978 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:40.728 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.724+0000 7f8628c0ef40 1 -- 172.21.15.32:0/3775086005 >> 172.21.15.32:0/3775086005 conn(0x55fffc25bac0 msgr2=0x55fffc40ff30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:54:40.728 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.724+0000 7f8628c0ef40 1 -- 172.21.15.32:0/3775086005 shutdown_connections 2024-08-03T20:54:40.728 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.724+0000 7f8628c0ef40 1 -- 172.21.15.32:0/3775086005 wait complete. 2024-08-03T20:54:40.728 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.725+0000 7f8628c0ef40 1 Processor -- start 2024-08-03T20:54:40.728 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.726+0000 7f8628c0ef40 1 -- start start 2024-08-03T20:54:40.730 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.727+0000 7f8628c0ef40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55fffc383010 0x55fffc403fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:40.730 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.727+0000 7f8628c0ef40 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55fffc384f50 0x55fffc40ad70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:40.730 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.727+0000 7f8628c0ef40 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55fffc40d820 0x55fffc408620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:40.730 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.727+0000 7f8628c0ef40 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55fffc409940 con 0x55fffc40d820 2024-08-03T20:54:40.730 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.727+0000 7f8628c0ef40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x55fffc365cd0 con 0x55fffc383010 2024-08-03T20:54:40.730 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.727+0000 7f8628c0ef40 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x55fffc40f5b0 con 0x55fffc384f50 2024-08-03T20:54:40.730 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.727+0000 7f8621a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55fffc40d820 0x55fffc408620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:40.731 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.727+0000 7f8621a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55fffc40d820 0x55fffc408620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:36746/0 (socket says 172.21.15.32:36746) 2024-08-03T20:54:40.731 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.727+0000 7f8621a00640 1 -- 172.21.15.32:0/3459113343 learned_addr learned my addr 172.21.15.32:0/3459113343 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:54:40.731 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.727+0000 7f8621000640 1 --2- 172.21.15.32:0/3459113343 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55fffc383010 0x55fffc403fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:40.731 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.728+0000 7f861be00640 1 --2- 172.21.15.32:0/3459113343 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55fffc384f50 0x55fffc40ad70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:40.731 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.728+0000 7f8621a00640 1 -- 172.21.15.32:0/3459113343 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55fffc384f50 msgr2=0x55fffc40ad70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:40.731 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.728+0000 7f8621a00640 1 --2- 172.21.15.32:0/3459113343 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55fffc384f50 0x55fffc40ad70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:40.731 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.728+0000 7f8621a00640 1 -- 172.21.15.32:0/3459113343 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55fffc383010 msgr2=0x55fffc403fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:40.731 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.728+0000 7f8621a00640 1 --2- 172.21.15.32:0/3459113343 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55fffc383010 0x55fffc403fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:40.731 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.728+0000 7f8621a00640 1 -- 172.21.15.32:0/3459113343 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fffc40ddc0 con 0x55fffc40d820 2024-08-03T20:54:40.731 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.728+0000 7f8621a00640 1 --2- 172.21.15.32:0/3459113343 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55fffc40d820 0x55fffc408620 secure :-1 s=READY pgs=979 cs=0 l=1 rev1=1 crypto rx=0x7f861c006e90 tx=0x7f861c0027a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:54:40.731 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.728+0000 7f8619600640 1 -- 172.21.15.32:0/3459113343 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f861c013480 con 0x55fffc40d820 2024-08-03T20:54:40.731 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.728+0000 7f8619600640 1 -- 172.21.15.32:0/3459113343 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f861c018070 con 0x55fffc40d820 2024-08-03T20:54:40.732 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.728+0000 7f8619600640 1 -- 172.21.15.32:0/3459113343 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f861c0139b0 con 0x55fffc40d820 2024-08-03T20:54:40.732 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.728+0000 7f8628c0ef40 1 -- 172.21.15.32:0/3459113343 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x55fffc40b2b0 con 0x55fffc40d820 2024-08-03T20:54:40.732 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.729+0000 7f8628c0ef40 1 -- 172.21.15.32:0/3459113343 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fffc40a420 con 0x55fffc40d820 2024-08-03T20:54:40.732 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.731+0000 7f8619600640 1 -- 172.21.15.32:0/3459113343 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f861c013b50 con 0x55fffc40d820 2024-08-03T20:54:40.733 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.733+0000 7f8619600640 1 --2- 172.21.15.32:0/3459113343 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f86000759f0 0x7f8600077f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:40.733 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.733+0000 7f8619600640 1 -- 172.21.15.32:0/3459113343 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 5795+0+0 (secure 0 0 0) 0x7f861c097530 con 0x55fffc40d820 2024-08-03T20:54:40.733 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.733+0000 7f8628c0ef40 1 --2- 172.21.15.32:0/3459113343 >> [v2:172.21.15.32:6800/9088303,v1:172.21.15.32:6801/9088303] conn(0x55fffc37fcb0 0x55fffc3800c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:40.733 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.733+0000 7f8621000640 1 --2- 172.21.15.32:0/3459113343 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f86000759f0 0x7f8600077f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:40.733 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.733+0000 7f8628c0ef40 1 -- 172.21.15.32:0/3459113343 --> [v2:172.21.15.32:6800/9088303,v1:172.21.15.32:6801/9088303] -- osd_op(unknown.0.0:1 12.b 12:d0630e4c:::mds0_sessionmap:head [omap-get-header] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84) -- 0x55fffc3638a0 con 0x55fffc37fcb0 2024-08-03T20:54:40.735 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.733+0000 7f861be00640 1 --2- 172.21.15.32:0/3459113343 >> [v2:172.21.15.32:6800/9088303,v1:172.21.15.32:6801/9088303] conn(0x55fffc37fcb0 0x55fffc3800c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:40.735 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.733+0000 7f8621000640 1 --2- 172.21.15.32:0/3459113343 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f86000759f0 0x7f8600077f00 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f8614025e40 tx=0x7f8614004880 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:54:40.735 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.733+0000 7f861be00640 1 --2- 172.21.15.32:0/3459113343 >> [v2:172.21.15.32:6800/9088303,v1:172.21.15.32:6801/9088303] conn(0x55fffc37fcb0 0x55fffc3800c0 crc :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:54:40.735 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.733+0000 7f861be00640 1 -- 172.21.15.32:0/3459113343 <== osd.3 v2:172.21.15.32:6800/9088303 1 ==== osd_op_reply(1 mds0_sessionmap [omap-get-header out=14b] v0'0 uv3 ondisk = 0) ==== 159+0+14 (crc 0 0 0) 0x7f860c00c830 con 0x55fffc37fcb0 2024-08-03T20:54:40.736 INFO:teuthology.orchestra.run.smithi032.stdout:header (14 bytes) : 2024-08-03T20:54:40.736 INFO:teuthology.orchestra.run.smithi032.stdout:00000000 01 01 08 00 00 00 04 00 00 00 00 00 00 00 |..............| 2024-08-03T20:54:40.736 INFO:teuthology.orchestra.run.smithi032.stdout:0000000e 2024-08-03T20:54:40.736 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-03T20:54:40.737 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.733+0000 7f8628c0ef40 1 -- 172.21.15.32:0/3459113343 >> [v2:172.21.15.32:6800/9088303,v1:172.21.15.32:6801/9088303] conn(0x55fffc37fcb0 msgr2=0x55fffc3800c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:40.737 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.733+0000 7f8628c0ef40 1 --2- 172.21.15.32:0/3459113343 >> [v2:172.21.15.32:6800/9088303,v1:172.21.15.32:6801/9088303] conn(0x55fffc37fcb0 0x55fffc3800c0 crc :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:40.737 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.733+0000 7f8628c0ef40 1 -- 172.21.15.32:0/3459113343 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f86000759f0 msgr2=0x7f8600077f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:40.737 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.733+0000 7f8628c0ef40 1 --2- 172.21.15.32:0/3459113343 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f86000759f0 0x7f8600077f00 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f8614025e40 tx=0x7f8614004880 comp rx=0 tx=0).stop 2024-08-03T20:54:40.737 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.733+0000 7f8628c0ef40 1 -- 172.21.15.32:0/3459113343 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55fffc40d820 msgr2=0x55fffc408620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:40.737 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.733+0000 7f8628c0ef40 1 --2- 172.21.15.32:0/3459113343 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55fffc40d820 0x55fffc408620 secure :-1 s=READY pgs=979 cs=0 l=1 rev1=1 crypto rx=0x7f861c006e90 tx=0x7f861c0027a0 comp rx=0 tx=0).stop 2024-08-03T20:54:40.737 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.733+0000 7f8621a00640 1 -- 172.21.15.32:0/3459113343 reap_dead start 2024-08-03T20:54:40.737 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.734+0000 7f8628c0ef40 1 -- 172.21.15.32:0/3459113343 shutdown_connections 2024-08-03T20:54:40.737 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.734+0000 7f8628c0ef40 1 -- 172.21.15.32:0/3459113343 >> 172.21.15.32:0/3459113343 conn(0x55fffc25bac0 msgr2=0x55fffc40ff30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:54:40.738 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.734+0000 7f8628c0ef40 1 -- 172.21.15.32:0/3459113343 shutdown_connections 2024-08-03T20:54:40.738 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.734+0000 7f8628c0ef40 1 -- 172.21.15.32:0/3459113343 wait complete. 2024-08-03T20:54:40.746 DEBUG:teuthology.orchestra.run.smithi032:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata getomapheader 400.00000000 2024-08-03T20:54:40.890 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.889+0000 7fe70a774f40 1 Processor -- start 2024-08-03T20:54:40.891 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.889+0000 7fe70a774f40 1 -- start start 2024-08-03T20:54:40.893 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.891+0000 7fe70a774f40 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55b54b018b40 0x55b54b137000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:40.893 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.891+0000 7fe70a774f40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55b54b08bbe0 0x55b54b137540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:40.893 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.891+0000 7fe70a774f40 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b54afd1680 0x55b54afd1a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:40.893 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.891+0000 7fe70a774f40 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55b54b08c510 con 0x55b54afd1680 2024-08-03T20:54:40.893 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.891+0000 7fe70a774f40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x55b54af21e30 con 0x55b54b08bbe0 2024-08-03T20:54:40.893 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.891+0000 7fe70a774f40 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x55b54b08c690 con 0x55b54b018b40 2024-08-03T20:54:40.893 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.891+0000 7fe704200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b54afd1680 0x55b54afd1a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:40.893 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.891+0000 7fe704200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b54afd1680 0x55b54afd1a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:36772/0 (socket says 172.21.15.32:36772) 2024-08-03T20:54:40.893 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.891+0000 7fe704200640 1 -- 172.21.15.32:0/210598752 learned_addr learned my addr 172.21.15.32:0/210598752 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:54:40.893 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.891+0000 7fe703800640 1 --2- 172.21.15.32:0/210598752 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55b54b018b40 0x55b54b137000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:40.893 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.891+0000 7fe702e00640 1 --2- 172.21.15.32:0/210598752 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55b54b08bbe0 0x55b54b137540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:40.893 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.891+0000 7fe704200640 1 -- 172.21.15.32:0/210598752 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55b54b018b40 msgr2=0x55b54b137000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:54:40.894 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.891+0000 7fe704200640 1 --2- 172.21.15.32:0/210598752 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55b54b018b40 0x55b54b137000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:40.894 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.891+0000 7fe704200640 1 -- 172.21.15.32:0/210598752 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55b54b08bbe0 msgr2=0x55b54b137540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:54:40.894 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.891+0000 7fe704200640 1 --2- 172.21.15.32:0/210598752 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55b54b08bbe0 0x55b54b137540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:40.894 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.891+0000 7fe704200640 1 -- 172.21.15.32:0/210598752 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b54aefaf10 con 0x55b54afd1680 2024-08-03T20:54:40.894 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.891+0000 7fe704200640 1 --2- 172.21.15.32:0/210598752 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b54afd1680 0x55b54afd1a90 secure :-1 s=READY pgs=981 cs=0 l=1 rev1=1 crypto rx=0x7fe6fc00b680 tx=0x7fe6fc00bb40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=47d98d1218144d80 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:54:40.895 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.892+0000 7fe701a00640 1 -- 172.21.15.32:0/210598752 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6fc009a30 con 0x55b54afd1680 2024-08-03T20:54:40.895 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.892+0000 7fe701a00640 1 -- 172.21.15.32:0/210598752 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fe6fc009ce0 con 0x55b54afd1680 2024-08-03T20:54:40.895 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.892+0000 7fe701a00640 1 -- 172.21.15.32:0/210598752 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6fc00e6b0 con 0x55b54afd1680 2024-08-03T20:54:40.896 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.892+0000 7fe70a774f40 1 -- 172.21.15.32:0/210598752 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b54afd1680 msgr2=0x55b54afd1a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:40.896 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.892+0000 7fe70a774f40 1 --2- 172.21.15.32:0/210598752 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b54afd1680 0x55b54afd1a90 secure :-1 s=READY pgs=981 cs=0 l=1 rev1=1 crypto rx=0x7fe6fc00b680 tx=0x7fe6fc00bb40 comp rx=0 tx=0).stop 2024-08-03T20:54:40.896 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.892+0000 7fe70a774f40 1 -- 172.21.15.32:0/210598752 shutdown_connections 2024-08-03T20:54:40.896 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.892+0000 7fe70a774f40 1 --2- 172.21.15.32:0/210598752 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55b54b08bbe0 0x55b54b137540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:40.896 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.892+0000 7fe70a774f40 1 --2- 172.21.15.32:0/210598752 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b54afd1680 0x55b54afd1a90 unknown :-1 s=CLOSED pgs=981 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:40.896 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.892+0000 7fe70a774f40 1 --2- 172.21.15.32:0/210598752 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55b54b018b40 0x55b54b137000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:40.896 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.892+0000 7fe70a774f40 1 -- 172.21.15.32:0/210598752 >> 172.21.15.32:0/210598752 conn(0x55b54aef3ac0 msgr2=0x55b54b0a9ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:54:40.896 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.893+0000 7fe70a774f40 1 -- 172.21.15.32:0/210598752 shutdown_connections 2024-08-03T20:54:40.896 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.893+0000 7fe70a774f40 1 -- 172.21.15.32:0/210598752 wait complete. 2024-08-03T20:54:40.896 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.894+0000 7fe70a774f40 1 Processor -- start 2024-08-03T20:54:40.896 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.894+0000 7fe70a774f40 1 -- start start 2024-08-03T20:54:40.898 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.895+0000 7fe70a774f40 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55b54afd1680 0x55b54afd2030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:40.898 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.895+0000 7fe70a774f40 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b54b018b40 0x55b54b0a7da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:40.898 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.895+0000 7fe70a774f40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55b54b08bbe0 0x55b54b140a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:40.898 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.895+0000 7fe70a774f40 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55b54b01c6a0 con 0x55b54b018b40 2024-08-03T20:54:40.898 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.895+0000 7fe70a774f40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x55b54b017290 con 0x55b54b08bbe0 2024-08-03T20:54:40.899 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.895+0000 7fe70a774f40 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x55b54b0a8fd0 con 0x55b54afd1680 2024-08-03T20:54:40.899 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.896+0000 7fe702e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b54b018b40 0x55b54b0a7da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:40.899 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.896+0000 7fe702e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b54b018b40 0x55b54b0a7da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:36780/0 (socket says 172.21.15.32:36780) 2024-08-03T20:54:40.899 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.896+0000 7fe702e00640 1 -- 172.21.15.32:0/2845845261 learned_addr learned my addr 172.21.15.32:0/2845845261 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:54:40.899 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.896+0000 7fe703800640 1 --2- 172.21.15.32:0/2845845261 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55b54afd1680 0x55b54afd2030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:40.899 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.896+0000 7fe704200640 1 --2- 172.21.15.32:0/2845845261 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55b54b08bbe0 0x55b54b140a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:40.899 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.896+0000 7fe702e00640 1 -- 172.21.15.32:0/2845845261 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55b54afd1680 msgr2=0x55b54afd2030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:40.899 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.896+0000 7fe702e00640 1 --2- 172.21.15.32:0/2845845261 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55b54afd1680 0x55b54afd2030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:40.899 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.896+0000 7fe702e00640 1 -- 172.21.15.32:0/2845845261 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55b54b08bbe0 msgr2=0x55b54b140a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:40.899 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.896+0000 7fe702e00640 1 --2- 172.21.15.32:0/2845845261 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55b54b08bbe0 0x55b54b140a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:40.899 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.896+0000 7fe702e00640 1 -- 172.21.15.32:0/2845845261 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b54b1337e0 con 0x55b54b018b40 2024-08-03T20:54:40.899 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.896+0000 7fe702e00640 1 --2- 172.21.15.32:0/2845845261 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b54b018b40 0x55b54b0a7da0 secure :-1 s=READY pgs=982 cs=0 l=1 rev1=1 crypto rx=0x7fe6f0025f80 tx=0x7fe6f0058c50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:54:40.899 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.897+0000 7fe6fbe00640 1 -- 172.21.15.32:0/2845845261 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6f0004640 con 0x55b54b018b40 2024-08-03T20:54:40.900 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.897+0000 7fe6fbe00640 1 -- 172.21.15.32:0/2845845261 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fe6f0058200 con 0x55b54b018b40 2024-08-03T20:54:40.900 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.897+0000 7fe70a774f40 1 -- 172.21.15.32:0/2845845261 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x55b54b133ac0 con 0x55b54b018b40 2024-08-03T20:54:40.900 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.897+0000 7fe6fbe00640 1 -- 172.21.15.32:0/2845845261 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6f0056470 con 0x55b54b018b40 2024-08-03T20:54:40.900 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.897+0000 7fe70a774f40 1 -- 172.21.15.32:0/2845845261 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b54b140550 con 0x55b54b018b40 2024-08-03T20:54:40.900 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.899+0000 7fe6fbe00640 1 -- 172.21.15.32:0/2845845261 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7fe6f006a020 con 0x55b54b018b40 2024-08-03T20:54:40.902 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.900+0000 7fe6fbe00640 1 --2- 172.21.15.32:0/2845845261 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fe6e40759f0 0x7fe6e4077f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:40.902 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.900+0000 7fe6fbe00640 1 -- 172.21.15.32:0/2845845261 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 5795+0+0 (secure 0 0 0) 0x7fe6f00dbbb0 con 0x55b54b018b40 2024-08-03T20:54:40.902 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.900+0000 7fe70a774f40 1 --2- 172.21.15.32:0/2845845261 >> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] conn(0x55b54b081490 0x55b54b0818a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:40.902 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.900+0000 7fe70a774f40 1 -- 172.21.15.32:0/2845845261 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:1 12.f 12:f1f69726:::400.00000000:head [omap-get-header] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84) -- 0x55b54b135910 con 0x55b54b081490 2024-08-03T20:54:40.902 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.900+0000 7fe703800640 1 --2- 172.21.15.32:0/2845845261 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fe6e40759f0 0x7fe6e4077f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:40.903 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.901+0000 7fe704200640 1 --2- 172.21.15.32:0/2845845261 >> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] conn(0x55b54b081490 0x55b54b0818a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:40.903 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.901+0000 7fe703800640 1 --2- 172.21.15.32:0/2845845261 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fe6e40759f0 0x7fe6e4077f00 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7fe6f4004230 tx=0x7fe6f400d040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:54:40.903 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.901+0000 7fe704200640 1 --2- 172.21.15.32:0/2845845261 >> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] conn(0x55b54b081490 0x55b54b0818a0 crc :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:54:40.903 INFO:teuthology.orchestra.run.smithi032.stdout:header (0 bytes) : 2024-08-03T20:54:40.903 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-03T20:54:40.904 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.902+0000 7fe704200640 1 -- 172.21.15.32:0/2845845261 <== osd.6 v2:172.21.15.118:6825/801947386 1 ==== osd_op_reply(1 400.00000000 [omap-get-header] v0'0 uv1 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe6fc009830 con 0x55b54b081490 2024-08-03T20:54:40.904 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.902+0000 7fe70a774f40 1 -- 172.21.15.32:0/2845845261 >> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] conn(0x55b54b081490 msgr2=0x55b54b0818a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:40.904 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.902+0000 7fe70a774f40 1 --2- 172.21.15.32:0/2845845261 >> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] conn(0x55b54b081490 0x55b54b0818a0 crc :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:40.905 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.902+0000 7fe70a774f40 1 -- 172.21.15.32:0/2845845261 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fe6e40759f0 msgr2=0x7fe6e4077f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:40.905 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.902+0000 7fe70a774f40 1 --2- 172.21.15.32:0/2845845261 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fe6e40759f0 0x7fe6e4077f00 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7fe6f4004230 tx=0x7fe6f400d040 comp rx=0 tx=0).stop 2024-08-03T20:54:40.905 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.902+0000 7fe70a774f40 1 -- 172.21.15.32:0/2845845261 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b54b018b40 msgr2=0x55b54b0a7da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:40.905 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.902+0000 7fe70a774f40 1 --2- 172.21.15.32:0/2845845261 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b54b018b40 0x55b54b0a7da0 secure :-1 s=READY pgs=982 cs=0 l=1 rev1=1 crypto rx=0x7fe6f0025f80 tx=0x7fe6f0058c50 comp rx=0 tx=0).stop 2024-08-03T20:54:40.905 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.902+0000 7fe704200640 1 -- 172.21.15.32:0/2845845261 reap_dead start 2024-08-03T20:54:40.905 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.902+0000 7fe70a774f40 1 -- 172.21.15.32:0/2845845261 shutdown_connections 2024-08-03T20:54:40.905 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.902+0000 7fe70a774f40 1 -- 172.21.15.32:0/2845845261 >> 172.21.15.32:0/2845845261 conn(0x55b54aef3ac0 msgr2=0x55b54b0a9ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:54:40.905 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.902+0000 7fe70a774f40 1 -- 172.21.15.32:0/2845845261 shutdown_connections 2024-08-03T20:54:40.905 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:40.902+0000 7fe70a774f40 1 -- 172.21.15.32:0/2845845261 wait complete. 2024-08-03T20:54:40.914 DEBUG:teuthology.orchestra.run.smithi032:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata getomapheader 100.00000000.inode 2024-08-03T20:54:41.055 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.054+0000 7f08d5a0ef40 1 Processor -- start 2024-08-03T20:54:41.056 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.054+0000 7f08d5a0ef40 1 -- start start 2024-08-03T20:54:41.058 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.056+0000 7f08d5a0ef40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x559f68862360 0x559f6887e680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:41.058 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.056+0000 7f08d5a0ef40 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x559f68863ea0 0x559f68860710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:41.058 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.056+0000 7f08d5a0ef40 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x559f6887fcb0 0x559f68860c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:41.058 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.056+0000 7f08d5a0ef40 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x559f686f9e30 con 0x559f68863ea0 2024-08-03T20:54:41.058 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.056+0000 7f08d5a0ef40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x559f68757bf0 con 0x559f68862360 2024-08-03T20:54:41.058 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.056+0000 7f08d5a0ef40 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x559f688631b0 con 0x559f6887fcb0 2024-08-03T20:54:41.058 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.056+0000 7f08cd600640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x559f68863ea0 0x559f68860710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:41.059 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.056+0000 7f08cd600640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x559f68863ea0 0x559f68860710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:36786/0 (socket says 172.21.15.32:36786) 2024-08-03T20:54:41.059 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.056+0000 7f08cd600640 1 -- 172.21.15.32:0/1486754430 learned_addr learned my addr 172.21.15.32:0/1486754430 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:54:41.059 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.056+0000 7f08ce000640 1 --2- 172.21.15.32:0/1486754430 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x559f68862360 0x559f6887e680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:41.059 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.056+0000 7f08cea00640 1 --2- 172.21.15.32:0/1486754430 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x559f6887fcb0 0x559f68860c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:41.059 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.056+0000 7f08cd600640 1 -- 172.21.15.32:0/1486754430 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x559f6887fcb0 msgr2=0x559f68860c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:54:41.059 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.056+0000 7f08cd600640 1 --2- 172.21.15.32:0/1486754430 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x559f6887fcb0 0x559f68860c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:41.059 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.056+0000 7f08cd600640 1 -- 172.21.15.32:0/1486754430 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x559f68862360 msgr2=0x559f6887e680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:54:41.059 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.056+0000 7f08cd600640 1 --2- 172.21.15.32:0/1486754430 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x559f68862360 0x559f6887e680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:41.059 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.056+0000 7f08cd600640 1 -- 172.21.15.32:0/1486754430 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559f687f34d0 con 0x559f68863ea0 2024-08-03T20:54:41.059 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.056+0000 7f08cd600640 1 --2- 172.21.15.32:0/1486754430 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x559f68863ea0 0x559f68860710 secure :-1 s=READY pgs=985 cs=0 l=1 rev1=1 crypto rx=0x7f08bc00c600 tx=0x7f08bc00cac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=72fc6d51e8ea4de5 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:54:41.059 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.056+0000 7f08c7e00640 1 -- 172.21.15.32:0/1486754430 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08bc00aa90 con 0x559f68863ea0 2024-08-03T20:54:41.059 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.057+0000 7f08c7e00640 1 -- 172.21.15.32:0/1486754430 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f08bc00ac30 con 0x559f68863ea0 2024-08-03T20:54:41.059 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.057+0000 7f08c7e00640 1 -- 172.21.15.32:0/1486754430 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08bc008420 con 0x559f68863ea0 2024-08-03T20:54:41.060 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.057+0000 7f08d5a0ef40 1 -- 172.21.15.32:0/1486754430 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x559f68863ea0 msgr2=0x559f68860710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:41.060 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.057+0000 7f08d5a0ef40 1 --2- 172.21.15.32:0/1486754430 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x559f68863ea0 0x559f68860710 secure :-1 s=READY pgs=985 cs=0 l=1 rev1=1 crypto rx=0x7f08bc00c600 tx=0x7f08bc00cac0 comp rx=0 tx=0).stop 2024-08-03T20:54:41.061 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.057+0000 7f08d5a0ef40 1 -- 172.21.15.32:0/1486754430 shutdown_connections 2024-08-03T20:54:41.061 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.057+0000 7f08d5a0ef40 1 --2- 172.21.15.32:0/1486754430 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x559f6887fcb0 0x559f68860c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:41.061 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.057+0000 7f08d5a0ef40 1 --2- 172.21.15.32:0/1486754430 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x559f68863ea0 0x559f68860710 unknown :-1 s=CLOSED pgs=985 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:41.061 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.057+0000 7f08d5a0ef40 1 --2- 172.21.15.32:0/1486754430 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x559f68862360 0x559f6887e680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:41.061 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.057+0000 7f08d5a0ef40 1 -- 172.21.15.32:0/1486754430 >> 172.21.15.32:0/1486754430 conn(0x559f686cbac0 msgr2=0x559f687f1ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:54:41.061 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.057+0000 7f08d5a0ef40 1 -- 172.21.15.32:0/1486754430 shutdown_connections 2024-08-03T20:54:41.061 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.057+0000 7f08d5a0ef40 1 -- 172.21.15.32:0/1486754430 wait complete. 2024-08-03T20:54:41.061 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.058+0000 7f08d5a0ef40 1 Processor -- start 2024-08-03T20:54:41.061 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.058+0000 7f08d5a0ef40 1 -- start start 2024-08-03T20:54:41.063 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.061+0000 7f08d5a0ef40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x559f68862360 0x559f6885e1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:41.063 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.061+0000 7f08d5a0ef40 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x559f68863ea0 0x559f687f1720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:41.063 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.061+0000 7f08d5a0ef40 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x559f6887fcb0 0x559f6885b620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:41.063 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.061+0000 7f08d5a0ef40 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x559f687f27a0 con 0x559f68863ea0 2024-08-03T20:54:41.063 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.061+0000 7f08d5a0ef40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x559f687f3040 con 0x559f68862360 2024-08-03T20:54:41.063 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.061+0000 7f08d5a0ef40 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x559f6887bcf0 con 0x559f6887fcb0 2024-08-03T20:54:41.063 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.061+0000 7f08cd600640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x559f68863ea0 0x559f687f1720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:41.064 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.061+0000 7f08cd600640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x559f68863ea0 0x559f687f1720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:36796/0 (socket says 172.21.15.32:36796) 2024-08-03T20:54:41.064 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.061+0000 7f08cd600640 1 -- 172.21.15.32:0/605963596 learned_addr learned my addr 172.21.15.32:0/605963596 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:54:41.064 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.061+0000 7f08cea00640 1 --2- 172.21.15.32:0/605963596 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x559f6887fcb0 0x559f6885b620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:41.064 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.061+0000 7f08ce000640 1 --2- 172.21.15.32:0/605963596 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x559f68862360 0x559f6885e1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:41.064 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.061+0000 7f08cd600640 1 -- 172.21.15.32:0/605963596 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x559f6887fcb0 msgr2=0x559f6885b620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:41.064 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.061+0000 7f08cd600640 1 --2- 172.21.15.32:0/605963596 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x559f6887fcb0 0x559f6885b620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:41.064 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.061+0000 7f08cd600640 1 -- 172.21.15.32:0/605963596 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x559f68862360 msgr2=0x559f6885e1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:41.064 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.061+0000 7f08cd600640 1 --2- 172.21.15.32:0/605963596 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x559f68862360 0x559f6885e1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:41.064 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.061+0000 7f08cd600640 1 -- 172.21.15.32:0/605963596 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559f687eeb50 con 0x559f68863ea0 2024-08-03T20:54:41.064 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.061+0000 7f08cd600640 1 --2- 172.21.15.32:0/605963596 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x559f68863ea0 0x559f687f1720 secure :-1 s=READY pgs=986 cs=0 l=1 rev1=1 crypto rx=0x7f08bc000c00 tx=0x7f08bc0146a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:54:41.064 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.061+0000 7f08c6a00640 1 -- 172.21.15.32:0/605963596 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08bc014aa0 con 0x559f68863ea0 2024-08-03T20:54:41.064 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.061+0000 7f08c6a00640 1 -- 172.21.15.32:0/605963596 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f08bc014d30 con 0x559f68863ea0 2024-08-03T20:54:41.065 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.061+0000 7f08c6a00640 1 -- 172.21.15.32:0/605963596 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08bc01d690 con 0x559f68863ea0 2024-08-03T20:54:41.065 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.061+0000 7f08d5a0ef40 1 -- 172.21.15.32:0/605963596 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x559f687eee10 con 0x559f68863ea0 2024-08-03T20:54:41.065 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.061+0000 7f08d5a0ef40 1 -- 172.21.15.32:0/605963596 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559f6885e9f0 con 0x559f68863ea0 2024-08-03T20:54:41.065 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.063+0000 7f08c6a00640 1 -- 172.21.15.32:0/605963596 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f08bc01d830 con 0x559f68863ea0 2024-08-03T20:54:41.066 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.065+0000 7f08c6a00640 1 --2- 172.21.15.32:0/605963596 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f08b00759f0 0x7f08b0077f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:41.066 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.065+0000 7f08c6a00640 1 -- 172.21.15.32:0/605963596 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 5795+0+0 (secure 0 0 0) 0x7f08bc098c50 con 0x559f68863ea0 2024-08-03T20:54:41.066 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.065+0000 7f08d5a0ef40 1 --2- 172.21.15.32:0/605963596 >> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] conn(0x559f68919900 0x559f68919d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:41.066 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.065+0000 7f08d5a0ef40 1 -- 172.21.15.32:0/605963596 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:1 12.1f 12:fe07bba1:::100.00000000.inode:head [omap-get-header] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84) -- 0x559f687ecc80 con 0x559f68919900 2024-08-03T20:54:41.067 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.065+0000 7f08ce000640 1 --2- 172.21.15.32:0/605963596 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f08b00759f0 0x7f08b0077f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:41.067 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.065+0000 7f08cea00640 1 --2- 172.21.15.32:0/605963596 >> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] conn(0x559f68919900 0x559f68919d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:41.067 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.066+0000 7f08cea00640 1 --2- 172.21.15.32:0/605963596 >> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] conn(0x559f68919900 0x559f68919d10 crc :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:54:41.067 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.066+0000 7f08ce000640 1 --2- 172.21.15.32:0/605963596 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f08b00759f0 0x7f08b0077f00 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f08c0055000 tx=0x7f08c0058040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:54:41.068 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.066+0000 7f08cea00640 1 -- 172.21.15.32:0/605963596 <== osd.6 v2:172.21.15.118:6825/801947386 1 ==== osd_op_reply(1 100.00000000.inode [omap-get-header] v0'0 uv3 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7f08c8002740 con 0x559f68919900 2024-08-03T20:54:41.068 INFO:teuthology.orchestra.run.smithi032.stdout:header (0 bytes) : 2024-08-03T20:54:41.068 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-03T20:54:41.069 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.066+0000 7f08d5a0ef40 1 -- 172.21.15.32:0/605963596 >> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] conn(0x559f68919900 msgr2=0x559f68919d10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:41.069 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.066+0000 7f08d5a0ef40 1 --2- 172.21.15.32:0/605963596 >> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] conn(0x559f68919900 0x559f68919d10 crc :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:41.069 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.067+0000 7f08d5a0ef40 1 -- 172.21.15.32:0/605963596 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f08b00759f0 msgr2=0x7f08b0077f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:41.069 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.067+0000 7f08d5a0ef40 1 --2- 172.21.15.32:0/605963596 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f08b00759f0 0x7f08b0077f00 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f08c0055000 tx=0x7f08c0058040 comp rx=0 tx=0).stop 2024-08-03T20:54:41.069 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.067+0000 7f08d5a0ef40 1 -- 172.21.15.32:0/605963596 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x559f68863ea0 msgr2=0x559f687f1720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:41.069 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.067+0000 7f08d5a0ef40 1 --2- 172.21.15.32:0/605963596 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x559f68863ea0 0x559f687f1720 secure :-1 s=READY pgs=986 cs=0 l=1 rev1=1 crypto rx=0x7f08bc000c00 tx=0x7f08bc0146a0 comp rx=0 tx=0).stop 2024-08-03T20:54:41.069 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.067+0000 7f08cea00640 1 -- 172.21.15.32:0/605963596 reap_dead start 2024-08-03T20:54:41.069 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.067+0000 7f08d5a0ef40 1 -- 172.21.15.32:0/605963596 shutdown_connections 2024-08-03T20:54:41.070 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.067+0000 7f08d5a0ef40 1 -- 172.21.15.32:0/605963596 >> 172.21.15.32:0/605963596 conn(0x559f686cbac0 msgr2=0x559f688610d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:54:41.070 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.067+0000 7f08d5a0ef40 1 -- 172.21.15.32:0/605963596 shutdown_connections 2024-08-03T20:54:41.070 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.067+0000 7f08d5a0ef40 1 -- 172.21.15.32:0/605963596 wait complete. 2024-08-03T20:54:41.078 DEBUG:teuthology.orchestra.run.smithi032:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata getomapheader 1.00000000 2024-08-03T20:54:41.235 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.220+0000 7f1124e3bf40 1 Processor -- start 2024-08-03T20:54:41.235 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.220+0000 7f1124e3bf40 1 -- start start 2024-08-03T20:54:41.235 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.222+0000 7f1124e3bf40 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5560de1f9010 0x5560de1fac00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:41.235 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.222+0000 7f1124e3bf40 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x5560de283820 0x5560de3149e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:41.235 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.222+0000 7f1124e3bf40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5560de1b1370 0x5560de314f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:41.235 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.222+0000 7f1124e3bf40 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x5560de0ffe30 con 0x5560de1f9010 2024-08-03T20:54:41.235 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.222+0000 7f1124e3bf40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x5560de15dbf0 con 0x5560de1b1370 2024-08-03T20:54:41.235 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.222+0000 7f1124e3bf40 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x5560de1f96f0 con 0x5560de283820 2024-08-03T20:54:41.236 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.222+0000 7f111e000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5560de1f9010 0x5560de1fac00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:41.236 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.222+0000 7f111e000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5560de1f9010 0x5560de1fac00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:36808/0 (socket says 172.21.15.32:36808) 2024-08-03T20:54:41.236 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.222+0000 7f111e000640 1 -- 172.21.15.32:0/4836472 learned_addr learned my addr 172.21.15.32:0/4836472 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:54:41.236 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.222+0000 7f111e000640 1 -- 172.21.15.32:0/4836472 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x5560de283820 msgr2=0x5560de3149e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:54:41.236 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.222+0000 7f111e000640 1 --2- 172.21.15.32:0/4836472 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x5560de283820 0x5560de3149e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:41.236 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.222+0000 7f111e000640 1 -- 172.21.15.32:0/4836472 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5560de1b1370 msgr2=0x5560de314f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:54:41.236 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.222+0000 7f111e000640 1 --2- 172.21.15.32:0/4836472 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5560de1b1370 0x5560de314f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:41.236 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.222+0000 7f111e000640 1 -- 172.21.15.32:0/4836472 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5560de26f370 con 0x5560de1f9010 2024-08-03T20:54:41.236 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.222+0000 7f111e000640 1 --2- 172.21.15.32:0/4836472 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5560de1f9010 0x5560de1fac00 secure :-1 s=READY pgs=989 cs=0 l=1 rev1=1 crypto rx=0x7f111000b630 tx=0x7f111000baf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f44a494ca0ff2254 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:54:41.236 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.222+0000 7f1117e00640 1 -- 172.21.15.32:0/4836472 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f111000a940 con 0x5560de1f9010 2024-08-03T20:54:41.236 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.222+0000 7f1117e00640 1 -- 172.21.15.32:0/4836472 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f111000aae0 con 0x5560de1f9010 2024-08-03T20:54:41.236 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.223+0000 7f1117e00640 1 -- 172.21.15.32:0/4836472 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1110008450 con 0x5560de1f9010 2024-08-03T20:54:41.236 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.223+0000 7f1124e3bf40 1 -- 172.21.15.32:0/4836472 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5560de1f9010 msgr2=0x5560de1fac00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:41.236 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.223+0000 7f1124e3bf40 1 --2- 172.21.15.32:0/4836472 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5560de1f9010 0x5560de1fac00 secure :-1 s=READY pgs=989 cs=0 l=1 rev1=1 crypto rx=0x7f111000b630 tx=0x7f111000baf0 comp rx=0 tx=0).stop 2024-08-03T20:54:41.237 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.223+0000 7f1124e3bf40 1 -- 172.21.15.32:0/4836472 shutdown_connections 2024-08-03T20:54:41.237 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.223+0000 7f1124e3bf40 1 --2- 172.21.15.32:0/4836472 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5560de1b1370 0x5560de314f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:41.237 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.223+0000 7f1124e3bf40 1 --2- 172.21.15.32:0/4836472 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x5560de283820 0x5560de3149e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:41.237 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.223+0000 7f1124e3bf40 1 --2- 172.21.15.32:0/4836472 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5560de1f9010 0x5560de1fac00 unknown :-1 s=CLOSED pgs=989 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:41.237 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.223+0000 7f1124e3bf40 1 -- 172.21.15.32:0/4836472 >> 172.21.15.32:0/4836472 conn(0x5560de0d1ac0 msgr2=0x5560de1dc150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:54:41.237 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.223+0000 7f1124e3bf40 1 -- 172.21.15.32:0/4836472 shutdown_connections 2024-08-03T20:54:41.237 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.223+0000 7f1124e3bf40 1 -- 172.21.15.32:0/4836472 wait complete. 2024-08-03T20:54:41.237 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.224+0000 7f1124e3bf40 1 Processor -- start 2024-08-03T20:54:41.237 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.224+0000 7f1124e3bf40 1 -- start start 2024-08-03T20:54:41.237 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.226+0000 7f1124e3bf40 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5560de1b1370 0x5560de2801b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:41.237 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.226+0000 7f1124e3bf40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5560de283820 0x5560de26dcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:41.237 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.226+0000 7f1124e3bf40 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x5560de31e1c0 0x5560de1d1a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:41.237 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.226+0000 7f1124e3bf40 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x5560de315460 con 0x5560de1b1370 2024-08-03T20:54:41.238 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.226+0000 7f1124e3bf40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x5560de26f090 con 0x5560de283820 2024-08-03T20:54:41.238 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.226+0000 7f1124e3bf40 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x5560de283dc0 con 0x5560de31e1c0 2024-08-03T20:54:41.238 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.226+0000 7f111e000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5560de1b1370 0x5560de2801b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:41.238 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.226+0000 7f111e000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5560de1b1370 0x5560de2801b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:36814/0 (socket says 172.21.15.32:36814) 2024-08-03T20:54:41.238 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.226+0000 7f111e000640 1 -- 172.21.15.32:0/870531344 learned_addr learned my addr 172.21.15.32:0/870531344 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:54:41.238 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.226+0000 7f111ea00640 1 --2- 172.21.15.32:0/870531344 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x5560de31e1c0 0x5560de1d1a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:41.238 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.226+0000 7f111d600640 1 --2- 172.21.15.32:0/870531344 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5560de283820 0x5560de26dcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:41.238 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.226+0000 7f111e000640 1 -- 172.21.15.32:0/870531344 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x5560de31e1c0 msgr2=0x5560de1d1a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:41.238 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.226+0000 7f111e000640 1 --2- 172.21.15.32:0/870531344 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x5560de31e1c0 0x5560de1d1a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:41.238 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.226+0000 7f111e000640 1 -- 172.21.15.32:0/870531344 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5560de283820 msgr2=0x5560de26dcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:41.238 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.226+0000 7f111e000640 1 --2- 172.21.15.32:0/870531344 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5560de283820 0x5560de26dcc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:41.238 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.227+0000 7f111e000640 1 -- 172.21.15.32:0/870531344 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5560de1f66d0 con 0x5560de1b1370 2024-08-03T20:54:41.238 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.227+0000 7f111e000640 1 --2- 172.21.15.32:0/870531344 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5560de1b1370 0x5560de2801b0 secure :-1 s=READY pgs=990 cs=0 l=1 rev1=1 crypto rx=0x5560de1deeb0 tx=0x7f1110016630 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:54:41.239 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.227+0000 7f1116a00640 1 -- 172.21.15.32:0/870531344 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1110016e00 con 0x5560de1b1370 2024-08-03T20:54:41.239 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.227+0000 7f1116a00640 1 -- 172.21.15.32:0/870531344 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f1110020420 con 0x5560de1b1370 2024-08-03T20:54:41.239 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.227+0000 7f1116a00640 1 -- 172.21.15.32:0/870531344 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1110020630 con 0x5560de1b1370 2024-08-03T20:54:41.239 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.227+0000 7f1124e3bf40 1 -- 172.21.15.32:0/870531344 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x5560de1d9280 con 0x5560de1b1370 2024-08-03T20:54:41.239 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.227+0000 7f1124e3bf40 1 -- 172.21.15.32:0/870531344 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5560de1f63f0 con 0x5560de1b1370 2024-08-03T20:54:41.239 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.231+0000 7f1116a00640 1 -- 172.21.15.32:0/870531344 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f1110029440 con 0x5560de1b1370 2024-08-03T20:54:41.239 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.231+0000 7f1116a00640 1 --2- 172.21.15.32:0/870531344 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f11000759f0 0x7f1100077f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:41.239 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.232+0000 7f1116a00640 1 -- 172.21.15.32:0/870531344 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 5795+0+0 (secure 0 0 0) 0x7f1110099830 con 0x5560de1b1370 2024-08-03T20:54:41.239 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.232+0000 7f1124e3bf40 1 --2- 172.21.15.32:0/870531344 >> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] conn(0x5560de285ee0 0x5560de1f48a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:41.239 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.232+0000 7f111d600640 1 --2- 172.21.15.32:0/870531344 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f11000759f0 0x7f1100077f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:41.239 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.232+0000 7f1124e3bf40 1 -- 172.21.15.32:0/870531344 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:1 12.1f 12:ff5b34d6:::1.00000000:head [omap-get-header] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84) -- 0x5560de1fad50 con 0x5560de285ee0 2024-08-03T20:54:41.239 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.232+0000 7f111ea00640 1 --2- 172.21.15.32:0/870531344 >> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] conn(0x5560de285ee0 0x5560de1f48a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:41.239 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.233+0000 7f111d600640 1 --2- 172.21.15.32:0/870531344 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f11000759f0 0x7f1100077f00 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x5560de1b1f10 tx=0x7f110c003480 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:54:41.240 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.233+0000 7f111ea00640 1 --2- 172.21.15.32:0/870531344 >> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] conn(0x5560de285ee0 0x5560de1f48a0 crc :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:54:41.240 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.233+0000 7f111ea00640 1 -- 172.21.15.32:0/870531344 <== osd.6 v2:172.21.15.118:6825/801947386 1 ==== osd_op_reply(1 1.00000000 [omap-get-header out=274b] v0'0 uv6 ondisk = 0) ==== 154+0+274 (crc 0 0 0) 0x7f111800b040 con 0x5560de285ee0 2024-08-03T20:54:41.240 INFO:teuthology.orchestra.run.smithi032.stdout:header (274 bytes) : 2024-08-03T20:54:41.240 INFO:teuthology.orchestra.run.smithi032.stdout:00000000 04 03 0c 01 00 00 0d 00 00 00 00 00 00 00 00 00 |................| 2024-08-03T20:54:41.240 INFO:teuthology.orchestra.run.smithi032.stdout:00000010 00 00 00 00 00 00 03 02 28 00 00 00 00 00 00 00 |........(.......| 2024-08-03T20:54:41.240 INFO:teuthology.orchestra.run.smithi032.stdout:00000020 00 00 00 00 0b 99 ae 66 8a d6 0e 38 00 00 00 00 |.......f...8....| 2024-08-03T20:54:41.240 INFO:teuthology.orchestra.run.smithi032.stdout:00000030 00 00 00 00 01 00 00 00 00 00 00 00 01 00 00 00 |................| 2024-08-03T20:54:41.240 INFO:teuthology.orchestra.run.smithi032.stdout:00000040 00 00 00 00 03 02 28 00 00 00 00 00 00 00 00 00 |......(.........| 2024-08-03T20:54:41.240 INFO:teuthology.orchestra.run.smithi032.stdout:00000050 00 00 0b 99 ae 66 8a d6 0e 38 00 00 00 00 00 00 |.....f...8......| 2024-08-03T20:54:41.241 INFO:teuthology.orchestra.run.smithi032.stdout:00000060 00 00 01 00 00 00 00 00 00 00 01 00 00 00 00 00 |................| 2024-08-03T20:54:41.241 INFO:teuthology.orchestra.run.smithi032.stdout:00000070 00 00 03 02 38 00 00 00 00 00 00 00 00 00 00 00 |....8...........| 2024-08-03T20:54:41.241 INFO:teuthology.orchestra.run.smithi032.stdout:00000080 00 00 60 00 00 00 00 00 01 00 00 00 00 00 00 00 |..`.............| 2024-08-03T20:54:41.241 INFO:teuthology.orchestra.run.smithi032.stdout:00000090 01 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| 2024-08-03T20:54:41.241 INFO:teuthology.orchestra.run.smithi032.stdout:000000a0 00 00 00 00 00 00 00 00 0c 99 ae 66 68 ad e5 07 |...........fh...| 2024-08-03T20:54:41.241 INFO:teuthology.orchestra.run.smithi032.stdout:000000b0 03 02 38 00 00 00 00 00 00 00 00 00 00 00 00 00 |..8.............| 2024-08-03T20:54:41.241 INFO:teuthology.orchestra.run.smithi032.stdout:000000c0 60 00 00 00 00 00 01 00 00 00 00 00 00 00 01 00 |`...............| 2024-08-03T20:54:41.241 INFO:teuthology.orchestra.run.smithi032.stdout:000000d0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| 2024-08-03T20:54:41.241 INFO:teuthology.orchestra.run.smithi032.stdout:000000e0 00 00 00 00 00 00 0c 99 ae 66 68 ad e5 07 00 00 |.........fh.....| 2024-08-03T20:54:41.241 INFO:teuthology.orchestra.run.smithi032.stdout:000000f0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| 2024-08-03T20:54:41.241 INFO:teuthology.orchestra.run.smithi032.stdout:* 2024-08-03T20:54:41.241 INFO:teuthology.orchestra.run.smithi032.stdout:00000110 00 00 |..| 2024-08-03T20:54:41.241 INFO:teuthology.orchestra.run.smithi032.stdout:00000112 2024-08-03T20:54:41.242 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-03T20:54:41.242 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.233+0000 7f1124e3bf40 1 -- 172.21.15.32:0/870531344 >> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] conn(0x5560de285ee0 msgr2=0x5560de1f48a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:41.243 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.233+0000 7f1124e3bf40 1 --2- 172.21.15.32:0/870531344 >> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] conn(0x5560de285ee0 0x5560de1f48a0 crc :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:41.243 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.233+0000 7f1124e3bf40 1 -- 172.21.15.32:0/870531344 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f11000759f0 msgr2=0x7f1100077f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:41.243 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.233+0000 7f1124e3bf40 1 --2- 172.21.15.32:0/870531344 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f11000759f0 0x7f1100077f00 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x5560de1b1f10 tx=0x7f110c003480 comp rx=0 tx=0).stop 2024-08-03T20:54:41.243 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.233+0000 7f1124e3bf40 1 -- 172.21.15.32:0/870531344 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5560de1b1370 msgr2=0x5560de2801b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:41.243 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.233+0000 7f1124e3bf40 1 --2- 172.21.15.32:0/870531344 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5560de1b1370 0x5560de2801b0 secure :-1 s=READY pgs=990 cs=0 l=1 rev1=1 crypto rx=0x5560de1deeb0 tx=0x7f1110016630 comp rx=0 tx=0).stop 2024-08-03T20:54:41.243 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.233+0000 7f111ea00640 1 -- 172.21.15.32:0/870531344 reap_dead start 2024-08-03T20:54:41.243 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.233+0000 7f1124e3bf40 1 -- 172.21.15.32:0/870531344 shutdown_connections 2024-08-03T20:54:41.243 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.233+0000 7f1124e3bf40 1 -- 172.21.15.32:0/870531344 >> 172.21.15.32:0/870531344 conn(0x5560de0d1ac0 msgr2=0x5560de1dc150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:54:41.243 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.233+0000 7f1124e3bf40 1 -- 172.21.15.32:0/870531344 shutdown_connections 2024-08-03T20:54:41.243 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.233+0000 7f1124e3bf40 1 -- 172.21.15.32:0/870531344 wait complete. 2024-08-03T20:54:41.244 DEBUG:teuthology.orchestra.run.smithi032:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata listomapkeys 10000000000.00000000 2024-08-03T20:54:41.386 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.385+0000 7f169515bf40 1 Processor -- start 2024-08-03T20:54:41.386 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.386+0000 7f169515bf40 1 -- start start 2024-08-03T20:54:41.388 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.387+0000 7f169515bf40 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x56275980c360 0x56275979d7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:41.388 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.387+0000 7f169515bf40 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56275980dea0 0x56275980b310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:41.388 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.387+0000 7f169515bf40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x562759829cb0 0x5627598b68a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:41.388 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.387+0000 7f169515bf40 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x5627596a3e30 con 0x56275980dea0 2024-08-03T20:54:41.388 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.387+0000 7f169515bf40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x562759701bf0 con 0x562759829cb0 2024-08-03T20:54:41.388 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.387+0000 7f169515bf40 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x56275980d1b0 con 0x56275980c360 2024-08-03T20:54:41.388 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.387+0000 7f168d800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56275980dea0 0x56275980b310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:41.388 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.387+0000 7f168d800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56275980dea0 0x56275980b310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:36824/0 (socket says 172.21.15.32:36824) 2024-08-03T20:54:41.388 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.387+0000 7f168d800640 1 -- 172.21.15.32:0/3285290620 learned_addr learned my addr 172.21.15.32:0/3285290620 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:54:41.390 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.388+0000 7f168e200640 1 --2- 172.21.15.32:0/3285290620 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x56275980c360 0x56275979d7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:41.390 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.388+0000 7f168ec00640 1 --2- 172.21.15.32:0/3285290620 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x562759829cb0 0x5627598b68a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:41.390 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.388+0000 7f168d800640 1 -- 172.21.15.32:0/3285290620 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x56275980c360 msgr2=0x56275979d7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:54:41.390 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.388+0000 7f168d800640 1 --2- 172.21.15.32:0/3285290620 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x56275980c360 0x56275979d7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:41.390 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.388+0000 7f168d800640 1 -- 172.21.15.32:0/3285290620 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x562759829cb0 msgr2=0x5627598b68a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:54:41.390 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.388+0000 7f168d800640 1 --2- 172.21.15.32:0/3285290620 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x562759829cb0 0x5627598b68a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:41.390 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.388+0000 7f168d800640 1 -- 172.21.15.32:0/3285290620 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56275979d490 con 0x56275980dea0 2024-08-03T20:54:41.390 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.388+0000 7f168d800640 1 --2- 172.21.15.32:0/3285290620 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56275980dea0 0x56275980b310 secure :-1 s=READY pgs=991 cs=0 l=1 rev1=1 crypto rx=0x7f167c00b600 tx=0x7f167c00bac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e5a4c022aff4ef2a server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:54:41.390 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.388+0000 7f1687e00640 1 -- 172.21.15.32:0/3285290620 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f167c00a8f0 con 0x56275980dea0 2024-08-03T20:54:41.391 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.388+0000 7f1687e00640 1 -- 172.21.15.32:0/3285290620 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f167c00aa90 con 0x56275980dea0 2024-08-03T20:54:41.391 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.388+0000 7f1687e00640 1 -- 172.21.15.32:0/3285290620 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f167c008450 con 0x56275980dea0 2024-08-03T20:54:41.391 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.389+0000 7f169515bf40 1 -- 172.21.15.32:0/3285290620 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56275980dea0 msgr2=0x56275980b310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:41.391 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.389+0000 7f169515bf40 1 --2- 172.21.15.32:0/3285290620 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56275980dea0 0x56275980b310 secure :-1 s=READY pgs=991 cs=0 l=1 rev1=1 crypto rx=0x7f167c00b600 tx=0x7f167c00bac0 comp rx=0 tx=0).stop 2024-08-03T20:54:41.391 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.389+0000 7f169515bf40 1 -- 172.21.15.32:0/3285290620 shutdown_connections 2024-08-03T20:54:41.391 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.389+0000 7f169515bf40 1 --2- 172.21.15.32:0/3285290620 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x562759829cb0 0x5627598b68a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:41.391 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.389+0000 7f169515bf40 1 --2- 172.21.15.32:0/3285290620 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56275980dea0 0x56275980b310 unknown :-1 s=CLOSED pgs=991 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:41.391 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.389+0000 7f169515bf40 1 --2- 172.21.15.32:0/3285290620 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x56275980c360 0x56275979d7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:41.391 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.389+0000 7f169515bf40 1 -- 172.21.15.32:0/3285290620 >> 172.21.15.32:0/3285290620 conn(0x562759675ac0 msgr2=0x56275979bff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:54:41.391 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.389+0000 7f169515bf40 1 -- 172.21.15.32:0/3285290620 shutdown_connections 2024-08-03T20:54:41.391 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.389+0000 7f169515bf40 1 -- 172.21.15.32:0/3285290620 wait complete. 2024-08-03T20:54:41.393 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.390+0000 7f169515bf40 1 Processor -- start 2024-08-03T20:54:41.393 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.390+0000 7f169515bf40 1 -- start start 2024-08-03T20:54:41.393 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.392+0000 7f169515bf40 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56275980c360 0x562759806300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:41.393 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.392+0000 7f169515bf40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56275980dea0 0x5627597961f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:41.393 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.392+0000 7f169515bf40 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x562759829cb0 0x562759796730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:41.393 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.392+0000 7f169515bf40 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x56275979d240 con 0x56275980c360 2024-08-03T20:54:41.393 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.392+0000 7f169515bf40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x562759826330 con 0x56275980dea0 2024-08-03T20:54:41.393 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.392+0000 7f169515bf40 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x562759826580 con 0x562759829cb0 2024-08-03T20:54:41.394 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.392+0000 7f168e200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56275980c360 0x562759806300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:41.394 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.392+0000 7f168e200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56275980c360 0x562759806300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:36840/0 (socket says 172.21.15.32:36840) 2024-08-03T20:54:41.394 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.392+0000 7f168e200640 1 -- 172.21.15.32:0/2448524815 learned_addr learned my addr 172.21.15.32:0/2448524815 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:54:41.394 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.392+0000 7f168d800640 1 --2- 172.21.15.32:0/2448524815 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56275980dea0 0x5627597961f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:41.394 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.392+0000 7f168ec00640 1 --2- 172.21.15.32:0/2448524815 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x562759829cb0 0x562759796730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:41.394 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.392+0000 7f168e200640 1 -- 172.21.15.32:0/2448524815 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x562759829cb0 msgr2=0x562759796730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:41.394 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.392+0000 7f168e200640 1 --2- 172.21.15.32:0/2448524815 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x562759829cb0 0x562759796730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:41.394 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.392+0000 7f168e200640 1 -- 172.21.15.32:0/2448524815 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56275980dea0 msgr2=0x5627597961f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:41.394 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.392+0000 7f168e200640 1 --2- 172.21.15.32:0/2448524815 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56275980dea0 0x5627597961f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:41.394 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.392+0000 7f168e200640 1 -- 172.21.15.32:0/2448524815 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5627598b35d0 con 0x56275980c360 2024-08-03T20:54:41.394 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.392+0000 7f168d800640 1 --2- 172.21.15.32:0/2448524815 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56275980dea0 0x5627597961f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-03T20:54:41.394 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.392+0000 7f168ec00640 1 --2- 172.21.15.32:0/2448524815 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x562759829cb0 0x562759796730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-03T20:54:41.394 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.393+0000 7f168e200640 1 --2- 172.21.15.32:0/2448524815 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56275980c360 0x562759806300 secure :-1 s=READY pgs=992 cs=0 l=1 rev1=1 crypto rx=0x7f1680046560 tx=0x7f16800046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:54:41.395 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.393+0000 7f1686a00640 1 -- 172.21.15.32:0/2448524815 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1680067070 con 0x56275980c360 2024-08-03T20:54:41.395 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.393+0000 7f1686a00640 1 -- 172.21.15.32:0/2448524815 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f1680062780 con 0x56275980c360 2024-08-03T20:54:41.395 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.393+0000 7f169515bf40 1 -- 172.21.15.32:0/2448524815 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x5627598b38b0 con 0x56275980c360 2024-08-03T20:54:41.395 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.393+0000 7f1686a00640 1 -- 172.21.15.32:0/2448524815 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1680062bd0 con 0x56275980c360 2024-08-03T20:54:41.395 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.393+0000 7f169515bf40 1 -- 172.21.15.32:0/2448524815 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562759755c10 con 0x56275980c360 2024-08-03T20:54:41.396 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.395+0000 7f1686a00640 1 -- 172.21.15.32:0/2448524815 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f168006a070 con 0x56275980c360 2024-08-03T20:54:41.397 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.396+0000 7f1686a00640 1 --2- 172.21.15.32:0/2448524815 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f1670075a20 0x7f1670077f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:41.397 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.396+0000 7f1686a00640 1 -- 172.21.15.32:0/2448524815 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 5795+0+0 (secure 0 0 0) 0x7f16800db1b0 con 0x56275980c360 2024-08-03T20:54:41.397 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.397+0000 7f169515bf40 1 --2- 172.21.15.32:0/2448524815 >> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] conn(0x5627598b5c80 0x5627598ce4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:41.398 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.397+0000 7f169515bf40 1 -- 172.21.15.32:0/2448524815 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:1 12.10 12:0cf6ad0f:::10000000000.00000000:head [omap-get-keys in=12b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84) -- 0x5627598b7070 con 0x5627598b5c80 2024-08-03T20:54:41.398 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.397+0000 7f168d800640 1 --2- 172.21.15.32:0/2448524815 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f1670075a20 0x7f1670077f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:41.398 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.397+0000 7f168ec00640 1 --2- 172.21.15.32:0/2448524815 >> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] conn(0x5627598b5c80 0x5627598ce4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:41.398 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.398+0000 7f168ec00640 1 --2- 172.21.15.32:0/2448524815 >> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] conn(0x5627598b5c80 0x5627598ce4c0 crc :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:54:41.398 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.398+0000 7f168d800640 1 --2- 172.21.15.32:0/2448524815 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f1670075a20 0x7f1670077f30 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f167c000c00 tx=0x7f167c002fb0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:54:41.399 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.398+0000 7f168ec00640 1 -- 172.21.15.32:0/2448524815 <== osd.0 v2:172.21.15.32:6816/3066156116 1 ==== osd_op_reply(1 10000000000.00000000 [omap-get-keys out=21b] v0'0 uv10 ondisk = 0) ==== 164+0+21 (crc 0 0 0) 0x7f16880015f0 con 0x5627598b5c80 2024-08-03T20:54:41.399 INFO:teuthology.orchestra.run.smithi032.stdout:sixmegs_head 2024-08-03T20:54:41.400 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.398+0000 7f169515bf40 1 -- 172.21.15.32:0/2448524815 >> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] conn(0x5627598b5c80 msgr2=0x5627598ce4c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:41.400 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.398+0000 7f169515bf40 1 --2- 172.21.15.32:0/2448524815 >> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] conn(0x5627598b5c80 0x5627598ce4c0 crc :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:41.400 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.398+0000 7f169515bf40 1 -- 172.21.15.32:0/2448524815 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f1670075a20 msgr2=0x7f1670077f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:41.400 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.398+0000 7f169515bf40 1 --2- 172.21.15.32:0/2448524815 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f1670075a20 0x7f1670077f30 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f167c000c00 tx=0x7f167c002fb0 comp rx=0 tx=0).stop 2024-08-03T20:54:41.400 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.398+0000 7f169515bf40 1 -- 172.21.15.32:0/2448524815 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56275980c360 msgr2=0x562759806300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:41.400 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.398+0000 7f169515bf40 1 --2- 172.21.15.32:0/2448524815 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56275980c360 0x562759806300 secure :-1 s=READY pgs=992 cs=0 l=1 rev1=1 crypto rx=0x7f1680046560 tx=0x7f16800046c0 comp rx=0 tx=0).stop 2024-08-03T20:54:41.401 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.398+0000 7f168ec00640 1 -- 172.21.15.32:0/2448524815 reap_dead start 2024-08-03T20:54:41.401 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.398+0000 7f169515bf40 1 -- 172.21.15.32:0/2448524815 shutdown_connections 2024-08-03T20:54:41.401 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.398+0000 7f169515bf40 1 -- 172.21.15.32:0/2448524815 >> 172.21.15.32:0/2448524815 conn(0x562759675ac0 msgr2=0x562759827880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:54:41.401 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.398+0000 7f169515bf40 1 -- 172.21.15.32:0/2448524815 shutdown_connections 2024-08-03T20:54:41.401 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.399+0000 7f169515bf40 1 -- 172.21.15.32:0/2448524815 wait complete. 2024-08-03T20:54:41.409 DEBUG:teuthology.orchestra.run.smithi032:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata listomapkeys 600.00000000 2024-08-03T20:54:41.552 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.551+0000 7fc963028f40 1 Processor -- start 2024-08-03T20:54:41.552 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.551+0000 7fc963028f40 1 -- start start 2024-08-03T20:54:41.554 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.553+0000 7fc963028f40 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fc940003680 0x55e2e792ec10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:41.555 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.553+0000 7fc963028f40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55e2e7978d50 0x55e2e7a927e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:41.555 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.553+0000 7fc963028f40 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55e2e7a92d20 0x55e2e7a93130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:41.555 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.553+0000 7fc963028f40 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55e2e78dbbf0 con 0x55e2e7a92d20 2024-08-03T20:54:41.555 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.553+0000 7fc963028f40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x55e2e78e86b0 con 0x55e2e7978d50 2024-08-03T20:54:41.555 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.553+0000 7fc963028f40 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x55e2e787de30 con 0x7fc940003680 2024-08-03T20:54:41.555 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.553+0000 7fc95cc00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55e2e7a92d20 0x55e2e7a93130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:41.555 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.553+0000 7fc95cc00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55e2e7a92d20 0x55e2e7a93130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:36858/0 (socket says 172.21.15.32:36858) 2024-08-03T20:54:41.555 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.553+0000 7fc95cc00640 1 -- 172.21.15.32:0/2909051389 learned_addr learned my addr 172.21.15.32:0/2909051389 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:54:41.555 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.553+0000 7fc95c200640 1 --2- 172.21.15.32:0/2909051389 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fc940003680 0x55e2e792ec10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:41.555 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.553+0000 7fc95b800640 1 --2- 172.21.15.32:0/2909051389 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55e2e7978d50 0x55e2e7a927e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:41.555 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.554+0000 7fc95cc00640 1 -- 172.21.15.32:0/2909051389 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fc940003680 msgr2=0x55e2e792ec10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:54:41.556 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.554+0000 7fc95cc00640 1 --2- 172.21.15.32:0/2909051389 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fc940003680 0x55e2e792ec10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:41.556 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.554+0000 7fc95cc00640 1 -- 172.21.15.32:0/2909051389 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55e2e7978d50 msgr2=0x55e2e7a927e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:54:41.556 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.554+0000 7fc95cc00640 1 --2- 172.21.15.32:0/2909051389 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55e2e7978d50 0x55e2e7a927e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:41.556 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.554+0000 7fc95cc00640 1 -- 172.21.15.32:0/2909051389 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e2e792f9d0 con 0x55e2e7a92d20 2024-08-03T20:54:41.556 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.554+0000 7fc95cc00640 1 --2- 172.21.15.32:0/2909051389 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55e2e7a92d20 0x55e2e7a93130 secure :-1 s=READY pgs=995 cs=0 l=1 rev1=1 crypto rx=0x7fc9540469c0 tx=0x7fc954001d30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=42f7cc1ea915e79b server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:54:41.556 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.554+0000 7fc95a400640 1 -- 172.21.15.32:0/2909051389 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc954001a70 con 0x55e2e7a92d20 2024-08-03T20:54:41.556 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.554+0000 7fc95a400640 1 -- 172.21.15.32:0/2909051389 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fc9540090d0 con 0x55e2e7a92d20 2024-08-03T20:54:41.556 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.554+0000 7fc95a400640 1 -- 172.21.15.32:0/2909051389 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc95404db20 con 0x55e2e7a92d20 2024-08-03T20:54:41.557 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.555+0000 7fc963028f40 1 -- 172.21.15.32:0/2909051389 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55e2e7a92d20 msgr2=0x55e2e7a93130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:41.557 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.555+0000 7fc963028f40 1 --2- 172.21.15.32:0/2909051389 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55e2e7a92d20 0x55e2e7a93130 secure :-1 s=READY pgs=995 cs=0 l=1 rev1=1 crypto rx=0x7fc9540469c0 tx=0x7fc954001d30 comp rx=0 tx=0).stop 2024-08-03T20:54:41.557 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.555+0000 7fc963028f40 1 -- 172.21.15.32:0/2909051389 shutdown_connections 2024-08-03T20:54:41.557 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.555+0000 7fc963028f40 1 --2- 172.21.15.32:0/2909051389 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55e2e7978d50 0x55e2e7a927e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:41.557 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.555+0000 7fc963028f40 1 --2- 172.21.15.32:0/2909051389 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55e2e7a92d20 0x55e2e7a93130 unknown :-1 s=CLOSED pgs=995 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:41.557 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.555+0000 7fc963028f40 1 --2- 172.21.15.32:0/2909051389 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fc940003680 0x55e2e792ec10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:41.558 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.555+0000 7fc963028f40 1 -- 172.21.15.32:0/2909051389 >> 172.21.15.32:0/2909051389 conn(0x55e2e784fac0 msgr2=0x55e2e79e5190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:54:41.558 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.555+0000 7fc963028f40 1 -- 172.21.15.32:0/2909051389 shutdown_connections 2024-08-03T20:54:41.558 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.555+0000 7fc963028f40 1 -- 172.21.15.32:0/2909051389 wait complete. 2024-08-03T20:54:41.558 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.556+0000 7fc963028f40 1 Processor -- start 2024-08-03T20:54:41.558 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.556+0000 7fc963028f40 1 -- start start 2024-08-03T20:54:41.560 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.557+0000 7fc963028f40 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55e2e7978d50 0x55e2e79e4170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:41.560 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.557+0000 7fc963028f40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55e2e7a92d20 0x55e2e7a071e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:41.560 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.557+0000 7fc963028f40 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc940003680 0x55e2e7a07720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:41.560 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.557+0000 7fc963028f40 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55e2e7a93670 con 0x7fc940003680 2024-08-03T20:54:41.560 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.557+0000 7fc963028f40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x55e2e7a92b20 con 0x55e2e7a92d20 2024-08-03T20:54:41.560 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.558+0000 7fc963028f40 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x55e2e7978660 con 0x55e2e7978d50 2024-08-03T20:54:41.560 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.558+0000 7fc95cc00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc940003680 0x55e2e7a07720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:41.560 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.558+0000 7fc95cc00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc940003680 0x55e2e7a07720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:36868/0 (socket says 172.21.15.32:36868) 2024-08-03T20:54:41.560 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.558+0000 7fc95cc00640 1 -- 172.21.15.32:0/1049645833 learned_addr learned my addr 172.21.15.32:0/1049645833 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:54:41.560 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.558+0000 7fc95b800640 1 --2- 172.21.15.32:0/1049645833 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55e2e7a92d20 0x55e2e7a071e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:41.560 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.558+0000 7fc95cc00640 1 -- 172.21.15.32:0/1049645833 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55e2e7978d50 msgr2=0x55e2e79e4170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:41.561 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.558+0000 7fc95cc00640 1 --2- 172.21.15.32:0/1049645833 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55e2e7978d50 0x55e2e79e4170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:41.561 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.558+0000 7fc95cc00640 1 -- 172.21.15.32:0/1049645833 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55e2e7a92d20 msgr2=0x55e2e7a071e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:41.561 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.558+0000 7fc95cc00640 1 --2- 172.21.15.32:0/1049645833 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55e2e7a92d20 0x55e2e7a071e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:41.561 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.558+0000 7fc95cc00640 1 -- 172.21.15.32:0/1049645833 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e2e79e46b0 con 0x7fc940003680 2024-08-03T20:54:41.561 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.558+0000 7fc95cc00640 1 --2- 172.21.15.32:0/1049645833 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc940003680 0x55e2e7a07720 secure :-1 s=READY pgs=996 cs=0 l=1 rev1=1 crypto rx=0x7fc954009da0 tx=0x7fc954001580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:54:41.561 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.559+0000 7fc959000640 1 -- 172.21.15.32:0/1049645833 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9540090d0 con 0x7fc940003680 2024-08-03T20:54:41.561 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.559+0000 7fc963028f40 1 -- 172.21.15.32:0/1049645833 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x55e2e7a916a0 con 0x7fc940003680 2024-08-03T20:54:41.561 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.559+0000 7fc959000640 1 -- 172.21.15.32:0/1049645833 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fc954009270 con 0x7fc940003680 2024-08-03T20:54:41.561 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.559+0000 7fc959000640 1 -- 172.21.15.32:0/1049645833 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc95404dd80 con 0x7fc940003680 2024-08-03T20:54:41.561 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.559+0000 7fc963028f40 1 -- 172.21.15.32:0/1049645833 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e2e79788b0 con 0x7fc940003680 2024-08-03T20:54:41.562 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.561+0000 7fc959000640 1 -- 172.21.15.32:0/1049645833 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7fc954007580 con 0x7fc940003680 2024-08-03T20:54:41.562 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.562+0000 7fc959000640 1 --2- 172.21.15.32:0/1049645833 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fc93c075a20 0x7fc93c077f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:41.562 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.562+0000 7fc959000640 1 -- 172.21.15.32:0/1049645833 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 5795+0+0 (secure 0 0 0) 0x7fc9540dbe00 con 0x7fc940003680 2024-08-03T20:54:41.563 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.562+0000 7fc963028f40 1 --2- 172.21.15.32:0/1049645833 >> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] conn(0x55e2e79e48b0 0x55e2e7aa0260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:41.563 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.562+0000 7fc963028f40 1 -- 172.21.15.32:0/1049645833 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:1 12.10 12:0d82a743:::600.00000000:head [omap-get-keys in=12b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84) -- 0x55e2e792ed30 con 0x55e2e79e48b0 2024-08-03T20:54:41.563 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.562+0000 7fc95c200640 1 --2- 172.21.15.32:0/1049645833 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fc93c075a20 0x7fc93c077f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:41.563 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.562+0000 7fc95b800640 1 --2- 172.21.15.32:0/1049645833 >> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] conn(0x55e2e79e48b0 0x55e2e7aa0260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:41.564 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.563+0000 7fc95b800640 1 --2- 172.21.15.32:0/1049645833 >> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] conn(0x55e2e79e48b0 0x55e2e7aa0260 crc :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:54:41.564 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.563+0000 7fc95c200640 1 --2- 172.21.15.32:0/1049645833 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fc93c075a20 0x7fc93c077f30 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7fc94c00d570 tx=0x7fc94c004c60 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:54:41.564 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.563+0000 7fc95b800640 1 -- 172.21.15.32:0/1049645833 <== osd.0 v2:172.21.15.32:6816/3066156116 1 ==== osd_op_reply(1 600.00000000 [omap-get-keys out=5b] v0'0 uv5 ondisk = 0) ==== 156+0+5 (crc 0 0 0) 0x7fc94800b040 con 0x55e2e79e48b0 2024-08-03T20:54:41.564 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.563+0000 7fc963028f40 1 -- 172.21.15.32:0/1049645833 >> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] conn(0x55e2e79e48b0 msgr2=0x55e2e7aa0260 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:41.565 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.563+0000 7fc963028f40 1 --2- 172.21.15.32:0/1049645833 >> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] conn(0x55e2e79e48b0 0x55e2e7aa0260 crc :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:41.565 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.564+0000 7fc963028f40 1 -- 172.21.15.32:0/1049645833 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fc93c075a20 msgr2=0x7fc93c077f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:41.565 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.564+0000 7fc963028f40 1 --2- 172.21.15.32:0/1049645833 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fc93c075a20 0x7fc93c077f30 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7fc94c00d570 tx=0x7fc94c004c60 comp rx=0 tx=0).stop 2024-08-03T20:54:41.565 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.564+0000 7fc963028f40 1 -- 172.21.15.32:0/1049645833 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc940003680 msgr2=0x55e2e7a07720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:41.565 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.564+0000 7fc963028f40 1 --2- 172.21.15.32:0/1049645833 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc940003680 0x55e2e7a07720 secure :-1 s=READY pgs=996 cs=0 l=1 rev1=1 crypto rx=0x7fc954009da0 tx=0x7fc954001580 comp rx=0 tx=0).stop 2024-08-03T20:54:41.565 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.564+0000 7fc95cc00640 1 -- 172.21.15.32:0/1049645833 reap_dead start 2024-08-03T20:54:41.565 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.564+0000 7fc963028f40 1 -- 172.21.15.32:0/1049645833 shutdown_connections 2024-08-03T20:54:41.565 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.564+0000 7fc963028f40 1 -- 172.21.15.32:0/1049645833 >> 172.21.15.32:0/1049645833 conn(0x55e2e784fac0 msgr2=0x55e2e7931410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:54:41.565 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.564+0000 7fc963028f40 1 -- 172.21.15.32:0/1049645833 shutdown_connections 2024-08-03T20:54:41.565 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.564+0000 7fc963028f40 1 -- 172.21.15.32:0/1049645833 wait complete. 2024-08-03T20:54:41.575 DEBUG:teuthology.orchestra.run.smithi032:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata listomapkeys 1.00000000.inode 2024-08-03T20:54:41.716 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.715+0000 7f136ae65f40 1 Processor -- start 2024-08-03T20:54:41.716 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.716+0000 7f136ae65f40 1 -- start start 2024-08-03T20:54:41.719 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.717+0000 7f136ae65f40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x556c713e0360 0x556c713fa1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:41.719 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.717+0000 7f136ae65f40 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x556c713f7330 0x556c71368a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:41.719 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.717+0000 7f136ae65f40 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x556c713fe3f0 0x556c7148fb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:41.719 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.717+0000 7f136ae65f40 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x556c712d5bf0 con 0x556c713f7330 2024-08-03T20:54:41.719 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.717+0000 7f136ae65f40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x556c712e26b0 con 0x556c713e0360 2024-08-03T20:54:41.719 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.717+0000 7f136ae65f40 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x556c71277e30 con 0x556c713fe3f0 2024-08-03T20:54:41.719 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.717+0000 7f1363600640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x556c713f7330 0x556c71368a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:41.719 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.717+0000 7f1363600640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x556c713f7330 0x556c71368a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:36888/0 (socket says 172.21.15.32:36888) 2024-08-03T20:54:41.719 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.717+0000 7f1363600640 1 -- 172.21.15.32:0/797062661 learned_addr learned my addr 172.21.15.32:0/797062661 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:54:41.719 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.717+0000 7f1364a00640 1 --2- 172.21.15.32:0/797062661 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x556c713fe3f0 0x556c7148fb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:41.720 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.717+0000 7f1363600640 1 -- 172.21.15.32:0/797062661 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x556c713fe3f0 msgr2=0x556c7148fb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:54:41.720 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.717+0000 7f1364000640 1 --2- 172.21.15.32:0/797062661 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x556c713e0360 0x556c713fa1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:41.720 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.718+0000 7f1363600640 1 --2- 172.21.15.32:0/797062661 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x556c713fe3f0 0x556c7148fb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:41.720 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.718+0000 7f1363600640 1 -- 172.21.15.32:0/797062661 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x556c713e0360 msgr2=0x556c713fa1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:54:41.720 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.718+0000 7f1363600640 1 --2- 172.21.15.32:0/797062661 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x556c713e0360 0x556c713fa1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:41.720 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.718+0000 7f1363600640 1 -- 172.21.15.32:0/797062661 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556c7136a9d0 con 0x556c713f7330 2024-08-03T20:54:41.720 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.718+0000 7f1364a00640 1 --2- 172.21.15.32:0/797062661 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x556c713fe3f0 0x556c7148fb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-03T20:54:41.720 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.718+0000 7f1364000640 1 --2- 172.21.15.32:0/797062661 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x556c713e0360 0x556c713fa1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-03T20:54:41.720 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.718+0000 7f1363600640 1 --2- 172.21.15.32:0/797062661 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x556c713f7330 0x556c71368a10 secure :-1 s=READY pgs=999 cs=0 l=1 rev1=1 crypto rx=0x7f1350001e30 tx=0x7f135000b730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ae9cdefc3f762924 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:54:41.720 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.718+0000 7f1362200640 1 -- 172.21.15.32:0/797062661 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1350012070 con 0x556c713f7330 2024-08-03T20:54:41.720 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.718+0000 7f1362200640 1 -- 172.21.15.32:0/797062661 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f1350006050 con 0x556c713f7330 2024-08-03T20:54:41.720 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.718+0000 7f1362200640 1 -- 172.21.15.32:0/797062661 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f135000ebc0 con 0x556c713f7330 2024-08-03T20:54:41.721 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.719+0000 7f136ae65f40 1 -- 172.21.15.32:0/797062661 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x556c713f7330 msgr2=0x556c71368a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:41.721 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.719+0000 7f136ae65f40 1 --2- 172.21.15.32:0/797062661 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x556c713f7330 0x556c71368a10 secure :-1 s=READY pgs=999 cs=0 l=1 rev1=1 crypto rx=0x7f1350001e30 tx=0x7f135000b730 comp rx=0 tx=0).stop 2024-08-03T20:54:41.722 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.719+0000 7f136ae65f40 1 -- 172.21.15.32:0/797062661 shutdown_connections 2024-08-03T20:54:41.722 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.719+0000 7f136ae65f40 1 --2- 172.21.15.32:0/797062661 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x556c713fe3f0 0x556c7148fb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:41.722 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.719+0000 7f136ae65f40 1 --2- 172.21.15.32:0/797062661 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x556c713f7330 0x556c71368a10 unknown :-1 s=CLOSED pgs=999 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:41.722 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.719+0000 7f136ae65f40 1 --2- 172.21.15.32:0/797062661 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x556c713e0360 0x556c713fa1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:41.722 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.719+0000 7f136ae65f40 1 -- 172.21.15.32:0/797062661 >> 172.21.15.32:0/797062661 conn(0x556c71249ac0 msgr2=0x556c713f78d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:54:41.722 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.719+0000 7f136ae65f40 1 -- 172.21.15.32:0/797062661 shutdown_connections 2024-08-03T20:54:41.722 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.719+0000 7f136ae65f40 1 -- 172.21.15.32:0/797062661 wait complete. 2024-08-03T20:54:41.722 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.720+0000 7f136ae65f40 1 Processor -- start 2024-08-03T20:54:41.722 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.720+0000 7f136ae65f40 1 -- start start 2024-08-03T20:54:41.724 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.721+0000 7f136ae65f40 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x556c713e0360 0x556c71496100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:41.724 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.722+0000 7f136ae65f40 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x556c713f7330 0x556c71372770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:41.724 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.722+0000 7f136ae65f40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x556c713fe3f0 0x556c713280d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:41.724 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.722+0000 7f136ae65f40 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x556c71373050 con 0x556c713f7330 2024-08-03T20:54:41.724 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.722+0000 7f136ae65f40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x556c7136acb0 con 0x556c713fe3f0 2024-08-03T20:54:41.724 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.722+0000 7f136ae65f40 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x556c71372ce0 con 0x556c713e0360 2024-08-03T20:54:41.724 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.722+0000 7f1363600640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x556c713f7330 0x556c71372770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:41.725 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.722+0000 7f1363600640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x556c713f7330 0x556c71372770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:36898/0 (socket says 172.21.15.32:36898) 2024-08-03T20:54:41.725 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.722+0000 7f1363600640 1 -- 172.21.15.32:0/1042116309 learned_addr learned my addr 172.21.15.32:0/1042116309 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:54:41.725 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.722+0000 7f1364000640 1 --2- 172.21.15.32:0/1042116309 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x556c713e0360 0x556c71496100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:41.725 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.722+0000 7f1364a00640 1 --2- 172.21.15.32:0/1042116309 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x556c713fe3f0 0x556c713280d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:41.725 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.722+0000 7f1363600640 1 -- 172.21.15.32:0/1042116309 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x556c713e0360 msgr2=0x556c71496100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:41.725 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.722+0000 7f1363600640 1 --2- 172.21.15.32:0/1042116309 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x556c713e0360 0x556c71496100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:41.725 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.722+0000 7f1363600640 1 -- 172.21.15.32:0/1042116309 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x556c713fe3f0 msgr2=0x556c713280d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:41.725 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.722+0000 7f1363600640 1 --2- 172.21.15.32:0/1042116309 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x556c713fe3f0 0x556c713280d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:41.725 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.722+0000 7f1363600640 1 -- 172.21.15.32:0/1042116309 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556c713fcaa0 con 0x556c713f7330 2024-08-03T20:54:41.725 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.723+0000 7f1363600640 1 --2- 172.21.15.32:0/1042116309 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x556c713f7330 0x556c71372770 secure :-1 s=READY pgs=1000 cs=0 l=1 rev1=1 crypto rx=0x7f1350011000 tx=0x7f135000fbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:54:41.725 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.723+0000 7f1360e00640 1 -- 172.21.15.32:0/1042116309 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1350012040 con 0x556c713f7330 2024-08-03T20:54:41.725 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.723+0000 7f1360e00640 1 -- 172.21.15.32:0/1042116309 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f1350006070 con 0x556c713f7330 2024-08-03T20:54:41.725 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.723+0000 7f136ae65f40 1 -- 172.21.15.32:0/1042116309 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x556c713fb2b0 con 0x556c713f7330 2024-08-03T20:54:41.725 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.723+0000 7f1360e00640 1 -- 172.21.15.32:0/1042116309 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1350017a60 con 0x556c713f7330 2024-08-03T20:54:41.726 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.723+0000 7f136ae65f40 1 -- 172.21.15.32:0/1042116309 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556c713fb870 con 0x556c713f7330 2024-08-03T20:54:41.726 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.725+0000 7f1360e00640 1 -- 172.21.15.32:0/1042116309 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f1350015070 con 0x556c713f7330 2024-08-03T20:54:41.726 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.726+0000 7f1360e00640 1 --2- 172.21.15.32:0/1042116309 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f1344075a20 0x7f1344077f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:41.727 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.726+0000 7f1360e00640 1 -- 172.21.15.32:0/1042116309 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 5795+0+0 (secure 0 0 0) 0x7f1350098790 con 0x556c713f7330 2024-08-03T20:54:41.727 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.727+0000 7f1364000640 1 --2- 172.21.15.32:0/1042116309 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f1344075a20 0x7f1344077f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:41.727 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.727+0000 7f136ae65f40 1 --2- 172.21.15.32:0/1042116309 >> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] conn(0x556c713712f0 0x556c714a2880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:41.727 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.727+0000 7f136ae65f40 1 -- 172.21.15.32:0/1042116309 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:1 12.14 12:287034c4:::1.00000000.inode:head [omap-get-keys in=12b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84) -- 0x556c713fc350 con 0x556c713712f0 2024-08-03T20:54:41.727 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.727+0000 7f1364a00640 1 --2- 172.21.15.32:0/1042116309 >> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] conn(0x556c713712f0 0x556c714a2880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:41.728 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.727+0000 7f1364000640 1 --2- 172.21.15.32:0/1042116309 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f1344075a20 0x7f1344077f30 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x556c71400780 tx=0x7f1354003610 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:54:41.728 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.728+0000 7f1364a00640 1 --2- 172.21.15.32:0/1042116309 >> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] conn(0x556c713712f0 0x556c714a2880 crc :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:54:41.729 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.728+0000 7f1364a00640 1 -- 172.21.15.32:0/1042116309 <== osd.5 v2:172.21.15.118:6810/2724891770 1 ==== osd_op_reply(1 1.00000000.inode [omap-get-keys out=5b] v0'0 uv15 ondisk = 0) ==== 160+0+5 (crc 0 0 0) 0x7f135c00b040 con 0x556c713712f0 2024-08-03T20:54:41.729 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.728+0000 7f136ae65f40 1 -- 172.21.15.32:0/1042116309 >> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] conn(0x556c713712f0 msgr2=0x556c714a2880 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:41.729 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.728+0000 7f136ae65f40 1 --2- 172.21.15.32:0/1042116309 >> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] conn(0x556c713712f0 0x556c714a2880 crc :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:41.729 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.728+0000 7f136ae65f40 1 -- 172.21.15.32:0/1042116309 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f1344075a20 msgr2=0x7f1344077f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:41.730 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.728+0000 7f136ae65f40 1 --2- 172.21.15.32:0/1042116309 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f1344075a20 0x7f1344077f30 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x556c71400780 tx=0x7f1354003610 comp rx=0 tx=0).stop 2024-08-03T20:54:41.730 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.728+0000 7f136ae65f40 1 -- 172.21.15.32:0/1042116309 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x556c713f7330 msgr2=0x556c71372770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:41.730 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.728+0000 7f136ae65f40 1 --2- 172.21.15.32:0/1042116309 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x556c713f7330 0x556c71372770 secure :-1 s=READY pgs=1000 cs=0 l=1 rev1=1 crypto rx=0x7f1350011000 tx=0x7f135000fbe0 comp rx=0 tx=0).stop 2024-08-03T20:54:41.730 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.728+0000 7f1364a00640 1 -- 172.21.15.32:0/1042116309 reap_dead start 2024-08-03T20:54:41.730 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.729+0000 7f136ae65f40 1 -- 172.21.15.32:0/1042116309 shutdown_connections 2024-08-03T20:54:41.730 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.729+0000 7f136ae65f40 1 -- 172.21.15.32:0/1042116309 >> 172.21.15.32:0/1042116309 conn(0x556c71249ac0 msgr2=0x556c713df1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:54:41.730 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.729+0000 7f136ae65f40 1 -- 172.21.15.32:0/1042116309 shutdown_connections 2024-08-03T20:54:41.730 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.729+0000 7f136ae65f40 1 -- 172.21.15.32:0/1042116309 wait complete. 2024-08-03T20:54:41.739 DEBUG:teuthology.orchestra.run.smithi032:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata listomapkeys 200.00000000 2024-08-03T20:54:41.882 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.881+0000 7f32b8fedf40 1 Processor -- start 2024-08-03T20:54:41.882 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.881+0000 7f32b8fedf40 1 -- start start 2024-08-03T20:54:41.885 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.883+0000 7f32b8fedf40 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55c4635c8360 0x55c46355a830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:41.885 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.883+0000 7f32b8fedf40 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55c4635e5cb0 0x55c463558820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:41.885 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.883+0000 7f32b8fedf40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55c4635e63f0 0x55c463558d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:41.885 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.883+0000 7f32b8fedf40 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55c4634bdbf0 con 0x55c4635e5cb0 2024-08-03T20:54:41.885 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.883+0000 7f32b8fedf40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x55c4634ca6b0 con 0x55c4635e63f0 2024-08-03T20:54:41.885 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.883+0000 7f32b8fedf40 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x55c46345fe30 con 0x55c4635c8360 2024-08-03T20:54:41.885 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.883+0000 7f32b1800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55c4635e5cb0 0x55c463558820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:41.885 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.883+0000 7f32b1800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55c4635e5cb0 0x55c463558820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:36912/0 (socket says 172.21.15.32:36912) 2024-08-03T20:54:41.885 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.883+0000 7f32b1800640 1 -- 172.21.15.32:0/411896101 learned_addr learned my addr 172.21.15.32:0/411896101 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:54:41.885 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.883+0000 7f32b2200640 1 --2- 172.21.15.32:0/411896101 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55c4635c8360 0x55c46355a830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:41.885 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.883+0000 7f32b2c00640 1 --2- 172.21.15.32:0/411896101 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55c4635e63f0 0x55c463558d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:41.885 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.884+0000 7f32b1800640 1 -- 172.21.15.32:0/411896101 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55c4635c8360 msgr2=0x55c46355a830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:54:41.886 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.884+0000 7f32b1800640 1 --2- 172.21.15.32:0/411896101 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55c4635c8360 0x55c46355a830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:41.886 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.884+0000 7f32b1800640 1 -- 172.21.15.32:0/411896101 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55c4635e63f0 msgr2=0x55c463558d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:54:41.886 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.884+0000 7f32b1800640 1 --2- 172.21.15.32:0/411896101 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55c4635e63f0 0x55c463558d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:41.886 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.884+0000 7f32b1800640 1 -- 172.21.15.32:0/411896101 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c4635e24f0 con 0x55c4635e5cb0 2024-08-03T20:54:41.886 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.884+0000 7f32b2c00640 1 --2- 172.21.15.32:0/411896101 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55c4635e63f0 0x55c463558d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-03T20:54:41.886 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.884+0000 7f32b1800640 1 --2- 172.21.15.32:0/411896101 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55c4635e5cb0 0x55c463558820 secure :-1 s=READY pgs=1002 cs=0 l=1 rev1=1 crypto rx=0x7f32a0001d70 tx=0x7f32a000b5b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e75c278708e30d1 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:54:41.886 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.884+0000 7f32abe00640 1 -- 172.21.15.32:0/411896101 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32a000bd60 con 0x55c4635e5cb0 2024-08-03T20:54:41.886 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.884+0000 7f32abe00640 1 -- 172.21.15.32:0/411896101 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f32a000f400 con 0x55c4635e5cb0 2024-08-03T20:54:41.886 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.884+0000 7f32abe00640 1 -- 172.21.15.32:0/411896101 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32a000f850 con 0x55c4635e5cb0 2024-08-03T20:54:41.888 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.885+0000 7f32b8fedf40 1 -- 172.21.15.32:0/411896101 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55c4635e5cb0 msgr2=0x55c463558820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:41.888 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.885+0000 7f32b8fedf40 1 --2- 172.21.15.32:0/411896101 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55c4635e5cb0 0x55c463558820 secure :-1 s=READY pgs=1002 cs=0 l=1 rev1=1 crypto rx=0x7f32a0001d70 tx=0x7f32a000b5b0 comp rx=0 tx=0).stop 2024-08-03T20:54:41.888 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.885+0000 7f32b8fedf40 1 -- 172.21.15.32:0/411896101 shutdown_connections 2024-08-03T20:54:41.888 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.885+0000 7f32b8fedf40 1 --2- 172.21.15.32:0/411896101 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55c4635e63f0 0x55c463558d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:41.888 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.885+0000 7f32b8fedf40 1 --2- 172.21.15.32:0/411896101 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55c4635e5cb0 0x55c463558820 unknown :-1 s=CLOSED pgs=1002 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:41.889 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.885+0000 7f32b8fedf40 1 --2- 172.21.15.32:0/411896101 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55c4635c8360 0x55c46355a830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:41.889 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.885+0000 7f32b8fedf40 1 -- 172.21.15.32:0/411896101 >> 172.21.15.32:0/411896101 conn(0x55c463431ac0 msgr2=0x55c4635e3d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:54:41.889 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.885+0000 7f32b8fedf40 1 -- 172.21.15.32:0/411896101 shutdown_connections 2024-08-03T20:54:41.889 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.885+0000 7f32b8fedf40 1 -- 172.21.15.32:0/411896101 wait complete. 2024-08-03T20:54:41.889 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.886+0000 7f32b8fedf40 1 Processor -- start 2024-08-03T20:54:41.889 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.886+0000 7f32b8fedf40 1 -- start start 2024-08-03T20:54:41.889 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.887+0000 7f32b8fedf40 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55c4635c8360 0x55c4635c15b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:41.889 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.887+0000 7f32b8fedf40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55c4635e5cb0 0x55c463670810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:41.889 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.887+0000 7f32b8fedf40 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55c4635e63f0 0x55c4635e3d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:41.889 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.887+0000 7f32b8fedf40 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55c463559ed0 con 0x55c4635c8360 2024-08-03T20:54:41.890 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.887+0000 7f32b8fedf40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x55c4635580b0 con 0x55c4635e5cb0 2024-08-03T20:54:41.890 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.888+0000 7f32b8fedf40 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x55c463558390 con 0x55c4635e63f0 2024-08-03T20:54:41.890 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.888+0000 7f32b2200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55c4635c8360 0x55c4635c15b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:41.890 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.888+0000 7f32b2200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55c4635c8360 0x55c4635c15b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:36928/0 (socket says 172.21.15.32:36928) 2024-08-03T20:54:41.890 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.888+0000 7f32b2200640 1 -- 172.21.15.32:0/2364901702 learned_addr learned my addr 172.21.15.32:0/2364901702 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:54:41.891 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.888+0000 7f32b2c00640 1 --2- 172.21.15.32:0/2364901702 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55c4635e63f0 0x55c4635e3d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:41.891 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.888+0000 7f32b1800640 1 --2- 172.21.15.32:0/2364901702 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55c4635e5cb0 0x55c463670810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:41.891 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.888+0000 7f32b2200640 1 -- 172.21.15.32:0/2364901702 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55c4635e63f0 msgr2=0x55c4635e3d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:41.891 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.888+0000 7f32b2200640 1 --2- 172.21.15.32:0/2364901702 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55c4635e63f0 0x55c4635e3d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:41.891 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.888+0000 7f32b2200640 1 -- 172.21.15.32:0/2364901702 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55c4635e5cb0 msgr2=0x55c463670810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:41.891 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.888+0000 7f32b2200640 1 --2- 172.21.15.32:0/2364901702 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55c4635e5cb0 0x55c463670810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:41.892 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.888+0000 7f32b2200640 1 -- 172.21.15.32:0/2364901702 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c463671290 con 0x55c4635c8360 2024-08-03T20:54:41.892 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.888+0000 7f32b1800640 1 --2- 172.21.15.32:0/2364901702 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55c4635e5cb0 0x55c463670810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-03T20:54:41.892 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.888+0000 7f32b2c00640 1 --2- 172.21.15.32:0/2364901702 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55c4635e63f0 0x55c4635e3d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-03T20:54:41.892 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.889+0000 7f32b2200640 1 --2- 172.21.15.32:0/2364901702 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55c4635c8360 0x55c4635c15b0 secure :-1 s=READY pgs=1003 cs=0 l=1 rev1=1 crypto rx=0x7f32a4046430 tx=0x7f32a404ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:54:41.892 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.889+0000 7f32aaa00640 1 -- 172.21.15.32:0/2364901702 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32a4066020 con 0x55c4635c8360 2024-08-03T20:54:41.892 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.889+0000 7f32aaa00640 1 -- 172.21.15.32:0/2364901702 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f32a4056470 con 0x55c4635c8360 2024-08-03T20:54:41.892 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.889+0000 7f32b8fedf40 1 -- 172.21.15.32:0/2364901702 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x55c463671570 con 0x55c4635c8360 2024-08-03T20:54:41.892 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.889+0000 7f32aaa00640 1 -- 172.21.15.32:0/2364901702 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32a40568c0 con 0x55c4635c8360 2024-08-03T20:54:41.892 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.889+0000 7f32b8fedf40 1 -- 172.21.15.32:0/2364901702 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c463670d50 con 0x55c4635c8360 2024-08-03T20:54:41.893 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.891+0000 7f32aaa00640 1 -- 172.21.15.32:0/2364901702 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f32a4069020 con 0x55c4635c8360 2024-08-03T20:54:41.893 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.892+0000 7f32aaa00640 1 --2- 172.21.15.32:0/2364901702 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f3294075a20 0x7f3294077f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:41.893 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.892+0000 7f32aaa00640 1 -- 172.21.15.32:0/2364901702 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 5795+0+0 (secure 0 0 0) 0x7f32a40dbcd0 con 0x55c4635c8360 2024-08-03T20:54:41.893 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.892+0000 7f32b8fedf40 1 --2- 172.21.15.32:0/2364901702 >> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] conn(0x55c463510cc0 0x55c4635110d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:41.894 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.893+0000 7f32b8fedf40 1 -- 172.21.15.32:0/2364901702 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:1 12.14 12:292cf221:::200.00000000:head [omap-get-keys in=12b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84) -- 0x55c463511610 con 0x55c463510cc0 2024-08-03T20:54:41.894 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.893+0000 7f32b1800640 1 --2- 172.21.15.32:0/2364901702 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f3294075a20 0x7f3294077f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:41.894 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.893+0000 7f32b2c00640 1 --2- 172.21.15.32:0/2364901702 >> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] conn(0x55c463510cc0 0x55c4635110d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:41.895 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.893+0000 7f32b1800640 1 --2- 172.21.15.32:0/2364901702 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f3294075a20 0x7f3294077f30 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f32a0001d70 tx=0x7f32a0011040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:54:41.895 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.894+0000 7f32b2c00640 1 --2- 172.21.15.32:0/2364901702 >> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] conn(0x55c463510cc0 0x55c4635110d0 crc :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:54:41.895 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.894+0000 7f32b2c00640 1 -- 172.21.15.32:0/2364901702 <== osd.5 v2:172.21.15.118:6810/2724891770 1 ==== osd_op_reply(1 200.00000000 [omap-get-keys out=5b] v0'0 uv19 ondisk = 0) ==== 156+0+5 (crc 0 0 0) 0x7f32ac005500 con 0x55c463510cc0 2024-08-03T20:54:41.895 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.894+0000 7f32b8fedf40 1 -- 172.21.15.32:0/2364901702 >> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] conn(0x55c463510cc0 msgr2=0x55c4635110d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:41.895 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.894+0000 7f32b8fedf40 1 --2- 172.21.15.32:0/2364901702 >> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] conn(0x55c463510cc0 0x55c4635110d0 crc :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:41.895 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.894+0000 7f32b8fedf40 1 -- 172.21.15.32:0/2364901702 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f3294075a20 msgr2=0x7f3294077f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:41.895 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.894+0000 7f32b8fedf40 1 --2- 172.21.15.32:0/2364901702 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f3294075a20 0x7f3294077f30 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f32a0001d70 tx=0x7f32a0011040 comp rx=0 tx=0).stop 2024-08-03T20:54:41.896 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.894+0000 7f32b8fedf40 1 -- 172.21.15.32:0/2364901702 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55c4635c8360 msgr2=0x55c4635c15b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:41.896 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.894+0000 7f32b8fedf40 1 --2- 172.21.15.32:0/2364901702 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55c4635c8360 0x55c4635c15b0 secure :-1 s=READY pgs=1003 cs=0 l=1 rev1=1 crypto rx=0x7f32a4046430 tx=0x7f32a404ecb0 comp rx=0 tx=0).stop 2024-08-03T20:54:41.896 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.894+0000 7f32b2c00640 1 -- 172.21.15.32:0/2364901702 reap_dead start 2024-08-03T20:54:41.896 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.894+0000 7f32b8fedf40 1 -- 172.21.15.32:0/2364901702 shutdown_connections 2024-08-03T20:54:41.896 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.894+0000 7f32b8fedf40 1 -- 172.21.15.32:0/2364901702 >> 172.21.15.32:0/2364901702 conn(0x55c463431ac0 msgr2=0x55c4635df8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:54:41.896 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.894+0000 7f32b8fedf40 1 -- 172.21.15.32:0/2364901702 shutdown_connections 2024-08-03T20:54:41.896 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:41.894+0000 7f32b8fedf40 1 -- 172.21.15.32:0/2364901702 wait complete. 2024-08-03T20:54:41.906 DEBUG:teuthology.orchestra.run.smithi032:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata listomapkeys 200.00000001 2024-08-03T20:54:42.048 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.047+0000 7fb11ab72f40 1 Processor -- start 2024-08-03T20:54:42.048 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.047+0000 7fb11ab72f40 1 -- start start 2024-08-03T20:54:42.049 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.048+0000 7fb11ab72f40 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55c2a4800360 0x55c2a481b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:42.049 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.048+0000 7fb11ab72f40 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55c2a481dcb0 0x55c2a4819af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:42.049 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.048+0000 7fb11ab72f40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55c2a481e3f0 0x55c2a4790ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:42.049 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.048+0000 7fb11ab72f40 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55c2a4697e30 con 0x55c2a4800360 2024-08-03T20:54:42.050 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.048+0000 7fb11ab72f40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x55c2a46f5bf0 con 0x55c2a481e3f0 2024-08-03T20:54:42.050 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.048+0000 7fb11ab72f40 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x55c2a4801440 con 0x55c2a481dcb0 2024-08-03T20:54:42.050 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.049+0000 7fb113c00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55c2a4800360 0x55c2a481b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:42.050 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.049+0000 7fb113c00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55c2a4800360 0x55c2a481b0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:36944/0 (socket says 172.21.15.32:36944) 2024-08-03T20:54:42.050 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.049+0000 7fb113c00640 1 -- 172.21.15.32:0/2775308074 learned_addr learned my addr 172.21.15.32:0/2775308074 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:54:42.050 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.049+0000 7fb113200640 1 --2- 172.21.15.32:0/2775308074 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55c2a481dcb0 0x55c2a4819af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:42.050 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.049+0000 7fb114600640 1 --2- 172.21.15.32:0/2775308074 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55c2a481e3f0 0x55c2a4790ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:42.051 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.049+0000 7fb113c00640 1 -- 172.21.15.32:0/2775308074 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55c2a481dcb0 msgr2=0x55c2a4819af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:54:42.051 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.049+0000 7fb113c00640 1 --2- 172.21.15.32:0/2775308074 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55c2a481dcb0 0x55c2a4819af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:42.052 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.049+0000 7fb113c00640 1 -- 172.21.15.32:0/2775308074 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55c2a481e3f0 msgr2=0x55c2a4790ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:54:42.052 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.049+0000 7fb113c00640 1 --2- 172.21.15.32:0/2775308074 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55c2a481e3f0 0x55c2a4790ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:42.052 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.049+0000 7fb113c00640 1 -- 172.21.15.32:0/2775308074 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c2a478a170 con 0x55c2a4800360 2024-08-03T20:54:42.052 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.049+0000 7fb114600640 1 --2- 172.21.15.32:0/2775308074 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55c2a481e3f0 0x55c2a4790ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-03T20:54:42.052 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.049+0000 7fb113200640 1 --2- 172.21.15.32:0/2775308074 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55c2a481dcb0 0x55c2a4819af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-03T20:54:42.052 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.050+0000 7fb113c00640 1 --2- 172.21.15.32:0/2775308074 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55c2a4800360 0x55c2a481b0d0 secure :-1 s=READY pgs=1004 cs=0 l=1 rev1=1 crypto rx=0x7fb104025e80 tx=0x7fb104002fb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5b1561f7c5519db7 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:54:42.052 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.050+0000 7fb111e00640 1 -- 172.21.15.32:0/2775308074 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb10404db70 con 0x55c2a4800360 2024-08-03T20:54:42.052 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.050+0000 7fb111e00640 1 -- 172.21.15.32:0/2775308074 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fb10404dd10 con 0x55c2a4800360 2024-08-03T20:54:42.052 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.050+0000 7fb111e00640 1 -- 172.21.15.32:0/2775308074 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1040566d0 con 0x55c2a4800360 2024-08-03T20:54:42.052 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.051+0000 7fb11ab72f40 1 -- 172.21.15.32:0/2775308074 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55c2a4800360 msgr2=0x55c2a481b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:42.052 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.051+0000 7fb11ab72f40 1 --2- 172.21.15.32:0/2775308074 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55c2a4800360 0x55c2a481b0d0 secure :-1 s=READY pgs=1004 cs=0 l=1 rev1=1 crypto rx=0x7fb104025e80 tx=0x7fb104002fb0 comp rx=0 tx=0).stop 2024-08-03T20:54:42.052 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.051+0000 7fb11ab72f40 1 -- 172.21.15.32:0/2775308074 shutdown_connections 2024-08-03T20:54:42.053 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.051+0000 7fb11ab72f40 1 --2- 172.21.15.32:0/2775308074 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55c2a481e3f0 0x55c2a4790ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:42.053 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.051+0000 7fb11ab72f40 1 --2- 172.21.15.32:0/2775308074 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55c2a481dcb0 0x55c2a4819af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:42.053 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.051+0000 7fb11ab72f40 1 --2- 172.21.15.32:0/2775308074 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55c2a4800360 0x55c2a481b0d0 unknown :-1 s=CLOSED pgs=1004 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:42.053 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.051+0000 7fb11ab72f40 1 -- 172.21.15.32:0/2775308074 >> 172.21.15.32:0/2775308074 conn(0x55c2a4669ac0 msgr2=0x55c2a478ab80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:54:42.053 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.051+0000 7fb11ab72f40 1 -- 172.21.15.32:0/2775308074 shutdown_connections 2024-08-03T20:54:42.053 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.051+0000 7fb11ab72f40 1 -- 172.21.15.32:0/2775308074 wait complete. 2024-08-03T20:54:42.053 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.052+0000 7fb11ab72f40 1 Processor -- start 2024-08-03T20:54:42.053 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.052+0000 7fb11ab72f40 1 -- start start 2024-08-03T20:54:42.055 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.053+0000 7fb11ab72f40 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55c2a4800360 0x55c2a47f98b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:42.055 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.053+0000 7fb11ab72f40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55c2a481dcb0 0x55c2a47fa860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:42.055 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.053+0000 7fb11ab72f40 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55c2a481e3f0 0x55c2a47fada0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:42.056 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.053+0000 7fb11ab72f40 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55c2a481ac40 con 0x55c2a4800360 2024-08-03T20:54:42.056 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.053+0000 7fb11ab72f40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x55c2a47ffa20 con 0x55c2a481dcb0 2024-08-03T20:54:42.056 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.053+0000 7fb11ab72f40 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x55c2a47ff120 con 0x55c2a481e3f0 2024-08-03T20:54:42.056 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.054+0000 7fb113c00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55c2a4800360 0x55c2a47f98b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:42.056 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.054+0000 7fb113c00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55c2a4800360 0x55c2a47f98b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:36956/0 (socket says 172.21.15.32:36956) 2024-08-03T20:54:42.056 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.054+0000 7fb113c00640 1 -- 172.21.15.32:0/908011568 learned_addr learned my addr 172.21.15.32:0/908011568 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:54:42.056 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.054+0000 7fb113200640 1 --2- 172.21.15.32:0/908011568 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55c2a481dcb0 0x55c2a47fa860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:42.056 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.054+0000 7fb114600640 1 --2- 172.21.15.32:0/908011568 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55c2a481e3f0 0x55c2a47fada0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:42.056 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.054+0000 7fb113c00640 1 -- 172.21.15.32:0/908011568 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55c2a481e3f0 msgr2=0x55c2a47fada0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:42.056 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.054+0000 7fb113c00640 1 --2- 172.21.15.32:0/908011568 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55c2a481e3f0 0x55c2a47fada0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:42.056 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.054+0000 7fb113c00640 1 -- 172.21.15.32:0/908011568 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55c2a481dcb0 msgr2=0x55c2a47fa860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:42.056 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.054+0000 7fb113c00640 1 --2- 172.21.15.32:0/908011568 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55c2a481dcb0 0x55c2a47fa860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:42.056 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.054+0000 7fb113c00640 1 -- 172.21.15.32:0/908011568 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c2a47f9df0 con 0x55c2a4800360 2024-08-03T20:54:42.057 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.054+0000 7fb113c00640 1 --2- 172.21.15.32:0/908011568 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55c2a4800360 0x55c2a47f98b0 secure :-1 s=READY pgs=1005 cs=0 l=1 rev1=1 crypto rx=0x7fb104025e80 tx=0x7fb104057480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:54:42.057 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.055+0000 7fb110a00640 1 -- 172.21.15.32:0/908011568 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb10406f030 con 0x55c2a4800360 2024-08-03T20:54:42.057 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.055+0000 7fb11ab72f40 1 -- 172.21.15.32:0/908011568 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x55c2a481a2a0 con 0x55c2a4800360 2024-08-03T20:54:42.057 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.055+0000 7fb110a00640 1 -- 172.21.15.32:0/908011568 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fb104057720 con 0x55c2a4800360 2024-08-03T20:54:42.057 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.055+0000 7fb110a00640 1 -- 172.21.15.32:0/908011568 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb104057930 con 0x55c2a4800360 2024-08-03T20:54:42.057 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.055+0000 7fb11ab72f40 1 -- 172.21.15.32:0/908011568 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c2a481a540 con 0x55c2a4800360 2024-08-03T20:54:42.057 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.057+0000 7fb110a00640 1 -- 172.21.15.32:0/908011568 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7fb104072070 con 0x55c2a4800360 2024-08-03T20:54:42.059 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.058+0000 7fb110a00640 1 --2- 172.21.15.32:0/908011568 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fb0f40759f0 0x7fb0f4077f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:42.059 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.058+0000 7fb110a00640 1 -- 172.21.15.32:0/908011568 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 5795+0+0 (secure 0 0 0) 0x7fb1040dc6d0 con 0x55c2a4800360 2024-08-03T20:54:42.059 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.058+0000 7fb11ab72f40 1 --2- 172.21.15.32:0/908011568 >> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] conn(0x55c2a47fb2e0 0x55c2a48b99e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:42.059 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.058+0000 7fb11ab72f40 1 -- 172.21.15.32:0/908011568 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:1 12.14 12:2e2fa760:::200.00000001:head [omap-get-keys in=12b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84) -- 0x55c2a48b9f20 con 0x55c2a47fb2e0 2024-08-03T20:54:42.059 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.058+0000 7fb113200640 1 --2- 172.21.15.32:0/908011568 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fb0f40759f0 0x7fb0f4077f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:42.059 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.059+0000 7fb114600640 1 --2- 172.21.15.32:0/908011568 >> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] conn(0x55c2a47fb2e0 0x55c2a48b99e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:42.060 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.059+0000 7fb113200640 1 --2- 172.21.15.32:0/908011568 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fb0f40759f0 0x7fb0f4077f00 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7fb100003400 tx=0x7fb100003050 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:54:42.061 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.059+0000 7fb114600640 1 --2- 172.21.15.32:0/908011568 >> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] conn(0x55c2a47fb2e0 0x55c2a48b99e0 crc :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:54:42.061 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.060+0000 7fb114600640 1 -- 172.21.15.32:0/908011568 <== osd.5 v2:172.21.15.118:6810/2724891770 1 ==== osd_op_reply(1 200.00000001 [omap-get-keys out=5b] v0'0 uv16 ondisk = 0) ==== 156+0+5 (crc 0 0 0) 0x7fb10c00b040 con 0x55c2a47fb2e0 2024-08-03T20:54:42.061 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.060+0000 7fb11ab72f40 1 -- 172.21.15.32:0/908011568 >> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] conn(0x55c2a47fb2e0 msgr2=0x55c2a48b99e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:42.061 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.060+0000 7fb11ab72f40 1 --2- 172.21.15.32:0/908011568 >> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] conn(0x55c2a47fb2e0 0x55c2a48b99e0 crc :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:42.061 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.060+0000 7fb11ab72f40 1 -- 172.21.15.32:0/908011568 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fb0f40759f0 msgr2=0x7fb0f4077f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:42.061 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.060+0000 7fb11ab72f40 1 --2- 172.21.15.32:0/908011568 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fb0f40759f0 0x7fb0f4077f00 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7fb100003400 tx=0x7fb100003050 comp rx=0 tx=0).stop 2024-08-03T20:54:42.061 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.060+0000 7fb11ab72f40 1 -- 172.21.15.32:0/908011568 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55c2a4800360 msgr2=0x55c2a47f98b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:42.061 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.060+0000 7fb11ab72f40 1 --2- 172.21.15.32:0/908011568 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55c2a4800360 0x55c2a47f98b0 secure :-1 s=READY pgs=1005 cs=0 l=1 rev1=1 crypto rx=0x7fb104025e80 tx=0x7fb104057480 comp rx=0 tx=0).stop 2024-08-03T20:54:42.061 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.060+0000 7fb114600640 1 -- 172.21.15.32:0/908011568 reap_dead start 2024-08-03T20:54:42.061 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.060+0000 7fb11ab72f40 1 -- 172.21.15.32:0/908011568 shutdown_connections 2024-08-03T20:54:42.061 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.060+0000 7fb11ab72f40 1 -- 172.21.15.32:0/908011568 >> 172.21.15.32:0/908011568 conn(0x55c2a4669ac0 msgr2=0x55c2a4819af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:54:42.062 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.060+0000 7fb11ab72f40 1 -- 172.21.15.32:0/908011568 shutdown_connections 2024-08-03T20:54:42.062 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.060+0000 7fb11ab72f40 1 -- 172.21.15.32:0/908011568 wait complete. 2024-08-03T20:54:42.071 DEBUG:teuthology.orchestra.run.smithi032:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata listomapkeys mds0_openfiles.0 2024-08-03T20:54:42.212 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.212+0000 7f373f40cf40 1 Processor -- start 2024-08-03T20:54:42.212 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.212+0000 7f373f40cf40 1 -- start start 2024-08-03T20:54:42.214 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.213+0000 7f373f40cf40 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x555a46ca5360 0x555a46cbd2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:42.214 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.213+0000 7f373f40cf40 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555a46cbc330 0x555a46c9d630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:42.214 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.213+0000 7f373f40cf40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x555a46cc33f0 0x555a46c2e210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:42.214 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.213+0000 7f373f40cf40 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x555a46b9abf0 con 0x555a46cbc330 2024-08-03T20:54:42.214 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.213+0000 7f373f40cf40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x555a46ba76b0 con 0x555a46cc33f0 2024-08-03T20:54:42.214 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.213+0000 7f373f40cf40 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x555a46b3ce30 con 0x555a46ca5360 2024-08-03T20:54:42.215 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.213+0000 7f3736e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555a46cbc330 0x555a46c9d630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:42.215 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.213+0000 7f3736e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555a46cbc330 0x555a46c9d630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:36972/0 (socket says 172.21.15.32:36972) 2024-08-03T20:54:42.215 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.213+0000 7f3736e00640 1 -- 172.21.15.32:0/2412043084 learned_addr learned my addr 172.21.15.32:0/2412043084 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:54:42.215 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.213+0000 7f3738200640 1 --2- 172.21.15.32:0/2412043084 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x555a46cc33f0 0x555a46c2e210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:42.215 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.213+0000 7f3737800640 1 --2- 172.21.15.32:0/2412043084 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x555a46ca5360 0x555a46cbd2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:42.215 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.214+0000 7f3736e00640 1 -- 172.21.15.32:0/2412043084 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x555a46ca5360 msgr2=0x555a46cbd2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:54:42.215 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.214+0000 7f3736e00640 1 --2- 172.21.15.32:0/2412043084 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x555a46ca5360 0x555a46cbd2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:42.215 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.214+0000 7f3736e00640 1 -- 172.21.15.32:0/2412043084 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x555a46cc33f0 msgr2=0x555a46c2e210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:54:42.215 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.214+0000 7f3736e00640 1 --2- 172.21.15.32:0/2412043084 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x555a46cc33f0 0x555a46c2e210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:42.215 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.214+0000 7f3736e00640 1 -- 172.21.15.32:0/2412043084 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555a46cbf340 con 0x555a46cbc330 2024-08-03T20:54:42.217 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.214+0000 7f3736e00640 1 --2- 172.21.15.32:0/2412043084 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555a46cbc330 0x555a46c9d630 secure :-1 s=READY pgs=1008 cs=0 l=1 rev1=1 crypto rx=0x7f3724001e60 tx=0x7f372400b730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fca436578f4a92b9 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:54:42.217 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.214+0000 7f3735a00640 1 -- 172.21.15.32:0/2412043084 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f372400e830 con 0x555a46cbc330 2024-08-03T20:54:42.217 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.214+0000 7f3735a00640 1 -- 172.21.15.32:0/2412043084 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f3724006050 con 0x555a46cbc330 2024-08-03T20:54:42.217 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.214+0000 7f3735a00640 1 -- 172.21.15.32:0/2412043084 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f372400ed80 con 0x555a46cbc330 2024-08-03T20:54:42.217 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.215+0000 7f373f40cf40 1 -- 172.21.15.32:0/2412043084 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555a46cbc330 msgr2=0x555a46c9d630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:42.217 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.215+0000 7f373f40cf40 1 --2- 172.21.15.32:0/2412043084 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555a46cbc330 0x555a46c9d630 secure :-1 s=READY pgs=1008 cs=0 l=1 rev1=1 crypto rx=0x7f3724001e60 tx=0x7f372400b730 comp rx=0 tx=0).stop 2024-08-03T20:54:42.217 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.215+0000 7f373f40cf40 1 -- 172.21.15.32:0/2412043084 shutdown_connections 2024-08-03T20:54:42.217 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.215+0000 7f373f40cf40 1 --2- 172.21.15.32:0/2412043084 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x555a46cc33f0 0x555a46c2e210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:42.217 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.215+0000 7f373f40cf40 1 --2- 172.21.15.32:0/2412043084 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555a46cbc330 0x555a46c9d630 unknown :-1 s=CLOSED pgs=1008 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:42.217 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.215+0000 7f373f40cf40 1 --2- 172.21.15.32:0/2412043084 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x555a46ca5360 0x555a46cbd2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:42.217 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.215+0000 7f373f40cf40 1 -- 172.21.15.32:0/2412043084 >> 172.21.15.32:0/2412043084 conn(0x555a46b0eac0 msgr2=0x555a46cbc8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:54:42.217 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.215+0000 7f373f40cf40 1 -- 172.21.15.32:0/2412043084 shutdown_connections 2024-08-03T20:54:42.218 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.215+0000 7f373f40cf40 1 -- 172.21.15.32:0/2412043084 wait complete. 2024-08-03T20:54:42.218 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.216+0000 7f373f40cf40 1 Processor -- start 2024-08-03T20:54:42.218 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.216+0000 7f373f40cf40 1 -- start start 2024-08-03T20:54:42.220 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.218+0000 7f373f40cf40 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555a46ca5360 0x555a46c9cac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:42.220 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.218+0000 7f373f40cf40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x555a46cbc330 0x555a46d4b440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:42.220 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.218+0000 7f373f40cf40 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x555a46cc33f0 0x555a46d4b980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:42.220 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.218+0000 7f373f40cf40 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x555a46d4c0c0 con 0x555a46ca5360 2024-08-03T20:54:42.220 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.218+0000 7f373f40cf40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x555a46c37f80 con 0x555a46cbc330 2024-08-03T20:54:42.220 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.218+0000 7f373f40cf40 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x555a46c9f530 con 0x555a46cc33f0 2024-08-03T20:54:42.220 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.218+0000 7f3737800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555a46ca5360 0x555a46c9cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:42.220 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.218+0000 7f3737800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555a46ca5360 0x555a46c9cac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:36976/0 (socket says 172.21.15.32:36976) 2024-08-03T20:54:42.220 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.218+0000 7f3737800640 1 -- 172.21.15.32:0/1217317717 learned_addr learned my addr 172.21.15.32:0/1217317717 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:54:42.220 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.218+0000 7f3738200640 1 --2- 172.21.15.32:0/1217317717 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x555a46cc33f0 0x555a46d4b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:42.220 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.218+0000 7f3737800640 1 -- 172.21.15.32:0/1217317717 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x555a46cc33f0 msgr2=0x555a46d4b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:42.220 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.218+0000 7f3736e00640 1 --2- 172.21.15.32:0/1217317717 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x555a46cbc330 0x555a46d4b440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:42.221 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.218+0000 7f3737800640 1 --2- 172.21.15.32:0/1217317717 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x555a46cc33f0 0x555a46d4b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:42.221 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.218+0000 7f3737800640 1 -- 172.21.15.32:0/1217317717 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x555a46cbc330 msgr2=0x555a46d4b440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:42.221 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.218+0000 7f3737800640 1 --2- 172.21.15.32:0/1217317717 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x555a46cbc330 0x555a46d4b440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:42.221 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.218+0000 7f3737800640 1 -- 172.21.15.32:0/1217317717 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555a46c9d000 con 0x555a46ca5360 2024-08-03T20:54:42.221 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.218+0000 7f3738200640 1 --2- 172.21.15.32:0/1217317717 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x555a46cc33f0 0x555a46d4b980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-03T20:54:42.221 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.218+0000 7f3736e00640 1 --2- 172.21.15.32:0/1217317717 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x555a46cbc330 0x555a46d4b440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-03T20:54:42.221 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.219+0000 7f3737800640 1 --2- 172.21.15.32:0/1217317717 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555a46ca5360 0x555a46c9cac0 secure :-1 s=READY pgs=1009 cs=0 l=1 rev1=1 crypto rx=0x7f3728002cc0 tx=0x7f372800c620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:54:42.221 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.219+0000 7f372fe00640 1 -- 172.21.15.32:0/1217317717 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3728015020 con 0x555a46ca5360 2024-08-03T20:54:42.221 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.219+0000 7f372fe00640 1 -- 172.21.15.32:0/1217317717 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f3728011450 con 0x555a46ca5360 2024-08-03T20:54:42.221 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.219+0000 7f372fe00640 1 -- 172.21.15.32:0/1217317717 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3728011840 con 0x555a46ca5360 2024-08-03T20:54:42.221 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.219+0000 7f373f40cf40 1 -- 172.21.15.32:0/1217317717 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x555a46cc1f90 con 0x555a46ca5360 2024-08-03T20:54:42.221 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.219+0000 7f373f40cf40 1 -- 172.21.15.32:0/1217317717 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555a46ca4520 con 0x555a46ca5360 2024-08-03T20:54:42.222 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.221+0000 7f372fe00640 1 -- 172.21.15.32:0/1217317717 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f3728018020 con 0x555a46ca5360 2024-08-03T20:54:42.222 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.222+0000 7f372fe00640 1 --2- 172.21.15.32:0/1217317717 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f37180759f0 0x7f3718077f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:42.222 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.222+0000 7f372fe00640 1 -- 172.21.15.32:0/1217317717 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 5795+0+0 (secure 0 0 0) 0x7f37280976e0 con 0x555a46ca5360 2024-08-03T20:54:42.223 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.223+0000 7f373f40cf40 1 --2- 172.21.15.32:0/1217317717 >> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] conn(0x555a46cbb900 0x555a46c9ebb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:42.223 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.223+0000 7f373f40cf40 1 -- 172.21.15.32:0/1217317717 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:1 12.6 12:654134d2:::mds0_openfiles.0:head [omap-get-keys in=12b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84) -- 0x555a46d67970 con 0x555a46cbb900 2024-08-03T20:54:42.223 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.223+0000 7f3736e00640 1 --2- 172.21.15.32:0/1217317717 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f37180759f0 0x7f3718077f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:42.223 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.223+0000 7f3738200640 1 --2- 172.21.15.32:0/1217317717 >> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] conn(0x555a46cbb900 0x555a46c9ebb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:42.225 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.223+0000 7f3738200640 1 --2- 172.21.15.32:0/1217317717 >> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] conn(0x555a46cbb900 0x555a46c9ebb0 crc :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:54:42.225 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.223+0000 7f3736e00640 1 --2- 172.21.15.32:0/1217317717 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f37180759f0 0x7f3718077f00 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x555a46ca0710 tx=0x7f372400e000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:54:42.225 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.224+0000 7f3738200640 1 -- 172.21.15.32:0/1217317717 <== osd.1 v2:172.21.15.32:6824/1551241759 1 ==== osd_op_reply(1 mds0_openfiles.0 [omap-get-keys out=5b] v0'0 uv4 ondisk = 0) ==== 160+0+5 (crc 0 0 0) 0x7f373000b040 con 0x555a46cbb900 2024-08-03T20:54:42.225 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.224+0000 7f373f40cf40 1 -- 172.21.15.32:0/1217317717 >> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] conn(0x555a46cbb900 msgr2=0x555a46c9ebb0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:42.225 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.224+0000 7f373f40cf40 1 --2- 172.21.15.32:0/1217317717 >> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] conn(0x555a46cbb900 0x555a46c9ebb0 crc :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:42.225 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.224+0000 7f373f40cf40 1 -- 172.21.15.32:0/1217317717 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f37180759f0 msgr2=0x7f3718077f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:42.225 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.224+0000 7f373f40cf40 1 --2- 172.21.15.32:0/1217317717 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f37180759f0 0x7f3718077f00 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x555a46ca0710 tx=0x7f372400e000 comp rx=0 tx=0).stop 2024-08-03T20:54:42.225 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.224+0000 7f373f40cf40 1 -- 172.21.15.32:0/1217317717 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555a46ca5360 msgr2=0x555a46c9cac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:42.225 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.224+0000 7f373f40cf40 1 --2- 172.21.15.32:0/1217317717 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555a46ca5360 0x555a46c9cac0 secure :-1 s=READY pgs=1009 cs=0 l=1 rev1=1 crypto rx=0x7f3728002cc0 tx=0x7f372800c620 comp rx=0 tx=0).stop 2024-08-03T20:54:42.225 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.224+0000 7f3738200640 1 -- 172.21.15.32:0/1217317717 reap_dead start 2024-08-03T20:54:42.225 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.224+0000 7f373f40cf40 1 -- 172.21.15.32:0/1217317717 shutdown_connections 2024-08-03T20:54:42.226 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.224+0000 7f373f40cf40 1 -- 172.21.15.32:0/1217317717 >> 172.21.15.32:0/1217317717 conn(0x555a46b0eac0 msgr2=0x555a46cbc8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:54:42.226 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.224+0000 7f373f40cf40 1 -- 172.21.15.32:0/1217317717 shutdown_connections 2024-08-03T20:54:42.226 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.224+0000 7f373f40cf40 1 -- 172.21.15.32:0/1217317717 wait complete. 2024-08-03T20:54:42.239 DEBUG:teuthology.orchestra.run.smithi032:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata listomapkeys 500.00000000 2024-08-03T20:54:42.381 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.380+0000 7fbbdf91bf40 1 Processor -- start 2024-08-03T20:54:42.381 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.380+0000 7fbbdf91bf40 1 -- start start 2024-08-03T20:54:42.383 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.382+0000 7fbbdf91bf40 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x556b723e2360 0x556b72371ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:42.383 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.382+0000 7fbbdf91bf40 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x556b723e3ea0 0x556b72329da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:42.384 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.382+0000 7fbbdf91bf40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x556b723ffcb0 0x556b724943d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:42.384 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.382+0000 7fbbdf91bf40 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x556b72279e30 con 0x556b723e3ea0 2024-08-03T20:54:42.384 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.382+0000 7fbbdf91bf40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x556b722d7bf0 con 0x556b723ffcb0 2024-08-03T20:54:42.384 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.382+0000 7fbbdf91bf40 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x556b723e31b0 con 0x556b723e2360 2024-08-03T20:54:42.384 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.382+0000 7fbbd7600640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x556b723e3ea0 0x556b72329da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:42.384 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.382+0000 7fbbd7600640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x556b723e3ea0 0x556b72329da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:36994/0 (socket says 172.21.15.32:36994) 2024-08-03T20:54:42.384 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.382+0000 7fbbd7600640 1 -- 172.21.15.32:0/1727178768 learned_addr learned my addr 172.21.15.32:0/1727178768 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:54:42.384 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.382+0000 7fbbd8000640 1 --2- 172.21.15.32:0/1727178768 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x556b723e2360 0x556b72371ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:42.384 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.382+0000 7fbbd8a00640 1 --2- 172.21.15.32:0/1727178768 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x556b723ffcb0 0x556b724943d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:42.384 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.382+0000 7fbbd7600640 1 -- 172.21.15.32:0/1727178768 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x556b723e2360 msgr2=0x556b72371ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:54:42.384 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.382+0000 7fbbd7600640 1 --2- 172.21.15.32:0/1727178768 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x556b723e2360 0x556b72371ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:42.384 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.383+0000 7fbbd7600640 1 -- 172.21.15.32:0/1727178768 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x556b723ffcb0 msgr2=0x556b724943d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:54:42.384 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.383+0000 7fbbd7600640 1 --2- 172.21.15.32:0/1727178768 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x556b723ffcb0 0x556b724943d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:42.385 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.383+0000 7fbbd7600640 1 -- 172.21.15.32:0/1727178768 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556b7232a2e0 con 0x556b723e3ea0 2024-08-03T20:54:42.385 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.383+0000 7fbbd7600640 1 --2- 172.21.15.32:0/1727178768 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x556b723e3ea0 0x556b72329da0 secure :-1 s=READY pgs=1011 cs=0 l=1 rev1=1 crypto rx=0x7fbbc4025de0 tx=0x7fbbc4002fb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=17f8a9b32cb842a1 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:54:42.385 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.383+0000 7fbbd6200640 1 -- 172.21.15.32:0/1727178768 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbbc404d840 con 0x556b723e3ea0 2024-08-03T20:54:42.385 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.383+0000 7fbbd6200640 1 -- 172.21.15.32:0/1727178768 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fbbc404dcf0 con 0x556b723e3ea0 2024-08-03T20:54:42.385 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.383+0000 7fbbd6200640 1 -- 172.21.15.32:0/1727178768 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbbc40566d0 con 0x556b723e3ea0 2024-08-03T20:54:42.386 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.384+0000 7fbbdf91bf40 1 -- 172.21.15.32:0/1727178768 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x556b723e3ea0 msgr2=0x556b72329da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:42.386 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.384+0000 7fbbdf91bf40 1 --2- 172.21.15.32:0/1727178768 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x556b723e3ea0 0x556b72329da0 secure :-1 s=READY pgs=1011 cs=0 l=1 rev1=1 crypto rx=0x7fbbc4025de0 tx=0x7fbbc4002fb0 comp rx=0 tx=0).stop 2024-08-03T20:54:42.386 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.384+0000 7fbbdf91bf40 1 -- 172.21.15.32:0/1727178768 shutdown_connections 2024-08-03T20:54:42.386 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.384+0000 7fbbdf91bf40 1 --2- 172.21.15.32:0/1727178768 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x556b723ffcb0 0x556b724943d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:42.386 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.384+0000 7fbbdf91bf40 1 --2- 172.21.15.32:0/1727178768 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x556b723e3ea0 0x556b72329da0 unknown :-1 s=CLOSED pgs=1011 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:42.386 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.384+0000 7fbbdf91bf40 1 --2- 172.21.15.32:0/1727178768 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x556b723e2360 0x556b72371ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:42.386 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.384+0000 7fbbdf91bf40 1 -- 172.21.15.32:0/1727178768 >> 172.21.15.32:0/1727178768 conn(0x556b7224bac0 msgr2=0x556b7248cbd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:54:42.386 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.384+0000 7fbbdf91bf40 1 -- 172.21.15.32:0/1727178768 shutdown_connections 2024-08-03T20:54:42.387 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.384+0000 7fbbdf91bf40 1 -- 172.21.15.32:0/1727178768 wait complete. 2024-08-03T20:54:42.387 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.385+0000 7fbbdf91bf40 1 Processor -- start 2024-08-03T20:54:42.387 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.385+0000 7fbbdf91bf40 1 -- start start 2024-08-03T20:54:42.388 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.387+0000 7fbbdf91bf40 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x556b723e2360 0x556b7248b6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:42.388 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.387+0000 7fbbdf91bf40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x556b723e3ea0 0x556b723fddc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:42.388 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.387+0000 7fbbdf91bf40 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x556b723ffcb0 0x556b723fe300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:42.389 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.387+0000 7fbbdf91bf40 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x556b7248b1b0 con 0x556b723ffcb0 2024-08-03T20:54:42.389 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.387+0000 7fbbdf91bf40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x556b7236ffd0 con 0x556b723e3ea0 2024-08-03T20:54:42.389 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.387+0000 7fbbdf91bf40 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x556b723e07a0 con 0x556b723e2360 2024-08-03T20:54:42.389 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.387+0000 7fbbd8a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x556b723ffcb0 0x556b723fe300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:42.389 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.387+0000 7fbbd8a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x556b723ffcb0 0x556b723fe300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:36996/0 (socket says 172.21.15.32:36996) 2024-08-03T20:54:42.389 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.387+0000 7fbbd8a00640 1 -- 172.21.15.32:0/4090067366 learned_addr learned my addr 172.21.15.32:0/4090067366 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:54:42.389 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.387+0000 7fbbd7600640 1 --2- 172.21.15.32:0/4090067366 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x556b723e3ea0 0x556b723fddc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:42.389 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.387+0000 7fbbd8000640 1 --2- 172.21.15.32:0/4090067366 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x556b723e2360 0x556b7248b6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:42.389 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.387+0000 7fbbd8a00640 1 -- 172.21.15.32:0/4090067366 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x556b723e2360 msgr2=0x556b7248b6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:42.389 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.387+0000 7fbbd8a00640 1 --2- 172.21.15.32:0/4090067366 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x556b723e2360 0x556b7248b6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:42.389 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.387+0000 7fbbd8a00640 1 -- 172.21.15.32:0/4090067366 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x556b723e3ea0 msgr2=0x556b723fddc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:42.389 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.387+0000 7fbbd8a00640 1 --2- 172.21.15.32:0/4090067366 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x556b723e3ea0 0x556b723fddc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:42.389 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.387+0000 7fbbd8a00640 1 -- 172.21.15.32:0/4090067366 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556b72372090 con 0x556b723ffcb0 2024-08-03T20:54:42.390 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.388+0000 7fbbd8a00640 1 --2- 172.21.15.32:0/4090067366 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x556b723ffcb0 0x556b723fe300 secure :-1 s=READY pgs=1012 cs=0 l=1 rev1=1 crypto rx=0x7fbbd000b400 tx=0x7fbbd000b8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:54:42.390 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.388+0000 7fbbd4e00640 1 -- 172.21.15.32:0/4090067366 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbbd0010560 con 0x556b723ffcb0 2024-08-03T20:54:42.390 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.388+0000 7fbbd4e00640 1 -- 172.21.15.32:0/4090067366 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fbbd0010810 con 0x556b723ffcb0 2024-08-03T20:54:42.390 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.388+0000 7fbbdf91bf40 1 -- 172.21.15.32:0/4090067366 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x556b72372370 con 0x556b723ffcb0 2024-08-03T20:54:42.390 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.388+0000 7fbbd4e00640 1 -- 172.21.15.32:0/4090067366 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbbd0010c60 con 0x556b723ffcb0 2024-08-03T20:54:42.390 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.388+0000 7fbbdf91bf40 1 -- 172.21.15.32:0/4090067366 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556b7236fd10 con 0x556b723ffcb0 2024-08-03T20:54:42.390 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.390+0000 7fbbd4e00640 1 -- 172.21.15.32:0/4090067366 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7fbbd001c470 con 0x556b723ffcb0 2024-08-03T20:54:42.391 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.391+0000 7fbbd4e00640 1 --2- 172.21.15.32:0/4090067366 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fbbb80759f0 0x7fbbb8077f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:42.391 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.391+0000 7fbbd4e00640 1 -- 172.21.15.32:0/4090067366 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 5795+0+0 (secure 0 0 0) 0x7fbbd00979a0 con 0x556b723ffcb0 2024-08-03T20:54:42.392 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.391+0000 7fbbdf91bf40 1 --2- 172.21.15.32:0/4090067366 >> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] conn(0x556b7236f8e0 0x556b7249ce10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:42.392 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.391+0000 7fbbdf91bf40 1 -- 172.21.15.32:0/4090067366 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:1 12.15 12:aa448500:::500.00000000:head [omap-get-keys in=12b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84) -- 0x556b723e1240 con 0x556b7236f8e0 2024-08-03T20:54:42.392 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.391+0000 7fbbd8000640 1 --2- 172.21.15.32:0/4090067366 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fbbb80759f0 0x7fbbb8077f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:42.392 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.392+0000 7fbbd7600640 1 --2- 172.21.15.32:0/4090067366 >> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] conn(0x556b7236f8e0 0x556b7249ce10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:42.393 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.392+0000 7fbbd8000640 1 --2- 172.21.15.32:0/4090067366 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fbbb80759f0 0x7fbbb8077f00 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7fbbc8004fd0 tx=0x7fbbc800e040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:54:42.393 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.393+0000 7fbbd7600640 1 --2- 172.21.15.32:0/4090067366 >> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] conn(0x556b7236f8e0 0x556b7249ce10 crc :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:54:42.394 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.393+0000 7fbbd7600640 1 -- 172.21.15.32:0/4090067366 <== osd.6 v2:172.21.15.118:6825/801947386 1 ==== osd_op_reply(1 500.00000000 [omap-get-keys out=5b] v0'0 uv3 ondisk = 0) ==== 156+0+5 (crc 0 0 0) 0x7fbbc405f030 con 0x556b7236f8e0 2024-08-03T20:54:42.394 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.393+0000 7fbbdf91bf40 1 -- 172.21.15.32:0/4090067366 >> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] conn(0x556b7236f8e0 msgr2=0x556b7249ce10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:42.394 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.393+0000 7fbbdf91bf40 1 --2- 172.21.15.32:0/4090067366 >> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] conn(0x556b7236f8e0 0x556b7249ce10 crc :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:42.394 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.393+0000 7fbbdf91bf40 1 -- 172.21.15.32:0/4090067366 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fbbb80759f0 msgr2=0x7fbbb8077f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:42.394 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.393+0000 7fbbdf91bf40 1 --2- 172.21.15.32:0/4090067366 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fbbb80759f0 0x7fbbb8077f00 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7fbbc8004fd0 tx=0x7fbbc800e040 comp rx=0 tx=0).stop 2024-08-03T20:54:42.398 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.393+0000 7fbbdf91bf40 1 -- 172.21.15.32:0/4090067366 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x556b723ffcb0 msgr2=0x556b723fe300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:42.398 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.393+0000 7fbbdf91bf40 1 --2- 172.21.15.32:0/4090067366 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x556b723ffcb0 0x556b723fe300 secure :-1 s=READY pgs=1012 cs=0 l=1 rev1=1 crypto rx=0x7fbbd000b400 tx=0x7fbbd000b8c0 comp rx=0 tx=0).stop 2024-08-03T20:54:42.398 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.393+0000 7fbbd8a00640 1 -- 172.21.15.32:0/4090067366 reap_dead start 2024-08-03T20:54:42.398 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.393+0000 7fbbdf91bf40 1 -- 172.21.15.32:0/4090067366 shutdown_connections 2024-08-03T20:54:42.398 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.393+0000 7fbbdf91bf40 1 -- 172.21.15.32:0/4090067366 >> 172.21.15.32:0/4090067366 conn(0x556b7224bac0 msgr2=0x556b7248cbd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:54:42.398 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.393+0000 7fbbdf91bf40 1 -- 172.21.15.32:0/4090067366 shutdown_connections 2024-08-03T20:54:42.398 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.393+0000 7fbbdf91bf40 1 -- 172.21.15.32:0/4090067366 wait complete. 2024-08-03T20:54:42.404 DEBUG:teuthology.orchestra.run.smithi032:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata listomapkeys mds_snaptable 2024-08-03T20:54:42.549 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.548+0000 7f8f9091ef40 1 Processor -- start 2024-08-03T20:54:42.549 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.548+0000 7f8f9091ef40 1 -- start start 2024-08-03T20:54:42.551 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.550+0000 7f8f9091ef40 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x562aeafbbb40 0x562aeaf75f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:42.551 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.550+0000 7f8f9091ef40 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x562aeb02ebe0 0x562aeb0d5060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:42.551 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.550+0000 7f8f9091ef40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x562aeaf74cb0 0x562aeb0d7500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:42.551 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.550+0000 7f8f9091ef40 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x562aeaec4e30 con 0x562aeafbbb40 2024-08-03T20:54:42.552 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.550+0000 7f8f9091ef40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x562aeaf22bf0 con 0x562aeaf74cb0 2024-08-03T20:54:42.552 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.550+0000 7f8f9091ef40 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x562aeb02f510 con 0x562aeb02ebe0 2024-08-03T20:54:42.552 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.550+0000 7f8f89000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x562aeafbbb40 0x562aeaf75f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:42.552 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.550+0000 7f8f89000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x562aeafbbb40 0x562aeaf75f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:37012/0 (socket says 172.21.15.32:37012) 2024-08-03T20:54:42.552 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.550+0000 7f8f89000640 1 -- 172.21.15.32:0/2721641348 learned_addr learned my addr 172.21.15.32:0/2721641348 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:54:42.552 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.550+0000 7f8f89a00640 1 --2- 172.21.15.32:0/2721641348 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x562aeaf74cb0 0x562aeb0d7500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:42.552 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.550+0000 7f8f83e00640 1 --2- 172.21.15.32:0/2721641348 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x562aeb02ebe0 0x562aeb0d5060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:42.552 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.550+0000 7f8f89000640 1 -- 172.21.15.32:0/2721641348 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x562aeb02ebe0 msgr2=0x562aeb0d5060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:54:42.552 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.550+0000 7f8f89000640 1 --2- 172.21.15.32:0/2721641348 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x562aeb02ebe0 0x562aeb0d5060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:42.552 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.550+0000 7f8f89000640 1 -- 172.21.15.32:0/2721641348 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x562aeaf74cb0 msgr2=0x562aeb0d7500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:54:42.552 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.550+0000 7f8f89000640 1 --2- 172.21.15.32:0/2721641348 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x562aeaf74cb0 0x562aeb0d7500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:42.552 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.550+0000 7f8f89000640 1 -- 172.21.15.32:0/2721641348 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562aeb024c80 con 0x562aeafbbb40 2024-08-03T20:54:42.552 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.551+0000 7f8f89000640 1 --2- 172.21.15.32:0/2721641348 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x562aeafbbb40 0x562aeaf75f00 secure :-1 s=READY pgs=1015 cs=0 l=1 rev1=1 crypto rx=0x7f8f7c001df0 tx=0x7f8f7c00b670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1565f05dd977ecc6 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:54:42.552 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.551+0000 7f8f82a00640 1 -- 172.21.15.32:0/2721641348 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f7c012070 con 0x562aeafbbb40 2024-08-03T20:54:42.553 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.551+0000 7f8f82a00640 1 -- 172.21.15.32:0/2721641348 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f8f7c006050 con 0x562aeafbbb40 2024-08-03T20:54:42.553 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.551+0000 7f8f82a00640 1 -- 172.21.15.32:0/2721641348 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f7c00ec40 con 0x562aeafbbb40 2024-08-03T20:54:42.554 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.552+0000 7f8f9091ef40 1 -- 172.21.15.32:0/2721641348 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x562aeafbbb40 msgr2=0x562aeaf75f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:42.554 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.552+0000 7f8f9091ef40 1 --2- 172.21.15.32:0/2721641348 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x562aeafbbb40 0x562aeaf75f00 secure :-1 s=READY pgs=1015 cs=0 l=1 rev1=1 crypto rx=0x7f8f7c001df0 tx=0x7f8f7c00b670 comp rx=0 tx=0).stop 2024-08-03T20:54:42.554 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.552+0000 7f8f9091ef40 1 -- 172.21.15.32:0/2721641348 shutdown_connections 2024-08-03T20:54:42.554 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.552+0000 7f8f9091ef40 1 --2- 172.21.15.32:0/2721641348 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x562aeaf74cb0 0x562aeb0d7500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:42.554 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.552+0000 7f8f9091ef40 1 --2- 172.21.15.32:0/2721641348 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x562aeb02ebe0 0x562aeb0d5060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:42.554 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.552+0000 7f8f9091ef40 1 --2- 172.21.15.32:0/2721641348 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x562aeafbbb40 0x562aeaf75f00 unknown :-1 s=CLOSED pgs=1015 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:42.554 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.552+0000 7f8f9091ef40 1 -- 172.21.15.32:0/2721641348 >> 172.21.15.32:0/2721641348 conn(0x562aeae96ac0 msgr2=0x562aeb02c190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:54:42.554 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.552+0000 7f8f9091ef40 1 -- 172.21.15.32:0/2721641348 shutdown_connections 2024-08-03T20:54:42.554 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.552+0000 7f8f9091ef40 1 -- 172.21.15.32:0/2721641348 wait complete. 2024-08-03T20:54:42.554 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.553+0000 7f8f9091ef40 1 Processor -- start 2024-08-03T20:54:42.554 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.553+0000 7f8f9091ef40 1 -- start start 2024-08-03T20:54:42.557 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.554+0000 7f8f9091ef40 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x562aeaf74cb0 0x562aeafbe070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:42.557 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.554+0000 7f8f9091ef40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x562aeafbbb40 0x562aeb024590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:42.557 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.554+0000 7f8f9091ef40 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x562aeb02ebe0 0x562aeb04cd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:42.557 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.554+0000 7f8f9091ef40 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x562aeafba090 con 0x562aeb02ebe0 2024-08-03T20:54:42.557 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.554+0000 7f8f9091ef40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x562aeb02c940 con 0x562aeafbbb40 2024-08-03T20:54:42.557 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.554+0000 7f8f9091ef40 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x562aeafbe900 con 0x562aeaf74cb0 2024-08-03T20:54:42.557 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.555+0000 7f8f89a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x562aeb02ebe0 0x562aeb04cd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:42.557 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.555+0000 7f8f89a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x562aeb02ebe0 0x562aeb04cd10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:37016/0 (socket says 172.21.15.32:37016) 2024-08-03T20:54:42.558 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.555+0000 7f8f89a00640 1 -- 172.21.15.32:0/3600076971 learned_addr learned my addr 172.21.15.32:0/3600076971 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:54:42.558 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.555+0000 7f8f83e00640 1 --2- 172.21.15.32:0/3600076971 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x562aeafbbb40 0x562aeb024590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:42.558 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.555+0000 7f8f89000640 1 --2- 172.21.15.32:0/3600076971 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x562aeaf74cb0 0x562aeafbe070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:42.558 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.555+0000 7f8f89a00640 1 -- 172.21.15.32:0/3600076971 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x562aeaf74cb0 msgr2=0x562aeafbe070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:42.558 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.555+0000 7f8f89a00640 1 --2- 172.21.15.32:0/3600076971 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x562aeaf74cb0 0x562aeafbe070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:42.558 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.555+0000 7f8f89a00640 1 -- 172.21.15.32:0/3600076971 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x562aeafbbb40 msgr2=0x562aeb024590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:42.558 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.555+0000 7f8f89a00640 1 --2- 172.21.15.32:0/3600076971 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x562aeafbbb40 0x562aeb024590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:42.558 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.555+0000 7f8f89a00640 1 -- 172.21.15.32:0/3600076971 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562aeaf75ea0 con 0x562aeb02ebe0 2024-08-03T20:54:42.558 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.555+0000 7f8f89000640 1 --2- 172.21.15.32:0/3600076971 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x562aeaf74cb0 0x562aeafbe070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-03T20:54:42.558 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.555+0000 7f8f89a00640 1 --2- 172.21.15.32:0/3600076971 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x562aeb02ebe0 0x562aeb04cd10 secure :-1 s=READY pgs=1016 cs=0 l=1 rev1=1 crypto rx=0x7f8f8400f650 tx=0x7f8f8400df90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:54:42.558 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.556+0000 7f8f81600640 1 -- 172.21.15.32:0/3600076971 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f84010970 con 0x562aeb02ebe0 2024-08-03T20:54:42.558 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.556+0000 7f8f81600640 1 -- 172.21.15.32:0/3600076971 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f8f84014070 con 0x562aeb02ebe0 2024-08-03T20:54:42.558 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.556+0000 7f8f9091ef40 1 -- 172.21.15.32:0/3600076971 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x562aeb027cd0 con 0x562aeb02ebe0 2024-08-03T20:54:42.559 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.556+0000 7f8f81600640 1 -- 172.21.15.32:0/3600076971 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f84018410 con 0x562aeb02ebe0 2024-08-03T20:54:42.559 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.556+0000 7f8f9091ef40 1 -- 172.21.15.32:0/3600076971 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562aeb04c510 con 0x562aeb02ebe0 2024-08-03T20:54:42.559 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.558+0000 7f8f81600640 1 -- 172.21.15.32:0/3600076971 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f8f84016030 con 0x562aeb02ebe0 2024-08-03T20:54:42.559 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.559+0000 7f8f81600640 1 --2- 172.21.15.32:0/3600076971 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f8f680759f0 0x7f8f68077f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:42.560 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.559+0000 7f8f81600640 1 -- 172.21.15.32:0/3600076971 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 5795+0+0 (secure 0 0 0) 0x7f8f84097fe0 con 0x562aeb02ebe0 2024-08-03T20:54:42.560 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.559+0000 7f8f9091ef40 1 --2- 172.21.15.32:0/3600076971 >> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] conn(0x562aeb0e6ba0 0x562aeb0e6fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:42.560 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.559+0000 7f8f89000640 1 --2- 172.21.15.32:0/3600076971 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f8f680759f0 0x7f8f68077f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:42.560 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.559+0000 7f8f9091ef40 1 -- 172.21.15.32:0/3600076971 --> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] -- osd_op(unknown.0.0:1 12.d 12:b50e409b:::mds_snaptable:head [omap-get-keys in=12b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84) -- 0x562aeb0e74f0 con 0x562aeb0e6ba0 2024-08-03T20:54:42.560 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.560+0000 7f8f83e00640 1 --2- 172.21.15.32:0/3600076971 >> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] conn(0x562aeb0e6ba0 0x562aeb0e6fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:42.561 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.560+0000 7f8f89000640 1 --2- 172.21.15.32:0/3600076971 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f8f680759f0 0x7f8f68077f00 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f8f7c001df0 tx=0x7f8f7c000ed0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:54:42.561 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.561+0000 7f8f83e00640 1 --2- 172.21.15.32:0/3600076971 >> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] conn(0x562aeb0e6ba0 0x562aeb0e6fb0 crc :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:54:42.562 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.561+0000 7f8f83e00640 1 -- 172.21.15.32:0/3600076971 <== osd.7 v2:172.21.15.118:6818/653000048 1 ==== osd_op_reply(1 mds_snaptable [omap-get-keys out=5b] v0'0 uv1 ondisk = 0) ==== 157+0+5 (crc 0 0 0) 0x7f8f7404dad0 con 0x562aeb0e6ba0 2024-08-03T20:54:42.562 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.561+0000 7f8f9091ef40 1 -- 172.21.15.32:0/3600076971 >> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] conn(0x562aeb0e6ba0 msgr2=0x562aeb0e6fb0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:42.562 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.561+0000 7f8f9091ef40 1 --2- 172.21.15.32:0/3600076971 >> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] conn(0x562aeb0e6ba0 0x562aeb0e6fb0 crc :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:42.562 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.561+0000 7f8f9091ef40 1 -- 172.21.15.32:0/3600076971 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f8f680759f0 msgr2=0x7f8f68077f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:42.562 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.561+0000 7f8f9091ef40 1 --2- 172.21.15.32:0/3600076971 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f8f680759f0 0x7f8f68077f00 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f8f7c001df0 tx=0x7f8f7c000ed0 comp rx=0 tx=0).stop 2024-08-03T20:54:42.563 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.561+0000 7f8f9091ef40 1 -- 172.21.15.32:0/3600076971 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x562aeb02ebe0 msgr2=0x562aeb04cd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:42.563 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.561+0000 7f8f9091ef40 1 --2- 172.21.15.32:0/3600076971 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x562aeb02ebe0 0x562aeb04cd10 secure :-1 s=READY pgs=1016 cs=0 l=1 rev1=1 crypto rx=0x7f8f8400f650 tx=0x7f8f8400df90 comp rx=0 tx=0).stop 2024-08-03T20:54:42.563 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.561+0000 7f8f89a00640 1 -- 172.21.15.32:0/3600076971 reap_dead start 2024-08-03T20:54:42.563 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.561+0000 7f8f9091ef40 1 -- 172.21.15.32:0/3600076971 shutdown_connections 2024-08-03T20:54:42.563 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.561+0000 7f8f9091ef40 1 -- 172.21.15.32:0/3600076971 >> 172.21.15.32:0/3600076971 conn(0x562aeae96ac0 msgr2=0x562aeb02c190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:54:42.563 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.561+0000 7f8f9091ef40 1 -- 172.21.15.32:0/3600076971 shutdown_connections 2024-08-03T20:54:42.563 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.562+0000 7f8f9091ef40 1 -- 172.21.15.32:0/3600076971 wait complete. 2024-08-03T20:54:42.576 DEBUG:teuthology.orchestra.run.smithi032:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata listomapkeys mds0_inotable 2024-08-03T20:54:42.718 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.717+0000 7ff973b26f40 1 Processor -- start 2024-08-03T20:54:42.718 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.718+0000 7ff973b26f40 1 -- start start 2024-08-03T20:54:42.720 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.719+0000 7ff973b26f40 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55e38ca27360 0x55e38ca21ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:42.721 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.719+0000 7ff973b26f40 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55e38ca28ea0 0x55e38ca42810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:42.721 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.719+0000 7ff973b26f40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55e38ca44cb0 0x55e38ca42d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:42.721 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.719+0000 7ff973b26f40 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55e38c8bee30 con 0x55e38ca27360 2024-08-03T20:54:42.721 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.719+0000 7ff973b26f40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x55e38c91cbf0 con 0x55e38ca44cb0 2024-08-03T20:54:42.721 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.719+0000 7ff973b26f40 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x55e38ca281b0 con 0x55e38ca28ea0 2024-08-03T20:54:42.721 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.719+0000 7ff96cc00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55e38ca27360 0x55e38ca21ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:42.721 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.719+0000 7ff96cc00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55e38ca27360 0x55e38ca21ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:37034/0 (socket says 172.21.15.32:37034) 2024-08-03T20:54:42.721 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.719+0000 7ff96cc00640 1 -- 172.21.15.32:0/1941408851 learned_addr learned my addr 172.21.15.32:0/1941408851 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:54:42.721 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.720+0000 7ff967e00640 1 --2- 172.21.15.32:0/1941408851 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55e38ca28ea0 0x55e38ca42810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:42.721 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.720+0000 7ff96d600640 1 --2- 172.21.15.32:0/1941408851 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55e38ca44cb0 0x55e38ca42d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:42.721 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.720+0000 7ff96cc00640 1 -- 172.21.15.32:0/1941408851 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55e38ca28ea0 msgr2=0x55e38ca42810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:54:42.721 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.720+0000 7ff96cc00640 1 --2- 172.21.15.32:0/1941408851 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55e38ca28ea0 0x55e38ca42810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:42.722 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.720+0000 7ff96cc00640 1 -- 172.21.15.32:0/1941408851 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55e38ca44cb0 msgr2=0x55e38ca42d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:54:42.722 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.720+0000 7ff96cc00640 1 --2- 172.21.15.32:0/1941408851 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55e38ca44cb0 0x55e38ca42d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:42.722 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.720+0000 7ff96cc00640 1 -- 172.21.15.32:0/1941408851 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e38ca43c90 con 0x55e38ca27360 2024-08-03T20:54:42.722 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.720+0000 7ff96cc00640 1 --2- 172.21.15.32:0/1941408851 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55e38ca27360 0x55e38ca21ea0 secure :-1 s=READY pgs=1019 cs=0 l=1 rev1=1 crypto rx=0x7ff960004e90 tx=0x7ff960002fb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bdcfc707b0eb7034 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:54:42.722 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.720+0000 7ff966a00640 1 -- 172.21.15.32:0/1941408851 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff96004dcb0 con 0x55e38ca27360 2024-08-03T20:54:42.722 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.720+0000 7ff966a00640 1 -- 172.21.15.32:0/1941408851 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7ff960056470 con 0x55e38ca27360 2024-08-03T20:54:42.722 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.720+0000 7ff966a00640 1 -- 172.21.15.32:0/1941408851 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9600568c0 con 0x55e38ca27360 2024-08-03T20:54:42.723 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.721+0000 7ff973b26f40 1 -- 172.21.15.32:0/1941408851 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55e38ca27360 msgr2=0x55e38ca21ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:42.723 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.721+0000 7ff973b26f40 1 --2- 172.21.15.32:0/1941408851 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55e38ca27360 0x55e38ca21ea0 secure :-1 s=READY pgs=1019 cs=0 l=1 rev1=1 crypto rx=0x7ff960004e90 tx=0x7ff960002fb0 comp rx=0 tx=0).stop 2024-08-03T20:54:42.723 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.721+0000 7ff973b26f40 1 -- 172.21.15.32:0/1941408851 shutdown_connections 2024-08-03T20:54:42.723 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.721+0000 7ff973b26f40 1 --2- 172.21.15.32:0/1941408851 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55e38ca44cb0 0x55e38ca42d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:42.724 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.721+0000 7ff973b26f40 1 --2- 172.21.15.32:0/1941408851 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55e38ca28ea0 0x55e38ca42810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:42.724 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.721+0000 7ff973b26f40 1 --2- 172.21.15.32:0/1941408851 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55e38ca27360 0x55e38ca21ea0 unknown :-1 s=CLOSED pgs=1019 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:42.724 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.721+0000 7ff973b26f40 1 -- 172.21.15.32:0/1941408851 >> 172.21.15.32:0/1941408851 conn(0x55e38c890ac0 msgr2=0x55e38c9b6ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:54:42.724 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.721+0000 7ff973b26f40 1 -- 172.21.15.32:0/1941408851 shutdown_connections 2024-08-03T20:54:42.724 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.721+0000 7ff973b26f40 1 -- 172.21.15.32:0/1941408851 wait complete. 2024-08-03T20:54:42.724 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.722+0000 7ff973b26f40 1 Processor -- start 2024-08-03T20:54:42.724 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.722+0000 7ff973b26f40 1 -- start start 2024-08-03T20:54:42.726 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.724+0000 7ff973b26f40 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55e38ca27360 0x55e38cad5aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:42.726 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.724+0000 7ff973b26f40 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55e38ca28ea0 0x55e38ca41010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:42.726 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.724+0000 7ff973b26f40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55e38ca44cb0 0x55e38cadea90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:42.726 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.724+0000 7ff973b26f40 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55e38c9b7b90 con 0x55e38ca27360 2024-08-03T20:54:42.726 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.724+0000 7ff973b26f40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x55e38c9b7e70 con 0x55e38ca44cb0 2024-08-03T20:54:42.726 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.724+0000 7ff973b26f40 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x55e38ca3f7b0 con 0x55e38ca28ea0 2024-08-03T20:54:42.726 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.724+0000 7ff96cc00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55e38ca27360 0x55e38cad5aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:42.727 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.724+0000 7ff96cc00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55e38ca27360 0x55e38cad5aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:37050/0 (socket says 172.21.15.32:37050) 2024-08-03T20:54:42.727 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.724+0000 7ff96cc00640 1 -- 172.21.15.32:0/1299296834 learned_addr learned my addr 172.21.15.32:0/1299296834 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:54:42.727 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.724+0000 7ff96d600640 1 --2- 172.21.15.32:0/1299296834 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55e38ca44cb0 0x55e38cadea90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:42.727 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.724+0000 7ff967e00640 1 --2- 172.21.15.32:0/1299296834 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55e38ca28ea0 0x55e38ca41010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:42.727 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.725+0000 7ff96cc00640 1 -- 172.21.15.32:0/1299296834 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55e38ca28ea0 msgr2=0x55e38ca41010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:42.727 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.725+0000 7ff96cc00640 1 --2- 172.21.15.32:0/1299296834 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55e38ca28ea0 0x55e38ca41010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:42.727 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.725+0000 7ff96cc00640 1 -- 172.21.15.32:0/1299296834 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55e38ca44cb0 msgr2=0x55e38cadea90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:42.727 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.725+0000 7ff96cc00640 1 --2- 172.21.15.32:0/1299296834 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55e38ca44cb0 0x55e38cadea90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:42.727 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.725+0000 7ff96cc00640 1 -- 172.21.15.32:0/1299296834 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e38ca1bfa0 con 0x55e38ca27360 2024-08-03T20:54:42.727 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.725+0000 7ff96cc00640 1 --2- 172.21.15.32:0/1299296834 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55e38ca27360 0x55e38cad5aa0 secure :-1 s=READY pgs=1020 cs=0 l=1 rev1=1 crypto rx=0x7ff960025dc0 tx=0x7ff960057590 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:54:42.727 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.725+0000 7ff965600640 1 -- 172.21.15.32:0/1299296834 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff960057840 con 0x55e38ca27360 2024-08-03T20:54:42.727 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.725+0000 7ff965600640 1 -- 172.21.15.32:0/1299296834 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7ff960057ad0 con 0x55e38ca27360 2024-08-03T20:54:42.727 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.725+0000 7ff973b26f40 1 -- 172.21.15.32:0/1299296834 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x55e38ca43080 con 0x55e38ca27360 2024-08-03T20:54:42.727 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.725+0000 7ff965600640 1 -- 172.21.15.32:0/1299296834 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff960057d20 con 0x55e38ca27360 2024-08-03T20:54:42.728 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.725+0000 7ff973b26f40 1 -- 172.21.15.32:0/1299296834 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e38ca1c8c0 con 0x55e38ca27360 2024-08-03T20:54:42.728 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.728+0000 7ff965600640 1 -- 172.21.15.32:0/1299296834 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7ff960072020 con 0x55e38ca27360 2024-08-03T20:54:42.729 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.728+0000 7ff965600640 1 --2- 172.21.15.32:0/1299296834 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7ff94c0759f0 0x7ff94c077f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:42.729 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.729+0000 7ff965600640 1 -- 172.21.15.32:0/1299296834 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 5795+0+0 (secure 0 0 0) 0x7ff9600dcba0 con 0x55e38ca27360 2024-08-03T20:54:42.729 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.729+0000 7ff973b26f40 1 --2- 172.21.15.32:0/1299296834 >> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] conn(0x55e38ca1c370 0x55e38cadefd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:42.729 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.729+0000 7ff973b26f40 1 -- 172.21.15.32:0/1299296834 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:1 12.13 12:c91d4a1d:::mds0_inotable:head [omap-get-keys in=12b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84) -- 0x55e38cad5fe0 con 0x55e38ca1c370 2024-08-03T20:54:42.729 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.729+0000 7ff967e00640 1 --2- 172.21.15.32:0/1299296834 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7ff94c0759f0 0x7ff94c077f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:42.730 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.729+0000 7ff96d600640 1 --2- 172.21.15.32:0/1299296834 >> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] conn(0x55e38ca1c370 0x55e38cadefd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:42.730 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.730+0000 7ff96d600640 1 --2- 172.21.15.32:0/1299296834 >> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] conn(0x55e38ca1c370 0x55e38cadefd0 crc :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:54:42.730 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.730+0000 7ff967e00640 1 --2- 172.21.15.32:0/1299296834 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7ff94c0759f0 0x7ff94c077f00 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x55e38ca419f0 tx=0x7ff958004a40 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:54:42.731 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.730+0000 7ff96d600640 1 -- 172.21.15.32:0/1299296834 <== osd.1 v2:172.21.15.32:6824/1551241759 1 ==== osd_op_reply(1 mds0_inotable [omap-get-keys out=5b] v0'0 uv3 ondisk = 0) ==== 157+0+5 (crc 0 0 0) 0x7ff968001740 con 0x55e38ca1c370 2024-08-03T20:54:42.731 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.730+0000 7ff973b26f40 1 -- 172.21.15.32:0/1299296834 >> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] conn(0x55e38ca1c370 msgr2=0x55e38cadefd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:42.731 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.730+0000 7ff973b26f40 1 --2- 172.21.15.32:0/1299296834 >> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] conn(0x55e38ca1c370 0x55e38cadefd0 crc :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:42.732 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.730+0000 7ff973b26f40 1 -- 172.21.15.32:0/1299296834 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7ff94c0759f0 msgr2=0x7ff94c077f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:42.732 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.730+0000 7ff973b26f40 1 --2- 172.21.15.32:0/1299296834 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7ff94c0759f0 0x7ff94c077f00 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x55e38ca419f0 tx=0x7ff958004a40 comp rx=0 tx=0).stop 2024-08-03T20:54:42.732 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.730+0000 7ff973b26f40 1 -- 172.21.15.32:0/1299296834 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55e38ca27360 msgr2=0x55e38cad5aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:42.732 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.730+0000 7ff973b26f40 1 --2- 172.21.15.32:0/1299296834 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55e38ca27360 0x55e38cad5aa0 secure :-1 s=READY pgs=1020 cs=0 l=1 rev1=1 crypto rx=0x7ff960025dc0 tx=0x7ff960057590 comp rx=0 tx=0).stop 2024-08-03T20:54:42.732 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.730+0000 7ff96d600640 1 -- 172.21.15.32:0/1299296834 reap_dead start 2024-08-03T20:54:42.732 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.730+0000 7ff973b26f40 1 -- 172.21.15.32:0/1299296834 shutdown_connections 2024-08-03T20:54:42.732 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.730+0000 7ff973b26f40 1 -- 172.21.15.32:0/1299296834 >> 172.21.15.32:0/1299296834 conn(0x55e38c890ac0 msgr2=0x55e38cad5390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:54:42.732 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.730+0000 7ff973b26f40 1 -- 172.21.15.32:0/1299296834 shutdown_connections 2024-08-03T20:54:42.732 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.731+0000 7ff973b26f40 1 -- 172.21.15.32:0/1299296834 wait complete. 2024-08-03T20:54:42.742 DEBUG:teuthology.orchestra.run.smithi032:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata listomapkeys 100.00000000 2024-08-03T20:54:42.885 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.885+0000 7fed87d1ff40 1 Processor -- start 2024-08-03T20:54:42.886 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.885+0000 7fed87d1ff40 1 -- start start 2024-08-03T20:54:42.887 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.886+0000 7fed87d1ff40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55f96f350360 0x55f96f36bf50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:42.887 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.886+0000 7fed87d1ff40 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55f96f36dcb0 0x55f96f298be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:42.888 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.886+0000 7fed87d1ff40 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55f96f36e3f0 0x55f96f299120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:42.888 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.886+0000 7fed87d1ff40 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55f96f1e7e30 con 0x55f96f36dcb0 2024-08-03T20:54:42.888 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.886+0000 7fed87d1ff40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x55f96f245bf0 con 0x55f96f350360 2024-08-03T20:54:42.888 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.886+0000 7fed87d1ff40 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x55f96f351440 con 0x55f96f36e3f0 2024-08-03T20:54:42.888 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.886+0000 7fed7fa00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55f96f36dcb0 0x55f96f298be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:42.888 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.886+0000 7fed7fa00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55f96f36dcb0 0x55f96f298be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:37058/0 (socket says 172.21.15.32:37058) 2024-08-03T20:54:42.888 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.886+0000 7fed7fa00640 1 -- 172.21.15.32:0/4068110389 learned_addr learned my addr 172.21.15.32:0/4068110389 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:54:42.888 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.887+0000 7fed80400640 1 --2- 172.21.15.32:0/4068110389 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55f96f350360 0x55f96f36bf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:42.888 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.887+0000 7fed80e00640 1 --2- 172.21.15.32:0/4068110389 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55f96f36e3f0 0x55f96f299120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:42.888 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.887+0000 7fed7fa00640 1 -- 172.21.15.32:0/4068110389 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55f96f36e3f0 msgr2=0x55f96f299120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:54:42.888 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.887+0000 7fed7fa00640 1 --2- 172.21.15.32:0/4068110389 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55f96f36e3f0 0x55f96f299120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:42.889 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.887+0000 7fed7fa00640 1 -- 172.21.15.32:0/4068110389 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55f96f350360 msgr2=0x55f96f36bf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:54:42.889 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.887+0000 7fed7fa00640 1 --2- 172.21.15.32:0/4068110389 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55f96f350360 0x55f96f36bf50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:42.889 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.887+0000 7fed7fa00640 1 -- 172.21.15.32:0/4068110389 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f96f367480 con 0x55f96f36dcb0 2024-08-03T20:54:42.889 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.887+0000 7fed7fa00640 1 --2- 172.21.15.32:0/4068110389 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55f96f36dcb0 0x55f96f298be0 secure :-1 s=READY pgs=1023 cs=0 l=1 rev1=1 crypto rx=0x7fed6c001e60 tx=0x7fed6c00b730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b2c92f68216cb0a4 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:54:42.890 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.887+0000 7fed7e600640 1 -- 172.21.15.32:0/4068110389 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed6c00e830 con 0x55f96f36dcb0 2024-08-03T20:54:42.890 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.887+0000 7fed7e600640 1 -- 172.21.15.32:0/4068110389 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fed6c006050 con 0x55f96f36dcb0 2024-08-03T20:54:42.890 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.888+0000 7fed7e600640 1 -- 172.21.15.32:0/4068110389 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed6c00ed80 con 0x55f96f36dcb0 2024-08-03T20:54:42.890 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.888+0000 7fed87d1ff40 1 -- 172.21.15.32:0/4068110389 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55f96f36dcb0 msgr2=0x55f96f298be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:42.891 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.888+0000 7fed87d1ff40 1 --2- 172.21.15.32:0/4068110389 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55f96f36dcb0 0x55f96f298be0 secure :-1 s=READY pgs=1023 cs=0 l=1 rev1=1 crypto rx=0x7fed6c001e60 tx=0x7fed6c00b730 comp rx=0 tx=0).stop 2024-08-03T20:54:42.891 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.888+0000 7fed87d1ff40 1 -- 172.21.15.32:0/4068110389 shutdown_connections 2024-08-03T20:54:42.891 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.888+0000 7fed87d1ff40 1 --2- 172.21.15.32:0/4068110389 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55f96f36e3f0 0x55f96f299120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:42.891 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.888+0000 7fed87d1ff40 1 --2- 172.21.15.32:0/4068110389 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55f96f36dcb0 0x55f96f298be0 unknown :-1 s=CLOSED pgs=1023 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:42.891 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.888+0000 7fed87d1ff40 1 --2- 172.21.15.32:0/4068110389 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55f96f350360 0x55f96f36bf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:42.891 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.888+0000 7fed87d1ff40 1 -- 172.21.15.32:0/4068110389 >> 172.21.15.32:0/4068110389 conn(0x55f96f1b9ac0 msgr2=0x55f96f36cb40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:54:42.891 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.888+0000 7fed87d1ff40 1 -- 172.21.15.32:0/4068110389 shutdown_connections 2024-08-03T20:54:42.891 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.888+0000 7fed87d1ff40 1 -- 172.21.15.32:0/4068110389 wait complete. 2024-08-03T20:54:42.891 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.889+0000 7fed87d1ff40 1 Processor -- start 2024-08-03T20:54:42.891 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.889+0000 7fed87d1ff40 1 -- start start 2024-08-03T20:54:42.893 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.891+0000 7fed87d1ff40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55f96f350360 0x55f96f2df4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:42.893 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.891+0000 7fed87d1ff40 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55f96f36dcb0 0x55f96f366440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:42.894 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.891+0000 7fed87d1ff40 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55f96f36e3f0 0x55f96f2e0c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:42.894 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.891+0000 7fed87d1ff40 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55f96f34f0d0 con 0x55f96f36dcb0 2024-08-03T20:54:42.894 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.891+0000 7fed87d1ff40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x55f96f34eb10 con 0x55f96f350360 2024-08-03T20:54:42.894 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.891+0000 7fed87d1ff40 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x55f96f2db010 con 0x55f96f36e3f0 2024-08-03T20:54:42.894 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.891+0000 7fed7fa00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55f96f36dcb0 0x55f96f366440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:42.894 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.891+0000 7fed7fa00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55f96f36dcb0 0x55f96f366440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:37066/0 (socket says 172.21.15.32:37066) 2024-08-03T20:54:42.894 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.891+0000 7fed7fa00640 1 -- 172.21.15.32:0/344594630 learned_addr learned my addr 172.21.15.32:0/344594630 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:54:42.894 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.891+0000 7fed80e00640 1 --2- 172.21.15.32:0/344594630 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55f96f36e3f0 0x55f96f2e0c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:42.894 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.891+0000 7fed80400640 1 --2- 172.21.15.32:0/344594630 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55f96f350360 0x55f96f2df4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:42.894 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.892+0000 7fed7fa00640 1 -- 172.21.15.32:0/344594630 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55f96f36e3f0 msgr2=0x55f96f2e0c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:42.894 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.892+0000 7fed7fa00640 1 --2- 172.21.15.32:0/344594630 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55f96f36e3f0 0x55f96f2e0c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:42.894 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.892+0000 7fed7fa00640 1 -- 172.21.15.32:0/344594630 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55f96f350360 msgr2=0x55f96f2df4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:42.895 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.892+0000 7fed7fa00640 1 --2- 172.21.15.32:0/344594630 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55f96f350360 0x55f96f2df4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:42.895 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.892+0000 7fed7fa00640 1 -- 172.21.15.32:0/344594630 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f96f34bf70 con 0x55f96f36dcb0 2024-08-03T20:54:42.895 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.892+0000 7fed80400640 1 --2- 172.21.15.32:0/344594630 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55f96f350360 0x55f96f2df4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-03T20:54:42.895 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.892+0000 7fed7fa00640 1 --2- 172.21.15.32:0/344594630 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55f96f36dcb0 0x55f96f366440 secure :-1 s=READY pgs=1024 cs=0 l=1 rev1=1 crypto rx=0x7fed6c00bc30 tx=0x7fed6c00e8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:54:42.895 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.892+0000 7fed7d200640 1 -- 172.21.15.32:0/344594630 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed6c024070 con 0x55f96f36dcb0 2024-08-03T20:54:42.895 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.892+0000 7fed7d200640 1 -- 172.21.15.32:0/344594630 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fed6c006050 con 0x55f96f36dcb0 2024-08-03T20:54:42.895 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.892+0000 7fed7d200640 1 -- 172.21.15.32:0/344594630 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed6c01f650 con 0x55f96f36dcb0 2024-08-03T20:54:42.895 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.892+0000 7fed87d1ff40 1 -- 172.21.15.32:0/344594630 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x55f96f2d8350 con 0x55f96f36dcb0 2024-08-03T20:54:42.895 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.892+0000 7fed87d1ff40 1 -- 172.21.15.32:0/344594630 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f96f2d8020 con 0x55f96f36dcb0 2024-08-03T20:54:42.896 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.895+0000 7fed7d200640 1 -- 172.21.15.32:0/344594630 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7fed6c0163f0 con 0x55f96f36dcb0 2024-08-03T20:54:42.896 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.895+0000 7fed7d200640 1 --2- 172.21.15.32:0/344594630 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fed60075a20 0x7fed60077f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:42.896 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.896+0000 7fed7d200640 1 -- 172.21.15.32:0/344594630 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 5795+0+0 (secure 0 0 0) 0x7fed6c0981f0 con 0x55f96f36dcb0 2024-08-03T20:54:42.896 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.896+0000 7fed87d1ff40 1 --2- 172.21.15.32:0/344594630 >> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] conn(0x55f96f3398e0 0x55f96f339cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:42.896 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.896+0000 7fed87d1ff40 1 -- 172.21.15.32:0/344594630 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:1 12.13 12:cd5a64a3:::100.00000000:head [omap-get-keys in=12b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84) -- 0x55f96f34cb00 con 0x55f96f3398e0 2024-08-03T20:54:42.897 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.896+0000 7fed80400640 1 --2- 172.21.15.32:0/344594630 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fed60075a20 0x7fed60077f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:42.897 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.896+0000 7fed80e00640 1 --2- 172.21.15.32:0/344594630 >> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] conn(0x55f96f3398e0 0x55f96f339cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:42.897 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.897+0000 7fed80e00640 1 --2- 172.21.15.32:0/344594630 >> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] conn(0x55f96f3398e0 0x55f96f339cf0 crc :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:54:42.898 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.897+0000 7fed80400640 1 --2- 172.21.15.32:0/344594630 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fed60075a20 0x7fed60077f30 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7fed70055000 tx=0x7fed70003020 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:54:42.898 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.897+0000 7fed80e00640 1 -- 172.21.15.32:0/344594630 <== osd.1 v2:172.21.15.32:6824/1551241759 1 ==== osd_op_reply(1 100.00000000 [omap-get-keys out=155b] v0'0 uv1 ondisk = 0) ==== 156+0+155 (crc 0 0 0) 0x7fed78003040 con 0x55f96f3398e0 2024-08-03T20:54:42.898 INFO:teuthology.orchestra.run.smithi032.stdout:stray0_head 2024-08-03T20:54:42.898 INFO:teuthology.orchestra.run.smithi032.stdout:stray1_head 2024-08-03T20:54:42.898 INFO:teuthology.orchestra.run.smithi032.stdout:stray2_head 2024-08-03T20:54:42.898 INFO:teuthology.orchestra.run.smithi032.stdout:stray3_head 2024-08-03T20:54:42.898 INFO:teuthology.orchestra.run.smithi032.stdout:stray4_head 2024-08-03T20:54:42.899 INFO:teuthology.orchestra.run.smithi032.stdout:stray5_head 2024-08-03T20:54:42.899 INFO:teuthology.orchestra.run.smithi032.stdout:stray6_head 2024-08-03T20:54:42.899 INFO:teuthology.orchestra.run.smithi032.stdout:stray7_head 2024-08-03T20:54:42.899 INFO:teuthology.orchestra.run.smithi032.stdout:stray8_head 2024-08-03T20:54:42.899 INFO:teuthology.orchestra.run.smithi032.stdout:stray9_head 2024-08-03T20:54:42.900 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.897+0000 7fed87d1ff40 1 -- 172.21.15.32:0/344594630 >> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] conn(0x55f96f3398e0 msgr2=0x55f96f339cf0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:42.900 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.897+0000 7fed87d1ff40 1 --2- 172.21.15.32:0/344594630 >> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] conn(0x55f96f3398e0 0x55f96f339cf0 crc :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:42.900 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.897+0000 7fed87d1ff40 1 -- 172.21.15.32:0/344594630 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fed60075a20 msgr2=0x7fed60077f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:42.900 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.897+0000 7fed87d1ff40 1 --2- 172.21.15.32:0/344594630 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fed60075a20 0x7fed60077f30 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7fed70055000 tx=0x7fed70003020 comp rx=0 tx=0).stop 2024-08-03T20:54:42.900 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.897+0000 7fed87d1ff40 1 -- 172.21.15.32:0/344594630 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55f96f36dcb0 msgr2=0x55f96f366440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:42.900 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.897+0000 7fed87d1ff40 1 --2- 172.21.15.32:0/344594630 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55f96f36dcb0 0x55f96f366440 secure :-1 s=READY pgs=1024 cs=0 l=1 rev1=1 crypto rx=0x7fed6c00bc30 tx=0x7fed6c00e8d0 comp rx=0 tx=0).stop 2024-08-03T20:54:42.900 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.897+0000 7fed80e00640 1 -- 172.21.15.32:0/344594630 reap_dead start 2024-08-03T20:54:42.900 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.897+0000 7fed87d1ff40 1 -- 172.21.15.32:0/344594630 shutdown_connections 2024-08-03T20:54:42.900 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.897+0000 7fed87d1ff40 1 -- 172.21.15.32:0/344594630 >> 172.21.15.32:0/344594630 conn(0x55f96f1b9ac0 msgr2=0x55f96f34e020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:54:42.901 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.898+0000 7fed87d1ff40 1 -- 172.21.15.32:0/344594630 shutdown_connections 2024-08-03T20:54:42.901 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:42.898+0000 7fed87d1ff40 1 -- 172.21.15.32:0/344594630 wait complete. 2024-08-03T20:54:42.912 DEBUG:teuthology.orchestra.run.smithi032:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata listomapkeys mds0_sessionmap 2024-08-03T20:54:43.054 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.053+0000 7f565b572f40 1 Processor -- start 2024-08-03T20:54:43.054 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.053+0000 7f565b572f40 1 -- start start 2024-08-03T20:54:43.056 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.055+0000 7f565b572f40 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x5559db11d0e0 0x5559db1ab210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:43.056 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.055+0000 7f565b572f40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5559db11ee20 0x5559db0fe970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:43.056 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.055+0000 7f565b572f40 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5559db1a9840 0x5559db0feeb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:43.056 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.055+0000 7f565b572f40 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x5559db023e30 con 0x5559db1a9840 2024-08-03T20:54:43.057 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.055+0000 7f565b572f40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x5559db081bf0 con 0x5559db11ee20 2024-08-03T20:54:43.057 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.055+0000 7f565b572f40 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x5559db11bc10 con 0x5559db11d0e0 2024-08-03T20:54:43.057 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.055+0000 7f5655000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5559db1a9840 0x5559db0feeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:43.057 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.055+0000 7f5655000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5559db1a9840 0x5559db0feeb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:37090/0 (socket says 172.21.15.32:37090) 2024-08-03T20:54:43.057 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.055+0000 7f5655000640 1 -- 172.21.15.32:0/2773143195 learned_addr learned my addr 172.21.15.32:0/2773143195 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:54:43.057 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.055+0000 7f5655000640 1 -- 172.21.15.32:0/2773143195 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x5559db11d0e0 msgr2=0x5559db1ab210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:54:43.057 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.055+0000 7f5655000640 1 --2- 172.21.15.32:0/2773143195 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x5559db11d0e0 0x5559db1ab210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:43.057 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.055+0000 7f5655000640 1 -- 172.21.15.32:0/2773143195 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5559db11ee20 msgr2=0x5559db0fe970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:54:43.057 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.055+0000 7f5655000640 1 --2- 172.21.15.32:0/2773143195 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5559db11ee20 0x5559db0fe970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:43.057 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.055+0000 7f5655000640 1 -- 172.21.15.32:0/2773143195 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5559db1abbf0 con 0x5559db1a9840 2024-08-03T20:54:43.057 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.056+0000 7f5655000640 1 --2- 172.21.15.32:0/2773143195 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5559db1a9840 0x5559db0feeb0 secure :-1 s=READY pgs=1027 cs=0 l=1 rev1=1 crypto rx=0x7f564c0015b0 tx=0x7f564c001a70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=437f4dfc27b5caad server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:54:43.057 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.056+0000 7f5652800640 1 -- 172.21.15.32:0/2773143195 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f564c017070 con 0x5559db1a9840 2024-08-03T20:54:43.057 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.056+0000 7f5652800640 1 -- 172.21.15.32:0/2773143195 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f564c0027a0 con 0x5559db1a9840 2024-08-03T20:54:43.058 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.056+0000 7f5652800640 1 -- 172.21.15.32:0/2773143195 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f564c002bf0 con 0x5559db1a9840 2024-08-03T20:54:43.058 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.056+0000 7f565b572f40 1 -- 172.21.15.32:0/2773143195 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5559db1a9840 msgr2=0x5559db0feeb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:43.059 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.056+0000 7f565b572f40 1 --2- 172.21.15.32:0/2773143195 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5559db1a9840 0x5559db0feeb0 secure :-1 s=READY pgs=1027 cs=0 l=1 rev1=1 crypto rx=0x7f564c0015b0 tx=0x7f564c001a70 comp rx=0 tx=0).stop 2024-08-03T20:54:43.059 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.056+0000 7f565b572f40 1 -- 172.21.15.32:0/2773143195 shutdown_connections 2024-08-03T20:54:43.059 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.056+0000 7f565b572f40 1 --2- 172.21.15.32:0/2773143195 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5559db11ee20 0x5559db0fe970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:43.059 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.057+0000 7f565b572f40 1 --2- 172.21.15.32:0/2773143195 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5559db1a9840 0x5559db0feeb0 unknown :-1 s=CLOSED pgs=1027 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:43.059 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.057+0000 7f565b572f40 1 --2- 172.21.15.32:0/2773143195 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x5559db11d0e0 0x5559db1ab210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:43.059 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.057+0000 7f565b572f40 1 -- 172.21.15.32:0/2773143195 >> 172.21.15.32:0/2773143195 conn(0x5559daff5ac0 msgr2=0x5559db0fe140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:54:43.059 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.057+0000 7f565b572f40 1 -- 172.21.15.32:0/2773143195 shutdown_connections 2024-08-03T20:54:43.059 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.057+0000 7f565b572f40 1 -- 172.21.15.32:0/2773143195 wait complete. 2024-08-03T20:54:43.059 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.058+0000 7f565b572f40 1 Processor -- start 2024-08-03T20:54:43.059 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.058+0000 7f565b572f40 1 -- start start 2024-08-03T20:54:43.060 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.059+0000 7f565b572f40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5559db11d0e0 0x5559db189d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:43.061 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.059+0000 7f565b572f40 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x5559db11ee20 0x5559db1a65f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:43.061 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.059+0000 7f565b572f40 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5559db1a9840 0x5559db1a6b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:43.061 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.059+0000 7f565b572f40 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x5559db233a00 con 0x5559db1a9840 2024-08-03T20:54:43.061 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.059+0000 7f565b572f40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x5559db233ce0 con 0x5559db11d0e0 2024-08-03T20:54:43.061 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.059+0000 7f565b572f40 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x5559db190920 con 0x5559db11ee20 2024-08-03T20:54:43.061 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.059+0000 7f5655000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5559db1a9840 0x5559db1a6b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:43.061 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.059+0000 7f5655000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5559db1a9840 0x5559db1a6b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:37102/0 (socket says 172.21.15.32:37102) 2024-08-03T20:54:43.061 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.059+0000 7f5655000640 1 -- 172.21.15.32:0/84353834 learned_addr learned my addr 172.21.15.32:0/84353834 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:54:43.061 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.059+0000 7f5654600640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5559db11d0e0 0x5559db189d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:43.062 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.060+0000 7f5653c00640 1 --2- 172.21.15.32:0/84353834 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x5559db11ee20 0x5559db1a65f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:43.062 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.060+0000 7f5655000640 1 -- 172.21.15.32:0/84353834 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x5559db11ee20 msgr2=0x5559db1a65f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:43.062 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.060+0000 7f5655000640 1 --2- 172.21.15.32:0/84353834 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x5559db11ee20 0x5559db1a65f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:43.062 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.060+0000 7f5655000640 1 -- 172.21.15.32:0/84353834 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5559db11d0e0 msgr2=0x5559db189d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:43.062 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.060+0000 7f5655000640 1 --2- 172.21.15.32:0/84353834 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5559db11d0e0 0x5559db189d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:43.062 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.060+0000 7f5655000640 1 -- 172.21.15.32:0/84353834 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5559db190b70 con 0x5559db1a9840 2024-08-03T20:54:43.062 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.060+0000 7f5655000640 1 --2- 172.21.15.32:0/84353834 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5559db1a9840 0x5559db1a6b30 secure :-1 s=READY pgs=1028 cs=0 l=1 rev1=1 crypto rx=0x7f564c001f70 tx=0x7f564c003000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:54:43.063 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.061+0000 7f5651400640 1 -- 172.21.15.32:0/84353834 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f564c023070 con 0x5559db1a9840 2024-08-03T20:54:43.063 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.061+0000 7f565b572f40 1 -- 172.21.15.32:0/84353834 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x5559db1ab340 con 0x5559db1a9840 2024-08-03T20:54:43.063 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.061+0000 7f5651400640 1 -- 172.21.15.32:0/84353834 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f564c018040 con 0x5559db1a9840 2024-08-03T20:54:43.063 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.061+0000 7f5651400640 1 -- 172.21.15.32:0/84353834 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f564c017070 con 0x5559db1a9840 2024-08-03T20:54:43.063 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.061+0000 7f565b572f40 1 -- 172.21.15.32:0/84353834 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5559db18a290 con 0x5559db1a9840 2024-08-03T20:54:43.064 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.063+0000 7f5651400640 1 -- 172.21.15.32:0/84353834 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f564c026070 con 0x5559db1a9840 2024-08-03T20:54:43.064 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.064+0000 7f5651400640 1 --2- 172.21.15.32:0/84353834 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f56340759f0 0x7f5634077f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:43.065 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.064+0000 7f5651400640 1 -- 172.21.15.32:0/84353834 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 5795+0+0 (secure 0 0 0) 0x7f564c098350 con 0x5559db1a9840 2024-08-03T20:54:43.065 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.065+0000 7f5654600640 1 --2- 172.21.15.32:0/84353834 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f56340759f0 0x7f5634077f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:43.065 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.065+0000 7f565b572f40 1 --2- 172.21.15.32:0/84353834 >> [v2:172.21.15.32:6800/9088303,v1:172.21.15.32:6801/9088303] conn(0x5559db1a40b0 0x5559db1a44c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:43.065 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.065+0000 7f565b572f40 1 -- 172.21.15.32:0/84353834 --> [v2:172.21.15.32:6800/9088303,v1:172.21.15.32:6801/9088303] -- osd_op(unknown.0.0:1 12.b 12:d0630e4c:::mds0_sessionmap:head [omap-get-keys in=12b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84) -- 0x5559db1a4a00 con 0x5559db1a40b0 2024-08-03T20:54:43.066 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.065+0000 7f5653c00640 1 --2- 172.21.15.32:0/84353834 >> [v2:172.21.15.32:6800/9088303,v1:172.21.15.32:6801/9088303] conn(0x5559db1a40b0 0x5559db1a44c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:43.067 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.065+0000 7f5654600640 1 --2- 172.21.15.32:0/84353834 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f56340759f0 0x7f5634077f00 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f56440040b0 tx=0x7f5644004b50 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:54:43.067 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.066+0000 7f5653c00640 1 --2- 172.21.15.32:0/84353834 >> [v2:172.21.15.32:6800/9088303,v1:172.21.15.32:6801/9088303] conn(0x5559db1a40b0 0x5559db1a44c0 crc :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:54:43.067 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.066+0000 7f5653c00640 1 -- 172.21.15.32:0/84353834 <== osd.3 v2:172.21.15.32:6800/9088303 1 ==== osd_op_reply(1 mds0_sessionmap [omap-get-keys out=5b] v0'0 uv3 ondisk = 0) ==== 159+0+5 (crc 0 0 0) 0x7f564004dab0 con 0x5559db1a40b0 2024-08-03T20:54:43.067 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.066+0000 7f565b572f40 1 -- 172.21.15.32:0/84353834 >> [v2:172.21.15.32:6800/9088303,v1:172.21.15.32:6801/9088303] conn(0x5559db1a40b0 msgr2=0x5559db1a44c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:43.067 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.066+0000 7f565b572f40 1 --2- 172.21.15.32:0/84353834 >> [v2:172.21.15.32:6800/9088303,v1:172.21.15.32:6801/9088303] conn(0x5559db1a40b0 0x5559db1a44c0 crc :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:43.067 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.066+0000 7f565b572f40 1 -- 172.21.15.32:0/84353834 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f56340759f0 msgr2=0x7f5634077f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:43.068 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.066+0000 7f565b572f40 1 --2- 172.21.15.32:0/84353834 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f56340759f0 0x7f5634077f00 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f56440040b0 tx=0x7f5644004b50 comp rx=0 tx=0).stop 2024-08-03T20:54:43.068 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.066+0000 7f565b572f40 1 -- 172.21.15.32:0/84353834 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5559db1a9840 msgr2=0x5559db1a6b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:43.068 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.066+0000 7f565b572f40 1 --2- 172.21.15.32:0/84353834 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5559db1a9840 0x5559db1a6b30 secure :-1 s=READY pgs=1028 cs=0 l=1 rev1=1 crypto rx=0x7f564c001f70 tx=0x7f564c003000 comp rx=0 tx=0).stop 2024-08-03T20:54:43.068 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.066+0000 7f5655000640 1 -- 172.21.15.32:0/84353834 reap_dead start 2024-08-03T20:54:43.068 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.066+0000 7f565b572f40 1 -- 172.21.15.32:0/84353834 shutdown_connections 2024-08-03T20:54:43.068 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.066+0000 7f565b572f40 1 -- 172.21.15.32:0/84353834 >> 172.21.15.32:0/84353834 conn(0x5559daff5ac0 msgr2=0x5559db1a7c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:54:43.068 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.066+0000 7f565b572f40 1 -- 172.21.15.32:0/84353834 shutdown_connections 2024-08-03T20:54:43.068 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.066+0000 7f565b572f40 1 -- 172.21.15.32:0/84353834 wait complete. 2024-08-03T20:54:43.078 DEBUG:teuthology.orchestra.run.smithi032:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata listomapkeys 400.00000000 2024-08-03T20:54:43.221 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.220+0000 7fcba891bf40 1 Processor -- start 2024-08-03T20:54:43.221 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.220+0000 7fcba891bf40 1 -- start start 2024-08-03T20:54:43.224 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.222+0000 7fcba891bf40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55b58afba870 0x55b58b030460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:43.224 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.222+0000 7fcba891bf40 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55b58afbc370 0x55b58afa2a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:43.224 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.222+0000 7fcba891bf40 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b58b031270 0x55b58afa2f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:43.224 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.222+0000 7fcba891bf40 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55b58aec0e30 con 0x55b58b031270 2024-08-03T20:54:43.224 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.222+0000 7fcba891bf40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x55b58af1ebf0 con 0x55b58afba870 2024-08-03T20:54:43.224 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.222+0000 7fcba891bf40 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x55b58afb9590 con 0x55b58afbc370 2024-08-03T20:54:43.224 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.222+0000 7fcba1a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b58b031270 0x55b58afa2f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:43.224 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.222+0000 7fcba1a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b58b031270 0x55b58afa2f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:37104/0 (socket says 172.21.15.32:37104) 2024-08-03T20:54:43.224 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.222+0000 7fcba1a00640 1 -- 172.21.15.32:0/4202612629 learned_addr learned my addr 172.21.15.32:0/4202612629 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:54:43.224 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.222+0000 7fcb9be00640 1 --2- 172.21.15.32:0/4202612629 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55b58afbc370 0x55b58afa2a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:43.224 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.222+0000 7fcba1000640 1 --2- 172.21.15.32:0/4202612629 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55b58afba870 0x55b58b030460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:43.225 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.222+0000 7fcba1a00640 1 -- 172.21.15.32:0/4202612629 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55b58afbc370 msgr2=0x55b58afa2a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:54:43.225 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.222+0000 7fcba1a00640 1 --2- 172.21.15.32:0/4202612629 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55b58afbc370 0x55b58afa2a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:43.225 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.222+0000 7fcba1a00640 1 -- 172.21.15.32:0/4202612629 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55b58afba870 msgr2=0x55b58b030460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:54:43.225 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.222+0000 7fcba1a00640 1 --2- 172.21.15.32:0/4202612629 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55b58afba870 0x55b58b030460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:43.225 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.222+0000 7fcba1a00640 1 -- 172.21.15.32:0/4202612629 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b58b048f70 con 0x55b58b031270 2024-08-03T20:54:43.225 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.223+0000 7fcba1a00640 1 --2- 172.21.15.32:0/4202612629 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b58b031270 0x55b58afa2f40 secure :-1 s=READY pgs=1029 cs=0 l=1 rev1=1 crypto rx=0x7fcb9c025e30 tx=0x7fcb9c003330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4f73f4e64dd8163a server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:54:43.225 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.223+0000 7fcb9aa00640 1 -- 172.21.15.32:0/4202612629 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb9c064030 con 0x55b58b031270 2024-08-03T20:54:43.225 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.223+0000 7fcb9aa00640 1 -- 172.21.15.32:0/4202612629 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fcb9c004020 con 0x55b58b031270 2024-08-03T20:54:43.225 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.223+0000 7fcb9aa00640 1 -- 172.21.15.32:0/4202612629 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb9c004270 con 0x55b58b031270 2024-08-03T20:54:43.226 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.224+0000 7fcba891bf40 1 -- 172.21.15.32:0/4202612629 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b58b031270 msgr2=0x55b58afa2f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:43.226 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.224+0000 7fcba891bf40 1 --2- 172.21.15.32:0/4202612629 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b58b031270 0x55b58afa2f40 secure :-1 s=READY pgs=1029 cs=0 l=1 rev1=1 crypto rx=0x7fcb9c025e30 tx=0x7fcb9c003330 comp rx=0 tx=0).stop 2024-08-03T20:54:43.227 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.224+0000 7fcba891bf40 1 -- 172.21.15.32:0/4202612629 shutdown_connections 2024-08-03T20:54:43.227 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.224+0000 7fcba891bf40 1 --2- 172.21.15.32:0/4202612629 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b58b031270 0x55b58afa2f40 unknown :-1 s=CLOSED pgs=1029 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:43.227 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.224+0000 7fcba891bf40 1 --2- 172.21.15.32:0/4202612629 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55b58afbc370 0x55b58afa2a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:43.227 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.224+0000 7fcba891bf40 1 --2- 172.21.15.32:0/4202612629 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55b58afba870 0x55b58b030460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:43.227 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.224+0000 7fcba891bf40 1 -- 172.21.15.32:0/4202612629 >> 172.21.15.32:0/4202612629 conn(0x55b58ae92ac0 msgr2=0x55b58afa3b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:54:43.227 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.224+0000 7fcba891bf40 1 -- 172.21.15.32:0/4202612629 shutdown_connections 2024-08-03T20:54:43.227 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.224+0000 7fcba891bf40 1 -- 172.21.15.32:0/4202612629 wait complete. 2024-08-03T20:54:43.227 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.225+0000 7fcba891bf40 1 Processor -- start 2024-08-03T20:54:43.227 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.225+0000 7fcba891bf40 1 -- start start 2024-08-03T20:54:43.229 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.226+0000 7fcba891bf40 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55b58afba870 0x55b58b02bf50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:43.229 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.226+0000 7fcba891bf40 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b58afbc370 0x55b58afb7a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:43.229 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.227+0000 7fcba891bf40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55b58b031270 0x55b58afb7f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:43.230 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.227+0000 7fcba891bf40 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55b58afa1930 con 0x55b58afbc370 2024-08-03T20:54:43.230 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.227+0000 7fcba891bf40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x55b58afa26b0 con 0x55b58b031270 2024-08-03T20:54:43.230 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.227+0000 7fcba891bf40 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x55b58afb5570 con 0x55b58afba870 2024-08-03T20:54:43.230 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.227+0000 7fcb9be00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b58afbc370 0x55b58afb7a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:43.230 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.227+0000 7fcb9be00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b58afbc370 0x55b58afb7a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:37112/0 (socket says 172.21.15.32:37112) 2024-08-03T20:54:43.230 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.227+0000 7fcb9be00640 1 -- 172.21.15.32:0/3235775728 learned_addr learned my addr 172.21.15.32:0/3235775728 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:54:43.230 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.227+0000 7fcba1a00640 1 --2- 172.21.15.32:0/3235775728 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55b58b031270 0x55b58afb7f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:43.230 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.227+0000 7fcba1000640 1 --2- 172.21.15.32:0/3235775728 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55b58afba870 0x55b58b02bf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:43.230 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.227+0000 7fcb9be00640 1 -- 172.21.15.32:0/3235775728 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55b58afba870 msgr2=0x55b58b02bf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:43.230 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.227+0000 7fcb9be00640 1 --2- 172.21.15.32:0/3235775728 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55b58afba870 0x55b58b02bf50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:43.230 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.227+0000 7fcb9be00640 1 -- 172.21.15.32:0/3235775728 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55b58b031270 msgr2=0x55b58afb7f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:43.230 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.227+0000 7fcb9be00640 1 --2- 172.21.15.32:0/3235775728 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55b58b031270 0x55b58afb7f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:43.231 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.227+0000 7fcb9be00640 1 -- 172.21.15.32:0/3235775728 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b58b03f830 con 0x55b58afbc370 2024-08-03T20:54:43.231 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.227+0000 7fcba1000640 1 --2- 172.21.15.32:0/3235775728 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55b58afba870 0x55b58b02bf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-03T20:54:43.231 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.227+0000 7fcba1a00640 1 --2- 172.21.15.32:0/3235775728 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55b58b031270 0x55b58afb7f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-03T20:54:43.231 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.228+0000 7fcb9be00640 1 --2- 172.21.15.32:0/3235775728 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b58afbc370 0x55b58afb7a20 secure :-1 s=READY pgs=1030 cs=0 l=1 rev1=1 crypto rx=0x7fcb8c00a6a0 tx=0x7fcb8c00ab60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:54:43.231 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.228+0000 7fcb99600640 1 -- 172.21.15.32:0/3235775728 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb8c008da0 con 0x55b58afbc370 2024-08-03T20:54:43.231 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.228+0000 7fcb99600640 1 -- 172.21.15.32:0/3235775728 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fcb8c01a070 con 0x55b58afbc370 2024-08-03T20:54:43.231 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.228+0000 7fcba891bf40 1 -- 172.21.15.32:0/3235775728 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x55b58afbb350 con 0x55b58afbc370 2024-08-03T20:54:43.231 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.228+0000 7fcb99600640 1 -- 172.21.15.32:0/3235775728 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb8c015640 con 0x55b58afbc370 2024-08-03T20:54:43.231 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.228+0000 7fcba891bf40 1 -- 172.21.15.32:0/3235775728 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b58b03f550 con 0x55b58afbc370 2024-08-03T20:54:43.232 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.230+0000 7fcb99600640 1 -- 172.21.15.32:0/3235775728 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7fcb8c014070 con 0x55b58afbc370 2024-08-03T20:54:43.232 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.231+0000 7fcb99600640 1 --2- 172.21.15.32:0/3235775728 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fcb800759f0 0x7fcb80077f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:43.232 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.231+0000 7fcb99600640 1 -- 172.21.15.32:0/3235775728 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 5795+0+0 (secure 0 0 0) 0x7fcb8c097ee0 con 0x55b58afbc370 2024-08-03T20:54:43.233 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.231+0000 7fcba891bf40 1 --2- 172.21.15.32:0/3235775728 >> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] conn(0x55b58b02c9a0 0x55b58b02cdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:43.233 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.231+0000 7fcba891bf40 1 -- 172.21.15.32:0/3235775728 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:1 12.f 12:f1f69726:::400.00000000:head [omap-get-keys in=12b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84) -- 0x55b58b02d2f0 con 0x55b58b02c9a0 2024-08-03T20:54:43.233 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.232+0000 7fcba1000640 1 --2- 172.21.15.32:0/3235775728 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fcb800759f0 0x7fcb80077f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:43.233 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.232+0000 7fcba1a00640 1 --2- 172.21.15.32:0/3235775728 >> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] conn(0x55b58b02c9a0 0x55b58b02cdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:43.234 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.232+0000 7fcba1000640 1 --2- 172.21.15.32:0/3235775728 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fcb800759f0 0x7fcb80077f00 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7fcb94002f50 tx=0x7fcb94002550 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:54:43.234 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.233+0000 7fcba1a00640 1 --2- 172.21.15.32:0/3235775728 >> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] conn(0x55b58b02c9a0 0x55b58b02cdb0 crc :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:54:43.234 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.233+0000 7fcba1a00640 1 -- 172.21.15.32:0/3235775728 <== osd.6 v2:172.21.15.118:6825/801947386 1 ==== osd_op_reply(1 400.00000000 [omap-get-keys out=5b] v0'0 uv1 ondisk = 0) ==== 156+0+5 (crc 0 0 0) 0x7fcb9c00b040 con 0x55b58b02c9a0 2024-08-03T20:54:43.235 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.233+0000 7fcba891bf40 1 -- 172.21.15.32:0/3235775728 >> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] conn(0x55b58b02c9a0 msgr2=0x55b58b02cdb0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:43.235 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.233+0000 7fcba891bf40 1 --2- 172.21.15.32:0/3235775728 >> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] conn(0x55b58b02c9a0 0x55b58b02cdb0 crc :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:43.235 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.233+0000 7fcba891bf40 1 -- 172.21.15.32:0/3235775728 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fcb800759f0 msgr2=0x7fcb80077f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:43.235 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.233+0000 7fcba891bf40 1 --2- 172.21.15.32:0/3235775728 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fcb800759f0 0x7fcb80077f00 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7fcb94002f50 tx=0x7fcb94002550 comp rx=0 tx=0).stop 2024-08-03T20:54:43.235 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.233+0000 7fcba891bf40 1 -- 172.21.15.32:0/3235775728 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b58afbc370 msgr2=0x55b58afb7a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:43.235 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.233+0000 7fcba891bf40 1 --2- 172.21.15.32:0/3235775728 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b58afbc370 0x55b58afb7a20 secure :-1 s=READY pgs=1030 cs=0 l=1 rev1=1 crypto rx=0x7fcb8c00a6a0 tx=0x7fcb8c00ab60 comp rx=0 tx=0).stop 2024-08-03T20:54:43.235 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.233+0000 7fcba1a00640 1 -- 172.21.15.32:0/3235775728 reap_dead start 2024-08-03T20:54:43.235 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.233+0000 7fcba891bf40 1 -- 172.21.15.32:0/3235775728 shutdown_connections 2024-08-03T20:54:43.235 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.233+0000 7fcba891bf40 1 -- 172.21.15.32:0/3235775728 >> 172.21.15.32:0/3235775728 conn(0x55b58ae92ac0 msgr2=0x55b58b030460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:54:43.235 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.233+0000 7fcba891bf40 1 -- 172.21.15.32:0/3235775728 shutdown_connections 2024-08-03T20:54:43.235 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.233+0000 7fcba891bf40 1 -- 172.21.15.32:0/3235775728 wait complete. 2024-08-03T20:54:43.245 DEBUG:teuthology.orchestra.run.smithi032:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata listomapkeys 100.00000000.inode 2024-08-03T20:54:43.388 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.387+0000 7f65f240ef40 1 Processor -- start 2024-08-03T20:54:43.388 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.387+0000 7f65f240ef40 1 -- start start 2024-08-03T20:54:43.393 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.388+0000 7f65f240ef40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5561a5e36b40 0x5561a5ea7380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:43.393 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.388+0000 7f65f240ef40 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x5561a5ea9be0 0x5561a5ec54e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:43.393 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.388+0000 7f65f240ef40 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5561a5def3e0 0x5561a5ec5a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:43.393 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.388+0000 7f65f240ef40 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x5561a5d3fe30 con 0x5561a5def3e0 2024-08-03T20:54:43.393 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.388+0000 7f65f240ef40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x5561a5d9dbf0 con 0x5561a5e36b40 2024-08-03T20:54:43.393 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.388+0000 7f65f240ef40 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x5561a5eaa510 con 0x5561a5ea9be0 2024-08-03T20:54:43.393 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.389+0000 7f65eb200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5561a5def3e0 0x5561a5ec5a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:43.394 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.389+0000 7f65eb200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5561a5def3e0 0x5561a5ec5a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:37142/0 (socket says 172.21.15.32:37142) 2024-08-03T20:54:43.394 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.389+0000 7f65eb200640 1 -- 172.21.15.32:0/2021530533 learned_addr learned my addr 172.21.15.32:0/2021530533 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:54:43.394 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.389+0000 7f65ea800640 1 --2- 172.21.15.32:0/2021530533 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5561a5e36b40 0x5561a5ea7380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:43.394 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.389+0000 7f65e9e00640 1 --2- 172.21.15.32:0/2021530533 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x5561a5ea9be0 0x5561a5ec54e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:43.394 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.389+0000 7f65eb200640 1 -- 172.21.15.32:0/2021530533 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x5561a5ea9be0 msgr2=0x5561a5ec54e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:54:43.394 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.389+0000 7f65eb200640 1 --2- 172.21.15.32:0/2021530533 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x5561a5ea9be0 0x5561a5ec54e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:43.394 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.389+0000 7f65eb200640 1 -- 172.21.15.32:0/2021530533 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5561a5e36b40 msgr2=0x5561a5ea7380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:54:43.394 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.389+0000 7f65eb200640 1 --2- 172.21.15.32:0/2021530533 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5561a5e36b40 0x5561a5ea7380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:43.394 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.389+0000 7f65eb200640 1 -- 172.21.15.32:0/2021530533 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5561a5ea6ca0 con 0x5561a5def3e0 2024-08-03T20:54:43.394 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.389+0000 7f65e9e00640 1 --2- 172.21.15.32:0/2021530533 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x5561a5ea9be0 0x5561a5ec54e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-03T20:54:43.394 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.390+0000 7f65eb200640 1 --2- 172.21.15.32:0/2021530533 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5561a5def3e0 0x5561a5ec5a20 secure :-1 s=READY pgs=1033 cs=0 l=1 rev1=1 crypto rx=0x7f65e40015b0 tx=0x7f65e4001a70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d9df4e633fad033c server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:54:43.394 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.390+0000 7f65e8a00640 1 -- 172.21.15.32:0/2021530533 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65e4017070 con 0x5561a5def3e0 2024-08-03T20:54:43.395 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.390+0000 7f65e8a00640 1 -- 172.21.15.32:0/2021530533 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f65e4002780 con 0x5561a5def3e0 2024-08-03T20:54:43.395 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.390+0000 7f65e8a00640 1 -- 172.21.15.32:0/2021530533 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65e4002bb0 con 0x5561a5def3e0 2024-08-03T20:54:43.395 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.390+0000 7f65f240ef40 1 -- 172.21.15.32:0/2021530533 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5561a5def3e0 msgr2=0x5561a5ec5a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:43.395 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.390+0000 7f65f240ef40 1 --2- 172.21.15.32:0/2021530533 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5561a5def3e0 0x5561a5ec5a20 secure :-1 s=READY pgs=1033 cs=0 l=1 rev1=1 crypto rx=0x7f65e40015b0 tx=0x7f65e4001a70 comp rx=0 tx=0).stop 2024-08-03T20:54:43.395 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.390+0000 7f65f240ef40 1 -- 172.21.15.32:0/2021530533 shutdown_connections 2024-08-03T20:54:43.395 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.390+0000 7f65f240ef40 1 --2- 172.21.15.32:0/2021530533 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5561a5def3e0 0x5561a5ec5a20 unknown :-1 s=CLOSED pgs=1033 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:43.395 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.390+0000 7f65f240ef40 1 --2- 172.21.15.32:0/2021530533 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x5561a5ea9be0 0x5561a5ec54e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:43.395 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.390+0000 7f65f240ef40 1 --2- 172.21.15.32:0/2021530533 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5561a5e36b40 0x5561a5ea7380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:43.395 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.390+0000 7f65f240ef40 1 -- 172.21.15.32:0/2021530533 >> 172.21.15.32:0/2021530533 conn(0x5561a5d11ac0 msgr2=0x5561a5ec9d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:54:43.395 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.391+0000 7f65f240ef40 1 -- 172.21.15.32:0/2021530533 shutdown_connections 2024-08-03T20:54:43.395 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.391+0000 7f65f240ef40 1 -- 172.21.15.32:0/2021530533 wait complete. 2024-08-03T20:54:43.395 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.392+0000 7f65f240ef40 1 Processor -- start 2024-08-03T20:54:43.395 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.392+0000 7f65f240ef40 1 -- start start 2024-08-03T20:54:43.397 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.393+0000 7f65f240ef40 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5561a5def3e0 0x5561a5ea2950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:43.397 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.393+0000 7f65f240ef40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5561a5e36b40 0x5561a5e39f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:43.397 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.393+0000 7f65f240ef40 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x5561a5ea9be0 0x5561a5e3a4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:43.398 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.393+0000 7f65f240ef40 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x5561a5df0090 con 0x5561a5def3e0 2024-08-03T20:54:43.398 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.393+0000 7f65f240ef40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x5561a5f52fe0 con 0x5561a5e36b40 2024-08-03T20:54:43.398 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.393+0000 7f65f240ef40 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x5561a5ec6a10 con 0x5561a5ea9be0 2024-08-03T20:54:43.398 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.393+0000 7f65ea800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5561a5def3e0 0x5561a5ea2950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:43.398 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.393+0000 7f65ea800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5561a5def3e0 0x5561a5ea2950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:37156/0 (socket says 172.21.15.32:37156) 2024-08-03T20:54:43.398 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.393+0000 7f65ea800640 1 -- 172.21.15.32:0/388544910 learned_addr learned my addr 172.21.15.32:0/388544910 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:54:43.398 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.393+0000 7f65e9e00640 1 --2- 172.21.15.32:0/388544910 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5561a5e36b40 0x5561a5e39f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:43.398 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.394+0000 7f65eb200640 1 --2- 172.21.15.32:0/388544910 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x5561a5ea9be0 0x5561a5e3a4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:43.398 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.394+0000 7f65ea800640 1 -- 172.21.15.32:0/388544910 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x5561a5ea9be0 msgr2=0x5561a5e3a4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:43.398 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.394+0000 7f65ea800640 1 --2- 172.21.15.32:0/388544910 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x5561a5ea9be0 0x5561a5e3a4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:43.398 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.394+0000 7f65ea800640 1 -- 172.21.15.32:0/388544910 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5561a5e36b40 msgr2=0x5561a5e39f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:43.398 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.394+0000 7f65ea800640 1 --2- 172.21.15.32:0/388544910 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5561a5e36b40 0x5561a5e39f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:43.399 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.394+0000 7f65ea800640 1 -- 172.21.15.32:0/388544910 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5561a5ea62d0 con 0x5561a5def3e0 2024-08-03T20:54:43.399 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.394+0000 7f65ea800640 1 --2- 172.21.15.32:0/388544910 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5561a5def3e0 0x5561a5ea2950 secure :-1 s=READY pgs=1034 cs=0 l=1 rev1=1 crypto rx=0x7f65dc025db0 tx=0x7f65dc04d770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:54:43.399 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.394+0000 7f65e3400640 1 -- 172.21.15.32:0/388544910 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65dc068070 con 0x5561a5def3e0 2024-08-03T20:54:43.399 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.394+0000 7f65f240ef40 1 -- 172.21.15.32:0/388544910 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x5561a5ea59a0 con 0x5561a5def3e0 2024-08-03T20:54:43.399 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.394+0000 7f65e3400640 1 -- 172.21.15.32:0/388544910 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f65dc056620 con 0x5561a5def3e0 2024-08-03T20:54:43.399 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.394+0000 7f65e3400640 1 -- 172.21.15.32:0/388544910 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65dc056a10 con 0x5561a5def3e0 2024-08-03T20:54:43.399 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.395+0000 7f65f240ef40 1 -- 172.21.15.32:0/388544910 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5561a5ea5f60 con 0x5561a5def3e0 2024-08-03T20:54:43.399 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.397+0000 7f65e3400640 1 -- 172.21.15.32:0/388544910 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f65dc06b070 con 0x5561a5def3e0 2024-08-03T20:54:43.401 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.398+0000 7f65e3400640 1 --2- 172.21.15.32:0/388544910 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f65cc0759f0 0x7f65cc077f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:43.401 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.398+0000 7f65e3400640 1 -- 172.21.15.32:0/388544910 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 5795+0+0 (secure 0 0 0) 0x7f65dc0db560 con 0x5561a5def3e0 2024-08-03T20:54:43.401 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.398+0000 7f65e9e00640 1 --2- 172.21.15.32:0/388544910 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f65cc0759f0 0x7f65cc077f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:43.401 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.398+0000 7f65f240ef40 1 --2- 172.21.15.32:0/388544910 >> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] conn(0x5561a5df0310 0x5561a5f52660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:43.401 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.398+0000 7f65f240ef40 1 -- 172.21.15.32:0/388544910 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:1 12.1f 12:fe07bba1:::100.00000000.inode:head [omap-get-keys in=12b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84) -- 0x5561a5f52ba0 con 0x5561a5df0310 2024-08-03T20:54:43.401 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.398+0000 7f65eb200640 1 --2- 172.21.15.32:0/388544910 >> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] conn(0x5561a5df0310 0x5561a5f52660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:43.401 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.399+0000 7f65e9e00640 1 --2- 172.21.15.32:0/388544910 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f65cc0759f0 0x7f65cc077f00 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x5561a5ec60f0 tx=0x7f65d8004a40 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:54:43.401 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.399+0000 7f65eb200640 1 --2- 172.21.15.32:0/388544910 >> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] conn(0x5561a5df0310 0x5561a5f52660 crc :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:54:43.401 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.399+0000 7f65eb200640 1 -- 172.21.15.32:0/388544910 <== osd.6 v2:172.21.15.118:6825/801947386 1 ==== osd_op_reply(1 100.00000000.inode [omap-get-keys out=5b] v0'0 uv3 ondisk = 0) ==== 162+0+5 (crc 0 0 0) 0x7f65e400f070 con 0x5561a5df0310 2024-08-03T20:54:43.401 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.399+0000 7f65f240ef40 1 -- 172.21.15.32:0/388544910 >> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] conn(0x5561a5df0310 msgr2=0x5561a5f52660 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:43.402 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.399+0000 7f65f240ef40 1 --2- 172.21.15.32:0/388544910 >> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] conn(0x5561a5df0310 0x5561a5f52660 crc :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:43.402 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.400+0000 7f65f240ef40 1 -- 172.21.15.32:0/388544910 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f65cc0759f0 msgr2=0x7f65cc077f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:43.402 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.400+0000 7f65f240ef40 1 --2- 172.21.15.32:0/388544910 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f65cc0759f0 0x7f65cc077f00 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x5561a5ec60f0 tx=0x7f65d8004a40 comp rx=0 tx=0).stop 2024-08-03T20:54:43.402 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.400+0000 7f65f240ef40 1 -- 172.21.15.32:0/388544910 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5561a5def3e0 msgr2=0x5561a5ea2950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:43.402 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.400+0000 7f65f240ef40 1 --2- 172.21.15.32:0/388544910 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5561a5def3e0 0x5561a5ea2950 secure :-1 s=READY pgs=1034 cs=0 l=1 rev1=1 crypto rx=0x7f65dc025db0 tx=0x7f65dc04d770 comp rx=0 tx=0).stop 2024-08-03T20:54:43.402 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.400+0000 7f65eb200640 1 -- 172.21.15.32:0/388544910 reap_dead start 2024-08-03T20:54:43.402 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.400+0000 7f65f240ef40 1 -- 172.21.15.32:0/388544910 shutdown_connections 2024-08-03T20:54:43.402 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.400+0000 7f65f240ef40 1 -- 172.21.15.32:0/388544910 >> 172.21.15.32:0/388544910 conn(0x5561a5d11ac0 msgr2=0x5561a5df14d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:54:43.402 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.400+0000 7f65f240ef40 1 -- 172.21.15.32:0/388544910 shutdown_connections 2024-08-03T20:54:43.402 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.400+0000 7f65f240ef40 1 -- 172.21.15.32:0/388544910 wait complete. 2024-08-03T20:54:43.413 DEBUG:teuthology.orchestra.run.smithi032:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata listomapkeys 1.00000000 2024-08-03T20:54:43.557 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.557+0000 7f1ee20aff40 1 Processor -- start 2024-08-03T20:54:43.557 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.557+0000 7f1ee20aff40 1 -- start start 2024-08-03T20:54:43.559 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.558+0000 7f1ee20aff40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55e779113360 0x55e77912b000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:43.559 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.558+0000 7f1ee20aff40 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55e77912a330 0x55e7790a3de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:43.559 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.558+0000 7f1ee20aff40 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55e7791313f0 0x55e7790a4320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:43.559 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.558+0000 7f1ee20aff40 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55e779008bf0 con 0x55e7791313f0 2024-08-03T20:54:43.560 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.558+0000 7f1ee20aff40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x55e7790156b0 con 0x55e779113360 2024-08-03T20:54:43.560 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.558+0000 7f1ee20aff40 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x55e778faae30 con 0x55e77912a330 2024-08-03T20:54:43.560 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.559+0000 7f1edbc00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55e7791313f0 0x55e7790a4320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:43.560 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.559+0000 7f1edbc00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55e7791313f0 0x55e7790a4320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:37174/0 (socket says 172.21.15.32:37174) 2024-08-03T20:54:43.560 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.559+0000 7f1edbc00640 1 -- 172.21.15.32:0/630942115 learned_addr learned my addr 172.21.15.32:0/630942115 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:54:43.560 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.559+0000 7f1edb200640 1 --2- 172.21.15.32:0/630942115 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55e779113360 0x55e77912b000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:43.560 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.559+0000 7f1eda800640 1 --2- 172.21.15.32:0/630942115 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55e77912a330 0x55e7790a3de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:43.560 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.559+0000 7f1edbc00640 1 -- 172.21.15.32:0/630942115 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55e77912a330 msgr2=0x55e7790a3de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:54:43.560 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.559+0000 7f1edbc00640 1 --2- 172.21.15.32:0/630942115 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55e77912a330 0x55e7790a3de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:43.560 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.559+0000 7f1edbc00640 1 -- 172.21.15.32:0/630942115 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55e779113360 msgr2=0x55e77912b000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:54:43.560 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.559+0000 7f1edbc00640 1 --2- 172.21.15.32:0/630942115 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55e779113360 0x55e77912b000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:43.560 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.559+0000 7f1edbc00640 1 -- 172.21.15.32:0/630942115 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e779129a30 con 0x55e7791313f0 2024-08-03T20:54:43.562 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.559+0000 7f1edbc00640 1 --2- 172.21.15.32:0/630942115 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55e7791313f0 0x55e7790a4320 secure :-1 s=READY pgs=1037 cs=0 l=1 rev1=1 crypto rx=0x7f1ed400a550 tx=0x7f1ed4003330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8e403fb2ff2d6033 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:54:43.562 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.560+0000 7f1ed9400640 1 -- 172.21.15.32:0/630942115 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ed4064020 con 0x55e7791313f0 2024-08-03T20:54:43.562 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.560+0000 7f1ed9400640 1 -- 172.21.15.32:0/630942115 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f1ed404dae0 con 0x55e7791313f0 2024-08-03T20:54:43.562 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.560+0000 7f1ed9400640 1 -- 172.21.15.32:0/630942115 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ed4060470 con 0x55e7791313f0 2024-08-03T20:54:43.562 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.560+0000 7f1ee20aff40 1 -- 172.21.15.32:0/630942115 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55e7791313f0 msgr2=0x55e7790a4320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:43.562 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.560+0000 7f1ee20aff40 1 --2- 172.21.15.32:0/630942115 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55e7791313f0 0x55e7790a4320 secure :-1 s=READY pgs=1037 cs=0 l=1 rev1=1 crypto rx=0x7f1ed400a550 tx=0x7f1ed4003330 comp rx=0 tx=0).stop 2024-08-03T20:54:43.562 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.560+0000 7f1ee20aff40 1 -- 172.21.15.32:0/630942115 shutdown_connections 2024-08-03T20:54:43.563 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.560+0000 7f1ee20aff40 1 --2- 172.21.15.32:0/630942115 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55e7791313f0 0x55e7790a4320 unknown :-1 s=CLOSED pgs=1037 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:43.563 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.560+0000 7f1ee20aff40 1 --2- 172.21.15.32:0/630942115 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55e77912a330 0x55e7790a3de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:43.563 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.560+0000 7f1ee20aff40 1 --2- 172.21.15.32:0/630942115 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55e779113360 0x55e77912b000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:43.563 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.560+0000 7f1ee20aff40 1 -- 172.21.15.32:0/630942115 >> 172.21.15.32:0/630942115 conn(0x55e778f7cac0 msgr2=0x55e77912a8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:54:43.563 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.560+0000 7f1ee20aff40 1 -- 172.21.15.32:0/630942115 shutdown_connections 2024-08-03T20:54:43.563 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.561+0000 7f1ee20aff40 1 -- 172.21.15.32:0/630942115 wait complete. 2024-08-03T20:54:43.563 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.562+0000 7f1ee20aff40 1 Processor -- start 2024-08-03T20:54:43.563 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.562+0000 7f1ee20aff40 1 -- start start 2024-08-03T20:54:43.565 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.563+0000 7f1ee20aff40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55e779113360 0x55e77910b0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:43.565 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.563+0000 7f1ee20aff40 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55e77912a330 0x55e77910c670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:43.565 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.563+0000 7f1ee20aff40 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55e7791313f0 0x55e77910cbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:43.565 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.563+0000 7f1ee20aff40 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55e77909b020 con 0x55e77912a330 2024-08-03T20:54:43.565 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.563+0000 7f1ee20aff40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x55e77905c700 con 0x55e779113360 2024-08-03T20:54:43.565 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.563+0000 7f1ee20aff40 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x55e77905c9e0 con 0x55e7791313f0 2024-08-03T20:54:43.565 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.563+0000 7f1eda800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55e77912a330 0x55e77910c670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:43.565 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.563+0000 7f1eda800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55e77912a330 0x55e77910c670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:37184/0 (socket says 172.21.15.32:37184) 2024-08-03T20:54:43.565 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.563+0000 7f1eda800640 1 -- 172.21.15.32:0/1721752433 learned_addr learned my addr 172.21.15.32:0/1721752433 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:54:43.566 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.564+0000 7f1edb200640 1 --2- 172.21.15.32:0/1721752433 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55e779113360 0x55e77910b0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:43.566 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.564+0000 7f1eda800640 1 -- 172.21.15.32:0/1721752433 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55e7791313f0 msgr2=0x55e77910cbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:43.566 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.564+0000 7f1edbc00640 1 --2- 172.21.15.32:0/1721752433 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55e7791313f0 0x55e77910cbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:43.566 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.564+0000 7f1eda800640 1 --2- 172.21.15.32:0/1721752433 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55e7791313f0 0x55e77910cbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:43.566 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.564+0000 7f1eda800640 1 -- 172.21.15.32:0/1721752433 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55e779113360 msgr2=0x55e77910b0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:43.566 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.564+0000 7f1eda800640 1 --2- 172.21.15.32:0/1721752433 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55e779113360 0x55e77910b0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:43.566 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.564+0000 7f1eda800640 1 -- 172.21.15.32:0/1721752433 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e779112500 con 0x55e77912a330 2024-08-03T20:54:43.566 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.564+0000 7f1edb200640 1 --2- 172.21.15.32:0/1721752433 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55e779113360 0x55e77910b0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-03T20:54:43.566 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.564+0000 7f1edbc00640 1 --2- 172.21.15.32:0/1721752433 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55e7791313f0 0x55e77910cbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-03T20:54:43.566 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.564+0000 7f1eda800640 1 --2- 172.21.15.32:0/1721752433 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55e77912a330 0x55e77910c670 secure :-1 s=READY pgs=1038 cs=0 l=1 rev1=1 crypto rx=0x7f1ec800be70 tx=0x7f1ec800a750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:54:43.566 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.564+0000 7f1ed3e00640 1 -- 172.21.15.32:0/1721752433 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ec80085e0 con 0x55e77912a330 2024-08-03T20:54:43.566 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.564+0000 7f1ee20aff40 1 -- 172.21.15.32:0/1721752433 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x55e7791b8710 con 0x55e77912a330 2024-08-03T20:54:43.566 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.564+0000 7f1ed3e00640 1 -- 172.21.15.32:0/1721752433 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f1ec8008780 con 0x55e77912a330 2024-08-03T20:54:43.567 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.564+0000 7f1ed3e00640 1 -- 172.21.15.32:0/1721752433 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ec8008b90 con 0x55e77912a330 2024-08-03T20:54:43.567 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.565+0000 7f1ee20aff40 1 -- 172.21.15.32:0/1721752433 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e77909c790 con 0x55e77912a330 2024-08-03T20:54:43.567 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.567+0000 7f1ed3e00640 1 -- 172.21.15.32:0/1721752433 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f1ec801e440 con 0x55e77912a330 2024-08-03T20:54:43.568 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.568+0000 7f1ed3e00640 1 --2- 172.21.15.32:0/1721752433 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f1ebc0759f0 0x7f1ebc077f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:43.569 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.568+0000 7f1ed3e00640 1 -- 172.21.15.32:0/1721752433 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 5795+0+0 (secure 0 0 0) 0x7f1ec8097770 con 0x55e77912a330 2024-08-03T20:54:43.569 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.568+0000 7f1ee20aff40 1 --2- 172.21.15.32:0/1721752433 >> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] conn(0x55e77912b090 0x55e7791d4e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:43.569 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.568+0000 7f1ee20aff40 1 -- 172.21.15.32:0/1721752433 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:1 12.1f 12:ff5b34d6:::1.00000000:head [omap-get-keys in=12b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84) -- 0x55e7791d5340 con 0x55e77912b090 2024-08-03T20:54:43.569 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.568+0000 7f1edb200640 1 --2- 172.21.15.32:0/1721752433 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f1ebc0759f0 0x7f1ebc077f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:43.569 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.569+0000 7f1edbc00640 1 --2- 172.21.15.32:0/1721752433 >> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] conn(0x55e77912b090 0x55e7791d4e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:43.569 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.569+0000 7f1edb200640 1 --2- 172.21.15.32:0/1721752433 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f1ebc0759f0 0x7f1ebc077f00 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f1ecc000c00 tx=0x7f1ecc001690 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:54:43.570 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.569+0000 7f1edbc00640 1 --2- 172.21.15.32:0/1721752433 >> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] conn(0x55e77912b090 0x55e7791d4e00 crc :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:54:43.570 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.570+0000 7f1edbc00640 1 -- 172.21.15.32:0/1721752433 <== osd.6 v2:172.21.15.118:6825/801947386 1 ==== osd_op_reply(1 1.00000000 [omap-get-keys out=20b] v0'0 uv6 ondisk = 0) ==== 154+0+20 (crc 0 0 0) 0x7f1ed400b040 con 0x55e77912b090 2024-08-03T20:54:43.570 INFO:teuthology.orchestra.run.smithi032.stdout:subdir_head 2024-08-03T20:54:43.571 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.570+0000 7f1ee20aff40 1 -- 172.21.15.32:0/1721752433 >> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] conn(0x55e77912b090 msgr2=0x55e7791d4e00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:43.571 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.570+0000 7f1ee20aff40 1 --2- 172.21.15.32:0/1721752433 >> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] conn(0x55e77912b090 0x55e7791d4e00 crc :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:43.571 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.570+0000 7f1ee20aff40 1 -- 172.21.15.32:0/1721752433 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f1ebc0759f0 msgr2=0x7f1ebc077f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:43.571 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.570+0000 7f1ee20aff40 1 --2- 172.21.15.32:0/1721752433 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f1ebc0759f0 0x7f1ebc077f00 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f1ecc000c00 tx=0x7f1ecc001690 comp rx=0 tx=0).stop 2024-08-03T20:54:43.572 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.570+0000 7f1ee20aff40 1 -- 172.21.15.32:0/1721752433 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55e77912a330 msgr2=0x55e77910c670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:43.572 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.570+0000 7f1ee20aff40 1 --2- 172.21.15.32:0/1721752433 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55e77912a330 0x55e77910c670 secure :-1 s=READY pgs=1038 cs=0 l=1 rev1=1 crypto rx=0x7f1ec800be70 tx=0x7f1ec800a750 comp rx=0 tx=0).stop 2024-08-03T20:54:43.572 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.570+0000 7f1edbc00640 1 -- 172.21.15.32:0/1721752433 reap_dead start 2024-08-03T20:54:43.572 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.570+0000 7f1ee20aff40 1 -- 172.21.15.32:0/1721752433 shutdown_connections 2024-08-03T20:54:43.572 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.570+0000 7f1ee20aff40 1 -- 172.21.15.32:0/1721752433 >> 172.21.15.32:0/1721752433 conn(0x55e778f7cac0 msgr2=0x55e77912a8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:54:43.572 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.570+0000 7f1ee20aff40 1 -- 172.21.15.32:0/1721752433 shutdown_connections 2024-08-03T20:54:43.572 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.570+0000 7f1ee20aff40 1 -- 172.21.15.32:0/1721752433 wait complete. 2024-08-03T20:54:43.580 DEBUG:teuthology.orchestra.run.smithi032:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata stat 10000000000.00000000 2024-08-03T20:54:43.723 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.723+0000 7fdedfd1df40 1 Processor -- start 2024-08-03T20:54:43.724 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.723+0000 7fdedfd1df40 1 -- start start 2024-08-03T20:54:43.726 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.724+0000 7fdedfd1df40 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55ef72450360 0x55ef724686a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:43.726 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.724+0000 7fdedfd1df40 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ef72451ea0 0x55ef724691f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:43.726 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.724+0000 7fdedfd1df40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55ef7246dcb0 0x55ef72398970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:43.727 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.724+0000 7fdedfd1df40 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55ef722e7e30 con 0x55ef72451ea0 2024-08-03T20:54:43.727 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.724+0000 7fdedfd1df40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x55ef72345bf0 con 0x55ef7246dcb0 2024-08-03T20:54:43.727 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.724+0000 7fdedfd1df40 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x55ef724511b0 con 0x55ef72450360 2024-08-03T20:54:43.727 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.725+0000 7fded7a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ef72451ea0 0x55ef724691f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:43.727 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.725+0000 7fded7a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ef72451ea0 0x55ef724691f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:37204/0 (socket says 172.21.15.32:37204) 2024-08-03T20:54:43.727 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.725+0000 7fded7a00640 1 -- 172.21.15.32:0/3126883204 learned_addr learned my addr 172.21.15.32:0/3126883204 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:54:43.727 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.725+0000 7fded8400640 1 --2- 172.21.15.32:0/3126883204 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55ef72450360 0x55ef724686a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:43.727 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.725+0000 7fded8e00640 1 --2- 172.21.15.32:0/3126883204 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55ef7246dcb0 0x55ef72398970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:43.728 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.725+0000 7fded7a00640 1 -- 172.21.15.32:0/3126883204 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55ef72450360 msgr2=0x55ef724686a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:54:43.728 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.725+0000 7fded7a00640 1 --2- 172.21.15.32:0/3126883204 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55ef72450360 0x55ef724686a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:43.728 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.725+0000 7fded7a00640 1 -- 172.21.15.32:0/3126883204 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55ef7246dcb0 msgr2=0x55ef72398970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:54:43.728 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.725+0000 7fded7a00640 1 --2- 172.21.15.32:0/3126883204 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55ef7246dcb0 0x55ef72398970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:43.728 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.725+0000 7fded7a00640 1 -- 172.21.15.32:0/3126883204 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ef7246eec0 con 0x55ef72451ea0 2024-08-03T20:54:43.728 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.725+0000 7fded8e00640 1 --2- 172.21.15.32:0/3126883204 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55ef7246dcb0 0x55ef72398970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-03T20:54:43.728 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.725+0000 7fded7a00640 1 --2- 172.21.15.32:0/3126883204 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ef72451ea0 0x55ef724691f0 secure :-1 s=READY pgs=1041 cs=0 l=1 rev1=1 crypto rx=0x7fdec40464c0 tx=0x7fdec4002fb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f61c935d43af401d server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:54:43.728 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.726+0000 7fded6600640 1 -- 172.21.15.32:0/3126883204 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdec404d820 con 0x55ef72451ea0 2024-08-03T20:54:43.728 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.726+0000 7fded6600640 1 -- 172.21.15.32:0/3126883204 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fdec404dcd0 con 0x55ef72451ea0 2024-08-03T20:54:43.728 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.726+0000 7fded6600640 1 -- 172.21.15.32:0/3126883204 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdec40566d0 con 0x55ef72451ea0 2024-08-03T20:54:43.730 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.726+0000 7fdedfd1df40 1 -- 172.21.15.32:0/3126883204 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ef72451ea0 msgr2=0x55ef724691f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:43.730 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.726+0000 7fdedfd1df40 1 --2- 172.21.15.32:0/3126883204 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ef72451ea0 0x55ef724691f0 secure :-1 s=READY pgs=1041 cs=0 l=1 rev1=1 crypto rx=0x7fdec40464c0 tx=0x7fdec4002fb0 comp rx=0 tx=0).stop 2024-08-03T20:54:43.730 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.726+0000 7fdedfd1df40 1 -- 172.21.15.32:0/3126883204 shutdown_connections 2024-08-03T20:54:43.730 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.726+0000 7fdedfd1df40 1 --2- 172.21.15.32:0/3126883204 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55ef7246dcb0 0x55ef72398970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:43.730 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.726+0000 7fdedfd1df40 1 --2- 172.21.15.32:0/3126883204 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ef72451ea0 0x55ef724691f0 unknown :-1 s=CLOSED pgs=1041 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:43.731 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.726+0000 7fdedfd1df40 1 --2- 172.21.15.32:0/3126883204 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55ef72450360 0x55ef724686a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:43.731 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.726+0000 7fdedfd1df40 1 -- 172.21.15.32:0/3126883204 >> 172.21.15.32:0/3126883204 conn(0x55ef722b9ac0 msgr2=0x55ef72469e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:54:43.731 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.726+0000 7fdedfd1df40 1 -- 172.21.15.32:0/3126883204 shutdown_connections 2024-08-03T20:54:43.731 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.726+0000 7fdedfd1df40 1 -- 172.21.15.32:0/3126883204 wait complete. 2024-08-03T20:54:43.731 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.727+0000 7fdedfd1df40 1 Processor -- start 2024-08-03T20:54:43.731 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.728+0000 7fdedfd1df40 1 -- start start 2024-08-03T20:54:43.731 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.729+0000 7fdedfd1df40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55ef72450360 0x55ef7244d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:43.731 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.729+0000 7fdedfd1df40 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55ef72451ea0 0x55ef72449160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:43.731 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.729+0000 7fdedfd1df40 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ef7246dcb0 0x55ef723dd640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:43.731 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.729+0000 7fdedfd1df40 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55ef7246ce60 con 0x55ef7246dcb0 2024-08-03T20:54:43.731 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.729+0000 7fdedfd1df40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x55ef723993d0 con 0x55ef72450360 2024-08-03T20:54:43.731 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.729+0000 7fdedfd1df40 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x55ef7244daf0 con 0x55ef72451ea0 2024-08-03T20:54:43.731 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.729+0000 7fded8e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ef7246dcb0 0x55ef723dd640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:43.732 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.729+0000 7fded8e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ef7246dcb0 0x55ef723dd640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:37214/0 (socket says 172.21.15.32:37214) 2024-08-03T20:54:43.732 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.729+0000 7fded8e00640 1 -- 172.21.15.32:0/3200991834 learned_addr learned my addr 172.21.15.32:0/3200991834 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:54:43.732 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.729+0000 7fded8400640 1 --2- 172.21.15.32:0/3200991834 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55ef72450360 0x55ef7244d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:43.732 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.729+0000 7fded7a00640 1 --2- 172.21.15.32:0/3200991834 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55ef72451ea0 0x55ef72449160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:43.732 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.730+0000 7fded8e00640 1 -- 172.21.15.32:0/3200991834 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55ef72451ea0 msgr2=0x55ef72449160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:43.732 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.730+0000 7fded8e00640 1 --2- 172.21.15.32:0/3200991834 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55ef72451ea0 0x55ef72449160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:43.732 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.730+0000 7fded8e00640 1 -- 172.21.15.32:0/3200991834 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55ef72450360 msgr2=0x55ef7244d330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:43.732 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.730+0000 7fded8e00640 1 --2- 172.21.15.32:0/3200991834 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55ef72450360 0x55ef7244d330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:43.732 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.730+0000 7fded8e00640 1 -- 172.21.15.32:0/3200991834 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ef723dce80 con 0x55ef7246dcb0 2024-08-03T20:54:43.732 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.730+0000 7fded8400640 1 --2- 172.21.15.32:0/3200991834 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55ef72450360 0x55ef7244d330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-03T20:54:43.732 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.730+0000 7fded7a00640 1 --2- 172.21.15.32:0/3200991834 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55ef72451ea0 0x55ef72449160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-03T20:54:43.732 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.730+0000 7fded8e00640 1 --2- 172.21.15.32:0/3200991834 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ef7246dcb0 0x55ef723dd640 secure :-1 s=READY pgs=1042 cs=0 l=1 rev1=1 crypto rx=0x7fded0006f90 tx=0x7fded000d5f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:54:43.733 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.730+0000 7fded5200640 1 -- 172.21.15.32:0/3200991834 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fded000de20 con 0x55ef7246dcb0 2024-08-03T20:54:43.733 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.730+0000 7fdedfd1df40 1 -- 172.21.15.32:0/3200991834 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x55ef7244c050 con 0x55ef7246dcb0 2024-08-03T20:54:43.733 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.730+0000 7fded5200640 1 -- 172.21.15.32:0/3200991834 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fded0018070 con 0x55ef7246dcb0 2024-08-03T20:54:43.733 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.730+0000 7fded5200640 1 -- 172.21.15.32:0/3200991834 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fded0012830 con 0x55ef7246dcb0 2024-08-03T20:54:43.734 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.730+0000 7fdedfd1df40 1 -- 172.21.15.32:0/3200991834 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ef723dd120 con 0x55ef7246dcb0 2024-08-03T20:54:43.734 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.733+0000 7fded5200640 1 -- 172.21.15.32:0/3200991834 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7fded00129d0 con 0x55ef7246dcb0 2024-08-03T20:54:43.735 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.734+0000 7fded5200640 1 --2- 172.21.15.32:0/3200991834 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fdeb8075a20 0x7fdeb8077f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:43.735 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.734+0000 7fded5200640 1 -- 172.21.15.32:0/3200991834 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 5795+0+0 (secure 0 0 0) 0x7fded0097800 con 0x55ef7246dcb0 2024-08-03T20:54:43.735 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.734+0000 7fdedfd1df40 1 --2- 172.21.15.32:0/3200991834 >> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] conn(0x55ef7246e380 0x55ef7246e790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:43.735 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.734+0000 7fdedfd1df40 1 -- 172.21.15.32:0/3200991834 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:1 12.10 12:0cf6ad0f:::10000000000.00000000:head [stat] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84) -- 0x55ef72509880 con 0x55ef7246e380 2024-08-03T20:54:43.735 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.734+0000 7fded7a00640 1 --2- 172.21.15.32:0/3200991834 >> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] conn(0x55ef7246e380 0x55ef7246e790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:43.735 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.734+0000 7fded8400640 1 --2- 172.21.15.32:0/3200991834 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fdeb8075a20 0x7fdeb8077f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:43.736 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.735+0000 7fded8400640 1 --2- 172.21.15.32:0/3200991834 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fdeb8075a20 0x7fdeb8077f30 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7fdec8004fd0 tx=0x7fdec800f040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:54:43.736 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.735+0000 7fded7a00640 1 --2- 172.21.15.32:0/3200991834 >> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] conn(0x55ef7246e380 0x55ef7246e790 crc :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:54:43.736 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.735+0000 7fded7a00640 1 -- 172.21.15.32:0/3200991834 <== osd.0 v2:172.21.15.32:6816/3066156116 1 ==== osd_op_reply(1 10000000000.00000000 [stat out=16b] v0'0 uv10 ondisk = 0) ==== 164+0+16 (crc 0 0 0) 0x7fdec405f030 con 0x55ef7246e380 2024-08-03T20:54:43.737 INFO:teuthology.orchestra.run.smithi032.stdout:cephfs_metadata/10000000000.00000000 mtime 2024-08-03T20:54:37.000000+0000, size 0 2024-08-03T20:54:43.737 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.736+0000 7fdedfd1df40 1 -- 172.21.15.32:0/3200991834 >> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] conn(0x55ef7246e380 msgr2=0x55ef7246e790 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:43.737 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.736+0000 7fdedfd1df40 1 --2- 172.21.15.32:0/3200991834 >> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] conn(0x55ef7246e380 0x55ef7246e790 crc :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:43.738 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.736+0000 7fdedfd1df40 1 -- 172.21.15.32:0/3200991834 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fdeb8075a20 msgr2=0x7fdeb8077f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:43.738 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.736+0000 7fdedfd1df40 1 --2- 172.21.15.32:0/3200991834 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fdeb8075a20 0x7fdeb8077f30 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7fdec8004fd0 tx=0x7fdec800f040 comp rx=0 tx=0).stop 2024-08-03T20:54:43.738 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.736+0000 7fdedfd1df40 1 -- 172.21.15.32:0/3200991834 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ef7246dcb0 msgr2=0x55ef723dd640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:43.738 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.736+0000 7fdedfd1df40 1 --2- 172.21.15.32:0/3200991834 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ef7246dcb0 0x55ef723dd640 secure :-1 s=READY pgs=1042 cs=0 l=1 rev1=1 crypto rx=0x7fded0006f90 tx=0x7fded000d5f0 comp rx=0 tx=0).stop 2024-08-03T20:54:43.738 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.736+0000 7fded8e00640 1 -- 172.21.15.32:0/3200991834 reap_dead start 2024-08-03T20:54:43.738 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.736+0000 7fdedfd1df40 1 -- 172.21.15.32:0/3200991834 shutdown_connections 2024-08-03T20:54:43.738 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.736+0000 7fdedfd1df40 1 -- 172.21.15.32:0/3200991834 >> 172.21.15.32:0/3200991834 conn(0x55ef722b9ac0 msgr2=0x55ef72468610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:54:43.738 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.736+0000 7fdedfd1df40 1 -- 172.21.15.32:0/3200991834 shutdown_connections 2024-08-03T20:54:43.738 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.736+0000 7fdedfd1df40 1 -- 172.21.15.32:0/3200991834 wait complete. 2024-08-03T20:54:43.750 DEBUG:teuthology.orchestra.run.smithi032:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata stat 600.00000000 2024-08-03T20:54:43.893 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.892+0000 7f8500630f40 1 Processor -- start 2024-08-03T20:54:43.893 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.893+0000 7f8500630f40 1 -- start start 2024-08-03T20:54:43.896 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.894+0000 7f8500630f40 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55d9a21b2360 0x55d9a20fa5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:43.896 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.894+0000 7f8500630f40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55d9a21b3ea0 0x55d9a21aa480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:43.896 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.894+0000 7f8500630f40 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55d9a21cfcb0 0x55d9a21d0810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:43.896 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.894+0000 7f8500630f40 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55d9a2049e30 con 0x55d9a21cfcb0 2024-08-03T20:54:43.896 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.894+0000 7f8500630f40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x55d9a20a7bf0 con 0x55d9a21b3ea0 2024-08-03T20:54:43.896 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.894+0000 7f8500630f40 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x55d9a21b31b0 con 0x55d9a21b2360 2024-08-03T20:54:43.896 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.894+0000 7f84fa200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55d9a21cfcb0 0x55d9a21d0810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:43.896 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.894+0000 7f84fa200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55d9a21cfcb0 0x55d9a21d0810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:37234/0 (socket says 172.21.15.32:37234) 2024-08-03T20:54:43.896 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.894+0000 7f84fa200640 1 -- 172.21.15.32:0/337936493 learned_addr learned my addr 172.21.15.32:0/337936493 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:54:43.896 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.895+0000 7f84f9800640 1 --2- 172.21.15.32:0/337936493 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55d9a21b2360 0x55d9a20fa5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:43.896 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.895+0000 7f84f8e00640 1 --2- 172.21.15.32:0/337936493 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55d9a21b3ea0 0x55d9a21aa480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:43.896 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.895+0000 7f84fa200640 1 -- 172.21.15.32:0/337936493 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55d9a21b2360 msgr2=0x55d9a20fa5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:54:43.897 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.895+0000 7f84fa200640 1 --2- 172.21.15.32:0/337936493 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55d9a21b2360 0x55d9a20fa5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:43.897 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.895+0000 7f84fa200640 1 -- 172.21.15.32:0/337936493 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55d9a21b3ea0 msgr2=0x55d9a21aa480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:54:43.897 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.895+0000 7f84fa200640 1 --2- 172.21.15.32:0/337936493 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55d9a21b3ea0 0x55d9a21aa480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:43.897 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.895+0000 7f84fa200640 1 -- 172.21.15.32:0/337936493 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d9a21c8890 con 0x55d9a21cfcb0 2024-08-03T20:54:43.897 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.895+0000 7f84f9800640 1 --2- 172.21.15.32:0/337936493 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55d9a21b2360 0x55d9a20fa5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-03T20:54:43.897 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.895+0000 7f84f8e00640 1 --2- 172.21.15.32:0/337936493 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55d9a21b3ea0 0x55d9a21aa480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-03T20:54:43.897 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.895+0000 7f84fa200640 1 --2- 172.21.15.32:0/337936493 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55d9a21cfcb0 0x55d9a21d0810 secure :-1 s=READY pgs=1045 cs=0 l=1 rev1=1 crypto rx=0x7f84f4006f60 tx=0x7f84f4006f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ea9cb032ff78ca88 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:54:43.897 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.895+0000 7f84f3400640 1 -- 172.21.15.32:0/337936493 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84f4015020 con 0x55d9a21cfcb0 2024-08-03T20:54:43.897 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.895+0000 7f84f3400640 1 -- 172.21.15.32:0/337936493 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f84f400f800 con 0x55d9a21cfcb0 2024-08-03T20:54:43.897 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.895+0000 7f84f3400640 1 -- 172.21.15.32:0/337936493 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84f400fc90 con 0x55d9a21cfcb0 2024-08-03T20:54:43.898 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.896+0000 7f8500630f40 1 -- 172.21.15.32:0/337936493 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55d9a21cfcb0 msgr2=0x55d9a21d0810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:43.898 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.896+0000 7f8500630f40 1 --2- 172.21.15.32:0/337936493 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55d9a21cfcb0 0x55d9a21d0810 secure :-1 s=READY pgs=1045 cs=0 l=1 rev1=1 crypto rx=0x7f84f4006f60 tx=0x7f84f4006f90 comp rx=0 tx=0).stop 2024-08-03T20:54:43.898 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.896+0000 7f8500630f40 1 -- 172.21.15.32:0/337936493 shutdown_connections 2024-08-03T20:54:43.898 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.896+0000 7f8500630f40 1 --2- 172.21.15.32:0/337936493 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55d9a21b3ea0 0x55d9a21aa480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:43.899 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.896+0000 7f8500630f40 1 --2- 172.21.15.32:0/337936493 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55d9a21cfcb0 0x55d9a21d0810 unknown :-1 s=CLOSED pgs=1045 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:43.899 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.896+0000 7f8500630f40 1 --2- 172.21.15.32:0/337936493 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55d9a21b2360 0x55d9a20fa5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:43.899 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.896+0000 7f8500630f40 1 -- 172.21.15.32:0/337936493 >> 172.21.15.32:0/337936493 conn(0x55d9a201bac0 msgr2=0x55d9a21cbdd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:54:43.899 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.896+0000 7f8500630f40 1 -- 172.21.15.32:0/337936493 shutdown_connections 2024-08-03T20:54:43.899 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.896+0000 7f8500630f40 1 -- 172.21.15.32:0/337936493 wait complete. 2024-08-03T20:54:43.899 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.897+0000 7f8500630f40 1 Processor -- start 2024-08-03T20:54:43.899 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.897+0000 7f8500630f40 1 -- start start 2024-08-03T20:54:43.901 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.899+0000 7f8500630f40 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55d9a21b2360 0x55d9a2144570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:43.901 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.899+0000 7f8500630f40 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55d9a21b3ea0 0x55d9a20fb010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:43.901 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.899+0000 7f8500630f40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55d9a21cfcb0 0x55d9a20fb550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:43.901 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.899+0000 7f8500630f40 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55d9a21c8b00 con 0x55d9a21b3ea0 2024-08-03T20:54:43.902 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.899+0000 7f8500630f40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x55d9a2144c70 con 0x55d9a21cfcb0 2024-08-03T20:54:43.902 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.899+0000 7f8500630f40 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x55d9a2143ca0 con 0x55d9a21b2360 2024-08-03T20:54:43.902 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.899+0000 7f84f8e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55d9a21b3ea0 0x55d9a20fb010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:43.902 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.899+0000 7f84f8e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55d9a21b3ea0 0x55d9a20fb010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:37240/0 (socket says 172.21.15.32:37240) 2024-08-03T20:54:43.902 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.899+0000 7f84f8e00640 1 -- 172.21.15.32:0/3976829209 learned_addr learned my addr 172.21.15.32:0/3976829209 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:54:43.902 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.899+0000 7f84fa200640 1 --2- 172.21.15.32:0/3976829209 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55d9a21cfcb0 0x55d9a20fb550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:43.902 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.899+0000 7f84f9800640 1 --2- 172.21.15.32:0/3976829209 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55d9a21b2360 0x55d9a2144570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:43.902 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.900+0000 7f84f8e00640 1 -- 172.21.15.32:0/3976829209 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55d9a21b2360 msgr2=0x55d9a2144570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:43.902 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.900+0000 7f84f8e00640 1 --2- 172.21.15.32:0/3976829209 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55d9a21b2360 0x55d9a2144570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:43.902 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.900+0000 7f84f8e00640 1 -- 172.21.15.32:0/3976829209 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55d9a21cfcb0 msgr2=0x55d9a20fb550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:43.902 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.900+0000 7f84f8e00640 1 --2- 172.21.15.32:0/3976829209 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55d9a21cfcb0 0x55d9a20fb550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:43.902 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.900+0000 7f84f8e00640 1 -- 172.21.15.32:0/3976829209 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d9a226b760 con 0x55d9a21b3ea0 2024-08-03T20:54:43.903 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.900+0000 7f84f8e00640 1 --2- 172.21.15.32:0/3976829209 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55d9a21b3ea0 0x55d9a20fb010 secure :-1 s=READY pgs=1046 cs=0 l=1 rev1=1 crypto rx=0x7f84e8000c00 tx=0x7f84e8004690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:54:43.903 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.900+0000 7f84f2000640 1 -- 172.21.15.32:0/3976829209 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84e8066070 con 0x55d9a21b3ea0 2024-08-03T20:54:43.903 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.900+0000 7f84f2000640 1 -- 172.21.15.32:0/3976829209 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f84e8056560 con 0x55d9a21b3ea0 2024-08-03T20:54:43.903 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.900+0000 7f84f2000640 1 -- 172.21.15.32:0/3976829209 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84e8056970 con 0x55d9a21b3ea0 2024-08-03T20:54:43.903 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.900+0000 7f8500630f40 1 -- 172.21.15.32:0/3976829209 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x55d9a226bc60 con 0x55d9a21b3ea0 2024-08-03T20:54:43.903 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.900+0000 7f8500630f40 1 -- 172.21.15.32:0/3976829209 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d9a226c220 con 0x55d9a21b3ea0 2024-08-03T20:54:43.903 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.903+0000 7f84f2000640 1 -- 172.21.15.32:0/3976829209 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f84e8069070 con 0x55d9a21b3ea0 2024-08-03T20:54:43.904 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.903+0000 7f84f2000640 1 --2- 172.21.15.32:0/3976829209 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f84dc075a20 0x7f84dc077f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:43.904 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.904+0000 7f84f2000640 1 -- 172.21.15.32:0/3976829209 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 5795+0+0 (secure 0 0 0) 0x7f84e80daf40 con 0x55d9a21b3ea0 2024-08-03T20:54:43.905 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.904+0000 7f84f9800640 1 --2- 172.21.15.32:0/3976829209 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f84dc075a20 0x7f84dc077f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:43.905 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.904+0000 7f8500630f40 1 --2- 172.21.15.32:0/3976829209 >> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] conn(0x55d9a21435a0 0x55d9a225e480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:43.905 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.904+0000 7f8500630f40 1 -- 172.21.15.32:0/3976829209 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:1 12.10 12:0d82a743:::600.00000000:head [stat] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84) -- 0x55d9a21ca940 con 0x55d9a21435a0 2024-08-03T20:54:43.905 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.904+0000 7f84fa200640 1 --2- 172.21.15.32:0/3976829209 >> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] conn(0x55d9a21435a0 0x55d9a225e480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:43.905 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.905+0000 7f84f9800640 1 --2- 172.21.15.32:0/3976829209 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f84dc075a20 0x7f84dc077f30 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x55d9a21d2780 tx=0x7f84ec004880 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:54:43.906 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.905+0000 7f84fa200640 1 --2- 172.21.15.32:0/3976829209 >> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] conn(0x55d9a21435a0 0x55d9a225e480 crc :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:54:43.906 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.905+0000 7f84fa200640 1 -- 172.21.15.32:0/3976829209 <== osd.0 v2:172.21.15.32:6816/3066156116 1 ==== osd_op_reply(1 600.00000000 [stat out=16b] v0'0 uv5 ondisk = 0) ==== 156+0+16 (crc 0 0 0) 0x7f84f4017040 con 0x55d9a21435a0 2024-08-03T20:54:43.906 INFO:teuthology.orchestra.run.smithi032.stdout:cephfs_metadata/600.00000000 mtime 2024-08-03T20:54:27.000000+0000, size 0 2024-08-03T20:54:43.907 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.905+0000 7f8500630f40 1 -- 172.21.15.32:0/3976829209 >> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] conn(0x55d9a21435a0 msgr2=0x55d9a225e480 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:43.907 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.905+0000 7f8500630f40 1 --2- 172.21.15.32:0/3976829209 >> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] conn(0x55d9a21435a0 0x55d9a225e480 crc :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:43.907 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.905+0000 7f8500630f40 1 -- 172.21.15.32:0/3976829209 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f84dc075a20 msgr2=0x7f84dc077f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:43.907 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.905+0000 7f8500630f40 1 --2- 172.21.15.32:0/3976829209 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f84dc075a20 0x7f84dc077f30 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x55d9a21d2780 tx=0x7f84ec004880 comp rx=0 tx=0).stop 2024-08-03T20:54:43.907 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.905+0000 7f8500630f40 1 -- 172.21.15.32:0/3976829209 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55d9a21b3ea0 msgr2=0x55d9a20fb010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:43.907 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.905+0000 7f8500630f40 1 --2- 172.21.15.32:0/3976829209 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55d9a21b3ea0 0x55d9a20fb010 secure :-1 s=READY pgs=1046 cs=0 l=1 rev1=1 crypto rx=0x7f84e8000c00 tx=0x7f84e8004690 comp rx=0 tx=0).stop 2024-08-03T20:54:43.908 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.905+0000 7f84fa200640 1 -- 172.21.15.32:0/3976829209 reap_dead start 2024-08-03T20:54:43.908 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.905+0000 7f8500630f40 1 -- 172.21.15.32:0/3976829209 shutdown_connections 2024-08-03T20:54:43.908 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.905+0000 7f8500630f40 1 -- 172.21.15.32:0/3976829209 >> 172.21.15.32:0/3976829209 conn(0x55d9a201bac0 msgr2=0x55d9a20fa530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:54:43.908 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.905+0000 7f8500630f40 1 -- 172.21.15.32:0/3976829209 shutdown_connections 2024-08-03T20:54:43.908 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:43.906+0000 7f8500630f40 1 -- 172.21.15.32:0/3976829209 wait complete. 2024-08-03T20:54:43.917 DEBUG:teuthology.orchestra.run.smithi032:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata stat 1.00000000.inode 2024-08-03T20:54:44.059 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.058+0000 7f54161b5f40 1 Processor -- start 2024-08-03T20:54:44.059 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.059+0000 7f54161b5f40 1 -- start start 2024-08-03T20:54:44.061 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.060+0000 7f54161b5f40 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55591134e360 0x5559113696a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:44.061 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.060+0000 7f54161b5f40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55591134fea0 0x55591136ce90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:44.061 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.060+0000 7f54161b5f40 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x555911365330 0x55591136d3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:44.061 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.060+0000 7f54161b5f40 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x5559111e5e30 con 0x55591134e360 2024-08-03T20:54:44.061 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.060+0000 7f54161b5f40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x555911243bf0 con 0x55591134fea0 2024-08-03T20:54:44.061 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.060+0000 7f54161b5f40 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x55591134f440 con 0x555911365330 2024-08-03T20:54:44.061 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.060+0000 7f540f400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55591134e360 0x5559113696a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:44.062 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.060+0000 7f540f400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55591134e360 0x5559113696a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:37260/0 (socket says 172.21.15.32:37260) 2024-08-03T20:54:44.062 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.060+0000 7f540f400640 1 -- 172.21.15.32:0/489025732 learned_addr learned my addr 172.21.15.32:0/489025732 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:54:44.062 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.060+0000 7f540ea00640 1 --2- 172.21.15.32:0/489025732 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55591134fea0 0x55591136ce90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:44.062 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.060+0000 7f540fe00640 1 --2- 172.21.15.32:0/489025732 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x555911365330 0x55591136d3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:44.062 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.061+0000 7f540f400640 1 -- 172.21.15.32:0/489025732 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x555911365330 msgr2=0x55591136d3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:54:44.062 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.061+0000 7f540f400640 1 --2- 172.21.15.32:0/489025732 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x555911365330 0x55591136d3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:44.062 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.061+0000 7f540f400640 1 -- 172.21.15.32:0/489025732 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55591134fea0 msgr2=0x55591136ce90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:54:44.062 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.061+0000 7f540f400640 1 --2- 172.21.15.32:0/489025732 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55591134fea0 0x55591136ce90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:44.062 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.061+0000 7f540f400640 1 -- 172.21.15.32:0/489025732 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555911368d20 con 0x55591134e360 2024-08-03T20:54:44.064 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.061+0000 7f540f400640 1 --2- 172.21.15.32:0/489025732 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55591134e360 0x5559113696a0 secure :-1 s=READY pgs=1048 cs=0 l=1 rev1=1 crypto rx=0x7f5400001d20 tx=0x7f540000b5b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9682ca56ef5b6814 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:54:44.064 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.061+0000 7f540d600640 1 -- 172.21.15.32:0/489025732 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f540000bd80 con 0x55591134e360 2024-08-03T20:54:44.064 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.061+0000 7f540d600640 1 -- 172.21.15.32:0/489025732 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f540000f400 con 0x55591134e360 2024-08-03T20:54:44.064 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.061+0000 7f540d600640 1 -- 172.21.15.32:0/489025732 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f540000f850 con 0x55591134e360 2024-08-03T20:54:44.064 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.062+0000 7f54161b5f40 1 -- 172.21.15.32:0/489025732 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55591134e360 msgr2=0x5559113696a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:44.064 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.062+0000 7f54161b5f40 1 --2- 172.21.15.32:0/489025732 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55591134e360 0x5559113696a0 secure :-1 s=READY pgs=1048 cs=0 l=1 rev1=1 crypto rx=0x7f5400001d20 tx=0x7f540000b5b0 comp rx=0 tx=0).stop 2024-08-03T20:54:44.064 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.062+0000 7f54161b5f40 1 -- 172.21.15.32:0/489025732 shutdown_connections 2024-08-03T20:54:44.064 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.062+0000 7f54161b5f40 1 --2- 172.21.15.32:0/489025732 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55591134fea0 0x55591136ce90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:44.064 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.062+0000 7f54161b5f40 1 --2- 172.21.15.32:0/489025732 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x555911365330 0x55591136d3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:44.064 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.062+0000 7f54161b5f40 1 --2- 172.21.15.32:0/489025732 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55591134e360 0x5559113696a0 unknown :-1 s=CLOSED pgs=1048 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:44.064 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.062+0000 7f54161b5f40 1 -- 172.21.15.32:0/489025732 >> 172.21.15.32:0/489025732 conn(0x5559111b7ac0 msgr2=0x555911368040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:54:44.065 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.062+0000 7f54161b5f40 1 -- 172.21.15.32:0/489025732 shutdown_connections 2024-08-03T20:54:44.065 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.062+0000 7f54161b5f40 1 -- 172.21.15.32:0/489025732 wait complete. 2024-08-03T20:54:44.065 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.063+0000 7f54161b5f40 1 Processor -- start 2024-08-03T20:54:44.065 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.063+0000 7f54161b5f40 1 -- start start 2024-08-03T20:54:44.067 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.065+0000 7f54161b5f40 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55591134e360 0x555911345630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:44.067 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.065+0000 7f54161b5f40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55591134fea0 0x5559113670c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:44.067 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.065+0000 7f54161b5f40 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555911365330 0x555911367600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:44.067 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.065+0000 7f54161b5f40 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55591136dc70 con 0x555911365330 2024-08-03T20:54:44.067 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.065+0000 7f54161b5f40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x55591136df50 con 0x55591134fea0 2024-08-03T20:54:44.067 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.065+0000 7f54161b5f40 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x555911369090 con 0x55591134e360 2024-08-03T20:54:44.067 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.065+0000 7f540fe00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555911365330 0x555911367600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:44.067 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.065+0000 7f540fe00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555911365330 0x555911367600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:37276/0 (socket says 172.21.15.32:37276) 2024-08-03T20:54:44.067 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.065+0000 7f540fe00640 1 -- 172.21.15.32:0/2172169747 learned_addr learned my addr 172.21.15.32:0/2172169747 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:54:44.068 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.065+0000 7f540f400640 1 --2- 172.21.15.32:0/2172169747 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55591134e360 0x555911345630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:44.068 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.065+0000 7f540ea00640 1 --2- 172.21.15.32:0/2172169747 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55591134fea0 0x5559113670c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:44.068 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.066+0000 7f540fe00640 1 -- 172.21.15.32:0/2172169747 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55591134e360 msgr2=0x555911345630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:44.068 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.066+0000 7f540fe00640 1 --2- 172.21.15.32:0/2172169747 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55591134e360 0x555911345630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:44.068 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.066+0000 7f540fe00640 1 -- 172.21.15.32:0/2172169747 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55591134fea0 msgr2=0x5559113670c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:44.068 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.066+0000 7f540fe00640 1 --2- 172.21.15.32:0/2172169747 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55591134fea0 0x5559113670c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:44.068 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.066+0000 7f540fe00640 1 -- 172.21.15.32:0/2172169747 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555911295fd0 con 0x555911365330 2024-08-03T20:54:44.068 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.066+0000 7f540ea00640 1 --2- 172.21.15.32:0/2172169747 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55591134fea0 0x5559113670c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-03T20:54:44.068 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.066+0000 7f540fe00640 1 --2- 172.21.15.32:0/2172169747 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555911365330 0x555911367600 secure :-1 s=READY pgs=1049 cs=0 l=1 rev1=1 crypto rx=0x7f5408057040 tx=0x7f540804d7e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:54:44.068 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.066+0000 7f5407e00640 1 -- 172.21.15.32:0/2172169747 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5408065030 con 0x555911365330 2024-08-03T20:54:44.068 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.066+0000 7f5407e00640 1 -- 172.21.15.32:0/2172169747 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f5408004110 con 0x555911365330 2024-08-03T20:54:44.068 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.066+0000 7f5407e00640 1 -- 172.21.15.32:0/2172169747 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5408061510 con 0x555911365330 2024-08-03T20:54:44.068 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.066+0000 7f54161b5f40 1 -- 172.21.15.32:0/2172169747 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x55591136d170 con 0x555911365330 2024-08-03T20:54:44.069 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.066+0000 7f54161b5f40 1 -- 172.21.15.32:0/2172169747 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5559112969d0 con 0x555911365330 2024-08-03T20:54:44.069 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.068+0000 7f5407e00640 1 -- 172.21.15.32:0/2172169747 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f540805e050 con 0x555911365330 2024-08-03T20:54:44.070 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.069+0000 7f5407e00640 1 --2- 172.21.15.32:0/2172169747 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f53f00759f0 0x7f53f0077f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:44.070 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.069+0000 7f5407e00640 1 -- 172.21.15.32:0/2172169747 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 5795+0+0 (secure 0 0 0) 0x7f54080db520 con 0x555911365330 2024-08-03T20:54:44.070 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.070+0000 7f54161b5f40 1 --2- 172.21.15.32:0/2172169747 >> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] conn(0x555911408060 0x555911408470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:44.071 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.070+0000 7f54161b5f40 1 -- 172.21.15.32:0/2172169747 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:1 12.14 12:287034c4:::1.00000000.inode:head [stat] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84) -- 0x55591136a9c0 con 0x555911408060 2024-08-03T20:54:44.071 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.070+0000 7f540f400640 1 --2- 172.21.15.32:0/2172169747 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f53f00759f0 0x7f53f0077f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:44.071 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.070+0000 7f540ea00640 1 --2- 172.21.15.32:0/2172169747 >> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] conn(0x555911408060 0x555911408470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:44.071 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.071+0000 7f540f400640 1 --2- 172.21.15.32:0/2172169747 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f53f00759f0 0x7f53f0077f00 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f54000012b0 tx=0x7f5400000ed0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:54:44.071 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.071+0000 7f540ea00640 1 --2- 172.21.15.32:0/2172169747 >> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] conn(0x555911408060 0x555911408470 crc :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:54:44.072 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.071+0000 7f540ea00640 1 -- 172.21.15.32:0/2172169747 <== osd.5 v2:172.21.15.118:6810/2724891770 1 ==== osd_op_reply(1 1.00000000.inode [stat out=16b] v0'0 uv15 ondisk = 0) ==== 160+0+16 (crc 0 0 0) 0x7f53fc00b820 con 0x555911408060 2024-08-03T20:54:44.072 INFO:teuthology.orchestra.run.smithi032.stdout:cephfs_metadata/1.00000000.inode mtime 2024-08-03T20:54:37.000000+0000, size 582 2024-08-03T20:54:44.073 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.071+0000 7f54161b5f40 1 -- 172.21.15.32:0/2172169747 >> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] conn(0x555911408060 msgr2=0x555911408470 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:44.073 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.071+0000 7f54161b5f40 1 --2- 172.21.15.32:0/2172169747 >> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] conn(0x555911408060 0x555911408470 crc :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:44.073 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.071+0000 7f54161b5f40 1 -- 172.21.15.32:0/2172169747 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f53f00759f0 msgr2=0x7f53f0077f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:44.073 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.071+0000 7f54161b5f40 1 --2- 172.21.15.32:0/2172169747 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f53f00759f0 0x7f53f0077f00 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f54000012b0 tx=0x7f5400000ed0 comp rx=0 tx=0).stop 2024-08-03T20:54:44.073 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.071+0000 7f54161b5f40 1 -- 172.21.15.32:0/2172169747 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555911365330 msgr2=0x555911367600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:44.073 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.071+0000 7f54161b5f40 1 --2- 172.21.15.32:0/2172169747 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555911365330 0x555911367600 secure :-1 s=READY pgs=1049 cs=0 l=1 rev1=1 crypto rx=0x7f5408057040 tx=0x7f540804d7e0 comp rx=0 tx=0).stop 2024-08-03T20:54:44.073 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.072+0000 7f540fe00640 1 -- 172.21.15.32:0/2172169747 reap_dead start 2024-08-03T20:54:44.073 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.072+0000 7f54161b5f40 1 -- 172.21.15.32:0/2172169747 shutdown_connections 2024-08-03T20:54:44.074 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.072+0000 7f54161b5f40 1 -- 172.21.15.32:0/2172169747 >> 172.21.15.32:0/2172169747 conn(0x5559111b7ac0 msgr2=0x555911367e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:54:44.074 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.072+0000 7f54161b5f40 1 -- 172.21.15.32:0/2172169747 shutdown_connections 2024-08-03T20:54:44.074 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.072+0000 7f54161b5f40 1 -- 172.21.15.32:0/2172169747 wait complete. 2024-08-03T20:54:44.087 DEBUG:teuthology.orchestra.run.smithi032:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata stat 200.00000000 2024-08-03T20:54:44.228 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.228+0000 7f6b75f28f40 1 Processor -- start 2024-08-03T20:54:44.228 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.228+0000 7f6b75f28f40 1 -- start start 2024-08-03T20:54:44.230 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.229+0000 7f6b75f28f40 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55a41a84d360 0x55a41a869330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:44.231 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.229+0000 7f6b75f28f40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55a41a84eea0 0x55a41a7d5670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:44.231 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.229+0000 7f6b75f28f40 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a41a864330 0x55a41a7d5bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:44.231 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.229+0000 7f6b75f28f40 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55a41a6e4e30 con 0x55a41a864330 2024-08-03T20:54:44.231 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.229+0000 7f6b75f28f40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x55a41a742bf0 con 0x55a41a84eea0 2024-08-03T20:54:44.231 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.229+0000 7f6b75f28f40 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x55a41a84e440 con 0x55a41a84d360 2024-08-03T20:54:44.231 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.229+0000 7f6b6fa00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a41a864330 0x55a41a7d5bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:44.231 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.229+0000 7f6b6fa00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a41a864330 0x55a41a7d5bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:37292/0 (socket says 172.21.15.32:37292) 2024-08-03T20:54:44.231 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.229+0000 7f6b6fa00640 1 -- 172.21.15.32:0/1411174374 learned_addr learned my addr 172.21.15.32:0/1411174374 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:54:44.231 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.230+0000 7f6b6fa00640 1 -- 172.21.15.32:0/1411174374 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55a41a84d360 msgr2=0x55a41a869330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:54:44.231 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.230+0000 7f6b6fa00640 1 --2- 172.21.15.32:0/1411174374 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55a41a84d360 0x55a41a869330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:44.231 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.230+0000 7f6b6fa00640 1 -- 172.21.15.32:0/1411174374 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55a41a84eea0 msgr2=0x55a41a7d5670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:54:44.231 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.230+0000 7f6b6fa00640 1 --2- 172.21.15.32:0/1411174374 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55a41a84eea0 0x55a41a7d5670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:44.232 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.230+0000 7f6b6fa00640 1 -- 172.21.15.32:0/1411174374 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a41a7d5210 con 0x55a41a864330 2024-08-03T20:54:44.232 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.230+0000 7f6b6fa00640 1 --2- 172.21.15.32:0/1411174374 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a41a864330 0x55a41a7d5bb0 secure :-1 s=READY pgs=1051 cs=0 l=1 rev1=1 crypto rx=0x7f6b6800d790 tx=0x7f6b6800dc50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ec5b8ab42a73f89c server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:54:44.232 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.230+0000 7f6b6d200640 1 -- 172.21.15.32:0/1411174374 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b6800dea0 con 0x55a41a864330 2024-08-03T20:54:44.232 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.230+0000 7f6b6d200640 1 -- 172.21.15.32:0/1411174374 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f6b6800bbe0 con 0x55a41a864330 2024-08-03T20:54:44.232 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.230+0000 7f6b6d200640 1 -- 172.21.15.32:0/1411174374 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b6800f430 con 0x55a41a864330 2024-08-03T20:54:44.233 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.231+0000 7f6b75f28f40 1 -- 172.21.15.32:0/1411174374 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a41a864330 msgr2=0x55a41a7d5bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:44.233 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.231+0000 7f6b75f28f40 1 --2- 172.21.15.32:0/1411174374 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a41a864330 0x55a41a7d5bb0 secure :-1 s=READY pgs=1051 cs=0 l=1 rev1=1 crypto rx=0x7f6b6800d790 tx=0x7f6b6800dc50 comp rx=0 tx=0).stop 2024-08-03T20:54:44.233 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.231+0000 7f6b75f28f40 1 -- 172.21.15.32:0/1411174374 shutdown_connections 2024-08-03T20:54:44.233 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.231+0000 7f6b75f28f40 1 --2- 172.21.15.32:0/1411174374 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55a41a84eea0 0x55a41a7d5670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:44.233 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.231+0000 7f6b75f28f40 1 --2- 172.21.15.32:0/1411174374 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a41a864330 0x55a41a7d5bb0 unknown :-1 s=CLOSED pgs=1051 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:44.233 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.231+0000 7f6b75f28f40 1 --2- 172.21.15.32:0/1411174374 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55a41a84d360 0x55a41a869330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:44.233 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.231+0000 7f6b75f28f40 1 -- 172.21.15.32:0/1411174374 >> 172.21.15.32:0/1411174374 conn(0x55a41a6b6ac0 msgr2=0x55a41a866110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:54:44.233 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.231+0000 7f6b75f28f40 1 -- 172.21.15.32:0/1411174374 shutdown_connections 2024-08-03T20:54:44.233 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.231+0000 7f6b75f28f40 1 -- 172.21.15.32:0/1411174374 wait complete. 2024-08-03T20:54:44.233 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.232+0000 7f6b75f28f40 1 Processor -- start 2024-08-03T20:54:44.234 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.232+0000 7f6b75f28f40 1 -- start start 2024-08-03T20:54:44.235 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.234+0000 7f6b75f28f40 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a41a84d360 0x55a41a866260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:44.235 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.234+0000 7f6b75f28f40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55a41a84eea0 0x55a41a794da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:44.235 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.234+0000 7f6b75f28f40 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55a41a864330 0x55a41a7952e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:44.235 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.234+0000 7f6b75f28f40 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55a41a8699d0 con 0x55a41a84d360 2024-08-03T20:54:44.235 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.234+0000 7f6b75f28f40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x55a41a84ea70 con 0x55a41a84eea0 2024-08-03T20:54:44.235 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.234+0000 7f6b75f28f40 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x55a41a795ba0 con 0x55a41a864330 2024-08-03T20:54:44.235 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.234+0000 7f6b6f000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a41a84d360 0x55a41a866260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:44.235 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.234+0000 7f6b6f000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a41a84d360 0x55a41a866260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:37298/0 (socket says 172.21.15.32:37298) 2024-08-03T20:54:44.235 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.234+0000 7f6b6f000640 1 -- 172.21.15.32:0/2160395164 learned_addr learned my addr 172.21.15.32:0/2160395164 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:54:44.235 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.234+0000 7f6b6e600640 1 --2- 172.21.15.32:0/2160395164 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55a41a84eea0 0x55a41a794da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:44.236 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.234+0000 7f6b6fa00640 1 --2- 172.21.15.32:0/2160395164 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55a41a864330 0x55a41a7952e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:44.236 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.234+0000 7f6b6f000640 1 -- 172.21.15.32:0/2160395164 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55a41a864330 msgr2=0x55a41a7952e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:44.236 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.234+0000 7f6b6f000640 1 --2- 172.21.15.32:0/2160395164 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55a41a864330 0x55a41a7952e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:44.236 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.234+0000 7f6b6f000640 1 -- 172.21.15.32:0/2160395164 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55a41a84eea0 msgr2=0x55a41a794da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:44.236 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.234+0000 7f6b6f000640 1 --2- 172.21.15.32:0/2160395164 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55a41a84eea0 0x55a41a794da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:44.236 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.235+0000 7f6b6f000640 1 -- 172.21.15.32:0/2160395164 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a41a7d8310 con 0x55a41a84d360 2024-08-03T20:54:44.236 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.235+0000 7f6b6e600640 1 --2- 172.21.15.32:0/2160395164 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55a41a84eea0 0x55a41a794da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-03T20:54:44.237 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.235+0000 7f6b6f000640 1 --2- 172.21.15.32:0/2160395164 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a41a84d360 0x55a41a866260 secure :-1 s=READY pgs=1052 cs=0 l=1 rev1=1 crypto rx=0x7f6b6000bd60 tx=0x7f6b600095f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:54:44.237 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.235+0000 7f6b67400640 1 -- 172.21.15.32:0/2160395164 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b6000e620 con 0x55a41a84d360 2024-08-03T20:54:44.237 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.235+0000 7f6b67400640 1 -- 172.21.15.32:0/2160395164 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f6b60006050 con 0x55a41a84d360 2024-08-03T20:54:44.237 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.235+0000 7f6b67400640 1 -- 172.21.15.32:0/2160395164 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b6000ea60 con 0x55a41a84d360 2024-08-03T20:54:44.237 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.235+0000 7f6b75f28f40 1 -- 172.21.15.32:0/2160395164 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x55a41a86b460 con 0x55a41a84d360 2024-08-03T20:54:44.237 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.235+0000 7f6b75f28f40 1 -- 172.21.15.32:0/2160395164 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a41a7d8030 con 0x55a41a84d360 2024-08-03T20:54:44.238 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.237+0000 7f6b67400640 1 -- 172.21.15.32:0/2160395164 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f6b6000ec00 con 0x55a41a84d360 2024-08-03T20:54:44.239 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.238+0000 7f6b67400640 1 --2- 172.21.15.32:0/2160395164 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f6b500759f0 0x7f6b50077f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:44.239 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.239+0000 7f6b67400640 1 -- 172.21.15.32:0/2160395164 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 5795+0+0 (secure 0 0 0) 0x7f6b600975f0 con 0x55a41a84d360 2024-08-03T20:54:44.239 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.239+0000 7f6b75f28f40 1 --2- 172.21.15.32:0/2160395164 >> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] conn(0x55a41a869450 0x55a41a7ddde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:44.239 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.239+0000 7f6b75f28f40 1 -- 172.21.15.32:0/2160395164 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:1 12.14 12:292cf221:::200.00000000:head [stat] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84) -- 0x55a41a8667a0 con 0x55a41a869450 2024-08-03T20:54:44.240 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.239+0000 7f6b6e600640 1 --2- 172.21.15.32:0/2160395164 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f6b500759f0 0x7f6b50077f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:44.240 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.239+0000 7f6b6fa00640 1 --2- 172.21.15.32:0/2160395164 >> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] conn(0x55a41a869450 0x55a41a7ddde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:44.240 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.240+0000 7f6b6e600640 1 --2- 172.21.15.32:0/2160395164 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f6b500759f0 0x7f6b50077f00 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f6b5c055000 tx=0x7f6b5c058040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:54:44.240 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.240+0000 7f6b6fa00640 1 --2- 172.21.15.32:0/2160395164 >> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] conn(0x55a41a869450 0x55a41a7ddde0 crc :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:54:44.241 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.240+0000 7f6b6fa00640 1 -- 172.21.15.32:0/2160395164 <== osd.5 v2:172.21.15.118:6810/2724891770 1 ==== osd_op_reply(1 200.00000000 [stat out=16b] v0'0 uv19 ondisk = 0) ==== 156+0+16 (crc 0 0 0) 0x7f6b68008070 con 0x55a41a869450 2024-08-03T20:54:44.241 INFO:teuthology.orchestra.run.smithi032.stdout:cephfs_metadata/200.00000000 mtime 2024-08-03T20:54:37.000000+0000, size 90 2024-08-03T20:54:44.242 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.240+0000 7f6b75f28f40 1 -- 172.21.15.32:0/2160395164 >> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] conn(0x55a41a869450 msgr2=0x55a41a7ddde0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:44.242 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.240+0000 7f6b75f28f40 1 --2- 172.21.15.32:0/2160395164 >> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] conn(0x55a41a869450 0x55a41a7ddde0 crc :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:44.242 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.240+0000 7f6b75f28f40 1 -- 172.21.15.32:0/2160395164 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f6b500759f0 msgr2=0x7f6b50077f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:44.242 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.240+0000 7f6b75f28f40 1 --2- 172.21.15.32:0/2160395164 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f6b500759f0 0x7f6b50077f00 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f6b5c055000 tx=0x7f6b5c058040 comp rx=0 tx=0).stop 2024-08-03T20:54:44.242 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.240+0000 7f6b75f28f40 1 -- 172.21.15.32:0/2160395164 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a41a84d360 msgr2=0x55a41a866260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:44.242 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.240+0000 7f6b75f28f40 1 --2- 172.21.15.32:0/2160395164 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a41a84d360 0x55a41a866260 secure :-1 s=READY pgs=1052 cs=0 l=1 rev1=1 crypto rx=0x7f6b6000bd60 tx=0x7f6b600095f0 comp rx=0 tx=0).stop 2024-08-03T20:54:44.242 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.241+0000 7f6b6fa00640 1 -- 172.21.15.32:0/2160395164 reap_dead start 2024-08-03T20:54:44.242 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.241+0000 7f6b75f28f40 1 -- 172.21.15.32:0/2160395164 shutdown_connections 2024-08-03T20:54:44.242 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.241+0000 7f6b75f28f40 1 -- 172.21.15.32:0/2160395164 >> 172.21.15.32:0/2160395164 conn(0x55a41a6b6ac0 msgr2=0x55a41a7d54c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:54:44.243 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.241+0000 7f6b75f28f40 1 -- 172.21.15.32:0/2160395164 shutdown_connections 2024-08-03T20:54:44.243 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.241+0000 7f6b75f28f40 1 -- 172.21.15.32:0/2160395164 wait complete. 2024-08-03T20:54:44.252 DEBUG:teuthology.orchestra.run.smithi032:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata stat 200.00000001 2024-08-03T20:54:44.395 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.395+0000 7fc66f71df40 1 Processor -- start 2024-08-03T20:54:44.395 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.395+0000 7fc66f71df40 1 -- start start 2024-08-03T20:54:44.397 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.396+0000 7fc66f71df40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x560d6fffb360 0x560d70015260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:44.397 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.396+0000 7fc66f71df40 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x560d6fffcea0 0x560d7001a060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:44.397 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.396+0000 7fc66f71df40 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x560d70012330 0x560d700ab0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:44.397 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.396+0000 7fc66f71df40 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x560d6fe92e30 con 0x560d70012330 2024-08-03T20:54:44.397 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.396+0000 7fc66f71df40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x560d6fef0bf0 con 0x560d6fffb360 2024-08-03T20:54:44.398 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.396+0000 7fc66f71df40 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x560d6fffc440 con 0x560d6fffcea0 2024-08-03T20:54:44.398 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.396+0000 7fc668800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x560d70012330 0x560d700ab0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:44.398 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.396+0000 7fc667e00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x560d6fffb360 0x560d70015260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:44.398 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.396+0000 7fc668800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x560d70012330 0x560d700ab0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:37300/0 (socket says 172.21.15.32:37300) 2024-08-03T20:54:44.398 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.396+0000 7fc667400640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x560d6fffcea0 0x560d7001a060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:44.398 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.396+0000 7fc668800640 1 -- 172.21.15.32:0/830671563 learned_addr learned my addr 172.21.15.32:0/830671563 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:54:44.398 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.397+0000 7fc668800640 1 -- 172.21.15.32:0/830671563 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x560d6fffcea0 msgr2=0x560d7001a060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:54:44.398 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.397+0000 7fc668800640 1 --2- 172.21.15.32:0/830671563 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x560d6fffcea0 0x560d7001a060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:44.398 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.397+0000 7fc668800640 1 -- 172.21.15.32:0/830671563 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x560d6fffb360 msgr2=0x560d70015260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:54:44.398 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.397+0000 7fc668800640 1 --2- 172.21.15.32:0/830671563 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x560d6fffb360 0x560d70015260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:44.398 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.397+0000 7fc668800640 1 -- 172.21.15.32:0/830671563 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560d6fffcc60 con 0x560d70012330 2024-08-03T20:54:44.400 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.397+0000 7fc668800640 1 --2- 172.21.15.32:0/830671563 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x560d70012330 0x560d700ab0c0 secure :-1 s=READY pgs=1053 cs=0 l=1 rev1=1 crypto rx=0x7fc66000d790 tx=0x7fc66000dc50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9b93472755fe330b server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:54:44.400 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.397+0000 7fc666000640 1 -- 172.21.15.32:0/830671563 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc66000dea0 con 0x560d70012330 2024-08-03T20:54:44.400 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.397+0000 7fc666000640 1 -- 172.21.15.32:0/830671563 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fc66000bbe0 con 0x560d70012330 2024-08-03T20:54:44.400 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.397+0000 7fc666000640 1 -- 172.21.15.32:0/830671563 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc66000f430 con 0x560d70012330 2024-08-03T20:54:44.400 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.398+0000 7fc66f71df40 1 -- 172.21.15.32:0/830671563 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x560d70012330 msgr2=0x560d700ab0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:44.400 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.398+0000 7fc66f71df40 1 --2- 172.21.15.32:0/830671563 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x560d70012330 0x560d700ab0c0 secure :-1 s=READY pgs=1053 cs=0 l=1 rev1=1 crypto rx=0x7fc66000d790 tx=0x7fc66000dc50 comp rx=0 tx=0).stop 2024-08-03T20:54:44.400 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.398+0000 7fc66f71df40 1 -- 172.21.15.32:0/830671563 shutdown_connections 2024-08-03T20:54:44.400 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.398+0000 7fc66f71df40 1 --2- 172.21.15.32:0/830671563 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x560d70012330 0x560d700ab0c0 unknown :-1 s=CLOSED pgs=1053 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:44.400 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.398+0000 7fc66f71df40 1 --2- 172.21.15.32:0/830671563 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x560d6fffcea0 0x560d7001a060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:44.400 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.398+0000 7fc66f71df40 1 --2- 172.21.15.32:0/830671563 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x560d6fffb360 0x560d70015260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:44.400 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.398+0000 7fc66f71df40 1 -- 172.21.15.32:0/830671563 >> 172.21.15.32:0/830671563 conn(0x560d6fe64ac0 msgr2=0x560d700128d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:54:44.401 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.398+0000 7fc66f71df40 1 -- 172.21.15.32:0/830671563 shutdown_connections 2024-08-03T20:54:44.401 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.398+0000 7fc66f71df40 1 -- 172.21.15.32:0/830671563 wait complete. 2024-08-03T20:54:44.401 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.399+0000 7fc66f71df40 1 Processor -- start 2024-08-03T20:54:44.401 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.399+0000 7fc66f71df40 1 -- start start 2024-08-03T20:54:44.402 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.401+0000 7fc66f71df40 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x560d6fffb360 0x560d6fff50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:44.402 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.401+0000 7fc66f71df40 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x560d6fffcea0 0x560d6fff5d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:44.402 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.401+0000 7fc66f71df40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x560d70012330 0x560d6fff62a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:44.402 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.401+0000 7fc66f71df40 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x560d70013bf0 con 0x560d6fffb360 2024-08-03T20:54:44.402 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.401+0000 7fc66f71df40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x560d70013a20 con 0x560d70012330 2024-08-03T20:54:44.403 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.401+0000 7fc66f71df40 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x560d6ff888f0 con 0x560d6fffcea0 2024-08-03T20:54:44.403 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.401+0000 7fc667e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x560d6fffb360 0x560d6fff50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:44.403 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.401+0000 7fc667e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x560d6fffb360 0x560d6fff50e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:37302/0 (socket says 172.21.15.32:37302) 2024-08-03T20:54:44.403 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.401+0000 7fc667e00640 1 -- 172.21.15.32:0/468273791 learned_addr learned my addr 172.21.15.32:0/468273791 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:54:44.403 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.401+0000 7fc667400640 1 --2- 172.21.15.32:0/468273791 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x560d6fffcea0 0x560d6fff5d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:44.403 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.401+0000 7fc668800640 1 --2- 172.21.15.32:0/468273791 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x560d70012330 0x560d6fff62a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:44.403 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.402+0000 7fc667e00640 1 -- 172.21.15.32:0/468273791 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x560d6fffcea0 msgr2=0x560d6fff5d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:44.403 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.402+0000 7fc667e00640 1 --2- 172.21.15.32:0/468273791 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x560d6fffcea0 0x560d6fff5d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:44.403 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.402+0000 7fc667e00640 1 -- 172.21.15.32:0/468273791 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x560d70012330 msgr2=0x560d6fff62a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:44.404 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.402+0000 7fc667e00640 1 --2- 172.21.15.32:0/468273791 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x560d70012330 0x560d6fff62a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:44.404 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.402+0000 7fc667e00640 1 -- 172.21.15.32:0/468273791 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560d700a3580 con 0x560d6fffb360 2024-08-03T20:54:44.404 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.402+0000 7fc667e00640 1 --2- 172.21.15.32:0/468273791 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x560d6fffb360 0x560d6fff50e0 secure :-1 s=READY pgs=1054 cs=0 l=1 rev1=1 crypto rx=0x7fc658002cc0 tx=0x7fc65800c620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:54:44.404 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.402+0000 7fc664c00640 1 -- 172.21.15.32:0/468273791 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc658015020 con 0x560d6fffb360 2024-08-03T20:54:44.404 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.402+0000 7fc664c00640 1 -- 172.21.15.32:0/468273791 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fc658011450 con 0x560d6fffb360 2024-08-03T20:54:44.405 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.402+0000 7fc664c00640 1 -- 172.21.15.32:0/468273791 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc658011840 con 0x560d6fffb360 2024-08-03T20:54:44.405 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.402+0000 7fc66f71df40 1 -- 172.21.15.32:0/468273791 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x560d700b55a0 con 0x560d6fffb360 2024-08-03T20:54:44.405 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.402+0000 7fc66f71df40 1 -- 172.21.15.32:0/468273791 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560d6ff88630 con 0x560d6fffb360 2024-08-03T20:54:44.405 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.405+0000 7fc664c00640 1 -- 172.21.15.32:0/468273791 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7fc658018020 con 0x560d6fffb360 2024-08-03T20:54:44.406 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.406+0000 7fc664c00640 1 --2- 172.21.15.32:0/468273791 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fc6480759f0 0x7fc648077f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:44.406 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.406+0000 7fc664c00640 1 -- 172.21.15.32:0/468273791 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 5795+0+0 (secure 0 0 0) 0x7fc6580976c0 con 0x560d6fffb360 2024-08-03T20:54:44.407 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.406+0000 7fc66f71df40 1 --2- 172.21.15.32:0/468273791 >> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] conn(0x560d6ff8a5b0 0x560d700b14b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:44.407 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.406+0000 7fc66f71df40 1 -- 172.21.15.32:0/468273791 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:1 12.14 12:2e2fa760:::200.00000001:head [stat] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84) -- 0x560d700154d0 con 0x560d6ff8a5b0 2024-08-03T20:54:44.407 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.406+0000 7fc667400640 1 --2- 172.21.15.32:0/468273791 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fc6480759f0 0x7fc648077f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:44.407 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.406+0000 7fc668800640 1 --2- 172.21.15.32:0/468273791 >> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] conn(0x560d6ff8a5b0 0x560d700b14b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:44.407 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.407+0000 7fc667400640 1 --2- 172.21.15.32:0/468273791 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fc6480759f0 0x7fc648077f00 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7fc654055000 tx=0x7fc654003020 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:54:44.407 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.407+0000 7fc668800640 1 --2- 172.21.15.32:0/468273791 >> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] conn(0x560d6ff8a5b0 0x560d700b14b0 crc :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:54:44.408 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.408+0000 7fc668800640 1 -- 172.21.15.32:0/468273791 <== osd.5 v2:172.21.15.118:6810/2724891770 1 ==== osd_op_reply(1 200.00000001 [stat out=16b] v0'0 uv16 ondisk = 0) ==== 156+0+16 (crc 0 0 0) 0x7fc660015020 con 0x560d6ff8a5b0 2024-08-03T20:54:44.408 INFO:teuthology.orchestra.run.smithi032.stdout:cephfs_metadata/200.00000001 mtime 2024-08-03T20:54:37.000000+0000, size 18118 2024-08-03T20:54:44.409 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.408+0000 7fc66f71df40 1 -- 172.21.15.32:0/468273791 >> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] conn(0x560d6ff8a5b0 msgr2=0x560d700b14b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:44.409 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.408+0000 7fc66f71df40 1 --2- 172.21.15.32:0/468273791 >> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] conn(0x560d6ff8a5b0 0x560d700b14b0 crc :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:44.409 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.408+0000 7fc66f71df40 1 -- 172.21.15.32:0/468273791 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fc6480759f0 msgr2=0x7fc648077f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:44.409 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.408+0000 7fc66f71df40 1 --2- 172.21.15.32:0/468273791 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fc6480759f0 0x7fc648077f00 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7fc654055000 tx=0x7fc654003020 comp rx=0 tx=0).stop 2024-08-03T20:54:44.409 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.408+0000 7fc66f71df40 1 -- 172.21.15.32:0/468273791 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x560d6fffb360 msgr2=0x560d6fff50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:44.410 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.408+0000 7fc66f71df40 1 --2- 172.21.15.32:0/468273791 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x560d6fffb360 0x560d6fff50e0 secure :-1 s=READY pgs=1054 cs=0 l=1 rev1=1 crypto rx=0x7fc658002cc0 tx=0x7fc65800c620 comp rx=0 tx=0).stop 2024-08-03T20:54:44.410 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.408+0000 7fc668800640 1 -- 172.21.15.32:0/468273791 reap_dead start 2024-08-03T20:54:44.410 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.408+0000 7fc66f71df40 1 -- 172.21.15.32:0/468273791 shutdown_connections 2024-08-03T20:54:44.410 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.408+0000 7fc66f71df40 1 -- 172.21.15.32:0/468273791 >> 172.21.15.32:0/468273791 conn(0x560d6fe64ac0 msgr2=0x560d7001a060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:54:44.410 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.408+0000 7fc66f71df40 1 -- 172.21.15.32:0/468273791 shutdown_connections 2024-08-03T20:54:44.410 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.408+0000 7fc66f71df40 1 -- 172.21.15.32:0/468273791 wait complete. 2024-08-03T20:54:44.419 DEBUG:teuthology.orchestra.run.smithi032:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata stat mds0_openfiles.0 2024-08-03T20:54:44.564 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.563+0000 7f2aac1c7f40 1 Processor -- start 2024-08-03T20:54:44.564 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.563+0000 7f2aac1c7f40 1 -- start start 2024-08-03T20:54:44.566 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.565+0000 7f2aac1c7f40 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55f5b3fc5360 0x55f5b406f150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:44.566 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.565+0000 7f2aac1c7f40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55f5b3fe2cb0 0x55f5b3fc40a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:44.566 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.565+0000 7f2aac1c7f40 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55f5b3fe33f0 0x55f5b3f57680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:44.566 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.565+0000 7f2aac1c7f40 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55f5b3ebabf0 con 0x55f5b3fc5360 2024-08-03T20:54:44.566 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.565+0000 7f2aac1c7f40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x55f5b3ec76b0 con 0x55f5b3fe2cb0 2024-08-03T20:54:44.567 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.565+0000 7f2aac1c7f40 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x55f5b3e5ce30 con 0x55f5b3fe33f0 2024-08-03T20:54:44.567 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.565+0000 7f2aa5400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55f5b3fc5360 0x55f5b406f150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:44.567 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.565+0000 7f2aa5400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55f5b3fc5360 0x55f5b406f150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:37304/0 (socket says 172.21.15.32:37304) 2024-08-03T20:54:44.567 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.565+0000 7f2aa5400640 1 -- 172.21.15.32:0/2190813026 learned_addr learned my addr 172.21.15.32:0/2190813026 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:54:44.567 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.565+0000 7f2aa5e00640 1 --2- 172.21.15.32:0/2190813026 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55f5b3fe33f0 0x55f5b3f57680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:44.567 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.565+0000 7f2aa4a00640 1 --2- 172.21.15.32:0/2190813026 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55f5b3fe2cb0 0x55f5b3fc40a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:44.567 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.566+0000 7f2aa5400640 1 -- 172.21.15.32:0/2190813026 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55f5b3fe33f0 msgr2=0x55f5b3f57680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:54:44.567 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.566+0000 7f2aa5400640 1 --2- 172.21.15.32:0/2190813026 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55f5b3fe33f0 0x55f5b3f57680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:44.567 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.566+0000 7f2aa5400640 1 -- 172.21.15.32:0/2190813026 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55f5b3fe2cb0 msgr2=0x55f5b3fc40a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:54:44.567 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.566+0000 7f2aa5400640 1 --2- 172.21.15.32:0/2190813026 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55f5b3fe2cb0 0x55f5b3fc40a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:44.567 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.566+0000 7f2aa5400640 1 -- 172.21.15.32:0/2190813026 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f5b3f56e90 con 0x55f5b3fc5360 2024-08-03T20:54:44.567 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.566+0000 7f2aa5400640 1 --2- 172.21.15.32:0/2190813026 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55f5b3fc5360 0x55f5b406f150 secure :-1 s=READY pgs=1056 cs=0 l=1 rev1=1 crypto rx=0x7f2a98025e40 tx=0x7f2a98002fb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ba1f948a76269ba4 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:54:44.569 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.566+0000 7f2a9f400640 1 -- 172.21.15.32:0/2190813026 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a9804dd30 con 0x55f5b3fc5360 2024-08-03T20:54:44.569 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.566+0000 7f2a9f400640 1 -- 172.21.15.32:0/2190813026 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f2a98065070 con 0x55f5b3fc5360 2024-08-03T20:54:44.569 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.566+0000 7f2a9f400640 1 -- 172.21.15.32:0/2190813026 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a980566d0 con 0x55f5b3fc5360 2024-08-03T20:54:44.569 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.567+0000 7f2aac1c7f40 1 -- 172.21.15.32:0/2190813026 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55f5b3fc5360 msgr2=0x55f5b406f150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:44.569 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.567+0000 7f2aac1c7f40 1 --2- 172.21.15.32:0/2190813026 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55f5b3fc5360 0x55f5b406f150 secure :-1 s=READY pgs=1056 cs=0 l=1 rev1=1 crypto rx=0x7f2a98025e40 tx=0x7f2a98002fb0 comp rx=0 tx=0).stop 2024-08-03T20:54:44.569 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.567+0000 7f2aac1c7f40 1 -- 172.21.15.32:0/2190813026 shutdown_connections 2024-08-03T20:54:44.569 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.567+0000 7f2aac1c7f40 1 --2- 172.21.15.32:0/2190813026 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55f5b3fe2cb0 0x55f5b3fc40a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:44.569 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.567+0000 7f2aac1c7f40 1 --2- 172.21.15.32:0/2190813026 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55f5b3fe33f0 0x55f5b3f57680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:44.569 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.567+0000 7f2aac1c7f40 1 --2- 172.21.15.32:0/2190813026 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55f5b3fc5360 0x55f5b406f150 unknown :-1 s=CLOSED pgs=1056 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:44.569 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.567+0000 7f2aac1c7f40 1 -- 172.21.15.32:0/2190813026 >> 172.21.15.32:0/2190813026 conn(0x55f5b3e2eac0 msgr2=0x55f5b3fe0d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:54:44.569 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.567+0000 7f2aac1c7f40 1 -- 172.21.15.32:0/2190813026 shutdown_connections 2024-08-03T20:54:44.570 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.567+0000 7f2aac1c7f40 1 -- 172.21.15.32:0/2190813026 wait complete. 2024-08-03T20:54:44.570 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.568+0000 7f2aac1c7f40 1 Processor -- start 2024-08-03T20:54:44.570 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.568+0000 7f2aac1c7f40 1 -- start start 2024-08-03T20:54:44.571 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.570+0000 7f2aac1c7f40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55f5b3fc5360 0x55f5b3f57870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:44.572 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.570+0000 7f2aac1c7f40 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55f5b3fe2cb0 0x55f5b3fc2270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:44.572 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.570+0000 7f2aac1c7f40 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55f5b3fe33f0 0x55f5b3fc27b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:44.572 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.570+0000 7f2aac1c7f40 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55f5b3fc45e0 con 0x55f5b3fe33f0 2024-08-03T20:54:44.572 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.570+0000 7f2aac1c7f40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x55f5b3f56680 con 0x55f5b3fc5360 2024-08-03T20:54:44.572 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.570+0000 7f2aac1c7f40 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x55f5b407c4f0 con 0x55f5b3fe2cb0 2024-08-03T20:54:44.572 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.570+0000 7f2aa5e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55f5b3fe33f0 0x55f5b3fc27b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:44.572 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.570+0000 7f2aa5e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55f5b3fe33f0 0x55f5b3fc27b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:37310/0 (socket says 172.21.15.32:37310) 2024-08-03T20:54:44.572 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.570+0000 7f2aa5e00640 1 -- 172.21.15.32:0/2293828571 learned_addr learned my addr 172.21.15.32:0/2293828571 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:54:44.572 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.570+0000 7f2aa5400640 1 --2- 172.21.15.32:0/2293828571 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55f5b3fc5360 0x55f5b3f57870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:44.572 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.570+0000 7f2aa4a00640 1 --2- 172.21.15.32:0/2293828571 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55f5b3fe2cb0 0x55f5b3fc2270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:44.572 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.570+0000 7f2aa5e00640 1 -- 172.21.15.32:0/2293828571 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55f5b3fe2cb0 msgr2=0x55f5b3fc2270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:44.572 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.570+0000 7f2aa5e00640 1 --2- 172.21.15.32:0/2293828571 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55f5b3fe2cb0 0x55f5b3fc2270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:44.572 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.570+0000 7f2aa5e00640 1 -- 172.21.15.32:0/2293828571 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55f5b3fc5360 msgr2=0x55f5b3f57870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:44.573 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.570+0000 7f2aa5e00640 1 --2- 172.21.15.32:0/2293828571 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55f5b3fc5360 0x55f5b3f57870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:44.573 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.570+0000 7f2aa5e00640 1 -- 172.21.15.32:0/2293828571 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f5b3f4f600 con 0x55f5b3fe33f0 2024-08-03T20:54:44.573 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.570+0000 7f2aa4a00640 1 --2- 172.21.15.32:0/2293828571 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55f5b3fe2cb0 0x55f5b3fc2270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-03T20:54:44.573 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.570+0000 7f2aa5400640 1 --2- 172.21.15.32:0/2293828571 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55f5b3fc5360 0x55f5b3f57870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-03T20:54:44.573 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.571+0000 7f2aa5e00640 1 --2- 172.21.15.32:0/2293828571 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55f5b3fe33f0 0x55f5b3fc27b0 secure :-1 s=READY pgs=1057 cs=0 l=1 rev1=1 crypto rx=0x7f2aa0003220 tx=0x7f2aa000e660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:54:44.573 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.571+0000 7f2a9e000640 1 -- 172.21.15.32:0/2293828571 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2aa0017020 con 0x55f5b3fe33f0 2024-08-03T20:54:44.573 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.571+0000 7f2a9e000640 1 -- 172.21.15.32:0/2293828571 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f2aa000ee90 con 0x55f5b3fe33f0 2024-08-03T20:54:44.573 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.571+0000 7f2a9e000640 1 -- 172.21.15.32:0/2293828571 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2aa00138e0 con 0x55f5b3fe33f0 2024-08-03T20:54:44.573 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.571+0000 7f2aac1c7f40 1 -- 172.21.15.32:0/2293828571 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x55f5b3f4f320 con 0x55f5b3fe33f0 2024-08-03T20:54:44.573 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.571+0000 7f2aac1c7f40 1 -- 172.21.15.32:0/2293828571 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f5b3f527d0 con 0x55f5b3fe33f0 2024-08-03T20:54:44.574 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.573+0000 7f2a9e000640 1 -- 172.21.15.32:0/2293828571 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f2aa001a020 con 0x55f5b3fe33f0 2024-08-03T20:54:44.574 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.574+0000 7f2a9e000640 1 --2- 172.21.15.32:0/2293828571 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f2a880759f0 0x7f2a88077f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:44.575 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.574+0000 7f2a9e000640 1 -- 172.21.15.32:0/2293828571 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 5795+0+0 (secure 0 0 0) 0x7f2aa0097900 con 0x55f5b3fe33f0 2024-08-03T20:54:44.575 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.575+0000 7f2aac1c7f40 1 --2- 172.21.15.32:0/2293828571 >> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] conn(0x55f5b3f57db0 0x55f5b3f53200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:44.575 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.575+0000 7f2aa5400640 1 --2- 172.21.15.32:0/2293828571 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f2a880759f0 0x7f2a88077f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:44.575 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.575+0000 7f2aac1c7f40 1 -- 172.21.15.32:0/2293828571 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:1 12.6 12:654134d2:::mds0_openfiles.0:head [stat] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84) -- 0x55f5b407be40 con 0x55f5b3f57db0 2024-08-03T20:54:44.575 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.575+0000 7f2aa4a00640 1 --2- 172.21.15.32:0/2293828571 >> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] conn(0x55f5b3f57db0 0x55f5b3f53200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:44.576 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.576+0000 7f2aa5400640 1 --2- 172.21.15.32:0/2293828571 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f2a880759f0 0x7f2a88077f00 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f2a98064000 tx=0x7f2a98004880 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:54:44.576 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.576+0000 7f2aa4a00640 1 --2- 172.21.15.32:0/2293828571 >> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] conn(0x55f5b3f57db0 0x55f5b3f53200 crc :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:54:44.576 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.576+0000 7f2aa4a00640 1 -- 172.21.15.32:0/2293828571 <== osd.1 v2:172.21.15.32:6824/1551241759 1 ==== osd_op_reply(1 mds0_openfiles.0 [stat out=16b] v0'0 uv4 ondisk = 0) ==== 160+0+16 (crc 0 0 0) 0x7f2a94001850 con 0x55f5b3f57db0 2024-08-03T20:54:44.577 INFO:teuthology.orchestra.run.smithi032.stdout:cephfs_metadata/mds0_openfiles.0 mtime 2024-08-03T20:54:37.000000+0000, size 0 2024-08-03T20:54:44.577 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.576+0000 7f2aac1c7f40 1 -- 172.21.15.32:0/2293828571 >> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] conn(0x55f5b3f57db0 msgr2=0x55f5b3f53200 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:44.578 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.576+0000 7f2aac1c7f40 1 --2- 172.21.15.32:0/2293828571 >> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] conn(0x55f5b3f57db0 0x55f5b3f53200 crc :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:44.578 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.576+0000 7f2aac1c7f40 1 -- 172.21.15.32:0/2293828571 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f2a880759f0 msgr2=0x7f2a88077f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:44.578 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.576+0000 7f2aac1c7f40 1 --2- 172.21.15.32:0/2293828571 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f2a880759f0 0x7f2a88077f00 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f2a98064000 tx=0x7f2a98004880 comp rx=0 tx=0).stop 2024-08-03T20:54:44.578 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.576+0000 7f2aac1c7f40 1 -- 172.21.15.32:0/2293828571 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55f5b3fe33f0 msgr2=0x55f5b3fc27b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:44.578 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.576+0000 7f2aac1c7f40 1 --2- 172.21.15.32:0/2293828571 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55f5b3fe33f0 0x55f5b3fc27b0 secure :-1 s=READY pgs=1057 cs=0 l=1 rev1=1 crypto rx=0x7f2aa0003220 tx=0x7f2aa000e660 comp rx=0 tx=0).stop 2024-08-03T20:54:44.578 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.576+0000 7f2aa5e00640 1 -- 172.21.15.32:0/2293828571 reap_dead start 2024-08-03T20:54:44.578 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.576+0000 7f2aac1c7f40 1 -- 172.21.15.32:0/2293828571 shutdown_connections 2024-08-03T20:54:44.578 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.576+0000 7f2aac1c7f40 1 -- 172.21.15.32:0/2293828571 >> 172.21.15.32:0/2293828571 conn(0x55f5b3e2eac0 msgr2=0x55f5b3f55e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:54:44.578 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.576+0000 7f2aac1c7f40 1 -- 172.21.15.32:0/2293828571 shutdown_connections 2024-08-03T20:54:44.578 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.576+0000 7f2aac1c7f40 1 -- 172.21.15.32:0/2293828571 wait complete. 2024-08-03T20:54:44.591 DEBUG:teuthology.orchestra.run.smithi032:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata stat 500.00000000 2024-08-03T20:54:44.734 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.733+0000 7fa5cbf1bf40 1 Processor -- start 2024-08-03T20:54:44.734 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.733+0000 7fa5cbf1bf40 1 -- start start 2024-08-03T20:54:44.736 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.735+0000 7fa5cbf1bf40 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5600ef2aeb40 0x5600ef341cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:44.736 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.735+0000 7fa5cbf1bf40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5600ef321be0 0x5600ef322aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:44.736 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.735+0000 7fa5cbf1bf40 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x5600ef2673e0 0x5600ef3d1df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:44.736 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.735+0000 7fa5cbf1bf40 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x5600ef322510 con 0x5600ef2aeb40 2024-08-03T20:54:44.736 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.735+0000 7fa5cbf1bf40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x5600ef1b7e30 con 0x5600ef321be0 2024-08-03T20:54:44.736 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.735+0000 7fa5cbf1bf40 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x5600ef322690 con 0x5600ef2673e0 2024-08-03T20:54:44.736 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.735+0000 7fa5c4600640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5600ef2aeb40 0x5600ef341cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:44.736 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.735+0000 7fa5c4600640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5600ef2aeb40 0x5600ef341cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:37334/0 (socket says 172.21.15.32:37334) 2024-08-03T20:54:44.736 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.735+0000 7fa5c4600640 1 -- 172.21.15.32:0/3658414957 learned_addr learned my addr 172.21.15.32:0/3658414957 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:54:44.736 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.735+0000 7fa5c5000640 1 --2- 172.21.15.32:0/3658414957 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x5600ef2673e0 0x5600ef3d1df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:44.737 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.735+0000 7fa5c3c00640 1 --2- 172.21.15.32:0/3658414957 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5600ef321be0 0x5600ef322aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:44.737 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.735+0000 7fa5c4600640 1 -- 172.21.15.32:0/3658414957 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x5600ef2673e0 msgr2=0x5600ef3d1df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:54:44.737 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.735+0000 7fa5c4600640 1 --2- 172.21.15.32:0/3658414957 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x5600ef2673e0 0x5600ef3d1df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:44.737 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.735+0000 7fa5c4600640 1 -- 172.21.15.32:0/3658414957 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5600ef321be0 msgr2=0x5600ef322aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:54:44.737 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.735+0000 7fa5c4600640 1 --2- 172.21.15.32:0/3658414957 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5600ef321be0 0x5600ef322aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:44.737 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.735+0000 7fa5c4600640 1 -- 172.21.15.32:0/3658414957 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5600ef190f10 con 0x5600ef2aeb40 2024-08-03T20:54:44.737 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.735+0000 7fa5c3c00640 1 --2- 172.21.15.32:0/3658414957 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5600ef321be0 0x5600ef322aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-03T20:54:44.738 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.736+0000 7fa5c4600640 1 --2- 172.21.15.32:0/3658414957 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5600ef2aeb40 0x5600ef341cc0 secure :-1 s=READY pgs=1059 cs=0 l=1 rev1=1 crypto rx=0x7fa5b400b5a0 tx=0x7fa5b4004f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=84f93a78f8f925eb server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:54:44.738 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.736+0000 7fa5c2800640 1 -- 172.21.15.32:0/3658414957 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5b400a860 con 0x5600ef2aeb40 2024-08-03T20:54:44.739 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.736+0000 7fa5c2800640 1 -- 172.21.15.32:0/3658414957 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fa5b400aa00 con 0x5600ef2aeb40 2024-08-03T20:54:44.739 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.736+0000 7fa5c2800640 1 -- 172.21.15.32:0/3658414957 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5b400ae30 con 0x5600ef2aeb40 2024-08-03T20:54:44.739 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.736+0000 7fa5cbf1bf40 1 -- 172.21.15.32:0/3658414957 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5600ef2aeb40 msgr2=0x5600ef341cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:44.739 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.736+0000 7fa5cbf1bf40 1 --2- 172.21.15.32:0/3658414957 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5600ef2aeb40 0x5600ef341cc0 secure :-1 s=READY pgs=1059 cs=0 l=1 rev1=1 crypto rx=0x7fa5b400b5a0 tx=0x7fa5b4004f90 comp rx=0 tx=0).stop 2024-08-03T20:54:44.739 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.737+0000 7fa5cbf1bf40 1 -- 172.21.15.32:0/3658414957 shutdown_connections 2024-08-03T20:54:44.739 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.737+0000 7fa5cbf1bf40 1 --2- 172.21.15.32:0/3658414957 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5600ef321be0 0x5600ef322aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:44.739 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.737+0000 7fa5cbf1bf40 1 --2- 172.21.15.32:0/3658414957 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x5600ef2673e0 0x5600ef3d1df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:44.739 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.737+0000 7fa5cbf1bf40 1 --2- 172.21.15.32:0/3658414957 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5600ef2aeb40 0x5600ef341cc0 unknown :-1 s=CLOSED pgs=1059 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:44.739 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.737+0000 7fa5cbf1bf40 1 -- 172.21.15.32:0/3658414957 >> 172.21.15.32:0/3658414957 conn(0x5600ef189ac0 msgr2=0x5600ef33cf20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:54:44.739 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.737+0000 7fa5cbf1bf40 1 -- 172.21.15.32:0/3658414957 shutdown_connections 2024-08-03T20:54:44.739 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.737+0000 7fa5cbf1bf40 1 -- 172.21.15.32:0/3658414957 wait complete. 2024-08-03T20:54:44.739 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.738+0000 7fa5cbf1bf40 1 Processor -- start 2024-08-03T20:54:44.739 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.738+0000 7fa5cbf1bf40 1 -- start start 2024-08-03T20:54:44.741 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.739+0000 7fa5cbf1bf40 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5600ef2aeb40 0x5600ef31f7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:44.741 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.739+0000 7fa5cbf1bf40 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x5600ef321be0 0x5600ef33d620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:44.742 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.739+0000 7fa5cbf1bf40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5600ef3d14d0 0x5600ef33db60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:44.742 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.739+0000 7fa5cbf1bf40 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x5600ef2b2b30 con 0x5600ef2aeb40 2024-08-03T20:54:44.742 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.739+0000 7fa5cbf1bf40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x5600ef2b2dd0 con 0x5600ef3d14d0 2024-08-03T20:54:44.742 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.739+0000 7fa5cbf1bf40 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x5600ef267ae0 con 0x5600ef321be0 2024-08-03T20:54:44.742 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.740+0000 7fa5c4600640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5600ef2aeb40 0x5600ef31f7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:44.742 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.740+0000 7fa5c4600640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5600ef2aeb40 0x5600ef31f7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:37336/0 (socket says 172.21.15.32:37336) 2024-08-03T20:54:44.742 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.740+0000 7fa5c4600640 1 -- 172.21.15.32:0/2089574169 learned_addr learned my addr 172.21.15.32:0/2089574169 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:54:44.742 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.740+0000 7fa5c3c00640 1 --2- 172.21.15.32:0/2089574169 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x5600ef321be0 0x5600ef33d620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:44.742 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.740+0000 7fa5c5000640 1 --2- 172.21.15.32:0/2089574169 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5600ef3d14d0 0x5600ef33db60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:44.742 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.740+0000 7fa5c4600640 1 -- 172.21.15.32:0/2089574169 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x5600ef321be0 msgr2=0x5600ef33d620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:44.742 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.740+0000 7fa5c4600640 1 --2- 172.21.15.32:0/2089574169 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x5600ef321be0 0x5600ef33d620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:44.742 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.740+0000 7fa5c4600640 1 -- 172.21.15.32:0/2089574169 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5600ef3d14d0 msgr2=0x5600ef33db60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:44.743 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.740+0000 7fa5c4600640 1 --2- 172.21.15.32:0/2089574169 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5600ef3d14d0 0x5600ef33db60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:44.743 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.740+0000 7fa5c4600640 1 -- 172.21.15.32:0/2089574169 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5600ef31fd20 con 0x5600ef2aeb40 2024-08-03T20:54:44.743 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.741+0000 7fa5c4600640 1 --2- 172.21.15.32:0/2089574169 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5600ef2aeb40 0x5600ef31f7e0 secure :-1 s=READY pgs=1060 cs=0 l=1 rev1=1 crypto rx=0x5600ef317ed0 tx=0x7fa5b4016650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:54:44.743 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.741+0000 7fa5c1400640 1 -- 172.21.15.32:0/2089574169 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5b4016a50 con 0x5600ef2aeb40 2024-08-03T20:54:44.743 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.741+0000 7fa5c1400640 1 -- 172.21.15.32:0/2089574169 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fa5b4016bf0 con 0x5600ef2aeb40 2024-08-03T20:54:44.743 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.741+0000 7fa5cbf1bf40 1 -- 172.21.15.32:0/2089574169 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x5600ef3d5eb0 con 0x5600ef2aeb40 2024-08-03T20:54:44.743 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.741+0000 7fa5c1400640 1 -- 172.21.15.32:0/2089574169 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5b4016e40 con 0x5600ef2aeb40 2024-08-03T20:54:44.743 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.741+0000 7fa5cbf1bf40 1 -- 172.21.15.32:0/2089574169 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5600ef2b2840 con 0x5600ef2aeb40 2024-08-03T20:54:44.743 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.743+0000 7fa5c1400640 1 -- 172.21.15.32:0/2089574169 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7fa5b4029440 con 0x5600ef2aeb40 2024-08-03T20:54:44.745 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.744+0000 7fa5c1400640 1 --2- 172.21.15.32:0/2089574169 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fa5a40759f0 0x7fa5a4077f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:44.745 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.744+0000 7fa5c1400640 1 -- 172.21.15.32:0/2089574169 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 5795+0+0 (secure 0 0 0) 0x7fa5b40993f0 con 0x5600ef2aeb40 2024-08-03T20:54:44.745 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.745+0000 7fa5cbf1bf40 1 --2- 172.21.15.32:0/2089574169 >> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] conn(0x5600ef3d6880 0x5600ef3d94d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:44.745 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.745+0000 7fa5cbf1bf40 1 -- 172.21.15.32:0/2089574169 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:1 12.15 12:aa448500:::500.00000000:head [stat] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84) -- 0x5600ef3d6c90 con 0x5600ef3d6880 2024-08-03T20:54:44.745 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.745+0000 7fa5c3c00640 1 --2- 172.21.15.32:0/2089574169 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fa5a40759f0 0x7fa5a4077f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:44.746 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.745+0000 7fa5c5000640 1 --2- 172.21.15.32:0/2089574169 >> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] conn(0x5600ef3d6880 0x5600ef3d94d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:44.746 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.746+0000 7fa5c3c00640 1 --2- 172.21.15.32:0/2089574169 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fa5a40759f0 0x7fa5a4077f00 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7fa5b0055000 tx=0x7fa5b0058040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:54:44.746 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.746+0000 7fa5c5000640 1 --2- 172.21.15.32:0/2089574169 >> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] conn(0x5600ef3d6880 0x5600ef3d94d0 crc :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:54:44.747 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.746+0000 7fa5c5000640 1 -- 172.21.15.32:0/2089574169 <== osd.6 v2:172.21.15.118:6825/801947386 1 ==== osd_op_reply(1 500.00000000 [stat out=16b] v0'0 uv3 ondisk = 0) ==== 156+0+16 (crc 0 0 0) 0x7fa5bc003040 con 0x5600ef3d6880 2024-08-03T20:54:44.747 INFO:teuthology.orchestra.run.smithi032.stdout:cephfs_metadata/500.00000000 mtime 2024-08-03T20:54:27.000000+0000, size 90 2024-08-03T20:54:44.748 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.746+0000 7fa5cbf1bf40 1 -- 172.21.15.32:0/2089574169 >> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] conn(0x5600ef3d6880 msgr2=0x5600ef3d94d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:44.748 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.746+0000 7fa5cbf1bf40 1 --2- 172.21.15.32:0/2089574169 >> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] conn(0x5600ef3d6880 0x5600ef3d94d0 crc :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:44.748 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.746+0000 7fa5cbf1bf40 1 -- 172.21.15.32:0/2089574169 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fa5a40759f0 msgr2=0x7fa5a4077f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:44.748 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.746+0000 7fa5cbf1bf40 1 --2- 172.21.15.32:0/2089574169 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fa5a40759f0 0x7fa5a4077f00 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7fa5b0055000 tx=0x7fa5b0058040 comp rx=0 tx=0).stop 2024-08-03T20:54:44.748 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.746+0000 7fa5cbf1bf40 1 -- 172.21.15.32:0/2089574169 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5600ef2aeb40 msgr2=0x5600ef31f7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:44.748 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.746+0000 7fa5cbf1bf40 1 --2- 172.21.15.32:0/2089574169 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5600ef2aeb40 0x5600ef31f7e0 secure :-1 s=READY pgs=1060 cs=0 l=1 rev1=1 crypto rx=0x5600ef317ed0 tx=0x7fa5b4016650 comp rx=0 tx=0).stop 2024-08-03T20:54:44.748 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.746+0000 7fa5c5000640 1 -- 172.21.15.32:0/2089574169 reap_dead start 2024-08-03T20:54:44.748 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.747+0000 7fa5cbf1bf40 1 -- 172.21.15.32:0/2089574169 shutdown_connections 2024-08-03T20:54:44.748 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.747+0000 7fa5cbf1bf40 1 -- 172.21.15.32:0/2089574169 >> 172.21.15.32:0/2089574169 conn(0x5600ef189ac0 msgr2=0x5600ef31e800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:54:44.748 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.747+0000 7fa5cbf1bf40 1 -- 172.21.15.32:0/2089574169 shutdown_connections 2024-08-03T20:54:44.749 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.747+0000 7fa5cbf1bf40 1 -- 172.21.15.32:0/2089574169 wait complete. 2024-08-03T20:54:44.760 DEBUG:teuthology.orchestra.run.smithi032:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata stat mds_snaptable 2024-08-03T20:54:44.901 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.901+0000 7fe3f2228f40 1 Processor -- start 2024-08-03T20:54:44.902 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.901+0000 7fe3f2228f40 1 -- start start 2024-08-03T20:54:44.903 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.902+0000 7fe3f2228f40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x559a3ab83360 0x559a3aacb140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:44.904 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.902+0000 7fe3f2228f40 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x559a3aba0cb0 0x559a3ab7b710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:44.904 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.902+0000 7fe3f2228f40 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x559a3aba13f0 0x559a3ab9ba20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:44.904 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.902+0000 7fe3f2228f40 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x559a3aa1ae30 con 0x559a3aba13f0 2024-08-03T20:54:44.904 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.902+0000 7fe3f2228f40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x559a3aa78bf0 con 0x559a3ab83360 2024-08-03T20:54:44.904 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.902+0000 7fe3f2228f40 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x559a3ab84440 con 0x559a3aba0cb0 2024-08-03T20:54:44.904 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.903+0000 7fe3ebe00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x559a3aba13f0 0x559a3ab9ba20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:44.904 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.903+0000 7fe3ebe00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x559a3aba13f0 0x559a3ab9ba20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:37358/0 (socket says 172.21.15.32:37358) 2024-08-03T20:54:44.904 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.903+0000 7fe3ebe00640 1 -- 172.21.15.32:0/4014193036 learned_addr learned my addr 172.21.15.32:0/4014193036 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:54:44.904 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.903+0000 7fe3eb400640 1 --2- 172.21.15.32:0/4014193036 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x559a3ab83360 0x559a3aacb140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:44.904 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.903+0000 7fe3eaa00640 1 --2- 172.21.15.32:0/4014193036 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x559a3aba0cb0 0x559a3ab7b710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:44.905 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.903+0000 7fe3ebe00640 1 -- 172.21.15.32:0/4014193036 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x559a3aba0cb0 msgr2=0x559a3ab7b710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:54:44.905 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.903+0000 7fe3ebe00640 1 --2- 172.21.15.32:0/4014193036 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x559a3aba0cb0 0x559a3ab7b710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:44.905 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.903+0000 7fe3ebe00640 1 -- 172.21.15.32:0/4014193036 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x559a3ab83360 msgr2=0x559a3aacb140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:54:44.905 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.903+0000 7fe3ebe00640 1 --2- 172.21.15.32:0/4014193036 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x559a3ab83360 0x559a3aacb140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:44.905 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.903+0000 7fe3ebe00640 1 -- 172.21.15.32:0/4014193036 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559a3aba3570 con 0x559a3aba13f0 2024-08-03T20:54:44.905 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.903+0000 7fe3eaa00640 1 --2- 172.21.15.32:0/4014193036 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x559a3aba0cb0 0x559a3ab7b710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-03T20:54:44.906 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.904+0000 7fe3ebe00640 1 --2- 172.21.15.32:0/4014193036 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x559a3aba13f0 0x559a3ab9ba20 secure :-1 s=READY pgs=1063 cs=0 l=1 rev1=1 crypto rx=0x7fe3e4025cf0 tx=0x7fe3e4003330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=465784c61e3f01fc server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:54:44.906 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.904+0000 7fe3e9600640 1 -- 172.21.15.32:0/4014193036 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3e4064020 con 0x559a3aba13f0 2024-08-03T20:54:44.907 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.904+0000 7fe3e9600640 1 -- 172.21.15.32:0/4014193036 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fe3e404d9e0 con 0x559a3aba13f0 2024-08-03T20:54:44.907 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.904+0000 7fe3e9600640 1 -- 172.21.15.32:0/4014193036 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3e404de30 con 0x559a3aba13f0 2024-08-03T20:54:44.907 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.904+0000 7fe3f2228f40 1 -- 172.21.15.32:0/4014193036 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x559a3aba13f0 msgr2=0x559a3ab9ba20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:44.907 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.904+0000 7fe3f2228f40 1 --2- 172.21.15.32:0/4014193036 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x559a3aba13f0 0x559a3ab9ba20 secure :-1 s=READY pgs=1063 cs=0 l=1 rev1=1 crypto rx=0x7fe3e4025cf0 tx=0x7fe3e4003330 comp rx=0 tx=0).stop 2024-08-03T20:54:44.907 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.905+0000 7fe3f2228f40 1 -- 172.21.15.32:0/4014193036 shutdown_connections 2024-08-03T20:54:44.907 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.905+0000 7fe3f2228f40 1 --2- 172.21.15.32:0/4014193036 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x559a3aba13f0 0x559a3ab9ba20 unknown :-1 s=CLOSED pgs=1063 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:44.907 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.905+0000 7fe3f2228f40 1 --2- 172.21.15.32:0/4014193036 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x559a3aba0cb0 0x559a3ab7b710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:44.907 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.905+0000 7fe3f2228f40 1 --2- 172.21.15.32:0/4014193036 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x559a3ab83360 0x559a3aacb140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:44.907 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.905+0000 7fe3f2228f40 1 -- 172.21.15.32:0/4014193036 >> 172.21.15.32:0/4014193036 conn(0x559a3a9ecac0 msgr2=0x559a3ab15ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:54:44.907 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.905+0000 7fe3f2228f40 1 -- 172.21.15.32:0/4014193036 shutdown_connections 2024-08-03T20:54:44.907 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.905+0000 7fe3f2228f40 1 -- 172.21.15.32:0/4014193036 wait complete. 2024-08-03T20:54:44.907 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.906+0000 7fe3f2228f40 1 Processor -- start 2024-08-03T20:54:44.907 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.906+0000 7fe3f2228f40 1 -- start start 2024-08-03T20:54:44.909 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.907+0000 7fe3f2228f40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x559a3ab83360 0x559a3ab7bff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:44.909 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.907+0000 7fe3f2228f40 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x559a3aba0cb0 0x559a3ab79f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:44.909 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.907+0000 7fe3f2228f40 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x559a3aba13f0 0x559a3ab7a490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:44.910 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.907+0000 7fe3f2228f40 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x559a3ab7bc50 con 0x559a3aba13f0 2024-08-03T20:54:44.910 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.907+0000 7fe3f2228f40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x559a3ac2d6e0 con 0x559a3ab83360 2024-08-03T20:54:44.910 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.907+0000 7fe3f2228f40 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x559a3ab7edc0 con 0x559a3aba0cb0 2024-08-03T20:54:44.910 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.908+0000 7fe3ebe00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x559a3aba13f0 0x559a3ab7a490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:44.910 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.908+0000 7fe3eb400640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x559a3ab83360 0x559a3ab7bff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:44.910 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.908+0000 7fe3eaa00640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x559a3aba0cb0 0x559a3ab79f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:44.910 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.908+0000 7fe3eb400640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x559a3ab83360 0x559a3ab7bff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.32:56480/0 (socket says 172.21.15.32:56480) 2024-08-03T20:54:44.910 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.908+0000 7fe3ebe00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x559a3aba13f0 0x559a3ab7a490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:37370/0 (socket says 172.21.15.32:37370) 2024-08-03T20:54:44.910 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.908+0000 7fe3eb400640 1 -- 172.21.15.32:0/590549498 learned_addr learned my addr 172.21.15.32:0/590549498 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:54:44.910 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.908+0000 7fe3ebe00640 1 -- 172.21.15.32:0/590549498 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x559a3aba0cb0 msgr2=0x559a3ab79f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:44.911 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.908+0000 7fe3ebe00640 1 --2- 172.21.15.32:0/590549498 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x559a3aba0cb0 0x559a3ab79f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:44.911 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.908+0000 7fe3ebe00640 1 -- 172.21.15.32:0/590549498 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x559a3ab83360 msgr2=0x559a3ab7bff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:44.911 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.908+0000 7fe3ebe00640 1 --2- 172.21.15.32:0/590549498 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x559a3ab83360 0x559a3ab7bff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:44.911 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.908+0000 7fe3ebe00640 1 -- 172.21.15.32:0/590549498 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559a3ab9d140 con 0x559a3aba13f0 2024-08-03T20:54:44.911 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.908+0000 7fe3ebe00640 1 --2- 172.21.15.32:0/590549498 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x559a3aba13f0 0x559a3ab7a490 secure :-1 s=READY pgs=1064 cs=0 l=1 rev1=1 crypto rx=0x7fe3e400a550 tx=0x7fe3e4060470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:54:44.911 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.909+0000 7fe3e3e00640 1 -- 172.21.15.32:0/590549498 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3e4064030 con 0x559a3aba13f0 2024-08-03T20:54:44.911 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.909+0000 7fe3e3e00640 1 -- 172.21.15.32:0/590549498 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fe3e406e020 con 0x559a3aba13f0 2024-08-03T20:54:44.911 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.909+0000 7fe3e3e00640 1 -- 172.21.15.32:0/590549498 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3e4060a90 con 0x559a3aba13f0 2024-08-03T20:54:44.911 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.909+0000 7fe3f2228f40 1 -- 172.21.15.32:0/590549498 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x559a3ab9d420 con 0x559a3aba13f0 2024-08-03T20:54:44.911 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.909+0000 7fe3f2228f40 1 -- 172.21.15.32:0/590549498 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559a3ab9e170 con 0x559a3aba13f0 2024-08-03T20:54:44.912 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.911+0000 7fe3e3e00640 1 -- 172.21.15.32:0/590549498 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7fe3e4067020 con 0x559a3aba13f0 2024-08-03T20:54:44.912 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.912+0000 7fe3e3e00640 1 --2- 172.21.15.32:0/590549498 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fe3cc075a20 0x7fe3cc077f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:44.912 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.912+0000 7fe3e3e00640 1 -- 172.21.15.32:0/590549498 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 5795+0+0 (secure 0 0 0) 0x7fe3e40dd0c0 con 0x559a3aba13f0 2024-08-03T20:54:44.913 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.912+0000 7fe3f2228f40 1 --2- 172.21.15.32:0/590549498 >> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] conn(0x559a3ab78130 0x559a3ab79170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:44.913 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.912+0000 7fe3f2228f40 1 -- 172.21.15.32:0/590549498 --> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] -- osd_op(unknown.0.0:1 12.d 12:b50e409b:::mds_snaptable:head [stat] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84) -- 0x559a3ab7c530 con 0x559a3ab78130 2024-08-03T20:54:44.913 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.912+0000 7fe3eb400640 1 --2- 172.21.15.32:0/590549498 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fe3cc075a20 0x7fe3cc077f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:44.913 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.913+0000 7fe3eaa00640 1 --2- 172.21.15.32:0/590549498 >> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] conn(0x559a3ab78130 0x559a3ab79170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:44.914 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.913+0000 7fe3eb400640 1 --2- 172.21.15.32:0/590549498 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fe3cc075a20 0x7fe3cc077f30 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7fe3dc004fd0 tx=0x7fe3dc00f040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:54:44.914 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.914+0000 7fe3eaa00640 1 --2- 172.21.15.32:0/590549498 >> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] conn(0x559a3ab78130 0x559a3ab79170 crc :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:54:44.914 INFO:teuthology.orchestra.run.smithi032.stdout:cephfs_metadata/mds_snaptable mtime 2024-08-03T20:54:27.000000+0000, size 70 2024-08-03T20:54:44.915 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.914+0000 7fe3eaa00640 1 -- 172.21.15.32:0/590549498 <== osd.7 v2:172.21.15.118:6818/653000048 1 ==== osd_op_reply(1 mds_snaptable [stat out=16b] v0'0 uv1 ondisk = 0) ==== 157+0+16 (crc 0 0 0) 0x7fe3d8003020 con 0x559a3ab78130 2024-08-03T20:54:44.915 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.914+0000 7fe3f2228f40 1 -- 172.21.15.32:0/590549498 >> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] conn(0x559a3ab78130 msgr2=0x559a3ab79170 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:44.916 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.914+0000 7fe3f2228f40 1 --2- 172.21.15.32:0/590549498 >> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] conn(0x559a3ab78130 0x559a3ab79170 crc :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:44.916 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.914+0000 7fe3f2228f40 1 -- 172.21.15.32:0/590549498 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fe3cc075a20 msgr2=0x7fe3cc077f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:44.916 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.914+0000 7fe3f2228f40 1 --2- 172.21.15.32:0/590549498 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fe3cc075a20 0x7fe3cc077f30 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7fe3dc004fd0 tx=0x7fe3dc00f040 comp rx=0 tx=0).stop 2024-08-03T20:54:44.916 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.914+0000 7fe3f2228f40 1 -- 172.21.15.32:0/590549498 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x559a3aba13f0 msgr2=0x559a3ab7a490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:44.916 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.914+0000 7fe3f2228f40 1 --2- 172.21.15.32:0/590549498 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x559a3aba13f0 0x559a3ab7a490 secure :-1 s=READY pgs=1064 cs=0 l=1 rev1=1 crypto rx=0x7fe3e400a550 tx=0x7fe3e4060470 comp rx=0 tx=0).stop 2024-08-03T20:54:44.916 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.914+0000 7fe3ebe00640 1 -- 172.21.15.32:0/590549498 reap_dead start 2024-08-03T20:54:44.916 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.914+0000 7fe3f2228f40 1 -- 172.21.15.32:0/590549498 shutdown_connections 2024-08-03T20:54:44.916 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.914+0000 7fe3f2228f40 1 -- 172.21.15.32:0/590549498 >> 172.21.15.32:0/590549498 conn(0x559a3a9ecac0 msgr2=0x559a3ac2c8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:54:44.916 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.914+0000 7fe3f2228f40 1 -- 172.21.15.32:0/590549498 shutdown_connections 2024-08-03T20:54:44.916 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:44.914+0000 7fe3f2228f40 1 -- 172.21.15.32:0/590549498 wait complete. 2024-08-03T20:54:44.929 DEBUG:teuthology.orchestra.run.smithi032:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata stat mds0_inotable 2024-08-03T20:54:45.071 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.071+0000 7f8d6de0cf40 1 Processor -- start 2024-08-03T20:54:45.071 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.071+0000 7f8d6de0cf40 1 -- start start 2024-08-03T20:54:45.074 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.072+0000 7f8d6de0cf40 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a8e0cd0010 0x55a8e0d471a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:45.074 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.072+0000 7f8d6de0cf40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55a8e0d5a820 0x55a8e0d5ad20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:45.074 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.072+0000 7f8d6de0cf40 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55a8e0d57750 0x55a8e0d5b260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:45.074 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.072+0000 7f8d6de0cf40 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55a8e0bd6e30 con 0x55a8e0cd0010 2024-08-03T20:54:45.074 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.072+0000 7f8d6de0cf40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x55a8e0c34bf0 con 0x55a8e0d5a820 2024-08-03T20:54:45.074 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.072+0000 7f8d6de0cf40 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x55a8e0c87420 con 0x55a8e0d57750 2024-08-03T20:54:45.074 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.072+0000 7f8d67000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a8e0cd0010 0x55a8e0d471a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:45.074 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.072+0000 7f8d67000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a8e0cd0010 0x55a8e0d471a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:37394/0 (socket says 172.21.15.32:37394) 2024-08-03T20:54:45.074 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.072+0000 7f8d67000640 1 -- 172.21.15.32:0/3487538946 learned_addr learned my addr 172.21.15.32:0/3487538946 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:54:45.075 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.073+0000 7f8d67a00640 1 --2- 172.21.15.32:0/3487538946 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55a8e0d57750 0x55a8e0d5b260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:45.075 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.073+0000 7f8d66600640 1 --2- 172.21.15.32:0/3487538946 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55a8e0d5a820 0x55a8e0d5ad20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:45.075 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.073+0000 7f8d67000640 1 -- 172.21.15.32:0/3487538946 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55a8e0d57750 msgr2=0x55a8e0d5b260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:54:45.075 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.073+0000 7f8d67000640 1 --2- 172.21.15.32:0/3487538946 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55a8e0d57750 0x55a8e0d5b260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:45.075 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.073+0000 7f8d67000640 1 -- 172.21.15.32:0/3487538946 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55a8e0d5a820 msgr2=0x55a8e0d5ad20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:54:45.075 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.073+0000 7f8d67000640 1 --2- 172.21.15.32:0/3487538946 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55a8e0d5a820 0x55a8e0d5ad20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:45.075 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.073+0000 7f8d67000640 1 -- 172.21.15.32:0/3487538946 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a8e0d57380 con 0x55a8e0cd0010 2024-08-03T20:54:45.075 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.073+0000 7f8d66600640 1 --2- 172.21.15.32:0/3487538946 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55a8e0d5a820 0x55a8e0d5ad20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-03T20:54:45.075 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.073+0000 7f8d67000640 1 --2- 172.21.15.32:0/3487538946 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a8e0cd0010 0x55a8e0d471a0 secure :-1 s=READY pgs=1067 cs=0 l=1 rev1=1 crypto rx=0x7f8d5800b600 tx=0x7f8d5800bac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=74cd66495eb8edcd server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:54:45.075 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.073+0000 7f8d65200640 1 -- 172.21.15.32:0/3487538946 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d5800a8f0 con 0x55a8e0cd0010 2024-08-03T20:54:45.075 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.074+0000 7f8d65200640 1 -- 172.21.15.32:0/3487538946 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f8d5800aa90 con 0x55a8e0cd0010 2024-08-03T20:54:45.075 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.074+0000 7f8d65200640 1 -- 172.21.15.32:0/3487538946 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d58008450 con 0x55a8e0cd0010 2024-08-03T20:54:45.076 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.074+0000 7f8d6de0cf40 1 -- 172.21.15.32:0/3487538946 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a8e0cd0010 msgr2=0x55a8e0d471a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:45.076 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.074+0000 7f8d6de0cf40 1 --2- 172.21.15.32:0/3487538946 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a8e0cd0010 0x55a8e0d471a0 secure :-1 s=READY pgs=1067 cs=0 l=1 rev1=1 crypto rx=0x7f8d5800b600 tx=0x7f8d5800bac0 comp rx=0 tx=0).stop 2024-08-03T20:54:45.076 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.074+0000 7f8d6de0cf40 1 -- 172.21.15.32:0/3487538946 shutdown_connections 2024-08-03T20:54:45.077 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.074+0000 7f8d6de0cf40 1 --2- 172.21.15.32:0/3487538946 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55a8e0d5a820 0x55a8e0d5ad20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:45.077 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.074+0000 7f8d6de0cf40 1 --2- 172.21.15.32:0/3487538946 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55a8e0d57750 0x55a8e0d5b260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:45.077 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.074+0000 7f8d6de0cf40 1 --2- 172.21.15.32:0/3487538946 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a8e0cd0010 0x55a8e0d471a0 unknown :-1 s=CLOSED pgs=1067 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:45.077 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.074+0000 7f8d6de0cf40 1 -- 172.21.15.32:0/3487538946 >> 172.21.15.32:0/3487538946 conn(0x55a8e0ba8ac0 msgr2=0x55a8e0d57d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:54:45.077 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.074+0000 7f8d6de0cf40 1 -- 172.21.15.32:0/3487538946 shutdown_connections 2024-08-03T20:54:45.077 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.074+0000 7f8d6de0cf40 1 -- 172.21.15.32:0/3487538946 wait complete. 2024-08-03T20:54:45.077 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.075+0000 7f8d6de0cf40 1 Processor -- start 2024-08-03T20:54:45.077 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.075+0000 7f8d6de0cf40 1 -- start start 2024-08-03T20:54:45.079 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.077+0000 7f8d6de0cf40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55a8e0cd0010 0x55a8e0cd1f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:45.079 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.077+0000 7f8d6de0cf40 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55a8e0d57750 0x55a8e0c885d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:45.079 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.077+0000 7f8d6de0cf40 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a8e0d5a820 0x55a8e0c88b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:45.079 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.077+0000 7f8d6de0cf40 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55a8e0d5c7e0 con 0x55a8e0d5a820 2024-08-03T20:54:45.079 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.077+0000 7f8d6de0cf40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x55a8e0cb57a0 con 0x55a8e0cd0010 2024-08-03T20:54:45.079 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.077+0000 7f8d6de0cf40 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x55a8e0d52d40 con 0x55a8e0d57750 2024-08-03T20:54:45.080 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.077+0000 7f8d67a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a8e0d5a820 0x55a8e0c88b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:45.080 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.077+0000 7f8d67a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a8e0d5a820 0x55a8e0c88b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:37402/0 (socket says 172.21.15.32:37402) 2024-08-03T20:54:45.080 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.077+0000 7f8d67a00640 1 -- 172.21.15.32:0/2013556486 learned_addr learned my addr 172.21.15.32:0/2013556486 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:54:45.080 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.077+0000 7f8d66600640 1 --2- 172.21.15.32:0/2013556486 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55a8e0d57750 0x55a8e0c885d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:45.080 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.077+0000 7f8d67000640 1 --2- 172.21.15.32:0/2013556486 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55a8e0cd0010 0x55a8e0cd1f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:45.080 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.077+0000 7f8d67a00640 1 -- 172.21.15.32:0/2013556486 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55a8e0d57750 msgr2=0x55a8e0c885d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:45.080 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.077+0000 7f8d67a00640 1 --2- 172.21.15.32:0/2013556486 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55a8e0d57750 0x55a8e0c885d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:45.080 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.077+0000 7f8d67a00640 1 -- 172.21.15.32:0/2013556486 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55a8e0cd0010 msgr2=0x55a8e0cd1f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:45.080 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.077+0000 7f8d67a00640 1 --2- 172.21.15.32:0/2013556486 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55a8e0cd0010 0x55a8e0cd1f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:45.080 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.078+0000 7f8d67a00640 1 -- 172.21.15.32:0/2013556486 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a8e0cb2d80 con 0x55a8e0d5a820 2024-08-03T20:54:45.080 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.078+0000 7f8d67000640 1 --2- 172.21.15.32:0/2013556486 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55a8e0cd0010 0x55a8e0cd1f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-03T20:54:45.080 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.078+0000 7f8d67a00640 1 --2- 172.21.15.32:0/2013556486 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a8e0d5a820 0x55a8e0c88b10 secure :-1 s=READY pgs=1068 cs=0 l=1 rev1=1 crypto rx=0x7f8d6000a550 tx=0x7f8d60004900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:54:45.081 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.078+0000 7f8d5f400640 1 -- 172.21.15.32:0/2013556486 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d60064030 con 0x55a8e0d5a820 2024-08-03T20:54:45.081 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.078+0000 7f8d5f400640 1 -- 172.21.15.32:0/2013556486 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f8d6004d780 con 0x55a8e0d5a820 2024-08-03T20:54:45.081 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.078+0000 7f8d5f400640 1 -- 172.21.15.32:0/2013556486 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d60060440 con 0x55a8e0d5a820 2024-08-03T20:54:45.081 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.078+0000 7f8d6de0cf40 1 -- 172.21.15.32:0/2013556486 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x55a8e0cb1ce0 con 0x55a8e0d5a820 2024-08-03T20:54:45.081 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.078+0000 7f8d6de0cf40 1 -- 172.21.15.32:0/2013556486 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a8e0cb22a0 con 0x55a8e0d5a820 2024-08-03T20:54:45.081 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.080+0000 7f8d5f400640 1 -- 172.21.15.32:0/2013556486 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f8d60067030 con 0x55a8e0d5a820 2024-08-03T20:54:45.082 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.081+0000 7f8d5f400640 1 --2- 172.21.15.32:0/2013556486 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f8d48075a20 0x7f8d48077f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:45.082 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.081+0000 7f8d5f400640 1 -- 172.21.15.32:0/2013556486 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 5795+0+0 (secure 0 0 0) 0x7f8d600db5e0 con 0x55a8e0d5a820 2024-08-03T20:54:45.082 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.082+0000 7f8d6de0cf40 1 --2- 172.21.15.32:0/2013556486 >> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] conn(0x55a8e0d548e0 0x55a8e0d4f560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:45.082 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.082+0000 7f8d6de0cf40 1 -- 172.21.15.32:0/2013556486 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:1 12.13 12:c91d4a1d:::mds0_inotable:head [stat] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84) -- 0x55a8e0d4faa0 con 0x55a8e0d548e0 2024-08-03T20:54:45.082 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.082+0000 7f8d67000640 1 --2- 172.21.15.32:0/2013556486 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f8d48075a20 0x7f8d48077f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:45.083 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.082+0000 7f8d66600640 1 --2- 172.21.15.32:0/2013556486 >> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] conn(0x55a8e0d548e0 0x55a8e0d4f560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:45.083 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.083+0000 7f8d66600640 1 --2- 172.21.15.32:0/2013556486 >> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] conn(0x55a8e0d548e0 0x55a8e0d4f560 crc :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:54:45.083 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.083+0000 7f8d67000640 1 --2- 172.21.15.32:0/2013556486 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f8d48075a20 0x7f8d48077f30 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f8d58002040 tx=0x7f8d58002fb0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:54:45.083 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.083+0000 7f8d66600640 1 -- 172.21.15.32:0/2013556486 <== osd.1 v2:172.21.15.32:6824/1551241759 1 ==== osd_op_reply(1 mds0_inotable [stat out=16b] v0'0 uv3 ondisk = 0) ==== 157+0+16 (crc 0 0 0) 0x7f8d5400f600 con 0x55a8e0d548e0 2024-08-03T20:54:45.084 INFO:teuthology.orchestra.run.smithi032.stdout:cephfs_metadata/mds0_inotable mtime 2024-08-03T20:54:37.000000+0000, size 34 2024-08-03T20:54:45.085 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.083+0000 7f8d6de0cf40 1 -- 172.21.15.32:0/2013556486 >> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] conn(0x55a8e0d548e0 msgr2=0x55a8e0d4f560 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:45.085 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.083+0000 7f8d6de0cf40 1 --2- 172.21.15.32:0/2013556486 >> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] conn(0x55a8e0d548e0 0x55a8e0d4f560 crc :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:45.085 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.083+0000 7f8d6de0cf40 1 -- 172.21.15.32:0/2013556486 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f8d48075a20 msgr2=0x7f8d48077f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:45.085 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.083+0000 7f8d6de0cf40 1 --2- 172.21.15.32:0/2013556486 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f8d48075a20 0x7f8d48077f30 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f8d58002040 tx=0x7f8d58002fb0 comp rx=0 tx=0).stop 2024-08-03T20:54:45.085 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.083+0000 7f8d6de0cf40 1 -- 172.21.15.32:0/2013556486 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a8e0d5a820 msgr2=0x55a8e0c88b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:45.085 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.083+0000 7f8d6de0cf40 1 --2- 172.21.15.32:0/2013556486 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a8e0d5a820 0x55a8e0c88b10 secure :-1 s=READY pgs=1068 cs=0 l=1 rev1=1 crypto rx=0x7f8d6000a550 tx=0x7f8d60004900 comp rx=0 tx=0).stop 2024-08-03T20:54:45.085 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.083+0000 7f8d67a00640 1 -- 172.21.15.32:0/2013556486 reap_dead start 2024-08-03T20:54:45.085 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.083+0000 7f8d6de0cf40 1 -- 172.21.15.32:0/2013556486 shutdown_connections 2024-08-03T20:54:45.085 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.083+0000 7f8d6de0cf40 1 -- 172.21.15.32:0/2013556486 >> 172.21.15.32:0/2013556486 conn(0x55a8e0ba8ac0 msgr2=0x55a8e0d57d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:54:45.085 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.083+0000 7f8d6de0cf40 1 -- 172.21.15.32:0/2013556486 shutdown_connections 2024-08-03T20:54:45.085 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.083+0000 7f8d6de0cf40 1 -- 172.21.15.32:0/2013556486 wait complete. 2024-08-03T20:54:45.095 DEBUG:teuthology.orchestra.run.smithi032:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata stat 100.00000000 2024-08-03T20:54:45.238 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.237+0000 7f8b84b8cf40 1 Processor -- start 2024-08-03T20:54:45.238 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.237+0000 7f8b84b8cf40 1 -- start start 2024-08-03T20:54:45.240 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.238+0000 7f8b84b8cf40 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b4eda10010 0x55b4eda9c3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:45.240 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.238+0000 7f8b84b8cf40 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55b4eda9a820 0x55b4ed9f15f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:45.240 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.238+0000 7f8b84b8cf40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55b4ed9f3d90 0x55b4edb29a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:45.240 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.238+0000 7f8b84b8cf40 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55b4ed974bf0 con 0x55b4eda10010 2024-08-03T20:54:45.240 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.238+0000 7f8b84b8cf40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x55b4ed9816b0 con 0x55b4ed9f3d90 2024-08-03T20:54:45.240 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.238+0000 7f8b84b8cf40 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x55b4ed916e30 con 0x55b4eda9a820 2024-08-03T20:54:45.241 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.239+0000 7f8b7de00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b4eda10010 0x55b4eda9c3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:45.241 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.239+0000 7f8b7de00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b4eda10010 0x55b4eda9c3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:37416/0 (socket says 172.21.15.32:37416) 2024-08-03T20:54:45.241 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.239+0000 7f8b7de00640 1 -- 172.21.15.32:0/519672462 learned_addr learned my addr 172.21.15.32:0/519672462 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:54:45.241 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.239+0000 7f8b7d400640 1 --2- 172.21.15.32:0/519672462 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55b4eda9a820 0x55b4ed9f15f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:45.241 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.239+0000 7f8b7e800640 1 --2- 172.21.15.32:0/519672462 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55b4ed9f3d90 0x55b4edb29a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:45.241 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.239+0000 7f8b7de00640 1 -- 172.21.15.32:0/519672462 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55b4eda9a820 msgr2=0x55b4ed9f15f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:54:45.241 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.239+0000 7f8b7de00640 1 --2- 172.21.15.32:0/519672462 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55b4eda9a820 0x55b4ed9f15f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:45.241 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.239+0000 7f8b7de00640 1 -- 172.21.15.32:0/519672462 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55b4ed9f3d90 msgr2=0x55b4edb29a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:54:45.241 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.239+0000 7f8b7de00640 1 --2- 172.21.15.32:0/519672462 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55b4ed9f3d90 0x55b4edb29a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:45.241 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.239+0000 7f8b7de00640 1 -- 172.21.15.32:0/519672462 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b4eda9ad40 con 0x55b4eda10010 2024-08-03T20:54:45.241 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.239+0000 7f8b7e800640 1 --2- 172.21.15.32:0/519672462 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55b4ed9f3d90 0x55b4edb29a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-03T20:54:45.241 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.239+0000 7f8b7de00640 1 --2- 172.21.15.32:0/519672462 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b4eda10010 0x55b4eda9c3d0 secure :-1 s=READY pgs=1070 cs=0 l=1 rev1=1 crypto rx=0x7f8b7000b2a0 tx=0x7f8b7000b6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f98c0c81cce5f5b0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:54:45.241 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.240+0000 7f8b77e00640 1 -- 172.21.15.32:0/519672462 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b70016070 con 0x55b4eda10010 2024-08-03T20:54:45.241 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.240+0000 7f8b77e00640 1 -- 172.21.15.32:0/519672462 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f8b7000d7a0 con 0x55b4eda10010 2024-08-03T20:54:45.242 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.240+0000 7f8b77e00640 1 -- 172.21.15.32:0/519672462 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b7000dbf0 con 0x55b4eda10010 2024-08-03T20:54:45.245 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.240+0000 7f8b84b8cf40 1 -- 172.21.15.32:0/519672462 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b4eda10010 msgr2=0x55b4eda9c3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:45.245 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.240+0000 7f8b84b8cf40 1 --2- 172.21.15.32:0/519672462 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b4eda10010 0x55b4eda9c3d0 secure :-1 s=READY pgs=1070 cs=0 l=1 rev1=1 crypto rx=0x7f8b7000b2a0 tx=0x7f8b7000b6a0 comp rx=0 tx=0).stop 2024-08-03T20:54:45.245 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.240+0000 7f8b84b8cf40 1 -- 172.21.15.32:0/519672462 shutdown_connections 2024-08-03T20:54:45.245 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.240+0000 7f8b84b8cf40 1 --2- 172.21.15.32:0/519672462 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55b4ed9f3d90 0x55b4edb29a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:45.245 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.240+0000 7f8b84b8cf40 1 --2- 172.21.15.32:0/519672462 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55b4eda9a820 0x55b4ed9f15f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:45.245 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.240+0000 7f8b84b8cf40 1 --2- 172.21.15.32:0/519672462 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b4eda10010 0x55b4eda9c3d0 unknown :-1 s=CLOSED pgs=1070 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:45.246 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.240+0000 7f8b84b8cf40 1 -- 172.21.15.32:0/519672462 >> 172.21.15.32:0/519672462 conn(0x55b4ed8e8ac0 msgr2=0x55b4eda11a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:54:45.246 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.241+0000 7f8b84b8cf40 1 -- 172.21.15.32:0/519672462 shutdown_connections 2024-08-03T20:54:45.246 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.241+0000 7f8b84b8cf40 1 -- 172.21.15.32:0/519672462 wait complete. 2024-08-03T20:54:45.246 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.242+0000 7f8b84b8cf40 1 Processor -- start 2024-08-03T20:54:45.246 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.242+0000 7f8b84b8cf40 1 -- start start 2024-08-03T20:54:45.246 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.243+0000 7f8b84b8cf40 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55b4ed9f3d90 0x55b4eda82e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:45.246 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.243+0000 7f8b84b8cf40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55b4eda10010 0x55b4edb362e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:45.246 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.243+0000 7f8b84b8cf40 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b4eda9a820 0x55b4edb36820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:45.246 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.243+0000 7f8b84b8cf40 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55b4eda975a0 con 0x55b4eda9a820 2024-08-03T20:54:45.246 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.243+0000 7f8b84b8cf40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x55b4eda0d7b0 con 0x55b4eda10010 2024-08-03T20:54:45.246 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.243+0000 7f8b84b8cf40 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x55b4eda949f0 con 0x55b4ed9f3d90 2024-08-03T20:54:45.246 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.243+0000 7f8b7e800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b4eda9a820 0x55b4edb36820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:45.246 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.243+0000 7f8b7e800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b4eda9a820 0x55b4edb36820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:37424/0 (socket says 172.21.15.32:37424) 2024-08-03T20:54:45.247 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.243+0000 7f8b7e800640 1 -- 172.21.15.32:0/3284238967 learned_addr learned my addr 172.21.15.32:0/3284238967 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:54:45.247 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.244+0000 7f8b7de00640 1 --2- 172.21.15.32:0/3284238967 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55b4ed9f3d90 0x55b4eda82e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:45.247 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.244+0000 7f8b7d400640 1 --2- 172.21.15.32:0/3284238967 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55b4eda10010 0x55b4edb362e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:45.247 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.244+0000 7f8b7e800640 1 -- 172.21.15.32:0/3284238967 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55b4ed9f3d90 msgr2=0x55b4eda82e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:45.247 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.244+0000 7f8b7e800640 1 --2- 172.21.15.32:0/3284238967 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55b4ed9f3d90 0x55b4eda82e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:45.247 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.244+0000 7f8b7e800640 1 -- 172.21.15.32:0/3284238967 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55b4eda10010 msgr2=0x55b4edb362e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:45.247 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.244+0000 7f8b7e800640 1 --2- 172.21.15.32:0/3284238967 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55b4eda10010 0x55b4edb362e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:45.247 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.244+0000 7f8b7e800640 1 -- 172.21.15.32:0/3284238967 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b4edb370f0 con 0x55b4eda9a820 2024-08-03T20:54:45.247 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.244+0000 7f8b7d400640 1 --2- 172.21.15.32:0/3284238967 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55b4eda10010 0x55b4edb362e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-03T20:54:45.247 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.244+0000 7f8b7e800640 1 --2- 172.21.15.32:0/3284238967 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b4eda9a820 0x55b4edb36820 secure :-1 s=READY pgs=1071 cs=0 l=1 rev1=1 crypto rx=0x7f8b7800c410 tx=0x7f8b7800c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:54:45.247 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.244+0000 7f8b76a00640 1 -- 172.21.15.32:0/3284238967 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b780032e0 con 0x55b4eda9a820 2024-08-03T20:54:45.247 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.244+0000 7f8b76a00640 1 -- 172.21.15.32:0/3284238967 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f8b78016070 con 0x55b4eda9a820 2024-08-03T20:54:45.247 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.244+0000 7f8b84b8cf40 1 -- 172.21.15.32:0/3284238967 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x55b4eda94d90 con 0x55b4eda9a820 2024-08-03T20:54:45.248 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.244+0000 7f8b76a00640 1 -- 172.21.15.32:0/3284238967 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b780037d0 con 0x55b4eda9a820 2024-08-03T20:54:45.248 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.245+0000 7f8b84b8cf40 1 -- 172.21.15.32:0/3284238967 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b4eda83380 con 0x55b4eda9a820 2024-08-03T20:54:45.249 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.247+0000 7f8b76a00640 1 -- 172.21.15.32:0/3284238967 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f8b78001d10 con 0x55b4eda9a820 2024-08-03T20:54:45.249 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.248+0000 7f8b76a00640 1 --2- 172.21.15.32:0/3284238967 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f8b60075a20 0x7f8b60077f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:45.249 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.248+0000 7f8b76a00640 1 -- 172.21.15.32:0/3284238967 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 5795+0+0 (secure 0 0 0) 0x7f8b780973a0 con 0x55b4eda9a820 2024-08-03T20:54:45.249 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.248+0000 7f8b84b8cf40 1 --2- 172.21.15.32:0/3284238967 >> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] conn(0x55b4eda0cf20 0x55b4eda06950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:45.249 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.248+0000 7f8b84b8cf40 1 -- 172.21.15.32:0/3284238967 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:1 12.13 12:cd5a64a3:::100.00000000:head [stat] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84) -- 0x55b4eda06e90 con 0x55b4eda0cf20 2024-08-03T20:54:45.249 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.248+0000 7f8b7de00640 1 --2- 172.21.15.32:0/3284238967 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f8b60075a20 0x7f8b60077f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:45.249 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.248+0000 7f8b7d400640 1 --2- 172.21.15.32:0/3284238967 >> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] conn(0x55b4eda0cf20 0x55b4eda06950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:45.250 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.249+0000 7f8b7d400640 1 --2- 172.21.15.32:0/3284238967 >> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] conn(0x55b4eda0cf20 0x55b4eda06950 crc :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:54:45.250 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.249+0000 7f8b7de00640 1 --2- 172.21.15.32:0/3284238967 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f8b60075a20 0x7f8b60077f30 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f8b7000b2a0 tx=0x7f8b700011d0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:54:45.250 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.249+0000 7f8b7d400640 1 -- 172.21.15.32:0/3284238967 <== osd.1 v2:172.21.15.32:6824/1551241759 1 ==== osd_op_reply(1 100.00000000 [stat out=16b] v0'0 uv1 ondisk = 0) ==== 156+0+16 (crc 0 0 0) 0x7f8b6c04e040 con 0x55b4eda0cf20 2024-08-03T20:54:45.250 INFO:teuthology.orchestra.run.smithi032.stdout:cephfs_metadata/100.00000000 mtime 2024-08-03T20:54:27.000000+0000, size 0 2024-08-03T20:54:45.251 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.249+0000 7f8b84b8cf40 1 -- 172.21.15.32:0/3284238967 >> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] conn(0x55b4eda0cf20 msgr2=0x55b4eda06950 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:45.251 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.249+0000 7f8b84b8cf40 1 --2- 172.21.15.32:0/3284238967 >> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] conn(0x55b4eda0cf20 0x55b4eda06950 crc :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:45.251 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.250+0000 7f8b84b8cf40 1 -- 172.21.15.32:0/3284238967 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f8b60075a20 msgr2=0x7f8b60077f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:45.251 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.250+0000 7f8b84b8cf40 1 --2- 172.21.15.32:0/3284238967 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f8b60075a20 0x7f8b60077f30 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f8b7000b2a0 tx=0x7f8b700011d0 comp rx=0 tx=0).stop 2024-08-03T20:54:45.251 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.250+0000 7f8b84b8cf40 1 -- 172.21.15.32:0/3284238967 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b4eda9a820 msgr2=0x55b4edb36820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:45.252 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.250+0000 7f8b84b8cf40 1 --2- 172.21.15.32:0/3284238967 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b4eda9a820 0x55b4edb36820 secure :-1 s=READY pgs=1071 cs=0 l=1 rev1=1 crypto rx=0x7f8b7800c410 tx=0x7f8b7800c8d0 comp rx=0 tx=0).stop 2024-08-03T20:54:45.252 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.250+0000 7f8b7e800640 1 -- 172.21.15.32:0/3284238967 reap_dead start 2024-08-03T20:54:45.252 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.250+0000 7f8b84b8cf40 1 -- 172.21.15.32:0/3284238967 shutdown_connections 2024-08-03T20:54:45.252 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.250+0000 7f8b84b8cf40 1 -- 172.21.15.32:0/3284238967 >> 172.21.15.32:0/3284238967 conn(0x55b4ed8e8ac0 msgr2=0x55b4eda9c390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:54:45.252 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.250+0000 7f8b84b8cf40 1 -- 172.21.15.32:0/3284238967 shutdown_connections 2024-08-03T20:54:45.252 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.250+0000 7f8b84b8cf40 1 -- 172.21.15.32:0/3284238967 wait complete. 2024-08-03T20:54:45.264 DEBUG:teuthology.orchestra.run.smithi032:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata stat mds0_sessionmap 2024-08-03T20:54:45.406 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.406+0000 7f312040df40 1 Processor -- start 2024-08-03T20:54:45.406 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.406+0000 7f312040df40 1 -- start start 2024-08-03T20:54:45.408 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.407+0000 7f312040df40 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55b4c23d9cb0 0x55b4c2448e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:45.408 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.407+0000 7f312040df40 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b4c23dae20 0x55b4c238f3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:45.408 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.407+0000 7f312040df40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55b4c245dba0 0x55b4c238f900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:45.408 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.407+0000 7f312040df40 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55b4c2466fc0 con 0x55b4c23dae20 2024-08-03T20:54:45.409 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.407+0000 7f312040df40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x55b4c22dfe30 con 0x55b4c245dba0 2024-08-03T20:54:45.409 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.407+0000 7f312040df40 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x55b4c2467df0 con 0x55b4c23d9cb0 2024-08-03T20:54:45.409 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.407+0000 7f3113e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b4c23dae20 0x55b4c238f3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:45.409 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.407+0000 7f3113e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b4c23dae20 0x55b4c238f3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:37438/0 (socket says 172.21.15.32:37438) 2024-08-03T20:54:45.409 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.407+0000 7f3113e00640 1 -- 172.21.15.32:0/3248862814 learned_addr learned my addr 172.21.15.32:0/3248862814 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:54:45.409 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.407+0000 7f3118a00640 1 --2- 172.21.15.32:0/3248862814 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55b4c23d9cb0 0x55b4c2448e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:45.409 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.407+0000 7f3119400640 1 --2- 172.21.15.32:0/3248862814 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55b4c245dba0 0x55b4c238f900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:45.409 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.408+0000 7f3113e00640 1 -- 172.21.15.32:0/3248862814 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55b4c23d9cb0 msgr2=0x55b4c2448e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:54:45.409 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.408+0000 7f3113e00640 1 --2- 172.21.15.32:0/3248862814 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55b4c23d9cb0 0x55b4c2448e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:45.409 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.408+0000 7f3113e00640 1 -- 172.21.15.32:0/3248862814 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55b4c245dba0 msgr2=0x55b4c238f900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:54:45.409 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.408+0000 7f3113e00640 1 --2- 172.21.15.32:0/3248862814 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55b4c245dba0 0x55b4c238f900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:45.409 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.408+0000 7f3113e00640 1 -- 172.21.15.32:0/3248862814 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b4c22b8f10 con 0x55b4c23dae20 2024-08-03T20:54:45.409 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.408+0000 7f3119400640 1 --2- 172.21.15.32:0/3248862814 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55b4c245dba0 0x55b4c238f900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-03T20:54:45.410 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.408+0000 7f3113e00640 1 --2- 172.21.15.32:0/3248862814 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b4c23dae20 0x55b4c238f3c0 secure :-1 s=READY pgs=1073 cs=0 l=1 rev1=1 crypto rx=0x7f3104025e80 tx=0x7f3104002fb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=125a690391df79fd server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:54:45.411 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.408+0000 7f3112a00640 1 -- 172.21.15.32:0/3248862814 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f310404d8e0 con 0x55b4c23dae20 2024-08-03T20:54:45.411 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.408+0000 7f3112a00640 1 -- 172.21.15.32:0/3248862814 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f310404dd70 con 0x55b4c23dae20 2024-08-03T20:54:45.411 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.409+0000 7f3112a00640 1 -- 172.21.15.32:0/3248862814 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31040566d0 con 0x55b4c23dae20 2024-08-03T20:54:45.411 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.409+0000 7f312040df40 1 -- 172.21.15.32:0/3248862814 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b4c23dae20 msgr2=0x55b4c238f3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:45.411 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.409+0000 7f312040df40 1 --2- 172.21.15.32:0/3248862814 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b4c23dae20 0x55b4c238f3c0 secure :-1 s=READY pgs=1073 cs=0 l=1 rev1=1 crypto rx=0x7f3104025e80 tx=0x7f3104002fb0 comp rx=0 tx=0).stop 2024-08-03T20:54:45.411 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.409+0000 7f312040df40 1 -- 172.21.15.32:0/3248862814 shutdown_connections 2024-08-03T20:54:45.411 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.409+0000 7f312040df40 1 --2- 172.21.15.32:0/3248862814 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55b4c245dba0 0x55b4c238f900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:45.412 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.409+0000 7f312040df40 1 --2- 172.21.15.32:0/3248862814 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b4c23dae20 0x55b4c238f3c0 unknown :-1 s=CLOSED pgs=1073 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:45.412 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.409+0000 7f312040df40 1 --2- 172.21.15.32:0/3248862814 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55b4c23d9cb0 0x55b4c2448e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:45.412 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.409+0000 7f312040df40 1 -- 172.21.15.32:0/3248862814 >> 172.21.15.32:0/3248862814 conn(0x55b4c22b1ac0 msgr2=0x55b4c23d3850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:54:45.412 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.409+0000 7f312040df40 1 -- 172.21.15.32:0/3248862814 shutdown_connections 2024-08-03T20:54:45.412 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.409+0000 7f312040df40 1 -- 172.21.15.32:0/3248862814 wait complete. 2024-08-03T20:54:45.412 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.410+0000 7f312040df40 1 Processor -- start 2024-08-03T20:54:45.412 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.411+0000 7f312040df40 1 -- start start 2024-08-03T20:54:45.414 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.412+0000 7f312040df40 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b4c23d9cb0 0x55b4c23ba310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:45.414 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.412+0000 7f312040df40 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55b4c23dae20 0x55b4c23ba850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:45.414 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.412+0000 7f312040df40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55b4c245dba0 0x55b4c245e050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:45.414 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.412+0000 7f312040df40 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55b4c23d4b60 con 0x55b4c23d9cb0 2024-08-03T20:54:45.414 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.412+0000 7f312040df40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x55b4c23d45a0 con 0x55b4c245dba0 2024-08-03T20:54:45.414 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.412+0000 7f312040df40 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x55b4c23d4880 con 0x55b4c23dae20 2024-08-03T20:54:45.414 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.412+0000 7f3118a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b4c23d9cb0 0x55b4c23ba310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:45.414 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.412+0000 7f3118a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b4c23d9cb0 0x55b4c23ba310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:37450/0 (socket says 172.21.15.32:37450) 2024-08-03T20:54:45.415 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.412+0000 7f3118a00640 1 -- 172.21.15.32:0/2208201093 learned_addr learned my addr 172.21.15.32:0/2208201093 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:54:45.415 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.412+0000 7f3119400640 1 --2- 172.21.15.32:0/2208201093 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55b4c245dba0 0x55b4c245e050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:45.415 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.412+0000 7f3113e00640 1 --2- 172.21.15.32:0/2208201093 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55b4c23dae20 0x55b4c23ba850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:45.415 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.413+0000 7f3118a00640 1 -- 172.21.15.32:0/2208201093 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55b4c23dae20 msgr2=0x55b4c23ba850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:45.415 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.413+0000 7f3118a00640 1 --2- 172.21.15.32:0/2208201093 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55b4c23dae20 0x55b4c23ba850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:45.415 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.413+0000 7f3118a00640 1 -- 172.21.15.32:0/2208201093 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55b4c245dba0 msgr2=0x55b4c245e050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:45.415 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.413+0000 7f3118a00640 1 --2- 172.21.15.32:0/2208201093 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55b4c245dba0 0x55b4c245e050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:45.415 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.413+0000 7f3118a00640 1 -- 172.21.15.32:0/2208201093 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b4c2461150 con 0x55b4c23d9cb0 2024-08-03T20:54:45.415 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.413+0000 7f3118a00640 1 --2- 172.21.15.32:0/2208201093 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b4c23d9cb0 0x55b4c23ba310 secure :-1 s=READY pgs=1074 cs=0 l=1 rev1=1 crypto rx=0x7f310c0086f0 tx=0x7f310c0087e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:54:45.415 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.413+0000 7f3111600640 1 -- 172.21.15.32:0/2208201093 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f310c00a850 con 0x55b4c23d9cb0 2024-08-03T20:54:45.415 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.413+0000 7f3111600640 1 -- 172.21.15.32:0/2208201093 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f310c00a9f0 con 0x55b4c23d9cb0 2024-08-03T20:54:45.415 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.413+0000 7f3111600640 1 -- 172.21.15.32:0/2208201093 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f310c00ae00 con 0x55b4c23d9cb0 2024-08-03T20:54:45.415 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.413+0000 7f312040df40 1 -- 172.21.15.32:0/2208201093 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x55b4c23bad90 con 0x55b4c23d9cb0 2024-08-03T20:54:45.416 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.413+0000 7f312040df40 1 -- 172.21.15.32:0/2208201093 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b4c245f840 con 0x55b4c23d9cb0 2024-08-03T20:54:45.416 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.415+0000 7f3111600640 1 -- 172.21.15.32:0/2208201093 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f310c016430 con 0x55b4c23d9cb0 2024-08-03T20:54:45.417 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.416+0000 7f3111600640 1 --2- 172.21.15.32:0/2208201093 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f30f80759f0 0x7f30f8077f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:45.417 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.417+0000 7f3111600640 1 -- 172.21.15.32:0/2208201093 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 5795+0+0 (secure 0 0 0) 0x7f310c097eb0 con 0x55b4c23d9cb0 2024-08-03T20:54:45.417 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.417+0000 7f3113e00640 1 --2- 172.21.15.32:0/2208201093 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f30f80759f0 0x7f30f8077f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:45.418 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.417+0000 7f312040df40 1 --2- 172.21.15.32:0/2208201093 >> [v2:172.21.15.32:6800/9088303,v1:172.21.15.32:6801/9088303] conn(0x55b4c245eaa0 0x55b4c2501d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:45.418 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.417+0000 7f312040df40 1 -- 172.21.15.32:0/2208201093 --> [v2:172.21.15.32:6800/9088303,v1:172.21.15.32:6801/9088303] -- osd_op(unknown.0.0:1 12.b 12:d0630e4c:::mds0_sessionmap:head [stat] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84) -- 0x55b4c25022d0 con 0x55b4c245eaa0 2024-08-03T20:54:45.418 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.417+0000 7f3119400640 1 --2- 172.21.15.32:0/2208201093 >> [v2:172.21.15.32:6800/9088303,v1:172.21.15.32:6801/9088303] conn(0x55b4c245eaa0 0x55b4c2501d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:45.418 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.418+0000 7f3113e00640 1 --2- 172.21.15.32:0/2208201093 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f30f80759f0 0x7f30f8077f00 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x55b4c245a6d0 tx=0x7f3104004940 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:54:45.418 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.418+0000 7f3119400640 1 --2- 172.21.15.32:0/2208201093 >> [v2:172.21.15.32:6800/9088303,v1:172.21.15.32:6801/9088303] conn(0x55b4c245eaa0 0x55b4c2501d90 crc :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:54:45.419 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.418+0000 7f3119400640 1 -- 172.21.15.32:0/2208201093 <== osd.3 v2:172.21.15.32:6800/9088303 1 ==== osd_op_reply(1 mds0_sessionmap [stat out=16b] v0'0 uv3 ondisk = 0) ==== 159+0+16 (crc 0 0 0) 0x7f311400b040 con 0x55b4c245eaa0 2024-08-03T20:54:45.419 INFO:teuthology.orchestra.run.smithi032.stdout:cephfs_metadata/mds0_sessionmap mtime 2024-08-03T20:54:37.000000+0000, size 0 2024-08-03T20:54:45.420 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.418+0000 7f312040df40 1 -- 172.21.15.32:0/2208201093 >> [v2:172.21.15.32:6800/9088303,v1:172.21.15.32:6801/9088303] conn(0x55b4c245eaa0 msgr2=0x55b4c2501d90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:45.420 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.418+0000 7f312040df40 1 --2- 172.21.15.32:0/2208201093 >> [v2:172.21.15.32:6800/9088303,v1:172.21.15.32:6801/9088303] conn(0x55b4c245eaa0 0x55b4c2501d90 crc :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:45.420 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.418+0000 7f312040df40 1 -- 172.21.15.32:0/2208201093 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f30f80759f0 msgr2=0x7f30f8077f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:45.420 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.418+0000 7f312040df40 1 --2- 172.21.15.32:0/2208201093 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f30f80759f0 0x7f30f8077f00 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x55b4c245a6d0 tx=0x7f3104004940 comp rx=0 tx=0).stop 2024-08-03T20:54:45.420 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.418+0000 7f312040df40 1 -- 172.21.15.32:0/2208201093 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b4c23d9cb0 msgr2=0x55b4c23ba310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:45.420 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.418+0000 7f312040df40 1 --2- 172.21.15.32:0/2208201093 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b4c23d9cb0 0x55b4c23ba310 secure :-1 s=READY pgs=1074 cs=0 l=1 rev1=1 crypto rx=0x7f310c0086f0 tx=0x7f310c0087e0 comp rx=0 tx=0).stop 2024-08-03T20:54:45.420 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.418+0000 7f3119400640 1 -- 172.21.15.32:0/2208201093 reap_dead start 2024-08-03T20:54:45.420 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.418+0000 7f312040df40 1 -- 172.21.15.32:0/2208201093 shutdown_connections 2024-08-03T20:54:45.420 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.418+0000 7f312040df40 1 -- 172.21.15.32:0/2208201093 >> 172.21.15.32:0/2208201093 conn(0x55b4c22b1ac0 msgr2=0x55b4c238f6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:54:45.420 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.418+0000 7f312040df40 1 -- 172.21.15.32:0/2208201093 shutdown_connections 2024-08-03T20:54:45.421 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.419+0000 7f312040df40 1 -- 172.21.15.32:0/2208201093 wait complete. 2024-08-03T20:54:45.433 DEBUG:teuthology.orchestra.run.smithi032:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata stat 400.00000000 2024-08-03T20:54:45.575 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.574+0000 7f6aa0e34f40 1 Processor -- start 2024-08-03T20:54:45.575 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.574+0000 7f6aa0e34f40 1 -- start start 2024-08-03T20:54:45.577 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.576+0000 7f6aa0e34f40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5563712b5360 0x5563712d47a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:45.578 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.576+0000 7f6aa0e34f40 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5563712b6ea0 0x5563711fdb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:45.578 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.576+0000 7f6aa0e34f40 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x5563712cc330 0x5563712cd970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:45.578 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.576+0000 7f6aa0e34f40 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55637114ce30 con 0x5563712b6ea0 2024-08-03T20:54:45.578 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.576+0000 7f6aa0e34f40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x5563711aabf0 con 0x5563712b5360 2024-08-03T20:54:45.578 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.576+0000 7f6aa0e34f40 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x5563712b6440 con 0x5563712cc330 2024-08-03T20:54:45.578 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.576+0000 7f6a99600640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5563712b6ea0 0x5563711fdb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:45.578 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.576+0000 7f6a99600640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5563712b6ea0 0x5563711fdb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:37464/0 (socket says 172.21.15.32:37464) 2024-08-03T20:54:45.578 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.576+0000 7f6a99600640 1 -- 172.21.15.32:0/3075078428 learned_addr learned my addr 172.21.15.32:0/3075078428 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:54:45.578 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.576+0000 7f6a9aa00640 1 --2- 172.21.15.32:0/3075078428 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x5563712cc330 0x5563712cd970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:45.578 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.576+0000 7f6a9a000640 1 --2- 172.21.15.32:0/3075078428 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5563712b5360 0x5563712d47a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:45.578 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.577+0000 7f6a99600640 1 -- 172.21.15.32:0/3075078428 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x5563712cc330 msgr2=0x5563712cd970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:54:45.579 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.577+0000 7f6a99600640 1 --2- 172.21.15.32:0/3075078428 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x5563712cc330 0x5563712cd970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:45.579 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.577+0000 7f6a99600640 1 -- 172.21.15.32:0/3075078428 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5563712b5360 msgr2=0x5563712d47a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:54:45.579 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.577+0000 7f6a99600640 1 --2- 172.21.15.32:0/3075078428 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5563712b5360 0x5563712d47a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:45.579 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.577+0000 7f6a99600640 1 -- 172.21.15.32:0/3075078428 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5563712cea80 con 0x5563712b6ea0 2024-08-03T20:54:45.579 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.577+0000 7f6a9a000640 1 --2- 172.21.15.32:0/3075078428 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5563712b5360 0x5563712d47a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-03T20:54:45.579 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.577+0000 7f6a99600640 1 --2- 172.21.15.32:0/3075078428 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5563712b6ea0 0x5563711fdb60 secure :-1 s=READY pgs=1077 cs=0 l=1 rev1=1 crypto rx=0x7f6a8800b5d0 tx=0x7f6a8800ba90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7af4a89abd01d42c server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:54:45.579 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.577+0000 7f6a93e00640 1 -- 172.21.15.32:0/3075078428 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a8800a860 con 0x5563712b6ea0 2024-08-03T20:54:45.579 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.577+0000 7f6a93e00640 1 -- 172.21.15.32:0/3075078428 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f6a8800aa00 con 0x5563712b6ea0 2024-08-03T20:54:45.579 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.577+0000 7f6a93e00640 1 -- 172.21.15.32:0/3075078428 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a8800ae30 con 0x5563712b6ea0 2024-08-03T20:54:45.580 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.578+0000 7f6aa0e34f40 1 -- 172.21.15.32:0/3075078428 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5563712b6ea0 msgr2=0x5563711fdb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:45.580 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.578+0000 7f6aa0e34f40 1 --2- 172.21.15.32:0/3075078428 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5563712b6ea0 0x5563711fdb60 secure :-1 s=READY pgs=1077 cs=0 l=1 rev1=1 crypto rx=0x7f6a8800b5d0 tx=0x7f6a8800ba90 comp rx=0 tx=0).stop 2024-08-03T20:54:45.580 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.578+0000 7f6aa0e34f40 1 -- 172.21.15.32:0/3075078428 shutdown_connections 2024-08-03T20:54:45.580 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.578+0000 7f6aa0e34f40 1 --2- 172.21.15.32:0/3075078428 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x5563712cc330 0x5563712cd970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:45.581 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.578+0000 7f6aa0e34f40 1 --2- 172.21.15.32:0/3075078428 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5563712b6ea0 0x5563711fdb60 unknown :-1 s=CLOSED pgs=1077 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:45.581 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.578+0000 7f6aa0e34f40 1 --2- 172.21.15.32:0/3075078428 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5563712b5360 0x5563712d47a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:45.581 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.578+0000 7f6aa0e34f40 1 -- 172.21.15.32:0/3075078428 >> 172.21.15.32:0/3075078428 conn(0x55637111eac0 msgr2=0x5563712cc8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:54:45.581 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.578+0000 7f6aa0e34f40 1 -- 172.21.15.32:0/3075078428 shutdown_connections 2024-08-03T20:54:45.581 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.578+0000 7f6aa0e34f40 1 -- 172.21.15.32:0/3075078428 wait complete. 2024-08-03T20:54:45.581 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.579+0000 7f6aa0e34f40 1 Processor -- start 2024-08-03T20:54:45.581 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.579+0000 7f6aa0e34f40 1 -- start start 2024-08-03T20:54:45.583 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.581+0000 7f6aa0e34f40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5563712b5360 0x55637123d020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:45.583 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.581+0000 7f6aa0e34f40 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5563712b6ea0 0x5563712d42a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:45.583 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.581+0000 7f6aa0e34f40 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x5563712cc330 0x5563712b0970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:45.583 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.581+0000 7f6aa0e34f40 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x5563712cd5a0 con 0x5563712b6ea0 2024-08-03T20:54:45.583 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.581+0000 7f6aa0e34f40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x5563712d4ce0 con 0x5563712b5360 2024-08-03T20:54:45.583 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.581+0000 7f6aa0e34f40 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x55637123d990 con 0x5563712cc330 2024-08-03T20:54:45.583 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.581+0000 7f6a99600640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5563712b6ea0 0x5563712d42a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:45.584 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.581+0000 7f6a99600640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5563712b6ea0 0x5563712d42a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:37472/0 (socket says 172.21.15.32:37472) 2024-08-03T20:54:45.584 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.581+0000 7f6a99600640 1 -- 172.21.15.32:0/3882399331 learned_addr learned my addr 172.21.15.32:0/3882399331 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:54:45.584 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.581+0000 7f6a9a000640 1 --2- 172.21.15.32:0/3882399331 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5563712b5360 0x55637123d020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:45.584 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.581+0000 7f6a9aa00640 1 --2- 172.21.15.32:0/3882399331 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x5563712cc330 0x5563712b0970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:45.584 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.581+0000 7f6a99600640 1 -- 172.21.15.32:0/3882399331 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x5563712cc330 msgr2=0x5563712b0970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:45.584 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.581+0000 7f6a99600640 1 --2- 172.21.15.32:0/3882399331 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x5563712cc330 0x5563712b0970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:45.584 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.581+0000 7f6a99600640 1 -- 172.21.15.32:0/3882399331 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5563712b5360 msgr2=0x55637123d020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:45.584 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.581+0000 7f6a99600640 1 --2- 172.21.15.32:0/3882399331 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5563712b5360 0x55637123d020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:45.584 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.581+0000 7f6a99600640 1 -- 172.21.15.32:0/3882399331 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5563712d3420 con 0x5563712b6ea0 2024-08-03T20:54:45.584 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.581+0000 7f6a9aa00640 1 --2- 172.21.15.32:0/3882399331 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x5563712cc330 0x5563712b0970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-03T20:54:45.584 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.582+0000 7f6a99600640 1 --2- 172.21.15.32:0/3882399331 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5563712b6ea0 0x5563712d42a0 secure :-1 s=READY pgs=1078 cs=0 l=1 rev1=1 crypto rx=0x7f6a8800cfd0 tx=0x7f6a88004f10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:54:45.584 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.582+0000 7f6a92a00640 1 -- 172.21.15.32:0/3882399331 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a88016a30 con 0x5563712b6ea0 2024-08-03T20:54:45.584 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.582+0000 7f6a92a00640 1 -- 172.21.15.32:0/3882399331 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f6a88016bf0 con 0x5563712b6ea0 2024-08-03T20:54:45.584 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.582+0000 7f6a92a00640 1 -- 172.21.15.32:0/3882399331 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a88016e40 con 0x5563712b6ea0 2024-08-03T20:54:45.585 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.582+0000 7f6aa0e34f40 1 -- 172.21.15.32:0/3882399331 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x5563712d3700 con 0x5563712b6ea0 2024-08-03T20:54:45.585 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.582+0000 7f6aa0e34f40 1 -- 172.21.15.32:0/3882399331 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55637123d6d0 con 0x5563712b6ea0 2024-08-03T20:54:45.585 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.584+0000 7f6a92a00640 1 -- 172.21.15.32:0/3882399331 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f6a88029440 con 0x5563712b6ea0 2024-08-03T20:54:45.585 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.585+0000 7f6a92a00640 1 --2- 172.21.15.32:0/3882399331 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f6a7c075a20 0x7f6a7c077f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:45.586 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.585+0000 7f6a92a00640 1 -- 172.21.15.32:0/3882399331 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 5795+0+0 (secure 0 0 0) 0x7f6a88098f80 con 0x5563712b6ea0 2024-08-03T20:54:45.586 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.585+0000 7f6aa0e34f40 1 --2- 172.21.15.32:0/3882399331 >> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] conn(0x5563712d47e0 0x55637136f3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:45.586 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.586+0000 7f6aa0e34f40 1 -- 172.21.15.32:0/3882399331 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:1 12.f 12:f1f69726:::400.00000000:head [stat] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84) -- 0x5563712d2cb0 con 0x5563712d47e0 2024-08-03T20:54:45.586 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.586+0000 7f6a9a000640 1 --2- 172.21.15.32:0/3882399331 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f6a7c075a20 0x7f6a7c077f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:45.587 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.586+0000 7f6a9aa00640 1 --2- 172.21.15.32:0/3882399331 >> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] conn(0x5563712d47e0 0x55637136f3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:45.587 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.586+0000 7f6a9a000640 1 --2- 172.21.15.32:0/3882399331 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f6a7c075a20 0x7f6a7c077f30 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f6a8c001220 tx=0x7f6a8c001760 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:54:45.587 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.587+0000 7f6a9aa00640 1 --2- 172.21.15.32:0/3882399331 >> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] conn(0x5563712d47e0 0x55637136f3b0 crc :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:54:45.588 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.587+0000 7f6a9aa00640 1 -- 172.21.15.32:0/3882399331 <== osd.6 v2:172.21.15.118:6825/801947386 1 ==== osd_op_reply(1 400.00000000 [stat out=16b] v0'0 uv1 ondisk = 0) ==== 156+0+16 (crc 0 0 0) 0x7f6a9400b040 con 0x5563712d47e0 2024-08-03T20:54:45.589 INFO:teuthology.orchestra.run.smithi032.stdout:cephfs_metadata/400.00000000 mtime 2024-08-03T20:54:27.000000+0000, size 22 2024-08-03T20:54:45.590 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.587+0000 7f6aa0e34f40 1 -- 172.21.15.32:0/3882399331 >> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] conn(0x5563712d47e0 msgr2=0x55637136f3b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:45.590 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.587+0000 7f6aa0e34f40 1 --2- 172.21.15.32:0/3882399331 >> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] conn(0x5563712d47e0 0x55637136f3b0 crc :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:45.590 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.587+0000 7f6aa0e34f40 1 -- 172.21.15.32:0/3882399331 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f6a7c075a20 msgr2=0x7f6a7c077f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:45.590 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.587+0000 7f6aa0e34f40 1 --2- 172.21.15.32:0/3882399331 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f6a7c075a20 0x7f6a7c077f30 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f6a8c001220 tx=0x7f6a8c001760 comp rx=0 tx=0).stop 2024-08-03T20:54:45.590 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.587+0000 7f6aa0e34f40 1 -- 172.21.15.32:0/3882399331 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5563712b6ea0 msgr2=0x5563712d42a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:45.590 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.587+0000 7f6aa0e34f40 1 --2- 172.21.15.32:0/3882399331 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5563712b6ea0 0x5563712d42a0 secure :-1 s=READY pgs=1078 cs=0 l=1 rev1=1 crypto rx=0x7f6a8800cfd0 tx=0x7f6a88004f10 comp rx=0 tx=0).stop 2024-08-03T20:54:45.590 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.587+0000 7f6a9aa00640 1 -- 172.21.15.32:0/3882399331 reap_dead start 2024-08-03T20:54:45.590 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.587+0000 7f6aa0e34f40 1 -- 172.21.15.32:0/3882399331 shutdown_connections 2024-08-03T20:54:45.590 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.587+0000 7f6aa0e34f40 1 -- 172.21.15.32:0/3882399331 >> 172.21.15.32:0/3882399331 conn(0x55637111eac0 msgr2=0x5563712cc8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:54:45.591 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.588+0000 7f6aa0e34f40 1 -- 172.21.15.32:0/3882399331 shutdown_connections 2024-08-03T20:54:45.591 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.588+0000 7f6aa0e34f40 1 -- 172.21.15.32:0/3882399331 wait complete. 2024-08-03T20:54:45.602 DEBUG:teuthology.orchestra.run.smithi032:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata stat 100.00000000.inode 2024-08-03T20:54:45.743 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.743+0000 7f6d5e1cef40 1 Processor -- start 2024-08-03T20:54:45.744 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.743+0000 7f6d5e1cef40 1 -- start start 2024-08-03T20:54:45.745 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.744+0000 7f6d5e1cef40 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55e8799c3360 0x55e8799bb520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:45.745 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.744+0000 7f6d5e1cef40 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55e8799da330 0x55e87994b7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:45.745 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.744+0000 7f6d5e1cef40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55e8799e13f0 0x55e879a72ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:45.745 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.744+0000 7f6d5e1cef40 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55e8798b8bf0 con 0x55e8799c3360 2024-08-03T20:54:45.745 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.744+0000 7f6d5e1cef40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x55e8798c56b0 con 0x55e8799e13f0 2024-08-03T20:54:45.745 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.744+0000 7f6d5e1cef40 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x55e87985ae30 con 0x55e8799da330 2024-08-03T20:54:45.745 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.744+0000 7f6d57400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55e8799c3360 0x55e8799bb520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:45.745 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.745+0000 7f6d57400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55e8799c3360 0x55e8799bb520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:37474/0 (socket says 172.21.15.32:37474) 2024-08-03T20:54:45.745 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.745+0000 7f6d57400640 1 -- 172.21.15.32:0/397414167 learned_addr learned my addr 172.21.15.32:0/397414167 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:54:45.747 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.745+0000 7f6d56a00640 1 --2- 172.21.15.32:0/397414167 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55e8799da330 0x55e87994b7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:45.748 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.745+0000 7f6d57400640 1 -- 172.21.15.32:0/397414167 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55e8799da330 msgr2=0x55e87994b7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:54:45.748 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.745+0000 7f6d57e00640 1 --2- 172.21.15.32:0/397414167 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55e8799e13f0 0x55e879a72ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:45.748 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.745+0000 7f6d57400640 1 --2- 172.21.15.32:0/397414167 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55e8799da330 0x55e87994b7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:45.748 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.745+0000 7f6d57400640 1 -- 172.21.15.32:0/397414167 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55e8799e13f0 msgr2=0x55e879a72ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:54:45.748 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.745+0000 7f6d57400640 1 --2- 172.21.15.32:0/397414167 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55e8799e13f0 0x55e879a72ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:45.748 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.745+0000 7f6d57400640 1 -- 172.21.15.32:0/397414167 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e8799dd650 con 0x55e8799c3360 2024-08-03T20:54:45.748 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.745+0000 7f6d56a00640 1 --2- 172.21.15.32:0/397414167 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55e8799da330 0x55e87994b7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-03T20:54:45.748 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.745+0000 7f6d57e00640 1 --2- 172.21.15.32:0/397414167 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55e8799e13f0 0x55e879a72ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-03T20:54:45.748 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.745+0000 7f6d57400640 1 --2- 172.21.15.32:0/397414167 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55e8799c3360 0x55e8799bb520 secure :-1 s=READY pgs=1080 cs=0 l=1 rev1=1 crypto rx=0x7f6d48001d00 tx=0x7f6d4800b5b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4fbc106348c0ffa6 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:54:45.748 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.745+0000 7f6d55600640 1 -- 172.21.15.32:0/397414167 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d4800bd80 con 0x55e8799c3360 2024-08-03T20:54:45.748 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.745+0000 7f6d55600640 1 -- 172.21.15.32:0/397414167 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f6d48001e50 con 0x55e8799c3360 2024-08-03T20:54:45.748 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.746+0000 7f6d55600640 1 -- 172.21.15.32:0/397414167 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d4800f660 con 0x55e8799c3360 2024-08-03T20:54:45.749 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.746+0000 7f6d5e1cef40 1 -- 172.21.15.32:0/397414167 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55e8799c3360 msgr2=0x55e8799bb520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:45.749 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.746+0000 7f6d5e1cef40 1 --2- 172.21.15.32:0/397414167 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55e8799c3360 0x55e8799bb520 secure :-1 s=READY pgs=1080 cs=0 l=1 rev1=1 crypto rx=0x7f6d48001d00 tx=0x7f6d4800b5b0 comp rx=0 tx=0).stop 2024-08-03T20:54:45.749 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.746+0000 7f6d5e1cef40 1 -- 172.21.15.32:0/397414167 shutdown_connections 2024-08-03T20:54:45.749 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.746+0000 7f6d5e1cef40 1 --2- 172.21.15.32:0/397414167 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55e8799e13f0 0x55e879a72ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:45.749 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.746+0000 7f6d5e1cef40 1 --2- 172.21.15.32:0/397414167 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55e8799da330 0x55e87994b7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:45.749 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.746+0000 7f6d5e1cef40 1 --2- 172.21.15.32:0/397414167 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55e8799c3360 0x55e8799bb520 unknown :-1 s=CLOSED pgs=1080 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:45.749 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.746+0000 7f6d5e1cef40 1 -- 172.21.15.32:0/397414167 >> 172.21.15.32:0/397414167 conn(0x55e87982cac0 msgr2=0x55e8799dc1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:54:45.749 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.746+0000 7f6d5e1cef40 1 -- 172.21.15.32:0/397414167 shutdown_connections 2024-08-03T20:54:45.749 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.746+0000 7f6d5e1cef40 1 -- 172.21.15.32:0/397414167 wait complete. 2024-08-03T20:54:45.749 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.747+0000 7f6d5e1cef40 1 Processor -- start 2024-08-03T20:54:45.749 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.747+0000 7f6d5e1cef40 1 -- start start 2024-08-03T20:54:45.751 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.749+0000 7f6d5e1cef40 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55e8799c3360 0x55e879a7bee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:45.751 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.749+0000 7f6d5e1cef40 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55e8799da330 0x55e87994d640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:45.751 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.749+0000 7f6d5e1cef40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55e8799e13f0 0x55e8799dbfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:45.751 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.749+0000 7f6d5e1cef40 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55e8799dbcc0 con 0x55e8799c3360 2024-08-03T20:54:45.751 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.749+0000 7f6d5e1cef40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x55e879955b70 con 0x55e8799e13f0 2024-08-03T20:54:45.751 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.749+0000 7f6d5e1cef40 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x55e879a7a3b0 con 0x55e8799da330 2024-08-03T20:54:45.752 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.749+0000 7f6d57400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55e8799c3360 0x55e879a7bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:45.752 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.749+0000 7f6d57400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55e8799c3360 0x55e879a7bee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:37476/0 (socket says 172.21.15.32:37476) 2024-08-03T20:54:45.752 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.749+0000 7f6d57400640 1 -- 172.21.15.32:0/262919076 learned_addr learned my addr 172.21.15.32:0/262919076 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:54:45.752 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.749+0000 7f6d56a00640 1 --2- 172.21.15.32:0/262919076 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55e8799da330 0x55e87994d640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:45.752 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.749+0000 7f6d57e00640 1 --2- 172.21.15.32:0/262919076 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55e8799e13f0 0x55e8799dbfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:45.752 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.749+0000 7f6d57400640 1 -- 172.21.15.32:0/262919076 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55e8799da330 msgr2=0x55e87994d640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:45.752 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.749+0000 7f6d57400640 1 --2- 172.21.15.32:0/262919076 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55e8799da330 0x55e87994d640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:45.752 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.749+0000 7f6d57400640 1 -- 172.21.15.32:0/262919076 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55e8799e13f0 msgr2=0x55e8799dbfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:45.752 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.749+0000 7f6d57400640 1 --2- 172.21.15.32:0/262919076 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55e8799e13f0 0x55e8799dbfa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:45.752 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.750+0000 7f6d57400640 1 -- 172.21.15.32:0/262919076 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e8799558b0 con 0x55e8799c3360 2024-08-03T20:54:45.752 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.750+0000 7f6d56a00640 1 --2- 172.21.15.32:0/262919076 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55e8799da330 0x55e87994d640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-03T20:54:45.752 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.750+0000 7f6d57e00640 1 --2- 172.21.15.32:0/262919076 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55e8799e13f0 0x55e8799dbfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-03T20:54:45.752 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.750+0000 7f6d57400640 1 --2- 172.21.15.32:0/262919076 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55e8799c3360 0x55e879a7bee0 secure :-1 s=READY pgs=1081 cs=0 l=1 rev1=1 crypto rx=0x7f6d4800ff30 tx=0x7f6d4800ff60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:54:45.752 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.750+0000 7f6d4fe00640 1 -- 172.21.15.32:0/262919076 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d4801b430 con 0x55e8799c3360 2024-08-03T20:54:45.753 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.750+0000 7f6d4fe00640 1 -- 172.21.15.32:0/262919076 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f6d48020020 con 0x55e8799c3360 2024-08-03T20:54:45.753 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.750+0000 7f6d5e1cef40 1 -- 172.21.15.32:0/262919076 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x55e8799b9890 con 0x55e8799c3360 2024-08-03T20:54:45.753 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.750+0000 7f6d4fe00640 1 -- 172.21.15.32:0/262919076 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d4801b740 con 0x55e8799c3360 2024-08-03T20:54:45.753 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.750+0000 7f6d5e1cef40 1 -- 172.21.15.32:0/262919076 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e8799b9e10 con 0x55e8799c3360 2024-08-03T20:54:45.753 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.752+0000 7f6d4fe00640 1 -- 172.21.15.32:0/262919076 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f6d48017070 con 0x55e8799c3360 2024-08-03T20:54:45.754 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.753+0000 7f6d4fe00640 1 --2- 172.21.15.32:0/262919076 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f6d38075a20 0x7f6d38077f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:45.754 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.753+0000 7f6d4fe00640 1 -- 172.21.15.32:0/262919076 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 5795+0+0 (secure 0 0 0) 0x7f6d48098c20 con 0x55e8799c3360 2024-08-03T20:54:45.754 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.754+0000 7f6d5e1cef40 1 --2- 172.21.15.32:0/262919076 >> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] conn(0x55e8799542f0 0x55e879954700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:45.754 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.754+0000 7f6d5e1cef40 1 -- 172.21.15.32:0/262919076 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:1 12.1f 12:fe07bba1:::100.00000000.inode:head [stat] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84) -- 0x55e87994db80 con 0x55e8799542f0 2024-08-03T20:54:45.755 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.754+0000 7f6d56a00640 1 --2- 172.21.15.32:0/262919076 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f6d38075a20 0x7f6d38077f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:45.755 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.754+0000 7f6d57e00640 1 --2- 172.21.15.32:0/262919076 >> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] conn(0x55e8799542f0 0x55e879954700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:45.755 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.755+0000 7f6d56a00640 1 --2- 172.21.15.32:0/262919076 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f6d38075a20 0x7f6d38077f30 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7f6d44001860 tx=0x7f6d44001590 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:54:45.755 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.755+0000 7f6d57e00640 1 --2- 172.21.15.32:0/262919076 >> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] conn(0x55e8799542f0 0x55e879954700 crc :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:54:45.756 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.755+0000 7f6d57e00640 1 -- 172.21.15.32:0/262919076 <== osd.6 v2:172.21.15.118:6825/801947386 1 ==== osd_op_reply(1 100.00000000.inode [stat out=16b] v0'0 uv3 ondisk = 0) ==== 162+0+16 (crc 0 0 0) 0x7f6d5000b040 con 0x55e8799542f0 2024-08-03T20:54:45.756 INFO:teuthology.orchestra.run.smithi032.stdout:cephfs_metadata/100.00000000.inode mtime 2024-08-03T20:54:27.000000+0000, size 582 2024-08-03T20:54:45.757 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.756+0000 7f6d5e1cef40 1 -- 172.21.15.32:0/262919076 >> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] conn(0x55e8799542f0 msgr2=0x55e879954700 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:45.757 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.756+0000 7f6d5e1cef40 1 --2- 172.21.15.32:0/262919076 >> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] conn(0x55e8799542f0 0x55e879954700 crc :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:45.757 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.756+0000 7f6d5e1cef40 1 -- 172.21.15.32:0/262919076 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f6d38075a20 msgr2=0x7f6d38077f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:45.757 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.756+0000 7f6d5e1cef40 1 --2- 172.21.15.32:0/262919076 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f6d38075a20 0x7f6d38077f30 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7f6d44001860 tx=0x7f6d44001590 comp rx=0 tx=0).stop 2024-08-03T20:54:45.757 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.756+0000 7f6d5e1cef40 1 -- 172.21.15.32:0/262919076 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55e8799c3360 msgr2=0x55e879a7bee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:45.757 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.756+0000 7f6d5e1cef40 1 --2- 172.21.15.32:0/262919076 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55e8799c3360 0x55e879a7bee0 secure :-1 s=READY pgs=1081 cs=0 l=1 rev1=1 crypto rx=0x7f6d4800ff30 tx=0x7f6d4800ff60 comp rx=0 tx=0).stop 2024-08-03T20:54:45.757 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.756+0000 7f6d57e00640 1 -- 172.21.15.32:0/262919076 reap_dead start 2024-08-03T20:54:45.757 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.756+0000 7f6d5e1cef40 1 -- 172.21.15.32:0/262919076 shutdown_connections 2024-08-03T20:54:45.758 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.756+0000 7f6d5e1cef40 1 -- 172.21.15.32:0/262919076 >> 172.21.15.32:0/262919076 conn(0x55e87982cac0 msgr2=0x55e8799bb520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:54:45.758 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.756+0000 7f6d5e1cef40 1 -- 172.21.15.32:0/262919076 shutdown_connections 2024-08-03T20:54:45.758 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.756+0000 7f6d5e1cef40 1 -- 172.21.15.32:0/262919076 wait complete. 2024-08-03T20:54:45.768 DEBUG:teuthology.orchestra.run.smithi032:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata stat 1.00000000 2024-08-03T20:54:45.911 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.910+0000 7ff7b951df40 1 Processor -- start 2024-08-03T20:54:45.911 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.910+0000 7ff7b951df40 1 -- start start 2024-08-03T20:54:45.913 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.911+0000 7ff7b951df40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5584bddc3360 0x5584bdd4dfc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:45.913 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.912+0000 7ff7b951df40 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x5584bdde0cb0 0x5584bdddaae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:45.913 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.912+0000 7ff7b951df40 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5584bdde13f0 0x5584bdd0ada0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:45.914 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.912+0000 7ff7b951df40 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x5584bdc5ae30 con 0x5584bdde13f0 2024-08-03T20:54:45.914 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.912+0000 7ff7b951df40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x5584bdcb8bf0 con 0x5584bddc3360 2024-08-03T20:54:45.914 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.912+0000 7ff7b951df40 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x5584bddc4440 con 0x5584bdde0cb0 2024-08-03T20:54:45.914 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.912+0000 7ff7b2600640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5584bdde13f0 0x5584bdd0ada0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:45.914 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.912+0000 7ff7b2600640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5584bdde13f0 0x5584bdd0ada0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:37484/0 (socket says 172.21.15.32:37484) 2024-08-03T20:54:45.914 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.912+0000 7ff7b2600640 1 -- 172.21.15.32:0/1168074823 learned_addr learned my addr 172.21.15.32:0/1168074823 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:54:45.914 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.912+0000 7ff7b1200640 1 --2- 172.21.15.32:0/1168074823 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x5584bdde0cb0 0x5584bdddaae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:45.914 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.912+0000 7ff7b1c00640 1 --2- 172.21.15.32:0/1168074823 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5584bddc3360 0x5584bdd4dfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:45.914 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.912+0000 7ff7b2600640 1 -- 172.21.15.32:0/1168074823 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x5584bdde0cb0 msgr2=0x5584bdddaae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:54:45.914 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.912+0000 7ff7b2600640 1 --2- 172.21.15.32:0/1168074823 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x5584bdde0cb0 0x5584bdddaae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:45.914 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.912+0000 7ff7b2600640 1 -- 172.21.15.32:0/1168074823 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5584bddc3360 msgr2=0x5584bdd4dfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:54:45.914 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.912+0000 7ff7b2600640 1 --2- 172.21.15.32:0/1168074823 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5584bddc3360 0x5584bdd4dfc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:45.914 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.912+0000 7ff7b2600640 1 -- 172.21.15.32:0/1168074823 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5584bdd0c3b0 con 0x5584bdde13f0 2024-08-03T20:54:45.915 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.913+0000 7ff7b2600640 1 --2- 172.21.15.32:0/1168074823 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5584bdde13f0 0x5584bdd0ada0 secure :-1 s=READY pgs=1083 cs=0 l=1 rev1=1 crypto rx=0x7ff7ac00a550 tx=0x7ff7ac003330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=17792463c6a40aa3 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:54:45.915 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.913+0000 7ff7ab400640 1 -- 172.21.15.32:0/1168074823 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7ac064030 con 0x5584bdde13f0 2024-08-03T20:54:45.915 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.913+0000 7ff7ab400640 1 -- 172.21.15.32:0/1168074823 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7ff7ac004020 con 0x5584bdde13f0 2024-08-03T20:54:45.915 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.913+0000 7ff7ab400640 1 -- 172.21.15.32:0/1168074823 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7ac004270 con 0x5584bdde13f0 2024-08-03T20:54:45.916 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.914+0000 7ff7b951df40 1 -- 172.21.15.32:0/1168074823 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5584bdde13f0 msgr2=0x5584bdd0ada0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:45.916 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.914+0000 7ff7b951df40 1 --2- 172.21.15.32:0/1168074823 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5584bdde13f0 0x5584bdd0ada0 secure :-1 s=READY pgs=1083 cs=0 l=1 rev1=1 crypto rx=0x7ff7ac00a550 tx=0x7ff7ac003330 comp rx=0 tx=0).stop 2024-08-03T20:54:45.916 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.914+0000 7ff7b951df40 1 -- 172.21.15.32:0/1168074823 shutdown_connections 2024-08-03T20:54:45.916 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.914+0000 7ff7b951df40 1 --2- 172.21.15.32:0/1168074823 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5584bdde13f0 0x5584bdd0ada0 unknown :-1 s=CLOSED pgs=1083 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:45.916 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.914+0000 7ff7b951df40 1 --2- 172.21.15.32:0/1168074823 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x5584bdde0cb0 0x5584bdddaae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:45.916 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.914+0000 7ff7b951df40 1 --2- 172.21.15.32:0/1168074823 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5584bddc3360 0x5584bdd4dfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:45.916 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.914+0000 7ff7b951df40 1 -- 172.21.15.32:0/1168074823 >> 172.21.15.32:0/1168074823 conn(0x5584bdc2cac0 msgr2=0x5584bdd4d5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:54:45.916 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.914+0000 7ff7b951df40 1 -- 172.21.15.32:0/1168074823 shutdown_connections 2024-08-03T20:54:45.917 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.914+0000 7ff7b951df40 1 -- 172.21.15.32:0/1168074823 wait complete. 2024-08-03T20:54:45.917 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.915+0000 7ff7b951df40 1 Processor -- start 2024-08-03T20:54:45.917 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.915+0000 7ff7b951df40 1 -- start start 2024-08-03T20:54:45.918 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.916+0000 7ff7b951df40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5584bddc3360 0x5584bde78d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:45.919 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.916+0000 7ff7b951df40 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5584bdde0cb0 0x5584bdddc1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:45.919 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.916+0000 7ff7b951df40 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x5584bdde13f0 0x5584bdddc700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:45.919 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.916+0000 7ff7b951df40 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x5584bddc2880 con 0x5584bdde0cb0 2024-08-03T20:54:45.919 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.916+0000 7ff7b951df40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x5584bdd0cb10 con 0x5584bddc3360 2024-08-03T20:54:45.919 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.916+0000 7ff7b951df40 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x5584bde793e0 con 0x5584bdde13f0 2024-08-03T20:54:45.919 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.917+0000 7ff7b1200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5584bdde0cb0 0x5584bdddc1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:45.919 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.917+0000 7ff7b1200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5584bdde0cb0 0x5584bdddc1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:37498/0 (socket says 172.21.15.32:37498) 2024-08-03T20:54:45.919 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.917+0000 7ff7b1200640 1 -- 172.21.15.32:0/2885821000 learned_addr learned my addr 172.21.15.32:0/2885821000 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:54:45.919 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.917+0000 7ff7b2600640 1 --2- 172.21.15.32:0/2885821000 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x5584bdde13f0 0x5584bdddc700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:45.919 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.917+0000 7ff7b1c00640 1 --2- 172.21.15.32:0/2885821000 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5584bddc3360 0x5584bde78d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:45.919 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.917+0000 7ff7b1200640 1 -- 172.21.15.32:0/2885821000 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x5584bdde13f0 msgr2=0x5584bdddc700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:45.919 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.917+0000 7ff7b1200640 1 --2- 172.21.15.32:0/2885821000 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x5584bdde13f0 0x5584bdddc700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:45.919 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.917+0000 7ff7b1200640 1 -- 172.21.15.32:0/2885821000 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5584bddc3360 msgr2=0x5584bde78d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:45.920 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.917+0000 7ff7b1200640 1 --2- 172.21.15.32:0/2885821000 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5584bddc3360 0x5584bde78d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:45.920 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.917+0000 7ff7b1200640 1 -- 172.21.15.32:0/2885821000 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5584bddb18d0 con 0x5584bdde0cb0 2024-08-03T20:54:45.920 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.917+0000 7ff7b1c00640 1 --2- 172.21.15.32:0/2885821000 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5584bddc3360 0x5584bde78d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-03T20:54:45.920 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.918+0000 7ff7b1200640 1 --2- 172.21.15.32:0/2885821000 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5584bdde0cb0 0x5584bdddc1c0 secure :-1 s=READY pgs=1084 cs=0 l=1 rev1=1 crypto rx=0x7ff7a000c310 tx=0x7ff7a000c7d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:54:45.920 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.918+0000 7ff7aa000640 1 -- 172.21.15.32:0/2885821000 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7a000f810 con 0x5584bdde0cb0 2024-08-03T20:54:45.920 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.918+0000 7ff7aa000640 1 -- 172.21.15.32:0/2885821000 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7ff7a0013070 con 0x5584bdde0cb0 2024-08-03T20:54:45.920 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.918+0000 7ff7aa000640 1 -- 172.21.15.32:0/2885821000 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7a000fd60 con 0x5584bdde0cb0 2024-08-03T20:54:45.920 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.918+0000 7ff7b951df40 1 -- 172.21.15.32:0/2885821000 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x5584bddbdcf0 con 0x5584bdde0cb0 2024-08-03T20:54:45.920 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.918+0000 7ff7b951df40 1 -- 172.21.15.32:0/2885821000 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5584bdd0bb30 con 0x5584bdde0cb0 2024-08-03T20:54:45.920 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.920+0000 7ff7aa000640 1 -- 172.21.15.32:0/2885821000 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7ff7a00213f0 con 0x5584bdde0cb0 2024-08-03T20:54:45.922 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.921+0000 7ff7aa000640 1 --2- 172.21.15.32:0/2885821000 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7ff7940759f0 0x7ff794077f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:45.922 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.921+0000 7ff7aa000640 1 -- 172.21.15.32:0/2885821000 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 5795+0+0 (secure 0 0 0) 0x7ff7a0097c70 con 0x5584bdde0cb0 2024-08-03T20:54:45.922 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.922+0000 7ff7b1c00640 1 --2- 172.21.15.32:0/2885821000 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7ff7940759f0 0x7ff794077f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:45.922 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.922+0000 7ff7b951df40 1 --2- 172.21.15.32:0/2885821000 >> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] conn(0x5584bddd95f0 0x5584bdd4d410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:45.922 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.922+0000 7ff7b951df40 1 -- 172.21.15.32:0/2885821000 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:1 12.1f 12:ff5b34d6:::1.00000000:head [stat] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84) -- 0x5584bdd4b410 con 0x5584bddd95f0 2024-08-03T20:54:45.923 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.922+0000 7ff7b2600640 1 --2- 172.21.15.32:0/2885821000 >> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] conn(0x5584bddd95f0 0x5584bdd4d410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:45.923 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.922+0000 7ff7b1c00640 1 --2- 172.21.15.32:0/2885821000 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7ff7940759f0 0x7ff794077f00 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x5584bdde3780 tx=0x7ff7a400f040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:54:45.923 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.923+0000 7ff7b2600640 1 --2- 172.21.15.32:0/2885821000 >> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] conn(0x5584bddd95f0 0x5584bdd4d410 crc :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:54:45.924 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.923+0000 7ff7b2600640 1 -- 172.21.15.32:0/2885821000 <== osd.6 v2:172.21.15.118:6825/801947386 1 ==== osd_op_reply(1 1.00000000 [stat out=16b] v0'0 uv6 ondisk = 0) ==== 154+0+16 (crc 0 0 0) 0x7ff7ac00b040 con 0x5584bddd95f0 2024-08-03T20:54:45.924 INFO:teuthology.orchestra.run.smithi032.stdout:cephfs_metadata/1.00000000 mtime 2024-08-03T20:54:37.000000+0000, size 0 2024-08-03T20:54:45.925 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.923+0000 7ff7b951df40 1 -- 172.21.15.32:0/2885821000 >> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] conn(0x5584bddd95f0 msgr2=0x5584bdd4d410 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:45.925 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.923+0000 7ff7b951df40 1 --2- 172.21.15.32:0/2885821000 >> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] conn(0x5584bddd95f0 0x5584bdd4d410 crc :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:45.925 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.924+0000 7ff7b951df40 1 -- 172.21.15.32:0/2885821000 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7ff7940759f0 msgr2=0x7ff794077f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:45.926 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.924+0000 7ff7b951df40 1 --2- 172.21.15.32:0/2885821000 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7ff7940759f0 0x7ff794077f00 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x5584bdde3780 tx=0x7ff7a400f040 comp rx=0 tx=0).stop 2024-08-03T20:54:45.926 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.924+0000 7ff7b951df40 1 -- 172.21.15.32:0/2885821000 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5584bdde0cb0 msgr2=0x5584bdddc1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:45.926 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.924+0000 7ff7b951df40 1 --2- 172.21.15.32:0/2885821000 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5584bdde0cb0 0x5584bdddc1c0 secure :-1 s=READY pgs=1084 cs=0 l=1 rev1=1 crypto rx=0x7ff7a000c310 tx=0x7ff7a000c7d0 comp rx=0 tx=0).stop 2024-08-03T20:54:45.926 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.924+0000 7ff7b2600640 1 -- 172.21.15.32:0/2885821000 reap_dead start 2024-08-03T20:54:45.926 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.924+0000 7ff7b951df40 1 -- 172.21.15.32:0/2885821000 shutdown_connections 2024-08-03T20:54:45.926 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.924+0000 7ff7b951df40 1 -- 172.21.15.32:0/2885821000 >> 172.21.15.32:0/2885821000 conn(0x5584bdc2cac0 msgr2=0x5584bddc2010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:54:45.926 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.924+0000 7ff7b951df40 1 -- 172.21.15.32:0/2885821000 shutdown_connections 2024-08-03T20:54:45.926 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:45.924+0000 7ff7b951df40 1 -- 172.21.15.32:0/2885821000 wait complete. 2024-08-03T20:54:45.936 INFO:tasks.cephfs.test_damage:Expectation on rm '10000000000.00000000' will be 'no damage' 2024-08-03T20:54:45.936 INFO:tasks.cephfs.test_damage:Expectation on rm '600.00000000' will be 'damaged_on_start' 2024-08-03T20:54:45.936 INFO:tasks.cephfs.test_damage:Expectation on rm '1.00000000.inode' will be 'damaged_on_start' 2024-08-03T20:54:45.936 INFO:tasks.cephfs.test_damage:Expectation on rm '200.00000000' will be 'damaged_on_start' 2024-08-03T20:54:45.936 INFO:tasks.cephfs.test_damage:Expectation on rm '200.00000001' will be 'damaged_on_start' 2024-08-03T20:54:45.936 INFO:tasks.cephfs.test_damage:Expectation on rm 'mds0_openfiles.0' will be 'no damage' 2024-08-03T20:54:45.936 INFO:tasks.cephfs.test_damage:Expectation on rm '500.00000000' will be 'no damage' 2024-08-03T20:54:45.936 INFO:tasks.cephfs.test_damage:Expectation on rm 'mds_snaptable' will be 'damaged_on_start' 2024-08-03T20:54:45.936 INFO:tasks.cephfs.test_damage:Expectation on rm 'mds0_inotable' will be 'damaged_on_start' 2024-08-03T20:54:45.936 INFO:tasks.cephfs.test_damage:Expectation on rm '100.00000000' will be 'damaged_on_start' 2024-08-03T20:54:45.936 INFO:tasks.cephfs.test_damage:Expectation on rm 'mds0_sessionmap' will be 'damaged_on_start' 2024-08-03T20:54:45.937 INFO:tasks.cephfs.test_damage:Expectation on rm '400.00000000' will be 'no damage' 2024-08-03T20:54:45.937 INFO:tasks.cephfs.test_damage:Expectation on rm '100.00000000.inode' will be 'damaged_on_start' 2024-08-03T20:54:45.937 INFO:tasks.cephfs.test_damage:Expectation on rm '1.00000000' will be 'damaged_on_start' 2024-08-03T20:54:45.937 INFO:tasks.cephfs.test_damage:Expectation on corrupt header '10000000000.00000000' will be 'no damage' 2024-08-03T20:54:45.937 INFO:tasks.cephfs.test_damage:Expectation on corrupt header '600.00000000' will be 'damaged_on_start' 2024-08-03T20:54:45.937 INFO:tasks.cephfs.test_damage:Expectation on corrupt header 'mds0_openfiles.0' will be 'no damage' 2024-08-03T20:54:45.937 INFO:tasks.cephfs.test_damage:Expectation on corrupt header '100.00000000' will be 'damaged_on_start' 2024-08-03T20:54:45.937 INFO:tasks.cephfs.test_damage:Expectation on corrupt header 'mds0_sessionmap' will be 'damaged_on_start' 2024-08-03T20:54:45.938 INFO:tasks.cephfs.test_damage:Expectation on corrupt header '1.00000000' will be 'damaged_on_start' 2024-08-03T20:54:45.938 INFO:tasks.cephfs.test_damage:Applying mutation 'Delete 10000000000.00000000' 2024-08-03T20:54:45.938 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-08-03T20:54:45.938 DEBUG:teuthology.orchestra.run.smithi032:> dd if=/proc/self/mounts of=/dev/stdout 2024-08-03T20:54:45.974 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053453,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6499208k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 none /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249604k,nr_inodes=812401,mode=700,uid=1000,gid=1316 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6499208k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-08-03T20:54:45.974 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi032.front.sepia.ceph.com 2024-08-03T20:54:45.974 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-08-03T20:54:45.974 DEBUG:teuthology.orchestra.run.smithi032:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-08-03T20:54:46.050 INFO:teuthology.orchestra.run.smithi032.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.0': No such file or directory 2024-08-03T20:54:46.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-03T20:54:46.051 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-08-03T20:54:46.320 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:46.319+0000 7f9bf8a00640 1 -- 172.21.15.32:0/892969238 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f9bf4144f90 msgr2=0x7f9bf4074570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:46.321 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:46.319+0000 7f9bf8a00640 1 --2- 172.21.15.32:0/892969238 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f9bf4144f90 0x7f9bf4074570 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f9bdc009830 tx=0x7f9bdc02a510 comp rx=0 tx=0).stop 2024-08-03T20:54:46.321 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:46.320+0000 7f9bf8a00640 1 -- 172.21.15.32:0/892969238 shutdown_connections 2024-08-03T20:54:46.321 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:46.320+0000 7f9bf8a00640 1 --2- 172.21.15.32:0/892969238 >> v2:172.21.15.118:3300/0 conn(0x7f9bf415fff0 0x7f9bf4160400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:46.321 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:46.320+0000 7f9bf8a00640 1 --2- 172.21.15.32:0/892969238 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f9bf4144f90 0x7f9bf4074570 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:46.321 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:46.320+0000 7f9bf8a00640 1 -- 172.21.15.32:0/892969238 >> 172.21.15.32:0/892969238 conn(0x7f9bf40b1bd0 msgr2=0x7f9bf40b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:54:46.321 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:46.320+0000 7f9bf8a00640 1 -- 172.21.15.32:0/892969238 shutdown_connections 2024-08-03T20:54:46.321 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:46.320+0000 7f9bf8a00640 1 -- 172.21.15.32:0/892969238 wait complete. 2024-08-03T20:54:46.322 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:46.322+0000 7f9bf8a00640 1 Processor -- start 2024-08-03T20:54:46.322 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:46.322+0000 7f9bf8a00640 1 -- start start 2024-08-03T20:54:46.325 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:46.323+0000 7f9bf8a00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f9bf406e930 0x7f9bf40ad8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:46.325 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:46.324+0000 7f9bf8a00640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f9bf4144f90 0x7f9bf41f8e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:46.325 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:46.324+0000 7f9bf8a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9bf415fff0 0x7f9bf415b640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:46.325 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:46.324+0000 7f9bf8a00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f9bf40cf840 con 0x7f9bf415fff0 2024-08-03T20:54:46.326 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:46.324+0000 7f9bf8a00640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f9bf4145f30 con 0x7f9bf406e930 2024-08-03T20:54:46.326 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:46.324+0000 7f9bf8a00640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7f9bf40cf9c0 con 0x7f9bf4144f90 2024-08-03T20:54:46.326 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:46.324+0000 7f9bf2800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9bf415fff0 0x7f9bf415b640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:46.326 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:46.324+0000 7f9bf2800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9bf415fff0 0x7f9bf415b640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:37512/0 (socket says 172.21.15.32:37512) 2024-08-03T20:54:46.326 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:46.324+0000 7f9bf2800640 1 -- 172.21.15.32:0/452622551 learned_addr learned my addr 172.21.15.32:0/452622551 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:54:46.326 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:46.324+0000 7f9bf1e00640 1 --2- 172.21.15.32:0/452622551 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f9bf406e930 0x7f9bf40ad8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:46.326 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:46.324+0000 7f9be9400640 1 --2- 172.21.15.32:0/452622551 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f9bf4144f90 0x7f9bf41f8e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:46.326 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:46.324+0000 7f9bf2800640 1 -- 172.21.15.32:0/452622551 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f9bf4144f90 msgr2=0x7f9bf41f8e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:46.326 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:46.324+0000 7f9bf2800640 1 --2- 172.21.15.32:0/452622551 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f9bf4144f90 0x7f9bf41f8e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:46.326 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:46.324+0000 7f9bf2800640 1 -- 172.21.15.32:0/452622551 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f9bf406e930 msgr2=0x7f9bf40ad8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:46.326 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:46.324+0000 7f9bf2800640 1 --2- 172.21.15.32:0/452622551 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f9bf406e930 0x7f9bf40ad8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:46.327 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:46.324+0000 7f9bf2800640 1 -- 172.21.15.32:0/452622551 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9bf415bb80 con 0x7f9bf415fff0 2024-08-03T20:54:46.327 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:46.324+0000 7f9be9400640 1 --2- 172.21.15.32:0/452622551 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f9bf4144f90 0x7f9bf41f8e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-03T20:54:46.327 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:46.325+0000 7f9bf2800640 1 --2- 172.21.15.32:0/452622551 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9bf415fff0 0x7f9bf415b640 secure :-1 s=READY pgs=1085 cs=0 l=1 rev1=1 crypto rx=0x7f9bec00ea70 tx=0x7f9bec00ef30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:54:46.327 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:46.325+0000 7f9beaa00640 1 -- 172.21.15.32:0/452622551 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9bec00ce40 con 0x7f9bf415fff0 2024-08-03T20:54:46.327 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:46.325+0000 7f9beaa00640 1 -- 172.21.15.32:0/452622551 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f9bec004c10 con 0x7f9bf415fff0 2024-08-03T20:54:46.327 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:46.325+0000 7f9bf8a00640 1 -- 172.21.15.32:0/452622551 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9bf40cba60 con 0x7f9bf415fff0 2024-08-03T20:54:46.327 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:46.325+0000 7f9beaa00640 1 -- 172.21.15.32:0/452622551 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9bec005020 con 0x7f9bf415fff0 2024-08-03T20:54:46.327 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:46.325+0000 7f9bf8a00640 1 -- 172.21.15.32:0/452622551 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9bf41f4f20 con 0x7f9bf415fff0 2024-08-03T20:54:46.327 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:46.326+0000 7f9bf8a00640 1 -- 172.21.15.32:0/452622551 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9bf4142260 con 0x7f9bf415fff0 2024-08-03T20:54:46.328 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:46.327+0000 7f9beaa00640 1 -- 172.21.15.32:0/452622551 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f9bec0051c0 con 0x7f9bf415fff0 2024-08-03T20:54:46.329 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:46.329+0000 7f9beaa00640 1 --2- 172.21.15.32:0/452622551 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f9bd0077010 0x7f9bd0079520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:46.329 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:46.329+0000 7f9beaa00640 1 -- 172.21.15.32:0/452622551 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 5795+0+0 (secure 0 0 0) 0x7f9bec099800 con 0x7f9bf415fff0 2024-08-03T20:54:46.330 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:46.329+0000 7f9bf1e00640 1 --2- 172.21.15.32:0/452622551 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f9bd0077010 0x7f9bd0079520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:46.330 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:46.330+0000 7f9bf1e00640 1 --2- 172.21.15.32:0/452622551 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f9bd0077010 0x7f9bd0079520 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f9be4007fd0 tx=0x7f9be4001370 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:54:46.334 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:46.334+0000 7f9beaa00640 1 -- 172.21.15.32:0/452622551 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7f9bec062880 con 0x7f9bf415fff0 2024-08-03T20:54:46.579 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:46.579+0000 7f9bf8a00640 1 -- 172.21.15.32:0/452622551 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7f9bf41f4d30 con 0x7f9bf415fff0 2024-08-03T20:54:47.513 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:47.512+0000 7f9beaa00640 1 -- 172.21.15.32:0/452622551 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v94) ==== 161+0+0 (secure 0 0 0) 0x7f9bec062180 con 0x7f9bf415fff0 2024-08-03T20:54:47.513 INFO:teuthology.orchestra.run.smithi032.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-08-03T20:54:47.517 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:47.516+0000 7f9bf8a00640 1 -- 172.21.15.32:0/452622551 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f9bd0077010 msgr2=0x7f9bd0079520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:47.517 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:47.516+0000 7f9bf8a00640 1 --2- 172.21.15.32:0/452622551 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f9bd0077010 0x7f9bd0079520 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f9be4007fd0 tx=0x7f9be4001370 comp rx=0 tx=0).stop 2024-08-03T20:54:47.517 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:47.516+0000 7f9bf8a00640 1 -- 172.21.15.32:0/452622551 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9bf415fff0 msgr2=0x7f9bf415b640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:47.517 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:47.516+0000 7f9bf8a00640 1 --2- 172.21.15.32:0/452622551 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9bf415fff0 0x7f9bf415b640 secure :-1 s=READY pgs=1085 cs=0 l=1 rev1=1 crypto rx=0x7f9bec00ea70 tx=0x7f9bec00ef30 comp rx=0 tx=0).stop 2024-08-03T20:54:47.517 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:47.517+0000 7f9bf8a00640 1 -- 172.21.15.32:0/452622551 shutdown_connections 2024-08-03T20:54:47.518 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:47.517+0000 7f9bf8a00640 1 --2- 172.21.15.32:0/452622551 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f9bd0077010 0x7f9bd0079520 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:47.518 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:47.517+0000 7f9bf8a00640 1 --2- 172.21.15.32:0/452622551 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9bf415fff0 0x7f9bf415b640 unknown :-1 s=CLOSED pgs=1085 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:47.518 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:47.517+0000 7f9bf8a00640 1 --2- 172.21.15.32:0/452622551 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f9bf4144f90 0x7f9bf41f8e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:47.518 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:47.517+0000 7f9bf8a00640 1 --2- 172.21.15.32:0/452622551 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f9bf406e930 0x7f9bf40ad8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:47.518 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:47.517+0000 7f9bf8a00640 1 -- 172.21.15.32:0/452622551 >> 172.21.15.32:0/452622551 conn(0x7f9bf40b1bd0 msgr2=0x7f9bf40b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:54:47.518 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:47.517+0000 7f9bf8a00640 1 -- 172.21.15.32:0/452622551 shutdown_connections 2024-08-03T20:54:47.518 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:47.517+0000 7f9bf8a00640 1 -- 172.21.15.32:0/452622551 wait complete. 2024-08-03T20:54:47.546 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mds repaired 0 2024-08-03T20:54:47.779 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:47.778+0000 7f6cd6c00640 1 -- 172.21.15.32:0/466785036 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6cd006e100 msgr2=0x7f6cd015b1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:47.779 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:47.778+0000 7f6cd6c00640 1 --2- 172.21.15.32:0/466785036 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6cd006e100 0x7f6cd015b1b0 secure :-1 s=READY pgs=1087 cs=0 l=1 rev1=1 crypto rx=0x7f6cc8005fd0 tx=0x7f6cc802a930 comp rx=0 tx=0).stop 2024-08-03T20:54:47.779 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:47.778+0000 7f6cd6c00640 1 -- 172.21.15.32:0/466785036 shutdown_connections 2024-08-03T20:54:47.779 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:47.778+0000 7f6cd6c00640 1 --2- 172.21.15.32:0/466785036 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6cd006e100 0x7f6cd015b1b0 unknown :-1 s=CLOSED pgs=1087 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:47.779 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:47.778+0000 7f6cd6c00640 1 -- 172.21.15.32:0/466785036 >> 172.21.15.32:0/466785036 conn(0x7f6cd0072e70 msgr2=0x7f6cd0073280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:54:47.780 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:47.778+0000 7f6cd6c00640 1 -- 172.21.15.32:0/466785036 shutdown_connections 2024-08-03T20:54:47.780 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:47.778+0000 7f6cd6c00640 1 -- 172.21.15.32:0/466785036 wait complete. 2024-08-03T20:54:47.781 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:47.781+0000 7f6cd6c00640 1 Processor -- start 2024-08-03T20:54:47.781 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:47.781+0000 7f6cd6c00640 1 -- start start 2024-08-03T20:54:47.783 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:47.782+0000 7f6cd6c00640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f6cd0073e30 0x7f6cd01f92c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:47.784 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:47.782+0000 7f6cd6c00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6cd00ce050 0x7f6cd00cf7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:47.784 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:47.782+0000 7f6cd6c00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f6cd00cfd10 0x7f6cd0143700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:47.784 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:47.782+0000 7f6cd6c00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f6cd01666a0 con 0x7f6cd00ce050 2024-08-03T20:54:47.784 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:47.782+0000 7f6cd6c00640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f6cd0166520 con 0x7f6cd00cfd10 2024-08-03T20:54:47.784 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:47.782+0000 7f6cd6c00640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7f6cd0166820 con 0x7f6cd0073e30 2024-08-03T20:54:47.784 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:47.783+0000 7f6cd4e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6cd00ce050 0x7f6cd00cf7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:47.784 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:47.783+0000 7f6cd5800640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f6cd0073e30 0x7f6cd01f92c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:47.784 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:47.783+0000 7f6cd4e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6cd00ce050 0x7f6cd00cf7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:37530/0 (socket says 172.21.15.32:37530) 2024-08-03T20:54:47.784 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:47.783+0000 7f6cd4e00640 1 -- 172.21.15.32:0/4223127457 learned_addr learned my addr 172.21.15.32:0/4223127457 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:54:47.784 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:47.783+0000 7f6cd6200640 1 --2- 172.21.15.32:0/4223127457 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f6cd00cfd10 0x7f6cd0143700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:47.784 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:47.783+0000 7f6cd4e00640 1 -- 172.21.15.32:0/4223127457 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f6cd0073e30 msgr2=0x7f6cd01f92c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:47.784 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:47.783+0000 7f6cd4e00640 1 --2- 172.21.15.32:0/4223127457 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f6cd0073e30 0x7f6cd01f92c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:47.785 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:47.783+0000 7f6cd4e00640 1 -- 172.21.15.32:0/4223127457 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f6cd00cfd10 msgr2=0x7f6cd0143700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:47.785 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:47.783+0000 7f6cd4e00640 1 --2- 172.21.15.32:0/4223127457 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f6cd00cfd10 0x7f6cd0143700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:47.785 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:47.783+0000 7f6cd4e00640 1 -- 172.21.15.32:0/4223127457 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6cd01f9800 con 0x7f6cd00ce050 2024-08-03T20:54:47.786 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:47.784+0000 7f6cd4e00640 1 --2- 172.21.15.32:0/4223127457 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6cd00ce050 0x7f6cd00cf7d0 secure :-1 s=READY pgs=1088 cs=0 l=1 rev1=1 crypto rx=0x7f6cbc00c370 tx=0x7f6cbc00c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:54:47.786 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:47.784+0000 7f6ccd600640 1 -- 172.21.15.32:0/4223127457 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6cbc017070 con 0x7f6cd00ce050 2024-08-03T20:54:47.786 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:47.784+0000 7f6ccd600640 1 -- 172.21.15.32:0/4223127457 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f6cbc0053c0 con 0x7f6cd00ce050 2024-08-03T20:54:47.786 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:47.784+0000 7f6cd6c00640 1 -- 172.21.15.32:0/4223127457 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6cd00b0cf0 con 0x7f6cd00ce050 2024-08-03T20:54:47.786 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:47.784+0000 7f6ccd600640 1 -- 172.21.15.32:0/4223127457 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6cbc003850 con 0x7f6cd00ce050 2024-08-03T20:54:47.786 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:47.784+0000 7f6cd6c00640 1 -- 172.21.15.32:0/4223127457 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6cd00b12b0 con 0x7f6cd00ce050 2024-08-03T20:54:47.786 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:47.785+0000 7f6cd6c00640 1 -- 172.21.15.32:0/4223127457 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6cd015b1b0 con 0x7f6cd00ce050 2024-08-03T20:54:47.787 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:47.786+0000 7f6ccd600640 1 -- 172.21.15.32:0/4223127457 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f6cbc01a070 con 0x7f6cd00ce050 2024-08-03T20:54:47.788 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:47.787+0000 7f6ccd600640 1 --2- 172.21.15.32:0/4223127457 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f6cb0077010 0x7f6cb0079520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:47.788 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:47.787+0000 7f6ccd600640 1 -- 172.21.15.32:0/4223127457 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 5795+0+0 (secure 0 0 0) 0x7f6cbc097d40 con 0x7f6cd00ce050 2024-08-03T20:54:47.788 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:47.788+0000 7f6cd5800640 1 --2- 172.21.15.32:0/4223127457 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f6cb0077010 0x7f6cb0079520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:47.789 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:47.789+0000 7f6cd5800640 1 --2- 172.21.15.32:0/4223127457 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f6cb0077010 0x7f6cb0079520 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f6cc40046b0 tx=0x7f6cc4007e90 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:54:47.792 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:47.792+0000 7f6ccd600640 1 -- 172.21.15.32:0/4223127457 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7f6cd01f9800 con 0x7f6cd00ce050 2024-08-03T20:54:48.073 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:48.073+0000 7f6cd6c00640 1 -- 172.21.15.32:0/4223127457 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "mds repaired", "role": "0"} v 0) -- 0x7f6cd006dd30 con 0x7f6cd00ce050 2024-08-03T20:54:48.519 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:48.518+0000 7f6ccd600640 1 -- 172.21.15.32:0/4223127457 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "mds repaired", "role": "0"}]=0 nothing to do: rank is not damaged v95) ==== 107+0+0 (secure 0 0 0) 0x7f6cbc0691b0 con 0x7f6cd00ce050 2024-08-03T20:54:48.519 INFO:teuthology.orchestra.run.smithi032.stderr:nothing to do: rank is not damaged 2024-08-03T20:54:48.522 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:48.521+0000 7f6cd6c00640 1 -- 172.21.15.32:0/4223127457 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f6cb0077010 msgr2=0x7f6cb0079520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:48.522 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:48.521+0000 7f6cd6c00640 1 --2- 172.21.15.32:0/4223127457 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f6cb0077010 0x7f6cb0079520 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f6cc40046b0 tx=0x7f6cc4007e90 comp rx=0 tx=0).stop 2024-08-03T20:54:48.522 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:48.521+0000 7f6cd6c00640 1 -- 172.21.15.32:0/4223127457 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6cd00ce050 msgr2=0x7f6cd00cf7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:48.522 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:48.521+0000 7f6cd6c00640 1 --2- 172.21.15.32:0/4223127457 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6cd00ce050 0x7f6cd00cf7d0 secure :-1 s=READY pgs=1088 cs=0 l=1 rev1=1 crypto rx=0x7f6cbc00c370 tx=0x7f6cbc00c830 comp rx=0 tx=0).stop 2024-08-03T20:54:48.522 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:48.521+0000 7f6cd6c00640 1 -- 172.21.15.32:0/4223127457 shutdown_connections 2024-08-03T20:54:48.522 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:48.521+0000 7f6cd6c00640 1 --2- 172.21.15.32:0/4223127457 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f6cb0077010 0x7f6cb0079520 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:48.522 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:48.521+0000 7f6cd6c00640 1 --2- 172.21.15.32:0/4223127457 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f6cd00cfd10 0x7f6cd0143700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:48.522 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:48.521+0000 7f6cd6c00640 1 --2- 172.21.15.32:0/4223127457 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6cd00ce050 0x7f6cd00cf7d0 unknown :-1 s=CLOSED pgs=1088 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:48.522 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:48.521+0000 7f6cd6c00640 1 --2- 172.21.15.32:0/4223127457 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f6cd0073e30 0x7f6cd01f92c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:48.522 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:48.521+0000 7f6cd6c00640 1 -- 172.21.15.32:0/4223127457 >> 172.21.15.32:0/4223127457 conn(0x7f6cd0072e70 msgr2=0x7f6cd006cae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:54:48.523 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:48.521+0000 7f6cd6c00640 1 -- 172.21.15.32:0/4223127457 shutdown_connections 2024-08-03T20:54:48.523 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:48.521+0000 7f6cd6c00640 1 -- 172.21.15.32:0/4223127457 wait complete. 2024-08-03T20:54:48.546 DEBUG:teuthology.orchestra.run.smithi032:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata import - 2024-08-03T20:54:48.691 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:48.690+0000 7f0904e30f40 1 Processor -- start 2024-08-03T20:54:48.691 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:48.690+0000 7f0904e30f40 1 -- start start 2024-08-03T20:54:48.693 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:48.691+0000 7f0904e30f40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x557d0a01cb40 0x557d0a0ac640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:48.693 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:48.691+0000 7f0904e30f40 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x557d0a08fbe0 0x557d09fd57c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:48.693 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:48.692+0000 7f0904e30f40 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x557d0a0b0090 0x557d09fd5d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:48.693 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:48.692+0000 7f0904e30f40 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x557d09f25e30 con 0x557d0a08fbe0 2024-08-03T20:54:48.694 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:48.692+0000 7f0904e30f40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x557d09f83bf0 con 0x557d0a01cb40 2024-08-03T20:54:48.694 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:48.692+0000 7f0904e30f40 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x557d0a090510 con 0x557d0a0b0090 2024-08-03T20:54:48.694 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:48.692+0000 7f08fd600640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x557d0a08fbe0 0x557d09fd57c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:48.694 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:48.692+0000 7f08fd600640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x557d0a08fbe0 0x557d09fd57c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:37538/0 (socket says 172.21.15.32:37538) 2024-08-03T20:54:48.694 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:48.692+0000 7f08fd600640 1 -- 172.21.15.32:0/3286445835 learned_addr learned my addr 172.21.15.32:0/3286445835 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:54:48.694 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:48.692+0000 7f08fe000640 1 --2- 172.21.15.32:0/3286445835 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x557d0a01cb40 0x557d0a0ac640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:48.694 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:48.692+0000 7f08fea00640 1 --2- 172.21.15.32:0/3286445835 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x557d0a0b0090 0x557d09fd5d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:48.694 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:48.692+0000 7f08fd600640 1 -- 172.21.15.32:0/3286445835 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x557d0a0b0090 msgr2=0x557d09fd5d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:54:48.694 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:48.692+0000 7f08fd600640 1 --2- 172.21.15.32:0/3286445835 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x557d0a0b0090 0x557d09fd5d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:48.694 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:48.692+0000 7f08fd600640 1 -- 172.21.15.32:0/3286445835 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x557d0a01cb40 msgr2=0x557d0a0ac640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:54:48.694 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:48.692+0000 7f08fd600640 1 --2- 172.21.15.32:0/3286445835 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x557d0a01cb40 0x557d0a0ac640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:48.694 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:48.692+0000 7f08fd600640 1 -- 172.21.15.32:0/3286445835 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557d0a0ab270 con 0x557d0a08fbe0 2024-08-03T20:54:48.695 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:48.692+0000 7f08fd600640 1 --2- 172.21.15.32:0/3286445835 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x557d0a08fbe0 0x557d09fd57c0 secure :-1 s=READY pgs=1091 cs=0 l=1 rev1=1 crypto rx=0x7f08ec004e90 tx=0x7f08ec002fb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e9bb6e9af9cb0bb1 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:54:48.695 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:48.693+0000 7f08f7e00640 1 -- 172.21.15.32:0/3286445835 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08ec04d8c0 con 0x557d0a08fbe0 2024-08-03T20:54:48.695 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:48.693+0000 7f08f7e00640 1 -- 172.21.15.32:0/3286445835 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f08ec04dd70 con 0x557d0a08fbe0 2024-08-03T20:54:48.695 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:48.693+0000 7f08f7e00640 1 -- 172.21.15.32:0/3286445835 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08ec0566d0 con 0x557d0a08fbe0 2024-08-03T20:54:48.696 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:48.693+0000 7f0904e30f40 1 -- 172.21.15.32:0/3286445835 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x557d0a08fbe0 msgr2=0x557d09fd57c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:48.696 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:48.693+0000 7f0904e30f40 1 --2- 172.21.15.32:0/3286445835 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x557d0a08fbe0 0x557d09fd57c0 secure :-1 s=READY pgs=1091 cs=0 l=1 rev1=1 crypto rx=0x7f08ec004e90 tx=0x7f08ec002fb0 comp rx=0 tx=0).stop 2024-08-03T20:54:48.696 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:48.693+0000 7f0904e30f40 1 -- 172.21.15.32:0/3286445835 shutdown_connections 2024-08-03T20:54:48.696 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:48.693+0000 7f0904e30f40 1 --2- 172.21.15.32:0/3286445835 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x557d0a0b0090 0x557d09fd5d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:48.696 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:48.693+0000 7f0904e30f40 1 --2- 172.21.15.32:0/3286445835 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x557d0a08fbe0 0x557d09fd57c0 unknown :-1 s=CLOSED pgs=1091 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:48.696 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:48.693+0000 7f0904e30f40 1 --2- 172.21.15.32:0/3286445835 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x557d0a01cb40 0x557d0a0ac640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:48.696 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:48.693+0000 7f0904e30f40 1 -- 172.21.15.32:0/3286445835 >> 172.21.15.32:0/3286445835 conn(0x557d09ef7ac0 msgr2=0x557d0a090820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:54:48.696 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:48.694+0000 7f0904e30f40 1 -- 172.21.15.32:0/3286445835 shutdown_connections 2024-08-03T20:54:48.697 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:48.694+0000 7f0904e30f40 1 -- 172.21.15.32:0/3286445835 wait complete. 2024-08-03T20:54:48.697 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:48.695+0000 7f0904e30f40 1 Processor -- start 2024-08-03T20:54:48.697 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:48.695+0000 7f0904e30f40 1 -- start start 2024-08-03T20:54:48.698 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:48.696+0000 7f0904e30f40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x557d0a01cb40 0x557d0a13b020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:48.698 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:48.696+0000 7f0904e30f40 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x557d0a08fbe0 0x557d0a143c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:48.698 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:48.696+0000 7f0904e30f40 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x557d0a0b0090 0x557d0a01e960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:48.699 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:48.696+0000 7f0904e30f40 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x557d0a090f30 con 0x557d0a08fbe0 2024-08-03T20:54:48.699 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:48.696+0000 7f0904e30f40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x557d0a0913e0 con 0x557d0a01cb40 2024-08-03T20:54:48.699 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:48.696+0000 7f0904e30f40 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x557d0a01fca0 con 0x557d0a0b0090 2024-08-03T20:54:48.699 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:48.696+0000 7f08fd600640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x557d0a08fbe0 0x557d0a143c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:48.699 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:48.696+0000 7f08fd600640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x557d0a08fbe0 0x557d0a143c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:37554/0 (socket says 172.21.15.32:37554) 2024-08-03T20:54:48.699 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:48.696+0000 7f08fd600640 1 -- 172.21.15.32:0/3693760922 learned_addr learned my addr 172.21.15.32:0/3693760922 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:54:48.699 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:48.696+0000 7f08fe000640 1 --2- 172.21.15.32:0/3693760922 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x557d0a01cb40 0x557d0a13b020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:48.699 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:48.697+0000 7f08fea00640 1 --2- 172.21.15.32:0/3693760922 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x557d0a0b0090 0x557d0a01e960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:48.699 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:48.697+0000 7f08fd600640 1 -- 172.21.15.32:0/3693760922 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x557d0a0b0090 msgr2=0x557d0a01e960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:48.699 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:48.697+0000 7f08fd600640 1 --2- 172.21.15.32:0/3693760922 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x557d0a0b0090 0x557d0a01e960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:48.699 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:48.697+0000 7f08fd600640 1 -- 172.21.15.32:0/3693760922 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x557d0a01cb40 msgr2=0x557d0a13b020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:48.699 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:48.697+0000 7f08fd600640 1 --2- 172.21.15.32:0/3693760922 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x557d0a01cb40 0x557d0a13b020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:48.699 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:48.697+0000 7f08fd600640 1 -- 172.21.15.32:0/3693760922 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557d0a0867e0 con 0x557d0a08fbe0 2024-08-03T20:54:48.700 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:48.697+0000 7f08fd600640 1 --2- 172.21.15.32:0/3693760922 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x557d0a08fbe0 0x557d0a143c90 secure :-1 s=READY pgs=1092 cs=0 l=1 rev1=1 crypto rx=0x7f08ec025e30 tx=0x7f08ec0575e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:54:48.700 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:48.698+0000 7f08f6a00640 1 -- 172.21.15.32:0/3693760922 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08ec06f030 con 0x557d0a08fbe0 2024-08-03T20:54:48.700 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:48.698+0000 7f08f6a00640 1 -- 172.21.15.32:0/3693760922 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f08ec057880 con 0x557d0a08fbe0 2024-08-03T20:54:48.700 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:48.698+0000 7f08f6a00640 1 -- 172.21.15.32:0/3693760922 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08ec057ad0 con 0x557d0a08fbe0 2024-08-03T20:54:48.700 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:48.698+0000 7f0904e30f40 1 -- 172.21.15.32:0/3693760922 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x557d0a086ac0 con 0x557d0a08fbe0 2024-08-03T20:54:48.700 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:48.698+0000 7f0904e30f40 1 -- 172.21.15.32:0/3693760922 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557d0a01e440 con 0x557d0a08fbe0 2024-08-03T20:54:48.700 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:48.700+0000 7f08f6a00640 1 -- 172.21.15.32:0/3693760922 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f08ec072070 con 0x557d0a08fbe0 2024-08-03T20:54:48.701 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:48.701+0000 7f08f6a00640 1 --2- 172.21.15.32:0/3693760922 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f08e00759f0 0x7f08e0077f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:48.701 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:48.701+0000 7f08f6a00640 1 -- 172.21.15.32:0/3693760922 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 5795+0+0 (secure 0 0 0) 0x7f08ec0dc9f0 con 0x557d0a08fbe0 2024-08-03T20:54:48.702 INFO:teuthology.orchestra.run.smithi032.stdout:Importing pool 2024-08-03T20:54:48.702 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:48.701+0000 7f08fe000640 1 --2- 172.21.15.32:0/3693760922 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f08e00759f0 0x7f08e0077f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:48.702 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:48.701+0000 7f0904e30f40 1 --2- 172.21.15.32:0/3693760922 >> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] conn(0x557d0a089370 0x557d0a089780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:48.702 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:48.701+0000 7f0904e30f40 1 -- 172.21.15.32:0/3693760922 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:1 12.10 12:09401a81:::601.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e84) -- 0x557d0a1441d0 con 0x557d0a089370 2024-08-03T20:54:48.703 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:48.702+0000 7f08fea00640 1 --2- 172.21.15.32:0/3693760922 >> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] conn(0x557d0a089370 0x557d0a089780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:48.703 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:48.702+0000 7f08fe000640 1 --2- 172.21.15.32:0/3693760922 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f08e00759f0 0x7f08e0077f00 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f08f000c740 tx=0x7f08f0004880 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:54:48.703 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:48.702+0000 7f08fea00640 1 --2- 172.21.15.32:0/3693760922 >> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] conn(0x557d0a089370 0x557d0a089780 crc :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:54:48.705 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:48.704+0000 7f08fea00640 1 -- 172.21.15.32:0/3693760922 <== osd.0 v2:172.21.15.32:6816/3066156116 1 ==== osd_op_reply(1 601.00000000 [create] v84'11 uv6 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f08f8001770 con 0x557d0a089370 2024-08-03T20:54:48.705 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:48.704+0000 7f0904e30f40 1 -- 172.21.15.32:0/3693760922 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:2 12.10 12:09401a81:::601.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e84) -- 0x557d0a089f80 con 0x557d0a089370 2024-08-03T20:54:48.706 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:48.706+0000 7f08fea00640 1 -- 172.21.15.32:0/3693760922 <== osd.0 v2:172.21.15.32:6816/3066156116 2 ==== osd_op_reply(2 601.00000000 [delete] v84'12 uv12 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f08f8001770 con 0x557d0a089370 2024-08-03T20:54:48.706 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:48.706+0000 7f0904e30f40 1 -- 172.21.15.32:0/3693760922 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:3 12.10 12:09401a81:::601.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e84) -- 0x557d0a150ac0 con 0x557d0a089370 2024-08-03T20:54:48.708 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::601.00000000:head# 2024-08-03T20:54:48.708 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:48.707+0000 7f08fea00640 1 -- 172.21.15.32:0/3693760922 <== osd.0 v2:172.21.15.32:6816/3066156116 3 ==== osd_op_reply(3 601.00000000 [create] v84'13 uv13 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f08f8001770 con 0x557d0a089370 2024-08-03T20:54:48.708 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:48.707+0000 7f0904e30f40 1 -- 172.21.15.32:0/3693760922 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:4 12.10 12:09401a81:::601.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e84) -- 0x557d0a151e70 con 0x557d0a089370 2024-08-03T20:54:48.709 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:48.709+0000 7f08fea00640 1 -- 172.21.15.32:0/3693760922 <== osd.0 v2:172.21.15.32:6816/3066156116 4 ==== osd_op_reply(4 601.00000000 [setxattr (30)] v84'14 uv14 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f08f8001770 con 0x557d0a089370 2024-08-03T20:54:48.709 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:48.709+0000 7f0904e30f40 1 -- 172.21.15.32:0/3693760922 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:5 12.10 12:09401a81:::601.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e84) -- 0x557d0a152210 con 0x557d0a089370 2024-08-03T20:54:48.711 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:48.710+0000 7f08fea00640 1 -- 172.21.15.32:0/3693760922 <== osd.0 v2:172.21.15.32:6816/3066156116 5 ==== osd_op_reply(5 601.00000000 [setxattr (62)] v84'15 uv15 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f08f8001770 con 0x557d0a089370 2024-08-03T20:54:48.711 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:48.710+0000 7f0904e30f40 1 -- 172.21.15.32:0/3693760922 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:6 12.10 12:09401a81:::601.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e84) -- 0x557d0a150e60 con 0x557d0a089370 2024-08-03T20:54:48.712 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:48.712+0000 7f08fea00640 1 -- 172.21.15.32:0/3693760922 <== osd.0 v2:172.21.15.32:6816/3066156116 6 ==== osd_op_reply(6 601.00000000 [omap-set-header] v84'16 uv16 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f08f8001770 con 0x557d0a089370 2024-08-03T20:54:48.712 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:48.712+0000 7f0904e30f40 1 -- 172.21.15.32:0/3693760922 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:7 12.10 12:0bd6d154:::602.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e84) -- 0x557d0a151250 con 0x557d0a089370 2024-08-03T20:54:48.714 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:48.713+0000 7f08fea00640 1 -- 172.21.15.32:0/3693760922 <== osd.0 v2:172.21.15.32:6816/3066156116 7 ==== osd_op_reply(7 602.00000000 [create] v84'17 uv7 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f08f8001770 con 0x557d0a089370 2024-08-03T20:54:48.714 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:48.713+0000 7f0904e30f40 1 -- 172.21.15.32:0/3693760922 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:8 12.10 12:0bd6d154:::602.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e84) -- 0x557d0a151640 con 0x557d0a089370 2024-08-03T20:54:48.715 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:48.715+0000 7f08fea00640 1 -- 172.21.15.32:0/3693760922 <== osd.0 v2:172.21.15.32:6816/3066156116 8 ==== osd_op_reply(8 602.00000000 [delete] v84'18 uv18 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f08f8001770 con 0x557d0a089370 2024-08-03T20:54:48.715 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:48.715+0000 7f0904e30f40 1 -- 172.21.15.32:0/3693760922 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:9 12.10 12:0bd6d154:::602.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e84) -- 0x557d0a151640 con 0x557d0a089370 2024-08-03T20:54:48.717 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:48.716+0000 7f08fea00640 1 -- 172.21.15.32:0/3693760922 <== osd.0 v2:172.21.15.32:6816/3066156116 9 ==== osd_op_reply(9 602.00000000 [create] v84'19 uv19 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f08f8001770 con 0x557d0a089370 2024-08-03T20:54:48.717 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::602.00000000:head# 2024-08-03T20:54:48.717 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:48.717+0000 7f0904e30f40 1 -- 172.21.15.32:0/3693760922 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:10 12.10 12:0bd6d154:::602.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e84) -- 0x557d0a1515f0 con 0x557d0a089370 2024-08-03T20:54:48.719 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:48.718+0000 7f08fea00640 1 -- 172.21.15.32:0/3693760922 <== osd.0 v2:172.21.15.32:6816/3066156116 10 ==== osd_op_reply(10 602.00000000 [setxattr (30)] v84'20 uv20 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f08f8001770 con 0x557d0a089370 2024-08-03T20:54:48.719 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:48.718+0000 7f0904e30f40 1 -- 172.21.15.32:0/3693760922 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:11 12.10 12:0bd6d154:::602.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e84) -- 0x557d0a1515f0 con 0x557d0a089370 2024-08-03T20:54:48.720 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:48.720+0000 7f08fea00640 1 -- 172.21.15.32:0/3693760922 <== osd.0 v2:172.21.15.32:6816/3066156116 11 ==== osd_op_reply(11 602.00000000 [setxattr (62)] v84'21 uv21 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f08f8001770 con 0x557d0a089370 2024-08-03T20:54:48.721 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:48.720+0000 7f0904e30f40 1 -- 172.21.15.32:0/3693760922 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:12 12.10 12:0bd6d154:::602.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e84) -- 0x557d0a1515f0 con 0x557d0a089370 2024-08-03T20:54:48.722 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:48.722+0000 7f08fea00640 1 -- 172.21.15.32:0/3693760922 <== osd.0 v2:172.21.15.32:6816/3066156116 12 ==== osd_op_reply(12 602.00000000 [omap-set-header] v84'22 uv22 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f08f8001770 con 0x557d0a089370 2024-08-03T20:54:48.722 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:48.722+0000 7f0904e30f40 1 -- 172.21.15.32:0/3693760922 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:13 12.10 12:0cf6ad0f:::10000000000.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e84) -- 0x557d0a1515f0 con 0x557d0a089370 2024-08-03T20:54:48.724 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:48.723+0000 7f08fea00640 1 -- 172.21.15.32:0/3693760922 <== osd.0 v2:172.21.15.32:6816/3066156116 13 ==== osd_op_reply(13 10000000000.00000000 [create] v84'23 uv10 ondisk = -17 ((17) File exists)) ==== 164+0+0 (crc 0 0 0) 0x7f08f8001770 con 0x557d0a089370 2024-08-03T20:54:48.724 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:48.723+0000 7f0904e30f40 1 -- 172.21.15.32:0/3693760922 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:14 12.10 12:0cf6ad0f:::10000000000.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e84) -- 0x557d0a1515f0 con 0x557d0a089370 2024-08-03T20:54:48.725 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:48.725+0000 7f08fea00640 1 -- 172.21.15.32:0/3693760922 <== osd.0 v2:172.21.15.32:6816/3066156116 14 ==== osd_op_reply(14 10000000000.00000000 [delete] v84'24 uv24 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f08f8001770 con 0x557d0a089370 2024-08-03T20:54:48.725 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:48.725+0000 7f0904e30f40 1 -- 172.21.15.32:0/3693760922 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:15 12.10 12:0cf6ad0f:::10000000000.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e84) -- 0x557d0a1515f0 con 0x557d0a089370 2024-08-03T20:54:48.727 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:48.726+0000 7f08fea00640 1 -- 172.21.15.32:0/3693760922 <== osd.0 v2:172.21.15.32:6816/3066156116 15 ==== osd_op_reply(15 10000000000.00000000 [create] v84'25 uv25 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f08f8001770 con 0x557d0a089370 2024-08-03T20:54:48.727 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::10000000000.00000000:head# 2024-08-03T20:54:48.727 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:48.726+0000 7f0904e30f40 1 -- 172.21.15.32:0/3693760922 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:16 12.10 12:0cf6ad0f:::10000000000.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e84) -- 0x557d0a1515f0 con 0x557d0a089370 2024-08-03T20:54:48.728 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:48.728+0000 7f08fea00640 1 -- 172.21.15.32:0/3693760922 <== osd.0 v2:172.21.15.32:6816/3066156116 16 ==== osd_op_reply(16 10000000000.00000000 [setxattr (30)] v84'26 uv26 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f08f8001770 con 0x557d0a089370 2024-08-03T20:54:48.728 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:48.728+0000 7f0904e30f40 1 -- 172.21.15.32:0/3693760922 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:17 12.10 12:0cf6ad0f:::10000000000.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e84) -- 0x557d0a1515f0 con 0x557d0a089370 2024-08-03T20:54:48.730 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:48.730+0000 7f08fea00640 1 -- 172.21.15.32:0/3693760922 <== osd.0 v2:172.21.15.32:6816/3066156116 17 ==== osd_op_reply(17 10000000000.00000000 [setxattr (62)] v84'27 uv27 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f08f8001770 con 0x557d0a089370 2024-08-03T20:54:48.730 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:48.730+0000 7f0904e30f40 1 -- 172.21.15.32:0/3693760922 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:18 12.10 12:0cf6ad0f:::10000000000.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e84) -- 0x557d0a1515f0 con 0x557d0a089370 2024-08-03T20:54:48.731 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:48.731+0000 7f08fea00640 1 -- 172.21.15.32:0/3693760922 <== osd.0 v2:172.21.15.32:6816/3066156116 18 ==== osd_op_reply(18 10000000000.00000000 [omap-set-header] v84'28 uv28 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f08f8001770 con 0x557d0a089370 2024-08-03T20:54:48.731 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:48.731+0000 7f0904e30f40 1 -- 172.21.15.32:0/3693760922 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:19 12.10 12:0cf6ad0f:::10000000000.00000000:head [omap-set-vals in=593b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e84) -- 0x557d0a1525b0 con 0x557d0a089370 2024-08-03T20:54:48.733 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:48.733+0000 7f08fea00640 1 -- 172.21.15.32:0/3693760922 <== osd.0 v2:172.21.15.32:6816/3066156116 19 ==== osd_op_reply(19 10000000000.00000000 [omap-set-vals] v84'29 uv29 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f08f8001770 con 0x557d0a089370 2024-08-03T20:54:48.733 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:48.733+0000 7f0904e30f40 1 -- 172.21.15.32:0/3693760922 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:20 12.10 12:0d82a743:::600.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e84) -- 0x557d0a1525b0 con 0x557d0a089370 2024-08-03T20:54:48.734 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:48.734+0000 7f08fea00640 1 -- 172.21.15.32:0/3693760922 <== osd.0 v2:172.21.15.32:6816/3066156116 20 ==== osd_op_reply(20 600.00000000 [create] v84'30 uv5 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f08f8001770 con 0x557d0a089370 2024-08-03T20:54:48.735 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:48.734+0000 7f0904e30f40 1 -- 172.21.15.32:0/3693760922 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:21 12.10 12:0d82a743:::600.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e84) -- 0x557d0a1525b0 con 0x557d0a089370 2024-08-03T20:54:48.736 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:48.736+0000 7f08fea00640 1 -- 172.21.15.32:0/3693760922 <== osd.0 v2:172.21.15.32:6816/3066156116 21 ==== osd_op_reply(21 600.00000000 [delete] v84'31 uv31 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f08f8001770 con 0x557d0a089370 2024-08-03T20:54:48.736 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:48.736+0000 7f0904e30f40 1 -- 172.21.15.32:0/3693760922 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:22 12.10 12:0d82a743:::600.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e84) -- 0x557d0a1525b0 con 0x557d0a089370 2024-08-03T20:54:48.737 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:48.737+0000 7f08fea00640 1 -- 172.21.15.32:0/3693760922 <== osd.0 v2:172.21.15.32:6816/3066156116 22 ==== osd_op_reply(22 600.00000000 [create] v84'32 uv32 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f08f8001770 con 0x557d0a089370 2024-08-03T20:54:48.738 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::600.00000000:head# 2024-08-03T20:54:48.738 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:48.737+0000 7f0904e30f40 1 -- 172.21.15.32:0/3693760922 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:23 12.10 12:0d82a743:::600.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e84) -- 0x557d0a1535c0 con 0x557d0a089370 2024-08-03T20:54:48.739 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:48.739+0000 7f08fea00640 1 -- 172.21.15.32:0/3693760922 <== osd.0 v2:172.21.15.32:6816/3066156116 23 ==== osd_op_reply(23 600.00000000 [setxattr (30)] v84'33 uv33 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f08f8001770 con 0x557d0a089370 2024-08-03T20:54:48.739 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:48.739+0000 7f0904e30f40 1 -- 172.21.15.32:0/3693760922 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:24 12.10 12:0d82a743:::600.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e84) -- 0x557d0a1535c0 con 0x557d0a089370 2024-08-03T20:54:48.740 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:48.740+0000 7f08fea00640 1 -- 172.21.15.32:0/3693760922 <== osd.0 v2:172.21.15.32:6816/3066156116 24 ==== osd_op_reply(24 600.00000000 [setxattr (62)] v84'34 uv34 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f08f8001770 con 0x557d0a089370 2024-08-03T20:54:48.740 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:48.740+0000 7f0904e30f40 1 -- 172.21.15.32:0/3693760922 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:25 12.10 12:0d82a743:::600.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e84) -- 0x557d0a1525b0 con 0x557d0a089370 2024-08-03T20:54:48.742 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:48.742+0000 7f08fea00640 1 -- 172.21.15.32:0/3693760922 <== osd.0 v2:172.21.15.32:6816/3066156116 25 ==== osd_op_reply(25 600.00000000 [omap-set-header] v84'35 uv35 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f08f8001770 con 0x557d0a089370 2024-08-03T20:54:48.742 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:48.742+0000 7f0904e30f40 1 -- 172.21.15.32:0/3693760922 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:26 12.10 12:0d89b25e:::603.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e84) -- 0x557d0a1525b0 con 0x557d0a089370 2024-08-03T20:54:48.743 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:48.743+0000 7f08fea00640 1 -- 172.21.15.32:0/3693760922 <== osd.0 v2:172.21.15.32:6816/3066156116 26 ==== osd_op_reply(26 603.00000000 [create] v84'36 uv8 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f08f8001770 con 0x557d0a089370 2024-08-03T20:54:48.744 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:48.743+0000 7f0904e30f40 1 -- 172.21.15.32:0/3693760922 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:27 12.10 12:0d89b25e:::603.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e84) -- 0x557d0a1525b0 con 0x557d0a089370 2024-08-03T20:54:48.745 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:48.744+0000 7f08fea00640 1 -- 172.21.15.32:0/3693760922 <== osd.0 v2:172.21.15.32:6816/3066156116 27 ==== osd_op_reply(27 603.00000000 [delete] v84'37 uv37 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f08f8001770 con 0x557d0a089370 2024-08-03T20:54:48.745 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:48.744+0000 7f0904e30f40 1 -- 172.21.15.32:0/3693760922 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:28 12.10 12:0d89b25e:::603.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e84) -- 0x557d0a1525b0 con 0x557d0a089370 2024-08-03T20:54:48.746 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:48.746+0000 7f08fea00640 1 -- 172.21.15.32:0/3693760922 <== osd.0 v2:172.21.15.32:6816/3066156116 28 ==== osd_op_reply(28 603.00000000 [create] v84'38 uv38 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f08f8001770 con 0x557d0a089370 2024-08-03T20:54:48.746 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::603.00000000:head# 2024-08-03T20:54:48.747 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:48.746+0000 7f0904e30f40 1 -- 172.21.15.32:0/3693760922 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:29 12.10 12:0d89b25e:::603.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e84) -- 0x557d0a1535c0 con 0x557d0a089370 2024-08-03T20:54:48.747 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:48.747+0000 7f08fea00640 1 -- 172.21.15.32:0/3693760922 <== osd.0 v2:172.21.15.32:6816/3066156116 29 ==== osd_op_reply(29 603.00000000 [setxattr (30)] v84'39 uv39 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f08f8001770 con 0x557d0a089370 2024-08-03T20:54:48.748 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:48.747+0000 7f0904e30f40 1 -- 172.21.15.32:0/3693760922 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:30 12.10 12:0d89b25e:::603.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e84) -- 0x557d0a1535c0 con 0x557d0a089370 2024-08-03T20:54:48.749 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:48.749+0000 7f08fea00640 1 -- 172.21.15.32:0/3693760922 <== osd.0 v2:172.21.15.32:6816/3066156116 30 ==== osd_op_reply(30 603.00000000 [setxattr (62)] v84'40 uv40 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f08f8001770 con 0x557d0a089370 2024-08-03T20:54:48.749 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:48.749+0000 7f0904e30f40 1 -- 172.21.15.32:0/3693760922 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:31 12.10 12:0d89b25e:::603.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e84) -- 0x557d0a1525b0 con 0x557d0a089370 2024-08-03T20:54:48.750 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:48.750+0000 7f08fea00640 1 -- 172.21.15.32:0/3693760922 <== osd.0 v2:172.21.15.32:6816/3066156116 31 ==== osd_op_reply(31 603.00000000 [omap-set-header] v84'41 uv41 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f08f8001770 con 0x557d0a089370 2024-08-03T20:54:48.751 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:48.750+0000 7f0904e30f40 1 --2- 172.21.15.32:0/3693760922 >> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] conn(0x557d0a152ac0 0x557d0a154ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:48.751 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:48.750+0000 7f0904e30f40 1 -- 172.21.15.32:0/3693760922 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:32 12.14 12:287034c4:::1.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e84) -- 0x557d0a155420 con 0x557d0a152ac0 2024-08-03T20:54:48.751 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:48.751+0000 7f08fe000640 1 --2- 172.21.15.32:0/3693760922 >> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] conn(0x557d0a152ac0 0x557d0a154ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:48.751 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:48.751+0000 7f08fe000640 1 --2- 172.21.15.32:0/3693760922 >> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] conn(0x557d0a152ac0 0x557d0a154ee0 crc :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:54:48.753 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:48.753+0000 7f08fe000640 1 -- 172.21.15.32:0/3693760922 <== osd.5 v2:172.21.15.118:6810/2724891770 1 ==== osd_op_reply(32 1.00000000.inode [create] v84'20 uv15 ondisk = -17 ((17) File exists)) ==== 160+0+0 (crc 0 0 0) 0x7f08f000a8a0 con 0x557d0a152ac0 2024-08-03T20:54:48.753 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:48.753+0000 7f0904e30f40 1 -- 172.21.15.32:0/3693760922 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:33 12.14 12:287034c4:::1.00000000.inode:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e84) -- 0x557d0a1557c0 con 0x557d0a152ac0 2024-08-03T20:54:48.755 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:48.755+0000 7f08fe000640 1 -- 172.21.15.32:0/3693760922 <== osd.5 v2:172.21.15.118:6810/2724891770 2 ==== osd_op_reply(33 1.00000000.inode [delete] v84'21 uv21 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f08f000a8a0 con 0x557d0a152ac0 2024-08-03T20:54:48.755 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:48.755+0000 7f0904e30f40 1 -- 172.21.15.32:0/3693760922 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:34 12.14 12:287034c4:::1.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e84) -- 0x557d0a155b60 con 0x557d0a152ac0 2024-08-03T20:54:48.757 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:48.756+0000 7f08fe000640 1 -- 172.21.15.32:0/3693760922 <== osd.5 v2:172.21.15.118:6810/2724891770 3 ==== osd_op_reply(34 1.00000000.inode [create] v84'22 uv22 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f08f000a8a0 con 0x557d0a152ac0 2024-08-03T20:54:48.757 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::1.00000000.inode:head# 2024-08-03T20:54:48.757 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:48.756+0000 7f0904e30f40 1 -- 172.21.15.32:0/3693760922 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:35 12.14 12:287034c4:::1.00000000.inode:head [write 0~582 in=582b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e84) -- 0x557d0a155f00 con 0x557d0a152ac0 2024-08-03T20:54:48.758 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:48.758+0000 7f08fe000640 1 -- 172.21.15.32:0/3693760922 <== osd.5 v2:172.21.15.118:6810/2724891770 4 ==== osd_op_reply(35 1.00000000.inode [write 0~582] v84'23 uv23 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f08f000a8a0 con 0x557d0a152ac0 2024-08-03T20:54:48.759 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:48.758+0000 7f0904e30f40 1 -- 172.21.15.32:0/3693760922 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:36 12.14 12:287034c4:::1.00000000.inode:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e84) -- 0x557d0a1562a0 con 0x557d0a152ac0 2024-08-03T20:54:48.760 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:48.760+0000 7f08fe000640 1 -- 172.21.15.32:0/3693760922 <== osd.5 v2:172.21.15.118:6810/2724891770 5 ==== osd_op_reply(36 1.00000000.inode [omap-set-header] v84'24 uv24 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f08f000a8a0 con 0x557d0a152ac0 2024-08-03T20:54:48.760 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:48.760+0000 7f0904e30f40 1 -- 172.21.15.32:0/3693760922 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:37 12.14 12:292cf221:::200.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e84) -- 0x557d0a156640 con 0x557d0a152ac0 2024-08-03T20:54:48.761 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:48.761+0000 7f08fe000640 1 -- 172.21.15.32:0/3693760922 <== osd.5 v2:172.21.15.118:6810/2724891770 6 ==== osd_op_reply(37 200.00000000 [create] v84'25 uv19 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f08f000a8a0 con 0x557d0a152ac0 2024-08-03T20:54:48.762 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:48.761+0000 7f0904e30f40 1 -- 172.21.15.32:0/3693760922 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:38 12.14 12:292cf221:::200.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e84) -- 0x557d0a1569e0 con 0x557d0a152ac0 2024-08-03T20:54:48.763 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:48.762+0000 7f08fe000640 1 -- 172.21.15.32:0/3693760922 <== osd.5 v2:172.21.15.118:6810/2724891770 7 ==== osd_op_reply(38 200.00000000 [delete] v84'26 uv26 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f08f000a8a0 con 0x557d0a152ac0 2024-08-03T20:54:48.763 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:48.762+0000 7f0904e30f40 1 -- 172.21.15.32:0/3693760922 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:39 12.14 12:292cf221:::200.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e84) -- 0x557d0a1569e0 con 0x557d0a152ac0 2024-08-03T20:54:48.764 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:48.764+0000 7f08fe000640 1 -- 172.21.15.32:0/3693760922 <== osd.5 v2:172.21.15.118:6810/2724891770 8 ==== osd_op_reply(39 200.00000000 [create] v84'27 uv27 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f08f000a8a0 con 0x557d0a152ac0 2024-08-03T20:54:48.764 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::200.00000000:head# 2024-08-03T20:54:48.765 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:48.764+0000 7f0904e30f40 1 -- 172.21.15.32:0/3693760922 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:40 12.14 12:292cf221:::200.00000000:head [write 0~90 in=90b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e84) -- 0x557d0a1569e0 con 0x557d0a152ac0 2024-08-03T20:54:48.766 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:48.766+0000 7f08fe000640 1 -- 172.21.15.32:0/3693760922 <== osd.5 v2:172.21.15.118:6810/2724891770 9 ==== osd_op_reply(40 200.00000000 [write 0~90] v84'28 uv28 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f08f000a8a0 con 0x557d0a152ac0 2024-08-03T20:54:48.766 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:48.766+0000 7f0904e30f40 1 -- 172.21.15.32:0/3693760922 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:41 12.14 12:292cf221:::200.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e84) -- 0x557d0a1569e0 con 0x557d0a152ac0 2024-08-03T20:54:48.768 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:48.767+0000 7f08fe000640 1 -- 172.21.15.32:0/3693760922 <== osd.5 v2:172.21.15.118:6810/2724891770 10 ==== osd_op_reply(41 200.00000000 [omap-set-header] v84'29 uv29 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f08f000a8a0 con 0x557d0a152ac0 2024-08-03T20:54:48.768 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:48.767+0000 7f0904e30f40 1 -- 172.21.15.32:0/3693760922 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:42 12.14 12:2e2fa760:::200.00000001:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e84) -- 0x557d0a1569e0 con 0x557d0a152ac0 2024-08-03T20:54:48.769 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:48.769+0000 7f08fe000640 1 -- 172.21.15.32:0/3693760922 <== osd.5 v2:172.21.15.118:6810/2724891770 11 ==== osd_op_reply(42 200.00000001 [create] v84'30 uv16 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f08f000a8a0 con 0x557d0a152ac0 2024-08-03T20:54:48.769 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:48.769+0000 7f0904e30f40 1 -- 172.21.15.32:0/3693760922 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:43 12.14 12:2e2fa760:::200.00000001:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e84) -- 0x557d0a1569e0 con 0x557d0a152ac0 2024-08-03T20:54:48.771 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:48.770+0000 7f08fe000640 1 -- 172.21.15.32:0/3693760922 <== osd.5 v2:172.21.15.118:6810/2724891770 12 ==== osd_op_reply(43 200.00000001 [delete] v84'31 uv31 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f08f000a8a0 con 0x557d0a152ac0 2024-08-03T20:54:48.771 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:48.770+0000 7f0904e30f40 1 -- 172.21.15.32:0/3693760922 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:44 12.14 12:2e2fa760:::200.00000001:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e84) -- 0x557d0a1569e0 con 0x557d0a152ac0 2024-08-03T20:54:48.772 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:48.772+0000 7f08fe000640 1 -- 172.21.15.32:0/3693760922 <== osd.5 v2:172.21.15.118:6810/2724891770 13 ==== osd_op_reply(44 200.00000001 [create] v84'32 uv32 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f08f000a8a0 con 0x557d0a152ac0 2024-08-03T20:54:48.772 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::200.00000001:head# 2024-08-03T20:54:48.772 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:48.772+0000 7f0904e30f40 1 -- 172.21.15.32:0/3693760922 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:45 12.14 12:2e2fa760:::200.00000001:head [write 0~18118 in=18118b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e84) -- 0x557d0a15b0d0 con 0x557d0a152ac0 2024-08-03T20:54:48.774 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:48.774+0000 7f08fe000640 1 -- 172.21.15.32:0/3693760922 <== osd.5 v2:172.21.15.118:6810/2724891770 14 ==== osd_op_reply(45 200.00000001 [write 0~18118] v84'33 uv33 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f08f000a8a0 con 0x557d0a152ac0 2024-08-03T20:54:48.774 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:48.774+0000 7f0904e30f40 1 -- 172.21.15.32:0/3693760922 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:46 12.14 12:2e2fa760:::200.00000001:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e84) -- 0x557d0a15b0d0 con 0x557d0a152ac0 2024-08-03T20:54:48.776 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:48.775+0000 7f08fe000640 1 -- 172.21.15.32:0/3693760922 <== osd.5 v2:172.21.15.118:6810/2724891770 15 ==== osd_op_reply(46 200.00000001 [omap-set-header] v84'34 uv34 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f08f000a8a0 con 0x557d0a152ac0 2024-08-03T20:54:48.776 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:48.775+0000 7f0904e30f40 1 -- 172.21.15.32:0/3693760922 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:47 12.14 12:2f55791f:::606.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e84) -- 0x557d0a1569e0 con 0x557d0a152ac0 2024-08-03T20:54:48.777 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:48.777+0000 7f08fe000640 1 -- 172.21.15.32:0/3693760922 <== osd.5 v2:172.21.15.118:6810/2724891770 16 ==== osd_op_reply(47 606.00000000 [create] v84'35 uv2 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f08f000a8a0 con 0x557d0a152ac0 2024-08-03T20:54:48.778 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:48.777+0000 7f0904e30f40 1 -- 172.21.15.32:0/3693760922 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:48 12.14 12:2f55791f:::606.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e84) -- 0x557d0a1569e0 con 0x557d0a152ac0 2024-08-03T20:54:48.779 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:48.778+0000 7f08fe000640 1 -- 172.21.15.32:0/3693760922 <== osd.5 v2:172.21.15.118:6810/2724891770 17 ==== osd_op_reply(48 606.00000000 [delete] v84'36 uv36 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f08f000a8a0 con 0x557d0a152ac0 2024-08-03T20:54:48.779 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:48.778+0000 7f0904e30f40 1 -- 172.21.15.32:0/3693760922 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:49 12.14 12:2f55791f:::606.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e84) -- 0x557d0a1569e0 con 0x557d0a152ac0 2024-08-03T20:54:48.780 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:48.780+0000 7f08fe000640 1 -- 172.21.15.32:0/3693760922 <== osd.5 v2:172.21.15.118:6810/2724891770 18 ==== osd_op_reply(49 606.00000000 [create] v84'37 uv37 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f08f000a8a0 con 0x557d0a152ac0 2024-08-03T20:54:48.780 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::606.00000000:head# 2024-08-03T20:54:48.781 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:48.780+0000 7f0904e30f40 1 -- 172.21.15.32:0/3693760922 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:50 12.14 12:2f55791f:::606.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e84) -- 0x557d0a1579f0 con 0x557d0a152ac0 2024-08-03T20:54:48.782 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:48.781+0000 7f08fe000640 1 -- 172.21.15.32:0/3693760922 <== osd.5 v2:172.21.15.118:6810/2724891770 19 ==== osd_op_reply(50 606.00000000 [setxattr (30)] v84'38 uv38 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f08f000a8a0 con 0x557d0a152ac0 2024-08-03T20:54:48.782 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:48.781+0000 7f0904e30f40 1 -- 172.21.15.32:0/3693760922 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:51 12.14 12:2f55791f:::606.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e84) -- 0x557d0a1579f0 con 0x557d0a152ac0 2024-08-03T20:54:48.783 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:48.783+0000 7f08fe000640 1 -- 172.21.15.32:0/3693760922 <== osd.5 v2:172.21.15.118:6810/2724891770 20 ==== osd_op_reply(51 606.00000000 [setxattr (62)] v84'39 uv39 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f08f000a8a0 con 0x557d0a152ac0 2024-08-03T20:54:48.783 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:48.783+0000 7f0904e30f40 1 -- 172.21.15.32:0/3693760922 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:52 12.14 12:2f55791f:::606.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e84) -- 0x557d0a1569e0 con 0x557d0a152ac0 2024-08-03T20:54:48.785 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:48.785+0000 7f08fe000640 1 -- 172.21.15.32:0/3693760922 <== osd.5 v2:172.21.15.118:6810/2724891770 21 ==== osd_op_reply(52 606.00000000 [omap-set-header] v84'40 uv40 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f08f000a8a0 con 0x557d0a152ac0 2024-08-03T20:54:48.785 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:48.785+0000 7f0904e30f40 1 -- 172.21.15.32:0/3693760922 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:53 12.1c 12:3ed09add:::607.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e84) -- 0x557d0a1569e0 con 0x557d0a089370 2024-08-03T20:54:48.787 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:48.786+0000 7f08fea00640 1 -- 172.21.15.32:0/3693760922 <== osd.0 v2:172.21.15.32:6816/3066156116 32 ==== osd_op_reply(53 607.00000000 [create] v84'3 uv2 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f08f8001770 con 0x557d0a089370 2024-08-03T20:54:48.787 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:48.786+0000 7f0904e30f40 1 -- 172.21.15.32:0/3693760922 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:54 12.1c 12:3ed09add:::607.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e84) -- 0x557d0a1569e0 con 0x557d0a089370 2024-08-03T20:54:48.788 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:48.788+0000 7f08fea00640 1 -- 172.21.15.32:0/3693760922 <== osd.0 v2:172.21.15.32:6816/3066156116 33 ==== osd_op_reply(54 607.00000000 [delete] v84'4 uv4 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f08f8001770 con 0x557d0a089370 2024-08-03T20:54:48.788 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:48.788+0000 7f0904e30f40 1 -- 172.21.15.32:0/3693760922 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:55 12.1c 12:3ed09add:::607.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e84) -- 0x557d0a1569e0 con 0x557d0a089370 2024-08-03T20:54:48.789 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::607.00000000:head# 2024-08-03T20:54:48.790 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:48.789+0000 7f08fea00640 1 -- 172.21.15.32:0/3693760922 <== osd.0 v2:172.21.15.32:6816/3066156116 34 ==== osd_op_reply(55 607.00000000 [create] v84'5 uv5 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f08f8001770 con 0x557d0a089370 2024-08-03T20:54:48.790 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:48.789+0000 7f0904e30f40 1 -- 172.21.15.32:0/3693760922 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:56 12.1c 12:3ed09add:::607.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e84) -- 0x557d0a1579f0 con 0x557d0a089370 2024-08-03T20:54:48.791 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:48.790+0000 7f08fea00640 1 -- 172.21.15.32:0/3693760922 <== osd.0 v2:172.21.15.32:6816/3066156116 35 ==== osd_op_reply(56 607.00000000 [setxattr (30)] v84'6 uv6 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f08f8001770 con 0x557d0a089370 2024-08-03T20:54:48.791 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:48.790+0000 7f0904e30f40 1 -- 172.21.15.32:0/3693760922 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:57 12.1c 12:3ed09add:::607.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e84) -- 0x557d0a1579f0 con 0x557d0a089370 2024-08-03T20:54:48.792 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:48.792+0000 7f08fea00640 1 -- 172.21.15.32:0/3693760922 <== osd.0 v2:172.21.15.32:6816/3066156116 36 ==== osd_op_reply(57 607.00000000 [setxattr (62)] v84'7 uv7 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f08f8001770 con 0x557d0a089370 2024-08-03T20:54:48.792 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:48.792+0000 7f0904e30f40 1 -- 172.21.15.32:0/3693760922 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:58 12.1c 12:3ed09add:::607.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e84) -- 0x557d0a1569e0 con 0x557d0a089370 2024-08-03T20:54:48.794 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:48.793+0000 7f08fea00640 1 -- 172.21.15.32:0/3693760922 <== osd.0 v2:172.21.15.32:6816/3066156116 37 ==== osd_op_reply(58 607.00000000 [omap-set-header] v84'8 uv8 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f08f8001770 con 0x557d0a089370 2024-08-03T20:54:48.794 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:48.793+0000 7f0904e30f40 1 --2- 172.21.15.32:0/3693760922 >> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] conn(0x557d0a1572b0 0x557d0a1596d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:48.794 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:48.793+0000 7f0904e30f40 1 -- 172.21.15.32:0/3693760922 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:59 12.6 12:654134d2:::mds0_openfiles.0:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e84) -- 0x557d0a159c10 con 0x557d0a1572b0 2024-08-03T20:54:48.794 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:48.794+0000 7f08fd600640 1 --2- 172.21.15.32:0/3693760922 >> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] conn(0x557d0a1572b0 0x557d0a1596d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:48.794 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:48.794+0000 7f08fd600640 1 --2- 172.21.15.32:0/3693760922 >> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] conn(0x557d0a1572b0 0x557d0a1596d0 crc :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:54:48.796 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:48.795+0000 7f08fd600640 1 -- 172.21.15.32:0/3693760922 <== osd.1 v2:172.21.15.32:6824/1551241759 1 ==== osd_op_reply(59 mds0_openfiles.0 [create] v84'5 uv4 ondisk = -17 ((17) File exists)) ==== 160+0+0 (crc 0 0 0) 0x7f08ec05f030 con 0x557d0a1572b0 2024-08-03T20:54:48.796 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:48.795+0000 7f0904e30f40 1 -- 172.21.15.32:0/3693760922 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:60 12.6 12:654134d2:::mds0_openfiles.0:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e84) -- 0x557d0a15a070 con 0x557d0a1572b0 2024-08-03T20:54:48.797 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:48.797+0000 7f08fd600640 1 -- 172.21.15.32:0/3693760922 <== osd.1 v2:172.21.15.32:6824/1551241759 2 ==== osd_op_reply(60 mds0_openfiles.0 [delete] v84'6 uv6 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f08ec05f030 con 0x557d0a1572b0 2024-08-03T20:54:48.797 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:48.797+0000 7f0904e30f40 1 -- 172.21.15.32:0/3693760922 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:61 12.6 12:654134d2:::mds0_openfiles.0:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e84) -- 0x557d0a15a560 con 0x557d0a1572b0 2024-08-03T20:54:48.798 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:48.798+0000 7f08fd600640 1 -- 172.21.15.32:0/3693760922 <== osd.1 v2:172.21.15.32:6824/1551241759 3 ==== osd_op_reply(61 mds0_openfiles.0 [create] v84'7 uv7 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f08ec05f030 con 0x557d0a1572b0 2024-08-03T20:54:48.799 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds0_openfiles.0:head# 2024-08-03T20:54:48.799 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:48.798+0000 7f0904e30f40 1 -- 172.21.15.32:0/3693760922 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:62 12.6 12:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e84) -- 0x557d0a01ef80 con 0x557d0a1572b0 2024-08-03T20:54:48.800 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:48.800+0000 7f08fd600640 1 -- 172.21.15.32:0/3693760922 <== osd.1 v2:172.21.15.32:6824/1551241759 4 ==== osd_op_reply(62 mds0_openfiles.0 [omap-set-header] v84'8 uv8 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f08ec05f030 con 0x557d0a1572b0 2024-08-03T20:54:48.800 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:48.800+0000 7f0904e30f40 1 -- 172.21.15.32:0/3693760922 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:63 12.e 12:717a0223:::608.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e84) -- 0x557d0a15ae90 con 0x557d0a089370 2024-08-03T20:54:48.802 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:48.801+0000 7f08fea00640 1 -- 172.21.15.32:0/3693760922 <== osd.0 v2:172.21.15.32:6816/3066156116 38 ==== osd_op_reply(63 608.00000000 [create] v84'3 uv2 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f08f8001770 con 0x557d0a089370 2024-08-03T20:54:48.802 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:48.801+0000 7f0904e30f40 1 -- 172.21.15.32:0/3693760922 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:64 12.e 12:717a0223:::608.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e84) -- 0x557d0a15ae90 con 0x557d0a089370 2024-08-03T20:54:48.803 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:48.803+0000 7f08fea00640 1 -- 172.21.15.32:0/3693760922 <== osd.0 v2:172.21.15.32:6816/3066156116 39 ==== osd_op_reply(64 608.00000000 [delete] v84'4 uv4 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f08f8001770 con 0x557d0a089370 2024-08-03T20:54:48.803 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:48.803+0000 7f0904e30f40 1 -- 172.21.15.32:0/3693760922 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:65 12.e 12:717a0223:::608.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e84) -- 0x557d0a15ae90 con 0x557d0a089370 2024-08-03T20:54:48.805 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::608.00000000:head# 2024-08-03T20:54:48.805 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:48.804+0000 7f08fea00640 1 -- 172.21.15.32:0/3693760922 <== osd.0 v2:172.21.15.32:6816/3066156116 40 ==== osd_op_reply(65 608.00000000 [create] v84'5 uv5 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f08f8001770 con 0x557d0a089370 2024-08-03T20:54:48.805 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:48.804+0000 7f0904e30f40 1 -- 172.21.15.32:0/3693760922 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:66 12.e 12:717a0223:::608.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e84) -- 0x557d0a15bee0 con 0x557d0a089370 2024-08-03T20:54:48.806 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:48.806+0000 7f08fea00640 1 -- 172.21.15.32:0/3693760922 <== osd.0 v2:172.21.15.32:6816/3066156116 41 ==== osd_op_reply(66 608.00000000 [setxattr (30)] v84'6 uv6 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f08f8001770 con 0x557d0a089370 2024-08-03T20:54:48.806 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:48.806+0000 7f0904e30f40 1 -- 172.21.15.32:0/3693760922 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:67 12.e 12:717a0223:::608.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e84) -- 0x557d0a15bea0 con 0x557d0a089370 2024-08-03T20:54:48.808 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:48.808+0000 7f08fea00640 1 -- 172.21.15.32:0/3693760922 <== osd.0 v2:172.21.15.32:6816/3066156116 42 ==== osd_op_reply(67 608.00000000 [setxattr (62)] v84'7 uv7 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f08f8001770 con 0x557d0a089370 2024-08-03T20:54:48.808 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:48.808+0000 7f0904e30f40 1 -- 172.21.15.32:0/3693760922 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:68 12.e 12:717a0223:::608.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e84) -- 0x557d0a15ae90 con 0x557d0a089370 2024-08-03T20:54:48.810 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:48.809+0000 7f08fea00640 1 -- 172.21.15.32:0/3693760922 <== osd.0 v2:172.21.15.32:6816/3066156116 43 ==== osd_op_reply(68 608.00000000 [omap-set-header] v84'8 uv8 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f08f8001770 con 0x557d0a089370 2024-08-03T20:54:48.810 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:48.809+0000 7f0904e30f40 1 --2- 172.21.15.32:0/3693760922 >> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] conn(0x557d0a15b560 0x557d0a15b970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:48.810 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:48.809+0000 7f0904e30f40 1 -- 172.21.15.32:0/3693760922 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:69 12.15 12:a93a17c2:::604.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e84) -- 0x557d0a15e2d0 con 0x557d0a15b560 2024-08-03T20:54:48.810 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:48.810+0000 7f08fea00640 1 --2- 172.21.15.32:0/3693760922 >> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] conn(0x557d0a15b560 0x557d0a15b970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:48.810 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:48.810+0000 7f08fea00640 1 --2- 172.21.15.32:0/3693760922 >> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] conn(0x557d0a15b560 0x557d0a15b970 crc :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:54:48.812 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:48.812+0000 7f08fea00640 1 -- 172.21.15.32:0/3693760922 <== osd.6 v2:172.21.15.118:6825/801947386 1 ==== osd_op_reply(69 604.00000000 [create] v84'4 uv2 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f08f8001770 con 0x557d0a15b560 2024-08-03T20:54:48.812 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:48.812+0000 7f0904e30f40 1 -- 172.21.15.32:0/3693760922 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:70 12.15 12:a93a17c2:::604.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e84) -- 0x557d0a15e2d0 con 0x557d0a15b560 2024-08-03T20:54:48.814 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:48.813+0000 7f08fea00640 1 -- 172.21.15.32:0/3693760922 <== osd.6 v2:172.21.15.118:6825/801947386 2 ==== osd_op_reply(70 604.00000000 [delete] v84'5 uv5 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f08f8001770 con 0x557d0a15b560 2024-08-03T20:54:48.814 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:48.813+0000 7f0904e30f40 1 -- 172.21.15.32:0/3693760922 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:71 12.15 12:a93a17c2:::604.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e84) -- 0x557d0a15e2d0 con 0x557d0a15b560 2024-08-03T20:54:48.815 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::604.00000000:head# 2024-08-03T20:54:48.816 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:48.815+0000 7f08fea00640 1 -- 172.21.15.32:0/3693760922 <== osd.6 v2:172.21.15.118:6825/801947386 3 ==== osd_op_reply(71 604.00000000 [create] v84'6 uv6 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f08f8001770 con 0x557d0a15b560 2024-08-03T20:54:48.816 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:48.815+0000 7f0904e30f40 1 -- 172.21.15.32:0/3693760922 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:72 12.15 12:a93a17c2:::604.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e84) -- 0x557d0a15f6d0 con 0x557d0a15b560 2024-08-03T20:54:48.817 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:48.816+0000 7f08fea00640 1 -- 172.21.15.32:0/3693760922 <== osd.6 v2:172.21.15.118:6825/801947386 4 ==== osd_op_reply(72 604.00000000 [setxattr (30)] v84'7 uv7 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f08f8001770 con 0x557d0a15b560 2024-08-03T20:54:48.817 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:48.817+0000 7f0904e30f40 1 -- 172.21.15.32:0/3693760922 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:73 12.15 12:a93a17c2:::604.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e84) -- 0x557d0a15f6d0 con 0x557d0a15b560 2024-08-03T20:54:48.818 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:48.818+0000 7f08fea00640 1 -- 172.21.15.32:0/3693760922 <== osd.6 v2:172.21.15.118:6825/801947386 5 ==== osd_op_reply(73 604.00000000 [setxattr (62)] v84'8 uv8 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f08f8001770 con 0x557d0a15b560 2024-08-03T20:54:48.818 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:48.818+0000 7f0904e30f40 1 -- 172.21.15.32:0/3693760922 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:74 12.15 12:a93a17c2:::604.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e84) -- 0x557d0a15e6c0 con 0x557d0a15b560 2024-08-03T20:54:48.820 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:48.819+0000 7f08fea00640 1 -- 172.21.15.32:0/3693760922 <== osd.6 v2:172.21.15.118:6825/801947386 6 ==== osd_op_reply(74 604.00000000 [omap-set-header] v84'9 uv9 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f08f8001770 con 0x557d0a15b560 2024-08-03T20:54:48.820 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:48.819+0000 7f0904e30f40 1 -- 172.21.15.32:0/3693760922 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:75 12.15 12:aa448500:::500.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e84) -- 0x557d0a15e6c0 con 0x557d0a15b560 2024-08-03T20:54:48.821 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:48.821+0000 7f08fea00640 1 -- 172.21.15.32:0/3693760922 <== osd.6 v2:172.21.15.118:6825/801947386 7 ==== osd_op_reply(75 500.00000000 [create] v84'10 uv3 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f08f8001770 con 0x557d0a15b560 2024-08-03T20:54:48.821 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:48.821+0000 7f0904e30f40 1 -- 172.21.15.32:0/3693760922 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:76 12.15 12:aa448500:::500.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e84) -- 0x557d0a15e6c0 con 0x557d0a15b560 2024-08-03T20:54:48.823 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:48.823+0000 7f08fea00640 1 -- 172.21.15.32:0/3693760922 <== osd.6 v2:172.21.15.118:6825/801947386 8 ==== osd_op_reply(76 500.00000000 [delete] v84'11 uv11 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f08f8001770 con 0x557d0a15b560 2024-08-03T20:54:48.823 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:48.823+0000 7f0904e30f40 1 -- 172.21.15.32:0/3693760922 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:77 12.15 12:aa448500:::500.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e84) -- 0x557d0a15e6c0 con 0x557d0a15b560 2024-08-03T20:54:48.824 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::500.00000000:head# 2024-08-03T20:54:48.824 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:48.824+0000 7f08fea00640 1 -- 172.21.15.32:0/3693760922 <== osd.6 v2:172.21.15.118:6825/801947386 9 ==== osd_op_reply(77 500.00000000 [create] v84'12 uv12 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f08f8001770 con 0x557d0a15b560 2024-08-03T20:54:48.825 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:48.824+0000 7f0904e30f40 1 -- 172.21.15.32:0/3693760922 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:78 12.15 12:aa448500:::500.00000000:head [write 0~90 in=90b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e84) -- 0x557d0a15e6c0 con 0x557d0a15b560 2024-08-03T20:54:48.826 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:48.825+0000 7f08fea00640 1 -- 172.21.15.32:0/3693760922 <== osd.6 v2:172.21.15.118:6825/801947386 10 ==== osd_op_reply(78 500.00000000 [write 0~90] v84'13 uv13 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f08f8001770 con 0x557d0a15b560 2024-08-03T20:54:48.826 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:48.825+0000 7f0904e30f40 1 -- 172.21.15.32:0/3693760922 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:79 12.15 12:aa448500:::500.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e84) -- 0x557d0a15e6c0 con 0x557d0a15b560 2024-08-03T20:54:48.827 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:48.827+0000 7f08fea00640 1 -- 172.21.15.32:0/3693760922 <== osd.6 v2:172.21.15.118:6825/801947386 11 ==== osd_op_reply(79 500.00000000 [omap-set-header] v84'14 uv14 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f08f8001770 con 0x557d0a15b560 2024-08-03T20:54:48.827 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:48.827+0000 7f0904e30f40 1 --2- 172.21.15.32:0/3693760922 >> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] conn(0x557d0a15ebd0 0x557d0a160ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:48.827 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:48.827+0000 7f0904e30f40 1 -- 172.21.15.32:0/3693760922 --> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] -- osd_op(unknown.0.0:80 12.d 12:b50e409b:::mds_snaptable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e84) -- 0x557d0a161680 con 0x557d0a15ebd0 2024-08-03T20:54:48.828 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:48.827+0000 7f08fe000640 1 --2- 172.21.15.32:0/3693760922 >> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] conn(0x557d0a15ebd0 0x557d0a160ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:48.829 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:48.828+0000 7f08fe000640 1 --2- 172.21.15.32:0/3693760922 >> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] conn(0x557d0a15ebd0 0x557d0a160ff0 crc :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:54:48.831 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:48.829+0000 7f08fe000640 1 -- 172.21.15.32:0/3693760922 <== osd.7 v2:172.21.15.118:6818/653000048 1 ==== osd_op_reply(80 mds_snaptable [create] v84'2 uv1 ondisk = -17 ((17) File exists)) ==== 157+0+0 (crc 0 0 0) 0x7f08f000a8a0 con 0x557d0a15ebd0 2024-08-03T20:54:48.831 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:48.830+0000 7f0904e30f40 1 -- 172.21.15.32:0/3693760922 --> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] -- osd_op(unknown.0.0:81 12.d 12:b50e409b:::mds_snaptable:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e84) -- 0x557d0a161a50 con 0x557d0a15ebd0 2024-08-03T20:54:48.832 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:48.831+0000 7f08fe000640 1 -- 172.21.15.32:0/3693760922 <== osd.7 v2:172.21.15.118:6818/653000048 2 ==== osd_op_reply(81 mds_snaptable [delete] v84'3 uv3 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f08f000a8a0 con 0x557d0a15ebd0 2024-08-03T20:54:48.832 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:48.831+0000 7f0904e30f40 1 -- 172.21.15.32:0/3693760922 --> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] -- osd_op(unknown.0.0:82 12.d 12:b50e409b:::mds_snaptable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e84) -- 0x557d0a161a50 con 0x557d0a15ebd0 2024-08-03T20:54:48.833 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds_snaptable:head# 2024-08-03T20:54:48.833 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:48.832+0000 7f08fe000640 1 -- 172.21.15.32:0/3693760922 <== osd.7 v2:172.21.15.118:6818/653000048 3 ==== osd_op_reply(82 mds_snaptable [create] v84'4 uv4 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f08f000a8a0 con 0x557d0a15ebd0 2024-08-03T20:54:48.834 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:48.832+0000 7f0904e30f40 1 -- 172.21.15.32:0/3693760922 --> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] -- osd_op(unknown.0.0:83 12.d 12:b50e409b:::mds_snaptable:head [write 0~70 in=70b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e84) -- 0x557d0a161a50 con 0x557d0a15ebd0 2024-08-03T20:54:48.838 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:48.834+0000 7f08fe000640 1 -- 172.21.15.32:0/3693760922 <== osd.7 v2:172.21.15.118:6818/653000048 4 ==== osd_op_reply(83 mds_snaptable [write 0~70] v84'5 uv5 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f08f000a8a0 con 0x557d0a15ebd0 2024-08-03T20:54:48.839 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:48.834+0000 7f0904e30f40 1 -- 172.21.15.32:0/3693760922 --> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] -- osd_op(unknown.0.0:84 12.d 12:b50e409b:::mds_snaptable:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e84) -- 0x557d0a161a50 con 0x557d0a15ebd0 2024-08-03T20:54:48.840 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:48.835+0000 7f08fe000640 1 -- 172.21.15.32:0/3693760922 <== osd.7 v2:172.21.15.118:6818/653000048 5 ==== osd_op_reply(84 mds_snaptable [omap-set-header] v84'6 uv6 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f08f000a8a0 con 0x557d0a15ebd0 2024-08-03T20:54:48.840 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:48.835+0000 7f0904e30f40 1 -- 172.21.15.32:0/3693760922 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:85 12.1d 12:b871830b:::605.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e84) -- 0x557d0a161a50 con 0x557d0a089370 2024-08-03T20:54:48.840 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:48.837+0000 7f08fea00640 1 -- 172.21.15.32:0/3693760922 <== osd.0 v2:172.21.15.32:6816/3066156116 44 ==== osd_op_reply(85 605.00000000 [create] v84'3 uv2 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f08f8001770 con 0x557d0a089370 2024-08-03T20:54:48.840 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:48.837+0000 7f0904e30f40 1 -- 172.21.15.32:0/3693760922 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:86 12.1d 12:b871830b:::605.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e84) -- 0x557d0a161a50 con 0x557d0a089370 2024-08-03T20:54:48.841 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:48.838+0000 7f08fea00640 1 -- 172.21.15.32:0/3693760922 <== osd.0 v2:172.21.15.32:6816/3066156116 45 ==== osd_op_reply(86 605.00000000 [delete] v84'4 uv4 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f08f8001770 con 0x557d0a089370 2024-08-03T20:54:48.841 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:48.838+0000 7f0904e30f40 1 -- 172.21.15.32:0/3693760922 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:87 12.1d 12:b871830b:::605.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e84) -- 0x557d0a161a50 con 0x557d0a089370 2024-08-03T20:54:48.841 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::605.00000000:head# 2024-08-03T20:54:48.843 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:48.840+0000 7f08fea00640 1 -- 172.21.15.32:0/3693760922 <== osd.0 v2:172.21.15.32:6816/3066156116 46 ==== osd_op_reply(87 605.00000000 [create] v84'5 uv5 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f08f8001770 con 0x557d0a089370 2024-08-03T20:54:48.843 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:48.840+0000 7f0904e30f40 1 -- 172.21.15.32:0/3693760922 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:88 12.1d 12:b871830b:::605.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e84) -- 0x557d0a162a60 con 0x557d0a089370 2024-08-03T20:54:48.843 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:48.841+0000 7f08fea00640 1 -- 172.21.15.32:0/3693760922 <== osd.0 v2:172.21.15.32:6816/3066156116 47 ==== osd_op_reply(88 605.00000000 [setxattr (30)] v84'6 uv6 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f08f8001770 con 0x557d0a089370 2024-08-03T20:54:48.844 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:48.841+0000 7f0904e30f40 1 -- 172.21.15.32:0/3693760922 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:89 12.1d 12:b871830b:::605.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e84) -- 0x557d0a162a60 con 0x557d0a089370 2024-08-03T20:54:48.844 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:48.842+0000 7f08fea00640 1 -- 172.21.15.32:0/3693760922 <== osd.0 v2:172.21.15.32:6816/3066156116 48 ==== osd_op_reply(89 605.00000000 [setxattr (62)] v84'7 uv7 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f08f8001770 con 0x557d0a089370 2024-08-03T20:54:48.844 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:48.842+0000 7f0904e30f40 1 -- 172.21.15.32:0/3693760922 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:90 12.1d 12:b871830b:::605.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e84) -- 0x557d0a161a50 con 0x557d0a089370 2024-08-03T20:54:48.846 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:48.844+0000 7f08fea00640 1 -- 172.21.15.32:0/3693760922 <== osd.0 v2:172.21.15.32:6816/3066156116 49 ==== osd_op_reply(90 605.00000000 [omap-set-header] v84'8 uv8 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f08f8001770 con 0x557d0a089370 2024-08-03T20:54:48.847 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:48.844+0000 7f0904e30f40 1 -- 172.21.15.32:0/3693760922 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:91 12.13 12:c91d4a1d:::mds0_inotable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e84) -- 0x557d0a161a50 con 0x557d0a1572b0 2024-08-03T20:54:48.847 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:48.845+0000 7f08fd600640 1 -- 172.21.15.32:0/3693760922 <== osd.1 v2:172.21.15.32:6824/1551241759 5 ==== osd_op_reply(91 mds0_inotable [create] v84'4 uv3 ondisk = -17 ((17) File exists)) ==== 157+0+0 (crc 0 0 0) 0x7f08ec05f030 con 0x557d0a1572b0 2024-08-03T20:54:48.847 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:48.845+0000 7f0904e30f40 1 -- 172.21.15.32:0/3693760922 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:92 12.13 12:c91d4a1d:::mds0_inotable:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e84) -- 0x557d0a161df0 con 0x557d0a1572b0 2024-08-03T20:54:48.847 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:48.846+0000 7f08fd600640 1 -- 172.21.15.32:0/3693760922 <== osd.1 v2:172.21.15.32:6824/1551241759 6 ==== osd_op_reply(92 mds0_inotable [delete] v84'5 uv5 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f08ec05f030 con 0x557d0a1572b0 2024-08-03T20:54:48.847 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:48.846+0000 7f0904e30f40 1 -- 172.21.15.32:0/3693760922 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:93 12.13 12:c91d4a1d:::mds0_inotable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e84) -- 0x557d0a1621e0 con 0x557d0a1572b0 2024-08-03T20:54:48.848 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds0_inotable:head# 2024-08-03T20:54:48.848 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:48.847+0000 7f08fd600640 1 -- 172.21.15.32:0/3693760922 <== osd.1 v2:172.21.15.32:6824/1551241759 7 ==== osd_op_reply(93 mds0_inotable [create] v84'6 uv6 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f08ec05f030 con 0x557d0a1572b0 2024-08-03T20:54:48.848 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:48.848+0000 7f0904e30f40 1 -- 172.21.15.32:0/3693760922 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:94 12.13 12:c91d4a1d:::mds0_inotable:head [write 0~34 in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e84) -- 0x557d0a1625d0 con 0x557d0a1572b0 2024-08-03T20:54:48.849 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:48.849+0000 7f08fd600640 1 -- 172.21.15.32:0/3693760922 <== osd.1 v2:172.21.15.32:6824/1551241759 8 ==== osd_op_reply(94 mds0_inotable [write 0~34] v84'7 uv7 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f08ec05f030 con 0x557d0a1572b0 2024-08-03T20:54:48.849 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:48.849+0000 7f0904e30f40 1 -- 172.21.15.32:0/3693760922 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:95 12.13 12:c91d4a1d:::mds0_inotable:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e84) -- 0x557d0a1625d0 con 0x557d0a1572b0 2024-08-03T20:54:48.851 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:48.850+0000 7f08fd600640 1 -- 172.21.15.32:0/3693760922 <== osd.1 v2:172.21.15.32:6824/1551241759 9 ==== osd_op_reply(95 mds0_inotable [omap-set-header] v84'8 uv8 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f08ec05f030 con 0x557d0a1572b0 2024-08-03T20:54:48.851 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:48.850+0000 7f0904e30f40 1 -- 172.21.15.32:0/3693760922 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:96 12.13 12:cd5a64a3:::100.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e84) -- 0x557d0a1622d0 con 0x557d0a1572b0 2024-08-03T20:54:48.852 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:48.852+0000 7f08fd600640 1 -- 172.21.15.32:0/3693760922 <== osd.1 v2:172.21.15.32:6824/1551241759 10 ==== osd_op_reply(96 100.00000000 [create] v84'9 uv1 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f08ec05f030 con 0x557d0a1572b0 2024-08-03T20:54:48.852 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:48.852+0000 7f0904e30f40 1 -- 172.21.15.32:0/3693760922 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:97 12.13 12:cd5a64a3:::100.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e84) -- 0x557d0a162280 con 0x557d0a1572b0 2024-08-03T20:54:48.853 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:48.853+0000 7f08fd600640 1 -- 172.21.15.32:0/3693760922 <== osd.1 v2:172.21.15.32:6824/1551241759 11 ==== osd_op_reply(97 100.00000000 [delete] v84'10 uv10 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f08ec05f030 con 0x557d0a1572b0 2024-08-03T20:54:48.854 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:48.853+0000 7f0904e30f40 1 -- 172.21.15.32:0/3693760922 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:98 12.13 12:cd5a64a3:::100.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e84) -- 0x557d0a162280 con 0x557d0a1572b0 2024-08-03T20:54:48.854 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:48.854+0000 7f08fd600640 1 -- 172.21.15.32:0/3693760922 <== osd.1 v2:172.21.15.32:6824/1551241759 12 ==== osd_op_reply(98 100.00000000 [create] v84'11 uv11 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f08ec05f030 con 0x557d0a1572b0 2024-08-03T20:54:48.855 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::100.00000000:head# 2024-08-03T20:54:48.855 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:48.854+0000 7f0904e30f40 1 -- 172.21.15.32:0/3693760922 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:99 12.13 12:cd5a64a3:::100.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e84) -- 0x557d0a162670 con 0x557d0a1572b0 2024-08-03T20:54:48.856 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:48.856+0000 7f08fd600640 1 -- 172.21.15.32:0/3693760922 <== osd.1 v2:172.21.15.32:6824/1551241759 13 ==== osd_op_reply(99 100.00000000 [omap-set-header] v84'12 uv12 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f08ec05f030 con 0x557d0a1572b0 2024-08-03T20:54:48.856 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:48.856+0000 7f0904e30f40 1 -- 172.21.15.32:0/3693760922 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:100 12.13 12:cd5a64a3:::100.00000000:head [omap-set-vals in=5194b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e84) -- 0x557d0a1650d0 con 0x557d0a1572b0 2024-08-03T20:54:48.857 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:48.857+0000 7f08fd600640 1 -- 172.21.15.32:0/3693760922 <== osd.1 v2:172.21.15.32:6824/1551241759 14 ==== osd_op_reply(100 100.00000000 [omap-set-vals] v84'13 uv13 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f08ec05f030 con 0x557d0a1572b0 2024-08-03T20:54:48.858 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:48.857+0000 7f0904e30f40 1 --2- 172.21.15.32:0/3693760922 >> [v2:172.21.15.32:6800/9088303,v1:172.21.15.32:6801/9088303] conn(0x557d0a162db0 0x557d0a165750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:48.858 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:48.857+0000 7f0904e30f40 1 -- 172.21.15.32:0/3693760922 --> [v2:172.21.15.32:6800/9088303,v1:172.21.15.32:6801/9088303] -- osd_op(unknown.0.0:101 12.b 12:d0630e4c:::mds0_sessionmap:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e84) -- 0x557d0a1651d0 con 0x557d0a162db0 2024-08-03T20:54:48.858 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:48.857+0000 7f08fd600640 1 --2- 172.21.15.32:0/3693760922 >> [v2:172.21.15.32:6800/9088303,v1:172.21.15.32:6801/9088303] conn(0x557d0a162db0 0x557d0a165750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:48.858 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:48.858+0000 7f08fd600640 1 --2- 172.21.15.32:0/3693760922 >> [v2:172.21.15.32:6800/9088303,v1:172.21.15.32:6801/9088303] conn(0x557d0a162db0 0x557d0a165750 crc :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:54:48.860 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:48.859+0000 7f08fd600640 1 -- 172.21.15.32:0/3693760922 <== osd.3 v2:172.21.15.32:6800/9088303 1 ==== osd_op_reply(101 mds0_sessionmap [create] v84'4 uv3 ondisk = -17 ((17) File exists)) ==== 159+0+0 (crc 0 0 0) 0x7f08ec05f030 con 0x557d0a162db0 2024-08-03T20:54:48.860 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:48.859+0000 7f0904e30f40 1 -- 172.21.15.32:0/3693760922 --> [v2:172.21.15.32:6800/9088303,v1:172.21.15.32:6801/9088303] -- osd_op(unknown.0.0:102 12.b 12:d0630e4c:::mds0_sessionmap:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e84) -- 0x557d0a1651d0 con 0x557d0a162db0 2024-08-03T20:54:48.861 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:48.860+0000 7f08fd600640 1 -- 172.21.15.32:0/3693760922 <== osd.3 v2:172.21.15.32:6800/9088303 2 ==== osd_op_reply(102 mds0_sessionmap [delete] v84'5 uv5 ondisk = 0) ==== 159+0+0 (crc 0 0 0) 0x7f08ec05f030 con 0x557d0a162db0 2024-08-03T20:54:48.861 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:48.861+0000 7f0904e30f40 1 -- 172.21.15.32:0/3693760922 --> [v2:172.21.15.32:6800/9088303,v1:172.21.15.32:6801/9088303] -- osd_op(unknown.0.0:103 12.b 12:d0630e4c:::mds0_sessionmap:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e84) -- 0x557d0a1651d0 con 0x557d0a162db0 2024-08-03T20:54:48.862 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:48.862+0000 7f08fd600640 1 -- 172.21.15.32:0/3693760922 <== osd.3 v2:172.21.15.32:6800/9088303 3 ==== osd_op_reply(103 mds0_sessionmap [create] v84'6 uv6 ondisk = 0) ==== 159+0+0 (crc 0 0 0) 0x7f08ec05f030 con 0x557d0a162db0 2024-08-03T20:54:48.863 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds0_sessionmap:head# 2024-08-03T20:54:48.863 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:48.862+0000 7f0904e30f40 1 -- 172.21.15.32:0/3693760922 --> [v2:172.21.15.32:6800/9088303,v1:172.21.15.32:6801/9088303] -- osd_op(unknown.0.0:104 12.b 12:d0630e4c:::mds0_sessionmap:head [omap-set-header in=14b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e84) -- 0x557d0a1651d0 con 0x557d0a162db0 2024-08-03T20:54:48.864 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:48.863+0000 7f08fd600640 1 -- 172.21.15.32:0/3693760922 <== osd.3 v2:172.21.15.32:6800/9088303 4 ==== osd_op_reply(104 mds0_sessionmap [omap-set-header] v84'7 uv7 ondisk = 0) ==== 159+0+0 (crc 0 0 0) 0x7f08ec05f030 con 0x557d0a162db0 2024-08-03T20:54:48.864 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:48.863+0000 7f0904e30f40 1 -- 172.21.15.32:0/3693760922 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:105 12.7 12:e0b41d06:::609.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e84) -- 0x557d0a1651d0 con 0x557d0a1572b0 2024-08-03T20:54:48.866 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:48.865+0000 7f08fd600640 1 -- 172.21.15.32:0/3693760922 <== osd.1 v2:172.21.15.32:6824/1551241759 15 ==== osd_op_reply(105 609.00000000 [create] v84'3 uv2 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f08ec05f030 con 0x557d0a1572b0 2024-08-03T20:54:48.866 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:48.865+0000 7f0904e30f40 1 -- 172.21.15.32:0/3693760922 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:106 12.7 12:e0b41d06:::609.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e84) -- 0x557d0a1651d0 con 0x557d0a1572b0 2024-08-03T20:54:48.867 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:48.866+0000 7f08fd600640 1 -- 172.21.15.32:0/3693760922 <== osd.1 v2:172.21.15.32:6824/1551241759 16 ==== osd_op_reply(106 609.00000000 [delete] v84'4 uv4 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f08ec05f030 con 0x557d0a1572b0 2024-08-03T20:54:48.867 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:48.866+0000 7f0904e30f40 1 -- 172.21.15.32:0/3693760922 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:107 12.7 12:e0b41d06:::609.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e84) -- 0x557d0a1651d0 con 0x557d0a1572b0 2024-08-03T20:54:48.868 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:48.868+0000 7f08fd600640 1 -- 172.21.15.32:0/3693760922 <== osd.1 v2:172.21.15.32:6824/1551241759 17 ==== osd_op_reply(107 609.00000000 [create] v84'5 uv5 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f08ec05f030 con 0x557d0a1572b0 2024-08-03T20:54:48.869 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::609.00000000:head# 2024-08-03T20:54:48.869 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:48.868+0000 7f0904e30f40 1 -- 172.21.15.32:0/3693760922 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:108 12.7 12:e0b41d06:::609.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e84) -- 0x557d0a1651d0 con 0x557d0a1572b0 2024-08-03T20:54:48.870 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:48.869+0000 7f08fd600640 1 -- 172.21.15.32:0/3693760922 <== osd.1 v2:172.21.15.32:6824/1551241759 18 ==== osd_op_reply(108 609.00000000 [setxattr (30)] v84'6 uv6 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f08ec05f030 con 0x557d0a1572b0 2024-08-03T20:54:48.870 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:48.869+0000 7f0904e30f40 1 -- 172.21.15.32:0/3693760922 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:109 12.7 12:e0b41d06:::609.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e84) -- 0x557d0a1651d0 con 0x557d0a1572b0 2024-08-03T20:54:48.871 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:48.871+0000 7f08fd600640 1 -- 172.21.15.32:0/3693760922 <== osd.1 v2:172.21.15.32:6824/1551241759 19 ==== osd_op_reply(109 609.00000000 [setxattr (62)] v84'7 uv7 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f08ec05f030 con 0x557d0a1572b0 2024-08-03T20:54:48.872 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:48.871+0000 7f0904e30f40 1 -- 172.21.15.32:0/3693760922 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:110 12.7 12:e0b41d06:::609.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e84) -- 0x557d0a1651d0 con 0x557d0a1572b0 2024-08-03T20:54:48.872 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:48.872+0000 7f08fd600640 1 -- 172.21.15.32:0/3693760922 <== osd.1 v2:172.21.15.32:6824/1551241759 20 ==== osd_op_reply(110 609.00000000 [omap-set-header] v84'8 uv8 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f08ec05f030 con 0x557d0a1572b0 2024-08-03T20:54:48.872 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:48.872+0000 7f0904e30f40 1 -- 172.21.15.32:0/3693760922 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:111 12.f 12:f1f69726:::400.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e84) -- 0x557d0a1651d0 con 0x557d0a15b560 2024-08-03T20:54:48.874 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:48.874+0000 7f08fea00640 1 -- 172.21.15.32:0/3693760922 <== osd.6 v2:172.21.15.118:6825/801947386 12 ==== osd_op_reply(111 400.00000000 [create] v84'2 uv1 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f08f8001770 con 0x557d0a15b560 2024-08-03T20:54:48.874 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:48.874+0000 7f0904e30f40 1 -- 172.21.15.32:0/3693760922 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:112 12.f 12:f1f69726:::400.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e84) -- 0x557d0a1651d0 con 0x557d0a15b560 2024-08-03T20:54:48.876 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:48.875+0000 7f08fea00640 1 -- 172.21.15.32:0/3693760922 <== osd.6 v2:172.21.15.118:6825/801947386 13 ==== osd_op_reply(112 400.00000000 [delete] v84'3 uv3 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f08f8001770 con 0x557d0a15b560 2024-08-03T20:54:48.876 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:48.875+0000 7f0904e30f40 1 -- 172.21.15.32:0/3693760922 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:113 12.f 12:f1f69726:::400.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e84) -- 0x557d0a1651d0 con 0x557d0a15b560 2024-08-03T20:54:48.877 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::400.00000000:head# 2024-08-03T20:54:48.878 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:48.877+0000 7f08fea00640 1 -- 172.21.15.32:0/3693760922 <== osd.6 v2:172.21.15.118:6825/801947386 14 ==== osd_op_reply(113 400.00000000 [create] v84'4 uv4 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f08f8001770 con 0x557d0a15b560 2024-08-03T20:54:48.878 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:48.877+0000 7f0904e30f40 1 -- 172.21.15.32:0/3693760922 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:114 12.f 12:f1f69726:::400.00000000:head [write 0~22 in=22b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e84) -- 0x557d0a1651d0 con 0x557d0a15b560 2024-08-03T20:54:48.879 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:48.878+0000 7f08fea00640 1 -- 172.21.15.32:0/3693760922 <== osd.6 v2:172.21.15.118:6825/801947386 15 ==== osd_op_reply(114 400.00000000 [write 0~22] v84'5 uv5 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f08f8001770 con 0x557d0a15b560 2024-08-03T20:54:48.879 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:48.879+0000 7f0904e30f40 1 -- 172.21.15.32:0/3693760922 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:115 12.f 12:f1f69726:::400.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e84) -- 0x557d0a1651d0 con 0x557d0a15b560 2024-08-03T20:54:48.880 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:48.880+0000 7f08fea00640 1 -- 172.21.15.32:0/3693760922 <== osd.6 v2:172.21.15.118:6825/801947386 16 ==== osd_op_reply(115 400.00000000 [omap-set-header] v84'6 uv6 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f08f8001770 con 0x557d0a15b560 2024-08-03T20:54:48.880 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:48.880+0000 7f0904e30f40 1 -- 172.21.15.32:0/3693760922 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:116 12.1f 12:fe07bba1:::100.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e84) -- 0x557d0a1651d0 con 0x557d0a15b560 2024-08-03T20:54:48.882 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:48.881+0000 7f08fea00640 1 -- 172.21.15.32:0/3693760922 <== osd.6 v2:172.21.15.118:6825/801947386 17 ==== osd_op_reply(116 100.00000000.inode [create] v84'7 uv3 ondisk = -17 ((17) File exists)) ==== 162+0+0 (crc 0 0 0) 0x7f08f8001770 con 0x557d0a15b560 2024-08-03T20:54:48.882 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:48.881+0000 7f0904e30f40 1 -- 172.21.15.32:0/3693760922 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:117 12.1f 12:fe07bba1:::100.00000000.inode:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e84) -- 0x557d0a165c90 con 0x557d0a15b560 2024-08-03T20:54:48.883 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:48.883+0000 7f08fea00640 1 -- 172.21.15.32:0/3693760922 <== osd.6 v2:172.21.15.118:6825/801947386 18 ==== osd_op_reply(117 100.00000000.inode [delete] v84'8 uv8 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7f08f8001770 con 0x557d0a15b560 2024-08-03T20:54:48.883 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:48.883+0000 7f0904e30f40 1 -- 172.21.15.32:0/3693760922 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:118 12.1f 12:fe07bba1:::100.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e84) -- 0x557d0a165c90 con 0x557d0a15b560 2024-08-03T20:54:48.884 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::100.00000000.inode:head# 2024-08-03T20:54:48.885 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:48.884+0000 7f08fea00640 1 -- 172.21.15.32:0/3693760922 <== osd.6 v2:172.21.15.118:6825/801947386 19 ==== osd_op_reply(118 100.00000000.inode [create] v84'9 uv9 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7f08f8001770 con 0x557d0a15b560 2024-08-03T20:54:48.885 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:48.884+0000 7f0904e30f40 1 -- 172.21.15.32:0/3693760922 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:119 12.1f 12:fe07bba1:::100.00000000.inode:head [write 0~582 in=582b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e84) -- 0x557d0a165c90 con 0x557d0a15b560 2024-08-03T20:54:48.886 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:48.886+0000 7f08fea00640 1 -- 172.21.15.32:0/3693760922 <== osd.6 v2:172.21.15.118:6825/801947386 20 ==== osd_op_reply(119 100.00000000.inode [write 0~582] v84'10 uv10 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7f08f8001770 con 0x557d0a15b560 2024-08-03T20:54:48.886 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:48.886+0000 7f0904e30f40 1 -- 172.21.15.32:0/3693760922 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:120 12.1f 12:fe07bba1:::100.00000000.inode:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e84) -- 0x557d0a165c90 con 0x557d0a15b560 2024-08-03T20:54:48.888 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:48.887+0000 7f08fea00640 1 -- 172.21.15.32:0/3693760922 <== osd.6 v2:172.21.15.118:6825/801947386 21 ==== osd_op_reply(120 100.00000000.inode [omap-set-header] v84'11 uv11 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7f08f8001770 con 0x557d0a15b560 2024-08-03T20:54:48.888 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:48.887+0000 7f0904e30f40 1 -- 172.21.15.32:0/3693760922 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:121 12.1f 12:ff5b34d6:::1.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e84) -- 0x557d0a165c90 con 0x557d0a15b560 2024-08-03T20:54:48.889 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:48.889+0000 7f08fea00640 1 -- 172.21.15.32:0/3693760922 <== osd.6 v2:172.21.15.118:6825/801947386 22 ==== osd_op_reply(121 1.00000000 [create] v84'12 uv6 ondisk = -17 ((17) File exists)) ==== 154+0+0 (crc 0 0 0) 0x7f08f8001770 con 0x557d0a15b560 2024-08-03T20:54:48.889 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:48.889+0000 7f0904e30f40 1 -- 172.21.15.32:0/3693760922 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:122 12.1f 12:ff5b34d6:::1.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e84) -- 0x557d0a1651d0 con 0x557d0a15b560 2024-08-03T20:54:48.891 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:48.890+0000 7f08fea00640 1 -- 172.21.15.32:0/3693760922 <== osd.6 v2:172.21.15.118:6825/801947386 23 ==== osd_op_reply(122 1.00000000 [delete] v84'13 uv13 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7f08f8001770 con 0x557d0a15b560 2024-08-03T20:54:48.891 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:48.890+0000 7f0904e30f40 1 -- 172.21.15.32:0/3693760922 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:123 12.1f 12:ff5b34d6:::1.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e84) -- 0x557d0a1651d0 con 0x557d0a15b560 2024-08-03T20:54:48.892 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::1.00000000:head# 2024-08-03T20:54:48.892 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:48.892+0000 7f08fea00640 1 -- 172.21.15.32:0/3693760922 <== osd.6 v2:172.21.15.118:6825/801947386 24 ==== osd_op_reply(123 1.00000000 [create] v84'14 uv14 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7f08f8001770 con 0x557d0a15b560 2024-08-03T20:54:48.893 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:48.892+0000 7f0904e30f40 1 -- 172.21.15.32:0/3693760922 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:124 12.1f 12:ff5b34d6:::1.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e84) -- 0x557d0a166cf0 con 0x557d0a15b560 2024-08-03T20:54:48.894 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:48.893+0000 7f08fea00640 1 -- 172.21.15.32:0/3693760922 <== osd.6 v2:172.21.15.118:6825/801947386 25 ==== osd_op_reply(124 1.00000000 [setxattr (30)] v84'15 uv15 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7f08f8001770 con 0x557d0a15b560 2024-08-03T20:54:48.894 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:48.894+0000 7f0904e30f40 1 -- 172.21.15.32:0/3693760922 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:125 12.1f 12:ff5b34d6:::1.00000000:head [setxattr parent (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e84) -- 0x557d0a166ca0 con 0x557d0a15b560 2024-08-03T20:54:48.895 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:48.895+0000 7f08fea00640 1 -- 172.21.15.32:0/3693760922 <== osd.6 v2:172.21.15.118:6825/801947386 26 ==== osd_op_reply(125 1.00000000 [setxattr (30)] v84'16 uv16 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7f08f8001770 con 0x557d0a15b560 2024-08-03T20:54:48.895 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:48.895+0000 7f0904e30f40 1 -- 172.21.15.32:0/3693760922 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:126 12.1f 12:ff5b34d6:::1.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e84) -- 0x557d0a165c90 con 0x557d0a15b560 2024-08-03T20:54:48.897 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:48.897+0000 7f08fea00640 1 -- 172.21.15.32:0/3693760922 <== osd.6 v2:172.21.15.118:6825/801947386 27 ==== osd_op_reply(126 1.00000000 [omap-set-header] v84'17 uv17 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7f08f8001770 con 0x557d0a15b560 2024-08-03T20:54:48.897 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:48.897+0000 7f0904e30f40 1 -- 172.21.15.32:0/3693760922 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:127 12.1f 12:ff5b34d6:::1.00000000:head [omap-set-vals in=584b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e84) -- 0x557d0a166140 con 0x557d0a15b560 2024-08-03T20:54:48.899 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:48.898+0000 7f08fea00640 1 -- 172.21.15.32:0/3693760922 <== osd.6 v2:172.21.15.118:6825/801947386 28 ==== osd_op_reply(127 1.00000000 [omap-set-vals] v84'18 uv18 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7f08f8001770 con 0x557d0a15b560 2024-08-03T20:54:48.899 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:48.898+0000 7f0904e30f40 1 -- 172.21.15.32:0/3693760922 >> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] conn(0x557d0a089370 msgr2=0x557d0a089780 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:48.899 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:48.898+0000 7f0904e30f40 1 --2- 172.21.15.32:0/3693760922 >> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] conn(0x557d0a089370 0x557d0a089780 crc :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:48.899 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:48.898+0000 7f0904e30f40 1 -- 172.21.15.32:0/3693760922 >> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] conn(0x557d0a1572b0 msgr2=0x557d0a1596d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:48.900 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:48.898+0000 7f0904e30f40 1 --2- 172.21.15.32:0/3693760922 >> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] conn(0x557d0a1572b0 0x557d0a1596d0 crc :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:48.900 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:48.898+0000 7f0904e30f40 1 -- 172.21.15.32:0/3693760922 >> [v2:172.21.15.32:6800/9088303,v1:172.21.15.32:6801/9088303] conn(0x557d0a162db0 msgr2=0x557d0a165750 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:48.900 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:48.898+0000 7f0904e30f40 1 --2- 172.21.15.32:0/3693760922 >> [v2:172.21.15.32:6800/9088303,v1:172.21.15.32:6801/9088303] conn(0x557d0a162db0 0x557d0a165750 crc :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:48.900 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:48.898+0000 7f0904e30f40 1 -- 172.21.15.32:0/3693760922 >> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] conn(0x557d0a152ac0 msgr2=0x557d0a154ee0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:48.900 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:48.898+0000 7f0904e30f40 1 --2- 172.21.15.32:0/3693760922 >> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] conn(0x557d0a152ac0 0x557d0a154ee0 crc :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:48.900 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:48.898+0000 7f0904e30f40 1 -- 172.21.15.32:0/3693760922 >> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] conn(0x557d0a15b560 msgr2=0x557d0a15b970 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:48.900 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:48.898+0000 7f0904e30f40 1 --2- 172.21.15.32:0/3693760922 >> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] conn(0x557d0a15b560 0x557d0a15b970 crc :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:48.900 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:48.898+0000 7f08fea00640 1 -- 172.21.15.32:0/3693760922 reap_dead start 2024-08-03T20:54:48.900 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:48.898+0000 7f0904e30f40 1 -- 172.21.15.32:0/3693760922 >> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] conn(0x557d0a15ebd0 msgr2=0x557d0a160ff0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:48.900 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:48.898+0000 7f0904e30f40 1 --2- 172.21.15.32:0/3693760922 >> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] conn(0x557d0a15ebd0 0x557d0a160ff0 crc :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:48.900 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:48.898+0000 7f08fea00640 1 -- 172.21.15.32:0/3693760922 reap_dead start 2024-08-03T20:54:48.900 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:48.898+0000 7f0904e30f40 1 -- 172.21.15.32:0/3693760922 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f08e00759f0 msgr2=0x7f08e0077f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:48.900 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:48.898+0000 7f0904e30f40 1 --2- 172.21.15.32:0/3693760922 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f08e00759f0 0x7f08e0077f00 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f08f000c740 tx=0x7f08f0004880 comp rx=0 tx=0).stop 2024-08-03T20:54:48.901 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:48.898+0000 7f0904e30f40 1 -- 172.21.15.32:0/3693760922 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x557d0a08fbe0 msgr2=0x557d0a143c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:48.901 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:48.898+0000 7f0904e30f40 1 --2- 172.21.15.32:0/3693760922 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x557d0a08fbe0 0x557d0a143c90 secure :-1 s=READY pgs=1092 cs=0 l=1 rev1=1 crypto rx=0x7f08ec025e30 tx=0x7f08ec0575e0 comp rx=0 tx=0).stop 2024-08-03T20:54:48.901 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:48.898+0000 7f0904e30f40 1 -- 172.21.15.32:0/3693760922 shutdown_connections 2024-08-03T20:54:48.901 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:48.898+0000 7f0904e30f40 1 --2- 172.21.15.32:0/3693760922 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f08e00759f0 0x7f08e0077f00 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:48.901 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:48.898+0000 7f0904e30f40 1 --2- 172.21.15.32:0/3693760922 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x557d0a08fbe0 0x557d0a143c90 unknown :-1 s=CLOSED pgs=1092 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:48.901 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:48.898+0000 7f0904e30f40 1 -- 172.21.15.32:0/3693760922 >> 172.21.15.32:0/3693760922 conn(0x557d09ef7ac0 msgr2=0x557d0a0907b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:54:48.901 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:48.898+0000 7f0904e30f40 1 -- 172.21.15.32:0/3693760922 shutdown_connections 2024-08-03T20:54:48.901 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:48.898+0000 7f0904e30f40 1 -- 172.21.15.32:0/3693760922 wait complete. 2024-08-03T20:54:48.913 DEBUG:teuthology.orchestra.run.smithi032:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata rm 10000000000.00000000 2024-08-03T20:54:49.100 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:49.100+0000 7f2876dd4f40 1 Processor -- start 2024-08-03T20:54:49.101 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:49.100+0000 7f2876dd4f40 1 -- start start 2024-08-03T20:54:49.103 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:49.101+0000 7f2876dd4f40 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x5594d3e9f360 0x5594d3e2fde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:49.103 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:49.101+0000 7f2876dd4f40 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5594d3ea0ea0 0x5594d3eb6ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:49.103 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:49.101+0000 7f2876dd4f40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5594d3eb6330 0x5594d3eb8180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:49.103 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:49.102+0000 7f2876dd4f40 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x5594d3d36e30 con 0x5594d3ea0ea0 2024-08-03T20:54:49.103 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:49.102+0000 7f2876dd4f40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x5594d3d94bf0 con 0x5594d3eb6330 2024-08-03T20:54:49.103 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:49.102+0000 7f2876dd4f40 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x5594d3ea0440 con 0x5594d3e9f360 2024-08-03T20:54:49.104 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:49.102+0000 7f286f600640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5594d3ea0ea0 0x5594d3eb6ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:49.104 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:49.102+0000 7f286f600640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5594d3ea0ea0 0x5594d3eb6ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:37568/0 (socket says 172.21.15.32:37568) 2024-08-03T20:54:49.104 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:49.102+0000 7f286f600640 1 -- 172.21.15.32:0/635209388 learned_addr learned my addr 172.21.15.32:0/635209388 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:54:49.104 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:49.102+0000 7f2870a00640 1 --2- 172.21.15.32:0/635209388 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5594d3eb6330 0x5594d3eb8180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:49.104 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:49.102+0000 7f2870000640 1 --2- 172.21.15.32:0/635209388 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x5594d3e9f360 0x5594d3e2fde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:49.104 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:49.102+0000 7f286f600640 1 -- 172.21.15.32:0/635209388 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x5594d3e9f360 msgr2=0x5594d3e2fde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:54:49.104 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:49.102+0000 7f286f600640 1 --2- 172.21.15.32:0/635209388 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x5594d3e9f360 0x5594d3e2fde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:49.104 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:49.102+0000 7f286f600640 1 -- 172.21.15.32:0/635209388 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5594d3eb6330 msgr2=0x5594d3eb8180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:54:49.104 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:49.102+0000 7f286f600640 1 --2- 172.21.15.32:0/635209388 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5594d3eb6330 0x5594d3eb8180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:49.104 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:49.102+0000 7f286f600640 1 -- 172.21.15.32:0/635209388 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5594d3eb55f0 con 0x5594d3ea0ea0 2024-08-03T20:54:49.104 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:49.102+0000 7f2870000640 1 --2- 172.21.15.32:0/635209388 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x5594d3e9f360 0x5594d3e2fde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-03T20:54:49.104 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:49.103+0000 7f286f600640 1 --2- 172.21.15.32:0/635209388 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5594d3ea0ea0 0x5594d3eb6ae0 secure :-1 s=READY pgs=1093 cs=0 l=1 rev1=1 crypto rx=0x7f285c025dc0 tx=0x7f285c002fb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=db142694a394d748 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:54:49.104 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:49.103+0000 7f286e200640 1 -- 172.21.15.32:0/635209388 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f285c04d820 con 0x5594d3ea0ea0 2024-08-03T20:54:49.105 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:49.103+0000 7f286e200640 1 -- 172.21.15.32:0/635209388 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f285c04dcd0 con 0x5594d3ea0ea0 2024-08-03T20:54:49.105 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:49.103+0000 7f286e200640 1 -- 172.21.15.32:0/635209388 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f285c0566d0 con 0x5594d3ea0ea0 2024-08-03T20:54:49.106 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:49.104+0000 7f2876dd4f40 1 -- 172.21.15.32:0/635209388 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5594d3ea0ea0 msgr2=0x5594d3eb6ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:49.106 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:49.104+0000 7f2876dd4f40 1 --2- 172.21.15.32:0/635209388 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5594d3ea0ea0 0x5594d3eb6ae0 secure :-1 s=READY pgs=1093 cs=0 l=1 rev1=1 crypto rx=0x7f285c025dc0 tx=0x7f285c002fb0 comp rx=0 tx=0).stop 2024-08-03T20:54:49.106 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:49.104+0000 7f2876dd4f40 1 -- 172.21.15.32:0/635209388 shutdown_connections 2024-08-03T20:54:49.106 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:49.104+0000 7f2876dd4f40 1 --2- 172.21.15.32:0/635209388 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5594d3eb6330 0x5594d3eb8180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:49.106 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:49.104+0000 7f2876dd4f40 1 --2- 172.21.15.32:0/635209388 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5594d3ea0ea0 0x5594d3eb6ae0 unknown :-1 s=CLOSED pgs=1093 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:49.106 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:49.104+0000 7f2876dd4f40 1 --2- 172.21.15.32:0/635209388 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x5594d3e9f360 0x5594d3e2fde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:49.106 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:49.104+0000 7f2876dd4f40 1 -- 172.21.15.32:0/635209388 >> 172.21.15.32:0/635209388 conn(0x5594d3d08ac0 msgr2=0x5594d3eb8e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:54:49.106 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:49.104+0000 7f2876dd4f40 1 -- 172.21.15.32:0/635209388 shutdown_connections 2024-08-03T20:54:49.106 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:49.104+0000 7f2876dd4f40 1 -- 172.21.15.32:0/635209388 wait complete. 2024-08-03T20:54:49.106 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:49.105+0000 7f2876dd4f40 1 Processor -- start 2024-08-03T20:54:49.106 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:49.105+0000 7f2876dd4f40 1 -- start start 2024-08-03T20:54:49.107 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:49.106+0000 7f2876dd4f40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5594d3e9f360 0x5594d3eba9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:49.107 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:49.106+0000 7f2876dd4f40 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5594d3ea0ea0 0x5594d3e91c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:49.107 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:49.106+0000 7f2876dd4f40 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x5594d3eb6330 0x5594d3eb9c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:49.107 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:49.106+0000 7f2876dd4f40 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x5594d3e31890 con 0x5594d3ea0ea0 2024-08-03T20:54:49.107 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:49.106+0000 7f2876dd4f40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x5594d3eb58d0 con 0x5594d3e9f360 2024-08-03T20:54:49.108 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:49.106+0000 7f2876dd4f40 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x5594d3f561b0 con 0x5594d3eb6330 2024-08-03T20:54:49.108 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:49.107+0000 7f286f600640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5594d3ea0ea0 0x5594d3e91c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:49.108 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:49.107+0000 7f2870000640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5594d3e9f360 0x5594d3eba9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:49.108 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:49.107+0000 7f286f600640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5594d3ea0ea0 0x5594d3e91c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:37576/0 (socket says 172.21.15.32:37576) 2024-08-03T20:54:49.108 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:49.107+0000 7f286f600640 1 -- 172.21.15.32:0/1508769103 learned_addr learned my addr 172.21.15.32:0/1508769103 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:54:49.108 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:49.107+0000 7f2870a00640 1 --2- 172.21.15.32:0/1508769103 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x5594d3eb6330 0x5594d3eb9c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:49.109 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:49.107+0000 7f286f600640 1 -- 172.21.15.32:0/1508769103 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x5594d3eb6330 msgr2=0x5594d3eb9c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:49.109 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:49.107+0000 7f286f600640 1 --2- 172.21.15.32:0/1508769103 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x5594d3eb6330 0x5594d3eb9c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:49.109 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:49.107+0000 7f286f600640 1 -- 172.21.15.32:0/1508769103 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5594d3e9f360 msgr2=0x5594d3eba9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:49.109 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:49.107+0000 7f286f600640 1 --2- 172.21.15.32:0/1508769103 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5594d3e9f360 0x5594d3eba9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:49.110 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:49.107+0000 7f286f600640 1 -- 172.21.15.32:0/1508769103 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5594d3f56400 con 0x5594d3ea0ea0 2024-08-03T20:54:49.110 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:49.108+0000 7f286f600640 1 --2- 172.21.15.32:0/1508769103 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5594d3ea0ea0 0x5594d3e91c50 secure :-1 s=READY pgs=1094 cs=0 l=1 rev1=1 crypto rx=0x7f285c004f40 tx=0x7f285c057480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:54:49.110 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:49.108+0000 7f286ce00640 1 -- 172.21.15.32:0/1508769103 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f285c0576b0 con 0x5594d3ea0ea0 2024-08-03T20:54:49.110 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:49.108+0000 7f286ce00640 1 -- 172.21.15.32:0/1508769103 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f285c057940 con 0x5594d3ea0ea0 2024-08-03T20:54:49.110 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:49.108+0000 7f2876dd4f40 1 -- 172.21.15.32:0/1508769103 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x5594d3f566e0 con 0x5594d3ea0ea0 2024-08-03T20:54:49.110 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:49.108+0000 7f286ce00640 1 -- 172.21.15.32:0/1508769103 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f285c057b90 con 0x5594d3ea0ea0 2024-08-03T20:54:49.110 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:49.108+0000 7f2876dd4f40 1 -- 172.21.15.32:0/1508769103 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5594d3f56ca0 con 0x5594d3ea0ea0 2024-08-03T20:54:49.110 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:49.110+0000 7f286ce00640 1 -- 172.21.15.32:0/1508769103 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f285c072020 con 0x5594d3ea0ea0 2024-08-03T20:54:49.111 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:49.111+0000 7f286ce00640 1 --2- 172.21.15.32:0/1508769103 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f2850075a20 0x7f2850077f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:49.111 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:49.111+0000 7f286ce00640 1 -- 172.21.15.32:0/1508769103 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 5795+0+0 (secure 0 0 0) 0x7f285c0dc9f0 con 0x5594d3ea0ea0 2024-08-03T20:54:49.112 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:49.111+0000 7f2876dd4f40 1 --2- 172.21.15.32:0/1508769103 >> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] conn(0x5594d3e93430 0x5594d3e93840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:49.112 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:49.111+0000 7f2876dd4f40 1 -- 172.21.15.32:0/1508769103 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:1 12.10 12:0cf6ad0f:::10000000000.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e84) -- 0x5594d3f52b70 con 0x5594d3e93430 2024-08-03T20:54:49.112 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:49.111+0000 7f2870000640 1 --2- 172.21.15.32:0/1508769103 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f2850075a20 0x7f2850077f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:49.112 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:49.112+0000 7f2870a00640 1 --2- 172.21.15.32:0/1508769103 >> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] conn(0x5594d3e93430 0x5594d3e93840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:49.113 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:49.112+0000 7f2870000640 1 --2- 172.21.15.32:0/1508769103 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f2850075a20 0x7f2850077f30 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f2860000c00 tx=0x7f2860003750 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:54:49.113 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:49.112+0000 7f2870a00640 1 --2- 172.21.15.32:0/1508769103 >> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] conn(0x5594d3e93430 0x5594d3e93840 crc :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:54:49.115 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:49.114+0000 7f2870a00640 1 -- 172.21.15.32:0/1508769103 <== osd.0 v2:172.21.15.32:6816/3066156116 1 ==== osd_op_reply(1 10000000000.00000000 [delete] v84'42 uv42 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f286800b040 con 0x5594d3e93430 2024-08-03T20:54:49.115 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:49.114+0000 7f2876dd4f40 1 -- 172.21.15.32:0/1508769103 >> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] conn(0x5594d3e93430 msgr2=0x5594d3e93840 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:49.115 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:49.114+0000 7f2876dd4f40 1 --2- 172.21.15.32:0/1508769103 >> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] conn(0x5594d3e93430 0x5594d3e93840 crc :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:49.115 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:49.114+0000 7f2876dd4f40 1 -- 172.21.15.32:0/1508769103 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f2850075a20 msgr2=0x7f2850077f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:49.115 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:49.114+0000 7f2876dd4f40 1 --2- 172.21.15.32:0/1508769103 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f2850075a20 0x7f2850077f30 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f2860000c00 tx=0x7f2860003750 comp rx=0 tx=0).stop 2024-08-03T20:54:49.116 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:49.114+0000 7f2876dd4f40 1 -- 172.21.15.32:0/1508769103 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5594d3ea0ea0 msgr2=0x5594d3e91c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:49.116 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:49.114+0000 7f2876dd4f40 1 --2- 172.21.15.32:0/1508769103 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5594d3ea0ea0 0x5594d3e91c50 secure :-1 s=READY pgs=1094 cs=0 l=1 rev1=1 crypto rx=0x7f285c004f40 tx=0x7f285c057480 comp rx=0 tx=0).stop 2024-08-03T20:54:49.116 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:49.114+0000 7f2870a00640 1 -- 172.21.15.32:0/1508769103 reap_dead start 2024-08-03T20:54:49.116 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:49.114+0000 7f2876dd4f40 1 -- 172.21.15.32:0/1508769103 shutdown_connections 2024-08-03T20:54:49.116 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:49.114+0000 7f2876dd4f40 1 -- 172.21.15.32:0/1508769103 >> 172.21.15.32:0/1508769103 conn(0x5594d3d08ac0 msgr2=0x5594d3e2fda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:54:49.116 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:49.114+0000 7f2876dd4f40 1 -- 172.21.15.32:0/1508769103 shutdown_connections 2024-08-03T20:54:49.116 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:49.114+0000 7f2876dd4f40 1 -- 172.21.15.32:0/1508769103 wait complete. 2024-08-03T20:54:49.129 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs joinable true 2024-08-03T20:54:49.360 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:49.359+0000 7f1482400640 1 -- 172.21.15.32:0/4205003912 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f147c075960 msgr2=0x7f147c0cf5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:49.360 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:49.359+0000 7f1482400640 1 --2- 172.21.15.32:0/4205003912 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f147c075960 0x7f147c0cf5e0 secure :-1 s=READY pgs=1095 cs=0 l=1 rev1=1 crypto rx=0x7f1464005fd0 tx=0x7f146402a420 comp rx=0 tx=0).stop 2024-08-03T20:54:49.360 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:49.359+0000 7f1482400640 1 -- 172.21.15.32:0/4205003912 shutdown_connections 2024-08-03T20:54:49.360 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:49.359+0000 7f1482400640 1 --2- 172.21.15.32:0/4205003912 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f147c075960 0x7f147c0cf5e0 unknown :-1 s=CLOSED pgs=1095 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:49.360 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:49.359+0000 7f1482400640 1 --2- 172.21.15.32:0/4205003912 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f147c06f130 0x7f147c152510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:49.360 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:49.359+0000 7f1482400640 1 -- 172.21.15.32:0/4205003912 >> 172.21.15.32:0/4205003912 conn(0x7f147c0cd1e0 msgr2=0x7f147c0cd5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:54:49.360 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:49.359+0000 7f1482400640 1 -- 172.21.15.32:0/4205003912 shutdown_connections 2024-08-03T20:54:49.360 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:49.359+0000 7f1482400640 1 -- 172.21.15.32:0/4205003912 wait complete. 2024-08-03T20:54:49.361 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:49.361+0000 7f1482400640 1 Processor -- start 2024-08-03T20:54:49.361 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:49.361+0000 7f1482400640 1 -- start start 2024-08-03T20:54:49.368 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:49.363+0000 7f1482400640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f147c06f130 0x7f147c1499d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:49.368 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:49.363+0000 7f1482400640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f147c075960 0x7f147c0b5800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:49.368 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:49.363+0000 7f1482400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f147c075d70 0x7f147c0b5d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:49.368 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:49.363+0000 7f1482400640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f147c0cea40 con 0x7f147c075d70 2024-08-03T20:54:49.368 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:49.363+0000 7f1482400640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f147c0cfeb0 con 0x7f147c075960 2024-08-03T20:54:49.368 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:49.363+0000 7f1482400640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7f147c0cebc0 con 0x7f147c06f130 2024-08-03T20:54:49.368 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:49.363+0000 7f147be00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f147c075d70 0x7f147c0b5d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:49.368 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:49.363+0000 7f147be00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f147c075d70 0x7f147c0b5d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:37596/0 (socket says 172.21.15.32:37596) 2024-08-03T20:54:49.368 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:49.363+0000 7f147be00640 1 -- 172.21.15.32:0/2666028862 learned_addr learned my addr 172.21.15.32:0/2666028862 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:54:49.368 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:49.363+0000 7f1472a00640 1 --2- 172.21.15.32:0/2666028862 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f147c075960 0x7f147c0b5800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:49.369 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:49.363+0000 7f147b400640 1 --2- 172.21.15.32:0/2666028862 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f147c06f130 0x7f147c1499d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:49.369 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:49.363+0000 7f147be00640 1 -- 172.21.15.32:0/2666028862 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f147c06f130 msgr2=0x7f147c1499d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:49.369 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:49.363+0000 7f147be00640 1 --2- 172.21.15.32:0/2666028862 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f147c06f130 0x7f147c1499d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:49.369 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:49.363+0000 7f147be00640 1 -- 172.21.15.32:0/2666028862 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f147c075960 msgr2=0x7f147c0b5800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:49.369 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:49.363+0000 7f147be00640 1 --2- 172.21.15.32:0/2666028862 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f147c075960 0x7f147c0b5800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:49.369 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:49.363+0000 7f147be00640 1 -- 172.21.15.32:0/2666028862 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f147c0c6f30 con 0x7f147c075d70 2024-08-03T20:54:49.369 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:49.364+0000 7f147be00640 1 --2- 172.21.15.32:0/2666028862 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f147c075d70 0x7f147c0b5d40 secure :-1 s=READY pgs=1096 cs=0 l=1 rev1=1 crypto rx=0x7f147400ba10 tx=0x7f147400bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:54:49.369 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:49.364+0000 7f1473e00640 1 -- 172.21.15.32:0/2666028862 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1474019070 con 0x7f147c075d70 2024-08-03T20:54:49.369 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:49.364+0000 7f1473e00640 1 -- 172.21.15.32:0/2666028862 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f147400cc30 con 0x7f147c075d70 2024-08-03T20:54:49.370 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:49.364+0000 7f1473e00640 1 -- 172.21.15.32:0/2666028862 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14740146a0 con 0x7f147c075d70 2024-08-03T20:54:49.370 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:49.364+0000 7f1482400640 1 -- 172.21.15.32:0/2666028862 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f147c0c4ed0 con 0x7f147c075d70 2024-08-03T20:54:49.370 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:49.364+0000 7f1482400640 1 -- 172.21.15.32:0/2666028862 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f147c0c7940 con 0x7f147c075d70 2024-08-03T20:54:49.370 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:49.365+0000 7f1482400640 1 -- 172.21.15.32:0/2666028862 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f147c0cf070 con 0x7f147c075d70 2024-08-03T20:54:49.370 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:49.367+0000 7f1473e00640 1 -- 172.21.15.32:0/2666028862 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f147401c070 con 0x7f147c075d70 2024-08-03T20:54:49.370 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:49.368+0000 7f1473e00640 1 --2- 172.21.15.32:0/2666028862 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f1458077010 0x7f1458079520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:49.371 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:49.368+0000 7f1473e00640 1 -- 172.21.15.32:0/2666028862 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 5795+0+0 (secure 0 0 0) 0x7f1474098630 con 0x7f147c075d70 2024-08-03T20:54:49.371 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:49.368+0000 7f147b400640 1 --2- 172.21.15.32:0/2666028862 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f1458077010 0x7f1458079520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:49.371 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:49.369+0000 7f147b400640 1 --2- 172.21.15.32:0/2666028862 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f1458077010 0x7f1458079520 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7f146c005d70 tx=0x7f146c005d00 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:54:49.372 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:49.372+0000 7f1473e00640 1 -- 172.21.15.32:0/2666028862 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7f147400ce20 con 0x7f147c075d70 2024-08-03T20:54:49.623 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:49.622+0000 7f1482400640 1 -- 172.21.15.32:0/2666028862 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "joinable", "val": "true"} v 0) -- 0x7f147c0ce4b0 con 0x7f147c075d70 2024-08-03T20:54:50.003 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:50.002+0000 7f1473e00640 1 -- 172.21.15.32:0/2666028862 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "joinable", "val": "true"}]=0 cephfs marked joinable; MDS may join as newly active. v96) ==== 162+0+0 (secure 0 0 0) 0x7f1474061850 con 0x7f147c075d70 2024-08-03T20:54:50.003 INFO:teuthology.orchestra.run.smithi032.stderr:cephfs marked joinable; MDS may join as newly active. 2024-08-03T20:54:50.007 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:50.006+0000 7f1482400640 1 -- 172.21.15.32:0/2666028862 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f1458077010 msgr2=0x7f1458079520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:50.007 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:50.006+0000 7f1482400640 1 --2- 172.21.15.32:0/2666028862 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f1458077010 0x7f1458079520 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7f146c005d70 tx=0x7f146c005d00 comp rx=0 tx=0).stop 2024-08-03T20:54:50.007 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:50.006+0000 7f1482400640 1 -- 172.21.15.32:0/2666028862 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f147c075d70 msgr2=0x7f147c0b5d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:50.007 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:50.006+0000 7f1482400640 1 --2- 172.21.15.32:0/2666028862 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f147c075d70 0x7f147c0b5d40 secure :-1 s=READY pgs=1096 cs=0 l=1 rev1=1 crypto rx=0x7f147400ba10 tx=0x7f147400bed0 comp rx=0 tx=0).stop 2024-08-03T20:54:50.007 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:50.006+0000 7f1482400640 1 -- 172.21.15.32:0/2666028862 shutdown_connections 2024-08-03T20:54:50.007 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:50.006+0000 7f1482400640 1 --2- 172.21.15.32:0/2666028862 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f1458077010 0x7f1458079520 unknown :-1 s=CLOSED pgs=455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:50.007 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:50.006+0000 7f1482400640 1 --2- 172.21.15.32:0/2666028862 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f147c075960 0x7f147c0b5800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:50.007 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:50.006+0000 7f1482400640 1 --2- 172.21.15.32:0/2666028862 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f147c075d70 0x7f147c0b5d40 unknown :-1 s=CLOSED pgs=1096 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:50.007 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:50.006+0000 7f1482400640 1 --2- 172.21.15.32:0/2666028862 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f147c06f130 0x7f147c1499d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:50.007 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:50.006+0000 7f1482400640 1 -- 172.21.15.32:0/2666028862 >> 172.21.15.32:0/2666028862 conn(0x7f147c0cd1e0 msgr2=0x7f147c15e580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:54:50.007 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:50.006+0000 7f1482400640 1 -- 172.21.15.32:0/2666028862 shutdown_connections 2024-08-03T20:54:50.008 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:50.006+0000 7f1482400640 1 -- 172.21.15.32:0/2666028862 wait complete. 2024-08-03T20:54:50.435 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-08-03T20:54:50.673 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:50.672+0000 7fecad800640 1 -- 172.21.15.32:0/162347239 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7feca80d4bb0 msgr2=0x7feca80d1c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:50.673 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:50.672+0000 7fecad800640 1 --2- 172.21.15.32:0/162347239 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7feca80d4bb0 0x7feca80d1c60 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fec98009860 tx=0x7fec9802a520 comp rx=0 tx=0).stop 2024-08-03T20:54:50.673 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:50.672+0000 7fecad800640 1 -- 172.21.15.32:0/162347239 shutdown_connections 2024-08-03T20:54:50.673 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:50.672+0000 7fecad800640 1 --2- 172.21.15.32:0/162347239 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7feca80d4bb0 0x7feca80d1c60 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:50.673 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:50.672+0000 7fecad800640 1 -- 172.21.15.32:0/162347239 >> 172.21.15.32:0/162347239 conn(0x7feca80d4090 msgr2=0x7feca80d44a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:54:50.674 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:50.672+0000 7fecad800640 1 -- 172.21.15.32:0/162347239 shutdown_connections 2024-08-03T20:54:50.674 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:50.672+0000 7fecad800640 1 -- 172.21.15.32:0/162347239 wait complete. 2024-08-03T20:54:50.675 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:50.674+0000 7fecad800640 1 Processor -- start 2024-08-03T20:54:50.675 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:50.675+0000 7fecad800640 1 -- start start 2024-08-03T20:54:50.678 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:50.676+0000 7fecad800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7feca80cec30 0x7feca80cb980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:50.679 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:50.676+0000 7fecad800640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7feca80d2290 0x7feca815e180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:50.679 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:50.676+0000 7fecad800640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7feca80d4bb0 0x7feca80b7300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:50.679 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:50.676+0000 7fecad800640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7feca80cfde0 con 0x7feca80cec30 2024-08-03T20:54:50.679 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:50.676+0000 7fecad800640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7feca80cfc60 con 0x7feca80d4bb0 2024-08-03T20:54:50.679 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:50.676+0000 7fecad800640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7feca8169d70 con 0x7feca80d2290 2024-08-03T20:54:50.679 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:50.676+0000 7feca7e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7feca80cec30 0x7feca80cb980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:50.679 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:50.676+0000 7feca7e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7feca80cec30 0x7feca80cb980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:49942/0 (socket says 172.21.15.32:49942) 2024-08-03T20:54:50.679 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:50.676+0000 7feca7e00640 1 -- 172.21.15.32:0/3216907391 learned_addr learned my addr 172.21.15.32:0/3216907391 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:54:50.679 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:50.677+0000 7fecace00640 1 --2- 172.21.15.32:0/3216907391 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7feca80d4bb0 0x7feca80b7300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:50.679 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:50.677+0000 7feca7400640 1 --2- 172.21.15.32:0/3216907391 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7feca80d2290 0x7feca815e180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:50.680 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:50.677+0000 7feca7e00640 1 -- 172.21.15.32:0/3216907391 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7feca80d2290 msgr2=0x7feca815e180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:50.680 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:50.677+0000 7feca7e00640 1 --2- 172.21.15.32:0/3216907391 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7feca80d2290 0x7feca815e180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:50.680 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:50.677+0000 7feca7e00640 1 -- 172.21.15.32:0/3216907391 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7feca80d4bb0 msgr2=0x7feca80b7300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:50.680 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:50.677+0000 7feca7e00640 1 --2- 172.21.15.32:0/3216907391 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7feca80d4bb0 0x7feca80b7300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:50.680 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:50.677+0000 7feca7e00640 1 -- 172.21.15.32:0/3216907391 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feca815e6c0 con 0x7feca80cec30 2024-08-03T20:54:50.680 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:50.677+0000 7fecace00640 1 --2- 172.21.15.32:0/3216907391 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7feca80d4bb0 0x7feca80b7300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-03T20:54:50.680 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:50.677+0000 7feca7e00640 1 --2- 172.21.15.32:0/3216907391 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7feca80cec30 0x7feca80cb980 secure :-1 s=READY pgs=1098 cs=0 l=1 rev1=1 crypto rx=0x7fec9802ab10 tx=0x7fec98033ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:54:50.680 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:50.677+0000 7fec9fe00640 1 -- 172.21.15.32:0/3216907391 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec98031b50 con 0x7feca80cec30 2024-08-03T20:54:50.680 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:50.678+0000 7fec9fe00640 1 -- 172.21.15.32:0/3216907391 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fec98031de0 con 0x7feca80cec30 2024-08-03T20:54:50.680 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:50.678+0000 7fec9fe00640 1 -- 172.21.15.32:0/3216907391 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec98033330 con 0x7feca80cec30 2024-08-03T20:54:50.680 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:50.678+0000 7fecad800640 1 -- 172.21.15.32:0/3216907391 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7feca80ca490 con 0x7feca80cec30 2024-08-03T20:54:50.680 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:50.678+0000 7fecad800640 1 -- 172.21.15.32:0/3216907391 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7feca80c9280 con 0x7feca80cec30 2024-08-03T20:54:50.682 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:50.679+0000 7fecad800640 1 -- 172.21.15.32:0/3216907391 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7feca808c100 con 0x7feca80cec30 2024-08-03T20:54:50.682 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:50.680+0000 7fec9fe00640 1 -- 172.21.15.32:0/3216907391 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7fec980334d0 con 0x7feca80cec30 2024-08-03T20:54:50.682 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:50.681+0000 7fec9fe00640 1 --2- 172.21.15.32:0/3216907391 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fec88077040 0x7fec88079550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:50.682 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:50.681+0000 7fec9fe00640 1 -- 172.21.15.32:0/3216907391 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 5795+0+0 (secure 0 0 0) 0x7fec980b9d30 con 0x7feca80cec30 2024-08-03T20:54:50.682 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:50.681+0000 7feca7400640 1 --2- 172.21.15.32:0/3216907391 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fec88077040 0x7fec88079550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:50.682 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:50.682+0000 7feca7400640 1 --2- 172.21.15.32:0/3216907391 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fec88077040 0x7fec88079550 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7feca80cf650 tx=0x7fec94009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:54:50.686 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:50.685+0000 7fec9fe00640 1 -- 172.21.15.32:0/3216907391 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7fec9803f030 con 0x7feca80cec30 2024-08-03T20:54:50.934 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:50.934+0000 7fecad800640 1 -- 172.21.15.32:0/3216907391 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7feca80d1c40 con 0x7feca80cec30 2024-08-03T20:54:50.935 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:50.934+0000 7fec9fe00640 1 -- 172.21.15.32:0/3216907391 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 96 v96) ==== 94+0+5222 (secure 0 0 0) 0x7fec98082e40 con 0x7feca80cec30 2024-08-03T20:54:50.935 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-03T20:54:50.935 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":96,"btime":"2024-08-03T20:54:50:001517+0000","default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6439,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6835/531080160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6834","nonce":531080160},{"type":"v1","addr":"172.21.15.118:6835","nonce":531080160}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":84},{"gid":6454,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6837/2796882319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6836","nonce":2796882319},{"type":"v1","addr":"172.21.15.118:6837","nonce":2796882319}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":86},{"gid":6813,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6834/2686933890","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":2686933890},{"type":"v1","addr":"172.21.15.32:6834","nonce":2686933890}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":85}],"filesystems":[{"mdsmap":{"epoch":96,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-08-03T20:54:26.950518+0000","modified":"2024-08-03T20:54:50.001499+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":84,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6975},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6975":{"gid":6975,"name":"a","rank":0,"incarnation":96,"state":"up:replay","state_seq":1,"addr":"172.21.15.32:6835/2194902489","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6833","nonce":2194902489},{"type":"v1","addr":"172.21.15.32:6835","nonce":2194902489}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":6}]} 2024-08-03T20:54:50.936 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 96 2024-08-03T20:54:50.937 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:50.937+0000 7fecad800640 1 -- 172.21.15.32:0/3216907391 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fec88077040 msgr2=0x7fec88079550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:50.938 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:50.937+0000 7fecad800640 1 --2- 172.21.15.32:0/3216907391 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fec88077040 0x7fec88079550 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7feca80cf650 tx=0x7fec94009040 comp rx=0 tx=0).stop 2024-08-03T20:54:50.938 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:50.937+0000 7fecad800640 1 -- 172.21.15.32:0/3216907391 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7feca80cec30 msgr2=0x7feca80cb980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:50.938 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:50.937+0000 7fecad800640 1 --2- 172.21.15.32:0/3216907391 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7feca80cec30 0x7feca80cb980 secure :-1 s=READY pgs=1098 cs=0 l=1 rev1=1 crypto rx=0x7fec9802ab10 tx=0x7fec98033ba0 comp rx=0 tx=0).stop 2024-08-03T20:54:50.938 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:50.937+0000 7fecad800640 1 -- 172.21.15.32:0/3216907391 shutdown_connections 2024-08-03T20:54:50.938 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:50.937+0000 7fecad800640 1 --2- 172.21.15.32:0/3216907391 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fec88077040 0x7fec88079550 unknown :-1 s=CLOSED pgs=456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:50.938 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:50.937+0000 7fecad800640 1 --2- 172.21.15.32:0/3216907391 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7feca80d4bb0 0x7feca80b7300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:50.938 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:50.937+0000 7fecad800640 1 --2- 172.21.15.32:0/3216907391 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7feca80d2290 0x7feca815e180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:50.938 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:50.937+0000 7fecad800640 1 --2- 172.21.15.32:0/3216907391 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7feca80cec30 0x7feca80cb980 unknown :-1 s=CLOSED pgs=1098 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:50.938 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:50.937+0000 7fecad800640 1 -- 172.21.15.32:0/3216907391 >> 172.21.15.32:0/3216907391 conn(0x7feca80d4090 msgr2=0x7feca806e7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:54:50.938 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:50.937+0000 7fecad800640 1 -- 172.21.15.32:0/3216907391 shutdown_connections 2024-08-03T20:54:50.938 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:50.937+0000 7fecad800640 1 -- 172.21.15.32:0/3216907391 wait complete. 2024-08-03T20:54:50.960 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 96, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-08-03T20:54:26.950518+0000', 'modified': '2024-08-03T20:54:50.001499+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 84, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 6975}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6975': {'gid': 6975, 'name': 'a', 'rank': 0, 'incarnation': 96, 'state': 'up:replay', 'state_seq': 1, 'addr': '172.21.15.32:6835/2194902489', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6833', 'nonce': 2194902489}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 2194902489}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [13], 'metadata_pool': 12, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 0, 'qdb_cluster': []} 2024-08-03T20:54:50.961 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6975:up:replay 2024-08-03T20:54:50.961 DEBUG:tasks.ceph_test_case:wait_until_true: waiting (timeout=60 retry_count=0)... 2024-08-03T20:54:54.490 INFO:tasks.ceph.mds.a.smithi032.stderr:2024-08-03T20:54:54.489+0000 7f1425200640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-08-03T20:54:56.364 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-08-03T20:54:56.605 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:56.604+0000 7f0156e00640 1 -- 172.21.15.32:0/1017564808 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f01500b38b0 msgr2=0x7f01501523f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:56.606 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:56.604+0000 7f0156e00640 1 --2- 172.21.15.32:0/1017564808 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f01500b38b0 0x7f01501523f0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f0140005fd0 tx=0x7f014002a5d0 comp rx=0 tx=0).stop 2024-08-03T20:54:56.606 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:56.604+0000 7f0156e00640 1 -- 172.21.15.32:0/1017564808 shutdown_connections 2024-08-03T20:54:56.606 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:56.604+0000 7f0156e00640 1 --2- 172.21.15.32:0/1017564808 >> v2:172.21.15.118:3300/0 conn(0x7f015015f610 0x7f01500cdf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:56.606 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:56.604+0000 7f0156e00640 1 --2- 172.21.15.32:0/1017564808 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f01500b38b0 0x7f01501523f0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:56.606 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:56.604+0000 7f0156e00640 1 -- 172.21.15.32:0/1017564808 >> 172.21.15.32:0/1017564808 conn(0x7f01500b1bd0 msgr2=0x7f01500b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:54:56.606 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:56.604+0000 7f0156e00640 1 -- 172.21.15.32:0/1017564808 shutdown_connections 2024-08-03T20:54:56.607 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:56.604+0000 7f0156e00640 1 -- 172.21.15.32:0/1017564808 wait complete. 2024-08-03T20:54:56.607 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:56.607+0000 7f0156e00640 1 Processor -- start 2024-08-03T20:54:56.607 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:56.607+0000 7f0156e00640 1 -- start start 2024-08-03T20:54:56.609 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:56.608+0000 7f0156e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f01500b38b0 0x7f0150141290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:56.609 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:56.608+0000 7f0156e00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f01500b4ce0 0x7f01501515e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:56.610 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:56.608+0000 7f0156e00640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f0150151b20 0x7f01501f8930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:56.610 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:56.608+0000 7f0156e00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f01500cea90 con 0x7f01500b38b0 2024-08-03T20:54:56.610 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:56.608+0000 7f0156e00640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f01500ce910 con 0x7f01500b4ce0 2024-08-03T20:54:56.610 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:56.608+0000 7f0156e00640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7f01500cec10 con 0x7f0150151b20 2024-08-03T20:54:56.610 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:56.608+0000 7f014fe00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f01500b38b0 0x7f0150141290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:56.610 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:56.609+0000 7f014fe00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f01500b38b0 0x7f0150141290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:49944/0 (socket says 172.21.15.32:49944) 2024-08-03T20:54:56.610 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:56.609+0000 7f014fe00640 1 -- 172.21.15.32:0/2296809340 learned_addr learned my addr 172.21.15.32:0/2296809340 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:54:56.610 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:56.609+0000 7f014f400640 1 --2- 172.21.15.32:0/2296809340 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f01500b4ce0 0x7f01501515e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:56.610 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:56.609+0000 7f0154e00640 1 --2- 172.21.15.32:0/2296809340 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f0150151b20 0x7f01501f8930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:56.610 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:56.609+0000 7f014fe00640 1 -- 172.21.15.32:0/2296809340 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f0150151b20 msgr2=0x7f01501f8930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:56.611 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:56.609+0000 7f014fe00640 1 --2- 172.21.15.32:0/2296809340 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f0150151b20 0x7f01501f8930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:56.611 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:56.609+0000 7f014fe00640 1 -- 172.21.15.32:0/2296809340 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f01500b4ce0 msgr2=0x7f01501515e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:56.611 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:56.609+0000 7f014fe00640 1 --2- 172.21.15.32:0/2296809340 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f01500b4ce0 0x7f01501515e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:56.611 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:56.609+0000 7f014fe00640 1 -- 172.21.15.32:0/2296809340 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f01501f5100 con 0x7f01500b38b0 2024-08-03T20:54:56.612 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:56.609+0000 7f014fe00640 1 --2- 172.21.15.32:0/2296809340 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f01500b38b0 0x7f0150141290 secure :-1 s=READY pgs=1099 cs=0 l=1 rev1=1 crypto rx=0x7f0140034040 tx=0x7f01400035f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:54:56.612 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:56.610+0000 7f0147e00640 1 -- 172.21.15.32:0/2296809340 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f014003e030 con 0x7f01500b38b0 2024-08-03T20:54:56.612 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:56.610+0000 7f0147e00640 1 -- 172.21.15.32:0/2296809340 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f0140003ae0 con 0x7f01500b38b0 2024-08-03T20:54:56.612 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:56.610+0000 7f0147e00640 1 -- 172.21.15.32:0/2296809340 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0140035040 con 0x7f01500b38b0 2024-08-03T20:54:56.613 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:56.610+0000 7f0156e00640 1 -- 172.21.15.32:0/2296809340 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0150151f30 con 0x7f01500b38b0 2024-08-03T20:54:56.613 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:56.610+0000 7f0156e00640 1 -- 172.21.15.32:0/2296809340 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f01501f5660 con 0x7f01500b38b0 2024-08-03T20:54:56.613 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:56.611+0000 7f0156e00640 1 -- 172.21.15.32:0/2296809340 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f01500cf150 con 0x7f01500b38b0 2024-08-03T20:54:56.614 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:56.612+0000 7f0147e00640 1 -- 172.21.15.32:0/2296809340 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f0140041030 con 0x7f01500b38b0 2024-08-03T20:54:56.614 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:56.613+0000 7f0147e00640 1 --2- 172.21.15.32:0/2296809340 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f0130077010 0x7f0130079520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:56.614 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:56.613+0000 7f0147e00640 1 -- 172.21.15.32:0/2296809340 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 5795+0+0 (secure 0 0 0) 0x7f01400b90a0 con 0x7f01500b38b0 2024-08-03T20:54:56.614 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:56.614+0000 7f014f400640 1 --2- 172.21.15.32:0/2296809340 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f0130077010 0x7f0130079520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:56.615 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:56.615+0000 7f014f400640 1 --2- 172.21.15.32:0/2296809340 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f0130077010 0x7f0130079520 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f0150152590 tx=0x7f013c009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:54:56.618 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:56.618+0000 7f0147e00640 1 -- 172.21.15.32:0/2296809340 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7f0150151f30 con 0x7f01500b38b0 2024-08-03T20:54:56.864 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:56.863+0000 7f0156e00640 1 -- 172.21.15.32:0/2296809340 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f0150152710 con 0x7f01500b38b0 2024-08-03T20:54:56.864 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:56.864+0000 7f0147e00640 1 -- 172.21.15.32:0/2296809340 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 99 v99) ==== 94+0+5229 (secure 0 0 0) 0x7f01501f5100 con 0x7f01500b38b0 2024-08-03T20:54:56.865 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-03T20:54:56.865 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":99,"btime":"2024-08-03T20:54:53:011658+0000","default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6439,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6835/531080160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6834","nonce":531080160},{"type":"v1","addr":"172.21.15.118:6835","nonce":531080160}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":84},{"gid":6454,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6837/2796882319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6836","nonce":2796882319},{"type":"v1","addr":"172.21.15.118:6837","nonce":2796882319}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":86},{"gid":6813,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6834/2686933890","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":2686933890},{"type":"v1","addr":"172.21.15.32:6834","nonce":2686933890}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":85}],"filesystems":[{"mdsmap":{"epoch":99,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-08-03T20:54:26.950518+0000","modified":"2024-08-03T20:54:53.011648+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":84,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6975},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6975":{"gid":6975,"name":"a","rank":0,"incarnation":96,"state":"up:active","state_seq":6,"addr":"172.21.15.32:6835/2194902489","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6833","nonce":2194902489},{"type":"v1","addr":"172.21.15.32:6835","nonce":2194902489}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6975,"qdb_cluster":[6975]},"id":6}]} 2024-08-03T20:54:56.865 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 99 2024-08-03T20:54:56.867 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:56.866+0000 7f0156e00640 1 -- 172.21.15.32:0/2296809340 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f0130077010 msgr2=0x7f0130079520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:56.867 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:56.866+0000 7f0156e00640 1 --2- 172.21.15.32:0/2296809340 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f0130077010 0x7f0130079520 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f0150152590 tx=0x7f013c009040 comp rx=0 tx=0).stop 2024-08-03T20:54:56.867 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:56.866+0000 7f0156e00640 1 -- 172.21.15.32:0/2296809340 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f01500b38b0 msgr2=0x7f0150141290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:56.867 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:56.866+0000 7f0156e00640 1 --2- 172.21.15.32:0/2296809340 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f01500b38b0 0x7f0150141290 secure :-1 s=READY pgs=1099 cs=0 l=1 rev1=1 crypto rx=0x7f0140034040 tx=0x7f01400035f0 comp rx=0 tx=0).stop 2024-08-03T20:54:56.868 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:56.866+0000 7f0156e00640 1 -- 172.21.15.32:0/2296809340 shutdown_connections 2024-08-03T20:54:56.868 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:56.866+0000 7f0156e00640 1 --2- 172.21.15.32:0/2296809340 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f0130077010 0x7f0130079520 unknown :-1 s=CLOSED pgs=457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:56.868 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:56.866+0000 7f0156e00640 1 --2- 172.21.15.32:0/2296809340 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f01500b4ce0 0x7f01501515e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:56.868 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:56.866+0000 7f0156e00640 1 --2- 172.21.15.32:0/2296809340 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f0150151b20 0x7f01501f8930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:56.868 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:56.866+0000 7f0156e00640 1 --2- 172.21.15.32:0/2296809340 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f01500b38b0 0x7f0150141290 unknown :-1 s=CLOSED pgs=1099 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:56.868 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:56.866+0000 7f0156e00640 1 -- 172.21.15.32:0/2296809340 >> 172.21.15.32:0/2296809340 conn(0x7f01500b1bd0 msgr2=0x7f015015a140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:54:56.868 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:56.866+0000 7f0156e00640 1 -- 172.21.15.32:0/2296809340 shutdown_connections 2024-08-03T20:54:56.868 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:56.867+0000 7f0156e00640 1 -- 172.21.15.32:0/2296809340 wait complete. 2024-08-03T20:54:56.889 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 99, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-08-03T20:54:26.950518+0000', 'modified': '2024-08-03T20:54:53.011648+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 84, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 6975}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6975': {'gid': 6975, 'name': 'a', 'rank': 0, 'incarnation': 96, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.32:6835/2194902489', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6833', 'nonce': 2194902489}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 2194902489}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [13], 'metadata_pool': 12, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 6975, 'qdb_cluster': [6975]} 2024-08-03T20:54:56.889 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-08-03T20:54:56.889 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2024-08-03T20:54:57.119 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:57.118+0000 7fb8a9c00640 1 -- 172.21.15.32:0/4280032182 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb8a40b7760 msgr2=0x7fb8a40cdba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:57.119 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:57.118+0000 7fb8a9c00640 1 --2- 172.21.15.32:0/4280032182 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb8a40b7760 0x7fb8a40cdba0 secure :-1 s=READY pgs=1100 cs=0 l=1 rev1=1 crypto rx=0x7fb8a40742d0 tx=0x7fb89402a3f0 comp rx=0 tx=0).stop 2024-08-03T20:54:57.120 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:57.119+0000 7fb8a9c00640 1 -- 172.21.15.32:0/4280032182 shutdown_connections 2024-08-03T20:54:57.120 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:57.119+0000 7fb8a9c00640 1 --2- 172.21.15.32:0/4280032182 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fb8a409ac30 0x7fb8a409b040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:57.120 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:57.119+0000 7fb8a9c00640 1 --2- 172.21.15.32:0/4280032182 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb8a40b7760 0x7fb8a40cdba0 unknown :-1 s=CLOSED pgs=1100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:57.120 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:57.119+0000 7fb8a9c00640 1 -- 172.21.15.32:0/4280032182 >> 172.21.15.32:0/4280032182 conn(0x7fb8a40b3bb0 msgr2=0x7fb8a40b3fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:54:57.120 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:57.119+0000 7fb8a9c00640 1 -- 172.21.15.32:0/4280032182 shutdown_connections 2024-08-03T20:54:57.120 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:57.119+0000 7fb8a9c00640 1 -- 172.21.15.32:0/4280032182 wait complete. 2024-08-03T20:54:57.121 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:57.121+0000 7fb8a9c00640 1 Processor -- start 2024-08-03T20:54:57.121 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:57.121+0000 7fb8a9c00640 1 -- start start 2024-08-03T20:54:57.124 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:57.122+0000 7fb8a9c00640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fb8a409ac30 0x7fb8a4163e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:57.124 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:57.122+0000 7fb8a9c00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb8a40b7760 0x7fb8a4166200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:57.124 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:57.122+0000 7fb8a9c00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb8a40cfc90 0x7fb8a4166740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:57.124 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:57.123+0000 7fb8a9c00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fb8a40b2b70 con 0x7fb8a40b7760 2024-08-03T20:54:57.124 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:57.123+0000 7fb8a9c00640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7fb8a40b29f0 con 0x7fb8a40cfc90 2024-08-03T20:54:57.124 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:57.123+0000 7fb8a9c00640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7fb8a4166d90 con 0x7fb8a409ac30 2024-08-03T20:54:57.124 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:57.123+0000 7fb89be00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb8a40b7760 0x7fb8a4166200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:57.124 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:57.123+0000 7fb89be00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb8a40b7760 0x7fb8a4166200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:49958/0 (socket says 172.21.15.32:49958) 2024-08-03T20:54:57.124 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:57.123+0000 7fb89be00640 1 -- 172.21.15.32:0/3855884075 learned_addr learned my addr 172.21.15.32:0/3855884075 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:54:57.124 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:57.123+0000 7fb8a2a00640 1 --2- 172.21.15.32:0/3855884075 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fb8a409ac30 0x7fb8a4163e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:57.125 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:57.123+0000 7fb8a3400640 1 --2- 172.21.15.32:0/3855884075 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb8a40cfc90 0x7fb8a4166740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:57.125 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:57.123+0000 7fb89be00640 1 -- 172.21.15.32:0/3855884075 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fb8a409ac30 msgr2=0x7fb8a4163e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:57.125 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:57.123+0000 7fb89be00640 1 --2- 172.21.15.32:0/3855884075 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fb8a409ac30 0x7fb8a4163e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:57.125 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:57.123+0000 7fb89be00640 1 -- 172.21.15.32:0/3855884075 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb8a40cfc90 msgr2=0x7fb8a4166740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:57.125 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:57.123+0000 7fb89be00640 1 --2- 172.21.15.32:0/3855884075 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb8a40cfc90 0x7fb8a4166740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:57.125 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:57.123+0000 7fb89be00640 1 -- 172.21.15.32:0/3855884075 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb8a4165430 con 0x7fb8a40b7760 2024-08-03T20:54:57.125 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:57.124+0000 7fb89be00640 1 --2- 172.21.15.32:0/3855884075 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb8a40b7760 0x7fb8a4166200 secure :-1 s=READY pgs=1101 cs=0 l=1 rev1=1 crypto rx=0x7fb88c00d6b0 tx=0x7fb88c00db70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:54:57.126 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:57.124+0000 7fb89aa00640 1 -- 172.21.15.32:0/3855884075 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb88c017070 con 0x7fb8a40b7760 2024-08-03T20:54:57.126 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:57.124+0000 7fb89aa00640 1 -- 172.21.15.32:0/3855884075 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fb88c005310 con 0x7fb8a40b7760 2024-08-03T20:54:57.126 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:57.124+0000 7fb8a9c00640 1 -- 172.21.15.32:0/3855884075 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb8a4165710 con 0x7fb8a40b7760 2024-08-03T20:54:57.126 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:57.124+0000 7fb89aa00640 1 -- 172.21.15.32:0/3855884075 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb88c0097c0 con 0x7fb8a40b7760 2024-08-03T20:54:57.126 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:57.124+0000 7fb8a9c00640 1 -- 172.21.15.32:0/3855884075 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb8a40ca260 con 0x7fb8a40b7760 2024-08-03T20:54:57.128 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:57.128+0000 7fb89aa00640 1 -- 172.21.15.32:0/3855884075 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7fb88c009960 con 0x7fb8a40b7760 2024-08-03T20:54:57.129 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:57.129+0000 7fb89aa00640 1 --2- 172.21.15.32:0/3855884075 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fb880077010 0x7fb880079520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:57.129 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:57.129+0000 7fb89aa00640 1 -- 172.21.15.32:0/3855884075 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 5795+0+0 (secure 0 0 0) 0x7fb88c098920 con 0x7fb8a40b7760 2024-08-03T20:54:57.129 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:57.129+0000 7fb8b0c08740 1 Processor -- start 2024-08-03T20:54:57.129 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:57.129+0000 7fb8b0c08740 1 -- start start 2024-08-03T20:54:57.130 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:57.129+0000 7fb8a2a00640 1 --2- 172.21.15.32:0/3855884075 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fb880077010 0x7fb880079520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:57.131 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:57.130+0000 7fb8a2a00640 1 --2- 172.21.15.32:0/3855884075 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fb880077010 0x7fb880079520 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7fb894034000 tx=0x7fb894004730 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:54:57.132 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:57.130+0000 7fb8b0c08740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5610bde0a270 0x5610be1f4e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:57.132 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:57.130+0000 7fb8b0c08740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5610bdde13b0 0x5610be1c9ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:57.132 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:57.130+0000 7fb8b0c08740 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x5610bdde9590 0x5610be1ca220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:57.132 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:57.130+0000 7fb8b0c08740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x5610be122a60 con 0x5610bde0a270 2024-08-03T20:54:57.132 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:57.130+0000 7fb8b0c08740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x5610be111f60 con 0x5610bdde13b0 2024-08-03T20:54:57.132 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:57.130+0000 7fb8b0c08740 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x5610be115fa0 con 0x5610bdde9590 2024-08-03T20:54:57.132 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:57.130+0000 7fb8a2a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5610bde0a270 0x5610be1f4e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:57.132 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:57.131+0000 7fb8a2a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5610bde0a270 0x5610be1f4e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:49966/0 (socket says 172.21.15.32:49966) 2024-08-03T20:54:57.132 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:57.131+0000 7fb8a2a00640 1 -- 172.21.15.32:0/3767617940 learned_addr learned my addr 172.21.15.32:0/3767617940 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:54:57.132 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:57.131+0000 7fb89be00640 1 --2- 172.21.15.32:0/3767617940 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5610bdde13b0 0x5610be1c9ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:57.132 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:57.131+0000 7fb8a3400640 1 --2- 172.21.15.32:0/3767617940 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x5610bdde9590 0x5610be1ca220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:57.132 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:57.131+0000 7fb8a2a00640 1 -- 172.21.15.32:0/3767617940 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x5610bdde9590 msgr2=0x5610be1ca220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:54:57.132 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:57.131+0000 7fb8a2a00640 1 --2- 172.21.15.32:0/3767617940 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x5610bdde9590 0x5610be1ca220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:57.133 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:57.131+0000 7fb8a2a00640 1 -- 172.21.15.32:0/3767617940 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5610bdde13b0 msgr2=0x5610be1c9ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:54:57.133 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:57.131+0000 7fb8a2a00640 1 --2- 172.21.15.32:0/3767617940 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5610bdde13b0 0x5610be1c9ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:57.133 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:57.131+0000 7fb8a2a00640 1 -- 172.21.15.32:0/3767617940 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5610bde75850 con 0x5610bde0a270 2024-08-03T20:54:57.133 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:57.131+0000 7fb8a2a00640 1 --2- 172.21.15.32:0/3767617940 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5610bde0a270 0x5610be1f4e90 secure :-1 s=READY pgs=1102 cs=0 l=1 rev1=1 crypto rx=0x7fb8940099a0 tx=0x7fb89403e040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=361e4df7a6a8eada server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:54:57.134 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:57.132+0000 7fb893400640 1 -- 172.21.15.32:0/3767617940 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb89403cd70 con 0x5610bde0a270 2024-08-03T20:54:57.134 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:57.132+0000 7fb893400640 1 -- 172.21.15.32:0/3767617940 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fb894045480 con 0x5610bde0a270 2024-08-03T20:54:57.134 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:57.132+0000 7fb893400640 1 -- 172.21.15.32:0/3767617940 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8940458b0 con 0x5610bde0a270 2024-08-03T20:54:57.134 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:57.132+0000 7fb8b0c08740 1 -- 172.21.15.32:0/3767617940 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5610bde0a270 msgr2=0x5610be1f4e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:57.134 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:57.132+0000 7fb8b0c08740 1 --2- 172.21.15.32:0/3767617940 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5610bde0a270 0x5610be1f4e90 secure :-1 s=READY pgs=1102 cs=0 l=1 rev1=1 crypto rx=0x7fb8940099a0 tx=0x7fb89403e040 comp rx=0 tx=0).stop 2024-08-03T20:54:57.134 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:57.132+0000 7fb8b0c08740 1 -- 172.21.15.32:0/3767617940 shutdown_connections 2024-08-03T20:54:57.134 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:57.132+0000 7fb8b0c08740 1 --2- 172.21.15.32:0/3767617940 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5610bdde13b0 0x5610be1c9ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:57.135 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:57.132+0000 7fb8b0c08740 1 --2- 172.21.15.32:0/3767617940 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x5610bdde9590 0x5610be1ca220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:57.135 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:57.132+0000 7fb8b0c08740 1 --2- 172.21.15.32:0/3767617940 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5610bde0a270 0x5610be1f4e90 unknown :-1 s=CLOSED pgs=1102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:57.135 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:57.132+0000 7fb8b0c08740 1 -- 172.21.15.32:0/3767617940 >> 172.21.15.32:0/3767617940 conn(0x5610bddfe3c0 msgr2=0x5610be1b0180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:54:57.135 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:57.133+0000 7fb8b0c08740 1 -- 172.21.15.32:0/3767617940 shutdown_connections 2024-08-03T20:54:57.135 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:57.133+0000 7fb8b0c08740 1 -- 172.21.15.32:0/3767617940 wait complete. 2024-08-03T20:54:57.135 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:57.134+0000 7fb8b0c08740 1 -- start start 2024-08-03T20:54:57.135 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:57.134+0000 7fb8b0c08740 1 Processor -- start 2024-08-03T20:54:57.136 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:57.135+0000 7fb8b0c08740 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x5610bdde13b0 0x5610be1f09b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:57.136 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:57.135+0000 7fb8b0c08740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5610bdde9590 0x5610be1f0ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:57.136 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:57.135+0000 7fb8b0c08740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5610bde0a270 0x5610be1d7ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:57.136 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:57.135+0000 7fb8b0c08740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x5610bde84620 con 0x5610bde0a270 2024-08-03T20:54:57.136 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:57.135+0000 7fb8b0c08740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x5610be0af160 con 0x5610bdde9590 2024-08-03T20:54:57.136 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:57.135+0000 7fb8b0c08740 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x5610be1edf80 con 0x5610bdde13b0 2024-08-03T20:54:57.136 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:57.135+0000 7fb8b0c08740 10 client.0 _ll_register_callbacks cb 0x5610be0be6b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-03T20:54:57.136 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:57.136+0000 7fb8a3400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5610bde0a270 0x5610be1d7ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:57.137 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:57.136+0000 7fb8a2a00640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x5610bdde13b0 0x5610be1f09b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:57.137 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:57.136+0000 7fb8a3400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5610bde0a270 0x5610be1d7ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:49974/0 (socket says 172.21.15.32:49974) 2024-08-03T20:54:57.137 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:57.136+0000 7fb89be00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5610bdde9590 0x5610be1f0ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:57.137 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:57.136+0000 7fb8a3400640 1 -- 172.21.15.32:0/1944405024 learned_addr learned my addr 172.21.15.32:0/1944405024 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:54:57.139 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:57.136+0000 7fb8a3400640 1 -- 172.21.15.32:0/1944405024 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x5610bdde13b0 msgr2=0x5610be1f09b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:54:57.139 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:57.136+0000 7fb8a3400640 1 --2- 172.21.15.32:0/1944405024 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x5610bdde13b0 0x5610be1f09b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:57.139 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:57.136+0000 7fb8a3400640 1 -- 172.21.15.32:0/1944405024 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5610bdde9590 msgr2=0x5610be1f0ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:54:57.139 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:57.136+0000 7fb8a3400640 1 --2- 172.21.15.32:0/1944405024 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5610bdde9590 0x5610be1f0ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:57.139 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:57.136+0000 7fb8a3400640 1 -- 172.21.15.32:0/1944405024 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5610bdeec6b0 con 0x5610bde0a270 2024-08-03T20:54:57.139 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:57.136+0000 7fb8a3400640 1 --2- 172.21.15.32:0/1944405024 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5610bde0a270 0x5610be1d7ea0 secure :-1 s=READY pgs=1103 cs=0 l=1 rev1=1 crypto rx=0x7fb89c00bbe0 tx=0x7fb89c00a7b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fac365d4f99b41ed server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:54:57.139 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:57.136+0000 7fb887e00640 10 client.0 ms_handle_connect on v2:172.21.15.32:3300/0 2024-08-03T20:54:57.139 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:57.137+0000 7fb887e00640 1 -- 172.21.15.32:0/1944405024 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb89c016610 con 0x5610bde0a270 2024-08-03T20:54:57.139 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:57.137+0000 7fb887e00640 1 -- 172.21.15.32:0/1944405024 <== mon.0 v2:172.21.15.32:3300/0 2 ==== osd_map(84..84 src has 1..84) ==== 5795+0+0 (secure 0 0 0) 0x7fb89c0167b0 con 0x5610bde0a270 2024-08-03T20:54:57.139 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:57.137+0000 7fb8b0c08740 1 -- 172.21.15.32:0/1944405024 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb894045480 con 0x5610bde0a270 2024-08-03T20:54:57.139 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:57.137+0000 7fb887e00640 1 -- 172.21.15.32:0/1944405024 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_get_version_reply(handle=1 version=99) ==== 24+0+0 (secure 0 0 0) 0x7fb89c0169b0 con 0x5610bde0a270 2024-08-03T20:54:57.139 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:57.137+0000 7fb8b0c08740 10 client.7470 fetch_fsmap learned FSMap version 99 2024-08-03T20:54:57.139 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:57.137+0000 7fb8b0c08740 1 -- 172.21.15.32:0/1944405024 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,fsmap=99,monmap=2+}) -- 0x5610be1d88b0 con 0x5610bde0a270 2024-08-03T20:54:57.140 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:57.137+0000 7fb887e00640 1 -- 172.21.15.32:0/1944405024 <== mon.0 v2:172.21.15.32:3300/0 4 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fb894045480 con 0x5610bde0a270 2024-08-03T20:54:57.140 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:57.137+0000 7fb887e00640 1 -- 172.21.15.32:0/1944405024 <== mon.0 v2:172.21.15.32:3300/0 5 ==== fsmap(e 99) ==== 3502+0+0 (secure 0 0 0) 0x7fb89c026080 con 0x5610bde0a270 2024-08-03T20:54:57.140 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:57.137+0000 7fb8b0c08740 10 client.7470 fetch_fsmap finished waiting for FSMap version 99 2024-08-03T20:54:57.140 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:57.137+0000 7fb8b0c08740 10 client.7470 resolve_mds: resolved name 'a' to daemon mds.a 2024-08-03T20:54:57.140 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:57.137+0000 7fb8b0c08740 20 client.7470 populate_metadata read hostname 'smithi032' 2024-08-03T20:54:57.140 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:57.137+0000 7fb8b0c08740 1 --2- 172.21.15.32:0/1944405024 >> [v2:172.21.15.32:6833/2194902489,v1:172.21.15.32:6835/2194902489] conn(0x5610bddc8e40 0x5610be1bb620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:57.140 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:57.137+0000 7fb8b0c08740 4 client.7470 mds_command: new command op to 6975 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-08-03T20:54:57.140 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:57.137+0000 7fb8b0c08740 1 -- 172.21.15.32:0/1944405024 --> [v2:172.21.15.32:6833/2194902489,v1:172.21.15.32:6835/2194902489] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5610be1249b0 con 0x5610bddc8e40 2024-08-03T20:54:57.140 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:57.138+0000 7fb8a2a00640 1 --2- 172.21.15.32:0/1944405024 >> [v2:172.21.15.32:6833/2194902489,v1:172.21.15.32:6835/2194902489] conn(0x5610bddc8e40 0x5610be1bb620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:57.140 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:57.138+0000 7fb8a2a00640 1 --2- 172.21.15.32:0/1944405024 >> [v2:172.21.15.32:6833/2194902489,v1:172.21.15.32:6835/2194902489] conn(0x5610bddc8e40 0x5610be1bb620 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e3bf0e1719739365 server_cookie=e7aa16cc80d8fc61 in_seq=0 out_seq=0 2024-08-03T20:54:57.140 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:57.138+0000 7fb887e00640 10 client.7470 ms_handle_connect on v2:172.21.15.32:6833/2194902489 2024-08-03T20:54:57.141 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:57.139+0000 7fb887e00640 1 -- 172.21.15.32:0/1944405024 <== mds.0 v2:172.21.15.32:6833/2194902489 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x5610be1249b0 con 0x5610bddc8e40 2024-08-03T20:54:57.141 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:57.139+0000 7fb887e00640 10 client.7470 handle_command_reply: tid=0 2024-08-03T20:54:57.141 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:57.139+0000 7fb8b0c08740 1 client.7470 shutdown 2024-08-03T20:54:57.142 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:57.139+0000 7fb8b0c08740 1 -- 172.21.15.32:0/1944405024 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5610bde0a270 msgr2=0x5610be1d7ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:57.142 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:57.139+0000 7fb8b0c08740 1 --2- 172.21.15.32:0/1944405024 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5610bde0a270 0x5610be1d7ea0 secure :-1 s=READY pgs=1103 cs=0 l=1 rev1=1 crypto rx=0x7fb89c00bbe0 tx=0x7fb89c00a7b0 comp rx=0 tx=0).stop 2024-08-03T20:54:57.142 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:57.140+0000 7fb8b0c08740 1 -- 172.21.15.32:0/1944405024 shutdown_connections 2024-08-03T20:54:57.142 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:57.140+0000 7fb8b0c08740 1 --2- 172.21.15.32:0/1944405024 >> [v2:172.21.15.32:6833/2194902489,v1:172.21.15.32:6835/2194902489] conn(0x5610bddc8e40 0x5610be1bb620 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:57.142 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:57.140+0000 7fb8b0c08740 1 --2- 172.21.15.32:0/1944405024 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5610bdde9590 0x5610be1f0ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:57.142 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:57.140+0000 7fb8b0c08740 1 --2- 172.21.15.32:0/1944405024 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5610bde0a270 0x5610be1d7ea0 unknown :-1 s=CLOSED pgs=1103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:57.142 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:57.140+0000 7fb887e00640 0 client.7470 ms_handle_reset on v2:172.21.15.32:6833/2194902489 2024-08-03T20:54:57.142 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:57.140+0000 7fb8b0c08740 1 --2- 172.21.15.32:0/1944405024 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x5610bdde13b0 0x5610be1f09b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:57.142 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:57.140+0000 7fb8b0c08740 1 -- 172.21.15.32:0/1944405024 >> 172.21.15.32:0/1944405024 conn(0x5610bddfe3c0 msgr2=0x5610be1afbf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:54:57.142 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:57.140+0000 7fb8b0c08740 1 -- 172.21.15.32:0/1944405024 shutdown_connections 2024-08-03T20:54:57.142 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:57.140+0000 7fb8b0c08740 1 -- 172.21.15.32:0/1944405024 wait complete. 2024-08-03T20:54:57.142 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:57.140+0000 7fb8b0c08740 20 client.7470 trim_cache size 0 max 16384 2024-08-03T20:54:57.160 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:57.160+0000 7fb8b0c08740 1 Processor -- start 2024-08-03T20:54:57.160 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:57.160+0000 7fb8b0c08740 1 -- start start 2024-08-03T20:54:57.161 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:57.161+0000 7fb8b0c08740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5610bddfe3c0 0x5610be1b01c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:57.162 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:57.161+0000 7fb8b0c08740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5610bdde13b0 0x5610be1c9aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:57.162 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:57.161+0000 7fb8b0c08740 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x5610bdde9590 0x5610be1c9fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:57.162 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:57.161+0000 7fb8b0c08740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fb8a40cb910 con 0x5610bddfe3c0 2024-08-03T20:54:57.162 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:57.161+0000 7fb8b0c08740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7fb8a40cac70 con 0x5610bdde13b0 2024-08-03T20:54:57.162 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:57.161+0000 7fb8b0c08740 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7fb8a40cb2c0 con 0x5610bdde9590 2024-08-03T20:54:57.163 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:57.161+0000 7fb8a2a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5610bddfe3c0 0x5610be1b01c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:57.163 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:57.161+0000 7fb8a2a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5610bddfe3c0 0x5610be1b01c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:49980/0 (socket says 172.21.15.32:49980) 2024-08-03T20:54:57.163 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:57.162+0000 7fb8a2a00640 1 -- 172.21.15.32:0/2894644877 learned_addr learned my addr 172.21.15.32:0/2894644877 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:54:57.163 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:57.162+0000 7fb8a3400640 1 --2- 172.21.15.32:0/2894644877 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x5610bdde9590 0x5610be1c9fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:57.163 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:57.162+0000 7fb89be00640 1 --2- 172.21.15.32:0/2894644877 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5610bdde13b0 0x5610be1c9aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:57.163 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:57.162+0000 7fb8a2a00640 1 -- 172.21.15.32:0/2894644877 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x5610bdde9590 msgr2=0x5610be1c9fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:54:57.163 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:57.162+0000 7fb8a2a00640 1 --2- 172.21.15.32:0/2894644877 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x5610bdde9590 0x5610be1c9fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:57.163 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:57.162+0000 7fb8a2a00640 1 -- 172.21.15.32:0/2894644877 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5610bdde13b0 msgr2=0x5610be1c9aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:54:57.164 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:57.162+0000 7fb8a2a00640 1 --2- 172.21.15.32:0/2894644877 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5610bdde13b0 0x5610be1c9aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:57.164 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:57.162+0000 7fb8a2a00640 1 -- 172.21.15.32:0/2894644877 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5610be1bf570 con 0x5610bddfe3c0 2024-08-03T20:54:57.164 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:57.162+0000 7fb8a2a00640 1 --2- 172.21.15.32:0/2894644877 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5610bddfe3c0 0x5610be1b01c0 secure :-1 s=READY pgs=1104 cs=0 l=1 rev1=1 crypto rx=0x7fb894005ee0 tx=0x7fb8940046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bc2963d4120f6720 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:54:57.164 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:57.162+0000 7fb892a00640 1 -- 172.21.15.32:0/2894644877 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb894004240 con 0x5610bddfe3c0 2024-08-03T20:54:57.164 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:57.162+0000 7fb892a00640 1 -- 172.21.15.32:0/2894644877 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fb8940044d0 con 0x5610bddfe3c0 2024-08-03T20:54:57.164 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:57.163+0000 7fb892a00640 1 -- 172.21.15.32:0/2894644877 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb894003830 con 0x5610bddfe3c0 2024-08-03T20:54:57.165 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:57.163+0000 7fb8b0c08740 1 -- 172.21.15.32:0/2894644877 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5610bddfe3c0 msgr2=0x5610be1b01c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:57.165 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:57.163+0000 7fb8b0c08740 1 --2- 172.21.15.32:0/2894644877 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5610bddfe3c0 0x5610be1b01c0 secure :-1 s=READY pgs=1104 cs=0 l=1 rev1=1 crypto rx=0x7fb894005ee0 tx=0x7fb8940046c0 comp rx=0 tx=0).stop 2024-08-03T20:54:57.165 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:57.163+0000 7fb8b0c08740 1 -- 172.21.15.32:0/2894644877 shutdown_connections 2024-08-03T20:54:57.165 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:57.163+0000 7fb8b0c08740 1 --2- 172.21.15.32:0/2894644877 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5610bdde13b0 0x5610be1c9aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:57.165 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:57.163+0000 7fb8b0c08740 1 --2- 172.21.15.32:0/2894644877 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x5610bdde9590 0x5610be1c9fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:57.165 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:57.163+0000 7fb8b0c08740 1 --2- 172.21.15.32:0/2894644877 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5610bddfe3c0 0x5610be1b01c0 unknown :-1 s=CLOSED pgs=1104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:57.165 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:57.163+0000 7fb8b0c08740 1 -- 172.21.15.32:0/2894644877 >> 172.21.15.32:0/2894644877 conn(0x5610be1d7fd0 msgr2=0x5610be1eaf20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:54:57.165 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:57.163+0000 7fb8b0c08740 1 -- 172.21.15.32:0/2894644877 shutdown_connections 2024-08-03T20:54:57.166 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:57.163+0000 7fb8b0c08740 1 -- 172.21.15.32:0/2894644877 wait complete. 2024-08-03T20:54:57.166 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:57.164+0000 7fb8b0c08740 1 -- start start 2024-08-03T20:54:57.166 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:57.165+0000 7fb8b0c08740 1 Processor -- start 2024-08-03T20:54:57.167 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:57.166+0000 7fb8b0c08740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5610bdde13b0 0x5610be271680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:57.167 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:57.166+0000 7fb8b0c08740 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x5610bdde9590 0x5610be1bd630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:57.168 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:57.166+0000 7fb8b0c08740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5610bdd9e280 0x5610be1b7ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:57.168 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:57.166+0000 7fb8b0c08740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x5610be1ed920 con 0x5610bdde13b0 2024-08-03T20:54:57.168 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:57.166+0000 7fb8b0c08740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x5610be1bf850 con 0x5610bdd9e280 2024-08-03T20:54:57.168 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:57.166+0000 7fb8b0c08740 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x5610be207930 con 0x5610bdde9590 2024-08-03T20:54:57.168 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:57.166+0000 7fb8b0c08740 10 client.0 _ll_register_callbacks cb 0x7fb89c00b7c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-03T20:54:57.168 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:57.166+0000 7fb8a2a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5610bdde13b0 0x5610be271680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:57.168 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:57.166+0000 7fb8a2a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5610bdde13b0 0x5610be271680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:49984/0 (socket says 172.21.15.32:49984) 2024-08-03T20:54:57.168 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:57.166+0000 7fb8a2a00640 1 -- 172.21.15.32:0/3037780294 learned_addr learned my addr 172.21.15.32:0/3037780294 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:54:57.168 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:57.167+0000 7fb8a3400640 1 --2- 172.21.15.32:0/3037780294 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5610bdd9e280 0x5610be1b7ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:57.168 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:57.167+0000 7fb89be00640 1 --2- 172.21.15.32:0/3037780294 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x5610bdde9590 0x5610be1bd630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:57.168 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:57.167+0000 7fb8a2a00640 1 -- 172.21.15.32:0/3037780294 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x5610bdde9590 msgr2=0x5610be1bd630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:54:57.168 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:57.167+0000 7fb8a2a00640 1 --2- 172.21.15.32:0/3037780294 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x5610bdde9590 0x5610be1bd630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:57.168 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:57.167+0000 7fb8a2a00640 1 -- 172.21.15.32:0/3037780294 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5610bdd9e280 msgr2=0x5610be1b7ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:54:57.168 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:57.167+0000 7fb8a2a00640 1 --2- 172.21.15.32:0/3037780294 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5610bdd9e280 0x5610be1b7ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:57.169 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:57.167+0000 7fb8a2a00640 1 -- 172.21.15.32:0/3037780294 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5610be1b0440 con 0x5610bdde13b0 2024-08-03T20:54:57.169 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:57.167+0000 7fb8a2a00640 1 --2- 172.21.15.32:0/3037780294 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5610bdde13b0 0x5610be271680 secure :-1 s=READY pgs=1105 cs=0 l=1 rev1=1 crypto rx=0x7fb89403cda0 tx=0x7fb89403c420 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ada27f83583be5cf server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:54:57.169 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:57.167+0000 7fb887e00640 10 client.0 ms_handle_connect on v2:172.21.15.32:3300/0 2024-08-03T20:54:57.171 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:57.167+0000 7fb887e00640 1 -- 172.21.15.32:0/3037780294 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb89406e060 con 0x5610bdde13b0 2024-08-03T20:54:57.171 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:57.167+0000 7fb887e00640 1 -- 172.21.15.32:0/3037780294 <== mon.0 v2:172.21.15.32:3300/0 2 ==== osd_map(84..84 src has 1..84) ==== 5795+0+0 (secure 0 0 0) 0x7fb8940037f0 con 0x5610bdde13b0 2024-08-03T20:54:57.171 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:57.168+0000 7fb8b0c08740 1 -- 172.21.15.32:0/3037780294 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb8940044d0 con 0x5610bdde13b0 2024-08-03T20:54:57.171 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:57.168+0000 7fb887e00640 1 -- 172.21.15.32:0/3037780294 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_get_version_reply(handle=1 version=99) ==== 24+0+0 (secure 0 0 0) 0x7fb894074050 con 0x5610bdde13b0 2024-08-03T20:54:57.171 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:57.168+0000 7fb8b0c08740 10 client.7476 fetch_fsmap learned FSMap version 99 2024-08-03T20:54:57.171 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:57.168+0000 7fb8b0c08740 1 -- 172.21.15.32:0/3037780294 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,fsmap=99,monmap=2+}) -- 0x5610be1bdb70 con 0x5610bdde13b0 2024-08-03T20:54:57.172 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:57.168+0000 7fb887e00640 1 -- 172.21.15.32:0/3037780294 <== mon.0 v2:172.21.15.32:3300/0 4 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fb8940044d0 con 0x5610bdde13b0 2024-08-03T20:54:57.172 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:57.168+0000 7fb887e00640 1 -- 172.21.15.32:0/3037780294 <== mon.0 v2:172.21.15.32:3300/0 5 ==== fsmap(e 99) ==== 3502+0+0 (secure 0 0 0) 0x7fb894077020 con 0x5610bdde13b0 2024-08-03T20:54:57.172 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:57.168+0000 7fb8b0c08740 10 client.7476 fetch_fsmap finished waiting for FSMap version 99 2024-08-03T20:54:57.172 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:57.168+0000 7fb8b0c08740 10 client.7476 resolve_mds: resolved name 'a' to daemon mds.a 2024-08-03T20:54:57.172 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:57.168+0000 7fb8b0c08740 20 client.7476 populate_metadata read hostname 'smithi032' 2024-08-03T20:54:57.172 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:57.168+0000 7fb8b0c08740 1 --2- 172.21.15.32:0/3037780294 >> [v2:172.21.15.32:6833/2194902489,v1:172.21.15.32:6835/2194902489] conn(0x5610be1f5390 0x5610be1f48e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:57.172 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:57.168+0000 7fb8b0c08740 4 client.7476 mds_command: new command op to 6975 tid=0 multi_id=0 [{"prefix": "status"}] 2024-08-03T20:54:57.172 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:57.168+0000 7fb8b0c08740 1 -- 172.21.15.32:0/3037780294 --> [v2:172.21.15.32:6833/2194902489,v1:172.21.15.32:6835/2194902489] -- command(tid 0: {"prefix": "status"}) -- 0x5610be120450 con 0x5610be1f5390 2024-08-03T20:54:57.172 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:57.169+0000 7fb89be00640 1 --2- 172.21.15.32:0/3037780294 >> [v2:172.21.15.32:6833/2194902489,v1:172.21.15.32:6835/2194902489] conn(0x5610be1f5390 0x5610be1f48e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:57.172 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:57.169+0000 7fb89be00640 1 --2- 172.21.15.32:0/3037780294 >> [v2:172.21.15.32:6833/2194902489,v1:172.21.15.32:6835/2194902489] conn(0x5610be1f5390 0x5610be1f48e0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=41da30f1301ec2cf server_cookie=e5ac6f1ba0b6ee2c in_seq=0 out_seq=0 2024-08-03T20:54:57.172 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:57.169+0000 7fb887e00640 10 client.7476 ms_handle_connect on v2:172.21.15.32:6833/2194902489 2024-08-03T20:54:57.172 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:57.169+0000 7fb887e00640 1 -- 172.21.15.32:0/3037780294 <== mds.0 v2:172.21.15.32:6833/2194902489 1 ==== command_reply(tid 0: 0 ) ==== 8+0+331 (crc 0 0 0) 0x5610be120450 con 0x5610be1f5390 2024-08-03T20:54:57.172 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:57.169+0000 7fb887e00640 10 client.7476 handle_command_reply: tid=0 2024-08-03T20:54:57.172 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:57.169+0000 7fb8b0c08740 1 client.7476 shutdown 2024-08-03T20:54:57.173 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:57.169+0000 7fb8b0c08740 1 -- 172.21.15.32:0/3037780294 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5610bdde13b0 msgr2=0x5610be271680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:57.173 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:57.169+0000 7fb8b0c08740 1 --2- 172.21.15.32:0/3037780294 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5610bdde13b0 0x5610be271680 secure :-1 s=READY pgs=1105 cs=0 l=1 rev1=1 crypto rx=0x7fb89403cda0 tx=0x7fb89403c420 comp rx=0 tx=0).stop 2024-08-03T20:54:57.173 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:57.170+0000 7fb8b0c08740 1 -- 172.21.15.32:0/3037780294 shutdown_connections 2024-08-03T20:54:57.173 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:57.170+0000 7fb8b0c08740 1 --2- 172.21.15.32:0/3037780294 >> [v2:172.21.15.32:6833/2194902489,v1:172.21.15.32:6835/2194902489] conn(0x5610be1f5390 0x5610be1f48e0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:57.173 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:57.170+0000 7fb8b0c08740 1 --2- 172.21.15.32:0/3037780294 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5610bdd9e280 0x5610be1b7ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:57.173 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:57.170+0000 7fb8b0c08740 1 --2- 172.21.15.32:0/3037780294 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x5610bdde9590 0x5610be1bd630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:57.173 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:57.170+0000 7fb8b0c08740 1 --2- 172.21.15.32:0/3037780294 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5610bdde13b0 0x5610be271680 unknown :-1 s=CLOSED pgs=1105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:57.173 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:57.170+0000 7fb887e00640 0 client.7476 ms_handle_reset on v2:172.21.15.32:6833/2194902489 2024-08-03T20:54:57.173 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:57.170+0000 7fb8b0c08740 1 -- 172.21.15.32:0/3037780294 >> 172.21.15.32:0/3037780294 conn(0x5610be1d7fd0 msgr2=0x5610be1f13e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:54:57.173 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:57.170+0000 7fb8b0c08740 1 -- 172.21.15.32:0/3037780294 shutdown_connections 2024-08-03T20:54:57.173 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:57.170+0000 7fb8b0c08740 1 -- 172.21.15.32:0/3037780294 wait complete. 2024-08-03T20:54:57.173 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:57.170+0000 7fb8b0c08740 20 client.7476 trim_cache size 0 max 16384 2024-08-03T20:54:57.174 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-08-03T20:54:57.174 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_fsid": "40ab141a-01d7-49b9-ae3c-2edec9876e95", 2024-08-03T20:54:57.174 INFO:teuthology.orchestra.run.smithi032.stdout: "whoami": 0, 2024-08-03T20:54:57.174 INFO:teuthology.orchestra.run.smithi032.stdout: "id": 6975, 2024-08-03T20:54:57.174 INFO:teuthology.orchestra.run.smithi032.stdout: "want_state": "up:active", 2024-08-03T20:54:57.174 INFO:teuthology.orchestra.run.smithi032.stdout: "state": "up:active", 2024-08-03T20:54:57.174 INFO:teuthology.orchestra.run.smithi032.stdout: "fs_name": "cephfs", 2024-08-03T20:54:57.174 INFO:teuthology.orchestra.run.smithi032.stdout: "rank_uptime": 7.1660778279999997, 2024-08-03T20:54:57.174 INFO:teuthology.orchestra.run.smithi032.stdout: "mdsmap_epoch": 99, 2024-08-03T20:54:57.174 INFO:teuthology.orchestra.run.smithi032.stdout: "osdmap_epoch": 84, 2024-08-03T20:54:57.174 INFO:teuthology.orchestra.run.smithi032.stdout: "osdmap_epoch_barrier": 84, 2024-08-03T20:54:57.174 INFO:teuthology.orchestra.run.smithi032.stdout: "uptime": 18.613711886000001 2024-08-03T20:54:57.174 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-08-03T20:54:57.175 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:57.170+0000 7fb8a9c00640 1 -- 172.21.15.32:0/3855884075 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fb880077010 msgr2=0x7fb880079520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:57.175 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:57.170+0000 7fb8a9c00640 1 --2- 172.21.15.32:0/3855884075 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fb880077010 0x7fb880079520 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7fb894034000 tx=0x7fb894004730 comp rx=0 tx=0).stop 2024-08-03T20:54:57.176 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:57.170+0000 7fb8a9c00640 1 -- 172.21.15.32:0/3855884075 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb8a40b7760 msgr2=0x7fb8a4166200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:57.176 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:57.170+0000 7fb8a9c00640 1 --2- 172.21.15.32:0/3855884075 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb8a40b7760 0x7fb8a4166200 secure :-1 s=READY pgs=1101 cs=0 l=1 rev1=1 crypto rx=0x7fb88c00d6b0 tx=0x7fb88c00db70 comp rx=0 tx=0).stop 2024-08-03T20:54:57.176 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:57.171+0000 7fb8a9c00640 1 -- 172.21.15.32:0/3855884075 shutdown_connections 2024-08-03T20:54:57.176 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:57.171+0000 7fb8a9c00640 1 --2- 172.21.15.32:0/3855884075 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fb880077010 0x7fb880079520 unknown :-1 s=CLOSED pgs=458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:57.176 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:57.171+0000 7fb8a9c00640 1 --2- 172.21.15.32:0/3855884075 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb8a40cfc90 0x7fb8a4166740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:57.176 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:57.171+0000 7fb8a9c00640 1 --2- 172.21.15.32:0/3855884075 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb8a40b7760 0x7fb8a4166200 unknown :-1 s=CLOSED pgs=1101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:57.176 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:57.171+0000 7fb8a9c00640 1 --2- 172.21.15.32:0/3855884075 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fb8a409ac30 0x7fb8a4163e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:57.176 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:57.171+0000 7fb8a9c00640 1 -- 172.21.15.32:0/3855884075 >> 172.21.15.32:0/3855884075 conn(0x7fb8a40b3bb0 msgr2=0x7fb8a40b6f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:54:57.176 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:57.171+0000 7fb8a9c00640 1 -- 172.21.15.32:0/3855884075 shutdown_connections 2024-08-03T20:54:57.176 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:57.171+0000 7fb8a9c00640 1 -- 172.21.15.32:0/3855884075 wait complete. 2024-08-03T20:54:57.196 DEBUG:tasks.ceph_test_case:wait_until_true: success in 5s and 0 retries 2024-08-03T20:54:57.196 INFO:tasks.cephfs.test_damage:Daemons came up after mutation 'Delete 10000000000.00000000', proceeding to ls 2024-08-03T20:54:57.196 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-08-03T20:54:57.426 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:57.424+0000 7fa813200640 1 -- 172.21.15.32:0/3467528279 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa80c144f90 msgr2=0x7fa80c074570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:57.426 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:57.424+0000 7fa813200640 1 --2- 172.21.15.32:0/3467528279 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa80c144f90 0x7fa80c074570 secure :-1 s=READY pgs=1107 cs=0 l=1 rev1=1 crypto rx=0x7fa7f4009830 tx=0x7fa7f402a510 comp rx=0 tx=0).stop 2024-08-03T20:54:57.426 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:57.424+0000 7fa813200640 1 -- 172.21.15.32:0/3467528279 shutdown_connections 2024-08-03T20:54:57.426 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:57.424+0000 7fa813200640 1 --2- 172.21.15.32:0/3467528279 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fa80c15fff0 0x7fa80c160400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:57.426 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:57.424+0000 7fa813200640 1 --2- 172.21.15.32:0/3467528279 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa80c144f90 0x7fa80c074570 unknown :-1 s=CLOSED pgs=1107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:57.427 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:57.424+0000 7fa813200640 1 -- 172.21.15.32:0/3467528279 >> 172.21.15.32:0/3467528279 conn(0x7fa80c0b1bd0 msgr2=0x7fa80c0b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:54:57.427 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:57.425+0000 7fa813200640 1 -- 172.21.15.32:0/3467528279 shutdown_connections 2024-08-03T20:54:57.427 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:57.425+0000 7fa813200640 1 -- 172.21.15.32:0/3467528279 wait complete. 2024-08-03T20:54:57.427 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:57.427+0000 7fa813200640 1 Processor -- start 2024-08-03T20:54:57.427 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:57.427+0000 7fa813200640 1 -- start start 2024-08-03T20:54:57.431 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:57.428+0000 7fa813200640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa80c06e930 0x7fa80c15bcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:57.431 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:57.428+0000 7fa813200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa80c144f90 0x7fa80c1f86c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:57.431 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:57.428+0000 7fa813200640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fa80c15fff0 0x7fa80c1f8c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:57.431 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:57.429+0000 7fa813200640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fa80c0cf840 con 0x7fa80c144f90 2024-08-03T20:54:57.431 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:57.429+0000 7fa813200640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7fa80c145f30 con 0x7fa80c06e930 2024-08-03T20:54:57.431 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:57.429+0000 7fa813200640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7fa80c0cf9c0 con 0x7fa80c15fff0 2024-08-03T20:54:57.431 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:57.429+0000 7fa803400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa80c144f90 0x7fa80c1f86c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:57.431 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:57.429+0000 7fa803400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa80c144f90 0x7fa80c1f86c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:50002/0 (socket says 172.21.15.32:50002) 2024-08-03T20:54:57.431 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:57.429+0000 7fa803400640 1 -- 172.21.15.32:0/4167303988 learned_addr learned my addr 172.21.15.32:0/4167303988 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:54:57.431 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:57.429+0000 7fa811200640 1 --2- 172.21.15.32:0/4167303988 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fa80c15fff0 0x7fa80c1f8c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:57.432 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:57.429+0000 7fa80be00640 1 --2- 172.21.15.32:0/4167303988 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa80c06e930 0x7fa80c15bcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:57.432 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:57.429+0000 7fa803400640 1 -- 172.21.15.32:0/4167303988 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fa80c15fff0 msgr2=0x7fa80c1f8c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:57.432 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:57.429+0000 7fa803400640 1 --2- 172.21.15.32:0/4167303988 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fa80c15fff0 0x7fa80c1f8c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:57.432 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:57.429+0000 7fa803400640 1 -- 172.21.15.32:0/4167303988 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa80c06e930 msgr2=0x7fa80c15bcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:57.432 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:57.429+0000 7fa803400640 1 --2- 172.21.15.32:0/4167303988 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa80c06e930 0x7fa80c15bcc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:57.432 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:57.429+0000 7fa803400640 1 -- 172.21.15.32:0/4167303988 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa80c142350 con 0x7fa80c144f90 2024-08-03T20:54:57.432 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:57.430+0000 7fa803400640 1 --2- 172.21.15.32:0/4167303988 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa80c144f90 0x7fa80c1f86c0 secure :-1 s=READY pgs=1108 cs=0 l=1 rev1=1 crypto rx=0x7fa7f4009950 tx=0x7fa7f4033c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:54:57.432 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:57.430+0000 7fa808c00640 1 -- 172.21.15.32:0/4167303988 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7f4005c90 con 0x7fa80c144f90 2024-08-03T20:54:57.432 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:57.430+0000 7fa808c00640 1 -- 172.21.15.32:0/4167303988 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fa7f4031e50 con 0x7fa80c144f90 2024-08-03T20:54:57.432 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:57.430+0000 7fa813200640 1 -- 172.21.15.32:0/4167303988 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa80c142630 con 0x7fa80c144f90 2024-08-03T20:54:57.432 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:57.430+0000 7fa808c00640 1 -- 172.21.15.32:0/4167303988 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7f4005260 con 0x7fa80c144f90 2024-08-03T20:54:57.432 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:57.430+0000 7fa813200640 1 -- 172.21.15.32:0/4167303988 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa80c13ea20 con 0x7fa80c144f90 2024-08-03T20:54:57.433 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:57.431+0000 7fa813200640 1 -- 172.21.15.32:0/4167303988 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa80c160cc0 con 0x7fa80c144f90 2024-08-03T20:54:57.433 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:57.432+0000 7fa808c00640 1 -- 172.21.15.32:0/4167303988 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7fa7f4005400 con 0x7fa80c144f90 2024-08-03T20:54:57.434 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:57.434+0000 7fa808c00640 1 --2- 172.21.15.32:0/4167303988 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fa7e8077010 0x7fa7e8079520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:57.434 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:57.434+0000 7fa808c00640 1 -- 172.21.15.32:0/4167303988 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 5795+0+0 (secure 0 0 0) 0x7fa7f40b9b80 con 0x7fa80c144f90 2024-08-03T20:54:57.435 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:57.434+0000 7fa80be00640 1 --2- 172.21.15.32:0/4167303988 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fa7e8077010 0x7fa7e8079520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:57.435 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:57.435+0000 7fa80be00640 1 --2- 172.21.15.32:0/4167303988 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fa7e8077010 0x7fa7e8079520 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7fa7fc001610 tx=0x7fa7fc009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:54:57.439 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:57.438+0000 7fa808c00640 1 -- 172.21.15.32:0/4167303988 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7fa7f403f030 con 0x7fa80c144f90 2024-08-03T20:54:57.688 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:57.687+0000 7fa813200640 1 -- 172.21.15.32:0/4167303988 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7fa80c1457e0 con 0x7fa80c144f90 2024-08-03T20:54:57.688 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:57.687+0000 7fa808c00640 1 -- 172.21.15.32:0/4167303988 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v99) ==== 53+0+73 (secure 0 0 0) 0x7fa7f4082c80 con 0x7fa80c144f90 2024-08-03T20:54:57.688 INFO:teuthology.orchestra.run.smithi032.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-08-03T20:54:57.690 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:57.690+0000 7fa813200640 1 -- 172.21.15.32:0/4167303988 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fa7e8077010 msgr2=0x7fa7e8079520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:57.691 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:57.690+0000 7fa813200640 1 --2- 172.21.15.32:0/4167303988 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fa7e8077010 0x7fa7e8079520 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7fa7fc001610 tx=0x7fa7fc009040 comp rx=0 tx=0).stop 2024-08-03T20:54:57.691 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:57.690+0000 7fa813200640 1 -- 172.21.15.32:0/4167303988 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa80c144f90 msgr2=0x7fa80c1f86c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:57.691 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:57.690+0000 7fa813200640 1 --2- 172.21.15.32:0/4167303988 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa80c144f90 0x7fa80c1f86c0 secure :-1 s=READY pgs=1108 cs=0 l=1 rev1=1 crypto rx=0x7fa7f4009950 tx=0x7fa7f4033c20 comp rx=0 tx=0).stop 2024-08-03T20:54:57.691 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:57.690+0000 7fa813200640 1 -- 172.21.15.32:0/4167303988 shutdown_connections 2024-08-03T20:54:57.691 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:57.690+0000 7fa813200640 1 --2- 172.21.15.32:0/4167303988 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fa7e8077010 0x7fa7e8079520 unknown :-1 s=CLOSED pgs=459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:57.691 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:57.690+0000 7fa813200640 1 --2- 172.21.15.32:0/4167303988 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fa80c15fff0 0x7fa80c1f8c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:57.691 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:57.690+0000 7fa813200640 1 --2- 172.21.15.32:0/4167303988 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa80c144f90 0x7fa80c1f86c0 unknown :-1 s=CLOSED pgs=1108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:57.691 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:57.690+0000 7fa813200640 1 --2- 172.21.15.32:0/4167303988 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa80c06e930 0x7fa80c15bcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:57.691 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:57.690+0000 7fa813200640 1 -- 172.21.15.32:0/4167303988 >> 172.21.15.32:0/4167303988 conn(0x7fa80c0b1bd0 msgr2=0x7fa80c0b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:54:57.691 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:57.690+0000 7fa813200640 1 -- 172.21.15.32:0/4167303988 shutdown_connections 2024-08-03T20:54:57.691 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:57.690+0000 7fa813200640 1 -- 172.21.15.32:0/4167303988 wait complete. 2024-08-03T20:54:57.716 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-08-03T20:54:57.716 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-08-03T20:54:57.716 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi032.front.sepia.ceph.com 2024-08-03T20:54:57.716 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-08-03T20:54:57.716 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-08-03T20:54:57.716 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-08-03T20:54:57.716 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2024-08-03T20:54:57.716 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-08-03T20:54:57.716 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-08-03T20:54:57.716 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-08-03T20:54:57.717 DEBUG:teuthology.orchestra.run.smithi032:> ip addr 2024-08-03T20:54:57.752 INFO:teuthology.orchestra.run.smithi032.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-08-03T20:54:57.752 INFO:teuthology.orchestra.run.smithi032.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-08-03T20:54:57.752 INFO:teuthology.orchestra.run.smithi032.stdout: inet 127.0.0.1/8 scope host lo 2024-08-03T20:54:57.752 INFO:teuthology.orchestra.run.smithi032.stdout: valid_lft forever preferred_lft forever 2024-08-03T20:54:57.752 INFO:teuthology.orchestra.run.smithi032.stdout: inet6 ::1/128 scope host 2024-08-03T20:54:57.752 INFO:teuthology.orchestra.run.smithi032.stdout: valid_lft forever preferred_lft forever 2024-08-03T20:54:57.753 INFO:teuthology.orchestra.run.smithi032.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-08-03T20:54:57.753 INFO:teuthology.orchestra.run.smithi032.stdout: link/ether 0c:c4:7a:6b:fd:0c brd ff:ff:ff:ff:ff:ff 2024-08-03T20:54:57.753 INFO:teuthology.orchestra.run.smithi032.stdout: altname enp7s0f0 2024-08-03T20:54:57.753 INFO:teuthology.orchestra.run.smithi032.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-08-03T20:54:57.753 INFO:teuthology.orchestra.run.smithi032.stdout: link/ether 0c:c4:7a:6b:fd:0d brd ff:ff:ff:ff:ff:ff 2024-08-03T20:54:57.753 INFO:teuthology.orchestra.run.smithi032.stdout: altname enp7s0f1 2024-08-03T20:54:57.753 INFO:teuthology.orchestra.run.smithi032.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-08-03T20:54:57.753 INFO:teuthology.orchestra.run.smithi032.stdout: link/ether ac:1f:6b:f8:26:02 brd ff:ff:ff:ff:ff:ff 2024-08-03T20:54:57.753 INFO:teuthology.orchestra.run.smithi032.stdout: altname enp1s0f0 2024-08-03T20:54:57.753 INFO:teuthology.orchestra.run.smithi032.stdout: inet 172.21.15.32/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-08-03T20:54:57.753 INFO:teuthology.orchestra.run.smithi032.stdout: valid_lft 42058sec preferred_lft 42058sec 2024-08-03T20:54:57.753 INFO:teuthology.orchestra.run.smithi032.stdout: inet6 fe80::ae1f:6bff:fef8:2602/64 scope link noprefixroute 2024-08-03T20:54:57.753 INFO:teuthology.orchestra.run.smithi032.stdout: valid_lft forever preferred_lft forever 2024-08-03T20:54:57.754 INFO:teuthology.orchestra.run.smithi032.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-08-03T20:54:57.754 INFO:teuthology.orchestra.run.smithi032.stdout: link/ether ac:1f:6b:f8:26:03 brd ff:ff:ff:ff:ff:ff 2024-08-03T20:54:57.754 INFO:teuthology.orchestra.run.smithi032.stdout: altname enp1s0f1 2024-08-03T20:54:57.754 INFO:teuthology.orchestra.run.smithi032.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-08-03T20:54:57.754 INFO:teuthology.orchestra.run.smithi032.stdout: link/ether 0a:23:2b:ca:2a:93 brd ff:ff:ff:ff:ff:ff 2024-08-03T20:54:57.754 INFO:teuthology.orchestra.run.smithi032.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-08-03T20:54:57.754 INFO:teuthology.orchestra.run.smithi032.stdout: valid_lft forever preferred_lft forever 2024-08-03T20:54:57.754 INFO:teuthology.orchestra.run.smithi032.stdout: inet6 fe80::d804:7cff:fed9:7ba4/64 scope link 2024-08-03T20:54:57.754 INFO:teuthology.orchestra.run.smithi032.stdout: valid_lft forever preferred_lft forever 2024-08-03T20:54:57.754 INFO:teuthology.orchestra.run.smithi032.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-08-03T20:54:57.754 INFO:teuthology.orchestra.run.smithi032.stdout: link/ether 22:86:28:30:4e:62 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-08-03T20:54:57.754 INFO:teuthology.orchestra.run.smithi032.stdout: inet6 fe80::2086:28ff:fe30:4e62/64 scope link 2024-08-03T20:54:57.755 INFO:teuthology.orchestra.run.smithi032.stdout: valid_lft forever preferred_lft forever 2024-08-03T20:54:57.755 INFO:teuthology.orchestra.run.smithi032.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-08-03T20:54:57.755 INFO:teuthology.orchestra.run.smithi032.stdout: link/ether 0a:23:2b:ca:2a:93 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-08-03T20:54:57.755 INFO:teuthology.orchestra.run.smithi032.stdout: inet6 fe80::823:2bff:feca:2a93/64 scope link 2024-08-03T20:54:57.755 INFO:teuthology.orchestra.run.smithi032.stdout: valid_lft forever preferred_lft forever 2024-08-03T20:54:57.755 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-08-03T20:54:57.755 DEBUG:teuthology.orchestra.run.smithi032:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-08-03T20:54:57.846 INFO:teuthology.orchestra.run.smithi032.stdout:1 2024-08-03T20:54:57.849 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-08-03T20:54:57.849 DEBUG:teuthology.orchestra.run.smithi032:> route 2024-08-03T20:54:57.885 INFO:teuthology.orchestra.run.smithi032.stdout:Kernel IP routing table 2024-08-03T20:54:57.886 INFO:teuthology.orchestra.run.smithi032.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-08-03T20:54:57.886 INFO:teuthology.orchestra.run.smithi032.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-08-03T20:54:57.886 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-08-03T20:54:57.886 INFO:teuthology.orchestra.run.smithi032.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-08-03T20:54:57.886 DEBUG:teuthology.orchestra.run.smithi032:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-08-03T20:54:57.886 DEBUG:teuthology.orchestra.run.smithi032:> set -e 2024-08-03T20:54:57.886 DEBUG:teuthology.orchestra.run.smithi032:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-08-03T20:54:57.886 DEBUG:teuthology.orchestra.run.smithi032:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-08-03T20:54:57.887 DEBUG:teuthology.orchestra.run.smithi032:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-08-03T20:54:57.887 DEBUG:teuthology.orchestra.run.smithi032:> ') 2024-08-03T20:54:58.005 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:58 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-08-03T20:54:58.076 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:58 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-08-03T20:54:58.085 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-08-03T20:54:58.085 DEBUG:teuthology.orchestra.run.smithi032:> ip netns list 2024-08-03T20:54:58.120 INFO:teuthology.orchestra.run.smithi032.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-08-03T20:54:58.120 INFO:teuthology.orchestra.run.smithi032.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-08-03T20:54:58.120 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-08-03T20:54:58.120 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-08-03T20:54:58.121 DEBUG:teuthology.orchestra.run.smithi032:> sudo ip link set brx.0 up 2024-08-03T20:54:58.216 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-08-03T20:54:58.216 DEBUG:teuthology.orchestra.run.smithi032:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-08-03T20:54:58.252 INFO:teuthology.orchestra.run.smithi032.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-08-03T20:54:58.252 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-08-03T20:54:58.252 DEBUG:teuthology.orchestra.run.smithi032:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-08-03T20:54:58.327 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2024-08-03T20:54:58.328 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-08-03T20:54:58.328 DEBUG:teuthology.orchestra.run.smithi032:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-08-03T20:54:58.526 INFO:teuthology.orchestra.run.smithi032.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-08-03T20:54:58.526 INFO:teuthology.orchestra.run.smithi032.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-08-03T20:54:58.526 INFO:teuthology.orchestra.run.smithi032.stdout:invalid new device string format 2024-08-03T20:54:58.546 INFO:teuthology.orchestra.run.smithi032.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-08-03T20:54:58.546 INFO:teuthology.orchestra.run.smithi032.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-08-03T20:54:58.546 INFO:teuthology.orchestra.run.smithi032.stdout:invalid new device string format 2024-08-03T20:54:58.546 INFO:teuthology.orchestra.run.smithi032.stdout:mount.ceph: resolved to: "172.21.15.32:3300,172.21.15.118:3300,172.21.15.118:3301" 2024-08-03T20:54:58.546 INFO:teuthology.orchestra.run.smithi032.stdout:mount.ceph: switching to using v1 address with old syntax 2024-08-03T20:54:58.575 INFO:teuthology.orchestra.run.smithi032.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-08-03T20:54:58.575 INFO:teuthology.orchestra.run.smithi032.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-08-03T20:54:58.575 INFO:teuthology.orchestra.run.smithi032.stdout:invalid new device string format 2024-08-03T20:54:58.575 INFO:teuthology.orchestra.run.smithi032.stdout:mount.ceph: resolved to: "172.21.15.32:3300,172.21.15.118:3300,172.21.15.118:3301" 2024-08-03T20:54:58.575 INFO:teuthology.orchestra.run.smithi032.stdout:mount.ceph: switching to using v1 address with old syntax 2024-08-03T20:54:58.575 INFO:teuthology.orchestra.run.smithi032.stdout:mount.ceph: resolved to: "172.21.15.32:6789,172.21.15.118:6789,172.21.15.118:6790" 2024-08-03T20:54:58.575 INFO:teuthology.orchestra.run.smithi032.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.32:6789,172.21.15.118:6789,172.21.15.118:6790:/ 2024-08-03T20:54:58.575 INFO:teuthology.orchestra.run.smithi032.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,key=0,fsid=40ab141a-01d7-49b9-ae3c-2edec9876e95" will pass to kernel 2024-08-03T20:54:58.577 INFO:teuthology.orchestra.run.smithi032.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2024-08-03T20:54:58.577 INFO:teuthology.orchestra.run.smithi032.stdout: You just mounted a file system that supports labels which does not 2024-08-03T20:54:58.577 INFO:teuthology.orchestra.run.smithi032.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-08-03T20:54:58.577 INFO:teuthology.orchestra.run.smithi032.stdout: applications will generate AVC messages and not be allowed access to 2024-08-03T20:54:58.577 INFO:teuthology.orchestra.run.smithi032.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-08-03T20:54:58.580 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-08-03T20:54:58.580 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-08-03T20:54:58.580 DEBUG:teuthology.orchestra.run.smithi032:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-08-03T20:54:58.636 DEBUG:teuthology.orchestra.run.smithi032:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-08-03T20:54:58.716 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:58 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-08-03T20:54:58.719 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:58 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-08-03T20:54:58.727 INFO:teuthology.orchestra.run.smithi032.stdout:client6434 2024-08-03T20:54:58.728 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-08-03T20:54:58.728 DEBUG:teuthology.orchestra.run.smithi032:> dd if=/proc/self/mounts of=/dev/stdout 2024-08-03T20:54:58.763 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-08-03T20:54:58.764 DEBUG:teuthology.orchestra.run.smithi032:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls -R .) 2024-08-03T20:54:58.807 INFO:teuthology.orchestra.run:waiting for 20 2024-08-03T20:54:58.840 INFO:tasks.ceph.mds.a.smithi032.stderr:2024-08-03T20:54:58.839+0000 7f1420200640 -1 log_channel(cluster) log [ERR] : dir 0x10000000000 object missing on disk; some files may be lost (/subdir) 2024-08-03T20:54:58.840 INFO:teuthology.orchestra.run.smithi032.stdout:.: 2024-08-03T20:54:58.840 INFO:teuthology.orchestra.run.smithi032.stdout:subdir 2024-08-03T20:54:58.840 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-03T20:54:58.840 INFO:teuthology.orchestra.run.smithi032.stdout:./subdir: 2024-08-03T20:54:58.841 INFO:tasks.cephfs.test_damage:Result: Mutation 'Delete 10000000000.00000000' did not caused DAMAGED state 2024-08-03T20:54:58.841 INFO:tasks.cephfs.test_damage:Applying mutation 'Delete 600.00000000' 2024-08-03T20:54:58.841 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-08-03T20:54:58.842 DEBUG:teuthology.orchestra.run.smithi032:> dd if=/proc/self/mounts of=/dev/stdout 2024-08-03T20:54:58.917 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-08-03T20:54:58.917 DEBUG:teuthology.orchestra.run.smithi032:> dd if=/proc/self/mounts of=/dev/stdout 2024-08-03T20:54:58.993 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-08-03T20:54:59.263 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:59.262+0000 7f47b0a00640 1 -- 172.21.15.32:0/3968410362 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f47ac15fff0 msgr2=0x7f47ac160400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:59.263 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:59.262+0000 7f47b0a00640 1 --2- 172.21.15.32:0/3968410362 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f47ac15fff0 0x7f47ac160400 secure :-1 s=READY pgs=1109 cs=0 l=1 rev1=1 crypto rx=0x7f47a4005fd0 tx=0x7f47a402aa90 comp rx=0 tx=0).stop 2024-08-03T20:54:59.263 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:59.262+0000 7f47b0a00640 1 -- 172.21.15.32:0/3968410362 shutdown_connections 2024-08-03T20:54:59.263 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:59.262+0000 7f47b0a00640 1 --2- 172.21.15.32:0/3968410362 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f47ac144f90 0x7f47ac074570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:59.263 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:59.262+0000 7f47b0a00640 1 --2- 172.21.15.32:0/3968410362 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f47ac15fff0 0x7f47ac160400 unknown :-1 s=CLOSED pgs=1109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:59.263 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:59.262+0000 7f47b0a00640 1 -- 172.21.15.32:0/3968410362 >> 172.21.15.32:0/3968410362 conn(0x7f47ac0b1bd0 msgr2=0x7f47ac0b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:54:59.263 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:59.262+0000 7f47b0a00640 1 -- 172.21.15.32:0/3968410362 shutdown_connections 2024-08-03T20:54:59.263 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:59.262+0000 7f47b0a00640 1 -- 172.21.15.32:0/3968410362 wait complete. 2024-08-03T20:54:59.264 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:59.264+0000 7f47b0a00640 1 Processor -- start 2024-08-03T20:54:59.265 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:59.264+0000 7f47b0a00640 1 -- start start 2024-08-03T20:54:59.267 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:59.266+0000 7f47b0a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f47ac06e930 0x7f47ac0cb2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:59.267 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:59.266+0000 7f47b0a00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f47ac144f90 0x7f47ac1e7350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:59.267 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:59.266+0000 7f47b0a00640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f47ac15fff0 0x7f47ac13fb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:59.267 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:59.266+0000 7f47b0a00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f47ac0cf840 con 0x7f47ac06e930 2024-08-03T20:54:59.267 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:59.266+0000 7f47b0a00640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f47ac145f30 con 0x7f47ac144f90 2024-08-03T20:54:59.267 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:59.266+0000 7f47b0a00640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7f47ac0cf9c0 con 0x7f47ac15fff0 2024-08-03T20:54:59.267 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:59.266+0000 7f47a9e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f47ac06e930 0x7f47ac0cb2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:59.267 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:59.266+0000 7f47a9e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f47ac06e930 0x7f47ac0cb2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:50030/0 (socket says 172.21.15.32:50030) 2024-08-03T20:54:59.268 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:59.266+0000 7f47a9e00640 1 -- 172.21.15.32:0/2343835145 learned_addr learned my addr 172.21.15.32:0/2343835145 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:54:59.268 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:59.266+0000 7f47a9400640 1 --2- 172.21.15.32:0/2343835145 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f47ac144f90 0x7f47ac1e7350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:59.268 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:59.266+0000 7f47aa800640 1 --2- 172.21.15.32:0/2343835145 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f47ac15fff0 0x7f47ac13fb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:59.268 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:59.266+0000 7f47a9e00640 1 -- 172.21.15.32:0/2343835145 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f47ac15fff0 msgr2=0x7f47ac13fb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:59.268 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:59.266+0000 7f47a9e00640 1 --2- 172.21.15.32:0/2343835145 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f47ac15fff0 0x7f47ac13fb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:59.268 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:59.266+0000 7f47a9e00640 1 -- 172.21.15.32:0/2343835145 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f47ac144f90 msgr2=0x7f47ac1e7350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:59.268 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:59.266+0000 7f47a9e00640 1 --2- 172.21.15.32:0/2343835145 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f47ac144f90 0x7f47ac1e7350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:59.268 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:59.266+0000 7f47a9e00640 1 -- 172.21.15.32:0/2343835145 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f47ac1f6e10 con 0x7f47ac06e930 2024-08-03T20:54:59.268 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:59.267+0000 7f47a9e00640 1 --2- 172.21.15.32:0/2343835145 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f47ac06e930 0x7f47ac0cb2a0 secure :-1 s=READY pgs=1110 cs=0 l=1 rev1=1 crypto rx=0x7f479c00c3a0 tx=0x7f479c00c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:54:59.269 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:59.267+0000 7f47a2000640 1 -- 172.21.15.32:0/2343835145 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f479c017070 con 0x7f47ac06e930 2024-08-03T20:54:59.269 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:59.267+0000 7f47a2000640 1 -- 172.21.15.32:0/2343835145 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f479c0053c0 con 0x7f47ac06e930 2024-08-03T20:54:59.269 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:59.267+0000 7f47a2000640 1 -- 172.21.15.32:0/2343835145 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f479c003850 con 0x7f47ac06e930 2024-08-03T20:54:59.269 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:59.267+0000 7f47b0a00640 1 -- 172.21.15.32:0/2343835145 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f47ac1f70f0 con 0x7f47ac06e930 2024-08-03T20:54:59.269 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:59.267+0000 7f47b0a00640 1 -- 172.21.15.32:0/2343835145 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f47ac1f76b0 con 0x7f47ac06e930 2024-08-03T20:54:59.269 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:59.268+0000 7f47b0a00640 1 -- 172.21.15.32:0/2343835145 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f47ac160c50 con 0x7f47ac06e930 2024-08-03T20:54:59.270 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:59.270+0000 7f47a2000640 1 -- 172.21.15.32:0/2343835145 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f479c01a070 con 0x7f47ac06e930 2024-08-03T20:54:59.271 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:59.271+0000 7f47a2000640 1 --2- 172.21.15.32:0/2343835145 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f478c077010 0x7f478c079520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:54:59.272 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:59.271+0000 7f47a2000640 1 -- 172.21.15.32:0/2343835145 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 5795+0+0 (secure 0 0 0) 0x7f479c097e40 con 0x7f47ac06e930 2024-08-03T20:54:59.272 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:59.271+0000 7f47a9400640 1 --2- 172.21.15.32:0/2343835145 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f478c077010 0x7f478c079520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:54:59.273 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:59.272+0000 7f47a9400640 1 --2- 172.21.15.32:0/2343835145 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f478c077010 0x7f478c079520 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f4798005fd0 tx=0x7f4798005850 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:54:59.276 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:59.276+0000 7f47a2000640 1 -- 172.21.15.32:0/2343835145 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7f479c060f40 con 0x7f47ac06e930 2024-08-03T20:54:59.497 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:59.497+0000 7f47b0a00640 1 -- 172.21.15.32:0/2343835145 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f47ac141250 con 0x7f47ac06e930 2024-08-03T20:54:59.720 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:59.497+0000 7f47a2000640 1 -- 172.21.15.32:0/2343835145 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 2 entries v84) ==== 80+0+122 (secure 0 0 0) 0x7f479c069210 con 0x7f47ac06e930 2024-08-03T20:54:59.720 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6835/1707454554 2024-08-04T20:54:38.409411+0000 2024-08-03T20:54:59.720 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6833/1707454554 2024-08-04T20:54:38.409411+0000 2024-08-03T20:54:59.722 INFO:teuthology.orchestra.run.smithi032.stderr:listed 2 entries 2024-08-03T20:54:59.722 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:59.500+0000 7f47b0a00640 1 -- 172.21.15.32:0/2343835145 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f478c077010 msgr2=0x7f478c079520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:59.722 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:59.500+0000 7f47b0a00640 1 --2- 172.21.15.32:0/2343835145 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f478c077010 0x7f478c079520 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f4798005fd0 tx=0x7f4798005850 comp rx=0 tx=0).stop 2024-08-03T20:54:59.722 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:59.500+0000 7f47b0a00640 1 -- 172.21.15.32:0/2343835145 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f47ac06e930 msgr2=0x7f47ac0cb2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:54:59.722 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:59.500+0000 7f47b0a00640 1 --2- 172.21.15.32:0/2343835145 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f47ac06e930 0x7f47ac0cb2a0 secure :-1 s=READY pgs=1110 cs=0 l=1 rev1=1 crypto rx=0x7f479c00c3a0 tx=0x7f479c00c860 comp rx=0 tx=0).stop 2024-08-03T20:54:59.722 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:59.500+0000 7f47b0a00640 1 -- 172.21.15.32:0/2343835145 shutdown_connections 2024-08-03T20:54:59.722 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:59.500+0000 7f47b0a00640 1 --2- 172.21.15.32:0/2343835145 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f478c077010 0x7f478c079520 unknown :-1 s=CLOSED pgs=460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:59.722 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:59.500+0000 7f47b0a00640 1 --2- 172.21.15.32:0/2343835145 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f47ac144f90 0x7f47ac1e7350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:59.723 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:59.500+0000 7f47b0a00640 1 --2- 172.21.15.32:0/2343835145 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f47ac15fff0 0x7f47ac13fb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:59.723 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:59.500+0000 7f47b0a00640 1 --2- 172.21.15.32:0/2343835145 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f47ac06e930 0x7f47ac0cb2a0 unknown :-1 s=CLOSED pgs=1110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:54:59.723 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:59.500+0000 7f47b0a00640 1 -- 172.21.15.32:0/2343835145 >> 172.21.15.32:0/2343835145 conn(0x7f47ac0b1bd0 msgr2=0x7f47ac15d790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:54:59.723 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:59.500+0000 7f47b0a00640 1 -- 172.21.15.32:0/2343835145 shutdown_connections 2024-08-03T20:54:59.723 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:54:59.500+0000 7f47b0a00640 1 -- 172.21.15.32:0/2343835145 wait complete. 2024-08-03T20:54:59.723 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-08-03T20:54:59.723 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-08-03T20:54:59.723 DEBUG:teuthology.orchestra.run.smithi032:> sudo umount /home/ubuntu/cephtest/mnt.0 -f 2024-08-03T20:54:59.795 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi032.front.sepia.ceph.com 2024-08-03T20:54:59.795 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-08-03T20:54:59.795 DEBUG:teuthology.orchestra.run.smithi032:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-08-03T20:54:59.831 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-08-03T20:55:00.102 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:00.101+0000 7fbf5a800640 1 -- 172.21.15.32:0/82223359 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbf540747b0 msgr2=0x7fbf540cc8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:55:00.102 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:00.101+0000 7fbf5a800640 1 --2- 172.21.15.32:0/82223359 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbf540747b0 0x7fbf540cc8e0 secure :-1 s=READY pgs=1111 cs=0 l=1 rev1=1 crypto rx=0x7fbf54071700 tx=0x7fbf4c02aa30 comp rx=0 tx=0).stop 2024-08-03T20:55:00.102 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:00.101+0000 7fbf5a800640 1 -- 172.21.15.32:0/82223359 shutdown_connections 2024-08-03T20:55:00.102 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:00.101+0000 7fbf5a800640 1 --2- 172.21.15.32:0/82223359 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbf540747b0 0x7fbf540cc8e0 unknown :-1 s=CLOSED pgs=1111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:00.102 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:00.101+0000 7fbf5a800640 1 --2- 172.21.15.32:0/82223359 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fbf54073e60 0x7fbf54074270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:00.102 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:00.101+0000 7fbf5a800640 1 --2- 172.21.15.32:0/82223359 >> v2:172.21.15.118:3300/0 conn(0x7fbf5406e930 0x7fbf54144f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:00.102 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:00.101+0000 7fbf5a800640 1 -- 172.21.15.32:0/82223359 >> 172.21.15.32:0/82223359 conn(0x7fbf540b1bd0 msgr2=0x7fbf540b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:55:00.102 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:00.101+0000 7fbf5a800640 1 -- 172.21.15.32:0/82223359 shutdown_connections 2024-08-03T20:55:00.102 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:00.101+0000 7fbf5a800640 1 -- 172.21.15.32:0/82223359 wait complete. 2024-08-03T20:55:00.103 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:00.103+0000 7fbf5a800640 1 Processor -- start 2024-08-03T20:55:00.104 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:00.103+0000 7fbf5a800640 1 -- start start 2024-08-03T20:55:00.107 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:00.105+0000 7fbf5a800640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fbf5406e930 0x7fbf540ca2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:55:00.107 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:00.105+0000 7fbf5a800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbf54073e60 0x7fbf54150890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:55:00.107 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:00.105+0000 7fbf5a800640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fbf540747b0 0x7fbf54150dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:55:00.107 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:00.105+0000 7fbf5a800640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fbf540cf9c0 con 0x7fbf54073e60 2024-08-03T20:55:00.107 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:00.105+0000 7fbf5a800640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7fbf540cf840 con 0x7fbf5406e930 2024-08-03T20:55:00.107 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:00.105+0000 7fbf5a800640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7fbf540cce20 con 0x7fbf540747b0 2024-08-03T20:55:00.107 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:00.105+0000 7fbf52a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbf54073e60 0x7fbf54150890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:55:00.107 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:00.105+0000 7fbf52a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbf54073e60 0x7fbf54150890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:47826/0 (socket says 172.21.15.32:47826) 2024-08-03T20:55:00.107 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:00.105+0000 7fbf52a00640 1 -- 172.21.15.32:0/242879679 learned_addr learned my addr 172.21.15.32:0/242879679 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:55:00.108 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:00.105+0000 7fbf53400640 1 --2- 172.21.15.32:0/242879679 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fbf5406e930 0x7fbf540ca2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:55:00.108 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:00.105+0000 7fbf53e00640 1 --2- 172.21.15.32:0/242879679 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fbf540747b0 0x7fbf54150dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:55:00.108 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:00.106+0000 7fbf52a00640 1 -- 172.21.15.32:0/242879679 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fbf540747b0 msgr2=0x7fbf54150dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:55:00.108 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:00.106+0000 7fbf52a00640 1 --2- 172.21.15.32:0/242879679 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fbf540747b0 0x7fbf54150dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:00.108 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:00.106+0000 7fbf52a00640 1 -- 172.21.15.32:0/242879679 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fbf5406e930 msgr2=0x7fbf540ca2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:55:00.108 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:00.106+0000 7fbf52a00640 1 --2- 172.21.15.32:0/242879679 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fbf5406e930 0x7fbf540ca2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:00.108 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:00.106+0000 7fbf52a00640 1 -- 172.21.15.32:0/242879679 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbf540c65c0 con 0x7fbf54073e60 2024-08-03T20:55:00.108 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:00.106+0000 7fbf53e00640 1 --2- 172.21.15.32:0/242879679 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fbf540747b0 0x7fbf54150dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-03T20:55:00.108 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:00.106+0000 7fbf52a00640 1 --2- 172.21.15.32:0/242879679 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbf54073e60 0x7fbf54150890 secure :-1 s=READY pgs=1112 cs=0 l=1 rev1=1 crypto rx=0x7fbf4000e7f0 tx=0x7fbf4000ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:55:00.108 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:00.106+0000 7fbf4b400640 1 -- 172.21.15.32:0/242879679 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf4000ccc0 con 0x7fbf54073e60 2024-08-03T20:55:00.108 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:00.106+0000 7fbf4b400640 1 -- 172.21.15.32:0/242879679 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fbf4000ce80 con 0x7fbf54073e60 2024-08-03T20:55:00.109 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:00.106+0000 7fbf4b400640 1 -- 172.21.15.32:0/242879679 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf40005280 con 0x7fbf54073e60 2024-08-03T20:55:00.109 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:00.106+0000 7fbf5a800640 1 -- 172.21.15.32:0/242879679 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbf540c68a0 con 0x7fbf54073e60 2024-08-03T20:55:00.109 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:00.106+0000 7fbf5a800640 1 -- 172.21.15.32:0/242879679 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbf54150090 con 0x7fbf54073e60 2024-08-03T20:55:00.109 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:00.108+0000 7fbf5a800640 1 -- 172.21.15.32:0/242879679 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbf540c6330 con 0x7fbf54073e60 2024-08-03T20:55:00.109 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:00.109+0000 7fbf4b400640 1 -- 172.21.15.32:0/242879679 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7fbf40005420 con 0x7fbf54073e60 2024-08-03T20:55:00.110 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:00.110+0000 7fbf4b400640 1 --2- 172.21.15.32:0/242879679 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fbf34077040 0x7fbf34079550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:55:00.110 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:00.110+0000 7fbf4b400640 1 -- 172.21.15.32:0/242879679 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 5795+0+0 (secure 0 0 0) 0x7fbf400991d0 con 0x7fbf54073e60 2024-08-03T20:55:00.110 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:00.110+0000 7fbf53400640 1 --2- 172.21.15.32:0/242879679 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fbf34077040 0x7fbf34079550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:55:00.111 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:00.111+0000 7fbf53400640 1 --2- 172.21.15.32:0/242879679 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fbf34077040 0x7fbf34079550 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7fbf44009b30 tx=0x7fbf44005f90 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:55:00.115 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:00.114+0000 7fbf4b400640 1 -- 172.21.15.32:0/242879679 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7fbf400622d0 con 0x7fbf54073e60 2024-08-03T20:55:00.358 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:00.358+0000 7fbf5a800640 1 -- 172.21.15.32:0/242879679 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7fbf54074270 con 0x7fbf54073e60 2024-08-03T20:55:00.359 INFO:tasks.ceph.mon.a.smithi032.stderr:2024-08-03T20:55:00.359+0000 7f4e6d600640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-08-03T20:55:00.361 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:00.361+0000 7fbf4b400640 1 -- 172.21.15.32:0/242879679 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v100) ==== 161+0+0 (secure 0 0 0) 0x7fbf40061bd0 con 0x7fbf54073e60 2024-08-03T20:55:00.361 INFO:teuthology.orchestra.run.smithi032.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-08-03T20:55:00.364 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:00.363+0000 7fbf5a800640 1 -- 172.21.15.32:0/242879679 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fbf34077040 msgr2=0x7fbf34079550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:55:00.364 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:00.363+0000 7fbf5a800640 1 --2- 172.21.15.32:0/242879679 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fbf34077040 0x7fbf34079550 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7fbf44009b30 tx=0x7fbf44005f90 comp rx=0 tx=0).stop 2024-08-03T20:55:00.364 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:00.364+0000 7fbf5a800640 1 -- 172.21.15.32:0/242879679 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbf54073e60 msgr2=0x7fbf54150890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:55:00.364 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:00.364+0000 7fbf5a800640 1 --2- 172.21.15.32:0/242879679 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbf54073e60 0x7fbf54150890 secure :-1 s=READY pgs=1112 cs=0 l=1 rev1=1 crypto rx=0x7fbf4000e7f0 tx=0x7fbf4000ecb0 comp rx=0 tx=0).stop 2024-08-03T20:55:00.365 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:00.364+0000 7fbf5a800640 1 -- 172.21.15.32:0/242879679 shutdown_connections 2024-08-03T20:55:00.365 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:00.364+0000 7fbf5a800640 1 --2- 172.21.15.32:0/242879679 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fbf34077040 0x7fbf34079550 unknown :-1 s=CLOSED pgs=461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:00.365 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:00.364+0000 7fbf5a800640 1 --2- 172.21.15.32:0/242879679 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fbf540747b0 0x7fbf54150dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:00.365 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:00.364+0000 7fbf5a800640 1 --2- 172.21.15.32:0/242879679 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbf54073e60 0x7fbf54150890 unknown :-1 s=CLOSED pgs=1112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:00.365 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:00.364+0000 7fbf5a800640 1 --2- 172.21.15.32:0/242879679 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fbf5406e930 0x7fbf540ca2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:00.365 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:00.364+0000 7fbf5a800640 1 -- 172.21.15.32:0/242879679 >> 172.21.15.32:0/242879679 conn(0x7fbf540b1bd0 msgr2=0x7fbf54068940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:55:00.365 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:00.365+0000 7fbf5a800640 1 -- 172.21.15.32:0/242879679 shutdown_connections 2024-08-03T20:55:00.365 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:00.365+0000 7fbf5a800640 1 -- 172.21.15.32:0/242879679 wait complete. 2024-08-03T20:55:00.369 INFO:tasks.ceph.mds.a.smithi032.stderr: -707> 2024-08-03T20:54:54.489+0000 7f1425200640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-08-03T20:55:00.370 INFO:tasks.ceph.mds.a.smithi032.stderr: -182> 2024-08-03T20:54:58.839+0000 7f1420200640 -1 log_channel(cluster) log [ERR] : dir 0x10000000000 object missing on disk; some files may be lost (/subdir) 2024-08-03T20:55:00.392 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mds repaired 0 2024-08-03T20:55:00.487 INFO:tasks.ceph.mds.a.smithi032.stdout:starting mds.a at 2024-08-03T20:55:00.625 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:00.623+0000 7f951e400640 1 -- 172.21.15.32:0/4055969495 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f951806e930 msgr2=0x7f9518144f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:55:00.625 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:00.623+0000 7f951e400640 1 --2- 172.21.15.32:0/4055969495 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f951806e930 0x7f9518144f90 secure :-1 s=READY pgs=1115 cs=0 l=1 rev1=1 crypto rx=0x7f95180afaf0 tx=0x7f950802a560 comp rx=0 tx=0).stop 2024-08-03T20:55:00.625 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:00.623+0000 7f951e400640 1 -- 172.21.15.32:0/4055969495 shutdown_connections 2024-08-03T20:55:00.625 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:00.623+0000 7f951e400640 1 --2- 172.21.15.32:0/4055969495 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f95180747b0 0x7f95180cc8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:00.625 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:00.623+0000 7f951e400640 1 --2- 172.21.15.32:0/4055969495 >> v2:172.21.15.118:3300/0 conn(0x7f9518073e60 0x7f9518074270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:00.625 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:00.623+0000 7f951e400640 1 --2- 172.21.15.32:0/4055969495 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f951806e930 0x7f9518144f90 unknown :-1 s=CLOSED pgs=1115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:00.625 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:00.623+0000 7f951e400640 1 -- 172.21.15.32:0/4055969495 >> 172.21.15.32:0/4055969495 conn(0x7f95180b1bd0 msgr2=0x7f95180b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:55:00.625 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:00.623+0000 7f951e400640 1 -- 172.21.15.32:0/4055969495 shutdown_connections 2024-08-03T20:55:00.625 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:00.623+0000 7f951e400640 1 -- 172.21.15.32:0/4055969495 wait complete. 2024-08-03T20:55:00.626 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:00.625+0000 7f951e400640 1 Processor -- start 2024-08-03T20:55:00.626 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:00.625+0000 7f951e400640 1 -- start start 2024-08-03T20:55:00.632 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:00.626+0000 7f951e400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f951806e930 0x7f95180caaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:55:00.632 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:00.627+0000 7f951e400640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f9518073e60 0x7f951815bbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:55:00.632 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:00.627+0000 7f951e400640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f95180747b0 0x7f95181e4c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:55:00.633 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:00.627+0000 7f951e400640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f95180cf9c0 con 0x7f951806e930 2024-08-03T20:55:00.633 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:00.627+0000 7f951e400640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f95180cf840 con 0x7f95180747b0 2024-08-03T20:55:00.633 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:00.627+0000 7f951e400640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7f95180cce20 con 0x7f9518073e60 2024-08-03T20:55:00.633 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:00.627+0000 7f9517400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f951806e930 0x7f95180caaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:55:00.633 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:00.627+0000 7f9517400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f951806e930 0x7f95180caaa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:47868/0 (socket says 172.21.15.32:47868) 2024-08-03T20:55:00.633 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:00.627+0000 7f9517400640 1 -- 172.21.15.32:0/3899583877 learned_addr learned my addr 172.21.15.32:0/3899583877 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:55:00.633 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:00.627+0000 7f9517e00640 1 --2- 172.21.15.32:0/3899583877 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f95180747b0 0x7f95181e4c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:55:00.633 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:00.627+0000 7f9516a00640 1 --2- 172.21.15.32:0/3899583877 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f9518073e60 0x7f951815bbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:55:00.633 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:00.627+0000 7f9517400640 1 -- 172.21.15.32:0/3899583877 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f9518073e60 msgr2=0x7f951815bbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:55:00.633 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:00.627+0000 7f9517400640 1 --2- 172.21.15.32:0/3899583877 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f9518073e60 0x7f951815bbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:00.633 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:00.627+0000 7f9517400640 1 -- 172.21.15.32:0/3899583877 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f95180747b0 msgr2=0x7f95181e4c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:55:00.633 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:00.627+0000 7f9517400640 1 --2- 172.21.15.32:0/3899583877 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f95180747b0 0x7f95181e4c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:00.633 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:00.627+0000 7f9517400640 1 -- 172.21.15.32:0/3899583877 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f95180c9a00 con 0x7f951806e930 2024-08-03T20:55:00.634 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:00.627+0000 7f9517e00640 1 --2- 172.21.15.32:0/3899583877 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f95180747b0 0x7f95181e4c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-03T20:55:00.634 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:00.627+0000 7f9516a00640 1 --2- 172.21.15.32:0/3899583877 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f9518073e60 0x7f951815bbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-03T20:55:00.634 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:00.628+0000 7f9517400640 1 --2- 172.21.15.32:0/3899583877 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f951806e930 0x7f95180caaa0 secure :-1 s=READY pgs=1116 cs=0 l=1 rev1=1 crypto rx=0x7f9508034040 tx=0x7f95080038a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:55:00.634 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:00.628+0000 7f950f400640 1 -- 172.21.15.32:0/3899583877 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f950803e030 con 0x7f951806e930 2024-08-03T20:55:00.634 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:00.628+0000 7f950f400640 1 -- 172.21.15.32:0/3899583877 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f9508003ca0 con 0x7f951806e930 2024-08-03T20:55:00.634 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:00.628+0000 7f950f400640 1 -- 172.21.15.32:0/3899583877 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9508035040 con 0x7f951806e930 2024-08-03T20:55:00.634 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:00.628+0000 7f951e400640 1 -- 172.21.15.32:0/3899583877 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f95180c9ce0 con 0x7f951806e930 2024-08-03T20:55:00.634 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:00.628+0000 7f951e400640 1 -- 172.21.15.32:0/3899583877 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f95180ca2a0 con 0x7f951806e930 2024-08-03T20:55:00.634 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:00.629+0000 7f951e400640 1 -- 172.21.15.32:0/3899583877 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9518074270 con 0x7f951806e930 2024-08-03T20:55:00.634 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:00.631+0000 7f950f400640 1 -- 172.21.15.32:0/3899583877 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f9508041030 con 0x7f951806e930 2024-08-03T20:55:00.635 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:00.631+0000 7f950f400640 1 --2- 172.21.15.32:0/3899583877 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f94f8077010 0x7f94f8079520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:55:00.635 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:00.632+0000 7f950f400640 1 -- 172.21.15.32:0/3899583877 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 5881+0+0 (secure 0 0 0) 0x7f95080b9050 con 0x7f951806e930 2024-08-03T20:55:00.635 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:00.632+0000 7f9516a00640 1 --2- 172.21.15.32:0/3899583877 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f94f8077010 0x7f94f8079520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:55:00.635 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:00.633+0000 7f9516a00640 1 --2- 172.21.15.32:0/3899583877 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f94f8077010 0x7f94f8079520 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7f95180cd360 tx=0x7f9504005f90 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:55:00.637 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:00.636+0000 7f950f400640 1 -- 172.21.15.32:0/3899583877 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7f9508082100 con 0x7f951806e930 2024-08-03T20:55:00.926 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:00.925+0000 7f951e400640 1 -- 172.21.15.32:0/3899583877 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "mds repaired", "role": "0"} v 0) -- 0x7f95180b2250 con 0x7f951806e930 2024-08-03T20:55:01.363 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:01.361+0000 7f950f400640 1 -- 172.21.15.32:0/3899583877 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "mds repaired", "role": "0"}]=0 nothing to do: rank is not damaged v101) ==== 107+0+0 (secure 0 0 0) 0x7f9508081a00 con 0x7f951806e930 2024-08-03T20:55:01.363 INFO:teuthology.orchestra.run.smithi032.stderr:nothing to do: rank is not damaged 2024-08-03T20:55:01.366 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:01.364+0000 7f951e400640 1 -- 172.21.15.32:0/3899583877 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f94f8077010 msgr2=0x7f94f8079520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:55:01.366 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:01.364+0000 7f951e400640 1 --2- 172.21.15.32:0/3899583877 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f94f8077010 0x7f94f8079520 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7f95180cd360 tx=0x7f9504005f90 comp rx=0 tx=0).stop 2024-08-03T20:55:01.367 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:01.364+0000 7f951e400640 1 -- 172.21.15.32:0/3899583877 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f951806e930 msgr2=0x7f95180caaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:55:01.367 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:01.364+0000 7f951e400640 1 --2- 172.21.15.32:0/3899583877 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f951806e930 0x7f95180caaa0 secure :-1 s=READY pgs=1116 cs=0 l=1 rev1=1 crypto rx=0x7f9508034040 tx=0x7f95080038a0 comp rx=0 tx=0).stop 2024-08-03T20:55:01.367 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:01.364+0000 7f951e400640 1 -- 172.21.15.32:0/3899583877 shutdown_connections 2024-08-03T20:55:01.367 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:01.364+0000 7f951e400640 1 --2- 172.21.15.32:0/3899583877 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f94f8077010 0x7f94f8079520 unknown :-1 s=CLOSED pgs=462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:01.367 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:01.364+0000 7f951e400640 1 --2- 172.21.15.32:0/3899583877 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f95180747b0 0x7f95181e4c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:01.367 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:01.364+0000 7f951e400640 1 --2- 172.21.15.32:0/3899583877 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f9518073e60 0x7f951815bbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:01.367 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:01.364+0000 7f951e400640 1 --2- 172.21.15.32:0/3899583877 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f951806e930 0x7f95180caaa0 unknown :-1 s=CLOSED pgs=1116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:01.367 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:01.364+0000 7f951e400640 1 -- 172.21.15.32:0/3899583877 >> 172.21.15.32:0/3899583877 conn(0x7f95180b1bd0 msgr2=0x7f951815f410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:55:01.367 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:01.365+0000 7f951e400640 1 -- 172.21.15.32:0/3899583877 shutdown_connections 2024-08-03T20:55:01.367 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:01.365+0000 7f951e400640 1 -- 172.21.15.32:0/3899583877 wait complete. 2024-08-03T20:55:01.394 DEBUG:teuthology.orchestra.run.smithi032:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata import - 2024-08-03T20:55:01.539 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:01.537+0000 7f003480ef40 1 Processor -- start 2024-08-03T20:55:01.539 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:01.537+0000 7f003480ef40 1 -- start start 2024-08-03T20:55:01.541 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:01.538+0000 7f003480ef40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5616cccc9360 0x5616ccc51230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:55:01.541 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:01.538+0000 7f003480ef40 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5616ccce6cb0 0x5616cccba410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:55:01.541 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:01.539+0000 7f003480ef40 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x5616ccce73f0 0x5616ccce5360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:55:01.541 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:01.539+0000 7f003480ef40 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x5616ccb60e30 con 0x5616ccce6cb0 2024-08-03T20:55:01.542 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:01.539+0000 7f003480ef40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x5616ccbbebf0 con 0x5616cccc9360 2024-08-03T20:55:01.542 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:01.539+0000 7f003480ef40 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x5616cccca440 con 0x5616ccce73f0 2024-08-03T20:55:01.542 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:01.539+0000 7f002d000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5616ccce6cb0 0x5616cccba410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:55:01.542 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:01.539+0000 7f002d000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5616ccce6cb0 0x5616cccba410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:47892/0 (socket says 172.21.15.32:47892) 2024-08-03T20:55:01.542 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:01.539+0000 7f002d000640 1 -- 172.21.15.32:0/464777876 learned_addr learned my addr 172.21.15.32:0/464777876 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:55:01.542 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:01.539+0000 7f002e400640 1 --2- 172.21.15.32:0/464777876 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x5616ccce73f0 0x5616ccce5360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:55:01.542 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:01.539+0000 7f002da00640 1 --2- 172.21.15.32:0/464777876 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5616cccc9360 0x5616ccc51230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:55:01.542 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:01.539+0000 7f002d000640 1 -- 172.21.15.32:0/464777876 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x5616ccce73f0 msgr2=0x5616ccce5360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:55:01.542 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:01.539+0000 7f002d000640 1 --2- 172.21.15.32:0/464777876 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x5616ccce73f0 0x5616ccce5360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:01.542 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:01.539+0000 7f002d000640 1 -- 172.21.15.32:0/464777876 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5616cccc9360 msgr2=0x5616ccc51230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:55:01.542 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:01.539+0000 7f002d000640 1 --2- 172.21.15.32:0/464777876 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5616cccc9360 0x5616ccc51230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:01.542 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:01.539+0000 7f002d000640 1 -- 172.21.15.32:0/464777876 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5616cccdfa30 con 0x5616ccce6cb0 2024-08-03T20:55:01.543 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:01.540+0000 7f002d000640 1 --2- 172.21.15.32:0/464777876 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5616ccce6cb0 0x5616cccba410 secure :-1 s=READY pgs=1118 cs=0 l=1 rev1=1 crypto rx=0x7f001c001d50 tx=0x7f001c00b5b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ce9c3fca2dd232cf server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:55:01.543 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:01.540+0000 7f0027400640 1 -- 172.21.15.32:0/464777876 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f001c00bd80 con 0x5616ccce6cb0 2024-08-03T20:55:01.543 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:01.540+0000 7f0027400640 1 -- 172.21.15.32:0/464777876 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f001c00f400 con 0x5616ccce6cb0 2024-08-03T20:55:01.543 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:01.540+0000 7f0027400640 1 -- 172.21.15.32:0/464777876 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f001c00f850 con 0x5616ccce6cb0 2024-08-03T20:55:01.544 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:01.540+0000 7f003480ef40 1 -- 172.21.15.32:0/464777876 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5616ccce6cb0 msgr2=0x5616cccba410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:55:01.544 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:01.540+0000 7f003480ef40 1 --2- 172.21.15.32:0/464777876 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5616ccce6cb0 0x5616cccba410 secure :-1 s=READY pgs=1118 cs=0 l=1 rev1=1 crypto rx=0x7f001c001d50 tx=0x7f001c00b5b0 comp rx=0 tx=0).stop 2024-08-03T20:55:01.544 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:01.541+0000 7f003480ef40 1 -- 172.21.15.32:0/464777876 shutdown_connections 2024-08-03T20:55:01.544 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:01.541+0000 7f003480ef40 1 --2- 172.21.15.32:0/464777876 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x5616ccce73f0 0x5616ccce5360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:01.544 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:01.541+0000 7f003480ef40 1 --2- 172.21.15.32:0/464777876 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5616ccce6cb0 0x5616cccba410 unknown :-1 s=CLOSED pgs=1118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:01.544 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:01.541+0000 7f003480ef40 1 --2- 172.21.15.32:0/464777876 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5616cccc9360 0x5616ccc51230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:01.544 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:01.541+0000 7f003480ef40 1 -- 172.21.15.32:0/464777876 >> 172.21.15.32:0/464777876 conn(0x5616ccb32ac0 msgr2=0x5616ccce3c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:55:01.544 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:01.541+0000 7f003480ef40 1 -- 172.21.15.32:0/464777876 shutdown_connections 2024-08-03T20:55:01.545 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:01.541+0000 7f003480ef40 1 -- 172.21.15.32:0/464777876 wait complete. 2024-08-03T20:55:01.545 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:01.542+0000 7f003480ef40 1 Processor -- start 2024-08-03T20:55:01.545 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:01.542+0000 7f003480ef40 1 -- start start 2024-08-03T20:55:01.547 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:01.543+0000 7f003480ef40 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5616cccc9360 0x5616ccc52b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:55:01.547 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:01.543+0000 7f003480ef40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5616ccce6cb0 0x5616ccd80d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:55:01.547 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:01.543+0000 7f003480ef40 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x5616ccce73f0 0x5616ccd812a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:55:01.547 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:01.543+0000 7f003480ef40 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x5616ccc5bd30 con 0x5616cccc9360 2024-08-03T20:55:01.547 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:01.543+0000 7f003480ef40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x5616cccc58d0 con 0x5616ccce6cb0 2024-08-03T20:55:01.547 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:01.543+0000 7f003480ef40 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x5616cccbd900 con 0x5616ccce73f0 2024-08-03T20:55:01.547 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:01.544+0000 7f002da00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5616cccc9360 0x5616ccc52b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:55:01.547 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:01.544+0000 7f002da00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5616cccc9360 0x5616ccc52b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:47908/0 (socket says 172.21.15.32:47908) 2024-08-03T20:55:01.547 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:01.544+0000 7f002da00640 1 -- 172.21.15.32:0/451983607 learned_addr learned my addr 172.21.15.32:0/451983607 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:55:01.547 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:01.544+0000 7f002e400640 1 --2- 172.21.15.32:0/451983607 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x5616ccce73f0 0x5616ccd812a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:55:01.547 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:01.544+0000 7f002d000640 1 --2- 172.21.15.32:0/451983607 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5616ccce6cb0 0x5616ccd80d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:55:01.548 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:01.544+0000 7f002da00640 1 -- 172.21.15.32:0/451983607 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x5616ccce73f0 msgr2=0x5616ccd812a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:55:01.548 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:01.544+0000 7f002da00640 1 --2- 172.21.15.32:0/451983607 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x5616ccce73f0 0x5616ccd812a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:01.548 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:01.544+0000 7f002da00640 1 -- 172.21.15.32:0/451983607 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5616ccce6cb0 msgr2=0x5616ccd80d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:55:01.548 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:01.544+0000 7f002da00640 1 --2- 172.21.15.32:0/451983607 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5616ccce6cb0 0x5616ccd80d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:01.548 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:01.544+0000 7f002da00640 1 -- 172.21.15.32:0/451983607 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5616cccbdb50 con 0x5616cccc9360 2024-08-03T20:55:01.548 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:01.544+0000 7f002d000640 1 --2- 172.21.15.32:0/451983607 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5616ccce6cb0 0x5616ccd80d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-03T20:55:01.548 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:01.544+0000 7f002da00640 1 --2- 172.21.15.32:0/451983607 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5616cccc9360 0x5616ccc52b60 secure :-1 s=READY pgs=1119 cs=0 l=1 rev1=1 crypto rx=0x7f00200086c0 tx=0x7f00200086f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:55:01.548 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:01.545+0000 7f0026000640 1 -- 172.21.15.32:0/451983607 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f002000a650 con 0x5616cccc9360 2024-08-03T20:55:01.548 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:01.545+0000 7f0026000640 1 -- 172.21.15.32:0/451983607 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f002000a7f0 con 0x5616cccc9360 2024-08-03T20:55:01.548 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:01.545+0000 7f0026000640 1 -- 172.21.15.32:0/451983607 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f002000ac40 con 0x5616cccc9360 2024-08-03T20:55:01.548 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:01.545+0000 7f003480ef40 1 -- 172.21.15.32:0/451983607 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x5616cccbde30 con 0x5616cccc9360 2024-08-03T20:55:01.548 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:01.545+0000 7f003480ef40 1 -- 172.21.15.32:0/451983607 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5616cccc5610 con 0x5616cccc9360 2024-08-03T20:55:01.549 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:01.547+0000 7f0026000640 1 -- 172.21.15.32:0/451983607 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f0020016430 con 0x5616cccc9360 2024-08-03T20:55:01.549 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:01.548+0000 7f0026000640 1 --2- 172.21.15.32:0/451983607 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f00100759f0 0x7f0010077f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:55:01.549 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:01.548+0000 7f0026000640 1 -- 172.21.15.32:0/451983607 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 5881+0+0 (secure 0 0 0) 0x7f0020097dd0 con 0x5616cccc9360 2024-08-03T20:55:01.550 INFO:teuthology.orchestra.run.smithi032.stdout:Importing pool 2024-08-03T20:55:01.551 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:01.548+0000 7f003480ef40 1 --2- 172.21.15.32:0/451983607 >> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] conn(0x5616cccbb090 0x5616cccbb4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:55:01.551 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:01.548+0000 7f002d000640 1 --2- 172.21.15.32:0/451983607 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f00100759f0 0x7f0010077f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:55:01.551 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:01.548+0000 7f003480ef40 1 -- 172.21.15.32:0/451983607 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:1 12.10 12:09401a81:::601.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e85) -- 0x5616cccded00 con 0x5616cccbb090 2024-08-03T20:55:01.551 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:01.549+0000 7f002e400640 1 --2- 172.21.15.32:0/451983607 >> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] conn(0x5616cccbb090 0x5616cccbb4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:55:01.551 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:01.549+0000 7f002d000640 1 --2- 172.21.15.32:0/451983607 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f00100759f0 0x7f0010077f00 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7f001c001d50 tx=0x7f001c000ed0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:55:01.551 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:01.549+0000 7f002e400640 1 --2- 172.21.15.32:0/451983607 >> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] conn(0x5616cccbb090 0x5616cccbb4a0 crc :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:55:01.553 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:01.551+0000 7f002e400640 1 -- 172.21.15.32:0/451983607 <== osd.0 v2:172.21.15.32:6816/3066156116 1 ==== osd_op_reply(1 601.00000000 [create] v85'43 uv16 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f002800b040 con 0x5616cccbb090 2024-08-03T20:55:01.553 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:01.551+0000 7f003480ef40 1 -- 172.21.15.32:0/451983607 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:2 12.10 12:09401a81:::601.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) -- 0x5616ccc530a0 con 0x5616cccbb090 2024-08-03T20:55:01.555 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:01.553+0000 7f002e400640 1 -- 172.21.15.32:0/451983607 <== osd.0 v2:172.21.15.32:6816/3066156116 2 ==== osd_op_reply(2 601.00000000 [delete] v85'44 uv44 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f002800b040 con 0x5616cccbb090 2024-08-03T20:55:01.555 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:01.553+0000 7f003480ef40 1 -- 172.21.15.32:0/451983607 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:3 12.10 12:09401a81:::601.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e85) -- 0x5616cccbb9e0 con 0x5616cccbb090 2024-08-03T20:55:01.556 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::601.00000000:head# 2024-08-03T20:55:01.557 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:01.555+0000 7f002e400640 1 -- 172.21.15.32:0/451983607 <== osd.0 v2:172.21.15.32:6816/3066156116 3 ==== osd_op_reply(3 601.00000000 [create] v85'45 uv45 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f002800b040 con 0x5616cccbb090 2024-08-03T20:55:01.557 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:01.555+0000 7f003480ef40 1 -- 172.21.15.32:0/451983607 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:4 12.10 12:09401a81:::601.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e85) -- 0x5616cccbbd80 con 0x5616cccbb090 2024-08-03T20:55:01.558 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:01.557+0000 7f002e400640 1 -- 172.21.15.32:0/451983607 <== osd.0 v2:172.21.15.32:6816/3066156116 4 ==== osd_op_reply(4 601.00000000 [setxattr (30)] v85'46 uv46 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f002800b040 con 0x5616cccbb090 2024-08-03T20:55:01.558 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:01.557+0000 7f003480ef40 1 -- 172.21.15.32:0/451983607 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:5 12.10 12:09401a81:::601.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e85) -- 0x5616ccd8b040 con 0x5616cccbb090 2024-08-03T20:55:01.560 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:01.559+0000 7f002e400640 1 -- 172.21.15.32:0/451983607 <== osd.0 v2:172.21.15.32:6816/3066156116 5 ==== osd_op_reply(5 601.00000000 [setxattr (62)] v85'47 uv47 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f002800b040 con 0x5616cccbb090 2024-08-03T20:55:01.560 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:01.559+0000 7f003480ef40 1 -- 172.21.15.32:0/451983607 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:6 12.10 12:09401a81:::601.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e85) -- 0x5616ccd8a030 con 0x5616cccbb090 2024-08-03T20:55:01.562 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:01.560+0000 7f002e400640 1 -- 172.21.15.32:0/451983607 <== osd.0 v2:172.21.15.32:6816/3066156116 6 ==== osd_op_reply(6 601.00000000 [omap-set-header] v85'48 uv48 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f002800b040 con 0x5616cccbb090 2024-08-03T20:55:01.562 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:01.560+0000 7f003480ef40 1 -- 172.21.15.32:0/451983607 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:7 12.10 12:0bd6d154:::602.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e85) -- 0x5616ccd8a3d0 con 0x5616cccbb090 2024-08-03T20:55:01.563 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:01.562+0000 7f002e400640 1 -- 172.21.15.32:0/451983607 <== osd.0 v2:172.21.15.32:6816/3066156116 7 ==== osd_op_reply(7 602.00000000 [create] v85'49 uv22 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f002800b040 con 0x5616cccbb090 2024-08-03T20:55:01.564 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:01.562+0000 7f003480ef40 1 -- 172.21.15.32:0/451983607 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:8 12.10 12:0bd6d154:::602.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) -- 0x5616ccd8a770 con 0x5616cccbb090 2024-08-03T20:55:01.565 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:01.564+0000 7f002e400640 1 -- 172.21.15.32:0/451983607 <== osd.0 v2:172.21.15.32:6816/3066156116 8 ==== osd_op_reply(8 602.00000000 [delete] v85'50 uv50 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f002800b040 con 0x5616cccbb090 2024-08-03T20:55:01.565 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:01.564+0000 7f003480ef40 1 -- 172.21.15.32:0/451983607 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:9 12.10 12:0bd6d154:::602.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e85) -- 0x5616ccd8a770 con 0x5616cccbb090 2024-08-03T20:55:01.567 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::602.00000000:head# 2024-08-03T20:55:01.567 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:01.565+0000 7f002e400640 1 -- 172.21.15.32:0/451983607 <== osd.0 v2:172.21.15.32:6816/3066156116 9 ==== osd_op_reply(9 602.00000000 [create] v85'51 uv51 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f002800b040 con 0x5616cccbb090 2024-08-03T20:55:01.567 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:01.565+0000 7f003480ef40 1 -- 172.21.15.32:0/451983607 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:10 12.10 12:0bd6d154:::602.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e85) -- 0x5616ccd8a770 con 0x5616cccbb090 2024-08-03T20:55:01.569 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:01.567+0000 7f002e400640 1 -- 172.21.15.32:0/451983607 <== osd.0 v2:172.21.15.32:6816/3066156116 10 ==== osd_op_reply(10 602.00000000 [setxattr (30)] v85'52 uv52 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f002800b040 con 0x5616cccbb090 2024-08-03T20:55:01.569 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:01.567+0000 7f003480ef40 1 -- 172.21.15.32:0/451983607 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:11 12.10 12:0bd6d154:::602.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e85) -- 0x5616ccd8a770 con 0x5616cccbb090 2024-08-03T20:55:01.570 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:01.569+0000 7f002e400640 1 -- 172.21.15.32:0/451983607 <== osd.0 v2:172.21.15.32:6816/3066156116 11 ==== osd_op_reply(11 602.00000000 [setxattr (62)] v85'53 uv53 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f002800b040 con 0x5616cccbb090 2024-08-03T20:55:01.571 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:01.569+0000 7f003480ef40 1 -- 172.21.15.32:0/451983607 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:12 12.10 12:0bd6d154:::602.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e85) -- 0x5616ccd8a770 con 0x5616cccbb090 2024-08-03T20:55:01.572 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:01.571+0000 7f002e400640 1 -- 172.21.15.32:0/451983607 <== osd.0 v2:172.21.15.32:6816/3066156116 12 ==== osd_op_reply(12 602.00000000 [omap-set-header] v85'54 uv54 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f002800b040 con 0x5616cccbb090 2024-08-03T20:55:01.572 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:01.571+0000 7f003480ef40 1 -- 172.21.15.32:0/451983607 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:13 12.10 12:0cf6ad0f:::10000000000.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e85) -- 0x5616ccd8a770 con 0x5616cccbb090 2024-08-03T20:55:01.573 INFO:teuthology.orchestra.run.smithi032.stdout:Write #-9223372036854775808:00000000:::10000000000.00000000:head# 2024-08-03T20:55:01.574 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:01.572+0000 7f002e400640 1 -- 172.21.15.32:0/451983607 <== osd.0 v2:172.21.15.32:6816/3066156116 13 ==== osd_op_reply(13 10000000000.00000000 [create] v85'55 uv55 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f002800b040 con 0x5616cccbb090 2024-08-03T20:55:01.574 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:01.572+0000 7f003480ef40 1 -- 172.21.15.32:0/451983607 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:14 12.10 12:0cf6ad0f:::10000000000.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e85) -- 0x5616ccd8a770 con 0x5616cccbb090 2024-08-03T20:55:01.575 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:01.574+0000 7f002e400640 1 -- 172.21.15.32:0/451983607 <== osd.0 v2:172.21.15.32:6816/3066156116 14 ==== osd_op_reply(14 10000000000.00000000 [setxattr (30)] v85'56 uv56 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f002800b040 con 0x5616cccbb090 2024-08-03T20:55:01.575 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:01.574+0000 7f003480ef40 1 -- 172.21.15.32:0/451983607 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:15 12.10 12:0cf6ad0f:::10000000000.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e85) -- 0x5616ccd8a770 con 0x5616cccbb090 2024-08-03T20:55:01.577 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:01.575+0000 7f002e400640 1 -- 172.21.15.32:0/451983607 <== osd.0 v2:172.21.15.32:6816/3066156116 15 ==== osd_op_reply(15 10000000000.00000000 [setxattr (62)] v85'57 uv57 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f002800b040 con 0x5616cccbb090 2024-08-03T20:55:01.577 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:01.575+0000 7f003480ef40 1 -- 172.21.15.32:0/451983607 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:16 12.10 12:0cf6ad0f:::10000000000.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e85) -- 0x5616ccd8a770 con 0x5616cccbb090 2024-08-03T20:55:01.578 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:01.577+0000 7f002e400640 1 -- 172.21.15.32:0/451983607 <== osd.0 v2:172.21.15.32:6816/3066156116 16 ==== osd_op_reply(16 10000000000.00000000 [omap-set-header] v85'58 uv58 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f002800b040 con 0x5616cccbb090 2024-08-03T20:55:01.578 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:01.577+0000 7f003480ef40 1 -- 172.21.15.32:0/451983607 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:17 12.10 12:0cf6ad0f:::10000000000.00000000:head [omap-set-vals in=593b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e85) -- 0x5616ccd8b3e0 con 0x5616cccbb090 2024-08-03T20:55:01.580 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:01.578+0000 7f002e400640 1 -- 172.21.15.32:0/451983607 <== osd.0 v2:172.21.15.32:6816/3066156116 17 ==== osd_op_reply(17 10000000000.00000000 [omap-set-vals] v85'59 uv59 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f002800b040 con 0x5616cccbb090 2024-08-03T20:55:01.580 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:01.578+0000 7f003480ef40 1 -- 172.21.15.32:0/451983607 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:18 12.10 12:0d82a743:::600.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e85) -- 0x5616ccd8b3e0 con 0x5616cccbb090 2024-08-03T20:55:01.581 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:01.580+0000 7f002e400640 1 -- 172.21.15.32:0/451983607 <== osd.0 v2:172.21.15.32:6816/3066156116 18 ==== osd_op_reply(18 600.00000000 [create] v85'60 uv35 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f002800b040 con 0x5616cccbb090 2024-08-03T20:55:01.582 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:01.580+0000 7f003480ef40 1 -- 172.21.15.32:0/451983607 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:19 12.10 12:0d82a743:::600.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) -- 0x5616ccd8b3e0 con 0x5616cccbb090 2024-08-03T20:55:01.583 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:01.582+0000 7f002e400640 1 -- 172.21.15.32:0/451983607 <== osd.0 v2:172.21.15.32:6816/3066156116 19 ==== osd_op_reply(19 600.00000000 [delete] v85'61 uv61 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f002800b040 con 0x5616cccbb090 2024-08-03T20:55:01.583 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:01.582+0000 7f003480ef40 1 -- 172.21.15.32:0/451983607 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:20 12.10 12:0d82a743:::600.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e85) -- 0x5616ccd8b3e0 con 0x5616cccbb090 2024-08-03T20:55:01.585 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::600.00000000:head# 2024-08-03T20:55:01.585 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:01.583+0000 7f002e400640 1 -- 172.21.15.32:0/451983607 <== osd.0 v2:172.21.15.32:6816/3066156116 20 ==== osd_op_reply(20 600.00000000 [create] v85'62 uv62 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f002800b040 con 0x5616cccbb090 2024-08-03T20:55:01.585 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:01.584+0000 7f003480ef40 1 -- 172.21.15.32:0/451983607 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:21 12.10 12:0d82a743:::600.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e85) -- 0x5616ccd8c3f0 con 0x5616cccbb090 2024-08-03T20:55:01.586 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:01.585+0000 7f002e400640 1 -- 172.21.15.32:0/451983607 <== osd.0 v2:172.21.15.32:6816/3066156116 21 ==== osd_op_reply(21 600.00000000 [setxattr (30)] v85'63 uv63 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f002800b040 con 0x5616cccbb090 2024-08-03T20:55:01.587 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:01.585+0000 7f003480ef40 1 -- 172.21.15.32:0/451983607 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:22 12.10 12:0d82a743:::600.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e85) -- 0x5616ccd8c3f0 con 0x5616cccbb090 2024-08-03T20:55:01.588 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:01.587+0000 7f002e400640 1 -- 172.21.15.32:0/451983607 <== osd.0 v2:172.21.15.32:6816/3066156116 22 ==== osd_op_reply(22 600.00000000 [setxattr (62)] v85'64 uv64 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f002800b040 con 0x5616cccbb090 2024-08-03T20:55:01.588 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:01.587+0000 7f003480ef40 1 -- 172.21.15.32:0/451983607 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:23 12.10 12:0d82a743:::600.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e85) -- 0x5616ccd8b3e0 con 0x5616cccbb090 2024-08-03T20:55:01.590 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:01.589+0000 7f002e400640 1 -- 172.21.15.32:0/451983607 <== osd.0 v2:172.21.15.32:6816/3066156116 23 ==== osd_op_reply(23 600.00000000 [omap-set-header] v85'65 uv65 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f002800b040 con 0x5616cccbb090 2024-08-03T20:55:01.590 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:01.589+0000 7f003480ef40 1 -- 172.21.15.32:0/451983607 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:24 12.10 12:0d89b25e:::603.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e85) -- 0x5616ccd8b3e0 con 0x5616cccbb090 2024-08-03T20:55:01.592 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:01.590+0000 7f002e400640 1 -- 172.21.15.32:0/451983607 <== osd.0 v2:172.21.15.32:6816/3066156116 24 ==== osd_op_reply(24 603.00000000 [create] v85'66 uv41 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f002800b040 con 0x5616cccbb090 2024-08-03T20:55:01.592 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:01.590+0000 7f003480ef40 1 -- 172.21.15.32:0/451983607 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:25 12.10 12:0d89b25e:::603.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) -- 0x5616ccd8b3e0 con 0x5616cccbb090 2024-08-03T20:55:01.593 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:01.592+0000 7f002e400640 1 -- 172.21.15.32:0/451983607 <== osd.0 v2:172.21.15.32:6816/3066156116 25 ==== osd_op_reply(25 603.00000000 [delete] v85'67 uv67 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f002800b040 con 0x5616cccbb090 2024-08-03T20:55:01.593 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:01.592+0000 7f003480ef40 1 -- 172.21.15.32:0/451983607 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:26 12.10 12:0d89b25e:::603.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e85) -- 0x5616ccd8b3e0 con 0x5616cccbb090 2024-08-03T20:55:01.595 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::603.00000000:head# 2024-08-03T20:55:01.595 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:01.594+0000 7f002e400640 1 -- 172.21.15.32:0/451983607 <== osd.0 v2:172.21.15.32:6816/3066156116 26 ==== osd_op_reply(26 603.00000000 [create] v85'68 uv68 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f002800b040 con 0x5616cccbb090 2024-08-03T20:55:01.595 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:01.594+0000 7f003480ef40 1 -- 172.21.15.32:0/451983607 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:27 12.10 12:0d89b25e:::603.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e85) -- 0x5616ccd8c3f0 con 0x5616cccbb090 2024-08-03T20:55:01.597 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:01.595+0000 7f002e400640 1 -- 172.21.15.32:0/451983607 <== osd.0 v2:172.21.15.32:6816/3066156116 27 ==== osd_op_reply(27 603.00000000 [setxattr (30)] v85'69 uv69 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f002800b040 con 0x5616cccbb090 2024-08-03T20:55:01.597 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:01.595+0000 7f003480ef40 1 -- 172.21.15.32:0/451983607 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:28 12.10 12:0d89b25e:::603.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e85) -- 0x5616ccd8c3f0 con 0x5616cccbb090 2024-08-03T20:55:01.599 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:01.597+0000 7f002e400640 1 -- 172.21.15.32:0/451983607 <== osd.0 v2:172.21.15.32:6816/3066156116 28 ==== osd_op_reply(28 603.00000000 [setxattr (62)] v85'70 uv70 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f002800b040 con 0x5616cccbb090 2024-08-03T20:55:01.599 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:01.597+0000 7f003480ef40 1 -- 172.21.15.32:0/451983607 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:29 12.10 12:0d89b25e:::603.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e85) -- 0x5616ccd8b3e0 con 0x5616cccbb090 2024-08-03T20:55:01.600 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:01.599+0000 7f002e400640 1 -- 172.21.15.32:0/451983607 <== osd.0 v2:172.21.15.32:6816/3066156116 29 ==== osd_op_reply(29 603.00000000 [omap-set-header] v85'71 uv71 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f002800b040 con 0x5616cccbb090 2024-08-03T20:55:01.600 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:01.599+0000 7f003480ef40 1 --2- 172.21.15.32:0/451983607 >> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] conn(0x5616ccd8b8f0 0x5616ccd8dd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:55:01.601 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:01.599+0000 7f003480ef40 1 -- 172.21.15.32:0/451983607 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:30 12.14 12:287034c4:::1.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e85) -- 0x5616ccd8e250 con 0x5616ccd8b8f0 2024-08-03T20:55:01.601 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:01.599+0000 7f002da00640 1 --2- 172.21.15.32:0/451983607 >> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] conn(0x5616ccd8b8f0 0x5616ccd8dd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:55:01.601 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:01.600+0000 7f002da00640 1 --2- 172.21.15.32:0/451983607 >> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] conn(0x5616ccd8b8f0 0x5616ccd8dd10 crc :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:55:01.603 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:01.602+0000 7f002da00640 1 -- 172.21.15.32:0/451983607 <== osd.5 v2:172.21.15.118:6810/2724891770 1 ==== osd_op_reply(30 1.00000000.inode [create] v85'45 uv24 ondisk = -17 ((17) File exists)) ==== 160+0+0 (crc 0 0 0) 0x7f002005cb30 con 0x5616ccd8b8f0 2024-08-03T20:55:01.603 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:01.602+0000 7f003480ef40 1 -- 172.21.15.32:0/451983607 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:31 12.14 12:287034c4:::1.00000000.inode:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) -- 0x5616ccd8e5f0 con 0x5616ccd8b8f0 2024-08-03T20:55:01.605 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:01.604+0000 7f002da00640 1 -- 172.21.15.32:0/451983607 <== osd.5 v2:172.21.15.118:6810/2724891770 2 ==== osd_op_reply(31 1.00000000.inode [delete] v85'46 uv46 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f002005cb30 con 0x5616ccd8b8f0 2024-08-03T20:55:01.605 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:01.604+0000 7f003480ef40 1 -- 172.21.15.32:0/451983607 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:32 12.14 12:287034c4:::1.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e85) -- 0x5616ccd8e990 con 0x5616ccd8b8f0 2024-08-03T20:55:01.607 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::1.00000000.inode:head# 2024-08-03T20:55:01.607 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:01.605+0000 7f002da00640 1 -- 172.21.15.32:0/451983607 <== osd.5 v2:172.21.15.118:6810/2724891770 3 ==== osd_op_reply(32 1.00000000.inode [create] v85'47 uv47 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f002005cb30 con 0x5616ccd8b8f0 2024-08-03T20:55:01.607 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:01.605+0000 7f003480ef40 1 -- 172.21.15.32:0/451983607 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:33 12.14 12:287034c4:::1.00000000.inode:head [write 0~582 in=582b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e85) -- 0x5616ccd8ed30 con 0x5616ccd8b8f0 2024-08-03T20:55:01.608 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:01.607+0000 7f002da00640 1 -- 172.21.15.32:0/451983607 <== osd.5 v2:172.21.15.118:6810/2724891770 4 ==== osd_op_reply(33 1.00000000.inode [write 0~582] v85'48 uv48 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f002005cb30 con 0x5616ccd8b8f0 2024-08-03T20:55:01.608 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:01.607+0000 7f003480ef40 1 -- 172.21.15.32:0/451983607 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:34 12.14 12:287034c4:::1.00000000.inode:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e85) -- 0x5616ccd8f140 con 0x5616ccd8b8f0 2024-08-03T20:55:01.610 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:01.609+0000 7f002da00640 1 -- 172.21.15.32:0/451983607 <== osd.5 v2:172.21.15.118:6810/2724891770 5 ==== osd_op_reply(34 1.00000000.inode [omap-set-header] v85'49 uv49 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f002005cb30 con 0x5616ccd8b8f0 2024-08-03T20:55:01.610 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:01.609+0000 7f003480ef40 1 -- 172.21.15.32:0/451983607 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:35 12.14 12:292cf221:::200.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e85) -- 0x5616ccd8f4e0 con 0x5616ccd8b8f0 2024-08-03T20:55:01.612 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:01.610+0000 7f002da00640 1 -- 172.21.15.32:0/451983607 <== osd.5 v2:172.21.15.118:6810/2724891770 6 ==== osd_op_reply(35 200.00000000 [create] v85'50 uv43 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f002005cb30 con 0x5616ccd8b8f0 2024-08-03T20:55:01.612 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:01.610+0000 7f003480ef40 1 -- 172.21.15.32:0/451983607 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:36 12.14 12:292cf221:::200.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) -- 0x5616ccd8f8d0 con 0x5616ccd8b8f0 2024-08-03T20:55:01.613 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:01.612+0000 7f002da00640 1 -- 172.21.15.32:0/451983607 <== osd.5 v2:172.21.15.118:6810/2724891770 7 ==== osd_op_reply(36 200.00000000 [delete] v85'51 uv51 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f002005cb30 con 0x5616ccd8b8f0 2024-08-03T20:55:01.613 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:01.612+0000 7f003480ef40 1 -- 172.21.15.32:0/451983607 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:37 12.14 12:292cf221:::200.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e85) -- 0x5616ccd8fcc0 con 0x5616ccd8b8f0 2024-08-03T20:55:01.615 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::200.00000000:head# 2024-08-03T20:55:01.615 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:01.613+0000 7f002da00640 1 -- 172.21.15.32:0/451983607 <== osd.5 v2:172.21.15.118:6810/2724891770 8 ==== osd_op_reply(37 200.00000000 [create] v85'52 uv52 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f002005cb30 con 0x5616ccd8b8f0 2024-08-03T20:55:01.615 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:01.613+0000 7f003480ef40 1 -- 172.21.15.32:0/451983607 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:38 12.14 12:292cf221:::200.00000000:head [write 0~90 in=90b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e85) -- 0x5616ccd8fcc0 con 0x5616ccd8b8f0 2024-08-03T20:55:01.617 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:01.616+0000 7f002da00640 1 -- 172.21.15.32:0/451983607 <== osd.5 v2:172.21.15.118:6810/2724891770 9 ==== osd_op_reply(38 200.00000000 [write 0~90] v85'53 uv53 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f002005cb30 con 0x5616ccd8b8f0 2024-08-03T20:55:01.618 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:01.616+0000 7f003480ef40 1 -- 172.21.15.32:0/451983607 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:39 12.14 12:292cf221:::200.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e85) -- 0x5616ccd8fc70 con 0x5616ccd8b8f0 2024-08-03T20:55:01.619 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:01.618+0000 7f002da00640 1 -- 172.21.15.32:0/451983607 <== osd.5 v2:172.21.15.118:6810/2724891770 10 ==== osd_op_reply(39 200.00000000 [omap-set-header] v85'54 uv54 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f002005cb30 con 0x5616ccd8b8f0 2024-08-03T20:55:01.619 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:01.618+0000 7f003480ef40 1 -- 172.21.15.32:0/451983607 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:40 12.14 12:2e2fa760:::200.00000001:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e85) -- 0x5616ccd8fc70 con 0x5616ccd8b8f0 2024-08-03T20:55:01.621 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:01.619+0000 7f002da00640 1 -- 172.21.15.32:0/451983607 <== osd.5 v2:172.21.15.118:6810/2724891770 11 ==== osd_op_reply(40 200.00000001 [create] v85'55 uv44 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f002005cb30 con 0x5616ccd8b8f0 2024-08-03T20:55:01.621 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:01.619+0000 7f003480ef40 1 -- 172.21.15.32:0/451983607 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:41 12.14 12:2e2fa760:::200.00000001:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) -- 0x5616ccd8fc70 con 0x5616ccd8b8f0 2024-08-03T20:55:01.623 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:01.621+0000 7f002da00640 1 -- 172.21.15.32:0/451983607 <== osd.5 v2:172.21.15.118:6810/2724891770 12 ==== osd_op_reply(41 200.00000001 [delete] v85'56 uv56 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f002005cb30 con 0x5616ccd8b8f0 2024-08-03T20:55:01.623 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:01.621+0000 7f003480ef40 1 -- 172.21.15.32:0/451983607 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:42 12.14 12:2e2fa760:::200.00000001:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e85) -- 0x5616ccd8fc70 con 0x5616ccd8b8f0 2024-08-03T20:55:01.624 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::200.00000001:head# 2024-08-03T20:55:01.625 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:01.623+0000 7f002da00640 1 -- 172.21.15.32:0/451983607 <== osd.5 v2:172.21.15.118:6810/2724891770 13 ==== osd_op_reply(42 200.00000001 [create] v85'57 uv57 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f002005cb30 con 0x5616ccd8b8f0 2024-08-03T20:55:01.625 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:01.623+0000 7f003480ef40 1 -- 172.21.15.32:0/451983607 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:43 12.14 12:2e2fa760:::200.00000001:head [write 0~18118 in=18118b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e85) -- 0x5616ccd94360 con 0x5616ccd8b8f0 2024-08-03T20:55:01.626 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:01.625+0000 7f002da00640 1 -- 172.21.15.32:0/451983607 <== osd.5 v2:172.21.15.118:6810/2724891770 14 ==== osd_op_reply(43 200.00000001 [write 0~18118] v85'58 uv58 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f002005cb30 con 0x5616ccd8b8f0 2024-08-03T20:55:01.626 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:01.625+0000 7f003480ef40 1 -- 172.21.15.32:0/451983607 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:44 12.14 12:2e2fa760:::200.00000001:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e85) -- 0x5616ccd94360 con 0x5616ccd8b8f0 2024-08-03T20:55:01.628 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:01.627+0000 7f002da00640 1 -- 172.21.15.32:0/451983607 <== osd.5 v2:172.21.15.118:6810/2724891770 15 ==== osd_op_reply(44 200.00000001 [omap-set-header] v85'59 uv59 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f002005cb30 con 0x5616ccd8b8f0 2024-08-03T20:55:01.628 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:01.627+0000 7f003480ef40 1 -- 172.21.15.32:0/451983607 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:45 12.14 12:2f55791f:::606.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e85) -- 0x5616ccd8fc70 con 0x5616ccd8b8f0 2024-08-03T20:55:01.629 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:01.628+0000 7f002da00640 1 -- 172.21.15.32:0/451983607 <== osd.5 v2:172.21.15.118:6810/2724891770 16 ==== osd_op_reply(45 606.00000000 [create] v85'60 uv40 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f002005cb30 con 0x5616ccd8b8f0 2024-08-03T20:55:01.630 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:01.628+0000 7f003480ef40 1 -- 172.21.15.32:0/451983607 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:46 12.14 12:2f55791f:::606.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) -- 0x5616ccd8fc70 con 0x5616ccd8b8f0 2024-08-03T20:55:01.631 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:01.630+0000 7f002da00640 1 -- 172.21.15.32:0/451983607 <== osd.5 v2:172.21.15.118:6810/2724891770 17 ==== osd_op_reply(46 606.00000000 [delete] v85'61 uv61 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f002005cb30 con 0x5616ccd8b8f0 2024-08-03T20:55:01.631 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:01.630+0000 7f003480ef40 1 -- 172.21.15.32:0/451983607 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:47 12.14 12:2f55791f:::606.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e85) -- 0x5616ccd8fc70 con 0x5616ccd8b8f0 2024-08-03T20:55:01.633 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::606.00000000:head# 2024-08-03T20:55:01.633 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:01.632+0000 7f002da00640 1 -- 172.21.15.32:0/451983607 <== osd.5 v2:172.21.15.118:6810/2724891770 18 ==== osd_op_reply(47 606.00000000 [create] v85'62 uv62 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f002005cb30 con 0x5616ccd8b8f0 2024-08-03T20:55:01.633 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:01.632+0000 7f003480ef40 1 -- 172.21.15.32:0/451983607 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:48 12.14 12:2f55791f:::606.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e85) -- 0x5616ccd8fc70 con 0x5616ccd8b8f0 2024-08-03T20:55:01.635 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:01.633+0000 7f002da00640 1 -- 172.21.15.32:0/451983607 <== osd.5 v2:172.21.15.118:6810/2724891770 19 ==== osd_op_reply(48 606.00000000 [setxattr (30)] v85'63 uv63 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f002005cb30 con 0x5616ccd8b8f0 2024-08-03T20:55:01.635 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:01.633+0000 7f003480ef40 1 -- 172.21.15.32:0/451983607 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:49 12.14 12:2f55791f:::606.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e85) -- 0x5616ccd8fc70 con 0x5616ccd8b8f0 2024-08-03T20:55:01.636 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:01.635+0000 7f002da00640 1 -- 172.21.15.32:0/451983607 <== osd.5 v2:172.21.15.118:6810/2724891770 20 ==== osd_op_reply(49 606.00000000 [setxattr (62)] v85'64 uv64 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f002005cb30 con 0x5616ccd8b8f0 2024-08-03T20:55:01.636 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:01.635+0000 7f003480ef40 1 -- 172.21.15.32:0/451983607 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:50 12.14 12:2f55791f:::606.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e85) -- 0x5616ccd8fcb0 con 0x5616ccd8b8f0 2024-08-03T20:55:01.638 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:01.637+0000 7f002da00640 1 -- 172.21.15.32:0/451983607 <== osd.5 v2:172.21.15.118:6810/2724891770 21 ==== osd_op_reply(50 606.00000000 [omap-set-header] v85'65 uv65 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f002005cb30 con 0x5616ccd8b8f0 2024-08-03T20:55:01.638 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:01.637+0000 7f003480ef40 1 -- 172.21.15.32:0/451983607 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:51 12.1c 12:3ed09add:::607.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e85) -- 0x5616ccd8fcb0 con 0x5616cccbb090 2024-08-03T20:55:01.639 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:01.638+0000 7f002e400640 1 -- 172.21.15.32:0/451983607 <== osd.0 v2:172.21.15.32:6816/3066156116 30 ==== osd_op_reply(51 607.00000000 [create] v85'9 uv8 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f002800b040 con 0x5616cccbb090 2024-08-03T20:55:01.640 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:01.638+0000 7f003480ef40 1 -- 172.21.15.32:0/451983607 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:52 12.1c 12:3ed09add:::607.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) -- 0x5616ccd8fc70 con 0x5616cccbb090 2024-08-03T20:55:01.641 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:01.639+0000 7f002e400640 1 -- 172.21.15.32:0/451983607 <== osd.0 v2:172.21.15.32:6816/3066156116 31 ==== osd_op_reply(52 607.00000000 [delete] v85'10 uv10 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f002800b040 con 0x5616cccbb090 2024-08-03T20:55:01.641 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:01.640+0000 7f003480ef40 1 -- 172.21.15.32:0/451983607 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:53 12.1c 12:3ed09add:::607.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e85) -- 0x5616ccd8fc70 con 0x5616cccbb090 2024-08-03T20:55:01.642 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::607.00000000:head# 2024-08-03T20:55:01.642 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:01.641+0000 7f002e400640 1 -- 172.21.15.32:0/451983607 <== osd.0 v2:172.21.15.32:6816/3066156116 32 ==== osd_op_reply(53 607.00000000 [create] v85'11 uv11 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f002800b040 con 0x5616cccbb090 2024-08-03T20:55:01.642 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:01.641+0000 7f003480ef40 1 -- 172.21.15.32:0/451983607 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:54 12.1c 12:3ed09add:::607.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e85) -- 0x5616ccd8fc70 con 0x5616cccbb090 2024-08-03T20:55:01.643 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:01.642+0000 7f002e400640 1 -- 172.21.15.32:0/451983607 <== osd.0 v2:172.21.15.32:6816/3066156116 33 ==== osd_op_reply(54 607.00000000 [setxattr (30)] v85'12 uv12 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f002800b040 con 0x5616cccbb090 2024-08-03T20:55:01.644 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:01.642+0000 7f003480ef40 1 -- 172.21.15.32:0/451983607 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:55 12.1c 12:3ed09add:::607.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e85) -- 0x5616ccd8fc70 con 0x5616cccbb090 2024-08-03T20:55:01.645 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:01.643+0000 7f002e400640 1 -- 172.21.15.32:0/451983607 <== osd.0 v2:172.21.15.32:6816/3066156116 34 ==== osd_op_reply(55 607.00000000 [setxattr (62)] v85'13 uv13 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f002800b040 con 0x5616cccbb090 2024-08-03T20:55:01.645 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:01.643+0000 7f003480ef40 1 -- 172.21.15.32:0/451983607 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:56 12.1c 12:3ed09add:::607.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e85) -- 0x5616ccd8fc70 con 0x5616cccbb090 2024-08-03T20:55:01.646 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:01.645+0000 7f002e400640 1 -- 172.21.15.32:0/451983607 <== osd.0 v2:172.21.15.32:6816/3066156116 35 ==== osd_op_reply(56 607.00000000 [omap-set-header] v85'14 uv14 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f002800b040 con 0x5616cccbb090 2024-08-03T20:55:01.646 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:01.645+0000 7f003480ef40 1 --2- 172.21.15.32:0/451983607 >> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] conn(0x5616ccd90620 0x5616ccd90a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:55:01.646 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:01.645+0000 7f003480ef40 1 -- 172.21.15.32:0/451983607 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:57 12.6 12:654134d2:::mds0_openfiles.0:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e85) -- 0x5616ccd931d0 con 0x5616ccd90620 2024-08-03T20:55:01.647 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:01.645+0000 7f002d000640 1 --2- 172.21.15.32:0/451983607 >> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] conn(0x5616ccd90620 0x5616ccd90a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:55:01.647 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:01.646+0000 7f002d000640 1 --2- 172.21.15.32:0/451983607 >> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] conn(0x5616ccd90620 0x5616ccd90a30 crc :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:55:01.648 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:01.647+0000 7f002d000640 1 -- 172.21.15.32:0/451983607 <== osd.1 v2:172.21.15.32:6824/1551241759 1 ==== osd_op_reply(57 mds0_openfiles.0 [create] v85'12 uv11 ondisk = -17 ((17) File exists)) ==== 160+0+0 (crc 0 0 0) 0x7f001c00f830 con 0x5616ccd90620 2024-08-03T20:55:01.648 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:01.647+0000 7f003480ef40 1 -- 172.21.15.32:0/451983607 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:58 12.6 12:654134d2:::mds0_openfiles.0:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) -- 0x5616ccd93570 con 0x5616ccd90620 2024-08-03T20:55:01.650 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:01.648+0000 7f002d000640 1 -- 172.21.15.32:0/451983607 <== osd.1 v2:172.21.15.32:6824/1551241759 2 ==== osd_op_reply(58 mds0_openfiles.0 [delete] v85'13 uv13 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f001c00f830 con 0x5616ccd90620 2024-08-03T20:55:01.650 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:01.648+0000 7f003480ef40 1 -- 172.21.15.32:0/451983607 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:59 12.6 12:654134d2:::mds0_openfiles.0:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e85) -- 0x5616ccd93910 con 0x5616ccd90620 2024-08-03T20:55:01.654 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds0_openfiles.0:head# 2024-08-03T20:55:01.655 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:01.650+0000 7f002d000640 1 -- 172.21.15.32:0/451983607 <== osd.1 v2:172.21.15.32:6824/1551241759 3 ==== osd_op_reply(59 mds0_openfiles.0 [create] v85'14 uv14 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f001c00f830 con 0x5616ccd90620 2024-08-03T20:55:01.655 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:01.650+0000 7f003480ef40 1 -- 172.21.15.32:0/451983607 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:60 12.6 12:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e85) -- 0x5616cccde340 con 0x5616ccd90620 2024-08-03T20:55:01.655 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:01.651+0000 7f002d000640 1 -- 172.21.15.32:0/451983607 <== osd.1 v2:172.21.15.32:6824/1551241759 4 ==== osd_op_reply(60 mds0_openfiles.0 [omap-set-header] v85'15 uv15 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f001c00f830 con 0x5616ccd90620 2024-08-03T20:55:01.656 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:01.651+0000 7f003480ef40 1 -- 172.21.15.32:0/451983607 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:61 12.e 12:717a0223:::608.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e85) -- 0x5616ccd94240 con 0x5616cccbb090 2024-08-03T20:55:01.656 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:01.653+0000 7f002e400640 1 -- 172.21.15.32:0/451983607 <== osd.0 v2:172.21.15.32:6816/3066156116 36 ==== osd_op_reply(61 608.00000000 [create] v85'9 uv8 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f002800b040 con 0x5616cccbb090 2024-08-03T20:55:01.656 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:01.653+0000 7f003480ef40 1 -- 172.21.15.32:0/451983607 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:62 12.e 12:717a0223:::608.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) -- 0x5616ccd94240 con 0x5616cccbb090 2024-08-03T20:55:01.656 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:01.655+0000 7f002e400640 1 -- 172.21.15.32:0/451983607 <== osd.0 v2:172.21.15.32:6816/3066156116 37 ==== osd_op_reply(62 608.00000000 [delete] v85'10 uv10 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f002800b040 con 0x5616cccbb090 2024-08-03T20:55:01.656 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:01.655+0000 7f003480ef40 1 -- 172.21.15.32:0/451983607 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:63 12.e 12:717a0223:::608.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e85) -- 0x5616ccd94240 con 0x5616cccbb090 2024-08-03T20:55:01.657 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::608.00000000:head# 2024-08-03T20:55:01.658 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:01.656+0000 7f002e400640 1 -- 172.21.15.32:0/451983607 <== osd.0 v2:172.21.15.32:6816/3066156116 38 ==== osd_op_reply(63 608.00000000 [create] v85'11 uv11 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f002800b040 con 0x5616cccbb090 2024-08-03T20:55:01.658 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:01.656+0000 7f003480ef40 1 -- 172.21.15.32:0/451983607 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:64 12.e 12:717a0223:::608.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e85) -- 0x5616ccd95250 con 0x5616cccbb090 2024-08-03T20:55:01.659 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:01.658+0000 7f002e400640 1 -- 172.21.15.32:0/451983607 <== osd.0 v2:172.21.15.32:6816/3066156116 39 ==== osd_op_reply(64 608.00000000 [setxattr (30)] v85'12 uv12 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f002800b040 con 0x5616cccbb090 2024-08-03T20:55:01.659 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:01.658+0000 7f003480ef40 1 -- 172.21.15.32:0/451983607 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:65 12.e 12:717a0223:::608.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e85) -- 0x5616ccd95250 con 0x5616cccbb090 2024-08-03T20:55:01.660 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:01.659+0000 7f002e400640 1 -- 172.21.15.32:0/451983607 <== osd.0 v2:172.21.15.32:6816/3066156116 40 ==== osd_op_reply(65 608.00000000 [setxattr (62)] v85'13 uv13 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f002800b040 con 0x5616cccbb090 2024-08-03T20:55:01.661 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:01.659+0000 7f003480ef40 1 -- 172.21.15.32:0/451983607 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:66 12.e 12:717a0223:::608.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e85) -- 0x5616ccd94240 con 0x5616cccbb090 2024-08-03T20:55:01.663 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:01.661+0000 7f002e400640 1 -- 172.21.15.32:0/451983607 <== osd.0 v2:172.21.15.32:6816/3066156116 41 ==== osd_op_reply(66 608.00000000 [omap-set-header] v85'14 uv14 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f002800b040 con 0x5616cccbb090 2024-08-03T20:55:01.663 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:01.661+0000 7f003480ef40 1 --2- 172.21.15.32:0/451983607 >> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] conn(0x5616ccd94910 0x5616ccd94d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:55:01.663 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:01.661+0000 7f003480ef40 1 -- 172.21.15.32:0/451983607 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:67 12.15 12:a93a17c2:::604.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e85) -- 0x5616ccd976a0 con 0x5616ccd94910 2024-08-03T20:55:01.663 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:01.661+0000 7f002e400640 1 --2- 172.21.15.32:0/451983607 >> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] conn(0x5616ccd94910 0x5616ccd94d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:55:01.663 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:01.662+0000 7f002e400640 1 --2- 172.21.15.32:0/451983607 >> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] conn(0x5616ccd94910 0x5616ccd94d40 crc :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:55:01.665 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:01.663+0000 7f002e400640 1 -- 172.21.15.32:0/451983607 <== osd.6 v2:172.21.15.118:6825/801947386 1 ==== osd_op_reply(67 604.00000000 [create] v85'15 uv9 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f002800b040 con 0x5616ccd94910 2024-08-03T20:55:01.665 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:01.664+0000 7f003480ef40 1 -- 172.21.15.32:0/451983607 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:68 12.15 12:a93a17c2:::604.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) -- 0x5616ccd976a0 con 0x5616ccd94910 2024-08-03T20:55:01.667 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:01.665+0000 7f002e400640 1 -- 172.21.15.32:0/451983607 <== osd.6 v2:172.21.15.118:6825/801947386 2 ==== osd_op_reply(68 604.00000000 [delete] v85'16 uv16 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f002800b040 con 0x5616ccd94910 2024-08-03T20:55:01.667 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:01.665+0000 7f003480ef40 1 -- 172.21.15.32:0/451983607 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:69 12.15 12:a93a17c2:::604.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e85) -- 0x5616ccd976a0 con 0x5616ccd94910 2024-08-03T20:55:01.668 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::604.00000000:head# 2024-08-03T20:55:01.668 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:01.667+0000 7f002e400640 1 -- 172.21.15.32:0/451983607 <== osd.6 v2:172.21.15.118:6825/801947386 3 ==== osd_op_reply(69 604.00000000 [create] v85'17 uv17 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f002800b040 con 0x5616ccd94910 2024-08-03T20:55:01.668 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:01.667+0000 7f003480ef40 1 -- 172.21.15.32:0/451983607 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:70 12.15 12:a93a17c2:::604.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e85) -- 0x5616ccd986f0 con 0x5616ccd94910 2024-08-03T20:55:01.670 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:01.668+0000 7f002e400640 1 -- 172.21.15.32:0/451983607 <== osd.6 v2:172.21.15.118:6825/801947386 4 ==== osd_op_reply(70 604.00000000 [setxattr (30)] v85'18 uv18 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f002800b040 con 0x5616ccd94910 2024-08-03T20:55:01.670 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:01.668+0000 7f003480ef40 1 -- 172.21.15.32:0/451983607 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:71 12.15 12:a93a17c2:::604.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e85) -- 0x5616ccd986b0 con 0x5616ccd94910 2024-08-03T20:55:01.671 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:01.669+0000 7f002e400640 1 -- 172.21.15.32:0/451983607 <== osd.6 v2:172.21.15.118:6825/801947386 5 ==== osd_op_reply(71 604.00000000 [setxattr (62)] v85'19 uv19 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f002800b040 con 0x5616ccd94910 2024-08-03T20:55:01.671 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:01.669+0000 7f003480ef40 1 -- 172.21.15.32:0/451983607 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:72 12.15 12:a93a17c2:::604.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e85) -- 0x5616ccd976a0 con 0x5616ccd94910 2024-08-03T20:55:01.672 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:01.671+0000 7f002e400640 1 -- 172.21.15.32:0/451983607 <== osd.6 v2:172.21.15.118:6825/801947386 6 ==== osd_op_reply(72 604.00000000 [omap-set-header] v85'20 uv20 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f002800b040 con 0x5616ccd94910 2024-08-03T20:55:01.673 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:01.671+0000 7f003480ef40 1 -- 172.21.15.32:0/451983607 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:73 12.15 12:aa448500:::500.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e85) -- 0x5616ccd976a0 con 0x5616ccd94910 2024-08-03T20:55:01.674 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:01.672+0000 7f002e400640 1 -- 172.21.15.32:0/451983607 <== osd.6 v2:172.21.15.118:6825/801947386 7 ==== osd_op_reply(73 500.00000000 [create] v85'21 uv14 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f002800b040 con 0x5616ccd94910 2024-08-03T20:55:01.674 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:01.672+0000 7f003480ef40 1 -- 172.21.15.32:0/451983607 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:74 12.15 12:aa448500:::500.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) -- 0x5616ccd976a0 con 0x5616ccd94910 2024-08-03T20:55:01.675 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:01.674+0000 7f002e400640 1 -- 172.21.15.32:0/451983607 <== osd.6 v2:172.21.15.118:6825/801947386 8 ==== osd_op_reply(74 500.00000000 [delete] v85'22 uv22 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f002800b040 con 0x5616ccd94910 2024-08-03T20:55:01.675 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:01.674+0000 7f003480ef40 1 -- 172.21.15.32:0/451983607 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:75 12.15 12:aa448500:::500.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e85) -- 0x5616ccd976a0 con 0x5616ccd94910 2024-08-03T20:55:01.677 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::500.00000000:head# 2024-08-03T20:55:01.677 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:01.675+0000 7f002e400640 1 -- 172.21.15.32:0/451983607 <== osd.6 v2:172.21.15.118:6825/801947386 9 ==== osd_op_reply(75 500.00000000 [create] v85'23 uv23 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f002800b040 con 0x5616ccd94910 2024-08-03T20:55:01.677 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:01.675+0000 7f003480ef40 1 -- 172.21.15.32:0/451983607 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:76 12.15 12:aa448500:::500.00000000:head [write 0~90 in=90b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e85) -- 0x5616ccd976a0 con 0x5616ccd94910 2024-08-03T20:55:01.678 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:01.677+0000 7f002e400640 1 -- 172.21.15.32:0/451983607 <== osd.6 v2:172.21.15.118:6825/801947386 10 ==== osd_op_reply(76 500.00000000 [write 0~90] v85'24 uv24 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f002800b040 con 0x5616ccd94910 2024-08-03T20:55:01.678 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:01.677+0000 7f003480ef40 1 -- 172.21.15.32:0/451983607 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:77 12.15 12:aa448500:::500.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e85) -- 0x5616ccd976a0 con 0x5616ccd94910 2024-08-03T20:55:01.680 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:01.678+0000 7f002e400640 1 -- 172.21.15.32:0/451983607 <== osd.6 v2:172.21.15.118:6825/801947386 11 ==== osd_op_reply(77 500.00000000 [omap-set-header] v85'25 uv25 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f002800b040 con 0x5616ccd94910 2024-08-03T20:55:01.680 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:01.678+0000 7f003480ef40 1 --2- 172.21.15.32:0/451983607 >> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] conn(0x5616ccd97ff0 0x5616ccd98400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:55:01.680 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:01.678+0000 7f003480ef40 1 -- 172.21.15.32:0/451983607 --> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] -- osd_op(unknown.0.0:78 12.d 12:b50e409b:::mds_snaptable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e85) -- 0x5616ccd9ab00 con 0x5616ccd97ff0 2024-08-03T20:55:01.680 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:01.679+0000 7f002da00640 1 --2- 172.21.15.32:0/451983607 >> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] conn(0x5616ccd97ff0 0x5616ccd98400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:55:01.681 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:01.679+0000 7f002da00640 1 --2- 172.21.15.32:0/451983607 >> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] conn(0x5616ccd97ff0 0x5616ccd98400 crc :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:55:01.682 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:01.681+0000 7f002da00640 1 -- 172.21.15.32:0/451983607 <== osd.7 v2:172.21.15.118:6818/653000048 1 ==== osd_op_reply(78 mds_snaptable [create] v85'7 uv6 ondisk = -17 ((17) File exists)) ==== 157+0+0 (crc 0 0 0) 0x7f002005cb30 con 0x5616ccd97ff0 2024-08-03T20:55:01.682 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:01.681+0000 7f003480ef40 1 -- 172.21.15.32:0/451983607 --> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] -- osd_op(unknown.0.0:79 12.d 12:b50e409b:::mds_snaptable:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) -- 0x5616ccd9ab00 con 0x5616ccd97ff0 2024-08-03T20:55:01.684 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:01.682+0000 7f002da00640 1 -- 172.21.15.32:0/451983607 <== osd.7 v2:172.21.15.118:6818/653000048 2 ==== osd_op_reply(79 mds_snaptable [delete] v85'8 uv8 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f002005cb30 con 0x5616ccd97ff0 2024-08-03T20:55:01.684 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:01.682+0000 7f003480ef40 1 -- 172.21.15.32:0/451983607 --> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] -- osd_op(unknown.0.0:80 12.d 12:b50e409b:::mds_snaptable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e85) -- 0x5616ccd9ab80 con 0x5616ccd97ff0 2024-08-03T20:55:01.685 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds_snaptable:head# 2024-08-03T20:55:01.686 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:01.684+0000 7f002da00640 1 -- 172.21.15.32:0/451983607 <== osd.7 v2:172.21.15.118:6818/653000048 3 ==== osd_op_reply(80 mds_snaptable [create] v85'9 uv9 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f002005cb30 con 0x5616ccd97ff0 2024-08-03T20:55:01.686 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:01.684+0000 7f003480ef40 1 -- 172.21.15.32:0/451983607 --> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] -- osd_op(unknown.0.0:81 12.d 12:b50e409b:::mds_snaptable:head [write 0~70 in=70b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e85) -- 0x5616ccd9afa0 con 0x5616ccd97ff0 2024-08-03T20:55:01.687 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:01.686+0000 7f002da00640 1 -- 172.21.15.32:0/451983607 <== osd.7 v2:172.21.15.118:6818/653000048 4 ==== osd_op_reply(81 mds_snaptable [write 0~70] v85'10 uv10 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f002005cb30 con 0x5616ccd97ff0 2024-08-03T20:55:01.687 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:01.686+0000 7f003480ef40 1 -- 172.21.15.32:0/451983607 --> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] -- osd_op(unknown.0.0:82 12.d 12:b50e409b:::mds_snaptable:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e85) -- 0x5616ccd9afa0 con 0x5616ccd97ff0 2024-08-03T20:55:01.689 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:01.687+0000 7f002da00640 1 -- 172.21.15.32:0/451983607 <== osd.7 v2:172.21.15.118:6818/653000048 5 ==== osd_op_reply(82 mds_snaptable [omap-set-header] v85'11 uv11 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f002005cb30 con 0x5616ccd97ff0 2024-08-03T20:55:01.689 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:01.687+0000 7f003480ef40 1 -- 172.21.15.32:0/451983607 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:83 12.1d 12:b871830b:::605.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e85) -- 0x5616ccd9afa0 con 0x5616cccbb090 2024-08-03T20:55:01.690 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:01.689+0000 7f002e400640 1 -- 172.21.15.32:0/451983607 <== osd.0 v2:172.21.15.32:6816/3066156116 42 ==== osd_op_reply(83 605.00000000 [create] v85'9 uv8 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f002800b040 con 0x5616cccbb090 2024-08-03T20:55:01.690 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:01.689+0000 7f003480ef40 1 -- 172.21.15.32:0/451983607 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:84 12.1d 12:b871830b:::605.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) -- 0x5616ccd9afa0 con 0x5616cccbb090 2024-08-03T20:55:01.691 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:01.690+0000 7f002e400640 1 -- 172.21.15.32:0/451983607 <== osd.0 v2:172.21.15.32:6816/3066156116 43 ==== osd_op_reply(84 605.00000000 [delete] v85'10 uv10 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f002800b040 con 0x5616cccbb090 2024-08-03T20:55:01.692 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:01.690+0000 7f003480ef40 1 -- 172.21.15.32:0/451983607 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:85 12.1d 12:b871830b:::605.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e85) -- 0x5616ccd9afa0 con 0x5616cccbb090 2024-08-03T20:55:01.693 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::605.00000000:head# 2024-08-03T20:55:01.693 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:01.692+0000 7f002e400640 1 -- 172.21.15.32:0/451983607 <== osd.0 v2:172.21.15.32:6816/3066156116 44 ==== osd_op_reply(85 605.00000000 [create] v85'11 uv11 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f002800b040 con 0x5616cccbb090 2024-08-03T20:55:01.693 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:01.692+0000 7f003480ef40 1 -- 172.21.15.32:0/451983607 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:86 12.1d 12:b871830b:::605.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e85) -- 0x5616ccd9bfb0 con 0x5616cccbb090 2024-08-03T20:55:01.694 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:01.693+0000 7f002e400640 1 -- 172.21.15.32:0/451983607 <== osd.0 v2:172.21.15.32:6816/3066156116 45 ==== osd_op_reply(86 605.00000000 [setxattr (30)] v85'12 uv12 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f002800b040 con 0x5616cccbb090 2024-08-03T20:55:01.694 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:01.693+0000 7f003480ef40 1 -- 172.21.15.32:0/451983607 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:87 12.1d 12:b871830b:::605.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e85) -- 0x5616ccd9bfb0 con 0x5616cccbb090 2024-08-03T20:55:01.696 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:01.694+0000 7f002e400640 1 -- 172.21.15.32:0/451983607 <== osd.0 v2:172.21.15.32:6816/3066156116 46 ==== osd_op_reply(87 605.00000000 [setxattr (62)] v85'13 uv13 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f002800b040 con 0x5616cccbb090 2024-08-03T20:55:01.696 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:01.694+0000 7f003480ef40 1 -- 172.21.15.32:0/451983607 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:88 12.1d 12:b871830b:::605.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e85) -- 0x5616ccd9afa0 con 0x5616cccbb090 2024-08-03T20:55:01.697 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:01.695+0000 7f002e400640 1 -- 172.21.15.32:0/451983607 <== osd.0 v2:172.21.15.32:6816/3066156116 47 ==== osd_op_reply(88 605.00000000 [omap-set-header] v85'14 uv14 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f002800b040 con 0x5616cccbb090 2024-08-03T20:55:01.697 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:01.696+0000 7f003480ef40 1 -- 172.21.15.32:0/451983607 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:89 12.13 12:c91d4a1d:::mds0_inotable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e85) -- 0x5616ccd9afa0 con 0x5616ccd90620 2024-08-03T20:55:01.698 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:01.697+0000 7f002d000640 1 -- 172.21.15.32:0/451983607 <== osd.1 v2:172.21.15.32:6824/1551241759 5 ==== osd_op_reply(89 mds0_inotable [create] v85'14 uv8 ondisk = -17 ((17) File exists)) ==== 157+0+0 (crc 0 0 0) 0x7f001c00f830 con 0x5616ccd90620 2024-08-03T20:55:01.698 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:01.697+0000 7f003480ef40 1 -- 172.21.15.32:0/451983607 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:90 12.13 12:c91d4a1d:::mds0_inotable:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) -- 0x5616ccd9b340 con 0x5616ccd90620 2024-08-03T20:55:01.699 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:01.698+0000 7f002d000640 1 -- 172.21.15.32:0/451983607 <== osd.1 v2:172.21.15.32:6824/1551241759 6 ==== osd_op_reply(90 mds0_inotable [delete] v85'15 uv15 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f001c00f830 con 0x5616ccd90620 2024-08-03T20:55:01.699 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:01.698+0000 7f003480ef40 1 -- 172.21.15.32:0/451983607 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:91 12.13 12:c91d4a1d:::mds0_inotable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e85) -- 0x5616ccd9b6e0 con 0x5616ccd90620 2024-08-03T20:55:01.701 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds0_inotable:head# 2024-08-03T20:55:01.701 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:01.699+0000 7f002d000640 1 -- 172.21.15.32:0/451983607 <== osd.1 v2:172.21.15.32:6824/1551241759 7 ==== osd_op_reply(91 mds0_inotable [create] v85'16 uv16 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f001c00f830 con 0x5616ccd90620 2024-08-03T20:55:01.701 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:01.699+0000 7f003480ef40 1 -- 172.21.15.32:0/451983607 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:92 12.13 12:c91d4a1d:::mds0_inotable:head [write 0~34 in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e85) -- 0x5616ccd9ba80 con 0x5616ccd90620 2024-08-03T20:55:01.702 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:01.701+0000 7f002d000640 1 -- 172.21.15.32:0/451983607 <== osd.1 v2:172.21.15.32:6824/1551241759 8 ==== osd_op_reply(92 mds0_inotable [write 0~34] v85'17 uv17 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f001c00f830 con 0x5616ccd90620 2024-08-03T20:55:01.702 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:01.701+0000 7f003480ef40 1 -- 172.21.15.32:0/451983607 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:93 12.13 12:c91d4a1d:::mds0_inotable:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e85) -- 0x5616ccd9ba80 con 0x5616ccd90620 2024-08-03T20:55:01.703 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:01.702+0000 7f002d000640 1 -- 172.21.15.32:0/451983607 <== osd.1 v2:172.21.15.32:6824/1551241759 9 ==== osd_op_reply(93 mds0_inotable [omap-set-header] v85'18 uv18 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f001c00f830 con 0x5616ccd90620 2024-08-03T20:55:01.703 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:01.702+0000 7f003480ef40 1 -- 172.21.15.32:0/451983607 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:94 12.13 12:cd5a64a3:::100.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e85) -- 0x5616ccd9ba80 con 0x5616ccd90620 2024-08-03T20:55:01.705 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:01.703+0000 7f002d000640 1 -- 172.21.15.32:0/451983607 <== osd.1 v2:172.21.15.32:6824/1551241759 10 ==== osd_op_reply(94 100.00000000 [create] v85'19 uv13 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f001c00f830 con 0x5616ccd90620 2024-08-03T20:55:01.705 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:01.703+0000 7f003480ef40 1 -- 172.21.15.32:0/451983607 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:95 12.13 12:cd5a64a3:::100.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) -- 0x5616ccd9ba80 con 0x5616ccd90620 2024-08-03T20:55:01.706 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:01.705+0000 7f002d000640 1 -- 172.21.15.32:0/451983607 <== osd.1 v2:172.21.15.32:6824/1551241759 11 ==== osd_op_reply(95 100.00000000 [delete] v85'20 uv20 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f001c00f830 con 0x5616ccd90620 2024-08-03T20:55:01.706 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:01.705+0000 7f003480ef40 1 -- 172.21.15.32:0/451983607 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:96 12.13 12:cd5a64a3:::100.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e85) -- 0x5616ccd9ba80 con 0x5616ccd90620 2024-08-03T20:55:01.707 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::100.00000000:head# 2024-08-03T20:55:01.708 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:01.706+0000 7f002d000640 1 -- 172.21.15.32:0/451983607 <== osd.1 v2:172.21.15.32:6824/1551241759 12 ==== osd_op_reply(96 100.00000000 [create] v85'21 uv21 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f001c00f830 con 0x5616ccd90620 2024-08-03T20:55:01.708 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:01.706+0000 7f003480ef40 1 -- 172.21.15.32:0/451983607 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:97 12.13 12:cd5a64a3:::100.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e85) -- 0x5616ccd9ba80 con 0x5616ccd90620 2024-08-03T20:55:01.709 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:01.707+0000 7f002d000640 1 -- 172.21.15.32:0/451983607 <== osd.1 v2:172.21.15.32:6824/1551241759 13 ==== osd_op_reply(97 100.00000000 [omap-set-header] v85'22 uv22 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f001c00f830 con 0x5616ccd90620 2024-08-03T20:55:01.709 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:01.707+0000 7f003480ef40 1 -- 172.21.15.32:0/451983607 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:98 12.13 12:cd5a64a3:::100.00000000:head [omap-set-vals in=5194b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e85) -- 0x5616ccd9e890 con 0x5616ccd90620 2024-08-03T20:55:01.710 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:01.709+0000 7f002d000640 1 -- 172.21.15.32:0/451983607 <== osd.1 v2:172.21.15.32:6824/1551241759 14 ==== osd_op_reply(98 100.00000000 [omap-set-vals] v85'23 uv23 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f001c00f830 con 0x5616ccd90620 2024-08-03T20:55:01.710 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:01.709+0000 7f003480ef40 1 --2- 172.21.15.32:0/451983607 >> [v2:172.21.15.32:6800/9088303,v1:172.21.15.32:6801/9088303] conn(0x5616ccd9c5b0 0x5616ccd9eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:55:01.711 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:01.709+0000 7f003480ef40 1 -- 172.21.15.32:0/451983607 --> [v2:172.21.15.32:6800/9088303,v1:172.21.15.32:6801/9088303] -- osd_op(unknown.0.0:99 12.b 12:d0630e4c:::mds0_sessionmap:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e85) -- 0x5616ccd9e9d0 con 0x5616ccd9c5b0 2024-08-03T20:55:01.711 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:01.709+0000 7f002d000640 1 --2- 172.21.15.32:0/451983607 >> [v2:172.21.15.32:6800/9088303,v1:172.21.15.32:6801/9088303] conn(0x5616ccd9c5b0 0x5616ccd9eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:55:01.711 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:01.710+0000 7f002d000640 1 --2- 172.21.15.32:0/451983607 >> [v2:172.21.15.32:6800/9088303,v1:172.21.15.32:6801/9088303] conn(0x5616ccd9c5b0 0x5616ccd9eed0 crc :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:55:01.712 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:01.711+0000 7f002d000640 1 -- 172.21.15.32:0/451983607 <== osd.3 v2:172.21.15.32:6800/9088303 1 ==== osd_op_reply(99 mds0_sessionmap [create] v85'9 uv7 ondisk = -17 ((17) File exists)) ==== 159+0+0 (crc 0 0 0) 0x7f001c00f830 con 0x5616ccd9c5b0 2024-08-03T20:55:01.713 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:01.711+0000 7f003480ef40 1 -- 172.21.15.32:0/451983607 --> [v2:172.21.15.32:6800/9088303,v1:172.21.15.32:6801/9088303] -- osd_op(unknown.0.0:100 12.b 12:d0630e4c:::mds0_sessionmap:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) -- 0x5616ccd9e9d0 con 0x5616ccd9c5b0 2024-08-03T20:55:01.714 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:01.713+0000 7f002d000640 1 -- 172.21.15.32:0/451983607 <== osd.3 v2:172.21.15.32:6800/9088303 2 ==== osd_op_reply(100 mds0_sessionmap [delete] v85'10 uv10 ondisk = 0) ==== 159+0+0 (crc 0 0 0) 0x7f001c00f830 con 0x5616ccd9c5b0 2024-08-03T20:55:01.714 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:01.713+0000 7f003480ef40 1 -- 172.21.15.32:0/451983607 --> [v2:172.21.15.32:6800/9088303,v1:172.21.15.32:6801/9088303] -- osd_op(unknown.0.0:101 12.b 12:d0630e4c:::mds0_sessionmap:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e85) -- 0x5616ccd9e9d0 con 0x5616ccd9c5b0 2024-08-03T20:55:01.715 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:01.714+0000 7f002d000640 1 -- 172.21.15.32:0/451983607 <== osd.3 v2:172.21.15.32:6800/9088303 3 ==== osd_op_reply(101 mds0_sessionmap [create] v85'11 uv11 ondisk = 0) ==== 159+0+0 (crc 0 0 0) 0x7f001c00f830 con 0x5616ccd9c5b0 2024-08-03T20:55:01.715 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds0_sessionmap:head# 2024-08-03T20:55:01.716 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:01.714+0000 7f003480ef40 1 -- 172.21.15.32:0/451983607 --> [v2:172.21.15.32:6800/9088303,v1:172.21.15.32:6801/9088303] -- osd_op(unknown.0.0:102 12.b 12:d0630e4c:::mds0_sessionmap:head [omap-set-header in=14b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e85) -- 0x5616ccd9e9d0 con 0x5616ccd9c5b0 2024-08-03T20:55:01.716 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:01.715+0000 7f002d000640 1 -- 172.21.15.32:0/451983607 <== osd.3 v2:172.21.15.32:6800/9088303 4 ==== osd_op_reply(102 mds0_sessionmap [omap-set-header] v85'12 uv12 ondisk = 0) ==== 159+0+0 (crc 0 0 0) 0x7f001c00f830 con 0x5616ccd9c5b0 2024-08-03T20:55:01.717 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:01.715+0000 7f003480ef40 1 -- 172.21.15.32:0/451983607 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:103 12.7 12:e0b41d06:::609.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e85) -- 0x5616ccd9e9d0 con 0x5616ccd90620 2024-08-03T20:55:01.718 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:01.717+0000 7f002d000640 1 -- 172.21.15.32:0/451983607 <== osd.1 v2:172.21.15.32:6824/1551241759 15 ==== osd_op_reply(103 609.00000000 [create] v85'9 uv8 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f001c00f830 con 0x5616ccd90620 2024-08-03T20:55:01.718 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:01.717+0000 7f003480ef40 1 -- 172.21.15.32:0/451983607 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:104 12.7 12:e0b41d06:::609.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) -- 0x5616ccd9e9d0 con 0x5616ccd90620 2024-08-03T20:55:01.720 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:01.718+0000 7f002d000640 1 -- 172.21.15.32:0/451983607 <== osd.1 v2:172.21.15.32:6824/1551241759 16 ==== osd_op_reply(104 609.00000000 [delete] v85'10 uv10 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f001c00f830 con 0x5616ccd90620 2024-08-03T20:55:01.720 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:01.718+0000 7f003480ef40 1 -- 172.21.15.32:0/451983607 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:105 12.7 12:e0b41d06:::609.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e85) -- 0x5616ccd9e9d0 con 0x5616ccd90620 2024-08-03T20:55:01.721 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::609.00000000:head# 2024-08-03T20:55:01.721 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:01.720+0000 7f002d000640 1 -- 172.21.15.32:0/451983607 <== osd.1 v2:172.21.15.32:6824/1551241759 17 ==== osd_op_reply(105 609.00000000 [create] v85'11 uv11 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f001c00f830 con 0x5616ccd90620 2024-08-03T20:55:01.722 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:01.720+0000 7f003480ef40 1 -- 172.21.15.32:0/451983607 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:106 12.7 12:e0b41d06:::609.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e85) -- 0x5616ccd9e9d0 con 0x5616ccd90620 2024-08-03T20:55:01.723 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:01.721+0000 7f002d000640 1 -- 172.21.15.32:0/451983607 <== osd.1 v2:172.21.15.32:6824/1551241759 18 ==== osd_op_reply(106 609.00000000 [setxattr (30)] v85'12 uv12 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f001c00f830 con 0x5616ccd90620 2024-08-03T20:55:01.723 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:01.721+0000 7f003480ef40 1 -- 172.21.15.32:0/451983607 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:107 12.7 12:e0b41d06:::609.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e85) -- 0x5616ccd9e9d0 con 0x5616ccd90620 2024-08-03T20:55:01.724 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:01.723+0000 7f002d000640 1 -- 172.21.15.32:0/451983607 <== osd.1 v2:172.21.15.32:6824/1551241759 19 ==== osd_op_reply(107 609.00000000 [setxattr (62)] v85'13 uv13 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f001c00f830 con 0x5616ccd90620 2024-08-03T20:55:01.724 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:01.723+0000 7f003480ef40 1 -- 172.21.15.32:0/451983607 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:108 12.7 12:e0b41d06:::609.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e85) -- 0x5616ccd9e9d0 con 0x5616ccd90620 2024-08-03T20:55:01.726 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:01.725+0000 7f002d000640 1 -- 172.21.15.32:0/451983607 <== osd.1 v2:172.21.15.32:6824/1551241759 20 ==== osd_op_reply(108 609.00000000 [omap-set-header] v85'14 uv14 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f001c00f830 con 0x5616ccd90620 2024-08-03T20:55:01.726 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:01.725+0000 7f003480ef40 1 -- 172.21.15.32:0/451983607 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:109 12.f 12:f1f69726:::400.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e85) -- 0x5616ccd9e9d0 con 0x5616ccd94910 2024-08-03T20:55:01.728 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:01.726+0000 7f002e400640 1 -- 172.21.15.32:0/451983607 <== osd.6 v2:172.21.15.118:6825/801947386 12 ==== osd_op_reply(109 400.00000000 [create] v85'7 uv6 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f002800b040 con 0x5616ccd94910 2024-08-03T20:55:01.728 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:01.726+0000 7f003480ef40 1 -- 172.21.15.32:0/451983607 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:110 12.f 12:f1f69726:::400.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) -- 0x5616ccd9e9d0 con 0x5616ccd94910 2024-08-03T20:55:01.730 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:01.728+0000 7f002e400640 1 -- 172.21.15.32:0/451983607 <== osd.6 v2:172.21.15.118:6825/801947386 13 ==== osd_op_reply(110 400.00000000 [delete] v85'8 uv8 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f002800b040 con 0x5616ccd94910 2024-08-03T20:55:01.730 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:01.728+0000 7f003480ef40 1 -- 172.21.15.32:0/451983607 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:111 12.f 12:f1f69726:::400.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e85) -- 0x5616ccd9e9d0 con 0x5616ccd94910 2024-08-03T20:55:01.731 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::400.00000000:head# 2024-08-03T20:55:01.731 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:01.730+0000 7f002e400640 1 -- 172.21.15.32:0/451983607 <== osd.6 v2:172.21.15.118:6825/801947386 14 ==== osd_op_reply(111 400.00000000 [create] v85'9 uv9 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f002800b040 con 0x5616ccd94910 2024-08-03T20:55:01.731 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:01.730+0000 7f003480ef40 1 -- 172.21.15.32:0/451983607 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:112 12.f 12:f1f69726:::400.00000000:head [write 0~22 in=22b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e85) -- 0x5616ccd9e9d0 con 0x5616ccd94910 2024-08-03T20:55:01.733 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:01.732+0000 7f002e400640 1 -- 172.21.15.32:0/451983607 <== osd.6 v2:172.21.15.118:6825/801947386 15 ==== osd_op_reply(112 400.00000000 [write 0~22] v85'10 uv10 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f002800b040 con 0x5616ccd94910 2024-08-03T20:55:01.733 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:01.732+0000 7f003480ef40 1 -- 172.21.15.32:0/451983607 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:113 12.f 12:f1f69726:::400.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e85) -- 0x5616ccd9e9d0 con 0x5616ccd94910 2024-08-03T20:55:01.735 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:01.733+0000 7f002e400640 1 -- 172.21.15.32:0/451983607 <== osd.6 v2:172.21.15.118:6825/801947386 16 ==== osd_op_reply(113 400.00000000 [omap-set-header] v85'11 uv11 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f002800b040 con 0x5616ccd94910 2024-08-03T20:55:01.735 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:01.733+0000 7f003480ef40 1 -- 172.21.15.32:0/451983607 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:114 12.1f 12:fe07bba1:::100.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e85) -- 0x5616ccd9e9d0 con 0x5616ccd94910 2024-08-03T20:55:01.736 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:01.735+0000 7f002e400640 1 -- 172.21.15.32:0/451983607 <== osd.6 v2:172.21.15.118:6825/801947386 17 ==== osd_op_reply(114 100.00000000.inode [create] v85'20 uv11 ondisk = -17 ((17) File exists)) ==== 162+0+0 (crc 0 0 0) 0x7f002800b040 con 0x5616ccd94910 2024-08-03T20:55:01.736 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:01.735+0000 7f003480ef40 1 -- 172.21.15.32:0/451983607 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:115 12.1f 12:fe07bba1:::100.00000000.inode:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) -- 0x5616ccd9e9d0 con 0x5616ccd94910 2024-08-03T20:55:01.738 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:01.737+0000 7f002e400640 1 -- 172.21.15.32:0/451983607 <== osd.6 v2:172.21.15.118:6825/801947386 18 ==== osd_op_reply(115 100.00000000.inode [delete] v85'21 uv21 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7f002800b040 con 0x5616ccd94910 2024-08-03T20:55:01.738 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:01.737+0000 7f003480ef40 1 -- 172.21.15.32:0/451983607 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:116 12.1f 12:fe07bba1:::100.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e85) -- 0x5616ccd9e9d0 con 0x5616ccd94910 2024-08-03T20:55:01.739 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::100.00000000.inode:head# 2024-08-03T20:55:01.740 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:01.738+0000 7f002e400640 1 -- 172.21.15.32:0/451983607 <== osd.6 v2:172.21.15.118:6825/801947386 19 ==== osd_op_reply(116 100.00000000.inode [create] v85'22 uv22 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7f002800b040 con 0x5616ccd94910 2024-08-03T20:55:01.740 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:01.738+0000 7f003480ef40 1 -- 172.21.15.32:0/451983607 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:117 12.1f 12:fe07bba1:::100.00000000.inode:head [write 0~582 in=582b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e85) -- 0x5616ccd9e9d0 con 0x5616ccd94910 2024-08-03T20:55:01.742 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:01.740+0000 7f002e400640 1 -- 172.21.15.32:0/451983607 <== osd.6 v2:172.21.15.118:6825/801947386 20 ==== osd_op_reply(117 100.00000000.inode [write 0~582] v85'23 uv23 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7f002800b040 con 0x5616ccd94910 2024-08-03T20:55:01.742 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:01.740+0000 7f003480ef40 1 -- 172.21.15.32:0/451983607 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:118 12.1f 12:fe07bba1:::100.00000000.inode:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e85) -- 0x5616ccd9e9d0 con 0x5616ccd94910 2024-08-03T20:55:01.743 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:01.742+0000 7f002e400640 1 -- 172.21.15.32:0/451983607 <== osd.6 v2:172.21.15.118:6825/801947386 21 ==== osd_op_reply(118 100.00000000.inode [omap-set-header] v85'24 uv24 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7f002800b040 con 0x5616ccd94910 2024-08-03T20:55:01.743 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:01.742+0000 7f003480ef40 1 -- 172.21.15.32:0/451983607 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:119 12.1f 12:ff5b34d6:::1.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e85) -- 0x5616ccd9e9d0 con 0x5616ccd94910 2024-08-03T20:55:01.744 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:01.743+0000 7f002e400640 1 -- 172.21.15.32:0/451983607 <== osd.6 v2:172.21.15.118:6825/801947386 22 ==== osd_op_reply(119 1.00000000 [create] v85'25 uv18 ondisk = -17 ((17) File exists)) ==== 154+0+0 (crc 0 0 0) 0x7f002800b040 con 0x5616ccd94910 2024-08-03T20:55:01.744 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:01.743+0000 7f003480ef40 1 -- 172.21.15.32:0/451983607 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:120 12.1f 12:ff5b34d6:::1.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) -- 0x5616ccd9e9d0 con 0x5616ccd94910 2024-08-03T20:55:01.746 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:01.745+0000 7f002e400640 1 -- 172.21.15.32:0/451983607 <== osd.6 v2:172.21.15.118:6825/801947386 23 ==== osd_op_reply(120 1.00000000 [delete] v85'26 uv26 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7f002800b040 con 0x5616ccd94910 2024-08-03T20:55:01.746 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:01.745+0000 7f003480ef40 1 -- 172.21.15.32:0/451983607 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:121 12.1f 12:ff5b34d6:::1.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e85) -- 0x5616ccd9e9d0 con 0x5616ccd94910 2024-08-03T20:55:01.747 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::1.00000000:head# 2024-08-03T20:55:01.747 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:01.746+0000 7f002e400640 1 -- 172.21.15.32:0/451983607 <== osd.6 v2:172.21.15.118:6825/801947386 24 ==== osd_op_reply(121 1.00000000 [create] v85'27 uv27 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7f002800b040 con 0x5616ccd94910 2024-08-03T20:55:01.748 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:01.746+0000 7f003480ef40 1 -- 172.21.15.32:0/451983607 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:122 12.1f 12:ff5b34d6:::1.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e85) -- 0x5616ccd9e9d0 con 0x5616ccd94910 2024-08-03T20:55:01.749 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:01.747+0000 7f002e400640 1 -- 172.21.15.32:0/451983607 <== osd.6 v2:172.21.15.118:6825/801947386 25 ==== osd_op_reply(122 1.00000000 [setxattr (30)] v85'28 uv28 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7f002800b040 con 0x5616ccd94910 2024-08-03T20:55:01.749 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:01.747+0000 7f003480ef40 1 -- 172.21.15.32:0/451983607 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:123 12.1f 12:ff5b34d6:::1.00000000:head [setxattr parent (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e85) -- 0x5616ccd9e9d0 con 0x5616ccd94910 2024-08-03T20:55:01.750 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:01.749+0000 7f002e400640 1 -- 172.21.15.32:0/451983607 <== osd.6 v2:172.21.15.118:6825/801947386 26 ==== osd_op_reply(123 1.00000000 [setxattr (30)] v85'29 uv29 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7f002800b040 con 0x5616ccd94910 2024-08-03T20:55:01.750 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:01.749+0000 7f003480ef40 1 -- 172.21.15.32:0/451983607 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:124 12.1f 12:ff5b34d6:::1.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e85) -- 0x5616ccd9e9d0 con 0x5616ccd94910 2024-08-03T20:55:01.752 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:01.750+0000 7f002e400640 1 -- 172.21.15.32:0/451983607 <== osd.6 v2:172.21.15.118:6825/801947386 27 ==== osd_op_reply(124 1.00000000 [omap-set-header] v85'30 uv30 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7f002800b040 con 0x5616ccd94910 2024-08-03T20:55:01.752 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:01.750+0000 7f003480ef40 1 -- 172.21.15.32:0/451983607 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:125 12.1f 12:ff5b34d6:::1.00000000:head [omap-set-vals in=584b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e85) -- 0x5616ccd9f8c0 con 0x5616ccd94910 2024-08-03T20:55:01.754 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:01.752+0000 7f002e400640 1 -- 172.21.15.32:0/451983607 <== osd.6 v2:172.21.15.118:6825/801947386 28 ==== osd_op_reply(125 1.00000000 [omap-set-vals] v85'31 uv31 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7f002800b040 con 0x5616ccd94910 2024-08-03T20:55:01.754 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:01.752+0000 7f003480ef40 1 -- 172.21.15.32:0/451983607 >> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] conn(0x5616cccbb090 msgr2=0x5616cccbb4a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:55:01.754 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:01.752+0000 7f003480ef40 1 --2- 172.21.15.32:0/451983607 >> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] conn(0x5616cccbb090 0x5616cccbb4a0 crc :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:01.754 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:01.752+0000 7f003480ef40 1 -- 172.21.15.32:0/451983607 >> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] conn(0x5616ccd90620 msgr2=0x5616ccd90a30 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:55:01.755 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:01.752+0000 7f003480ef40 1 --2- 172.21.15.32:0/451983607 >> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] conn(0x5616ccd90620 0x5616ccd90a30 crc :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:01.755 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:01.752+0000 7f003480ef40 1 -- 172.21.15.32:0/451983607 >> [v2:172.21.15.32:6800/9088303,v1:172.21.15.32:6801/9088303] conn(0x5616ccd9c5b0 msgr2=0x5616ccd9eed0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:55:01.755 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:01.752+0000 7f003480ef40 1 --2- 172.21.15.32:0/451983607 >> [v2:172.21.15.32:6800/9088303,v1:172.21.15.32:6801/9088303] conn(0x5616ccd9c5b0 0x5616ccd9eed0 crc :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:01.755 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:01.752+0000 7f003480ef40 1 -- 172.21.15.32:0/451983607 >> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] conn(0x5616ccd8b8f0 msgr2=0x5616ccd8dd10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:55:01.755 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:01.752+0000 7f003480ef40 1 --2- 172.21.15.32:0/451983607 >> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] conn(0x5616ccd8b8f0 0x5616ccd8dd10 crc :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:01.755 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:01.752+0000 7f003480ef40 1 -- 172.21.15.32:0/451983607 >> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] conn(0x5616ccd94910 msgr2=0x5616ccd94d40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:55:01.755 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:01.752+0000 7f002e400640 1 -- 172.21.15.32:0/451983607 reap_dead start 2024-08-03T20:55:01.755 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:01.752+0000 7f003480ef40 1 --2- 172.21.15.32:0/451983607 >> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] conn(0x5616ccd94910 0x5616ccd94d40 crc :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:01.755 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:01.752+0000 7f003480ef40 1 -- 172.21.15.32:0/451983607 >> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] conn(0x5616ccd97ff0 msgr2=0x5616ccd98400 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:55:01.755 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:01.752+0000 7f003480ef40 1 --2- 172.21.15.32:0/451983607 >> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] conn(0x5616ccd97ff0 0x5616ccd98400 crc :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:01.755 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:01.752+0000 7f003480ef40 1 -- 172.21.15.32:0/451983607 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f00100759f0 msgr2=0x7f0010077f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:55:01.755 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:01.752+0000 7f003480ef40 1 --2- 172.21.15.32:0/451983607 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f00100759f0 0x7f0010077f00 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7f001c001d50 tx=0x7f001c000ed0 comp rx=0 tx=0).stop 2024-08-03T20:55:01.755 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:01.752+0000 7f003480ef40 1 -- 172.21.15.32:0/451983607 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5616cccc9360 msgr2=0x5616ccc52b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:55:01.755 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:01.752+0000 7f003480ef40 1 --2- 172.21.15.32:0/451983607 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5616cccc9360 0x5616ccc52b60 secure :-1 s=READY pgs=1119 cs=0 l=1 rev1=1 crypto rx=0x7f00200086c0 tx=0x7f00200086f0 comp rx=0 tx=0).stop 2024-08-03T20:55:01.756 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:01.752+0000 7f003480ef40 1 -- 172.21.15.32:0/451983607 shutdown_connections 2024-08-03T20:55:01.756 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:01.752+0000 7f003480ef40 1 --2- 172.21.15.32:0/451983607 >> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] conn(0x5616ccd97ff0 0x5616ccd98400 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:01.756 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:01.752+0000 7f003480ef40 1 --2- 172.21.15.32:0/451983607 >> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] conn(0x5616ccd94910 0x5616ccd94d40 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:01.756 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:01.752+0000 7f003480ef40 1 --2- 172.21.15.32:0/451983607 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f00100759f0 0x7f0010077f00 unknown :-1 s=CLOSED pgs=464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:01.756 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:01.752+0000 7f003480ef40 1 --2- 172.21.15.32:0/451983607 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5616cccc9360 0x5616ccc52b60 unknown :-1 s=CLOSED pgs=1119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:01.756 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:01.752+0000 7f003480ef40 1 -- 172.21.15.32:0/451983607 >> 172.21.15.32:0/451983607 conn(0x5616ccb32ac0 msgr2=0x5616ccc59000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:55:01.756 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:01.752+0000 7f003480ef40 1 -- 172.21.15.32:0/451983607 shutdown_connections 2024-08-03T20:55:01.756 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:01.752+0000 7f003480ef40 1 -- 172.21.15.32:0/451983607 wait complete. 2024-08-03T20:55:01.765 DEBUG:teuthology.orchestra.run.smithi032:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata rm 600.00000000 2024-08-03T20:55:01.947 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:01.946+0000 7f217d034f40 1 Processor -- start 2024-08-03T20:55:01.947 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:01.946+0000 7f217d034f40 1 -- start start 2024-08-03T20:55:01.949 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:01.947+0000 7f217d034f40 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x5602bbdbccb0 0x5602bbda1470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:55:01.949 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:01.947+0000 7f217d034f40 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5602bbdbde20 0x5602bbe38070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:55:01.950 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:01.947+0000 7f217d034f40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5602bbe40ba0 0x5602bbe385b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:55:01.950 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:01.947+0000 7f217d034f40 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x5602bbcc2e30 con 0x5602bbdbde20 2024-08-03T20:55:01.950 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:01.947+0000 7f217d034f40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x5602bbd20bf0 con 0x5602bbe40ba0 2024-08-03T20:55:01.950 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:01.947+0000 7f217d034f40 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x5602bbe49fc0 con 0x5602bbdbccb0 2024-08-03T20:55:01.950 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:01.947+0000 7f2175800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5602bbdbde20 0x5602bbe38070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:55:01.950 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:01.947+0000 7f2175800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5602bbdbde20 0x5602bbe38070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:47922/0 (socket says 172.21.15.32:47922) 2024-08-03T20:55:01.950 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:01.947+0000 7f2175800640 1 -- 172.21.15.32:0/3846703895 learned_addr learned my addr 172.21.15.32:0/3846703895 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:55:01.950 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:01.947+0000 7f2176200640 1 --2- 172.21.15.32:0/3846703895 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x5602bbdbccb0 0x5602bbda1470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:55:01.950 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:01.947+0000 7f2176c00640 1 --2- 172.21.15.32:0/3846703895 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5602bbe40ba0 0x5602bbe385b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:55:01.950 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:01.948+0000 7f2175800640 1 -- 172.21.15.32:0/3846703895 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x5602bbdbccb0 msgr2=0x5602bbda1470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:55:01.950 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:01.948+0000 7f2175800640 1 --2- 172.21.15.32:0/3846703895 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x5602bbdbccb0 0x5602bbda1470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:01.950 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:01.948+0000 7f2175800640 1 -- 172.21.15.32:0/3846703895 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5602bbe40ba0 msgr2=0x5602bbe385b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:55:01.951 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:01.948+0000 7f2175800640 1 --2- 172.21.15.32:0/3846703895 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5602bbe40ba0 0x5602bbe385b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:01.951 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:01.948+0000 7f2175800640 1 -- 172.21.15.32:0/3846703895 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5602bbdb9060 con 0x5602bbdbde20 2024-08-03T20:55:01.951 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:01.948+0000 7f2175800640 1 --2- 172.21.15.32:0/3846703895 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5602bbdbde20 0x5602bbe38070 secure :-1 s=READY pgs=1120 cs=0 l=1 rev1=1 crypto rx=0x7f2164025e60 tx=0x7f2164002fb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1d59fe3a091b8a86 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:55:01.952 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:01.948+0000 7f216fe00640 1 -- 172.21.15.32:0/3846703895 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f216404d8c0 con 0x5602bbdbde20 2024-08-03T20:55:01.952 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:01.948+0000 7f216fe00640 1 -- 172.21.15.32:0/3846703895 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f216404dd70 con 0x5602bbdbde20 2024-08-03T20:55:01.952 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:01.948+0000 7f216fe00640 1 -- 172.21.15.32:0/3846703895 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21640566d0 con 0x5602bbdbde20 2024-08-03T20:55:01.953 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:01.949+0000 7f217d034f40 1 -- 172.21.15.32:0/3846703895 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5602bbdbde20 msgr2=0x5602bbe38070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:55:01.953 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:01.949+0000 7f217d034f40 1 --2- 172.21.15.32:0/3846703895 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5602bbdbde20 0x5602bbe38070 secure :-1 s=READY pgs=1120 cs=0 l=1 rev1=1 crypto rx=0x7f2164025e60 tx=0x7f2164002fb0 comp rx=0 tx=0).stop 2024-08-03T20:55:01.953 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:01.949+0000 7f217d034f40 1 -- 172.21.15.32:0/3846703895 shutdown_connections 2024-08-03T20:55:01.953 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:01.949+0000 7f217d034f40 1 --2- 172.21.15.32:0/3846703895 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5602bbe40ba0 0x5602bbe385b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:01.953 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:01.949+0000 7f217d034f40 1 --2- 172.21.15.32:0/3846703895 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5602bbdbde20 0x5602bbe38070 unknown :-1 s=CLOSED pgs=1120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:01.953 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:01.949+0000 7f217d034f40 1 --2- 172.21.15.32:0/3846703895 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x5602bbdbccb0 0x5602bbda1470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:01.953 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:01.949+0000 7f217d034f40 1 -- 172.21.15.32:0/3846703895 >> 172.21.15.32:0/3846703895 conn(0x5602bbc94ac0 msgr2=0x5602bbda1eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:55:01.953 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:01.949+0000 7f217d034f40 1 -- 172.21.15.32:0/3846703895 shutdown_connections 2024-08-03T20:55:01.953 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:01.949+0000 7f217d034f40 1 -- 172.21.15.32:0/3846703895 wait complete. 2024-08-03T20:55:01.953 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:01.950+0000 7f217d034f40 1 Processor -- start 2024-08-03T20:55:01.953 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:01.951+0000 7f217d034f40 1 -- start start 2024-08-03T20:55:01.955 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:01.952+0000 7f217d034f40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5602bbdbccb0 0x5602bbe33130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:55:01.955 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:01.952+0000 7f217d034f40 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5602bbdbde20 0x5602bbe357b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:55:01.956 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:01.952+0000 7f217d034f40 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x5602bbe40ba0 0x5602bbe35cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:55:01.956 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:01.952+0000 7f217d034f40 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x5602bbdb7a80 con 0x5602bbdbde20 2024-08-03T20:55:01.956 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:01.952+0000 7f217d034f40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x5602bbe3f9d0 con 0x5602bbdbccb0 2024-08-03T20:55:01.956 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:01.952+0000 7f217d034f40 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x5602bbe34620 con 0x5602bbe40ba0 2024-08-03T20:55:01.956 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:01.952+0000 7f2175800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5602bbdbde20 0x5602bbe357b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:55:01.956 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:01.952+0000 7f2175800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5602bbdbde20 0x5602bbe357b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:47924/0 (socket says 172.21.15.32:47924) 2024-08-03T20:55:01.956 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:01.952+0000 7f2176200640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5602bbdbccb0 0x5602bbe33130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:55:01.956 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:01.952+0000 7f2175800640 1 -- 172.21.15.32:0/1462709121 learned_addr learned my addr 172.21.15.32:0/1462709121 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:55:01.957 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:01.952+0000 7f2176c00640 1 --2- 172.21.15.32:0/1462709121 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x5602bbe40ba0 0x5602bbe35cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:55:01.957 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:01.953+0000 7f2175800640 1 -- 172.21.15.32:0/1462709121 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x5602bbe40ba0 msgr2=0x5602bbe35cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:55:01.957 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:01.953+0000 7f2175800640 1 --2- 172.21.15.32:0/1462709121 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x5602bbe40ba0 0x5602bbe35cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:01.957 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:01.953+0000 7f2175800640 1 -- 172.21.15.32:0/1462709121 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5602bbdbccb0 msgr2=0x5602bbe33130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:55:01.957 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:01.953+0000 7f2175800640 1 --2- 172.21.15.32:0/1462709121 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5602bbdbccb0 0x5602bbe33130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:01.957 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:01.953+0000 7f2175800640 1 -- 172.21.15.32:0/1462709121 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5602bbe34da0 con 0x5602bbdbde20 2024-08-03T20:55:01.957 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:01.953+0000 7f2176200640 1 --2- 172.21.15.32:0/1462709121 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5602bbdbccb0 0x5602bbe33130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-03T20:55:01.957 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:01.953+0000 7f2175800640 1 --2- 172.21.15.32:0/1462709121 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5602bbdbde20 0x5602bbe357b0 secure :-1 s=READY pgs=1121 cs=0 l=1 rev1=1 crypto rx=0x7f2164025e30 tx=0x7f2164046470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:55:01.957 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:01.954+0000 7f216ea00640 1 -- 172.21.15.32:0/1462709121 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f216406f030 con 0x5602bbdbde20 2024-08-03T20:55:01.957 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:01.954+0000 7f216ea00640 1 -- 172.21.15.32:0/1462709121 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f2164057690 con 0x5602bbdbde20 2024-08-03T20:55:01.957 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:01.954+0000 7f217d034f40 1 -- 172.21.15.32:0/1462709121 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x5602bbe35060 con 0x5602bbdbde20 2024-08-03T20:55:01.957 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:01.954+0000 7f216ea00640 1 -- 172.21.15.32:0/1462709121 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21640578e0 con 0x5602bbdbde20 2024-08-03T20:55:01.957 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:01.954+0000 7f217d034f40 1 -- 172.21.15.32:0/1462709121 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5602bbe36660 con 0x5602bbdbde20 2024-08-03T20:55:01.958 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:01.956+0000 7f216ea00640 1 -- 172.21.15.32:0/1462709121 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f2164072070 con 0x5602bbdbde20 2024-08-03T20:55:01.958 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:01.957+0000 7f216ea00640 1 --2- 172.21.15.32:0/1462709121 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f2158075a20 0x7f2158077f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:55:01.959 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:01.957+0000 7f216ea00640 1 -- 172.21.15.32:0/1462709121 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 5881+0+0 (secure 0 0 0) 0x7f21640dc9f0 con 0x5602bbdbde20 2024-08-03T20:55:01.959 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:01.957+0000 7f217d034f40 1 --2- 172.21.15.32:0/1462709121 >> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] conn(0x5602bbda2720 0x5602bbe33d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:55:01.960 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:01.957+0000 7f217d034f40 1 -- 172.21.15.32:0/1462709121 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:1 12.10 12:0d82a743:::600.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) -- 0x5602bbe352f0 con 0x5602bbda2720 2024-08-03T20:55:01.960 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:01.957+0000 7f2176200640 1 --2- 172.21.15.32:0/1462709121 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f2158075a20 0x7f2158077f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:55:01.960 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:01.957+0000 7f2176c00640 1 --2- 172.21.15.32:0/1462709121 >> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] conn(0x5602bbda2720 0x5602bbe33d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:55:01.960 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:01.958+0000 7f2176c00640 1 --2- 172.21.15.32:0/1462709121 >> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] conn(0x5602bbda2720 0x5602bbe33d50 crc :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:55:01.960 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:01.958+0000 7f2176200640 1 --2- 172.21.15.32:0/1462709121 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f2158075a20 0x7f2158077f30 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7f216800cdf0 tx=0x7f216800f040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:55:01.962 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:01.960+0000 7f2176c00640 1 -- 172.21.15.32:0/1462709121 <== osd.0 v2:172.21.15.32:6816/3066156116 1 ==== osd_op_reply(1 600.00000000 [delete] v85'72 uv72 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f21700025f0 con 0x5602bbda2720 2024-08-03T20:55:01.962 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:01.960+0000 7f217d034f40 1 -- 172.21.15.32:0/1462709121 >> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] conn(0x5602bbda2720 msgr2=0x5602bbe33d50 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:55:01.962 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:01.960+0000 7f217d034f40 1 --2- 172.21.15.32:0/1462709121 >> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] conn(0x5602bbda2720 0x5602bbe33d50 crc :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:01.962 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:01.960+0000 7f217d034f40 1 -- 172.21.15.32:0/1462709121 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f2158075a20 msgr2=0x7f2158077f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:55:01.962 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:01.960+0000 7f217d034f40 1 --2- 172.21.15.32:0/1462709121 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f2158075a20 0x7f2158077f30 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7f216800cdf0 tx=0x7f216800f040 comp rx=0 tx=0).stop 2024-08-03T20:55:01.962 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:01.960+0000 7f217d034f40 1 -- 172.21.15.32:0/1462709121 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5602bbdbde20 msgr2=0x5602bbe357b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:55:01.962 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:01.960+0000 7f217d034f40 1 --2- 172.21.15.32:0/1462709121 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5602bbdbde20 0x5602bbe357b0 secure :-1 s=READY pgs=1121 cs=0 l=1 rev1=1 crypto rx=0x7f2164025e30 tx=0x7f2164046470 comp rx=0 tx=0).stop 2024-08-03T20:55:01.963 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:01.960+0000 7f2176c00640 1 -- 172.21.15.32:0/1462709121 reap_dead start 2024-08-03T20:55:01.963 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:01.960+0000 7f217d034f40 1 -- 172.21.15.32:0/1462709121 shutdown_connections 2024-08-03T20:55:01.963 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:01.960+0000 7f217d034f40 1 -- 172.21.15.32:0/1462709121 >> 172.21.15.32:0/1462709121 conn(0x5602bbc94ac0 msgr2=0x5602bbe3f050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:55:01.963 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:01.960+0000 7f217d034f40 1 -- 172.21.15.32:0/1462709121 shutdown_connections 2024-08-03T20:55:01.963 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:01.960+0000 7f217d034f40 1 -- 172.21.15.32:0/1462709121 wait complete. 2024-08-03T20:55:01.972 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs joinable true 2024-08-03T20:55:02.204 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:02.202+0000 7f40d3200640 1 -- 172.21.15.32:0/2193470958 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f40d41609d0 msgr2=0x7f40d40cc050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:55:02.204 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:02.202+0000 7f40d3200640 1 --2- 172.21.15.32:0/2193470958 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f40d41609d0 0x7f40d40cc050 secure :-1 s=READY pgs=1123 cs=0 l=1 rev1=1 crypto rx=0x7f40d40cb2c0 tx=0x7f40c402a5c0 comp rx=0 tx=0).stop 2024-08-03T20:55:02.204 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:02.202+0000 7f40d3200640 1 -- 172.21.15.32:0/2193470958 shutdown_connections 2024-08-03T20:55:02.204 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:02.202+0000 7f40d3200640 1 --2- 172.21.15.32:0/2193470958 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f40d41609d0 0x7f40d40cc050 unknown :-1 s=CLOSED pgs=1123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:02.204 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:02.202+0000 7f40d3200640 1 -- 172.21.15.32:0/2193470958 >> 172.21.15.32:0/2193470958 conn(0x7f40d40cfd80 msgr2=0x7f40d40d0190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:55:02.204 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:02.202+0000 7f40d3200640 1 -- 172.21.15.32:0/2193470958 shutdown_connections 2024-08-03T20:55:02.204 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:02.202+0000 7f40d3200640 1 -- 172.21.15.32:0/2193470958 wait complete. 2024-08-03T20:55:02.206 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:02.204+0000 7f40d3200640 1 Processor -- start 2024-08-03T20:55:02.206 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:02.205+0000 7f40d3200640 1 -- start start 2024-08-03T20:55:02.209 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:02.206+0000 7f40d3200640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f40d4159ff0 0x7f40d4156a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:55:02.209 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:02.206+0000 7f40d3200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f40d41609d0 0x7f40d40afff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:55:02.209 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:02.206+0000 7f40d3200640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f40d4157290 0x7f40d40b0530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:55:02.209 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:02.206+0000 7f40d3200640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f40d406f0d0 con 0x7f40d41609d0 2024-08-03T20:55:02.209 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:02.206+0000 7f40d3200640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f40d406ef50 con 0x7f40d4159ff0 2024-08-03T20:55:02.209 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:02.206+0000 7f40d3200640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7f40d406f250 con 0x7f40d4157290 2024-08-03T20:55:02.209 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:02.206+0000 7f40c9400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f40d41609d0 0x7f40d40afff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:55:02.210 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:02.206+0000 7f40c9400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f40d41609d0 0x7f40d40afff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:47942/0 (socket says 172.21.15.32:47942) 2024-08-03T20:55:02.210 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:02.206+0000 7f40c9400640 1 -- 172.21.15.32:0/787189521 learned_addr learned my addr 172.21.15.32:0/787189521 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:55:02.210 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:02.206+0000 7f40d1e00640 1 --2- 172.21.15.32:0/787189521 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f40d4159ff0 0x7f40d4156a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:55:02.210 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:02.207+0000 7f40d2800640 1 --2- 172.21.15.32:0/787189521 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f40d4157290 0x7f40d40b0530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:55:02.210 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:02.207+0000 7f40c9400640 1 -- 172.21.15.32:0/787189521 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f40d4157290 msgr2=0x7f40d40b0530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:55:02.210 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:02.207+0000 7f40c9400640 1 --2- 172.21.15.32:0/787189521 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f40d4157290 0x7f40d40b0530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:02.210 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:02.207+0000 7f40c9400640 1 -- 172.21.15.32:0/787189521 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f40d4159ff0 msgr2=0x7f40d4156a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:55:02.210 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:02.207+0000 7f40c9400640 1 --2- 172.21.15.32:0/787189521 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f40d4159ff0 0x7f40d4156a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:02.210 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:02.207+0000 7f40c9400640 1 -- 172.21.15.32:0/787189521 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f40d40b3b10 con 0x7f40d41609d0 2024-08-03T20:55:02.210 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:02.207+0000 7f40c9400640 1 --2- 172.21.15.32:0/787189521 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f40d41609d0 0x7f40d40afff0 secure :-1 s=READY pgs=1124 cs=0 l=1 rev1=1 crypto rx=0x7f40bc00c400 tx=0x7f40bc00c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:55:02.210 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:02.207+0000 7f40caa00640 1 -- 172.21.15.32:0/787189521 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40bc017070 con 0x7f40d41609d0 2024-08-03T20:55:02.210 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:02.207+0000 7f40caa00640 1 -- 172.21.15.32:0/787189521 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f40bc0053c0 con 0x7f40d41609d0 2024-08-03T20:55:02.211 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:02.207+0000 7f40d3200640 1 -- 172.21.15.32:0/787189521 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f40d40b3df0 con 0x7f40d41609d0 2024-08-03T20:55:02.211 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:02.207+0000 7f40caa00640 1 -- 172.21.15.32:0/787189521 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40bc003850 con 0x7f40d41609d0 2024-08-03T20:55:02.211 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:02.208+0000 7f40d3200640 1 -- 172.21.15.32:0/787189521 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f40d415d0a0 con 0x7f40d41609d0 2024-08-03T20:55:02.211 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:02.209+0000 7f40d3200640 1 -- 172.21.15.32:0/787189521 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f40d40cc050 con 0x7f40d41609d0 2024-08-03T20:55:02.211 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:02.210+0000 7f40caa00640 1 -- 172.21.15.32:0/787189521 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f40bc01a070 con 0x7f40d41609d0 2024-08-03T20:55:02.212 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:02.211+0000 7f40caa00640 1 --2- 172.21.15.32:0/787189521 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f40b0077010 0x7f40b0079520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:55:02.212 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:02.211+0000 7f40caa00640 1 -- 172.21.15.32:0/787189521 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 5881+0+0 (secure 0 0 0) 0x7f40bc097d90 con 0x7f40d41609d0 2024-08-03T20:55:02.212 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:02.211+0000 7f40d1e00640 1 --2- 172.21.15.32:0/787189521 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f40b0077010 0x7f40b0079520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:55:02.213 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:02.212+0000 7f40d1e00640 1 --2- 172.21.15.32:0/787189521 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f40b0077010 0x7f40b0079520 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f40c4034000 tx=0x7f40c4004730 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:55:02.216 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:02.215+0000 7f40caa00640 1 -- 172.21.15.32:0/787189521 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7f40bc060880 con 0x7f40d41609d0 2024-08-03T20:55:02.459 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:02.458+0000 7f40d3200640 1 -- 172.21.15.32:0/787189521 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "joinable", "val": "true"} v 0) -- 0x7f40d40d09e0 con 0x7f40d41609d0 2024-08-03T20:55:03.045 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:03.043+0000 7f40caa00640 1 -- 172.21.15.32:0/787189521 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "joinable", "val": "true"}]=0 cephfs marked joinable; MDS may join as newly active. v102) ==== 162+0+0 (secure 0 0 0) 0x7f40bc069110 con 0x7f40d41609d0 2024-08-03T20:55:03.045 INFO:teuthology.orchestra.run.smithi032.stderr:cephfs marked joinable; MDS may join as newly active. 2024-08-03T20:55:03.048 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:03.046+0000 7f40d3200640 1 -- 172.21.15.32:0/787189521 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f40b0077010 msgr2=0x7f40b0079520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:55:03.048 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:03.046+0000 7f40d3200640 1 --2- 172.21.15.32:0/787189521 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f40b0077010 0x7f40b0079520 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f40c4034000 tx=0x7f40c4004730 comp rx=0 tx=0).stop 2024-08-03T20:55:03.048 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:03.046+0000 7f40d3200640 1 -- 172.21.15.32:0/787189521 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f40d41609d0 msgr2=0x7f40d40afff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:55:03.049 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:03.046+0000 7f40d3200640 1 --2- 172.21.15.32:0/787189521 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f40d41609d0 0x7f40d40afff0 secure :-1 s=READY pgs=1124 cs=0 l=1 rev1=1 crypto rx=0x7f40bc00c400 tx=0x7f40bc00c8c0 comp rx=0 tx=0).stop 2024-08-03T20:55:03.049 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:03.047+0000 7f40d3200640 1 -- 172.21.15.32:0/787189521 shutdown_connections 2024-08-03T20:55:03.049 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:03.047+0000 7f40d3200640 1 --2- 172.21.15.32:0/787189521 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f40b0077010 0x7f40b0079520 unknown :-1 s=CLOSED pgs=466 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:03.049 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:03.047+0000 7f40d3200640 1 --2- 172.21.15.32:0/787189521 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f40d4157290 0x7f40d40b0530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:03.049 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:03.047+0000 7f40d3200640 1 --2- 172.21.15.32:0/787189521 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f40d41609d0 0x7f40d40afff0 unknown :-1 s=CLOSED pgs=1124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:03.049 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:03.047+0000 7f40d3200640 1 --2- 172.21.15.32:0/787189521 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f40d4159ff0 0x7f40d4156a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:03.049 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:03.047+0000 7f40d3200640 1 -- 172.21.15.32:0/787189521 >> 172.21.15.32:0/787189521 conn(0x7f40d40cfd80 msgr2=0x7f40d4159700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:55:03.049 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:03.047+0000 7f40d3200640 1 -- 172.21.15.32:0/787189521 shutdown_connections 2024-08-03T20:55:03.049 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:03.047+0000 7f40d3200640 1 -- 172.21.15.32:0/787189521 wait complete. 2024-08-03T20:55:03.076 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-08-03T20:55:03.307 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:03.305+0000 7fa80de00640 1 -- 172.21.15.32:0/3976208480 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa80806e930 msgr2=0x7fa808144f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:55:03.308 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:03.305+0000 7fa80de00640 1 --2- 172.21.15.32:0/3976208480 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa80806e930 0x7fa808144f90 secure :-1 s=READY pgs=1125 cs=0 l=1 rev1=1 crypto rx=0x7fa8080b2f80 tx=0x7fa7f802a560 comp rx=0 tx=0).stop 2024-08-03T20:55:03.308 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:03.305+0000 7fa80de00640 1 -- 172.21.15.32:0/3976208480 shutdown_connections 2024-08-03T20:55:03.308 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:03.305+0000 7fa80de00640 1 --2- 172.21.15.32:0/3976208480 >> v2:172.21.15.118:3300/0 conn(0x7fa8080747b0 0x7fa8080cc8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:03.308 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:03.305+0000 7fa80de00640 1 --2- 172.21.15.32:0/3976208480 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fa808073e60 0x7fa808074270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:03.308 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:03.305+0000 7fa80de00640 1 --2- 172.21.15.32:0/3976208480 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa80806e930 0x7fa808144f90 unknown :-1 s=CLOSED pgs=1125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:03.308 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:03.305+0000 7fa80de00640 1 -- 172.21.15.32:0/3976208480 >> 172.21.15.32:0/3976208480 conn(0x7fa8080b1bd0 msgr2=0x7fa8080b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:55:03.308 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:03.305+0000 7fa80de00640 1 -- 172.21.15.32:0/3976208480 shutdown_connections 2024-08-03T20:55:03.308 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:03.305+0000 7fa80de00640 1 -- 172.21.15.32:0/3976208480 wait complete. 2024-08-03T20:55:03.309 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:03.308+0000 7fa80de00640 1 Processor -- start 2024-08-03T20:55:03.309 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:03.308+0000 7fa80de00640 1 -- start start 2024-08-03T20:55:03.311 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:03.309+0000 7fa80de00640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fa80806e930 0x7fa8080ae450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:55:03.311 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:03.309+0000 7fa80de00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa808073e60 0x7fa8080a9dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:55:03.311 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:03.309+0000 7fa80de00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa8080747b0 0x7fa80813e490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:55:03.311 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:03.309+0000 7fa80de00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fa8080cf9c0 con 0x7fa8080747b0 2024-08-03T20:55:03.311 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:03.309+0000 7fa80de00640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7fa8080cf840 con 0x7fa808073e60 2024-08-03T20:55:03.312 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:03.309+0000 7fa80de00640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7fa8080cce20 con 0x7fa80806e930 2024-08-03T20:55:03.312 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:03.309+0000 7fa807400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa8080747b0 0x7fa80813e490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:55:03.312 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:03.310+0000 7fa7ffe00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa808073e60 0x7fa8080a9dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:55:03.312 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:03.310+0000 7fa806a00640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fa80806e930 0x7fa8080ae450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:55:03.312 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:03.310+0000 7fa7ffe00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa808073e60 0x7fa8080a9dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.32:54550/0 (socket says 172.21.15.32:54550) 2024-08-03T20:55:03.312 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:03.310+0000 7fa7ffe00640 1 -- 172.21.15.32:0/1337027708 learned_addr learned my addr 172.21.15.32:0/1337027708 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:55:03.312 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:03.310+0000 7fa807400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa8080747b0 0x7fa80813e490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:47960/0 (socket says 172.21.15.32:47960) 2024-08-03T20:55:03.312 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:03.310+0000 7fa806a00640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fa80806e930 0x7fa8080ae450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3301/0 says I am v2:172.21.15.32:34684/0 (socket says 172.21.15.32:34684) 2024-08-03T20:55:03.313 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:03.310+0000 7fa807400640 1 -- 172.21.15.32:0/1337027708 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fa80806e930 msgr2=0x7fa8080ae450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:55:03.314 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:03.310+0000 7fa807400640 1 --2- 172.21.15.32:0/1337027708 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fa80806e930 0x7fa8080ae450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:03.314 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:03.310+0000 7fa807400640 1 -- 172.21.15.32:0/1337027708 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa808073e60 msgr2=0x7fa8080a9dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:55:03.314 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:03.310+0000 7fa807400640 1 --2- 172.21.15.32:0/1337027708 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa808073e60 0x7fa8080a9dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:03.314 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:03.310+0000 7fa807400640 1 -- 172.21.15.32:0/1337027708 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa8080aa310 con 0x7fa8080747b0 2024-08-03T20:55:03.314 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:03.310+0000 7fa7ffe00640 1 --2- 172.21.15.32:0/1337027708 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa808073e60 0x7fa8080a9dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-03T20:55:03.314 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:03.310+0000 7fa807400640 1 --2- 172.21.15.32:0/1337027708 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa8080747b0 0x7fa80813e490 secure :-1 s=READY pgs=1126 cs=0 l=1 rev1=1 crypto rx=0x7fa800009ca0 tx=0x7fa800007590 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:55:03.314 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:03.311+0000 7fa7ff400640 1 -- 172.21.15.32:0/1337027708 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa800007ca0 con 0x7fa8080747b0 2024-08-03T20:55:03.314 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:03.311+0000 7fa7ff400640 1 -- 172.21.15.32:0/1337027708 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fa800007e40 con 0x7fa8080747b0 2024-08-03T20:55:03.314 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:03.311+0000 7fa80de00640 1 -- 172.21.15.32:0/1337027708 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa8080aec10 con 0x7fa8080747b0 2024-08-03T20:55:03.315 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:03.311+0000 7fa7ff400640 1 -- 172.21.15.32:0/1337027708 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa800004d80 con 0x7fa8080747b0 2024-08-03T20:55:03.315 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:03.311+0000 7fa80de00640 1 -- 172.21.15.32:0/1337027708 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa8080af1d0 con 0x7fa8080747b0 2024-08-03T20:55:03.315 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:03.312+0000 7fa80de00640 1 -- 172.21.15.32:0/1337027708 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa808074270 con 0x7fa8080747b0 2024-08-03T20:55:03.315 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:03.313+0000 7fa7ff400640 1 -- 172.21.15.32:0/1337027708 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7fa8000100d0 con 0x7fa8080747b0 2024-08-03T20:55:03.316 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:03.315+0000 7fa7ff400640 1 --2- 172.21.15.32:0/1337027708 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fa7e4077040 0x7fa7e4079550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:55:03.316 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:03.315+0000 7fa7ff400640 1 -- 172.21.15.32:0/1337027708 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 5881+0+0 (secure 0 0 0) 0x7fa80009c6b0 con 0x7fa8080747b0 2024-08-03T20:55:03.316 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:03.315+0000 7fa806a00640 1 --2- 172.21.15.32:0/1337027708 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fa7e4077040 0x7fa7e4079550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:55:03.317 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:03.316+0000 7fa806a00640 1 --2- 172.21.15.32:0/1337027708 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fa7e4077040 0x7fa7e4079550 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7fa7f8033000 tx=0x7fa7f8004730 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:55:03.320 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:03.319+0000 7fa7ff400640 1 -- 172.21.15.32:0/1337027708 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7fa800065110 con 0x7fa8080747b0 2024-08-03T20:55:03.562 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:03.561+0000 7fa80de00640 1 -- 172.21.15.32:0/1337027708 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa8080ae9a0 con 0x7fa8080747b0 2024-08-03T20:55:03.563 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:03.561+0000 7fa7ff400640 1 -- 172.21.15.32:0/1337027708 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 103 v103) ==== 95+0+5225 (secure 0 0 0) 0x7fa800069660 con 0x7fa8080747b0 2024-08-03T20:55:03.563 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-03T20:55:03.564 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":103,"btime":"2024-08-03T20:55:03:048134+0000","default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6439,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6835/531080160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6834","nonce":531080160},{"type":"v1","addr":"172.21.15.118:6835","nonce":531080160}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":84},{"gid":6454,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6837/2796882319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6836","nonce":2796882319},{"type":"v1","addr":"172.21.15.118:6837","nonce":2796882319}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":86},{"gid":6813,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6834/2686933890","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":2686933890},{"type":"v1","addr":"172.21.15.32:6834","nonce":2686933890}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":85}],"filesystems":[{"mdsmap":{"epoch":103,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-08-03T20:54:26.950518+0000","modified":"2024-08-03T20:55:03.048116+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":85,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7503},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7503":{"gid":7503,"name":"a","rank":0,"incarnation":103,"state":"up:replay","state_seq":1,"addr":"172.21.15.32:6835/1016338829","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6833","nonce":1016338829},{"type":"v1","addr":"172.21.15.32:6835","nonce":1016338829}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":6}]} 2024-08-03T20:55:03.564 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 103 2024-08-03T20:55:03.566 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:03.564+0000 7fa80de00640 1 -- 172.21.15.32:0/1337027708 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fa7e4077040 msgr2=0x7fa7e4079550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:55:03.566 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:03.564+0000 7fa80de00640 1 --2- 172.21.15.32:0/1337027708 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fa7e4077040 0x7fa7e4079550 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7fa7f8033000 tx=0x7fa7f8004730 comp rx=0 tx=0).stop 2024-08-03T20:55:03.566 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:03.564+0000 7fa80de00640 1 -- 172.21.15.32:0/1337027708 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa8080747b0 msgr2=0x7fa80813e490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:55:03.566 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:03.564+0000 7fa80de00640 1 --2- 172.21.15.32:0/1337027708 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa8080747b0 0x7fa80813e490 secure :-1 s=READY pgs=1126 cs=0 l=1 rev1=1 crypto rx=0x7fa800009ca0 tx=0x7fa800007590 comp rx=0 tx=0).stop 2024-08-03T20:55:03.566 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:03.564+0000 7fa80de00640 1 -- 172.21.15.32:0/1337027708 shutdown_connections 2024-08-03T20:55:03.566 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:03.564+0000 7fa80de00640 1 --2- 172.21.15.32:0/1337027708 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fa7e4077040 0x7fa7e4079550 unknown :-1 s=CLOSED pgs=467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:03.566 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:03.564+0000 7fa80de00640 1 --2- 172.21.15.32:0/1337027708 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa808073e60 0x7fa8080a9dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:03.566 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:03.564+0000 7fa80de00640 1 --2- 172.21.15.32:0/1337027708 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa8080747b0 0x7fa80813e490 unknown :-1 s=CLOSED pgs=1126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:03.566 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:03.564+0000 7fa80de00640 1 --2- 172.21.15.32:0/1337027708 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fa80806e930 0x7fa8080ae450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:03.567 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:03.564+0000 7fa80de00640 1 -- 172.21.15.32:0/1337027708 >> 172.21.15.32:0/1337027708 conn(0x7fa8080b1bd0 msgr2=0x7fa8080afb40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:55:03.567 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:03.564+0000 7fa80de00640 1 -- 172.21.15.32:0/1337027708 shutdown_connections 2024-08-03T20:55:03.567 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:03.564+0000 7fa80de00640 1 -- 172.21.15.32:0/1337027708 wait complete. 2024-08-03T20:55:03.589 DEBUG:tasks.ceph_test_case:wait_until_true: waiting (timeout=60 retry_count=0)... 2024-08-03T20:55:06.071 INFO:tasks.ceph.mds.a.smithi032.stderr:2024-08-03T20:55:06.069+0000 7fcadf600640 -1 log_channel(cluster) log [ERR] : dir 0x600 object missing on disk; some files may be lost (~mds0/stray0) 2024-08-03T20:55:06.071 INFO:tasks.ceph.mds.a.smithi032.stderr:2024-08-03T20:55:06.069+0000 7fcadf600640 -1 mds.0.damage notify_dirfrag Damage to fragment * of ino 0x600 is fatal because it is a system directory for this rank 2024-08-03T20:55:06.072 INFO:tasks.ceph.mon.a.smithi032.stderr:2024-08-03T20:55:06.070+0000 7f4e6d600640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-08-03T20:55:06.072 INFO:tasks.ceph.mon.a.smithi032.stderr:2024-08-03T20:55:06.070+0000 7f4e6d600640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 mds daemon damaged (MDS_DAMAGE) 2024-08-03T20:55:06.373 INFO:tasks.ceph.mds.a.smithi032.stderr:2024-08-03T20:55:06.371+0000 7fcae4600640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-08-03T20:55:07.987 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-08-03T20:55:07.993 DEBUG:teuthology.orchestra.run.smithi118:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-08-03T20:55:08.589 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-08-03T20:55:08.829 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:08.827+0000 7f7501200640 1 -- 172.21.15.32:0/1287254251 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f74fc0b93d0 msgr2=0x7f74fc0b39e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:55:08.830 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:08.827+0000 7f7501200640 1 --2- 172.21.15.32:0/1287254251 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f74fc0b93d0 0x7f74fc0b39e0 secure :-1 s=READY pgs=1127 cs=0 l=1 rev1=1 crypto rx=0x7f74e4005fd0 tx=0x7f74e402a5d0 comp rx=0 tx=0).stop 2024-08-03T20:55:08.830 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:08.827+0000 7f7501200640 1 -- 172.21.15.32:0/1287254251 shutdown_connections 2024-08-03T20:55:08.830 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:08.827+0000 7f7501200640 1 --2- 172.21.15.32:0/1287254251 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f74fc0b93d0 0x7f74fc0b39e0 unknown :-1 s=CLOSED pgs=1127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:08.830 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:08.827+0000 7f7501200640 1 --2- 172.21.15.32:0/1287254251 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f74fc0d4ba0 0x7f74fc0b3240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:08.830 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:08.827+0000 7f7501200640 1 -- 172.21.15.32:0/1287254251 >> 172.21.15.32:0/1287254251 conn(0x7f74fc0d4080 msgr2=0x7f74fc0d4490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:55:08.830 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:08.828+0000 7f7501200640 1 -- 172.21.15.32:0/1287254251 shutdown_connections 2024-08-03T20:55:08.830 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:08.828+0000 7f7501200640 1 -- 172.21.15.32:0/1287254251 wait complete. 2024-08-03T20:55:08.831 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:08.830+0000 7f7501200640 1 Processor -- start 2024-08-03T20:55:08.831 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:08.830+0000 7f7501200640 1 -- start start 2024-08-03T20:55:08.834 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:08.831+0000 7f7501200640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f74fc0b3f20 0x7f74fc09f620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:55:08.834 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:08.831+0000 7f7501200640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f74fc0b93d0 0x7f74fc1549e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:55:08.834 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:08.831+0000 7f7501200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f74fc0d4ba0 0x7f74fc151c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:55:08.834 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:08.831+0000 7f7501200640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f74fc0d08f0 con 0x7f74fc0d4ba0 2024-08-03T20:55:08.834 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:08.831+0000 7f7501200640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f74fc0d0770 con 0x7f74fc0b3f20 2024-08-03T20:55:08.834 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:08.831+0000 7f7501200640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7f74fc0d0a70 con 0x7f74fc0b93d0 2024-08-03T20:55:08.834 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:08.832+0000 7f74fbe00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f74fc0d4ba0 0x7f74fc151c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:55:08.834 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:08.832+0000 7f74fbe00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f74fc0d4ba0 0x7f74fc151c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:47978/0 (socket says 172.21.15.32:47978) 2024-08-03T20:55:08.834 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:08.832+0000 7f74fbe00640 1 -- 172.21.15.32:0/3735980188 learned_addr learned my addr 172.21.15.32:0/3735980188 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:55:08.834 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:08.832+0000 7f74fb400640 1 --2- 172.21.15.32:0/3735980188 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f74fc0b3f20 0x7f74fc09f620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:55:08.834 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:08.832+0000 7f74f3e00640 1 --2- 172.21.15.32:0/3735980188 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f74fc0b93d0 0x7f74fc1549e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:55:08.835 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:08.832+0000 7f74fbe00640 1 -- 172.21.15.32:0/3735980188 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f74fc0b93d0 msgr2=0x7f74fc1549e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:55:08.835 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:08.832+0000 7f74fbe00640 1 --2- 172.21.15.32:0/3735980188 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f74fc0b93d0 0x7f74fc1549e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:08.835 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:08.832+0000 7f74fbe00640 1 -- 172.21.15.32:0/3735980188 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f74fc0b3f20 msgr2=0x7f74fc09f620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:55:08.835 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:08.832+0000 7f74fbe00640 1 --2- 172.21.15.32:0/3735980188 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f74fc0b3f20 0x7f74fc09f620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:08.835 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:08.832+0000 7f74fbe00640 1 -- 172.21.15.32:0/3735980188 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f74fc0a0200 con 0x7f74fc0d4ba0 2024-08-03T20:55:08.835 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:08.832+0000 7f74f3e00640 1 --2- 172.21.15.32:0/3735980188 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f74fc0b93d0 0x7f74fc1549e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-03T20:55:08.835 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:08.832+0000 7f74fb400640 1 --2- 172.21.15.32:0/3735980188 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f74fc0b3f20 0x7f74fc09f620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-03T20:55:08.835 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:08.832+0000 7f74fbe00640 1 --2- 172.21.15.32:0/3735980188 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f74fc0d4ba0 0x7f74fc151c10 secure :-1 s=READY pgs=1128 cs=0 l=1 rev1=1 crypto rx=0x7f74f400c6c0 tx=0x7f74f400cb80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:55:08.836 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:08.833+0000 7f74f3400640 1 -- 172.21.15.32:0/3735980188 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74f4003b70 con 0x7f74fc0d4ba0 2024-08-03T20:55:08.836 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:08.833+0000 7f74f3400640 1 -- 172.21.15.32:0/3735980188 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f74f4003d30 con 0x7f74fc0d4ba0 2024-08-03T20:55:08.836 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:08.833+0000 7f74f3400640 1 -- 172.21.15.32:0/3735980188 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74f4005330 con 0x7f74fc0d4ba0 2024-08-03T20:55:08.836 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:08.833+0000 7f7501200640 1 -- 172.21.15.32:0/3735980188 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f74fc0a04e0 con 0x7f74fc0d4ba0 2024-08-03T20:55:08.836 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:08.833+0000 7f7501200640 1 -- 172.21.15.32:0/3735980188 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f74fc0a0aa0 con 0x7f74fc0d4ba0 2024-08-03T20:55:08.836 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:08.834+0000 7f7501200640 1 -- 172.21.15.32:0/3735980188 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f74fc09ff70 con 0x7f74fc0d4ba0 2024-08-03T20:55:08.837 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:08.835+0000 7f74f3400640 1 -- 172.21.15.32:0/3735980188 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f74f40054d0 con 0x7f74fc0d4ba0 2024-08-03T20:55:08.837 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:08.836+0000 7f74f3400640 1 --2- 172.21.15.32:0/3735980188 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f74d8077010 0x7f74d8079520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:55:08.838 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:08.836+0000 7f74f3400640 1 -- 172.21.15.32:0/3735980188 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 5967+0+0 (secure 0 0 0) 0x7f74f4098840 con 0x7f74fc0d4ba0 2024-08-03T20:55:08.838 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:08.837+0000 7f74fb400640 1 --2- 172.21.15.32:0/3735980188 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f74d8077010 0x7f74d8079520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:55:08.839 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:08.837+0000 7f74fb400640 1 --2- 172.21.15.32:0/3735980188 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f74d8077010 0x7f74d8079520 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f74ec005fd0 tx=0x7f74ec00f360 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:55:08.842 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:08.841+0000 7f74f3400640 1 -- 172.21.15.32:0/3735980188 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7f74f4061930 con 0x7f74fc0d4ba0 2024-08-03T20:55:09.085 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:09.083+0000 7f7501200640 1 -- 172.21.15.32:0/3735980188 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f74fc0d0f90 con 0x7f74fc0d4ba0 2024-08-03T20:55:09.085 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:09.084+0000 7f74f3400640 1 -- 172.21.15.32:0/3735980188 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 107 v107) ==== 95+0+4390 (secure 0 0 0) 0x7f74f4061230 con 0x7f74fc0d4ba0 2024-08-03T20:55:09.086 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-03T20:55:09.086 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":107,"btime":"2024-08-03T20:55:06:071734+0000","default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6439,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6835/531080160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6834","nonce":531080160},{"type":"v1","addr":"172.21.15.118:6835","nonce":531080160}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":84},{"gid":6454,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6837/2796882319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6836","nonce":2796882319},{"type":"v1","addr":"172.21.15.118:6837","nonce":2796882319}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":86},{"gid":6813,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6834/2686933890","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":2686933890},{"type":"v1","addr":"172.21.15.32:6834","nonce":2686933890}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":85}],"filesystems":[{"mdsmap":{"epoch":107,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-08-03T20:54:26.950518+0000","modified":"2024-08-03T20:55:06.071728+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":86,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":6}]} 2024-08-03T20:55:09.086 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 107 2024-08-03T20:55:09.088 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:09.086+0000 7f7501200640 1 -- 172.21.15.32:0/3735980188 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f74d8077010 msgr2=0x7f74d8079520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:55:09.088 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:09.086+0000 7f7501200640 1 --2- 172.21.15.32:0/3735980188 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f74d8077010 0x7f74d8079520 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f74ec005fd0 tx=0x7f74ec00f360 comp rx=0 tx=0).stop 2024-08-03T20:55:09.088 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:09.086+0000 7f7501200640 1 -- 172.21.15.32:0/3735980188 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f74fc0d4ba0 msgr2=0x7f74fc151c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:55:09.088 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:09.086+0000 7f7501200640 1 --2- 172.21.15.32:0/3735980188 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f74fc0d4ba0 0x7f74fc151c10 secure :-1 s=READY pgs=1128 cs=0 l=1 rev1=1 crypto rx=0x7f74f400c6c0 tx=0x7f74f400cb80 comp rx=0 tx=0).stop 2024-08-03T20:55:09.088 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:09.086+0000 7f7501200640 1 -- 172.21.15.32:0/3735980188 shutdown_connections 2024-08-03T20:55:09.089 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:09.086+0000 7f7501200640 1 --2- 172.21.15.32:0/3735980188 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f74d8077010 0x7f74d8079520 unknown :-1 s=CLOSED pgs=468 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:09.089 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:09.086+0000 7f7501200640 1 --2- 172.21.15.32:0/3735980188 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f74fc0d4ba0 0x7f74fc151c10 unknown :-1 s=CLOSED pgs=1128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:09.089 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:09.086+0000 7f7501200640 1 --2- 172.21.15.32:0/3735980188 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f74fc0b93d0 0x7f74fc1549e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:09.089 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:09.086+0000 7f7501200640 1 --2- 172.21.15.32:0/3735980188 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f74fc0b3f20 0x7f74fc09f620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:09.089 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:09.086+0000 7f7501200640 1 -- 172.21.15.32:0/3735980188 >> 172.21.15.32:0/3735980188 conn(0x7f74fc0d4080 msgr2=0x7f74fc0b86c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:55:09.089 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:09.086+0000 7f7501200640 1 -- 172.21.15.32:0/3735980188 shutdown_connections 2024-08-03T20:55:09.089 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:09.087+0000 7f7501200640 1 -- 172.21.15.32:0/3735980188 wait complete. 2024-08-03T20:55:09.112 DEBUG:tasks.ceph_test_case:wait_until_true: success in 5s and 0 retries 2024-08-03T20:55:09.112 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-08-03T20:55:09.341 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:09.339+0000 7f1811200640 1 -- 172.21.15.32:0/1832084226 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f180c073e60 msgr2=0x7f180c074270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:55:09.341 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:09.339+0000 7f1811200640 1 --2- 172.21.15.32:0/1832084226 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f180c073e60 0x7f180c074270 secure :-1 s=READY pgs=1129 cs=0 l=1 rev1=1 crypto rx=0x7f180c0b2f80 tx=0x7f17f402a560 comp rx=0 tx=0).stop 2024-08-03T20:55:09.341 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:09.339+0000 7f1811200640 1 -- 172.21.15.32:0/1832084226 shutdown_connections 2024-08-03T20:55:09.342 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:09.339+0000 7f1811200640 1 --2- 172.21.15.32:0/1832084226 >> v2:172.21.15.118:3300/0 conn(0x7f180c0747b0 0x7f180c0cc8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:09.342 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:09.339+0000 7f1811200640 1 --2- 172.21.15.32:0/1832084226 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f180c073e60 0x7f180c074270 unknown :-1 s=CLOSED pgs=1129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:09.342 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:09.339+0000 7f1811200640 1 --2- 172.21.15.32:0/1832084226 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f180c06e930 0x7f180c144f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:09.342 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:09.339+0000 7f1811200640 1 -- 172.21.15.32:0/1832084226 >> 172.21.15.32:0/1832084226 conn(0x7f180c0b1bd0 msgr2=0x7f180c0b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:55:09.342 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:09.339+0000 7f1811200640 1 -- 172.21.15.32:0/1832084226 shutdown_connections 2024-08-03T20:55:09.342 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:09.340+0000 7f1811200640 1 -- 172.21.15.32:0/1832084226 wait complete. 2024-08-03T20:55:09.343 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:09.342+0000 7f1811200640 1 Processor -- start 2024-08-03T20:55:09.343 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:09.342+0000 7f1811200640 1 -- start start 2024-08-03T20:55:09.345 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:09.343+0000 7f1811200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f180c06e930 0x7f180c1f5070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:55:09.345 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:09.343+0000 7f1811200640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f180c073e60 0x7f180c1e4c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:55:09.345 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:09.343+0000 7f1811200640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f180c0747b0 0x7f180c0b0b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:55:09.345 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:09.343+0000 7f1811200640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f180c0cf9c0 con 0x7f180c06e930 2024-08-03T20:55:09.346 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:09.343+0000 7f1811200640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f180c0cf840 con 0x7f180c073e60 2024-08-03T20:55:09.346 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:09.343+0000 7f1811200640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7f180c0cce20 con 0x7f180c0747b0 2024-08-03T20:55:09.346 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:09.344+0000 7f1809e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f180c06e930 0x7f180c1f5070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:55:09.346 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:09.344+0000 7f1809e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f180c06e930 0x7f180c1f5070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:47988/0 (socket says 172.21.15.32:47988) 2024-08-03T20:55:09.346 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:09.344+0000 7f1809e00640 1 -- 172.21.15.32:0/781592624 learned_addr learned my addr 172.21.15.32:0/781592624 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:55:09.346 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:09.344+0000 7f1801400640 1 --2- 172.21.15.32:0/781592624 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f180c073e60 0x7f180c1e4c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:55:09.346 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:09.344+0000 7f180a800640 1 --2- 172.21.15.32:0/781592624 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f180c0747b0 0x7f180c0b0b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:55:09.346 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:09.344+0000 7f1809e00640 1 -- 172.21.15.32:0/781592624 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f180c0747b0 msgr2=0x7f180c0b0b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:55:09.346 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:09.344+0000 7f1809e00640 1 --2- 172.21.15.32:0/781592624 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f180c0747b0 0x7f180c0b0b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:09.346 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:09.344+0000 7f1809e00640 1 -- 172.21.15.32:0/781592624 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f180c073e60 msgr2=0x7f180c1e4c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:55:09.346 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:09.344+0000 7f1809e00640 1 --2- 172.21.15.32:0/781592624 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f180c073e60 0x7f180c1e4c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:09.346 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:09.344+0000 7f1809e00640 1 -- 172.21.15.32:0/781592624 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f180c1f55b0 con 0x7f180c06e930 2024-08-03T20:55:09.348 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:09.344+0000 7f1809e00640 1 --2- 172.21.15.32:0/781592624 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f180c06e930 0x7f180c1f5070 secure :-1 s=READY pgs=1130 cs=0 l=1 rev1=1 crypto rx=0x7f17fc00d700 tx=0x7f17fc00dbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:55:09.348 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:09.345+0000 7f1802a00640 1 -- 172.21.15.32:0/781592624 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17fc017070 con 0x7f180c06e930 2024-08-03T20:55:09.348 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:09.345+0000 7f1802a00640 1 -- 172.21.15.32:0/781592624 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f17fc005310 con 0x7f180c06e930 2024-08-03T20:55:09.348 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:09.345+0000 7f1802a00640 1 -- 172.21.15.32:0/781592624 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17fc009820 con 0x7f180c06e930 2024-08-03T20:55:09.348 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:09.345+0000 7f1811200640 1 -- 172.21.15.32:0/781592624 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f180c15cb80 con 0x7f180c06e930 2024-08-03T20:55:09.348 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:09.345+0000 7f1811200640 1 -- 172.21.15.32:0/781592624 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f180c15d140 con 0x7f180c06e930 2024-08-03T20:55:09.348 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:09.346+0000 7f1811200640 1 -- 172.21.15.32:0/781592624 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f180c0728b0 con 0x7f180c06e930 2024-08-03T20:55:09.349 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:09.347+0000 7f1802a00640 1 -- 172.21.15.32:0/781592624 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f17fc0099c0 con 0x7f180c06e930 2024-08-03T20:55:09.349 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:09.348+0000 7f1802a00640 1 --2- 172.21.15.32:0/781592624 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f17e8077040 0x7f17e8079550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:55:09.350 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:09.348+0000 7f1802a00640 1 -- 172.21.15.32:0/781592624 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 5967+0+0 (secure 0 0 0) 0x7f17fc098b00 con 0x7f180c06e930 2024-08-03T20:55:09.350 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:09.349+0000 7f1801400640 1 --2- 172.21.15.32:0/781592624 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f17e8077040 0x7f17e8079550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:55:09.351 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:09.350+0000 7f1801400640 1 --2- 172.21.15.32:0/781592624 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f17e8077040 0x7f17e8079550 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7f17f4033000 tx=0x7f17f4004730 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:55:09.354 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:09.353+0000 7f1802a00640 1 -- 172.21.15.32:0/781592624 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7f180c0728b0 con 0x7f180c06e930 2024-08-03T20:55:09.596 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:09.595+0000 7f1811200640 1 -- 172.21.15.32:0/781592624 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f180c074270 con 0x7f180c06e930 2024-08-03T20:55:09.597 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:09.595+0000 7f1802a00640 1 -- 172.21.15.32:0/781592624 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 107 v107) ==== 95+0+4390 (secure 0 0 0) 0x7f17fc098e20 con 0x7f180c06e930 2024-08-03T20:55:09.597 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-03T20:55:09.597 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":107,"btime":"2024-08-03T20:55:06:071734+0000","default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6439,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6835/531080160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6834","nonce":531080160},{"type":"v1","addr":"172.21.15.118:6835","nonce":531080160}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":84},{"gid":6454,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6837/2796882319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6836","nonce":2796882319},{"type":"v1","addr":"172.21.15.118:6837","nonce":2796882319}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":86},{"gid":6813,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6834/2686933890","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":2686933890},{"type":"v1","addr":"172.21.15.32:6834","nonce":2686933890}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":85}],"filesystems":[{"mdsmap":{"epoch":107,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-08-03T20:54:26.950518+0000","modified":"2024-08-03T20:55:06.071728+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":86,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":6}]} 2024-08-03T20:55:09.597 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 107 2024-08-03T20:55:09.599 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:09.597+0000 7f1811200640 1 -- 172.21.15.32:0/781592624 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f17e8077040 msgr2=0x7f17e8079550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:55:09.599 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:09.597+0000 7f1811200640 1 --2- 172.21.15.32:0/781592624 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f17e8077040 0x7f17e8079550 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7f17f4033000 tx=0x7f17f4004730 comp rx=0 tx=0).stop 2024-08-03T20:55:09.599 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:09.597+0000 7f1811200640 1 -- 172.21.15.32:0/781592624 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f180c06e930 msgr2=0x7f180c1f5070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:55:09.599 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:09.597+0000 7f1811200640 1 --2- 172.21.15.32:0/781592624 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f180c06e930 0x7f180c1f5070 secure :-1 s=READY pgs=1130 cs=0 l=1 rev1=1 crypto rx=0x7f17fc00d700 tx=0x7f17fc00dbc0 comp rx=0 tx=0).stop 2024-08-03T20:55:09.599 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:09.597+0000 7f1811200640 1 -- 172.21.15.32:0/781592624 shutdown_connections 2024-08-03T20:55:09.599 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:09.597+0000 7f1811200640 1 --2- 172.21.15.32:0/781592624 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f17e8077040 0x7f17e8079550 unknown :-1 s=CLOSED pgs=469 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:09.600 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:09.597+0000 7f1811200640 1 --2- 172.21.15.32:0/781592624 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f180c073e60 0x7f180c1e4c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:09.600 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:09.597+0000 7f1811200640 1 --2- 172.21.15.32:0/781592624 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f180c0747b0 0x7f180c0b0b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:09.600 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:09.597+0000 7f1811200640 1 --2- 172.21.15.32:0/781592624 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f180c06e930 0x7f180c1f5070 unknown :-1 s=CLOSED pgs=1130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:09.600 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:09.597+0000 7f1811200640 1 -- 172.21.15.32:0/781592624 >> 172.21.15.32:0/781592624 conn(0x7f180c0b1bd0 msgr2=0x7f180c06f930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:55:09.600 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:09.598+0000 7f1811200640 1 -- 172.21.15.32:0/781592624 shutdown_connections 2024-08-03T20:55:09.600 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:09.598+0000 7f1811200640 1 -- 172.21.15.32:0/781592624 wait complete. 2024-08-03T20:55:09.622 DEBUG:tasks.ceph_test_case:wait_until_true: success in 0s and 0 retries 2024-08-03T20:55:09.622 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-08-03T20:55:09.853 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:09.851+0000 7f0713e00640 1 -- 172.21.15.32:0/613751408 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f06f8003a30 msgr2=0x7f06f8003ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:55:09.853 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:09.851+0000 7f0713e00640 1 --2- 172.21.15.32:0/613751408 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f06f8003a30 0x7f06f8003ea0 secure :-1 s=READY pgs=1132 cs=0 l=1 rev1=1 crypto rx=0x7f0704009b50 tx=0x7f06f8005830 comp rx=0 tx=0).stop 2024-08-03T20:55:09.853 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:09.851+0000 7f0713400640 1 -- 172.21.15.32:0/613751408 reap_dead start 2024-08-03T20:55:09.854 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:09.851+0000 7f0713e00640 1 -- 172.21.15.32:0/613751408 shutdown_connections 2024-08-03T20:55:09.854 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:09.851+0000 7f0713e00640 1 -- 172.21.15.32:0/613751408 >> 172.21.15.32:0/613751408 conn(0x7f071408d7a0 msgr2=0x7f071409d3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:55:09.854 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:09.851+0000 7f0713e00640 1 -- 172.21.15.32:0/613751408 shutdown_connections 2024-08-03T20:55:09.854 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:09.851+0000 7f0713e00640 1 -- 172.21.15.32:0/613751408 wait complete. 2024-08-03T20:55:09.855 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:09.854+0000 7f0713e00640 1 Processor -- start 2024-08-03T20:55:09.855 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:09.854+0000 7f0713e00640 1 -- start start 2024-08-03T20:55:09.859 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:09.855+0000 7f0713e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f06f8003a30 0x7f07140cb180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:55:09.859 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:09.855+0000 7f0713e00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f07141f9510 0x7f07140b7fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:55:09.859 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:09.855+0000 7f0713e00640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f07141556c0 0x7f07140c8f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:55:09.859 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:09.855+0000 7f0713e00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f0714161b80 con 0x7f06f8003a30 2024-08-03T20:55:09.859 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:09.855+0000 7f0713e00640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f07140d48a0 con 0x7f07141f9510 2024-08-03T20:55:09.860 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:09.855+0000 7f0713e00640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7f0714161d00 con 0x7f07141556c0 2024-08-03T20:55:09.860 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:09.855+0000 7f0712a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f06f8003a30 0x7f07140cb180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:55:09.860 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:09.855+0000 7f0712a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f06f8003a30 0x7f07140cb180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:50346/0 (socket says 172.21.15.32:50346) 2024-08-03T20:55:09.860 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:09.855+0000 7f0712a00640 1 -- 172.21.15.32:0/3412453301 learned_addr learned my addr 172.21.15.32:0/3412453301 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:55:09.860 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:09.856+0000 7f0713400640 1 --2- 172.21.15.32:0/3412453301 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f07141556c0 0x7f07140c8f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:55:09.860 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:09.856+0000 7f070a000640 1 --2- 172.21.15.32:0/3412453301 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f07141f9510 0x7f07140b7fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:55:09.860 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:09.856+0000 7f0712a00640 1 -- 172.21.15.32:0/3412453301 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f07141556c0 msgr2=0x7f07140c8f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:55:09.860 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:09.856+0000 7f0712a00640 1 --2- 172.21.15.32:0/3412453301 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f07141556c0 0x7f07140c8f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:09.860 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:09.856+0000 7f0712a00640 1 -- 172.21.15.32:0/3412453301 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f07141f9510 msgr2=0x7f07140b7fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:55:09.860 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:09.856+0000 7f0712a00640 1 --2- 172.21.15.32:0/3412453301 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f07141f9510 0x7f07140b7fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:09.860 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:09.856+0000 7f0712a00640 1 -- 172.21.15.32:0/3412453301 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f07141f9920 con 0x7f06f8003a30 2024-08-03T20:55:09.860 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:09.856+0000 7f0713400640 1 --2- 172.21.15.32:0/3412453301 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f07141556c0 0x7f07140c8f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-03T20:55:09.860 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:09.856+0000 7f0712a00640 1 --2- 172.21.15.32:0/3412453301 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f06f8003a30 0x7f07140cb180 secure :-1 s=READY pgs=1133 cs=0 l=1 rev1=1 crypto rx=0x7f0704036040 tx=0x7f0704033670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:55:09.861 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:09.856+0000 7f070b400640 1 -- 172.21.15.32:0/3412453301 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0704038040 con 0x7f06f8003a30 2024-08-03T20:55:09.861 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:09.856+0000 7f070b400640 1 -- 172.21.15.32:0/3412453301 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f0704004020 con 0x7f06f8003a30 2024-08-03T20:55:09.861 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:09.857+0000 7f0713e00640 1 -- 172.21.15.32:0/3412453301 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f07141f8a70 con 0x7f06f8003a30 2024-08-03T20:55:09.861 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:09.857+0000 7f070b400640 1 -- 172.21.15.32:0/3412453301 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0704004470 con 0x7f06f8003a30 2024-08-03T20:55:09.861 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:09.857+0000 7f0713e00640 1 -- 172.21.15.32:0/3412453301 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f07141f8d10 con 0x7f06f8003a30 2024-08-03T20:55:09.861 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:09.858+0000 7f0713e00640 1 -- 172.21.15.32:0/3412453301 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0714162080 con 0x7f06f8003a30 2024-08-03T20:55:09.861 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:09.859+0000 7f070b400640 1 -- 172.21.15.32:0/3412453301 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f0704045020 con 0x7f06f8003a30 2024-08-03T20:55:09.862 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:09.860+0000 7f070b400640 1 --2- 172.21.15.32:0/3412453301 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f06f0077010 0x7f06f0079520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:55:09.862 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:09.861+0000 7f070b400640 1 -- 172.21.15.32:0/3412453301 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 5967+0+0 (secure 0 0 0) 0x7f07040b95a0 con 0x7f06f8003a30 2024-08-03T20:55:09.862 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:09.861+0000 7f070a000640 1 --2- 172.21.15.32:0/3412453301 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f06f0077010 0x7f06f0079520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:55:09.863 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:09.862+0000 7f070a000640 1 --2- 172.21.15.32:0/3412453301 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f06f0077010 0x7f06f0079520 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7f06fc001860 tx=0x7f06fc001cf0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:55:09.866 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:09.865+0000 7f070b400640 1 -- 172.21.15.32:0/3412453301 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7f07141f9920 con 0x7f06f8003a30 2024-08-03T20:55:10.109 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:10.107+0000 7f0713e00640 1 -- 172.21.15.32:0/3412453301 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f0714068c20 con 0x7f06f8003a30 2024-08-03T20:55:10.109 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:10.107+0000 7f070b400640 1 -- 172.21.15.32:0/3412453301 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 107 v107) ==== 95+0+4390 (secure 0 0 0) 0x7f0704082600 con 0x7f06f8003a30 2024-08-03T20:55:10.110 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-03T20:55:10.110 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":107,"btime":"2024-08-03T20:55:06:071734+0000","default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6439,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6835/531080160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6834","nonce":531080160},{"type":"v1","addr":"172.21.15.118:6835","nonce":531080160}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":84},{"gid":6454,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6837/2796882319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6836","nonce":2796882319},{"type":"v1","addr":"172.21.15.118:6837","nonce":2796882319}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":86},{"gid":6813,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6834/2686933890","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":2686933890},{"type":"v1","addr":"172.21.15.32:6834","nonce":2686933890}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":85}],"filesystems":[{"mdsmap":{"epoch":107,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-08-03T20:54:26.950518+0000","modified":"2024-08-03T20:55:06.071728+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":86,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":6}]} 2024-08-03T20:55:10.110 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 107 2024-08-03T20:55:10.112 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:10.110+0000 7f0713e00640 1 -- 172.21.15.32:0/3412453301 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f06f0077010 msgr2=0x7f06f0079520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:55:10.113 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:10.110+0000 7f0713e00640 1 --2- 172.21.15.32:0/3412453301 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f06f0077010 0x7f06f0079520 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7f06fc001860 tx=0x7f06fc001cf0 comp rx=0 tx=0).stop 2024-08-03T20:55:10.113 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:10.110+0000 7f0713e00640 1 -- 172.21.15.32:0/3412453301 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f06f8003a30 msgr2=0x7f07140cb180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:55:10.113 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:10.110+0000 7f0713e00640 1 --2- 172.21.15.32:0/3412453301 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f06f8003a30 0x7f07140cb180 secure :-1 s=READY pgs=1133 cs=0 l=1 rev1=1 crypto rx=0x7f0704036040 tx=0x7f0704033670 comp rx=0 tx=0).stop 2024-08-03T20:55:10.113 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:10.110+0000 7f0713e00640 1 -- 172.21.15.32:0/3412453301 shutdown_connections 2024-08-03T20:55:10.113 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:10.110+0000 7f0713e00640 1 --2- 172.21.15.32:0/3412453301 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f06f0077010 0x7f06f0079520 unknown :-1 s=CLOSED pgs=470 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:10.113 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:10.110+0000 7f0713e00640 1 --2- 172.21.15.32:0/3412453301 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f07141f9510 0x7f07140b7fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:10.114 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:10.110+0000 7f0713e00640 1 --2- 172.21.15.32:0/3412453301 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f07141556c0 0x7f07140c8f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:10.114 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:10.110+0000 7f0713e00640 1 --2- 172.21.15.32:0/3412453301 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f06f8003a30 0x7f07140cb180 unknown :-1 s=CLOSED pgs=1133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:10.114 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:10.110+0000 7f0713e00640 1 -- 172.21.15.32:0/3412453301 >> 172.21.15.32:0/3412453301 conn(0x7f071408d7a0 msgr2=0x7f071415f380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:55:10.114 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:10.110+0000 7f0713e00640 1 -- 172.21.15.32:0/3412453301 shutdown_connections 2024-08-03T20:55:10.114 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:10.110+0000 7f0713e00640 1 -- 172.21.15.32:0/3412453301 wait complete. 2024-08-03T20:55:10.136 INFO:tasks.cephfs.test_damage:Result: Mutation 'Delete 600.00000000' led to DAMAGED state 2024-08-03T20:55:10.137 INFO:tasks.cephfs.test_damage:Applying mutation 'Delete 1.00000000.inode' 2024-08-03T20:55:10.137 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-08-03T20:55:10.137 DEBUG:teuthology.orchestra.run.smithi032:> dd if=/proc/self/mounts of=/dev/stdout 2024-08-03T20:55:10.173 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053453,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6499208k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 none /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249604k,nr_inodes=812401,mode=700,uid=1000,gid=1316 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6499208k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-08-03T20:55:10.173 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi032.front.sepia.ceph.com 2024-08-03T20:55:10.173 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-08-03T20:55:10.173 DEBUG:teuthology.orchestra.run.smithi032:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-08-03T20:55:10.248 INFO:teuthology.orchestra.run.smithi032.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.0': No such file or directory 2024-08-03T20:55:10.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-03T20:55:10.249 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-08-03T20:55:10.518 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:10.516+0000 7ffab9200640 1 -- 172.21.15.32:0/1815076077 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ffab40747b0 msgr2=0x7ffab40cc8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:55:10.518 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:10.516+0000 7ffab9200640 1 --2- 172.21.15.32:0/1815076077 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ffab40747b0 0x7ffab40cc8e0 secure :-1 s=READY pgs=1134 cs=0 l=1 rev1=1 crypto rx=0x7ffab40b2860 tx=0x7ffaac02aa30 comp rx=0 tx=0).stop 2024-08-03T20:55:10.518 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:10.516+0000 7ffab9200640 1 -- 172.21.15.32:0/1815076077 shutdown_connections 2024-08-03T20:55:10.519 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:10.516+0000 7ffab9200640 1 --2- 172.21.15.32:0/1815076077 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ffab40747b0 0x7ffab40cc8e0 unknown :-1 s=CLOSED pgs=1134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:10.519 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:10.516+0000 7ffab9200640 1 --2- 172.21.15.32:0/1815076077 >> v2:172.21.15.118:3300/0 conn(0x7ffab4073e60 0x7ffab4074270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:10.519 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:10.516+0000 7ffab9200640 1 --2- 172.21.15.32:0/1815076077 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7ffab406e930 0x7ffab4144f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:10.519 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:10.516+0000 7ffab9200640 1 -- 172.21.15.32:0/1815076077 >> 172.21.15.32:0/1815076077 conn(0x7ffab40b1bd0 msgr2=0x7ffab40b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:55:10.519 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:10.516+0000 7ffab9200640 1 -- 172.21.15.32:0/1815076077 shutdown_connections 2024-08-03T20:55:10.519 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:10.516+0000 7ffab9200640 1 -- 172.21.15.32:0/1815076077 wait complete. 2024-08-03T20:55:10.520 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:10.518+0000 7ffab9200640 1 Processor -- start 2024-08-03T20:55:10.520 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:10.519+0000 7ffab9200640 1 -- start start 2024-08-03T20:55:10.523 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:10.520+0000 7ffab9200640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ffab406e930 0x7ffab4141590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:55:10.523 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:10.520+0000 7ffab9200640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7ffab4073e60 0x7ffab4141ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:55:10.523 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:10.520+0000 7ffab9200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ffab40747b0 0x7ffab41f8ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:55:10.524 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:10.520+0000 7ffab9200640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7ffab40cf9c0 con 0x7ffab40747b0 2024-08-03T20:55:10.524 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:10.520+0000 7ffab9200640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7ffab40cf840 con 0x7ffab406e930 2024-08-03T20:55:10.524 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:10.520+0000 7ffab9200640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7ffab40cce20 con 0x7ffab4073e60 2024-08-03T20:55:10.524 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:10.520+0000 7ffab2800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ffab40747b0 0x7ffab41f8ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:55:10.524 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:10.520+0000 7ffab2800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ffab40747b0 0x7ffab41f8ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:50374/0 (socket says 172.21.15.32:50374) 2024-08-03T20:55:10.524 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:10.520+0000 7ffab2800640 1 -- 172.21.15.32:0/1365804137 learned_addr learned my addr 172.21.15.32:0/1365804137 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:55:10.524 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:10.520+0000 7ffab1e00640 1 --2- 172.21.15.32:0/1365804137 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ffab406e930 0x7ffab4141590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:55:10.524 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:10.520+0000 7ffaabe00640 1 --2- 172.21.15.32:0/1365804137 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7ffab4073e60 0x7ffab4141ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:55:10.524 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:10.521+0000 7ffab2800640 1 -- 172.21.15.32:0/1365804137 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7ffab4073e60 msgr2=0x7ffab4141ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:55:10.524 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:10.521+0000 7ffab2800640 1 --2- 172.21.15.32:0/1365804137 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7ffab4073e60 0x7ffab4141ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:10.524 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:10.521+0000 7ffab2800640 1 -- 172.21.15.32:0/1365804137 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ffab406e930 msgr2=0x7ffab4141590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:55:10.524 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:10.521+0000 7ffab2800640 1 --2- 172.21.15.32:0/1365804137 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ffab406e930 0x7ffab4141590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:10.525 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:10.521+0000 7ffab2800640 1 -- 172.21.15.32:0/1365804137 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffab4143b80 con 0x7ffab40747b0 2024-08-03T20:55:10.525 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:10.521+0000 7ffab2800640 1 --2- 172.21.15.32:0/1365804137 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ffab40747b0 0x7ffab41f8ae0 secure :-1 s=READY pgs=1135 cs=0 l=1 rev1=1 crypto rx=0x7ffaac009d60 tx=0x7ffaac003f10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:55:10.525 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:10.521+0000 7ffaaa000640 1 -- 172.21.15.32:0/1365804137 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffaac033050 con 0x7ffab40747b0 2024-08-03T20:55:10.525 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:10.521+0000 7ffaaa000640 1 -- 172.21.15.32:0/1365804137 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7ffaac004450 con 0x7ffab40747b0 2024-08-03T20:55:10.525 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:10.521+0000 7ffaaa000640 1 -- 172.21.15.32:0/1365804137 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffaac038450 con 0x7ffab40747b0 2024-08-03T20:55:10.525 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:10.522+0000 7ffab9200640 1 -- 172.21.15.32:0/1365804137 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffab4143e60 con 0x7ffab40747b0 2024-08-03T20:55:10.525 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:10.522+0000 7ffab9200640 1 -- 172.21.15.32:0/1365804137 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffab4144420 con 0x7ffab40747b0 2024-08-03T20:55:10.525 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:10.523+0000 7ffab9200640 1 -- 172.21.15.32:0/1365804137 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ffab4144800 con 0x7ffab40747b0 2024-08-03T20:55:10.526 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:10.524+0000 7ffaaa000640 1 -- 172.21.15.32:0/1365804137 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7ffaac036070 con 0x7ffab40747b0 2024-08-03T20:55:10.526 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:10.525+0000 7ffaaa000640 1 --2- 172.21.15.32:0/1365804137 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7ffa90077010 0x7ffa90079520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:55:10.526 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:10.525+0000 7ffaaa000640 1 -- 172.21.15.32:0/1365804137 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 5967+0+0 (secure 0 0 0) 0x7ffaac0b9c90 con 0x7ffab40747b0 2024-08-03T20:55:10.527 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:10.525+0000 7ffab1e00640 1 --2- 172.21.15.32:0/1365804137 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7ffa90077010 0x7ffa90079520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:55:10.527 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:10.526+0000 7ffab1e00640 1 --2- 172.21.15.32:0/1365804137 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7ffa90077010 0x7ffa90079520 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7ffaa4009b30 tx=0x7ffaa4005f90 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:55:10.531 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:10.529+0000 7ffaaa000640 1 -- 172.21.15.32:0/1365804137 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7ffab4144800 con 0x7ffab40747b0 2024-08-03T20:55:10.774 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:10.772+0000 7ffab9200640 1 -- 172.21.15.32:0/1365804137 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7ffab4143910 con 0x7ffab40747b0 2024-08-03T20:55:10.825 INFO:tasks.ceph.mds.a.smithi032.stderr: -53> 2024-08-03T20:55:06.069+0000 7fcadf600640 -1 log_channel(cluster) log [ERR] : dir 0x600 object missing on disk; some files may be lost (~mds0/stray0) 2024-08-03T20:55:10.825 INFO:tasks.ceph.mds.a.smithi032.stderr: -51> 2024-08-03T20:55:06.069+0000 7fcadf600640 -1 mds.0.damage notify_dirfrag Damage to fragment * of ino 0x600 is fatal because it is a system directory for this rank 2024-08-03T20:55:10.825 INFO:tasks.ceph.mds.a.smithi032.stderr: -37> 2024-08-03T20:55:06.371+0000 7fcae4600640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-08-03T20:55:10.940 INFO:tasks.ceph.mds.a.smithi032.stdout:starting mds.a at 2024-08-03T20:55:11.086 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:11.084+0000 7ffaaa000640 1 -- 172.21.15.32:0/1365804137 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v108) ==== 161+0+0 (secure 0 0 0) 0x7ffaac0826b0 con 0x7ffab40747b0 2024-08-03T20:55:11.086 INFO:teuthology.orchestra.run.smithi032.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-08-03T20:55:11.089 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:11.087+0000 7ffab9200640 1 -- 172.21.15.32:0/1365804137 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7ffa90077010 msgr2=0x7ffa90079520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:55:11.089 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:11.087+0000 7ffab9200640 1 --2- 172.21.15.32:0/1365804137 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7ffa90077010 0x7ffa90079520 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7ffaa4009b30 tx=0x7ffaa4005f90 comp rx=0 tx=0).stop 2024-08-03T20:55:11.089 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:11.087+0000 7ffab9200640 1 -- 172.21.15.32:0/1365804137 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ffab40747b0 msgr2=0x7ffab41f8ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:55:11.089 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:11.087+0000 7ffab9200640 1 --2- 172.21.15.32:0/1365804137 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ffab40747b0 0x7ffab41f8ae0 secure :-1 s=READY pgs=1135 cs=0 l=1 rev1=1 crypto rx=0x7ffaac009d60 tx=0x7ffaac003f10 comp rx=0 tx=0).stop 2024-08-03T20:55:11.090 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:11.087+0000 7ffab9200640 1 -- 172.21.15.32:0/1365804137 shutdown_connections 2024-08-03T20:55:11.090 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:11.087+0000 7ffab9200640 1 --2- 172.21.15.32:0/1365804137 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7ffa90077010 0x7ffa90079520 unknown :-1 s=CLOSED pgs=471 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:11.090 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:11.087+0000 7ffab9200640 1 --2- 172.21.15.32:0/1365804137 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ffab40747b0 0x7ffab41f8ae0 unknown :-1 s=CLOSED pgs=1135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:11.090 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:11.087+0000 7ffab9200640 1 --2- 172.21.15.32:0/1365804137 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7ffab4073e60 0x7ffab4141ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:11.090 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:11.087+0000 7ffab9200640 1 --2- 172.21.15.32:0/1365804137 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ffab406e930 0x7ffab4141590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:11.090 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:11.087+0000 7ffab9200640 1 -- 172.21.15.32:0/1365804137 >> 172.21.15.32:0/1365804137 conn(0x7ffab40b1bd0 msgr2=0x7ffab40b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:55:11.090 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:11.087+0000 7ffab9200640 1 -- 172.21.15.32:0/1365804137 shutdown_connections 2024-08-03T20:55:11.090 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:11.088+0000 7ffab9200640 1 -- 172.21.15.32:0/1365804137 wait complete. 2024-08-03T20:55:11.113 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mds repaired 0 2024-08-03T20:55:11.344 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:11.342+0000 7fed12e00640 1 -- 172.21.15.32:0/1486107803 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fed0c0b38b0 msgr2=0x7fecf0003c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:55:11.344 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:11.342+0000 7fed12e00640 1 --2- 172.21.15.32:0/1486107803 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fed0c0b38b0 0x7fecf0003c10 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fecfc009b50 tx=0x7fecf0005830 comp rx=0 tx=0).stop 2024-08-03T20:55:11.344 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:11.342+0000 7fed10e00640 1 -- 172.21.15.32:0/1486107803 reap_dead start 2024-08-03T20:55:11.344 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:11.342+0000 7fed12e00640 1 -- 172.21.15.32:0/1486107803 shutdown_connections 2024-08-03T20:55:11.345 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:11.342+0000 7fed12e00640 1 -- 172.21.15.32:0/1486107803 >> 172.21.15.32:0/1486107803 conn(0x7fed0c0b1bd0 msgr2=0x7fed0c0b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:55:11.345 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:11.342+0000 7fed12e00640 1 -- 172.21.15.32:0/1486107803 shutdown_connections 2024-08-03T20:55:11.345 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:11.342+0000 7fed12e00640 1 -- 172.21.15.32:0/1486107803 wait complete. 2024-08-03T20:55:11.346 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:11.345+0000 7fed12e00640 1 Processor -- start 2024-08-03T20:55:11.346 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:11.345+0000 7fed12e00640 1 -- start start 2024-08-03T20:55:11.349 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:11.346+0000 7fed12e00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fed0c0b38b0 0x7fed0c0cb400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:55:11.349 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:11.346+0000 7fed12e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fed0c06f2a0 0x7fed0c06f6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:55:11.349 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:11.346+0000 7fed12e00640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fed0c15b910 0x7fed0c15bd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:55:11.349 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:11.346+0000 7fed12e00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fed0c0ce800 con 0x7fed0c06f2a0 2024-08-03T20:55:11.349 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:11.346+0000 7fed12e00640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7fed0c0ce680 con 0x7fed0c0b38b0 2024-08-03T20:55:11.350 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:11.346+0000 7fed12e00640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7fed0c0ce980 con 0x7fed0c15b910 2024-08-03T20:55:11.350 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:11.346+0000 7fed0b400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fed0c06f2a0 0x7fed0c06f6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:55:11.350 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:11.347+0000 7fed0b400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fed0c06f2a0 0x7fed0c06f6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:50404/0 (socket says 172.21.15.32:50404) 2024-08-03T20:55:11.350 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:11.347+0000 7fed0b400640 1 -- 172.21.15.32:0/1650848064 learned_addr learned my addr 172.21.15.32:0/1650848064 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:55:11.350 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:11.347+0000 7fed0be00640 1 --2- 172.21.15.32:0/1650848064 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fed0c0b38b0 0x7fed0c0cb400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:55:11.350 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:11.347+0000 7fed10e00640 1 --2- 172.21.15.32:0/1650848064 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fed0c15b910 0x7fed0c15bd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:55:11.350 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:11.347+0000 7fed0b400640 1 -- 172.21.15.32:0/1650848064 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fed0c15b910 msgr2=0x7fed0c15bd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:55:11.350 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:11.347+0000 7fed0b400640 1 --2- 172.21.15.32:0/1650848064 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fed0c15b910 0x7fed0c15bd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:11.350 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:11.347+0000 7fed0b400640 1 -- 172.21.15.32:0/1650848064 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fed0c0b38b0 msgr2=0x7fed0c0cb400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:55:11.350 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:11.347+0000 7fed0b400640 1 --2- 172.21.15.32:0/1650848064 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fed0c0b38b0 0x7fed0c0cb400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:11.350 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:11.347+0000 7fed0b400640 1 -- 172.21.15.32:0/1650848064 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fed0c15c260 con 0x7fed0c06f2a0 2024-08-03T20:55:11.350 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:11.347+0000 7fed0b400640 1 --2- 172.21.15.32:0/1650848064 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fed0c06f2a0 0x7fed0c06f6b0 secure :-1 s=READY pgs=1139 cs=0 l=1 rev1=1 crypto rx=0x7fecf8006630 tx=0x7fecf80039c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:55:11.350 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:11.348+0000 7fed03e00640 1 -- 172.21.15.32:0/1650848064 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fecf8015070 con 0x7fed0c06f2a0 2024-08-03T20:55:11.351 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:11.348+0000 7fed03e00640 1 -- 172.21.15.32:0/1650848064 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fecf8011900 con 0x7fed0c06f2a0 2024-08-03T20:55:11.351 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:11.348+0000 7fed03e00640 1 -- 172.21.15.32:0/1650848064 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fecf8011d10 con 0x7fed0c06f2a0 2024-08-03T20:55:11.351 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:11.348+0000 7fed12e00640 1 -- 172.21.15.32:0/1650848064 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fed0c1f79b0 con 0x7fed0c06f2a0 2024-08-03T20:55:11.351 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:11.348+0000 7fed12e00640 1 -- 172.21.15.32:0/1650848064 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fed0c1f39f0 con 0x7fed0c06f2a0 2024-08-03T20:55:11.351 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:11.349+0000 7fed12e00640 1 -- 172.21.15.32:0/1650848064 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fed0c06fdd0 con 0x7fed0c06f2a0 2024-08-03T20:55:11.351 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:11.350+0000 7fed03e00640 1 -- 172.21.15.32:0/1650848064 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7fecf8018070 con 0x7fed0c06f2a0 2024-08-03T20:55:11.352 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:11.351+0000 7fed03e00640 1 --2- 172.21.15.32:0/1650848064 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fecec077040 0x7fecec079550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:55:11.352 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:11.351+0000 7fed03e00640 1 -- 172.21.15.32:0/1650848064 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 5967+0+0 (secure 0 0 0) 0x7fecf8099740 con 0x7fed0c06f2a0 2024-08-03T20:55:11.353 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:11.352+0000 7fed0be00640 1 --2- 172.21.15.32:0/1650848064 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fecec077040 0x7fecec079550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:55:11.353 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:11.352+0000 7fed0be00640 1 --2- 172.21.15.32:0/1650848064 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fecec077040 0x7fecec079550 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7fecfc035000 tx=0x7fecfc0046c0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:55:11.357 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:11.355+0000 7fed03e00640 1 -- 172.21.15.32:0/1650848064 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7fed0c15c260 con 0x7fed0c06f2a0 2024-08-03T20:55:11.637 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:11.636+0000 7fed12e00640 1 -- 172.21.15.32:0/1650848064 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "mds repaired", "role": "0"} v 0) -- 0x7fed0c0cef10 con 0x7fed0c06f2a0 2024-08-03T20:55:12.095 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.092+0000 7fed03e00640 1 -- 172.21.15.32:0/1650848064 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "mds repaired", "role": "0"}]=0 repaired: restoring rank 6:0 v109) ==== 101+0+0 (secure 0 0 0) 0x7fecf80621f0 con 0x7fed0c06f2a0 2024-08-03T20:55:12.095 INFO:teuthology.orchestra.run.smithi032.stderr:repaired: restoring rank 6:0 2024-08-03T20:55:12.096 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.094+0000 7fed12e00640 1 -- 172.21.15.32:0/1650848064 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fecec077040 msgr2=0x7fecec079550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:55:12.096 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.094+0000 7fed12e00640 1 --2- 172.21.15.32:0/1650848064 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fecec077040 0x7fecec079550 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7fecfc035000 tx=0x7fecfc0046c0 comp rx=0 tx=0).stop 2024-08-03T20:55:12.097 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.094+0000 7fed12e00640 1 -- 172.21.15.32:0/1650848064 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fed0c06f2a0 msgr2=0x7fed0c06f6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:55:12.097 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.094+0000 7fed12e00640 1 --2- 172.21.15.32:0/1650848064 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fed0c06f2a0 0x7fed0c06f6b0 secure :-1 s=READY pgs=1139 cs=0 l=1 rev1=1 crypto rx=0x7fecf8006630 tx=0x7fecf80039c0 comp rx=0 tx=0).stop 2024-08-03T20:55:12.097 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.095+0000 7fed12e00640 1 -- 172.21.15.32:0/1650848064 shutdown_connections 2024-08-03T20:55:12.097 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.095+0000 7fed12e00640 1 --2- 172.21.15.32:0/1650848064 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fecec077040 0x7fecec079550 unknown :-1 s=CLOSED pgs=473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:12.097 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.095+0000 7fed12e00640 1 --2- 172.21.15.32:0/1650848064 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fed0c15b910 0x7fed0c15bd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:12.097 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.095+0000 7fed12e00640 1 --2- 172.21.15.32:0/1650848064 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fed0c06f2a0 0x7fed0c06f6b0 unknown :-1 s=CLOSED pgs=1139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:12.097 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.095+0000 7fed12e00640 1 --2- 172.21.15.32:0/1650848064 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fed0c0b38b0 0x7fed0c0cb400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:12.097 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.095+0000 7fed12e00640 1 -- 172.21.15.32:0/1650848064 >> 172.21.15.32:0/1650848064 conn(0x7fed0c0b1bd0 msgr2=0x7fed0c159f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:55:12.097 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.095+0000 7fed12e00640 1 -- 172.21.15.32:0/1650848064 shutdown_connections 2024-08-03T20:55:12.097 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.095+0000 7fed12e00640 1 -- 172.21.15.32:0/1650848064 wait complete. 2024-08-03T20:55:12.123 DEBUG:teuthology.orchestra.run.smithi032:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata import - 2024-08-03T20:55:12.268 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.266+0000 7fbcc5a28f40 1 Processor -- start 2024-08-03T20:55:12.268 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.267+0000 7fbcc5a28f40 1 -- start start 2024-08-03T20:55:12.270 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.268+0000 7fbcc5a28f40 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55cac4678360 0x55cac4691af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:55:12.270 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.268+0000 7fbcc5a28f40 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55cac4695cb0 0x55cac4672af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:55:12.270 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.268+0000 7fbcc5a28f40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55cac46963f0 0x55cac4673030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:55:12.270 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.268+0000 7fbcc5a28f40 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55cac450fe30 con 0x55cac4695cb0 2024-08-03T20:55:12.270 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.268+0000 7fbcc5a28f40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x55cac456dbf0 con 0x55cac46963f0 2024-08-03T20:55:12.270 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.268+0000 7fbcc5a28f40 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x55cac4679440 con 0x55cac4678360 2024-08-03T20:55:12.271 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.268+0000 7fbcbe200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55cac4695cb0 0x55cac4672af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:55:12.271 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.268+0000 7fbcbe200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55cac4695cb0 0x55cac4672af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:50416/0 (socket says 172.21.15.32:50416) 2024-08-03T20:55:12.271 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.268+0000 7fbcbe200640 1 -- 172.21.15.32:0/2667442771 learned_addr learned my addr 172.21.15.32:0/2667442771 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:55:12.271 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.268+0000 7fbcbec00640 1 --2- 172.21.15.32:0/2667442771 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55cac4678360 0x55cac4691af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:55:12.271 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.268+0000 7fbcbf600640 1 --2- 172.21.15.32:0/2667442771 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55cac46963f0 0x55cac4673030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:55:12.271 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.269+0000 7fbcbe200640 1 -- 172.21.15.32:0/2667442771 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55cac4678360 msgr2=0x55cac4691af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:55:12.271 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.269+0000 7fbcbe200640 1 --2- 172.21.15.32:0/2667442771 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55cac4678360 0x55cac4691af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:12.271 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.269+0000 7fbcbe200640 1 -- 172.21.15.32:0/2667442771 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55cac46963f0 msgr2=0x55cac4673030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:55:12.271 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.269+0000 7fbcbe200640 1 --2- 172.21.15.32:0/2667442771 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55cac46963f0 0x55cac4673030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:12.271 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.269+0000 7fbcbe200640 1 -- 172.21.15.32:0/2667442771 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cac46020d0 con 0x55cac4695cb0 2024-08-03T20:55:12.272 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.269+0000 7fbcbe200640 1 --2- 172.21.15.32:0/2667442771 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55cac4695cb0 0x55cac4672af0 secure :-1 s=READY pgs=1140 cs=0 l=1 rev1=1 crypto rx=0x7fbcac001cf0 tx=0x7fbcac00b5b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f9a11171947ff180 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:55:12.272 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.269+0000 7fbcbce00640 1 -- 172.21.15.32:0/2667442771 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbcac00bd80 con 0x55cac4695cb0 2024-08-03T20:55:12.272 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.269+0000 7fbcbce00640 1 -- 172.21.15.32:0/2667442771 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fbcac00f400 con 0x55cac4695cb0 2024-08-03T20:55:12.272 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.269+0000 7fbcbce00640 1 -- 172.21.15.32:0/2667442771 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbcac00f890 con 0x55cac4695cb0 2024-08-03T20:55:12.273 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.270+0000 7fbcc5a28f40 1 -- 172.21.15.32:0/2667442771 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55cac4695cb0 msgr2=0x55cac4672af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:55:12.273 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.270+0000 7fbcc5a28f40 1 --2- 172.21.15.32:0/2667442771 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55cac4695cb0 0x55cac4672af0 secure :-1 s=READY pgs=1140 cs=0 l=1 rev1=1 crypto rx=0x7fbcac001cf0 tx=0x7fbcac00b5b0 comp rx=0 tx=0).stop 2024-08-03T20:55:12.273 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.270+0000 7fbcc5a28f40 1 -- 172.21.15.32:0/2667442771 shutdown_connections 2024-08-03T20:55:12.273 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.270+0000 7fbcc5a28f40 1 --2- 172.21.15.32:0/2667442771 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55cac46963f0 0x55cac4673030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:12.273 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.270+0000 7fbcc5a28f40 1 --2- 172.21.15.32:0/2667442771 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55cac4695cb0 0x55cac4672af0 unknown :-1 s=CLOSED pgs=1140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:12.273 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.270+0000 7fbcc5a28f40 1 --2- 172.21.15.32:0/2667442771 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55cac4678360 0x55cac4691af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:12.273 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.270+0000 7fbcc5a28f40 1 -- 172.21.15.32:0/2667442771 >> 172.21.15.32:0/2667442771 conn(0x55cac44e1ac0 msgr2=0x55cac4602b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:55:12.273 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.270+0000 7fbcc5a28f40 1 -- 172.21.15.32:0/2667442771 shutdown_connections 2024-08-03T20:55:12.273 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.270+0000 7fbcc5a28f40 1 -- 172.21.15.32:0/2667442771 wait complete. 2024-08-03T20:55:12.274 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.271+0000 7fbcc5a28f40 1 Processor -- start 2024-08-03T20:55:12.274 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.272+0000 7fbcc5a28f40 1 -- start start 2024-08-03T20:55:12.276 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.273+0000 7fbcc5a28f40 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55cac4678360 0x55cac466f470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:55:12.276 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.273+0000 7fbcc5a28f40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55cac4695cb0 0x55cac4691af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:55:12.276 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.273+0000 7fbcc5a28f40 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55cac46963f0 0x55cac4692030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:55:12.276 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.273+0000 7fbcc5a28f40 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55cac4602370 con 0x55cac46963f0 2024-08-03T20:55:12.276 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.273+0000 7fbcc5a28f40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x55cac4672420 con 0x55cac4695cb0 2024-08-03T20:55:12.276 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.273+0000 7fbcc5a28f40 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x55cac4677100 con 0x55cac4678360 2024-08-03T20:55:12.276 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.273+0000 7fbcbf600640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55cac46963f0 0x55cac4692030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:55:12.276 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.273+0000 7fbcbf600640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55cac46963f0 0x55cac4692030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:50428/0 (socket says 172.21.15.32:50428) 2024-08-03T20:55:12.276 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.273+0000 7fbcbf600640 1 -- 172.21.15.32:0/801923352 learned_addr learned my addr 172.21.15.32:0/801923352 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:55:12.276 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.273+0000 7fbcbec00640 1 --2- 172.21.15.32:0/801923352 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55cac4678360 0x55cac466f470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:55:12.276 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.273+0000 7fbcbe200640 1 --2- 172.21.15.32:0/801923352 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55cac4695cb0 0x55cac4691af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:55:12.277 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.274+0000 7fbcbf600640 1 -- 172.21.15.32:0/801923352 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55cac4678360 msgr2=0x55cac466f470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:55:12.277 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.274+0000 7fbcbf600640 1 --2- 172.21.15.32:0/801923352 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55cac4678360 0x55cac466f470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:12.277 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.274+0000 7fbcbf600640 1 -- 172.21.15.32:0/801923352 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55cac4695cb0 msgr2=0x55cac4691af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:55:12.277 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.274+0000 7fbcbf600640 1 --2- 172.21.15.32:0/801923352 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55cac4695cb0 0x55cac4691af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:12.277 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.274+0000 7fbcbf600640 1 -- 172.21.15.32:0/801923352 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cac4694d20 con 0x55cac46963f0 2024-08-03T20:55:12.277 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.274+0000 7fbcbf600640 1 --2- 172.21.15.32:0/801923352 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55cac46963f0 0x55cac4692030 secure :-1 s=READY pgs=1141 cs=0 l=1 rev1=1 crypto rx=0x7fbcb80106f0 tx=0x7fbcb8010bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:55:12.277 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.274+0000 7fbcb7400640 1 -- 172.21.15.32:0/801923352 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbcb800eb30 con 0x55cac46963f0 2024-08-03T20:55:12.277 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.274+0000 7fbcb7400640 1 -- 172.21.15.32:0/801923352 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fbcb800edc0 con 0x55cac46963f0 2024-08-03T20:55:12.277 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.274+0000 7fbcc5a28f40 1 -- 172.21.15.32:0/801923352 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x55cac4730540 con 0x55cac46963f0 2024-08-03T20:55:12.277 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.274+0000 7fbcb7400640 1 -- 172.21.15.32:0/801923352 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbcb8019680 con 0x55cac46963f0 2024-08-03T20:55:12.277 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.274+0000 7fbcc5a28f40 1 -- 172.21.15.32:0/801923352 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cac46910d0 con 0x55cac46963f0 2024-08-03T20:55:12.278 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.276+0000 7fbcb7400640 1 -- 172.21.15.32:0/801923352 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7fbcb8019820 con 0x55cac46963f0 2024-08-03T20:55:12.279 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.277+0000 7fbcb7400640 1 --2- 172.21.15.32:0/801923352 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fbca00759f0 0x7fbca0077f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:55:12.279 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.278+0000 7fbcb7400640 1 -- 172.21.15.32:0/801923352 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 5967+0+0 (secure 0 0 0) 0x7fbcb80987c0 con 0x55cac46963f0 2024-08-03T20:55:12.279 INFO:teuthology.orchestra.run.smithi032.stdout:Importing pool 2024-08-03T20:55:12.280 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.278+0000 7fbcc5a28f40 1 --2- 172.21.15.32:0/801923352 >> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] conn(0x55cac4695830 0x55cac4690290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:55:12.280 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.278+0000 7fbcc5a28f40 1 -- 172.21.15.32:0/801923352 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:1 12.10 12:09401a81:::601.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e86) -- 0x55cac46733b0 con 0x55cac4695830 2024-08-03T20:55:12.280 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.278+0000 7fbcbec00640 1 --2- 172.21.15.32:0/801923352 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fbca00759f0 0x7fbca0077f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:55:12.280 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.278+0000 7fbcbe200640 1 --2- 172.21.15.32:0/801923352 >> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] conn(0x55cac4695830 0x55cac4690290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:55:12.280 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.279+0000 7fbcbe200640 1 --2- 172.21.15.32:0/801923352 >> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] conn(0x55cac4695830 0x55cac4690290 crc :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:55:12.280 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.279+0000 7fbcbec00640 1 --2- 172.21.15.32:0/801923352 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fbca00759f0 0x7fbca0077f00 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7fbcb0055000 tx=0x7fbcb0003020 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:55:12.282 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.281+0000 7fbcbe200640 1 -- 172.21.15.32:0/801923352 <== osd.0 v2:172.21.15.32:6816/3066156116 1 ==== osd_op_reply(1 601.00000000 [create] v86'73 uv48 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fbcac00f500 con 0x55cac4695830 2024-08-03T20:55:12.282 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.281+0000 7fbcc5a28f40 1 -- 172.21.15.32:0/801923352 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:2 12.10 12:09401a81:::601.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e86) -- 0x55cac4738b50 con 0x55cac4695830 2024-08-03T20:55:12.284 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.283+0000 7fbcbe200640 1 -- 172.21.15.32:0/801923352 <== osd.0 v2:172.21.15.32:6816/3066156116 2 ==== osd_op_reply(2 601.00000000 [delete] v86'74 uv74 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fbcac00f500 con 0x55cac4695830 2024-08-03T20:55:12.284 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.283+0000 7fbcc5a28f40 1 -- 172.21.15.32:0/801923352 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:3 12.10 12:09401a81:::601.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e86) -- 0x55cac4738ef0 con 0x55cac4695830 2024-08-03T20:55:12.286 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.284+0000 7fbcbe200640 1 -- 172.21.15.32:0/801923352 <== osd.0 v2:172.21.15.32:6816/3066156116 3 ==== osd_op_reply(3 601.00000000 [create] v86'75 uv75 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fbcac00f500 con 0x55cac4695830 2024-08-03T20:55:12.286 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::601.00000000:head# 2024-08-03T20:55:12.286 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.285+0000 7fbcc5a28f40 1 -- 172.21.15.32:0/801923352 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:4 12.10 12:09401a81:::601.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e86) -- 0x55cac473a2a0 con 0x55cac4695830 2024-08-03T20:55:12.288 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.286+0000 7fbcbe200640 1 -- 172.21.15.32:0/801923352 <== osd.0 v2:172.21.15.32:6816/3066156116 4 ==== osd_op_reply(4 601.00000000 [setxattr (30)] v86'76 uv76 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fbcac00f500 con 0x55cac4695830 2024-08-03T20:55:12.288 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.286+0000 7fbcc5a28f40 1 -- 172.21.15.32:0/801923352 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:5 12.10 12:09401a81:::601.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e86) -- 0x55cac473a640 con 0x55cac4695830 2024-08-03T20:55:12.289 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.288+0000 7fbcbe200640 1 -- 172.21.15.32:0/801923352 <== osd.0 v2:172.21.15.32:6816/3066156116 5 ==== osd_op_reply(5 601.00000000 [setxattr (62)] v86'77 uv77 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fbcac00f500 con 0x55cac4695830 2024-08-03T20:55:12.289 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.288+0000 7fbcc5a28f40 1 -- 172.21.15.32:0/801923352 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:6 12.10 12:09401a81:::601.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e86) -- 0x55cac4739290 con 0x55cac4695830 2024-08-03T20:55:12.291 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.289+0000 7fbcbe200640 1 -- 172.21.15.32:0/801923352 <== osd.0 v2:172.21.15.32:6816/3066156116 6 ==== osd_op_reply(6 601.00000000 [omap-set-header] v86'78 uv78 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fbcac00f500 con 0x55cac4695830 2024-08-03T20:55:12.291 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.289+0000 7fbcc5a28f40 1 -- 172.21.15.32:0/801923352 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:7 12.10 12:0bd6d154:::602.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e86) -- 0x55cac4739650 con 0x55cac4695830 2024-08-03T20:55:12.292 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.291+0000 7fbcbe200640 1 -- 172.21.15.32:0/801923352 <== osd.0 v2:172.21.15.32:6816/3066156116 7 ==== osd_op_reply(7 602.00000000 [create] v86'79 uv54 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fbcac00f500 con 0x55cac4695830 2024-08-03T20:55:12.292 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.291+0000 7fbcc5a28f40 1 -- 172.21.15.32:0/801923352 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:8 12.10 12:0bd6d154:::602.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e86) -- 0x55cac47399f0 con 0x55cac4695830 2024-08-03T20:55:12.294 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.292+0000 7fbcbe200640 1 -- 172.21.15.32:0/801923352 <== osd.0 v2:172.21.15.32:6816/3066156116 8 ==== osd_op_reply(8 602.00000000 [delete] v86'80 uv80 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fbcac00f500 con 0x55cac4695830 2024-08-03T20:55:12.294 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.292+0000 7fbcc5a28f40 1 -- 172.21.15.32:0/801923352 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:9 12.10 12:0bd6d154:::602.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e86) -- 0x55cac47399f0 con 0x55cac4695830 2024-08-03T20:55:12.298 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::602.00000000:head# 2024-08-03T20:55:12.299 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.294+0000 7fbcbe200640 1 -- 172.21.15.32:0/801923352 <== osd.0 v2:172.21.15.32:6816/3066156116 9 ==== osd_op_reply(9 602.00000000 [create] v86'81 uv81 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fbcac00f500 con 0x55cac4695830 2024-08-03T20:55:12.299 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.294+0000 7fbcc5a28f40 1 -- 172.21.15.32:0/801923352 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:10 12.10 12:0bd6d154:::602.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e86) -- 0x55cac47399f0 con 0x55cac4695830 2024-08-03T20:55:12.299 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.296+0000 7fbcbe200640 1 -- 172.21.15.32:0/801923352 <== osd.0 v2:172.21.15.32:6816/3066156116 10 ==== osd_op_reply(10 602.00000000 [setxattr (30)] v86'82 uv82 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fbcac00f500 con 0x55cac4695830 2024-08-03T20:55:12.300 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.296+0000 7fbcc5a28f40 1 -- 172.21.15.32:0/801923352 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:11 12.10 12:0bd6d154:::602.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e86) -- 0x55cac47399f0 con 0x55cac4695830 2024-08-03T20:55:12.300 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.297+0000 7fbcbe200640 1 -- 172.21.15.32:0/801923352 <== osd.0 v2:172.21.15.32:6816/3066156116 11 ==== osd_op_reply(11 602.00000000 [setxattr (62)] v86'83 uv83 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fbcac00f500 con 0x55cac4695830 2024-08-03T20:55:12.300 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.297+0000 7fbcc5a28f40 1 -- 172.21.15.32:0/801923352 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:12 12.10 12:0bd6d154:::602.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e86) -- 0x55cac47399f0 con 0x55cac4695830 2024-08-03T20:55:12.300 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.299+0000 7fbcbe200640 1 -- 172.21.15.32:0/801923352 <== osd.0 v2:172.21.15.32:6816/3066156116 12 ==== osd_op_reply(12 602.00000000 [omap-set-header] v86'84 uv84 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fbcac00f500 con 0x55cac4695830 2024-08-03T20:55:12.300 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.299+0000 7fbcc5a28f40 1 -- 172.21.15.32:0/801923352 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:13 12.10 12:0cf6ad0f:::10000000000.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e86) -- 0x55cac47399f0 con 0x55cac4695830 2024-08-03T20:55:12.302 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.300+0000 7fbcbe200640 1 -- 172.21.15.32:0/801923352 <== osd.0 v2:172.21.15.32:6816/3066156116 13 ==== osd_op_reply(13 10000000000.00000000 [create] v86'85 uv59 ondisk = -17 ((17) File exists)) ==== 164+0+0 (crc 0 0 0) 0x7fbcac00f500 con 0x55cac4695830 2024-08-03T20:55:12.302 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.300+0000 7fbcc5a28f40 1 -- 172.21.15.32:0/801923352 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:14 12.10 12:0cf6ad0f:::10000000000.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e86) -- 0x55cac47399f0 con 0x55cac4695830 2024-08-03T20:55:12.303 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.302+0000 7fbcbe200640 1 -- 172.21.15.32:0/801923352 <== osd.0 v2:172.21.15.32:6816/3066156116 14 ==== osd_op_reply(14 10000000000.00000000 [delete] v86'86 uv86 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7fbcac00f500 con 0x55cac4695830 2024-08-03T20:55:12.303 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.302+0000 7fbcc5a28f40 1 -- 172.21.15.32:0/801923352 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:15 12.10 12:0cf6ad0f:::10000000000.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e86) -- 0x55cac47399f0 con 0x55cac4695830 2024-08-03T20:55:12.305 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.304+0000 7fbcbe200640 1 -- 172.21.15.32:0/801923352 <== osd.0 v2:172.21.15.32:6816/3066156116 15 ==== osd_op_reply(15 10000000000.00000000 [create] v86'87 uv87 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7fbcac00f500 con 0x55cac4695830 2024-08-03T20:55:12.305 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::10000000000.00000000:head# 2024-08-03T20:55:12.305 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.304+0000 7fbcc5a28f40 1 -- 172.21.15.32:0/801923352 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:16 12.10 12:0cf6ad0f:::10000000000.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e86) -- 0x55cac47399f0 con 0x55cac4695830 2024-08-03T20:55:12.306 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.305+0000 7fbcbe200640 1 -- 172.21.15.32:0/801923352 <== osd.0 v2:172.21.15.32:6816/3066156116 16 ==== osd_op_reply(16 10000000000.00000000 [setxattr (30)] v86'88 uv88 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7fbcac00f500 con 0x55cac4695830 2024-08-03T20:55:12.307 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.305+0000 7fbcc5a28f40 1 -- 172.21.15.32:0/801923352 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:17 12.10 12:0cf6ad0f:::10000000000.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e86) -- 0x55cac47399f0 con 0x55cac4695830 2024-08-03T20:55:12.308 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.307+0000 7fbcbe200640 1 -- 172.21.15.32:0/801923352 <== osd.0 v2:172.21.15.32:6816/3066156116 17 ==== osd_op_reply(17 10000000000.00000000 [setxattr (62)] v86'89 uv89 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7fbcac00f500 con 0x55cac4695830 2024-08-03T20:55:12.308 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.307+0000 7fbcc5a28f40 1 -- 172.21.15.32:0/801923352 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:18 12.10 12:0cf6ad0f:::10000000000.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e86) -- 0x55cac47399f0 con 0x55cac4695830 2024-08-03T20:55:12.310 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.308+0000 7fbcbe200640 1 -- 172.21.15.32:0/801923352 <== osd.0 v2:172.21.15.32:6816/3066156116 18 ==== osd_op_reply(18 10000000000.00000000 [omap-set-header] v86'90 uv90 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7fbcac00f500 con 0x55cac4695830 2024-08-03T20:55:12.310 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.308+0000 7fbcc5a28f40 1 -- 172.21.15.32:0/801923352 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:19 12.10 12:0cf6ad0f:::10000000000.00000000:head [omap-set-vals in=593b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e86) -- 0x55cac47399f0 con 0x55cac4695830 2024-08-03T20:55:12.311 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.310+0000 7fbcbe200640 1 -- 172.21.15.32:0/801923352 <== osd.0 v2:172.21.15.32:6816/3066156116 19 ==== osd_op_reply(19 10000000000.00000000 [omap-set-vals] v86'91 uv91 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7fbcac00f500 con 0x55cac4695830 2024-08-03T20:55:12.311 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.310+0000 7fbcc5a28f40 1 -- 172.21.15.32:0/801923352 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:20 12.10 12:0d82a743:::600.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e86) -- 0x55cac47399f0 con 0x55cac4695830 2024-08-03T20:55:12.313 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.312+0000 7fbcbe200640 1 -- 172.21.15.32:0/801923352 <== osd.0 v2:172.21.15.32:6816/3066156116 20 ==== osd_op_reply(20 600.00000000 [create] v86'92 uv92 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fbcac00f500 con 0x55cac4695830 2024-08-03T20:55:12.313 INFO:teuthology.orchestra.run.smithi032.stdout:Write #-9223372036854775808:00000000:::600.00000000:head# 2024-08-03T20:55:12.314 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.312+0000 7fbcc5a28f40 1 -- 172.21.15.32:0/801923352 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:21 12.10 12:0d82a743:::600.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e86) -- 0x55cac47399f0 con 0x55cac4695830 2024-08-03T20:55:12.314 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.313+0000 7fbcbe200640 1 -- 172.21.15.32:0/801923352 <== osd.0 v2:172.21.15.32:6816/3066156116 21 ==== osd_op_reply(21 600.00000000 [setxattr (30)] v86'93 uv93 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fbcac00f500 con 0x55cac4695830 2024-08-03T20:55:12.314 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.313+0000 7fbcc5a28f40 1 -- 172.21.15.32:0/801923352 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:22 12.10 12:0d82a743:::600.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e86) -- 0x55cac47399f0 con 0x55cac4695830 2024-08-03T20:55:12.316 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.315+0000 7fbcbe200640 1 -- 172.21.15.32:0/801923352 <== osd.0 v2:172.21.15.32:6816/3066156116 22 ==== osd_op_reply(22 600.00000000 [setxattr (62)] v86'94 uv94 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fbcac00f500 con 0x55cac4695830 2024-08-03T20:55:12.316 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.315+0000 7fbcc5a28f40 1 -- 172.21.15.32:0/801923352 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:23 12.10 12:0d82a743:::600.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e86) -- 0x55cac47399f0 con 0x55cac4695830 2024-08-03T20:55:12.318 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.316+0000 7fbcbe200640 1 -- 172.21.15.32:0/801923352 <== osd.0 v2:172.21.15.32:6816/3066156116 23 ==== osd_op_reply(23 600.00000000 [omap-set-header] v86'95 uv95 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fbcac00f500 con 0x55cac4695830 2024-08-03T20:55:12.318 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.316+0000 7fbcc5a28f40 1 -- 172.21.15.32:0/801923352 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:24 12.10 12:0d89b25e:::603.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e86) -- 0x55cac47399f0 con 0x55cac4695830 2024-08-03T20:55:12.319 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.318+0000 7fbcbe200640 1 -- 172.21.15.32:0/801923352 <== osd.0 v2:172.21.15.32:6816/3066156116 24 ==== osd_op_reply(24 603.00000000 [create] v86'96 uv71 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fbcac00f500 con 0x55cac4695830 2024-08-03T20:55:12.319 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.318+0000 7fbcc5a28f40 1 -- 172.21.15.32:0/801923352 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:25 12.10 12:0d89b25e:::603.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e86) -- 0x55cac47399f0 con 0x55cac4695830 2024-08-03T20:55:12.321 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.319+0000 7fbcbe200640 1 -- 172.21.15.32:0/801923352 <== osd.0 v2:172.21.15.32:6816/3066156116 25 ==== osd_op_reply(25 603.00000000 [delete] v86'97 uv97 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fbcac00f500 con 0x55cac4695830 2024-08-03T20:55:12.321 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.319+0000 7fbcc5a28f40 1 -- 172.21.15.32:0/801923352 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:26 12.10 12:0d89b25e:::603.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e86) -- 0x55cac47399f0 con 0x55cac4695830 2024-08-03T20:55:12.322 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::603.00000000:head# 2024-08-03T20:55:12.322 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.321+0000 7fbcbe200640 1 -- 172.21.15.32:0/801923352 <== osd.0 v2:172.21.15.32:6816/3066156116 26 ==== osd_op_reply(26 603.00000000 [create] v86'98 uv98 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fbcac00f500 con 0x55cac4695830 2024-08-03T20:55:12.323 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.321+0000 7fbcc5a28f40 1 -- 172.21.15.32:0/801923352 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:27 12.10 12:0d89b25e:::603.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e86) -- 0x55cac47399f0 con 0x55cac4695830 2024-08-03T20:55:12.324 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.322+0000 7fbcbe200640 1 -- 172.21.15.32:0/801923352 <== osd.0 v2:172.21.15.32:6816/3066156116 27 ==== osd_op_reply(27 603.00000000 [setxattr (30)] v86'99 uv99 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fbcac00f500 con 0x55cac4695830 2024-08-03T20:55:12.324 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.322+0000 7fbcc5a28f40 1 -- 172.21.15.32:0/801923352 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:28 12.10 12:0d89b25e:::603.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e86) -- 0x55cac47399f0 con 0x55cac4695830 2024-08-03T20:55:12.325 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.324+0000 7fbcbe200640 1 -- 172.21.15.32:0/801923352 <== osd.0 v2:172.21.15.32:6816/3066156116 28 ==== osd_op_reply(28 603.00000000 [setxattr (62)] v86'100 uv100 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fbcac00f500 con 0x55cac4695830 2024-08-03T20:55:12.325 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.324+0000 7fbcc5a28f40 1 -- 172.21.15.32:0/801923352 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:29 12.10 12:0d89b25e:::603.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e86) -- 0x55cac47399f0 con 0x55cac4695830 2024-08-03T20:55:12.327 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.326+0000 7fbcbe200640 1 -- 172.21.15.32:0/801923352 <== osd.0 v2:172.21.15.32:6816/3066156116 29 ==== osd_op_reply(29 603.00000000 [omap-set-header] v86'101 uv101 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fbcac00f500 con 0x55cac4695830 2024-08-03T20:55:12.327 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.326+0000 7fbcc5a28f40 1 --2- 172.21.15.32:0/801923352 >> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] conn(0x55cac4739db0 0x55cac473cf00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:55:12.328 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.326+0000 7fbcc5a28f40 1 -- 172.21.15.32:0/801923352 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:30 12.14 12:287034c4:::1.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e86) -- 0x55cac47399f0 con 0x55cac4739db0 2024-08-03T20:55:12.328 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.326+0000 7fbcbec00640 1 --2- 172.21.15.32:0/801923352 >> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] conn(0x55cac4739db0 0x55cac473cf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:55:12.328 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.327+0000 7fbcbec00640 1 --2- 172.21.15.32:0/801923352 >> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] conn(0x55cac4739db0 0x55cac473cf00 crc :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:55:12.330 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.328+0000 7fbcbec00640 1 -- 172.21.15.32:0/801923352 <== osd.5 v2:172.21.15.118:6810/2724891770 1 ==== osd_op_reply(30 1.00000000.inode [create] v86'67 uv49 ondisk = -17 ((17) File exists)) ==== 160+0+0 (crc 0 0 0) 0x7fbcb0057480 con 0x55cac4739db0 2024-08-03T20:55:12.330 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.328+0000 7fbcc5a28f40 1 -- 172.21.15.32:0/801923352 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:31 12.14 12:287034c4:::1.00000000.inode:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e86) -- 0x55cac473d440 con 0x55cac4739db0 2024-08-03T20:55:12.331 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.330+0000 7fbcbec00640 1 -- 172.21.15.32:0/801923352 <== osd.5 v2:172.21.15.118:6810/2724891770 2 ==== osd_op_reply(31 1.00000000.inode [delete] v86'68 uv68 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7fbcb0057480 con 0x55cac4739db0 2024-08-03T20:55:12.331 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.330+0000 7fbcc5a28f40 1 -- 172.21.15.32:0/801923352 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:32 12.14 12:287034c4:::1.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e86) -- 0x55cac473d7e0 con 0x55cac4739db0 2024-08-03T20:55:12.333 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.332+0000 7fbcbec00640 1 -- 172.21.15.32:0/801923352 <== osd.5 v2:172.21.15.118:6810/2724891770 3 ==== osd_op_reply(32 1.00000000.inode [create] v86'69 uv69 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7fbcb0057480 con 0x55cac4739db0 2024-08-03T20:55:12.333 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::1.00000000.inode:head# 2024-08-03T20:55:12.333 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.332+0000 7fbcc5a28f40 1 -- 172.21.15.32:0/801923352 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:33 12.14 12:287034c4:::1.00000000.inode:head [write 0~582 in=582b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e86) -- 0x55cac473db80 con 0x55cac4739db0 2024-08-03T20:55:12.335 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.333+0000 7fbcbec00640 1 -- 172.21.15.32:0/801923352 <== osd.5 v2:172.21.15.118:6810/2724891770 4 ==== osd_op_reply(33 1.00000000.inode [write 0~582] v86'70 uv70 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7fbcb0057480 con 0x55cac4739db0 2024-08-03T20:55:12.335 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.333+0000 7fbcc5a28f40 1 -- 172.21.15.32:0/801923352 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:34 12.14 12:287034c4:::1.00000000.inode:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e86) -- 0x55cac473df20 con 0x55cac4739db0 2024-08-03T20:55:12.337 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.335+0000 7fbcbec00640 1 -- 172.21.15.32:0/801923352 <== osd.5 v2:172.21.15.118:6810/2724891770 5 ==== osd_op_reply(34 1.00000000.inode [omap-set-header] v86'71 uv71 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7fbcb0057480 con 0x55cac4739db0 2024-08-03T20:55:12.337 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.335+0000 7fbcc5a28f40 1 -- 172.21.15.32:0/801923352 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:35 12.14 12:292cf221:::200.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e86) -- 0x55cac473e2c0 con 0x55cac4739db0 2024-08-03T20:55:12.338 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.336+0000 7fbcbec00640 1 -- 172.21.15.32:0/801923352 <== osd.5 v2:172.21.15.118:6810/2724891770 6 ==== osd_op_reply(35 200.00000000 [create] v86'72 uv54 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fbcb0057480 con 0x55cac4739db0 2024-08-03T20:55:12.338 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.337+0000 7fbcc5a28f40 1 -- 172.21.15.32:0/801923352 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:36 12.14 12:292cf221:::200.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e86) -- 0x55cac473e660 con 0x55cac4739db0 2024-08-03T20:55:12.339 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.338+0000 7fbcbec00640 1 -- 172.21.15.32:0/801923352 <== osd.5 v2:172.21.15.118:6810/2724891770 7 ==== osd_op_reply(36 200.00000000 [delete] v86'73 uv73 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fbcb0057480 con 0x55cac4739db0 2024-08-03T20:55:12.340 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.338+0000 7fbcc5a28f40 1 -- 172.21.15.32:0/801923352 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:37 12.14 12:292cf221:::200.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e86) -- 0x55cac473e660 con 0x55cac4739db0 2024-08-03T20:55:12.341 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.339+0000 7fbcbec00640 1 -- 172.21.15.32:0/801923352 <== osd.5 v2:172.21.15.118:6810/2724891770 8 ==== osd_op_reply(37 200.00000000 [create] v86'74 uv74 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fbcb0057480 con 0x55cac4739db0 2024-08-03T20:55:12.341 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::200.00000000:head# 2024-08-03T20:55:12.341 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.340+0000 7fbcc5a28f40 1 -- 172.21.15.32:0/801923352 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:38 12.14 12:292cf221:::200.00000000:head [write 0~90 in=90b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e86) -- 0x55cac468f9b0 con 0x55cac4739db0 2024-08-03T20:55:12.342 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.341+0000 7fbcbec00640 1 -- 172.21.15.32:0/801923352 <== osd.5 v2:172.21.15.118:6810/2724891770 9 ==== osd_op_reply(38 200.00000000 [write 0~90] v86'75 uv75 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fbcb0057480 con 0x55cac4739db0 2024-08-03T20:55:12.343 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.341+0000 7fbcc5a28f40 1 -- 172.21.15.32:0/801923352 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:39 12.14 12:292cf221:::200.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e86) -- 0x55cac468f9b0 con 0x55cac4739db0 2024-08-03T20:55:12.344 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.343+0000 7fbcbec00640 1 -- 172.21.15.32:0/801923352 <== osd.5 v2:172.21.15.118:6810/2724891770 10 ==== osd_op_reply(39 200.00000000 [omap-set-header] v86'76 uv76 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fbcb0057480 con 0x55cac4739db0 2024-08-03T20:55:12.344 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.343+0000 7fbcc5a28f40 1 -- 172.21.15.32:0/801923352 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:40 12.14 12:2e2fa760:::200.00000001:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e86) -- 0x55cac468f9b0 con 0x55cac4739db0 2024-08-03T20:55:12.345 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.344+0000 7fbcbec00640 1 -- 172.21.15.32:0/801923352 <== osd.5 v2:172.21.15.118:6810/2724891770 11 ==== osd_op_reply(40 200.00000001 [create] v86'77 uv59 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fbcb0057480 con 0x55cac4739db0 2024-08-03T20:55:12.345 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.344+0000 7fbcc5a28f40 1 -- 172.21.15.32:0/801923352 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:41 12.14 12:2e2fa760:::200.00000001:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e86) -- 0x55cac468f9b0 con 0x55cac4739db0 2024-08-03T20:55:12.347 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.346+0000 7fbcbec00640 1 -- 172.21.15.32:0/801923352 <== osd.5 v2:172.21.15.118:6810/2724891770 12 ==== osd_op_reply(41 200.00000001 [delete] v86'78 uv78 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fbcb0057480 con 0x55cac4739db0 2024-08-03T20:55:12.347 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.346+0000 7fbcc5a28f40 1 -- 172.21.15.32:0/801923352 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:42 12.14 12:2e2fa760:::200.00000001:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e86) -- 0x55cac468f9b0 con 0x55cac4739db0 2024-08-03T20:55:12.348 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.347+0000 7fbcbec00640 1 -- 172.21.15.32:0/801923352 <== osd.5 v2:172.21.15.118:6810/2724891770 13 ==== osd_op_reply(42 200.00000001 [create] v86'79 uv79 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fbcb0057480 con 0x55cac4739db0 2024-08-03T20:55:12.349 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::200.00000001:head# 2024-08-03T20:55:12.349 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.347+0000 7fbcc5a28f40 1 -- 172.21.15.32:0/801923352 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:43 12.14 12:2e2fa760:::200.00000001:head [write 0~18118 in=18118b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e86) -- 0x55cac468f9b0 con 0x55cac4739db0 2024-08-03T20:55:12.351 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.349+0000 7fbcbec00640 1 -- 172.21.15.32:0/801923352 <== osd.5 v2:172.21.15.118:6810/2724891770 14 ==== osd_op_reply(43 200.00000001 [write 0~18118] v86'80 uv80 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fbcb0057480 con 0x55cac4739db0 2024-08-03T20:55:12.351 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.349+0000 7fbcc5a28f40 1 -- 172.21.15.32:0/801923352 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:44 12.14 12:2e2fa760:::200.00000001:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e86) -- 0x55cac468f9b0 con 0x55cac4739db0 2024-08-03T20:55:12.352 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.351+0000 7fbcbec00640 1 -- 172.21.15.32:0/801923352 <== osd.5 v2:172.21.15.118:6810/2724891770 15 ==== osd_op_reply(44 200.00000001 [omap-set-header] v86'81 uv81 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fbcb0057480 con 0x55cac4739db0 2024-08-03T20:55:12.352 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.351+0000 7fbcc5a28f40 1 -- 172.21.15.32:0/801923352 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:45 12.14 12:2f55791f:::606.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e86) -- 0x55cac468f9b0 con 0x55cac4739db0 2024-08-03T20:55:12.354 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.352+0000 7fbcbec00640 1 -- 172.21.15.32:0/801923352 <== osd.5 v2:172.21.15.118:6810/2724891770 16 ==== osd_op_reply(45 606.00000000 [create] v86'82 uv65 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fbcb0057480 con 0x55cac4739db0 2024-08-03T20:55:12.354 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.352+0000 7fbcc5a28f40 1 -- 172.21.15.32:0/801923352 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:46 12.14 12:2f55791f:::606.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e86) -- 0x55cac468f9b0 con 0x55cac4739db0 2024-08-03T20:55:12.355 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.354+0000 7fbcbec00640 1 -- 172.21.15.32:0/801923352 <== osd.5 v2:172.21.15.118:6810/2724891770 17 ==== osd_op_reply(46 606.00000000 [delete] v86'83 uv83 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fbcb0057480 con 0x55cac4739db0 2024-08-03T20:55:12.355 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.354+0000 7fbcc5a28f40 1 -- 172.21.15.32:0/801923352 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:47 12.14 12:2f55791f:::606.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e86) -- 0x55cac468f9b0 con 0x55cac4739db0 2024-08-03T20:55:12.357 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.355+0000 7fbcbec00640 1 -- 172.21.15.32:0/801923352 <== osd.5 v2:172.21.15.118:6810/2724891770 18 ==== osd_op_reply(47 606.00000000 [create] v86'84 uv84 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fbcb0057480 con 0x55cac4739db0 2024-08-03T20:55:12.357 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::606.00000000:head# 2024-08-03T20:55:12.357 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.355+0000 7fbcc5a28f40 1 -- 172.21.15.32:0/801923352 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:48 12.14 12:2f55791f:::606.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e86) -- 0x55cac468f9b0 con 0x55cac4739db0 2024-08-03T20:55:12.358 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.356+0000 7fbcbec00640 1 -- 172.21.15.32:0/801923352 <== osd.5 v2:172.21.15.118:6810/2724891770 19 ==== osd_op_reply(48 606.00000000 [setxattr (30)] v86'85 uv85 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fbcb0057480 con 0x55cac4739db0 2024-08-03T20:55:12.358 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.356+0000 7fbcc5a28f40 1 -- 172.21.15.32:0/801923352 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:49 12.14 12:2f55791f:::606.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e86) -- 0x55cac468f9b0 con 0x55cac4739db0 2024-08-03T20:55:12.359 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.358+0000 7fbcbec00640 1 -- 172.21.15.32:0/801923352 <== osd.5 v2:172.21.15.118:6810/2724891770 20 ==== osd_op_reply(49 606.00000000 [setxattr (62)] v86'86 uv86 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fbcb0057480 con 0x55cac4739db0 2024-08-03T20:55:12.359 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.358+0000 7fbcc5a28f40 1 -- 172.21.15.32:0/801923352 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:50 12.14 12:2f55791f:::606.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e86) -- 0x55cac468f9b0 con 0x55cac4739db0 2024-08-03T20:55:12.361 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.359+0000 7fbcbec00640 1 -- 172.21.15.32:0/801923352 <== osd.5 v2:172.21.15.118:6810/2724891770 21 ==== osd_op_reply(50 606.00000000 [omap-set-header] v86'87 uv87 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fbcb0057480 con 0x55cac4739db0 2024-08-03T20:55:12.361 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.359+0000 7fbcc5a28f40 1 -- 172.21.15.32:0/801923352 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:51 12.1c 12:3ed09add:::607.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e86) -- 0x55cac468f9b0 con 0x55cac4695830 2024-08-03T20:55:12.362 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.361+0000 7fbcbe200640 1 -- 172.21.15.32:0/801923352 <== osd.0 v2:172.21.15.32:6816/3066156116 30 ==== osd_op_reply(51 607.00000000 [create] v86'15 uv14 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fbcac00f500 con 0x55cac4695830 2024-08-03T20:55:12.362 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.361+0000 7fbcc5a28f40 1 -- 172.21.15.32:0/801923352 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:52 12.1c 12:3ed09add:::607.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e86) -- 0x55cac468f9b0 con 0x55cac4695830 2024-08-03T20:55:12.364 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.362+0000 7fbcbe200640 1 -- 172.21.15.32:0/801923352 <== osd.0 v2:172.21.15.32:6816/3066156116 31 ==== osd_op_reply(52 607.00000000 [delete] v86'16 uv16 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fbcac00f500 con 0x55cac4695830 2024-08-03T20:55:12.364 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.362+0000 7fbcc5a28f40 1 -- 172.21.15.32:0/801923352 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:53 12.1c 12:3ed09add:::607.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e86) -- 0x55cac468f9b0 con 0x55cac4695830 2024-08-03T20:55:12.365 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.364+0000 7fbcbe200640 1 -- 172.21.15.32:0/801923352 <== osd.0 v2:172.21.15.32:6816/3066156116 32 ==== osd_op_reply(53 607.00000000 [create] v86'17 uv17 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fbcac00f500 con 0x55cac4695830 2024-08-03T20:55:12.365 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::607.00000000:head# 2024-08-03T20:55:12.365 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.364+0000 7fbcc5a28f40 1 -- 172.21.15.32:0/801923352 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:54 12.1c 12:3ed09add:::607.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e86) -- 0x55cac468f9b0 con 0x55cac4695830 2024-08-03T20:55:12.366 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.365+0000 7fbcbe200640 1 -- 172.21.15.32:0/801923352 <== osd.0 v2:172.21.15.32:6816/3066156116 33 ==== osd_op_reply(54 607.00000000 [setxattr (30)] v86'18 uv18 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fbcac00f500 con 0x55cac4695830 2024-08-03T20:55:12.366 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.365+0000 7fbcc5a28f40 1 -- 172.21.15.32:0/801923352 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:55 12.1c 12:3ed09add:::607.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e86) -- 0x55cac468f9b0 con 0x55cac4695830 2024-08-03T20:55:12.367 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.366+0000 7fbcbe200640 1 -- 172.21.15.32:0/801923352 <== osd.0 v2:172.21.15.32:6816/3066156116 34 ==== osd_op_reply(55 607.00000000 [setxattr (62)] v86'19 uv19 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fbcac00f500 con 0x55cac4695830 2024-08-03T20:55:12.368 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.366+0000 7fbcc5a28f40 1 -- 172.21.15.32:0/801923352 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:56 12.1c 12:3ed09add:::607.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e86) -- 0x55cac468f9b0 con 0x55cac4695830 2024-08-03T20:55:12.369 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.368+0000 7fbcbe200640 1 -- 172.21.15.32:0/801923352 <== osd.0 v2:172.21.15.32:6816/3066156116 35 ==== osd_op_reply(56 607.00000000 [omap-set-header] v86'20 uv20 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fbcac00f500 con 0x55cac4695830 2024-08-03T20:55:12.369 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.368+0000 7fbcc5a28f40 1 --2- 172.21.15.32:0/801923352 >> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] conn(0x55cac473f020 0x55cac473f430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:55:12.369 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.368+0000 7fbcc5a28f40 1 -- 172.21.15.32:0/801923352 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:57 12.6 12:654134d2:::mds0_openfiles.0:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e86) -- 0x55cac4741bd0 con 0x55cac473f020 2024-08-03T20:55:12.369 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.368+0000 7fbcbe200640 1 --2- 172.21.15.32:0/801923352 >> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] conn(0x55cac473f020 0x55cac473f430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:55:12.370 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.369+0000 7fbcbe200640 1 --2- 172.21.15.32:0/801923352 >> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] conn(0x55cac473f020 0x55cac473f430 crc :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:55:12.371 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.370+0000 7fbcbe200640 1 -- 172.21.15.32:0/801923352 <== osd.1 v2:172.21.15.32:6824/1551241759 1 ==== osd_op_reply(57 mds0_openfiles.0 [create] v86'16 uv15 ondisk = -17 ((17) File exists)) ==== 160+0+0 (crc 0 0 0) 0x7fbcac00f500 con 0x55cac473f020 2024-08-03T20:55:12.371 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.370+0000 7fbcc5a28f40 1 -- 172.21.15.32:0/801923352 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:58 12.6 12:654134d2:::mds0_openfiles.0:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e86) -- 0x55cac4741bd0 con 0x55cac473f020 2024-08-03T20:55:12.373 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.371+0000 7fbcbe200640 1 -- 172.21.15.32:0/801923352 <== osd.1 v2:172.21.15.32:6824/1551241759 2 ==== osd_op_reply(58 mds0_openfiles.0 [delete] v86'17 uv17 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7fbcac00f500 con 0x55cac473f020 2024-08-03T20:55:12.373 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.371+0000 7fbcc5a28f40 1 -- 172.21.15.32:0/801923352 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:59 12.6 12:654134d2:::mds0_openfiles.0:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e86) -- 0x55cac4741bd0 con 0x55cac473f020 2024-08-03T20:55:12.374 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.373+0000 7fbcbe200640 1 -- 172.21.15.32:0/801923352 <== osd.1 v2:172.21.15.32:6824/1551241759 3 ==== osd_op_reply(59 mds0_openfiles.0 [create] v86'18 uv18 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7fbcac00f500 con 0x55cac473f020 2024-08-03T20:55:12.374 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds0_openfiles.0:head# 2024-08-03T20:55:12.374 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.373+0000 7fbcc5a28f40 1 -- 172.21.15.32:0/801923352 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:60 12.6 12:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e86) -- 0x55cac4741fc0 con 0x55cac473f020 2024-08-03T20:55:12.375 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.374+0000 7fbcbe200640 1 -- 172.21.15.32:0/801923352 <== osd.1 v2:172.21.15.32:6824/1551241759 4 ==== osd_op_reply(60 mds0_openfiles.0 [omap-set-header] v86'19 uv19 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7fbcac00f500 con 0x55cac473f020 2024-08-03T20:55:12.375 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.374+0000 7fbcc5a28f40 1 -- 172.21.15.32:0/801923352 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:61 12.e 12:717a0223:::608.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e86) -- 0x55cac4741fc0 con 0x55cac4695830 2024-08-03T20:55:12.377 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.376+0000 7fbcbe200640 1 -- 172.21.15.32:0/801923352 <== osd.0 v2:172.21.15.32:6816/3066156116 36 ==== osd_op_reply(61 608.00000000 [create] v86'15 uv14 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fbcac00f500 con 0x55cac4695830 2024-08-03T20:55:12.377 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.376+0000 7fbcc5a28f40 1 -- 172.21.15.32:0/801923352 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:62 12.e 12:717a0223:::608.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e86) -- 0x55cac4741fc0 con 0x55cac4695830 2024-08-03T20:55:12.378 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.377+0000 7fbcbe200640 1 -- 172.21.15.32:0/801923352 <== osd.0 v2:172.21.15.32:6816/3066156116 37 ==== osd_op_reply(62 608.00000000 [delete] v86'16 uv16 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fbcac00f500 con 0x55cac4695830 2024-08-03T20:55:12.378 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.377+0000 7fbcc5a28f40 1 -- 172.21.15.32:0/801923352 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:63 12.e 12:717a0223:::608.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e86) -- 0x55cac4741fc0 con 0x55cac4695830 2024-08-03T20:55:12.380 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::608.00000000:head# 2024-08-03T20:55:12.380 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.379+0000 7fbcbe200640 1 -- 172.21.15.32:0/801923352 <== osd.0 v2:172.21.15.32:6816/3066156116 38 ==== osd_op_reply(63 608.00000000 [create] v86'17 uv17 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fbcac00f500 con 0x55cac4695830 2024-08-03T20:55:12.380 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.379+0000 7fbcc5a28f40 1 -- 172.21.15.32:0/801923352 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:64 12.e 12:717a0223:::608.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e86) -- 0x55cac4742fd0 con 0x55cac4695830 2024-08-03T20:55:12.381 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.380+0000 7fbcbe200640 1 -- 172.21.15.32:0/801923352 <== osd.0 v2:172.21.15.32:6816/3066156116 39 ==== osd_op_reply(64 608.00000000 [setxattr (30)] v86'18 uv18 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fbcac00f500 con 0x55cac4695830 2024-08-03T20:55:12.381 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.380+0000 7fbcc5a28f40 1 -- 172.21.15.32:0/801923352 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:65 12.e 12:717a0223:::608.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e86) -- 0x55cac4742fd0 con 0x55cac4695830 2024-08-03T20:55:12.383 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.382+0000 7fbcbe200640 1 -- 172.21.15.32:0/801923352 <== osd.0 v2:172.21.15.32:6816/3066156116 40 ==== osd_op_reply(65 608.00000000 [setxattr (62)] v86'19 uv19 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fbcac00f500 con 0x55cac4695830 2024-08-03T20:55:12.383 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.382+0000 7fbcc5a28f40 1 -- 172.21.15.32:0/801923352 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:66 12.e 12:717a0223:::608.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e86) -- 0x55cac4741fc0 con 0x55cac4695830 2024-08-03T20:55:12.385 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.383+0000 7fbcbe200640 1 -- 172.21.15.32:0/801923352 <== osd.0 v2:172.21.15.32:6816/3066156116 41 ==== osd_op_reply(66 608.00000000 [omap-set-header] v86'20 uv20 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fbcac00f500 con 0x55cac4695830 2024-08-03T20:55:12.385 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.383+0000 7fbcc5a28f40 1 --2- 172.21.15.32:0/801923352 >> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] conn(0x55cac47424d0 0x55cac47448f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:55:12.385 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.383+0000 7fbcc5a28f40 1 -- 172.21.15.32:0/801923352 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:67 12.15 12:a93a17c2:::604.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e86) -- 0x55cac4744e30 con 0x55cac47424d0 2024-08-03T20:55:12.385 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.384+0000 7fbcbf600640 1 --2- 172.21.15.32:0/801923352 >> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] conn(0x55cac47424d0 0x55cac47448f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:55:12.386 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.385+0000 7fbcbf600640 1 --2- 172.21.15.32:0/801923352 >> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] conn(0x55cac47424d0 0x55cac47448f0 crc :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:55:12.388 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.387+0000 7fbcbf600640 1 -- 172.21.15.32:0/801923352 <== osd.6 v2:172.21.15.118:6825/801947386 1 ==== osd_op_reply(67 604.00000000 [create] v86'26 uv20 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fbcb8003040 con 0x55cac47424d0 2024-08-03T20:55:12.388 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.387+0000 7fbcc5a28f40 1 -- 172.21.15.32:0/801923352 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:68 12.15 12:a93a17c2:::604.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e86) -- 0x55cac4745320 con 0x55cac47424d0 2024-08-03T20:55:12.390 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.388+0000 7fbcbf600640 1 -- 172.21.15.32:0/801923352 <== osd.6 v2:172.21.15.118:6825/801947386 2 ==== osd_op_reply(68 604.00000000 [delete] v86'27 uv27 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fbcb8003040 con 0x55cac47424d0 2024-08-03T20:55:12.390 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.388+0000 7fbcc5a28f40 1 -- 172.21.15.32:0/801923352 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:69 12.15 12:a93a17c2:::604.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e86) -- 0x55cac4745800 con 0x55cac47424d0 2024-08-03T20:55:12.391 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.390+0000 7fbcbf600640 1 -- 172.21.15.32:0/801923352 <== osd.6 v2:172.21.15.118:6825/801947386 3 ==== osd_op_reply(69 604.00000000 [create] v86'28 uv28 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fbcb8003040 con 0x55cac47424d0 2024-08-03T20:55:12.391 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::604.00000000:head# 2024-08-03T20:55:12.392 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.390+0000 7fbcc5a28f40 1 -- 172.21.15.32:0/801923352 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:70 12.15 12:a93a17c2:::604.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e86) -- 0x55cac47470b0 con 0x55cac47424d0 2024-08-03T20:55:12.393 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.391+0000 7fbcbf600640 1 -- 172.21.15.32:0/801923352 <== osd.6 v2:172.21.15.118:6825/801947386 4 ==== osd_op_reply(70 604.00000000 [setxattr (30)] v86'29 uv29 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fbcb8003040 con 0x55cac47424d0 2024-08-03T20:55:12.393 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.391+0000 7fbcc5a28f40 1 -- 172.21.15.32:0/801923352 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:71 12.15 12:a93a17c2:::604.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e86) -- 0x55cac4747450 con 0x55cac47424d0 2024-08-03T20:55:12.394 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.393+0000 7fbcbf600640 1 -- 172.21.15.32:0/801923352 <== osd.6 v2:172.21.15.118:6825/801947386 5 ==== osd_op_reply(71 604.00000000 [setxattr (62)] v86'30 uv30 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fbcb8003040 con 0x55cac47424d0 2024-08-03T20:55:12.394 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.393+0000 7fbcc5a28f40 1 -- 172.21.15.32:0/801923352 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:72 12.15 12:a93a17c2:::604.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e86) -- 0x55cac47460a0 con 0x55cac47424d0 2024-08-03T20:55:12.396 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.394+0000 7fbcbf600640 1 -- 172.21.15.32:0/801923352 <== osd.6 v2:172.21.15.118:6825/801947386 6 ==== osd_op_reply(72 604.00000000 [omap-set-header] v86'31 uv31 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fbcb8003040 con 0x55cac47424d0 2024-08-03T20:55:12.396 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.394+0000 7fbcc5a28f40 1 -- 172.21.15.32:0/801923352 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:73 12.15 12:aa448500:::500.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e86) -- 0x55cac4746490 con 0x55cac47424d0 2024-08-03T20:55:12.397 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.396+0000 7fbcbf600640 1 -- 172.21.15.32:0/801923352 <== osd.6 v2:172.21.15.118:6825/801947386 7 ==== osd_op_reply(73 500.00000000 [create] v86'32 uv25 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fbcb8003040 con 0x55cac47424d0 2024-08-03T20:55:12.397 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.396+0000 7fbcc5a28f40 1 -- 172.21.15.32:0/801923352 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:74 12.15 12:aa448500:::500.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e86) -- 0x55cac4746830 con 0x55cac47424d0 2024-08-03T20:55:12.399 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.398+0000 7fbcbf600640 1 -- 172.21.15.32:0/801923352 <== osd.6 v2:172.21.15.118:6825/801947386 8 ==== osd_op_reply(74 500.00000000 [delete] v86'33 uv33 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fbcb8003040 con 0x55cac47424d0 2024-08-03T20:55:12.399 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.398+0000 7fbcc5a28f40 1 -- 172.21.15.32:0/801923352 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:75 12.15 12:aa448500:::500.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e86) -- 0x55cac4746830 con 0x55cac47424d0 2024-08-03T20:55:12.400 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.399+0000 7fbcbf600640 1 -- 172.21.15.32:0/801923352 <== osd.6 v2:172.21.15.118:6825/801947386 9 ==== osd_op_reply(75 500.00000000 [create] v86'34 uv34 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fbcb8003040 con 0x55cac47424d0 2024-08-03T20:55:12.401 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::500.00000000:head# 2024-08-03T20:55:12.401 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.399+0000 7fbcc5a28f40 1 -- 172.21.15.32:0/801923352 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:76 12.15 12:aa448500:::500.00000000:head [write 0~90 in=90b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e86) -- 0x55cac4746830 con 0x55cac47424d0 2024-08-03T20:55:12.402 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.401+0000 7fbcbf600640 1 -- 172.21.15.32:0/801923352 <== osd.6 v2:172.21.15.118:6825/801947386 10 ==== osd_op_reply(76 500.00000000 [write 0~90] v86'35 uv35 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fbcb8003040 con 0x55cac47424d0 2024-08-03T20:55:12.402 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.401+0000 7fbcc5a28f40 1 -- 172.21.15.32:0/801923352 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:77 12.15 12:aa448500:::500.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e86) -- 0x55cac4746830 con 0x55cac47424d0 2024-08-03T20:55:12.403 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.402+0000 7fbcbf600640 1 -- 172.21.15.32:0/801923352 <== osd.6 v2:172.21.15.118:6825/801947386 11 ==== osd_op_reply(77 500.00000000 [omap-set-header] v86'36 uv36 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fbcb8003040 con 0x55cac47424d0 2024-08-03T20:55:12.404 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.402+0000 7fbcc5a28f40 1 --2- 172.21.15.32:0/801923352 >> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] conn(0x55cac4746c20 0x55cac4749d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:55:12.404 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.402+0000 7fbcc5a28f40 1 -- 172.21.15.32:0/801923352 --> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] -- osd_op(unknown.0.0:78 12.d 12:b50e409b:::mds_snaptable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e86) -- 0x55cac474a250 con 0x55cac4746c20 2024-08-03T20:55:12.404 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.403+0000 7fbcbec00640 1 --2- 172.21.15.32:0/801923352 >> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] conn(0x55cac4746c20 0x55cac4749d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:55:12.405 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.403+0000 7fbcbec00640 1 --2- 172.21.15.32:0/801923352 >> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] conn(0x55cac4746c20 0x55cac4749d10 crc :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:55:12.406 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.405+0000 7fbcbec00640 1 -- 172.21.15.32:0/801923352 <== osd.7 v2:172.21.15.118:6818/653000048 1 ==== osd_op_reply(78 mds_snaptable [create] v86'12 uv11 ondisk = -17 ((17) File exists)) ==== 157+0+0 (crc 0 0 0) 0x7fbcb0057480 con 0x55cac4746c20 2024-08-03T20:55:12.406 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.405+0000 7fbcc5a28f40 1 -- 172.21.15.32:0/801923352 --> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] -- osd_op(unknown.0.0:79 12.d 12:b50e409b:::mds_snaptable:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e86) -- 0x55cac474a250 con 0x55cac4746c20 2024-08-03T20:55:12.408 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.406+0000 7fbcbec00640 1 -- 172.21.15.32:0/801923352 <== osd.7 v2:172.21.15.118:6818/653000048 2 ==== osd_op_reply(79 mds_snaptable [delete] v86'13 uv13 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7fbcb0057480 con 0x55cac4746c20 2024-08-03T20:55:12.408 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.406+0000 7fbcc5a28f40 1 -- 172.21.15.32:0/801923352 --> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] -- osd_op(unknown.0.0:80 12.d 12:b50e409b:::mds_snaptable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e86) -- 0x55cac474a250 con 0x55cac4746c20 2024-08-03T20:55:12.409 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds_snaptable:head# 2024-08-03T20:55:12.410 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.408+0000 7fbcbec00640 1 -- 172.21.15.32:0/801923352 <== osd.7 v2:172.21.15.118:6818/653000048 3 ==== osd_op_reply(80 mds_snaptable [create] v86'14 uv14 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7fbcb0057480 con 0x55cac4746c20 2024-08-03T20:55:12.410 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.408+0000 7fbcc5a28f40 1 -- 172.21.15.32:0/801923352 --> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] -- osd_op(unknown.0.0:81 12.d 12:b50e409b:::mds_snaptable:head [write 0~70 in=70b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e86) -- 0x55cac474a640 con 0x55cac4746c20 2024-08-03T20:55:12.411 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.410+0000 7fbcbec00640 1 -- 172.21.15.32:0/801923352 <== osd.7 v2:172.21.15.118:6818/653000048 4 ==== osd_op_reply(81 mds_snaptable [write 0~70] v86'15 uv15 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7fbcb0057480 con 0x55cac4746c20 2024-08-03T20:55:12.411 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.410+0000 7fbcc5a28f40 1 -- 172.21.15.32:0/801923352 --> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] -- osd_op(unknown.0.0:82 12.d 12:b50e409b:::mds_snaptable:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e86) -- 0x55cac474a640 con 0x55cac4746c20 2024-08-03T20:55:12.412 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.411+0000 7fbcbec00640 1 -- 172.21.15.32:0/801923352 <== osd.7 v2:172.21.15.118:6818/653000048 5 ==== osd_op_reply(82 mds_snaptable [omap-set-header] v86'16 uv16 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7fbcb0057480 con 0x55cac4746c20 2024-08-03T20:55:12.413 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.411+0000 7fbcc5a28f40 1 -- 172.21.15.32:0/801923352 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:83 12.1d 12:b871830b:::605.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e86) -- 0x55cac474a640 con 0x55cac4695830 2024-08-03T20:55:12.414 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.413+0000 7fbcbe200640 1 -- 172.21.15.32:0/801923352 <== osd.0 v2:172.21.15.32:6816/3066156116 42 ==== osd_op_reply(83 605.00000000 [create] v86'15 uv14 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fbcac00f500 con 0x55cac4695830 2024-08-03T20:55:12.414 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.413+0000 7fbcc5a28f40 1 -- 172.21.15.32:0/801923352 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:84 12.1d 12:b871830b:::605.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e86) -- 0x55cac474a640 con 0x55cac4695830 2024-08-03T20:55:12.415 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.414+0000 7fbcbe200640 1 -- 172.21.15.32:0/801923352 <== osd.0 v2:172.21.15.32:6816/3066156116 43 ==== osd_op_reply(84 605.00000000 [delete] v86'16 uv16 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fbcac00f500 con 0x55cac4695830 2024-08-03T20:55:12.416 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.414+0000 7fbcc5a28f40 1 -- 172.21.15.32:0/801923352 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:85 12.1d 12:b871830b:::605.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e86) -- 0x55cac474a640 con 0x55cac4695830 2024-08-03T20:55:12.417 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.416+0000 7fbcbe200640 1 -- 172.21.15.32:0/801923352 <== osd.0 v2:172.21.15.32:6816/3066156116 44 ==== osd_op_reply(85 605.00000000 [create] v86'17 uv17 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fbcac00f500 con 0x55cac4695830 2024-08-03T20:55:12.417 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::605.00000000:head# 2024-08-03T20:55:12.417 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.416+0000 7fbcc5a28f40 1 -- 172.21.15.32:0/801923352 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:86 12.1d 12:b871830b:::605.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e86) -- 0x55cac474b650 con 0x55cac4695830 2024-08-03T20:55:12.418 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.417+0000 7fbcbe200640 1 -- 172.21.15.32:0/801923352 <== osd.0 v2:172.21.15.32:6816/3066156116 45 ==== osd_op_reply(86 605.00000000 [setxattr (30)] v86'18 uv18 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fbcac00f500 con 0x55cac4695830 2024-08-03T20:55:12.418 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.417+0000 7fbcc5a28f40 1 -- 172.21.15.32:0/801923352 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:87 12.1d 12:b871830b:::605.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e86) -- 0x55cac474b650 con 0x55cac4695830 2024-08-03T20:55:12.419 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.418+0000 7fbcbe200640 1 -- 172.21.15.32:0/801923352 <== osd.0 v2:172.21.15.32:6816/3066156116 46 ==== osd_op_reply(87 605.00000000 [setxattr (62)] v86'19 uv19 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fbcac00f500 con 0x55cac4695830 2024-08-03T20:55:12.420 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.418+0000 7fbcc5a28f40 1 -- 172.21.15.32:0/801923352 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:88 12.1d 12:b871830b:::605.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e86) -- 0x55cac474a640 con 0x55cac4695830 2024-08-03T20:55:12.421 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.420+0000 7fbcbe200640 1 -- 172.21.15.32:0/801923352 <== osd.0 v2:172.21.15.32:6816/3066156116 47 ==== osd_op_reply(88 605.00000000 [omap-set-header] v86'20 uv20 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fbcac00f500 con 0x55cac4695830 2024-08-03T20:55:12.421 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.420+0000 7fbcc5a28f40 1 -- 172.21.15.32:0/801923352 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:89 12.13 12:c91d4a1d:::mds0_inotable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e86) -- 0x55cac474a640 con 0x55cac473f020 2024-08-03T20:55:12.422 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.421+0000 7fbcbe200640 1 -- 172.21.15.32:0/801923352 <== osd.1 v2:172.21.15.32:6824/1551241759 5 ==== osd_op_reply(89 mds0_inotable [create] v86'24 uv18 ondisk = -17 ((17) File exists)) ==== 157+0+0 (crc 0 0 0) 0x7fbcac00f500 con 0x55cac473f020 2024-08-03T20:55:12.422 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.421+0000 7fbcc5a28f40 1 -- 172.21.15.32:0/801923352 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:90 12.13 12:c91d4a1d:::mds0_inotable:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e86) -- 0x55cac474a640 con 0x55cac473f020 2024-08-03T20:55:12.424 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.422+0000 7fbcbe200640 1 -- 172.21.15.32:0/801923352 <== osd.1 v2:172.21.15.32:6824/1551241759 6 ==== osd_op_reply(90 mds0_inotable [delete] v86'25 uv25 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7fbcac00f500 con 0x55cac473f020 2024-08-03T20:55:12.424 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.422+0000 7fbcc5a28f40 1 -- 172.21.15.32:0/801923352 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:91 12.13 12:c91d4a1d:::mds0_inotable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e86) -- 0x55cac474a640 con 0x55cac473f020 2024-08-03T20:55:12.425 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.423+0000 7fbcbe200640 1 -- 172.21.15.32:0/801923352 <== osd.1 v2:172.21.15.32:6824/1551241759 7 ==== osd_op_reply(91 mds0_inotable [create] v86'26 uv26 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7fbcac00f500 con 0x55cac473f020 2024-08-03T20:55:12.425 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds0_inotable:head# 2024-08-03T20:55:12.425 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.424+0000 7fbcc5a28f40 1 -- 172.21.15.32:0/801923352 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:92 12.13 12:c91d4a1d:::mds0_inotable:head [write 0~34 in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e86) -- 0x55cac474a640 con 0x55cac473f020 2024-08-03T20:55:12.426 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.425+0000 7fbcbe200640 1 -- 172.21.15.32:0/801923352 <== osd.1 v2:172.21.15.32:6824/1551241759 8 ==== osd_op_reply(92 mds0_inotable [write 0~34] v86'27 uv27 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7fbcac00f500 con 0x55cac473f020 2024-08-03T20:55:12.426 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.425+0000 7fbcc5a28f40 1 -- 172.21.15.32:0/801923352 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:93 12.13 12:c91d4a1d:::mds0_inotable:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e86) -- 0x55cac474a640 con 0x55cac473f020 2024-08-03T20:55:12.427 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.426+0000 7fbcbe200640 1 -- 172.21.15.32:0/801923352 <== osd.1 v2:172.21.15.32:6824/1551241759 9 ==== osd_op_reply(93 mds0_inotable [omap-set-header] v86'28 uv28 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7fbcac00f500 con 0x55cac473f020 2024-08-03T20:55:12.428 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.426+0000 7fbcc5a28f40 1 -- 172.21.15.32:0/801923352 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:94 12.13 12:cd5a64a3:::100.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e86) -- 0x55cac474a640 con 0x55cac473f020 2024-08-03T20:55:12.429 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.427+0000 7fbcbe200640 1 -- 172.21.15.32:0/801923352 <== osd.1 v2:172.21.15.32:6824/1551241759 10 ==== osd_op_reply(94 100.00000000 [create] v86'29 uv23 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fbcac00f500 con 0x55cac473f020 2024-08-03T20:55:12.429 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.427+0000 7fbcc5a28f40 1 -- 172.21.15.32:0/801923352 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:95 12.13 12:cd5a64a3:::100.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e86) -- 0x55cac474a640 con 0x55cac473f020 2024-08-03T20:55:12.430 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.429+0000 7fbcbe200640 1 -- 172.21.15.32:0/801923352 <== osd.1 v2:172.21.15.32:6824/1551241759 11 ==== osd_op_reply(95 100.00000000 [delete] v86'30 uv30 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fbcac00f500 con 0x55cac473f020 2024-08-03T20:55:12.430 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.429+0000 7fbcc5a28f40 1 -- 172.21.15.32:0/801923352 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:96 12.13 12:cd5a64a3:::100.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e86) -- 0x55cac474a640 con 0x55cac473f020 2024-08-03T20:55:12.431 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.430+0000 7fbcbe200640 1 -- 172.21.15.32:0/801923352 <== osd.1 v2:172.21.15.32:6824/1551241759 12 ==== osd_op_reply(96 100.00000000 [create] v86'31 uv31 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fbcac00f500 con 0x55cac473f020 2024-08-03T20:55:12.431 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::100.00000000:head# 2024-08-03T20:55:12.432 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.430+0000 7fbcc5a28f40 1 -- 172.21.15.32:0/801923352 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:97 12.13 12:cd5a64a3:::100.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e86) -- 0x55cac474a640 con 0x55cac473f020 2024-08-03T20:55:12.432 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.431+0000 7fbcbe200640 1 -- 172.21.15.32:0/801923352 <== osd.1 v2:172.21.15.32:6824/1551241759 13 ==== osd_op_reply(97 100.00000000 [omap-set-header] v86'32 uv32 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fbcac00f500 con 0x55cac473f020 2024-08-03T20:55:12.432 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.431+0000 7fbcc5a28f40 1 -- 172.21.15.32:0/801923352 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:98 12.13 12:cd5a64a3:::100.00000000:head [omap-set-vals in=5194b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e86) -- 0x55cac474d450 con 0x55cac473f020 2024-08-03T20:55:12.434 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.432+0000 7fbcbe200640 1 -- 172.21.15.32:0/801923352 <== osd.1 v2:172.21.15.32:6824/1551241759 14 ==== osd_op_reply(98 100.00000000 [omap-set-vals] v86'33 uv33 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fbcac00f500 con 0x55cac473f020 2024-08-03T20:55:12.434 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.432+0000 7fbcc5a28f40 1 --2- 172.21.15.32:0/801923352 >> [v2:172.21.15.32:6800/9088303,v1:172.21.15.32:6801/9088303] conn(0x55cac474b000 0x55cac474d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:55:12.434 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.432+0000 7fbcc5a28f40 1 -- 172.21.15.32:0/801923352 --> [v2:172.21.15.32:6800/9088303,v1:172.21.15.32:6801/9088303] -- osd_op(unknown.0.0:99 12.b 12:d0630e4c:::mds0_sessionmap:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e86) -- 0x55cac474d420 con 0x55cac474b000 2024-08-03T20:55:12.434 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.433+0000 7fbcbe200640 1 --2- 172.21.15.32:0/801923352 >> [v2:172.21.15.32:6800/9088303,v1:172.21.15.32:6801/9088303] conn(0x55cac474b000 0x55cac474d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:55:12.435 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.433+0000 7fbcbe200640 1 --2- 172.21.15.32:0/801923352 >> [v2:172.21.15.32:6800/9088303,v1:172.21.15.32:6801/9088303] conn(0x55cac474b000 0x55cac474d990 crc :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:55:12.436 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.435+0000 7fbcbe200640 1 -- 172.21.15.32:0/801923352 <== osd.3 v2:172.21.15.32:6800/9088303 1 ==== osd_op_reply(99 mds0_sessionmap [create] v86'14 uv12 ondisk = -17 ((17) File exists)) ==== 159+0+0 (crc 0 0 0) 0x7fbcac00f500 con 0x55cac474b000 2024-08-03T20:55:12.436 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.435+0000 7fbcc5a28f40 1 -- 172.21.15.32:0/801923352 --> [v2:172.21.15.32:6800/9088303,v1:172.21.15.32:6801/9088303] -- osd_op(unknown.0.0:100 12.b 12:d0630e4c:::mds0_sessionmap:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e86) -- 0x55cac474d420 con 0x55cac474b000 2024-08-03T20:55:12.437 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.436+0000 7fbcbe200640 1 -- 172.21.15.32:0/801923352 <== osd.3 v2:172.21.15.32:6800/9088303 2 ==== osd_op_reply(100 mds0_sessionmap [delete] v86'15 uv15 ondisk = 0) ==== 159+0+0 (crc 0 0 0) 0x7fbcac00f500 con 0x55cac474b000 2024-08-03T20:55:12.438 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.436+0000 7fbcc5a28f40 1 -- 172.21.15.32:0/801923352 --> [v2:172.21.15.32:6800/9088303,v1:172.21.15.32:6801/9088303] -- osd_op(unknown.0.0:101 12.b 12:d0630e4c:::mds0_sessionmap:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e86) -- 0x55cac474d420 con 0x55cac474b000 2024-08-03T20:55:12.439 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.437+0000 7fbcbe200640 1 -- 172.21.15.32:0/801923352 <== osd.3 v2:172.21.15.32:6800/9088303 3 ==== osd_op_reply(101 mds0_sessionmap [create] v86'16 uv16 ondisk = 0) ==== 159+0+0 (crc 0 0 0) 0x7fbcac00f500 con 0x55cac474b000 2024-08-03T20:55:12.439 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds0_sessionmap:head# 2024-08-03T20:55:12.439 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.437+0000 7fbcc5a28f40 1 -- 172.21.15.32:0/801923352 --> [v2:172.21.15.32:6800/9088303,v1:172.21.15.32:6801/9088303] -- osd_op(unknown.0.0:102 12.b 12:d0630e4c:::mds0_sessionmap:head [omap-set-header in=14b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e86) -- 0x55cac474d420 con 0x55cac474b000 2024-08-03T20:55:12.440 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.439+0000 7fbcbe200640 1 -- 172.21.15.32:0/801923352 <== osd.3 v2:172.21.15.32:6800/9088303 4 ==== osd_op_reply(102 mds0_sessionmap [omap-set-header] v86'17 uv17 ondisk = 0) ==== 159+0+0 (crc 0 0 0) 0x7fbcac00f500 con 0x55cac474b000 2024-08-03T20:55:12.440 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.439+0000 7fbcc5a28f40 1 -- 172.21.15.32:0/801923352 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:103 12.7 12:e0b41d06:::609.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e86) -- 0x55cac474d420 con 0x55cac473f020 2024-08-03T20:55:12.442 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.440+0000 7fbcbe200640 1 -- 172.21.15.32:0/801923352 <== osd.1 v2:172.21.15.32:6824/1551241759 15 ==== osd_op_reply(103 609.00000000 [create] v86'15 uv14 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fbcac00f500 con 0x55cac473f020 2024-08-03T20:55:12.442 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.440+0000 7fbcc5a28f40 1 -- 172.21.15.32:0/801923352 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:104 12.7 12:e0b41d06:::609.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e86) -- 0x55cac474d420 con 0x55cac473f020 2024-08-03T20:55:12.443 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.442+0000 7fbcbe200640 1 -- 172.21.15.32:0/801923352 <== osd.1 v2:172.21.15.32:6824/1551241759 16 ==== osd_op_reply(104 609.00000000 [delete] v86'16 uv16 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fbcac00f500 con 0x55cac473f020 2024-08-03T20:55:12.443 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.442+0000 7fbcc5a28f40 1 -- 172.21.15.32:0/801923352 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:105 12.7 12:e0b41d06:::609.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e86) -- 0x55cac474d420 con 0x55cac473f020 2024-08-03T20:55:12.445 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::609.00000000:head# 2024-08-03T20:55:12.445 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.444+0000 7fbcbe200640 1 -- 172.21.15.32:0/801923352 <== osd.1 v2:172.21.15.32:6824/1551241759 17 ==== osd_op_reply(105 609.00000000 [create] v86'17 uv17 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fbcac00f500 con 0x55cac473f020 2024-08-03T20:55:12.445 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.444+0000 7fbcc5a28f40 1 -- 172.21.15.32:0/801923352 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:106 12.7 12:e0b41d06:::609.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e86) -- 0x55cac474d420 con 0x55cac473f020 2024-08-03T20:55:12.446 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.445+0000 7fbcbe200640 1 -- 172.21.15.32:0/801923352 <== osd.1 v2:172.21.15.32:6824/1551241759 18 ==== osd_op_reply(106 609.00000000 [setxattr (30)] v86'18 uv18 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fbcac00f500 con 0x55cac473f020 2024-08-03T20:55:12.447 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.445+0000 7fbcc5a28f40 1 -- 172.21.15.32:0/801923352 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:107 12.7 12:e0b41d06:::609.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e86) -- 0x55cac474d420 con 0x55cac473f020 2024-08-03T20:55:12.448 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.446+0000 7fbcbe200640 1 -- 172.21.15.32:0/801923352 <== osd.1 v2:172.21.15.32:6824/1551241759 19 ==== osd_op_reply(107 609.00000000 [setxattr (62)] v86'19 uv19 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fbcac00f500 con 0x55cac473f020 2024-08-03T20:55:12.448 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.446+0000 7fbcc5a28f40 1 -- 172.21.15.32:0/801923352 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:108 12.7 12:e0b41d06:::609.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e86) -- 0x55cac474d420 con 0x55cac473f020 2024-08-03T20:55:12.449 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.448+0000 7fbcbe200640 1 -- 172.21.15.32:0/801923352 <== osd.1 v2:172.21.15.32:6824/1551241759 20 ==== osd_op_reply(108 609.00000000 [omap-set-header] v86'20 uv20 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fbcac00f500 con 0x55cac473f020 2024-08-03T20:55:12.449 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.448+0000 7fbcc5a28f40 1 -- 172.21.15.32:0/801923352 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:109 12.f 12:f1f69726:::400.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e86) -- 0x55cac474d420 con 0x55cac47424d0 2024-08-03T20:55:12.451 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.449+0000 7fbcbf600640 1 -- 172.21.15.32:0/801923352 <== osd.6 v2:172.21.15.118:6825/801947386 12 ==== osd_op_reply(109 400.00000000 [create] v86'12 uv11 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fbcb8003040 con 0x55cac47424d0 2024-08-03T20:55:12.451 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.450+0000 7fbcc5a28f40 1 -- 172.21.15.32:0/801923352 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:110 12.f 12:f1f69726:::400.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e86) -- 0x55cac474d420 con 0x55cac47424d0 2024-08-03T20:55:12.452 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.451+0000 7fbcbf600640 1 -- 172.21.15.32:0/801923352 <== osd.6 v2:172.21.15.118:6825/801947386 13 ==== osd_op_reply(110 400.00000000 [delete] v86'13 uv13 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fbcb8003040 con 0x55cac47424d0 2024-08-03T20:55:12.453 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.451+0000 7fbcc5a28f40 1 -- 172.21.15.32:0/801923352 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:111 12.f 12:f1f69726:::400.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e86) -- 0x55cac474d420 con 0x55cac47424d0 2024-08-03T20:55:12.454 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::400.00000000:head# 2024-08-03T20:55:12.455 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.453+0000 7fbcbf600640 1 -- 172.21.15.32:0/801923352 <== osd.6 v2:172.21.15.118:6825/801947386 14 ==== osd_op_reply(111 400.00000000 [create] v86'14 uv14 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fbcb8003040 con 0x55cac47424d0 2024-08-03T20:55:12.455 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.453+0000 7fbcc5a28f40 1 -- 172.21.15.32:0/801923352 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:112 12.f 12:f1f69726:::400.00000000:head [write 0~22 in=22b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e86) -- 0x55cac474d420 con 0x55cac47424d0 2024-08-03T20:55:12.456 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.455+0000 7fbcbf600640 1 -- 172.21.15.32:0/801923352 <== osd.6 v2:172.21.15.118:6825/801947386 15 ==== osd_op_reply(112 400.00000000 [write 0~22] v86'15 uv15 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fbcb8003040 con 0x55cac47424d0 2024-08-03T20:55:12.456 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.455+0000 7fbcc5a28f40 1 -- 172.21.15.32:0/801923352 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:113 12.f 12:f1f69726:::400.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e86) -- 0x55cac474d420 con 0x55cac47424d0 2024-08-03T20:55:12.458 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.456+0000 7fbcbf600640 1 -- 172.21.15.32:0/801923352 <== osd.6 v2:172.21.15.118:6825/801947386 16 ==== osd_op_reply(113 400.00000000 [omap-set-header] v86'16 uv16 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fbcb8003040 con 0x55cac47424d0 2024-08-03T20:55:12.458 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.456+0000 7fbcc5a28f40 1 -- 172.21.15.32:0/801923352 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:114 12.1f 12:fe07bba1:::100.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e86) -- 0x55cac474d420 con 0x55cac47424d0 2024-08-03T20:55:12.459 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.458+0000 7fbcbf600640 1 -- 172.21.15.32:0/801923352 <== osd.6 v2:172.21.15.118:6825/801947386 17 ==== osd_op_reply(114 100.00000000.inode [create] v86'33 uv24 ondisk = -17 ((17) File exists)) ==== 162+0+0 (crc 0 0 0) 0x7fbcb8003040 con 0x55cac47424d0 2024-08-03T20:55:12.459 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.458+0000 7fbcc5a28f40 1 -- 172.21.15.32:0/801923352 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:115 12.1f 12:fe07bba1:::100.00000000.inode:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e86) -- 0x55cac474d420 con 0x55cac47424d0 2024-08-03T20:55:12.461 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.459+0000 7fbcbf600640 1 -- 172.21.15.32:0/801923352 <== osd.6 v2:172.21.15.118:6825/801947386 18 ==== osd_op_reply(115 100.00000000.inode [delete] v86'34 uv34 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7fbcb8003040 con 0x55cac47424d0 2024-08-03T20:55:12.461 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.459+0000 7fbcc5a28f40 1 -- 172.21.15.32:0/801923352 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:116 12.1f 12:fe07bba1:::100.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e86) -- 0x55cac474d420 con 0x55cac47424d0 2024-08-03T20:55:12.462 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.461+0000 7fbcbf600640 1 -- 172.21.15.32:0/801923352 <== osd.6 v2:172.21.15.118:6825/801947386 19 ==== osd_op_reply(116 100.00000000.inode [create] v86'35 uv35 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7fbcb8003040 con 0x55cac47424d0 2024-08-03T20:55:12.462 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::100.00000000.inode:head# 2024-08-03T20:55:12.463 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.461+0000 7fbcc5a28f40 1 -- 172.21.15.32:0/801923352 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:117 12.1f 12:fe07bba1:::100.00000000.inode:head [write 0~582 in=582b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e86) -- 0x55cac474d420 con 0x55cac47424d0 2024-08-03T20:55:12.464 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.463+0000 7fbcbf600640 1 -- 172.21.15.32:0/801923352 <== osd.6 v2:172.21.15.118:6825/801947386 20 ==== osd_op_reply(117 100.00000000.inode [write 0~582] v86'36 uv36 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7fbcb8003040 con 0x55cac47424d0 2024-08-03T20:55:12.464 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.463+0000 7fbcc5a28f40 1 -- 172.21.15.32:0/801923352 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:118 12.1f 12:fe07bba1:::100.00000000.inode:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e86) -- 0x55cac474d420 con 0x55cac47424d0 2024-08-03T20:55:12.465 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.464+0000 7fbcbf600640 1 -- 172.21.15.32:0/801923352 <== osd.6 v2:172.21.15.118:6825/801947386 21 ==== osd_op_reply(118 100.00000000.inode [omap-set-header] v86'37 uv37 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7fbcb8003040 con 0x55cac47424d0 2024-08-03T20:55:12.465 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.464+0000 7fbcc5a28f40 1 -- 172.21.15.32:0/801923352 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:119 12.1f 12:ff5b34d6:::1.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e86) -- 0x55cac474d420 con 0x55cac47424d0 2024-08-03T20:55:12.467 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.465+0000 7fbcbf600640 1 -- 172.21.15.32:0/801923352 <== osd.6 v2:172.21.15.118:6825/801947386 22 ==== osd_op_reply(119 1.00000000 [create] v86'38 uv31 ondisk = -17 ((17) File exists)) ==== 154+0+0 (crc 0 0 0) 0x7fbcb8003040 con 0x55cac47424d0 2024-08-03T20:55:12.467 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.465+0000 7fbcc5a28f40 1 -- 172.21.15.32:0/801923352 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:120 12.1f 12:ff5b34d6:::1.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e86) -- 0x55cac474d420 con 0x55cac47424d0 2024-08-03T20:55:12.468 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.467+0000 7fbcbf600640 1 -- 172.21.15.32:0/801923352 <== osd.6 v2:172.21.15.118:6825/801947386 23 ==== osd_op_reply(120 1.00000000 [delete] v86'39 uv39 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7fbcb8003040 con 0x55cac47424d0 2024-08-03T20:55:12.468 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.467+0000 7fbcc5a28f40 1 -- 172.21.15.32:0/801923352 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:121 12.1f 12:ff5b34d6:::1.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e86) -- 0x55cac474d420 con 0x55cac47424d0 2024-08-03T20:55:12.469 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.468+0000 7fbcbf600640 1 -- 172.21.15.32:0/801923352 <== osd.6 v2:172.21.15.118:6825/801947386 24 ==== osd_op_reply(121 1.00000000 [create] v86'40 uv40 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7fbcb8003040 con 0x55cac47424d0 2024-08-03T20:55:12.470 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::1.00000000:head# 2024-08-03T20:55:12.470 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.468+0000 7fbcc5a28f40 1 -- 172.21.15.32:0/801923352 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:122 12.1f 12:ff5b34d6:::1.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e86) -- 0x55cac474d420 con 0x55cac47424d0 2024-08-03T20:55:12.471 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.470+0000 7fbcbf600640 1 -- 172.21.15.32:0/801923352 <== osd.6 v2:172.21.15.118:6825/801947386 25 ==== osd_op_reply(122 1.00000000 [setxattr (30)] v86'41 uv41 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7fbcb8003040 con 0x55cac47424d0 2024-08-03T20:55:12.471 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.470+0000 7fbcc5a28f40 1 -- 172.21.15.32:0/801923352 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:123 12.1f 12:ff5b34d6:::1.00000000:head [setxattr parent (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e86) -- 0x55cac474d420 con 0x55cac47424d0 2024-08-03T20:55:12.473 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.472+0000 7fbcbf600640 1 -- 172.21.15.32:0/801923352 <== osd.6 v2:172.21.15.118:6825/801947386 26 ==== osd_op_reply(123 1.00000000 [setxattr (30)] v86'42 uv42 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7fbcb8003040 con 0x55cac47424d0 2024-08-03T20:55:12.473 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.472+0000 7fbcc5a28f40 1 -- 172.21.15.32:0/801923352 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:124 12.1f 12:ff5b34d6:::1.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e86) -- 0x55cac474d420 con 0x55cac47424d0 2024-08-03T20:55:12.475 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.473+0000 7fbcbf600640 1 -- 172.21.15.32:0/801923352 <== osd.6 v2:172.21.15.118:6825/801947386 27 ==== osd_op_reply(124 1.00000000 [omap-set-header] v86'43 uv43 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7fbcb8003040 con 0x55cac47424d0 2024-08-03T20:55:12.475 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.473+0000 7fbcc5a28f40 1 -- 172.21.15.32:0/801923352 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:125 12.1f 12:ff5b34d6:::1.00000000:head [omap-set-vals in=584b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e86) -- 0x55cac474e3f0 con 0x55cac47424d0 2024-08-03T20:55:12.477 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.475+0000 7fbcbf600640 1 -- 172.21.15.32:0/801923352 <== osd.6 v2:172.21.15.118:6825/801947386 28 ==== osd_op_reply(125 1.00000000 [omap-set-vals] v86'44 uv44 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7fbcb8003040 con 0x55cac47424d0 2024-08-03T20:55:12.477 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.475+0000 7fbcc5a28f40 1 -- 172.21.15.32:0/801923352 >> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] conn(0x55cac4695830 msgr2=0x55cac4690290 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:55:12.477 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.475+0000 7fbcc5a28f40 1 --2- 172.21.15.32:0/801923352 >> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] conn(0x55cac4695830 0x55cac4690290 crc :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:12.477 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.475+0000 7fbcc5a28f40 1 -- 172.21.15.32:0/801923352 >> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] conn(0x55cac473f020 msgr2=0x55cac473f430 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:55:12.477 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.475+0000 7fbcc5a28f40 1 --2- 172.21.15.32:0/801923352 >> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] conn(0x55cac473f020 0x55cac473f430 crc :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:12.477 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.475+0000 7fbcc5a28f40 1 -- 172.21.15.32:0/801923352 >> [v2:172.21.15.32:6800/9088303,v1:172.21.15.32:6801/9088303] conn(0x55cac474b000 msgr2=0x55cac474d990 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:55:12.477 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.475+0000 7fbcc5a28f40 1 --2- 172.21.15.32:0/801923352 >> [v2:172.21.15.32:6800/9088303,v1:172.21.15.32:6801/9088303] conn(0x55cac474b000 0x55cac474d990 crc :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:12.478 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.475+0000 7fbcc5a28f40 1 -- 172.21.15.32:0/801923352 >> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] conn(0x55cac4739db0 msgr2=0x55cac473cf00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:55:12.478 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.475+0000 7fbcc5a28f40 1 --2- 172.21.15.32:0/801923352 >> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] conn(0x55cac4739db0 0x55cac473cf00 crc :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:12.478 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.475+0000 7fbcbf600640 1 -- 172.21.15.32:0/801923352 reap_dead start 2024-08-03T20:55:12.478 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.475+0000 7fbcc5a28f40 1 -- 172.21.15.32:0/801923352 >> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] conn(0x55cac47424d0 msgr2=0x55cac47448f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:55:12.478 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.475+0000 7fbcc5a28f40 1 --2- 172.21.15.32:0/801923352 >> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] conn(0x55cac47424d0 0x55cac47448f0 crc :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:12.478 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.475+0000 7fbcc5a28f40 1 -- 172.21.15.32:0/801923352 >> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] conn(0x55cac4746c20 msgr2=0x55cac4749d10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:55:12.478 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.475+0000 7fbcc5a28f40 1 --2- 172.21.15.32:0/801923352 >> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] conn(0x55cac4746c20 0x55cac4749d10 crc :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:12.478 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.475+0000 7fbcc5a28f40 1 -- 172.21.15.32:0/801923352 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fbca00759f0 msgr2=0x7fbca0077f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:55:12.478 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.475+0000 7fbcc5a28f40 1 --2- 172.21.15.32:0/801923352 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fbca00759f0 0x7fbca0077f00 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7fbcb0055000 tx=0x7fbcb0003020 comp rx=0 tx=0).stop 2024-08-03T20:55:12.478 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.475+0000 7fbcc5a28f40 1 -- 172.21.15.32:0/801923352 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55cac46963f0 msgr2=0x55cac4692030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:55:12.478 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.475+0000 7fbcc5a28f40 1 --2- 172.21.15.32:0/801923352 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55cac46963f0 0x55cac4692030 secure :-1 s=READY pgs=1141 cs=0 l=1 rev1=1 crypto rx=0x7fbcb80106f0 tx=0x7fbcb8010bb0 comp rx=0 tx=0).stop 2024-08-03T20:55:12.478 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.475+0000 7fbcbf600640 1 -- 172.21.15.32:0/801923352 reap_dead start 2024-08-03T20:55:12.478 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.475+0000 7fbcc5a28f40 1 -- 172.21.15.32:0/801923352 shutdown_connections 2024-08-03T20:55:12.479 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.475+0000 7fbcc5a28f40 1 -- 172.21.15.32:0/801923352 >> 172.21.15.32:0/801923352 conn(0x55cac44e1ac0 msgr2=0x55cac4692c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:55:12.479 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.475+0000 7fbcc5a28f40 1 -- 172.21.15.32:0/801923352 shutdown_connections 2024-08-03T20:55:12.479 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.475+0000 7fbcc5a28f40 1 -- 172.21.15.32:0/801923352 wait complete. 2024-08-03T20:55:12.492 DEBUG:teuthology.orchestra.run.smithi032:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata rm 1.00000000.inode 2024-08-03T20:55:12.674 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.673+0000 7f684a14df40 1 Processor -- start 2024-08-03T20:55:12.675 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.673+0000 7f684a14df40 1 -- start start 2024-08-03T20:55:12.677 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.674+0000 7f684a14df40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55bd92203360 0x55bd921fb610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:55:12.677 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.674+0000 7f684a14df40 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55bd9221a330 0x55bd9214ada0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:55:12.677 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.674+0000 7f684a14df40 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55bd922213f0 0x55bd922a9be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:55:12.677 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.674+0000 7f684a14df40 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55bd920f8bf0 con 0x55bd9221a330 2024-08-03T20:55:12.677 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.674+0000 7f684a14df40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x55bd921056b0 con 0x55bd92203360 2024-08-03T20:55:12.677 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.674+0000 7f684a14df40 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x55bd9209ae30 con 0x55bd922213f0 2024-08-03T20:55:12.677 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.674+0000 7f6842800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55bd9221a330 0x55bd9214ada0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:55:12.677 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.674+0000 7f6842800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55bd9221a330 0x55bd9214ada0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:50446/0 (socket says 172.21.15.32:50446) 2024-08-03T20:55:12.677 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.674+0000 7f6842800640 1 -- 172.21.15.32:0/1630836193 learned_addr learned my addr 172.21.15.32:0/1630836193 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:55:12.677 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.674+0000 7f6843c00640 1 --2- 172.21.15.32:0/1630836193 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55bd922213f0 0x55bd922a9be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:55:12.678 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.674+0000 7f6843200640 1 --2- 172.21.15.32:0/1630836193 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55bd92203360 0x55bd921fb610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:55:12.678 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.675+0000 7f6842800640 1 -- 172.21.15.32:0/1630836193 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55bd922213f0 msgr2=0x55bd922a9be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:55:12.678 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.675+0000 7f6842800640 1 --2- 172.21.15.32:0/1630836193 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55bd922213f0 0x55bd922a9be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:12.678 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.675+0000 7f6842800640 1 -- 172.21.15.32:0/1630836193 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55bd92203360 msgr2=0x55bd921fb610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:55:12.678 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.675+0000 7f6842800640 1 --2- 172.21.15.32:0/1630836193 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55bd92203360 0x55bd921fb610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:12.678 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.675+0000 7f6842800640 1 -- 172.21.15.32:0/1630836193 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bd9218b570 con 0x55bd9221a330 2024-08-03T20:55:12.678 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.675+0000 7f6842800640 1 --2- 172.21.15.32:0/1630836193 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55bd9221a330 0x55bd9214ada0 secure :-1 s=READY pgs=1144 cs=0 l=1 rev1=1 crypto rx=0x7f6830001d50 tx=0x7f683000b5b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4e48c0b5dba62d9 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:55:12.678 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.675+0000 7f6841400640 1 -- 172.21.15.32:0/1630836193 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f683000bd80 con 0x55bd9221a330 2024-08-03T20:55:12.678 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.675+0000 7f6841400640 1 -- 172.21.15.32:0/1630836193 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f683000f400 con 0x55bd9221a330 2024-08-03T20:55:12.678 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.675+0000 7f6841400640 1 -- 172.21.15.32:0/1630836193 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f683000f850 con 0x55bd9221a330 2024-08-03T20:55:12.679 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.676+0000 7f684a14df40 1 -- 172.21.15.32:0/1630836193 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55bd9221a330 msgr2=0x55bd9214ada0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:55:12.680 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.676+0000 7f684a14df40 1 --2- 172.21.15.32:0/1630836193 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55bd9221a330 0x55bd9214ada0 secure :-1 s=READY pgs=1144 cs=0 l=1 rev1=1 crypto rx=0x7f6830001d50 tx=0x7f683000b5b0 comp rx=0 tx=0).stop 2024-08-03T20:55:12.680 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.676+0000 7f684a14df40 1 -- 172.21.15.32:0/1630836193 shutdown_connections 2024-08-03T20:55:12.680 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.676+0000 7f684a14df40 1 --2- 172.21.15.32:0/1630836193 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55bd922213f0 0x55bd922a9be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:12.680 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.676+0000 7f684a14df40 1 --2- 172.21.15.32:0/1630836193 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55bd9221a330 0x55bd9214ada0 unknown :-1 s=CLOSED pgs=1144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:12.680 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.676+0000 7f684a14df40 1 --2- 172.21.15.32:0/1630836193 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55bd92203360 0x55bd921fb610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:12.680 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.676+0000 7f684a14df40 1 -- 172.21.15.32:0/1630836193 >> 172.21.15.32:0/1630836193 conn(0x55bd9206cac0 msgr2=0x55bd922a84c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:55:12.680 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.676+0000 7f684a14df40 1 -- 172.21.15.32:0/1630836193 shutdown_connections 2024-08-03T20:55:12.680 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.676+0000 7f684a14df40 1 -- 172.21.15.32:0/1630836193 wait complete. 2024-08-03T20:55:12.680 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.677+0000 7f684a14df40 1 Processor -- start 2024-08-03T20:55:12.680 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.677+0000 7f684a14df40 1 -- start start 2024-08-03T20:55:12.682 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.679+0000 7f684a14df40 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55bd92203360 0x55bd921f67e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:55:12.682 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.679+0000 7f684a14df40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55bd9221a330 0x55bd921fc6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:55:12.682 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.679+0000 7f684a14df40 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55bd922213f0 0x55bd921fcc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:55:12.682 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.679+0000 7f684a14df40 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55bd9218b0e0 con 0x55bd92203360 2024-08-03T20:55:12.682 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.679+0000 7f684a14df40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x55bd9214cb60 con 0x55bd9221a330 2024-08-03T20:55:12.683 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.679+0000 7f684a14df40 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x55bd92195d40 con 0x55bd922213f0 2024-08-03T20:55:12.683 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.679+0000 7f6843200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55bd92203360 0x55bd921f67e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:55:12.683 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.679+0000 7f6843200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55bd92203360 0x55bd921f67e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:50462/0 (socket says 172.21.15.32:50462) 2024-08-03T20:55:12.683 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.679+0000 7f6843200640 1 -- 172.21.15.32:0/1556939735 learned_addr learned my addr 172.21.15.32:0/1556939735 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:55:12.683 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.679+0000 7f6842800640 1 --2- 172.21.15.32:0/1556939735 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55bd9221a330 0x55bd921fc6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:55:12.683 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.679+0000 7f6843c00640 1 --2- 172.21.15.32:0/1556939735 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55bd922213f0 0x55bd921fcc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:55:12.683 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.679+0000 7f6843200640 1 -- 172.21.15.32:0/1556939735 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55bd922213f0 msgr2=0x55bd921fcc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:55:12.683 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.679+0000 7f6843200640 1 --2- 172.21.15.32:0/1556939735 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55bd922213f0 0x55bd921fcc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:12.683 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.679+0000 7f6843200640 1 -- 172.21.15.32:0/1556939735 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55bd9221a330 msgr2=0x55bd921fc6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:55:12.683 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.679+0000 7f6843200640 1 --2- 172.21.15.32:0/1556939735 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55bd9221a330 0x55bd921fc6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:12.683 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.679+0000 7f6843200640 1 -- 172.21.15.32:0/1556939735 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bd921f6d20 con 0x55bd92203360 2024-08-03T20:55:12.683 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.679+0000 7f6843c00640 1 --2- 172.21.15.32:0/1556939735 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55bd922213f0 0x55bd921fcc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-03T20:55:12.684 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.679+0000 7f6842800640 1 --2- 172.21.15.32:0/1556939735 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55bd9221a330 0x55bd921fc6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-03T20:55:12.684 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.680+0000 7f6843200640 1 --2- 172.21.15.32:0/1556939735 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55bd92203360 0x55bd921f67e0 secure :-1 s=READY pgs=1145 cs=0 l=1 rev1=1 crypto rx=0x7f683400efd0 tx=0x7f683400c7a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:55:12.684 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.680+0000 7f683be00640 1 -- 172.21.15.32:0/1556939735 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6834001d40 con 0x55bd92203360 2024-08-03T20:55:12.684 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.680+0000 7f684a14df40 1 -- 172.21.15.32:0/1556939735 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x55bd921fea60 con 0x55bd92203360 2024-08-03T20:55:12.684 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.680+0000 7f683be00640 1 -- 172.21.15.32:0/1556939735 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f6834013030 con 0x55bd92203360 2024-08-03T20:55:12.684 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.680+0000 7f683be00640 1 -- 172.21.15.32:0/1556939735 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6834002270 con 0x55bd92203360 2024-08-03T20:55:12.684 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.680+0000 7f684a14df40 1 -- 172.21.15.32:0/1556939735 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bd921ff020 con 0x55bd92203360 2024-08-03T20:55:12.685 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.682+0000 7f683be00640 1 -- 172.21.15.32:0/1556939735 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f6834002410 con 0x55bd92203360 2024-08-03T20:55:12.685 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.683+0000 7f683be00640 1 --2- 172.21.15.32:0/1556939735 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f6824075a20 0x7f6824077f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:55:12.685 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.683+0000 7f683be00640 1 -- 172.21.15.32:0/1556939735 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 5967+0+0 (secure 0 0 0) 0x7f68340980d0 con 0x55bd92203360 2024-08-03T20:55:12.685 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.683+0000 7f684a14df40 1 --2- 172.21.15.32:0/1556939735 >> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] conn(0x55bd921fa9e0 0x55bd9221ade0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:55:12.686 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.683+0000 7f684a14df40 1 -- 172.21.15.32:0/1556939735 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:1 12.14 12:287034c4:::1.00000000.inode:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e86) -- 0x55bd9221b320 con 0x55bd921fa9e0 2024-08-03T20:55:12.686 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.684+0000 7f6842800640 1 --2- 172.21.15.32:0/1556939735 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f6824075a20 0x7f6824077f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:55:12.686 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.684+0000 7f6843c00640 1 --2- 172.21.15.32:0/1556939735 >> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] conn(0x55bd921fa9e0 0x55bd9221ade0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:55:12.686 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.684+0000 7f6842800640 1 --2- 172.21.15.32:0/1556939735 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f6824075a20 0x7f6824077f30 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x55bd9214c090 tx=0x7f6830000ed0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:55:12.686 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.685+0000 7f6843c00640 1 --2- 172.21.15.32:0/1556939735 >> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] conn(0x55bd921fa9e0 0x55bd9221ade0 crc :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:55:12.688 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.686+0000 7f6843c00640 1 -- 172.21.15.32:0/1556939735 <== osd.5 v2:172.21.15.118:6810/2724891770 1 ==== osd_op_reply(1 1.00000000.inode [delete] v86'88 uv88 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f683c00b040 con 0x55bd921fa9e0 2024-08-03T20:55:12.688 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.686+0000 7f684a14df40 1 -- 172.21.15.32:0/1556939735 >> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] conn(0x55bd921fa9e0 msgr2=0x55bd9221ade0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:55:12.688 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.686+0000 7f684a14df40 1 --2- 172.21.15.32:0/1556939735 >> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] conn(0x55bd921fa9e0 0x55bd9221ade0 crc :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:12.689 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.687+0000 7f684a14df40 1 -- 172.21.15.32:0/1556939735 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f6824075a20 msgr2=0x7f6824077f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:55:12.689 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.687+0000 7f684a14df40 1 --2- 172.21.15.32:0/1556939735 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f6824075a20 0x7f6824077f30 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x55bd9214c090 tx=0x7f6830000ed0 comp rx=0 tx=0).stop 2024-08-03T20:55:12.689 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.687+0000 7f684a14df40 1 -- 172.21.15.32:0/1556939735 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55bd92203360 msgr2=0x55bd921f67e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:55:12.689 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.687+0000 7f684a14df40 1 --2- 172.21.15.32:0/1556939735 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55bd92203360 0x55bd921f67e0 secure :-1 s=READY pgs=1145 cs=0 l=1 rev1=1 crypto rx=0x7f683400efd0 tx=0x7f683400c7a0 comp rx=0 tx=0).stop 2024-08-03T20:55:12.689 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.687+0000 7f6843c00640 1 -- 172.21.15.32:0/1556939735 reap_dead start 2024-08-03T20:55:12.689 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.687+0000 7f684a14df40 1 -- 172.21.15.32:0/1556939735 shutdown_connections 2024-08-03T20:55:12.689 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.687+0000 7f684a14df40 1 -- 172.21.15.32:0/1556939735 >> 172.21.15.32:0/1556939735 conn(0x55bd9206cac0 msgr2=0x55bd9221cf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:55:12.689 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.687+0000 7f684a14df40 1 -- 172.21.15.32:0/1556939735 shutdown_connections 2024-08-03T20:55:12.689 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.687+0000 7f684a14df40 1 -- 172.21.15.32:0/1556939735 wait complete. 2024-08-03T20:55:12.699 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs joinable true 2024-08-03T20:55:12.930 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.928+0000 7f2f6cc00640 1 -- 172.21.15.32:0/1461153556 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2f68145c50 msgr2=0x7f2f6806e950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:55:12.931 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.928+0000 7f2f6cc00640 1 --2- 172.21.15.32:0/1461153556 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2f68145c50 0x7f2f6806e950 secure :-1 s=READY pgs=1146 cs=0 l=1 rev1=1 crypto rx=0x7f2f680cff30 tx=0x7f2f5802a550 comp rx=0 tx=0).stop 2024-08-03T20:55:12.931 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.928+0000 7f2f6cc00640 1 -- 172.21.15.32:0/1461153556 shutdown_connections 2024-08-03T20:55:12.931 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.928+0000 7f2f6cc00640 1 --2- 172.21.15.32:0/1461153556 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f2f680ced30 0x7f2f680cf140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:12.931 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.928+0000 7f2f6cc00640 1 --2- 172.21.15.32:0/1461153556 >> v2:172.21.15.118:3300/0 conn(0x7f2f680cccf0 0x7f2f680cd100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:12.931 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.928+0000 7f2f6cc00640 1 --2- 172.21.15.32:0/1461153556 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2f68145c50 0x7f2f6806e950 unknown :-1 s=CLOSED pgs=1146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:12.931 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.928+0000 7f2f6cc00640 1 -- 172.21.15.32:0/1461153556 >> 172.21.15.32:0/1461153556 conn(0x7f2f680b1bd0 msgr2=0x7f2f680b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:55:12.931 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.928+0000 7f2f6cc00640 1 -- 172.21.15.32:0/1461153556 shutdown_connections 2024-08-03T20:55:12.931 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.929+0000 7f2f6cc00640 1 -- 172.21.15.32:0/1461153556 wait complete. 2024-08-03T20:55:12.932 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.931+0000 7f2f6cc00640 1 Processor -- start 2024-08-03T20:55:12.932 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.931+0000 7f2f6cc00640 1 -- start start 2024-08-03T20:55:12.935 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.932+0000 7f2f6cc00640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f2f680cccf0 0x7f2f681f98d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:55:12.935 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.932+0000 7f2f6cc00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f2f680ced30 0x7f2f6814c7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:55:12.935 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.932+0000 7f2f6cc00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2f68145c50 0x7f2f68144320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:55:12.935 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.932+0000 7f2f6cc00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f2f680b3e60 con 0x7f2f68145c50 2024-08-03T20:55:12.936 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.932+0000 7f2f6cc00640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f2f680b3ce0 con 0x7f2f680ced30 2024-08-03T20:55:12.936 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.932+0000 7f2f6cc00640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7f2f680b3fe0 con 0x7f2f680cccf0 2024-08-03T20:55:12.936 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.933+0000 7f2f66800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2f68145c50 0x7f2f68144320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:55:12.936 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.933+0000 7f2f65e00640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f2f680cccf0 0x7f2f681f98d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:55:12.936 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.933+0000 7f2f5d400640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f2f680ced30 0x7f2f6814c7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:55:12.936 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.933+0000 7f2f66800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2f68145c50 0x7f2f68144320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:50484/0 (socket says 172.21.15.32:50484) 2024-08-03T20:55:12.936 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.933+0000 7f2f65e00640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f2f680cccf0 0x7f2f681f98d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3301/0 says I am v2:172.21.15.32:42448/0 (socket says 172.21.15.32:42448) 2024-08-03T20:55:12.936 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.933+0000 7f2f66800640 1 -- 172.21.15.32:0/361982585 learned_addr learned my addr 172.21.15.32:0/361982585 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:55:12.936 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.933+0000 7f2f65e00640 1 -- 172.21.15.32:0/361982585 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f2f680ced30 msgr2=0x7f2f6814c7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:55:12.936 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.933+0000 7f2f65e00640 1 --2- 172.21.15.32:0/361982585 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f2f680ced30 0x7f2f6814c7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:12.936 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.933+0000 7f2f65e00640 1 -- 172.21.15.32:0/361982585 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2f68145c50 msgr2=0x7f2f68144320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:55:12.936 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.933+0000 7f2f65e00640 1 --2- 172.21.15.32:0/361982585 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2f68145c50 0x7f2f68144320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:12.936 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.933+0000 7f2f65e00640 1 -- 172.21.15.32:0/361982585 --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2f681f9e10 con 0x7f2f680cccf0 2024-08-03T20:55:12.937 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.933+0000 7f2f66800640 1 --2- 172.21.15.32:0/361982585 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2f68145c50 0x7f2f68144320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-03T20:55:12.937 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.934+0000 7f2f65e00640 1 --2- 172.21.15.32:0/361982585 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f2f680cccf0 0x7f2f681f98d0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f2f58031000 tx=0x7f2f58003970 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:55:12.938 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.934+0000 7f2f5ea00640 1 -- 172.21.15.32:0/361982585 <== mon.2 v2:172.21.15.118:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f5803e030 con 0x7f2f680cccf0 2024-08-03T20:55:12.938 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.934+0000 7f2f5ea00640 1 -- 172.21.15.32:0/361982585 <== mon.2 v2:172.21.15.118:3301/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f2f58003e60 con 0x7f2f680cccf0 2024-08-03T20:55:12.938 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.934+0000 7f2f6cc00640 1 -- 172.21.15.32:0/361982585 --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2f68146580 con 0x7f2f680cccf0 2024-08-03T20:55:12.938 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.934+0000 7f2f5ea00640 1 -- 172.21.15.32:0/361982585 <== mon.2 v2:172.21.15.118:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f58035040 con 0x7f2f680cccf0 2024-08-03T20:55:12.938 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.934+0000 7f2f6cc00640 1 -- 172.21.15.32:0/361982585 --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f2f68146b40 con 0x7f2f680cccf0 2024-08-03T20:55:12.938 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.935+0000 7f2f6cc00640 1 -- 172.21.15.32:0/361982585 --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2f68070150 con 0x7f2f680cccf0 2024-08-03T20:55:12.939 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.937+0000 7f2f5ea00640 1 -- 172.21.15.32:0/361982585 <== mon.2 v2:172.21.15.118:3301/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f2f58041030 con 0x7f2f680cccf0 2024-08-03T20:55:12.939 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.938+0000 7f2f5ea00640 1 --2- 172.21.15.32:0/361982585 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f2f44077010 0x7f2f44079520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:55:12.939 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.938+0000 7f2f5ea00640 1 -- 172.21.15.32:0/361982585 <== mon.2 v2:172.21.15.118:3301/0 5 ==== osd_map(86..86 src has 1..86) ==== 5967+0+0 (secure 0 0 0) 0x7f2f580b9440 con 0x7f2f680cccf0 2024-08-03T20:55:12.940 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.938+0000 7f2f5d400640 1 --2- 172.21.15.32:0/361982585 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f2f44077010 0x7f2f44079520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:55:12.940 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.939+0000 7f2f5d400640 1 --2- 172.21.15.32:0/361982585 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f2f44077010 0x7f2f44079520 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f2f6815a460 tx=0x7f2f5000a250 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:55:12.943 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:12.941+0000 7f2f5ea00640 1 -- 172.21.15.32:0/361982585 <== mon.2 v2:172.21.15.118:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7f2f68070150 con 0x7f2f680cccf0 2024-08-03T20:55:13.191 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:13.190+0000 7f2f6cc00640 1 -- 172.21.15.32:0/361982585 --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "joinable", "val": "true"} v 0) -- 0x7f2f680b4ca0 con 0x7f2f680cccf0 2024-08-03T20:55:14.105 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:14.103+0000 7f2f5ea00640 1 -- 172.21.15.32:0/361982585 <== mon.2 v2:172.21.15.118:3301/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "joinable", "val": "true"}]=0 cephfs marked joinable; MDS may join as newly active. v110) ==== 162+0+0 (secure 0 0 0) 0x7f2f680b4ca0 con 0x7f2f680cccf0 2024-08-03T20:55:14.105 INFO:teuthology.orchestra.run.smithi032.stderr:cephfs marked joinable; MDS may join as newly active. 2024-08-03T20:55:14.110 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:14.108+0000 7f2f6cc00640 1 -- 172.21.15.32:0/361982585 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f2f44077010 msgr2=0x7f2f44079520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:55:14.110 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:14.108+0000 7f2f6cc00640 1 --2- 172.21.15.32:0/361982585 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f2f44077010 0x7f2f44079520 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f2f6815a460 tx=0x7f2f5000a250 comp rx=0 tx=0).stop 2024-08-03T20:55:14.110 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:14.108+0000 7f2f6cc00640 1 -- 172.21.15.32:0/361982585 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f2f680cccf0 msgr2=0x7f2f681f98d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:55:14.110 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:14.108+0000 7f2f6cc00640 1 --2- 172.21.15.32:0/361982585 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f2f680cccf0 0x7f2f681f98d0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f2f58031000 tx=0x7f2f58003970 comp rx=0 tx=0).stop 2024-08-03T20:55:14.110 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:14.108+0000 7f2f6cc00640 1 -- 172.21.15.32:0/361982585 shutdown_connections 2024-08-03T20:55:14.110 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:14.108+0000 7f2f6cc00640 1 --2- 172.21.15.32:0/361982585 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f2f44077010 0x7f2f44079520 unknown :-1 s=CLOSED pgs=476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:14.110 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:14.108+0000 7f2f6cc00640 1 --2- 172.21.15.32:0/361982585 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f2f680ced30 0x7f2f6814c7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:14.110 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:14.108+0000 7f2f6cc00640 1 --2- 172.21.15.32:0/361982585 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2f68145c50 0x7f2f68144320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:14.110 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:14.108+0000 7f2f6cc00640 1 --2- 172.21.15.32:0/361982585 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f2f680cccf0 0x7f2f681f98d0 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:14.111 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:14.108+0000 7f2f6cc00640 1 -- 172.21.15.32:0/361982585 >> 172.21.15.32:0/361982585 conn(0x7f2f680b1bd0 msgr2=0x7f2f6806f890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:55:14.111 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:14.108+0000 7f2f6cc00640 1 -- 172.21.15.32:0/361982585 shutdown_connections 2024-08-03T20:55:14.111 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:14.108+0000 7f2f6cc00640 1 -- 172.21.15.32:0/361982585 wait complete. 2024-08-03T20:55:14.118 INFO:tasks.ceph.mds.a.smithi032.stderr:2024-08-03T20:55:14.117+0000 7f6ee1400640 -1 mds.0.cache.ino(0x1) No data while reading inode 0x1 2024-08-03T20:55:14.119 INFO:tasks.ceph.mds.a.smithi032.stderr:2024-08-03T20:55:14.117+0000 7f6ee1400640 -1 log_channel(cluster) log [ERR] : Error loading MDS rank 0: (2) No such file or directory 2024-08-03T20:55:14.119 INFO:tasks.ceph.mon.a.smithi032.stderr:2024-08-03T20:55:14.118+0000 7f4e6d600640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-08-03T20:55:14.119 INFO:tasks.ceph.mon.a.smithi032.stderr:2024-08-03T20:55:14.118+0000 7f4e6d600640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 mds daemon damaged (MDS_DAMAGE) 2024-08-03T20:55:14.143 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-08-03T20:55:14.376 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:14.374+0000 7f3d91a00640 1 -- 172.21.15.32:0/615662496 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3d8c0d26e0 msgr2=0x7f3d8c161b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:55:14.376 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:14.374+0000 7f3d91a00640 1 --2- 172.21.15.32:0/615662496 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3d8c0d26e0 0x7f3d8c161b80 secure :-1 s=READY pgs=1148 cs=0 l=1 rev1=1 crypto rx=0x7f3d7c009830 tx=0x7f3d7c02a5c0 comp rx=0 tx=0).stop 2024-08-03T20:55:14.376 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:14.374+0000 7f3d91a00640 1 -- 172.21.15.32:0/615662496 shutdown_connections 2024-08-03T20:55:14.376 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:14.374+0000 7f3d91a00640 1 --2- 172.21.15.32:0/615662496 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3d8c0d26e0 0x7f3d8c161b80 unknown :-1 s=CLOSED pgs=1148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:14.376 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:14.374+0000 7f3d91a00640 1 -- 172.21.15.32:0/615662496 >> 172.21.15.32:0/615662496 conn(0x7f3d8c08d7a0 msgr2=0x7f3d8c09d3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:55:14.376 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:14.374+0000 7f3d91a00640 1 -- 172.21.15.32:0/615662496 shutdown_connections 2024-08-03T20:55:14.376 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:14.374+0000 7f3d91a00640 1 -- 172.21.15.32:0/615662496 wait complete. 2024-08-03T20:55:14.378 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:14.377+0000 7f3d91a00640 1 Processor -- start 2024-08-03T20:55:14.378 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:14.377+0000 7f3d91a00640 1 -- start start 2024-08-03T20:55:14.380 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:14.378+0000 7f3d91a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3d8c0cf510 0x7f3d8c0ccfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:55:14.380 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:14.378+0000 7f3d91a00640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f3d8c0cffe0 0x7f3d8c0ca4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:55:14.380 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:14.378+0000 7f3d91a00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f3d8c0d26e0 0x7f3d8c0b47c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:55:14.380 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:14.378+0000 7f3d91a00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f3d8c0cf920 con 0x7f3d8c0cf510 2024-08-03T20:55:14.381 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:14.378+0000 7f3d91a00640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f3d8c0d48a0 con 0x7f3d8c0d26e0 2024-08-03T20:55:14.381 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:14.378+0000 7f3d91a00640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7f3d8c0cfaa0 con 0x7f3d8c0cffe0 2024-08-03T20:55:14.381 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:14.378+0000 7f3d8be00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3d8c0cf510 0x7f3d8c0ccfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:55:14.381 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:14.379+0000 7f3d8be00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3d8c0cf510 0x7f3d8c0ccfe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:50514/0 (socket says 172.21.15.32:50514) 2024-08-03T20:55:14.381 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:14.379+0000 7f3d8be00640 1 -- 172.21.15.32:0/4075651315 learned_addr learned my addr 172.21.15.32:0/4075651315 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:55:14.381 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:14.379+0000 7f3d83e00640 1 --2- 172.21.15.32:0/4075651315 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f3d8c0cffe0 0x7f3d8c0ca4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:55:14.381 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:14.379+0000 7f3d91000640 1 --2- 172.21.15.32:0/4075651315 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f3d8c0d26e0 0x7f3d8c0b47c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:55:14.381 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:14.379+0000 7f3d8be00640 1 -- 172.21.15.32:0/4075651315 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f3d8c0cffe0 msgr2=0x7f3d8c0ca4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:55:14.381 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:14.379+0000 7f3d8be00640 1 --2- 172.21.15.32:0/4075651315 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f3d8c0cffe0 0x7f3d8c0ca4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:14.381 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:14.379+0000 7f3d8be00640 1 -- 172.21.15.32:0/4075651315 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f3d8c0d26e0 msgr2=0x7f3d8c0b47c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:55:14.381 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:14.379+0000 7f3d8be00640 1 --2- 172.21.15.32:0/4075651315 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f3d8c0d26e0 0x7f3d8c0b47c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:14.381 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:14.379+0000 7f3d8be00640 1 -- 172.21.15.32:0/4075651315 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3d8c1f8f70 con 0x7f3d8c0cf510 2024-08-03T20:55:14.381 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:14.379+0000 7f3d91000640 1 --2- 172.21.15.32:0/4075651315 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f3d8c0d26e0 0x7f3d8c0b47c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-03T20:55:14.382 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:14.379+0000 7f3d8be00640 1 --2- 172.21.15.32:0/4075651315 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3d8c0cf510 0x7f3d8c0ccfe0 secure :-1 s=READY pgs=1149 cs=0 l=1 rev1=1 crypto rx=0x7f3d7c009950 tx=0x7f3d7c033ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:55:14.382 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:14.380+0000 7f3d88c00640 1 -- 172.21.15.32:0/4075651315 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d7c031b50 con 0x7f3d8c0cf510 2024-08-03T20:55:14.383 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:14.380+0000 7f3d88c00640 1 -- 172.21.15.32:0/4075651315 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f3d7c031de0 con 0x7f3d8c0cf510 2024-08-03T20:55:14.383 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:14.380+0000 7f3d88c00640 1 -- 172.21.15.32:0/4075651315 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d7c005280 con 0x7f3d8c0cf510 2024-08-03T20:55:14.383 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:14.380+0000 7f3d91a00640 1 -- 172.21.15.32:0/4075651315 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3d8c1f9250 con 0x7f3d8c0cf510 2024-08-03T20:55:14.383 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:14.380+0000 7f3d91a00640 1 -- 172.21.15.32:0/4075651315 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3d8c1f9810 con 0x7f3d8c0cf510 2024-08-03T20:55:14.383 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:14.381+0000 7f3d91a00640 1 -- 172.21.15.32:0/4075651315 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3d8c0d3f80 con 0x7f3d8c0cf510 2024-08-03T20:55:14.383 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:14.382+0000 7f3d88c00640 1 -- 172.21.15.32:0/4075651315 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f3d7c005420 con 0x7f3d8c0cf510 2024-08-03T20:55:14.385 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:14.383+0000 7f3d88c00640 1 --2- 172.21.15.32:0/4075651315 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f3d68077040 0x7f3d68079550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:55:14.385 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:14.383+0000 7f3d88c00640 1 -- 172.21.15.32:0/4075651315 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 6053+0+0 (secure 0 0 0) 0x7f3d7c0b9a90 con 0x7f3d8c0cf510 2024-08-03T20:55:14.385 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:14.383+0000 7f3d83e00640 1 --2- 172.21.15.32:0/4075651315 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f3d68077040 0x7f3d68079550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:55:14.385 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:14.384+0000 7f3d83e00640 1 --2- 172.21.15.32:0/4075651315 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f3d68077040 0x7f3d68079550 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f3d8c068d90 tx=0x7f3d74009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:55:14.391 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:14.389+0000 7f3d88c00640 1 -- 172.21.15.32:0/4075651315 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7f3d7c03f030 con 0x7f3d8c0cf510 2024-08-03T20:55:14.637 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:14.635+0000 7f3d91a00640 1 -- 172.21.15.32:0/4075651315 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f3d8c161c20 con 0x7f3d8c0cf510 2024-08-03T20:55:14.638 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:14.636+0000 7f3d88c00640 1 -- 172.21.15.32:0/4075651315 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 112 v112) ==== 95+0+4390 (secure 0 0 0) 0x7f3d7c031770 con 0x7f3d8c0cf510 2024-08-03T20:55:14.638 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-03T20:55:14.638 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":112,"btime":"2024-08-03T20:55:14:119574+0000","default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6439,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6835/531080160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6834","nonce":531080160},{"type":"v1","addr":"172.21.15.118:6835","nonce":531080160}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":84},{"gid":6454,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6837/2796882319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6836","nonce":2796882319},{"type":"v1","addr":"172.21.15.118:6837","nonce":2796882319}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":86},{"gid":6813,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6834/2686933890","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":2686933890},{"type":"v1","addr":"172.21.15.32:6834","nonce":2686933890}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":85}],"filesystems":[{"mdsmap":{"epoch":112,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-08-03T20:54:26.950518+0000","modified":"2024-08-03T20:55:14.119472+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":87,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":6}]} 2024-08-03T20:55:14.638 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 112 2024-08-03T20:55:14.640 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:14.638+0000 7f3d91a00640 1 -- 172.21.15.32:0/4075651315 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f3d68077040 msgr2=0x7f3d68079550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:55:14.640 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:14.638+0000 7f3d91a00640 1 --2- 172.21.15.32:0/4075651315 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f3d68077040 0x7f3d68079550 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f3d8c068d90 tx=0x7f3d74009040 comp rx=0 tx=0).stop 2024-08-03T20:55:14.640 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:14.638+0000 7f3d91a00640 1 -- 172.21.15.32:0/4075651315 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3d8c0cf510 msgr2=0x7f3d8c0ccfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:55:14.640 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:14.638+0000 7f3d91a00640 1 --2- 172.21.15.32:0/4075651315 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3d8c0cf510 0x7f3d8c0ccfe0 secure :-1 s=READY pgs=1149 cs=0 l=1 rev1=1 crypto rx=0x7f3d7c009950 tx=0x7f3d7c033ba0 comp rx=0 tx=0).stop 2024-08-03T20:55:14.640 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:14.638+0000 7f3d91a00640 1 -- 172.21.15.32:0/4075651315 shutdown_connections 2024-08-03T20:55:14.641 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:14.638+0000 7f3d91a00640 1 --2- 172.21.15.32:0/4075651315 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f3d68077040 0x7f3d68079550 unknown :-1 s=CLOSED pgs=477 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:14.641 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:14.638+0000 7f3d91a00640 1 --2- 172.21.15.32:0/4075651315 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f3d8c0d26e0 0x7f3d8c0b47c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:14.641 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:14.638+0000 7f3d91a00640 1 --2- 172.21.15.32:0/4075651315 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f3d8c0cffe0 0x7f3d8c0ca4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:14.641 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:14.638+0000 7f3d91a00640 1 --2- 172.21.15.32:0/4075651315 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3d8c0cf510 0x7f3d8c0ccfe0 unknown :-1 s=CLOSED pgs=1149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:14.641 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:14.638+0000 7f3d91a00640 1 -- 172.21.15.32:0/4075651315 >> 172.21.15.32:0/4075651315 conn(0x7f3d8c08d7a0 msgr2=0x7f3d8c0b3e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:55:14.641 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:14.639+0000 7f3d91a00640 1 -- 172.21.15.32:0/4075651315 shutdown_connections 2024-08-03T20:55:14.641 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:14.639+0000 7f3d91a00640 1 -- 172.21.15.32:0/4075651315 wait complete. 2024-08-03T20:55:14.663 DEBUG:tasks.ceph_test_case:wait_until_true: success in 0s and 0 retries 2024-08-03T20:55:14.663 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-08-03T20:55:14.891 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:14.889+0000 7fd115c00640 1 -- 172.21.15.32:0/770590799 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd1100b4ce0 msgr2=0x7fd1100cdf70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:55:14.891 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:14.889+0000 7fd115c00640 1 --2- 172.21.15.32:0/770590799 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd1100b4ce0 0x7fd1100cdf70 secure :-1 s=READY pgs=1151 cs=0 l=1 rev1=1 crypto rx=0x7fd11015c920 tx=0x7fd10802a820 comp rx=0 tx=0).stop 2024-08-03T20:55:14.891 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:14.889+0000 7fd115c00640 1 -- 172.21.15.32:0/770590799 shutdown_connections 2024-08-03T20:55:14.891 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:14.889+0000 7fd115c00640 1 --2- 172.21.15.32:0/770590799 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd1100b4ce0 0x7fd1100cdf70 unknown :-1 s=CLOSED pgs=1151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:14.891 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:14.889+0000 7fd115c00640 1 --2- 172.21.15.32:0/770590799 >> v2:172.21.15.118:3300/0 conn(0x7fd11015f610 0x7fd1101523f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:14.891 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:14.889+0000 7fd115c00640 1 -- 172.21.15.32:0/770590799 >> 172.21.15.32:0/770590799 conn(0x7fd1100b1bd0 msgr2=0x7fd1100b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:55:14.892 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:14.889+0000 7fd115c00640 1 -- 172.21.15.32:0/770590799 shutdown_connections 2024-08-03T20:55:14.892 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:14.889+0000 7fd115c00640 1 -- 172.21.15.32:0/770590799 wait complete. 2024-08-03T20:55:14.893 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:14.891+0000 7fd115c00640 1 Processor -- start 2024-08-03T20:55:14.893 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:14.892+0000 7fd115c00640 1 -- start start 2024-08-03T20:55:14.895 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:14.893+0000 7fd115c00640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fd1100b38b0 0x7fd1101505f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:55:14.895 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:14.893+0000 7fd115c00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd1100b4ce0 0x7fd11015d4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:55:14.895 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:14.893+0000 7fd115c00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fd11014f420 0x7fd1100b0b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:55:14.895 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:14.893+0000 7fd115c00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fd1100cea90 con 0x7fd1100b4ce0 2024-08-03T20:55:14.895 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:14.893+0000 7fd115c00640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7fd1100ce910 con 0x7fd11014f420 2024-08-03T20:55:14.895 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:14.893+0000 7fd115c00640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7fd1100cec10 con 0x7fd1100b38b0 2024-08-03T20:55:14.896 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:14.893+0000 7fd107e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd1100b4ce0 0x7fd11015d4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:55:14.896 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:14.893+0000 7fd107e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd1100b4ce0 0x7fd11015d4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:50534/0 (socket says 172.21.15.32:50534) 2024-08-03T20:55:14.896 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:14.893+0000 7fd107e00640 1 -- 172.21.15.32:0/3060789454 learned_addr learned my addr 172.21.15.32:0/3060789454 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:55:14.896 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:14.893+0000 7fd10ea00640 1 --2- 172.21.15.32:0/3060789454 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fd1100b38b0 0x7fd1101505f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:55:14.896 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:14.894+0000 7fd10f400640 1 --2- 172.21.15.32:0/3060789454 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fd11014f420 0x7fd1100b0b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:55:14.896 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:14.894+0000 7fd107e00640 1 -- 172.21.15.32:0/3060789454 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fd1100b38b0 msgr2=0x7fd1101505f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:55:14.896 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:14.894+0000 7fd107e00640 1 --2- 172.21.15.32:0/3060789454 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fd1100b38b0 0x7fd1101505f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:14.897 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:14.894+0000 7fd107e00640 1 -- 172.21.15.32:0/3060789454 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fd11014f420 msgr2=0x7fd1100b0b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:55:14.898 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:14.894+0000 7fd107e00640 1 --2- 172.21.15.32:0/3060789454 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fd11014f420 0x7fd1100b0b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:14.898 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:14.894+0000 7fd107e00640 1 -- 172.21.15.32:0/3060789454 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd11015da00 con 0x7fd1100b4ce0 2024-08-03T20:55:14.898 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:14.894+0000 7fd10f400640 1 --2- 172.21.15.32:0/3060789454 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fd11014f420 0x7fd1100b0b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-03T20:55:14.898 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:14.894+0000 7fd107e00640 1 --2- 172.21.15.32:0/3060789454 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd1100b4ce0 0x7fd11015d4c0 secure :-1 s=READY pgs=1152 cs=0 l=1 rev1=1 crypto rx=0x7fd0f800d750 tx=0x7fd0f800dc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:55:14.898 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:14.894+0000 7fd106a00640 1 -- 172.21.15.32:0/3060789454 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0f8017070 con 0x7fd1100b4ce0 2024-08-03T20:55:14.898 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:14.894+0000 7fd106a00640 1 -- 172.21.15.32:0/3060789454 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fd0f8004310 con 0x7fd1100b4ce0 2024-08-03T20:55:14.898 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:14.894+0000 7fd106a00640 1 -- 172.21.15.32:0/3060789454 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0f800de60 con 0x7fd1100b4ce0 2024-08-03T20:55:14.898 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:14.894+0000 7fd115c00640 1 -- 172.21.15.32:0/3060789454 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd11015e600 con 0x7fd1100b4ce0 2024-08-03T20:55:14.898 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:14.895+0000 7fd115c00640 1 -- 172.21.15.32:0/3060789454 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd110157c00 con 0x7fd1100b4ce0 2024-08-03T20:55:14.898 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:14.896+0000 7fd115c00640 1 -- 172.21.15.32:0/3060789454 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd1100cf130 con 0x7fd1100b4ce0 2024-08-03T20:55:14.899 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:14.897+0000 7fd106a00640 1 -- 172.21.15.32:0/3060789454 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7fd0f8009c30 con 0x7fd1100b4ce0 2024-08-03T20:55:14.900 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:14.898+0000 7fd106a00640 1 --2- 172.21.15.32:0/3060789454 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fd0ec077040 0x7fd0ec079550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:55:14.900 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:14.898+0000 7fd106a00640 1 -- 172.21.15.32:0/3060789454 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 6053+0+0 (secure 0 0 0) 0x7fd0f8098e50 con 0x7fd1100b4ce0 2024-08-03T20:55:14.900 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:14.899+0000 7fd10ea00640 1 --2- 172.21.15.32:0/3060789454 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fd0ec077040 0x7fd0ec079550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:55:14.902 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:14.900+0000 7fd10ea00640 1 --2- 172.21.15.32:0/3060789454 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fd0ec077040 0x7fd0ec079550 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7fd100000f60 tx=0x7fd100009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:55:14.904 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:14.903+0000 7fd106a00640 1 -- 172.21.15.32:0/3060789454 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7fd0f8020020 con 0x7fd1100b4ce0 2024-08-03T20:55:15.146 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:15.144+0000 7fd115c00640 1 -- 172.21.15.32:0/3060789454 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd1100ce060 con 0x7fd1100b4ce0 2024-08-03T20:55:15.146 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:15.145+0000 7fd106a00640 1 -- 172.21.15.32:0/3060789454 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 112 v112) ==== 95+0+4390 (secure 0 0 0) 0x7fd0f8061e50 con 0x7fd1100b4ce0 2024-08-03T20:55:15.147 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-03T20:55:15.147 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":112,"btime":"2024-08-03T20:55:14:119574+0000","default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6439,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6835/531080160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6834","nonce":531080160},{"type":"v1","addr":"172.21.15.118:6835","nonce":531080160}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":84},{"gid":6454,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6837/2796882319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6836","nonce":2796882319},{"type":"v1","addr":"172.21.15.118:6837","nonce":2796882319}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":86},{"gid":6813,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6834/2686933890","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":2686933890},{"type":"v1","addr":"172.21.15.32:6834","nonce":2686933890}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":85}],"filesystems":[{"mdsmap":{"epoch":112,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-08-03T20:54:26.950518+0000","modified":"2024-08-03T20:55:14.119472+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":87,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":6}]} 2024-08-03T20:55:15.147 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 112 2024-08-03T20:55:15.149 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:15.147+0000 7fd115c00640 1 -- 172.21.15.32:0/3060789454 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fd0ec077040 msgr2=0x7fd0ec079550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:55:15.149 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:15.147+0000 7fd115c00640 1 --2- 172.21.15.32:0/3060789454 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fd0ec077040 0x7fd0ec079550 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7fd100000f60 tx=0x7fd100009040 comp rx=0 tx=0).stop 2024-08-03T20:55:15.149 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:15.147+0000 7fd115c00640 1 -- 172.21.15.32:0/3060789454 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd1100b4ce0 msgr2=0x7fd11015d4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:55:15.149 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:15.147+0000 7fd115c00640 1 --2- 172.21.15.32:0/3060789454 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd1100b4ce0 0x7fd11015d4c0 secure :-1 s=READY pgs=1152 cs=0 l=1 rev1=1 crypto rx=0x7fd0f800d750 tx=0x7fd0f800dc10 comp rx=0 tx=0).stop 2024-08-03T20:55:15.149 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:15.147+0000 7fd115c00640 1 -- 172.21.15.32:0/3060789454 shutdown_connections 2024-08-03T20:55:15.149 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:15.147+0000 7fd115c00640 1 --2- 172.21.15.32:0/3060789454 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fd0ec077040 0x7fd0ec079550 unknown :-1 s=CLOSED pgs=478 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:15.149 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:15.147+0000 7fd115c00640 1 --2- 172.21.15.32:0/3060789454 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fd11014f420 0x7fd1100b0b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:15.149 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:15.147+0000 7fd115c00640 1 --2- 172.21.15.32:0/3060789454 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd1100b4ce0 0x7fd11015d4c0 unknown :-1 s=CLOSED pgs=1152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:15.150 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:15.147+0000 7fd115c00640 1 --2- 172.21.15.32:0/3060789454 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fd1100b38b0 0x7fd1101505f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:15.150 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:15.147+0000 7fd115c00640 1 -- 172.21.15.32:0/3060789454 >> 172.21.15.32:0/3060789454 conn(0x7fd1100b1bd0 msgr2=0x7fd110151c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:55:15.150 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:15.147+0000 7fd115c00640 1 -- 172.21.15.32:0/3060789454 shutdown_connections 2024-08-03T20:55:15.150 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:15.147+0000 7fd115c00640 1 -- 172.21.15.32:0/3060789454 wait complete. 2024-08-03T20:55:15.172 DEBUG:tasks.ceph_test_case:wait_until_true: success in 0s and 0 retries 2024-08-03T20:55:15.172 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-08-03T20:55:15.401 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:15.399+0000 7f0dbe200640 1 -- 172.21.15.32:0/1171538872 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0db80acd80 msgr2=0x7f0db80ad190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:55:15.401 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:15.399+0000 7f0dbe200640 1 --2- 172.21.15.32:0/1171538872 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0db80acd80 0x7f0db80ad190 secure :-1 s=READY pgs=1154 cs=0 l=1 rev1=1 crypto rx=0x7f0db80720a0 tx=0x7f0da402a4a0 comp rx=0 tx=0).stop 2024-08-03T20:55:15.401 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:15.399+0000 7f0dbe200640 1 -- 172.21.15.32:0/1171538872 shutdown_connections 2024-08-03T20:55:15.401 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:15.399+0000 7f0dbe200640 1 --2- 172.21.15.32:0/1171538872 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0db80acd80 0x7f0db80ad190 unknown :-1 s=CLOSED pgs=1154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:15.401 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:15.399+0000 7f0dbe200640 1 --2- 172.21.15.32:0/1171538872 >> v2:172.21.15.118:3300/0 conn(0x7f0db80af020 0x7f0db80742b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:15.401 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:15.399+0000 7f0dbe200640 1 -- 172.21.15.32:0/1171538872 >> 172.21.15.32:0/1171538872 conn(0x7f0db80b2120 msgr2=0x7f0db80cfd60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:55:15.401 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:15.399+0000 7f0dbe200640 1 -- 172.21.15.32:0/1171538872 shutdown_connections 2024-08-03T20:55:15.401 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:15.399+0000 7f0dbe200640 1 -- 172.21.15.32:0/1171538872 wait complete. 2024-08-03T20:55:15.402 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:15.401+0000 7f0dbe200640 1 Processor -- start 2024-08-03T20:55:15.403 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:15.401+0000 7f0dbe200640 1 -- start start 2024-08-03T20:55:15.406 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:15.403+0000 7f0dbe200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0db806f440 0x7f0db8154890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:55:15.406 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:15.403+0000 7f0dbe200640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f0db80acd80 0x7f0db80b4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:55:15.406 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:15.403+0000 7f0dbe200640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f0db80af020 0x7f0db80b53a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:55:15.406 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:15.403+0000 7f0dbe200640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f0db81601e0 con 0x7f0db806f440 2024-08-03T20:55:15.406 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:15.403+0000 7f0dbe200640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f0db8160060 con 0x7f0db80af020 2024-08-03T20:55:15.406 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:15.403+0000 7f0dbe200640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7f0db8160360 con 0x7f0db80acd80 2024-08-03T20:55:15.406 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:15.403+0000 7f0dbce00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0db806f440 0x7f0db8154890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:55:15.406 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:15.403+0000 7f0dbce00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0db806f440 0x7f0db8154890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:50554/0 (socket says 172.21.15.32:50554) 2024-08-03T20:55:15.406 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:15.403+0000 7f0dbce00640 1 -- 172.21.15.32:0/564984224 learned_addr learned my addr 172.21.15.32:0/564984224 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:55:15.406 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:15.403+0000 7f0dbd800640 1 --2- 172.21.15.32:0/564984224 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f0db80af020 0x7f0db80b53a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:55:15.407 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:15.403+0000 7f0db7e00640 1 --2- 172.21.15.32:0/564984224 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f0db80acd80 0x7f0db80b4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:55:15.407 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:15.404+0000 7f0dbce00640 1 -- 172.21.15.32:0/564984224 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f0db80acd80 msgr2=0x7f0db80b4e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:55:15.407 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:15.404+0000 7f0dbce00640 1 --2- 172.21.15.32:0/564984224 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f0db80acd80 0x7f0db80b4e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:15.407 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:15.404+0000 7f0dbce00640 1 -- 172.21.15.32:0/564984224 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f0db80af020 msgr2=0x7f0db80b53a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:55:15.407 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:15.404+0000 7f0dbce00640 1 --2- 172.21.15.32:0/564984224 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f0db80af020 0x7f0db80b53a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:15.407 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:15.404+0000 7f0dbce00640 1 -- 172.21.15.32:0/564984224 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0db80c9e40 con 0x7f0db806f440 2024-08-03T20:55:15.407 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:15.404+0000 7f0dbd800640 1 --2- 172.21.15.32:0/564984224 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f0db80af020 0x7f0db80b53a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-03T20:55:15.407 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:15.404+0000 7f0dbce00640 1 --2- 172.21.15.32:0/564984224 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0db806f440 0x7f0db8154890 secure :-1 s=READY pgs=1155 cs=0 l=1 rev1=1 crypto rx=0x7f0dac00d750 tx=0x7f0dac00dc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:55:15.407 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:15.404+0000 7f0db4c00640 1 -- 172.21.15.32:0/564984224 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0dac017070 con 0x7f0db806f440 2024-08-03T20:55:15.407 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:15.404+0000 7f0db4c00640 1 -- 172.21.15.32:0/564984224 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f0dac004310 con 0x7f0db806f440 2024-08-03T20:55:15.407 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:15.404+0000 7f0dbe200640 1 -- 172.21.15.32:0/564984224 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0db80c92c0 con 0x7f0db806f440 2024-08-03T20:55:15.407 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:15.404+0000 7f0db4c00640 1 -- 172.21.15.32:0/564984224 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0dac00de60 con 0x7f0db806f440 2024-08-03T20:55:15.407 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:15.404+0000 7f0dbe200640 1 -- 172.21.15.32:0/564984224 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0db80c9880 con 0x7f0db806f440 2024-08-03T20:55:15.408 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:15.405+0000 7f0dbe200640 1 -- 172.21.15.32:0/564984224 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0db80ad190 con 0x7f0db806f440 2024-08-03T20:55:15.408 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:15.407+0000 7f0db4c00640 1 -- 172.21.15.32:0/564984224 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f0dac009c30 con 0x7f0db806f440 2024-08-03T20:55:15.409 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:15.408+0000 7f0db4c00640 1 --2- 172.21.15.32:0/564984224 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f0d98077010 0x7f0d98079520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:55:15.409 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:15.408+0000 7f0db4c00640 1 -- 172.21.15.32:0/564984224 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 6053+0+0 (secure 0 0 0) 0x7f0dac098e80 con 0x7f0db806f440 2024-08-03T20:55:15.409 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:15.408+0000 7f0db7e00640 1 --2- 172.21.15.32:0/564984224 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f0d98077010 0x7f0d98079520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:55:15.410 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:15.409+0000 7f0db7e00640 1 --2- 172.21.15.32:0/564984224 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f0d98077010 0x7f0d98079520 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7f0da4034000 tx=0x7f0da4004730 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:55:15.414 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:15.412+0000 7f0db4c00640 1 -- 172.21.15.32:0/564984224 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7f0dac020020 con 0x7f0db806f440 2024-08-03T20:55:15.658 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:15.657+0000 7f0dbe200640 1 -- 172.21.15.32:0/564984224 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f0db80d0930 con 0x7f0db806f440 2024-08-03T20:55:15.659 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:15.657+0000 7f0db4c00640 1 -- 172.21.15.32:0/564984224 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 112 v112) ==== 95+0+4390 (secure 0 0 0) 0x7f0dac061e80 con 0x7f0db806f440 2024-08-03T20:55:15.659 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-03T20:55:15.659 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":112,"btime":"2024-08-03T20:55:14:119574+0000","default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6439,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6835/531080160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6834","nonce":531080160},{"type":"v1","addr":"172.21.15.118:6835","nonce":531080160}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":84},{"gid":6454,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6837/2796882319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6836","nonce":2796882319},{"type":"v1","addr":"172.21.15.118:6837","nonce":2796882319}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":86},{"gid":6813,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6834/2686933890","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":2686933890},{"type":"v1","addr":"172.21.15.32:6834","nonce":2686933890}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":85}],"filesystems":[{"mdsmap":{"epoch":112,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-08-03T20:54:26.950518+0000","modified":"2024-08-03T20:55:14.119472+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":87,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":6}]} 2024-08-03T20:55:15.660 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 112 2024-08-03T20:55:15.661 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:15.660+0000 7f0dbe200640 1 -- 172.21.15.32:0/564984224 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f0d98077010 msgr2=0x7f0d98079520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:55:15.662 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:15.660+0000 7f0dbe200640 1 --2- 172.21.15.32:0/564984224 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f0d98077010 0x7f0d98079520 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7f0da4034000 tx=0x7f0da4004730 comp rx=0 tx=0).stop 2024-08-03T20:55:15.662 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:15.660+0000 7f0dbe200640 1 -- 172.21.15.32:0/564984224 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0db806f440 msgr2=0x7f0db8154890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:55:15.662 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:15.660+0000 7f0dbe200640 1 --2- 172.21.15.32:0/564984224 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0db806f440 0x7f0db8154890 secure :-1 s=READY pgs=1155 cs=0 l=1 rev1=1 crypto rx=0x7f0dac00d750 tx=0x7f0dac00dc10 comp rx=0 tx=0).stop 2024-08-03T20:55:15.662 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:15.660+0000 7f0dbe200640 1 -- 172.21.15.32:0/564984224 shutdown_connections 2024-08-03T20:55:15.662 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:15.660+0000 7f0dbe200640 1 --2- 172.21.15.32:0/564984224 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f0d98077010 0x7f0d98079520 unknown :-1 s=CLOSED pgs=479 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:15.662 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:15.660+0000 7f0dbe200640 1 --2- 172.21.15.32:0/564984224 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f0db80af020 0x7f0db80b53a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:15.662 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:15.660+0000 7f0dbe200640 1 --2- 172.21.15.32:0/564984224 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f0db80acd80 0x7f0db80b4e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:15.662 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:15.660+0000 7f0dbe200640 1 --2- 172.21.15.32:0/564984224 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0db806f440 0x7f0db8154890 unknown :-1 s=CLOSED pgs=1155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:15.662 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:15.660+0000 7f0dbe200640 1 -- 172.21.15.32:0/564984224 >> 172.21.15.32:0/564984224 conn(0x7f0db80b2120 msgr2=0x7f0db80ccb00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:55:15.662 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:15.660+0000 7f0dbe200640 1 -- 172.21.15.32:0/564984224 shutdown_connections 2024-08-03T20:55:15.662 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:15.660+0000 7f0dbe200640 1 -- 172.21.15.32:0/564984224 wait complete. 2024-08-03T20:55:15.683 INFO:tasks.cephfs.test_damage:Result: Mutation 'Delete 1.00000000.inode' led to DAMAGED state 2024-08-03T20:55:15.684 INFO:tasks.cephfs.test_damage:Applying mutation 'Delete 200.00000000' 2024-08-03T20:55:15.684 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-08-03T20:55:15.684 DEBUG:teuthology.orchestra.run.smithi032:> dd if=/proc/self/mounts of=/dev/stdout 2024-08-03T20:55:15.719 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053453,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6499208k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 none /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249604k,nr_inodes=812401,mode=700,uid=1000,gid=1316 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6499208k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-08-03T20:55:15.719 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi032.front.sepia.ceph.com 2024-08-03T20:55:15.719 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-08-03T20:55:15.719 DEBUG:teuthology.orchestra.run.smithi032:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-08-03T20:55:15.794 INFO:teuthology.orchestra.run.smithi032.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.0': No such file or directory 2024-08-03T20:55:15.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-03T20:55:15.795 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-08-03T20:55:16.065 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:16.063+0000 7fbd14800640 1 -- 172.21.15.32:0/1287594814 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbd0c06e930 msgr2=0x7fbd0c144f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:55:16.065 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:16.063+0000 7fbd14800640 1 --2- 172.21.15.32:0/1287594814 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbd0c06e930 0x7fbd0c144f90 secure :-1 s=READY pgs=1156 cs=0 l=1 rev1=1 crypto rx=0x7fbd00005fd0 tx=0x7fbd0002a5d0 comp rx=0 tx=0).stop 2024-08-03T20:55:16.066 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:16.063+0000 7fbd14800640 1 -- 172.21.15.32:0/1287594814 shutdown_connections 2024-08-03T20:55:16.066 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:16.063+0000 7fbd14800640 1 --2- 172.21.15.32:0/1287594814 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fbd0c15fff0 0x7fbd0c160400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:16.066 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:16.063+0000 7fbd14800640 1 --2- 172.21.15.32:0/1287594814 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbd0c06e930 0x7fbd0c144f90 unknown :-1 s=CLOSED pgs=1156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:16.066 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:16.063+0000 7fbd14800640 1 -- 172.21.15.32:0/1287594814 >> 172.21.15.32:0/1287594814 conn(0x7fbd0c0b1bd0 msgr2=0x7fbd0c0b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:55:16.066 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:16.063+0000 7fbd14800640 1 -- 172.21.15.32:0/1287594814 shutdown_connections 2024-08-03T20:55:16.066 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:16.063+0000 7fbd14800640 1 -- 172.21.15.32:0/1287594814 wait complete. 2024-08-03T20:55:16.067 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:16.065+0000 7fbd14800640 1 Processor -- start 2024-08-03T20:55:16.067 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:16.066+0000 7fbd14800640 1 -- start start 2024-08-03T20:55:16.069 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:16.067+0000 7fbd14800640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fbd0c06e930 0x7fbd0c15be90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:55:16.069 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:16.067+0000 7fbd14800640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fbd0c073e60 0x7fbd0c142610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:55:16.069 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:16.067+0000 7fbd14800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbd0c15fff0 0x7fbd0c15df60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:55:16.069 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:16.067+0000 7fbd14800640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fbd0c0cf840 con 0x7fbd0c15fff0 2024-08-03T20:55:16.069 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:16.067+0000 7fbd14800640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7fbd0c145f30 con 0x7fbd0c06e930 2024-08-03T20:55:16.070 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:16.067+0000 7fbd14800640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7fbd0c0cf9c0 con 0x7fbd0c073e60 2024-08-03T20:55:16.070 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:16.067+0000 7fbd12800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbd0c15fff0 0x7fbd0c15df60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:55:16.070 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:16.067+0000 7fbd12800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbd0c15fff0 0x7fbd0c15df60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:50568/0 (socket says 172.21.15.32:50568) 2024-08-03T20:55:16.070 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:16.067+0000 7fbd12800640 1 -- 172.21.15.32:0/4294723834 learned_addr learned my addr 172.21.15.32:0/4294723834 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:55:16.070 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:16.067+0000 7fbd11400640 1 --2- 172.21.15.32:0/4294723834 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fbd0c073e60 0x7fbd0c142610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:55:16.070 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:16.067+0000 7fbd11e00640 1 --2- 172.21.15.32:0/4294723834 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fbd0c06e930 0x7fbd0c15be90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:55:16.070 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:16.068+0000 7fbd12800640 1 -- 172.21.15.32:0/4294723834 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fbd0c073e60 msgr2=0x7fbd0c142610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:55:16.070 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:16.068+0000 7fbd12800640 1 --2- 172.21.15.32:0/4294723834 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fbd0c073e60 0x7fbd0c142610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:16.070 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:16.068+0000 7fbd12800640 1 -- 172.21.15.32:0/4294723834 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fbd0c06e930 msgr2=0x7fbd0c15be90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:55:16.070 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:16.068+0000 7fbd12800640 1 --2- 172.21.15.32:0/4294723834 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fbd0c06e930 0x7fbd0c15be90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:16.070 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:16.068+0000 7fbd12800640 1 -- 172.21.15.32:0/4294723834 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbd0c142f60 con 0x7fbd0c15fff0 2024-08-03T20:55:16.071 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:16.068+0000 7fbd12800640 1 --2- 172.21.15.32:0/4294723834 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbd0c15fff0 0x7fbd0c15df60 secure :-1 s=READY pgs=1157 cs=0 l=1 rev1=1 crypto rx=0x7fbd0800ea80 tx=0x7fbd0800ef40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:55:16.072 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:16.068+0000 7fbd06000640 1 -- 172.21.15.32:0/4294723834 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd0800cde0 con 0x7fbd0c15fff0 2024-08-03T20:55:16.072 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:16.068+0000 7fbd06000640 1 -- 172.21.15.32:0/4294723834 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fbd08004b20 con 0x7fbd0c15fff0 2024-08-03T20:55:16.072 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:16.068+0000 7fbd14800640 1 -- 172.21.15.32:0/4294723834 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbd0c143240 con 0x7fbd0c15fff0 2024-08-03T20:55:16.072 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:16.068+0000 7fbd06000640 1 -- 172.21.15.32:0/4294723834 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd08005020 con 0x7fbd0c15fff0 2024-08-03T20:55:16.072 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:16.069+0000 7fbd14800640 1 -- 172.21.15.32:0/4294723834 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbd0c143800 con 0x7fbd0c15fff0 2024-08-03T20:55:16.072 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:16.070+0000 7fbd14800640 1 -- 172.21.15.32:0/4294723834 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbd0c160c50 con 0x7fbd0c15fff0 2024-08-03T20:55:16.072 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:16.071+0000 7fbd06000640 1 -- 172.21.15.32:0/4294723834 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7fbd080051c0 con 0x7fbd0c15fff0 2024-08-03T20:55:16.073 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:16.072+0000 7fbd06000640 1 --2- 172.21.15.32:0/4294723834 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fbcf0077040 0x7fbcf0079550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:55:16.073 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:16.072+0000 7fbd06000640 1 -- 172.21.15.32:0/4294723834 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 6053+0+0 (secure 0 0 0) 0x7fbd08099500 con 0x7fbd0c15fff0 2024-08-03T20:55:16.074 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:16.072+0000 7fbd11e00640 1 --2- 172.21.15.32:0/4294723834 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fbcf0077040 0x7fbcf0079550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:55:16.074 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:16.073+0000 7fbd11e00640 1 --2- 172.21.15.32:0/4294723834 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fbcf0077040 0x7fbcf0079550 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7fbd00033000 tx=0x7fbd00004730 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:55:16.077 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:16.076+0000 7fbd06000640 1 -- 172.21.15.32:0/4294723834 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7fbd08061eb0 con 0x7fbd0c15fff0 2024-08-03T20:55:16.319 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:16.317+0000 7fbd14800640 1 -- 172.21.15.32:0/4294723834 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7fbd0c074480 con 0x7fbd0c15fff0 2024-08-03T20:55:17.133 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:17.131+0000 7fbd06000640 1 -- 172.21.15.32:0/4294723834 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v113) ==== 161+0+0 (secure 0 0 0) 0x7fbd08066400 con 0x7fbd0c15fff0 2024-08-03T20:55:17.133 INFO:teuthology.orchestra.run.smithi032.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-08-03T20:55:17.137 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:17.134+0000 7fbd14800640 1 -- 172.21.15.32:0/4294723834 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fbcf0077040 msgr2=0x7fbcf0079550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:55:17.137 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:17.134+0000 7fbd14800640 1 --2- 172.21.15.32:0/4294723834 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fbcf0077040 0x7fbcf0079550 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7fbd00033000 tx=0x7fbd00004730 comp rx=0 tx=0).stop 2024-08-03T20:55:17.137 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:17.134+0000 7fbd14800640 1 -- 172.21.15.32:0/4294723834 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbd0c15fff0 msgr2=0x7fbd0c15df60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:55:17.137 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:17.134+0000 7fbd14800640 1 --2- 172.21.15.32:0/4294723834 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbd0c15fff0 0x7fbd0c15df60 secure :-1 s=READY pgs=1157 cs=0 l=1 rev1=1 crypto rx=0x7fbd0800ea80 tx=0x7fbd0800ef40 comp rx=0 tx=0).stop 2024-08-03T20:55:17.137 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:17.134+0000 7fbd14800640 1 -- 172.21.15.32:0/4294723834 shutdown_connections 2024-08-03T20:55:17.137 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:17.134+0000 7fbd14800640 1 --2- 172.21.15.32:0/4294723834 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fbcf0077040 0x7fbcf0079550 unknown :-1 s=CLOSED pgs=480 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:17.138 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:17.134+0000 7fbd14800640 1 --2- 172.21.15.32:0/4294723834 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbd0c15fff0 0x7fbd0c15df60 unknown :-1 s=CLOSED pgs=1157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:17.138 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:17.134+0000 7fbd14800640 1 --2- 172.21.15.32:0/4294723834 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fbd0c073e60 0x7fbd0c142610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:17.138 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:17.134+0000 7fbd14800640 1 --2- 172.21.15.32:0/4294723834 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fbd0c06e930 0x7fbd0c15be90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:17.138 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:17.134+0000 7fbd14800640 1 -- 172.21.15.32:0/4294723834 >> 172.21.15.32:0/4294723834 conn(0x7fbd0c0b1bd0 msgr2=0x7fbd0c0afb20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:55:17.138 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:17.135+0000 7fbd14800640 1 -- 172.21.15.32:0/4294723834 shutdown_connections 2024-08-03T20:55:17.138 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:17.135+0000 7fbd14800640 1 -- 172.21.15.32:0/4294723834 wait complete. 2024-08-03T20:55:17.161 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mds repaired 0 2024-08-03T20:55:17.393 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:17.391+0000 7f3d77200640 1 -- 172.21.15.32:0/969013365 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3d7006e930 msgr2=0x7f3d70144f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:55:17.394 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:17.391+0000 7f3d77200640 1 --2- 172.21.15.32:0/969013365 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3d7006e930 0x7f3d70144f90 secure :-1 s=READY pgs=1158 cs=0 l=1 rev1=1 crypto rx=0x7f3d700b2f80 tx=0x7f3d6002a560 comp rx=0 tx=0).stop 2024-08-03T20:55:17.394 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:17.392+0000 7f3d77200640 1 -- 172.21.15.32:0/969013365 shutdown_connections 2024-08-03T20:55:17.394 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:17.392+0000 7f3d77200640 1 --2- 172.21.15.32:0/969013365 >> v2:172.21.15.118:3300/0 conn(0x7f3d700747b0 0x7f3d700cc8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:17.394 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:17.392+0000 7f3d77200640 1 --2- 172.21.15.32:0/969013365 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f3d70073e60 0x7f3d70074270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:17.394 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:17.392+0000 7f3d77200640 1 --2- 172.21.15.32:0/969013365 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3d7006e930 0x7f3d70144f90 unknown :-1 s=CLOSED pgs=1158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:17.394 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:17.392+0000 7f3d77200640 1 -- 172.21.15.32:0/969013365 >> 172.21.15.32:0/969013365 conn(0x7f3d700b1bd0 msgr2=0x7f3d700b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:55:17.394 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:17.392+0000 7f3d77200640 1 -- 172.21.15.32:0/969013365 shutdown_connections 2024-08-03T20:55:17.394 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:17.392+0000 7f3d77200640 1 -- 172.21.15.32:0/969013365 wait complete. 2024-08-03T20:55:17.395 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:17.394+0000 7f3d77200640 1 Processor -- start 2024-08-03T20:55:17.395 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:17.394+0000 7f3d77200640 1 -- start start 2024-08-03T20:55:17.398 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:17.395+0000 7f3d77200640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f3d7006e930 0x7f3d700abf50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:55:17.398 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:17.395+0000 7f3d77200640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f3d70073e60 0x7f3d701f4ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:55:17.398 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:17.395+0000 7f3d77200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3d700747b0 0x7f3d700ae670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:55:17.398 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:17.395+0000 7f3d77200640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f3d700cf9c0 con 0x7f3d700747b0 2024-08-03T20:55:17.398 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:17.395+0000 7f3d77200640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f3d700cf840 con 0x7f3d7006e930 2024-08-03T20:55:17.398 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:17.395+0000 7f3d77200640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7f3d700cce20 con 0x7f3d70073e60 2024-08-03T20:55:17.398 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:17.396+0000 7f3d75200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3d700747b0 0x7f3d700ae670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:55:17.398 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:17.396+0000 7f3d75200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3d700747b0 0x7f3d700ae670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:50590/0 (socket says 172.21.15.32:50590) 2024-08-03T20:55:17.398 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:17.396+0000 7f3d75200640 1 -- 172.21.15.32:0/2301394735 learned_addr learned my addr 172.21.15.32:0/2301394735 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:55:17.399 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:17.396+0000 7f3d6fe00640 1 --2- 172.21.15.32:0/2301394735 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f3d7006e930 0x7f3d700abf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:55:17.399 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:17.396+0000 7f3d6f400640 1 --2- 172.21.15.32:0/2301394735 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f3d70073e60 0x7f3d701f4ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:55:17.399 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:17.396+0000 7f3d75200640 1 -- 172.21.15.32:0/2301394735 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f3d70073e60 msgr2=0x7f3d701f4ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:55:17.399 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:17.396+0000 7f3d75200640 1 --2- 172.21.15.32:0/2301394735 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f3d70073e60 0x7f3d701f4ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:17.399 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:17.396+0000 7f3d75200640 1 -- 172.21.15.32:0/2301394735 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f3d7006e930 msgr2=0x7f3d700abf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:55:17.399 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:17.396+0000 7f3d75200640 1 --2- 172.21.15.32:0/2301394735 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f3d7006e930 0x7f3d700abf50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:17.399 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:17.396+0000 7f3d75200640 1 -- 172.21.15.32:0/2301394735 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3d700ac9c0 con 0x7f3d700747b0 2024-08-03T20:55:17.399 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:17.397+0000 7f3d75200640 1 --2- 172.21.15.32:0/2301394735 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3d700747b0 0x7f3d700ae670 secure :-1 s=READY pgs=1159 cs=0 l=1 rev1=1 crypto rx=0x7f3d6800e990 tx=0x7f3d6800ee50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:55:17.400 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:17.397+0000 7f3d67e00640 1 -- 172.21.15.32:0/2301394735 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d68004d20 con 0x7f3d700747b0 2024-08-03T20:55:17.400 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:17.397+0000 7f3d67e00640 1 -- 172.21.15.32:0/2301394735 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f3d6800caa0 con 0x7f3d700747b0 2024-08-03T20:55:17.400 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:17.397+0000 7f3d67e00640 1 -- 172.21.15.32:0/2301394735 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d68005020 con 0x7f3d700747b0 2024-08-03T20:55:17.400 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:17.397+0000 7f3d77200640 1 -- 172.21.15.32:0/2301394735 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3d700acec0 con 0x7f3d700747b0 2024-08-03T20:55:17.400 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:17.397+0000 7f3d77200640 1 -- 172.21.15.32:0/2301394735 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3d700ca1d0 con 0x7f3d700747b0 2024-08-03T20:55:17.400 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:17.398+0000 7f3d77200640 1 -- 172.21.15.32:0/2301394735 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3d700acc30 con 0x7f3d700747b0 2024-08-03T20:55:17.401 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:17.399+0000 7f3d67e00640 1 -- 172.21.15.32:0/2301394735 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f3d680051c0 con 0x7f3d700747b0 2024-08-03T20:55:17.402 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:17.401+0000 7f3d67e00640 1 --2- 172.21.15.32:0/2301394735 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f3d50077010 0x7f3d50079520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:55:17.402 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:17.401+0000 7f3d67e00640 1 -- 172.21.15.32:0/2301394735 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 6053+0+0 (secure 0 0 0) 0x7f3d680991c0 con 0x7f3d700747b0 2024-08-03T20:55:17.403 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:17.401+0000 7f3d6fe00640 1 --2- 172.21.15.32:0/2301394735 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f3d50077010 0x7f3d50079520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:55:17.403 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:17.402+0000 7f3d6fe00640 1 --2- 172.21.15.32:0/2301394735 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f3d50077010 0x7f3d50079520 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7f3d60033000 tx=0x7f3d60004730 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:55:17.407 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:17.405+0000 7f3d67e00640 1 -- 172.21.15.32:0/2301394735 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7f3d68016030 con 0x7f3d700747b0 2024-08-03T20:55:17.691 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:17.690+0000 7f3d77200640 1 -- 172.21.15.32:0/2301394735 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "mds repaired", "role": "0"} v 0) -- 0x7f3d700c9ce0 con 0x7f3d700747b0 2024-08-03T20:55:18.135 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.133+0000 7f3d67e00640 1 -- 172.21.15.32:0/2301394735 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "mds repaired", "role": "0"}]=0 repaired: restoring rank 6:0 v114) ==== 101+0+0 (secure 0 0 0) 0x7f3d68061c00 con 0x7f3d700747b0 2024-08-03T20:55:18.135 INFO:teuthology.orchestra.run.smithi032.stderr:repaired: restoring rank 6:0 2024-08-03T20:55:18.138 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.136+0000 7f3d77200640 1 -- 172.21.15.32:0/2301394735 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f3d50077010 msgr2=0x7f3d50079520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:55:18.138 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.136+0000 7f3d77200640 1 --2- 172.21.15.32:0/2301394735 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f3d50077010 0x7f3d50079520 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7f3d60033000 tx=0x7f3d60004730 comp rx=0 tx=0).stop 2024-08-03T20:55:18.138 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.136+0000 7f3d77200640 1 -- 172.21.15.32:0/2301394735 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3d700747b0 msgr2=0x7f3d700ae670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:55:18.138 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.136+0000 7f3d77200640 1 --2- 172.21.15.32:0/2301394735 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3d700747b0 0x7f3d700ae670 secure :-1 s=READY pgs=1159 cs=0 l=1 rev1=1 crypto rx=0x7f3d6800e990 tx=0x7f3d6800ee50 comp rx=0 tx=0).stop 2024-08-03T20:55:18.139 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.136+0000 7f3d77200640 1 -- 172.21.15.32:0/2301394735 shutdown_connections 2024-08-03T20:55:18.139 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.136+0000 7f3d77200640 1 --2- 172.21.15.32:0/2301394735 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f3d50077010 0x7f3d50079520 unknown :-1 s=CLOSED pgs=481 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:18.139 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.136+0000 7f3d77200640 1 --2- 172.21.15.32:0/2301394735 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3d700747b0 0x7f3d700ae670 unknown :-1 s=CLOSED pgs=1159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:18.139 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.136+0000 7f3d77200640 1 --2- 172.21.15.32:0/2301394735 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f3d70073e60 0x7f3d701f4ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:18.139 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.136+0000 7f3d77200640 1 --2- 172.21.15.32:0/2301394735 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f3d7006e930 0x7f3d700abf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:18.139 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.136+0000 7f3d77200640 1 -- 172.21.15.32:0/2301394735 >> 172.21.15.32:0/2301394735 conn(0x7f3d700b1bd0 msgr2=0x7f3d700b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:55:18.139 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.137+0000 7f3d77200640 1 -- 172.21.15.32:0/2301394735 shutdown_connections 2024-08-03T20:55:18.139 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.137+0000 7f3d77200640 1 -- 172.21.15.32:0/2301394735 wait complete. 2024-08-03T20:55:18.167 DEBUG:teuthology.orchestra.run.smithi032:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata import - 2024-08-03T20:55:18.310 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.309+0000 7f1d28c90f40 1 Processor -- start 2024-08-03T20:55:18.310 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.309+0000 7f1d28c90f40 1 -- start start 2024-08-03T20:55:18.313 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.310+0000 7f1d28c90f40 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x563b99b78360 0x563b99b76930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:55:18.313 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.310+0000 7f1d28c90f40 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x563b99b79ea0 0x563b99b089a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:55:18.313 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.310+0000 7f1d28c90f40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x563b99b95cb0 0x563b99ac1520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:55:18.313 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.310+0000 7f1d28c90f40 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x563b99a0fe30 con 0x563b99b79ea0 2024-08-03T20:55:18.313 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.310+0000 7f1d28c90f40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x563b99a6dbf0 con 0x563b99b95cb0 2024-08-03T20:55:18.313 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.310+0000 7f1d28c90f40 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x563b99b791b0 con 0x563b99b78360 2024-08-03T20:55:18.313 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.310+0000 7f1d21400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x563b99b79ea0 0x563b99b089a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:55:18.313 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.310+0000 7f1d21400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x563b99b79ea0 0x563b99b089a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:50608/0 (socket says 172.21.15.32:50608) 2024-08-03T20:55:18.313 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.310+0000 7f1d21400640 1 -- 172.21.15.32:0/3143504986 learned_addr learned my addr 172.21.15.32:0/3143504986 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:55:18.313 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.311+0000 7f1d22800640 1 --2- 172.21.15.32:0/3143504986 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x563b99b95cb0 0x563b99ac1520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:55:18.314 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.311+0000 7f1d21400640 1 -- 172.21.15.32:0/3143504986 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x563b99b78360 msgr2=0x563b99b76930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:55:18.314 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.311+0000 7f1d21400640 1 --2- 172.21.15.32:0/3143504986 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x563b99b78360 0x563b99b76930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:18.314 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.311+0000 7f1d21400640 1 -- 172.21.15.32:0/3143504986 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x563b99b95cb0 msgr2=0x563b99ac1520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:55:18.314 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.311+0000 7f1d21400640 1 --2- 172.21.15.32:0/3143504986 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x563b99b95cb0 0x563b99ac1520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:18.314 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.311+0000 7f1d21400640 1 -- 172.21.15.32:0/3143504986 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563b99b93a20 con 0x563b99b79ea0 2024-08-03T20:55:18.314 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.311+0000 7f1d21400640 1 --2- 172.21.15.32:0/3143504986 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x563b99b79ea0 0x563b99b089a0 secure :-1 s=READY pgs=1162 cs=0 l=1 rev1=1 crypto rx=0x7f1d1000b600 tx=0x7f1d1000bac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4655f8a7d318e5a9 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:55:18.314 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.311+0000 7f1d1be00640 1 -- 172.21.15.32:0/3143504986 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d1000a8f0 con 0x563b99b79ea0 2024-08-03T20:55:18.314 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.311+0000 7f1d1be00640 1 -- 172.21.15.32:0/3143504986 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f1d1000aa90 con 0x563b99b79ea0 2024-08-03T20:55:18.314 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.311+0000 7f1d1be00640 1 -- 172.21.15.32:0/3143504986 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d10008450 con 0x563b99b79ea0 2024-08-03T20:55:18.315 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.312+0000 7f1d28c90f40 1 -- 172.21.15.32:0/3143504986 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x563b99b79ea0 msgr2=0x563b99b089a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:55:18.315 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.312+0000 7f1d28c90f40 1 --2- 172.21.15.32:0/3143504986 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x563b99b79ea0 0x563b99b089a0 secure :-1 s=READY pgs=1162 cs=0 l=1 rev1=1 crypto rx=0x7f1d1000b600 tx=0x7f1d1000bac0 comp rx=0 tx=0).stop 2024-08-03T20:55:18.315 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.312+0000 7f1d28c90f40 1 -- 172.21.15.32:0/3143504986 shutdown_connections 2024-08-03T20:55:18.315 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.312+0000 7f1d28c90f40 1 --2- 172.21.15.32:0/3143504986 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x563b99b95cb0 0x563b99ac1520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:18.316 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.312+0000 7f1d28c90f40 1 --2- 172.21.15.32:0/3143504986 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x563b99b79ea0 0x563b99b089a0 unknown :-1 s=CLOSED pgs=1162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:18.316 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.312+0000 7f1d28c90f40 1 --2- 172.21.15.32:0/3143504986 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x563b99b78360 0x563b99b76930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:18.316 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.312+0000 7f1d28c90f40 1 -- 172.21.15.32:0/3143504986 >> 172.21.15.32:0/3143504986 conn(0x563b999e1ac0 msgr2=0x563b99b08000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:55:18.316 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.312+0000 7f1d28c90f40 1 -- 172.21.15.32:0/3143504986 shutdown_connections 2024-08-03T20:55:18.316 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.312+0000 7f1d28c90f40 1 -- 172.21.15.32:0/3143504986 wait complete. 2024-08-03T20:55:18.316 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.313+0000 7f1d28c90f40 1 Processor -- start 2024-08-03T20:55:18.316 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.313+0000 7f1d28c90f40 1 -- start start 2024-08-03T20:55:18.318 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.315+0000 7f1d28c90f40 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x563b99b78360 0x563b99b75fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:55:18.318 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.315+0000 7f1d28c90f40 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x563b99b79ea0 0x563b99b94030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:55:18.318 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.315+0000 7f1d28c90f40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x563b99b95cb0 0x563b99b94570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:55:18.318 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.315+0000 7f1d28c90f40 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x563b99b758e0 con 0x563b99b78360 2024-08-03T20:55:18.318 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.315+0000 7f1d28c90f40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x563b99b022f0 con 0x563b99b95cb0 2024-08-03T20:55:18.318 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.315+0000 7f1d28c90f40 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x563b99b01a50 con 0x563b99b79ea0 2024-08-03T20:55:18.319 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.315+0000 7f1d21e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x563b99b78360 0x563b99b75fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:55:18.319 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.315+0000 7f1d21e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x563b99b78360 0x563b99b75fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:50612/0 (socket says 172.21.15.32:50612) 2024-08-03T20:55:18.319 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.315+0000 7f1d21e00640 1 -- 172.21.15.32:0/3176098597 learned_addr learned my addr 172.21.15.32:0/3176098597 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:55:18.319 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.315+0000 7f1d22800640 1 --2- 172.21.15.32:0/3176098597 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x563b99b95cb0 0x563b99b94570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:55:18.319 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.315+0000 7f1d21400640 1 --2- 172.21.15.32:0/3176098597 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x563b99b79ea0 0x563b99b94030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:55:18.319 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.315+0000 7f1d21e00640 1 -- 172.21.15.32:0/3176098597 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x563b99b79ea0 msgr2=0x563b99b94030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:55:18.319 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.315+0000 7f1d21e00640 1 --2- 172.21.15.32:0/3176098597 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x563b99b79ea0 0x563b99b94030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:18.319 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.315+0000 7f1d21e00640 1 -- 172.21.15.32:0/3176098597 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x563b99b95cb0 msgr2=0x563b99b94570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:55:18.319 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.315+0000 7f1d21e00640 1 --2- 172.21.15.32:0/3176098597 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x563b99b95cb0 0x563b99b94570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:18.319 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.315+0000 7f1d21e00640 1 -- 172.21.15.32:0/3176098597 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563b99b01d30 con 0x563b99b78360 2024-08-03T20:55:18.319 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.315+0000 7f1d21400640 1 --2- 172.21.15.32:0/3176098597 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x563b99b79ea0 0x563b99b94030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-03T20:55:18.319 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.316+0000 7f1d21e00640 1 --2- 172.21.15.32:0/3176098597 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x563b99b78360 0x563b99b75fb0 secure :-1 s=READY pgs=1163 cs=0 l=1 rev1=1 crypto rx=0x7f1d14000c00 tx=0x7f1d140041f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:55:18.319 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.316+0000 7f1d1aa00640 1 -- 172.21.15.32:0/3176098597 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d14066070 con 0x563b99b78360 2024-08-03T20:55:18.320 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.316+0000 7f1d1aa00640 1 -- 172.21.15.32:0/3176098597 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f1d14056560 con 0x563b99b78360 2024-08-03T20:55:18.320 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.316+0000 7f1d28c90f40 1 -- 172.21.15.32:0/3176098597 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x563b99b02010 con 0x563b99b78360 2024-08-03T20:55:18.320 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.316+0000 7f1d1aa00640 1 -- 172.21.15.32:0/3176098597 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d14056990 con 0x563b99b78360 2024-08-03T20:55:18.320 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.316+0000 7f1d28c90f40 1 -- 172.21.15.32:0/3176098597 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563b99b75620 con 0x563b99b78360 2024-08-03T20:55:18.320 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.318+0000 7f1d1aa00640 1 -- 172.21.15.32:0/3176098597 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f1d14056b30 con 0x563b99b78360 2024-08-03T20:55:18.321 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.319+0000 7f1d1aa00640 1 --2- 172.21.15.32:0/3176098597 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f1d04075a20 0x7f1d04077f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:55:18.321 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.319+0000 7f1d1aa00640 1 -- 172.21.15.32:0/3176098597 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 6053+0+0 (secure 0 0 0) 0x7f1d140db050 con 0x563b99b78360 2024-08-03T20:55:18.321 INFO:teuthology.orchestra.run.smithi032.stdout:Importing pool 2024-08-03T20:55:18.322 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.320+0000 7f1d21400640 1 --2- 172.21.15.32:0/3176098597 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f1d04075a20 0x7f1d04077f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:55:18.322 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.320+0000 7f1d28c90f40 1 --2- 172.21.15.32:0/3176098597 >> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] conn(0x563b99c31e80 0x563b99c3a790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:55:18.322 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.320+0000 7f1d28c90f40 1 -- 172.21.15.32:0/3176098597 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:1 12.10 12:09401a81:::601.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e87) -- 0x563b99b764f0 con 0x563b99c31e80 2024-08-03T20:55:18.322 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.320+0000 7f1d22800640 1 --2- 172.21.15.32:0/3176098597 >> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] conn(0x563b99c31e80 0x563b99c3a790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:55:18.322 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.321+0000 7f1d21400640 1 --2- 172.21.15.32:0/3176098597 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f1d04075a20 0x7f1d04077f30 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x563b99c264f0 tx=0x7f1d10002fb0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:55:18.322 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.321+0000 7f1d22800640 1 --2- 172.21.15.32:0/3176098597 >> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] conn(0x563b99c31e80 0x563b99c3a790 crc :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:55:18.324 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.323+0000 7f1d22800640 1 -- 172.21.15.32:0/3176098597 <== osd.0 v2:172.21.15.32:6816/3066156116 1 ==== osd_op_reply(1 601.00000000 [create] v87'102 uv78 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f1d1c0026e0 con 0x563b99c31e80 2024-08-03T20:55:18.324 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.323+0000 7f1d28c90f40 1 -- 172.21.15.32:0/3176098597 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:2 12.10 12:09401a81:::601.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e87) -- 0x563b99ac1590 con 0x563b99c31e80 2024-08-03T20:55:18.326 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.325+0000 7f1d22800640 1 -- 172.21.15.32:0/3176098597 <== osd.0 v2:172.21.15.32:6816/3066156116 2 ==== osd_op_reply(2 601.00000000 [delete] v87'103 uv103 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f1d1c0026e0 con 0x563b99c31e80 2024-08-03T20:55:18.326 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.325+0000 7f1d28c90f40 1 -- 172.21.15.32:0/3176098597 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:3 12.10 12:09401a81:::601.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e87) -- 0x563b99c32290 con 0x563b99c31e80 2024-08-03T20:55:18.328 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::601.00000000:head# 2024-08-03T20:55:18.328 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.326+0000 7f1d22800640 1 -- 172.21.15.32:0/3176098597 <== osd.0 v2:172.21.15.32:6816/3066156116 3 ==== osd_op_reply(3 601.00000000 [create] v87'104 uv104 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f1d1c0026e0 con 0x563b99c31e80 2024-08-03T20:55:18.328 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.326+0000 7f1d28c90f40 1 -- 172.21.15.32:0/3176098597 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:4 12.10 12:09401a81:::601.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e87) -- 0x563b99c3bce0 con 0x563b99c31e80 2024-08-03T20:55:18.329 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.328+0000 7f1d22800640 1 -- 172.21.15.32:0/3176098597 <== osd.0 v2:172.21.15.32:6816/3066156116 4 ==== osd_op_reply(4 601.00000000 [setxattr (30)] v87'105 uv105 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f1d1c0026e0 con 0x563b99c31e80 2024-08-03T20:55:18.330 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.328+0000 7f1d28c90f40 1 -- 172.21.15.32:0/3176098597 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:5 12.10 12:09401a81:::601.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e87) -- 0x563b99c3c080 con 0x563b99c31e80 2024-08-03T20:55:18.331 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.330+0000 7f1d22800640 1 -- 172.21.15.32:0/3176098597 <== osd.0 v2:172.21.15.32:6816/3066156116 5 ==== osd_op_reply(5 601.00000000 [setxattr (62)] v87'106 uv106 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f1d1c0026e0 con 0x563b99c31e80 2024-08-03T20:55:18.331 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.330+0000 7f1d28c90f40 1 -- 172.21.15.32:0/3176098597 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:6 12.10 12:09401a81:::601.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e87) -- 0x563b99c3acd0 con 0x563b99c31e80 2024-08-03T20:55:18.333 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.331+0000 7f1d22800640 1 -- 172.21.15.32:0/3176098597 <== osd.0 v2:172.21.15.32:6816/3066156116 6 ==== osd_op_reply(6 601.00000000 [omap-set-header] v87'107 uv107 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f1d1c0026e0 con 0x563b99c31e80 2024-08-03T20:55:18.333 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.331+0000 7f1d28c90f40 1 -- 172.21.15.32:0/3176098597 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:7 12.10 12:0bd6d154:::602.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e87) -- 0x563b99c3b0c0 con 0x563b99c31e80 2024-08-03T20:55:18.334 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.333+0000 7f1d22800640 1 -- 172.21.15.32:0/3176098597 <== osd.0 v2:172.21.15.32:6816/3066156116 7 ==== osd_op_reply(7 602.00000000 [create] v87'108 uv84 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f1d1c0026e0 con 0x563b99c31e80 2024-08-03T20:55:18.334 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.333+0000 7f1d28c90f40 1 -- 172.21.15.32:0/3176098597 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:8 12.10 12:0bd6d154:::602.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e87) -- 0x563b99c3b4b0 con 0x563b99c31e80 2024-08-03T20:55:18.336 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.335+0000 7f1d22800640 1 -- 172.21.15.32:0/3176098597 <== osd.0 v2:172.21.15.32:6816/3066156116 8 ==== osd_op_reply(8 602.00000000 [delete] v87'109 uv109 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f1d1c0026e0 con 0x563b99c31e80 2024-08-03T20:55:18.336 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.335+0000 7f1d28c90f40 1 -- 172.21.15.32:0/3176098597 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:9 12.10 12:0bd6d154:::602.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e87) -- 0x563b99c3b4b0 con 0x563b99c31e80 2024-08-03T20:55:18.338 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::602.00000000:head# 2024-08-03T20:55:18.338 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.336+0000 7f1d22800640 1 -- 172.21.15.32:0/3176098597 <== osd.0 v2:172.21.15.32:6816/3066156116 9 ==== osd_op_reply(9 602.00000000 [create] v87'110 uv110 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f1d1c0026e0 con 0x563b99c31e80 2024-08-03T20:55:18.338 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.336+0000 7f1d28c90f40 1 -- 172.21.15.32:0/3176098597 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:10 12.10 12:0bd6d154:::602.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e87) -- 0x563b99c3b460 con 0x563b99c31e80 2024-08-03T20:55:18.339 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.338+0000 7f1d22800640 1 -- 172.21.15.32:0/3176098597 <== osd.0 v2:172.21.15.32:6816/3066156116 10 ==== osd_op_reply(10 602.00000000 [setxattr (30)] v87'111 uv111 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f1d1c0026e0 con 0x563b99c31e80 2024-08-03T20:55:18.340 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.338+0000 7f1d28c90f40 1 -- 172.21.15.32:0/3176098597 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:11 12.10 12:0bd6d154:::602.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e87) -- 0x563b99c3b460 con 0x563b99c31e80 2024-08-03T20:55:18.341 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.340+0000 7f1d22800640 1 -- 172.21.15.32:0/3176098597 <== osd.0 v2:172.21.15.32:6816/3066156116 11 ==== osd_op_reply(11 602.00000000 [setxattr (62)] v87'112 uv112 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f1d1c0026e0 con 0x563b99c31e80 2024-08-03T20:55:18.341 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.340+0000 7f1d28c90f40 1 -- 172.21.15.32:0/3176098597 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:12 12.10 12:0bd6d154:::602.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e87) -- 0x563b99c3b460 con 0x563b99c31e80 2024-08-03T20:55:18.343 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.341+0000 7f1d22800640 1 -- 172.21.15.32:0/3176098597 <== osd.0 v2:172.21.15.32:6816/3066156116 12 ==== osd_op_reply(12 602.00000000 [omap-set-header] v87'113 uv113 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f1d1c0026e0 con 0x563b99c31e80 2024-08-03T20:55:18.343 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.342+0000 7f1d28c90f40 1 -- 172.21.15.32:0/3176098597 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:13 12.10 12:0cf6ad0f:::10000000000.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e87) -- 0x563b99c3b460 con 0x563b99c31e80 2024-08-03T20:55:18.344 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.343+0000 7f1d22800640 1 -- 172.21.15.32:0/3176098597 <== osd.0 v2:172.21.15.32:6816/3066156116 13 ==== osd_op_reply(13 10000000000.00000000 [create] v87'114 uv91 ondisk = -17 ((17) File exists)) ==== 164+0+0 (crc 0 0 0) 0x7f1d1c0026e0 con 0x563b99c31e80 2024-08-03T20:55:18.345 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.343+0000 7f1d28c90f40 1 -- 172.21.15.32:0/3176098597 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:14 12.10 12:0cf6ad0f:::10000000000.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e87) -- 0x563b99c3b460 con 0x563b99c31e80 2024-08-03T20:55:18.346 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.345+0000 7f1d22800640 1 -- 172.21.15.32:0/3176098597 <== osd.0 v2:172.21.15.32:6816/3066156116 14 ==== osd_op_reply(14 10000000000.00000000 [delete] v87'115 uv115 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f1d1c0026e0 con 0x563b99c31e80 2024-08-03T20:55:18.346 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.345+0000 7f1d28c90f40 1 -- 172.21.15.32:0/3176098597 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:15 12.10 12:0cf6ad0f:::10000000000.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e87) -- 0x563b99c3b460 con 0x563b99c31e80 2024-08-03T20:55:18.348 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::10000000000.00000000:head# 2024-08-03T20:55:18.348 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.347+0000 7f1d22800640 1 -- 172.21.15.32:0/3176098597 <== osd.0 v2:172.21.15.32:6816/3066156116 15 ==== osd_op_reply(15 10000000000.00000000 [create] v87'116 uv116 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f1d1c0026e0 con 0x563b99c31e80 2024-08-03T20:55:18.348 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.347+0000 7f1d28c90f40 1 -- 172.21.15.32:0/3176098597 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:16 12.10 12:0cf6ad0f:::10000000000.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e87) -- 0x563b99c3b460 con 0x563b99c31e80 2024-08-03T20:55:18.349 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.348+0000 7f1d22800640 1 -- 172.21.15.32:0/3176098597 <== osd.0 v2:172.21.15.32:6816/3066156116 16 ==== osd_op_reply(16 10000000000.00000000 [setxattr (30)] v87'117 uv117 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f1d1c0026e0 con 0x563b99c31e80 2024-08-03T20:55:18.350 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.348+0000 7f1d28c90f40 1 -- 172.21.15.32:0/3176098597 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:17 12.10 12:0cf6ad0f:::10000000000.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e87) -- 0x563b99c3b460 con 0x563b99c31e80 2024-08-03T20:55:18.351 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.350+0000 7f1d22800640 1 -- 172.21.15.32:0/3176098597 <== osd.0 v2:172.21.15.32:6816/3066156116 17 ==== osd_op_reply(17 10000000000.00000000 [setxattr (62)] v87'118 uv118 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f1d1c0026e0 con 0x563b99c31e80 2024-08-03T20:55:18.351 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.350+0000 7f1d28c90f40 1 -- 172.21.15.32:0/3176098597 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:18 12.10 12:0cf6ad0f:::10000000000.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e87) -- 0x563b99c3b460 con 0x563b99c31e80 2024-08-03T20:55:18.353 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.352+0000 7f1d22800640 1 -- 172.21.15.32:0/3176098597 <== osd.0 v2:172.21.15.32:6816/3066156116 18 ==== osd_op_reply(18 10000000000.00000000 [omap-set-header] v87'119 uv119 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f1d1c0026e0 con 0x563b99c31e80 2024-08-03T20:55:18.353 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.352+0000 7f1d28c90f40 1 -- 172.21.15.32:0/3176098597 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:19 12.10 12:0cf6ad0f:::10000000000.00000000:head [omap-set-vals in=593b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e87) -- 0x563b99c3c420 con 0x563b99c31e80 2024-08-03T20:55:18.355 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.353+0000 7f1d22800640 1 -- 172.21.15.32:0/3176098597 <== osd.0 v2:172.21.15.32:6816/3066156116 19 ==== osd_op_reply(19 10000000000.00000000 [omap-set-vals] v87'120 uv120 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f1d1c0026e0 con 0x563b99c31e80 2024-08-03T20:55:18.355 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.353+0000 7f1d28c90f40 1 -- 172.21.15.32:0/3176098597 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:20 12.10 12:0d82a743:::600.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e87) -- 0x563b99c3c420 con 0x563b99c31e80 2024-08-03T20:55:18.356 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.355+0000 7f1d22800640 1 -- 172.21.15.32:0/3176098597 <== osd.0 v2:172.21.15.32:6816/3066156116 20 ==== osd_op_reply(20 600.00000000 [create] v87'121 uv95 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f1d1c0026e0 con 0x563b99c31e80 2024-08-03T20:55:18.356 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.355+0000 7f1d28c90f40 1 -- 172.21.15.32:0/3176098597 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:21 12.10 12:0d82a743:::600.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e87) -- 0x563b99c3c420 con 0x563b99c31e80 2024-08-03T20:55:18.358 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.356+0000 7f1d22800640 1 -- 172.21.15.32:0/3176098597 <== osd.0 v2:172.21.15.32:6816/3066156116 21 ==== osd_op_reply(21 600.00000000 [delete] v87'122 uv122 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f1d1c0026e0 con 0x563b99c31e80 2024-08-03T20:55:18.358 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.356+0000 7f1d28c90f40 1 -- 172.21.15.32:0/3176098597 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:22 12.10 12:0d82a743:::600.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e87) -- 0x563b99c3c420 con 0x563b99c31e80 2024-08-03T20:55:18.359 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.358+0000 7f1d22800640 1 -- 172.21.15.32:0/3176098597 <== osd.0 v2:172.21.15.32:6816/3066156116 22 ==== osd_op_reply(22 600.00000000 [create] v87'123 uv123 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f1d1c0026e0 con 0x563b99c31e80 2024-08-03T20:55:18.360 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::600.00000000:head# 2024-08-03T20:55:18.360 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.358+0000 7f1d28c90f40 1 -- 172.21.15.32:0/3176098597 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:23 12.10 12:0d82a743:::600.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e87) -- 0x563b99c3d430 con 0x563b99c31e80 2024-08-03T20:55:18.361 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.359+0000 7f1d22800640 1 -- 172.21.15.32:0/3176098597 <== osd.0 v2:172.21.15.32:6816/3066156116 23 ==== osd_op_reply(23 600.00000000 [setxattr (30)] v87'124 uv124 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f1d1c0026e0 con 0x563b99c31e80 2024-08-03T20:55:18.361 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.359+0000 7f1d28c90f40 1 -- 172.21.15.32:0/3176098597 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:24 12.10 12:0d82a743:::600.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e87) -- 0x563b99c3d430 con 0x563b99c31e80 2024-08-03T20:55:18.362 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.361+0000 7f1d22800640 1 -- 172.21.15.32:0/3176098597 <== osd.0 v2:172.21.15.32:6816/3066156116 24 ==== osd_op_reply(24 600.00000000 [setxattr (62)] v87'125 uv125 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f1d1c0026e0 con 0x563b99c31e80 2024-08-03T20:55:18.363 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.361+0000 7f1d28c90f40 1 -- 172.21.15.32:0/3176098597 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:25 12.10 12:0d82a743:::600.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e87) -- 0x563b99c3c420 con 0x563b99c31e80 2024-08-03T20:55:18.364 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.363+0000 7f1d22800640 1 -- 172.21.15.32:0/3176098597 <== osd.0 v2:172.21.15.32:6816/3066156116 25 ==== osd_op_reply(25 600.00000000 [omap-set-header] v87'126 uv126 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f1d1c0026e0 con 0x563b99c31e80 2024-08-03T20:55:18.364 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.363+0000 7f1d28c90f40 1 -- 172.21.15.32:0/3176098597 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:26 12.10 12:0d89b25e:::603.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e87) -- 0x563b99c3c420 con 0x563b99c31e80 2024-08-03T20:55:18.366 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.364+0000 7f1d22800640 1 -- 172.21.15.32:0/3176098597 <== osd.0 v2:172.21.15.32:6816/3066156116 26 ==== osd_op_reply(26 603.00000000 [create] v87'127 uv101 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f1d1c0026e0 con 0x563b99c31e80 2024-08-03T20:55:18.366 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.364+0000 7f1d28c90f40 1 -- 172.21.15.32:0/3176098597 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:27 12.10 12:0d89b25e:::603.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e87) -- 0x563b99c3c420 con 0x563b99c31e80 2024-08-03T20:55:18.367 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.366+0000 7f1d22800640 1 -- 172.21.15.32:0/3176098597 <== osd.0 v2:172.21.15.32:6816/3066156116 27 ==== osd_op_reply(27 603.00000000 [delete] v87'128 uv128 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f1d1c0026e0 con 0x563b99c31e80 2024-08-03T20:55:18.367 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.366+0000 7f1d28c90f40 1 -- 172.21.15.32:0/3176098597 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:28 12.10 12:0d89b25e:::603.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e87) -- 0x563b99c3c420 con 0x563b99c31e80 2024-08-03T20:55:18.369 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.367+0000 7f1d22800640 1 -- 172.21.15.32:0/3176098597 <== osd.0 v2:172.21.15.32:6816/3066156116 28 ==== osd_op_reply(28 603.00000000 [create] v87'129 uv129 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f1d1c0026e0 con 0x563b99c31e80 2024-08-03T20:55:18.369 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::603.00000000:head# 2024-08-03T20:55:18.369 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.367+0000 7f1d28c90f40 1 -- 172.21.15.32:0/3176098597 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:29 12.10 12:0d89b25e:::603.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e87) -- 0x563b99c3d430 con 0x563b99c31e80 2024-08-03T20:55:18.370 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.369+0000 7f1d22800640 1 -- 172.21.15.32:0/3176098597 <== osd.0 v2:172.21.15.32:6816/3066156116 29 ==== osd_op_reply(29 603.00000000 [setxattr (30)] v87'130 uv130 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f1d1c0026e0 con 0x563b99c31e80 2024-08-03T20:55:18.371 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.369+0000 7f1d28c90f40 1 -- 172.21.15.32:0/3176098597 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:30 12.10 12:0d89b25e:::603.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e87) -- 0x563b99c3d430 con 0x563b99c31e80 2024-08-03T20:55:18.372 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.371+0000 7f1d22800640 1 -- 172.21.15.32:0/3176098597 <== osd.0 v2:172.21.15.32:6816/3066156116 30 ==== osd_op_reply(30 603.00000000 [setxattr (62)] v87'131 uv131 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f1d1c0026e0 con 0x563b99c31e80 2024-08-03T20:55:18.372 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.371+0000 7f1d28c90f40 1 -- 172.21.15.32:0/3176098597 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:31 12.10 12:0d89b25e:::603.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e87) -- 0x563b99c3c420 con 0x563b99c31e80 2024-08-03T20:55:18.374 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.372+0000 7f1d22800640 1 -- 172.21.15.32:0/3176098597 <== osd.0 v2:172.21.15.32:6816/3066156116 31 ==== osd_op_reply(31 603.00000000 [omap-set-header] v87'132 uv132 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f1d1c0026e0 con 0x563b99c31e80 2024-08-03T20:55:18.374 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.372+0000 7f1d28c90f40 1 --2- 172.21.15.32:0/3176098597 >> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] conn(0x563b99c3c930 0x563b99c3ed50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:55:18.374 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.372+0000 7f1d28c90f40 1 -- 172.21.15.32:0/3176098597 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:32 12.14 12:287034c4:::1.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e87) -- 0x563b99c3f290 con 0x563b99c3c930 2024-08-03T20:55:18.374 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.373+0000 7f1d21e00640 1 --2- 172.21.15.32:0/3176098597 >> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] conn(0x563b99c3c930 0x563b99c3ed50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:55:18.375 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.374+0000 7f1d21e00640 1 --2- 172.21.15.32:0/3176098597 >> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] conn(0x563b99c3c930 0x563b99c3ed50 crc :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:55:18.377 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.376+0000 7f1d21e00640 1 -- 172.21.15.32:0/3176098597 <== osd.5 v2:172.21.15.118:6810/2724891770 1 ==== osd_op_reply(32 1.00000000.inode [create] v87'89 uv89 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f1d1405f070 con 0x563b99c3c930 2024-08-03T20:55:18.377 INFO:teuthology.orchestra.run.smithi032.stdout:Write #-9223372036854775808:00000000:::1.00000000.inode:head# 2024-08-03T20:55:18.377 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.376+0000 7f1d28c90f40 1 -- 172.21.15.32:0/3176098597 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:33 12.14 12:287034c4:::1.00000000.inode:head [write 0~582 in=582b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e87) -- 0x563b99c3f630 con 0x563b99c3c930 2024-08-03T20:55:18.379 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.377+0000 7f1d21e00640 1 -- 172.21.15.32:0/3176098597 <== osd.5 v2:172.21.15.118:6810/2724891770 2 ==== osd_op_reply(33 1.00000000.inode [write 0~582] v87'90 uv90 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f1d1405f070 con 0x563b99c3c930 2024-08-03T20:55:18.379 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.378+0000 7f1d28c90f40 1 -- 172.21.15.32:0/3176098597 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:34 12.14 12:287034c4:::1.00000000.inode:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e87) -- 0x563b99c3f9d0 con 0x563b99c3c930 2024-08-03T20:55:18.380 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.379+0000 7f1d21e00640 1 -- 172.21.15.32:0/3176098597 <== osd.5 v2:172.21.15.118:6810/2724891770 3 ==== osd_op_reply(34 1.00000000.inode [omap-set-header] v87'91 uv91 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f1d1405f070 con 0x563b99c3c930 2024-08-03T20:55:18.380 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.379+0000 7f1d28c90f40 1 -- 172.21.15.32:0/3176098597 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:35 12.14 12:292cf221:::200.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e87) -- 0x563b99c3fd70 con 0x563b99c3c930 2024-08-03T20:55:18.382 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.381+0000 7f1d21e00640 1 -- 172.21.15.32:0/3176098597 <== osd.5 v2:172.21.15.118:6810/2724891770 4 ==== osd_op_reply(35 200.00000000 [create] v87'92 uv76 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f1d1405f070 con 0x563b99c3c930 2024-08-03T20:55:18.382 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.381+0000 7f1d28c90f40 1 -- 172.21.15.32:0/3176098597 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:36 12.14 12:292cf221:::200.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e87) -- 0x563b99c40110 con 0x563b99c3c930 2024-08-03T20:55:18.384 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.382+0000 7f1d21e00640 1 -- 172.21.15.32:0/3176098597 <== osd.5 v2:172.21.15.118:6810/2724891770 5 ==== osd_op_reply(36 200.00000000 [delete] v87'93 uv93 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f1d1405f070 con 0x563b99c3c930 2024-08-03T20:55:18.384 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.382+0000 7f1d28c90f40 1 -- 172.21.15.32:0/3176098597 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:37 12.14 12:292cf221:::200.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e87) -- 0x563b99c404b0 con 0x563b99c3c930 2024-08-03T20:55:18.385 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.384+0000 7f1d21e00640 1 -- 172.21.15.32:0/3176098597 <== osd.5 v2:172.21.15.118:6810/2724891770 6 ==== osd_op_reply(37 200.00000000 [create] v87'94 uv94 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f1d1405f070 con 0x563b99c3c930 2024-08-03T20:55:18.385 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::200.00000000:head# 2024-08-03T20:55:18.386 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.384+0000 7f1d28c90f40 1 -- 172.21.15.32:0/3176098597 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:38 12.14 12:292cf221:::200.00000000:head [write 0~90 in=90b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e87) -- 0x563b99c40850 con 0x563b99c3c930 2024-08-03T20:55:18.387 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.386+0000 7f1d21e00640 1 -- 172.21.15.32:0/3176098597 <== osd.5 v2:172.21.15.118:6810/2724891770 7 ==== osd_op_reply(38 200.00000000 [write 0~90] v87'95 uv95 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f1d1405f070 con 0x563b99c3c930 2024-08-03T20:55:18.387 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.386+0000 7f1d28c90f40 1 -- 172.21.15.32:0/3176098597 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:39 12.14 12:292cf221:::200.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e87) -- 0x563b99c40850 con 0x563b99c3c930 2024-08-03T20:55:18.388 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.387+0000 7f1d21e00640 1 -- 172.21.15.32:0/3176098597 <== osd.5 v2:172.21.15.118:6810/2724891770 8 ==== osd_op_reply(39 200.00000000 [omap-set-header] v87'96 uv96 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f1d1405f070 con 0x563b99c3c930 2024-08-03T20:55:18.389 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.387+0000 7f1d28c90f40 1 -- 172.21.15.32:0/3176098597 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:40 12.14 12:2e2fa760:::200.00000001:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e87) -- 0x563b99c40850 con 0x563b99c3c930 2024-08-03T20:55:18.390 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.389+0000 7f1d21e00640 1 -- 172.21.15.32:0/3176098597 <== osd.5 v2:172.21.15.118:6810/2724891770 9 ==== osd_op_reply(40 200.00000001 [create] v87'97 uv81 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f1d1405f070 con 0x563b99c3c930 2024-08-03T20:55:18.390 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.389+0000 7f1d28c90f40 1 -- 172.21.15.32:0/3176098597 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:41 12.14 12:2e2fa760:::200.00000001:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e87) -- 0x563b99c40850 con 0x563b99c3c930 2024-08-03T20:55:18.392 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.390+0000 7f1d21e00640 1 -- 172.21.15.32:0/3176098597 <== osd.5 v2:172.21.15.118:6810/2724891770 10 ==== osd_op_reply(41 200.00000001 [delete] v87'98 uv98 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f1d1405f070 con 0x563b99c3c930 2024-08-03T20:55:18.392 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.390+0000 7f1d28c90f40 1 -- 172.21.15.32:0/3176098597 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:42 12.14 12:2e2fa760:::200.00000001:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e87) -- 0x563b99c40850 con 0x563b99c3c930 2024-08-03T20:55:18.393 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.391+0000 7f1d21e00640 1 -- 172.21.15.32:0/3176098597 <== osd.5 v2:172.21.15.118:6810/2724891770 11 ==== osd_op_reply(42 200.00000001 [create] v87'99 uv99 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f1d1405f070 con 0x563b99c3c930 2024-08-03T20:55:18.393 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::200.00000001:head# 2024-08-03T20:55:18.394 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.392+0000 7f1d28c90f40 1 -- 172.21.15.32:0/3176098597 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:43 12.14 12:2e2fa760:::200.00000001:head [write 0~18118 in=18118b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e87) -- 0x563b99c44f40 con 0x563b99c3c930 2024-08-03T20:55:18.395 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.393+0000 7f1d21e00640 1 -- 172.21.15.32:0/3176098597 <== osd.5 v2:172.21.15.118:6810/2724891770 12 ==== osd_op_reply(43 200.00000001 [write 0~18118] v87'100 uv100 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f1d1405f070 con 0x563b99c3c930 2024-08-03T20:55:18.395 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.393+0000 7f1d28c90f40 1 -- 172.21.15.32:0/3176098597 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:44 12.14 12:2e2fa760:::200.00000001:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e87) -- 0x563b99c44f40 con 0x563b99c3c930 2024-08-03T20:55:18.396 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.395+0000 7f1d21e00640 1 -- 172.21.15.32:0/3176098597 <== osd.5 v2:172.21.15.118:6810/2724891770 13 ==== osd_op_reply(44 200.00000001 [omap-set-header] v87'101 uv101 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f1d1405f070 con 0x563b99c3c930 2024-08-03T20:55:18.396 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.395+0000 7f1d28c90f40 1 -- 172.21.15.32:0/3176098597 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:45 12.14 12:2f55791f:::606.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e87) -- 0x563b99c40850 con 0x563b99c3c930 2024-08-03T20:55:18.398 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.396+0000 7f1d21e00640 1 -- 172.21.15.32:0/3176098597 <== osd.5 v2:172.21.15.118:6810/2724891770 14 ==== osd_op_reply(45 606.00000000 [create] v87'102 uv87 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f1d1405f070 con 0x563b99c3c930 2024-08-03T20:55:18.398 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.396+0000 7f1d28c90f40 1 -- 172.21.15.32:0/3176098597 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:46 12.14 12:2f55791f:::606.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e87) -- 0x563b99c40850 con 0x563b99c3c930 2024-08-03T20:55:18.399 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.398+0000 7f1d21e00640 1 -- 172.21.15.32:0/3176098597 <== osd.5 v2:172.21.15.118:6810/2724891770 15 ==== osd_op_reply(46 606.00000000 [delete] v87'103 uv103 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f1d1405f070 con 0x563b99c3c930 2024-08-03T20:55:18.399 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.398+0000 7f1d28c90f40 1 -- 172.21.15.32:0/3176098597 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:47 12.14 12:2f55791f:::606.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e87) -- 0x563b99c40850 con 0x563b99c3c930 2024-08-03T20:55:18.400 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.399+0000 7f1d21e00640 1 -- 172.21.15.32:0/3176098597 <== osd.5 v2:172.21.15.118:6810/2724891770 16 ==== osd_op_reply(47 606.00000000 [create] v87'104 uv104 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f1d1405f070 con 0x563b99c3c930 2024-08-03T20:55:18.401 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::606.00000000:head# 2024-08-03T20:55:18.401 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.399+0000 7f1d28c90f40 1 -- 172.21.15.32:0/3176098597 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:48 12.14 12:2f55791f:::606.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e87) -- 0x563b99c41860 con 0x563b99c3c930 2024-08-03T20:55:18.402 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.401+0000 7f1d21e00640 1 -- 172.21.15.32:0/3176098597 <== osd.5 v2:172.21.15.118:6810/2724891770 17 ==== osd_op_reply(48 606.00000000 [setxattr (30)] v87'105 uv105 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f1d1405f070 con 0x563b99c3c930 2024-08-03T20:55:18.402 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.401+0000 7f1d28c90f40 1 -- 172.21.15.32:0/3176098597 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:49 12.14 12:2f55791f:::606.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e87) -- 0x563b99c41860 con 0x563b99c3c930 2024-08-03T20:55:18.403 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.402+0000 7f1d21e00640 1 -- 172.21.15.32:0/3176098597 <== osd.5 v2:172.21.15.118:6810/2724891770 18 ==== osd_op_reply(49 606.00000000 [setxattr (62)] v87'106 uv106 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f1d1405f070 con 0x563b99c3c930 2024-08-03T20:55:18.404 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.402+0000 7f1d28c90f40 1 -- 172.21.15.32:0/3176098597 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:50 12.14 12:2f55791f:::606.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e87) -- 0x563b99c40850 con 0x563b99c3c930 2024-08-03T20:55:18.405 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.404+0000 7f1d21e00640 1 -- 172.21.15.32:0/3176098597 <== osd.5 v2:172.21.15.118:6810/2724891770 19 ==== osd_op_reply(50 606.00000000 [omap-set-header] v87'107 uv107 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f1d1405f070 con 0x563b99c3c930 2024-08-03T20:55:18.405 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.404+0000 7f1d28c90f40 1 -- 172.21.15.32:0/3176098597 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:51 12.1c 12:3ed09add:::607.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e87) -- 0x563b99c40850 con 0x563b99c31e80 2024-08-03T20:55:18.407 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.405+0000 7f1d22800640 1 -- 172.21.15.32:0/3176098597 <== osd.0 v2:172.21.15.32:6816/3066156116 32 ==== osd_op_reply(51 607.00000000 [create] v87'21 uv20 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f1d1c0026e0 con 0x563b99c31e80 2024-08-03T20:55:18.408 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.405+0000 7f1d28c90f40 1 -- 172.21.15.32:0/3176098597 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:52 12.1c 12:3ed09add:::607.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e87) -- 0x563b99c40850 con 0x563b99c31e80 2024-08-03T20:55:18.408 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.407+0000 7f1d22800640 1 -- 172.21.15.32:0/3176098597 <== osd.0 v2:172.21.15.32:6816/3066156116 33 ==== osd_op_reply(52 607.00000000 [delete] v87'22 uv22 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f1d1c0026e0 con 0x563b99c31e80 2024-08-03T20:55:18.408 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.407+0000 7f1d28c90f40 1 -- 172.21.15.32:0/3176098597 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:53 12.1c 12:3ed09add:::607.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e87) -- 0x563b99c40850 con 0x563b99c31e80 2024-08-03T20:55:18.409 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.408+0000 7f1d22800640 1 -- 172.21.15.32:0/3176098597 <== osd.0 v2:172.21.15.32:6816/3066156116 34 ==== osd_op_reply(53 607.00000000 [create] v87'23 uv23 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f1d1c0026e0 con 0x563b99c31e80 2024-08-03T20:55:18.410 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::607.00000000:head# 2024-08-03T20:55:18.410 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.408+0000 7f1d28c90f40 1 -- 172.21.15.32:0/3176098597 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:54 12.1c 12:3ed09add:::607.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e87) -- 0x563b99c41860 con 0x563b99c31e80 2024-08-03T20:55:18.411 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.409+0000 7f1d22800640 1 -- 172.21.15.32:0/3176098597 <== osd.0 v2:172.21.15.32:6816/3066156116 35 ==== osd_op_reply(54 607.00000000 [setxattr (30)] v87'24 uv24 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f1d1c0026e0 con 0x563b99c31e80 2024-08-03T20:55:18.411 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.409+0000 7f1d28c90f40 1 -- 172.21.15.32:0/3176098597 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:55 12.1c 12:3ed09add:::607.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e87) -- 0x563b99c41860 con 0x563b99c31e80 2024-08-03T20:55:18.412 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.411+0000 7f1d22800640 1 -- 172.21.15.32:0/3176098597 <== osd.0 v2:172.21.15.32:6816/3066156116 36 ==== osd_op_reply(55 607.00000000 [setxattr (62)] v87'25 uv25 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f1d1c0026e0 con 0x563b99c31e80 2024-08-03T20:55:18.412 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.411+0000 7f1d28c90f40 1 -- 172.21.15.32:0/3176098597 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:56 12.1c 12:3ed09add:::607.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e87) -- 0x563b99c40850 con 0x563b99c31e80 2024-08-03T20:55:18.413 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.412+0000 7f1d22800640 1 -- 172.21.15.32:0/3176098597 <== osd.0 v2:172.21.15.32:6816/3066156116 37 ==== osd_op_reply(56 607.00000000 [omap-set-header] v87'26 uv26 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f1d1c0026e0 con 0x563b99c31e80 2024-08-03T20:55:18.414 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.412+0000 7f1d28c90f40 1 --2- 172.21.15.32:0/3176098597 >> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] conn(0x563b99c40ed0 0x563b99c41340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:55:18.414 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.412+0000 7f1d28c90f40 1 -- 172.21.15.32:0/3176098597 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:57 12.6 12:654134d2:::mds0_openfiles.0:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e87) -- 0x563b99c43cc0 con 0x563b99c40ed0 2024-08-03T20:55:18.414 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.412+0000 7f1d21400640 1 --2- 172.21.15.32:0/3176098597 >> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] conn(0x563b99c40ed0 0x563b99c41340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:55:18.414 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.413+0000 7f1d21400640 1 --2- 172.21.15.32:0/3176098597 >> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] conn(0x563b99c40ed0 0x563b99c41340 crc :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:55:18.416 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.414+0000 7f1d21400640 1 -- 172.21.15.32:0/3176098597 <== osd.1 v2:172.21.15.32:6824/1551241759 1 ==== osd_op_reply(57 mds0_openfiles.0 [create] v87'20 uv19 ondisk = -17 ((17) File exists)) ==== 160+0+0 (crc 0 0 0) 0x7f1d10025430 con 0x563b99c40ed0 2024-08-03T20:55:18.416 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.414+0000 7f1d28c90f40 1 -- 172.21.15.32:0/3176098597 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:58 12.6 12:654134d2:::mds0_openfiles.0:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e87) -- 0x563b99c44060 con 0x563b99c40ed0 2024-08-03T20:55:18.417 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.416+0000 7f1d21400640 1 -- 172.21.15.32:0/3176098597 <== osd.1 v2:172.21.15.32:6824/1551241759 2 ==== osd_op_reply(58 mds0_openfiles.0 [delete] v87'21 uv21 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f1d10025430 con 0x563b99c40ed0 2024-08-03T20:55:18.417 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.416+0000 7f1d28c90f40 1 -- 172.21.15.32:0/3176098597 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:59 12.6 12:654134d2:::mds0_openfiles.0:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e87) -- 0x563b99c44480 con 0x563b99c40ed0 2024-08-03T20:55:18.418 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.417+0000 7f1d21400640 1 -- 172.21.15.32:0/3176098597 <== osd.1 v2:172.21.15.32:6824/1551241759 3 ==== osd_op_reply(59 mds0_openfiles.0 [create] v87'22 uv22 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f1d10025430 con 0x563b99c40ed0 2024-08-03T20:55:18.418 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds0_openfiles.0:head# 2024-08-03T20:55:18.419 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.417+0000 7f1d28c90f40 1 -- 172.21.15.32:0/3176098597 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:60 12.6 12:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e87) -- 0x563b99b8ecf0 con 0x563b99c40ed0 2024-08-03T20:55:18.419 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.418+0000 7f1d21400640 1 -- 172.21.15.32:0/3176098597 <== osd.1 v2:172.21.15.32:6824/1551241759 4 ==== osd_op_reply(60 mds0_openfiles.0 [omap-set-header] v87'23 uv23 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f1d10025430 con 0x563b99c40ed0 2024-08-03T20:55:18.419 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.418+0000 7f1d28c90f40 1 -- 172.21.15.32:0/3176098597 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:61 12.e 12:717a0223:::608.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e87) -- 0x563b99c44d90 con 0x563b99c31e80 2024-08-03T20:55:18.421 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.420+0000 7f1d22800640 1 -- 172.21.15.32:0/3176098597 <== osd.0 v2:172.21.15.32:6816/3066156116 38 ==== osd_op_reply(61 608.00000000 [create] v87'21 uv20 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f1d1c0026e0 con 0x563b99c31e80 2024-08-03T20:55:18.421 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.420+0000 7f1d28c90f40 1 -- 172.21.15.32:0/3176098597 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:62 12.e 12:717a0223:::608.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e87) -- 0x563b99c44d90 con 0x563b99c31e80 2024-08-03T20:55:18.423 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.421+0000 7f1d22800640 1 -- 172.21.15.32:0/3176098597 <== osd.0 v2:172.21.15.32:6816/3066156116 39 ==== osd_op_reply(62 608.00000000 [delete] v87'22 uv22 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f1d1c0026e0 con 0x563b99c31e80 2024-08-03T20:55:18.423 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.421+0000 7f1d28c90f40 1 -- 172.21.15.32:0/3176098597 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:63 12.e 12:717a0223:::608.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e87) -- 0x563b99c44d90 con 0x563b99c31e80 2024-08-03T20:55:18.424 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::608.00000000:head# 2024-08-03T20:55:18.424 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.423+0000 7f1d22800640 1 -- 172.21.15.32:0/3176098597 <== osd.0 v2:172.21.15.32:6816/3066156116 40 ==== osd_op_reply(63 608.00000000 [create] v87'23 uv23 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f1d1c0026e0 con 0x563b99c31e80 2024-08-03T20:55:18.425 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.423+0000 7f1d28c90f40 1 -- 172.21.15.32:0/3176098597 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:64 12.e 12:717a0223:::608.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e87) -- 0x563b99c45da0 con 0x563b99c31e80 2024-08-03T20:55:18.426 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.424+0000 7f1d22800640 1 -- 172.21.15.32:0/3176098597 <== osd.0 v2:172.21.15.32:6816/3066156116 41 ==== osd_op_reply(64 608.00000000 [setxattr (30)] v87'24 uv24 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f1d1c0026e0 con 0x563b99c31e80 2024-08-03T20:55:18.426 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.424+0000 7f1d28c90f40 1 -- 172.21.15.32:0/3176098597 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:65 12.e 12:717a0223:::608.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e87) -- 0x563b99c45da0 con 0x563b99c31e80 2024-08-03T20:55:18.427 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.426+0000 7f1d22800640 1 -- 172.21.15.32:0/3176098597 <== osd.0 v2:172.21.15.32:6816/3066156116 42 ==== osd_op_reply(65 608.00000000 [setxattr (62)] v87'25 uv25 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f1d1c0026e0 con 0x563b99c31e80 2024-08-03T20:55:18.427 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.426+0000 7f1d28c90f40 1 -- 172.21.15.32:0/3176098597 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:66 12.e 12:717a0223:::608.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e87) -- 0x563b99c44d90 con 0x563b99c31e80 2024-08-03T20:55:18.429 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.427+0000 7f1d22800640 1 -- 172.21.15.32:0/3176098597 <== osd.0 v2:172.21.15.32:6816/3066156116 43 ==== osd_op_reply(66 608.00000000 [omap-set-header] v87'26 uv26 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f1d1c0026e0 con 0x563b99c31e80 2024-08-03T20:55:18.429 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.427+0000 7f1d28c90f40 1 --2- 172.21.15.32:0/3176098597 >> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] conn(0x563b99c45410 0x563b99c45860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:55:18.429 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.427+0000 7f1d28c90f40 1 -- 172.21.15.32:0/3176098597 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:67 12.15 12:a93a17c2:::604.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e87) -- 0x563b99c481a0 con 0x563b99c45410 2024-08-03T20:55:18.429 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.428+0000 7f1d22800640 1 --2- 172.21.15.32:0/3176098597 >> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] conn(0x563b99c45410 0x563b99c45860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:55:18.430 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.428+0000 7f1d22800640 1 --2- 172.21.15.32:0/3176098597 >> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] conn(0x563b99c45410 0x563b99c45860 crc :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:55:18.431 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.430+0000 7f1d22800640 1 -- 172.21.15.32:0/3176098597 <== osd.6 v2:172.21.15.118:6825/801947386 1 ==== osd_op_reply(67 604.00000000 [create] v87'37 uv31 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f1d1c0026e0 con 0x563b99c45410 2024-08-03T20:55:18.431 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.430+0000 7f1d28c90f40 1 -- 172.21.15.32:0/3176098597 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:68 12.15 12:a93a17c2:::604.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e87) -- 0x563b99c481a0 con 0x563b99c45410 2024-08-03T20:55:18.433 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.431+0000 7f1d22800640 1 -- 172.21.15.32:0/3176098597 <== osd.6 v2:172.21.15.118:6825/801947386 2 ==== osd_op_reply(68 604.00000000 [delete] v87'38 uv38 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f1d1c0026e0 con 0x563b99c45410 2024-08-03T20:55:18.433 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.431+0000 7f1d28c90f40 1 -- 172.21.15.32:0/3176098597 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:69 12.15 12:a93a17c2:::604.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e87) -- 0x563b99c481a0 con 0x563b99c45410 2024-08-03T20:55:18.434 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::604.00000000:head# 2024-08-03T20:55:18.435 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.433+0000 7f1d22800640 1 -- 172.21.15.32:0/3176098597 <== osd.6 v2:172.21.15.118:6825/801947386 3 ==== osd_op_reply(69 604.00000000 [create] v87'39 uv39 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f1d1c0026e0 con 0x563b99c45410 2024-08-03T20:55:18.435 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.433+0000 7f1d28c90f40 1 -- 172.21.15.32:0/3176098597 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:70 12.15 12:a93a17c2:::604.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e87) -- 0x563b99c49210 con 0x563b99c45410 2024-08-03T20:55:18.436 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.435+0000 7f1d22800640 1 -- 172.21.15.32:0/3176098597 <== osd.6 v2:172.21.15.118:6825/801947386 4 ==== osd_op_reply(70 604.00000000 [setxattr (30)] v87'40 uv40 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f1d1c0026e0 con 0x563b99c45410 2024-08-03T20:55:18.436 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.435+0000 7f1d28c90f40 1 -- 172.21.15.32:0/3176098597 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:71 12.15 12:a93a17c2:::604.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e87) -- 0x563b99c491d0 con 0x563b99c45410 2024-08-03T20:55:18.437 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.436+0000 7f1d22800640 1 -- 172.21.15.32:0/3176098597 <== osd.6 v2:172.21.15.118:6825/801947386 5 ==== osd_op_reply(71 604.00000000 [setxattr (62)] v87'41 uv41 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f1d1c0026e0 con 0x563b99c45410 2024-08-03T20:55:18.437 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.436+0000 7f1d28c90f40 1 -- 172.21.15.32:0/3176098597 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:72 12.15 12:a93a17c2:::604.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e87) -- 0x563b99c491d0 con 0x563b99c45410 2024-08-03T20:55:18.439 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.438+0000 7f1d22800640 1 -- 172.21.15.32:0/3176098597 <== osd.6 v2:172.21.15.118:6825/801947386 6 ==== osd_op_reply(72 604.00000000 [omap-set-header] v87'42 uv42 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f1d1c0026e0 con 0x563b99c45410 2024-08-03T20:55:18.439 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.438+0000 7f1d28c90f40 1 -- 172.21.15.32:0/3176098597 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:73 12.15 12:aa448500:::500.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e87) -- 0x563b99c491d0 con 0x563b99c45410 2024-08-03T20:55:18.440 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.439+0000 7f1d22800640 1 -- 172.21.15.32:0/3176098597 <== osd.6 v2:172.21.15.118:6825/801947386 7 ==== osd_op_reply(73 500.00000000 [create] v87'43 uv36 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f1d1c0026e0 con 0x563b99c45410 2024-08-03T20:55:18.441 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.439+0000 7f1d28c90f40 1 -- 172.21.15.32:0/3176098597 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:74 12.15 12:aa448500:::500.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e87) -- 0x563b99c491d0 con 0x563b99c45410 2024-08-03T20:55:18.442 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.441+0000 7f1d22800640 1 -- 172.21.15.32:0/3176098597 <== osd.6 v2:172.21.15.118:6825/801947386 8 ==== osd_op_reply(74 500.00000000 [delete] v87'44 uv44 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f1d1c0026e0 con 0x563b99c45410 2024-08-03T20:55:18.442 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.441+0000 7f1d28c90f40 1 -- 172.21.15.32:0/3176098597 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:75 12.15 12:aa448500:::500.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e87) -- 0x563b99c491d0 con 0x563b99c45410 2024-08-03T20:55:18.443 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::500.00000000:head# 2024-08-03T20:55:18.444 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.442+0000 7f1d22800640 1 -- 172.21.15.32:0/3176098597 <== osd.6 v2:172.21.15.118:6825/801947386 9 ==== osd_op_reply(75 500.00000000 [create] v87'45 uv45 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f1d1c0026e0 con 0x563b99c45410 2024-08-03T20:55:18.444 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.442+0000 7f1d28c90f40 1 -- 172.21.15.32:0/3176098597 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:76 12.15 12:aa448500:::500.00000000:head [write 0~90 in=90b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e87) -- 0x563b99c491d0 con 0x563b99c45410 2024-08-03T20:55:18.445 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.444+0000 7f1d22800640 1 -- 172.21.15.32:0/3176098597 <== osd.6 v2:172.21.15.118:6825/801947386 10 ==== osd_op_reply(76 500.00000000 [write 0~90] v87'46 uv46 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f1d1c0026e0 con 0x563b99c45410 2024-08-03T20:55:18.445 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.444+0000 7f1d28c90f40 1 -- 172.21.15.32:0/3176098597 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:77 12.15 12:aa448500:::500.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e87) -- 0x563b99c491d0 con 0x563b99c45410 2024-08-03T20:55:18.447 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.445+0000 7f1d22800640 1 -- 172.21.15.32:0/3176098597 <== osd.6 v2:172.21.15.118:6825/801947386 11 ==== osd_op_reply(77 500.00000000 [omap-set-header] v87'47 uv47 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f1d1c0026e0 con 0x563b99c45410 2024-08-03T20:55:18.447 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.445+0000 7f1d28c90f40 1 --2- 172.21.15.32:0/3176098597 >> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] conn(0x563b99c48720 0x563b99c48b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:55:18.447 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.445+0000 7f1d28c90f40 1 -- 172.21.15.32:0/3176098597 --> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] -- osd_op(unknown.0.0:78 12.d 12:b50e409b:::mds_snaptable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e87) -- 0x563b99c4b620 con 0x563b99c48720 2024-08-03T20:55:18.447 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.446+0000 7f1d21e00640 1 --2- 172.21.15.32:0/3176098597 >> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] conn(0x563b99c48720 0x563b99c48b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:55:18.447 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.446+0000 7f1d21e00640 1 --2- 172.21.15.32:0/3176098597 >> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] conn(0x563b99c48720 0x563b99c48b30 crc :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:55:18.449 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.447+0000 7f1d21e00640 1 -- 172.21.15.32:0/3176098597 <== osd.7 v2:172.21.15.118:6818/653000048 1 ==== osd_op_reply(78 mds_snaptable [create] v87'17 uv16 ondisk = -17 ((17) File exists)) ==== 157+0+0 (crc 0 0 0) 0x7f1d1405f070 con 0x563b99c48720 2024-08-03T20:55:18.449 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.448+0000 7f1d28c90f40 1 -- 172.21.15.32:0/3176098597 --> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] -- osd_op(unknown.0.0:79 12.d 12:b50e409b:::mds_snaptable:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e87) -- 0x563b99c4b620 con 0x563b99c48720 2024-08-03T20:55:18.450 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.449+0000 7f1d21e00640 1 -- 172.21.15.32:0/3176098597 <== osd.7 v2:172.21.15.118:6818/653000048 2 ==== osd_op_reply(79 mds_snaptable [delete] v87'18 uv18 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f1d1405f070 con 0x563b99c48720 2024-08-03T20:55:18.450 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.449+0000 7f1d28c90f40 1 -- 172.21.15.32:0/3176098597 --> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] -- osd_op(unknown.0.0:80 12.d 12:b50e409b:::mds_snaptable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e87) -- 0x563b99c4b6a0 con 0x563b99c48720 2024-08-03T20:55:18.452 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds_snaptable:head# 2024-08-03T20:55:18.452 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.450+0000 7f1d21e00640 1 -- 172.21.15.32:0/3176098597 <== osd.7 v2:172.21.15.118:6818/653000048 3 ==== osd_op_reply(80 mds_snaptable [create] v87'19 uv19 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f1d1405f070 con 0x563b99c48720 2024-08-03T20:55:18.452 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.451+0000 7f1d28c90f40 1 -- 172.21.15.32:0/3176098597 --> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] -- osd_op(unknown.0.0:81 12.d 12:b50e409b:::mds_snaptable:head [write 0~70 in=70b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e87) -- 0x563b99c4ba90 con 0x563b99c48720 2024-08-03T20:55:18.453 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.452+0000 7f1d21e00640 1 -- 172.21.15.32:0/3176098597 <== osd.7 v2:172.21.15.118:6818/653000048 4 ==== osd_op_reply(81 mds_snaptable [write 0~70] v87'20 uv20 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f1d1405f070 con 0x563b99c48720 2024-08-03T20:55:18.454 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.452+0000 7f1d28c90f40 1 -- 172.21.15.32:0/3176098597 --> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] -- osd_op(unknown.0.0:82 12.d 12:b50e409b:::mds_snaptable:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e87) -- 0x563b99c4ba90 con 0x563b99c48720 2024-08-03T20:55:18.455 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.454+0000 7f1d21e00640 1 -- 172.21.15.32:0/3176098597 <== osd.7 v2:172.21.15.118:6818/653000048 5 ==== osd_op_reply(82 mds_snaptable [omap-set-header] v87'21 uv21 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f1d1405f070 con 0x563b99c48720 2024-08-03T20:55:18.455 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.454+0000 7f1d28c90f40 1 -- 172.21.15.32:0/3176098597 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:83 12.1d 12:b871830b:::605.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e87) -- 0x563b99c4ba90 con 0x563b99c31e80 2024-08-03T20:55:18.457 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.455+0000 7f1d22800640 1 -- 172.21.15.32:0/3176098597 <== osd.0 v2:172.21.15.32:6816/3066156116 44 ==== osd_op_reply(83 605.00000000 [create] v87'21 uv20 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f1d1c0026e0 con 0x563b99c31e80 2024-08-03T20:55:18.457 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.455+0000 7f1d28c90f40 1 -- 172.21.15.32:0/3176098597 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:84 12.1d 12:b871830b:::605.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e87) -- 0x563b99c4ba90 con 0x563b99c31e80 2024-08-03T20:55:18.458 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.457+0000 7f1d22800640 1 -- 172.21.15.32:0/3176098597 <== osd.0 v2:172.21.15.32:6816/3066156116 45 ==== osd_op_reply(84 605.00000000 [delete] v87'22 uv22 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f1d1c0026e0 con 0x563b99c31e80 2024-08-03T20:55:18.458 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.457+0000 7f1d28c90f40 1 -- 172.21.15.32:0/3176098597 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:85 12.1d 12:b871830b:::605.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e87) -- 0x563b99c4ba90 con 0x563b99c31e80 2024-08-03T20:55:18.459 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::605.00000000:head# 2024-08-03T20:55:18.460 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.458+0000 7f1d22800640 1 -- 172.21.15.32:0/3176098597 <== osd.0 v2:172.21.15.32:6816/3066156116 46 ==== osd_op_reply(85 605.00000000 [create] v87'23 uv23 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f1d1c0026e0 con 0x563b99c31e80 2024-08-03T20:55:18.460 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.458+0000 7f1d28c90f40 1 -- 172.21.15.32:0/3176098597 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:86 12.1d 12:b871830b:::605.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e87) -- 0x563b99c4caa0 con 0x563b99c31e80 2024-08-03T20:55:18.461 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.460+0000 7f1d22800640 1 -- 172.21.15.32:0/3176098597 <== osd.0 v2:172.21.15.32:6816/3066156116 47 ==== osd_op_reply(86 605.00000000 [setxattr (30)] v87'24 uv24 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f1d1c0026e0 con 0x563b99c31e80 2024-08-03T20:55:18.461 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.460+0000 7f1d28c90f40 1 -- 172.21.15.32:0/3176098597 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:87 12.1d 12:b871830b:::605.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e87) -- 0x563b99c4caa0 con 0x563b99c31e80 2024-08-03T20:55:18.462 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.461+0000 7f1d22800640 1 -- 172.21.15.32:0/3176098597 <== osd.0 v2:172.21.15.32:6816/3066156116 48 ==== osd_op_reply(87 605.00000000 [setxattr (62)] v87'25 uv25 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f1d1c0026e0 con 0x563b99c31e80 2024-08-03T20:55:18.462 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.461+0000 7f1d28c90f40 1 -- 172.21.15.32:0/3176098597 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:88 12.1d 12:b871830b:::605.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e87) -- 0x563b99c4ba90 con 0x563b99c31e80 2024-08-03T20:55:18.464 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.462+0000 7f1d22800640 1 -- 172.21.15.32:0/3176098597 <== osd.0 v2:172.21.15.32:6816/3066156116 49 ==== osd_op_reply(88 605.00000000 [omap-set-header] v87'26 uv26 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f1d1c0026e0 con 0x563b99c31e80 2024-08-03T20:55:18.464 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.462+0000 7f1d28c90f40 1 -- 172.21.15.32:0/3176098597 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:89 12.13 12:c91d4a1d:::mds0_inotable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e87) -- 0x563b99c4ba90 con 0x563b99c40ed0 2024-08-03T20:55:18.465 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.464+0000 7f1d21400640 1 -- 172.21.15.32:0/3176098597 <== osd.1 v2:172.21.15.32:6824/1551241759 5 ==== osd_op_reply(89 mds0_inotable [create] v87'34 uv28 ondisk = -17 ((17) File exists)) ==== 157+0+0 (crc 0 0 0) 0x7f1d10025430 con 0x563b99c40ed0 2024-08-03T20:55:18.465 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.464+0000 7f1d28c90f40 1 -- 172.21.15.32:0/3176098597 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:90 12.13 12:c91d4a1d:::mds0_inotable:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e87) -- 0x563b99c4be30 con 0x563b99c40ed0 2024-08-03T20:55:18.466 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.465+0000 7f1d21400640 1 -- 172.21.15.32:0/3176098597 <== osd.1 v2:172.21.15.32:6824/1551241759 6 ==== osd_op_reply(90 mds0_inotable [delete] v87'35 uv35 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f1d10025430 con 0x563b99c40ed0 2024-08-03T20:55:18.466 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.465+0000 7f1d28c90f40 1 -- 172.21.15.32:0/3176098597 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:91 12.13 12:c91d4a1d:::mds0_inotable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e87) -- 0x563b99c4c1d0 con 0x563b99c40ed0 2024-08-03T20:55:18.467 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds0_inotable:head# 2024-08-03T20:55:18.468 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.466+0000 7f1d21400640 1 -- 172.21.15.32:0/3176098597 <== osd.1 v2:172.21.15.32:6824/1551241759 7 ==== osd_op_reply(91 mds0_inotable [create] v87'36 uv36 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f1d10025430 con 0x563b99c40ed0 2024-08-03T20:55:18.468 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.466+0000 7f1d28c90f40 1 -- 172.21.15.32:0/3176098597 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:92 12.13 12:c91d4a1d:::mds0_inotable:head [write 0~34 in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e87) -- 0x563b99c4c570 con 0x563b99c40ed0 2024-08-03T20:55:18.469 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.468+0000 7f1d21400640 1 -- 172.21.15.32:0/3176098597 <== osd.1 v2:172.21.15.32:6824/1551241759 8 ==== osd_op_reply(92 mds0_inotable [write 0~34] v87'37 uv37 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f1d10025430 con 0x563b99c40ed0 2024-08-03T20:55:18.469 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.468+0000 7f1d28c90f40 1 -- 172.21.15.32:0/3176098597 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:93 12.13 12:c91d4a1d:::mds0_inotable:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e87) -- 0x563b99c4c570 con 0x563b99c40ed0 2024-08-03T20:55:18.470 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.469+0000 7f1d21400640 1 -- 172.21.15.32:0/3176098597 <== osd.1 v2:172.21.15.32:6824/1551241759 9 ==== osd_op_reply(93 mds0_inotable [omap-set-header] v87'38 uv38 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f1d10025430 con 0x563b99c40ed0 2024-08-03T20:55:18.470 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.469+0000 7f1d28c90f40 1 -- 172.21.15.32:0/3176098597 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:94 12.13 12:cd5a64a3:::100.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e87) -- 0x563b99c4c570 con 0x563b99c40ed0 2024-08-03T20:55:18.471 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.470+0000 7f1d21400640 1 -- 172.21.15.32:0/3176098597 <== osd.1 v2:172.21.15.32:6824/1551241759 10 ==== osd_op_reply(94 100.00000000 [create] v87'39 uv33 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f1d10025430 con 0x563b99c40ed0 2024-08-03T20:55:18.472 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.470+0000 7f1d28c90f40 1 -- 172.21.15.32:0/3176098597 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:95 12.13 12:cd5a64a3:::100.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e87) -- 0x563b99c4c570 con 0x563b99c40ed0 2024-08-03T20:55:18.473 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.471+0000 7f1d21400640 1 -- 172.21.15.32:0/3176098597 <== osd.1 v2:172.21.15.32:6824/1551241759 11 ==== osd_op_reply(95 100.00000000 [delete] v87'40 uv40 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f1d10025430 con 0x563b99c40ed0 2024-08-03T20:55:18.473 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.471+0000 7f1d28c90f40 1 -- 172.21.15.32:0/3176098597 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:96 12.13 12:cd5a64a3:::100.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e87) -- 0x563b99c4c570 con 0x563b99c40ed0 2024-08-03T20:55:18.474 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::100.00000000:head# 2024-08-03T20:55:18.474 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.473+0000 7f1d21400640 1 -- 172.21.15.32:0/3176098597 <== osd.1 v2:172.21.15.32:6824/1551241759 12 ==== osd_op_reply(96 100.00000000 [create] v87'41 uv41 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f1d10025430 con 0x563b99c40ed0 2024-08-03T20:55:18.474 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.473+0000 7f1d28c90f40 1 -- 172.21.15.32:0/3176098597 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:97 12.13 12:cd5a64a3:::100.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e87) -- 0x563b99c4c570 con 0x563b99c40ed0 2024-08-03T20:55:18.475 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.474+0000 7f1d21400640 1 -- 172.21.15.32:0/3176098597 <== osd.1 v2:172.21.15.32:6824/1551241759 13 ==== osd_op_reply(97 100.00000000 [omap-set-header] v87'42 uv42 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f1d10025430 con 0x563b99c40ed0 2024-08-03T20:55:18.475 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.474+0000 7f1d28c90f40 1 -- 172.21.15.32:0/3176098597 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:98 12.13 12:cd5a64a3:::100.00000000:head [omap-set-vals in=5194b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e87) -- 0x563b99c4f370 con 0x563b99c40ed0 2024-08-03T20:55:18.477 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.476+0000 7f1d21400640 1 -- 172.21.15.32:0/3176098597 <== osd.1 v2:172.21.15.32:6824/1551241759 14 ==== osd_op_reply(98 100.00000000 [omap-set-vals] v87'43 uv43 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f1d10025430 con 0x563b99c40ed0 2024-08-03T20:55:18.477 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.476+0000 7f1d28c90f40 1 --2- 172.21.15.32:0/3176098597 >> [v2:172.21.15.32:6800/9088303,v1:172.21.15.32:6801/9088303] conn(0x563b99c4c940 0x563b99c4ed60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:55:18.477 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.476+0000 7f1d28c90f40 1 -- 172.21.15.32:0/3176098597 --> [v2:172.21.15.32:6800/9088303,v1:172.21.15.32:6801/9088303] -- osd_op(unknown.0.0:99 12.b 12:d0630e4c:::mds0_sessionmap:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e87) -- 0x563b99c4fa30 con 0x563b99c4c940 2024-08-03T20:55:18.477 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.476+0000 7f1d21400640 1 --2- 172.21.15.32:0/3176098597 >> [v2:172.21.15.32:6800/9088303,v1:172.21.15.32:6801/9088303] conn(0x563b99c4c940 0x563b99c4ed60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:55:18.478 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.477+0000 7f1d21400640 1 --2- 172.21.15.32:0/3176098597 >> [v2:172.21.15.32:6800/9088303,v1:172.21.15.32:6801/9088303] conn(0x563b99c4c940 0x563b99c4ed60 crc :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:55:18.480 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.478+0000 7f1d21400640 1 -- 172.21.15.32:0/3176098597 <== osd.3 v2:172.21.15.32:6800/9088303 1 ==== osd_op_reply(99 mds0_sessionmap [create] v87'18 uv17 ondisk = -17 ((17) File exists)) ==== 159+0+0 (crc 0 0 0) 0x7f1d10025430 con 0x563b99c4c940 2024-08-03T20:55:18.480 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.478+0000 7f1d28c90f40 1 -- 172.21.15.32:0/3176098597 --> [v2:172.21.15.32:6800/9088303,v1:172.21.15.32:6801/9088303] -- osd_op(unknown.0.0:100 12.b 12:d0630e4c:::mds0_sessionmap:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e87) -- 0x563b99c4fa30 con 0x563b99c4c940 2024-08-03T20:55:18.481 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.480+0000 7f1d21400640 1 -- 172.21.15.32:0/3176098597 <== osd.3 v2:172.21.15.32:6800/9088303 2 ==== osd_op_reply(100 mds0_sessionmap [delete] v87'19 uv19 ondisk = 0) ==== 159+0+0 (crc 0 0 0) 0x7f1d10025430 con 0x563b99c4c940 2024-08-03T20:55:18.481 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.480+0000 7f1d28c90f40 1 -- 172.21.15.32:0/3176098597 --> [v2:172.21.15.32:6800/9088303,v1:172.21.15.32:6801/9088303] -- osd_op(unknown.0.0:101 12.b 12:d0630e4c:::mds0_sessionmap:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e87) -- 0x563b99c4fab0 con 0x563b99c4c940 2024-08-03T20:55:18.482 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds0_sessionmap:head# 2024-08-03T20:55:18.483 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.481+0000 7f1d21400640 1 -- 172.21.15.32:0/3176098597 <== osd.3 v2:172.21.15.32:6800/9088303 3 ==== osd_op_reply(101 mds0_sessionmap [create] v87'20 uv20 ondisk = 0) ==== 159+0+0 (crc 0 0 0) 0x7f1d10025430 con 0x563b99c4c940 2024-08-03T20:55:18.483 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.481+0000 7f1d28c90f40 1 -- 172.21.15.32:0/3176098597 --> [v2:172.21.15.32:6800/9088303,v1:172.21.15.32:6801/9088303] -- osd_op(unknown.0.0:102 12.b 12:d0630e4c:::mds0_sessionmap:head [omap-set-header in=14b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e87) -- 0x563b99c4fb30 con 0x563b99c4c940 2024-08-03T20:55:18.484 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.483+0000 7f1d21400640 1 -- 172.21.15.32:0/3176098597 <== osd.3 v2:172.21.15.32:6800/9088303 4 ==== osd_op_reply(102 mds0_sessionmap [omap-set-header] v87'21 uv21 ondisk = 0) ==== 159+0+0 (crc 0 0 0) 0x7f1d10025430 con 0x563b99c4c940 2024-08-03T20:55:18.484 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.483+0000 7f1d28c90f40 1 -- 172.21.15.32:0/3176098597 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:103 12.7 12:e0b41d06:::609.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e87) -- 0x563b99c4fb30 con 0x563b99c40ed0 2024-08-03T20:55:18.486 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.485+0000 7f1d21400640 1 -- 172.21.15.32:0/3176098597 <== osd.1 v2:172.21.15.32:6824/1551241759 15 ==== osd_op_reply(103 609.00000000 [create] v87'21 uv20 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f1d10025430 con 0x563b99c40ed0 2024-08-03T20:55:18.486 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.485+0000 7f1d28c90f40 1 -- 172.21.15.32:0/3176098597 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:104 12.7 12:e0b41d06:::609.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e87) -- 0x563b99c4fb30 con 0x563b99c40ed0 2024-08-03T20:55:18.487 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.486+0000 7f1d21400640 1 -- 172.21.15.32:0/3176098597 <== osd.1 v2:172.21.15.32:6824/1551241759 16 ==== osd_op_reply(104 609.00000000 [delete] v87'22 uv22 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f1d10025430 con 0x563b99c40ed0 2024-08-03T20:55:18.488 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.486+0000 7f1d28c90f40 1 -- 172.21.15.32:0/3176098597 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:105 12.7 12:e0b41d06:::609.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e87) -- 0x563b99c4fab0 con 0x563b99c40ed0 2024-08-03T20:55:18.489 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::609.00000000:head# 2024-08-03T20:55:18.489 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.488+0000 7f1d21400640 1 -- 172.21.15.32:0/3176098597 <== osd.1 v2:172.21.15.32:6824/1551241759 17 ==== osd_op_reply(105 609.00000000 [create] v87'23 uv23 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f1d10025430 con 0x563b99c40ed0 2024-08-03T20:55:18.489 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.488+0000 7f1d28c90f40 1 -- 172.21.15.32:0/3176098597 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:106 12.7 12:e0b41d06:::609.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e87) -- 0x563b99c4fab0 con 0x563b99c40ed0 2024-08-03T20:55:18.490 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.489+0000 7f1d21400640 1 -- 172.21.15.32:0/3176098597 <== osd.1 v2:172.21.15.32:6824/1551241759 18 ==== osd_op_reply(106 609.00000000 [setxattr (30)] v87'24 uv24 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f1d10025430 con 0x563b99c40ed0 2024-08-03T20:55:18.490 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.489+0000 7f1d28c90f40 1 -- 172.21.15.32:0/3176098597 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:107 12.7 12:e0b41d06:::609.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e87) -- 0x563b99c4fab0 con 0x563b99c40ed0 2024-08-03T20:55:18.492 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.491+0000 7f1d21400640 1 -- 172.21.15.32:0/3176098597 <== osd.1 v2:172.21.15.32:6824/1551241759 19 ==== osd_op_reply(107 609.00000000 [setxattr (62)] v87'25 uv25 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f1d10025430 con 0x563b99c40ed0 2024-08-03T20:55:18.492 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.491+0000 7f1d28c90f40 1 -- 172.21.15.32:0/3176098597 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:108 12.7 12:e0b41d06:::609.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e87) -- 0x563b99c4faf0 con 0x563b99c40ed0 2024-08-03T20:55:18.493 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.492+0000 7f1d21400640 1 -- 172.21.15.32:0/3176098597 <== osd.1 v2:172.21.15.32:6824/1551241759 20 ==== osd_op_reply(108 609.00000000 [omap-set-header] v87'26 uv26 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f1d10025430 con 0x563b99c40ed0 2024-08-03T20:55:18.493 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.492+0000 7f1d28c90f40 1 -- 172.21.15.32:0/3176098597 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:109 12.f 12:f1f69726:::400.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e87) -- 0x563b99c4faf0 con 0x563b99c45410 2024-08-03T20:55:18.495 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.494+0000 7f1d22800640 1 -- 172.21.15.32:0/3176098597 <== osd.6 v2:172.21.15.118:6825/801947386 12 ==== osd_op_reply(109 400.00000000 [create] v87'17 uv16 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f1d1c0026e0 con 0x563b99c45410 2024-08-03T20:55:18.495 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.494+0000 7f1d28c90f40 1 -- 172.21.15.32:0/3176098597 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:110 12.f 12:f1f69726:::400.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e87) -- 0x563b99c4fab0 con 0x563b99c45410 2024-08-03T20:55:18.496 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.495+0000 7f1d22800640 1 -- 172.21.15.32:0/3176098597 <== osd.6 v2:172.21.15.118:6825/801947386 13 ==== osd_op_reply(110 400.00000000 [delete] v87'18 uv18 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f1d1c0026e0 con 0x563b99c45410 2024-08-03T20:55:18.497 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.495+0000 7f1d28c90f40 1 -- 172.21.15.32:0/3176098597 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:111 12.f 12:f1f69726:::400.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e87) -- 0x563b99c4fab0 con 0x563b99c45410 2024-08-03T20:55:18.498 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.497+0000 7f1d22800640 1 -- 172.21.15.32:0/3176098597 <== osd.6 v2:172.21.15.118:6825/801947386 14 ==== osd_op_reply(111 400.00000000 [create] v87'19 uv19 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f1d1c0026e0 con 0x563b99c45410 2024-08-03T20:55:18.498 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::400.00000000:head# 2024-08-03T20:55:18.498 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.497+0000 7f1d28c90f40 1 -- 172.21.15.32:0/3176098597 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:112 12.f 12:f1f69726:::400.00000000:head [write 0~22 in=22b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e87) -- 0x563b99c4ff50 con 0x563b99c45410 2024-08-03T20:55:18.500 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.498+0000 7f1d22800640 1 -- 172.21.15.32:0/3176098597 <== osd.6 v2:172.21.15.118:6825/801947386 15 ==== osd_op_reply(112 400.00000000 [write 0~22] v87'20 uv20 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f1d1c0026e0 con 0x563b99c45410 2024-08-03T20:55:18.500 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.499+0000 7f1d28c90f40 1 -- 172.21.15.32:0/3176098597 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:113 12.f 12:f1f69726:::400.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e87) -- 0x563b99c4ff00 con 0x563b99c45410 2024-08-03T20:55:18.502 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.500+0000 7f1d22800640 1 -- 172.21.15.32:0/3176098597 <== osd.6 v2:172.21.15.118:6825/801947386 16 ==== osd_op_reply(113 400.00000000 [omap-set-header] v87'21 uv21 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f1d1c0026e0 con 0x563b99c45410 2024-08-03T20:55:18.502 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.500+0000 7f1d28c90f40 1 -- 172.21.15.32:0/3176098597 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:114 12.1f 12:fe07bba1:::100.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e87) -- 0x563b99c4ff00 con 0x563b99c45410 2024-08-03T20:55:18.503 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.502+0000 7f1d22800640 1 -- 172.21.15.32:0/3176098597 <== osd.6 v2:172.21.15.118:6825/801947386 17 ==== osd_op_reply(114 100.00000000.inode [create] v87'45 uv37 ondisk = -17 ((17) File exists)) ==== 162+0+0 (crc 0 0 0) 0x7f1d1c0026e0 con 0x563b99c45410 2024-08-03T20:55:18.503 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.502+0000 7f1d28c90f40 1 -- 172.21.15.32:0/3176098597 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:115 12.1f 12:fe07bba1:::100.00000000.inode:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e87) -- 0x563b99c4feb0 con 0x563b99c45410 2024-08-03T20:55:18.505 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.503+0000 7f1d22800640 1 -- 172.21.15.32:0/3176098597 <== osd.6 v2:172.21.15.118:6825/801947386 18 ==== osd_op_reply(115 100.00000000.inode [delete] v87'46 uv46 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7f1d1c0026e0 con 0x563b99c45410 2024-08-03T20:55:18.505 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.503+0000 7f1d28c90f40 1 -- 172.21.15.32:0/3176098597 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:116 12.1f 12:fe07bba1:::100.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e87) -- 0x563b99c4feb0 con 0x563b99c45410 2024-08-03T20:55:18.506 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::100.00000000.inode:head# 2024-08-03T20:55:18.506 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.505+0000 7f1d22800640 1 -- 172.21.15.32:0/3176098597 <== osd.6 v2:172.21.15.118:6825/801947386 19 ==== osd_op_reply(116 100.00000000.inode [create] v87'47 uv47 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7f1d1c0026e0 con 0x563b99c45410 2024-08-03T20:55:18.507 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.505+0000 7f1d28c90f40 1 -- 172.21.15.32:0/3176098597 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:117 12.1f 12:fe07bba1:::100.00000000.inode:head [write 0~582 in=582b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e87) -- 0x563b99c4feb0 con 0x563b99c45410 2024-08-03T20:55:18.508 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.506+0000 7f1d22800640 1 -- 172.21.15.32:0/3176098597 <== osd.6 v2:172.21.15.118:6825/801947386 20 ==== osd_op_reply(117 100.00000000.inode [write 0~582] v87'48 uv48 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7f1d1c0026e0 con 0x563b99c45410 2024-08-03T20:55:18.508 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.506+0000 7f1d28c90f40 1 -- 172.21.15.32:0/3176098597 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:118 12.1f 12:fe07bba1:::100.00000000.inode:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e87) -- 0x563b99c4feb0 con 0x563b99c45410 2024-08-03T20:55:18.509 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.508+0000 7f1d22800640 1 -- 172.21.15.32:0/3176098597 <== osd.6 v2:172.21.15.118:6825/801947386 21 ==== osd_op_reply(118 100.00000000.inode [omap-set-header] v87'49 uv49 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7f1d1c0026e0 con 0x563b99c45410 2024-08-03T20:55:18.509 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.508+0000 7f1d28c90f40 1 -- 172.21.15.32:0/3176098597 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:119 12.1f 12:ff5b34d6:::1.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e87) -- 0x563b99c4feb0 con 0x563b99c45410 2024-08-03T20:55:18.510 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.509+0000 7f1d22800640 1 -- 172.21.15.32:0/3176098597 <== osd.6 v2:172.21.15.118:6825/801947386 22 ==== osd_op_reply(119 1.00000000 [create] v87'50 uv44 ondisk = -17 ((17) File exists)) ==== 154+0+0 (crc 0 0 0) 0x7f1d1c0026e0 con 0x563b99c45410 2024-08-03T20:55:18.510 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.509+0000 7f1d28c90f40 1 -- 172.21.15.32:0/3176098597 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:120 12.1f 12:ff5b34d6:::1.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e87) -- 0x563b99c4feb0 con 0x563b99c45410 2024-08-03T20:55:18.512 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.511+0000 7f1d22800640 1 -- 172.21.15.32:0/3176098597 <== osd.6 v2:172.21.15.118:6825/801947386 23 ==== osd_op_reply(120 1.00000000 [delete] v87'51 uv51 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7f1d1c0026e0 con 0x563b99c45410 2024-08-03T20:55:18.512 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.511+0000 7f1d28c90f40 1 -- 172.21.15.32:0/3176098597 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:121 12.1f 12:ff5b34d6:::1.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e87) -- 0x563b99c4feb0 con 0x563b99c45410 2024-08-03T20:55:18.513 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::1.00000000:head# 2024-08-03T20:55:18.514 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.512+0000 7f1d22800640 1 -- 172.21.15.32:0/3176098597 <== osd.6 v2:172.21.15.118:6825/801947386 24 ==== osd_op_reply(121 1.00000000 [create] v87'52 uv52 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7f1d1c0026e0 con 0x563b99c45410 2024-08-03T20:55:18.514 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.512+0000 7f1d28c90f40 1 -- 172.21.15.32:0/3176098597 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:122 12.1f 12:ff5b34d6:::1.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e87) -- 0x563b99c50ec0 con 0x563b99c45410 2024-08-03T20:55:18.515 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.513+0000 7f1d22800640 1 -- 172.21.15.32:0/3176098597 <== osd.6 v2:172.21.15.118:6825/801947386 25 ==== osd_op_reply(122 1.00000000 [setxattr (30)] v87'53 uv53 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7f1d1c0026e0 con 0x563b99c45410 2024-08-03T20:55:18.515 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.513+0000 7f1d28c90f40 1 -- 172.21.15.32:0/3176098597 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:123 12.1f 12:ff5b34d6:::1.00000000:head [setxattr parent (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e87) -- 0x563b99c50ec0 con 0x563b99c45410 2024-08-03T20:55:18.516 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.515+0000 7f1d22800640 1 -- 172.21.15.32:0/3176098597 <== osd.6 v2:172.21.15.118:6825/801947386 26 ==== osd_op_reply(123 1.00000000 [setxattr (30)] v87'54 uv54 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7f1d1c0026e0 con 0x563b99c45410 2024-08-03T20:55:18.516 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.515+0000 7f1d28c90f40 1 -- 172.21.15.32:0/3176098597 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:124 12.1f 12:ff5b34d6:::1.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e87) -- 0x563b99c4feb0 con 0x563b99c45410 2024-08-03T20:55:18.517 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.516+0000 7f1d22800640 1 -- 172.21.15.32:0/3176098597 <== osd.6 v2:172.21.15.118:6825/801947386 27 ==== osd_op_reply(124 1.00000000 [omap-set-header] v87'55 uv55 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7f1d1c0026e0 con 0x563b99c45410 2024-08-03T20:55:18.517 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.516+0000 7f1d28c90f40 1 -- 172.21.15.32:0/3176098597 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:125 12.1f 12:ff5b34d6:::1.00000000:head [omap-set-vals in=584b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e87) -- 0x563b99c50610 con 0x563b99c45410 2024-08-03T20:55:18.520 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.518+0000 7f1d22800640 1 -- 172.21.15.32:0/3176098597 <== osd.6 v2:172.21.15.118:6825/801947386 28 ==== osd_op_reply(125 1.00000000 [omap-set-vals] v87'56 uv56 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7f1d1c0026e0 con 0x563b99c45410 2024-08-03T20:55:18.520 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.518+0000 7f1d28c90f40 1 -- 172.21.15.32:0/3176098597 >> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] conn(0x563b99c31e80 msgr2=0x563b99c3a790 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:55:18.520 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.518+0000 7f1d28c90f40 1 --2- 172.21.15.32:0/3176098597 >> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] conn(0x563b99c31e80 0x563b99c3a790 crc :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:18.520 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.518+0000 7f1d28c90f40 1 -- 172.21.15.32:0/3176098597 >> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] conn(0x563b99c40ed0 msgr2=0x563b99c41340 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:55:18.520 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.518+0000 7f1d28c90f40 1 --2- 172.21.15.32:0/3176098597 >> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] conn(0x563b99c40ed0 0x563b99c41340 crc :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:18.520 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.518+0000 7f1d28c90f40 1 -- 172.21.15.32:0/3176098597 >> [v2:172.21.15.32:6800/9088303,v1:172.21.15.32:6801/9088303] conn(0x563b99c4c940 msgr2=0x563b99c4ed60 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:55:18.520 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.518+0000 7f1d28c90f40 1 --2- 172.21.15.32:0/3176098597 >> [v2:172.21.15.32:6800/9088303,v1:172.21.15.32:6801/9088303] conn(0x563b99c4c940 0x563b99c4ed60 crc :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:18.520 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.518+0000 7f1d28c90f40 1 -- 172.21.15.32:0/3176098597 >> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] conn(0x563b99c3c930 msgr2=0x563b99c3ed50 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:55:18.520 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.518+0000 7f1d28c90f40 1 --2- 172.21.15.32:0/3176098597 >> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] conn(0x563b99c3c930 0x563b99c3ed50 crc :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:18.520 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.518+0000 7f1d28c90f40 1 -- 172.21.15.32:0/3176098597 >> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] conn(0x563b99c45410 msgr2=0x563b99c45860 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:55:18.520 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.518+0000 7f1d28c90f40 1 --2- 172.21.15.32:0/3176098597 >> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] conn(0x563b99c45410 0x563b99c45860 crc :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:18.521 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.518+0000 7f1d28c90f40 1 -- 172.21.15.32:0/3176098597 >> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] conn(0x563b99c48720 msgr2=0x563b99c48b30 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:55:18.521 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.518+0000 7f1d22800640 1 -- 172.21.15.32:0/3176098597 reap_dead start 2024-08-03T20:55:18.521 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.518+0000 7f1d28c90f40 1 --2- 172.21.15.32:0/3176098597 >> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] conn(0x563b99c48720 0x563b99c48b30 crc :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:18.521 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.518+0000 7f1d22800640 1 -- 172.21.15.32:0/3176098597 reap_dead start 2024-08-03T20:55:18.521 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.518+0000 7f1d28c90f40 1 -- 172.21.15.32:0/3176098597 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f1d04075a20 msgr2=0x7f1d04077f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:55:18.521 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.518+0000 7f1d28c90f40 1 --2- 172.21.15.32:0/3176098597 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f1d04075a20 0x7f1d04077f30 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x563b99c264f0 tx=0x7f1d10002fb0 comp rx=0 tx=0).stop 2024-08-03T20:55:18.521 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.518+0000 7f1d28c90f40 1 -- 172.21.15.32:0/3176098597 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x563b99b78360 msgr2=0x563b99b75fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:55:18.521 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.518+0000 7f1d28c90f40 1 --2- 172.21.15.32:0/3176098597 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x563b99b78360 0x563b99b75fb0 secure :-1 s=READY pgs=1163 cs=0 l=1 rev1=1 crypto rx=0x7f1d14000c00 tx=0x7f1d140041f0 comp rx=0 tx=0).stop 2024-08-03T20:55:18.521 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.518+0000 7f1d28c90f40 1 -- 172.21.15.32:0/3176098597 shutdown_connections 2024-08-03T20:55:18.521 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.518+0000 7f1d28c90f40 1 --2- 172.21.15.32:0/3176098597 >> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] conn(0x563b99c48720 0x563b99c48b30 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:18.521 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.518+0000 7f1d28c90f40 1 --2- 172.21.15.32:0/3176098597 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f1d04075a20 0x7f1d04077f30 unknown :-1 s=CLOSED pgs=482 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:18.521 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.518+0000 7f1d28c90f40 1 --2- 172.21.15.32:0/3176098597 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x563b99b78360 0x563b99b75fb0 unknown :-1 s=CLOSED pgs=1163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:18.521 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.518+0000 7f1d28c90f40 1 -- 172.21.15.32:0/3176098597 >> 172.21.15.32:0/3176098597 conn(0x563b999e1ac0 msgr2=0x563b99b77a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:55:18.522 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.518+0000 7f1d28c90f40 1 -- 172.21.15.32:0/3176098597 shutdown_connections 2024-08-03T20:55:18.522 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.518+0000 7f1d28c90f40 1 -- 172.21.15.32:0/3176098597 wait complete. 2024-08-03T20:55:18.535 DEBUG:teuthology.orchestra.run.smithi032:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata rm 200.00000000 2024-08-03T20:55:18.718 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.717+0000 7f6280f8af40 1 Processor -- start 2024-08-03T20:55:18.718 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.717+0000 7f6280f8af40 1 -- start start 2024-08-03T20:55:18.720 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.718+0000 7f6280f8af40 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x555cd3751360 0x555cd3699140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:55:18.720 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.718+0000 7f6280f8af40 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555cd376ecb0 0x555cd3749710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:55:18.720 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.718+0000 7f6280f8af40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x555cd376f3f0 0x555cd3769a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:55:18.720 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.718+0000 7f6280f8af40 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x555cd35e8e30 con 0x555cd376ecb0 2024-08-03T20:55:18.721 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.718+0000 7f6280f8af40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x555cd3646bf0 con 0x555cd376f3f0 2024-08-03T20:55:18.721 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.718+0000 7f6280f8af40 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x555cd3752440 con 0x555cd3751360 2024-08-03T20:55:18.721 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.718+0000 7f6279800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555cd376ecb0 0x555cd3749710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:55:18.721 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.718+0000 7f6279800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555cd376ecb0 0x555cd3749710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:50622/0 (socket says 172.21.15.32:50622) 2024-08-03T20:55:18.721 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.718+0000 7f6279800640 1 -- 172.21.15.32:0/1397492319 learned_addr learned my addr 172.21.15.32:0/1397492319 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:55:18.721 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.718+0000 7f627ac00640 1 --2- 172.21.15.32:0/1397492319 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x555cd376f3f0 0x555cd3769a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:55:18.721 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.718+0000 7f627a200640 1 --2- 172.21.15.32:0/1397492319 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x555cd3751360 0x555cd3699140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:55:18.721 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.719+0000 7f6279800640 1 -- 172.21.15.32:0/1397492319 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x555cd3751360 msgr2=0x555cd3699140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:55:18.721 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.719+0000 7f6279800640 1 --2- 172.21.15.32:0/1397492319 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x555cd3751360 0x555cd3699140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:18.721 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.719+0000 7f6279800640 1 -- 172.21.15.32:0/1397492319 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x555cd376f3f0 msgr2=0x555cd3769a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:55:18.721 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.719+0000 7f6279800640 1 --2- 172.21.15.32:0/1397492319 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x555cd376f3f0 0x555cd3769a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:18.722 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.719+0000 7f6279800640 1 -- 172.21.15.32:0/1397492319 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555cd3771570 con 0x555cd376ecb0 2024-08-03T20:55:18.722 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.719+0000 7f6279800640 1 --2- 172.21.15.32:0/1397492319 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555cd376ecb0 0x555cd3749710 secure :-1 s=READY pgs=1165 cs=0 l=1 rev1=1 crypto rx=0x7f6268001e10 tx=0x7f626800b650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7a54804a1a14fac3 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:55:18.723 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.719+0000 7f6273e00640 1 -- 172.21.15.32:0/1397492319 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6268016070 con 0x555cd376ecb0 2024-08-03T20:55:18.723 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.719+0000 7f6273e00640 1 -- 172.21.15.32:0/1397492319 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f626800d7c0 con 0x555cd376ecb0 2024-08-03T20:55:18.723 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.719+0000 7f6273e00640 1 -- 172.21.15.32:0/1397492319 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f626800dc10 con 0x555cd376ecb0 2024-08-03T20:55:18.723 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.720+0000 7f6280f8af40 1 -- 172.21.15.32:0/1397492319 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555cd376ecb0 msgr2=0x555cd3749710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:55:18.724 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.720+0000 7f6280f8af40 1 --2- 172.21.15.32:0/1397492319 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555cd376ecb0 0x555cd3749710 secure :-1 s=READY pgs=1165 cs=0 l=1 rev1=1 crypto rx=0x7f6268001e10 tx=0x7f626800b650 comp rx=0 tx=0).stop 2024-08-03T20:55:18.724 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.720+0000 7f6280f8af40 1 -- 172.21.15.32:0/1397492319 shutdown_connections 2024-08-03T20:55:18.724 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.720+0000 7f6280f8af40 1 --2- 172.21.15.32:0/1397492319 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x555cd376f3f0 0x555cd3769a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:18.724 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.720+0000 7f6280f8af40 1 --2- 172.21.15.32:0/1397492319 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555cd376ecb0 0x555cd3749710 unknown :-1 s=CLOSED pgs=1165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:18.724 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.720+0000 7f6280f8af40 1 --2- 172.21.15.32:0/1397492319 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x555cd3751360 0x555cd3699140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:18.724 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.720+0000 7f6280f8af40 1 -- 172.21.15.32:0/1397492319 >> 172.21.15.32:0/1397492319 conn(0x555cd35baac0 msgr2=0x555cd36e3ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:55:18.724 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.720+0000 7f6280f8af40 1 -- 172.21.15.32:0/1397492319 shutdown_connections 2024-08-03T20:55:18.724 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.720+0000 7f6280f8af40 1 -- 172.21.15.32:0/1397492319 wait complete. 2024-08-03T20:55:18.724 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.721+0000 7f6280f8af40 1 Processor -- start 2024-08-03T20:55:18.724 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.721+0000 7f6280f8af40 1 -- start start 2024-08-03T20:55:18.726 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.723+0000 7f6280f8af40 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x555cd3751360 0x555cd3699120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:55:18.726 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.723+0000 7f6280f8af40 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555cd376ecb0 0x555cd37670b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:55:18.726 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.723+0000 7f6280f8af40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x555cd376f3f0 0x555cd3747cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:55:18.726 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.723+0000 7f6280f8af40 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x555cd3749c50 con 0x555cd376ecb0 2024-08-03T20:55:18.727 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.723+0000 7f6280f8af40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x555cd37fb6e0 con 0x555cd376f3f0 2024-08-03T20:55:18.727 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.723+0000 7f6280f8af40 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x555cd376ca30 con 0x555cd3751360 2024-08-03T20:55:18.727 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.723+0000 7f6279800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555cd376ecb0 0x555cd37670b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:55:18.727 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.723+0000 7f6279800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555cd376ecb0 0x555cd37670b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:50630/0 (socket says 172.21.15.32:50630) 2024-08-03T20:55:18.727 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.723+0000 7f6279800640 1 -- 172.21.15.32:0/2593030325 learned_addr learned my addr 172.21.15.32:0/2593030325 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:55:18.727 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.723+0000 7f627a200640 1 --2- 172.21.15.32:0/2593030325 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x555cd3751360 0x555cd3699120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:55:18.727 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.723+0000 7f627ac00640 1 --2- 172.21.15.32:0/2593030325 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x555cd376f3f0 0x555cd3747cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:55:18.727 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.724+0000 7f6279800640 1 -- 172.21.15.32:0/2593030325 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x555cd3751360 msgr2=0x555cd3699120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:55:18.727 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.724+0000 7f6279800640 1 --2- 172.21.15.32:0/2593030325 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x555cd3751360 0x555cd3699120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:18.727 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.724+0000 7f6279800640 1 -- 172.21.15.32:0/2593030325 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x555cd376f3f0 msgr2=0x555cd3747cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:55:18.727 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.724+0000 7f6279800640 1 --2- 172.21.15.32:0/2593030325 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x555cd376f3f0 0x555cd3747cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:18.727 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.724+0000 7f6279800640 1 -- 172.21.15.32:0/2593030325 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555cd376ae10 con 0x555cd376ecb0 2024-08-03T20:55:18.727 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.724+0000 7f627ac00640 1 --2- 172.21.15.32:0/2593030325 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x555cd376f3f0 0x555cd3747cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-03T20:55:18.728 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.724+0000 7f627a200640 1 --2- 172.21.15.32:0/2593030325 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x555cd3751360 0x555cd3699120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-03T20:55:18.728 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.724+0000 7f6279800640 1 --2- 172.21.15.32:0/2593030325 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555cd376ecb0 0x555cd37670b0 secure :-1 s=READY pgs=1166 cs=0 l=1 rev1=1 crypto rx=0x7f6268014040 tx=0x7f626800dec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:55:18.728 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.724+0000 7f6272a00640 1 -- 172.21.15.32:0/2593030325 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6268022070 con 0x555cd376ecb0 2024-08-03T20:55:18.728 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.724+0000 7f6272a00640 1 -- 172.21.15.32:0/2593030325 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f6268017040 con 0x555cd376ecb0 2024-08-03T20:55:18.728 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.724+0000 7f6272a00640 1 -- 172.21.15.32:0/2593030325 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6268016070 con 0x555cd376ecb0 2024-08-03T20:55:18.728 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.724+0000 7f6280f8af40 1 -- 172.21.15.32:0/2593030325 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x555cd3750250 con 0x555cd376ecb0 2024-08-03T20:55:18.728 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.724+0000 7f6280f8af40 1 -- 172.21.15.32:0/2593030325 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555cd3743290 con 0x555cd376ecb0 2024-08-03T20:55:18.729 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.726+0000 7f6272a00640 1 -- 172.21.15.32:0/2593030325 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f6268010440 con 0x555cd376ecb0 2024-08-03T20:55:18.730 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.727+0000 7f6272a00640 1 --2- 172.21.15.32:0/2593030325 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f625c075a20 0x7f625c077f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:55:18.730 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.727+0000 7f6272a00640 1 -- 172.21.15.32:0/2593030325 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 6053+0+0 (secure 0 0 0) 0x7f6268098140 con 0x555cd376ecb0 2024-08-03T20:55:18.730 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.728+0000 7f627a200640 1 --2- 172.21.15.32:0/2593030325 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f625c075a20 0x7f625c077f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:55:18.730 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.728+0000 7f6280f8af40 1 --2- 172.21.15.32:0/2593030325 >> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] conn(0x555cd36dde60 0x555cd36de270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:55:18.730 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.728+0000 7f6280f8af40 1 -- 172.21.15.32:0/2593030325 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:1 12.14 12:292cf221:::200.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e87) -- 0x555cd36de7b0 con 0x555cd36dde60 2024-08-03T20:55:18.730 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.728+0000 7f627ac00640 1 --2- 172.21.15.32:0/2593030325 >> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] conn(0x555cd36dde60 0x555cd36de270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:55:18.731 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.729+0000 7f627a200640 1 --2- 172.21.15.32:0/2593030325 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f625c075a20 0x7f625c077f30 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7f626c004fd0 tx=0x7f626c00f040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:55:18.731 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.729+0000 7f627ac00640 1 --2- 172.21.15.32:0/2593030325 >> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] conn(0x555cd36dde60 0x555cd36de270 crc :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:55:18.733 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.731+0000 7f627ac00640 1 -- 172.21.15.32:0/2593030325 <== osd.5 v2:172.21.15.118:6810/2724891770 1 ==== osd_op_reply(1 200.00000000 [delete] v87'108 uv108 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f627400b040 con 0x555cd36dde60 2024-08-03T20:55:18.733 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.731+0000 7f6280f8af40 1 -- 172.21.15.32:0/2593030325 >> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] conn(0x555cd36dde60 msgr2=0x555cd36de270 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:55:18.733 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.731+0000 7f6280f8af40 1 --2- 172.21.15.32:0/2593030325 >> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] conn(0x555cd36dde60 0x555cd36de270 crc :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:18.733 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.731+0000 7f6280f8af40 1 -- 172.21.15.32:0/2593030325 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f625c075a20 msgr2=0x7f625c077f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:55:18.733 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.731+0000 7f6280f8af40 1 --2- 172.21.15.32:0/2593030325 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f625c075a20 0x7f625c077f30 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7f626c004fd0 tx=0x7f626c00f040 comp rx=0 tx=0).stop 2024-08-03T20:55:18.733 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.731+0000 7f6280f8af40 1 -- 172.21.15.32:0/2593030325 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555cd376ecb0 msgr2=0x555cd37670b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:55:18.733 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.731+0000 7f6280f8af40 1 --2- 172.21.15.32:0/2593030325 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555cd376ecb0 0x555cd37670b0 secure :-1 s=READY pgs=1166 cs=0 l=1 rev1=1 crypto rx=0x7f6268014040 tx=0x7f626800dec0 comp rx=0 tx=0).stop 2024-08-03T20:55:18.733 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.731+0000 7f627ac00640 1 -- 172.21.15.32:0/2593030325 reap_dead start 2024-08-03T20:55:18.733 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.731+0000 7f6280f8af40 1 -- 172.21.15.32:0/2593030325 shutdown_connections 2024-08-03T20:55:18.733 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.731+0000 7f6280f8af40 1 -- 172.21.15.32:0/2593030325 >> 172.21.15.32:0/2593030325 conn(0x555cd35baac0 msgr2=0x555cd376bc40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:55:18.733 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.731+0000 7f6280f8af40 1 -- 172.21.15.32:0/2593030325 shutdown_connections 2024-08-03T20:55:18.733 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.731+0000 7f6280f8af40 1 -- 172.21.15.32:0/2593030325 wait complete. 2024-08-03T20:55:18.747 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs joinable true 2024-08-03T20:55:18.873 INFO:tasks.ceph.mds.a.smithi032.stderr: -40> 2024-08-03T20:55:14.117+0000 7f6ee1400640 -1 mds.0.cache.ino(0x1) No data while reading inode 0x1 2024-08-03T20:55:18.873 INFO:tasks.ceph.mds.a.smithi032.stderr: -38> 2024-08-03T20:55:14.117+0000 7f6ee1400640 -1 log_channel(cluster) log [ERR] : Error loading MDS rank 0: (2) No such file or directory 2024-08-03T20:55:18.983 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.981+0000 7f4a94000640 1 -- 172.21.15.32:0/2522049815 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4a8c160520 msgr2=0x7f4a8c1450f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:55:18.984 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.981+0000 7f4a94000640 1 --2- 172.21.15.32:0/2522049815 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4a8c160520 0x7f4a8c1450f0 secure :-1 s=READY pgs=1167 cs=0 l=1 rev1=1 crypto rx=0x7f4a8c15da00 tx=0x7f4a8402a9a0 comp rx=0 tx=0).stop 2024-08-03T20:55:18.984 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.982+0000 7f4a94000640 1 -- 172.21.15.32:0/2522049815 shutdown_connections 2024-08-03T20:55:18.984 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.982+0000 7f4a94000640 1 --2- 172.21.15.32:0/2522049815 >> v2:172.21.15.118:3300/0 conn(0x7f4a8c0cc850 0x7f4a8c15ffe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:18.984 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.982+0000 7f4a94000640 1 --2- 172.21.15.32:0/2522049815 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4a8c160520 0x7f4a8c1450f0 unknown :-1 s=CLOSED pgs=1167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:18.984 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.982+0000 7f4a94000640 1 -- 172.21.15.32:0/2522049815 >> 172.21.15.32:0/2522049815 conn(0x7f4a8c0b1bd0 msgr2=0x7f4a8c0b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:55:18.984 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.982+0000 7f4a94000640 1 -- 172.21.15.32:0/2522049815 shutdown_connections 2024-08-03T20:55:18.984 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.982+0000 7f4a94000640 1 -- 172.21.15.32:0/2522049815 wait complete. 2024-08-03T20:55:18.985 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.984+0000 7f4a94000640 1 Processor -- start 2024-08-03T20:55:18.985 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.984+0000 7f4a94000640 1 -- start start 2024-08-03T20:55:18.988 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.986+0000 7f4a94000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4a8c0cc850 0x7f4a8c0c5e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:55:18.988 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.986+0000 7f4a94000640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f4a8c0cf8d0 0x7f4a8c0cad70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:55:18.988 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.986+0000 7f4a94000640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f4a8c160520 0x7f4a8c0cb2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:55:18.988 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.986+0000 7f4a94000640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f4a8c1459f0 con 0x7f4a8c0cc850 2024-08-03T20:55:18.988 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.986+0000 7f4a94000640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f4a8c145870 con 0x7f4a8c0cf8d0 2024-08-03T20:55:18.988 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.986+0000 7f4a94000640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7f4a8c145b70 con 0x7f4a8c160520 2024-08-03T20:55:18.989 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.986+0000 7f4a91600640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4a8c0cc850 0x7f4a8c0c5e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:55:18.989 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.986+0000 7f4a91600640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4a8c0cc850 0x7f4a8c0c5e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:50648/0 (socket says 172.21.15.32:50648) 2024-08-03T20:55:18.989 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.986+0000 7f4a91600640 1 -- 172.21.15.32:0/1769637778 learned_addr learned my addr 172.21.15.32:0/1769637778 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:55:18.989 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.986+0000 7f4a92000640 1 --2- 172.21.15.32:0/1769637778 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f4a8c160520 0x7f4a8c0cb2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:55:18.989 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.986+0000 7f4a90c00640 1 --2- 172.21.15.32:0/1769637778 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f4a8c0cf8d0 0x7f4a8c0cad70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:55:18.989 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.987+0000 7f4a91600640 1 -- 172.21.15.32:0/1769637778 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f4a8c160520 msgr2=0x7f4a8c0cb2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:55:18.989 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.987+0000 7f4a91600640 1 --2- 172.21.15.32:0/1769637778 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f4a8c160520 0x7f4a8c0cb2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:18.989 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.987+0000 7f4a91600640 1 -- 172.21.15.32:0/1769637778 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f4a8c0cf8d0 msgr2=0x7f4a8c0cad70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:55:18.989 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.987+0000 7f4a91600640 1 --2- 172.21.15.32:0/1769637778 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f4a8c0cf8d0 0x7f4a8c0cad70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:18.989 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.987+0000 7f4a91600640 1 -- 172.21.15.32:0/1769637778 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4a8c143c70 con 0x7f4a8c0cc850 2024-08-03T20:55:18.990 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.987+0000 7f4a92000640 1 --2- 172.21.15.32:0/1769637778 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f4a8c160520 0x7f4a8c0cb2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-03T20:55:18.990 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.987+0000 7f4a90c00640 1 --2- 172.21.15.32:0/1769637778 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f4a8c0cf8d0 0x7f4a8c0cad70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-03T20:55:18.990 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.987+0000 7f4a91600640 1 --2- 172.21.15.32:0/1769637778 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4a8c0cc850 0x7f4a8c0c5e90 secure :-1 s=READY pgs=1169 cs=0 l=1 rev1=1 crypto rx=0x7f4a8000c3a0 tx=0x7f4a8000c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:55:18.990 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.987+0000 7f4a89600640 1 -- 172.21.15.32:0/1769637778 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a80017070 con 0x7f4a8c0cc850 2024-08-03T20:55:18.991 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.987+0000 7f4a89600640 1 -- 172.21.15.32:0/1769637778 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f4a800053c0 con 0x7f4a8c0cc850 2024-08-03T20:55:18.991 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.987+0000 7f4a89600640 1 -- 172.21.15.32:0/1769637778 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a80003850 con 0x7f4a8c0cc850 2024-08-03T20:55:18.991 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.987+0000 7f4a94000640 1 -- 172.21.15.32:0/1769637778 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4a8c0c6a70 con 0x7f4a8c0cc850 2024-08-03T20:55:18.991 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.987+0000 7f4a94000640 1 -- 172.21.15.32:0/1769637778 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4a8c0c7030 con 0x7f4a8c0cc850 2024-08-03T20:55:18.991 INFO:tasks.ceph.mds.a.smithi032.stdout:starting mds.a at 2024-08-03T20:55:18.992 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.989+0000 7f4a94000640 1 -- 172.21.15.32:0/1769637778 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4a8c15ffe0 con 0x7f4a8c0cc850 2024-08-03T20:55:18.992 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.990+0000 7f4a89600640 1 -- 172.21.15.32:0/1769637778 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f4a8001a070 con 0x7f4a8c0cc850 2024-08-03T20:55:18.993 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.992+0000 7f4a89600640 1 --2- 172.21.15.32:0/1769637778 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f4a6c077040 0x7f4a6c079550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:55:18.993 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.992+0000 7f4a89600640 1 -- 172.21.15.32:0/1769637778 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 6053+0+0 (secure 0 0 0) 0x7f4a80097eb0 con 0x7f4a8c0cc850 2024-08-03T20:55:18.993 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.992+0000 7f4a90c00640 1 --2- 172.21.15.32:0/1769637778 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f4a6c077040 0x7f4a6c079550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:55:18.994 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.993+0000 7f4a90c00640 1 --2- 172.21.15.32:0/1769637778 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f4a6c077040 0x7f4a6c079550 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f4a8c144090 tx=0x7f4a78005f90 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:55:18.998 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:18.997+0000 7f4a89600640 1 -- 172.21.15.32:0/1769637778 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7f4a8c143c70 con 0x7f4a8c0cc850 2024-08-03T20:55:19.246 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:19.244+0000 7f4a94000640 1 -- 172.21.15.32:0/1769637778 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "joinable", "val": "true"} v 0) -- 0x7f4a8c1450f0 con 0x7f4a8c0cc850 2024-08-03T20:55:20.011 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:20.009+0000 7f4a89600640 1 -- 172.21.15.32:0/1769637778 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "joinable", "val": "true"}]=0 cephfs marked joinable; MDS may join as newly active. v116) ==== 162+0+0 (secure 0 0 0) 0x7f4a8c160930 con 0x7f4a8c0cc850 2024-08-03T20:55:20.011 INFO:teuthology.orchestra.run.smithi032.stderr:cephfs marked joinable; MDS may join as newly active. 2024-08-03T20:55:20.014 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:20.012+0000 7f4a94000640 1 -- 172.21.15.32:0/1769637778 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f4a6c077040 msgr2=0x7f4a6c079550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:55:20.014 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:20.012+0000 7f4a94000640 1 --2- 172.21.15.32:0/1769637778 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f4a6c077040 0x7f4a6c079550 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f4a8c144090 tx=0x7f4a78005f90 comp rx=0 tx=0).stop 2024-08-03T20:55:20.015 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:20.013+0000 7f4a94000640 1 -- 172.21.15.32:0/1769637778 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4a8c0cc850 msgr2=0x7f4a8c0c5e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:55:20.015 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:20.013+0000 7f4a94000640 1 --2- 172.21.15.32:0/1769637778 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4a8c0cc850 0x7f4a8c0c5e90 secure :-1 s=READY pgs=1169 cs=0 l=1 rev1=1 crypto rx=0x7f4a8000c3a0 tx=0x7f4a8000c860 comp rx=0 tx=0).stop 2024-08-03T20:55:20.015 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:20.013+0000 7f4a94000640 1 -- 172.21.15.32:0/1769637778 shutdown_connections 2024-08-03T20:55:20.015 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:20.013+0000 7f4a94000640 1 --2- 172.21.15.32:0/1769637778 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f4a6c077040 0x7f4a6c079550 unknown :-1 s=CLOSED pgs=484 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:20.015 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:20.013+0000 7f4a94000640 1 --2- 172.21.15.32:0/1769637778 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f4a8c0cf8d0 0x7f4a8c0cad70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:20.015 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:20.013+0000 7f4a94000640 1 --2- 172.21.15.32:0/1769637778 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f4a8c160520 0x7f4a8c0cb2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:20.015 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:20.013+0000 7f4a94000640 1 --2- 172.21.15.32:0/1769637778 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4a8c0cc850 0x7f4a8c0c5e90 unknown :-1 s=CLOSED pgs=1169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:20.015 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:20.013+0000 7f4a94000640 1 -- 172.21.15.32:0/1769637778 >> 172.21.15.32:0/1769637778 conn(0x7f4a8c0b1bd0 msgr2=0x7f4a8c0b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:55:20.015 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:20.013+0000 7f4a94000640 1 -- 172.21.15.32:0/1769637778 shutdown_connections 2024-08-03T20:55:20.016 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:20.013+0000 7f4a94000640 1 -- 172.21.15.32:0/1769637778 wait complete. 2024-08-03T20:55:20.021 INFO:tasks.ceph.mds.a.smithi032.stderr:2024-08-03T20:55:20.019+0000 7f9932000640 -1 log_channel(cluster) log [ERR] : Error recovering journal 0x200: (2) No such file or directory 2024-08-03T20:55:20.021 INFO:tasks.ceph.mon.a.smithi032.stderr:2024-08-03T20:55:20.020+0000 7f4e6d600640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-08-03T20:55:20.021 INFO:tasks.ceph.mon.a.smithi032.stderr:2024-08-03T20:55:20.020+0000 7f4e6d600640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 mds daemon damaged (MDS_DAMAGE) 2024-08-03T20:55:20.043 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-08-03T20:55:20.288 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:20.283+0000 7f3897e00640 1 -- 172.21.15.32:0/332416519 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f38980cee30 msgr2=0x7f38980b15e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:55:20.288 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:20.283+0000 7f3897e00640 1 --2- 172.21.15.32:0/332416519 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f38980cee30 0x7f38980b15e0 secure :-1 s=READY pgs=1172 cs=0 l=1 rev1=1 crypto rx=0x7f389806a580 tx=0x7f389002a820 comp rx=0 tx=0).stop 2024-08-03T20:55:20.288 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:20.283+0000 7f3897e00640 1 -- 172.21.15.32:0/332416519 shutdown_connections 2024-08-03T20:55:20.288 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:20.283+0000 7f3897e00640 1 --2- 172.21.15.32:0/332416519 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f38980cee30 0x7f38980b15e0 unknown :-1 s=CLOSED pgs=1172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:20.288 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:20.283+0000 7f3897e00640 1 --2- 172.21.15.32:0/332416519 >> v2:172.21.15.118:3300/0 conn(0x7f38980b7180 0x7f38980d2970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:20.288 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:20.283+0000 7f3897e00640 1 -- 172.21.15.32:0/332416519 >> 172.21.15.32:0/332416519 conn(0x7f38980d1e50 msgr2=0x7f38980d2260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:55:20.288 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:20.283+0000 7f3897e00640 1 -- 172.21.15.32:0/332416519 shutdown_connections 2024-08-03T20:55:20.288 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:20.283+0000 7f3897e00640 1 -- 172.21.15.32:0/332416519 wait complete. 2024-08-03T20:55:20.289 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:20.285+0000 7f3897e00640 1 Processor -- start 2024-08-03T20:55:20.289 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:20.285+0000 7f3897e00640 1 -- start start 2024-08-03T20:55:20.291 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:20.287+0000 7f3897e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f38980b3380 0x7f38981ec600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:55:20.291 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:20.287+0000 7f3897e00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f38980b7180 0x7f389809f2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:55:20.291 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:20.287+0000 7f3897e00640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f38981f58a0 0x7f389809f830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:55:20.291 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:20.287+0000 7f3897e00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f38980d0560 con 0x7f38980b3380 2024-08-03T20:55:20.291 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:20.287+0000 7f3897e00640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f38980d03e0 con 0x7f38980b7180 2024-08-03T20:55:20.291 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:20.287+0000 7f3897e00640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7f38980d06e0 con 0x7f38981f58a0 2024-08-03T20:55:20.292 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:20.287+0000 7f3896a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f38980b3380 0x7f38981ec600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:55:20.292 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:20.287+0000 7f3896a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f38980b3380 0x7f38981ec600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:53232/0 (socket says 172.21.15.32:53232) 2024-08-03T20:55:20.292 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:20.287+0000 7f3896a00640 1 -- 172.21.15.32:0/3072481616 learned_addr learned my addr 172.21.15.32:0/3072481616 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:55:20.292 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:20.287+0000 7f388fe00640 1 --2- 172.21.15.32:0/3072481616 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f38980b7180 0x7f389809f2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:55:20.292 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:20.287+0000 7f3896a00640 1 -- 172.21.15.32:0/3072481616 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f38981f58a0 msgr2=0x7f389809f830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:55:20.293 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:20.287+0000 7f3896a00640 1 --2- 172.21.15.32:0/3072481616 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f38981f58a0 0x7f389809f830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:20.293 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:20.287+0000 7f3896a00640 1 -- 172.21.15.32:0/3072481616 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f38980b7180 msgr2=0x7f389809f2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:55:20.293 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:20.287+0000 7f3896a00640 1 --2- 172.21.15.32:0/3072481616 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f38980b7180 0x7f389809f2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:20.293 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:20.287+0000 7f3896a00640 1 -- 172.21.15.32:0/3072481616 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f38980b0590 con 0x7f38980b3380 2024-08-03T20:55:20.294 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:20.287+0000 7f388fe00640 1 --2- 172.21.15.32:0/3072481616 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f38980b7180 0x7f389809f2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-03T20:55:20.294 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:20.288+0000 7f3896a00640 1 --2- 172.21.15.32:0/3072481616 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f38980b3380 0x7f38981ec600 secure :-1 s=READY pgs=1173 cs=0 l=1 rev1=1 crypto rx=0x7f388800d700 tx=0x7f388800dbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:55:20.295 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:20.288+0000 7f388ea00640 1 -- 172.21.15.32:0/3072481616 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3888017070 con 0x7f38980b3380 2024-08-03T20:55:20.295 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:20.288+0000 7f388ea00640 1 -- 172.21.15.32:0/3072481616 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f3888009cc0 con 0x7f38980b3380 2024-08-03T20:55:20.295 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:20.288+0000 7f388ea00640 1 -- 172.21.15.32:0/3072481616 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38880106c0 con 0x7f38980b3380 2024-08-03T20:55:20.296 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:20.288+0000 7f3897e00640 1 -- 172.21.15.32:0/3072481616 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f38980b0870 con 0x7f38980b3380 2024-08-03T20:55:20.296 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:20.288+0000 7f3897e00640 1 -- 172.21.15.32:0/3072481616 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f38980b0e30 con 0x7f38980b3380 2024-08-03T20:55:20.296 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:20.289+0000 7f3897e00640 1 -- 172.21.15.32:0/3072481616 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f38980d0b90 con 0x7f38980b3380 2024-08-03T20:55:20.301 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:20.290+0000 7f388ea00640 1 -- 172.21.15.32:0/3072481616 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f388801a070 con 0x7f38980b3380 2024-08-03T20:55:20.301 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:20.291+0000 7f388ea00640 1 --2- 172.21.15.32:0/3072481616 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f3874077010 0x7f3874079520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:55:20.302 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:20.292+0000 7f388ea00640 1 -- 172.21.15.32:0/3072481616 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 6139+0+0 (secure 0 0 0) 0x7f3888098c30 con 0x7f38980b3380 2024-08-03T20:55:20.302 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:20.292+0000 7f388fe00640 1 --2- 172.21.15.32:0/3072481616 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f3874077010 0x7f3874079520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:55:20.302 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:20.293+0000 7f388fe00640 1 --2- 172.21.15.32:0/3072481616 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f3874077010 0x7f3874079520 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7f38981f6980 tx=0x7f3880009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:55:20.302 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:20.296+0000 7f388ea00640 1 -- 172.21.15.32:0/3072481616 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7f3888061c70 con 0x7f38980b3380 2024-08-03T20:55:20.540 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:20.538+0000 7f3897e00640 1 -- 172.21.15.32:0/3072481616 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f38981f8e90 con 0x7f38980b3380 2024-08-03T20:55:20.540 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:20.539+0000 7f388ea00640 1 -- 172.21.15.32:0/3072481616 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 117 v117) ==== 95+0+4390 (secure 0 0 0) 0x7f3888061570 con 0x7f38980b3380 2024-08-03T20:55:20.541 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-03T20:55:20.541 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":117,"btime":"2024-08-03T20:55:20:021609+0000","default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6439,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6835/531080160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6834","nonce":531080160},{"type":"v1","addr":"172.21.15.118:6835","nonce":531080160}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":84},{"gid":6454,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6837/2796882319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6836","nonce":2796882319},{"type":"v1","addr":"172.21.15.118:6837","nonce":2796882319}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":86},{"gid":6813,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6834/2686933890","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":2686933890},{"type":"v1","addr":"172.21.15.32:6834","nonce":2686933890}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":85}],"filesystems":[{"mdsmap":{"epoch":117,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-08-03T20:54:26.950518+0000","modified":"2024-08-03T20:55:20.021534+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":88,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":6}]} 2024-08-03T20:55:20.541 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 117 2024-08-03T20:55:20.543 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:20.541+0000 7f3897e00640 1 -- 172.21.15.32:0/3072481616 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f3874077010 msgr2=0x7f3874079520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:55:20.543 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:20.541+0000 7f3897e00640 1 --2- 172.21.15.32:0/3072481616 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f3874077010 0x7f3874079520 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7f38981f6980 tx=0x7f3880009040 comp rx=0 tx=0).stop 2024-08-03T20:55:20.543 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:20.541+0000 7f3897e00640 1 -- 172.21.15.32:0/3072481616 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f38980b3380 msgr2=0x7f38981ec600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:55:20.543 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:20.541+0000 7f3897e00640 1 --2- 172.21.15.32:0/3072481616 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f38980b3380 0x7f38981ec600 secure :-1 s=READY pgs=1173 cs=0 l=1 rev1=1 crypto rx=0x7f388800d700 tx=0x7f388800dbc0 comp rx=0 tx=0).stop 2024-08-03T20:55:20.543 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:20.541+0000 7f3897e00640 1 -- 172.21.15.32:0/3072481616 shutdown_connections 2024-08-03T20:55:20.543 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:20.541+0000 7f3897e00640 1 --2- 172.21.15.32:0/3072481616 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f3874077010 0x7f3874079520 unknown :-1 s=CLOSED pgs=486 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:20.543 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:20.541+0000 7f3897e00640 1 --2- 172.21.15.32:0/3072481616 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f38980b7180 0x7f389809f2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:20.544 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:20.541+0000 7f3897e00640 1 --2- 172.21.15.32:0/3072481616 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f38981f58a0 0x7f389809f830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:20.544 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:20.541+0000 7f3897e00640 1 --2- 172.21.15.32:0/3072481616 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f38980b3380 0x7f38981ec600 unknown :-1 s=CLOSED pgs=1173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:20.544 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:20.541+0000 7f3897e00640 1 -- 172.21.15.32:0/3072481616 >> 172.21.15.32:0/3072481616 conn(0x7f38980d1e50 msgr2=0x7f3898160520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:55:20.544 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:20.542+0000 7f3897e00640 1 -- 172.21.15.32:0/3072481616 shutdown_connections 2024-08-03T20:55:20.544 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:20.542+0000 7f3897e00640 1 -- 172.21.15.32:0/3072481616 wait complete. 2024-08-03T20:55:20.564 DEBUG:tasks.ceph_test_case:wait_until_true: success in 0s and 0 retries 2024-08-03T20:55:20.565 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-08-03T20:55:20.793 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:20.791+0000 7f1defe00640 1 -- 172.21.15.32:0/622669916 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1dd0004800 msgr2=0x7f1dd0004c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:55:20.793 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:20.791+0000 7f1defe00640 1 --2- 172.21.15.32:0/622669916 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1dd0004800 0x7f1dd0004c10 secure :-1 s=READY pgs=1175 cs=0 l=1 rev1=1 crypto rx=0x7f1dd4009b50 tx=0x7f1dd0005830 comp rx=0 tx=0).stop 2024-08-03T20:55:20.793 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:20.791+0000 7f1dede00640 1 -- 172.21.15.32:0/622669916 reap_dead start 2024-08-03T20:55:20.793 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:20.791+0000 7f1defe00640 1 -- 172.21.15.32:0/622669916 shutdown_connections 2024-08-03T20:55:20.793 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:20.791+0000 7f1defe00640 1 -- 172.21.15.32:0/622669916 >> 172.21.15.32:0/622669916 conn(0x7f1de80b1bd0 msgr2=0x7f1de80b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:55:20.793 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:20.791+0000 7f1defe00640 1 -- 172.21.15.32:0/622669916 shutdown_connections 2024-08-03T20:55:20.793 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:20.791+0000 7f1defe00640 1 -- 172.21.15.32:0/622669916 wait complete. 2024-08-03T20:55:20.795 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:20.793+0000 7f1defe00640 1 Processor -- start 2024-08-03T20:55:20.795 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:20.794+0000 7f1defe00640 1 -- start start 2024-08-03T20:55:20.797 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:20.795+0000 7f1defe00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1dd0004800 0x7f1de81f55a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:55:20.797 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:20.795+0000 7f1defe00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f1de815ca00 0x7f1de8141df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:55:20.797 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:20.795+0000 7f1defe00640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f1de815f040 0x7f1de815ab60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:55:20.798 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:20.795+0000 7f1defe00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f1de80cf9c0 con 0x7f1dd0004800 2024-08-03T20:55:20.798 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:20.795+0000 7f1defe00640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f1de80cf840 con 0x7f1de815ca00 2024-08-03T20:55:20.798 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:20.795+0000 7f1defe00640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7f1de8074270 con 0x7f1de815f040 2024-08-03T20:55:20.798 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:20.795+0000 7f1ded400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1dd0004800 0x7f1de81f55a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:55:20.798 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:20.795+0000 7f1ded400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1dd0004800 0x7f1de81f55a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:53250/0 (socket says 172.21.15.32:53250) 2024-08-03T20:55:20.798 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:20.795+0000 7f1ded400640 1 -- 172.21.15.32:0/3698627559 learned_addr learned my addr 172.21.15.32:0/3698627559 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:55:20.798 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:20.795+0000 7f1deca00640 1 --2- 172.21.15.32:0/3698627559 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f1de815ca00 0x7f1de8141df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:55:20.798 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:20.796+0000 7f1dede00640 1 --2- 172.21.15.32:0/3698627559 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f1de815f040 0x7f1de815ab60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:55:20.798 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:20.796+0000 7f1ded400640 1 -- 172.21.15.32:0/3698627559 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f1de815f040 msgr2=0x7f1de815ab60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:55:20.798 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:20.796+0000 7f1ded400640 1 --2- 172.21.15.32:0/3698627559 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f1de815f040 0x7f1de815ab60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:20.798 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:20.796+0000 7f1ded400640 1 -- 172.21.15.32:0/3698627559 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f1de815ca00 msgr2=0x7f1de8141df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:55:20.798 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:20.796+0000 7f1ded400640 1 --2- 172.21.15.32:0/3698627559 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f1de815ca00 0x7f1de8141df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:20.798 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:20.796+0000 7f1ded400640 1 -- 172.21.15.32:0/3698627559 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1de81f6540 con 0x7f1dd0004800 2024-08-03T20:55:20.799 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:20.796+0000 7f1ded400640 1 --2- 172.21.15.32:0/3698627559 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1dd0004800 0x7f1de81f55a0 secure :-1 s=READY pgs=1176 cs=0 l=1 rev1=1 crypto rx=0x7f1ddc011ae0 tx=0x7f1ddc009f10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:55:20.799 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:20.796+0000 7f1de5600640 1 -- 172.21.15.32:0/3698627559 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ddc00fc30 con 0x7f1dd0004800 2024-08-03T20:55:20.799 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:20.796+0000 7f1de5600640 1 -- 172.21.15.32:0/3698627559 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f1ddc00fdd0 con 0x7f1dd0004800 2024-08-03T20:55:20.800 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:20.796+0000 7f1defe00640 1 -- 172.21.15.32:0/3698627559 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1de81f5f80 con 0x7f1dd0004800 2024-08-03T20:55:20.800 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:20.796+0000 7f1de5600640 1 -- 172.21.15.32:0/3698627559 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ddc004280 con 0x7f1dd0004800 2024-08-03T20:55:20.800 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:20.797+0000 7f1defe00640 1 -- 172.21.15.32:0/3698627559 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1de8141610 con 0x7f1dd0004800 2024-08-03T20:55:20.800 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:20.798+0000 7f1defe00640 1 -- 172.21.15.32:0/3698627559 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1de8144f90 con 0x7f1dd0004800 2024-08-03T20:55:20.800 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:20.799+0000 7f1de5600640 1 -- 172.21.15.32:0/3698627559 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f1ddc0035f0 con 0x7f1dd0004800 2024-08-03T20:55:20.801 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:20.800+0000 7f1de5600640 1 --2- 172.21.15.32:0/3698627559 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f1dc8077010 0x7f1dc8079520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:55:20.802 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:20.801+0000 7f1de5600640 1 -- 172.21.15.32:0/3698627559 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 6139+0+0 (secure 0 0 0) 0x7f1ddc099470 con 0x7f1dd0004800 2024-08-03T20:55:20.802 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:20.801+0000 7f1deca00640 1 --2- 172.21.15.32:0/3698627559 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f1dc8077010 0x7f1dc8079520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:55:20.803 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:20.801+0000 7f1deca00640 1 --2- 172.21.15.32:0/3698627559 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f1dc8077010 0x7f1dc8079520 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7f1dd4009b50 tx=0x7f1dd402d040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:55:20.806 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:20.805+0000 7f1de5600640 1 -- 172.21.15.32:0/3698627559 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7f1ddc061e60 con 0x7f1dd0004800 2024-08-03T20:55:21.049 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:21.048+0000 7f1defe00640 1 -- 172.21.15.32:0/3698627559 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f1de8070020 con 0x7f1dd0004800 2024-08-03T20:55:21.050 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:21.048+0000 7f1de5600640 1 -- 172.21.15.32:0/3698627559 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 117 v117) ==== 95+0+4390 (secure 0 0 0) 0x7f1ddc0663b0 con 0x7f1dd0004800 2024-08-03T20:55:21.050 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-03T20:55:21.050 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":117,"btime":"2024-08-03T20:55:20:021609+0000","default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6439,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6835/531080160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6834","nonce":531080160},{"type":"v1","addr":"172.21.15.118:6835","nonce":531080160}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":84},{"gid":6454,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6837/2796882319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6836","nonce":2796882319},{"type":"v1","addr":"172.21.15.118:6837","nonce":2796882319}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":86},{"gid":6813,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6834/2686933890","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":2686933890},{"type":"v1","addr":"172.21.15.32:6834","nonce":2686933890}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":85}],"filesystems":[{"mdsmap":{"epoch":117,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-08-03T20:54:26.950518+0000","modified":"2024-08-03T20:55:20.021534+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":88,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":6}]} 2024-08-03T20:55:21.050 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 117 2024-08-03T20:55:21.052 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:21.050+0000 7f1defe00640 1 -- 172.21.15.32:0/3698627559 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f1dc8077010 msgr2=0x7f1dc8079520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:55:21.052 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:21.050+0000 7f1defe00640 1 --2- 172.21.15.32:0/3698627559 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f1dc8077010 0x7f1dc8079520 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7f1dd4009b50 tx=0x7f1dd402d040 comp rx=0 tx=0).stop 2024-08-03T20:55:21.052 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:21.050+0000 7f1defe00640 1 -- 172.21.15.32:0/3698627559 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1dd0004800 msgr2=0x7f1de81f55a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:55:21.052 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:21.050+0000 7f1defe00640 1 --2- 172.21.15.32:0/3698627559 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1dd0004800 0x7f1de81f55a0 secure :-1 s=READY pgs=1176 cs=0 l=1 rev1=1 crypto rx=0x7f1ddc011ae0 tx=0x7f1ddc009f10 comp rx=0 tx=0).stop 2024-08-03T20:55:21.052 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:21.050+0000 7f1defe00640 1 -- 172.21.15.32:0/3698627559 shutdown_connections 2024-08-03T20:55:21.052 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:21.050+0000 7f1defe00640 1 --2- 172.21.15.32:0/3698627559 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f1dc8077010 0x7f1dc8079520 unknown :-1 s=CLOSED pgs=487 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:21.053 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:21.050+0000 7f1defe00640 1 --2- 172.21.15.32:0/3698627559 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f1de815ca00 0x7f1de8141df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:21.053 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:21.050+0000 7f1defe00640 1 --2- 172.21.15.32:0/3698627559 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f1de815f040 0x7f1de815ab60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:21.053 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:21.050+0000 7f1defe00640 1 --2- 172.21.15.32:0/3698627559 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1dd0004800 0x7f1de81f55a0 unknown :-1 s=CLOSED pgs=1176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:21.053 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:21.051+0000 7f1defe00640 1 -- 172.21.15.32:0/3698627559 >> 172.21.15.32:0/3698627559 conn(0x7f1de80b1bd0 msgr2=0x7f1de80b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:55:21.053 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:21.051+0000 7f1defe00640 1 -- 172.21.15.32:0/3698627559 shutdown_connections 2024-08-03T20:55:21.053 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:21.051+0000 7f1defe00640 1 -- 172.21.15.32:0/3698627559 wait complete. 2024-08-03T20:55:21.075 DEBUG:tasks.ceph_test_case:wait_until_true: success in 0s and 0 retries 2024-08-03T20:55:21.075 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-08-03T20:55:21.310 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:21.308+0000 7f3d9a600640 1 -- 172.21.15.32:0/254119698 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3d78004d40 msgr2=0x7f3d780051f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:55:21.310 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:21.308+0000 7f3d9a600640 1 --2- 172.21.15.32:0/254119698 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3d78004d40 0x7f3d780051f0 secure :-1 s=READY pgs=1178 cs=0 l=1 rev1=1 crypto rx=0x7f3d940afb10 tx=0x7f3d78005830 comp rx=0 tx=0).stop 2024-08-03T20:55:21.310 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:21.308+0000 7f3d93e00640 1 -- 172.21.15.32:0/254119698 reap_dead start 2024-08-03T20:55:21.311 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:21.308+0000 7f3d9a600640 1 -- 172.21.15.32:0/254119698 shutdown_connections 2024-08-03T20:55:21.311 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:21.308+0000 7f3d9a600640 1 -- 172.21.15.32:0/254119698 >> 172.21.15.32:0/254119698 conn(0x7f3d940b1bd0 msgr2=0x7f3d940b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:55:21.311 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:21.309+0000 7f3d9a600640 1 -- 172.21.15.32:0/254119698 shutdown_connections 2024-08-03T20:55:21.311 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:21.309+0000 7f3d9a600640 1 -- 172.21.15.32:0/254119698 wait complete. 2024-08-03T20:55:21.312 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:21.311+0000 7f3d9a600640 1 Processor -- start 2024-08-03T20:55:21.312 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:21.311+0000 7f3d9a600640 1 -- start start 2024-08-03T20:55:21.315 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:21.312+0000 7f3d9a600640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3d78004d40 0x7f3d940b3790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:55:21.315 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:21.312+0000 7f3d9a600640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f3d940b3cd0 0x7f3d941332f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:55:21.315 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:21.312+0000 7f3d9a600640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f3d94133830 0x7f3d941f46a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:55:21.316 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:21.312+0000 7f3d9a600640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f3d940cf840 con 0x7f3d78004d40 2024-08-03T20:55:21.316 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:21.312+0000 7f3d9a600640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f3d94145f30 con 0x7f3d940b3cd0 2024-08-03T20:55:21.316 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:21.312+0000 7f3d9a600640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7f3d940cf9c0 con 0x7f3d94133830 2024-08-03T20:55:21.316 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:21.313+0000 7f3d93400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3d78004d40 0x7f3d940b3790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:55:21.316 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:21.313+0000 7f3d93400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3d78004d40 0x7f3d940b3790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:53272/0 (socket says 172.21.15.32:53272) 2024-08-03T20:55:21.316 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:21.313+0000 7f3d93400640 1 -- 172.21.15.32:0/3232753028 learned_addr learned my addr 172.21.15.32:0/3232753028 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:55:21.316 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:21.313+0000 7f3d8be00640 1 --2- 172.21.15.32:0/3232753028 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f3d940b3cd0 0x7f3d941332f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:55:21.316 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:21.313+0000 7f3d93e00640 1 --2- 172.21.15.32:0/3232753028 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f3d94133830 0x7f3d941f46a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:55:21.316 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:21.313+0000 7f3d93400640 1 -- 172.21.15.32:0/3232753028 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f3d94133830 msgr2=0x7f3d941f46a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:55:21.316 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:21.313+0000 7f3d93400640 1 --2- 172.21.15.32:0/3232753028 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f3d94133830 0x7f3d941f46a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:21.316 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:21.313+0000 7f3d93400640 1 -- 172.21.15.32:0/3232753028 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f3d940b3cd0 msgr2=0x7f3d941332f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:55:21.316 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:21.313+0000 7f3d93400640 1 --2- 172.21.15.32:0/3232753028 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f3d940b3cd0 0x7f3d941332f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:21.316 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:21.313+0000 7f3d93400640 1 -- 172.21.15.32:0/3232753028 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3d941340c0 con 0x7f3d78004d40 2024-08-03T20:55:21.317 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:21.313+0000 7f3d93400640 1 --2- 172.21.15.32:0/3232753028 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3d78004d40 0x7f3d940b3790 secure :-1 s=READY pgs=1179 cs=0 l=1 rev1=1 crypto rx=0x7f3d84003f50 tx=0x7f3d840016f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:55:21.317 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:21.313+0000 7f3d8b400640 1 -- 172.21.15.32:0/3232753028 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d84011640 con 0x7f3d78004d40 2024-08-03T20:55:21.317 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:21.314+0000 7f3d8b400640 1 -- 172.21.15.32:0/3232753028 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f3d84016020 con 0x7f3d78004d40 2024-08-03T20:55:21.317 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:21.314+0000 7f3d8b400640 1 -- 172.21.15.32:0/3232753028 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d84011a60 con 0x7f3d78004d40 2024-08-03T20:55:21.317 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:21.314+0000 7f3d9a600640 1 -- 172.21.15.32:0/3232753028 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3d941f8b50 con 0x7f3d78004d40 2024-08-03T20:55:21.317 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:21.314+0000 7f3d9a600640 1 -- 172.21.15.32:0/3232753028 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3d94142250 con 0x7f3d78004d40 2024-08-03T20:55:21.317 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:21.315+0000 7f3d9a600640 1 -- 172.21.15.32:0/3232753028 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3d941424b0 con 0x7f3d78004d40 2024-08-03T20:55:21.317 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:21.316+0000 7f3d8b400640 1 -- 172.21.15.32:0/3232753028 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f3d84011c00 con 0x7f3d78004d40 2024-08-03T20:55:21.318 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:21.317+0000 7f3d8b400640 1 --2- 172.21.15.32:0/3232753028 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f3d70077010 0x7f3d70079520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:55:21.318 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:21.317+0000 7f3d8b400640 1 -- 172.21.15.32:0/3232753028 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 6284+0+0 (secure 0 0 0) 0x7f3d84099840 con 0x7f3d78004d40 2024-08-03T20:55:21.319 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:21.317+0000 7f3d8be00640 1 --2- 172.21.15.32:0/3232753028 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f3d70077010 0x7f3d70079520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:55:21.319 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:21.318+0000 7f3d8be00640 1 --2- 172.21.15.32:0/3232753028 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f3d70077010 0x7f3d70079520 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7f3d9415c5b0 tx=0x7f3d7c005f90 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:55:21.323 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:21.322+0000 7f3d8b400640 1 -- 172.21.15.32:0/3232753028 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7f3d84020020 con 0x7f3d78004d40 2024-08-03T20:55:21.565 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:21.564+0000 7f3d9a600640 1 -- 172.21.15.32:0/3232753028 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f3d94074860 con 0x7f3d78004d40 2024-08-03T20:55:21.566 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:21.564+0000 7f3d8b400640 1 -- 172.21.15.32:0/3232753028 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 117 v117) ==== 95+0+4390 (secure 0 0 0) 0x7f3d84062760 con 0x7f3d78004d40 2024-08-03T20:55:21.566 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-03T20:55:21.566 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":117,"btime":"2024-08-03T20:55:20:021609+0000","default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6439,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6835/531080160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6834","nonce":531080160},{"type":"v1","addr":"172.21.15.118:6835","nonce":531080160}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":84},{"gid":6454,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6837/2796882319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6836","nonce":2796882319},{"type":"v1","addr":"172.21.15.118:6837","nonce":2796882319}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":86},{"gid":6813,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6834/2686933890","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":2686933890},{"type":"v1","addr":"172.21.15.32:6834","nonce":2686933890}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":85}],"filesystems":[{"mdsmap":{"epoch":117,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-08-03T20:54:26.950518+0000","modified":"2024-08-03T20:55:20.021534+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":88,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":6}]} 2024-08-03T20:55:21.566 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 117 2024-08-03T20:55:21.568 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:21.567+0000 7f3d9a600640 1 -- 172.21.15.32:0/3232753028 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f3d70077010 msgr2=0x7f3d70079520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:55:21.569 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:21.567+0000 7f3d9a600640 1 --2- 172.21.15.32:0/3232753028 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f3d70077010 0x7f3d70079520 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7f3d9415c5b0 tx=0x7f3d7c005f90 comp rx=0 tx=0).stop 2024-08-03T20:55:21.569 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:21.567+0000 7f3d9a600640 1 -- 172.21.15.32:0/3232753028 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3d78004d40 msgr2=0x7f3d940b3790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:55:21.569 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:21.567+0000 7f3d9a600640 1 --2- 172.21.15.32:0/3232753028 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3d78004d40 0x7f3d940b3790 secure :-1 s=READY pgs=1179 cs=0 l=1 rev1=1 crypto rx=0x7f3d84003f50 tx=0x7f3d840016f0 comp rx=0 tx=0).stop 2024-08-03T20:55:21.569 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:21.567+0000 7f3d9a600640 1 -- 172.21.15.32:0/3232753028 shutdown_connections 2024-08-03T20:55:21.569 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:21.567+0000 7f3d9a600640 1 --2- 172.21.15.32:0/3232753028 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f3d70077010 0x7f3d70079520 unknown :-1 s=CLOSED pgs=488 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:21.569 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:21.567+0000 7f3d9a600640 1 --2- 172.21.15.32:0/3232753028 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f3d940b3cd0 0x7f3d941332f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:21.569 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:21.567+0000 7f3d9a600640 1 --2- 172.21.15.32:0/3232753028 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f3d94133830 0x7f3d941f46a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:21.569 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:21.567+0000 7f3d9a600640 1 --2- 172.21.15.32:0/3232753028 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3d78004d40 0x7f3d940b3790 unknown :-1 s=CLOSED pgs=1179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:21.570 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:21.567+0000 7f3d9a600640 1 -- 172.21.15.32:0/3232753028 >> 172.21.15.32:0/3232753028 conn(0x7f3d940b1bd0 msgr2=0x7f3d940cb660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:55:21.570 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:21.567+0000 7f3d9a600640 1 -- 172.21.15.32:0/3232753028 shutdown_connections 2024-08-03T20:55:21.570 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:21.567+0000 7f3d9a600640 1 -- 172.21.15.32:0/3232753028 wait complete. 2024-08-03T20:55:21.593 INFO:tasks.cephfs.test_damage:Result: Mutation 'Delete 200.00000000' led to DAMAGED state 2024-08-03T20:55:21.593 INFO:tasks.cephfs.test_damage:Applying mutation 'Delete 200.00000001' 2024-08-03T20:55:21.593 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-08-03T20:55:21.593 DEBUG:teuthology.orchestra.run.smithi032:> dd if=/proc/self/mounts of=/dev/stdout 2024-08-03T20:55:21.629 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053453,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6499208k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 none /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249604k,nr_inodes=812401,mode=700,uid=1000,gid=1316 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6499208k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-08-03T20:55:21.629 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi032.front.sepia.ceph.com 2024-08-03T20:55:21.629 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-08-03T20:55:21.629 DEBUG:teuthology.orchestra.run.smithi032:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-08-03T20:55:21.704 INFO:teuthology.orchestra.run.smithi032.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.0': No such file or directory 2024-08-03T20:55:21.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-03T20:55:21.705 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-08-03T20:55:21.975 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:21.973+0000 7f3d0ec00640 1 -- 172.21.15.32:0/3255577120 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3d0806fb50 msgr2=0x7f3d080ad190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:55:21.975 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:21.973+0000 7f3d0ec00640 1 --2- 172.21.15.32:0/3255577120 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3d0806fb50 0x7f3d080ad190 secure :-1 s=READY pgs=1181 cs=0 l=1 rev1=1 crypto rx=0x7f3d080720a0 tx=0x7f3d0002a820 comp rx=0 tx=0).stop 2024-08-03T20:55:21.975 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:21.973+0000 7f3d0ec00640 1 -- 172.21.15.32:0/3255577120 shutdown_connections 2024-08-03T20:55:21.975 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:21.973+0000 7f3d0ec00640 1 --2- 172.21.15.32:0/3255577120 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3d0806fb50 0x7f3d080ad190 unknown :-1 s=CLOSED pgs=1181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:21.975 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:21.973+0000 7f3d0ec00640 1 --2- 172.21.15.32:0/3255577120 >> v2:172.21.15.118:3300/0 conn(0x7f3d080af020 0x7f3d080742b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:21.975 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:21.973+0000 7f3d0ec00640 1 -- 172.21.15.32:0/3255577120 >> 172.21.15.32:0/3255577120 conn(0x7f3d080b2120 msgr2=0x7f3d080cfd60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:55:21.976 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:21.973+0000 7f3d0ec00640 1 -- 172.21.15.32:0/3255577120 shutdown_connections 2024-08-03T20:55:21.976 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:21.973+0000 7f3d0ec00640 1 -- 172.21.15.32:0/3255577120 wait complete. 2024-08-03T20:55:21.977 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:21.975+0000 7f3d0ec00640 1 Processor -- start 2024-08-03T20:55:21.977 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:21.976+0000 7f3d0ec00640 1 -- start start 2024-08-03T20:55:21.979 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:21.977+0000 7f3d0ec00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3d0806fb50 0x7f3d080b3a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:55:21.979 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:21.977+0000 7f3d0ec00640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f3d080acd80 0x7f3d08155500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:55:21.979 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:21.977+0000 7f3d0ec00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f3d080af020 0x7f3d08155a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:55:21.979 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:21.977+0000 7f3d0ec00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f3d081601e0 con 0x7f3d0806fb50 2024-08-03T20:55:21.979 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:21.977+0000 7f3d0ec00640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f3d08160060 con 0x7f3d080af020 2024-08-03T20:55:21.979 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:21.977+0000 7f3d0ec00640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7f3d08160360 con 0x7f3d080acd80 2024-08-03T20:55:21.979 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:21.977+0000 7f3d0d800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3d0806fb50 0x7f3d080b3a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:55:21.980 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:21.978+0000 7f3d0d800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3d0806fb50 0x7f3d080b3a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:53296/0 (socket says 172.21.15.32:53296) 2024-08-03T20:55:21.980 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:21.978+0000 7f3d0d800640 1 -- 172.21.15.32:0/284166066 learned_addr learned my addr 172.21.15.32:0/284166066 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:55:21.980 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:21.978+0000 7f3d0ce00640 1 --2- 172.21.15.32:0/284166066 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f3d080acd80 0x7f3d08155500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:55:21.980 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:21.978+0000 7f3d0e200640 1 --2- 172.21.15.32:0/284166066 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f3d080af020 0x7f3d08155a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:55:21.981 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:21.978+0000 7f3d0d800640 1 -- 172.21.15.32:0/284166066 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f3d080acd80 msgr2=0x7f3d08155500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:55:21.981 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:21.978+0000 7f3d0d800640 1 --2- 172.21.15.32:0/284166066 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f3d080acd80 0x7f3d08155500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:21.981 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:21.978+0000 7f3d0d800640 1 -- 172.21.15.32:0/284166066 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f3d080af020 msgr2=0x7f3d08155a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:55:21.981 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:21.978+0000 7f3d0d800640 1 --2- 172.21.15.32:0/284166066 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f3d080af020 0x7f3d08155a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:21.981 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:21.978+0000 7f3d0d800640 1 -- 172.21.15.32:0/284166066 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3d081f8c20 con 0x7f3d0806fb50 2024-08-03T20:55:21.981 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:21.978+0000 7f3d0d800640 1 --2- 172.21.15.32:0/284166066 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3d0806fb50 0x7f3d080b3a40 secure :-1 s=READY pgs=1182 cs=0 l=1 rev1=1 crypto rx=0x7f3cfc00e7c0 tx=0x7f3cfc00ec80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:55:21.981 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:21.979+0000 7f3d05600640 1 -- 172.21.15.32:0/284166066 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3cfc00ccc0 con 0x7f3d0806fb50 2024-08-03T20:55:21.981 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:21.979+0000 7f3d05600640 1 -- 172.21.15.32:0/284166066 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f3cfc010430 con 0x7f3d0806fb50 2024-08-03T20:55:21.981 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:21.979+0000 7f3d05600640 1 -- 172.21.15.32:0/284166066 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3cfc010860 con 0x7f3d0806fb50 2024-08-03T20:55:21.981 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:21.979+0000 7f3d0ec00640 1 -- 172.21.15.32:0/284166066 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3d081f8f00 con 0x7f3d0806fb50 2024-08-03T20:55:21.981 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:21.979+0000 7f3d0ec00640 1 -- 172.21.15.32:0/284166066 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3d081f94c0 con 0x7f3d0806fb50 2024-08-03T20:55:21.982 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:21.980+0000 7f3d0ec00640 1 -- 172.21.15.32:0/284166066 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3d080ad190 con 0x7f3d0806fb50 2024-08-03T20:55:21.982 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:21.981+0000 7f3d05600640 1 -- 172.21.15.32:0/284166066 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f3cfc010a00 con 0x7f3d0806fb50 2024-08-03T20:55:21.983 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:21.982+0000 7f3d05600640 1 --2- 172.21.15.32:0/284166066 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f3ce8077010 0x7f3ce8079520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:55:21.983 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:21.982+0000 7f3d05600640 1 -- 172.21.15.32:0/284166066 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 6284+0+0 (secure 0 0 0) 0x7f3cfc099460 con 0x7f3d0806fb50 2024-08-03T20:55:21.984 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:21.982+0000 7f3d0ce00640 1 --2- 172.21.15.32:0/284166066 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f3ce8077010 0x7f3ce8079520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:55:21.984 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:21.983+0000 7f3d0ce00640 1 --2- 172.21.15.32:0/284166066 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f3ce8077010 0x7f3ce8079520 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7f3d0815bb10 tx=0x7f3cf4007e90 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:55:21.987 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:21.986+0000 7f3d05600640 1 -- 172.21.15.32:0/284166066 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7f3cfc061dc0 con 0x7f3d0806fb50 2024-08-03T20:55:22.238 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:22.236+0000 7f3d0ec00640 1 -- 172.21.15.32:0/284166066 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7f3d080d0930 con 0x7f3d0806fb50 2024-08-03T20:55:23.171 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:23.169+0000 7f3d05600640 1 -- 172.21.15.32:0/284166066 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v118) ==== 161+0+0 (secure 0 0 0) 0x7f3cfc066310 con 0x7f3d0806fb50 2024-08-03T20:55:23.171 INFO:teuthology.orchestra.run.smithi032.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-08-03T20:55:23.175 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:23.173+0000 7f3d0ec00640 1 -- 172.21.15.32:0/284166066 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f3ce8077010 msgr2=0x7f3ce8079520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:55:23.175 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:23.173+0000 7f3d0ec00640 1 --2- 172.21.15.32:0/284166066 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f3ce8077010 0x7f3ce8079520 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7f3d0815bb10 tx=0x7f3cf4007e90 comp rx=0 tx=0).stop 2024-08-03T20:55:23.175 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:23.173+0000 7f3d0ec00640 1 -- 172.21.15.32:0/284166066 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3d0806fb50 msgr2=0x7f3d080b3a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:55:23.175 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:23.173+0000 7f3d0ec00640 1 --2- 172.21.15.32:0/284166066 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3d0806fb50 0x7f3d080b3a40 secure :-1 s=READY pgs=1182 cs=0 l=1 rev1=1 crypto rx=0x7f3cfc00e7c0 tx=0x7f3cfc00ec80 comp rx=0 tx=0).stop 2024-08-03T20:55:23.175 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:23.173+0000 7f3d0ec00640 1 -- 172.21.15.32:0/284166066 shutdown_connections 2024-08-03T20:55:23.175 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:23.173+0000 7f3d0ec00640 1 --2- 172.21.15.32:0/284166066 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f3ce8077010 0x7f3ce8079520 unknown :-1 s=CLOSED pgs=489 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:23.175 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:23.173+0000 7f3d0ec00640 1 --2- 172.21.15.32:0/284166066 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f3d080af020 0x7f3d08155a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:23.175 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:23.173+0000 7f3d0ec00640 1 --2- 172.21.15.32:0/284166066 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f3d080acd80 0x7f3d08155500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:23.175 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:23.173+0000 7f3d0ec00640 1 --2- 172.21.15.32:0/284166066 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3d0806fb50 0x7f3d080b3a40 unknown :-1 s=CLOSED pgs=1182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:23.175 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:23.173+0000 7f3d0ec00640 1 -- 172.21.15.32:0/284166066 >> 172.21.15.32:0/284166066 conn(0x7f3d080b2120 msgr2=0x7f3d0815bd00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:55:23.176 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:23.173+0000 7f3d0ec00640 1 -- 172.21.15.32:0/284166066 shutdown_connections 2024-08-03T20:55:23.176 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:23.174+0000 7f3d0ec00640 1 -- 172.21.15.32:0/284166066 wait complete. 2024-08-03T20:55:23.203 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mds repaired 0 2024-08-03T20:55:23.436 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:23.435+0000 7fa75b000640 1 -- 172.21.15.32:0/662178937 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fa7540b40b0 msgr2=0x7fa75406fb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:55:23.437 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:23.435+0000 7fa75b000640 1 --2- 172.21.15.32:0/662178937 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fa7540b40b0 0x7fa75406fb30 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fa73c009830 tx=0x7fa73c02a510 comp rx=0 tx=0).stop 2024-08-03T20:55:23.437 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:23.435+0000 7fa75b000640 1 -- 172.21.15.32:0/662178937 shutdown_connections 2024-08-03T20:55:23.437 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:23.435+0000 7fa75b000640 1 --2- 172.21.15.32:0/662178937 >> v2:172.21.15.118:3300/0 conn(0x7fa7540ade80 0x7fa7540ae290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:23.437 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:23.435+0000 7fa75b000640 1 --2- 172.21.15.32:0/662178937 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fa7540b40b0 0x7fa75406fb30 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:23.437 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:23.435+0000 7fa75b000640 1 -- 172.21.15.32:0/662178937 >> 172.21.15.32:0/662178937 conn(0x7fa7540b1bd0 msgr2=0x7fa7540b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:55:23.437 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:23.435+0000 7fa75b000640 1 -- 172.21.15.32:0/662178937 shutdown_connections 2024-08-03T20:55:23.437 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:23.435+0000 7fa75b000640 1 -- 172.21.15.32:0/662178937 wait complete. 2024-08-03T20:55:23.438 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:23.437+0000 7fa75b000640 1 Processor -- start 2024-08-03T20:55:23.438 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:23.437+0000 7fa75b000640 1 -- start start 2024-08-03T20:55:23.441 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:23.439+0000 7fa75b000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa75406f720 0x7fa75414e8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:55:23.441 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:23.439+0000 7fa75b000640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa7540ade80 0x7fa7541611b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:55:23.441 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:23.439+0000 7fa75b000640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fa7540b40b0 0x7fa7541616f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:55:23.441 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:23.439+0000 7fa75b000640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fa7540aebe0 con 0x7fa75406f720 2024-08-03T20:55:23.441 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:23.439+0000 7fa75b000640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7fa7540aea60 con 0x7fa7540ade80 2024-08-03T20:55:23.441 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:23.439+0000 7fa75b000640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7fa7540aed60 con 0x7fa7540b40b0 2024-08-03T20:55:23.441 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:23.439+0000 7fa753e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa75406f720 0x7fa75414e8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:55:23.441 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:23.439+0000 7fa753e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa75406f720 0x7fa75414e8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:53312/0 (socket says 172.21.15.32:53312) 2024-08-03T20:55:23.442 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:23.439+0000 7fa753e00640 1 -- 172.21.15.32:0/2619458520 learned_addr learned my addr 172.21.15.32:0/2619458520 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:55:23.442 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:23.439+0000 7fa759000640 1 --2- 172.21.15.32:0/2619458520 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fa7540b40b0 0x7fa7541616f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:55:23.442 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:23.439+0000 7fa74be00640 1 --2- 172.21.15.32:0/2619458520 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa7540ade80 0x7fa7541611b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:55:23.442 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:23.439+0000 7fa753e00640 1 -- 172.21.15.32:0/2619458520 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fa7540b40b0 msgr2=0x7fa7541616f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:55:23.442 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:23.439+0000 7fa753e00640 1 --2- 172.21.15.32:0/2619458520 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fa7540b40b0 0x7fa7541616f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:23.442 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:23.440+0000 7fa753e00640 1 -- 172.21.15.32:0/2619458520 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa7540ade80 msgr2=0x7fa7541611b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:55:23.442 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:23.440+0000 7fa753e00640 1 --2- 172.21.15.32:0/2619458520 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa7540ade80 0x7fa7541611b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:23.442 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:23.440+0000 7fa753e00640 1 -- 172.21.15.32:0/2619458520 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa75415ab00 con 0x7fa75406f720 2024-08-03T20:55:23.442 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:23.440+0000 7fa753e00640 1 --2- 172.21.15.32:0/2619458520 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa75406f720 0x7fa75414e8b0 secure :-1 s=READY pgs=1183 cs=0 l=1 rev1=1 crypto rx=0x7fa74400c3c0 tx=0x7fa74400c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:55:23.443 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:23.440+0000 7fa750c00640 1 -- 172.21.15.32:0/2619458520 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa744017070 con 0x7fa75406f720 2024-08-03T20:55:23.443 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:23.440+0000 7fa750c00640 1 -- 172.21.15.32:0/2619458520 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fa7440035f0 con 0x7fa75406f720 2024-08-03T20:55:23.443 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:23.440+0000 7fa750c00640 1 -- 172.21.15.32:0/2619458520 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa744003a40 con 0x7fa75406f720 2024-08-03T20:55:23.443 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:23.440+0000 7fa75b000640 1 -- 172.21.15.32:0/2619458520 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa75415ade0 con 0x7fa75406f720 2024-08-03T20:55:23.443 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:23.440+0000 7fa75b000640 1 -- 172.21.15.32:0/2619458520 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa75415b3a0 con 0x7fa75406f720 2024-08-03T20:55:23.443 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:23.442+0000 7fa75b000640 1 -- 172.21.15.32:0/2619458520 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa7540b5100 con 0x7fa75406f720 2024-08-03T20:55:23.444 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:23.443+0000 7fa750c00640 1 -- 172.21.15.32:0/2619458520 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7fa74401a070 con 0x7fa75406f720 2024-08-03T20:55:23.445 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:23.444+0000 7fa750c00640 1 --2- 172.21.15.32:0/2619458520 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fa730077010 0x7fa730079520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:55:23.445 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:23.444+0000 7fa750c00640 1 -- 172.21.15.32:0/2619458520 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 6284+0+0 (secure 0 0 0) 0x7fa744097fb0 con 0x7fa75406f720 2024-08-03T20:55:23.446 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:23.444+0000 7fa74be00640 1 --2- 172.21.15.32:0/2619458520 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fa730077010 0x7fa730079520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:55:23.447 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:23.445+0000 7fa74be00640 1 --2- 172.21.15.32:0/2619458520 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fa730077010 0x7fa730079520 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7fa73c034000 tx=0x7fa73c004730 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:55:23.450 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:23.449+0000 7fa750c00640 1 -- 172.21.15.32:0/2619458520 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7fa7540b5100 con 0x7fa75406f720 2024-08-03T20:55:23.733 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:23.732+0000 7fa75b000640 1 -- 172.21.15.32:0/2619458520 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "mds repaired", "role": "0"} v 0) -- 0x7fa7540ae290 con 0x7fa75406f720 2024-08-03T20:55:24.173 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:24.171+0000 7fa750c00640 1 -- 172.21.15.32:0/2619458520 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "mds repaired", "role": "0"}]=0 repaired: restoring rank 6:0 v119) ==== 101+0+0 (secure 0 0 0) 0x7fa744060e50 con 0x7fa75406f720 2024-08-03T20:55:24.173 INFO:teuthology.orchestra.run.smithi032.stderr:repaired: restoring rank 6:0 2024-08-03T20:55:24.176 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:24.174+0000 7fa75b000640 1 -- 172.21.15.32:0/2619458520 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fa730077010 msgr2=0x7fa730079520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:55:24.176 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:24.174+0000 7fa75b000640 1 --2- 172.21.15.32:0/2619458520 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fa730077010 0x7fa730079520 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7fa73c034000 tx=0x7fa73c004730 comp rx=0 tx=0).stop 2024-08-03T20:55:24.176 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:24.174+0000 7fa75b000640 1 -- 172.21.15.32:0/2619458520 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa75406f720 msgr2=0x7fa75414e8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:55:24.176 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:24.174+0000 7fa75b000640 1 --2- 172.21.15.32:0/2619458520 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa75406f720 0x7fa75414e8b0 secure :-1 s=READY pgs=1183 cs=0 l=1 rev1=1 crypto rx=0x7fa74400c3c0 tx=0x7fa74400c880 comp rx=0 tx=0).stop 2024-08-03T20:55:24.176 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:24.174+0000 7fa75b000640 1 -- 172.21.15.32:0/2619458520 shutdown_connections 2024-08-03T20:55:24.176 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:24.174+0000 7fa75b000640 1 --2- 172.21.15.32:0/2619458520 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fa730077010 0x7fa730079520 unknown :-1 s=CLOSED pgs=490 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:24.176 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:24.174+0000 7fa75b000640 1 --2- 172.21.15.32:0/2619458520 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa7540ade80 0x7fa7541611b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:24.176 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:24.174+0000 7fa75b000640 1 --2- 172.21.15.32:0/2619458520 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fa7540b40b0 0x7fa7541616f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:24.176 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:24.174+0000 7fa75b000640 1 --2- 172.21.15.32:0/2619458520 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa75406f720 0x7fa75414e8b0 unknown :-1 s=CLOSED pgs=1183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:24.176 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:24.174+0000 7fa75b000640 1 -- 172.21.15.32:0/2619458520 >> 172.21.15.32:0/2619458520 conn(0x7fa7540b1bd0 msgr2=0x7fa754072d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:55:24.177 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:24.174+0000 7fa75b000640 1 -- 172.21.15.32:0/2619458520 shutdown_connections 2024-08-03T20:55:24.177 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:24.174+0000 7fa75b000640 1 -- 172.21.15.32:0/2619458520 wait complete. 2024-08-03T20:55:24.203 DEBUG:teuthology.orchestra.run.smithi032:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata import - 2024-08-03T20:55:24.348 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:24.347+0000 7f2052a0ef40 1 Processor -- start 2024-08-03T20:55:24.349 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:24.347+0000 7f2052a0ef40 1 -- start start 2024-08-03T20:55:24.350 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:24.348+0000 7f2052a0ef40 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x556a9deb3b40 0x556a9df1c200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:55:24.350 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:24.348+0000 7f2052a0ef40 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x556a9df26be0 0x556a9de6e640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:55:24.350 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:24.348+0000 7f2052a0ef40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x556a9df42860 0x556a9df42c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:55:24.350 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:24.348+0000 7f2052a0ef40 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x556a9ddbce30 con 0x556a9df26be0 2024-08-03T20:55:24.350 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:24.348+0000 7f2052a0ef40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x556a9de1abf0 con 0x556a9df42860 2024-08-03T20:55:24.351 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:24.348+0000 7f2052a0ef40 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x556a9df27510 con 0x556a9deb3b40 2024-08-03T20:55:24.351 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:24.349+0000 7f204a400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x556a9df26be0 0x556a9de6e640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:55:24.351 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:24.349+0000 7f204a400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x556a9df26be0 0x556a9de6e640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:53318/0 (socket says 172.21.15.32:53318) 2024-08-03T20:55:24.351 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:24.349+0000 7f204a400640 1 -- 172.21.15.32:0/2246438332 learned_addr learned my addr 172.21.15.32:0/2246438332 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:55:24.352 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:24.349+0000 7f204a400640 1 -- 172.21.15.32:0/2246438332 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x556a9deb3b40 msgr2=0x556a9df1c200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:55:24.352 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:24.349+0000 7f204a400640 1 --2- 172.21.15.32:0/2246438332 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x556a9deb3b40 0x556a9df1c200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:24.352 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:24.349+0000 7f204a400640 1 -- 172.21.15.32:0/2246438332 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x556a9df42860 msgr2=0x556a9df42c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:55:24.352 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:24.349+0000 7f204a400640 1 --2- 172.21.15.32:0/2246438332 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x556a9df42860 0x556a9df42c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:24.352 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:24.349+0000 7f204a400640 1 -- 172.21.15.32:0/2246438332 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556a9df239d0 con 0x556a9df26be0 2024-08-03T20:55:24.352 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:24.349+0000 7f204a400640 1 --2- 172.21.15.32:0/2246438332 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x556a9df26be0 0x556a9de6e640 secure :-1 s=READY pgs=1184 cs=0 l=1 rev1=1 crypto rx=0x7f2038001e60 tx=0x7f203800b730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=229ad8873d9052e server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:55:24.353 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:24.350+0000 7f2049000640 1 -- 172.21.15.32:0/2246438332 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f203800e830 con 0x556a9df26be0 2024-08-03T20:55:24.353 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:24.350+0000 7f2049000640 1 -- 172.21.15.32:0/2246438332 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f2038006050 con 0x556a9df26be0 2024-08-03T20:55:24.353 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:24.350+0000 7f2049000640 1 -- 172.21.15.32:0/2246438332 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f203800ed80 con 0x556a9df26be0 2024-08-03T20:55:24.353 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:24.350+0000 7f2052a0ef40 1 -- 172.21.15.32:0/2246438332 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x556a9df26be0 msgr2=0x556a9de6e640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:55:24.353 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:24.350+0000 7f2052a0ef40 1 --2- 172.21.15.32:0/2246438332 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x556a9df26be0 0x556a9de6e640 secure :-1 s=READY pgs=1184 cs=0 l=1 rev1=1 crypto rx=0x7f2038001e60 tx=0x7f203800b730 comp rx=0 tx=0).stop 2024-08-03T20:55:24.353 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:24.350+0000 7f2052a0ef40 1 -- 172.21.15.32:0/2246438332 shutdown_connections 2024-08-03T20:55:24.353 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:24.350+0000 7f2052a0ef40 1 --2- 172.21.15.32:0/2246438332 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x556a9df42860 0x556a9df42c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:24.353 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:24.350+0000 7f2052a0ef40 1 --2- 172.21.15.32:0/2246438332 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x556a9df26be0 0x556a9de6e640 unknown :-1 s=CLOSED pgs=1184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:24.353 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:24.350+0000 7f2052a0ef40 1 --2- 172.21.15.32:0/2246438332 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x556a9deb3b40 0x556a9df1c200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:24.353 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:24.350+0000 7f2052a0ef40 1 -- 172.21.15.32:0/2246438332 >> 172.21.15.32:0/2246438332 conn(0x556a9dd8eac0 msgr2=0x556a9df46a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:55:24.353 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:24.351+0000 7f2052a0ef40 1 -- 172.21.15.32:0/2246438332 shutdown_connections 2024-08-03T20:55:24.353 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:24.351+0000 7f2052a0ef40 1 -- 172.21.15.32:0/2246438332 wait complete. 2024-08-03T20:55:24.354 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:24.352+0000 7f2052a0ef40 1 Processor -- start 2024-08-03T20:55:24.354 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:24.352+0000 7f2052a0ef40 1 -- start start 2024-08-03T20:55:24.355 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:24.353+0000 7f2052a0ef40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x556a9deb3b40 0x556a9df17490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:55:24.355 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:24.353+0000 7f2052a0ef40 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x556a9df26be0 0x556a9de6cf80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:55:24.355 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:24.353+0000 7f2052a0ef40 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x556a9df42860 0x556a9de6c3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:55:24.355 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:24.353+0000 7f2052a0ef40 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x556a9df42200 con 0x556a9df42860 2024-08-03T20:55:24.355 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:24.353+0000 7f2052a0ef40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x556a9dfd23e0 con 0x556a9deb3b40 2024-08-03T20:55:24.355 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:24.353+0000 7f2052a0ef40 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x556a9df41f20 con 0x556a9df26be0 2024-08-03T20:55:24.355 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:24.353+0000 7f204b800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x556a9df42860 0x556a9de6c3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:55:24.356 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:24.354+0000 7f204b800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x556a9df42860 0x556a9de6c3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:53324/0 (socket says 172.21.15.32:53324) 2024-08-03T20:55:24.356 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:24.354+0000 7f204b800640 1 -- 172.21.15.32:0/2905321537 learned_addr learned my addr 172.21.15.32:0/2905321537 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:55:24.356 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:24.354+0000 7f204a400640 1 --2- 172.21.15.32:0/2905321537 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x556a9df26be0 0x556a9de6cf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:55:24.356 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:24.354+0000 7f204ae00640 1 --2- 172.21.15.32:0/2905321537 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x556a9deb3b40 0x556a9df17490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:55:24.357 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:24.354+0000 7f204b800640 1 -- 172.21.15.32:0/2905321537 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x556a9df26be0 msgr2=0x556a9de6cf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:55:24.357 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:24.354+0000 7f204b800640 1 --2- 172.21.15.32:0/2905321537 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x556a9df26be0 0x556a9de6cf80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:24.357 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:24.354+0000 7f204b800640 1 -- 172.21.15.32:0/2905321537 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x556a9deb3b40 msgr2=0x556a9df17490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:55:24.357 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:24.354+0000 7f204b800640 1 --2- 172.21.15.32:0/2905321537 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x556a9deb3b40 0x556a9df17490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:24.357 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:24.354+0000 7f204b800640 1 -- 172.21.15.32:0/2905321537 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556a9df1d160 con 0x556a9df42860 2024-08-03T20:55:24.358 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:24.354+0000 7f204b800640 1 --2- 172.21.15.32:0/2905321537 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x556a9df42860 0x556a9de6c3e0 secure :-1 s=READY pgs=1185 cs=0 l=1 rev1=1 crypto rx=0x7f2044025d70 tx=0x7f204400a5c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:55:24.358 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:24.355+0000 7f2043400640 1 -- 172.21.15.32:0/2905321537 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2044064070 con 0x556a9df42860 2024-08-03T20:55:24.358 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:24.355+0000 7f2043400640 1 -- 172.21.15.32:0/2905321537 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f204404d6c0 con 0x556a9df42860 2024-08-03T20:55:24.358 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:24.355+0000 7f2052a0ef40 1 -- 172.21.15.32:0/2905321537 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x556a9df45fb0 con 0x556a9df42860 2024-08-03T20:55:24.358 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:24.355+0000 7f2043400640 1 -- 172.21.15.32:0/2905321537 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2044004280 con 0x556a9df42860 2024-08-03T20:55:24.358 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:24.355+0000 7f2052a0ef40 1 -- 172.21.15.32:0/2905321537 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556a9df1ff80 con 0x556a9df42860 2024-08-03T20:55:24.358 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:24.357+0000 7f2043400640 1 -- 172.21.15.32:0/2905321537 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f204405e040 con 0x556a9df42860 2024-08-03T20:55:24.359 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:24.358+0000 7f2043400640 1 --2- 172.21.15.32:0/2905321537 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f202c0759f0 0x7f202c077f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:55:24.359 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:24.358+0000 7f2043400640 1 -- 172.21.15.32:0/2905321537 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 6284+0+0 (secure 0 0 0) 0x7f20440db700 con 0x556a9df42860 2024-08-03T20:55:24.360 INFO:teuthology.orchestra.run.smithi032.stdout:Importing pool 2024-08-03T20:55:24.360 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:24.358+0000 7f2052a0ef40 1 --2- 172.21.15.32:0/2905321537 >> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] conn(0x556a9df45a60 0x556a9df21d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:55:24.361 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:24.358+0000 7f204ae00640 1 --2- 172.21.15.32:0/2905321537 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f202c0759f0 0x7f202c077f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:55:24.361 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:24.358+0000 7f2052a0ef40 1 -- 172.21.15.32:0/2905321537 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:1 12.10 12:09401a81:::601.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e90) -- 0x556a9df222a0 con 0x556a9df45a60 2024-08-03T20:55:24.361 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:24.359+0000 7f204a400640 1 --2- 172.21.15.32:0/2905321537 >> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] conn(0x556a9df45a60 0x556a9df21d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:55:24.361 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:24.359+0000 7f204ae00640 1 --2- 172.21.15.32:0/2905321537 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f202c0759f0 0x7f202c077f00 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7f203c0040b0 tx=0x7f203c004b50 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:55:24.361 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:24.359+0000 7f204a400640 1 --2- 172.21.15.32:0/2905321537 >> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] conn(0x556a9df45a60 0x556a9df21d60 crc :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:55:24.363 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:24.361+0000 7f204a400640 1 -- 172.21.15.32:0/2905321537 <== osd.0 v2:172.21.15.32:6816/3066156116 1 ==== osd_op_reply(1 601.00000000 [create] v90'133 uv107 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f2038014070 con 0x556a9df45a60 2024-08-03T20:55:24.363 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:24.361+0000 7f2052a0ef40 1 -- 172.21.15.32:0/2905321537 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:2 12.10 12:09401a81:::601.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e90) -- 0x556a9dfebd00 con 0x556a9df45a60 2024-08-03T20:55:24.364 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:24.363+0000 7f204a400640 1 -- 172.21.15.32:0/2905321537 <== osd.0 v2:172.21.15.32:6816/3066156116 2 ==== osd_op_reply(2 601.00000000 [delete] v90'134 uv134 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f2038014070 con 0x556a9df45a60 2024-08-03T20:55:24.364 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:24.363+0000 7f2052a0ef40 1 -- 172.21.15.32:0/2905321537 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:3 12.10 12:09401a81:::601.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e90) -- 0x556a9dfec0a0 con 0x556a9df45a60 2024-08-03T20:55:24.366 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::601.00000000:head# 2024-08-03T20:55:24.366 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:24.365+0000 7f204a400640 1 -- 172.21.15.32:0/2905321537 <== osd.0 v2:172.21.15.32:6816/3066156116 3 ==== osd_op_reply(3 601.00000000 [create] v90'135 uv135 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f2038014070 con 0x556a9df45a60 2024-08-03T20:55:24.366 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:24.365+0000 7f2052a0ef40 1 -- 172.21.15.32:0/2905321537 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:4 12.10 12:09401a81:::601.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e90) -- 0x556a9dfed450 con 0x556a9df45a60 2024-08-03T20:55:24.368 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:24.366+0000 7f204a400640 1 -- 172.21.15.32:0/2905321537 <== osd.0 v2:172.21.15.32:6816/3066156116 4 ==== osd_op_reply(4 601.00000000 [setxattr (30)] v90'136 uv136 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f2038014070 con 0x556a9df45a60 2024-08-03T20:55:24.368 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:24.366+0000 7f2052a0ef40 1 -- 172.21.15.32:0/2905321537 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:5 12.10 12:09401a81:::601.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e90) -- 0x556a9df1d350 con 0x556a9df45a60 2024-08-03T20:55:24.369 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:24.368+0000 7f204a400640 1 -- 172.21.15.32:0/2905321537 <== osd.0 v2:172.21.15.32:6816/3066156116 5 ==== osd_op_reply(5 601.00000000 [setxattr (62)] v90'137 uv137 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f2038014070 con 0x556a9df45a60 2024-08-03T20:55:24.370 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:24.368+0000 7f2052a0ef40 1 -- 172.21.15.32:0/2905321537 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:6 12.10 12:09401a81:::601.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e90) -- 0x556a9dfec440 con 0x556a9df45a60 2024-08-03T20:55:24.371 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:24.369+0000 7f204a400640 1 -- 172.21.15.32:0/2905321537 <== osd.0 v2:172.21.15.32:6816/3066156116 6 ==== osd_op_reply(6 601.00000000 [omap-set-header] v90'138 uv138 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f2038014070 con 0x556a9df45a60 2024-08-03T20:55:24.371 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:24.370+0000 7f2052a0ef40 1 -- 172.21.15.32:0/2905321537 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:7 12.10 12:0bd6d154:::602.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e90) -- 0x556a9dfec7e0 con 0x556a9df45a60 2024-08-03T20:55:24.372 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:24.371+0000 7f204a400640 1 -- 172.21.15.32:0/2905321537 <== osd.0 v2:172.21.15.32:6816/3066156116 7 ==== osd_op_reply(7 602.00000000 [create] v90'139 uv113 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f2038014070 con 0x556a9df45a60 2024-08-03T20:55:24.372 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:24.371+0000 7f2052a0ef40 1 -- 172.21.15.32:0/2905321537 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:8 12.10 12:0bd6d154:::602.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e90) -- 0x556a9dfec7e0 con 0x556a9df45a60 2024-08-03T20:55:24.374 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:24.373+0000 7f204a400640 1 -- 172.21.15.32:0/2905321537 <== osd.0 v2:172.21.15.32:6816/3066156116 8 ==== osd_op_reply(8 602.00000000 [delete] v90'140 uv140 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f2038014070 con 0x556a9df45a60 2024-08-03T20:55:24.374 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:24.373+0000 7f2052a0ef40 1 -- 172.21.15.32:0/2905321537 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:9 12.10 12:0bd6d154:::602.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e90) -- 0x556a9dfec7e0 con 0x556a9df45a60 2024-08-03T20:55:24.375 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::602.00000000:head# 2024-08-03T20:55:24.376 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:24.374+0000 7f204a400640 1 -- 172.21.15.32:0/2905321537 <== osd.0 v2:172.21.15.32:6816/3066156116 9 ==== osd_op_reply(9 602.00000000 [create] v90'141 uv141 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f2038014070 con 0x556a9df45a60 2024-08-03T20:55:24.376 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:24.374+0000 7f2052a0ef40 1 -- 172.21.15.32:0/2905321537 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:10 12.10 12:0bd6d154:::602.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e90) -- 0x556a9dfec7e0 con 0x556a9df45a60 2024-08-03T20:55:24.377 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:24.376+0000 7f204a400640 1 -- 172.21.15.32:0/2905321537 <== osd.0 v2:172.21.15.32:6816/3066156116 10 ==== osd_op_reply(10 602.00000000 [setxattr (30)] v90'142 uv142 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f2038014070 con 0x556a9df45a60 2024-08-03T20:55:24.378 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:24.376+0000 7f2052a0ef40 1 -- 172.21.15.32:0/2905321537 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:11 12.10 12:0bd6d154:::602.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e90) -- 0x556a9dfec7e0 con 0x556a9df45a60 2024-08-03T20:55:24.379 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:24.378+0000 7f204a400640 1 -- 172.21.15.32:0/2905321537 <== osd.0 v2:172.21.15.32:6816/3066156116 11 ==== osd_op_reply(11 602.00000000 [setxattr (62)] v90'143 uv143 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f2038014070 con 0x556a9df45a60 2024-08-03T20:55:24.379 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:24.378+0000 7f2052a0ef40 1 -- 172.21.15.32:0/2905321537 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:12 12.10 12:0bd6d154:::602.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e90) -- 0x556a9dfec7e0 con 0x556a9df45a60 2024-08-03T20:55:24.381 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:24.379+0000 7f204a400640 1 -- 172.21.15.32:0/2905321537 <== osd.0 v2:172.21.15.32:6816/3066156116 12 ==== osd_op_reply(12 602.00000000 [omap-set-header] v90'144 uv144 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f2038014070 con 0x556a9df45a60 2024-08-03T20:55:24.381 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:24.379+0000 7f2052a0ef40 1 -- 172.21.15.32:0/2905321537 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:13 12.10 12:0cf6ad0f:::10000000000.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e90) -- 0x556a9dfec7e0 con 0x556a9df45a60 2024-08-03T20:55:24.382 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:24.381+0000 7f204a400640 1 -- 172.21.15.32:0/2905321537 <== osd.0 v2:172.21.15.32:6816/3066156116 13 ==== osd_op_reply(13 10000000000.00000000 [create] v90'145 uv120 ondisk = -17 ((17) File exists)) ==== 164+0+0 (crc 0 0 0) 0x7f2038014070 con 0x556a9df45a60 2024-08-03T20:55:24.382 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:24.381+0000 7f2052a0ef40 1 -- 172.21.15.32:0/2905321537 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:14 12.10 12:0cf6ad0f:::10000000000.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e90) -- 0x556a9dfec7e0 con 0x556a9df45a60 2024-08-03T20:55:24.384 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:24.383+0000 7f204a400640 1 -- 172.21.15.32:0/2905321537 <== osd.0 v2:172.21.15.32:6816/3066156116 14 ==== osd_op_reply(14 10000000000.00000000 [delete] v90'146 uv146 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f2038014070 con 0x556a9df45a60 2024-08-03T20:55:24.384 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:24.383+0000 7f2052a0ef40 1 -- 172.21.15.32:0/2905321537 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:15 12.10 12:0cf6ad0f:::10000000000.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e90) -- 0x556a9dfec7e0 con 0x556a9df45a60 2024-08-03T20:55:24.385 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::10000000000.00000000:head# 2024-08-03T20:55:24.386 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:24.384+0000 7f204a400640 1 -- 172.21.15.32:0/2905321537 <== osd.0 v2:172.21.15.32:6816/3066156116 15 ==== osd_op_reply(15 10000000000.00000000 [create] v90'147 uv147 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f2038014070 con 0x556a9df45a60 2024-08-03T20:55:24.386 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:24.384+0000 7f2052a0ef40 1 -- 172.21.15.32:0/2905321537 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:16 12.10 12:0cf6ad0f:::10000000000.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e90) -- 0x556a9dfec7e0 con 0x556a9df45a60 2024-08-03T20:55:24.387 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:24.386+0000 7f204a400640 1 -- 172.21.15.32:0/2905321537 <== osd.0 v2:172.21.15.32:6816/3066156116 16 ==== osd_op_reply(16 10000000000.00000000 [setxattr (30)] v90'148 uv148 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f2038014070 con 0x556a9df45a60 2024-08-03T20:55:24.387 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:24.386+0000 7f2052a0ef40 1 -- 172.21.15.32:0/2905321537 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:17 12.10 12:0cf6ad0f:::10000000000.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e90) -- 0x556a9dfec7e0 con 0x556a9df45a60 2024-08-03T20:55:24.389 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:24.388+0000 7f204a400640 1 -- 172.21.15.32:0/2905321537 <== osd.0 v2:172.21.15.32:6816/3066156116 17 ==== osd_op_reply(17 10000000000.00000000 [setxattr (62)] v90'149 uv149 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f2038014070 con 0x556a9df45a60 2024-08-03T20:55:24.389 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:24.388+0000 7f2052a0ef40 1 -- 172.21.15.32:0/2905321537 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:18 12.10 12:0cf6ad0f:::10000000000.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e90) -- 0x556a9dfec7e0 con 0x556a9df45a60 2024-08-03T20:55:24.390 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:24.389+0000 7f204a400640 1 -- 172.21.15.32:0/2905321537 <== osd.0 v2:172.21.15.32:6816/3066156116 18 ==== osd_op_reply(18 10000000000.00000000 [omap-set-header] v90'150 uv150 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f2038014070 con 0x556a9df45a60 2024-08-03T20:55:24.390 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:24.389+0000 7f2052a0ef40 1 -- 172.21.15.32:0/2905321537 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:19 12.10 12:0cf6ad0f:::10000000000.00000000:head [omap-set-vals in=593b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e90) -- 0x556a9dfecbc0 con 0x556a9df45a60 2024-08-03T20:55:24.392 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:24.391+0000 7f204a400640 1 -- 172.21.15.32:0/2905321537 <== osd.0 v2:172.21.15.32:6816/3066156116 19 ==== osd_op_reply(19 10000000000.00000000 [omap-set-vals] v90'151 uv151 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f2038014070 con 0x556a9df45a60 2024-08-03T20:55:24.392 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:24.391+0000 7f2052a0ef40 1 -- 172.21.15.32:0/2905321537 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:20 12.10 12:0d82a743:::600.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e90) -- 0x556a9dfecbc0 con 0x556a9df45a60 2024-08-03T20:55:24.394 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:24.393+0000 7f204a400640 1 -- 172.21.15.32:0/2905321537 <== osd.0 v2:172.21.15.32:6816/3066156116 20 ==== osd_op_reply(20 600.00000000 [create] v90'152 uv126 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f2038014070 con 0x556a9df45a60 2024-08-03T20:55:24.394 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:24.393+0000 7f2052a0ef40 1 -- 172.21.15.32:0/2905321537 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:21 12.10 12:0d82a743:::600.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e90) -- 0x556a9dfecbc0 con 0x556a9df45a60 2024-08-03T20:55:24.395 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:24.394+0000 7f204a400640 1 -- 172.21.15.32:0/2905321537 <== osd.0 v2:172.21.15.32:6816/3066156116 21 ==== osd_op_reply(21 600.00000000 [delete] v90'153 uv153 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f2038014070 con 0x556a9df45a60 2024-08-03T20:55:24.396 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:24.394+0000 7f2052a0ef40 1 -- 172.21.15.32:0/2905321537 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:22 12.10 12:0d82a743:::600.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e90) -- 0x556a9dfecbc0 con 0x556a9df45a60 2024-08-03T20:55:24.397 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::600.00000000:head# 2024-08-03T20:55:24.397 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:24.396+0000 7f204a400640 1 -- 172.21.15.32:0/2905321537 <== osd.0 v2:172.21.15.32:6816/3066156116 22 ==== osd_op_reply(22 600.00000000 [create] v90'154 uv154 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f2038014070 con 0x556a9df45a60 2024-08-03T20:55:24.397 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:24.396+0000 7f2052a0ef40 1 -- 172.21.15.32:0/2905321537 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:23 12.10 12:0d82a743:::600.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e90) -- 0x556a9dfecbc0 con 0x556a9df45a60 2024-08-03T20:55:24.399 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:24.397+0000 7f204a400640 1 -- 172.21.15.32:0/2905321537 <== osd.0 v2:172.21.15.32:6816/3066156116 23 ==== osd_op_reply(23 600.00000000 [setxattr (30)] v90'155 uv155 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f2038014070 con 0x556a9df45a60 2024-08-03T20:55:24.399 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:24.397+0000 7f2052a0ef40 1 -- 172.21.15.32:0/2905321537 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:24 12.10 12:0d82a743:::600.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e90) -- 0x556a9dfecaa0 con 0x556a9df45a60 2024-08-03T20:55:24.400 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:24.399+0000 7f204a400640 1 -- 172.21.15.32:0/2905321537 <== osd.0 v2:172.21.15.32:6816/3066156116 24 ==== osd_op_reply(24 600.00000000 [setxattr (62)] v90'156 uv156 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f2038014070 con 0x556a9df45a60 2024-08-03T20:55:24.400 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:24.399+0000 7f2052a0ef40 1 -- 172.21.15.32:0/2905321537 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:25 12.10 12:0d82a743:::600.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e90) -- 0x556a9dfecaa0 con 0x556a9df45a60 2024-08-03T20:55:24.402 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:24.401+0000 7f204a400640 1 -- 172.21.15.32:0/2905321537 <== osd.0 v2:172.21.15.32:6816/3066156116 25 ==== osd_op_reply(25 600.00000000 [omap-set-header] v90'157 uv157 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f2038014070 con 0x556a9df45a60 2024-08-03T20:55:24.402 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:24.401+0000 7f2052a0ef40 1 -- 172.21.15.32:0/2905321537 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:26 12.10 12:0d89b25e:::603.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e90) -- 0x556a9dfecaa0 con 0x556a9df45a60 2024-08-03T20:55:24.403 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:24.402+0000 7f204a400640 1 -- 172.21.15.32:0/2905321537 <== osd.0 v2:172.21.15.32:6816/3066156116 26 ==== osd_op_reply(26 603.00000000 [create] v90'158 uv132 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f2038014070 con 0x556a9df45a60 2024-08-03T20:55:24.404 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:24.402+0000 7f2052a0ef40 1 -- 172.21.15.32:0/2905321537 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:27 12.10 12:0d89b25e:::603.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e90) -- 0x556a9dfecaa0 con 0x556a9df45a60 2024-08-03T20:55:24.405 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:24.404+0000 7f204a400640 1 -- 172.21.15.32:0/2905321537 <== osd.0 v2:172.21.15.32:6816/3066156116 27 ==== osd_op_reply(27 603.00000000 [delete] v90'159 uv159 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f2038014070 con 0x556a9df45a60 2024-08-03T20:55:24.405 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:24.404+0000 7f2052a0ef40 1 -- 172.21.15.32:0/2905321537 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:28 12.10 12:0d89b25e:::603.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e90) -- 0x556a9dfecaa0 con 0x556a9df45a60 2024-08-03T20:55:24.407 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:24.405+0000 7f204a400640 1 -- 172.21.15.32:0/2905321537 <== osd.0 v2:172.21.15.32:6816/3066156116 28 ==== osd_op_reply(28 603.00000000 [create] v90'160 uv160 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f2038014070 con 0x556a9df45a60 2024-08-03T20:55:24.407 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::603.00000000:head# 2024-08-03T20:55:24.407 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:24.405+0000 7f2052a0ef40 1 -- 172.21.15.32:0/2905321537 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:29 12.10 12:0d89b25e:::603.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e90) -- 0x556a9dfecaa0 con 0x556a9df45a60 2024-08-03T20:55:24.409 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:24.407+0000 7f204a400640 1 -- 172.21.15.32:0/2905321537 <== osd.0 v2:172.21.15.32:6816/3066156116 29 ==== osd_op_reply(29 603.00000000 [setxattr (30)] v90'161 uv161 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f2038014070 con 0x556a9df45a60 2024-08-03T20:55:24.409 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:24.407+0000 7f2052a0ef40 1 -- 172.21.15.32:0/2905321537 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:30 12.10 12:0d89b25e:::603.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e90) -- 0x556a9dfecaa0 con 0x556a9df45a60 2024-08-03T20:55:24.410 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:24.409+0000 7f204a400640 1 -- 172.21.15.32:0/2905321537 <== osd.0 v2:172.21.15.32:6816/3066156116 30 ==== osd_op_reply(30 603.00000000 [setxattr (62)] v90'162 uv162 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f2038014070 con 0x556a9df45a60 2024-08-03T20:55:24.410 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:24.409+0000 7f2052a0ef40 1 -- 172.21.15.32:0/2905321537 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:31 12.10 12:0d89b25e:::603.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e90) -- 0x556a9dfecaa0 con 0x556a9df45a60 2024-08-03T20:55:24.412 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:24.410+0000 7f204a400640 1 -- 172.21.15.32:0/2905321537 <== osd.0 v2:172.21.15.32:6816/3066156116 31 ==== osd_op_reply(31 603.00000000 [omap-set-header] v90'163 uv163 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f2038014070 con 0x556a9df45a60 2024-08-03T20:55:24.412 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:24.411+0000 7f2052a0ef40 1 --2- 172.21.15.32:0/2905321537 >> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] conn(0x556a9dfed7f0 0x556a9dfefc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:55:24.412 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:24.411+0000 7f2052a0ef40 1 -- 172.21.15.32:0/2905321537 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:32 12.14 12:287034c4:::1.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e90) -- 0x556a9dff0150 con 0x556a9dfed7f0 2024-08-03T20:55:24.412 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:24.411+0000 7f204ae00640 1 --2- 172.21.15.32:0/2905321537 >> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] conn(0x556a9dfed7f0 0x556a9dfefc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:55:24.413 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:24.412+0000 7f204ae00640 1 --2- 172.21.15.32:0/2905321537 >> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] conn(0x556a9dfed7f0 0x556a9dfefc10 crc :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:55:24.415 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:24.414+0000 7f204ae00640 1 -- 172.21.15.32:0/2905321537 <== osd.5 v2:172.21.15.118:6810/2724891770 1 ==== osd_op_reply(32 1.00000000.inode [create] v90'109 uv91 ondisk = -17 ((17) File exists)) ==== 160+0+0 (crc 0 0 0) 0x7f203c00aa70 con 0x556a9dfed7f0 2024-08-03T20:55:24.415 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:24.414+0000 7f2052a0ef40 1 -- 172.21.15.32:0/2905321537 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:33 12.14 12:287034c4:::1.00000000.inode:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e90) -- 0x556a9dff04f0 con 0x556a9dfed7f0 2024-08-03T20:55:24.417 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:24.415+0000 7f204ae00640 1 -- 172.21.15.32:0/2905321537 <== osd.5 v2:172.21.15.118:6810/2724891770 2 ==== osd_op_reply(33 1.00000000.inode [delete] v90'110 uv110 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f203c00aa70 con 0x556a9dfed7f0 2024-08-03T20:55:24.417 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:24.415+0000 7f2052a0ef40 1 -- 172.21.15.32:0/2905321537 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:34 12.14 12:287034c4:::1.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e90) -- 0x556a9dff0890 con 0x556a9dfed7f0 2024-08-03T20:55:24.418 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:24.417+0000 7f204ae00640 1 -- 172.21.15.32:0/2905321537 <== osd.5 v2:172.21.15.118:6810/2724891770 3 ==== osd_op_reply(34 1.00000000.inode [create] v90'111 uv111 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f203c00aa70 con 0x556a9dfed7f0 2024-08-03T20:55:24.418 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::1.00000000.inode:head# 2024-08-03T20:55:24.419 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:24.417+0000 7f2052a0ef40 1 -- 172.21.15.32:0/2905321537 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:35 12.14 12:287034c4:::1.00000000.inode:head [write 0~582 in=582b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e90) -- 0x556a9dff0c30 con 0x556a9dfed7f0 2024-08-03T20:55:24.420 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:24.418+0000 7f204ae00640 1 -- 172.21.15.32:0/2905321537 <== osd.5 v2:172.21.15.118:6810/2724891770 4 ==== osd_op_reply(35 1.00000000.inode [write 0~582] v90'112 uv112 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f203c00aa70 con 0x556a9dfed7f0 2024-08-03T20:55:24.420 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:24.418+0000 7f2052a0ef40 1 -- 172.21.15.32:0/2905321537 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:36 12.14 12:287034c4:::1.00000000.inode:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e90) -- 0x556a9dff0fd0 con 0x556a9dfed7f0 2024-08-03T20:55:24.421 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:24.420+0000 7f204ae00640 1 -- 172.21.15.32:0/2905321537 <== osd.5 v2:172.21.15.118:6810/2724891770 5 ==== osd_op_reply(36 1.00000000.inode [omap-set-header] v90'113 uv113 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f203c00aa70 con 0x556a9dfed7f0 2024-08-03T20:55:24.421 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:24.420+0000 7f2052a0ef40 1 -- 172.21.15.32:0/2905321537 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:37 12.14 12:292cf221:::200.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e90) -- 0x556a9dff1370 con 0x556a9dfed7f0 2024-08-03T20:55:24.422 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:24.421+0000 7f204ae00640 1 -- 172.21.15.32:0/2905321537 <== osd.5 v2:172.21.15.118:6810/2724891770 6 ==== osd_op_reply(37 200.00000000 [create] v90'114 uv114 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f203c00aa70 con 0x556a9dfed7f0 2024-08-03T20:55:24.423 INFO:teuthology.orchestra.run.smithi032.stdout:Write #-9223372036854775808:00000000:::200.00000000:head# 2024-08-03T20:55:24.423 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:24.421+0000 7f2052a0ef40 1 -- 172.21.15.32:0/2905321537 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:38 12.14 12:292cf221:::200.00000000:head [write 0~90 in=90b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e90) -- 0x556a9dff1710 con 0x556a9dfed7f0 2024-08-03T20:55:24.424 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:24.423+0000 7f204ae00640 1 -- 172.21.15.32:0/2905321537 <== osd.5 v2:172.21.15.118:6810/2724891770 7 ==== osd_op_reply(38 200.00000000 [write 0~90] v90'115 uv115 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f203c00aa70 con 0x556a9dfed7f0 2024-08-03T20:55:24.424 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:24.423+0000 7f2052a0ef40 1 -- 172.21.15.32:0/2905321537 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:39 12.14 12:292cf221:::200.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e90) -- 0x556a9dff1ab0 con 0x556a9dfed7f0 2024-08-03T20:55:24.426 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:24.424+0000 7f204ae00640 1 -- 172.21.15.32:0/2905321537 <== osd.5 v2:172.21.15.118:6810/2724891770 8 ==== osd_op_reply(39 200.00000000 [omap-set-header] v90'116 uv116 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f203c00aa70 con 0x556a9dfed7f0 2024-08-03T20:55:24.426 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:24.424+0000 7f2052a0ef40 1 -- 172.21.15.32:0/2905321537 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:40 12.14 12:2e2fa760:::200.00000001:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e90) -- 0x556a9dff1ab0 con 0x556a9dfed7f0 2024-08-03T20:55:24.427 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:24.426+0000 7f204ae00640 1 -- 172.21.15.32:0/2905321537 <== osd.5 v2:172.21.15.118:6810/2724891770 9 ==== osd_op_reply(40 200.00000001 [create] v90'117 uv101 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f203c00aa70 con 0x556a9dfed7f0 2024-08-03T20:55:24.427 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:24.426+0000 7f2052a0ef40 1 -- 172.21.15.32:0/2905321537 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:41 12.14 12:2e2fa760:::200.00000001:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e90) -- 0x556a9dff1ab0 con 0x556a9dfed7f0 2024-08-03T20:55:24.429 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:24.427+0000 7f204ae00640 1 -- 172.21.15.32:0/2905321537 <== osd.5 v2:172.21.15.118:6810/2724891770 10 ==== osd_op_reply(41 200.00000001 [delete] v90'118 uv118 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f203c00aa70 con 0x556a9dfed7f0 2024-08-03T20:55:24.429 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:24.427+0000 7f2052a0ef40 1 -- 172.21.15.32:0/2905321537 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:42 12.14 12:2e2fa760:::200.00000001:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e90) -- 0x556a9dff1ab0 con 0x556a9dfed7f0 2024-08-03T20:55:24.430 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:24.429+0000 7f204ae00640 1 -- 172.21.15.32:0/2905321537 <== osd.5 v2:172.21.15.118:6810/2724891770 11 ==== osd_op_reply(42 200.00000001 [create] v90'119 uv119 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f203c00aa70 con 0x556a9dfed7f0 2024-08-03T20:55:24.430 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::200.00000001:head# 2024-08-03T20:55:24.430 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:24.429+0000 7f2052a0ef40 1 -- 172.21.15.32:0/2905321537 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:43 12.14 12:2e2fa760:::200.00000001:head [write 0~18118 in=18118b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e90) -- 0x556a9dff61a0 con 0x556a9dfed7f0 2024-08-03T20:55:24.432 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:24.430+0000 7f204ae00640 1 -- 172.21.15.32:0/2905321537 <== osd.5 v2:172.21.15.118:6810/2724891770 12 ==== osd_op_reply(43 200.00000001 [write 0~18118] v90'120 uv120 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f203c00aa70 con 0x556a9dfed7f0 2024-08-03T20:55:24.432 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:24.431+0000 7f2052a0ef40 1 -- 172.21.15.32:0/2905321537 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:44 12.14 12:2e2fa760:::200.00000001:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e90) -- 0x556a9dff61a0 con 0x556a9dfed7f0 2024-08-03T20:55:24.433 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:24.432+0000 7f204ae00640 1 -- 172.21.15.32:0/2905321537 <== osd.5 v2:172.21.15.118:6810/2724891770 13 ==== osd_op_reply(44 200.00000001 [omap-set-header] v90'121 uv121 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f203c00aa70 con 0x556a9dfed7f0 2024-08-03T20:55:24.433 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:24.432+0000 7f2052a0ef40 1 -- 172.21.15.32:0/2905321537 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:45 12.14 12:2f55791f:::606.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e90) -- 0x556a9dff1ab0 con 0x556a9dfed7f0 2024-08-03T20:55:24.435 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:24.433+0000 7f204ae00640 1 -- 172.21.15.32:0/2905321537 <== osd.5 v2:172.21.15.118:6810/2724891770 14 ==== osd_op_reply(45 606.00000000 [create] v90'122 uv107 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f203c00aa70 con 0x556a9dfed7f0 2024-08-03T20:55:24.435 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:24.433+0000 7f2052a0ef40 1 -- 172.21.15.32:0/2905321537 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:46 12.14 12:2f55791f:::606.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e90) -- 0x556a9dff1ab0 con 0x556a9dfed7f0 2024-08-03T20:55:24.436 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:24.435+0000 7f204ae00640 1 -- 172.21.15.32:0/2905321537 <== osd.5 v2:172.21.15.118:6810/2724891770 15 ==== osd_op_reply(46 606.00000000 [delete] v90'123 uv123 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f203c00aa70 con 0x556a9dfed7f0 2024-08-03T20:55:24.436 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:24.435+0000 7f2052a0ef40 1 -- 172.21.15.32:0/2905321537 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:47 12.14 12:2f55791f:::606.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e90) -- 0x556a9dff1ab0 con 0x556a9dfed7f0 2024-08-03T20:55:24.437 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::606.00000000:head# 2024-08-03T20:55:24.438 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:24.436+0000 7f204ae00640 1 -- 172.21.15.32:0/2905321537 <== osd.5 v2:172.21.15.118:6810/2724891770 16 ==== osd_op_reply(47 606.00000000 [create] v90'124 uv124 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f203c00aa70 con 0x556a9dfed7f0 2024-08-03T20:55:24.438 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:24.436+0000 7f2052a0ef40 1 -- 172.21.15.32:0/2905321537 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:48 12.14 12:2f55791f:::606.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e90) -- 0x556a9dff2ac0 con 0x556a9dfed7f0 2024-08-03T20:55:24.439 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:24.438+0000 7f204ae00640 1 -- 172.21.15.32:0/2905321537 <== osd.5 v2:172.21.15.118:6810/2724891770 17 ==== osd_op_reply(48 606.00000000 [setxattr (30)] v90'125 uv125 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f203c00aa70 con 0x556a9dfed7f0 2024-08-03T20:55:24.439 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:24.438+0000 7f2052a0ef40 1 -- 172.21.15.32:0/2905321537 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:49 12.14 12:2f55791f:::606.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e90) -- 0x556a9dff2ac0 con 0x556a9dfed7f0 2024-08-03T20:55:24.441 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:24.440+0000 7f204ae00640 1 -- 172.21.15.32:0/2905321537 <== osd.5 v2:172.21.15.118:6810/2724891770 18 ==== osd_op_reply(49 606.00000000 [setxattr (62)] v90'126 uv126 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f203c00aa70 con 0x556a9dfed7f0 2024-08-03T20:55:24.441 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:24.440+0000 7f2052a0ef40 1 -- 172.21.15.32:0/2905321537 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:50 12.14 12:2f55791f:::606.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e90) -- 0x556a9dff1ab0 con 0x556a9dfed7f0 2024-08-03T20:55:24.442 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:24.441+0000 7f204ae00640 1 -- 172.21.15.32:0/2905321537 <== osd.5 v2:172.21.15.118:6810/2724891770 19 ==== osd_op_reply(50 606.00000000 [omap-set-header] v90'127 uv127 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f203c00aa70 con 0x556a9dfed7f0 2024-08-03T20:55:24.442 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:24.441+0000 7f2052a0ef40 1 -- 172.21.15.32:0/2905321537 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:51 12.1c 12:3ed09add:::607.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e90) -- 0x556a9dff1ab0 con 0x556a9df45a60 2024-08-03T20:55:24.444 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:24.442+0000 7f204a400640 1 -- 172.21.15.32:0/2905321537 <== osd.0 v2:172.21.15.32:6816/3066156116 32 ==== osd_op_reply(51 607.00000000 [create] v90'27 uv26 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f2038014070 con 0x556a9df45a60 2024-08-03T20:55:24.444 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:24.442+0000 7f2052a0ef40 1 -- 172.21.15.32:0/2905321537 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:52 12.1c 12:3ed09add:::607.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e90) -- 0x556a9dff1ab0 con 0x556a9df45a60 2024-08-03T20:55:24.445 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:24.444+0000 7f204a400640 1 -- 172.21.15.32:0/2905321537 <== osd.0 v2:172.21.15.32:6816/3066156116 33 ==== osd_op_reply(52 607.00000000 [delete] v90'28 uv28 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f2038014070 con 0x556a9df45a60 2024-08-03T20:55:24.445 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:24.444+0000 7f2052a0ef40 1 -- 172.21.15.32:0/2905321537 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:53 12.1c 12:3ed09add:::607.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e90) -- 0x556a9dff1ab0 con 0x556a9df45a60 2024-08-03T20:55:24.446 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:24.445+0000 7f204a400640 1 -- 172.21.15.32:0/2905321537 <== osd.0 v2:172.21.15.32:6816/3066156116 34 ==== osd_op_reply(53 607.00000000 [create] v90'29 uv29 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f2038014070 con 0x556a9df45a60 2024-08-03T20:55:24.447 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::607.00000000:head# 2024-08-03T20:55:24.447 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:24.445+0000 7f2052a0ef40 1 -- 172.21.15.32:0/2905321537 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:54 12.1c 12:3ed09add:::607.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e90) -- 0x556a9dff2ac0 con 0x556a9df45a60 2024-08-03T20:55:24.448 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:24.447+0000 7f204a400640 1 -- 172.21.15.32:0/2905321537 <== osd.0 v2:172.21.15.32:6816/3066156116 35 ==== osd_op_reply(54 607.00000000 [setxattr (30)] v90'30 uv30 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f2038014070 con 0x556a9df45a60 2024-08-03T20:55:24.448 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:24.447+0000 7f2052a0ef40 1 -- 172.21.15.32:0/2905321537 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:55 12.1c 12:3ed09add:::607.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e90) -- 0x556a9dff2ac0 con 0x556a9df45a60 2024-08-03T20:55:24.449 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:24.448+0000 7f204a400640 1 -- 172.21.15.32:0/2905321537 <== osd.0 v2:172.21.15.32:6816/3066156116 36 ==== osd_op_reply(55 607.00000000 [setxattr (62)] v90'31 uv31 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f2038014070 con 0x556a9df45a60 2024-08-03T20:55:24.449 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:24.448+0000 7f2052a0ef40 1 -- 172.21.15.32:0/2905321537 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:56 12.1c 12:3ed09add:::607.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e90) -- 0x556a9dff1ab0 con 0x556a9df45a60 2024-08-03T20:55:24.451 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:24.449+0000 7f204a400640 1 -- 172.21.15.32:0/2905321537 <== osd.0 v2:172.21.15.32:6816/3066156116 37 ==== osd_op_reply(56 607.00000000 [omap-set-header] v90'32 uv32 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f2038014070 con 0x556a9df45a60 2024-08-03T20:55:24.451 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:24.449+0000 7f2052a0ef40 1 --2- 172.21.15.32:0/2905321537 >> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] conn(0x556a9dff23b0 0x556a9dff47d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:55:24.451 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:24.449+0000 7f2052a0ef40 1 -- 172.21.15.32:0/2905321537 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:57 12.6 12:654134d2:::mds0_openfiles.0:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e90) -- 0x556a9dff4d10 con 0x556a9dff23b0 2024-08-03T20:55:24.451 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:24.450+0000 7f204a400640 1 --2- 172.21.15.32:0/2905321537 >> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] conn(0x556a9dff23b0 0x556a9dff47d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:55:24.452 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:24.450+0000 7f204a400640 1 --2- 172.21.15.32:0/2905321537 >> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] conn(0x556a9dff23b0 0x556a9dff47d0 crc :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:55:24.453 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:24.452+0000 7f204a400640 1 -- 172.21.15.32:0/2905321537 <== osd.1 v2:172.21.15.32:6824/1551241759 1 ==== osd_op_reply(57 mds0_openfiles.0 [create] v90'24 uv23 ondisk = -17 ((17) File exists)) ==== 160+0+0 (crc 0 0 0) 0x7f2038014070 con 0x556a9dff23b0 2024-08-03T20:55:24.453 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:24.452+0000 7f2052a0ef40 1 -- 172.21.15.32:0/2905321537 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:58 12.6 12:654134d2:::mds0_openfiles.0:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e90) -- 0x556a9dff4d10 con 0x556a9dff23b0 2024-08-03T20:55:24.454 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:24.453+0000 7f204a400640 1 -- 172.21.15.32:0/2905321537 <== osd.1 v2:172.21.15.32:6824/1551241759 2 ==== osd_op_reply(58 mds0_openfiles.0 [delete] v90'25 uv25 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f2038014070 con 0x556a9dff23b0 2024-08-03T20:55:24.454 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:24.453+0000 7f2052a0ef40 1 -- 172.21.15.32:0/2905321537 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:59 12.6 12:654134d2:::mds0_openfiles.0:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e90) -- 0x556a9dff5150 con 0x556a9dff23b0 2024-08-03T20:55:24.455 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:24.454+0000 7f204a400640 1 -- 172.21.15.32:0/2905321537 <== osd.1 v2:172.21.15.32:6824/1551241759 3 ==== osd_op_reply(59 mds0_openfiles.0 [create] v90'26 uv26 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f2038014070 con 0x556a9dff23b0 2024-08-03T20:55:24.456 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds0_openfiles.0:head# 2024-08-03T20:55:24.456 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:24.454+0000 7f2052a0ef40 1 -- 172.21.15.32:0/2905321537 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:60 12.6 12:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e90) -- 0x556a9dff5150 con 0x556a9dff23b0 2024-08-03T20:55:24.457 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:24.455+0000 7f204a400640 1 -- 172.21.15.32:0/2905321537 <== osd.1 v2:172.21.15.32:6824/1551241759 4 ==== osd_op_reply(60 mds0_openfiles.0 [omap-set-header] v90'27 uv27 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f2038014070 con 0x556a9dff23b0 2024-08-03T20:55:24.457 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:24.455+0000 7f2052a0ef40 1 -- 172.21.15.32:0/2905321537 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:61 12.e 12:717a0223:::608.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e90) -- 0x556a9dff5150 con 0x556a9df45a60 2024-08-03T20:55:24.458 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:24.457+0000 7f204a400640 1 -- 172.21.15.32:0/2905321537 <== osd.0 v2:172.21.15.32:6816/3066156116 38 ==== osd_op_reply(61 608.00000000 [create] v90'27 uv26 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f2038014070 con 0x556a9df45a60 2024-08-03T20:55:24.459 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:24.457+0000 7f2052a0ef40 1 -- 172.21.15.32:0/2905321537 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:62 12.e 12:717a0223:::608.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e90) -- 0x556a9dff5150 con 0x556a9df45a60 2024-08-03T20:55:24.460 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:24.459+0000 7f204a400640 1 -- 172.21.15.32:0/2905321537 <== osd.0 v2:172.21.15.32:6816/3066156116 39 ==== osd_op_reply(62 608.00000000 [delete] v90'28 uv28 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f2038014070 con 0x556a9df45a60 2024-08-03T20:55:24.460 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:24.459+0000 7f2052a0ef40 1 -- 172.21.15.32:0/2905321537 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:63 12.e 12:717a0223:::608.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e90) -- 0x556a9dff5150 con 0x556a9df45a60 2024-08-03T20:55:24.461 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::608.00000000:head# 2024-08-03T20:55:24.462 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:24.460+0000 7f204a400640 1 -- 172.21.15.32:0/2905321537 <== osd.0 v2:172.21.15.32:6816/3066156116 40 ==== osd_op_reply(63 608.00000000 [create] v90'29 uv29 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f2038014070 con 0x556a9df45a60 2024-08-03T20:55:24.462 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:24.460+0000 7f2052a0ef40 1 -- 172.21.15.32:0/2905321537 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:64 12.e 12:717a0223:::608.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e90) -- 0x556a9dff6160 con 0x556a9df45a60 2024-08-03T20:55:24.463 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:24.462+0000 7f204a400640 1 -- 172.21.15.32:0/2905321537 <== osd.0 v2:172.21.15.32:6816/3066156116 41 ==== osd_op_reply(64 608.00000000 [setxattr (30)] v90'30 uv30 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f2038014070 con 0x556a9df45a60 2024-08-03T20:55:24.463 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:24.462+0000 7f2052a0ef40 1 -- 172.21.15.32:0/2905321537 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:65 12.e 12:717a0223:::608.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e90) -- 0x556a9dff6160 con 0x556a9df45a60 2024-08-03T20:55:24.464 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:24.463+0000 7f204a400640 1 -- 172.21.15.32:0/2905321537 <== osd.0 v2:172.21.15.32:6816/3066156116 42 ==== osd_op_reply(65 608.00000000 [setxattr (62)] v90'31 uv31 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f2038014070 con 0x556a9df45a60 2024-08-03T20:55:24.465 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:24.463+0000 7f2052a0ef40 1 -- 172.21.15.32:0/2905321537 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:66 12.e 12:717a0223:::608.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e90) -- 0x556a9dff5150 con 0x556a9df45a60 2024-08-03T20:55:24.466 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:24.464+0000 7f204a400640 1 -- 172.21.15.32:0/2905321537 <== osd.0 v2:172.21.15.32:6816/3066156116 43 ==== osd_op_reply(66 608.00000000 [omap-set-header] v90'32 uv32 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f2038014070 con 0x556a9df45a60 2024-08-03T20:55:24.466 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:24.464+0000 7f2052a0ef40 1 --2- 172.21.15.32:0/2905321537 >> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] conn(0x556a9dff5660 0x556a9dff7a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:55:24.466 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:24.464+0000 7f2052a0ef40 1 -- 172.21.15.32:0/2905321537 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:67 12.15 12:a93a17c2:::604.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e90) -- 0x556a9dff8090 con 0x556a9dff5660 2024-08-03T20:55:24.466 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:24.465+0000 7f204b800640 1 --2- 172.21.15.32:0/2905321537 >> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] conn(0x556a9dff5660 0x556a9dff7a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:55:24.467 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:24.465+0000 7f204b800640 1 --2- 172.21.15.32:0/2905321537 >> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] conn(0x556a9dff5660 0x556a9dff7a80 crc :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:55:24.468 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:24.467+0000 7f204b800640 1 -- 172.21.15.32:0/2905321537 <== osd.6 v2:172.21.15.118:6825/801947386 1 ==== osd_op_reply(67 604.00000000 [create] v90'48 uv42 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f204406f070 con 0x556a9dff5660 2024-08-03T20:55:24.468 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:24.467+0000 7f2052a0ef40 1 -- 172.21.15.32:0/2905321537 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:68 12.15 12:a93a17c2:::604.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e90) -- 0x556a9dff8530 con 0x556a9dff5660 2024-08-03T20:55:24.469 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:24.468+0000 7f204b800640 1 -- 172.21.15.32:0/2905321537 <== osd.6 v2:172.21.15.118:6825/801947386 2 ==== osd_op_reply(68 604.00000000 [delete] v90'49 uv49 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f204406f070 con 0x556a9dff5660 2024-08-03T20:55:24.470 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:24.468+0000 7f2052a0ef40 1 -- 172.21.15.32:0/2905321537 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:69 12.15 12:a93a17c2:::604.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e90) -- 0x556a9dff8a10 con 0x556a9dff5660 2024-08-03T20:55:24.471 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::604.00000000:head# 2024-08-03T20:55:24.471 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:24.470+0000 7f204b800640 1 -- 172.21.15.32:0/2905321537 <== osd.6 v2:172.21.15.118:6825/801947386 3 ==== osd_op_reply(69 604.00000000 [create] v90'50 uv50 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f204406f070 con 0x556a9dff5660 2024-08-03T20:55:24.471 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:24.470+0000 7f2052a0ef40 1 -- 172.21.15.32:0/2905321537 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:70 12.15 12:a93a17c2:::604.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e90) -- 0x556a9dffa2c0 con 0x556a9dff5660 2024-08-03T20:55:24.472 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:24.471+0000 7f204b800640 1 -- 172.21.15.32:0/2905321537 <== osd.6 v2:172.21.15.118:6825/801947386 4 ==== osd_op_reply(70 604.00000000 [setxattr (30)] v90'51 uv51 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f204406f070 con 0x556a9dff5660 2024-08-03T20:55:24.472 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:24.471+0000 7f2052a0ef40 1 -- 172.21.15.32:0/2905321537 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:71 12.15 12:a93a17c2:::604.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e90) -- 0x556a9dffa660 con 0x556a9dff5660 2024-08-03T20:55:24.474 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:24.472+0000 7f204b800640 1 -- 172.21.15.32:0/2905321537 <== osd.6 v2:172.21.15.118:6825/801947386 5 ==== osd_op_reply(71 604.00000000 [setxattr (62)] v90'52 uv52 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f204406f070 con 0x556a9dff5660 2024-08-03T20:55:24.474 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:24.472+0000 7f2052a0ef40 1 -- 172.21.15.32:0/2905321537 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:72 12.15 12:a93a17c2:::604.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e90) -- 0x556a9dff92b0 con 0x556a9dff5660 2024-08-03T20:55:24.475 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:24.474+0000 7f204b800640 1 -- 172.21.15.32:0/2905321537 <== osd.6 v2:172.21.15.118:6825/801947386 6 ==== osd_op_reply(72 604.00000000 [omap-set-header] v90'53 uv53 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f204406f070 con 0x556a9dff5660 2024-08-03T20:55:24.475 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:24.474+0000 7f2052a0ef40 1 -- 172.21.15.32:0/2905321537 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:73 12.15 12:aa448500:::500.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e90) -- 0x556a9dff96c0 con 0x556a9dff5660 2024-08-03T20:55:24.477 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:24.475+0000 7f204b800640 1 -- 172.21.15.32:0/2905321537 <== osd.6 v2:172.21.15.118:6825/801947386 7 ==== osd_op_reply(73 500.00000000 [create] v90'54 uv47 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f204406f070 con 0x556a9dff5660 2024-08-03T20:55:24.477 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:24.475+0000 7f2052a0ef40 1 -- 172.21.15.32:0/2905321537 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:74 12.15 12:aa448500:::500.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e90) -- 0x556a9dff96c0 con 0x556a9dff5660 2024-08-03T20:55:24.478 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:24.477+0000 7f204b800640 1 -- 172.21.15.32:0/2905321537 <== osd.6 v2:172.21.15.118:6825/801947386 8 ==== osd_op_reply(74 500.00000000 [delete] v90'55 uv55 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f204406f070 con 0x556a9dff5660 2024-08-03T20:55:24.478 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:24.477+0000 7f2052a0ef40 1 -- 172.21.15.32:0/2905321537 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:75 12.15 12:aa448500:::500.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e90) -- 0x556a9dff96c0 con 0x556a9dff5660 2024-08-03T20:55:24.480 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::500.00000000:head# 2024-08-03T20:55:24.480 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:24.478+0000 7f204b800640 1 -- 172.21.15.32:0/2905321537 <== osd.6 v2:172.21.15.118:6825/801947386 9 ==== osd_op_reply(75 500.00000000 [create] v90'56 uv56 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f204406f070 con 0x556a9dff5660 2024-08-03T20:55:24.480 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:24.478+0000 7f2052a0ef40 1 -- 172.21.15.32:0/2905321537 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:76 12.15 12:aa448500:::500.00000000:head [write 0~90 in=90b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e90) -- 0x556a9dff96c0 con 0x556a9dff5660 2024-08-03T20:55:24.482 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:24.480+0000 7f204b800640 1 -- 172.21.15.32:0/2905321537 <== osd.6 v2:172.21.15.118:6825/801947386 10 ==== osd_op_reply(76 500.00000000 [write 0~90] v90'57 uv57 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f204406f070 con 0x556a9dff5660 2024-08-03T20:55:24.482 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:24.480+0000 7f2052a0ef40 1 -- 172.21.15.32:0/2905321537 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:77 12.15 12:aa448500:::500.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e90) -- 0x556a9dff96c0 con 0x556a9dff5660 2024-08-03T20:55:24.483 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:24.481+0000 7f204b800640 1 -- 172.21.15.32:0/2905321537 <== osd.6 v2:172.21.15.118:6825/801947386 11 ==== osd_op_reply(77 500.00000000 [omap-set-header] v90'58 uv58 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f204406f070 con 0x556a9dff5660 2024-08-03T20:55:24.483 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:24.481+0000 7f2052a0ef40 1 --2- 172.21.15.32:0/2905321537 >> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] conn(0x556a9dffaa00 0x556a9dffce20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:55:24.483 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:24.481+0000 7f2052a0ef40 1 -- 172.21.15.32:0/2905321537 --> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] -- osd_op(unknown.0.0:78 12.d 12:b50e409b:::mds_snaptable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e90) -- 0x556a9dff96c0 con 0x556a9dffaa00 2024-08-03T20:55:24.483 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:24.482+0000 7f204ae00640 1 --2- 172.21.15.32:0/2905321537 >> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] conn(0x556a9dffaa00 0x556a9dffce20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:55:24.484 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:24.482+0000 7f204ae00640 1 --2- 172.21.15.32:0/2905321537 >> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] conn(0x556a9dffaa00 0x556a9dffce20 crc :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:55:24.485 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:24.484+0000 7f204ae00640 1 -- 172.21.15.32:0/2905321537 <== osd.7 v2:172.21.15.118:6818/653000048 1 ==== osd_op_reply(78 mds_snaptable [create] v90'22 uv21 ondisk = -17 ((17) File exists)) ==== 157+0+0 (crc 0 0 0) 0x7f203c00aa70 con 0x556a9dffaa00 2024-08-03T20:55:24.485 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:24.484+0000 7f2052a0ef40 1 -- 172.21.15.32:0/2905321537 --> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] -- osd_op(unknown.0.0:79 12.d 12:b50e409b:::mds_snaptable:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e90) -- 0x556a9dff96c0 con 0x556a9dffaa00 2024-08-03T20:55:24.486 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:24.485+0000 7f204ae00640 1 -- 172.21.15.32:0/2905321537 <== osd.7 v2:172.21.15.118:6818/653000048 2 ==== osd_op_reply(79 mds_snaptable [delete] v90'23 uv23 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f203c00aa70 con 0x556a9dffaa00 2024-08-03T20:55:24.486 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:24.485+0000 7f2052a0ef40 1 -- 172.21.15.32:0/2905321537 --> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] -- osd_op(unknown.0.0:80 12.d 12:b50e409b:::mds_snaptable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e90) -- 0x556a9dffd360 con 0x556a9dffaa00 2024-08-03T20:55:24.488 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:24.486+0000 7f204ae00640 1 -- 172.21.15.32:0/2905321537 <== osd.7 v2:172.21.15.118:6818/653000048 3 ==== osd_op_reply(80 mds_snaptable [create] v90'24 uv24 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f203c00aa70 con 0x556a9dffaa00 2024-08-03T20:55:24.488 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds_snaptable:head# 2024-08-03T20:55:24.488 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:24.486+0000 7f2052a0ef40 1 -- 172.21.15.32:0/2905321537 --> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] -- osd_op(unknown.0.0:81 12.d 12:b50e409b:::mds_snaptable:head [write 0~70 in=70b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e90) -- 0x556a9dffd360 con 0x556a9dffaa00 2024-08-03T20:55:24.489 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:24.488+0000 7f204ae00640 1 -- 172.21.15.32:0/2905321537 <== osd.7 v2:172.21.15.118:6818/653000048 4 ==== osd_op_reply(81 mds_snaptable [write 0~70] v90'25 uv25 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f203c00aa70 con 0x556a9dffaa00 2024-08-03T20:55:24.489 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:24.488+0000 7f2052a0ef40 1 -- 172.21.15.32:0/2905321537 --> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] -- osd_op(unknown.0.0:82 12.d 12:b50e409b:::mds_snaptable:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e90) -- 0x556a9dffd360 con 0x556a9dffaa00 2024-08-03T20:55:24.491 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:24.489+0000 7f204ae00640 1 -- 172.21.15.32:0/2905321537 <== osd.7 v2:172.21.15.118:6818/653000048 5 ==== osd_op_reply(82 mds_snaptable [omap-set-header] v90'26 uv26 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f203c00aa70 con 0x556a9dffaa00 2024-08-03T20:55:24.491 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:24.489+0000 7f2052a0ef40 1 -- 172.21.15.32:0/2905321537 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:83 12.1d 12:b871830b:::605.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e90) -- 0x556a9dffd360 con 0x556a9df45a60 2024-08-03T20:55:24.492 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:24.491+0000 7f204a400640 1 -- 172.21.15.32:0/2905321537 <== osd.0 v2:172.21.15.32:6816/3066156116 44 ==== osd_op_reply(83 605.00000000 [create] v90'27 uv26 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f2038014070 con 0x556a9df45a60 2024-08-03T20:55:24.492 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:24.491+0000 7f2052a0ef40 1 -- 172.21.15.32:0/2905321537 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:84 12.1d 12:b871830b:::605.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e90) -- 0x556a9dffd360 con 0x556a9df45a60 2024-08-03T20:55:24.493 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:24.492+0000 7f204a400640 1 -- 172.21.15.32:0/2905321537 <== osd.0 v2:172.21.15.32:6816/3066156116 45 ==== osd_op_reply(84 605.00000000 [delete] v90'28 uv28 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f2038014070 con 0x556a9df45a60 2024-08-03T20:55:24.493 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:24.492+0000 7f2052a0ef40 1 -- 172.21.15.32:0/2905321537 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:85 12.1d 12:b871830b:::605.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e90) -- 0x556a9dffd360 con 0x556a9df45a60 2024-08-03T20:55:24.495 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:24.493+0000 7f204a400640 1 -- 172.21.15.32:0/2905321537 <== osd.0 v2:172.21.15.32:6816/3066156116 46 ==== osd_op_reply(85 605.00000000 [create] v90'29 uv29 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f2038014070 con 0x556a9df45a60 2024-08-03T20:55:24.495 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::605.00000000:head# 2024-08-03T20:55:24.495 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:24.493+0000 7f2052a0ef40 1 -- 172.21.15.32:0/2905321537 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:86 12.1d 12:b871830b:::605.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e90) -- 0x556a9dffe370 con 0x556a9df45a60 2024-08-03T20:55:24.496 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:24.495+0000 7f204a400640 1 -- 172.21.15.32:0/2905321537 <== osd.0 v2:172.21.15.32:6816/3066156116 47 ==== osd_op_reply(86 605.00000000 [setxattr (30)] v90'30 uv30 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f2038014070 con 0x556a9df45a60 2024-08-03T20:55:24.496 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:24.495+0000 7f2052a0ef40 1 -- 172.21.15.32:0/2905321537 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:87 12.1d 12:b871830b:::605.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e90) -- 0x556a9dffe370 con 0x556a9df45a60 2024-08-03T20:55:24.497 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:24.496+0000 7f204a400640 1 -- 172.21.15.32:0/2905321537 <== osd.0 v2:172.21.15.32:6816/3066156116 48 ==== osd_op_reply(87 605.00000000 [setxattr (62)] v90'31 uv31 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f2038014070 con 0x556a9df45a60 2024-08-03T20:55:24.498 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:24.496+0000 7f2052a0ef40 1 -- 172.21.15.32:0/2905321537 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:88 12.1d 12:b871830b:::605.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e90) -- 0x556a9dffd360 con 0x556a9df45a60 2024-08-03T20:55:24.499 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:24.498+0000 7f204a400640 1 -- 172.21.15.32:0/2905321537 <== osd.0 v2:172.21.15.32:6816/3066156116 49 ==== osd_op_reply(88 605.00000000 [omap-set-header] v90'32 uv32 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f2038014070 con 0x556a9df45a60 2024-08-03T20:55:24.499 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:24.498+0000 7f2052a0ef40 1 -- 172.21.15.32:0/2905321537 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:89 12.13 12:c91d4a1d:::mds0_inotable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e90) -- 0x556a9dffd360 con 0x556a9dff23b0 2024-08-03T20:55:24.500 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:24.499+0000 7f204a400640 1 -- 172.21.15.32:0/2905321537 <== osd.1 v2:172.21.15.32:6824/1551241759 5 ==== osd_op_reply(89 mds0_inotable [create] v90'44 uv38 ondisk = -17 ((17) File exists)) ==== 157+0+0 (crc 0 0 0) 0x7f2038014070 con 0x556a9dff23b0 2024-08-03T20:55:24.500 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:24.499+0000 7f2052a0ef40 1 -- 172.21.15.32:0/2905321537 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:90 12.13 12:c91d4a1d:::mds0_inotable:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e90) -- 0x556a9dffd360 con 0x556a9dff23b0 2024-08-03T20:55:24.502 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:24.500+0000 7f204a400640 1 -- 172.21.15.32:0/2905321537 <== osd.1 v2:172.21.15.32:6824/1551241759 6 ==== osd_op_reply(90 mds0_inotable [delete] v90'45 uv45 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f2038014070 con 0x556a9dff23b0 2024-08-03T20:55:24.502 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:24.500+0000 7f2052a0ef40 1 -- 172.21.15.32:0/2905321537 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:91 12.13 12:c91d4a1d:::mds0_inotable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e90) -- 0x556a9dffd360 con 0x556a9dff23b0 2024-08-03T20:55:24.503 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:24.502+0000 7f204a400640 1 -- 172.21.15.32:0/2905321537 <== osd.1 v2:172.21.15.32:6824/1551241759 7 ==== osd_op_reply(91 mds0_inotable [create] v90'46 uv46 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f2038014070 con 0x556a9dff23b0 2024-08-03T20:55:24.503 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds0_inotable:head# 2024-08-03T20:55:24.503 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:24.502+0000 7f2052a0ef40 1 -- 172.21.15.32:0/2905321537 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:92 12.13 12:c91d4a1d:::mds0_inotable:head [write 0~34 in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e90) -- 0x556a9dffd720 con 0x556a9dff23b0 2024-08-03T20:55:24.504 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:24.503+0000 7f204a400640 1 -- 172.21.15.32:0/2905321537 <== osd.1 v2:172.21.15.32:6824/1551241759 8 ==== osd_op_reply(92 mds0_inotable [write 0~34] v90'47 uv47 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f2038014070 con 0x556a9dff23b0 2024-08-03T20:55:24.504 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:24.503+0000 7f2052a0ef40 1 -- 172.21.15.32:0/2905321537 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:93 12.13 12:c91d4a1d:::mds0_inotable:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e90) -- 0x556a9dffd720 con 0x556a9dff23b0 2024-08-03T20:55:24.505 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:24.504+0000 7f204a400640 1 -- 172.21.15.32:0/2905321537 <== osd.1 v2:172.21.15.32:6824/1551241759 9 ==== osd_op_reply(93 mds0_inotable [omap-set-header] v90'48 uv48 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f2038014070 con 0x556a9dff23b0 2024-08-03T20:55:24.506 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:24.504+0000 7f2052a0ef40 1 -- 172.21.15.32:0/2905321537 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:94 12.13 12:cd5a64a3:::100.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e90) -- 0x556a9dffd720 con 0x556a9dff23b0 2024-08-03T20:55:24.507 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:24.505+0000 7f204a400640 1 -- 172.21.15.32:0/2905321537 <== osd.1 v2:172.21.15.32:6824/1551241759 10 ==== osd_op_reply(94 100.00000000 [create] v90'49 uv43 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f2038014070 con 0x556a9dff23b0 2024-08-03T20:55:24.507 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:24.506+0000 7f2052a0ef40 1 -- 172.21.15.32:0/2905321537 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:95 12.13 12:cd5a64a3:::100.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e90) -- 0x556a9dffd720 con 0x556a9dff23b0 2024-08-03T20:55:24.508 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:24.507+0000 7f204a400640 1 -- 172.21.15.32:0/2905321537 <== osd.1 v2:172.21.15.32:6824/1551241759 11 ==== osd_op_reply(95 100.00000000 [delete] v90'50 uv50 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f2038014070 con 0x556a9dff23b0 2024-08-03T20:55:24.508 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:24.507+0000 7f2052a0ef40 1 -- 172.21.15.32:0/2905321537 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:96 12.13 12:cd5a64a3:::100.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e90) -- 0x556a9dffd720 con 0x556a9dff23b0 2024-08-03T20:55:24.509 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:24.508+0000 7f204a400640 1 -- 172.21.15.32:0/2905321537 <== osd.1 v2:172.21.15.32:6824/1551241759 12 ==== osd_op_reply(96 100.00000000 [create] v90'51 uv51 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f2038014070 con 0x556a9dff23b0 2024-08-03T20:55:24.510 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::100.00000000:head# 2024-08-03T20:55:24.510 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:24.508+0000 7f2052a0ef40 1 -- 172.21.15.32:0/2905321537 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:97 12.13 12:cd5a64a3:::100.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e90) -- 0x556a9dffd720 con 0x556a9dff23b0 2024-08-03T20:55:24.511 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:24.509+0000 7f204a400640 1 -- 172.21.15.32:0/2905321537 <== osd.1 v2:172.21.15.32:6824/1551241759 13 ==== osd_op_reply(97 100.00000000 [omap-set-header] v90'52 uv52 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f2038014070 con 0x556a9dff23b0 2024-08-03T20:55:24.511 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:24.509+0000 7f2052a0ef40 1 -- 172.21.15.32:0/2905321537 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:98 12.13 12:cd5a64a3:::100.00000000:head [omap-set-vals in=5194b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e90) -- 0x556a9e000530 con 0x556a9dff23b0 2024-08-03T20:55:24.512 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:24.511+0000 7f204a400640 1 -- 172.21.15.32:0/2905321537 <== osd.1 v2:172.21.15.32:6824/1551241759 14 ==== osd_op_reply(98 100.00000000 [omap-set-vals] v90'53 uv53 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f2038014070 con 0x556a9dff23b0 2024-08-03T20:55:24.512 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:24.511+0000 7f2052a0ef40 1 --2- 172.21.15.32:0/2905321537 >> [v2:172.21.15.32:6800/9088303,v1:172.21.15.32:6801/9088303] conn(0x556a9dffe0e0 0x556a9e000500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:55:24.513 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:24.511+0000 7f2052a0ef40 1 -- 172.21.15.32:0/2905321537 --> [v2:172.21.15.32:6800/9088303,v1:172.21.15.32:6801/9088303] -- osd_op(unknown.0.0:99 12.b 12:d0630e4c:::mds0_sessionmap:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e90) -- 0x556a9e000a40 con 0x556a9dffe0e0 2024-08-03T20:55:24.513 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:24.511+0000 7f204a400640 1 --2- 172.21.15.32:0/2905321537 >> [v2:172.21.15.32:6800/9088303,v1:172.21.15.32:6801/9088303] conn(0x556a9dffe0e0 0x556a9e000500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:55:24.513 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:24.512+0000 7f204a400640 1 --2- 172.21.15.32:0/2905321537 >> [v2:172.21.15.32:6800/9088303,v1:172.21.15.32:6801/9088303] conn(0x556a9dffe0e0 0x556a9e000500 crc :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:55:24.514 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:24.513+0000 7f204a400640 1 -- 172.21.15.32:0/2905321537 <== osd.3 v2:172.21.15.32:6800/9088303 1 ==== osd_op_reply(99 mds0_sessionmap [create] v90'22 uv21 ondisk = -17 ((17) File exists)) ==== 159+0+0 (crc 0 0 0) 0x7f2038014070 con 0x556a9dffe0e0 2024-08-03T20:55:24.515 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:24.513+0000 7f2052a0ef40 1 -- 172.21.15.32:0/2905321537 --> [v2:172.21.15.32:6800/9088303,v1:172.21.15.32:6801/9088303] -- osd_op(unknown.0.0:100 12.b 12:d0630e4c:::mds0_sessionmap:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e90) -- 0x556a9e000a40 con 0x556a9dffe0e0 2024-08-03T20:55:24.516 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:24.515+0000 7f204a400640 1 -- 172.21.15.32:0/2905321537 <== osd.3 v2:172.21.15.32:6800/9088303 2 ==== osd_op_reply(100 mds0_sessionmap [delete] v90'23 uv23 ondisk = 0) ==== 159+0+0 (crc 0 0 0) 0x7f2038014070 con 0x556a9dffe0e0 2024-08-03T20:55:24.516 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:24.515+0000 7f2052a0ef40 1 -- 172.21.15.32:0/2905321537 --> [v2:172.21.15.32:6800/9088303,v1:172.21.15.32:6801/9088303] -- osd_op(unknown.0.0:101 12.b 12:d0630e4c:::mds0_sessionmap:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e90) -- 0x556a9e000a40 con 0x556a9dffe0e0 2024-08-03T20:55:24.517 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:24.516+0000 7f204a400640 1 -- 172.21.15.32:0/2905321537 <== osd.3 v2:172.21.15.32:6800/9088303 3 ==== osd_op_reply(101 mds0_sessionmap [create] v90'24 uv24 ondisk = 0) ==== 159+0+0 (crc 0 0 0) 0x7f2038014070 con 0x556a9dffe0e0 2024-08-03T20:55:24.517 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds0_sessionmap:head# 2024-08-03T20:55:24.518 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:24.516+0000 7f2052a0ef40 1 -- 172.21.15.32:0/2905321537 --> [v2:172.21.15.32:6800/9088303,v1:172.21.15.32:6801/9088303] -- osd_op(unknown.0.0:102 12.b 12:d0630e4c:::mds0_sessionmap:head [omap-set-header in=14b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e90) -- 0x556a9e000a40 con 0x556a9dffe0e0 2024-08-03T20:55:24.519 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:24.518+0000 7f204a400640 1 -- 172.21.15.32:0/2905321537 <== osd.3 v2:172.21.15.32:6800/9088303 4 ==== osd_op_reply(102 mds0_sessionmap [omap-set-header] v90'25 uv25 ondisk = 0) ==== 159+0+0 (crc 0 0 0) 0x7f2038014070 con 0x556a9dffe0e0 2024-08-03T20:55:24.519 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:24.518+0000 7f2052a0ef40 1 -- 172.21.15.32:0/2905321537 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:103 12.7 12:e0b41d06:::609.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e90) -- 0x556a9e000a40 con 0x556a9dff23b0 2024-08-03T20:55:24.521 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:24.519+0000 7f204a400640 1 -- 172.21.15.32:0/2905321537 <== osd.1 v2:172.21.15.32:6824/1551241759 15 ==== osd_op_reply(103 609.00000000 [create] v90'27 uv26 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f2038014070 con 0x556a9dff23b0 2024-08-03T20:55:24.521 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:24.519+0000 7f2052a0ef40 1 -- 172.21.15.32:0/2905321537 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:104 12.7 12:e0b41d06:::609.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e90) -- 0x556a9e000a40 con 0x556a9dff23b0 2024-08-03T20:55:24.522 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:24.521+0000 7f204a400640 1 -- 172.21.15.32:0/2905321537 <== osd.1 v2:172.21.15.32:6824/1551241759 16 ==== osd_op_reply(104 609.00000000 [delete] v90'28 uv28 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f2038014070 con 0x556a9dff23b0 2024-08-03T20:55:24.522 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:24.521+0000 7f2052a0ef40 1 -- 172.21.15.32:0/2905321537 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:105 12.7 12:e0b41d06:::609.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e90) -- 0x556a9e000a40 con 0x556a9dff23b0 2024-08-03T20:55:24.523 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:24.522+0000 7f204a400640 1 -- 172.21.15.32:0/2905321537 <== osd.1 v2:172.21.15.32:6824/1551241759 17 ==== osd_op_reply(105 609.00000000 [create] v90'29 uv29 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f2038014070 con 0x556a9dff23b0 2024-08-03T20:55:24.524 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::609.00000000:head# 2024-08-03T20:55:24.524 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:24.522+0000 7f2052a0ef40 1 -- 172.21.15.32:0/2905321537 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:106 12.7 12:e0b41d06:::609.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e90) -- 0x556a9e001a50 con 0x556a9dff23b0 2024-08-03T20:55:24.525 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:24.523+0000 7f204a400640 1 -- 172.21.15.32:0/2905321537 <== osd.1 v2:172.21.15.32:6824/1551241759 18 ==== osd_op_reply(106 609.00000000 [setxattr (30)] v90'30 uv30 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f2038014070 con 0x556a9dff23b0 2024-08-03T20:55:24.525 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:24.523+0000 7f2052a0ef40 1 -- 172.21.15.32:0/2905321537 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:107 12.7 12:e0b41d06:::609.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e90) -- 0x556a9e001a50 con 0x556a9dff23b0 2024-08-03T20:55:24.526 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:24.525+0000 7f204a400640 1 -- 172.21.15.32:0/2905321537 <== osd.1 v2:172.21.15.32:6824/1551241759 19 ==== osd_op_reply(107 609.00000000 [setxattr (62)] v90'31 uv31 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f2038014070 con 0x556a9dff23b0 2024-08-03T20:55:24.526 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:24.525+0000 7f2052a0ef40 1 -- 172.21.15.32:0/2905321537 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:108 12.7 12:e0b41d06:::609.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e90) -- 0x556a9e000a40 con 0x556a9dff23b0 2024-08-03T20:55:24.528 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:24.526+0000 7f204a400640 1 -- 172.21.15.32:0/2905321537 <== osd.1 v2:172.21.15.32:6824/1551241759 20 ==== osd_op_reply(108 609.00000000 [omap-set-header] v90'32 uv32 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f2038014070 con 0x556a9dff23b0 2024-08-03T20:55:24.528 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:24.526+0000 7f2052a0ef40 1 -- 172.21.15.32:0/2905321537 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:109 12.f 12:f1f69726:::400.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e90) -- 0x556a9e000a40 con 0x556a9dff5660 2024-08-03T20:55:24.529 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:24.528+0000 7f204b800640 1 -- 172.21.15.32:0/2905321537 <== osd.6 v2:172.21.15.118:6825/801947386 12 ==== osd_op_reply(109 400.00000000 [create] v90'22 uv21 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f204406f070 con 0x556a9dff5660 2024-08-03T20:55:24.529 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:24.528+0000 7f2052a0ef40 1 -- 172.21.15.32:0/2905321537 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:110 12.f 12:f1f69726:::400.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e90) -- 0x556a9e000a40 con 0x556a9dff5660 2024-08-03T20:55:24.531 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:24.530+0000 7f204b800640 1 -- 172.21.15.32:0/2905321537 <== osd.6 v2:172.21.15.118:6825/801947386 13 ==== osd_op_reply(110 400.00000000 [delete] v90'23 uv23 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f204406f070 con 0x556a9dff5660 2024-08-03T20:55:24.531 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:24.530+0000 7f2052a0ef40 1 -- 172.21.15.32:0/2905321537 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:111 12.f 12:f1f69726:::400.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e90) -- 0x556a9e000a40 con 0x556a9dff5660 2024-08-03T20:55:24.533 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::400.00000000:head# 2024-08-03T20:55:24.533 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:24.531+0000 7f204b800640 1 -- 172.21.15.32:0/2905321537 <== osd.6 v2:172.21.15.118:6825/801947386 14 ==== osd_op_reply(111 400.00000000 [create] v90'24 uv24 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f204406f070 con 0x556a9dff5660 2024-08-03T20:55:24.533 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:24.531+0000 7f2052a0ef40 1 -- 172.21.15.32:0/2905321537 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:112 12.f 12:f1f69726:::400.00000000:head [write 0~22 in=22b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e90) -- 0x556a9e000a40 con 0x556a9dff5660 2024-08-03T20:55:24.535 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:24.533+0000 7f204b800640 1 -- 172.21.15.32:0/2905321537 <== osd.6 v2:172.21.15.118:6825/801947386 15 ==== osd_op_reply(112 400.00000000 [write 0~22] v90'25 uv25 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f204406f070 con 0x556a9dff5660 2024-08-03T20:55:24.535 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:24.533+0000 7f2052a0ef40 1 -- 172.21.15.32:0/2905321537 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:113 12.f 12:f1f69726:::400.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e90) -- 0x556a9e000a40 con 0x556a9dff5660 2024-08-03T20:55:24.536 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:24.535+0000 7f204b800640 1 -- 172.21.15.32:0/2905321537 <== osd.6 v2:172.21.15.118:6825/801947386 16 ==== osd_op_reply(113 400.00000000 [omap-set-header] v90'26 uv26 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f204406f070 con 0x556a9dff5660 2024-08-03T20:55:24.536 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:24.535+0000 7f2052a0ef40 1 -- 172.21.15.32:0/2905321537 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:114 12.1f 12:fe07bba1:::100.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e90) -- 0x556a9e000a40 con 0x556a9dff5660 2024-08-03T20:55:24.538 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:24.537+0000 7f204b800640 1 -- 172.21.15.32:0/2905321537 <== osd.6 v2:172.21.15.118:6825/801947386 17 ==== osd_op_reply(114 100.00000000.inode [create] v90'57 uv49 ondisk = -17 ((17) File exists)) ==== 162+0+0 (crc 0 0 0) 0x7f204406f070 con 0x556a9dff5660 2024-08-03T20:55:24.538 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:24.537+0000 7f2052a0ef40 1 -- 172.21.15.32:0/2905321537 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:115 12.1f 12:fe07bba1:::100.00000000.inode:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e90) -- 0x556a9e000a40 con 0x556a9dff5660 2024-08-03T20:55:24.539 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:24.538+0000 7f204b800640 1 -- 172.21.15.32:0/2905321537 <== osd.6 v2:172.21.15.118:6825/801947386 18 ==== osd_op_reply(115 100.00000000.inode [delete] v90'58 uv58 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7f204406f070 con 0x556a9dff5660 2024-08-03T20:55:24.539 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:24.538+0000 7f2052a0ef40 1 -- 172.21.15.32:0/2905321537 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:116 12.1f 12:fe07bba1:::100.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e90) -- 0x556a9e000a40 con 0x556a9dff5660 2024-08-03T20:55:24.541 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:24.540+0000 7f204b800640 1 -- 172.21.15.32:0/2905321537 <== osd.6 v2:172.21.15.118:6825/801947386 19 ==== osd_op_reply(116 100.00000000.inode [create] v90'59 uv59 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7f204406f070 con 0x556a9dff5660 2024-08-03T20:55:24.541 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::100.00000000.inode:head# 2024-08-03T20:55:24.541 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:24.540+0000 7f2052a0ef40 1 -- 172.21.15.32:0/2905321537 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:117 12.1f 12:fe07bba1:::100.00000000.inode:head [write 0~582 in=582b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e90) -- 0x556a9e000a40 con 0x556a9dff5660 2024-08-03T20:55:24.542 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:24.541+0000 7f204b800640 1 -- 172.21.15.32:0/2905321537 <== osd.6 v2:172.21.15.118:6825/801947386 20 ==== osd_op_reply(117 100.00000000.inode [write 0~582] v90'60 uv60 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7f204406f070 con 0x556a9dff5660 2024-08-03T20:55:24.543 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:24.541+0000 7f2052a0ef40 1 -- 172.21.15.32:0/2905321537 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:118 12.1f 12:fe07bba1:::100.00000000.inode:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e90) -- 0x556a9e000a40 con 0x556a9dff5660 2024-08-03T20:55:24.544 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:24.542+0000 7f204b800640 1 -- 172.21.15.32:0/2905321537 <== osd.6 v2:172.21.15.118:6825/801947386 21 ==== osd_op_reply(118 100.00000000.inode [omap-set-header] v90'61 uv61 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7f204406f070 con 0x556a9dff5660 2024-08-03T20:55:24.544 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:24.542+0000 7f2052a0ef40 1 -- 172.21.15.32:0/2905321537 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:119 12.1f 12:ff5b34d6:::1.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e90) -- 0x556a9e000a40 con 0x556a9dff5660 2024-08-03T20:55:24.545 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:24.544+0000 7f204b800640 1 -- 172.21.15.32:0/2905321537 <== osd.6 v2:172.21.15.118:6825/801947386 22 ==== osd_op_reply(119 1.00000000 [create] v90'62 uv56 ondisk = -17 ((17) File exists)) ==== 154+0+0 (crc 0 0 0) 0x7f204406f070 con 0x556a9dff5660 2024-08-03T20:55:24.545 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:24.544+0000 7f2052a0ef40 1 -- 172.21.15.32:0/2905321537 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:120 12.1f 12:ff5b34d6:::1.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e90) -- 0x556a9e000a40 con 0x556a9dff5660 2024-08-03T20:55:24.547 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:24.545+0000 7f204b800640 1 -- 172.21.15.32:0/2905321537 <== osd.6 v2:172.21.15.118:6825/801947386 23 ==== osd_op_reply(120 1.00000000 [delete] v90'63 uv63 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7f204406f070 con 0x556a9dff5660 2024-08-03T20:55:24.547 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:24.545+0000 7f2052a0ef40 1 -- 172.21.15.32:0/2905321537 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:121 12.1f 12:ff5b34d6:::1.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e90) -- 0x556a9e000a40 con 0x556a9dff5660 2024-08-03T20:55:24.548 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:24.547+0000 7f204b800640 1 -- 172.21.15.32:0/2905321537 <== osd.6 v2:172.21.15.118:6825/801947386 24 ==== osd_op_reply(121 1.00000000 [create] v90'64 uv64 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7f204406f070 con 0x556a9dff5660 2024-08-03T20:55:24.548 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::1.00000000:head# 2024-08-03T20:55:24.549 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:24.547+0000 7f2052a0ef40 1 -- 172.21.15.32:0/2905321537 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:122 12.1f 12:ff5b34d6:::1.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e90) -- 0x556a9e000a40 con 0x556a9dff5660 2024-08-03T20:55:24.550 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:24.548+0000 7f204b800640 1 -- 172.21.15.32:0/2905321537 <== osd.6 v2:172.21.15.118:6825/801947386 25 ==== osd_op_reply(122 1.00000000 [setxattr (30)] v90'65 uv65 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7f204406f070 con 0x556a9dff5660 2024-08-03T20:55:24.550 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:24.548+0000 7f2052a0ef40 1 -- 172.21.15.32:0/2905321537 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:123 12.1f 12:ff5b34d6:::1.00000000:head [setxattr parent (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e90) -- 0x556a9e001a50 con 0x556a9dff5660 2024-08-03T20:55:24.551 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:24.550+0000 7f204b800640 1 -- 172.21.15.32:0/2905321537 <== osd.6 v2:172.21.15.118:6825/801947386 26 ==== osd_op_reply(123 1.00000000 [setxattr (30)] v90'66 uv66 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7f204406f070 con 0x556a9dff5660 2024-08-03T20:55:24.551 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:24.550+0000 7f2052a0ef40 1 -- 172.21.15.32:0/2905321537 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:124 12.1f 12:ff5b34d6:::1.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e90) -- 0x556a9e000a40 con 0x556a9dff5660 2024-08-03T20:55:24.553 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:24.551+0000 7f204b800640 1 -- 172.21.15.32:0/2905321537 <== osd.6 v2:172.21.15.118:6825/801947386 27 ==== osd_op_reply(124 1.00000000 [omap-set-header] v90'67 uv67 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7f204406f070 con 0x556a9dff5660 2024-08-03T20:55:24.553 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:24.552+0000 7f2052a0ef40 1 -- 172.21.15.32:0/2905321537 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:125 12.1f 12:ff5b34d6:::1.00000000:head [omap-set-vals in=584b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e90) -- 0x556a9e0011a0 con 0x556a9dff5660 2024-08-03T20:55:24.555 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:24.553+0000 7f204b800640 1 -- 172.21.15.32:0/2905321537 <== osd.6 v2:172.21.15.118:6825/801947386 28 ==== osd_op_reply(125 1.00000000 [omap-set-vals] v90'68 uv68 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7f204406f070 con 0x556a9dff5660 2024-08-03T20:55:24.556 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:24.553+0000 7f2052a0ef40 1 -- 172.21.15.32:0/2905321537 >> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] conn(0x556a9df45a60 msgr2=0x556a9df21d60 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:55:24.556 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:24.553+0000 7f2052a0ef40 1 --2- 172.21.15.32:0/2905321537 >> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] conn(0x556a9df45a60 0x556a9df21d60 crc :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:24.556 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:24.553+0000 7f2052a0ef40 1 -- 172.21.15.32:0/2905321537 >> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] conn(0x556a9dff23b0 msgr2=0x556a9dff47d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:55:24.556 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:24.553+0000 7f2052a0ef40 1 --2- 172.21.15.32:0/2905321537 >> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] conn(0x556a9dff23b0 0x556a9dff47d0 crc :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:24.556 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:24.553+0000 7f2052a0ef40 1 -- 172.21.15.32:0/2905321537 >> [v2:172.21.15.32:6800/9088303,v1:172.21.15.32:6801/9088303] conn(0x556a9dffe0e0 msgr2=0x556a9e000500 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:55:24.556 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:24.553+0000 7f2052a0ef40 1 --2- 172.21.15.32:0/2905321537 >> [v2:172.21.15.32:6800/9088303,v1:172.21.15.32:6801/9088303] conn(0x556a9dffe0e0 0x556a9e000500 crc :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:24.556 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:24.553+0000 7f2052a0ef40 1 -- 172.21.15.32:0/2905321537 >> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] conn(0x556a9dfed7f0 msgr2=0x556a9dfefc10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:55:24.556 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:24.553+0000 7f204b800640 1 -- 172.21.15.32:0/2905321537 reap_dead start 2024-08-03T20:55:24.556 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:24.553+0000 7f2052a0ef40 1 --2- 172.21.15.32:0/2905321537 >> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] conn(0x556a9dfed7f0 0x556a9dfefc10 crc :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:24.556 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:24.553+0000 7f2052a0ef40 1 -- 172.21.15.32:0/2905321537 >> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] conn(0x556a9dff5660 msgr2=0x556a9dff7a80 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:55:24.556 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:24.553+0000 7f2052a0ef40 1 --2- 172.21.15.32:0/2905321537 >> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] conn(0x556a9dff5660 0x556a9dff7a80 crc :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:24.556 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:24.553+0000 7f2052a0ef40 1 -- 172.21.15.32:0/2905321537 >> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] conn(0x556a9dffaa00 msgr2=0x556a9dffce20 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:55:24.557 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:24.553+0000 7f2052a0ef40 1 --2- 172.21.15.32:0/2905321537 >> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] conn(0x556a9dffaa00 0x556a9dffce20 crc :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:24.557 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:24.553+0000 7f2052a0ef40 1 -- 172.21.15.32:0/2905321537 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f202c0759f0 msgr2=0x7f202c077f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:55:24.557 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:24.553+0000 7f2052a0ef40 1 --2- 172.21.15.32:0/2905321537 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f202c0759f0 0x7f202c077f00 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7f203c0040b0 tx=0x7f203c004b50 comp rx=0 tx=0).stop 2024-08-03T20:55:24.557 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:24.553+0000 7f2052a0ef40 1 -- 172.21.15.32:0/2905321537 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x556a9df42860 msgr2=0x556a9de6c3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:55:24.557 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:24.553+0000 7f2052a0ef40 1 --2- 172.21.15.32:0/2905321537 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x556a9df42860 0x556a9de6c3e0 secure :-1 s=READY pgs=1185 cs=0 l=1 rev1=1 crypto rx=0x7f2044025d70 tx=0x7f204400a5c0 comp rx=0 tx=0).stop 2024-08-03T20:55:24.557 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:24.553+0000 7f204b800640 1 -- 172.21.15.32:0/2905321537 reap_dead start 2024-08-03T20:55:24.557 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:24.553+0000 7f2052a0ef40 1 -- 172.21.15.32:0/2905321537 shutdown_connections 2024-08-03T20:55:24.557 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:24.553+0000 7f2052a0ef40 1 -- 172.21.15.32:0/2905321537 >> 172.21.15.32:0/2905321537 conn(0x556a9dd8eac0 msgr2=0x556a9de6e5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:55:24.557 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:24.554+0000 7f2052a0ef40 1 -- 172.21.15.32:0/2905321537 shutdown_connections 2024-08-03T20:55:24.557 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:24.554+0000 7f2052a0ef40 1 -- 172.21.15.32:0/2905321537 wait complete. 2024-08-03T20:55:24.569 DEBUG:teuthology.orchestra.run.smithi032:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata rm 200.00000001 2024-08-03T20:55:24.752 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:24.750+0000 7f86bed6df40 1 Processor -- start 2024-08-03T20:55:24.752 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:24.751+0000 7f86bed6df40 1 -- start start 2024-08-03T20:55:24.755 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:24.752+0000 7f86bed6df40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55efd160d360 0x55efd16268f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:55:24.756 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:24.752+0000 7f86bed6df40 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55efd160eea0 0x55efd162bf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:55:24.756 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:24.752+0000 7f86bed6df40 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55efd162acb0 0x55efd1627c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:55:24.756 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:24.752+0000 7f86bed6df40 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55efd14a4e30 con 0x55efd162acb0 2024-08-03T20:55:24.756 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:24.752+0000 7f86bed6df40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x55efd1502bf0 con 0x55efd160d360 2024-08-03T20:55:24.756 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:24.752+0000 7f86bed6df40 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x55efd162b0c0 con 0x55efd160eea0 2024-08-03T20:55:24.756 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:24.752+0000 7f86b8800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55efd162acb0 0x55efd1627c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:55:24.756 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:24.752+0000 7f86b8800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55efd162acb0 0x55efd1627c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:53350/0 (socket says 172.21.15.32:53350) 2024-08-03T20:55:24.756 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:24.752+0000 7f86b8800640 1 -- 172.21.15.32:0/336084727 learned_addr learned my addr 172.21.15.32:0/336084727 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:55:24.757 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:24.752+0000 7f86b7e00640 1 --2- 172.21.15.32:0/336084727 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55efd160d360 0x55efd16268f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:55:24.757 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:24.752+0000 7f86b7400640 1 --2- 172.21.15.32:0/336084727 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55efd160eea0 0x55efd162bf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:55:24.757 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:24.753+0000 7f86b8800640 1 -- 172.21.15.32:0/336084727 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55efd160eea0 msgr2=0x55efd162bf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:55:24.757 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:24.753+0000 7f86b8800640 1 --2- 172.21.15.32:0/336084727 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55efd160eea0 0x55efd162bf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:24.757 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:24.753+0000 7f86b8800640 1 -- 172.21.15.32:0/336084727 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55efd160d360 msgr2=0x55efd16268f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:55:24.757 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:24.753+0000 7f86b8800640 1 --2- 172.21.15.32:0/336084727 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55efd160d360 0x55efd16268f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:24.757 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:24.753+0000 7f86b8800640 1 -- 172.21.15.32:0/336084727 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55efd1625e00 con 0x55efd162acb0 2024-08-03T20:55:24.757 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:24.753+0000 7f86b8800640 1 --2- 172.21.15.32:0/336084727 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55efd162acb0 0x55efd1627c40 secure :-1 s=READY pgs=1188 cs=0 l=1 rev1=1 crypto rx=0x7f86b0001560 tx=0x7f86b0001a20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c18986abd1ae0715 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:55:24.757 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:24.753+0000 7f86b6000640 1 -- 172.21.15.32:0/336084727 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f86b0016070 con 0x55efd162acb0 2024-08-03T20:55:24.757 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:24.753+0000 7f86b6000640 1 -- 172.21.15.32:0/336084727 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f86b000f800 con 0x55efd162acb0 2024-08-03T20:55:24.758 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:24.753+0000 7f86b6000640 1 -- 172.21.15.32:0/336084727 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f86b000fbf0 con 0x55efd162acb0 2024-08-03T20:55:24.758 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:24.754+0000 7f86bed6df40 1 -- 172.21.15.32:0/336084727 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55efd162acb0 msgr2=0x55efd1627c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:55:24.758 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:24.754+0000 7f86bed6df40 1 --2- 172.21.15.32:0/336084727 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55efd162acb0 0x55efd1627c40 secure :-1 s=READY pgs=1188 cs=0 l=1 rev1=1 crypto rx=0x7f86b0001560 tx=0x7f86b0001a20 comp rx=0 tx=0).stop 2024-08-03T20:55:24.758 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:24.754+0000 7f86bed6df40 1 -- 172.21.15.32:0/336084727 shutdown_connections 2024-08-03T20:55:24.758 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:24.754+0000 7f86bed6df40 1 --2- 172.21.15.32:0/336084727 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55efd162acb0 0x55efd1627c40 unknown :-1 s=CLOSED pgs=1188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:24.758 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:24.754+0000 7f86bed6df40 1 --2- 172.21.15.32:0/336084727 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55efd160eea0 0x55efd162bf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:24.758 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:24.754+0000 7f86bed6df40 1 --2- 172.21.15.32:0/336084727 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55efd160d360 0x55efd16268f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:24.758 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:24.754+0000 7f86bed6df40 1 -- 172.21.15.32:0/336084727 >> 172.21.15.32:0/336084727 conn(0x55efd1476ac0 msgr2=0x55efd162cf20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:55:24.758 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:24.754+0000 7f86bed6df40 1 -- 172.21.15.32:0/336084727 shutdown_connections 2024-08-03T20:55:24.759 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:24.754+0000 7f86bed6df40 1 -- 172.21.15.32:0/336084727 wait complete. 2024-08-03T20:55:24.761 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:24.755+0000 7f86bed6df40 1 Processor -- start 2024-08-03T20:55:24.761 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:24.755+0000 7f86bed6df40 1 -- start start 2024-08-03T20:55:24.761 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:24.756+0000 7f86bed6df40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55efd160d360 0x55efd15545d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:55:24.761 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:24.756+0000 7f86bed6df40 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55efd160eea0 0x55efd159e050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:55:24.761 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:24.756+0000 7f86bed6df40 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55efd162acb0 0x55efd159e590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:55:24.761 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:24.756+0000 7f86bed6df40 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55efd1627880 con 0x55efd162acb0 2024-08-03T20:55:24.762 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:24.756+0000 7f86bed6df40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x55efd159f920 con 0x55efd160d360 2024-08-03T20:55:24.762 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:24.756+0000 7f86bed6df40 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x55efd16274a0 con 0x55efd160eea0 2024-08-03T20:55:24.762 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:24.757+0000 7f86b8800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55efd162acb0 0x55efd159e590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:55:24.762 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:24.757+0000 7f86b8800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55efd162acb0 0x55efd159e590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:53356/0 (socket says 172.21.15.32:53356) 2024-08-03T20:55:24.762 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:24.757+0000 7f86b8800640 1 -- 172.21.15.32:0/174959275 learned_addr learned my addr 172.21.15.32:0/174959275 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:55:24.762 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:24.757+0000 7f86b7e00640 1 --2- 172.21.15.32:0/174959275 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55efd160d360 0x55efd15545d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:55:24.763 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:24.757+0000 7f86b7400640 1 --2- 172.21.15.32:0/174959275 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55efd160eea0 0x55efd159e050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:55:24.763 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:24.757+0000 7f86b8800640 1 -- 172.21.15.32:0/174959275 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55efd160eea0 msgr2=0x55efd159e050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:55:24.763 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:24.757+0000 7f86b8800640 1 --2- 172.21.15.32:0/174959275 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55efd160eea0 0x55efd159e050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:24.763 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:24.757+0000 7f86b8800640 1 -- 172.21.15.32:0/174959275 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55efd160d360 msgr2=0x55efd15545d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:55:24.763 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:24.757+0000 7f86b8800640 1 --2- 172.21.15.32:0/174959275 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55efd160d360 0x55efd15545d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:24.764 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:24.757+0000 7f86b8800640 1 -- 172.21.15.32:0/174959275 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55efd159bcd0 con 0x55efd162acb0 2024-08-03T20:55:24.764 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:24.757+0000 7f86b7400640 1 --2- 172.21.15.32:0/174959275 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55efd160eea0 0x55efd159e050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-03T20:55:24.764 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:24.757+0000 7f86b8800640 1 --2- 172.21.15.32:0/174959275 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55efd162acb0 0x55efd159e590 secure :-1 s=READY pgs=1189 cs=0 l=1 rev1=1 crypto rx=0x7f86b0014040 tx=0x7f86b0006d60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:55:24.764 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:24.758+0000 7f86b4c00640 1 -- 172.21.15.32:0/174959275 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f86b0021020 con 0x55efd162acb0 2024-08-03T20:55:24.764 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:24.758+0000 7f86b4c00640 1 -- 172.21.15.32:0/174959275 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f86b001b680 con 0x55efd162acb0 2024-08-03T20:55:24.764 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:24.758+0000 7f86b4c00640 1 -- 172.21.15.32:0/174959275 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f86b0016040 con 0x55efd162acb0 2024-08-03T20:55:24.765 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:24.758+0000 7f86bed6df40 1 -- 172.21.15.32:0/174959275 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x55efd159bfb0 con 0x55efd162acb0 2024-08-03T20:55:24.765 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:24.758+0000 7f86bed6df40 1 -- 172.21.15.32:0/174959275 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55efd162d390 con 0x55efd162acb0 2024-08-03T20:55:24.768 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:24.760+0000 7f86b4c00640 1 -- 172.21.15.32:0/174959275 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f86b0024020 con 0x55efd162acb0 2024-08-03T20:55:24.768 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:24.761+0000 7f86b4c00640 1 --2- 172.21.15.32:0/174959275 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f86980759f0 0x7f8698077f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:55:24.768 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:24.761+0000 7f86b4c00640 1 -- 172.21.15.32:0/174959275 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 6284+0+0 (secure 0 0 0) 0x7f86b0098d50 con 0x55efd162acb0 2024-08-03T20:55:24.768 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:24.761+0000 7f86b7e00640 1 --2- 172.21.15.32:0/174959275 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f86980759f0 0x7f8698077f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:55:24.768 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:24.761+0000 7f86bed6df40 1 --2- 172.21.15.32:0/174959275 >> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] conn(0x55efd1608520 0x55efd16c7230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:55:24.769 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:24.761+0000 7f86bed6df40 1 -- 172.21.15.32:0/174959275 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:1 12.14 12:2e2fa760:::200.00000001:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e90) -- 0x55efd1554b10 con 0x55efd1608520 2024-08-03T20:55:24.769 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:24.762+0000 7f86b7400640 1 --2- 172.21.15.32:0/174959275 >> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] conn(0x55efd1608520 0x55efd16c7230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:55:24.769 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:24.762+0000 7f86b7e00640 1 --2- 172.21.15.32:0/174959275 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f86980759f0 0x7f8698077f00 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7f86a8004fd0 tx=0x7f86a800e040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:55:24.769 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:24.762+0000 7f86b7400640 1 --2- 172.21.15.32:0/174959275 >> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] conn(0x55efd1608520 0x55efd16c7230 crc :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:55:24.769 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:24.764+0000 7f86b7400640 1 -- 172.21.15.32:0/174959275 <== osd.5 v2:172.21.15.118:6810/2724891770 1 ==== osd_op_reply(1 200.00000001 [delete] v90'128 uv128 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f86a404eb40 con 0x55efd1608520 2024-08-03T20:55:24.769 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:24.764+0000 7f86bed6df40 1 -- 172.21.15.32:0/174959275 >> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] conn(0x55efd1608520 msgr2=0x55efd16c7230 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:55:24.769 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:24.764+0000 7f86bed6df40 1 --2- 172.21.15.32:0/174959275 >> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] conn(0x55efd1608520 0x55efd16c7230 crc :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:24.769 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:24.764+0000 7f86bed6df40 1 -- 172.21.15.32:0/174959275 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f86980759f0 msgr2=0x7f8698077f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:55:24.769 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:24.764+0000 7f86bed6df40 1 --2- 172.21.15.32:0/174959275 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f86980759f0 0x7f8698077f00 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7f86a8004fd0 tx=0x7f86a800e040 comp rx=0 tx=0).stop 2024-08-03T20:55:24.769 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:24.764+0000 7f86bed6df40 1 -- 172.21.15.32:0/174959275 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55efd162acb0 msgr2=0x55efd159e590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:55:24.769 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:24.764+0000 7f86bed6df40 1 --2- 172.21.15.32:0/174959275 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55efd162acb0 0x55efd159e590 secure :-1 s=READY pgs=1189 cs=0 l=1 rev1=1 crypto rx=0x7f86b0014040 tx=0x7f86b0006d60 comp rx=0 tx=0).stop 2024-08-03T20:55:24.769 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:24.764+0000 7f86b8800640 1 -- 172.21.15.32:0/174959275 reap_dead start 2024-08-03T20:55:24.769 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:24.764+0000 7f86bed6df40 1 -- 172.21.15.32:0/174959275 shutdown_connections 2024-08-03T20:55:24.770 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:24.764+0000 7f86bed6df40 1 -- 172.21.15.32:0/174959275 >> 172.21.15.32:0/174959275 conn(0x55efd1476ac0 msgr2=0x55efd15565a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:55:24.770 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:24.765+0000 7f86bed6df40 1 -- 172.21.15.32:0/174959275 shutdown_connections 2024-08-03T20:55:24.770 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:24.765+0000 7f86bed6df40 1 -- 172.21.15.32:0/174959275 wait complete. 2024-08-03T20:55:24.773 INFO:tasks.ceph.mds.a.smithi032.stderr: -35> 2024-08-03T20:55:20.019+0000 7f9932000640 -1 log_channel(cluster) log [ERR] : Error recovering journal 0x200: (2) No such file or directory 2024-08-03T20:55:24.777 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs joinable true 2024-08-03T20:55:24.887 INFO:tasks.ceph.mds.a.smithi032.stdout:starting mds.a at 2024-08-03T20:55:25.012 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:25.010+0000 7f0b1ee00640 1 -- 172.21.15.32:0/2097678686 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f0b180cf070 msgr2=0x7f0b180cf480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:55:25.012 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:25.010+0000 7f0b1ee00640 1 --2- 172.21.15.32:0/2097678686 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f0b180cf070 0x7f0b180cf480 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f0b10005fd0 tx=0x7f0b1002a9a0 comp rx=0 tx=0).stop 2024-08-03T20:55:25.012 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:25.010+0000 7f0b1ee00640 1 -- 172.21.15.32:0/2097678686 shutdown_connections 2024-08-03T20:55:25.012 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:25.010+0000 7f0b1ee00640 1 --2- 172.21.15.32:0/2097678686 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f0b180cf070 0x7f0b180cf480 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:25.012 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:25.010+0000 7f0b1ee00640 1 --2- 172.21.15.32:0/2097678686 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f0b180d47a0 0x7f0b180ceb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:25.012 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:25.010+0000 7f0b1ee00640 1 -- 172.21.15.32:0/2097678686 >> 172.21.15.32:0/2097678686 conn(0x7f0b1808d7e0 msgr2=0x7f0b1809d410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:55:25.013 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:25.010+0000 7f0b1ee00640 1 -- 172.21.15.32:0/2097678686 shutdown_connections 2024-08-03T20:55:25.013 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:25.010+0000 7f0b1ee00640 1 -- 172.21.15.32:0/2097678686 wait complete. 2024-08-03T20:55:25.014 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:25.012+0000 7f0b1ee00640 1 Processor -- start 2024-08-03T20:55:25.014 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:25.013+0000 7f0b1ee00640 1 -- start start 2024-08-03T20:55:25.019 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:25.014+0000 7f0b1ee00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0b180cf070 0x7f0b1806e000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:55:25.019 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:25.014+0000 7f0b1ee00640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f0b180d26e0 0x7f0b18157120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:55:25.019 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:25.014+0000 7f0b1ee00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f0b18160aa0 0x7f0b181f8e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:55:25.020 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:25.014+0000 7f0b1ee00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f0b180cfb40 con 0x7f0b180cf070 2024-08-03T20:55:25.020 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:25.014+0000 7f0b1ee00640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f0b180cf9c0 con 0x7f0b18160aa0 2024-08-03T20:55:25.020 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:25.014+0000 7f0b1ee00640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7f0b180cfcc0 con 0x7f0b180d26e0 2024-08-03T20:55:25.020 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:25.014+0000 7f0b1da00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0b180cf070 0x7f0b1806e000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:55:25.020 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:25.014+0000 7f0b1da00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0b180cf070 0x7f0b1806e000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:53374/0 (socket says 172.21.15.32:53374) 2024-08-03T20:55:25.020 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:25.014+0000 7f0b1da00640 1 -- 172.21.15.32:0/2330074287 learned_addr learned my addr 172.21.15.32:0/2330074287 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:55:25.021 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:25.014+0000 7f0b1d000640 1 --2- 172.21.15.32:0/2330074287 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f0b180d26e0 0x7f0b18157120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:55:25.021 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:25.014+0000 7f0b1e400640 1 --2- 172.21.15.32:0/2330074287 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f0b18160aa0 0x7f0b181f8e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:55:25.021 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:25.015+0000 7f0b1da00640 1 -- 172.21.15.32:0/2330074287 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f0b180d26e0 msgr2=0x7f0b18157120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:55:25.021 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:25.015+0000 7f0b1da00640 1 --2- 172.21.15.32:0/2330074287 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f0b180d26e0 0x7f0b18157120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:25.021 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:25.015+0000 7f0b1da00640 1 -- 172.21.15.32:0/2330074287 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f0b18160aa0 msgr2=0x7f0b181f8e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:55:25.022 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:25.015+0000 7f0b1da00640 1 --2- 172.21.15.32:0/2330074287 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f0b18160aa0 0x7f0b181f8e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:25.022 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:25.015+0000 7f0b1da00640 1 -- 172.21.15.32:0/2330074287 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0b1809dd80 con 0x7f0b180cf070 2024-08-03T20:55:25.022 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:25.015+0000 7f0b1da00640 1 --2- 172.21.15.32:0/2330074287 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0b180cf070 0x7f0b1806e000 secure :-1 s=READY pgs=1193 cs=0 l=1 rev1=1 crypto rx=0x7f0b0c00c520 tx=0x7f0b0c00c9e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:55:25.022 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:25.015+0000 7f0b15600640 1 -- 172.21.15.32:0/2330074287 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b0c017070 con 0x7f0b180cf070 2024-08-03T20:55:25.022 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:25.015+0000 7f0b15600640 1 -- 172.21.15.32:0/2330074287 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f0b0c0035f0 con 0x7f0b180cf070 2024-08-03T20:55:25.022 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:25.015+0000 7f0b1ee00640 1 -- 172.21.15.32:0/2330074287 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0b181f9c30 con 0x7f0b180cf070 2024-08-03T20:55:25.023 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:25.015+0000 7f0b15600640 1 -- 172.21.15.32:0/2330074287 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b0c003a00 con 0x7f0b180cf070 2024-08-03T20:55:25.023 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:25.016+0000 7f0b1ee00640 1 -- 172.21.15.32:0/2330074287 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0b1809e320 con 0x7f0b180cf070 2024-08-03T20:55:25.023 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:25.017+0000 7f0b1ee00640 1 -- 172.21.15.32:0/2330074287 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0b18076d80 con 0x7f0b180cf070 2024-08-03T20:55:25.024 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:25.018+0000 7f0b15600640 1 -- 172.21.15.32:0/2330074287 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f0b0c01a070 con 0x7f0b180cf070 2024-08-03T20:55:25.024 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:25.019+0000 7f0b15600640 1 --2- 172.21.15.32:0/2330074287 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f0af8077040 0x7f0af8079550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:55:25.025 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:25.019+0000 7f0b15600640 1 -- 172.21.15.32:0/2330074287 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 6284+0+0 (secure 0 0 0) 0x7f0b0c097f80 con 0x7f0b180cf070 2024-08-03T20:55:25.025 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:25.020+0000 7f0b1d000640 1 --2- 172.21.15.32:0/2330074287 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f0af8077040 0x7f0af8079550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:55:25.025 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:25.020+0000 7f0b1d000640 1 --2- 172.21.15.32:0/2330074287 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f0af8077040 0x7f0af8079550 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7f0b180d0820 tx=0x7f0b04005f90 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:55:25.025 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:25.024+0000 7f0b15600640 1 -- 172.21.15.32:0/2330074287 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7f0b18076d80 con 0x7f0b180cf070 2024-08-03T20:55:25.280 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:25.278+0000 7f0b1ee00640 1 -- 172.21.15.32:0/2330074287 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "joinable", "val": "true"} v 0) -- 0x7f0b180d5250 con 0x7f0b180cf070 2024-08-03T20:55:26.195 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:26.193+0000 7f0b15600640 1 -- 172.21.15.32:0/2330074287 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "joinable", "val": "true"}]=0 cephfs marked joinable; MDS may join as newly active. v121) ==== 162+0+0 (secure 0 0 0) 0x7f0b180d5250 con 0x7f0b180cf070 2024-08-03T20:55:26.195 INFO:teuthology.orchestra.run.smithi032.stderr:cephfs marked joinable; MDS may join as newly active. 2024-08-03T20:55:26.199 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:26.197+0000 7f0b1ee00640 1 -- 172.21.15.32:0/2330074287 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f0af8077040 msgr2=0x7f0af8079550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:55:26.200 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:26.197+0000 7f0b1ee00640 1 --2- 172.21.15.32:0/2330074287 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f0af8077040 0x7f0af8079550 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7f0b180d0820 tx=0x7f0b04005f90 comp rx=0 tx=0).stop 2024-08-03T20:55:26.200 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:26.197+0000 7f0b1ee00640 1 -- 172.21.15.32:0/2330074287 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0b180cf070 msgr2=0x7f0b1806e000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:55:26.200 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:26.197+0000 7f0b1ee00640 1 --2- 172.21.15.32:0/2330074287 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0b180cf070 0x7f0b1806e000 secure :-1 s=READY pgs=1193 cs=0 l=1 rev1=1 crypto rx=0x7f0b0c00c520 tx=0x7f0b0c00c9e0 comp rx=0 tx=0).stop 2024-08-03T20:55:26.200 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:26.198+0000 7f0b1ee00640 1 -- 172.21.15.32:0/2330074287 shutdown_connections 2024-08-03T20:55:26.200 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:26.198+0000 7f0b1ee00640 1 --2- 172.21.15.32:0/2330074287 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f0af8077040 0x7f0af8079550 unknown :-1 s=CLOSED pgs=493 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:26.200 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:26.198+0000 7f0b1ee00640 1 --2- 172.21.15.32:0/2330074287 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f0b18160aa0 0x7f0b181f8e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:26.200 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:26.198+0000 7f0b1ee00640 1 --2- 172.21.15.32:0/2330074287 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f0b180d26e0 0x7f0b18157120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:26.200 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:26.198+0000 7f0b1ee00640 1 --2- 172.21.15.32:0/2330074287 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0b180cf070 0x7f0b1806e000 unknown :-1 s=CLOSED pgs=1193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:26.200 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:26.198+0000 7f0b1ee00640 1 -- 172.21.15.32:0/2330074287 >> 172.21.15.32:0/2330074287 conn(0x7f0b1808d7e0 msgr2=0x7f0b180d4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:55:26.200 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:26.198+0000 7f0b1ee00640 1 -- 172.21.15.32:0/2330074287 shutdown_connections 2024-08-03T20:55:26.200 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:26.198+0000 7f0b1ee00640 1 -- 172.21.15.32:0/2330074287 wait complete. 2024-08-03T20:55:26.214 INFO:tasks.ceph.mds.a.smithi032.stderr:2024-08-03T20:55:26.212+0000 7f0bd3800640 -1 log_channel(cluster) log [ERR] : missing journal object 2024-08-03T20:55:26.214 INFO:tasks.ceph.mon.a.smithi032.stderr:2024-08-03T20:55:26.213+0000 7f4e6d600640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-08-03T20:55:26.215 INFO:tasks.ceph.mon.a.smithi032.stderr:2024-08-03T20:55:26.213+0000 7f4e6d600640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 mds daemon damaged (MDS_DAMAGE) 2024-08-03T20:55:26.230 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-08-03T20:55:26.461 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:26.459+0000 7f029aa00640 1 -- 172.21.15.32:0/1025168287 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f029406f440 msgr2=0x7f029406f850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:55:26.461 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:26.459+0000 7f029aa00640 1 --2- 172.21.15.32:0/1025168287 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f029406f440 0x7f029406f850 secure :-1 s=READY pgs=1194 cs=0 l=1 rev1=1 crypto rx=0x7f02940720a0 tx=0x7f028c02a980 comp rx=0 tx=0).stop 2024-08-03T20:55:26.462 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:26.459+0000 7f029aa00640 1 -- 172.21.15.32:0/1025168287 shutdown_connections 2024-08-03T20:55:26.462 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:26.459+0000 7f029aa00640 1 --2- 172.21.15.32:0/1025168287 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f029406f440 0x7f029406f850 unknown :-1 s=CLOSED pgs=1194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:26.462 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:26.459+0000 7f029aa00640 1 --2- 172.21.15.32:0/1025168287 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f02940acd80 0x7f02940ad190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:26.462 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:26.459+0000 7f029aa00640 1 --2- 172.21.15.32:0/1025168287 >> v2:172.21.15.118:3300/0 conn(0x7f02940af020 0x7f02940742b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:26.462 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:26.459+0000 7f029aa00640 1 -- 172.21.15.32:0/1025168287 >> 172.21.15.32:0/1025168287 conn(0x7f02940b2120 msgr2=0x7f02940cfd60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:55:26.462 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:26.460+0000 7f029aa00640 1 -- 172.21.15.32:0/1025168287 shutdown_connections 2024-08-03T20:55:26.462 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:26.460+0000 7f029aa00640 1 -- 172.21.15.32:0/1025168287 wait complete. 2024-08-03T20:55:26.463 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:26.462+0000 7f029aa00640 1 Processor -- start 2024-08-03T20:55:26.463 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:26.462+0000 7f029aa00640 1 -- start start 2024-08-03T20:55:26.466 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:26.463+0000 7f029aa00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f029406f440 0x7f0294156560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:55:26.466 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:26.463+0000 7f029aa00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f02940acd80 0x7f02940c8700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:55:26.466 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:26.463+0000 7f029aa00640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f02940af020 0x7f029415ea80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:55:26.466 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:26.463+0000 7f029aa00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f02941601e0 con 0x7f029406f440 2024-08-03T20:55:26.467 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:26.463+0000 7f029aa00640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f0294160060 con 0x7f02940acd80 2024-08-03T20:55:26.467 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:26.463+0000 7f029aa00640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7f0294160360 con 0x7f02940af020 2024-08-03T20:55:26.467 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:26.464+0000 7f0299600640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f029406f440 0x7f0294156560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:55:26.467 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:26.464+0000 7f0299600640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f029406f440 0x7f0294156560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:53382/0 (socket says 172.21.15.32:53382) 2024-08-03T20:55:26.467 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:26.464+0000 7f0299600640 1 -- 172.21.15.32:0/577959504 learned_addr learned my addr 172.21.15.32:0/577959504 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:55:26.467 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:26.464+0000 7f0298c00640 1 --2- 172.21.15.32:0/577959504 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f02940acd80 0x7f02940c8700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:55:26.467 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:26.464+0000 7f029a000640 1 --2- 172.21.15.32:0/577959504 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f02940af020 0x7f029415ea80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:55:26.467 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:26.464+0000 7f0299600640 1 -- 172.21.15.32:0/577959504 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f02940af020 msgr2=0x7f029415ea80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:55:26.467 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:26.464+0000 7f0299600640 1 --2- 172.21.15.32:0/577959504 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f02940af020 0x7f029415ea80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:26.467 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:26.464+0000 7f0299600640 1 -- 172.21.15.32:0/577959504 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f02940acd80 msgr2=0x7f02940c8700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:55:26.468 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:26.464+0000 7f0299600640 1 --2- 172.21.15.32:0/577959504 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f02940acd80 0x7f02940c8700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:26.468 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:26.464+0000 7f0299600640 1 -- 172.21.15.32:0/577959504 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f02940c77d0 con 0x7f029406f440 2024-08-03T20:55:26.468 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:26.464+0000 7f0299600640 1 --2- 172.21.15.32:0/577959504 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f029406f440 0x7f0294156560 secure :-1 s=READY pgs=1195 cs=0 l=1 rev1=1 crypto rx=0x7f028800d780 tx=0x7f028800dc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:55:26.468 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:26.465+0000 7f0291600640 1 -- 172.21.15.32:0/577959504 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0288017070 con 0x7f029406f440 2024-08-03T20:55:26.468 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:26.465+0000 7f0291600640 1 -- 172.21.15.32:0/577959504 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f028800de90 con 0x7f029406f440 2024-08-03T20:55:26.468 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:26.465+0000 7f029aa00640 1 -- 172.21.15.32:0/577959504 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f02940c7a70 con 0x7f029406f440 2024-08-03T20:55:26.468 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:26.465+0000 7f0291600640 1 -- 172.21.15.32:0/577959504 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02880099d0 con 0x7f029406f440 2024-08-03T20:55:26.468 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:26.465+0000 7f029aa00640 1 -- 172.21.15.32:0/577959504 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f02940c7d50 con 0x7f029406f440 2024-08-03T20:55:26.468 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:26.466+0000 7f029aa00640 1 -- 172.21.15.32:0/577959504 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f02940ad190 con 0x7f029406f440 2024-08-03T20:55:26.469 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:26.468+0000 7f0291600640 1 -- 172.21.15.32:0/577959504 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f0288009b70 con 0x7f029406f440 2024-08-03T20:55:26.470 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:26.469+0000 7f0291600640 1 --2- 172.21.15.32:0/577959504 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f0274077040 0x7f0274079550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:55:26.470 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:26.469+0000 7f0291600640 1 -- 172.21.15.32:0/577959504 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 6370+0+0 (secure 0 0 0) 0x7f0288098e40 con 0x7f029406f440 2024-08-03T20:55:26.470 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:26.469+0000 7f0298c00640 1 --2- 172.21.15.32:0/577959504 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f0274077040 0x7f0274079550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:55:26.471 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:26.470+0000 7f0298c00640 1 --2- 172.21.15.32:0/577959504 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f0274077040 0x7f0274079550 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7f029409e0d0 tx=0x7f028000b260 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:55:26.474 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:26.473+0000 7f0291600640 1 -- 172.21.15.32:0/577959504 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7f0288020020 con 0x7f029406f440 2024-08-03T20:55:26.716 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:26.715+0000 7f029aa00640 1 -- 172.21.15.32:0/577959504 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f02940d0930 con 0x7f029406f440 2024-08-03T20:55:26.717 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:26.716+0000 7f0291600640 1 -- 172.21.15.32:0/577959504 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 123 v123) ==== 95+0+4390 (secure 0 0 0) 0x7f02940c77d0 con 0x7f029406f440 2024-08-03T20:55:26.718 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-03T20:55:26.718 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":123,"btime":"2024-08-03T20:55:26:214420+0000","default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6439,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6835/531080160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6834","nonce":531080160},{"type":"v1","addr":"172.21.15.118:6835","nonce":531080160}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":84},{"gid":6454,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6837/2796882319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6836","nonce":2796882319},{"type":"v1","addr":"172.21.15.118:6837","nonce":2796882319}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":86},{"gid":6813,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6834/2686933890","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":2686933890},{"type":"v1","addr":"172.21.15.32:6834","nonce":2686933890}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":85}],"filesystems":[{"mdsmap":{"epoch":123,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-08-03T20:54:26.950518+0000","modified":"2024-08-03T20:55:26.214330+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":91,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":6}]} 2024-08-03T20:55:26.719 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 123 2024-08-03T20:55:26.721 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:26.718+0000 7f029aa00640 1 -- 172.21.15.32:0/577959504 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f0274077040 msgr2=0x7f0274079550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:55:26.721 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:26.718+0000 7f029aa00640 1 --2- 172.21.15.32:0/577959504 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f0274077040 0x7f0274079550 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7f029409e0d0 tx=0x7f028000b260 comp rx=0 tx=0).stop 2024-08-03T20:55:26.721 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:26.718+0000 7f029aa00640 1 -- 172.21.15.32:0/577959504 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f029406f440 msgr2=0x7f0294156560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:55:26.722 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:26.718+0000 7f029aa00640 1 --2- 172.21.15.32:0/577959504 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f029406f440 0x7f0294156560 secure :-1 s=READY pgs=1195 cs=0 l=1 rev1=1 crypto rx=0x7f028800d780 tx=0x7f028800dc40 comp rx=0 tx=0).stop 2024-08-03T20:55:26.722 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:26.718+0000 7f029aa00640 1 -- 172.21.15.32:0/577959504 shutdown_connections 2024-08-03T20:55:26.722 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:26.718+0000 7f029aa00640 1 --2- 172.21.15.32:0/577959504 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f0274077040 0x7f0274079550 unknown :-1 s=CLOSED pgs=495 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:26.722 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:26.718+0000 7f029aa00640 1 --2- 172.21.15.32:0/577959504 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f02940acd80 0x7f02940c8700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:26.722 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:26.718+0000 7f029aa00640 1 --2- 172.21.15.32:0/577959504 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f02940af020 0x7f029415ea80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:26.722 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:26.718+0000 7f029aa00640 1 --2- 172.21.15.32:0/577959504 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f029406f440 0x7f0294156560 unknown :-1 s=CLOSED pgs=1195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:26.722 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:26.718+0000 7f029aa00640 1 -- 172.21.15.32:0/577959504 >> 172.21.15.32:0/577959504 conn(0x7f02940b2120 msgr2=0x7f02940cc7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:55:26.722 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:26.718+0000 7f029aa00640 1 -- 172.21.15.32:0/577959504 shutdown_connections 2024-08-03T20:55:26.722 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:26.718+0000 7f029aa00640 1 -- 172.21.15.32:0/577959504 wait complete. 2024-08-03T20:55:26.746 DEBUG:tasks.ceph_test_case:wait_until_true: success in 0s and 0 retries 2024-08-03T20:55:26.746 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-08-03T20:55:26.975 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:26.974+0000 7f0fbf200640 1 -- 172.21.15.32:0/2797705088 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f0fb815fff0 msgr2=0x7f0fb8160400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:55:26.976 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:26.974+0000 7f0fbf200640 1 --2- 172.21.15.32:0/2797705088 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f0fb815fff0 0x7f0fb8160400 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f0fb0005fd0 tx=0x7f0fb002aa90 comp rx=0 tx=0).stop 2024-08-03T20:55:26.976 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:26.974+0000 7f0fbf200640 1 -- 172.21.15.32:0/2797705088 shutdown_connections 2024-08-03T20:55:26.976 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:26.974+0000 7f0fbf200640 1 --2- 172.21.15.32:0/2797705088 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f0fb815fff0 0x7f0fb8160400 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:26.976 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:26.974+0000 7f0fbf200640 1 --2- 172.21.15.32:0/2797705088 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f0fb8144f90 0x7f0fb8074570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:26.976 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:26.974+0000 7f0fbf200640 1 -- 172.21.15.32:0/2797705088 >> 172.21.15.32:0/2797705088 conn(0x7f0fb80b1bd0 msgr2=0x7f0fb80b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:55:26.976 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:26.974+0000 7f0fbf200640 1 -- 172.21.15.32:0/2797705088 shutdown_connections 2024-08-03T20:55:26.976 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:26.974+0000 7f0fbf200640 1 -- 172.21.15.32:0/2797705088 wait complete. 2024-08-03T20:55:26.978 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:26.976+0000 7f0fbf200640 1 Processor -- start 2024-08-03T20:55:26.978 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:26.976+0000 7f0fbf200640 1 -- start start 2024-08-03T20:55:26.981 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:26.978+0000 7f0fbf200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0fb806e930 0x7f0fb815bd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:55:26.981 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:26.978+0000 7f0fbf200640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f0fb8144f90 0x7f0fb81e7440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:55:26.981 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:26.978+0000 7f0fbf200640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f0fb815fff0 0x7f0fb81e7980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:55:26.981 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:26.978+0000 7f0fbf200640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f0fb80cf840 con 0x7f0fb806e930 2024-08-03T20:55:26.981 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:26.978+0000 7f0fbf200640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f0fb8145f30 con 0x7f0fb8144f90 2024-08-03T20:55:26.981 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:26.978+0000 7f0fbf200640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7f0fb80cf9c0 con 0x7f0fb815fff0 2024-08-03T20:55:26.981 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:26.978+0000 7f0fb7e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0fb806e930 0x7f0fb815bd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:55:26.981 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:26.978+0000 7f0fb7e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0fb806e930 0x7f0fb815bd90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:53394/0 (socket says 172.21.15.32:53394) 2024-08-03T20:55:26.981 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:26.978+0000 7f0fb7e00640 1 -- 172.21.15.32:0/1914034698 learned_addr learned my addr 172.21.15.32:0/1914034698 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:55:26.981 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:26.978+0000 7f0faf400640 1 --2- 172.21.15.32:0/1914034698 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f0fb8144f90 0x7f0fb81e7440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:55:26.982 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:26.978+0000 7f0fbd200640 1 --2- 172.21.15.32:0/1914034698 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f0fb815fff0 0x7f0fb81e7980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:55:26.982 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:26.979+0000 7f0fb7e00640 1 -- 172.21.15.32:0/1914034698 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f0fb815fff0 msgr2=0x7f0fb81e7980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:55:26.982 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:26.979+0000 7f0fb7e00640 1 --2- 172.21.15.32:0/1914034698 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f0fb815fff0 0x7f0fb81e7980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:26.982 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:26.979+0000 7f0fb7e00640 1 -- 172.21.15.32:0/1914034698 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f0fb8144f90 msgr2=0x7f0fb81e7440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:55:26.982 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:26.979+0000 7f0fb7e00640 1 --2- 172.21.15.32:0/1914034698 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f0fb8144f90 0x7f0fb81e7440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:26.983 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:26.979+0000 7f0fb7e00640 1 -- 172.21.15.32:0/1914034698 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0fb815c2d0 con 0x7f0fb806e930 2024-08-03T20:55:26.983 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:26.979+0000 7f0fb7e00640 1 --2- 172.21.15.32:0/1914034698 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0fb806e930 0x7f0fb815bd90 secure :-1 s=READY pgs=1196 cs=0 l=1 rev1=1 crypto rx=0x7f0fa800c3a0 tx=0x7f0fa800c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:55:26.983 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:26.979+0000 7f0fb4c00640 1 -- 172.21.15.32:0/1914034698 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0fa8017070 con 0x7f0fb806e930 2024-08-03T20:55:26.983 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:26.979+0000 7f0fb4c00640 1 -- 172.21.15.32:0/1914034698 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f0fa80053c0 con 0x7f0fb806e930 2024-08-03T20:55:26.983 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:26.979+0000 7f0fbf200640 1 -- 172.21.15.32:0/1914034698 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0fb81f5cd0 con 0x7f0fb806e930 2024-08-03T20:55:26.983 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:26.979+0000 7f0fb4c00640 1 -- 172.21.15.32:0/1914034698 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0fa8003850 con 0x7f0fb806e930 2024-08-03T20:55:26.983 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:26.979+0000 7f0fbf200640 1 -- 172.21.15.32:0/1914034698 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0fb81e5810 con 0x7f0fb806e930 2024-08-03T20:55:26.984 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:26.981+0000 7f0fbf200640 1 -- 172.21.15.32:0/1914034698 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0fb8160400 con 0x7f0fb806e930 2024-08-03T20:55:26.984 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:26.982+0000 7f0fb4c00640 1 -- 172.21.15.32:0/1914034698 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f0fa801a070 con 0x7f0fb806e930 2024-08-03T20:55:26.984 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:26.983+0000 7f0fb4c00640 1 --2- 172.21.15.32:0/1914034698 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f0f94077040 0x7f0f94079550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:55:26.985 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:26.983+0000 7f0fb4c00640 1 -- 172.21.15.32:0/1914034698 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 6370+0+0 (secure 0 0 0) 0x7f0fa8097f80 con 0x7f0fb806e930 2024-08-03T20:55:26.985 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:26.983+0000 7f0faf400640 1 --2- 172.21.15.32:0/1914034698 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f0f94077040 0x7f0f94079550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:55:26.986 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:26.984+0000 7f0faf400640 1 --2- 172.21.15.32:0/1914034698 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f0f94077040 0x7f0f94079550 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7f0fb81f97d0 tx=0x7f0fa0005f90 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:55:26.988 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:26.987+0000 7f0fb4c00640 1 -- 172.21.15.32:0/1914034698 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7f0fb815c2d0 con 0x7f0fb806e930 2024-08-03T20:55:27.236 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:27.234+0000 7f0fbf200640 1 -- 172.21.15.32:0/1914034698 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f0fb80b2280 con 0x7f0fb806e930 2024-08-03T20:55:27.236 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:27.235+0000 7f0fb4c00640 1 -- 172.21.15.32:0/1914034698 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 123 v123) ==== 95+0+4390 (secure 0 0 0) 0x7f0fa8069090 con 0x7f0fb806e930 2024-08-03T20:55:27.236 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-03T20:55:27.237 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":123,"btime":"2024-08-03T20:55:26:214420+0000","default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6439,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6835/531080160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6834","nonce":531080160},{"type":"v1","addr":"172.21.15.118:6835","nonce":531080160}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":84},{"gid":6454,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6837/2796882319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6836","nonce":2796882319},{"type":"v1","addr":"172.21.15.118:6837","nonce":2796882319}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":86},{"gid":6813,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6834/2686933890","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":2686933890},{"type":"v1","addr":"172.21.15.32:6834","nonce":2686933890}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":85}],"filesystems":[{"mdsmap":{"epoch":123,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-08-03T20:54:26.950518+0000","modified":"2024-08-03T20:55:26.214330+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":91,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":6}]} 2024-08-03T20:55:27.237 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 123 2024-08-03T20:55:27.242 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:27.237+0000 7f0fbf200640 1 -- 172.21.15.32:0/1914034698 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f0f94077040 msgr2=0x7f0f94079550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:55:27.242 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:27.237+0000 7f0fbf200640 1 --2- 172.21.15.32:0/1914034698 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f0f94077040 0x7f0f94079550 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7f0fb81f97d0 tx=0x7f0fa0005f90 comp rx=0 tx=0).stop 2024-08-03T20:55:27.242 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:27.237+0000 7f0fbf200640 1 -- 172.21.15.32:0/1914034698 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0fb806e930 msgr2=0x7f0fb815bd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:55:27.242 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:27.237+0000 7f0fbf200640 1 --2- 172.21.15.32:0/1914034698 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0fb806e930 0x7f0fb815bd90 secure :-1 s=READY pgs=1196 cs=0 l=1 rev1=1 crypto rx=0x7f0fa800c3a0 tx=0x7f0fa800c860 comp rx=0 tx=0).stop 2024-08-03T20:55:27.242 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:27.237+0000 7f0fbf200640 1 -- 172.21.15.32:0/1914034698 shutdown_connections 2024-08-03T20:55:27.242 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:27.237+0000 7f0fbf200640 1 --2- 172.21.15.32:0/1914034698 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f0f94077040 0x7f0f94079550 unknown :-1 s=CLOSED pgs=496 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:27.242 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:27.237+0000 7f0fbf200640 1 --2- 172.21.15.32:0/1914034698 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f0fb8144f90 0x7f0fb81e7440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:27.243 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:27.237+0000 7f0fbf200640 1 --2- 172.21.15.32:0/1914034698 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f0fb815fff0 0x7f0fb81e7980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:27.243 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:27.237+0000 7f0fbf200640 1 --2- 172.21.15.32:0/1914034698 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0fb806e930 0x7f0fb815bd90 unknown :-1 s=CLOSED pgs=1196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:27.243 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:27.237+0000 7f0fbf200640 1 -- 172.21.15.32:0/1914034698 >> 172.21.15.32:0/1914034698 conn(0x7f0fb80b1bd0 msgr2=0x7f0fb80b3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:55:27.243 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:27.238+0000 7f0fbf200640 1 -- 172.21.15.32:0/1914034698 shutdown_connections 2024-08-03T20:55:27.243 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:27.238+0000 7f0fbf200640 1 -- 172.21.15.32:0/1914034698 wait complete. 2024-08-03T20:55:27.264 DEBUG:tasks.ceph_test_case:wait_until_true: success in 0s and 0 retries 2024-08-03T20:55:27.264 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-08-03T20:55:27.492 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:27.490+0000 7f976d400640 1 -- 172.21.15.32:0/2238154478 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f976815fff0 msgr2=0x7f9768160400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:55:27.492 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:27.490+0000 7f976d400640 1 --2- 172.21.15.32:0/2238154478 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f976815fff0 0x7f9768160400 secure :-1 s=READY pgs=1197 cs=0 l=1 rev1=1 crypto rx=0x7f97680afaf0 tx=0x7f976002aac0 comp rx=0 tx=0).stop 2024-08-03T20:55:27.492 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:27.490+0000 7f976d400640 1 -- 172.21.15.32:0/2238154478 shutdown_connections 2024-08-03T20:55:27.492 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:27.490+0000 7f976d400640 1 --2- 172.21.15.32:0/2238154478 >> v2:172.21.15.118:3300/0 conn(0x7f9768144f90 0x7f9768074570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:27.492 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:27.490+0000 7f976d400640 1 --2- 172.21.15.32:0/2238154478 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f976815fff0 0x7f9768160400 unknown :-1 s=CLOSED pgs=1197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:27.492 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:27.490+0000 7f976d400640 1 -- 172.21.15.32:0/2238154478 >> 172.21.15.32:0/2238154478 conn(0x7f97680b1bd0 msgr2=0x7f97680b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:55:27.492 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:27.490+0000 7f976d400640 1 -- 172.21.15.32:0/2238154478 shutdown_connections 2024-08-03T20:55:27.492 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:27.490+0000 7f976d400640 1 -- 172.21.15.32:0/2238154478 wait complete. 2024-08-03T20:55:27.494 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:27.492+0000 7f976d400640 1 Processor -- start 2024-08-03T20:55:27.494 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:27.493+0000 7f976d400640 1 -- start start 2024-08-03T20:55:27.497 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:27.494+0000 7f976d400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f976806e930 0x7f976815ca40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:55:27.497 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:27.494+0000 7f976d400640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f9768144f90 0x7f97681f82a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:55:27.497 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:27.494+0000 7f976d400640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f976815fff0 0x7f97681f9070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:55:27.497 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:27.494+0000 7f976d400640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f97680cf840 con 0x7f976806e930 2024-08-03T20:55:27.497 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:27.494+0000 7f976d400640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f9768145f30 con 0x7f976815fff0 2024-08-03T20:55:27.497 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:27.494+0000 7f976d400640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7f97680cf9c0 con 0x7f9768144f90 2024-08-03T20:55:27.497 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:27.494+0000 7f9766800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f976806e930 0x7f976815ca40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:55:27.498 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:27.494+0000 7f9766800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f976806e930 0x7f976815ca40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:53424/0 (socket says 172.21.15.32:53424) 2024-08-03T20:55:27.498 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:27.494+0000 7f9766800640 1 -- 172.21.15.32:0/1965791897 learned_addr learned my addr 172.21.15.32:0/1965791897 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:55:27.498 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:27.495+0000 7f9767200640 1 --2- 172.21.15.32:0/1965791897 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f976815fff0 0x7f97681f9070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:55:27.498 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:27.495+0000 7f975fe00640 1 --2- 172.21.15.32:0/1965791897 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f9768144f90 0x7f97681f82a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:55:27.498 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:27.495+0000 7f9766800640 1 -- 172.21.15.32:0/1965791897 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f9768144f90 msgr2=0x7f97681f82a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:55:27.498 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:27.495+0000 7f9766800640 1 --2- 172.21.15.32:0/1965791897 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f9768144f90 0x7f97681f82a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:27.498 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:27.495+0000 7f9766800640 1 -- 172.21.15.32:0/1965791897 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f976815fff0 msgr2=0x7f97681f9070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:55:27.498 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:27.495+0000 7f9766800640 1 --2- 172.21.15.32:0/1965791897 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f976815fff0 0x7f97681f9070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:27.498 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:27.495+0000 7f9766800640 1 -- 172.21.15.32:0/1965791897 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9768135410 con 0x7f976806e930 2024-08-03T20:55:27.498 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:27.495+0000 7f9766800640 1 --2- 172.21.15.32:0/1965791897 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f976806e930 0x7f976815ca40 secure :-1 s=READY pgs=1198 cs=0 l=1 rev1=1 crypto rx=0x7f975800c3d0 tx=0x7f975800c890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:55:27.498 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:27.495+0000 7f975ea00640 1 -- 172.21.15.32:0/1965791897 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9758017070 con 0x7f976806e930 2024-08-03T20:55:27.498 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:27.496+0000 7f975ea00640 1 -- 172.21.15.32:0/1965791897 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f97580053c0 con 0x7f976806e930 2024-08-03T20:55:27.499 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:27.496+0000 7f976d400640 1 -- 172.21.15.32:0/1965791897 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f97681356f0 con 0x7f976806e930 2024-08-03T20:55:27.499 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:27.496+0000 7f975ea00640 1 -- 172.21.15.32:0/1965791897 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9758003850 con 0x7f976806e930 2024-08-03T20:55:27.499 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:27.496+0000 7f976d400640 1 -- 172.21.15.32:0/1965791897 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f97681f8b50 con 0x7f976806e930 2024-08-03T20:55:27.499 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:27.497+0000 7f976d400640 1 -- 172.21.15.32:0/1965791897 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9768160cc0 con 0x7f976806e930 2024-08-03T20:55:27.499 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:27.498+0000 7f975ea00640 1 -- 172.21.15.32:0/1965791897 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f975801a070 con 0x7f976806e930 2024-08-03T20:55:27.500 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:27.499+0000 7f975ea00640 1 --2- 172.21.15.32:0/1965791897 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f9744077010 0x7f9744079520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:55:27.501 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:27.499+0000 7f975ea00640 1 -- 172.21.15.32:0/1965791897 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 6370+0+0 (secure 0 0 0) 0x7f9758097fd0 con 0x7f976806e930 2024-08-03T20:55:27.501 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:27.500+0000 7f975fe00640 1 --2- 172.21.15.32:0/1965791897 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f9744077010 0x7f9744079520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:55:27.502 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:27.500+0000 7f975fe00640 1 --2- 172.21.15.32:0/1965791897 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f9744077010 0x7f9744079520 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7f97680b1ba0 tx=0x7f975000b2e0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:55:27.505 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:27.504+0000 7f975ea00640 1 -- 172.21.15.32:0/1965791897 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7f9758060840 con 0x7f976806e930 2024-08-03T20:55:27.749 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:27.748+0000 7f976d400640 1 -- 172.21.15.32:0/1965791897 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f97680b2230 con 0x7f976806e930 2024-08-03T20:55:27.750 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:27.749+0000 7f975ea00640 1 -- 172.21.15.32:0/1965791897 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 123 v123) ==== 95+0+4390 (secure 0 0 0) 0x7f9758069160 con 0x7f976806e930 2024-08-03T20:55:27.751 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-03T20:55:27.751 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":123,"btime":"2024-08-03T20:55:26:214420+0000","default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6439,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6835/531080160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6834","nonce":531080160},{"type":"v1","addr":"172.21.15.118:6835","nonce":531080160}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":84},{"gid":6454,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6837/2796882319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6836","nonce":2796882319},{"type":"v1","addr":"172.21.15.118:6837","nonce":2796882319}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":86},{"gid":6813,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6834/2686933890","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":2686933890},{"type":"v1","addr":"172.21.15.32:6834","nonce":2686933890}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":85}],"filesystems":[{"mdsmap":{"epoch":123,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-08-03T20:54:26.950518+0000","modified":"2024-08-03T20:55:26.214330+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":91,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":6}]} 2024-08-03T20:55:27.751 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 123 2024-08-03T20:55:27.752 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:27.751+0000 7f976d400640 1 -- 172.21.15.32:0/1965791897 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f9744077010 msgr2=0x7f9744079520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:55:27.753 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:27.751+0000 7f976d400640 1 --2- 172.21.15.32:0/1965791897 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f9744077010 0x7f9744079520 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7f97680b1ba0 tx=0x7f975000b2e0 comp rx=0 tx=0).stop 2024-08-03T20:55:27.753 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:27.751+0000 7f976d400640 1 -- 172.21.15.32:0/1965791897 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f976806e930 msgr2=0x7f976815ca40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:55:27.753 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:27.751+0000 7f976d400640 1 --2- 172.21.15.32:0/1965791897 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f976806e930 0x7f976815ca40 secure :-1 s=READY pgs=1198 cs=0 l=1 rev1=1 crypto rx=0x7f975800c3d0 tx=0x7f975800c890 comp rx=0 tx=0).stop 2024-08-03T20:55:27.753 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:27.751+0000 7f976d400640 1 -- 172.21.15.32:0/1965791897 shutdown_connections 2024-08-03T20:55:27.753 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:27.751+0000 7f976d400640 1 --2- 172.21.15.32:0/1965791897 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f9744077010 0x7f9744079520 unknown :-1 s=CLOSED pgs=497 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:27.753 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:27.751+0000 7f976d400640 1 --2- 172.21.15.32:0/1965791897 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f976815fff0 0x7f97681f9070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:27.753 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:27.751+0000 7f976d400640 1 --2- 172.21.15.32:0/1965791897 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f9768144f90 0x7f97681f82a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:27.753 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:27.751+0000 7f976d400640 1 --2- 172.21.15.32:0/1965791897 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f976806e930 0x7f976815ca40 unknown :-1 s=CLOSED pgs=1198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:27.753 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:27.751+0000 7f976d400640 1 -- 172.21.15.32:0/1965791897 >> 172.21.15.32:0/1965791897 conn(0x7f97680b1bd0 msgr2=0x7f9768142310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:55:27.753 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:27.751+0000 7f976d400640 1 -- 172.21.15.32:0/1965791897 shutdown_connections 2024-08-03T20:55:27.753 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:27.751+0000 7f976d400640 1 -- 172.21.15.32:0/1965791897 wait complete. 2024-08-03T20:55:27.775 INFO:tasks.cephfs.test_damage:Result: Mutation 'Delete 200.00000001' led to DAMAGED state 2024-08-03T20:55:27.775 INFO:tasks.cephfs.test_damage:Applying mutation 'Delete mds0_openfiles.0' 2024-08-03T20:55:27.775 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-08-03T20:55:27.775 DEBUG:teuthology.orchestra.run.smithi032:> dd if=/proc/self/mounts of=/dev/stdout 2024-08-03T20:55:27.810 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053453,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6499208k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 none /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249604k,nr_inodes=812401,mode=700,uid=1000,gid=1316 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6499208k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-08-03T20:55:27.811 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi032.front.sepia.ceph.com 2024-08-03T20:55:27.811 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-08-03T20:55:27.811 DEBUG:teuthology.orchestra.run.smithi032:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-08-03T20:55:27.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-03T20:55:27.886 INFO:teuthology.orchestra.run.smithi032.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.0': No such file or directory 2024-08-03T20:55:27.886 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-08-03T20:55:28.156 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:28.154+0000 7f74c8c00640 1 -- 172.21.15.32:0/451639525 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f74c4073e60 msgr2=0x7f74c4074270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:55:28.157 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:28.154+0000 7f74c8c00640 1 --2- 172.21.15.32:0/451639525 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f74c4073e60 0x7f74c4074270 secure :-1 s=READY pgs=1200 cs=0 l=1 rev1=1 crypto rx=0x7f74c4071700 tx=0x7f74b002a4b0 comp rx=0 tx=0).stop 2024-08-03T20:55:28.157 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:28.154+0000 7f74c8c00640 1 -- 172.21.15.32:0/451639525 shutdown_connections 2024-08-03T20:55:28.157 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:28.154+0000 7f74c8c00640 1 --2- 172.21.15.32:0/451639525 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f74c4073e60 0x7f74c4074270 unknown :-1 s=CLOSED pgs=1200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:28.157 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:28.154+0000 7f74c8c00640 1 --2- 172.21.15.32:0/451639525 >> v2:172.21.15.118:3300/0 conn(0x7f74c406e930 0x7f74c4144f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:28.157 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:28.155+0000 7f74c8c00640 1 -- 172.21.15.32:0/451639525 >> 172.21.15.32:0/451639525 conn(0x7f74c40b1bd0 msgr2=0x7f74c40b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:55:28.157 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:28.155+0000 7f74c8c00640 1 -- 172.21.15.32:0/451639525 shutdown_connections 2024-08-03T20:55:28.157 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:28.155+0000 7f74c8c00640 1 -- 172.21.15.32:0/451639525 wait complete. 2024-08-03T20:55:28.158 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:28.157+0000 7f74c8c00640 1 Processor -- start 2024-08-03T20:55:28.158 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:28.157+0000 7f74c8c00640 1 -- start start 2024-08-03T20:55:28.161 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:28.158+0000 7f74c8c00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f74c406e930 0x7f74c41442d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:55:28.161 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:28.158+0000 7f74c8c00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f74c4073e60 0x7f74c413e270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:55:28.162 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:28.158+0000 7f74c8c00640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f74c40747b0 0x7f74c413e7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:55:28.162 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:28.158+0000 7f74c8c00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f74c40cf9c0 con 0x7f74c4073e60 2024-08-03T20:55:28.162 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:28.158+0000 7f74c8c00640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f74c40cf840 con 0x7f74c406e930 2024-08-03T20:55:28.162 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:28.158+0000 7f74c8c00640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7f74c40ccff0 con 0x7f74c40747b0 2024-08-03T20:55:28.162 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:28.159+0000 7f74c1400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f74c4073e60 0x7f74c413e270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:55:28.162 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:28.159+0000 7f74c1400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f74c4073e60 0x7f74c413e270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:53438/0 (socket says 172.21.15.32:53438) 2024-08-03T20:55:28.162 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:28.159+0000 7f74c1400640 1 -- 172.21.15.32:0/54997398 learned_addr learned my addr 172.21.15.32:0/54997398 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:55:28.163 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:28.159+0000 7f74c1e00640 1 --2- 172.21.15.32:0/54997398 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f74c406e930 0x7f74c41442d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:55:28.163 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:28.159+0000 7f74c2800640 1 --2- 172.21.15.32:0/54997398 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f74c40747b0 0x7f74c413e7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:55:28.163 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:28.159+0000 7f74c1400640 1 -- 172.21.15.32:0/54997398 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f74c40747b0 msgr2=0x7f74c413e7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:55:28.163 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:28.159+0000 7f74c1400640 1 --2- 172.21.15.32:0/54997398 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f74c40747b0 0x7f74c413e7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:28.163 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:28.159+0000 7f74c1400640 1 -- 172.21.15.32:0/54997398 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f74c406e930 msgr2=0x7f74c41442d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:55:28.163 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:28.159+0000 7f74c1400640 1 --2- 172.21.15.32:0/54997398 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f74c406e930 0x7f74c41442d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:28.163 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:28.159+0000 7f74c1400640 1 -- 172.21.15.32:0/54997398 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f74c4141a60 con 0x7f74c4073e60 2024-08-03T20:55:28.163 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:28.159+0000 7f74c1e00640 1 --2- 172.21.15.32:0/54997398 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f74c406e930 0x7f74c41442d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-03T20:55:28.163 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:28.159+0000 7f74c1400640 1 --2- 172.21.15.32:0/54997398 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f74c4073e60 0x7f74c413e270 secure :-1 s=READY pgs=1201 cs=0 l=1 rev1=1 crypto rx=0x7f74b0009920 tx=0x7f74b0033c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:55:28.163 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:28.160+0000 7f74ba000640 1 -- 172.21.15.32:0/54997398 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74b0031d60 con 0x7f74c4073e60 2024-08-03T20:55:28.163 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:28.160+0000 7f74ba000640 1 -- 172.21.15.32:0/54997398 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f74b0033e10 con 0x7f74c4073e60 2024-08-03T20:55:28.164 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:28.160+0000 7f74ba000640 1 -- 172.21.15.32:0/54997398 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74b0005260 con 0x7f74c4073e60 2024-08-03T20:55:28.164 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:28.160+0000 7f74c8c00640 1 -- 172.21.15.32:0/54997398 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f74c4141d40 con 0x7f74c4073e60 2024-08-03T20:55:28.164 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:28.160+0000 7f74c8c00640 1 -- 172.21.15.32:0/54997398 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f74c4142300 con 0x7f74c4073e60 2024-08-03T20:55:28.165 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:28.161+0000 7f74c8c00640 1 -- 172.21.15.32:0/54997398 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f74c40cddf0 con 0x7f74c4073e60 2024-08-03T20:55:28.165 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:28.162+0000 7f74ba000640 1 -- 172.21.15.32:0/54997398 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f74b0005400 con 0x7f74c4073e60 2024-08-03T20:55:28.165 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:28.163+0000 7f74ba000640 1 --2- 172.21.15.32:0/54997398 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f74a4077010 0x7f74a4079520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:55:28.165 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:28.163+0000 7f74ba000640 1 -- 172.21.15.32:0/54997398 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 6370+0+0 (secure 0 0 0) 0x7f74b00b9de0 con 0x7f74c4073e60 2024-08-03T20:55:28.165 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:28.163+0000 7f74c1e00640 1 --2- 172.21.15.32:0/54997398 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f74a4077010 0x7f74a4079520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:55:28.166 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:28.164+0000 7f74c1e00640 1 --2- 172.21.15.32:0/54997398 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f74a4077010 0x7f74a4079520 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7f74b4005fd0 tx=0x7f74b400f360 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:55:28.169 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:28.167+0000 7f74ba000640 1 -- 172.21.15.32:0/54997398 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7f74c40cddf0 con 0x7f74c4073e60 2024-08-03T20:55:28.415 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:28.413+0000 7f74c8c00640 1 -- 172.21.15.32:0/54997398 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7f74c40cd580 con 0x7f74c4073e60 2024-08-03T20:55:29.226 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:29.224+0000 7f74ba000640 1 -- 172.21.15.32:0/54997398 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v124) ==== 161+0+0 (secure 0 0 0) 0x7f74b003f030 con 0x7f74c4073e60 2024-08-03T20:55:29.226 INFO:teuthology.orchestra.run.smithi032.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-08-03T20:55:29.229 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:29.227+0000 7f74c8c00640 1 -- 172.21.15.32:0/54997398 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f74a4077010 msgr2=0x7f74a4079520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:55:29.229 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:29.227+0000 7f74c8c00640 1 --2- 172.21.15.32:0/54997398 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f74a4077010 0x7f74a4079520 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7f74b4005fd0 tx=0x7f74b400f360 comp rx=0 tx=0).stop 2024-08-03T20:55:29.229 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:29.227+0000 7f74c8c00640 1 -- 172.21.15.32:0/54997398 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f74c4073e60 msgr2=0x7f74c413e270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:55:29.229 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:29.227+0000 7f74c8c00640 1 --2- 172.21.15.32:0/54997398 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f74c4073e60 0x7f74c413e270 secure :-1 s=READY pgs=1201 cs=0 l=1 rev1=1 crypto rx=0x7f74b0009920 tx=0x7f74b0033c20 comp rx=0 tx=0).stop 2024-08-03T20:55:29.229 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:29.227+0000 7f74c8c00640 1 -- 172.21.15.32:0/54997398 shutdown_connections 2024-08-03T20:55:29.229 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:29.227+0000 7f74c8c00640 1 --2- 172.21.15.32:0/54997398 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f74a4077010 0x7f74a4079520 unknown :-1 s=CLOSED pgs=498 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:29.229 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:29.227+0000 7f74c8c00640 1 --2- 172.21.15.32:0/54997398 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f74c40747b0 0x7f74c413e7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:29.230 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:29.227+0000 7f74c8c00640 1 --2- 172.21.15.32:0/54997398 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f74c4073e60 0x7f74c413e270 unknown :-1 s=CLOSED pgs=1201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:29.230 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:29.227+0000 7f74c8c00640 1 --2- 172.21.15.32:0/54997398 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f74c406e930 0x7f74c41442d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:29.230 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:29.227+0000 7f74c8c00640 1 -- 172.21.15.32:0/54997398 >> 172.21.15.32:0/54997398 conn(0x7f74c40b1bd0 msgr2=0x7f74c40b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:55:29.230 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:29.228+0000 7f74c8c00640 1 -- 172.21.15.32:0/54997398 shutdown_connections 2024-08-03T20:55:29.230 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:29.228+0000 7f74c8c00640 1 -- 172.21.15.32:0/54997398 wait complete. 2024-08-03T20:55:29.255 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mds repaired 0 2024-08-03T20:55:29.487 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:29.484+0000 7fe51b000640 1 -- 172.21.15.32:0/49335473 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe51406fb90 msgr2=0x7fe5140ca3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:55:29.487 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:29.485+0000 7fe51b000640 1 --2- 172.21.15.32:0/49335473 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe51406fb90 0x7fe5140ca3b0 secure :-1 s=READY pgs=1203 cs=0 l=1 rev1=1 crypto rx=0x7fe5140720a0 tx=0x7fe50c02a8b0 comp rx=0 tx=0).stop 2024-08-03T20:55:29.487 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:29.485+0000 7fe51b000640 1 -- 172.21.15.32:0/49335473 shutdown_connections 2024-08-03T20:55:29.487 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:29.485+0000 7fe51b000640 1 --2- 172.21.15.32:0/49335473 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe51406fb90 0x7fe5140ca3b0 unknown :-1 s=CLOSED pgs=1203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:29.487 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:29.485+0000 7fe51b000640 1 --2- 172.21.15.32:0/49335473 >> v2:172.21.15.118:3300/0 conn(0x7fe5140cee10 0x7fe51415c0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:29.487 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:29.485+0000 7fe51b000640 1 -- 172.21.15.32:0/49335473 >> 172.21.15.32:0/49335473 conn(0x7fe5140b2120 msgr2=0x7fe5140cfd60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:55:29.487 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:29.485+0000 7fe51b000640 1 -- 172.21.15.32:0/49335473 shutdown_connections 2024-08-03T20:55:29.487 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:29.485+0000 7fe51b000640 1 -- 172.21.15.32:0/49335473 wait complete. 2024-08-03T20:55:29.488 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:29.487+0000 7fe51b000640 1 Processor -- start 2024-08-03T20:55:29.488 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:29.487+0000 7fe51b000640 1 -- start start 2024-08-03T20:55:29.490 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:29.489+0000 7fe51b000640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fe5140af040 0x7fe51414c3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:55:29.491 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:29.489+0000 7fe51b000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe5140cee10 0x7fe5140b3d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:55:29.491 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:29.489+0000 7fe51b000640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fe5140b42b0 0x7fe51415d820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:55:29.491 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:29.489+0000 7fe51b000640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fe5140ca8f0 con 0x7fe5140cee10 2024-08-03T20:55:29.491 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:29.489+0000 7fe51b000640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7fe5140af660 con 0x7fe5140af040 2024-08-03T20:55:29.491 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:29.489+0000 7fe51b000640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7fe5140caa70 con 0x7fe5140b42b0 2024-08-03T20:55:29.491 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:29.489+0000 7fe519200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe5140cee10 0x7fe5140b3d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:55:29.491 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:29.489+0000 7fe519200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe5140cee10 0x7fe5140b3d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:53460/0 (socket says 172.21.15.32:53460) 2024-08-03T20:55:29.491 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:29.489+0000 7fe519200640 1 -- 172.21.15.32:0/892829524 learned_addr learned my addr 172.21.15.32:0/892829524 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:55:29.494 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:29.489+0000 7fe519c00640 1 --2- 172.21.15.32:0/892829524 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fe5140af040 0x7fe51414c3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:55:29.494 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:29.489+0000 7fe51a600640 1 --2- 172.21.15.32:0/892829524 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fe5140b42b0 0x7fe51415d820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:55:29.494 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:29.490+0000 7fe519200640 1 -- 172.21.15.32:0/892829524 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fe5140b42b0 msgr2=0x7fe51415d820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:55:29.494 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:29.490+0000 7fe519200640 1 --2- 172.21.15.32:0/892829524 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fe5140b42b0 0x7fe51415d820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:29.494 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:29.490+0000 7fe519200640 1 -- 172.21.15.32:0/892829524 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fe5140af040 msgr2=0x7fe51414c3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:55:29.494 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:29.490+0000 7fe519200640 1 --2- 172.21.15.32:0/892829524 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fe5140af040 0x7fe51414c3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:29.494 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:29.490+0000 7fe519200640 1 -- 172.21.15.32:0/892829524 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe51414c920 con 0x7fe5140cee10 2024-08-03T20:55:29.495 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:29.490+0000 7fe519c00640 1 --2- 172.21.15.32:0/892829524 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fe5140af040 0x7fe51414c3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-03T20:55:29.495 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:29.490+0000 7fe519200640 1 --2- 172.21.15.32:0/892829524 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe5140cee10 0x7fe5140b3d70 secure :-1 s=READY pgs=1204 cs=0 l=1 rev1=1 crypto rx=0x7fe50000c3a0 tx=0x7fe50000c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:55:29.495 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:29.490+0000 7fe511600640 1 -- 172.21.15.32:0/892829524 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe500011790 con 0x7fe5140cee10 2024-08-03T20:55:29.495 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:29.490+0000 7fe511600640 1 -- 172.21.15.32:0/892829524 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fe500017070 con 0x7fe5140cee10 2024-08-03T20:55:29.495 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:29.490+0000 7fe511600640 1 -- 172.21.15.32:0/892829524 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe500011cc0 con 0x7fe5140cee10 2024-08-03T20:55:29.495 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:29.490+0000 7fe51b000640 1 -- 172.21.15.32:0/892829524 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe5140b49f0 con 0x7fe5140cee10 2024-08-03T20:55:29.495 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:29.490+0000 7fe51b000640 1 -- 172.21.15.32:0/892829524 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe5140b4fb0 con 0x7fe5140cee10 2024-08-03T20:55:29.495 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:29.491+0000 7fe51b000640 1 -- 172.21.15.32:0/892829524 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe5140ca3b0 con 0x7fe5140cee10 2024-08-03T20:55:29.496 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:29.493+0000 7fe511600640 1 -- 172.21.15.32:0/892829524 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7fe50001c460 con 0x7fe5140cee10 2024-08-03T20:55:29.496 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:29.494+0000 7fe511600640 1 --2- 172.21.15.32:0/892829524 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fe4f4077010 0x7fe4f4079520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:55:29.496 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:29.494+0000 7fe511600640 1 -- 172.21.15.32:0/892829524 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 6370+0+0 (secure 0 0 0) 0x7fe500098a70 con 0x7fe5140cee10 2024-08-03T20:55:29.496 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:29.494+0000 7fe519c00640 1 --2- 172.21.15.32:0/892829524 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fe4f4077010 0x7fe4f4079520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:55:29.496 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:29.495+0000 7fe519c00640 1 --2- 172.21.15.32:0/892829524 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fe4f4077010 0x7fe4f4079520 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7fe508005fd0 tx=0x7fe50800f360 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:55:29.500 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:29.499+0000 7fe511600640 1 -- 172.21.15.32:0/892829524 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7fe51415c5f0 con 0x7fe5140cee10 2024-08-03T20:55:29.784 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:29.782+0000 7fe51b000640 1 -- 172.21.15.32:0/892829524 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "mds repaired", "role": "0"} v 0) -- 0x7fe5140b4740 con 0x7fe5140cee10 2024-08-03T20:55:30.228 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:30.226+0000 7fe511600640 1 -- 172.21.15.32:0/892829524 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "mds repaired", "role": "0"}]=0 repaired: restoring rank 6:0 v125) ==== 101+0+0 (secure 0 0 0) 0x7fe500061930 con 0x7fe5140cee10 2024-08-03T20:55:30.228 INFO:teuthology.orchestra.run.smithi032.stderr:repaired: restoring rank 6:0 2024-08-03T20:55:30.230 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:30.228+0000 7fe51b000640 1 -- 172.21.15.32:0/892829524 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fe4f4077010 msgr2=0x7fe4f4079520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:55:30.230 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:30.228+0000 7fe51b000640 1 --2- 172.21.15.32:0/892829524 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fe4f4077010 0x7fe4f4079520 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7fe508005fd0 tx=0x7fe50800f360 comp rx=0 tx=0).stop 2024-08-03T20:55:30.230 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:30.229+0000 7fe51b000640 1 -- 172.21.15.32:0/892829524 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe5140cee10 msgr2=0x7fe5140b3d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:55:30.231 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:30.229+0000 7fe51b000640 1 --2- 172.21.15.32:0/892829524 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe5140cee10 0x7fe5140b3d70 secure :-1 s=READY pgs=1204 cs=0 l=1 rev1=1 crypto rx=0x7fe50000c3a0 tx=0x7fe50000c860 comp rx=0 tx=0).stop 2024-08-03T20:55:30.231 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:30.229+0000 7fe51b000640 1 -- 172.21.15.32:0/892829524 shutdown_connections 2024-08-03T20:55:30.231 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:30.229+0000 7fe51b000640 1 --2- 172.21.15.32:0/892829524 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fe4f4077010 0x7fe4f4079520 unknown :-1 s=CLOSED pgs=499 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:30.231 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:30.229+0000 7fe51b000640 1 --2- 172.21.15.32:0/892829524 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fe5140b42b0 0x7fe51415d820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:30.231 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:30.229+0000 7fe51b000640 1 --2- 172.21.15.32:0/892829524 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe5140cee10 0x7fe5140b3d70 unknown :-1 s=CLOSED pgs=1204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:30.231 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:30.229+0000 7fe51b000640 1 --2- 172.21.15.32:0/892829524 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fe5140af040 0x7fe51414c3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:30.231 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:30.229+0000 7fe51b000640 1 -- 172.21.15.32:0/892829524 >> 172.21.15.32:0/892829524 conn(0x7fe5140b2120 msgr2=0x7fe514158150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:55:30.231 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:30.229+0000 7fe51b000640 1 -- 172.21.15.32:0/892829524 shutdown_connections 2024-08-03T20:55:30.231 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:30.229+0000 7fe51b000640 1 -- 172.21.15.32:0/892829524 wait complete. 2024-08-03T20:55:30.253 DEBUG:teuthology.orchestra.run.smithi032:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata import - 2024-08-03T20:55:30.396 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:30.394+0000 7fe3f7c0cf40 1 Processor -- start 2024-08-03T20:55:30.543 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:30.394+0000 7fe3f7c0cf40 1 -- start start 2024-08-03T20:55:30.547 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:30.395+0000 7fe3f7c0cf40 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x562ea1c17360 0x562ea1c34ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:55:30.548 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:30.395+0000 7fe3f7c0cf40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x562ea1c18ea0 0x562ea1c30e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:55:30.548 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:30.396+0000 7fe3f7c0cf40 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x562ea1c2e330 0x562ea1c35d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:55:30.548 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:30.396+0000 7fe3f7c0cf40 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x562ea1b0cbf0 con 0x562ea1c17360 2024-08-03T20:55:30.548 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:30.396+0000 7fe3f7c0cf40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x562ea1b196b0 con 0x562ea1c18ea0 2024-08-03T20:55:30.548 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:30.396+0000 7fe3f7c0cf40 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x562ea1aaee30 con 0x562ea1c2e330 2024-08-03T20:55:30.548 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:30.396+0000 7fe3f0200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x562ea1c17360 0x562ea1c34ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:55:30.548 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:30.396+0000 7fe3f0200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x562ea1c17360 0x562ea1c34ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:59390/0 (socket says 172.21.15.32:59390) 2024-08-03T20:55:30.548 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:30.396+0000 7fe3f0200640 1 -- 172.21.15.32:0/2913456242 learned_addr learned my addr 172.21.15.32:0/2913456242 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:55:30.548 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:30.396+0000 7fe3ef800640 1 --2- 172.21.15.32:0/2913456242 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x562ea1c18ea0 0x562ea1c30e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:55:30.548 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:30.396+0000 7fe3f0c00640 1 --2- 172.21.15.32:0/2913456242 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x562ea1c2e330 0x562ea1c35d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:55:30.548 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:30.396+0000 7fe3f0200640 1 -- 172.21.15.32:0/2913456242 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x562ea1c2e330 msgr2=0x562ea1c35d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:55:30.548 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:30.396+0000 7fe3f0200640 1 --2- 172.21.15.32:0/2913456242 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x562ea1c2e330 0x562ea1c35d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:30.549 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:30.396+0000 7fe3f0200640 1 -- 172.21.15.32:0/2913456242 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x562ea1c18ea0 msgr2=0x562ea1c30e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:55:30.549 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:30.396+0000 7fe3f0200640 1 --2- 172.21.15.32:0/2913456242 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x562ea1c18ea0 0x562ea1c30e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:30.549 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:30.396+0000 7fe3f0200640 1 -- 172.21.15.32:0/2913456242 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562ea1c2e8f0 con 0x562ea1c17360 2024-08-03T20:55:30.549 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:30.396+0000 7fe3ef800640 1 --2- 172.21.15.32:0/2913456242 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x562ea1c18ea0 0x562ea1c30e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-03T20:55:30.549 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:30.396+0000 7fe3f0200640 1 --2- 172.21.15.32:0/2913456242 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x562ea1c17360 0x562ea1c34ea0 secure :-1 s=READY pgs=1207 cs=0 l=1 rev1=1 crypto rx=0x7fe3e0001d80 tx=0x7fe3e0001e70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6b598666d7a94522 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:55:30.549 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:30.397+0000 7fe3ee400640 1 -- 172.21.15.32:0/2913456242 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3e000bd70 con 0x562ea1c17360 2024-08-03T20:55:30.549 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:30.397+0000 7fe3ee400640 1 -- 172.21.15.32:0/2913456242 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fe3e000f400 con 0x562ea1c17360 2024-08-03T20:55:30.549 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:30.397+0000 7fe3ee400640 1 -- 172.21.15.32:0/2913456242 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3e000f850 con 0x562ea1c17360 2024-08-03T20:55:30.549 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:30.397+0000 7fe3f7c0cf40 1 -- 172.21.15.32:0/2913456242 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x562ea1c17360 msgr2=0x562ea1c34ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:55:30.549 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:30.397+0000 7fe3f7c0cf40 1 --2- 172.21.15.32:0/2913456242 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x562ea1c17360 0x562ea1c34ea0 secure :-1 s=READY pgs=1207 cs=0 l=1 rev1=1 crypto rx=0x7fe3e0001d80 tx=0x7fe3e0001e70 comp rx=0 tx=0).stop 2024-08-03T20:55:30.549 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:30.397+0000 7fe3f7c0cf40 1 -- 172.21.15.32:0/2913456242 shutdown_connections 2024-08-03T20:55:30.549 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:30.397+0000 7fe3f7c0cf40 1 --2- 172.21.15.32:0/2913456242 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x562ea1c18ea0 0x562ea1c30e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:30.549 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:30.397+0000 7fe3f7c0cf40 1 --2- 172.21.15.32:0/2913456242 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x562ea1c2e330 0x562ea1c35d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:30.550 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:30.397+0000 7fe3f7c0cf40 1 --2- 172.21.15.32:0/2913456242 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x562ea1c17360 0x562ea1c34ea0 unknown :-1 s=CLOSED pgs=1207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:30.550 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:30.397+0000 7fe3f7c0cf40 1 -- 172.21.15.32:0/2913456242 >> 172.21.15.32:0/2913456242 conn(0x562ea1a80ac0 msgr2=0x562ea1c32100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:55:30.550 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:30.398+0000 7fe3f7c0cf40 1 -- 172.21.15.32:0/2913456242 shutdown_connections 2024-08-03T20:55:30.550 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:30.398+0000 7fe3f7c0cf40 1 -- 172.21.15.32:0/2913456242 wait complete. 2024-08-03T20:55:30.550 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:30.400+0000 7fe3f7c0cf40 1 Processor -- start 2024-08-03T20:55:30.550 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:30.400+0000 7fe3f7c0cf40 1 -- start start 2024-08-03T20:55:30.550 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:30.401+0000 7fe3f7c0cf40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x562ea1c17360 0x562ea1b9f900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:55:30.550 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:30.401+0000 7fe3f7c0cf40 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x562ea1c18ea0 0x562ea1c0e260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:55:30.550 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:30.401+0000 7fe3f7c0cf40 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x562ea1c2e330 0x562ea1ba8230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:55:30.550 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:30.401+0000 7fe3f7c0cf40 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x562ea1c2f200 con 0x562ea1c2e330 2024-08-03T20:55:30.550 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:30.401+0000 7fe3f7c0cf40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x562ea1c2ebb0 con 0x562ea1c17360 2024-08-03T20:55:30.550 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:30.401+0000 7fe3f7c0cf40 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x562ea1ba0310 con 0x562ea1c18ea0 2024-08-03T20:55:30.550 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:30.401+0000 7fe3f0c00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x562ea1c2e330 0x562ea1ba8230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:55:30.551 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:30.401+0000 7fe3f0200640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x562ea1c17360 0x562ea1b9f900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:55:30.551 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:30.401+0000 7fe3ef800640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x562ea1c18ea0 0x562ea1c0e260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:55:30.551 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:30.401+0000 7fe3f0c00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x562ea1c2e330 0x562ea1ba8230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:59392/0 (socket says 172.21.15.32:59392) 2024-08-03T20:55:30.551 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:30.401+0000 7fe3ef800640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x562ea1c18ea0 0x562ea1c0e260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3301/0 says I am v2:172.21.15.32:39778/0 (socket says 172.21.15.32:39778) 2024-08-03T20:55:30.551 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:30.401+0000 7fe3f0c00640 1 -- 172.21.15.32:0/747014091 learned_addr learned my addr 172.21.15.32:0/747014091 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:55:30.551 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:30.401+0000 7fe3f0200640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x562ea1c17360 0x562ea1b9f900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.32:47288/0 (socket says 172.21.15.32:47288) 2024-08-03T20:55:30.551 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:30.401+0000 7fe3f0c00640 1 -- 172.21.15.32:0/747014091 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x562ea1c18ea0 msgr2=0x562ea1c0e260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:55:30.551 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:30.401+0000 7fe3f0c00640 1 --2- 172.21.15.32:0/747014091 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x562ea1c18ea0 0x562ea1c0e260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:30.551 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:30.401+0000 7fe3f0c00640 1 -- 172.21.15.32:0/747014091 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x562ea1c17360 msgr2=0x562ea1b9f900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:55:30.551 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:30.401+0000 7fe3f0c00640 1 --2- 172.21.15.32:0/747014091 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x562ea1c17360 0x562ea1b9f900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:30.551 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:30.401+0000 7fe3f0c00640 1 -- 172.21.15.32:0/747014091 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562ea1b9fe40 con 0x562ea1c2e330 2024-08-03T20:55:30.551 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:30.401+0000 7fe3f0c00640 1 --2- 172.21.15.32:0/747014091 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x562ea1c2e330 0x562ea1ba8230 secure :-1 s=READY pgs=1208 cs=0 l=1 rev1=1 crypto rx=0x7fe3e8004170 tx=0x7fe3e800d530 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:55:30.552 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:30.401+0000 7fe3ed000640 1 -- 172.21.15.32:0/747014091 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3e8018070 con 0x562ea1c2e330 2024-08-03T20:55:30.552 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:30.402+0000 7fe3f7c0cf40 1 -- 172.21.15.32:0/747014091 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x562ea1ba99a0 con 0x562ea1c2e330 2024-08-03T20:55:30.552 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:30.402+0000 7fe3ed000640 1 -- 172.21.15.32:0/747014091 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fe3e80030d0 con 0x562ea1c2e330 2024-08-03T20:55:30.552 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:30.402+0000 7fe3ed000640 1 -- 172.21.15.32:0/747014091 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3e80032e0 con 0x562ea1c2e330 2024-08-03T20:55:30.552 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:30.402+0000 7fe3f7c0cf40 1 -- 172.21.15.32:0/747014091 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562ea1ba9f60 con 0x562ea1c2e330 2024-08-03T20:55:30.552 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:30.405+0000 7fe3ed000640 1 -- 172.21.15.32:0/747014091 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7fe3e801b070 con 0x562ea1c2e330 2024-08-03T20:55:30.552 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:30.405+0000 7fe3ed000640 1 --2- 172.21.15.32:0/747014091 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fe3d00759f0 0x7fe3d0077f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:55:30.552 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:30.405+0000 7fe3ed000640 1 -- 172.21.15.32:0/747014091 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 6370+0+0 (secure 0 0 0) 0x7fe3e8096ea0 con 0x562ea1c2e330 2024-08-03T20:55:30.552 INFO:teuthology.orchestra.run.smithi032.stdout:Importing pool 2024-08-03T20:55:30.553 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:30.405+0000 7fe3f7c0cf40 1 --2- 172.21.15.32:0/747014091 >> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] conn(0x562ea1c0e7a0 0x562ea1c14eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:55:30.554 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:30.405+0000 7fe3f7c0cf40 1 -- 172.21.15.32:0/747014091 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:1 12.10 12:09401a81:::601.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e91) -- 0x562ea1c12190 con 0x562ea1c0e7a0 2024-08-03T20:55:30.554 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:30.405+0000 7fe3f0200640 1 --2- 172.21.15.32:0/747014091 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fe3d00759f0 0x7fe3d0077f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:55:30.554 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:30.406+0000 7fe3ef800640 1 --2- 172.21.15.32:0/747014091 >> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] conn(0x562ea1c0e7a0 0x562ea1c14eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:55:30.554 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:30.406+0000 7fe3f0200640 1 --2- 172.21.15.32:0/747014091 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fe3d00759f0 0x7fe3d0077f00 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7fe3e0001d80 tx=0x7fe3e0001080 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:55:30.554 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:30.406+0000 7fe3ef800640 1 --2- 172.21.15.32:0/747014091 >> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] conn(0x562ea1c0e7a0 0x562ea1c14eb0 crc :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:55:30.554 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:30.409+0000 7fe3ef800640 1 -- 172.21.15.32:0/747014091 <== osd.0 v2:172.21.15.32:6816/3066156116 1 ==== osd_op_reply(1 601.00000000 [create] v91'164 uv138 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fe3dc056470 con 0x562ea1c0e7a0 2024-08-03T20:55:30.554 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:30.409+0000 7fe3f7c0cf40 1 -- 172.21.15.32:0/747014091 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:2 12.10 12:09401a81:::601.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e91) -- 0x562ea1c153f0 con 0x562ea1c0e7a0 2024-08-03T20:55:30.554 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:30.410+0000 7fe3ef800640 1 -- 172.21.15.32:0/747014091 <== osd.0 v2:172.21.15.32:6816/3066156116 2 ==== osd_op_reply(2 601.00000000 [delete] v91'165 uv165 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe3dc056470 con 0x562ea1c0e7a0 2024-08-03T20:55:30.554 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:30.410+0000 7fe3f7c0cf40 1 -- 172.21.15.32:0/747014091 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:3 12.10 12:09401a81:::601.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e91) -- 0x562ea1cda0b0 con 0x562ea1c0e7a0 2024-08-03T20:55:30.554 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:30.413+0000 7fe3ef800640 1 -- 172.21.15.32:0/747014091 <== osd.0 v2:172.21.15.32:6816/3066156116 3 ==== osd_op_reply(3 601.00000000 [create] v91'166 uv166 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe3dc056470 con 0x562ea1c0e7a0 2024-08-03T20:55:30.555 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::601.00000000:head# 2024-08-03T20:55:30.556 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:30.413+0000 7fe3f7c0cf40 1 -- 172.21.15.32:0/747014091 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:4 12.10 12:09401a81:::601.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e91) -- 0x562ea1cdb460 con 0x562ea1c0e7a0 2024-08-03T20:55:30.556 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:30.414+0000 7fe3ef800640 1 -- 172.21.15.32:0/747014091 <== osd.0 v2:172.21.15.32:6816/3066156116 4 ==== osd_op_reply(4 601.00000000 [setxattr (30)] v91'167 uv167 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe3dc056470 con 0x562ea1c0e7a0 2024-08-03T20:55:30.556 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:30.414+0000 7fe3f7c0cf40 1 -- 172.21.15.32:0/747014091 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:5 12.10 12:09401a81:::601.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e91) -- 0x562ea1cdb800 con 0x562ea1c0e7a0 2024-08-03T20:55:30.556 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:30.416+0000 7fe3ef800640 1 -- 172.21.15.32:0/747014091 <== osd.0 v2:172.21.15.32:6816/3066156116 5 ==== osd_op_reply(5 601.00000000 [setxattr (62)] v91'168 uv168 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe3dc056470 con 0x562ea1c0e7a0 2024-08-03T20:55:30.556 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:30.416+0000 7fe3f7c0cf40 1 -- 172.21.15.32:0/747014091 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:6 12.10 12:09401a81:::601.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e91) -- 0x562ea1cda450 con 0x562ea1c0e7a0 2024-08-03T20:55:30.556 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:30.418+0000 7fe3ef800640 1 -- 172.21.15.32:0/747014091 <== osd.0 v2:172.21.15.32:6816/3066156116 6 ==== osd_op_reply(6 601.00000000 [omap-set-header] v91'169 uv169 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe3dc056470 con 0x562ea1c0e7a0 2024-08-03T20:55:30.557 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:30.418+0000 7fe3f7c0cf40 1 -- 172.21.15.32:0/747014091 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:7 12.10 12:0bd6d154:::602.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e91) -- 0x562ea1cda860 con 0x562ea1c0e7a0 2024-08-03T20:55:30.557 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:30.418+0000 7fe3ef800640 1 -- 172.21.15.32:0/747014091 <== osd.0 v2:172.21.15.32:6816/3066156116 7 ==== osd_op_reply(7 602.00000000 [create] v91'170 uv144 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fe3dc056470 con 0x562ea1c0e7a0 2024-08-03T20:55:30.557 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:30.419+0000 7fe3f7c0cf40 1 -- 172.21.15.32:0/747014091 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:8 12.10 12:0bd6d154:::602.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e91) -- 0x562ea1cda860 con 0x562ea1c0e7a0 2024-08-03T20:55:30.557 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:30.420+0000 7fe3ef800640 1 -- 172.21.15.32:0/747014091 <== osd.0 v2:172.21.15.32:6816/3066156116 8 ==== osd_op_reply(8 602.00000000 [delete] v91'171 uv171 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe3dc056470 con 0x562ea1c0e7a0 2024-08-03T20:55:30.557 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:30.420+0000 7fe3f7c0cf40 1 -- 172.21.15.32:0/747014091 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:9 12.10 12:0bd6d154:::602.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e91) -- 0x562ea1cda810 con 0x562ea1c0e7a0 2024-08-03T20:55:30.557 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::602.00000000:head# 2024-08-03T20:55:30.559 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:30.422+0000 7fe3ef800640 1 -- 172.21.15.32:0/747014091 <== osd.0 v2:172.21.15.32:6816/3066156116 9 ==== osd_op_reply(9 602.00000000 [create] v91'172 uv172 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe3dc056470 con 0x562ea1c0e7a0 2024-08-03T20:55:30.559 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:30.422+0000 7fe3f7c0cf40 1 -- 172.21.15.32:0/747014091 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:10 12.10 12:0bd6d154:::602.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e91) -- 0x562ea1cda810 con 0x562ea1c0e7a0 2024-08-03T20:55:30.559 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:30.424+0000 7fe3ef800640 1 -- 172.21.15.32:0/747014091 <== osd.0 v2:172.21.15.32:6816/3066156116 10 ==== osd_op_reply(10 602.00000000 [setxattr (30)] v91'173 uv173 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe3dc056470 con 0x562ea1c0e7a0 2024-08-03T20:55:30.559 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:30.424+0000 7fe3f7c0cf40 1 -- 172.21.15.32:0/747014091 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:11 12.10 12:0bd6d154:::602.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e91) -- 0x562ea1cda7f0 con 0x562ea1c0e7a0 2024-08-03T20:55:30.559 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:30.425+0000 7fe3ef800640 1 -- 172.21.15.32:0/747014091 <== osd.0 v2:172.21.15.32:6816/3066156116 11 ==== osd_op_reply(11 602.00000000 [setxattr (62)] v91'174 uv174 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe3dc056470 con 0x562ea1c0e7a0 2024-08-03T20:55:30.559 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:30.425+0000 7fe3f7c0cf40 1 -- 172.21.15.32:0/747014091 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:12 12.10 12:0bd6d154:::602.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e91) -- 0x562ea1cda7f0 con 0x562ea1c0e7a0 2024-08-03T20:55:30.559 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:30.427+0000 7fe3ef800640 1 -- 172.21.15.32:0/747014091 <== osd.0 v2:172.21.15.32:6816/3066156116 12 ==== osd_op_reply(12 602.00000000 [omap-set-header] v91'175 uv175 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe3dc056470 con 0x562ea1c0e7a0 2024-08-03T20:55:30.559 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:30.427+0000 7fe3f7c0cf40 1 -- 172.21.15.32:0/747014091 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:13 12.10 12:0cf6ad0f:::10000000000.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e91) -- 0x562ea1cda7f0 con 0x562ea1c0e7a0 2024-08-03T20:55:30.559 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:30.429+0000 7fe3ef800640 1 -- 172.21.15.32:0/747014091 <== osd.0 v2:172.21.15.32:6816/3066156116 13 ==== osd_op_reply(13 10000000000.00000000 [create] v91'176 uv151 ondisk = -17 ((17) File exists)) ==== 164+0+0 (crc 0 0 0) 0x7fe3dc056470 con 0x562ea1c0e7a0 2024-08-03T20:55:30.559 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:30.429+0000 7fe3f7c0cf40 1 -- 172.21.15.32:0/747014091 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:14 12.10 12:0cf6ad0f:::10000000000.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e91) -- 0x562ea1cda7f0 con 0x562ea1c0e7a0 2024-08-03T20:55:30.559 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:30.430+0000 7fe3ef800640 1 -- 172.21.15.32:0/747014091 <== osd.0 v2:172.21.15.32:6816/3066156116 14 ==== osd_op_reply(14 10000000000.00000000 [delete] v91'177 uv177 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7fe3dc056470 con 0x562ea1c0e7a0 2024-08-03T20:55:30.560 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:30.430+0000 7fe3f7c0cf40 1 -- 172.21.15.32:0/747014091 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:15 12.10 12:0cf6ad0f:::10000000000.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e91) -- 0x562ea1cda7f0 con 0x562ea1c0e7a0 2024-08-03T20:55:30.560 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:30.432+0000 7fe3ef800640 1 -- 172.21.15.32:0/747014091 <== osd.0 v2:172.21.15.32:6816/3066156116 15 ==== osd_op_reply(15 10000000000.00000000 [create] v91'178 uv178 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7fe3dc056470 con 0x562ea1c0e7a0 2024-08-03T20:55:30.560 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::10000000000.00000000:head# 2024-08-03T20:55:30.561 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:30.432+0000 7fe3f7c0cf40 1 -- 172.21.15.32:0/747014091 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:16 12.10 12:0cf6ad0f:::10000000000.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e91) -- 0x562ea1cda7f0 con 0x562ea1c0e7a0 2024-08-03T20:55:30.562 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:30.433+0000 7fe3ef800640 1 -- 172.21.15.32:0/747014091 <== osd.0 v2:172.21.15.32:6816/3066156116 16 ==== osd_op_reply(16 10000000000.00000000 [setxattr (30)] v91'179 uv179 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7fe3dc056470 con 0x562ea1c0e7a0 2024-08-03T20:55:30.562 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:30.433+0000 7fe3f7c0cf40 1 -- 172.21.15.32:0/747014091 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:17 12.10 12:0cf6ad0f:::10000000000.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e91) -- 0x562ea1cda7f0 con 0x562ea1c0e7a0 2024-08-03T20:55:30.562 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:30.436+0000 7fe3ef800640 1 -- 172.21.15.32:0/747014091 <== osd.0 v2:172.21.15.32:6816/3066156116 17 ==== osd_op_reply(17 10000000000.00000000 [setxattr (62)] v91'180 uv180 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7fe3dc056470 con 0x562ea1c0e7a0 2024-08-03T20:55:30.562 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:30.436+0000 7fe3f7c0cf40 1 -- 172.21.15.32:0/747014091 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:18 12.10 12:0cf6ad0f:::10000000000.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e91) -- 0x562ea1cda7f0 con 0x562ea1c0e7a0 2024-08-03T20:55:30.562 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:30.436+0000 7fe3ef800640 1 -- 172.21.15.32:0/747014091 <== osd.0 v2:172.21.15.32:6816/3066156116 18 ==== osd_op_reply(18 10000000000.00000000 [omap-set-header] v91'181 uv181 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7fe3dc056470 con 0x562ea1c0e7a0 2024-08-03T20:55:30.562 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:30.436+0000 7fe3f7c0cf40 1 -- 172.21.15.32:0/747014091 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:19 12.10 12:0cf6ad0f:::10000000000.00000000:head [omap-set-vals in=593b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e91) -- 0x562ea1cdabe0 con 0x562ea1c0e7a0 2024-08-03T20:55:30.562 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:30.438+0000 7fe3ef800640 1 -- 172.21.15.32:0/747014091 <== osd.0 v2:172.21.15.32:6816/3066156116 19 ==== osd_op_reply(19 10000000000.00000000 [omap-set-vals] v91'182 uv182 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7fe3dc056470 con 0x562ea1c0e7a0 2024-08-03T20:55:30.562 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:30.438+0000 7fe3f7c0cf40 1 -- 172.21.15.32:0/747014091 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:20 12.10 12:0d82a743:::600.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e91) -- 0x562ea1cdabe0 con 0x562ea1c0e7a0 2024-08-03T20:55:30.562 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:30.440+0000 7fe3ef800640 1 -- 172.21.15.32:0/747014091 <== osd.0 v2:172.21.15.32:6816/3066156116 20 ==== osd_op_reply(20 600.00000000 [create] v91'183 uv157 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fe3dc056470 con 0x562ea1c0e7a0 2024-08-03T20:55:30.563 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:30.440+0000 7fe3f7c0cf40 1 -- 172.21.15.32:0/747014091 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:21 12.10 12:0d82a743:::600.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e91) -- 0x562ea1cdaab0 con 0x562ea1c0e7a0 2024-08-03T20:55:30.563 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:30.442+0000 7fe3ef800640 1 -- 172.21.15.32:0/747014091 <== osd.0 v2:172.21.15.32:6816/3066156116 21 ==== osd_op_reply(21 600.00000000 [delete] v91'184 uv184 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe3dc056470 con 0x562ea1c0e7a0 2024-08-03T20:55:30.563 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:30.442+0000 7fe3f7c0cf40 1 -- 172.21.15.32:0/747014091 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:22 12.10 12:0d82a743:::600.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e91) -- 0x562ea1cdaab0 con 0x562ea1c0e7a0 2024-08-03T20:55:30.563 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:30.444+0000 7fe3ef800640 1 -- 172.21.15.32:0/747014091 <== osd.0 v2:172.21.15.32:6816/3066156116 22 ==== osd_op_reply(22 600.00000000 [create] v91'185 uv185 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe3dc056470 con 0x562ea1c0e7a0 2024-08-03T20:55:30.563 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::600.00000000:head# 2024-08-03T20:55:30.564 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:30.444+0000 7fe3f7c0cf40 1 -- 172.21.15.32:0/747014091 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:23 12.10 12:0d82a743:::600.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e91) -- 0x562ea1cdaab0 con 0x562ea1c0e7a0 2024-08-03T20:55:30.564 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:30.445+0000 7fe3ef800640 1 -- 172.21.15.32:0/747014091 <== osd.0 v2:172.21.15.32:6816/3066156116 23 ==== osd_op_reply(23 600.00000000 [setxattr (30)] v91'186 uv186 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe3dc056470 con 0x562ea1c0e7a0 2024-08-03T20:55:30.565 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:30.445+0000 7fe3f7c0cf40 1 -- 172.21.15.32:0/747014091 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:24 12.10 12:0d82a743:::600.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e91) -- 0x562ea1cdaab0 con 0x562ea1c0e7a0 2024-08-03T20:55:30.565 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:30.446+0000 7fe3ef800640 1 -- 172.21.15.32:0/747014091 <== osd.0 v2:172.21.15.32:6816/3066156116 24 ==== osd_op_reply(24 600.00000000 [setxattr (62)] v91'187 uv187 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe3dc056470 con 0x562ea1c0e7a0 2024-08-03T20:55:30.565 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:30.446+0000 7fe3f7c0cf40 1 -- 172.21.15.32:0/747014091 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:25 12.10 12:0d82a743:::600.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e91) -- 0x562ea1cdaab0 con 0x562ea1c0e7a0 2024-08-03T20:55:30.565 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:30.448+0000 7fe3ef800640 1 -- 172.21.15.32:0/747014091 <== osd.0 v2:172.21.15.32:6816/3066156116 25 ==== osd_op_reply(25 600.00000000 [omap-set-header] v91'188 uv188 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe3dc056470 con 0x562ea1c0e7a0 2024-08-03T20:55:30.565 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:30.448+0000 7fe3f7c0cf40 1 -- 172.21.15.32:0/747014091 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:26 12.10 12:0d89b25e:::603.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e91) -- 0x562ea1cdaab0 con 0x562ea1c0e7a0 2024-08-03T20:55:30.565 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:30.449+0000 7fe3ef800640 1 -- 172.21.15.32:0/747014091 <== osd.0 v2:172.21.15.32:6816/3066156116 26 ==== osd_op_reply(26 603.00000000 [create] v91'189 uv163 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fe3dc056470 con 0x562ea1c0e7a0 2024-08-03T20:55:30.565 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:30.449+0000 7fe3f7c0cf40 1 -- 172.21.15.32:0/747014091 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:27 12.10 12:0d89b25e:::603.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e91) -- 0x562ea1cdaab0 con 0x562ea1c0e7a0 2024-08-03T20:55:30.565 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:30.451+0000 7fe3ef800640 1 -- 172.21.15.32:0/747014091 <== osd.0 v2:172.21.15.32:6816/3066156116 27 ==== osd_op_reply(27 603.00000000 [delete] v91'190 uv190 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe3dc056470 con 0x562ea1c0e7a0 2024-08-03T20:55:30.565 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:30.451+0000 7fe3f7c0cf40 1 -- 172.21.15.32:0/747014091 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:28 12.10 12:0d89b25e:::603.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e91) -- 0x562ea1cdaab0 con 0x562ea1c0e7a0 2024-08-03T20:55:30.566 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::603.00000000:head# 2024-08-03T20:55:30.567 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:30.452+0000 7fe3ef800640 1 -- 172.21.15.32:0/747014091 <== osd.0 v2:172.21.15.32:6816/3066156116 28 ==== osd_op_reply(28 603.00000000 [create] v91'191 uv191 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe3dc056470 con 0x562ea1c0e7a0 2024-08-03T20:55:30.567 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:30.453+0000 7fe3f7c0cf40 1 -- 172.21.15.32:0/747014091 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:29 12.10 12:0d89b25e:::603.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e91) -- 0x562ea1cdaab0 con 0x562ea1c0e7a0 2024-08-03T20:55:30.568 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:30.454+0000 7fe3ef800640 1 -- 172.21.15.32:0/747014091 <== osd.0 v2:172.21.15.32:6816/3066156116 29 ==== osd_op_reply(29 603.00000000 [setxattr (30)] v91'192 uv192 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe3dc056470 con 0x562ea1c0e7a0 2024-08-03T20:55:30.568 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:30.454+0000 7fe3f7c0cf40 1 -- 172.21.15.32:0/747014091 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:30 12.10 12:0d89b25e:::603.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e91) -- 0x562ea1cdaab0 con 0x562ea1c0e7a0 2024-08-03T20:55:30.568 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:30.456+0000 7fe3ef800640 1 -- 172.21.15.32:0/747014091 <== osd.0 v2:172.21.15.32:6816/3066156116 30 ==== osd_op_reply(30 603.00000000 [setxattr (62)] v91'193 uv193 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe3dc056470 con 0x562ea1c0e7a0 2024-08-03T20:55:30.568 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:30.456+0000 7fe3f7c0cf40 1 -- 172.21.15.32:0/747014091 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:31 12.10 12:0d89b25e:::603.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e91) -- 0x562ea1cdaab0 con 0x562ea1c0e7a0 2024-08-03T20:55:30.568 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:30.457+0000 7fe3ef800640 1 -- 172.21.15.32:0/747014091 <== osd.0 v2:172.21.15.32:6816/3066156116 31 ==== osd_op_reply(31 603.00000000 [omap-set-header] v91'194 uv194 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe3dc056470 con 0x562ea1c0e7a0 2024-08-03T20:55:30.568 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:30.457+0000 7fe3f7c0cf40 1 --2- 172.21.15.32:0/747014091 >> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] conn(0x562ea1cdbba0 0x562ea1cddfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:55:30.568 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:30.457+0000 7fe3f7c0cf40 1 -- 172.21.15.32:0/747014091 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:32 12.14 12:287034c4:::1.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e91) -- 0x562ea1cdaab0 con 0x562ea1cdbba0 2024-08-03T20:55:30.569 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:30.458+0000 7fe3f0200640 1 --2- 172.21.15.32:0/747014091 >> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] conn(0x562ea1cdbba0 0x562ea1cddfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:55:30.569 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:30.458+0000 7fe3f0200640 1 --2- 172.21.15.32:0/747014091 >> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] conn(0x562ea1cdbba0 0x562ea1cddfc0 crc :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:55:30.569 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:30.461+0000 7fe3f0200640 1 -- 172.21.15.32:0/747014091 <== osd.5 v2:172.21.15.118:6810/2724891770 1 ==== osd_op_reply(32 1.00000000.inode [create] v91'129 uv113 ondisk = -17 ((17) File exists)) ==== 160+0+0 (crc 0 0 0) 0x7fe3e001c690 con 0x562ea1cdbba0 2024-08-03T20:55:30.569 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:30.461+0000 7fe3f7c0cf40 1 -- 172.21.15.32:0/747014091 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:33 12.14 12:287034c4:::1.00000000.inode:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e91) -- 0x562ea1cde500 con 0x562ea1cdbba0 2024-08-03T20:55:30.569 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:30.462+0000 7fe3f0200640 1 -- 172.21.15.32:0/747014091 <== osd.5 v2:172.21.15.118:6810/2724891770 2 ==== osd_op_reply(33 1.00000000.inode [delete] v91'130 uv130 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7fe3e001c690 con 0x562ea1cdbba0 2024-08-03T20:55:30.569 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:30.462+0000 7fe3f7c0cf40 1 -- 172.21.15.32:0/747014091 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:34 12.14 12:287034c4:::1.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e91) -- 0x562ea1cde8a0 con 0x562ea1cdbba0 2024-08-03T20:55:30.570 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::1.00000000.inode:head# 2024-08-03T20:55:30.571 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:30.463+0000 7fe3f0200640 1 -- 172.21.15.32:0/747014091 <== osd.5 v2:172.21.15.118:6810/2724891770 3 ==== osd_op_reply(34 1.00000000.inode [create] v91'131 uv131 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7fe3e001c690 con 0x562ea1cdbba0 2024-08-03T20:55:30.571 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:30.463+0000 7fe3f7c0cf40 1 -- 172.21.15.32:0/747014091 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:35 12.14 12:287034c4:::1.00000000.inode:head [write 0~582 in=582b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e91) -- 0x562ea1cdec60 con 0x562ea1cdbba0 2024-08-03T20:55:30.571 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:30.465+0000 7fe3f0200640 1 -- 172.21.15.32:0/747014091 <== osd.5 v2:172.21.15.118:6810/2724891770 4 ==== osd_op_reply(35 1.00000000.inode [write 0~582] v91'132 uv132 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7fe3e001c690 con 0x562ea1cdbba0 2024-08-03T20:55:30.571 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:30.465+0000 7fe3f7c0cf40 1 -- 172.21.15.32:0/747014091 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:36 12.14 12:287034c4:::1.00000000.inode:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e91) -- 0x562ea1cdf090 con 0x562ea1cdbba0 2024-08-03T20:55:30.572 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:30.467+0000 7fe3f0200640 1 -- 172.21.15.32:0/747014091 <== osd.5 v2:172.21.15.118:6810/2724891770 5 ==== osd_op_reply(36 1.00000000.inode [omap-set-header] v91'133 uv133 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7fe3e001c690 con 0x562ea1cdbba0 2024-08-03T20:55:30.572 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:30.467+0000 7fe3f7c0cf40 1 -- 172.21.15.32:0/747014091 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:37 12.14 12:292cf221:::200.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e91) -- 0x562ea1cdf480 con 0x562ea1cdbba0 2024-08-03T20:55:30.572 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:30.469+0000 7fe3f0200640 1 -- 172.21.15.32:0/747014091 <== osd.5 v2:172.21.15.118:6810/2724891770 6 ==== osd_op_reply(37 200.00000000 [create] v91'134 uv116 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fe3e001c690 con 0x562ea1cdbba0 2024-08-03T20:55:30.572 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:30.469+0000 7fe3f7c0cf40 1 -- 172.21.15.32:0/747014091 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:38 12.14 12:292cf221:::200.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e91) -- 0x562ea1cdf870 con 0x562ea1cdbba0 2024-08-03T20:55:30.572 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:30.469+0000 7fe3f0200640 1 -- 172.21.15.32:0/747014091 <== osd.5 v2:172.21.15.118:6810/2724891770 7 ==== osd_op_reply(38 200.00000000 [delete] v91'135 uv135 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe3e001c690 con 0x562ea1cdbba0 2024-08-03T20:55:30.572 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:30.469+0000 7fe3f7c0cf40 1 -- 172.21.15.32:0/747014091 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:39 12.14 12:292cf221:::200.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e91) -- 0x562ea1cdfc60 con 0x562ea1cdbba0 2024-08-03T20:55:30.572 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:30.471+0000 7fe3f0200640 1 -- 172.21.15.32:0/747014091 <== osd.5 v2:172.21.15.118:6810/2724891770 8 ==== osd_op_reply(39 200.00000000 [create] v91'136 uv136 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe3e001c690 con 0x562ea1cdbba0 2024-08-03T20:55:30.572 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::200.00000000:head# 2024-08-03T20:55:30.573 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:30.471+0000 7fe3f7c0cf40 1 -- 172.21.15.32:0/747014091 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:40 12.14 12:292cf221:::200.00000000:head [write 0~90 in=90b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e91) -- 0x562ea1cdfc60 con 0x562ea1cdbba0 2024-08-03T20:55:30.573 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:30.472+0000 7fe3f0200640 1 -- 172.21.15.32:0/747014091 <== osd.5 v2:172.21.15.118:6810/2724891770 9 ==== osd_op_reply(40 200.00000000 [write 0~90] v91'137 uv137 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe3e001c690 con 0x562ea1cdbba0 2024-08-03T20:55:30.573 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:30.472+0000 7fe3f7c0cf40 1 -- 172.21.15.32:0/747014091 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:41 12.14 12:292cf221:::200.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e91) -- 0x562ea1cdfc10 con 0x562ea1cdbba0 2024-08-03T20:55:30.573 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:30.474+0000 7fe3f0200640 1 -- 172.21.15.32:0/747014091 <== osd.5 v2:172.21.15.118:6810/2724891770 10 ==== osd_op_reply(41 200.00000000 [omap-set-header] v91'138 uv138 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe3e001c690 con 0x562ea1cdbba0 2024-08-03T20:55:30.573 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:30.474+0000 7fe3f7c0cf40 1 -- 172.21.15.32:0/747014091 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:42 12.14 12:2e2fa760:::200.00000001:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e91) -- 0x562ea1cdfc10 con 0x562ea1cdbba0 2024-08-03T20:55:30.573 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:30.475+0000 7fe3f0200640 1 -- 172.21.15.32:0/747014091 <== osd.5 v2:172.21.15.118:6810/2724891770 11 ==== osd_op_reply(42 200.00000001 [create] v91'139 uv139 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe3e001c690 con 0x562ea1cdbba0 2024-08-03T20:55:30.574 INFO:teuthology.orchestra.run.smithi032.stdout:Write #-9223372036854775808:00000000:::200.00000001:head# 2024-08-03T20:55:30.575 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:30.476+0000 7fe3f7c0cf40 1 -- 172.21.15.32:0/747014091 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:43 12.14 12:2e2fa760:::200.00000001:head [write 0~18118 in=18118b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e91) -- 0x562ea1cdfc10 con 0x562ea1cdbba0 2024-08-03T20:55:30.575 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:30.477+0000 7fe3f0200640 1 -- 172.21.15.32:0/747014091 <== osd.5 v2:172.21.15.118:6810/2724891770 12 ==== osd_op_reply(43 200.00000001 [write 0~18118] v91'140 uv140 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe3e001c690 con 0x562ea1cdbba0 2024-08-03T20:55:30.575 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:30.477+0000 7fe3f7c0cf40 1 -- 172.21.15.32:0/747014091 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:44 12.14 12:2e2fa760:::200.00000001:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e91) -- 0x562ea1cdfc10 con 0x562ea1cdbba0 2024-08-03T20:55:30.575 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:30.479+0000 7fe3f0200640 1 -- 172.21.15.32:0/747014091 <== osd.5 v2:172.21.15.118:6810/2724891770 13 ==== osd_op_reply(44 200.00000001 [omap-set-header] v91'141 uv141 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe3e001c690 con 0x562ea1cdbba0 2024-08-03T20:55:30.575 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:30.479+0000 7fe3f7c0cf40 1 -- 172.21.15.32:0/747014091 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:45 12.14 12:2f55791f:::606.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e91) -- 0x562ea1cdfc10 con 0x562ea1cdbba0 2024-08-03T20:55:30.575 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:30.481+0000 7fe3f0200640 1 -- 172.21.15.32:0/747014091 <== osd.5 v2:172.21.15.118:6810/2724891770 14 ==== osd_op_reply(45 606.00000000 [create] v91'142 uv127 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fe3e001c690 con 0x562ea1cdbba0 2024-08-03T20:55:30.575 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:30.481+0000 7fe3f7c0cf40 1 -- 172.21.15.32:0/747014091 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:46 12.14 12:2f55791f:::606.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e91) -- 0x562ea1cdfc10 con 0x562ea1cdbba0 2024-08-03T20:55:30.575 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:30.482+0000 7fe3f0200640 1 -- 172.21.15.32:0/747014091 <== osd.5 v2:172.21.15.118:6810/2724891770 15 ==== osd_op_reply(46 606.00000000 [delete] v91'143 uv143 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe3e001c690 con 0x562ea1cdbba0 2024-08-03T20:55:30.576 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:30.482+0000 7fe3f7c0cf40 1 -- 172.21.15.32:0/747014091 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:47 12.14 12:2f55791f:::606.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e91) -- 0x562ea1cdfc10 con 0x562ea1cdbba0 2024-08-03T20:55:30.576 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:30.484+0000 7fe3f0200640 1 -- 172.21.15.32:0/747014091 <== osd.5 v2:172.21.15.118:6810/2724891770 16 ==== osd_op_reply(47 606.00000000 [create] v91'144 uv144 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe3e001c690 con 0x562ea1cdbba0 2024-08-03T20:55:30.576 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::606.00000000:head# 2024-08-03T20:55:30.577 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:30.484+0000 7fe3f7c0cf40 1 -- 172.21.15.32:0/747014091 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:48 12.14 12:2f55791f:::606.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e91) -- 0x562ea1cdfc10 con 0x562ea1cdbba0 2024-08-03T20:55:30.577 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:30.485+0000 7fe3f0200640 1 -- 172.21.15.32:0/747014091 <== osd.5 v2:172.21.15.118:6810/2724891770 17 ==== osd_op_reply(48 606.00000000 [setxattr (30)] v91'145 uv145 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe3e001c690 con 0x562ea1cdbba0 2024-08-03T20:55:30.577 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:30.485+0000 7fe3f7c0cf40 1 -- 172.21.15.32:0/747014091 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:49 12.14 12:2f55791f:::606.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e91) -- 0x562ea1cdfc10 con 0x562ea1cdbba0 2024-08-03T20:55:30.577 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:30.486+0000 7fe3f0200640 1 -- 172.21.15.32:0/747014091 <== osd.5 v2:172.21.15.118:6810/2724891770 18 ==== osd_op_reply(49 606.00000000 [setxattr (62)] v91'146 uv146 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe3e001c690 con 0x562ea1cdbba0 2024-08-03T20:55:30.577 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:30.486+0000 7fe3f7c0cf40 1 -- 172.21.15.32:0/747014091 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:50 12.14 12:2f55791f:::606.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e91) -- 0x562ea1cdfc10 con 0x562ea1cdbba0 2024-08-03T20:55:30.578 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:30.488+0000 7fe3f0200640 1 -- 172.21.15.32:0/747014091 <== osd.5 v2:172.21.15.118:6810/2724891770 19 ==== osd_op_reply(50 606.00000000 [omap-set-header] v91'147 uv147 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe3e001c690 con 0x562ea1cdbba0 2024-08-03T20:55:30.578 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:30.488+0000 7fe3f7c0cf40 1 -- 172.21.15.32:0/747014091 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:51 12.1c 12:3ed09add:::607.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e91) -- 0x562ea1cdfc10 con 0x562ea1c0e7a0 2024-08-03T20:55:30.578 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:30.490+0000 7fe3ef800640 1 -- 172.21.15.32:0/747014091 <== osd.0 v2:172.21.15.32:6816/3066156116 32 ==== osd_op_reply(51 607.00000000 [create] v91'33 uv32 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fe3dc056470 con 0x562ea1c0e7a0 2024-08-03T20:55:30.578 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:30.490+0000 7fe3f7c0cf40 1 -- 172.21.15.32:0/747014091 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:52 12.1c 12:3ed09add:::607.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e91) -- 0x562ea1cdfc10 con 0x562ea1c0e7a0 2024-08-03T20:55:30.578 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:30.491+0000 7fe3ef800640 1 -- 172.21.15.32:0/747014091 <== osd.0 v2:172.21.15.32:6816/3066156116 33 ==== osd_op_reply(52 607.00000000 [delete] v91'34 uv34 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe3dc056470 con 0x562ea1c0e7a0 2024-08-03T20:55:30.578 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:30.491+0000 7fe3f7c0cf40 1 -- 172.21.15.32:0/747014091 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:53 12.1c 12:3ed09add:::607.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e91) -- 0x562ea1cdfc10 con 0x562ea1c0e7a0 2024-08-03T20:55:30.578 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::607.00000000:head# 2024-08-03T20:55:30.580 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:30.492+0000 7fe3ef800640 1 -- 172.21.15.32:0/747014091 <== osd.0 v2:172.21.15.32:6816/3066156116 34 ==== osd_op_reply(53 607.00000000 [create] v91'35 uv35 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe3dc056470 con 0x562ea1c0e7a0 2024-08-03T20:55:30.580 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:30.493+0000 7fe3f7c0cf40 1 -- 172.21.15.32:0/747014091 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:54 12.1c 12:3ed09add:::607.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e91) -- 0x562ea1cdfc10 con 0x562ea1c0e7a0 2024-08-03T20:55:30.580 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:30.494+0000 7fe3ef800640 1 -- 172.21.15.32:0/747014091 <== osd.0 v2:172.21.15.32:6816/3066156116 35 ==== osd_op_reply(54 607.00000000 [setxattr (30)] v91'36 uv36 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe3dc056470 con 0x562ea1c0e7a0 2024-08-03T20:55:30.581 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:30.494+0000 7fe3f7c0cf40 1 -- 172.21.15.32:0/747014091 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:55 12.1c 12:3ed09add:::607.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e91) -- 0x562ea1cdfc10 con 0x562ea1c0e7a0 2024-08-03T20:55:30.581 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:30.496+0000 7fe3ef800640 1 -- 172.21.15.32:0/747014091 <== osd.0 v2:172.21.15.32:6816/3066156116 36 ==== osd_op_reply(55 607.00000000 [setxattr (62)] v91'37 uv37 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe3dc056470 con 0x562ea1c0e7a0 2024-08-03T20:55:30.581 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:30.496+0000 7fe3f7c0cf40 1 -- 172.21.15.32:0/747014091 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:56 12.1c 12:3ed09add:::607.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e91) -- 0x562ea1cdfc10 con 0x562ea1c0e7a0 2024-08-03T20:55:30.581 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:30.497+0000 7fe3ef800640 1 -- 172.21.15.32:0/747014091 <== osd.0 v2:172.21.15.32:6816/3066156116 37 ==== osd_op_reply(56 607.00000000 [omap-set-header] v91'38 uv38 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe3dc056470 con 0x562ea1c0e7a0 2024-08-03T20:55:30.581 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:30.497+0000 7fe3f7c0cf40 1 --2- 172.21.15.32:0/747014091 >> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] conn(0x562ea1ce0510 0x562ea1ce0920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:55:30.581 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:30.497+0000 7fe3f7c0cf40 1 -- 172.21.15.32:0/747014091 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:57 12.6 12:654134d2:::mds0_openfiles.0:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e91) -- 0x562ea1ce31d0 con 0x562ea1ce0510 2024-08-03T20:55:30.581 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:30.497+0000 7fe3ef800640 1 --2- 172.21.15.32:0/747014091 >> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] conn(0x562ea1ce0510 0x562ea1ce0920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:55:30.581 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:30.498+0000 7fe3ef800640 1 --2- 172.21.15.32:0/747014091 >> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] conn(0x562ea1ce0510 0x562ea1ce0920 crc :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:55:30.581 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:30.499+0000 7fe3ef800640 1 -- 172.21.15.32:0/747014091 <== osd.1 v2:172.21.15.32:6824/1551241759 1 ==== osd_op_reply(57 mds0_openfiles.0 [create] v91'28 uv27 ondisk = -17 ((17) File exists)) ==== 160+0+0 (crc 0 0 0) 0x7fe3dc056470 con 0x562ea1ce0510 2024-08-03T20:55:30.581 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:30.499+0000 7fe3f7c0cf40 1 -- 172.21.15.32:0/747014091 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:58 12.6 12:654134d2:::mds0_openfiles.0:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e91) -- 0x562ea1ce31d0 con 0x562ea1ce0510 2024-08-03T20:55:30.581 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:30.500+0000 7fe3ef800640 1 -- 172.21.15.32:0/747014091 <== osd.1 v2:172.21.15.32:6824/1551241759 2 ==== osd_op_reply(58 mds0_openfiles.0 [delete] v91'29 uv29 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7fe3dc056470 con 0x562ea1ce0510 2024-08-03T20:55:30.581 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:30.500+0000 7fe3f7c0cf40 1 -- 172.21.15.32:0/747014091 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:59 12.6 12:654134d2:::mds0_openfiles.0:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e91) -- 0x562ea1ce31d0 con 0x562ea1ce0510 2024-08-03T20:55:30.581 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:30.502+0000 7fe3ef800640 1 -- 172.21.15.32:0/747014091 <== osd.1 v2:172.21.15.32:6824/1551241759 3 ==== osd_op_reply(59 mds0_openfiles.0 [create] v91'30 uv30 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7fe3dc056470 con 0x562ea1ce0510 2024-08-03T20:55:30.582 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds0_openfiles.0:head# 2024-08-03T20:55:30.583 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:30.502+0000 7fe3f7c0cf40 1 -- 172.21.15.32:0/747014091 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:60 12.6 12:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e91) -- 0x562ea1ce35f0 con 0x562ea1ce0510 2024-08-03T20:55:30.583 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:30.503+0000 7fe3ef800640 1 -- 172.21.15.32:0/747014091 <== osd.1 v2:172.21.15.32:6824/1551241759 4 ==== osd_op_reply(60 mds0_openfiles.0 [omap-set-header] v91'31 uv31 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7fe3dc056470 con 0x562ea1ce0510 2024-08-03T20:55:30.583 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:30.503+0000 7fe3f7c0cf40 1 -- 172.21.15.32:0/747014091 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:61 12.e 12:717a0223:::608.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e91) -- 0x562ea1ce35f0 con 0x562ea1c0e7a0 2024-08-03T20:55:30.583 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:30.504+0000 7fe3ef800640 1 -- 172.21.15.32:0/747014091 <== osd.0 v2:172.21.15.32:6816/3066156116 38 ==== osd_op_reply(61 608.00000000 [create] v91'33 uv32 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fe3dc056470 con 0x562ea1c0e7a0 2024-08-03T20:55:30.583 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:30.504+0000 7fe3f7c0cf40 1 -- 172.21.15.32:0/747014091 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:62 12.e 12:717a0223:::608.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e91) -- 0x562ea1ce35f0 con 0x562ea1c0e7a0 2024-08-03T20:55:30.583 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:30.507+0000 7fe3ef800640 1 -- 172.21.15.32:0/747014091 <== osd.0 v2:172.21.15.32:6816/3066156116 39 ==== osd_op_reply(62 608.00000000 [delete] v91'34 uv34 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe3dc056470 con 0x562ea1c0e7a0 2024-08-03T20:55:30.583 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:30.507+0000 7fe3f7c0cf40 1 -- 172.21.15.32:0/747014091 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:63 12.e 12:717a0223:::608.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e91) -- 0x562ea1ce35f0 con 0x562ea1c0e7a0 2024-08-03T20:55:30.583 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:30.508+0000 7fe3ef800640 1 -- 172.21.15.32:0/747014091 <== osd.0 v2:172.21.15.32:6816/3066156116 40 ==== osd_op_reply(63 608.00000000 [create] v91'35 uv35 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe3dc056470 con 0x562ea1c0e7a0 2024-08-03T20:55:30.584 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::608.00000000:head# 2024-08-03T20:55:30.586 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:30.508+0000 7fe3f7c0cf40 1 -- 172.21.15.32:0/747014091 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:64 12.e 12:717a0223:::608.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e91) -- 0x562ea1ce4600 con 0x562ea1c0e7a0 2024-08-03T20:55:30.586 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:30.509+0000 7fe3ef800640 1 -- 172.21.15.32:0/747014091 <== osd.0 v2:172.21.15.32:6816/3066156116 41 ==== osd_op_reply(64 608.00000000 [setxattr (30)] v91'36 uv36 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe3dc056470 con 0x562ea1c0e7a0 2024-08-03T20:55:30.586 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:30.509+0000 7fe3f7c0cf40 1 -- 172.21.15.32:0/747014091 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:65 12.e 12:717a0223:::608.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e91) -- 0x562ea1ce4600 con 0x562ea1c0e7a0 2024-08-03T20:55:30.586 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:30.511+0000 7fe3ef800640 1 -- 172.21.15.32:0/747014091 <== osd.0 v2:172.21.15.32:6816/3066156116 42 ==== osd_op_reply(65 608.00000000 [setxattr (62)] v91'37 uv37 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe3dc056470 con 0x562ea1c0e7a0 2024-08-03T20:55:30.586 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:30.511+0000 7fe3f7c0cf40 1 -- 172.21.15.32:0/747014091 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:66 12.e 12:717a0223:::608.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e91) -- 0x562ea1ce35f0 con 0x562ea1c0e7a0 2024-08-03T20:55:30.586 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:30.513+0000 7fe3ef800640 1 -- 172.21.15.32:0/747014091 <== osd.0 v2:172.21.15.32:6816/3066156116 43 ==== osd_op_reply(66 608.00000000 [omap-set-header] v91'38 uv38 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe3dc056470 con 0x562ea1c0e7a0 2024-08-03T20:55:30.586 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:30.513+0000 7fe3f7c0cf40 1 --2- 172.21.15.32:0/747014091 >> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] conn(0x562ea1ce3b00 0x562ea1ce5f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:55:30.587 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:30.513+0000 7fe3f7c0cf40 1 -- 172.21.15.32:0/747014091 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:67 12.15 12:a93a17c2:::604.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e91) -- 0x562ea1ce6460 con 0x562ea1ce3b00 2024-08-03T20:55:30.587 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:30.513+0000 7fe3f0c00640 1 --2- 172.21.15.32:0/747014091 >> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] conn(0x562ea1ce3b00 0x562ea1ce5f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:55:30.587 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:30.513+0000 7fe3f0c00640 1 --2- 172.21.15.32:0/747014091 >> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] conn(0x562ea1ce3b00 0x562ea1ce5f20 crc :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:55:30.587 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:30.515+0000 7fe3f0c00640 1 -- 172.21.15.32:0/747014091 <== osd.6 v2:172.21.15.118:6825/801947386 1 ==== osd_op_reply(67 604.00000000 [create] v91'59 uv53 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fe3e801a030 con 0x562ea1ce3b00 2024-08-03T20:55:30.587 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:30.515+0000 7fe3f7c0cf40 1 -- 172.21.15.32:0/747014091 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:68 12.15 12:a93a17c2:::604.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e91) -- 0x562ea1ce6910 con 0x562ea1ce3b00 2024-08-03T20:55:30.587 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:30.516+0000 7fe3f0c00640 1 -- 172.21.15.32:0/747014091 <== osd.6 v2:172.21.15.118:6825/801947386 2 ==== osd_op_reply(68 604.00000000 [delete] v91'60 uv60 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe3e801a030 con 0x562ea1ce3b00 2024-08-03T20:55:30.587 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:30.516+0000 7fe3f7c0cf40 1 -- 172.21.15.32:0/747014091 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:69 12.15 12:a93a17c2:::604.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e91) -- 0x562ea1ce6df0 con 0x562ea1ce3b00 2024-08-03T20:55:30.587 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:30.517+0000 7fe3f0c00640 1 -- 172.21.15.32:0/747014091 <== osd.6 v2:172.21.15.118:6825/801947386 3 ==== osd_op_reply(69 604.00000000 [create] v91'61 uv61 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe3e801a030 con 0x562ea1ce3b00 2024-08-03T20:55:30.587 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::604.00000000:head# 2024-08-03T20:55:30.589 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:30.517+0000 7fe3f7c0cf40 1 -- 172.21.15.32:0/747014091 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:70 12.15 12:a93a17c2:::604.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e91) -- 0x562ea1ce86a0 con 0x562ea1ce3b00 2024-08-03T20:55:30.589 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:30.518+0000 7fe3f0c00640 1 -- 172.21.15.32:0/747014091 <== osd.6 v2:172.21.15.118:6825/801947386 4 ==== osd_op_reply(70 604.00000000 [setxattr (30)] v91'62 uv62 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe3e801a030 con 0x562ea1ce3b00 2024-08-03T20:55:30.589 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:30.518+0000 7fe3f7c0cf40 1 -- 172.21.15.32:0/747014091 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:71 12.15 12:a93a17c2:::604.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e91) -- 0x562ea1ce8a40 con 0x562ea1ce3b00 2024-08-03T20:55:30.589 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:30.519+0000 7fe3f0c00640 1 -- 172.21.15.32:0/747014091 <== osd.6 v2:172.21.15.118:6825/801947386 5 ==== osd_op_reply(71 604.00000000 [setxattr (62)] v91'63 uv63 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe3e801a030 con 0x562ea1ce3b00 2024-08-03T20:55:30.589 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:30.519+0000 7fe3f7c0cf40 1 -- 172.21.15.32:0/747014091 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:72 12.15 12:a93a17c2:::604.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e91) -- 0x562ea1ce7690 con 0x562ea1ce3b00 2024-08-03T20:55:30.589 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:30.521+0000 7fe3f0c00640 1 -- 172.21.15.32:0/747014091 <== osd.6 v2:172.21.15.118:6825/801947386 6 ==== osd_op_reply(72 604.00000000 [omap-set-header] v91'64 uv64 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe3e801a030 con 0x562ea1ce3b00 2024-08-03T20:55:30.589 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:30.521+0000 7fe3f7c0cf40 1 -- 172.21.15.32:0/747014091 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:73 12.15 12:aa448500:::500.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e91) -- 0x562ea1ce7a80 con 0x562ea1ce3b00 2024-08-03T20:55:30.589 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:30.522+0000 7fe3f0c00640 1 -- 172.21.15.32:0/747014091 <== osd.6 v2:172.21.15.118:6825/801947386 7 ==== osd_op_reply(73 500.00000000 [create] v91'65 uv58 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fe3e801a030 con 0x562ea1ce3b00 2024-08-03T20:55:30.589 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:30.522+0000 7fe3f7c0cf40 1 -- 172.21.15.32:0/747014091 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:74 12.15 12:aa448500:::500.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e91) -- 0x562ea1ce7e70 con 0x562ea1ce3b00 2024-08-03T20:55:30.589 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:30.523+0000 7fe3f0c00640 1 -- 172.21.15.32:0/747014091 <== osd.6 v2:172.21.15.118:6825/801947386 8 ==== osd_op_reply(74 500.00000000 [delete] v91'66 uv66 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe3e801a030 con 0x562ea1ce3b00 2024-08-03T20:55:30.589 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:30.523+0000 7fe3f7c0cf40 1 -- 172.21.15.32:0/747014091 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:75 12.15 12:aa448500:::500.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e91) -- 0x562ea1ce7e70 con 0x562ea1ce3b00 2024-08-03T20:55:30.589 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:30.524+0000 7fe3f0c00640 1 -- 172.21.15.32:0/747014091 <== osd.6 v2:172.21.15.118:6825/801947386 9 ==== osd_op_reply(75 500.00000000 [create] v91'67 uv67 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe3e801a030 con 0x562ea1ce3b00 2024-08-03T20:55:30.590 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::500.00000000:head# 2024-08-03T20:55:30.591 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:30.524+0000 7fe3f7c0cf40 1 -- 172.21.15.32:0/747014091 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:76 12.15 12:aa448500:::500.00000000:head [write 0~90 in=90b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e91) -- 0x562ea1ce7e20 con 0x562ea1ce3b00 2024-08-03T20:55:30.591 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:30.526+0000 7fe3f0c00640 1 -- 172.21.15.32:0/747014091 <== osd.6 v2:172.21.15.118:6825/801947386 10 ==== osd_op_reply(76 500.00000000 [write 0~90] v91'68 uv68 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe3e801a030 con 0x562ea1ce3b00 2024-08-03T20:55:30.591 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:30.526+0000 7fe3f7c0cf40 1 -- 172.21.15.32:0/747014091 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:77 12.15 12:aa448500:::500.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e91) -- 0x562ea1ce7e20 con 0x562ea1ce3b00 2024-08-03T20:55:30.591 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:30.527+0000 7fe3f0c00640 1 -- 172.21.15.32:0/747014091 <== osd.6 v2:172.21.15.118:6825/801947386 11 ==== osd_op_reply(77 500.00000000 [omap-set-header] v91'69 uv69 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe3e801a030 con 0x562ea1ce3b00 2024-08-03T20:55:30.591 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:30.527+0000 7fe3f7c0cf40 1 --2- 172.21.15.32:0/747014091 >> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] conn(0x562ea1ce8210 0x562ea1ceb300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:55:30.591 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:30.527+0000 7fe3f7c0cf40 1 -- 172.21.15.32:0/747014091 --> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] -- osd_op(unknown.0.0:78 12.d 12:b50e409b:::mds_snaptable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e91) -- 0x562ea1ceb840 con 0x562ea1ce8210 2024-08-03T20:55:30.591 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:30.528+0000 7fe3f0200640 1 --2- 172.21.15.32:0/747014091 >> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] conn(0x562ea1ce8210 0x562ea1ceb300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:55:30.591 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:30.528+0000 7fe3f0200640 1 --2- 172.21.15.32:0/747014091 >> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] conn(0x562ea1ce8210 0x562ea1ceb300 crc :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:55:30.591 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:30.530+0000 7fe3f0200640 1 -- 172.21.15.32:0/747014091 <== osd.7 v2:172.21.15.118:6818/653000048 1 ==== osd_op_reply(78 mds_snaptable [create] v91'27 uv26 ondisk = -17 ((17) File exists)) ==== 157+0+0 (crc 0 0 0) 0x7fe3e001c690 con 0x562ea1ce8210 2024-08-03T20:55:30.592 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:30.530+0000 7fe3f7c0cf40 1 -- 172.21.15.32:0/747014091 --> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] -- osd_op(unknown.0.0:79 12.d 12:b50e409b:::mds_snaptable:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e91) -- 0x562ea1ceb840 con 0x562ea1ce8210 2024-08-03T20:55:30.592 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:30.531+0000 7fe3f0200640 1 -- 172.21.15.32:0/747014091 <== osd.7 v2:172.21.15.118:6818/653000048 2 ==== osd_op_reply(79 mds_snaptable [delete] v91'28 uv28 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7fe3e001c690 con 0x562ea1ce8210 2024-08-03T20:55:30.592 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:30.531+0000 7fe3f7c0cf40 1 -- 172.21.15.32:0/747014091 --> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] -- osd_op(unknown.0.0:80 12.d 12:b50e409b:::mds_snaptable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e91) -- 0x562ea1ceb840 con 0x562ea1ce8210 2024-08-03T20:55:30.592 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds_snaptable:head# 2024-08-03T20:55:30.593 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:30.532+0000 7fe3f0200640 1 -- 172.21.15.32:0/747014091 <== osd.7 v2:172.21.15.118:6818/653000048 3 ==== osd_op_reply(80 mds_snaptable [create] v91'29 uv29 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7fe3e001c690 con 0x562ea1ce8210 2024-08-03T20:55:30.593 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:30.532+0000 7fe3f7c0cf40 1 -- 172.21.15.32:0/747014091 --> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] -- osd_op(unknown.0.0:81 12.d 12:b50e409b:::mds_snaptable:head [write 0~70 in=70b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e91) -- 0x562ea1ceb840 con 0x562ea1ce8210 2024-08-03T20:55:30.593 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:30.534+0000 7fe3f0200640 1 -- 172.21.15.32:0/747014091 <== osd.7 v2:172.21.15.118:6818/653000048 4 ==== osd_op_reply(81 mds_snaptable [write 0~70] v91'30 uv30 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7fe3e001c690 con 0x562ea1ce8210 2024-08-03T20:55:30.593 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:30.534+0000 7fe3f7c0cf40 1 -- 172.21.15.32:0/747014091 --> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] -- osd_op(unknown.0.0:82 12.d 12:b50e409b:::mds_snaptable:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e91) -- 0x562ea1ceb840 con 0x562ea1ce8210 2024-08-03T20:55:30.593 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:30.535+0000 7fe3f0200640 1 -- 172.21.15.32:0/747014091 <== osd.7 v2:172.21.15.118:6818/653000048 5 ==== osd_op_reply(82 mds_snaptable [omap-set-header] v91'31 uv31 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7fe3e001c690 con 0x562ea1ce8210 2024-08-03T20:55:30.594 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:30.535+0000 7fe3f7c0cf40 1 -- 172.21.15.32:0/747014091 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:83 12.1d 12:b871830b:::605.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e91) -- 0x562ea1ceb840 con 0x562ea1c0e7a0 2024-08-03T20:55:30.594 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:30.537+0000 7fe3ef800640 1 -- 172.21.15.32:0/747014091 <== osd.0 v2:172.21.15.32:6816/3066156116 44 ==== osd_op_reply(83 605.00000000 [create] v91'33 uv32 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fe3dc056470 con 0x562ea1c0e7a0 2024-08-03T20:55:30.594 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:30.537+0000 7fe3f7c0cf40 1 -- 172.21.15.32:0/747014091 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:84 12.1d 12:b871830b:::605.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e91) -- 0x562ea1ceb840 con 0x562ea1c0e7a0 2024-08-03T20:55:30.594 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:30.538+0000 7fe3ef800640 1 -- 172.21.15.32:0/747014091 <== osd.0 v2:172.21.15.32:6816/3066156116 45 ==== osd_op_reply(84 605.00000000 [delete] v91'34 uv34 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe3dc056470 con 0x562ea1c0e7a0 2024-08-03T20:55:30.594 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:30.538+0000 7fe3f7c0cf40 1 -- 172.21.15.32:0/747014091 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:85 12.1d 12:b871830b:::605.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e91) -- 0x562ea1ceb840 con 0x562ea1c0e7a0 2024-08-03T20:55:30.594 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::605.00000000:head# 2024-08-03T20:55:30.596 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:30.540+0000 7fe3ef800640 1 -- 172.21.15.32:0/747014091 <== osd.0 v2:172.21.15.32:6816/3066156116 46 ==== osd_op_reply(85 605.00000000 [create] v91'35 uv35 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe3dc056470 con 0x562ea1c0e7a0 2024-08-03T20:55:30.596 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:30.540+0000 7fe3f7c0cf40 1 -- 172.21.15.32:0/747014091 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:86 12.1d 12:b871830b:::605.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e91) -- 0x562ea1cecc90 con 0x562ea1c0e7a0 2024-08-03T20:55:30.596 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:30.541+0000 7fe3ef800640 1 -- 172.21.15.32:0/747014091 <== osd.0 v2:172.21.15.32:6816/3066156116 47 ==== osd_op_reply(86 605.00000000 [setxattr (30)] v91'36 uv36 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe3dc056470 con 0x562ea1c0e7a0 2024-08-03T20:55:30.596 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:30.541+0000 7fe3f7c0cf40 1 -- 172.21.15.32:0/747014091 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:87 12.1d 12:b871830b:::605.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e91) -- 0x562ea1cecc90 con 0x562ea1c0e7a0 2024-08-03T20:55:30.596 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:30.542+0000 7fe3ef800640 1 -- 172.21.15.32:0/747014091 <== osd.0 v2:172.21.15.32:6816/3066156116 48 ==== osd_op_reply(87 605.00000000 [setxattr (62)] v91'37 uv37 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe3dc056470 con 0x562ea1c0e7a0 2024-08-03T20:55:30.596 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:30.542+0000 7fe3f7c0cf40 1 -- 172.21.15.32:0/747014091 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:88 12.1d 12:b871830b:::605.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e91) -- 0x562ea1cebc80 con 0x562ea1c0e7a0 2024-08-03T20:55:30.596 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:30.544+0000 7fe3ef800640 1 -- 172.21.15.32:0/747014091 <== osd.0 v2:172.21.15.32:6816/3066156116 49 ==== osd_op_reply(88 605.00000000 [omap-set-header] v91'38 uv38 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe3dc056470 con 0x562ea1c0e7a0 2024-08-03T20:55:30.596 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:30.544+0000 7fe3f7c0cf40 1 -- 172.21.15.32:0/747014091 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:89 12.13 12:c91d4a1d:::mds0_inotable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e91) -- 0x562ea1cebc80 con 0x562ea1ce0510 2024-08-03T20:55:30.596 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:30.545+0000 7fe3ef800640 1 -- 172.21.15.32:0/747014091 <== osd.1 v2:172.21.15.32:6824/1551241759 5 ==== osd_op_reply(89 mds0_inotable [create] v91'54 uv48 ondisk = -17 ((17) File exists)) ==== 157+0+0 (crc 0 0 0) 0x7fe3dc056470 con 0x562ea1ce0510 2024-08-03T20:55:30.596 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:30.545+0000 7fe3f7c0cf40 1 -- 172.21.15.32:0/747014091 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:90 12.13 12:c91d4a1d:::mds0_inotable:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e91) -- 0x562ea1cebc80 con 0x562ea1ce0510 2024-08-03T20:55:30.596 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:30.546+0000 7fe3ef800640 1 -- 172.21.15.32:0/747014091 <== osd.1 v2:172.21.15.32:6824/1551241759 6 ==== osd_op_reply(90 mds0_inotable [delete] v91'55 uv55 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7fe3dc056470 con 0x562ea1ce0510 2024-08-03T20:55:30.597 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:30.546+0000 7fe3f7c0cf40 1 -- 172.21.15.32:0/747014091 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:91 12.13 12:c91d4a1d:::mds0_inotable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e91) -- 0x562ea1cebc80 con 0x562ea1ce0510 2024-08-03T20:55:30.597 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds0_inotable:head# 2024-08-03T20:55:30.599 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:30.548+0000 7fe3ef800640 1 -- 172.21.15.32:0/747014091 <== osd.1 v2:172.21.15.32:6824/1551241759 7 ==== osd_op_reply(91 mds0_inotable [create] v91'56 uv56 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7fe3dc056470 con 0x562ea1ce0510 2024-08-03T20:55:30.599 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:30.548+0000 7fe3f7c0cf40 1 -- 172.21.15.32:0/747014091 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:92 12.13 12:c91d4a1d:::mds0_inotable:head [write 0~34 in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e91) -- 0x562ea1cebc80 con 0x562ea1ce0510 2024-08-03T20:55:30.599 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:30.549+0000 7fe3ef800640 1 -- 172.21.15.32:0/747014091 <== osd.1 v2:172.21.15.32:6824/1551241759 8 ==== osd_op_reply(92 mds0_inotable [write 0~34] v91'57 uv57 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7fe3dc056470 con 0x562ea1ce0510 2024-08-03T20:55:30.599 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:30.549+0000 7fe3f7c0cf40 1 -- 172.21.15.32:0/747014091 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:93 12.13 12:c91d4a1d:::mds0_inotable:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e91) -- 0x562ea1cebc80 con 0x562ea1ce0510 2024-08-03T20:55:30.599 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:30.550+0000 7fe3ef800640 1 -- 172.21.15.32:0/747014091 <== osd.1 v2:172.21.15.32:6824/1551241759 9 ==== osd_op_reply(93 mds0_inotable [omap-set-header] v91'58 uv58 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7fe3dc056470 con 0x562ea1ce0510 2024-08-03T20:55:30.599 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:30.550+0000 7fe3f7c0cf40 1 -- 172.21.15.32:0/747014091 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:94 12.13 12:cd5a64a3:::100.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e91) -- 0x562ea1cebc80 con 0x562ea1ce0510 2024-08-03T20:55:30.599 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:30.552+0000 7fe3ef800640 1 -- 172.21.15.32:0/747014091 <== osd.1 v2:172.21.15.32:6824/1551241759 10 ==== osd_op_reply(94 100.00000000 [create] v91'59 uv53 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fe3dc056470 con 0x562ea1ce0510 2024-08-03T20:55:30.600 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:30.552+0000 7fe3f7c0cf40 1 -- 172.21.15.32:0/747014091 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:95 12.13 12:cd5a64a3:::100.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e91) -- 0x562ea1cebc80 con 0x562ea1ce0510 2024-08-03T20:55:30.600 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:30.553+0000 7fe3ef800640 1 -- 172.21.15.32:0/747014091 <== osd.1 v2:172.21.15.32:6824/1551241759 11 ==== osd_op_reply(95 100.00000000 [delete] v91'60 uv60 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe3dc056470 con 0x562ea1ce0510 2024-08-03T20:55:30.600 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:30.553+0000 7fe3f7c0cf40 1 -- 172.21.15.32:0/747014091 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:96 12.13 12:cd5a64a3:::100.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e91) -- 0x562ea1cebc80 con 0x562ea1ce0510 2024-08-03T20:55:30.600 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::100.00000000:head# 2024-08-03T20:55:30.601 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:30.554+0000 7fe3ef800640 1 -- 172.21.15.32:0/747014091 <== osd.1 v2:172.21.15.32:6824/1551241759 12 ==== osd_op_reply(96 100.00000000 [create] v91'61 uv61 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe3dc056470 con 0x562ea1ce0510 2024-08-03T20:55:30.602 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:30.554+0000 7fe3f7c0cf40 1 -- 172.21.15.32:0/747014091 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:97 12.13 12:cd5a64a3:::100.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e91) -- 0x562ea1cebc80 con 0x562ea1ce0510 2024-08-03T20:55:30.602 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:30.555+0000 7fe3ef800640 1 -- 172.21.15.32:0/747014091 <== osd.1 v2:172.21.15.32:6824/1551241759 13 ==== osd_op_reply(97 100.00000000 [omap-set-header] v91'62 uv62 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe3dc056470 con 0x562ea1ce0510 2024-08-03T20:55:30.602 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:30.555+0000 7fe3f7c0cf40 1 -- 172.21.15.32:0/747014091 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:98 12.13 12:cd5a64a3:::100.00000000:head [omap-set-vals in=5194b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e91) -- 0x562ea1ceea90 con 0x562ea1ce0510 2024-08-03T20:55:30.602 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:30.557+0000 7fe3ef800640 1 -- 172.21.15.32:0/747014091 <== osd.1 v2:172.21.15.32:6824/1551241759 14 ==== osd_op_reply(98 100.00000000 [omap-set-vals] v91'63 uv63 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe3dc056470 con 0x562ea1ce0510 2024-08-03T20:55:30.602 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:30.557+0000 7fe3f7c0cf40 1 --2- 172.21.15.32:0/747014091 >> [v2:172.21.15.32:6800/9088303,v1:172.21.15.32:6801/9088303] conn(0x562ea1cec640 0x562ea1ceee80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:55:30.602 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:30.557+0000 7fe3f7c0cf40 1 -- 172.21.15.32:0/747014091 --> [v2:172.21.15.32:6800/9088303,v1:172.21.15.32:6801/9088303] -- osd_op(unknown.0.0:99 12.b 12:d0630e4c:::mds0_sessionmap:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e91) -- 0x562ea1ceea60 con 0x562ea1cec640 2024-08-03T20:55:30.602 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:30.557+0000 7fe3ef800640 1 --2- 172.21.15.32:0/747014091 >> [v2:172.21.15.32:6800/9088303,v1:172.21.15.32:6801/9088303] conn(0x562ea1cec640 0x562ea1ceee80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:55:30.602 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:30.558+0000 7fe3ef800640 1 --2- 172.21.15.32:0/747014091 >> [v2:172.21.15.32:6800/9088303,v1:172.21.15.32:6801/9088303] conn(0x562ea1cec640 0x562ea1ceee80 crc :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:55:30.602 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:30.559+0000 7fe3ef800640 1 -- 172.21.15.32:0/747014091 <== osd.3 v2:172.21.15.32:6800/9088303 1 ==== osd_op_reply(99 mds0_sessionmap [create] v91'26 uv25 ondisk = -17 ((17) File exists)) ==== 159+0+0 (crc 0 0 0) 0x7fe3dc056470 con 0x562ea1cec640 2024-08-03T20:55:30.602 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:30.559+0000 7fe3f7c0cf40 1 -- 172.21.15.32:0/747014091 --> [v2:172.21.15.32:6800/9088303,v1:172.21.15.32:6801/9088303] -- osd_op(unknown.0.0:100 12.b 12:d0630e4c:::mds0_sessionmap:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e91) -- 0x562ea1ceea60 con 0x562ea1cec640 2024-08-03T20:55:30.602 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:30.561+0000 7fe3ef800640 1 -- 172.21.15.32:0/747014091 <== osd.3 v2:172.21.15.32:6800/9088303 2 ==== osd_op_reply(100 mds0_sessionmap [delete] v91'27 uv27 ondisk = 0) ==== 159+0+0 (crc 0 0 0) 0x7fe3dc056470 con 0x562ea1cec640 2024-08-03T20:55:30.602 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:30.561+0000 7fe3f7c0cf40 1 -- 172.21.15.32:0/747014091 --> [v2:172.21.15.32:6800/9088303,v1:172.21.15.32:6801/9088303] -- osd_op(unknown.0.0:101 12.b 12:d0630e4c:::mds0_sessionmap:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e91) -- 0x562ea1ceea60 con 0x562ea1cec640 2024-08-03T20:55:30.603 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:30.562+0000 7fe3ef800640 1 -- 172.21.15.32:0/747014091 <== osd.3 v2:172.21.15.32:6800/9088303 3 ==== osd_op_reply(101 mds0_sessionmap [create] v91'28 uv28 ondisk = 0) ==== 159+0+0 (crc 0 0 0) 0x7fe3dc056470 con 0x562ea1cec640 2024-08-03T20:55:30.603 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds0_sessionmap:head# 2024-08-03T20:55:30.604 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:30.562+0000 7fe3f7c0cf40 1 -- 172.21.15.32:0/747014091 --> [v2:172.21.15.32:6800/9088303,v1:172.21.15.32:6801/9088303] -- osd_op(unknown.0.0:102 12.b 12:d0630e4c:::mds0_sessionmap:head [omap-set-header in=14b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e91) -- 0x562ea1cef3c0 con 0x562ea1cec640 2024-08-03T20:55:30.604 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:30.564+0000 7fe3ef800640 1 -- 172.21.15.32:0/747014091 <== osd.3 v2:172.21.15.32:6800/9088303 4 ==== osd_op_reply(102 mds0_sessionmap [omap-set-header] v91'29 uv29 ondisk = 0) ==== 159+0+0 (crc 0 0 0) 0x7fe3dc056470 con 0x562ea1cec640 2024-08-03T20:55:30.604 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:30.564+0000 7fe3f7c0cf40 1 -- 172.21.15.32:0/747014091 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:103 12.7 12:e0b41d06:::609.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e91) -- 0x562ea1cef3c0 con 0x562ea1ce0510 2024-08-03T20:55:30.604 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:30.565+0000 7fe3ef800640 1 -- 172.21.15.32:0/747014091 <== osd.1 v2:172.21.15.32:6824/1551241759 15 ==== osd_op_reply(103 609.00000000 [create] v91'33 uv32 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fe3dc056470 con 0x562ea1ce0510 2024-08-03T20:55:30.604 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:30.565+0000 7fe3f7c0cf40 1 -- 172.21.15.32:0/747014091 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:104 12.7 12:e0b41d06:::609.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e91) -- 0x562ea1cef3c0 con 0x562ea1ce0510 2024-08-03T20:55:30.604 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:30.567+0000 7fe3ef800640 1 -- 172.21.15.32:0/747014091 <== osd.1 v2:172.21.15.32:6824/1551241759 16 ==== osd_op_reply(104 609.00000000 [delete] v91'34 uv34 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe3dc056470 con 0x562ea1ce0510 2024-08-03T20:55:30.604 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:30.567+0000 7fe3f7c0cf40 1 -- 172.21.15.32:0/747014091 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:105 12.7 12:e0b41d06:::609.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e91) -- 0x562ea1cef3c0 con 0x562ea1ce0510 2024-08-03T20:55:30.604 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::609.00000000:head# 2024-08-03T20:55:30.606 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:30.568+0000 7fe3ef800640 1 -- 172.21.15.32:0/747014091 <== osd.1 v2:172.21.15.32:6824/1551241759 17 ==== osd_op_reply(105 609.00000000 [create] v91'35 uv35 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe3dc056470 con 0x562ea1ce0510 2024-08-03T20:55:30.606 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:30.568+0000 7fe3f7c0cf40 1 -- 172.21.15.32:0/747014091 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:106 12.7 12:e0b41d06:::609.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e91) -- 0x562ea1cf03d0 con 0x562ea1ce0510 2024-08-03T20:55:30.606 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:30.569+0000 7fe3ef800640 1 -- 172.21.15.32:0/747014091 <== osd.1 v2:172.21.15.32:6824/1551241759 18 ==== osd_op_reply(106 609.00000000 [setxattr (30)] v91'36 uv36 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe3dc056470 con 0x562ea1ce0510 2024-08-03T20:55:30.606 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:30.569+0000 7fe3f7c0cf40 1 -- 172.21.15.32:0/747014091 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:107 12.7 12:e0b41d06:::609.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e91) -- 0x562ea1cf03d0 con 0x562ea1ce0510 2024-08-03T20:55:30.606 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:30.571+0000 7fe3ef800640 1 -- 172.21.15.32:0/747014091 <== osd.1 v2:172.21.15.32:6824/1551241759 19 ==== osd_op_reply(107 609.00000000 [setxattr (62)] v91'37 uv37 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe3dc056470 con 0x562ea1ce0510 2024-08-03T20:55:30.607 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:30.571+0000 7fe3f7c0cf40 1 -- 172.21.15.32:0/747014091 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:108 12.7 12:e0b41d06:::609.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e91) -- 0x562ea1cef3c0 con 0x562ea1ce0510 2024-08-03T20:55:30.607 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:30.572+0000 7fe3ef800640 1 -- 172.21.15.32:0/747014091 <== osd.1 v2:172.21.15.32:6824/1551241759 20 ==== osd_op_reply(108 609.00000000 [omap-set-header] v91'38 uv38 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe3dc056470 con 0x562ea1ce0510 2024-08-03T20:55:30.607 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:30.572+0000 7fe3f7c0cf40 1 -- 172.21.15.32:0/747014091 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:109 12.f 12:f1f69726:::400.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e91) -- 0x562ea1cef3c0 con 0x562ea1ce3b00 2024-08-03T20:55:30.607 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:30.575+0000 7fe3f0c00640 1 -- 172.21.15.32:0/747014091 <== osd.6 v2:172.21.15.118:6825/801947386 12 ==== osd_op_reply(109 400.00000000 [create] v91'27 uv26 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fe3e801a030 con 0x562ea1ce3b00 2024-08-03T20:55:30.607 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:30.575+0000 7fe3f7c0cf40 1 -- 172.21.15.32:0/747014091 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:110 12.f 12:f1f69726:::400.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e91) -- 0x562ea1cef3c0 con 0x562ea1ce3b00 2024-08-03T20:55:30.607 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:30.576+0000 7fe3f0c00640 1 -- 172.21.15.32:0/747014091 <== osd.6 v2:172.21.15.118:6825/801947386 13 ==== osd_op_reply(110 400.00000000 [delete] v91'28 uv28 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe3e801a030 con 0x562ea1ce3b00 2024-08-03T20:55:30.607 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:30.577+0000 7fe3f7c0cf40 1 -- 172.21.15.32:0/747014091 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:111 12.f 12:f1f69726:::400.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e91) -- 0x562ea1cef3c0 con 0x562ea1ce3b00 2024-08-03T20:55:30.607 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:30.578+0000 7fe3f0c00640 1 -- 172.21.15.32:0/747014091 <== osd.6 v2:172.21.15.118:6825/801947386 14 ==== osd_op_reply(111 400.00000000 [create] v91'29 uv29 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe3e801a030 con 0x562ea1ce3b00 2024-08-03T20:55:30.607 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::400.00000000:head# 2024-08-03T20:55:30.608 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:30.578+0000 7fe3f7c0cf40 1 -- 172.21.15.32:0/747014091 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:112 12.f 12:f1f69726:::400.00000000:head [write 0~22 in=22b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e91) -- 0x562ea1cef3c0 con 0x562ea1ce3b00 2024-08-03T20:55:30.608 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:30.580+0000 7fe3f0c00640 1 -- 172.21.15.32:0/747014091 <== osd.6 v2:172.21.15.118:6825/801947386 15 ==== osd_op_reply(112 400.00000000 [write 0~22] v91'30 uv30 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe3e801a030 con 0x562ea1ce3b00 2024-08-03T20:55:30.609 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:30.580+0000 7fe3f7c0cf40 1 -- 172.21.15.32:0/747014091 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:113 12.f 12:f1f69726:::400.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e91) -- 0x562ea1cef3c0 con 0x562ea1ce3b00 2024-08-03T20:55:30.609 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:30.580+0000 7fe3f0c00640 1 -- 172.21.15.32:0/747014091 <== osd.6 v2:172.21.15.118:6825/801947386 16 ==== osd_op_reply(113 400.00000000 [omap-set-header] v91'31 uv31 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe3e801a030 con 0x562ea1ce3b00 2024-08-03T20:55:30.609 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:30.581+0000 7fe3f7c0cf40 1 -- 172.21.15.32:0/747014091 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:114 12.1f 12:fe07bba1:::100.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e91) -- 0x562ea1cef3c0 con 0x562ea1ce3b00 2024-08-03T20:55:30.609 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:30.582+0000 7fe3f0c00640 1 -- 172.21.15.32:0/747014091 <== osd.6 v2:172.21.15.118:6825/801947386 17 ==== osd_op_reply(114 100.00000000.inode [create] v91'69 uv61 ondisk = -17 ((17) File exists)) ==== 162+0+0 (crc 0 0 0) 0x7fe3e801a030 con 0x562ea1ce3b00 2024-08-03T20:55:30.609 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:30.582+0000 7fe3f7c0cf40 1 -- 172.21.15.32:0/747014091 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:115 12.1f 12:fe07bba1:::100.00000000.inode:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e91) -- 0x562ea1cef3c0 con 0x562ea1ce3b00 2024-08-03T20:55:30.609 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:30.584+0000 7fe3f0c00640 1 -- 172.21.15.32:0/747014091 <== osd.6 v2:172.21.15.118:6825/801947386 18 ==== osd_op_reply(115 100.00000000.inode [delete] v91'70 uv70 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7fe3e801a030 con 0x562ea1ce3b00 2024-08-03T20:55:30.609 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:30.584+0000 7fe3f7c0cf40 1 -- 172.21.15.32:0/747014091 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:116 12.1f 12:fe07bba1:::100.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e91) -- 0x562ea1cef3c0 con 0x562ea1ce3b00 2024-08-03T20:55:30.609 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::100.00000000.inode:head# 2024-08-03T20:55:30.610 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:30.585+0000 7fe3f0c00640 1 -- 172.21.15.32:0/747014091 <== osd.6 v2:172.21.15.118:6825/801947386 19 ==== osd_op_reply(116 100.00000000.inode [create] v91'71 uv71 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7fe3e801a030 con 0x562ea1ce3b00 2024-08-03T20:55:30.611 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:30.585+0000 7fe3f7c0cf40 1 -- 172.21.15.32:0/747014091 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:117 12.1f 12:fe07bba1:::100.00000000.inode:head [write 0~582 in=582b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e91) -- 0x562ea1cef3c0 con 0x562ea1ce3b00 2024-08-03T20:55:30.611 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:30.587+0000 7fe3f0c00640 1 -- 172.21.15.32:0/747014091 <== osd.6 v2:172.21.15.118:6825/801947386 20 ==== osd_op_reply(117 100.00000000.inode [write 0~582] v91'72 uv72 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7fe3e801a030 con 0x562ea1ce3b00 2024-08-03T20:55:30.611 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:30.587+0000 7fe3f7c0cf40 1 -- 172.21.15.32:0/747014091 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:118 12.1f 12:fe07bba1:::100.00000000.inode:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e91) -- 0x562ea1cef3c0 con 0x562ea1ce3b00 2024-08-03T20:55:30.611 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:30.588+0000 7fe3f0c00640 1 -- 172.21.15.32:0/747014091 <== osd.6 v2:172.21.15.118:6825/801947386 21 ==== osd_op_reply(118 100.00000000.inode [omap-set-header] v91'73 uv73 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7fe3e801a030 con 0x562ea1ce3b00 2024-08-03T20:55:30.611 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:30.588+0000 7fe3f7c0cf40 1 -- 172.21.15.32:0/747014091 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:119 12.1f 12:ff5b34d6:::1.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e91) -- 0x562ea1cef3c0 con 0x562ea1ce3b00 2024-08-03T20:55:30.611 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:30.590+0000 7fe3f0c00640 1 -- 172.21.15.32:0/747014091 <== osd.6 v2:172.21.15.118:6825/801947386 22 ==== osd_op_reply(119 1.00000000 [create] v91'74 uv68 ondisk = -17 ((17) File exists)) ==== 154+0+0 (crc 0 0 0) 0x7fe3e801a030 con 0x562ea1ce3b00 2024-08-03T20:55:30.611 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:30.590+0000 7fe3f7c0cf40 1 -- 172.21.15.32:0/747014091 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:120 12.1f 12:ff5b34d6:::1.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e91) -- 0x562ea1cef3c0 con 0x562ea1ce3b00 2024-08-03T20:55:30.611 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:30.591+0000 7fe3f0c00640 1 -- 172.21.15.32:0/747014091 <== osd.6 v2:172.21.15.118:6825/801947386 23 ==== osd_op_reply(120 1.00000000 [delete] v91'75 uv75 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7fe3e801a030 con 0x562ea1ce3b00 2024-08-03T20:55:30.611 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:30.592+0000 7fe3f7c0cf40 1 -- 172.21.15.32:0/747014091 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:121 12.1f 12:ff5b34d6:::1.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e91) -- 0x562ea1cef3c0 con 0x562ea1ce3b00 2024-08-03T20:55:30.611 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::1.00000000:head# 2024-08-03T20:55:30.614 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:30.593+0000 7fe3f0c00640 1 -- 172.21.15.32:0/747014091 <== osd.6 v2:172.21.15.118:6825/801947386 24 ==== osd_op_reply(121 1.00000000 [create] v91'76 uv76 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7fe3e801a030 con 0x562ea1ce3b00 2024-08-03T20:55:30.614 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:30.593+0000 7fe3f7c0cf40 1 -- 172.21.15.32:0/747014091 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:122 12.1f 12:ff5b34d6:::1.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e91) -- 0x562ea1cf03d0 con 0x562ea1ce3b00 2024-08-03T20:55:30.614 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:30.595+0000 7fe3f0c00640 1 -- 172.21.15.32:0/747014091 <== osd.6 v2:172.21.15.118:6825/801947386 25 ==== osd_op_reply(122 1.00000000 [setxattr (30)] v91'77 uv77 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7fe3e801a030 con 0x562ea1ce3b00 2024-08-03T20:55:30.614 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:30.595+0000 7fe3f7c0cf40 1 -- 172.21.15.32:0/747014091 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:123 12.1f 12:ff5b34d6:::1.00000000:head [setxattr parent (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e91) -- 0x562ea1cf03d0 con 0x562ea1ce3b00 2024-08-03T20:55:30.614 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:30.597+0000 7fe3f0c00640 1 -- 172.21.15.32:0/747014091 <== osd.6 v2:172.21.15.118:6825/801947386 26 ==== osd_op_reply(123 1.00000000 [setxattr (30)] v91'78 uv78 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7fe3e801a030 con 0x562ea1ce3b00 2024-08-03T20:55:30.614 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:30.597+0000 7fe3f7c0cf40 1 -- 172.21.15.32:0/747014091 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:124 12.1f 12:ff5b34d6:::1.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e91) -- 0x562ea1cef3c0 con 0x562ea1ce3b00 2024-08-03T20:55:30.614 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:30.598+0000 7fe3f0c00640 1 -- 172.21.15.32:0/747014091 <== osd.6 v2:172.21.15.118:6825/801947386 27 ==== osd_op_reply(124 1.00000000 [omap-set-header] v91'79 uv79 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7fe3e801a030 con 0x562ea1ce3b00 2024-08-03T20:55:30.614 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:30.598+0000 7fe3f7c0cf40 1 -- 172.21.15.32:0/747014091 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:125 12.1f 12:ff5b34d6:::1.00000000:head [omap-set-vals in=584b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e91) -- 0x562ea1cef870 con 0x562ea1ce3b00 2024-08-03T20:55:30.615 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:30.599+0000 7fe3f0c00640 1 -- 172.21.15.32:0/747014091 <== osd.6 v2:172.21.15.118:6825/801947386 28 ==== osd_op_reply(125 1.00000000 [omap-set-vals] v91'80 uv80 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7fe3e801a030 con 0x562ea1ce3b00 2024-08-03T20:55:30.615 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:30.600+0000 7fe3f7c0cf40 1 -- 172.21.15.32:0/747014091 >> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] conn(0x562ea1c0e7a0 msgr2=0x562ea1c14eb0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:55:30.615 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:30.600+0000 7fe3f7c0cf40 1 --2- 172.21.15.32:0/747014091 >> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] conn(0x562ea1c0e7a0 0x562ea1c14eb0 crc :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:30.615 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:30.600+0000 7fe3f7c0cf40 1 -- 172.21.15.32:0/747014091 >> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] conn(0x562ea1ce0510 msgr2=0x562ea1ce0920 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:55:30.615 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:30.600+0000 7fe3f7c0cf40 1 --2- 172.21.15.32:0/747014091 >> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] conn(0x562ea1ce0510 0x562ea1ce0920 crc :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:30.615 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:30.600+0000 7fe3f7c0cf40 1 -- 172.21.15.32:0/747014091 >> [v2:172.21.15.32:6800/9088303,v1:172.21.15.32:6801/9088303] conn(0x562ea1cec640 msgr2=0x562ea1ceee80 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:55:30.615 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:30.600+0000 7fe3f7c0cf40 1 --2- 172.21.15.32:0/747014091 >> [v2:172.21.15.32:6800/9088303,v1:172.21.15.32:6801/9088303] conn(0x562ea1cec640 0x562ea1ceee80 crc :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:30.615 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:30.600+0000 7fe3f7c0cf40 1 -- 172.21.15.32:0/747014091 >> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] conn(0x562ea1cdbba0 msgr2=0x562ea1cddfc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:55:30.615 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:30.600+0000 7fe3f7c0cf40 1 --2- 172.21.15.32:0/747014091 >> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] conn(0x562ea1cdbba0 0x562ea1cddfc0 crc :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:30.616 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:30.600+0000 7fe3f7c0cf40 1 -- 172.21.15.32:0/747014091 >> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] conn(0x562ea1ce3b00 msgr2=0x562ea1ce5f20 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:55:30.616 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:30.600+0000 7fe3f0c00640 1 -- 172.21.15.32:0/747014091 reap_dead start 2024-08-03T20:55:30.616 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:30.600+0000 7fe3f7c0cf40 1 --2- 172.21.15.32:0/747014091 >> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] conn(0x562ea1ce3b00 0x562ea1ce5f20 crc :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:30.616 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:30.600+0000 7fe3f7c0cf40 1 -- 172.21.15.32:0/747014091 >> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] conn(0x562ea1ce8210 msgr2=0x562ea1ceb300 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:55:30.616 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:30.600+0000 7fe3f7c0cf40 1 --2- 172.21.15.32:0/747014091 >> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] conn(0x562ea1ce8210 0x562ea1ceb300 crc :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:30.616 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:30.600+0000 7fe3f7c0cf40 1 -- 172.21.15.32:0/747014091 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fe3d00759f0 msgr2=0x7fe3d0077f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:55:30.616 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:30.600+0000 7fe3f7c0cf40 1 --2- 172.21.15.32:0/747014091 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fe3d00759f0 0x7fe3d0077f00 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7fe3e0001d80 tx=0x7fe3e0001080 comp rx=0 tx=0).stop 2024-08-03T20:55:30.616 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:30.600+0000 7fe3f7c0cf40 1 -- 172.21.15.32:0/747014091 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x562ea1c2e330 msgr2=0x562ea1ba8230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:55:30.616 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:30.600+0000 7fe3f7c0cf40 1 --2- 172.21.15.32:0/747014091 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x562ea1c2e330 0x562ea1ba8230 secure :-1 s=READY pgs=1208 cs=0 l=1 rev1=1 crypto rx=0x7fe3e8004170 tx=0x7fe3e800d530 comp rx=0 tx=0).stop 2024-08-03T20:55:30.616 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:30.600+0000 7fe3f7c0cf40 1 -- 172.21.15.32:0/747014091 shutdown_connections 2024-08-03T20:55:30.616 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:30.600+0000 7fe3f7c0cf40 1 --2- 172.21.15.32:0/747014091 >> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] conn(0x562ea1ce8210 0x562ea1ceb300 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:30.616 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:30.600+0000 7fe3f7c0cf40 1 --2- 172.21.15.32:0/747014091 >> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] conn(0x562ea1ce3b00 0x562ea1ce5f20 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:30.617 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:30.600+0000 7fe3f7c0cf40 1 --2- 172.21.15.32:0/747014091 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fe3d00759f0 0x7fe3d0077f00 unknown :-1 s=CLOSED pgs=500 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:30.617 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:30.600+0000 7fe3f7c0cf40 1 --2- 172.21.15.32:0/747014091 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x562ea1c2e330 0x562ea1ba8230 unknown :-1 s=CLOSED pgs=1208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:30.617 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:30.600+0000 7fe3f7c0cf40 1 -- 172.21.15.32:0/747014091 >> 172.21.15.32:0/747014091 conn(0x562ea1a80ac0 msgr2=0x562ea1c30e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:55:30.617 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:30.600+0000 7fe3f7c0cf40 1 -- 172.21.15.32:0/747014091 shutdown_connections 2024-08-03T20:55:30.617 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:30.600+0000 7fe3f7c0cf40 1 -- 172.21.15.32:0/747014091 wait complete. 2024-08-03T20:55:30.617 DEBUG:teuthology.orchestra.run.smithi032:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata rm mds0_openfiles.0 2024-08-03T20:55:30.798 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:30.796+0000 7faac10e2f40 1 Processor -- start 2024-08-03T20:55:30.798 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:30.798+0000 7faac10e2f40 1 -- start start 2024-08-03T20:55:30.800 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:30.799+0000 7faac10e2f40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55894bb94360 0x55894bc3dac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:55:30.800 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:30.799+0000 7faac10e2f40 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55894bb95ea0 0x55894bc46190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:55:30.801 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:30.799+0000 7faac10e2f40 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55894bbb1cb0 0x55894bc466d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:55:30.801 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:30.799+0000 7faac10e2f40 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55894bbb20c0 con 0x55894bb95ea0 2024-08-03T20:55:30.801 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:30.799+0000 7faac10e2f40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x55894ba2be30 con 0x55894bb94360 2024-08-03T20:55:30.801 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:30.799+0000 7faac10e2f40 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x55894bbabd60 con 0x55894bbb1cb0 2024-08-03T20:55:30.801 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:30.799+0000 7faab9800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55894bb95ea0 0x55894bc46190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:55:30.801 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:30.799+0000 7faab9800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55894bb95ea0 0x55894bc46190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:59412/0 (socket says 172.21.15.32:59412) 2024-08-03T20:55:30.801 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:30.799+0000 7faab9800640 1 -- 172.21.15.32:0/3186353073 learned_addr learned my addr 172.21.15.32:0/3186353073 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:55:30.801 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:30.799+0000 7faaba200640 1 --2- 172.21.15.32:0/3186353073 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55894bb94360 0x55894bc3dac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:55:30.801 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:30.799+0000 7faabac00640 1 --2- 172.21.15.32:0/3186353073 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55894bbb1cb0 0x55894bc466d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:55:30.801 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:30.799+0000 7faab9800640 1 -- 172.21.15.32:0/3186353073 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55894bbb1cb0 msgr2=0x55894bc466d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:55:30.801 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:30.799+0000 7faab9800640 1 --2- 172.21.15.32:0/3186353073 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55894bbb1cb0 0x55894bc466d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:30.801 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:30.799+0000 7faab9800640 1 -- 172.21.15.32:0/3186353073 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55894bb94360 msgr2=0x55894bc3dac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:55:30.802 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:30.799+0000 7faab9800640 1 --2- 172.21.15.32:0/3186353073 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55894bb94360 0x55894bc3dac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:30.802 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:30.799+0000 7faab9800640 1 -- 172.21.15.32:0/3186353073 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55894ba04f10 con 0x55894bb95ea0 2024-08-03T20:55:30.802 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:30.799+0000 7faabac00640 1 --2- 172.21.15.32:0/3186353073 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55894bbb1cb0 0x55894bc466d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-03T20:55:30.802 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:30.799+0000 7faab9800640 1 --2- 172.21.15.32:0/3186353073 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55894bb95ea0 0x55894bc46190 secure :-1 s=READY pgs=1210 cs=0 l=1 rev1=1 crypto rx=0x7faaa8046500 tx=0x7faaa8002fb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bd5dccbb3ce3f022 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:55:30.803 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:30.799+0000 7faab3e00640 1 -- 172.21.15.32:0/3186353073 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faaa804d860 con 0x55894bb95ea0 2024-08-03T20:55:30.803 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:30.799+0000 7faab3e00640 1 -- 172.21.15.32:0/3186353073 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7faaa804dcf0 con 0x55894bb95ea0 2024-08-03T20:55:30.803 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:30.799+0000 7faab3e00640 1 -- 172.21.15.32:0/3186353073 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faaa80566d0 con 0x55894bb95ea0 2024-08-03T20:55:30.803 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:30.801+0000 7faac10e2f40 1 -- 172.21.15.32:0/3186353073 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55894bb95ea0 msgr2=0x55894bc46190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:55:30.804 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:30.801+0000 7faac10e2f40 1 --2- 172.21.15.32:0/3186353073 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55894bb95ea0 0x55894bc46190 secure :-1 s=READY pgs=1210 cs=0 l=1 rev1=1 crypto rx=0x7faaa8046500 tx=0x7faaa8002fb0 comp rx=0 tx=0).stop 2024-08-03T20:55:30.804 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:30.801+0000 7faac10e2f40 1 -- 172.21.15.32:0/3186353073 shutdown_connections 2024-08-03T20:55:30.804 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:30.801+0000 7faac10e2f40 1 --2- 172.21.15.32:0/3186353073 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55894bbb1cb0 0x55894bc466d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:30.804 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:30.801+0000 7faac10e2f40 1 --2- 172.21.15.32:0/3186353073 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55894bb95ea0 0x55894bc46190 unknown :-1 s=CLOSED pgs=1210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:30.804 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:30.801+0000 7faac10e2f40 1 --2- 172.21.15.32:0/3186353073 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55894bb94360 0x55894bc3dac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:30.804 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:30.801+0000 7faac10e2f40 1 -- 172.21.15.32:0/3186353073 >> 172.21.15.32:0/3186353073 conn(0x55894b9fdac0 msgr2=0x55894bbb3fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:55:30.804 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:30.801+0000 7faac10e2f40 1 -- 172.21.15.32:0/3186353073 shutdown_connections 2024-08-03T20:55:30.804 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:30.801+0000 7faac10e2f40 1 -- 172.21.15.32:0/3186353073 wait complete. 2024-08-03T20:55:30.804 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:30.802+0000 7faac10e2f40 1 Processor -- start 2024-08-03T20:55:30.804 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:30.802+0000 7faac10e2f40 1 -- start start 2024-08-03T20:55:30.807 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:30.804+0000 7faac10e2f40 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55894bb94360 0x55894bb8ac20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:55:30.807 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:30.804+0000 7faac10e2f40 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55894bb95ea0 0x55894bc4db10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:55:30.807 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:30.804+0000 7faac10e2f40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55894bbb1cb0 0x55894bc4e050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:55:30.807 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:30.804+0000 7faac10e2f40 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55894bbab890 con 0x55894bb94360 2024-08-03T20:55:30.807 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:30.804+0000 7faac10e2f40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x55894bbaabd0 con 0x55894bbb1cb0 2024-08-03T20:55:30.807 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:30.804+0000 7faac10e2f40 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x55894bbadfb0 con 0x55894bb95ea0 2024-08-03T20:55:30.807 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:30.804+0000 7faaba200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55894bb94360 0x55894bb8ac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:55:30.807 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:30.804+0000 7faaba200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55894bb94360 0x55894bb8ac20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:59420/0 (socket says 172.21.15.32:59420) 2024-08-03T20:55:30.807 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:30.804+0000 7faaba200640 1 -- 172.21.15.32:0/85127735 learned_addr learned my addr 172.21.15.32:0/85127735 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:55:30.808 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:30.804+0000 7faabac00640 1 --2- 172.21.15.32:0/85127735 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55894bbb1cb0 0x55894bc4e050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:55:30.808 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:30.804+0000 7faab9800640 1 --2- 172.21.15.32:0/85127735 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55894bb95ea0 0x55894bc4db10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:55:30.808 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:30.804+0000 7faaba200640 1 -- 172.21.15.32:0/85127735 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55894bb95ea0 msgr2=0x55894bc4db10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:55:30.808 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:30.804+0000 7faaba200640 1 --2- 172.21.15.32:0/85127735 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55894bb95ea0 0x55894bc4db10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:30.808 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:30.804+0000 7faaba200640 1 -- 172.21.15.32:0/85127735 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55894bbb1cb0 msgr2=0x55894bc4e050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:55:30.808 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:30.804+0000 7faaba200640 1 --2- 172.21.15.32:0/85127735 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55894bbb1cb0 0x55894bc4e050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:30.808 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:30.805+0000 7faaba200640 1 -- 172.21.15.32:0/85127735 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55894bbabef0 con 0x55894bb94360 2024-08-03T20:55:30.808 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:30.805+0000 7faabac00640 1 --2- 172.21.15.32:0/85127735 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55894bbb1cb0 0x55894bc4e050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-03T20:55:30.808 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:30.805+0000 7faaba200640 1 --2- 172.21.15.32:0/85127735 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55894bb94360 0x55894bb8ac20 secure :-1 s=READY pgs=1211 cs=0 l=1 rev1=1 crypto rx=0x7faaac0086f0 tx=0x7faaac0087e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:55:30.809 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:30.805+0000 7faab2a00640 1 -- 172.21.15.32:0/85127735 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faaac00a850 con 0x55894bb94360 2024-08-03T20:55:30.809 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:30.805+0000 7faab2a00640 1 -- 172.21.15.32:0/85127735 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7faaac00a9f0 con 0x55894bb94360 2024-08-03T20:55:30.809 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:30.805+0000 7faab2a00640 1 -- 172.21.15.32:0/85127735 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faaac00ae20 con 0x55894bb94360 2024-08-03T20:55:30.809 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:30.805+0000 7faac10e2f40 1 -- 172.21.15.32:0/85127735 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x55894bb8cb50 con 0x55894bb94360 2024-08-03T20:55:30.809 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:30.805+0000 7faac10e2f40 1 -- 172.21.15.32:0/85127735 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55894bbac190 con 0x55894bb94360 2024-08-03T20:55:30.810 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:30.807+0000 7faab2a00640 1 -- 172.21.15.32:0/85127735 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7faaac016430 con 0x55894bb94360 2024-08-03T20:55:30.810 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:30.807+0000 7faab2a00640 1 --2- 172.21.15.32:0/85127735 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7faa9c0759f0 0x7faa9c077f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:55:30.810 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:30.809+0000 7faab2a00640 1 -- 172.21.15.32:0/85127735 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 6370+0+0 (secure 0 0 0) 0x7faaac097fc0 con 0x55894bb94360 2024-08-03T20:55:30.811 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:30.809+0000 7faab9800640 1 --2- 172.21.15.32:0/85127735 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7faa9c0759f0 0x7faa9c077f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:55:30.811 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:30.809+0000 7faac10e2f40 1 --2- 172.21.15.32:0/85127735 >> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] conn(0x55894baddad0 0x55894bbad180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:55:30.811 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:30.809+0000 7faac10e2f40 1 -- 172.21.15.32:0/85127735 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:1 12.6 12:654134d2:::mds0_openfiles.0:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e91) -- 0x55894bb8b160 con 0x55894baddad0 2024-08-03T20:55:30.811 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:30.809+0000 7faabac00640 1 --2- 172.21.15.32:0/85127735 >> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] conn(0x55894baddad0 0x55894bbad180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:55:30.811 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:30.810+0000 7faab9800640 1 --2- 172.21.15.32:0/85127735 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7faa9c0759f0 0x7faa9c077f00 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7faaa8046500 tx=0x7faaa8004940 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:55:30.811 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:30.810+0000 7faabac00640 1 --2- 172.21.15.32:0/85127735 >> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] conn(0x55894baddad0 0x55894bbad180 crc :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:55:30.812 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:30.811+0000 7faabac00640 1 -- 172.21.15.32:0/85127735 <== osd.1 v2:172.21.15.32:6824/1551241759 1 ==== osd_op_reply(1 mds0_openfiles.0 [delete] v91'32 uv32 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7faab400b040 con 0x55894baddad0 2024-08-03T20:55:30.812 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:30.811+0000 7faac10e2f40 1 -- 172.21.15.32:0/85127735 >> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] conn(0x55894baddad0 msgr2=0x55894bbad180 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:55:30.812 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:30.811+0000 7faac10e2f40 1 --2- 172.21.15.32:0/85127735 >> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] conn(0x55894baddad0 0x55894bbad180 crc :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:30.813 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:30.811+0000 7faac10e2f40 1 -- 172.21.15.32:0/85127735 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7faa9c0759f0 msgr2=0x7faa9c077f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:55:30.813 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:30.811+0000 7faac10e2f40 1 --2- 172.21.15.32:0/85127735 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7faa9c0759f0 0x7faa9c077f00 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7faaa8046500 tx=0x7faaa8004940 comp rx=0 tx=0).stop 2024-08-03T20:55:30.813 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:30.811+0000 7faac10e2f40 1 -- 172.21.15.32:0/85127735 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55894bb94360 msgr2=0x55894bb8ac20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:55:30.813 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:30.811+0000 7faac10e2f40 1 --2- 172.21.15.32:0/85127735 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55894bb94360 0x55894bb8ac20 secure :-1 s=READY pgs=1211 cs=0 l=1 rev1=1 crypto rx=0x7faaac0086f0 tx=0x7faaac0087e0 comp rx=0 tx=0).stop 2024-08-03T20:55:30.813 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:30.811+0000 7faabac00640 1 -- 172.21.15.32:0/85127735 reap_dead start 2024-08-03T20:55:30.813 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:30.811+0000 7faac10e2f40 1 -- 172.21.15.32:0/85127735 shutdown_connections 2024-08-03T20:55:30.813 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:30.811+0000 7faac10e2f40 1 -- 172.21.15.32:0/85127735 >> 172.21.15.32:0/85127735 conn(0x55894b9fdac0 msgr2=0x55894badd3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:55:30.813 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:30.812+0000 7faac10e2f40 1 -- 172.21.15.32:0/85127735 shutdown_connections 2024-08-03T20:55:30.813 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:30.812+0000 7faac10e2f40 1 -- 172.21.15.32:0/85127735 wait complete. 2024-08-03T20:55:30.823 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs joinable true 2024-08-03T20:55:30.966 INFO:tasks.ceph.mds.a.smithi032.stderr: -38> 2024-08-03T20:55:26.212+0000 7f0bd3800640 -1 log_channel(cluster) log [ERR] : missing journal object 2024-08-03T20:55:31.057 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:31.056+0000 7f1c0d400640 1 -- 172.21.15.32:0/4281177367 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1c0806e950 msgr2=0x7f1c0806ed60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:55:31.057 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:31.056+0000 7f1c0d400640 1 --2- 172.21.15.32:0/4281177367 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1c0806e950 0x7f1c0806ed60 secure :-1 s=READY pgs=1213 cs=0 l=1 rev1=1 crypto rx=0x7f1c08070690 tx=0x7f1bf402a5f0 comp rx=0 tx=0).stop 2024-08-03T20:55:31.057 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:31.056+0000 7f1c0d400640 1 -- 172.21.15.32:0/4281177367 shutdown_connections 2024-08-03T20:55:31.057 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:31.056+0000 7f1c0d400640 1 --2- 172.21.15.32:0/4281177367 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1c0806e950 0x7f1c0806ed60 unknown :-1 s=CLOSED pgs=1213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:31.057 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:31.056+0000 7f1c0d400640 1 --2- 172.21.15.32:0/4281177367 >> v2:172.21.15.118:3300/0 conn(0x7f1c0815d2b0 0x7f1c080ced30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:31.057 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:31.056+0000 7f1c0d400640 1 -- 172.21.15.32:0/4281177367 >> 172.21.15.32:0/4281177367 conn(0x7f1c080b1bd0 msgr2=0x7f1c080b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:55:31.057 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:31.056+0000 7f1c0d400640 1 -- 172.21.15.32:0/4281177367 shutdown_connections 2024-08-03T20:55:31.057 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:31.056+0000 7f1c0d400640 1 -- 172.21.15.32:0/4281177367 wait complete. 2024-08-03T20:55:31.059 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:31.058+0000 7f1c0d400640 1 Processor -- start 2024-08-03T20:55:31.059 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:31.059+0000 7f1c0d400640 1 -- start start 2024-08-03T20:55:31.061 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:31.060+0000 7f1c0d400640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f1c0806e950 0x7f1c080ab970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:55:31.061 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:31.060+0000 7f1c0d400640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f1c080b3e80 0x7f1c08154220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:55:31.061 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:31.060+0000 7f1c0d400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1c0815d2b0 0x7f1c081f5f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:55:31.061 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:31.060+0000 7f1c0d400640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f1c080cd510 con 0x7f1c0815d2b0 2024-08-03T20:55:31.061 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:31.060+0000 7f1c0d400640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f1c080cd390 con 0x7f1c0806e950 2024-08-03T20:55:31.061 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:31.060+0000 7f1c0d400640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7f1c080b4290 con 0x7f1c080b3e80 2024-08-03T20:55:31.062 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:31.060+0000 7f1c07200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1c0815d2b0 0x7f1c081f5f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:55:31.062 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:31.061+0000 7f1c07200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1c0815d2b0 0x7f1c081f5f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:59440/0 (socket says 172.21.15.32:59440) 2024-08-03T20:55:31.062 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:31.061+0000 7f1c06800640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f1c0806e950 0x7f1c080ab970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:55:31.062 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:31.061+0000 7f1c07200640 1 -- 172.21.15.32:0/3123430072 learned_addr learned my addr 172.21.15.32:0/3123430072 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:55:31.062 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:31.061+0000 7f1c05e00640 1 --2- 172.21.15.32:0/3123430072 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f1c080b3e80 0x7f1c08154220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:55:31.063 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:31.061+0000 7f1c06800640 1 -- 172.21.15.32:0/3123430072 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f1c080b3e80 msgr2=0x7f1c08154220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:55:31.064 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:31.061+0000 7f1c06800640 1 --2- 172.21.15.32:0/3123430072 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f1c080b3e80 0x7f1c08154220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:31.064 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:31.061+0000 7f1c06800640 1 -- 172.21.15.32:0/3123430072 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1c0815d2b0 msgr2=0x7f1c081f5f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:55:31.064 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:31.061+0000 7f1c06800640 1 --2- 172.21.15.32:0/3123430072 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1c0815d2b0 0x7f1c081f5f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:31.064 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:31.061+0000 7f1c06800640 1 -- 172.21.15.32:0/3123430072 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1c081548e0 con 0x7f1c0806e950 2024-08-03T20:55:31.065 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:31.061+0000 7f1c07200640 1 --2- 172.21.15.32:0/3123430072 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1c0815d2b0 0x7f1c081f5f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-03T20:55:31.065 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:31.061+0000 7f1c06800640 1 --2- 172.21.15.32:0/3123430072 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f1c0806e950 0x7f1c080ab970 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f1bf800e6d0 tx=0x7f1bf800eb90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:55:31.065 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:31.062+0000 7f1bfea00640 1 -- 172.21.15.32:0/3123430072 <== mon.1 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1bf800cab0 con 0x7f1c0806e950 2024-08-03T20:55:31.065 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:31.062+0000 7f1bfea00640 1 -- 172.21.15.32:0/3123430072 <== mon.1 v2:172.21.15.118:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f1bf800cd40 con 0x7f1c0806e950 2024-08-03T20:55:31.065 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:31.062+0000 7f1c0d400640 1 -- 172.21.15.32:0/3123430072 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1c080aae90 con 0x7f1c0806e950 2024-08-03T20:55:31.065 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:31.062+0000 7f1bfea00640 1 -- 172.21.15.32:0/3123430072 <== mon.1 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1bf8010690 con 0x7f1c0806e950 2024-08-03T20:55:31.065 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:31.062+0000 7f1c0d400640 1 -- 172.21.15.32:0/3123430072 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1c08152240 con 0x7f1c0806e950 2024-08-03T20:55:31.065 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:31.063+0000 7f1c0d400640 1 -- 172.21.15.32:0/3123430072 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1c080b5300 con 0x7f1c0806e950 2024-08-03T20:55:31.066 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:31.066+0000 7f1bfea00640 1 -- 172.21.15.32:0/3123430072 <== mon.1 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f1bf8010830 con 0x7f1c0806e950 2024-08-03T20:55:31.067 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:31.067+0000 7f1bfea00640 1 --2- 172.21.15.32:0/3123430072 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f1be8077010 0x7f1be8079520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:55:31.067 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:31.067+0000 7f1bfea00640 1 -- 172.21.15.32:0/3123430072 <== mon.1 v2:172.21.15.118:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 6370+0+0 (secure 0 0 0) 0x7f1bf8099690 con 0x7f1c0806e950 2024-08-03T20:55:31.067 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:31.067+0000 7f1c05e00640 1 --2- 172.21.15.32:0/3123430072 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f1be8077010 0x7f1be8079520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:55:31.068 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:31.068+0000 7f1c05e00640 1 --2- 172.21.15.32:0/3123430072 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f1be8077010 0x7f1be8079520 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7f1bf4033000 tx=0x7f1bf4004730 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:55:31.070 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:31.070+0000 7f1bfea00640 1 -- 172.21.15.32:0/3123430072 <== mon.1 v2:172.21.15.118:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7f1c080b5300 con 0x7f1c0806e950 2024-08-03T20:55:31.084 INFO:tasks.ceph.mds.a.smithi032.stdout:starting mds.a at 2024-08-03T20:55:31.323 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:31.322+0000 7f1c0d400640 1 -- 172.21.15.32:0/3123430072 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "joinable", "val": "true"} v 0) -- 0x7f1c080b46f0 con 0x7f1c0806e950 2024-08-03T20:55:32.243 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:32.243+0000 7f1bfea00640 1 -- 172.21.15.32:0/3123430072 <== mon.1 v2:172.21.15.118:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "joinable", "val": "true"}]=0 cephfs marked joinable; MDS may join as newly active. v127) ==== 162+0+0 (secure 0 0 0) 0x7f1bf8061f00 con 0x7f1c0806e950 2024-08-03T20:55:32.244 INFO:teuthology.orchestra.run.smithi032.stderr:cephfs marked joinable; MDS may join as newly active. 2024-08-03T20:55:32.247 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:32.246+0000 7f1c0d400640 1 -- 172.21.15.32:0/3123430072 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f1be8077010 msgr2=0x7f1be8079520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:55:32.247 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:32.246+0000 7f1c0d400640 1 --2- 172.21.15.32:0/3123430072 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f1be8077010 0x7f1be8079520 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7f1bf4033000 tx=0x7f1bf4004730 comp rx=0 tx=0).stop 2024-08-03T20:55:32.247 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:32.246+0000 7f1c0d400640 1 -- 172.21.15.32:0/3123430072 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f1c0806e950 msgr2=0x7f1c080ab970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:55:32.247 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:32.246+0000 7f1c0d400640 1 --2- 172.21.15.32:0/3123430072 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f1c0806e950 0x7f1c080ab970 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f1bf800e6d0 tx=0x7f1bf800eb90 comp rx=0 tx=0).stop 2024-08-03T20:55:32.247 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:32.246+0000 7f1c0d400640 1 -- 172.21.15.32:0/3123430072 shutdown_connections 2024-08-03T20:55:32.247 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:32.246+0000 7f1c0d400640 1 --2- 172.21.15.32:0/3123430072 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f1be8077010 0x7f1be8079520 unknown :-1 s=CLOSED pgs=502 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:32.247 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:32.246+0000 7f1c0d400640 1 --2- 172.21.15.32:0/3123430072 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1c0815d2b0 0x7f1c081f5f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:32.247 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:32.246+0000 7f1c0d400640 1 --2- 172.21.15.32:0/3123430072 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f1c080b3e80 0x7f1c08154220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:32.247 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:32.246+0000 7f1c0d400640 1 --2- 172.21.15.32:0/3123430072 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f1c0806e950 0x7f1c080ab970 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:32.248 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:32.246+0000 7f1c0d400640 1 -- 172.21.15.32:0/3123430072 >> 172.21.15.32:0/3123430072 conn(0x7f1c080b1bd0 msgr2=0x7f1c080d0920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:55:32.248 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:32.246+0000 7f1c0d400640 1 -- 172.21.15.32:0/3123430072 shutdown_connections 2024-08-03T20:55:32.248 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:32.246+0000 7f1c0d400640 1 -- 172.21.15.32:0/3123430072 wait complete. 2024-08-03T20:55:32.251 INFO:tasks.ceph.mds.a.smithi032.stderr:2024-08-03T20:55:32.251+0000 7f64fda00640 -1 mds.0.openfiles _load_finish got (2) No such file or directory 2024-08-03T20:55:32.682 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-08-03T20:55:32.922 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:32.921+0000 7f4cbe600640 1 -- 172.21.15.32:0/3493215468 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f4cb80acd80 msgr2=0x7f4cb80ad190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:55:32.922 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:32.921+0000 7f4cbe600640 1 --2- 172.21.15.32:0/3493215468 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f4cb80acd80 0x7f4cb80ad190 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f4ca4009830 tx=0x7f4ca402a480 comp rx=0 tx=0).stop 2024-08-03T20:55:32.923 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:32.921+0000 7f4cbe600640 1 -- 172.21.15.32:0/3493215468 shutdown_connections 2024-08-03T20:55:32.923 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:32.921+0000 7f4cbe600640 1 --2- 172.21.15.32:0/3493215468 >> v2:172.21.15.118:3300/0 conn(0x7f4cb806f440 0x7f4cb806f850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:32.923 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:32.921+0000 7f4cbe600640 1 --2- 172.21.15.32:0/3493215468 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f4cb80acd80 0x7f4cb80ad190 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:32.923 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:32.922+0000 7f4cbe600640 1 -- 172.21.15.32:0/3493215468 >> 172.21.15.32:0/3493215468 conn(0x7f4cb80b2120 msgr2=0x7f4cb80cfd60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:55:32.923 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:32.922+0000 7f4cbe600640 1 -- 172.21.15.32:0/3493215468 shutdown_connections 2024-08-03T20:55:32.923 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:32.922+0000 7f4cbe600640 1 -- 172.21.15.32:0/3493215468 wait complete. 2024-08-03T20:55:32.924 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:32.924+0000 7f4cbe600640 1 Processor -- start 2024-08-03T20:55:32.924 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:32.924+0000 7f4cbe600640 1 -- start start 2024-08-03T20:55:32.926 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:32.925+0000 7f4cbe600640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4cb806f440 0x7f4cb80b3640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:55:32.926 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:32.925+0000 7f4cbe600640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f4cb80acd80 0x7f4cb80c55b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:55:32.926 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:32.925+0000 7f4cbe600640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f4cb80af020 0x7f4cb814f160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:55:32.926 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:32.925+0000 7f4cbe600640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f4cb81602f0 con 0x7f4cb806f440 2024-08-03T20:55:32.926 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:32.925+0000 7f4cbe600640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f4cb8160170 con 0x7f4cb80acd80 2024-08-03T20:55:32.926 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:32.925+0000 7f4cbe600640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7f4cb8160470 con 0x7f4cb80af020 2024-08-03T20:55:32.926 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:32.926+0000 7f4cbd200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4cb806f440 0x7f4cb80b3640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:55:32.926 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:32.926+0000 7f4cbd200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4cb806f440 0x7f4cb80b3640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:59476/0 (socket says 172.21.15.32:59476) 2024-08-03T20:55:32.927 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:32.926+0000 7f4cbd200640 1 -- 172.21.15.32:0/1848412087 learned_addr learned my addr 172.21.15.32:0/1848412087 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:55:32.928 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:32.926+0000 7f4cbdc00640 1 --2- 172.21.15.32:0/1848412087 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f4cb80af020 0x7f4cb814f160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:55:32.928 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:32.926+0000 7f4cb7e00640 1 --2- 172.21.15.32:0/1848412087 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f4cb80acd80 0x7f4cb80c55b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:55:32.928 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:32.926+0000 7f4cbd200640 1 -- 172.21.15.32:0/1848412087 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f4cb80af020 msgr2=0x7f4cb814f160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:55:32.928 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:32.926+0000 7f4cbd200640 1 --2- 172.21.15.32:0/1848412087 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f4cb80af020 0x7f4cb814f160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:32.928 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:32.926+0000 7f4cbd200640 1 -- 172.21.15.32:0/1848412087 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f4cb80acd80 msgr2=0x7f4cb80c55b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:55:32.928 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:32.926+0000 7f4cbd200640 1 --2- 172.21.15.32:0/1848412087 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f4cb80acd80 0x7f4cb80c55b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:32.928 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:32.926+0000 7f4cbd200640 1 -- 172.21.15.32:0/1848412087 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4cb80c5af0 con 0x7f4cb806f440 2024-08-03T20:55:32.928 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:32.927+0000 7f4cbd200640 1 --2- 172.21.15.32:0/1848412087 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4cb806f440 0x7f4cb80b3640 secure :-1 s=READY pgs=1214 cs=0 l=1 rev1=1 crypto rx=0x7f4cac00c460 tx=0x7f4cac00c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:55:32.929 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:32.927+0000 7f4cb4c00640 1 -- 172.21.15.32:0/1848412087 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4cac016020 con 0x7f4cb806f440 2024-08-03T20:55:32.929 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:32.927+0000 7f4cbe600640 1 -- 172.21.15.32:0/1848412087 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4cb80c4f00 con 0x7f4cb806f440 2024-08-03T20:55:32.929 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:32.927+0000 7f4cb4c00640 1 -- 172.21.15.32:0/1848412087 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f4cac012940 con 0x7f4cb806f440 2024-08-03T20:55:32.929 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:32.927+0000 7f4cb4c00640 1 -- 172.21.15.32:0/1848412087 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4cac012d90 con 0x7f4cb806f440 2024-08-03T20:55:32.929 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:32.927+0000 7f4cbe600640 1 -- 172.21.15.32:0/1848412087 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4cb80b2e40 con 0x7f4cb806f440 2024-08-03T20:55:32.929 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:32.928+0000 7f4cbe600640 1 -- 172.21.15.32:0/1848412087 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4cb80b2c30 con 0x7f4cb806f440 2024-08-03T20:55:32.929 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:32.929+0000 7f4cb4c00640 1 -- 172.21.15.32:0/1848412087 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f4cac019020 con 0x7f4cb806f440 2024-08-03T20:55:32.930 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:32.930+0000 7f4cb4c00640 1 --2- 172.21.15.32:0/1848412087 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f4c98077010 0x7f4c98079520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:55:32.931 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:32.930+0000 7f4cb4c00640 1 -- 172.21.15.32:0/1848412087 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 6370+0+0 (secure 0 0 0) 0x7f4cac098620 con 0x7f4cb806f440 2024-08-03T20:55:32.931 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:32.931+0000 7f4cb7e00640 1 --2- 172.21.15.32:0/1848412087 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f4c98077010 0x7f4c98079520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:55:32.932 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:32.932+0000 7f4cb7e00640 1 --2- 172.21.15.32:0/1848412087 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f4c98077010 0x7f4c98079520 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7f4ca4034000 tx=0x7f4ca4004730 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:55:32.935 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:32.935+0000 7f4cb4c00640 1 -- 172.21.15.32:0/1848412087 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7f4cac020070 con 0x7f4cb806f440 2024-08-03T20:55:33.178 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:33.177+0000 7f4cbe600640 1 -- 172.21.15.32:0/1848412087 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f4cb80ad190 con 0x7f4cb806f440 2024-08-03T20:55:33.178 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:33.178+0000 7f4cb4c00640 1 -- 172.21.15.32:0/1848412087 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 128 v128) ==== 95+0+5225 (secure 0 0 0) 0x7f4cac0614f0 con 0x7f4cb806f440 2024-08-03T20:55:33.179 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-03T20:55:33.179 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":128,"btime":"2024-08-03T20:55:32:242459+0000","default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6439,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6835/531080160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6834","nonce":531080160},{"type":"v1","addr":"172.21.15.118:6835","nonce":531080160}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":84},{"gid":6454,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6837/2796882319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6836","nonce":2796882319},{"type":"v1","addr":"172.21.15.118:6837","nonce":2796882319}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":86},{"gid":6813,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6834/2686933890","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":2686933890},{"type":"v1","addr":"172.21.15.32:6834","nonce":2686933890}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":85}],"filesystems":[{"mdsmap":{"epoch":128,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-08-03T20:54:26.950518+0000","modified":"2024-08-03T20:55:32.242444+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":91,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7273},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7273":{"gid":7273,"name":"a","rank":0,"incarnation":128,"state":"up:replay","state_seq":1,"addr":"172.21.15.32:6835/3191654276","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6833","nonce":3191654276},{"type":"v1","addr":"172.21.15.32:6835","nonce":3191654276}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":6}]} 2024-08-03T20:55:33.179 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 128 2024-08-03T20:55:33.181 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:33.180+0000 7f4cbe600640 1 -- 172.21.15.32:0/1848412087 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f4c98077010 msgr2=0x7f4c98079520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:55:33.181 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:33.180+0000 7f4cbe600640 1 --2- 172.21.15.32:0/1848412087 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f4c98077010 0x7f4c98079520 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7f4ca4034000 tx=0x7f4ca4004730 comp rx=0 tx=0).stop 2024-08-03T20:55:33.182 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:33.181+0000 7f4cbe600640 1 -- 172.21.15.32:0/1848412087 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4cb806f440 msgr2=0x7f4cb80b3640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:55:33.182 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:33.181+0000 7f4cbe600640 1 --2- 172.21.15.32:0/1848412087 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4cb806f440 0x7f4cb80b3640 secure :-1 s=READY pgs=1214 cs=0 l=1 rev1=1 crypto rx=0x7f4cac00c460 tx=0x7f4cac00c920 comp rx=0 tx=0).stop 2024-08-03T20:55:33.182 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:33.181+0000 7f4cbe600640 1 -- 172.21.15.32:0/1848412087 shutdown_connections 2024-08-03T20:55:33.182 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:33.181+0000 7f4cbe600640 1 --2- 172.21.15.32:0/1848412087 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f4c98077010 0x7f4c98079520 unknown :-1 s=CLOSED pgs=504 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:33.182 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:33.181+0000 7f4cbe600640 1 --2- 172.21.15.32:0/1848412087 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f4cb80acd80 0x7f4cb80c55b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:33.182 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:33.181+0000 7f4cbe600640 1 --2- 172.21.15.32:0/1848412087 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f4cb80af020 0x7f4cb814f160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:33.182 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:33.181+0000 7f4cbe600640 1 --2- 172.21.15.32:0/1848412087 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4cb806f440 0x7f4cb80b3640 unknown :-1 s=CLOSED pgs=1214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:33.182 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:33.181+0000 7f4cbe600640 1 -- 172.21.15.32:0/1848412087 >> 172.21.15.32:0/1848412087 conn(0x7f4cb80b2120 msgr2=0x7f4cb80ca390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:55:33.182 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:33.181+0000 7f4cbe600640 1 -- 172.21.15.32:0/1848412087 shutdown_connections 2024-08-03T20:55:33.182 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:33.181+0000 7f4cbe600640 1 -- 172.21.15.32:0/1848412087 wait complete. 2024-08-03T20:55:33.203 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 128, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-08-03T20:54:26.950518+0000', 'modified': '2024-08-03T20:55:32.242444+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 91, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 7273}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7273': {'gid': 7273, 'name': 'a', 'rank': 0, 'incarnation': 128, 'state': 'up:replay', 'state_seq': 1, 'addr': '172.21.15.32:6835/3191654276', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6833', 'nonce': 3191654276}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 3191654276}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [13], 'metadata_pool': 12, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 0, 'qdb_cluster': []} 2024-08-03T20:55:33.204 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_7273:up:replay 2024-08-03T20:55:33.204 DEBUG:tasks.ceph_test_case:wait_until_true: waiting (timeout=60 retry_count=0)... 2024-08-03T20:55:36.246 INFO:tasks.ceph.mds.a.smithi032.stderr:2024-08-03T20:55:36.245+0000 7f6502a00640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-08-03T20:55:38.105 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-08-03T20:55:38.110 DEBUG:teuthology.orchestra.run.smithi118:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-08-03T20:55:38.607 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-08-03T20:55:38.846 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:38.845+0000 7faf7be00640 1 -- 172.21.15.32:0/2588446961 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faf74073e60 msgr2=0x7faf74074270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:55:38.847 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:38.845+0000 7faf7be00640 1 --2- 172.21.15.32:0/2588446961 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faf74073e60 0x7faf74074270 secure :-1 s=READY pgs=1215 cs=0 l=1 rev1=1 crypto rx=0x7faf74071700 tx=0x7faf6002a560 comp rx=0 tx=0).stop 2024-08-03T20:55:38.847 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:38.846+0000 7faf7be00640 1 -- 172.21.15.32:0/2588446961 shutdown_connections 2024-08-03T20:55:38.847 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:38.846+0000 7faf7be00640 1 --2- 172.21.15.32:0/2588446961 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7faf740747b0 0x7faf740cc8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:38.847 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:38.846+0000 7faf7be00640 1 --2- 172.21.15.32:0/2588446961 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faf74073e60 0x7faf74074270 unknown :-1 s=CLOSED pgs=1215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:38.847 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:38.846+0000 7faf7be00640 1 --2- 172.21.15.32:0/2588446961 >> v2:172.21.15.118:3300/0 conn(0x7faf7406e930 0x7faf74144f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:38.847 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:38.846+0000 7faf7be00640 1 -- 172.21.15.32:0/2588446961 >> 172.21.15.32:0/2588446961 conn(0x7faf740b1bd0 msgr2=0x7faf740b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:55:38.847 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:38.846+0000 7faf7be00640 1 -- 172.21.15.32:0/2588446961 shutdown_connections 2024-08-03T20:55:38.847 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:38.846+0000 7faf7be00640 1 -- 172.21.15.32:0/2588446961 wait complete. 2024-08-03T20:55:38.848 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:38.848+0000 7faf7be00640 1 Processor -- start 2024-08-03T20:55:38.849 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:38.848+0000 7faf7be00640 1 -- start start 2024-08-03T20:55:38.851 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:38.850+0000 7faf7be00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7faf7406e930 0x7faf7413e950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:55:38.851 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:38.850+0000 7faf7be00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faf74073e60 0x7faf7415e420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:55:38.851 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:38.850+0000 7faf7be00640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7faf740747b0 0x7faf7415e960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:55:38.851 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:38.850+0000 7faf7be00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7faf740cf9c0 con 0x7faf74073e60 2024-08-03T20:55:38.851 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:38.850+0000 7faf7be00640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7faf740cf840 con 0x7faf7406e930 2024-08-03T20:55:38.852 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:38.850+0000 7faf7be00640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7faf740cce20 con 0x7faf740747b0 2024-08-03T20:55:38.852 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:38.850+0000 7faf78a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faf74073e60 0x7faf7415e420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:55:38.852 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:38.850+0000 7faf78a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faf74073e60 0x7faf7415e420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:59484/0 (socket says 172.21.15.32:59484) 2024-08-03T20:55:38.852 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:38.850+0000 7faf78a00640 1 -- 172.21.15.32:0/1850002307 learned_addr learned my addr 172.21.15.32:0/1850002307 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:55:38.852 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:38.850+0000 7faf79e00640 1 --2- 172.21.15.32:0/1850002307 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7faf740747b0 0x7faf7415e960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:55:38.852 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:38.850+0000 7faf79400640 1 --2- 172.21.15.32:0/1850002307 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7faf7406e930 0x7faf7413e950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:55:38.852 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:38.850+0000 7faf78a00640 1 -- 172.21.15.32:0/1850002307 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7faf740747b0 msgr2=0x7faf7415e960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:55:38.852 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:38.850+0000 7faf78a00640 1 --2- 172.21.15.32:0/1850002307 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7faf740747b0 0x7faf7415e960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:38.852 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:38.850+0000 7faf78a00640 1 -- 172.21.15.32:0/1850002307 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7faf7406e930 msgr2=0x7faf7413e950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:55:38.852 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:38.850+0000 7faf78a00640 1 --2- 172.21.15.32:0/1850002307 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7faf7406e930 0x7faf7413e950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:38.853 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:38.850+0000 7faf78a00640 1 -- 172.21.15.32:0/1850002307 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faf74144990 con 0x7faf74073e60 2024-08-03T20:55:38.853 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:38.851+0000 7faf78a00640 1 --2- 172.21.15.32:0/1850002307 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faf74073e60 0x7faf7415e420 secure :-1 s=READY pgs=1216 cs=0 l=1 rev1=1 crypto rx=0x7faf60000c00 tx=0x7faf60003910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:55:38.854 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:38.851+0000 7faf71600640 1 -- 172.21.15.32:0/1850002307 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf6003e030 con 0x7faf74073e60 2024-08-03T20:55:38.854 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:38.851+0000 7faf71600640 1 -- 172.21.15.32:0/1850002307 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7faf60003e00 con 0x7faf74073e60 2024-08-03T20:55:38.854 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:38.851+0000 7faf71600640 1 -- 172.21.15.32:0/1850002307 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf60035040 con 0x7faf74073e60 2024-08-03T20:55:38.854 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:38.851+0000 7faf7be00640 1 -- 172.21.15.32:0/1850002307 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faf74141bd0 con 0x7faf74073e60 2024-08-03T20:55:38.854 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:38.851+0000 7faf7be00640 1 -- 172.21.15.32:0/1850002307 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faf74142220 con 0x7faf74073e60 2024-08-03T20:55:38.854 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:38.852+0000 7faf7be00640 1 -- 172.21.15.32:0/1850002307 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faf740ce3b0 con 0x7faf74073e60 2024-08-03T20:55:38.854 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:38.854+0000 7faf71600640 1 -- 172.21.15.32:0/1850002307 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7faf60041030 con 0x7faf74073e60 2024-08-03T20:55:38.856 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:38.855+0000 7faf71600640 1 --2- 172.21.15.32:0/1850002307 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7faf54077010 0x7faf54079520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:55:38.856 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:38.855+0000 7faf71600640 1 -- 172.21.15.32:0/1850002307 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 6370+0+0 (secure 0 0 0) 0x7faf600b9290 con 0x7faf74073e60 2024-08-03T20:55:38.856 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:38.855+0000 7faf79400640 1 --2- 172.21.15.32:0/1850002307 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7faf54077010 0x7faf54079520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:55:38.857 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:38.856+0000 7faf79400640 1 --2- 172.21.15.32:0/1850002307 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7faf54077010 0x7faf54079520 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7faf68005fd0 tx=0x7faf6800a250 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:55:38.860 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:38.859+0000 7faf71600640 1 -- 172.21.15.32:0/1850002307 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7faf740ce3b0 con 0x7faf74073e60 2024-08-03T20:55:39.102 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:39.101+0000 7faf7be00640 1 -- 172.21.15.32:0/1850002307 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7faf74074270 con 0x7faf74073e60 2024-08-03T20:55:39.103 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:39.102+0000 7faf71600640 1 -- 172.21.15.32:0/1850002307 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 131 v131) ==== 95+0+5232 (secure 0 0 0) 0x7faf6008a420 con 0x7faf74073e60 2024-08-03T20:55:39.103 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-03T20:55:39.103 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":131,"btime":"2024-08-03T20:55:35:252566+0000","default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6439,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6835/531080160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6834","nonce":531080160},{"type":"v1","addr":"172.21.15.118:6835","nonce":531080160}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":84},{"gid":6454,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6837/2796882319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6836","nonce":2796882319},{"type":"v1","addr":"172.21.15.118:6837","nonce":2796882319}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":86},{"gid":6813,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6834/2686933890","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":2686933890},{"type":"v1","addr":"172.21.15.32:6834","nonce":2686933890}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":85}],"filesystems":[{"mdsmap":{"epoch":131,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-08-03T20:54:26.950518+0000","modified":"2024-08-03T20:55:35.252556+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":91,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7273},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7273":{"gid":7273,"name":"a","rank":0,"incarnation":128,"state":"up:active","state_seq":4,"addr":"172.21.15.32:6835/3191654276","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6833","nonce":3191654276},{"type":"v1","addr":"172.21.15.32:6835","nonce":3191654276}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7273,"qdb_cluster":[7273]},"id":6}]} 2024-08-03T20:55:39.103 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 131 2024-08-03T20:55:39.106 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:39.105+0000 7faf7be00640 1 -- 172.21.15.32:0/1850002307 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7faf54077010 msgr2=0x7faf54079520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:55:39.106 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:39.105+0000 7faf7be00640 1 --2- 172.21.15.32:0/1850002307 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7faf54077010 0x7faf54079520 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7faf68005fd0 tx=0x7faf6800a250 comp rx=0 tx=0).stop 2024-08-03T20:55:39.106 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:39.105+0000 7faf7be00640 1 -- 172.21.15.32:0/1850002307 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faf74073e60 msgr2=0x7faf7415e420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:55:39.106 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:39.105+0000 7faf7be00640 1 --2- 172.21.15.32:0/1850002307 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faf74073e60 0x7faf7415e420 secure :-1 s=READY pgs=1216 cs=0 l=1 rev1=1 crypto rx=0x7faf60000c00 tx=0x7faf60003910 comp rx=0 tx=0).stop 2024-08-03T20:55:39.106 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:39.105+0000 7faf7be00640 1 -- 172.21.15.32:0/1850002307 shutdown_connections 2024-08-03T20:55:39.106 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:39.105+0000 7faf7be00640 1 --2- 172.21.15.32:0/1850002307 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7faf54077010 0x7faf54079520 unknown :-1 s=CLOSED pgs=505 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:39.106 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:39.105+0000 7faf7be00640 1 --2- 172.21.15.32:0/1850002307 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7faf740747b0 0x7faf7415e960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:39.106 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:39.105+0000 7faf7be00640 1 --2- 172.21.15.32:0/1850002307 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faf74073e60 0x7faf7415e420 unknown :-1 s=CLOSED pgs=1216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:39.106 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:39.105+0000 7faf7be00640 1 --2- 172.21.15.32:0/1850002307 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7faf7406e930 0x7faf7413e950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:39.106 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:39.105+0000 7faf7be00640 1 -- 172.21.15.32:0/1850002307 >> 172.21.15.32:0/1850002307 conn(0x7faf740b1bd0 msgr2=0x7faf7413fbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:55:39.107 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:39.105+0000 7faf7be00640 1 -- 172.21.15.32:0/1850002307 shutdown_connections 2024-08-03T20:55:39.107 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:39.105+0000 7faf7be00640 1 -- 172.21.15.32:0/1850002307 wait complete. 2024-08-03T20:55:39.127 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 131, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-08-03T20:54:26.950518+0000', 'modified': '2024-08-03T20:55:35.252556+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 91, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 7273}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7273': {'gid': 7273, 'name': 'a', 'rank': 0, 'incarnation': 128, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.32:6835/3191654276', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6833', 'nonce': 3191654276}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 3191654276}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [13], 'metadata_pool': 12, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 7273, 'qdb_cluster': [7273]} 2024-08-03T20:55:39.128 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-08-03T20:55:39.128 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2024-08-03T20:55:39.358 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:39.357+0000 7fcd4f200640 1 -- 172.21.15.32:0/3394298845 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fcd500ceef0 msgr2=0x7fcd500b72e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:55:39.358 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:39.357+0000 7fcd4f200640 1 --2- 172.21.15.32:0/3394298845 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fcd500ceef0 0x7fcd500b72e0 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7fcd38009830 tx=0x7fcd3802a470 comp rx=0 tx=0).stop 2024-08-03T20:55:39.359 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:39.357+0000 7fcd4f200640 1 -- 172.21.15.32:0/3394298845 shutdown_connections 2024-08-03T20:55:39.359 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:39.357+0000 7fcd4f200640 1 --2- 172.21.15.32:0/3394298845 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fcd500d29b0 0x7fcd500b1f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:39.359 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:39.357+0000 7fcd4f200640 1 --2- 172.21.15.32:0/3394298845 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fcd500ceef0 0x7fcd500b72e0 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:39.359 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:39.357+0000 7fcd4f200640 1 -- 172.21.15.32:0/3394298845 >> 172.21.15.32:0/3394298845 conn(0x7fcd500d1e90 msgr2=0x7fcd500d22a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:55:39.359 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:39.358+0000 7fcd4f200640 1 -- 172.21.15.32:0/3394298845 shutdown_connections 2024-08-03T20:55:39.359 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:39.358+0000 7fcd4f200640 1 -- 172.21.15.32:0/3394298845 wait complete. 2024-08-03T20:55:39.360 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:39.360+0000 7fcd4f200640 1 Processor -- start 2024-08-03T20:55:39.360 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:39.360+0000 7fcd4f200640 1 -- start start 2024-08-03T20:55:39.362 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:39.361+0000 7fcd4f200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcd500b1300 0x7fcd500cb720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:55:39.362 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:39.361+0000 7fcd4f200640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fcd500d29b0 0x7fcd50148fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:55:39.362 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:39.361+0000 7fcd4f200640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fcd500afc00 0x7fcd500b0010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:55:39.362 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:39.361+0000 7fcd4f200640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fcd500b2a50 con 0x7fcd500b1300 2024-08-03T20:55:39.362 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:39.361+0000 7fcd4f200640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7fcd500b28d0 con 0x7fcd500afc00 2024-08-03T20:55:39.362 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:39.361+0000 7fcd4f200640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7fcd500b2bd0 con 0x7fcd500d29b0 2024-08-03T20:55:39.362 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:39.362+0000 7fcd4de00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcd500b1300 0x7fcd500cb720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:55:39.362 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:39.362+0000 7fcd4de00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcd500b1300 0x7fcd500cb720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:59488/0 (socket says 172.21.15.32:59488) 2024-08-03T20:55:39.362 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:39.362+0000 7fcd4de00640 1 -- 172.21.15.32:0/767965005 learned_addr learned my addr 172.21.15.32:0/767965005 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:55:39.364 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:39.362+0000 7fcd4e800640 1 --2- 172.21.15.32:0/767965005 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fcd500afc00 0x7fcd500b0010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:55:39.364 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:39.362+0000 7fcd47e00640 1 --2- 172.21.15.32:0/767965005 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fcd500d29b0 0x7fcd50148fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:55:39.364 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:39.362+0000 7fcd4de00640 1 -- 172.21.15.32:0/767965005 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fcd500d29b0 msgr2=0x7fcd50148fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:55:39.364 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:39.362+0000 7fcd4de00640 1 --2- 172.21.15.32:0/767965005 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fcd500d29b0 0x7fcd50148fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:39.364 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:39.362+0000 7fcd4de00640 1 -- 172.21.15.32:0/767965005 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fcd500afc00 msgr2=0x7fcd500b0010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:55:39.364 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:39.362+0000 7fcd4de00640 1 --2- 172.21.15.32:0/767965005 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fcd500afc00 0x7fcd500b0010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:39.364 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:39.362+0000 7fcd4de00640 1 -- 172.21.15.32:0/767965005 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcd500b0550 con 0x7fcd500b1300 2024-08-03T20:55:39.365 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:39.362+0000 7fcd4de00640 1 --2- 172.21.15.32:0/767965005 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcd500b1300 0x7fcd500cb720 secure :-1 s=READY pgs=1218 cs=0 l=1 rev1=1 crypto rx=0x7fcd4000c430 tx=0x7fcd4000c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:55:39.365 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:39.363+0000 7fcd46000640 1 -- 172.21.15.32:0/767965005 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd40011790 con 0x7fcd500b1300 2024-08-03T20:55:39.365 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:39.363+0000 7fcd46000640 1 -- 172.21.15.32:0/767965005 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fcd40017070 con 0x7fcd500b1300 2024-08-03T20:55:39.365 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:39.363+0000 7fcd46000640 1 -- 172.21.15.32:0/767965005 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd40011ce0 con 0x7fcd500b1300 2024-08-03T20:55:39.365 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:39.363+0000 7fcd4f200640 1 -- 172.21.15.32:0/767965005 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcd50155420 con 0x7fcd500b1300 2024-08-03T20:55:39.365 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:39.363+0000 7fcd4f200640 1 -- 172.21.15.32:0/767965005 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcd5015d010 con 0x7fcd500b1300 2024-08-03T20:55:39.366 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:39.365+0000 7fcd46000640 1 -- 172.21.15.32:0/767965005 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7fcd4001c460 con 0x7fcd500b1300 2024-08-03T20:55:39.367 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:39.366+0000 7fcd46000640 1 --2- 172.21.15.32:0/767965005 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fcd2c077040 0x7fcd2c079550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:55:39.367 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:39.366+0000 7fcd46000640 1 -- 172.21.15.32:0/767965005 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 6370+0+0 (secure 0 0 0) 0x7fcd40098ab0 con 0x7fcd500b1300 2024-08-03T20:55:39.367 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:39.367+0000 7fcd5bbb3740 1 Processor -- start 2024-08-03T20:55:39.367 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:39.367+0000 7fcd5bbb3740 1 -- start start 2024-08-03T20:55:39.368 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:39.367+0000 7fcd47e00640 1 --2- 172.21.15.32:0/767965005 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fcd2c077040 0x7fcd2c079550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:55:39.368 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:39.368+0000 7fcd47e00640 1 --2- 172.21.15.32:0/767965005 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fcd2c077040 0x7fcd2c079550 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7fcd38034000 tx=0x7fcd38004730 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:55:39.369 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:39.368+0000 7fcd5bbb3740 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55610ec87270 0x55610f0742c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:55:39.369 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:39.368+0000 7fcd5bbb3740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55610ec5e3b0 0x55610f056f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:55:39.369 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:39.368+0000 7fcd5bbb3740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55610ec66590 0x55610f0574d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:55:39.369 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:39.368+0000 7fcd5bbb3740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55610ef9fa60 con 0x55610ec5e3b0 2024-08-03T20:55:39.370 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:39.368+0000 7fcd5bbb3740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x55610ef8ef60 con 0x55610ec66590 2024-08-03T20:55:39.370 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:39.368+0000 7fcd5bbb3740 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x55610ef92fa0 con 0x55610ec87270 2024-08-03T20:55:39.370 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:39.368+0000 7fcd47e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55610ec5e3b0 0x55610f056f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:55:39.370 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:39.368+0000 7fcd4e800640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55610ec66590 0x55610f0574d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:55:39.370 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:39.369+0000 7fcd47e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55610ec5e3b0 0x55610f056f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:59494/0 (socket says 172.21.15.32:59494) 2024-08-03T20:55:39.370 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:39.369+0000 7fcd4de00640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55610ec87270 0x55610f0742c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:55:39.370 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:39.369+0000 7fcd47e00640 1 -- 172.21.15.32:0/406161840 learned_addr learned my addr 172.21.15.32:0/406161840 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:55:39.370 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:39.369+0000 7fcd4de00640 1 -- 172.21.15.32:0/406161840 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55610ec66590 msgr2=0x55610f0574d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:55:39.370 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:39.369+0000 7fcd4de00640 1 --2- 172.21.15.32:0/406161840 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55610ec66590 0x55610f0574d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:39.370 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:39.369+0000 7fcd4de00640 1 -- 172.21.15.32:0/406161840 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55610ec5e3b0 msgr2=0x55610f056f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:55:39.370 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:39.369+0000 7fcd4de00640 1 --2- 172.21.15.32:0/406161840 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55610ec5e3b0 0x55610f056f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:39.370 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:39.369+0000 7fcd4de00640 1 -- 172.21.15.32:0/406161840 --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55610ecf2850 con 0x55610ec87270 2024-08-03T20:55:39.372 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:39.369+0000 7fcd4de00640 1 --2- 172.21.15.32:0/406161840 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55610ec87270 0x55610f0742c0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fcd40096c70 tx=0x7fcd40060cb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4d7dcc5cc5f1be52 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:55:39.372 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:39.370+0000 7fcd3e600640 1 -- 172.21.15.32:0/406161840 <== mon.2 v2:172.21.15.118:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd40061270 con 0x55610ec87270 2024-08-03T20:55:39.372 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:39.370+0000 7fcd3e600640 1 -- 172.21.15.32:0/406161840 <== mon.2 v2:172.21.15.118:3301/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fcd40065870 con 0x55610ec87270 2024-08-03T20:55:39.372 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:39.370+0000 7fcd3e600640 1 -- 172.21.15.32:0/406161840 <== mon.2 v2:172.21.15.118:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd400606d0 con 0x55610ec87270 2024-08-03T20:55:39.372 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:39.370+0000 7fcd5bbb3740 1 -- 172.21.15.32:0/406161840 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55610ec87270 msgr2=0x55610f0742c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:55:39.372 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:39.370+0000 7fcd5bbb3740 1 --2- 172.21.15.32:0/406161840 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55610ec87270 0x55610f0742c0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fcd40096c70 tx=0x7fcd40060cb0 comp rx=0 tx=0).stop 2024-08-03T20:55:39.372 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:39.370+0000 7fcd5bbb3740 1 -- 172.21.15.32:0/406161840 shutdown_connections 2024-08-03T20:55:39.372 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:39.370+0000 7fcd5bbb3740 1 --2- 172.21.15.32:0/406161840 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55610ec66590 0x55610f0574d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:39.372 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:39.370+0000 7fcd5bbb3740 1 --2- 172.21.15.32:0/406161840 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55610ec5e3b0 0x55610f056f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:39.372 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:39.370+0000 7fcd5bbb3740 1 --2- 172.21.15.32:0/406161840 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55610ec87270 0x55610f0742c0 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:39.372 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:39.370+0000 7fcd5bbb3740 1 -- 172.21.15.32:0/406161840 >> 172.21.15.32:0/406161840 conn(0x55610ec7b3c0 msgr2=0x55610f02d1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:55:39.372 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:39.371+0000 7fcd5bbb3740 1 -- 172.21.15.32:0/406161840 shutdown_connections 2024-08-03T20:55:39.373 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:39.371+0000 7fcd5bbb3740 1 -- 172.21.15.32:0/406161840 wait complete. 2024-08-03T20:55:39.373 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:39.372+0000 7fcd5bbb3740 1 -- start start 2024-08-03T20:55:39.373 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:39.372+0000 7fcd5bbb3740 1 Processor -- start 2024-08-03T20:55:39.374 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:39.373+0000 7fcd5bbb3740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55610ec5e3b0 0x55610f085370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:55:39.374 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:39.373+0000 7fcd5bbb3740 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55610ec66590 0x55610f045620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:55:39.374 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:39.373+0000 7fcd5bbb3740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55610ec87270 0x55610f071000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:55:39.374 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:39.373+0000 7fcd5bbb3740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55610ed01620 con 0x55610ec87270 2024-08-03T20:55:39.374 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:39.373+0000 7fcd5bbb3740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x55610ef2c160 con 0x55610ec5e3b0 2024-08-03T20:55:39.375 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:39.373+0000 7fcd5bbb3740 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x55610f06d280 con 0x55610ec66590 2024-08-03T20:55:39.375 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:39.373+0000 7fcd5bbb3740 10 client.0 _ll_register_callbacks cb 0x55610ef3b6b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-03T20:55:39.375 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:39.373+0000 7fcd4e800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55610ec87270 0x55610f071000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:55:39.375 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:39.374+0000 7fcd4e800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55610ec87270 0x55610f071000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:59498/0 (socket says 172.21.15.32:59498) 2024-08-03T20:55:39.375 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:39.374+0000 7fcd4e800640 1 -- 172.21.15.32:0/3098594718 learned_addr learned my addr 172.21.15.32:0/3098594718 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:55:39.375 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:39.374+0000 7fcd47e00640 1 --2- 172.21.15.32:0/3098594718 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55610ec66590 0x55610f045620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:55:39.375 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:39.374+0000 7fcd4de00640 1 --2- 172.21.15.32:0/3098594718 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55610ec5e3b0 0x55610f085370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:55:39.375 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:39.374+0000 7fcd4e800640 1 -- 172.21.15.32:0/3098594718 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55610ec66590 msgr2=0x55610f045620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:55:39.375 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:39.374+0000 7fcd4e800640 1 --2- 172.21.15.32:0/3098594718 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55610ec66590 0x55610f045620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:39.375 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:39.374+0000 7fcd4e800640 1 -- 172.21.15.32:0/3098594718 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55610ec5e3b0 msgr2=0x55610f085370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:55:39.375 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:39.374+0000 7fcd4e800640 1 --2- 172.21.15.32:0/3098594718 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55610ec5e3b0 0x55610f085370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:39.375 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:39.374+0000 7fcd4e800640 1 -- 172.21.15.32:0/3098594718 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55610ed696b0 con 0x55610ec87270 2024-08-03T20:55:39.378 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:39.374+0000 7fcd4e800640 1 --2- 172.21.15.32:0/3098594718 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55610ec87270 0x55610f071000 secure :-1 s=READY pgs=1219 cs=0 l=1 rev1=1 crypto rx=0x7fcd5015cba0 tx=0x7fcd48011db0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c39dd67ad064080d server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:55:39.378 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:39.374+0000 7fcd32a00640 10 client.0 ms_handle_connect on v2:172.21.15.32:3300/0 2024-08-03T20:55:39.378 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:39.374+0000 7fcd32a00640 1 -- 172.21.15.32:0/3098594718 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd48010bb0 con 0x55610ec87270 2024-08-03T20:55:39.378 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:39.374+0000 7fcd32a00640 1 -- 172.21.15.32:0/3098594718 <== mon.0 v2:172.21.15.32:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 6370+0+0 (secure 0 0 0) 0x7fcd4801bd60 con 0x55610ec87270 2024-08-03T20:55:39.378 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:39.375+0000 7fcd5bbb3740 1 -- 172.21.15.32:0/3098594718 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcd40065870 con 0x55610ec87270 2024-08-03T20:55:39.379 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:39.375+0000 7fcd32a00640 1 -- 172.21.15.32:0/3098594718 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_get_version_reply(handle=1 version=131) ==== 24+0+0 (secure 0 0 0) 0x7fcd48010d50 con 0x55610ec87270 2024-08-03T20:55:39.379 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:39.375+0000 7fcd5bbb3740 10 client.7824 fetch_fsmap learned FSMap version 131 2024-08-03T20:55:39.379 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:39.375+0000 7fcd5bbb3740 1 -- 172.21.15.32:0/3098594718 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,fsmap=131,monmap=2+}) -- 0x55610f045b60 con 0x55610ec87270 2024-08-03T20:55:39.379 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:39.375+0000 7fcd32a00640 1 -- 172.21.15.32:0/3098594718 <== mon.0 v2:172.21.15.32:3300/0 4 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fcd40065870 con 0x55610ec87270 2024-08-03T20:55:39.379 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:39.375+0000 7fcd32a00640 1 -- 172.21.15.32:0/3098594718 <== mon.0 v2:172.21.15.32:3300/0 5 ==== fsmap(e 131) ==== 3502+0+0 (secure 0 0 0) 0x7fcd48022050 con 0x55610ec87270 2024-08-03T20:55:39.379 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:39.375+0000 7fcd5bbb3740 10 client.7824 fetch_fsmap finished waiting for FSMap version 131 2024-08-03T20:55:39.379 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:39.375+0000 7fcd5bbb3740 10 client.7824 resolve_mds: resolved name 'a' to daemon mds.a 2024-08-03T20:55:39.379 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:39.375+0000 7fcd5bbb3740 20 client.7824 populate_metadata read hostname 'smithi032' 2024-08-03T20:55:39.379 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:39.375+0000 7fcd5bbb3740 1 --2- 172.21.15.32:0/3098594718 >> [v2:172.21.15.32:6833/3191654276,v1:172.21.15.32:6835/3191654276] conn(0x55610ec45e40 0x55610f06e870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:55:39.379 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:39.375+0000 7fcd5bbb3740 4 client.7824 mds_command: new command op to 7273 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-08-03T20:55:39.379 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:39.375+0000 7fcd5bbb3740 1 -- 172.21.15.32:0/3098594718 --> [v2:172.21.15.32:6833/3191654276,v1:172.21.15.32:6835/3191654276] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55610efa19b0 con 0x55610ec45e40 2024-08-03T20:55:39.379 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:39.376+0000 7fcd4de00640 1 --2- 172.21.15.32:0/3098594718 >> [v2:172.21.15.32:6833/3191654276,v1:172.21.15.32:6835/3191654276] conn(0x55610ec45e40 0x55610f06e870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:55:39.379 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:39.376+0000 7fcd32a00640 10 client.7824 ms_handle_connect on v2:172.21.15.32:6833/3191654276 2024-08-03T20:55:39.379 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:39.376+0000 7fcd4de00640 1 --2- 172.21.15.32:0/3098594718 >> [v2:172.21.15.32:6833/3191654276,v1:172.21.15.32:6835/3191654276] conn(0x55610ec45e40 0x55610f06e870 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9323c6aa57030f42 server_cookie=1baabb0cdd1216ca in_seq=0 out_seq=0 2024-08-03T20:55:39.380 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:39.377+0000 7fcd32a00640 1 -- 172.21.15.32:0/3098594718 <== mds.0 v2:172.21.15.32:6833/3191654276 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55610efa19b0 con 0x55610ec45e40 2024-08-03T20:55:39.380 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:39.377+0000 7fcd32a00640 10 client.7824 handle_command_reply: tid=0 2024-08-03T20:55:39.380 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:39.377+0000 7fcd5bbb3740 1 client.7824 shutdown 2024-08-03T20:55:39.380 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:39.377+0000 7fcd5bbb3740 1 -- 172.21.15.32:0/3098594718 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55610ec87270 msgr2=0x55610f071000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:55:39.380 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:39.377+0000 7fcd5bbb3740 1 --2- 172.21.15.32:0/3098594718 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55610ec87270 0x55610f071000 secure :-1 s=READY pgs=1219 cs=0 l=1 rev1=1 crypto rx=0x7fcd5015cba0 tx=0x7fcd48011db0 comp rx=0 tx=0).stop 2024-08-03T20:55:39.380 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:39.377+0000 7fcd5bbb3740 1 -- 172.21.15.32:0/3098594718 shutdown_connections 2024-08-03T20:55:39.380 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:39.377+0000 7fcd5bbb3740 1 --2- 172.21.15.32:0/3098594718 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55610ec87270 0x55610f071000 unknown :-1 s=CLOSED pgs=1219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:39.380 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:39.377+0000 7fcd5bbb3740 1 --2- 172.21.15.32:0/3098594718 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55610ec66590 0x55610f045620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:39.380 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:39.377+0000 7fcd5bbb3740 1 --2- 172.21.15.32:0/3098594718 >> [v2:172.21.15.32:6833/3191654276,v1:172.21.15.32:6835/3191654276] conn(0x55610ec45e40 0x55610f06e870 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:39.380 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:39.377+0000 7fcd5bbb3740 1 --2- 172.21.15.32:0/3098594718 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55610ec5e3b0 0x55610f085370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:39.380 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:39.377+0000 7fcd32a00640 0 client.7824 ms_handle_reset on v2:172.21.15.32:6833/3191654276 2024-08-03T20:55:39.380 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:39.377+0000 7fcd5bbb3740 1 -- 172.21.15.32:0/3098594718 >> 172.21.15.32:0/3098594718 conn(0x55610ec7b3c0 msgr2=0x55610f026780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:55:39.381 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:39.378+0000 7fcd5bbb3740 1 -- 172.21.15.32:0/3098594718 shutdown_connections 2024-08-03T20:55:39.381 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:39.378+0000 7fcd5bbb3740 1 -- 172.21.15.32:0/3098594718 wait complete. 2024-08-03T20:55:39.381 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:39.378+0000 7fcd5bbb3740 20 client.7824 trim_cache size 0 max 16384 2024-08-03T20:55:39.398 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:39.398+0000 7fcd5bbb3740 1 Processor -- start 2024-08-03T20:55:39.398 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:39.398+0000 7fcd5bbb3740 1 -- start start 2024-08-03T20:55:39.401 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:39.399+0000 7fcd5bbb3740 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55610ec5e3b0 0x55610f02cd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:55:39.401 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:39.399+0000 7fcd5bbb3740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55610ec66590 0x55610f070ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:55:39.401 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:39.399+0000 7fcd5bbb3740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55610ec87270 0x55610f095050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:55:39.401 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:39.399+0000 7fcd5bbb3740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fcd50201240 con 0x55610ec66590 2024-08-03T20:55:39.401 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:39.399+0000 7fcd5bbb3740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7fcd50156f90 con 0x55610ec87270 2024-08-03T20:55:39.401 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:39.399+0000 7fcd5bbb3740 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7fcd501575e0 con 0x55610ec5e3b0 2024-08-03T20:55:39.401 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:39.400+0000 7fcd47e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55610ec66590 0x55610f070ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:55:39.401 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:39.400+0000 7fcd47e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55610ec66590 0x55610f070ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:59504/0 (socket says 172.21.15.32:59504) 2024-08-03T20:55:39.402 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:39.400+0000 7fcd47e00640 1 -- 172.21.15.32:0/947313369 learned_addr learned my addr 172.21.15.32:0/947313369 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:55:39.402 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:39.400+0000 7fcd4e800640 1 --2- 172.21.15.32:0/947313369 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55610ec87270 0x55610f095050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:55:39.402 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:39.400+0000 7fcd4de00640 1 --2- 172.21.15.32:0/947313369 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55610ec5e3b0 0x55610f02cd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:55:39.402 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:39.400+0000 7fcd47e00640 1 -- 172.21.15.32:0/947313369 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55610ec5e3b0 msgr2=0x55610f02cd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:55:39.402 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:39.400+0000 7fcd47e00640 1 --2- 172.21.15.32:0/947313369 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55610ec5e3b0 0x55610f02cd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:39.402 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:39.400+0000 7fcd47e00640 1 -- 172.21.15.32:0/947313369 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55610ec87270 msgr2=0x55610f095050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:55:39.402 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:39.400+0000 7fcd47e00640 1 --2- 172.21.15.32:0/947313369 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55610ec87270 0x55610f095050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:39.402 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:39.400+0000 7fcd47e00640 1 -- 172.21.15.32:0/947313369 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55610f045990 con 0x55610ec66590 2024-08-03T20:55:39.402 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:39.400+0000 7fcd4e800640 1 --2- 172.21.15.32:0/947313369 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55610ec87270 0x55610f095050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-03T20:55:39.402 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:39.400+0000 7fcd4de00640 1 --2- 172.21.15.32:0/947313369 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55610ec5e3b0 0x55610f02cd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-03T20:55:39.403 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:39.400+0000 7fcd47e00640 1 --2- 172.21.15.32:0/947313369 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55610ec66590 0x55610f070ca0 secure :-1 s=READY pgs=1220 cs=0 l=1 rev1=1 crypto rx=0x7fcd380035f0 tx=0x7fcd3803e070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ab61697ae3252b8a server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:55:39.403 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:39.401+0000 7fcd3dc00640 1 -- 172.21.15.32:0/947313369 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd38003e20 con 0x55610ec66590 2024-08-03T20:55:39.403 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:39.401+0000 7fcd3dc00640 1 -- 172.21.15.32:0/947313369 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fcd3804a430 con 0x55610ec66590 2024-08-03T20:55:39.403 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:39.401+0000 7fcd3dc00640 1 -- 172.21.15.32:0/947313369 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd3804a820 con 0x55610ec66590 2024-08-03T20:55:39.404 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:39.401+0000 7fcd5bbb3740 1 -- 172.21.15.32:0/947313369 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55610ec66590 msgr2=0x55610f070ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:55:39.404 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:39.401+0000 7fcd5bbb3740 1 --2- 172.21.15.32:0/947313369 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55610ec66590 0x55610f070ca0 secure :-1 s=READY pgs=1220 cs=0 l=1 rev1=1 crypto rx=0x7fcd380035f0 tx=0x7fcd3803e070 comp rx=0 tx=0).stop 2024-08-03T20:55:39.404 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:39.401+0000 7fcd5bbb3740 1 -- 172.21.15.32:0/947313369 shutdown_connections 2024-08-03T20:55:39.404 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:39.401+0000 7fcd5bbb3740 1 --2- 172.21.15.32:0/947313369 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55610ec87270 0x55610f095050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:39.404 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:39.401+0000 7fcd5bbb3740 1 --2- 172.21.15.32:0/947313369 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55610ec66590 0x55610f070ca0 unknown :-1 s=CLOSED pgs=1220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:39.404 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:39.401+0000 7fcd5bbb3740 1 --2- 172.21.15.32:0/947313369 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55610ec5e3b0 0x55610f02cd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:39.404 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:39.401+0000 7fcd5bbb3740 1 -- 172.21.15.32:0/947313369 >> 172.21.15.32:0/947313369 conn(0x55610ec7b3c0 msgr2=0x55610f028490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:55:39.404 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:39.401+0000 7fcd5bbb3740 1 -- 172.21.15.32:0/947313369 shutdown_connections 2024-08-03T20:55:39.404 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:39.402+0000 7fcd5bbb3740 1 -- 172.21.15.32:0/947313369 wait complete. 2024-08-03T20:55:39.404 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:39.402+0000 7fcd5bbb3740 1 -- start start 2024-08-03T20:55:39.405 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:39.403+0000 7fcd5bbb3740 1 Processor -- start 2024-08-03T20:55:39.407 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:39.404+0000 7fcd5bbb3740 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55610ec5e3b0 0x55610f056f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:55:39.407 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:39.404+0000 7fcd5bbb3740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55610ec66590 0x55610f0f2ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:55:39.407 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:39.404+0000 7fcd5bbb3740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55610ec87270 0x55610f08aa80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:55:39.407 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:39.404+0000 7fcd5bbb3740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55610f071680 con 0x55610ec66590 2024-08-03T20:55:39.407 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:39.404+0000 7fcd5bbb3740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x55610f070740 con 0x55610ec87270 2024-08-03T20:55:39.407 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:39.404+0000 7fcd5bbb3740 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x55610f06c4d0 con 0x55610ec5e3b0 2024-08-03T20:55:39.407 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:39.404+0000 7fcd5bbb3740 10 client.0 _ll_register_callbacks cb 0x7fcd48011560 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-03T20:55:39.407 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:39.404+0000 7fcd47e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55610ec66590 0x55610f0f2ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:55:39.408 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:39.404+0000 7fcd47e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55610ec66590 0x55610f0f2ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:59520/0 (socket says 172.21.15.32:59520) 2024-08-03T20:55:39.408 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:39.404+0000 7fcd47e00640 1 -- 172.21.15.32:0/292254142 learned_addr learned my addr 172.21.15.32:0/292254142 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:55:39.408 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:39.404+0000 7fcd4e800640 1 --2- 172.21.15.32:0/292254142 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55610ec87270 0x55610f08aa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:55:39.408 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:39.404+0000 7fcd4de00640 1 --2- 172.21.15.32:0/292254142 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55610ec5e3b0 0x55610f056f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:55:39.408 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:39.405+0000 7fcd47e00640 1 -- 172.21.15.32:0/292254142 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55610ec5e3b0 msgr2=0x55610f056f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:55:39.408 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:39.405+0000 7fcd47e00640 1 --2- 172.21.15.32:0/292254142 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55610ec5e3b0 0x55610f056f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:39.408 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:39.405+0000 7fcd47e00640 1 -- 172.21.15.32:0/292254142 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55610ec87270 msgr2=0x55610f08aa80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:55:39.408 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:39.405+0000 7fcd47e00640 1 --2- 172.21.15.32:0/292254142 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55610ec87270 0x55610f08aa80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:39.408 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:39.405+0000 7fcd47e00640 1 -- 172.21.15.32:0/292254142 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55610f06d4b0 con 0x55610ec66590 2024-08-03T20:55:39.408 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:39.405+0000 7fcd47e00640 1 --2- 172.21.15.32:0/292254142 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55610ec66590 0x55610f0f2ec0 secure :-1 s=READY pgs=1221 cs=0 l=1 rev1=1 crypto rx=0x7fcd38009950 tx=0x7fcd3804add0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=da710dc3febcb736 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:55:39.408 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:39.405+0000 7fcd32a00640 10 client.0 ms_handle_connect on v2:172.21.15.32:3300/0 2024-08-03T20:55:39.408 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:39.405+0000 7fcd32a00640 1 -- 172.21.15.32:0/292254142 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd3805c070 con 0x55610ec66590 2024-08-03T20:55:39.408 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:39.405+0000 7fcd32a00640 1 -- 172.21.15.32:0/292254142 <== mon.0 v2:172.21.15.32:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 6370+0+0 (secure 0 0 0) 0x7fcd3803c420 con 0x55610ec66590 2024-08-03T20:55:39.408 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:39.405+0000 7fcd5bbb3740 1 -- 172.21.15.32:0/292254142 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcd3804a430 con 0x55610ec66590 2024-08-03T20:55:39.409 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:39.406+0000 7fcd32a00640 1 -- 172.21.15.32:0/292254142 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_get_version_reply(handle=1 version=131) ==== 24+0+0 (secure 0 0 0) 0x7fcd38067050 con 0x55610ec66590 2024-08-03T20:55:39.409 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:39.406+0000 7fcd5bbb3740 10 client.7830 fetch_fsmap learned FSMap version 131 2024-08-03T20:55:39.409 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:39.406+0000 7fcd5bbb3740 1 -- 172.21.15.32:0/292254142 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,fsmap=131,monmap=2+}) -- 0x55610f0f3500 con 0x55610ec66590 2024-08-03T20:55:39.409 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:39.406+0000 7fcd32a00640 1 -- 172.21.15.32:0/292254142 <== mon.0 v2:172.21.15.32:3300/0 4 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fcd3804a430 con 0x55610ec66590 2024-08-03T20:55:39.409 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:39.406+0000 7fcd32a00640 1 -- 172.21.15.32:0/292254142 <== mon.0 v2:172.21.15.32:3300/0 5 ==== fsmap(e 131) ==== 3502+0+0 (secure 0 0 0) 0x7fcd3806a020 con 0x55610ec66590 2024-08-03T20:55:39.409 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:39.406+0000 7fcd5bbb3740 10 client.7830 fetch_fsmap finished waiting for FSMap version 131 2024-08-03T20:55:39.409 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:39.406+0000 7fcd5bbb3740 10 client.7830 resolve_mds: resolved name 'a' to daemon mds.a 2024-08-03T20:55:39.409 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:39.406+0000 7fcd5bbb3740 20 client.7830 populate_metadata read hostname 'smithi032' 2024-08-03T20:55:39.409 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:39.406+0000 7fcd5bbb3740 1 --2- 172.21.15.32:0/292254142 >> [v2:172.21.15.32:6833/3191654276,v1:172.21.15.32:6835/3191654276] conn(0x55610f028880 0x55610f0271d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:55:39.409 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:39.406+0000 7fcd5bbb3740 4 client.7830 mds_command: new command op to 7273 tid=0 multi_id=0 [{"prefix": "status"}] 2024-08-03T20:55:39.409 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:39.406+0000 7fcd5bbb3740 1 -- 172.21.15.32:0/292254142 --> [v2:172.21.15.32:6833/3191654276,v1:172.21.15.32:6835/3191654276] -- command(tid 0: {"prefix": "status"}) -- 0x55610f072b30 con 0x55610f028880 2024-08-03T20:55:39.409 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:39.406+0000 7fcd4de00640 1 --2- 172.21.15.32:0/292254142 >> [v2:172.21.15.32:6833/3191654276,v1:172.21.15.32:6835/3191654276] conn(0x55610f028880 0x55610f0271d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:55:39.411 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:39.407+0000 7fcd4de00640 1 --2- 172.21.15.32:0/292254142 >> [v2:172.21.15.32:6833/3191654276,v1:172.21.15.32:6835/3191654276] conn(0x55610f028880 0x55610f0271d0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7b6f188f07b00d31 server_cookie=c4f3e0293a37215d in_seq=0 out_seq=0 2024-08-03T20:55:39.411 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:39.407+0000 7fcd32a00640 10 client.7830 ms_handle_connect on v2:172.21.15.32:6833/3191654276 2024-08-03T20:55:39.411 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:39.407+0000 7fcd32a00640 1 -- 172.21.15.32:0/292254142 <== mds.0 v2:172.21.15.32:6833/3191654276 1 ==== command_reply(tid 0: 0 ) ==== 8+0+332 (crc 0 0 0) 0x55610f072b30 con 0x55610f028880 2024-08-03T20:55:39.411 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:39.407+0000 7fcd32a00640 10 client.7830 handle_command_reply: tid=0 2024-08-03T20:55:39.411 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:39.407+0000 7fcd5bbb3740 1 client.7830 shutdown 2024-08-03T20:55:39.411 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:39.407+0000 7fcd5bbb3740 1 -- 172.21.15.32:0/292254142 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55610ec66590 msgr2=0x55610f0f2ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:55:39.411 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:39.407+0000 7fcd5bbb3740 1 --2- 172.21.15.32:0/292254142 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55610ec66590 0x55610f0f2ec0 secure :-1 s=READY pgs=1221 cs=0 l=1 rev1=1 crypto rx=0x7fcd38009950 tx=0x7fcd3804add0 comp rx=0 tx=0).stop 2024-08-03T20:55:39.411 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:39.407+0000 7fcd5bbb3740 1 -- 172.21.15.32:0/292254142 shutdown_connections 2024-08-03T20:55:39.411 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:39.408+0000 7fcd5bbb3740 1 --2- 172.21.15.32:0/292254142 >> [v2:172.21.15.32:6833/3191654276,v1:172.21.15.32:6835/3191654276] conn(0x55610f028880 0x55610f0271d0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:39.411 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:39.408+0000 7fcd5bbb3740 1 --2- 172.21.15.32:0/292254142 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55610ec87270 0x55610f08aa80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:39.411 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:39.408+0000 7fcd5bbb3740 1 --2- 172.21.15.32:0/292254142 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55610ec66590 0x55610f0f2ec0 unknown :-1 s=CLOSED pgs=1221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:39.411 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:39.408+0000 7fcd5bbb3740 1 --2- 172.21.15.32:0/292254142 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55610ec5e3b0 0x55610f056f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:39.411 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:39.408+0000 7fcd32a00640 0 client.7830 ms_handle_reset on v2:172.21.15.32:6833/3191654276 2024-08-03T20:55:39.411 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:39.408+0000 7fcd5bbb3740 1 -- 172.21.15.32:0/292254142 >> 172.21.15.32:0/292254142 conn(0x55610ec7b3c0 msgr2=0x55610f02cbf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:55:39.412 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:39.408+0000 7fcd5bbb3740 1 -- 172.21.15.32:0/292254142 shutdown_connections 2024-08-03T20:55:39.412 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:39.408+0000 7fcd5bbb3740 1 -- 172.21.15.32:0/292254142 wait complete. 2024-08-03T20:55:39.412 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:39.408+0000 7fcd5bbb3740 20 client.7830 trim_cache size 0 max 16384 2024-08-03T20:55:39.412 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-08-03T20:55:39.412 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_fsid": "40ab141a-01d7-49b9-ae3c-2edec9876e95", 2024-08-03T20:55:39.412 INFO:teuthology.orchestra.run.smithi032.stdout: "whoami": 0, 2024-08-03T20:55:39.412 INFO:teuthology.orchestra.run.smithi032.stdout: "id": 7273, 2024-08-03T20:55:39.412 INFO:teuthology.orchestra.run.smithi032.stdout: "want_state": "up:active", 2024-08-03T20:55:39.412 INFO:teuthology.orchestra.run.smithi032.stdout: "state": "up:active", 2024-08-03T20:55:39.413 INFO:teuthology.orchestra.run.smithi032.stdout: "fs_name": "cephfs", 2024-08-03T20:55:39.413 INFO:teuthology.orchestra.run.smithi032.stdout: "rank_uptime": 7.1620570199999998, 2024-08-03T20:55:39.413 INFO:teuthology.orchestra.run.smithi032.stdout: "mdsmap_epoch": 131, 2024-08-03T20:55:39.413 INFO:teuthology.orchestra.run.smithi032.stdout: "osdmap_epoch": 91, 2024-08-03T20:55:39.413 INFO:teuthology.orchestra.run.smithi032.stdout: "osdmap_epoch_barrier": 91, 2024-08-03T20:55:39.413 INFO:teuthology.orchestra.run.smithi032.stdout: "uptime": 8.3198919930000006 2024-08-03T20:55:39.413 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-08-03T20:55:39.414 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:39.408+0000 7fcd4f200640 1 -- 172.21.15.32:0/767965005 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fcd2c077040 msgr2=0x7fcd2c079550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:55:39.414 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:39.408+0000 7fcd4f200640 1 --2- 172.21.15.32:0/767965005 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fcd2c077040 0x7fcd2c079550 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7fcd38034000 tx=0x7fcd38004730 comp rx=0 tx=0).stop 2024-08-03T20:55:39.414 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:39.408+0000 7fcd4f200640 1 -- 172.21.15.32:0/767965005 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcd500b1300 msgr2=0x7fcd500cb720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:55:39.414 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:39.408+0000 7fcd4f200640 1 --2- 172.21.15.32:0/767965005 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcd500b1300 0x7fcd500cb720 secure :-1 s=READY pgs=1218 cs=0 l=1 rev1=1 crypto rx=0x7fcd4000c430 tx=0x7fcd4000c8f0 comp rx=0 tx=0).stop 2024-08-03T20:55:39.414 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:39.409+0000 7fcd4f200640 1 -- 172.21.15.32:0/767965005 shutdown_connections 2024-08-03T20:55:39.414 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:39.409+0000 7fcd4f200640 1 --2- 172.21.15.32:0/767965005 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fcd2c077040 0x7fcd2c079550 unknown :-1 s=CLOSED pgs=506 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:39.414 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:39.409+0000 7fcd4f200640 1 --2- 172.21.15.32:0/767965005 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fcd500afc00 0x7fcd500b0010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:39.414 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:39.409+0000 7fcd4f200640 1 --2- 172.21.15.32:0/767965005 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fcd500d29b0 0x7fcd50148fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:39.415 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:39.409+0000 7fcd4f200640 1 --2- 172.21.15.32:0/767965005 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcd500b1300 0x7fcd500cb720 unknown :-1 s=CLOSED pgs=1218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:39.415 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:39.409+0000 7fcd4f200640 1 -- 172.21.15.32:0/767965005 >> 172.21.15.32:0/767965005 conn(0x7fcd500d1e90 msgr2=0x7fcd500b3e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:55:39.415 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:39.409+0000 7fcd4f200640 1 -- 172.21.15.32:0/767965005 shutdown_connections 2024-08-03T20:55:39.415 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:39.409+0000 7fcd4f200640 1 -- 172.21.15.32:0/767965005 wait complete. 2024-08-03T20:55:39.434 DEBUG:tasks.ceph_test_case:wait_until_true: success in 5s and 0 retries 2024-08-03T20:55:39.435 INFO:tasks.cephfs.test_damage:Daemons came up after mutation 'Delete mds0_openfiles.0', proceeding to ls 2024-08-03T20:55:39.435 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-08-03T20:55:39.665 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:39.664+0000 7f2503600640 1 -- 172.21.15.32:0/3450208105 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f24fc073e60 msgr2=0x7f24fc074270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:55:39.665 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:39.664+0000 7f2503600640 1 --2- 172.21.15.32:0/3450208105 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f24fc073e60 0x7f24fc074270 secure :-1 s=READY pgs=1222 cs=0 l=1 rev1=1 crypto rx=0x7f24e8005fd0 tx=0x7f24e802a540 comp rx=0 tx=0).stop 2024-08-03T20:55:39.665 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:39.664+0000 7f2503600640 1 -- 172.21.15.32:0/3450208105 shutdown_connections 2024-08-03T20:55:39.665 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:39.664+0000 7f2503600640 1 --2- 172.21.15.32:0/3450208105 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f24fc073e60 0x7f24fc074270 unknown :-1 s=CLOSED pgs=1222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:39.665 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:39.664+0000 7f2503600640 1 --2- 172.21.15.32:0/3450208105 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f24fc06e930 0x7f24fc144f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:39.666 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:39.664+0000 7f2503600640 1 -- 172.21.15.32:0/3450208105 >> 172.21.15.32:0/3450208105 conn(0x7f24fc0b1bd0 msgr2=0x7f24fc0b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:55:39.666 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:39.664+0000 7f2503600640 1 -- 172.21.15.32:0/3450208105 shutdown_connections 2024-08-03T20:55:39.666 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:39.664+0000 7f2503600640 1 -- 172.21.15.32:0/3450208105 wait complete. 2024-08-03T20:55:39.667 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:39.666+0000 7f2503600640 1 Processor -- start 2024-08-03T20:55:39.668 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:39.667+0000 7f2503600640 1 -- start start 2024-08-03T20:55:39.669 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:39.668+0000 7f2503600640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f24fc06e930 0x7f24fc1f2520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:55:39.669 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:39.668+0000 7f2503600640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f24fc073e60 0x7f24fc15ec40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:55:39.669 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:39.668+0000 7f2503600640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f24fc0747b0 0x7f24fc15f180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:55:39.669 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:39.668+0000 7f2503600640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f24fc0cf9c0 con 0x7f24fc0747b0 2024-08-03T20:55:39.669 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:39.668+0000 7f2503600640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f24fc0cf840 con 0x7f24fc06e930 2024-08-03T20:55:39.669 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:39.668+0000 7f2503600640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7f24fc0ccff0 con 0x7f24fc073e60 2024-08-03T20:55:39.669 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:39.668+0000 7f2501600640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f24fc0747b0 0x7f24fc15f180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:55:39.669 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:39.668+0000 7f2501600640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f24fc0747b0 0x7f24fc15f180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:59540/0 (socket says 172.21.15.32:59540) 2024-08-03T20:55:39.669 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:39.668+0000 7f2501600640 1 -- 172.21.15.32:0/3808771532 learned_addr learned my addr 172.21.15.32:0/3808771532 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:55:39.669 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:39.669+0000 7f24fbe00640 1 --2- 172.21.15.32:0/3808771532 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f24fc073e60 0x7f24fc15ec40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:55:39.670 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:39.669+0000 7f2500c00640 1 --2- 172.21.15.32:0/3808771532 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f24fc06e930 0x7f24fc1f2520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:55:39.671 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:39.669+0000 7f2501600640 1 -- 172.21.15.32:0/3808771532 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f24fc073e60 msgr2=0x7f24fc15ec40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:55:39.671 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:39.669+0000 7f2501600640 1 --2- 172.21.15.32:0/3808771532 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f24fc073e60 0x7f24fc15ec40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:39.671 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:39.669+0000 7f2501600640 1 -- 172.21.15.32:0/3808771532 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f24fc06e930 msgr2=0x7f24fc1f2520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:55:39.671 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:39.669+0000 7f2501600640 1 --2- 172.21.15.32:0/3808771532 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f24fc06e930 0x7f24fc1f2520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:39.671 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:39.669+0000 7f2501600640 1 -- 172.21.15.32:0/3808771532 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f24fc15f850 con 0x7f24fc0747b0 2024-08-03T20:55:39.671 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:39.669+0000 7f2500c00640 1 --2- 172.21.15.32:0/3808771532 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f24fc06e930 0x7f24fc1f2520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-03T20:55:39.671 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:39.669+0000 7f2501600640 1 --2- 172.21.15.32:0/3808771532 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f24fc0747b0 0x7f24fc15f180 secure :-1 s=READY pgs=1223 cs=0 l=1 rev1=1 crypto rx=0x7f24f4002230 tx=0x7f24f400cb90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:55:39.671 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:39.669+0000 7f24f8c00640 1 -- 172.21.15.32:0/3808771532 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24f4003b70 con 0x7f24fc0747b0 2024-08-03T20:55:39.671 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:39.669+0000 7f24f8c00640 1 -- 172.21.15.32:0/3808771532 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f24f4003d10 con 0x7f24fc0747b0 2024-08-03T20:55:39.671 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:39.669+0000 7f24f8c00640 1 -- 172.21.15.32:0/3808771532 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24f4005330 con 0x7f24fc0747b0 2024-08-03T20:55:39.671 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:39.670+0000 7f2503600640 1 -- 172.21.15.32:0/3808771532 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f24fc142c90 con 0x7f24fc0747b0 2024-08-03T20:55:39.671 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:39.670+0000 7f2503600640 1 -- 172.21.15.32:0/3808771532 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f24fc143250 con 0x7f24fc0747b0 2024-08-03T20:55:39.672 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:39.671+0000 7f2503600640 1 -- 172.21.15.32:0/3808771532 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f24fc15fac0 con 0x7f24fc0747b0 2024-08-03T20:55:39.672 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:39.672+0000 7f24f8c00640 1 -- 172.21.15.32:0/3808771532 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f24f40054d0 con 0x7f24fc0747b0 2024-08-03T20:55:39.673 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:39.673+0000 7f24f8c00640 1 --2- 172.21.15.32:0/3808771532 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f24dc077040 0x7f24dc079550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:55:39.673 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:39.673+0000 7f24f8c00640 1 -- 172.21.15.32:0/3808771532 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 6370+0+0 (secure 0 0 0) 0x7f24f4098890 con 0x7f24fc0747b0 2024-08-03T20:55:39.674 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:39.674+0000 7f2500c00640 1 --2- 172.21.15.32:0/3808771532 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f24dc077040 0x7f24dc079550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:55:39.675 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:39.674+0000 7f2500c00640 1 --2- 172.21.15.32:0/3808771532 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f24dc077040 0x7f24dc079550 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7f24f0005fd0 tx=0x7f24f000f360 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:55:39.678 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:39.677+0000 7f24f8c00640 1 -- 172.21.15.32:0/3808771532 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7f24f4061870 con 0x7f24fc0747b0 2024-08-03T20:55:39.921 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:39.921+0000 7f2503600640 1 -- 172.21.15.32:0/3808771532 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f24fc0cd510 con 0x7f24fc0747b0 2024-08-03T20:55:39.922 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:39.921+0000 7f24f8c00640 1 -- 172.21.15.32:0/3808771532 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v131) ==== 53+0+73 (secure 0 0 0) 0x7f24f4061170 con 0x7f24fc0747b0 2024-08-03T20:55:39.922 INFO:teuthology.orchestra.run.smithi032.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-08-03T20:55:39.924 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:39.923+0000 7f2503600640 1 -- 172.21.15.32:0/3808771532 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f24dc077040 msgr2=0x7f24dc079550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:55:39.924 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:39.923+0000 7f2503600640 1 --2- 172.21.15.32:0/3808771532 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f24dc077040 0x7f24dc079550 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7f24f0005fd0 tx=0x7f24f000f360 comp rx=0 tx=0).stop 2024-08-03T20:55:39.924 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:39.923+0000 7f2503600640 1 -- 172.21.15.32:0/3808771532 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f24fc0747b0 msgr2=0x7f24fc15f180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:55:39.924 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:39.923+0000 7f2503600640 1 --2- 172.21.15.32:0/3808771532 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f24fc0747b0 0x7f24fc15f180 secure :-1 s=READY pgs=1223 cs=0 l=1 rev1=1 crypto rx=0x7f24f4002230 tx=0x7f24f400cb90 comp rx=0 tx=0).stop 2024-08-03T20:55:39.924 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:39.923+0000 7f2503600640 1 -- 172.21.15.32:0/3808771532 shutdown_connections 2024-08-03T20:55:39.924 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:39.923+0000 7f2503600640 1 --2- 172.21.15.32:0/3808771532 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f24dc077040 0x7f24dc079550 unknown :-1 s=CLOSED pgs=507 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:39.925 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:39.923+0000 7f2503600640 1 --2- 172.21.15.32:0/3808771532 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f24fc0747b0 0x7f24fc15f180 unknown :-1 s=CLOSED pgs=1223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:39.925 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:39.923+0000 7f2503600640 1 --2- 172.21.15.32:0/3808771532 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f24fc073e60 0x7f24fc15ec40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:39.925 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:39.923+0000 7f2503600640 1 --2- 172.21.15.32:0/3808771532 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f24fc06e930 0x7f24fc1f2520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:39.925 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:39.923+0000 7f2503600640 1 -- 172.21.15.32:0/3808771532 >> 172.21.15.32:0/3808771532 conn(0x7f24fc0b1bd0 msgr2=0x7f24fc0b02b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:55:39.925 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:39.923+0000 7f2503600640 1 -- 172.21.15.32:0/3808771532 shutdown_connections 2024-08-03T20:55:39.925 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:39.923+0000 7f2503600640 1 -- 172.21.15.32:0/3808771532 wait complete. 2024-08-03T20:55:39.946 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-08-03T20:55:39.946 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-08-03T20:55:39.947 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi032.front.sepia.ceph.com 2024-08-03T20:55:39.947 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-08-03T20:55:39.947 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-08-03T20:55:39.947 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-08-03T20:55:39.947 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2024-08-03T20:55:39.947 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-08-03T20:55:39.947 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-08-03T20:55:39.947 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-08-03T20:55:39.947 DEBUG:teuthology.orchestra.run.smithi032:> ip addr 2024-08-03T20:55:39.983 INFO:teuthology.orchestra.run.smithi032.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-08-03T20:55:39.983 INFO:teuthology.orchestra.run.smithi032.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-08-03T20:55:39.984 INFO:teuthology.orchestra.run.smithi032.stdout: inet 127.0.0.1/8 scope host lo 2024-08-03T20:55:39.984 INFO:teuthology.orchestra.run.smithi032.stdout: valid_lft forever preferred_lft forever 2024-08-03T20:55:39.984 INFO:teuthology.orchestra.run.smithi032.stdout: inet6 ::1/128 scope host 2024-08-03T20:55:39.984 INFO:teuthology.orchestra.run.smithi032.stdout: valid_lft forever preferred_lft forever 2024-08-03T20:55:39.984 INFO:teuthology.orchestra.run.smithi032.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-08-03T20:55:39.984 INFO:teuthology.orchestra.run.smithi032.stdout: link/ether 0c:c4:7a:6b:fd:0c brd ff:ff:ff:ff:ff:ff 2024-08-03T20:55:39.984 INFO:teuthology.orchestra.run.smithi032.stdout: altname enp7s0f0 2024-08-03T20:55:39.984 INFO:teuthology.orchestra.run.smithi032.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-08-03T20:55:39.984 INFO:teuthology.orchestra.run.smithi032.stdout: link/ether 0c:c4:7a:6b:fd:0d brd ff:ff:ff:ff:ff:ff 2024-08-03T20:55:39.984 INFO:teuthology.orchestra.run.smithi032.stdout: altname enp7s0f1 2024-08-03T20:55:39.984 INFO:teuthology.orchestra.run.smithi032.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-08-03T20:55:39.985 INFO:teuthology.orchestra.run.smithi032.stdout: link/ether ac:1f:6b:f8:26:02 brd ff:ff:ff:ff:ff:ff 2024-08-03T20:55:39.985 INFO:teuthology.orchestra.run.smithi032.stdout: altname enp1s0f0 2024-08-03T20:55:39.985 INFO:teuthology.orchestra.run.smithi032.stdout: inet 172.21.15.32/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-08-03T20:55:39.985 INFO:teuthology.orchestra.run.smithi032.stdout: valid_lft 42016sec preferred_lft 42016sec 2024-08-03T20:55:39.985 INFO:teuthology.orchestra.run.smithi032.stdout: inet6 fe80::ae1f:6bff:fef8:2602/64 scope link noprefixroute 2024-08-03T20:55:39.985 INFO:teuthology.orchestra.run.smithi032.stdout: valid_lft forever preferred_lft forever 2024-08-03T20:55:39.985 INFO:teuthology.orchestra.run.smithi032.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-08-03T20:55:39.985 INFO:teuthology.orchestra.run.smithi032.stdout: link/ether ac:1f:6b:f8:26:03 brd ff:ff:ff:ff:ff:ff 2024-08-03T20:55:39.985 INFO:teuthology.orchestra.run.smithi032.stdout: altname enp1s0f1 2024-08-03T20:55:39.985 INFO:teuthology.orchestra.run.smithi032.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-08-03T20:55:39.985 INFO:teuthology.orchestra.run.smithi032.stdout: link/ether 0a:23:2b:ca:2a:93 brd ff:ff:ff:ff:ff:ff 2024-08-03T20:55:39.986 INFO:teuthology.orchestra.run.smithi032.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-08-03T20:55:39.986 INFO:teuthology.orchestra.run.smithi032.stdout: valid_lft forever preferred_lft forever 2024-08-03T20:55:39.986 INFO:teuthology.orchestra.run.smithi032.stdout: inet6 fe80::d804:7cff:fed9:7ba4/64 scope link 2024-08-03T20:55:39.986 INFO:teuthology.orchestra.run.smithi032.stdout: valid_lft forever preferred_lft forever 2024-08-03T20:55:39.986 INFO:teuthology.orchestra.run.smithi032.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-08-03T20:55:39.986 INFO:teuthology.orchestra.run.smithi032.stdout: link/ether 22:86:28:30:4e:62 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-08-03T20:55:39.986 INFO:teuthology.orchestra.run.smithi032.stdout: inet6 fe80::2086:28ff:fe30:4e62/64 scope link 2024-08-03T20:55:39.986 INFO:teuthology.orchestra.run.smithi032.stdout: valid_lft forever preferred_lft forever 2024-08-03T20:55:39.986 INFO:teuthology.orchestra.run.smithi032.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-08-03T20:55:39.986 INFO:teuthology.orchestra.run.smithi032.stdout: link/ether 0a:23:2b:ca:2a:93 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-08-03T20:55:39.986 INFO:teuthology.orchestra.run.smithi032.stdout: inet6 fe80::823:2bff:feca:2a93/64 scope link 2024-08-03T20:55:39.986 INFO:teuthology.orchestra.run.smithi032.stdout: valid_lft forever preferred_lft forever 2024-08-03T20:55:39.987 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-08-03T20:55:39.987 DEBUG:teuthology.orchestra.run.smithi032:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-08-03T20:55:40.077 INFO:teuthology.orchestra.run.smithi032.stdout:1 2024-08-03T20:55:40.080 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-08-03T20:55:40.080 DEBUG:teuthology.orchestra.run.smithi032:> route 2024-08-03T20:55:40.116 INFO:teuthology.orchestra.run.smithi032.stdout:Kernel IP routing table 2024-08-03T20:55:40.116 INFO:teuthology.orchestra.run.smithi032.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-08-03T20:55:40.116 INFO:teuthology.orchestra.run.smithi032.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-08-03T20:55:40.117 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-08-03T20:55:40.117 INFO:teuthology.orchestra.run.smithi032.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-08-03T20:55:40.117 DEBUG:teuthology.orchestra.run.smithi032:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-08-03T20:55:40.117 DEBUG:teuthology.orchestra.run.smithi032:> set -e 2024-08-03T20:55:40.117 DEBUG:teuthology.orchestra.run.smithi032:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-08-03T20:55:40.117 DEBUG:teuthology.orchestra.run.smithi032:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-08-03T20:55:40.117 DEBUG:teuthology.orchestra.run.smithi032:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-08-03T20:55:40.117 DEBUG:teuthology.orchestra.run.smithi032:> ') 2024-08-03T20:55:40.235 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:40 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-08-03T20:55:40.305 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:40 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-08-03T20:55:40.314 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-08-03T20:55:40.314 DEBUG:teuthology.orchestra.run.smithi032:> ip netns list 2024-08-03T20:55:40.349 INFO:teuthology.orchestra.run.smithi032.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-08-03T20:55:40.349 INFO:teuthology.orchestra.run.smithi032.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-08-03T20:55:40.349 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-08-03T20:55:40.349 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-08-03T20:55:40.350 DEBUG:teuthology.orchestra.run.smithi032:> sudo ip link set brx.0 up 2024-08-03T20:55:40.446 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-08-03T20:55:40.446 DEBUG:teuthology.orchestra.run.smithi032:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-08-03T20:55:40.481 INFO:teuthology.orchestra.run.smithi032.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-08-03T20:55:40.482 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-08-03T20:55:40.482 DEBUG:teuthology.orchestra.run.smithi032:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-08-03T20:55:40.558 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2024-08-03T20:55:40.558 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-08-03T20:55:40.559 DEBUG:teuthology.orchestra.run.smithi032:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-08-03T20:55:40.764 INFO:teuthology.orchestra.run.smithi032.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-08-03T20:55:40.764 INFO:teuthology.orchestra.run.smithi032.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-08-03T20:55:40.764 INFO:teuthology.orchestra.run.smithi032.stdout:invalid new device string format 2024-08-03T20:55:40.785 INFO:teuthology.orchestra.run.smithi032.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-08-03T20:55:40.785 INFO:teuthology.orchestra.run.smithi032.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-08-03T20:55:40.785 INFO:teuthology.orchestra.run.smithi032.stdout:invalid new device string format 2024-08-03T20:55:40.785 INFO:teuthology.orchestra.run.smithi032.stdout:mount.ceph: resolved to: "172.21.15.32:3300,172.21.15.118:3300,172.21.15.118:3301" 2024-08-03T20:55:40.785 INFO:teuthology.orchestra.run.smithi032.stdout:mount.ceph: switching to using v1 address with old syntax 2024-08-03T20:55:40.818 INFO:teuthology.orchestra.run.smithi032.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-08-03T20:55:40.818 INFO:teuthology.orchestra.run.smithi032.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-08-03T20:55:40.818 INFO:teuthology.orchestra.run.smithi032.stdout:invalid new device string format 2024-08-03T20:55:40.818 INFO:teuthology.orchestra.run.smithi032.stdout:mount.ceph: resolved to: "172.21.15.32:3300,172.21.15.118:3300,172.21.15.118:3301" 2024-08-03T20:55:40.818 INFO:teuthology.orchestra.run.smithi032.stdout:mount.ceph: switching to using v1 address with old syntax 2024-08-03T20:55:40.819 INFO:teuthology.orchestra.run.smithi032.stdout:mount.ceph: resolved to: "172.21.15.32:6789,172.21.15.118:6789,172.21.15.118:6790" 2024-08-03T20:55:40.819 INFO:teuthology.orchestra.run.smithi032.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.32:6789,172.21.15.118:6789,172.21.15.118:6790:/ 2024-08-03T20:55:40.819 INFO:teuthology.orchestra.run.smithi032.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,key=0,fsid=40ab141a-01d7-49b9-ae3c-2edec9876e95" will pass to kernel 2024-08-03T20:55:40.820 INFO:teuthology.orchestra.run.smithi032.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2024-08-03T20:55:40.820 INFO:teuthology.orchestra.run.smithi032.stdout: You just mounted a file system that supports labels which does not 2024-08-03T20:55:40.820 INFO:teuthology.orchestra.run.smithi032.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-08-03T20:55:40.820 INFO:teuthology.orchestra.run.smithi032.stdout: applications will generate AVC messages and not be allowed access to 2024-08-03T20:55:40.821 INFO:teuthology.orchestra.run.smithi032.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-08-03T20:55:40.823 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-08-03T20:55:40.823 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-08-03T20:55:40.823 DEBUG:teuthology.orchestra.run.smithi032:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-08-03T20:55:40.880 DEBUG:teuthology.orchestra.run.smithi032:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-08-03T20:55:40.963 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:40 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-08-03T20:55:40.966 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:40 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-08-03T20:55:40.976 INFO:teuthology.orchestra.run.smithi032.stdout:client7839 2024-08-03T20:55:40.977 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-08-03T20:55:40.977 DEBUG:teuthology.orchestra.run.smithi032:> dd if=/proc/self/mounts of=/dev/stdout 2024-08-03T20:55:41.013 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-08-03T20:55:41.013 DEBUG:teuthology.orchestra.run.smithi032:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls -R .) 2024-08-03T20:55:41.059 INFO:teuthology.orchestra.run:waiting for 20 2024-08-03T20:55:41.092 INFO:teuthology.orchestra.run.smithi032.stdout:.: 2024-08-03T20:55:41.092 INFO:teuthology.orchestra.run.smithi032.stdout:subdir 2024-08-03T20:55:41.092 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-03T20:55:41.092 INFO:teuthology.orchestra.run.smithi032.stdout:./subdir: 2024-08-03T20:55:41.093 INFO:teuthology.orchestra.run.smithi032.stdout:sixmegs 2024-08-03T20:55:41.094 INFO:tasks.cephfs.test_damage:Result: Mutation 'Delete mds0_openfiles.0' did not caused DAMAGED state 2024-08-03T20:55:41.094 INFO:tasks.cephfs.test_damage:Applying mutation 'Delete 500.00000000' 2024-08-03T20:55:41.095 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-08-03T20:55:41.095 DEBUG:teuthology.orchestra.run.smithi032:> dd if=/proc/self/mounts of=/dev/stdout 2024-08-03T20:55:41.174 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-08-03T20:55:41.174 DEBUG:teuthology.orchestra.run.smithi032:> dd if=/proc/self/mounts of=/dev/stdout 2024-08-03T20:55:41.250 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-08-03T20:55:41.521 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:41.520+0000 7f6c91800640 1 -- 172.21.15.32:0/2421932724 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6c70003a30 msgr2=0x7f6c70003ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:55:41.521 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:41.520+0000 7f6c91800640 1 --2- 172.21.15.32:0/2421932724 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6c70003a30 0x7f6c70003ea0 secure :-1 s=READY pgs=1226 cs=0 l=1 rev1=1 crypto rx=0x7f6c7c009b50 tx=0x7f6c70005830 comp rx=0 tx=0).stop 2024-08-03T20:55:41.521 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:41.520+0000 7f6c90e00640 1 -- 172.21.15.32:0/2421932724 reap_dead start 2024-08-03T20:55:41.521 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:41.520+0000 7f6c91800640 1 -- 172.21.15.32:0/2421932724 shutdown_connections 2024-08-03T20:55:41.521 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:41.520+0000 7f6c91800640 1 -- 172.21.15.32:0/2421932724 >> 172.21.15.32:0/2421932724 conn(0x7f6c8c0b2120 msgr2=0x7f6c8c0cfd60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:55:41.521 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:41.520+0000 7f6c91800640 1 -- 172.21.15.32:0/2421932724 shutdown_connections 2024-08-03T20:55:41.521 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:41.520+0000 7f6c91800640 1 -- 172.21.15.32:0/2421932724 wait complete. 2024-08-03T20:55:41.523 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:41.522+0000 7f6c91800640 1 Processor -- start 2024-08-03T20:55:41.523 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:41.523+0000 7f6c91800640 1 -- start start 2024-08-03T20:55:41.525 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:41.524+0000 7f6c91800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6c70003a30 0x7f6c8c155a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:55:41.525 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:41.524+0000 7f6c91800640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f6c8c156070 0x7f6c8c15b950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:55:41.525 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:41.524+0000 7f6c91800640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f6c8c15be90 0x7f6c8c15ea80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:55:41.526 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:41.524+0000 7f6c91800640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f6c8c0742b0 con 0x7f6c70003a30 2024-08-03T20:55:41.526 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:41.524+0000 7f6c91800640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f6c8c160800 con 0x7f6c8c156070 2024-08-03T20:55:41.526 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:41.524+0000 7f6c91800640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7f6c8c074430 con 0x7f6c8c15be90 2024-08-03T20:55:41.526 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:41.524+0000 7f6c8be00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6c70003a30 0x7f6c8c155a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:55:41.526 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:41.524+0000 7f6c8be00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6c70003a30 0x7f6c8c155a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:59758/0 (socket says 172.21.15.32:59758) 2024-08-03T20:55:41.526 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:41.524+0000 7f6c8be00640 1 -- 172.21.15.32:0/3643552324 learned_addr learned my addr 172.21.15.32:0/3643552324 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:55:41.526 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:41.525+0000 7f6c83400640 1 --2- 172.21.15.32:0/3643552324 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f6c8c156070 0x7f6c8c15b950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:55:41.526 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:41.525+0000 7f6c90e00640 1 --2- 172.21.15.32:0/3643552324 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f6c8c15be90 0x7f6c8c15ea80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:55:41.526 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:41.525+0000 7f6c8be00640 1 -- 172.21.15.32:0/3643552324 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f6c8c15be90 msgr2=0x7f6c8c15ea80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:55:41.526 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:41.525+0000 7f6c8be00640 1 --2- 172.21.15.32:0/3643552324 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f6c8c15be90 0x7f6c8c15ea80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:41.526 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:41.525+0000 7f6c8be00640 1 -- 172.21.15.32:0/3643552324 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f6c8c156070 msgr2=0x7f6c8c15b950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:55:41.526 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:41.525+0000 7f6c8be00640 1 --2- 172.21.15.32:0/3643552324 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f6c8c156070 0x7f6c8c15b950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:41.527 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:41.525+0000 7f6c8be00640 1 -- 172.21.15.32:0/3643552324 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6c8c15c2a0 con 0x7f6c70003a30 2024-08-03T20:55:41.527 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:41.525+0000 7f6c90e00640 1 --2- 172.21.15.32:0/3643552324 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f6c8c15be90 0x7f6c8c15ea80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-03T20:55:41.527 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:41.525+0000 7f6c83400640 1 --2- 172.21.15.32:0/3643552324 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f6c8c156070 0x7f6c8c15b950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-03T20:55:41.527 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:41.525+0000 7f6c8be00640 1 --2- 172.21.15.32:0/3643552324 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6c70003a30 0x7f6c8c155a30 secure :-1 s=READY pgs=1227 cs=0 l=1 rev1=1 crypto rx=0x7f6c7c009c70 tx=0x7f6c7c033670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:55:41.527 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:41.525+0000 7f6c88c00640 1 -- 172.21.15.32:0/3643552324 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c7c038040 con 0x7f6c70003a30 2024-08-03T20:55:41.528 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:41.525+0000 7f6c88c00640 1 -- 172.21.15.32:0/3643552324 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f6c7c004020 con 0x7f6c70003a30 2024-08-03T20:55:41.528 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:41.526+0000 7f6c91800640 1 -- 172.21.15.32:0/3643552324 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6c8c09ba40 con 0x7f6c70003a30 2024-08-03T20:55:41.528 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:41.526+0000 7f6c88c00640 1 -- 172.21.15.32:0/3643552324 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c7c004470 con 0x7f6c70003a30 2024-08-03T20:55:41.528 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:41.526+0000 7f6c91800640 1 -- 172.21.15.32:0/3643552324 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6c8c0b5b30 con 0x7f6c70003a30 2024-08-03T20:55:41.528 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:41.527+0000 7f6c91800640 1 -- 172.21.15.32:0/3643552324 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6c8c0ad350 con 0x7f6c70003a30 2024-08-03T20:55:41.528 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:41.528+0000 7f6c88c00640 1 -- 172.21.15.32:0/3643552324 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f6c7c045020 con 0x7f6c70003a30 2024-08-03T20:55:41.529 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:41.529+0000 7f6c88c00640 1 --2- 172.21.15.32:0/3643552324 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f6c68077040 0x7f6c68079550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:55:41.529 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:41.529+0000 7f6c88c00640 1 -- 172.21.15.32:0/3643552324 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 6370+0+0 (secure 0 0 0) 0x7f6c7c0b9560 con 0x7f6c70003a30 2024-08-03T20:55:41.530 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:41.529+0000 7f6c83400640 1 --2- 172.21.15.32:0/3643552324 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f6c68077040 0x7f6c68079550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:55:41.530 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:41.530+0000 7f6c83400640 1 --2- 172.21.15.32:0/3643552324 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f6c68077040 0x7f6c68079550 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7f6c8c09b530 tx=0x7f6c74006c00 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:55:41.534 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:41.533+0000 7f6c88c00640 1 -- 172.21.15.32:0/3643552324 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7f6c8c15c2a0 con 0x7f6c70003a30 2024-08-03T20:55:41.752 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:41.752+0000 7f6c91800640 1 -- 172.21.15.32:0/3643552324 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f6c8c09db10 con 0x7f6c70003a30 2024-08-03T20:55:41.753 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:41.752+0000 7f6c88c00640 1 -- 172.21.15.32:0/3643552324 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 12 entries v91) ==== 81+0+730 (secure 0 0 0) 0x7f6c7c082540 con 0x7f6c70003a30 2024-08-03T20:55:41.753 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6833/1172050551 2024-08-04T20:55:26.214113+0000 2024-08-03T20:55:41.753 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6835/2292690833 2024-08-04T20:55:20.021329+0000 2024-08-03T20:55:41.753 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6833/2292690833 2024-08-04T20:55:20.021329+0000 2024-08-03T20:55:41.753 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6835/756683988 2024-08-04T20:55:14.119228+0000 2024-08-03T20:55:41.753 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6833/756683988 2024-08-04T20:55:14.119228+0000 2024-08-03T20:55:41.753 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6835/1016338829 2024-08-04T20:55:06.071485+0000 2024-08-03T20:55:41.754 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6835/1172050551 2024-08-04T20:55:26.214113+0000 2024-08-03T20:55:41.754 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6833/1016338829 2024-08-04T20:55:06.071485+0000 2024-08-03T20:55:41.754 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6833/2194902489 2024-08-04T20:55:00.359126+0000 2024-08-03T20:55:41.754 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6835/1707454554 2024-08-04T20:54:38.409411+0000 2024-08-03T20:55:41.754 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6835/2194902489 2024-08-04T20:55:00.359126+0000 2024-08-03T20:55:41.754 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6833/1707454554 2024-08-04T20:54:38.409411+0000 2024-08-03T20:55:41.754 INFO:teuthology.orchestra.run.smithi032.stderr:listed 12 entries 2024-08-03T20:55:41.756 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:41.755+0000 7f6c91800640 1 -- 172.21.15.32:0/3643552324 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f6c68077040 msgr2=0x7f6c68079550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:55:41.756 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:41.755+0000 7f6c91800640 1 --2- 172.21.15.32:0/3643552324 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f6c68077040 0x7f6c68079550 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7f6c8c09b530 tx=0x7f6c74006c00 comp rx=0 tx=0).stop 2024-08-03T20:55:41.756 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:41.755+0000 7f6c91800640 1 -- 172.21.15.32:0/3643552324 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6c70003a30 msgr2=0x7f6c8c155a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:55:41.756 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:41.755+0000 7f6c91800640 1 --2- 172.21.15.32:0/3643552324 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6c70003a30 0x7f6c8c155a30 secure :-1 s=READY pgs=1227 cs=0 l=1 rev1=1 crypto rx=0x7f6c7c009c70 tx=0x7f6c7c033670 comp rx=0 tx=0).stop 2024-08-03T20:55:41.756 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:41.755+0000 7f6c91800640 1 -- 172.21.15.32:0/3643552324 shutdown_connections 2024-08-03T20:55:41.756 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:41.755+0000 7f6c91800640 1 --2- 172.21.15.32:0/3643552324 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f6c68077040 0x7f6c68079550 unknown :-1 s=CLOSED pgs=508 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:41.756 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:41.755+0000 7f6c91800640 1 --2- 172.21.15.32:0/3643552324 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f6c8c156070 0x7f6c8c15b950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:41.756 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:41.755+0000 7f6c91800640 1 --2- 172.21.15.32:0/3643552324 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f6c8c15be90 0x7f6c8c15ea80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:41.756 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:41.755+0000 7f6c91800640 1 --2- 172.21.15.32:0/3643552324 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6c70003a30 0x7f6c8c155a30 unknown :-1 s=CLOSED pgs=1227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:41.756 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:41.755+0000 7f6c91800640 1 -- 172.21.15.32:0/3643552324 >> 172.21.15.32:0/3643552324 conn(0x7f6c8c0b2120 msgr2=0x7f6c8c0ad560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:55:41.756 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:41.755+0000 7f6c91800640 1 -- 172.21.15.32:0/3643552324 shutdown_connections 2024-08-03T20:55:41.757 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:41.755+0000 7f6c91800640 1 -- 172.21.15.32:0/3643552324 wait complete. 2024-08-03T20:55:41.777 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-08-03T20:55:41.778 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-08-03T20:55:41.778 DEBUG:teuthology.orchestra.run.smithi032:> sudo umount /home/ubuntu/cephtest/mnt.0 -f 2024-08-03T20:55:41.861 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi032.front.sepia.ceph.com 2024-08-03T20:55:41.861 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-08-03T20:55:41.861 DEBUG:teuthology.orchestra.run.smithi032:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-08-03T20:55:41.897 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-08-03T20:55:42.167 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:42.166+0000 7f5453600640 1 -- 172.21.15.32:0/3752297984 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f544c073e60 msgr2=0x7f544c074270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:55:42.167 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:42.166+0000 7f5453600640 1 --2- 172.21.15.32:0/3752297984 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f544c073e60 0x7f544c074270 secure :-1 s=READY pgs=1229 cs=0 l=1 rev1=1 crypto rx=0x7f5438005fd0 tx=0x7f543802a540 comp rx=0 tx=0).stop 2024-08-03T20:55:42.167 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:42.166+0000 7f5453600640 1 -- 172.21.15.32:0/3752297984 shutdown_connections 2024-08-03T20:55:42.167 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:42.166+0000 7f5453600640 1 --2- 172.21.15.32:0/3752297984 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f544c073e60 0x7f544c074270 unknown :-1 s=CLOSED pgs=1229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:42.167 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:42.166+0000 7f5453600640 1 --2- 172.21.15.32:0/3752297984 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f544c06e930 0x7f544c144f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:42.168 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:42.166+0000 7f5453600640 1 -- 172.21.15.32:0/3752297984 >> 172.21.15.32:0/3752297984 conn(0x7f544c0b1bd0 msgr2=0x7f544c0b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:55:42.168 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:42.166+0000 7f5453600640 1 -- 172.21.15.32:0/3752297984 shutdown_connections 2024-08-03T20:55:42.168 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:42.166+0000 7f5453600640 1 -- 172.21.15.32:0/3752297984 wait complete. 2024-08-03T20:55:42.169 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:42.168+0000 7f5453600640 1 Processor -- start 2024-08-03T20:55:42.169 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:42.169+0000 7f5453600640 1 -- start start 2024-08-03T20:55:42.171 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:42.170+0000 7f5453600640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f544c06e930 0x7f544c1444c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:55:42.172 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:42.170+0000 7f5453600640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f544c073e60 0x7f544c15b3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:55:42.172 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:42.170+0000 7f5453600640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f544c0747b0 0x7f544c0b2fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:55:42.172 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:42.170+0000 7f5453600640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f544c0cf9c0 con 0x7f544c06e930 2024-08-03T20:55:42.172 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:42.170+0000 7f5453600640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f544c0cf840 con 0x7f544c073e60 2024-08-03T20:55:42.172 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:42.170+0000 7f5453600640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7f544c0ccff0 con 0x7f544c0747b0 2024-08-03T20:55:42.172 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:42.170+0000 7f5450c00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f544c06e930 0x7f544c1444c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:55:42.172 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:42.170+0000 7f5450c00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f544c06e930 0x7f544c1444c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:59788/0 (socket says 172.21.15.32:59788) 2024-08-03T20:55:42.172 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:42.170+0000 7f5450c00640 1 -- 172.21.15.32:0/1799620934 learned_addr learned my addr 172.21.15.32:0/1799620934 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:55:42.172 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:42.171+0000 7f544be00640 1 --2- 172.21.15.32:0/1799620934 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f544c073e60 0x7f544c15b3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:55:42.172 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:42.171+0000 7f5451600640 1 --2- 172.21.15.32:0/1799620934 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f544c0747b0 0x7f544c0b2fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:55:42.172 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:42.171+0000 7f5450c00640 1 -- 172.21.15.32:0/1799620934 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f544c0747b0 msgr2=0x7f544c0b2fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:55:42.172 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:42.171+0000 7f5450c00640 1 --2- 172.21.15.32:0/1799620934 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f544c0747b0 0x7f544c0b2fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:42.172 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:42.171+0000 7f5450c00640 1 -- 172.21.15.32:0/1799620934 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f544c073e60 msgr2=0x7f544c15b3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:55:42.173 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:42.171+0000 7f5450c00640 1 --2- 172.21.15.32:0/1799620934 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f544c073e60 0x7f544c15b3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:42.173 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:42.171+0000 7f5450c00640 1 -- 172.21.15.32:0/1799620934 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f544c143770 con 0x7f544c06e930 2024-08-03T20:55:42.173 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:42.171+0000 7f5450c00640 1 --2- 172.21.15.32:0/1799620934 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f544c06e930 0x7f544c1444c0 secure :-1 s=READY pgs=1230 cs=0 l=1 rev1=1 crypto rx=0x7f544000d740 tx=0x7f544000dc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:55:42.173 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:42.172+0000 7f5448c00640 1 -- 172.21.15.32:0/1799620934 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5440017070 con 0x7f544c06e930 2024-08-03T20:55:42.174 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:42.172+0000 7f5448c00640 1 -- 172.21.15.32:0/1799620934 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f5440009d20 con 0x7f544c06e930 2024-08-03T20:55:42.174 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:42.172+0000 7f5453600640 1 -- 172.21.15.32:0/1799620934 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f544c1f5d70 con 0x7f544c06e930 2024-08-03T20:55:42.174 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:42.172+0000 7f5448c00640 1 -- 172.21.15.32:0/1799620934 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54400106c0 con 0x7f544c06e930 2024-08-03T20:55:42.174 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:42.172+0000 7f5453600640 1 -- 172.21.15.32:0/1799620934 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f544c143cc0 con 0x7f544c06e930 2024-08-03T20:55:42.174 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:42.173+0000 7f5453600640 1 -- 172.21.15.32:0/1799620934 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f544c0cd830 con 0x7f544c06e930 2024-08-03T20:55:42.174 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:42.174+0000 7f5448c00640 1 -- 172.21.15.32:0/1799620934 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f544001a070 con 0x7f544c06e930 2024-08-03T20:55:42.175 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:42.175+0000 7f5448c00640 1 --2- 172.21.15.32:0/1799620934 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f542c077010 0x7f542c079520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:55:42.176 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:42.175+0000 7f5448c00640 1 -- 172.21.15.32:0/1799620934 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 6370+0+0 (secure 0 0 0) 0x7f5440098c60 con 0x7f544c06e930 2024-08-03T20:55:42.176 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:42.175+0000 7f544be00640 1 --2- 172.21.15.32:0/1799620934 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f542c077010 0x7f542c079520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:55:42.176 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:42.176+0000 7f544be00640 1 --2- 172.21.15.32:0/1799620934 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f542c077010 0x7f542c079520 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7f5438033000 tx=0x7f5438004730 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:55:42.179 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:42.179+0000 7f5448c00640 1 -- 172.21.15.32:0/1799620934 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7f544c0cd830 con 0x7f544c06e930 2024-08-03T20:55:42.433 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:42.432+0000 7f5453600640 1 -- 172.21.15.32:0/1799620934 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7f544c0cd530 con 0x7f544c06e930 2024-08-03T20:55:42.433 INFO:tasks.ceph.mon.a.smithi032.stderr:2024-08-03T20:55:42.433+0000 7f4e6d600640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-08-03T20:55:42.435 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:42.435+0000 7f5448c00640 1 -- 172.21.15.32:0/1799620934 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v132) ==== 161+0+0 (secure 0 0 0) 0x7f5440061560 con 0x7f544c06e930 2024-08-03T20:55:42.435 INFO:teuthology.orchestra.run.smithi032.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-08-03T20:55:42.439 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:42.438+0000 7f5453600640 1 -- 172.21.15.32:0/1799620934 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f542c077010 msgr2=0x7f542c079520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:55:42.439 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:42.438+0000 7f5453600640 1 --2- 172.21.15.32:0/1799620934 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f542c077010 0x7f542c079520 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7f5438033000 tx=0x7f5438004730 comp rx=0 tx=0).stop 2024-08-03T20:55:42.439 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:42.438+0000 7f5453600640 1 -- 172.21.15.32:0/1799620934 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f544c06e930 msgr2=0x7f544c1444c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:55:42.439 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:42.438+0000 7f5453600640 1 --2- 172.21.15.32:0/1799620934 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f544c06e930 0x7f544c1444c0 secure :-1 s=READY pgs=1230 cs=0 l=1 rev1=1 crypto rx=0x7f544000d740 tx=0x7f544000dc00 comp rx=0 tx=0).stop 2024-08-03T20:55:42.440 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:42.439+0000 7f5453600640 1 -- 172.21.15.32:0/1799620934 shutdown_connections 2024-08-03T20:55:42.440 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:42.439+0000 7f5453600640 1 --2- 172.21.15.32:0/1799620934 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f542c077010 0x7f542c079520 unknown :-1 s=CLOSED pgs=509 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:42.440 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:42.439+0000 7f5453600640 1 --2- 172.21.15.32:0/1799620934 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f544c073e60 0x7f544c15b3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:42.440 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:42.439+0000 7f5453600640 1 --2- 172.21.15.32:0/1799620934 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f544c0747b0 0x7f544c0b2fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:42.440 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:42.439+0000 7f5453600640 1 --2- 172.21.15.32:0/1799620934 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f544c06e930 0x7f544c1444c0 unknown :-1 s=CLOSED pgs=1230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:42.440 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:42.439+0000 7f5453600640 1 -- 172.21.15.32:0/1799620934 >> 172.21.15.32:0/1799620934 conn(0x7f544c0b1bd0 msgr2=0x7f544c140d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:55:42.440 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:42.439+0000 7f5453600640 1 -- 172.21.15.32:0/1799620934 shutdown_connections 2024-08-03T20:55:42.440 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:42.439+0000 7f5453600640 1 -- 172.21.15.32:0/1799620934 wait complete. 2024-08-03T20:55:42.441 INFO:tasks.ceph.mds.a.smithi032.stderr: -1939> 2024-08-03T20:55:32.251+0000 7f64fda00640 -1 mds.0.openfiles _load_finish got (2) No such file or directory 2024-08-03T20:55:42.444 INFO:tasks.ceph.mds.a.smithi032.stderr: -764> 2024-08-03T20:55:36.245+0000 7f6502a00640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-08-03T20:55:42.465 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mds repaired 0 2024-08-03T20:55:42.562 INFO:tasks.ceph.mds.a.smithi032.stdout:starting mds.a at 2024-08-03T20:55:42.698 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:42.697+0000 7fce7ac00640 1 -- 172.21.15.32:0/999584507 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fce7406e930 msgr2=0x7fce74144f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:55:42.698 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:42.697+0000 7fce7ac00640 1 --2- 172.21.15.32:0/999584507 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fce7406e930 0x7fce74144f90 secure :-1 s=READY pgs=1233 cs=0 l=1 rev1=1 crypto rx=0x7fce64009830 tx=0x7fce6402a510 comp rx=0 tx=0).stop 2024-08-03T20:55:42.698 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:42.697+0000 7fce7ac00640 1 -- 172.21.15.32:0/999584507 shutdown_connections 2024-08-03T20:55:42.698 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:42.697+0000 7fce7ac00640 1 --2- 172.21.15.32:0/999584507 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fce7406e930 0x7fce74144f90 unknown :-1 s=CLOSED pgs=1233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:42.698 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:42.697+0000 7fce7ac00640 1 -- 172.21.15.32:0/999584507 >> 172.21.15.32:0/999584507 conn(0x7fce740b1bd0 msgr2=0x7fce740b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:55:42.698 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:42.697+0000 7fce7ac00640 1 -- 172.21.15.32:0/999584507 shutdown_connections 2024-08-03T20:55:42.699 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:42.697+0000 7fce7ac00640 1 -- 172.21.15.32:0/999584507 wait complete. 2024-08-03T20:55:42.700 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:42.699+0000 7fce7ac00640 1 Processor -- start 2024-08-03T20:55:42.700 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:42.700+0000 7fce7ac00640 1 -- start start 2024-08-03T20:55:42.703 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:42.701+0000 7fce7ac00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fce7406e930 0x7fce741f54f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:55:42.703 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:42.701+0000 7fce7ac00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fce74073e60 0x7fce741f9aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:55:42.703 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:42.701+0000 7fce7ac00640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fce7415fff0 0x7fce741eb000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:55:42.703 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:42.701+0000 7fce7ac00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fce740cf840 con 0x7fce74073e60 2024-08-03T20:55:42.703 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:42.701+0000 7fce7ac00640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7fce74145f30 con 0x7fce7406e930 2024-08-03T20:55:42.704 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:42.701+0000 7fce7ac00640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7fce740cf9c0 con 0x7fce7415fff0 2024-08-03T20:55:42.704 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:42.701+0000 7fce73400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fce74073e60 0x7fce741f9aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:55:42.704 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:42.701+0000 7fce73400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fce74073e60 0x7fce741f9aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:59814/0 (socket says 172.21.15.32:59814) 2024-08-03T20:55:42.704 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:42.701+0000 7fce73400640 1 -- 172.21.15.32:0/2445594352 learned_addr learned my addr 172.21.15.32:0/2445594352 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:55:42.704 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:42.702+0000 7fce73e00640 1 --2- 172.21.15.32:0/2445594352 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fce7406e930 0x7fce741f54f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:55:42.704 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:42.702+0000 7fce78c00640 1 --2- 172.21.15.32:0/2445594352 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fce7415fff0 0x7fce741eb000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:55:42.704 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:42.702+0000 7fce73400640 1 -- 172.21.15.32:0/2445594352 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fce7415fff0 msgr2=0x7fce741eb000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:55:42.704 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:42.702+0000 7fce73400640 1 --2- 172.21.15.32:0/2445594352 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fce7415fff0 0x7fce741eb000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:42.704 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:42.702+0000 7fce73400640 1 -- 172.21.15.32:0/2445594352 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fce7406e930 msgr2=0x7fce741f54f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:55:42.704 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:42.702+0000 7fce73400640 1 --2- 172.21.15.32:0/2445594352 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fce7406e930 0x7fce741f54f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:42.704 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:42.702+0000 7fce73400640 1 -- 172.21.15.32:0/2445594352 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fce741f9fe0 con 0x7fce74073e60 2024-08-03T20:55:42.705 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:42.702+0000 7fce73400640 1 --2- 172.21.15.32:0/2445594352 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fce74073e60 0x7fce741f9aa0 secure :-1 s=READY pgs=1234 cs=0 l=1 rev1=1 crypto rx=0x7fce60009fd0 tx=0x7fce6000b870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:55:42.705 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:42.702+0000 7fce6be00640 1 -- 172.21.15.32:0/2445594352 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce60015020 con 0x7fce74073e60 2024-08-03T20:55:42.705 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:42.702+0000 7fce6be00640 1 -- 172.21.15.32:0/2445594352 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fce60011920 con 0x7fce74073e60 2024-08-03T20:55:42.705 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:42.702+0000 7fce7ac00640 1 -- 172.21.15.32:0/2445594352 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fce7414b610 con 0x7fce74073e60 2024-08-03T20:55:42.705 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:42.702+0000 7fce6be00640 1 -- 172.21.15.32:0/2445594352 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce60011d50 con 0x7fce74073e60 2024-08-03T20:55:42.705 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:42.703+0000 7fce7ac00640 1 -- 172.21.15.32:0/2445594352 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fce7414bbd0 con 0x7fce74073e60 2024-08-03T20:55:42.705 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:42.703+0000 7fce7ac00640 1 -- 172.21.15.32:0/2445594352 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fce74160e20 con 0x7fce74073e60 2024-08-03T20:55:42.706 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:42.705+0000 7fce6be00640 1 -- 172.21.15.32:0/2445594352 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7fce60018020 con 0x7fce74073e60 2024-08-03T20:55:42.706 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:42.706+0000 7fce6be00640 1 --2- 172.21.15.32:0/2445594352 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fce54077010 0x7fce54079520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:55:42.706 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:42.706+0000 7fce6be00640 1 -- 172.21.15.32:0/2445594352 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 6456+0+0 (secure 0 0 0) 0x7fce60098bf0 con 0x7fce74073e60 2024-08-03T20:55:42.707 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:42.706+0000 7fce73e00640 1 --2- 172.21.15.32:0/2445594352 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fce54077010 0x7fce54079520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:55:42.707 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:42.707+0000 7fce73e00640 1 --2- 172.21.15.32:0/2445594352 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fce54077010 0x7fce54079520 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7fce64034000 tx=0x7fce64004730 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:55:42.710 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:42.710+0000 7fce6be00640 1 -- 172.21.15.32:0/2445594352 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7fce6001f020 con 0x7fce74073e60 2024-08-03T20:55:42.996 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:42.995+0000 7fce7ac00640 1 -- 172.21.15.32:0/2445594352 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "mds repaired", "role": "0"} v 0) -- 0x7fce74144b90 con 0x7fce74073e60 2024-08-03T20:55:43.438 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:43.438+0000 7fce6be00640 1 -- 172.21.15.32:0/2445594352 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "mds repaired", "role": "0"}]=0 nothing to do: rank is not damaged v133) ==== 107+0+0 (secure 0 0 0) 0x7fce60098020 con 0x7fce74073e60 2024-08-03T20:55:43.439 INFO:teuthology.orchestra.run.smithi032.stderr:nothing to do: rank is not damaged 2024-08-03T20:55:43.441 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:43.440+0000 7fce7ac00640 1 -- 172.21.15.32:0/2445594352 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fce54077010 msgr2=0x7fce54079520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:55:43.441 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:43.440+0000 7fce7ac00640 1 --2- 172.21.15.32:0/2445594352 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fce54077010 0x7fce54079520 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7fce64034000 tx=0x7fce64004730 comp rx=0 tx=0).stop 2024-08-03T20:55:43.441 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:43.440+0000 7fce7ac00640 1 -- 172.21.15.32:0/2445594352 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fce74073e60 msgr2=0x7fce741f9aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:55:43.441 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:43.440+0000 7fce7ac00640 1 --2- 172.21.15.32:0/2445594352 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fce74073e60 0x7fce741f9aa0 secure :-1 s=READY pgs=1234 cs=0 l=1 rev1=1 crypto rx=0x7fce60009fd0 tx=0x7fce6000b870 comp rx=0 tx=0).stop 2024-08-03T20:55:43.441 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:43.440+0000 7fce7ac00640 1 -- 172.21.15.32:0/2445594352 shutdown_connections 2024-08-03T20:55:43.442 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:43.440+0000 7fce7ac00640 1 --2- 172.21.15.32:0/2445594352 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fce54077010 0x7fce54079520 unknown :-1 s=CLOSED pgs=510 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:43.442 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:43.440+0000 7fce7ac00640 1 --2- 172.21.15.32:0/2445594352 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fce7415fff0 0x7fce741eb000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:43.442 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:43.440+0000 7fce7ac00640 1 --2- 172.21.15.32:0/2445594352 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fce74073e60 0x7fce741f9aa0 unknown :-1 s=CLOSED pgs=1234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:43.442 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:43.440+0000 7fce7ac00640 1 --2- 172.21.15.32:0/2445594352 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fce7406e930 0x7fce741f54f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:43.442 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:43.440+0000 7fce7ac00640 1 -- 172.21.15.32:0/2445594352 >> 172.21.15.32:0/2445594352 conn(0x7fce740b1bd0 msgr2=0x7fce740b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:55:43.442 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:43.441+0000 7fce7ac00640 1 -- 172.21.15.32:0/2445594352 shutdown_connections 2024-08-03T20:55:43.442 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:43.441+0000 7fce7ac00640 1 -- 172.21.15.32:0/2445594352 wait complete. 2024-08-03T20:55:43.468 DEBUG:teuthology.orchestra.run.smithi032:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata import - 2024-08-03T20:55:43.612 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:43.611+0000 7fa8f5a30f40 1 Processor -- start 2024-08-03T20:55:43.612 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:43.611+0000 7fa8f5a30f40 1 -- start start 2024-08-03T20:55:43.614 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:43.613+0000 7fa8f5a30f40 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x5631acfd5360 0x5631acf5d1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:55:43.615 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:43.613+0000 7fa8f5a30f40 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5631acfec330 0x5631acf1e880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:55:43.615 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:43.613+0000 7fa8f5a30f40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5631acff33f0 0x5631acfcd510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:55:43.615 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:43.613+0000 7fa8f5a30f40 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x5631acecabf0 con 0x5631acfec330 2024-08-03T20:55:43.615 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:43.613+0000 7fa8f5a30f40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x5631aced76b0 con 0x5631acff33f0 2024-08-03T20:55:43.615 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:43.613+0000 7fa8f5a30f40 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x5631ace6ce30 con 0x5631acfd5360 2024-08-03T20:55:43.615 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:43.613+0000 7fa8ee200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5631acfec330 0x5631acf1e880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:55:43.615 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:43.613+0000 7fa8ee200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5631acfec330 0x5631acf1e880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:59832/0 (socket says 172.21.15.32:59832) 2024-08-03T20:55:43.615 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:43.613+0000 7fa8ee200640 1 -- 172.21.15.32:0/272584295 learned_addr learned my addr 172.21.15.32:0/272584295 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:55:43.615 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:43.613+0000 7fa8ef600640 1 --2- 172.21.15.32:0/272584295 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5631acff33f0 0x5631acfcd510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:55:43.615 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:43.613+0000 7fa8eec00640 1 --2- 172.21.15.32:0/272584295 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x5631acfd5360 0x5631acf5d1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:55:43.615 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:43.613+0000 7fa8ee200640 1 -- 172.21.15.32:0/272584295 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x5631acfd5360 msgr2=0x5631acf5d1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:55:43.615 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:43.614+0000 7fa8ee200640 1 --2- 172.21.15.32:0/272584295 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x5631acfd5360 0x5631acf5d1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:43.616 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:43.614+0000 7fa8ee200640 1 -- 172.21.15.32:0/272584295 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5631acff33f0 msgr2=0x5631acfcd510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:55:43.616 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:43.614+0000 7fa8ee200640 1 --2- 172.21.15.32:0/272584295 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5631acff33f0 0x5631acfcd510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:43.616 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:43.614+0000 7fa8ee200640 1 -- 172.21.15.32:0/272584295 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5631acf1edc0 con 0x5631acfec330 2024-08-03T20:55:43.616 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:43.614+0000 7fa8eec00640 1 --2- 172.21.15.32:0/272584295 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x5631acfd5360 0x5631acf5d1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-03T20:55:43.616 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:43.614+0000 7fa8ee200640 1 --2- 172.21.15.32:0/272584295 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5631acfec330 0x5631acf1e880 secure :-1 s=READY pgs=1237 cs=0 l=1 rev1=1 crypto rx=0x7fa8dc001d50 tx=0x7fa8dc00b5b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2e6602e69cf97d05 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:55:43.616 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:43.614+0000 7fa8ece00640 1 -- 172.21.15.32:0/272584295 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8dc00bd80 con 0x5631acfec330 2024-08-03T20:55:43.616 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:43.614+0000 7fa8ece00640 1 -- 172.21.15.32:0/272584295 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fa8dc00f400 con 0x5631acfec330 2024-08-03T20:55:43.616 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:43.614+0000 7fa8ece00640 1 -- 172.21.15.32:0/272584295 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8dc00f850 con 0x5631acfec330 2024-08-03T20:55:43.617 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:43.615+0000 7fa8f5a30f40 1 -- 172.21.15.32:0/272584295 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5631acfec330 msgr2=0x5631acf1e880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:55:43.617 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:43.615+0000 7fa8f5a30f40 1 --2- 172.21.15.32:0/272584295 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5631acfec330 0x5631acf1e880 secure :-1 s=READY pgs=1237 cs=0 l=1 rev1=1 crypto rx=0x7fa8dc001d50 tx=0x7fa8dc00b5b0 comp rx=0 tx=0).stop 2024-08-03T20:55:43.617 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:43.615+0000 7fa8f5a30f40 1 -- 172.21.15.32:0/272584295 shutdown_connections 2024-08-03T20:55:43.617 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:43.615+0000 7fa8f5a30f40 1 --2- 172.21.15.32:0/272584295 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5631acff33f0 0x5631acfcd510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:43.617 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:43.615+0000 7fa8f5a30f40 1 --2- 172.21.15.32:0/272584295 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5631acfec330 0x5631acf1e880 unknown :-1 s=CLOSED pgs=1237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:43.618 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:43.615+0000 7fa8f5a30f40 1 --2- 172.21.15.32:0/272584295 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x5631acfd5360 0x5631acf5d1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:43.618 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:43.615+0000 7fa8f5a30f40 1 -- 172.21.15.32:0/272584295 >> 172.21.15.32:0/272584295 conn(0x5631ace3eac0 msgr2=0x5631acfeb550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:55:43.618 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:43.615+0000 7fa8f5a30f40 1 -- 172.21.15.32:0/272584295 shutdown_connections 2024-08-03T20:55:43.618 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:43.615+0000 7fa8f5a30f40 1 -- 172.21.15.32:0/272584295 wait complete. 2024-08-03T20:55:43.618 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:43.616+0000 7fa8f5a30f40 1 Processor -- start 2024-08-03T20:55:43.618 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:43.616+0000 7fa8f5a30f40 1 -- start start 2024-08-03T20:55:43.620 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:43.617+0000 7fa8f5a30f40 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x5631acfd5360 0x5631acfcc890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:55:43.620 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:43.617+0000 7fa8f5a30f40 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5631acfec330 0x5631acfcd510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:55:43.620 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:43.617+0000 7fa8f5a30f40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5631acff33f0 0x5631acfcda50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:55:43.620 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:43.617+0000 7fa8f5a30f40 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x5631acff1130 con 0x5631acfec330 2024-08-03T20:55:43.620 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:43.617+0000 7fa8f5a30f40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x5631acf1cda0 con 0x5631acff33f0 2024-08-03T20:55:43.620 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:43.617+0000 7fa8f5a30f40 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x5631acf601c0 con 0x5631acfd5360 2024-08-03T20:55:43.620 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:43.618+0000 7fa8ee200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5631acfec330 0x5631acfcd510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:55:43.621 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:43.618+0000 7fa8ee200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5631acfec330 0x5631acfcd510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:59844/0 (socket says 172.21.15.32:59844) 2024-08-03T20:55:43.621 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:43.618+0000 7fa8ee200640 1 -- 172.21.15.32:0/30469862 learned_addr learned my addr 172.21.15.32:0/30469862 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:55:43.621 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:43.618+0000 7fa8eec00640 1 --2- 172.21.15.32:0/30469862 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x5631acfd5360 0x5631acfcc890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:55:43.621 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:43.618+0000 7fa8ef600640 1 --2- 172.21.15.32:0/30469862 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5631acff33f0 0x5631acfcda50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:55:43.621 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:43.618+0000 7fa8ee200640 1 -- 172.21.15.32:0/30469862 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x5631acfd5360 msgr2=0x5631acfcc890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:55:43.621 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:43.618+0000 7fa8ee200640 1 --2- 172.21.15.32:0/30469862 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x5631acfd5360 0x5631acfcc890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:43.621 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:43.618+0000 7fa8ee200640 1 -- 172.21.15.32:0/30469862 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5631acff33f0 msgr2=0x5631acfcda50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:55:43.621 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:43.618+0000 7fa8ee200640 1 --2- 172.21.15.32:0/30469862 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5631acff33f0 0x5631acfcda50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:43.621 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:43.618+0000 7fa8ee200640 1 -- 172.21.15.32:0/30469862 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5631acfef110 con 0x5631acfec330 2024-08-03T20:55:43.621 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:43.618+0000 7fa8ee200640 1 --2- 172.21.15.32:0/30469862 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5631acfec330 0x5631acfcd510 secure :-1 s=READY pgs=1238 cs=0 l=1 rev1=1 crypto rx=0x7fa8dc002000 tx=0x7fa8dc00ff50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:55:43.621 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:43.619+0000 7fa8e7400640 1 -- 172.21.15.32:0/30469862 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8dc01b430 con 0x5631acfec330 2024-08-03T20:55:43.621 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:43.619+0000 7fa8e7400640 1 -- 172.21.15.32:0/30469862 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fa8dc020020 con 0x5631acfec330 2024-08-03T20:55:43.621 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:43.619+0000 7fa8e7400640 1 -- 172.21.15.32:0/30469862 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8dc01b700 con 0x5631acfec330 2024-08-03T20:55:43.621 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:43.619+0000 7fa8f5a30f40 1 -- 172.21.15.32:0/30469862 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x5631acf5d840 con 0x5631acfec330 2024-08-03T20:55:43.622 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:43.619+0000 7fa8f5a30f40 1 -- 172.21.15.32:0/30469862 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5631acf5de00 con 0x5631acfec330 2024-08-03T20:55:43.622 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:43.621+0000 7fa8e7400640 1 -- 172.21.15.32:0/30469862 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7fa8dc017070 con 0x5631acfec330 2024-08-03T20:55:43.623 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:43.622+0000 7fa8e7400640 1 --2- 172.21.15.32:0/30469862 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fa8d00759f0 0x7fa8d0077f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:55:43.623 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:43.622+0000 7fa8e7400640 1 -- 172.21.15.32:0/30469862 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 6456+0+0 (secure 0 0 0) 0x7fa8dc098dd0 con 0x5631acfec330 2024-08-03T20:55:43.623 INFO:teuthology.orchestra.run.smithi032.stdout:Importing pool 2024-08-03T20:55:43.624 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:43.622+0000 7fa8f5a30f40 1 --2- 172.21.15.32:0/30469862 >> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] conn(0x5631acfd43e0 0x5631acf5eb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:55:43.624 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:43.622+0000 7fa8f5a30f40 1 -- 172.21.15.32:0/30469862 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:1 12.10 12:09401a81:::601.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e92) -- 0x5631acf5f050 con 0x5631acfd43e0 2024-08-03T20:55:43.624 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:43.623+0000 7fa8eec00640 1 --2- 172.21.15.32:0/30469862 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fa8d00759f0 0x7fa8d0077f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:55:43.624 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:43.623+0000 7fa8ef600640 1 --2- 172.21.15.32:0/30469862 >> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] conn(0x5631acfd43e0 0x5631acf5eb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:55:43.624 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:43.623+0000 7fa8eec00640 1 --2- 172.21.15.32:0/30469862 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fa8d00759f0 0x7fa8d0077f00 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7fa8e0000c00 tx=0x7fa8e0002590 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:55:43.624 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:43.623+0000 7fa8ef600640 1 --2- 172.21.15.32:0/30469862 >> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] conn(0x5631acfd43e0 0x5631acf5eb10 crc :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:55:43.626 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:43.625+0000 7fa8ef600640 1 -- 172.21.15.32:0/30469862 <== osd.0 v2:172.21.15.32:6816/3066156116 1 ==== osd_op_reply(1 601.00000000 [create] v92'195 uv169 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fa8e800b040 con 0x5631acfd43e0 2024-08-03T20:55:43.626 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:43.625+0000 7fa8f5a30f40 1 -- 172.21.15.32:0/30469862 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:2 12.10 12:09401a81:::601.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e92) -- 0x5631ad097c00 con 0x5631acfd43e0 2024-08-03T20:55:43.627 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:43.627+0000 7fa8ef600640 1 -- 172.21.15.32:0/30469862 <== osd.0 v2:172.21.15.32:6816/3066156116 2 ==== osd_op_reply(2 601.00000000 [delete] v92'196 uv196 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fa8e800b040 con 0x5631acfd43e0 2024-08-03T20:55:43.627 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:43.627+0000 7fa8f5a30f40 1 -- 172.21.15.32:0/30469862 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:3 12.10 12:09401a81:::601.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e92) -- 0x5631ad097fa0 con 0x5631acfd43e0 2024-08-03T20:55:43.629 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::601.00000000:head# 2024-08-03T20:55:43.629 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:43.629+0000 7fa8ef600640 1 -- 172.21.15.32:0/30469862 <== osd.0 v2:172.21.15.32:6816/3066156116 3 ==== osd_op_reply(3 601.00000000 [create] v92'197 uv197 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fa8e800b040 con 0x5631acfd43e0 2024-08-03T20:55:43.629 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:43.629+0000 7fa8f5a30f40 1 -- 172.21.15.32:0/30469862 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:4 12.10 12:09401a81:::601.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e92) -- 0x5631ad099350 con 0x5631acfd43e0 2024-08-03T20:55:43.631 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:43.630+0000 7fa8ef600640 1 -- 172.21.15.32:0/30469862 <== osd.0 v2:172.21.15.32:6816/3066156116 4 ==== osd_op_reply(4 601.00000000 [setxattr (30)] v92'198 uv198 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fa8e800b040 con 0x5631acfd43e0 2024-08-03T20:55:43.631 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:43.630+0000 7fa8f5a30f40 1 -- 172.21.15.32:0/30469862 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:5 12.10 12:09401a81:::601.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e92) -- 0x5631ad0996f0 con 0x5631acfd43e0 2024-08-03T20:55:43.632 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:43.632+0000 7fa8ef600640 1 -- 172.21.15.32:0/30469862 <== osd.0 v2:172.21.15.32:6816/3066156116 5 ==== osd_op_reply(5 601.00000000 [setxattr (62)] v92'199 uv199 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fa8e800b040 con 0x5631acfd43e0 2024-08-03T20:55:43.632 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:43.632+0000 7fa8f5a30f40 1 -- 172.21.15.32:0/30469862 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:6 12.10 12:09401a81:::601.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e92) -- 0x5631ad098340 con 0x5631acfd43e0 2024-08-03T20:55:43.634 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:43.634+0000 7fa8ef600640 1 -- 172.21.15.32:0/30469862 <== osd.0 v2:172.21.15.32:6816/3066156116 6 ==== osd_op_reply(6 601.00000000 [omap-set-header] v92'200 uv200 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fa8e800b040 con 0x5631acfd43e0 2024-08-03T20:55:43.634 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:43.634+0000 7fa8f5a30f40 1 -- 172.21.15.32:0/30469862 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:7 12.10 12:0bd6d154:::602.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e92) -- 0x5631ad0986e0 con 0x5631acfd43e0 2024-08-03T20:55:43.635 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:43.635+0000 7fa8ef600640 1 -- 172.21.15.32:0/30469862 <== osd.0 v2:172.21.15.32:6816/3066156116 7 ==== osd_op_reply(7 602.00000000 [create] v92'201 uv175 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fa8e800b040 con 0x5631acfd43e0 2024-08-03T20:55:43.636 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:43.635+0000 7fa8f5a30f40 1 -- 172.21.15.32:0/30469862 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:8 12.10 12:0bd6d154:::602.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e92) -- 0x5631ad098ad0 con 0x5631acfd43e0 2024-08-03T20:55:43.637 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:43.637+0000 7fa8ef600640 1 -- 172.21.15.32:0/30469862 <== osd.0 v2:172.21.15.32:6816/3066156116 8 ==== osd_op_reply(8 602.00000000 [delete] v92'202 uv202 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fa8e800b040 con 0x5631acfd43e0 2024-08-03T20:55:43.637 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:43.637+0000 7fa8f5a30f40 1 -- 172.21.15.32:0/30469862 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:9 12.10 12:0bd6d154:::602.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e92) -- 0x5631ad098ad0 con 0x5631acfd43e0 2024-08-03T20:55:43.638 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::602.00000000:head# 2024-08-03T20:55:43.639 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:43.638+0000 7fa8ef600640 1 -- 172.21.15.32:0/30469862 <== osd.0 v2:172.21.15.32:6816/3066156116 9 ==== osd_op_reply(9 602.00000000 [create] v92'203 uv203 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fa8e800b040 con 0x5631acfd43e0 2024-08-03T20:55:43.639 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:43.638+0000 7fa8f5a30f40 1 -- 172.21.15.32:0/30469862 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:10 12.10 12:0bd6d154:::602.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e92) -- 0x5631ad098a80 con 0x5631acfd43e0 2024-08-03T20:55:43.640 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:43.640+0000 7fa8ef600640 1 -- 172.21.15.32:0/30469862 <== osd.0 v2:172.21.15.32:6816/3066156116 10 ==== osd_op_reply(10 602.00000000 [setxattr (30)] v92'204 uv204 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fa8e800b040 con 0x5631acfd43e0 2024-08-03T20:55:43.640 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:43.640+0000 7fa8f5a30f40 1 -- 172.21.15.32:0/30469862 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:11 12.10 12:0bd6d154:::602.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e92) -- 0x5631ad098a80 con 0x5631acfd43e0 2024-08-03T20:55:43.642 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:43.641+0000 7fa8ef600640 1 -- 172.21.15.32:0/30469862 <== osd.0 v2:172.21.15.32:6816/3066156116 11 ==== osd_op_reply(11 602.00000000 [setxattr (62)] v92'205 uv205 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fa8e800b040 con 0x5631acfd43e0 2024-08-03T20:55:43.642 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:43.641+0000 7fa8f5a30f40 1 -- 172.21.15.32:0/30469862 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:12 12.10 12:0bd6d154:::602.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e92) -- 0x5631ad098a80 con 0x5631acfd43e0 2024-08-03T20:55:43.643 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:43.643+0000 7fa8ef600640 1 -- 172.21.15.32:0/30469862 <== osd.0 v2:172.21.15.32:6816/3066156116 12 ==== osd_op_reply(12 602.00000000 [omap-set-header] v92'206 uv206 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fa8e800b040 con 0x5631acfd43e0 2024-08-03T20:55:43.643 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:43.643+0000 7fa8f5a30f40 1 -- 172.21.15.32:0/30469862 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:13 12.10 12:0cf6ad0f:::10000000000.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e92) -- 0x5631ad098a80 con 0x5631acfd43e0 2024-08-03T20:55:43.645 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:43.645+0000 7fa8ef600640 1 -- 172.21.15.32:0/30469862 <== osd.0 v2:172.21.15.32:6816/3066156116 13 ==== osd_op_reply(13 10000000000.00000000 [create] v92'207 uv182 ondisk = -17 ((17) File exists)) ==== 164+0+0 (crc 0 0 0) 0x7fa8e800b040 con 0x5631acfd43e0 2024-08-03T20:55:43.645 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:43.645+0000 7fa8f5a30f40 1 -- 172.21.15.32:0/30469862 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:14 12.10 12:0cf6ad0f:::10000000000.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e92) -- 0x5631ad098a80 con 0x5631acfd43e0 2024-08-03T20:55:43.647 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:43.646+0000 7fa8ef600640 1 -- 172.21.15.32:0/30469862 <== osd.0 v2:172.21.15.32:6816/3066156116 14 ==== osd_op_reply(14 10000000000.00000000 [delete] v92'208 uv208 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7fa8e800b040 con 0x5631acfd43e0 2024-08-03T20:55:43.647 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:43.646+0000 7fa8f5a30f40 1 -- 172.21.15.32:0/30469862 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:15 12.10 12:0cf6ad0f:::10000000000.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e92) -- 0x5631ad098a80 con 0x5631acfd43e0 2024-08-03T20:55:43.648 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::10000000000.00000000:head# 2024-08-03T20:55:43.648 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:43.648+0000 7fa8ef600640 1 -- 172.21.15.32:0/30469862 <== osd.0 v2:172.21.15.32:6816/3066156116 15 ==== osd_op_reply(15 10000000000.00000000 [create] v92'209 uv209 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7fa8e800b040 con 0x5631acfd43e0 2024-08-03T20:55:43.648 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:43.648+0000 7fa8f5a30f40 1 -- 172.21.15.32:0/30469862 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:16 12.10 12:0cf6ad0f:::10000000000.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e92) -- 0x5631ad098a80 con 0x5631acfd43e0 2024-08-03T20:55:43.650 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:43.649+0000 7fa8ef600640 1 -- 172.21.15.32:0/30469862 <== osd.0 v2:172.21.15.32:6816/3066156116 16 ==== osd_op_reply(16 10000000000.00000000 [setxattr (30)] v92'210 uv210 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7fa8e800b040 con 0x5631acfd43e0 2024-08-03T20:55:43.650 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:43.649+0000 7fa8f5a30f40 1 -- 172.21.15.32:0/30469862 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:17 12.10 12:0cf6ad0f:::10000000000.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e92) -- 0x5631ad098a80 con 0x5631acfd43e0 2024-08-03T20:55:43.651 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:43.651+0000 7fa8ef600640 1 -- 172.21.15.32:0/30469862 <== osd.0 v2:172.21.15.32:6816/3066156116 17 ==== osd_op_reply(17 10000000000.00000000 [setxattr (62)] v92'211 uv211 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7fa8e800b040 con 0x5631acfd43e0 2024-08-03T20:55:43.651 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:43.651+0000 7fa8f5a30f40 1 -- 172.21.15.32:0/30469862 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:18 12.10 12:0cf6ad0f:::10000000000.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e92) -- 0x5631ad098a80 con 0x5631acfd43e0 2024-08-03T20:55:43.653 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:43.652+0000 7fa8ef600640 1 -- 172.21.15.32:0/30469862 <== osd.0 v2:172.21.15.32:6816/3066156116 18 ==== osd_op_reply(18 10000000000.00000000 [omap-set-header] v92'212 uv212 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7fa8e800b040 con 0x5631acfd43e0 2024-08-03T20:55:43.653 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:43.652+0000 7fa8f5a30f40 1 -- 172.21.15.32:0/30469862 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:19 12.10 12:0cf6ad0f:::10000000000.00000000:head [omap-set-vals in=593b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e92) -- 0x5631ad098a80 con 0x5631acfd43e0 2024-08-03T20:55:43.655 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:43.654+0000 7fa8ef600640 1 -- 172.21.15.32:0/30469862 <== osd.0 v2:172.21.15.32:6816/3066156116 19 ==== osd_op_reply(19 10000000000.00000000 [omap-set-vals] v92'213 uv213 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7fa8e800b040 con 0x5631acfd43e0 2024-08-03T20:55:43.655 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:43.654+0000 7fa8f5a30f40 1 -- 172.21.15.32:0/30469862 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:20 12.10 12:0d82a743:::600.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e92) -- 0x5631ad098a80 con 0x5631acfd43e0 2024-08-03T20:55:43.656 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:43.655+0000 7fa8ef600640 1 -- 172.21.15.32:0/30469862 <== osd.0 v2:172.21.15.32:6816/3066156116 20 ==== osd_op_reply(20 600.00000000 [create] v92'214 uv188 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fa8e800b040 con 0x5631acfd43e0 2024-08-03T20:55:43.656 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:43.656+0000 7fa8f5a30f40 1 -- 172.21.15.32:0/30469862 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:21 12.10 12:0d82a743:::600.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e92) -- 0x5631ad098a80 con 0x5631acfd43e0 2024-08-03T20:55:43.657 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:43.657+0000 7fa8ef600640 1 -- 172.21.15.32:0/30469862 <== osd.0 v2:172.21.15.32:6816/3066156116 21 ==== osd_op_reply(21 600.00000000 [delete] v92'215 uv215 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fa8e800b040 con 0x5631acfd43e0 2024-08-03T20:55:43.658 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:43.657+0000 7fa8f5a30f40 1 -- 172.21.15.32:0/30469862 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:22 12.10 12:0d82a743:::600.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e92) -- 0x5631ad098a80 con 0x5631acfd43e0 2024-08-03T20:55:43.659 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::600.00000000:head# 2024-08-03T20:55:43.659 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:43.659+0000 7fa8ef600640 1 -- 172.21.15.32:0/30469862 <== osd.0 v2:172.21.15.32:6816/3066156116 22 ==== osd_op_reply(22 600.00000000 [create] v92'216 uv216 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fa8e800b040 con 0x5631acfd43e0 2024-08-03T20:55:43.659 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:43.659+0000 7fa8f5a30f40 1 -- 172.21.15.32:0/30469862 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:23 12.10 12:0d82a743:::600.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e92) -- 0x5631ad098a80 con 0x5631acfd43e0 2024-08-03T20:55:43.661 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:43.660+0000 7fa8ef600640 1 -- 172.21.15.32:0/30469862 <== osd.0 v2:172.21.15.32:6816/3066156116 23 ==== osd_op_reply(23 600.00000000 [setxattr (30)] v92'217 uv217 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fa8e800b040 con 0x5631acfd43e0 2024-08-03T20:55:43.661 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:43.660+0000 7fa8f5a30f40 1 -- 172.21.15.32:0/30469862 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:24 12.10 12:0d82a743:::600.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e92) -- 0x5631ad098a80 con 0x5631acfd43e0 2024-08-03T20:55:43.662 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:43.662+0000 7fa8ef600640 1 -- 172.21.15.32:0/30469862 <== osd.0 v2:172.21.15.32:6816/3066156116 24 ==== osd_op_reply(24 600.00000000 [setxattr (62)] v92'218 uv218 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fa8e800b040 con 0x5631acfd43e0 2024-08-03T20:55:43.662 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:43.662+0000 7fa8f5a30f40 1 -- 172.21.15.32:0/30469862 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:25 12.10 12:0d82a743:::600.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e92) -- 0x5631ad098a80 con 0x5631acfd43e0 2024-08-03T20:55:43.664 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:43.663+0000 7fa8ef600640 1 -- 172.21.15.32:0/30469862 <== osd.0 v2:172.21.15.32:6816/3066156116 25 ==== osd_op_reply(25 600.00000000 [omap-set-header] v92'219 uv219 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fa8e800b040 con 0x5631acfd43e0 2024-08-03T20:55:43.664 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:43.663+0000 7fa8f5a30f40 1 -- 172.21.15.32:0/30469862 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:26 12.10 12:0d89b25e:::603.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e92) -- 0x5631ad098a80 con 0x5631acfd43e0 2024-08-03T20:55:43.665 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:43.665+0000 7fa8ef600640 1 -- 172.21.15.32:0/30469862 <== osd.0 v2:172.21.15.32:6816/3066156116 26 ==== osd_op_reply(26 603.00000000 [create] v92'220 uv194 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fa8e800b040 con 0x5631acfd43e0 2024-08-03T20:55:43.665 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:43.665+0000 7fa8f5a30f40 1 -- 172.21.15.32:0/30469862 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:27 12.10 12:0d89b25e:::603.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e92) -- 0x5631ad098a80 con 0x5631acfd43e0 2024-08-03T20:55:43.667 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:43.666+0000 7fa8ef600640 1 -- 172.21.15.32:0/30469862 <== osd.0 v2:172.21.15.32:6816/3066156116 27 ==== osd_op_reply(27 603.00000000 [delete] v92'221 uv221 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fa8e800b040 con 0x5631acfd43e0 2024-08-03T20:55:43.667 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:43.666+0000 7fa8f5a30f40 1 -- 172.21.15.32:0/30469862 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:28 12.10 12:0d89b25e:::603.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e92) -- 0x5631ad098a80 con 0x5631acfd43e0 2024-08-03T20:55:43.668 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::603.00000000:head# 2024-08-03T20:55:43.668 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:43.668+0000 7fa8ef600640 1 -- 172.21.15.32:0/30469862 <== osd.0 v2:172.21.15.32:6816/3066156116 28 ==== osd_op_reply(28 603.00000000 [create] v92'222 uv222 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fa8e800b040 con 0x5631acfd43e0 2024-08-03T20:55:43.669 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:43.668+0000 7fa8f5a30f40 1 -- 172.21.15.32:0/30469862 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:29 12.10 12:0d89b25e:::603.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e92) -- 0x5631ad098a80 con 0x5631acfd43e0 2024-08-03T20:55:43.670 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:43.669+0000 7fa8ef600640 1 -- 172.21.15.32:0/30469862 <== osd.0 v2:172.21.15.32:6816/3066156116 29 ==== osd_op_reply(29 603.00000000 [setxattr (30)] v92'223 uv223 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fa8e800b040 con 0x5631acfd43e0 2024-08-03T20:55:43.670 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:43.669+0000 7fa8f5a30f40 1 -- 172.21.15.32:0/30469862 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:30 12.10 12:0d89b25e:::603.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e92) -- 0x5631ad098a80 con 0x5631acfd43e0 2024-08-03T20:55:43.671 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:43.671+0000 7fa8ef600640 1 -- 172.21.15.32:0/30469862 <== osd.0 v2:172.21.15.32:6816/3066156116 30 ==== osd_op_reply(30 603.00000000 [setxattr (62)] v92'224 uv224 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fa8e800b040 con 0x5631acfd43e0 2024-08-03T20:55:43.671 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:43.671+0000 7fa8f5a30f40 1 -- 172.21.15.32:0/30469862 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:31 12.10 12:0d89b25e:::603.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e92) -- 0x5631ad098a80 con 0x5631acfd43e0 2024-08-03T20:55:43.673 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:43.672+0000 7fa8ef600640 1 -- 172.21.15.32:0/30469862 <== osd.0 v2:172.21.15.32:6816/3066156116 31 ==== osd_op_reply(31 603.00000000 [omap-set-header] v92'225 uv225 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fa8e800b040 con 0x5631acfd43e0 2024-08-03T20:55:43.673 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:43.673+0000 7fa8f5a30f40 1 --2- 172.21.15.32:0/30469862 >> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] conn(0x5631ad098e90 0x5631ad09bfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:55:43.673 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:43.673+0000 7fa8f5a30f40 1 -- 172.21.15.32:0/30469862 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:32 12.14 12:287034c4:::1.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e92) -- 0x5631ad098a80 con 0x5631ad098e90 2024-08-03T20:55:43.673 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:43.673+0000 7fa8eec00640 1 --2- 172.21.15.32:0/30469862 >> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] conn(0x5631ad098e90 0x5631ad09bfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:55:43.674 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:43.674+0000 7fa8eec00640 1 --2- 172.21.15.32:0/30469862 >> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] conn(0x5631ad098e90 0x5631ad09bfb0 crc :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:55:43.678 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:43.678+0000 7fa8eec00640 1 -- 172.21.15.32:0/30469862 <== osd.5 v2:172.21.15.118:6810/2724891770 1 ==== osd_op_reply(32 1.00000000.inode [create] v92'152 uv133 ondisk = -17 ((17) File exists)) ==== 160+0+0 (crc 0 0 0) 0x7fa8e000f780 con 0x5631ad098e90 2024-08-03T20:55:43.678 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:43.678+0000 7fa8f5a30f40 1 -- 172.21.15.32:0/30469862 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:33 12.14 12:287034c4:::1.00000000.inode:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e92) -- 0x5631ad09c4f0 con 0x5631ad098e90 2024-08-03T20:55:43.680 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:43.680+0000 7fa8eec00640 1 -- 172.21.15.32:0/30469862 <== osd.5 v2:172.21.15.118:6810/2724891770 2 ==== osd_op_reply(33 1.00000000.inode [delete] v92'153 uv153 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7fa8e000f780 con 0x5631ad098e90 2024-08-03T20:55:43.680 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:43.680+0000 7fa8f5a30f40 1 -- 172.21.15.32:0/30469862 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:34 12.14 12:287034c4:::1.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e92) -- 0x5631ad09c890 con 0x5631ad098e90 2024-08-03T20:55:43.682 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::1.00000000.inode:head# 2024-08-03T20:55:43.682 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:43.681+0000 7fa8eec00640 1 -- 172.21.15.32:0/30469862 <== osd.5 v2:172.21.15.118:6810/2724891770 3 ==== osd_op_reply(34 1.00000000.inode [create] v92'154 uv154 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7fa8e000f780 con 0x5631ad098e90 2024-08-03T20:55:43.682 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:43.681+0000 7fa8f5a30f40 1 -- 172.21.15.32:0/30469862 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:35 12.14 12:287034c4:::1.00000000.inode:head [write 0~582 in=582b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e92) -- 0x5631ad09cc30 con 0x5631ad098e90 2024-08-03T20:55:43.684 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:43.683+0000 7fa8eec00640 1 -- 172.21.15.32:0/30469862 <== osd.5 v2:172.21.15.118:6810/2724891770 4 ==== osd_op_reply(35 1.00000000.inode [write 0~582] v92'155 uv155 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7fa8e000f780 con 0x5631ad098e90 2024-08-03T20:55:43.684 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:43.683+0000 7fa8f5a30f40 1 -- 172.21.15.32:0/30469862 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:36 12.14 12:287034c4:::1.00000000.inode:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e92) -- 0x5631ad09cfd0 con 0x5631ad098e90 2024-08-03T20:55:43.685 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:43.685+0000 7fa8eec00640 1 -- 172.21.15.32:0/30469862 <== osd.5 v2:172.21.15.118:6810/2724891770 5 ==== osd_op_reply(36 1.00000000.inode [omap-set-header] v92'156 uv156 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7fa8e000f780 con 0x5631ad098e90 2024-08-03T20:55:43.685 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:43.685+0000 7fa8f5a30f40 1 -- 172.21.15.32:0/30469862 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:37 12.14 12:292cf221:::200.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e92) -- 0x5631ad09d370 con 0x5631ad098e90 2024-08-03T20:55:43.687 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:43.686+0000 7fa8eec00640 1 -- 172.21.15.32:0/30469862 <== osd.5 v2:172.21.15.118:6810/2724891770 6 ==== osd_op_reply(37 200.00000000 [create] v92'157 uv150 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fa8e000f780 con 0x5631ad098e90 2024-08-03T20:55:43.687 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:43.686+0000 7fa8f5a30f40 1 -- 172.21.15.32:0/30469862 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:38 12.14 12:292cf221:::200.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e92) -- 0x5631ad09d710 con 0x5631ad098e90 2024-08-03T20:55:43.688 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:43.688+0000 7fa8eec00640 1 -- 172.21.15.32:0/30469862 <== osd.5 v2:172.21.15.118:6810/2724891770 7 ==== osd_op_reply(38 200.00000000 [delete] v92'158 uv158 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fa8e000f780 con 0x5631ad098e90 2024-08-03T20:55:43.688 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:43.688+0000 7fa8f5a30f40 1 -- 172.21.15.32:0/30469862 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:39 12.14 12:292cf221:::200.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e92) -- 0x5631ad09dab0 con 0x5631ad098e90 2024-08-03T20:55:43.690 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::200.00000000:head# 2024-08-03T20:55:43.690 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:43.689+0000 7fa8eec00640 1 -- 172.21.15.32:0/30469862 <== osd.5 v2:172.21.15.118:6810/2724891770 8 ==== osd_op_reply(39 200.00000000 [create] v92'159 uv159 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fa8e000f780 con 0x5631ad098e90 2024-08-03T20:55:43.690 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:43.689+0000 7fa8f5a30f40 1 -- 172.21.15.32:0/30469862 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:40 12.14 12:292cf221:::200.00000000:head [write 0~90 in=90b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e92) -- 0x5631acfedf10 con 0x5631ad098e90 2024-08-03T20:55:43.692 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:43.691+0000 7fa8eec00640 1 -- 172.21.15.32:0/30469862 <== osd.5 v2:172.21.15.118:6810/2724891770 9 ==== osd_op_reply(40 200.00000000 [write 0~90] v92'160 uv160 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fa8e000f780 con 0x5631ad098e90 2024-08-03T20:55:43.692 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:43.691+0000 7fa8f5a30f40 1 -- 172.21.15.32:0/30469862 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:41 12.14 12:292cf221:::200.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e92) -- 0x5631acfedf10 con 0x5631ad098e90 2024-08-03T20:55:43.693 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:43.693+0000 7fa8eec00640 1 -- 172.21.15.32:0/30469862 <== osd.5 v2:172.21.15.118:6810/2724891770 10 ==== osd_op_reply(41 200.00000000 [omap-set-header] v92'161 uv161 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fa8e000f780 con 0x5631ad098e90 2024-08-03T20:55:43.693 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:43.693+0000 7fa8f5a30f40 1 -- 172.21.15.32:0/30469862 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:42 12.14 12:2e2fa760:::200.00000001:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e92) -- 0x5631acfedf10 con 0x5631ad098e90 2024-08-03T20:55:43.694 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:43.694+0000 7fa8eec00640 1 -- 172.21.15.32:0/30469862 <== osd.5 v2:172.21.15.118:6810/2724891770 11 ==== osd_op_reply(42 200.00000001 [create] v92'162 uv151 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fa8e000f780 con 0x5631ad098e90 2024-08-03T20:55:43.695 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:43.694+0000 7fa8f5a30f40 1 -- 172.21.15.32:0/30469862 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:43 12.14 12:2e2fa760:::200.00000001:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e92) -- 0x5631acfedf10 con 0x5631ad098e90 2024-08-03T20:55:43.696 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:43.696+0000 7fa8eec00640 1 -- 172.21.15.32:0/30469862 <== osd.5 v2:172.21.15.118:6810/2724891770 12 ==== osd_op_reply(43 200.00000001 [delete] v92'163 uv163 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fa8e000f780 con 0x5631ad098e90 2024-08-03T20:55:43.696 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:43.696+0000 7fa8f5a30f40 1 -- 172.21.15.32:0/30469862 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:44 12.14 12:2e2fa760:::200.00000001:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e92) -- 0x5631acfedf10 con 0x5631ad098e90 2024-08-03T20:55:43.698 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:43.697+0000 7fa8eec00640 1 -- 172.21.15.32:0/30469862 <== osd.5 v2:172.21.15.118:6810/2724891770 13 ==== osd_op_reply(44 200.00000001 [create] v92'164 uv164 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fa8e000f780 con 0x5631ad098e90 2024-08-03T20:55:43.698 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::200.00000001:head# 2024-08-03T20:55:43.699 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:43.698+0000 7fa8f5a30f40 1 -- 172.21.15.32:0/30469862 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:45 12.14 12:2e2fa760:::200.00000001:head [write 0~18118 in=18118b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e92) -- 0x5631ad0a26f0 con 0x5631ad098e90 2024-08-03T20:55:43.700 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:43.699+0000 7fa8eec00640 1 -- 172.21.15.32:0/30469862 <== osd.5 v2:172.21.15.118:6810/2724891770 14 ==== osd_op_reply(45 200.00000001 [write 0~18118] v92'165 uv165 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fa8e000f780 con 0x5631ad098e90 2024-08-03T20:55:43.700 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:43.700+0000 7fa8f5a30f40 1 -- 172.21.15.32:0/30469862 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:46 12.14 12:2e2fa760:::200.00000001:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e92) -- 0x5631acfedf30 con 0x5631ad098e90 2024-08-03T20:55:43.701 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:43.701+0000 7fa8eec00640 1 -- 172.21.15.32:0/30469862 <== osd.5 v2:172.21.15.118:6810/2724891770 15 ==== osd_op_reply(46 200.00000001 [omap-set-header] v92'166 uv166 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fa8e000f780 con 0x5631ad098e90 2024-08-03T20:55:43.701 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:43.701+0000 7fa8f5a30f40 1 -- 172.21.15.32:0/30469862 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:47 12.14 12:2f55791f:::606.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e92) -- 0x5631ad09e000 con 0x5631ad098e90 2024-08-03T20:55:43.703 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:43.703+0000 7fa8eec00640 1 -- 172.21.15.32:0/30469862 <== osd.5 v2:172.21.15.118:6810/2724891770 16 ==== osd_op_reply(47 606.00000000 [create] v92'167 uv147 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fa8e000f780 con 0x5631ad098e90 2024-08-03T20:55:43.703 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:43.703+0000 7fa8f5a30f40 1 -- 172.21.15.32:0/30469862 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:48 12.14 12:2f55791f:::606.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e92) -- 0x5631ad09dfb0 con 0x5631ad098e90 2024-08-03T20:55:43.705 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:43.704+0000 7fa8eec00640 1 -- 172.21.15.32:0/30469862 <== osd.5 v2:172.21.15.118:6810/2724891770 17 ==== osd_op_reply(48 606.00000000 [delete] v92'168 uv168 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fa8e000f780 con 0x5631ad098e90 2024-08-03T20:55:43.705 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:43.704+0000 7fa8f5a30f40 1 -- 172.21.15.32:0/30469862 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:49 12.14 12:2f55791f:::606.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e92) -- 0x5631ad09dfb0 con 0x5631ad098e90 2024-08-03T20:55:43.706 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:43.706+0000 7fa8eec00640 1 -- 172.21.15.32:0/30469862 <== osd.5 v2:172.21.15.118:6810/2724891770 18 ==== osd_op_reply(49 606.00000000 [create] v92'169 uv169 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fa8e000f780 con 0x5631ad098e90 2024-08-03T20:55:43.707 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::606.00000000:head# 2024-08-03T20:55:43.707 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:43.706+0000 7fa8f5a30f40 1 -- 172.21.15.32:0/30469862 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:50 12.14 12:2f55791f:::606.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e92) -- 0x5631ad09f3b0 con 0x5631ad098e90 2024-08-03T20:55:43.708 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:43.708+0000 7fa8eec00640 1 -- 172.21.15.32:0/30469862 <== osd.5 v2:172.21.15.118:6810/2724891770 19 ==== osd_op_reply(50 606.00000000 [setxattr (30)] v92'170 uv170 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fa8e000f780 con 0x5631ad098e90 2024-08-03T20:55:43.708 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:43.708+0000 7fa8f5a30f40 1 -- 172.21.15.32:0/30469862 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:51 12.14 12:2f55791f:::606.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e92) -- 0x5631ad09f0c0 con 0x5631ad098e90 2024-08-03T20:55:43.709 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:43.709+0000 7fa8eec00640 1 -- 172.21.15.32:0/30469862 <== osd.5 v2:172.21.15.118:6810/2724891770 20 ==== osd_op_reply(51 606.00000000 [setxattr (62)] v92'171 uv171 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fa8e000f780 con 0x5631ad098e90 2024-08-03T20:55:43.710 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:43.709+0000 7fa8f5a30f40 1 -- 172.21.15.32:0/30469862 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:52 12.14 12:2f55791f:::606.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e92) -- 0x5631ad09e070 con 0x5631ad098e90 2024-08-03T20:55:43.712 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:43.711+0000 7fa8eec00640 1 -- 172.21.15.32:0/30469862 <== osd.5 v2:172.21.15.118:6810/2724891770 21 ==== osd_op_reply(52 606.00000000 [omap-set-header] v92'172 uv172 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fa8e000f780 con 0x5631ad098e90 2024-08-03T20:55:43.712 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:43.711+0000 7fa8f5a30f40 1 -- 172.21.15.32:0/30469862 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:53 12.1c 12:3ed09add:::607.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e92) -- 0x5631ad09e020 con 0x5631acfd43e0 2024-08-03T20:55:43.713 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:43.712+0000 7fa8ef600640 1 -- 172.21.15.32:0/30469862 <== osd.0 v2:172.21.15.32:6816/3066156116 32 ==== osd_op_reply(53 607.00000000 [create] v92'39 uv38 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fa8e800b040 con 0x5631acfd43e0 2024-08-03T20:55:43.713 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:43.713+0000 7fa8f5a30f40 1 -- 172.21.15.32:0/30469862 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:54 12.1c 12:3ed09add:::607.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e92) -- 0x5631ad09e020 con 0x5631acfd43e0 2024-08-03T20:55:43.714 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:43.714+0000 7fa8ef600640 1 -- 172.21.15.32:0/30469862 <== osd.0 v2:172.21.15.32:6816/3066156116 33 ==== osd_op_reply(54 607.00000000 [delete] v92'40 uv40 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fa8e800b040 con 0x5631acfd43e0 2024-08-03T20:55:43.714 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:43.714+0000 7fa8f5a30f40 1 -- 172.21.15.32:0/30469862 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:55 12.1c 12:3ed09add:::607.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e92) -- 0x5631ad09e020 con 0x5631acfd43e0 2024-08-03T20:55:43.716 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::607.00000000:head# 2024-08-03T20:55:43.716 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:43.715+0000 7fa8ef600640 1 -- 172.21.15.32:0/30469862 <== osd.0 v2:172.21.15.32:6816/3066156116 34 ==== osd_op_reply(55 607.00000000 [create] v92'41 uv41 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fa8e800b040 con 0x5631acfd43e0 2024-08-03T20:55:43.716 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:43.715+0000 7fa8f5a30f40 1 -- 172.21.15.32:0/30469862 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:56 12.1c 12:3ed09add:::607.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e92) -- 0x5631ad09e020 con 0x5631acfd43e0 2024-08-03T20:55:43.717 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:43.717+0000 7fa8ef600640 1 -- 172.21.15.32:0/30469862 <== osd.0 v2:172.21.15.32:6816/3066156116 35 ==== osd_op_reply(56 607.00000000 [setxattr (30)] v92'42 uv42 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fa8e800b040 con 0x5631acfd43e0 2024-08-03T20:55:43.717 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:43.717+0000 7fa8f5a30f40 1 -- 172.21.15.32:0/30469862 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:57 12.1c 12:3ed09add:::607.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e92) -- 0x5631ad09e020 con 0x5631acfd43e0 2024-08-03T20:55:43.718 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:43.718+0000 7fa8ef600640 1 -- 172.21.15.32:0/30469862 <== osd.0 v2:172.21.15.32:6816/3066156116 36 ==== osd_op_reply(57 607.00000000 [setxattr (62)] v92'43 uv43 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fa8e800b040 con 0x5631acfd43e0 2024-08-03T20:55:43.718 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:43.718+0000 7fa8f5a30f40 1 -- 172.21.15.32:0/30469862 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:58 12.1c 12:3ed09add:::607.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e92) -- 0x5631ad09e020 con 0x5631acfd43e0 2024-08-03T20:55:43.720 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:43.719+0000 7fa8ef600640 1 -- 172.21.15.32:0/30469862 <== osd.0 v2:172.21.15.32:6816/3066156116 37 ==== osd_op_reply(58 607.00000000 [omap-set-header] v92'44 uv44 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fa8e800b040 con 0x5631acfd43e0 2024-08-03T20:55:43.720 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:43.719+0000 7fa8f5a30f40 1 --2- 172.21.15.32:0/30469862 >> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] conn(0x5631ad09e970 0x5631ad09ed80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:55:43.720 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:43.719+0000 7fa8f5a30f40 1 -- 172.21.15.32:0/30469862 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:59 12.6 12:654134d2:::mds0_openfiles.0:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e92) -- 0x5631ad0a14c0 con 0x5631ad09e970 2024-08-03T20:55:43.720 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:43.720+0000 7fa8ee200640 1 --2- 172.21.15.32:0/30469862 >> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] conn(0x5631ad09e970 0x5631ad09ed80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:55:43.721 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:43.720+0000 7fa8ee200640 1 --2- 172.21.15.32:0/30469862 >> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] conn(0x5631ad09e970 0x5631ad09ed80 crc :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:55:43.724 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:43.724+0000 7fa8ee200640 1 -- 172.21.15.32:0/30469862 <== osd.1 v2:172.21.15.32:6824/1551241759 1 ==== osd_op_reply(59 mds0_openfiles.0 [create] v92'36 uv35 ondisk = -17 ((17) File exists)) ==== 160+0+0 (crc 0 0 0) 0x7fa8dc02c070 con 0x5631ad09e970 2024-08-03T20:55:43.724 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:43.724+0000 7fa8f5a30f40 1 -- 172.21.15.32:0/30469862 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:60 12.6 12:654134d2:::mds0_openfiles.0:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e92) -- 0x5631ad0a1860 con 0x5631ad09e970 2024-08-03T20:55:43.725 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:43.725+0000 7fa8ee200640 1 -- 172.21.15.32:0/30469862 <== osd.1 v2:172.21.15.32:6824/1551241759 2 ==== osd_op_reply(60 mds0_openfiles.0 [delete] v92'37 uv37 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7fa8dc02c070 con 0x5631ad09e970 2024-08-03T20:55:43.726 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:43.725+0000 7fa8f5a30f40 1 -- 172.21.15.32:0/30469862 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:61 12.6 12:654134d2:::mds0_openfiles.0:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e92) -- 0x5631ad0a1d10 con 0x5631ad09e970 2024-08-03T20:55:43.726 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds0_openfiles.0:head# 2024-08-03T20:55:43.727 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:43.726+0000 7fa8ee200640 1 -- 172.21.15.32:0/30469862 <== osd.1 v2:172.21.15.32:6824/1551241759 3 ==== osd_op_reply(61 mds0_openfiles.0 [create] v92'38 uv38 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7fa8dc02c070 con 0x5631ad09e970 2024-08-03T20:55:43.727 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:43.726+0000 7fa8f5a30f40 1 -- 172.21.15.32:0/30469862 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:62 12.6 12:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e92) -- 0x5631ad09db70 con 0x5631ad09e970 2024-08-03T20:55:43.728 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:43.727+0000 7fa8ee200640 1 -- 172.21.15.32:0/30469862 <== osd.1 v2:172.21.15.32:6824/1551241759 4 ==== osd_op_reply(62 mds0_openfiles.0 [omap-set-header] v92'39 uv39 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7fa8dc02c070 con 0x5631ad09e970 2024-08-03T20:55:43.728 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:43.728+0000 7fa8f5a30f40 1 -- 172.21.15.32:0/30469862 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:63 12.e 12:717a0223:::608.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e92) -- 0x5631ad0a2650 con 0x5631acfd43e0 2024-08-03T20:55:43.730 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:43.729+0000 7fa8ef600640 1 -- 172.21.15.32:0/30469862 <== osd.0 v2:172.21.15.32:6816/3066156116 38 ==== osd_op_reply(63 608.00000000 [create] v92'39 uv38 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fa8e800b040 con 0x5631acfd43e0 2024-08-03T20:55:43.730 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:43.729+0000 7fa8f5a30f40 1 -- 172.21.15.32:0/30469862 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:64 12.e 12:717a0223:::608.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e92) -- 0x5631ad0a2650 con 0x5631acfd43e0 2024-08-03T20:55:43.731 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:43.731+0000 7fa8ef600640 1 -- 172.21.15.32:0/30469862 <== osd.0 v2:172.21.15.32:6816/3066156116 39 ==== osd_op_reply(64 608.00000000 [delete] v92'40 uv40 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fa8e800b040 con 0x5631acfd43e0 2024-08-03T20:55:43.731 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:43.731+0000 7fa8f5a30f40 1 -- 172.21.15.32:0/30469862 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:65 12.e 12:717a0223:::608.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e92) -- 0x5631ad0a2650 con 0x5631acfd43e0 2024-08-03T20:55:43.732 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::608.00000000:head# 2024-08-03T20:55:43.733 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:43.732+0000 7fa8ef600640 1 -- 172.21.15.32:0/30469862 <== osd.0 v2:172.21.15.32:6816/3066156116 40 ==== osd_op_reply(65 608.00000000 [create] v92'41 uv41 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fa8e800b040 con 0x5631acfd43e0 2024-08-03T20:55:43.733 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:43.732+0000 7fa8f5a30f40 1 -- 172.21.15.32:0/30469862 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:66 12.e 12:717a0223:::608.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e92) -- 0x5631ad0a3660 con 0x5631acfd43e0 2024-08-03T20:55:43.734 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:43.734+0000 7fa8ef600640 1 -- 172.21.15.32:0/30469862 <== osd.0 v2:172.21.15.32:6816/3066156116 41 ==== osd_op_reply(66 608.00000000 [setxattr (30)] v92'42 uv42 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fa8e800b040 con 0x5631acfd43e0 2024-08-03T20:55:43.734 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:43.734+0000 7fa8f5a30f40 1 -- 172.21.15.32:0/30469862 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:67 12.e 12:717a0223:::608.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e92) -- 0x5631ad0a3660 con 0x5631acfd43e0 2024-08-03T20:55:43.736 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:43.735+0000 7fa8ef600640 1 -- 172.21.15.32:0/30469862 <== osd.0 v2:172.21.15.32:6816/3066156116 42 ==== osd_op_reply(67 608.00000000 [setxattr (62)] v92'43 uv43 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fa8e800b040 con 0x5631acfd43e0 2024-08-03T20:55:43.736 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:43.735+0000 7fa8f5a30f40 1 -- 172.21.15.32:0/30469862 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:68 12.e 12:717a0223:::608.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e92) -- 0x5631ad0a2650 con 0x5631acfd43e0 2024-08-03T20:55:43.737 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:43.737+0000 7fa8ef600640 1 -- 172.21.15.32:0/30469862 <== osd.0 v2:172.21.15.32:6816/3066156116 43 ==== osd_op_reply(68 608.00000000 [omap-set-header] v92'44 uv44 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fa8e800b040 con 0x5631acfd43e0 2024-08-03T20:55:43.737 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:43.737+0000 7fa8f5a30f40 1 --2- 172.21.15.32:0/30469862 >> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] conn(0x5631ad0a2cd0 0x5631ad0a50f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:55:43.737 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:43.737+0000 7fa8f5a30f40 1 -- 172.21.15.32:0/30469862 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:69 12.15 12:a93a17c2:::604.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e92) -- 0x5631ad0a5700 con 0x5631ad0a2cd0 2024-08-03T20:55:43.737 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:43.737+0000 7fa8ef600640 1 --2- 172.21.15.32:0/30469862 >> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] conn(0x5631ad0a2cd0 0x5631ad0a50f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:55:43.738 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:43.738+0000 7fa8ef600640 1 --2- 172.21.15.32:0/30469862 >> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] conn(0x5631ad0a2cd0 0x5631ad0a50f0 crc :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:55:43.740 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:43.739+0000 7fa8ef600640 1 -- 172.21.15.32:0/30469862 <== osd.6 v2:172.21.15.118:6825/801947386 1 ==== osd_op_reply(69 604.00000000 [create] v92'70 uv64 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fa8e800b040 con 0x5631ad0a2cd0 2024-08-03T20:55:43.740 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:43.739+0000 7fa8f5a30f40 1 -- 172.21.15.32:0/30469862 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:70 12.15 12:a93a17c2:::604.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e92) -- 0x5631ad0a5ad0 con 0x5631ad0a2cd0 2024-08-03T20:55:43.741 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:43.741+0000 7fa8ef600640 1 -- 172.21.15.32:0/30469862 <== osd.6 v2:172.21.15.118:6825/801947386 2 ==== osd_op_reply(70 604.00000000 [delete] v92'71 uv71 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fa8e800b040 con 0x5631ad0a2cd0 2024-08-03T20:55:43.741 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:43.741+0000 7fa8f5a30f40 1 -- 172.21.15.32:0/30469862 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:71 12.15 12:a93a17c2:::604.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e92) -- 0x5631ad0a5ad0 con 0x5631ad0a2cd0 2024-08-03T20:55:43.743 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::604.00000000:head# 2024-08-03T20:55:43.743 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:43.742+0000 7fa8ef600640 1 -- 172.21.15.32:0/30469862 <== osd.6 v2:172.21.15.118:6825/801947386 3 ==== osd_op_reply(71 604.00000000 [create] v92'72 uv72 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fa8e800b040 con 0x5631ad0a2cd0 2024-08-03T20:55:43.743 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:43.742+0000 7fa8f5a30f40 1 -- 172.21.15.32:0/30469862 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:72 12.15 12:a93a17c2:::604.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e92) -- 0x5631ad0a6ae0 con 0x5631ad0a2cd0 2024-08-03T20:55:43.744 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:43.744+0000 7fa8ef600640 1 -- 172.21.15.32:0/30469862 <== osd.6 v2:172.21.15.118:6825/801947386 4 ==== osd_op_reply(72 604.00000000 [setxattr (30)] v92'73 uv73 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fa8e800b040 con 0x5631ad0a2cd0 2024-08-03T20:55:43.744 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:43.744+0000 7fa8f5a30f40 1 -- 172.21.15.32:0/30469862 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:73 12.15 12:a93a17c2:::604.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e92) -- 0x5631ad0a6ae0 con 0x5631ad0a2cd0 2024-08-03T20:55:43.745 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:43.745+0000 7fa8ef600640 1 -- 172.21.15.32:0/30469862 <== osd.6 v2:172.21.15.118:6825/801947386 5 ==== osd_op_reply(73 604.00000000 [setxattr (62)] v92'74 uv74 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fa8e800b040 con 0x5631ad0a2cd0 2024-08-03T20:55:43.745 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:43.745+0000 7fa8f5a30f40 1 -- 172.21.15.32:0/30469862 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:74 12.15 12:a93a17c2:::604.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e92) -- 0x5631ad0a5ad0 con 0x5631ad0a2cd0 2024-08-03T20:55:43.746 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:43.746+0000 7fa8ef600640 1 -- 172.21.15.32:0/30469862 <== osd.6 v2:172.21.15.118:6825/801947386 6 ==== osd_op_reply(74 604.00000000 [omap-set-header] v92'75 uv75 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fa8e800b040 con 0x5631ad0a2cd0 2024-08-03T20:55:43.747 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:43.746+0000 7fa8f5a30f40 1 -- 172.21.15.32:0/30469862 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:75 12.15 12:aa448500:::500.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e92) -- 0x5631ad0a5ad0 con 0x5631ad0a2cd0 2024-08-03T20:55:43.748 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:43.747+0000 7fa8ef600640 1 -- 172.21.15.32:0/30469862 <== osd.6 v2:172.21.15.118:6825/801947386 7 ==== osd_op_reply(75 500.00000000 [create] v92'76 uv69 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fa8e800b040 con 0x5631ad0a2cd0 2024-08-03T20:55:43.748 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:43.747+0000 7fa8f5a30f40 1 -- 172.21.15.32:0/30469862 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:76 12.15 12:aa448500:::500.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e92) -- 0x5631ad0a5ad0 con 0x5631ad0a2cd0 2024-08-03T20:55:43.749 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:43.749+0000 7fa8ef600640 1 -- 172.21.15.32:0/30469862 <== osd.6 v2:172.21.15.118:6825/801947386 8 ==== osd_op_reply(76 500.00000000 [delete] v92'77 uv77 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fa8e800b040 con 0x5631ad0a2cd0 2024-08-03T20:55:43.749 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:43.749+0000 7fa8f5a30f40 1 -- 172.21.15.32:0/30469862 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:77 12.15 12:aa448500:::500.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e92) -- 0x5631ad0a5ad0 con 0x5631ad0a2cd0 2024-08-03T20:55:43.750 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::500.00000000:head# 2024-08-03T20:55:43.751 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:43.750+0000 7fa8ef600640 1 -- 172.21.15.32:0/30469862 <== osd.6 v2:172.21.15.118:6825/801947386 9 ==== osd_op_reply(77 500.00000000 [create] v92'78 uv78 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fa8e800b040 con 0x5631ad0a2cd0 2024-08-03T20:55:43.751 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:43.750+0000 7fa8f5a30f40 1 -- 172.21.15.32:0/30469862 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:78 12.15 12:aa448500:::500.00000000:head [write 0~90 in=90b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e92) -- 0x5631ad0a5ad0 con 0x5631ad0a2cd0 2024-08-03T20:55:43.752 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:43.752+0000 7fa8ef600640 1 -- 172.21.15.32:0/30469862 <== osd.6 v2:172.21.15.118:6825/801947386 10 ==== osd_op_reply(78 500.00000000 [write 0~90] v92'79 uv79 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fa8e800b040 con 0x5631ad0a2cd0 2024-08-03T20:55:43.752 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:43.752+0000 7fa8f5a30f40 1 -- 172.21.15.32:0/30469862 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:79 12.15 12:aa448500:::500.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e92) -- 0x5631ad0a5ad0 con 0x5631ad0a2cd0 2024-08-03T20:55:43.753 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:43.753+0000 7fa8ef600640 1 -- 172.21.15.32:0/30469862 <== osd.6 v2:172.21.15.118:6825/801947386 11 ==== osd_op_reply(79 500.00000000 [omap-set-header] v92'80 uv80 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fa8e800b040 con 0x5631ad0a2cd0 2024-08-03T20:55:43.753 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:43.753+0000 7fa8f5a30f40 1 --2- 172.21.15.32:0/30469862 >> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] conn(0x5631ad0a6420 0x5631ad0a6830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:55:43.754 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:43.753+0000 7fa8f5a30f40 1 -- 172.21.15.32:0/30469862 --> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] -- osd_op(unknown.0.0:80 12.d 12:b50e409b:::mds_snaptable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e92) -- 0x5631ad0a8eb0 con 0x5631ad0a6420 2024-08-03T20:55:43.754 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:43.753+0000 7fa8eec00640 1 --2- 172.21.15.32:0/30469862 >> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] conn(0x5631ad0a6420 0x5631ad0a6830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:55:43.754 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:43.754+0000 7fa8eec00640 1 --2- 172.21.15.32:0/30469862 >> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] conn(0x5631ad0a6420 0x5631ad0a6830 crc :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:55:43.756 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:43.756+0000 7fa8eec00640 1 -- 172.21.15.32:0/30469862 <== osd.7 v2:172.21.15.118:6818/653000048 1 ==== osd_op_reply(80 mds_snaptable [create] v92'32 uv31 ondisk = -17 ((17) File exists)) ==== 157+0+0 (crc 0 0 0) 0x7fa8e000f780 con 0x5631ad0a6420 2024-08-03T20:55:43.756 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:43.756+0000 7fa8f5a30f40 1 -- 172.21.15.32:0/30469862 --> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] -- osd_op(unknown.0.0:81 12.d 12:b50e409b:::mds_snaptable:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e92) -- 0x5631ad0a8eb0 con 0x5631ad0a6420 2024-08-03T20:55:43.758 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:43.758+0000 7fa8eec00640 1 -- 172.21.15.32:0/30469862 <== osd.7 v2:172.21.15.118:6818/653000048 2 ==== osd_op_reply(81 mds_snaptable [delete] v92'33 uv33 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7fa8e000f780 con 0x5631ad0a6420 2024-08-03T20:55:43.758 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:43.758+0000 7fa8f5a30f40 1 -- 172.21.15.32:0/30469862 --> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] -- osd_op(unknown.0.0:82 12.d 12:b50e409b:::mds_snaptable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e92) -- 0x5631ad0a92d0 con 0x5631ad0a6420 2024-08-03T20:55:43.759 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:43.759+0000 7fa8eec00640 1 -- 172.21.15.32:0/30469862 <== osd.7 v2:172.21.15.118:6818/653000048 3 ==== osd_op_reply(82 mds_snaptable [create] v92'34 uv34 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7fa8e000f780 con 0x5631ad0a6420 2024-08-03T20:55:43.760 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds_snaptable:head# 2024-08-03T20:55:43.760 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:43.759+0000 7fa8f5a30f40 1 -- 172.21.15.32:0/30469862 --> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] -- osd_op(unknown.0.0:83 12.d 12:b50e409b:::mds_snaptable:head [write 0~70 in=70b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e92) -- 0x5631ad0a92d0 con 0x5631ad0a6420 2024-08-03T20:55:43.761 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:43.761+0000 7fa8eec00640 1 -- 172.21.15.32:0/30469862 <== osd.7 v2:172.21.15.118:6818/653000048 4 ==== osd_op_reply(83 mds_snaptable [write 0~70] v92'35 uv35 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7fa8e000f780 con 0x5631ad0a6420 2024-08-03T20:55:43.761 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:43.761+0000 7fa8f5a30f40 1 -- 172.21.15.32:0/30469862 --> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] -- osd_op(unknown.0.0:84 12.d 12:b50e409b:::mds_snaptable:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e92) -- 0x5631ad0a92d0 con 0x5631ad0a6420 2024-08-03T20:55:43.762 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:43.762+0000 7fa8eec00640 1 -- 172.21.15.32:0/30469862 <== osd.7 v2:172.21.15.118:6818/653000048 5 ==== osd_op_reply(84 mds_snaptable [omap-set-header] v92'36 uv36 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7fa8e000f780 con 0x5631ad0a6420 2024-08-03T20:55:43.763 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:43.762+0000 7fa8f5a30f40 1 -- 172.21.15.32:0/30469862 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:85 12.1d 12:b871830b:::605.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e92) -- 0x5631ad0a92d0 con 0x5631acfd43e0 2024-08-03T20:55:43.764 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:43.764+0000 7fa8ef600640 1 -- 172.21.15.32:0/30469862 <== osd.0 v2:172.21.15.32:6816/3066156116 44 ==== osd_op_reply(85 605.00000000 [create] v92'39 uv38 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fa8e800b040 con 0x5631acfd43e0 2024-08-03T20:55:43.764 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:43.764+0000 7fa8f5a30f40 1 -- 172.21.15.32:0/30469862 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:86 12.1d 12:b871830b:::605.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e92) -- 0x5631ad0a92d0 con 0x5631acfd43e0 2024-08-03T20:55:43.766 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:43.765+0000 7fa8ef600640 1 -- 172.21.15.32:0/30469862 <== osd.0 v2:172.21.15.32:6816/3066156116 45 ==== osd_op_reply(86 605.00000000 [delete] v92'40 uv40 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fa8e800b040 con 0x5631acfd43e0 2024-08-03T20:55:43.766 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:43.765+0000 7fa8f5a30f40 1 -- 172.21.15.32:0/30469862 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:87 12.1d 12:b871830b:::605.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e92) -- 0x5631ad0a92d0 con 0x5631acfd43e0 2024-08-03T20:55:43.767 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::605.00000000:head# 2024-08-03T20:55:43.767 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:43.766+0000 7fa8ef600640 1 -- 172.21.15.32:0/30469862 <== osd.0 v2:172.21.15.32:6816/3066156116 46 ==== osd_op_reply(87 605.00000000 [create] v92'41 uv41 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fa8e800b040 con 0x5631acfd43e0 2024-08-03T20:55:43.767 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:43.767+0000 7fa8f5a30f40 1 -- 172.21.15.32:0/30469862 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:88 12.1d 12:b871830b:::605.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e92) -- 0x5631ad0aa2e0 con 0x5631acfd43e0 2024-08-03T20:55:43.768 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:43.768+0000 7fa8ef600640 1 -- 172.21.15.32:0/30469862 <== osd.0 v2:172.21.15.32:6816/3066156116 47 ==== osd_op_reply(88 605.00000000 [setxattr (30)] v92'42 uv42 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fa8e800b040 con 0x5631acfd43e0 2024-08-03T20:55:43.768 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:43.768+0000 7fa8f5a30f40 1 -- 172.21.15.32:0/30469862 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:89 12.1d 12:b871830b:::605.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e92) -- 0x5631ad0aa2e0 con 0x5631acfd43e0 2024-08-03T20:55:43.770 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:43.769+0000 7fa8ef600640 1 -- 172.21.15.32:0/30469862 <== osd.0 v2:172.21.15.32:6816/3066156116 48 ==== osd_op_reply(89 605.00000000 [setxattr (62)] v92'43 uv43 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fa8e800b040 con 0x5631acfd43e0 2024-08-03T20:55:43.770 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:43.769+0000 7fa8f5a30f40 1 -- 172.21.15.32:0/30469862 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:90 12.1d 12:b871830b:::605.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e92) -- 0x5631ad0a92d0 con 0x5631acfd43e0 2024-08-03T20:55:43.771 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:43.771+0000 7fa8ef600640 1 -- 172.21.15.32:0/30469862 <== osd.0 v2:172.21.15.32:6816/3066156116 49 ==== osd_op_reply(90 605.00000000 [omap-set-header] v92'44 uv44 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fa8e800b040 con 0x5631acfd43e0 2024-08-03T20:55:43.771 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:43.771+0000 7fa8f5a30f40 1 -- 172.21.15.32:0/30469862 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:91 12.13 12:c91d4a1d:::mds0_inotable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e92) -- 0x5631ad0a92d0 con 0x5631ad09e970 2024-08-03T20:55:43.772 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:43.772+0000 7fa8ee200640 1 -- 172.21.15.32:0/30469862 <== osd.1 v2:172.21.15.32:6824/1551241759 5 ==== osd_op_reply(91 mds0_inotable [create] v92'64 uv58 ondisk = -17 ((17) File exists)) ==== 157+0+0 (crc 0 0 0) 0x7fa8dc02c070 con 0x5631ad09e970 2024-08-03T20:55:43.772 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:43.772+0000 7fa8f5a30f40 1 -- 172.21.15.32:0/30469862 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:92 12.13 12:c91d4a1d:::mds0_inotable:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e92) -- 0x5631ad0a9670 con 0x5631ad09e970 2024-08-03T20:55:43.773 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:43.773+0000 7fa8ee200640 1 -- 172.21.15.32:0/30469862 <== osd.1 v2:172.21.15.32:6824/1551241759 6 ==== osd_op_reply(92 mds0_inotable [delete] v92'65 uv65 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7fa8dc02c070 con 0x5631ad09e970 2024-08-03T20:55:43.774 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:43.773+0000 7fa8f5a30f40 1 -- 172.21.15.32:0/30469862 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:93 12.13 12:c91d4a1d:::mds0_inotable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e92) -- 0x5631ad0a9a10 con 0x5631ad09e970 2024-08-03T20:55:43.775 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:43.775+0000 7fa8ee200640 1 -- 172.21.15.32:0/30469862 <== osd.1 v2:172.21.15.32:6824/1551241759 7 ==== osd_op_reply(93 mds0_inotable [create] v92'66 uv66 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7fa8dc02c070 con 0x5631ad09e970 2024-08-03T20:55:43.775 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds0_inotable:head# 2024-08-03T20:55:43.776 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:43.775+0000 7fa8f5a30f40 1 -- 172.21.15.32:0/30469862 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:94 12.13 12:c91d4a1d:::mds0_inotable:head [write 0~34 in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e92) -- 0x5631ad0a9db0 con 0x5631ad09e970 2024-08-03T20:55:43.776 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:43.776+0000 7fa8ee200640 1 -- 172.21.15.32:0/30469862 <== osd.1 v2:172.21.15.32:6824/1551241759 8 ==== osd_op_reply(94 mds0_inotable [write 0~34] v92'67 uv67 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7fa8dc02c070 con 0x5631ad09e970 2024-08-03T20:55:43.776 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:43.776+0000 7fa8f5a30f40 1 -- 172.21.15.32:0/30469862 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:95 12.13 12:c91d4a1d:::mds0_inotable:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e92) -- 0x5631ad0a9db0 con 0x5631ad09e970 2024-08-03T20:55:43.778 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:43.777+0000 7fa8ee200640 1 -- 172.21.15.32:0/30469862 <== osd.1 v2:172.21.15.32:6824/1551241759 9 ==== osd_op_reply(95 mds0_inotable [omap-set-header] v92'68 uv68 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7fa8dc02c070 con 0x5631ad09e970 2024-08-03T20:55:43.778 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:43.777+0000 7fa8f5a30f40 1 -- 172.21.15.32:0/30469862 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:96 12.13 12:cd5a64a3:::100.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e92) -- 0x5631ad0a9db0 con 0x5631ad09e970 2024-08-03T20:55:43.779 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:43.778+0000 7fa8ee200640 1 -- 172.21.15.32:0/30469862 <== osd.1 v2:172.21.15.32:6824/1551241759 10 ==== osd_op_reply(96 100.00000000 [create] v92'69 uv63 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fa8dc02c070 con 0x5631ad09e970 2024-08-03T20:55:43.779 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:43.778+0000 7fa8f5a30f40 1 -- 172.21.15.32:0/30469862 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:97 12.13 12:cd5a64a3:::100.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e92) -- 0x5631ad0a9db0 con 0x5631ad09e970 2024-08-03T20:55:43.780 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:43.780+0000 7fa8ee200640 1 -- 172.21.15.32:0/30469862 <== osd.1 v2:172.21.15.32:6824/1551241759 11 ==== osd_op_reply(97 100.00000000 [delete] v92'70 uv70 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fa8dc02c070 con 0x5631ad09e970 2024-08-03T20:55:43.780 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:43.780+0000 7fa8f5a30f40 1 -- 172.21.15.32:0/30469862 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:98 12.13 12:cd5a64a3:::100.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e92) -- 0x5631ad0a9db0 con 0x5631ad09e970 2024-08-03T20:55:43.781 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::100.00000000:head# 2024-08-03T20:55:43.781 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:43.781+0000 7fa8ee200640 1 -- 172.21.15.32:0/30469862 <== osd.1 v2:172.21.15.32:6824/1551241759 12 ==== osd_op_reply(98 100.00000000 [create] v92'71 uv71 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fa8dc02c070 con 0x5631ad09e970 2024-08-03T20:55:43.781 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:43.781+0000 7fa8f5a30f40 1 -- 172.21.15.32:0/30469862 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:99 12.13 12:cd5a64a3:::100.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e92) -- 0x5631ad0a9db0 con 0x5631ad09e970 2024-08-03T20:55:43.783 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:43.782+0000 7fa8ee200640 1 -- 172.21.15.32:0/30469862 <== osd.1 v2:172.21.15.32:6824/1551241759 13 ==== osd_op_reply(99 100.00000000 [omap-set-header] v92'72 uv72 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fa8dc02c070 con 0x5631ad09e970 2024-08-03T20:55:43.783 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:43.782+0000 7fa8f5a30f40 1 -- 172.21.15.32:0/30469862 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:100 12.13 12:cd5a64a3:::100.00000000:head [omap-set-vals in=5194b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e92) -- 0x5631ad0a2210 con 0x5631ad09e970 2024-08-03T20:55:43.784 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:43.784+0000 7fa8ee200640 1 -- 172.21.15.32:0/30469862 <== osd.1 v2:172.21.15.32:6824/1551241759 14 ==== osd_op_reply(100 100.00000000 [omap-set-vals] v92'73 uv73 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fa8dc02c070 con 0x5631ad09e970 2024-08-03T20:55:43.784 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:43.784+0000 7fa8f5a30f40 1 --2- 172.21.15.32:0/30469862 >> [v2:172.21.15.32:6800/9088303,v1:172.21.15.32:6801/9088303] conn(0x5631ad0aabb0 0x5631ad0ad300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:55:43.784 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:43.784+0000 7fa8f5a30f40 1 -- 172.21.15.32:0/30469862 --> [v2:172.21.15.32:6800/9088303,v1:172.21.15.32:6801/9088303] -- osd_op(unknown.0.0:101 12.b 12:d0630e4c:::mds0_sessionmap:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e92) -- 0x5631ad0ad840 con 0x5631ad0aabb0 2024-08-03T20:55:43.784 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:43.784+0000 7fa8ee200640 1 --2- 172.21.15.32:0/30469862 >> [v2:172.21.15.32:6800/9088303,v1:172.21.15.32:6801/9088303] conn(0x5631ad0aabb0 0x5631ad0ad300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:55:43.785 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:43.784+0000 7fa8ee200640 1 --2- 172.21.15.32:0/30469862 >> [v2:172.21.15.32:6800/9088303,v1:172.21.15.32:6801/9088303] conn(0x5631ad0aabb0 0x5631ad0ad300 crc :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:55:43.786 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:43.786+0000 7fa8ee200640 1 -- 172.21.15.32:0/30469862 <== osd.3 v2:172.21.15.32:6800/9088303 1 ==== osd_op_reply(101 mds0_sessionmap [create] v92'31 uv29 ondisk = -17 ((17) File exists)) ==== 159+0+0 (crc 0 0 0) 0x7fa8dc02c070 con 0x5631ad0aabb0 2024-08-03T20:55:43.786 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:43.786+0000 7fa8f5a30f40 1 -- 172.21.15.32:0/30469862 --> [v2:172.21.15.32:6800/9088303,v1:172.21.15.32:6801/9088303] -- osd_op(unknown.0.0:102 12.b 12:d0630e4c:::mds0_sessionmap:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e92) -- 0x5631ad0ad840 con 0x5631ad0aabb0 2024-08-03T20:55:43.788 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:43.787+0000 7fa8ee200640 1 -- 172.21.15.32:0/30469862 <== osd.3 v2:172.21.15.32:6800/9088303 2 ==== osd_op_reply(102 mds0_sessionmap [delete] v92'32 uv32 ondisk = 0) ==== 159+0+0 (crc 0 0 0) 0x7fa8dc02c070 con 0x5631ad0aabb0 2024-08-03T20:55:43.788 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:43.788+0000 7fa8f5a30f40 1 -- 172.21.15.32:0/30469862 --> [v2:172.21.15.32:6800/9088303,v1:172.21.15.32:6801/9088303] -- osd_op(unknown.0.0:103 12.b 12:d0630e4c:::mds0_sessionmap:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e92) -- 0x5631ad0ad840 con 0x5631ad0aabb0 2024-08-03T20:55:43.789 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds0_sessionmap:head# 2024-08-03T20:55:43.789 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:43.789+0000 7fa8ee200640 1 -- 172.21.15.32:0/30469862 <== osd.3 v2:172.21.15.32:6800/9088303 3 ==== osd_op_reply(103 mds0_sessionmap [create] v92'33 uv33 ondisk = 0) ==== 159+0+0 (crc 0 0 0) 0x7fa8dc02c070 con 0x5631ad0aabb0 2024-08-03T20:55:43.790 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:43.789+0000 7fa8f5a30f40 1 -- 172.21.15.32:0/30469862 --> [v2:172.21.15.32:6800/9088303,v1:172.21.15.32:6801/9088303] -- osd_op(unknown.0.0:104 12.b 12:d0630e4c:::mds0_sessionmap:head [omap-set-header in=14b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e92) -- 0x5631ad0ad840 con 0x5631ad0aabb0 2024-08-03T20:55:43.790 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:43.790+0000 7fa8ee200640 1 -- 172.21.15.32:0/30469862 <== osd.3 v2:172.21.15.32:6800/9088303 4 ==== osd_op_reply(104 mds0_sessionmap [omap-set-header] v92'34 uv34 ondisk = 0) ==== 159+0+0 (crc 0 0 0) 0x7fa8dc02c070 con 0x5631ad0aabb0 2024-08-03T20:55:43.791 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:43.790+0000 7fa8f5a30f40 1 -- 172.21.15.32:0/30469862 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:105 12.7 12:e0b41d06:::609.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e92) -- 0x5631ad0ad840 con 0x5631ad09e970 2024-08-03T20:55:43.792 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:43.792+0000 7fa8ee200640 1 -- 172.21.15.32:0/30469862 <== osd.1 v2:172.21.15.32:6824/1551241759 15 ==== osd_op_reply(105 609.00000000 [create] v92'39 uv38 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fa8dc02c070 con 0x5631ad09e970 2024-08-03T20:55:43.792 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:43.792+0000 7fa8f5a30f40 1 -- 172.21.15.32:0/30469862 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:106 12.7 12:e0b41d06:::609.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e92) -- 0x5631ad0ad840 con 0x5631ad09e970 2024-08-03T20:55:43.793 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:43.793+0000 7fa8ee200640 1 -- 172.21.15.32:0/30469862 <== osd.1 v2:172.21.15.32:6824/1551241759 16 ==== osd_op_reply(106 609.00000000 [delete] v92'40 uv40 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fa8dc02c070 con 0x5631ad09e970 2024-08-03T20:55:43.794 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:43.793+0000 7fa8f5a30f40 1 -- 172.21.15.32:0/30469862 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:107 12.7 12:e0b41d06:::609.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e92) -- 0x5631ad0ad840 con 0x5631ad09e970 2024-08-03T20:55:43.795 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::609.00000000:head# 2024-08-03T20:55:43.795 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:43.794+0000 7fa8ee200640 1 -- 172.21.15.32:0/30469862 <== osd.1 v2:172.21.15.32:6824/1551241759 17 ==== osd_op_reply(107 609.00000000 [create] v92'41 uv41 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fa8dc02c070 con 0x5631ad09e970 2024-08-03T20:55:43.795 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:43.795+0000 7fa8f5a30f40 1 -- 172.21.15.32:0/30469862 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:108 12.7 12:e0b41d06:::609.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e92) -- 0x5631ad0ae850 con 0x5631ad09e970 2024-08-03T20:55:43.796 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:43.796+0000 7fa8ee200640 1 -- 172.21.15.32:0/30469862 <== osd.1 v2:172.21.15.32:6824/1551241759 18 ==== osd_op_reply(108 609.00000000 [setxattr (30)] v92'42 uv42 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fa8dc02c070 con 0x5631ad09e970 2024-08-03T20:55:43.796 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:43.796+0000 7fa8f5a30f40 1 -- 172.21.15.32:0/30469862 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:109 12.7 12:e0b41d06:::609.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e92) -- 0x5631ad0ae850 con 0x5631ad09e970 2024-08-03T20:55:43.797 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:43.797+0000 7fa8ee200640 1 -- 172.21.15.32:0/30469862 <== osd.1 v2:172.21.15.32:6824/1551241759 19 ==== osd_op_reply(109 609.00000000 [setxattr (62)] v92'43 uv43 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fa8dc02c070 con 0x5631ad09e970 2024-08-03T20:55:43.798 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:43.797+0000 7fa8f5a30f40 1 -- 172.21.15.32:0/30469862 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:110 12.7 12:e0b41d06:::609.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e92) -- 0x5631ad0ad840 con 0x5631ad09e970 2024-08-03T20:55:43.799 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:43.799+0000 7fa8ee200640 1 -- 172.21.15.32:0/30469862 <== osd.1 v2:172.21.15.32:6824/1551241759 20 ==== osd_op_reply(110 609.00000000 [omap-set-header] v92'44 uv44 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fa8dc02c070 con 0x5631ad09e970 2024-08-03T20:55:43.799 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:43.799+0000 7fa8f5a30f40 1 -- 172.21.15.32:0/30469862 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:111 12.f 12:f1f69726:::400.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e92) -- 0x5631ad0ad840 con 0x5631ad0a2cd0 2024-08-03T20:55:43.801 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:43.800+0000 7fa8ef600640 1 -- 172.21.15.32:0/30469862 <== osd.6 v2:172.21.15.118:6825/801947386 12 ==== osd_op_reply(111 400.00000000 [create] v92'32 uv31 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fa8e800b040 con 0x5631ad0a2cd0 2024-08-03T20:55:43.801 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:43.800+0000 7fa8f5a30f40 1 -- 172.21.15.32:0/30469862 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:112 12.f 12:f1f69726:::400.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e92) -- 0x5631ad0ad840 con 0x5631ad0a2cd0 2024-08-03T20:55:43.802 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:43.802+0000 7fa8ef600640 1 -- 172.21.15.32:0/30469862 <== osd.6 v2:172.21.15.118:6825/801947386 13 ==== osd_op_reply(112 400.00000000 [delete] v92'33 uv33 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fa8e800b040 con 0x5631ad0a2cd0 2024-08-03T20:55:43.802 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:43.802+0000 7fa8f5a30f40 1 -- 172.21.15.32:0/30469862 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:113 12.f 12:f1f69726:::400.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e92) -- 0x5631ad0ad840 con 0x5631ad0a2cd0 2024-08-03T20:55:43.804 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::400.00000000:head# 2024-08-03T20:55:43.804 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:43.803+0000 7fa8ef600640 1 -- 172.21.15.32:0/30469862 <== osd.6 v2:172.21.15.118:6825/801947386 14 ==== osd_op_reply(113 400.00000000 [create] v92'34 uv34 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fa8e800b040 con 0x5631ad0a2cd0 2024-08-03T20:55:43.804 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:43.804+0000 7fa8f5a30f40 1 -- 172.21.15.32:0/30469862 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:114 12.f 12:f1f69726:::400.00000000:head [write 0~22 in=22b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e92) -- 0x5631ad0ad840 con 0x5631ad0a2cd0 2024-08-03T20:55:43.806 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:43.805+0000 7fa8ef600640 1 -- 172.21.15.32:0/30469862 <== osd.6 v2:172.21.15.118:6825/801947386 15 ==== osd_op_reply(114 400.00000000 [write 0~22] v92'35 uv35 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fa8e800b040 con 0x5631ad0a2cd0 2024-08-03T20:55:43.806 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:43.805+0000 7fa8f5a30f40 1 -- 172.21.15.32:0/30469862 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:115 12.f 12:f1f69726:::400.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e92) -- 0x5631ad0ad840 con 0x5631ad0a2cd0 2024-08-03T20:55:43.807 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:43.807+0000 7fa8ef600640 1 -- 172.21.15.32:0/30469862 <== osd.6 v2:172.21.15.118:6825/801947386 16 ==== osd_op_reply(115 400.00000000 [omap-set-header] v92'36 uv36 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fa8e800b040 con 0x5631ad0a2cd0 2024-08-03T20:55:43.807 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:43.807+0000 7fa8f5a30f40 1 -- 172.21.15.32:0/30469862 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:116 12.1f 12:fe07bba1:::100.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e92) -- 0x5631ad0ad840 con 0x5631ad0a2cd0 2024-08-03T20:55:43.809 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:43.809+0000 7fa8ef600640 1 -- 172.21.15.32:0/30469862 <== osd.6 v2:172.21.15.118:6825/801947386 17 ==== osd_op_reply(116 100.00000000.inode [create] v92'82 uv73 ondisk = -17 ((17) File exists)) ==== 162+0+0 (crc 0 0 0) 0x7fa8e800b040 con 0x5631ad0a2cd0 2024-08-03T20:55:43.809 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:43.809+0000 7fa8f5a30f40 1 -- 172.21.15.32:0/30469862 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:117 12.1f 12:fe07bba1:::100.00000000.inode:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e92) -- 0x5631ad0ad840 con 0x5631ad0a2cd0 2024-08-03T20:55:43.810 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:43.810+0000 7fa8ef600640 1 -- 172.21.15.32:0/30469862 <== osd.6 v2:172.21.15.118:6825/801947386 18 ==== osd_op_reply(117 100.00000000.inode [delete] v92'83 uv83 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7fa8e800b040 con 0x5631ad0a2cd0 2024-08-03T20:55:43.810 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:43.810+0000 7fa8f5a30f40 1 -- 172.21.15.32:0/30469862 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:118 12.1f 12:fe07bba1:::100.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e92) -- 0x5631ad0ad840 con 0x5631ad0a2cd0 2024-08-03T20:55:43.812 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::100.00000000.inode:head# 2024-08-03T20:55:43.812 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:43.811+0000 7fa8ef600640 1 -- 172.21.15.32:0/30469862 <== osd.6 v2:172.21.15.118:6825/801947386 19 ==== osd_op_reply(118 100.00000000.inode [create] v92'84 uv84 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7fa8e800b040 con 0x5631ad0a2cd0 2024-08-03T20:55:43.812 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:43.811+0000 7fa8f5a30f40 1 -- 172.21.15.32:0/30469862 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:119 12.1f 12:fe07bba1:::100.00000000.inode:head [write 0~582 in=582b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e92) -- 0x5631ad0ad840 con 0x5631ad0a2cd0 2024-08-03T20:55:43.814 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:43.814+0000 7fa8ef600640 1 -- 172.21.15.32:0/30469862 <== osd.6 v2:172.21.15.118:6825/801947386 20 ==== osd_op_reply(119 100.00000000.inode [write 0~582] v92'85 uv85 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7fa8e800b040 con 0x5631ad0a2cd0 2024-08-03T20:55:43.814 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:43.814+0000 7fa8f5a30f40 1 -- 172.21.15.32:0/30469862 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:120 12.1f 12:fe07bba1:::100.00000000.inode:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e92) -- 0x5631ad0ad840 con 0x5631ad0a2cd0 2024-08-03T20:55:43.816 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:43.815+0000 7fa8ef600640 1 -- 172.21.15.32:0/30469862 <== osd.6 v2:172.21.15.118:6825/801947386 21 ==== osd_op_reply(120 100.00000000.inode [omap-set-header] v92'86 uv86 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7fa8e800b040 con 0x5631ad0a2cd0 2024-08-03T20:55:43.816 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:43.816+0000 7fa8f5a30f40 1 -- 172.21.15.32:0/30469862 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:121 12.1f 12:ff5b34d6:::1.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e92) -- 0x5631ad0ad840 con 0x5631ad0a2cd0 2024-08-03T20:55:43.817 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:43.817+0000 7fa8ef600640 1 -- 172.21.15.32:0/30469862 <== osd.6 v2:172.21.15.118:6825/801947386 22 ==== osd_op_reply(121 1.00000000 [create] v92'87 uv80 ondisk = -17 ((17) File exists)) ==== 154+0+0 (crc 0 0 0) 0x7fa8e800b040 con 0x5631ad0a2cd0 2024-08-03T20:55:43.817 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:43.817+0000 7fa8f5a30f40 1 -- 172.21.15.32:0/30469862 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:122 12.1f 12:ff5b34d6:::1.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e92) -- 0x5631ad0ad840 con 0x5631ad0a2cd0 2024-08-03T20:55:43.819 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:43.818+0000 7fa8ef600640 1 -- 172.21.15.32:0/30469862 <== osd.6 v2:172.21.15.118:6825/801947386 23 ==== osd_op_reply(122 1.00000000 [delete] v92'88 uv88 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7fa8e800b040 con 0x5631ad0a2cd0 2024-08-03T20:55:43.819 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:43.818+0000 7fa8f5a30f40 1 -- 172.21.15.32:0/30469862 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:123 12.1f 12:ff5b34d6:::1.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e92) -- 0x5631ad0ad840 con 0x5631ad0a2cd0 2024-08-03T20:55:43.820 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::1.00000000:head# 2024-08-03T20:55:43.821 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:43.820+0000 7fa8ef600640 1 -- 172.21.15.32:0/30469862 <== osd.6 v2:172.21.15.118:6825/801947386 24 ==== osd_op_reply(123 1.00000000 [create] v92'89 uv89 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7fa8e800b040 con 0x5631ad0a2cd0 2024-08-03T20:55:43.821 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:43.820+0000 7fa8f5a30f40 1 -- 172.21.15.32:0/30469862 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:124 12.1f 12:ff5b34d6:::1.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e92) -- 0x5631ad0ae850 con 0x5631ad0a2cd0 2024-08-03T20:55:43.822 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:43.821+0000 7fa8ef600640 1 -- 172.21.15.32:0/30469862 <== osd.6 v2:172.21.15.118:6825/801947386 25 ==== osd_op_reply(124 1.00000000 [setxattr (30)] v92'90 uv90 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7fa8e800b040 con 0x5631ad0a2cd0 2024-08-03T20:55:43.822 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:43.821+0000 7fa8f5a30f40 1 -- 172.21.15.32:0/30469862 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:125 12.1f 12:ff5b34d6:::1.00000000:head [setxattr parent (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e92) -- 0x5631ad0ae850 con 0x5631ad0a2cd0 2024-08-03T20:55:43.823 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:43.823+0000 7fa8ef600640 1 -- 172.21.15.32:0/30469862 <== osd.6 v2:172.21.15.118:6825/801947386 26 ==== osd_op_reply(125 1.00000000 [setxattr (30)] v92'91 uv91 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7fa8e800b040 con 0x5631ad0a2cd0 2024-08-03T20:55:43.823 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:43.823+0000 7fa8f5a30f40 1 -- 172.21.15.32:0/30469862 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:126 12.1f 12:ff5b34d6:::1.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e92) -- 0x5631ad0ad840 con 0x5631ad0a2cd0 2024-08-03T20:55:43.825 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:43.824+0000 7fa8ef600640 1 -- 172.21.15.32:0/30469862 <== osd.6 v2:172.21.15.118:6825/801947386 27 ==== osd_op_reply(126 1.00000000 [omap-set-header] v92'92 uv92 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7fa8e800b040 con 0x5631ad0a2cd0 2024-08-03T20:55:43.825 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:43.824+0000 7fa8f5a30f40 1 -- 172.21.15.32:0/30469862 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:127 12.1f 12:ff5b34d6:::1.00000000:head [omap-set-vals in=584b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e92) -- 0x5631ad0adcf0 con 0x5631ad0a2cd0 2024-08-03T20:55:43.827 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:43.826+0000 7fa8ef600640 1 -- 172.21.15.32:0/30469862 <== osd.6 v2:172.21.15.118:6825/801947386 28 ==== osd_op_reply(127 1.00000000 [omap-set-vals] v92'93 uv93 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7fa8e800b040 con 0x5631ad0a2cd0 2024-08-03T20:55:43.827 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:43.826+0000 7fa8f5a30f40 1 -- 172.21.15.32:0/30469862 >> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] conn(0x5631acfd43e0 msgr2=0x5631acf5eb10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:55:43.827 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:43.826+0000 7fa8f5a30f40 1 --2- 172.21.15.32:0/30469862 >> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] conn(0x5631acfd43e0 0x5631acf5eb10 crc :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:43.827 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:43.826+0000 7fa8f5a30f40 1 -- 172.21.15.32:0/30469862 >> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] conn(0x5631ad09e970 msgr2=0x5631ad09ed80 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:55:43.827 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:43.826+0000 7fa8f5a30f40 1 --2- 172.21.15.32:0/30469862 >> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] conn(0x5631ad09e970 0x5631ad09ed80 crc :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:43.827 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:43.826+0000 7fa8f5a30f40 1 -- 172.21.15.32:0/30469862 >> [v2:172.21.15.32:6800/9088303,v1:172.21.15.32:6801/9088303] conn(0x5631ad0aabb0 msgr2=0x5631ad0ad300 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:55:43.827 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:43.826+0000 7fa8f5a30f40 1 --2- 172.21.15.32:0/30469862 >> [v2:172.21.15.32:6800/9088303,v1:172.21.15.32:6801/9088303] conn(0x5631ad0aabb0 0x5631ad0ad300 crc :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:43.828 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:43.826+0000 7fa8f5a30f40 1 -- 172.21.15.32:0/30469862 >> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] conn(0x5631ad098e90 msgr2=0x5631ad09bfb0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:55:43.828 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:43.826+0000 7fa8f5a30f40 1 --2- 172.21.15.32:0/30469862 >> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] conn(0x5631ad098e90 0x5631ad09bfb0 crc :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:43.828 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:43.826+0000 7fa8f5a30f40 1 -- 172.21.15.32:0/30469862 >> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] conn(0x5631ad0a2cd0 msgr2=0x5631ad0a50f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:55:43.828 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:43.826+0000 7fa8f5a30f40 1 --2- 172.21.15.32:0/30469862 >> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] conn(0x5631ad0a2cd0 0x5631ad0a50f0 crc :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:43.829 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:43.826+0000 7fa8f5a30f40 1 -- 172.21.15.32:0/30469862 >> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] conn(0x5631ad0a6420 msgr2=0x5631ad0a6830 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:55:43.829 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:43.826+0000 7fa8f5a30f40 1 --2- 172.21.15.32:0/30469862 >> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] conn(0x5631ad0a6420 0x5631ad0a6830 crc :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:43.829 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:43.826+0000 7fa8ef600640 1 -- 172.21.15.32:0/30469862 reap_dead start 2024-08-03T20:55:43.830 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:43.826+0000 7fa8ef600640 1 -- 172.21.15.32:0/30469862 reap_dead start 2024-08-03T20:55:43.830 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:43.826+0000 7fa8ef600640 1 -- 172.21.15.32:0/30469862 reap_dead start 2024-08-03T20:55:43.830 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:43.826+0000 7fa8f5a30f40 1 -- 172.21.15.32:0/30469862 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fa8d00759f0 msgr2=0x7fa8d0077f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:55:43.830 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:43.826+0000 7fa8f5a30f40 1 --2- 172.21.15.32:0/30469862 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fa8d00759f0 0x7fa8d0077f00 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7fa8e0000c00 tx=0x7fa8e0002590 comp rx=0 tx=0).stop 2024-08-03T20:55:43.830 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:43.826+0000 7fa8f5a30f40 1 -- 172.21.15.32:0/30469862 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5631acfec330 msgr2=0x5631acfcd510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:55:43.830 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:43.826+0000 7fa8f5a30f40 1 --2- 172.21.15.32:0/30469862 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5631acfec330 0x5631acfcd510 secure :-1 s=READY pgs=1238 cs=0 l=1 rev1=1 crypto rx=0x7fa8dc002000 tx=0x7fa8dc00ff50 comp rx=0 tx=0).stop 2024-08-03T20:55:43.830 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:43.826+0000 7fa8f5a30f40 1 -- 172.21.15.32:0/30469862 shutdown_connections 2024-08-03T20:55:43.830 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:43.826+0000 7fa8f5a30f40 1 --2- 172.21.15.32:0/30469862 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fa8d00759f0 0x7fa8d0077f00 unknown :-1 s=CLOSED pgs=512 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:43.830 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:43.826+0000 7fa8f5a30f40 1 --2- 172.21.15.32:0/30469862 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5631acfec330 0x5631acfcd510 unknown :-1 s=CLOSED pgs=1238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:43.830 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:43.826+0000 7fa8f5a30f40 1 -- 172.21.15.32:0/30469862 >> 172.21.15.32:0/30469862 conn(0x5631ace3eac0 msgr2=0x5631acfeb530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:55:43.830 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:43.826+0000 7fa8f5a30f40 1 -- 172.21.15.32:0/30469862 shutdown_connections 2024-08-03T20:55:43.830 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:43.826+0000 7fa8f5a30f40 1 -- 172.21.15.32:0/30469862 wait complete. 2024-08-03T20:55:43.839 DEBUG:teuthology.orchestra.run.smithi032:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata rm 500.00000000 2024-08-03T20:55:43.983 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:43.982+0000 7fa7d7e1ef40 1 Processor -- start 2024-08-03T20:55:43.983 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:43.982+0000 7fa7d7e1ef40 1 -- start start 2024-08-03T20:55:43.985 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:43.984+0000 7fa7d7e1ef40 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55be39a45360 0x55be39a3f6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:55:43.985 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:43.984+0000 7fa7d7e1ef40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55be39a62cb0 0x55be399d7790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:55:43.986 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:43.984+0000 7fa7d7e1ef40 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55be39a633f0 0x55be39a5e070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:55:43.986 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:43.984+0000 7fa7d7e1ef40 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55be3993abf0 con 0x55be39a633f0 2024-08-03T20:55:43.986 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:43.984+0000 7fa7d7e1ef40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x55be399476b0 con 0x55be39a62cb0 2024-08-03T20:55:43.986 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:43.984+0000 7fa7d7e1ef40 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x55be398dce30 con 0x55be39a45360 2024-08-03T20:55:43.986 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:43.984+0000 7fa7d0e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55be39a633f0 0x55be39a5e070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:55:43.986 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:43.984+0000 7fa7d0e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55be39a633f0 0x55be39a5e070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:59850/0 (socket says 172.21.15.32:59850) 2024-08-03T20:55:43.986 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:43.984+0000 7fa7d0e00640 1 -- 172.21.15.32:0/3280168631 learned_addr learned my addr 172.21.15.32:0/3280168631 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:55:43.986 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:43.984+0000 7fa7d0400640 1 --2- 172.21.15.32:0/3280168631 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55be39a45360 0x55be39a3f6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:55:43.986 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:43.984+0000 7fa7cfa00640 1 --2- 172.21.15.32:0/3280168631 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55be39a62cb0 0x55be399d7790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:55:43.986 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:43.984+0000 7fa7d0e00640 1 -- 172.21.15.32:0/3280168631 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55be39a45360 msgr2=0x55be39a3f6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:55:43.986 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:43.984+0000 7fa7d0e00640 1 --2- 172.21.15.32:0/3280168631 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55be39a45360 0x55be39a3f6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:43.986 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:43.984+0000 7fa7d0e00640 1 -- 172.21.15.32:0/3280168631 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55be39a62cb0 msgr2=0x55be399d7790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:55:43.987 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:43.984+0000 7fa7d0e00640 1 --2- 172.21.15.32:0/3280168631 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55be39a62cb0 0x55be399d7790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:43.987 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:43.985+0000 7fa7d0e00640 1 -- 172.21.15.32:0/3280168631 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55be39a5dd70 con 0x55be39a633f0 2024-08-03T20:55:43.987 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:43.985+0000 7fa7d0e00640 1 --2- 172.21.15.32:0/3280168631 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55be39a633f0 0x55be39a5e070 secure :-1 s=READY pgs=1240 cs=0 l=1 rev1=1 crypto rx=0x7fa7c8001550 tx=0x7fa7c8001a10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=71ffaebdca557a1a server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:55:43.987 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:43.985+0000 7fa7ce600640 1 -- 172.21.15.32:0/3280168631 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7c8017070 con 0x55be39a633f0 2024-08-03T20:55:43.987 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:43.985+0000 7fa7ce600640 1 -- 172.21.15.32:0/3280168631 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fa7c800f820 con 0x55be39a633f0 2024-08-03T20:55:43.987 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:43.985+0000 7fa7ce600640 1 -- 172.21.15.32:0/3280168631 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7c800fc70 con 0x55be39a633f0 2024-08-03T20:55:43.988 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:43.986+0000 7fa7d7e1ef40 1 -- 172.21.15.32:0/3280168631 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55be39a633f0 msgr2=0x55be39a5e070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:55:43.988 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:43.986+0000 7fa7d7e1ef40 1 --2- 172.21.15.32:0/3280168631 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55be39a633f0 0x55be39a5e070 secure :-1 s=READY pgs=1240 cs=0 l=1 rev1=1 crypto rx=0x7fa7c8001550 tx=0x7fa7c8001a10 comp rx=0 tx=0).stop 2024-08-03T20:55:43.988 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:43.986+0000 7fa7d7e1ef40 1 -- 172.21.15.32:0/3280168631 shutdown_connections 2024-08-03T20:55:43.988 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:43.986+0000 7fa7d7e1ef40 1 --2- 172.21.15.32:0/3280168631 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55be39a62cb0 0x55be399d7790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:43.988 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:43.986+0000 7fa7d7e1ef40 1 --2- 172.21.15.32:0/3280168631 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55be39a633f0 0x55be39a5e070 unknown :-1 s=CLOSED pgs=1240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:43.988 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:43.986+0000 7fa7d7e1ef40 1 --2- 172.21.15.32:0/3280168631 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55be39a45360 0x55be39a3f6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:43.989 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:43.986+0000 7fa7d7e1ef40 1 -- 172.21.15.32:0/3280168631 >> 172.21.15.32:0/3280168631 conn(0x55be398aeac0 msgr2=0x55be39a60d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:55:43.989 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:43.986+0000 7fa7d7e1ef40 1 -- 172.21.15.32:0/3280168631 shutdown_connections 2024-08-03T20:55:43.989 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:43.986+0000 7fa7d7e1ef40 1 -- 172.21.15.32:0/3280168631 wait complete. 2024-08-03T20:55:43.989 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:43.987+0000 7fa7d7e1ef40 1 Processor -- start 2024-08-03T20:55:43.989 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:43.987+0000 7fa7d7e1ef40 1 -- start start 2024-08-03T20:55:43.990 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:43.989+0000 7fa7d7e1ef40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55be39a45360 0x55be39a408c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:55:43.990 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:43.989+0000 7fa7d7e1ef40 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55be39a62cb0 0x55be39a61120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:55:43.990 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:43.989+0000 7fa7d7e1ef40 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55be39a633f0 0x55be39a423a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:55:43.990 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:43.989+0000 7fa7d7e1ef40 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55be399d6ce0 con 0x55be39a633f0 2024-08-03T20:55:43.990 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:43.989+0000 7fa7d7e1ef40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x55be399d6fc0 con 0x55be39a45360 2024-08-03T20:55:43.990 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:43.989+0000 7fa7d7e1ef40 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x55be39a42e90 con 0x55be39a62cb0 2024-08-03T20:55:43.991 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:43.989+0000 7fa7d0e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55be39a633f0 0x55be39a423a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:55:43.991 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:43.989+0000 7fa7cfa00640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55be39a62cb0 0x55be39a61120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:55:43.991 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:43.989+0000 7fa7d0400640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55be39a45360 0x55be39a408c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:55:43.991 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:43.989+0000 7fa7cfa00640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55be39a62cb0 0x55be39a61120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3301/0 says I am v2:172.21.15.32:39662/0 (socket says 172.21.15.32:39662) 2024-08-03T20:55:43.991 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:43.989+0000 7fa7d0400640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55be39a45360 0x55be39a408c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.32:60322/0 (socket says 172.21.15.32:60322) 2024-08-03T20:55:43.991 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:43.989+0000 7fa7cfa00640 1 -- 172.21.15.32:0/3655707692 learned_addr learned my addr 172.21.15.32:0/3655707692 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:55:43.991 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:43.989+0000 7fa7cfa00640 1 -- 172.21.15.32:0/3655707692 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55be39a45360 msgr2=0x55be39a408c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:55:43.991 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:43.989+0000 7fa7cfa00640 1 --2- 172.21.15.32:0/3655707692 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55be39a45360 0x55be39a408c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:43.991 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:43.989+0000 7fa7cfa00640 1 -- 172.21.15.32:0/3655707692 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55be39a633f0 msgr2=0x55be39a423a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:55:43.991 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:43.989+0000 7fa7cfa00640 1 --2- 172.21.15.32:0/3655707692 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55be39a633f0 0x55be39a423a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:43.991 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:43.989+0000 7fa7cfa00640 1 -- 172.21.15.32:0/3655707692 --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55be39af8e80 con 0x55be39a62cb0 2024-08-03T20:55:43.991 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:43.990+0000 7fa7cfa00640 1 --2- 172.21.15.32:0/3655707692 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55be39a62cb0 0x55be39a61120 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fa7bc00b2d0 tx=0x7fa7bc00b790 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:55:43.992 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:43.990+0000 7fa7cd200640 1 -- 172.21.15.32:0/3655707692 <== mon.2 v2:172.21.15.118:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7bc013070 con 0x55be39a62cb0 2024-08-03T20:55:43.992 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:43.990+0000 7fa7cd200640 1 -- 172.21.15.32:0/3655707692 <== mon.2 v2:172.21.15.118:3301/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fa7bc006070 con 0x55be39a62cb0 2024-08-03T20:55:43.992 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:43.990+0000 7fa7cd200640 1 -- 172.21.15.32:0/3655707692 <== mon.2 v2:172.21.15.118:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7bc00fb80 con 0x55be39a62cb0 2024-08-03T20:55:43.992 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:43.990+0000 7fa7d7e1ef40 1 -- 172.21.15.32:0/3655707692 --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x55be39a400b0 con 0x55be39a62cb0 2024-08-03T20:55:43.992 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:43.990+0000 7fa7d7e1ef40 1 -- 172.21.15.32:0/3655707692 --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_subscribe({osdmap=0}) -- 0x55be39a42bd0 con 0x55be39a62cb0 2024-08-03T20:55:43.993 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:43.993+0000 7fa7cd200640 1 -- 172.21.15.32:0/3655707692 <== mon.2 v2:172.21.15.118:3301/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7fa7bc018460 con 0x55be39a62cb0 2024-08-03T20:55:43.994 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:43.994+0000 7fa7cd200640 1 --2- 172.21.15.32:0/3655707692 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fa7b00759f0 0x7fa7b0077f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:55:43.995 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:43.994+0000 7fa7cd200640 1 -- 172.21.15.32:0/3655707692 <== mon.2 v2:172.21.15.118:3301/0 5 ==== osd_map(92..92 src has 1..92) ==== 6456+0+0 (secure 0 0 0) 0x7fa7bc097e30 con 0x55be39a62cb0 2024-08-03T20:55:43.995 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:43.994+0000 7fa7d0400640 1 --2- 172.21.15.32:0/3655707692 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fa7b00759f0 0x7fa7b0077f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:55:43.995 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:43.994+0000 7fa7d7e1ef40 1 --2- 172.21.15.32:0/3655707692 >> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] conn(0x55be39af9b70 0x55be39a5e070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:55:43.995 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:43.994+0000 7fa7d7e1ef40 1 -- 172.21.15.32:0/3655707692 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:1 12.15 12:aa448500:::500.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e92) -- 0x55be39a5e5b0 con 0x55be39af9b70 2024-08-03T20:55:43.995 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:43.995+0000 7fa7d0e00640 1 --2- 172.21.15.32:0/3655707692 >> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] conn(0x55be39af9b70 0x55be39a5e070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:55:43.996 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:43.995+0000 7fa7d0400640 1 --2- 172.21.15.32:0/3655707692 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fa7b00759f0 0x7fa7b0077f00 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7fa7c0055000 tx=0x7fa7c0058040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:55:43.996 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:43.995+0000 7fa7d0e00640 1 --2- 172.21.15.32:0/3655707692 >> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] conn(0x55be39af9b70 0x55be39a5e070 crc :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:55:43.998 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:43.997+0000 7fa7d0e00640 1 -- 172.21.15.32:0/3655707692 <== osd.6 v2:172.21.15.118:6825/801947386 1 ==== osd_op_reply(1 500.00000000 [delete] v92'81 uv81 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fa7c8003030 con 0x55be39af9b70 2024-08-03T20:55:43.998 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:43.997+0000 7fa7d7e1ef40 1 -- 172.21.15.32:0/3655707692 >> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] conn(0x55be39af9b70 msgr2=0x55be39a5e070 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:55:43.998 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:43.997+0000 7fa7d7e1ef40 1 --2- 172.21.15.32:0/3655707692 >> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] conn(0x55be39af9b70 0x55be39a5e070 crc :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:43.998 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:43.997+0000 7fa7d7e1ef40 1 -- 172.21.15.32:0/3655707692 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fa7b00759f0 msgr2=0x7fa7b0077f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:55:43.998 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:43.997+0000 7fa7d7e1ef40 1 --2- 172.21.15.32:0/3655707692 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fa7b00759f0 0x7fa7b0077f00 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7fa7c0055000 tx=0x7fa7c0058040 comp rx=0 tx=0).stop 2024-08-03T20:55:43.998 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:43.997+0000 7fa7d7e1ef40 1 -- 172.21.15.32:0/3655707692 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55be39a62cb0 msgr2=0x55be39a61120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:55:43.998 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:43.997+0000 7fa7d7e1ef40 1 --2- 172.21.15.32:0/3655707692 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55be39a62cb0 0x55be39a61120 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fa7bc00b2d0 tx=0x7fa7bc00b790 comp rx=0 tx=0).stop 2024-08-03T20:55:43.998 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:43.997+0000 7fa7d0e00640 1 -- 172.21.15.32:0/3655707692 reap_dead start 2024-08-03T20:55:43.999 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:43.997+0000 7fa7d7e1ef40 1 -- 172.21.15.32:0/3655707692 shutdown_connections 2024-08-03T20:55:43.999 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:43.997+0000 7fa7d7e1ef40 1 -- 172.21.15.32:0/3655707692 >> 172.21.15.32:0/3655707692 conn(0x55be398aeac0 msgr2=0x55be399d4ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:55:43.999 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:43.997+0000 7fa7d7e1ef40 1 -- 172.21.15.32:0/3655707692 shutdown_connections 2024-08-03T20:55:43.999 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:43.997+0000 7fa7d7e1ef40 1 -- 172.21.15.32:0/3655707692 wait complete. 2024-08-03T20:55:44.013 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs joinable true 2024-08-03T20:55:44.246 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:44.245+0000 7faad4a00640 1 -- 172.21.15.32:0/470818072 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7faab4003c10 msgr2=0x7faab4004060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:55:44.246 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:44.245+0000 7faad4a00640 1 --2- 172.21.15.32:0/470818072 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7faab4003c10 0x7faab4004060 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7faac0006f90 tx=0x7faab4005830 comp rx=0 tx=0).stop 2024-08-03T20:55:44.246 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:44.245+0000 7faacf200640 1 -- 172.21.15.32:0/470818072 reap_dead start 2024-08-03T20:55:44.246 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:44.245+0000 7faad4a00640 1 -- 172.21.15.32:0/470818072 shutdown_connections 2024-08-03T20:55:44.246 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:44.245+0000 7faad4a00640 1 -- 172.21.15.32:0/470818072 >> 172.21.15.32:0/470818072 conn(0x7faad00cfd80 msgr2=0x7faad00d0190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:55:44.247 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:44.245+0000 7faad4a00640 1 -- 172.21.15.32:0/470818072 shutdown_connections 2024-08-03T20:55:44.247 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:44.245+0000 7faad4a00640 1 -- 172.21.15.32:0/470818072 wait complete. 2024-08-03T20:55:44.248 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:44.247+0000 7faad4a00640 1 Processor -- start 2024-08-03T20:55:44.248 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:44.248+0000 7faad4a00640 1 -- start start 2024-08-03T20:55:44.251 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:44.249+0000 7faad4a00640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7faab4003c10 0x7faad015c470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:55:44.251 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:44.249+0000 7faad4a00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7faad00b3580 0x7faad014d470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:55:44.251 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:44.249+0000 7faad4a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faad00b4210 0x7faad0154740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:55:44.251 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:44.249+0000 7faad4a00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7faad0165190 con 0x7faad00b4210 2024-08-03T20:55:44.251 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:44.249+0000 7faad4a00640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7faad0160730 con 0x7faad00b3580 2024-08-03T20:55:44.251 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:44.249+0000 7faad4a00640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7faad0165310 con 0x7faab4003c10 2024-08-03T20:55:44.251 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:44.249+0000 7faacf200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faad00b4210 0x7faad0154740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:55:44.251 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:44.249+0000 7faacf200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faad00b4210 0x7faad0154740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:59870/0 (socket says 172.21.15.32:59870) 2024-08-03T20:55:44.251 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:44.249+0000 7faacf200640 1 -- 172.21.15.32:0/929325980 learned_addr learned my addr 172.21.15.32:0/929325980 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:55:44.252 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:44.250+0000 7faace800640 1 --2- 172.21.15.32:0/929325980 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7faab4003c10 0x7faad015c470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:55:44.252 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:44.250+0000 7faac7e00640 1 --2- 172.21.15.32:0/929325980 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7faad00b3580 0x7faad014d470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:55:44.252 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:44.250+0000 7faacf200640 1 -- 172.21.15.32:0/929325980 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7faab4003c10 msgr2=0x7faad015c470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:55:44.252 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:44.250+0000 7faacf200640 1 --2- 172.21.15.32:0/929325980 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7faab4003c10 0x7faad015c470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:44.252 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:44.250+0000 7faacf200640 1 -- 172.21.15.32:0/929325980 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7faad00b3580 msgr2=0x7faad014d470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:55:44.252 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:44.250+0000 7faacf200640 1 --2- 172.21.15.32:0/929325980 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7faad00b3580 0x7faad014d470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:44.252 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:44.250+0000 7faacf200640 1 -- 172.21.15.32:0/929325980 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faad00b3310 con 0x7faad00b4210 2024-08-03T20:55:44.252 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:44.250+0000 7faacf200640 1 --2- 172.21.15.32:0/929325980 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faad00b4210 0x7faad0154740 secure :-1 s=READY pgs=1242 cs=0 l=1 rev1=1 crypto rx=0x7faad00cd840 tx=0x7faac800f790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:55:44.252 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:44.250+0000 7faac6a00640 1 -- 172.21.15.32:0/929325980 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faac8003d80 con 0x7faad00b4210 2024-08-03T20:55:44.252 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:44.250+0000 7faac6a00640 1 -- 172.21.15.32:0/929325980 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7faac8004120 con 0x7faad00b4210 2024-08-03T20:55:44.252 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:44.250+0000 7faac6a00640 1 -- 172.21.15.32:0/929325980 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faac8004550 con 0x7faad00b4210 2024-08-03T20:55:44.252 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:44.250+0000 7faad4a00640 1 -- 172.21.15.32:0/929325980 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faad014de30 con 0x7faad00b4210 2024-08-03T20:55:44.252 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:44.251+0000 7faad4a00640 1 -- 172.21.15.32:0/929325980 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faad00b5630 con 0x7faad00b4210 2024-08-03T20:55:44.253 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:44.252+0000 7faad4a00640 1 -- 172.21.15.32:0/929325980 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faad015fe10 con 0x7faad00b4210 2024-08-03T20:55:44.253 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:44.253+0000 7faac6a00640 1 -- 172.21.15.32:0/929325980 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7faac80046f0 con 0x7faad00b4210 2024-08-03T20:55:44.254 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:44.254+0000 7faac6a00640 1 --2- 172.21.15.32:0/929325980 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7faaac077010 0x7faaac079520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:55:44.254 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:44.254+0000 7faac6a00640 1 -- 172.21.15.32:0/929325980 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 6456+0+0 (secure 0 0 0) 0x7faac8099900 con 0x7faad00b4210 2024-08-03T20:55:44.255 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:44.254+0000 7faace800640 1 --2- 172.21.15.32:0/929325980 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7faaac077010 0x7faaac079520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:55:44.255 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:44.255+0000 7faace800640 1 --2- 172.21.15.32:0/929325980 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7faaac077010 0x7faaac079520 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7faac0033000 tx=0x7faac002d000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:55:44.259 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:44.259+0000 7faac6a00640 1 -- 172.21.15.32:0/929325980 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7faac8021070 con 0x7faad00b4210 2024-08-03T20:55:44.504 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:44.503+0000 7faad4a00640 1 -- 172.21.15.32:0/929325980 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "joinable", "val": "true"} v 0) -- 0x7faad00b59e0 con 0x7faad00b4210 2024-08-03T20:55:45.015 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:45.015+0000 7faac6a00640 1 -- 172.21.15.32:0/929325980 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "joinable", "val": "true"}]=0 cephfs marked joinable; MDS may join as newly active. v134) ==== 162+0+0 (secure 0 0 0) 0x7faac8062800 con 0x7faad00b4210 2024-08-03T20:55:45.015 INFO:teuthology.orchestra.run.smithi032.stderr:cephfs marked joinable; MDS may join as newly active. 2024-08-03T20:55:45.018 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:45.017+0000 7faad4a00640 1 -- 172.21.15.32:0/929325980 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7faaac077010 msgr2=0x7faaac079520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:55:45.018 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:45.017+0000 7faad4a00640 1 --2- 172.21.15.32:0/929325980 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7faaac077010 0x7faaac079520 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7faac0033000 tx=0x7faac002d000 comp rx=0 tx=0).stop 2024-08-03T20:55:45.018 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:45.017+0000 7faad4a00640 1 -- 172.21.15.32:0/929325980 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faad00b4210 msgr2=0x7faad0154740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:55:45.018 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:45.017+0000 7faad4a00640 1 --2- 172.21.15.32:0/929325980 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faad00b4210 0x7faad0154740 secure :-1 s=READY pgs=1242 cs=0 l=1 rev1=1 crypto rx=0x7faad00cd840 tx=0x7faac800f790 comp rx=0 tx=0).stop 2024-08-03T20:55:45.018 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:45.017+0000 7faad4a00640 1 -- 172.21.15.32:0/929325980 shutdown_connections 2024-08-03T20:55:45.018 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:45.017+0000 7faad4a00640 1 --2- 172.21.15.32:0/929325980 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7faaac077010 0x7faaac079520 unknown :-1 s=CLOSED pgs=514 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:45.019 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:45.017+0000 7faad4a00640 1 --2- 172.21.15.32:0/929325980 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7faad00b3580 0x7faad014d470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:45.019 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:45.017+0000 7faad4a00640 1 --2- 172.21.15.32:0/929325980 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faad00b4210 0x7faad0154740 unknown :-1 s=CLOSED pgs=1242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:45.019 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:45.017+0000 7faad4a00640 1 --2- 172.21.15.32:0/929325980 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7faab4003c10 0x7faad015c470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:45.019 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:45.017+0000 7faad4a00640 1 -- 172.21.15.32:0/929325980 >> 172.21.15.32:0/929325980 conn(0x7faad00cfd80 msgr2=0x7faad015e250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:55:45.019 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:45.018+0000 7faad4a00640 1 -- 172.21.15.32:0/929325980 shutdown_connections 2024-08-03T20:55:45.019 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:45.018+0000 7faad4a00640 1 -- 172.21.15.32:0/929325980 wait complete. 2024-08-03T20:55:45.447 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-08-03T20:55:45.683 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:45.682+0000 7f9f13400640 1 -- 172.21.15.32:0/318069200 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9ef4003a30 msgr2=0x7f9ef4003ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:55:45.683 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:45.682+0000 7f9f13400640 1 --2- 172.21.15.32:0/318069200 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9ef4003a30 0x7f9ef4003ea0 secure :-1 s=READY pgs=1244 cs=0 l=1 rev1=1 crypto rx=0x7f9f000016c0 tx=0x7f9ef4005830 comp rx=0 tx=0).stop 2024-08-03T20:55:45.684 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:45.682+0000 7f9f11400640 1 -- 172.21.15.32:0/318069200 reap_dead start 2024-08-03T20:55:45.684 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:45.682+0000 7f9f13400640 1 -- 172.21.15.32:0/318069200 shutdown_connections 2024-08-03T20:55:45.684 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:45.682+0000 7f9f13400640 1 -- 172.21.15.32:0/318069200 >> 172.21.15.32:0/318069200 conn(0x7f9f0c0b1bd0 msgr2=0x7f9f0c0b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:55:45.684 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:45.683+0000 7f9f13400640 1 -- 172.21.15.32:0/318069200 shutdown_connections 2024-08-03T20:55:45.684 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:45.683+0000 7f9f13400640 1 -- 172.21.15.32:0/318069200 wait complete. 2024-08-03T20:55:45.685 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:45.685+0000 7f9f13400640 1 Processor -- start 2024-08-03T20:55:45.685 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:45.685+0000 7f9f13400640 1 -- start start 2024-08-03T20:55:45.688 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:45.686+0000 7f9f13400640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f9ef4003a30 0x7f9f0c0af260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:55:45.689 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:45.686+0000 7f9f13400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9f0c1436b0 0x7f9f0c143ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:55:45.689 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:45.686+0000 7f9f13400640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f9f0c131eb0 0x7f9f0c1322c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:55:45.689 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:45.686+0000 7f9f13400640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f9f0c0cf840 con 0x7f9f0c1436b0 2024-08-03T20:55:45.689 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:45.686+0000 7f9f13400640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f9f0c145f30 con 0x7f9ef4003a30 2024-08-03T20:55:45.689 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:45.686+0000 7f9f13400640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7f9f0c0cf9c0 con 0x7f9f0c131eb0 2024-08-03T20:55:45.689 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:45.687+0000 7f9f0be00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9f0c1436b0 0x7f9f0c143ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:55:45.689 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:45.687+0000 7f9f0be00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9f0c1436b0 0x7f9f0c143ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:59894/0 (socket says 172.21.15.32:59894) 2024-08-03T20:55:45.689 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:45.687+0000 7f9f0be00640 1 -- 172.21.15.32:0/1648928045 learned_addr learned my addr 172.21.15.32:0/1648928045 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:55:45.689 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:45.687+0000 7f9f11400640 1 --2- 172.21.15.32:0/1648928045 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f9f0c131eb0 0x7f9f0c1322c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:55:45.689 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:45.687+0000 7f9f10a00640 1 --2- 172.21.15.32:0/1648928045 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f9ef4003a30 0x7f9f0c0af260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:55:45.689 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:45.687+0000 7f9f0be00640 1 -- 172.21.15.32:0/1648928045 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f9f0c131eb0 msgr2=0x7f9f0c1322c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:55:45.689 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:45.687+0000 7f9f0be00640 1 --2- 172.21.15.32:0/1648928045 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f9f0c131eb0 0x7f9f0c1322c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:45.689 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:45.687+0000 7f9f0be00640 1 -- 172.21.15.32:0/1648928045 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f9ef4003a30 msgr2=0x7f9f0c0af260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:55:45.690 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:45.687+0000 7f9f0be00640 1 --2- 172.21.15.32:0/1648928045 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f9ef4003a30 0x7f9f0c0af260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:45.690 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:45.687+0000 7f9f0be00640 1 -- 172.21.15.32:0/1648928045 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9f0c144000 con 0x7f9f0c1436b0 2024-08-03T20:55:45.690 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:45.687+0000 7f9f11400640 1 --2- 172.21.15.32:0/1648928045 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f9f0c131eb0 0x7f9f0c1322c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-03T20:55:45.690 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:45.687+0000 7f9f10a00640 1 --2- 172.21.15.32:0/1648928045 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f9ef4003a30 0x7f9f0c0af260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-03T20:55:45.690 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:45.687+0000 7f9f0be00640 1 --2- 172.21.15.32:0/1648928045 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9f0c1436b0 0x7f9f0c143ac0 secure :-1 s=READY pgs=1245 cs=0 l=1 rev1=1 crypto rx=0x7f9ef8011ae0 tx=0x7f9ef8009e90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:55:45.690 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:45.687+0000 7f9f08c00640 1 -- 172.21.15.32:0/1648928045 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ef801e070 con 0x7f9f0c1436b0 2024-08-03T20:55:45.690 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:45.688+0000 7f9f08c00640 1 -- 172.21.15.32:0/1648928045 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f9ef800fe70 con 0x7f9f0c1436b0 2024-08-03T20:55:45.690 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:45.688+0000 7f9f08c00640 1 -- 172.21.15.32:0/1648928045 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ef8004290 con 0x7f9f0c1436b0 2024-08-03T20:55:45.690 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:45.688+0000 7f9f13400640 1 -- 172.21.15.32:0/1648928045 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9f0c136c30 con 0x7f9f0c1436b0 2024-08-03T20:55:45.690 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:45.688+0000 7f9f13400640 1 -- 172.21.15.32:0/1648928045 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9f0c132ee0 con 0x7f9f0c1436b0 2024-08-03T20:55:45.691 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:45.689+0000 7f9f13400640 1 -- 172.21.15.32:0/1648928045 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9f0c0cf580 con 0x7f9f0c1436b0 2024-08-03T20:55:45.691 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:45.690+0000 7f9f08c00640 1 -- 172.21.15.32:0/1648928045 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f9ef8021070 con 0x7f9f0c1436b0 2024-08-03T20:55:45.691 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:45.691+0000 7f9f08c00640 1 --2- 172.21.15.32:0/1648928045 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f9eec077010 0x7f9eec079520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:55:45.692 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:45.691+0000 7f9f08c00640 1 -- 172.21.15.32:0/1648928045 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 6456+0+0 (secure 0 0 0) 0x7f9ef80995d0 con 0x7f9f0c1436b0 2024-08-03T20:55:45.692 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:45.692+0000 7f9f10a00640 1 --2- 172.21.15.32:0/1648928045 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f9eec077010 0x7f9eec079520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:55:45.693 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:45.692+0000 7f9f10a00640 1 --2- 172.21.15.32:0/1648928045 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f9eec077010 0x7f9eec079520 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7f9f00032000 tx=0x7f9f00001190 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:55:45.696 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:45.696+0000 7f9f08c00640 1 -- 172.21.15.32:0/1648928045 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7f9f0c0cf580 con 0x7f9f0c1436b0 2024-08-03T20:55:45.937 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:45.937+0000 7f9f13400640 1 -- 172.21.15.32:0/1648928045 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f9f0c073e60 con 0x7f9f0c1436b0 2024-08-03T20:55:45.938 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:45.938+0000 7f9f08c00640 1 -- 172.21.15.32:0/1648928045 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 134 v134) ==== 95+0+5225 (secure 0 0 0) 0x7f9f0c144000 con 0x7f9f0c1436b0 2024-08-03T20:55:45.938 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-03T20:55:45.939 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":134,"btime":"2024-08-03T20:55:45:013753+0000","default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6439,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6835/531080160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6834","nonce":531080160},{"type":"v1","addr":"172.21.15.118:6835","nonce":531080160}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":84},{"gid":6454,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6837/2796882319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6836","nonce":2796882319},{"type":"v1","addr":"172.21.15.118:6837","nonce":2796882319}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":86},{"gid":6813,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6834/2686933890","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":2686933890},{"type":"v1","addr":"172.21.15.32:6834","nonce":2686933890}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":85}],"filesystems":[{"mdsmap":{"epoch":134,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-08-03T20:54:26.950518+0000","modified":"2024-08-03T20:55:45.013734+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":92,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7857},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7857":{"gid":7857,"name":"a","rank":0,"incarnation":134,"state":"up:replay","state_seq":1,"addr":"172.21.15.32:6835/2803901497","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6833","nonce":2803901497},{"type":"v1","addr":"172.21.15.32:6835","nonce":2803901497}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":6}]} 2024-08-03T20:55:45.939 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 134 2024-08-03T20:55:45.941 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:45.940+0000 7f9f13400640 1 -- 172.21.15.32:0/1648928045 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f9eec077010 msgr2=0x7f9eec079520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:55:45.941 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:45.940+0000 7f9f13400640 1 --2- 172.21.15.32:0/1648928045 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f9eec077010 0x7f9eec079520 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7f9f00032000 tx=0x7f9f00001190 comp rx=0 tx=0).stop 2024-08-03T20:55:45.941 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:45.940+0000 7f9f13400640 1 -- 172.21.15.32:0/1648928045 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9f0c1436b0 msgr2=0x7f9f0c143ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:55:45.941 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:45.940+0000 7f9f13400640 1 --2- 172.21.15.32:0/1648928045 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9f0c1436b0 0x7f9f0c143ac0 secure :-1 s=READY pgs=1245 cs=0 l=1 rev1=1 crypto rx=0x7f9ef8011ae0 tx=0x7f9ef8009e90 comp rx=0 tx=0).stop 2024-08-03T20:55:45.941 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:45.940+0000 7f9f13400640 1 -- 172.21.15.32:0/1648928045 shutdown_connections 2024-08-03T20:55:45.941 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:45.940+0000 7f9f13400640 1 --2- 172.21.15.32:0/1648928045 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f9eec077010 0x7f9eec079520 unknown :-1 s=CLOSED pgs=515 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:45.941 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:45.940+0000 7f9f13400640 1 --2- 172.21.15.32:0/1648928045 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f9f0c131eb0 0x7f9f0c1322c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:45.941 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:45.940+0000 7f9f13400640 1 --2- 172.21.15.32:0/1648928045 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9f0c1436b0 0x7f9f0c143ac0 unknown :-1 s=CLOSED pgs=1245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:45.941 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:45.940+0000 7f9f13400640 1 --2- 172.21.15.32:0/1648928045 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f9ef4003a30 0x7f9f0c0af260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:45.942 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:45.940+0000 7f9f13400640 1 -- 172.21.15.32:0/1648928045 >> 172.21.15.32:0/1648928045 conn(0x7f9f0c0b1bd0 msgr2=0x7f9f0c0afb40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:55:45.942 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:45.940+0000 7f9f13400640 1 -- 172.21.15.32:0/1648928045 shutdown_connections 2024-08-03T20:55:45.942 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:45.940+0000 7f9f13400640 1 -- 172.21.15.32:0/1648928045 wait complete. 2024-08-03T20:55:45.962 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 134, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-08-03T20:54:26.950518+0000', 'modified': '2024-08-03T20:55:45.013734+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 92, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 7857}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7857': {'gid': 7857, 'name': 'a', 'rank': 0, 'incarnation': 134, 'state': 'up:replay', 'state_seq': 1, 'addr': '172.21.15.32:6835/2803901497', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6833', 'nonce': 2803901497}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 2803901497}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [13], 'metadata_pool': 12, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 0, 'qdb_cluster': []} 2024-08-03T20:55:45.963 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_7857:up:replay 2024-08-03T20:55:45.963 DEBUG:tasks.ceph_test_case:wait_until_true: waiting (timeout=60 retry_count=0)... 2024-08-03T20:55:48.447 INFO:tasks.ceph.mds.a.smithi032.stderr:2024-08-03T20:55:48.446+0000 7f4ec5a00640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-08-03T20:55:51.367 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-08-03T20:55:51.613 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:51.610+0000 7f9155e00640 1 -- 172.21.15.32:0/2625993759 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f9134005280 msgr2=0x7f913400b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:55:51.613 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:51.610+0000 7f9155e00640 1 --2- 172.21.15.32:0/2625993759 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f9134005280 0x7f913400b9c0 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f914800c590 tx=0x7f9134005830 comp rx=0 tx=0).stop 2024-08-03T20:55:51.613 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:51.610+0000 7f9155e00640 1 -- 172.21.15.32:0/2625993759 shutdown_connections 2024-08-03T20:55:51.613 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:51.610+0000 7f9155e00640 1 --2- 172.21.15.32:0/2625993759 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f9134005280 0x7f913400b9c0 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:51.614 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:51.610+0000 7f9155e00640 1 -- 172.21.15.32:0/2625993759 >> 172.21.15.32:0/2625993759 conn(0x7f91500b1bd0 msgr2=0x7f91500b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:55:51.614 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:51.610+0000 7f9155e00640 1 -- 172.21.15.32:0/2625993759 shutdown_connections 2024-08-03T20:55:51.614 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:51.610+0000 7f9155e00640 1 -- 172.21.15.32:0/2625993759 wait complete. 2024-08-03T20:55:51.615 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:51.613+0000 7f9155e00640 1 Processor -- start 2024-08-03T20:55:51.615 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:51.613+0000 7f9155e00640 1 -- start start 2024-08-03T20:55:51.619 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:51.614+0000 7f9155e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9134005280 0x7f915015b940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:55:51.619 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:51.614+0000 7f9155e00640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f9150141ef0 0x7f91500ca860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:55:51.619 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:51.614+0000 7f9155e00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f91500cada0 0x7f91501f84b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:55:51.619 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:51.614+0000 7f9155e00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f9150145110 con 0x7f9134005280 2024-08-03T20:55:51.619 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:51.614+0000 7f9155e00640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f9150144f90 con 0x7f91500cada0 2024-08-03T20:55:51.619 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:51.614+0000 7f9155e00640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7f9150145290 con 0x7f9150141ef0 2024-08-03T20:55:51.620 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:51.614+0000 7f914ea00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9134005280 0x7f915015b940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:55:51.620 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:51.615+0000 7f914ea00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9134005280 0x7f915015b940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:50556/0 (socket says 172.21.15.32:50556) 2024-08-03T20:55:51.620 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:51.615+0000 7f914ea00640 1 -- 172.21.15.32:0/3807202744 learned_addr learned my addr 172.21.15.32:0/3807202744 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:55:51.620 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:51.615+0000 7f914e000640 1 --2- 172.21.15.32:0/3807202744 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f9150141ef0 0x7f91500ca860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:55:51.620 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:51.615+0000 7f914f400640 1 --2- 172.21.15.32:0/3807202744 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f91500cada0 0x7f91501f84b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:55:51.620 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:51.615+0000 7f914ea00640 1 -- 172.21.15.32:0/3807202744 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f9150141ef0 msgr2=0x7f91500ca860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:55:51.620 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:51.615+0000 7f914ea00640 1 --2- 172.21.15.32:0/3807202744 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f9150141ef0 0x7f91500ca860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:51.621 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:51.615+0000 7f914ea00640 1 -- 172.21.15.32:0/3807202744 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f91500cada0 msgr2=0x7f91501f84b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:55:51.621 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:51.615+0000 7f914ea00640 1 --2- 172.21.15.32:0/3807202744 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f91500cada0 0x7f91501f84b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:51.621 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:51.615+0000 7f914ea00640 1 -- 172.21.15.32:0/3807202744 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f91500cb1b0 con 0x7f9134005280 2024-08-03T20:55:51.621 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:51.615+0000 7f914ea00640 1 --2- 172.21.15.32:0/3807202744 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9134005280 0x7f915015b940 secure :-1 s=READY pgs=1247 cs=0 l=1 rev1=1 crypto rx=0x7f9140011ae0 tx=0x7f9140009e60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:55:51.621 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:51.616+0000 7f9147400640 1 -- 172.21.15.32:0/3807202744 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f914001e070 con 0x7f9134005280 2024-08-03T20:55:51.621 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:51.616+0000 7f9147400640 1 -- 172.21.15.32:0/3807202744 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f914000fc10 con 0x7f9134005280 2024-08-03T20:55:51.622 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:51.616+0000 7f9147400640 1 -- 172.21.15.32:0/3807202744 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9140019410 con 0x7f9134005280 2024-08-03T20:55:51.622 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:51.616+0000 7f9155e00640 1 -- 172.21.15.32:0/3807202744 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f915015d510 con 0x7f9134005280 2024-08-03T20:55:51.622 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:51.616+0000 7f9155e00640 1 -- 172.21.15.32:0/3807202744 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f915015dad0 con 0x7f9134005280 2024-08-03T20:55:51.622 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:51.617+0000 7f9155e00640 1 -- 172.21.15.32:0/3807202744 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9150074350 con 0x7f9134005280 2024-08-03T20:55:51.623 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:51.618+0000 7f9147400640 1 -- 172.21.15.32:0/3807202744 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f9140021070 con 0x7f9134005280 2024-08-03T20:55:51.623 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:51.619+0000 7f9147400640 1 --2- 172.21.15.32:0/3807202744 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f9130077010 0x7f9130079520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:55:51.624 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:51.619+0000 7f9147400640 1 -- 172.21.15.32:0/3807202744 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 6456+0+0 (secure 0 0 0) 0x7f91400993f0 con 0x7f9134005280 2024-08-03T20:55:51.624 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:51.619+0000 7f914e000640 1 --2- 172.21.15.32:0/3807202744 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f9130077010 0x7f9130079520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:55:51.624 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:51.620+0000 7f914e000640 1 --2- 172.21.15.32:0/3807202744 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f9130077010 0x7f9130079520 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7f9150144a90 tx=0x7f913c006b10 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:55:51.625 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:51.624+0000 7f9147400640 1 -- 172.21.15.32:0/3807202744 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7f91500cb1b0 con 0x7f9134005280 2024-08-03T20:55:51.867 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:51.865+0000 7f9155e00640 1 -- 172.21.15.32:0/3807202744 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f91500cd540 con 0x7f9134005280 2024-08-03T20:55:51.868 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:51.866+0000 7f9147400640 1 -- 172.21.15.32:0/3807202744 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 137 v137) ==== 95+0+5232 (secure 0 0 0) 0x7f9140061ca0 con 0x7f9134005280 2024-08-03T20:55:51.868 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-03T20:55:51.868 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":137,"btime":"2024-08-03T20:55:48:024216+0000","default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6439,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6835/531080160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6834","nonce":531080160},{"type":"v1","addr":"172.21.15.118:6835","nonce":531080160}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":84},{"gid":6454,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6837/2796882319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6836","nonce":2796882319},{"type":"v1","addr":"172.21.15.118:6837","nonce":2796882319}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":86},{"gid":6813,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6834/2686933890","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":2686933890},{"type":"v1","addr":"172.21.15.32:6834","nonce":2686933890}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":85}],"filesystems":[{"mdsmap":{"epoch":137,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-08-03T20:54:26.950518+0000","modified":"2024-08-03T20:55:48.024206+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":92,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7857},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7857":{"gid":7857,"name":"a","rank":0,"incarnation":134,"state":"up:active","state_seq":4,"addr":"172.21.15.32:6835/2803901497","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6833","nonce":2803901497},{"type":"v1","addr":"172.21.15.32:6835","nonce":2803901497}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7857,"qdb_cluster":[7857]},"id":6}]} 2024-08-03T20:55:51.869 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 137 2024-08-03T20:55:51.870 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:51.869+0000 7f9155e00640 1 -- 172.21.15.32:0/3807202744 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f9130077010 msgr2=0x7f9130079520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:55:51.871 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:51.869+0000 7f9155e00640 1 --2- 172.21.15.32:0/3807202744 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f9130077010 0x7f9130079520 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7f9150144a90 tx=0x7f913c006b10 comp rx=0 tx=0).stop 2024-08-03T20:55:51.871 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:51.869+0000 7f9155e00640 1 -- 172.21.15.32:0/3807202744 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9134005280 msgr2=0x7f915015b940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:55:51.871 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:51.869+0000 7f9155e00640 1 --2- 172.21.15.32:0/3807202744 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9134005280 0x7f915015b940 secure :-1 s=READY pgs=1247 cs=0 l=1 rev1=1 crypto rx=0x7f9140011ae0 tx=0x7f9140009e60 comp rx=0 tx=0).stop 2024-08-03T20:55:51.871 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:51.869+0000 7f9155e00640 1 -- 172.21.15.32:0/3807202744 shutdown_connections 2024-08-03T20:55:51.871 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:51.869+0000 7f9155e00640 1 --2- 172.21.15.32:0/3807202744 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f9130077010 0x7f9130079520 unknown :-1 s=CLOSED pgs=516 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:51.871 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:51.869+0000 7f9155e00640 1 --2- 172.21.15.32:0/3807202744 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f91500cada0 0x7f91501f84b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:51.871 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:51.869+0000 7f9155e00640 1 --2- 172.21.15.32:0/3807202744 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f9150141ef0 0x7f91500ca860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:51.871 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:51.869+0000 7f9155e00640 1 --2- 172.21.15.32:0/3807202744 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9134005280 0x7f915015b940 unknown :-1 s=CLOSED pgs=1247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:51.871 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:51.869+0000 7f9155e00640 1 -- 172.21.15.32:0/3807202744 >> 172.21.15.32:0/3807202744 conn(0x7f91500b1bd0 msgr2=0x7f9150140a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:55:51.871 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:51.869+0000 7f9155e00640 1 -- 172.21.15.32:0/3807202744 shutdown_connections 2024-08-03T20:55:51.871 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:51.869+0000 7f9155e00640 1 -- 172.21.15.32:0/3807202744 wait complete. 2024-08-03T20:55:51.897 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 137, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-08-03T20:54:26.950518+0000', 'modified': '2024-08-03T20:55:48.024206+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 92, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 7857}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7857': {'gid': 7857, 'name': 'a', 'rank': 0, 'incarnation': 134, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.32:6835/2803901497', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6833', 'nonce': 2803901497}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 2803901497}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [13], 'metadata_pool': 12, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 7857, 'qdb_cluster': [7857]} 2024-08-03T20:55:51.897 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-08-03T20:55:51.897 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2024-08-03T20:55:52.127 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:52.125+0000 7f77fa400640 1 -- 172.21.15.32:0/3453714253 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f77d80039e0 msgr2=0x7f77d8003e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:55:52.128 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:52.125+0000 7f77fa400640 1 --2- 172.21.15.32:0/3453714253 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f77d80039e0 0x7f77d8003e10 secure :-1 s=READY pgs=1249 cs=0 l=1 rev1=1 crypto rx=0x7f77e40017c0 tx=0x7f77d8005830 comp rx=0 tx=0).stop 2024-08-03T20:55:52.128 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:52.125+0000 7f77f3e00640 1 -- 172.21.15.32:0/3453714253 reap_dead start 2024-08-03T20:55:52.128 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:52.126+0000 7f77fa400640 1 -- 172.21.15.32:0/3453714253 shutdown_connections 2024-08-03T20:55:52.128 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:52.126+0000 7f77fa400640 1 -- 172.21.15.32:0/3453714253 >> 172.21.15.32:0/3453714253 conn(0x7f77f40b1bd0 msgr2=0x7f77f40b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:55:52.128 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:52.126+0000 7f77fa400640 1 -- 172.21.15.32:0/3453714253 shutdown_connections 2024-08-03T20:55:52.128 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:52.126+0000 7f77fa400640 1 -- 172.21.15.32:0/3453714253 wait complete. 2024-08-03T20:55:52.129 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:52.128+0000 7f77fa400640 1 Processor -- start 2024-08-03T20:55:52.129 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:52.128+0000 7f77fa400640 1 -- start start 2024-08-03T20:55:52.137 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:52.129+0000 7f77fa400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f77d80039e0 0x7f77f4143ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:55:52.137 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:52.129+0000 7f77fa400640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f77f413a940 0x7f77f413ad50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:55:52.137 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:52.130+0000 7f77fa400640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f77f4138530 0x7f77f4138940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:55:52.137 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:52.130+0000 7f77fa400640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f77f4145eb0 con 0x7f77d80039e0 2024-08-03T20:55:52.137 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:52.130+0000 7f77fa400640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f77f40cfa50 con 0x7f77f4138530 2024-08-03T20:55:52.137 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:52.130+0000 7f77fa400640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7f77f4165570 con 0x7f77f413a940 2024-08-03T20:55:52.137 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:52.130+0000 7f77f3400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f77d80039e0 0x7f77f4143ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:55:52.137 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:52.130+0000 7f77f3400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f77d80039e0 0x7f77f4143ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:50574/0 (socket says 172.21.15.32:50574) 2024-08-03T20:55:52.137 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:52.130+0000 7f77f3400640 1 -- 172.21.15.32:0/1231718604 learned_addr learned my addr 172.21.15.32:0/1231718604 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:55:52.138 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:52.130+0000 7f77f2a00640 1 --2- 172.21.15.32:0/1231718604 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f77f413a940 0x7f77f413ad50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:55:52.138 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:52.130+0000 7f77f3e00640 1 --2- 172.21.15.32:0/1231718604 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f77f4138530 0x7f77f4138940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:55:52.138 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:52.130+0000 7f77f3400640 1 -- 172.21.15.32:0/1231718604 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f77f413a940 msgr2=0x7f77f413ad50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:55:52.138 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:52.130+0000 7f77f3400640 1 --2- 172.21.15.32:0/1231718604 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f77f413a940 0x7f77f413ad50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:52.138 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:52.130+0000 7f77f3400640 1 -- 172.21.15.32:0/1231718604 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f77f4138530 msgr2=0x7f77f4138940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:55:52.138 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:52.130+0000 7f77f3400640 1 --2- 172.21.15.32:0/1231718604 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f77f4138530 0x7f77f4138940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:52.138 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:52.130+0000 7f77f3400640 1 -- 172.21.15.32:0/1231718604 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f77f4136230 con 0x7f77d80039e0 2024-08-03T20:55:52.138 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:52.130+0000 7f77f2a00640 1 --2- 172.21.15.32:0/1231718604 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f77f413a940 0x7f77f413ad50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-03T20:55:52.138 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:52.130+0000 7f77f3e00640 1 --2- 172.21.15.32:0/1231718604 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f77f4138530 0x7f77f4138940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-03T20:55:52.138 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:52.131+0000 7f77f3400640 1 --2- 172.21.15.32:0/1231718604 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f77d80039e0 0x7f77f4143ed0 secure :-1 s=READY pgs=1250 cs=0 l=1 rev1=1 crypto rx=0x7f77e40018e0 tx=0x7f77e40060f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:55:52.138 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:52.131+0000 7f77eb400640 1 -- 172.21.15.32:0/1231718604 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77e4037040 con 0x7f77d80039e0 2024-08-03T20:55:52.139 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:52.131+0000 7f77eb400640 1 -- 172.21.15.32:0/1231718604 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f77e4041070 con 0x7f77d80039e0 2024-08-03T20:55:52.139 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:52.131+0000 7f77eb400640 1 -- 172.21.15.32:0/1231718604 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77e4003a90 con 0x7f77d80039e0 2024-08-03T20:55:52.139 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:52.131+0000 7f77fa400640 1 -- 172.21.15.32:0/1231718604 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f77f4136510 con 0x7f77d80039e0 2024-08-03T20:55:52.139 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:52.131+0000 7f77fa400640 1 -- 172.21.15.32:0/1231718604 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f77f4136ad0 con 0x7f77d80039e0 2024-08-03T20:55:52.139 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:52.134+0000 7f77eb400640 1 -- 172.21.15.32:0/1231718604 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f77e4003c30 con 0x7f77d80039e0 2024-08-03T20:55:52.139 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:52.135+0000 7f77eb400640 1 --2- 172.21.15.32:0/1231718604 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f77d4077010 0x7f77d4079520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:55:52.139 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:52.135+0000 7f77eb400640 1 -- 172.21.15.32:0/1231718604 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 6456+0+0 (secure 0 0 0) 0x7f77e40b9180 con 0x7f77d80039e0 2024-08-03T20:55:52.139 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:52.135+0000 7f77f2a00640 1 --2- 172.21.15.32:0/1231718604 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f77d4077010 0x7f77d4079520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:55:52.139 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:52.135+0000 7f78013e5740 1 Processor -- start 2024-08-03T20:55:52.139 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:52.136+0000 7f78013e5740 1 -- start start 2024-08-03T20:55:52.142 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:52.136+0000 7f77f2a00640 1 --2- 172.21.15.32:0/1231718604 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f77d4077010 0x7f77d4079520 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7f77e0001240 tx=0x7f77e0001e00 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:55:52.142 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:52.137+0000 7f78013e5740 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x5646a4e71270 0x5646a525be90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:55:52.142 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:52.137+0000 7f78013e5740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5646a4e483b0 0x5646a5230ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:55:52.142 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:52.137+0000 7f78013e5740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5646a4e50590 0x5646a5231220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:55:52.142 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:52.137+0000 7f78013e5740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x5646a5189a60 con 0x5646a4e483b0 2024-08-03T20:55:52.142 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:52.137+0000 7f78013e5740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x5646a5178f60 con 0x5646a4e50590 2024-08-03T20:55:52.142 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:52.137+0000 7f78013e5740 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x5646a517cfa0 con 0x5646a4e71270 2024-08-03T20:55:52.142 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:52.137+0000 7f77f2a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5646a4e483b0 0x5646a5230ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:55:52.143 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:52.137+0000 7f77f2a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5646a4e483b0 0x5646a5230ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:50582/0 (socket says 172.21.15.32:50582) 2024-08-03T20:55:52.143 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:52.137+0000 7f77f2a00640 1 -- 172.21.15.32:0/611711463 learned_addr learned my addr 172.21.15.32:0/611711463 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:55:52.143 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:52.138+0000 7f77f3e00640 1 --2- 172.21.15.32:0/611711463 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5646a4e50590 0x5646a5231220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:55:52.143 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:52.138+0000 7f77f2a00640 1 -- 172.21.15.32:0/611711463 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x5646a4e71270 msgr2=0x5646a525be90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:55:52.143 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:52.138+0000 7f77f2a00640 1 --2- 172.21.15.32:0/611711463 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x5646a4e71270 0x5646a525be90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:52.143 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:52.138+0000 7f77f2a00640 1 -- 172.21.15.32:0/611711463 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5646a4e50590 msgr2=0x5646a5231220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:55:52.143 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:52.138+0000 7f77f2a00640 1 --2- 172.21.15.32:0/611711463 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5646a4e50590 0x5646a5231220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:52.143 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:52.138+0000 7f77f2a00640 1 -- 172.21.15.32:0/611711463 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5646a4edc850 con 0x5646a4e483b0 2024-08-03T20:55:52.143 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:52.138+0000 7f77f3e00640 1 --2- 172.21.15.32:0/611711463 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5646a4e50590 0x5646a5231220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-03T20:55:52.143 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:52.138+0000 7f77f2a00640 1 --2- 172.21.15.32:0/611711463 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5646a4e483b0 0x5646a5230ce0 secure :-1 s=READY pgs=1251 cs=0 l=1 rev1=1 crypto rx=0x7f77f4072880 tx=0x7f77e000d000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=84a9f0e0cf64d604 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:55:52.143 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:52.138+0000 7f77dfe00640 1 -- 172.21.15.32:0/611711463 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77e0025070 con 0x5646a4e483b0 2024-08-03T20:55:52.144 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:52.138+0000 7f77dfe00640 1 -- 172.21.15.32:0/611711463 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f77e0004370 con 0x5646a4e483b0 2024-08-03T20:55:52.144 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:52.138+0000 7f77dfe00640 1 -- 172.21.15.32:0/611711463 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77e0003b70 con 0x5646a4e483b0 2024-08-03T20:55:52.144 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:52.139+0000 7f78013e5740 1 -- 172.21.15.32:0/611711463 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5646a4e483b0 msgr2=0x5646a5230ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:55:52.144 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:52.139+0000 7f78013e5740 1 --2- 172.21.15.32:0/611711463 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5646a4e483b0 0x5646a5230ce0 secure :-1 s=READY pgs=1251 cs=0 l=1 rev1=1 crypto rx=0x7f77f4072880 tx=0x7f77e000d000 comp rx=0 tx=0).stop 2024-08-03T20:55:52.144 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:52.139+0000 7f78013e5740 1 -- 172.21.15.32:0/611711463 shutdown_connections 2024-08-03T20:55:52.144 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:52.139+0000 7f78013e5740 1 --2- 172.21.15.32:0/611711463 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5646a4e50590 0x5646a5231220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:52.144 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:52.139+0000 7f78013e5740 1 --2- 172.21.15.32:0/611711463 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5646a4e483b0 0x5646a5230ce0 unknown :-1 s=CLOSED pgs=1251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:52.144 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:52.139+0000 7f78013e5740 1 --2- 172.21.15.32:0/611711463 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x5646a4e71270 0x5646a525be90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:52.144 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:52.139+0000 7f78013e5740 1 -- 172.21.15.32:0/611711463 >> 172.21.15.32:0/611711463 conn(0x5646a4e653c0 msgr2=0x5646a5217180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:55:52.144 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:52.139+0000 7f78013e5740 1 -- 172.21.15.32:0/611711463 shutdown_connections 2024-08-03T20:55:52.145 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:52.139+0000 7f78013e5740 1 -- 172.21.15.32:0/611711463 wait complete. 2024-08-03T20:55:52.145 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:52.140+0000 7f78013e5740 1 -- start start 2024-08-03T20:55:52.145 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:52.141+0000 7f78013e5740 1 Processor -- start 2024-08-03T20:55:52.148 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:52.142+0000 7f78013e5740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5646a4e483b0 0x5646a522aa60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:55:52.149 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:52.142+0000 7f78013e5740 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x5646a4e50590 0x5646a522b170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:55:52.149 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:52.142+0000 7f78013e5740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5646a4e71270 0x5646a522ea60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:55:52.149 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:52.142+0000 7f78013e5740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x5646a4eeb620 con 0x5646a4e483b0 2024-08-03T20:55:52.149 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:52.142+0000 7f78013e5740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x5646a5116160 con 0x5646a4e71270 2024-08-03T20:55:52.149 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:52.142+0000 7f78013e5740 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x5646a522bc60 con 0x5646a4e50590 2024-08-03T20:55:52.149 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:52.142+0000 7f78013e5740 10 client.0 _ll_register_callbacks cb 0x5646a51256b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-03T20:55:52.149 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:52.142+0000 7f77f3400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5646a4e483b0 0x5646a522aa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:55:52.149 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:52.142+0000 7f77f3400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5646a4e483b0 0x5646a522aa60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:50592/0 (socket says 172.21.15.32:50592) 2024-08-03T20:55:52.149 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:52.142+0000 7f77f3400640 1 -- 172.21.15.32:0/3771905380 learned_addr learned my addr 172.21.15.32:0/3771905380 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:55:52.149 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:52.143+0000 7f77f3e00640 1 --2- 172.21.15.32:0/3771905380 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5646a4e71270 0x5646a522ea60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:55:52.149 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:52.143+0000 7f77f2a00640 1 --2- 172.21.15.32:0/3771905380 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x5646a4e50590 0x5646a522b170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:55:52.149 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:52.143+0000 7f77f3400640 1 -- 172.21.15.32:0/3771905380 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x5646a4e50590 msgr2=0x5646a522b170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:55:52.149 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:52.143+0000 7f77f3400640 1 --2- 172.21.15.32:0/3771905380 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x5646a4e50590 0x5646a522b170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:52.150 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:52.143+0000 7f77f3400640 1 -- 172.21.15.32:0/3771905380 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5646a4e71270 msgr2=0x5646a522ea60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:55:52.150 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:52.143+0000 7f77f3400640 1 --2- 172.21.15.32:0/3771905380 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5646a4e71270 0x5646a522ea60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:52.150 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:52.143+0000 7f77f3400640 1 -- 172.21.15.32:0/3771905380 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5646a4f536b0 con 0x5646a4e483b0 2024-08-03T20:55:52.150 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:52.143+0000 7f77f3e00640 1 --2- 172.21.15.32:0/3771905380 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5646a4e71270 0x5646a522ea60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-03T20:55:52.150 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:52.143+0000 7f77f3400640 1 --2- 172.21.15.32:0/3771905380 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5646a4e483b0 0x5646a522aa60 secure :-1 s=READY pgs=1252 cs=0 l=1 rev1=1 crypto rx=0x7f77e407da10 tx=0x7f77e4081850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f2bc7a5d1d1cc101 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:55:52.150 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:52.143+0000 7f77dcc00640 10 client.0 ms_handle_connect on v2:172.21.15.32:3300/0 2024-08-03T20:55:52.150 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:52.143+0000 7f77dcc00640 1 -- 172.21.15.32:0/3771905380 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77e409b060 con 0x5646a4e483b0 2024-08-03T20:55:52.150 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:52.143+0000 7f77dcc00640 1 -- 172.21.15.32:0/3771905380 <== mon.0 v2:172.21.15.32:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 6456+0+0 (secure 0 0 0) 0x7f77e40a32d0 con 0x5646a4e483b0 2024-08-03T20:55:52.150 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:52.143+0000 7f78013e5740 1 -- 172.21.15.32:0/3771905380 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f77e0004370 con 0x5646a4e483b0 2024-08-03T20:55:52.150 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:52.144+0000 7f77dcc00640 1 -- 172.21.15.32:0/3771905380 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_get_version_reply(handle=1 version=137) ==== 24+0+0 (secure 0 0 0) 0x7f77e40a1050 con 0x5646a4e483b0 2024-08-03T20:55:52.150 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:52.144+0000 7f78013e5740 10 client.7917 fetch_fsmap learned FSMap version 137 2024-08-03T20:55:52.150 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:52.144+0000 7f78013e5740 1 -- 172.21.15.32:0/3771905380 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,fsmap=137,monmap=2+}) -- 0x5646a5258850 con 0x5646a4e483b0 2024-08-03T20:55:52.150 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:52.144+0000 7f77dcc00640 1 -- 172.21.15.32:0/3771905380 <== mon.0 v2:172.21.15.32:3300/0 4 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f77e0004370 con 0x5646a4e483b0 2024-08-03T20:55:52.151 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:52.144+0000 7f77dcc00640 1 -- 172.21.15.32:0/3771905380 <== mon.0 v2:172.21.15.32:3300/0 5 ==== fsmap(e 137) ==== 3502+0+0 (secure 0 0 0) 0x7f77e409cce0 con 0x5646a4e483b0 2024-08-03T20:55:52.151 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:52.144+0000 7f78013e5740 10 client.7917 fetch_fsmap finished waiting for FSMap version 137 2024-08-03T20:55:52.151 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:52.144+0000 7f78013e5740 10 client.7917 resolve_mds: resolved name 'a' to daemon mds.a 2024-08-03T20:55:52.151 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:52.144+0000 7f78013e5740 20 client.7917 populate_metadata read hostname 'smithi032' 2024-08-03T20:55:52.151 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:52.144+0000 7f78013e5740 1 --2- 172.21.15.32:0/3771905380 >> [v2:172.21.15.32:6833/2803901497,v1:172.21.15.32:6835/2803901497] conn(0x5646a4e2fe40 0x5646a52563d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:55:52.151 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:52.144+0000 7f78013e5740 4 client.7917 mds_command: new command op to 7857 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-08-03T20:55:52.151 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:52.144+0000 7f78013e5740 1 -- 172.21.15.32:0/3771905380 --> [v2:172.21.15.32:6833/2803901497,v1:172.21.15.32:6835/2803901497] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5646a518b9b0 con 0x5646a4e2fe40 2024-08-03T20:55:52.151 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:52.144+0000 7f77f2a00640 1 --2- 172.21.15.32:0/3771905380 >> [v2:172.21.15.32:6833/2803901497,v1:172.21.15.32:6835/2803901497] conn(0x5646a4e2fe40 0x5646a52563d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:55:52.151 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:52.145+0000 7f77f2a00640 1 --2- 172.21.15.32:0/3771905380 >> [v2:172.21.15.32:6833/2803901497,v1:172.21.15.32:6835/2803901497] conn(0x5646a4e2fe40 0x5646a52563d0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=79203407fdf0e6cd server_cookie=3f225cab555b5022 in_seq=0 out_seq=0 2024-08-03T20:55:52.151 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:52.145+0000 7f77dcc00640 10 client.7917 ms_handle_connect on v2:172.21.15.32:6833/2803901497 2024-08-03T20:55:52.151 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:52.146+0000 7f77dcc00640 1 -- 172.21.15.32:0/3771905380 <== mds.0 v2:172.21.15.32:6833/2803901497 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x5646a518b9b0 con 0x5646a4e2fe40 2024-08-03T20:55:52.151 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:52.146+0000 7f77dcc00640 10 client.7917 handle_command_reply: tid=0 2024-08-03T20:55:52.151 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:52.146+0000 7f78013e5740 1 client.7917 shutdown 2024-08-03T20:55:52.152 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:52.146+0000 7f78013e5740 1 -- 172.21.15.32:0/3771905380 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5646a4e483b0 msgr2=0x5646a522aa60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:55:52.152 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:52.146+0000 7f78013e5740 1 --2- 172.21.15.32:0/3771905380 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5646a4e483b0 0x5646a522aa60 secure :-1 s=READY pgs=1252 cs=0 l=1 rev1=1 crypto rx=0x7f77e407da10 tx=0x7f77e4081850 comp rx=0 tx=0).stop 2024-08-03T20:55:52.152 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:52.146+0000 7f78013e5740 1 -- 172.21.15.32:0/3771905380 shutdown_connections 2024-08-03T20:55:52.152 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:52.146+0000 7f78013e5740 1 --2- 172.21.15.32:0/3771905380 >> [v2:172.21.15.32:6833/2803901497,v1:172.21.15.32:6835/2803901497] conn(0x5646a4e2fe40 0x5646a52563d0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:52.152 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:52.146+0000 7f78013e5740 1 --2- 172.21.15.32:0/3771905380 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5646a4e71270 0x5646a522ea60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:52.152 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:52.146+0000 7f78013e5740 1 --2- 172.21.15.32:0/3771905380 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x5646a4e50590 0x5646a522b170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:52.152 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:52.146+0000 7f78013e5740 1 --2- 172.21.15.32:0/3771905380 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5646a4e483b0 0x5646a522aa60 unknown :-1 s=CLOSED pgs=1252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:52.152 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:52.146+0000 7f78013e5740 1 -- 172.21.15.32:0/3771905380 >> 172.21.15.32:0/3771905380 conn(0x5646a4e653c0 msgr2=0x5646a5211ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:55:52.152 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:52.146+0000 7f77dcc00640 0 client.7917 ms_handle_reset on v2:172.21.15.32:6833/2803901497 2024-08-03T20:55:52.152 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:52.146+0000 7f78013e5740 1 -- 172.21.15.32:0/3771905380 shutdown_connections 2024-08-03T20:55:52.152 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:52.146+0000 7f78013e5740 1 -- 172.21.15.32:0/3771905380 wait complete. 2024-08-03T20:55:52.152 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:52.147+0000 7f78013e5740 20 client.7917 trim_cache size 0 max 16384 2024-08-03T20:55:52.168 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:52.167+0000 7f78013e5740 1 Processor -- start 2024-08-03T20:55:52.168 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:52.167+0000 7f78013e5740 1 -- start start 2024-08-03T20:55:52.170 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:52.168+0000 7f78013e5740 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x5646a4e483b0 0x5646a527f120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:55:52.170 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:52.168+0000 7f78013e5740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5646a4e50590 0x5646a527f660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:55:52.171 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:52.168+0000 7f78013e5740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5646a4e71270 0x5646a523cb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:55:52.171 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:52.168+0000 7f78013e5740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f77f41ffa80 con 0x5646a4e71270 2024-08-03T20:55:52.171 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:52.168+0000 7f78013e5740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f77f41f3f70 con 0x5646a4e50590 2024-08-03T20:55:52.171 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:52.168+0000 7f78013e5740 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7f77f41421f0 con 0x5646a4e483b0 2024-08-03T20:55:52.171 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:52.168+0000 7f77f3e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5646a4e71270 0x5646a523cb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:55:52.171 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:52.168+0000 7f77f3e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5646a4e71270 0x5646a523cb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:50610/0 (socket says 172.21.15.32:50610) 2024-08-03T20:55:52.171 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:52.168+0000 7f77f3e00640 1 -- 172.21.15.32:0/4126729031 learned_addr learned my addr 172.21.15.32:0/4126729031 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:55:52.171 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:52.168+0000 7f77f3400640 1 --2- 172.21.15.32:0/4126729031 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x5646a4e483b0 0x5646a527f120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:55:52.171 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:52.169+0000 7f77f2a00640 1 --2- 172.21.15.32:0/4126729031 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5646a4e50590 0x5646a527f660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:55:52.171 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:52.169+0000 7f77f3e00640 1 -- 172.21.15.32:0/4126729031 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x5646a4e483b0 msgr2=0x5646a527f120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:55:52.171 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:52.169+0000 7f77f3e00640 1 --2- 172.21.15.32:0/4126729031 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x5646a4e483b0 0x5646a527f120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:52.171 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:52.169+0000 7f77f3e00640 1 -- 172.21.15.32:0/4126729031 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5646a4e50590 msgr2=0x5646a527f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:55:52.172 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:52.169+0000 7f77f3e00640 1 --2- 172.21.15.32:0/4126729031 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5646a4e50590 0x5646a527f660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:52.172 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:52.169+0000 7f77f3e00640 1 -- 172.21.15.32:0/4126729031 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5646a523eb40 con 0x5646a4e71270 2024-08-03T20:55:52.172 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:52.169+0000 7f77f3e00640 1 --2- 172.21.15.32:0/4126729031 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5646a4e71270 0x5646a523cb00 secure :-1 s=READY pgs=1253 cs=0 l=1 rev1=1 crypto rx=0x7f77ec009f10 tx=0x7f77ec003d40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4d5ac0e94cf495f1 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:55:52.172 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:52.169+0000 7f77df400640 1 -- 172.21.15.32:0/4126729031 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77ec004520 con 0x5646a4e71270 2024-08-03T20:55:52.172 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:52.169+0000 7f77df400640 1 -- 172.21.15.32:0/4126729031 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f77ec0046c0 con 0x5646a4e71270 2024-08-03T20:55:52.172 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:52.169+0000 7f77df400640 1 -- 172.21.15.32:0/4126729031 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77ec0126b0 con 0x5646a4e71270 2024-08-03T20:55:52.173 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:52.170+0000 7f78013e5740 1 -- 172.21.15.32:0/4126729031 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5646a4e71270 msgr2=0x5646a523cb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:55:52.173 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:52.170+0000 7f78013e5740 1 --2- 172.21.15.32:0/4126729031 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5646a4e71270 0x5646a523cb00 secure :-1 s=READY pgs=1253 cs=0 l=1 rev1=1 crypto rx=0x7f77ec009f10 tx=0x7f77ec003d40 comp rx=0 tx=0).stop 2024-08-03T20:55:52.173 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:52.170+0000 7f78013e5740 1 -- 172.21.15.32:0/4126729031 shutdown_connections 2024-08-03T20:55:52.173 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:52.170+0000 7f78013e5740 1 --2- 172.21.15.32:0/4126729031 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5646a4e50590 0x5646a527f660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:52.173 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:52.170+0000 7f78013e5740 1 --2- 172.21.15.32:0/4126729031 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5646a4e71270 0x5646a523cb00 unknown :-1 s=CLOSED pgs=1253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:52.173 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:52.170+0000 7f78013e5740 1 --2- 172.21.15.32:0/4126729031 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x5646a4e483b0 0x5646a527f120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:52.173 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:52.170+0000 7f78013e5740 1 -- 172.21.15.32:0/4126729031 >> 172.21.15.32:0/4126729031 conn(0x5646a4e653c0 msgr2=0x5646a5241460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:55:52.173 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:52.170+0000 7f78013e5740 1 -- 172.21.15.32:0/4126729031 shutdown_connections 2024-08-03T20:55:52.174 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:52.170+0000 7f78013e5740 1 -- 172.21.15.32:0/4126729031 wait complete. 2024-08-03T20:55:52.174 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:52.171+0000 7f78013e5740 1 -- start start 2024-08-03T20:55:52.174 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:52.171+0000 7f78013e5740 1 Processor -- start 2024-08-03T20:55:52.176 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:52.173+0000 7f78013e5740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5646a4e483b0 0x5646a5257050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:55:52.177 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:52.173+0000 7f78013e5740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5646a4e50590 0x5646a52cf730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:55:52.177 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:52.173+0000 7f78013e5740 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x5646a4e71270 0x5646a5275120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:55:52.177 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:52.173+0000 7f78013e5740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x5646a522efa0 con 0x5646a4e483b0 2024-08-03T20:55:52.177 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:52.173+0000 7f78013e5740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x5646a4eeb620 con 0x5646a4e50590 2024-08-03T20:55:52.177 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:52.173+0000 7f78013e5740 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x5646a522c860 con 0x5646a4e71270 2024-08-03T20:55:52.177 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:52.173+0000 7f78013e5740 10 client.0 _ll_register_callbacks cb 0x7f77e4003f20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-03T20:55:52.177 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:52.173+0000 7f77f3400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5646a4e483b0 0x5646a5257050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:55:52.177 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:52.173+0000 7f77f3400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5646a4e483b0 0x5646a5257050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:50624/0 (socket says 172.21.15.32:50624) 2024-08-03T20:55:52.177 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:52.173+0000 7f77f3400640 1 -- 172.21.15.32:0/76256469 learned_addr learned my addr 172.21.15.32:0/76256469 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:55:52.177 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:52.173+0000 7f77f2a00640 1 --2- 172.21.15.32:0/76256469 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5646a4e50590 0x5646a52cf730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:55:52.177 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:52.173+0000 7f77f3e00640 1 --2- 172.21.15.32:0/76256469 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x5646a4e71270 0x5646a5275120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:55:52.178 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:52.173+0000 7f77f3400640 1 -- 172.21.15.32:0/76256469 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x5646a4e71270 msgr2=0x5646a5275120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:55:52.178 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:52.173+0000 7f77f3400640 1 --2- 172.21.15.32:0/76256469 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x5646a4e71270 0x5646a5275120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:52.178 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:52.173+0000 7f77f3400640 1 -- 172.21.15.32:0/76256469 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5646a4e50590 msgr2=0x5646a52cf730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:55:52.178 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:52.173+0000 7f77f3400640 1 --2- 172.21.15.32:0/76256469 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5646a4e50590 0x5646a52cf730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:52.178 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:52.173+0000 7f77f3400640 1 -- 172.21.15.32:0/76256469 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5646a522f350 con 0x5646a4e483b0 2024-08-03T20:55:52.178 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:52.174+0000 7f77f3e00640 1 --2- 172.21.15.32:0/76256469 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x5646a4e71270 0x5646a5275120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-03T20:55:52.178 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:52.174+0000 7f77f3400640 1 --2- 172.21.15.32:0/76256469 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5646a4e483b0 0x5646a5257050 secure :-1 s=READY pgs=1254 cs=0 l=1 rev1=1 crypto rx=0x7f77e4006b40 tx=0x7f77e4088450 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7cecb8703ed0aa68 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:55:52.178 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:52.174+0000 7f77dcc00640 10 client.0 ms_handle_connect on v2:172.21.15.32:3300/0 2024-08-03T20:55:52.178 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:52.174+0000 7f77dcc00640 1 -- 172.21.15.32:0/76256469 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77e409b060 con 0x5646a4e483b0 2024-08-03T20:55:52.178 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:52.174+0000 7f77dcc00640 1 -- 172.21.15.32:0/76256469 <== mon.0 v2:172.21.15.32:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 6456+0+0 (secure 0 0 0) 0x7f77e40a3240 con 0x5646a4e483b0 2024-08-03T20:55:52.178 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:52.174+0000 7f78013e5740 1 -- 172.21.15.32:0/76256469 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f77ec0046c0 con 0x5646a4e483b0 2024-08-03T20:55:52.178 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:52.174+0000 7f77dcc00640 1 -- 172.21.15.32:0/76256469 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_get_version_reply(handle=1 version=137) ==== 24+0+0 (secure 0 0 0) 0x7f77e4091050 con 0x5646a4e483b0 2024-08-03T20:55:52.179 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:52.174+0000 7f78013e5740 10 client.7923 fetch_fsmap learned FSMap version 137 2024-08-03T20:55:52.179 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:52.174+0000 7f78013e5740 1 -- 172.21.15.32:0/76256469 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,fsmap=137,monmap=2+}) -- 0x5646a52d0080 con 0x5646a4e483b0 2024-08-03T20:55:52.179 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:52.174+0000 7f77dcc00640 1 -- 172.21.15.32:0/76256469 <== mon.0 v2:172.21.15.32:3300/0 4 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f77ec0046c0 con 0x5646a4e483b0 2024-08-03T20:55:52.179 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:52.175+0000 7f77dcc00640 1 -- 172.21.15.32:0/76256469 <== mon.0 v2:172.21.15.32:3300/0 5 ==== fsmap(e 137) ==== 3502+0+0 (secure 0 0 0) 0x7f77e407d300 con 0x5646a4e483b0 2024-08-03T20:55:52.179 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:52.175+0000 7f78013e5740 10 client.7923 fetch_fsmap finished waiting for FSMap version 137 2024-08-03T20:55:52.179 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:52.175+0000 7f78013e5740 10 client.7923 resolve_mds: resolved name 'a' to daemon mds.a 2024-08-03T20:55:52.179 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:52.175+0000 7f78013e5740 20 client.7923 populate_metadata read hostname 'smithi032' 2024-08-03T20:55:52.179 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:52.175+0000 7f78013e5740 1 --2- 172.21.15.32:0/76256469 >> [v2:172.21.15.32:6833/2803901497,v1:172.21.15.32:6835/2803901497] conn(0x5646a522af60 0x5646a5254e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:55:52.179 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:52.175+0000 7f78013e5740 4 client.7923 mds_command: new command op to 7857 tid=0 multi_id=0 [{"prefix": "status"}] 2024-08-03T20:55:52.179 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:52.175+0000 7f78013e5740 1 -- 172.21.15.32:0/76256469 --> [v2:172.21.15.32:6833/2803901497,v1:172.21.15.32:6835/2803901497] -- command(tid 0: {"prefix": "status"}) -- 0x5646a5187450 con 0x5646a522af60 2024-08-03T20:55:52.179 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:52.175+0000 7f77f2a00640 1 --2- 172.21.15.32:0/76256469 >> [v2:172.21.15.32:6833/2803901497,v1:172.21.15.32:6835/2803901497] conn(0x5646a522af60 0x5646a5254e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:55:52.181 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:52.175+0000 7f77f2a00640 1 --2- 172.21.15.32:0/76256469 >> [v2:172.21.15.32:6833/2803901497,v1:172.21.15.32:6835/2803901497] conn(0x5646a522af60 0x5646a5254e20 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=63e212098e97cf3 server_cookie=3968521a7a2c0958 in_seq=0 out_seq=0 2024-08-03T20:55:52.181 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:52.175+0000 7f77dcc00640 10 client.7923 ms_handle_connect on v2:172.21.15.32:6833/2803901497 2024-08-03T20:55:52.181 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:52.176+0000 7f77dcc00640 1 -- 172.21.15.32:0/76256469 <== mds.0 v2:172.21.15.32:6833/2803901497 1 ==== command_reply(tid 0: 0 ) ==== 8+0+332 (crc 0 0 0) 0x5646a5187450 con 0x5646a522af60 2024-08-03T20:55:52.181 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:52.176+0000 7f77dcc00640 10 client.7923 handle_command_reply: tid=0 2024-08-03T20:55:52.181 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:52.176+0000 7f78013e5740 1 client.7923 shutdown 2024-08-03T20:55:52.181 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:52.176+0000 7f78013e5740 1 -- 172.21.15.32:0/76256469 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5646a4e483b0 msgr2=0x5646a5257050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:55:52.181 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:52.176+0000 7f78013e5740 1 --2- 172.21.15.32:0/76256469 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5646a4e483b0 0x5646a5257050 secure :-1 s=READY pgs=1254 cs=0 l=1 rev1=1 crypto rx=0x7f77e4006b40 tx=0x7f77e4088450 comp rx=0 tx=0).stop 2024-08-03T20:55:52.181 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:52.176+0000 7f78013e5740 1 -- 172.21.15.32:0/76256469 shutdown_connections 2024-08-03T20:55:52.181 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:52.176+0000 7f78013e5740 1 --2- 172.21.15.32:0/76256469 >> [v2:172.21.15.32:6833/2803901497,v1:172.21.15.32:6835/2803901497] conn(0x5646a522af60 0x5646a5254e20 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:52.181 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:52.176+0000 7f78013e5740 1 --2- 172.21.15.32:0/76256469 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5646a4e50590 0x5646a52cf730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:52.181 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:52.176+0000 7f78013e5740 1 --2- 172.21.15.32:0/76256469 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x5646a4e71270 0x5646a5275120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:52.182 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:52.176+0000 7f78013e5740 1 --2- 172.21.15.32:0/76256469 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5646a4e483b0 0x5646a5257050 unknown :-1 s=CLOSED pgs=1254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:52.182 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:52.176+0000 7f77dcc00640 0 client.7923 ms_handle_reset on v2:172.21.15.32:6833/2803901497 2024-08-03T20:55:52.182 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:52.176+0000 7f78013e5740 1 -- 172.21.15.32:0/76256469 >> 172.21.15.32:0/76256469 conn(0x5646a4e653c0 msgr2=0x5646a522bdf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:55:52.182 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:52.176+0000 7f78013e5740 1 -- 172.21.15.32:0/76256469 shutdown_connections 2024-08-03T20:55:52.182 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:52.176+0000 7f78013e5740 1 -- 172.21.15.32:0/76256469 wait complete. 2024-08-03T20:55:52.182 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:52.176+0000 7f78013e5740 20 client.7923 trim_cache size 0 max 16384 2024-08-03T20:55:52.182 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-08-03T20:55:52.182 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_fsid": "40ab141a-01d7-49b9-ae3c-2edec9876e95", 2024-08-03T20:55:52.182 INFO:teuthology.orchestra.run.smithi032.stdout: "whoami": 0, 2024-08-03T20:55:52.182 INFO:teuthology.orchestra.run.smithi032.stdout: "id": 7857, 2024-08-03T20:55:52.183 INFO:teuthology.orchestra.run.smithi032.stdout: "want_state": "up:active", 2024-08-03T20:55:52.183 INFO:teuthology.orchestra.run.smithi032.stdout: "state": "up:active", 2024-08-03T20:55:52.183 INFO:teuthology.orchestra.run.smithi032.stdout: "fs_name": "cephfs", 2024-08-03T20:55:52.183 INFO:teuthology.orchestra.run.smithi032.stdout: "rank_uptime": 7.1610929849999998, 2024-08-03T20:55:52.183 INFO:teuthology.orchestra.run.smithi032.stdout: "mdsmap_epoch": 137, 2024-08-03T20:55:52.183 INFO:teuthology.orchestra.run.smithi032.stdout: "osdmap_epoch": 92, 2024-08-03T20:55:52.183 INFO:teuthology.orchestra.run.smithi032.stdout: "osdmap_epoch_barrier": 92, 2024-08-03T20:55:52.183 INFO:teuthology.orchestra.run.smithi032.stdout: "uptime": 9.6112061329999996 2024-08-03T20:55:52.183 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-08-03T20:55:52.184 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:52.177+0000 7f77fa400640 1 -- 172.21.15.32:0/1231718604 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f77d4077010 msgr2=0x7f77d4079520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:55:52.184 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:52.177+0000 7f77fa400640 1 --2- 172.21.15.32:0/1231718604 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f77d4077010 0x7f77d4079520 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7f77e0001240 tx=0x7f77e0001e00 comp rx=0 tx=0).stop 2024-08-03T20:55:52.184 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:52.177+0000 7f77fa400640 1 -- 172.21.15.32:0/1231718604 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f77d80039e0 msgr2=0x7f77f4143ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:55:52.184 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:52.177+0000 7f77fa400640 1 --2- 172.21.15.32:0/1231718604 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f77d80039e0 0x7f77f4143ed0 secure :-1 s=READY pgs=1250 cs=0 l=1 rev1=1 crypto rx=0x7f77e40018e0 tx=0x7f77e40060f0 comp rx=0 tx=0).stop 2024-08-03T20:55:52.184 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:52.177+0000 7f77fa400640 1 -- 172.21.15.32:0/1231718604 shutdown_connections 2024-08-03T20:55:52.184 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:52.177+0000 7f77fa400640 1 --2- 172.21.15.32:0/1231718604 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f77d4077010 0x7f77d4079520 unknown :-1 s=CLOSED pgs=517 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:52.185 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:52.177+0000 7f77fa400640 1 --2- 172.21.15.32:0/1231718604 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f77f4138530 0x7f77f4138940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:52.185 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:52.177+0000 7f77fa400640 1 --2- 172.21.15.32:0/1231718604 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f77f413a940 0x7f77f413ad50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:52.185 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:52.177+0000 7f77fa400640 1 --2- 172.21.15.32:0/1231718604 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f77d80039e0 0x7f77f4143ed0 unknown :-1 s=CLOSED pgs=1250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:52.185 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:52.177+0000 7f77fa400640 1 -- 172.21.15.32:0/1231718604 >> 172.21.15.32:0/1231718604 conn(0x7f77f40b1bd0 msgr2=0x7f77f413c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:55:52.185 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:52.177+0000 7f77fa400640 1 -- 172.21.15.32:0/1231718604 shutdown_connections 2024-08-03T20:55:52.185 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:52.177+0000 7f77fa400640 1 -- 172.21.15.32:0/1231718604 wait complete. 2024-08-03T20:55:52.202 DEBUG:tasks.ceph_test_case:wait_until_true: success in 5s and 0 retries 2024-08-03T20:55:52.203 INFO:tasks.cephfs.test_damage:Daemons came up after mutation 'Delete 500.00000000', proceeding to ls 2024-08-03T20:55:52.203 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-08-03T20:55:52.431 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:52.429+0000 7fba24a00640 1 -- 172.21.15.32:0/209528035 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fba200b39a0 msgr2=0x7fba200b3db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:55:52.431 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:52.429+0000 7fba24a00640 1 --2- 172.21.15.32:0/209528035 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fba200b39a0 0x7fba200b3db0 secure :-1 s=READY pgs=1255 cs=0 l=1 rev1=1 crypto rx=0x7fba200b2840 tx=0x7fba0c02a560 comp rx=0 tx=0).stop 2024-08-03T20:55:52.431 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:52.429+0000 7fba24a00640 1 -- 172.21.15.32:0/209528035 shutdown_connections 2024-08-03T20:55:52.432 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:52.429+0000 7fba24a00640 1 --2- 172.21.15.32:0/209528035 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fba20162210 0x7fba20162620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:52.432 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:52.429+0000 7fba24a00640 1 --2- 172.21.15.32:0/209528035 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fba200b39a0 0x7fba200b3db0 unknown :-1 s=CLOSED pgs=1255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:52.432 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:52.429+0000 7fba24a00640 1 --2- 172.21.15.32:0/209528035 >> v2:172.21.15.118:3300/0 conn(0x7fba2006f720 0x7fba2006fb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:52.432 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:52.429+0000 7fba24a00640 1 -- 172.21.15.32:0/209528035 >> 172.21.15.32:0/209528035 conn(0x7fba200b1bd0 msgr2=0x7fba200b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:55:52.432 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:52.429+0000 7fba24a00640 1 -- 172.21.15.32:0/209528035 shutdown_connections 2024-08-03T20:55:52.432 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:52.429+0000 7fba24a00640 1 -- 172.21.15.32:0/209528035 wait complete. 2024-08-03T20:55:52.433 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:52.432+0000 7fba24a00640 1 Processor -- start 2024-08-03T20:55:52.433 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:52.432+0000 7fba24a00640 1 -- start start 2024-08-03T20:55:52.435 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:52.433+0000 7fba24a00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fba2006f720 0x7fba200ceec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:55:52.436 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:52.433+0000 7fba24a00640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fba200b39a0 0x7fba20153950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:55:52.436 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:52.433+0000 7fba24a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fba20162210 0x7fba201525a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:55:52.436 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:52.433+0000 7fba24a00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fba20162ce0 con 0x7fba20162210 2024-08-03T20:55:52.436 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:52.433+0000 7fba24a00640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7fba20162b60 con 0x7fba2006f720 2024-08-03T20:55:52.436 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:52.433+0000 7fba24a00640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7fba20162e60 con 0x7fba200b39a0 2024-08-03T20:55:52.436 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:52.434+0000 7fba1e800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fba20162210 0x7fba201525a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:55:52.436 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:52.434+0000 7fba1e800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fba20162210 0x7fba201525a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:50638/0 (socket says 172.21.15.32:50638) 2024-08-03T20:55:52.436 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:52.434+0000 7fba1e800640 1 -- 172.21.15.32:0/3376269524 learned_addr learned my addr 172.21.15.32:0/3376269524 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:55:52.436 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:52.434+0000 7fba1de00640 1 --2- 172.21.15.32:0/3376269524 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fba2006f720 0x7fba200ceec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:55:52.436 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:52.434+0000 7fba1d400640 1 --2- 172.21.15.32:0/3376269524 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fba200b39a0 0x7fba20153950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:55:52.436 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:52.434+0000 7fba1e800640 1 -- 172.21.15.32:0/3376269524 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fba200b39a0 msgr2=0x7fba20153950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:55:52.436 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:52.434+0000 7fba1e800640 1 --2- 172.21.15.32:0/3376269524 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fba200b39a0 0x7fba20153950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:52.436 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:52.434+0000 7fba1e800640 1 -- 172.21.15.32:0/3376269524 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fba2006f720 msgr2=0x7fba200ceec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:55:52.437 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:52.434+0000 7fba1e800640 1 --2- 172.21.15.32:0/3376269524 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fba2006f720 0x7fba200ceec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:52.437 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:52.434+0000 7fba1e800640 1 -- 172.21.15.32:0/3376269524 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fba200cde20 con 0x7fba20162210 2024-08-03T20:55:52.437 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:52.434+0000 7fba1e800640 1 --2- 172.21.15.32:0/3376269524 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fba20162210 0x7fba201525a0 secure :-1 s=READY pgs=1256 cs=0 l=1 rev1=1 crypto rx=0x7fba1800d9a0 tx=0x7fba1800de60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:55:52.438 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:52.435+0000 7fba16000640 1 -- 172.21.15.32:0/3376269524 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba18017070 con 0x7fba20162210 2024-08-03T20:55:52.438 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:52.435+0000 7fba16000640 1 -- 172.21.15.32:0/3376269524 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fba1800be00 con 0x7fba20162210 2024-08-03T20:55:52.438 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:52.435+0000 7fba16000640 1 -- 172.21.15.32:0/3376269524 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba18003bd0 con 0x7fba20162210 2024-08-03T20:55:52.438 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:52.435+0000 7fba24a00640 1 -- 172.21.15.32:0/3376269524 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fba200ce100 con 0x7fba20162210 2024-08-03T20:55:52.438 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:52.435+0000 7fba24a00640 1 -- 172.21.15.32:0/3376269524 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fba200ce6c0 con 0x7fba20162210 2024-08-03T20:55:52.438 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:52.436+0000 7fba24a00640 1 -- 172.21.15.32:0/3376269524 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fba200b3db0 con 0x7fba20162210 2024-08-03T20:55:52.438 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:52.437+0000 7fba16000640 1 -- 172.21.15.32:0/3376269524 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7fba18010460 con 0x7fba20162210 2024-08-03T20:55:52.439 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:52.438+0000 7fba16000640 1 --2- 172.21.15.32:0/3376269524 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fba00077010 0x7fba00079520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:55:52.439 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:52.438+0000 7fba16000640 1 -- 172.21.15.32:0/3376269524 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 6456+0+0 (secure 0 0 0) 0x7fba18098f80 con 0x7fba20162210 2024-08-03T20:55:52.440 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:52.439+0000 7fba1de00640 1 --2- 172.21.15.32:0/3376269524 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fba00077010 0x7fba00079520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:55:52.440 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:52.439+0000 7fba1de00640 1 --2- 172.21.15.32:0/3376269524 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fba00077010 0x7fba00079520 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7fba10005fd0 tx=0x7fba1000b260 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:55:52.444 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:52.443+0000 7fba16000640 1 -- 172.21.15.32:0/3376269524 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7fba180617a0 con 0x7fba20162210 2024-08-03T20:55:52.689 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:52.688+0000 7fba24a00640 1 -- 172.21.15.32:0/3376269524 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7fba20074a90 con 0x7fba20162210 2024-08-03T20:55:52.690 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:52.688+0000 7fba16000640 1 -- 172.21.15.32:0/3376269524 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v137) ==== 53+0+73 (secure 0 0 0) 0x7fba18065cf0 con 0x7fba20162210 2024-08-03T20:55:52.690 INFO:teuthology.orchestra.run.smithi032.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-08-03T20:55:52.692 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:52.691+0000 7fba24a00640 1 -- 172.21.15.32:0/3376269524 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fba00077010 msgr2=0x7fba00079520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:55:52.693 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:52.691+0000 7fba24a00640 1 --2- 172.21.15.32:0/3376269524 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fba00077010 0x7fba00079520 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7fba10005fd0 tx=0x7fba1000b260 comp rx=0 tx=0).stop 2024-08-03T20:55:52.693 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:52.691+0000 7fba24a00640 1 -- 172.21.15.32:0/3376269524 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fba20162210 msgr2=0x7fba201525a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:55:52.693 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:52.691+0000 7fba24a00640 1 --2- 172.21.15.32:0/3376269524 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fba20162210 0x7fba201525a0 secure :-1 s=READY pgs=1256 cs=0 l=1 rev1=1 crypto rx=0x7fba1800d9a0 tx=0x7fba1800de60 comp rx=0 tx=0).stop 2024-08-03T20:55:52.693 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:52.691+0000 7fba24a00640 1 -- 172.21.15.32:0/3376269524 shutdown_connections 2024-08-03T20:55:52.693 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:52.691+0000 7fba24a00640 1 --2- 172.21.15.32:0/3376269524 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fba00077010 0x7fba00079520 unknown :-1 s=CLOSED pgs=518 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:52.693 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:52.691+0000 7fba24a00640 1 --2- 172.21.15.32:0/3376269524 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fba20162210 0x7fba201525a0 unknown :-1 s=CLOSED pgs=1256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:52.693 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:52.691+0000 7fba24a00640 1 --2- 172.21.15.32:0/3376269524 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fba200b39a0 0x7fba20153950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:52.693 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:52.691+0000 7fba24a00640 1 --2- 172.21.15.32:0/3376269524 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fba2006f720 0x7fba200ceec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:52.693 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:52.691+0000 7fba24a00640 1 -- 172.21.15.32:0/3376269524 >> 172.21.15.32:0/3376269524 conn(0x7fba200b1bd0 msgr2=0x7fba20073e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:55:52.693 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:52.691+0000 7fba24a00640 1 -- 172.21.15.32:0/3376269524 shutdown_connections 2024-08-03T20:55:52.693 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:52.691+0000 7fba24a00640 1 -- 172.21.15.32:0/3376269524 wait complete. 2024-08-03T20:55:52.715 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-08-03T20:55:52.715 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-08-03T20:55:52.715 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi032.front.sepia.ceph.com 2024-08-03T20:55:52.715 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-08-03T20:55:52.715 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-08-03T20:55:52.715 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-08-03T20:55:52.716 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2024-08-03T20:55:52.716 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-08-03T20:55:52.716 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-08-03T20:55:52.716 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-08-03T20:55:52.716 DEBUG:teuthology.orchestra.run.smithi032:> ip addr 2024-08-03T20:55:52.751 INFO:teuthology.orchestra.run.smithi032.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-08-03T20:55:52.751 INFO:teuthology.orchestra.run.smithi032.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-08-03T20:55:52.752 INFO:teuthology.orchestra.run.smithi032.stdout: inet 127.0.0.1/8 scope host lo 2024-08-03T20:55:52.752 INFO:teuthology.orchestra.run.smithi032.stdout: valid_lft forever preferred_lft forever 2024-08-03T20:55:52.752 INFO:teuthology.orchestra.run.smithi032.stdout: inet6 ::1/128 scope host 2024-08-03T20:55:52.752 INFO:teuthology.orchestra.run.smithi032.stdout: valid_lft forever preferred_lft forever 2024-08-03T20:55:52.752 INFO:teuthology.orchestra.run.smithi032.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-08-03T20:55:52.752 INFO:teuthology.orchestra.run.smithi032.stdout: link/ether 0c:c4:7a:6b:fd:0c brd ff:ff:ff:ff:ff:ff 2024-08-03T20:55:52.752 INFO:teuthology.orchestra.run.smithi032.stdout: altname enp7s0f0 2024-08-03T20:55:52.752 INFO:teuthology.orchestra.run.smithi032.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-08-03T20:55:52.752 INFO:teuthology.orchestra.run.smithi032.stdout: link/ether 0c:c4:7a:6b:fd:0d brd ff:ff:ff:ff:ff:ff 2024-08-03T20:55:52.752 INFO:teuthology.orchestra.run.smithi032.stdout: altname enp7s0f1 2024-08-03T20:55:52.752 INFO:teuthology.orchestra.run.smithi032.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-08-03T20:55:52.752 INFO:teuthology.orchestra.run.smithi032.stdout: link/ether ac:1f:6b:f8:26:02 brd ff:ff:ff:ff:ff:ff 2024-08-03T20:55:52.752 INFO:teuthology.orchestra.run.smithi032.stdout: altname enp1s0f0 2024-08-03T20:55:52.752 INFO:teuthology.orchestra.run.smithi032.stdout: inet 172.21.15.32/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-08-03T20:55:52.753 INFO:teuthology.orchestra.run.smithi032.stdout: valid_lft 42003sec preferred_lft 42003sec 2024-08-03T20:55:52.753 INFO:teuthology.orchestra.run.smithi032.stdout: inet6 fe80::ae1f:6bff:fef8:2602/64 scope link noprefixroute 2024-08-03T20:55:52.753 INFO:teuthology.orchestra.run.smithi032.stdout: valid_lft forever preferred_lft forever 2024-08-03T20:55:52.753 INFO:teuthology.orchestra.run.smithi032.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-08-03T20:55:52.753 INFO:teuthology.orchestra.run.smithi032.stdout: link/ether ac:1f:6b:f8:26:03 brd ff:ff:ff:ff:ff:ff 2024-08-03T20:55:52.753 INFO:teuthology.orchestra.run.smithi032.stdout: altname enp1s0f1 2024-08-03T20:55:52.753 INFO:teuthology.orchestra.run.smithi032.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-08-03T20:55:52.753 INFO:teuthology.orchestra.run.smithi032.stdout: link/ether 0a:23:2b:ca:2a:93 brd ff:ff:ff:ff:ff:ff 2024-08-03T20:55:52.753 INFO:teuthology.orchestra.run.smithi032.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-08-03T20:55:52.753 INFO:teuthology.orchestra.run.smithi032.stdout: valid_lft forever preferred_lft forever 2024-08-03T20:55:52.753 INFO:teuthology.orchestra.run.smithi032.stdout: inet6 fe80::d804:7cff:fed9:7ba4/64 scope link 2024-08-03T20:55:52.753 INFO:teuthology.orchestra.run.smithi032.stdout: valid_lft forever preferred_lft forever 2024-08-03T20:55:52.753 INFO:teuthology.orchestra.run.smithi032.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-08-03T20:55:52.754 INFO:teuthology.orchestra.run.smithi032.stdout: link/ether 22:86:28:30:4e:62 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-08-03T20:55:52.754 INFO:teuthology.orchestra.run.smithi032.stdout: inet6 fe80::2086:28ff:fe30:4e62/64 scope link 2024-08-03T20:55:52.754 INFO:teuthology.orchestra.run.smithi032.stdout: valid_lft forever preferred_lft forever 2024-08-03T20:55:52.754 INFO:teuthology.orchestra.run.smithi032.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-08-03T20:55:52.754 INFO:teuthology.orchestra.run.smithi032.stdout: link/ether 0a:23:2b:ca:2a:93 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-08-03T20:55:52.754 INFO:teuthology.orchestra.run.smithi032.stdout: inet6 fe80::823:2bff:feca:2a93/64 scope link 2024-08-03T20:55:52.754 INFO:teuthology.orchestra.run.smithi032.stdout: valid_lft forever preferred_lft forever 2024-08-03T20:55:52.754 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-08-03T20:55:52.754 DEBUG:teuthology.orchestra.run.smithi032:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-08-03T20:55:52.844 INFO:teuthology.orchestra.run.smithi032.stdout:1 2024-08-03T20:55:52.847 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-08-03T20:55:52.848 DEBUG:teuthology.orchestra.run.smithi032:> route 2024-08-03T20:55:52.884 INFO:teuthology.orchestra.run.smithi032.stdout:Kernel IP routing table 2024-08-03T20:55:52.884 INFO:teuthology.orchestra.run.smithi032.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-08-03T20:55:52.884 INFO:teuthology.orchestra.run.smithi032.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-08-03T20:55:52.884 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-08-03T20:55:52.884 INFO:teuthology.orchestra.run.smithi032.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-08-03T20:55:52.884 DEBUG:teuthology.orchestra.run.smithi032:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-08-03T20:55:52.884 DEBUG:teuthology.orchestra.run.smithi032:> set -e 2024-08-03T20:55:52.885 DEBUG:teuthology.orchestra.run.smithi032:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-08-03T20:55:52.885 DEBUG:teuthology.orchestra.run.smithi032:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-08-03T20:55:52.885 DEBUG:teuthology.orchestra.run.smithi032:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-08-03T20:55:52.885 DEBUG:teuthology.orchestra.run.smithi032:> ') 2024-08-03T20:55:53.003 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:53 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-08-03T20:55:53.077 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:53 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-08-03T20:55:53.086 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-08-03T20:55:53.086 DEBUG:teuthology.orchestra.run.smithi032:> ip netns list 2024-08-03T20:55:53.122 INFO:teuthology.orchestra.run.smithi032.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-08-03T20:55:53.122 INFO:teuthology.orchestra.run.smithi032.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-08-03T20:55:53.122 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-08-03T20:55:53.122 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-08-03T20:55:53.123 DEBUG:teuthology.orchestra.run.smithi032:> sudo ip link set brx.0 up 2024-08-03T20:55:53.218 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-08-03T20:55:53.218 DEBUG:teuthology.orchestra.run.smithi032:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-08-03T20:55:53.253 INFO:teuthology.orchestra.run.smithi032.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-08-03T20:55:53.254 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-08-03T20:55:53.254 DEBUG:teuthology.orchestra.run.smithi032:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-08-03T20:55:53.330 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2024-08-03T20:55:53.330 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-08-03T20:55:53.330 DEBUG:teuthology.orchestra.run.smithi032:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-08-03T20:55:53.530 INFO:teuthology.orchestra.run.smithi032.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-08-03T20:55:53.530 INFO:teuthology.orchestra.run.smithi032.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-08-03T20:55:53.530 INFO:teuthology.orchestra.run.smithi032.stdout:invalid new device string format 2024-08-03T20:55:53.550 INFO:teuthology.orchestra.run.smithi032.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-08-03T20:55:53.550 INFO:teuthology.orchestra.run.smithi032.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-08-03T20:55:53.551 INFO:teuthology.orchestra.run.smithi032.stdout:invalid new device string format 2024-08-03T20:55:53.551 INFO:teuthology.orchestra.run.smithi032.stdout:mount.ceph: resolved to: "172.21.15.32:3300,172.21.15.118:3300,172.21.15.118:3301" 2024-08-03T20:55:53.551 INFO:teuthology.orchestra.run.smithi032.stdout:mount.ceph: switching to using v1 address with old syntax 2024-08-03T20:55:53.580 INFO:teuthology.orchestra.run.smithi032.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-08-03T20:55:53.581 INFO:teuthology.orchestra.run.smithi032.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-08-03T20:55:53.581 INFO:teuthology.orchestra.run.smithi032.stdout:invalid new device string format 2024-08-03T20:55:53.581 INFO:teuthology.orchestra.run.smithi032.stdout:mount.ceph: resolved to: "172.21.15.32:3300,172.21.15.118:3300,172.21.15.118:3301" 2024-08-03T20:55:53.581 INFO:teuthology.orchestra.run.smithi032.stdout:mount.ceph: switching to using v1 address with old syntax 2024-08-03T20:55:53.581 INFO:teuthology.orchestra.run.smithi032.stdout:mount.ceph: resolved to: "172.21.15.32:6789,172.21.15.118:6789,172.21.15.118:6790" 2024-08-03T20:55:53.581 INFO:teuthology.orchestra.run.smithi032.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.32:6789,172.21.15.118:6789,172.21.15.118:6790:/ 2024-08-03T20:55:53.581 INFO:teuthology.orchestra.run.smithi032.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,key=0,fsid=40ab141a-01d7-49b9-ae3c-2edec9876e95" will pass to kernel 2024-08-03T20:55:53.582 INFO:teuthology.orchestra.run.smithi032.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2024-08-03T20:55:53.582 INFO:teuthology.orchestra.run.smithi032.stdout: You just mounted a file system that supports labels which does not 2024-08-03T20:55:53.582 INFO:teuthology.orchestra.run.smithi032.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-08-03T20:55:53.582 INFO:teuthology.orchestra.run.smithi032.stdout: applications will generate AVC messages and not be allowed access to 2024-08-03T20:55:53.582 INFO:teuthology.orchestra.run.smithi032.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-08-03T20:55:53.585 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-08-03T20:55:53.585 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-08-03T20:55:53.586 DEBUG:teuthology.orchestra.run.smithi032:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-08-03T20:55:53.642 DEBUG:teuthology.orchestra.run.smithi032:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-08-03T20:55:53.721 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:53 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-08-03T20:55:53.724 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:53 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-08-03T20:55:53.732 INFO:teuthology.orchestra.run.smithi032.stdout:client6752 2024-08-03T20:55:53.733 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-08-03T20:55:53.733 DEBUG:teuthology.orchestra.run.smithi032:> dd if=/proc/self/mounts of=/dev/stdout 2024-08-03T20:55:53.769 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-08-03T20:55:53.770 DEBUG:teuthology.orchestra.run.smithi032:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls -R .) 2024-08-03T20:55:53.815 INFO:teuthology.orchestra.run:waiting for 20 2024-08-03T20:55:53.847 INFO:teuthology.orchestra.run.smithi032.stdout:.: 2024-08-03T20:55:53.848 INFO:teuthology.orchestra.run.smithi032.stdout:subdir 2024-08-03T20:55:53.848 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-03T20:55:53.848 INFO:teuthology.orchestra.run.smithi032.stdout:./subdir: 2024-08-03T20:55:53.848 INFO:teuthology.orchestra.run.smithi032.stdout:sixmegs 2024-08-03T20:55:53.849 INFO:tasks.cephfs.test_damage:Result: Mutation 'Delete 500.00000000' did not caused DAMAGED state 2024-08-03T20:55:53.849 INFO:tasks.cephfs.test_damage:Applying mutation 'Delete mds_snaptable' 2024-08-03T20:55:53.849 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-08-03T20:55:53.849 DEBUG:teuthology.orchestra.run.smithi032:> dd if=/proc/self/mounts of=/dev/stdout 2024-08-03T20:55:53.924 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-08-03T20:55:53.925 DEBUG:teuthology.orchestra.run.smithi032:> dd if=/proc/self/mounts of=/dev/stdout 2024-08-03T20:55:54.000 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-08-03T20:55:54.269 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:54.267+0000 7f4c03c00640 1 -- 172.21.15.32:0/1952976554 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4be40052e0 msgr2=0x7f4be400b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:55:54.269 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:54.267+0000 7f4c03c00640 1 --2- 172.21.15.32:0/1952976554 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4be40052e0 0x7f4be400b9c0 secure :-1 s=READY pgs=1258 cs=0 l=1 rev1=1 crypto rx=0x7f4bf4005fd0 tx=0x7f4be4005830 comp rx=0 tx=0).stop 2024-08-03T20:55:54.269 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:54.267+0000 7f4c03c00640 1 -- 172.21.15.32:0/1952976554 shutdown_connections 2024-08-03T20:55:54.269 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:54.267+0000 7f4c03c00640 1 --2- 172.21.15.32:0/1952976554 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4be40052e0 0x7f4be400b9c0 unknown :-1 s=CLOSED pgs=1258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:54.270 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:54.267+0000 7f4c03c00640 1 -- 172.21.15.32:0/1952976554 >> 172.21.15.32:0/1952976554 conn(0x7f4bfc0b1bd0 msgr2=0x7f4bfc0b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:55:54.270 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:54.267+0000 7f4c03c00640 1 -- 172.21.15.32:0/1952976554 shutdown_connections 2024-08-03T20:55:54.270 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:54.267+0000 7f4c03c00640 1 -- 172.21.15.32:0/1952976554 wait complete. 2024-08-03T20:55:54.271 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:54.270+0000 7f4c03c00640 1 Processor -- start 2024-08-03T20:55:54.271 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:54.270+0000 7f4c03c00640 1 -- start start 2024-08-03T20:55:54.274 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:54.271+0000 7f4c03c00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4be40052e0 0x7f4bfc1f5590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:55:54.274 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:54.271+0000 7f4c03c00640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f4bfc159300 0x7f4bfc0cee10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:55:54.274 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:54.271+0000 7f4c03c00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f4bfc14a320 0x7f4bfc1f92d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:55:54.274 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:54.271+0000 7f4c03c00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f4bfc0b3d30 con 0x7f4be40052e0 2024-08-03T20:55:54.274 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:54.271+0000 7f4c03c00640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f4bfc0b3bb0 con 0x7f4bfc14a320 2024-08-03T20:55:54.274 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:54.271+0000 7f4c03c00640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7f4bfc0b3eb0 con 0x7f4bfc159300 2024-08-03T20:55:54.274 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:54.271+0000 7f4c01200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4be40052e0 0x7f4bfc1f5590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:55:54.275 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:54.271+0000 7f4c01200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4be40052e0 0x7f4bfc1f5590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:50664/0 (socket says 172.21.15.32:50664) 2024-08-03T20:55:54.275 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:54.271+0000 7f4c01200640 1 -- 172.21.15.32:0/1746559330 learned_addr learned my addr 172.21.15.32:0/1746559330 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:55:54.275 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:54.272+0000 7f4bfbe00640 1 --2- 172.21.15.32:0/1746559330 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f4bfc159300 0x7f4bfc0cee10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:55:54.275 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:54.272+0000 7f4c01c00640 1 --2- 172.21.15.32:0/1746559330 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f4bfc14a320 0x7f4bfc1f92d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:55:54.275 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:54.272+0000 7f4c01200640 1 -- 172.21.15.32:0/1746559330 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f4bfc159300 msgr2=0x7f4bfc0cee10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:55:54.275 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:54.272+0000 7f4c01200640 1 --2- 172.21.15.32:0/1746559330 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f4bfc159300 0x7f4bfc0cee10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:54.275 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:54.272+0000 7f4c01200640 1 -- 172.21.15.32:0/1746559330 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f4bfc14a320 msgr2=0x7f4bfc1f92d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:55:54.275 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:54.272+0000 7f4c01200640 1 --2- 172.21.15.32:0/1746559330 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f4bfc14a320 0x7f4bfc1f92d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:54.275 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:54.272+0000 7f4c01200640 1 -- 172.21.15.32:0/1746559330 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4bfc0cf350 con 0x7f4be40052e0 2024-08-03T20:55:54.275 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:54.272+0000 7f4c01200640 1 --2- 172.21.15.32:0/1746559330 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4be40052e0 0x7f4bfc1f5590 secure :-1 s=READY pgs=1259 cs=0 l=1 rev1=1 crypto rx=0x7f4bf0003d80 tx=0x7f4bf0001800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:55:54.275 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:54.272+0000 7f4bf8c00640 1 -- 172.21.15.32:0/1746559330 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4bf0011700 con 0x7f4be40052e0 2024-08-03T20:55:54.275 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:54.272+0000 7f4bf8c00640 1 -- 172.21.15.32:0/1746559330 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f4bf0016020 con 0x7f4be40052e0 2024-08-03T20:55:54.275 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:54.272+0000 7f4bf8c00640 1 -- 172.21.15.32:0/1746559330 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4bf0011b60 con 0x7f4be40052e0 2024-08-03T20:55:54.276 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:54.272+0000 7f4c03c00640 1 -- 172.21.15.32:0/1746559330 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4bfc1f5de0 con 0x7f4be40052e0 2024-08-03T20:55:54.276 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:54.273+0000 7f4c03c00640 1 -- 172.21.15.32:0/1746559330 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4bfc1f63a0 con 0x7f4be40052e0 2024-08-03T20:55:54.276 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:54.274+0000 7f4c03c00640 1 -- 172.21.15.32:0/1746559330 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4bfc162420 con 0x7f4be40052e0 2024-08-03T20:55:54.276 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:54.275+0000 7f4bf8c00640 1 -- 172.21.15.32:0/1746559330 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f4bf0011d00 con 0x7f4be40052e0 2024-08-03T20:55:54.277 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:54.276+0000 7f4bf8c00640 1 --2- 172.21.15.32:0/1746559330 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f4bdc077010 0x7f4bdc079520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:55:54.278 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:54.276+0000 7f4bf8c00640 1 -- 172.21.15.32:0/1746559330 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 6456+0+0 (secure 0 0 0) 0x7f4bf00997f0 con 0x7f4be40052e0 2024-08-03T20:55:54.278 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:54.276+0000 7f4bfbe00640 1 --2- 172.21.15.32:0/1746559330 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f4bdc077010 0x7f4bdc079520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:55:54.278 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:54.277+0000 7f4bfbe00640 1 --2- 172.21.15.32:0/1746559330 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f4bdc077010 0x7f4bdc079520 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7f4bfc0b18b0 tx=0x7f4be8001de0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:55:54.281 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:54.280+0000 7f4bf8c00640 1 -- 172.21.15.32:0/1746559330 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7f4bf0020020 con 0x7f4be40052e0 2024-08-03T20:55:54.501 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:54.500+0000 7f4c03c00640 1 -- 172.21.15.32:0/1746559330 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f4bfc074a90 con 0x7f4be40052e0 2024-08-03T20:55:54.502 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:54.501+0000 7f4bf8c00640 1 -- 172.21.15.32:0/1746559330 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 14 entries v92) ==== 81+0+852 (secure 0 0 0) 0x7f4bf0062130 con 0x7f4be40052e0 2024-08-03T20:55:54.502 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6835/3191654276 2024-08-04T20:55:42.433364+0000 2024-08-03T20:55:54.502 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6833/1707454554 2024-08-04T20:54:38.409411+0000 2024-08-03T20:55:54.502 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6835/2194902489 2024-08-04T20:55:00.359126+0000 2024-08-03T20:55:54.503 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6835/1707454554 2024-08-04T20:54:38.409411+0000 2024-08-03T20:55:54.503 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6833/2194902489 2024-08-04T20:55:00.359126+0000 2024-08-03T20:55:54.503 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6833/1016338829 2024-08-04T20:55:06.071485+0000 2024-08-03T20:55:54.503 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6835/1172050551 2024-08-04T20:55:26.214113+0000 2024-08-03T20:55:54.503 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6835/1016338829 2024-08-04T20:55:06.071485+0000 2024-08-03T20:55:54.503 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6833/756683988 2024-08-04T20:55:14.119228+0000 2024-08-03T20:55:54.503 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6835/756683988 2024-08-04T20:55:14.119228+0000 2024-08-03T20:55:54.503 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6833/2292690833 2024-08-04T20:55:20.021329+0000 2024-08-03T20:55:54.503 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6835/2292690833 2024-08-04T20:55:20.021329+0000 2024-08-03T20:55:54.503 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6833/1172050551 2024-08-04T20:55:26.214113+0000 2024-08-03T20:55:54.503 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6833/3191654276 2024-08-04T20:55:42.433364+0000 2024-08-03T20:55:54.503 INFO:teuthology.orchestra.run.smithi032.stderr:listed 14 entries 2024-08-03T20:55:54.505 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:54.503+0000 7f4c03c00640 1 -- 172.21.15.32:0/1746559330 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f4bdc077010 msgr2=0x7f4bdc079520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:55:54.505 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:54.503+0000 7f4c03c00640 1 --2- 172.21.15.32:0/1746559330 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f4bdc077010 0x7f4bdc079520 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7f4bfc0b18b0 tx=0x7f4be8001de0 comp rx=0 tx=0).stop 2024-08-03T20:55:54.505 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:54.503+0000 7f4c03c00640 1 -- 172.21.15.32:0/1746559330 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4be40052e0 msgr2=0x7f4bfc1f5590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:55:54.505 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:54.503+0000 7f4c03c00640 1 --2- 172.21.15.32:0/1746559330 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4be40052e0 0x7f4bfc1f5590 secure :-1 s=READY pgs=1259 cs=0 l=1 rev1=1 crypto rx=0x7f4bf0003d80 tx=0x7f4bf0001800 comp rx=0 tx=0).stop 2024-08-03T20:55:54.505 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:54.503+0000 7f4c03c00640 1 -- 172.21.15.32:0/1746559330 shutdown_connections 2024-08-03T20:55:54.505 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:54.503+0000 7f4c03c00640 1 --2- 172.21.15.32:0/1746559330 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f4bdc077010 0x7f4bdc079520 unknown :-1 s=CLOSED pgs=519 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:54.505 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:54.503+0000 7f4c03c00640 1 --2- 172.21.15.32:0/1746559330 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f4bfc14a320 0x7f4bfc1f92d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:54.505 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:54.503+0000 7f4c03c00640 1 --2- 172.21.15.32:0/1746559330 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f4bfc159300 0x7f4bfc0cee10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:54.505 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:54.503+0000 7f4c03c00640 1 --2- 172.21.15.32:0/1746559330 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4be40052e0 0x7f4bfc1f5590 unknown :-1 s=CLOSED pgs=1259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:54.505 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:54.503+0000 7f4c03c00640 1 -- 172.21.15.32:0/1746559330 >> 172.21.15.32:0/1746559330 conn(0x7f4bfc0b1bd0 msgr2=0x7f4bfc0afd50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:55:54.506 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:54.503+0000 7f4c03c00640 1 -- 172.21.15.32:0/1746559330 shutdown_connections 2024-08-03T20:55:54.506 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:54.503+0000 7f4c03c00640 1 -- 172.21.15.32:0/1746559330 wait complete. 2024-08-03T20:55:54.529 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-08-03T20:55:54.529 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-08-03T20:55:54.529 DEBUG:teuthology.orchestra.run.smithi032:> sudo umount /home/ubuntu/cephtest/mnt.0 -f 2024-08-03T20:55:54.610 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi032.front.sepia.ceph.com 2024-08-03T20:55:54.610 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-08-03T20:55:54.610 DEBUG:teuthology.orchestra.run.smithi032:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-08-03T20:55:54.646 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-08-03T20:55:54.915 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:54.913+0000 7f820b200640 1 -- 172.21.15.32:0/2094288678 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f820406e930 msgr2=0x7f8204144f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:55:54.915 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:54.913+0000 7f820b200640 1 --2- 172.21.15.32:0/2094288678 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f820406e930 0x7f8204144f90 secure :-1 s=READY pgs=1260 cs=0 l=1 rev1=1 crypto rx=0x7f82040afaf0 tx=0x7f81f402a4d0 comp rx=0 tx=0).stop 2024-08-03T20:55:54.915 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:54.913+0000 7f820b200640 1 -- 172.21.15.32:0/2094288678 shutdown_connections 2024-08-03T20:55:54.915 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:54.913+0000 7f820b200640 1 --2- 172.21.15.32:0/2094288678 >> v2:172.21.15.118:3300/0 conn(0x7f8204073e60 0x7f8204074270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:54.915 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:54.913+0000 7f820b200640 1 --2- 172.21.15.32:0/2094288678 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f820406e930 0x7f8204144f90 unknown :-1 s=CLOSED pgs=1260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:54.915 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:54.913+0000 7f820b200640 1 -- 172.21.15.32:0/2094288678 >> 172.21.15.32:0/2094288678 conn(0x7f82040b1bd0 msgr2=0x7f82040b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:55:54.916 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:54.913+0000 7f820b200640 1 -- 172.21.15.32:0/2094288678 shutdown_connections 2024-08-03T20:55:54.916 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:54.913+0000 7f820b200640 1 -- 172.21.15.32:0/2094288678 wait complete. 2024-08-03T20:55:54.917 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:54.915+0000 7f820b200640 1 Processor -- start 2024-08-03T20:55:54.917 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:54.916+0000 7f820b200640 1 -- start start 2024-08-03T20:55:54.920 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:54.917+0000 7f820b200640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f820406e930 0x7f820413da60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:55:54.920 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:54.917+0000 7f820b200640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f8204073e60 0x7f820414e740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:55:54.920 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:54.917+0000 7f820b200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f82040747b0 0x7f8204143d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:55:54.920 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:54.917+0000 7f820b200640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f82040cf9c0 con 0x7f82040747b0 2024-08-03T20:55:54.920 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:54.917+0000 7f820b200640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f82040cf840 con 0x7f820406e930 2024-08-03T20:55:54.921 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:54.917+0000 7f820b200640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7f82040ccff0 con 0x7f8204073e60 2024-08-03T20:55:54.921 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:54.917+0000 7f8209200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f82040747b0 0x7f8204143d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:55:54.921 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:54.917+0000 7f8209200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f82040747b0 0x7f8204143d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:50684/0 (socket says 172.21.15.32:50684) 2024-08-03T20:55:54.921 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:54.917+0000 7f8209200640 1 -- 172.21.15.32:0/1666379157 learned_addr learned my addr 172.21.15.32:0/1666379157 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:55:54.921 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:54.917+0000 7f8203e00640 1 --2- 172.21.15.32:0/1666379157 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f820406e930 0x7f820413da60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:55:54.921 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:54.917+0000 7f81fb400640 1 --2- 172.21.15.32:0/1666379157 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f8204073e60 0x7f820414e740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:55:54.921 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:54.918+0000 7f8209200640 1 -- 172.21.15.32:0/1666379157 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f8204073e60 msgr2=0x7f820414e740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:55:54.921 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:54.918+0000 7f8209200640 1 --2- 172.21.15.32:0/1666379157 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f8204073e60 0x7f820414e740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:54.921 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:54.918+0000 7f8209200640 1 -- 172.21.15.32:0/1666379157 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f820406e930 msgr2=0x7f820413da60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:55:54.921 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:54.918+0000 7f8209200640 1 --2- 172.21.15.32:0/1666379157 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f820406e930 0x7f820413da60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:54.921 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:54.918+0000 7f8209200640 1 -- 172.21.15.32:0/1666379157 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f82041fa030 con 0x7f82040747b0 2024-08-03T20:55:54.921 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:54.918+0000 7f8209200640 1 --2- 172.21.15.32:0/1666379157 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f82040747b0 0x7f8204143d50 secure :-1 s=READY pgs=1261 cs=0 l=1 rev1=1 crypto rx=0x7f81fc00b9a0 tx=0x7f81fc00be60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:55:54.921 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:54.918+0000 7f8200c00640 1 -- 172.21.15.32:0/1666379157 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f81fc019070 con 0x7f82040747b0 2024-08-03T20:55:54.922 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:54.918+0000 7f8200c00640 1 -- 172.21.15.32:0/1666379157 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f81fc00cc90 con 0x7f82040747b0 2024-08-03T20:55:54.922 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:54.918+0000 7f820b200640 1 -- 172.21.15.32:0/1666379157 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f82041f9270 con 0x7f82040747b0 2024-08-03T20:55:54.922 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:54.918+0000 7f8200c00640 1 -- 172.21.15.32:0/1666379157 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f81fc0146a0 con 0x7f82040747b0 2024-08-03T20:55:54.922 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:54.919+0000 7f820b200640 1 -- 172.21.15.32:0/1666379157 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f82041f9830 con 0x7f82040747b0 2024-08-03T20:55:54.922 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:54.920+0000 7f820b200640 1 -- 172.21.15.32:0/1666379157 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f82040cd5e0 con 0x7f82040747b0 2024-08-03T20:55:54.922 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:54.921+0000 7f8200c00640 1 -- 172.21.15.32:0/1666379157 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f81fc01c070 con 0x7f82040747b0 2024-08-03T20:55:54.924 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:54.922+0000 7f8200c00640 1 --2- 172.21.15.32:0/1666379157 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f81e0077040 0x7f81e0079550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:55:54.924 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:54.923+0000 7f8200c00640 1 -- 172.21.15.32:0/1666379157 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 6456+0+0 (secure 0 0 0) 0x7f81fc098840 con 0x7f82040747b0 2024-08-03T20:55:54.924 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:54.923+0000 7f8203e00640 1 --2- 172.21.15.32:0/1666379157 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f81e0077040 0x7f81e0079550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:55:54.925 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:54.924+0000 7f8203e00640 1 --2- 172.21.15.32:0/1666379157 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f81e0077040 0x7f81e0079550 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7f81f4033000 tx=0x7f81f4004730 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:55:54.928 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:54.927+0000 7f8200c00640 1 -- 172.21.15.32:0/1666379157 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7f81fc0610f0 con 0x7f82040747b0 2024-08-03T20:55:55.180 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:55.179+0000 7f820b200640 1 -- 172.21.15.32:0/1666379157 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7f820414f240 con 0x7f82040747b0 2024-08-03T20:55:55.181 INFO:tasks.ceph.mon.a.smithi032.stderr:2024-08-03T20:55:55.180+0000 7f4e6d600640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-08-03T20:55:55.183 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:55.181+0000 7f8200c00640 1 -- 172.21.15.32:0/1666379157 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v138) ==== 161+0+0 (secure 0 0 0) 0x7f81fc065640 con 0x7f82040747b0 2024-08-03T20:55:55.183 INFO:teuthology.orchestra.run.smithi032.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-08-03T20:55:55.187 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:55.185+0000 7f820b200640 1 -- 172.21.15.32:0/1666379157 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f81e0077040 msgr2=0x7f81e0079550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:55:55.188 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:55.185+0000 7f820b200640 1 --2- 172.21.15.32:0/1666379157 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f81e0077040 0x7f81e0079550 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7f81f4033000 tx=0x7f81f4004730 comp rx=0 tx=0).stop 2024-08-03T20:55:55.188 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:55.185+0000 7f820b200640 1 -- 172.21.15.32:0/1666379157 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f82040747b0 msgr2=0x7f8204143d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:55:55.188 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:55.185+0000 7f820b200640 1 --2- 172.21.15.32:0/1666379157 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f82040747b0 0x7f8204143d50 secure :-1 s=READY pgs=1261 cs=0 l=1 rev1=1 crypto rx=0x7f81fc00b9a0 tx=0x7f81fc00be60 comp rx=0 tx=0).stop 2024-08-03T20:55:55.188 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:55.185+0000 7f820b200640 1 -- 172.21.15.32:0/1666379157 shutdown_connections 2024-08-03T20:55:55.188 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:55.185+0000 7f820b200640 1 --2- 172.21.15.32:0/1666379157 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f81e0077040 0x7f81e0079550 unknown :-1 s=CLOSED pgs=520 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:55.188 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:55.185+0000 7f820b200640 1 --2- 172.21.15.32:0/1666379157 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f82040747b0 0x7f8204143d50 unknown :-1 s=CLOSED pgs=1261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:55.189 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:55.185+0000 7f820b200640 1 --2- 172.21.15.32:0/1666379157 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f8204073e60 0x7f820414e740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:55.189 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:55.185+0000 7f820b200640 1 --2- 172.21.15.32:0/1666379157 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f820406e930 0x7f820413da60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:55.189 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:55.185+0000 7f820b200640 1 -- 172.21.15.32:0/1666379157 >> 172.21.15.32:0/1666379157 conn(0x7f82040b1bd0 msgr2=0x7f82040b3950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:55:55.189 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:55.185+0000 7f820b200640 1 -- 172.21.15.32:0/1666379157 shutdown_connections 2024-08-03T20:55:55.189 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:55.186+0000 7f820b200640 1 -- 172.21.15.32:0/1666379157 wait complete. 2024-08-03T20:55:55.191 INFO:tasks.ceph.mds.a.smithi032.stderr: -753> 2024-08-03T20:55:48.446+0000 7f4ec5a00640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-08-03T20:55:55.211 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mds repaired 0 2024-08-03T20:55:55.309 INFO:tasks.ceph.mds.a.smithi032.stdout:starting mds.a at 2024-08-03T20:55:55.445 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:55.443+0000 7fb2ac800640 1 -- 172.21.15.32:0/1578697382 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb2a415fff0 msgr2=0x7fb2a4160400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:55:55.445 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:55.443+0000 7fb2ac800640 1 --2- 172.21.15.32:0/1578697382 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb2a415fff0 0x7fb2a4160400 secure :-1 s=READY pgs=1264 cs=0 l=1 rev1=1 crypto rx=0x7fb2a0005fd0 tx=0x7fb2a002aa90 comp rx=0 tx=0).stop 2024-08-03T20:55:55.446 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:55.443+0000 7fb2ac800640 1 -- 172.21.15.32:0/1578697382 shutdown_connections 2024-08-03T20:55:55.446 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:55.443+0000 7fb2ac800640 1 --2- 172.21.15.32:0/1578697382 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb2a415fff0 0x7fb2a4160400 unknown :-1 s=CLOSED pgs=1264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:55.446 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:55.443+0000 7fb2ac800640 1 --2- 172.21.15.32:0/1578697382 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fb2a406e930 0x7fb2a4144f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:55.446 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:55.443+0000 7fb2ac800640 1 -- 172.21.15.32:0/1578697382 >> 172.21.15.32:0/1578697382 conn(0x7fb2a40b1bd0 msgr2=0x7fb2a40b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:55:55.446 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:55.444+0000 7fb2ac800640 1 -- 172.21.15.32:0/1578697382 shutdown_connections 2024-08-03T20:55:55.446 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:55.444+0000 7fb2ac800640 1 -- 172.21.15.32:0/1578697382 wait complete. 2024-08-03T20:55:55.448 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:55.446+0000 7fb2ac800640 1 Processor -- start 2024-08-03T20:55:55.448 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:55.446+0000 7fb2ac800640 1 -- start start 2024-08-03T20:55:55.449 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:55.448+0000 7fb2ac800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb2a406e930 0x7fb2a413a030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:55:55.450 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:55.448+0000 7fb2ac800640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb2a4073e60 0x7fb2a41f3330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:55:55.450 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:55.448+0000 7fb2ac800640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fb2a415fff0 0x7fb2a41f3870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:55:55.450 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:55.448+0000 7fb2ac800640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fb2a40cf840 con 0x7fb2a406e930 2024-08-03T20:55:55.450 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:55.448+0000 7fb2ac800640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7fb2a4145f30 con 0x7fb2a4073e60 2024-08-03T20:55:55.450 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:55.448+0000 7fb2ac800640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7fb2a40cf9c0 con 0x7fb2a415fff0 2024-08-03T20:55:55.450 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:55.448+0000 7fb2a9e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb2a406e930 0x7fb2a413a030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:55:55.450 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:55.448+0000 7fb2a9e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb2a406e930 0x7fb2a413a030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:50718/0 (socket says 172.21.15.32:50718) 2024-08-03T20:55:55.450 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:55.448+0000 7fb2a9e00640 1 -- 172.21.15.32:0/325765240 learned_addr learned my addr 172.21.15.32:0/325765240 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:55:55.450 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:55.448+0000 7fb2a9400640 1 --2- 172.21.15.32:0/325765240 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb2a4073e60 0x7fb2a41f3330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:55:55.450 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:55.448+0000 7fb2aa800640 1 --2- 172.21.15.32:0/325765240 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fb2a415fff0 0x7fb2a41f3870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:55:55.452 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:55.449+0000 7fb2a9e00640 1 -- 172.21.15.32:0/325765240 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fb2a415fff0 msgr2=0x7fb2a41f3870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:55:55.452 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:55.449+0000 7fb2a9e00640 1 --2- 172.21.15.32:0/325765240 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fb2a415fff0 0x7fb2a41f3870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:55.452 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:55.449+0000 7fb2a9e00640 1 -- 172.21.15.32:0/325765240 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb2a4073e60 msgr2=0x7fb2a41f3330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:55:55.452 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:55.449+0000 7fb2a9e00640 1 --2- 172.21.15.32:0/325765240 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb2a4073e60 0x7fb2a41f3330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:55.452 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:55.449+0000 7fb2a9e00640 1 -- 172.21.15.32:0/325765240 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb2a41392d0 con 0x7fb2a406e930 2024-08-03T20:55:55.452 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:55.449+0000 7fb2a9e00640 1 --2- 172.21.15.32:0/325765240 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb2a406e930 0x7fb2a413a030 secure :-1 s=READY pgs=1265 cs=0 l=1 rev1=1 crypto rx=0x7fb29800d740 tx=0x7fb29800dc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:55:55.452 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:55.449+0000 7fb29e000640 1 -- 172.21.15.32:0/325765240 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb298017070 con 0x7fb2a406e930 2024-08-03T20:55:55.452 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:55.449+0000 7fb29e000640 1 -- 172.21.15.32:0/325765240 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fb298009cc0 con 0x7fb2a406e930 2024-08-03T20:55:55.452 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:55.449+0000 7fb29e000640 1 -- 172.21.15.32:0/325765240 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2980106c0 con 0x7fb2a406e930 2024-08-03T20:55:55.452 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:55.449+0000 7fb2ac800640 1 -- 172.21.15.32:0/325765240 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb2a413a570 con 0x7fb2a406e930 2024-08-03T20:55:55.452 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:55.449+0000 7fb2ac800640 1 -- 172.21.15.32:0/325765240 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb2a4138510 con 0x7fb2a406e930 2024-08-03T20:55:55.453 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:55.451+0000 7fb2ac800640 1 -- 172.21.15.32:0/325765240 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb2a4160c50 con 0x7fb2a406e930 2024-08-03T20:55:55.453 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:55.452+0000 7fb29e000640 1 -- 172.21.15.32:0/325765240 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7fb29801a070 con 0x7fb2a406e930 2024-08-03T20:55:55.455 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:55.454+0000 7fb29e000640 1 --2- 172.21.15.32:0/325765240 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fb288077040 0x7fb288079550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:55:55.455 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:55.454+0000 7fb29e000640 1 -- 172.21.15.32:0/325765240 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 6542+0+0 (secure 0 0 0) 0x7fb298098d20 con 0x7fb2a406e930 2024-08-03T20:55:55.456 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:55.454+0000 7fb2a9400640 1 --2- 172.21.15.32:0/325765240 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fb288077040 0x7fb288079550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:55:55.456 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:55.455+0000 7fb2a9400640 1 --2- 172.21.15.32:0/325765240 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fb288077040 0x7fb288079550 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7fb2a40d0840 tx=0x7fb294009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:55:55.460 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:55.459+0000 7fb29e000640 1 -- 172.21.15.32:0/325765240 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7fb2980614f0 con 0x7fb2a406e930 2024-08-03T20:55:55.749 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:55.747+0000 7fb2ac800640 1 -- 172.21.15.32:0/325765240 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "mds repaired", "role": "0"} v 0) -- 0x7fb2a4145960 con 0x7fb2a406e930 2024-08-03T20:55:56.185 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:56.183+0000 7fb29e000640 1 -- 172.21.15.32:0/325765240 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "mds repaired", "role": "0"}]=0 nothing to do: rank is not damaged v139) ==== 107+0+0 (secure 0 0 0) 0x7fb298065a40 con 0x7fb2a406e930 2024-08-03T20:55:56.185 INFO:teuthology.orchestra.run.smithi032.stderr:nothing to do: rank is not damaged 2024-08-03T20:55:56.188 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:56.186+0000 7fb2ac800640 1 -- 172.21.15.32:0/325765240 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fb288077040 msgr2=0x7fb288079550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:55:56.188 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:56.186+0000 7fb2ac800640 1 --2- 172.21.15.32:0/325765240 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fb288077040 0x7fb288079550 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7fb2a40d0840 tx=0x7fb294009040 comp rx=0 tx=0).stop 2024-08-03T20:55:56.189 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:56.186+0000 7fb2ac800640 1 -- 172.21.15.32:0/325765240 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb2a406e930 msgr2=0x7fb2a413a030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:55:56.189 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:56.186+0000 7fb2ac800640 1 --2- 172.21.15.32:0/325765240 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb2a406e930 0x7fb2a413a030 secure :-1 s=READY pgs=1265 cs=0 l=1 rev1=1 crypto rx=0x7fb29800d740 tx=0x7fb29800dc00 comp rx=0 tx=0).stop 2024-08-03T20:55:56.189 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:56.186+0000 7fb2ac800640 1 -- 172.21.15.32:0/325765240 shutdown_connections 2024-08-03T20:55:56.189 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:56.186+0000 7fb2ac800640 1 --2- 172.21.15.32:0/325765240 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fb288077040 0x7fb288079550 unknown :-1 s=CLOSED pgs=521 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:56.189 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:56.186+0000 7fb2ac800640 1 --2- 172.21.15.32:0/325765240 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb2a4073e60 0x7fb2a41f3330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:56.189 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:56.186+0000 7fb2ac800640 1 --2- 172.21.15.32:0/325765240 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fb2a415fff0 0x7fb2a41f3870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:56.189 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:56.186+0000 7fb2ac800640 1 --2- 172.21.15.32:0/325765240 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb2a406e930 0x7fb2a413a030 unknown :-1 s=CLOSED pgs=1265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:56.189 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:56.186+0000 7fb2ac800640 1 -- 172.21.15.32:0/325765240 >> 172.21.15.32:0/325765240 conn(0x7fb2a40b1bd0 msgr2=0x7fb2a415ea80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:55:56.189 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:56.187+0000 7fb2ac800640 1 -- 172.21.15.32:0/325765240 shutdown_connections 2024-08-03T20:55:56.189 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:56.187+0000 7fb2ac800640 1 -- 172.21.15.32:0/325765240 wait complete. 2024-08-03T20:55:56.212 DEBUG:teuthology.orchestra.run.smithi032:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata import - 2024-08-03T20:55:56.354 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:56.353+0000 7f7660b12f40 1 Processor -- start 2024-08-03T20:55:56.354 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:56.353+0000 7f7660b12f40 1 -- start start 2024-08-03T20:55:56.356 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:56.354+0000 7f7660b12f40 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x557f01e8bb40 0x557f01ef7e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:55:56.356 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:56.354+0000 7f7660b12f40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x557f01efebe0 0x557f01efc380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:55:56.356 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:56.354+0000 7f7660b12f40 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x557f01f1b650 0x557f01f1ba60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:55:56.357 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:56.354+0000 7f7660b12f40 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x557f01eff510 con 0x557f01f1b650 2024-08-03T20:55:56.357 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:56.354+0000 7f7660b12f40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x557f01d94e30 con 0x557f01efebe0 2024-08-03T20:55:56.357 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:56.354+0000 7f7660b12f40 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x557f01eff690 con 0x557f01e8bb40 2024-08-03T20:55:56.357 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:56.354+0000 7f765a600640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x557f01f1b650 0x557f01f1ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:55:56.357 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:56.354+0000 7f765a600640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x557f01f1b650 0x557f01f1ba60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:50722/0 (socket says 172.21.15.32:50722) 2024-08-03T20:55:56.357 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:56.354+0000 7f765a600640 1 -- 172.21.15.32:0/1483049902 learned_addr learned my addr 172.21.15.32:0/1483049902 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:55:56.357 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:56.355+0000 7f7659200640 1 --2- 172.21.15.32:0/1483049902 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x557f01efebe0 0x557f01efc380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:55:56.357 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:56.355+0000 7f7659c00640 1 --2- 172.21.15.32:0/1483049902 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x557f01e8bb40 0x557f01ef7e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:55:56.357 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:56.355+0000 7f765a600640 1 -- 172.21.15.32:0/1483049902 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x557f01e8bb40 msgr2=0x557f01ef7e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:55:56.357 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:56.355+0000 7f765a600640 1 --2- 172.21.15.32:0/1483049902 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x557f01e8bb40 0x557f01ef7e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:56.357 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:56.355+0000 7f765a600640 1 -- 172.21.15.32:0/1483049902 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x557f01efebe0 msgr2=0x557f01efc380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:55:56.357 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:56.355+0000 7f765a600640 1 --2- 172.21.15.32:0/1483049902 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x557f01efebe0 0x557f01efc380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:56.357 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:56.355+0000 7f765a600640 1 -- 172.21.15.32:0/1483049902 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557f01d6df10 con 0x557f01f1b650 2024-08-03T20:55:56.357 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:56.355+0000 7f7659200640 1 --2- 172.21.15.32:0/1483049902 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x557f01efebe0 0x557f01efc380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-03T20:55:56.358 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:56.355+0000 7f765a600640 1 --2- 172.21.15.32:0/1483049902 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x557f01f1b650 0x557f01f1ba60 secure :-1 s=READY pgs=1267 cs=0 l=1 rev1=1 crypto rx=0x7f765400b680 tx=0x7f765400bb40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8a90a78e139072b9 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:55:56.359 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:56.355+0000 7f7653400640 1 -- 172.21.15.32:0/1483049902 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7654009a30 con 0x557f01f1b650 2024-08-03T20:55:56.359 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:56.355+0000 7f7653400640 1 -- 172.21.15.32:0/1483049902 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f7654009ce0 con 0x557f01f1b650 2024-08-03T20:55:56.359 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:56.355+0000 7f7653400640 1 -- 172.21.15.32:0/1483049902 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f765400e6b0 con 0x557f01f1b650 2024-08-03T20:55:56.359 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:56.356+0000 7f7660b12f40 1 -- 172.21.15.32:0/1483049902 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x557f01f1b650 msgr2=0x557f01f1ba60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:55:56.359 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:56.356+0000 7f7660b12f40 1 --2- 172.21.15.32:0/1483049902 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x557f01f1b650 0x557f01f1ba60 secure :-1 s=READY pgs=1267 cs=0 l=1 rev1=1 crypto rx=0x7f765400b680 tx=0x7f765400bb40 comp rx=0 tx=0).stop 2024-08-03T20:55:56.359 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:56.356+0000 7f7660b12f40 1 -- 172.21.15.32:0/1483049902 shutdown_connections 2024-08-03T20:55:56.359 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:56.356+0000 7f7660b12f40 1 --2- 172.21.15.32:0/1483049902 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x557f01efebe0 0x557f01efc380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:56.359 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:56.356+0000 7f7660b12f40 1 --2- 172.21.15.32:0/1483049902 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x557f01f1b650 0x557f01f1ba60 unknown :-1 s=CLOSED pgs=1267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:56.359 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:56.356+0000 7f7660b12f40 1 --2- 172.21.15.32:0/1483049902 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x557f01e8bb40 0x557f01ef7e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:56.359 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:56.356+0000 7f7660b12f40 1 -- 172.21.15.32:0/1483049902 >> 172.21.15.32:0/1483049902 conn(0x557f01d66ac0 msgr2=0x557f01f1e5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:55:56.360 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:56.356+0000 7f7660b12f40 1 -- 172.21.15.32:0/1483049902 shutdown_connections 2024-08-03T20:55:56.360 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:56.356+0000 7f7660b12f40 1 -- 172.21.15.32:0/1483049902 wait complete. 2024-08-03T20:55:56.360 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:56.357+0000 7f7660b12f40 1 Processor -- start 2024-08-03T20:55:56.360 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:56.357+0000 7f7660b12f40 1 -- start start 2024-08-03T20:55:56.361 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:56.359+0000 7f7660b12f40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x557f01e8bb40 0x557f01ef4d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:55:56.361 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:56.359+0000 7f7660b12f40 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x557f01efebe0 0x557f01ef9510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:55:56.361 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:56.359+0000 7f7660b12f40 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x557f01f1b650 0x557f01fa4420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:55:56.361 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:56.359+0000 7f7660b12f40 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x557f01f1ede0 con 0x557f01efebe0 2024-08-03T20:55:56.361 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:56.359+0000 7f7660b12f40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x557f01fa75c0 con 0x557f01e8bb40 2024-08-03T20:55:56.361 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:56.359+0000 7f7660b12f40 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x557f01fa4fc0 con 0x557f01f1b650 2024-08-03T20:55:56.361 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:56.359+0000 7f7659200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x557f01efebe0 0x557f01ef9510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:55:56.361 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:56.359+0000 7f7659200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x557f01efebe0 0x557f01ef9510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:50738/0 (socket says 172.21.15.32:50738) 2024-08-03T20:55:56.361 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:56.359+0000 7f7659200640 1 -- 172.21.15.32:0/3265918504 learned_addr learned my addr 172.21.15.32:0/3265918504 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:55:56.362 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:56.359+0000 7f7659200640 1 -- 172.21.15.32:0/3265918504 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x557f01f1b650 msgr2=0x557f01fa4420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:55:56.362 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:56.359+0000 7f7659200640 1 --2- 172.21.15.32:0/3265918504 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x557f01f1b650 0x557f01fa4420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:56.362 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:56.359+0000 7f7659200640 1 -- 172.21.15.32:0/3265918504 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x557f01e8bb40 msgr2=0x557f01ef4d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:55:56.362 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:56.359+0000 7f7659200640 1 --2- 172.21.15.32:0/3265918504 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x557f01e8bb40 0x557f01ef4d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:56.362 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:56.359+0000 7f7659200640 1 -- 172.21.15.32:0/3265918504 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557f01f1ad30 con 0x557f01efebe0 2024-08-03T20:55:56.362 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:56.360+0000 7f7659200640 1 --2- 172.21.15.32:0/3265918504 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x557f01efebe0 0x557f01ef9510 secure :-1 s=READY pgs=1268 cs=0 l=1 rev1=1 crypto rx=0x7f7648009480 tx=0x7f7648009570 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:55:56.363 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:56.360+0000 7f7652000640 1 -- 172.21.15.32:0/3265918504 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7648015470 con 0x557f01efebe0 2024-08-03T20:55:56.363 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:56.360+0000 7f7652000640 1 -- 172.21.15.32:0/3265918504 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f7648015610 con 0x557f01efebe0 2024-08-03T20:55:56.363 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:56.360+0000 7f7652000640 1 -- 172.21.15.32:0/3265918504 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7648015a40 con 0x557f01efebe0 2024-08-03T20:55:56.363 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:56.360+0000 7f7660b12f40 1 -- 172.21.15.32:0/3265918504 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x557f01efb260 con 0x557f01efebe0 2024-08-03T20:55:56.363 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:56.360+0000 7f7660b12f40 1 -- 172.21.15.32:0/3265918504 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557f01fa4d00 con 0x557f01efebe0 2024-08-03T20:55:56.364 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:56.362+0000 7f7652000640 1 -- 172.21.15.32:0/3265918504 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f7648015be0 con 0x557f01efebe0 2024-08-03T20:55:56.365 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:56.363+0000 7f7652000640 1 --2- 172.21.15.32:0/3265918504 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f763c0759f0 0x7f763c077f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:55:56.365 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:56.363+0000 7f7652000640 1 -- 172.21.15.32:0/3265918504 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 6542+0+0 (secure 0 0 0) 0x7f7648097510 con 0x557f01efebe0 2024-08-03T20:55:56.365 INFO:teuthology.orchestra.run.smithi032.stdout:Importing pool 2024-08-03T20:55:56.366 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:56.364+0000 7f7660b12f40 1 --2- 172.21.15.32:0/3265918504 >> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] conn(0x557f01fbd760 0x557f01fbfb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:55:56.366 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:56.364+0000 7f7660b12f40 1 -- 172.21.15.32:0/3265918504 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:1 12.10 12:09401a81:::601.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e93) -- 0x557f01e49830 con 0x557f01fbd760 2024-08-03T20:55:56.366 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:56.364+0000 7f7659c00640 1 --2- 172.21.15.32:0/3265918504 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f763c0759f0 0x7f763c077f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:55:56.366 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:56.364+0000 7f765a600640 1 --2- 172.21.15.32:0/3265918504 >> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] conn(0x557f01fbd760 0x557f01fbfb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:55:56.367 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:56.365+0000 7f7659c00640 1 --2- 172.21.15.32:0/3265918504 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f763c0759f0 0x7f763c077f00 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7f764c055000 tx=0x7f764c002fb0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:55:56.367 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:56.365+0000 7f765a600640 1 --2- 172.21.15.32:0/3265918504 >> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] conn(0x557f01fbd760 0x557f01fbfb80 crc :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:55:56.368 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:56.367+0000 7f765a600640 1 -- 172.21.15.32:0/3265918504 <== osd.0 v2:172.21.15.32:6816/3066156116 1 ==== osd_op_reply(1 601.00000000 [create] v93'226 uv200 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f7654009830 con 0x557f01fbd760 2024-08-03T20:55:56.368 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:56.367+0000 7f7660b12f40 1 -- 172.21.15.32:0/3265918504 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:2 12.10 12:09401a81:::601.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) -- 0x557f01ef5510 con 0x557f01fbd760 2024-08-03T20:55:56.370 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:56.369+0000 7f765a600640 1 -- 172.21.15.32:0/3265918504 <== osd.0 v2:172.21.15.32:6816/3066156116 2 ==== osd_op_reply(2 601.00000000 [delete] v93'227 uv227 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f7654009830 con 0x557f01fbd760 2024-08-03T20:55:56.370 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:56.369+0000 7f7660b12f40 1 -- 172.21.15.32:0/3265918504 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:3 12.10 12:09401a81:::601.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e93) -- 0x557f01fc00c0 con 0x557f01fbd760 2024-08-03T20:55:56.371 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:56.370+0000 7f765a600640 1 -- 172.21.15.32:0/3265918504 <== osd.0 v2:172.21.15.32:6816/3066156116 3 ==== osd_op_reply(3 601.00000000 [create] v93'228 uv228 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f7654009830 con 0x557f01fbd760 2024-08-03T20:55:56.372 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::601.00000000:head# 2024-08-03T20:55:56.372 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:56.370+0000 7f7660b12f40 1 -- 172.21.15.32:0/3265918504 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:4 12.10 12:09401a81:::601.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e93) -- 0x557f01fc1470 con 0x557f01fbd760 2024-08-03T20:55:56.373 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:56.372+0000 7f765a600640 1 -- 172.21.15.32:0/3265918504 <== osd.0 v2:172.21.15.32:6816/3066156116 4 ==== osd_op_reply(4 601.00000000 [setxattr (30)] v93'229 uv229 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f7654009830 con 0x557f01fbd760 2024-08-03T20:55:56.373 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:56.372+0000 7f7660b12f40 1 -- 172.21.15.32:0/3265918504 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:5 12.10 12:09401a81:::601.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e93) -- 0x557f01fc1810 con 0x557f01fbd760 2024-08-03T20:55:56.375 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:56.373+0000 7f765a600640 1 -- 172.21.15.32:0/3265918504 <== osd.0 v2:172.21.15.32:6816/3066156116 5 ==== osd_op_reply(5 601.00000000 [setxattr (62)] v93'230 uv230 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f7654009830 con 0x557f01fbd760 2024-08-03T20:55:56.375 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:56.373+0000 7f7660b12f40 1 -- 172.21.15.32:0/3265918504 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:6 12.10 12:09401a81:::601.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e93) -- 0x557f01fc0460 con 0x557f01fbd760 2024-08-03T20:55:56.377 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:56.375+0000 7f765a600640 1 -- 172.21.15.32:0/3265918504 <== osd.0 v2:172.21.15.32:6816/3066156116 6 ==== osd_op_reply(6 601.00000000 [omap-set-header] v93'231 uv231 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f7654009830 con 0x557f01fbd760 2024-08-03T20:55:56.377 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:56.375+0000 7f7660b12f40 1 -- 172.21.15.32:0/3265918504 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:7 12.10 12:0bd6d154:::602.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e93) -- 0x557f01fc0850 con 0x557f01fbd760 2024-08-03T20:55:56.378 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:56.377+0000 7f765a600640 1 -- 172.21.15.32:0/3265918504 <== osd.0 v2:172.21.15.32:6816/3066156116 7 ==== osd_op_reply(7 602.00000000 [create] v93'232 uv206 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f7654009830 con 0x557f01fbd760 2024-08-03T20:55:56.379 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:56.377+0000 7f7660b12f40 1 -- 172.21.15.32:0/3265918504 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:8 12.10 12:0bd6d154:::602.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) -- 0x557f01fc0c40 con 0x557f01fbd760 2024-08-03T20:55:56.380 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:56.379+0000 7f765a600640 1 -- 172.21.15.32:0/3265918504 <== osd.0 v2:172.21.15.32:6816/3066156116 8 ==== osd_op_reply(8 602.00000000 [delete] v93'233 uv233 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f7654009830 con 0x557f01fbd760 2024-08-03T20:55:56.381 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:56.379+0000 7f7660b12f40 1 -- 172.21.15.32:0/3265918504 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:9 12.10 12:0bd6d154:::602.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e93) -- 0x557f01fc0c40 con 0x557f01fbd760 2024-08-03T20:55:56.382 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::602.00000000:head# 2024-08-03T20:55:56.383 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:56.381+0000 7f765a600640 1 -- 172.21.15.32:0/3265918504 <== osd.0 v2:172.21.15.32:6816/3066156116 9 ==== osd_op_reply(9 602.00000000 [create] v93'234 uv234 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f7654009830 con 0x557f01fbd760 2024-08-03T20:55:56.383 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:56.381+0000 7f7660b12f40 1 -- 172.21.15.32:0/3265918504 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:10 12.10 12:0bd6d154:::602.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e93) -- 0x557f01fc0bf0 con 0x557f01fbd760 2024-08-03T20:55:56.384 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:56.383+0000 7f765a600640 1 -- 172.21.15.32:0/3265918504 <== osd.0 v2:172.21.15.32:6816/3066156116 10 ==== osd_op_reply(10 602.00000000 [setxattr (30)] v93'235 uv235 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f7654009830 con 0x557f01fbd760 2024-08-03T20:55:56.385 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:56.383+0000 7f7660b12f40 1 -- 172.21.15.32:0/3265918504 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:11 12.10 12:0bd6d154:::602.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e93) -- 0x557f01fc0bf0 con 0x557f01fbd760 2024-08-03T20:55:56.386 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:56.385+0000 7f765a600640 1 -- 172.21.15.32:0/3265918504 <== osd.0 v2:172.21.15.32:6816/3066156116 11 ==== osd_op_reply(11 602.00000000 [setxattr (62)] v93'236 uv236 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f7654009830 con 0x557f01fbd760 2024-08-03T20:55:56.386 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:56.385+0000 7f7660b12f40 1 -- 172.21.15.32:0/3265918504 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:12 12.10 12:0bd6d154:::602.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e93) -- 0x557f01fc0bf0 con 0x557f01fbd760 2024-08-03T20:55:56.389 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:56.387+0000 7f765a600640 1 -- 172.21.15.32:0/3265918504 <== osd.0 v2:172.21.15.32:6816/3066156116 12 ==== osd_op_reply(12 602.00000000 [omap-set-header] v93'237 uv237 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f7654009830 con 0x557f01fbd760 2024-08-03T20:55:56.389 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:56.387+0000 7f7660b12f40 1 -- 172.21.15.32:0/3265918504 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:13 12.10 12:0cf6ad0f:::10000000000.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e93) -- 0x557f01fc0bf0 con 0x557f01fbd760 2024-08-03T20:55:56.390 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:56.388+0000 7f765a600640 1 -- 172.21.15.32:0/3265918504 <== osd.0 v2:172.21.15.32:6816/3066156116 13 ==== osd_op_reply(13 10000000000.00000000 [create] v93'238 uv213 ondisk = -17 ((17) File exists)) ==== 164+0+0 (crc 0 0 0) 0x7f7654009830 con 0x557f01fbd760 2024-08-03T20:55:56.390 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:56.388+0000 7f7660b12f40 1 -- 172.21.15.32:0/3265918504 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:14 12.10 12:0cf6ad0f:::10000000000.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) -- 0x557f01fc0bf0 con 0x557f01fbd760 2024-08-03T20:55:56.392 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:56.390+0000 7f765a600640 1 -- 172.21.15.32:0/3265918504 <== osd.0 v2:172.21.15.32:6816/3066156116 14 ==== osd_op_reply(14 10000000000.00000000 [delete] v93'239 uv239 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f7654009830 con 0x557f01fbd760 2024-08-03T20:55:56.392 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:56.390+0000 7f7660b12f40 1 -- 172.21.15.32:0/3265918504 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:15 12.10 12:0cf6ad0f:::10000000000.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e93) -- 0x557f01fc0bf0 con 0x557f01fbd760 2024-08-03T20:55:56.393 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::10000000000.00000000:head# 2024-08-03T20:55:56.394 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:56.392+0000 7f765a600640 1 -- 172.21.15.32:0/3265918504 <== osd.0 v2:172.21.15.32:6816/3066156116 15 ==== osd_op_reply(15 10000000000.00000000 [create] v93'240 uv240 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f7654009830 con 0x557f01fbd760 2024-08-03T20:55:56.394 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:56.392+0000 7f7660b12f40 1 -- 172.21.15.32:0/3265918504 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:16 12.10 12:0cf6ad0f:::10000000000.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e93) -- 0x557f01fc0bf0 con 0x557f01fbd760 2024-08-03T20:55:56.395 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:56.394+0000 7f765a600640 1 -- 172.21.15.32:0/3265918504 <== osd.0 v2:172.21.15.32:6816/3066156116 16 ==== osd_op_reply(16 10000000000.00000000 [setxattr (30)] v93'241 uv241 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f7654009830 con 0x557f01fbd760 2024-08-03T20:55:56.396 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:56.394+0000 7f7660b12f40 1 -- 172.21.15.32:0/3265918504 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:17 12.10 12:0cf6ad0f:::10000000000.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e93) -- 0x557f01fc0bf0 con 0x557f01fbd760 2024-08-03T20:55:56.397 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:56.396+0000 7f765a600640 1 -- 172.21.15.32:0/3265918504 <== osd.0 v2:172.21.15.32:6816/3066156116 17 ==== osd_op_reply(17 10000000000.00000000 [setxattr (62)] v93'242 uv242 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f7654009830 con 0x557f01fbd760 2024-08-03T20:55:56.397 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:56.396+0000 7f7660b12f40 1 -- 172.21.15.32:0/3265918504 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:18 12.10 12:0cf6ad0f:::10000000000.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e93) -- 0x557f01fc0bf0 con 0x557f01fbd760 2024-08-03T20:55:56.399 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:56.398+0000 7f765a600640 1 -- 172.21.15.32:0/3265918504 <== osd.0 v2:172.21.15.32:6816/3066156116 18 ==== osd_op_reply(18 10000000000.00000000 [omap-set-header] v93'243 uv243 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f7654009830 con 0x557f01fbd760 2024-08-03T20:55:56.399 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:56.398+0000 7f7660b12f40 1 -- 172.21.15.32:0/3265918504 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:19 12.10 12:0cf6ad0f:::10000000000.00000000:head [omap-set-vals in=593b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e93) -- 0x557f01fc0bf0 con 0x557f01fbd760 2024-08-03T20:55:56.401 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:56.400+0000 7f765a600640 1 -- 172.21.15.32:0/3265918504 <== osd.0 v2:172.21.15.32:6816/3066156116 19 ==== osd_op_reply(19 10000000000.00000000 [omap-set-vals] v93'244 uv244 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f7654009830 con 0x557f01fbd760 2024-08-03T20:55:56.401 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:56.400+0000 7f7660b12f40 1 -- 172.21.15.32:0/3265918504 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:20 12.10 12:0d82a743:::600.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e93) -- 0x557f01fc0bf0 con 0x557f01fbd760 2024-08-03T20:55:56.403 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:56.401+0000 7f765a600640 1 -- 172.21.15.32:0/3265918504 <== osd.0 v2:172.21.15.32:6816/3066156116 20 ==== osd_op_reply(20 600.00000000 [create] v93'245 uv219 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f7654009830 con 0x557f01fbd760 2024-08-03T20:55:56.403 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:56.401+0000 7f7660b12f40 1 -- 172.21.15.32:0/3265918504 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:21 12.10 12:0d82a743:::600.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) -- 0x557f01fc0bf0 con 0x557f01fbd760 2024-08-03T20:55:56.404 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:56.403+0000 7f765a600640 1 -- 172.21.15.32:0/3265918504 <== osd.0 v2:172.21.15.32:6816/3066156116 21 ==== osd_op_reply(21 600.00000000 [delete] v93'246 uv246 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f7654009830 con 0x557f01fbd760 2024-08-03T20:55:56.405 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:56.403+0000 7f7660b12f40 1 -- 172.21.15.32:0/3265918504 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:22 12.10 12:0d82a743:::600.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e93) -- 0x557f01fc0bf0 con 0x557f01fbd760 2024-08-03T20:55:56.406 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::600.00000000:head# 2024-08-03T20:55:56.406 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:56.405+0000 7f765a600640 1 -- 172.21.15.32:0/3265918504 <== osd.0 v2:172.21.15.32:6816/3066156116 22 ==== osd_op_reply(22 600.00000000 [create] v93'247 uv247 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f7654009830 con 0x557f01fbd760 2024-08-03T20:55:56.407 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:56.405+0000 7f7660b12f40 1 -- 172.21.15.32:0/3265918504 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:23 12.10 12:0d82a743:::600.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e93) -- 0x557f01fc0bf0 con 0x557f01fbd760 2024-08-03T20:55:56.408 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:56.406+0000 7f765a600640 1 -- 172.21.15.32:0/3265918504 <== osd.0 v2:172.21.15.32:6816/3066156116 23 ==== osd_op_reply(23 600.00000000 [setxattr (30)] v93'248 uv248 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f7654009830 con 0x557f01fbd760 2024-08-03T20:55:56.408 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:56.407+0000 7f7660b12f40 1 -- 172.21.15.32:0/3265918504 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:24 12.10 12:0d82a743:::600.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e93) -- 0x557f01fc0bf0 con 0x557f01fbd760 2024-08-03T20:55:56.409 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:56.408+0000 7f765a600640 1 -- 172.21.15.32:0/3265918504 <== osd.0 v2:172.21.15.32:6816/3066156116 24 ==== osd_op_reply(24 600.00000000 [setxattr (62)] v93'249 uv249 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f7654009830 con 0x557f01fbd760 2024-08-03T20:55:56.410 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:56.408+0000 7f7660b12f40 1 -- 172.21.15.32:0/3265918504 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:25 12.10 12:0d82a743:::600.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e93) -- 0x557f01fc0bf0 con 0x557f01fbd760 2024-08-03T20:55:56.411 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:56.410+0000 7f765a600640 1 -- 172.21.15.32:0/3265918504 <== osd.0 v2:172.21.15.32:6816/3066156116 25 ==== osd_op_reply(25 600.00000000 [omap-set-header] v93'250 uv250 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f7654009830 con 0x557f01fbd760 2024-08-03T20:55:56.411 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:56.410+0000 7f7660b12f40 1 -- 172.21.15.32:0/3265918504 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:26 12.10 12:0d89b25e:::603.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e93) -- 0x557f01fc0bf0 con 0x557f01fbd760 2024-08-03T20:55:56.413 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:56.412+0000 7f765a600640 1 -- 172.21.15.32:0/3265918504 <== osd.0 v2:172.21.15.32:6816/3066156116 26 ==== osd_op_reply(26 603.00000000 [create] v93'251 uv225 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f7654009830 con 0x557f01fbd760 2024-08-03T20:55:56.413 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:56.412+0000 7f7660b12f40 1 -- 172.21.15.32:0/3265918504 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:27 12.10 12:0d89b25e:::603.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) -- 0x557f01fc0bf0 con 0x557f01fbd760 2024-08-03T20:55:56.415 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:56.413+0000 7f765a600640 1 -- 172.21.15.32:0/3265918504 <== osd.0 v2:172.21.15.32:6816/3066156116 27 ==== osd_op_reply(27 603.00000000 [delete] v93'252 uv252 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f7654009830 con 0x557f01fbd760 2024-08-03T20:55:56.415 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:56.413+0000 7f7660b12f40 1 -- 172.21.15.32:0/3265918504 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:28 12.10 12:0d89b25e:::603.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e93) -- 0x557f01fc0bf0 con 0x557f01fbd760 2024-08-03T20:55:56.416 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::603.00000000:head# 2024-08-03T20:55:56.417 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:56.415+0000 7f765a600640 1 -- 172.21.15.32:0/3265918504 <== osd.0 v2:172.21.15.32:6816/3066156116 28 ==== osd_op_reply(28 603.00000000 [create] v93'253 uv253 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f7654009830 con 0x557f01fbd760 2024-08-03T20:55:56.417 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:56.415+0000 7f7660b12f40 1 -- 172.21.15.32:0/3265918504 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:29 12.10 12:0d89b25e:::603.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e93) -- 0x557f01fc0bf0 con 0x557f01fbd760 2024-08-03T20:55:56.418 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:56.416+0000 7f765a600640 1 -- 172.21.15.32:0/3265918504 <== osd.0 v2:172.21.15.32:6816/3066156116 29 ==== osd_op_reply(29 603.00000000 [setxattr (30)] v93'254 uv254 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f7654009830 con 0x557f01fbd760 2024-08-03T20:55:56.418 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:56.416+0000 7f7660b12f40 1 -- 172.21.15.32:0/3265918504 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:30 12.10 12:0d89b25e:::603.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e93) -- 0x557f01fc0bf0 con 0x557f01fbd760 2024-08-03T20:55:56.420 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:56.418+0000 7f765a600640 1 -- 172.21.15.32:0/3265918504 <== osd.0 v2:172.21.15.32:6816/3066156116 30 ==== osd_op_reply(30 603.00000000 [setxattr (62)] v93'255 uv255 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f7654009830 con 0x557f01fbd760 2024-08-03T20:55:56.421 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:56.418+0000 7f7660b12f40 1 -- 172.21.15.32:0/3265918504 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:31 12.10 12:0d89b25e:::603.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e93) -- 0x557f01fc0bf0 con 0x557f01fbd760 2024-08-03T20:55:56.421 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:56.420+0000 7f765a600640 1 -- 172.21.15.32:0/3265918504 <== osd.0 v2:172.21.15.32:6816/3066156116 31 ==== osd_op_reply(31 603.00000000 [omap-set-header] v93'256 uv256 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f7654009830 con 0x557f01fbd760 2024-08-03T20:55:56.421 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:56.420+0000 7f7660b12f40 1 --2- 172.21.15.32:0/3265918504 >> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] conn(0x557f01fc0fb0 0x557f01fc40d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:55:56.421 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:56.420+0000 7f7660b12f40 1 -- 172.21.15.32:0/3265918504 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:32 12.14 12:287034c4:::1.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e93) -- 0x557f01fc0bf0 con 0x557f01fc0fb0 2024-08-03T20:55:56.422 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:56.420+0000 7f7659c00640 1 --2- 172.21.15.32:0/3265918504 >> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] conn(0x557f01fc0fb0 0x557f01fc40d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:55:56.422 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:56.421+0000 7f7659c00640 1 --2- 172.21.15.32:0/3265918504 >> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] conn(0x557f01fc0fb0 0x557f01fc40d0 crc :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:55:56.424 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:56.423+0000 7f7659c00640 1 -- 172.21.15.32:0/3265918504 <== osd.5 v2:172.21.15.118:6810/2724891770 1 ==== osd_op_reply(32 1.00000000.inode [create] v93'177 uv156 ondisk = -17 ((17) File exists)) ==== 160+0+0 (crc 0 0 0) 0x7f764c056470 con 0x557f01fc0fb0 2024-08-03T20:55:56.424 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:56.423+0000 7f7660b12f40 1 -- 172.21.15.32:0/3265918504 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:33 12.14 12:287034c4:::1.00000000.inode:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) -- 0x557f01fc4610 con 0x557f01fc0fb0 2024-08-03T20:55:56.426 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:56.424+0000 7f7659c00640 1 -- 172.21.15.32:0/3265918504 <== osd.5 v2:172.21.15.118:6810/2724891770 2 ==== osd_op_reply(33 1.00000000.inode [delete] v93'178 uv178 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f764c056470 con 0x557f01fc0fb0 2024-08-03T20:55:56.426 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:56.424+0000 7f7660b12f40 1 -- 172.21.15.32:0/3265918504 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:34 12.14 12:287034c4:::1.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e93) -- 0x557f01fc49b0 con 0x557f01fc0fb0 2024-08-03T20:55:56.427 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:56.426+0000 7f7659c00640 1 -- 172.21.15.32:0/3265918504 <== osd.5 v2:172.21.15.118:6810/2724891770 3 ==== osd_op_reply(34 1.00000000.inode [create] v93'179 uv179 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f764c056470 con 0x557f01fc0fb0 2024-08-03T20:55:56.428 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::1.00000000.inode:head# 2024-08-03T20:55:56.428 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:56.426+0000 7f7660b12f40 1 -- 172.21.15.32:0/3265918504 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:35 12.14 12:287034c4:::1.00000000.inode:head [write 0~582 in=582b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e93) -- 0x557f01fc4d50 con 0x557f01fc0fb0 2024-08-03T20:55:56.429 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:56.428+0000 7f7659c00640 1 -- 172.21.15.32:0/3265918504 <== osd.5 v2:172.21.15.118:6810/2724891770 4 ==== osd_op_reply(35 1.00000000.inode [write 0~582] v93'180 uv180 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f764c056470 con 0x557f01fc0fb0 2024-08-03T20:55:56.429 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:56.428+0000 7f7660b12f40 1 -- 172.21.15.32:0/3265918504 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:36 12.14 12:287034c4:::1.00000000.inode:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e93) -- 0x557f01fc50f0 con 0x557f01fc0fb0 2024-08-03T20:55:56.431 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:56.429+0000 7f7659c00640 1 -- 172.21.15.32:0/3265918504 <== osd.5 v2:172.21.15.118:6810/2724891770 5 ==== osd_op_reply(36 1.00000000.inode [omap-set-header] v93'181 uv181 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f764c056470 con 0x557f01fc0fb0 2024-08-03T20:55:56.431 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:56.429+0000 7f7660b12f40 1 -- 172.21.15.32:0/3265918504 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:37 12.14 12:292cf221:::200.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e93) -- 0x557f01fc5490 con 0x557f01fc0fb0 2024-08-03T20:55:56.432 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:56.431+0000 7f7659c00640 1 -- 172.21.15.32:0/3265918504 <== osd.5 v2:172.21.15.118:6810/2724891770 6 ==== osd_op_reply(37 200.00000000 [create] v93'182 uv175 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f764c056470 con 0x557f01fc0fb0 2024-08-03T20:55:56.432 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:56.431+0000 7f7660b12f40 1 -- 172.21.15.32:0/3265918504 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:38 12.14 12:292cf221:::200.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) -- 0x557f01fc5830 con 0x557f01fc0fb0 2024-08-03T20:55:56.434 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:56.433+0000 7f7659c00640 1 -- 172.21.15.32:0/3265918504 <== osd.5 v2:172.21.15.118:6810/2724891770 7 ==== osd_op_reply(38 200.00000000 [delete] v93'183 uv183 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f764c056470 con 0x557f01fc0fb0 2024-08-03T20:55:56.434 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:56.433+0000 7f7660b12f40 1 -- 172.21.15.32:0/3265918504 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:39 12.14 12:292cf221:::200.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e93) -- 0x557f01fc5830 con 0x557f01fc0fb0 2024-08-03T20:55:56.435 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:56.434+0000 7f7659c00640 1 -- 172.21.15.32:0/3265918504 <== osd.5 v2:172.21.15.118:6810/2724891770 8 ==== osd_op_reply(39 200.00000000 [create] v93'184 uv184 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f764c056470 con 0x557f01fc0fb0 2024-08-03T20:55:56.436 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::200.00000000:head# 2024-08-03T20:55:56.436 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:56.434+0000 7f7660b12f40 1 -- 172.21.15.32:0/3265918504 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:40 12.14 12:292cf221:::200.00000000:head [write 0~90 in=90b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e93) -- 0x557f01fc5830 con 0x557f01fc0fb0 2024-08-03T20:55:56.437 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:56.436+0000 7f7659c00640 1 -- 172.21.15.32:0/3265918504 <== osd.5 v2:172.21.15.118:6810/2724891770 9 ==== osd_op_reply(40 200.00000000 [write 0~90] v93'185 uv185 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f764c056470 con 0x557f01fc0fb0 2024-08-03T20:55:56.437 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:56.436+0000 7f7660b12f40 1 -- 172.21.15.32:0/3265918504 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:41 12.14 12:292cf221:::200.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e93) -- 0x557f01fc5830 con 0x557f01fc0fb0 2024-08-03T20:55:56.439 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:56.438+0000 7f7659c00640 1 -- 172.21.15.32:0/3265918504 <== osd.5 v2:172.21.15.118:6810/2724891770 10 ==== osd_op_reply(41 200.00000000 [omap-set-header] v93'186 uv186 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f764c056470 con 0x557f01fc0fb0 2024-08-03T20:55:56.439 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:56.438+0000 7f7660b12f40 1 -- 172.21.15.32:0/3265918504 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:42 12.14 12:2e2fa760:::200.00000001:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e93) -- 0x557f01fc5830 con 0x557f01fc0fb0 2024-08-03T20:55:56.441 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:56.439+0000 7f7659c00640 1 -- 172.21.15.32:0/3265918504 <== osd.5 v2:172.21.15.118:6810/2724891770 11 ==== osd_op_reply(42 200.00000001 [create] v93'187 uv176 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f764c056470 con 0x557f01fc0fb0 2024-08-03T20:55:56.441 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:56.439+0000 7f7660b12f40 1 -- 172.21.15.32:0/3265918504 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:43 12.14 12:2e2fa760:::200.00000001:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) -- 0x557f01fc5830 con 0x557f01fc0fb0 2024-08-03T20:55:56.443 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:56.441+0000 7f7659c00640 1 -- 172.21.15.32:0/3265918504 <== osd.5 v2:172.21.15.118:6810/2724891770 12 ==== osd_op_reply(43 200.00000001 [delete] v93'188 uv188 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f764c056470 con 0x557f01fc0fb0 2024-08-03T20:55:56.443 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:56.441+0000 7f7660b12f40 1 -- 172.21.15.32:0/3265918504 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:44 12.14 12:2e2fa760:::200.00000001:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e93) -- 0x557f01fc5830 con 0x557f01fc0fb0 2024-08-03T20:55:56.444 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:56.443+0000 7f7659c00640 1 -- 172.21.15.32:0/3265918504 <== osd.5 v2:172.21.15.118:6810/2724891770 13 ==== osd_op_reply(44 200.00000001 [create] v93'189 uv189 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f764c056470 con 0x557f01fc0fb0 2024-08-03T20:55:56.444 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::200.00000001:head# 2024-08-03T20:55:56.444 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:56.443+0000 7f7660b12f40 1 -- 172.21.15.32:0/3265918504 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:45 12.14 12:2e2fa760:::200.00000001:head [write 0~18118 in=18118b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e93) -- 0x557f01fc9f20 con 0x557f01fc0fb0 2024-08-03T20:55:56.446 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:56.445+0000 7f7659c00640 1 -- 172.21.15.32:0/3265918504 <== osd.5 v2:172.21.15.118:6810/2724891770 14 ==== osd_op_reply(45 200.00000001 [write 0~18118] v93'190 uv190 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f764c056470 con 0x557f01fc0fb0 2024-08-03T20:55:56.446 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:56.445+0000 7f7660b12f40 1 -- 172.21.15.32:0/3265918504 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:46 12.14 12:2e2fa760:::200.00000001:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e93) -- 0x557f01fc9f20 con 0x557f01fc0fb0 2024-08-03T20:55:56.448 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:56.447+0000 7f7659c00640 1 -- 172.21.15.32:0/3265918504 <== osd.5 v2:172.21.15.118:6810/2724891770 15 ==== osd_op_reply(46 200.00000001 [omap-set-header] v93'191 uv191 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f764c056470 con 0x557f01fc0fb0 2024-08-03T20:55:56.448 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:56.447+0000 7f7660b12f40 1 -- 172.21.15.32:0/3265918504 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:47 12.14 12:2f55791f:::606.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e93) -- 0x557f01fc5830 con 0x557f01fc0fb0 2024-08-03T20:55:56.450 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:56.449+0000 7f7659c00640 1 -- 172.21.15.32:0/3265918504 <== osd.5 v2:172.21.15.118:6810/2724891770 16 ==== osd_op_reply(47 606.00000000 [create] v93'192 uv172 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f764c056470 con 0x557f01fc0fb0 2024-08-03T20:55:56.450 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:56.449+0000 7f7660b12f40 1 -- 172.21.15.32:0/3265918504 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:48 12.14 12:2f55791f:::606.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) -- 0x557f01fc5830 con 0x557f01fc0fb0 2024-08-03T20:55:56.452 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:56.450+0000 7f7659c00640 1 -- 172.21.15.32:0/3265918504 <== osd.5 v2:172.21.15.118:6810/2724891770 17 ==== osd_op_reply(48 606.00000000 [delete] v93'193 uv193 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f764c056470 con 0x557f01fc0fb0 2024-08-03T20:55:56.452 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:56.450+0000 7f7660b12f40 1 -- 172.21.15.32:0/3265918504 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:49 12.14 12:2f55791f:::606.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e93) -- 0x557f01fc5830 con 0x557f01fc0fb0 2024-08-03T20:55:56.453 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:56.452+0000 7f7659c00640 1 -- 172.21.15.32:0/3265918504 <== osd.5 v2:172.21.15.118:6810/2724891770 18 ==== osd_op_reply(49 606.00000000 [create] v93'194 uv194 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f764c056470 con 0x557f01fc0fb0 2024-08-03T20:55:56.454 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::606.00000000:head# 2024-08-03T20:55:56.454 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:56.452+0000 7f7660b12f40 1 -- 172.21.15.32:0/3265918504 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:50 12.14 12:2f55791f:::606.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e93) -- 0x557f01fc6840 con 0x557f01fc0fb0 2024-08-03T20:55:56.455 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:56.453+0000 7f7659c00640 1 -- 172.21.15.32:0/3265918504 <== osd.5 v2:172.21.15.118:6810/2724891770 19 ==== osd_op_reply(50 606.00000000 [setxattr (30)] v93'195 uv195 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f764c056470 con 0x557f01fc0fb0 2024-08-03T20:55:56.455 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:56.453+0000 7f7660b12f40 1 -- 172.21.15.32:0/3265918504 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:51 12.14 12:2f55791f:::606.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e93) -- 0x557f01fc6840 con 0x557f01fc0fb0 2024-08-03T20:55:56.456 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:56.455+0000 7f7659c00640 1 -- 172.21.15.32:0/3265918504 <== osd.5 v2:172.21.15.118:6810/2724891770 20 ==== osd_op_reply(51 606.00000000 [setxattr (62)] v93'196 uv196 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f764c056470 con 0x557f01fc0fb0 2024-08-03T20:55:56.456 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:56.455+0000 7f7660b12f40 1 -- 172.21.15.32:0/3265918504 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:52 12.14 12:2f55791f:::606.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e93) -- 0x557f01fc5830 con 0x557f01fc0fb0 2024-08-03T20:55:56.458 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:56.457+0000 7f7659c00640 1 -- 172.21.15.32:0/3265918504 <== osd.5 v2:172.21.15.118:6810/2724891770 21 ==== osd_op_reply(52 606.00000000 [omap-set-header] v93'197 uv197 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f764c056470 con 0x557f01fc0fb0 2024-08-03T20:55:56.459 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:56.457+0000 7f7660b12f40 1 -- 172.21.15.32:0/3265918504 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:53 12.1c 12:3ed09add:::607.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e93) -- 0x557f01fc5830 con 0x557f01fbd760 2024-08-03T20:55:56.460 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:56.459+0000 7f765a600640 1 -- 172.21.15.32:0/3265918504 <== osd.0 v2:172.21.15.32:6816/3066156116 32 ==== osd_op_reply(53 607.00000000 [create] v93'45 uv44 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f7654009830 con 0x557f01fbd760 2024-08-03T20:55:56.460 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:56.459+0000 7f7660b12f40 1 -- 172.21.15.32:0/3265918504 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:54 12.1c 12:3ed09add:::607.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) -- 0x557f01fc5830 con 0x557f01fbd760 2024-08-03T20:55:56.462 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:56.460+0000 7f765a600640 1 -- 172.21.15.32:0/3265918504 <== osd.0 v2:172.21.15.32:6816/3066156116 33 ==== osd_op_reply(54 607.00000000 [delete] v93'46 uv46 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f7654009830 con 0x557f01fbd760 2024-08-03T20:55:56.462 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:56.460+0000 7f7660b12f40 1 -- 172.21.15.32:0/3265918504 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:55 12.1c 12:3ed09add:::607.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e93) -- 0x557f01fc5830 con 0x557f01fbd760 2024-08-03T20:55:56.463 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::607.00000000:head# 2024-08-03T20:55:56.463 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:56.462+0000 7f765a600640 1 -- 172.21.15.32:0/3265918504 <== osd.0 v2:172.21.15.32:6816/3066156116 34 ==== osd_op_reply(55 607.00000000 [create] v93'47 uv47 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f7654009830 con 0x557f01fbd760 2024-08-03T20:55:56.463 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:56.462+0000 7f7660b12f40 1 -- 172.21.15.32:0/3265918504 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:56 12.1c 12:3ed09add:::607.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e93) -- 0x557f01fc6840 con 0x557f01fbd760 2024-08-03T20:55:56.464 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:56.463+0000 7f765a600640 1 -- 172.21.15.32:0/3265918504 <== osd.0 v2:172.21.15.32:6816/3066156116 35 ==== osd_op_reply(56 607.00000000 [setxattr (30)] v93'48 uv48 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f7654009830 con 0x557f01fbd760 2024-08-03T20:55:56.464 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:56.463+0000 7f7660b12f40 1 -- 172.21.15.32:0/3265918504 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:57 12.1c 12:3ed09add:::607.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e93) -- 0x557f01fc6840 con 0x557f01fbd760 2024-08-03T20:55:56.466 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:56.464+0000 7f765a600640 1 -- 172.21.15.32:0/3265918504 <== osd.0 v2:172.21.15.32:6816/3066156116 36 ==== osd_op_reply(57 607.00000000 [setxattr (62)] v93'49 uv49 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f7654009830 con 0x557f01fbd760 2024-08-03T20:55:56.466 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:56.464+0000 7f7660b12f40 1 -- 172.21.15.32:0/3265918504 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:58 12.1c 12:3ed09add:::607.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e93) -- 0x557f01fc5830 con 0x557f01fbd760 2024-08-03T20:55:56.467 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:56.466+0000 7f765a600640 1 -- 172.21.15.32:0/3265918504 <== osd.0 v2:172.21.15.32:6816/3066156116 37 ==== osd_op_reply(58 607.00000000 [omap-set-header] v93'50 uv50 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f7654009830 con 0x557f01fbd760 2024-08-03T20:55:56.467 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:56.466+0000 7f7660b12f40 1 --2- 172.21.15.32:0/3265918504 >> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] conn(0x557f01fc5eb0 0x557f01fc62c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:55:56.467 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:56.466+0000 7f7660b12f40 1 -- 172.21.15.32:0/3265918504 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:59 12.6 12:654134d2:::mds0_openfiles.0:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e93) -- 0x557f01fc8c40 con 0x557f01fc5eb0 2024-08-03T20:55:56.468 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:56.466+0000 7f7659200640 1 --2- 172.21.15.32:0/3265918504 >> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] conn(0x557f01fc5eb0 0x557f01fc62c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:55:56.468 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:56.467+0000 7f7659200640 1 --2- 172.21.15.32:0/3265918504 >> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] conn(0x557f01fc5eb0 0x557f01fc62c0 crc :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:55:56.469 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:56.468+0000 7f7659200640 1 -- 172.21.15.32:0/3265918504 <== osd.1 v2:172.21.15.32:6824/1551241759 1 ==== osd_op_reply(59 mds0_openfiles.0 [create] v93'43 uv42 ondisk = -17 ((17) File exists)) ==== 160+0+0 (crc 0 0 0) 0x7f764805c0a0 con 0x557f01fc5eb0 2024-08-03T20:55:56.469 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:56.468+0000 7f7660b12f40 1 -- 172.21.15.32:0/3265918504 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:60 12.6 12:654134d2:::mds0_openfiles.0:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) -- 0x557f01fc8fe0 con 0x557f01fc5eb0 2024-08-03T20:55:56.471 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:56.469+0000 7f7659200640 1 -- 172.21.15.32:0/3265918504 <== osd.1 v2:172.21.15.32:6824/1551241759 2 ==== osd_op_reply(60 mds0_openfiles.0 [delete] v93'44 uv44 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f764805c0a0 con 0x557f01fc5eb0 2024-08-03T20:55:56.471 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:56.469+0000 7f7660b12f40 1 -- 172.21.15.32:0/3265918504 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:61 12.6 12:654134d2:::mds0_openfiles.0:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e93) -- 0x557f01fc9380 con 0x557f01fc5eb0 2024-08-03T20:55:56.472 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds0_openfiles.0:head# 2024-08-03T20:55:56.472 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:56.470+0000 7f7659200640 1 -- 172.21.15.32:0/3265918504 <== osd.1 v2:172.21.15.32:6824/1551241759 3 ==== osd_op_reply(61 mds0_openfiles.0 [create] v93'45 uv45 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f764805c0a0 con 0x557f01fc5eb0 2024-08-03T20:55:56.473 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:56.470+0000 7f7660b12f40 1 -- 172.21.15.32:0/3265918504 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:62 12.6 12:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e93) -- 0x557f01f00230 con 0x557f01fc5eb0 2024-08-03T20:55:56.473 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:56.472+0000 7f7659200640 1 -- 172.21.15.32:0/3265918504 <== osd.1 v2:172.21.15.32:6824/1551241759 4 ==== osd_op_reply(62 mds0_openfiles.0 [omap-set-header] v93'46 uv46 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f764805c0a0 con 0x557f01fc5eb0 2024-08-03T20:55:56.473 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:56.472+0000 7f7660b12f40 1 -- 172.21.15.32:0/3265918504 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:63 12.e 12:717a0223:::608.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e93) -- 0x557f01fc9cb0 con 0x557f01fbd760 2024-08-03T20:55:56.475 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:56.473+0000 7f765a600640 1 -- 172.21.15.32:0/3265918504 <== osd.0 v2:172.21.15.32:6816/3066156116 38 ==== osd_op_reply(63 608.00000000 [create] v93'45 uv44 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f7654009830 con 0x557f01fbd760 2024-08-03T20:55:56.475 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:56.473+0000 7f7660b12f40 1 -- 172.21.15.32:0/3265918504 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:64 12.e 12:717a0223:::608.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) -- 0x557f01fc9cb0 con 0x557f01fbd760 2024-08-03T20:55:56.476 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:56.475+0000 7f765a600640 1 -- 172.21.15.32:0/3265918504 <== osd.0 v2:172.21.15.32:6816/3066156116 39 ==== osd_op_reply(64 608.00000000 [delete] v93'46 uv46 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f7654009830 con 0x557f01fbd760 2024-08-03T20:55:56.476 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:56.475+0000 7f7660b12f40 1 -- 172.21.15.32:0/3265918504 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:65 12.e 12:717a0223:::608.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e93) -- 0x557f01fc9cb0 con 0x557f01fbd760 2024-08-03T20:55:56.478 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:56.476+0000 7f765a600640 1 -- 172.21.15.32:0/3265918504 <== osd.0 v2:172.21.15.32:6816/3066156116 40 ==== osd_op_reply(65 608.00000000 [create] v93'47 uv47 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f7654009830 con 0x557f01fbd760 2024-08-03T20:55:56.478 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::608.00000000:head# 2024-08-03T20:55:56.478 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:56.476+0000 7f7660b12f40 1 -- 172.21.15.32:0/3265918504 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:66 12.e 12:717a0223:::608.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e93) -- 0x557f01fcacc0 con 0x557f01fbd760 2024-08-03T20:55:56.479 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:56.478+0000 7f765a600640 1 -- 172.21.15.32:0/3265918504 <== osd.0 v2:172.21.15.32:6816/3066156116 41 ==== osd_op_reply(66 608.00000000 [setxattr (30)] v93'48 uv48 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f7654009830 con 0x557f01fbd760 2024-08-03T20:55:56.479 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:56.478+0000 7f7660b12f40 1 -- 172.21.15.32:0/3265918504 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:67 12.e 12:717a0223:::608.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e93) -- 0x557f01fcacc0 con 0x557f01fbd760 2024-08-03T20:55:56.481 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:56.479+0000 7f765a600640 1 -- 172.21.15.32:0/3265918504 <== osd.0 v2:172.21.15.32:6816/3066156116 42 ==== osd_op_reply(67 608.00000000 [setxattr (62)] v93'49 uv49 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f7654009830 con 0x557f01fbd760 2024-08-03T20:55:56.481 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:56.479+0000 7f7660b12f40 1 -- 172.21.15.32:0/3265918504 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:68 12.e 12:717a0223:::608.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e93) -- 0x557f01fc9cb0 con 0x557f01fbd760 2024-08-03T20:55:56.483 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:56.481+0000 7f765a600640 1 -- 172.21.15.32:0/3265918504 <== osd.0 v2:172.21.15.32:6816/3066156116 43 ==== osd_op_reply(68 608.00000000 [omap-set-header] v93'50 uv50 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f7654009830 con 0x557f01fbd760 2024-08-03T20:55:56.483 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:56.481+0000 7f7660b12f40 1 --2- 172.21.15.32:0/3265918504 >> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] conn(0x557f01fca580 0x557f01fca990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:55:56.483 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:56.481+0000 7f7660b12f40 1 -- 172.21.15.32:0/3265918504 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:69 12.15 12:a93a17c2:::604.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e93) -- 0x557f01fcd130 con 0x557f01fca580 2024-08-03T20:55:56.483 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:56.481+0000 7f765a600640 1 --2- 172.21.15.32:0/3265918504 >> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] conn(0x557f01fca580 0x557f01fca990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:55:56.483 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:56.482+0000 7f765a600640 1 --2- 172.21.15.32:0/3265918504 >> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] conn(0x557f01fca580 0x557f01fca990 crc :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:55:56.485 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:56.483+0000 7f765a600640 1 -- 172.21.15.32:0/3265918504 <== osd.6 v2:172.21.15.118:6825/801947386 1 ==== osd_op_reply(69 604.00000000 [create] v93'83 uv75 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f7654009830 con 0x557f01fca580 2024-08-03T20:55:56.485 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:56.483+0000 7f7660b12f40 1 -- 172.21.15.32:0/3265918504 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:70 12.15 12:a93a17c2:::604.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) -- 0x557f01fcd130 con 0x557f01fca580 2024-08-03T20:55:56.486 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:56.485+0000 7f765a600640 1 -- 172.21.15.32:0/3265918504 <== osd.6 v2:172.21.15.118:6825/801947386 2 ==== osd_op_reply(70 604.00000000 [delete] v93'84 uv84 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f7654009830 con 0x557f01fca580 2024-08-03T20:55:56.486 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:56.485+0000 7f7660b12f40 1 -- 172.21.15.32:0/3265918504 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:71 12.15 12:a93a17c2:::604.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e93) -- 0x557f01fcd130 con 0x557f01fca580 2024-08-03T20:55:56.487 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::604.00000000:head# 2024-08-03T20:55:56.488 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:56.486+0000 7f765a600640 1 -- 172.21.15.32:0/3265918504 <== osd.6 v2:172.21.15.118:6825/801947386 3 ==== osd_op_reply(71 604.00000000 [create] v93'85 uv85 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f7654009830 con 0x557f01fca580 2024-08-03T20:55:56.488 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:56.486+0000 7f7660b12f40 1 -- 172.21.15.32:0/3265918504 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:72 12.15 12:a93a17c2:::604.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e93) -- 0x557f01fce140 con 0x557f01fca580 2024-08-03T20:55:56.489 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:56.487+0000 7f765a600640 1 -- 172.21.15.32:0/3265918504 <== osd.6 v2:172.21.15.118:6825/801947386 4 ==== osd_op_reply(72 604.00000000 [setxattr (30)] v93'86 uv86 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f7654009830 con 0x557f01fca580 2024-08-03T20:55:56.489 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:56.487+0000 7f7660b12f40 1 -- 172.21.15.32:0/3265918504 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:73 12.15 12:a93a17c2:::604.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e93) -- 0x557f01fce140 con 0x557f01fca580 2024-08-03T20:55:56.490 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:56.488+0000 7f765a600640 1 -- 172.21.15.32:0/3265918504 <== osd.6 v2:172.21.15.118:6825/801947386 5 ==== osd_op_reply(73 604.00000000 [setxattr (62)] v93'87 uv87 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f7654009830 con 0x557f01fca580 2024-08-03T20:55:56.490 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:56.489+0000 7f7660b12f40 1 -- 172.21.15.32:0/3265918504 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:74 12.15 12:a93a17c2:::604.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e93) -- 0x557f01fcd130 con 0x557f01fca580 2024-08-03T20:55:56.491 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:56.490+0000 7f765a600640 1 -- 172.21.15.32:0/3265918504 <== osd.6 v2:172.21.15.118:6825/801947386 6 ==== osd_op_reply(74 604.00000000 [omap-set-header] v93'88 uv88 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f7654009830 con 0x557f01fca580 2024-08-03T20:55:56.491 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:56.490+0000 7f7660b12f40 1 -- 172.21.15.32:0/3265918504 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:75 12.15 12:aa448500:::500.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e93) -- 0x557f01fcd130 con 0x557f01fca580 2024-08-03T20:55:56.492 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:56.491+0000 7f765a600640 1 -- 172.21.15.32:0/3265918504 <== osd.6 v2:172.21.15.118:6825/801947386 7 ==== osd_op_reply(75 500.00000000 [create] v93'89 uv82 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f7654009830 con 0x557f01fca580 2024-08-03T20:55:56.492 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:56.491+0000 7f7660b12f40 1 -- 172.21.15.32:0/3265918504 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:76 12.15 12:aa448500:::500.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) -- 0x557f01fcd130 con 0x557f01fca580 2024-08-03T20:55:56.494 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:56.492+0000 7f765a600640 1 -- 172.21.15.32:0/3265918504 <== osd.6 v2:172.21.15.118:6825/801947386 8 ==== osd_op_reply(76 500.00000000 [delete] v93'90 uv90 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f7654009830 con 0x557f01fca580 2024-08-03T20:55:56.494 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:56.492+0000 7f7660b12f40 1 -- 172.21.15.32:0/3265918504 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:77 12.15 12:aa448500:::500.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e93) -- 0x557f01fcd130 con 0x557f01fca580 2024-08-03T20:55:56.495 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::500.00000000:head# 2024-08-03T20:55:56.495 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:56.493+0000 7f765a600640 1 -- 172.21.15.32:0/3265918504 <== osd.6 v2:172.21.15.118:6825/801947386 9 ==== osd_op_reply(77 500.00000000 [create] v93'91 uv91 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f7654009830 con 0x557f01fca580 2024-08-03T20:55:56.495 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:56.494+0000 7f7660b12f40 1 -- 172.21.15.32:0/3265918504 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:78 12.15 12:aa448500:::500.00000000:head [write 0~90 in=90b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e93) -- 0x557f01fcd130 con 0x557f01fca580 2024-08-03T20:55:56.497 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:56.495+0000 7f765a600640 1 -- 172.21.15.32:0/3265918504 <== osd.6 v2:172.21.15.118:6825/801947386 10 ==== osd_op_reply(78 500.00000000 [write 0~90] v93'92 uv92 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f7654009830 con 0x557f01fca580 2024-08-03T20:55:56.497 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:56.495+0000 7f7660b12f40 1 -- 172.21.15.32:0/3265918504 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:79 12.15 12:aa448500:::500.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e93) -- 0x557f01fcd130 con 0x557f01fca580 2024-08-03T20:55:56.498 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:56.496+0000 7f765a600640 1 -- 172.21.15.32:0/3265918504 <== osd.6 v2:172.21.15.118:6825/801947386 11 ==== osd_op_reply(79 500.00000000 [omap-set-header] v93'93 uv93 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f7654009830 con 0x557f01fca580 2024-08-03T20:55:56.498 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:56.496+0000 7f7660b12f40 1 --2- 172.21.15.32:0/3265918504 >> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] conn(0x557f01fcda50 0x557f01fcde60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:55:56.498 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:56.496+0000 7f7660b12f40 1 -- 172.21.15.32:0/3265918504 --> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] -- osd_op(unknown.0.0:80 12.d 12:b50e409b:::mds_snaptable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e93) -- 0x557f01fd0580 con 0x557f01fcda50 2024-08-03T20:55:56.498 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:56.497+0000 7f7659c00640 1 --2- 172.21.15.32:0/3265918504 >> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] conn(0x557f01fcda50 0x557f01fcde60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:55:56.499 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:56.498+0000 7f7659c00640 1 --2- 172.21.15.32:0/3265918504 >> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] conn(0x557f01fcda50 0x557f01fcde60 crc :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:55:56.500 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:56.499+0000 7f7659c00640 1 -- 172.21.15.32:0/3265918504 <== osd.7 v2:172.21.15.118:6818/653000048 1 ==== osd_op_reply(80 mds_snaptable [create] v93'37 uv36 ondisk = -17 ((17) File exists)) ==== 157+0+0 (crc 0 0 0) 0x7f764c056470 con 0x557f01fcda50 2024-08-03T20:55:56.500 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:56.499+0000 7f7660b12f40 1 -- 172.21.15.32:0/3265918504 --> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] -- osd_op(unknown.0.0:81 12.d 12:b50e409b:::mds_snaptable:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) -- 0x557f01fd0580 con 0x557f01fcda50 2024-08-03T20:55:56.501 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:56.500+0000 7f7659c00640 1 -- 172.21.15.32:0/3265918504 <== osd.7 v2:172.21.15.118:6818/653000048 2 ==== osd_op_reply(81 mds_snaptable [delete] v93'38 uv38 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f764c056470 con 0x557f01fcda50 2024-08-03T20:55:56.501 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:56.500+0000 7f7660b12f40 1 -- 172.21.15.32:0/3265918504 --> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] -- osd_op(unknown.0.0:82 12.d 12:b50e409b:::mds_snaptable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e93) -- 0x557f01fd0970 con 0x557f01fcda50 2024-08-03T20:55:56.502 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:56.501+0000 7f7659c00640 1 -- 172.21.15.32:0/3265918504 <== osd.7 v2:172.21.15.118:6818/653000048 3 ==== osd_op_reply(82 mds_snaptable [create] v93'39 uv39 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f764c056470 con 0x557f01fcda50 2024-08-03T20:55:56.503 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds_snaptable:head# 2024-08-03T20:55:56.503 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:56.501+0000 7f7660b12f40 1 -- 172.21.15.32:0/3265918504 --> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] -- osd_op(unknown.0.0:83 12.d 12:b50e409b:::mds_snaptable:head [write 0~70 in=70b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e93) -- 0x557f01fd0970 con 0x557f01fcda50 2024-08-03T20:55:56.504 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:56.503+0000 7f7659c00640 1 -- 172.21.15.32:0/3265918504 <== osd.7 v2:172.21.15.118:6818/653000048 4 ==== osd_op_reply(83 mds_snaptable [write 0~70] v93'40 uv40 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f764c056470 con 0x557f01fcda50 2024-08-03T20:55:56.504 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:56.503+0000 7f7660b12f40 1 -- 172.21.15.32:0/3265918504 --> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] -- osd_op(unknown.0.0:84 12.d 12:b50e409b:::mds_snaptable:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e93) -- 0x557f01fd0970 con 0x557f01fcda50 2024-08-03T20:55:56.506 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:56.504+0000 7f7659c00640 1 -- 172.21.15.32:0/3265918504 <== osd.7 v2:172.21.15.118:6818/653000048 5 ==== osd_op_reply(84 mds_snaptable [omap-set-header] v93'41 uv41 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f764c056470 con 0x557f01fcda50 2024-08-03T20:55:56.506 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:56.505+0000 7f7660b12f40 1 -- 172.21.15.32:0/3265918504 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:85 12.1d 12:b871830b:::605.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e93) -- 0x557f01fd0970 con 0x557f01fbd760 2024-08-03T20:55:56.507 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:56.506+0000 7f765a600640 1 -- 172.21.15.32:0/3265918504 <== osd.0 v2:172.21.15.32:6816/3066156116 44 ==== osd_op_reply(85 605.00000000 [create] v93'45 uv44 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f7654009830 con 0x557f01fbd760 2024-08-03T20:55:56.507 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:56.506+0000 7f7660b12f40 1 -- 172.21.15.32:0/3265918504 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:86 12.1d 12:b871830b:::605.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) -- 0x557f01fd0970 con 0x557f01fbd760 2024-08-03T20:55:56.509 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:56.507+0000 7f765a600640 1 -- 172.21.15.32:0/3265918504 <== osd.0 v2:172.21.15.32:6816/3066156116 45 ==== osd_op_reply(86 605.00000000 [delete] v93'46 uv46 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f7654009830 con 0x557f01fbd760 2024-08-03T20:55:56.509 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:56.507+0000 7f7660b12f40 1 -- 172.21.15.32:0/3265918504 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:87 12.1d 12:b871830b:::605.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e93) -- 0x557f01fd0970 con 0x557f01fbd760 2024-08-03T20:55:56.510 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::605.00000000:head# 2024-08-03T20:55:56.510 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:56.509+0000 7f765a600640 1 -- 172.21.15.32:0/3265918504 <== osd.0 v2:172.21.15.32:6816/3066156116 46 ==== osd_op_reply(87 605.00000000 [create] v93'47 uv47 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f7654009830 con 0x557f01fbd760 2024-08-03T20:55:56.510 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:56.509+0000 7f7660b12f40 1 -- 172.21.15.32:0/3265918504 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:88 12.1d 12:b871830b:::605.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e93) -- 0x557f01fd1980 con 0x557f01fbd760 2024-08-03T20:55:56.511 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:56.510+0000 7f765a600640 1 -- 172.21.15.32:0/3265918504 <== osd.0 v2:172.21.15.32:6816/3066156116 47 ==== osd_op_reply(88 605.00000000 [setxattr (30)] v93'48 uv48 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f7654009830 con 0x557f01fbd760 2024-08-03T20:55:56.511 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:56.510+0000 7f7660b12f40 1 -- 172.21.15.32:0/3265918504 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:89 12.1d 12:b871830b:::605.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e93) -- 0x557f01fd1980 con 0x557f01fbd760 2024-08-03T20:55:56.513 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:56.511+0000 7f765a600640 1 -- 172.21.15.32:0/3265918504 <== osd.0 v2:172.21.15.32:6816/3066156116 48 ==== osd_op_reply(89 605.00000000 [setxattr (62)] v93'49 uv49 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f7654009830 con 0x557f01fbd760 2024-08-03T20:55:56.513 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:56.511+0000 7f7660b12f40 1 -- 172.21.15.32:0/3265918504 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:90 12.1d 12:b871830b:::605.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e93) -- 0x557f01fd0970 con 0x557f01fbd760 2024-08-03T20:55:56.514 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:56.513+0000 7f765a600640 1 -- 172.21.15.32:0/3265918504 <== osd.0 v2:172.21.15.32:6816/3066156116 49 ==== osd_op_reply(90 605.00000000 [omap-set-header] v93'50 uv50 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f7654009830 con 0x557f01fbd760 2024-08-03T20:55:56.514 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:56.513+0000 7f7660b12f40 1 -- 172.21.15.32:0/3265918504 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:91 12.13 12:c91d4a1d:::mds0_inotable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e93) -- 0x557f01fd0970 con 0x557f01fc5eb0 2024-08-03T20:55:56.518 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:56.514+0000 7f7659200640 1 -- 172.21.15.32:0/3265918504 <== osd.1 v2:172.21.15.32:6824/1551241759 5 ==== osd_op_reply(91 mds0_inotable [create] v93'74 uv68 ondisk = -17 ((17) File exists)) ==== 157+0+0 (crc 0 0 0) 0x7f764805c0a0 con 0x557f01fc5eb0 2024-08-03T20:55:56.519 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:56.514+0000 7f7660b12f40 1 -- 172.21.15.32:0/3265918504 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:92 12.13 12:c91d4a1d:::mds0_inotable:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) -- 0x557f01fd0d10 con 0x557f01fc5eb0 2024-08-03T20:55:56.519 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:56.515+0000 7f7659200640 1 -- 172.21.15.32:0/3265918504 <== osd.1 v2:172.21.15.32:6824/1551241759 6 ==== osd_op_reply(92 mds0_inotable [delete] v93'75 uv75 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f764805c0a0 con 0x557f01fc5eb0 2024-08-03T20:55:56.519 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:56.515+0000 7f7660b12f40 1 -- 172.21.15.32:0/3265918504 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:93 12.13 12:c91d4a1d:::mds0_inotable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e93) -- 0x557f01fd10b0 con 0x557f01fc5eb0 2024-08-03T20:55:56.519 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:56.517+0000 7f7659200640 1 -- 172.21.15.32:0/3265918504 <== osd.1 v2:172.21.15.32:6824/1551241759 7 ==== osd_op_reply(93 mds0_inotable [create] v93'76 uv76 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f764805c0a0 con 0x557f01fc5eb0 2024-08-03T20:55:56.519 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds0_inotable:head# 2024-08-03T20:55:56.520 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:56.517+0000 7f7660b12f40 1 -- 172.21.15.32:0/3265918504 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:94 12.13 12:c91d4a1d:::mds0_inotable:head [write 0~34 in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e93) -- 0x557f01fc9830 con 0x557f01fc5eb0 2024-08-03T20:55:56.520 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:56.518+0000 7f7659200640 1 -- 172.21.15.32:0/3265918504 <== osd.1 v2:172.21.15.32:6824/1551241759 8 ==== osd_op_reply(94 mds0_inotable [write 0~34] v93'77 uv77 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f764805c0a0 con 0x557f01fc5eb0 2024-08-03T20:55:56.520 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:56.518+0000 7f7660b12f40 1 -- 172.21.15.32:0/3265918504 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:95 12.13 12:c91d4a1d:::mds0_inotable:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e93) -- 0x557f01fc9830 con 0x557f01fc5eb0 2024-08-03T20:55:56.521 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:56.519+0000 7f7659200640 1 -- 172.21.15.32:0/3265918504 <== osd.1 v2:172.21.15.32:6824/1551241759 9 ==== osd_op_reply(95 mds0_inotable [omap-set-header] v93'78 uv78 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f764805c0a0 con 0x557f01fc5eb0 2024-08-03T20:55:56.521 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:56.519+0000 7f7660b12f40 1 -- 172.21.15.32:0/3265918504 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:96 12.13 12:cd5a64a3:::100.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e93) -- 0x557f01fc9830 con 0x557f01fc5eb0 2024-08-03T20:55:56.522 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:56.521+0000 7f7659200640 1 -- 172.21.15.32:0/3265918504 <== osd.1 v2:172.21.15.32:6824/1551241759 10 ==== osd_op_reply(96 100.00000000 [create] v93'79 uv73 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f764805c0a0 con 0x557f01fc5eb0 2024-08-03T20:55:56.522 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:56.521+0000 7f7660b12f40 1 -- 172.21.15.32:0/3265918504 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:97 12.13 12:cd5a64a3:::100.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) -- 0x557f01fc9830 con 0x557f01fc5eb0 2024-08-03T20:55:56.523 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:56.522+0000 7f7659200640 1 -- 172.21.15.32:0/3265918504 <== osd.1 v2:172.21.15.32:6824/1551241759 11 ==== osd_op_reply(97 100.00000000 [delete] v93'80 uv80 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f764805c0a0 con 0x557f01fc5eb0 2024-08-03T20:55:56.523 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:56.522+0000 7f7660b12f40 1 -- 172.21.15.32:0/3265918504 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:98 12.13 12:cd5a64a3:::100.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e93) -- 0x557f01fc9830 con 0x557f01fc5eb0 2024-08-03T20:55:56.524 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::100.00000000:head# 2024-08-03T20:55:56.525 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:56.523+0000 7f7659200640 1 -- 172.21.15.32:0/3265918504 <== osd.1 v2:172.21.15.32:6824/1551241759 12 ==== osd_op_reply(98 100.00000000 [create] v93'81 uv81 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f764805c0a0 con 0x557f01fc5eb0 2024-08-03T20:55:56.525 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:56.523+0000 7f7660b12f40 1 -- 172.21.15.32:0/3265918504 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:99 12.13 12:cd5a64a3:::100.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e93) -- 0x557f01fc9830 con 0x557f01fc5eb0 2024-08-03T20:55:56.526 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:56.524+0000 7f7659200640 1 -- 172.21.15.32:0/3265918504 <== osd.1 v2:172.21.15.32:6824/1551241759 13 ==== osd_op_reply(99 100.00000000 [omap-set-header] v93'82 uv82 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f764805c0a0 con 0x557f01fc5eb0 2024-08-03T20:55:56.526 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:56.525+0000 7f7660b12f40 1 -- 172.21.15.32:0/3265918504 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:100 12.13 12:cd5a64a3:::100.00000000:head [omap-set-vals in=5194b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e93) -- 0x557f01fd3f70 con 0x557f01fc5eb0 2024-08-03T20:55:56.527 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:56.526+0000 7f7659200640 1 -- 172.21.15.32:0/3265918504 <== osd.1 v2:172.21.15.32:6824/1551241759 14 ==== osd_op_reply(100 100.00000000 [omap-set-vals] v93'83 uv83 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f764805c0a0 con 0x557f01fc5eb0 2024-08-03T20:55:56.527 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:56.526+0000 7f7660b12f40 1 --2- 172.21.15.32:0/3265918504 >> [v2:172.21.15.32:6800/9088303,v1:172.21.15.32:6801/9088303] conn(0x557f01fd10b0 0x557f01fd34d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:55:56.528 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:56.526+0000 7f7660b12f40 1 -- 172.21.15.32:0/3265918504 --> [v2:172.21.15.32:6800/9088303,v1:172.21.15.32:6801/9088303] -- osd_op(unknown.0.0:101 12.b 12:d0630e4c:::mds0_sessionmap:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e93) -- 0x557f01fd4630 con 0x557f01fd10b0 2024-08-03T20:55:56.528 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:56.526+0000 7f7659200640 1 --2- 172.21.15.32:0/3265918504 >> [v2:172.21.15.32:6800/9088303,v1:172.21.15.32:6801/9088303] conn(0x557f01fd10b0 0x557f01fd34d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:55:56.528 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:56.527+0000 7f7659200640 1 --2- 172.21.15.32:0/3265918504 >> [v2:172.21.15.32:6800/9088303,v1:172.21.15.32:6801/9088303] conn(0x557f01fd10b0 0x557f01fd34d0 crc :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:55:56.529 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:56.528+0000 7f7659200640 1 -- 172.21.15.32:0/3265918504 <== osd.3 v2:172.21.15.32:6800/9088303 1 ==== osd_op_reply(101 mds0_sessionmap [create] v93'36 uv34 ondisk = -17 ((17) File exists)) ==== 159+0+0 (crc 0 0 0) 0x7f764805c0a0 con 0x557f01fd10b0 2024-08-03T20:55:56.530 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:56.528+0000 7f7660b12f40 1 -- 172.21.15.32:0/3265918504 --> [v2:172.21.15.32:6800/9088303,v1:172.21.15.32:6801/9088303] -- osd_op(unknown.0.0:102 12.b 12:d0630e4c:::mds0_sessionmap:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) -- 0x557f01fd4630 con 0x557f01fd10b0 2024-08-03T20:55:56.531 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:56.530+0000 7f7659200640 1 -- 172.21.15.32:0/3265918504 <== osd.3 v2:172.21.15.32:6800/9088303 2 ==== osd_op_reply(102 mds0_sessionmap [delete] v93'37 uv37 ondisk = 0) ==== 159+0+0 (crc 0 0 0) 0x7f764805c0a0 con 0x557f01fd10b0 2024-08-03T20:55:56.531 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:56.530+0000 7f7660b12f40 1 -- 172.21.15.32:0/3265918504 --> [v2:172.21.15.32:6800/9088303,v1:172.21.15.32:6801/9088303] -- osd_op(unknown.0.0:103 12.b 12:d0630e4c:::mds0_sessionmap:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e93) -- 0x557f01fd4630 con 0x557f01fd10b0 2024-08-03T20:55:56.532 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:56.531+0000 7f7659200640 1 -- 172.21.15.32:0/3265918504 <== osd.3 v2:172.21.15.32:6800/9088303 3 ==== osd_op_reply(103 mds0_sessionmap [create] v93'38 uv38 ondisk = 0) ==== 159+0+0 (crc 0 0 0) 0x7f764805c0a0 con 0x557f01fd10b0 2024-08-03T20:55:56.533 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds0_sessionmap:head# 2024-08-03T20:55:56.533 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:56.531+0000 7f7660b12f40 1 -- 172.21.15.32:0/3265918504 --> [v2:172.21.15.32:6800/9088303,v1:172.21.15.32:6801/9088303] -- osd_op(unknown.0.0:104 12.b 12:d0630e4c:::mds0_sessionmap:head [omap-set-header in=14b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e93) -- 0x557f01fd46b0 con 0x557f01fd10b0 2024-08-03T20:55:56.534 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:56.532+0000 7f7659200640 1 -- 172.21.15.32:0/3265918504 <== osd.3 v2:172.21.15.32:6800/9088303 4 ==== osd_op_reply(104 mds0_sessionmap [omap-set-header] v93'39 uv39 ondisk = 0) ==== 159+0+0 (crc 0 0 0) 0x7f764805c0a0 con 0x557f01fd10b0 2024-08-03T20:55:56.534 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:56.532+0000 7f7660b12f40 1 -- 172.21.15.32:0/3265918504 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:105 12.7 12:e0b41d06:::609.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e93) -- 0x557f01fd46b0 con 0x557f01fc5eb0 2024-08-03T20:55:56.535 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:56.534+0000 7f7659200640 1 -- 172.21.15.32:0/3265918504 <== osd.1 v2:172.21.15.32:6824/1551241759 15 ==== osd_op_reply(105 609.00000000 [create] v93'45 uv44 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f764805c0a0 con 0x557f01fc5eb0 2024-08-03T20:55:56.535 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:56.534+0000 7f7660b12f40 1 -- 172.21.15.32:0/3265918504 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:106 12.7 12:e0b41d06:::609.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) -- 0x557f01fd46b0 con 0x557f01fc5eb0 2024-08-03T20:55:56.537 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:56.535+0000 7f7659200640 1 -- 172.21.15.32:0/3265918504 <== osd.1 v2:172.21.15.32:6824/1551241759 16 ==== osd_op_reply(106 609.00000000 [delete] v93'46 uv46 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f764805c0a0 con 0x557f01fc5eb0 2024-08-03T20:55:56.537 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:56.535+0000 7f7660b12f40 1 -- 172.21.15.32:0/3265918504 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:107 12.7 12:e0b41d06:::609.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e93) -- 0x557f01fd46b0 con 0x557f01fc5eb0 2024-08-03T20:55:56.538 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:56.537+0000 7f7659200640 1 -- 172.21.15.32:0/3265918504 <== osd.1 v2:172.21.15.32:6824/1551241759 17 ==== osd_op_reply(107 609.00000000 [create] v93'47 uv47 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f764805c0a0 con 0x557f01fc5eb0 2024-08-03T20:55:56.538 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::609.00000000:head# 2024-08-03T20:55:56.538 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:56.537+0000 7f7660b12f40 1 -- 172.21.15.32:0/3265918504 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:108 12.7 12:e0b41d06:::609.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e93) -- 0x557f01fd56c0 con 0x557f01fc5eb0 2024-08-03T20:55:56.539 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:56.538+0000 7f7659200640 1 -- 172.21.15.32:0/3265918504 <== osd.1 v2:172.21.15.32:6824/1551241759 18 ==== osd_op_reply(108 609.00000000 [setxattr (30)] v93'48 uv48 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f764805c0a0 con 0x557f01fc5eb0 2024-08-03T20:55:56.539 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:56.538+0000 7f7660b12f40 1 -- 172.21.15.32:0/3265918504 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:109 12.7 12:e0b41d06:::609.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e93) -- 0x557f01fd56c0 con 0x557f01fc5eb0 2024-08-03T20:55:56.541 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:56.539+0000 7f7659200640 1 -- 172.21.15.32:0/3265918504 <== osd.1 v2:172.21.15.32:6824/1551241759 19 ==== osd_op_reply(109 609.00000000 [setxattr (62)] v93'49 uv49 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f764805c0a0 con 0x557f01fc5eb0 2024-08-03T20:55:56.541 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:56.539+0000 7f7660b12f40 1 -- 172.21.15.32:0/3265918504 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:110 12.7 12:e0b41d06:::609.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e93) -- 0x557f01fd46b0 con 0x557f01fc5eb0 2024-08-03T20:55:56.542 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:56.541+0000 7f7659200640 1 -- 172.21.15.32:0/3265918504 <== osd.1 v2:172.21.15.32:6824/1551241759 20 ==== osd_op_reply(110 609.00000000 [omap-set-header] v93'50 uv50 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f764805c0a0 con 0x557f01fc5eb0 2024-08-03T20:55:56.542 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:56.541+0000 7f7660b12f40 1 -- 172.21.15.32:0/3265918504 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:111 12.f 12:f1f69726:::400.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e93) -- 0x557f01fd46b0 con 0x557f01fca580 2024-08-03T20:55:56.544 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:56.542+0000 7f765a600640 1 -- 172.21.15.32:0/3265918504 <== osd.6 v2:172.21.15.118:6825/801947386 12 ==== osd_op_reply(111 400.00000000 [create] v93'37 uv36 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f7654009830 con 0x557f01fca580 2024-08-03T20:55:56.544 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:56.543+0000 7f7660b12f40 1 -- 172.21.15.32:0/3265918504 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:112 12.f 12:f1f69726:::400.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) -- 0x557f01fd46b0 con 0x557f01fca580 2024-08-03T20:55:56.545 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:56.544+0000 7f765a600640 1 -- 172.21.15.32:0/3265918504 <== osd.6 v2:172.21.15.118:6825/801947386 13 ==== osd_op_reply(112 400.00000000 [delete] v93'38 uv38 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f7654009830 con 0x557f01fca580 2024-08-03T20:55:56.545 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:56.544+0000 7f7660b12f40 1 -- 172.21.15.32:0/3265918504 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:113 12.f 12:f1f69726:::400.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e93) -- 0x557f01fd46b0 con 0x557f01fca580 2024-08-03T20:55:56.547 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:56.546+0000 7f765a600640 1 -- 172.21.15.32:0/3265918504 <== osd.6 v2:172.21.15.118:6825/801947386 14 ==== osd_op_reply(113 400.00000000 [create] v93'39 uv39 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f7654009830 con 0x557f01fca580 2024-08-03T20:55:56.547 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::400.00000000:head# 2024-08-03T20:55:56.548 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:56.546+0000 7f7660b12f40 1 -- 172.21.15.32:0/3265918504 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:114 12.f 12:f1f69726:::400.00000000:head [write 0~22 in=22b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e93) -- 0x557f01fd3ac0 con 0x557f01fca580 2024-08-03T20:55:56.549 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:56.547+0000 7f765a600640 1 -- 172.21.15.32:0/3265918504 <== osd.6 v2:172.21.15.118:6825/801947386 15 ==== osd_op_reply(114 400.00000000 [write 0~22] v93'40 uv40 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f7654009830 con 0x557f01fca580 2024-08-03T20:55:56.549 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:56.547+0000 7f7660b12f40 1 -- 172.21.15.32:0/3265918504 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:115 12.f 12:f1f69726:::400.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e93) -- 0x557f01fd3ac0 con 0x557f01fca580 2024-08-03T20:55:56.550 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:56.549+0000 7f765a600640 1 -- 172.21.15.32:0/3265918504 <== osd.6 v2:172.21.15.118:6825/801947386 16 ==== osd_op_reply(115 400.00000000 [omap-set-header] v93'41 uv41 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f7654009830 con 0x557f01fca580 2024-08-03T20:55:56.550 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:56.549+0000 7f7660b12f40 1 -- 172.21.15.32:0/3265918504 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:116 12.1f 12:fe07bba1:::100.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e93) -- 0x557f01fd3aa0 con 0x557f01fca580 2024-08-03T20:55:56.552 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:56.551+0000 7f765a600640 1 -- 172.21.15.32:0/3265918504 <== osd.6 v2:172.21.15.118:6825/801947386 17 ==== osd_op_reply(116 100.00000000.inode [create] v93'95 uv86 ondisk = -17 ((17) File exists)) ==== 162+0+0 (crc 0 0 0) 0x7f7654009830 con 0x557f01fca580 2024-08-03T20:55:56.552 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:56.551+0000 7f7660b12f40 1 -- 172.21.15.32:0/3265918504 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:117 12.1f 12:fe07bba1:::100.00000000.inode:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) -- 0x557f01fd3aa0 con 0x557f01fca580 2024-08-03T20:55:56.553 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:56.552+0000 7f765a600640 1 -- 172.21.15.32:0/3265918504 <== osd.6 v2:172.21.15.118:6825/801947386 18 ==== osd_op_reply(117 100.00000000.inode [delete] v93'96 uv96 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7f7654009830 con 0x557f01fca580 2024-08-03T20:55:56.553 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:56.552+0000 7f7660b12f40 1 -- 172.21.15.32:0/3265918504 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:118 12.1f 12:fe07bba1:::100.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e93) -- 0x557f01fd3aa0 con 0x557f01fca580 2024-08-03T20:55:56.555 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::100.00000000.inode:head# 2024-08-03T20:55:56.555 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:56.553+0000 7f765a600640 1 -- 172.21.15.32:0/3265918504 <== osd.6 v2:172.21.15.118:6825/801947386 19 ==== osd_op_reply(118 100.00000000.inode [create] v93'97 uv97 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7f7654009830 con 0x557f01fca580 2024-08-03T20:55:56.555 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:56.553+0000 7f7660b12f40 1 -- 172.21.15.32:0/3265918504 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:119 12.1f 12:fe07bba1:::100.00000000.inode:head [write 0~582 in=582b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e93) -- 0x557f01fd3aa0 con 0x557f01fca580 2024-08-03T20:55:56.556 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:56.555+0000 7f765a600640 1 -- 172.21.15.32:0/3265918504 <== osd.6 v2:172.21.15.118:6825/801947386 20 ==== osd_op_reply(119 100.00000000.inode [write 0~582] v93'98 uv98 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7f7654009830 con 0x557f01fca580 2024-08-03T20:55:56.556 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:56.555+0000 7f7660b12f40 1 -- 172.21.15.32:0/3265918504 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:120 12.1f 12:fe07bba1:::100.00000000.inode:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e93) -- 0x557f01fd3aa0 con 0x557f01fca580 2024-08-03T20:55:56.557 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:56.556+0000 7f765a600640 1 -- 172.21.15.32:0/3265918504 <== osd.6 v2:172.21.15.118:6825/801947386 21 ==== osd_op_reply(120 100.00000000.inode [omap-set-header] v93'99 uv99 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7f7654009830 con 0x557f01fca580 2024-08-03T20:55:56.557 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:56.556+0000 7f7660b12f40 1 -- 172.21.15.32:0/3265918504 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:121 12.1f 12:ff5b34d6:::1.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e93) -- 0x557f01fd3aa0 con 0x557f01fca580 2024-08-03T20:55:56.558 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:56.557+0000 7f765a600640 1 -- 172.21.15.32:0/3265918504 <== osd.6 v2:172.21.15.118:6825/801947386 22 ==== osd_op_reply(121 1.00000000 [create] v93'100 uv93 ondisk = -17 ((17) File exists)) ==== 154+0+0 (crc 0 0 0) 0x7f7654009830 con 0x557f01fca580 2024-08-03T20:55:56.559 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:56.557+0000 7f7660b12f40 1 -- 172.21.15.32:0/3265918504 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:122 12.1f 12:ff5b34d6:::1.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) -- 0x557f01fd3aa0 con 0x557f01fca580 2024-08-03T20:55:56.560 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:56.559+0000 7f765a600640 1 -- 172.21.15.32:0/3265918504 <== osd.6 v2:172.21.15.118:6825/801947386 23 ==== osd_op_reply(122 1.00000000 [delete] v93'101 uv101 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7f7654009830 con 0x557f01fca580 2024-08-03T20:55:56.560 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:56.559+0000 7f7660b12f40 1 -- 172.21.15.32:0/3265918504 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:123 12.1f 12:ff5b34d6:::1.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e93) -- 0x557f01fd3aa0 con 0x557f01fca580 2024-08-03T20:55:56.561 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::1.00000000:head# 2024-08-03T20:55:56.562 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:56.560+0000 7f765a600640 1 -- 172.21.15.32:0/3265918504 <== osd.6 v2:172.21.15.118:6825/801947386 24 ==== osd_op_reply(123 1.00000000 [create] v93'102 uv102 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7f7654009830 con 0x557f01fca580 2024-08-03T20:55:56.562 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:56.560+0000 7f7660b12f40 1 -- 172.21.15.32:0/3265918504 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:124 12.1f 12:ff5b34d6:::1.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e93) -- 0x557f01fd3aa0 con 0x557f01fca580 2024-08-03T20:55:56.563 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:56.561+0000 7f765a600640 1 -- 172.21.15.32:0/3265918504 <== osd.6 v2:172.21.15.118:6825/801947386 25 ==== osd_op_reply(124 1.00000000 [setxattr (30)] v93'103 uv103 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7f7654009830 con 0x557f01fca580 2024-08-03T20:55:56.563 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:56.561+0000 7f7660b12f40 1 -- 172.21.15.32:0/3265918504 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:125 12.1f 12:ff5b34d6:::1.00000000:head [setxattr parent (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e93) -- 0x557f01fd3aa0 con 0x557f01fca580 2024-08-03T20:55:56.564 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:56.563+0000 7f765a600640 1 -- 172.21.15.32:0/3265918504 <== osd.6 v2:172.21.15.118:6825/801947386 26 ==== osd_op_reply(125 1.00000000 [setxattr (30)] v93'104 uv104 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7f7654009830 con 0x557f01fca580 2024-08-03T20:55:56.564 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:56.563+0000 7f7660b12f40 1 -- 172.21.15.32:0/3265918504 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:126 12.1f 12:ff5b34d6:::1.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e93) -- 0x557f01fd3aa0 con 0x557f01fca580 2024-08-03T20:55:56.566 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:56.565+0000 7f765a600640 1 -- 172.21.15.32:0/3265918504 <== osd.6 v2:172.21.15.118:6825/801947386 27 ==== osd_op_reply(126 1.00000000 [omap-set-header] v93'105 uv105 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7f7654009830 con 0x557f01fca580 2024-08-03T20:55:56.566 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:56.565+0000 7f7660b12f40 1 -- 172.21.15.32:0/3265918504 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:127 12.1f 12:ff5b34d6:::1.00000000:head [omap-set-vals in=584b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e93) -- 0x557f01fd4e10 con 0x557f01fca580 2024-08-03T20:55:56.569 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:56.566+0000 7f765a600640 1 -- 172.21.15.32:0/3265918504 <== osd.6 v2:172.21.15.118:6825/801947386 28 ==== osd_op_reply(127 1.00000000 [omap-set-vals] v93'106 uv106 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7f7654009830 con 0x557f01fca580 2024-08-03T20:55:56.569 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:56.566+0000 7f7660b12f40 1 -- 172.21.15.32:0/3265918504 >> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] conn(0x557f01fbd760 msgr2=0x557f01fbfb80 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:55:56.569 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:56.566+0000 7f7660b12f40 1 --2- 172.21.15.32:0/3265918504 >> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] conn(0x557f01fbd760 0x557f01fbfb80 crc :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:56.569 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:56.566+0000 7f7660b12f40 1 -- 172.21.15.32:0/3265918504 >> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] conn(0x557f01fc5eb0 msgr2=0x557f01fc62c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:55:56.569 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:56.566+0000 7f7660b12f40 1 --2- 172.21.15.32:0/3265918504 >> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] conn(0x557f01fc5eb0 0x557f01fc62c0 crc :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:56.569 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:56.566+0000 7f7660b12f40 1 -- 172.21.15.32:0/3265918504 >> [v2:172.21.15.32:6800/9088303,v1:172.21.15.32:6801/9088303] conn(0x557f01fd10b0 msgr2=0x557f01fd34d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:55:56.569 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:56.566+0000 7f7660b12f40 1 --2- 172.21.15.32:0/3265918504 >> [v2:172.21.15.32:6800/9088303,v1:172.21.15.32:6801/9088303] conn(0x557f01fd10b0 0x557f01fd34d0 crc :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:56.569 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:56.566+0000 7f7660b12f40 1 -- 172.21.15.32:0/3265918504 >> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] conn(0x557f01fc0fb0 msgr2=0x557f01fc40d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:55:56.569 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:56.566+0000 7f7660b12f40 1 --2- 172.21.15.32:0/3265918504 >> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] conn(0x557f01fc0fb0 0x557f01fc40d0 crc :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:56.569 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:56.566+0000 7f7660b12f40 1 -- 172.21.15.32:0/3265918504 >> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] conn(0x557f01fca580 msgr2=0x557f01fca990 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:55:56.569 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:56.566+0000 7f7660b12f40 1 --2- 172.21.15.32:0/3265918504 >> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] conn(0x557f01fca580 0x557f01fca990 crc :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:56.569 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:56.566+0000 7f7660b12f40 1 -- 172.21.15.32:0/3265918504 >> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] conn(0x557f01fcda50 msgr2=0x557f01fcde60 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:55:56.569 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:56.566+0000 7f7660b12f40 1 --2- 172.21.15.32:0/3265918504 >> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] conn(0x557f01fcda50 0x557f01fcde60 crc :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:56.570 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:56.566+0000 7f765a600640 1 -- 172.21.15.32:0/3265918504 reap_dead start 2024-08-03T20:55:56.570 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:56.566+0000 7f765a600640 1 -- 172.21.15.32:0/3265918504 reap_dead start 2024-08-03T20:55:56.570 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:56.566+0000 7f765a600640 1 -- 172.21.15.32:0/3265918504 reap_dead start 2024-08-03T20:55:56.570 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:56.567+0000 7f7660b12f40 1 -- 172.21.15.32:0/3265918504 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f763c0759f0 msgr2=0x7f763c077f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:55:56.570 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:56.567+0000 7f7660b12f40 1 --2- 172.21.15.32:0/3265918504 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f763c0759f0 0x7f763c077f00 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7f764c055000 tx=0x7f764c002fb0 comp rx=0 tx=0).stop 2024-08-03T20:55:56.570 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:56.567+0000 7f7660b12f40 1 -- 172.21.15.32:0/3265918504 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x557f01efebe0 msgr2=0x557f01ef9510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:55:56.570 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:56.567+0000 7f7660b12f40 1 --2- 172.21.15.32:0/3265918504 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x557f01efebe0 0x557f01ef9510 secure :-1 s=READY pgs=1268 cs=0 l=1 rev1=1 crypto rx=0x7f7648009480 tx=0x7f7648009570 comp rx=0 tx=0).stop 2024-08-03T20:55:56.570 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:56.567+0000 7f7660b12f40 1 -- 172.21.15.32:0/3265918504 shutdown_connections 2024-08-03T20:55:56.570 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:56.567+0000 7f7660b12f40 1 --2- 172.21.15.32:0/3265918504 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f763c0759f0 0x7f763c077f00 unknown :-1 s=CLOSED pgs=523 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:56.570 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:56.567+0000 7f7660b12f40 1 --2- 172.21.15.32:0/3265918504 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x557f01efebe0 0x557f01ef9510 unknown :-1 s=CLOSED pgs=1268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:56.570 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:56.567+0000 7f7660b12f40 1 -- 172.21.15.32:0/3265918504 >> 172.21.15.32:0/3265918504 conn(0x557f01d66ac0 msgr2=0x557f01f1e5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:55:56.570 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:56.567+0000 7f7660b12f40 1 -- 172.21.15.32:0/3265918504 shutdown_connections 2024-08-03T20:55:56.570 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:56.567+0000 7f7660b12f40 1 -- 172.21.15.32:0/3265918504 wait complete. 2024-08-03T20:55:56.583 DEBUG:teuthology.orchestra.run.smithi032:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata rm mds_snaptable 2024-08-03T20:55:56.766 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:56.765+0000 7f1a35a67f40 1 Processor -- start 2024-08-03T20:55:56.766 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:56.765+0000 7f1a35a67f40 1 -- start start 2024-08-03T20:55:56.768 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:56.766+0000 7f1a35a67f40 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b47ac1e360 0x55b47ab65f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:55:56.768 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:56.766+0000 7f1a35a67f40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55b47ac3bcb0 0x55b47ac358f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:55:56.768 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:56.766+0000 7f1a35a67f40 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55b47ac3c3f0 0x55b47ac35e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:55:56.768 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:56.766+0000 7f1a35a67f40 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55b47aab5e30 con 0x55b47ac1e360 2024-08-03T20:55:56.768 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:56.766+0000 7f1a35a67f40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x55b47ab13bf0 con 0x55b47ac3bcb0 2024-08-03T20:55:56.769 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:56.766+0000 7f1a35a67f40 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x55b47ac1f440 con 0x55b47ac3c3f0 2024-08-03T20:55:56.769 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:56.766+0000 7f1a2ec00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b47ac1e360 0x55b47ab65f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:55:56.769 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:56.766+0000 7f1a2ec00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b47ac1e360 0x55b47ab65f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:50756/0 (socket says 172.21.15.32:50756) 2024-08-03T20:55:56.769 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:56.766+0000 7f1a2ec00640 1 -- 172.21.15.32:0/1728453048 learned_addr learned my addr 172.21.15.32:0/1728453048 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:55:56.769 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:56.766+0000 7f1a2f600640 1 --2- 172.21.15.32:0/1728453048 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55b47ac3c3f0 0x55b47ac35e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:55:56.769 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:56.766+0000 7f1a2e200640 1 --2- 172.21.15.32:0/1728453048 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55b47ac3bcb0 0x55b47ac358f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:55:56.769 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:56.767+0000 7f1a2ec00640 1 -- 172.21.15.32:0/1728453048 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55b47ac3c3f0 msgr2=0x55b47ac35e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:55:56.769 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:56.767+0000 7f1a2ec00640 1 --2- 172.21.15.32:0/1728453048 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55b47ac3c3f0 0x55b47ac35e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:56.769 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:56.767+0000 7f1a2ec00640 1 -- 172.21.15.32:0/1728453048 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55b47ac3bcb0 msgr2=0x55b47ac358f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:55:56.769 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:56.767+0000 7f1a2ec00640 1 --2- 172.21.15.32:0/1728453048 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55b47ac3bcb0 0x55b47ac358f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:56.769 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:56.767+0000 7f1a2ec00640 1 -- 172.21.15.32:0/1728453048 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b47ac3a910 con 0x55b47ac1e360 2024-08-03T20:55:56.769 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:56.767+0000 7f1a2e200640 1 --2- 172.21.15.32:0/1728453048 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55b47ac3bcb0 0x55b47ac358f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-03T20:55:56.769 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:56.767+0000 7f1a2ec00640 1 --2- 172.21.15.32:0/1728453048 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b47ac1e360 0x55b47ab65f90 secure :-1 s=READY pgs=1270 cs=0 l=1 rev1=1 crypto rx=0x7f1a2000b600 tx=0x7f1a2000bac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4a9816a1638c824c server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:55:56.771 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:56.767+0000 7f1a2ce00640 1 -- 172.21.15.32:0/1728453048 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a2000a8f0 con 0x55b47ac1e360 2024-08-03T20:55:56.771 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:56.767+0000 7f1a2ce00640 1 -- 172.21.15.32:0/1728453048 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f1a2000aa90 con 0x55b47ac1e360 2024-08-03T20:55:56.771 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:56.767+0000 7f1a2ce00640 1 -- 172.21.15.32:0/1728453048 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a20008450 con 0x55b47ac1e360 2024-08-03T20:55:56.771 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:56.768+0000 7f1a35a67f40 1 -- 172.21.15.32:0/1728453048 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b47ac1e360 msgr2=0x55b47ab65f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:55:56.771 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:56.768+0000 7f1a35a67f40 1 --2- 172.21.15.32:0/1728453048 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b47ac1e360 0x55b47ab65f90 secure :-1 s=READY pgs=1270 cs=0 l=1 rev1=1 crypto rx=0x7f1a2000b600 tx=0x7f1a2000bac0 comp rx=0 tx=0).stop 2024-08-03T20:55:56.771 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:56.768+0000 7f1a35a67f40 1 -- 172.21.15.32:0/1728453048 shutdown_connections 2024-08-03T20:55:56.771 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:56.768+0000 7f1a35a67f40 1 --2- 172.21.15.32:0/1728453048 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55b47ac3bcb0 0x55b47ac358f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:56.771 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:56.768+0000 7f1a35a67f40 1 --2- 172.21.15.32:0/1728453048 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55b47ac3c3f0 0x55b47ac35e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:56.771 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:56.768+0000 7f1a35a67f40 1 --2- 172.21.15.32:0/1728453048 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b47ac1e360 0x55b47ab65f90 unknown :-1 s=CLOSED pgs=1270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:56.771 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:56.768+0000 7f1a35a67f40 1 -- 172.21.15.32:0/1728453048 >> 172.21.15.32:0/1728453048 conn(0x55b47aa87ac0 msgr2=0x55b47ac34610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:55:56.772 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:56.768+0000 7f1a35a67f40 1 -- 172.21.15.32:0/1728453048 shutdown_connections 2024-08-03T20:55:56.772 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:56.768+0000 7f1a35a67f40 1 -- 172.21.15.32:0/1728453048 wait complete. 2024-08-03T20:55:56.772 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:56.769+0000 7f1a35a67f40 1 Processor -- start 2024-08-03T20:55:56.772 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:56.769+0000 7f1a35a67f40 1 -- start start 2024-08-03T20:55:56.774 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:56.771+0000 7f1a35a67f40 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b47ac1e360 0x55b47ac169f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:55:56.774 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:56.771+0000 7f1a35a67f40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55b47ac3bcb0 0x55b47aba8310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:55:56.774 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:56.771+0000 7f1a35a67f40 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55b47ac3c3f0 0x55b47aba8850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:55:56.774 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:56.771+0000 7f1a35a67f40 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55b47ac3e300 con 0x55b47ac1e360 2024-08-03T20:55:56.774 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:56.771+0000 7f1a35a67f40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x55b47ac3a660 con 0x55b47ac3bcb0 2024-08-03T20:55:56.774 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:56.771+0000 7f1a35a67f40 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x55b47ac3acb0 con 0x55b47ac3c3f0 2024-08-03T20:55:56.774 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:56.771+0000 7f1a2ec00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b47ac1e360 0x55b47ac169f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:55:56.774 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:56.771+0000 7f1a2ec00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b47ac1e360 0x55b47ac169f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:50768/0 (socket says 172.21.15.32:50768) 2024-08-03T20:55:56.774 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:56.771+0000 7f1a2ec00640 1 -- 172.21.15.32:0/2026773555 learned_addr learned my addr 172.21.15.32:0/2026773555 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:55:56.774 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:56.771+0000 7f1a2e200640 1 --2- 172.21.15.32:0/2026773555 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55b47ac3bcb0 0x55b47aba8310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:55:56.775 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:56.771+0000 7f1a2f600640 1 --2- 172.21.15.32:0/2026773555 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55b47ac3c3f0 0x55b47aba8850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:55:56.775 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:56.772+0000 7f1a2ec00640 1 -- 172.21.15.32:0/2026773555 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55b47ac3c3f0 msgr2=0x55b47aba8850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:55:56.775 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:56.772+0000 7f1a2ec00640 1 --2- 172.21.15.32:0/2026773555 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55b47ac3c3f0 0x55b47aba8850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:56.775 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:56.772+0000 7f1a2ec00640 1 -- 172.21.15.32:0/2026773555 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55b47ac3bcb0 msgr2=0x55b47aba8310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:55:56.775 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:56.772+0000 7f1a2ec00640 1 --2- 172.21.15.32:0/2026773555 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55b47ac3bcb0 0x55b47aba8310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:56.775 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:56.772+0000 7f1a2ec00640 1 -- 172.21.15.32:0/2026773555 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b47ac363c0 con 0x55b47ac1e360 2024-08-03T20:55:56.775 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:56.772+0000 7f1a2ec00640 1 --2- 172.21.15.32:0/2026773555 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b47ac1e360 0x55b47ac169f0 secure :-1 s=READY pgs=1271 cs=0 l=1 rev1=1 crypto rx=0x7f1a20016630 tx=0x7f1a20016660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:55:56.775 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:56.772+0000 7f1a27400640 1 -- 172.21.15.32:0/2026773555 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a20016a90 con 0x55b47ac1e360 2024-08-03T20:55:56.775 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:56.772+0000 7f1a27400640 1 -- 172.21.15.32:0/2026773555 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f1a20016c30 con 0x55b47ac1e360 2024-08-03T20:55:56.775 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:56.772+0000 7f1a27400640 1 -- 172.21.15.32:0/2026773555 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a20020420 con 0x55b47ac1e360 2024-08-03T20:55:56.775 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:56.772+0000 7f1a35a67f40 1 -- 172.21.15.32:0/2026773555 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x55b47ac19840 con 0x55b47ac1e360 2024-08-03T20:55:56.775 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:56.772+0000 7f1a35a67f40 1 -- 172.21.15.32:0/2026773555 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b47ac1a7b0 con 0x55b47ac1e360 2024-08-03T20:55:56.776 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:56.774+0000 7f1a27400640 1 -- 172.21.15.32:0/2026773555 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f1a20029440 con 0x55b47ac1e360 2024-08-03T20:55:56.777 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:56.775+0000 7f1a27400640 1 --2- 172.21.15.32:0/2026773555 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f1a100759f0 0x7f1a10077f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:55:56.777 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:56.775+0000 7f1a27400640 1 -- 172.21.15.32:0/2026773555 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 6542+0+0 (secure 0 0 0) 0x7f1a200995f0 con 0x55b47ac1e360 2024-08-03T20:55:56.777 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:56.776+0000 7f1a35a67f40 1 --2- 172.21.15.32:0/2026773555 >> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] conn(0x55b47ac18ca0 0x55b47ac190b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:55:56.777 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:56.776+0000 7f1a2e200640 1 --2- 172.21.15.32:0/2026773555 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f1a100759f0 0x7f1a10077f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:55:56.778 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:56.776+0000 7f1a35a67f40 1 -- 172.21.15.32:0/2026773555 --> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] -- osd_op(unknown.0.0:1 12.d 12:b50e409b:::mds_snaptable:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) -- 0x55b47ac1acd0 con 0x55b47ac18ca0 2024-08-03T20:55:56.778 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:56.776+0000 7f1a2f600640 1 --2- 172.21.15.32:0/2026773555 >> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] conn(0x55b47ac18ca0 0x55b47ac190b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:55:56.778 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:56.777+0000 7f1a2e200640 1 --2- 172.21.15.32:0/2026773555 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f1a100759f0 0x7f1a10077f00 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x55b47abb1400 tx=0x7f1a1c003810 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:55:56.778 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:56.777+0000 7f1a2f600640 1 --2- 172.21.15.32:0/2026773555 >> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] conn(0x55b47ac18ca0 0x55b47ac190b0 crc :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:55:56.781 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:56.779+0000 7f1a2f600640 1 -- 172.21.15.32:0/2026773555 <== osd.7 v2:172.21.15.118:6818/653000048 1 ==== osd_op_reply(1 mds_snaptable [delete] v93'42 uv42 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f1a2800b040 con 0x55b47ac18ca0 2024-08-03T20:55:56.781 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:56.779+0000 7f1a35a67f40 1 -- 172.21.15.32:0/2026773555 >> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] conn(0x55b47ac18ca0 msgr2=0x55b47ac190b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:55:56.781 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:56.779+0000 7f1a35a67f40 1 --2- 172.21.15.32:0/2026773555 >> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] conn(0x55b47ac18ca0 0x55b47ac190b0 crc :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:56.781 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:56.779+0000 7f1a35a67f40 1 -- 172.21.15.32:0/2026773555 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f1a100759f0 msgr2=0x7f1a10077f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:55:56.781 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:56.779+0000 7f1a35a67f40 1 --2- 172.21.15.32:0/2026773555 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f1a100759f0 0x7f1a10077f00 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x55b47abb1400 tx=0x7f1a1c003810 comp rx=0 tx=0).stop 2024-08-03T20:55:56.781 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:56.779+0000 7f1a35a67f40 1 -- 172.21.15.32:0/2026773555 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b47ac1e360 msgr2=0x55b47ac169f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:55:56.781 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:56.779+0000 7f1a35a67f40 1 --2- 172.21.15.32:0/2026773555 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b47ac1e360 0x55b47ac169f0 secure :-1 s=READY pgs=1271 cs=0 l=1 rev1=1 crypto rx=0x7f1a20016630 tx=0x7f1a20016660 comp rx=0 tx=0).stop 2024-08-03T20:55:56.781 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:56.779+0000 7f1a2f600640 1 -- 172.21.15.32:0/2026773555 reap_dead start 2024-08-03T20:55:56.781 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:56.779+0000 7f1a35a67f40 1 -- 172.21.15.32:0/2026773555 shutdown_connections 2024-08-03T20:55:56.781 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:56.779+0000 7f1a35a67f40 1 -- 172.21.15.32:0/2026773555 >> 172.21.15.32:0/2026773555 conn(0x55b47aa87ac0 msgr2=0x55b47ab65f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:55:56.781 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:56.779+0000 7f1a35a67f40 1 -- 172.21.15.32:0/2026773555 shutdown_connections 2024-08-03T20:55:56.782 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:56.779+0000 7f1a35a67f40 1 -- 172.21.15.32:0/2026773555 wait complete. 2024-08-03T20:55:56.792 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs joinable true 2024-08-03T20:55:57.024 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:57.022+0000 7f5b2f200640 1 -- 172.21.15.32:0/1125922023 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f5b280af520 msgr2=0x7f5b280af930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:55:57.024 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:57.022+0000 7f5b2f200640 1 --2- 172.21.15.32:0/1125922023 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f5b280af520 0x7f5b280af930 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f5b10009830 tx=0x7f5b1002a540 comp rx=0 tx=0).stop 2024-08-03T20:55:57.024 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:57.022+0000 7f5b2f200640 1 -- 172.21.15.32:0/1125922023 shutdown_connections 2024-08-03T20:55:57.024 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:57.022+0000 7f5b2f200640 1 --2- 172.21.15.32:0/1125922023 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f5b280af520 0x7f5b280af930 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:57.024 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:57.022+0000 7f5b2f200640 1 -- 172.21.15.32:0/1125922023 >> 172.21.15.32:0/1125922023 conn(0x7f5b280cd1e0 msgr2=0x7f5b280cd5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:55:57.024 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:57.022+0000 7f5b2f200640 1 -- 172.21.15.32:0/1125922023 shutdown_connections 2024-08-03T20:55:57.025 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:57.022+0000 7f5b2f200640 1 -- 172.21.15.32:0/1125922023 wait complete. 2024-08-03T20:55:57.026 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:57.024+0000 7f5b2f200640 1 Processor -- start 2024-08-03T20:55:57.026 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:57.025+0000 7f5b2f200640 1 -- start start 2024-08-03T20:55:57.028 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:57.026+0000 7f5b2f200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5b2806f600 0x7f5b2814cdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:55:57.028 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:57.026+0000 7f5b2f200640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f5b28075a10 0x7f5b28152d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:55:57.029 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:57.026+0000 7f5b2f200640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f5b280af520 0x7f5b281f6df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:55:57.029 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:57.026+0000 7f5b2f200640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f5b280b0060 con 0x7f5b2806f600 2024-08-03T20:55:57.029 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:57.026+0000 7f5b2f200640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f5b2806de40 con 0x7f5b28075a10 2024-08-03T20:55:57.029 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:57.026+0000 7f5b2f200640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7f5b28070120 con 0x7f5b280af520 2024-08-03T20:55:57.029 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:57.026+0000 7f5b27e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5b2806f600 0x7f5b2814cdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:55:57.029 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:57.026+0000 7f5b27e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5b2806f600 0x7f5b2814cdd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:50774/0 (socket says 172.21.15.32:50774) 2024-08-03T20:55:57.029 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:57.026+0000 7f5b27e00640 1 -- 172.21.15.32:0/3481755578 learned_addr learned my addr 172.21.15.32:0/3481755578 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:55:57.029 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:57.027+0000 7f5b1f400640 1 --2- 172.21.15.32:0/3481755578 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f5b28075a10 0x7f5b28152d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:55:57.029 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:57.027+0000 7f5b2d200640 1 --2- 172.21.15.32:0/3481755578 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f5b280af520 0x7f5b281f6df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:55:57.029 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:57.027+0000 7f5b27e00640 1 -- 172.21.15.32:0/3481755578 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f5b280af520 msgr2=0x7f5b281f6df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:55:57.029 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:57.027+0000 7f5b27e00640 1 --2- 172.21.15.32:0/3481755578 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f5b280af520 0x7f5b281f6df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:57.029 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:57.027+0000 7f5b27e00640 1 -- 172.21.15.32:0/3481755578 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f5b28075a10 msgr2=0x7f5b28152d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:55:57.029 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:57.027+0000 7f5b27e00640 1 --2- 172.21.15.32:0/3481755578 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f5b28075a10 0x7f5b28152d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:57.030 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:57.027+0000 7f5b27e00640 1 -- 172.21.15.32:0/3481755578 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5b28153aa0 con 0x7f5b2806f600 2024-08-03T20:55:57.030 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:57.027+0000 7f5b27e00640 1 --2- 172.21.15.32:0/3481755578 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5b2806f600 0x7f5b2814cdd0 secure :-1 s=READY pgs=1273 cs=0 l=1 rev1=1 crypto rx=0x7f5b1800c4d0 tx=0x7f5b1800c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:55:57.030 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:57.027+0000 7f5b24c00640 1 -- 172.21.15.32:0/3481755578 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b18016020 con 0x7f5b2806f600 2024-08-03T20:55:57.030 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:57.027+0000 7f5b24c00640 1 -- 172.21.15.32:0/3481755578 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f5b18012920 con 0x7f5b2806f600 2024-08-03T20:55:57.031 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:57.027+0000 7f5b2f200640 1 -- 172.21.15.32:0/3481755578 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5b280b4640 con 0x7f5b2806f600 2024-08-03T20:55:57.031 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:57.028+0000 7f5b24c00640 1 -- 172.21.15.32:0/3481755578 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b18012d30 con 0x7f5b2806f600 2024-08-03T20:55:57.031 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:57.028+0000 7f5b2f200640 1 -- 172.21.15.32:0/3481755578 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5b28154060 con 0x7f5b2806f600 2024-08-03T20:55:57.031 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:57.029+0000 7f5b2f200640 1 -- 172.21.15.32:0/3481755578 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5b280af930 con 0x7f5b2806f600 2024-08-03T20:55:57.031 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:57.030+0000 7f5b24c00640 1 -- 172.21.15.32:0/3481755578 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f5b18019020 con 0x7f5b2806f600 2024-08-03T20:55:57.032 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:57.031+0000 7f5b24c00640 1 --2- 172.21.15.32:0/3481755578 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f5b04077010 0x7f5b04079520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:55:57.032 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:57.031+0000 7f5b24c00640 1 -- 172.21.15.32:0/3481755578 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 6542+0+0 (secure 0 0 0) 0x7f5b18098cb0 con 0x7f5b2806f600 2024-08-03T20:55:57.033 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:57.031+0000 7f5b1f400640 1 --2- 172.21.15.32:0/3481755578 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f5b04077010 0x7f5b04079520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:55:57.033 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:57.032+0000 7f5b1f400640 1 --2- 172.21.15.32:0/3481755578 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f5b04077010 0x7f5b04079520 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7f5b10034000 tx=0x7f5b10004730 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:55:57.036 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:57.035+0000 7f5b24c00640 1 -- 172.21.15.32:0/3481755578 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7f5b18020020 con 0x7f5b2806f600 2024-08-03T20:55:57.286 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:57.285+0000 7f5b2f200640 1 -- 172.21.15.32:0/3481755578 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "joinable", "val": "true"} v 0) -- 0x7f5b2806aa50 con 0x7f5b2806f600 2024-08-03T20:55:58.198 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:58.196+0000 7f5b24c00640 1 -- 172.21.15.32:0/3481755578 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "joinable", "val": "true"}]=0 cephfs marked joinable; MDS may join as newly active. v140) ==== 162+0+0 (secure 0 0 0) 0x7f5b18061510 con 0x7f5b2806f600 2024-08-03T20:55:58.198 INFO:teuthology.orchestra.run.smithi032.stderr:cephfs marked joinable; MDS may join as newly active. 2024-08-03T20:55:58.203 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:58.201+0000 7f5b2f200640 1 -- 172.21.15.32:0/3481755578 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f5b04077010 msgr2=0x7f5b04079520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:55:58.204 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:58.201+0000 7f5b2f200640 1 --2- 172.21.15.32:0/3481755578 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f5b04077010 0x7f5b04079520 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7f5b10034000 tx=0x7f5b10004730 comp rx=0 tx=0).stop 2024-08-03T20:55:58.204 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:58.201+0000 7f5b2f200640 1 -- 172.21.15.32:0/3481755578 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5b2806f600 msgr2=0x7f5b2814cdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:55:58.204 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:58.201+0000 7f5b2f200640 1 --2- 172.21.15.32:0/3481755578 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5b2806f600 0x7f5b2814cdd0 secure :-1 s=READY pgs=1273 cs=0 l=1 rev1=1 crypto rx=0x7f5b1800c4d0 tx=0x7f5b1800c990 comp rx=0 tx=0).stop 2024-08-03T20:55:58.204 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:58.201+0000 7f5b2f200640 1 -- 172.21.15.32:0/3481755578 shutdown_connections 2024-08-03T20:55:58.204 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:58.201+0000 7f5b2f200640 1 --2- 172.21.15.32:0/3481755578 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f5b04077010 0x7f5b04079520 unknown :-1 s=CLOSED pgs=525 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:58.204 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:58.201+0000 7f5b2f200640 1 --2- 172.21.15.32:0/3481755578 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f5b28075a10 0x7f5b28152d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:58.204 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:58.201+0000 7f5b2f200640 1 --2- 172.21.15.32:0/3481755578 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f5b280af520 0x7f5b281f6df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:58.205 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:58.201+0000 7f5b2f200640 1 --2- 172.21.15.32:0/3481755578 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5b2806f600 0x7f5b2814cdd0 unknown :-1 s=CLOSED pgs=1273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:58.205 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:58.201+0000 7f5b2f200640 1 -- 172.21.15.32:0/3481755578 >> 172.21.15.32:0/3481755578 conn(0x7f5b280cd1e0 msgr2=0x7f5b28072fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:55:58.205 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:58.202+0000 7f5b2f200640 1 -- 172.21.15.32:0/3481755578 shutdown_connections 2024-08-03T20:55:58.205 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:58.202+0000 7f5b2f200640 1 -- 172.21.15.32:0/3481755578 wait complete. 2024-08-03T20:55:58.212 INFO:tasks.ceph.mds.a.smithi032.stderr:2024-08-03T20:55:58.210+0000 7f1945a00640 -1 mds.0.snaptable: load_2 could not read table: -2 2024-08-03T20:55:58.212 INFO:tasks.ceph.mds.a.smithi032.stderr:2024-08-03T20:55:58.210+0000 7f1945a00640 -1 log_channel(cluster) log [ERR] : error reading table object 'mds_snaptable' -2 ((2) No such file or directory) 2024-08-03T20:55:58.213 INFO:tasks.ceph.mon.a.smithi032.stderr:2024-08-03T20:55:58.211+0000 7f4e6d600640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-08-03T20:55:58.213 INFO:tasks.ceph.mon.a.smithi032.stderr:2024-08-03T20:55:58.211+0000 7f4e6d600640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 mds daemon damaged (MDS_DAMAGE) 2024-08-03T20:55:58.238 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-08-03T20:55:58.473 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:58.471+0000 7f6c9f200640 1 -- 172.21.15.32:0/3339886929 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f6ca00af020 msgr2=0x7f6ca00742b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:55:58.473 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:58.471+0000 7f6c9f200640 1 --2- 172.21.15.32:0/3339886929 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f6ca00af020 0x7f6ca00742b0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f6c90005fd0 tx=0x7f6c9002a540 comp rx=0 tx=0).stop 2024-08-03T20:55:58.473 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:58.471+0000 7f6c9f200640 1 -- 172.21.15.32:0/3339886929 shutdown_connections 2024-08-03T20:55:58.473 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:58.471+0000 7f6c9f200640 1 --2- 172.21.15.32:0/3339886929 >> v2:172.21.15.118:3300/0 conn(0x7f6ca00acd80 0x7f6ca00ad190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:58.473 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:58.471+0000 7f6c9f200640 1 --2- 172.21.15.32:0/3339886929 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f6ca00af020 0x7f6ca00742b0 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:58.473 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:58.471+0000 7f6c9f200640 1 -- 172.21.15.32:0/3339886929 >> 172.21.15.32:0/3339886929 conn(0x7f6ca00b2120 msgr2=0x7f6ca00cfd60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:55:58.473 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:58.471+0000 7f6c9f200640 1 -- 172.21.15.32:0/3339886929 shutdown_connections 2024-08-03T20:55:58.473 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:58.471+0000 7f6c9f200640 1 -- 172.21.15.32:0/3339886929 wait complete. 2024-08-03T20:55:58.475 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:58.473+0000 7f6c9f200640 1 Processor -- start 2024-08-03T20:55:58.475 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:58.473+0000 7f6c9f200640 1 -- start start 2024-08-03T20:55:58.477 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:58.475+0000 7f6c9f200640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f6ca006f440 0x7f6ca00c6e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:55:58.477 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:58.475+0000 7f6c9f200640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f6ca00acd80 0x7f6ca0152d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:55:58.477 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:58.475+0000 7f6c9f200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6ca00af020 0x7f6ca01f6e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:55:58.477 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:58.475+0000 7f6c9f200640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f6ca01602c0 con 0x7f6ca00af020 2024-08-03T20:55:58.477 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:58.475+0000 7f6c9f200640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f6ca0160140 con 0x7f6ca006f440 2024-08-03T20:55:58.477 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:58.475+0000 7f6c9f200640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7f6ca0160440 con 0x7f6ca00acd80 2024-08-03T20:55:58.477 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:58.475+0000 7f6c9e800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6ca00af020 0x7f6ca01f6e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:55:58.477 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:58.475+0000 7f6c9e800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6ca00af020 0x7f6ca01f6e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:50788/0 (socket says 172.21.15.32:50788) 2024-08-03T20:55:58.478 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:58.475+0000 7f6c9e800640 1 -- 172.21.15.32:0/2227332439 learned_addr learned my addr 172.21.15.32:0/2227332439 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:55:58.478 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:58.475+0000 7f6c9de00640 1 --2- 172.21.15.32:0/2227332439 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f6ca006f440 0x7f6ca00c6e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:55:58.478 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:58.475+0000 7f6c9d400640 1 --2- 172.21.15.32:0/2227332439 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f6ca00acd80 0x7f6ca0152d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:55:58.479 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:58.476+0000 7f6c9e800640 1 -- 172.21.15.32:0/2227332439 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f6ca00acd80 msgr2=0x7f6ca0152d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:55:58.479 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:58.476+0000 7f6c9e800640 1 --2- 172.21.15.32:0/2227332439 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f6ca00acd80 0x7f6ca0152d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:58.479 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:58.476+0000 7f6c9e800640 1 -- 172.21.15.32:0/2227332439 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f6ca006f440 msgr2=0x7f6ca00c6e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:55:58.479 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:58.476+0000 7f6c9e800640 1 --2- 172.21.15.32:0/2227332439 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f6ca006f440 0x7f6ca00c6e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:58.480 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:58.476+0000 7f6c9e800640 1 -- 172.21.15.32:0/2227332439 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6ca00c7350 con 0x7f6ca00af020 2024-08-03T20:55:58.480 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:58.476+0000 7f6c9e800640 1 --2- 172.21.15.32:0/2227332439 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6ca00af020 0x7f6ca01f6e20 secure :-1 s=READY pgs=1274 cs=0 l=1 rev1=1 crypto rx=0x7f6c9800c6b0 tx=0x7f6c9800cb70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:55:58.480 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:58.477+0000 7f6c96000640 1 -- 172.21.15.32:0/2227332439 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c98003b70 con 0x7f6ca00af020 2024-08-03T20:55:58.480 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:58.477+0000 7f6c96000640 1 -- 172.21.15.32:0/2227332439 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f6c98003d10 con 0x7f6ca00af020 2024-08-03T20:55:58.480 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:58.477+0000 7f6c96000640 1 -- 172.21.15.32:0/2227332439 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c98005330 con 0x7f6ca00af020 2024-08-03T20:55:58.480 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:58.477+0000 7f6c9f200640 1 -- 172.21.15.32:0/2227332439 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6ca00c7e20 con 0x7f6ca00af020 2024-08-03T20:55:58.480 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:58.477+0000 7f6c9f200640 1 -- 172.21.15.32:0/2227332439 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6ca00c7b40 con 0x7f6ca00af020 2024-08-03T20:55:58.480 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:58.478+0000 7f6c9f200640 1 -- 172.21.15.32:0/2227332439 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6ca00c75d0 con 0x7f6ca00af020 2024-08-03T20:55:58.481 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:58.479+0000 7f6c96000640 1 -- 172.21.15.32:0/2227332439 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f6c980054d0 con 0x7f6ca00af020 2024-08-03T20:55:58.481 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:58.480+0000 7f6c96000640 1 --2- 172.21.15.32:0/2227332439 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f6c80077040 0x7f6c80079550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:55:58.482 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:58.480+0000 7f6c96000640 1 -- 172.21.15.32:0/2227332439 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 6628+0+0 (secure 0 0 0) 0x7f6c980989a0 con 0x7f6ca00af020 2024-08-03T20:55:58.482 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:58.481+0000 7f6c9de00640 1 --2- 172.21.15.32:0/2227332439 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f6c80077040 0x7f6c80079550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:55:58.482 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:58.481+0000 7f6c9de00640 1 --2- 172.21.15.32:0/2227332439 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f6c80077040 0x7f6c80079550 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7f6c90033000 tx=0x7f6c90004730 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:55:58.486 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:58.484+0000 7f6c96000640 1 -- 172.21.15.32:0/2227332439 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7f6c98061120 con 0x7f6ca00af020 2024-08-03T20:55:58.738 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:58.736+0000 7f6c9f200640 1 -- 172.21.15.32:0/2227332439 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f6ca00ad190 con 0x7f6ca00af020 2024-08-03T20:55:58.739 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:58.737+0000 7f6c96000640 1 -- 172.21.15.32:0/2227332439 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 142 v142) ==== 95+0+4390 (secure 0 0 0) 0x7f6c98065670 con 0x7f6ca00af020 2024-08-03T20:55:58.739 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-03T20:55:58.739 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":142,"btime":"2024-08-03T20:55:58:213120+0000","default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6439,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6835/531080160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6834","nonce":531080160},{"type":"v1","addr":"172.21.15.118:6835","nonce":531080160}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":84},{"gid":6454,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6837/2796882319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6836","nonce":2796882319},{"type":"v1","addr":"172.21.15.118:6837","nonce":2796882319}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":86},{"gid":6813,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6834/2686933890","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":2686933890},{"type":"v1","addr":"172.21.15.32:6834","nonce":2686933890}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":85}],"filesystems":[{"mdsmap":{"epoch":142,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-08-03T20:54:26.950518+0000","modified":"2024-08-03T20:55:58.213029+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":94,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":6}]} 2024-08-03T20:55:58.739 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 142 2024-08-03T20:55:58.741 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:58.739+0000 7f6c9f200640 1 -- 172.21.15.32:0/2227332439 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f6c80077040 msgr2=0x7f6c80079550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:55:58.741 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:58.739+0000 7f6c9f200640 1 --2- 172.21.15.32:0/2227332439 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f6c80077040 0x7f6c80079550 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7f6c90033000 tx=0x7f6c90004730 comp rx=0 tx=0).stop 2024-08-03T20:55:58.741 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:58.739+0000 7f6c9f200640 1 -- 172.21.15.32:0/2227332439 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6ca00af020 msgr2=0x7f6ca01f6e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:55:58.741 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:58.739+0000 7f6c9f200640 1 --2- 172.21.15.32:0/2227332439 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6ca00af020 0x7f6ca01f6e20 secure :-1 s=READY pgs=1274 cs=0 l=1 rev1=1 crypto rx=0x7f6c9800c6b0 tx=0x7f6c9800cb70 comp rx=0 tx=0).stop 2024-08-03T20:55:58.741 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:58.739+0000 7f6c9f200640 1 -- 172.21.15.32:0/2227332439 shutdown_connections 2024-08-03T20:55:58.741 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:58.739+0000 7f6c9f200640 1 --2- 172.21.15.32:0/2227332439 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f6c80077040 0x7f6c80079550 unknown :-1 s=CLOSED pgs=526 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:58.742 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:58.739+0000 7f6c9f200640 1 --2- 172.21.15.32:0/2227332439 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6ca00af020 0x7f6ca01f6e20 unknown :-1 s=CLOSED pgs=1274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:58.742 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:58.739+0000 7f6c9f200640 1 --2- 172.21.15.32:0/2227332439 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f6ca00acd80 0x7f6ca0152d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:58.742 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:58.739+0000 7f6c9f200640 1 --2- 172.21.15.32:0/2227332439 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f6ca006f440 0x7f6ca00c6e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:58.742 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:58.739+0000 7f6c9f200640 1 -- 172.21.15.32:0/2227332439 >> 172.21.15.32:0/2227332439 conn(0x7f6ca00b2120 msgr2=0x7f6ca0156530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:55:58.742 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:58.739+0000 7f6c9f200640 1 -- 172.21.15.32:0/2227332439 shutdown_connections 2024-08-03T20:55:58.742 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:58.739+0000 7f6c9f200640 1 -- 172.21.15.32:0/2227332439 wait complete. 2024-08-03T20:55:58.767 DEBUG:tasks.ceph_test_case:wait_until_true: success in 0s and 0 retries 2024-08-03T20:55:58.767 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-08-03T20:55:58.996 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:58.994+0000 7f632b400640 1 -- 172.21.15.32:0/2505797732 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f632406e930 msgr2=0x7f6324144f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:55:58.997 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:58.994+0000 7f632b400640 1 --2- 172.21.15.32:0/2505797732 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f632406e930 0x7f6324144f90 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f6318009830 tx=0x7f631802a480 comp rx=0 tx=0).stop 2024-08-03T20:55:58.997 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:58.995+0000 7f632b400640 1 -- 172.21.15.32:0/2505797732 shutdown_connections 2024-08-03T20:55:58.997 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:58.995+0000 7f632b400640 1 --2- 172.21.15.32:0/2505797732 >> v2:172.21.15.118:3300/0 conn(0x7f632415fff0 0x7f6324160400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:58.997 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:58.995+0000 7f632b400640 1 --2- 172.21.15.32:0/2505797732 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f632406e930 0x7f6324144f90 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:58.997 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:58.995+0000 7f632b400640 1 -- 172.21.15.32:0/2505797732 >> 172.21.15.32:0/2505797732 conn(0x7f63240b1bd0 msgr2=0x7f63240b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:55:58.997 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:58.995+0000 7f632b400640 1 -- 172.21.15.32:0/2505797732 shutdown_connections 2024-08-03T20:55:58.997 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:58.995+0000 7f632b400640 1 -- 172.21.15.32:0/2505797732 wait complete. 2024-08-03T20:55:58.998 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:58.997+0000 7f632b400640 1 Processor -- start 2024-08-03T20:55:58.998 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:58.997+0000 7f632b400640 1 -- start start 2024-08-03T20:55:59.000 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:58.999+0000 7f632b400640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f632406e930 0x7f632415d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:55:59.001 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:58.999+0000 7f632b400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6324073e60 0x7f632415e290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:55:59.001 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:58.999+0000 7f632b400640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f632415fff0 0x7f632415dc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:55:59.001 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:58.999+0000 7f632b400640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f63240cf840 con 0x7f6324073e60 2024-08-03T20:55:59.001 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:58.999+0000 7f632b400640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f6324145f30 con 0x7f632415fff0 2024-08-03T20:55:59.001 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:58.999+0000 7f632b400640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7f63240cf9c0 con 0x7f632406e930 2024-08-03T20:55:59.001 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:58.999+0000 7f6323e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6324073e60 0x7f632415e290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:55:59.001 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:58.999+0000 7f6323e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6324073e60 0x7f632415e290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:50798/0 (socket says 172.21.15.32:50798) 2024-08-03T20:55:59.002 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:58.999+0000 7f6323e00640 1 -- 172.21.15.32:0/661790665 learned_addr learned my addr 172.21.15.32:0/661790665 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:55:59.002 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:58.999+0000 7f6328a00640 1 --2- 172.21.15.32:0/661790665 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f632406e930 0x7f632415d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:55:59.002 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:58.999+0000 7f6329400640 1 --2- 172.21.15.32:0/661790665 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f632415fff0 0x7f632415dc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:55:59.005 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:58.999+0000 7f6323e00640 1 -- 172.21.15.32:0/661790665 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f632406e930 msgr2=0x7f632415d450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:55:59.005 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:58.999+0000 7f6323e00640 1 --2- 172.21.15.32:0/661790665 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f632406e930 0x7f632415d450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:59.005 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:58.999+0000 7f6323e00640 1 -- 172.21.15.32:0/661790665 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f632415fff0 msgr2=0x7f632415dc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:55:59.005 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:58.999+0000 7f6323e00640 1 --2- 172.21.15.32:0/661790665 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f632415fff0 0x7f632415dc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:59.006 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:59.000+0000 7f6323e00640 1 -- 172.21.15.32:0/661790665 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f63240a9f40 con 0x7f6324073e60 2024-08-03T20:55:59.006 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:59.000+0000 7f6323e00640 1 --2- 172.21.15.32:0/661790665 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6324073e60 0x7f632415e290 secure :-1 s=READY pgs=1275 cs=0 l=1 rev1=1 crypto rx=0x7f631000c3c0 tx=0x7f631000c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:55:59.006 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:59.000+0000 7f6320c00640 1 -- 172.21.15.32:0/661790665 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6310017070 con 0x7f6324073e60 2024-08-03T20:55:59.006 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:59.000+0000 7f6320c00640 1 -- 172.21.15.32:0/661790665 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f63100035f0 con 0x7f6324073e60 2024-08-03T20:55:59.006 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:59.000+0000 7f632b400640 1 -- 172.21.15.32:0/661790665 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f63240aa220 con 0x7f6324073e60 2024-08-03T20:55:59.007 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:59.000+0000 7f6320c00640 1 -- 172.21.15.32:0/661790665 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6310003a40 con 0x7f6324073e60 2024-08-03T20:55:59.007 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:59.000+0000 7f632b400640 1 -- 172.21.15.32:0/661790665 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f632415e8d0 con 0x7f6324073e60 2024-08-03T20:55:59.007 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:59.001+0000 7f632b400640 1 -- 172.21.15.32:0/661790665 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6324072ac0 con 0x7f6324073e60 2024-08-03T20:55:59.007 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:59.003+0000 7f6320c00640 1 -- 172.21.15.32:0/661790665 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f631001a070 con 0x7f6324073e60 2024-08-03T20:55:59.007 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:59.003+0000 7f6320c00640 1 --2- 172.21.15.32:0/661790665 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f6304077010 0x7f6304079520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:55:59.008 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:59.004+0000 7f6320c00640 1 -- 172.21.15.32:0/661790665 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 6628+0+0 (secure 0 0 0) 0x7f6310098080 con 0x7f6324073e60 2024-08-03T20:55:59.009 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:59.004+0000 7f6328a00640 1 --2- 172.21.15.32:0/661790665 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f6304077010 0x7f6304079520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:55:59.009 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:59.005+0000 7f6328a00640 1 --2- 172.21.15.32:0/661790665 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f6304077010 0x7f6304079520 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7f6318034000 tx=0x7f6318004730 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:55:59.009 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:59.008+0000 7f6320c00640 1 -- 172.21.15.32:0/661790665 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7f6324072ac0 con 0x7f6324073e60 2024-08-03T20:55:59.255 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:59.254+0000 7f632b400640 1 -- 172.21.15.32:0/661790665 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f6324160bc0 con 0x7f6324073e60 2024-08-03T20:55:59.256 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:59.254+0000 7f6320c00640 1 -- 172.21.15.32:0/661790665 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 142 v142) ==== 95+0+4390 (secure 0 0 0) 0x7f6310069090 con 0x7f6324073e60 2024-08-03T20:55:59.256 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-03T20:55:59.256 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":142,"btime":"2024-08-03T20:55:58:213120+0000","default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6439,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6835/531080160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6834","nonce":531080160},{"type":"v1","addr":"172.21.15.118:6835","nonce":531080160}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":84},{"gid":6454,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6837/2796882319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6836","nonce":2796882319},{"type":"v1","addr":"172.21.15.118:6837","nonce":2796882319}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":86},{"gid":6813,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6834/2686933890","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":2686933890},{"type":"v1","addr":"172.21.15.32:6834","nonce":2686933890}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":85}],"filesystems":[{"mdsmap":{"epoch":142,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-08-03T20:54:26.950518+0000","modified":"2024-08-03T20:55:58.213029+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":94,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":6}]} 2024-08-03T20:55:59.256 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 142 2024-08-03T20:55:59.258 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:59.257+0000 7f632b400640 1 -- 172.21.15.32:0/661790665 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f6304077010 msgr2=0x7f6304079520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:55:59.259 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:59.257+0000 7f632b400640 1 --2- 172.21.15.32:0/661790665 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f6304077010 0x7f6304079520 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7f6318034000 tx=0x7f6318004730 comp rx=0 tx=0).stop 2024-08-03T20:55:59.259 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:59.257+0000 7f632b400640 1 -- 172.21.15.32:0/661790665 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6324073e60 msgr2=0x7f632415e290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:55:59.259 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:59.257+0000 7f632b400640 1 --2- 172.21.15.32:0/661790665 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6324073e60 0x7f632415e290 secure :-1 s=READY pgs=1275 cs=0 l=1 rev1=1 crypto rx=0x7f631000c3c0 tx=0x7f631000c880 comp rx=0 tx=0).stop 2024-08-03T20:55:59.259 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:59.257+0000 7f632b400640 1 -- 172.21.15.32:0/661790665 shutdown_connections 2024-08-03T20:55:59.259 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:59.257+0000 7f632b400640 1 --2- 172.21.15.32:0/661790665 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f6304077010 0x7f6304079520 unknown :-1 s=CLOSED pgs=527 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:59.259 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:59.257+0000 7f632b400640 1 --2- 172.21.15.32:0/661790665 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f632415fff0 0x7f632415dc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:59.259 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:59.257+0000 7f632b400640 1 --2- 172.21.15.32:0/661790665 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6324073e60 0x7f632415e290 unknown :-1 s=CLOSED pgs=1275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:59.259 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:59.257+0000 7f632b400640 1 --2- 172.21.15.32:0/661790665 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f632406e930 0x7f632415d450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:59.259 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:59.257+0000 7f632b400640 1 -- 172.21.15.32:0/661790665 >> 172.21.15.32:0/661790665 conn(0x7f63240b1bd0 msgr2=0x7f63240b2ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:55:59.259 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:59.257+0000 7f632b400640 1 -- 172.21.15.32:0/661790665 shutdown_connections 2024-08-03T20:55:59.259 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:59.257+0000 7f632b400640 1 -- 172.21.15.32:0/661790665 wait complete. 2024-08-03T20:55:59.285 DEBUG:tasks.ceph_test_case:wait_until_true: success in 0s and 0 retries 2024-08-03T20:55:59.285 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-08-03T20:55:59.514 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:59.512+0000 7fece3a00640 1 -- 172.21.15.32:0/582021608 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fecdc1515c0 msgr2=0x7fecdc15e170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:55:59.514 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:59.512+0000 7fece3a00640 1 --2- 172.21.15.32:0/582021608 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fecdc1515c0 0x7fecdc15e170 secure :-1 s=READY pgs=1277 cs=0 l=1 rev1=1 crypto rx=0x7fecc8009830 tx=0x7fecc802a480 comp rx=0 tx=0).stop 2024-08-03T20:55:59.514 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:59.512+0000 7fece3a00640 1 -- 172.21.15.32:0/582021608 shutdown_connections 2024-08-03T20:55:59.514 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:59.512+0000 7fece3a00640 1 --2- 172.21.15.32:0/582021608 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fecdc15e6b0 0x7fecdc06e930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:59.514 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:59.512+0000 7fece3a00640 1 --2- 172.21.15.32:0/582021608 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fecdc1515c0 0x7fecdc15e170 unknown :-1 s=CLOSED pgs=1277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:59.514 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:59.512+0000 7fece3a00640 1 -- 172.21.15.32:0/582021608 >> 172.21.15.32:0/582021608 conn(0x7fecdc0b1bd0 msgr2=0x7fecdc0b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:55:59.514 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:59.512+0000 7fece3a00640 1 -- 172.21.15.32:0/582021608 shutdown_connections 2024-08-03T20:55:59.515 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:59.512+0000 7fece3a00640 1 -- 172.21.15.32:0/582021608 wait complete. 2024-08-03T20:55:59.516 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:59.514+0000 7fece3a00640 1 Processor -- start 2024-08-03T20:55:59.516 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:59.514+0000 7fece3a00640 1 -- start start 2024-08-03T20:55:59.519 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:59.516+0000 7fece3a00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fecdc0b4de0 0x7fecdc14e100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:55:59.519 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:59.516+0000 7fece3a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fecdc1515c0 0x7fecdc146510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:55:59.519 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:59.516+0000 7fece3a00640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fecdc15e6b0 0x7fecdc159710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:55:59.519 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:59.516+0000 7fece3a00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fecdc0708c0 con 0x7fecdc1515c0 2024-08-03T20:55:59.519 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:59.516+0000 7fece3a00640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7fecdc15f000 con 0x7fecdc0b4de0 2024-08-03T20:55:59.519 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:59.516+0000 7fece3a00640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7fecdc070a40 con 0x7fecdc15e6b0 2024-08-03T20:55:59.519 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:59.516+0000 7fecdbe00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fecdc1515c0 0x7fecdc146510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:55:59.519 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:59.516+0000 7fecdbe00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fecdc1515c0 0x7fecdc146510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:50820/0 (socket says 172.21.15.32:50820) 2024-08-03T20:55:59.519 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:59.516+0000 7fecdbe00640 1 -- 172.21.15.32:0/2753878061 learned_addr learned my addr 172.21.15.32:0/2753878061 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:55:59.519 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:59.516+0000 7fece1a00640 1 --2- 172.21.15.32:0/2753878061 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fecdc15e6b0 0x7fecdc159710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:55:59.519 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:59.516+0000 7fece1000640 1 --2- 172.21.15.32:0/2753878061 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fecdc0b4de0 0x7fecdc14e100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:55:59.519 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:59.517+0000 7fecdbe00640 1 -- 172.21.15.32:0/2753878061 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fecdc15e6b0 msgr2=0x7fecdc159710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:55:59.520 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:59.517+0000 7fecdbe00640 1 --2- 172.21.15.32:0/2753878061 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fecdc15e6b0 0x7fecdc159710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:59.520 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:59.517+0000 7fecdbe00640 1 -- 172.21.15.32:0/2753878061 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fecdc0b4de0 msgr2=0x7fecdc14e100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:55:59.520 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:59.517+0000 7fecdbe00640 1 --2- 172.21.15.32:0/2753878061 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fecdc0b4de0 0x7fecdc14e100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:59.520 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:59.517+0000 7fecdbe00640 1 -- 172.21.15.32:0/2753878061 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fecdc0d0cd0 con 0x7fecdc1515c0 2024-08-03T20:55:59.520 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:59.517+0000 7fece1a00640 1 --2- 172.21.15.32:0/2753878061 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fecdc15e6b0 0x7fecdc159710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-03T20:55:59.520 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:59.517+0000 7fece1000640 1 --2- 172.21.15.32:0/2753878061 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fecdc0b4de0 0x7fecdc14e100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-03T20:55:59.520 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:59.517+0000 7fecdbe00640 1 --2- 172.21.15.32:0/2753878061 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fecdc1515c0 0x7fecdc146510 secure :-1 s=READY pgs=1278 cs=0 l=1 rev1=1 crypto rx=0x7fecc802aa70 tx=0x7fecc80042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:55:59.520 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:59.517+0000 7fecd8c00640 1 -- 172.21.15.32:0/2753878061 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fecc80044e0 con 0x7fecdc1515c0 2024-08-03T20:55:59.520 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:59.517+0000 7fecd8c00640 1 -- 172.21.15.32:0/2753878061 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fecc8005310 con 0x7fecdc1515c0 2024-08-03T20:55:59.520 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:59.517+0000 7fecd8c00640 1 -- 172.21.15.32:0/2753878061 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fecc8003800 con 0x7fecdc1515c0 2024-08-03T20:55:59.520 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:59.517+0000 7fece3a00640 1 -- 172.21.15.32:0/2753878061 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fecdc150a00 con 0x7fecdc1515c0 2024-08-03T20:55:59.520 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:59.517+0000 7fece3a00640 1 -- 172.21.15.32:0/2753878061 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fecdc150fc0 con 0x7fecdc1515c0 2024-08-03T20:55:59.521 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:59.519+0000 7fece3a00640 1 -- 172.21.15.32:0/2753878061 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fecdc150770 con 0x7fecdc1515c0 2024-08-03T20:55:59.521 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:59.520+0000 7fecd8c00640 1 -- 172.21.15.32:0/2753878061 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7fecc80039a0 con 0x7fecdc1515c0 2024-08-03T20:55:59.522 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:59.521+0000 7fecd8c00640 1 --2- 172.21.15.32:0/2753878061 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fecbc077010 0x7fecbc079520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:55:59.523 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:59.521+0000 7fecd8c00640 1 -- 172.21.15.32:0/2753878061 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 6628+0+0 (secure 0 0 0) 0x7fecc80ba0a0 con 0x7fecdc1515c0 2024-08-03T20:55:59.523 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:59.521+0000 7fece1000640 1 --2- 172.21.15.32:0/2753878061 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fecbc077010 0x7fecbc079520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:55:59.523 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:59.522+0000 7fece1000640 1 --2- 172.21.15.32:0/2753878061 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fecbc077010 0x7fecbc079520 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7fecd0000f60 tx=0x7fecd0009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:55:59.526 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:59.525+0000 7fecd8c00640 1 -- 172.21.15.32:0/2753878061 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7fecc803f030 con 0x7fecdc1515c0 2024-08-03T20:55:59.773 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:59.772+0000 7fece3a00640 1 -- 172.21.15.32:0/2753878061 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fecdc050ba0 con 0x7fecdc1515c0 2024-08-03T20:55:59.774 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:59.772+0000 7fecd8c00640 1 -- 172.21.15.32:0/2753878061 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 142 v142) ==== 95+0+4390 (secure 0 0 0) 0x7fecdc050ba0 con 0x7fecdc1515c0 2024-08-03T20:55:59.774 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-03T20:55:59.774 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":142,"btime":"2024-08-03T20:55:58:213120+0000","default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6439,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6835/531080160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6834","nonce":531080160},{"type":"v1","addr":"172.21.15.118:6835","nonce":531080160}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":84},{"gid":6454,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6837/2796882319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6836","nonce":2796882319},{"type":"v1","addr":"172.21.15.118:6837","nonce":2796882319}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":86},{"gid":6813,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6834/2686933890","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":2686933890},{"type":"v1","addr":"172.21.15.32:6834","nonce":2686933890}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":85}],"filesystems":[{"mdsmap":{"epoch":142,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-08-03T20:54:26.950518+0000","modified":"2024-08-03T20:55:58.213029+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":94,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":6}]} 2024-08-03T20:55:59.775 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 142 2024-08-03T20:55:59.777 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:59.775+0000 7fece3a00640 1 -- 172.21.15.32:0/2753878061 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fecbc077010 msgr2=0x7fecbc079520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:55:59.777 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:59.775+0000 7fece3a00640 1 --2- 172.21.15.32:0/2753878061 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fecbc077010 0x7fecbc079520 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7fecd0000f60 tx=0x7fecd0009040 comp rx=0 tx=0).stop 2024-08-03T20:55:59.777 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:59.775+0000 7fece3a00640 1 -- 172.21.15.32:0/2753878061 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fecdc1515c0 msgr2=0x7fecdc146510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:55:59.777 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:59.775+0000 7fece3a00640 1 --2- 172.21.15.32:0/2753878061 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fecdc1515c0 0x7fecdc146510 secure :-1 s=READY pgs=1278 cs=0 l=1 rev1=1 crypto rx=0x7fecc802aa70 tx=0x7fecc80042f0 comp rx=0 tx=0).stop 2024-08-03T20:55:59.777 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:59.775+0000 7fece3a00640 1 -- 172.21.15.32:0/2753878061 shutdown_connections 2024-08-03T20:55:59.777 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:59.775+0000 7fece3a00640 1 --2- 172.21.15.32:0/2753878061 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fecbc077010 0x7fecbc079520 unknown :-1 s=CLOSED pgs=528 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:59.777 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:59.775+0000 7fece3a00640 1 --2- 172.21.15.32:0/2753878061 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fecdc15e6b0 0x7fecdc159710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:59.777 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:59.775+0000 7fece3a00640 1 --2- 172.21.15.32:0/2753878061 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fecdc1515c0 0x7fecdc146510 unknown :-1 s=CLOSED pgs=1278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:59.777 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:59.775+0000 7fece3a00640 1 --2- 172.21.15.32:0/2753878061 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fecdc0b4de0 0x7fecdc14e100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:55:59.777 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:59.775+0000 7fece3a00640 1 -- 172.21.15.32:0/2753878061 >> 172.21.15.32:0/2753878061 conn(0x7fecdc0b1bd0 msgr2=0x7fecdc0b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:55:59.777 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:59.775+0000 7fece3a00640 1 -- 172.21.15.32:0/2753878061 shutdown_connections 2024-08-03T20:55:59.778 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:55:59.775+0000 7fece3a00640 1 -- 172.21.15.32:0/2753878061 wait complete. 2024-08-03T20:55:59.799 INFO:tasks.cephfs.test_damage:Result: Mutation 'Delete mds_snaptable' led to DAMAGED state 2024-08-03T20:55:59.799 INFO:tasks.cephfs.test_damage:Applying mutation 'Delete mds0_inotable' 2024-08-03T20:55:59.799 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-08-03T20:55:59.799 DEBUG:teuthology.orchestra.run.smithi032:> dd if=/proc/self/mounts of=/dev/stdout 2024-08-03T20:55:59.835 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053453,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6499208k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 none /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249604k,nr_inodes=812401,mode=700,uid=1000,gid=1316 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6499208k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-08-03T20:55:59.835 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi032.front.sepia.ceph.com 2024-08-03T20:55:59.835 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-08-03T20:55:59.835 DEBUG:teuthology.orchestra.run.smithi032:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-08-03T20:55:59.910 INFO:teuthology.orchestra.run.smithi032.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.0': No such file or directory 2024-08-03T20:55:59.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-03T20:55:59.911 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-08-03T20:56:00.186 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:00.184+0000 7f8adf200640 1 -- 172.21.15.32:0/704578139 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8ac4004d40 msgr2=0x7f8ac4005150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:56:00.186 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:00.184+0000 7f8adf200640 1 --2- 172.21.15.32:0/704578139 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8ac4004d40 0x7f8ac4005150 secure :-1 s=READY pgs=1280 cs=0 l=1 rev1=1 crypto rx=0x7f8ae00d0a30 tx=0x7f8ac4005830 comp rx=0 tx=0).stop 2024-08-03T20:56:00.186 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:00.184+0000 7f8ade800640 1 -- 172.21.15.32:0/704578139 reap_dead start 2024-08-03T20:56:00.186 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:00.184+0000 7f8adf200640 1 -- 172.21.15.32:0/704578139 shutdown_connections 2024-08-03T20:56:00.186 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:00.184+0000 7f8adf200640 1 -- 172.21.15.32:0/704578139 >> 172.21.15.32:0/704578139 conn(0x7f8ae008d7a0 msgr2=0x7f8ae009d3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:56:00.186 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:00.184+0000 7f8adf200640 1 -- 172.21.15.32:0/704578139 shutdown_connections 2024-08-03T20:56:00.186 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:00.184+0000 7f8adf200640 1 -- 172.21.15.32:0/704578139 wait complete. 2024-08-03T20:56:00.188 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:00.186+0000 7f8adf200640 1 Processor -- start 2024-08-03T20:56:00.188 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:00.187+0000 7f8adf200640 1 -- start start 2024-08-03T20:56:00.196 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:00.188+0000 7f8adf200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8ac4004d40 0x7f8ae01f95d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:56:00.196 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:00.188+0000 7f8adf200640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f8ae015d870 0x7f8ae0155600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:56:00.196 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:00.188+0000 7f8adf200640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8ae00c8ee0 0x7f8ae00c92f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:56:00.196 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:00.188+0000 7f8adf200640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f8ae0161d00 con 0x7f8ac4004d40 2024-08-03T20:56:00.196 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:00.188+0000 7f8adf200640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f8ae0161b80 con 0x7f8ae00c8ee0 2024-08-03T20:56:00.196 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:00.188+0000 7f8adf200640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7f8ae0161e80 con 0x7f8ae015d870 2024-08-03T20:56:00.196 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:00.188+0000 7f8adde00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8ac4004d40 0x7f8ae01f95d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:56:00.196 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:00.188+0000 7f8adde00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8ac4004d40 0x7f8ae01f95d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:42508/0 (socket says 172.21.15.32:42508) 2024-08-03T20:56:00.196 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:00.188+0000 7f8adde00640 1 -- 172.21.15.32:0/1949491493 learned_addr learned my addr 172.21.15.32:0/1949491493 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:56:00.197 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:00.188+0000 7f8ad7e00640 1 --2- 172.21.15.32:0/1949491493 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f8ae015d870 0x7f8ae0155600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:56:00.197 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:00.188+0000 7f8ade800640 1 --2- 172.21.15.32:0/1949491493 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8ae00c8ee0 0x7f8ae00c92f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:56:00.197 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:00.189+0000 7f8adde00640 1 -- 172.21.15.32:0/1949491493 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f8ae015d870 msgr2=0x7f8ae0155600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:56:00.197 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:00.189+0000 7f8adde00640 1 --2- 172.21.15.32:0/1949491493 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f8ae015d870 0x7f8ae0155600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:00.197 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:00.189+0000 7f8adde00640 1 -- 172.21.15.32:0/1949491493 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8ae00c8ee0 msgr2=0x7f8ae00c92f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:56:00.197 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:00.189+0000 7f8adde00640 1 --2- 172.21.15.32:0/1949491493 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8ae00c8ee0 0x7f8ae00c92f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:00.197 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:00.189+0000 7f8adde00640 1 -- 172.21.15.32:0/1949491493 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8ae00cb410 con 0x7f8ac4004d40 2024-08-03T20:56:00.197 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:00.189+0000 7f8ad7e00640 1 --2- 172.21.15.32:0/1949491493 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f8ae015d870 0x7f8ae0155600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-03T20:56:00.197 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:00.189+0000 7f8adde00640 1 --2- 172.21.15.32:0/1949491493 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8ac4004d40 0x7f8ae01f95d0 secure :-1 s=READY pgs=1281 cs=0 l=1 rev1=1 crypto rx=0x7f8ad0011a80 tx=0x7f8ad0011f40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:56:00.197 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:00.189+0000 7f8ad6000640 1 -- 172.21.15.32:0/1949491493 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ad000fe10 con 0x7f8ac4004d40 2024-08-03T20:56:00.197 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:00.189+0000 7f8ad6000640 1 -- 172.21.15.32:0/1949491493 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f8ad0016070 con 0x7f8ac4004d40 2024-08-03T20:56:00.197 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:00.189+0000 7f8ad6000640 1 -- 172.21.15.32:0/1949491493 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ad0012690 con 0x7f8ac4004d40 2024-08-03T20:56:00.197 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:00.189+0000 7f8adf200640 1 -- 172.21.15.32:0/1949491493 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8ae00cc2b0 con 0x7f8ac4004d40 2024-08-03T20:56:00.197 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:00.190+0000 7f8adf200640 1 -- 172.21.15.32:0/1949491493 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8ae01f8af0 con 0x7f8ac4004d40 2024-08-03T20:56:00.198 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:00.191+0000 7f8adf200640 1 -- 172.21.15.32:0/1949491493 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8ae00d24b0 con 0x7f8ac4004d40 2024-08-03T20:56:00.198 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:00.192+0000 7f8ad6000640 1 -- 172.21.15.32:0/1949491493 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f8ad0018030 con 0x7f8ac4004d40 2024-08-03T20:56:00.198 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:00.193+0000 7f8ad6000640 1 --2- 172.21.15.32:0/1949491493 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f8abc077010 0x7f8abc079520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:56:00.198 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:00.193+0000 7f8ad6000640 1 -- 172.21.15.32:0/1949491493 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 6628+0+0 (secure 0 0 0) 0x7f8ad0099690 con 0x7f8ac4004d40 2024-08-03T20:56:00.198 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:00.193+0000 7f8ad7e00640 1 --2- 172.21.15.32:0/1949491493 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f8abc077010 0x7f8abc079520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:56:00.198 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:00.194+0000 7f8ad7e00640 1 --2- 172.21.15.32:0/1949491493 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f8abc077010 0x7f8abc079520 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7f8ac8000fb0 tx=0x7f8ac80067a0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:56:00.198 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:00.197+0000 7f8ad6000640 1 -- 172.21.15.32:0/1949491493 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7f8ae00d24b0 con 0x7f8ac4004d40 2024-08-03T20:56:00.451 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:00.449+0000 7f8adf200640 1 -- 172.21.15.32:0/1949491493 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7f8ae00cf920 con 0x7f8ac4004d40 2024-08-03T20:56:01.225 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:01.223+0000 7f8ad6000640 1 -- 172.21.15.32:0/1949491493 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v143) ==== 161+0+0 (secure 0 0 0) 0x7f8ad0061e00 con 0x7f8ac4004d40 2024-08-03T20:56:01.225 INFO:teuthology.orchestra.run.smithi032.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-08-03T20:56:01.230 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:01.227+0000 7f8adf200640 1 -- 172.21.15.32:0/1949491493 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f8abc077010 msgr2=0x7f8abc079520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:56:01.230 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:01.227+0000 7f8adf200640 1 --2- 172.21.15.32:0/1949491493 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f8abc077010 0x7f8abc079520 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7f8ac8000fb0 tx=0x7f8ac80067a0 comp rx=0 tx=0).stop 2024-08-03T20:56:01.230 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:01.227+0000 7f8adf200640 1 -- 172.21.15.32:0/1949491493 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8ac4004d40 msgr2=0x7f8ae01f95d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:56:01.230 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:01.227+0000 7f8adf200640 1 --2- 172.21.15.32:0/1949491493 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8ac4004d40 0x7f8ae01f95d0 secure :-1 s=READY pgs=1281 cs=0 l=1 rev1=1 crypto rx=0x7f8ad0011a80 tx=0x7f8ad0011f40 comp rx=0 tx=0).stop 2024-08-03T20:56:01.230 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:01.228+0000 7f8adf200640 1 -- 172.21.15.32:0/1949491493 shutdown_connections 2024-08-03T20:56:01.230 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:01.228+0000 7f8adf200640 1 --2- 172.21.15.32:0/1949491493 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f8abc077010 0x7f8abc079520 unknown :-1 s=CLOSED pgs=529 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:01.230 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:01.228+0000 7f8adf200640 1 --2- 172.21.15.32:0/1949491493 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8ae00c8ee0 0x7f8ae00c92f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:01.231 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:01.228+0000 7f8adf200640 1 --2- 172.21.15.32:0/1949491493 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f8ae015d870 0x7f8ae0155600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:01.231 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:01.228+0000 7f8adf200640 1 --2- 172.21.15.32:0/1949491493 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8ac4004d40 0x7f8ae01f95d0 unknown :-1 s=CLOSED pgs=1281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:01.231 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:01.228+0000 7f8adf200640 1 -- 172.21.15.32:0/1949491493 >> 172.21.15.32:0/1949491493 conn(0x7f8ae008d7a0 msgr2=0x7f8ae015e900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:56:01.231 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:01.228+0000 7f8adf200640 1 -- 172.21.15.32:0/1949491493 shutdown_connections 2024-08-03T20:56:01.231 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:01.228+0000 7f8adf200640 1 -- 172.21.15.32:0/1949491493 wait complete. 2024-08-03T20:56:01.256 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mds repaired 0 2024-08-03T20:56:01.491 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:01.489+0000 7fbd90c00640 1 -- 172.21.15.32:0/2955249074 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbd8c0af140 msgr2=0x7fbd8c0cde40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:56:01.491 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:01.489+0000 7fbd90c00640 1 --2- 172.21.15.32:0/2955249074 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbd8c0af140 0x7fbd8c0cde40 secure :-1 s=READY pgs=1282 cs=0 l=1 rev1=1 crypto rx=0x7fbd74005fd0 tx=0x7fbd7402a5a0 comp rx=0 tx=0).stop 2024-08-03T20:56:01.491 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:01.489+0000 7fbd90c00640 1 -- 172.21.15.32:0/2955249074 shutdown_connections 2024-08-03T20:56:01.491 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:01.489+0000 7fbd90c00640 1 --2- 172.21.15.32:0/2955249074 >> v2:172.21.15.118:3300/0 conn(0x7fbd8c0ce380 0x7fbd8c161fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:01.491 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:01.489+0000 7fbd90c00640 1 --2- 172.21.15.32:0/2955249074 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbd8c0af140 0x7fbd8c0cde40 unknown :-1 s=CLOSED pgs=1282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:01.491 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:01.489+0000 7fbd90c00640 1 -- 172.21.15.32:0/2955249074 >> 172.21.15.32:0/2955249074 conn(0x7fbd8c0b2120 msgr2=0x7fbd8c0cfd60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:56:01.491 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:01.489+0000 7fbd90c00640 1 -- 172.21.15.32:0/2955249074 shutdown_connections 2024-08-03T20:56:01.491 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:01.489+0000 7fbd90c00640 1 -- 172.21.15.32:0/2955249074 wait complete. 2024-08-03T20:56:01.492 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:01.491+0000 7fbd90c00640 1 Processor -- start 2024-08-03T20:56:01.493 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:01.491+0000 7fbd90c00640 1 -- start start 2024-08-03T20:56:01.495 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:01.493+0000 7fbd90c00640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fbd8c0af140 0x7fbd8c09cfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:56:01.495 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:01.493+0000 7fbd90c00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbd8c0ca8c0 0x7fbd8c15afb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:56:01.495 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:01.493+0000 7fbd90c00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fbd8c0ce380 0x7fbd8c15b4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:56:01.495 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:01.493+0000 7fbd90c00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fbd8c162880 con 0x7fbd8c0ca8c0 2024-08-03T20:56:01.496 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:01.493+0000 7fbd90c00640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7fbd8c162700 con 0x7fbd8c0ce380 2024-08-03T20:56:01.496 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:01.493+0000 7fbd90c00640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7fbd8c162a00 con 0x7fbd8c0af140 2024-08-03T20:56:01.496 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:01.493+0000 7fbd82a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbd8c0ca8c0 0x7fbd8c15afb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:56:01.496 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:01.493+0000 7fbd82a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbd8c0ca8c0 0x7fbd8c15afb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:42526/0 (socket says 172.21.15.32:42526) 2024-08-03T20:56:01.496 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:01.493+0000 7fbd82a00640 1 -- 172.21.15.32:0/842380011 learned_addr learned my addr 172.21.15.32:0/842380011 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:56:01.496 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:01.493+0000 7fbd8b400640 1 --2- 172.21.15.32:0/842380011 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fbd8c0af140 0x7fbd8c09cfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:56:01.496 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:01.493+0000 7fbd8be00640 1 --2- 172.21.15.32:0/842380011 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fbd8c0ce380 0x7fbd8c15b4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:56:01.496 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:01.494+0000 7fbd82a00640 1 -- 172.21.15.32:0/842380011 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fbd8c0af140 msgr2=0x7fbd8c09cfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:56:01.496 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:01.494+0000 7fbd82a00640 1 --2- 172.21.15.32:0/842380011 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fbd8c0af140 0x7fbd8c09cfa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:01.496 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:01.494+0000 7fbd82a00640 1 -- 172.21.15.32:0/842380011 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fbd8c0ce380 msgr2=0x7fbd8c15b4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:56:01.496 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:01.494+0000 7fbd82a00640 1 --2- 172.21.15.32:0/842380011 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fbd8c0ce380 0x7fbd8c15b4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:01.497 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:01.494+0000 7fbd82a00640 1 -- 172.21.15.32:0/842380011 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbd8c1f9f10 con 0x7fbd8c0ca8c0 2024-08-03T20:56:01.497 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:01.494+0000 7fbd8be00640 1 --2- 172.21.15.32:0/842380011 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fbd8c0ce380 0x7fbd8c15b4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-03T20:56:01.497 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:01.494+0000 7fbd82a00640 1 --2- 172.21.15.32:0/842380011 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbd8c0ca8c0 0x7fbd8c15afb0 secure :-1 s=READY pgs=1283 cs=0 l=1 rev1=1 crypto rx=0x7fbd74034040 tx=0x7fbd740038a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:56:01.498 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:01.494+0000 7fbd83e00640 1 -- 172.21.15.32:0/842380011 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd7403e030 con 0x7fbd8c0ca8c0 2024-08-03T20:56:01.498 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:01.494+0000 7fbd83e00640 1 -- 172.21.15.32:0/842380011 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fbd74003d90 con 0x7fbd8c0ca8c0 2024-08-03T20:56:01.498 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:01.494+0000 7fbd83e00640 1 -- 172.21.15.32:0/842380011 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd74035040 con 0x7fbd8c0ca8c0 2024-08-03T20:56:01.498 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:01.494+0000 7fbd90c00640 1 -- 172.21.15.32:0/842380011 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbd8c15bb50 con 0x7fbd8c0ca8c0 2024-08-03T20:56:01.498 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:01.494+0000 7fbd90c00640 1 -- 172.21.15.32:0/842380011 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbd8c152ea0 con 0x7fbd8c0ca8c0 2024-08-03T20:56:01.498 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:01.495+0000 7fbd90c00640 1 -- 172.21.15.32:0/842380011 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbd8c15f870 con 0x7fbd8c0ca8c0 2024-08-03T20:56:01.498 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:01.497+0000 7fbd83e00640 1 -- 172.21.15.32:0/842380011 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7fbd74041030 con 0x7fbd8c0ca8c0 2024-08-03T20:56:01.500 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:01.498+0000 7fbd83e00640 1 --2- 172.21.15.32:0/842380011 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fbd68077010 0x7fbd68079520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:56:01.500 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:01.498+0000 7fbd83e00640 1 -- 172.21.15.32:0/842380011 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 6628+0+0 (secure 0 0 0) 0x7fbd740b97b0 con 0x7fbd8c0ca8c0 2024-08-03T20:56:01.500 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:01.499+0000 7fbd8b400640 1 --2- 172.21.15.32:0/842380011 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fbd68077010 0x7fbd68079520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:56:01.501 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:01.500+0000 7fbd8b400640 1 --2- 172.21.15.32:0/842380011 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fbd68077010 0x7fbd68079520 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7fbd7c001720 tx=0x7fbd7c009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:56:01.504 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:01.503+0000 7fbd83e00640 1 -- 172.21.15.32:0/842380011 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7fbd74081f20 con 0x7fbd8c0ca8c0 2024-08-03T20:56:01.890 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:01.785+0000 7fbd90c00640 1 -- 172.21.15.32:0/842380011 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "mds repaired", "role": "0"} v 0) -- 0x7fbd8c0748b0 con 0x7fbd8c0ca8c0 2024-08-03T20:56:02.231 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:02.230+0000 7fbd83e00640 1 -- 172.21.15.32:0/842380011 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "mds repaired", "role": "0"}]=0 repaired: restoring rank 6:0 v144) ==== 101+0+0 (secure 0 0 0) 0x7fbd74086470 con 0x7fbd8c0ca8c0 2024-08-03T20:56:02.232 INFO:teuthology.orchestra.run.smithi032.stderr:repaired: restoring rank 6:0 2024-08-03T20:56:02.236 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:02.234+0000 7fbd90c00640 1 -- 172.21.15.32:0/842380011 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fbd68077010 msgr2=0x7fbd68079520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:56:02.236 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:02.234+0000 7fbd90c00640 1 --2- 172.21.15.32:0/842380011 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fbd68077010 0x7fbd68079520 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7fbd7c001720 tx=0x7fbd7c009040 comp rx=0 tx=0).stop 2024-08-03T20:56:02.236 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:02.234+0000 7fbd90c00640 1 -- 172.21.15.32:0/842380011 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbd8c0ca8c0 msgr2=0x7fbd8c15afb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:56:02.236 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:02.234+0000 7fbd90c00640 1 --2- 172.21.15.32:0/842380011 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbd8c0ca8c0 0x7fbd8c15afb0 secure :-1 s=READY pgs=1283 cs=0 l=1 rev1=1 crypto rx=0x7fbd74034040 tx=0x7fbd740038a0 comp rx=0 tx=0).stop 2024-08-03T20:56:02.236 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:02.234+0000 7fbd90c00640 1 -- 172.21.15.32:0/842380011 shutdown_connections 2024-08-03T20:56:02.236 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:02.234+0000 7fbd90c00640 1 --2- 172.21.15.32:0/842380011 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fbd68077010 0x7fbd68079520 unknown :-1 s=CLOSED pgs=530 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:02.236 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:02.234+0000 7fbd90c00640 1 --2- 172.21.15.32:0/842380011 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fbd8c0ce380 0x7fbd8c15b4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:02.236 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:02.234+0000 7fbd90c00640 1 --2- 172.21.15.32:0/842380011 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbd8c0ca8c0 0x7fbd8c15afb0 unknown :-1 s=CLOSED pgs=1283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:02.236 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:02.234+0000 7fbd90c00640 1 --2- 172.21.15.32:0/842380011 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fbd8c0af140 0x7fbd8c09cfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:02.236 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:02.234+0000 7fbd90c00640 1 -- 172.21.15.32:0/842380011 >> 172.21.15.32:0/842380011 conn(0x7fbd8c0b2120 msgr2=0x7fbd8c0ae6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:56:02.236 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:02.234+0000 7fbd90c00640 1 -- 172.21.15.32:0/842380011 shutdown_connections 2024-08-03T20:56:02.236 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:02.234+0000 7fbd90c00640 1 -- 172.21.15.32:0/842380011 wait complete. 2024-08-03T20:56:02.264 DEBUG:teuthology.orchestra.run.smithi032:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata import - 2024-08-03T20:56:02.411 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:02.409+0000 7f0809904f40 1 Processor -- start 2024-08-03T20:56:02.411 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:02.409+0000 7f0809904f40 1 -- start start 2024-08-03T20:56:02.413 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:02.410+0000 7f0809904f40 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f07e80030f0 0x5560df04ada0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:56:02.413 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:02.410+0000 7f0809904f40 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5560df11d5f0 0x5560df04bda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:56:02.413 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:02.411+0000 7f0809904f40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5560df04c2e0 0x5560df11bed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:56:02.413 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:02.411+0000 7f0809904f40 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x5560def9ae30 con 0x5560df11d5f0 2024-08-03T20:56:02.413 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:02.411+0000 7f0809904f40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x5560deff8bf0 con 0x5560df04c2e0 2024-08-03T20:56:02.413 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:02.411+0000 7f0809904f40 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x5560df104440 con 0x7f07e80030f0 2024-08-03T20:56:02.414 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:02.411+0000 7f0802000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5560df11d5f0 0x5560df04bda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:56:02.414 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:02.411+0000 7f0802000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5560df11d5f0 0x5560df04bda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:42550/0 (socket says 172.21.15.32:42550) 2024-08-03T20:56:02.414 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:02.411+0000 7f0802000640 1 -- 172.21.15.32:0/432521278 learned_addr learned my addr 172.21.15.32:0/432521278 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:56:02.414 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:02.411+0000 7f0802a00640 1 --2- 172.21.15.32:0/432521278 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f07e80030f0 0x5560df04ada0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:56:02.414 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:02.411+0000 7f0803400640 1 --2- 172.21.15.32:0/432521278 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5560df04c2e0 0x5560df11bed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:56:02.414 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:02.411+0000 7f0802000640 1 -- 172.21.15.32:0/432521278 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f07e80030f0 msgr2=0x5560df04ada0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:56:02.414 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:02.411+0000 7f0802000640 1 --2- 172.21.15.32:0/432521278 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f07e80030f0 0x5560df04ada0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:02.414 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:02.411+0000 7f0802000640 1 -- 172.21.15.32:0/432521278 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5560df04c2e0 msgr2=0x5560df11bed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:56:02.414 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:02.411+0000 7f0802000640 1 --2- 172.21.15.32:0/432521278 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5560df04c2e0 0x5560df11bed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:02.414 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:02.411+0000 7f0802000640 1 -- 172.21.15.32:0/432521278 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5560df096040 con 0x5560df11d5f0 2024-08-03T20:56:02.415 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:02.412+0000 7f0802000640 1 --2- 172.21.15.32:0/432521278 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5560df11d5f0 0x5560df04bda0 secure :-1 s=READY pgs=1286 cs=0 l=1 rev1=1 crypto rx=0x7f07f0026270 tx=0x7f07f0003020 comp rx=0 tx=0).ready entity=mon.0 client_cookie=471ff8b9eb866352 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:56:02.415 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:02.412+0000 7f0800c00640 1 -- 172.21.15.32:0/432521278 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07f0062030 con 0x5560df11d5f0 2024-08-03T20:56:02.415 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:02.412+0000 7f0800c00640 1 -- 172.21.15.32:0/432521278 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f07f005e400 con 0x5560df11d5f0 2024-08-03T20:56:02.415 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:02.412+0000 7f0800c00640 1 -- 172.21.15.32:0/432521278 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07f005e630 con 0x5560df11d5f0 2024-08-03T20:56:02.416 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:02.412+0000 7f0809904f40 1 -- 172.21.15.32:0/432521278 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5560df11d5f0 msgr2=0x5560df04bda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:56:02.416 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:02.412+0000 7f0809904f40 1 --2- 172.21.15.32:0/432521278 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5560df11d5f0 0x5560df04bda0 secure :-1 s=READY pgs=1286 cs=0 l=1 rev1=1 crypto rx=0x7f07f0026270 tx=0x7f07f0003020 comp rx=0 tx=0).stop 2024-08-03T20:56:02.416 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:02.413+0000 7f0809904f40 1 -- 172.21.15.32:0/432521278 shutdown_connections 2024-08-03T20:56:02.416 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:02.413+0000 7f0809904f40 1 --2- 172.21.15.32:0/432521278 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5560df04c2e0 0x5560df11bed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:02.416 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:02.413+0000 7f0809904f40 1 --2- 172.21.15.32:0/432521278 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5560df11d5f0 0x5560df04bda0 unknown :-1 s=CLOSED pgs=1286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:02.416 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:02.413+0000 7f0809904f40 1 --2- 172.21.15.32:0/432521278 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f07e80030f0 0x5560df04ada0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:02.416 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:02.413+0000 7f0809904f40 1 -- 172.21.15.32:0/432521278 >> 172.21.15.32:0/432521278 conn(0x5560def6cac0 msgr2=0x5560df11cee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:56:02.416 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:02.413+0000 7f0809904f40 1 -- 172.21.15.32:0/432521278 shutdown_connections 2024-08-03T20:56:02.417 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:02.413+0000 7f0809904f40 1 -- 172.21.15.32:0/432521278 wait complete. 2024-08-03T20:56:02.417 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:02.414+0000 7f0809904f40 1 Processor -- start 2024-08-03T20:56:02.417 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:02.414+0000 7f0809904f40 1 -- start start 2024-08-03T20:56:02.419 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:02.415+0000 7f0809904f40 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x5560df04c2e0 0x5560df101a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:56:02.419 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:02.415+0000 7f0809904f40 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5560df11d5f0 0x5560df11be40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:56:02.419 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:02.415+0000 7f0809904f40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f07e80030f0 0x5560df11c380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:56:02.419 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:02.415+0000 7f0809904f40 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x5560df04c8b0 con 0x5560df11d5f0 2024-08-03T20:56:02.419 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:02.415+0000 7f0809904f40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x5560df095df0 con 0x7f07e80030f0 2024-08-03T20:56:02.419 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:02.415+0000 7f0809904f40 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x5560df08c460 con 0x5560df04c2e0 2024-08-03T20:56:02.419 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:02.415+0000 7f0802000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5560df11d5f0 0x5560df11be40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:56:02.419 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:02.415+0000 7f0802000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5560df11d5f0 0x5560df11be40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:42562/0 (socket says 172.21.15.32:42562) 2024-08-03T20:56:02.420 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:02.416+0000 7f0802000640 1 -- 172.21.15.32:0/3180561690 learned_addr learned my addr 172.21.15.32:0/3180561690 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:56:02.420 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:02.416+0000 7f0802a00640 1 --2- 172.21.15.32:0/3180561690 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x5560df04c2e0 0x5560df101a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:56:02.420 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:02.416+0000 7f0803400640 1 --2- 172.21.15.32:0/3180561690 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f07e80030f0 0x5560df11c380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:56:02.420 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:02.416+0000 7f0802000640 1 -- 172.21.15.32:0/3180561690 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x5560df04c2e0 msgr2=0x5560df101a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:56:02.420 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:02.416+0000 7f0802000640 1 --2- 172.21.15.32:0/3180561690 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x5560df04c2e0 0x5560df101a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:02.420 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:02.416+0000 7f0802000640 1 -- 172.21.15.32:0/3180561690 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f07e80030f0 msgr2=0x5560df11c380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:56:02.420 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:02.416+0000 7f0802000640 1 --2- 172.21.15.32:0/3180561690 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f07e80030f0 0x5560df11c380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:02.420 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:02.416+0000 7f0802000640 1 -- 172.21.15.32:0/3180561690 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5560df102010 con 0x5560df11d5f0 2024-08-03T20:56:02.420 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:02.416+0000 7f0802000640 1 --2- 172.21.15.32:0/3180561690 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5560df11d5f0 0x5560df11be40 secure :-1 s=READY pgs=1287 cs=0 l=1 rev1=1 crypto rx=0x7f07f000e090 tx=0x7f07f0056a80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:56:02.420 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:02.416+0000 7f07fb400640 1 -- 172.21.15.32:0/3180561690 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07f0067470 con 0x5560df11d5f0 2024-08-03T20:56:02.420 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:02.417+0000 7f07fb400640 1 -- 172.21.15.32:0/3180561690 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f07f0067700 con 0x5560df11d5f0 2024-08-03T20:56:02.421 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:02.417+0000 7f07fb400640 1 -- 172.21.15.32:0/3180561690 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07f0067b30 con 0x5560df11d5f0 2024-08-03T20:56:02.421 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:02.417+0000 7f0809904f40 1 -- 172.21.15.32:0/3180561690 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x5560df08f9d0 con 0x5560df11d5f0 2024-08-03T20:56:02.421 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:02.417+0000 7f0809904f40 1 -- 172.21.15.32:0/3180561690 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5560df08ff90 con 0x5560df11d5f0 2024-08-03T20:56:02.421 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:02.419+0000 7f07fb400640 1 -- 172.21.15.32:0/3180561690 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f07f006e030 con 0x5560df11d5f0 2024-08-03T20:56:02.422 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:02.420+0000 7f07fb400640 1 --2- 172.21.15.32:0/3180561690 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f07e40759f0 0x7f07e4077f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:56:02.422 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:02.420+0000 7f07fb400640 1 -- 172.21.15.32:0/3180561690 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 6628+0+0 (secure 0 0 0) 0x7f07f00dc7d0 con 0x5560df11d5f0 2024-08-03T20:56:02.422 INFO:teuthology.orchestra.run.smithi032.stdout:Importing pool 2024-08-03T20:56:02.423 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:02.420+0000 7f0802a00640 1 --2- 172.21.15.32:0/3180561690 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f07e40759f0 0x7f07e4077f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:56:02.423 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:02.420+0000 7f0809904f40 1 --2- 172.21.15.32:0/3180561690 >> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] conn(0x5560df092530 0x5560df092940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:56:02.423 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:02.420+0000 7f0809904f40 1 -- 172.21.15.32:0/3180561690 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:1 12.10 12:09401a81:::601.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e94) -- 0x5560df1c6480 con 0x5560df092530 2024-08-03T20:56:02.423 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:02.421+0000 7f0803400640 1 --2- 172.21.15.32:0/3180561690 >> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] conn(0x5560df092530 0x5560df092940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:56:02.423 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:02.421+0000 7f0802a00640 1 --2- 172.21.15.32:0/3180561690 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f07e40759f0 0x7f07e4077f00 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7f07f4000fb0 tx=0x7f07f4003040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:56:02.423 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:02.421+0000 7f0803400640 1 --2- 172.21.15.32:0/3180561690 >> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] conn(0x5560df092530 0x5560df092940 crc :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:56:02.425 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:02.423+0000 7f0803400640 1 -- 172.21.15.32:0/3180561690 <== osd.0 v2:172.21.15.32:6816/3066156116 1 ==== osd_op_reply(1 601.00000000 [create] v94'257 uv231 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f07fc003040 con 0x5560df092530 2024-08-03T20:56:02.425 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:02.423+0000 7f0809904f40 1 -- 172.21.15.32:0/3180561690 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:2 12.10 12:09401a81:::601.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e94) -- 0x5560df1c6820 con 0x5560df092530 2024-08-03T20:56:02.427 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:02.425+0000 7f0803400640 1 -- 172.21.15.32:0/3180561690 <== osd.0 v2:172.21.15.32:6816/3066156116 2 ==== osd_op_reply(2 601.00000000 [delete] v94'258 uv258 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f07fc003040 con 0x5560df092530 2024-08-03T20:56:02.427 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:02.425+0000 7f0809904f40 1 -- 172.21.15.32:0/3180561690 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:3 12.10 12:09401a81:::601.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e94) -- 0x5560df1c6bc0 con 0x5560df092530 2024-08-03T20:56:02.428 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::601.00000000:head# 2024-08-03T20:56:02.429 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:02.427+0000 7f0803400640 1 -- 172.21.15.32:0/3180561690 <== osd.0 v2:172.21.15.32:6816/3066156116 3 ==== osd_op_reply(3 601.00000000 [create] v94'259 uv259 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f07fc003040 con 0x5560df092530 2024-08-03T20:56:02.429 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:02.427+0000 7f0809904f40 1 -- 172.21.15.32:0/3180561690 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:4 12.10 12:09401a81:::601.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e94) -- 0x5560df1c7f70 con 0x5560df092530 2024-08-03T20:56:02.430 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:02.429+0000 7f0803400640 1 -- 172.21.15.32:0/3180561690 <== osd.0 v2:172.21.15.32:6816/3066156116 4 ==== osd_op_reply(4 601.00000000 [setxattr (30)] v94'260 uv260 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f07fc003040 con 0x5560df092530 2024-08-03T20:56:02.430 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:02.429+0000 7f0809904f40 1 -- 172.21.15.32:0/3180561690 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:5 12.10 12:09401a81:::601.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e94) -- 0x5560df1c8310 con 0x5560df092530 2024-08-03T20:56:02.432 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:02.430+0000 7f0803400640 1 -- 172.21.15.32:0/3180561690 <== osd.0 v2:172.21.15.32:6816/3066156116 5 ==== osd_op_reply(5 601.00000000 [setxattr (62)] v94'261 uv261 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f07fc003040 con 0x5560df092530 2024-08-03T20:56:02.432 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:02.430+0000 7f0809904f40 1 -- 172.21.15.32:0/3180561690 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:6 12.10 12:09401a81:::601.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e94) -- 0x5560df1c6f60 con 0x5560df092530 2024-08-03T20:56:02.434 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:02.432+0000 7f0803400640 1 -- 172.21.15.32:0/3180561690 <== osd.0 v2:172.21.15.32:6816/3066156116 6 ==== osd_op_reply(6 601.00000000 [omap-set-header] v94'262 uv262 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f07fc003040 con 0x5560df092530 2024-08-03T20:56:02.434 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:02.432+0000 7f0809904f40 1 -- 172.21.15.32:0/3180561690 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:7 12.10 12:0bd6d154:::602.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e94) -- 0x5560df1c7350 con 0x5560df092530 2024-08-03T20:56:02.435 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:02.434+0000 7f0803400640 1 -- 172.21.15.32:0/3180561690 <== osd.0 v2:172.21.15.32:6816/3066156116 7 ==== osd_op_reply(7 602.00000000 [create] v94'263 uv237 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f07fc003040 con 0x5560df092530 2024-08-03T20:56:02.435 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:02.434+0000 7f0809904f40 1 -- 172.21.15.32:0/3180561690 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:8 12.10 12:0bd6d154:::602.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e94) -- 0x5560df1c7350 con 0x5560df092530 2024-08-03T20:56:02.437 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:02.435+0000 7f0803400640 1 -- 172.21.15.32:0/3180561690 <== osd.0 v2:172.21.15.32:6816/3066156116 8 ==== osd_op_reply(8 602.00000000 [delete] v94'264 uv264 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f07fc003040 con 0x5560df092530 2024-08-03T20:56:02.437 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:02.436+0000 7f0809904f40 1 -- 172.21.15.32:0/3180561690 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:9 12.10 12:0bd6d154:::602.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e94) -- 0x5560df1c7300 con 0x5560df092530 2024-08-03T20:56:02.438 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::602.00000000:head# 2024-08-03T20:56:02.439 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:02.437+0000 7f0803400640 1 -- 172.21.15.32:0/3180561690 <== osd.0 v2:172.21.15.32:6816/3066156116 9 ==== osd_op_reply(9 602.00000000 [create] v94'265 uv265 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f07fc003040 con 0x5560df092530 2024-08-03T20:56:02.439 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:02.437+0000 7f0809904f40 1 -- 172.21.15.32:0/3180561690 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:10 12.10 12:0bd6d154:::602.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e94) -- 0x5560df1c7300 con 0x5560df092530 2024-08-03T20:56:02.440 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:02.439+0000 7f0803400640 1 -- 172.21.15.32:0/3180561690 <== osd.0 v2:172.21.15.32:6816/3066156116 10 ==== osd_op_reply(10 602.00000000 [setxattr (30)] v94'266 uv266 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f07fc003040 con 0x5560df092530 2024-08-03T20:56:02.440 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:02.439+0000 7f0809904f40 1 -- 172.21.15.32:0/3180561690 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:11 12.10 12:0bd6d154:::602.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e94) -- 0x5560df1c7300 con 0x5560df092530 2024-08-03T20:56:02.442 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:02.440+0000 7f0803400640 1 -- 172.21.15.32:0/3180561690 <== osd.0 v2:172.21.15.32:6816/3066156116 11 ==== osd_op_reply(11 602.00000000 [setxattr (62)] v94'267 uv267 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f07fc003040 con 0x5560df092530 2024-08-03T20:56:02.442 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:02.440+0000 7f0809904f40 1 -- 172.21.15.32:0/3180561690 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:12 12.10 12:0bd6d154:::602.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e94) -- 0x5560df1c7300 con 0x5560df092530 2024-08-03T20:56:02.443 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:02.442+0000 7f0803400640 1 -- 172.21.15.32:0/3180561690 <== osd.0 v2:172.21.15.32:6816/3066156116 12 ==== osd_op_reply(12 602.00000000 [omap-set-header] v94'268 uv268 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f07fc003040 con 0x5560df092530 2024-08-03T20:56:02.443 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:02.442+0000 7f0809904f40 1 -- 172.21.15.32:0/3180561690 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:13 12.10 12:0cf6ad0f:::10000000000.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e94) -- 0x5560df1c7300 con 0x5560df092530 2024-08-03T20:56:02.445 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:02.443+0000 7f0803400640 1 -- 172.21.15.32:0/3180561690 <== osd.0 v2:172.21.15.32:6816/3066156116 13 ==== osd_op_reply(13 10000000000.00000000 [create] v94'269 uv244 ondisk = -17 ((17) File exists)) ==== 164+0+0 (crc 0 0 0) 0x7f07fc003040 con 0x5560df092530 2024-08-03T20:56:02.445 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:02.443+0000 7f0809904f40 1 -- 172.21.15.32:0/3180561690 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:14 12.10 12:0cf6ad0f:::10000000000.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e94) -- 0x5560df1c7300 con 0x5560df092530 2024-08-03T20:56:02.447 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:02.445+0000 7f0803400640 1 -- 172.21.15.32:0/3180561690 <== osd.0 v2:172.21.15.32:6816/3066156116 14 ==== osd_op_reply(14 10000000000.00000000 [delete] v94'270 uv270 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f07fc003040 con 0x5560df092530 2024-08-03T20:56:02.447 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:02.445+0000 7f0809904f40 1 -- 172.21.15.32:0/3180561690 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:15 12.10 12:0cf6ad0f:::10000000000.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e94) -- 0x5560df1c7300 con 0x5560df092530 2024-08-03T20:56:02.448 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:02.447+0000 7f0803400640 1 -- 172.21.15.32:0/3180561690 <== osd.0 v2:172.21.15.32:6816/3066156116 15 ==== osd_op_reply(15 10000000000.00000000 [create] v94'271 uv271 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f07fc003040 con 0x5560df092530 2024-08-03T20:56:02.448 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::10000000000.00000000:head# 2024-08-03T20:56:02.449 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:02.447+0000 7f0809904f40 1 -- 172.21.15.32:0/3180561690 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:16 12.10 12:0cf6ad0f:::10000000000.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e94) -- 0x5560df1c7300 con 0x5560df092530 2024-08-03T20:56:02.449 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:02.448+0000 7f0803400640 1 -- 172.21.15.32:0/3180561690 <== osd.0 v2:172.21.15.32:6816/3066156116 16 ==== osd_op_reply(16 10000000000.00000000 [setxattr (30)] v94'272 uv272 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f07fc003040 con 0x5560df092530 2024-08-03T20:56:02.450 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:02.448+0000 7f0809904f40 1 -- 172.21.15.32:0/3180561690 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:17 12.10 12:0cf6ad0f:::10000000000.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e94) -- 0x5560df1c7300 con 0x5560df092530 2024-08-03T20:56:02.451 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:02.450+0000 7f0803400640 1 -- 172.21.15.32:0/3180561690 <== osd.0 v2:172.21.15.32:6816/3066156116 17 ==== osd_op_reply(17 10000000000.00000000 [setxattr (62)] v94'273 uv273 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f07fc003040 con 0x5560df092530 2024-08-03T20:56:02.451 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:02.450+0000 7f0809904f40 1 -- 172.21.15.32:0/3180561690 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:18 12.10 12:0cf6ad0f:::10000000000.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e94) -- 0x5560df1c7300 con 0x5560df092530 2024-08-03T20:56:02.453 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:02.451+0000 7f0803400640 1 -- 172.21.15.32:0/3180561690 <== osd.0 v2:172.21.15.32:6816/3066156116 18 ==== osd_op_reply(18 10000000000.00000000 [omap-set-header] v94'274 uv274 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f07fc003040 con 0x5560df092530 2024-08-03T20:56:02.453 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:02.451+0000 7f0809904f40 1 -- 172.21.15.32:0/3180561690 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:19 12.10 12:0cf6ad0f:::10000000000.00000000:head [omap-set-vals in=593b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e94) -- 0x5560df1c7770 con 0x5560df092530 2024-08-03T20:56:02.454 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:02.453+0000 7f0803400640 1 -- 172.21.15.32:0/3180561690 <== osd.0 v2:172.21.15.32:6816/3066156116 19 ==== osd_op_reply(19 10000000000.00000000 [omap-set-vals] v94'275 uv275 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f07fc003040 con 0x5560df092530 2024-08-03T20:56:02.454 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:02.453+0000 7f0809904f40 1 -- 172.21.15.32:0/3180561690 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:20 12.10 12:0d82a743:::600.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e94) -- 0x5560df1c76c0 con 0x5560df092530 2024-08-03T20:56:02.456 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:02.455+0000 7f0803400640 1 -- 172.21.15.32:0/3180561690 <== osd.0 v2:172.21.15.32:6816/3066156116 20 ==== osd_op_reply(20 600.00000000 [create] v94'276 uv250 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f07fc003040 con 0x5560df092530 2024-08-03T20:56:02.456 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:02.455+0000 7f0809904f40 1 -- 172.21.15.32:0/3180561690 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:21 12.10 12:0d82a743:::600.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e94) -- 0x5560df1c75c0 con 0x5560df092530 2024-08-03T20:56:02.458 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:02.456+0000 7f0803400640 1 -- 172.21.15.32:0/3180561690 <== osd.0 v2:172.21.15.32:6816/3066156116 21 ==== osd_op_reply(21 600.00000000 [delete] v94'277 uv277 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f07fc003040 con 0x5560df092530 2024-08-03T20:56:02.458 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:02.456+0000 7f0809904f40 1 -- 172.21.15.32:0/3180561690 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:22 12.10 12:0d82a743:::600.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e94) -- 0x5560df1c75c0 con 0x5560df092530 2024-08-03T20:56:02.459 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:02.458+0000 7f0803400640 1 -- 172.21.15.32:0/3180561690 <== osd.0 v2:172.21.15.32:6816/3066156116 22 ==== osd_op_reply(22 600.00000000 [create] v94'278 uv278 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f07fc003040 con 0x5560df092530 2024-08-03T20:56:02.459 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::600.00000000:head# 2024-08-03T20:56:02.459 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:02.458+0000 7f0809904f40 1 -- 172.21.15.32:0/3180561690 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:23 12.10 12:0d82a743:::600.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e94) -- 0x5560df1c75c0 con 0x5560df092530 2024-08-03T20:56:02.460 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:02.459+0000 7f0803400640 1 -- 172.21.15.32:0/3180561690 <== osd.0 v2:172.21.15.32:6816/3066156116 23 ==== osd_op_reply(23 600.00000000 [setxattr (30)] v94'279 uv279 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f07fc003040 con 0x5560df092530 2024-08-03T20:56:02.460 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:02.459+0000 7f0809904f40 1 -- 172.21.15.32:0/3180561690 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:24 12.10 12:0d82a743:::600.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e94) -- 0x5560df1c75c0 con 0x5560df092530 2024-08-03T20:56:02.462 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:02.461+0000 7f0803400640 1 -- 172.21.15.32:0/3180561690 <== osd.0 v2:172.21.15.32:6816/3066156116 24 ==== osd_op_reply(24 600.00000000 [setxattr (62)] v94'280 uv280 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f07fc003040 con 0x5560df092530 2024-08-03T20:56:02.462 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:02.461+0000 7f0809904f40 1 -- 172.21.15.32:0/3180561690 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:25 12.10 12:0d82a743:::600.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e94) -- 0x5560df1c75c0 con 0x5560df092530 2024-08-03T20:56:02.464 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:02.462+0000 7f0803400640 1 -- 172.21.15.32:0/3180561690 <== osd.0 v2:172.21.15.32:6816/3066156116 25 ==== osd_op_reply(25 600.00000000 [omap-set-header] v94'281 uv281 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f07fc003040 con 0x5560df092530 2024-08-03T20:56:02.464 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:02.462+0000 7f0809904f40 1 -- 172.21.15.32:0/3180561690 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:26 12.10 12:0d89b25e:::603.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e94) -- 0x5560df1c75c0 con 0x5560df092530 2024-08-03T20:56:02.465 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:02.464+0000 7f0803400640 1 -- 172.21.15.32:0/3180561690 <== osd.0 v2:172.21.15.32:6816/3066156116 26 ==== osd_op_reply(26 603.00000000 [create] v94'282 uv256 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f07fc003040 con 0x5560df092530 2024-08-03T20:56:02.465 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:02.464+0000 7f0809904f40 1 -- 172.21.15.32:0/3180561690 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:27 12.10 12:0d89b25e:::603.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e94) -- 0x5560df1c75c0 con 0x5560df092530 2024-08-03T20:56:02.467 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:02.465+0000 7f0803400640 1 -- 172.21.15.32:0/3180561690 <== osd.0 v2:172.21.15.32:6816/3066156116 27 ==== osd_op_reply(27 603.00000000 [delete] v94'283 uv283 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f07fc003040 con 0x5560df092530 2024-08-03T20:56:02.467 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:02.465+0000 7f0809904f40 1 -- 172.21.15.32:0/3180561690 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:28 12.10 12:0d89b25e:::603.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e94) -- 0x5560df1c75c0 con 0x5560df092530 2024-08-03T20:56:02.470 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::603.00000000:head# 2024-08-03T20:56:02.472 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:02.467+0000 7f0803400640 1 -- 172.21.15.32:0/3180561690 <== osd.0 v2:172.21.15.32:6816/3066156116 28 ==== osd_op_reply(28 603.00000000 [create] v94'284 uv284 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f07fc003040 con 0x5560df092530 2024-08-03T20:56:02.472 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:02.467+0000 7f0809904f40 1 -- 172.21.15.32:0/3180561690 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:29 12.10 12:0d89b25e:::603.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e94) -- 0x5560df1c75c0 con 0x5560df092530 2024-08-03T20:56:02.472 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:02.468+0000 7f0803400640 1 -- 172.21.15.32:0/3180561690 <== osd.0 v2:172.21.15.32:6816/3066156116 29 ==== osd_op_reply(29 603.00000000 [setxattr (30)] v94'285 uv285 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f07fc003040 con 0x5560df092530 2024-08-03T20:56:02.472 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:02.468+0000 7f0809904f40 1 -- 172.21.15.32:0/3180561690 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:30 12.10 12:0d89b25e:::603.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e94) -- 0x5560df1c75c0 con 0x5560df092530 2024-08-03T20:56:02.472 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:02.470+0000 7f0803400640 1 -- 172.21.15.32:0/3180561690 <== osd.0 v2:172.21.15.32:6816/3066156116 30 ==== osd_op_reply(30 603.00000000 [setxattr (62)] v94'286 uv286 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f07fc003040 con 0x5560df092530 2024-08-03T20:56:02.472 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:02.470+0000 7f0809904f40 1 -- 172.21.15.32:0/3180561690 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:31 12.10 12:0d89b25e:::603.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e94) -- 0x5560df1c75c0 con 0x5560df092530 2024-08-03T20:56:02.473 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:02.471+0000 7f0803400640 1 -- 172.21.15.32:0/3180561690 <== osd.0 v2:172.21.15.32:6816/3066156116 31 ==== osd_op_reply(31 603.00000000 [omap-set-header] v94'287 uv287 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f07fc003040 con 0x5560df092530 2024-08-03T20:56:02.473 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:02.472+0000 7f0809904f40 1 --2- 172.21.15.32:0/3180561690 >> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] conn(0x5560df1c86b0 0x5560df1caad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:56:02.473 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:02.472+0000 7f0809904f40 1 -- 172.21.15.32:0/3180561690 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:32 12.14 12:287034c4:::1.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e94) -- 0x5560df1c75c0 con 0x5560df1c86b0 2024-08-03T20:56:02.473 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:02.472+0000 7f0802a00640 1 --2- 172.21.15.32:0/3180561690 >> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] conn(0x5560df1c86b0 0x5560df1caad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:56:02.474 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:02.473+0000 7f0802a00640 1 --2- 172.21.15.32:0/3180561690 >> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] conn(0x5560df1c86b0 0x5560df1caad0 crc :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:56:02.475 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:02.474+0000 7f0802a00640 1 -- 172.21.15.32:0/3180561690 <== osd.5 v2:172.21.15.118:6810/2724891770 1 ==== osd_op_reply(32 1.00000000.inode [create] v94'198 uv181 ondisk = -17 ((17) File exists)) ==== 160+0+0 (crc 0 0 0) 0x7f07f401b410 con 0x5560df1c86b0 2024-08-03T20:56:02.476 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:02.474+0000 7f0809904f40 1 -- 172.21.15.32:0/3180561690 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:33 12.14 12:287034c4:::1.00000000.inode:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e94) -- 0x5560df1cb010 con 0x5560df1c86b0 2024-08-03T20:56:02.477 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:02.476+0000 7f0802a00640 1 -- 172.21.15.32:0/3180561690 <== osd.5 v2:172.21.15.118:6810/2724891770 2 ==== osd_op_reply(33 1.00000000.inode [delete] v94'199 uv199 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f07f401b410 con 0x5560df1c86b0 2024-08-03T20:56:02.477 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:02.476+0000 7f0809904f40 1 -- 172.21.15.32:0/3180561690 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:34 12.14 12:287034c4:::1.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e94) -- 0x5560df1cb3b0 con 0x5560df1c86b0 2024-08-03T20:56:02.478 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:02.477+0000 7f0802a00640 1 -- 172.21.15.32:0/3180561690 <== osd.5 v2:172.21.15.118:6810/2724891770 3 ==== osd_op_reply(34 1.00000000.inode [create] v94'200 uv200 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f07f401b410 con 0x5560df1c86b0 2024-08-03T20:56:02.479 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::1.00000000.inode:head# 2024-08-03T20:56:02.479 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:02.477+0000 7f0809904f40 1 -- 172.21.15.32:0/3180561690 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:35 12.14 12:287034c4:::1.00000000.inode:head [write 0~582 in=582b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e94) -- 0x5560df1cb750 con 0x5560df1c86b0 2024-08-03T20:56:02.480 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:02.479+0000 7f0802a00640 1 -- 172.21.15.32:0/3180561690 <== osd.5 v2:172.21.15.118:6810/2724891770 4 ==== osd_op_reply(35 1.00000000.inode [write 0~582] v94'201 uv201 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f07f401b410 con 0x5560df1c86b0 2024-08-03T20:56:02.480 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:02.479+0000 7f0809904f40 1 -- 172.21.15.32:0/3180561690 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:36 12.14 12:287034c4:::1.00000000.inode:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e94) -- 0x5560df1cbaf0 con 0x5560df1c86b0 2024-08-03T20:56:02.482 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:02.481+0000 7f0802a00640 1 -- 172.21.15.32:0/3180561690 <== osd.5 v2:172.21.15.118:6810/2724891770 5 ==== osd_op_reply(36 1.00000000.inode [omap-set-header] v94'202 uv202 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f07f401b410 con 0x5560df1c86b0 2024-08-03T20:56:02.482 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:02.481+0000 7f0809904f40 1 -- 172.21.15.32:0/3180561690 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:37 12.14 12:292cf221:::200.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e94) -- 0x5560df1cbe90 con 0x5560df1c86b0 2024-08-03T20:56:02.483 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:02.482+0000 7f0802a00640 1 -- 172.21.15.32:0/3180561690 <== osd.5 v2:172.21.15.118:6810/2724891770 6 ==== osd_op_reply(37 200.00000000 [create] v94'203 uv186 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f07f401b410 con 0x5560df1c86b0 2024-08-03T20:56:02.484 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:02.482+0000 7f0809904f40 1 -- 172.21.15.32:0/3180561690 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:38 12.14 12:292cf221:::200.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e94) -- 0x5560df1cc230 con 0x5560df1c86b0 2024-08-03T20:56:02.485 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:02.484+0000 7f0802a00640 1 -- 172.21.15.32:0/3180561690 <== osd.5 v2:172.21.15.118:6810/2724891770 7 ==== osd_op_reply(38 200.00000000 [delete] v94'204 uv204 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f07f401b410 con 0x5560df1c86b0 2024-08-03T20:56:02.485 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:02.484+0000 7f0809904f40 1 -- 172.21.15.32:0/3180561690 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:39 12.14 12:292cf221:::200.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e94) -- 0x5560df1cc5d0 con 0x5560df1c86b0 2024-08-03T20:56:02.486 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:02.485+0000 7f0802a00640 1 -- 172.21.15.32:0/3180561690 <== osd.5 v2:172.21.15.118:6810/2724891770 8 ==== osd_op_reply(39 200.00000000 [create] v94'205 uv205 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f07f401b410 con 0x5560df1c86b0 2024-08-03T20:56:02.487 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::200.00000000:head# 2024-08-03T20:56:02.487 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:02.485+0000 7f0809904f40 1 -- 172.21.15.32:0/3180561690 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:40 12.14 12:292cf221:::200.00000000:head [write 0~90 in=90b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e94) -- 0x5560df1cc5d0 con 0x5560df1c86b0 2024-08-03T20:56:02.488 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:02.487+0000 7f0802a00640 1 -- 172.21.15.32:0/3180561690 <== osd.5 v2:172.21.15.118:6810/2724891770 9 ==== osd_op_reply(40 200.00000000 [write 0~90] v94'206 uv206 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f07f401b410 con 0x5560df1c86b0 2024-08-03T20:56:02.488 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:02.487+0000 7f0809904f40 1 -- 172.21.15.32:0/3180561690 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:41 12.14 12:292cf221:::200.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e94) -- 0x5560df1cc5d0 con 0x5560df1c86b0 2024-08-03T20:56:02.489 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:02.488+0000 7f0802a00640 1 -- 172.21.15.32:0/3180561690 <== osd.5 v2:172.21.15.118:6810/2724891770 10 ==== osd_op_reply(41 200.00000000 [omap-set-header] v94'207 uv207 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f07f401b410 con 0x5560df1c86b0 2024-08-03T20:56:02.490 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:02.488+0000 7f0809904f40 1 -- 172.21.15.32:0/3180561690 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:42 12.14 12:2e2fa760:::200.00000001:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e94) -- 0x5560df1cc5d0 con 0x5560df1c86b0 2024-08-03T20:56:02.491 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:02.490+0000 7f0802a00640 1 -- 172.21.15.32:0/3180561690 <== osd.5 v2:172.21.15.118:6810/2724891770 11 ==== osd_op_reply(42 200.00000001 [create] v94'208 uv191 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f07f401b410 con 0x5560df1c86b0 2024-08-03T20:56:02.491 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:02.490+0000 7f0809904f40 1 -- 172.21.15.32:0/3180561690 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:43 12.14 12:2e2fa760:::200.00000001:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e94) -- 0x5560df1cc5d0 con 0x5560df1c86b0 2024-08-03T20:56:02.493 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:02.491+0000 7f0802a00640 1 -- 172.21.15.32:0/3180561690 <== osd.5 v2:172.21.15.118:6810/2724891770 12 ==== osd_op_reply(43 200.00000001 [delete] v94'209 uv209 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f07f401b410 con 0x5560df1c86b0 2024-08-03T20:56:02.493 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:02.491+0000 7f0809904f40 1 -- 172.21.15.32:0/3180561690 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:44 12.14 12:2e2fa760:::200.00000001:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e94) -- 0x5560df1cc5d0 con 0x5560df1c86b0 2024-08-03T20:56:02.494 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:02.493+0000 7f0802a00640 1 -- 172.21.15.32:0/3180561690 <== osd.5 v2:172.21.15.118:6810/2724891770 13 ==== osd_op_reply(44 200.00000001 [create] v94'210 uv210 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f07f401b410 con 0x5560df1c86b0 2024-08-03T20:56:02.495 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::200.00000001:head# 2024-08-03T20:56:02.495 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:02.493+0000 7f0809904f40 1 -- 172.21.15.32:0/3180561690 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:45 12.14 12:2e2fa760:::200.00000001:head [write 0~18118 in=18118b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e94) -- 0x5560df1d0cc0 con 0x5560df1c86b0 2024-08-03T20:56:02.496 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:02.495+0000 7f0802a00640 1 -- 172.21.15.32:0/3180561690 <== osd.5 v2:172.21.15.118:6810/2724891770 14 ==== osd_op_reply(45 200.00000001 [write 0~18118] v94'211 uv211 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f07f401b410 con 0x5560df1c86b0 2024-08-03T20:56:02.496 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:02.495+0000 7f0809904f40 1 -- 172.21.15.32:0/3180561690 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:46 12.14 12:2e2fa760:::200.00000001:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e94) -- 0x5560df1d0cc0 con 0x5560df1c86b0 2024-08-03T20:56:02.498 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:02.496+0000 7f0802a00640 1 -- 172.21.15.32:0/3180561690 <== osd.5 v2:172.21.15.118:6810/2724891770 15 ==== osd_op_reply(46 200.00000001 [omap-set-header] v94'212 uv212 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f07f401b410 con 0x5560df1c86b0 2024-08-03T20:56:02.498 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:02.497+0000 7f0809904f40 1 -- 172.21.15.32:0/3180561690 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:47 12.14 12:2f55791f:::606.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e94) -- 0x5560df1cc5d0 con 0x5560df1c86b0 2024-08-03T20:56:02.499 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:02.498+0000 7f0802a00640 1 -- 172.21.15.32:0/3180561690 <== osd.5 v2:172.21.15.118:6810/2724891770 16 ==== osd_op_reply(47 606.00000000 [create] v94'213 uv197 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f07f401b410 con 0x5560df1c86b0 2024-08-03T20:56:02.499 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:02.498+0000 7f0809904f40 1 -- 172.21.15.32:0/3180561690 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:48 12.14 12:2f55791f:::606.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e94) -- 0x5560df1cc5d0 con 0x5560df1c86b0 2024-08-03T20:56:02.501 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:02.500+0000 7f0802a00640 1 -- 172.21.15.32:0/3180561690 <== osd.5 v2:172.21.15.118:6810/2724891770 17 ==== osd_op_reply(48 606.00000000 [delete] v94'214 uv214 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f07f401b410 con 0x5560df1c86b0 2024-08-03T20:56:02.501 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:02.500+0000 7f0809904f40 1 -- 172.21.15.32:0/3180561690 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:49 12.14 12:2f55791f:::606.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e94) -- 0x5560df1cc5d0 con 0x5560df1c86b0 2024-08-03T20:56:02.502 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:02.501+0000 7f0802a00640 1 -- 172.21.15.32:0/3180561690 <== osd.5 v2:172.21.15.118:6810/2724891770 18 ==== osd_op_reply(49 606.00000000 [create] v94'215 uv215 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f07f401b410 con 0x5560df1c86b0 2024-08-03T20:56:02.502 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::606.00000000:head# 2024-08-03T20:56:02.503 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:02.501+0000 7f0809904f40 1 -- 172.21.15.32:0/3180561690 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:50 12.14 12:2f55791f:::606.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e94) -- 0x5560df1cd5e0 con 0x5560df1c86b0 2024-08-03T20:56:02.504 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:02.502+0000 7f0802a00640 1 -- 172.21.15.32:0/3180561690 <== osd.5 v2:172.21.15.118:6810/2724891770 19 ==== osd_op_reply(50 606.00000000 [setxattr (30)] v94'216 uv216 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f07f401b410 con 0x5560df1c86b0 2024-08-03T20:56:02.504 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:02.502+0000 7f0809904f40 1 -- 172.21.15.32:0/3180561690 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:51 12.14 12:2f55791f:::606.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e94) -- 0x5560df1cd5e0 con 0x5560df1c86b0 2024-08-03T20:56:02.505 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:02.504+0000 7f0802a00640 1 -- 172.21.15.32:0/3180561690 <== osd.5 v2:172.21.15.118:6810/2724891770 20 ==== osd_op_reply(51 606.00000000 [setxattr (62)] v94'217 uv217 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f07f401b410 con 0x5560df1c86b0 2024-08-03T20:56:02.505 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:02.504+0000 7f0809904f40 1 -- 172.21.15.32:0/3180561690 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:52 12.14 12:2f55791f:::606.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e94) -- 0x5560df1cc5d0 con 0x5560df1c86b0 2024-08-03T20:56:02.507 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:02.505+0000 7f0802a00640 1 -- 172.21.15.32:0/3180561690 <== osd.5 v2:172.21.15.118:6810/2724891770 21 ==== osd_op_reply(52 606.00000000 [omap-set-header] v94'218 uv218 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f07f401b410 con 0x5560df1c86b0 2024-08-03T20:56:02.507 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:02.505+0000 7f0809904f40 1 -- 172.21.15.32:0/3180561690 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:53 12.1c 12:3ed09add:::607.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e94) -- 0x5560df1cc5d0 con 0x5560df092530 2024-08-03T20:56:02.508 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:02.507+0000 7f0803400640 1 -- 172.21.15.32:0/3180561690 <== osd.0 v2:172.21.15.32:6816/3066156116 32 ==== osd_op_reply(53 607.00000000 [create] v94'51 uv50 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f07fc003040 con 0x5560df092530 2024-08-03T20:56:02.508 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:02.507+0000 7f0809904f40 1 -- 172.21.15.32:0/3180561690 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:54 12.1c 12:3ed09add:::607.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e94) -- 0x5560df1cc5d0 con 0x5560df092530 2024-08-03T20:56:02.509 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:02.508+0000 7f0803400640 1 -- 172.21.15.32:0/3180561690 <== osd.0 v2:172.21.15.32:6816/3066156116 33 ==== osd_op_reply(54 607.00000000 [delete] v94'52 uv52 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f07fc003040 con 0x5560df092530 2024-08-03T20:56:02.510 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:02.508+0000 7f0809904f40 1 -- 172.21.15.32:0/3180561690 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:55 12.1c 12:3ed09add:::607.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e94) -- 0x5560df1cc5d0 con 0x5560df092530 2024-08-03T20:56:02.511 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:02.509+0000 7f0803400640 1 -- 172.21.15.32:0/3180561690 <== osd.0 v2:172.21.15.32:6816/3066156116 34 ==== osd_op_reply(55 607.00000000 [create] v94'53 uv53 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f07fc003040 con 0x5560df092530 2024-08-03T20:56:02.511 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::607.00000000:head# 2024-08-03T20:56:02.511 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:02.509+0000 7f0809904f40 1 -- 172.21.15.32:0/3180561690 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:56 12.1c 12:3ed09add:::607.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e94) -- 0x5560df1cd5e0 con 0x5560df092530 2024-08-03T20:56:02.512 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:02.511+0000 7f0803400640 1 -- 172.21.15.32:0/3180561690 <== osd.0 v2:172.21.15.32:6816/3066156116 35 ==== osd_op_reply(56 607.00000000 [setxattr (30)] v94'54 uv54 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f07fc003040 con 0x5560df092530 2024-08-03T20:56:02.512 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:02.511+0000 7f0809904f40 1 -- 172.21.15.32:0/3180561690 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:57 12.1c 12:3ed09add:::607.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e94) -- 0x5560df1cd5e0 con 0x5560df092530 2024-08-03T20:56:02.514 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:02.512+0000 7f0803400640 1 -- 172.21.15.32:0/3180561690 <== osd.0 v2:172.21.15.32:6816/3066156116 36 ==== osd_op_reply(57 607.00000000 [setxattr (62)] v94'55 uv55 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f07fc003040 con 0x5560df092530 2024-08-03T20:56:02.514 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:02.512+0000 7f0809904f40 1 -- 172.21.15.32:0/3180561690 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:58 12.1c 12:3ed09add:::607.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e94) -- 0x5560df1cc5d0 con 0x5560df092530 2024-08-03T20:56:02.515 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:02.514+0000 7f0803400640 1 -- 172.21.15.32:0/3180561690 <== osd.0 v2:172.21.15.32:6816/3066156116 37 ==== osd_op_reply(58 607.00000000 [omap-set-header] v94'56 uv56 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f07fc003040 con 0x5560df092530 2024-08-03T20:56:02.515 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:02.514+0000 7f0809904f40 1 --2- 172.21.15.32:0/3180561690 >> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] conn(0x5560df1ccae0 0x5560df1cef00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:56:02.515 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:02.514+0000 7f0809904f40 1 -- 172.21.15.32:0/3180561690 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:59 12.6 12:654134d2:::mds0_openfiles.0:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e94) -- 0x5560df1cf440 con 0x5560df1ccae0 2024-08-03T20:56:02.515 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:02.514+0000 7f0802000640 1 --2- 172.21.15.32:0/3180561690 >> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] conn(0x5560df1ccae0 0x5560df1cef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:56:02.516 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:02.515+0000 7f0802000640 1 --2- 172.21.15.32:0/3180561690 >> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] conn(0x5560df1ccae0 0x5560df1cef00 crc :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:56:02.517 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:02.516+0000 7f0802000640 1 -- 172.21.15.32:0/3180561690 <== osd.1 v2:172.21.15.32:6824/1551241759 1 ==== osd_op_reply(59 mds0_openfiles.0 [create] v94'47 uv46 ondisk = -17 ((17) File exists)) ==== 160+0+0 (crc 0 0 0) 0x7f07f00a1410 con 0x5560df1ccae0 2024-08-03T20:56:02.518 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:02.516+0000 7f0809904f40 1 -- 172.21.15.32:0/3180561690 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:60 12.6 12:654134d2:::mds0_openfiles.0:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e94) -- 0x5560df1cf840 con 0x5560df1ccae0 2024-08-03T20:56:02.519 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:02.517+0000 7f0802000640 1 -- 172.21.15.32:0/3180561690 <== osd.1 v2:172.21.15.32:6824/1551241759 2 ==== osd_op_reply(60 mds0_openfiles.0 [delete] v94'48 uv48 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f07f00a1410 con 0x5560df1ccae0 2024-08-03T20:56:02.519 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:02.517+0000 7f0809904f40 1 -- 172.21.15.32:0/3180561690 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:61 12.6 12:654134d2:::mds0_openfiles.0:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e94) -- 0x5560df1cfd40 con 0x5560df1ccae0 2024-08-03T20:56:02.520 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:02.519+0000 7f0802000640 1 -- 172.21.15.32:0/3180561690 <== osd.1 v2:172.21.15.32:6824/1551241759 3 ==== osd_op_reply(61 mds0_openfiles.0 [create] v94'49 uv49 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f07f00a1410 con 0x5560df1ccae0 2024-08-03T20:56:02.520 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds0_openfiles.0:head# 2024-08-03T20:56:02.520 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:02.519+0000 7f0809904f40 1 -- 172.21.15.32:0/3180561690 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:62 12.6 12:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e94) -- 0x5560df102620 con 0x5560df1ccae0 2024-08-03T20:56:02.521 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:02.520+0000 7f0802000640 1 -- 172.21.15.32:0/3180561690 <== osd.1 v2:172.21.15.32:6824/1551241759 4 ==== osd_op_reply(62 mds0_openfiles.0 [omap-set-header] v94'50 uv50 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f07f00a1410 con 0x5560df1ccae0 2024-08-03T20:56:02.521 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:02.520+0000 7f0809904f40 1 -- 172.21.15.32:0/3180561690 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:63 12.e 12:717a0223:::608.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e94) -- 0x5560df1d0670 con 0x5560df092530 2024-08-03T20:56:02.523 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:02.522+0000 7f0803400640 1 -- 172.21.15.32:0/3180561690 <== osd.0 v2:172.21.15.32:6816/3066156116 38 ==== osd_op_reply(63 608.00000000 [create] v94'51 uv50 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f07fc003040 con 0x5560df092530 2024-08-03T20:56:02.523 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:02.522+0000 7f0809904f40 1 -- 172.21.15.32:0/3180561690 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:64 12.e 12:717a0223:::608.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e94) -- 0x5560df1d0670 con 0x5560df092530 2024-08-03T20:56:02.525 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:02.523+0000 7f0803400640 1 -- 172.21.15.32:0/3180561690 <== osd.0 v2:172.21.15.32:6816/3066156116 39 ==== osd_op_reply(64 608.00000000 [delete] v94'52 uv52 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f07fc003040 con 0x5560df092530 2024-08-03T20:56:02.525 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:02.523+0000 7f0809904f40 1 -- 172.21.15.32:0/3180561690 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:65 12.e 12:717a0223:::608.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e94) -- 0x5560df1d0670 con 0x5560df092530 2024-08-03T20:56:02.526 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::608.00000000:head# 2024-08-03T20:56:02.527 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:02.525+0000 7f0803400640 1 -- 172.21.15.32:0/3180561690 <== osd.0 v2:172.21.15.32:6816/3066156116 40 ==== osd_op_reply(65 608.00000000 [create] v94'53 uv53 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f07fc003040 con 0x5560df092530 2024-08-03T20:56:02.527 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:02.525+0000 7f0809904f40 1 -- 172.21.15.32:0/3180561690 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:66 12.e 12:717a0223:::608.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e94) -- 0x5560df1d1680 con 0x5560df092530 2024-08-03T20:56:02.528 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:02.526+0000 7f0803400640 1 -- 172.21.15.32:0/3180561690 <== osd.0 v2:172.21.15.32:6816/3066156116 41 ==== osd_op_reply(66 608.00000000 [setxattr (30)] v94'54 uv54 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f07fc003040 con 0x5560df092530 2024-08-03T20:56:02.528 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:02.526+0000 7f0809904f40 1 -- 172.21.15.32:0/3180561690 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:67 12.e 12:717a0223:::608.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e94) -- 0x5560df1d1680 con 0x5560df092530 2024-08-03T20:56:02.529 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:02.528+0000 7f0803400640 1 -- 172.21.15.32:0/3180561690 <== osd.0 v2:172.21.15.32:6816/3066156116 42 ==== osd_op_reply(67 608.00000000 [setxattr (62)] v94'55 uv55 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f07fc003040 con 0x5560df092530 2024-08-03T20:56:02.529 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:02.528+0000 7f0809904f40 1 -- 172.21.15.32:0/3180561690 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:68 12.e 12:717a0223:::608.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e94) -- 0x5560df1d0670 con 0x5560df092530 2024-08-03T20:56:02.531 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:02.529+0000 7f0803400640 1 -- 172.21.15.32:0/3180561690 <== osd.0 v2:172.21.15.32:6816/3066156116 43 ==== osd_op_reply(68 608.00000000 [omap-set-header] v94'56 uv56 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f07fc003040 con 0x5560df092530 2024-08-03T20:56:02.531 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:02.530+0000 7f0809904f40 1 --2- 172.21.15.32:0/3180561690 >> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] conn(0x5560df1d0f40 0x5560df1d3360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:56:02.531 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:02.530+0000 7f0809904f40 1 -- 172.21.15.32:0/3180561690 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:69 12.15 12:a93a17c2:::604.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e94) -- 0x5560df1d38a0 con 0x5560df1d0f40 2024-08-03T20:56:02.531 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:02.530+0000 7f0803400640 1 --2- 172.21.15.32:0/3180561690 >> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] conn(0x5560df1d0f40 0x5560df1d3360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:56:02.532 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:02.531+0000 7f0803400640 1 --2- 172.21.15.32:0/3180561690 >> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] conn(0x5560df1d0f40 0x5560df1d3360 crc :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:56:02.533 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:02.532+0000 7f0803400640 1 -- 172.21.15.32:0/3180561690 <== osd.6 v2:172.21.15.118:6825/801947386 1 ==== osd_op_reply(69 604.00000000 [create] v94'94 uv88 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f07fc003040 con 0x5560df1d0f40 2024-08-03T20:56:02.533 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:02.532+0000 7f0809904f40 1 -- 172.21.15.32:0/3180561690 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:70 12.15 12:a93a17c2:::604.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e94) -- 0x5560df1d38e0 con 0x5560df1d0f40 2024-08-03T20:56:02.535 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:02.533+0000 7f0803400640 1 -- 172.21.15.32:0/3180561690 <== osd.6 v2:172.21.15.118:6825/801947386 2 ==== osd_op_reply(70 604.00000000 [delete] v94'95 uv95 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f07fc003040 con 0x5560df1d0f40 2024-08-03T20:56:02.535 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:02.533+0000 7f0809904f40 1 -- 172.21.15.32:0/3180561690 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:71 12.15 12:a93a17c2:::604.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e94) -- 0x5560df1d3cd0 con 0x5560df1d0f40 2024-08-03T20:56:02.536 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::604.00000000:head# 2024-08-03T20:56:02.536 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:02.535+0000 7f0803400640 1 -- 172.21.15.32:0/3180561690 <== osd.6 v2:172.21.15.118:6825/801947386 3 ==== osd_op_reply(71 604.00000000 [create] v94'96 uv96 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f07fc003040 con 0x5560df1d0f40 2024-08-03T20:56:02.536 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:02.535+0000 7f0809904f40 1 -- 172.21.15.32:0/3180561690 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:72 12.15 12:a93a17c2:::604.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e94) -- 0x5560df1d4ce0 con 0x5560df1d0f40 2024-08-03T20:56:02.537 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:02.536+0000 7f0803400640 1 -- 172.21.15.32:0/3180561690 <== osd.6 v2:172.21.15.118:6825/801947386 4 ==== osd_op_reply(72 604.00000000 [setxattr (30)] v94'97 uv97 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f07fc003040 con 0x5560df1d0f40 2024-08-03T20:56:02.537 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:02.536+0000 7f0809904f40 1 -- 172.21.15.32:0/3180561690 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:73 12.15 12:a93a17c2:::604.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e94) -- 0x5560df1d4ce0 con 0x5560df1d0f40 2024-08-03T20:56:02.538 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:02.537+0000 7f0803400640 1 -- 172.21.15.32:0/3180561690 <== osd.6 v2:172.21.15.118:6825/801947386 5 ==== osd_op_reply(73 604.00000000 [setxattr (62)] v94'98 uv98 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f07fc003040 con 0x5560df1d0f40 2024-08-03T20:56:02.538 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:02.537+0000 7f0809904f40 1 -- 172.21.15.32:0/3180561690 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:74 12.15 12:a93a17c2:::604.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e94) -- 0x5560df1d3cd0 con 0x5560df1d0f40 2024-08-03T20:56:02.540 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:02.538+0000 7f0803400640 1 -- 172.21.15.32:0/3180561690 <== osd.6 v2:172.21.15.118:6825/801947386 6 ==== osd_op_reply(74 604.00000000 [omap-set-header] v94'99 uv99 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f07fc003040 con 0x5560df1d0f40 2024-08-03T20:56:02.540 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:02.538+0000 7f0809904f40 1 -- 172.21.15.32:0/3180561690 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:75 12.15 12:aa448500:::500.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e94) -- 0x5560df1d3cd0 con 0x5560df1d0f40 2024-08-03T20:56:02.541 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:02.540+0000 7f0803400640 1 -- 172.21.15.32:0/3180561690 <== osd.6 v2:172.21.15.118:6825/801947386 7 ==== osd_op_reply(75 500.00000000 [create] v94'100 uv93 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f07fc003040 con 0x5560df1d0f40 2024-08-03T20:56:02.541 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:02.540+0000 7f0809904f40 1 -- 172.21.15.32:0/3180561690 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:76 12.15 12:aa448500:::500.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e94) -- 0x5560df1d3cd0 con 0x5560df1d0f40 2024-08-03T20:56:02.543 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:02.541+0000 7f0803400640 1 -- 172.21.15.32:0/3180561690 <== osd.6 v2:172.21.15.118:6825/801947386 8 ==== osd_op_reply(76 500.00000000 [delete] v94'101 uv101 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f07fc003040 con 0x5560df1d0f40 2024-08-03T20:56:02.543 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:02.541+0000 7f0809904f40 1 -- 172.21.15.32:0/3180561690 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:77 12.15 12:aa448500:::500.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e94) -- 0x5560df1d3cd0 con 0x5560df1d0f40 2024-08-03T20:56:02.544 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::500.00000000:head# 2024-08-03T20:56:02.545 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:02.543+0000 7f0803400640 1 -- 172.21.15.32:0/3180561690 <== osd.6 v2:172.21.15.118:6825/801947386 9 ==== osd_op_reply(77 500.00000000 [create] v94'102 uv102 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f07fc003040 con 0x5560df1d0f40 2024-08-03T20:56:02.545 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:02.543+0000 7f0809904f40 1 -- 172.21.15.32:0/3180561690 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:78 12.15 12:aa448500:::500.00000000:head [write 0~90 in=90b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e94) -- 0x5560df1d3cd0 con 0x5560df1d0f40 2024-08-03T20:56:02.545 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:02.544+0000 7f0803400640 1 -- 172.21.15.32:0/3180561690 <== osd.6 v2:172.21.15.118:6825/801947386 10 ==== osd_op_reply(78 500.00000000 [write 0~90] v94'103 uv103 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f07fc003040 con 0x5560df1d0f40 2024-08-03T20:56:02.546 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:02.544+0000 7f0809904f40 1 -- 172.21.15.32:0/3180561690 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:79 12.15 12:aa448500:::500.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e94) -- 0x5560df1d3cd0 con 0x5560df1d0f40 2024-08-03T20:56:02.547 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:02.545+0000 7f0803400640 1 -- 172.21.15.32:0/3180561690 <== osd.6 v2:172.21.15.118:6825/801947386 11 ==== osd_op_reply(79 500.00000000 [omap-set-header] v94'104 uv104 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f07fc003040 con 0x5560df1d0f40 2024-08-03T20:56:02.547 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:02.545+0000 7f0809904f40 1 --2- 172.21.15.32:0/3180561690 >> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] conn(0x5560df1d41e0 0x5560df1d6680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:56:02.547 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:02.546+0000 7f0809904f40 1 -- 172.21.15.32:0/3180561690 --> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] -- osd_op(unknown.0.0:80 12.d 12:b50e409b:::mds_snaptable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e94) -- 0x5560df1d6d50 con 0x5560df1d41e0 2024-08-03T20:56:02.547 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:02.546+0000 7f0802a00640 1 --2- 172.21.15.32:0/3180561690 >> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] conn(0x5560df1d41e0 0x5560df1d6680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:56:02.548 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:02.547+0000 7f0802a00640 1 --2- 172.21.15.32:0/3180561690 >> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] conn(0x5560df1d41e0 0x5560df1d6680 crc :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:56:02.549 INFO:teuthology.orchestra.run.smithi032.stdout:Write #-9223372036854775808:00000000:::mds_snaptable:head# 2024-08-03T20:56:02.550 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:02.548+0000 7f0802a00640 1 -- 172.21.15.32:0/3180561690 <== osd.7 v2:172.21.15.118:6818/653000048 1 ==== osd_op_reply(80 mds_snaptable [create] v94'43 uv43 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f07f401b410 con 0x5560df1d41e0 2024-08-03T20:56:02.550 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:02.548+0000 7f0809904f40 1 -- 172.21.15.32:0/3180561690 --> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] -- osd_op(unknown.0.0:81 12.d 12:b50e409b:::mds_snaptable:head [write 0~70 in=70b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e94) -- 0x5560df1d7120 con 0x5560df1d41e0 2024-08-03T20:56:02.551 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:02.550+0000 7f0802a00640 1 -- 172.21.15.32:0/3180561690 <== osd.7 v2:172.21.15.118:6818/653000048 2 ==== osd_op_reply(81 mds_snaptable [write 0~70] v94'44 uv44 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f07f401b410 con 0x5560df1d41e0 2024-08-03T20:56:02.551 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:02.550+0000 7f0809904f40 1 -- 172.21.15.32:0/3180561690 --> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] -- osd_op(unknown.0.0:82 12.d 12:b50e409b:::mds_snaptable:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e94) -- 0x5560df1d7120 con 0x5560df1d41e0 2024-08-03T20:56:02.552 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:02.551+0000 7f0802a00640 1 -- 172.21.15.32:0/3180561690 <== osd.7 v2:172.21.15.118:6818/653000048 3 ==== osd_op_reply(82 mds_snaptable [omap-set-header] v94'45 uv45 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f07f401b410 con 0x5560df1d41e0 2024-08-03T20:56:02.552 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:02.551+0000 7f0809904f40 1 -- 172.21.15.32:0/3180561690 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:83 12.1d 12:b871830b:::605.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e94) -- 0x5560df1d7120 con 0x5560df092530 2024-08-03T20:56:02.553 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:02.552+0000 7f0803400640 1 -- 172.21.15.32:0/3180561690 <== osd.0 v2:172.21.15.32:6816/3066156116 44 ==== osd_op_reply(83 605.00000000 [create] v94'51 uv50 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f07fc003040 con 0x5560df092530 2024-08-03T20:56:02.554 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:02.552+0000 7f0809904f40 1 -- 172.21.15.32:0/3180561690 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:84 12.1d 12:b871830b:::605.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e94) -- 0x5560df1d7120 con 0x5560df092530 2024-08-03T20:56:02.555 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:02.554+0000 7f0803400640 1 -- 172.21.15.32:0/3180561690 <== osd.0 v2:172.21.15.32:6816/3066156116 45 ==== osd_op_reply(84 605.00000000 [delete] v94'52 uv52 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f07fc003040 con 0x5560df092530 2024-08-03T20:56:02.555 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:02.554+0000 7f0809904f40 1 -- 172.21.15.32:0/3180561690 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:85 12.1d 12:b871830b:::605.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e94) -- 0x5560df1d7120 con 0x5560df092530 2024-08-03T20:56:02.556 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:02.555+0000 7f0803400640 1 -- 172.21.15.32:0/3180561690 <== osd.0 v2:172.21.15.32:6816/3066156116 46 ==== osd_op_reply(85 605.00000000 [create] v94'53 uv53 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f07fc003040 con 0x5560df092530 2024-08-03T20:56:02.557 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::605.00000000:head# 2024-08-03T20:56:02.557 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:02.555+0000 7f0809904f40 1 -- 172.21.15.32:0/3180561690 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:86 12.1d 12:b871830b:::605.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e94) -- 0x5560df1d8130 con 0x5560df092530 2024-08-03T20:56:02.558 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:02.557+0000 7f0803400640 1 -- 172.21.15.32:0/3180561690 <== osd.0 v2:172.21.15.32:6816/3066156116 47 ==== osd_op_reply(86 605.00000000 [setxattr (30)] v94'54 uv54 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f07fc003040 con 0x5560df092530 2024-08-03T20:56:02.558 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:02.557+0000 7f0809904f40 1 -- 172.21.15.32:0/3180561690 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:87 12.1d 12:b871830b:::605.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e94) -- 0x5560df1d8130 con 0x5560df092530 2024-08-03T20:56:02.559 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:02.558+0000 7f0803400640 1 -- 172.21.15.32:0/3180561690 <== osd.0 v2:172.21.15.32:6816/3066156116 48 ==== osd_op_reply(87 605.00000000 [setxattr (62)] v94'55 uv55 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f07fc003040 con 0x5560df092530 2024-08-03T20:56:02.559 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:02.558+0000 7f0809904f40 1 -- 172.21.15.32:0/3180561690 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:88 12.1d 12:b871830b:::605.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e94) -- 0x5560df1d7120 con 0x5560df092530 2024-08-03T20:56:02.561 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:02.559+0000 7f0803400640 1 -- 172.21.15.32:0/3180561690 <== osd.0 v2:172.21.15.32:6816/3066156116 49 ==== osd_op_reply(88 605.00000000 [omap-set-header] v94'56 uv56 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f07fc003040 con 0x5560df092530 2024-08-03T20:56:02.561 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:02.559+0000 7f0809904f40 1 -- 172.21.15.32:0/3180561690 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:89 12.13 12:c91d4a1d:::mds0_inotable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e94) -- 0x5560df1d7120 con 0x5560df1ccae0 2024-08-03T20:56:02.562 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:02.561+0000 7f0802000640 1 -- 172.21.15.32:0/3180561690 <== osd.1 v2:172.21.15.32:6824/1551241759 5 ==== osd_op_reply(89 mds0_inotable [create] v94'84 uv78 ondisk = -17 ((17) File exists)) ==== 157+0+0 (crc 0 0 0) 0x7f07f00a1410 con 0x5560df1ccae0 2024-08-03T20:56:02.562 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:02.561+0000 7f0809904f40 1 -- 172.21.15.32:0/3180561690 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:90 12.13 12:c91d4a1d:::mds0_inotable:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e94) -- 0x5560df1d7530 con 0x5560df1ccae0 2024-08-03T20:56:02.564 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:02.562+0000 7f0802000640 1 -- 172.21.15.32:0/3180561690 <== osd.1 v2:172.21.15.32:6824/1551241759 6 ==== osd_op_reply(90 mds0_inotable [delete] v94'85 uv85 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f07f00a1410 con 0x5560df1ccae0 2024-08-03T20:56:02.564 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:02.562+0000 7f0809904f40 1 -- 172.21.15.32:0/3180561690 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:91 12.13 12:c91d4a1d:::mds0_inotable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e94) -- 0x5560df1d7920 con 0x5560df1ccae0 2024-08-03T20:56:02.565 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:02.563+0000 7f0802000640 1 -- 172.21.15.32:0/3180561690 <== osd.1 v2:172.21.15.32:6824/1551241759 7 ==== osd_op_reply(91 mds0_inotable [create] v94'86 uv86 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f07f00a1410 con 0x5560df1ccae0 2024-08-03T20:56:02.565 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds0_inotable:head# 2024-08-03T20:56:02.565 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:02.563+0000 7f0809904f40 1 -- 172.21.15.32:0/3180561690 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:92 12.13 12:c91d4a1d:::mds0_inotable:head [write 0~34 in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e94) -- 0x5560df1d7d50 con 0x5560df1ccae0 2024-08-03T20:56:02.566 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:02.565+0000 7f0802000640 1 -- 172.21.15.32:0/3180561690 <== osd.1 v2:172.21.15.32:6824/1551241759 8 ==== osd_op_reply(92 mds0_inotable [write 0~34] v94'87 uv87 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f07f00a1410 con 0x5560df1ccae0 2024-08-03T20:56:02.566 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:02.565+0000 7f0809904f40 1 -- 172.21.15.32:0/3180561690 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:93 12.13 12:c91d4a1d:::mds0_inotable:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e94) -- 0x5560df1d7d10 con 0x5560df1ccae0 2024-08-03T20:56:02.567 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:02.566+0000 7f0802000640 1 -- 172.21.15.32:0/3180561690 <== osd.1 v2:172.21.15.32:6824/1551241759 9 ==== osd_op_reply(93 mds0_inotable [omap-set-header] v94'88 uv88 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f07f00a1410 con 0x5560df1ccae0 2024-08-03T20:56:02.567 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:02.566+0000 7f0809904f40 1 -- 172.21.15.32:0/3180561690 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:94 12.13 12:cd5a64a3:::100.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e94) -- 0x5560df1d7cc0 con 0x5560df1ccae0 2024-08-03T20:56:02.569 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:02.567+0000 7f0802000640 1 -- 172.21.15.32:0/3180561690 <== osd.1 v2:172.21.15.32:6824/1551241759 10 ==== osd_op_reply(94 100.00000000 [create] v94'89 uv83 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f07f00a1410 con 0x5560df1ccae0 2024-08-03T20:56:02.569 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:02.567+0000 7f0809904f40 1 -- 172.21.15.32:0/3180561690 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:95 12.13 12:cd5a64a3:::100.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e94) -- 0x5560df1d7cc0 con 0x5560df1ccae0 2024-08-03T20:56:02.570 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:02.569+0000 7f0802000640 1 -- 172.21.15.32:0/3180561690 <== osd.1 v2:172.21.15.32:6824/1551241759 11 ==== osd_op_reply(95 100.00000000 [delete] v94'90 uv90 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f07f00a1410 con 0x5560df1ccae0 2024-08-03T20:56:02.570 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:02.569+0000 7f0809904f40 1 -- 172.21.15.32:0/3180561690 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:96 12.13 12:cd5a64a3:::100.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e94) -- 0x5560df1d7cc0 con 0x5560df1ccae0 2024-08-03T20:56:02.571 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::100.00000000:head# 2024-08-03T20:56:02.571 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:02.570+0000 7f0802000640 1 -- 172.21.15.32:0/3180561690 <== osd.1 v2:172.21.15.32:6824/1551241759 12 ==== osd_op_reply(96 100.00000000 [create] v94'91 uv91 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f07f00a1410 con 0x5560df1ccae0 2024-08-03T20:56:02.571 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:02.570+0000 7f0809904f40 1 -- 172.21.15.32:0/3180561690 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:97 12.13 12:cd5a64a3:::100.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e94) -- 0x5560df1d7cc0 con 0x5560df1ccae0 2024-08-03T20:56:02.572 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:02.571+0000 7f0802000640 1 -- 172.21.15.32:0/3180561690 <== osd.1 v2:172.21.15.32:6824/1551241759 13 ==== osd_op_reply(97 100.00000000 [omap-set-header] v94'92 uv92 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f07f00a1410 con 0x5560df1ccae0 2024-08-03T20:56:02.573 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:02.571+0000 7f0809904f40 1 -- 172.21.15.32:0/3180561690 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:98 12.13 12:cd5a64a3:::100.00000000:head [omap-set-vals in=5194b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e94) -- 0x5560df1daf40 con 0x5560df1ccae0 2024-08-03T20:56:02.574 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:02.573+0000 7f0802000640 1 -- 172.21.15.32:0/3180561690 <== osd.1 v2:172.21.15.32:6824/1551241759 14 ==== osd_op_reply(98 100.00000000 [omap-set-vals] v94'93 uv93 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f07f00a1410 con 0x5560df1ccae0 2024-08-03T20:56:02.574 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:02.573+0000 7f0809904f40 1 --2- 172.21.15.32:0/3180561690 >> [v2:172.21.15.32:6800/9088303,v1:172.21.15.32:6801/9088303] conn(0x5560df1d8af0 0x5560df1db3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:56:02.575 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:02.573+0000 7f0809904f40 1 -- 172.21.15.32:0/3180561690 --> [v2:172.21.15.32:6800/9088303,v1:172.21.15.32:6801/9088303] -- osd_op(unknown.0.0:99 12.b 12:d0630e4c:::mds0_sessionmap:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e94) -- 0x5560df1daf10 con 0x5560df1d8af0 2024-08-03T20:56:02.575 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:02.573+0000 7f0802000640 1 --2- 172.21.15.32:0/3180561690 >> [v2:172.21.15.32:6800/9088303,v1:172.21.15.32:6801/9088303] conn(0x5560df1d8af0 0x5560df1db3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:56:02.575 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:02.574+0000 7f0802000640 1 --2- 172.21.15.32:0/3180561690 >> [v2:172.21.15.32:6800/9088303,v1:172.21.15.32:6801/9088303] conn(0x5560df1d8af0 0x5560df1db3c0 crc :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:56:02.576 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:02.575+0000 7f0802000640 1 -- 172.21.15.32:0/3180561690 <== osd.3 v2:172.21.15.32:6800/9088303 1 ==== osd_op_reply(99 mds0_sessionmap [create] v94'40 uv39 ondisk = -17 ((17) File exists)) ==== 159+0+0 (crc 0 0 0) 0x7f07f00a1410 con 0x5560df1d8af0 2024-08-03T20:56:02.576 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:02.575+0000 7f0809904f40 1 -- 172.21.15.32:0/3180561690 --> [v2:172.21.15.32:6800/9088303,v1:172.21.15.32:6801/9088303] -- osd_op(unknown.0.0:100 12.b 12:d0630e4c:::mds0_sessionmap:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e94) -- 0x5560df1daf10 con 0x5560df1d8af0 2024-08-03T20:56:02.578 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:02.576+0000 7f0802000640 1 -- 172.21.15.32:0/3180561690 <== osd.3 v2:172.21.15.32:6800/9088303 2 ==== osd_op_reply(100 mds0_sessionmap [delete] v94'41 uv41 ondisk = 0) ==== 159+0+0 (crc 0 0 0) 0x7f07f00a1410 con 0x5560df1d8af0 2024-08-03T20:56:02.578 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:02.576+0000 7f0809904f40 1 -- 172.21.15.32:0/3180561690 --> [v2:172.21.15.32:6800/9088303,v1:172.21.15.32:6801/9088303] -- osd_op(unknown.0.0:101 12.b 12:d0630e4c:::mds0_sessionmap:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e94) -- 0x5560df1daf10 con 0x5560df1d8af0 2024-08-03T20:56:02.579 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:02.578+0000 7f0802000640 1 -- 172.21.15.32:0/3180561690 <== osd.3 v2:172.21.15.32:6800/9088303 3 ==== osd_op_reply(101 mds0_sessionmap [create] v94'42 uv42 ondisk = 0) ==== 159+0+0 (crc 0 0 0) 0x7f07f00a1410 con 0x5560df1d8af0 2024-08-03T20:56:02.579 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds0_sessionmap:head# 2024-08-03T20:56:02.579 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:02.578+0000 7f0809904f40 1 -- 172.21.15.32:0/3180561690 --> [v2:172.21.15.32:6800/9088303,v1:172.21.15.32:6801/9088303] -- osd_op(unknown.0.0:102 12.b 12:d0630e4c:::mds0_sessionmap:head [omap-set-header in=14b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e94) -- 0x5560df1daf10 con 0x5560df1d8af0 2024-08-03T20:56:02.580 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:02.579+0000 7f0802000640 1 -- 172.21.15.32:0/3180561690 <== osd.3 v2:172.21.15.32:6800/9088303 4 ==== osd_op_reply(102 mds0_sessionmap [omap-set-header] v94'43 uv43 ondisk = 0) ==== 159+0+0 (crc 0 0 0) 0x7f07f00a1410 con 0x5560df1d8af0 2024-08-03T20:56:02.580 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:02.579+0000 7f0809904f40 1 -- 172.21.15.32:0/3180561690 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:103 12.7 12:e0b41d06:::609.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e94) -- 0x5560df1daf10 con 0x5560df1ccae0 2024-08-03T20:56:02.582 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:02.580+0000 7f0802000640 1 -- 172.21.15.32:0/3180561690 <== osd.1 v2:172.21.15.32:6824/1551241759 15 ==== osd_op_reply(103 609.00000000 [create] v94'51 uv50 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f07f00a1410 con 0x5560df1ccae0 2024-08-03T20:56:02.582 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:02.580+0000 7f0809904f40 1 -- 172.21.15.32:0/3180561690 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:104 12.7 12:e0b41d06:::609.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e94) -- 0x5560df1daf10 con 0x5560df1ccae0 2024-08-03T20:56:02.583 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:02.582+0000 7f0802000640 1 -- 172.21.15.32:0/3180561690 <== osd.1 v2:172.21.15.32:6824/1551241759 16 ==== osd_op_reply(104 609.00000000 [delete] v94'52 uv52 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f07f00a1410 con 0x5560df1ccae0 2024-08-03T20:56:02.583 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:02.582+0000 7f0809904f40 1 -- 172.21.15.32:0/3180561690 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:105 12.7 12:e0b41d06:::609.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e94) -- 0x5560df1daf10 con 0x5560df1ccae0 2024-08-03T20:56:02.585 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::609.00000000:head# 2024-08-03T20:56:02.585 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:02.584+0000 7f0802000640 1 -- 172.21.15.32:0/3180561690 <== osd.1 v2:172.21.15.32:6824/1551241759 17 ==== osd_op_reply(105 609.00000000 [create] v94'53 uv53 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f07f00a1410 con 0x5560df1ccae0 2024-08-03T20:56:02.585 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:02.584+0000 7f0809904f40 1 -- 172.21.15.32:0/3180561690 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:106 12.7 12:e0b41d06:::609.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e94) -- 0x5560df1daf10 con 0x5560df1ccae0 2024-08-03T20:56:02.586 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:02.585+0000 7f0802000640 1 -- 172.21.15.32:0/3180561690 <== osd.1 v2:172.21.15.32:6824/1551241759 18 ==== osd_op_reply(106 609.00000000 [setxattr (30)] v94'54 uv54 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f07f00a1410 con 0x5560df1ccae0 2024-08-03T20:56:02.586 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:02.585+0000 7f0809904f40 1 -- 172.21.15.32:0/3180561690 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:107 12.7 12:e0b41d06:::609.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e94) -- 0x5560df1daf10 con 0x5560df1ccae0 2024-08-03T20:56:02.588 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:02.587+0000 7f0802000640 1 -- 172.21.15.32:0/3180561690 <== osd.1 v2:172.21.15.32:6824/1551241759 19 ==== osd_op_reply(107 609.00000000 [setxattr (62)] v94'55 uv55 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f07f00a1410 con 0x5560df1ccae0 2024-08-03T20:56:02.588 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:02.587+0000 7f0809904f40 1 -- 172.21.15.32:0/3180561690 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:108 12.7 12:e0b41d06:::609.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e94) -- 0x5560df1daf10 con 0x5560df1ccae0 2024-08-03T20:56:02.589 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:02.588+0000 7f0802000640 1 -- 172.21.15.32:0/3180561690 <== osd.1 v2:172.21.15.32:6824/1551241759 20 ==== osd_op_reply(108 609.00000000 [omap-set-header] v94'56 uv56 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f07f00a1410 con 0x5560df1ccae0 2024-08-03T20:56:02.590 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:02.588+0000 7f0809904f40 1 -- 172.21.15.32:0/3180561690 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:109 12.f 12:f1f69726:::400.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e94) -- 0x5560df1daf10 con 0x5560df1d0f40 2024-08-03T20:56:02.591 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:02.590+0000 7f0803400640 1 -- 172.21.15.32:0/3180561690 <== osd.6 v2:172.21.15.118:6825/801947386 12 ==== osd_op_reply(109 400.00000000 [create] v94'42 uv41 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f07fc003040 con 0x5560df1d0f40 2024-08-03T20:56:02.591 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:02.590+0000 7f0809904f40 1 -- 172.21.15.32:0/3180561690 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:110 12.f 12:f1f69726:::400.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e94) -- 0x5560df1daf10 con 0x5560df1d0f40 2024-08-03T20:56:02.593 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:02.592+0000 7f0803400640 1 -- 172.21.15.32:0/3180561690 <== osd.6 v2:172.21.15.118:6825/801947386 13 ==== osd_op_reply(110 400.00000000 [delete] v94'43 uv43 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f07fc003040 con 0x5560df1d0f40 2024-08-03T20:56:02.593 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:02.592+0000 7f0809904f40 1 -- 172.21.15.32:0/3180561690 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:111 12.f 12:f1f69726:::400.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e94) -- 0x5560df1daf10 con 0x5560df1d0f40 2024-08-03T20:56:02.595 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::400.00000000:head# 2024-08-03T20:56:02.595 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:02.593+0000 7f0803400640 1 -- 172.21.15.32:0/3180561690 <== osd.6 v2:172.21.15.118:6825/801947386 14 ==== osd_op_reply(111 400.00000000 [create] v94'44 uv44 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f07fc003040 con 0x5560df1d0f40 2024-08-03T20:56:02.595 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:02.593+0000 7f0809904f40 1 -- 172.21.15.32:0/3180561690 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:112 12.f 12:f1f69726:::400.00000000:head [write 0~22 in=22b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e94) -- 0x5560df1daf10 con 0x5560df1d0f40 2024-08-03T20:56:02.596 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:02.595+0000 7f0803400640 1 -- 172.21.15.32:0/3180561690 <== osd.6 v2:172.21.15.118:6825/801947386 15 ==== osd_op_reply(112 400.00000000 [write 0~22] v94'45 uv45 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f07fc003040 con 0x5560df1d0f40 2024-08-03T20:56:02.597 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:02.595+0000 7f0809904f40 1 -- 172.21.15.32:0/3180561690 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:113 12.f 12:f1f69726:::400.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e94) -- 0x5560df1daf10 con 0x5560df1d0f40 2024-08-03T20:56:02.598 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:02.597+0000 7f0803400640 1 -- 172.21.15.32:0/3180561690 <== osd.6 v2:172.21.15.118:6825/801947386 16 ==== osd_op_reply(113 400.00000000 [omap-set-header] v94'46 uv46 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f07fc003040 con 0x5560df1d0f40 2024-08-03T20:56:02.598 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:02.597+0000 7f0809904f40 1 -- 172.21.15.32:0/3180561690 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:114 12.1f 12:fe07bba1:::100.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e94) -- 0x5560df1daf10 con 0x5560df1d0f40 2024-08-03T20:56:02.600 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:02.598+0000 7f0803400640 1 -- 172.21.15.32:0/3180561690 <== osd.6 v2:172.21.15.118:6825/801947386 17 ==== osd_op_reply(114 100.00000000.inode [create] v94'107 uv99 ondisk = -17 ((17) File exists)) ==== 162+0+0 (crc 0 0 0) 0x7f07fc003040 con 0x5560df1d0f40 2024-08-03T20:56:02.600 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:02.598+0000 7f0809904f40 1 -- 172.21.15.32:0/3180561690 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:115 12.1f 12:fe07bba1:::100.00000000.inode:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e94) -- 0x5560df1daf10 con 0x5560df1d0f40 2024-08-03T20:56:02.602 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:02.600+0000 7f0803400640 1 -- 172.21.15.32:0/3180561690 <== osd.6 v2:172.21.15.118:6825/801947386 18 ==== osd_op_reply(115 100.00000000.inode [delete] v94'108 uv108 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7f07fc003040 con 0x5560df1d0f40 2024-08-03T20:56:02.602 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:02.600+0000 7f0809904f40 1 -- 172.21.15.32:0/3180561690 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:116 12.1f 12:fe07bba1:::100.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e94) -- 0x5560df1daf10 con 0x5560df1d0f40 2024-08-03T20:56:02.603 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:02.602+0000 7f0803400640 1 -- 172.21.15.32:0/3180561690 <== osd.6 v2:172.21.15.118:6825/801947386 19 ==== osd_op_reply(116 100.00000000.inode [create] v94'109 uv109 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7f07fc003040 con 0x5560df1d0f40 2024-08-03T20:56:02.603 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::100.00000000.inode:head# 2024-08-03T20:56:02.604 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:02.602+0000 7f0809904f40 1 -- 172.21.15.32:0/3180561690 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:117 12.1f 12:fe07bba1:::100.00000000.inode:head [write 0~582 in=582b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e94) -- 0x5560df1daf10 con 0x5560df1d0f40 2024-08-03T20:56:02.605 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:02.603+0000 7f0803400640 1 -- 172.21.15.32:0/3180561690 <== osd.6 v2:172.21.15.118:6825/801947386 20 ==== osd_op_reply(117 100.00000000.inode [write 0~582] v94'110 uv110 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7f07fc003040 con 0x5560df1d0f40 2024-08-03T20:56:02.605 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:02.603+0000 7f0809904f40 1 -- 172.21.15.32:0/3180561690 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:118 12.1f 12:fe07bba1:::100.00000000.inode:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e94) -- 0x5560df1daf10 con 0x5560df1d0f40 2024-08-03T20:56:02.606 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:02.605+0000 7f0803400640 1 -- 172.21.15.32:0/3180561690 <== osd.6 v2:172.21.15.118:6825/801947386 21 ==== osd_op_reply(118 100.00000000.inode [omap-set-header] v94'111 uv111 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7f07fc003040 con 0x5560df1d0f40 2024-08-03T20:56:02.606 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:02.605+0000 7f0809904f40 1 -- 172.21.15.32:0/3180561690 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:119 12.1f 12:ff5b34d6:::1.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e94) -- 0x5560df1daf10 con 0x5560df1d0f40 2024-08-03T20:56:02.608 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:02.606+0000 7f0803400640 1 -- 172.21.15.32:0/3180561690 <== osd.6 v2:172.21.15.118:6825/801947386 22 ==== osd_op_reply(119 1.00000000 [create] v94'112 uv106 ondisk = -17 ((17) File exists)) ==== 154+0+0 (crc 0 0 0) 0x7f07fc003040 con 0x5560df1d0f40 2024-08-03T20:56:02.608 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:02.606+0000 7f0809904f40 1 -- 172.21.15.32:0/3180561690 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:120 12.1f 12:ff5b34d6:::1.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e94) -- 0x5560df1daf10 con 0x5560df1d0f40 2024-08-03T20:56:02.609 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:02.608+0000 7f0803400640 1 -- 172.21.15.32:0/3180561690 <== osd.6 v2:172.21.15.118:6825/801947386 23 ==== osd_op_reply(120 1.00000000 [delete] v94'113 uv113 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7f07fc003040 con 0x5560df1d0f40 2024-08-03T20:56:02.609 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:02.608+0000 7f0809904f40 1 -- 172.21.15.32:0/3180561690 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:121 12.1f 12:ff5b34d6:::1.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e94) -- 0x5560df1daf10 con 0x5560df1d0f40 2024-08-03T20:56:02.611 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:02.610+0000 7f0803400640 1 -- 172.21.15.32:0/3180561690 <== osd.6 v2:172.21.15.118:6825/801947386 24 ==== osd_op_reply(121 1.00000000 [create] v94'114 uv114 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7f07fc003040 con 0x5560df1d0f40 2024-08-03T20:56:02.611 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::1.00000000:head# 2024-08-03T20:56:02.611 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:02.610+0000 7f0809904f40 1 -- 172.21.15.32:0/3180561690 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:122 12.1f 12:ff5b34d6:::1.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e94) -- 0x5560df1daf10 con 0x5560df1d0f40 2024-08-03T20:56:02.612 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:02.611+0000 7f0803400640 1 -- 172.21.15.32:0/3180561690 <== osd.6 v2:172.21.15.118:6825/801947386 25 ==== osd_op_reply(122 1.00000000 [setxattr (30)] v94'115 uv115 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7f07fc003040 con 0x5560df1d0f40 2024-08-03T20:56:02.612 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:02.611+0000 7f0809904f40 1 -- 172.21.15.32:0/3180561690 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:123 12.1f 12:ff5b34d6:::1.00000000:head [setxattr parent (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e94) -- 0x5560df1daf10 con 0x5560df1d0f40 2024-08-03T20:56:02.614 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:02.612+0000 7f0803400640 1 -- 172.21.15.32:0/3180561690 <== osd.6 v2:172.21.15.118:6825/801947386 26 ==== osd_op_reply(123 1.00000000 [setxattr (30)] v94'116 uv116 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7f07fc003040 con 0x5560df1d0f40 2024-08-03T20:56:02.614 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:02.612+0000 7f0809904f40 1 -- 172.21.15.32:0/3180561690 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:124 12.1f 12:ff5b34d6:::1.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e94) -- 0x5560df1daf10 con 0x5560df1d0f40 2024-08-03T20:56:02.615 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:02.614+0000 7f0803400640 1 -- 172.21.15.32:0/3180561690 <== osd.6 v2:172.21.15.118:6825/801947386 27 ==== osd_op_reply(124 1.00000000 [omap-set-header] v94'117 uv117 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7f07fc003040 con 0x5560df1d0f40 2024-08-03T20:56:02.615 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:02.614+0000 7f0809904f40 1 -- 172.21.15.32:0/3180561690 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:125 12.1f 12:ff5b34d6:::1.00000000:head [omap-set-vals in=584b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e94) -- 0x5560df1dbe10 con 0x5560df1d0f40 2024-08-03T20:56:02.617 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:02.615+0000 7f0803400640 1 -- 172.21.15.32:0/3180561690 <== osd.6 v2:172.21.15.118:6825/801947386 28 ==== osd_op_reply(125 1.00000000 [omap-set-vals] v94'118 uv118 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7f07fc003040 con 0x5560df1d0f40 2024-08-03T20:56:02.617 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:02.615+0000 7f0809904f40 1 -- 172.21.15.32:0/3180561690 >> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] conn(0x5560df092530 msgr2=0x5560df092940 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:56:02.618 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:02.615+0000 7f0809904f40 1 --2- 172.21.15.32:0/3180561690 >> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] conn(0x5560df092530 0x5560df092940 crc :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:02.618 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:02.615+0000 7f0809904f40 1 -- 172.21.15.32:0/3180561690 >> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] conn(0x5560df1ccae0 msgr2=0x5560df1cef00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:56:02.618 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:02.615+0000 7f0809904f40 1 --2- 172.21.15.32:0/3180561690 >> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] conn(0x5560df1ccae0 0x5560df1cef00 crc :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:02.618 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:02.615+0000 7f0809904f40 1 -- 172.21.15.32:0/3180561690 >> [v2:172.21.15.32:6800/9088303,v1:172.21.15.32:6801/9088303] conn(0x5560df1d8af0 msgr2=0x5560df1db3c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:56:02.618 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:02.615+0000 7f0809904f40 1 --2- 172.21.15.32:0/3180561690 >> [v2:172.21.15.32:6800/9088303,v1:172.21.15.32:6801/9088303] conn(0x5560df1d8af0 0x5560df1db3c0 crc :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:02.618 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:02.615+0000 7f0809904f40 1 -- 172.21.15.32:0/3180561690 >> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] conn(0x5560df1c86b0 msgr2=0x5560df1caad0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:56:02.618 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:02.615+0000 7f0809904f40 1 --2- 172.21.15.32:0/3180561690 >> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] conn(0x5560df1c86b0 0x5560df1caad0 crc :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:02.618 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:02.615+0000 7f0809904f40 1 -- 172.21.15.32:0/3180561690 >> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] conn(0x5560df1d0f40 msgr2=0x5560df1d3360 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:56:02.618 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:02.615+0000 7f0809904f40 1 --2- 172.21.15.32:0/3180561690 >> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] conn(0x5560df1d0f40 0x5560df1d3360 crc :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:02.618 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:02.615+0000 7f0809904f40 1 -- 172.21.15.32:0/3180561690 >> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] conn(0x5560df1d41e0 msgr2=0x5560df1d6680 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:56:02.618 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:02.615+0000 7f0809904f40 1 --2- 172.21.15.32:0/3180561690 >> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] conn(0x5560df1d41e0 0x5560df1d6680 crc :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:02.618 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:02.615+0000 7f0803400640 1 -- 172.21.15.32:0/3180561690 reap_dead start 2024-08-03T20:56:02.619 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:02.615+0000 7f0803400640 1 -- 172.21.15.32:0/3180561690 reap_dead start 2024-08-03T20:56:02.619 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:02.615+0000 7f0803400640 1 -- 172.21.15.32:0/3180561690 reap_dead start 2024-08-03T20:56:02.619 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:02.616+0000 7f0809904f40 1 -- 172.21.15.32:0/3180561690 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f07e40759f0 msgr2=0x7f07e4077f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:56:02.619 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:02.616+0000 7f0809904f40 1 --2- 172.21.15.32:0/3180561690 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f07e40759f0 0x7f07e4077f00 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7f07f4000fb0 tx=0x7f07f4003040 comp rx=0 tx=0).stop 2024-08-03T20:56:02.619 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:02.616+0000 7f0809904f40 1 -- 172.21.15.32:0/3180561690 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5560df11d5f0 msgr2=0x5560df11be40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:56:02.619 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:02.616+0000 7f0809904f40 1 --2- 172.21.15.32:0/3180561690 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5560df11d5f0 0x5560df11be40 secure :-1 s=READY pgs=1287 cs=0 l=1 rev1=1 crypto rx=0x7f07f000e090 tx=0x7f07f0056a80 comp rx=0 tx=0).stop 2024-08-03T20:56:02.619 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:02.616+0000 7f0809904f40 1 -- 172.21.15.32:0/3180561690 shutdown_connections 2024-08-03T20:56:02.619 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:02.616+0000 7f0809904f40 1 --2- 172.21.15.32:0/3180561690 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f07e40759f0 0x7f07e4077f00 unknown :-1 s=CLOSED pgs=531 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:02.619 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:02.616+0000 7f0809904f40 1 --2- 172.21.15.32:0/3180561690 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5560df11d5f0 0x5560df11be40 unknown :-1 s=CLOSED pgs=1287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:02.619 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:02.616+0000 7f0809904f40 1 -- 172.21.15.32:0/3180561690 >> 172.21.15.32:0/3180561690 conn(0x5560def6cac0 msgr2=0x5560df11cd30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:56:02.619 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:02.616+0000 7f0809904f40 1 -- 172.21.15.32:0/3180561690 shutdown_connections 2024-08-03T20:56:02.619 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:02.616+0000 7f0809904f40 1 -- 172.21.15.32:0/3180561690 wait complete. 2024-08-03T20:56:02.629 DEBUG:teuthology.orchestra.run.smithi032:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata rm mds0_inotable 2024-08-03T20:56:02.811 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:02.810+0000 7fa8c6a19f40 1 Processor -- start 2024-08-03T20:56:02.812 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:02.810+0000 7fa8c6a19f40 1 -- start start 2024-08-03T20:56:02.814 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:02.811+0000 7fa8c6a19f40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56161a536360 0x56161a5e8310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:56:02.814 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:02.811+0000 7fa8c6a19f40 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56161a537ea0 0x56161a5e8850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:56:02.814 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:02.811+0000 7fa8c6a19f40 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x56161a553cb0 0x56161a5e8d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:56:02.814 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:02.811+0000 7fa8c6a19f40 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x56161a5540c0 con 0x56161a537ea0 2024-08-03T20:56:02.814 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:02.811+0000 7fa8c6a19f40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x56161a3cde30 con 0x56161a536360 2024-08-03T20:56:02.814 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:02.811+0000 7fa8c6a19f40 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x56161a54ee40 con 0x56161a553cb0 2024-08-03T20:56:02.815 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:02.812+0000 7fa8bf200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56161a537ea0 0x56161a5e8850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:56:02.815 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:02.812+0000 7fa8bf200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56161a537ea0 0x56161a5e8850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:42576/0 (socket says 172.21.15.32:42576) 2024-08-03T20:56:02.815 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:02.812+0000 7fa8bf200640 1 -- 172.21.15.32:0/4229282126 learned_addr learned my addr 172.21.15.32:0/4229282126 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:56:02.815 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:02.812+0000 7fa8c0600640 1 --2- 172.21.15.32:0/4229282126 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x56161a553cb0 0x56161a5e8d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:56:02.815 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:02.812+0000 7fa8bfc00640 1 --2- 172.21.15.32:0/4229282126 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56161a536360 0x56161a5e8310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:56:02.815 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:02.812+0000 7fa8bf200640 1 -- 172.21.15.32:0/4229282126 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x56161a553cb0 msgr2=0x56161a5e8d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:56:02.815 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:02.812+0000 7fa8bf200640 1 --2- 172.21.15.32:0/4229282126 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x56161a553cb0 0x56161a5e8d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:02.815 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:02.812+0000 7fa8bf200640 1 -- 172.21.15.32:0/4229282126 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56161a536360 msgr2=0x56161a5e8310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:56:02.815 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:02.812+0000 7fa8bf200640 1 --2- 172.21.15.32:0/4229282126 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56161a536360 0x56161a5e8310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:02.815 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:02.812+0000 7fa8bf200640 1 -- 172.21.15.32:0/4229282126 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56161a3a6f10 con 0x56161a537ea0 2024-08-03T20:56:02.815 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:02.812+0000 7fa8bf200640 1 --2- 172.21.15.32:0/4229282126 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56161a537ea0 0x56161a5e8850 secure :-1 s=READY pgs=1289 cs=0 l=1 rev1=1 crypto rx=0x7fa8ac025e00 tx=0x7fa8ac002fb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=190d446007d33f19 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:56:02.815 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:02.813+0000 7fa8bde00640 1 -- 172.21.15.32:0/4229282126 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8ac04daf0 con 0x56161a537ea0 2024-08-03T20:56:02.815 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:02.813+0000 7fa8bde00640 1 -- 172.21.15.32:0/4229282126 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fa8ac04dc90 con 0x56161a537ea0 2024-08-03T20:56:02.816 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:02.813+0000 7fa8bde00640 1 -- 172.21.15.32:0/4229282126 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8ac0566d0 con 0x56161a537ea0 2024-08-03T20:56:02.816 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:02.813+0000 7fa8c6a19f40 1 -- 172.21.15.32:0/4229282126 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56161a537ea0 msgr2=0x56161a5e8850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:56:02.817 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:02.813+0000 7fa8c6a19f40 1 --2- 172.21.15.32:0/4229282126 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56161a537ea0 0x56161a5e8850 secure :-1 s=READY pgs=1289 cs=0 l=1 rev1=1 crypto rx=0x7fa8ac025e00 tx=0x7fa8ac002fb0 comp rx=0 tx=0).stop 2024-08-03T20:56:02.817 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:02.813+0000 7fa8c6a19f40 1 -- 172.21.15.32:0/4229282126 shutdown_connections 2024-08-03T20:56:02.817 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:02.813+0000 7fa8c6a19f40 1 --2- 172.21.15.32:0/4229282126 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x56161a553cb0 0x56161a5e8d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:02.817 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:02.813+0000 7fa8c6a19f40 1 --2- 172.21.15.32:0/4229282126 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56161a537ea0 0x56161a5e8850 unknown :-1 s=CLOSED pgs=1289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:02.817 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:02.813+0000 7fa8c6a19f40 1 --2- 172.21.15.32:0/4229282126 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56161a536360 0x56161a5e8310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:02.817 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:02.813+0000 7fa8c6a19f40 1 -- 172.21.15.32:0/4229282126 >> 172.21.15.32:0/4229282126 conn(0x56161a39fac0 msgr2=0x56161a551810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:56:02.817 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:02.813+0000 7fa8c6a19f40 1 -- 172.21.15.32:0/4229282126 shutdown_connections 2024-08-03T20:56:02.817 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:02.813+0000 7fa8c6a19f40 1 -- 172.21.15.32:0/4229282126 wait complete. 2024-08-03T20:56:02.817 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:02.814+0000 7fa8c6a19f40 1 Processor -- start 2024-08-03T20:56:02.818 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:02.815+0000 7fa8c6a19f40 1 -- start start 2024-08-03T20:56:02.820 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:02.816+0000 7fa8c6a19f40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56161a536360 0x56161a4c29d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:56:02.820 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:02.816+0000 7fa8c6a19f40 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56161a537ea0 0x56161a5e82e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:56:02.820 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:02.816+0000 7fa8c6a19f40 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x56161a553cb0 0x56161a5e75d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:56:02.820 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:02.816+0000 7fa8c6a19f40 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x56161a52eec0 con 0x56161a537ea0 2024-08-03T20:56:02.820 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:02.816+0000 7fa8c6a19f40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x56161a5378d0 con 0x56161a536360 2024-08-03T20:56:02.820 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:02.816+0000 7fa8c6a19f40 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x56161a537b70 con 0x56161a553cb0 2024-08-03T20:56:02.820 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:02.816+0000 7fa8bf200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56161a537ea0 0x56161a5e82e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:56:02.820 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:02.816+0000 7fa8bf200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56161a537ea0 0x56161a5e82e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:42578/0 (socket says 172.21.15.32:42578) 2024-08-03T20:56:02.820 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:02.816+0000 7fa8bf200640 1 -- 172.21.15.32:0/4255045544 learned_addr learned my addr 172.21.15.32:0/4255045544 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:56:02.821 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:02.816+0000 7fa8bfc00640 1 --2- 172.21.15.32:0/4255045544 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56161a536360 0x56161a4c29d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:56:02.821 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:02.816+0000 7fa8c0600640 1 --2- 172.21.15.32:0/4255045544 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x56161a553cb0 0x56161a5e75d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:56:02.821 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:02.817+0000 7fa8bf200640 1 -- 172.21.15.32:0/4255045544 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x56161a553cb0 msgr2=0x56161a5e75d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:56:02.821 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:02.817+0000 7fa8bf200640 1 --2- 172.21.15.32:0/4255045544 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x56161a553cb0 0x56161a5e75d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:02.821 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:02.817+0000 7fa8bf200640 1 -- 172.21.15.32:0/4255045544 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56161a536360 msgr2=0x56161a4c29d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:56:02.821 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:02.817+0000 7fa8bf200640 1 --2- 172.21.15.32:0/4255045544 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56161a536360 0x56161a4c29d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:02.821 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:02.817+0000 7fa8bf200640 1 -- 172.21.15.32:0/4255045544 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56161a54dfe0 con 0x56161a537ea0 2024-08-03T20:56:02.821 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:02.817+0000 7fa8bf200640 1 --2- 172.21.15.32:0/4255045544 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56161a537ea0 0x56161a5e82e0 secure :-1 s=READY pgs=1290 cs=0 l=1 rev1=1 crypto rx=0x7fa8ac004f50 tx=0x7fa8ac04d940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:56:02.821 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:02.817+0000 7fa8bca00640 1 -- 172.21.15.32:0/4255045544 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8ac056530 con 0x56161a537ea0 2024-08-03T20:56:02.821 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:02.817+0000 7fa8bca00640 1 -- 172.21.15.32:0/4255045544 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fa8ac0567c0 con 0x56161a537ea0 2024-08-03T20:56:02.821 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:02.817+0000 7fa8c6a19f40 1 -- 172.21.15.32:0/4255045544 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x56161a52db60 con 0x56161a537ea0 2024-08-03T20:56:02.822 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:02.817+0000 7fa8bca00640 1 -- 172.21.15.32:0/4255045544 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8ac056bd0 con 0x56161a537ea0 2024-08-03T20:56:02.822 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:02.818+0000 7fa8c6a19f40 1 -- 172.21.15.32:0/4255045544 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56161a4c2fa0 con 0x56161a537ea0 2024-08-03T20:56:02.823 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:02.820+0000 7fa8bca00640 1 -- 172.21.15.32:0/4255045544 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7fa8ac06a070 con 0x56161a537ea0 2024-08-03T20:56:02.823 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:02.820+0000 7fa8bca00640 1 --2- 172.21.15.32:0/4255045544 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fa8a0075a20 0x7fa8a0077f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:56:02.823 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:02.821+0000 7fa8bca00640 1 -- 172.21.15.32:0/4255045544 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 6628+0+0 (secure 0 0 0) 0x7fa8ac0dca40 con 0x56161a537ea0 2024-08-03T20:56:02.823 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:02.821+0000 7fa8c6a19f40 1 --2- 172.21.15.32:0/4255045544 >> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] conn(0x56161a5f08a0 0x56161a5f0cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:56:02.823 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:02.821+0000 7fa8c6a19f40 1 -- 172.21.15.32:0/4255045544 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:1 12.13 12:c91d4a1d:::mds0_inotable:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e94) -- 0x56161a52e4f0 con 0x56161a5f08a0 2024-08-03T20:56:02.823 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:02.821+0000 7fa8bfc00640 1 --2- 172.21.15.32:0/4255045544 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fa8a0075a20 0x7fa8a0077f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:56:02.823 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:02.821+0000 7fa8c0600640 1 --2- 172.21.15.32:0/4255045544 >> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] conn(0x56161a5f08a0 0x56161a5f0cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:56:02.823 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:02.822+0000 7fa8bfc00640 1 --2- 172.21.15.32:0/4255045544 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fa8a0075a20 0x7fa8a0077f30 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x56161a556780 tx=0x7fa8b0004880 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:56:02.824 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:02.822+0000 7fa8c0600640 1 --2- 172.21.15.32:0/4255045544 >> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] conn(0x56161a5f08a0 0x56161a5f0cb0 crc :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:56:02.825 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:02.823+0000 7fa8c0600640 1 -- 172.21.15.32:0/4255045544 <== osd.1 v2:172.21.15.32:6824/1551241759 1 ==== osd_op_reply(1 mds0_inotable [delete] v94'94 uv94 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7fa8b800b040 con 0x56161a5f08a0 2024-08-03T20:56:02.825 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:02.823+0000 7fa8c6a19f40 1 -- 172.21.15.32:0/4255045544 >> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] conn(0x56161a5f08a0 msgr2=0x56161a5f0cb0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:56:02.826 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:02.823+0000 7fa8c6a19f40 1 --2- 172.21.15.32:0/4255045544 >> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] conn(0x56161a5f08a0 0x56161a5f0cb0 crc :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:02.826 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:02.824+0000 7fa8c6a19f40 1 -- 172.21.15.32:0/4255045544 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fa8a0075a20 msgr2=0x7fa8a0077f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:56:02.826 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:02.824+0000 7fa8c6a19f40 1 --2- 172.21.15.32:0/4255045544 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fa8a0075a20 0x7fa8a0077f30 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x56161a556780 tx=0x7fa8b0004880 comp rx=0 tx=0).stop 2024-08-03T20:56:02.826 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:02.824+0000 7fa8c6a19f40 1 -- 172.21.15.32:0/4255045544 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56161a537ea0 msgr2=0x56161a5e82e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:56:02.826 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:02.824+0000 7fa8c6a19f40 1 --2- 172.21.15.32:0/4255045544 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56161a537ea0 0x56161a5e82e0 secure :-1 s=READY pgs=1290 cs=0 l=1 rev1=1 crypto rx=0x7fa8ac004f50 tx=0x7fa8ac04d940 comp rx=0 tx=0).stop 2024-08-03T20:56:02.826 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:02.824+0000 7fa8c0600640 1 -- 172.21.15.32:0/4255045544 reap_dead start 2024-08-03T20:56:02.826 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:02.824+0000 7fa8c6a19f40 1 -- 172.21.15.32:0/4255045544 shutdown_connections 2024-08-03T20:56:02.826 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:02.824+0000 7fa8c6a19f40 1 -- 172.21.15.32:0/4255045544 >> 172.21.15.32:0/4255045544 conn(0x56161a39fac0 msgr2=0x56161a54e3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:56:02.826 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:02.824+0000 7fa8c6a19f40 1 -- 172.21.15.32:0/4255045544 shutdown_connections 2024-08-03T20:56:02.826 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:02.824+0000 7fa8c6a19f40 1 -- 172.21.15.32:0/4255045544 wait complete. 2024-08-03T20:56:02.838 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs joinable true 2024-08-03T20:56:02.965 INFO:tasks.ceph.mds.a.smithi032.stderr: -52> 2024-08-03T20:55:58.210+0000 7f1945a00640 -1 mds.0.snaptable: load_2 could not read table: -2 2024-08-03T20:56:02.965 INFO:tasks.ceph.mds.a.smithi032.stderr: -51> 2024-08-03T20:55:58.210+0000 7f1945a00640 -1 log_channel(cluster) log [ERR] : error reading table object 'mds_snaptable' -2 ((2) No such file or directory) 2024-08-03T20:56:03.073 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:03.071+0000 7f6966200640 1 -- 172.21.15.32:0/4170042691 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f69440039e0 msgr2=0x7f6944003e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:56:03.073 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:03.071+0000 7f6966200640 1 --2- 172.21.15.32:0/4170042691 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f69440039e0 0x7f6944003e10 secure :-1 s=READY pgs=1292 cs=0 l=1 rev1=1 crypto rx=0x7f695000c4a0 tx=0x7f6944005830 comp rx=0 tx=0).stop 2024-08-03T20:56:03.073 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:03.071+0000 7f695fe00640 1 -- 172.21.15.32:0/4170042691 reap_dead start 2024-08-03T20:56:03.073 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:03.071+0000 7f6966200640 1 -- 172.21.15.32:0/4170042691 shutdown_connections 2024-08-03T20:56:03.073 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:03.071+0000 7f6966200640 1 -- 172.21.15.32:0/4170042691 >> 172.21.15.32:0/4170042691 conn(0x7f69600b1bd0 msgr2=0x7f69600b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:56:03.073 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:03.071+0000 7f6966200640 1 -- 172.21.15.32:0/4170042691 shutdown_connections 2024-08-03T20:56:03.073 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:03.071+0000 7f6966200640 1 -- 172.21.15.32:0/4170042691 wait complete. 2024-08-03T20:56:03.075 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:03.073+0000 7f6966200640 1 Processor -- start 2024-08-03T20:56:03.075 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:03.074+0000 7f6966200640 1 -- start start 2024-08-03T20:56:03.077 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:03.076+0000 7f6966200640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f69440039e0 0x7f69600c9660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:56:03.078 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:03.076+0000 7f6966200640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f696013f550 0x7f69600cb8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:56:03.078 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:03.076+0000 7f6966200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f696013faa0 0x7f69600cbde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:56:03.078 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:03.076+0000 7f6966200640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f696015ffe0 con 0x7f696013faa0 2024-08-03T20:56:03.078 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:03.076+0000 7f6966200640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f69600700d0 con 0x7f696013f550 2024-08-03T20:56:03.078 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:03.076+0000 7f6966200640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7f6960160160 con 0x7f69440039e0 2024-08-03T20:56:03.078 INFO:tasks.ceph.mds.a.smithi032.stdout:starting mds.a at 2024-08-03T20:56:03.080 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:03.076+0000 7f695fe00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f696013faa0 0x7f69600cbde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:56:03.080 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:03.076+0000 7f695fe00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f696013faa0 0x7f69600cbde0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:42592/0 (socket says 172.21.15.32:42592) 2024-08-03T20:56:03.080 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:03.076+0000 7f695fe00640 1 -- 172.21.15.32:0/3888794667 learned_addr learned my addr 172.21.15.32:0/3888794667 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:56:03.080 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:03.077+0000 7f6957e00640 1 --2- 172.21.15.32:0/3888794667 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f696013f550 0x7f69600cb8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:56:03.080 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:03.077+0000 7f695f400640 1 --2- 172.21.15.32:0/3888794667 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f69440039e0 0x7f69600c9660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:56:03.080 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:03.077+0000 7f695fe00640 1 -- 172.21.15.32:0/3888794667 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f69440039e0 msgr2=0x7f69600c9660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:56:03.080 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:03.077+0000 7f695fe00640 1 --2- 172.21.15.32:0/3888794667 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f69440039e0 0x7f69600c9660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:03.080 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:03.077+0000 7f695fe00640 1 -- 172.21.15.32:0/3888794667 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f696013f550 msgr2=0x7f69600cb8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:56:03.080 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:03.077+0000 7f695fe00640 1 --2- 172.21.15.32:0/3888794667 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f696013f550 0x7f69600cb8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:03.080 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:03.077+0000 7f695fe00640 1 -- 172.21.15.32:0/3888794667 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f69600c9fd0 con 0x7f696013faa0 2024-08-03T20:56:03.080 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:03.077+0000 7f6957e00640 1 --2- 172.21.15.32:0/3888794667 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f696013f550 0x7f69600cb8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-03T20:56:03.080 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:03.077+0000 7f695fe00640 1 --2- 172.21.15.32:0/3888794667 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f696013faa0 0x7f69600cbde0 secure :-1 s=READY pgs=1294 cs=0 l=1 rev1=1 crypto rx=0x7f6944004780 tx=0x7f6958006430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:56:03.081 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:03.077+0000 7f6957400640 1 -- 172.21.15.32:0/3888794667 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6958015070 con 0x7f696013faa0 2024-08-03T20:56:03.081 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:03.077+0000 7f6957400640 1 -- 172.21.15.32:0/3888794667 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f6958011900 con 0x7f696013faa0 2024-08-03T20:56:03.081 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:03.077+0000 7f6957400640 1 -- 172.21.15.32:0/3888794667 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6958011d50 con 0x7f696013faa0 2024-08-03T20:56:03.081 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:03.077+0000 7f6966200640 1 -- 172.21.15.32:0/3888794667 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6960072900 con 0x7f696013faa0 2024-08-03T20:56:03.081 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:03.078+0000 7f6966200640 1 -- 172.21.15.32:0/3888794667 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6960072ec0 con 0x7f696013faa0 2024-08-03T20:56:03.081 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:03.079+0000 7f6966200640 1 -- 172.21.15.32:0/3888794667 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f69600cd210 con 0x7f696013faa0 2024-08-03T20:56:03.081 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:03.080+0000 7f6957400640 1 -- 172.21.15.32:0/3888794667 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f6958018070 con 0x7f696013faa0 2024-08-03T20:56:03.083 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:03.081+0000 7f6957400640 1 --2- 172.21.15.32:0/3888794667 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f693c077010 0x7f693c079520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:56:03.083 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:03.081+0000 7f6957400640 1 -- 172.21.15.32:0/3888794667 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 6628+0+0 (secure 0 0 0) 0x7f6958099910 con 0x7f696013faa0 2024-08-03T20:56:03.083 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:03.081+0000 7f695f400640 1 --2- 172.21.15.32:0/3888794667 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f693c077010 0x7f693c079520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:56:03.084 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:03.082+0000 7f695f400640 1 --2- 172.21.15.32:0/3888794667 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f693c077010 0x7f693c079520 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7f6950033000 tx=0x7f695002d070 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:56:03.087 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:03.086+0000 7f6957400640 1 -- 172.21.15.32:0/3888794667 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7f69580627f0 con 0x7f696013faa0 2024-08-03T20:56:03.333 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:03.332+0000 7f6966200640 1 -- 172.21.15.32:0/3888794667 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "joinable", "val": "true"} v 0) -- 0x7f69600cffb0 con 0x7f696013faa0 2024-08-03T20:56:04.244 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:04.242+0000 7f6957400640 1 -- 172.21.15.32:0/3888794667 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "joinable", "val": "true"}]=0 cephfs marked joinable; MDS may join as newly active. v146) ==== 162+0+0 (secure 0 0 0) 0x7f6958097e20 con 0x7f696013faa0 2024-08-03T20:56:04.244 INFO:teuthology.orchestra.run.smithi032.stderr:cephfs marked joinable; MDS may join as newly active. 2024-08-03T20:56:04.247 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:04.245+0000 7f6966200640 1 -- 172.21.15.32:0/3888794667 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f693c077010 msgr2=0x7f693c079520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:56:04.247 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:04.245+0000 7f6966200640 1 --2- 172.21.15.32:0/3888794667 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f693c077010 0x7f693c079520 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7f6950033000 tx=0x7f695002d070 comp rx=0 tx=0).stop 2024-08-03T20:56:04.247 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:04.245+0000 7f6966200640 1 -- 172.21.15.32:0/3888794667 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f696013faa0 msgr2=0x7f69600cbde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:56:04.248 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:04.245+0000 7f6966200640 1 --2- 172.21.15.32:0/3888794667 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f696013faa0 0x7f69600cbde0 secure :-1 s=READY pgs=1294 cs=0 l=1 rev1=1 crypto rx=0x7f6944004780 tx=0x7f6958006430 comp rx=0 tx=0).stop 2024-08-03T20:56:04.248 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:04.246+0000 7f6966200640 1 -- 172.21.15.32:0/3888794667 shutdown_connections 2024-08-03T20:56:04.248 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:04.246+0000 7f6966200640 1 --2- 172.21.15.32:0/3888794667 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f693c077010 0x7f693c079520 unknown :-1 s=CLOSED pgs=533 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:04.248 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:04.246+0000 7f6966200640 1 --2- 172.21.15.32:0/3888794667 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f696013f550 0x7f69600cb8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:04.248 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:04.246+0000 7f6966200640 1 --2- 172.21.15.32:0/3888794667 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f696013faa0 0x7f69600cbde0 unknown :-1 s=CLOSED pgs=1294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:04.248 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:04.246+0000 7f6966200640 1 --2- 172.21.15.32:0/3888794667 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f69440039e0 0x7f69600c9660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:04.248 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:04.246+0000 7f6966200640 1 -- 172.21.15.32:0/3888794667 >> 172.21.15.32:0/3888794667 conn(0x7f69600b1bd0 msgr2=0x7f69600d04d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:56:04.248 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:04.246+0000 7f6966200640 1 -- 172.21.15.32:0/3888794667 shutdown_connections 2024-08-03T20:56:04.248 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:04.246+0000 7f6966200640 1 -- 172.21.15.32:0/3888794667 wait complete. 2024-08-03T20:56:04.255 INFO:tasks.ceph.mds.a.smithi032.stderr:2024-08-03T20:56:04.253+0000 7f53fa200640 -1 mds.0.inotable: load_2 could not read table: -2 2024-08-03T20:56:04.255 INFO:tasks.ceph.mds.a.smithi032.stderr:2024-08-03T20:56:04.254+0000 7f53fa200640 -1 log_channel(cluster) log [ERR] : error reading table object 'mds0_inotable' -2 ((2) No such file or directory) 2024-08-03T20:56:04.256 INFO:tasks.ceph.mon.a.smithi032.stderr:2024-08-03T20:56:04.255+0000 7f4e6d600640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-08-03T20:56:04.256 INFO:tasks.ceph.mon.a.smithi032.stderr:2024-08-03T20:56:04.255+0000 7f4e6d600640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 mds daemon damaged (MDS_DAMAGE) 2024-08-03T20:56:04.277 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-08-03T20:56:04.509 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:04.507+0000 7f8c0be00640 1 -- 172.21.15.32:0/216842126 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8c0406f720 msgr2=0x7f8c0406fb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:56:04.509 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:04.507+0000 7f8c0be00640 1 --2- 172.21.15.32:0/216842126 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8c0406f720 0x7f8c0406fb30 secure :-1 s=READY pgs=1295 cs=0 l=1 rev1=1 crypto rx=0x7f8bf8005fd0 tx=0x7f8bf802a5d0 comp rx=0 tx=0).stop 2024-08-03T20:56:04.509 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:04.507+0000 7f8c0be00640 1 -- 172.21.15.32:0/216842126 shutdown_connections 2024-08-03T20:56:04.509 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:04.507+0000 7f8c0be00640 1 --2- 172.21.15.32:0/216842126 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f8c040b39a0 0x7f8c040b3db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:04.509 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:04.507+0000 7f8c0be00640 1 --2- 172.21.15.32:0/216842126 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8c0406f720 0x7f8c0406fb30 unknown :-1 s=CLOSED pgs=1295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:04.509 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:04.507+0000 7f8c0be00640 1 -- 172.21.15.32:0/216842126 >> 172.21.15.32:0/216842126 conn(0x7f8c040b1bd0 msgr2=0x7f8c040b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:56:04.510 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:04.507+0000 7f8c0be00640 1 -- 172.21.15.32:0/216842126 shutdown_connections 2024-08-03T20:56:04.510 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:04.507+0000 7f8c0be00640 1 -- 172.21.15.32:0/216842126 wait complete. 2024-08-03T20:56:04.511 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:04.509+0000 7f8c0be00640 1 Processor -- start 2024-08-03T20:56:04.511 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:04.510+0000 7f8c0be00640 1 -- start start 2024-08-03T20:56:04.513 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:04.511+0000 7f8c0be00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8c0406f720 0x7f8c040cf600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:56:04.513 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:04.511+0000 7f8c0be00640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f8c040b39a0 0x7f8c0415fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:56:04.513 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:04.511+0000 7f8c0be00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8c04162210 0x7f8c041601e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:56:04.513 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:04.511+0000 7f8c0be00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f8c04162d30 con 0x7f8c04162210 2024-08-03T20:56:04.513 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:04.511+0000 7f8c0be00640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f8c040af0a0 con 0x7f8c0406f720 2024-08-03T20:56:04.513 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:04.511+0000 7f8c0be00640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7f8c04162eb0 con 0x7f8c040b39a0 2024-08-03T20:56:04.513 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:04.511+0000 7f8c09e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8c04162210 0x7f8c041601e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:56:04.513 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:04.511+0000 7f8c09e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8c04162210 0x7f8c041601e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:42630/0 (socket says 172.21.15.32:42630) 2024-08-03T20:56:04.513 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:04.511+0000 7f8c08a00640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f8c040b39a0 0x7f8c0415fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:56:04.514 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:04.511+0000 7f8c09400640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8c0406f720 0x7f8c040cf600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:56:04.514 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:04.511+0000 7f8c09e00640 1 -- 172.21.15.32:0/4183414241 learned_addr learned my addr 172.21.15.32:0/4183414241 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:56:04.519 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:04.512+0000 7f8c09e00640 1 -- 172.21.15.32:0/4183414241 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f8c040b39a0 msgr2=0x7f8c0415fca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:56:04.519 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:04.512+0000 7f8c09e00640 1 --2- 172.21.15.32:0/4183414241 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f8c040b39a0 0x7f8c0415fca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:04.519 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:04.512+0000 7f8c09e00640 1 -- 172.21.15.32:0/4183414241 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8c0406f720 msgr2=0x7f8c040cf600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:56:04.519 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:04.512+0000 7f8c09e00640 1 --2- 172.21.15.32:0/4183414241 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8c0406f720 0x7f8c040cf600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:04.519 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:04.512+0000 7f8c09e00640 1 -- 172.21.15.32:0/4183414241 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8c04158070 con 0x7f8c04162210 2024-08-03T20:56:04.519 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:04.512+0000 7f8c09e00640 1 --2- 172.21.15.32:0/4183414241 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8c04162210 0x7f8c041601e0 secure :-1 s=READY pgs=1296 cs=0 l=1 rev1=1 crypto rx=0x7f8bfc00c700 tx=0x7f8bfc00cbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:56:04.519 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:04.513+0000 7f8c01600640 1 -- 172.21.15.32:0/4183414241 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8bfc003b70 con 0x7f8c04162210 2024-08-03T20:56:04.519 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:04.513+0000 7f8c01600640 1 -- 172.21.15.32:0/4183414241 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f8bfc003d10 con 0x7f8c04162210 2024-08-03T20:56:04.520 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:04.513+0000 7f8c01600640 1 -- 172.21.15.32:0/4183414241 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8bfc00aab0 con 0x7f8c04162210 2024-08-03T20:56:04.520 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:04.513+0000 7f8c0be00640 1 -- 172.21.15.32:0/4183414241 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8c04158330 con 0x7f8c04162210 2024-08-03T20:56:04.520 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:04.513+0000 7f8c0be00640 1 -- 172.21.15.32:0/4183414241 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8c041588f0 con 0x7f8c04162210 2024-08-03T20:56:04.520 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:04.514+0000 7f8c0be00640 1 -- 172.21.15.32:0/4183414241 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8c04155f90 con 0x7f8c04162210 2024-08-03T20:56:04.520 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:04.515+0000 7f8c01600640 1 -- 172.21.15.32:0/4183414241 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f8bfc00ac50 con 0x7f8c04162210 2024-08-03T20:56:04.520 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:04.516+0000 7f8c01600640 1 --2- 172.21.15.32:0/4183414241 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f8be4077040 0x7f8be4079550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:56:04.520 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:04.516+0000 7f8c01600640 1 -- 172.21.15.32:0/4183414241 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 6714+0+0 (secure 0 0 0) 0x7f8bfc098a80 con 0x7f8c04162210 2024-08-03T20:56:04.520 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:04.516+0000 7f8c09400640 1 --2- 172.21.15.32:0/4183414241 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f8be4077040 0x7f8be4079550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:56:04.520 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:04.517+0000 7f8c09400640 1 --2- 172.21.15.32:0/4183414241 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f8be4077040 0x7f8be4079550 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7f8bf8033000 tx=0x7f8bf8004730 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:56:04.522 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:04.520+0000 7f8c01600640 1 -- 172.21.15.32:0/4183414241 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7f8bfc09ae20 con 0x7f8c04162210 2024-08-03T20:56:04.771 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:04.769+0000 7f8c0be00640 1 -- 172.21.15.32:0/4183414241 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f8c04160ee0 con 0x7f8c04162210 2024-08-03T20:56:04.771 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:04.770+0000 7f8c01600640 1 -- 172.21.15.32:0/4183414241 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 148 v148) ==== 95+0+4390 (secure 0 0 0) 0x7f8bfc0611a0 con 0x7f8c04162210 2024-08-03T20:56:04.772 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-03T20:56:04.772 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":148,"btime":"2024-08-03T20:56:04:256619+0000","default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6439,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6835/531080160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6834","nonce":531080160},{"type":"v1","addr":"172.21.15.118:6835","nonce":531080160}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":84},{"gid":6454,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6837/2796882319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6836","nonce":2796882319},{"type":"v1","addr":"172.21.15.118:6837","nonce":2796882319}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":86},{"gid":6813,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6834/2686933890","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":2686933890},{"type":"v1","addr":"172.21.15.32:6834","nonce":2686933890}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":85}],"filesystems":[{"mdsmap":{"epoch":148,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-08-03T20:54:26.950518+0000","modified":"2024-08-03T20:56:04.256526+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":95,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":6}]} 2024-08-03T20:56:04.772 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 148 2024-08-03T20:56:04.774 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:04.772+0000 7f8c0be00640 1 -- 172.21.15.32:0/4183414241 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f8be4077040 msgr2=0x7f8be4079550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:56:04.774 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:04.772+0000 7f8c0be00640 1 --2- 172.21.15.32:0/4183414241 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f8be4077040 0x7f8be4079550 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7f8bf8033000 tx=0x7f8bf8004730 comp rx=0 tx=0).stop 2024-08-03T20:56:04.774 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:04.772+0000 7f8c0be00640 1 -- 172.21.15.32:0/4183414241 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8c04162210 msgr2=0x7f8c041601e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:56:04.774 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:04.772+0000 7f8c0be00640 1 --2- 172.21.15.32:0/4183414241 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8c04162210 0x7f8c041601e0 secure :-1 s=READY pgs=1296 cs=0 l=1 rev1=1 crypto rx=0x7f8bfc00c700 tx=0x7f8bfc00cbc0 comp rx=0 tx=0).stop 2024-08-03T20:56:04.775 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:04.772+0000 7f8c0be00640 1 -- 172.21.15.32:0/4183414241 shutdown_connections 2024-08-03T20:56:04.775 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:04.773+0000 7f8c0be00640 1 --2- 172.21.15.32:0/4183414241 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f8be4077040 0x7f8be4079550 unknown :-1 s=CLOSED pgs=535 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:04.775 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:04.773+0000 7f8c0be00640 1 --2- 172.21.15.32:0/4183414241 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8c04162210 0x7f8c041601e0 unknown :-1 s=CLOSED pgs=1296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:04.775 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:04.773+0000 7f8c0be00640 1 --2- 172.21.15.32:0/4183414241 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f8c040b39a0 0x7f8c0415fca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:04.775 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:04.773+0000 7f8c0be00640 1 --2- 172.21.15.32:0/4183414241 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8c0406f720 0x7f8c040cf600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:04.775 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:04.773+0000 7f8c0be00640 1 -- 172.21.15.32:0/4183414241 >> 172.21.15.32:0/4183414241 conn(0x7f8c040b1bd0 msgr2=0x7f8c040b0d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:56:04.775 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:04.773+0000 7f8c0be00640 1 -- 172.21.15.32:0/4183414241 shutdown_connections 2024-08-03T20:56:04.775 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:04.773+0000 7f8c0be00640 1 -- 172.21.15.32:0/4183414241 wait complete. 2024-08-03T20:56:04.796 DEBUG:tasks.ceph_test_case:wait_until_true: success in 0s and 0 retries 2024-08-03T20:56:04.796 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-08-03T20:56:05.025 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:05.023+0000 7fdf75e00640 1 -- 172.21.15.32:0/3154545927 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdf7015f610 msgr2=0x7fdf700cdf70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:56:05.025 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:05.023+0000 7fdf75e00640 1 --2- 172.21.15.32:0/3154545927 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdf7015f610 0x7fdf700cdf70 secure :-1 s=READY pgs=1298 cs=0 l=1 rev1=1 crypto rx=0x7fdf68005fd0 tx=0x7fdf6802a8b0 comp rx=0 tx=0).stop 2024-08-03T20:56:05.025 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:05.023+0000 7fdf75e00640 1 -- 172.21.15.32:0/3154545927 shutdown_connections 2024-08-03T20:56:05.025 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:05.023+0000 7fdf75e00640 1 --2- 172.21.15.32:0/3154545927 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdf7015f610 0x7fdf700cdf70 unknown :-1 s=CLOSED pgs=1298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:05.026 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:05.023+0000 7fdf75e00640 1 --2- 172.21.15.32:0/3154545927 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fdf700b38b0 0x7fdf701523f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:05.026 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:05.023+0000 7fdf75e00640 1 -- 172.21.15.32:0/3154545927 >> 172.21.15.32:0/3154545927 conn(0x7fdf700b1bd0 msgr2=0x7fdf700b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:56:05.026 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:05.023+0000 7fdf75e00640 1 -- 172.21.15.32:0/3154545927 shutdown_connections 2024-08-03T20:56:05.026 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:05.023+0000 7fdf75e00640 1 -- 172.21.15.32:0/3154545927 wait complete. 2024-08-03T20:56:05.027 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:05.026+0000 7fdf75e00640 1 Processor -- start 2024-08-03T20:56:05.027 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:05.026+0000 7fdf75e00640 1 -- start start 2024-08-03T20:56:05.030 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:05.027+0000 7fdf75e00640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fdf700b38b0 0x7fdf70146a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:56:05.030 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:05.027+0000 7fdf75e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdf700b4ce0 0x7fdf70158910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:56:05.030 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:05.027+0000 7fdf75e00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fdf70153ba0 0x7fdf70152ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:56:05.030 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:05.027+0000 7fdf75e00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fdf700cea90 con 0x7fdf700b4ce0 2024-08-03T20:56:05.030 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:05.027+0000 7fdf75e00640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7fdf700ce910 con 0x7fdf70153ba0 2024-08-03T20:56:05.030 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:05.027+0000 7fdf75e00640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7fdf700cec10 con 0x7fdf700b38b0 2024-08-03T20:56:05.030 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:05.028+0000 7fdf67e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdf700b4ce0 0x7fdf70158910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:56:05.030 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:05.028+0000 7fdf67e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdf700b4ce0 0x7fdf70158910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:42646/0 (socket says 172.21.15.32:42646) 2024-08-03T20:56:05.030 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:05.028+0000 7fdf67e00640 1 -- 172.21.15.32:0/3183861195 learned_addr learned my addr 172.21.15.32:0/3183861195 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:56:05.031 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:05.028+0000 7fdf6ea00640 1 --2- 172.21.15.32:0/3183861195 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fdf700b38b0 0x7fdf70146a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:56:05.031 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:05.028+0000 7fdf6f400640 1 --2- 172.21.15.32:0/3183861195 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fdf70153ba0 0x7fdf70152ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:56:05.031 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:05.028+0000 7fdf67e00640 1 -- 172.21.15.32:0/3183861195 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fdf700b38b0 msgr2=0x7fdf70146a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:56:05.031 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:05.028+0000 7fdf67e00640 1 --2- 172.21.15.32:0/3183861195 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fdf700b38b0 0x7fdf70146a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:05.031 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:05.028+0000 7fdf67e00640 1 -- 172.21.15.32:0/3183861195 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fdf70153ba0 msgr2=0x7fdf70152ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:56:05.031 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:05.028+0000 7fdf67e00640 1 --2- 172.21.15.32:0/3183861195 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fdf70153ba0 0x7fdf70152ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:05.031 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:05.028+0000 7fdf67e00640 1 -- 172.21.15.32:0/3183861195 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdf701f31a0 con 0x7fdf700b4ce0 2024-08-03T20:56:05.031 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:05.028+0000 7fdf67e00640 1 --2- 172.21.15.32:0/3183861195 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdf700b4ce0 0x7fdf70158910 secure :-1 s=READY pgs=1299 cs=0 l=1 rev1=1 crypto rx=0x7fdf5800c3a0 tx=0x7fdf5800c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:56:05.032 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:05.029+0000 7fdf67400640 1 -- 172.21.15.32:0/3183861195 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf58011790 con 0x7fdf700b4ce0 2024-08-03T20:56:05.032 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:05.029+0000 7fdf67400640 1 -- 172.21.15.32:0/3183861195 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fdf58017070 con 0x7fdf700b4ce0 2024-08-03T20:56:05.032 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:05.029+0000 7fdf75e00640 1 -- 172.21.15.32:0/3183861195 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdf701f3480 con 0x7fdf700b4ce0 2024-08-03T20:56:05.032 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:05.029+0000 7fdf67400640 1 -- 172.21.15.32:0/3183861195 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf58011cc0 con 0x7fdf700b4ce0 2024-08-03T20:56:05.032 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:05.029+0000 7fdf75e00640 1 -- 172.21.15.32:0/3183861195 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdf701f3a40 con 0x7fdf700b4ce0 2024-08-03T20:56:05.032 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:05.030+0000 7fdf75e00640 1 -- 172.21.15.32:0/3183861195 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdf700cf130 con 0x7fdf700b4ce0 2024-08-03T20:56:05.033 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:05.031+0000 7fdf67400640 1 -- 172.21.15.32:0/3183861195 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7fdf5801c460 con 0x7fdf700b4ce0 2024-08-03T20:56:05.034 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:05.032+0000 7fdf67400640 1 --2- 172.21.15.32:0/3183861195 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fdf4c077010 0x7fdf4c079520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:56:05.034 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:05.032+0000 7fdf67400640 1 -- 172.21.15.32:0/3183861195 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 6714+0+0 (secure 0 0 0) 0x7fdf58098980 con 0x7fdf700b4ce0 2024-08-03T20:56:05.034 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:05.033+0000 7fdf6ea00640 1 --2- 172.21.15.32:0/3183861195 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fdf4c077010 0x7fdf4c079520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:56:05.035 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:05.033+0000 7fdf6ea00640 1 --2- 172.21.15.32:0/3183861195 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fdf4c077010 0x7fdf4c079520 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7fdf6000bb50 tx=0x7fdf6000b2e0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:56:05.038 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:05.037+0000 7fdf67400640 1 -- 172.21.15.32:0/3183861195 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7fdf58061140 con 0x7fdf700b4ce0 2024-08-03T20:56:05.283 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:05.282+0000 7fdf75e00640 1 -- 172.21.15.32:0/3183861195 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fdf701f2f30 con 0x7fdf700b4ce0 2024-08-03T20:56:05.284 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:05.282+0000 7fdf67400640 1 -- 172.21.15.32:0/3183861195 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 148 v148) ==== 95+0+4390 (secure 0 0 0) 0x7fdf58065690 con 0x7fdf700b4ce0 2024-08-03T20:56:05.284 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-03T20:56:05.284 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":148,"btime":"2024-08-03T20:56:04:256619+0000","default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6439,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6835/531080160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6834","nonce":531080160},{"type":"v1","addr":"172.21.15.118:6835","nonce":531080160}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":84},{"gid":6454,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6837/2796882319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6836","nonce":2796882319},{"type":"v1","addr":"172.21.15.118:6837","nonce":2796882319}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":86},{"gid":6813,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6834/2686933890","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":2686933890},{"type":"v1","addr":"172.21.15.32:6834","nonce":2686933890}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":85}],"filesystems":[{"mdsmap":{"epoch":148,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-08-03T20:54:26.950518+0000","modified":"2024-08-03T20:56:04.256526+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":95,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":6}]} 2024-08-03T20:56:05.285 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 148 2024-08-03T20:56:05.287 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:05.285+0000 7fdf75e00640 1 -- 172.21.15.32:0/3183861195 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fdf4c077010 msgr2=0x7fdf4c079520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:56:05.287 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:05.285+0000 7fdf75e00640 1 --2- 172.21.15.32:0/3183861195 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fdf4c077010 0x7fdf4c079520 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7fdf6000bb50 tx=0x7fdf6000b2e0 comp rx=0 tx=0).stop 2024-08-03T20:56:05.287 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:05.285+0000 7fdf75e00640 1 -- 172.21.15.32:0/3183861195 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdf700b4ce0 msgr2=0x7fdf70158910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:56:05.287 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:05.285+0000 7fdf75e00640 1 --2- 172.21.15.32:0/3183861195 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdf700b4ce0 0x7fdf70158910 secure :-1 s=READY pgs=1299 cs=0 l=1 rev1=1 crypto rx=0x7fdf5800c3a0 tx=0x7fdf5800c860 comp rx=0 tx=0).stop 2024-08-03T20:56:05.287 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:05.285+0000 7fdf75e00640 1 -- 172.21.15.32:0/3183861195 shutdown_connections 2024-08-03T20:56:05.287 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:05.285+0000 7fdf75e00640 1 --2- 172.21.15.32:0/3183861195 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fdf4c077010 0x7fdf4c079520 unknown :-1 s=CLOSED pgs=536 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:05.287 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:05.285+0000 7fdf75e00640 1 --2- 172.21.15.32:0/3183861195 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fdf70153ba0 0x7fdf70152ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:05.287 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:05.285+0000 7fdf75e00640 1 --2- 172.21.15.32:0/3183861195 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdf700b4ce0 0x7fdf70158910 unknown :-1 s=CLOSED pgs=1299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:05.287 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:05.285+0000 7fdf75e00640 1 --2- 172.21.15.32:0/3183861195 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fdf700b38b0 0x7fdf70146a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:05.287 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:05.285+0000 7fdf75e00640 1 -- 172.21.15.32:0/3183861195 >> 172.21.15.32:0/3183861195 conn(0x7fdf700b1bd0 msgr2=0x7fdf7015a2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:56:05.287 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:05.285+0000 7fdf75e00640 1 -- 172.21.15.32:0/3183861195 shutdown_connections 2024-08-03T20:56:05.288 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:05.285+0000 7fdf75e00640 1 -- 172.21.15.32:0/3183861195 wait complete. 2024-08-03T20:56:05.309 DEBUG:tasks.ceph_test_case:wait_until_true: success in 0s and 0 retries 2024-08-03T20:56:05.309 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-08-03T20:56:05.538 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:05.536+0000 7fe0dda00640 1 -- 172.21.15.32:0/3471029352 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe0d8073e30 msgr2=0x7fe0d80ce050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:56:05.539 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:05.536+0000 7fe0dda00640 1 --2- 172.21.15.32:0/3471029352 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe0d8073e30 0x7fe0d80ce050 secure :-1 s=READY pgs=1300 cs=0 l=1 rev1=1 crypto rx=0x7fe0c8005fd0 tx=0x7fe0c802a530 comp rx=0 tx=0).stop 2024-08-03T20:56:05.539 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:05.537+0000 7fe0dda00640 1 -- 172.21.15.32:0/3471029352 shutdown_connections 2024-08-03T20:56:05.539 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:05.537+0000 7fe0dda00640 1 --2- 172.21.15.32:0/3471029352 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fe0d806d9f0 0x7fe0d806de00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:05.539 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:05.537+0000 7fe0dda00640 1 --2- 172.21.15.32:0/3471029352 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe0d8073e30 0x7fe0d80ce050 unknown :-1 s=CLOSED pgs=1300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:05.539 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:05.537+0000 7fe0dda00640 1 -- 172.21.15.32:0/3471029352 >> 172.21.15.32:0/3471029352 conn(0x7fe0d8072e70 msgr2=0x7fe0d8073280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:56:05.539 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:05.537+0000 7fe0dda00640 1 -- 172.21.15.32:0/3471029352 shutdown_connections 2024-08-03T20:56:05.539 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:05.537+0000 7fe0dda00640 1 -- 172.21.15.32:0/3471029352 wait complete. 2024-08-03T20:56:05.540 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:05.539+0000 7fe0dda00640 1 Processor -- start 2024-08-03T20:56:05.540 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:05.539+0000 7fe0dda00640 1 -- start start 2024-08-03T20:56:05.543 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:05.540+0000 7fe0dda00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fe0d806d9f0 0x7fe0d8154160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:56:05.543 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:05.540+0000 7fe0dda00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe0d8073e30 0x7fe0d80a94b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:56:05.543 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:05.540+0000 7fe0dda00640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fe0d815b1b0 0x7fe0d81f5bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:56:05.543 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:05.540+0000 7fe0dda00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fe0d80b4dd0 con 0x7fe0d8073e30 2024-08-03T20:56:05.543 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:05.540+0000 7fe0dda00640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7fe0d80b4c50 con 0x7fe0d806d9f0 2024-08-03T20:56:05.543 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:05.540+0000 7fe0dda00640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7fe0d80b4f50 con 0x7fe0d815b1b0 2024-08-03T20:56:05.544 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:05.541+0000 7fe0d7400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe0d8073e30 0x7fe0d80a94b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:56:05.544 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:05.541+0000 7fe0d7400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe0d8073e30 0x7fe0d80a94b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:42674/0 (socket says 172.21.15.32:42674) 2024-08-03T20:56:05.544 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:05.541+0000 7fe0d7400640 1 -- 172.21.15.32:0/2910198457 learned_addr learned my addr 172.21.15.32:0/2910198457 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:56:05.544 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:05.541+0000 7fe0dd000640 1 --2- 172.21.15.32:0/2910198457 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fe0d815b1b0 0x7fe0d81f5bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:56:05.544 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:05.541+0000 7fe0d7e00640 1 --2- 172.21.15.32:0/2910198457 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fe0d806d9f0 0x7fe0d8154160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:56:05.544 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:05.541+0000 7fe0d7400640 1 -- 172.21.15.32:0/2910198457 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fe0d815b1b0 msgr2=0x7fe0d81f5bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:56:05.544 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:05.541+0000 7fe0d7400640 1 --2- 172.21.15.32:0/2910198457 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fe0d815b1b0 0x7fe0d81f5bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:05.544 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:05.541+0000 7fe0d7400640 1 -- 172.21.15.32:0/2910198457 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fe0d806d9f0 msgr2=0x7fe0d8154160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:56:05.544 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:05.541+0000 7fe0d7400640 1 --2- 172.21.15.32:0/2910198457 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fe0d806d9f0 0x7fe0d8154160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:05.544 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:05.541+0000 7fe0d7400640 1 -- 172.21.15.32:0/2910198457 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe0d80a99f0 con 0x7fe0d8073e30 2024-08-03T20:56:05.544 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:05.541+0000 7fe0dd000640 1 --2- 172.21.15.32:0/2910198457 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fe0d815b1b0 0x7fe0d81f5bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-03T20:56:05.544 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:05.541+0000 7fe0d7400640 1 --2- 172.21.15.32:0/2910198457 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe0d8073e30 0x7fe0d80a94b0 secure :-1 s=READY pgs=1301 cs=0 l=1 rev1=1 crypto rx=0x7fe0c400b3c0 tx=0x7fe0c400b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:56:05.544 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:05.542+0000 7fe0cfe00640 1 -- 172.21.15.32:0/2910198457 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe0c4011830 con 0x7fe0d8073e30 2024-08-03T20:56:05.545 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:05.542+0000 7fe0cfe00640 1 -- 172.21.15.32:0/2910198457 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fe0c4016070 con 0x7fe0d8073e30 2024-08-03T20:56:05.545 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:05.542+0000 7fe0cfe00640 1 -- 172.21.15.32:0/2910198457 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe0c4011da0 con 0x7fe0d8073e30 2024-08-03T20:56:05.545 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:05.542+0000 7fe0dda00640 1 -- 172.21.15.32:0/2910198457 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe0d80b3720 con 0x7fe0d8073e30 2024-08-03T20:56:05.545 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:05.542+0000 7fe0dda00640 1 -- 172.21.15.32:0/2910198457 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe0d80b3ce0 con 0x7fe0d8073e30 2024-08-03T20:56:05.545 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:05.543+0000 7fe0dda00640 1 -- 172.21.15.32:0/2910198457 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe0d80b5440 con 0x7fe0d8073e30 2024-08-03T20:56:05.545 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:05.544+0000 7fe0cfe00640 1 -- 172.21.15.32:0/2910198457 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7fe0c401b460 con 0x7fe0d8073e30 2024-08-03T20:56:05.546 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:05.545+0000 7fe0cfe00640 1 --2- 172.21.15.32:0/2910198457 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fe0b8077010 0x7fe0b8079520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:56:05.546 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:05.545+0000 7fe0cfe00640 1 -- 172.21.15.32:0/2910198457 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 6714+0+0 (secure 0 0 0) 0x7fe0c4098740 con 0x7fe0d8073e30 2024-08-03T20:56:05.547 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:05.545+0000 7fe0d7e00640 1 --2- 172.21.15.32:0/2910198457 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fe0b8077010 0x7fe0b8079520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:56:05.548 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:05.546+0000 7fe0d7e00640 1 --2- 172.21.15.32:0/2910198457 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fe0b8077010 0x7fe0b8079520 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7fe0c8033000 tx=0x7fe0c8004730 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:56:05.551 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:05.550+0000 7fe0cfe00640 1 -- 172.21.15.32:0/2910198457 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7fe0c4020020 con 0x7fe0d8073e30 2024-08-03T20:56:05.794 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:05.792+0000 7fe0dda00640 1 -- 172.21.15.32:0/2910198457 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe0d80a9c50 con 0x7fe0d8073e30 2024-08-03T20:56:05.794 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:05.793+0000 7fe0cfe00640 1 -- 172.21.15.32:0/2910198457 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 148 v148) ==== 95+0+4390 (secure 0 0 0) 0x7fe0c40615d0 con 0x7fe0d8073e30 2024-08-03T20:56:05.795 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-03T20:56:05.795 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":148,"btime":"2024-08-03T20:56:04:256619+0000","default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6439,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6835/531080160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6834","nonce":531080160},{"type":"v1","addr":"172.21.15.118:6835","nonce":531080160}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":84},{"gid":6454,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6837/2796882319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6836","nonce":2796882319},{"type":"v1","addr":"172.21.15.118:6837","nonce":2796882319}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":86},{"gid":6813,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6834/2686933890","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":2686933890},{"type":"v1","addr":"172.21.15.32:6834","nonce":2686933890}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":85}],"filesystems":[{"mdsmap":{"epoch":148,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-08-03T20:54:26.950518+0000","modified":"2024-08-03T20:56:04.256526+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":95,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":6}]} 2024-08-03T20:56:05.795 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 148 2024-08-03T20:56:05.797 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:05.795+0000 7fe0dda00640 1 -- 172.21.15.32:0/2910198457 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fe0b8077010 msgr2=0x7fe0b8079520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:56:05.797 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:05.795+0000 7fe0dda00640 1 --2- 172.21.15.32:0/2910198457 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fe0b8077010 0x7fe0b8079520 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7fe0c8033000 tx=0x7fe0c8004730 comp rx=0 tx=0).stop 2024-08-03T20:56:05.797 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:05.795+0000 7fe0dda00640 1 -- 172.21.15.32:0/2910198457 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe0d8073e30 msgr2=0x7fe0d80a94b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:56:05.797 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:05.795+0000 7fe0dda00640 1 --2- 172.21.15.32:0/2910198457 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe0d8073e30 0x7fe0d80a94b0 secure :-1 s=READY pgs=1301 cs=0 l=1 rev1=1 crypto rx=0x7fe0c400b3c0 tx=0x7fe0c400b880 comp rx=0 tx=0).stop 2024-08-03T20:56:05.797 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:05.795+0000 7fe0dda00640 1 -- 172.21.15.32:0/2910198457 shutdown_connections 2024-08-03T20:56:05.798 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:05.795+0000 7fe0dda00640 1 --2- 172.21.15.32:0/2910198457 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fe0b8077010 0x7fe0b8079520 unknown :-1 s=CLOSED pgs=537 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:05.798 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:05.795+0000 7fe0dda00640 1 --2- 172.21.15.32:0/2910198457 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fe0d815b1b0 0x7fe0d81f5bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:05.798 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:05.795+0000 7fe0dda00640 1 --2- 172.21.15.32:0/2910198457 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe0d8073e30 0x7fe0d80a94b0 unknown :-1 s=CLOSED pgs=1301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:05.798 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:05.795+0000 7fe0dda00640 1 --2- 172.21.15.32:0/2910198457 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fe0d806d9f0 0x7fe0d8154160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:05.798 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:05.796+0000 7fe0dda00640 1 -- 172.21.15.32:0/2910198457 >> 172.21.15.32:0/2910198457 conn(0x7fe0d8072e70 msgr2=0x7fe0d80701c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:56:05.798 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:05.796+0000 7fe0dda00640 1 -- 172.21.15.32:0/2910198457 shutdown_connections 2024-08-03T20:56:05.798 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:05.796+0000 7fe0dda00640 1 -- 172.21.15.32:0/2910198457 wait complete. 2024-08-03T20:56:05.823 INFO:tasks.cephfs.test_damage:Result: Mutation 'Delete mds0_inotable' led to DAMAGED state 2024-08-03T20:56:05.823 INFO:tasks.cephfs.test_damage:Applying mutation 'Delete 100.00000000' 2024-08-03T20:56:05.823 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-08-03T20:56:05.823 DEBUG:teuthology.orchestra.run.smithi032:> dd if=/proc/self/mounts of=/dev/stdout 2024-08-03T20:56:05.858 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053453,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6499208k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 none /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249604k,nr_inodes=812401,mode=700,uid=1000,gid=1316 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6499208k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-08-03T20:56:05.858 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi032.front.sepia.ceph.com 2024-08-03T20:56:05.858 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-08-03T20:56:05.859 DEBUG:teuthology.orchestra.run.smithi032:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-08-03T20:56:05.933 INFO:teuthology.orchestra.run.smithi032.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.0': No such file or directory 2024-08-03T20:56:05.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-03T20:56:05.934 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-08-03T20:56:06.204 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:06.202+0000 7f38bba00640 1 -- 172.21.15.32:0/4058997587 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f38b4144f90 msgr2=0x7f38b4074570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:56:06.204 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:06.202+0000 7f38bba00640 1 --2- 172.21.15.32:0/4058997587 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f38b4144f90 0x7f38b4074570 secure :-1 s=READY pgs=1303 cs=0 l=1 rev1=1 crypto rx=0x7f38b4073b50 tx=0x7f389c02a670 comp rx=0 tx=0).stop 2024-08-03T20:56:06.204 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:06.202+0000 7f38bba00640 1 -- 172.21.15.32:0/4058997587 shutdown_connections 2024-08-03T20:56:06.204 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:06.202+0000 7f38bba00640 1 --2- 172.21.15.32:0/4058997587 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f38b4144f90 0x7f38b4074570 unknown :-1 s=CLOSED pgs=1303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:06.204 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:06.202+0000 7f38bba00640 1 -- 172.21.15.32:0/4058997587 >> 172.21.15.32:0/4058997587 conn(0x7f38b40b1bd0 msgr2=0x7f38b40b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:56:06.205 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:06.202+0000 7f38bba00640 1 -- 172.21.15.32:0/4058997587 shutdown_connections 2024-08-03T20:56:06.205 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:06.202+0000 7f38bba00640 1 -- 172.21.15.32:0/4058997587 wait complete. 2024-08-03T20:56:06.206 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:06.205+0000 7f38bba00640 1 Processor -- start 2024-08-03T20:56:06.206 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:06.205+0000 7f38bba00640 1 -- start start 2024-08-03T20:56:06.209 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:06.206+0000 7f38bba00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f38b406e930 0x7f38b4141ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:56:06.209 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:06.206+0000 7f38bba00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f38b4144f90 0x7f38b415ebf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:56:06.209 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:06.206+0000 7f38bba00640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f38b415fff0 0x7f38b415f130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:56:06.209 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:06.206+0000 7f38bba00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f38b40cf840 con 0x7f38b4144f90 2024-08-03T20:56:06.209 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:06.206+0000 7f38bba00640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f38b4145f30 con 0x7f38b406e930 2024-08-03T20:56:06.210 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:06.206+0000 7f38bba00640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7f38b40cf9c0 con 0x7f38b415fff0 2024-08-03T20:56:06.210 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:06.207+0000 7f38abe00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f38b4144f90 0x7f38b415ebf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:56:06.210 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:06.207+0000 7f38abe00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f38b4144f90 0x7f38b415ebf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:42692/0 (socket says 172.21.15.32:42692) 2024-08-03T20:56:06.210 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:06.207+0000 7f38abe00640 1 -- 172.21.15.32:0/2582003068 learned_addr learned my addr 172.21.15.32:0/2582003068 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:56:06.210 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:06.207+0000 7f38b9000640 1 --2- 172.21.15.32:0/2582003068 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f38b406e930 0x7f38b4141ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:56:06.210 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:06.207+0000 7f38b9a00640 1 --2- 172.21.15.32:0/2582003068 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f38b415fff0 0x7f38b415f130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:56:06.210 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:06.207+0000 7f38abe00640 1 -- 172.21.15.32:0/2582003068 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f38b415fff0 msgr2=0x7f38b415f130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:56:06.210 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:06.207+0000 7f38abe00640 1 --2- 172.21.15.32:0/2582003068 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f38b415fff0 0x7f38b415f130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:06.210 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:06.207+0000 7f38abe00640 1 -- 172.21.15.32:0/2582003068 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f38b406e930 msgr2=0x7f38b4141ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:56:06.210 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:06.207+0000 7f38abe00640 1 --2- 172.21.15.32:0/2582003068 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f38b406e930 0x7f38b4141ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:06.210 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:06.207+0000 7f38abe00640 1 -- 172.21.15.32:0/2582003068 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f38b41f6970 con 0x7f38b4144f90 2024-08-03T20:56:06.210 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:06.207+0000 7f38abe00640 1 --2- 172.21.15.32:0/2582003068 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f38b4144f90 0x7f38b415ebf0 secure :-1 s=READY pgs=1304 cs=0 l=1 rev1=1 crypto rx=0x7f389c0099c0 tx=0x7f389c003910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:56:06.210 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:06.208+0000 7f38b1600640 1 -- 172.21.15.32:0/2582003068 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f389c03e030 con 0x7f38b4144f90 2024-08-03T20:56:06.211 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:06.208+0000 7f38b1600640 1 -- 172.21.15.32:0/2582003068 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f389c003e00 con 0x7f38b4144f90 2024-08-03T20:56:06.211 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:06.208+0000 7f38b1600640 1 -- 172.21.15.32:0/2582003068 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f389c035040 con 0x7f38b4144f90 2024-08-03T20:56:06.211 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:06.208+0000 7f38bba00640 1 -- 172.21.15.32:0/2582003068 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f38b41f6c50 con 0x7f38b4144f90 2024-08-03T20:56:06.211 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:06.208+0000 7f38bba00640 1 -- 172.21.15.32:0/2582003068 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f38b41f7210 con 0x7f38b4144f90 2024-08-03T20:56:06.211 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:06.209+0000 7f38bba00640 1 -- 172.21.15.32:0/2582003068 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f38b4160400 con 0x7f38b4144f90 2024-08-03T20:56:06.211 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:06.210+0000 7f38b1600640 1 -- 172.21.15.32:0/2582003068 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f389c041030 con 0x7f38b4144f90 2024-08-03T20:56:06.212 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:06.211+0000 7f38b1600640 1 --2- 172.21.15.32:0/2582003068 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f3890077040 0x7f3890079550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:56:06.213 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:06.211+0000 7f38b1600640 1 -- 172.21.15.32:0/2582003068 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 6714+0+0 (secure 0 0 0) 0x7f389c0b94d0 con 0x7f38b4144f90 2024-08-03T20:56:06.213 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:06.212+0000 7f38b9000640 1 --2- 172.21.15.32:0/2582003068 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f3890077040 0x7f3890079550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:56:06.214 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:06.212+0000 7f38b9000640 1 --2- 172.21.15.32:0/2582003068 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f3890077040 0x7f3890079550 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7f38a4000f60 tx=0x7f38a4009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:56:06.217 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:06.216+0000 7f38b1600640 1 -- 172.21.15.32:0/2582003068 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7f389c081c80 con 0x7f38b4144f90 2024-08-03T20:56:06.458 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:06.457+0000 7f38bba00640 1 -- 172.21.15.32:0/2582003068 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7f38b415a910 con 0x7f38b4144f90 2024-08-03T20:56:07.269 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:07.267+0000 7f38b1600640 1 -- 172.21.15.32:0/2582003068 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v149) ==== 161+0+0 (secure 0 0 0) 0x7f389c0861d0 con 0x7f38b4144f90 2024-08-03T20:56:07.269 INFO:teuthology.orchestra.run.smithi032.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-08-03T20:56:07.271 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:07.270+0000 7f38bba00640 1 -- 172.21.15.32:0/2582003068 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f3890077040 msgr2=0x7f3890079550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:56:07.272 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:07.270+0000 7f38bba00640 1 --2- 172.21.15.32:0/2582003068 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f3890077040 0x7f3890079550 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7f38a4000f60 tx=0x7f38a4009040 comp rx=0 tx=0).stop 2024-08-03T20:56:07.272 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:07.270+0000 7f38bba00640 1 -- 172.21.15.32:0/2582003068 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f38b4144f90 msgr2=0x7f38b415ebf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:56:07.272 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:07.270+0000 7f38bba00640 1 --2- 172.21.15.32:0/2582003068 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f38b4144f90 0x7f38b415ebf0 secure :-1 s=READY pgs=1304 cs=0 l=1 rev1=1 crypto rx=0x7f389c0099c0 tx=0x7f389c003910 comp rx=0 tx=0).stop 2024-08-03T20:56:07.272 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:07.270+0000 7f38bba00640 1 -- 172.21.15.32:0/2582003068 shutdown_connections 2024-08-03T20:56:07.272 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:07.270+0000 7f38bba00640 1 --2- 172.21.15.32:0/2582003068 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f3890077040 0x7f3890079550 unknown :-1 s=CLOSED pgs=538 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:07.272 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:07.270+0000 7f38bba00640 1 --2- 172.21.15.32:0/2582003068 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f38b415fff0 0x7f38b415f130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:07.272 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:07.270+0000 7f38bba00640 1 --2- 172.21.15.32:0/2582003068 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f38b4144f90 0x7f38b415ebf0 unknown :-1 s=CLOSED pgs=1304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:07.272 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:07.270+0000 7f38bba00640 1 --2- 172.21.15.32:0/2582003068 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f38b406e930 0x7f38b4141ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:07.272 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:07.270+0000 7f38bba00640 1 -- 172.21.15.32:0/2582003068 >> 172.21.15.32:0/2582003068 conn(0x7f38b40b1bd0 msgr2=0x7f38b40b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:56:07.272 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:07.270+0000 7f38bba00640 1 -- 172.21.15.32:0/2582003068 shutdown_connections 2024-08-03T20:56:07.272 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:07.270+0000 7f38bba00640 1 -- 172.21.15.32:0/2582003068 wait complete. 2024-08-03T20:56:07.299 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mds repaired 0 2024-08-03T20:56:07.529 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:07.527+0000 7f445ea00640 1 -- 172.21.15.32:0/572357699 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f445806e930 msgr2=0x7f4458144f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:56:07.529 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:07.527+0000 7f445ea00640 1 --2- 172.21.15.32:0/572357699 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f445806e930 0x7f4458144f90 secure :-1 s=READY pgs=1306 cs=0 l=1 rev1=1 crypto rx=0x7f4448005fd0 tx=0x7f444802a670 comp rx=0 tx=0).stop 2024-08-03T20:56:07.529 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:07.527+0000 7f445ea00640 1 -- 172.21.15.32:0/572357699 shutdown_connections 2024-08-03T20:56:07.529 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:07.527+0000 7f445ea00640 1 --2- 172.21.15.32:0/572357699 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f445806e930 0x7f4458144f90 unknown :-1 s=CLOSED pgs=1306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:07.529 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:07.527+0000 7f445ea00640 1 -- 172.21.15.32:0/572357699 >> 172.21.15.32:0/572357699 conn(0x7f44580b1bd0 msgr2=0x7f44580b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:56:07.529 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:07.527+0000 7f445ea00640 1 -- 172.21.15.32:0/572357699 shutdown_connections 2024-08-03T20:56:07.529 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:07.527+0000 7f445ea00640 1 -- 172.21.15.32:0/572357699 wait complete. 2024-08-03T20:56:07.531 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:07.529+0000 7f445ea00640 1 Processor -- start 2024-08-03T20:56:07.531 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:07.530+0000 7f445ea00640 1 -- start start 2024-08-03T20:56:07.533 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:07.531+0000 7f445ea00640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f445806e930 0x7f44581f8300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:56:07.533 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:07.531+0000 7f445ea00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f4458073e60 0x7f44581f8ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:56:07.533 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:07.531+0000 7f445ea00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f445815fff0 0x7f44580ccd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:56:07.533 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:07.531+0000 7f445ea00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f44580cf840 con 0x7f445815fff0 2024-08-03T20:56:07.533 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:07.531+0000 7f445ea00640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f4458145f30 con 0x7f4458073e60 2024-08-03T20:56:07.534 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:07.531+0000 7f445ea00640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7f44580cf9c0 con 0x7f445806e930 2024-08-03T20:56:07.534 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:07.531+0000 7f445ca00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f445815fff0 0x7f44580ccd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:56:07.534 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:07.531+0000 7f445ca00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f445815fff0 0x7f44580ccd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:42708/0 (socket says 172.21.15.32:42708) 2024-08-03T20:56:07.534 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:07.531+0000 7f445ca00640 1 -- 172.21.15.32:0/1735376223 learned_addr learned my addr 172.21.15.32:0/1735376223 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:56:07.534 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:07.532+0000 7f4457e00640 1 --2- 172.21.15.32:0/1735376223 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f445806e930 0x7f44581f8300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:56:07.534 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:07.532+0000 7f444f400640 1 --2- 172.21.15.32:0/1735376223 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f4458073e60 0x7f44581f8ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:56:07.534 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:07.532+0000 7f445ca00640 1 -- 172.21.15.32:0/1735376223 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f445806e930 msgr2=0x7f44581f8300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:56:07.534 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:07.532+0000 7f445ca00640 1 --2- 172.21.15.32:0/1735376223 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f445806e930 0x7f44581f8300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:07.534 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:07.532+0000 7f445ca00640 1 -- 172.21.15.32:0/1735376223 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f4458073e60 msgr2=0x7f44581f8ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:56:07.534 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:07.532+0000 7f445ca00640 1 --2- 172.21.15.32:0/1735376223 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f4458073e60 0x7f44581f8ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:07.534 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:07.532+0000 7f445ca00640 1 -- 172.21.15.32:0/1735376223 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f44581f9780 con 0x7f445815fff0 2024-08-03T20:56:07.535 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:07.532+0000 7f445ca00640 1 --2- 172.21.15.32:0/1735376223 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f445815fff0 0x7f44580ccd00 secure :-1 s=READY pgs=1307 cs=0 l=1 rev1=1 crypto rx=0x7f445000c760 tx=0x7f445000cb60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:56:07.535 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:07.532+0000 7f4454c00640 1 -- 172.21.15.32:0/1735376223 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4450003b70 con 0x7f445815fff0 2024-08-03T20:56:07.536 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:07.532+0000 7f4454c00640 1 -- 172.21.15.32:0/1735376223 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f4450003d10 con 0x7f445815fff0 2024-08-03T20:56:07.536 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:07.532+0000 7f4454c00640 1 -- 172.21.15.32:0/1735376223 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44500052d0 con 0x7f445815fff0 2024-08-03T20:56:07.536 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:07.533+0000 7f445ea00640 1 -- 172.21.15.32:0/1735376223 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4458150f40 con 0x7f445815fff0 2024-08-03T20:56:07.536 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:07.533+0000 7f445ea00640 1 -- 172.21.15.32:0/1735376223 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4458151500 con 0x7f445815fff0 2024-08-03T20:56:07.536 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:07.534+0000 7f445ea00640 1 -- 172.21.15.32:0/1735376223 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4458160e20 con 0x7f445815fff0 2024-08-03T20:56:07.536 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:07.535+0000 7f4454c00640 1 -- 172.21.15.32:0/1735376223 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f4450005470 con 0x7f445815fff0 2024-08-03T20:56:07.537 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:07.536+0000 7f4454c00640 1 --2- 172.21.15.32:0/1735376223 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f4434077040 0x7f4434079550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:56:07.537 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:07.536+0000 7f4454c00640 1 -- 172.21.15.32:0/1735376223 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 6714+0+0 (secure 0 0 0) 0x7f4450098a00 con 0x7f445815fff0 2024-08-03T20:56:07.538 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:07.536+0000 7f4457e00640 1 --2- 172.21.15.32:0/1735376223 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f4434077040 0x7f4434079550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:56:07.538 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:07.537+0000 7f4457e00640 1 --2- 172.21.15.32:0/1735376223 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f4434077040 0x7f4434079550 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7f4448033000 tx=0x7f4448004730 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:56:07.541 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:07.540+0000 7f4454c00640 1 -- 172.21.15.32:0/1735376223 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7f44500611b0 con 0x7f445815fff0 2024-08-03T20:56:07.823 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:07.822+0000 7f445ea00640 1 -- 172.21.15.32:0/1735376223 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "mds repaired", "role": "0"} v 0) -- 0x7f44581f9510 con 0x7f445815fff0 2024-08-03T20:56:08.219 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-08-03T20:56:08.224 DEBUG:teuthology.orchestra.run.smithi118:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-08-03T20:56:08.271 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:08.269+0000 7f4454c00640 1 -- 172.21.15.32:0/1735376223 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "mds repaired", "role": "0"}]=0 repaired: restoring rank 6:0 v150) ==== 101+0+0 (secure 0 0 0) 0x7f4450065700 con 0x7f445815fff0 2024-08-03T20:56:08.271 INFO:teuthology.orchestra.run.smithi032.stderr:repaired: restoring rank 6:0 2024-08-03T20:56:08.274 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:08.272+0000 7f445ea00640 1 -- 172.21.15.32:0/1735376223 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f4434077040 msgr2=0x7f4434079550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:56:08.274 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:08.272+0000 7f445ea00640 1 --2- 172.21.15.32:0/1735376223 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f4434077040 0x7f4434079550 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7f4448033000 tx=0x7f4448004730 comp rx=0 tx=0).stop 2024-08-03T20:56:08.274 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:08.272+0000 7f445ea00640 1 -- 172.21.15.32:0/1735376223 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f445815fff0 msgr2=0x7f44580ccd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:56:08.274 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:08.272+0000 7f445ea00640 1 --2- 172.21.15.32:0/1735376223 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f445815fff0 0x7f44580ccd00 secure :-1 s=READY pgs=1307 cs=0 l=1 rev1=1 crypto rx=0x7f445000c760 tx=0x7f445000cb60 comp rx=0 tx=0).stop 2024-08-03T20:56:08.274 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:08.272+0000 7f445ea00640 1 -- 172.21.15.32:0/1735376223 shutdown_connections 2024-08-03T20:56:08.274 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:08.272+0000 7f445ea00640 1 --2- 172.21.15.32:0/1735376223 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f4434077040 0x7f4434079550 unknown :-1 s=CLOSED pgs=539 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:08.275 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:08.272+0000 7f445ea00640 1 --2- 172.21.15.32:0/1735376223 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f4458073e60 0x7f44581f8ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:08.275 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:08.272+0000 7f445ea00640 1 --2- 172.21.15.32:0/1735376223 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f445815fff0 0x7f44580ccd00 unknown :-1 s=CLOSED pgs=1307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:08.275 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:08.272+0000 7f445ea00640 1 --2- 172.21.15.32:0/1735376223 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f445806e930 0x7f44581f8300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:08.275 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:08.272+0000 7f445ea00640 1 -- 172.21.15.32:0/1735376223 >> 172.21.15.32:0/1735376223 conn(0x7f44580b1bd0 msgr2=0x7f4458140c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:56:08.275 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:08.272+0000 7f445ea00640 1 -- 172.21.15.32:0/1735376223 shutdown_connections 2024-08-03T20:56:08.275 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:08.272+0000 7f445ea00640 1 -- 172.21.15.32:0/1735376223 wait complete. 2024-08-03T20:56:08.302 DEBUG:teuthology.orchestra.run.smithi032:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata import - 2024-08-03T20:56:08.444 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:08.443+0000 7fea6022af40 1 Processor -- start 2024-08-03T20:56:08.445 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:08.443+0000 7fea6022af40 1 -- start start 2024-08-03T20:56:08.446 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:08.444+0000 7fea6022af40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55e321492360 0x55e3214a7bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:56:08.447 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:08.444+0000 7fea6022af40 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55e3214afcb0 0x55e321422000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:56:08.447 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:08.444+0000 7fea6022af40 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55e3214b03f0 0x55e321422540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:56:08.447 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:08.444+0000 7fea6022af40 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55e321329e30 con 0x55e3214b03f0 2024-08-03T20:56:08.447 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:08.444+0000 7fea6022af40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x55e321387bf0 con 0x55e321492360 2024-08-03T20:56:08.447 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:08.444+0000 7fea6022af40 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x55e321493440 con 0x55e3214afcb0 2024-08-03T20:56:08.447 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:08.445+0000 7fea59e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55e3214b03f0 0x55e321422540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:56:08.447 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:08.445+0000 7fea59e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55e3214b03f0 0x55e321422540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:42730/0 (socket says 172.21.15.32:42730) 2024-08-03T20:56:08.447 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:08.445+0000 7fea59e00640 1 -- 172.21.15.32:0/2357694590 learned_addr learned my addr 172.21.15.32:0/2357694590 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:56:08.447 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:08.445+0000 7fea59400640 1 --2- 172.21.15.32:0/2357694590 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55e321492360 0x55e3214a7bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:56:08.447 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:08.445+0000 7fea58a00640 1 --2- 172.21.15.32:0/2357694590 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55e3214afcb0 0x55e321422000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:56:08.447 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:08.445+0000 7fea59e00640 1 -- 172.21.15.32:0/2357694590 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55e3214afcb0 msgr2=0x55e321422000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:56:08.448 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:08.445+0000 7fea59e00640 1 --2- 172.21.15.32:0/2357694590 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55e3214afcb0 0x55e321422000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:08.448 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:08.445+0000 7fea59e00640 1 -- 172.21.15.32:0/2357694590 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55e321492360 msgr2=0x55e3214a7bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:56:08.448 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:08.445+0000 7fea59e00640 1 --2- 172.21.15.32:0/2357694590 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55e321492360 0x55e3214a7bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:08.448 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:08.445+0000 7fea59e00640 1 -- 172.21.15.32:0/2357694590 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e321423970 con 0x55e3214b03f0 2024-08-03T20:56:08.448 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:08.445+0000 7fea59e00640 1 --2- 172.21.15.32:0/2357694590 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55e3214b03f0 0x55e321422540 secure :-1 s=READY pgs=1310 cs=0 l=1 rev1=1 crypto rx=0x7fea5400b680 tx=0x7fea5400bb40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d12634ec8223b8e3 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:56:08.449 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:08.446+0000 7fea53400640 1 -- 172.21.15.32:0/2357694590 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea540099d0 con 0x55e3214b03f0 2024-08-03T20:56:08.449 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:08.446+0000 7fea53400640 1 -- 172.21.15.32:0/2357694590 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fea54009c80 con 0x55e3214b03f0 2024-08-03T20:56:08.449 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:08.446+0000 7fea53400640 1 -- 172.21.15.32:0/2357694590 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea5400e6b0 con 0x55e3214b03f0 2024-08-03T20:56:08.449 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:08.446+0000 7fea6022af40 1 -- 172.21.15.32:0/2357694590 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55e3214b03f0 msgr2=0x55e321422540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:56:08.449 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:08.446+0000 7fea6022af40 1 --2- 172.21.15.32:0/2357694590 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55e3214b03f0 0x55e321422540 secure :-1 s=READY pgs=1310 cs=0 l=1 rev1=1 crypto rx=0x7fea5400b680 tx=0x7fea5400bb40 comp rx=0 tx=0).stop 2024-08-03T20:56:08.450 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:08.446+0000 7fea6022af40 1 -- 172.21.15.32:0/2357694590 shutdown_connections 2024-08-03T20:56:08.450 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:08.446+0000 7fea6022af40 1 --2- 172.21.15.32:0/2357694590 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55e3214b03f0 0x55e321422540 unknown :-1 s=CLOSED pgs=1310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:08.450 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:08.446+0000 7fea6022af40 1 --2- 172.21.15.32:0/2357694590 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55e3214afcb0 0x55e321422000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:08.450 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:08.446+0000 7fea6022af40 1 --2- 172.21.15.32:0/2357694590 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55e321492360 0x55e3214a7bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:08.450 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:08.446+0000 7fea6022af40 1 -- 172.21.15.32:0/2357694590 >> 172.21.15.32:0/2357694590 conn(0x55e3212fbac0 msgr2=0x55e3214add60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:56:08.450 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:08.446+0000 7fea6022af40 1 -- 172.21.15.32:0/2357694590 shutdown_connections 2024-08-03T20:56:08.450 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:08.447+0000 7fea6022af40 1 -- 172.21.15.32:0/2357694590 wait complete. 2024-08-03T20:56:08.450 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:08.448+0000 7fea6022af40 1 Processor -- start 2024-08-03T20:56:08.450 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:08.448+0000 7fea6022af40 1 -- start start 2024-08-03T20:56:08.452 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:08.449+0000 7fea6022af40 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55e321492360 0x55e3214ac310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:56:08.452 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:08.449+0000 7fea6022af40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55e3214afcb0 0x55e321484670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:56:08.452 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:08.449+0000 7fea6022af40 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55e3214b03f0 0x55e321485f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:56:08.452 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:08.449+0000 7fea6022af40 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55e321490220 con 0x55e321492360 2024-08-03T20:56:08.452 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:08.449+0000 7fea6022af40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x55e3214aeac0 con 0x55e3214afcb0 2024-08-03T20:56:08.452 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:08.449+0000 7fea6022af40 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x55e321423690 con 0x55e3214b03f0 2024-08-03T20:56:08.453 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:08.449+0000 7fea59400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55e321492360 0x55e3214ac310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:56:08.453 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:08.449+0000 7fea59400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55e321492360 0x55e3214ac310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:42734/0 (socket says 172.21.15.32:42734) 2024-08-03T20:56:08.453 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:08.449+0000 7fea59400640 1 -- 172.21.15.32:0/1724914972 learned_addr learned my addr 172.21.15.32:0/1724914972 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:56:08.453 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:08.449+0000 7fea58a00640 1 --2- 172.21.15.32:0/1724914972 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55e3214afcb0 0x55e321484670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:56:08.453 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:08.450+0000 7fea59e00640 1 --2- 172.21.15.32:0/1724914972 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55e3214b03f0 0x55e321485f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:56:08.453 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:08.450+0000 7fea59400640 1 -- 172.21.15.32:0/1724914972 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55e3214b03f0 msgr2=0x55e321485f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:56:08.453 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:08.450+0000 7fea59400640 1 --2- 172.21.15.32:0/1724914972 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55e3214b03f0 0x55e321485f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:08.453 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:08.450+0000 7fea59400640 1 -- 172.21.15.32:0/1724914972 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55e3214afcb0 msgr2=0x55e321484670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:56:08.453 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:08.450+0000 7fea59400640 1 --2- 172.21.15.32:0/1724914972 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55e3214afcb0 0x55e321484670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:08.453 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:08.450+0000 7fea59400640 1 -- 172.21.15.32:0/1724914972 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e3214ae430 con 0x55e321492360 2024-08-03T20:56:08.453 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:08.450+0000 7fea59400640 1 --2- 172.21.15.32:0/1724914972 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55e321492360 0x55e3214ac310 secure :-1 s=READY pgs=1311 cs=0 l=1 rev1=1 crypto rx=0x7fea4c025fa0 tx=0x7fea4c056c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:56:08.453 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:08.450+0000 7fea52000640 1 -- 172.21.15.32:0/1724914972 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea4c067070 con 0x55e321492360 2024-08-03T20:56:08.454 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:08.450+0000 7fea52000640 1 -- 172.21.15.32:0/1724914972 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fea4c04d800 con 0x55e321492360 2024-08-03T20:56:08.454 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:08.450+0000 7fea52000640 1 -- 172.21.15.32:0/1724914972 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea4c004640 con 0x55e321492360 2024-08-03T20:56:08.454 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:08.450+0000 7fea6022af40 1 -- 172.21.15.32:0/1724914972 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x55e3214a9990 con 0x55e321492360 2024-08-03T20:56:08.454 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:08.451+0000 7fea6022af40 1 -- 172.21.15.32:0/1724914972 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e3214a9f50 con 0x55e321492360 2024-08-03T20:56:08.454 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:08.453+0000 7fea52000640 1 -- 172.21.15.32:0/1724914972 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7fea4c06a070 con 0x55e321492360 2024-08-03T20:56:08.455 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:08.453+0000 7fea52000640 1 --2- 172.21.15.32:0/1724914972 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fea3c075a20 0x7fea3c077f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:56:08.455 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:08.454+0000 7fea52000640 1 -- 172.21.15.32:0/1724914972 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 6714+0+0 (secure 0 0 0) 0x7fea4c0dbc80 con 0x55e321492360 2024-08-03T20:56:08.455 INFO:teuthology.orchestra.run.smithi032.stdout:Importing pool 2024-08-03T20:56:08.456 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:08.454+0000 7fea58a00640 1 --2- 172.21.15.32:0/1724914972 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fea3c075a20 0x7fea3c077f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:56:08.456 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:08.454+0000 7fea6022af40 1 --2- 172.21.15.32:0/1724914972 >> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] conn(0x55e321486450 0x55e3214a80e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:56:08.456 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:08.454+0000 7fea6022af40 1 -- 172.21.15.32:0/1724914972 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:1 12.10 12:09401a81:::601.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e95) -- 0x55e321552650 con 0x55e321486450 2024-08-03T20:56:08.456 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:08.454+0000 7fea59e00640 1 --2- 172.21.15.32:0/1724914972 >> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] conn(0x55e321486450 0x55e3214a80e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:56:08.456 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:08.455+0000 7fea58a00640 1 --2- 172.21.15.32:0/1724914972 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fea3c075a20 0x7fea3c077f30 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7fea48003ef0 tx=0x7fea480011f0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:56:08.457 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:08.455+0000 7fea59e00640 1 --2- 172.21.15.32:0/1724914972 >> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] conn(0x55e321486450 0x55e3214a80e0 crc :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:56:08.458 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:08.457+0000 7fea59e00640 1 -- 172.21.15.32:0/1724914972 <== osd.0 v2:172.21.15.32:6816/3066156116 1 ==== osd_op_reply(1 601.00000000 [create] v95'288 uv262 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fea54009c60 con 0x55e321486450 2024-08-03T20:56:08.458 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:08.457+0000 7fea6022af40 1 -- 172.21.15.32:0/1724914972 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:2 12.10 12:09401a81:::601.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) -- 0x55e3215529f0 con 0x55e321486450 2024-08-03T20:56:08.460 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:08.459+0000 7fea59e00640 1 -- 172.21.15.32:0/1724914972 <== osd.0 v2:172.21.15.32:6816/3066156116 2 ==== osd_op_reply(2 601.00000000 [delete] v95'289 uv289 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fea5401e040 con 0x55e321486450 2024-08-03T20:56:08.460 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:08.459+0000 7fea6022af40 1 -- 172.21.15.32:0/1724914972 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:3 12.10 12:09401a81:::601.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e95) -- 0x55e321486860 con 0x55e321486450 2024-08-03T20:56:08.461 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::601.00000000:head# 2024-08-03T20:56:08.462 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:08.460+0000 7fea59e00640 1 -- 172.21.15.32:0/1724914972 <== osd.0 v2:172.21.15.32:6816/3066156116 3 ==== osd_op_reply(3 601.00000000 [create] v95'290 uv290 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fea5401e040 con 0x55e321486450 2024-08-03T20:56:08.462 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:08.460+0000 7fea6022af40 1 -- 172.21.15.32:0/1724914972 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:4 12.10 12:09401a81:::601.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e95) -- 0x55e321553da0 con 0x55e321486450 2024-08-03T20:56:08.463 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:08.462+0000 7fea59e00640 1 -- 172.21.15.32:0/1724914972 <== osd.0 v2:172.21.15.32:6816/3066156116 4 ==== osd_op_reply(4 601.00000000 [setxattr (30)] v95'291 uv291 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fea5401e040 con 0x55e321486450 2024-08-03T20:56:08.463 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:08.462+0000 7fea6022af40 1 -- 172.21.15.32:0/1724914972 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:5 12.10 12:09401a81:::601.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e95) -- 0x55e321554140 con 0x55e321486450 2024-08-03T20:56:08.465 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:08.463+0000 7fea59e00640 1 -- 172.21.15.32:0/1724914972 <== osd.0 v2:172.21.15.32:6816/3066156116 5 ==== osd_op_reply(5 601.00000000 [setxattr (62)] v95'292 uv292 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fea5401e040 con 0x55e321486450 2024-08-03T20:56:08.465 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:08.464+0000 7fea6022af40 1 -- 172.21.15.32:0/1724914972 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:6 12.10 12:09401a81:::601.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e95) -- 0x55e321552d90 con 0x55e321486450 2024-08-03T20:56:08.467 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:08.465+0000 7fea59e00640 1 -- 172.21.15.32:0/1724914972 <== osd.0 v2:172.21.15.32:6816/3066156116 6 ==== osd_op_reply(6 601.00000000 [omap-set-header] v95'293 uv293 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fea5401e040 con 0x55e321486450 2024-08-03T20:56:08.467 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:08.465+0000 7fea6022af40 1 -- 172.21.15.32:0/1724914972 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:7 12.10 12:0bd6d154:::602.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e95) -- 0x55e321553130 con 0x55e321486450 2024-08-03T20:56:08.468 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:08.467+0000 7fea59e00640 1 -- 172.21.15.32:0/1724914972 <== osd.0 v2:172.21.15.32:6816/3066156116 7 ==== osd_op_reply(7 602.00000000 [create] v95'294 uv268 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fea5401e040 con 0x55e321486450 2024-08-03T20:56:08.469 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:08.467+0000 7fea6022af40 1 -- 172.21.15.32:0/1724914972 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:8 12.10 12:0bd6d154:::602.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) -- 0x55e3215534d0 con 0x55e321486450 2024-08-03T20:56:08.470 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:08.468+0000 7fea59e00640 1 -- 172.21.15.32:0/1724914972 <== osd.0 v2:172.21.15.32:6816/3066156116 8 ==== osd_op_reply(8 602.00000000 [delete] v95'295 uv295 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fea5401e040 con 0x55e321486450 2024-08-03T20:56:08.470 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:08.468+0000 7fea6022af40 1 -- 172.21.15.32:0/1724914972 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:9 12.10 12:0bd6d154:::602.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e95) -- 0x55e3215538c0 con 0x55e321486450 2024-08-03T20:56:08.471 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::602.00000000:head# 2024-08-03T20:56:08.472 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:08.470+0000 7fea59e00640 1 -- 172.21.15.32:0/1724914972 <== osd.0 v2:172.21.15.32:6816/3066156116 9 ==== osd_op_reply(9 602.00000000 [create] v95'296 uv296 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fea5401e040 con 0x55e321486450 2024-08-03T20:56:08.472 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:08.470+0000 7fea6022af40 1 -- 172.21.15.32:0/1724914972 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:10 12.10 12:0bd6d154:::602.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e95) -- 0x55e3215538c0 con 0x55e321486450 2024-08-03T20:56:08.473 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:08.471+0000 7fea59e00640 1 -- 172.21.15.32:0/1724914972 <== osd.0 v2:172.21.15.32:6816/3066156116 10 ==== osd_op_reply(10 602.00000000 [setxattr (30)] v95'297 uv297 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fea5401e040 con 0x55e321486450 2024-08-03T20:56:08.473 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:08.472+0000 7fea6022af40 1 -- 172.21.15.32:0/1724914972 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:11 12.10 12:0bd6d154:::602.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e95) -- 0x55e321553870 con 0x55e321486450 2024-08-03T20:56:08.475 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:08.473+0000 7fea59e00640 1 -- 172.21.15.32:0/1724914972 <== osd.0 v2:172.21.15.32:6816/3066156116 11 ==== osd_op_reply(11 602.00000000 [setxattr (62)] v95'298 uv298 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fea5401e040 con 0x55e321486450 2024-08-03T20:56:08.475 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:08.473+0000 7fea6022af40 1 -- 172.21.15.32:0/1724914972 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:12 12.10 12:0bd6d154:::602.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e95) -- 0x55e321553870 con 0x55e321486450 2024-08-03T20:56:08.476 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:08.475+0000 7fea59e00640 1 -- 172.21.15.32:0/1724914972 <== osd.0 v2:172.21.15.32:6816/3066156116 12 ==== osd_op_reply(12 602.00000000 [omap-set-header] v95'299 uv299 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fea5401e040 con 0x55e321486450 2024-08-03T20:56:08.476 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:08.475+0000 7fea6022af40 1 -- 172.21.15.32:0/1724914972 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:13 12.10 12:0cf6ad0f:::10000000000.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e95) -- 0x55e321553870 con 0x55e321486450 2024-08-03T20:56:08.478 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:08.476+0000 7fea59e00640 1 -- 172.21.15.32:0/1724914972 <== osd.0 v2:172.21.15.32:6816/3066156116 13 ==== osd_op_reply(13 10000000000.00000000 [create] v95'300 uv275 ondisk = -17 ((17) File exists)) ==== 164+0+0 (crc 0 0 0) 0x7fea5401e040 con 0x55e321486450 2024-08-03T20:56:08.478 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:08.476+0000 7fea6022af40 1 -- 172.21.15.32:0/1724914972 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:14 12.10 12:0cf6ad0f:::10000000000.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) -- 0x55e321553870 con 0x55e321486450 2024-08-03T20:56:08.479 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:08.478+0000 7fea59e00640 1 -- 172.21.15.32:0/1724914972 <== osd.0 v2:172.21.15.32:6816/3066156116 14 ==== osd_op_reply(14 10000000000.00000000 [delete] v95'301 uv301 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7fea5401e040 con 0x55e321486450 2024-08-03T20:56:08.479 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:08.478+0000 7fea6022af40 1 -- 172.21.15.32:0/1724914972 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:15 12.10 12:0cf6ad0f:::10000000000.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e95) -- 0x55e321553870 con 0x55e321486450 2024-08-03T20:56:08.481 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::10000000000.00000000:head# 2024-08-03T20:56:08.481 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:08.480+0000 7fea59e00640 1 -- 172.21.15.32:0/1724914972 <== osd.0 v2:172.21.15.32:6816/3066156116 15 ==== osd_op_reply(15 10000000000.00000000 [create] v95'302 uv302 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7fea5401e040 con 0x55e321486450 2024-08-03T20:56:08.481 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:08.480+0000 7fea6022af40 1 -- 172.21.15.32:0/1724914972 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:16 12.10 12:0cf6ad0f:::10000000000.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e95) -- 0x55e321553870 con 0x55e321486450 2024-08-03T20:56:08.482 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:08.481+0000 7fea59e00640 1 -- 172.21.15.32:0/1724914972 <== osd.0 v2:172.21.15.32:6816/3066156116 16 ==== osd_op_reply(16 10000000000.00000000 [setxattr (30)] v95'303 uv303 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7fea5401e040 con 0x55e321486450 2024-08-03T20:56:08.482 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:08.481+0000 7fea6022af40 1 -- 172.21.15.32:0/1724914972 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:17 12.10 12:0cf6ad0f:::10000000000.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e95) -- 0x55e321553870 con 0x55e321486450 2024-08-03T20:56:08.484 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:08.482+0000 7fea59e00640 1 -- 172.21.15.32:0/1724914972 <== osd.0 v2:172.21.15.32:6816/3066156116 17 ==== osd_op_reply(17 10000000000.00000000 [setxattr (62)] v95'304 uv304 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7fea5401e040 con 0x55e321486450 2024-08-03T20:56:08.484 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:08.483+0000 7fea6022af40 1 -- 172.21.15.32:0/1724914972 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:18 12.10 12:0cf6ad0f:::10000000000.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e95) -- 0x55e321553870 con 0x55e321486450 2024-08-03T20:56:08.486 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:08.484+0000 7fea59e00640 1 -- 172.21.15.32:0/1724914972 <== osd.0 v2:172.21.15.32:6816/3066156116 18 ==== osd_op_reply(18 10000000000.00000000 [omap-set-header] v95'305 uv305 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7fea5401e040 con 0x55e321486450 2024-08-03T20:56:08.486 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:08.484+0000 7fea6022af40 1 -- 172.21.15.32:0/1724914972 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:19 12.10 12:0cf6ad0f:::10000000000.00000000:head [omap-set-vals in=593b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e95) -- 0x55e321553870 con 0x55e321486450 2024-08-03T20:56:08.487 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:08.486+0000 7fea59e00640 1 -- 172.21.15.32:0/1724914972 <== osd.0 v2:172.21.15.32:6816/3066156116 19 ==== osd_op_reply(19 10000000000.00000000 [omap-set-vals] v95'306 uv306 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7fea5401e040 con 0x55e321486450 2024-08-03T20:56:08.487 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:08.486+0000 7fea6022af40 1 -- 172.21.15.32:0/1724914972 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:20 12.10 12:0d82a743:::600.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e95) -- 0x55e321553870 con 0x55e321486450 2024-08-03T20:56:08.489 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:08.487+0000 7fea59e00640 1 -- 172.21.15.32:0/1724914972 <== osd.0 v2:172.21.15.32:6816/3066156116 20 ==== osd_op_reply(20 600.00000000 [create] v95'307 uv281 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fea5401e040 con 0x55e321486450 2024-08-03T20:56:08.489 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:08.487+0000 7fea6022af40 1 -- 172.21.15.32:0/1724914972 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:21 12.10 12:0d82a743:::600.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) -- 0x55e321553870 con 0x55e321486450 2024-08-03T20:56:08.490 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:08.489+0000 7fea59e00640 1 -- 172.21.15.32:0/1724914972 <== osd.0 v2:172.21.15.32:6816/3066156116 21 ==== osd_op_reply(21 600.00000000 [delete] v95'308 uv308 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fea5401e040 con 0x55e321486450 2024-08-03T20:56:08.491 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:08.489+0000 7fea6022af40 1 -- 172.21.15.32:0/1724914972 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:22 12.10 12:0d82a743:::600.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e95) -- 0x55e321553870 con 0x55e321486450 2024-08-03T20:56:08.492 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:08.491+0000 7fea59e00640 1 -- 172.21.15.32:0/1724914972 <== osd.0 v2:172.21.15.32:6816/3066156116 22 ==== osd_op_reply(22 600.00000000 [create] v95'309 uv309 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fea5401e040 con 0x55e321486450 2024-08-03T20:56:08.492 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::600.00000000:head# 2024-08-03T20:56:08.492 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:08.491+0000 7fea6022af40 1 -- 172.21.15.32:0/1724914972 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:23 12.10 12:0d82a743:::600.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e95) -- 0x55e321553870 con 0x55e321486450 2024-08-03T20:56:08.493 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:08.492+0000 7fea59e00640 1 -- 172.21.15.32:0/1724914972 <== osd.0 v2:172.21.15.32:6816/3066156116 23 ==== osd_op_reply(23 600.00000000 [setxattr (30)] v95'310 uv310 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fea5401e040 con 0x55e321486450 2024-08-03T20:56:08.494 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:08.492+0000 7fea6022af40 1 -- 172.21.15.32:0/1724914972 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:24 12.10 12:0d82a743:::600.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e95) -- 0x55e321553870 con 0x55e321486450 2024-08-03T20:56:08.495 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:08.494+0000 7fea59e00640 1 -- 172.21.15.32:0/1724914972 <== osd.0 v2:172.21.15.32:6816/3066156116 24 ==== osd_op_reply(24 600.00000000 [setxattr (62)] v95'311 uv311 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fea5401e040 con 0x55e321486450 2024-08-03T20:56:08.495 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:08.494+0000 7fea6022af40 1 -- 172.21.15.32:0/1724914972 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:25 12.10 12:0d82a743:::600.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e95) -- 0x55e321553870 con 0x55e321486450 2024-08-03T20:56:08.497 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:08.495+0000 7fea59e00640 1 -- 172.21.15.32:0/1724914972 <== osd.0 v2:172.21.15.32:6816/3066156116 25 ==== osd_op_reply(25 600.00000000 [omap-set-header] v95'312 uv312 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fea5401e040 con 0x55e321486450 2024-08-03T20:56:08.497 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:08.495+0000 7fea6022af40 1 -- 172.21.15.32:0/1724914972 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:26 12.10 12:0d89b25e:::603.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e95) -- 0x55e321553870 con 0x55e321486450 2024-08-03T20:56:08.498 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:08.497+0000 7fea59e00640 1 -- 172.21.15.32:0/1724914972 <== osd.0 v2:172.21.15.32:6816/3066156116 26 ==== osd_op_reply(26 603.00000000 [create] v95'313 uv287 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fea5401e040 con 0x55e321486450 2024-08-03T20:56:08.498 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:08.497+0000 7fea6022af40 1 -- 172.21.15.32:0/1724914972 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:27 12.10 12:0d89b25e:::603.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) -- 0x55e321553870 con 0x55e321486450 2024-08-03T20:56:08.500 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:08.498+0000 7fea59e00640 1 -- 172.21.15.32:0/1724914972 <== osd.0 v2:172.21.15.32:6816/3066156116 27 ==== osd_op_reply(27 603.00000000 [delete] v95'314 uv314 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fea5401e040 con 0x55e321486450 2024-08-03T20:56:08.500 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:08.498+0000 7fea6022af40 1 -- 172.21.15.32:0/1724914972 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:28 12.10 12:0d89b25e:::603.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e95) -- 0x55e321553870 con 0x55e321486450 2024-08-03T20:56:08.502 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::603.00000000:head# 2024-08-03T20:56:08.502 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:08.500+0000 7fea59e00640 1 -- 172.21.15.32:0/1724914972 <== osd.0 v2:172.21.15.32:6816/3066156116 28 ==== osd_op_reply(28 603.00000000 [create] v95'315 uv315 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fea5401e040 con 0x55e321486450 2024-08-03T20:56:08.502 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:08.500+0000 7fea6022af40 1 -- 172.21.15.32:0/1724914972 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:29 12.10 12:0d89b25e:::603.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e95) -- 0x55e321553870 con 0x55e321486450 2024-08-03T20:56:08.503 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:08.502+0000 7fea59e00640 1 -- 172.21.15.32:0/1724914972 <== osd.0 v2:172.21.15.32:6816/3066156116 29 ==== osd_op_reply(29 603.00000000 [setxattr (30)] v95'316 uv316 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fea5401e040 con 0x55e321486450 2024-08-03T20:56:08.503 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:08.502+0000 7fea6022af40 1 -- 172.21.15.32:0/1724914972 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:30 12.10 12:0d89b25e:::603.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e95) -- 0x55e321553870 con 0x55e321486450 2024-08-03T20:56:08.504 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:08.503+0000 7fea59e00640 1 -- 172.21.15.32:0/1724914972 <== osd.0 v2:172.21.15.32:6816/3066156116 30 ==== osd_op_reply(30 603.00000000 [setxattr (62)] v95'317 uv317 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fea5401e040 con 0x55e321486450 2024-08-03T20:56:08.505 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:08.503+0000 7fea6022af40 1 -- 172.21.15.32:0/1724914972 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:31 12.10 12:0d89b25e:::603.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e95) -- 0x55e321553870 con 0x55e321486450 2024-08-03T20:56:08.506 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:08.505+0000 7fea59e00640 1 -- 172.21.15.32:0/1724914972 <== osd.0 v2:172.21.15.32:6816/3066156116 31 ==== osd_op_reply(31 603.00000000 [omap-set-header] v95'318 uv318 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fea5401e040 con 0x55e321486450 2024-08-03T20:56:08.506 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:08.505+0000 7fea6022af40 1 --2- 172.21.15.32:0/1724914972 >> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] conn(0x55e3215549f0 0x55e321556e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:56:08.507 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:08.505+0000 7fea6022af40 1 -- 172.21.15.32:0/1724914972 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:32 12.14 12:287034c4:::1.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e95) -- 0x55e321553870 con 0x55e3215549f0 2024-08-03T20:56:08.507 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:08.505+0000 7fea59400640 1 --2- 172.21.15.32:0/1724914972 >> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] conn(0x55e3215549f0 0x55e321556e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:56:08.507 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:08.506+0000 7fea59400640 1 --2- 172.21.15.32:0/1724914972 >> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] conn(0x55e3215549f0 0x55e321556e10 crc :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:56:08.509 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:08.508+0000 7fea59400640 1 -- 172.21.15.32:0/1724914972 <== osd.5 v2:172.21.15.118:6810/2724891770 1 ==== osd_op_reply(32 1.00000000.inode [create] v95'219 uv202 ondisk = -17 ((17) File exists)) ==== 160+0+0 (crc 0 0 0) 0x7fea4c069030 con 0x55e3215549f0 2024-08-03T20:56:08.509 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:08.508+0000 7fea6022af40 1 -- 172.21.15.32:0/1724914972 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:33 12.14 12:287034c4:::1.00000000.inode:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) -- 0x55e321557350 con 0x55e3215549f0 2024-08-03T20:56:08.511 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:08.509+0000 7fea59400640 1 -- 172.21.15.32:0/1724914972 <== osd.5 v2:172.21.15.118:6810/2724891770 2 ==== osd_op_reply(33 1.00000000.inode [delete] v95'220 uv220 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7fea4c069030 con 0x55e3215549f0 2024-08-03T20:56:08.511 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:08.509+0000 7fea6022af40 1 -- 172.21.15.32:0/1724914972 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:34 12.14 12:287034c4:::1.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e95) -- 0x55e3215576f0 con 0x55e3215549f0 2024-08-03T20:56:08.512 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::1.00000000.inode:head# 2024-08-03T20:56:08.513 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:08.511+0000 7fea59400640 1 -- 172.21.15.32:0/1724914972 <== osd.5 v2:172.21.15.118:6810/2724891770 3 ==== osd_op_reply(34 1.00000000.inode [create] v95'221 uv221 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7fea4c069030 con 0x55e3215549f0 2024-08-03T20:56:08.513 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:08.511+0000 7fea6022af40 1 -- 172.21.15.32:0/1724914972 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:35 12.14 12:287034c4:::1.00000000.inode:head [write 0~582 in=582b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e95) -- 0x55e321557a90 con 0x55e3215549f0 2024-08-03T20:56:08.514 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:08.513+0000 7fea59400640 1 -- 172.21.15.32:0/1724914972 <== osd.5 v2:172.21.15.118:6810/2724891770 4 ==== osd_op_reply(35 1.00000000.inode [write 0~582] v95'222 uv222 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7fea4c069030 con 0x55e3215549f0 2024-08-03T20:56:08.514 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:08.513+0000 7fea6022af40 1 -- 172.21.15.32:0/1724914972 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:36 12.14 12:287034c4:::1.00000000.inode:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e95) -- 0x55e321557e30 con 0x55e3215549f0 2024-08-03T20:56:08.516 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:08.514+0000 7fea59400640 1 -- 172.21.15.32:0/1724914972 <== osd.5 v2:172.21.15.118:6810/2724891770 5 ==== osd_op_reply(36 1.00000000.inode [omap-set-header] v95'223 uv223 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7fea4c069030 con 0x55e3215549f0 2024-08-03T20:56:08.516 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:08.514+0000 7fea6022af40 1 -- 172.21.15.32:0/1724914972 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:37 12.14 12:292cf221:::200.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e95) -- 0x55e321557e30 con 0x55e3215549f0 2024-08-03T20:56:08.517 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:08.516+0000 7fea59400640 1 -- 172.21.15.32:0/1724914972 <== osd.5 v2:172.21.15.118:6810/2724891770 6 ==== osd_op_reply(37 200.00000000 [create] v95'224 uv207 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fea4c069030 con 0x55e3215549f0 2024-08-03T20:56:08.517 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:08.516+0000 7fea6022af40 1 -- 172.21.15.32:0/1724914972 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:38 12.14 12:292cf221:::200.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) -- 0x55e321557e30 con 0x55e3215549f0 2024-08-03T20:56:08.519 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:08.517+0000 7fea59400640 1 -- 172.21.15.32:0/1724914972 <== osd.5 v2:172.21.15.118:6810/2724891770 7 ==== osd_op_reply(38 200.00000000 [delete] v95'225 uv225 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fea4c069030 con 0x55e3215549f0 2024-08-03T20:56:08.519 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:08.517+0000 7fea6022af40 1 -- 172.21.15.32:0/1724914972 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:39 12.14 12:292cf221:::200.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e95) -- 0x55e321557e30 con 0x55e3215549f0 2024-08-03T20:56:08.520 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::200.00000000:head# 2024-08-03T20:56:08.520 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:08.519+0000 7fea59400640 1 -- 172.21.15.32:0/1724914972 <== osd.5 v2:172.21.15.118:6810/2724891770 8 ==== osd_op_reply(39 200.00000000 [create] v95'226 uv226 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fea4c069030 con 0x55e3215549f0 2024-08-03T20:56:08.520 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:08.519+0000 7fea6022af40 1 -- 172.21.15.32:0/1724914972 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:40 12.14 12:292cf221:::200.00000000:head [write 0~90 in=90b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e95) -- 0x55e321557e30 con 0x55e3215549f0 2024-08-03T20:56:08.522 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:08.520+0000 7fea59400640 1 -- 172.21.15.32:0/1724914972 <== osd.5 v2:172.21.15.118:6810/2724891770 9 ==== osd_op_reply(40 200.00000000 [write 0~90] v95'227 uv227 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fea4c069030 con 0x55e3215549f0 2024-08-03T20:56:08.522 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:08.520+0000 7fea6022af40 1 -- 172.21.15.32:0/1724914972 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:41 12.14 12:292cf221:::200.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e95) -- 0x55e321557e30 con 0x55e3215549f0 2024-08-03T20:56:08.523 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:08.522+0000 7fea59400640 1 -- 172.21.15.32:0/1724914972 <== osd.5 v2:172.21.15.118:6810/2724891770 10 ==== osd_op_reply(41 200.00000000 [omap-set-header] v95'228 uv228 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fea4c069030 con 0x55e3215549f0 2024-08-03T20:56:08.523 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:08.522+0000 7fea6022af40 1 -- 172.21.15.32:0/1724914972 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:42 12.14 12:2e2fa760:::200.00000001:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e95) -- 0x55e321557e30 con 0x55e3215549f0 2024-08-03T20:56:08.525 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:08.523+0000 7fea59400640 1 -- 172.21.15.32:0/1724914972 <== osd.5 v2:172.21.15.118:6810/2724891770 11 ==== osd_op_reply(42 200.00000001 [create] v95'229 uv212 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fea4c069030 con 0x55e3215549f0 2024-08-03T20:56:08.525 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:08.523+0000 7fea6022af40 1 -- 172.21.15.32:0/1724914972 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:43 12.14 12:2e2fa760:::200.00000001:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) -- 0x55e321557e30 con 0x55e3215549f0 2024-08-03T20:56:08.527 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:08.525+0000 7fea59400640 1 -- 172.21.15.32:0/1724914972 <== osd.5 v2:172.21.15.118:6810/2724891770 12 ==== osd_op_reply(43 200.00000001 [delete] v95'230 uv230 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fea4c069030 con 0x55e3215549f0 2024-08-03T20:56:08.527 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:08.525+0000 7fea6022af40 1 -- 172.21.15.32:0/1724914972 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:44 12.14 12:2e2fa760:::200.00000001:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e95) -- 0x55e321557e30 con 0x55e3215549f0 2024-08-03T20:56:08.528 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:08.526+0000 7fea59400640 1 -- 172.21.15.32:0/1724914972 <== osd.5 v2:172.21.15.118:6810/2724891770 13 ==== osd_op_reply(44 200.00000001 [create] v95'231 uv231 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fea4c069030 con 0x55e3215549f0 2024-08-03T20:56:08.528 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::200.00000001:head# 2024-08-03T20:56:08.528 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:08.526+0000 7fea6022af40 1 -- 172.21.15.32:0/1724914972 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:45 12.14 12:2e2fa760:::200.00000001:head [write 0~18118 in=18118b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e95) -- 0x55e32155c520 con 0x55e3215549f0 2024-08-03T20:56:08.530 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:08.528+0000 7fea59400640 1 -- 172.21.15.32:0/1724914972 <== osd.5 v2:172.21.15.118:6810/2724891770 14 ==== osd_op_reply(45 200.00000001 [write 0~18118] v95'232 uv232 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fea4c069030 con 0x55e3215549f0 2024-08-03T20:56:08.530 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:08.528+0000 7fea6022af40 1 -- 172.21.15.32:0/1724914972 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:46 12.14 12:2e2fa760:::200.00000001:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e95) -- 0x55e32155c520 con 0x55e3215549f0 2024-08-03T20:56:08.531 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:08.530+0000 7fea59400640 1 -- 172.21.15.32:0/1724914972 <== osd.5 v2:172.21.15.118:6810/2724891770 15 ==== osd_op_reply(46 200.00000001 [omap-set-header] v95'233 uv233 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fea4c069030 con 0x55e3215549f0 2024-08-03T20:56:08.531 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:08.530+0000 7fea6022af40 1 -- 172.21.15.32:0/1724914972 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:47 12.14 12:2f55791f:::606.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e95) -- 0x55e321557e30 con 0x55e3215549f0 2024-08-03T20:56:08.533 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:08.531+0000 7fea59400640 1 -- 172.21.15.32:0/1724914972 <== osd.5 v2:172.21.15.118:6810/2724891770 16 ==== osd_op_reply(47 606.00000000 [create] v95'234 uv218 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fea4c069030 con 0x55e3215549f0 2024-08-03T20:56:08.533 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:08.531+0000 7fea6022af40 1 -- 172.21.15.32:0/1724914972 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:48 12.14 12:2f55791f:::606.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) -- 0x55e321557e30 con 0x55e3215549f0 2024-08-03T20:56:08.534 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:08.533+0000 7fea59400640 1 -- 172.21.15.32:0/1724914972 <== osd.5 v2:172.21.15.118:6810/2724891770 17 ==== osd_op_reply(48 606.00000000 [delete] v95'235 uv235 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fea4c069030 con 0x55e3215549f0 2024-08-03T20:56:08.534 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:08.533+0000 7fea6022af40 1 -- 172.21.15.32:0/1724914972 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:49 12.14 12:2f55791f:::606.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e95) -- 0x55e321557e30 con 0x55e3215549f0 2024-08-03T20:56:08.536 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::606.00000000:head# 2024-08-03T20:56:08.536 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:08.534+0000 7fea59400640 1 -- 172.21.15.32:0/1724914972 <== osd.5 v2:172.21.15.118:6810/2724891770 18 ==== osd_op_reply(49 606.00000000 [create] v95'236 uv236 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fea4c069030 con 0x55e3215549f0 2024-08-03T20:56:08.536 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:08.534+0000 7fea6022af40 1 -- 172.21.15.32:0/1724914972 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:50 12.14 12:2f55791f:::606.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e95) -- 0x55e321558e40 con 0x55e3215549f0 2024-08-03T20:56:08.537 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:08.536+0000 7fea59400640 1 -- 172.21.15.32:0/1724914972 <== osd.5 v2:172.21.15.118:6810/2724891770 19 ==== osd_op_reply(50 606.00000000 [setxattr (30)] v95'237 uv237 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fea4c069030 con 0x55e3215549f0 2024-08-03T20:56:08.537 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:08.536+0000 7fea6022af40 1 -- 172.21.15.32:0/1724914972 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:51 12.14 12:2f55791f:::606.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e95) -- 0x55e321558e40 con 0x55e3215549f0 2024-08-03T20:56:08.539 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:08.537+0000 7fea59400640 1 -- 172.21.15.32:0/1724914972 <== osd.5 v2:172.21.15.118:6810/2724891770 20 ==== osd_op_reply(51 606.00000000 [setxattr (62)] v95'238 uv238 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fea4c069030 con 0x55e3215549f0 2024-08-03T20:56:08.539 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:08.537+0000 7fea6022af40 1 -- 172.21.15.32:0/1724914972 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:52 12.14 12:2f55791f:::606.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e95) -- 0x55e321557e30 con 0x55e3215549f0 2024-08-03T20:56:08.540 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:08.539+0000 7fea59400640 1 -- 172.21.15.32:0/1724914972 <== osd.5 v2:172.21.15.118:6810/2724891770 21 ==== osd_op_reply(52 606.00000000 [omap-set-header] v95'239 uv239 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fea4c069030 con 0x55e3215549f0 2024-08-03T20:56:08.540 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:08.539+0000 7fea6022af40 1 -- 172.21.15.32:0/1724914972 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:53 12.1c 12:3ed09add:::607.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e95) -- 0x55e321557e30 con 0x55e321486450 2024-08-03T20:56:08.542 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:08.541+0000 7fea59e00640 1 -- 172.21.15.32:0/1724914972 <== osd.0 v2:172.21.15.32:6816/3066156116 32 ==== osd_op_reply(53 607.00000000 [create] v95'57 uv56 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fea5401e040 con 0x55e321486450 2024-08-03T20:56:08.542 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:08.541+0000 7fea6022af40 1 -- 172.21.15.32:0/1724914972 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:54 12.1c 12:3ed09add:::607.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) -- 0x55e321557e30 con 0x55e321486450 2024-08-03T20:56:08.543 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:08.542+0000 7fea59e00640 1 -- 172.21.15.32:0/1724914972 <== osd.0 v2:172.21.15.32:6816/3066156116 33 ==== osd_op_reply(54 607.00000000 [delete] v95'58 uv58 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fea5401e040 con 0x55e321486450 2024-08-03T20:56:08.544 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:08.542+0000 7fea6022af40 1 -- 172.21.15.32:0/1724914972 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:55 12.1c 12:3ed09add:::607.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e95) -- 0x55e321557e30 con 0x55e321486450 2024-08-03T20:56:08.545 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::607.00000000:head# 2024-08-03T20:56:08.546 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:08.544+0000 7fea59e00640 1 -- 172.21.15.32:0/1724914972 <== osd.0 v2:172.21.15.32:6816/3066156116 34 ==== osd_op_reply(55 607.00000000 [create] v95'59 uv59 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fea5401e040 con 0x55e321486450 2024-08-03T20:56:08.546 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:08.544+0000 7fea6022af40 1 -- 172.21.15.32:0/1724914972 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:56 12.1c 12:3ed09add:::607.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e95) -- 0x55e321557e30 con 0x55e321486450 2024-08-03T20:56:08.547 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:08.545+0000 7fea59e00640 1 -- 172.21.15.32:0/1724914972 <== osd.0 v2:172.21.15.32:6816/3066156116 35 ==== osd_op_reply(56 607.00000000 [setxattr (30)] v95'60 uv60 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fea5401e040 con 0x55e321486450 2024-08-03T20:56:08.547 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:08.545+0000 7fea6022af40 1 -- 172.21.15.32:0/1724914972 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:57 12.1c 12:3ed09add:::607.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e95) -- 0x55e321557e30 con 0x55e321486450 2024-08-03T20:56:08.548 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:08.546+0000 7fea59e00640 1 -- 172.21.15.32:0/1724914972 <== osd.0 v2:172.21.15.32:6816/3066156116 36 ==== osd_op_reply(57 607.00000000 [setxattr (62)] v95'61 uv61 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fea5401e040 con 0x55e321486450 2024-08-03T20:56:08.548 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:08.546+0000 7fea6022af40 1 -- 172.21.15.32:0/1724914972 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:58 12.1c 12:3ed09add:::607.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e95) -- 0x55e321557e30 con 0x55e321486450 2024-08-03T20:56:08.549 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:08.548+0000 7fea59e00640 1 -- 172.21.15.32:0/1724914972 <== osd.0 v2:172.21.15.32:6816/3066156116 37 ==== osd_op_reply(58 607.00000000 [omap-set-header] v95'62 uv62 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fea5401e040 con 0x55e321486450 2024-08-03T20:56:08.549 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:08.548+0000 7fea6022af40 1 --2- 172.21.15.32:0/1724914972 >> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] conn(0x55e321558780 0x55e321558b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:56:08.550 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:08.548+0000 7fea6022af40 1 -- 172.21.15.32:0/1724914972 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:59 12.6 12:654134d2:::mds0_openfiles.0:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e95) -- 0x55e32155b240 con 0x55e321558780 2024-08-03T20:56:08.550 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:08.548+0000 7fea58a00640 1 --2- 172.21.15.32:0/1724914972 >> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] conn(0x55e321558780 0x55e321558b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:56:08.550 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:08.549+0000 7fea58a00640 1 --2- 172.21.15.32:0/1724914972 >> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] conn(0x55e321558780 0x55e321558b90 crc :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:56:08.551 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:08.550+0000 7fea58a00640 1 -- 172.21.15.32:0/1724914972 <== osd.1 v2:172.21.15.32:6824/1551241759 1 ==== osd_op_reply(59 mds0_openfiles.0 [create] v95'51 uv50 ondisk = -17 ((17) File exists)) ==== 160+0+0 (crc 0 0 0) 0x7fea4801b040 con 0x55e321558780 2024-08-03T20:56:08.551 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:08.550+0000 7fea6022af40 1 -- 172.21.15.32:0/1724914972 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:60 12.6 12:654134d2:::mds0_openfiles.0:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) -- 0x55e32155b5e0 con 0x55e321558780 2024-08-03T20:56:08.553 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:08.551+0000 7fea58a00640 1 -- 172.21.15.32:0/1724914972 <== osd.1 v2:172.21.15.32:6824/1551241759 2 ==== osd_op_reply(60 mds0_openfiles.0 [delete] v95'52 uv52 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7fea4801b040 con 0x55e321558780 2024-08-03T20:56:08.553 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:08.551+0000 7fea6022af40 1 -- 172.21.15.32:0/1724914972 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:61 12.6 12:654134d2:::mds0_openfiles.0:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e95) -- 0x55e32155baa0 con 0x55e321558780 2024-08-03T20:56:08.554 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds0_openfiles.0:head# 2024-08-03T20:56:08.554 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:08.553+0000 7fea58a00640 1 -- 172.21.15.32:0/1724914972 <== osd.1 v2:172.21.15.32:6824/1551241759 3 ==== osd_op_reply(61 mds0_openfiles.0 [create] v95'53 uv53 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7fea4801b040 con 0x55e321558780 2024-08-03T20:56:08.554 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:08.553+0000 7fea6022af40 1 -- 172.21.15.32:0/1724914972 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:62 12.6 12:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e95) -- 0x55e321485580 con 0x55e321558780 2024-08-03T20:56:08.555 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:08.554+0000 7fea58a00640 1 -- 172.21.15.32:0/1724914972 <== osd.1 v2:172.21.15.32:6824/1551241759 4 ==== osd_op_reply(62 mds0_openfiles.0 [omap-set-header] v95'54 uv54 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7fea4801b040 con 0x55e321558780 2024-08-03T20:56:08.555 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:08.554+0000 7fea6022af40 1 -- 172.21.15.32:0/1724914972 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:63 12.e 12:717a0223:::608.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e95) -- 0x55e32155c3d0 con 0x55e321486450 2024-08-03T20:56:08.557 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:08.556+0000 7fea59e00640 1 -- 172.21.15.32:0/1724914972 <== osd.0 v2:172.21.15.32:6816/3066156116 38 ==== osd_op_reply(63 608.00000000 [create] v95'57 uv56 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fea5401e040 con 0x55e321486450 2024-08-03T20:56:08.557 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:08.556+0000 7fea6022af40 1 -- 172.21.15.32:0/1724914972 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:64 12.e 12:717a0223:::608.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) -- 0x55e32155c3d0 con 0x55e321486450 2024-08-03T20:56:08.558 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:08.557+0000 7fea59e00640 1 -- 172.21.15.32:0/1724914972 <== osd.0 v2:172.21.15.32:6816/3066156116 39 ==== osd_op_reply(64 608.00000000 [delete] v95'58 uv58 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fea5401e040 con 0x55e321486450 2024-08-03T20:56:08.559 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:08.557+0000 7fea6022af40 1 -- 172.21.15.32:0/1724914972 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:65 12.e 12:717a0223:::608.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e95) -- 0x55e32155c3d0 con 0x55e321486450 2024-08-03T20:56:08.560 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:08.559+0000 7fea59e00640 1 -- 172.21.15.32:0/1724914972 <== osd.0 v2:172.21.15.32:6816/3066156116 40 ==== osd_op_reply(65 608.00000000 [create] v95'59 uv59 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fea5401e040 con 0x55e321486450 2024-08-03T20:56:08.560 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::608.00000000:head# 2024-08-03T20:56:08.561 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:08.559+0000 7fea6022af40 1 -- 172.21.15.32:0/1724914972 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:66 12.e 12:717a0223:::608.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e95) -- 0x55e32155d3e0 con 0x55e321486450 2024-08-03T20:56:08.561 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:08.560+0000 7fea59e00640 1 -- 172.21.15.32:0/1724914972 <== osd.0 v2:172.21.15.32:6816/3066156116 41 ==== osd_op_reply(66 608.00000000 [setxattr (30)] v95'60 uv60 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fea5401e040 con 0x55e321486450 2024-08-03T20:56:08.561 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:08.560+0000 7fea6022af40 1 -- 172.21.15.32:0/1724914972 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:67 12.e 12:717a0223:::608.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e95) -- 0x55e32155d3e0 con 0x55e321486450 2024-08-03T20:56:08.563 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:08.562+0000 7fea59e00640 1 -- 172.21.15.32:0/1724914972 <== osd.0 v2:172.21.15.32:6816/3066156116 42 ==== osd_op_reply(67 608.00000000 [setxattr (62)] v95'61 uv61 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fea5401e040 con 0x55e321486450 2024-08-03T20:56:08.563 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:08.562+0000 7fea6022af40 1 -- 172.21.15.32:0/1724914972 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:68 12.e 12:717a0223:::608.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e95) -- 0x55e32155c3d0 con 0x55e321486450 2024-08-03T20:56:08.565 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:08.563+0000 7fea59e00640 1 -- 172.21.15.32:0/1724914972 <== osd.0 v2:172.21.15.32:6816/3066156116 43 ==== osd_op_reply(68 608.00000000 [omap-set-header] v95'62 uv62 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fea5401e040 con 0x55e321486450 2024-08-03T20:56:08.565 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:08.563+0000 7fea6022af40 1 --2- 172.21.15.32:0/1724914972 >> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] conn(0x55e32155ca50 0x55e32155ce60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:56:08.565 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:08.563+0000 7fea6022af40 1 -- 172.21.15.32:0/1724914972 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:69 12.15 12:a93a17c2:::604.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e95) -- 0x55e32155f860 con 0x55e32155ca50 2024-08-03T20:56:08.565 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:08.564+0000 7fea59e00640 1 --2- 172.21.15.32:0/1724914972 >> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] conn(0x55e32155ca50 0x55e32155ce60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:56:08.566 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:08.564+0000 7fea59e00640 1 --2- 172.21.15.32:0/1724914972 >> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] conn(0x55e32155ca50 0x55e32155ce60 crc :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:56:08.567 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:08.566+0000 7fea59e00640 1 -- 172.21.15.32:0/1724914972 <== osd.6 v2:172.21.15.118:6825/801947386 1 ==== osd_op_reply(69 604.00000000 [create] v95'105 uv99 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fea5401e040 con 0x55e32155ca50 2024-08-03T20:56:08.568 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:08.566+0000 7fea6022af40 1 -- 172.21.15.32:0/1724914972 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:70 12.15 12:a93a17c2:::604.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) -- 0x55e32155f860 con 0x55e32155ca50 2024-08-03T20:56:08.569 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:08.567+0000 7fea59e00640 1 -- 172.21.15.32:0/1724914972 <== osd.6 v2:172.21.15.118:6825/801947386 2 ==== osd_op_reply(70 604.00000000 [delete] v95'106 uv106 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fea5401e040 con 0x55e32155ca50 2024-08-03T20:56:08.569 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:08.567+0000 7fea6022af40 1 -- 172.21.15.32:0/1724914972 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:71 12.15 12:a93a17c2:::604.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e95) -- 0x55e32155f860 con 0x55e32155ca50 2024-08-03T20:56:08.570 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::604.00000000:head# 2024-08-03T20:56:08.570 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:08.569+0000 7fea59e00640 1 -- 172.21.15.32:0/1724914972 <== osd.6 v2:172.21.15.118:6825/801947386 3 ==== osd_op_reply(71 604.00000000 [create] v95'107 uv107 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fea5401e040 con 0x55e32155ca50 2024-08-03T20:56:08.571 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:08.569+0000 7fea6022af40 1 -- 172.21.15.32:0/1724914972 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:72 12.15 12:a93a17c2:::604.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e95) -- 0x55e321560870 con 0x55e32155ca50 2024-08-03T20:56:08.572 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:08.570+0000 7fea59e00640 1 -- 172.21.15.32:0/1724914972 <== osd.6 v2:172.21.15.118:6825/801947386 4 ==== osd_op_reply(72 604.00000000 [setxattr (30)] v95'108 uv108 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fea5401e040 con 0x55e32155ca50 2024-08-03T20:56:08.572 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:08.570+0000 7fea6022af40 1 -- 172.21.15.32:0/1724914972 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:73 12.15 12:a93a17c2:::604.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e95) -- 0x55e321560870 con 0x55e32155ca50 2024-08-03T20:56:08.573 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:08.572+0000 7fea59e00640 1 -- 172.21.15.32:0/1724914972 <== osd.6 v2:172.21.15.118:6825/801947386 5 ==== osd_op_reply(73 604.00000000 [setxattr (62)] v95'109 uv109 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fea5401e040 con 0x55e32155ca50 2024-08-03T20:56:08.573 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:08.572+0000 7fea6022af40 1 -- 172.21.15.32:0/1724914972 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:74 12.15 12:a93a17c2:::604.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e95) -- 0x55e32155f860 con 0x55e32155ca50 2024-08-03T20:56:08.575 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:08.574+0000 7fea59e00640 1 -- 172.21.15.32:0/1724914972 <== osd.6 v2:172.21.15.118:6825/801947386 6 ==== osd_op_reply(74 604.00000000 [omap-set-header] v95'110 uv110 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fea5401e040 con 0x55e32155ca50 2024-08-03T20:56:08.575 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:08.574+0000 7fea6022af40 1 -- 172.21.15.32:0/1724914972 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:75 12.15 12:aa448500:::500.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e95) -- 0x55e32155f860 con 0x55e32155ca50 2024-08-03T20:56:08.577 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:08.575+0000 7fea59e00640 1 -- 172.21.15.32:0/1724914972 <== osd.6 v2:172.21.15.118:6825/801947386 7 ==== osd_op_reply(75 500.00000000 [create] v95'111 uv104 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fea5401e040 con 0x55e32155ca50 2024-08-03T20:56:08.577 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:08.575+0000 7fea6022af40 1 -- 172.21.15.32:0/1724914972 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:76 12.15 12:aa448500:::500.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) -- 0x55e32155f860 con 0x55e32155ca50 2024-08-03T20:56:08.578 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:08.577+0000 7fea59e00640 1 -- 172.21.15.32:0/1724914972 <== osd.6 v2:172.21.15.118:6825/801947386 8 ==== osd_op_reply(76 500.00000000 [delete] v95'112 uv112 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fea5401e040 con 0x55e32155ca50 2024-08-03T20:56:08.578 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:08.577+0000 7fea6022af40 1 -- 172.21.15.32:0/1724914972 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:77 12.15 12:aa448500:::500.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e95) -- 0x55e32155f860 con 0x55e32155ca50 2024-08-03T20:56:08.580 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:08.578+0000 7fea59e00640 1 -- 172.21.15.32:0/1724914972 <== osd.6 v2:172.21.15.118:6825/801947386 9 ==== osd_op_reply(77 500.00000000 [create] v95'113 uv113 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fea5401e040 con 0x55e32155ca50 2024-08-03T20:56:08.580 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::500.00000000:head# 2024-08-03T20:56:08.580 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:08.578+0000 7fea6022af40 1 -- 172.21.15.32:0/1724914972 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:78 12.15 12:aa448500:::500.00000000:head [write 0~90 in=90b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e95) -- 0x55e32155f940 con 0x55e32155ca50 2024-08-03T20:56:08.581 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:08.580+0000 7fea59e00640 1 -- 172.21.15.32:0/1724914972 <== osd.6 v2:172.21.15.118:6825/801947386 10 ==== osd_op_reply(78 500.00000000 [write 0~90] v95'114 uv114 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fea5401e040 con 0x55e32155ca50 2024-08-03T20:56:08.581 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:08.580+0000 7fea6022af40 1 -- 172.21.15.32:0/1724914972 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:79 12.15 12:aa448500:::500.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e95) -- 0x55e32155f940 con 0x55e32155ca50 2024-08-03T20:56:08.583 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:08.581+0000 7fea59e00640 1 -- 172.21.15.32:0/1724914972 <== osd.6 v2:172.21.15.118:6825/801947386 11 ==== osd_op_reply(79 500.00000000 [omap-set-header] v95'115 uv115 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fea5401e040 con 0x55e32155ca50 2024-08-03T20:56:08.583 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:08.581+0000 7fea6022af40 1 --2- 172.21.15.32:0/1724914972 >> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] conn(0x55e321560240 0x55e321560650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:56:08.583 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:08.581+0000 7fea6022af40 1 -- 172.21.15.32:0/1724914972 --> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] -- osd_op(unknown.0.0:80 12.d 12:b50e409b:::mds_snaptable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e95) -- 0x55e321562c90 con 0x55e321560240 2024-08-03T20:56:08.583 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:08.581+0000 7fea59400640 1 --2- 172.21.15.32:0/1724914972 >> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] conn(0x55e321560240 0x55e321560650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:56:08.583 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:08.582+0000 7fea59400640 1 --2- 172.21.15.32:0/1724914972 >> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] conn(0x55e321560240 0x55e321560650 crc :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:56:08.585 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:08.583+0000 7fea59400640 1 -- 172.21.15.32:0/1724914972 <== osd.7 v2:172.21.15.118:6818/653000048 1 ==== osd_op_reply(80 mds_snaptable [create] v95'46 uv45 ondisk = -17 ((17) File exists)) ==== 157+0+0 (crc 0 0 0) 0x7fea4c069030 con 0x55e321560240 2024-08-03T20:56:08.585 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:08.583+0000 7fea6022af40 1 -- 172.21.15.32:0/1724914972 --> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] -- osd_op(unknown.0.0:81 12.d 12:b50e409b:::mds_snaptable:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) -- 0x55e321562cd0 con 0x55e321560240 2024-08-03T20:56:08.586 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:08.585+0000 7fea59400640 1 -- 172.21.15.32:0/1724914972 <== osd.7 v2:172.21.15.118:6818/653000048 2 ==== osd_op_reply(81 mds_snaptable [delete] v95'47 uv47 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7fea4c069030 con 0x55e321560240 2024-08-03T20:56:08.586 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:08.585+0000 7fea6022af40 1 -- 172.21.15.32:0/1724914972 --> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] -- osd_op(unknown.0.0:82 12.d 12:b50e409b:::mds_snaptable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e95) -- 0x55e3215630c0 con 0x55e321560240 2024-08-03T20:56:08.587 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:08.586+0000 7fea59400640 1 -- 172.21.15.32:0/1724914972 <== osd.7 v2:172.21.15.118:6818/653000048 3 ==== osd_op_reply(82 mds_snaptable [create] v95'48 uv48 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7fea4c069030 con 0x55e321560240 2024-08-03T20:56:08.587 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds_snaptable:head# 2024-08-03T20:56:08.588 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:08.586+0000 7fea6022af40 1 -- 172.21.15.32:0/1724914972 --> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] -- osd_op(unknown.0.0:83 12.d 12:b50e409b:::mds_snaptable:head [write 0~70 in=70b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e95) -- 0x55e3215630c0 con 0x55e321560240 2024-08-03T20:56:08.588 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:08.587+0000 7fea59400640 1 -- 172.21.15.32:0/1724914972 <== osd.7 v2:172.21.15.118:6818/653000048 4 ==== osd_op_reply(83 mds_snaptable [write 0~70] v95'49 uv49 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7fea4c069030 con 0x55e321560240 2024-08-03T20:56:08.589 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:08.587+0000 7fea6022af40 1 -- 172.21.15.32:0/1724914972 --> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] -- osd_op(unknown.0.0:84 12.d 12:b50e409b:::mds_snaptable:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e95) -- 0x55e3215630c0 con 0x55e321560240 2024-08-03T20:56:08.590 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:08.588+0000 7fea59400640 1 -- 172.21.15.32:0/1724914972 <== osd.7 v2:172.21.15.118:6818/653000048 5 ==== osd_op_reply(84 mds_snaptable [omap-set-header] v95'50 uv50 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7fea4c069030 con 0x55e321560240 2024-08-03T20:56:08.590 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:08.588+0000 7fea6022af40 1 -- 172.21.15.32:0/1724914972 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:85 12.1d 12:b871830b:::605.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e95) -- 0x55e3215630c0 con 0x55e321486450 2024-08-03T20:56:08.591 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:08.590+0000 7fea59e00640 1 -- 172.21.15.32:0/1724914972 <== osd.0 v2:172.21.15.32:6816/3066156116 44 ==== osd_op_reply(85 605.00000000 [create] v95'57 uv56 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fea5401e040 con 0x55e321486450 2024-08-03T20:56:08.591 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:08.590+0000 7fea6022af40 1 -- 172.21.15.32:0/1724914972 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:86 12.1d 12:b871830b:::605.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) -- 0x55e3215630c0 con 0x55e321486450 2024-08-03T20:56:08.592 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:08.591+0000 7fea59e00640 1 -- 172.21.15.32:0/1724914972 <== osd.0 v2:172.21.15.32:6816/3066156116 45 ==== osd_op_reply(86 605.00000000 [delete] v95'58 uv58 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fea5401e040 con 0x55e321486450 2024-08-03T20:56:08.593 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:08.591+0000 7fea6022af40 1 -- 172.21.15.32:0/1724914972 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:87 12.1d 12:b871830b:::605.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e95) -- 0x55e3215630c0 con 0x55e321486450 2024-08-03T20:56:08.594 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::605.00000000:head# 2024-08-03T20:56:08.594 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:08.592+0000 7fea59e00640 1 -- 172.21.15.32:0/1724914972 <== osd.0 v2:172.21.15.32:6816/3066156116 46 ==== osd_op_reply(87 605.00000000 [create] v95'59 uv59 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fea5401e040 con 0x55e321486450 2024-08-03T20:56:08.594 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:08.593+0000 7fea6022af40 1 -- 172.21.15.32:0/1724914972 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:88 12.1d 12:b871830b:::605.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e95) -- 0x55e3215640d0 con 0x55e321486450 2024-08-03T20:56:08.595 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:08.594+0000 7fea59e00640 1 -- 172.21.15.32:0/1724914972 <== osd.0 v2:172.21.15.32:6816/3066156116 47 ==== osd_op_reply(88 605.00000000 [setxattr (30)] v95'60 uv60 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fea5401e040 con 0x55e321486450 2024-08-03T20:56:08.595 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:08.594+0000 7fea6022af40 1 -- 172.21.15.32:0/1724914972 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:89 12.1d 12:b871830b:::605.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e95) -- 0x55e3215640d0 con 0x55e321486450 2024-08-03T20:56:08.597 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:08.595+0000 7fea59e00640 1 -- 172.21.15.32:0/1724914972 <== osd.0 v2:172.21.15.32:6816/3066156116 48 ==== osd_op_reply(89 605.00000000 [setxattr (62)] v95'61 uv61 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fea5401e040 con 0x55e321486450 2024-08-03T20:56:08.597 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:08.595+0000 7fea6022af40 1 -- 172.21.15.32:0/1724914972 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:90 12.1d 12:b871830b:::605.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e95) -- 0x55e3215630c0 con 0x55e321486450 2024-08-03T20:56:08.598 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:08.597+0000 7fea59e00640 1 -- 172.21.15.32:0/1724914972 <== osd.0 v2:172.21.15.32:6816/3066156116 49 ==== osd_op_reply(90 605.00000000 [omap-set-header] v95'62 uv62 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fea5401e040 con 0x55e321486450 2024-08-03T20:56:08.598 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:08.597+0000 7fea6022af40 1 -- 172.21.15.32:0/1724914972 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:91 12.13 12:c91d4a1d:::mds0_inotable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e95) -- 0x55e3215630c0 con 0x55e321558780 2024-08-03T20:56:08.599 INFO:teuthology.orchestra.run.smithi032.stdout:Write #-9223372036854775808:00000000:::mds0_inotable:head# 2024-08-03T20:56:08.600 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:08.598+0000 7fea58a00640 1 -- 172.21.15.32:0/1724914972 <== osd.1 v2:172.21.15.32:6824/1551241759 5 ==== osd_op_reply(91 mds0_inotable [create] v95'95 uv95 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7fea4801b040 con 0x55e321558780 2024-08-03T20:56:08.600 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:08.598+0000 7fea6022af40 1 -- 172.21.15.32:0/1724914972 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:92 12.13 12:c91d4a1d:::mds0_inotable:head [write 0~34 in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e95) -- 0x55e321563460 con 0x55e321558780 2024-08-03T20:56:08.601 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:08.600+0000 7fea58a00640 1 -- 172.21.15.32:0/1724914972 <== osd.1 v2:172.21.15.32:6824/1551241759 6 ==== osd_op_reply(92 mds0_inotable [write 0~34] v95'96 uv96 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7fea4801b040 con 0x55e321558780 2024-08-03T20:56:08.601 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:08.600+0000 7fea6022af40 1 -- 172.21.15.32:0/1724914972 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:93 12.13 12:c91d4a1d:::mds0_inotable:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e95) -- 0x55e321563800 con 0x55e321558780 2024-08-03T20:56:08.602 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:08.601+0000 7fea58a00640 1 -- 172.21.15.32:0/1724914972 <== osd.1 v2:172.21.15.32:6824/1551241759 7 ==== osd_op_reply(93 mds0_inotable [omap-set-header] v95'97 uv97 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7fea4801b040 con 0x55e321558780 2024-08-03T20:56:08.602 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:08.601+0000 7fea6022af40 1 -- 172.21.15.32:0/1724914972 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:94 12.13 12:cd5a64a3:::100.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e95) -- 0x55e321563ba0 con 0x55e321558780 2024-08-03T20:56:08.603 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:08.602+0000 7fea58a00640 1 -- 172.21.15.32:0/1724914972 <== osd.1 v2:172.21.15.32:6824/1551241759 8 ==== osd_op_reply(94 100.00000000 [create] v95'98 uv93 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fea4801b040 con 0x55e321558780 2024-08-03T20:56:08.603 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:08.602+0000 7fea6022af40 1 -- 172.21.15.32:0/1724914972 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:95 12.13 12:cd5a64a3:::100.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) -- 0x55e321563ba0 con 0x55e321558780 2024-08-03T20:56:08.605 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:08.603+0000 7fea58a00640 1 -- 172.21.15.32:0/1724914972 <== osd.1 v2:172.21.15.32:6824/1551241759 9 ==== osd_op_reply(95 100.00000000 [delete] v95'99 uv99 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fea4801b040 con 0x55e321558780 2024-08-03T20:56:08.605 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:08.604+0000 7fea6022af40 1 -- 172.21.15.32:0/1724914972 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:96 12.13 12:cd5a64a3:::100.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e95) -- 0x55e321563ba0 con 0x55e321558780 2024-08-03T20:56:08.606 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::100.00000000:head# 2024-08-03T20:56:08.606 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:08.605+0000 7fea58a00640 1 -- 172.21.15.32:0/1724914972 <== osd.1 v2:172.21.15.32:6824/1551241759 10 ==== osd_op_reply(96 100.00000000 [create] v95'100 uv100 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fea4801b040 con 0x55e321558780 2024-08-03T20:56:08.607 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:08.605+0000 7fea6022af40 1 -- 172.21.15.32:0/1724914972 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:97 12.13 12:cd5a64a3:::100.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e95) -- 0x55e321563ba0 con 0x55e321558780 2024-08-03T20:56:08.607 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:08.606+0000 7fea58a00640 1 -- 172.21.15.32:0/1724914972 <== osd.1 v2:172.21.15.32:6824/1551241759 11 ==== osd_op_reply(97 100.00000000 [omap-set-header] v95'101 uv101 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fea4801b040 con 0x55e321558780 2024-08-03T20:56:08.607 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:08.606+0000 7fea6022af40 1 -- 172.21.15.32:0/1724914972 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:98 12.13 12:cd5a64a3:::100.00000000:head [omap-set-vals in=5194b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e95) -- 0x55e3215669b0 con 0x55e321558780 2024-08-03T20:56:08.609 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:08.607+0000 7fea58a00640 1 -- 172.21.15.32:0/1724914972 <== osd.1 v2:172.21.15.32:6824/1551241759 12 ==== osd_op_reply(98 100.00000000 [omap-set-vals] v95'102 uv102 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fea4801b040 con 0x55e321558780 2024-08-03T20:56:08.609 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:08.608+0000 7fea6022af40 1 --2- 172.21.15.32:0/1724914972 >> [v2:172.21.15.32:6800/9088303,v1:172.21.15.32:6801/9088303] conn(0x55e3215646d0 0x55e321567050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:56:08.609 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:08.608+0000 7fea6022af40 1 -- 172.21.15.32:0/1724914972 --> [v2:172.21.15.32:6800/9088303,v1:172.21.15.32:6801/9088303] -- osd_op(unknown.0.0:99 12.b 12:d0630e4c:::mds0_sessionmap:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e95) -- 0x55e321566af0 con 0x55e3215646d0 2024-08-03T20:56:08.609 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:08.608+0000 7fea58a00640 1 --2- 172.21.15.32:0/1724914972 >> [v2:172.21.15.32:6800/9088303,v1:172.21.15.32:6801/9088303] conn(0x55e3215646d0 0x55e321567050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:56:08.610 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:08.608+0000 7fea58a00640 1 --2- 172.21.15.32:0/1724914972 >> [v2:172.21.15.32:6800/9088303,v1:172.21.15.32:6801/9088303] conn(0x55e3215646d0 0x55e321567050 crc :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:56:08.614 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:08.610+0000 7fea58a00640 1 -- 172.21.15.32:0/1724914972 <== osd.3 v2:172.21.15.32:6800/9088303 1 ==== osd_op_reply(99 mds0_sessionmap [create] v95'44 uv43 ondisk = -17 ((17) File exists)) ==== 159+0+0 (crc 0 0 0) 0x7fea4801b040 con 0x55e3215646d0 2024-08-03T20:56:08.615 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:08.610+0000 7fea6022af40 1 -- 172.21.15.32:0/1724914972 --> [v2:172.21.15.32:6800/9088303,v1:172.21.15.32:6801/9088303] -- osd_op(unknown.0.0:100 12.b 12:d0630e4c:::mds0_sessionmap:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) -- 0x55e321566af0 con 0x55e3215646d0 2024-08-03T20:56:08.615 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:08.612+0000 7fea58a00640 1 -- 172.21.15.32:0/1724914972 <== osd.3 v2:172.21.15.32:6800/9088303 2 ==== osd_op_reply(100 mds0_sessionmap [delete] v95'45 uv45 ondisk = 0) ==== 159+0+0 (crc 0 0 0) 0x7fea4801b040 con 0x55e3215646d0 2024-08-03T20:56:08.615 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:08.612+0000 7fea6022af40 1 -- 172.21.15.32:0/1724914972 --> [v2:172.21.15.32:6800/9088303,v1:172.21.15.32:6801/9088303] -- osd_op(unknown.0.0:101 12.b 12:d0630e4c:::mds0_sessionmap:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e95) -- 0x55e321566af0 con 0x55e3215646d0 2024-08-03T20:56:08.615 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds0_sessionmap:head# 2024-08-03T20:56:08.615 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:08.613+0000 7fea58a00640 1 -- 172.21.15.32:0/1724914972 <== osd.3 v2:172.21.15.32:6800/9088303 3 ==== osd_op_reply(101 mds0_sessionmap [create] v95'46 uv46 ondisk = 0) ==== 159+0+0 (crc 0 0 0) 0x7fea4801b040 con 0x55e3215646d0 2024-08-03T20:56:08.615 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:08.613+0000 7fea6022af40 1 -- 172.21.15.32:0/1724914972 --> [v2:172.21.15.32:6800/9088303,v1:172.21.15.32:6801/9088303] -- osd_op(unknown.0.0:102 12.b 12:d0630e4c:::mds0_sessionmap:head [omap-set-header in=14b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e95) -- 0x55e321566af0 con 0x55e3215646d0 2024-08-03T20:56:08.616 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:08.614+0000 7fea58a00640 1 -- 172.21.15.32:0/1724914972 <== osd.3 v2:172.21.15.32:6800/9088303 4 ==== osd_op_reply(102 mds0_sessionmap [omap-set-header] v95'47 uv47 ondisk = 0) ==== 159+0+0 (crc 0 0 0) 0x7fea4801b040 con 0x55e3215646d0 2024-08-03T20:56:08.616 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:08.614+0000 7fea6022af40 1 -- 172.21.15.32:0/1724914972 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:103 12.7 12:e0b41d06:::609.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e95) -- 0x55e321566af0 con 0x55e321558780 2024-08-03T20:56:08.617 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:08.616+0000 7fea58a00640 1 -- 172.21.15.32:0/1724914972 <== osd.1 v2:172.21.15.32:6824/1551241759 13 ==== osd_op_reply(103 609.00000000 [create] v95'57 uv56 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fea4801b040 con 0x55e321558780 2024-08-03T20:56:08.617 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:08.616+0000 7fea6022af40 1 -- 172.21.15.32:0/1724914972 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:104 12.7 12:e0b41d06:::609.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) -- 0x55e321566af0 con 0x55e321558780 2024-08-03T20:56:08.618 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:08.617+0000 7fea58a00640 1 -- 172.21.15.32:0/1724914972 <== osd.1 v2:172.21.15.32:6824/1551241759 14 ==== osd_op_reply(104 609.00000000 [delete] v95'58 uv58 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fea4801b040 con 0x55e321558780 2024-08-03T20:56:08.619 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:08.617+0000 7fea6022af40 1 -- 172.21.15.32:0/1724914972 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:105 12.7 12:e0b41d06:::609.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e95) -- 0x55e321566af0 con 0x55e321558780 2024-08-03T20:56:08.620 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:08.619+0000 7fea58a00640 1 -- 172.21.15.32:0/1724914972 <== osd.1 v2:172.21.15.32:6824/1551241759 15 ==== osd_op_reply(105 609.00000000 [create] v95'59 uv59 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fea4801b040 con 0x55e321558780 2024-08-03T20:56:08.620 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::609.00000000:head# 2024-08-03T20:56:08.620 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:08.619+0000 7fea6022af40 1 -- 172.21.15.32:0/1724914972 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:106 12.7 12:e0b41d06:::609.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e95) -- 0x55e321566af0 con 0x55e321558780 2024-08-03T20:56:08.621 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:08.620+0000 7fea58a00640 1 -- 172.21.15.32:0/1724914972 <== osd.1 v2:172.21.15.32:6824/1551241759 16 ==== osd_op_reply(106 609.00000000 [setxattr (30)] v95'60 uv60 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fea4801b040 con 0x55e321558780 2024-08-03T20:56:08.621 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:08.620+0000 7fea6022af40 1 -- 172.21.15.32:0/1724914972 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:107 12.7 12:e0b41d06:::609.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e95) -- 0x55e321566af0 con 0x55e321558780 2024-08-03T20:56:08.622 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:08.621+0000 7fea58a00640 1 -- 172.21.15.32:0/1724914972 <== osd.1 v2:172.21.15.32:6824/1551241759 17 ==== osd_op_reply(107 609.00000000 [setxattr (62)] v95'61 uv61 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fea4801b040 con 0x55e321558780 2024-08-03T20:56:08.622 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:08.621+0000 7fea6022af40 1 -- 172.21.15.32:0/1724914972 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:108 12.7 12:e0b41d06:::609.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e95) -- 0x55e321566af0 con 0x55e321558780 2024-08-03T20:56:08.624 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:08.622+0000 7fea58a00640 1 -- 172.21.15.32:0/1724914972 <== osd.1 v2:172.21.15.32:6824/1551241759 18 ==== osd_op_reply(108 609.00000000 [omap-set-header] v95'62 uv62 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fea4801b040 con 0x55e321558780 2024-08-03T20:56:08.624 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:08.622+0000 7fea6022af40 1 -- 172.21.15.32:0/1724914972 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:109 12.f 12:f1f69726:::400.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e95) -- 0x55e321566af0 con 0x55e32155ca50 2024-08-03T20:56:08.625 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:08.624+0000 7fea59e00640 1 -- 172.21.15.32:0/1724914972 <== osd.6 v2:172.21.15.118:6825/801947386 12 ==== osd_op_reply(109 400.00000000 [create] v95'47 uv46 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fea5401e040 con 0x55e32155ca50 2024-08-03T20:56:08.625 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:08.624+0000 7fea6022af40 1 -- 172.21.15.32:0/1724914972 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:110 12.f 12:f1f69726:::400.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) -- 0x55e321566af0 con 0x55e32155ca50 2024-08-03T20:56:08.627 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:08.625+0000 7fea59e00640 1 -- 172.21.15.32:0/1724914972 <== osd.6 v2:172.21.15.118:6825/801947386 13 ==== osd_op_reply(110 400.00000000 [delete] v95'48 uv48 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fea5401e040 con 0x55e32155ca50 2024-08-03T20:56:08.627 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:08.626+0000 7fea6022af40 1 -- 172.21.15.32:0/1724914972 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:111 12.f 12:f1f69726:::400.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e95) -- 0x55e321566af0 con 0x55e32155ca50 2024-08-03T20:56:08.628 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::400.00000000:head# 2024-08-03T20:56:08.629 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:08.627+0000 7fea59e00640 1 -- 172.21.15.32:0/1724914972 <== osd.6 v2:172.21.15.118:6825/801947386 14 ==== osd_op_reply(111 400.00000000 [create] v95'49 uv49 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fea5401e040 con 0x55e32155ca50 2024-08-03T20:56:08.629 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:08.627+0000 7fea6022af40 1 -- 172.21.15.32:0/1724914972 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:112 12.f 12:f1f69726:::400.00000000:head [write 0~22 in=22b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e95) -- 0x55e321567590 con 0x55e32155ca50 2024-08-03T20:56:08.630 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:08.629+0000 7fea59e00640 1 -- 172.21.15.32:0/1724914972 <== osd.6 v2:172.21.15.118:6825/801947386 15 ==== osd_op_reply(112 400.00000000 [write 0~22] v95'50 uv50 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fea5401e040 con 0x55e32155ca50 2024-08-03T20:56:08.630 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:08.629+0000 7fea6022af40 1 -- 172.21.15.32:0/1724914972 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:113 12.f 12:f1f69726:::400.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e95) -- 0x55e321567590 con 0x55e32155ca50 2024-08-03T20:56:08.631 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:08.630+0000 7fea59e00640 1 -- 172.21.15.32:0/1724914972 <== osd.6 v2:172.21.15.118:6825/801947386 16 ==== osd_op_reply(113 400.00000000 [omap-set-header] v95'51 uv51 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fea5401e040 con 0x55e32155ca50 2024-08-03T20:56:08.631 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:08.630+0000 7fea6022af40 1 -- 172.21.15.32:0/1724914972 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:114 12.1f 12:fe07bba1:::100.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e95) -- 0x55e321567590 con 0x55e32155ca50 2024-08-03T20:56:08.633 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:08.631+0000 7fea59e00640 1 -- 172.21.15.32:0/1724914972 <== osd.6 v2:172.21.15.118:6825/801947386 17 ==== osd_op_reply(114 100.00000000.inode [create] v95'119 uv111 ondisk = -17 ((17) File exists)) ==== 162+0+0 (crc 0 0 0) 0x7fea5401e040 con 0x55e32155ca50 2024-08-03T20:56:08.633 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:08.632+0000 7fea6022af40 1 -- 172.21.15.32:0/1724914972 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:115 12.1f 12:fe07bba1:::100.00000000.inode:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) -- 0x55e321567590 con 0x55e32155ca50 2024-08-03T20:56:08.634 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:08.633+0000 7fea59e00640 1 -- 172.21.15.32:0/1724914972 <== osd.6 v2:172.21.15.118:6825/801947386 18 ==== osd_op_reply(115 100.00000000.inode [delete] v95'120 uv120 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7fea5401e040 con 0x55e32155ca50 2024-08-03T20:56:08.635 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:08.633+0000 7fea6022af40 1 -- 172.21.15.32:0/1724914972 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:116 12.1f 12:fe07bba1:::100.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e95) -- 0x55e321567590 con 0x55e32155ca50 2024-08-03T20:56:08.636 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::100.00000000.inode:head# 2024-08-03T20:56:08.636 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:08.635+0000 7fea59e00640 1 -- 172.21.15.32:0/1724914972 <== osd.6 v2:172.21.15.118:6825/801947386 19 ==== osd_op_reply(116 100.00000000.inode [create] v95'121 uv121 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7fea5401e040 con 0x55e32155ca50 2024-08-03T20:56:08.636 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:08.635+0000 7fea6022af40 1 -- 172.21.15.32:0/1724914972 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:117 12.1f 12:fe07bba1:::100.00000000.inode:head [write 0~582 in=582b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e95) -- 0x55e321567590 con 0x55e32155ca50 2024-08-03T20:56:08.637 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:08.636+0000 7fea59e00640 1 -- 172.21.15.32:0/1724914972 <== osd.6 v2:172.21.15.118:6825/801947386 20 ==== osd_op_reply(117 100.00000000.inode [write 0~582] v95'122 uv122 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7fea5401e040 con 0x55e32155ca50 2024-08-03T20:56:08.638 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:08.636+0000 7fea6022af40 1 -- 172.21.15.32:0/1724914972 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:118 12.1f 12:fe07bba1:::100.00000000.inode:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e95) -- 0x55e321567590 con 0x55e32155ca50 2024-08-03T20:56:08.639 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:08.638+0000 7fea59e00640 1 -- 172.21.15.32:0/1724914972 <== osd.6 v2:172.21.15.118:6825/801947386 21 ==== osd_op_reply(118 100.00000000.inode [omap-set-header] v95'123 uv123 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7fea5401e040 con 0x55e32155ca50 2024-08-03T20:56:08.639 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:08.638+0000 7fea6022af40 1 -- 172.21.15.32:0/1724914972 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:119 12.1f 12:ff5b34d6:::1.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e95) -- 0x55e321567590 con 0x55e32155ca50 2024-08-03T20:56:08.641 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:08.639+0000 7fea59e00640 1 -- 172.21.15.32:0/1724914972 <== osd.6 v2:172.21.15.118:6825/801947386 22 ==== osd_op_reply(119 1.00000000 [create] v95'124 uv118 ondisk = -17 ((17) File exists)) ==== 154+0+0 (crc 0 0 0) 0x7fea5401e040 con 0x55e32155ca50 2024-08-03T20:56:08.641 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:08.639+0000 7fea6022af40 1 -- 172.21.15.32:0/1724914972 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:120 12.1f 12:ff5b34d6:::1.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) -- 0x55e321567590 con 0x55e32155ca50 2024-08-03T20:56:08.642 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:08.641+0000 7fea59e00640 1 -- 172.21.15.32:0/1724914972 <== osd.6 v2:172.21.15.118:6825/801947386 23 ==== osd_op_reply(120 1.00000000 [delete] v95'125 uv125 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7fea5401e040 con 0x55e32155ca50 2024-08-03T20:56:08.642 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:08.641+0000 7fea6022af40 1 -- 172.21.15.32:0/1724914972 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:121 12.1f 12:ff5b34d6:::1.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e95) -- 0x55e321567590 con 0x55e32155ca50 2024-08-03T20:56:08.643 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::1.00000000:head# 2024-08-03T20:56:08.643 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:08.642+0000 7fea59e00640 1 -- 172.21.15.32:0/1724914972 <== osd.6 v2:172.21.15.118:6825/801947386 24 ==== osd_op_reply(121 1.00000000 [create] v95'126 uv126 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7fea5401e040 con 0x55e32155ca50 2024-08-03T20:56:08.644 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:08.642+0000 7fea6022af40 1 -- 172.21.15.32:0/1724914972 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:122 12.1f 12:ff5b34d6:::1.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e95) -- 0x55e321567590 con 0x55e32155ca50 2024-08-03T20:56:08.645 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:08.643+0000 7fea59e00640 1 -- 172.21.15.32:0/1724914972 <== osd.6 v2:172.21.15.118:6825/801947386 25 ==== osd_op_reply(122 1.00000000 [setxattr (30)] v95'127 uv127 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7fea5401e040 con 0x55e32155ca50 2024-08-03T20:56:08.645 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:08.643+0000 7fea6022af40 1 -- 172.21.15.32:0/1724914972 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:123 12.1f 12:ff5b34d6:::1.00000000:head [setxattr parent (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e95) -- 0x55e321567590 con 0x55e32155ca50 2024-08-03T20:56:08.647 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:08.645+0000 7fea59e00640 1 -- 172.21.15.32:0/1724914972 <== osd.6 v2:172.21.15.118:6825/801947386 26 ==== osd_op_reply(123 1.00000000 [setxattr (30)] v95'128 uv128 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7fea5401e040 con 0x55e32155ca50 2024-08-03T20:56:08.647 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:08.645+0000 7fea6022af40 1 -- 172.21.15.32:0/1724914972 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:124 12.1f 12:ff5b34d6:::1.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e95) -- 0x55e321567590 con 0x55e32155ca50 2024-08-03T20:56:08.648 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:08.646+0000 7fea59e00640 1 -- 172.21.15.32:0/1724914972 <== osd.6 v2:172.21.15.118:6825/801947386 27 ==== osd_op_reply(124 1.00000000 [omap-set-header] v95'129 uv129 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7fea5401e040 con 0x55e32155ca50 2024-08-03T20:56:08.648 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:08.646+0000 7fea6022af40 1 -- 172.21.15.32:0/1724914972 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:125 12.1f 12:ff5b34d6:::1.00000000:head [omap-set-vals in=584b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e95) -- 0x55e321567980 con 0x55e32155ca50 2024-08-03T20:56:08.650 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:08.648+0000 7fea59e00640 1 -- 172.21.15.32:0/1724914972 <== osd.6 v2:172.21.15.118:6825/801947386 28 ==== osd_op_reply(125 1.00000000 [omap-set-vals] v95'130 uv130 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7fea5401e040 con 0x55e32155ca50 2024-08-03T20:56:08.651 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:08.648+0000 7fea6022af40 1 -- 172.21.15.32:0/1724914972 >> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] conn(0x55e321486450 msgr2=0x55e3214a80e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:56:08.651 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:08.648+0000 7fea6022af40 1 --2- 172.21.15.32:0/1724914972 >> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] conn(0x55e321486450 0x55e3214a80e0 crc :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:08.651 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:08.648+0000 7fea6022af40 1 -- 172.21.15.32:0/1724914972 >> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] conn(0x55e321558780 msgr2=0x55e321558b90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:56:08.651 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:08.648+0000 7fea6022af40 1 --2- 172.21.15.32:0/1724914972 >> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] conn(0x55e321558780 0x55e321558b90 crc :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:08.651 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:08.648+0000 7fea6022af40 1 -- 172.21.15.32:0/1724914972 >> [v2:172.21.15.32:6800/9088303,v1:172.21.15.32:6801/9088303] conn(0x55e3215646d0 msgr2=0x55e321567050 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:56:08.651 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:08.648+0000 7fea6022af40 1 --2- 172.21.15.32:0/1724914972 >> [v2:172.21.15.32:6800/9088303,v1:172.21.15.32:6801/9088303] conn(0x55e3215646d0 0x55e321567050 crc :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:08.651 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:08.648+0000 7fea6022af40 1 -- 172.21.15.32:0/1724914972 >> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] conn(0x55e3215549f0 msgr2=0x55e321556e10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:56:08.651 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:08.648+0000 7fea6022af40 1 --2- 172.21.15.32:0/1724914972 >> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] conn(0x55e3215549f0 0x55e321556e10 crc :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:08.651 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:08.648+0000 7fea6022af40 1 -- 172.21.15.32:0/1724914972 >> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] conn(0x55e32155ca50 msgr2=0x55e32155ce60 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:56:08.651 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:08.648+0000 7fea6022af40 1 --2- 172.21.15.32:0/1724914972 >> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] conn(0x55e32155ca50 0x55e32155ce60 crc :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:08.651 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:08.648+0000 7fea6022af40 1 -- 172.21.15.32:0/1724914972 >> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] conn(0x55e321560240 msgr2=0x55e321560650 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:56:08.651 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:08.648+0000 7fea6022af40 1 --2- 172.21.15.32:0/1724914972 >> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] conn(0x55e321560240 0x55e321560650 crc :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:08.652 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:08.648+0000 7fea59e00640 1 -- 172.21.15.32:0/1724914972 reap_dead start 2024-08-03T20:56:08.652 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:08.648+0000 7fea59e00640 1 -- 172.21.15.32:0/1724914972 reap_dead start 2024-08-03T20:56:08.652 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:08.648+0000 7fea59e00640 1 -- 172.21.15.32:0/1724914972 reap_dead start 2024-08-03T20:56:08.652 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:08.648+0000 7fea6022af40 1 -- 172.21.15.32:0/1724914972 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fea3c075a20 msgr2=0x7fea3c077f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:56:08.652 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:08.648+0000 7fea6022af40 1 --2- 172.21.15.32:0/1724914972 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fea3c075a20 0x7fea3c077f30 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7fea48003ef0 tx=0x7fea480011f0 comp rx=0 tx=0).stop 2024-08-03T20:56:08.652 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:08.648+0000 7fea6022af40 1 -- 172.21.15.32:0/1724914972 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55e321492360 msgr2=0x55e3214ac310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:56:08.652 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:08.648+0000 7fea6022af40 1 --2- 172.21.15.32:0/1724914972 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55e321492360 0x55e3214ac310 secure :-1 s=READY pgs=1311 cs=0 l=1 rev1=1 crypto rx=0x7fea4c025fa0 tx=0x7fea4c056c10 comp rx=0 tx=0).stop 2024-08-03T20:56:08.652 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:08.648+0000 7fea6022af40 1 -- 172.21.15.32:0/1724914972 shutdown_connections 2024-08-03T20:56:08.652 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:08.648+0000 7fea6022af40 1 --2- 172.21.15.32:0/1724914972 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fea3c075a20 0x7fea3c077f30 unknown :-1 s=CLOSED pgs=540 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:08.652 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:08.648+0000 7fea6022af40 1 --2- 172.21.15.32:0/1724914972 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55e321492360 0x55e3214ac310 unknown :-1 s=CLOSED pgs=1311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:08.652 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:08.648+0000 7fea6022af40 1 -- 172.21.15.32:0/1724914972 >> 172.21.15.32:0/1724914972 conn(0x55e3212fbac0 msgr2=0x55e3214227c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:56:08.652 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:08.648+0000 7fea6022af40 1 -- 172.21.15.32:0/1724914972 shutdown_connections 2024-08-03T20:56:08.652 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:08.648+0000 7fea6022af40 1 -- 172.21.15.32:0/1724914972 wait complete. 2024-08-03T20:56:08.662 DEBUG:teuthology.orchestra.run.smithi032:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata rm 100.00000000 2024-08-03T20:56:08.807 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:08.806+0000 7ff16651cf40 1 Processor -- start 2024-08-03T20:56:08.808 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:08.806+0000 7ff16651cf40 1 -- start start 2024-08-03T20:56:08.809 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:08.807+0000 7ff16651cf40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55b55fe8bb40 0x55b55fefc8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:56:08.809 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:08.807+0000 7ff16651cf40 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b55fefebe0 0x55b55fe44830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:56:08.809 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:08.807+0000 7ff16651cf40 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55b55fe8de30 0x55b55fe45030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:56:08.809 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:08.807+0000 7ff16651cf40 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55b55fd94e30 con 0x55b55fefebe0 2024-08-03T20:56:08.809 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:08.807+0000 7ff16651cf40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x55b55fdf2bf0 con 0x55b55fe8bb40 2024-08-03T20:56:08.809 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:08.807+0000 7ff16651cf40 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x55b55fe8e5b0 con 0x55b55fe8de30 2024-08-03T20:56:08.809 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:08.808+0000 7ff15e200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b55fefebe0 0x55b55fe44830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:56:08.809 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:08.808+0000 7ff15ec00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55b55fe8bb40 0x55b55fefc8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:56:08.810 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:08.808+0000 7ff15f600640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55b55fe8de30 0x55b55fe45030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:56:08.810 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:08.808+0000 7ff15ec00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55b55fe8bb40 0x55b55fefc8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.32:57218/0 (socket says 172.21.15.32:57218) 2024-08-03T20:56:08.810 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:08.808+0000 7ff15f600640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55b55fe8de30 0x55b55fe45030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3301/0 says I am v2:172.21.15.32:38344/0 (socket says 172.21.15.32:38344) 2024-08-03T20:56:08.810 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:08.808+0000 7ff15ec00640 1 -- 172.21.15.32:0/135289128 learned_addr learned my addr 172.21.15.32:0/135289128 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:56:08.810 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:08.808+0000 7ff15e200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b55fefebe0 0x55b55fe44830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:42736/0 (socket says 172.21.15.32:42736) 2024-08-03T20:56:08.811 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:08.808+0000 7ff15ec00640 1 -- 172.21.15.32:0/135289128 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55b55fe8de30 msgr2=0x55b55fe45030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:56:08.811 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:08.808+0000 7ff15ec00640 1 --2- 172.21.15.32:0/135289128 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55b55fe8de30 0x55b55fe45030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:08.811 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:08.808+0000 7ff15ec00640 1 -- 172.21.15.32:0/135289128 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b55fefebe0 msgr2=0x55b55fe44830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:56:08.811 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:08.808+0000 7ff15ec00640 1 --2- 172.21.15.32:0/135289128 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b55fefebe0 0x55b55fe44830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:08.812 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:08.808+0000 7ff15ec00640 1 -- 172.21.15.32:0/135289128 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b55ff00410 con 0x55b55fe8bb40 2024-08-03T20:56:08.812 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:08.808+0000 7ff15ec00640 1 --2- 172.21.15.32:0/135289128 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55b55fe8bb40 0x55b55fefc8a0 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7ff150002ea0 tx=0x7ff15000b670 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8cdaf40cc22f4d23 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:56:08.812 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:08.809+0000 7ff15ce00640 1 -- 172.21.15.32:0/135289128 <== mon.1 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff15000e830 con 0x55b55fe8bb40 2024-08-03T20:56:08.812 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:08.809+0000 7ff15ce00640 1 -- 172.21.15.32:0/135289128 <== mon.1 v2:172.21.15.118:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7ff150006050 con 0x55b55fe8bb40 2024-08-03T20:56:08.812 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:08.809+0000 7ff15ce00640 1 -- 172.21.15.32:0/135289128 <== mon.1 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff15000ed80 con 0x55b55fe8bb40 2024-08-03T20:56:08.812 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:08.809+0000 7ff16651cf40 1 -- 172.21.15.32:0/135289128 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55b55fe8bb40 msgr2=0x55b55fefc8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:56:08.812 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:08.809+0000 7ff16651cf40 1 --2- 172.21.15.32:0/135289128 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55b55fe8bb40 0x55b55fefc8a0 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7ff150002ea0 tx=0x7ff15000b670 comp rx=0 tx=0).stop 2024-08-03T20:56:08.812 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:08.809+0000 7ff16651cf40 1 -- 172.21.15.32:0/135289128 shutdown_connections 2024-08-03T20:56:08.812 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:08.809+0000 7ff16651cf40 1 --2- 172.21.15.32:0/135289128 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55b55fe8de30 0x55b55fe45030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:08.812 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:08.809+0000 7ff16651cf40 1 --2- 172.21.15.32:0/135289128 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b55fefebe0 0x55b55fe44830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:08.812 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:08.809+0000 7ff16651cf40 1 --2- 172.21.15.32:0/135289128 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55b55fe8bb40 0x55b55fefc8a0 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:08.812 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:08.809+0000 7ff16651cf40 1 -- 172.21.15.32:0/135289128 >> 172.21.15.32:0/135289128 conn(0x55b55fd66ac0 msgr2=0x55b55fefc190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:56:08.812 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:08.810+0000 7ff16651cf40 1 -- 172.21.15.32:0/135289128 shutdown_connections 2024-08-03T20:56:08.812 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:08.810+0000 7ff16651cf40 1 -- 172.21.15.32:0/135289128 wait complete. 2024-08-03T20:56:08.813 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:08.811+0000 7ff16651cf40 1 Processor -- start 2024-08-03T20:56:08.813 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:08.811+0000 7ff16651cf40 1 -- start start 2024-08-03T20:56:08.815 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:08.812+0000 7ff16651cf40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55b55fe8bb40 0x55b55fefa410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:56:08.815 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:08.812+0000 7ff16651cf40 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b55fe8de30 0x55b55ffb3c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:56:08.815 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:08.812+0000 7ff16651cf40 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55b55fefebe0 0x55b55fe45030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:56:08.815 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:08.812+0000 7ff16651cf40 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55b55fe8f490 con 0x55b55fe8de30 2024-08-03T20:56:08.815 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:08.812+0000 7ff16651cf40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x55b55fe8f770 con 0x55b55fe8bb40 2024-08-03T20:56:08.815 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:08.812+0000 7ff16651cf40 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x55b55fe463a0 con 0x55b55fefebe0 2024-08-03T20:56:08.815 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:08.813+0000 7ff15e200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b55fe8de30 0x55b55ffb3c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:56:08.815 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:08.813+0000 7ff15f600640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55b55fefebe0 0x55b55fe45030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:56:08.815 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:08.813+0000 7ff15e200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b55fe8de30 0x55b55ffb3c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:42752/0 (socket says 172.21.15.32:42752) 2024-08-03T20:56:08.815 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:08.813+0000 7ff15e200640 1 -- 172.21.15.32:0/802940886 learned_addr learned my addr 172.21.15.32:0/802940886 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:56:08.816 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:08.813+0000 7ff15ec00640 1 --2- 172.21.15.32:0/802940886 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55b55fe8bb40 0x55b55fefa410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:56:08.816 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:08.813+0000 7ff15f600640 1 -- 172.21.15.32:0/802940886 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55b55fe8bb40 msgr2=0x55b55fefa410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:56:08.816 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:08.813+0000 7ff15f600640 1 --2- 172.21.15.32:0/802940886 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55b55fe8bb40 0x55b55fefa410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:08.816 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:08.813+0000 7ff15f600640 1 -- 172.21.15.32:0/802940886 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b55fe8de30 msgr2=0x55b55ffb3c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:56:08.816 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:08.813+0000 7ff15f600640 1 --2- 172.21.15.32:0/802940886 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b55fe8de30 0x55b55ffb3c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:08.816 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:08.813+0000 7ff15f600640 1 -- 172.21.15.32:0/802940886 --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b55fefc430 con 0x55b55fefebe0 2024-08-03T20:56:08.816 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:08.813+0000 7ff15ec00640 1 --2- 172.21.15.32:0/802940886 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55b55fe8bb40 0x55b55fefa410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-03T20:56:08.816 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:08.814+0000 7ff15f600640 1 --2- 172.21.15.32:0/802940886 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55b55fefebe0 0x55b55fe45030 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7ff1580106c0 tx=0x7ff158010b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:56:08.817 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:08.814+0000 7ff157400640 1 -- 172.21.15.32:0/802940886 <== mon.2 v2:172.21.15.118:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff158004220 con 0x55b55fefebe0 2024-08-03T20:56:08.817 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:08.814+0000 7ff157400640 1 -- 172.21.15.32:0/802940886 <== mon.2 v2:172.21.15.118:3301/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7ff15800ebf0 con 0x55b55fefebe0 2024-08-03T20:56:08.817 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:08.814+0000 7ff16651cf40 1 -- 172.21.15.32:0/802940886 --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x55b55fefa950 con 0x55b55fefebe0 2024-08-03T20:56:08.817 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:08.814+0000 7ff157400640 1 -- 172.21.15.32:0/802940886 <== mon.2 v2:172.21.15.118:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff158003410 con 0x55b55fefebe0 2024-08-03T20:56:08.817 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:08.814+0000 7ff16651cf40 1 -- 172.21.15.32:0/802940886 --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_subscribe({osdmap=0}) -- 0x55b55fe8eab0 con 0x55b55fefebe0 2024-08-03T20:56:08.818 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:08.817+0000 7ff157400640 1 -- 172.21.15.32:0/802940886 <== mon.2 v2:172.21.15.118:3301/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7ff1580035b0 con 0x55b55fefebe0 2024-08-03T20:56:08.819 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:08.817+0000 7ff157400640 1 --2- 172.21.15.32:0/802940886 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7ff1400759f0 0x7ff140077f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:56:08.819 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:08.818+0000 7ff157400640 1 -- 172.21.15.32:0/802940886 <== mon.2 v2:172.21.15.118:3301/0 5 ==== osd_map(95..95 src has 1..95) ==== 6714+0+0 (secure 0 0 0) 0x7ff158098820 con 0x55b55fefebe0 2024-08-03T20:56:08.820 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:08.818+0000 7ff15ec00640 1 --2- 172.21.15.32:0/802940886 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7ff1400759f0 0x7ff140077f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:56:08.820 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:08.818+0000 7ff16651cf40 1 --2- 172.21.15.32:0/802940886 >> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] conn(0x55b55ff1e520 0x55b55ffb6a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:56:08.820 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:08.818+0000 7ff16651cf40 1 -- 172.21.15.32:0/802940886 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:1 12.13 12:cd5a64a3:::100.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) -- 0x55b55ffb41b0 con 0x55b55ff1e520 2024-08-03T20:56:08.820 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:08.818+0000 7ff15e200640 1 --2- 172.21.15.32:0/802940886 >> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] conn(0x55b55ff1e520 0x55b55ffb6a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:56:08.821 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:08.819+0000 7ff15ec00640 1 --2- 172.21.15.32:0/802940886 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7ff1400759f0 0x7ff140077f00 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7ff150002fd0 tx=0x7ff150000f80 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:56:08.821 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:08.819+0000 7ff15e200640 1 --2- 172.21.15.32:0/802940886 >> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] conn(0x55b55ff1e520 0x55b55ffb6a80 crc :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:56:08.823 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:08.821+0000 7ff15e200640 1 -- 172.21.15.32:0/802940886 <== osd.1 v2:172.21.15.32:6824/1551241759 1 ==== osd_op_reply(1 100.00000000 [delete] v95'103 uv103 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7ff14c04db50 con 0x55b55ff1e520 2024-08-03T20:56:08.823 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:08.821+0000 7ff16651cf40 1 -- 172.21.15.32:0/802940886 >> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] conn(0x55b55ff1e520 msgr2=0x55b55ffb6a80 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:56:08.823 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:08.821+0000 7ff16651cf40 1 --2- 172.21.15.32:0/802940886 >> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] conn(0x55b55ff1e520 0x55b55ffb6a80 crc :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:08.823 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:08.821+0000 7ff16651cf40 1 -- 172.21.15.32:0/802940886 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7ff1400759f0 msgr2=0x7ff140077f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:56:08.823 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:08.821+0000 7ff16651cf40 1 --2- 172.21.15.32:0/802940886 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7ff1400759f0 0x7ff140077f00 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7ff150002fd0 tx=0x7ff150000f80 comp rx=0 tx=0).stop 2024-08-03T20:56:08.823 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:08.821+0000 7ff16651cf40 1 -- 172.21.15.32:0/802940886 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55b55fefebe0 msgr2=0x55b55fe45030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:56:08.823 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:08.821+0000 7ff16651cf40 1 --2- 172.21.15.32:0/802940886 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55b55fefebe0 0x55b55fe45030 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7ff1580106c0 tx=0x7ff158010b80 comp rx=0 tx=0).stop 2024-08-03T20:56:08.823 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:08.821+0000 7ff15f600640 1 -- 172.21.15.32:0/802940886 reap_dead start 2024-08-03T20:56:08.823 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:08.821+0000 7ff16651cf40 1 -- 172.21.15.32:0/802940886 shutdown_connections 2024-08-03T20:56:08.823 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:08.821+0000 7ff16651cf40 1 -- 172.21.15.32:0/802940886 >> 172.21.15.32:0/802940886 conn(0x55b55fd66ac0 msgr2=0x55b55fefc670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:56:08.823 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:08.821+0000 7ff16651cf40 1 -- 172.21.15.32:0/802940886 shutdown_connections 2024-08-03T20:56:08.824 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:08.821+0000 7ff16651cf40 1 -- 172.21.15.32:0/802940886 wait complete. 2024-08-03T20:56:08.834 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs joinable true 2024-08-03T20:56:09.008 INFO:tasks.ceph.mds.a.smithi032.stderr: -51> 2024-08-03T20:56:04.253+0000 7f53fa200640 -1 mds.0.inotable: load_2 could not read table: -2 2024-08-03T20:56:09.008 INFO:tasks.ceph.mds.a.smithi032.stderr: -50> 2024-08-03T20:56:04.254+0000 7f53fa200640 -1 log_channel(cluster) log [ERR] : error reading table object 'mds0_inotable' -2 ((2) No such file or directory) 2024-08-03T20:56:09.080 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:09.077+0000 7f063b400640 1 -- 172.21.15.32:0/3874721971 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f06340cfa10 msgr2=0x7f063406f1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:56:09.080 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:09.077+0000 7f063b400640 1 --2- 172.21.15.32:0/3874721971 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f06340cfa10 0x7f063406f1e0 secure :-1 s=READY pgs=1314 cs=0 l=1 rev1=1 crypto rx=0x7f06340703f0 tx=0x7f062002b930 comp rx=0 tx=0).stop 2024-08-03T20:56:09.080 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:09.077+0000 7f063b400640 1 -- 172.21.15.32:0/3874721971 shutdown_connections 2024-08-03T20:56:09.080 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:09.077+0000 7f063b400640 1 --2- 172.21.15.32:0/3874721971 >> v2:172.21.15.118:3300/0 conn(0x7f063406f720 0x7f06341511b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:09.080 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:09.077+0000 7f063b400640 1 --2- 172.21.15.32:0/3874721971 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f06340cfa10 0x7f063406f1e0 unknown :-1 s=CLOSED pgs=1314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:09.080 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:09.077+0000 7f063b400640 1 -- 172.21.15.32:0/3874721971 >> 172.21.15.32:0/3874721971 conn(0x7f06340b1bd0 msgr2=0x7f06340b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:56:09.081 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:09.078+0000 7f063b400640 1 -- 172.21.15.32:0/3874721971 shutdown_connections 2024-08-03T20:56:09.081 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:09.078+0000 7f063b400640 1 -- 172.21.15.32:0/3874721971 wait complete. 2024-08-03T20:56:09.081 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:09.080+0000 7f063b400640 1 Processor -- start 2024-08-03T20:56:09.081 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:09.080+0000 7f063b400640 1 -- start start 2024-08-03T20:56:09.084 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:09.081+0000 7f063b400640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f063406f720 0x7f06341f5f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:56:09.084 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:09.082+0000 7f063b400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f06340727f0 0x7f0634151b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:56:09.084 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:09.082+0000 7f063b400640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f06340cfa10 0x7f06341eac70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:56:09.084 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:09.082+0000 7f063b400640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f063406ef30 con 0x7f06340727f0 2024-08-03T20:56:09.084 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:09.082+0000 7f063b400640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f063406edb0 con 0x7f063406f720 2024-08-03T20:56:09.084 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:09.082+0000 7f063b400640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7f06341516f0 con 0x7f06340cfa10 2024-08-03T20:56:09.084 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:09.082+0000 7f0633e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f06340727f0 0x7f0634151b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:56:09.084 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:09.082+0000 7f0633e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f06340727f0 0x7f0634151b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:42776/0 (socket says 172.21.15.32:42776) 2024-08-03T20:56:09.084 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:09.082+0000 7f0633e00640 1 -- 172.21.15.32:0/2724698509 learned_addr learned my addr 172.21.15.32:0/2724698509 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:56:09.084 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:09.082+0000 7f0639400640 1 --2- 172.21.15.32:0/2724698509 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f06340cfa10 0x7f06341eac70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:56:09.085 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:09.082+0000 7f0638a00640 1 --2- 172.21.15.32:0/2724698509 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f063406f720 0x7f06341f5f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:56:09.085 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:09.082+0000 7f0633e00640 1 -- 172.21.15.32:0/2724698509 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f06340cfa10 msgr2=0x7f06341eac70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:56:09.085 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:09.082+0000 7f0633e00640 1 --2- 172.21.15.32:0/2724698509 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f06340cfa10 0x7f06341eac70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:09.085 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:09.082+0000 7f0633e00640 1 -- 172.21.15.32:0/2724698509 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f063406f720 msgr2=0x7f06341f5f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:56:09.085 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:09.082+0000 7f0633e00640 1 --2- 172.21.15.32:0/2724698509 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f063406f720 0x7f06341f5f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:09.085 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:09.083+0000 7f0633e00640 1 -- 172.21.15.32:0/2724698509 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f06341f6470 con 0x7f06340727f0 2024-08-03T20:56:09.085 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:09.083+0000 7f0639400640 1 --2- 172.21.15.32:0/2724698509 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f06340cfa10 0x7f06341eac70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-03T20:56:09.086 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:09.083+0000 7f0633e00640 1 --2- 172.21.15.32:0/2724698509 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f06340727f0 0x7f0634151b50 secure :-1 s=READY pgs=1315 cs=0 l=1 rev1=1 crypto rx=0x7f0620034040 tx=0x7f06200042d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:56:09.086 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:09.083+0000 7f0630c00640 1 -- 172.21.15.32:0/2724698509 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f062003e070 con 0x7f06340727f0 2024-08-03T20:56:09.086 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:09.083+0000 7f0630c00640 1 -- 172.21.15.32:0/2724698509 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f0620004ca0 con 0x7f06340727f0 2024-08-03T20:56:09.086 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:09.083+0000 7f0630c00640 1 -- 172.21.15.32:0/2724698509 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0620031850 con 0x7f06340727f0 2024-08-03T20:56:09.086 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:09.083+0000 7f063b400640 1 -- 172.21.15.32:0/2724698509 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0634152090 con 0x7f06340727f0 2024-08-03T20:56:09.086 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:09.084+0000 7f063b400640 1 -- 172.21.15.32:0/2724698509 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f06341e9010 con 0x7f06340727f0 2024-08-03T20:56:09.087 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:09.085+0000 7f063b400640 1 -- 172.21.15.32:0/2724698509 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0634154730 con 0x7f06340727f0 2024-08-03T20:56:09.087 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:09.086+0000 7f0630c00640 1 -- 172.21.15.32:0/2724698509 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f0620038070 con 0x7f06340727f0 2024-08-03T20:56:09.088 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:09.087+0000 7f0630c00640 1 --2- 172.21.15.32:0/2724698509 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f0614077010 0x7f0614079520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:56:09.089 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:09.087+0000 7f0630c00640 1 -- 172.21.15.32:0/2724698509 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 6714+0+0 (secure 0 0 0) 0x7f06200b99e0 con 0x7f06340727f0 2024-08-03T20:56:09.089 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:09.088+0000 7f0638a00640 1 --2- 172.21.15.32:0/2724698509 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f0614077010 0x7f0614079520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:56:09.090 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:09.089+0000 7f0638a00640 1 --2- 172.21.15.32:0/2724698509 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f0614077010 0x7f0614079520 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7f0628001720 tx=0x7f0628009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:56:09.097 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:09.096+0000 7f0630c00640 1 -- 172.21.15.32:0/2724698509 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7f06341f6470 con 0x7f06340727f0 2024-08-03T20:56:09.142 INFO:tasks.ceph.mds.a.smithi032.stdout:starting mds.a at 2024-08-03T20:56:09.349 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:09.347+0000 7f063b400640 1 -- 172.21.15.32:0/2724698509 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "joinable", "val": "true"} v 0) -- 0x7f06341511b0 con 0x7f06340727f0 2024-08-03T20:56:10.027 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:10.025+0000 7f0630c00640 1 -- 172.21.15.32:0/2724698509 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "joinable", "val": "true"}]=0 cephfs marked joinable; MDS may join as newly active. v152) ==== 162+0+0 (secure 0 0 0) 0x7f0620082760 con 0x7f06340727f0 2024-08-03T20:56:10.027 INFO:teuthology.orchestra.run.smithi032.stderr:cephfs marked joinable; MDS may join as newly active. 2024-08-03T20:56:10.029 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:10.027+0000 7f063b400640 1 -- 172.21.15.32:0/2724698509 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f0614077010 msgr2=0x7f0614079520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:56:10.029 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:10.027+0000 7f063b400640 1 --2- 172.21.15.32:0/2724698509 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f0614077010 0x7f0614079520 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7f0628001720 tx=0x7f0628009040 comp rx=0 tx=0).stop 2024-08-03T20:56:10.029 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:10.027+0000 7f063b400640 1 -- 172.21.15.32:0/2724698509 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f06340727f0 msgr2=0x7f0634151b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:56:10.029 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:10.027+0000 7f063b400640 1 --2- 172.21.15.32:0/2724698509 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f06340727f0 0x7f0634151b50 secure :-1 s=READY pgs=1315 cs=0 l=1 rev1=1 crypto rx=0x7f0620034040 tx=0x7f06200042d0 comp rx=0 tx=0).stop 2024-08-03T20:56:10.030 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:10.027+0000 7f063b400640 1 -- 172.21.15.32:0/2724698509 shutdown_connections 2024-08-03T20:56:10.030 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:10.027+0000 7f063b400640 1 --2- 172.21.15.32:0/2724698509 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f0614077010 0x7f0614079520 unknown :-1 s=CLOSED pgs=542 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:10.030 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:10.027+0000 7f063b400640 1 --2- 172.21.15.32:0/2724698509 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f06340cfa10 0x7f06341eac70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:10.030 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:10.027+0000 7f063b400640 1 --2- 172.21.15.32:0/2724698509 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f06340727f0 0x7f0634151b50 unknown :-1 s=CLOSED pgs=1315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:10.030 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:10.027+0000 7f063b400640 1 --2- 172.21.15.32:0/2724698509 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f063406f720 0x7f06341f5f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:10.030 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:10.028+0000 7f063b400640 1 -- 172.21.15.32:0/2724698509 >> 172.21.15.32:0/2724698509 conn(0x7f06340b1bd0 msgr2=0x7f063415a760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:56:10.030 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:10.028+0000 7f063b400640 1 -- 172.21.15.32:0/2724698509 shutdown_connections 2024-08-03T20:56:10.030 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:10.028+0000 7f063b400640 1 -- 172.21.15.32:0/2724698509 wait complete. 2024-08-03T20:56:10.058 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-08-03T20:56:10.287 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:10.285+0000 7f67e4400640 1 -- 172.21.15.32:0/12427536 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f67dc0b38b0 msgr2=0x7f67dc1523f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:56:10.287 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:10.285+0000 7f67e4400640 1 --2- 172.21.15.32:0/12427536 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f67dc0b38b0 0x7f67dc1523f0 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f67d0009830 tx=0x7f67d002a520 comp rx=0 tx=0).stop 2024-08-03T20:56:10.288 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:10.286+0000 7f67e4400640 1 -- 172.21.15.32:0/12427536 shutdown_connections 2024-08-03T20:56:10.288 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:10.286+0000 7f67e4400640 1 --2- 172.21.15.32:0/12427536 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f67dc0b38b0 0x7f67dc1523f0 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:10.288 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:10.286+0000 7f67e4400640 1 -- 172.21.15.32:0/12427536 >> 172.21.15.32:0/12427536 conn(0x7f67dc0b1bd0 msgr2=0x7f67dc0b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:56:10.288 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:10.286+0000 7f67e4400640 1 -- 172.21.15.32:0/12427536 shutdown_connections 2024-08-03T20:56:10.288 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:10.286+0000 7f67e4400640 1 -- 172.21.15.32:0/12427536 wait complete. 2024-08-03T20:56:10.289 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:10.288+0000 7f67e4400640 1 Processor -- start 2024-08-03T20:56:10.289 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:10.288+0000 7f67e4400640 1 -- start start 2024-08-03T20:56:10.292 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:10.290+0000 7f67e4400640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f67dc0b38b0 0x7f67dc158980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:56:10.292 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:10.290+0000 7f67e4400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f67dc0b4ce0 0x7f67dc14d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:56:10.292 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:10.290+0000 7f67e4400640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f67dc1f6bc0 0x7f67dc1f6fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:56:10.293 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:10.290+0000 7f67e4400640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f67dc0cec60 con 0x7f67dc0b4ce0 2024-08-03T20:56:10.293 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:10.290+0000 7f67e4400640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f67dc0ceae0 con 0x7f67dc0b38b0 2024-08-03T20:56:10.293 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:10.290+0000 7f67e4400640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7f67dc0cede0 con 0x7f67dc1f6bc0 2024-08-03T20:56:10.293 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:10.290+0000 7f67e1000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f67dc0b4ce0 0x7f67dc14d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:56:10.293 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:10.290+0000 7f67e1000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f67dc0b4ce0 0x7f67dc14d810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:46924/0 (socket says 172.21.15.32:46924) 2024-08-03T20:56:10.293 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:10.290+0000 7f67e1000640 1 -- 172.21.15.32:0/3921212495 learned_addr learned my addr 172.21.15.32:0/3921212495 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:56:10.293 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:10.290+0000 7f67e1a00640 1 --2- 172.21.15.32:0/3921212495 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f67dc0b38b0 0x7f67dc158980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:56:10.293 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:10.290+0000 7f67e2400640 1 --2- 172.21.15.32:0/3921212495 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f67dc1f6bc0 0x7f67dc1f6fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:56:10.293 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:10.290+0000 7f67e1000640 1 -- 172.21.15.32:0/3921212495 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f67dc1f6bc0 msgr2=0x7f67dc1f6fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:56:10.293 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:10.290+0000 7f67e1000640 1 --2- 172.21.15.32:0/3921212495 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f67dc1f6bc0 0x7f67dc1f6fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:10.293 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:10.290+0000 7f67e1000640 1 -- 172.21.15.32:0/3921212495 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f67dc0b38b0 msgr2=0x7f67dc158980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:56:10.293 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:10.290+0000 7f67e1000640 1 --2- 172.21.15.32:0/3921212495 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f67dc0b38b0 0x7f67dc158980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:10.293 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:10.290+0000 7f67e1000640 1 -- 172.21.15.32:0/3921212495 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f67dc0aee90 con 0x7f67dc0b4ce0 2024-08-03T20:56:10.294 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:10.291+0000 7f67e1000640 1 --2- 172.21.15.32:0/3921212495 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f67dc0b4ce0 0x7f67dc14d810 secure :-1 s=READY pgs=1318 cs=0 l=1 rev1=1 crypto rx=0x7f67c800c400 tx=0x7f67c800c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:56:10.294 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:10.291+0000 7f67d9600640 1 -- 172.21.15.32:0/3921212495 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67c8011790 con 0x7f67dc0b4ce0 2024-08-03T20:56:10.294 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:10.291+0000 7f67d9600640 1 -- 172.21.15.32:0/3921212495 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f67c8017070 con 0x7f67dc0b4ce0 2024-08-03T20:56:10.294 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:10.291+0000 7f67e4400640 1 -- 172.21.15.32:0/3921212495 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f67dc0af130 con 0x7f67dc0b4ce0 2024-08-03T20:56:10.294 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:10.291+0000 7f67d9600640 1 -- 172.21.15.32:0/3921212495 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67c8011ce0 con 0x7f67dc0b4ce0 2024-08-03T20:56:10.294 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:10.291+0000 7f67e4400640 1 -- 172.21.15.32:0/3921212495 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f67dc157980 con 0x7f67dc0b4ce0 2024-08-03T20:56:10.294 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:10.292+0000 7f67e4400640 1 -- 172.21.15.32:0/3921212495 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f67dc1523f0 con 0x7f67dc0b4ce0 2024-08-03T20:56:10.295 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:10.294+0000 7f67d9600640 1 -- 172.21.15.32:0/3921212495 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f67c801c460 con 0x7f67dc0b4ce0 2024-08-03T20:56:10.296 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:10.294+0000 7f67d9600640 1 --2- 172.21.15.32:0/3921212495 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f67bc077010 0x7f67bc079520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:56:10.296 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:10.295+0000 7f67d9600640 1 -- 172.21.15.32:0/3921212495 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 6714+0+0 (secure 0 0 0) 0x7f67c8098a60 con 0x7f67dc0b4ce0 2024-08-03T20:56:10.296 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:10.295+0000 7f67e1a00640 1 --2- 172.21.15.32:0/3921212495 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f67bc077010 0x7f67bc079520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:56:10.297 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:10.296+0000 7f67e1a00640 1 --2- 172.21.15.32:0/3921212495 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f67bc077010 0x7f67bc079520 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7f67d0034000 tx=0x7f67d0004730 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:56:10.300 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:10.299+0000 7f67d9600640 1 -- 172.21.15.32:0/3921212495 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7f67c8061210 con 0x7f67dc0b4ce0 2024-08-03T20:56:10.544 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:10.543+0000 7f67e4400640 1 -- 172.21.15.32:0/3921212495 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f67dc158f40 con 0x7f67dc0b4ce0 2024-08-03T20:56:10.545 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:10.543+0000 7f67d9600640 1 -- 172.21.15.32:0/3921212495 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 152 v152) ==== 95+0+5225 (secure 0 0 0) 0x7f67c8065760 con 0x7f67dc0b4ce0 2024-08-03T20:56:10.545 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-03T20:56:10.545 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":152,"btime":"2024-08-03T20:56:10:024076+0000","default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6439,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6835/531080160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6834","nonce":531080160},{"type":"v1","addr":"172.21.15.118:6835","nonce":531080160}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":84},{"gid":6454,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6837/2796882319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6836","nonce":2796882319},{"type":"v1","addr":"172.21.15.118:6837","nonce":2796882319}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":86},{"gid":6813,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6834/2686933890","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":2686933890},{"type":"v1","addr":"172.21.15.32:6834","nonce":2686933890}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":85}],"filesystems":[{"mdsmap":{"epoch":152,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-08-03T20:54:26.950518+0000","modified":"2024-08-03T20:56:10.024060+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":95,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6884},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6884":{"gid":6884,"name":"a","rank":0,"incarnation":152,"state":"up:replay","state_seq":1,"addr":"172.21.15.32:6835/2278473729","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6833","nonce":2278473729},{"type":"v1","addr":"172.21.15.32:6835","nonce":2278473729}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":6}]} 2024-08-03T20:56:10.546 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 152 2024-08-03T20:56:10.548 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:10.546+0000 7f67e4400640 1 -- 172.21.15.32:0/3921212495 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f67bc077010 msgr2=0x7f67bc079520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:56:10.548 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:10.546+0000 7f67e4400640 1 --2- 172.21.15.32:0/3921212495 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f67bc077010 0x7f67bc079520 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7f67d0034000 tx=0x7f67d0004730 comp rx=0 tx=0).stop 2024-08-03T20:56:10.548 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:10.546+0000 7f67e4400640 1 -- 172.21.15.32:0/3921212495 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f67dc0b4ce0 msgr2=0x7f67dc14d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:56:10.548 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:10.546+0000 7f67e4400640 1 --2- 172.21.15.32:0/3921212495 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f67dc0b4ce0 0x7f67dc14d810 secure :-1 s=READY pgs=1318 cs=0 l=1 rev1=1 crypto rx=0x7f67c800c400 tx=0x7f67c800c8c0 comp rx=0 tx=0).stop 2024-08-03T20:56:10.548 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:10.546+0000 7f67e4400640 1 -- 172.21.15.32:0/3921212495 shutdown_connections 2024-08-03T20:56:10.549 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:10.546+0000 7f67e4400640 1 --2- 172.21.15.32:0/3921212495 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f67bc077010 0x7f67bc079520 unknown :-1 s=CLOSED pgs=544 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:10.549 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:10.546+0000 7f67e4400640 1 --2- 172.21.15.32:0/3921212495 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f67dc1f6bc0 0x7f67dc1f6fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:10.549 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:10.546+0000 7f67e4400640 1 --2- 172.21.15.32:0/3921212495 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f67dc0b4ce0 0x7f67dc14d810 unknown :-1 s=CLOSED pgs=1318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:10.549 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:10.546+0000 7f67e4400640 1 --2- 172.21.15.32:0/3921212495 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f67dc0b38b0 0x7f67dc158980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:10.549 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:10.546+0000 7f67e4400640 1 -- 172.21.15.32:0/3921212495 >> 172.21.15.32:0/3921212495 conn(0x7f67dc0b1bd0 msgr2=0x7f67dc0cc160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:56:10.549 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:10.546+0000 7f67e4400640 1 -- 172.21.15.32:0/3921212495 shutdown_connections 2024-08-03T20:56:10.549 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:10.546+0000 7f67e4400640 1 -- 172.21.15.32:0/3921212495 wait complete. 2024-08-03T20:56:10.569 DEBUG:tasks.ceph_test_case:wait_until_true: waiting (timeout=60 retry_count=0)... 2024-08-03T20:56:13.039 INFO:tasks.ceph.mds.a.smithi032.stderr:2024-08-03T20:56:13.037+0000 7f15eac00640 -1 log_channel(cluster) log [ERR] : dir 0x100 object missing on disk; some files may be lost (~mds0) 2024-08-03T20:56:13.042 INFO:tasks.ceph.mon.a.smithi032.stderr:2024-08-03T20:56:13.040+0000 7f4e6d600640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-08-03T20:56:13.042 INFO:tasks.ceph.mon.a.smithi032.stderr:2024-08-03T20:56:13.040+0000 7f4e6d600640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 mds daemon damaged (MDS_DAMAGE) 2024-08-03T20:56:14.288 INFO:tasks.ceph.mds.a.smithi032.stderr:2024-08-03T20:56:14.287+0000 7f15efc00640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-08-03T20:56:15.570 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-08-03T20:56:15.811 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:15.809+0000 7fdb96800640 1 -- 172.21.15.32:0/416723177 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdb9006e930 msgr2=0x7fdb90144f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:56:15.811 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:15.809+0000 7fdb96800640 1 --2- 172.21.15.32:0/416723177 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdb9006e930 0x7fdb90144f90 secure :-1 s=READY pgs=1320 cs=0 l=1 rev1=1 crypto rx=0x7fdb80005fd0 tx=0x7fdb8002a670 comp rx=0 tx=0).stop 2024-08-03T20:56:15.811 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:15.809+0000 7fdb96800640 1 -- 172.21.15.32:0/416723177 shutdown_connections 2024-08-03T20:56:15.811 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:15.809+0000 7fdb96800640 1 --2- 172.21.15.32:0/416723177 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdb9006e930 0x7fdb90144f90 unknown :-1 s=CLOSED pgs=1320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:15.811 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:15.809+0000 7fdb96800640 1 -- 172.21.15.32:0/416723177 >> 172.21.15.32:0/416723177 conn(0x7fdb900b1bd0 msgr2=0x7fdb900b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:56:15.811 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:15.809+0000 7fdb96800640 1 -- 172.21.15.32:0/416723177 shutdown_connections 2024-08-03T20:56:15.812 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:15.809+0000 7fdb96800640 1 -- 172.21.15.32:0/416723177 wait complete. 2024-08-03T20:56:15.812 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:15.811+0000 7fdb96800640 1 Processor -- start 2024-08-03T20:56:15.813 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:15.811+0000 7fdb96800640 1 -- start start 2024-08-03T20:56:15.816 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:15.813+0000 7fdb96800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdb9006e930 0x7fdb90156eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:56:15.816 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:15.813+0000 7fdb96800640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fdb90073e60 0x7fdb90157cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:56:15.816 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:15.813+0000 7fdb96800640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fdb9015fff0 0x7fdb90158200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:56:15.816 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:15.813+0000 7fdb96800640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fdb900cf840 con 0x7fdb9006e930 2024-08-03T20:56:15.816 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:15.813+0000 7fdb96800640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7fdb90145f30 con 0x7fdb9015fff0 2024-08-03T20:56:15.816 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:15.813+0000 7fdb96800640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7fdb900cf9c0 con 0x7fdb90073e60 2024-08-03T20:56:15.816 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:15.813+0000 7fdb8f400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdb9006e930 0x7fdb90156eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:56:15.816 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:15.813+0000 7fdb8f400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdb9006e930 0x7fdb90156eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:46930/0 (socket says 172.21.15.32:46930) 2024-08-03T20:56:15.816 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:15.813+0000 7fdb8f400640 1 -- 172.21.15.32:0/3581181406 learned_addr learned my addr 172.21.15.32:0/3581181406 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:56:15.816 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:15.813+0000 7fdb87e00640 1 --2- 172.21.15.32:0/3581181406 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fdb90073e60 0x7fdb90157cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:56:15.816 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:15.813+0000 7fdb8fe00640 1 --2- 172.21.15.32:0/3581181406 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fdb9015fff0 0x7fdb90158200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:56:15.816 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:15.814+0000 7fdb8f400640 1 -- 172.21.15.32:0/3581181406 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fdb90073e60 msgr2=0x7fdb90157cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:56:15.817 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:15.814+0000 7fdb8f400640 1 --2- 172.21.15.32:0/3581181406 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fdb90073e60 0x7fdb90157cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:15.817 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:15.814+0000 7fdb8f400640 1 -- 172.21.15.32:0/3581181406 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fdb9015fff0 msgr2=0x7fdb90158200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:56:15.817 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:15.814+0000 7fdb8f400640 1 --2- 172.21.15.32:0/3581181406 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fdb9015fff0 0x7fdb90158200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:15.817 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:15.814+0000 7fdb8f400640 1 -- 172.21.15.32:0/3581181406 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdb90143af0 con 0x7fdb9006e930 2024-08-03T20:56:15.817 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:15.814+0000 7fdb8fe00640 1 --2- 172.21.15.32:0/3581181406 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fdb9015fff0 0x7fdb90158200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-03T20:56:15.817 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:15.814+0000 7fdb8f400640 1 --2- 172.21.15.32:0/3581181406 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdb9006e930 0x7fdb90156eb0 secure :-1 s=READY pgs=1321 cs=0 l=1 rev1=1 crypto rx=0x7fdb80034040 tx=0x7fdb800039e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:56:15.817 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:15.814+0000 7fdb87400640 1 -- 172.21.15.32:0/3581181406 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb8003e030 con 0x7fdb9006e930 2024-08-03T20:56:15.817 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:15.814+0000 7fdb87400640 1 -- 172.21.15.32:0/3581181406 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fdb800047a0 con 0x7fdb9006e930 2024-08-03T20:56:15.817 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:15.814+0000 7fdb96800640 1 -- 172.21.15.32:0/3581181406 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdb901f3740 con 0x7fdb9006e930 2024-08-03T20:56:15.817 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:15.814+0000 7fdb87400640 1 -- 172.21.15.32:0/3581181406 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb80035040 con 0x7fdb9006e930 2024-08-03T20:56:15.817 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:15.814+0000 7fdb96800640 1 -- 172.21.15.32:0/3581181406 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdb90143810 con 0x7fdb9006e930 2024-08-03T20:56:15.818 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:15.815+0000 7fdb96800640 1 -- 172.21.15.32:0/3581181406 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdb90070880 con 0x7fdb9006e930 2024-08-03T20:56:15.818 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:15.817+0000 7fdb87400640 1 -- 172.21.15.32:0/3581181406 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7fdb80041030 con 0x7fdb9006e930 2024-08-03T20:56:15.819 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:15.817+0000 7fdb87400640 1 --2- 172.21.15.32:0/3581181406 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fdb6c077010 0x7fdb6c079520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:56:15.819 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:15.818+0000 7fdb87400640 1 -- 172.21.15.32:0/3581181406 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 6800+0+0 (secure 0 0 0) 0x7fdb800b9650 con 0x7fdb9006e930 2024-08-03T20:56:15.819 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:15.818+0000 7fdb87e00640 1 --2- 172.21.15.32:0/3581181406 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fdb6c077010 0x7fdb6c079520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:56:15.821 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:15.820+0000 7fdb87e00640 1 --2- 172.21.15.32:0/3581181406 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fdb6c077010 0x7fdb6c079520 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7fdb9015feb0 tx=0x7fdb78009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:56:15.823 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:15.822+0000 7fdb87400640 1 -- 172.21.15.32:0/3581181406 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7fdb90070880 con 0x7fdb9006e930 2024-08-03T20:56:16.074 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:16.072+0000 7fdb96800640 1 -- 172.21.15.32:0/3581181406 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fdb90160e20 con 0x7fdb9006e930 2024-08-03T20:56:16.074 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:16.073+0000 7fdb87400640 1 -- 172.21.15.32:0/3581181406 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 156 v156) ==== 95+0+4390 (secure 0 0 0) 0x7fdb80081d40 con 0x7fdb9006e930 2024-08-03T20:56:16.075 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-03T20:56:16.075 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":156,"btime":"2024-08-03T20:56:13:041986+0000","default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6439,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6835/531080160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6834","nonce":531080160},{"type":"v1","addr":"172.21.15.118:6835","nonce":531080160}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":84},{"gid":6454,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6837/2796882319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6836","nonce":2796882319},{"type":"v1","addr":"172.21.15.118:6837","nonce":2796882319}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":86},{"gid":6813,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6834/2686933890","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":2686933890},{"type":"v1","addr":"172.21.15.32:6834","nonce":2686933890}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":85}],"filesystems":[{"mdsmap":{"epoch":156,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-08-03T20:54:26.950518+0000","modified":"2024-08-03T20:56:13.041980+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":96,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":6}]} 2024-08-03T20:56:16.075 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 156 2024-08-03T20:56:16.077 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:16.075+0000 7fdb96800640 1 -- 172.21.15.32:0/3581181406 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fdb6c077010 msgr2=0x7fdb6c079520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:56:16.077 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:16.075+0000 7fdb96800640 1 --2- 172.21.15.32:0/3581181406 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fdb6c077010 0x7fdb6c079520 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7fdb9015feb0 tx=0x7fdb78009040 comp rx=0 tx=0).stop 2024-08-03T20:56:16.077 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:16.075+0000 7fdb96800640 1 -- 172.21.15.32:0/3581181406 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdb9006e930 msgr2=0x7fdb90156eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:56:16.077 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:16.075+0000 7fdb96800640 1 --2- 172.21.15.32:0/3581181406 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdb9006e930 0x7fdb90156eb0 secure :-1 s=READY pgs=1321 cs=0 l=1 rev1=1 crypto rx=0x7fdb80034040 tx=0x7fdb800039e0 comp rx=0 tx=0).stop 2024-08-03T20:56:16.077 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:16.075+0000 7fdb96800640 1 -- 172.21.15.32:0/3581181406 shutdown_connections 2024-08-03T20:56:16.077 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:16.075+0000 7fdb96800640 1 --2- 172.21.15.32:0/3581181406 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fdb6c077010 0x7fdb6c079520 unknown :-1 s=CLOSED pgs=545 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:16.077 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:16.075+0000 7fdb96800640 1 --2- 172.21.15.32:0/3581181406 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fdb9015fff0 0x7fdb90158200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:16.077 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:16.075+0000 7fdb96800640 1 --2- 172.21.15.32:0/3581181406 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fdb90073e60 0x7fdb90157cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:16.077 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:16.075+0000 7fdb96800640 1 --2- 172.21.15.32:0/3581181406 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdb9006e930 0x7fdb90156eb0 unknown :-1 s=CLOSED pgs=1321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:16.077 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:16.075+0000 7fdb96800640 1 -- 172.21.15.32:0/3581181406 >> 172.21.15.32:0/3581181406 conn(0x7fdb900b1bd0 msgr2=0x7fdb900b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:56:16.078 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:16.075+0000 7fdb96800640 1 -- 172.21.15.32:0/3581181406 shutdown_connections 2024-08-03T20:56:16.078 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:16.075+0000 7fdb96800640 1 -- 172.21.15.32:0/3581181406 wait complete. 2024-08-03T20:56:16.099 DEBUG:tasks.ceph_test_case:wait_until_true: success in 5s and 0 retries 2024-08-03T20:56:16.099 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-08-03T20:56:16.329 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:16.327+0000 7fe90be00640 1 -- 172.21.15.32:0/641470257 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe90c0d26e0 msgr2=0x7fe90c161b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:56:16.329 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:16.327+0000 7fe90be00640 1 --2- 172.21.15.32:0/641470257 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe90c0d26e0 0x7fe90c161b80 secure :-1 s=READY pgs=1322 cs=0 l=1 rev1=1 crypto rx=0x7fe90c0d3730 tx=0x7fe8fc02a560 comp rx=0 tx=0).stop 2024-08-03T20:56:16.329 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:16.327+0000 7fe90be00640 1 -- 172.21.15.32:0/641470257 shutdown_connections 2024-08-03T20:56:16.329 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:16.327+0000 7fe90be00640 1 --2- 172.21.15.32:0/641470257 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fe90c0b43d0 0x7fe90c0cf510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:16.329 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:16.327+0000 7fe90be00640 1 --2- 172.21.15.32:0/641470257 >> v2:172.21.15.118:3300/0 conn(0x7fe90c0cffe0 0x7fe90c0b3e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:16.329 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:16.327+0000 7fe90be00640 1 --2- 172.21.15.32:0/641470257 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe90c0d26e0 0x7fe90c161b80 unknown :-1 s=CLOSED pgs=1322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:16.329 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:16.327+0000 7fe90be00640 1 -- 172.21.15.32:0/641470257 >> 172.21.15.32:0/641470257 conn(0x7fe90c08d7a0 msgr2=0x7fe90c09d3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:56:16.329 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:16.327+0000 7fe90be00640 1 -- 172.21.15.32:0/641470257 shutdown_connections 2024-08-03T20:56:16.329 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:16.327+0000 7fe90be00640 1 -- 172.21.15.32:0/641470257 wait complete. 2024-08-03T20:56:16.330 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:16.329+0000 7fe90be00640 1 Processor -- start 2024-08-03T20:56:16.331 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:16.329+0000 7fe90be00640 1 -- start start 2024-08-03T20:56:16.333 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:16.331+0000 7fe90be00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fe90c0b43d0 0x7fe90c1568d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:56:16.334 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:16.331+0000 7fe90be00640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fe90c0cffe0 0x7fe90c0c8e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:56:16.334 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:16.331+0000 7fe90be00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe90c0d26e0 0x7fe90c0cd830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:56:16.334 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:16.331+0000 7fe90be00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fe90c0d3f80 con 0x7fe90c0d26e0 2024-08-03T20:56:16.334 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:16.331+0000 7fe90be00640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7fe90c0cfce0 con 0x7fe90c0b43d0 2024-08-03T20:56:16.334 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:16.331+0000 7fe90be00640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7fe90c0d4100 con 0x7fe90c0cffe0 2024-08-03T20:56:16.334 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:16.331+0000 7fe90b400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe90c0d26e0 0x7fe90c0cd830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:56:16.334 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:16.331+0000 7fe90b400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe90c0d26e0 0x7fe90c0cd830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:46950/0 (socket says 172.21.15.32:46950) 2024-08-03T20:56:16.334 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:16.331+0000 7fe90b400640 1 -- 172.21.15.32:0/3523035535 learned_addr learned my addr 172.21.15.32:0/3523035535 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:56:16.334 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:16.331+0000 7fe90aa00640 1 --2- 172.21.15.32:0/3523035535 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fe90c0b43d0 0x7fe90c1568d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:56:16.334 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:16.331+0000 7fe903e00640 1 --2- 172.21.15.32:0/3523035535 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fe90c0cffe0 0x7fe90c0c8e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:56:16.334 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:16.331+0000 7fe90b400640 1 -- 172.21.15.32:0/3523035535 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fe90c0cffe0 msgr2=0x7fe90c0c8e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:56:16.335 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:16.331+0000 7fe90b400640 1 --2- 172.21.15.32:0/3523035535 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fe90c0cffe0 0x7fe90c0c8e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:16.335 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:16.331+0000 7fe90b400640 1 -- 172.21.15.32:0/3523035535 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fe90c0b43d0 msgr2=0x7fe90c1568d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:56:16.335 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:16.331+0000 7fe90b400640 1 --2- 172.21.15.32:0/3523035535 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fe90c0b43d0 0x7fe90c1568d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:16.335 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:16.331+0000 7fe90b400640 1 -- 172.21.15.32:0/3523035535 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe90c0cb8b0 con 0x7fe90c0d26e0 2024-08-03T20:56:16.335 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:16.331+0000 7fe903e00640 1 --2- 172.21.15.32:0/3523035535 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fe90c0cffe0 0x7fe90c0c8e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-03T20:56:16.335 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:16.332+0000 7fe90b400640 1 --2- 172.21.15.32:0/3523035535 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe90c0d26e0 0x7fe90c0cd830 secure :-1 s=READY pgs=1323 cs=0 l=1 rev1=1 crypto rx=0x7fe90400e9d0 tx=0x7fe90400ee90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:56:16.335 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:16.332+0000 7fe902a00640 1 -- 172.21.15.32:0/3523035535 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe90400cd80 con 0x7fe90c0d26e0 2024-08-03T20:56:16.335 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:16.332+0000 7fe902a00640 1 -- 172.21.15.32:0/3523035535 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fe904004b20 con 0x7fe90c0d26e0 2024-08-03T20:56:16.335 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:16.332+0000 7fe90be00640 1 -- 172.21.15.32:0/3523035535 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe90c0cbb90 con 0x7fe90c0d26e0 2024-08-03T20:56:16.335 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:16.332+0000 7fe902a00640 1 -- 172.21.15.32:0/3523035535 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe904005020 con 0x7fe90c0d26e0 2024-08-03T20:56:16.335 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:16.332+0000 7fe90be00640 1 -- 172.21.15.32:0/3523035535 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe90c0cc150 con 0x7fe90c0d26e0 2024-08-03T20:56:16.336 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:16.333+0000 7fe90be00640 1 -- 172.21.15.32:0/3523035535 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe90c0cb620 con 0x7fe90c0d26e0 2024-08-03T20:56:16.336 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:16.335+0000 7fe902a00640 1 -- 172.21.15.32:0/3523035535 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7fe9040051c0 con 0x7fe90c0d26e0 2024-08-03T20:56:16.337 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:16.335+0000 7fe902a00640 1 --2- 172.21.15.32:0/3523035535 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fe8e8077010 0x7fe8e8079520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:56:16.337 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:16.336+0000 7fe902a00640 1 -- 172.21.15.32:0/3523035535 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 6800+0+0 (secure 0 0 0) 0x7fe904099780 con 0x7fe90c0d26e0 2024-08-03T20:56:16.337 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:16.336+0000 7fe90aa00640 1 --2- 172.21.15.32:0/3523035535 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fe8e8077010 0x7fe8e8079520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:56:16.338 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:16.337+0000 7fe90aa00640 1 --2- 172.21.15.32:0/3523035535 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fe8e8077010 0x7fe8e8079520 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7fe8fc033000 tx=0x7fe8fc004730 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:56:16.341 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:16.340+0000 7fe902a00640 1 -- 172.21.15.32:0/3523035535 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7fe904062550 con 0x7fe90c0d26e0 2024-08-03T20:56:16.581 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:16.580+0000 7fe90be00640 1 -- 172.21.15.32:0/3523035535 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe90c0b3e90 con 0x7fe90c0d26e0 2024-08-03T20:56:16.582 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:16.581+0000 7fe902a00640 1 -- 172.21.15.32:0/3523035535 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 156 v156) ==== 95+0+4390 (secure 0 0 0) 0x7fe904061e50 con 0x7fe90c0d26e0 2024-08-03T20:56:16.582 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-03T20:56:16.583 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":156,"btime":"2024-08-03T20:56:13:041986+0000","default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6439,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6835/531080160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6834","nonce":531080160},{"type":"v1","addr":"172.21.15.118:6835","nonce":531080160}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":84},{"gid":6454,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6837/2796882319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6836","nonce":2796882319},{"type":"v1","addr":"172.21.15.118:6837","nonce":2796882319}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":86},{"gid":6813,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6834/2686933890","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":2686933890},{"type":"v1","addr":"172.21.15.32:6834","nonce":2686933890}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":85}],"filesystems":[{"mdsmap":{"epoch":156,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-08-03T20:54:26.950518+0000","modified":"2024-08-03T20:56:13.041980+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":96,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":6}]} 2024-08-03T20:56:16.583 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 156 2024-08-03T20:56:16.585 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:16.583+0000 7fe90be00640 1 -- 172.21.15.32:0/3523035535 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fe8e8077010 msgr2=0x7fe8e8079520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:56:16.585 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:16.583+0000 7fe90be00640 1 --2- 172.21.15.32:0/3523035535 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fe8e8077010 0x7fe8e8079520 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7fe8fc033000 tx=0x7fe8fc004730 comp rx=0 tx=0).stop 2024-08-03T20:56:16.585 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:16.583+0000 7fe90be00640 1 -- 172.21.15.32:0/3523035535 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe90c0d26e0 msgr2=0x7fe90c0cd830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:56:16.585 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:16.583+0000 7fe90be00640 1 --2- 172.21.15.32:0/3523035535 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe90c0d26e0 0x7fe90c0cd830 secure :-1 s=READY pgs=1323 cs=0 l=1 rev1=1 crypto rx=0x7fe90400e9d0 tx=0x7fe90400ee90 comp rx=0 tx=0).stop 2024-08-03T20:56:16.585 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:16.583+0000 7fe90be00640 1 -- 172.21.15.32:0/3523035535 shutdown_connections 2024-08-03T20:56:16.585 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:16.583+0000 7fe90be00640 1 --2- 172.21.15.32:0/3523035535 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fe8e8077010 0x7fe8e8079520 unknown :-1 s=CLOSED pgs=546 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:16.585 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:16.583+0000 7fe90be00640 1 --2- 172.21.15.32:0/3523035535 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe90c0d26e0 0x7fe90c0cd830 unknown :-1 s=CLOSED pgs=1323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:16.585 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:16.583+0000 7fe90be00640 1 --2- 172.21.15.32:0/3523035535 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fe90c0cffe0 0x7fe90c0c8e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:16.585 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:16.583+0000 7fe90be00640 1 --2- 172.21.15.32:0/3523035535 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fe90c0b43d0 0x7fe90c1568d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:16.586 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:16.583+0000 7fe90be00640 1 -- 172.21.15.32:0/3523035535 >> 172.21.15.32:0/3523035535 conn(0x7fe90c08d7a0 msgr2=0x7fe90c160360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:56:16.586 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:16.583+0000 7fe90be00640 1 -- 172.21.15.32:0/3523035535 shutdown_connections 2024-08-03T20:56:16.586 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:16.583+0000 7fe90be00640 1 -- 172.21.15.32:0/3523035535 wait complete. 2024-08-03T20:56:16.610 DEBUG:tasks.ceph_test_case:wait_until_true: success in 0s and 0 retries 2024-08-03T20:56:16.610 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-08-03T20:56:16.838 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:16.836+0000 7f8df0e00640 1 -- 172.21.15.32:0/1713889094 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f8dec0d1c60 msgr2=0x7f8dec0cf340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:56:16.838 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:16.836+0000 7f8df0e00640 1 --2- 172.21.15.32:0/1713889094 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f8dec0d1c60 0x7f8dec0cf340 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f8de4009be0 tx=0x7f8de402a820 comp rx=0 tx=0).stop 2024-08-03T20:56:16.838 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:16.836+0000 7f8df0e00640 1 -- 172.21.15.32:0/1713889094 shutdown_connections 2024-08-03T20:56:16.838 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:16.836+0000 7f8df0e00640 1 --2- 172.21.15.32:0/1713889094 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f8dec0d1c60 0x7f8dec0cf340 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:16.838 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:16.836+0000 7f8df0e00640 1 -- 172.21.15.32:0/1713889094 >> 172.21.15.32:0/1713889094 conn(0x7f8dec0d4090 msgr2=0x7f8dec0d44a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:56:16.838 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:16.836+0000 7f8df0e00640 1 -- 172.21.15.32:0/1713889094 shutdown_connections 2024-08-03T20:56:16.839 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:16.836+0000 7f8df0e00640 1 -- 172.21.15.32:0/1713889094 wait complete. 2024-08-03T20:56:16.840 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:16.838+0000 7f8df0e00640 1 Processor -- start 2024-08-03T20:56:16.840 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:16.839+0000 7f8df0e00640 1 -- start start 2024-08-03T20:56:16.843 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:16.840+0000 7f8df0e00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8dec0d1c60 0x7f8dec15fa10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:56:16.843 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:16.840+0000 7f8df0e00640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f8dec0d4bb0 0x7f8dec167910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:56:16.843 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:16.840+0000 7f8df0e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8dec1698d0 0x7f8dec167e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:56:16.843 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:16.840+0000 7f8df0e00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f8dec0cfd50 con 0x7f8dec1698d0 2024-08-03T20:56:16.843 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:16.840+0000 7f8df0e00640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f8dec0cfbd0 con 0x7f8dec0d1c60 2024-08-03T20:56:16.843 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:16.840+0000 7f8df0e00640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7f8dec0cfed0 con 0x7f8dec0d4bb0 2024-08-03T20:56:16.843 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:16.840+0000 7f8debe00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8dec1698d0 0x7f8dec167e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:56:16.843 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:16.840+0000 7f8debe00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8dec1698d0 0x7f8dec167e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:46966/0 (socket says 172.21.15.32:46966) 2024-08-03T20:56:16.843 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:16.840+0000 7f8debe00640 1 -- 172.21.15.32:0/1218080528 learned_addr learned my addr 172.21.15.32:0/1218080528 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:56:16.843 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:16.841+0000 7f8deb400640 1 --2- 172.21.15.32:0/1218080528 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8dec0d1c60 0x7f8dec15fa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:56:16.843 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:16.841+0000 7f8de3e00640 1 --2- 172.21.15.32:0/1218080528 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f8dec0d4bb0 0x7f8dec167910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:56:16.844 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:16.841+0000 7f8debe00640 1 -- 172.21.15.32:0/1218080528 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f8dec0d4bb0 msgr2=0x7f8dec167910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:56:16.844 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:16.841+0000 7f8debe00640 1 --2- 172.21.15.32:0/1218080528 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f8dec0d4bb0 0x7f8dec167910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:16.844 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:16.841+0000 7f8debe00640 1 -- 172.21.15.32:0/1218080528 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8dec0d1c60 msgr2=0x7f8dec15fa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:56:16.844 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:16.841+0000 7f8debe00640 1 --2- 172.21.15.32:0/1218080528 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8dec0d1c60 0x7f8dec15fa10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:16.844 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:16.841+0000 7f8debe00640 1 -- 172.21.15.32:0/1218080528 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8dec0cd2b0 con 0x7f8dec1698d0 2024-08-03T20:56:16.844 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:16.841+0000 7f8deb400640 1 --2- 172.21.15.32:0/1218080528 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8dec0d1c60 0x7f8dec15fa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-03T20:56:16.844 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:16.841+0000 7f8debe00640 1 --2- 172.21.15.32:0/1218080528 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8dec1698d0 0x7f8dec167e50 secure :-1 s=READY pgs=1325 cs=0 l=1 rev1=1 crypto rx=0x7f8de4009be0 tx=0x7f8de4032c60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:56:16.845 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:16.842+0000 7f8de3400640 1 -- 172.21.15.32:0/1218080528 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8de4041020 con 0x7f8dec1698d0 2024-08-03T20:56:16.845 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:16.842+0000 7f8de3400640 1 -- 172.21.15.32:0/1218080528 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f8de4032e50 con 0x7f8dec1698d0 2024-08-03T20:56:16.845 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:16.842+0000 7f8de3400640 1 -- 172.21.15.32:0/1218080528 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8de4036040 con 0x7f8dec1698d0 2024-08-03T20:56:16.845 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:16.842+0000 7f8df0e00640 1 -- 172.21.15.32:0/1218080528 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8dec0cd590 con 0x7f8dec1698d0 2024-08-03T20:56:16.845 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:16.842+0000 7f8df0e00640 1 -- 172.21.15.32:0/1218080528 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8dec167110 con 0x7f8dec1698d0 2024-08-03T20:56:16.845 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:16.843+0000 7f8df0e00640 1 -- 172.21.15.32:0/1218080528 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8dec0d2290 con 0x7f8dec1698d0 2024-08-03T20:56:16.846 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:16.844+0000 7f8de3400640 1 -- 172.21.15.32:0/1218080528 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f8de4044020 con 0x7f8dec1698d0 2024-08-03T20:56:16.846 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:16.845+0000 7f8de3400640 1 --2- 172.21.15.32:0/1218080528 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f8dc8077040 0x7f8dc8079550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:56:16.846 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:16.845+0000 7f8de3400640 1 -- 172.21.15.32:0/1218080528 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 6800+0+0 (secure 0 0 0) 0x7f8de40b9c90 con 0x7f8dec1698d0 2024-08-03T20:56:16.847 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:16.845+0000 7f8deb400640 1 --2- 172.21.15.32:0/1218080528 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f8dc8077040 0x7f8dc8079550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:56:16.847 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:16.846+0000 7f8deb400640 1 --2- 172.21.15.32:0/1218080528 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f8dc8077040 0x7f8dc8079550 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7f8ddc001220 tx=0x7f8ddc009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:56:16.851 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:16.850+0000 7f8de3400640 1 -- 172.21.15.32:0/1218080528 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7f8de404b030 con 0x7f8dec1698d0 2024-08-03T20:56:17.098 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:17.096+0000 7f8df0e00640 1 -- 172.21.15.32:0/1218080528 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f8dec08c100 con 0x7f8dec1698d0 2024-08-03T20:56:17.098 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:17.097+0000 7f8de3400640 1 -- 172.21.15.32:0/1218080528 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 156 v156) ==== 95+0+4390 (secure 0 0 0) 0x7f8de40829d0 con 0x7f8dec1698d0 2024-08-03T20:56:17.098 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-03T20:56:17.099 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":156,"btime":"2024-08-03T20:56:13:041986+0000","default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6439,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6835/531080160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6834","nonce":531080160},{"type":"v1","addr":"172.21.15.118:6835","nonce":531080160}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":84},{"gid":6454,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6837/2796882319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6836","nonce":2796882319},{"type":"v1","addr":"172.21.15.118:6837","nonce":2796882319}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":86},{"gid":6813,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6834/2686933890","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":2686933890},{"type":"v1","addr":"172.21.15.32:6834","nonce":2686933890}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":85}],"filesystems":[{"mdsmap":{"epoch":156,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-08-03T20:54:26.950518+0000","modified":"2024-08-03T20:56:13.041980+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":96,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":6}]} 2024-08-03T20:56:17.099 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 156 2024-08-03T20:56:17.101 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:17.099+0000 7f8df0e00640 1 -- 172.21.15.32:0/1218080528 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f8dc8077040 msgr2=0x7f8dc8079550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:56:17.101 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:17.099+0000 7f8df0e00640 1 --2- 172.21.15.32:0/1218080528 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f8dc8077040 0x7f8dc8079550 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7f8ddc001220 tx=0x7f8ddc009040 comp rx=0 tx=0).stop 2024-08-03T20:56:17.101 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:17.099+0000 7f8df0e00640 1 -- 172.21.15.32:0/1218080528 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8dec1698d0 msgr2=0x7f8dec167e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:56:17.101 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:17.099+0000 7f8df0e00640 1 --2- 172.21.15.32:0/1218080528 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8dec1698d0 0x7f8dec167e50 secure :-1 s=READY pgs=1325 cs=0 l=1 rev1=1 crypto rx=0x7f8de4009be0 tx=0x7f8de4032c60 comp rx=0 tx=0).stop 2024-08-03T20:56:17.101 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:17.099+0000 7f8df0e00640 1 -- 172.21.15.32:0/1218080528 shutdown_connections 2024-08-03T20:56:17.101 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:17.099+0000 7f8df0e00640 1 --2- 172.21.15.32:0/1218080528 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f8dc8077040 0x7f8dc8079550 unknown :-1 s=CLOSED pgs=547 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:17.102 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:17.099+0000 7f8df0e00640 1 --2- 172.21.15.32:0/1218080528 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8dec1698d0 0x7f8dec167e50 unknown :-1 s=CLOSED pgs=1325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:17.102 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:17.099+0000 7f8df0e00640 1 --2- 172.21.15.32:0/1218080528 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f8dec0d4bb0 0x7f8dec167910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:17.102 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:17.099+0000 7f8df0e00640 1 --2- 172.21.15.32:0/1218080528 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8dec0d1c60 0x7f8dec15fa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:17.102 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:17.099+0000 7f8df0e00640 1 -- 172.21.15.32:0/1218080528 >> 172.21.15.32:0/1218080528 conn(0x7f8dec0d4090 msgr2=0x7f8dec0d07c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:56:17.102 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:17.100+0000 7f8df0e00640 1 -- 172.21.15.32:0/1218080528 shutdown_connections 2024-08-03T20:56:17.102 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:17.100+0000 7f8df0e00640 1 -- 172.21.15.32:0/1218080528 wait complete. 2024-08-03T20:56:17.123 INFO:tasks.cephfs.test_damage:Result: Mutation 'Delete 100.00000000' led to DAMAGED state 2024-08-03T20:56:17.124 INFO:tasks.cephfs.test_damage:Applying mutation 'Delete mds0_sessionmap' 2024-08-03T20:56:17.124 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-08-03T20:56:17.124 DEBUG:teuthology.orchestra.run.smithi032:> dd if=/proc/self/mounts of=/dev/stdout 2024-08-03T20:56:17.159 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053453,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6499208k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 none /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249604k,nr_inodes=812401,mode=700,uid=1000,gid=1316 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6499208k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-08-03T20:56:17.160 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi032.front.sepia.ceph.com 2024-08-03T20:56:17.160 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-08-03T20:56:17.160 DEBUG:teuthology.orchestra.run.smithi032:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-08-03T20:56:17.234 INFO:teuthology.orchestra.run.smithi032.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.0': No such file or directory 2024-08-03T20:56:17.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-03T20:56:17.235 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-08-03T20:56:17.506 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:17.504+0000 7ff5fec00640 1 -- 172.21.15.32:0/4138743118 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff5dc004800 msgr2=0x7ff5dc004c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:56:17.506 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:17.504+0000 7ff5fec00640 1 --2- 172.21.15.32:0/4138743118 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff5dc004800 0x7ff5dc004c10 secure :-1 s=READY pgs=1327 cs=0 l=1 rev1=1 crypto rx=0x7ff5e0006fb0 tx=0x7ff5dc005830 comp rx=0 tx=0).stop 2024-08-03T20:56:17.506 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:17.504+0000 7ff5fcc00640 1 -- 172.21.15.32:0/4138743118 reap_dead start 2024-08-03T20:56:17.506 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:17.504+0000 7ff5fec00640 1 -- 172.21.15.32:0/4138743118 shutdown_connections 2024-08-03T20:56:17.507 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:17.504+0000 7ff5fec00640 1 -- 172.21.15.32:0/4138743118 >> 172.21.15.32:0/4138743118 conn(0x7ff5f80b1bd0 msgr2=0x7ff5f80b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:56:17.507 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:17.504+0000 7ff5fec00640 1 -- 172.21.15.32:0/4138743118 shutdown_connections 2024-08-03T20:56:17.507 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:17.504+0000 7ff5fec00640 1 -- 172.21.15.32:0/4138743118 wait complete. 2024-08-03T20:56:17.508 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:17.506+0000 7ff5fec00640 1 Processor -- start 2024-08-03T20:56:17.508 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:17.507+0000 7ff5fec00640 1 -- start start 2024-08-03T20:56:17.511 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:17.508+0000 7ff5fec00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff5dc004800 0x7ff5f81f9600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:56:17.511 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:17.508+0000 7ff5fec00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ff5f8143e40 0x7ff5f815b840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:56:17.511 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:17.508+0000 7ff5fec00640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7ff5f81f6220 0x7ff5f8142e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:56:17.511 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:17.508+0000 7ff5fec00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7ff5f8145eb0 con 0x7ff5dc004800 2024-08-03T20:56:17.511 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:17.508+0000 7ff5fec00640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7ff5f80cfa50 con 0x7ff5f8143e40 2024-08-03T20:56:17.511 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:17.508+0000 7ff5fec00640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7ff5f8160b10 con 0x7ff5f81f6220 2024-08-03T20:56:17.511 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:17.508+0000 7ff5f7e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff5dc004800 0x7ff5f81f9600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:56:17.511 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:17.508+0000 7ff5f7e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff5dc004800 0x7ff5f81f9600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:46984/0 (socket says 172.21.15.32:46984) 2024-08-03T20:56:17.511 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:17.508+0000 7ff5f7e00640 1 -- 172.21.15.32:0/2296049294 learned_addr learned my addr 172.21.15.32:0/2296049294 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:56:17.512 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:17.509+0000 7ff5ef400640 1 --2- 172.21.15.32:0/2296049294 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ff5f8143e40 0x7ff5f815b840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:56:17.512 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:17.509+0000 7ff5fcc00640 1 --2- 172.21.15.32:0/2296049294 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7ff5f81f6220 0x7ff5f8142e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:56:17.512 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:17.509+0000 7ff5f7e00640 1 -- 172.21.15.32:0/2296049294 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7ff5f81f6220 msgr2=0x7ff5f8142e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:56:17.512 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:17.509+0000 7ff5f7e00640 1 --2- 172.21.15.32:0/2296049294 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7ff5f81f6220 0x7ff5f8142e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:17.512 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:17.509+0000 7ff5f7e00640 1 -- 172.21.15.32:0/2296049294 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ff5f8143e40 msgr2=0x7ff5f815b840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:56:17.512 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:17.509+0000 7ff5f7e00640 1 --2- 172.21.15.32:0/2296049294 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ff5f8143e40 0x7ff5f815b840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:17.512 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:17.509+0000 7ff5f7e00640 1 -- 172.21.15.32:0/2296049294 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff5f81e5f80 con 0x7ff5dc004800 2024-08-03T20:56:17.512 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:17.509+0000 7ff5ef400640 1 --2- 172.21.15.32:0/2296049294 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ff5f8143e40 0x7ff5f815b840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-03T20:56:17.512 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:17.509+0000 7ff5f7e00640 1 --2- 172.21.15.32:0/2296049294 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff5dc004800 0x7ff5f81f9600 secure :-1 s=READY pgs=1328 cs=0 l=1 rev1=1 crypto rx=0x7ff5e8010a40 tx=0x7ff5e8010f00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:56:17.512 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:17.509+0000 7ff5f4c00640 1 -- 172.21.15.32:0/2296049294 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5e800ed20 con 0x7ff5dc004800 2024-08-03T20:56:17.512 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:17.509+0000 7ff5f4c00640 1 -- 172.21.15.32:0/2296049294 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7ff5e8005020 con 0x7ff5dc004800 2024-08-03T20:56:17.512 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:17.509+0000 7ff5fec00640 1 -- 172.21.15.32:0/2296049294 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff5f81f5740 con 0x7ff5dc004800 2024-08-03T20:56:17.512 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:17.509+0000 7ff5f4c00640 1 -- 172.21.15.32:0/2296049294 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5e8005450 con 0x7ff5dc004800 2024-08-03T20:56:17.512 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:17.510+0000 7ff5fec00640 1 -- 172.21.15.32:0/2296049294 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff5f81f5d00 con 0x7ff5dc004800 2024-08-03T20:56:17.513 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:17.510+0000 7ff5fec00640 1 -- 172.21.15.32:0/2296049294 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff5f8144f90 con 0x7ff5dc004800 2024-08-03T20:56:17.513 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:17.512+0000 7ff5f4c00640 1 -- 172.21.15.32:0/2296049294 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7ff5e80047a0 con 0x7ff5dc004800 2024-08-03T20:56:17.514 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:17.513+0000 7ff5f4c00640 1 --2- 172.21.15.32:0/2296049294 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7ff5d4077040 0x7ff5d4079550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:56:17.514 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:17.513+0000 7ff5f4c00640 1 -- 172.21.15.32:0/2296049294 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 6800+0+0 (secure 0 0 0) 0x7ff5e8099a60 con 0x7ff5dc004800 2024-08-03T20:56:17.515 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:17.513+0000 7ff5ef400640 1 --2- 172.21.15.32:0/2296049294 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7ff5d4077040 0x7ff5d4079550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:56:17.516 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:17.514+0000 7ff5ef400640 1 --2- 172.21.15.32:0/2296049294 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7ff5d4077040 0x7ff5d4079550 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7ff5e0033000 tx=0x7ff5e002d000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:56:17.518 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:17.517+0000 7ff5f4c00640 1 -- 172.21.15.32:0/2296049294 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7ff5e80627b0 con 0x7ff5dc004800 2024-08-03T20:56:17.770 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:17.768+0000 7ff5fec00640 1 -- 172.21.15.32:0/2296049294 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7ff5f80b1fe0 con 0x7ff5dc004800 2024-08-03T20:56:17.791 INFO:tasks.ceph.mds.a.smithi032.stderr: -57> 2024-08-03T20:56:13.037+0000 7f15eac00640 -1 log_channel(cluster) log [ERR] : dir 0x100 object missing on disk; some files may be lost (~mds0) 2024-08-03T20:56:17.792 INFO:tasks.ceph.mds.a.smithi032.stderr: -30> 2024-08-03T20:56:14.287+0000 7f15efc00640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-08-03T20:56:17.907 INFO:tasks.ceph.mds.a.smithi032.stdout:starting mds.a at 2024-08-03T20:56:18.056 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:18.055+0000 7ff5f4c00640 1 -- 172.21.15.32:0/2296049294 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v157) ==== 161+0+0 (secure 0 0 0) 0x7ff5e80620b0 con 0x7ff5dc004800 2024-08-03T20:56:18.056 INFO:teuthology.orchestra.run.smithi032.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-08-03T20:56:18.059 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:18.057+0000 7ff5fec00640 1 -- 172.21.15.32:0/2296049294 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7ff5d4077040 msgr2=0x7ff5d4079550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:56:18.059 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:18.057+0000 7ff5fec00640 1 --2- 172.21.15.32:0/2296049294 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7ff5d4077040 0x7ff5d4079550 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7ff5e0033000 tx=0x7ff5e002d000 comp rx=0 tx=0).stop 2024-08-03T20:56:18.059 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:18.057+0000 7ff5fec00640 1 -- 172.21.15.32:0/2296049294 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff5dc004800 msgr2=0x7ff5f81f9600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:56:18.059 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:18.057+0000 7ff5fec00640 1 --2- 172.21.15.32:0/2296049294 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff5dc004800 0x7ff5f81f9600 secure :-1 s=READY pgs=1328 cs=0 l=1 rev1=1 crypto rx=0x7ff5e8010a40 tx=0x7ff5e8010f00 comp rx=0 tx=0).stop 2024-08-03T20:56:18.059 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:18.057+0000 7ff5fec00640 1 -- 172.21.15.32:0/2296049294 shutdown_connections 2024-08-03T20:56:18.059 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:18.057+0000 7ff5fec00640 1 --2- 172.21.15.32:0/2296049294 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7ff5d4077040 0x7ff5d4079550 unknown :-1 s=CLOSED pgs=548 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:18.059 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:18.057+0000 7ff5fec00640 1 --2- 172.21.15.32:0/2296049294 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ff5f8143e40 0x7ff5f815b840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:18.060 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:18.057+0000 7ff5fec00640 1 --2- 172.21.15.32:0/2296049294 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7ff5f81f6220 0x7ff5f8142e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:18.060 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:18.057+0000 7ff5fec00640 1 --2- 172.21.15.32:0/2296049294 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff5dc004800 0x7ff5f81f9600 unknown :-1 s=CLOSED pgs=1328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:18.060 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:18.057+0000 7ff5fec00640 1 -- 172.21.15.32:0/2296049294 >> 172.21.15.32:0/2296049294 conn(0x7ff5f80b1bd0 msgr2=0x7ff5f80afc40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:56:18.060 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:18.057+0000 7ff5fec00640 1 -- 172.21.15.32:0/2296049294 shutdown_connections 2024-08-03T20:56:18.060 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:18.057+0000 7ff5fec00640 1 -- 172.21.15.32:0/2296049294 wait complete. 2024-08-03T20:56:18.081 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mds repaired 0 2024-08-03T20:56:18.311 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:18.310+0000 7f9d64800640 1 -- 172.21.15.32:0/2259817393 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9d5c0b4ce0 msgr2=0x7f9d5c15ef00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:56:18.312 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:18.310+0000 7f9d64800640 1 --2- 172.21.15.32:0/2259817393 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9d5c0b4ce0 0x7f9d5c15ef00 secure :-1 s=READY pgs=1331 cs=0 l=1 rev1=1 crypto rx=0x7f9d5c0d1030 tx=0x7f9d4c02a560 comp rx=0 tx=0).stop 2024-08-03T20:56:18.312 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:18.310+0000 7f9d64800640 1 -- 172.21.15.32:0/2259817393 shutdown_connections 2024-08-03T20:56:18.312 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:18.310+0000 7f9d64800640 1 --2- 172.21.15.32:0/2259817393 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f9d5c15f440 0x7f9d5c15b910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:18.312 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:18.310+0000 7f9d64800640 1 --2- 172.21.15.32:0/2259817393 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9d5c0b4ce0 0x7f9d5c15ef00 unknown :-1 s=CLOSED pgs=1331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:18.312 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:18.310+0000 7f9d64800640 1 --2- 172.21.15.32:0/2259817393 >> v2:172.21.15.118:3300/0 conn(0x7f9d5c0b38b0 0x7f9d5c1523f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:18.312 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:18.310+0000 7f9d64800640 1 -- 172.21.15.32:0/2259817393 >> 172.21.15.32:0/2259817393 conn(0x7f9d5c0b1bd0 msgr2=0x7f9d5c0b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:56:18.312 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:18.310+0000 7f9d64800640 1 -- 172.21.15.32:0/2259817393 shutdown_connections 2024-08-03T20:56:18.312 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:18.310+0000 7f9d64800640 1 -- 172.21.15.32:0/2259817393 wait complete. 2024-08-03T20:56:18.313 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:18.312+0000 7f9d64800640 1 Processor -- start 2024-08-03T20:56:18.313 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:18.312+0000 7f9d64800640 1 -- start start 2024-08-03T20:56:18.316 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:18.314+0000 7f9d64800640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f9d5c0b38b0 0x7f9d5c146af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:56:18.316 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:18.314+0000 7f9d64800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9d5c0b4ce0 0x7f9d5c0731e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:56:18.316 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:18.314+0000 7f9d64800640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f9d5c15f440 0x7f9d5c1e8b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:56:18.316 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:18.314+0000 7f9d64800640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f9d5c15c1e0 con 0x7f9d5c0b4ce0 2024-08-03T20:56:18.316 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:18.314+0000 7f9d64800640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f9d5c15c060 con 0x7f9d5c0b38b0 2024-08-03T20:56:18.316 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:18.314+0000 7f9d64800640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7f9d5c15c360 con 0x7f9d5c15f440 2024-08-03T20:56:18.316 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:18.314+0000 7f9d61400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9d5c0b4ce0 0x7f9d5c0731e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:56:18.317 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:18.314+0000 7f9d61400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9d5c0b4ce0 0x7f9d5c0731e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:47024/0 (socket says 172.21.15.32:47024) 2024-08-03T20:56:18.317 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:18.314+0000 7f9d61400640 1 -- 172.21.15.32:0/3926661789 learned_addr learned my addr 172.21.15.32:0/3926661789 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:56:18.317 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:18.314+0000 7f9d61e00640 1 --2- 172.21.15.32:0/3926661789 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f9d5c0b38b0 0x7f9d5c146af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:56:18.317 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:18.314+0000 7f9d62800640 1 --2- 172.21.15.32:0/3926661789 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f9d5c15f440 0x7f9d5c1e8b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:56:18.317 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:18.314+0000 7f9d61400640 1 -- 172.21.15.32:0/3926661789 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f9d5c15f440 msgr2=0x7f9d5c1e8b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:56:18.317 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:18.314+0000 7f9d61400640 1 --2- 172.21.15.32:0/3926661789 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f9d5c15f440 0x7f9d5c1e8b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:18.317 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:18.314+0000 7f9d61400640 1 -- 172.21.15.32:0/3926661789 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f9d5c0b38b0 msgr2=0x7f9d5c146af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:56:18.317 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:18.314+0000 7f9d61400640 1 --2- 172.21.15.32:0/3926661789 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f9d5c0b38b0 0x7f9d5c146af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:18.317 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:18.315+0000 7f9d61400640 1 -- 172.21.15.32:0/3926661789 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9d5c0c4fe0 con 0x7f9d5c0b4ce0 2024-08-03T20:56:18.317 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:18.315+0000 7f9d62800640 1 --2- 172.21.15.32:0/3926661789 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f9d5c15f440 0x7f9d5c1e8b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-03T20:56:18.317 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:18.315+0000 7f9d61400640 1 --2- 172.21.15.32:0/3926661789 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9d5c0b4ce0 0x7f9d5c0731e0 secure :-1 s=READY pgs=1332 cs=0 l=1 rev1=1 crypto rx=0x7f9d4c034040 tx=0x7f9d4c0038a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:56:18.318 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:18.315+0000 7f9d56000640 1 -- 172.21.15.32:0/3926661789 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d4c03e030 con 0x7f9d5c0b4ce0 2024-08-03T20:56:18.318 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:18.315+0000 7f9d56000640 1 -- 172.21.15.32:0/3926661789 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f9d4c003ca0 con 0x7f9d5c0b4ce0 2024-08-03T20:56:18.318 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:18.315+0000 7f9d56000640 1 -- 172.21.15.32:0/3926661789 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d4c035040 con 0x7f9d5c0b4ce0 2024-08-03T20:56:18.318 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:18.315+0000 7f9d64800640 1 -- 172.21.15.32:0/3926661789 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9d5c0c52c0 con 0x7f9d5c0b4ce0 2024-08-03T20:56:18.318 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:18.315+0000 7f9d64800640 1 -- 172.21.15.32:0/3926661789 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9d5c158940 con 0x7f9d5c0b4ce0 2024-08-03T20:56:18.318 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:18.317+0000 7f9d64800640 1 -- 172.21.15.32:0/3926661789 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9d5c15ef00 con 0x7f9d5c0b4ce0 2024-08-03T20:56:18.319 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:18.318+0000 7f9d56000640 1 -- 172.21.15.32:0/3926661789 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f9d4c041030 con 0x7f9d5c0b4ce0 2024-08-03T20:56:18.320 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:18.319+0000 7f9d56000640 1 --2- 172.21.15.32:0/3926661789 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f9d40077010 0x7f9d40079520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:56:18.320 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:18.319+0000 7f9d56000640 1 -- 172.21.15.32:0/3926661789 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 6800+0+0 (secure 0 0 0) 0x7f9d4c0b92d0 con 0x7f9d5c0b4ce0 2024-08-03T20:56:18.320 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:18.319+0000 7f9d61e00640 1 --2- 172.21.15.32:0/3926661789 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f9d40077010 0x7f9d40079520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:56:18.321 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:18.320+0000 7f9d61e00640 1 --2- 172.21.15.32:0/3926661789 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f9d40077010 0x7f9d40079520 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7f9d50009a40 tx=0x7f9d50005f90 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:56:18.324 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:18.323+0000 7f9d56000640 1 -- 172.21.15.32:0/3926661789 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7f9d4c082130 con 0x7f9d5c0b4ce0 2024-08-03T20:56:18.612 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:18.610+0000 7f9d64800640 1 -- 172.21.15.32:0/3926661789 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "mds repaired", "role": "0"} v 0) -- 0x7f9d5c15e940 con 0x7f9d5c0b4ce0 2024-08-03T20:56:19.067 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.066+0000 7f9d56000640 1 -- 172.21.15.32:0/3926661789 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "mds repaired", "role": "0"}]=0 repaired: restoring rank 6:0 v158) ==== 101+0+0 (secure 0 0 0) 0x7f9d4c081a30 con 0x7f9d5c0b4ce0 2024-08-03T20:56:19.067 INFO:teuthology.orchestra.run.smithi032.stderr:repaired: restoring rank 6:0 2024-08-03T20:56:19.070 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.068+0000 7f9d64800640 1 -- 172.21.15.32:0/3926661789 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f9d40077010 msgr2=0x7f9d40079520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:56:19.070 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.068+0000 7f9d64800640 1 --2- 172.21.15.32:0/3926661789 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f9d40077010 0x7f9d40079520 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7f9d50009a40 tx=0x7f9d50005f90 comp rx=0 tx=0).stop 2024-08-03T20:56:19.070 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.068+0000 7f9d64800640 1 -- 172.21.15.32:0/3926661789 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9d5c0b4ce0 msgr2=0x7f9d5c0731e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:56:19.070 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.068+0000 7f9d64800640 1 --2- 172.21.15.32:0/3926661789 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9d5c0b4ce0 0x7f9d5c0731e0 secure :-1 s=READY pgs=1332 cs=0 l=1 rev1=1 crypto rx=0x7f9d4c034040 tx=0x7f9d4c0038a0 comp rx=0 tx=0).stop 2024-08-03T20:56:19.071 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.069+0000 7f9d64800640 1 -- 172.21.15.32:0/3926661789 shutdown_connections 2024-08-03T20:56:19.071 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.069+0000 7f9d64800640 1 --2- 172.21.15.32:0/3926661789 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f9d40077010 0x7f9d40079520 unknown :-1 s=CLOSED pgs=550 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:19.071 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.069+0000 7f9d64800640 1 --2- 172.21.15.32:0/3926661789 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f9d5c15f440 0x7f9d5c1e8b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:19.071 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.069+0000 7f9d64800640 1 --2- 172.21.15.32:0/3926661789 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9d5c0b4ce0 0x7f9d5c0731e0 unknown :-1 s=CLOSED pgs=1332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:19.071 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.069+0000 7f9d64800640 1 --2- 172.21.15.32:0/3926661789 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f9d5c0b38b0 0x7f9d5c146af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:19.071 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.069+0000 7f9d64800640 1 -- 172.21.15.32:0/3926661789 >> 172.21.15.32:0/3926661789 conn(0x7f9d5c0b1bd0 msgr2=0x7f9d5c15ce90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:56:19.071 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.069+0000 7f9d64800640 1 -- 172.21.15.32:0/3926661789 shutdown_connections 2024-08-03T20:56:19.071 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.069+0000 7f9d64800640 1 -- 172.21.15.32:0/3926661789 wait complete. 2024-08-03T20:56:19.095 DEBUG:teuthology.orchestra.run.smithi032:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata import - 2024-08-03T20:56:19.239 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.238+0000 7f897bb93f40 1 Processor -- start 2024-08-03T20:56:19.239 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.238+0000 7f897bb93f40 1 -- start start 2024-08-03T20:56:19.242 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.239+0000 7f897bb93f40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f895c003680 0x5567d9374120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:56:19.242 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.239+0000 7f897bb93f40 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5567d9422eb0 0x5567d9374660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:56:19.242 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.239+0000 7f897bb93f40 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x5567d938f660 0x5567d938fa70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:56:19.242 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.239+0000 7f897bb93f40 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x5567d92f7bf0 con 0x5567d9422eb0 2024-08-03T20:56:19.242 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.239+0000 7f897bb93f40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x5567d93046b0 con 0x7f895c003680 2024-08-03T20:56:19.242 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.239+0000 7f897bb93f40 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x5567d9299e30 con 0x5567d938f660 2024-08-03T20:56:19.242 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.239+0000 7f896fe00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5567d9422eb0 0x5567d9374660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:56:19.242 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.239+0000 7f896fe00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5567d9422eb0 0x5567d9374660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:47050/0 (socket says 172.21.15.32:47050) 2024-08-03T20:56:19.242 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.239+0000 7f896fe00640 1 -- 172.21.15.32:0/490318987 learned_addr learned my addr 172.21.15.32:0/490318987 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:56:19.243 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.240+0000 7f8974e00640 1 --2- 172.21.15.32:0/490318987 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f895c003680 0x5567d9374120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:56:19.243 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.240+0000 7f8975800640 1 --2- 172.21.15.32:0/490318987 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x5567d938f660 0x5567d938fa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:56:19.243 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.240+0000 7f896fe00640 1 -- 172.21.15.32:0/490318987 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x5567d938f660 msgr2=0x5567d938fa70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:56:19.243 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.240+0000 7f896fe00640 1 --2- 172.21.15.32:0/490318987 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x5567d938f660 0x5567d938fa70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:19.243 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.240+0000 7f896fe00640 1 -- 172.21.15.32:0/490318987 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f895c003680 msgr2=0x5567d9374120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:56:19.243 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.240+0000 7f896fe00640 1 --2- 172.21.15.32:0/490318987 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f895c003680 0x5567d9374120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:19.243 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.240+0000 7f896fe00640 1 -- 172.21.15.32:0/490318987 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5567d94223d0 con 0x5567d9422eb0 2024-08-03T20:56:19.243 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.240+0000 7f896fe00640 1 --2- 172.21.15.32:0/490318987 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5567d9422eb0 0x5567d9374660 secure :-1 s=READY pgs=1335 cs=0 l=1 rev1=1 crypto rx=0x7f896000d790 tx=0x7f896000dc50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8b85e8a75ac55194 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:56:19.243 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.240+0000 7f896ea00640 1 -- 172.21.15.32:0/490318987 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f896000dea0 con 0x5567d9422eb0 2024-08-03T20:56:19.243 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.240+0000 7f896ea00640 1 -- 172.21.15.32:0/490318987 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f8960014020 con 0x5567d9422eb0 2024-08-03T20:56:19.243 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.240+0000 7f896ea00640 1 -- 172.21.15.32:0/490318987 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f896000b8b0 con 0x5567d9422eb0 2024-08-03T20:56:19.244 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.241+0000 7f897bb93f40 1 -- 172.21.15.32:0/490318987 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5567d9422eb0 msgr2=0x5567d9374660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:56:19.244 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.241+0000 7f897bb93f40 1 --2- 172.21.15.32:0/490318987 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5567d9422eb0 0x5567d9374660 secure :-1 s=READY pgs=1335 cs=0 l=1 rev1=1 crypto rx=0x7f896000d790 tx=0x7f896000dc50 comp rx=0 tx=0).stop 2024-08-03T20:56:19.244 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.241+0000 7f897bb93f40 1 -- 172.21.15.32:0/490318987 shutdown_connections 2024-08-03T20:56:19.245 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.241+0000 7f897bb93f40 1 --2- 172.21.15.32:0/490318987 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x5567d938f660 0x5567d938fa70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:19.245 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.241+0000 7f897bb93f40 1 --2- 172.21.15.32:0/490318987 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5567d9422eb0 0x5567d9374660 unknown :-1 s=CLOSED pgs=1335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:19.245 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.241+0000 7f897bb93f40 1 --2- 172.21.15.32:0/490318987 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f895c003680 0x5567d9374120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:19.245 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.241+0000 7f897bb93f40 1 -- 172.21.15.32:0/490318987 >> 172.21.15.32:0/490318987 conn(0x5567d926bac0 msgr2=0x5567d9394880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:56:19.245 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.241+0000 7f897bb93f40 1 -- 172.21.15.32:0/490318987 shutdown_connections 2024-08-03T20:56:19.245 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.241+0000 7f897bb93f40 1 -- 172.21.15.32:0/490318987 wait complete. 2024-08-03T20:56:19.245 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.242+0000 7f897bb93f40 1 Processor -- start 2024-08-03T20:56:19.245 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.242+0000 7f897bb93f40 1 -- start start 2024-08-03T20:56:19.247 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.244+0000 7f897bb93f40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5567d938f660 0x5567d94af190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:56:19.247 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.244+0000 7f897bb93f40 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x5567d9422eb0 0x5567d94166b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:56:19.247 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.244+0000 7f897bb93f40 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f895c003680 0x5567d938d850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:56:19.247 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.244+0000 7f897bb93f40 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x5567d9374ba0 con 0x7f895c003680 2024-08-03T20:56:19.247 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.244+0000 7f897bb93f40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x5567d9392bd0 con 0x5567d938f660 2024-08-03T20:56:19.247 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.244+0000 7f897bb93f40 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x5567d9392eb0 con 0x5567d9422eb0 2024-08-03T20:56:19.247 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.244+0000 7f8975800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f895c003680 0x5567d938d850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:56:19.248 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.244+0000 7f8975800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f895c003680 0x5567d938d850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:47058/0 (socket says 172.21.15.32:47058) 2024-08-03T20:56:19.248 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.244+0000 7f8975800640 1 -- 172.21.15.32:0/1013615250 learned_addr learned my addr 172.21.15.32:0/1013615250 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:56:19.248 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.244+0000 7f8974e00640 1 --2- 172.21.15.32:0/1013615250 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5567d938f660 0x5567d94af190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:56:19.248 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.244+0000 7f896fe00640 1 --2- 172.21.15.32:0/1013615250 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x5567d9422eb0 0x5567d94166b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:56:19.248 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.245+0000 7f8975800640 1 -- 172.21.15.32:0/1013615250 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x5567d9422eb0 msgr2=0x5567d94166b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:56:19.248 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.245+0000 7f8975800640 1 --2- 172.21.15.32:0/1013615250 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x5567d9422eb0 0x5567d94166b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:19.248 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.245+0000 7f8975800640 1 -- 172.21.15.32:0/1013615250 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5567d938f660 msgr2=0x5567d94af190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:56:19.248 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.245+0000 7f8975800640 1 --2- 172.21.15.32:0/1013615250 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5567d938f660 0x5567d94af190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:19.248 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.245+0000 7f8975800640 1 -- 172.21.15.32:0/1013615250 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5567d9420170 con 0x7f895c003680 2024-08-03T20:56:19.248 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.245+0000 7f8975800640 1 --2- 172.21.15.32:0/1013615250 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f895c003680 0x5567d938d850 secure :-1 s=READY pgs=1336 cs=0 l=1 rev1=1 crypto rx=0x7f8970009d60 tx=0x7f89700098b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:56:19.248 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.245+0000 7f896d600640 1 -- 172.21.15.32:0/1013615250 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89700011c0 con 0x7f895c003680 2024-08-03T20:56:19.248 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.245+0000 7f896d600640 1 -- 172.21.15.32:0/1013615250 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f89700093c0 con 0x7f895c003680 2024-08-03T20:56:19.248 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.245+0000 7f897bb93f40 1 -- 172.21.15.32:0/1013615250 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x5567d941e490 con 0x7f895c003680 2024-08-03T20:56:19.249 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.245+0000 7f896d600640 1 -- 172.21.15.32:0/1013615250 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f897004dd20 con 0x7f895c003680 2024-08-03T20:56:19.249 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.245+0000 7f897bb93f40 1 -- 172.21.15.32:0/1013615250 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5567d9420410 con 0x7f895c003680 2024-08-03T20:56:19.249 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.247+0000 7f896d600640 1 -- 172.21.15.32:0/1013615250 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f8970069060 con 0x7f895c003680 2024-08-03T20:56:19.249 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.248+0000 7f896d600640 1 --2- 172.21.15.32:0/1013615250 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f89540759f0 0x7f8954077f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:56:19.250 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.248+0000 7f896d600640 1 -- 172.21.15.32:0/1013615250 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 6800+0+0 (secure 0 0 0) 0x7f89700db310 con 0x7f895c003680 2024-08-03T20:56:19.250 INFO:teuthology.orchestra.run.smithi032.stdout:Importing pool 2024-08-03T20:56:19.251 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.249+0000 7f897bb93f40 1 --2- 172.21.15.32:0/1013615250 >> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] conn(0x5567d9420930 0x5567d941c5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:56:19.251 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.249+0000 7f8974e00640 1 --2- 172.21.15.32:0/1013615250 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f89540759f0 0x7f8954077f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:56:19.251 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.249+0000 7f897bb93f40 1 -- 172.21.15.32:0/1013615250 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:1 12.10 12:09401a81:::601.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96) -- 0x5567d941cb10 con 0x5567d9420930 2024-08-03T20:56:19.252 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.249+0000 7f896fe00640 1 --2- 172.21.15.32:0/1013615250 >> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] conn(0x5567d9420930 0x5567d941c5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:56:19.252 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.249+0000 7f896fe00640 1 --2- 172.21.15.32:0/1013615250 >> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] conn(0x5567d9420930 0x5567d941c5d0 crc :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:56:19.252 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.250+0000 7f8974e00640 1 --2- 172.21.15.32:0/1013615250 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f89540759f0 0x7f8954077f00 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7f8968004fd0 tx=0x7f8968004880 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:56:19.253 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.251+0000 7f896fe00640 1 -- 172.21.15.32:0/1013615250 <== osd.0 v2:172.21.15.32:6816/3066156116 1 ==== osd_op_reply(1 601.00000000 [create] v96'319 uv293 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f896000b040 con 0x5567d9420930 2024-08-03T20:56:19.253 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.251+0000 7f897bb93f40 1 -- 172.21.15.32:0/1013615250 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:2 12.10 12:09401a81:::601.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e96) -- 0x5567d941cf10 con 0x5567d9420930 2024-08-03T20:56:19.254 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.253+0000 7f896fe00640 1 -- 172.21.15.32:0/1013615250 <== osd.0 v2:172.21.15.32:6816/3066156116 2 ==== osd_op_reply(2 601.00000000 [delete] v96'320 uv320 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f896000b040 con 0x5567d9420930 2024-08-03T20:56:19.255 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.253+0000 7f897bb93f40 1 -- 172.21.15.32:0/1013615250 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:3 12.10 12:09401a81:::601.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96) -- 0x5567d941d2e0 con 0x5567d9420930 2024-08-03T20:56:19.256 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::601.00000000:head# 2024-08-03T20:56:19.256 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.255+0000 7f896fe00640 1 -- 172.21.15.32:0/1013615250 <== osd.0 v2:172.21.15.32:6816/3066156116 3 ==== osd_op_reply(3 601.00000000 [create] v96'321 uv321 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f896000b040 con 0x5567d9420930 2024-08-03T20:56:19.256 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.255+0000 7f897bb93f40 1 -- 172.21.15.32:0/1013615250 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:4 12.10 12:09401a81:::601.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96) -- 0x5567d94c5de0 con 0x5567d9420930 2024-08-03T20:56:19.258 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.256+0000 7f896fe00640 1 -- 172.21.15.32:0/1013615250 <== osd.0 v2:172.21.15.32:6816/3066156116 4 ==== osd_op_reply(4 601.00000000 [setxattr (30)] v96'322 uv322 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f896000b040 con 0x5567d9420930 2024-08-03T20:56:19.258 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.256+0000 7f897bb93f40 1 -- 172.21.15.32:0/1013615250 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:5 12.10 12:09401a81:::601.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96) -- 0x5567d94c6180 con 0x5567d9420930 2024-08-03T20:56:19.259 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.258+0000 7f896fe00640 1 -- 172.21.15.32:0/1013615250 <== osd.0 v2:172.21.15.32:6816/3066156116 5 ==== osd_op_reply(5 601.00000000 [setxattr (62)] v96'323 uv323 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f896000b040 con 0x5567d9420930 2024-08-03T20:56:19.259 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.258+0000 7f897bb93f40 1 -- 172.21.15.32:0/1013615250 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:6 12.10 12:09401a81:::601.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96) -- 0x5567d94c4dd0 con 0x5567d9420930 2024-08-03T20:56:19.261 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.260+0000 7f896fe00640 1 -- 172.21.15.32:0/1013615250 <== osd.0 v2:172.21.15.32:6816/3066156116 6 ==== osd_op_reply(6 601.00000000 [omap-set-header] v96'324 uv324 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f896000b040 con 0x5567d9420930 2024-08-03T20:56:19.261 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.260+0000 7f897bb93f40 1 -- 172.21.15.32:0/1013615250 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:7 12.10 12:0bd6d154:::602.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96) -- 0x5567d94c5170 con 0x5567d9420930 2024-08-03T20:56:19.263 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.261+0000 7f896fe00640 1 -- 172.21.15.32:0/1013615250 <== osd.0 v2:172.21.15.32:6816/3066156116 7 ==== osd_op_reply(7 602.00000000 [create] v96'325 uv299 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f896000b040 con 0x5567d9420930 2024-08-03T20:56:19.263 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.261+0000 7f897bb93f40 1 -- 172.21.15.32:0/1013615250 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:8 12.10 12:0bd6d154:::602.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e96) -- 0x5567d94c5510 con 0x5567d9420930 2024-08-03T20:56:19.264 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.263+0000 7f896fe00640 1 -- 172.21.15.32:0/1013615250 <== osd.0 v2:172.21.15.32:6816/3066156116 8 ==== osd_op_reply(8 602.00000000 [delete] v96'326 uv326 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f896000b040 con 0x5567d9420930 2024-08-03T20:56:19.264 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.263+0000 7f897bb93f40 1 -- 172.21.15.32:0/1013615250 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:9 12.10 12:0bd6d154:::602.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96) -- 0x5567d94c5510 con 0x5567d9420930 2024-08-03T20:56:19.266 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::602.00000000:head# 2024-08-03T20:56:19.266 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.265+0000 7f896fe00640 1 -- 172.21.15.32:0/1013615250 <== osd.0 v2:172.21.15.32:6816/3066156116 9 ==== osd_op_reply(9 602.00000000 [create] v96'327 uv327 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f896000b040 con 0x5567d9420930 2024-08-03T20:56:19.267 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.265+0000 7f897bb93f40 1 -- 172.21.15.32:0/1013615250 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:10 12.10 12:0bd6d154:::602.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96) -- 0x5567d94c5510 con 0x5567d9420930 2024-08-03T20:56:19.268 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.266+0000 7f896fe00640 1 -- 172.21.15.32:0/1013615250 <== osd.0 v2:172.21.15.32:6816/3066156116 10 ==== osd_op_reply(10 602.00000000 [setxattr (30)] v96'328 uv328 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f896000b040 con 0x5567d9420930 2024-08-03T20:56:19.268 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.266+0000 7f897bb93f40 1 -- 172.21.15.32:0/1013615250 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:11 12.10 12:0bd6d154:::602.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96) -- 0x5567d94c5510 con 0x5567d9420930 2024-08-03T20:56:19.269 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.268+0000 7f896fe00640 1 -- 172.21.15.32:0/1013615250 <== osd.0 v2:172.21.15.32:6816/3066156116 11 ==== osd_op_reply(11 602.00000000 [setxattr (62)] v96'329 uv329 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f896000b040 con 0x5567d9420930 2024-08-03T20:56:19.269 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.268+0000 7f897bb93f40 1 -- 172.21.15.32:0/1013615250 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:12 12.10 12:0bd6d154:::602.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96) -- 0x5567d94c5510 con 0x5567d9420930 2024-08-03T20:56:19.271 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.269+0000 7f896fe00640 1 -- 172.21.15.32:0/1013615250 <== osd.0 v2:172.21.15.32:6816/3066156116 12 ==== osd_op_reply(12 602.00000000 [omap-set-header] v96'330 uv330 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f896000b040 con 0x5567d9420930 2024-08-03T20:56:19.271 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.270+0000 7f897bb93f40 1 -- 172.21.15.32:0/1013615250 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:13 12.10 12:0cf6ad0f:::10000000000.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96) -- 0x5567d94c5510 con 0x5567d9420930 2024-08-03T20:56:19.273 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.271+0000 7f896fe00640 1 -- 172.21.15.32:0/1013615250 <== osd.0 v2:172.21.15.32:6816/3066156116 13 ==== osd_op_reply(13 10000000000.00000000 [create] v96'331 uv306 ondisk = -17 ((17) File exists)) ==== 164+0+0 (crc 0 0 0) 0x7f896000b040 con 0x5567d9420930 2024-08-03T20:56:19.273 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.271+0000 7f897bb93f40 1 -- 172.21.15.32:0/1013615250 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:14 12.10 12:0cf6ad0f:::10000000000.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e96) -- 0x5567d94c5510 con 0x5567d9420930 2024-08-03T20:56:19.274 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.273+0000 7f896fe00640 1 -- 172.21.15.32:0/1013615250 <== osd.0 v2:172.21.15.32:6816/3066156116 14 ==== osd_op_reply(14 10000000000.00000000 [delete] v96'332 uv332 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f896000b040 con 0x5567d9420930 2024-08-03T20:56:19.274 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.273+0000 7f897bb93f40 1 -- 172.21.15.32:0/1013615250 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:15 12.10 12:0cf6ad0f:::10000000000.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96) -- 0x5567d94c5510 con 0x5567d9420930 2024-08-03T20:56:19.275 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::10000000000.00000000:head# 2024-08-03T20:56:19.276 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.274+0000 7f896fe00640 1 -- 172.21.15.32:0/1013615250 <== osd.0 v2:172.21.15.32:6816/3066156116 15 ==== osd_op_reply(15 10000000000.00000000 [create] v96'333 uv333 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f896000b040 con 0x5567d9420930 2024-08-03T20:56:19.276 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.274+0000 7f897bb93f40 1 -- 172.21.15.32:0/1013615250 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:16 12.10 12:0cf6ad0f:::10000000000.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96) -- 0x5567d94c5510 con 0x5567d9420930 2024-08-03T20:56:19.277 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.276+0000 7f896fe00640 1 -- 172.21.15.32:0/1013615250 <== osd.0 v2:172.21.15.32:6816/3066156116 16 ==== osd_op_reply(16 10000000000.00000000 [setxattr (30)] v96'334 uv334 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f896000b040 con 0x5567d9420930 2024-08-03T20:56:19.277 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.276+0000 7f897bb93f40 1 -- 172.21.15.32:0/1013615250 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:17 12.10 12:0cf6ad0f:::10000000000.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96) -- 0x5567d94c5510 con 0x5567d9420930 2024-08-03T20:56:19.279 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.277+0000 7f896fe00640 1 -- 172.21.15.32:0/1013615250 <== osd.0 v2:172.21.15.32:6816/3066156116 17 ==== osd_op_reply(17 10000000000.00000000 [setxattr (62)] v96'335 uv335 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f896000b040 con 0x5567d9420930 2024-08-03T20:56:19.279 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.277+0000 7f897bb93f40 1 -- 172.21.15.32:0/1013615250 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:18 12.10 12:0cf6ad0f:::10000000000.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96) -- 0x5567d94c5510 con 0x5567d9420930 2024-08-03T20:56:19.280 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.279+0000 7f896fe00640 1 -- 172.21.15.32:0/1013615250 <== osd.0 v2:172.21.15.32:6816/3066156116 18 ==== osd_op_reply(18 10000000000.00000000 [omap-set-header] v96'336 uv336 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f896000b040 con 0x5567d9420930 2024-08-03T20:56:19.281 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.279+0000 7f897bb93f40 1 -- 172.21.15.32:0/1013615250 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:19 12.10 12:0cf6ad0f:::10000000000.00000000:head [omap-set-vals in=593b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96) -- 0x5567d94c6520 con 0x5567d9420930 2024-08-03T20:56:19.282 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.281+0000 7f896fe00640 1 -- 172.21.15.32:0/1013615250 <== osd.0 v2:172.21.15.32:6816/3066156116 19 ==== osd_op_reply(19 10000000000.00000000 [omap-set-vals] v96'337 uv337 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f896000b040 con 0x5567d9420930 2024-08-03T20:56:19.283 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.281+0000 7f897bb93f40 1 -- 172.21.15.32:0/1013615250 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:20 12.10 12:0d82a743:::600.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96) -- 0x5567d94c6520 con 0x5567d9420930 2024-08-03T20:56:19.284 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.283+0000 7f896fe00640 1 -- 172.21.15.32:0/1013615250 <== osd.0 v2:172.21.15.32:6816/3066156116 20 ==== osd_op_reply(20 600.00000000 [create] v96'338 uv312 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f896000b040 con 0x5567d9420930 2024-08-03T20:56:19.284 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.283+0000 7f897bb93f40 1 -- 172.21.15.32:0/1013615250 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:21 12.10 12:0d82a743:::600.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e96) -- 0x5567d94c6520 con 0x5567d9420930 2024-08-03T20:56:19.286 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.284+0000 7f896fe00640 1 -- 172.21.15.32:0/1013615250 <== osd.0 v2:172.21.15.32:6816/3066156116 21 ==== osd_op_reply(21 600.00000000 [delete] v96'339 uv339 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f896000b040 con 0x5567d9420930 2024-08-03T20:56:19.286 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.284+0000 7f897bb93f40 1 -- 172.21.15.32:0/1013615250 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:22 12.10 12:0d82a743:::600.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96) -- 0x5567d94c6520 con 0x5567d9420930 2024-08-03T20:56:19.287 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::600.00000000:head# 2024-08-03T20:56:19.287 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.286+0000 7f896fe00640 1 -- 172.21.15.32:0/1013615250 <== osd.0 v2:172.21.15.32:6816/3066156116 22 ==== osd_op_reply(22 600.00000000 [create] v96'340 uv340 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f896000b040 con 0x5567d9420930 2024-08-03T20:56:19.288 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.286+0000 7f897bb93f40 1 -- 172.21.15.32:0/1013615250 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:23 12.10 12:0d82a743:::600.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96) -- 0x5567d94c7530 con 0x5567d9420930 2024-08-03T20:56:19.289 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.287+0000 7f896fe00640 1 -- 172.21.15.32:0/1013615250 <== osd.0 v2:172.21.15.32:6816/3066156116 23 ==== osd_op_reply(23 600.00000000 [setxattr (30)] v96'341 uv341 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f896000b040 con 0x5567d9420930 2024-08-03T20:56:19.289 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.287+0000 7f897bb93f40 1 -- 172.21.15.32:0/1013615250 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:24 12.10 12:0d82a743:::600.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96) -- 0x5567d94c7530 con 0x5567d9420930 2024-08-03T20:56:19.290 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.289+0000 7f896fe00640 1 -- 172.21.15.32:0/1013615250 <== osd.0 v2:172.21.15.32:6816/3066156116 24 ==== osd_op_reply(24 600.00000000 [setxattr (62)] v96'342 uv342 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f896000b040 con 0x5567d9420930 2024-08-03T20:56:19.290 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.289+0000 7f897bb93f40 1 -- 172.21.15.32:0/1013615250 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:25 12.10 12:0d82a743:::600.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96) -- 0x5567d94c6520 con 0x5567d9420930 2024-08-03T20:56:19.292 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.290+0000 7f896fe00640 1 -- 172.21.15.32:0/1013615250 <== osd.0 v2:172.21.15.32:6816/3066156116 25 ==== osd_op_reply(25 600.00000000 [omap-set-header] v96'343 uv343 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f896000b040 con 0x5567d9420930 2024-08-03T20:56:19.292 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.290+0000 7f897bb93f40 1 -- 172.21.15.32:0/1013615250 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:26 12.10 12:0d89b25e:::603.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96) -- 0x5567d94c6520 con 0x5567d9420930 2024-08-03T20:56:19.293 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.292+0000 7f896fe00640 1 -- 172.21.15.32:0/1013615250 <== osd.0 v2:172.21.15.32:6816/3066156116 26 ==== osd_op_reply(26 603.00000000 [create] v96'344 uv318 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f896000b040 con 0x5567d9420930 2024-08-03T20:56:19.293 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.292+0000 7f897bb93f40 1 -- 172.21.15.32:0/1013615250 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:27 12.10 12:0d89b25e:::603.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e96) -- 0x5567d94c6520 con 0x5567d9420930 2024-08-03T20:56:19.295 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.293+0000 7f896fe00640 1 -- 172.21.15.32:0/1013615250 <== osd.0 v2:172.21.15.32:6816/3066156116 27 ==== osd_op_reply(27 603.00000000 [delete] v96'345 uv345 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f896000b040 con 0x5567d9420930 2024-08-03T20:56:19.295 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.293+0000 7f897bb93f40 1 -- 172.21.15.32:0/1013615250 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:28 12.10 12:0d89b25e:::603.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96) -- 0x5567d94c6520 con 0x5567d9420930 2024-08-03T20:56:19.296 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::603.00000000:head# 2024-08-03T20:56:19.296 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.295+0000 7f896fe00640 1 -- 172.21.15.32:0/1013615250 <== osd.0 v2:172.21.15.32:6816/3066156116 28 ==== osd_op_reply(28 603.00000000 [create] v96'346 uv346 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f896000b040 con 0x5567d9420930 2024-08-03T20:56:19.297 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.295+0000 7f897bb93f40 1 -- 172.21.15.32:0/1013615250 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:29 12.10 12:0d89b25e:::603.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96) -- 0x5567d94c7530 con 0x5567d9420930 2024-08-03T20:56:19.298 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.296+0000 7f896fe00640 1 -- 172.21.15.32:0/1013615250 <== osd.0 v2:172.21.15.32:6816/3066156116 29 ==== osd_op_reply(29 603.00000000 [setxattr (30)] v96'347 uv347 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f896000b040 con 0x5567d9420930 2024-08-03T20:56:19.298 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.296+0000 7f897bb93f40 1 -- 172.21.15.32:0/1013615250 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:30 12.10 12:0d89b25e:::603.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96) -- 0x5567d94c7530 con 0x5567d9420930 2024-08-03T20:56:19.299 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.298+0000 7f896fe00640 1 -- 172.21.15.32:0/1013615250 <== osd.0 v2:172.21.15.32:6816/3066156116 30 ==== osd_op_reply(30 603.00000000 [setxattr (62)] v96'348 uv348 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f896000b040 con 0x5567d9420930 2024-08-03T20:56:19.300 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.298+0000 7f897bb93f40 1 -- 172.21.15.32:0/1013615250 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:31 12.10 12:0d89b25e:::603.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96) -- 0x5567d94c6520 con 0x5567d9420930 2024-08-03T20:56:19.301 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.300+0000 7f896fe00640 1 -- 172.21.15.32:0/1013615250 <== osd.0 v2:172.21.15.32:6816/3066156116 31 ==== osd_op_reply(31 603.00000000 [omap-set-header] v96'349 uv349 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f896000b040 con 0x5567d9420930 2024-08-03T20:56:19.301 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.300+0000 7f897bb93f40 1 --2- 172.21.15.32:0/1013615250 >> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] conn(0x5567d94c6a30 0x5567d94c8e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:56:19.301 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.300+0000 7f897bb93f40 1 -- 172.21.15.32:0/1013615250 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:32 12.14 12:287034c4:::1.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96) -- 0x5567d94c9390 con 0x5567d94c6a30 2024-08-03T20:56:19.302 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.300+0000 7f8974e00640 1 --2- 172.21.15.32:0/1013615250 >> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] conn(0x5567d94c6a30 0x5567d94c8e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:56:19.302 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.301+0000 7f8974e00640 1 --2- 172.21.15.32:0/1013615250 >> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] conn(0x5567d94c6a30 0x5567d94c8e50 crc :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:56:19.305 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.303+0000 7f8974e00640 1 -- 172.21.15.32:0/1013615250 <== osd.5 v2:172.21.15.118:6810/2724891770 1 ==== osd_op_reply(32 1.00000000.inode [create] v96'241 uv223 ondisk = -17 ((17) File exists)) ==== 160+0+0 (crc 0 0 0) 0x7f896800aa50 con 0x5567d94c6a30 2024-08-03T20:56:19.305 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.303+0000 7f897bb93f40 1 -- 172.21.15.32:0/1013615250 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:33 12.14 12:287034c4:::1.00000000.inode:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e96) -- 0x5567d94c9730 con 0x5567d94c6a30 2024-08-03T20:56:19.307 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.305+0000 7f8974e00640 1 -- 172.21.15.32:0/1013615250 <== osd.5 v2:172.21.15.118:6810/2724891770 2 ==== osd_op_reply(33 1.00000000.inode [delete] v96'242 uv242 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f896800aa50 con 0x5567d94c6a30 2024-08-03T20:56:19.307 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.305+0000 7f897bb93f40 1 -- 172.21.15.32:0/1013615250 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:34 12.14 12:287034c4:::1.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96) -- 0x5567d94c9ad0 con 0x5567d94c6a30 2024-08-03T20:56:19.308 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.307+0000 7f8974e00640 1 -- 172.21.15.32:0/1013615250 <== osd.5 v2:172.21.15.118:6810/2724891770 3 ==== osd_op_reply(34 1.00000000.inode [create] v96'243 uv243 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f896800aa50 con 0x5567d94c6a30 2024-08-03T20:56:19.309 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::1.00000000.inode:head# 2024-08-03T20:56:19.309 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.307+0000 7f897bb93f40 1 -- 172.21.15.32:0/1013615250 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:35 12.14 12:287034c4:::1.00000000.inode:head [write 0~582 in=582b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96) -- 0x5567d94c9e70 con 0x5567d94c6a30 2024-08-03T20:56:19.310 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.309+0000 7f8974e00640 1 -- 172.21.15.32:0/1013615250 <== osd.5 v2:172.21.15.118:6810/2724891770 4 ==== osd_op_reply(35 1.00000000.inode [write 0~582] v96'244 uv244 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f896800aa50 con 0x5567d94c6a30 2024-08-03T20:56:19.310 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.309+0000 7f897bb93f40 1 -- 172.21.15.32:0/1013615250 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:36 12.14 12:287034c4:::1.00000000.inode:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96) -- 0x5567d94ca210 con 0x5567d94c6a30 2024-08-03T20:56:19.312 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.310+0000 7f8974e00640 1 -- 172.21.15.32:0/1013615250 <== osd.5 v2:172.21.15.118:6810/2724891770 5 ==== osd_op_reply(36 1.00000000.inode [omap-set-header] v96'245 uv245 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f896800aa50 con 0x5567d94c6a30 2024-08-03T20:56:19.312 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.310+0000 7f897bb93f40 1 -- 172.21.15.32:0/1013615250 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:37 12.14 12:292cf221:::200.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96) -- 0x5567d94ca5b0 con 0x5567d94c6a30 2024-08-03T20:56:19.313 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.312+0000 7f8974e00640 1 -- 172.21.15.32:0/1013615250 <== osd.5 v2:172.21.15.118:6810/2724891770 6 ==== osd_op_reply(37 200.00000000 [create] v96'246 uv228 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f896800aa50 con 0x5567d94c6a30 2024-08-03T20:56:19.313 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.312+0000 7f897bb93f40 1 -- 172.21.15.32:0/1013615250 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:38 12.14 12:292cf221:::200.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e96) -- 0x5567d94ca950 con 0x5567d94c6a30 2024-08-03T20:56:19.315 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.313+0000 7f8974e00640 1 -- 172.21.15.32:0/1013615250 <== osd.5 v2:172.21.15.118:6810/2724891770 7 ==== osd_op_reply(38 200.00000000 [delete] v96'247 uv247 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f896800aa50 con 0x5567d94c6a30 2024-08-03T20:56:19.315 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.313+0000 7f897bb93f40 1 -- 172.21.15.32:0/1013615250 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:39 12.14 12:292cf221:::200.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96) -- 0x5567d94cacf0 con 0x5567d94c6a30 2024-08-03T20:56:19.316 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.315+0000 7f8974e00640 1 -- 172.21.15.32:0/1013615250 <== osd.5 v2:172.21.15.118:6810/2724891770 8 ==== osd_op_reply(39 200.00000000 [create] v96'248 uv248 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f896800aa50 con 0x5567d94c6a30 2024-08-03T20:56:19.316 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::200.00000000:head# 2024-08-03T20:56:19.317 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.315+0000 7f897bb93f40 1 -- 172.21.15.32:0/1013615250 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:40 12.14 12:292cf221:::200.00000000:head [write 0~90 in=90b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96) -- 0x5567d9416cd0 con 0x5567d94c6a30 2024-08-03T20:56:19.318 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.316+0000 7f8974e00640 1 -- 172.21.15.32:0/1013615250 <== osd.5 v2:172.21.15.118:6810/2724891770 9 ==== osd_op_reply(40 200.00000000 [write 0~90] v96'249 uv249 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f896800aa50 con 0x5567d94c6a30 2024-08-03T20:56:19.318 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.317+0000 7f897bb93f40 1 -- 172.21.15.32:0/1013615250 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:41 12.14 12:292cf221:::200.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96) -- 0x5567d9416cd0 con 0x5567d94c6a30 2024-08-03T20:56:19.319 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.318+0000 7f8974e00640 1 -- 172.21.15.32:0/1013615250 <== osd.5 v2:172.21.15.118:6810/2724891770 10 ==== osd_op_reply(41 200.00000000 [omap-set-header] v96'250 uv250 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f896800aa50 con 0x5567d94c6a30 2024-08-03T20:56:19.320 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.318+0000 7f897bb93f40 1 -- 172.21.15.32:0/1013615250 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:42 12.14 12:2e2fa760:::200.00000001:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96) -- 0x5567d9416cd0 con 0x5567d94c6a30 2024-08-03T20:56:19.321 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.319+0000 7f8974e00640 1 -- 172.21.15.32:0/1013615250 <== osd.5 v2:172.21.15.118:6810/2724891770 11 ==== osd_op_reply(42 200.00000001 [create] v96'251 uv233 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f896800aa50 con 0x5567d94c6a30 2024-08-03T20:56:19.321 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.319+0000 7f897bb93f40 1 -- 172.21.15.32:0/1013615250 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:43 12.14 12:2e2fa760:::200.00000001:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e96) -- 0x5567d9416cd0 con 0x5567d94c6a30 2024-08-03T20:56:19.322 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.321+0000 7f8974e00640 1 -- 172.21.15.32:0/1013615250 <== osd.5 v2:172.21.15.118:6810/2724891770 12 ==== osd_op_reply(43 200.00000001 [delete] v96'252 uv252 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f896800aa50 con 0x5567d94c6a30 2024-08-03T20:56:19.322 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.321+0000 7f897bb93f40 1 -- 172.21.15.32:0/1013615250 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:44 12.14 12:2e2fa760:::200.00000001:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96) -- 0x5567d9416cd0 con 0x5567d94c6a30 2024-08-03T20:56:19.324 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.322+0000 7f8974e00640 1 -- 172.21.15.32:0/1013615250 <== osd.5 v2:172.21.15.118:6810/2724891770 13 ==== osd_op_reply(44 200.00000001 [create] v96'253 uv253 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f896800aa50 con 0x5567d94c6a30 2024-08-03T20:56:19.324 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::200.00000001:head# 2024-08-03T20:56:19.324 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.322+0000 7f897bb93f40 1 -- 172.21.15.32:0/1013615250 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:45 12.14 12:2e2fa760:::200.00000001:head [write 0~18118 in=18118b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96) -- 0x5567d9416cd0 con 0x5567d94c6a30 2024-08-03T20:56:19.326 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.324+0000 7f8974e00640 1 -- 172.21.15.32:0/1013615250 <== osd.5 v2:172.21.15.118:6810/2724891770 14 ==== osd_op_reply(45 200.00000001 [write 0~18118] v96'254 uv254 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f896800aa50 con 0x5567d94c6a30 2024-08-03T20:56:19.326 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.324+0000 7f897bb93f40 1 -- 172.21.15.32:0/1013615250 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:46 12.14 12:2e2fa760:::200.00000001:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96) -- 0x5567d94cf3e0 con 0x5567d94c6a30 2024-08-03T20:56:19.327 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.326+0000 7f8974e00640 1 -- 172.21.15.32:0/1013615250 <== osd.5 v2:172.21.15.118:6810/2724891770 15 ==== osd_op_reply(46 200.00000001 [omap-set-header] v96'255 uv255 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f896800aa50 con 0x5567d94c6a30 2024-08-03T20:56:19.327 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.326+0000 7f897bb93f40 1 -- 172.21.15.32:0/1013615250 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:47 12.14 12:2f55791f:::606.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96) -- 0x5567d94cacf0 con 0x5567d94c6a30 2024-08-03T20:56:19.329 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.327+0000 7f8974e00640 1 -- 172.21.15.32:0/1013615250 <== osd.5 v2:172.21.15.118:6810/2724891770 16 ==== osd_op_reply(47 606.00000000 [create] v96'256 uv239 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f896800aa50 con 0x5567d94c6a30 2024-08-03T20:56:19.329 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.327+0000 7f897bb93f40 1 -- 172.21.15.32:0/1013615250 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:48 12.14 12:2f55791f:::606.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e96) -- 0x5567d94cacf0 con 0x5567d94c6a30 2024-08-03T20:56:19.330 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.329+0000 7f8974e00640 1 -- 172.21.15.32:0/1013615250 <== osd.5 v2:172.21.15.118:6810/2724891770 17 ==== osd_op_reply(48 606.00000000 [delete] v96'257 uv257 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f896800aa50 con 0x5567d94c6a30 2024-08-03T20:56:19.330 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.329+0000 7f897bb93f40 1 -- 172.21.15.32:0/1013615250 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:49 12.14 12:2f55791f:::606.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96) -- 0x5567d94cacf0 con 0x5567d94c6a30 2024-08-03T20:56:19.332 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.330+0000 7f8974e00640 1 -- 172.21.15.32:0/1013615250 <== osd.5 v2:172.21.15.118:6810/2724891770 18 ==== osd_op_reply(49 606.00000000 [create] v96'258 uv258 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f896800aa50 con 0x5567d94c6a30 2024-08-03T20:56:19.332 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::606.00000000:head# 2024-08-03T20:56:19.332 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.330+0000 7f897bb93f40 1 -- 172.21.15.32:0/1013615250 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:50 12.14 12:2f55791f:::606.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96) -- 0x5567d94cc0f0 con 0x5567d94c6a30 2024-08-03T20:56:19.333 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.332+0000 7f8974e00640 1 -- 172.21.15.32:0/1013615250 <== osd.5 v2:172.21.15.118:6810/2724891770 19 ==== osd_op_reply(50 606.00000000 [setxattr (30)] v96'259 uv259 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f896800aa50 con 0x5567d94c6a30 2024-08-03T20:56:19.333 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.332+0000 7f897bb93f40 1 -- 172.21.15.32:0/1013615250 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:51 12.14 12:2f55791f:::606.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96) -- 0x5567d94cc0f0 con 0x5567d94c6a30 2024-08-03T20:56:19.335 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.333+0000 7f8974e00640 1 -- 172.21.15.32:0/1013615250 <== osd.5 v2:172.21.15.118:6810/2724891770 20 ==== osd_op_reply(51 606.00000000 [setxattr (62)] v96'260 uv260 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f896800aa50 con 0x5567d94c6a30 2024-08-03T20:56:19.335 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.333+0000 7f897bb93f40 1 -- 172.21.15.32:0/1013615250 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:52 12.14 12:2f55791f:::606.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96) -- 0x5567d94cae10 con 0x5567d94c6a30 2024-08-03T20:56:19.337 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.335+0000 7f8974e00640 1 -- 172.21.15.32:0/1013615250 <== osd.5 v2:172.21.15.118:6810/2724891770 21 ==== osd_op_reply(52 606.00000000 [omap-set-header] v96'261 uv261 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f896800aa50 con 0x5567d94c6a30 2024-08-03T20:56:19.337 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.336+0000 7f897bb93f40 1 -- 172.21.15.32:0/1013615250 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:53 12.1c 12:3ed09add:::607.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96) -- 0x5567d94cad90 con 0x5567d9420930 2024-08-03T20:56:19.338 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.337+0000 7f896fe00640 1 -- 172.21.15.32:0/1013615250 <== osd.0 v2:172.21.15.32:6816/3066156116 32 ==== osd_op_reply(53 607.00000000 [create] v96'63 uv62 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f896000b040 con 0x5567d9420930 2024-08-03T20:56:19.339 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.337+0000 7f897bb93f40 1 -- 172.21.15.32:0/1013615250 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:54 12.1c 12:3ed09add:::607.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e96) -- 0x5567d94cad90 con 0x5567d9420930 2024-08-03T20:56:19.340 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.339+0000 7f896fe00640 1 -- 172.21.15.32:0/1013615250 <== osd.0 v2:172.21.15.32:6816/3066156116 33 ==== osd_op_reply(54 607.00000000 [delete] v96'64 uv64 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f896000b040 con 0x5567d9420930 2024-08-03T20:56:19.340 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.339+0000 7f897bb93f40 1 -- 172.21.15.32:0/1013615250 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:55 12.1c 12:3ed09add:::607.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96) -- 0x5567d94cad90 con 0x5567d9420930 2024-08-03T20:56:19.341 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::607.00000000:head# 2024-08-03T20:56:19.342 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.340+0000 7f896fe00640 1 -- 172.21.15.32:0/1013615250 <== osd.0 v2:172.21.15.32:6816/3066156116 34 ==== osd_op_reply(55 607.00000000 [create] v96'65 uv65 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f896000b040 con 0x5567d9420930 2024-08-03T20:56:19.342 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.340+0000 7f897bb93f40 1 -- 172.21.15.32:0/1013615250 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:56 12.1c 12:3ed09add:::607.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96) -- 0x5567d94cadd0 con 0x5567d9420930 2024-08-03T20:56:19.343 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.341+0000 7f896fe00640 1 -- 172.21.15.32:0/1013615250 <== osd.0 v2:172.21.15.32:6816/3066156116 35 ==== osd_op_reply(56 607.00000000 [setxattr (30)] v96'66 uv66 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f896000b040 con 0x5567d9420930 2024-08-03T20:56:19.343 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.341+0000 7f897bb93f40 1 -- 172.21.15.32:0/1013615250 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:57 12.1c 12:3ed09add:::607.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96) -- 0x5567d94cad90 con 0x5567d9420930 2024-08-03T20:56:19.344 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.343+0000 7f896fe00640 1 -- 172.21.15.32:0/1013615250 <== osd.0 v2:172.21.15.32:6816/3066156116 36 ==== osd_op_reply(57 607.00000000 [setxattr (62)] v96'67 uv67 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f896000b040 con 0x5567d9420930 2024-08-03T20:56:19.344 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.343+0000 7f897bb93f40 1 -- 172.21.15.32:0/1013615250 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:58 12.1c 12:3ed09add:::607.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96) -- 0x5567d94cad90 con 0x5567d9420930 2024-08-03T20:56:19.346 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.344+0000 7f896fe00640 1 -- 172.21.15.32:0/1013615250 <== osd.0 v2:172.21.15.32:6816/3066156116 37 ==== osd_op_reply(58 607.00000000 [omap-set-header] v96'68 uv68 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f896000b040 con 0x5567d9420930 2024-08-03T20:56:19.346 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.344+0000 7f897bb93f40 1 --2- 172.21.15.32:0/1013615250 >> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] conn(0x5567d94cb710 0x5567d94cbb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:56:19.346 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.344+0000 7f897bb93f40 1 -- 172.21.15.32:0/1013615250 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:59 12.6 12:654134d2:::mds0_openfiles.0:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96) -- 0x5567d94ce2f0 con 0x5567d94cb710 2024-08-03T20:56:19.346 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.345+0000 7f896fe00640 1 --2- 172.21.15.32:0/1013615250 >> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] conn(0x5567d94cb710 0x5567d94cbb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:56:19.346 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.345+0000 7f896fe00640 1 --2- 172.21.15.32:0/1013615250 >> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] conn(0x5567d94cb710 0x5567d94cbb20 crc :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:56:19.348 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.347+0000 7f896fe00640 1 -- 172.21.15.32:0/1013615250 <== osd.1 v2:172.21.15.32:6824/1551241759 1 ==== osd_op_reply(59 mds0_openfiles.0 [create] v96'55 uv54 ondisk = -17 ((17) File exists)) ==== 160+0+0 (crc 0 0 0) 0x7f896000b040 con 0x5567d94cb710 2024-08-03T20:56:19.348 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.347+0000 7f897bb93f40 1 -- 172.21.15.32:0/1013615250 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:60 12.6 12:654134d2:::mds0_openfiles.0:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e96) -- 0x5567d94ce2f0 con 0x5567d94cb710 2024-08-03T20:56:19.349 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.348+0000 7f896fe00640 1 -- 172.21.15.32:0/1013615250 <== osd.1 v2:172.21.15.32:6824/1551241759 2 ==== osd_op_reply(60 mds0_openfiles.0 [delete] v96'56 uv56 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f896000b040 con 0x5567d94cb710 2024-08-03T20:56:19.349 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.348+0000 7f897bb93f40 1 -- 172.21.15.32:0/1013615250 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:61 12.6 12:654134d2:::mds0_openfiles.0:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96) -- 0x5567d94ce2f0 con 0x5567d94cb710 2024-08-03T20:56:19.351 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.349+0000 7f896fe00640 1 -- 172.21.15.32:0/1013615250 <== osd.1 v2:172.21.15.32:6824/1551241759 3 ==== osd_op_reply(61 mds0_openfiles.0 [create] v96'57 uv57 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f896000b040 con 0x5567d94cb710 2024-08-03T20:56:19.351 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds0_openfiles.0:head# 2024-08-03T20:56:19.351 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.349+0000 7f897bb93f40 1 -- 172.21.15.32:0/1013615250 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:62 12.6 12:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96) -- 0x5567d94ce6e0 con 0x5567d94cb710 2024-08-03T20:56:19.352 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.350+0000 7f896fe00640 1 -- 172.21.15.32:0/1013615250 <== osd.1 v2:172.21.15.32:6824/1551241759 4 ==== osd_op_reply(62 mds0_openfiles.0 [omap-set-header] v96'58 uv58 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f896000b040 con 0x5567d94cb710 2024-08-03T20:56:19.352 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.350+0000 7f897bb93f40 1 -- 172.21.15.32:0/1013615250 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:63 12.e 12:717a0223:::608.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96) -- 0x5567d94ce6e0 con 0x5567d9420930 2024-08-03T20:56:19.353 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.352+0000 7f896fe00640 1 -- 172.21.15.32:0/1013615250 <== osd.0 v2:172.21.15.32:6816/3066156116 38 ==== osd_op_reply(63 608.00000000 [create] v96'63 uv62 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f896000b040 con 0x5567d9420930 2024-08-03T20:56:19.353 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.352+0000 7f897bb93f40 1 -- 172.21.15.32:0/1013615250 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:64 12.e 12:717a0223:::608.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e96) -- 0x5567d94ce6e0 con 0x5567d9420930 2024-08-03T20:56:19.355 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.354+0000 7f896fe00640 1 -- 172.21.15.32:0/1013615250 <== osd.0 v2:172.21.15.32:6816/3066156116 39 ==== osd_op_reply(64 608.00000000 [delete] v96'64 uv64 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f896000b040 con 0x5567d9420930 2024-08-03T20:56:19.355 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.354+0000 7f897bb93f40 1 -- 172.21.15.32:0/1013615250 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:65 12.e 12:717a0223:::608.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96) -- 0x5567d94ce6e0 con 0x5567d9420930 2024-08-03T20:56:19.356 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::608.00000000:head# 2024-08-03T20:56:19.357 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.355+0000 7f896fe00640 1 -- 172.21.15.32:0/1013615250 <== osd.0 v2:172.21.15.32:6816/3066156116 40 ==== osd_op_reply(65 608.00000000 [create] v96'65 uv65 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f896000b040 con 0x5567d9420930 2024-08-03T20:56:19.357 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.355+0000 7f897bb93f40 1 -- 172.21.15.32:0/1013615250 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:66 12.e 12:717a0223:::608.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96) -- 0x5567d94cf6f0 con 0x5567d9420930 2024-08-03T20:56:19.358 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.356+0000 7f896fe00640 1 -- 172.21.15.32:0/1013615250 <== osd.0 v2:172.21.15.32:6816/3066156116 41 ==== osd_op_reply(66 608.00000000 [setxattr (30)] v96'66 uv66 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f896000b040 con 0x5567d9420930 2024-08-03T20:56:19.358 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.356+0000 7f897bb93f40 1 -- 172.21.15.32:0/1013615250 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:67 12.e 12:717a0223:::608.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96) -- 0x5567d94cf6f0 con 0x5567d9420930 2024-08-03T20:56:19.359 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.358+0000 7f896fe00640 1 -- 172.21.15.32:0/1013615250 <== osd.0 v2:172.21.15.32:6816/3066156116 42 ==== osd_op_reply(67 608.00000000 [setxattr (62)] v96'67 uv67 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f896000b040 con 0x5567d9420930 2024-08-03T20:56:19.359 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.358+0000 7f897bb93f40 1 -- 172.21.15.32:0/1013615250 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:68 12.e 12:717a0223:::608.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96) -- 0x5567d94ce6e0 con 0x5567d9420930 2024-08-03T20:56:19.361 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.359+0000 7f896fe00640 1 -- 172.21.15.32:0/1013615250 <== osd.0 v2:172.21.15.32:6816/3066156116 43 ==== osd_op_reply(68 608.00000000 [omap-set-header] v96'68 uv68 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f896000b040 con 0x5567d9420930 2024-08-03T20:56:19.361 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.359+0000 7f897bb93f40 1 --2- 172.21.15.32:0/1013615250 >> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] conn(0x5567d94cebf0 0x5567d94d1010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:56:19.361 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.359+0000 7f897bb93f40 1 -- 172.21.15.32:0/1013615250 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:69 12.15 12:a93a17c2:::604.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96) -- 0x5567d94d1550 con 0x5567d94cebf0 2024-08-03T20:56:19.361 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.360+0000 7f8975800640 1 --2- 172.21.15.32:0/1013615250 >> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] conn(0x5567d94cebf0 0x5567d94d1010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:56:19.361 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.360+0000 7f8975800640 1 --2- 172.21.15.32:0/1013615250 >> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] conn(0x5567d94cebf0 0x5567d94d1010 crc :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:56:19.363 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.362+0000 7f8975800640 1 -- 172.21.15.32:0/1013615250 <== osd.6 v2:172.21.15.118:6825/801947386 1 ==== osd_op_reply(69 604.00000000 [create] v96'116 uv110 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f897009fd00 con 0x5567d94cebf0 2024-08-03T20:56:19.363 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.362+0000 7f897bb93f40 1 -- 172.21.15.32:0/1013615250 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:70 12.15 12:a93a17c2:::604.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e96) -- 0x5567d94d1a20 con 0x5567d94cebf0 2024-08-03T20:56:19.365 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.363+0000 7f8975800640 1 -- 172.21.15.32:0/1013615250 <== osd.6 v2:172.21.15.118:6825/801947386 2 ==== osd_op_reply(70 604.00000000 [delete] v96'117 uv117 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f897009fd00 con 0x5567d94cebf0 2024-08-03T20:56:19.365 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.363+0000 7f897bb93f40 1 -- 172.21.15.32:0/1013615250 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:71 12.15 12:a93a17c2:::604.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96) -- 0x5567d94d1f00 con 0x5567d94cebf0 2024-08-03T20:56:19.366 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.365+0000 7f8975800640 1 -- 172.21.15.32:0/1013615250 <== osd.6 v2:172.21.15.118:6825/801947386 3 ==== osd_op_reply(71 604.00000000 [create] v96'118 uv118 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f897009fd00 con 0x5567d94cebf0 2024-08-03T20:56:19.366 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::604.00000000:head# 2024-08-03T20:56:19.367 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.365+0000 7f897bb93f40 1 -- 172.21.15.32:0/1013615250 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:72 12.15 12:a93a17c2:::604.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96) -- 0x5567d94d37b0 con 0x5567d94cebf0 2024-08-03T20:56:19.368 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.366+0000 7f8975800640 1 -- 172.21.15.32:0/1013615250 <== osd.6 v2:172.21.15.118:6825/801947386 4 ==== osd_op_reply(72 604.00000000 [setxattr (30)] v96'119 uv119 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f897009fd00 con 0x5567d94cebf0 2024-08-03T20:56:19.368 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.367+0000 7f897bb93f40 1 -- 172.21.15.32:0/1013615250 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:73 12.15 12:a93a17c2:::604.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96) -- 0x5567d94d3b50 con 0x5567d94cebf0 2024-08-03T20:56:19.369 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.368+0000 7f8975800640 1 -- 172.21.15.32:0/1013615250 <== osd.6 v2:172.21.15.118:6825/801947386 5 ==== osd_op_reply(73 604.00000000 [setxattr (62)] v96'120 uv120 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f897009fd00 con 0x5567d94cebf0 2024-08-03T20:56:19.369 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.368+0000 7f897bb93f40 1 -- 172.21.15.32:0/1013615250 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:74 12.15 12:a93a17c2:::604.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96) -- 0x5567d94d27a0 con 0x5567d94cebf0 2024-08-03T20:56:19.371 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.369+0000 7f8975800640 1 -- 172.21.15.32:0/1013615250 <== osd.6 v2:172.21.15.118:6825/801947386 6 ==== osd_op_reply(74 604.00000000 [omap-set-header] v96'121 uv121 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f897009fd00 con 0x5567d94cebf0 2024-08-03T20:56:19.371 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.369+0000 7f897bb93f40 1 -- 172.21.15.32:0/1013615250 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:75 12.15 12:aa448500:::500.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96) -- 0x5567d94d27a0 con 0x5567d94cebf0 2024-08-03T20:56:19.372 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.371+0000 7f8975800640 1 -- 172.21.15.32:0/1013615250 <== osd.6 v2:172.21.15.118:6825/801947386 7 ==== osd_op_reply(75 500.00000000 [create] v96'122 uv115 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f897009fd00 con 0x5567d94cebf0 2024-08-03T20:56:19.372 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.371+0000 7f897bb93f40 1 -- 172.21.15.32:0/1013615250 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:76 12.15 12:aa448500:::500.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e96) -- 0x5567d94d27a0 con 0x5567d94cebf0 2024-08-03T20:56:19.374 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.372+0000 7f8975800640 1 -- 172.21.15.32:0/1013615250 <== osd.6 v2:172.21.15.118:6825/801947386 8 ==== osd_op_reply(76 500.00000000 [delete] v96'123 uv123 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f897009fd00 con 0x5567d94cebf0 2024-08-03T20:56:19.374 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.372+0000 7f897bb93f40 1 -- 172.21.15.32:0/1013615250 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:77 12.15 12:aa448500:::500.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96) -- 0x5567d94d27a0 con 0x5567d94cebf0 2024-08-03T20:56:19.375 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.373+0000 7f8975800640 1 -- 172.21.15.32:0/1013615250 <== osd.6 v2:172.21.15.118:6825/801947386 9 ==== osd_op_reply(77 500.00000000 [create] v96'124 uv124 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f897009fd00 con 0x5567d94cebf0 2024-08-03T20:56:19.375 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::500.00000000:head# 2024-08-03T20:56:19.375 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.373+0000 7f897bb93f40 1 -- 172.21.15.32:0/1013615250 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:78 12.15 12:aa448500:::500.00000000:head [write 0~90 in=90b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96) -- 0x5567d94d27a0 con 0x5567d94cebf0 2024-08-03T20:56:19.376 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.375+0000 7f8975800640 1 -- 172.21.15.32:0/1013615250 <== osd.6 v2:172.21.15.118:6825/801947386 10 ==== osd_op_reply(78 500.00000000 [write 0~90] v96'125 uv125 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f897009fd00 con 0x5567d94cebf0 2024-08-03T20:56:19.376 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.375+0000 7f897bb93f40 1 -- 172.21.15.32:0/1013615250 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:79 12.15 12:aa448500:::500.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96) -- 0x5567d94d27a0 con 0x5567d94cebf0 2024-08-03T20:56:19.378 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.376+0000 7f8975800640 1 -- 172.21.15.32:0/1013615250 <== osd.6 v2:172.21.15.118:6825/801947386 11 ==== osd_op_reply(79 500.00000000 [omap-set-header] v96'126 uv126 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f897009fd00 con 0x5567d94cebf0 2024-08-03T20:56:19.378 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.376+0000 7f897bb93f40 1 --2- 172.21.15.32:0/1013615250 >> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] conn(0x5567d94d30a0 0x5567d94d5f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:56:19.378 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.376+0000 7f897bb93f40 1 -- 172.21.15.32:0/1013615250 --> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] -- osd_op(unknown.0.0:80 12.d 12:b50e409b:::mds_snaptable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96) -- 0x5567d94d27a0 con 0x5567d94d30a0 2024-08-03T20:56:19.379 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.377+0000 7f8974e00640 1 --2- 172.21.15.32:0/1013615250 >> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] conn(0x5567d94d30a0 0x5567d94d5f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:56:19.379 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.378+0000 7f8974e00640 1 --2- 172.21.15.32:0/1013615250 >> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] conn(0x5567d94d30a0 0x5567d94d5f00 crc :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:56:19.380 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.379+0000 7f8974e00640 1 -- 172.21.15.32:0/1013615250 <== osd.7 v2:172.21.15.118:6818/653000048 1 ==== osd_op_reply(80 mds_snaptable [create] v96'51 uv50 ondisk = -17 ((17) File exists)) ==== 157+0+0 (crc 0 0 0) 0x7f896800aa50 con 0x5567d94d30a0 2024-08-03T20:56:19.381 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.379+0000 7f897bb93f40 1 -- 172.21.15.32:0/1013615250 --> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] -- osd_op(unknown.0.0:81 12.d 12:b50e409b:::mds_snaptable:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e96) -- 0x5567d94d27a0 con 0x5567d94d30a0 2024-08-03T20:56:19.382 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.380+0000 7f8974e00640 1 -- 172.21.15.32:0/1013615250 <== osd.7 v2:172.21.15.118:6818/653000048 2 ==== osd_op_reply(81 mds_snaptable [delete] v96'52 uv52 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f896800aa50 con 0x5567d94d30a0 2024-08-03T20:56:19.382 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.380+0000 7f897bb93f40 1 -- 172.21.15.32:0/1013615250 --> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] -- osd_op(unknown.0.0:82 12.d 12:b50e409b:::mds_snaptable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96) -- 0x5567d94d64c0 con 0x5567d94d30a0 2024-08-03T20:56:19.383 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.382+0000 7f8974e00640 1 -- 172.21.15.32:0/1013615250 <== osd.7 v2:172.21.15.118:6818/653000048 3 ==== osd_op_reply(82 mds_snaptable [create] v96'53 uv53 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f896800aa50 con 0x5567d94d30a0 2024-08-03T20:56:19.383 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds_snaptable:head# 2024-08-03T20:56:19.384 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.382+0000 7f897bb93f40 1 -- 172.21.15.32:0/1013615250 --> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] -- osd_op(unknown.0.0:83 12.d 12:b50e409b:::mds_snaptable:head [write 0~70 in=70b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96) -- 0x5567d94d6490 con 0x5567d94d30a0 2024-08-03T20:56:19.385 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.383+0000 7f8974e00640 1 -- 172.21.15.32:0/1013615250 <== osd.7 v2:172.21.15.118:6818/653000048 4 ==== osd_op_reply(83 mds_snaptable [write 0~70] v96'54 uv54 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f896800aa50 con 0x5567d94d30a0 2024-08-03T20:56:19.385 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.383+0000 7f897bb93f40 1 -- 172.21.15.32:0/1013615250 --> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] -- osd_op(unknown.0.0:84 12.d 12:b50e409b:::mds_snaptable:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96) -- 0x5567d94d6490 con 0x5567d94d30a0 2024-08-03T20:56:19.386 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.385+0000 7f8974e00640 1 -- 172.21.15.32:0/1013615250 <== osd.7 v2:172.21.15.118:6818/653000048 5 ==== osd_op_reply(84 mds_snaptable [omap-set-header] v96'55 uv55 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f896800aa50 con 0x5567d94d30a0 2024-08-03T20:56:19.386 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.385+0000 7f897bb93f40 1 -- 172.21.15.32:0/1013615250 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:85 12.1d 12:b871830b:::605.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96) -- 0x5567d94d6490 con 0x5567d9420930 2024-08-03T20:56:19.388 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.386+0000 7f896fe00640 1 -- 172.21.15.32:0/1013615250 <== osd.0 v2:172.21.15.32:6816/3066156116 44 ==== osd_op_reply(85 605.00000000 [create] v96'63 uv62 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f896000b040 con 0x5567d9420930 2024-08-03T20:56:19.388 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.386+0000 7f897bb93f40 1 -- 172.21.15.32:0/1013615250 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:86 12.1d 12:b871830b:::605.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e96) -- 0x5567d94d6490 con 0x5567d9420930 2024-08-03T20:56:19.389 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.388+0000 7f896fe00640 1 -- 172.21.15.32:0/1013615250 <== osd.0 v2:172.21.15.32:6816/3066156116 45 ==== osd_op_reply(86 605.00000000 [delete] v96'64 uv64 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f896000b040 con 0x5567d9420930 2024-08-03T20:56:19.389 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.388+0000 7f897bb93f40 1 -- 172.21.15.32:0/1013615250 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:87 12.1d 12:b871830b:::605.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96) -- 0x5567d94d6490 con 0x5567d9420930 2024-08-03T20:56:19.390 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::605.00000000:head# 2024-08-03T20:56:19.391 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.389+0000 7f896fe00640 1 -- 172.21.15.32:0/1013615250 <== osd.0 v2:172.21.15.32:6816/3066156116 46 ==== osd_op_reply(87 605.00000000 [create] v96'65 uv65 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f896000b040 con 0x5567d9420930 2024-08-03T20:56:19.391 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.389+0000 7f897bb93f40 1 -- 172.21.15.32:0/1013615250 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:88 12.1d 12:b871830b:::605.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96) -- 0x5567d94d74a0 con 0x5567d9420930 2024-08-03T20:56:19.392 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.390+0000 7f896fe00640 1 -- 172.21.15.32:0/1013615250 <== osd.0 v2:172.21.15.32:6816/3066156116 47 ==== osd_op_reply(88 605.00000000 [setxattr (30)] v96'66 uv66 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f896000b040 con 0x5567d9420930 2024-08-03T20:56:19.392 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.390+0000 7f897bb93f40 1 -- 172.21.15.32:0/1013615250 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:89 12.1d 12:b871830b:::605.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96) -- 0x5567d94d74a0 con 0x5567d9420930 2024-08-03T20:56:19.393 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.392+0000 7f896fe00640 1 -- 172.21.15.32:0/1013615250 <== osd.0 v2:172.21.15.32:6816/3066156116 48 ==== osd_op_reply(89 605.00000000 [setxattr (62)] v96'67 uv67 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f896000b040 con 0x5567d9420930 2024-08-03T20:56:19.393 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.392+0000 7f897bb93f40 1 -- 172.21.15.32:0/1013615250 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:90 12.1d 12:b871830b:::605.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96) -- 0x5567d94d6490 con 0x5567d9420930 2024-08-03T20:56:19.394 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.393+0000 7f896fe00640 1 -- 172.21.15.32:0/1013615250 <== osd.0 v2:172.21.15.32:6816/3066156116 49 ==== osd_op_reply(90 605.00000000 [omap-set-header] v96'68 uv68 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f896000b040 con 0x5567d9420930 2024-08-03T20:56:19.394 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.393+0000 7f897bb93f40 1 -- 172.21.15.32:0/1013615250 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:91 12.13 12:c91d4a1d:::mds0_inotable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96) -- 0x5567d94d6490 con 0x5567d94cb710 2024-08-03T20:56:19.396 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.394+0000 7f896fe00640 1 -- 172.21.15.32:0/1013615250 <== osd.1 v2:172.21.15.32:6824/1551241759 5 ==== osd_op_reply(91 mds0_inotable [create] v96'104 uv97 ondisk = -17 ((17) File exists)) ==== 157+0+0 (crc 0 0 0) 0x7f896000b040 con 0x5567d94cb710 2024-08-03T20:56:19.396 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.394+0000 7f897bb93f40 1 -- 172.21.15.32:0/1013615250 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:92 12.13 12:c91d4a1d:::mds0_inotable:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e96) -- 0x5567d94d6490 con 0x5567d94cb710 2024-08-03T20:56:19.397 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.396+0000 7f896fe00640 1 -- 172.21.15.32:0/1013615250 <== osd.1 v2:172.21.15.32:6824/1551241759 6 ==== osd_op_reply(92 mds0_inotable [delete] v96'105 uv105 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f896000b040 con 0x5567d94cb710 2024-08-03T20:56:19.397 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.396+0000 7f897bb93f40 1 -- 172.21.15.32:0/1013615250 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:93 12.13 12:c91d4a1d:::mds0_inotable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96) -- 0x5567d94d6490 con 0x5567d94cb710 2024-08-03T20:56:19.398 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds0_inotable:head# 2024-08-03T20:56:19.399 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.397+0000 7f896fe00640 1 -- 172.21.15.32:0/1013615250 <== osd.1 v2:172.21.15.32:6824/1551241759 7 ==== osd_op_reply(93 mds0_inotable [create] v96'106 uv106 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f896000b040 con 0x5567d94cb710 2024-08-03T20:56:19.399 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.397+0000 7f897bb93f40 1 -- 172.21.15.32:0/1013615250 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:94 12.13 12:c91d4a1d:::mds0_inotable:head [write 0~34 in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96) -- 0x5567d94d6490 con 0x5567d94cb710 2024-08-03T20:56:19.400 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.398+0000 7f896fe00640 1 -- 172.21.15.32:0/1013615250 <== osd.1 v2:172.21.15.32:6824/1551241759 8 ==== osd_op_reply(94 mds0_inotable [write 0~34] v96'107 uv107 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f896000b040 con 0x5567d94cb710 2024-08-03T20:56:19.400 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.398+0000 7f897bb93f40 1 -- 172.21.15.32:0/1013615250 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:95 12.13 12:c91d4a1d:::mds0_inotable:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96) -- 0x5567d94d6490 con 0x5567d94cb710 2024-08-03T20:56:19.401 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.400+0000 7f896fe00640 1 -- 172.21.15.32:0/1013615250 <== osd.1 v2:172.21.15.32:6824/1551241759 9 ==== osd_op_reply(95 mds0_inotable [omap-set-header] v96'108 uv108 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f896000b040 con 0x5567d94cb710 2024-08-03T20:56:19.401 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.400+0000 7f897bb93f40 1 -- 172.21.15.32:0/1013615250 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:96 12.13 12:cd5a64a3:::100.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96) -- 0x5567d94d6490 con 0x5567d94cb710 2024-08-03T20:56:19.402 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.401+0000 7f896fe00640 1 -- 172.21.15.32:0/1013615250 <== osd.1 v2:172.21.15.32:6824/1551241759 10 ==== osd_op_reply(96 100.00000000 [create] v96'109 uv109 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f896000b040 con 0x5567d94cb710 2024-08-03T20:56:19.403 INFO:teuthology.orchestra.run.smithi032.stdout:Write #-9223372036854775808:00000000:::100.00000000:head# 2024-08-03T20:56:19.403 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.401+0000 7f897bb93f40 1 -- 172.21.15.32:0/1013615250 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:97 12.13 12:cd5a64a3:::100.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96) -- 0x5567d94d6490 con 0x5567d94cb710 2024-08-03T20:56:19.404 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.402+0000 7f896fe00640 1 -- 172.21.15.32:0/1013615250 <== osd.1 v2:172.21.15.32:6824/1551241759 11 ==== osd_op_reply(97 100.00000000 [omap-set-header] v96'110 uv110 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f896000b040 con 0x5567d94cb710 2024-08-03T20:56:19.404 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.403+0000 7f897bb93f40 1 -- 172.21.15.32:0/1013615250 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:98 12.13 12:cd5a64a3:::100.00000000:head [omap-set-vals in=5194b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96) -- 0x5567d94d92a0 con 0x5567d94cb710 2024-08-03T20:56:19.405 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.404+0000 7f896fe00640 1 -- 172.21.15.32:0/1013615250 <== osd.1 v2:172.21.15.32:6824/1551241759 12 ==== osd_op_reply(98 100.00000000 [omap-set-vals] v96'111 uv111 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f896000b040 con 0x5567d94cb710 2024-08-03T20:56:19.405 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.404+0000 7f897bb93f40 1 --2- 172.21.15.32:0/1013615250 >> [v2:172.21.15.32:6800/9088303,v1:172.21.15.32:6801/9088303] conn(0x5567d94d6fc0 0x5567d94d98e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:56:19.406 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.404+0000 7f897bb93f40 1 -- 172.21.15.32:0/1013615250 --> [v2:172.21.15.32:6800/9088303,v1:172.21.15.32:6801/9088303] -- osd_op(unknown.0.0:99 12.b 12:d0630e4c:::mds0_sessionmap:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96) -- 0x5567d94d93e0 con 0x5567d94d6fc0 2024-08-03T20:56:19.406 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.404+0000 7f896fe00640 1 --2- 172.21.15.32:0/1013615250 >> [v2:172.21.15.32:6800/9088303,v1:172.21.15.32:6801/9088303] conn(0x5567d94d6fc0 0x5567d94d98e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:56:19.406 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.405+0000 7f896fe00640 1 --2- 172.21.15.32:0/1013615250 >> [v2:172.21.15.32:6800/9088303,v1:172.21.15.32:6801/9088303] conn(0x5567d94d6fc0 0x5567d94d98e0 crc :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:56:19.408 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.406+0000 7f896fe00640 1 -- 172.21.15.32:0/1013615250 <== osd.3 v2:172.21.15.32:6800/9088303 1 ==== osd_op_reply(99 mds0_sessionmap [create] v96'49 uv47 ondisk = -17 ((17) File exists)) ==== 159+0+0 (crc 0 0 0) 0x7f896000b040 con 0x5567d94d6fc0 2024-08-03T20:56:19.408 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.406+0000 7f897bb93f40 1 -- 172.21.15.32:0/1013615250 --> [v2:172.21.15.32:6800/9088303,v1:172.21.15.32:6801/9088303] -- osd_op(unknown.0.0:100 12.b 12:d0630e4c:::mds0_sessionmap:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e96) -- 0x5567d94d93e0 con 0x5567d94d6fc0 2024-08-03T20:56:19.409 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.408+0000 7f896fe00640 1 -- 172.21.15.32:0/1013615250 <== osd.3 v2:172.21.15.32:6800/9088303 2 ==== osd_op_reply(100 mds0_sessionmap [delete] v96'50 uv50 ondisk = 0) ==== 159+0+0 (crc 0 0 0) 0x7f896000b040 con 0x5567d94d6fc0 2024-08-03T20:56:19.409 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.408+0000 7f897bb93f40 1 -- 172.21.15.32:0/1013615250 --> [v2:172.21.15.32:6800/9088303,v1:172.21.15.32:6801/9088303] -- osd_op(unknown.0.0:101 12.b 12:d0630e4c:::mds0_sessionmap:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96) -- 0x5567d94d93e0 con 0x5567d94d6fc0 2024-08-03T20:56:19.411 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.409+0000 7f896fe00640 1 -- 172.21.15.32:0/1013615250 <== osd.3 v2:172.21.15.32:6800/9088303 3 ==== osd_op_reply(101 mds0_sessionmap [create] v96'51 uv51 ondisk = 0) ==== 159+0+0 (crc 0 0 0) 0x7f896000b040 con 0x5567d94d6fc0 2024-08-03T20:56:19.411 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds0_sessionmap:head# 2024-08-03T20:56:19.411 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.409+0000 7f897bb93f40 1 -- 172.21.15.32:0/1013615250 --> [v2:172.21.15.32:6800/9088303,v1:172.21.15.32:6801/9088303] -- osd_op(unknown.0.0:102 12.b 12:d0630e4c:::mds0_sessionmap:head [omap-set-header in=14b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96) -- 0x5567d94d93e0 con 0x5567d94d6fc0 2024-08-03T20:56:19.412 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.411+0000 7f896fe00640 1 -- 172.21.15.32:0/1013615250 <== osd.3 v2:172.21.15.32:6800/9088303 4 ==== osd_op_reply(102 mds0_sessionmap [omap-set-header] v96'52 uv52 ondisk = 0) ==== 159+0+0 (crc 0 0 0) 0x7f896000b040 con 0x5567d94d6fc0 2024-08-03T20:56:19.412 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.411+0000 7f897bb93f40 1 -- 172.21.15.32:0/1013615250 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:103 12.7 12:e0b41d06:::609.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96) -- 0x5567d94d93e0 con 0x5567d94cb710 2024-08-03T20:56:19.414 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.412+0000 7f896fe00640 1 -- 172.21.15.32:0/1013615250 <== osd.1 v2:172.21.15.32:6824/1551241759 13 ==== osd_op_reply(103 609.00000000 [create] v96'63 uv62 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f896000b040 con 0x5567d94cb710 2024-08-03T20:56:19.414 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.412+0000 7f897bb93f40 1 -- 172.21.15.32:0/1013615250 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:104 12.7 12:e0b41d06:::609.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e96) -- 0x5567d94d93e0 con 0x5567d94cb710 2024-08-03T20:56:19.415 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.414+0000 7f896fe00640 1 -- 172.21.15.32:0/1013615250 <== osd.1 v2:172.21.15.32:6824/1551241759 14 ==== osd_op_reply(104 609.00000000 [delete] v96'64 uv64 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f896000b040 con 0x5567d94cb710 2024-08-03T20:56:19.415 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.414+0000 7f897bb93f40 1 -- 172.21.15.32:0/1013615250 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:105 12.7 12:e0b41d06:::609.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96) -- 0x5567d94d93e0 con 0x5567d94cb710 2024-08-03T20:56:19.416 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::609.00000000:head# 2024-08-03T20:56:19.417 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.415+0000 7f896fe00640 1 -- 172.21.15.32:0/1013615250 <== osd.1 v2:172.21.15.32:6824/1551241759 15 ==== osd_op_reply(105 609.00000000 [create] v96'65 uv65 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f896000b040 con 0x5567d94cb710 2024-08-03T20:56:19.417 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.415+0000 7f897bb93f40 1 -- 172.21.15.32:0/1013615250 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:106 12.7 12:e0b41d06:::609.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96) -- 0x5567d94d93e0 con 0x5567d94cb710 2024-08-03T20:56:19.418 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.417+0000 7f896fe00640 1 -- 172.21.15.32:0/1013615250 <== osd.1 v2:172.21.15.32:6824/1551241759 16 ==== osd_op_reply(106 609.00000000 [setxattr (30)] v96'66 uv66 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f896000b040 con 0x5567d94cb710 2024-08-03T20:56:19.418 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.417+0000 7f897bb93f40 1 -- 172.21.15.32:0/1013615250 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:107 12.7 12:e0b41d06:::609.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96) -- 0x5567d94d93e0 con 0x5567d94cb710 2024-08-03T20:56:19.419 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.418+0000 7f896fe00640 1 -- 172.21.15.32:0/1013615250 <== osd.1 v2:172.21.15.32:6824/1551241759 17 ==== osd_op_reply(107 609.00000000 [setxattr (62)] v96'67 uv67 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f896000b040 con 0x5567d94cb710 2024-08-03T20:56:19.419 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.418+0000 7f897bb93f40 1 -- 172.21.15.32:0/1013615250 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:108 12.7 12:e0b41d06:::609.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96) -- 0x5567d94d93e0 con 0x5567d94cb710 2024-08-03T20:56:19.421 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.419+0000 7f896fe00640 1 -- 172.21.15.32:0/1013615250 <== osd.1 v2:172.21.15.32:6824/1551241759 18 ==== osd_op_reply(108 609.00000000 [omap-set-header] v96'68 uv68 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f896000b040 con 0x5567d94cb710 2024-08-03T20:56:19.421 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.420+0000 7f897bb93f40 1 -- 172.21.15.32:0/1013615250 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:109 12.f 12:f1f69726:::400.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96) -- 0x5567d94d93e0 con 0x5567d94cebf0 2024-08-03T20:56:19.423 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.422+0000 7f8975800640 1 -- 172.21.15.32:0/1013615250 <== osd.6 v2:172.21.15.118:6825/801947386 12 ==== osd_op_reply(109 400.00000000 [create] v96'52 uv51 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f897009fd00 con 0x5567d94cebf0 2024-08-03T20:56:19.423 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.422+0000 7f897bb93f40 1 -- 172.21.15.32:0/1013615250 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:110 12.f 12:f1f69726:::400.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e96) -- 0x5567d94d93e0 con 0x5567d94cebf0 2024-08-03T20:56:19.424 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.424+0000 7f8975800640 1 -- 172.21.15.32:0/1013615250 <== osd.6 v2:172.21.15.118:6825/801947386 13 ==== osd_op_reply(110 400.00000000 [delete] v96'53 uv53 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f897009fd00 con 0x5567d94cebf0 2024-08-03T20:56:19.424 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.424+0000 7f897bb93f40 1 -- 172.21.15.32:0/1013615250 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:111 12.f 12:f1f69726:::400.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96) -- 0x5567d94d93e0 con 0x5567d94cebf0 2024-08-03T20:56:19.426 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::400.00000000:head# 2024-08-03T20:56:19.426 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.425+0000 7f8975800640 1 -- 172.21.15.32:0/1013615250 <== osd.6 v2:172.21.15.118:6825/801947386 14 ==== osd_op_reply(111 400.00000000 [create] v96'54 uv54 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f897009fd00 con 0x5567d94cebf0 2024-08-03T20:56:19.426 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.425+0000 7f897bb93f40 1 -- 172.21.15.32:0/1013615250 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:112 12.f 12:f1f69726:::400.00000000:head [write 0~22 in=22b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96) -- 0x5567d94d93e0 con 0x5567d94cebf0 2024-08-03T20:56:19.427 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.427+0000 7f8975800640 1 -- 172.21.15.32:0/1013615250 <== osd.6 v2:172.21.15.118:6825/801947386 15 ==== osd_op_reply(112 400.00000000 [write 0~22] v96'55 uv55 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f897009fd00 con 0x5567d94cebf0 2024-08-03T20:56:19.427 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.427+0000 7f897bb93f40 1 -- 172.21.15.32:0/1013615250 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:113 12.f 12:f1f69726:::400.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96) -- 0x5567d94d93e0 con 0x5567d94cebf0 2024-08-03T20:56:19.429 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.429+0000 7f8975800640 1 -- 172.21.15.32:0/1013615250 <== osd.6 v2:172.21.15.118:6825/801947386 16 ==== osd_op_reply(113 400.00000000 [omap-set-header] v96'56 uv56 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f897009fd00 con 0x5567d94cebf0 2024-08-03T20:56:19.429 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.429+0000 7f897bb93f40 1 -- 172.21.15.32:0/1013615250 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:114 12.1f 12:fe07bba1:::100.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96) -- 0x5567d94d93e0 con 0x5567d94cebf0 2024-08-03T20:56:19.431 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.430+0000 7f8975800640 1 -- 172.21.15.32:0/1013615250 <== osd.6 v2:172.21.15.118:6825/801947386 17 ==== osd_op_reply(114 100.00000000.inode [create] v96'132 uv123 ondisk = -17 ((17) File exists)) ==== 162+0+0 (crc 0 0 0) 0x7f897009fd00 con 0x5567d94cebf0 2024-08-03T20:56:19.431 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.430+0000 7f897bb93f40 1 -- 172.21.15.32:0/1013615250 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:115 12.1f 12:fe07bba1:::100.00000000.inode:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e96) -- 0x5567d94d93e0 con 0x5567d94cebf0 2024-08-03T20:56:19.433 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.432+0000 7f8975800640 1 -- 172.21.15.32:0/1013615250 <== osd.6 v2:172.21.15.118:6825/801947386 18 ==== osd_op_reply(115 100.00000000.inode [delete] v96'133 uv133 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7f897009fd00 con 0x5567d94cebf0 2024-08-03T20:56:19.433 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.432+0000 7f897bb93f40 1 -- 172.21.15.32:0/1013615250 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:116 12.1f 12:fe07bba1:::100.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96) -- 0x5567d94d9e70 con 0x5567d94cebf0 2024-08-03T20:56:19.434 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.432+0000 7f8975800640 1 -- 172.21.15.32:0/1013615250 <== osd.6 v2:172.21.15.118:6825/801947386 19 ==== osd_op_reply(116 100.00000000.inode [create] v96'134 uv134 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7f897009fd00 con 0x5567d94cebf0 2024-08-03T20:56:19.434 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::100.00000000.inode:head# 2024-08-03T20:56:19.434 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.433+0000 7f897bb93f40 1 -- 172.21.15.32:0/1013615250 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:117 12.1f 12:fe07bba1:::100.00000000.inode:head [write 0~582 in=582b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96) -- 0x5567d94d9e70 con 0x5567d94cebf0 2024-08-03T20:56:19.435 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.434+0000 7f8975800640 1 -- 172.21.15.32:0/1013615250 <== osd.6 v2:172.21.15.118:6825/801947386 20 ==== osd_op_reply(117 100.00000000.inode [write 0~582] v96'135 uv135 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7f897009fd00 con 0x5567d94cebf0 2024-08-03T20:56:19.436 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.434+0000 7f897bb93f40 1 -- 172.21.15.32:0/1013615250 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:118 12.1f 12:fe07bba1:::100.00000000.inode:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96) -- 0x5567d94d9e20 con 0x5567d94cebf0 2024-08-03T20:56:19.437 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.436+0000 7f8975800640 1 -- 172.21.15.32:0/1013615250 <== osd.6 v2:172.21.15.118:6825/801947386 21 ==== osd_op_reply(118 100.00000000.inode [omap-set-header] v96'136 uv136 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7f897009fd00 con 0x5567d94cebf0 2024-08-03T20:56:19.437 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.436+0000 7f897bb93f40 1 -- 172.21.15.32:0/1013615250 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:119 12.1f 12:ff5b34d6:::1.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96) -- 0x5567d94d9e20 con 0x5567d94cebf0 2024-08-03T20:56:19.438 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.437+0000 7f8975800640 1 -- 172.21.15.32:0/1013615250 <== osd.6 v2:172.21.15.118:6825/801947386 22 ==== osd_op_reply(119 1.00000000 [create] v96'137 uv130 ondisk = -17 ((17) File exists)) ==== 154+0+0 (crc 0 0 0) 0x7f897009fd00 con 0x5567d94cebf0 2024-08-03T20:56:19.439 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.437+0000 7f897bb93f40 1 -- 172.21.15.32:0/1013615250 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:120 12.1f 12:ff5b34d6:::1.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e96) -- 0x5567d94d9e20 con 0x5567d94cebf0 2024-08-03T20:56:19.440 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.439+0000 7f8975800640 1 -- 172.21.15.32:0/1013615250 <== osd.6 v2:172.21.15.118:6825/801947386 23 ==== osd_op_reply(120 1.00000000 [delete] v96'138 uv138 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7f897009fd00 con 0x5567d94cebf0 2024-08-03T20:56:19.440 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.439+0000 7f897bb93f40 1 -- 172.21.15.32:0/1013615250 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:121 12.1f 12:ff5b34d6:::1.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96) -- 0x5567d94d9e20 con 0x5567d94cebf0 2024-08-03T20:56:19.441 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.440+0000 7f8975800640 1 -- 172.21.15.32:0/1013615250 <== osd.6 v2:172.21.15.118:6825/801947386 24 ==== osd_op_reply(121 1.00000000 [create] v96'139 uv139 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7f897009fd00 con 0x5567d94cebf0 2024-08-03T20:56:19.442 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::1.00000000:head# 2024-08-03T20:56:19.442 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.440+0000 7f897bb93f40 1 -- 172.21.15.32:0/1013615250 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:122 12.1f 12:ff5b34d6:::1.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96) -- 0x5567d94dae30 con 0x5567d94cebf0 2024-08-03T20:56:19.443 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.441+0000 7f8975800640 1 -- 172.21.15.32:0/1013615250 <== osd.6 v2:172.21.15.118:6825/801947386 25 ==== osd_op_reply(122 1.00000000 [setxattr (30)] v96'140 uv140 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7f897009fd00 con 0x5567d94cebf0 2024-08-03T20:56:19.443 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.442+0000 7f897bb93f40 1 -- 172.21.15.32:0/1013615250 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:123 12.1f 12:ff5b34d6:::1.00000000:head [setxattr parent (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96) -- 0x5567d94dae30 con 0x5567d94cebf0 2024-08-03T20:56:19.444 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.443+0000 7f8975800640 1 -- 172.21.15.32:0/1013615250 <== osd.6 v2:172.21.15.118:6825/801947386 26 ==== osd_op_reply(123 1.00000000 [setxattr (30)] v96'141 uv141 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7f897009fd00 con 0x5567d94cebf0 2024-08-03T20:56:19.444 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.443+0000 7f897bb93f40 1 -- 172.21.15.32:0/1013615250 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:124 12.1f 12:ff5b34d6:::1.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96) -- 0x5567d94d9e20 con 0x5567d94cebf0 2024-08-03T20:56:19.446 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.445+0000 7f8975800640 1 -- 172.21.15.32:0/1013615250 <== osd.6 v2:172.21.15.118:6825/801947386 27 ==== osd_op_reply(124 1.00000000 [omap-set-header] v96'142 uv142 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7f897009fd00 con 0x5567d94cebf0 2024-08-03T20:56:19.446 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.445+0000 7f897bb93f40 1 -- 172.21.15.32:0/1013615250 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:125 12.1f 12:ff5b34d6:::1.00000000:head [omap-set-vals in=584b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96) -- 0x5567d94da2d0 con 0x5567d94cebf0 2024-08-03T20:56:19.449 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.446+0000 7f8975800640 1 -- 172.21.15.32:0/1013615250 <== osd.6 v2:172.21.15.118:6825/801947386 28 ==== osd_op_reply(125 1.00000000 [omap-set-vals] v96'143 uv143 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7f897009fd00 con 0x5567d94cebf0 2024-08-03T20:56:19.449 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.446+0000 7f897bb93f40 1 -- 172.21.15.32:0/1013615250 >> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] conn(0x5567d9420930 msgr2=0x5567d941c5d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:56:19.449 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.446+0000 7f897bb93f40 1 --2- 172.21.15.32:0/1013615250 >> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] conn(0x5567d9420930 0x5567d941c5d0 crc :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:19.449 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.446+0000 7f897bb93f40 1 -- 172.21.15.32:0/1013615250 >> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] conn(0x5567d94cb710 msgr2=0x5567d94cbb20 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:56:19.449 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.446+0000 7f897bb93f40 1 --2- 172.21.15.32:0/1013615250 >> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] conn(0x5567d94cb710 0x5567d94cbb20 crc :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:19.449 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.446+0000 7f897bb93f40 1 -- 172.21.15.32:0/1013615250 >> [v2:172.21.15.32:6800/9088303,v1:172.21.15.32:6801/9088303] conn(0x5567d94d6fc0 msgr2=0x5567d94d98e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:56:19.449 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.446+0000 7f897bb93f40 1 --2- 172.21.15.32:0/1013615250 >> [v2:172.21.15.32:6800/9088303,v1:172.21.15.32:6801/9088303] conn(0x5567d94d6fc0 0x5567d94d98e0 crc :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:19.449 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.446+0000 7f897bb93f40 1 -- 172.21.15.32:0/1013615250 >> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] conn(0x5567d94c6a30 msgr2=0x5567d94c8e50 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:56:19.449 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.446+0000 7f8975800640 1 -- 172.21.15.32:0/1013615250 reap_dead start 2024-08-03T20:56:19.449 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.446+0000 7f897bb93f40 1 --2- 172.21.15.32:0/1013615250 >> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] conn(0x5567d94c6a30 0x5567d94c8e50 crc :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:19.449 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.446+0000 7f897bb93f40 1 -- 172.21.15.32:0/1013615250 >> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] conn(0x5567d94cebf0 msgr2=0x5567d94d1010 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:56:19.450 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.446+0000 7f897bb93f40 1 --2- 172.21.15.32:0/1013615250 >> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] conn(0x5567d94cebf0 0x5567d94d1010 crc :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:19.450 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.446+0000 7f897bb93f40 1 -- 172.21.15.32:0/1013615250 >> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] conn(0x5567d94d30a0 msgr2=0x5567d94d5f00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:56:19.450 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.446+0000 7f897bb93f40 1 --2- 172.21.15.32:0/1013615250 >> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] conn(0x5567d94d30a0 0x5567d94d5f00 crc :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:19.450 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.447+0000 7f897bb93f40 1 -- 172.21.15.32:0/1013615250 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f89540759f0 msgr2=0x7f8954077f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:56:19.450 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.447+0000 7f897bb93f40 1 --2- 172.21.15.32:0/1013615250 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f89540759f0 0x7f8954077f00 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7f8968004fd0 tx=0x7f8968004880 comp rx=0 tx=0).stop 2024-08-03T20:56:19.450 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.447+0000 7f897bb93f40 1 -- 172.21.15.32:0/1013615250 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f895c003680 msgr2=0x5567d938d850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:56:19.450 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.447+0000 7f897bb93f40 1 --2- 172.21.15.32:0/1013615250 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f895c003680 0x5567d938d850 secure :-1 s=READY pgs=1336 cs=0 l=1 rev1=1 crypto rx=0x7f8970009d60 tx=0x7f89700098b0 comp rx=0 tx=0).stop 2024-08-03T20:56:19.450 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.447+0000 7f8975800640 1 -- 172.21.15.32:0/1013615250 reap_dead start 2024-08-03T20:56:19.450 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.447+0000 7f897bb93f40 1 -- 172.21.15.32:0/1013615250 shutdown_connections 2024-08-03T20:56:19.450 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.447+0000 7f897bb93f40 1 -- 172.21.15.32:0/1013615250 >> 172.21.15.32:0/1013615250 conn(0x5567d926bac0 msgr2=0x5567d9394880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:56:19.450 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.447+0000 7f897bb93f40 1 -- 172.21.15.32:0/1013615250 shutdown_connections 2024-08-03T20:56:19.450 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.447+0000 7f897bb93f40 1 -- 172.21.15.32:0/1013615250 wait complete. 2024-08-03T20:56:19.463 DEBUG:teuthology.orchestra.run.smithi032:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata rm mds0_sessionmap 2024-08-03T20:56:19.606 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.606+0000 7f5555625f40 1 Processor -- start 2024-08-03T20:56:19.607 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.606+0000 7f5555625f40 1 -- start start 2024-08-03T20:56:19.609 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.607+0000 7f5555625f40 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x563f183d0360 0x563f184787f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:56:19.609 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.607+0000 7f5555625f40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x563f183d1ea0 0x563f18317da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:56:19.609 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.607+0000 7f5555625f40 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x563f183edcb0 0x563f183ec070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:56:19.609 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.607+0000 7f5555625f40 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x563f18267e30 con 0x563f183edcb0 2024-08-03T20:56:19.609 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.607+0000 7f5555625f40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x563f183ee0c0 con 0x563f183d1ea0 2024-08-03T20:56:19.609 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.607+0000 7f5555625f40 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x563f18478fb0 con 0x563f183d0360 2024-08-03T20:56:19.609 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.607+0000 7f554f200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x563f183edcb0 0x563f183ec070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:56:19.609 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.607+0000 7f554f200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x563f183edcb0 0x563f183ec070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:47074/0 (socket says 172.21.15.32:47074) 2024-08-03T20:56:19.609 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.607+0000 7f554f200640 1 -- 172.21.15.32:0/805833350 learned_addr learned my addr 172.21.15.32:0/805833350 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:56:19.609 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.607+0000 7f554de00640 1 --2- 172.21.15.32:0/805833350 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x563f183d1ea0 0x563f18317da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:56:19.609 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.607+0000 7f554e800640 1 --2- 172.21.15.32:0/805833350 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x563f183d0360 0x563f184787f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:56:19.610 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.607+0000 7f554f200640 1 -- 172.21.15.32:0/805833350 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x563f183d0360 msgr2=0x563f184787f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:56:19.610 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.607+0000 7f554f200640 1 --2- 172.21.15.32:0/805833350 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x563f183d0360 0x563f184787f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:19.610 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.607+0000 7f554f200640 1 -- 172.21.15.32:0/805833350 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x563f183d1ea0 msgr2=0x563f18317da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:56:19.610 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.607+0000 7f554f200640 1 --2- 172.21.15.32:0/805833350 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x563f183d1ea0 0x563f18317da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:19.610 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.607+0000 7f554f200640 1 -- 172.21.15.32:0/805833350 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563f18240f10 con 0x563f183edcb0 2024-08-03T20:56:19.610 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.607+0000 7f554de00640 1 --2- 172.21.15.32:0/805833350 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x563f183d1ea0 0x563f18317da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-03T20:56:19.611 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.608+0000 7f554f200640 1 --2- 172.21.15.32:0/805833350 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x563f183edcb0 0x563f183ec070 secure :-1 s=READY pgs=1338 cs=0 l=1 rev1=1 crypto rx=0x7f5548025d30 tx=0x7f5548003330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=13ef94ff5c802675 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:56:19.613 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.608+0000 7f554ca00640 1 -- 172.21.15.32:0/805833350 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5548064030 con 0x563f183edcb0 2024-08-03T20:56:19.613 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.608+0000 7f554ca00640 1 -- 172.21.15.32:0/805833350 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f5548004020 con 0x563f183edcb0 2024-08-03T20:56:19.613 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.608+0000 7f554ca00640 1 -- 172.21.15.32:0/805833350 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5548004270 con 0x563f183edcb0 2024-08-03T20:56:19.613 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.609+0000 7f5555625f40 1 -- 172.21.15.32:0/805833350 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x563f183edcb0 msgr2=0x563f183ec070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:56:19.614 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.609+0000 7f5555625f40 1 --2- 172.21.15.32:0/805833350 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x563f183edcb0 0x563f183ec070 secure :-1 s=READY pgs=1338 cs=0 l=1 rev1=1 crypto rx=0x7f5548025d30 tx=0x7f5548003330 comp rx=0 tx=0).stop 2024-08-03T20:56:19.614 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.609+0000 7f5555625f40 1 -- 172.21.15.32:0/805833350 shutdown_connections 2024-08-03T20:56:19.614 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.609+0000 7f5555625f40 1 --2- 172.21.15.32:0/805833350 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x563f183d1ea0 0x563f18317da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:19.614 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.609+0000 7f5555625f40 1 --2- 172.21.15.32:0/805833350 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x563f183edcb0 0x563f183ec070 unknown :-1 s=CLOSED pgs=1338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:19.614 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.609+0000 7f5555625f40 1 --2- 172.21.15.32:0/805833350 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x563f183d0360 0x563f184787f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:19.614 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.609+0000 7f5555625f40 1 -- 172.21.15.32:0/805833350 >> 172.21.15.32:0/805833350 conn(0x563f18239ac0 msgr2=0x563f183e6680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:56:19.614 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.609+0000 7f5555625f40 1 -- 172.21.15.32:0/805833350 shutdown_connections 2024-08-03T20:56:19.614 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.609+0000 7f5555625f40 1 -- 172.21.15.32:0/805833350 wait complete. 2024-08-03T20:56:19.614 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.610+0000 7f5555625f40 1 Processor -- start 2024-08-03T20:56:19.616 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.610+0000 7f5555625f40 1 -- start start 2024-08-03T20:56:19.616 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.612+0000 7f5555625f40 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x563f183d0360 0x563f183c36e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:56:19.616 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.612+0000 7f5555625f40 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x563f183d1ea0 0x563f183e82a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:56:19.616 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.612+0000 7f5555625f40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x563f183edcb0 0x563f18360770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:56:19.616 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.612+0000 7f5555625f40 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x563f183ea500 con 0x563f183d1ea0 2024-08-03T20:56:19.616 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.612+0000 7f5555625f40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x563f183e9fc0 con 0x563f183edcb0 2024-08-03T20:56:19.617 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.612+0000 7f5555625f40 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x563f183e8d90 con 0x563f183d0360 2024-08-03T20:56:19.617 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.612+0000 7f554de00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x563f183d1ea0 0x563f183e82a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:56:19.617 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.612+0000 7f554de00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x563f183d1ea0 0x563f183e82a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:47084/0 (socket says 172.21.15.32:47084) 2024-08-03T20:56:19.617 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.612+0000 7f554de00640 1 -- 172.21.15.32:0/1087055714 learned_addr learned my addr 172.21.15.32:0/1087055714 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:56:19.617 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.612+0000 7f554e800640 1 --2- 172.21.15.32:0/1087055714 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x563f183d0360 0x563f183c36e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:56:19.617 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.612+0000 7f554f200640 1 --2- 172.21.15.32:0/1087055714 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x563f183edcb0 0x563f18360770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:56:19.617 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.613+0000 7f554de00640 1 -- 172.21.15.32:0/1087055714 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x563f183d0360 msgr2=0x563f183c36e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:56:19.617 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.613+0000 7f554de00640 1 --2- 172.21.15.32:0/1087055714 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x563f183d0360 0x563f183c36e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:19.617 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.613+0000 7f554de00640 1 -- 172.21.15.32:0/1087055714 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x563f183edcb0 msgr2=0x563f18360770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:56:19.618 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.613+0000 7f554de00640 1 --2- 172.21.15.32:0/1087055714 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x563f183edcb0 0x563f18360770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:19.618 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.613+0000 7f554de00640 1 -- 172.21.15.32:0/1087055714 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563f1847a410 con 0x563f183d1ea0 2024-08-03T20:56:19.618 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.613+0000 7f554de00640 1 --2- 172.21.15.32:0/1087055714 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x563f183d1ea0 0x563f183e82a0 secure :-1 s=READY pgs=1339 cs=0 l=1 rev1=1 crypto rx=0x7f553c0086c0 tx=0x7f553c0087b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:56:19.618 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.613+0000 7f5547400640 1 -- 172.21.15.32:0/1087055714 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f553c00a850 con 0x563f183d1ea0 2024-08-03T20:56:19.618 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.613+0000 7f5555625f40 1 -- 172.21.15.32:0/1087055714 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x563f18362400 con 0x563f183d1ea0 2024-08-03T20:56:19.618 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.613+0000 7f5547400640 1 -- 172.21.15.32:0/1087055714 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f553c00a9f0 con 0x563f183d1ea0 2024-08-03T20:56:19.618 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.613+0000 7f5547400640 1 -- 172.21.15.32:0/1087055714 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f553c00ae20 con 0x563f183d1ea0 2024-08-03T20:56:19.618 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.614+0000 7f5555625f40 1 -- 172.21.15.32:0/1087055714 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563f183e5be0 con 0x563f183d1ea0 2024-08-03T20:56:19.619 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.616+0000 7f5547400640 1 -- 172.21.15.32:0/1087055714 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f553c016430 con 0x563f183d1ea0 2024-08-03T20:56:19.619 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.617+0000 7f5547400640 1 --2- 172.21.15.32:0/1087055714 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f5530075a20 0x7f5530077f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:56:19.619 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.617+0000 7f5547400640 1 -- 172.21.15.32:0/1087055714 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 6800+0+0 (secure 0 0 0) 0x7f553c0981b0 con 0x563f183d1ea0 2024-08-03T20:56:19.619 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.617+0000 7f5555625f40 1 --2- 172.21.15.32:0/1087055714 >> [v2:172.21.15.32:6800/9088303,v1:172.21.15.32:6801/9088303] conn(0x563f183ec580 0x563f184899a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:56:19.619 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.617+0000 7f554e800640 1 --2- 172.21.15.32:0/1087055714 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f5530075a20 0x7f5530077f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:56:19.619 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.617+0000 7f5555625f40 1 -- 172.21.15.32:0/1087055714 --> [v2:172.21.15.32:6800/9088303,v1:172.21.15.32:6801/9088303] -- osd_op(unknown.0.0:1 12.b 12:d0630e4c:::mds0_sessionmap:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e96) -- 0x563f18360cb0 con 0x563f183ec580 2024-08-03T20:56:19.619 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.618+0000 7f554f200640 1 --2- 172.21.15.32:0/1087055714 >> [v2:172.21.15.32:6800/9088303,v1:172.21.15.32:6801/9088303] conn(0x563f183ec580 0x563f184899a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:56:19.620 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.618+0000 7f554e800640 1 --2- 172.21.15.32:0/1087055714 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f5530075a20 0x7f5530077f30 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x563f183f0780 tx=0x7f554000f040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:56:19.620 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.618+0000 7f554f200640 1 --2- 172.21.15.32:0/1087055714 >> [v2:172.21.15.32:6800/9088303,v1:172.21.15.32:6801/9088303] conn(0x563f183ec580 0x563f184899a0 crc :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:56:19.622 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.620+0000 7f554f200640 1 -- 172.21.15.32:0/1087055714 <== osd.3 v2:172.21.15.32:6800/9088303 1 ==== osd_op_reply(1 mds0_sessionmap [delete] v96'53 uv53 ondisk = 0) ==== 159+0+0 (crc 0 0 0) 0x7f554800b040 con 0x563f183ec580 2024-08-03T20:56:19.622 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.620+0000 7f5555625f40 1 -- 172.21.15.32:0/1087055714 >> [v2:172.21.15.32:6800/9088303,v1:172.21.15.32:6801/9088303] conn(0x563f183ec580 msgr2=0x563f184899a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:56:19.622 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.620+0000 7f5555625f40 1 --2- 172.21.15.32:0/1087055714 >> [v2:172.21.15.32:6800/9088303,v1:172.21.15.32:6801/9088303] conn(0x563f183ec580 0x563f184899a0 crc :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:19.622 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.620+0000 7f5555625f40 1 -- 172.21.15.32:0/1087055714 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f5530075a20 msgr2=0x7f5530077f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:56:19.622 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.620+0000 7f5555625f40 1 --2- 172.21.15.32:0/1087055714 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f5530075a20 0x7f5530077f30 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x563f183f0780 tx=0x7f554000f040 comp rx=0 tx=0).stop 2024-08-03T20:56:19.622 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.620+0000 7f5555625f40 1 -- 172.21.15.32:0/1087055714 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x563f183d1ea0 msgr2=0x563f183e82a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:56:19.622 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.620+0000 7f5555625f40 1 --2- 172.21.15.32:0/1087055714 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x563f183d1ea0 0x563f183e82a0 secure :-1 s=READY pgs=1339 cs=0 l=1 rev1=1 crypto rx=0x7f553c0086c0 tx=0x7f553c0087b0 comp rx=0 tx=0).stop 2024-08-03T20:56:19.623 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.620+0000 7f554f200640 1 -- 172.21.15.32:0/1087055714 reap_dead start 2024-08-03T20:56:19.623 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.620+0000 7f5555625f40 1 -- 172.21.15.32:0/1087055714 shutdown_connections 2024-08-03T20:56:19.623 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.620+0000 7f5555625f40 1 -- 172.21.15.32:0/1087055714 >> 172.21.15.32:0/1087055714 conn(0x563f18239ac0 msgr2=0x563f183e6680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:56:19.623 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.620+0000 7f5555625f40 1 -- 172.21.15.32:0/1087055714 shutdown_connections 2024-08-03T20:56:19.623 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.621+0000 7f5555625f40 1 -- 172.21.15.32:0/1087055714 wait complete. 2024-08-03T20:56:19.638 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs joinable true 2024-08-03T20:56:19.870 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.868+0000 7f41f6e00640 1 -- 172.21.15.32:0/998011721 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f41f01609d0 msgr2=0x7f41f00cc050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:56:19.870 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.868+0000 7f41f6e00640 1 --2- 172.21.15.32:0/998011721 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f41f01609d0 0x7f41f00cc050 secure :-1 s=READY pgs=1340 cs=0 l=1 rev1=1 crypto rx=0x7f41e4005fd0 tx=0x7f41e402a5d0 comp rx=0 tx=0).stop 2024-08-03T20:56:19.870 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.868+0000 7f41f6e00640 1 -- 172.21.15.32:0/998011721 shutdown_connections 2024-08-03T20:56:19.870 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.868+0000 7f41f6e00640 1 --2- 172.21.15.32:0/998011721 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f41f0159ff0 0x7f41f015fe10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:19.870 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.868+0000 7f41f6e00640 1 --2- 172.21.15.32:0/998011721 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f41f01609d0 0x7f41f00cc050 unknown :-1 s=CLOSED pgs=1340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:19.870 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.868+0000 7f41f6e00640 1 -- 172.21.15.32:0/998011721 >> 172.21.15.32:0/998011721 conn(0x7f41f00cfd80 msgr2=0x7f41f00d0190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:56:19.871 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.868+0000 7f41f6e00640 1 -- 172.21.15.32:0/998011721 shutdown_connections 2024-08-03T20:56:19.871 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.868+0000 7f41f6e00640 1 -- 172.21.15.32:0/998011721 wait complete. 2024-08-03T20:56:19.872 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.870+0000 7f41f6e00640 1 Processor -- start 2024-08-03T20:56:19.872 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.870+0000 7f41f6e00640 1 -- start start 2024-08-03T20:56:19.875 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.873+0000 7f41f6e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f41f0159ff0 0x7f41f00c8ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:56:19.875 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.873+0000 7f41f6e00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f41f0160350 0x7f41f014b900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:56:19.875 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.873+0000 7f41f6e00640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f41f01609d0 0x7f41f014e1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:56:19.875 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.873+0000 7f41f6e00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f41f015d0b0 con 0x7f41f0159ff0 2024-08-03T20:56:19.875 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.873+0000 7f41f6e00640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f41f015cf30 con 0x7f41f0160350 2024-08-03T20:56:19.875 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.873+0000 7f41f6e00640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7f41f015d230 con 0x7f41f01609d0 2024-08-03T20:56:19.875 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.873+0000 7f41f5a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f41f0159ff0 0x7f41f00c8ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:56:19.875 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.873+0000 7f41f5a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f41f0159ff0 0x7f41f00c8ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:50762/0 (socket says 172.21.15.32:50762) 2024-08-03T20:56:19.876 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.873+0000 7f41f5a00640 1 -- 172.21.15.32:0/1527692718 learned_addr learned my addr 172.21.15.32:0/1527692718 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:56:19.876 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.873+0000 7f41f5000640 1 --2- 172.21.15.32:0/1527692718 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f41f0160350 0x7f41f014b900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:56:19.876 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.873+0000 7f41f6400640 1 --2- 172.21.15.32:0/1527692718 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f41f01609d0 0x7f41f014e1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:56:19.876 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.873+0000 7f41f5a00640 1 -- 172.21.15.32:0/1527692718 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f41f01609d0 msgr2=0x7f41f014e1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:56:19.876 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.873+0000 7f41f5a00640 1 --2- 172.21.15.32:0/1527692718 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f41f01609d0 0x7f41f014e1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:19.876 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.873+0000 7f41f5a00640 1 -- 172.21.15.32:0/1527692718 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f41f0160350 msgr2=0x7f41f014b900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:56:19.876 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.873+0000 7f41f5a00640 1 --2- 172.21.15.32:0/1527692718 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f41f0160350 0x7f41f014b900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:19.876 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.873+0000 7f41f5a00640 1 -- 172.21.15.32:0/1527692718 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f41f014da30 con 0x7f41f0159ff0 2024-08-03T20:56:19.876 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.873+0000 7f41f6400640 1 --2- 172.21.15.32:0/1527692718 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f41f01609d0 0x7f41f014e1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-03T20:56:19.876 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.873+0000 7f41f5a00640 1 --2- 172.21.15.32:0/1527692718 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f41f0159ff0 0x7f41f00c8ec0 secure :-1 s=READY pgs=1341 cs=0 l=1 rev1=1 crypto rx=0x7f41e4034040 tx=0x7f41e40035f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:56:19.876 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.873+0000 7f41ed600640 1 -- 172.21.15.32:0/1527692718 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41e403e030 con 0x7f41f0159ff0 2024-08-03T20:56:19.876 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.873+0000 7f41ed600640 1 -- 172.21.15.32:0/1527692718 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f41e4003ae0 con 0x7f41f0159ff0 2024-08-03T20:56:19.876 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.873+0000 7f41ed600640 1 -- 172.21.15.32:0/1527692718 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41e4035040 con 0x7f41f0159ff0 2024-08-03T20:56:19.877 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.873+0000 7f41f6e00640 1 -- 172.21.15.32:0/1527692718 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f41f00c9400 con 0x7f41f0159ff0 2024-08-03T20:56:19.877 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.873+0000 7f41f6e00640 1 -- 172.21.15.32:0/1527692718 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f41f00c99b0 con 0x7f41f0159ff0 2024-08-03T20:56:19.877 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.875+0000 7f41f6e00640 1 -- 172.21.15.32:0/1527692718 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f41f015fe10 con 0x7f41f0159ff0 2024-08-03T20:56:19.877 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.877+0000 7f41ed600640 1 -- 172.21.15.32:0/1527692718 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f41e4041030 con 0x7f41f0159ff0 2024-08-03T20:56:19.878 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.877+0000 7f41ed600640 1 --2- 172.21.15.32:0/1527692718 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f41d0077010 0x7f41d0079520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:56:19.878 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.877+0000 7f41ed600640 1 -- 172.21.15.32:0/1527692718 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 6800+0+0 (secure 0 0 0) 0x7f41e40b9220 con 0x7f41f0159ff0 2024-08-03T20:56:19.878 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.877+0000 7f41f5000640 1 --2- 172.21.15.32:0/1527692718 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f41d0077010 0x7f41d0079520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:56:19.879 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.879+0000 7f41f5000640 1 --2- 172.21.15.32:0/1527692718 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f41d0077010 0x7f41d0079520 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7f41f00cd310 tx=0x7f41dc005f90 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:56:19.883 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:19.882+0000 7f41ed600640 1 -- 172.21.15.32:0/1527692718 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7f41e4081ff0 con 0x7f41f0159ff0 2024-08-03T20:56:20.130 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:20.129+0000 7f41f6e00640 1 -- 172.21.15.32:0/1527692718 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "joinable", "val": "true"} v 0) -- 0x7f41f00d09e0 con 0x7f41f0159ff0 2024-08-03T20:56:21.076 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:21.076+0000 7f41ed600640 1 -- 172.21.15.32:0/1527692718 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "joinable", "val": "true"}]=0 cephfs marked joinable; MDS may join as newly active. v159) ==== 162+0+0 (secure 0 0 0) 0x7f41e40818f0 con 0x7f41f0159ff0 2024-08-03T20:56:21.076 INFO:teuthology.orchestra.run.smithi032.stderr:cephfs marked joinable; MDS may join as newly active. 2024-08-03T20:56:21.079 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:21.079+0000 7f41f6e00640 1 -- 172.21.15.32:0/1527692718 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f41d0077010 msgr2=0x7f41d0079520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:56:21.080 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:21.079+0000 7f41f6e00640 1 --2- 172.21.15.32:0/1527692718 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f41d0077010 0x7f41d0079520 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7f41f00cd310 tx=0x7f41dc005f90 comp rx=0 tx=0).stop 2024-08-03T20:56:21.080 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:21.079+0000 7f41f6e00640 1 -- 172.21.15.32:0/1527692718 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f41f0159ff0 msgr2=0x7f41f00c8ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:56:21.080 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:21.079+0000 7f41f6e00640 1 --2- 172.21.15.32:0/1527692718 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f41f0159ff0 0x7f41f00c8ec0 secure :-1 s=READY pgs=1341 cs=0 l=1 rev1=1 crypto rx=0x7f41e4034040 tx=0x7f41e40035f0 comp rx=0 tx=0).stop 2024-08-03T20:56:21.080 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:21.079+0000 7f41f6e00640 1 -- 172.21.15.32:0/1527692718 shutdown_connections 2024-08-03T20:56:21.080 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:21.079+0000 7f41f6e00640 1 --2- 172.21.15.32:0/1527692718 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f41d0077010 0x7f41d0079520 unknown :-1 s=CLOSED pgs=553 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:21.080 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:21.079+0000 7f41f6e00640 1 --2- 172.21.15.32:0/1527692718 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f41f0160350 0x7f41f014b900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:21.080 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:21.079+0000 7f41f6e00640 1 --2- 172.21.15.32:0/1527692718 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f41f01609d0 0x7f41f014e1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:21.080 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:21.079+0000 7f41f6e00640 1 --2- 172.21.15.32:0/1527692718 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f41f0159ff0 0x7f41f00c8ec0 unknown :-1 s=CLOSED pgs=1341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:21.080 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:21.079+0000 7f41f6e00640 1 -- 172.21.15.32:0/1527692718 >> 172.21.15.32:0/1527692718 conn(0x7f41f00cfd80 msgr2=0x7f41f015f1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:56:21.080 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:21.079+0000 7f41f6e00640 1 -- 172.21.15.32:0/1527692718 shutdown_connections 2024-08-03T20:56:21.080 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:21.079+0000 7f41f6e00640 1 -- 172.21.15.32:0/1527692718 wait complete. 2024-08-03T20:56:21.088 INFO:tasks.ceph.mds.a.smithi032.stderr:2024-08-03T20:56:21.088+0000 7f5e6ea00640 -1 mds.0.sessionmap _load_finish got (2) No such file or directory 2024-08-03T20:56:21.088 INFO:tasks.ceph.mds.a.smithi032.stderr:2024-08-03T20:56:21.088+0000 7f5e6ea00640 -1 log_channel(cluster) log [ERR] : error reading sessionmap 'mds0_sessionmap' -2 ((2) No such file or directory) 2024-08-03T20:56:21.089 INFO:tasks.ceph.mon.a.smithi032.stderr:2024-08-03T20:56:21.088+0000 7f4e6d600640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-08-03T20:56:21.089 INFO:tasks.ceph.mon.a.smithi032.stderr:2024-08-03T20:56:21.088+0000 7f4e6d600640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 mds daemon damaged (MDS_DAMAGE) 2024-08-03T20:56:21.115 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-08-03T20:56:21.348 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:21.347+0000 7ff74be00640 1 -- 172.21.15.32:0/694978481 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff74c0d4bb0 msgr2=0x7ff74c0d1c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:56:21.348 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:21.347+0000 7ff74be00640 1 --2- 172.21.15.32:0/694978481 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff74c0d4bb0 0x7ff74c0d1c60 secure :-1 s=READY pgs=1343 cs=0 l=1 rev1=1 crypto rx=0x7ff74c0d5190 tx=0x7ff73c02a4a0 comp rx=0 tx=0).stop 2024-08-03T20:56:21.348 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:21.347+0000 7ff74be00640 1 -- 172.21.15.32:0/694978481 shutdown_connections 2024-08-03T20:56:21.348 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:21.347+0000 7ff74be00640 1 --2- 172.21.15.32:0/694978481 >> v2:172.21.15.118:3300/0 conn(0x7ff74c0d2290 0x7ff74c0d26a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:21.348 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:21.347+0000 7ff74be00640 1 --2- 172.21.15.32:0/694978481 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff74c0d4bb0 0x7ff74c0d1c60 unknown :-1 s=CLOSED pgs=1343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:21.348 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:21.347+0000 7ff74be00640 1 -- 172.21.15.32:0/694978481 >> 172.21.15.32:0/694978481 conn(0x7ff74c0d4090 msgr2=0x7ff74c0d44a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:56:21.348 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:21.347+0000 7ff74be00640 1 -- 172.21.15.32:0/694978481 shutdown_connections 2024-08-03T20:56:21.348 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:21.347+0000 7ff74be00640 1 -- 172.21.15.32:0/694978481 wait complete. 2024-08-03T20:56:21.350 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:21.349+0000 7ff74be00640 1 Processor -- start 2024-08-03T20:56:21.350 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:21.349+0000 7ff74be00640 1 -- start start 2024-08-03T20:56:21.352 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:21.351+0000 7ff74be00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff74c0d0aa0 0x7ff74c0ca250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:56:21.352 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:21.351+0000 7ff74be00640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7ff74c0d2290 0x7ff74c15de50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:56:21.352 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:21.351+0000 7ff74be00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ff74c0d4bb0 0x7ff74c15e390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:56:21.352 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:21.351+0000 7ff74be00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7ff74c0d15c0 con 0x7ff74c0d0aa0 2024-08-03T20:56:21.352 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:21.351+0000 7ff74be00640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7ff74c0cfe50 con 0x7ff74c0d4bb0 2024-08-03T20:56:21.353 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:21.351+0000 7ff74be00640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7ff74c0d1740 con 0x7ff74c0d2290 2024-08-03T20:56:21.353 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:21.351+0000 7ff74aa00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff74c0d0aa0 0x7ff74c0ca250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:56:21.353 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:21.351+0000 7ff74aa00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff74c0d0aa0 0x7ff74c0ca250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:50772/0 (socket says 172.21.15.32:50772) 2024-08-03T20:56:21.353 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:21.351+0000 7ff74aa00640 1 -- 172.21.15.32:0/1166962239 learned_addr learned my addr 172.21.15.32:0/1166962239 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:56:21.353 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:21.351+0000 7ff74b400640 1 --2- 172.21.15.32:0/1166962239 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ff74c0d4bb0 0x7ff74c15e390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:56:21.353 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:21.351+0000 7ff743e00640 1 --2- 172.21.15.32:0/1166962239 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7ff74c0d2290 0x7ff74c15de50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:56:21.353 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:21.352+0000 7ff74aa00640 1 -- 172.21.15.32:0/1166962239 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7ff74c0d2290 msgr2=0x7ff74c15de50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:56:21.353 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:21.352+0000 7ff74aa00640 1 --2- 172.21.15.32:0/1166962239 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7ff74c0d2290 0x7ff74c15de50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:21.353 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:21.352+0000 7ff74aa00640 1 -- 172.21.15.32:0/1166962239 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ff74c0d4bb0 msgr2=0x7ff74c15e390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:56:21.353 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:21.352+0000 7ff74aa00640 1 --2- 172.21.15.32:0/1166962239 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ff74c0d4bb0 0x7ff74c15e390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:21.353 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:21.352+0000 7ff74aa00640 1 -- 172.21.15.32:0/1166962239 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff74c15d6b0 con 0x7ff74c0d0aa0 2024-08-03T20:56:21.354 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:21.352+0000 7ff74aa00640 1 --2- 172.21.15.32:0/1166962239 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff74c0d0aa0 0x7ff74c0ca250 secure :-1 s=READY pgs=1344 cs=0 l=1 rev1=1 crypto rx=0x7ff73c02aa90 tx=0x7ff73c033c90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:56:21.354 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:21.352+0000 7ff742a00640 1 -- 172.21.15.32:0/1166962239 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff73c031d60 con 0x7ff74c0d0aa0 2024-08-03T20:56:21.354 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:21.352+0000 7ff742a00640 1 -- 172.21.15.32:0/1166962239 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7ff73c005020 con 0x7ff74c0d0aa0 2024-08-03T20:56:21.355 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:21.352+0000 7ff74be00640 1 -- 172.21.15.32:0/1166962239 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff74c161be0 con 0x7ff74c0d0aa0 2024-08-03T20:56:21.355 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:21.352+0000 7ff742a00640 1 -- 172.21.15.32:0/1166962239 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff73c005470 con 0x7ff74c0d0aa0 2024-08-03T20:56:21.355 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:21.352+0000 7ff74be00640 1 -- 172.21.15.32:0/1166962239 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff74c166b80 con 0x7ff74c0d0aa0 2024-08-03T20:56:21.355 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:21.353+0000 7ff74be00640 1 -- 172.21.15.32:0/1166962239 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff74c0d26a0 con 0x7ff74c0d0aa0 2024-08-03T20:56:21.355 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:21.355+0000 7ff742a00640 1 -- 172.21.15.32:0/1166962239 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7ff73c0047a0 con 0x7ff74c0d0aa0 2024-08-03T20:56:21.357 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:21.356+0000 7ff742a00640 1 --2- 172.21.15.32:0/1166962239 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7ff728077010 0x7ff728079520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:56:21.357 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:21.356+0000 7ff742a00640 1 -- 172.21.15.32:0/1166962239 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 6886+0+0 (secure 0 0 0) 0x7ff73c0b9f10 con 0x7ff74c0d0aa0 2024-08-03T20:56:21.357 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:21.357+0000 7ff743e00640 1 --2- 172.21.15.32:0/1166962239 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7ff728077010 0x7ff728079520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:56:21.358 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:21.357+0000 7ff743e00640 1 --2- 172.21.15.32:0/1166962239 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7ff728077010 0x7ff728079520 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7ff74c15fe00 tx=0x7ff73400b2e0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:56:21.361 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:21.361+0000 7ff742a00640 1 -- 172.21.15.32:0/1166962239 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7ff73c03f030 con 0x7ff74c0d0aa0 2024-08-03T20:56:21.605 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:21.605+0000 7ff74be00640 1 -- 172.21.15.32:0/1166962239 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff74c0d2fe0 con 0x7ff74c0d0aa0 2024-08-03T20:56:21.605 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:21.605+0000 7ff742a00640 1 -- 172.21.15.32:0/1166962239 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 161 v161) ==== 95+0+4390 (secure 0 0 0) 0x7ff73c082580 con 0x7ff74c0d0aa0 2024-08-03T20:56:21.606 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-03T20:56:21.606 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":161,"btime":"2024-08-03T20:56:21:089210+0000","default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6439,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6835/531080160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6834","nonce":531080160},{"type":"v1","addr":"172.21.15.118:6835","nonce":531080160}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":84},{"gid":6454,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6837/2796882319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6836","nonce":2796882319},{"type":"v1","addr":"172.21.15.118:6837","nonce":2796882319}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":86},{"gid":6813,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6834/2686933890","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":2686933890},{"type":"v1","addr":"172.21.15.32:6834","nonce":2686933890}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":85}],"filesystems":[{"mdsmap":{"epoch":161,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-08-03T20:54:26.950518+0000","modified":"2024-08-03T20:56:21.089141+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":97,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":6}]} 2024-08-03T20:56:21.606 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 161 2024-08-03T20:56:21.609 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:21.608+0000 7ff74be00640 1 -- 172.21.15.32:0/1166962239 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7ff728077010 msgr2=0x7ff728079520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:56:21.609 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:21.608+0000 7ff74be00640 1 --2- 172.21.15.32:0/1166962239 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7ff728077010 0x7ff728079520 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7ff74c15fe00 tx=0x7ff73400b2e0 comp rx=0 tx=0).stop 2024-08-03T20:56:21.609 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:21.608+0000 7ff74be00640 1 -- 172.21.15.32:0/1166962239 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff74c0d0aa0 msgr2=0x7ff74c0ca250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:56:21.609 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:21.608+0000 7ff74be00640 1 --2- 172.21.15.32:0/1166962239 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff74c0d0aa0 0x7ff74c0ca250 secure :-1 s=READY pgs=1344 cs=0 l=1 rev1=1 crypto rx=0x7ff73c02aa90 tx=0x7ff73c033c90 comp rx=0 tx=0).stop 2024-08-03T20:56:21.609 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:21.608+0000 7ff74be00640 1 -- 172.21.15.32:0/1166962239 shutdown_connections 2024-08-03T20:56:21.609 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:21.608+0000 7ff74be00640 1 --2- 172.21.15.32:0/1166962239 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7ff728077010 0x7ff728079520 unknown :-1 s=CLOSED pgs=554 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:21.609 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:21.608+0000 7ff74be00640 1 --2- 172.21.15.32:0/1166962239 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ff74c0d4bb0 0x7ff74c15e390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:21.609 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:21.608+0000 7ff74be00640 1 --2- 172.21.15.32:0/1166962239 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7ff74c0d2290 0x7ff74c15de50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:21.610 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:21.608+0000 7ff74be00640 1 --2- 172.21.15.32:0/1166962239 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff74c0d0aa0 0x7ff74c0ca250 unknown :-1 s=CLOSED pgs=1344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:21.610 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:21.608+0000 7ff74be00640 1 -- 172.21.15.32:0/1166962239 >> 172.21.15.32:0/1166962239 conn(0x7ff74c0d4090 msgr2=0x7ff74c0b0df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:56:21.610 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:21.608+0000 7ff74be00640 1 -- 172.21.15.32:0/1166962239 shutdown_connections 2024-08-03T20:56:21.610 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:21.608+0000 7ff74be00640 1 -- 172.21.15.32:0/1166962239 wait complete. 2024-08-03T20:56:21.630 DEBUG:tasks.ceph_test_case:wait_until_true: success in 0s and 0 retries 2024-08-03T20:56:21.631 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-08-03T20:56:21.858 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:21.857+0000 7fb53c000640 1 -- 172.21.15.32:0/1485877946 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb53415fff0 msgr2=0x7fb534160400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:56:21.858 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:21.857+0000 7fb53c000640 1 --2- 172.21.15.32:0/1485877946 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb53415fff0 0x7fb534160400 secure :-1 s=READY pgs=1345 cs=0 l=1 rev1=1 crypto rx=0x7fb534071700 tx=0x7fb52c02aa30 comp rx=0 tx=0).stop 2024-08-03T20:56:21.859 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:21.857+0000 7fb53c000640 1 -- 172.21.15.32:0/1485877946 shutdown_connections 2024-08-03T20:56:21.859 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:21.857+0000 7fb53c000640 1 --2- 172.21.15.32:0/1485877946 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb53415fff0 0x7fb534160400 unknown :-1 s=CLOSED pgs=1345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:21.859 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:21.857+0000 7fb53c000640 1 --2- 172.21.15.32:0/1485877946 >> v2:172.21.15.118:3300/0 conn(0x7fb53406e930 0x7fb534144f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:21.859 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:21.857+0000 7fb53c000640 1 -- 172.21.15.32:0/1485877946 >> 172.21.15.32:0/1485877946 conn(0x7fb5340b1bd0 msgr2=0x7fb5340b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:56:21.859 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:21.858+0000 7fb53c000640 1 -- 172.21.15.32:0/1485877946 shutdown_connections 2024-08-03T20:56:21.859 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:21.858+0000 7fb53c000640 1 -- 172.21.15.32:0/1485877946 wait complete. 2024-08-03T20:56:21.860 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:21.860+0000 7fb53c000640 1 Processor -- start 2024-08-03T20:56:21.860 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:21.860+0000 7fb53c000640 1 -- start start 2024-08-03T20:56:21.863 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:21.861+0000 7fb53c000640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fb53406e930 0x7fb53415ec10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:56:21.863 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:21.861+0000 7fb53c000640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb534073e60 0x7fb53415f150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:56:21.863 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:21.862+0000 7fb53c000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb53415fff0 0x7fb5340ad8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:56:21.863 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:21.862+0000 7fb53c000640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fb5340cf840 con 0x7fb53415fff0 2024-08-03T20:56:21.863 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:21.862+0000 7fb53c000640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7fb534145f30 con 0x7fb534073e60 2024-08-03T20:56:21.863 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:21.862+0000 7fb53c000640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7fb5340cf9c0 con 0x7fb53406e930 2024-08-03T20:56:21.863 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:21.862+0000 7fb53a000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb53415fff0 0x7fb5340ad8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:56:21.863 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:21.862+0000 7fb53a000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb53415fff0 0x7fb5340ad8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:50786/0 (socket says 172.21.15.32:50786) 2024-08-03T20:56:21.863 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:21.862+0000 7fb53a000640 1 -- 172.21.15.32:0/3331498059 learned_addr learned my addr 172.21.15.32:0/3331498059 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:56:21.863 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:21.862+0000 7fb539600640 1 --2- 172.21.15.32:0/3331498059 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fb53406e930 0x7fb53415ec10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:56:21.864 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:21.862+0000 7fb538c00640 1 --2- 172.21.15.32:0/3331498059 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb534073e60 0x7fb53415f150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:56:21.864 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:21.862+0000 7fb53a000640 1 -- 172.21.15.32:0/3331498059 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fb53406e930 msgr2=0x7fb53415ec10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:56:21.864 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:21.862+0000 7fb53a000640 1 --2- 172.21.15.32:0/3331498059 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fb53406e930 0x7fb53415ec10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:21.864 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:21.862+0000 7fb53a000640 1 -- 172.21.15.32:0/3331498059 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb534073e60 msgr2=0x7fb53415f150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:56:21.864 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:21.862+0000 7fb53a000640 1 --2- 172.21.15.32:0/3331498059 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb534073e60 0x7fb53415f150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:21.864 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:21.862+0000 7fb53a000640 1 -- 172.21.15.32:0/3331498059 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb5341f8890 con 0x7fb53415fff0 2024-08-03T20:56:21.864 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:21.862+0000 7fb538c00640 1 --2- 172.21.15.32:0/3331498059 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb534073e60 0x7fb53415f150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-03T20:56:21.864 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:21.863+0000 7fb53a000640 1 --2- 172.21.15.32:0/3331498059 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb53415fff0 0x7fb5340ad8c0 secure :-1 s=READY pgs=1346 cs=0 l=1 rev1=1 crypto rx=0x7fb52c02b020 tx=0x7fb52c003f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:56:21.865 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:21.863+0000 7fb531600640 1 -- 172.21.15.32:0/3331498059 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb52c033050 con 0x7fb53415fff0 2024-08-03T20:56:21.865 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:21.863+0000 7fb531600640 1 -- 172.21.15.32:0/3331498059 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fb52c004450 con 0x7fb53415fff0 2024-08-03T20:56:21.865 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:21.863+0000 7fb53c000640 1 -- 172.21.15.32:0/3331498059 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb5341f8b70 con 0x7fb53415fff0 2024-08-03T20:56:21.865 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:21.863+0000 7fb531600640 1 -- 172.21.15.32:0/3331498059 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb52c038450 con 0x7fb53415fff0 2024-08-03T20:56:21.865 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:21.863+0000 7fb53c000640 1 -- 172.21.15.32:0/3331498059 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb5340ae7c0 con 0x7fb53415fff0 2024-08-03T20:56:21.865 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:21.864+0000 7fb53c000640 1 -- 172.21.15.32:0/3331498059 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb5340aea20 con 0x7fb53415fff0 2024-08-03T20:56:21.866 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:21.865+0000 7fb531600640 1 -- 172.21.15.32:0/3331498059 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7fb52c036070 con 0x7fb53415fff0 2024-08-03T20:56:21.867 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:21.866+0000 7fb531600640 1 --2- 172.21.15.32:0/3331498059 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fb514077010 0x7fb514079520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:56:21.867 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:21.866+0000 7fb531600640 1 -- 172.21.15.32:0/3331498059 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 6886+0+0 (secure 0 0 0) 0x7fb52c0ba270 con 0x7fb53415fff0 2024-08-03T20:56:21.867 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:21.867+0000 7fb539600640 1 --2- 172.21.15.32:0/3331498059 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fb514077010 0x7fb514079520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:56:21.868 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:21.868+0000 7fb539600640 1 --2- 172.21.15.32:0/3331498059 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fb514077010 0x7fb514079520 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7fb528009b30 tx=0x7fb528005f90 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:56:21.871 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:21.871+0000 7fb531600640 1 -- 172.21.15.32:0/3331498059 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7fb52c082fc0 con 0x7fb53415fff0 2024-08-03T20:56:22.115 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:22.114+0000 7fb53c000640 1 -- 172.21.15.32:0/3331498059 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb534160c50 con 0x7fb53415fff0 2024-08-03T20:56:22.115 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:22.115+0000 7fb531600640 1 -- 172.21.15.32:0/3331498059 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 161 v161) ==== 95+0+4390 (secure 0 0 0) 0x7fb52c08b070 con 0x7fb53415fff0 2024-08-03T20:56:22.115 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-03T20:56:22.116 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":161,"btime":"2024-08-03T20:56:21:089210+0000","default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6439,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6835/531080160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6834","nonce":531080160},{"type":"v1","addr":"172.21.15.118:6835","nonce":531080160}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":84},{"gid":6454,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6837/2796882319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6836","nonce":2796882319},{"type":"v1","addr":"172.21.15.118:6837","nonce":2796882319}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":86},{"gid":6813,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6834/2686933890","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":2686933890},{"type":"v1","addr":"172.21.15.32:6834","nonce":2686933890}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":85}],"filesystems":[{"mdsmap":{"epoch":161,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-08-03T20:54:26.950518+0000","modified":"2024-08-03T20:56:21.089141+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":97,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":6}]} 2024-08-03T20:56:22.116 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 161 2024-08-03T20:56:22.118 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:22.117+0000 7fb53c000640 1 -- 172.21.15.32:0/3331498059 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fb514077010 msgr2=0x7fb514079520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:56:22.118 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:22.117+0000 7fb53c000640 1 --2- 172.21.15.32:0/3331498059 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fb514077010 0x7fb514079520 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7fb528009b30 tx=0x7fb528005f90 comp rx=0 tx=0).stop 2024-08-03T20:56:22.118 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:22.117+0000 7fb53c000640 1 -- 172.21.15.32:0/3331498059 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb53415fff0 msgr2=0x7fb5340ad8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:56:22.118 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:22.117+0000 7fb53c000640 1 --2- 172.21.15.32:0/3331498059 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb53415fff0 0x7fb5340ad8c0 secure :-1 s=READY pgs=1346 cs=0 l=1 rev1=1 crypto rx=0x7fb52c02b020 tx=0x7fb52c003f90 comp rx=0 tx=0).stop 2024-08-03T20:56:22.118 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:22.117+0000 7fb53c000640 1 -- 172.21.15.32:0/3331498059 shutdown_connections 2024-08-03T20:56:22.118 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:22.117+0000 7fb53c000640 1 --2- 172.21.15.32:0/3331498059 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fb514077010 0x7fb514079520 unknown :-1 s=CLOSED pgs=555 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:22.118 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:22.117+0000 7fb53c000640 1 --2- 172.21.15.32:0/3331498059 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb534073e60 0x7fb53415f150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:22.119 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:22.117+0000 7fb53c000640 1 --2- 172.21.15.32:0/3331498059 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb53415fff0 0x7fb5340ad8c0 unknown :-1 s=CLOSED pgs=1346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:22.119 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:22.117+0000 7fb53c000640 1 --2- 172.21.15.32:0/3331498059 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fb53406e930 0x7fb53415ec10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:22.119 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:22.117+0000 7fb53c000640 1 -- 172.21.15.32:0/3331498059 >> 172.21.15.32:0/3331498059 conn(0x7fb5340b1bd0 msgr2=0x7fb5340cc540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:56:22.119 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:22.117+0000 7fb53c000640 1 -- 172.21.15.32:0/3331498059 shutdown_connections 2024-08-03T20:56:22.119 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:22.117+0000 7fb53c000640 1 -- 172.21.15.32:0/3331498059 wait complete. 2024-08-03T20:56:22.141 DEBUG:tasks.ceph_test_case:wait_until_true: success in 0s and 0 retries 2024-08-03T20:56:22.141 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-08-03T20:56:22.368 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:22.367+0000 7f8874c00640 1 -- 172.21.15.32:0/489606439 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f88700b4de0 msgr2=0x7f88701515c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:56:22.368 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:22.367+0000 7f8874c00640 1 --2- 172.21.15.32:0/489606439 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f88700b4de0 0x7f88701515c0 secure :-1 s=READY pgs=1348 cs=0 l=1 rev1=1 crypto rx=0x7f8860005fd0 tx=0x7f886002a540 comp rx=0 tx=0).stop 2024-08-03T20:56:22.368 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:22.367+0000 7f8874c00640 1 -- 172.21.15.32:0/489606439 shutdown_connections 2024-08-03T20:56:22.368 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:22.367+0000 7f8874c00640 1 --2- 172.21.15.32:0/489606439 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f88700733f0 0x7f8870073800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:22.368 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:22.367+0000 7f8874c00640 1 --2- 172.21.15.32:0/489606439 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f88700b4de0 0x7f88701515c0 unknown :-1 s=CLOSED pgs=1348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:22.369 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:22.367+0000 7f8874c00640 1 -- 172.21.15.32:0/489606439 >> 172.21.15.32:0/489606439 conn(0x7f88700b1bd0 msgr2=0x7f88700b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:56:22.369 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:22.367+0000 7f8874c00640 1 -- 172.21.15.32:0/489606439 shutdown_connections 2024-08-03T20:56:22.369 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:22.367+0000 7f8874c00640 1 -- 172.21.15.32:0/489606439 wait complete. 2024-08-03T20:56:22.370 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:22.369+0000 7f8874c00640 1 Processor -- start 2024-08-03T20:56:22.370 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:22.369+0000 7f8874c00640 1 -- start start 2024-08-03T20:56:22.372 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:22.371+0000 7f8874c00640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f88700733f0 0x7f88701483a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:56:22.372 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:22.371+0000 7f8874c00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f88700b4de0 0x7f88701373c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:56:22.372 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:22.371+0000 7f8874c00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f887015e170 0x7f88701595c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:56:22.373 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:22.371+0000 7f8874c00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f887006e930 con 0x7f887015e170 2024-08-03T20:56:22.373 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:22.371+0000 7f8874c00640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f8870070b90 con 0x7f88700b4de0 2024-08-03T20:56:22.373 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:22.371+0000 7f8874c00640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7f887006eab0 con 0x7f88700733f0 2024-08-03T20:56:22.373 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:22.371+0000 7f886e800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f887015e170 0x7f88701595c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:56:22.373 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:22.371+0000 7f886e800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f887015e170 0x7f88701595c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:50794/0 (socket says 172.21.15.32:50794) 2024-08-03T20:56:22.373 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:22.371+0000 7f886e800640 1 -- 172.21.15.32:0/3865729611 learned_addr learned my addr 172.21.15.32:0/3865729611 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:56:22.373 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:22.371+0000 7f886de00640 1 --2- 172.21.15.32:0/3865729611 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f88700733f0 0x7f88701483a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:56:22.373 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:22.371+0000 7f8867e00640 1 --2- 172.21.15.32:0/3865729611 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f88700b4de0 0x7f88701373c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:56:22.373 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:22.372+0000 7f886e800640 1 -- 172.21.15.32:0/3865729611 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f88700733f0 msgr2=0x7f88701483a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:56:22.373 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:22.372+0000 7f886e800640 1 --2- 172.21.15.32:0/3865729611 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f88700733f0 0x7f88701483a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:22.374 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:22.372+0000 7f886e800640 1 -- 172.21.15.32:0/3865729611 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f88700b4de0 msgr2=0x7f88701373c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:56:22.374 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:22.372+0000 7f886e800640 1 --2- 172.21.15.32:0/3865729611 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f88700b4de0 0x7f88701373c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:22.374 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:22.372+0000 7f886e800640 1 -- 172.21.15.32:0/3865729611 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f88701f8f00 con 0x7f887015e170 2024-08-03T20:56:22.375 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:22.372+0000 7f886e800640 1 --2- 172.21.15.32:0/3865729611 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f887015e170 0x7f88701595c0 secure :-1 s=READY pgs=1349 cs=0 l=1 rev1=1 crypto rx=0x7f886800c760 tx=0x7f886800cb60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:56:22.375 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:22.372+0000 7f8866000640 1 -- 172.21.15.32:0/3865729611 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8868003b70 con 0x7f887015e170 2024-08-03T20:56:22.376 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:22.372+0000 7f8866000640 1 -- 172.21.15.32:0/3865729611 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f8868003d10 con 0x7f887015e170 2024-08-03T20:56:22.376 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:22.372+0000 7f8866000640 1 -- 172.21.15.32:0/3865729611 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88680052d0 con 0x7f887015e170 2024-08-03T20:56:22.376 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:22.372+0000 7f8874c00640 1 -- 172.21.15.32:0/3865729611 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f88701f91e0 con 0x7f887015e170 2024-08-03T20:56:22.376 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:22.372+0000 7f8874c00640 1 -- 172.21.15.32:0/3865729611 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f88701f97a0 con 0x7f887015e170 2024-08-03T20:56:22.376 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:22.374+0000 7f8874c00640 1 -- 172.21.15.32:0/3865729611 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8870073800 con 0x7f887015e170 2024-08-03T20:56:22.376 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:22.375+0000 7f8866000640 1 -- 172.21.15.32:0/3865729611 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f8868005470 con 0x7f887015e170 2024-08-03T20:56:22.376 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:22.376+0000 7f8866000640 1 --2- 172.21.15.32:0/3865729611 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f884c077010 0x7f884c079520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:56:22.377 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:22.376+0000 7f8866000640 1 -- 172.21.15.32:0/3865729611 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 6886+0+0 (secure 0 0 0) 0x7f8868098b40 con 0x7f887015e170 2024-08-03T20:56:22.377 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:22.376+0000 7f886de00640 1 --2- 172.21.15.32:0/3865729611 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f884c077010 0x7f884c079520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:56:22.378 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:22.377+0000 7f886de00640 1 --2- 172.21.15.32:0/3865729611 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f884c077010 0x7f884c079520 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7f8860033000 tx=0x7f8860004730 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:56:22.380 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:22.380+0000 7f8866000640 1 -- 172.21.15.32:0/3865729611 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7f88700b51f0 con 0x7f887015e170 2024-08-03T20:56:22.629 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:22.629+0000 7f8874c00640 1 -- 172.21.15.32:0/3865729611 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f88701f8c60 con 0x7f887015e170 2024-08-03T20:56:22.630 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:22.629+0000 7f8866000640 1 -- 172.21.15.32:0/3865729611 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 161 v161) ==== 95+0+4390 (secure 0 0 0) 0x7f88680611b0 con 0x7f887015e170 2024-08-03T20:56:22.630 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-03T20:56:22.630 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":161,"btime":"2024-08-03T20:56:21:089210+0000","default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6439,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6835/531080160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6834","nonce":531080160},{"type":"v1","addr":"172.21.15.118:6835","nonce":531080160}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":84},{"gid":6454,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6837/2796882319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6836","nonce":2796882319},{"type":"v1","addr":"172.21.15.118:6837","nonce":2796882319}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":86},{"gid":6813,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6834/2686933890","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":2686933890},{"type":"v1","addr":"172.21.15.32:6834","nonce":2686933890}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":85}],"filesystems":[{"mdsmap":{"epoch":161,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-08-03T20:54:26.950518+0000","modified":"2024-08-03T20:56:21.089141+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":97,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":6}]} 2024-08-03T20:56:22.630 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 161 2024-08-03T20:56:22.633 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:22.632+0000 7f8874c00640 1 -- 172.21.15.32:0/3865729611 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f884c077010 msgr2=0x7f884c079520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:56:22.633 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:22.632+0000 7f8874c00640 1 --2- 172.21.15.32:0/3865729611 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f884c077010 0x7f884c079520 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7f8860033000 tx=0x7f8860004730 comp rx=0 tx=0).stop 2024-08-03T20:56:22.633 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:22.632+0000 7f8874c00640 1 -- 172.21.15.32:0/3865729611 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f887015e170 msgr2=0x7f88701595c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:56:22.633 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:22.632+0000 7f8874c00640 1 --2- 172.21.15.32:0/3865729611 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f887015e170 0x7f88701595c0 secure :-1 s=READY pgs=1349 cs=0 l=1 rev1=1 crypto rx=0x7f886800c760 tx=0x7f886800cb60 comp rx=0 tx=0).stop 2024-08-03T20:56:22.633 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:22.632+0000 7f8874c00640 1 -- 172.21.15.32:0/3865729611 shutdown_connections 2024-08-03T20:56:22.633 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:22.632+0000 7f8874c00640 1 --2- 172.21.15.32:0/3865729611 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f884c077010 0x7f884c079520 unknown :-1 s=CLOSED pgs=556 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:22.633 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:22.632+0000 7f8874c00640 1 --2- 172.21.15.32:0/3865729611 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f88700b4de0 0x7f88701373c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:22.633 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:22.632+0000 7f8874c00640 1 --2- 172.21.15.32:0/3865729611 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f887015e170 0x7f88701595c0 unknown :-1 s=CLOSED pgs=1349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:22.633 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:22.632+0000 7f8874c00640 1 --2- 172.21.15.32:0/3865729611 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f88700733f0 0x7f88701483a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:22.633 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:22.632+0000 7f8874c00640 1 -- 172.21.15.32:0/3865729611 >> 172.21.15.32:0/3865729611 conn(0x7f88700b1bd0 msgr2=0x7f887015c510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:56:22.633 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:22.632+0000 7f8874c00640 1 -- 172.21.15.32:0/3865729611 shutdown_connections 2024-08-03T20:56:22.633 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:22.632+0000 7f8874c00640 1 -- 172.21.15.32:0/3865729611 wait complete. 2024-08-03T20:56:22.659 INFO:tasks.cephfs.test_damage:Result: Mutation 'Delete mds0_sessionmap' led to DAMAGED state 2024-08-03T20:56:22.659 INFO:tasks.cephfs.test_damage:Applying mutation 'Delete 400.00000000' 2024-08-03T20:56:22.660 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-08-03T20:56:22.660 DEBUG:teuthology.orchestra.run.smithi032:> dd if=/proc/self/mounts of=/dev/stdout 2024-08-03T20:56:22.699 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053453,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6499208k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 none /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249604k,nr_inodes=812401,mode=700,uid=1000,gid=1316 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6499208k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-08-03T20:56:22.699 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi032.front.sepia.ceph.com 2024-08-03T20:56:22.700 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-08-03T20:56:22.700 DEBUG:teuthology.orchestra.run.smithi032:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-08-03T20:56:22.773 INFO:teuthology.orchestra.run.smithi032.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.0': No such file or directory 2024-08-03T20:56:22.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-03T20:56:22.775 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-08-03T20:56:23.044 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:23.043+0000 7f098ee00640 1 -- 172.21.15.32:0/636634249 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f098806f440 msgr2=0x7f098806f850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:56:23.044 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:23.043+0000 7f098ee00640 1 --2- 172.21.15.32:0/636634249 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f098806f440 0x7f098806f850 secure :-1 s=READY pgs=1350 cs=0 l=1 rev1=1 crypto rx=0x7f09880720a0 tx=0x7f098002a980 comp rx=0 tx=0).stop 2024-08-03T20:56:23.044 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:23.043+0000 7f098ee00640 1 -- 172.21.15.32:0/636634249 shutdown_connections 2024-08-03T20:56:23.044 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:23.043+0000 7f098ee00640 1 --2- 172.21.15.32:0/636634249 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f098806f440 0x7f098806f850 unknown :-1 s=CLOSED pgs=1350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:23.044 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:23.043+0000 7f098ee00640 1 --2- 172.21.15.32:0/636634249 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f09880acd80 0x7f09880ad190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:23.044 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:23.043+0000 7f098ee00640 1 --2- 172.21.15.32:0/636634249 >> v2:172.21.15.118:3300/0 conn(0x7f09880af020 0x7f09880742b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:23.044 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:23.043+0000 7f098ee00640 1 -- 172.21.15.32:0/636634249 >> 172.21.15.32:0/636634249 conn(0x7f09880b2120 msgr2=0x7f09880cfd60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:56:23.044 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:23.043+0000 7f098ee00640 1 -- 172.21.15.32:0/636634249 shutdown_connections 2024-08-03T20:56:23.044 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:23.043+0000 7f098ee00640 1 -- 172.21.15.32:0/636634249 wait complete. 2024-08-03T20:56:23.045 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:23.045+0000 7f098ee00640 1 Processor -- start 2024-08-03T20:56:23.045 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:23.045+0000 7f098ee00640 1 -- start start 2024-08-03T20:56:23.048 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:23.047+0000 7f098ee00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f098806f440 0x7f09880c4290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:56:23.048 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:23.047+0000 7f098ee00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f09880acd80 0x7f098814d1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:56:23.048 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:23.047+0000 7f098ee00640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f09880af020 0x7f09881f35a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:56:23.048 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:23.047+0000 7f098ee00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f09881601e0 con 0x7f098806f440 2024-08-03T20:56:23.048 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:23.047+0000 7f098ee00640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f0988160060 con 0x7f09880acd80 2024-08-03T20:56:23.048 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:23.047+0000 7f098ee00640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7f0988160360 con 0x7f09880af020 2024-08-03T20:56:23.048 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:23.047+0000 7f098da00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f098806f440 0x7f09880c4290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:56:23.048 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:23.047+0000 7f098da00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f098806f440 0x7f09880c4290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:50806/0 (socket says 172.21.15.32:50806) 2024-08-03T20:56:23.048 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:23.047+0000 7f098da00640 1 -- 172.21.15.32:0/2223912833 learned_addr learned my addr 172.21.15.32:0/2223912833 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:56:23.049 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:23.047+0000 7f098e400640 1 --2- 172.21.15.32:0/2223912833 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f09880af020 0x7f09881f35a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:56:23.049 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:23.047+0000 7f098d000640 1 --2- 172.21.15.32:0/2223912833 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f09880acd80 0x7f098814d1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:56:23.049 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:23.047+0000 7f098da00640 1 -- 172.21.15.32:0/2223912833 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f09880af020 msgr2=0x7f09881f35a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:56:23.049 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:23.047+0000 7f098da00640 1 --2- 172.21.15.32:0/2223912833 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f09880af020 0x7f09881f35a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:23.049 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:23.047+0000 7f098da00640 1 -- 172.21.15.32:0/2223912833 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f09880acd80 msgr2=0x7f098814d1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:56:23.049 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:23.047+0000 7f098da00640 1 --2- 172.21.15.32:0/2223912833 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f09880acd80 0x7f098814d1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:23.049 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:23.047+0000 7f098da00640 1 -- 172.21.15.32:0/2223912833 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f09880b4ff0 con 0x7f098806f440 2024-08-03T20:56:23.050 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:23.048+0000 7f098da00640 1 --2- 172.21.15.32:0/2223912833 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f098806f440 0x7f09880c4290 secure :-1 s=READY pgs=1351 cs=0 l=1 rev1=1 crypto rx=0x7f097c00c970 tx=0x7f097c00ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:56:23.050 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:23.048+0000 7f0985600640 1 -- 172.21.15.32:0/2223912833 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f097c007bb0 con 0x7f098806f440 2024-08-03T20:56:23.050 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:23.048+0000 7f0985600640 1 -- 172.21.15.32:0/2223912833 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f097c007e40 con 0x7f098806f440 2024-08-03T20:56:23.050 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:23.048+0000 7f098ee00640 1 -- 172.21.15.32:0/2223912833 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f09880c34d0 con 0x7f098806f440 2024-08-03T20:56:23.050 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:23.048+0000 7f0985600640 1 -- 172.21.15.32:0/2223912833 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f097c00f690 con 0x7f098806f440 2024-08-03T20:56:23.051 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:23.048+0000 7f098ee00640 1 -- 172.21.15.32:0/2223912833 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f09880c3a90 con 0x7f098806f440 2024-08-03T20:56:23.051 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:23.049+0000 7f098ee00640 1 -- 172.21.15.32:0/2223912833 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f09880ad190 con 0x7f098806f440 2024-08-03T20:56:23.051 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:23.050+0000 7f0985600640 1 -- 172.21.15.32:0/2223912833 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f097c00f830 con 0x7f098806f440 2024-08-03T20:56:23.052 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:23.051+0000 7f0985600640 1 --2- 172.21.15.32:0/2223912833 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f0968077010 0x7f0968079520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:56:23.052 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:23.051+0000 7f0985600640 1 -- 172.21.15.32:0/2223912833 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 6886+0+0 (secure 0 0 0) 0x7f097c0994d0 con 0x7f098806f440 2024-08-03T20:56:23.052 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:23.052+0000 7f098d000640 1 --2- 172.21.15.32:0/2223912833 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f0968077010 0x7f0968079520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:56:23.053 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:23.053+0000 7f098d000640 1 --2- 172.21.15.32:0/2223912833 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f0968077010 0x7f0968079520 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7f09880b4180 tx=0x7f097400b260 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:56:23.056 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:23.056+0000 7f0985600640 1 -- 172.21.15.32:0/2223912833 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7f097c014030 con 0x7f098806f440 2024-08-03T20:56:23.300 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:23.300+0000 7f098ee00640 1 -- 172.21.15.32:0/2223912833 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7f098815c340 con 0x7f098806f440 2024-08-03T20:56:24.085 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:24.085+0000 7f0985600640 1 -- 172.21.15.32:0/2223912833 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v162) ==== 161+0+0 (secure 0 0 0) 0x7f097c061b40 con 0x7f098806f440 2024-08-03T20:56:24.085 INFO:teuthology.orchestra.run.smithi032.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-08-03T20:56:24.089 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:24.088+0000 7f098ee00640 1 -- 172.21.15.32:0/2223912833 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f0968077010 msgr2=0x7f0968079520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:56:24.089 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:24.088+0000 7f098ee00640 1 --2- 172.21.15.32:0/2223912833 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f0968077010 0x7f0968079520 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7f09880b4180 tx=0x7f097400b260 comp rx=0 tx=0).stop 2024-08-03T20:56:24.089 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:24.088+0000 7f098ee00640 1 -- 172.21.15.32:0/2223912833 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f098806f440 msgr2=0x7f09880c4290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:56:24.089 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:24.088+0000 7f098ee00640 1 --2- 172.21.15.32:0/2223912833 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f098806f440 0x7f09880c4290 secure :-1 s=READY pgs=1351 cs=0 l=1 rev1=1 crypto rx=0x7f097c00c970 tx=0x7f097c00ce30 comp rx=0 tx=0).stop 2024-08-03T20:56:24.089 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:24.088+0000 7f098ee00640 1 -- 172.21.15.32:0/2223912833 shutdown_connections 2024-08-03T20:56:24.089 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:24.088+0000 7f098ee00640 1 --2- 172.21.15.32:0/2223912833 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f0968077010 0x7f0968079520 unknown :-1 s=CLOSED pgs=557 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:24.089 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:24.088+0000 7f098ee00640 1 --2- 172.21.15.32:0/2223912833 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f09880acd80 0x7f098814d1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:24.089 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:24.088+0000 7f098ee00640 1 --2- 172.21.15.32:0/2223912833 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f09880af020 0x7f09881f35a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:24.090 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:24.088+0000 7f098ee00640 1 --2- 172.21.15.32:0/2223912833 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f098806f440 0x7f09880c4290 unknown :-1 s=CLOSED pgs=1351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:24.090 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:24.088+0000 7f098ee00640 1 -- 172.21.15.32:0/2223912833 >> 172.21.15.32:0/2223912833 conn(0x7f09880b2120 msgr2=0x7f09880cba10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:56:24.090 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:24.088+0000 7f098ee00640 1 -- 172.21.15.32:0/2223912833 shutdown_connections 2024-08-03T20:56:24.090 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:24.088+0000 7f098ee00640 1 -- 172.21.15.32:0/2223912833 wait complete. 2024-08-03T20:56:24.114 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mds repaired 0 2024-08-03T20:56:24.347 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:24.346+0000 7fb567000640 1 -- 172.21.15.32:0/593544796 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb560155970 msgr2=0x7fb56015c2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:56:24.347 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:24.346+0000 7fb567000640 1 --2- 172.21.15.32:0/593544796 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb560155970 0x7fb56015c2c0 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7fb550009830 tx=0x7fb55002a510 comp rx=0 tx=0).stop 2024-08-03T20:56:24.347 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:24.346+0000 7fb567000640 1 -- 172.21.15.32:0/593544796 shutdown_connections 2024-08-03T20:56:24.347 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:24.346+0000 7fb567000640 1 --2- 172.21.15.32:0/593544796 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fb560073ff0 0x7fb56015f290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:24.347 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:24.346+0000 7fb567000640 1 --2- 172.21.15.32:0/593544796 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb560155970 0x7fb56015c2c0 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:24.347 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:24.346+0000 7fb567000640 1 -- 172.21.15.32:0/593544796 >> 172.21.15.32:0/593544796 conn(0x7fb5600b34b0 msgr2=0x7fb5600d1050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:56:24.348 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:24.346+0000 7fb567000640 1 -- 172.21.15.32:0/593544796 shutdown_connections 2024-08-03T20:56:24.348 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:24.346+0000 7fb567000640 1 -- 172.21.15.32:0/593544796 wait complete. 2024-08-03T20:56:24.349 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:24.348+0000 7fb567000640 1 Processor -- start 2024-08-03T20:56:24.349 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:24.349+0000 7fb567000640 1 -- start start 2024-08-03T20:56:24.352 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:24.350+0000 7fb567000640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb560155970 0x7fb5600ccae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:56:24.352 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:24.350+0000 7fb567000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb56015ee80 0x7fb560147c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:56:24.352 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:24.350+0000 7fb567000640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fb5600caaa0 0x7fb5600caeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:56:24.352 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:24.350+0000 7fb567000640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fb5600b4350 con 0x7fb56015ee80 2024-08-03T20:56:24.352 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:24.350+0000 7fb567000640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7fb5600b41d0 con 0x7fb560155970 2024-08-03T20:56:24.352 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:24.350+0000 7fb567000640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7fb5600b44d0 con 0x7fb5600caaa0 2024-08-03T20:56:24.352 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:24.350+0000 7fb55f400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb56015ee80 0x7fb560147c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:56:24.352 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:24.350+0000 7fb55f400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb56015ee80 0x7fb560147c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:50818/0 (socket says 172.21.15.32:50818) 2024-08-03T20:56:24.352 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:24.350+0000 7fb55f400640 1 -- 172.21.15.32:0/39336265 learned_addr learned my addr 172.21.15.32:0/39336265 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:56:24.353 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:24.350+0000 7fb565000640 1 --2- 172.21.15.32:0/39336265 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fb5600caaa0 0x7fb5600caeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:56:24.353 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:24.350+0000 7fb55fe00640 1 --2- 172.21.15.32:0/39336265 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb560155970 0x7fb5600ccae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:56:24.353 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:24.351+0000 7fb55f400640 1 -- 172.21.15.32:0/39336265 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fb5600caaa0 msgr2=0x7fb5600caeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:56:24.353 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:24.351+0000 7fb55f400640 1 --2- 172.21.15.32:0/39336265 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fb5600caaa0 0x7fb5600caeb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:24.353 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:24.351+0000 7fb55f400640 1 -- 172.21.15.32:0/39336265 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb560155970 msgr2=0x7fb5600ccae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:56:24.353 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:24.351+0000 7fb55f400640 1 --2- 172.21.15.32:0/39336265 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb560155970 0x7fb5600ccae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:24.353 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:24.351+0000 7fb55f400640 1 -- 172.21.15.32:0/39336265 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb5600cb3f0 con 0x7fb56015ee80 2024-08-03T20:56:24.353 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:24.351+0000 7fb55f400640 1 --2- 172.21.15.32:0/39336265 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb56015ee80 0x7fb560147c50 secure :-1 s=READY pgs=1353 cs=0 l=1 rev1=1 crypto rx=0x7fb54c00c460 tx=0x7fb54c00c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:56:24.353 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:24.351+0000 7fb557e00640 1 -- 172.21.15.32:0/39336265 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb54c016020 con 0x7fb56015ee80 2024-08-03T20:56:24.353 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:24.351+0000 7fb557e00640 1 -- 172.21.15.32:0/39336265 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fb54c012940 con 0x7fb56015ee80 2024-08-03T20:56:24.353 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:24.351+0000 7fb557e00640 1 -- 172.21.15.32:0/39336265 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb54c012d90 con 0x7fb56015ee80 2024-08-03T20:56:24.353 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:24.351+0000 7fb567000640 1 -- 172.21.15.32:0/39336265 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb5600b1dd0 con 0x7fb56015ee80 2024-08-03T20:56:24.354 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:24.351+0000 7fb567000640 1 -- 172.21.15.32:0/39336265 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb5600b2520 con 0x7fb56015ee80 2024-08-03T20:56:24.354 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:24.352+0000 7fb567000640 1 -- 172.21.15.32:0/39336265 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb560151270 con 0x7fb56015ee80 2024-08-03T20:56:24.354 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:24.354+0000 7fb557e00640 1 -- 172.21.15.32:0/39336265 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7fb54c019020 con 0x7fb56015ee80 2024-08-03T20:56:24.355 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:24.354+0000 7fb557e00640 1 --2- 172.21.15.32:0/39336265 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fb540077010 0x7fb540079520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:56:24.355 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:24.355+0000 7fb557e00640 1 -- 172.21.15.32:0/39336265 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 6886+0+0 (secure 0 0 0) 0x7fb54c098af0 con 0x7fb56015ee80 2024-08-03T20:56:24.355 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:24.355+0000 7fb55fe00640 1 --2- 172.21.15.32:0/39336265 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fb540077010 0x7fb540079520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:56:24.356 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:24.356+0000 7fb55fe00640 1 --2- 172.21.15.32:0/39336265 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fb540077010 0x7fb540079520 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7fb550034000 tx=0x7fb550004730 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:56:24.359 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:24.359+0000 7fb557e00640 1 -- 172.21.15.32:0/39336265 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7fb54c020030 con 0x7fb56015ee80 2024-08-03T20:56:24.644 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:24.644+0000 7fb567000640 1 -- 172.21.15.32:0/39336265 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "mds repaired", "role": "0"} v 0) -- 0x7fb5600b1fa0 con 0x7fb56015ee80 2024-08-03T20:56:25.096 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.095+0000 7fb557e00640 1 -- 172.21.15.32:0/39336265 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "mds repaired", "role": "0"}]=0 repaired: restoring rank 6:0 v163) ==== 101+0+0 (secure 0 0 0) 0x7fb5600cb3f0 con 0x7fb56015ee80 2024-08-03T20:56:25.096 INFO:teuthology.orchestra.run.smithi032.stderr:repaired: restoring rank 6:0 2024-08-03T20:56:25.100 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.099+0000 7fb567000640 1 -- 172.21.15.32:0/39336265 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fb540077010 msgr2=0x7fb540079520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:56:25.100 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.099+0000 7fb567000640 1 --2- 172.21.15.32:0/39336265 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fb540077010 0x7fb540079520 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7fb550034000 tx=0x7fb550004730 comp rx=0 tx=0).stop 2024-08-03T20:56:25.100 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.099+0000 7fb567000640 1 -- 172.21.15.32:0/39336265 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb56015ee80 msgr2=0x7fb560147c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:56:25.100 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.099+0000 7fb567000640 1 --2- 172.21.15.32:0/39336265 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb56015ee80 0x7fb560147c50 secure :-1 s=READY pgs=1353 cs=0 l=1 rev1=1 crypto rx=0x7fb54c00c460 tx=0x7fb54c00c920 comp rx=0 tx=0).stop 2024-08-03T20:56:25.100 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.099+0000 7fb567000640 1 -- 172.21.15.32:0/39336265 shutdown_connections 2024-08-03T20:56:25.100 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.099+0000 7fb567000640 1 --2- 172.21.15.32:0/39336265 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fb540077010 0x7fb540079520 unknown :-1 s=CLOSED pgs=558 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:25.100 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.099+0000 7fb567000640 1 --2- 172.21.15.32:0/39336265 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fb5600caaa0 0x7fb5600caeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:25.100 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.099+0000 7fb567000640 1 --2- 172.21.15.32:0/39336265 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb56015ee80 0x7fb560147c50 unknown :-1 s=CLOSED pgs=1353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:25.100 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.099+0000 7fb567000640 1 --2- 172.21.15.32:0/39336265 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb560155970 0x7fb5600ccae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:25.101 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.099+0000 7fb567000640 1 -- 172.21.15.32:0/39336265 >> 172.21.15.32:0/39336265 conn(0x7fb5600b34b0 msgr2=0x7fb5601485e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:56:25.101 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.099+0000 7fb567000640 1 -- 172.21.15.32:0/39336265 shutdown_connections 2024-08-03T20:56:25.101 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.099+0000 7fb567000640 1 -- 172.21.15.32:0/39336265 wait complete. 2024-08-03T20:56:25.125 DEBUG:teuthology.orchestra.run.smithi032:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata import - 2024-08-03T20:56:25.270 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.269+0000 7f30a8487f40 1 Processor -- start 2024-08-03T20:56:25.270 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.269+0000 7f30a8487f40 1 -- start start 2024-08-03T20:56:25.272 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.271+0000 7f30a8487f40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x564aaba9e360 0x564aababcf80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:56:25.272 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.271+0000 7f30a8487f40 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x564aaba9fea0 0x564aababd4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:56:25.272 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.271+0000 7f30a8487f40 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x564aabab5330 0x564aabab8c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:56:25.272 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.271+0000 7f30a8487f40 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x564aab935e30 con 0x564aaba9fea0 2024-08-03T20:56:25.272 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.271+0000 7f30a8487f40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x564aab993bf0 con 0x564aaba9e360 2024-08-03T20:56:25.273 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.271+0000 7f30a8487f40 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x564aaba9f440 con 0x564aabab5330 2024-08-03T20:56:25.273 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.271+0000 7f30a0c00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x564aaba9fea0 0x564aababd4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:56:25.273 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.271+0000 7f30a0c00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x564aaba9fea0 0x564aababd4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:50832/0 (socket says 172.21.15.32:50832) 2024-08-03T20:56:25.273 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.271+0000 7f30a0c00640 1 -- 172.21.15.32:0/1438356139 learned_addr learned my addr 172.21.15.32:0/1438356139 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:56:25.273 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.271+0000 7f30a1600640 1 --2- 172.21.15.32:0/1438356139 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x564aaba9e360 0x564aababcf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:56:25.273 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.271+0000 7f30a2000640 1 --2- 172.21.15.32:0/1438356139 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x564aabab5330 0x564aabab8c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:56:25.273 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.272+0000 7f30a0c00640 1 -- 172.21.15.32:0/1438356139 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x564aabab5330 msgr2=0x564aabab8c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:56:25.273 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.272+0000 7f30a0c00640 1 --2- 172.21.15.32:0/1438356139 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x564aabab5330 0x564aabab8c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:25.273 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.272+0000 7f30a0c00640 1 -- 172.21.15.32:0/1438356139 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x564aaba9e360 msgr2=0x564aababcf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:56:25.273 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.272+0000 7f30a0c00640 1 --2- 172.21.15.32:0/1438356139 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x564aaba9e360 0x564aababcf80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:25.273 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.272+0000 7f30a0c00640 1 -- 172.21.15.32:0/1438356139 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564aababa0a0 con 0x564aaba9fea0 2024-08-03T20:56:25.273 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.272+0000 7f30a1600640 1 --2- 172.21.15.32:0/1438356139 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x564aaba9e360 0x564aababcf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-03T20:56:25.274 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.272+0000 7f30a0c00640 1 --2- 172.21.15.32:0/1438356139 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x564aaba9fea0 0x564aababd4c0 secure :-1 s=READY pgs=1356 cs=0 l=1 rev1=1 crypto rx=0x7f3090025dc0 tx=0x7f3090002fb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=18a663e9c539fa94 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:56:25.275 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.272+0000 7f309b400640 1 -- 172.21.15.32:0/1438356139 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f309004dcb0 con 0x564aaba9fea0 2024-08-03T20:56:25.275 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.272+0000 7f309b400640 1 -- 172.21.15.32:0/1438356139 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f3090056470 con 0x564aaba9fea0 2024-08-03T20:56:25.275 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.272+0000 7f309b400640 1 -- 172.21.15.32:0/1438356139 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30900568c0 con 0x564aaba9fea0 2024-08-03T20:56:25.275 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.273+0000 7f30a8487f40 1 -- 172.21.15.32:0/1438356139 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x564aaba9fea0 msgr2=0x564aababd4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:56:25.275 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.273+0000 7f30a8487f40 1 --2- 172.21.15.32:0/1438356139 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x564aaba9fea0 0x564aababd4c0 secure :-1 s=READY pgs=1356 cs=0 l=1 rev1=1 crypto rx=0x7f3090025dc0 tx=0x7f3090002fb0 comp rx=0 tx=0).stop 2024-08-03T20:56:25.275 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.273+0000 7f30a8487f40 1 -- 172.21.15.32:0/1438356139 shutdown_connections 2024-08-03T20:56:25.275 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.273+0000 7f30a8487f40 1 --2- 172.21.15.32:0/1438356139 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x564aabab5330 0x564aabab8c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:25.275 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.273+0000 7f30a8487f40 1 --2- 172.21.15.32:0/1438356139 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x564aaba9fea0 0x564aababd4c0 unknown :-1 s=CLOSED pgs=1356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:25.275 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.273+0000 7f30a8487f40 1 --2- 172.21.15.32:0/1438356139 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x564aaba9e360 0x564aababcf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:25.276 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.273+0000 7f30a8487f40 1 -- 172.21.15.32:0/1438356139 >> 172.21.15.32:0/1438356139 conn(0x564aab907ac0 msgr2=0x564aabab7e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:56:25.276 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.273+0000 7f30a8487f40 1 -- 172.21.15.32:0/1438356139 shutdown_connections 2024-08-03T20:56:25.276 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.273+0000 7f30a8487f40 1 -- 172.21.15.32:0/1438356139 wait complete. 2024-08-03T20:56:25.276 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.274+0000 7f30a8487f40 1 Processor -- start 2024-08-03T20:56:25.276 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.274+0000 7f30a8487f40 1 -- start start 2024-08-03T20:56:25.278 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.276+0000 7f30a8487f40 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x564aaba9e360 0x564aaba99a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:56:25.278 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.276+0000 7f30a8487f40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x564aaba9fea0 0x564aaba2f1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:56:25.278 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.276+0000 7f30a8487f40 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x564aabab5330 0x564aaba2f710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:56:25.278 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.276+0000 7f30a8487f40 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x564aabab7360 con 0x564aaba9e360 2024-08-03T20:56:25.278 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.276+0000 7f30a8487f40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x564aabab7640 con 0x564aaba9fea0 2024-08-03T20:56:25.278 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.276+0000 7f30a8487f40 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x564aab9e7130 con 0x564aabab5330 2024-08-03T20:56:25.278 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.276+0000 7f30a1600640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x564aaba9e360 0x564aaba99a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:56:25.278 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.276+0000 7f30a1600640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x564aaba9e360 0x564aaba99a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:50848/0 (socket says 172.21.15.32:50848) 2024-08-03T20:56:25.278 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.276+0000 7f30a1600640 1 -- 172.21.15.32:0/706711870 learned_addr learned my addr 172.21.15.32:0/706711870 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:56:25.278 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.276+0000 7f30a0c00640 1 --2- 172.21.15.32:0/706711870 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x564aaba9fea0 0x564aaba2f1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:56:25.278 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.276+0000 7f30a2000640 1 --2- 172.21.15.32:0/706711870 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x564aabab5330 0x564aaba2f710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:56:25.278 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.276+0000 7f30a1600640 1 -- 172.21.15.32:0/706711870 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x564aabab5330 msgr2=0x564aaba2f710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:56:25.279 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.276+0000 7f30a1600640 1 --2- 172.21.15.32:0/706711870 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x564aabab5330 0x564aaba2f710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:25.279 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.276+0000 7f30a1600640 1 -- 172.21.15.32:0/706711870 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x564aaba9fea0 msgr2=0x564aaba2f1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:56:25.279 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.276+0000 7f30a1600640 1 --2- 172.21.15.32:0/706711870 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x564aaba9fea0 0x564aaba2f1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:25.279 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.276+0000 7f30a1600640 1 -- 172.21.15.32:0/706711870 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564aaba260c0 con 0x564aaba9e360 2024-08-03T20:56:25.279 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.277+0000 7f30a1600640 1 --2- 172.21.15.32:0/706711870 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x564aaba9e360 0x564aaba99a20 secure :-1 s=READY pgs=1357 cs=0 l=1 rev1=1 crypto rx=0x7f309400dd30 tx=0x7f309400b590 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:56:25.279 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.277+0000 7f309a000640 1 -- 172.21.15.32:0/706711870 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3094015020 con 0x564aaba9e360 2024-08-03T20:56:25.279 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.277+0000 7f309a000640 1 -- 172.21.15.32:0/706711870 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f3094011750 con 0x564aaba9e360 2024-08-03T20:56:25.279 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.277+0000 7f30a8487f40 1 -- 172.21.15.32:0/706711870 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x564aaba263a0 con 0x564aaba9e360 2024-08-03T20:56:25.279 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.277+0000 7f309a000640 1 -- 172.21.15.32:0/706711870 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3094011b80 con 0x564aaba9e360 2024-08-03T20:56:25.279 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.277+0000 7f30a8487f40 1 -- 172.21.15.32:0/706711870 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564aaba26960 con 0x564aaba9e360 2024-08-03T20:56:25.280 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.279+0000 7f309a000640 1 -- 172.21.15.32:0/706711870 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f3094018020 con 0x564aaba9e360 2024-08-03T20:56:25.280 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.280+0000 7f309a000640 1 --2- 172.21.15.32:0/706711870 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f30840759f0 0x7f3084077f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:56:25.280 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.280+0000 7f309a000640 1 -- 172.21.15.32:0/706711870 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 6886+0+0 (secure 0 0 0) 0x7f3094097ff0 con 0x564aaba9e360 2024-08-03T20:56:25.281 INFO:teuthology.orchestra.run.smithi032.stdout:Importing pool 2024-08-03T20:56:25.282 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.280+0000 7f30a0c00640 1 --2- 172.21.15.32:0/706711870 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f30840759f0 0x7f3084077f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:56:25.282 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.281+0000 7f30a8487f40 1 --2- 172.21.15.32:0/706711870 >> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] conn(0x564aab9e6340 0x564aab9e6750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:56:25.282 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.281+0000 7f30a8487f40 1 -- 172.21.15.32:0/706711870 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:1 12.10 12:09401a81:::601.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97) -- 0x564aabb60870 con 0x564aab9e6340 2024-08-03T20:56:25.282 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.281+0000 7f30a2000640 1 --2- 172.21.15.32:0/706711870 >> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] conn(0x564aab9e6340 0x564aab9e6750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:56:25.282 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.281+0000 7f30a0c00640 1 --2- 172.21.15.32:0/706711870 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f30840759f0 0x7f3084077f00 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7f3090062000 tx=0x7f3090004880 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:56:25.282 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.281+0000 7f30a2000640 1 --2- 172.21.15.32:0/706711870 >> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] conn(0x564aab9e6340 0x564aab9e6750 crc :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:56:25.284 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.284+0000 7f30a2000640 1 -- 172.21.15.32:0/706711870 <== osd.0 v2:172.21.15.32:6816/3066156116 1 ==== osd_op_reply(1 601.00000000 [create] v97'350 uv324 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f309c00b040 con 0x564aab9e6340 2024-08-03T20:56:25.284 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.284+0000 7f30a8487f40 1 -- 172.21.15.32:0/706711870 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:2 12.10 12:09401a81:::601.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e97) -- 0x564aabb60c10 con 0x564aab9e6340 2024-08-03T20:56:25.286 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.285+0000 7f30a2000640 1 -- 172.21.15.32:0/706711870 <== osd.0 v2:172.21.15.32:6816/3066156116 2 ==== osd_op_reply(2 601.00000000 [delete] v97'351 uv351 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f309c00b040 con 0x564aab9e6340 2024-08-03T20:56:25.286 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.285+0000 7f30a8487f40 1 -- 172.21.15.32:0/706711870 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:3 12.10 12:09401a81:::601.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97) -- 0x564aabb60fb0 con 0x564aab9e6340 2024-08-03T20:56:25.287 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::601.00000000:head# 2024-08-03T20:56:25.287 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.287+0000 7f30a2000640 1 -- 172.21.15.32:0/706711870 <== osd.0 v2:172.21.15.32:6816/3066156116 3 ==== osd_op_reply(3 601.00000000 [create] v97'352 uv352 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f309c00b040 con 0x564aab9e6340 2024-08-03T20:56:25.288 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.287+0000 7f30a8487f40 1 -- 172.21.15.32:0/706711870 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:4 12.10 12:09401a81:::601.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97) -- 0x564aabb62360 con 0x564aab9e6340 2024-08-03T20:56:25.289 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.289+0000 7f30a2000640 1 -- 172.21.15.32:0/706711870 <== osd.0 v2:172.21.15.32:6816/3066156116 4 ==== osd_op_reply(4 601.00000000 [setxattr (30)] v97'353 uv353 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f309c00b040 con 0x564aab9e6340 2024-08-03T20:56:25.289 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.289+0000 7f30a8487f40 1 -- 172.21.15.32:0/706711870 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:5 12.10 12:09401a81:::601.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97) -- 0x564aaba28c40 con 0x564aab9e6340 2024-08-03T20:56:25.290 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.290+0000 7f30a2000640 1 -- 172.21.15.32:0/706711870 <== osd.0 v2:172.21.15.32:6816/3066156116 5 ==== osd_op_reply(5 601.00000000 [setxattr (62)] v97'354 uv354 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f309c00b040 con 0x564aab9e6340 2024-08-03T20:56:25.291 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.290+0000 7f30a8487f40 1 -- 172.21.15.32:0/706711870 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:6 12.10 12:09401a81:::601.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97) -- 0x564aabb61350 con 0x564aab9e6340 2024-08-03T20:56:25.292 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.292+0000 7f30a2000640 1 -- 172.21.15.32:0/706711870 <== osd.0 v2:172.21.15.32:6816/3066156116 6 ==== osd_op_reply(6 601.00000000 [omap-set-header] v97'355 uv355 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f309c00b040 con 0x564aab9e6340 2024-08-03T20:56:25.292 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.292+0000 7f30a8487f40 1 -- 172.21.15.32:0/706711870 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:7 12.10 12:0bd6d154:::602.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97) -- 0x564aabb61740 con 0x564aab9e6340 2024-08-03T20:56:25.294 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.293+0000 7f30a2000640 1 -- 172.21.15.32:0/706711870 <== osd.0 v2:172.21.15.32:6816/3066156116 7 ==== osd_op_reply(7 602.00000000 [create] v97'356 uv330 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f309c00b040 con 0x564aab9e6340 2024-08-03T20:56:25.294 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.294+0000 7f30a8487f40 1 -- 172.21.15.32:0/706711870 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:8 12.10 12:0bd6d154:::602.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e97) -- 0x564aabb61b30 con 0x564aab9e6340 2024-08-03T20:56:25.296 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.295+0000 7f30a2000640 1 -- 172.21.15.32:0/706711870 <== osd.0 v2:172.21.15.32:6816/3066156116 8 ==== osd_op_reply(8 602.00000000 [delete] v97'357 uv357 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f309c00b040 con 0x564aab9e6340 2024-08-03T20:56:25.296 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.295+0000 7f30a8487f40 1 -- 172.21.15.32:0/706711870 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:9 12.10 12:0bd6d154:::602.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97) -- 0x564aabb61b30 con 0x564aab9e6340 2024-08-03T20:56:25.297 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::602.00000000:head# 2024-08-03T20:56:25.297 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.297+0000 7f30a2000640 1 -- 172.21.15.32:0/706711870 <== osd.0 v2:172.21.15.32:6816/3066156116 9 ==== osd_op_reply(9 602.00000000 [create] v97'358 uv358 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f309c00b040 con 0x564aab9e6340 2024-08-03T20:56:25.297 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.297+0000 7f30a8487f40 1 -- 172.21.15.32:0/706711870 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:10 12.10 12:0bd6d154:::602.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97) -- 0x564aabb61ae0 con 0x564aab9e6340 2024-08-03T20:56:25.299 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.299+0000 7f30a2000640 1 -- 172.21.15.32:0/706711870 <== osd.0 v2:172.21.15.32:6816/3066156116 10 ==== osd_op_reply(10 602.00000000 [setxattr (30)] v97'359 uv359 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f309c00b040 con 0x564aab9e6340 2024-08-03T20:56:25.299 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.299+0000 7f30a8487f40 1 -- 172.21.15.32:0/706711870 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:11 12.10 12:0bd6d154:::602.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97) -- 0x564aabb61ae0 con 0x564aab9e6340 2024-08-03T20:56:25.300 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.300+0000 7f30a2000640 1 -- 172.21.15.32:0/706711870 <== osd.0 v2:172.21.15.32:6816/3066156116 11 ==== osd_op_reply(11 602.00000000 [setxattr (62)] v97'360 uv360 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f309c00b040 con 0x564aab9e6340 2024-08-03T20:56:25.301 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.300+0000 7f30a8487f40 1 -- 172.21.15.32:0/706711870 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:12 12.10 12:0bd6d154:::602.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97) -- 0x564aabb61ae0 con 0x564aab9e6340 2024-08-03T20:56:25.302 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.302+0000 7f30a2000640 1 -- 172.21.15.32:0/706711870 <== osd.0 v2:172.21.15.32:6816/3066156116 12 ==== osd_op_reply(12 602.00000000 [omap-set-header] v97'361 uv361 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f309c00b040 con 0x564aab9e6340 2024-08-03T20:56:25.302 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.302+0000 7f30a8487f40 1 -- 172.21.15.32:0/706711870 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:13 12.10 12:0cf6ad0f:::10000000000.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97) -- 0x564aabb61ae0 con 0x564aab9e6340 2024-08-03T20:56:25.304 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.303+0000 7f30a2000640 1 -- 172.21.15.32:0/706711870 <== osd.0 v2:172.21.15.32:6816/3066156116 13 ==== osd_op_reply(13 10000000000.00000000 [create] v97'362 uv337 ondisk = -17 ((17) File exists)) ==== 164+0+0 (crc 0 0 0) 0x7f309c00b040 con 0x564aab9e6340 2024-08-03T20:56:25.304 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.304+0000 7f30a8487f40 1 -- 172.21.15.32:0/706711870 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:14 12.10 12:0cf6ad0f:::10000000000.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e97) -- 0x564aabb61ae0 con 0x564aab9e6340 2024-08-03T20:56:25.305 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.305+0000 7f30a2000640 1 -- 172.21.15.32:0/706711870 <== osd.0 v2:172.21.15.32:6816/3066156116 14 ==== osd_op_reply(14 10000000000.00000000 [delete] v97'363 uv363 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f309c00b040 con 0x564aab9e6340 2024-08-03T20:56:25.306 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.305+0000 7f30a8487f40 1 -- 172.21.15.32:0/706711870 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:15 12.10 12:0cf6ad0f:::10000000000.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97) -- 0x564aabb61ae0 con 0x564aab9e6340 2024-08-03T20:56:25.307 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::10000000000.00000000:head# 2024-08-03T20:56:25.308 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.307+0000 7f30a2000640 1 -- 172.21.15.32:0/706711870 <== osd.0 v2:172.21.15.32:6816/3066156116 15 ==== osd_op_reply(15 10000000000.00000000 [create] v97'364 uv364 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f309c00b040 con 0x564aab9e6340 2024-08-03T20:56:25.308 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.307+0000 7f30a8487f40 1 -- 172.21.15.32:0/706711870 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:16 12.10 12:0cf6ad0f:::10000000000.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97) -- 0x564aabb61ae0 con 0x564aab9e6340 2024-08-03T20:56:25.309 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.308+0000 7f30a2000640 1 -- 172.21.15.32:0/706711870 <== osd.0 v2:172.21.15.32:6816/3066156116 16 ==== osd_op_reply(16 10000000000.00000000 [setxattr (30)] v97'365 uv365 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f309c00b040 con 0x564aab9e6340 2024-08-03T20:56:25.309 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.308+0000 7f30a8487f40 1 -- 172.21.15.32:0/706711870 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:17 12.10 12:0cf6ad0f:::10000000000.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97) -- 0x564aabb61ae0 con 0x564aab9e6340 2024-08-03T20:56:25.310 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.310+0000 7f30a2000640 1 -- 172.21.15.32:0/706711870 <== osd.0 v2:172.21.15.32:6816/3066156116 17 ==== osd_op_reply(17 10000000000.00000000 [setxattr (62)] v97'366 uv366 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f309c00b040 con 0x564aab9e6340 2024-08-03T20:56:25.311 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.310+0000 7f30a8487f40 1 -- 172.21.15.32:0/706711870 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:18 12.10 12:0cf6ad0f:::10000000000.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97) -- 0x564aabb61ae0 con 0x564aab9e6340 2024-08-03T20:56:25.312 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.311+0000 7f30a2000640 1 -- 172.21.15.32:0/706711870 <== osd.0 v2:172.21.15.32:6816/3066156116 18 ==== osd_op_reply(18 10000000000.00000000 [omap-set-header] v97'367 uv367 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f309c00b040 con 0x564aab9e6340 2024-08-03T20:56:25.312 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.311+0000 7f30a8487f40 1 -- 172.21.15.32:0/706711870 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:19 12.10 12:0cf6ad0f:::10000000000.00000000:head [omap-set-vals in=593b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97) -- 0x564aabb61ea0 con 0x564aab9e6340 2024-08-03T20:56:25.313 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.313+0000 7f30a2000640 1 -- 172.21.15.32:0/706711870 <== osd.0 v2:172.21.15.32:6816/3066156116 19 ==== osd_op_reply(19 10000000000.00000000 [omap-set-vals] v97'368 uv368 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f309c00b040 con 0x564aab9e6340 2024-08-03T20:56:25.313 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.313+0000 7f30a8487f40 1 -- 172.21.15.32:0/706711870 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:20 12.10 12:0d82a743:::600.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97) -- 0x564aabb61ea0 con 0x564aab9e6340 2024-08-03T20:56:25.315 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.314+0000 7f30a2000640 1 -- 172.21.15.32:0/706711870 <== osd.0 v2:172.21.15.32:6816/3066156116 20 ==== osd_op_reply(20 600.00000000 [create] v97'369 uv343 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f309c00b040 con 0x564aab9e6340 2024-08-03T20:56:25.315 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.314+0000 7f30a8487f40 1 -- 172.21.15.32:0/706711870 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:21 12.10 12:0d82a743:::600.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e97) -- 0x564aabb61da0 con 0x564aab9e6340 2024-08-03T20:56:25.317 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.316+0000 7f30a2000640 1 -- 172.21.15.32:0/706711870 <== osd.0 v2:172.21.15.32:6816/3066156116 21 ==== osd_op_reply(21 600.00000000 [delete] v97'370 uv370 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f309c00b040 con 0x564aab9e6340 2024-08-03T20:56:25.317 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.316+0000 7f30a8487f40 1 -- 172.21.15.32:0/706711870 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:22 12.10 12:0d82a743:::600.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97) -- 0x564aabb61da0 con 0x564aab9e6340 2024-08-03T20:56:25.318 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::600.00000000:head# 2024-08-03T20:56:25.318 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.318+0000 7f30a2000640 1 -- 172.21.15.32:0/706711870 <== osd.0 v2:172.21.15.32:6816/3066156116 22 ==== osd_op_reply(22 600.00000000 [create] v97'371 uv371 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f309c00b040 con 0x564aab9e6340 2024-08-03T20:56:25.318 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.318+0000 7f30a8487f40 1 -- 172.21.15.32:0/706711870 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:23 12.10 12:0d82a743:::600.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97) -- 0x564aabb61da0 con 0x564aab9e6340 2024-08-03T20:56:25.320 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.319+0000 7f30a2000640 1 -- 172.21.15.32:0/706711870 <== osd.0 v2:172.21.15.32:6816/3066156116 23 ==== osd_op_reply(23 600.00000000 [setxattr (30)] v97'372 uv372 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f309c00b040 con 0x564aab9e6340 2024-08-03T20:56:25.320 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.319+0000 7f30a8487f40 1 -- 172.21.15.32:0/706711870 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:24 12.10 12:0d82a743:::600.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97) -- 0x564aabb61da0 con 0x564aab9e6340 2024-08-03T20:56:25.321 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.321+0000 7f30a2000640 1 -- 172.21.15.32:0/706711870 <== osd.0 v2:172.21.15.32:6816/3066156116 24 ==== osd_op_reply(24 600.00000000 [setxattr (62)] v97'373 uv373 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f309c00b040 con 0x564aab9e6340 2024-08-03T20:56:25.321 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.321+0000 7f30a8487f40 1 -- 172.21.15.32:0/706711870 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:25 12.10 12:0d82a743:::600.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97) -- 0x564aabb61da0 con 0x564aab9e6340 2024-08-03T20:56:25.323 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.323+0000 7f30a2000640 1 -- 172.21.15.32:0/706711870 <== osd.0 v2:172.21.15.32:6816/3066156116 25 ==== osd_op_reply(25 600.00000000 [omap-set-header] v97'374 uv374 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f309c00b040 con 0x564aab9e6340 2024-08-03T20:56:25.323 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.323+0000 7f30a8487f40 1 -- 172.21.15.32:0/706711870 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:26 12.10 12:0d89b25e:::603.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97) -- 0x564aabb61da0 con 0x564aab9e6340 2024-08-03T20:56:25.324 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.324+0000 7f30a2000640 1 -- 172.21.15.32:0/706711870 <== osd.0 v2:172.21.15.32:6816/3066156116 26 ==== osd_op_reply(26 603.00000000 [create] v97'375 uv349 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f309c00b040 con 0x564aab9e6340 2024-08-03T20:56:25.324 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.324+0000 7f30a8487f40 1 -- 172.21.15.32:0/706711870 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:27 12.10 12:0d89b25e:::603.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e97) -- 0x564aabb61da0 con 0x564aab9e6340 2024-08-03T20:56:25.326 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.326+0000 7f30a2000640 1 -- 172.21.15.32:0/706711870 <== osd.0 v2:172.21.15.32:6816/3066156116 27 ==== osd_op_reply(27 603.00000000 [delete] v97'376 uv376 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f309c00b040 con 0x564aab9e6340 2024-08-03T20:56:25.326 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.326+0000 7f30a8487f40 1 -- 172.21.15.32:0/706711870 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:28 12.10 12:0d89b25e:::603.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97) -- 0x564aabb61da0 con 0x564aab9e6340 2024-08-03T20:56:25.327 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::603.00000000:head# 2024-08-03T20:56:25.328 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.327+0000 7f30a2000640 1 -- 172.21.15.32:0/706711870 <== osd.0 v2:172.21.15.32:6816/3066156116 28 ==== osd_op_reply(28 603.00000000 [create] v97'377 uv377 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f309c00b040 con 0x564aab9e6340 2024-08-03T20:56:25.328 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.327+0000 7f30a8487f40 1 -- 172.21.15.32:0/706711870 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:29 12.10 12:0d89b25e:::603.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97) -- 0x564aabb61da0 con 0x564aab9e6340 2024-08-03T20:56:25.329 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.329+0000 7f30a2000640 1 -- 172.21.15.32:0/706711870 <== osd.0 v2:172.21.15.32:6816/3066156116 29 ==== osd_op_reply(29 603.00000000 [setxattr (30)] v97'378 uv378 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f309c00b040 con 0x564aab9e6340 2024-08-03T20:56:25.329 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.329+0000 7f30a8487f40 1 -- 172.21.15.32:0/706711870 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:30 12.10 12:0d89b25e:::603.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97) -- 0x564aabb61da0 con 0x564aab9e6340 2024-08-03T20:56:25.330 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.330+0000 7f30a2000640 1 -- 172.21.15.32:0/706711870 <== osd.0 v2:172.21.15.32:6816/3066156116 30 ==== osd_op_reply(30 603.00000000 [setxattr (62)] v97'379 uv379 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f309c00b040 con 0x564aab9e6340 2024-08-03T20:56:25.331 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.330+0000 7f30a8487f40 1 -- 172.21.15.32:0/706711870 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:31 12.10 12:0d89b25e:::603.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97) -- 0x564aabb61da0 con 0x564aab9e6340 2024-08-03T20:56:25.332 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.332+0000 7f30a2000640 1 -- 172.21.15.32:0/706711870 <== osd.0 v2:172.21.15.32:6816/3066156116 31 ==== osd_op_reply(31 603.00000000 [omap-set-header] v97'380 uv380 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f309c00b040 con 0x564aab9e6340 2024-08-03T20:56:25.332 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.332+0000 7f30a8487f40 1 --2- 172.21.15.32:0/706711870 >> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] conn(0x564aabb62c10 0x564aabb65030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:56:25.332 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.332+0000 7f30a8487f40 1 -- 172.21.15.32:0/706711870 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:32 12.14 12:287034c4:::1.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97) -- 0x564aabb61da0 con 0x564aabb62c10 2024-08-03T20:56:25.333 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.332+0000 7f30a1600640 1 --2- 172.21.15.32:0/706711870 >> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] conn(0x564aabb62c10 0x564aabb65030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:56:25.333 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.333+0000 7f30a1600640 1 --2- 172.21.15.32:0/706711870 >> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] conn(0x564aabb62c10 0x564aabb65030 crc :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:56:25.336 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.335+0000 7f30a1600640 1 -- 172.21.15.32:0/706711870 <== osd.5 v2:172.21.15.118:6810/2724891770 1 ==== osd_op_reply(32 1.00000000.inode [create] v97'262 uv245 ondisk = -17 ((17) File exists)) ==== 160+0+0 (crc 0 0 0) 0x7f3094021070 con 0x564aabb62c10 2024-08-03T20:56:25.336 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.335+0000 7f30a8487f40 1 -- 172.21.15.32:0/706711870 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:33 12.14 12:287034c4:::1.00000000.inode:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e97) -- 0x564aabb65570 con 0x564aabb62c10 2024-08-03T20:56:25.338 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.337+0000 7f30a1600640 1 -- 172.21.15.32:0/706711870 <== osd.5 v2:172.21.15.118:6810/2724891770 2 ==== osd_op_reply(33 1.00000000.inode [delete] v97'263 uv263 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f3094021070 con 0x564aabb62c10 2024-08-03T20:56:25.338 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.337+0000 7f30a8487f40 1 -- 172.21.15.32:0/706711870 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:34 12.14 12:287034c4:::1.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97) -- 0x564aabb65910 con 0x564aabb62c10 2024-08-03T20:56:25.339 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::1.00000000.inode:head# 2024-08-03T20:56:25.339 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.338+0000 7f30a1600640 1 -- 172.21.15.32:0/706711870 <== osd.5 v2:172.21.15.118:6810/2724891770 3 ==== osd_op_reply(34 1.00000000.inode [create] v97'264 uv264 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f3094021070 con 0x564aabb62c10 2024-08-03T20:56:25.340 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.339+0000 7f30a8487f40 1 -- 172.21.15.32:0/706711870 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:35 12.14 12:287034c4:::1.00000000.inode:head [write 0~582 in=582b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97) -- 0x564aabb65cb0 con 0x564aabb62c10 2024-08-03T20:56:25.341 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.340+0000 7f30a1600640 1 -- 172.21.15.32:0/706711870 <== osd.5 v2:172.21.15.118:6810/2724891770 4 ==== osd_op_reply(35 1.00000000.inode [write 0~582] v97'265 uv265 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f3094021070 con 0x564aabb62c10 2024-08-03T20:56:25.341 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.340+0000 7f30a8487f40 1 -- 172.21.15.32:0/706711870 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:36 12.14 12:287034c4:::1.00000000.inode:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97) -- 0x564aabb66050 con 0x564aabb62c10 2024-08-03T20:56:25.342 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.342+0000 7f30a1600640 1 -- 172.21.15.32:0/706711870 <== osd.5 v2:172.21.15.118:6810/2724891770 5 ==== osd_op_reply(36 1.00000000.inode [omap-set-header] v97'266 uv266 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f3094021070 con 0x564aabb62c10 2024-08-03T20:56:25.342 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.342+0000 7f30a8487f40 1 -- 172.21.15.32:0/706711870 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:37 12.14 12:292cf221:::200.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97) -- 0x564aabb66460 con 0x564aabb62c10 2024-08-03T20:56:25.344 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.343+0000 7f30a1600640 1 -- 172.21.15.32:0/706711870 <== osd.5 v2:172.21.15.118:6810/2724891770 6 ==== osd_op_reply(37 200.00000000 [create] v97'267 uv250 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f3094021070 con 0x564aabb62c10 2024-08-03T20:56:25.344 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.343+0000 7f30a8487f40 1 -- 172.21.15.32:0/706711870 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:38 12.14 12:292cf221:::200.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e97) -- 0x564aabb66850 con 0x564aabb62c10 2024-08-03T20:56:25.345 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.344+0000 7f30a1600640 1 -- 172.21.15.32:0/706711870 <== osd.5 v2:172.21.15.118:6810/2724891770 7 ==== osd_op_reply(38 200.00000000 [delete] v97'268 uv268 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f3094021070 con 0x564aabb62c10 2024-08-03T20:56:25.345 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.345+0000 7f30a8487f40 1 -- 172.21.15.32:0/706711870 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:39 12.14 12:292cf221:::200.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97) -- 0x564aabb66850 con 0x564aabb62c10 2024-08-03T20:56:25.346 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.346+0000 7f30a1600640 1 -- 172.21.15.32:0/706711870 <== osd.5 v2:172.21.15.118:6810/2724891770 8 ==== osd_op_reply(39 200.00000000 [create] v97'269 uv269 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f3094021070 con 0x564aabb62c10 2024-08-03T20:56:25.347 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::200.00000000:head# 2024-08-03T20:56:25.347 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.346+0000 7f30a8487f40 1 -- 172.21.15.32:0/706711870 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:40 12.14 12:292cf221:::200.00000000:head [write 0~90 in=90b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97) -- 0x564aabb66c90 con 0x564aabb62c10 2024-08-03T20:56:25.348 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.347+0000 7f30a1600640 1 -- 172.21.15.32:0/706711870 <== osd.5 v2:172.21.15.118:6810/2724891770 9 ==== osd_op_reply(40 200.00000000 [write 0~90] v97'270 uv270 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f3094021070 con 0x564aabb62c10 2024-08-03T20:56:25.348 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.348+0000 7f30a8487f40 1 -- 172.21.15.32:0/706711870 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:41 12.14 12:292cf221:::200.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97) -- 0x564aabb66c90 con 0x564aabb62c10 2024-08-03T20:56:25.349 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.349+0000 7f30a1600640 1 -- 172.21.15.32:0/706711870 <== osd.5 v2:172.21.15.118:6810/2724891770 10 ==== osd_op_reply(41 200.00000000 [omap-set-header] v97'271 uv271 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f3094021070 con 0x564aabb62c10 2024-08-03T20:56:25.350 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.349+0000 7f30a8487f40 1 -- 172.21.15.32:0/706711870 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:42 12.14 12:2e2fa760:::200.00000001:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97) -- 0x564aabb669d0 con 0x564aabb62c10 2024-08-03T20:56:25.351 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.350+0000 7f30a1600640 1 -- 172.21.15.32:0/706711870 <== osd.5 v2:172.21.15.118:6810/2724891770 11 ==== osd_op_reply(42 200.00000001 [create] v97'272 uv255 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f3094021070 con 0x564aabb62c10 2024-08-03T20:56:25.351 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.350+0000 7f30a8487f40 1 -- 172.21.15.32:0/706711870 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:43 12.14 12:2e2fa760:::200.00000001:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e97) -- 0x564aabb669d0 con 0x564aabb62c10 2024-08-03T20:56:25.352 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.352+0000 7f30a1600640 1 -- 172.21.15.32:0/706711870 <== osd.5 v2:172.21.15.118:6810/2724891770 12 ==== osd_op_reply(43 200.00000001 [delete] v97'273 uv273 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f3094021070 con 0x564aabb62c10 2024-08-03T20:56:25.352 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.352+0000 7f30a8487f40 1 -- 172.21.15.32:0/706711870 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:44 12.14 12:2e2fa760:::200.00000001:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97) -- 0x564aabb66980 con 0x564aabb62c10 2024-08-03T20:56:25.354 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::200.00000001:head# 2024-08-03T20:56:25.354 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.353+0000 7f30a1600640 1 -- 172.21.15.32:0/706711870 <== osd.5 v2:172.21.15.118:6810/2724891770 13 ==== osd_op_reply(44 200.00000001 [create] v97'274 uv274 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f3094021070 con 0x564aabb62c10 2024-08-03T20:56:25.354 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.353+0000 7f30a8487f40 1 -- 172.21.15.32:0/706711870 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:45 12.14 12:2e2fa760:::200.00000001:head [write 0~18118 in=18118b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97) -- 0x564aabb66980 con 0x564aabb62c10 2024-08-03T20:56:25.355 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.355+0000 7f30a1600640 1 -- 172.21.15.32:0/706711870 <== osd.5 v2:172.21.15.118:6810/2724891770 14 ==== osd_op_reply(45 200.00000001 [write 0~18118] v97'275 uv275 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f3094021070 con 0x564aabb62c10 2024-08-03T20:56:25.356 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.355+0000 7f30a8487f40 1 -- 172.21.15.32:0/706711870 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:46 12.14 12:2e2fa760:::200.00000001:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97) -- 0x564aabb66930 con 0x564aabb62c10 2024-08-03T20:56:25.357 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.356+0000 7f30a1600640 1 -- 172.21.15.32:0/706711870 <== osd.5 v2:172.21.15.118:6810/2724891770 15 ==== osd_op_reply(46 200.00000001 [omap-set-header] v97'276 uv276 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f3094021070 con 0x564aabb62c10 2024-08-03T20:56:25.357 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.356+0000 7f30a8487f40 1 -- 172.21.15.32:0/706711870 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:47 12.14 12:2f55791f:::606.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97) -- 0x564aabb66930 con 0x564aabb62c10 2024-08-03T20:56:25.359 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.358+0000 7f30a1600640 1 -- 172.21.15.32:0/706711870 <== osd.5 v2:172.21.15.118:6810/2724891770 16 ==== osd_op_reply(47 606.00000000 [create] v97'277 uv261 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f3094021070 con 0x564aabb62c10 2024-08-03T20:56:25.359 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.358+0000 7f30a8487f40 1 -- 172.21.15.32:0/706711870 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:48 12.14 12:2f55791f:::606.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e97) -- 0x564aabb66930 con 0x564aabb62c10 2024-08-03T20:56:25.360 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.359+0000 7f30a1600640 1 -- 172.21.15.32:0/706711870 <== osd.5 v2:172.21.15.118:6810/2724891770 17 ==== osd_op_reply(48 606.00000000 [delete] v97'278 uv278 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f3094021070 con 0x564aabb62c10 2024-08-03T20:56:25.360 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.359+0000 7f30a8487f40 1 -- 172.21.15.32:0/706711870 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:49 12.14 12:2f55791f:::606.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97) -- 0x564aabb66930 con 0x564aabb62c10 2024-08-03T20:56:25.361 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::606.00000000:head# 2024-08-03T20:56:25.362 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.361+0000 7f30a1600640 1 -- 172.21.15.32:0/706711870 <== osd.5 v2:172.21.15.118:6810/2724891770 18 ==== osd_op_reply(49 606.00000000 [create] v97'279 uv279 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f3094021070 con 0x564aabb62c10 2024-08-03T20:56:25.362 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.361+0000 7f30a8487f40 1 -- 172.21.15.32:0/706711870 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:50 12.14 12:2f55791f:::606.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97) -- 0x564aabb67d30 con 0x564aabb62c10 2024-08-03T20:56:25.363 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.362+0000 7f30a1600640 1 -- 172.21.15.32:0/706711870 <== osd.5 v2:172.21.15.118:6810/2724891770 19 ==== osd_op_reply(50 606.00000000 [setxattr (30)] v97'280 uv280 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f3094021070 con 0x564aabb62c10 2024-08-03T20:56:25.363 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.362+0000 7f30a8487f40 1 -- 172.21.15.32:0/706711870 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:51 12.14 12:2f55791f:::606.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97) -- 0x564aabb67d30 con 0x564aabb62c10 2024-08-03T20:56:25.364 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.364+0000 7f30a1600640 1 -- 172.21.15.32:0/706711870 <== osd.5 v2:172.21.15.118:6810/2724891770 20 ==== osd_op_reply(51 606.00000000 [setxattr (62)] v97'281 uv281 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f3094021070 con 0x564aabb62c10 2024-08-03T20:56:25.364 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.364+0000 7f30a8487f40 1 -- 172.21.15.32:0/706711870 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:52 12.14 12:2f55791f:::606.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97) -- 0x564aabb66a40 con 0x564aabb62c10 2024-08-03T20:56:25.366 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.365+0000 7f30a1600640 1 -- 172.21.15.32:0/706711870 <== osd.5 v2:172.21.15.118:6810/2724891770 21 ==== osd_op_reply(52 606.00000000 [omap-set-header] v97'282 uv282 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f3094021070 con 0x564aabb62c10 2024-08-03T20:56:25.366 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.365+0000 7f30a8487f40 1 -- 172.21.15.32:0/706711870 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:53 12.1c 12:3ed09add:::607.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97) -- 0x564aabb669f0 con 0x564aab9e6340 2024-08-03T20:56:25.367 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.367+0000 7f30a2000640 1 -- 172.21.15.32:0/706711870 <== osd.0 v2:172.21.15.32:6816/3066156116 32 ==== osd_op_reply(53 607.00000000 [create] v97'69 uv68 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f309c00b040 con 0x564aab9e6340 2024-08-03T20:56:25.367 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.367+0000 7f30a8487f40 1 -- 172.21.15.32:0/706711870 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:54 12.1c 12:3ed09add:::607.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e97) -- 0x564aabb669f0 con 0x564aab9e6340 2024-08-03T20:56:25.368 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.368+0000 7f30a2000640 1 -- 172.21.15.32:0/706711870 <== osd.0 v2:172.21.15.32:6816/3066156116 33 ==== osd_op_reply(54 607.00000000 [delete] v97'70 uv70 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f309c00b040 con 0x564aab9e6340 2024-08-03T20:56:25.369 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.368+0000 7f30a8487f40 1 -- 172.21.15.32:0/706711870 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:55 12.1c 12:3ed09add:::607.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97) -- 0x564aabb669d0 con 0x564aab9e6340 2024-08-03T20:56:25.370 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::607.00000000:head# 2024-08-03T20:56:25.370 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.370+0000 7f30a2000640 1 -- 172.21.15.32:0/706711870 <== osd.0 v2:172.21.15.32:6816/3066156116 34 ==== osd_op_reply(55 607.00000000 [create] v97'71 uv71 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f309c00b040 con 0x564aab9e6340 2024-08-03T20:56:25.370 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.370+0000 7f30a8487f40 1 -- 172.21.15.32:0/706711870 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:56 12.1c 12:3ed09add:::607.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97) -- 0x564aabb679e0 con 0x564aab9e6340 2024-08-03T20:56:25.371 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.371+0000 7f30a2000640 1 -- 172.21.15.32:0/706711870 <== osd.0 v2:172.21.15.32:6816/3066156116 35 ==== osd_op_reply(56 607.00000000 [setxattr (30)] v97'72 uv72 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f309c00b040 con 0x564aab9e6340 2024-08-03T20:56:25.371 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.371+0000 7f30a8487f40 1 -- 172.21.15.32:0/706711870 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:57 12.1c 12:3ed09add:::607.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97) -- 0x564aabb679e0 con 0x564aab9e6340 2024-08-03T20:56:25.373 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.372+0000 7f30a2000640 1 -- 172.21.15.32:0/706711870 <== osd.0 v2:172.21.15.32:6816/3066156116 36 ==== osd_op_reply(57 607.00000000 [setxattr (62)] v97'73 uv73 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f309c00b040 con 0x564aab9e6340 2024-08-03T20:56:25.373 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.372+0000 7f30a8487f40 1 -- 172.21.15.32:0/706711870 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:58 12.1c 12:3ed09add:::607.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97) -- 0x564aabb669d0 con 0x564aab9e6340 2024-08-03T20:56:25.374 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.374+0000 7f30a2000640 1 -- 172.21.15.32:0/706711870 <== osd.0 v2:172.21.15.32:6816/3066156116 37 ==== osd_op_reply(58 607.00000000 [omap-set-header] v97'74 uv74 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f309c00b040 con 0x564aab9e6340 2024-08-03T20:56:25.374 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.374+0000 7f30a8487f40 1 --2- 172.21.15.32:0/706711870 >> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] conn(0x564aabb670a0 0x564aabb67560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:56:25.375 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.374+0000 7f30a8487f40 1 -- 172.21.15.32:0/706711870 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:59 12.6 12:654134d2:::mds0_openfiles.0:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97) -- 0x564aabb69df0 con 0x564aabb670a0 2024-08-03T20:56:25.375 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.374+0000 7f30a0c00640 1 --2- 172.21.15.32:0/706711870 >> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] conn(0x564aabb670a0 0x564aabb67560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:56:25.375 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.375+0000 7f30a0c00640 1 --2- 172.21.15.32:0/706711870 >> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] conn(0x564aabb670a0 0x564aabb67560 crc :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:56:25.376 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.376+0000 7f30a0c00640 1 -- 172.21.15.32:0/706711870 <== osd.1 v2:172.21.15.32:6824/1551241759 1 ==== osd_op_reply(59 mds0_openfiles.0 [create] v97'59 uv58 ondisk = -17 ((17) File exists)) ==== 160+0+0 (crc 0 0 0) 0x7f3090077040 con 0x564aabb670a0 2024-08-03T20:56:25.376 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.376+0000 7f30a8487f40 1 -- 172.21.15.32:0/706711870 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:60 12.6 12:654134d2:::mds0_openfiles.0:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e97) -- 0x564aabb6a1b0 con 0x564aabb670a0 2024-08-03T20:56:25.378 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.377+0000 7f30a0c00640 1 -- 172.21.15.32:0/706711870 <== osd.1 v2:172.21.15.32:6824/1551241759 2 ==== osd_op_reply(60 mds0_openfiles.0 [delete] v97'60 uv60 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f3090077040 con 0x564aabb670a0 2024-08-03T20:56:25.378 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.377+0000 7f30a8487f40 1 -- 172.21.15.32:0/706711870 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:61 12.6 12:654134d2:::mds0_openfiles.0:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97) -- 0x564aabb6a6f0 con 0x564aabb670a0 2024-08-03T20:56:25.379 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds0_openfiles.0:head# 2024-08-03T20:56:25.379 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.379+0000 7f30a0c00640 1 -- 172.21.15.32:0/706711870 <== osd.1 v2:172.21.15.32:6824/1551241759 3 ==== osd_op_reply(61 mds0_openfiles.0 [create] v97'61 uv61 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f3090077040 con 0x564aabb670a0 2024-08-03T20:56:25.379 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.379+0000 7f30a8487f40 1 -- 172.21.15.32:0/706711870 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:62 12.6 12:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97) -- 0x564aabb6ab60 con 0x564aabb670a0 2024-08-03T20:56:25.380 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.380+0000 7f30a0c00640 1 -- 172.21.15.32:0/706711870 <== osd.1 v2:172.21.15.32:6824/1551241759 4 ==== osd_op_reply(62 mds0_openfiles.0 [omap-set-header] v97'62 uv62 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f3090077040 con 0x564aabb670a0 2024-08-03T20:56:25.380 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.380+0000 7f30a8487f40 1 -- 172.21.15.32:0/706711870 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:63 12.e 12:717a0223:::608.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97) -- 0x564aabb6af90 con 0x564aab9e6340 2024-08-03T20:56:25.382 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.382+0000 7f30a2000640 1 -- 172.21.15.32:0/706711870 <== osd.0 v2:172.21.15.32:6816/3066156116 38 ==== osd_op_reply(63 608.00000000 [create] v97'69 uv68 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f309c00b040 con 0x564aab9e6340 2024-08-03T20:56:25.382 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.382+0000 7f30a8487f40 1 -- 172.21.15.32:0/706711870 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:64 12.e 12:717a0223:::608.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e97) -- 0x564aabb6af90 con 0x564aab9e6340 2024-08-03T20:56:25.384 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.383+0000 7f30a2000640 1 -- 172.21.15.32:0/706711870 <== osd.0 v2:172.21.15.32:6816/3066156116 39 ==== osd_op_reply(64 608.00000000 [delete] v97'70 uv70 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f309c00b040 con 0x564aab9e6340 2024-08-03T20:56:25.384 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.383+0000 7f30a8487f40 1 -- 172.21.15.32:0/706711870 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:65 12.e 12:717a0223:::608.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97) -- 0x564aabb6af90 con 0x564aab9e6340 2024-08-03T20:56:25.385 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.385+0000 7f30a2000640 1 -- 172.21.15.32:0/706711870 <== osd.0 v2:172.21.15.32:6816/3066156116 40 ==== osd_op_reply(65 608.00000000 [create] v97'71 uv71 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f309c00b040 con 0x564aab9e6340 2024-08-03T20:56:25.385 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::608.00000000:head# 2024-08-03T20:56:25.386 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.385+0000 7f30a8487f40 1 -- 172.21.15.32:0/706711870 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:66 12.e 12:717a0223:::608.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97) -- 0x564aabb6bfa0 con 0x564aab9e6340 2024-08-03T20:56:25.386 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.386+0000 7f30a2000640 1 -- 172.21.15.32:0/706711870 <== osd.0 v2:172.21.15.32:6816/3066156116 41 ==== osd_op_reply(66 608.00000000 [setxattr (30)] v97'72 uv72 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f309c00b040 con 0x564aab9e6340 2024-08-03T20:56:25.387 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.386+0000 7f30a8487f40 1 -- 172.21.15.32:0/706711870 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:67 12.e 12:717a0223:::608.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97) -- 0x564aabb6bfa0 con 0x564aab9e6340 2024-08-03T20:56:25.388 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.388+0000 7f30a2000640 1 -- 172.21.15.32:0/706711870 <== osd.0 v2:172.21.15.32:6816/3066156116 42 ==== osd_op_reply(67 608.00000000 [setxattr (62)] v97'73 uv73 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f309c00b040 con 0x564aab9e6340 2024-08-03T20:56:25.388 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.388+0000 7f30a8487f40 1 -- 172.21.15.32:0/706711870 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:68 12.e 12:717a0223:::608.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97) -- 0x564aabb6af90 con 0x564aab9e6340 2024-08-03T20:56:25.391 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.389+0000 7f30a2000640 1 -- 172.21.15.32:0/706711870 <== osd.0 v2:172.21.15.32:6816/3066156116 43 ==== osd_op_reply(68 608.00000000 [omap-set-header] v97'74 uv74 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f309c00b040 con 0x564aab9e6340 2024-08-03T20:56:25.391 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.389+0000 7f30a8487f40 1 --2- 172.21.15.32:0/706711870 >> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] conn(0x564aabb6b610 0x564aabb6da50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:56:25.391 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.389+0000 7f30a8487f40 1 -- 172.21.15.32:0/706711870 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:69 12.15 12:a93a17c2:::604.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97) -- 0x564aabb6e0d0 con 0x564aabb6b610 2024-08-03T20:56:25.391 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.390+0000 7f30a2000640 1 --2- 172.21.15.32:0/706711870 >> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] conn(0x564aabb6b610 0x564aabb6da50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:56:25.391 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.390+0000 7f30a2000640 1 --2- 172.21.15.32:0/706711870 >> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] conn(0x564aabb6b610 0x564aabb6da50 crc :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:56:25.393 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.392+0000 7f30a2000640 1 -- 172.21.15.32:0/706711870 <== osd.6 v2:172.21.15.118:6825/801947386 1 ==== osd_op_reply(69 604.00000000 [create] v97'127 uv121 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f309c00b040 con 0x564aabb6b610 2024-08-03T20:56:25.393 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.392+0000 7f30a8487f40 1 -- 172.21.15.32:0/706711870 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:70 12.15 12:a93a17c2:::604.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e97) -- 0x564aabb6e4a0 con 0x564aabb6b610 2024-08-03T20:56:25.394 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.394+0000 7f30a2000640 1 -- 172.21.15.32:0/706711870 <== osd.6 v2:172.21.15.118:6825/801947386 2 ==== osd_op_reply(70 604.00000000 [delete] v97'128 uv128 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f309c00b040 con 0x564aabb6b610 2024-08-03T20:56:25.395 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.394+0000 7f30a8487f40 1 -- 172.21.15.32:0/706711870 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:71 12.15 12:a93a17c2:::604.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97) -- 0x564aabb6e4a0 con 0x564aabb6b610 2024-08-03T20:56:25.395 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.395+0000 7f30a2000640 1 -- 172.21.15.32:0/706711870 <== osd.6 v2:172.21.15.118:6825/801947386 3 ==== osd_op_reply(71 604.00000000 [create] v97'129 uv129 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f309c00b040 con 0x564aabb6b610 2024-08-03T20:56:25.396 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::604.00000000:head# 2024-08-03T20:56:25.396 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.395+0000 7f30a8487f40 1 -- 172.21.15.32:0/706711870 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:72 12.15 12:a93a17c2:::604.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97) -- 0x564aabb6f4b0 con 0x564aabb6b610 2024-08-03T20:56:25.397 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.396+0000 7f30a2000640 1 -- 172.21.15.32:0/706711870 <== osd.6 v2:172.21.15.118:6825/801947386 4 ==== osd_op_reply(72 604.00000000 [setxattr (30)] v97'130 uv130 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f309c00b040 con 0x564aabb6b610 2024-08-03T20:56:25.397 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.397+0000 7f30a8487f40 1 -- 172.21.15.32:0/706711870 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:73 12.15 12:a93a17c2:::604.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97) -- 0x564aabb6f4b0 con 0x564aabb6b610 2024-08-03T20:56:25.398 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.398+0000 7f30a2000640 1 -- 172.21.15.32:0/706711870 <== osd.6 v2:172.21.15.118:6825/801947386 5 ==== osd_op_reply(73 604.00000000 [setxattr (62)] v97'131 uv131 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f309c00b040 con 0x564aabb6b610 2024-08-03T20:56:25.399 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.398+0000 7f30a8487f40 1 -- 172.21.15.32:0/706711870 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:74 12.15 12:a93a17c2:::604.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97) -- 0x564aabb6e4a0 con 0x564aabb6b610 2024-08-03T20:56:25.400 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.399+0000 7f30a2000640 1 -- 172.21.15.32:0/706711870 <== osd.6 v2:172.21.15.118:6825/801947386 6 ==== osd_op_reply(74 604.00000000 [omap-set-header] v97'132 uv132 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f309c00b040 con 0x564aabb6b610 2024-08-03T20:56:25.400 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.399+0000 7f30a8487f40 1 -- 172.21.15.32:0/706711870 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:75 12.15 12:aa448500:::500.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97) -- 0x564aabb6e4a0 con 0x564aabb6b610 2024-08-03T20:56:25.401 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.401+0000 7f30a2000640 1 -- 172.21.15.32:0/706711870 <== osd.6 v2:172.21.15.118:6825/801947386 7 ==== osd_op_reply(75 500.00000000 [create] v97'133 uv126 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f309c00b040 con 0x564aabb6b610 2024-08-03T20:56:25.401 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.401+0000 7f30a8487f40 1 -- 172.21.15.32:0/706711870 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:76 12.15 12:aa448500:::500.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e97) -- 0x564aabb6e4a0 con 0x564aabb6b610 2024-08-03T20:56:25.403 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.402+0000 7f30a2000640 1 -- 172.21.15.32:0/706711870 <== osd.6 v2:172.21.15.118:6825/801947386 8 ==== osd_op_reply(76 500.00000000 [delete] v97'134 uv134 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f309c00b040 con 0x564aabb6b610 2024-08-03T20:56:25.403 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.402+0000 7f30a8487f40 1 -- 172.21.15.32:0/706711870 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:77 12.15 12:aa448500:::500.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97) -- 0x564aabb6e4a0 con 0x564aabb6b610 2024-08-03T20:56:25.404 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.403+0000 7f30a2000640 1 -- 172.21.15.32:0/706711870 <== osd.6 v2:172.21.15.118:6825/801947386 9 ==== osd_op_reply(77 500.00000000 [create] v97'135 uv135 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f309c00b040 con 0x564aabb6b610 2024-08-03T20:56:25.404 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::500.00000000:head# 2024-08-03T20:56:25.405 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.403+0000 7f30a8487f40 1 -- 172.21.15.32:0/706711870 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:78 12.15 12:aa448500:::500.00000000:head [write 0~90 in=90b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97) -- 0x564aabb6e4a0 con 0x564aabb6b610 2024-08-03T20:56:25.406 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.405+0000 7f30a2000640 1 -- 172.21.15.32:0/706711870 <== osd.6 v2:172.21.15.118:6825/801947386 10 ==== osd_op_reply(78 500.00000000 [write 0~90] v97'136 uv136 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f309c00b040 con 0x564aabb6b610 2024-08-03T20:56:25.406 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.405+0000 7f30a8487f40 1 -- 172.21.15.32:0/706711870 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:79 12.15 12:aa448500:::500.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97) -- 0x564aababa800 con 0x564aabb6b610 2024-08-03T20:56:25.407 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.406+0000 7f30a2000640 1 -- 172.21.15.32:0/706711870 <== osd.6 v2:172.21.15.118:6825/801947386 11 ==== osd_op_reply(79 500.00000000 [omap-set-header] v97'137 uv137 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f309c00b040 con 0x564aabb6b610 2024-08-03T20:56:25.407 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.406+0000 7f30a8487f40 1 --2- 172.21.15.32:0/706711870 >> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] conn(0x564aabb6ee60 0x564aabb71280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:56:25.408 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.407+0000 7f30a8487f40 1 -- 172.21.15.32:0/706711870 --> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] -- osd_op(unknown.0.0:80 12.d 12:b50e409b:::mds_snaptable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97) -- 0x564aabb717c0 con 0x564aabb6ee60 2024-08-03T20:56:25.408 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.407+0000 7f30a1600640 1 --2- 172.21.15.32:0/706711870 >> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] conn(0x564aabb6ee60 0x564aabb71280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:56:25.409 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.407+0000 7f30a1600640 1 --2- 172.21.15.32:0/706711870 >> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] conn(0x564aabb6ee60 0x564aabb71280 crc :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:56:25.409 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.409+0000 7f30a1600640 1 -- 172.21.15.32:0/706711870 <== osd.7 v2:172.21.15.118:6818/653000048 1 ==== osd_op_reply(80 mds_snaptable [create] v97'56 uv55 ondisk = -17 ((17) File exists)) ==== 157+0+0 (crc 0 0 0) 0x7f3094021070 con 0x564aabb6ee60 2024-08-03T20:56:25.410 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.409+0000 7f30a8487f40 1 -- 172.21.15.32:0/706711870 --> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] -- osd_op(unknown.0.0:81 12.d 12:b50e409b:::mds_snaptable:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e97) -- 0x564aabb71820 con 0x564aabb6ee60 2024-08-03T20:56:25.410 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.410+0000 7f30a1600640 1 -- 172.21.15.32:0/706711870 <== osd.7 v2:172.21.15.118:6818/653000048 2 ==== osd_op_reply(81 mds_snaptable [delete] v97'57 uv57 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f3094021070 con 0x564aabb6ee60 2024-08-03T20:56:25.411 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.410+0000 7f30a8487f40 1 -- 172.21.15.32:0/706711870 --> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] -- osd_op(unknown.0.0:82 12.d 12:b50e409b:::mds_snaptable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97) -- 0x564aabb71c10 con 0x564aabb6ee60 2024-08-03T20:56:25.411 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.411+0000 7f30a1600640 1 -- 172.21.15.32:0/706711870 <== osd.7 v2:172.21.15.118:6818/653000048 3 ==== osd_op_reply(82 mds_snaptable [create] v97'58 uv58 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f3094021070 con 0x564aabb6ee60 2024-08-03T20:56:25.412 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds_snaptable:head# 2024-08-03T20:56:25.412 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.411+0000 7f30a8487f40 1 -- 172.21.15.32:0/706711870 --> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] -- osd_op(unknown.0.0:83 12.d 12:b50e409b:::mds_snaptable:head [write 0~70 in=70b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97) -- 0x564aabb71c10 con 0x564aabb6ee60 2024-08-03T20:56:25.413 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.412+0000 7f30a1600640 1 -- 172.21.15.32:0/706711870 <== osd.7 v2:172.21.15.118:6818/653000048 4 ==== osd_op_reply(83 mds_snaptable [write 0~70] v97'59 uv59 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f3094021070 con 0x564aabb6ee60 2024-08-03T20:56:25.413 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.412+0000 7f30a8487f40 1 -- 172.21.15.32:0/706711870 --> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] -- osd_op(unknown.0.0:84 12.d 12:b50e409b:::mds_snaptable:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97) -- 0x564aabb71c10 con 0x564aabb6ee60 2024-08-03T20:56:25.414 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.414+0000 7f30a1600640 1 -- 172.21.15.32:0/706711870 <== osd.7 v2:172.21.15.118:6818/653000048 5 ==== osd_op_reply(84 mds_snaptable [omap-set-header] v97'60 uv60 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f3094021070 con 0x564aabb6ee60 2024-08-03T20:56:25.414 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.414+0000 7f30a8487f40 1 -- 172.21.15.32:0/706711870 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:85 12.1d 12:b871830b:::605.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97) -- 0x564aabb71c10 con 0x564aab9e6340 2024-08-03T20:56:25.416 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.415+0000 7f30a2000640 1 -- 172.21.15.32:0/706711870 <== osd.0 v2:172.21.15.32:6816/3066156116 44 ==== osd_op_reply(85 605.00000000 [create] v97'69 uv68 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f309c00b040 con 0x564aab9e6340 2024-08-03T20:56:25.416 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.415+0000 7f30a8487f40 1 -- 172.21.15.32:0/706711870 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:86 12.1d 12:b871830b:::605.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e97) -- 0x564aabb71c10 con 0x564aab9e6340 2024-08-03T20:56:25.417 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.417+0000 7f30a2000640 1 -- 172.21.15.32:0/706711870 <== osd.0 v2:172.21.15.32:6816/3066156116 45 ==== osd_op_reply(86 605.00000000 [delete] v97'70 uv70 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f309c00b040 con 0x564aab9e6340 2024-08-03T20:56:25.417 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.417+0000 7f30a8487f40 1 -- 172.21.15.32:0/706711870 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:87 12.1d 12:b871830b:::605.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97) -- 0x564aabb71c10 con 0x564aab9e6340 2024-08-03T20:56:25.419 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.418+0000 7f30a2000640 1 -- 172.21.15.32:0/706711870 <== osd.0 v2:172.21.15.32:6816/3066156116 46 ==== osd_op_reply(87 605.00000000 [create] v97'71 uv71 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f309c00b040 con 0x564aab9e6340 2024-08-03T20:56:25.419 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::605.00000000:head# 2024-08-03T20:56:25.419 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.418+0000 7f30a8487f40 1 -- 172.21.15.32:0/706711870 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:88 12.1d 12:b871830b:::605.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97) -- 0x564aabb72c20 con 0x564aab9e6340 2024-08-03T20:56:25.420 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.420+0000 7f30a2000640 1 -- 172.21.15.32:0/706711870 <== osd.0 v2:172.21.15.32:6816/3066156116 47 ==== osd_op_reply(88 605.00000000 [setxattr (30)] v97'72 uv72 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f309c00b040 con 0x564aab9e6340 2024-08-03T20:56:25.420 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.420+0000 7f30a8487f40 1 -- 172.21.15.32:0/706711870 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:89 12.1d 12:b871830b:::605.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97) -- 0x564aabb72c20 con 0x564aab9e6340 2024-08-03T20:56:25.422 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.421+0000 7f30a2000640 1 -- 172.21.15.32:0/706711870 <== osd.0 v2:172.21.15.32:6816/3066156116 48 ==== osd_op_reply(89 605.00000000 [setxattr (62)] v97'73 uv73 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f309c00b040 con 0x564aab9e6340 2024-08-03T20:56:25.422 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.421+0000 7f30a8487f40 1 -- 172.21.15.32:0/706711870 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:90 12.1d 12:b871830b:::605.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97) -- 0x564aabb71c10 con 0x564aab9e6340 2024-08-03T20:56:25.423 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.423+0000 7f30a2000640 1 -- 172.21.15.32:0/706711870 <== osd.0 v2:172.21.15.32:6816/3066156116 49 ==== osd_op_reply(90 605.00000000 [omap-set-header] v97'74 uv74 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f309c00b040 con 0x564aab9e6340 2024-08-03T20:56:25.423 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.423+0000 7f30a8487f40 1 -- 172.21.15.32:0/706711870 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:91 12.13 12:c91d4a1d:::mds0_inotable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97) -- 0x564aabb71c10 con 0x564aabb670a0 2024-08-03T20:56:25.424 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.424+0000 7f30a0c00640 1 -- 172.21.15.32:0/706711870 <== osd.1 v2:172.21.15.32:6824/1551241759 5 ==== osd_op_reply(91 mds0_inotable [create] v97'112 uv108 ondisk = -17 ((17) File exists)) ==== 157+0+0 (crc 0 0 0) 0x7f3090077040 con 0x564aabb670a0 2024-08-03T20:56:25.425 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.424+0000 7f30a8487f40 1 -- 172.21.15.32:0/706711870 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:92 12.13 12:c91d4a1d:::mds0_inotable:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e97) -- 0x564aabb71fb0 con 0x564aabb670a0 2024-08-03T20:56:25.426 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.426+0000 7f30a0c00640 1 -- 172.21.15.32:0/706711870 <== osd.1 v2:172.21.15.32:6824/1551241759 6 ==== osd_op_reply(92 mds0_inotable [delete] v97'113 uv113 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f3090077040 con 0x564aabb670a0 2024-08-03T20:56:25.426 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.426+0000 7f30a8487f40 1 -- 172.21.15.32:0/706711870 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:93 12.13 12:c91d4a1d:::mds0_inotable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97) -- 0x564aabb72350 con 0x564aabb670a0 2024-08-03T20:56:25.427 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds0_inotable:head# 2024-08-03T20:56:25.428 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.427+0000 7f30a0c00640 1 -- 172.21.15.32:0/706711870 <== osd.1 v2:172.21.15.32:6824/1551241759 7 ==== osd_op_reply(93 mds0_inotable [create] v97'114 uv114 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f3090077040 con 0x564aabb670a0 2024-08-03T20:56:25.428 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.427+0000 7f30a8487f40 1 -- 172.21.15.32:0/706711870 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:94 12.13 12:c91d4a1d:::mds0_inotable:head [write 0~34 in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97) -- 0x564aabb72350 con 0x564aabb670a0 2024-08-03T20:56:25.429 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.428+0000 7f30a0c00640 1 -- 172.21.15.32:0/706711870 <== osd.1 v2:172.21.15.32:6824/1551241759 8 ==== osd_op_reply(94 mds0_inotable [write 0~34] v97'115 uv115 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f3090077040 con 0x564aabb670a0 2024-08-03T20:56:25.429 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.428+0000 7f30a8487f40 1 -- 172.21.15.32:0/706711870 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:95 12.13 12:c91d4a1d:::mds0_inotable:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97) -- 0x564aabb72350 con 0x564aabb670a0 2024-08-03T20:56:25.430 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.430+0000 7f30a0c00640 1 -- 172.21.15.32:0/706711870 <== osd.1 v2:172.21.15.32:6824/1551241759 9 ==== osd_op_reply(95 mds0_inotable [omap-set-header] v97'116 uv116 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f3090077040 con 0x564aabb670a0 2024-08-03T20:56:25.430 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.430+0000 7f30a8487f40 1 -- 172.21.15.32:0/706711870 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:96 12.13 12:cd5a64a3:::100.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97) -- 0x564aabb72350 con 0x564aabb670a0 2024-08-03T20:56:25.432 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.431+0000 7f30a0c00640 1 -- 172.21.15.32:0/706711870 <== osd.1 v2:172.21.15.32:6824/1551241759 10 ==== osd_op_reply(96 100.00000000 [create] v97'117 uv111 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f3090077040 con 0x564aabb670a0 2024-08-03T20:56:25.432 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.431+0000 7f30a8487f40 1 -- 172.21.15.32:0/706711870 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:97 12.13 12:cd5a64a3:::100.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e97) -- 0x564aabb72350 con 0x564aabb670a0 2024-08-03T20:56:25.433 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.433+0000 7f30a0c00640 1 -- 172.21.15.32:0/706711870 <== osd.1 v2:172.21.15.32:6824/1551241759 11 ==== osd_op_reply(97 100.00000000 [delete] v97'118 uv118 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f3090077040 con 0x564aabb670a0 2024-08-03T20:56:25.433 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.433+0000 7f30a8487f40 1 -- 172.21.15.32:0/706711870 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:98 12.13 12:cd5a64a3:::100.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97) -- 0x564aabb72350 con 0x564aabb670a0 2024-08-03T20:56:25.434 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.434+0000 7f30a0c00640 1 -- 172.21.15.32:0/706711870 <== osd.1 v2:172.21.15.32:6824/1551241759 12 ==== osd_op_reply(98 100.00000000 [create] v97'119 uv119 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f3090077040 con 0x564aabb670a0 2024-08-03T20:56:25.434 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::100.00000000:head# 2024-08-03T20:56:25.435 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.434+0000 7f30a8487f40 1 -- 172.21.15.32:0/706711870 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:99 12.13 12:cd5a64a3:::100.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97) -- 0x564aabb72350 con 0x564aabb670a0 2024-08-03T20:56:25.436 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.435+0000 7f30a0c00640 1 -- 172.21.15.32:0/706711870 <== osd.1 v2:172.21.15.32:6824/1551241759 13 ==== osd_op_reply(99 100.00000000 [omap-set-header] v97'120 uv120 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f3090077040 con 0x564aabb670a0 2024-08-03T20:56:25.436 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.435+0000 7f30a8487f40 1 -- 172.21.15.32:0/706711870 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:100 12.13 12:cd5a64a3:::100.00000000:head [omap-set-vals in=5194b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97) -- 0x564aabb75160 con 0x564aabb670a0 2024-08-03T20:56:25.437 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.437+0000 7f30a0c00640 1 -- 172.21.15.32:0/706711870 <== osd.1 v2:172.21.15.32:6824/1551241759 14 ==== osd_op_reply(100 100.00000000 [omap-set-vals] v97'121 uv121 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f3090077040 con 0x564aabb670a0 2024-08-03T20:56:25.437 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.437+0000 7f30a8487f40 1 --2- 172.21.15.32:0/706711870 >> [v2:172.21.15.32:6800/9088303,v1:172.21.15.32:6801/9088303] conn(0x564aabb72e80 0x564aabb75800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:56:25.437 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.437+0000 7f30a8487f40 1 -- 172.21.15.32:0/706711870 --> [v2:172.21.15.32:6800/9088303,v1:172.21.15.32:6801/9088303] -- osd_op(unknown.0.0:101 12.b 12:d0630e4c:::mds0_sessionmap:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97) -- 0x564aabb752a0 con 0x564aabb72e80 2024-08-03T20:56:25.437 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.437+0000 7f30a0c00640 1 --2- 172.21.15.32:0/706711870 >> [v2:172.21.15.32:6800/9088303,v1:172.21.15.32:6801/9088303] conn(0x564aabb72e80 0x564aabb75800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:56:25.438 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.438+0000 7f30a0c00640 1 --2- 172.21.15.32:0/706711870 >> [v2:172.21.15.32:6800/9088303,v1:172.21.15.32:6801/9088303] conn(0x564aabb72e80 0x564aabb75800 crc :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:56:25.439 INFO:teuthology.orchestra.run.smithi032.stdout:Write #-9223372036854775808:00000000:::mds0_sessionmap:head# 2024-08-03T20:56:25.440 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.439+0000 7f30a0c00640 1 -- 172.21.15.32:0/706711870 <== osd.3 v2:172.21.15.32:6800/9088303 1 ==== osd_op_reply(101 mds0_sessionmap [create] v97'54 uv54 ondisk = 0) ==== 159+0+0 (crc 0 0 0) 0x7f3090077040 con 0x564aabb72e80 2024-08-03T20:56:25.440 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.439+0000 7f30a8487f40 1 -- 172.21.15.32:0/706711870 --> [v2:172.21.15.32:6800/9088303,v1:172.21.15.32:6801/9088303] -- osd_op(unknown.0.0:102 12.b 12:d0630e4c:::mds0_sessionmap:head [omap-set-header in=14b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97) -- 0x564aabb75360 con 0x564aabb72e80 2024-08-03T20:56:25.441 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.440+0000 7f30a0c00640 1 -- 172.21.15.32:0/706711870 <== osd.3 v2:172.21.15.32:6800/9088303 2 ==== osd_op_reply(102 mds0_sessionmap [omap-set-header] v97'55 uv55 ondisk = 0) ==== 159+0+0 (crc 0 0 0) 0x7f3090077040 con 0x564aabb72e80 2024-08-03T20:56:25.441 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.440+0000 7f30a8487f40 1 -- 172.21.15.32:0/706711870 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:103 12.7 12:e0b41d06:::609.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97) -- 0x564aabb753e0 con 0x564aabb670a0 2024-08-03T20:56:25.442 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.442+0000 7f30a0c00640 1 -- 172.21.15.32:0/706711870 <== osd.1 v2:172.21.15.32:6824/1551241759 15 ==== osd_op_reply(103 609.00000000 [create] v97'69 uv68 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f3090077040 con 0x564aabb670a0 2024-08-03T20:56:25.442 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.442+0000 7f30a8487f40 1 -- 172.21.15.32:0/706711870 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:104 12.7 12:e0b41d06:::609.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e97) -- 0x564aabb753e0 con 0x564aabb670a0 2024-08-03T20:56:25.444 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.443+0000 7f30a0c00640 1 -- 172.21.15.32:0/706711870 <== osd.1 v2:172.21.15.32:6824/1551241759 16 ==== osd_op_reply(104 609.00000000 [delete] v97'70 uv70 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f3090077040 con 0x564aabb670a0 2024-08-03T20:56:25.444 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.443+0000 7f30a8487f40 1 -- 172.21.15.32:0/706711870 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:105 12.7 12:e0b41d06:::609.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97) -- 0x564aabb753e0 con 0x564aabb670a0 2024-08-03T20:56:25.445 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.445+0000 7f30a0c00640 1 -- 172.21.15.32:0/706711870 <== osd.1 v2:172.21.15.32:6824/1551241759 17 ==== osd_op_reply(105 609.00000000 [create] v97'71 uv71 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f3090077040 con 0x564aabb670a0 2024-08-03T20:56:25.445 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::609.00000000:head# 2024-08-03T20:56:25.445 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.445+0000 7f30a8487f40 1 -- 172.21.15.32:0/706711870 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:106 12.7 12:e0b41d06:::609.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97) -- 0x564aabb753e0 con 0x564aabb670a0 2024-08-03T20:56:25.446 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.446+0000 7f30a0c00640 1 -- 172.21.15.32:0/706711870 <== osd.1 v2:172.21.15.32:6824/1551241759 18 ==== osd_op_reply(106 609.00000000 [setxattr (30)] v97'72 uv72 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f3090077040 con 0x564aabb670a0 2024-08-03T20:56:25.446 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.446+0000 7f30a8487f40 1 -- 172.21.15.32:0/706711870 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:107 12.7 12:e0b41d06:::609.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97) -- 0x564aabb753e0 con 0x564aabb670a0 2024-08-03T20:56:25.447 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.447+0000 7f30a0c00640 1 -- 172.21.15.32:0/706711870 <== osd.1 v2:172.21.15.32:6824/1551241759 19 ==== osd_op_reply(107 609.00000000 [setxattr (62)] v97'73 uv73 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f3090077040 con 0x564aabb670a0 2024-08-03T20:56:25.448 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.447+0000 7f30a8487f40 1 -- 172.21.15.32:0/706711870 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:108 12.7 12:e0b41d06:::609.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97) -- 0x564aabb753e0 con 0x564aabb670a0 2024-08-03T20:56:25.449 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.449+0000 7f30a0c00640 1 -- 172.21.15.32:0/706711870 <== osd.1 v2:172.21.15.32:6824/1551241759 20 ==== osd_op_reply(108 609.00000000 [omap-set-header] v97'74 uv74 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f3090077040 con 0x564aabb670a0 2024-08-03T20:56:25.449 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.449+0000 7f30a8487f40 1 -- 172.21.15.32:0/706711870 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:109 12.f 12:f1f69726:::400.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97) -- 0x564aabb753e0 con 0x564aabb6b610 2024-08-03T20:56:25.451 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.450+0000 7f30a2000640 1 -- 172.21.15.32:0/706711870 <== osd.6 v2:172.21.15.118:6825/801947386 12 ==== osd_op_reply(109 400.00000000 [create] v97'57 uv56 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f309c00b040 con 0x564aabb6b610 2024-08-03T20:56:25.451 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.450+0000 7f30a8487f40 1 -- 172.21.15.32:0/706711870 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:110 12.f 12:f1f69726:::400.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e97) -- 0x564aabb753e0 con 0x564aabb6b610 2024-08-03T20:56:25.452 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.452+0000 7f30a2000640 1 -- 172.21.15.32:0/706711870 <== osd.6 v2:172.21.15.118:6825/801947386 13 ==== osd_op_reply(110 400.00000000 [delete] v97'58 uv58 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f309c00b040 con 0x564aabb6b610 2024-08-03T20:56:25.452 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.452+0000 7f30a8487f40 1 -- 172.21.15.32:0/706711870 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:111 12.f 12:f1f69726:::400.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97) -- 0x564aabb753e0 con 0x564aabb6b610 2024-08-03T20:56:25.454 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.454+0000 7f30a2000640 1 -- 172.21.15.32:0/706711870 <== osd.6 v2:172.21.15.118:6825/801947386 14 ==== osd_op_reply(111 400.00000000 [create] v97'59 uv59 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f309c00b040 con 0x564aabb6b610 2024-08-03T20:56:25.454 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::400.00000000:head# 2024-08-03T20:56:25.454 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.454+0000 7f30a8487f40 1 -- 172.21.15.32:0/706711870 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:112 12.f 12:f1f69726:::400.00000000:head [write 0~22 in=22b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97) -- 0x564aabb75d40 con 0x564aabb6b610 2024-08-03T20:56:25.456 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.455+0000 7f30a2000640 1 -- 172.21.15.32:0/706711870 <== osd.6 v2:172.21.15.118:6825/801947386 15 ==== osd_op_reply(112 400.00000000 [write 0~22] v97'60 uv60 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f309c00b040 con 0x564aabb6b610 2024-08-03T20:56:25.456 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.455+0000 7f30a8487f40 1 -- 172.21.15.32:0/706711870 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:113 12.f 12:f1f69726:::400.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97) -- 0x564aabb75d40 con 0x564aabb6b610 2024-08-03T20:56:25.457 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.457+0000 7f30a2000640 1 -- 172.21.15.32:0/706711870 <== osd.6 v2:172.21.15.118:6825/801947386 16 ==== osd_op_reply(113 400.00000000 [omap-set-header] v97'61 uv61 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f309c00b040 con 0x564aabb6b610 2024-08-03T20:56:25.457 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.457+0000 7f30a8487f40 1 -- 172.21.15.32:0/706711870 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:114 12.1f 12:fe07bba1:::100.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97) -- 0x564aabb75d40 con 0x564aabb6b610 2024-08-03T20:56:25.459 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.459+0000 7f30a2000640 1 -- 172.21.15.32:0/706711870 <== osd.6 v2:172.21.15.118:6825/801947386 17 ==== osd_op_reply(114 100.00000000.inode [create] v97'144 uv136 ondisk = -17 ((17) File exists)) ==== 162+0+0 (crc 0 0 0) 0x7f309c00b040 con 0x564aabb6b610 2024-08-03T20:56:25.459 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.459+0000 7f30a8487f40 1 -- 172.21.15.32:0/706711870 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:115 12.1f 12:fe07bba1:::100.00000000.inode:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e97) -- 0x564aabb75d40 con 0x564aabb6b610 2024-08-03T20:56:25.461 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.460+0000 7f30a2000640 1 -- 172.21.15.32:0/706711870 <== osd.6 v2:172.21.15.118:6825/801947386 18 ==== osd_op_reply(115 100.00000000.inode [delete] v97'145 uv145 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7f309c00b040 con 0x564aabb6b610 2024-08-03T20:56:25.461 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.460+0000 7f30a8487f40 1 -- 172.21.15.32:0/706711870 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:116 12.1f 12:fe07bba1:::100.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97) -- 0x564aabb75d40 con 0x564aabb6b610 2024-08-03T20:56:25.462 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::100.00000000.inode:head# 2024-08-03T20:56:25.462 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.462+0000 7f30a2000640 1 -- 172.21.15.32:0/706711870 <== osd.6 v2:172.21.15.118:6825/801947386 19 ==== osd_op_reply(116 100.00000000.inode [create] v97'146 uv146 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7f309c00b040 con 0x564aabb6b610 2024-08-03T20:56:25.463 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.462+0000 7f30a8487f40 1 -- 172.21.15.32:0/706711870 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:117 12.1f 12:fe07bba1:::100.00000000.inode:head [write 0~582 in=582b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97) -- 0x564aabb75d40 con 0x564aabb6b610 2024-08-03T20:56:25.464 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.464+0000 7f30a2000640 1 -- 172.21.15.32:0/706711870 <== osd.6 v2:172.21.15.118:6825/801947386 20 ==== osd_op_reply(117 100.00000000.inode [write 0~582] v97'147 uv147 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7f309c00b040 con 0x564aabb6b610 2024-08-03T20:56:25.464 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.464+0000 7f30a8487f40 1 -- 172.21.15.32:0/706711870 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:118 12.1f 12:fe07bba1:::100.00000000.inode:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97) -- 0x564aabb75d40 con 0x564aabb6b610 2024-08-03T20:56:25.466 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.465+0000 7f30a2000640 1 -- 172.21.15.32:0/706711870 <== osd.6 v2:172.21.15.118:6825/801947386 21 ==== osd_op_reply(118 100.00000000.inode [omap-set-header] v97'148 uv148 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7f309c00b040 con 0x564aabb6b610 2024-08-03T20:56:25.466 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.465+0000 7f30a8487f40 1 -- 172.21.15.32:0/706711870 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:119 12.1f 12:ff5b34d6:::1.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97) -- 0x564aabb75d40 con 0x564aabb6b610 2024-08-03T20:56:25.467 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.467+0000 7f30a2000640 1 -- 172.21.15.32:0/706711870 <== osd.6 v2:172.21.15.118:6825/801947386 22 ==== osd_op_reply(119 1.00000000 [create] v97'149 uv143 ondisk = -17 ((17) File exists)) ==== 154+0+0 (crc 0 0 0) 0x7f309c00b040 con 0x564aabb6b610 2024-08-03T20:56:25.467 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.467+0000 7f30a8487f40 1 -- 172.21.15.32:0/706711870 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:120 12.1f 12:ff5b34d6:::1.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e97) -- 0x564aabb75d40 con 0x564aabb6b610 2024-08-03T20:56:25.469 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.469+0000 7f30a2000640 1 -- 172.21.15.32:0/706711870 <== osd.6 v2:172.21.15.118:6825/801947386 23 ==== osd_op_reply(120 1.00000000 [delete] v97'150 uv150 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7f309c00b040 con 0x564aabb6b610 2024-08-03T20:56:25.469 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.469+0000 7f30a8487f40 1 -- 172.21.15.32:0/706711870 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:121 12.1f 12:ff5b34d6:::1.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97) -- 0x564aabb75d40 con 0x564aabb6b610 2024-08-03T20:56:25.471 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::1.00000000:head# 2024-08-03T20:56:25.471 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.470+0000 7f30a2000640 1 -- 172.21.15.32:0/706711870 <== osd.6 v2:172.21.15.118:6825/801947386 24 ==== osd_op_reply(121 1.00000000 [create] v97'151 uv151 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7f309c00b040 con 0x564aabb6b610 2024-08-03T20:56:25.471 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.470+0000 7f30a8487f40 1 -- 172.21.15.32:0/706711870 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:122 12.1f 12:ff5b34d6:::1.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97) -- 0x564aabb76d50 con 0x564aabb6b610 2024-08-03T20:56:25.472 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.472+0000 7f30a2000640 1 -- 172.21.15.32:0/706711870 <== osd.6 v2:172.21.15.118:6825/801947386 25 ==== osd_op_reply(122 1.00000000 [setxattr (30)] v97'152 uv152 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7f309c00b040 con 0x564aabb6b610 2024-08-03T20:56:25.472 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.472+0000 7f30a8487f40 1 -- 172.21.15.32:0/706711870 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:123 12.1f 12:ff5b34d6:::1.00000000:head [setxattr parent (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97) -- 0x564aabb76d50 con 0x564aabb6b610 2024-08-03T20:56:25.474 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.474+0000 7f30a2000640 1 -- 172.21.15.32:0/706711870 <== osd.6 v2:172.21.15.118:6825/801947386 26 ==== osd_op_reply(123 1.00000000 [setxattr (30)] v97'153 uv153 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7f309c00b040 con 0x564aabb6b610 2024-08-03T20:56:25.474 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.474+0000 7f30a8487f40 1 -- 172.21.15.32:0/706711870 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:124 12.1f 12:ff5b34d6:::1.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97) -- 0x564aabb75d40 con 0x564aabb6b610 2024-08-03T20:56:25.476 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.475+0000 7f30a2000640 1 -- 172.21.15.32:0/706711870 <== osd.6 v2:172.21.15.118:6825/801947386 27 ==== osd_op_reply(124 1.00000000 [omap-set-header] v97'154 uv154 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7f309c00b040 con 0x564aabb6b610 2024-08-03T20:56:25.476 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.475+0000 7f30a8487f40 1 -- 172.21.15.32:0/706711870 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:125 12.1f 12:ff5b34d6:::1.00000000:head [omap-set-vals in=584b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97) -- 0x564aabb764a0 con 0x564aabb6b610 2024-08-03T20:56:25.478 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.477+0000 7f30a2000640 1 -- 172.21.15.32:0/706711870 <== osd.6 v2:172.21.15.118:6825/801947386 28 ==== osd_op_reply(125 1.00000000 [omap-set-vals] v97'155 uv155 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7f309c00b040 con 0x564aabb6b610 2024-08-03T20:56:25.478 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.477+0000 7f30a8487f40 1 -- 172.21.15.32:0/706711870 >> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] conn(0x564aab9e6340 msgr2=0x564aab9e6750 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:56:25.478 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.477+0000 7f30a8487f40 1 --2- 172.21.15.32:0/706711870 >> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] conn(0x564aab9e6340 0x564aab9e6750 crc :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:25.479 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.477+0000 7f30a8487f40 1 -- 172.21.15.32:0/706711870 >> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] conn(0x564aabb670a0 msgr2=0x564aabb67560 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:56:25.479 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.477+0000 7f30a8487f40 1 --2- 172.21.15.32:0/706711870 >> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] conn(0x564aabb670a0 0x564aabb67560 crc :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:25.479 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.477+0000 7f30a8487f40 1 -- 172.21.15.32:0/706711870 >> [v2:172.21.15.32:6800/9088303,v1:172.21.15.32:6801/9088303] conn(0x564aabb72e80 msgr2=0x564aabb75800 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:56:25.479 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.477+0000 7f30a8487f40 1 --2- 172.21.15.32:0/706711870 >> [v2:172.21.15.32:6800/9088303,v1:172.21.15.32:6801/9088303] conn(0x564aabb72e80 0x564aabb75800 crc :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:25.479 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.477+0000 7f30a8487f40 1 -- 172.21.15.32:0/706711870 >> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] conn(0x564aabb62c10 msgr2=0x564aabb65030 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:56:25.479 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.477+0000 7f30a8487f40 1 --2- 172.21.15.32:0/706711870 >> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] conn(0x564aabb62c10 0x564aabb65030 crc :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:25.479 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.477+0000 7f30a8487f40 1 -- 172.21.15.32:0/706711870 >> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] conn(0x564aabb6b610 msgr2=0x564aabb6da50 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:56:25.479 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.477+0000 7f30a8487f40 1 --2- 172.21.15.32:0/706711870 >> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] conn(0x564aabb6b610 0x564aabb6da50 crc :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:25.479 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.477+0000 7f30a8487f40 1 -- 172.21.15.32:0/706711870 >> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] conn(0x564aabb6ee60 msgr2=0x564aabb71280 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:56:25.479 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.477+0000 7f30a8487f40 1 --2- 172.21.15.32:0/706711870 >> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] conn(0x564aabb6ee60 0x564aabb71280 crc :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:25.479 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.477+0000 7f30a2000640 1 -- 172.21.15.32:0/706711870 reap_dead start 2024-08-03T20:56:25.479 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.477+0000 7f30a2000640 1 -- 172.21.15.32:0/706711870 reap_dead start 2024-08-03T20:56:25.479 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.477+0000 7f30a2000640 1 -- 172.21.15.32:0/706711870 reap_dead start 2024-08-03T20:56:25.479 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.477+0000 7f30a8487f40 1 -- 172.21.15.32:0/706711870 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f30840759f0 msgr2=0x7f3084077f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:56:25.480 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.477+0000 7f30a8487f40 1 --2- 172.21.15.32:0/706711870 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f30840759f0 0x7f3084077f00 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7f3090062000 tx=0x7f3090004880 comp rx=0 tx=0).stop 2024-08-03T20:56:25.480 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.477+0000 7f30a8487f40 1 -- 172.21.15.32:0/706711870 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x564aaba9e360 msgr2=0x564aaba99a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:56:25.480 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.477+0000 7f30a8487f40 1 --2- 172.21.15.32:0/706711870 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x564aaba9e360 0x564aaba99a20 secure :-1 s=READY pgs=1357 cs=0 l=1 rev1=1 crypto rx=0x7f309400dd30 tx=0x7f309400b590 comp rx=0 tx=0).stop 2024-08-03T20:56:25.480 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.477+0000 7f30a8487f40 1 -- 172.21.15.32:0/706711870 shutdown_connections 2024-08-03T20:56:25.480 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.477+0000 7f30a8487f40 1 --2- 172.21.15.32:0/706711870 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f30840759f0 0x7f3084077f00 unknown :-1 s=CLOSED pgs=559 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:25.480 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.477+0000 7f30a8487f40 1 --2- 172.21.15.32:0/706711870 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x564aaba9e360 0x564aaba99a20 unknown :-1 s=CLOSED pgs=1357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:25.480 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.477+0000 7f30a8487f40 1 -- 172.21.15.32:0/706711870 >> 172.21.15.32:0/706711870 conn(0x564aab907ac0 msgr2=0x564aabab8c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:56:25.480 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.477+0000 7f30a8487f40 1 -- 172.21.15.32:0/706711870 shutdown_connections 2024-08-03T20:56:25.480 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.478+0000 7f30a8487f40 1 -- 172.21.15.32:0/706711870 wait complete. 2024-08-03T20:56:25.489 DEBUG:teuthology.orchestra.run.smithi032:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata rm 400.00000000 2024-08-03T20:56:25.673 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.672+0000 7f876c126f40 1 Processor -- start 2024-08-03T20:56:25.673 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.672+0000 7f876c126f40 1 -- start start 2024-08-03T20:56:25.675 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.674+0000 7f876c126f40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55a50f076360 0x55a50f0917c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:56:25.675 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.674+0000 7f876c126f40 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a50f077ea0 0x55a50f08e8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:56:25.675 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.674+0000 7f876c126f40 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55a50f08d330 0x55a50f094f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:56:25.675 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.674+0000 7f876c126f40 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55a50ef0de30 con 0x55a50f077ea0 2024-08-03T20:56:25.676 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.674+0000 7f876c126f40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x55a50ef6bbf0 con 0x55a50f076360 2024-08-03T20:56:25.676 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.674+0000 7f876c126f40 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x55a50f077440 con 0x55a50f08d330 2024-08-03T20:56:25.676 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.674+0000 7f875fe00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a50f077ea0 0x55a50f08e8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:56:25.676 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.674+0000 7f875fe00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a50f077ea0 0x55a50f08e8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:50866/0 (socket says 172.21.15.32:50866) 2024-08-03T20:56:25.676 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.674+0000 7f875fe00640 1 -- 172.21.15.32:0/495089897 learned_addr learned my addr 172.21.15.32:0/495089897 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:56:25.676 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.674+0000 7f8765c00640 1 --2- 172.21.15.32:0/495089897 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55a50f08d330 0x55a50f094f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:56:25.676 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.674+0000 7f8765200640 1 --2- 172.21.15.32:0/495089897 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55a50f076360 0x55a50f0917c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:56:25.676 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.674+0000 7f875fe00640 1 -- 172.21.15.32:0/495089897 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55a50f08d330 msgr2=0x55a50f094f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:56:25.676 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.674+0000 7f875fe00640 1 --2- 172.21.15.32:0/495089897 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55a50f08d330 0x55a50f094f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:25.676 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.674+0000 7f875fe00640 1 -- 172.21.15.32:0/495089897 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55a50f076360 msgr2=0x55a50f0917c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:56:25.676 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.674+0000 7f875fe00640 1 --2- 172.21.15.32:0/495089897 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55a50f076360 0x55a50f0917c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:25.676 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.674+0000 7f875fe00640 1 -- 172.21.15.32:0/495089897 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a50efbf3a0 con 0x55a50f077ea0 2024-08-03T20:56:25.676 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.674+0000 7f8765200640 1 --2- 172.21.15.32:0/495089897 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55a50f076360 0x55a50f0917c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-03T20:56:25.676 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.675+0000 7f875fe00640 1 --2- 172.21.15.32:0/495089897 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a50f077ea0 0x55a50f08e8e0 secure :-1 s=READY pgs=1360 cs=0 l=1 rev1=1 crypto rx=0x7f875000b5d0 tx=0x7f875000ba90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fa7ee8b5d9c839b1 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:56:25.677 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.675+0000 7f875ea00640 1 -- 172.21.15.32:0/495089897 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f875000a8c0 con 0x55a50f077ea0 2024-08-03T20:56:25.677 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.675+0000 7f875ea00640 1 -- 172.21.15.32:0/495089897 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f875000aa60 con 0x55a50f077ea0 2024-08-03T20:56:25.677 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.675+0000 7f875ea00640 1 -- 172.21.15.32:0/495089897 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8750015460 con 0x55a50f077ea0 2024-08-03T20:56:25.678 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.676+0000 7f876c126f40 1 -- 172.21.15.32:0/495089897 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a50f077ea0 msgr2=0x55a50f08e8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:56:25.678 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.676+0000 7f876c126f40 1 --2- 172.21.15.32:0/495089897 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a50f077ea0 0x55a50f08e8e0 secure :-1 s=READY pgs=1360 cs=0 l=1 rev1=1 crypto rx=0x7f875000b5d0 tx=0x7f875000ba90 comp rx=0 tx=0).stop 2024-08-03T20:56:25.678 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.676+0000 7f876c126f40 1 -- 172.21.15.32:0/495089897 shutdown_connections 2024-08-03T20:56:25.678 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.676+0000 7f876c126f40 1 --2- 172.21.15.32:0/495089897 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55a50f08d330 0x55a50f094f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:25.678 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.676+0000 7f876c126f40 1 --2- 172.21.15.32:0/495089897 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a50f077ea0 0x55a50f08e8e0 unknown :-1 s=CLOSED pgs=1360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:25.678 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.676+0000 7f876c126f40 1 --2- 172.21.15.32:0/495089897 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55a50f076360 0x55a50f0917c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:25.678 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.676+0000 7f876c126f40 1 -- 172.21.15.32:0/495089897 >> 172.21.15.32:0/495089897 conn(0x55a50eedfac0 msgr2=0x55a50f000e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:56:25.678 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.676+0000 7f876c126f40 1 -- 172.21.15.32:0/495089897 shutdown_connections 2024-08-03T20:56:25.678 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.676+0000 7f876c126f40 1 -- 172.21.15.32:0/495089897 wait complete. 2024-08-03T20:56:25.678 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.677+0000 7f876c126f40 1 Processor -- start 2024-08-03T20:56:25.679 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.677+0000 7f876c126f40 1 -- start start 2024-08-03T20:56:25.680 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.678+0000 7f876c126f40 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55a50f076360 0x55a50f093cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:56:25.680 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.678+0000 7f876c126f40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55a50f077ea0 0x55a50f0941f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:56:25.681 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.678+0000 7f876c126f40 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a50f08d330 0x55a50f072f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:56:25.681 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.678+0000 7f876c126f40 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55a50efbf7c0 con 0x55a50f08d330 2024-08-03T20:56:25.681 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.678+0000 7f876c126f40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x55a50efbfaa0 con 0x55a50f077ea0 2024-08-03T20:56:25.681 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.678+0000 7f876c126f40 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x55a50f095e10 con 0x55a50f076360 2024-08-03T20:56:25.681 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.679+0000 7f8765c00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a50f08d330 0x55a50f072f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:56:25.681 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.679+0000 7f8765c00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a50f08d330 0x55a50f072f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:50872/0 (socket says 172.21.15.32:50872) 2024-08-03T20:56:25.681 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.679+0000 7f8765c00640 1 -- 172.21.15.32:0/2824631697 learned_addr learned my addr 172.21.15.32:0/2824631697 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:56:25.681 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.679+0000 7f875fe00640 1 --2- 172.21.15.32:0/2824631697 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55a50f077ea0 0x55a50f0941f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:56:25.681 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.679+0000 7f8765200640 1 --2- 172.21.15.32:0/2824631697 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55a50f076360 0x55a50f093cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:56:25.681 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.679+0000 7f8765c00640 1 -- 172.21.15.32:0/2824631697 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55a50f076360 msgr2=0x55a50f093cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:56:25.681 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.679+0000 7f8765c00640 1 --2- 172.21.15.32:0/2824631697 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55a50f076360 0x55a50f093cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:25.681 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.679+0000 7f8765c00640 1 -- 172.21.15.32:0/2824631697 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55a50f077ea0 msgr2=0x55a50f0941f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:56:25.682 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.679+0000 7f8765c00640 1 --2- 172.21.15.32:0/2824631697 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55a50f077ea0 0x55a50f0941f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:25.682 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.679+0000 7f8765c00640 1 -- 172.21.15.32:0/2824631697 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a50f06e3b0 con 0x55a50f08d330 2024-08-03T20:56:25.682 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.679+0000 7f8765c00640 1 --2- 172.21.15.32:0/2824631697 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a50f08d330 0x55a50f072f00 secure :-1 s=READY pgs=1361 cs=0 l=1 rev1=1 crypto rx=0x7f876000a600 tx=0x7f876004d770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:56:25.682 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.680+0000 7f875d600640 1 -- 172.21.15.32:0/2824631697 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8760065030 con 0x55a50f08d330 2024-08-03T20:56:25.682 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.680+0000 7f875d600640 1 -- 172.21.15.32:0/2824631697 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f8760004110 con 0x55a50f08d330 2024-08-03T20:56:25.682 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.680+0000 7f876c126f40 1 -- 172.21.15.32:0/2824631697 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x55a50f072180 con 0x55a50f08d330 2024-08-03T20:56:25.682 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.680+0000 7f875d600640 1 -- 172.21.15.32:0/2824631697 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8760061450 con 0x55a50f08d330 2024-08-03T20:56:25.682 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.680+0000 7f876c126f40 1 -- 172.21.15.32:0/2824631697 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a50f06eab0 con 0x55a50f08d330 2024-08-03T20:56:25.682 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.682+0000 7f875d600640 1 -- 172.21.15.32:0/2824631697 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f876005e050 con 0x55a50f08d330 2024-08-03T20:56:25.684 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.683+0000 7f875d600640 1 --2- 172.21.15.32:0/2824631697 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f87440759f0 0x7f8744077f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:56:25.684 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.683+0000 7f875d600640 1 -- 172.21.15.32:0/2824631697 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 6886+0+0 (secure 0 0 0) 0x7f87600dc210 con 0x55a50f08d330 2024-08-03T20:56:25.684 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.683+0000 7f876c126f40 1 --2- 172.21.15.32:0/2824631697 >> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] conn(0x55a50f072710 0x55a50f12f6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:56:25.684 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.683+0000 7f876c126f40 1 -- 172.21.15.32:0/2824631697 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:1 12.f 12:f1f69726:::400.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e97) -- 0x55a50f06de90 con 0x55a50f072710 2024-08-03T20:56:25.684 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.683+0000 7f8765200640 1 --2- 172.21.15.32:0/2824631697 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f87440759f0 0x7f8744077f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:56:25.684 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.684+0000 7f875fe00640 1 --2- 172.21.15.32:0/2824631697 >> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] conn(0x55a50f072710 0x55a50f12f6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:56:25.685 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.684+0000 7f8765200640 1 --2- 172.21.15.32:0/2824631697 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f87440759f0 0x7f8744077f00 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x55a50f096780 tx=0x7f8758003810 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:56:25.685 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.684+0000 7f875fe00640 1 --2- 172.21.15.32:0/2824631697 >> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] conn(0x55a50f072710 0x55a50f12f6f0 crc :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:56:25.687 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.686+0000 7f875fe00640 1 -- 172.21.15.32:0/2824631697 <== osd.6 v2:172.21.15.118:6825/801947386 1 ==== osd_op_reply(1 400.00000000 [delete] v97'62 uv62 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f875000ab80 con 0x55a50f072710 2024-08-03T20:56:25.687 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.686+0000 7f876c126f40 1 -- 172.21.15.32:0/2824631697 >> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] conn(0x55a50f072710 msgr2=0x55a50f12f6f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:56:25.687 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.686+0000 7f876c126f40 1 --2- 172.21.15.32:0/2824631697 >> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] conn(0x55a50f072710 0x55a50f12f6f0 crc :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:25.687 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.686+0000 7f876c126f40 1 -- 172.21.15.32:0/2824631697 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f87440759f0 msgr2=0x7f8744077f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:56:25.687 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.686+0000 7f876c126f40 1 --2- 172.21.15.32:0/2824631697 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f87440759f0 0x7f8744077f00 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x55a50f096780 tx=0x7f8758003810 comp rx=0 tx=0).stop 2024-08-03T20:56:25.687 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.686+0000 7f876c126f40 1 -- 172.21.15.32:0/2824631697 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a50f08d330 msgr2=0x55a50f072f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:56:25.687 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.686+0000 7f876c126f40 1 --2- 172.21.15.32:0/2824631697 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a50f08d330 0x55a50f072f00 secure :-1 s=READY pgs=1361 cs=0 l=1 rev1=1 crypto rx=0x7f876000a600 tx=0x7f876004d770 comp rx=0 tx=0).stop 2024-08-03T20:56:25.687 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.686+0000 7f8765c00640 1 -- 172.21.15.32:0/2824631697 reap_dead start 2024-08-03T20:56:25.687 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.686+0000 7f876c126f40 1 -- 172.21.15.32:0/2824631697 shutdown_connections 2024-08-03T20:56:25.687 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.686+0000 7f876c126f40 1 -- 172.21.15.32:0/2824631697 >> 172.21.15.32:0/2824631697 conn(0x55a50eedfac0 msgr2=0x55a50f0923d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:56:25.687 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.686+0000 7f876c126f40 1 -- 172.21.15.32:0/2824631697 shutdown_connections 2024-08-03T20:56:25.688 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.686+0000 7f876c126f40 1 -- 172.21.15.32:0/2824631697 wait complete. 2024-08-03T20:56:25.700 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs joinable true 2024-08-03T20:56:25.841 INFO:tasks.ceph.mds.a.smithi032.stderr: -57> 2024-08-03T20:56:21.088+0000 7f5e6ea00640 -1 mds.0.sessionmap _load_finish got (2) No such file or directory 2024-08-03T20:56:25.841 INFO:tasks.ceph.mds.a.smithi032.stderr: -56> 2024-08-03T20:56:21.088+0000 7f5e6ea00640 -1 log_channel(cluster) log [ERR] : error reading sessionmap 'mds0_sessionmap' -2 ((2) No such file or directory) 2024-08-03T20:56:25.933 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.932+0000 7fddfe800640 1 -- 172.21.15.32:0/352243595 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fddf8155a40 msgr2=0x7fddf815eea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:56:25.933 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.932+0000 7fddfe800640 1 --2- 172.21.15.32:0/352243595 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fddf8155a40 0x7fddf815eea0 secure :-1 s=READY pgs=1363 cs=0 l=1 rev1=1 crypto rx=0x7fdde0005fd0 tx=0x7fdde002a620 comp rx=0 tx=0).stop 2024-08-03T20:56:25.934 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.933+0000 7fddfe800640 1 -- 172.21.15.32:0/352243595 shutdown_connections 2024-08-03T20:56:25.934 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.933+0000 7fddfe800640 1 --2- 172.21.15.32:0/352243595 >> v2:172.21.15.118:3300/0 conn(0x7fddf815f3e0 0x7fddf8072880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:25.934 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.933+0000 7fddfe800640 1 --2- 172.21.15.32:0/352243595 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fddf8155a40 0x7fddf815eea0 unknown :-1 s=CLOSED pgs=1363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:25.934 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.933+0000 7fddfe800640 1 -- 172.21.15.32:0/352243595 >> 172.21.15.32:0/352243595 conn(0x7fddf80b1bd0 msgr2=0x7fddf80b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:56:25.934 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.933+0000 7fddfe800640 1 -- 172.21.15.32:0/352243595 shutdown_connections 2024-08-03T20:56:25.934 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.933+0000 7fddfe800640 1 -- 172.21.15.32:0/352243595 wait complete. 2024-08-03T20:56:25.935 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.935+0000 7fddfe800640 1 Processor -- start 2024-08-03T20:56:25.935 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.935+0000 7fddfe800640 1 -- start start 2024-08-03T20:56:25.938 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.936+0000 7fddfe800640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fddf8155a40 0x7fddf81522c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:56:25.938 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.936+0000 7fddfe800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fddf815f3e0 0x7fddf8142480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:56:25.938 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.936+0000 7fddfe800640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fddf815fa60 0x7fddf81429c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:56:25.938 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.936+0000 7fddfe800640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fddf8073310 con 0x7fddf815f3e0 2024-08-03T20:56:25.938 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.936+0000 7fddfe800640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7fddf8073190 con 0x7fddf8155a40 2024-08-03T20:56:25.938 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.936+0000 7fddfe800640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7fddf8073490 con 0x7fddf815fa60 2024-08-03T20:56:25.938 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.937+0000 7fddefe00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fddf815f3e0 0x7fddf8142480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:56:25.938 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.937+0000 7fddefe00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fddf815f3e0 0x7fddf8142480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:50894/0 (socket says 172.21.15.32:50894) 2024-08-03T20:56:25.938 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.937+0000 7fddefe00640 1 -- 172.21.15.32:0/871833848 learned_addr learned my addr 172.21.15.32:0/871833848 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:56:25.938 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.937+0000 7fddf7400640 1 --2- 172.21.15.32:0/871833848 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fddf8155a40 0x7fddf81522c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:56:25.938 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.937+0000 7fddf7e00640 1 --2- 172.21.15.32:0/871833848 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fddf815fa60 0x7fddf81429c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:56:25.939 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.937+0000 7fddefe00640 1 -- 172.21.15.32:0/871833848 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fddf815fa60 msgr2=0x7fddf81429c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:56:25.939 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.937+0000 7fddefe00640 1 --2- 172.21.15.32:0/871833848 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fddf815fa60 0x7fddf81429c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:25.939 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.937+0000 7fddefe00640 1 -- 172.21.15.32:0/871833848 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fddf8155a40 msgr2=0x7fddf81522c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:56:25.939 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.937+0000 7fddefe00640 1 --2- 172.21.15.32:0/871833848 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fddf8155a40 0x7fddf81522c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:25.939 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.937+0000 7fddefe00640 1 -- 172.21.15.32:0/871833848 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fddf8152b10 con 0x7fddf815f3e0 2024-08-03T20:56:25.940 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.938+0000 7fddefe00640 1 --2- 172.21.15.32:0/871833848 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fddf815f3e0 0x7fddf8142480 secure :-1 s=READY pgs=1364 cs=0 l=1 rev1=1 crypto rx=0x7fdde0006010 tx=0x7fdde0004430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:56:25.940 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.938+0000 7fddef400640 1 -- 172.21.15.32:0/871833848 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdde003e030 con 0x7fddf815f3e0 2024-08-03T20:56:25.940 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.938+0000 7fddef400640 1 -- 172.21.15.32:0/871833848 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fdde0004aa0 con 0x7fddf815f3e0 2024-08-03T20:56:25.940 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.938+0000 7fddef400640 1 -- 172.21.15.32:0/871833848 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdde0035040 con 0x7fddf815f3e0 2024-08-03T20:56:25.940 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.938+0000 7fddfe800640 1 -- 172.21.15.32:0/871833848 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fddf8152df0 con 0x7fddf815f3e0 2024-08-03T20:56:25.940 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.938+0000 7fddfe800640 1 -- 172.21.15.32:0/871833848 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fddf81533b0 con 0x7fddf815f3e0 2024-08-03T20:56:25.940 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.940+0000 7fddfe800640 1 -- 172.21.15.32:0/871833848 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fddf8152880 con 0x7fddf815f3e0 2024-08-03T20:56:25.941 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.941+0000 7fddef400640 1 -- 172.21.15.32:0/871833848 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7fdde0041030 con 0x7fddf815f3e0 2024-08-03T20:56:25.942 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.942+0000 7fddef400640 1 --2- 172.21.15.32:0/871833848 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fddd4077040 0x7fddd4079550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:56:25.942 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.942+0000 7fddef400640 1 -- 172.21.15.32:0/871833848 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 6886+0+0 (secure 0 0 0) 0x7fdde00b91c0 con 0x7fddf815f3e0 2024-08-03T20:56:25.943 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.942+0000 7fddf7400640 1 --2- 172.21.15.32:0/871833848 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fddd4077040 0x7fddd4079550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:56:25.943 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.943+0000 7fddf7400640 1 --2- 172.21.15.32:0/871833848 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fddd4077040 0x7fddd4079550 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7fdde8001720 tx=0x7fdde8009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:56:25.947 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:25.946+0000 7fddef400640 1 -- 172.21.15.32:0/871833848 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7fdde0081840 con 0x7fddf815f3e0 2024-08-03T20:56:25.962 INFO:tasks.ceph.mds.a.smithi032.stdout:starting mds.a at 2024-08-03T20:56:26.206 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:26.205+0000 7fddfe800640 1 -- 172.21.15.32:0/871833848 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "joinable", "val": "true"} v 0) -- 0x7fddf80739b0 con 0x7fddf815f3e0 2024-08-03T20:56:27.108 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:27.108+0000 7fddef400640 1 -- 172.21.15.32:0/871833848 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "joinable", "val": "true"}]=0 cephfs marked joinable; MDS may join as newly active. v165) ==== 162+0+0 (secure 0 0 0) 0x7fdde008a0d0 con 0x7fddf815f3e0 2024-08-03T20:56:27.109 INFO:teuthology.orchestra.run.smithi032.stderr:cephfs marked joinable; MDS may join as newly active. 2024-08-03T20:56:27.112 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:27.112+0000 7fddfe800640 1 -- 172.21.15.32:0/871833848 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fddd4077040 msgr2=0x7fddd4079550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:56:27.112 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:27.112+0000 7fddfe800640 1 --2- 172.21.15.32:0/871833848 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fddd4077040 0x7fddd4079550 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7fdde8001720 tx=0x7fdde8009040 comp rx=0 tx=0).stop 2024-08-03T20:56:27.113 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:27.112+0000 7fddfe800640 1 -- 172.21.15.32:0/871833848 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fddf815f3e0 msgr2=0x7fddf8142480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:56:27.113 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:27.112+0000 7fddfe800640 1 --2- 172.21.15.32:0/871833848 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fddf815f3e0 0x7fddf8142480 secure :-1 s=READY pgs=1364 cs=0 l=1 rev1=1 crypto rx=0x7fdde0006010 tx=0x7fdde0004430 comp rx=0 tx=0).stop 2024-08-03T20:56:27.113 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:27.112+0000 7fddfe800640 1 -- 172.21.15.32:0/871833848 shutdown_connections 2024-08-03T20:56:27.113 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:27.112+0000 7fddfe800640 1 --2- 172.21.15.32:0/871833848 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fddd4077040 0x7fddd4079550 unknown :-1 s=CLOSED pgs=561 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:27.113 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:27.112+0000 7fddfe800640 1 --2- 172.21.15.32:0/871833848 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fddf815fa60 0x7fddf81429c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:27.113 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:27.112+0000 7fddfe800640 1 --2- 172.21.15.32:0/871833848 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fddf815f3e0 0x7fddf8142480 unknown :-1 s=CLOSED pgs=1364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:27.113 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:27.112+0000 7fddfe800640 1 --2- 172.21.15.32:0/871833848 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fddf8155a40 0x7fddf81522c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:27.113 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:27.112+0000 7fddfe800640 1 -- 172.21.15.32:0/871833848 >> 172.21.15.32:0/871833848 conn(0x7fddf80b1bd0 msgr2=0x7fddf815a380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:56:27.113 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:27.112+0000 7fddfe800640 1 -- 172.21.15.32:0/871833848 shutdown_connections 2024-08-03T20:56:27.113 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:27.112+0000 7fddfe800640 1 -- 172.21.15.32:0/871833848 wait complete. 2024-08-03T20:56:27.545 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-08-03T20:56:27.785 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:27.784+0000 7f3ce9c00640 1 -- 172.21.15.32:0/4116217024 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3ce40d29a0 msgr2=0x7f3ce40d1c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:56:27.785 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:27.784+0000 7f3ce9c00640 1 --2- 172.21.15.32:0/4116217024 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3ce40d29a0 0x7f3ce40d1c60 secure :-1 s=READY pgs=1368 cs=0 l=1 rev1=1 crypto rx=0x7f3ce40ba100 tx=0x7f3ccc02a4a0 comp rx=0 tx=0).stop 2024-08-03T20:56:27.785 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:27.784+0000 7f3ce9c00640 1 -- 172.21.15.32:0/4116217024 shutdown_connections 2024-08-03T20:56:27.785 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:27.784+0000 7f3ce9c00640 1 --2- 172.21.15.32:0/4116217024 >> v2:172.21.15.118:3300/0 conn(0x7f3ce40cec30 0x7f3ce40cf040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:27.785 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:27.784+0000 7f3ce9c00640 1 --2- 172.21.15.32:0/4116217024 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3ce40d29a0 0x7f3ce40d1c60 unknown :-1 s=CLOSED pgs=1368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:27.785 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:27.784+0000 7f3ce9c00640 1 -- 172.21.15.32:0/4116217024 >> 172.21.15.32:0/4116217024 conn(0x7f3ce40d4090 msgr2=0x7f3ce40d44a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:56:27.785 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:27.784+0000 7f3ce9c00640 1 -- 172.21.15.32:0/4116217024 shutdown_connections 2024-08-03T20:56:27.785 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:27.784+0000 7f3ce9c00640 1 -- 172.21.15.32:0/4116217024 wait complete. 2024-08-03T20:56:27.786 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:27.786+0000 7f3ce9c00640 1 Processor -- start 2024-08-03T20:56:27.787 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:27.786+0000 7f3ce9c00640 1 -- start start 2024-08-03T20:56:27.789 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:27.788+0000 7f3ce9c00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3ce40cec30 0x7f3ce40cdd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:56:27.789 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:27.788+0000 7f3ce9c00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f3ce40d4bb0 0x7f3ce40a1660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:56:27.789 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:27.788+0000 7f3ce9c00640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f3ce40a1ba0 0x7f3ce4166690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:56:27.789 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:27.788+0000 7f3ce9c00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f3ce40cfa80 con 0x7f3ce40cec30 2024-08-03T20:56:27.789 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:27.788+0000 7f3ce9c00640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f3ce40cf900 con 0x7f3ce40d4bb0 2024-08-03T20:56:27.789 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:27.788+0000 7f3ce9c00640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7f3ce40cfc00 con 0x7f3ce40a1ba0 2024-08-03T20:56:27.790 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:27.788+0000 7f3ce3e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3ce40cec30 0x7f3ce40cdd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:56:27.790 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:27.788+0000 7f3ce3e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3ce40cec30 0x7f3ce40cdd40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:50934/0 (socket says 172.21.15.32:50934) 2024-08-03T20:56:27.790 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:27.788+0000 7f3ce3e00640 1 -- 172.21.15.32:0/2999820703 learned_addr learned my addr 172.21.15.32:0/2999820703 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:56:27.790 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:27.788+0000 7f3cdb400640 1 --2- 172.21.15.32:0/2999820703 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f3ce40d4bb0 0x7f3ce40a1660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:56:27.790 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:27.788+0000 7f3ce9200640 1 --2- 172.21.15.32:0/2999820703 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f3ce40a1ba0 0x7f3ce4166690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:56:27.791 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:27.789+0000 7f3ce3e00640 1 -- 172.21.15.32:0/2999820703 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f3ce40a1ba0 msgr2=0x7f3ce4166690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:56:27.791 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:27.789+0000 7f3ce3e00640 1 --2- 172.21.15.32:0/2999820703 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f3ce40a1ba0 0x7f3ce4166690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:27.791 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:27.789+0000 7f3ce3e00640 1 -- 172.21.15.32:0/2999820703 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f3ce40d4bb0 msgr2=0x7f3ce40a1660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:56:27.791 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:27.789+0000 7f3ce3e00640 1 --2- 172.21.15.32:0/2999820703 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f3ce40d4bb0 0x7f3ce40a1660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:27.791 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:27.789+0000 7f3ce3e00640 1 -- 172.21.15.32:0/2999820703 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3ce41672f0 con 0x7f3ce40cec30 2024-08-03T20:56:27.794 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:27.789+0000 7f3ce3e00640 1 --2- 172.21.15.32:0/2999820703 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3ce40cec30 0x7f3ce40cdd40 secure :-1 s=READY pgs=1369 cs=0 l=1 rev1=1 crypto rx=0x7f3cd400c3d0 tx=0x7f3cd400c890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:56:27.795 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:27.789+0000 7f3ce0c00640 1 -- 172.21.15.32:0/2999820703 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3cd4011790 con 0x7f3ce40cec30 2024-08-03T20:56:27.795 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:27.789+0000 7f3ce0c00640 1 -- 172.21.15.32:0/2999820703 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f3cd4017070 con 0x7f3ce40cec30 2024-08-03T20:56:27.795 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:27.789+0000 7f3ce9c00640 1 -- 172.21.15.32:0/2999820703 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3ce41675d0 con 0x7f3ce40cec30 2024-08-03T20:56:27.795 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:27.789+0000 7f3ce0c00640 1 -- 172.21.15.32:0/2999820703 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3cd4011cc0 con 0x7f3ce40cec30 2024-08-03T20:56:27.795 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:27.790+0000 7f3ce9c00640 1 -- 172.21.15.32:0/2999820703 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3ce4167b90 con 0x7f3ce40cec30 2024-08-03T20:56:27.796 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:27.791+0000 7f3ce9c00640 1 -- 172.21.15.32:0/2999820703 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3ce40cf040 con 0x7f3ce40cec30 2024-08-03T20:56:27.796 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:27.792+0000 7f3ce0c00640 1 -- 172.21.15.32:0/2999820703 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f3cd401c460 con 0x7f3ce40cec30 2024-08-03T20:56:27.796 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:27.793+0000 7f3ce0c00640 1 --2- 172.21.15.32:0/2999820703 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f3cc0077010 0x7f3cc0079520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:56:27.796 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:27.793+0000 7f3ce0c00640 1 -- 172.21.15.32:0/2999820703 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 6886+0+0 (secure 0 0 0) 0x7f3cd4098c10 con 0x7f3ce40cec30 2024-08-03T20:56:27.796 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:27.793+0000 7f3cdb400640 1 --2- 172.21.15.32:0/2999820703 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f3cc0077010 0x7f3cc0079520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:56:27.797 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:27.794+0000 7f3cdb400640 1 --2- 172.21.15.32:0/2999820703 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f3cc0077010 0x7f3cc0079520 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7f3ccc034000 tx=0x7f3ccc004730 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:56:27.798 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:27.797+0000 7f3ce0c00640 1 -- 172.21.15.32:0/2999820703 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7f3cd4061960 con 0x7f3ce40cec30 2024-08-03T20:56:28.041 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:28.040+0000 7f3ce9c00640 1 -- 172.21.15.32:0/2999820703 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f3ce408c100 con 0x7f3ce40cec30 2024-08-03T20:56:28.042 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:28.041+0000 7f3ce0c00640 1 -- 172.21.15.32:0/2999820703 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 166 v166) ==== 95+0+5225 (secure 0 0 0) 0x7f3cd4061260 con 0x7f3ce40cec30 2024-08-03T20:56:28.042 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-03T20:56:28.042 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":166,"btime":"2024-08-03T20:56:27:113538+0000","default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6439,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6835/531080160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6834","nonce":531080160},{"type":"v1","addr":"172.21.15.118:6835","nonce":531080160}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":84},{"gid":6454,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6837/2796882319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6836","nonce":2796882319},{"type":"v1","addr":"172.21.15.118:6837","nonce":2796882319}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":86},{"gid":6813,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6834/2686933890","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":2686933890},{"type":"v1","addr":"172.21.15.32:6834","nonce":2686933890}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":85}],"filesystems":[{"mdsmap":{"epoch":166,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-08-03T20:54:26.950518+0000","modified":"2024-08-03T20:56:27.113521+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":97,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8250},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8250":{"gid":8250,"name":"a","rank":0,"incarnation":166,"state":"up:replay","state_seq":1,"addr":"172.21.15.32:6835/2806168786","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6833","nonce":2806168786},{"type":"v1","addr":"172.21.15.32:6835","nonce":2806168786}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":6}]} 2024-08-03T20:56:28.042 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 166 2024-08-03T20:56:28.046 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:28.043+0000 7f3ce9c00640 1 -- 172.21.15.32:0/2999820703 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f3cc0077010 msgr2=0x7f3cc0079520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:56:28.046 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:28.043+0000 7f3ce9c00640 1 --2- 172.21.15.32:0/2999820703 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f3cc0077010 0x7f3cc0079520 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7f3ccc034000 tx=0x7f3ccc004730 comp rx=0 tx=0).stop 2024-08-03T20:56:28.046 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:28.043+0000 7f3ce9c00640 1 -- 172.21.15.32:0/2999820703 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3ce40cec30 msgr2=0x7f3ce40cdd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:56:28.046 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:28.043+0000 7f3ce9c00640 1 --2- 172.21.15.32:0/2999820703 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3ce40cec30 0x7f3ce40cdd40 secure :-1 s=READY pgs=1369 cs=0 l=1 rev1=1 crypto rx=0x7f3cd400c3d0 tx=0x7f3cd400c890 comp rx=0 tx=0).stop 2024-08-03T20:56:28.046 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:28.044+0000 7f3ce9c00640 1 -- 172.21.15.32:0/2999820703 shutdown_connections 2024-08-03T20:56:28.046 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:28.044+0000 7f3ce9c00640 1 --2- 172.21.15.32:0/2999820703 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f3cc0077010 0x7f3cc0079520 unknown :-1 s=CLOSED pgs=563 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:28.047 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:28.044+0000 7f3ce9c00640 1 --2- 172.21.15.32:0/2999820703 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f3ce40d4bb0 0x7f3ce40a1660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:28.047 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:28.044+0000 7f3ce9c00640 1 --2- 172.21.15.32:0/2999820703 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f3ce40a1ba0 0x7f3ce4166690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:28.047 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:28.044+0000 7f3ce9c00640 1 --2- 172.21.15.32:0/2999820703 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3ce40cec30 0x7f3ce40cdd40 unknown :-1 s=CLOSED pgs=1369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:28.047 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:28.044+0000 7f3ce9c00640 1 -- 172.21.15.32:0/2999820703 >> 172.21.15.32:0/2999820703 conn(0x7f3ce40d4090 msgr2=0x7f3ce406e670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:56:28.047 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:28.044+0000 7f3ce9c00640 1 -- 172.21.15.32:0/2999820703 shutdown_connections 2024-08-03T20:56:28.047 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:28.044+0000 7f3ce9c00640 1 -- 172.21.15.32:0/2999820703 wait complete. 2024-08-03T20:56:28.066 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 166, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-08-03T20:54:26.950518+0000', 'modified': '2024-08-03T20:56:27.113521+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 97, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 8250}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8250': {'gid': 8250, 'name': 'a', 'rank': 0, 'incarnation': 166, 'state': 'up:replay', 'state_seq': 1, 'addr': '172.21.15.32:6835/2806168786', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6833', 'nonce': 2806168786}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 2806168786}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [13], 'metadata_pool': 12, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 0, 'qdb_cluster': []} 2024-08-03T20:56:28.066 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8250:up:replay 2024-08-03T20:56:28.066 DEBUG:tasks.ceph_test_case:wait_until_true: waiting (timeout=60 retry_count=0)... 2024-08-03T20:56:31.110 INFO:tasks.ceph.mds.a.smithi032.stderr:2024-08-03T20:56:31.110+0000 7fa5c5e00640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-08-03T20:56:33.470 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-08-03T20:56:33.709 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:33.709+0000 7f763ae00640 1 -- 172.21.15.32:0/2680167482 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f761c003c10 msgr2=0x7f761c004060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:56:33.710 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:33.709+0000 7f763ae00640 1 --2- 172.21.15.32:0/2680167482 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f761c003c10 0x7f761c004060 secure :-1 s=READY pgs=1371 cs=0 l=1 rev1=1 crypto rx=0x7f7628009b50 tx=0x7f761c005830 comp rx=0 tx=0).stop 2024-08-03T20:56:33.710 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:33.709+0000 7f763a400640 1 -- 172.21.15.32:0/2680167482 reap_dead start 2024-08-03T20:56:33.710 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:33.709+0000 7f763ae00640 1 -- 172.21.15.32:0/2680167482 shutdown_connections 2024-08-03T20:56:33.710 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:33.709+0000 7f763ae00640 1 -- 172.21.15.32:0/2680167482 >> 172.21.15.32:0/2680167482 conn(0x7f76340b2120 msgr2=0x7f76340cfd60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:56:33.710 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:33.709+0000 7f763ae00640 1 -- 172.21.15.32:0/2680167482 shutdown_connections 2024-08-03T20:56:33.710 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:33.709+0000 7f763ae00640 1 -- 172.21.15.32:0/2680167482 wait complete. 2024-08-03T20:56:33.712 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:33.711+0000 7f763ae00640 1 Processor -- start 2024-08-03T20:56:33.712 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:33.711+0000 7f763ae00640 1 -- start start 2024-08-03T20:56:33.714 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:33.713+0000 7f763ae00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f761c003c10 0x7f76340b4b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:56:33.714 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:33.713+0000 7f763ae00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f76340b5080 0x7f76341f8b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:56:33.714 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:33.713+0000 7f763ae00640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f76341ef770 0x7f76341f90d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:56:33.714 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:33.713+0000 7f763ae00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f7634074430 con 0x7f761c003c10 2024-08-03T20:56:33.714 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:33.713+0000 7f763ae00640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f76340742b0 con 0x7f76340b5080 2024-08-03T20:56:33.714 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:33.713+0000 7f763ae00640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7f76340745b0 con 0x7f76341ef770 2024-08-03T20:56:33.714 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:33.713+0000 7f7639a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f761c003c10 0x7f76340b4b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:56:33.714 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:33.713+0000 7f7639a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f761c003c10 0x7f76340b4b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:37002/0 (socket says 172.21.15.32:37002) 2024-08-03T20:56:33.714 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:33.713+0000 7f7639a00640 1 -- 172.21.15.32:0/943134207 learned_addr learned my addr 172.21.15.32:0/943134207 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:56:33.714 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:33.713+0000 7f7639a00640 1 -- 172.21.15.32:0/943134207 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f76341ef770 msgr2=0x7f76341f90d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:56:33.715 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:33.713+0000 7f7639a00640 1 --2- 172.21.15.32:0/943134207 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f76341ef770 0x7f76341f90d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:33.715 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:33.713+0000 7f7639a00640 1 -- 172.21.15.32:0/943134207 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f76340b5080 msgr2=0x7f76341f8b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:56:33.715 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:33.713+0000 7f7639a00640 1 --2- 172.21.15.32:0/943134207 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f76340b5080 0x7f76341f8b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:33.715 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:33.713+0000 7f7639a00640 1 -- 172.21.15.32:0/943134207 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f76340b3920 con 0x7f761c003c10 2024-08-03T20:56:33.716 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:33.714+0000 7f7639a00640 1 --2- 172.21.15.32:0/943134207 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f761c003c10 0x7f76340b4b40 secure :-1 s=READY pgs=1372 cs=0 l=1 rev1=1 crypto rx=0x7f7628036040 tx=0x7f7628033670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:56:33.716 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:33.714+0000 7f7631600640 1 -- 172.21.15.32:0/943134207 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7628038040 con 0x7f761c003c10 2024-08-03T20:56:33.716 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:33.714+0000 7f7631600640 1 -- 172.21.15.32:0/943134207 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f7628004020 con 0x7f761c003c10 2024-08-03T20:56:33.716 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:33.714+0000 7f763ae00640 1 -- 172.21.15.32:0/943134207 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f76340b3c00 con 0x7f761c003c10 2024-08-03T20:56:33.716 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:33.714+0000 7f7631600640 1 -- 172.21.15.32:0/943134207 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7628004470 con 0x7f761c003c10 2024-08-03T20:56:33.716 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:33.714+0000 7f763ae00640 1 -- 172.21.15.32:0/943134207 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f76340b41c0 con 0x7f761c003c10 2024-08-03T20:56:33.716 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:33.715+0000 7f763ae00640 1 -- 172.21.15.32:0/943134207 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f76340ad410 con 0x7f761c003c10 2024-08-03T20:56:33.717 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:33.717+0000 7f7631600640 1 -- 172.21.15.32:0/943134207 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f7628045020 con 0x7f761c003c10 2024-08-03T20:56:33.718 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:33.718+0000 7f7631600640 1 --2- 172.21.15.32:0/943134207 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f7614077010 0x7f7614079520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:56:33.718 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:33.718+0000 7f7631600640 1 -- 172.21.15.32:0/943134207 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 6886+0+0 (secure 0 0 0) 0x7f76280b99f0 con 0x7f761c003c10 2024-08-03T20:56:33.718 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:33.718+0000 7f7639000640 1 --2- 172.21.15.32:0/943134207 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f7614077010 0x7f7614079520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:56:33.719 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:33.719+0000 7f7639000640 1 --2- 172.21.15.32:0/943134207 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f7614077010 0x7f7614079520 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7f7620006fd0 tx=0x7f7620000ef0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:56:33.722 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:33.722+0000 7f7631600640 1 -- 172.21.15.32:0/943134207 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7f76280826b0 con 0x7f761c003c10 2024-08-03T20:56:33.965 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:33.965+0000 7f763ae00640 1 -- 172.21.15.32:0/943134207 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f76340d0930 con 0x7f761c003c10 2024-08-03T20:56:33.966 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:33.966+0000 7f7631600640 1 -- 172.21.15.32:0/943134207 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 169 v169) ==== 95+0+5232 (secure 0 0 0) 0x7f7628081fb0 con 0x7f761c003c10 2024-08-03T20:56:33.966 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-03T20:56:33.966 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":169,"btime":"2024-08-03T20:56:30:124185+0000","default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6439,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6835/531080160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6834","nonce":531080160},{"type":"v1","addr":"172.21.15.118:6835","nonce":531080160}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":84},{"gid":6454,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6837/2796882319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6836","nonce":2796882319},{"type":"v1","addr":"172.21.15.118:6837","nonce":2796882319}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":86},{"gid":6813,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6834/2686933890","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":2686933890},{"type":"v1","addr":"172.21.15.32:6834","nonce":2686933890}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":85}],"filesystems":[{"mdsmap":{"epoch":169,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-08-03T20:54:26.950518+0000","modified":"2024-08-03T20:56:30.124176+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":97,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8250},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8250":{"gid":8250,"name":"a","rank":0,"incarnation":166,"state":"up:active","state_seq":4,"addr":"172.21.15.32:6835/2806168786","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6833","nonce":2806168786},{"type":"v1","addr":"172.21.15.32:6835","nonce":2806168786}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8250,"qdb_cluster":[8250]},"id":6}]} 2024-08-03T20:56:33.966 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 169 2024-08-03T20:56:33.968 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:33.968+0000 7f763ae00640 1 -- 172.21.15.32:0/943134207 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f7614077010 msgr2=0x7f7614079520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:56:33.969 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:33.968+0000 7f763ae00640 1 --2- 172.21.15.32:0/943134207 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f7614077010 0x7f7614079520 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7f7620006fd0 tx=0x7f7620000ef0 comp rx=0 tx=0).stop 2024-08-03T20:56:33.969 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:33.968+0000 7f763ae00640 1 -- 172.21.15.32:0/943134207 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f761c003c10 msgr2=0x7f76340b4b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:56:33.969 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:33.968+0000 7f763ae00640 1 --2- 172.21.15.32:0/943134207 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f761c003c10 0x7f76340b4b40 secure :-1 s=READY pgs=1372 cs=0 l=1 rev1=1 crypto rx=0x7f7628036040 tx=0x7f7628033670 comp rx=0 tx=0).stop 2024-08-03T20:56:33.969 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:33.968+0000 7f763ae00640 1 -- 172.21.15.32:0/943134207 shutdown_connections 2024-08-03T20:56:33.969 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:33.968+0000 7f763ae00640 1 --2- 172.21.15.32:0/943134207 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f7614077010 0x7f7614079520 unknown :-1 s=CLOSED pgs=564 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:33.969 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:33.968+0000 7f763ae00640 1 --2- 172.21.15.32:0/943134207 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f76340b5080 0x7f76341f8b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:33.969 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:33.968+0000 7f763ae00640 1 --2- 172.21.15.32:0/943134207 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f76341ef770 0x7f76341f90d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:33.969 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:33.968+0000 7f763ae00640 1 --2- 172.21.15.32:0/943134207 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f761c003c10 0x7f76340b4b40 unknown :-1 s=CLOSED pgs=1372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:33.969 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:33.968+0000 7f763ae00640 1 -- 172.21.15.32:0/943134207 >> 172.21.15.32:0/943134207 conn(0x7f76340b2120 msgr2=0x7f76340ad620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:56:33.969 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:33.968+0000 7f763ae00640 1 -- 172.21.15.32:0/943134207 shutdown_connections 2024-08-03T20:56:33.969 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:33.968+0000 7f763ae00640 1 -- 172.21.15.32:0/943134207 wait complete. 2024-08-03T20:56:33.994 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 169, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-08-03T20:54:26.950518+0000', 'modified': '2024-08-03T20:56:30.124176+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 97, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 8250}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8250': {'gid': 8250, 'name': 'a', 'rank': 0, 'incarnation': 166, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.32:6835/2806168786', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6833', 'nonce': 2806168786}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 2806168786}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [13], 'metadata_pool': 12, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8250, 'qdb_cluster': [8250]} 2024-08-03T20:56:33.995 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-08-03T20:56:33.995 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2024-08-03T20:56:34.226 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:34.225+0000 7f065dc00640 1 -- 172.21.15.32:0/3818739895 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f06580afcd0 msgr2=0x7f065815e5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:56:34.226 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:34.225+0000 7f065dc00640 1 --2- 172.21.15.32:0/3818739895 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f06580afcd0 0x7f065815e5c0 secure :-1 s=READY pgs=1374 cs=0 l=1 rev1=1 crypto rx=0x7f0648005fd0 tx=0x7f064802a540 comp rx=0 tx=0).stop 2024-08-03T20:56:34.226 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:34.225+0000 7f065dc00640 1 -- 172.21.15.32:0/3818739895 shutdown_connections 2024-08-03T20:56:34.226 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:34.225+0000 7f065dc00640 1 --2- 172.21.15.32:0/3818739895 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f065806c9e0 0x7f065806cdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:34.226 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:34.225+0000 7f065dc00640 1 --2- 172.21.15.32:0/3818739895 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f06580afcd0 0x7f065815e5c0 unknown :-1 s=CLOSED pgs=1374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:34.226 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:34.225+0000 7f065dc00640 1 -- 172.21.15.32:0/3818739895 >> 172.21.15.32:0/3818739895 conn(0x7f06580d0dd0 msgr2=0x7f06580d11e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:56:34.226 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:34.225+0000 7f065dc00640 1 -- 172.21.15.32:0/3818739895 shutdown_connections 2024-08-03T20:56:34.227 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:34.225+0000 7f065dc00640 1 -- 172.21.15.32:0/3818739895 wait complete. 2024-08-03T20:56:34.228 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:34.227+0000 7f065dc00640 1 Processor -- start 2024-08-03T20:56:34.228 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:34.227+0000 7f065dc00640 1 -- start start 2024-08-03T20:56:34.231 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:34.229+0000 7f065dc00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f065806c9e0 0x7f06580c4900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:56:34.231 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:34.229+0000 7f065dc00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f06580afcd0 0x7f06581f8e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:56:34.231 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:34.229+0000 7f065dc00640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f06580b0220 0x7f06580aebd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:56:34.231 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:34.229+0000 7f065dc00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f0658162970 con 0x7f06580afcd0 2024-08-03T20:56:34.231 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:34.229+0000 7f065dc00640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f06581627f0 con 0x7f065806c9e0 2024-08-03T20:56:34.231 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:34.229+0000 7f065dc00640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7f0658162af0 con 0x7f06580b0220 2024-08-03T20:56:34.231 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:34.229+0000 7f064fe00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f06580afcd0 0x7f06581f8e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:56:34.231 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:34.229+0000 7f064fe00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f06580afcd0 0x7f06581f8e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:37022/0 (socket says 172.21.15.32:37022) 2024-08-03T20:56:34.232 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:34.229+0000 7f064fe00640 1 -- 172.21.15.32:0/1397619297 learned_addr learned my addr 172.21.15.32:0/1397619297 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:56:34.232 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:34.230+0000 7f0657e00640 1 --2- 172.21.15.32:0/1397619297 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f065806c9e0 0x7f06580c4900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:56:34.232 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:34.230+0000 7f065d200640 1 --2- 172.21.15.32:0/1397619297 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f06580b0220 0x7f06580aebd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:56:34.232 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:34.230+0000 7f064fe00640 1 -- 172.21.15.32:0/1397619297 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f06580b0220 msgr2=0x7f06580aebd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:56:34.232 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:34.230+0000 7f064fe00640 1 --2- 172.21.15.32:0/1397619297 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f06580b0220 0x7f06580aebd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:34.232 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:34.230+0000 7f064fe00640 1 -- 172.21.15.32:0/1397619297 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f065806c9e0 msgr2=0x7f06580c4900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:56:34.232 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:34.230+0000 7f064fe00640 1 --2- 172.21.15.32:0/1397619297 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f065806c9e0 0x7f06580c4900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:34.232 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:34.230+0000 7f064fe00640 1 -- 172.21.15.32:0/1397619297 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f06581f5420 con 0x7f06580afcd0 2024-08-03T20:56:34.232 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:34.230+0000 7f065d200640 1 --2- 172.21.15.32:0/1397619297 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f06580b0220 0x7f06580aebd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-03T20:56:34.232 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:34.230+0000 7f0657e00640 1 --2- 172.21.15.32:0/1397619297 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f065806c9e0 0x7f06580c4900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-03T20:56:34.232 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:34.230+0000 7f064fe00640 1 --2- 172.21.15.32:0/1397619297 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f06580afcd0 0x7f06581f8e30 secure :-1 s=READY pgs=1375 cs=0 l=1 rev1=1 crypto rx=0x7f064000c3c0 tx=0x7f064000c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:56:34.232 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:34.230+0000 7f0654c00640 1 -- 172.21.15.32:0/1397619297 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0640017070 con 0x7f06580afcd0 2024-08-03T20:56:34.233 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:34.230+0000 7f0654c00640 1 -- 172.21.15.32:0/1397619297 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f06400035f0 con 0x7f06580afcd0 2024-08-03T20:56:34.233 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:34.230+0000 7f0654c00640 1 -- 172.21.15.32:0/1397619297 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0640003a40 con 0x7f06580afcd0 2024-08-03T20:56:34.233 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:34.230+0000 7f065dc00640 1 -- 172.21.15.32:0/1397619297 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f06581f56c0 con 0x7f06580afcd0 2024-08-03T20:56:34.233 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:34.230+0000 7f065dc00640 1 -- 172.21.15.32:0/1397619297 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f06581f5c80 con 0x7f06580afcd0 2024-08-03T20:56:34.233 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:34.233+0000 7f0654c00640 1 -- 172.21.15.32:0/1397619297 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f064001a070 con 0x7f06580afcd0 2024-08-03T20:56:34.234 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:34.234+0000 7f0654c00640 1 --2- 172.21.15.32:0/1397619297 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f0634077010 0x7f0634079520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:56:34.234 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:34.234+0000 7f0654c00640 1 -- 172.21.15.32:0/1397619297 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 6886+0+0 (secure 0 0 0) 0x7f0640098100 con 0x7f06580afcd0 2024-08-03T20:56:34.235 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:34.234+0000 7f0657e00640 1 --2- 172.21.15.32:0/1397619297 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f0634077010 0x7f0634079520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:56:34.235 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:34.235+0000 7f0666208740 1 Processor -- start 2024-08-03T20:56:34.235 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:34.235+0000 7f0666208740 1 -- start start 2024-08-03T20:56:34.235 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:34.235+0000 7f0657e00640 1 --2- 172.21.15.32:0/1397619297 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f0634077010 0x7f0634079520 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7f0648033000 tx=0x7f0648004730 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:56:34.237 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:34.236+0000 7f0666208740 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55b866608270 0x55b8669f52c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:56:34.238 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:34.236+0000 7f0666208740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55b8665df3b0 0x55b8669d7f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:56:34.238 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:34.236+0000 7f0666208740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b8665e7590 0x55b8669d84d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:56:34.238 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:34.236+0000 7f0666208740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55b866920a60 con 0x55b8665e7590 2024-08-03T20:56:34.238 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:34.236+0000 7f0666208740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x55b86690ff60 con 0x55b8665df3b0 2024-08-03T20:56:34.238 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:34.236+0000 7f0666208740 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x55b866913fa0 con 0x55b866608270 2024-08-03T20:56:34.238 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:34.237+0000 7f065d200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b8665e7590 0x55b8669d84d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:56:34.238 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:34.237+0000 7f065d200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b8665e7590 0x55b8669d84d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:37026/0 (socket says 172.21.15.32:37026) 2024-08-03T20:56:34.238 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:34.237+0000 7f065d200640 1 -- 172.21.15.32:0/4133789274 learned_addr learned my addr 172.21.15.32:0/4133789274 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:56:34.238 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:34.237+0000 7f064fe00640 1 --2- 172.21.15.32:0/4133789274 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55b8665df3b0 0x55b8669d7f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:56:34.238 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:34.237+0000 7f0657e00640 1 --2- 172.21.15.32:0/4133789274 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55b866608270 0x55b8669f52c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:56:34.238 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:34.237+0000 7f065d200640 1 -- 172.21.15.32:0/4133789274 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55b866608270 msgr2=0x55b8669f52c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:56:34.238 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:34.237+0000 7f065d200640 1 --2- 172.21.15.32:0/4133789274 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55b866608270 0x55b8669f52c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:34.238 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:34.237+0000 7f065d200640 1 -- 172.21.15.32:0/4133789274 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55b8665df3b0 msgr2=0x55b8669d7f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:56:34.239 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:34.237+0000 7f065d200640 1 --2- 172.21.15.32:0/4133789274 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55b8665df3b0 0x55b8669d7f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:34.239 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:34.237+0000 7f065d200640 1 -- 172.21.15.32:0/4133789274 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b866673850 con 0x55b8665e7590 2024-08-03T20:56:34.240 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:34.237+0000 7f065d200640 1 --2- 172.21.15.32:0/4133789274 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b8665e7590 0x55b8669d84d0 secure :-1 s=READY pgs=1376 cs=0 l=1 rev1=1 crypto rx=0x7f0650009e60 tx=0x7f065000f730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=67a14742fc1794e9 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:56:34.240 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:34.238+0000 7f064cc00640 1 -- 172.21.15.32:0/4133789274 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0650004d20 con 0x55b8665e7590 2024-08-03T20:56:34.240 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:34.238+0000 7f064cc00640 1 -- 172.21.15.32:0/4133789274 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f0650005020 con 0x55b8665e7590 2024-08-03T20:56:34.240 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:34.238+0000 7f064cc00640 1 -- 172.21.15.32:0/4133789274 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0650005410 con 0x55b8665e7590 2024-08-03T20:56:34.240 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:34.238+0000 7f0666208740 1 -- 172.21.15.32:0/4133789274 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b8665e7590 msgr2=0x55b8669d84d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:56:34.240 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:34.238+0000 7f0666208740 1 --2- 172.21.15.32:0/4133789274 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b8665e7590 0x55b8669d84d0 secure :-1 s=READY pgs=1376 cs=0 l=1 rev1=1 crypto rx=0x7f0650009e60 tx=0x7f065000f730 comp rx=0 tx=0).stop 2024-08-03T20:56:34.240 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:34.239+0000 7f0666208740 1 -- 172.21.15.32:0/4133789274 shutdown_connections 2024-08-03T20:56:34.241 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:34.239+0000 7f0666208740 1 --2- 172.21.15.32:0/4133789274 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55b8665df3b0 0x55b8669d7f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:34.241 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:34.239+0000 7f0666208740 1 --2- 172.21.15.32:0/4133789274 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b8665e7590 0x55b8669d84d0 unknown :-1 s=CLOSED pgs=1376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:34.241 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:34.239+0000 7f0666208740 1 --2- 172.21.15.32:0/4133789274 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55b866608270 0x55b8669f52c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:34.241 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:34.239+0000 7f0666208740 1 -- 172.21.15.32:0/4133789274 >> 172.21.15.32:0/4133789274 conn(0x55b8665fc3c0 msgr2=0x55b8669ae1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:56:34.241 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:34.239+0000 7f0666208740 1 -- 172.21.15.32:0/4133789274 shutdown_connections 2024-08-03T20:56:34.241 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:34.239+0000 7f0666208740 1 -- 172.21.15.32:0/4133789274 wait complete. 2024-08-03T20:56:34.241 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:34.240+0000 7f0666208740 1 -- start start 2024-08-03T20:56:34.241 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:34.240+0000 7f0666208740 1 Processor -- start 2024-08-03T20:56:34.244 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:34.241+0000 7f0666208740 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55b8665df3b0 0x55b866a06340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:56:34.244 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:34.241+0000 7f0666208740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b8665e7590 0x55b8669c66b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:56:34.244 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:34.241+0000 7f0666208740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55b866608270 0x55b8669ad4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:56:34.244 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:34.241+0000 7f0666208740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55b866682620 con 0x55b8665e7590 2024-08-03T20:56:34.244 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:34.241+0000 7f0666208740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x55b8668ad160 con 0x55b866608270 2024-08-03T20:56:34.245 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:34.241+0000 7f0666208740 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x55b8669f0bc0 con 0x55b8665df3b0 2024-08-03T20:56:34.245 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:34.241+0000 7f0666208740 10 client.0 _ll_register_callbacks cb 0x55b8668bc6b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-03T20:56:34.245 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:34.242+0000 7f064fe00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b8665e7590 0x55b8669c66b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:56:34.245 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:34.242+0000 7f064fe00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b8665e7590 0x55b8669c66b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:37040/0 (socket says 172.21.15.32:37040) 2024-08-03T20:56:34.245 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:34.242+0000 7f064fe00640 1 -- 172.21.15.32:0/2576719726 learned_addr learned my addr 172.21.15.32:0/2576719726 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:56:34.245 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:34.242+0000 7f0657e00640 1 --2- 172.21.15.32:0/2576719726 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55b8665df3b0 0x55b866a06340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:56:34.245 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:34.242+0000 7f065d200640 1 --2- 172.21.15.32:0/2576719726 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55b866608270 0x55b8669ad4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:56:34.245 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:34.242+0000 7f064fe00640 1 -- 172.21.15.32:0/2576719726 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55b8665df3b0 msgr2=0x55b866a06340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:56:34.245 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:34.242+0000 7f064fe00640 1 --2- 172.21.15.32:0/2576719726 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55b8665df3b0 0x55b866a06340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:34.245 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:34.242+0000 7f064fe00640 1 -- 172.21.15.32:0/2576719726 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55b866608270 msgr2=0x55b8669ad4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:56:34.245 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:34.242+0000 7f064fe00640 1 --2- 172.21.15.32:0/2576719726 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55b866608270 0x55b8669ad4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:34.245 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:34.242+0000 7f064fe00640 1 -- 172.21.15.32:0/2576719726 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b8666ea6b0 con 0x55b8665e7590 2024-08-03T20:56:34.245 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:34.242+0000 7f064fe00640 1 --2- 172.21.15.32:0/2576719726 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b8665e7590 0x55b8669c66b0 secure :-1 s=READY pgs=1377 cs=0 l=1 rev1=1 crypto rx=0x7f064000cd80 tx=0x7f064005ffd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5d35996ddf030a01 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:56:34.246 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:34.243+0000 7f0645600640 10 client.0 ms_handle_connect on v2:172.21.15.32:3300/0 2024-08-03T20:56:34.246 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:34.243+0000 7f0645600640 1 -- 172.21.15.32:0/2576719726 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0640060750 con 0x55b8665e7590 2024-08-03T20:56:34.246 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:34.243+0000 7f0645600640 1 -- 172.21.15.32:0/2576719726 <== mon.0 v2:172.21.15.32:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 6886+0+0 (secure 0 0 0) 0x7f064005f9b0 con 0x55b8665e7590 2024-08-03T20:56:34.246 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:34.243+0000 7f0666208740 1 -- 172.21.15.32:0/2576719726 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0650005020 con 0x55b8665e7590 2024-08-03T20:56:34.246 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:34.243+0000 7f0645600640 1 -- 172.21.15.32:0/2576719726 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_get_version_reply(handle=1 version=169) ==== 24+0+0 (secure 0 0 0) 0x7f0640064d50 con 0x55b8665e7590 2024-08-03T20:56:34.246 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:34.243+0000 7f0666208740 10 client.8280 fetch_fsmap learned FSMap version 169 2024-08-03T20:56:34.246 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:34.243+0000 7f0666208740 1 -- 172.21.15.32:0/2576719726 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,fsmap=169,monmap=2+}) -- 0x55b8669c6bf0 con 0x55b8665e7590 2024-08-03T20:56:34.246 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:34.243+0000 7f0645600640 1 -- 172.21.15.32:0/2576719726 <== mon.0 v2:172.21.15.32:3300/0 4 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f0650005020 con 0x55b8665e7590 2024-08-03T20:56:34.246 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:34.243+0000 7f0645600640 1 -- 172.21.15.32:0/2576719726 <== mon.0 v2:172.21.15.32:3300/0 5 ==== fsmap(e 169) ==== 3502+0+0 (secure 0 0 0) 0x7f064005c0c0 con 0x55b8665e7590 2024-08-03T20:56:34.246 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:34.243+0000 7f0666208740 10 client.8280 fetch_fsmap finished waiting for FSMap version 169 2024-08-03T20:56:34.246 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:34.243+0000 7f0666208740 10 client.8280 resolve_mds: resolved name 'a' to daemon mds.a 2024-08-03T20:56:34.246 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:34.243+0000 7f0666208740 20 client.8280 populate_metadata read hostname 'smithi032' 2024-08-03T20:56:34.246 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:34.244+0000 7f0666208740 1 --2- 172.21.15.32:0/2576719726 >> [v2:172.21.15.32:6833/2806168786,v1:172.21.15.32:6835/2806168786] conn(0x55b8665c6e40 0x55b8669ed520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:56:34.247 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:34.244+0000 7f0666208740 4 client.8280 mds_command: new command op to 8250 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-08-03T20:56:34.247 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:34.244+0000 7f0666208740 1 -- 172.21.15.32:0/2576719726 --> [v2:172.21.15.32:6833/2806168786,v1:172.21.15.32:6835/2806168786] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b8669229b0 con 0x55b8665c6e40 2024-08-03T20:56:34.247 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:34.244+0000 7f0657e00640 1 --2- 172.21.15.32:0/2576719726 >> [v2:172.21.15.32:6833/2806168786,v1:172.21.15.32:6835/2806168786] conn(0x55b8665c6e40 0x55b8669ed520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:56:34.248 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:34.244+0000 7f0657e00640 1 --2- 172.21.15.32:0/2576719726 >> [v2:172.21.15.32:6833/2806168786,v1:172.21.15.32:6835/2806168786] conn(0x55b8665c6e40 0x55b8669ed520 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=da46bb414a48dad server_cookie=6392bb77de5a056c in_seq=0 out_seq=0 2024-08-03T20:56:34.248 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:34.244+0000 7f0645600640 10 client.8280 ms_handle_connect on v2:172.21.15.32:6833/2806168786 2024-08-03T20:56:34.248 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:34.245+0000 7f0645600640 1 -- 172.21.15.32:0/2576719726 <== mds.0 v2:172.21.15.32:6833/2806168786 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55b8669229b0 con 0x55b8665c6e40 2024-08-03T20:56:34.248 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:34.245+0000 7f0645600640 10 client.8280 handle_command_reply: tid=0 2024-08-03T20:56:34.248 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:34.245+0000 7f0666208740 1 client.8280 shutdown 2024-08-03T20:56:34.248 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:34.246+0000 7f0666208740 1 -- 172.21.15.32:0/2576719726 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b8665e7590 msgr2=0x55b8669c66b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:56:34.248 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:34.246+0000 7f0666208740 1 --2- 172.21.15.32:0/2576719726 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b8665e7590 0x55b8669c66b0 secure :-1 s=READY pgs=1377 cs=0 l=1 rev1=1 crypto rx=0x7f064000cd80 tx=0x7f064005ffd0 comp rx=0 tx=0).stop 2024-08-03T20:56:34.248 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:34.246+0000 7f0666208740 1 -- 172.21.15.32:0/2576719726 shutdown_connections 2024-08-03T20:56:34.249 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:34.246+0000 7f0666208740 1 --2- 172.21.15.32:0/2576719726 >> [v2:172.21.15.32:6833/2806168786,v1:172.21.15.32:6835/2806168786] conn(0x55b8665c6e40 0x55b8669ed520 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:34.249 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:34.246+0000 7f0666208740 1 --2- 172.21.15.32:0/2576719726 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55b866608270 0x55b8669ad4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:34.249 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:34.246+0000 7f0666208740 1 --2- 172.21.15.32:0/2576719726 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b8665e7590 0x55b8669c66b0 unknown :-1 s=CLOSED pgs=1377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:34.249 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:34.246+0000 7f0666208740 1 --2- 172.21.15.32:0/2576719726 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55b8665df3b0 0x55b866a06340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:34.249 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:34.246+0000 7f0645600640 0 client.8280 ms_handle_reset on v2:172.21.15.32:6833/2806168786 2024-08-03T20:56:34.249 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:34.246+0000 7f0666208740 1 -- 172.21.15.32:0/2576719726 >> 172.21.15.32:0/2576719726 conn(0x55b8665fc3c0 msgr2=0x55b8669a74e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:56:34.249 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:34.246+0000 7f0666208740 1 -- 172.21.15.32:0/2576719726 shutdown_connections 2024-08-03T20:56:34.249 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:34.246+0000 7f0666208740 1 -- 172.21.15.32:0/2576719726 wait complete. 2024-08-03T20:56:34.249 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:34.246+0000 7f0666208740 20 client.8280 trim_cache size 0 max 16384 2024-08-03T20:56:34.266 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:34.266+0000 7f0666208740 1 Processor -- start 2024-08-03T20:56:34.266 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:34.266+0000 7f0666208740 1 -- start start 2024-08-03T20:56:34.268 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:34.267+0000 7f0666208740 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55b8665df3b0 0x55b8669af520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:56:34.268 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:34.267+0000 7f0666208740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b8665e7590 0x55b866a160a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:56:34.268 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:34.267+0000 7f0666208740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55b866608270 0x55b866a165e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:56:34.268 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:34.267+0000 7f0666208740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f0658201790 con 0x55b8665e7590 2024-08-03T20:56:34.268 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:34.267+0000 7f0666208740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f0658200b90 con 0x55b866608270 2024-08-03T20:56:34.268 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:34.267+0000 7f0666208740 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7f0658201140 con 0x55b8665df3b0 2024-08-03T20:56:34.268 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:34.267+0000 7f064fe00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b8665e7590 0x55b866a160a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:56:34.268 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:34.268+0000 7f064fe00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b8665e7590 0x55b866a160a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:37044/0 (socket says 172.21.15.32:37044) 2024-08-03T20:56:34.268 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:34.268+0000 7f064fe00640 1 -- 172.21.15.32:0/2704362870 learned_addr learned my addr 172.21.15.32:0/2704362870 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:56:34.269 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:34.268+0000 7f0657e00640 1 --2- 172.21.15.32:0/2704362870 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55b8665df3b0 0x55b8669af520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:56:34.269 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:34.268+0000 7f065d200640 1 --2- 172.21.15.32:0/2704362870 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55b866608270 0x55b866a165e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:56:34.270 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:34.268+0000 7f064fe00640 1 -- 172.21.15.32:0/2704362870 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55b8665df3b0 msgr2=0x55b8669af520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:56:34.270 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:34.268+0000 7f064fe00640 1 --2- 172.21.15.32:0/2704362870 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55b8665df3b0 0x55b8669af520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:34.270 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:34.268+0000 7f064fe00640 1 -- 172.21.15.32:0/2704362870 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55b866608270 msgr2=0x55b866a165e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:56:34.270 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:34.268+0000 7f064fe00640 1 --2- 172.21.15.32:0/2704362870 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55b866608270 0x55b866a165e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:34.270 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:34.268+0000 7f064fe00640 1 -- 172.21.15.32:0/2704362870 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b8669aad10 con 0x55b8665e7590 2024-08-03T20:56:34.270 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:34.268+0000 7f065d200640 1 --2- 172.21.15.32:0/2704362870 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55b866608270 0x55b866a165e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-03T20:56:34.270 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:34.268+0000 7f064fe00640 1 --2- 172.21.15.32:0/2704362870 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b8665e7590 0x55b866a160a0 secure :-1 s=READY pgs=1378 cs=0 l=1 rev1=1 crypto rx=0x7f064005c7d0 tx=0x7f0640067390 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3e09f961097fbd9f server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:56:34.270 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:34.269+0000 7f0647e00640 1 -- 172.21.15.32:0/2704362870 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0640060750 con 0x55b8665e7590 2024-08-03T20:56:34.271 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:34.269+0000 7f0647e00640 1 -- 172.21.15.32:0/2704362870 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f064005c0c0 con 0x55b8665e7590 2024-08-03T20:56:34.271 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:34.269+0000 7f0647e00640 1 -- 172.21.15.32:0/2704362870 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f064005fbb0 con 0x55b8665e7590 2024-08-03T20:56:34.271 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:34.269+0000 7f0666208740 1 -- 172.21.15.32:0/2704362870 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b8665e7590 msgr2=0x55b866a160a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:56:34.271 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:34.269+0000 7f0666208740 1 --2- 172.21.15.32:0/2704362870 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b8665e7590 0x55b866a160a0 secure :-1 s=READY pgs=1378 cs=0 l=1 rev1=1 crypto rx=0x7f064005c7d0 tx=0x7f0640067390 comp rx=0 tx=0).stop 2024-08-03T20:56:34.271 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:34.269+0000 7f0666208740 1 -- 172.21.15.32:0/2704362870 shutdown_connections 2024-08-03T20:56:34.271 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:34.269+0000 7f0666208740 1 --2- 172.21.15.32:0/2704362870 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55b866608270 0x55b866a165e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:34.271 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:34.269+0000 7f0666208740 1 --2- 172.21.15.32:0/2704362870 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b8665e7590 0x55b866a160a0 unknown :-1 s=CLOSED pgs=1378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:34.271 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:34.269+0000 7f0666208740 1 --2- 172.21.15.32:0/2704362870 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55b8665df3b0 0x55b8669af520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:34.271 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:34.269+0000 7f0666208740 1 -- 172.21.15.32:0/2704362870 >> 172.21.15.32:0/2704362870 conn(0x55b8665fc3c0 msgr2=0x55b8669ed520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:56:34.271 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:34.269+0000 7f0666208740 1 -- 172.21.15.32:0/2704362870 shutdown_connections 2024-08-03T20:56:34.271 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:34.269+0000 7f0666208740 1 -- 172.21.15.32:0/2704362870 wait complete. 2024-08-03T20:56:34.272 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:34.270+0000 7f0666208740 1 -- start start 2024-08-03T20:56:34.272 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:34.270+0000 7f0666208740 1 Processor -- start 2024-08-03T20:56:34.273 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:34.272+0000 7f0666208740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55b8665df3b0 0x55b8669c6410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:56:34.273 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:34.272+0000 7f0666208740 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55b8665e7590 0x55b866a72570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:56:34.273 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:34.272+0000 7f0666208740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b866608270 0x55b866a15e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:56:34.273 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:34.272+0000 7f0666208740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55b8669ef340 con 0x55b866608270 2024-08-03T20:56:34.273 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:34.272+0000 7f0666208740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x55b8669f0950 con 0x55b8665df3b0 2024-08-03T20:56:34.273 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:34.272+0000 7f0666208740 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x55b8669ef900 con 0x55b8665e7590 2024-08-03T20:56:34.273 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:34.272+0000 7f0666208740 10 client.0 _ll_register_callbacks cb 0x7f06400690b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-03T20:56:34.273 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:34.272+0000 7f065d200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b866608270 0x55b866a15e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:56:34.273 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:34.272+0000 7f065d200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b866608270 0x55b866a15e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:37048/0 (socket says 172.21.15.32:37048) 2024-08-03T20:56:34.273 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:34.272+0000 7f065d200640 1 -- 172.21.15.32:0/2496624847 learned_addr learned my addr 172.21.15.32:0/2496624847 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:56:34.274 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:34.272+0000 7f0657e00640 1 --2- 172.21.15.32:0/2496624847 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55b8665df3b0 0x55b8669c6410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:56:34.274 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:34.272+0000 7f064fe00640 1 --2- 172.21.15.32:0/2496624847 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55b8665e7590 0x55b866a72570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:56:34.276 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:34.273+0000 7f065d200640 1 -- 172.21.15.32:0/2496624847 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55b8665e7590 msgr2=0x55b866a72570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:56:34.276 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:34.273+0000 7f065d200640 1 --2- 172.21.15.32:0/2496624847 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55b8665e7590 0x55b866a72570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:34.276 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:34.273+0000 7f065d200640 1 -- 172.21.15.32:0/2496624847 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55b8665df3b0 msgr2=0x55b8669c6410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:56:34.276 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:34.273+0000 7f065d200640 1 --2- 172.21.15.32:0/2496624847 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55b8665df3b0 0x55b8669c6410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:34.277 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:34.273+0000 7f065d200640 1 -- 172.21.15.32:0/2496624847 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b866919220 con 0x55b866608270 2024-08-03T20:56:34.277 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:34.273+0000 7f065d200640 1 --2- 172.21.15.32:0/2496624847 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b866608270 0x55b866a15e30 secure :-1 s=READY pgs=1379 cs=0 l=1 rev1=1 crypto rx=0x7f0650016000 tx=0x7f0650009b40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1f4a9dceba52ec96 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:56:34.277 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:34.273+0000 7f0645600640 10 client.0 ms_handle_connect on v2:172.21.15.32:3300/0 2024-08-03T20:56:34.277 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:34.273+0000 7f0645600640 1 -- 172.21.15.32:0/2496624847 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06500051b0 con 0x55b866608270 2024-08-03T20:56:34.277 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:34.273+0000 7f0645600640 1 -- 172.21.15.32:0/2496624847 <== mon.0 v2:172.21.15.32:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 6886+0+0 (secure 0 0 0) 0x7f0650005350 con 0x55b866608270 2024-08-03T20:56:34.277 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:34.273+0000 7f0666208740 1 -- 172.21.15.32:0/2496624847 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f064005c0c0 con 0x55b866608270 2024-08-03T20:56:34.277 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:34.274+0000 7f0645600640 1 -- 172.21.15.32:0/2496624847 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_get_version_reply(handle=1 version=169) ==== 24+0+0 (secure 0 0 0) 0x7f0650005550 con 0x55b866608270 2024-08-03T20:56:34.277 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:34.274+0000 7f0666208740 10 client.8286 fetch_fsmap learned FSMap version 169 2024-08-03T20:56:34.277 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:34.274+0000 7f0666208740 1 -- 172.21.15.32:0/2496624847 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,fsmap=169,monmap=2+}) -- 0x55b866a72fd0 con 0x55b866608270 2024-08-03T20:56:34.277 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:34.274+0000 7f0645600640 1 -- 172.21.15.32:0/2496624847 <== mon.0 v2:172.21.15.32:3300/0 4 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f064005c0c0 con 0x55b866608270 2024-08-03T20:56:34.277 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:34.274+0000 7f0645600640 1 -- 172.21.15.32:0/2496624847 <== mon.0 v2:172.21.15.32:3300/0 5 ==== fsmap(e 169) ==== 3502+0+0 (secure 0 0 0) 0x7f06500056f0 con 0x55b866608270 2024-08-03T20:56:34.277 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:34.274+0000 7f0666208740 10 client.8286 fetch_fsmap finished waiting for FSMap version 169 2024-08-03T20:56:34.277 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:34.274+0000 7f0666208740 10 client.8286 resolve_mds: resolved name 'a' to daemon mds.a 2024-08-03T20:56:34.278 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:34.274+0000 7f0666208740 20 client.8286 populate_metadata read hostname 'smithi032' 2024-08-03T20:56:34.278 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:34.274+0000 7f0666208740 1 --2- 172.21.15.32:0/2496624847 >> [v2:172.21.15.32:6833/2806168786,v1:172.21.15.32:6835/2806168786] conn(0x55b8669abc10 0x55b8669ac020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:56:34.278 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:34.274+0000 7f0666208740 4 client.8286 mds_command: new command op to 8250 tid=0 multi_id=0 [{"prefix": "status"}] 2024-08-03T20:56:34.278 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:34.274+0000 7f0666208740 1 -- 172.21.15.32:0/2496624847 --> [v2:172.21.15.32:6833/2806168786,v1:172.21.15.32:6835/2806168786] -- command(tid 0: {"prefix": "status"}) -- 0x55b86691e450 con 0x55b8669abc10 2024-08-03T20:56:34.278 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:34.274+0000 7f0657e00640 1 --2- 172.21.15.32:0/2496624847 >> [v2:172.21.15.32:6833/2806168786,v1:172.21.15.32:6835/2806168786] conn(0x55b8669abc10 0x55b8669ac020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:56:34.278 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:34.275+0000 7f0657e00640 1 --2- 172.21.15.32:0/2496624847 >> [v2:172.21.15.32:6833/2806168786,v1:172.21.15.32:6835/2806168786] conn(0x55b8669abc10 0x55b8669ac020 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6a683745966a3ffb server_cookie=8cc9d13f786c56d0 in_seq=0 out_seq=0 2024-08-03T20:56:34.278 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:34.275+0000 7f0645600640 10 client.8286 ms_handle_connect on v2:172.21.15.32:6833/2806168786 2024-08-03T20:56:34.278 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:34.275+0000 7f0645600640 1 -- 172.21.15.32:0/2496624847 <== mds.0 v2:172.21.15.32:6833/2806168786 1 ==== command_reply(tid 0: 0 ) ==== 8+0+325 (crc 0 0 0) 0x55b86691e450 con 0x55b8669abc10 2024-08-03T20:56:34.278 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:34.275+0000 7f0645600640 10 client.8286 handle_command_reply: tid=0 2024-08-03T20:56:34.278 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:34.275+0000 7f0666208740 1 client.8286 shutdown 2024-08-03T20:56:34.278 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:34.275+0000 7f0666208740 1 -- 172.21.15.32:0/2496624847 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b866608270 msgr2=0x55b866a15e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:56:34.278 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:34.275+0000 7f0666208740 1 --2- 172.21.15.32:0/2496624847 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b866608270 0x55b866a15e30 secure :-1 s=READY pgs=1379 cs=0 l=1 rev1=1 crypto rx=0x7f0650016000 tx=0x7f0650009b40 comp rx=0 tx=0).stop 2024-08-03T20:56:34.279 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:34.275+0000 7f0666208740 1 -- 172.21.15.32:0/2496624847 shutdown_connections 2024-08-03T20:56:34.279 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:34.275+0000 7f0666208740 1 --2- 172.21.15.32:0/2496624847 >> [v2:172.21.15.32:6833/2806168786,v1:172.21.15.32:6835/2806168786] conn(0x55b8669abc10 0x55b8669ac020 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:34.279 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:34.275+0000 7f0666208740 1 --2- 172.21.15.32:0/2496624847 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b866608270 0x55b866a15e30 unknown :-1 s=CLOSED pgs=1379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:34.279 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:34.275+0000 7f0666208740 1 --2- 172.21.15.32:0/2496624847 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55b8665e7590 0x55b866a72570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:34.279 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:34.275+0000 7f0666208740 1 --2- 172.21.15.32:0/2496624847 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55b8665df3b0 0x55b8669c6410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:34.279 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:34.275+0000 7f0645600640 0 client.8286 ms_handle_reset on v2:172.21.15.32:6833/2806168786 2024-08-03T20:56:34.279 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:34.275+0000 7f0666208740 1 -- 172.21.15.32:0/2496624847 >> 172.21.15.32:0/2496624847 conn(0x55b8665fc3c0 msgr2=0x55b8669f6320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:56:34.279 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:34.276+0000 7f0666208740 1 -- 172.21.15.32:0/2496624847 shutdown_connections 2024-08-03T20:56:34.279 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:34.276+0000 7f0666208740 1 -- 172.21.15.32:0/2496624847 wait complete. 2024-08-03T20:56:34.279 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:34.276+0000 7f0666208740 20 client.8286 trim_cache size 0 max 16384 2024-08-03T20:56:34.279 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-08-03T20:56:34.280 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_fsid": "40ab141a-01d7-49b9-ae3c-2edec9876e95", 2024-08-03T20:56:34.280 INFO:teuthology.orchestra.run.smithi032.stdout: "whoami": 0, 2024-08-03T20:56:34.280 INFO:teuthology.orchestra.run.smithi032.stdout: "id": 8250, 2024-08-03T20:56:34.280 INFO:teuthology.orchestra.run.smithi032.stdout: "want_state": "up:active", 2024-08-03T20:56:34.280 INFO:teuthology.orchestra.run.smithi032.stdout: "state": "up:active", 2024-08-03T20:56:34.280 INFO:teuthology.orchestra.run.smithi032.stdout: "fs_name": "cephfs", 2024-08-03T20:56:34.280 INFO:teuthology.orchestra.run.smithi032.stdout: "rank_uptime": 7.159784814, 2024-08-03T20:56:34.280 INFO:teuthology.orchestra.run.smithi032.stdout: "mdsmap_epoch": 169, 2024-08-03T20:56:34.280 INFO:teuthology.orchestra.run.smithi032.stdout: "osdmap_epoch": 97, 2024-08-03T20:56:34.280 INFO:teuthology.orchestra.run.smithi032.stdout: "osdmap_epoch_barrier": 97, 2024-08-03T20:56:34.280 INFO:teuthology.orchestra.run.smithi032.stdout: "uptime": 8.3097264400000004 2024-08-03T20:56:34.280 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-08-03T20:56:34.281 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:34.276+0000 7f065dc00640 1 -- 172.21.15.32:0/1397619297 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f0634077010 msgr2=0x7f0634079520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:56:34.281 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:34.276+0000 7f065dc00640 1 --2- 172.21.15.32:0/1397619297 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f0634077010 0x7f0634079520 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7f0648033000 tx=0x7f0648004730 comp rx=0 tx=0).stop 2024-08-03T20:56:34.281 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:34.276+0000 7f065dc00640 1 -- 172.21.15.32:0/1397619297 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f06580afcd0 msgr2=0x7f06581f8e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:56:34.281 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:34.276+0000 7f065dc00640 1 --2- 172.21.15.32:0/1397619297 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f06580afcd0 0x7f06581f8e30 secure :-1 s=READY pgs=1375 cs=0 l=1 rev1=1 crypto rx=0x7f064000c3c0 tx=0x7f064000c880 comp rx=0 tx=0).stop 2024-08-03T20:56:34.282 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:34.276+0000 7f065dc00640 1 -- 172.21.15.32:0/1397619297 shutdown_connections 2024-08-03T20:56:34.282 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:34.276+0000 7f065dc00640 1 --2- 172.21.15.32:0/1397619297 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f0634077010 0x7f0634079520 unknown :-1 s=CLOSED pgs=565 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:34.282 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:34.276+0000 7f065dc00640 1 --2- 172.21.15.32:0/1397619297 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f06580b0220 0x7f06580aebd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:34.282 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:34.276+0000 7f065dc00640 1 --2- 172.21.15.32:0/1397619297 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f06580afcd0 0x7f06581f8e30 unknown :-1 s=CLOSED pgs=1375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:34.282 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:34.276+0000 7f065dc00640 1 --2- 172.21.15.32:0/1397619297 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f065806c9e0 0x7f06580c4900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:34.282 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:34.277+0000 7f065dc00640 1 -- 172.21.15.32:0/1397619297 >> 172.21.15.32:0/1397619297 conn(0x7f06580d0dd0 msgr2=0x7f065806fad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:56:34.282 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:34.277+0000 7f065dc00640 1 -- 172.21.15.32:0/1397619297 shutdown_connections 2024-08-03T20:56:34.282 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:34.277+0000 7f065dc00640 1 -- 172.21.15.32:0/1397619297 wait complete. 2024-08-03T20:56:34.302 DEBUG:tasks.ceph_test_case:wait_until_true: success in 5s and 0 retries 2024-08-03T20:56:34.302 INFO:tasks.cephfs.test_damage:Daemons came up after mutation 'Delete 400.00000000', proceeding to ls 2024-08-03T20:56:34.303 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-08-03T20:56:34.533 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:34.532+0000 7fc0de600640 1 -- 172.21.15.32:0/640733463 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fc0d806fb50 msgr2=0x7fc0d80ad190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:56:34.533 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:34.532+0000 7fc0de600640 1 --2- 172.21.15.32:0/640733463 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fc0d806fb50 0x7fc0d80ad190 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fc0d0005fd0 tx=0x7fc0d002a7a0 comp rx=0 tx=0).stop 2024-08-03T20:56:34.534 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:34.532+0000 7fc0de600640 1 -- 172.21.15.32:0/640733463 shutdown_connections 2024-08-03T20:56:34.534 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:34.532+0000 7fc0de600640 1 --2- 172.21.15.32:0/640733463 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fc0d806fb50 0x7fc0d80ad190 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:34.534 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:34.532+0000 7fc0de600640 1 --2- 172.21.15.32:0/640733463 >> v2:172.21.15.118:3300/0 conn(0x7fc0d80af020 0x7fc0d80742b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:34.534 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:34.532+0000 7fc0de600640 1 -- 172.21.15.32:0/640733463 >> 172.21.15.32:0/640733463 conn(0x7fc0d80b2120 msgr2=0x7fc0d80cfd60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:56:34.534 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:34.533+0000 7fc0de600640 1 -- 172.21.15.32:0/640733463 shutdown_connections 2024-08-03T20:56:34.534 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:34.533+0000 7fc0de600640 1 -- 172.21.15.32:0/640733463 wait complete. 2024-08-03T20:56:34.535 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:34.535+0000 7fc0de600640 1 Processor -- start 2024-08-03T20:56:34.535 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:34.535+0000 7fc0de600640 1 -- start start 2024-08-03T20:56:34.538 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:34.536+0000 7fc0de600640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc0d806fb50 0x7fc0d809d890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:56:34.538 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:34.536+0000 7fc0de600640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fc0d80acd80 0x7fc0d8156140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:56:34.539 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:34.536+0000 7fc0de600640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc0d80af020 0x7fc0d80b49f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:56:34.539 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:34.536+0000 7fc0de600640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fc0d81601e0 con 0x7fc0d80af020 2024-08-03T20:56:34.539 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:34.536+0000 7fc0de600640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7fc0d8160060 con 0x7fc0d806fb50 2024-08-03T20:56:34.539 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:34.536+0000 7fc0de600640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7fc0d8160360 con 0x7fc0d80acd80 2024-08-03T20:56:34.539 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:34.537+0000 7fc0ddc00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc0d80af020 0x7fc0d80b49f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:56:34.539 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:34.537+0000 7fc0ddc00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc0d80af020 0x7fc0d80b49f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:37050/0 (socket says 172.21.15.32:37050) 2024-08-03T20:56:34.539 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:34.537+0000 7fc0ddc00640 1 -- 172.21.15.32:0/515490043 learned_addr learned my addr 172.21.15.32:0/515490043 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:56:34.539 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:34.537+0000 7fc0dd200640 1 --2- 172.21.15.32:0/515490043 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc0d806fb50 0x7fc0d809d890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:56:34.539 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:34.537+0000 7fc0d7e00640 1 --2- 172.21.15.32:0/515490043 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fc0d80acd80 0x7fc0d8156140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:56:34.539 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:34.537+0000 7fc0ddc00640 1 -- 172.21.15.32:0/515490043 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fc0d80acd80 msgr2=0x7fc0d8156140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:56:34.539 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:34.537+0000 7fc0ddc00640 1 --2- 172.21.15.32:0/515490043 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fc0d80acd80 0x7fc0d8156140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:34.539 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:34.537+0000 7fc0ddc00640 1 -- 172.21.15.32:0/515490043 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc0d806fb50 msgr2=0x7fc0d809d890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:56:34.539 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:34.537+0000 7fc0ddc00640 1 --2- 172.21.15.32:0/515490043 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc0d806fb50 0x7fc0d809d890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:34.540 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:34.537+0000 7fc0ddc00640 1 -- 172.21.15.32:0/515490043 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc0d809d570 con 0x7fc0d80af020 2024-08-03T20:56:34.540 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:34.537+0000 7fc0ddc00640 1 --2- 172.21.15.32:0/515490043 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc0d80af020 0x7fc0d80b49f0 secure :-1 s=READY pgs=1381 cs=0 l=1 rev1=1 crypto rx=0x7fc0d0005fd0 tx=0x7fc0d0032be0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:56:34.540 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:34.538+0000 7fc0d4c00640 1 -- 172.21.15.32:0/515490043 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0d0041020 con 0x7fc0d80af020 2024-08-03T20:56:34.540 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:34.538+0000 7fc0d4c00640 1 -- 172.21.15.32:0/515490043 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fc0d0004040 con 0x7fc0d80af020 2024-08-03T20:56:34.540 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:34.538+0000 7fc0d4c00640 1 -- 172.21.15.32:0/515490043 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0d0036040 con 0x7fc0d80af020 2024-08-03T20:56:34.540 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:34.538+0000 7fc0de600640 1 -- 172.21.15.32:0/515490043 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc0d81f8f70 con 0x7fc0d80af020 2024-08-03T20:56:34.540 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:34.538+0000 7fc0de600640 1 -- 172.21.15.32:0/515490043 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc0d81f9530 con 0x7fc0d80af020 2024-08-03T20:56:34.540 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:34.539+0000 7fc0de600640 1 -- 172.21.15.32:0/515490043 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc0d80ad190 con 0x7fc0d80af020 2024-08-03T20:56:34.541 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:34.541+0000 7fc0d4c00640 1 -- 172.21.15.32:0/515490043 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7fc0d0044020 con 0x7fc0d80af020 2024-08-03T20:56:34.542 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:34.541+0000 7fc0d4c00640 1 --2- 172.21.15.32:0/515490043 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fc0b8077040 0x7fc0b8079550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:56:34.542 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:34.542+0000 7fc0d4c00640 1 -- 172.21.15.32:0/515490043 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 6886+0+0 (secure 0 0 0) 0x7fc0d00b9da0 con 0x7fc0d80af020 2024-08-03T20:56:34.542 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:34.542+0000 7fc0dd200640 1 --2- 172.21.15.32:0/515490043 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fc0b8077040 0x7fc0b8079550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:56:34.543 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:34.543+0000 7fc0dd200640 1 --2- 172.21.15.32:0/515490043 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fc0b8077040 0x7fc0b8079550 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7fc0cc009ac0 tx=0x7fc0cc005f90 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:56:34.546 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:34.546+0000 7fc0d4c00640 1 -- 172.21.15.32:0/515490043 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7fc0d004b030 con 0x7fc0d80af020 2024-08-03T20:56:34.793 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:34.792+0000 7fc0de600640 1 -- 172.21.15.32:0/515490043 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7fc0d81f8d00 con 0x7fc0d80af020 2024-08-03T20:56:34.793 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:34.793+0000 7fc0d4c00640 1 -- 172.21.15.32:0/515490043 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v169) ==== 53+0+73 (secure 0 0 0) 0x7fc0d809d570 con 0x7fc0d80af020 2024-08-03T20:56:34.793 INFO:teuthology.orchestra.run.smithi032.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-08-03T20:56:34.796 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:34.795+0000 7fc0de600640 1 -- 172.21.15.32:0/515490043 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fc0b8077040 msgr2=0x7fc0b8079550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:56:34.796 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:34.795+0000 7fc0de600640 1 --2- 172.21.15.32:0/515490043 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fc0b8077040 0x7fc0b8079550 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7fc0cc009ac0 tx=0x7fc0cc005f90 comp rx=0 tx=0).stop 2024-08-03T20:56:34.796 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:34.795+0000 7fc0de600640 1 -- 172.21.15.32:0/515490043 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc0d80af020 msgr2=0x7fc0d80b49f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:56:34.796 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:34.795+0000 7fc0de600640 1 --2- 172.21.15.32:0/515490043 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc0d80af020 0x7fc0d80b49f0 secure :-1 s=READY pgs=1381 cs=0 l=1 rev1=1 crypto rx=0x7fc0d0005fd0 tx=0x7fc0d0032be0 comp rx=0 tx=0).stop 2024-08-03T20:56:34.796 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:34.795+0000 7fc0de600640 1 -- 172.21.15.32:0/515490043 shutdown_connections 2024-08-03T20:56:34.796 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:34.795+0000 7fc0de600640 1 --2- 172.21.15.32:0/515490043 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fc0b8077040 0x7fc0b8079550 unknown :-1 s=CLOSED pgs=566 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:34.796 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:34.795+0000 7fc0de600640 1 --2- 172.21.15.32:0/515490043 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc0d80af020 0x7fc0d80b49f0 unknown :-1 s=CLOSED pgs=1381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:34.796 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:34.795+0000 7fc0de600640 1 --2- 172.21.15.32:0/515490043 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fc0d80acd80 0x7fc0d8156140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:34.796 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:34.795+0000 7fc0de600640 1 --2- 172.21.15.32:0/515490043 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc0d806fb50 0x7fc0d809d890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:34.796 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:34.795+0000 7fc0de600640 1 -- 172.21.15.32:0/515490043 >> 172.21.15.32:0/515490043 conn(0x7fc0d80b2120 msgr2=0x7fc0d80cc7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:56:34.796 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:34.795+0000 7fc0de600640 1 -- 172.21.15.32:0/515490043 shutdown_connections 2024-08-03T20:56:34.797 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:34.795+0000 7fc0de600640 1 -- 172.21.15.32:0/515490043 wait complete. 2024-08-03T20:56:34.821 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-08-03T20:56:34.821 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-08-03T20:56:34.821 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi032.front.sepia.ceph.com 2024-08-03T20:56:34.821 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-08-03T20:56:34.821 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-08-03T20:56:34.821 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-08-03T20:56:34.821 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2024-08-03T20:56:34.822 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-08-03T20:56:34.822 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-08-03T20:56:34.822 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-08-03T20:56:34.822 DEBUG:teuthology.orchestra.run.smithi032:> ip addr 2024-08-03T20:56:34.858 INFO:teuthology.orchestra.run.smithi032.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-08-03T20:56:34.858 INFO:teuthology.orchestra.run.smithi032.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-08-03T20:56:34.858 INFO:teuthology.orchestra.run.smithi032.stdout: inet 127.0.0.1/8 scope host lo 2024-08-03T20:56:34.858 INFO:teuthology.orchestra.run.smithi032.stdout: valid_lft forever preferred_lft forever 2024-08-03T20:56:34.858 INFO:teuthology.orchestra.run.smithi032.stdout: inet6 ::1/128 scope host 2024-08-03T20:56:34.858 INFO:teuthology.orchestra.run.smithi032.stdout: valid_lft forever preferred_lft forever 2024-08-03T20:56:34.858 INFO:teuthology.orchestra.run.smithi032.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-08-03T20:56:34.858 INFO:teuthology.orchestra.run.smithi032.stdout: link/ether 0c:c4:7a:6b:fd:0c brd ff:ff:ff:ff:ff:ff 2024-08-03T20:56:34.858 INFO:teuthology.orchestra.run.smithi032.stdout: altname enp7s0f0 2024-08-03T20:56:34.858 INFO:teuthology.orchestra.run.smithi032.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-08-03T20:56:34.859 INFO:teuthology.orchestra.run.smithi032.stdout: link/ether 0c:c4:7a:6b:fd:0d brd ff:ff:ff:ff:ff:ff 2024-08-03T20:56:34.859 INFO:teuthology.orchestra.run.smithi032.stdout: altname enp7s0f1 2024-08-03T20:56:34.859 INFO:teuthology.orchestra.run.smithi032.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-08-03T20:56:34.859 INFO:teuthology.orchestra.run.smithi032.stdout: link/ether ac:1f:6b:f8:26:02 brd ff:ff:ff:ff:ff:ff 2024-08-03T20:56:34.859 INFO:teuthology.orchestra.run.smithi032.stdout: altname enp1s0f0 2024-08-03T20:56:34.859 INFO:teuthology.orchestra.run.smithi032.stdout: inet 172.21.15.32/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-08-03T20:56:34.859 INFO:teuthology.orchestra.run.smithi032.stdout: valid_lft 41961sec preferred_lft 41961sec 2024-08-03T20:56:34.859 INFO:teuthology.orchestra.run.smithi032.stdout: inet6 fe80::ae1f:6bff:fef8:2602/64 scope link noprefixroute 2024-08-03T20:56:34.859 INFO:teuthology.orchestra.run.smithi032.stdout: valid_lft forever preferred_lft forever 2024-08-03T20:56:34.859 INFO:teuthology.orchestra.run.smithi032.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-08-03T20:56:34.859 INFO:teuthology.orchestra.run.smithi032.stdout: link/ether ac:1f:6b:f8:26:03 brd ff:ff:ff:ff:ff:ff 2024-08-03T20:56:34.859 INFO:teuthology.orchestra.run.smithi032.stdout: altname enp1s0f1 2024-08-03T20:56:34.859 INFO:teuthology.orchestra.run.smithi032.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-08-03T20:56:34.859 INFO:teuthology.orchestra.run.smithi032.stdout: link/ether 0a:23:2b:ca:2a:93 brd ff:ff:ff:ff:ff:ff 2024-08-03T20:56:34.859 INFO:teuthology.orchestra.run.smithi032.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-08-03T20:56:34.860 INFO:teuthology.orchestra.run.smithi032.stdout: valid_lft forever preferred_lft forever 2024-08-03T20:56:34.860 INFO:teuthology.orchestra.run.smithi032.stdout: inet6 fe80::d804:7cff:fed9:7ba4/64 scope link 2024-08-03T20:56:34.860 INFO:teuthology.orchestra.run.smithi032.stdout: valid_lft forever preferred_lft forever 2024-08-03T20:56:34.860 INFO:teuthology.orchestra.run.smithi032.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-08-03T20:56:34.860 INFO:teuthology.orchestra.run.smithi032.stdout: link/ether 22:86:28:30:4e:62 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-08-03T20:56:34.860 INFO:teuthology.orchestra.run.smithi032.stdout: inet6 fe80::2086:28ff:fe30:4e62/64 scope link 2024-08-03T20:56:34.860 INFO:teuthology.orchestra.run.smithi032.stdout: valid_lft forever preferred_lft forever 2024-08-03T20:56:34.860 INFO:teuthology.orchestra.run.smithi032.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-08-03T20:56:34.860 INFO:teuthology.orchestra.run.smithi032.stdout: link/ether 0a:23:2b:ca:2a:93 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-08-03T20:56:34.860 INFO:teuthology.orchestra.run.smithi032.stdout: inet6 fe80::823:2bff:feca:2a93/64 scope link 2024-08-03T20:56:34.860 INFO:teuthology.orchestra.run.smithi032.stdout: valid_lft forever preferred_lft forever 2024-08-03T20:56:34.861 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-08-03T20:56:34.861 DEBUG:teuthology.orchestra.run.smithi032:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-08-03T20:56:34.951 INFO:teuthology.orchestra.run.smithi032.stdout:1 2024-08-03T20:56:34.954 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-08-03T20:56:34.955 DEBUG:teuthology.orchestra.run.smithi032:> route 2024-08-03T20:56:34.991 INFO:teuthology.orchestra.run.smithi032.stdout:Kernel IP routing table 2024-08-03T20:56:34.991 INFO:teuthology.orchestra.run.smithi032.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-08-03T20:56:34.991 INFO:teuthology.orchestra.run.smithi032.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-08-03T20:56:34.991 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-08-03T20:56:34.991 INFO:teuthology.orchestra.run.smithi032.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-08-03T20:56:34.991 DEBUG:teuthology.orchestra.run.smithi032:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-08-03T20:56:34.992 DEBUG:teuthology.orchestra.run.smithi032:> set -e 2024-08-03T20:56:34.992 DEBUG:teuthology.orchestra.run.smithi032:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-08-03T20:56:34.992 DEBUG:teuthology.orchestra.run.smithi032:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-08-03T20:56:34.992 DEBUG:teuthology.orchestra.run.smithi032:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-08-03T20:56:34.992 DEBUG:teuthology.orchestra.run.smithi032:> ') 2024-08-03T20:56:35.110 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:35 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-08-03T20:56:35.184 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:35 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-08-03T20:56:35.193 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-08-03T20:56:35.193 DEBUG:teuthology.orchestra.run.smithi032:> ip netns list 2024-08-03T20:56:35.228 INFO:teuthology.orchestra.run.smithi032.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-08-03T20:56:35.228 INFO:teuthology.orchestra.run.smithi032.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-08-03T20:56:35.228 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-08-03T20:56:35.228 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-08-03T20:56:35.228 DEBUG:teuthology.orchestra.run.smithi032:> sudo ip link set brx.0 up 2024-08-03T20:56:35.324 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-08-03T20:56:35.325 DEBUG:teuthology.orchestra.run.smithi032:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-08-03T20:56:35.360 INFO:teuthology.orchestra.run.smithi032.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-08-03T20:56:35.360 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-08-03T20:56:35.360 DEBUG:teuthology.orchestra.run.smithi032:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-08-03T20:56:35.435 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2024-08-03T20:56:35.435 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-08-03T20:56:35.435 DEBUG:teuthology.orchestra.run.smithi032:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-08-03T20:56:35.633 INFO:teuthology.orchestra.run.smithi032.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-08-03T20:56:35.633 INFO:teuthology.orchestra.run.smithi032.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-08-03T20:56:35.633 INFO:teuthology.orchestra.run.smithi032.stdout:invalid new device string format 2024-08-03T20:56:35.653 INFO:teuthology.orchestra.run.smithi032.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-08-03T20:56:35.653 INFO:teuthology.orchestra.run.smithi032.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-08-03T20:56:35.653 INFO:teuthology.orchestra.run.smithi032.stdout:invalid new device string format 2024-08-03T20:56:35.653 INFO:teuthology.orchestra.run.smithi032.stdout:mount.ceph: resolved to: "172.21.15.32:3300,172.21.15.118:3300,172.21.15.118:3301" 2024-08-03T20:56:35.654 INFO:teuthology.orchestra.run.smithi032.stdout:mount.ceph: switching to using v1 address with old syntax 2024-08-03T20:56:35.686 INFO:teuthology.orchestra.run.smithi032.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-08-03T20:56:35.686 INFO:teuthology.orchestra.run.smithi032.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-08-03T20:56:35.686 INFO:teuthology.orchestra.run.smithi032.stdout:invalid new device string format 2024-08-03T20:56:35.686 INFO:teuthology.orchestra.run.smithi032.stdout:mount.ceph: resolved to: "172.21.15.32:3300,172.21.15.118:3300,172.21.15.118:3301" 2024-08-03T20:56:35.686 INFO:teuthology.orchestra.run.smithi032.stdout:mount.ceph: switching to using v1 address with old syntax 2024-08-03T20:56:35.686 INFO:teuthology.orchestra.run.smithi032.stdout:mount.ceph: resolved to: "172.21.15.32:6789,172.21.15.118:6789,172.21.15.118:6790" 2024-08-03T20:56:35.687 INFO:teuthology.orchestra.run.smithi032.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.32:6789,172.21.15.118:6789,172.21.15.118:6790:/ 2024-08-03T20:56:35.687 INFO:teuthology.orchestra.run.smithi032.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,key=0,fsid=40ab141a-01d7-49b9-ae3c-2edec9876e95" will pass to kernel 2024-08-03T20:56:35.688 INFO:teuthology.orchestra.run.smithi032.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2024-08-03T20:56:35.688 INFO:teuthology.orchestra.run.smithi032.stdout: You just mounted a file system that supports labels which does not 2024-08-03T20:56:35.688 INFO:teuthology.orchestra.run.smithi032.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-08-03T20:56:35.688 INFO:teuthology.orchestra.run.smithi032.stdout: applications will generate AVC messages and not be allowed access to 2024-08-03T20:56:35.688 INFO:teuthology.orchestra.run.smithi032.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-08-03T20:56:35.691 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-08-03T20:56:35.691 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-08-03T20:56:35.691 DEBUG:teuthology.orchestra.run.smithi032:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-08-03T20:56:35.747 DEBUG:teuthology.orchestra.run.smithi032:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-08-03T20:56:35.827 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:35 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-08-03T20:56:35.830 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:35 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-08-03T20:56:35.838 INFO:teuthology.orchestra.run.smithi032.stdout:client8295 2024-08-03T20:56:35.838 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-08-03T20:56:35.838 DEBUG:teuthology.orchestra.run.smithi032:> dd if=/proc/self/mounts of=/dev/stdout 2024-08-03T20:56:35.874 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-08-03T20:56:35.874 DEBUG:teuthology.orchestra.run.smithi032:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls -R .) 2024-08-03T20:56:35.918 INFO:teuthology.orchestra.run:waiting for 20 2024-08-03T20:56:35.951 INFO:teuthology.orchestra.run.smithi032.stdout:.: 2024-08-03T20:56:35.952 INFO:teuthology.orchestra.run.smithi032.stdout:subdir 2024-08-03T20:56:35.952 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-03T20:56:35.952 INFO:teuthology.orchestra.run.smithi032.stdout:./subdir: 2024-08-03T20:56:35.952 INFO:teuthology.orchestra.run.smithi032.stdout:sixmegs 2024-08-03T20:56:35.953 INFO:tasks.cephfs.test_damage:Result: Mutation 'Delete 400.00000000' did not caused DAMAGED state 2024-08-03T20:56:35.953 INFO:tasks.cephfs.test_damage:Applying mutation 'Delete 100.00000000.inode' 2024-08-03T20:56:35.953 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-08-03T20:56:35.953 DEBUG:teuthology.orchestra.run.smithi032:> dd if=/proc/self/mounts of=/dev/stdout 2024-08-03T20:56:36.029 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-08-03T20:56:36.029 DEBUG:teuthology.orchestra.run.smithi032:> dd if=/proc/self/mounts of=/dev/stdout 2024-08-03T20:56:36.105 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-08-03T20:56:36.374 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:36.373+0000 7fecf8c00640 1 -- 172.21.15.32:0/2107818983 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fecf40acd80 msgr2=0x7fecf40ad190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:56:36.374 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:36.373+0000 7fecf8c00640 1 --2- 172.21.15.32:0/2107818983 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fecf40acd80 0x7fecf40ad190 secure :-1 s=READY pgs=1383 cs=0 l=1 rev1=1 crypto rx=0x7fecdc005fd0 tx=0x7fecdc02a540 comp rx=0 tx=0).stop 2024-08-03T20:56:36.374 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:36.373+0000 7fecf8c00640 1 -- 172.21.15.32:0/2107818983 shutdown_connections 2024-08-03T20:56:36.374 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:36.373+0000 7fecf8c00640 1 --2- 172.21.15.32:0/2107818983 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fecf40acd80 0x7fecf40ad190 unknown :-1 s=CLOSED pgs=1383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:36.374 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:36.373+0000 7fecf8c00640 1 --2- 172.21.15.32:0/2107818983 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fecf40af020 0x7fecf40742b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:36.374 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:36.373+0000 7fecf8c00640 1 -- 172.21.15.32:0/2107818983 >> 172.21.15.32:0/2107818983 conn(0x7fecf40b2120 msgr2=0x7fecf40cfd60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:56:36.374 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:36.373+0000 7fecf8c00640 1 -- 172.21.15.32:0/2107818983 shutdown_connections 2024-08-03T20:56:36.375 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:36.373+0000 7fecf8c00640 1 -- 172.21.15.32:0/2107818983 wait complete. 2024-08-03T20:56:36.375 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:36.375+0000 7fecf8c00640 1 Processor -- start 2024-08-03T20:56:36.376 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:36.375+0000 7fecf8c00640 1 -- start start 2024-08-03T20:56:36.378 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:36.377+0000 7fecf8c00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fecf406f440 0x7fecf40b2c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:56:36.378 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:36.377+0000 7fecf8c00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fecf40acd80 0x7fecf415ec60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:56:36.378 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:36.377+0000 7fecf8c00640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fecf40af020 0x7fecf41f6170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:56:36.378 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:36.377+0000 7fecf8c00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fecf4160260 con 0x7fecf406f440 2024-08-03T20:56:36.378 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:36.377+0000 7fecf8c00640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7fecf41600e0 con 0x7fecf40acd80 2024-08-03T20:56:36.378 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:36.377+0000 7fecf8c00640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7fecf41603e0 con 0x7fecf40af020 2024-08-03T20:56:36.378 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:36.377+0000 7fecf3400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fecf406f440 0x7fecf40b2c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:56:36.378 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:36.377+0000 7fecf3400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fecf406f440 0x7fecf40b2c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:37066/0 (socket says 172.21.15.32:37066) 2024-08-03T20:56:36.379 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:36.377+0000 7fecf3400640 1 -- 172.21.15.32:0/1098880922 learned_addr learned my addr 172.21.15.32:0/1098880922 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:56:36.379 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:36.377+0000 7feceaa00640 1 --2- 172.21.15.32:0/1098880922 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fecf40acd80 0x7fecf415ec60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:56:36.379 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:36.377+0000 7fecf3e00640 1 --2- 172.21.15.32:0/1098880922 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fecf40af020 0x7fecf41f6170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:56:36.379 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:36.377+0000 7fecf3400640 1 -- 172.21.15.32:0/1098880922 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fecf40af020 msgr2=0x7fecf41f6170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:56:36.379 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:36.377+0000 7fecf3400640 1 --2- 172.21.15.32:0/1098880922 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fecf40af020 0x7fecf41f6170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:36.379 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:36.377+0000 7fecf3400640 1 -- 172.21.15.32:0/1098880922 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fecf40acd80 msgr2=0x7fecf415ec60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:56:36.379 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:36.377+0000 7fecf3400640 1 --2- 172.21.15.32:0/1098880922 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fecf40acd80 0x7fecf415ec60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:36.379 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:36.377+0000 7fecf3400640 1 -- 172.21.15.32:0/1098880922 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fecf40c61d0 con 0x7fecf406f440 2024-08-03T20:56:36.379 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:36.377+0000 7fecf3e00640 1 --2- 172.21.15.32:0/1098880922 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fecf40af020 0x7fecf41f6170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-03T20:56:36.379 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:36.378+0000 7fecf3400640 1 --2- 172.21.15.32:0/1098880922 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fecf406f440 0x7fecf40b2c30 secure :-1 s=READY pgs=1384 cs=0 l=1 rev1=1 crypto rx=0x7fece400d700 tx=0x7fece400dbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:56:36.380 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:36.378+0000 7fecebe00640 1 -- 172.21.15.32:0/1098880922 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fece4017070 con 0x7fecf406f440 2024-08-03T20:56:36.380 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:36.378+0000 7fecebe00640 1 -- 172.21.15.32:0/1098880922 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fece4005310 con 0x7fecf406f440 2024-08-03T20:56:36.380 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:36.378+0000 7fecebe00640 1 -- 172.21.15.32:0/1098880922 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fece4009820 con 0x7fecf406f440 2024-08-03T20:56:36.380 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:36.378+0000 7fecf8c00640 1 -- 172.21.15.32:0/1098880922 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fecf40c64b0 con 0x7fecf406f440 2024-08-03T20:56:36.380 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:36.378+0000 7fecf8c00640 1 -- 172.21.15.32:0/1098880922 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fecf40b3200 con 0x7fecf406f440 2024-08-03T20:56:36.380 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:36.379+0000 7fecf8c00640 1 -- 172.21.15.32:0/1098880922 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fecf40cab60 con 0x7fecf406f440 2024-08-03T20:56:36.381 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:36.381+0000 7fecebe00640 1 -- 172.21.15.32:0/1098880922 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7fece40099c0 con 0x7fecf406f440 2024-08-03T20:56:36.382 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:36.382+0000 7fecebe00640 1 --2- 172.21.15.32:0/1098880922 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fecd0077010 0x7fecd0079520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:56:36.382 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:36.382+0000 7fecebe00640 1 -- 172.21.15.32:0/1098880922 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 6886+0+0 (secure 0 0 0) 0x7fece4098d70 con 0x7fecf406f440 2024-08-03T20:56:36.382 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:36.382+0000 7feceaa00640 1 --2- 172.21.15.32:0/1098880922 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fecd0077010 0x7fecd0079520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:56:36.383 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:36.383+0000 7feceaa00640 1 --2- 172.21.15.32:0/1098880922 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fecd0077010 0x7fecd0079520 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7fecdc033000 tx=0x7fecdc004730 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:56:36.386 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:36.386+0000 7fecebe00640 1 -- 172.21.15.32:0/1098880922 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7fecf40cab60 con 0x7fecf406f440 2024-08-03T20:56:36.607 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:36.606+0000 7fecf8c00640 1 -- 172.21.15.32:0/1098880922 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7fecf40ae310 con 0x7fecf406f440 2024-08-03T20:56:36.607 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:36.606+0000 7fecebe00640 1 -- 172.21.15.32:0/1098880922 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 24 entries v97) ==== 81+0+1462 (secure 0 0 0) 0x7fece4061480 con 0x7fecf406f440 2024-08-03T20:56:36.607 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6835/2278473729 2024-08-04T20:56:13.041693+0000 2024-08-03T20:56:36.607 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6833/2278473729 2024-08-04T20:56:13.041693+0000 2024-08-03T20:56:36.607 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6833/4172198298 2024-08-04T20:56:04.256182+0000 2024-08-03T20:56:36.608 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6833/3771304550 2024-08-04T20:55:58.212752+0000 2024-08-03T20:56:36.608 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6835/3771304550 2024-08-04T20:55:58.212752+0000 2024-08-03T20:56:36.608 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6835/2803901497 2024-08-04T20:55:55.181144+0000 2024-08-03T20:56:36.608 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6833/2803901497 2024-08-04T20:55:55.181144+0000 2024-08-03T20:56:36.608 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6835/3191654276 2024-08-04T20:55:42.433364+0000 2024-08-03T20:56:36.608 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6833/1707454554 2024-08-04T20:54:38.409411+0000 2024-08-03T20:56:36.608 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6833/3431300225 2024-08-04T20:56:21.088927+0000 2024-08-03T20:56:36.608 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6835/4172198298 2024-08-04T20:56:04.256182+0000 2024-08-03T20:56:36.608 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6835/2194902489 2024-08-04T20:55:00.359126+0000 2024-08-03T20:56:36.608 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6835/1707454554 2024-08-04T20:54:38.409411+0000 2024-08-03T20:56:36.608 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6833/2194902489 2024-08-04T20:55:00.359126+0000 2024-08-03T20:56:36.608 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6833/1016338829 2024-08-04T20:55:06.071485+0000 2024-08-03T20:56:36.608 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6835/1172050551 2024-08-04T20:55:26.214113+0000 2024-08-03T20:56:36.609 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6835/1016338829 2024-08-04T20:55:06.071485+0000 2024-08-03T20:56:36.609 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6833/756683988 2024-08-04T20:55:14.119228+0000 2024-08-03T20:56:36.609 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6835/756683988 2024-08-04T20:55:14.119228+0000 2024-08-03T20:56:36.609 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6833/2292690833 2024-08-04T20:55:20.021329+0000 2024-08-03T20:56:36.609 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6835/3431300225 2024-08-04T20:56:21.088927+0000 2024-08-03T20:56:36.609 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6835/2292690833 2024-08-04T20:55:20.021329+0000 2024-08-03T20:56:36.609 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6833/1172050551 2024-08-04T20:55:26.214113+0000 2024-08-03T20:56:36.609 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6833/3191654276 2024-08-04T20:55:42.433364+0000 2024-08-03T20:56:36.610 INFO:teuthology.orchestra.run.smithi032.stderr:listed 24 entries 2024-08-03T20:56:36.610 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:36.608+0000 7fecf8c00640 1 -- 172.21.15.32:0/1098880922 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fecd0077010 msgr2=0x7fecd0079520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:56:36.611 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:36.608+0000 7fecf8c00640 1 --2- 172.21.15.32:0/1098880922 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fecd0077010 0x7fecd0079520 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7fecdc033000 tx=0x7fecdc004730 comp rx=0 tx=0).stop 2024-08-03T20:56:36.611 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:36.608+0000 7fecf8c00640 1 -- 172.21.15.32:0/1098880922 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fecf406f440 msgr2=0x7fecf40b2c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:56:36.611 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:36.608+0000 7fecf8c00640 1 --2- 172.21.15.32:0/1098880922 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fecf406f440 0x7fecf40b2c30 secure :-1 s=READY pgs=1384 cs=0 l=1 rev1=1 crypto rx=0x7fece400d700 tx=0x7fece400dbc0 comp rx=0 tx=0).stop 2024-08-03T20:56:36.611 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:36.609+0000 7fecf8c00640 1 -- 172.21.15.32:0/1098880922 shutdown_connections 2024-08-03T20:56:36.611 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:36.609+0000 7fecf8c00640 1 --2- 172.21.15.32:0/1098880922 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fecd0077010 0x7fecd0079520 unknown :-1 s=CLOSED pgs=567 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:36.611 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:36.609+0000 7fecf8c00640 1 --2- 172.21.15.32:0/1098880922 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fecf40acd80 0x7fecf415ec60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:36.611 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:36.609+0000 7fecf8c00640 1 --2- 172.21.15.32:0/1098880922 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fecf40af020 0x7fecf41f6170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:36.611 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:36.609+0000 7fecf8c00640 1 --2- 172.21.15.32:0/1098880922 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fecf406f440 0x7fecf40b2c30 unknown :-1 s=CLOSED pgs=1384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:36.612 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:36.609+0000 7fecf8c00640 1 -- 172.21.15.32:0/1098880922 >> 172.21.15.32:0/1098880922 conn(0x7fecf40b2120 msgr2=0x7fecf40ca390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:56:36.612 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:36.609+0000 7fecf8c00640 1 -- 172.21.15.32:0/1098880922 shutdown_connections 2024-08-03T20:56:36.612 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:36.609+0000 7fecf8c00640 1 -- 172.21.15.32:0/1098880922 wait complete. 2024-08-03T20:56:36.631 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-08-03T20:56:36.632 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-08-03T20:56:36.632 DEBUG:teuthology.orchestra.run.smithi032:> sudo umount /home/ubuntu/cephtest/mnt.0 -f 2024-08-03T20:56:36.712 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi032.front.sepia.ceph.com 2024-08-03T20:56:36.712 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-08-03T20:56:36.712 DEBUG:teuthology.orchestra.run.smithi032:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-08-03T20:56:36.748 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-08-03T20:56:37.020 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:37.019+0000 7f49b7e00640 1 -- 172.21.15.32:0/2975715374 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f499c004020 msgr2=0x7f499c004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:56:37.020 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:37.019+0000 7f49b7e00640 1 --2- 172.21.15.32:0/2975715374 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f499c004020 0x7f499c004800 secure :-1 s=READY pgs=1386 cs=0 l=1 rev1=1 crypto rx=0x7f49b80703f0 tx=0x7f499c005830 comp rx=0 tx=0).stop 2024-08-03T20:56:37.021 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:37.019+0000 7f49b7400640 1 -- 172.21.15.32:0/2975715374 reap_dead start 2024-08-03T20:56:37.021 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:37.019+0000 7f49b7e00640 1 -- 172.21.15.32:0/2975715374 shutdown_connections 2024-08-03T20:56:37.021 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:37.019+0000 7f49b7e00640 1 -- 172.21.15.32:0/2975715374 >> 172.21.15.32:0/2975715374 conn(0x7f49b80b2120 msgr2=0x7f49b80cfd60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:56:37.021 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:37.019+0000 7f49b7e00640 1 -- 172.21.15.32:0/2975715374 shutdown_connections 2024-08-03T20:56:37.021 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:37.020+0000 7f49b7e00640 1 -- 172.21.15.32:0/2975715374 wait complete. 2024-08-03T20:56:37.022 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:37.022+0000 7f49b7e00640 1 Processor -- start 2024-08-03T20:56:37.022 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:37.022+0000 7f49b7e00640 1 -- start start 2024-08-03T20:56:37.025 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:37.023+0000 7f49b7e00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f499c004020 0x7f49b81f32b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:56:37.026 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:37.023+0000 7f49b7e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f49b81f0780 0x7f49b80c3580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:56:37.026 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:37.023+0000 7f49b7e00640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f49b80c3ac0 0x7f49b8154b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:56:37.026 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:37.023+0000 7f49b7e00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f49b80748f0 con 0x7f49b81f0780 2024-08-03T20:56:37.026 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:37.023+0000 7f49b7e00640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f49b8074770 con 0x7f499c004020 2024-08-03T20:56:37.026 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:37.023+0000 7f49b7e00640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7f49b815fe50 con 0x7f49b80c3ac0 2024-08-03T20:56:37.026 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:37.024+0000 7f49b6000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f49b81f0780 0x7f49b80c3580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:56:37.026 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:37.024+0000 7f49b6000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f49b81f0780 0x7f49b80c3580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:37082/0 (socket says 172.21.15.32:37082) 2024-08-03T20:56:37.026 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:37.024+0000 7f49b6000640 1 -- 172.21.15.32:0/3527771346 learned_addr learned my addr 172.21.15.32:0/3527771346 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:56:37.026 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:37.024+0000 7f49b6a00640 1 --2- 172.21.15.32:0/3527771346 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f499c004020 0x7f49b81f32b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:56:37.026 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:37.024+0000 7f49b7400640 1 --2- 172.21.15.32:0/3527771346 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f49b80c3ac0 0x7f49b8154b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:56:37.026 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:37.024+0000 7f49b6000640 1 -- 172.21.15.32:0/3527771346 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f49b80c3ac0 msgr2=0x7f49b8154b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:56:37.027 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:37.024+0000 7f49b6000640 1 --2- 172.21.15.32:0/3527771346 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f49b80c3ac0 0x7f49b8154b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:37.027 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:37.024+0000 7f49b6000640 1 -- 172.21.15.32:0/3527771346 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f499c004020 msgr2=0x7f49b81f32b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:56:37.027 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:37.024+0000 7f49b6000640 1 --2- 172.21.15.32:0/3527771346 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f499c004020 0x7f49b81f32b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:37.027 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:37.024+0000 7f49b6000640 1 -- 172.21.15.32:0/3527771346 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f49b80c3ed0 con 0x7f49b81f0780 2024-08-03T20:56:37.027 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:37.024+0000 7f49b6000640 1 --2- 172.21.15.32:0/3527771346 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f49b81f0780 0x7f49b80c3580 secure :-1 s=READY pgs=1387 cs=0 l=1 rev1=1 crypto rx=0x7f49a400c320 tx=0x7f49a4001d40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:56:37.027 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:37.025+0000 7f49af400640 1 -- 172.21.15.32:0/3527771346 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49a4031040 con 0x7f49b81f0780 2024-08-03T20:56:37.027 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:37.025+0000 7f49af400640 1 -- 172.21.15.32:0/3527771346 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f49a40035f0 con 0x7f49b81f0780 2024-08-03T20:56:37.027 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:37.025+0000 7f49af400640 1 -- 172.21.15.32:0/3527771346 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49a4003a20 con 0x7f49b81f0780 2024-08-03T20:56:37.027 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:37.025+0000 7f49b7e00640 1 -- 172.21.15.32:0/3527771346 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f49b81f3c90 con 0x7f49b81f0780 2024-08-03T20:56:37.027 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:37.025+0000 7f49b7e00640 1 -- 172.21.15.32:0/3527771346 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f49b80c7f90 con 0x7f49b81f0780 2024-08-03T20:56:37.028 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:37.026+0000 7f49b7e00640 1 -- 172.21.15.32:0/3527771346 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f49b81f3a00 con 0x7f49b81f0780 2024-08-03T20:56:37.028 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:37.027+0000 7f49af400640 1 -- 172.21.15.32:0/3527771346 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f49a4034070 con 0x7f49b81f0780 2024-08-03T20:56:37.028 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:37.028+0000 7f49af400640 1 --2- 172.21.15.32:0/3527771346 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f4998077010 0x7f4998079520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:56:37.029 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:37.028+0000 7f49af400640 1 -- 172.21.15.32:0/3527771346 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 6886+0+0 (secure 0 0 0) 0x7f49a40b8c30 con 0x7f49b81f0780 2024-08-03T20:56:37.029 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:37.029+0000 7f49b6a00640 1 --2- 172.21.15.32:0/3527771346 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f4998077010 0x7f4998079520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:56:37.030 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:37.030+0000 7f49b6a00640 1 --2- 172.21.15.32:0/3527771346 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f4998077010 0x7f4998079520 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7f49a8009bb0 tx=0x7f49a8005f90 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:56:37.033 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:37.033+0000 7f49af400640 1 -- 172.21.15.32:0/3527771346 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7f49b80c3ed0 con 0x7f49b81f0780 2024-08-03T20:56:37.279 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:37.278+0000 7f49b7e00640 1 -- 172.21.15.32:0/3527771346 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7f49b80ad190 con 0x7f49b81f0780 2024-08-03T20:56:37.279 INFO:tasks.ceph.mon.a.smithi032.stderr:2024-08-03T20:56:37.279+0000 7f4e6d600640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-08-03T20:56:37.281 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:37.281+0000 7f49af400640 1 -- 172.21.15.32:0/3527771346 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v170) ==== 161+0+0 (secure 0 0 0) 0x7f49a40819b0 con 0x7f49b81f0780 2024-08-03T20:56:37.282 INFO:teuthology.orchestra.run.smithi032.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-08-03T20:56:37.285 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:37.284+0000 7f49b7e00640 1 -- 172.21.15.32:0/3527771346 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f4998077010 msgr2=0x7f4998079520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:56:37.285 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:37.284+0000 7f49b7e00640 1 --2- 172.21.15.32:0/3527771346 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f4998077010 0x7f4998079520 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7f49a8009bb0 tx=0x7f49a8005f90 comp rx=0 tx=0).stop 2024-08-03T20:56:37.286 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:37.284+0000 7f49b7e00640 1 -- 172.21.15.32:0/3527771346 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f49b81f0780 msgr2=0x7f49b80c3580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:56:37.286 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:37.284+0000 7f49b7e00640 1 --2- 172.21.15.32:0/3527771346 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f49b81f0780 0x7f49b80c3580 secure :-1 s=READY pgs=1387 cs=0 l=1 rev1=1 crypto rx=0x7f49a400c320 tx=0x7f49a4001d40 comp rx=0 tx=0).stop 2024-08-03T20:56:37.286 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:37.284+0000 7f49b7e00640 1 -- 172.21.15.32:0/3527771346 shutdown_connections 2024-08-03T20:56:37.286 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:37.284+0000 7f49b7e00640 1 --2- 172.21.15.32:0/3527771346 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f4998077010 0x7f4998079520 unknown :-1 s=CLOSED pgs=568 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:37.286 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:37.284+0000 7f49b7e00640 1 --2- 172.21.15.32:0/3527771346 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f49b80c3ac0 0x7f49b8154b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:37.287 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:37.284+0000 7f49b7e00640 1 --2- 172.21.15.32:0/3527771346 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f49b81f0780 0x7f49b80c3580 unknown :-1 s=CLOSED pgs=1387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:37.287 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:37.284+0000 7f49b7e00640 1 --2- 172.21.15.32:0/3527771346 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f499c004020 0x7f49b81f32b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:37.287 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:37.284+0000 7f49b7e00640 1 -- 172.21.15.32:0/3527771346 >> 172.21.15.32:0/3527771346 conn(0x7f49b80b2120 msgr2=0x7f49b80cacf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:56:37.288 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:37.284+0000 7f49b7e00640 1 -- 172.21.15.32:0/3527771346 shutdown_connections 2024-08-03T20:56:37.288 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:37.284+0000 7f49b7e00640 1 -- 172.21.15.32:0/3527771346 wait complete. 2024-08-03T20:56:37.289 INFO:tasks.ceph.mds.a.smithi032.stderr: -751> 2024-08-03T20:56:31.110+0000 7fa5c5e00640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-08-03T20:56:37.308 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mds repaired 0 2024-08-03T20:56:37.403 INFO:tasks.ceph.mds.a.smithi032.stdout:starting mds.a at 2024-08-03T20:56:37.540 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:37.539+0000 7fcb3fe00640 1 -- 172.21.15.32:0/2897685134 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcb400af020 msgr2=0x7fcb400742b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:56:37.540 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:37.539+0000 7fcb3fe00640 1 --2- 172.21.15.32:0/2897685134 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcb400af020 0x7fcb400742b0 secure :-1 s=READY pgs=1390 cs=0 l=1 rev1=1 crypto rx=0x7fcb30009830 tx=0x7fcb3002a480 comp rx=0 tx=0).stop 2024-08-03T20:56:37.540 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:37.539+0000 7fcb3fe00640 1 -- 172.21.15.32:0/2897685134 shutdown_connections 2024-08-03T20:56:37.540 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:37.539+0000 7fcb3fe00640 1 --2- 172.21.15.32:0/2897685134 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fcb4006fb50 0x7fcb400ad190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:37.540 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:37.539+0000 7fcb3fe00640 1 --2- 172.21.15.32:0/2897685134 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcb400af020 0x7fcb400742b0 unknown :-1 s=CLOSED pgs=1390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:37.540 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:37.539+0000 7fcb3fe00640 1 -- 172.21.15.32:0/2897685134 >> 172.21.15.32:0/2897685134 conn(0x7fcb400b2120 msgr2=0x7fcb400cfd60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:56:37.540 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:37.539+0000 7fcb3fe00640 1 -- 172.21.15.32:0/2897685134 shutdown_connections 2024-08-03T20:56:37.540 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:37.539+0000 7fcb3fe00640 1 -- 172.21.15.32:0/2897685134 wait complete. 2024-08-03T20:56:37.542 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:37.541+0000 7fcb3fe00640 1 Processor -- start 2024-08-03T20:56:37.542 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:37.541+0000 7fcb3fe00640 1 -- start start 2024-08-03T20:56:37.544 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:37.543+0000 7fcb3fe00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcb4006fb50 0x7fcb401f3bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:56:37.544 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:37.543+0000 7fcb3fe00640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fcb400acd80 0x7fcb4009bc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:56:37.545 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:37.543+0000 7fcb3fe00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fcb400af020 0x7fcb400c5570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:56:37.545 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:37.543+0000 7fcb3fe00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fcb401601e0 con 0x7fcb4006fb50 2024-08-03T20:56:37.545 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:37.543+0000 7fcb3fe00640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7fcb40160060 con 0x7fcb400af020 2024-08-03T20:56:37.545 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:37.543+0000 7fcb3fe00640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7fcb40160360 con 0x7fcb400acd80 2024-08-03T20:56:37.545 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:37.543+0000 7fcb3ea00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcb4006fb50 0x7fcb401f3bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:56:37.545 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:37.543+0000 7fcb3ea00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcb4006fb50 0x7fcb401f3bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:37114/0 (socket says 172.21.15.32:37114) 2024-08-03T20:56:37.545 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:37.543+0000 7fcb3ea00640 1 -- 172.21.15.32:0/2065338543 learned_addr learned my addr 172.21.15.32:0/2065338543 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:56:37.545 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:37.543+0000 7fcb36000640 1 --2- 172.21.15.32:0/2065338543 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fcb400acd80 0x7fcb4009bc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:56:37.545 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:37.543+0000 7fcb3f400640 1 --2- 172.21.15.32:0/2065338543 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fcb400af020 0x7fcb400c5570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:56:37.545 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:37.544+0000 7fcb3ea00640 1 -- 172.21.15.32:0/2065338543 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fcb400acd80 msgr2=0x7fcb4009bc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:56:37.545 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:37.544+0000 7fcb3ea00640 1 --2- 172.21.15.32:0/2065338543 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fcb400acd80 0x7fcb4009bc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:37.545 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:37.544+0000 7fcb3ea00640 1 -- 172.21.15.32:0/2065338543 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fcb400af020 msgr2=0x7fcb400c5570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:56:37.545 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:37.544+0000 7fcb3ea00640 1 --2- 172.21.15.32:0/2065338543 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fcb400af020 0x7fcb400c5570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:37.546 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:37.544+0000 7fcb3ea00640 1 -- 172.21.15.32:0/2065338543 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcb401f40f0 con 0x7fcb4006fb50 2024-08-03T20:56:37.546 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:37.544+0000 7fcb3ea00640 1 --2- 172.21.15.32:0/2065338543 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcb4006fb50 0x7fcb401f3bb0 secure :-1 s=READY pgs=1391 cs=0 l=1 rev1=1 crypto rx=0x7fcb30009950 tx=0x7fcb30033c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:56:37.546 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:37.544+0000 7fcb37400640 1 -- 172.21.15.32:0/2065338543 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb30005c90 con 0x7fcb4006fb50 2024-08-03T20:56:37.547 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:37.544+0000 7fcb37400640 1 -- 172.21.15.32:0/2065338543 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fcb30031e50 con 0x7fcb4006fb50 2024-08-03T20:56:37.547 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:37.544+0000 7fcb3fe00640 1 -- 172.21.15.32:0/2065338543 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcb400c37f0 con 0x7fcb4006fb50 2024-08-03T20:56:37.547 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:37.545+0000 7fcb37400640 1 -- 172.21.15.32:0/2065338543 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb30005260 con 0x7fcb4006fb50 2024-08-03T20:56:37.547 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:37.545+0000 7fcb3fe00640 1 -- 172.21.15.32:0/2065338543 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcb400c3a00 con 0x7fcb4006fb50 2024-08-03T20:56:37.547 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:37.546+0000 7fcb3fe00640 1 -- 172.21.15.32:0/2065338543 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcb400ad190 con 0x7fcb4006fb50 2024-08-03T20:56:37.547 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:37.547+0000 7fcb37400640 1 -- 172.21.15.32:0/2065338543 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7fcb30005400 con 0x7fcb4006fb50 2024-08-03T20:56:37.548 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:37.548+0000 7fcb37400640 1 --2- 172.21.15.32:0/2065338543 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fcb1c077010 0x7fcb1c079520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:56:37.548 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:37.548+0000 7fcb37400640 1 -- 172.21.15.32:0/2065338543 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 6972+0+0 (secure 0 0 0) 0x7fcb300b9fe0 con 0x7fcb4006fb50 2024-08-03T20:56:37.549 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:37.548+0000 7fcb36000640 1 --2- 172.21.15.32:0/2065338543 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fcb1c077010 0x7fcb1c079520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:56:37.549 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:37.549+0000 7fcb36000640 1 --2- 172.21.15.32:0/2065338543 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fcb1c077010 0x7fcb1c079520 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7fcb4009d210 tx=0x7fcb28009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:56:37.552 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:37.552+0000 7fcb37400640 1 -- 172.21.15.32:0/2065338543 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7fcb401f40f0 con 0x7fcb4006fb50 2024-08-03T20:56:37.836 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:37.835+0000 7fcb3fe00640 1 -- 172.21.15.32:0/2065338543 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "mds repaired", "role": "0"} v 0) -- 0x7fcb400d0930 con 0x7fcb4006fb50 2024-08-03T20:56:38.283 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:38.283+0000 7fcb37400640 1 -- 172.21.15.32:0/2065338543 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "mds repaired", "role": "0"}]=0 nothing to do: rank is not damaged v171) ==== 107+0+0 (secure 0 0 0) 0x7fcb3003f030 con 0x7fcb4006fb50 2024-08-03T20:56:38.283 INFO:teuthology.orchestra.run.smithi032.stderr:nothing to do: rank is not damaged 2024-08-03T20:56:38.286 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:38.286+0000 7fcb3fe00640 1 -- 172.21.15.32:0/2065338543 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fcb1c077010 msgr2=0x7fcb1c079520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:56:38.286 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:38.286+0000 7fcb3fe00640 1 --2- 172.21.15.32:0/2065338543 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fcb1c077010 0x7fcb1c079520 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7fcb4009d210 tx=0x7fcb28009040 comp rx=0 tx=0).stop 2024-08-03T20:56:38.287 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:38.286+0000 7fcb3fe00640 1 -- 172.21.15.32:0/2065338543 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcb4006fb50 msgr2=0x7fcb401f3bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:56:38.287 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:38.286+0000 7fcb3fe00640 1 --2- 172.21.15.32:0/2065338543 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcb4006fb50 0x7fcb401f3bb0 secure :-1 s=READY pgs=1391 cs=0 l=1 rev1=1 crypto rx=0x7fcb30009950 tx=0x7fcb30033c20 comp rx=0 tx=0).stop 2024-08-03T20:56:38.287 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:38.286+0000 7fcb3fe00640 1 -- 172.21.15.32:0/2065338543 shutdown_connections 2024-08-03T20:56:38.287 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:38.286+0000 7fcb3fe00640 1 --2- 172.21.15.32:0/2065338543 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fcb1c077010 0x7fcb1c079520 unknown :-1 s=CLOSED pgs=569 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:38.287 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:38.286+0000 7fcb3fe00640 1 --2- 172.21.15.32:0/2065338543 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fcb400af020 0x7fcb400c5570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:38.287 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:38.286+0000 7fcb3fe00640 1 --2- 172.21.15.32:0/2065338543 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fcb400acd80 0x7fcb4009bc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:38.287 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:38.286+0000 7fcb3fe00640 1 --2- 172.21.15.32:0/2065338543 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcb4006fb50 0x7fcb401f3bb0 unknown :-1 s=CLOSED pgs=1391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:38.287 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:38.286+0000 7fcb3fe00640 1 -- 172.21.15.32:0/2065338543 >> 172.21.15.32:0/2065338543 conn(0x7fcb400b2120 msgr2=0x7fcb40068bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:56:38.287 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:38.286+0000 7fcb3fe00640 1 -- 172.21.15.32:0/2065338543 shutdown_connections 2024-08-03T20:56:38.287 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:38.286+0000 7fcb3fe00640 1 -- 172.21.15.32:0/2065338543 wait complete. 2024-08-03T20:56:38.314 DEBUG:teuthology.orchestra.run.smithi032:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata import - 2024-08-03T20:56:38.315 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-08-03T20:56:38.321 DEBUG:teuthology.orchestra.run.smithi118:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-08-03T20:56:38.461 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:38.460+0000 7fe698dcbf40 1 Processor -- start 2024-08-03T20:56:38.461 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:38.461+0000 7fe698dcbf40 1 -- start start 2024-08-03T20:56:38.463 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:38.462+0000 7fe698dcbf40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55bab646c8e0 0x55bab64faf60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:56:38.463 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:38.462+0000 7fe698dcbf40 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55bab64efba0 0x55bab646ac70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:56:38.463 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:38.462+0000 7fe698dcbf40 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55bab64fc5b0 0x55bab644e320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:56:38.463 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:38.462+0000 7fe698dcbf40 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55bab6371e30 con 0x55bab64efba0 2024-08-03T20:56:38.464 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:38.462+0000 7fe698dcbf40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x55bab63cfbf0 con 0x55bab646c8e0 2024-08-03T20:56:38.464 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:38.462+0000 7fe698dcbf40 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x55bab646be80 con 0x55bab64fc5b0 2024-08-03T20:56:38.464 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:38.462+0000 7fe691600640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55bab64efba0 0x55bab646ac70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:56:38.464 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:38.462+0000 7fe691600640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55bab64efba0 0x55bab646ac70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:37124/0 (socket says 172.21.15.32:37124) 2024-08-03T20:56:38.464 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:38.462+0000 7fe691600640 1 -- 172.21.15.32:0/1619102376 learned_addr learned my addr 172.21.15.32:0/1619102376 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:56:38.464 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:38.462+0000 7fe692a00640 1 --2- 172.21.15.32:0/1619102376 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55bab64fc5b0 0x55bab644e320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:56:38.464 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:38.462+0000 7fe692000640 1 --2- 172.21.15.32:0/1619102376 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55bab646c8e0 0x55bab64faf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:56:38.464 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:38.463+0000 7fe691600640 1 -- 172.21.15.32:0/1619102376 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55bab64fc5b0 msgr2=0x55bab644e320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:56:38.464 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:38.463+0000 7fe691600640 1 --2- 172.21.15.32:0/1619102376 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55bab64fc5b0 0x55bab644e320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:38.464 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:38.463+0000 7fe691600640 1 -- 172.21.15.32:0/1619102376 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55bab646c8e0 msgr2=0x55bab64faf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:56:38.464 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:38.463+0000 7fe691600640 1 --2- 172.21.15.32:0/1619102376 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55bab646c8e0 0x55bab64faf60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:38.464 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:38.463+0000 7fe691600640 1 -- 172.21.15.32:0/1619102376 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bab64ef200 con 0x55bab64efba0 2024-08-03T20:56:38.464 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:38.463+0000 7fe691600640 1 --2- 172.21.15.32:0/1619102376 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55bab64efba0 0x55bab646ac70 secure :-1 s=READY pgs=1393 cs=0 l=1 rev1=1 crypto rx=0x7fe68000b600 tx=0x7fe68000bac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=28c8b558fed1a3dc server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:56:38.466 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:38.463+0000 7fe68be00640 1 -- 172.21.15.32:0/1619102376 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe68000a8f0 con 0x55bab64efba0 2024-08-03T20:56:38.466 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:38.463+0000 7fe68be00640 1 -- 172.21.15.32:0/1619102376 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fe68000aa90 con 0x55bab64efba0 2024-08-03T20:56:38.466 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:38.463+0000 7fe68be00640 1 -- 172.21.15.32:0/1619102376 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe680008450 con 0x55bab64efba0 2024-08-03T20:56:38.466 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:38.464+0000 7fe698dcbf40 1 -- 172.21.15.32:0/1619102376 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55bab64efba0 msgr2=0x55bab646ac70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:56:38.466 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:38.464+0000 7fe698dcbf40 1 --2- 172.21.15.32:0/1619102376 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55bab64efba0 0x55bab646ac70 secure :-1 s=READY pgs=1393 cs=0 l=1 rev1=1 crypto rx=0x7fe68000b600 tx=0x7fe68000bac0 comp rx=0 tx=0).stop 2024-08-03T20:56:38.466 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:38.464+0000 7fe698dcbf40 1 -- 172.21.15.32:0/1619102376 shutdown_connections 2024-08-03T20:56:38.466 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:38.464+0000 7fe698dcbf40 1 --2- 172.21.15.32:0/1619102376 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55bab64fc5b0 0x55bab644e320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:38.466 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:38.464+0000 7fe698dcbf40 1 --2- 172.21.15.32:0/1619102376 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55bab64efba0 0x55bab646ac70 unknown :-1 s=CLOSED pgs=1393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:38.466 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:38.464+0000 7fe698dcbf40 1 --2- 172.21.15.32:0/1619102376 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55bab646c8e0 0x55bab64faf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:38.466 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:38.464+0000 7fe698dcbf40 1 -- 172.21.15.32:0/1619102376 >> 172.21.15.32:0/1619102376 conn(0x55bab6343ac0 msgr2=0x55bab6421340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:56:38.467 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:38.464+0000 7fe698dcbf40 1 -- 172.21.15.32:0/1619102376 shutdown_connections 2024-08-03T20:56:38.467 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:38.464+0000 7fe698dcbf40 1 -- 172.21.15.32:0/1619102376 wait complete. 2024-08-03T20:56:38.467 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:38.465+0000 7fe698dcbf40 1 Processor -- start 2024-08-03T20:56:38.467 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:38.465+0000 7fe698dcbf40 1 -- start start 2024-08-03T20:56:38.469 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:38.467+0000 7fe698dcbf40 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55bab646c8e0 0x55bab64fad20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:56:38.469 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:38.467+0000 7fe698dcbf40 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55bab64efba0 0x55bab644e510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:56:38.469 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:38.467+0000 7fe698dcbf40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55bab64fc5b0 0x55bab64f6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:56:38.469 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:38.467+0000 7fe698dcbf40 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55bab6468340 con 0x55bab64efba0 2024-08-03T20:56:38.469 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:38.467+0000 7fe698dcbf40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x55bab64659b0 con 0x55bab64fc5b0 2024-08-03T20:56:38.469 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:38.467+0000 7fe698dcbf40 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x55bab64fb4a0 con 0x55bab646c8e0 2024-08-03T20:56:38.469 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:38.467+0000 7fe691600640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55bab64efba0 0x55bab644e510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:56:38.469 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:38.467+0000 7fe691600640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55bab64efba0 0x55bab644e510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:37126/0 (socket says 172.21.15.32:37126) 2024-08-03T20:56:38.469 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:38.467+0000 7fe691600640 1 -- 172.21.15.32:0/4237013562 learned_addr learned my addr 172.21.15.32:0/4237013562 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:56:38.469 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:38.467+0000 7fe692000640 1 --2- 172.21.15.32:0/4237013562 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55bab646c8e0 0x55bab64fad20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:56:38.469 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:38.467+0000 7fe692a00640 1 --2- 172.21.15.32:0/4237013562 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55bab64fc5b0 0x55bab64f6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:56:38.470 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:38.467+0000 7fe691600640 1 -- 172.21.15.32:0/4237013562 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55bab646c8e0 msgr2=0x55bab64fad20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:56:38.470 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:38.467+0000 7fe691600640 1 --2- 172.21.15.32:0/4237013562 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55bab646c8e0 0x55bab64fad20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:38.470 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:38.467+0000 7fe691600640 1 -- 172.21.15.32:0/4237013562 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55bab64fc5b0 msgr2=0x55bab64f6070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:56:38.470 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:38.467+0000 7fe691600640 1 --2- 172.21.15.32:0/4237013562 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55bab64fc5b0 0x55bab64f6070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:38.470 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:38.467+0000 7fe691600640 1 -- 172.21.15.32:0/4237013562 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bab64ee4f0 con 0x55bab64efba0 2024-08-03T20:56:38.470 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:38.468+0000 7fe691600640 1 --2- 172.21.15.32:0/4237013562 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55bab64efba0 0x55bab644e510 secure :-1 s=READY pgs=1394 cs=0 l=1 rev1=1 crypto rx=0x55bab6469cf0 tx=0x7fe680016420 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:56:38.470 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:38.468+0000 7fe68aa00640 1 -- 172.21.15.32:0/4237013562 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe680016820 con 0x55bab64efba0 2024-08-03T20:56:38.470 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:38.468+0000 7fe68aa00640 1 -- 172.21.15.32:0/4237013562 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fe6800169e0 con 0x55bab64efba0 2024-08-03T20:56:38.470 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:38.468+0000 7fe68aa00640 1 -- 172.21.15.32:0/4237013562 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe680016c30 con 0x55bab64efba0 2024-08-03T20:56:38.470 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:38.468+0000 7fe698dcbf40 1 -- 172.21.15.32:0/4237013562 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x55bab64675f0 con 0x55bab64efba0 2024-08-03T20:56:38.470 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:38.468+0000 7fe698dcbf40 1 -- 172.21.15.32:0/4237013562 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bab6467890 con 0x55bab64efba0 2024-08-03T20:56:38.471 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:38.470+0000 7fe68aa00640 1 -- 172.21.15.32:0/4237013562 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7fe68001e030 con 0x55bab64efba0 2024-08-03T20:56:38.471 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:38.471+0000 7fe68aa00640 1 --2- 172.21.15.32:0/4237013562 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fe6740759f0 0x7fe674077f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:56:38.472 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:38.471+0000 7fe68aa00640 1 -- 172.21.15.32:0/4237013562 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 6972+0+0 (secure 0 0 0) 0x7fe680098fa0 con 0x55bab64efba0 2024-08-03T20:56:38.472 INFO:teuthology.orchestra.run.smithi032.stdout:Importing pool 2024-08-03T20:56:38.473 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:38.472+0000 7fe692000640 1 --2- 172.21.15.32:0/4237013562 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fe6740759f0 0x7fe674077f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:56:38.473 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:38.472+0000 7fe698dcbf40 1 --2- 172.21.15.32:0/4237013562 >> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] conn(0x55bab64f65b0 0x55bab64669c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:56:38.473 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:38.472+0000 7fe698dcbf40 1 -- 172.21.15.32:0/4237013562 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:1 12.10 12:09401a81:::601.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98) -- 0x55bab644ea50 con 0x55bab64f65b0 2024-08-03T20:56:38.473 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:38.472+0000 7fe692a00640 1 --2- 172.21.15.32:0/4237013562 >> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] conn(0x55bab64f65b0 0x55bab64669c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:56:38.473 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:38.473+0000 7fe692a00640 1 --2- 172.21.15.32:0/4237013562 >> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] conn(0x55bab64f65b0 0x55bab64669c0 crc :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:56:38.473 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:38.473+0000 7fe692000640 1 --2- 172.21.15.32:0/4237013562 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fe6740759f0 0x7fe674077f00 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7fe684011480 tx=0x7fe684001730 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:56:38.479 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:38.477+0000 7fe692a00640 1 -- 172.21.15.32:0/4237013562 <== osd.0 v2:172.21.15.32:6816/3066156116 1 ==== osd_op_reply(1 601.00000000 [create] v98'381 uv355 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fe68c00b040 con 0x55bab64f65b0 2024-08-03T20:56:38.479 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:38.477+0000 7fe698dcbf40 1 -- 172.21.15.32:0/4237013562 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:2 12.10 12:09401a81:::601.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e98) -- 0x55bab64edc00 con 0x55bab64f65b0 2024-08-03T20:56:38.479 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:38.478+0000 7fe692a00640 1 -- 172.21.15.32:0/4237013562 <== osd.0 v2:172.21.15.32:6816/3066156116 2 ==== osd_op_reply(2 601.00000000 [delete] v98'382 uv382 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe68c00b040 con 0x55bab64f65b0 2024-08-03T20:56:38.479 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:38.478+0000 7fe698dcbf40 1 -- 172.21.15.32:0/4237013562 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:3 12.10 12:09401a81:::601.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98) -- 0x55bab6466f00 con 0x55bab64f65b0 2024-08-03T20:56:38.480 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::601.00000000:head# 2024-08-03T20:56:38.480 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:38.480+0000 7fe692a00640 1 -- 172.21.15.32:0/4237013562 <== osd.0 v2:172.21.15.32:6816/3066156116 3 ==== osd_op_reply(3 601.00000000 [create] v98'383 uv383 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe68c00b040 con 0x55bab64f65b0 2024-08-03T20:56:38.481 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:38.480+0000 7fe698dcbf40 1 -- 172.21.15.32:0/4237013562 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:4 12.10 12:09401a81:::601.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98) -- 0x55bab659dd20 con 0x55bab64f65b0 2024-08-03T20:56:38.482 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:38.482+0000 7fe692a00640 1 -- 172.21.15.32:0/4237013562 <== osd.0 v2:172.21.15.32:6816/3066156116 4 ==== osd_op_reply(4 601.00000000 [setxattr (30)] v98'384 uv384 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe68c00b040 con 0x55bab64f65b0 2024-08-03T20:56:38.482 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:38.482+0000 7fe698dcbf40 1 -- 172.21.15.32:0/4237013562 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:5 12.10 12:09401a81:::601.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98) -- 0x55bab659e0c0 con 0x55bab64f65b0 2024-08-03T20:56:38.484 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:38.483+0000 7fe692a00640 1 -- 172.21.15.32:0/4237013562 <== osd.0 v2:172.21.15.32:6816/3066156116 5 ==== osd_op_reply(5 601.00000000 [setxattr (62)] v98'385 uv385 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe68c00b040 con 0x55bab64f65b0 2024-08-03T20:56:38.484 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:38.483+0000 7fe698dcbf40 1 -- 172.21.15.32:0/4237013562 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:6 12.10 12:09401a81:::601.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98) -- 0x55bab659cd10 con 0x55bab64f65b0 2024-08-03T20:56:38.486 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:38.485+0000 7fe692a00640 1 -- 172.21.15.32:0/4237013562 <== osd.0 v2:172.21.15.32:6816/3066156116 6 ==== osd_op_reply(6 601.00000000 [omap-set-header] v98'386 uv386 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe68c00b040 con 0x55bab64f65b0 2024-08-03T20:56:38.486 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:38.485+0000 7fe698dcbf40 1 -- 172.21.15.32:0/4237013562 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:7 12.10 12:0bd6d154:::602.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98) -- 0x55bab659d100 con 0x55bab64f65b0 2024-08-03T20:56:38.487 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:38.486+0000 7fe692a00640 1 -- 172.21.15.32:0/4237013562 <== osd.0 v2:172.21.15.32:6816/3066156116 7 ==== osd_op_reply(7 602.00000000 [create] v98'387 uv361 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fe68c00b040 con 0x55bab64f65b0 2024-08-03T20:56:38.487 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:38.486+0000 7fe698dcbf40 1 -- 172.21.15.32:0/4237013562 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:8 12.10 12:0bd6d154:::602.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e98) -- 0x55bab659d4f0 con 0x55bab64f65b0 2024-08-03T20:56:38.489 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:38.488+0000 7fe692a00640 1 -- 172.21.15.32:0/4237013562 <== osd.0 v2:172.21.15.32:6816/3066156116 8 ==== osd_op_reply(8 602.00000000 [delete] v98'388 uv388 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe68c00b040 con 0x55bab64f65b0 2024-08-03T20:56:38.489 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:38.488+0000 7fe698dcbf40 1 -- 172.21.15.32:0/4237013562 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:9 12.10 12:0bd6d154:::602.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98) -- 0x55bab659d4f0 con 0x55bab64f65b0 2024-08-03T20:56:38.490 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::602.00000000:head# 2024-08-03T20:56:38.490 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:38.490+0000 7fe692a00640 1 -- 172.21.15.32:0/4237013562 <== osd.0 v2:172.21.15.32:6816/3066156116 9 ==== osd_op_reply(9 602.00000000 [create] v98'389 uv389 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe68c00b040 con 0x55bab64f65b0 2024-08-03T20:56:38.490 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:38.490+0000 7fe698dcbf40 1 -- 172.21.15.32:0/4237013562 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:10 12.10 12:0bd6d154:::602.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98) -- 0x55bab659d4a0 con 0x55bab64f65b0 2024-08-03T20:56:38.492 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:38.491+0000 7fe692a00640 1 -- 172.21.15.32:0/4237013562 <== osd.0 v2:172.21.15.32:6816/3066156116 10 ==== osd_op_reply(10 602.00000000 [setxattr (30)] v98'390 uv390 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe68c00b040 con 0x55bab64f65b0 2024-08-03T20:56:38.492 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:38.491+0000 7fe698dcbf40 1 -- 172.21.15.32:0/4237013562 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:11 12.10 12:0bd6d154:::602.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98) -- 0x55bab659d4a0 con 0x55bab64f65b0 2024-08-03T20:56:38.493 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:38.493+0000 7fe692a00640 1 -- 172.21.15.32:0/4237013562 <== osd.0 v2:172.21.15.32:6816/3066156116 11 ==== osd_op_reply(11 602.00000000 [setxattr (62)] v98'391 uv391 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe68c00b040 con 0x55bab64f65b0 2024-08-03T20:56:38.493 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:38.493+0000 7fe698dcbf40 1 -- 172.21.15.32:0/4237013562 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:12 12.10 12:0bd6d154:::602.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98) -- 0x55bab659d4a0 con 0x55bab64f65b0 2024-08-03T20:56:38.495 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:38.494+0000 7fe692a00640 1 -- 172.21.15.32:0/4237013562 <== osd.0 v2:172.21.15.32:6816/3066156116 12 ==== osd_op_reply(12 602.00000000 [omap-set-header] v98'392 uv392 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe68c00b040 con 0x55bab64f65b0 2024-08-03T20:56:38.495 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:38.494+0000 7fe698dcbf40 1 -- 172.21.15.32:0/4237013562 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:13 12.10 12:0cf6ad0f:::10000000000.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98) -- 0x55bab659d4a0 con 0x55bab64f65b0 2024-08-03T20:56:38.497 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:38.496+0000 7fe692a00640 1 -- 172.21.15.32:0/4237013562 <== osd.0 v2:172.21.15.32:6816/3066156116 13 ==== osd_op_reply(13 10000000000.00000000 [create] v98'393 uv368 ondisk = -17 ((17) File exists)) ==== 164+0+0 (crc 0 0 0) 0x7fe68c00b040 con 0x55bab64f65b0 2024-08-03T20:56:38.497 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:38.496+0000 7fe698dcbf40 1 -- 172.21.15.32:0/4237013562 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:14 12.10 12:0cf6ad0f:::10000000000.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e98) -- 0x55bab659d4a0 con 0x55bab64f65b0 2024-08-03T20:56:38.498 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:38.498+0000 7fe692a00640 1 -- 172.21.15.32:0/4237013562 <== osd.0 v2:172.21.15.32:6816/3066156116 14 ==== osd_op_reply(14 10000000000.00000000 [delete] v98'394 uv394 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7fe68c00b040 con 0x55bab64f65b0 2024-08-03T20:56:38.498 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:38.498+0000 7fe698dcbf40 1 -- 172.21.15.32:0/4237013562 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:15 12.10 12:0cf6ad0f:::10000000000.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98) -- 0x55bab659d4a0 con 0x55bab64f65b0 2024-08-03T20:56:38.499 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:38.499+0000 7fe692a00640 1 -- 172.21.15.32:0/4237013562 <== osd.0 v2:172.21.15.32:6816/3066156116 15 ==== osd_op_reply(15 10000000000.00000000 [create] v98'395 uv395 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7fe68c00b040 con 0x55bab64f65b0 2024-08-03T20:56:38.500 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::10000000000.00000000:head# 2024-08-03T20:56:38.500 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:38.499+0000 7fe698dcbf40 1 -- 172.21.15.32:0/4237013562 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:16 12.10 12:0cf6ad0f:::10000000000.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98) -- 0x55bab659d4a0 con 0x55bab64f65b0 2024-08-03T20:56:38.501 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:38.501+0000 7fe692a00640 1 -- 172.21.15.32:0/4237013562 <== osd.0 v2:172.21.15.32:6816/3066156116 16 ==== osd_op_reply(16 10000000000.00000000 [setxattr (30)] v98'396 uv396 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7fe68c00b040 con 0x55bab64f65b0 2024-08-03T20:56:38.501 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:38.501+0000 7fe698dcbf40 1 -- 172.21.15.32:0/4237013562 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:17 12.10 12:0cf6ad0f:::10000000000.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98) -- 0x55bab659d4a0 con 0x55bab64f65b0 2024-08-03T20:56:38.502 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:38.502+0000 7fe692a00640 1 -- 172.21.15.32:0/4237013562 <== osd.0 v2:172.21.15.32:6816/3066156116 17 ==== osd_op_reply(17 10000000000.00000000 [setxattr (62)] v98'397 uv397 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7fe68c00b040 con 0x55bab64f65b0 2024-08-03T20:56:38.503 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:38.502+0000 7fe698dcbf40 1 -- 172.21.15.32:0/4237013562 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:18 12.10 12:0cf6ad0f:::10000000000.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98) -- 0x55bab659d4a0 con 0x55bab64f65b0 2024-08-03T20:56:38.504 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:38.504+0000 7fe692a00640 1 -- 172.21.15.32:0/4237013562 <== osd.0 v2:172.21.15.32:6816/3066156116 18 ==== osd_op_reply(18 10000000000.00000000 [omap-set-header] v98'398 uv398 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7fe68c00b040 con 0x55bab64f65b0 2024-08-03T20:56:38.504 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:38.504+0000 7fe698dcbf40 1 -- 172.21.15.32:0/4237013562 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:19 12.10 12:0cf6ad0f:::10000000000.00000000:head [omap-set-vals in=593b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98) -- 0x55bab659e460 con 0x55bab64f65b0 2024-08-03T20:56:38.506 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:38.505+0000 7fe692a00640 1 -- 172.21.15.32:0/4237013562 <== osd.0 v2:172.21.15.32:6816/3066156116 19 ==== osd_op_reply(19 10000000000.00000000 [omap-set-vals] v98'399 uv399 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7fe68c00b040 con 0x55bab64f65b0 2024-08-03T20:56:38.506 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:38.505+0000 7fe698dcbf40 1 -- 172.21.15.32:0/4237013562 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:20 12.10 12:0d82a743:::600.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98) -- 0x55bab659e460 con 0x55bab64f65b0 2024-08-03T20:56:38.507 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:38.507+0000 7fe692a00640 1 -- 172.21.15.32:0/4237013562 <== osd.0 v2:172.21.15.32:6816/3066156116 20 ==== osd_op_reply(20 600.00000000 [create] v98'400 uv374 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fe68c00b040 con 0x55bab64f65b0 2024-08-03T20:56:38.507 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:38.507+0000 7fe698dcbf40 1 -- 172.21.15.32:0/4237013562 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:21 12.10 12:0d82a743:::600.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e98) -- 0x55bab659e460 con 0x55bab64f65b0 2024-08-03T20:56:38.509 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:38.508+0000 7fe692a00640 1 -- 172.21.15.32:0/4237013562 <== osd.0 v2:172.21.15.32:6816/3066156116 21 ==== osd_op_reply(21 600.00000000 [delete] v98'401 uv401 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe68c00b040 con 0x55bab64f65b0 2024-08-03T20:56:38.509 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:38.508+0000 7fe698dcbf40 1 -- 172.21.15.32:0/4237013562 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:22 12.10 12:0d82a743:::600.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98) -- 0x55bab659e460 con 0x55bab64f65b0 2024-08-03T20:56:38.510 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::600.00000000:head# 2024-08-03T20:56:38.510 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:38.510+0000 7fe692a00640 1 -- 172.21.15.32:0/4237013562 <== osd.0 v2:172.21.15.32:6816/3066156116 22 ==== osd_op_reply(22 600.00000000 [create] v98'402 uv402 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe68c00b040 con 0x55bab64f65b0 2024-08-03T20:56:38.510 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:38.510+0000 7fe698dcbf40 1 -- 172.21.15.32:0/4237013562 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:23 12.10 12:0d82a743:::600.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98) -- 0x55bab659f470 con 0x55bab64f65b0 2024-08-03T20:56:38.511 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:38.511+0000 7fe692a00640 1 -- 172.21.15.32:0/4237013562 <== osd.0 v2:172.21.15.32:6816/3066156116 23 ==== osd_op_reply(23 600.00000000 [setxattr (30)] v98'403 uv403 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe68c00b040 con 0x55bab64f65b0 2024-08-03T20:56:38.512 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:38.511+0000 7fe698dcbf40 1 -- 172.21.15.32:0/4237013562 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:24 12.10 12:0d82a743:::600.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98) -- 0x55bab659f470 con 0x55bab64f65b0 2024-08-03T20:56:38.513 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:38.513+0000 7fe692a00640 1 -- 172.21.15.32:0/4237013562 <== osd.0 v2:172.21.15.32:6816/3066156116 24 ==== osd_op_reply(24 600.00000000 [setxattr (62)] v98'404 uv404 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe68c00b040 con 0x55bab64f65b0 2024-08-03T20:56:38.513 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:38.513+0000 7fe698dcbf40 1 -- 172.21.15.32:0/4237013562 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:25 12.10 12:0d82a743:::600.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98) -- 0x55bab659e460 con 0x55bab64f65b0 2024-08-03T20:56:38.514 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:38.514+0000 7fe692a00640 1 -- 172.21.15.32:0/4237013562 <== osd.0 v2:172.21.15.32:6816/3066156116 25 ==== osd_op_reply(25 600.00000000 [omap-set-header] v98'405 uv405 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe68c00b040 con 0x55bab64f65b0 2024-08-03T20:56:38.515 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:38.514+0000 7fe698dcbf40 1 -- 172.21.15.32:0/4237013562 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:26 12.10 12:0d89b25e:::603.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98) -- 0x55bab659e460 con 0x55bab64f65b0 2024-08-03T20:56:38.516 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:38.516+0000 7fe692a00640 1 -- 172.21.15.32:0/4237013562 <== osd.0 v2:172.21.15.32:6816/3066156116 26 ==== osd_op_reply(26 603.00000000 [create] v98'406 uv380 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fe68c00b040 con 0x55bab64f65b0 2024-08-03T20:56:38.516 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:38.516+0000 7fe698dcbf40 1 -- 172.21.15.32:0/4237013562 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:27 12.10 12:0d89b25e:::603.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e98) -- 0x55bab659e460 con 0x55bab64f65b0 2024-08-03T20:56:38.518 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:38.517+0000 7fe692a00640 1 -- 172.21.15.32:0/4237013562 <== osd.0 v2:172.21.15.32:6816/3066156116 27 ==== osd_op_reply(27 603.00000000 [delete] v98'407 uv407 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe68c00b040 con 0x55bab64f65b0 2024-08-03T20:56:38.518 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:38.517+0000 7fe698dcbf40 1 -- 172.21.15.32:0/4237013562 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:28 12.10 12:0d89b25e:::603.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98) -- 0x55bab659e460 con 0x55bab64f65b0 2024-08-03T20:56:38.519 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::603.00000000:head# 2024-08-03T20:56:38.519 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:38.519+0000 7fe692a00640 1 -- 172.21.15.32:0/4237013562 <== osd.0 v2:172.21.15.32:6816/3066156116 28 ==== osd_op_reply(28 603.00000000 [create] v98'408 uv408 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe68c00b040 con 0x55bab64f65b0 2024-08-03T20:56:38.520 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:38.519+0000 7fe698dcbf40 1 -- 172.21.15.32:0/4237013562 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:29 12.10 12:0d89b25e:::603.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98) -- 0x55bab659f470 con 0x55bab64f65b0 2024-08-03T20:56:38.521 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:38.520+0000 7fe692a00640 1 -- 172.21.15.32:0/4237013562 <== osd.0 v2:172.21.15.32:6816/3066156116 29 ==== osd_op_reply(29 603.00000000 [setxattr (30)] v98'409 uv409 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe68c00b040 con 0x55bab64f65b0 2024-08-03T20:56:38.521 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:38.520+0000 7fe698dcbf40 1 -- 172.21.15.32:0/4237013562 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:30 12.10 12:0d89b25e:::603.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98) -- 0x55bab659f470 con 0x55bab64f65b0 2024-08-03T20:56:38.522 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:38.522+0000 7fe692a00640 1 -- 172.21.15.32:0/4237013562 <== osd.0 v2:172.21.15.32:6816/3066156116 30 ==== osd_op_reply(30 603.00000000 [setxattr (62)] v98'410 uv410 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe68c00b040 con 0x55bab64f65b0 2024-08-03T20:56:38.522 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:38.522+0000 7fe698dcbf40 1 -- 172.21.15.32:0/4237013562 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:31 12.10 12:0d89b25e:::603.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98) -- 0x55bab659e460 con 0x55bab64f65b0 2024-08-03T20:56:38.524 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:38.523+0000 7fe692a00640 1 -- 172.21.15.32:0/4237013562 <== osd.0 v2:172.21.15.32:6816/3066156116 31 ==== osd_op_reply(31 603.00000000 [omap-set-header] v98'411 uv411 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe68c00b040 con 0x55bab64f65b0 2024-08-03T20:56:38.524 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:38.523+0000 7fe698dcbf40 1 --2- 172.21.15.32:0/4237013562 >> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] conn(0x55bab659e970 0x55bab65a0d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:56:38.524 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:38.523+0000 7fe698dcbf40 1 -- 172.21.15.32:0/4237013562 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:32 12.14 12:287034c4:::1.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98) -- 0x55bab65a12d0 con 0x55bab659e970 2024-08-03T20:56:38.524 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:38.524+0000 7fe692000640 1 --2- 172.21.15.32:0/4237013562 >> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] conn(0x55bab659e970 0x55bab65a0d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:56:38.525 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:38.524+0000 7fe692000640 1 --2- 172.21.15.32:0/4237013562 >> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] conn(0x55bab659e970 0x55bab65a0d90 crc :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:56:38.529 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:38.529+0000 7fe692000640 1 -- 172.21.15.32:0/4237013562 <== osd.5 v2:172.21.15.118:6810/2724891770 1 ==== osd_op_reply(32 1.00000000.inode [create] v98'287 uv266 ondisk = -17 ((17) File exists)) ==== 160+0+0 (crc 0 0 0) 0x7fe684012930 con 0x55bab659e970 2024-08-03T20:56:38.530 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:38.529+0000 7fe698dcbf40 1 -- 172.21.15.32:0/4237013562 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:33 12.14 12:287034c4:::1.00000000.inode:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e98) -- 0x55bab65a1670 con 0x55bab659e970 2024-08-03T20:56:38.531 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:38.531+0000 7fe692000640 1 -- 172.21.15.32:0/4237013562 <== osd.5 v2:172.21.15.118:6810/2724891770 2 ==== osd_op_reply(33 1.00000000.inode [delete] v98'288 uv288 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7fe684012930 con 0x55bab659e970 2024-08-03T20:56:38.531 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:38.531+0000 7fe698dcbf40 1 -- 172.21.15.32:0/4237013562 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:34 12.14 12:287034c4:::1.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98) -- 0x55bab65a1a10 con 0x55bab659e970 2024-08-03T20:56:38.533 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:38.533+0000 7fe692000640 1 -- 172.21.15.32:0/4237013562 <== osd.5 v2:172.21.15.118:6810/2724891770 3 ==== osd_op_reply(34 1.00000000.inode [create] v98'289 uv289 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7fe684012930 con 0x55bab659e970 2024-08-03T20:56:38.533 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::1.00000000.inode:head# 2024-08-03T20:56:38.533 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:38.533+0000 7fe698dcbf40 1 -- 172.21.15.32:0/4237013562 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:35 12.14 12:287034c4:::1.00000000.inode:head [write 0~582 in=582b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98) -- 0x55bab65a1db0 con 0x55bab659e970 2024-08-03T20:56:38.535 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:38.534+0000 7fe692000640 1 -- 172.21.15.32:0/4237013562 <== osd.5 v2:172.21.15.118:6810/2724891770 4 ==== osd_op_reply(35 1.00000000.inode [write 0~582] v98'290 uv290 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7fe684012930 con 0x55bab659e970 2024-08-03T20:56:38.535 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:38.534+0000 7fe698dcbf40 1 -- 172.21.15.32:0/4237013562 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:36 12.14 12:287034c4:::1.00000000.inode:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98) -- 0x55bab65a2150 con 0x55bab659e970 2024-08-03T20:56:38.536 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:38.536+0000 7fe692000640 1 -- 172.21.15.32:0/4237013562 <== osd.5 v2:172.21.15.118:6810/2724891770 5 ==== osd_op_reply(36 1.00000000.inode [omap-set-header] v98'291 uv291 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7fe684012930 con 0x55bab659e970 2024-08-03T20:56:38.536 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:38.536+0000 7fe698dcbf40 1 -- 172.21.15.32:0/4237013562 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:37 12.14 12:292cf221:::200.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98) -- 0x55bab65a24f0 con 0x55bab659e970 2024-08-03T20:56:38.538 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:38.538+0000 7fe692000640 1 -- 172.21.15.32:0/4237013562 <== osd.5 v2:172.21.15.118:6810/2724891770 6 ==== osd_op_reply(37 200.00000000 [create] v98'292 uv285 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fe684012930 con 0x55bab659e970 2024-08-03T20:56:38.538 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:38.538+0000 7fe698dcbf40 1 -- 172.21.15.32:0/4237013562 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:38 12.14 12:292cf221:::200.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e98) -- 0x55bab65a2890 con 0x55bab659e970 2024-08-03T20:56:38.540 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:38.539+0000 7fe692000640 1 -- 172.21.15.32:0/4237013562 <== osd.5 v2:172.21.15.118:6810/2724891770 7 ==== osd_op_reply(38 200.00000000 [delete] v98'293 uv293 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe684012930 con 0x55bab659e970 2024-08-03T20:56:38.540 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:38.539+0000 7fe698dcbf40 1 -- 172.21.15.32:0/4237013562 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:39 12.14 12:292cf221:::200.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98) -- 0x55bab65a2c30 con 0x55bab659e970 2024-08-03T20:56:38.541 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::200.00000000:head# 2024-08-03T20:56:38.541 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:38.541+0000 7fe692000640 1 -- 172.21.15.32:0/4237013562 <== osd.5 v2:172.21.15.118:6810/2724891770 8 ==== osd_op_reply(39 200.00000000 [create] v98'294 uv294 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe684012930 con 0x55bab659e970 2024-08-03T20:56:38.541 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:38.541+0000 7fe698dcbf40 1 -- 172.21.15.32:0/4237013562 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:40 12.14 12:292cf221:::200.00000000:head [write 0~90 in=90b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98) -- 0x55bab65a2c30 con 0x55bab659e970 2024-08-03T20:56:38.543 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:38.542+0000 7fe692000640 1 -- 172.21.15.32:0/4237013562 <== osd.5 v2:172.21.15.118:6810/2724891770 9 ==== osd_op_reply(40 200.00000000 [write 0~90] v98'295 uv295 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe684012930 con 0x55bab659e970 2024-08-03T20:56:38.543 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:38.542+0000 7fe698dcbf40 1 -- 172.21.15.32:0/4237013562 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:41 12.14 12:292cf221:::200.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98) -- 0x55bab65a2c30 con 0x55bab659e970 2024-08-03T20:56:38.544 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:38.544+0000 7fe692000640 1 -- 172.21.15.32:0/4237013562 <== osd.5 v2:172.21.15.118:6810/2724891770 10 ==== osd_op_reply(41 200.00000000 [omap-set-header] v98'296 uv296 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe684012930 con 0x55bab659e970 2024-08-03T20:56:38.544 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:38.544+0000 7fe698dcbf40 1 -- 172.21.15.32:0/4237013562 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:42 12.14 12:2e2fa760:::200.00000001:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98) -- 0x55bab65a2c30 con 0x55bab659e970 2024-08-03T20:56:38.545 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:38.545+0000 7fe692000640 1 -- 172.21.15.32:0/4237013562 <== osd.5 v2:172.21.15.118:6810/2724891770 11 ==== osd_op_reply(42 200.00000001 [create] v98'297 uv286 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fe684012930 con 0x55bab659e970 2024-08-03T20:56:38.546 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:38.545+0000 7fe698dcbf40 1 -- 172.21.15.32:0/4237013562 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:43 12.14 12:2e2fa760:::200.00000001:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e98) -- 0x55bab65a2c30 con 0x55bab659e970 2024-08-03T20:56:38.547 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:38.547+0000 7fe692000640 1 -- 172.21.15.32:0/4237013562 <== osd.5 v2:172.21.15.118:6810/2724891770 12 ==== osd_op_reply(43 200.00000001 [delete] v98'298 uv298 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe684012930 con 0x55bab659e970 2024-08-03T20:56:38.547 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:38.547+0000 7fe698dcbf40 1 -- 172.21.15.32:0/4237013562 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:44 12.14 12:2e2fa760:::200.00000001:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98) -- 0x55bab65a2c30 con 0x55bab659e970 2024-08-03T20:56:38.549 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:38.549+0000 7fe692000640 1 -- 172.21.15.32:0/4237013562 <== osd.5 v2:172.21.15.118:6810/2724891770 13 ==== osd_op_reply(44 200.00000001 [create] v98'299 uv299 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe684012930 con 0x55bab659e970 2024-08-03T20:56:38.549 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::200.00000001:head# 2024-08-03T20:56:38.549 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:38.549+0000 7fe698dcbf40 1 -- 172.21.15.32:0/4237013562 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:45 12.14 12:2e2fa760:::200.00000001:head [write 0~18118 in=18118b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98) -- 0x55bab65a7320 con 0x55bab659e970 2024-08-03T20:56:38.551 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:38.551+0000 7fe692000640 1 -- 172.21.15.32:0/4237013562 <== osd.5 v2:172.21.15.118:6810/2724891770 14 ==== osd_op_reply(45 200.00000001 [write 0~18118] v98'300 uv300 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe684012930 con 0x55bab659e970 2024-08-03T20:56:38.551 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:38.551+0000 7fe698dcbf40 1 -- 172.21.15.32:0/4237013562 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:46 12.14 12:2e2fa760:::200.00000001:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98) -- 0x55bab65a7320 con 0x55bab659e970 2024-08-03T20:56:38.552 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:38.552+0000 7fe692000640 1 -- 172.21.15.32:0/4237013562 <== osd.5 v2:172.21.15.118:6810/2724891770 15 ==== osd_op_reply(46 200.00000001 [omap-set-header] v98'301 uv301 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe684012930 con 0x55bab659e970 2024-08-03T20:56:38.552 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:38.552+0000 7fe698dcbf40 1 -- 172.21.15.32:0/4237013562 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:47 12.14 12:2f55791f:::606.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98) -- 0x55bab65a2c30 con 0x55bab659e970 2024-08-03T20:56:38.554 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:38.553+0000 7fe692000640 1 -- 172.21.15.32:0/4237013562 <== osd.5 v2:172.21.15.118:6810/2724891770 16 ==== osd_op_reply(47 606.00000000 [create] v98'302 uv282 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fe684012930 con 0x55bab659e970 2024-08-03T20:56:38.554 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:38.553+0000 7fe698dcbf40 1 -- 172.21.15.32:0/4237013562 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:48 12.14 12:2f55791f:::606.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e98) -- 0x55bab65a2c30 con 0x55bab659e970 2024-08-03T20:56:38.555 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:38.555+0000 7fe692000640 1 -- 172.21.15.32:0/4237013562 <== osd.5 v2:172.21.15.118:6810/2724891770 17 ==== osd_op_reply(48 606.00000000 [delete] v98'303 uv303 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe684012930 con 0x55bab659e970 2024-08-03T20:56:38.555 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:38.555+0000 7fe698dcbf40 1 -- 172.21.15.32:0/4237013562 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:49 12.14 12:2f55791f:::606.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98) -- 0x55bab65a2c30 con 0x55bab659e970 2024-08-03T20:56:38.557 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::606.00000000:head# 2024-08-03T20:56:38.557 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:38.556+0000 7fe692000640 1 -- 172.21.15.32:0/4237013562 <== osd.5 v2:172.21.15.118:6810/2724891770 18 ==== osd_op_reply(49 606.00000000 [create] v98'304 uv304 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe684012930 con 0x55bab659e970 2024-08-03T20:56:38.557 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:38.556+0000 7fe698dcbf40 1 -- 172.21.15.32:0/4237013562 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:50 12.14 12:2f55791f:::606.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98) -- 0x55bab65a3c40 con 0x55bab659e970 2024-08-03T20:56:38.558 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:38.558+0000 7fe692000640 1 -- 172.21.15.32:0/4237013562 <== osd.5 v2:172.21.15.118:6810/2724891770 19 ==== osd_op_reply(50 606.00000000 [setxattr (30)] v98'305 uv305 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe684012930 con 0x55bab659e970 2024-08-03T20:56:38.558 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:38.558+0000 7fe698dcbf40 1 -- 172.21.15.32:0/4237013562 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:51 12.14 12:2f55791f:::606.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98) -- 0x55bab65a3c40 con 0x55bab659e970 2024-08-03T20:56:38.560 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:38.559+0000 7fe692000640 1 -- 172.21.15.32:0/4237013562 <== osd.5 v2:172.21.15.118:6810/2724891770 20 ==== osd_op_reply(51 606.00000000 [setxattr (62)] v98'306 uv306 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe684012930 con 0x55bab659e970 2024-08-03T20:56:38.560 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:38.559+0000 7fe698dcbf40 1 -- 172.21.15.32:0/4237013562 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:52 12.14 12:2f55791f:::606.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98) -- 0x55bab65a2c30 con 0x55bab659e970 2024-08-03T20:56:38.561 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:38.561+0000 7fe692000640 1 -- 172.21.15.32:0/4237013562 <== osd.5 v2:172.21.15.118:6810/2724891770 21 ==== osd_op_reply(52 606.00000000 [omap-set-header] v98'307 uv307 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe684012930 con 0x55bab659e970 2024-08-03T20:56:38.561 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:38.561+0000 7fe698dcbf40 1 -- 172.21.15.32:0/4237013562 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:53 12.1c 12:3ed09add:::607.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98) -- 0x55bab65a2c30 con 0x55bab64f65b0 2024-08-03T20:56:38.563 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:38.562+0000 7fe692a00640 1 -- 172.21.15.32:0/4237013562 <== osd.0 v2:172.21.15.32:6816/3066156116 32 ==== osd_op_reply(53 607.00000000 [create] v98'75 uv74 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fe68c00b040 con 0x55bab64f65b0 2024-08-03T20:56:38.563 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:38.562+0000 7fe698dcbf40 1 -- 172.21.15.32:0/4237013562 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:54 12.1c 12:3ed09add:::607.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e98) -- 0x55bab65a2c30 con 0x55bab64f65b0 2024-08-03T20:56:38.564 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:38.564+0000 7fe692a00640 1 -- 172.21.15.32:0/4237013562 <== osd.0 v2:172.21.15.32:6816/3066156116 33 ==== osd_op_reply(54 607.00000000 [delete] v98'76 uv76 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe68c00b040 con 0x55bab64f65b0 2024-08-03T20:56:38.564 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:38.564+0000 7fe698dcbf40 1 -- 172.21.15.32:0/4237013562 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:55 12.1c 12:3ed09add:::607.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98) -- 0x55bab65a2c30 con 0x55bab64f65b0 2024-08-03T20:56:38.566 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::607.00000000:head# 2024-08-03T20:56:38.566 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:38.565+0000 7fe692a00640 1 -- 172.21.15.32:0/4237013562 <== osd.0 v2:172.21.15.32:6816/3066156116 34 ==== osd_op_reply(55 607.00000000 [create] v98'77 uv77 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe68c00b040 con 0x55bab64f65b0 2024-08-03T20:56:38.566 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:38.565+0000 7fe698dcbf40 1 -- 172.21.15.32:0/4237013562 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:56 12.1c 12:3ed09add:::607.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98) -- 0x55bab65a3c40 con 0x55bab64f65b0 2024-08-03T20:56:38.567 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:38.567+0000 7fe692a00640 1 -- 172.21.15.32:0/4237013562 <== osd.0 v2:172.21.15.32:6816/3066156116 35 ==== osd_op_reply(56 607.00000000 [setxattr (30)] v98'78 uv78 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe68c00b040 con 0x55bab64f65b0 2024-08-03T20:56:38.567 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:38.567+0000 7fe698dcbf40 1 -- 172.21.15.32:0/4237013562 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:57 12.1c 12:3ed09add:::607.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98) -- 0x55bab65a3c40 con 0x55bab64f65b0 2024-08-03T20:56:38.568 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:38.568+0000 7fe692a00640 1 -- 172.21.15.32:0/4237013562 <== osd.0 v2:172.21.15.32:6816/3066156116 36 ==== osd_op_reply(57 607.00000000 [setxattr (62)] v98'79 uv79 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe68c00b040 con 0x55bab64f65b0 2024-08-03T20:56:38.569 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:38.568+0000 7fe698dcbf40 1 -- 172.21.15.32:0/4237013562 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:58 12.1c 12:3ed09add:::607.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98) -- 0x55bab65a2c30 con 0x55bab64f65b0 2024-08-03T20:56:38.570 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:38.570+0000 7fe692a00640 1 -- 172.21.15.32:0/4237013562 <== osd.0 v2:172.21.15.32:6816/3066156116 37 ==== osd_op_reply(58 607.00000000 [omap-set-header] v98'80 uv80 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe68c00b040 con 0x55bab64f65b0 2024-08-03T20:56:38.570 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:38.570+0000 7fe698dcbf40 1 --2- 172.21.15.32:0/4237013562 >> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] conn(0x55bab65a3140 0x55bab65a5560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:56:38.570 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:38.570+0000 7fe698dcbf40 1 -- 172.21.15.32:0/4237013562 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:59 12.6 12:654134d2:::mds0_openfiles.0:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98) -- 0x55bab65a5ae0 con 0x55bab65a3140 2024-08-03T20:56:38.571 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:38.570+0000 7fe691600640 1 --2- 172.21.15.32:0/4237013562 >> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] conn(0x55bab65a3140 0x55bab65a5560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:56:38.571 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:38.571+0000 7fe691600640 1 --2- 172.21.15.32:0/4237013562 >> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] conn(0x55bab65a3140 0x55bab65a5560 crc :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:56:38.572 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:38.572+0000 7fe691600640 1 -- 172.21.15.32:0/4237013562 <== osd.1 v2:172.21.15.32:6824/1551241759 1 ==== osd_op_reply(59 mds0_openfiles.0 [create] v98'66 uv65 ondisk = -17 ((17) File exists)) ==== 160+0+0 (crc 0 0 0) 0x7fe680027070 con 0x55bab65a3140 2024-08-03T20:56:38.572 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:38.572+0000 7fe698dcbf40 1 -- 172.21.15.32:0/4237013562 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:60 12.6 12:654134d2:::mds0_openfiles.0:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e98) -- 0x55bab65a5fd0 con 0x55bab65a3140 2024-08-03T20:56:38.573 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:38.573+0000 7fe691600640 1 -- 172.21.15.32:0/4237013562 <== osd.1 v2:172.21.15.32:6824/1551241759 2 ==== osd_op_reply(60 mds0_openfiles.0 [delete] v98'67 uv67 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7fe680027070 con 0x55bab65a3140 2024-08-03T20:56:38.574 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:38.573+0000 7fe698dcbf40 1 -- 172.21.15.32:0/4237013562 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:61 12.6 12:654134d2:::mds0_openfiles.0:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98) -- 0x55bab65a64d0 con 0x55bab65a3140 2024-08-03T20:56:38.575 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds0_openfiles.0:head# 2024-08-03T20:56:38.575 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:38.574+0000 7fe691600640 1 -- 172.21.15.32:0/4237013562 <== osd.1 v2:172.21.15.32:6824/1551241759 3 ==== osd_op_reply(61 mds0_openfiles.0 [create] v98'68 uv68 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7fe680027070 con 0x55bab65a3140 2024-08-03T20:56:38.576 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:38.574+0000 7fe698dcbf40 1 -- 172.21.15.32:0/4237013562 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:62 12.6 12:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98) -- 0x55bab64fa1a0 con 0x55bab65a3140 2024-08-03T20:56:38.576 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:38.576+0000 7fe691600640 1 -- 172.21.15.32:0/4237013562 <== osd.1 v2:172.21.15.32:6824/1551241759 4 ==== osd_op_reply(62 mds0_openfiles.0 [omap-set-header] v98'69 uv69 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7fe680027070 con 0x55bab65a3140 2024-08-03T20:56:38.576 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:38.576+0000 7fe698dcbf40 1 -- 172.21.15.32:0/4237013562 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:63 12.e 12:717a0223:::608.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98) -- 0x55bab65a6e00 con 0x55bab64f65b0 2024-08-03T20:56:38.578 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:38.578+0000 7fe692a00640 1 -- 172.21.15.32:0/4237013562 <== osd.0 v2:172.21.15.32:6816/3066156116 38 ==== osd_op_reply(63 608.00000000 [create] v98'75 uv74 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fe68c00b040 con 0x55bab64f65b0 2024-08-03T20:56:38.578 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:38.578+0000 7fe698dcbf40 1 -- 172.21.15.32:0/4237013562 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:64 12.e 12:717a0223:::608.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e98) -- 0x55bab65a6e00 con 0x55bab64f65b0 2024-08-03T20:56:38.580 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:38.579+0000 7fe692a00640 1 -- 172.21.15.32:0/4237013562 <== osd.0 v2:172.21.15.32:6816/3066156116 39 ==== osd_op_reply(64 608.00000000 [delete] v98'76 uv76 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe68c00b040 con 0x55bab64f65b0 2024-08-03T20:56:38.580 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:38.579+0000 7fe698dcbf40 1 -- 172.21.15.32:0/4237013562 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:65 12.e 12:717a0223:::608.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98) -- 0x55bab65a6e00 con 0x55bab64f65b0 2024-08-03T20:56:38.581 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::608.00000000:head# 2024-08-03T20:56:38.581 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:38.581+0000 7fe692a00640 1 -- 172.21.15.32:0/4237013562 <== osd.0 v2:172.21.15.32:6816/3066156116 40 ==== osd_op_reply(65 608.00000000 [create] v98'77 uv77 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe68c00b040 con 0x55bab64f65b0 2024-08-03T20:56:38.581 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:38.581+0000 7fe698dcbf40 1 -- 172.21.15.32:0/4237013562 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:66 12.e 12:717a0223:::608.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98) -- 0x55bab65a6e00 con 0x55bab64f65b0 2024-08-03T20:56:38.583 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:38.583+0000 7fe692a00640 1 -- 172.21.15.32:0/4237013562 <== osd.0 v2:172.21.15.32:6816/3066156116 41 ==== osd_op_reply(66 608.00000000 [setxattr (30)] v98'78 uv78 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe68c00b040 con 0x55bab64f65b0 2024-08-03T20:56:38.583 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:38.583+0000 7fe698dcbf40 1 -- 172.21.15.32:0/4237013562 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:67 12.e 12:717a0223:::608.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98) -- 0x55bab65a6e00 con 0x55bab64f65b0 2024-08-03T20:56:38.584 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:38.584+0000 7fe692a00640 1 -- 172.21.15.32:0/4237013562 <== osd.0 v2:172.21.15.32:6816/3066156116 42 ==== osd_op_reply(67 608.00000000 [setxattr (62)] v98'79 uv79 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe68c00b040 con 0x55bab64f65b0 2024-08-03T20:56:38.584 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:38.584+0000 7fe698dcbf40 1 -- 172.21.15.32:0/4237013562 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:68 12.e 12:717a0223:::608.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98) -- 0x55bab65a6e00 con 0x55bab64f65b0 2024-08-03T20:56:38.586 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:38.586+0000 7fe692a00640 1 -- 172.21.15.32:0/4237013562 <== osd.0 v2:172.21.15.32:6816/3066156116 43 ==== osd_op_reply(68 608.00000000 [omap-set-header] v98'80 uv80 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe68c00b040 con 0x55bab64f65b0 2024-08-03T20:56:38.586 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:38.586+0000 7fe698dcbf40 1 --2- 172.21.15.32:0/4237013562 >> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] conn(0x55bab65a7700 0x55bab65a7b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:56:38.586 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:38.586+0000 7fe698dcbf40 1 -- 172.21.15.32:0/4237013562 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:69 12.15 12:a93a17c2:::604.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98) -- 0x55bab65aa230 con 0x55bab65a7700 2024-08-03T20:56:38.587 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:38.586+0000 7fe692a00640 1 --2- 172.21.15.32:0/4237013562 >> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] conn(0x55bab65a7700 0x55bab65a7b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:56:38.587 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:38.587+0000 7fe692a00640 1 --2- 172.21.15.32:0/4237013562 >> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] conn(0x55bab65a7700 0x55bab65a7b10 crc :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:56:38.589 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:38.588+0000 7fe692a00640 1 -- 172.21.15.32:0/4237013562 <== osd.6 v2:172.21.15.118:6825/801947386 1 ==== osd_op_reply(69 604.00000000 [create] v98'138 uv132 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fe68c00b040 con 0x55bab65a7700 2024-08-03T20:56:38.589 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:38.588+0000 7fe698dcbf40 1 -- 172.21.15.32:0/4237013562 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:70 12.15 12:a93a17c2:::604.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e98) -- 0x55bab65aa230 con 0x55bab65a7700 2024-08-03T20:56:38.590 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:38.590+0000 7fe692a00640 1 -- 172.21.15.32:0/4237013562 <== osd.6 v2:172.21.15.118:6825/801947386 2 ==== osd_op_reply(70 604.00000000 [delete] v98'139 uv139 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe68c00b040 con 0x55bab65a7700 2024-08-03T20:56:38.590 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:38.590+0000 7fe698dcbf40 1 -- 172.21.15.32:0/4237013562 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:71 12.15 12:a93a17c2:::604.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98) -- 0x55bab65aa230 con 0x55bab65a7700 2024-08-03T20:56:38.592 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::604.00000000:head# 2024-08-03T20:56:38.592 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:38.591+0000 7fe692a00640 1 -- 172.21.15.32:0/4237013562 <== osd.6 v2:172.21.15.118:6825/801947386 3 ==== osd_op_reply(71 604.00000000 [create] v98'140 uv140 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe68c00b040 con 0x55bab65a7700 2024-08-03T20:56:38.592 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:38.591+0000 7fe698dcbf40 1 -- 172.21.15.32:0/4237013562 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:72 12.15 12:a93a17c2:::604.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98) -- 0x55bab65ab630 con 0x55bab65a7700 2024-08-03T20:56:38.593 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:38.593+0000 7fe692a00640 1 -- 172.21.15.32:0/4237013562 <== osd.6 v2:172.21.15.118:6825/801947386 4 ==== osd_op_reply(72 604.00000000 [setxattr (30)] v98'141 uv141 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe68c00b040 con 0x55bab65a7700 2024-08-03T20:56:38.594 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:38.593+0000 7fe698dcbf40 1 -- 172.21.15.32:0/4237013562 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:73 12.15 12:a93a17c2:::604.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98) -- 0x55bab65ab630 con 0x55bab65a7700 2024-08-03T20:56:38.595 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:38.595+0000 7fe692a00640 1 -- 172.21.15.32:0/4237013562 <== osd.6 v2:172.21.15.118:6825/801947386 5 ==== osd_op_reply(73 604.00000000 [setxattr (62)] v98'142 uv142 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe68c00b040 con 0x55bab65a7700 2024-08-03T20:56:38.595 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:38.595+0000 7fe698dcbf40 1 -- 172.21.15.32:0/4237013562 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:74 12.15 12:a93a17c2:::604.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98) -- 0x55bab65aa620 con 0x55bab65a7700 2024-08-03T20:56:38.597 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:38.596+0000 7fe692a00640 1 -- 172.21.15.32:0/4237013562 <== osd.6 v2:172.21.15.118:6825/801947386 6 ==== osd_op_reply(74 604.00000000 [omap-set-header] v98'143 uv143 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe68c00b040 con 0x55bab65a7700 2024-08-03T20:56:38.597 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:38.596+0000 7fe698dcbf40 1 -- 172.21.15.32:0/4237013562 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:75 12.15 12:aa448500:::500.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98) -- 0x55bab65aa620 con 0x55bab65a7700 2024-08-03T20:56:38.598 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:38.598+0000 7fe692a00640 1 -- 172.21.15.32:0/4237013562 <== osd.6 v2:172.21.15.118:6825/801947386 7 ==== osd_op_reply(75 500.00000000 [create] v98'144 uv137 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fe68c00b040 con 0x55bab65a7700 2024-08-03T20:56:38.598 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:38.598+0000 7fe698dcbf40 1 -- 172.21.15.32:0/4237013562 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:76 12.15 12:aa448500:::500.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e98) -- 0x55bab65aa620 con 0x55bab65a7700 2024-08-03T20:56:38.600 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:38.599+0000 7fe692a00640 1 -- 172.21.15.32:0/4237013562 <== osd.6 v2:172.21.15.118:6825/801947386 8 ==== osd_op_reply(76 500.00000000 [delete] v98'145 uv145 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe68c00b040 con 0x55bab65a7700 2024-08-03T20:56:38.600 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:38.599+0000 7fe698dcbf40 1 -- 172.21.15.32:0/4237013562 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:77 12.15 12:aa448500:::500.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98) -- 0x55bab65aa620 con 0x55bab65a7700 2024-08-03T20:56:38.601 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::500.00000000:head# 2024-08-03T20:56:38.601 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:38.600+0000 7fe692a00640 1 -- 172.21.15.32:0/4237013562 <== osd.6 v2:172.21.15.118:6825/801947386 9 ==== osd_op_reply(77 500.00000000 [create] v98'146 uv146 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe68c00b040 con 0x55bab65a7700 2024-08-03T20:56:38.601 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:38.600+0000 7fe698dcbf40 1 -- 172.21.15.32:0/4237013562 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:78 12.15 12:aa448500:::500.00000000:head [write 0~90 in=90b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98) -- 0x55bab65aa620 con 0x55bab65a7700 2024-08-03T20:56:38.602 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:38.602+0000 7fe692a00640 1 -- 172.21.15.32:0/4237013562 <== osd.6 v2:172.21.15.118:6825/801947386 10 ==== osd_op_reply(78 500.00000000 [write 0~90] v98'147 uv147 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe68c00b040 con 0x55bab65a7700 2024-08-03T20:56:38.602 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:38.602+0000 7fe698dcbf40 1 -- 172.21.15.32:0/4237013562 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:79 12.15 12:aa448500:::500.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98) -- 0x55bab65aa620 con 0x55bab65a7700 2024-08-03T20:56:38.603 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:38.603+0000 7fe692a00640 1 -- 172.21.15.32:0/4237013562 <== osd.6 v2:172.21.15.118:6825/801947386 11 ==== osd_op_reply(79 500.00000000 [omap-set-header] v98'148 uv148 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe68c00b040 con 0x55bab65a7700 2024-08-03T20:56:38.603 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:38.603+0000 7fe698dcbf40 1 --2- 172.21.15.32:0/4237013562 >> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] conn(0x55bab65aab30 0x55bab65acf50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:56:38.604 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:38.603+0000 7fe698dcbf40 1 -- 172.21.15.32:0/4237013562 --> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] -- osd_op(unknown.0.0:80 12.d 12:b50e409b:::mds_snaptable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98) -- 0x55bab65ad560 con 0x55bab65aab30 2024-08-03T20:56:38.604 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:38.603+0000 7fe692000640 1 --2- 172.21.15.32:0/4237013562 >> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] conn(0x55bab65aab30 0x55bab65acf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:56:38.604 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:38.604+0000 7fe692000640 1 --2- 172.21.15.32:0/4237013562 >> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] conn(0x55bab65aab30 0x55bab65acf50 crc :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:56:38.606 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:38.606+0000 7fe692000640 1 -- 172.21.15.32:0/4237013562 <== osd.7 v2:172.21.15.118:6818/653000048 1 ==== osd_op_reply(80 mds_snaptable [create] v98'61 uv60 ondisk = -17 ((17) File exists)) ==== 157+0+0 (crc 0 0 0) 0x7fe684012930 con 0x55bab65aab30 2024-08-03T20:56:38.606 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:38.606+0000 7fe698dcbf40 1 -- 172.21.15.32:0/4237013562 --> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] -- osd_op(unknown.0.0:81 12.d 12:b50e409b:::mds_snaptable:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e98) -- 0x55bab65ad930 con 0x55bab65aab30 2024-08-03T20:56:38.607 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:38.607+0000 7fe692000640 1 -- 172.21.15.32:0/4237013562 <== osd.7 v2:172.21.15.118:6818/653000048 2 ==== osd_op_reply(81 mds_snaptable [delete] v98'62 uv62 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7fe684012930 con 0x55bab65aab30 2024-08-03T20:56:38.607 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:38.607+0000 7fe698dcbf40 1 -- 172.21.15.32:0/4237013562 --> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] -- osd_op(unknown.0.0:82 12.d 12:b50e409b:::mds_snaptable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98) -- 0x55bab65ad930 con 0x55bab65aab30 2024-08-03T20:56:38.608 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds_snaptable:head# 2024-08-03T20:56:38.609 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:38.608+0000 7fe692000640 1 -- 172.21.15.32:0/4237013562 <== osd.7 v2:172.21.15.118:6818/653000048 3 ==== osd_op_reply(82 mds_snaptable [create] v98'63 uv63 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7fe684012930 con 0x55bab65aab30 2024-08-03T20:56:38.609 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:38.608+0000 7fe698dcbf40 1 -- 172.21.15.32:0/4237013562 --> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] -- osd_op(unknown.0.0:83 12.d 12:b50e409b:::mds_snaptable:head [write 0~70 in=70b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98) -- 0x55bab65ad930 con 0x55bab65aab30 2024-08-03T20:56:38.610 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:38.610+0000 7fe692000640 1 -- 172.21.15.32:0/4237013562 <== osd.7 v2:172.21.15.118:6818/653000048 4 ==== osd_op_reply(83 mds_snaptable [write 0~70] v98'64 uv64 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7fe684012930 con 0x55bab65aab30 2024-08-03T20:56:38.610 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:38.610+0000 7fe698dcbf40 1 -- 172.21.15.32:0/4237013562 --> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] -- osd_op(unknown.0.0:84 12.d 12:b50e409b:::mds_snaptable:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98) -- 0x55bab65ad930 con 0x55bab65aab30 2024-08-03T20:56:38.611 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:38.611+0000 7fe692000640 1 -- 172.21.15.32:0/4237013562 <== osd.7 v2:172.21.15.118:6818/653000048 5 ==== osd_op_reply(84 mds_snaptable [omap-set-header] v98'65 uv65 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7fe684012930 con 0x55bab65aab30 2024-08-03T20:56:38.611 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:38.611+0000 7fe698dcbf40 1 -- 172.21.15.32:0/4237013562 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:85 12.1d 12:b871830b:::605.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98) -- 0x55bab65ad930 con 0x55bab64f65b0 2024-08-03T20:56:38.613 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:38.612+0000 7fe692a00640 1 -- 172.21.15.32:0/4237013562 <== osd.0 v2:172.21.15.32:6816/3066156116 44 ==== osd_op_reply(85 605.00000000 [create] v98'75 uv74 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fe68c00b040 con 0x55bab64f65b0 2024-08-03T20:56:38.613 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:38.612+0000 7fe698dcbf40 1 -- 172.21.15.32:0/4237013562 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:86 12.1d 12:b871830b:::605.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e98) -- 0x55bab65ad930 con 0x55bab64f65b0 2024-08-03T20:56:38.614 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:38.614+0000 7fe692a00640 1 -- 172.21.15.32:0/4237013562 <== osd.0 v2:172.21.15.32:6816/3066156116 45 ==== osd_op_reply(86 605.00000000 [delete] v98'76 uv76 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe68c00b040 con 0x55bab64f65b0 2024-08-03T20:56:38.614 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:38.614+0000 7fe698dcbf40 1 -- 172.21.15.32:0/4237013562 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:87 12.1d 12:b871830b:::605.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98) -- 0x55bab65ad930 con 0x55bab64f65b0 2024-08-03T20:56:38.615 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:38.615+0000 7fe692a00640 1 -- 172.21.15.32:0/4237013562 <== osd.0 v2:172.21.15.32:6816/3066156116 46 ==== osd_op_reply(87 605.00000000 [create] v98'77 uv77 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe68c00b040 con 0x55bab64f65b0 2024-08-03T20:56:38.616 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::605.00000000:head# 2024-08-03T20:56:38.616 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:38.615+0000 7fe698dcbf40 1 -- 172.21.15.32:0/4237013562 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:88 12.1d 12:b871830b:::605.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98) -- 0x55bab65aed30 con 0x55bab64f65b0 2024-08-03T20:56:38.617 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:38.617+0000 7fe692a00640 1 -- 172.21.15.32:0/4237013562 <== osd.0 v2:172.21.15.32:6816/3066156116 47 ==== osd_op_reply(88 605.00000000 [setxattr (30)] v98'78 uv78 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe68c00b040 con 0x55bab64f65b0 2024-08-03T20:56:38.617 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:38.617+0000 7fe698dcbf40 1 -- 172.21.15.32:0/4237013562 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:89 12.1d 12:b871830b:::605.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98) -- 0x55bab65aed30 con 0x55bab64f65b0 2024-08-03T20:56:38.619 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:38.618+0000 7fe692a00640 1 -- 172.21.15.32:0/4237013562 <== osd.0 v2:172.21.15.32:6816/3066156116 48 ==== osd_op_reply(89 605.00000000 [setxattr (62)] v98'79 uv79 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe68c00b040 con 0x55bab64f65b0 2024-08-03T20:56:38.619 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:38.618+0000 7fe698dcbf40 1 -- 172.21.15.32:0/4237013562 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:90 12.1d 12:b871830b:::605.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98) -- 0x55bab65add20 con 0x55bab64f65b0 2024-08-03T20:56:38.619 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:38.619+0000 7fe692a00640 1 -- 172.21.15.32:0/4237013562 <== osd.0 v2:172.21.15.32:6816/3066156116 49 ==== osd_op_reply(90 605.00000000 [omap-set-header] v98'80 uv80 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe68c00b040 con 0x55bab64f65b0 2024-08-03T20:56:38.620 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:38.619+0000 7fe698dcbf40 1 -- 172.21.15.32:0/4237013562 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:91 12.13 12:c91d4a1d:::mds0_inotable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98) -- 0x55bab65add20 con 0x55bab65a3140 2024-08-03T20:56:38.621 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:38.621+0000 7fe691600640 1 -- 172.21.15.32:0/4237013562 <== osd.1 v2:172.21.15.32:6824/1551241759 5 ==== osd_op_reply(91 mds0_inotable [create] v98'122 uv116 ondisk = -17 ((17) File exists)) ==== 157+0+0 (crc 0 0 0) 0x7fe680027070 con 0x55bab65a3140 2024-08-03T20:56:38.621 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:38.621+0000 7fe698dcbf40 1 -- 172.21.15.32:0/4237013562 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:92 12.13 12:c91d4a1d:::mds0_inotable:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e98) -- 0x55bab65ae0c0 con 0x55bab65a3140 2024-08-03T20:56:38.622 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:38.622+0000 7fe691600640 1 -- 172.21.15.32:0/4237013562 <== osd.1 v2:172.21.15.32:6824/1551241759 6 ==== osd_op_reply(92 mds0_inotable [delete] v98'123 uv123 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7fe680027070 con 0x55bab65a3140 2024-08-03T20:56:38.622 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:38.622+0000 7fe698dcbf40 1 -- 172.21.15.32:0/4237013562 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:93 12.13 12:c91d4a1d:::mds0_inotable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98) -- 0x55bab65ae460 con 0x55bab65a3140 2024-08-03T20:56:38.623 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds0_inotable:head# 2024-08-03T20:56:38.624 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:38.623+0000 7fe691600640 1 -- 172.21.15.32:0/4237013562 <== osd.1 v2:172.21.15.32:6824/1551241759 7 ==== osd_op_reply(93 mds0_inotable [create] v98'124 uv124 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7fe680027070 con 0x55bab65a3140 2024-08-03T20:56:38.624 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:38.623+0000 7fe698dcbf40 1 -- 172.21.15.32:0/4237013562 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:94 12.13 12:c91d4a1d:::mds0_inotable:head [write 0~34 in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98) -- 0x55bab65ae800 con 0x55bab65a3140 2024-08-03T20:56:38.625 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:38.624+0000 7fe691600640 1 -- 172.21.15.32:0/4237013562 <== osd.1 v2:172.21.15.32:6824/1551241759 8 ==== osd_op_reply(94 mds0_inotable [write 0~34] v98'125 uv125 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7fe680027070 con 0x55bab65a3140 2024-08-03T20:56:38.625 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:38.624+0000 7fe698dcbf40 1 -- 172.21.15.32:0/4237013562 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:95 12.13 12:c91d4a1d:::mds0_inotable:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98) -- 0x55bab65ae800 con 0x55bab65a3140 2024-08-03T20:56:38.626 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:38.626+0000 7fe691600640 1 -- 172.21.15.32:0/4237013562 <== osd.1 v2:172.21.15.32:6824/1551241759 9 ==== osd_op_reply(95 mds0_inotable [omap-set-header] v98'126 uv126 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7fe680027070 con 0x55bab65a3140 2024-08-03T20:56:38.626 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:38.626+0000 7fe698dcbf40 1 -- 172.21.15.32:0/4237013562 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:96 12.13 12:cd5a64a3:::100.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98) -- 0x55bab65ae800 con 0x55bab65a3140 2024-08-03T20:56:38.627 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:38.627+0000 7fe691600640 1 -- 172.21.15.32:0/4237013562 <== osd.1 v2:172.21.15.32:6824/1551241759 10 ==== osd_op_reply(96 100.00000000 [create] v98'127 uv121 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fe680027070 con 0x55bab65a3140 2024-08-03T20:56:38.627 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:38.627+0000 7fe698dcbf40 1 -- 172.21.15.32:0/4237013562 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:97 12.13 12:cd5a64a3:::100.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e98) -- 0x55bab65ae800 con 0x55bab65a3140 2024-08-03T20:56:38.629 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:38.628+0000 7fe691600640 1 -- 172.21.15.32:0/4237013562 <== osd.1 v2:172.21.15.32:6824/1551241759 11 ==== osd_op_reply(97 100.00000000 [delete] v98'128 uv128 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe680027070 con 0x55bab65a3140 2024-08-03T20:56:38.629 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:38.628+0000 7fe698dcbf40 1 -- 172.21.15.32:0/4237013562 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:98 12.13 12:cd5a64a3:::100.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98) -- 0x55bab65ae800 con 0x55bab65a3140 2024-08-03T20:56:38.630 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:38.629+0000 7fe691600640 1 -- 172.21.15.32:0/4237013562 <== osd.1 v2:172.21.15.32:6824/1551241759 12 ==== osd_op_reply(98 100.00000000 [create] v98'129 uv129 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe680027070 con 0x55bab65a3140 2024-08-03T20:56:38.630 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::100.00000000:head# 2024-08-03T20:56:38.630 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:38.630+0000 7fe698dcbf40 1 -- 172.21.15.32:0/4237013562 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:99 12.13 12:cd5a64a3:::100.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98) -- 0x55bab65ae800 con 0x55bab65a3140 2024-08-03T20:56:38.631 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:38.631+0000 7fe691600640 1 -- 172.21.15.32:0/4237013562 <== osd.1 v2:172.21.15.32:6824/1551241759 13 ==== osd_op_reply(99 100.00000000 [omap-set-header] v98'130 uv130 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe680027070 con 0x55bab65a3140 2024-08-03T20:56:38.631 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:38.631+0000 7fe698dcbf40 1 -- 172.21.15.32:0/4237013562 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:100 12.13 12:cd5a64a3:::100.00000000:head [omap-set-vals in=5194b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98) -- 0x55bab65b1610 con 0x55bab65a3140 2024-08-03T20:56:38.633 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:38.632+0000 7fe691600640 1 -- 172.21.15.32:0/4237013562 <== osd.1 v2:172.21.15.32:6824/1551241759 14 ==== osd_op_reply(100 100.00000000 [omap-set-vals] v98'131 uv131 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe680027070 con 0x55bab65a3140 2024-08-03T20:56:38.633 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:38.632+0000 7fe698dcbf40 1 --2- 172.21.15.32:0/4237013562 >> [v2:172.21.15.32:6800/9088303,v1:172.21.15.32:6801/9088303] conn(0x55bab65af1c0 0x55bab65b1af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:56:38.633 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:38.632+0000 7fe698dcbf40 1 -- 172.21.15.32:0/4237013562 --> [v2:172.21.15.32:6800/9088303,v1:172.21.15.32:6801/9088303] -- osd_op(unknown.0.0:101 12.b 12:d0630e4c:::mds0_sessionmap:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98) -- 0x55bab65b15e0 con 0x55bab65af1c0 2024-08-03T20:56:38.633 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:38.633+0000 7fe691600640 1 --2- 172.21.15.32:0/4237013562 >> [v2:172.21.15.32:6800/9088303,v1:172.21.15.32:6801/9088303] conn(0x55bab65af1c0 0x55bab65b1af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:56:38.633 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:38.633+0000 7fe691600640 1 --2- 172.21.15.32:0/4237013562 >> [v2:172.21.15.32:6800/9088303,v1:172.21.15.32:6801/9088303] conn(0x55bab65af1c0 0x55bab65b1af0 crc :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:56:38.635 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:38.635+0000 7fe691600640 1 -- 172.21.15.32:0/4237013562 <== osd.3 v2:172.21.15.32:6800/9088303 1 ==== osd_op_reply(101 mds0_sessionmap [create] v98'57 uv55 ondisk = -17 ((17) File exists)) ==== 159+0+0 (crc 0 0 0) 0x7fe680027070 con 0x55bab65af1c0 2024-08-03T20:56:38.635 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:38.635+0000 7fe698dcbf40 1 -- 172.21.15.32:0/4237013562 --> [v2:172.21.15.32:6800/9088303,v1:172.21.15.32:6801/9088303] -- osd_op(unknown.0.0:102 12.b 12:d0630e4c:::mds0_sessionmap:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e98) -- 0x55bab65b15e0 con 0x55bab65af1c0 2024-08-03T20:56:38.636 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:38.636+0000 7fe691600640 1 -- 172.21.15.32:0/4237013562 <== osd.3 v2:172.21.15.32:6800/9088303 2 ==== osd_op_reply(102 mds0_sessionmap [delete] v98'58 uv58 ondisk = 0) ==== 159+0+0 (crc 0 0 0) 0x7fe680027070 con 0x55bab65af1c0 2024-08-03T20:56:38.636 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:38.636+0000 7fe698dcbf40 1 -- 172.21.15.32:0/4237013562 --> [v2:172.21.15.32:6800/9088303,v1:172.21.15.32:6801/9088303] -- osd_op(unknown.0.0:103 12.b 12:d0630e4c:::mds0_sessionmap:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98) -- 0x55bab65b15e0 con 0x55bab65af1c0 2024-08-03T20:56:38.638 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:38.637+0000 7fe691600640 1 -- 172.21.15.32:0/4237013562 <== osd.3 v2:172.21.15.32:6800/9088303 3 ==== osd_op_reply(103 mds0_sessionmap [create] v98'59 uv59 ondisk = 0) ==== 159+0+0 (crc 0 0 0) 0x7fe680094040 con 0x55bab65af1c0 2024-08-03T20:56:38.638 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds0_sessionmap:head# 2024-08-03T20:56:38.638 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:38.637+0000 7fe698dcbf40 1 -- 172.21.15.32:0/4237013562 --> [v2:172.21.15.32:6800/9088303,v1:172.21.15.32:6801/9088303] -- osd_op(unknown.0.0:104 12.b 12:d0630e4c:::mds0_sessionmap:head [omap-set-header in=14b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98) -- 0x55bab65b15e0 con 0x55bab65af1c0 2024-08-03T20:56:38.639 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:38.639+0000 7fe691600640 1 -- 172.21.15.32:0/4237013562 <== osd.3 v2:172.21.15.32:6800/9088303 4 ==== osd_op_reply(104 mds0_sessionmap [omap-set-header] v98'60 uv60 ondisk = 0) ==== 159+0+0 (crc 0 0 0) 0x7fe680094040 con 0x55bab65af1c0 2024-08-03T20:56:38.639 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:38.639+0000 7fe698dcbf40 1 -- 172.21.15.32:0/4237013562 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:105 12.7 12:e0b41d06:::609.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98) -- 0x55bab65b15e0 con 0x55bab65a3140 2024-08-03T20:56:38.641 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:38.640+0000 7fe691600640 1 -- 172.21.15.32:0/4237013562 <== osd.1 v2:172.21.15.32:6824/1551241759 15 ==== osd_op_reply(105 609.00000000 [create] v98'75 uv74 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fe680094040 con 0x55bab65a3140 2024-08-03T20:56:38.641 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:38.640+0000 7fe698dcbf40 1 -- 172.21.15.32:0/4237013562 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:106 12.7 12:e0b41d06:::609.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e98) -- 0x55bab65b15e0 con 0x55bab65a3140 2024-08-03T20:56:38.642 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:38.642+0000 7fe691600640 1 -- 172.21.15.32:0/4237013562 <== osd.1 v2:172.21.15.32:6824/1551241759 16 ==== osd_op_reply(106 609.00000000 [delete] v98'76 uv76 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe680094040 con 0x55bab65a3140 2024-08-03T20:56:38.642 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:38.642+0000 7fe698dcbf40 1 -- 172.21.15.32:0/4237013562 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:107 12.7 12:e0b41d06:::609.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98) -- 0x55bab65b15e0 con 0x55bab65a3140 2024-08-03T20:56:38.643 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::609.00000000:head# 2024-08-03T20:56:38.644 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:38.643+0000 7fe691600640 1 -- 172.21.15.32:0/4237013562 <== osd.1 v2:172.21.15.32:6824/1551241759 17 ==== osd_op_reply(107 609.00000000 [create] v98'77 uv77 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe680094040 con 0x55bab65a3140 2024-08-03T20:56:38.644 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:38.643+0000 7fe698dcbf40 1 -- 172.21.15.32:0/4237013562 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:108 12.7 12:e0b41d06:::609.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98) -- 0x55bab65b15e0 con 0x55bab65a3140 2024-08-03T20:56:38.645 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:38.645+0000 7fe691600640 1 -- 172.21.15.32:0/4237013562 <== osd.1 v2:172.21.15.32:6824/1551241759 18 ==== osd_op_reply(108 609.00000000 [setxattr (30)] v98'78 uv78 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe680094040 con 0x55bab65a3140 2024-08-03T20:56:38.645 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:38.645+0000 7fe698dcbf40 1 -- 172.21.15.32:0/4237013562 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:109 12.7 12:e0b41d06:::609.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98) -- 0x55bab65b15e0 con 0x55bab65a3140 2024-08-03T20:56:38.646 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:38.646+0000 7fe691600640 1 -- 172.21.15.32:0/4237013562 <== osd.1 v2:172.21.15.32:6824/1551241759 19 ==== osd_op_reply(109 609.00000000 [setxattr (62)] v98'79 uv79 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe680094040 con 0x55bab65a3140 2024-08-03T20:56:38.646 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:38.646+0000 7fe698dcbf40 1 -- 172.21.15.32:0/4237013562 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:110 12.7 12:e0b41d06:::609.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98) -- 0x55bab65b15e0 con 0x55bab65a3140 2024-08-03T20:56:38.647 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:38.647+0000 7fe691600640 1 -- 172.21.15.32:0/4237013562 <== osd.1 v2:172.21.15.32:6824/1551241759 20 ==== osd_op_reply(110 609.00000000 [omap-set-header] v98'80 uv80 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe680094040 con 0x55bab65a3140 2024-08-03T20:56:38.648 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:38.647+0000 7fe698dcbf40 1 -- 172.21.15.32:0/4237013562 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:111 12.f 12:f1f69726:::400.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98) -- 0x55bab65b15e0 con 0x55bab65a7700 2024-08-03T20:56:38.649 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:38.649+0000 7fe692a00640 1 -- 172.21.15.32:0/4237013562 <== osd.6 v2:172.21.15.118:6825/801947386 12 ==== osd_op_reply(111 400.00000000 [create] v98'64 uv63 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fe68c00b040 con 0x55bab65a7700 2024-08-03T20:56:38.649 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:38.649+0000 7fe698dcbf40 1 -- 172.21.15.32:0/4237013562 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:112 12.f 12:f1f69726:::400.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e98) -- 0x55bab65b15e0 con 0x55bab65a7700 2024-08-03T20:56:38.650 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:38.650+0000 7fe692a00640 1 -- 172.21.15.32:0/4237013562 <== osd.6 v2:172.21.15.118:6825/801947386 13 ==== osd_op_reply(112 400.00000000 [delete] v98'65 uv65 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe68c00b040 con 0x55bab65a7700 2024-08-03T20:56:38.651 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:38.650+0000 7fe698dcbf40 1 -- 172.21.15.32:0/4237013562 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:113 12.f 12:f1f69726:::400.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98) -- 0x55bab65b15e0 con 0x55bab65a7700 2024-08-03T20:56:38.652 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::400.00000000:head# 2024-08-03T20:56:38.652 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:38.652+0000 7fe692a00640 1 -- 172.21.15.32:0/4237013562 <== osd.6 v2:172.21.15.118:6825/801947386 14 ==== osd_op_reply(113 400.00000000 [create] v98'66 uv66 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe68c00b040 con 0x55bab65a7700 2024-08-03T20:56:38.652 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:38.652+0000 7fe698dcbf40 1 -- 172.21.15.32:0/4237013562 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:114 12.f 12:f1f69726:::400.00000000:head [write 0~22 in=22b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98) -- 0x55bab65b2030 con 0x55bab65a7700 2024-08-03T20:56:38.654 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:38.653+0000 7fe692a00640 1 -- 172.21.15.32:0/4237013562 <== osd.6 v2:172.21.15.118:6825/801947386 15 ==== osd_op_reply(114 400.00000000 [write 0~22] v98'67 uv67 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe68c00b040 con 0x55bab65a7700 2024-08-03T20:56:38.654 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:38.653+0000 7fe698dcbf40 1 -- 172.21.15.32:0/4237013562 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:115 12.f 12:f1f69726:::400.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98) -- 0x55bab65b2030 con 0x55bab65a7700 2024-08-03T20:56:38.655 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:38.655+0000 7fe692a00640 1 -- 172.21.15.32:0/4237013562 <== osd.6 v2:172.21.15.118:6825/801947386 16 ==== osd_op_reply(115 400.00000000 [omap-set-header] v98'68 uv68 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe68c00b040 con 0x55bab65a7700 2024-08-03T20:56:38.656 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:38.655+0000 7fe698dcbf40 1 -- 172.21.15.32:0/4237013562 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:116 12.1f 12:fe07bba1:::100.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98) -- 0x55bab65b2030 con 0x55bab65a7700 2024-08-03T20:56:38.657 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:38.657+0000 7fe692a00640 1 -- 172.21.15.32:0/4237013562 <== osd.6 v2:172.21.15.118:6825/801947386 17 ==== osd_op_reply(116 100.00000000.inode [create] v98'157 uv148 ondisk = -17 ((17) File exists)) ==== 162+0+0 (crc 0 0 0) 0x7fe68c00b040 con 0x55bab65a7700 2024-08-03T20:56:38.657 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:38.657+0000 7fe698dcbf40 1 -- 172.21.15.32:0/4237013562 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:117 12.1f 12:fe07bba1:::100.00000000.inode:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e98) -- 0x55bab65b2030 con 0x55bab65a7700 2024-08-03T20:56:38.659 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:38.658+0000 7fe692a00640 1 -- 172.21.15.32:0/4237013562 <== osd.6 v2:172.21.15.118:6825/801947386 18 ==== osd_op_reply(117 100.00000000.inode [delete] v98'158 uv158 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7fe68c00b040 con 0x55bab65a7700 2024-08-03T20:56:38.659 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:38.658+0000 7fe698dcbf40 1 -- 172.21.15.32:0/4237013562 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:118 12.1f 12:fe07bba1:::100.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98) -- 0x55bab65b2030 con 0x55bab65a7700 2024-08-03T20:56:38.660 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::100.00000000.inode:head# 2024-08-03T20:56:38.661 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:38.660+0000 7fe692a00640 1 -- 172.21.15.32:0/4237013562 <== osd.6 v2:172.21.15.118:6825/801947386 19 ==== osd_op_reply(118 100.00000000.inode [create] v98'159 uv159 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7fe68c00b040 con 0x55bab65a7700 2024-08-03T20:56:38.661 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:38.660+0000 7fe698dcbf40 1 -- 172.21.15.32:0/4237013562 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:119 12.1f 12:fe07bba1:::100.00000000.inode:head [write 0~582 in=582b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98) -- 0x55bab65b2030 con 0x55bab65a7700 2024-08-03T20:56:38.662 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:38.661+0000 7fe692a00640 1 -- 172.21.15.32:0/4237013562 <== osd.6 v2:172.21.15.118:6825/801947386 20 ==== osd_op_reply(119 100.00000000.inode [write 0~582] v98'160 uv160 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7fe68c00b040 con 0x55bab65a7700 2024-08-03T20:56:38.662 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:38.662+0000 7fe698dcbf40 1 -- 172.21.15.32:0/4237013562 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:120 12.1f 12:fe07bba1:::100.00000000.inode:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98) -- 0x55bab65b2030 con 0x55bab65a7700 2024-08-03T20:56:38.663 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:38.663+0000 7fe692a00640 1 -- 172.21.15.32:0/4237013562 <== osd.6 v2:172.21.15.118:6825/801947386 21 ==== osd_op_reply(120 100.00000000.inode [omap-set-header] v98'161 uv161 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7fe68c00b040 con 0x55bab65a7700 2024-08-03T20:56:38.664 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:38.663+0000 7fe698dcbf40 1 -- 172.21.15.32:0/4237013562 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:121 12.1f 12:ff5b34d6:::1.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98) -- 0x55bab65b2030 con 0x55bab65a7700 2024-08-03T20:56:38.665 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:38.665+0000 7fe692a00640 1 -- 172.21.15.32:0/4237013562 <== osd.6 v2:172.21.15.118:6825/801947386 22 ==== osd_op_reply(121 1.00000000 [create] v98'162 uv155 ondisk = -17 ((17) File exists)) ==== 154+0+0 (crc 0 0 0) 0x7fe68c00b040 con 0x55bab65a7700 2024-08-03T20:56:38.665 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:38.665+0000 7fe698dcbf40 1 -- 172.21.15.32:0/4237013562 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:122 12.1f 12:ff5b34d6:::1.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e98) -- 0x55bab65b2030 con 0x55bab65a7700 2024-08-03T20:56:38.666 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:38.666+0000 7fe692a00640 1 -- 172.21.15.32:0/4237013562 <== osd.6 v2:172.21.15.118:6825/801947386 23 ==== osd_op_reply(122 1.00000000 [delete] v98'163 uv163 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7fe68c00b040 con 0x55bab65a7700 2024-08-03T20:56:38.667 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:38.666+0000 7fe698dcbf40 1 -- 172.21.15.32:0/4237013562 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:123 12.1f 12:ff5b34d6:::1.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98) -- 0x55bab65b2030 con 0x55bab65a7700 2024-08-03T20:56:38.668 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:38.668+0000 7fe692a00640 1 -- 172.21.15.32:0/4237013562 <== osd.6 v2:172.21.15.118:6825/801947386 24 ==== osd_op_reply(123 1.00000000 [create] v98'164 uv164 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7fe68c00b040 con 0x55bab65a7700 2024-08-03T20:56:38.668 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::1.00000000:head# 2024-08-03T20:56:38.668 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:38.668+0000 7fe698dcbf40 1 -- 172.21.15.32:0/4237013562 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:124 12.1f 12:ff5b34d6:::1.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98) -- 0x55bab65b3040 con 0x55bab65a7700 2024-08-03T20:56:38.669 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:38.669+0000 7fe692a00640 1 -- 172.21.15.32:0/4237013562 <== osd.6 v2:172.21.15.118:6825/801947386 25 ==== osd_op_reply(124 1.00000000 [setxattr (30)] v98'165 uv165 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7fe68c00b040 con 0x55bab65a7700 2024-08-03T20:56:38.669 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:38.669+0000 7fe698dcbf40 1 -- 172.21.15.32:0/4237013562 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:125 12.1f 12:ff5b34d6:::1.00000000:head [setxattr parent (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98) -- 0x55bab65b3040 con 0x55bab65a7700 2024-08-03T20:56:38.671 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:38.670+0000 7fe692a00640 1 -- 172.21.15.32:0/4237013562 <== osd.6 v2:172.21.15.118:6825/801947386 26 ==== osd_op_reply(125 1.00000000 [setxattr (30)] v98'166 uv166 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7fe68c00b040 con 0x55bab65a7700 2024-08-03T20:56:38.671 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:38.670+0000 7fe698dcbf40 1 -- 172.21.15.32:0/4237013562 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:126 12.1f 12:ff5b34d6:::1.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98) -- 0x55bab65b2030 con 0x55bab65a7700 2024-08-03T20:56:38.672 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:38.672+0000 7fe692a00640 1 -- 172.21.15.32:0/4237013562 <== osd.6 v2:172.21.15.118:6825/801947386 27 ==== osd_op_reply(126 1.00000000 [omap-set-header] v98'167 uv167 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7fe68c00b040 con 0x55bab65a7700 2024-08-03T20:56:38.672 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:38.672+0000 7fe698dcbf40 1 -- 172.21.15.32:0/4237013562 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:127 12.1f 12:ff5b34d6:::1.00000000:head [omap-set-vals in=584b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98) -- 0x55bab65b2790 con 0x55bab65a7700 2024-08-03T20:56:38.674 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:38.673+0000 7fe692a00640 1 -- 172.21.15.32:0/4237013562 <== osd.6 v2:172.21.15.118:6825/801947386 28 ==== osd_op_reply(127 1.00000000 [omap-set-vals] v98'168 uv168 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7fe68c00b040 con 0x55bab65a7700 2024-08-03T20:56:38.675 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:38.673+0000 7fe698dcbf40 1 -- 172.21.15.32:0/4237013562 >> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] conn(0x55bab64f65b0 msgr2=0x55bab64669c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:56:38.675 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:38.673+0000 7fe698dcbf40 1 --2- 172.21.15.32:0/4237013562 >> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] conn(0x55bab64f65b0 0x55bab64669c0 crc :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:38.675 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:38.673+0000 7fe698dcbf40 1 -- 172.21.15.32:0/4237013562 >> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] conn(0x55bab65a3140 msgr2=0x55bab65a5560 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:56:38.675 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:38.673+0000 7fe698dcbf40 1 --2- 172.21.15.32:0/4237013562 >> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] conn(0x55bab65a3140 0x55bab65a5560 crc :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:38.675 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:38.673+0000 7fe698dcbf40 1 -- 172.21.15.32:0/4237013562 >> [v2:172.21.15.32:6800/9088303,v1:172.21.15.32:6801/9088303] conn(0x55bab65af1c0 msgr2=0x55bab65b1af0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:56:38.675 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:38.673+0000 7fe698dcbf40 1 --2- 172.21.15.32:0/4237013562 >> [v2:172.21.15.32:6800/9088303,v1:172.21.15.32:6801/9088303] conn(0x55bab65af1c0 0x55bab65b1af0 crc :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:38.675 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:38.673+0000 7fe698dcbf40 1 -- 172.21.15.32:0/4237013562 >> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] conn(0x55bab659e970 msgr2=0x55bab65a0d90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:56:38.675 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:38.673+0000 7fe698dcbf40 1 --2- 172.21.15.32:0/4237013562 >> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] conn(0x55bab659e970 0x55bab65a0d90 crc :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:38.675 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:38.673+0000 7fe698dcbf40 1 -- 172.21.15.32:0/4237013562 >> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] conn(0x55bab65a7700 msgr2=0x55bab65a7b10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:56:38.675 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:38.673+0000 7fe698dcbf40 1 --2- 172.21.15.32:0/4237013562 >> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] conn(0x55bab65a7700 0x55bab65a7b10 crc :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:38.675 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:38.673+0000 7fe698dcbf40 1 -- 172.21.15.32:0/4237013562 >> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] conn(0x55bab65aab30 msgr2=0x55bab65acf50 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:56:38.675 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:38.673+0000 7fe698dcbf40 1 --2- 172.21.15.32:0/4237013562 >> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] conn(0x55bab65aab30 0x55bab65acf50 crc :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:38.675 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:38.673+0000 7fe692a00640 1 -- 172.21.15.32:0/4237013562 reap_dead start 2024-08-03T20:56:38.676 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:38.673+0000 7fe692a00640 1 -- 172.21.15.32:0/4237013562 reap_dead start 2024-08-03T20:56:38.676 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:38.673+0000 7fe692a00640 1 -- 172.21.15.32:0/4237013562 reap_dead start 2024-08-03T20:56:38.676 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:38.674+0000 7fe698dcbf40 1 -- 172.21.15.32:0/4237013562 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fe6740759f0 msgr2=0x7fe674077f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:56:38.676 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:38.674+0000 7fe698dcbf40 1 --2- 172.21.15.32:0/4237013562 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fe6740759f0 0x7fe674077f00 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7fe684011480 tx=0x7fe684001730 comp rx=0 tx=0).stop 2024-08-03T20:56:38.676 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:38.674+0000 7fe698dcbf40 1 -- 172.21.15.32:0/4237013562 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55bab64efba0 msgr2=0x55bab644e510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:56:38.676 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:38.674+0000 7fe698dcbf40 1 --2- 172.21.15.32:0/4237013562 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55bab64efba0 0x55bab644e510 secure :-1 s=READY pgs=1394 cs=0 l=1 rev1=1 crypto rx=0x55bab6469cf0 tx=0x7fe680016420 comp rx=0 tx=0).stop 2024-08-03T20:56:38.676 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:38.674+0000 7fe698dcbf40 1 -- 172.21.15.32:0/4237013562 shutdown_connections 2024-08-03T20:56:38.676 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:38.674+0000 7fe698dcbf40 1 --2- 172.21.15.32:0/4237013562 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fe6740759f0 0x7fe674077f00 unknown :-1 s=CLOSED pgs=571 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:38.676 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:38.674+0000 7fe698dcbf40 1 --2- 172.21.15.32:0/4237013562 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55bab64efba0 0x55bab644e510 unknown :-1 s=CLOSED pgs=1394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:38.676 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:38.674+0000 7fe698dcbf40 1 -- 172.21.15.32:0/4237013562 >> 172.21.15.32:0/4237013562 conn(0x55bab6343ac0 msgr2=0x55bab644d910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:56:38.676 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:38.674+0000 7fe698dcbf40 1 -- 172.21.15.32:0/4237013562 shutdown_connections 2024-08-03T20:56:38.676 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:38.674+0000 7fe698dcbf40 1 -- 172.21.15.32:0/4237013562 wait complete. 2024-08-03T20:56:38.685 DEBUG:teuthology.orchestra.run.smithi032:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata rm 100.00000000.inode 2024-08-03T20:56:38.829 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:38.828+0000 7f0b02ee5f40 1 Processor -- start 2024-08-03T20:56:38.829 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:38.828+0000 7f0b02ee5f40 1 -- start start 2024-08-03T20:56:38.833 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:38.830+0000 7f0b02ee5f40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f0ae00030f0 0x5612dba7b380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:56:38.834 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:38.830+0000 7f0b02ee5f40 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x5612dba0eca0 0x5612db9c44d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:56:38.834 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:38.830+0000 7f0b02ee5f40 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5612db9c4a10 0x5612dba9c840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:56:38.834 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:38.830+0000 7f0b02ee5f40 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x5612db971bf0 con 0x5612db9c4a10 2024-08-03T20:56:38.834 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:38.830+0000 7f0b02ee5f40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x5612db97e6b0 con 0x7f0ae00030f0 2024-08-03T20:56:38.835 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:38.830+0000 7f0b02ee5f40 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x5612db913e30 con 0x5612dba0eca0 2024-08-03T20:56:38.835 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:38.830+0000 7f0afca00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5612db9c4a10 0x5612dba9c840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:56:38.835 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:38.830+0000 7f0afca00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5612db9c4a10 0x5612dba9c840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:37138/0 (socket says 172.21.15.32:37138) 2024-08-03T20:56:38.835 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:38.830+0000 7f0afca00640 1 -- 172.21.15.32:0/3828908157 learned_addr learned my addr 172.21.15.32:0/3828908157 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:56:38.835 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:38.830+0000 7f0afc000640 1 --2- 172.21.15.32:0/3828908157 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f0ae00030f0 0x5612dba7b380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:56:38.835 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:38.830+0000 7f0afb600640 1 --2- 172.21.15.32:0/3828908157 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x5612dba0eca0 0x5612db9c44d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:56:38.836 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:38.830+0000 7f0afca00640 1 -- 172.21.15.32:0/3828908157 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x5612dba0eca0 msgr2=0x5612db9c44d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:56:38.836 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:38.830+0000 7f0afca00640 1 --2- 172.21.15.32:0/3828908157 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x5612dba0eca0 0x5612db9c44d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:38.836 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:38.830+0000 7f0afca00640 1 -- 172.21.15.32:0/3828908157 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f0ae00030f0 msgr2=0x5612dba7b380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:56:38.836 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:38.830+0000 7f0afca00640 1 --2- 172.21.15.32:0/3828908157 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f0ae00030f0 0x5612dba7b380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:38.836 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:38.830+0000 7f0afca00640 1 -- 172.21.15.32:0/3828908157 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5612dba0e080 con 0x5612db9c4a10 2024-08-03T20:56:38.836 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:38.831+0000 7f0afca00640 1 --2- 172.21.15.32:0/3828908157 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5612db9c4a10 0x5612dba9c840 secure :-1 s=READY pgs=1397 cs=0 l=1 rev1=1 crypto rx=0x5612dba09b00 tx=0x7f0af4009d00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=13d8ff2fc74fa9e3 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:56:38.837 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:38.831+0000 7f0afa200640 1 -- 172.21.15.32:0/3828908157 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0af4009e30 con 0x5612db9c4a10 2024-08-03T20:56:38.837 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:38.831+0000 7f0afa200640 1 -- 172.21.15.32:0/3828908157 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f0af4005820 con 0x5612db9c4a10 2024-08-03T20:56:38.837 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:38.831+0000 7f0afa200640 1 -- 172.21.15.32:0/3828908157 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0af4005c50 con 0x5612db9c4a10 2024-08-03T20:56:38.837 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:38.832+0000 7f0b02ee5f40 1 -- 172.21.15.32:0/3828908157 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5612db9c4a10 msgr2=0x5612dba9c840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:56:38.837 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:38.832+0000 7f0b02ee5f40 1 --2- 172.21.15.32:0/3828908157 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5612db9c4a10 0x5612dba9c840 secure :-1 s=READY pgs=1397 cs=0 l=1 rev1=1 crypto rx=0x5612dba09b00 tx=0x7f0af4009d00 comp rx=0 tx=0).stop 2024-08-03T20:56:38.838 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:38.832+0000 7f0b02ee5f40 1 -- 172.21.15.32:0/3828908157 shutdown_connections 2024-08-03T20:56:38.838 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:38.832+0000 7f0b02ee5f40 1 --2- 172.21.15.32:0/3828908157 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5612db9c4a10 0x5612dba9c840 unknown :-1 s=CLOSED pgs=1397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:38.838 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:38.832+0000 7f0b02ee5f40 1 --2- 172.21.15.32:0/3828908157 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x5612dba0eca0 0x5612db9c44d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:38.838 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:38.832+0000 7f0b02ee5f40 1 --2- 172.21.15.32:0/3828908157 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f0ae00030f0 0x5612dba7b380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:38.838 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:38.832+0000 7f0b02ee5f40 1 -- 172.21.15.32:0/3828908157 >> 172.21.15.32:0/3828908157 conn(0x5612db8e5ac0 msgr2=0x5612db9c3b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:56:38.839 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:38.832+0000 7f0b02ee5f40 1 -- 172.21.15.32:0/3828908157 shutdown_connections 2024-08-03T20:56:38.839 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:38.832+0000 7f0b02ee5f40 1 -- 172.21.15.32:0/3828908157 wait complete. 2024-08-03T20:56:38.839 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:38.833+0000 7f0b02ee5f40 1 Processor -- start 2024-08-03T20:56:38.839 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:38.833+0000 7f0b02ee5f40 1 -- start start 2024-08-03T20:56:38.846 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:38.834+0000 7f0b02ee5f40 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5612db9c4a10 0x5612dba84b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:56:38.846 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:38.834+0000 7f0b02ee5f40 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x5612dba0eca0 0x5612dba7aa40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:56:38.846 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:38.834+0000 7f0b02ee5f40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f0ae00030f0 0x5612dba796f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:56:38.846 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:38.834+0000 7f0b02ee5f40 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x5612dba7a750 con 0x5612db9c4a10 2024-08-03T20:56:38.846 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:38.834+0000 7f0b02ee5f40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x5612dba7bf70 con 0x7f0ae00030f0 2024-08-03T20:56:38.847 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:38.834+0000 7f0b02ee5f40 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x5612dba85490 con 0x5612dba0eca0 2024-08-03T20:56:38.847 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:38.835+0000 7f0afc000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5612db9c4a10 0x5612dba84b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:56:38.847 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:38.835+0000 7f0afc000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5612db9c4a10 0x5612dba84b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:37148/0 (socket says 172.21.15.32:37148) 2024-08-03T20:56:38.847 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:38.835+0000 7f0afc000640 1 -- 172.21.15.32:0/769572039 learned_addr learned my addr 172.21.15.32:0/769572039 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:56:38.847 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:38.835+0000 7f0afb600640 1 --2- 172.21.15.32:0/769572039 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x5612dba0eca0 0x5612dba7aa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:56:38.848 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:38.835+0000 7f0afca00640 1 --2- 172.21.15.32:0/769572039 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f0ae00030f0 0x5612dba796f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:56:38.848 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:38.835+0000 7f0afc000640 1 -- 172.21.15.32:0/769572039 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x5612dba0eca0 msgr2=0x5612dba7aa40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:56:38.848 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:38.835+0000 7f0afc000640 1 --2- 172.21.15.32:0/769572039 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x5612dba0eca0 0x5612dba7aa40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:38.848 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:38.835+0000 7f0afc000640 1 -- 172.21.15.32:0/769572039 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f0ae00030f0 msgr2=0x5612dba796f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:56:38.848 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:38.835+0000 7f0afc000640 1 --2- 172.21.15.32:0/769572039 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f0ae00030f0 0x5612dba796f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:38.848 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:38.835+0000 7f0afc000640 1 -- 172.21.15.32:0/769572039 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5612dba80630 con 0x5612db9c4a10 2024-08-03T20:56:38.849 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:38.835+0000 7f0afca00640 1 --2- 172.21.15.32:0/769572039 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f0ae00030f0 0x5612dba796f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-03T20:56:38.849 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:38.835+0000 7f0afc000640 1 --2- 172.21.15.32:0/769572039 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5612db9c4a10 0x5612dba84b90 secure :-1 s=READY pgs=1398 cs=0 l=1 rev1=1 crypto rx=0x7f0aec00a7d0 tx=0x7f0aec00ac90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:56:38.849 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:38.836+0000 7f0af8e00640 1 -- 172.21.15.32:0/769572039 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0aec01c070 con 0x5612db9c4a10 2024-08-03T20:56:38.849 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:38.836+0000 7f0af8e00640 1 -- 172.21.15.32:0/769572039 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f0aec009cd0 con 0x5612db9c4a10 2024-08-03T20:56:38.849 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:38.836+0000 7f0b02ee5f40 1 -- 172.21.15.32:0/769572039 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x5612dba80910 con 0x5612db9c4a10 2024-08-03T20:56:38.849 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:38.836+0000 7f0af8e00640 1 -- 172.21.15.32:0/769572039 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0aec0176a0 con 0x5612db9c4a10 2024-08-03T20:56:38.850 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:38.836+0000 7f0b02ee5f40 1 -- 172.21.15.32:0/769572039 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5612dba851d0 con 0x5612db9c4a10 2024-08-03T20:56:38.850 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:38.838+0000 7f0af8e00640 1 -- 172.21.15.32:0/769572039 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f0aec01f070 con 0x5612db9c4a10 2024-08-03T20:56:38.850 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:38.839+0000 7f0af8e00640 1 --2- 172.21.15.32:0/769572039 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f0adc0759f0 0x7f0adc077f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:56:38.850 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:38.839+0000 7f0af8e00640 1 -- 172.21.15.32:0/769572039 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 6972+0+0 (secure 0 0 0) 0x7f0aec098590 con 0x5612db9c4a10 2024-08-03T20:56:38.850 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:38.839+0000 7f0afb600640 1 --2- 172.21.15.32:0/769572039 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f0adc0759f0 0x7f0adc077f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:56:38.851 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:38.839+0000 7f0b02ee5f40 1 --2- 172.21.15.32:0/769572039 >> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] conn(0x5612dba08680 0x5612dba08a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:56:38.851 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:38.839+0000 7f0b02ee5f40 1 -- 172.21.15.32:0/769572039 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:1 12.1f 12:fe07bba1:::100.00000000.inode:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e98) -- 0x5612dba7b380 con 0x5612dba08680 2024-08-03T20:56:38.851 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:38.840+0000 7f0afca00640 1 --2- 172.21.15.32:0/769572039 >> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] conn(0x5612dba08680 0x5612dba08a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:56:38.851 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:38.840+0000 7f0afb600640 1 --2- 172.21.15.32:0/769572039 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f0adc0759f0 0x7f0adc077f00 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7f0ae8055000 tx=0x7f0ae805f040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:56:38.851 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:38.840+0000 7f0afca00640 1 --2- 172.21.15.32:0/769572039 >> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] conn(0x5612dba08680 0x5612dba08a90 crc :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:56:38.852 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:38.842+0000 7f0afca00640 1 -- 172.21.15.32:0/769572039 <== osd.6 v2:172.21.15.118:6825/801947386 1 ==== osd_op_reply(1 100.00000000.inode [delete] v98'169 uv169 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7f0af401e040 con 0x5612dba08680 2024-08-03T20:56:38.852 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:38.842+0000 7f0b02ee5f40 1 -- 172.21.15.32:0/769572039 >> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] conn(0x5612dba08680 msgr2=0x5612dba08a90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:56:38.852 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:38.842+0000 7f0b02ee5f40 1 --2- 172.21.15.32:0/769572039 >> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] conn(0x5612dba08680 0x5612dba08a90 crc :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:38.852 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:38.842+0000 7f0b02ee5f40 1 -- 172.21.15.32:0/769572039 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f0adc0759f0 msgr2=0x7f0adc077f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:56:38.852 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:38.842+0000 7f0b02ee5f40 1 --2- 172.21.15.32:0/769572039 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f0adc0759f0 0x7f0adc077f00 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7f0ae8055000 tx=0x7f0ae805f040 comp rx=0 tx=0).stop 2024-08-03T20:56:38.852 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:38.842+0000 7f0b02ee5f40 1 -- 172.21.15.32:0/769572039 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5612db9c4a10 msgr2=0x5612dba84b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:56:38.853 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:38.842+0000 7f0b02ee5f40 1 --2- 172.21.15.32:0/769572039 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5612db9c4a10 0x5612dba84b90 secure :-1 s=READY pgs=1398 cs=0 l=1 rev1=1 crypto rx=0x7f0aec00a7d0 tx=0x7f0aec00ac90 comp rx=0 tx=0).stop 2024-08-03T20:56:38.853 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:38.843+0000 7f0afca00640 1 -- 172.21.15.32:0/769572039 reap_dead start 2024-08-03T20:56:38.853 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:38.843+0000 7f0b02ee5f40 1 -- 172.21.15.32:0/769572039 shutdown_connections 2024-08-03T20:56:38.853 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:38.843+0000 7f0b02ee5f40 1 -- 172.21.15.32:0/769572039 >> 172.21.15.32:0/769572039 conn(0x5612db8e5ac0 msgr2=0x5612db9c7410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:56:38.853 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:38.843+0000 7f0b02ee5f40 1 -- 172.21.15.32:0/769572039 shutdown_connections 2024-08-03T20:56:38.853 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:38.843+0000 7f0b02ee5f40 1 -- 172.21.15.32:0/769572039 wait complete. 2024-08-03T20:56:38.856 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs joinable true 2024-08-03T20:56:39.088 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:39.087+0000 7f5c21200640 1 -- 172.21.15.32:0/2336915757 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5c1c15ce70 msgr2=0x7f5c1c159330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:56:39.088 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:39.087+0000 7f5c21200640 1 --2- 172.21.15.32:0/2336915757 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5c1c15ce70 0x7f5c1c159330 secure :-1 s=READY pgs=1399 cs=0 l=1 rev1=1 crypto rx=0x7f5c1c06eb40 tx=0x7f5c1402aac0 comp rx=0 tx=0).stop 2024-08-03T20:56:39.088 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:39.087+0000 7f5c21200640 1 -- 172.21.15.32:0/2336915757 shutdown_connections 2024-08-03T20:56:39.088 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:39.087+0000 7f5c21200640 1 --2- 172.21.15.32:0/2336915757 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5c1c15ce70 0x7f5c1c159330 unknown :-1 s=CLOSED pgs=1399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:39.088 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:39.087+0000 7f5c21200640 1 --2- 172.21.15.32:0/2336915757 >> v2:172.21.15.118:3300/0 conn(0x7f5c1c074330 0x7f5c1c06c9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:39.088 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:39.087+0000 7f5c21200640 1 -- 172.21.15.32:0/2336915757 >> 172.21.15.32:0/2336915757 conn(0x7f5c1c0d0e10 msgr2=0x7f5c1c0d1220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:56:39.088 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:39.087+0000 7f5c21200640 1 -- 172.21.15.32:0/2336915757 shutdown_connections 2024-08-03T20:56:39.089 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:39.087+0000 7f5c21200640 1 -- 172.21.15.32:0/2336915757 wait complete. 2024-08-03T20:56:39.089 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:39.089+0000 7f5c21200640 1 Processor -- start 2024-08-03T20:56:39.090 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:39.089+0000 7f5c21200640 1 -- start start 2024-08-03T20:56:39.097 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:39.091+0000 7f5c21200640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f5c1c06cf20 0x7f5c1c154df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:56:39.097 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:39.091+0000 7f5c21200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5c1c15ce70 0x7f5c1c0ae270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:56:39.097 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:39.091+0000 7f5c21200640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f5c1c0ae7b0 0x7f5c1c1e9040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:56:39.097 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:39.091+0000 7f5c21200640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f5c1c159c00 con 0x7f5c1c15ce70 2024-08-03T20:56:39.097 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:39.091+0000 7f5c21200640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f5c1c159a80 con 0x7f5c1c06cf20 2024-08-03T20:56:39.097 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:39.091+0000 7f5c21200640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7f5c1c159d80 con 0x7f5c1c0ae7b0 2024-08-03T20:56:39.097 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:39.091+0000 7f5c12a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5c1c15ce70 0x7f5c1c0ae270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:56:39.097 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:39.091+0000 7f5c12a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5c1c15ce70 0x7f5c1c0ae270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:37172/0 (socket says 172.21.15.32:37172) 2024-08-03T20:56:39.098 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:39.091+0000 7f5c12a00640 1 -- 172.21.15.32:0/4149283006 learned_addr learned my addr 172.21.15.32:0/4149283006 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:56:39.098 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:39.091+0000 7f5c1be00640 1 --2- 172.21.15.32:0/4149283006 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f5c1c0ae7b0 0x7f5c1c1e9040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:56:39.098 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:39.091+0000 7f5c1b400640 1 --2- 172.21.15.32:0/4149283006 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f5c1c06cf20 0x7f5c1c154df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:56:39.098 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:39.091+0000 7f5c12a00640 1 -- 172.21.15.32:0/4149283006 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f5c1c0ae7b0 msgr2=0x7f5c1c1e9040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:56:39.098 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:39.091+0000 7f5c12a00640 1 --2- 172.21.15.32:0/4149283006 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f5c1c0ae7b0 0x7f5c1c1e9040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:39.098 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:39.091+0000 7f5c12a00640 1 -- 172.21.15.32:0/4149283006 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f5c1c06cf20 msgr2=0x7f5c1c154df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:56:39.098 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:39.091+0000 7f5c12a00640 1 --2- 172.21.15.32:0/4149283006 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f5c1c06cf20 0x7f5c1c154df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:39.098 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:39.092+0000 7f5c12a00640 1 -- 172.21.15.32:0/4149283006 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5c1c0aebc0 con 0x7f5c1c15ce70 2024-08-03T20:56:39.098 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:39.092+0000 7f5c12a00640 1 --2- 172.21.15.32:0/4149283006 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5c1c15ce70 0x7f5c1c0ae270 secure :-1 s=READY pgs=1400 cs=0 l=1 rev1=1 crypto rx=0x7f5c04001eb0 tx=0x7f5c0400c890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:56:39.098 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:39.092+0000 7f5c13e00640 1 -- 172.21.15.32:0/4149283006 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c04017070 con 0x7f5c1c15ce70 2024-08-03T20:56:39.098 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:39.092+0000 7f5c13e00640 1 -- 172.21.15.32:0/4149283006 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f5c040053c0 con 0x7f5c1c15ce70 2024-08-03T20:56:39.098 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:39.092+0000 7f5c13e00640 1 -- 172.21.15.32:0/4149283006 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c04003830 con 0x7f5c1c15ce70 2024-08-03T20:56:39.099 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:39.092+0000 7f5c21200640 1 -- 172.21.15.32:0/4149283006 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5c1c1e9d10 con 0x7f5c1c15ce70 2024-08-03T20:56:39.099 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:39.092+0000 7f5c21200640 1 -- 172.21.15.32:0/4149283006 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5c1c1ea2d0 con 0x7f5c1c15ce70 2024-08-03T20:56:39.099 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:39.093+0000 7f5c21200640 1 -- 172.21.15.32:0/4149283006 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5c1c0cb1c0 con 0x7f5c1c15ce70 2024-08-03T20:56:39.099 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:39.095+0000 7f5c13e00640 1 -- 172.21.15.32:0/4149283006 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f5c0401a070 con 0x7f5c1c15ce70 2024-08-03T20:56:39.099 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:39.096+0000 7f5c13e00640 1 --2- 172.21.15.32:0/4149283006 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f5bf8077040 0x7f5bf8079550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:56:39.099 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:39.096+0000 7f5c13e00640 1 -- 172.21.15.32:0/4149283006 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 6972+0+0 (secure 0 0 0) 0x7f5c04098120 con 0x7f5c1c15ce70 2024-08-03T20:56:39.099 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:39.096+0000 7f5c1b400640 1 --2- 172.21.15.32:0/4149283006 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f5bf8077040 0x7f5bf8079550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:56:39.099 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:39.097+0000 7f5c1b400640 1 --2- 172.21.15.32:0/4149283006 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f5bf8077040 0x7f5bf8079550 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7f5c1c0725e0 tx=0x7f5c0c005f90 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:56:39.100 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:39.100+0000 7f5c13e00640 1 -- 172.21.15.32:0/4149283006 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7f5c1c0cb1c0 con 0x7f5c1c15ce70 2024-08-03T20:56:39.353 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:39.353+0000 7f5c21200640 1 -- 172.21.15.32:0/4149283006 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "joinable", "val": "true"} v 0) -- 0x7f5c1c074330 con 0x7f5c1c15ce70 2024-08-03T20:56:40.033 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:40.031+0000 7f5c13e00640 1 -- 172.21.15.32:0/4149283006 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "joinable", "val": "true"}]=0 cephfs marked joinable; MDS may join as newly active. v172) ==== 162+0+0 (secure 0 0 0) 0x7f5c1c0aebc0 con 0x7f5c1c15ce70 2024-08-03T20:56:40.033 INFO:teuthology.orchestra.run.smithi032.stderr:cephfs marked joinable; MDS may join as newly active. 2024-08-03T20:56:40.036 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:40.034+0000 7f5c21200640 1 -- 172.21.15.32:0/4149283006 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f5bf8077040 msgr2=0x7f5bf8079550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:56:40.036 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:40.034+0000 7f5c21200640 1 --2- 172.21.15.32:0/4149283006 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f5bf8077040 0x7f5bf8079550 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7f5c1c0725e0 tx=0x7f5c0c005f90 comp rx=0 tx=0).stop 2024-08-03T20:56:40.037 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:40.034+0000 7f5c21200640 1 -- 172.21.15.32:0/4149283006 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5c1c15ce70 msgr2=0x7f5c1c0ae270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:56:40.037 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:40.034+0000 7f5c21200640 1 --2- 172.21.15.32:0/4149283006 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5c1c15ce70 0x7f5c1c0ae270 secure :-1 s=READY pgs=1400 cs=0 l=1 rev1=1 crypto rx=0x7f5c04001eb0 tx=0x7f5c0400c890 comp rx=0 tx=0).stop 2024-08-03T20:56:40.037 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:40.034+0000 7f5c21200640 1 -- 172.21.15.32:0/4149283006 shutdown_connections 2024-08-03T20:56:40.037 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:40.034+0000 7f5c21200640 1 --2- 172.21.15.32:0/4149283006 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f5bf8077040 0x7f5bf8079550 unknown :-1 s=CLOSED pgs=573 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:40.037 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:40.034+0000 7f5c21200640 1 --2- 172.21.15.32:0/4149283006 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f5c1c0ae7b0 0x7f5c1c1e9040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:40.037 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:40.034+0000 7f5c21200640 1 --2- 172.21.15.32:0/4149283006 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5c1c15ce70 0x7f5c1c0ae270 unknown :-1 s=CLOSED pgs=1400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:40.037 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:40.034+0000 7f5c21200640 1 --2- 172.21.15.32:0/4149283006 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f5c1c06cf20 0x7f5c1c154df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:40.037 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:40.034+0000 7f5c21200640 1 -- 172.21.15.32:0/4149283006 >> 172.21.15.32:0/4149283006 conn(0x7f5c1c0d0e10 msgr2=0x7f5c1c0b4670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:56:40.037 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:40.035+0000 7f5c21200640 1 -- 172.21.15.32:0/4149283006 shutdown_connections 2024-08-03T20:56:40.037 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:40.035+0000 7f5c21200640 1 -- 172.21.15.32:0/4149283006 wait complete. 2024-08-03T20:56:40.044 INFO:tasks.ceph.mds.a.smithi032.stderr:2024-08-03T20:56:40.043+0000 7fccc0400640 -1 mds.0.cache.ino(0x100) No data while reading inode 0x100 2024-08-03T20:56:40.045 INFO:tasks.ceph.mds.a.smithi032.stderr:2024-08-03T20:56:40.043+0000 7fccc0400640 -1 log_channel(cluster) log [ERR] : Error loading MDS rank 0: (2) No such file or directory 2024-08-03T20:56:40.045 INFO:tasks.ceph.mon.a.smithi032.stderr:2024-08-03T20:56:40.044+0000 7f4e6d600640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-08-03T20:56:40.045 INFO:tasks.ceph.mon.a.smithi032.stderr:2024-08-03T20:56:40.044+0000 7f4e6d600640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 mds daemon damaged (MDS_DAMAGE) 2024-08-03T20:56:40.067 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-08-03T20:56:40.299 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:40.297+0000 7f3577200640 1 -- 172.21.15.32:0/905699598 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f35780cffe0 msgr2=0x7f35780b3e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:56:40.299 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:40.297+0000 7f3577200640 1 --2- 172.21.15.32:0/905699598 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f35780cffe0 0x7f35780b3e90 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f3564009830 tx=0x7f356402a530 comp rx=0 tx=0).stop 2024-08-03T20:56:40.299 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:40.297+0000 7f3577200640 1 -- 172.21.15.32:0/905699598 shutdown_connections 2024-08-03T20:56:40.299 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:40.297+0000 7f3577200640 1 --2- 172.21.15.32:0/905699598 >> v2:172.21.15.118:3300/0 conn(0x7f35780b43d0 0x7f35780cf510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:40.299 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:40.297+0000 7f3577200640 1 --2- 172.21.15.32:0/905699598 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f35780cffe0 0x7f35780b3e90 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:40.299 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:40.297+0000 7f3577200640 1 -- 172.21.15.32:0/905699598 >> 172.21.15.32:0/905699598 conn(0x7f357808d7a0 msgr2=0x7f357809d3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:56:40.300 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:40.297+0000 7f3577200640 1 -- 172.21.15.32:0/905699598 shutdown_connections 2024-08-03T20:56:40.300 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:40.297+0000 7f3577200640 1 -- 172.21.15.32:0/905699598 wait complete. 2024-08-03T20:56:40.301 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:40.300+0000 7f3577200640 1 Processor -- start 2024-08-03T20:56:40.301 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:40.300+0000 7f3577200640 1 -- start start 2024-08-03T20:56:40.304 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:40.301+0000 7f3577200640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f35780b43d0 0x7f3578157120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:56:40.304 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:40.301+0000 7f3577200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f35780cffe0 0x7f35781f5540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:56:40.305 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:40.301+0000 7f3577200640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f35780d26e0 0x7f35781f5a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:56:40.305 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:40.301+0000 7f3577200640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f35780d4100 con 0x7f35780cffe0 2024-08-03T20:56:40.305 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:40.301+0000 7f3577200640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f35780d3f80 con 0x7f35780b43d0 2024-08-03T20:56:40.305 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:40.301+0000 7f3577200640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7f35780d4280 con 0x7f35780d26e0 2024-08-03T20:56:40.305 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:40.301+0000 7f3575400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f35780cffe0 0x7f35781f5540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:56:40.305 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:40.301+0000 7f3575400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f35780cffe0 0x7f35781f5540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:49962/0 (socket says 172.21.15.32:49962) 2024-08-03T20:56:40.305 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:40.301+0000 7f3575400640 1 -- 172.21.15.32:0/2153172023 learned_addr learned my addr 172.21.15.32:0/2153172023 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:56:40.305 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:40.302+0000 7f3575e00640 1 --2- 172.21.15.32:0/2153172023 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f35780b43d0 0x7f3578157120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:56:40.305 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:40.302+0000 7f3576800640 1 --2- 172.21.15.32:0/2153172023 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f35780d26e0 0x7f35781f5a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:56:40.305 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:40.302+0000 7f3575400640 1 -- 172.21.15.32:0/2153172023 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f35780d26e0 msgr2=0x7f35781f5a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:56:40.305 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:40.302+0000 7f3575400640 1 --2- 172.21.15.32:0/2153172023 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f35780d26e0 0x7f35781f5a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:40.305 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:40.302+0000 7f3575400640 1 -- 172.21.15.32:0/2153172023 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f35780b43d0 msgr2=0x7f3578157120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:56:40.305 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:40.302+0000 7f3575400640 1 --2- 172.21.15.32:0/2153172023 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f35780b43d0 0x7f3578157120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:40.306 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:40.302+0000 7f3575400640 1 -- 172.21.15.32:0/2153172023 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f35780cd230 con 0x7f35780cffe0 2024-08-03T20:56:40.306 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:40.302+0000 7f3575400640 1 --2- 172.21.15.32:0/2153172023 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f35780cffe0 0x7f35781f5540 secure :-1 s=READY pgs=1402 cs=0 l=1 rev1=1 crypto rx=0x7f3564009950 tx=0x7f3564033c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:56:40.306 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:40.303+0000 7f356e000640 1 -- 172.21.15.32:0/2153172023 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3564005c90 con 0x7f35780cffe0 2024-08-03T20:56:40.306 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:40.303+0000 7f356e000640 1 -- 172.21.15.32:0/2153172023 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f3564031e50 con 0x7f35780cffe0 2024-08-03T20:56:40.306 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:40.303+0000 7f3577200640 1 -- 172.21.15.32:0/2153172023 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f35780cd4f0 con 0x7f35780cffe0 2024-08-03T20:56:40.306 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:40.303+0000 7f356e000640 1 -- 172.21.15.32:0/2153172023 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35640332f0 con 0x7f35780cffe0 2024-08-03T20:56:40.306 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:40.303+0000 7f3577200640 1 -- 172.21.15.32:0/2153172023 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f35780cdab0 con 0x7f35780cffe0 2024-08-03T20:56:40.307 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:40.304+0000 7f3577200640 1 -- 172.21.15.32:0/2153172023 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f35780d5250 con 0x7f35780cffe0 2024-08-03T20:56:40.307 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:40.305+0000 7f356e000640 1 -- 172.21.15.32:0/2153172023 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f3564033490 con 0x7f35780cffe0 2024-08-03T20:56:40.307 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:40.306+0000 7f356e000640 1 --2- 172.21.15.32:0/2153172023 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f3558077010 0x7f3558079520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:56:40.307 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:40.306+0000 7f356e000640 1 -- 172.21.15.32:0/2153172023 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 7058+0+0 (secure 0 0 0) 0x7f35640b9e30 con 0x7f35780cffe0 2024-08-03T20:56:40.308 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:40.306+0000 7f3575e00640 1 --2- 172.21.15.32:0/2153172023 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f3558077010 0x7f3558079520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:56:40.308 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:40.307+0000 7f3575e00640 1 --2- 172.21.15.32:0/2153172023 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f3558077010 0x7f3558079520 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7f3568007fd0 tx=0x7f3568001370 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:56:40.312 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:40.311+0000 7f356e000640 1 -- 172.21.15.32:0/2153172023 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7f35780d5250 con 0x7f35780cffe0 2024-08-03T20:56:40.554 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:40.552+0000 7f3577200640 1 -- 172.21.15.32:0/2153172023 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f3578076df0 con 0x7f35780cffe0 2024-08-03T20:56:40.555 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:40.553+0000 7f356e000640 1 -- 172.21.15.32:0/2153172023 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 173 v173) ==== 95+0+4390 (secure 0 0 0) 0x7f356404a030 con 0x7f35780cffe0 2024-08-03T20:56:40.555 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-03T20:56:40.555 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":173,"btime":"2024-08-03T20:56:40:045853+0000","default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6439,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6835/531080160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6834","nonce":531080160},{"type":"v1","addr":"172.21.15.118:6835","nonce":531080160}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":84},{"gid":6454,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6837/2796882319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6836","nonce":2796882319},{"type":"v1","addr":"172.21.15.118:6837","nonce":2796882319}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":86},{"gid":6813,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6834/2686933890","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":2686933890},{"type":"v1","addr":"172.21.15.32:6834","nonce":2686933890}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":85}],"filesystems":[{"mdsmap":{"epoch":173,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-08-03T20:54:26.950518+0000","modified":"2024-08-03T20:56:40.045781+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":99,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":6}]} 2024-08-03T20:56:40.556 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 173 2024-08-03T20:56:40.557 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:40.555+0000 7f3577200640 1 -- 172.21.15.32:0/2153172023 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f3558077010 msgr2=0x7f3558079520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:56:40.557 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:40.555+0000 7f3577200640 1 --2- 172.21.15.32:0/2153172023 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f3558077010 0x7f3558079520 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7f3568007fd0 tx=0x7f3568001370 comp rx=0 tx=0).stop 2024-08-03T20:56:40.557 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:40.555+0000 7f3577200640 1 -- 172.21.15.32:0/2153172023 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f35780cffe0 msgr2=0x7f35781f5540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:56:40.557 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:40.555+0000 7f3577200640 1 --2- 172.21.15.32:0/2153172023 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f35780cffe0 0x7f35781f5540 secure :-1 s=READY pgs=1402 cs=0 l=1 rev1=1 crypto rx=0x7f3564009950 tx=0x7f3564033c20 comp rx=0 tx=0).stop 2024-08-03T20:56:40.557 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:40.555+0000 7f3577200640 1 -- 172.21.15.32:0/2153172023 shutdown_connections 2024-08-03T20:56:40.558 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:40.555+0000 7f3577200640 1 --2- 172.21.15.32:0/2153172023 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f3558077010 0x7f3558079520 unknown :-1 s=CLOSED pgs=574 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:40.558 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:40.555+0000 7f3577200640 1 --2- 172.21.15.32:0/2153172023 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f35780d26e0 0x7f35781f5a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:40.558 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:40.555+0000 7f3577200640 1 --2- 172.21.15.32:0/2153172023 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f35780cffe0 0x7f35781f5540 unknown :-1 s=CLOSED pgs=1402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:40.558 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:40.555+0000 7f3577200640 1 --2- 172.21.15.32:0/2153172023 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f35780b43d0 0x7f3578157120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:40.558 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:40.555+0000 7f3577200640 1 -- 172.21.15.32:0/2153172023 >> 172.21.15.32:0/2153172023 conn(0x7f357808d7a0 msgr2=0x7f3578068940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:56:40.558 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:40.556+0000 7f3577200640 1 -- 172.21.15.32:0/2153172023 shutdown_connections 2024-08-03T20:56:40.558 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:40.556+0000 7f3577200640 1 -- 172.21.15.32:0/2153172023 wait complete. 2024-08-03T20:56:40.582 DEBUG:tasks.ceph_test_case:wait_until_true: success in 0s and 0 retries 2024-08-03T20:56:40.582 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-08-03T20:56:40.812 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:40.810+0000 7f29f1600640 1 -- 172.21.15.32:0/932648991 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f29ec15fff0 msgr2=0x7f29ec160400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:56:40.813 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:40.810+0000 7f29f1600640 1 --2- 172.21.15.32:0/932648991 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f29ec15fff0 0x7f29ec160400 secure :-1 s=READY pgs=1404 cs=0 l=1 rev1=1 crypto rx=0x7f29ec0afaf0 tx=0x7f29e402a8b0 comp rx=0 tx=0).stop 2024-08-03T20:56:40.813 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:40.811+0000 7f29f1600640 1 -- 172.21.15.32:0/932648991 shutdown_connections 2024-08-03T20:56:40.813 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:40.811+0000 7f29f1600640 1 --2- 172.21.15.32:0/932648991 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f29ec15fff0 0x7f29ec160400 unknown :-1 s=CLOSED pgs=1404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:40.813 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:40.811+0000 7f29f1600640 1 --2- 172.21.15.32:0/932648991 >> v2:172.21.15.118:3300/0 conn(0x7f29ec144f90 0x7f29ec074570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:40.813 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:40.811+0000 7f29f1600640 1 -- 172.21.15.32:0/932648991 >> 172.21.15.32:0/932648991 conn(0x7f29ec0b1bd0 msgr2=0x7f29ec0b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:56:40.813 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:40.811+0000 7f29f1600640 1 -- 172.21.15.32:0/932648991 shutdown_connections 2024-08-03T20:56:40.813 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:40.811+0000 7f29f1600640 1 -- 172.21.15.32:0/932648991 wait complete. 2024-08-03T20:56:40.814 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:40.813+0000 7f29f1600640 1 Processor -- start 2024-08-03T20:56:40.814 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:40.813+0000 7f29f1600640 1 -- start start 2024-08-03T20:56:40.818 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:40.815+0000 7f29f1600640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f29ec06e930 0x7f29ec1f26a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:56:40.818 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:40.815+0000 7f29f1600640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f29ec144f90 0x7f29ec15ca40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:56:40.818 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:40.815+0000 7f29f1600640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f29ec15fff0 0x7f29ec1f8870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:56:40.818 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:40.815+0000 7f29f1600640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f29ec0cf840 con 0x7f29ec06e930 2024-08-03T20:56:40.818 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:40.815+0000 7f29f1600640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f29ec145f30 con 0x7f29ec15fff0 2024-08-03T20:56:40.818 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:40.815+0000 7f29f1600640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7f29ec0cf9c0 con 0x7f29ec144f90 2024-08-03T20:56:40.818 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:40.815+0000 7f29eaa00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f29ec06e930 0x7f29ec1f26a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:56:40.818 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:40.815+0000 7f29eaa00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f29ec06e930 0x7f29ec1f26a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:49970/0 (socket says 172.21.15.32:49970) 2024-08-03T20:56:40.818 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:40.815+0000 7f29eaa00640 1 -- 172.21.15.32:0/2303416045 learned_addr learned my addr 172.21.15.32:0/2303416045 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:56:40.818 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:40.815+0000 7f29e3e00640 1 --2- 172.21.15.32:0/2303416045 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f29ec144f90 0x7f29ec15ca40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:56:40.819 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:40.815+0000 7f29eb400640 1 --2- 172.21.15.32:0/2303416045 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f29ec15fff0 0x7f29ec1f8870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:56:40.819 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:40.815+0000 7f29eaa00640 1 -- 172.21.15.32:0/2303416045 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f29ec144f90 msgr2=0x7f29ec15ca40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:56:40.819 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:40.815+0000 7f29eaa00640 1 --2- 172.21.15.32:0/2303416045 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f29ec144f90 0x7f29ec15ca40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:40.819 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:40.815+0000 7f29eaa00640 1 -- 172.21.15.32:0/2303416045 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f29ec15fff0 msgr2=0x7f29ec1f8870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:56:40.819 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:40.815+0000 7f29eaa00640 1 --2- 172.21.15.32:0/2303416045 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f29ec15fff0 0x7f29ec1f8870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:40.819 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:40.815+0000 7f29eaa00640 1 -- 172.21.15.32:0/2303416045 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f29ec15d6c0 con 0x7f29ec06e930 2024-08-03T20:56:40.819 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:40.816+0000 7f29eaa00640 1 --2- 172.21.15.32:0/2303416045 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f29ec06e930 0x7f29ec1f26a0 secure :-1 s=READY pgs=1405 cs=0 l=1 rev1=1 crypto rx=0x7f29dc00c430 tx=0x7f29dc00c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:56:40.819 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:40.816+0000 7f29e2a00640 1 -- 172.21.15.32:0/2303416045 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29dc011790 con 0x7f29ec06e930 2024-08-03T20:56:40.819 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:40.816+0000 7f29e2a00640 1 -- 172.21.15.32:0/2303416045 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f29dc017070 con 0x7f29ec06e930 2024-08-03T20:56:40.819 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:40.816+0000 7f29e2a00640 1 -- 172.21.15.32:0/2303416045 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29dc011ce0 con 0x7f29ec06e930 2024-08-03T20:56:40.819 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:40.816+0000 7f29f1600640 1 -- 172.21.15.32:0/2303416045 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f29ec15d9a0 con 0x7f29ec06e930 2024-08-03T20:56:40.819 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:40.816+0000 7f29f1600640 1 -- 172.21.15.32:0/2303416045 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f29ec15df60 con 0x7f29ec06e930 2024-08-03T20:56:40.820 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:40.817+0000 7f29f1600640 1 -- 172.21.15.32:0/2303416045 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f29ec160c30 con 0x7f29ec06e930 2024-08-03T20:56:40.820 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:40.818+0000 7f29e2a00640 1 -- 172.21.15.32:0/2303416045 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f29dc01c460 con 0x7f29ec06e930 2024-08-03T20:56:40.820 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:40.819+0000 7f29e2a00640 1 --2- 172.21.15.32:0/2303416045 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f29c8077010 0x7f29c8079520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:56:40.821 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:40.819+0000 7f29e2a00640 1 -- 172.21.15.32:0/2303416045 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 7058+0+0 (secure 0 0 0) 0x7f29dc098d50 con 0x7f29ec06e930 2024-08-03T20:56:40.821 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:40.820+0000 7f29e3e00640 1 --2- 172.21.15.32:0/2303416045 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f29c8077010 0x7f29c8079520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:56:40.822 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:40.821+0000 7f29e3e00640 1 --2- 172.21.15.32:0/2303416045 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f29c8077010 0x7f29c8079520 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7f29d4005fd0 tx=0x7f29d4005850 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:56:40.825 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:40.824+0000 7f29e2a00640 1 -- 172.21.15.32:0/2303416045 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7f29dc061960 con 0x7f29ec06e930 2024-08-03T20:56:41.070 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:41.068+0000 7f29f1600640 1 -- 172.21.15.32:0/2303416045 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f29ec0b2110 con 0x7f29ec06e930 2024-08-03T20:56:41.070 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:41.069+0000 7f29e2a00640 1 -- 172.21.15.32:0/2303416045 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 173 v173) ==== 95+0+4390 (secure 0 0 0) 0x7f29dc061260 con 0x7f29ec06e930 2024-08-03T20:56:41.071 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-03T20:56:41.071 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":173,"btime":"2024-08-03T20:56:40:045853+0000","default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6439,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6835/531080160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6834","nonce":531080160},{"type":"v1","addr":"172.21.15.118:6835","nonce":531080160}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":84},{"gid":6454,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6837/2796882319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6836","nonce":2796882319},{"type":"v1","addr":"172.21.15.118:6837","nonce":2796882319}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":86},{"gid":6813,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6834/2686933890","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":2686933890},{"type":"v1","addr":"172.21.15.32:6834","nonce":2686933890}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":85}],"filesystems":[{"mdsmap":{"epoch":173,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-08-03T20:54:26.950518+0000","modified":"2024-08-03T20:56:40.045781+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":99,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":6}]} 2024-08-03T20:56:41.071 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 173 2024-08-03T20:56:41.073 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:41.071+0000 7f29f1600640 1 -- 172.21.15.32:0/2303416045 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f29c8077010 msgr2=0x7f29c8079520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:56:41.073 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:41.071+0000 7f29f1600640 1 --2- 172.21.15.32:0/2303416045 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f29c8077010 0x7f29c8079520 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7f29d4005fd0 tx=0x7f29d4005850 comp rx=0 tx=0).stop 2024-08-03T20:56:41.073 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:41.071+0000 7f29f1600640 1 -- 172.21.15.32:0/2303416045 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f29ec06e930 msgr2=0x7f29ec1f26a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:56:41.073 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:41.071+0000 7f29f1600640 1 --2- 172.21.15.32:0/2303416045 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f29ec06e930 0x7f29ec1f26a0 secure :-1 s=READY pgs=1405 cs=0 l=1 rev1=1 crypto rx=0x7f29dc00c430 tx=0x7f29dc00c8f0 comp rx=0 tx=0).stop 2024-08-03T20:56:41.073 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:41.072+0000 7f29f1600640 1 -- 172.21.15.32:0/2303416045 shutdown_connections 2024-08-03T20:56:41.073 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:41.072+0000 7f29f1600640 1 --2- 172.21.15.32:0/2303416045 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f29c8077010 0x7f29c8079520 unknown :-1 s=CLOSED pgs=575 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:41.074 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:41.072+0000 7f29f1600640 1 --2- 172.21.15.32:0/2303416045 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f29ec15fff0 0x7f29ec1f8870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:41.074 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:41.072+0000 7f29f1600640 1 --2- 172.21.15.32:0/2303416045 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f29ec144f90 0x7f29ec15ca40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:41.074 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:41.072+0000 7f29f1600640 1 --2- 172.21.15.32:0/2303416045 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f29ec06e930 0x7f29ec1f26a0 unknown :-1 s=CLOSED pgs=1405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:41.074 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:41.072+0000 7f29f1600640 1 -- 172.21.15.32:0/2303416045 >> 172.21.15.32:0/2303416045 conn(0x7f29ec0b1bd0 msgr2=0x7f29ec0b4a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:56:41.074 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:41.072+0000 7f29f1600640 1 -- 172.21.15.32:0/2303416045 shutdown_connections 2024-08-03T20:56:41.074 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:41.072+0000 7f29f1600640 1 -- 172.21.15.32:0/2303416045 wait complete. 2024-08-03T20:56:41.096 DEBUG:tasks.ceph_test_case:wait_until_true: success in 0s and 0 retries 2024-08-03T20:56:41.096 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-08-03T20:56:41.327 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:41.325+0000 7f8e15c00640 1 -- 172.21.15.32:0/327343612 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8e1006e930 msgr2=0x7f8e10144f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:56:41.327 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:41.325+0000 7f8e15c00640 1 --2- 172.21.15.32:0/327343612 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8e1006e930 0x7f8e10144f90 secure :-1 s=READY pgs=1407 cs=0 l=1 rev1=1 crypto rx=0x7f8e00009830 tx=0x7f8e0002a480 comp rx=0 tx=0).stop 2024-08-03T20:56:41.327 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:41.325+0000 7f8e15c00640 1 -- 172.21.15.32:0/327343612 shutdown_connections 2024-08-03T20:56:41.328 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:41.325+0000 7f8e15c00640 1 --2- 172.21.15.32:0/327343612 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f8e10073e60 0x7f8e10074270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:41.328 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:41.325+0000 7f8e15c00640 1 --2- 172.21.15.32:0/327343612 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8e1006e930 0x7f8e10144f90 unknown :-1 s=CLOSED pgs=1407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:41.328 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:41.325+0000 7f8e15c00640 1 -- 172.21.15.32:0/327343612 >> 172.21.15.32:0/327343612 conn(0x7f8e100b1bd0 msgr2=0x7f8e100b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:56:41.328 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:41.326+0000 7f8e15c00640 1 -- 172.21.15.32:0/327343612 shutdown_connections 2024-08-03T20:56:41.328 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:41.326+0000 7f8e15c00640 1 -- 172.21.15.32:0/327343612 wait complete. 2024-08-03T20:56:41.329 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:41.328+0000 7f8e15c00640 1 Processor -- start 2024-08-03T20:56:41.329 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:41.328+0000 7f8e15c00640 1 -- start start 2024-08-03T20:56:41.332 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:41.329+0000 7f8e15c00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8e1006e930 0x7f8e1013f4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:56:41.333 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:41.329+0000 7f8e15c00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8e10073e60 0x7f8e100cb8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:56:41.333 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:41.329+0000 7f8e15c00640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f8e100747b0 0x7f8e101f8660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:56:41.333 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:41.329+0000 7f8e15c00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f8e100cf9c0 con 0x7f8e1006e930 2024-08-03T20:56:41.333 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:41.329+0000 7f8e15c00640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f8e100cf840 con 0x7f8e10073e60 2024-08-03T20:56:41.333 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:41.329+0000 7f8e15c00640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7f8e100ccff0 con 0x7f8e100747b0 2024-08-03T20:56:41.333 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:41.330+0000 7f8e0ea00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8e1006e930 0x7f8e1013f4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:56:41.333 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:41.330+0000 7f8e0ea00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8e1006e930 0x7f8e1013f4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:49990/0 (socket says 172.21.15.32:49990) 2024-08-03T20:56:41.333 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:41.330+0000 7f8e0ea00640 1 -- 172.21.15.32:0/1938179801 learned_addr learned my addr 172.21.15.32:0/1938179801 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:56:41.333 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:41.330+0000 7f8e0ea00640 1 -- 172.21.15.32:0/1938179801 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f8e100747b0 msgr2=0x7f8e101f8660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:56:41.333 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:41.330+0000 7f8e0f400640 1 --2- 172.21.15.32:0/1938179801 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f8e100747b0 0x7f8e101f8660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:56:41.333 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:41.330+0000 7f8e0ea00640 1 --2- 172.21.15.32:0/1938179801 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f8e100747b0 0x7f8e101f8660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:41.334 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:41.330+0000 7f8e0ea00640 1 -- 172.21.15.32:0/1938179801 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8e10073e60 msgr2=0x7f8e100cb8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:56:41.334 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:41.330+0000 7f8e07e00640 1 --2- 172.21.15.32:0/1938179801 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8e10073e60 0x7f8e100cb8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:56:41.334 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:41.330+0000 7f8e0ea00640 1 --2- 172.21.15.32:0/1938179801 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8e10073e60 0x7f8e100cb8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:41.334 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:41.330+0000 7f8e0ea00640 1 -- 172.21.15.32:0/1938179801 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8e1013f9e0 con 0x7f8e1006e930 2024-08-03T20:56:41.334 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:41.330+0000 7f8e07e00640 1 --2- 172.21.15.32:0/1938179801 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8e10073e60 0x7f8e100cb8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-03T20:56:41.334 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:41.330+0000 7f8e0ea00640 1 --2- 172.21.15.32:0/1938179801 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8e1006e930 0x7f8e1013f4a0 secure :-1 s=READY pgs=1408 cs=0 l=1 rev1=1 crypto rx=0x7f8e0002aa70 tx=0x7f8e00033c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:56:41.334 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:41.331+0000 7f8e06a00640 1 -- 172.21.15.32:0/1938179801 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e00005c90 con 0x7f8e1006e930 2024-08-03T20:56:41.334 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:41.331+0000 7f8e06a00640 1 -- 172.21.15.32:0/1938179801 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f8e00031e50 con 0x7f8e1006e930 2024-08-03T20:56:41.334 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:41.331+0000 7f8e06a00640 1 -- 172.21.15.32:0/1938179801 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e00005260 con 0x7f8e1006e930 2024-08-03T20:56:41.334 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:41.331+0000 7f8e15c00640 1 -- 172.21.15.32:0/1938179801 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8e100b3d00 con 0x7f8e1006e930 2024-08-03T20:56:41.334 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:41.331+0000 7f8e15c00640 1 -- 172.21.15.32:0/1938179801 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8e100b42c0 con 0x7f8e1006e930 2024-08-03T20:56:41.335 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:41.332+0000 7f8e15c00640 1 -- 172.21.15.32:0/1938179801 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8e100cd510 con 0x7f8e1006e930 2024-08-03T20:56:41.335 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:41.333+0000 7f8e06a00640 1 -- 172.21.15.32:0/1938179801 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f8e00005400 con 0x7f8e1006e930 2024-08-03T20:56:41.336 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:41.334+0000 7f8e06a00640 1 --2- 172.21.15.32:0/1938179801 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f8dec077010 0x7f8dec079520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:56:41.336 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:41.334+0000 7f8e06a00640 1 -- 172.21.15.32:0/1938179801 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 7058+0+0 (secure 0 0 0) 0x7f8e000b9f20 con 0x7f8e1006e930 2024-08-03T20:56:41.336 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:41.335+0000 7f8e07e00640 1 --2- 172.21.15.32:0/1938179801 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f8dec077010 0x7f8dec079520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:56:41.337 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:41.335+0000 7f8e07e00640 1 --2- 172.21.15.32:0/1938179801 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f8dec077010 0x7f8dec079520 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7f8df8009ab0 tx=0x7f8df8005f90 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:56:41.340 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:41.339+0000 7f8e06a00640 1 -- 172.21.15.32:0/1938179801 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7f8e1013f9e0 con 0x7f8e1006e930 2024-08-03T20:56:41.582 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:41.580+0000 7f8e15c00640 1 -- 172.21.15.32:0/1938179801 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f8e100cbe10 con 0x7f8e1006e930 2024-08-03T20:56:41.582 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:41.581+0000 7f8e06a00640 1 -- 172.21.15.32:0/1938179801 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 173 v173) ==== 95+0+4390 (secure 0 0 0) 0x7f8e0003f030 con 0x7f8e1006e930 2024-08-03T20:56:41.583 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-03T20:56:41.583 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":173,"btime":"2024-08-03T20:56:40:045853+0000","default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6439,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6835/531080160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6834","nonce":531080160},{"type":"v1","addr":"172.21.15.118:6835","nonce":531080160}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":84},{"gid":6454,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6837/2796882319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6836","nonce":2796882319},{"type":"v1","addr":"172.21.15.118:6837","nonce":2796882319}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":86},{"gid":6813,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6834/2686933890","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":2686933890},{"type":"v1","addr":"172.21.15.32:6834","nonce":2686933890}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":85}],"filesystems":[{"mdsmap":{"epoch":173,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-08-03T20:54:26.950518+0000","modified":"2024-08-03T20:56:40.045781+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":99,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":6}]} 2024-08-03T20:56:41.583 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 173 2024-08-03T20:56:41.585 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:41.583+0000 7f8e15c00640 1 -- 172.21.15.32:0/1938179801 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f8dec077010 msgr2=0x7f8dec079520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:56:41.585 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:41.583+0000 7f8e15c00640 1 --2- 172.21.15.32:0/1938179801 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f8dec077010 0x7f8dec079520 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7f8df8009ab0 tx=0x7f8df8005f90 comp rx=0 tx=0).stop 2024-08-03T20:56:41.585 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:41.583+0000 7f8e15c00640 1 -- 172.21.15.32:0/1938179801 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8e1006e930 msgr2=0x7f8e1013f4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:56:41.586 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:41.583+0000 7f8e15c00640 1 --2- 172.21.15.32:0/1938179801 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8e1006e930 0x7f8e1013f4a0 secure :-1 s=READY pgs=1408 cs=0 l=1 rev1=1 crypto rx=0x7f8e0002aa70 tx=0x7f8e00033c20 comp rx=0 tx=0).stop 2024-08-03T20:56:41.586 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:41.583+0000 7f8e15c00640 1 -- 172.21.15.32:0/1938179801 shutdown_connections 2024-08-03T20:56:41.586 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:41.583+0000 7f8e15c00640 1 --2- 172.21.15.32:0/1938179801 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f8dec077010 0x7f8dec079520 unknown :-1 s=CLOSED pgs=576 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:41.586 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:41.583+0000 7f8e15c00640 1 --2- 172.21.15.32:0/1938179801 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8e10073e60 0x7f8e100cb8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:41.586 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:41.583+0000 7f8e15c00640 1 --2- 172.21.15.32:0/1938179801 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f8e100747b0 0x7f8e101f8660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:41.586 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:41.583+0000 7f8e15c00640 1 --2- 172.21.15.32:0/1938179801 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8e1006e930 0x7f8e1013f4a0 unknown :-1 s=CLOSED pgs=1408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:41.586 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:41.583+0000 7f8e15c00640 1 -- 172.21.15.32:0/1938179801 >> 172.21.15.32:0/1938179801 conn(0x7f8e100b1bd0 msgr2=0x7f8e100afb40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:56:41.587 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:41.583+0000 7f8e15c00640 1 -- 172.21.15.32:0/1938179801 shutdown_connections 2024-08-03T20:56:41.587 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:41.583+0000 7f8e15c00640 1 -- 172.21.15.32:0/1938179801 wait complete. 2024-08-03T20:56:41.610 INFO:tasks.cephfs.test_damage:Result: Mutation 'Delete 100.00000000.inode' led to DAMAGED state 2024-08-03T20:56:41.610 INFO:tasks.cephfs.test_damage:Applying mutation 'Delete 1.00000000' 2024-08-03T20:56:41.610 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-08-03T20:56:41.610 DEBUG:teuthology.orchestra.run.smithi032:> dd if=/proc/self/mounts of=/dev/stdout 2024-08-03T20:56:41.646 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053453,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6499208k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 none /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249604k,nr_inodes=812401,mode=700,uid=1000,gid=1316 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6499208k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-08-03T20:56:41.646 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi032.front.sepia.ceph.com 2024-08-03T20:56:41.646 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-08-03T20:56:41.646 DEBUG:teuthology.orchestra.run.smithi032:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-08-03T20:56:41.722 INFO:teuthology.orchestra.run.smithi032.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.0': No such file or directory 2024-08-03T20:56:41.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-03T20:56:41.723 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-08-03T20:56:41.992 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:41.990+0000 7f3b8b200640 1 -- 172.21.15.32:0/1167496484 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3b8c0acd80 msgr2=0x7f3b8c0ad190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:56:41.992 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:41.990+0000 7f3b8b200640 1 --2- 172.21.15.32:0/1167496484 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3b8c0acd80 0x7f3b8c0ad190 secure :-1 s=READY pgs=1410 cs=0 l=1 rev1=1 crypto rx=0x7f3b8c0723c0 tx=0x7f3b7402a4a0 comp rx=0 tx=0).stop 2024-08-03T20:56:41.992 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:41.990+0000 7f3b8b200640 1 -- 172.21.15.32:0/1167496484 shutdown_connections 2024-08-03T20:56:41.992 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:41.990+0000 7f3b8b200640 1 --2- 172.21.15.32:0/1167496484 >> v2:172.21.15.118:3300/0 conn(0x7f3b8c06f440 0x7f3b8c06f850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:41.992 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:41.990+0000 7f3b8b200640 1 --2- 172.21.15.32:0/1167496484 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3b8c0acd80 0x7f3b8c0ad190 unknown :-1 s=CLOSED pgs=1410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:41.992 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:41.990+0000 7f3b8b200640 1 -- 172.21.15.32:0/1167496484 >> 172.21.15.32:0/1167496484 conn(0x7f3b8c0b2120 msgr2=0x7f3b8c0cfd60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:56:41.993 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:41.990+0000 7f3b8b200640 1 -- 172.21.15.32:0/1167496484 shutdown_connections 2024-08-03T20:56:41.993 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:41.990+0000 7f3b8b200640 1 -- 172.21.15.32:0/1167496484 wait complete. 2024-08-03T20:56:41.994 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:41.992+0000 7f3b8b200640 1 Processor -- start 2024-08-03T20:56:41.994 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:41.993+0000 7f3b8b200640 1 -- start start 2024-08-03T20:56:41.997 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:41.994+0000 7f3b8b200640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f3b8c06f440 0x7f3b8c0b3420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:56:41.997 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:41.994+0000 7f3b8b200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3b8c0acd80 0x7f3b8c1ef750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:56:41.997 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:41.994+0000 7f3b8b200640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f3b8c0af020 0x7f3b8c15b360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:56:41.997 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:41.994+0000 7f3b8b200640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f3b8c1601e0 con 0x7f3b8c0acd80 2024-08-03T20:56:41.997 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:41.994+0000 7f3b8b200640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f3b8c160060 con 0x7f3b8c06f440 2024-08-03T20:56:41.997 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:41.994+0000 7f3b8b200640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7f3b8c160360 con 0x7f3b8c0af020 2024-08-03T20:56:41.997 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:41.994+0000 7f3b83e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3b8c0acd80 0x7f3b8c1ef750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:56:41.997 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:41.994+0000 7f3b83e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3b8c0acd80 0x7f3b8c1ef750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:50010/0 (socket says 172.21.15.32:50010) 2024-08-03T20:56:41.998 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:41.994+0000 7f3b83e00640 1 -- 172.21.15.32:0/3330952834 learned_addr learned my addr 172.21.15.32:0/3330952834 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:56:41.998 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:41.994+0000 7f3b89e00640 1 --2- 172.21.15.32:0/3330952834 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f3b8c06f440 0x7f3b8c0b3420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:56:41.998 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:41.994+0000 7f3b8a800640 1 --2- 172.21.15.32:0/3330952834 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f3b8c0af020 0x7f3b8c15b360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:56:41.998 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:41.995+0000 7f3b83e00640 1 -- 172.21.15.32:0/3330952834 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f3b8c0af020 msgr2=0x7f3b8c15b360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:56:41.998 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:41.995+0000 7f3b83e00640 1 --2- 172.21.15.32:0/3330952834 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f3b8c0af020 0x7f3b8c15b360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:41.998 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:41.995+0000 7f3b83e00640 1 -- 172.21.15.32:0/3330952834 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f3b8c06f440 msgr2=0x7f3b8c0b3420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:56:41.998 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:41.995+0000 7f3b83e00640 1 --2- 172.21.15.32:0/3330952834 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f3b8c06f440 0x7f3b8c0b3420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:41.998 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:41.995+0000 7f3b83e00640 1 -- 172.21.15.32:0/3330952834 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3b8c1f8dd0 con 0x7f3b8c0acd80 2024-08-03T20:56:41.998 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:41.995+0000 7f3b8a800640 1 --2- 172.21.15.32:0/3330952834 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f3b8c0af020 0x7f3b8c15b360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-03T20:56:41.998 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:41.995+0000 7f3b83e00640 1 --2- 172.21.15.32:0/3330952834 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3b8c0acd80 0x7f3b8c1ef750 secure :-1 s=READY pgs=1411 cs=0 l=1 rev1=1 crypto rx=0x7f3b74009920 tx=0x7f3b74033c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:56:41.998 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:41.995+0000 7f3b82000640 1 -- 172.21.15.32:0/3330952834 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b74005c90 con 0x7f3b8c0acd80 2024-08-03T20:56:41.999 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:41.995+0000 7f3b82000640 1 -- 172.21.15.32:0/3330952834 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f3b74031e50 con 0x7f3b8c0acd80 2024-08-03T20:56:41.999 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:41.995+0000 7f3b82000640 1 -- 172.21.15.32:0/3330952834 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b74005240 con 0x7f3b8c0acd80 2024-08-03T20:56:41.999 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:41.995+0000 7f3b8b200640 1 -- 172.21.15.32:0/3330952834 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3b8c1f90b0 con 0x7f3b8c0acd80 2024-08-03T20:56:41.999 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:41.996+0000 7f3b8b200640 1 -- 172.21.15.32:0/3330952834 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3b8c1f9670 con 0x7f3b8c0acd80 2024-08-03T20:56:41.999 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:41.997+0000 7f3b8b200640 1 -- 172.21.15.32:0/3330952834 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3b8c0ad190 con 0x7f3b8c0acd80 2024-08-03T20:56:41.999 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:41.998+0000 7f3b82000640 1 -- 172.21.15.32:0/3330952834 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f3b740053e0 con 0x7f3b8c0acd80 2024-08-03T20:56:42.001 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:41.999+0000 7f3b82000640 1 --2- 172.21.15.32:0/3330952834 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f3b68077010 0x7f3b68079520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:56:42.001 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:41.999+0000 7f3b82000640 1 -- 172.21.15.32:0/3330952834 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 7058+0+0 (secure 0 0 0) 0x7f3b740b9ea0 con 0x7f3b8c0acd80 2024-08-03T20:56:42.001 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:41.999+0000 7f3b89e00640 1 --2- 172.21.15.32:0/3330952834 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f3b68077010 0x7f3b68079520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:56:42.002 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:42.000+0000 7f3b89e00640 1 --2- 172.21.15.32:0/3330952834 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f3b68077010 0x7f3b68079520 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7f3b7c000f60 tx=0x7f3b7c009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:56:42.005 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:42.003+0000 7f3b82000640 1 -- 172.21.15.32:0/3330952834 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7f3b7403f030 con 0x7f3b8c0acd80 2024-08-03T20:56:42.251 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:42.250+0000 7f3b8b200640 1 -- 172.21.15.32:0/3330952834 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7f3b8c0d0930 con 0x7f3b8c0acd80 2024-08-03T20:56:42.297 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:42.296+0000 7f3b82000640 1 -- 172.21.15.32:0/3330952834 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v174) ==== 161+0+0 (secure 0 0 0) 0x7f3b74031770 con 0x7f3b8c0acd80 2024-08-03T20:56:42.297 INFO:teuthology.orchestra.run.smithi032.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-08-03T20:56:42.300 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:42.298+0000 7f3b8b200640 1 -- 172.21.15.32:0/3330952834 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f3b68077010 msgr2=0x7f3b68079520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:56:42.300 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:42.298+0000 7f3b8b200640 1 --2- 172.21.15.32:0/3330952834 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f3b68077010 0x7f3b68079520 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7f3b7c000f60 tx=0x7f3b7c009040 comp rx=0 tx=0).stop 2024-08-03T20:56:42.300 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:42.298+0000 7f3b8b200640 1 -- 172.21.15.32:0/3330952834 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3b8c0acd80 msgr2=0x7f3b8c1ef750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:56:42.300 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:42.298+0000 7f3b8b200640 1 --2- 172.21.15.32:0/3330952834 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3b8c0acd80 0x7f3b8c1ef750 secure :-1 s=READY pgs=1411 cs=0 l=1 rev1=1 crypto rx=0x7f3b74009920 tx=0x7f3b74033c20 comp rx=0 tx=0).stop 2024-08-03T20:56:42.300 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:42.298+0000 7f3b8b200640 1 -- 172.21.15.32:0/3330952834 shutdown_connections 2024-08-03T20:56:42.301 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:42.298+0000 7f3b8b200640 1 --2- 172.21.15.32:0/3330952834 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f3b68077010 0x7f3b68079520 unknown :-1 s=CLOSED pgs=577 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:42.301 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:42.298+0000 7f3b8b200640 1 --2- 172.21.15.32:0/3330952834 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f3b8c0af020 0x7f3b8c15b360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:42.301 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:42.298+0000 7f3b8b200640 1 --2- 172.21.15.32:0/3330952834 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3b8c0acd80 0x7f3b8c1ef750 unknown :-1 s=CLOSED pgs=1411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:42.301 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:42.298+0000 7f3b8b200640 1 --2- 172.21.15.32:0/3330952834 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f3b8c06f440 0x7f3b8c0b3420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:42.301 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:42.298+0000 7f3b8b200640 1 -- 172.21.15.32:0/3330952834 >> 172.21.15.32:0/3330952834 conn(0x7f3b8c0b2120 msgr2=0x7f3b8c0b07f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:56:42.301 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:42.298+0000 7f3b8b200640 1 -- 172.21.15.32:0/3330952834 shutdown_connections 2024-08-03T20:56:42.301 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:42.299+0000 7f3b8b200640 1 -- 172.21.15.32:0/3330952834 wait complete. 2024-08-03T20:56:42.324 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mds repaired 0 2024-08-03T20:56:42.557 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:42.555+0000 7f2248e00640 1 -- 172.21.15.32:0/3550461965 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2228004800 msgr2=0x7f2228004c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:56:42.558 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:42.555+0000 7f2248e00640 1 --2- 172.21.15.32:0/3550461965 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2228004800 0x7f2228004c10 secure :-1 s=READY pgs=1413 cs=0 l=1 rev1=1 crypto rx=0x7f222c009b50 tx=0x7f2228005830 comp rx=0 tx=0).stop 2024-08-03T20:56:42.558 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:42.555+0000 7f2248e00640 1 -- 172.21.15.32:0/3550461965 shutdown_connections 2024-08-03T20:56:42.558 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:42.555+0000 7f2248e00640 1 --2- 172.21.15.32:0/3550461965 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2228004800 0x7f2228004c10 unknown :-1 s=CLOSED pgs=1413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:42.558 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:42.555+0000 7f2248e00640 1 -- 172.21.15.32:0/3550461965 >> 172.21.15.32:0/3550461965 conn(0x7f224408d7a0 msgr2=0x7f224409d3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:56:42.558 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:42.555+0000 7f2248e00640 1 -- 172.21.15.32:0/3550461965 shutdown_connections 2024-08-03T20:56:42.559 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:42.555+0000 7f2248e00640 1 -- 172.21.15.32:0/3550461965 wait complete. 2024-08-03T20:56:42.559 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:42.557+0000 7f2248e00640 1 Processor -- start 2024-08-03T20:56:42.560 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:42.558+0000 7f2248e00640 1 -- start start 2024-08-03T20:56:42.564 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:42.559+0000 7f2248e00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f2228004800 0x7f22441f5b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:56:42.564 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:42.559+0000 7f2248e00640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f22440cb1b0 0x7f22440c7c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:56:42.565 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:42.559+0000 7f2248e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f22440cb5c0 0x7f22440ca270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:56:42.565 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:42.559+0000 7f2248e00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f2244161d00 con 0x7f22440cb5c0 2024-08-03T20:56:42.565 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:42.559+0000 7f2248e00640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f2244161b80 con 0x7f2228004800 2024-08-03T20:56:42.565 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:42.559+0000 7f2248e00640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7f2244161e80 con 0x7f22440cb1b0 2024-08-03T20:56:42.565 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:42.559+0000 7f2243400640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f2228004800 0x7f22441f5b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:56:42.566 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:42.559+0000 7f2243e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f22440cb5c0 0x7f22440ca270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:56:42.566 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:42.559+0000 7f223aa00640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f22440cb1b0 0x7f22440c7c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:56:42.566 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:42.559+0000 7f2243400640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f2228004800 0x7f22441f5b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.32:54844/0 (socket says 172.21.15.32:54844) 2024-08-03T20:56:42.566 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:42.559+0000 7f2243400640 1 -- 172.21.15.32:0/2573207162 learned_addr learned my addr 172.21.15.32:0/2573207162 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:56:42.566 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:42.560+0000 7f2243400640 1 -- 172.21.15.32:0/2573207162 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f22440cb1b0 msgr2=0x7f22440c7c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:56:42.567 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:42.560+0000 7f2243400640 1 --2- 172.21.15.32:0/2573207162 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f22440cb1b0 0x7f22440c7c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:42.567 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:42.560+0000 7f2243400640 1 -- 172.21.15.32:0/2573207162 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f22440cb5c0 msgr2=0x7f22440ca270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:56:42.567 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:42.560+0000 7f2243400640 1 --2- 172.21.15.32:0/2573207162 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f22440cb5c0 0x7f22440ca270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:42.567 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:42.560+0000 7f2243400640 1 -- 172.21.15.32:0/2573207162 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f22440c8840 con 0x7f2228004800 2024-08-03T20:56:42.567 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:42.560+0000 7f223aa00640 1 --2- 172.21.15.32:0/2573207162 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f22440cb1b0 0x7f22440c7c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-03T20:56:42.567 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:42.560+0000 7f2243e00640 1 --2- 172.21.15.32:0/2573207162 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f22440cb5c0 0x7f22440ca270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-03T20:56:42.568 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:42.560+0000 7f2243400640 1 --2- 172.21.15.32:0/2573207162 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f2228004800 0x7f22441f5b90 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f2234006fc0 tx=0x7f2234001860 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:56:42.568 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:42.561+0000 7f223be00640 1 -- 172.21.15.32:0/2573207162 <== mon.1 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2234011700 con 0x7f2228004800 2024-08-03T20:56:42.568 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:42.561+0000 7f223be00640 1 -- 172.21.15.32:0/2573207162 <== mon.1 v2:172.21.15.118:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f2234016020 con 0x7f2228004800 2024-08-03T20:56:42.568 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:42.561+0000 7f223be00640 1 -- 172.21.15.32:0/2573207162 <== mon.1 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2234011b40 con 0x7f2228004800 2024-08-03T20:56:42.568 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:42.561+0000 7f2248e00640 1 -- 172.21.15.32:0/2573207162 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f22440c8b20 con 0x7f2228004800 2024-08-03T20:56:42.569 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:42.561+0000 7f2248e00640 1 -- 172.21.15.32:0/2573207162 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f22440c9a90 con 0x7f2228004800 2024-08-03T20:56:42.569 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:42.562+0000 7f2248e00640 1 -- 172.21.15.32:0/2573207162 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f22440cba30 con 0x7f2228004800 2024-08-03T20:56:42.570 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:42.563+0000 7f223be00640 1 -- 172.21.15.32:0/2573207162 <== mon.1 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f2234011ce0 con 0x7f2228004800 2024-08-03T20:56:42.571 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:42.564+0000 7f223be00640 1 --2- 172.21.15.32:0/2573207162 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f2220077010 0x7f2220079520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:56:42.571 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:42.564+0000 7f223be00640 1 -- 172.21.15.32:0/2573207162 <== mon.1 v2:172.21.15.118:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 7058+0+0 (secure 0 0 0) 0x7f2234099680 con 0x7f2228004800 2024-08-03T20:56:42.571 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:42.565+0000 7f223aa00640 1 --2- 172.21.15.32:0/2573207162 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f2220077010 0x7f2220079520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:56:42.571 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:42.566+0000 7f223aa00640 1 --2- 172.21.15.32:0/2573207162 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f2220077010 0x7f2220079520 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7f222c035000 tx=0x7f222c0046c0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:56:42.571 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:42.569+0000 7f223be00640 1 -- 172.21.15.32:0/2573207162 <== mon.1 v2:172.21.15.118:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7f2234020030 con 0x7f2228004800 2024-08-03T20:56:42.854 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:42.853+0000 7f2248e00640 1 -- 172.21.15.32:0/2573207162 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command({"prefix": "mds repaired", "role": "0"} v 0) -- 0x7f22440cf5f0 con 0x7f2228004800 2024-08-03T20:56:43.312 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:43.310+0000 7f223be00640 1 -- 172.21.15.32:0/2573207162 <== mon.1 v2:172.21.15.118:3300/0 7 ==== mon_command_ack([{"prefix": "mds repaired", "role": "0"}]=0 repaired: restoring rank 6:0 v175) ==== 101+0+0 (secure 0 0 0) 0x7f2234061ce0 con 0x7f2228004800 2024-08-03T20:56:43.312 INFO:teuthology.orchestra.run.smithi032.stderr:repaired: restoring rank 6:0 2024-08-03T20:56:43.315 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:43.313+0000 7f2248e00640 1 -- 172.21.15.32:0/2573207162 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f2220077010 msgr2=0x7f2220079520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:56:43.315 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:43.313+0000 7f2248e00640 1 --2- 172.21.15.32:0/2573207162 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f2220077010 0x7f2220079520 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7f222c035000 tx=0x7f222c0046c0 comp rx=0 tx=0).stop 2024-08-03T20:56:43.315 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:43.313+0000 7f2248e00640 1 -- 172.21.15.32:0/2573207162 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f2228004800 msgr2=0x7f22441f5b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:56:43.315 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:43.313+0000 7f2248e00640 1 --2- 172.21.15.32:0/2573207162 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f2228004800 0x7f22441f5b90 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f2234006fc0 tx=0x7f2234001860 comp rx=0 tx=0).stop 2024-08-03T20:56:43.315 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:43.313+0000 7f2248e00640 1 -- 172.21.15.32:0/2573207162 shutdown_connections 2024-08-03T20:56:43.315 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:43.313+0000 7f2248e00640 1 --2- 172.21.15.32:0/2573207162 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f2220077010 0x7f2220079520 unknown :-1 s=CLOSED pgs=578 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:43.315 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:43.313+0000 7f2248e00640 1 --2- 172.21.15.32:0/2573207162 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f22440cb5c0 0x7f22440ca270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:43.315 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:43.313+0000 7f2248e00640 1 --2- 172.21.15.32:0/2573207162 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f22440cb1b0 0x7f22440c7c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:43.315 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:43.313+0000 7f2248e00640 1 --2- 172.21.15.32:0/2573207162 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f2228004800 0x7f22441f5b90 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:43.315 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:43.313+0000 7f2248e00640 1 -- 172.21.15.32:0/2573207162 >> 172.21.15.32:0/2573207162 conn(0x7f224408d7a0 msgr2=0x7f22440d1cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:56:43.315 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:43.313+0000 7f2248e00640 1 -- 172.21.15.32:0/2573207162 shutdown_connections 2024-08-03T20:56:43.316 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:43.313+0000 7f2248e00640 1 -- 172.21.15.32:0/2573207162 wait complete. 2024-08-03T20:56:43.338 DEBUG:teuthology.orchestra.run.smithi032:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata import - 2024-08-03T20:56:43.485 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:43.483+0000 7f18123b7f40 1 Processor -- start 2024-08-03T20:56:43.485 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:43.483+0000 7f18123b7f40 1 -- start start 2024-08-03T20:56:43.488 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:43.485+0000 7f18123b7f40 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x555971493360 0x5559714ac340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:56:43.489 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:43.485+0000 7f18123b7f40 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555971494ea0 0x5559714af910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:56:43.489 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:43.485+0000 7f18123b7f40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5559714aa330 0x5559714adc40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:56:43.489 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:43.485+0000 7f18123b7f40 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x555971388bf0 con 0x555971494ea0 2024-08-03T20:56:43.489 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:43.485+0000 7f18123b7f40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x5559713956b0 con 0x5559714aa330 2024-08-03T20:56:43.489 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:43.485+0000 7f18123b7f40 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x55597132ae30 con 0x555971493360 2024-08-03T20:56:43.489 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:43.485+0000 7f180ac00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555971494ea0 0x5559714af910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:56:43.489 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:43.485+0000 7f180ac00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555971494ea0 0x5559714af910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:50052/0 (socket says 172.21.15.32:50052) 2024-08-03T20:56:43.489 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:43.485+0000 7f180ac00640 1 -- 172.21.15.32:0/737258235 learned_addr learned my addr 172.21.15.32:0/737258235 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:56:43.489 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:43.485+0000 7f180c000640 1 --2- 172.21.15.32:0/737258235 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5559714aa330 0x5559714adc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:56:43.489 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:43.485+0000 7f180b600640 1 --2- 172.21.15.32:0/737258235 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x555971493360 0x5559714ac340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:56:43.489 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:43.485+0000 7f180ac00640 1 -- 172.21.15.32:0/737258235 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x555971493360 msgr2=0x5559714ac340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:56:43.489 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:43.485+0000 7f180ac00640 1 --2- 172.21.15.32:0/737258235 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x555971493360 0x5559714ac340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:43.490 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:43.485+0000 7f180ac00640 1 -- 172.21.15.32:0/737258235 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5559714aa330 msgr2=0x5559714adc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:56:43.490 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:43.485+0000 7f180ac00640 1 --2- 172.21.15.32:0/737258235 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5559714aa330 0x5559714adc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:43.490 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:43.485+0000 7f180ac00640 1 -- 172.21.15.32:0/737258235 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555971425e50 con 0x555971494ea0 2024-08-03T20:56:43.490 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:43.485+0000 7f180b600640 1 --2- 172.21.15.32:0/737258235 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x555971493360 0x5559714ac340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-03T20:56:43.490 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:43.486+0000 7f180c000640 1 --2- 172.21.15.32:0/737258235 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5559714aa330 0x5559714adc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-03T20:56:43.490 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:43.486+0000 7f180ac00640 1 --2- 172.21.15.32:0/737258235 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555971494ea0 0x5559714af910 secure :-1 s=READY pgs=1416 cs=0 l=1 rev1=1 crypto rx=0x7f17f8004e90 tx=0x7f17f8002fb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3bd1f29db9d83fb server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:56:43.490 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:43.486+0000 7f1809800640 1 -- 172.21.15.32:0/737258235 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17f8066070 con 0x555971494ea0 2024-08-03T20:56:43.490 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:43.486+0000 7f1809800640 1 -- 172.21.15.32:0/737258235 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f17f8056760 con 0x555971494ea0 2024-08-03T20:56:43.490 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:43.486+0000 7f1809800640 1 -- 172.21.15.32:0/737258235 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17f8056bb0 con 0x555971494ea0 2024-08-03T20:56:43.490 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:43.487+0000 7f18123b7f40 1 -- 172.21.15.32:0/737258235 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555971494ea0 msgr2=0x5559714af910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:56:43.490 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:43.487+0000 7f18123b7f40 1 --2- 172.21.15.32:0/737258235 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555971494ea0 0x5559714af910 secure :-1 s=READY pgs=1416 cs=0 l=1 rev1=1 crypto rx=0x7f17f8004e90 tx=0x7f17f8002fb0 comp rx=0 tx=0).stop 2024-08-03T20:56:43.490 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:43.487+0000 7f18123b7f40 1 -- 172.21.15.32:0/737258235 shutdown_connections 2024-08-03T20:56:43.491 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:43.487+0000 7f18123b7f40 1 --2- 172.21.15.32:0/737258235 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5559714aa330 0x5559714adc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:43.491 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:43.487+0000 7f18123b7f40 1 --2- 172.21.15.32:0/737258235 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555971494ea0 0x5559714af910 unknown :-1 s=CLOSED pgs=1416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:43.491 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:43.487+0000 7f18123b7f40 1 --2- 172.21.15.32:0/737258235 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x555971493360 0x5559714ac340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:43.491 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:43.487+0000 7f18123b7f40 1 -- 172.21.15.32:0/737258235 >> 172.21.15.32:0/737258235 conn(0x5559712fcac0 msgr2=0x5559714acf90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:56:43.491 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:43.487+0000 7f18123b7f40 1 -- 172.21.15.32:0/737258235 shutdown_connections 2024-08-03T20:56:43.491 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:43.487+0000 7f18123b7f40 1 -- 172.21.15.32:0/737258235 wait complete. 2024-08-03T20:56:43.493 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:43.488+0000 7f18123b7f40 1 Processor -- start 2024-08-03T20:56:43.493 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:43.488+0000 7f18123b7f40 1 -- start start 2024-08-03T20:56:43.493 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:43.489+0000 7f18123b7f40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x555971493360 0x5559714b12e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:56:43.493 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:43.489+0000 7f18123b7f40 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x555971494ea0 0x5559713dc7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:56:43.493 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:43.489+0000 7f18123b7f40 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5559714aa330 0x55597148f610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:56:43.493 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:43.489+0000 7f18123b7f40 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55597148be90 con 0x5559714aa330 2024-08-03T20:56:43.493 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:43.489+0000 7f18123b7f40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x555971424660 con 0x555971493360 2024-08-03T20:56:43.493 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:43.489+0000 7f18123b7f40 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x555971490290 con 0x555971494ea0 2024-08-03T20:56:43.494 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:43.490+0000 7f180c000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5559714aa330 0x55597148f610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:56:43.494 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:43.490+0000 7f180c000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5559714aa330 0x55597148f610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:50058/0 (socket says 172.21.15.32:50058) 2024-08-03T20:56:43.494 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:43.490+0000 7f180c000640 1 -- 172.21.15.32:0/117654999 learned_addr learned my addr 172.21.15.32:0/117654999 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:56:43.494 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:43.490+0000 7f180ac00640 1 --2- 172.21.15.32:0/117654999 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x555971494ea0 0x5559713dc7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:56:43.494 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:43.490+0000 7f180b600640 1 --2- 172.21.15.32:0/117654999 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x555971493360 0x5559714b12e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:56:43.494 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:43.490+0000 7f180c000640 1 -- 172.21.15.32:0/117654999 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x555971494ea0 msgr2=0x5559713dc7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:56:43.494 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:43.490+0000 7f180c000640 1 --2- 172.21.15.32:0/117654999 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x555971494ea0 0x5559713dc7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:43.494 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:43.490+0000 7f180c000640 1 -- 172.21.15.32:0/117654999 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x555971493360 msgr2=0x5559714b12e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:56:43.494 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:43.490+0000 7f180c000640 1 --2- 172.21.15.32:0/117654999 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x555971493360 0x5559714b12e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:43.494 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:43.490+0000 7f180c000640 1 -- 172.21.15.32:0/117654999 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5559714b1820 con 0x5559714aa330 2024-08-03T20:56:43.494 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:43.490+0000 7f180c000640 1 --2- 172.21.15.32:0/117654999 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5559714aa330 0x55597148f610 secure :-1 s=READY pgs=1417 cs=0 l=1 rev1=1 crypto rx=0x7f180400d410 tx=0x7f180400d8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:56:43.494 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:43.491+0000 7f1803e00640 1 -- 172.21.15.32:0/117654999 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1804018070 con 0x5559714aa330 2024-08-03T20:56:43.494 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:43.491+0000 7f1803e00640 1 -- 172.21.15.32:0/117654999 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f18040030d0 con 0x5559714aa330 2024-08-03T20:56:43.495 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:43.491+0000 7f18123b7f40 1 -- 172.21.15.32:0/117654999 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x5559714b0fe0 con 0x5559714aa330 2024-08-03T20:56:43.495 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:43.491+0000 7f1803e00640 1 -- 172.21.15.32:0/117654999 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1804013480 con 0x5559714aa330 2024-08-03T20:56:43.495 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:43.491+0000 7f18123b7f40 1 -- 172.21.15.32:0/117654999 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555971425450 con 0x5559714aa330 2024-08-03T20:56:43.495 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:43.493+0000 7f1803e00640 1 -- 172.21.15.32:0/117654999 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f180401b070 con 0x5559714aa330 2024-08-03T20:56:43.496 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:43.494+0000 7f1803e00640 1 --2- 172.21.15.32:0/117654999 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f17ec0759f0 0x7f17ec077f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:56:43.496 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:43.494+0000 7f1803e00640 1 -- 172.21.15.32:0/117654999 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 7058+0+0 (secure 0 0 0) 0x7f18040971c0 con 0x5559714aa330 2024-08-03T20:56:43.496 INFO:teuthology.orchestra.run.smithi032.stdout:Importing pool 2024-08-03T20:56:43.497 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:43.494+0000 7f18123b7f40 1 --2- 172.21.15.32:0/117654999 >> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] conn(0x55597148fb50 0x555971555940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:56:43.497 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:43.495+0000 7f18123b7f40 1 -- 172.21.15.32:0/117654999 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:1 12.10 12:09401a81:::601.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99) -- 0x555971555e80 con 0x55597148fb50 2024-08-03T20:56:43.497 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:43.495+0000 7f180b600640 1 --2- 172.21.15.32:0/117654999 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f17ec0759f0 0x7f17ec077f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:56:43.497 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:43.495+0000 7f180ac00640 1 --2- 172.21.15.32:0/117654999 >> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] conn(0x55597148fb50 0x555971555940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:56:43.497 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:43.495+0000 7f180b600640 1 --2- 172.21.15.32:0/117654999 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f17ec0759f0 0x7f17ec077f00 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7f17fc000c00 tx=0x7f17fc003850 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:56:43.497 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:43.495+0000 7f180ac00640 1 --2- 172.21.15.32:0/117654999 >> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] conn(0x55597148fb50 0x555971555940 crc :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:56:43.499 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:43.497+0000 7f180ac00640 1 -- 172.21.15.32:0/117654999 <== osd.0 v2:172.21.15.32:6816/3066156116 1 ==== osd_op_reply(1 601.00000000 [create] v99'412 uv386 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f17f8069040 con 0x55597148fb50 2024-08-03T20:56:43.499 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:43.497+0000 7f18123b7f40 1 -- 172.21.15.32:0/117654999 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:2 12.10 12:09401a81:::601.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e99) -- 0x555971556220 con 0x55597148fb50 2024-08-03T20:56:43.500 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:43.499+0000 7f180ac00640 1 -- 172.21.15.32:0/117654999 <== osd.0 v2:172.21.15.32:6816/3066156116 2 ==== osd_op_reply(2 601.00000000 [delete] v99'413 uv413 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f17f8069040 con 0x55597148fb50 2024-08-03T20:56:43.500 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:43.499+0000 7f18123b7f40 1 -- 172.21.15.32:0/117654999 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:3 12.10 12:09401a81:::601.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99) -- 0x5559715565c0 con 0x55597148fb50 2024-08-03T20:56:43.502 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::601.00000000:head# 2024-08-03T20:56:43.502 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:43.500+0000 7f180ac00640 1 -- 172.21.15.32:0/117654999 <== osd.0 v2:172.21.15.32:6816/3066156116 3 ==== osd_op_reply(3 601.00000000 [create] v99'414 uv414 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f17f8069040 con 0x55597148fb50 2024-08-03T20:56:43.502 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:43.500+0000 7f18123b7f40 1 -- 172.21.15.32:0/117654999 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:4 12.10 12:09401a81:::601.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99) -- 0x555971557970 con 0x55597148fb50 2024-08-03T20:56:43.503 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:43.502+0000 7f180ac00640 1 -- 172.21.15.32:0/117654999 <== osd.0 v2:172.21.15.32:6816/3066156116 4 ==== osd_op_reply(4 601.00000000 [setxattr (30)] v99'415 uv415 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f17f8069040 con 0x55597148fb50 2024-08-03T20:56:43.504 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:43.502+0000 7f18123b7f40 1 -- 172.21.15.32:0/117654999 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:5 12.10 12:09401a81:::601.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99) -- 0x555971557d10 con 0x55597148fb50 2024-08-03T20:56:43.505 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:43.504+0000 7f180ac00640 1 -- 172.21.15.32:0/117654999 <== osd.0 v2:172.21.15.32:6816/3066156116 5 ==== osd_op_reply(5 601.00000000 [setxattr (62)] v99'416 uv416 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f17f8069040 con 0x55597148fb50 2024-08-03T20:56:43.505 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:43.504+0000 7f18123b7f40 1 -- 172.21.15.32:0/117654999 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:6 12.10 12:09401a81:::601.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99) -- 0x555971556960 con 0x55597148fb50 2024-08-03T20:56:43.507 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:43.505+0000 7f180ac00640 1 -- 172.21.15.32:0/117654999 <== osd.0 v2:172.21.15.32:6816/3066156116 6 ==== osd_op_reply(6 601.00000000 [omap-set-header] v99'417 uv417 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f17f8069040 con 0x55597148fb50 2024-08-03T20:56:43.507 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:43.505+0000 7f18123b7f40 1 -- 172.21.15.32:0/117654999 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:7 12.10 12:0bd6d154:::602.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99) -- 0x555971556d50 con 0x55597148fb50 2024-08-03T20:56:43.508 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:43.507+0000 7f180ac00640 1 -- 172.21.15.32:0/117654999 <== osd.0 v2:172.21.15.32:6816/3066156116 7 ==== osd_op_reply(7 602.00000000 [create] v99'418 uv392 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f17f8069040 con 0x55597148fb50 2024-08-03T20:56:43.508 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:43.507+0000 7f18123b7f40 1 -- 172.21.15.32:0/117654999 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:8 12.10 12:0bd6d154:::602.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e99) -- 0x555971556d50 con 0x55597148fb50 2024-08-03T20:56:43.510 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:43.508+0000 7f180ac00640 1 -- 172.21.15.32:0/117654999 <== osd.0 v2:172.21.15.32:6816/3066156116 8 ==== osd_op_reply(8 602.00000000 [delete] v99'419 uv419 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f17f8069040 con 0x55597148fb50 2024-08-03T20:56:43.510 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:43.509+0000 7f18123b7f40 1 -- 172.21.15.32:0/117654999 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:9 12.10 12:0bd6d154:::602.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99) -- 0x555971556d00 con 0x55597148fb50 2024-08-03T20:56:43.511 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::602.00000000:head# 2024-08-03T20:56:43.512 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:43.510+0000 7f180ac00640 1 -- 172.21.15.32:0/117654999 <== osd.0 v2:172.21.15.32:6816/3066156116 9 ==== osd_op_reply(9 602.00000000 [create] v99'420 uv420 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f17f8069040 con 0x55597148fb50 2024-08-03T20:56:43.512 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:43.510+0000 7f18123b7f40 1 -- 172.21.15.32:0/117654999 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:10 12.10 12:0bd6d154:::602.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99) -- 0x555971556d00 con 0x55597148fb50 2024-08-03T20:56:43.513 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:43.511+0000 7f180ac00640 1 -- 172.21.15.32:0/117654999 <== osd.0 v2:172.21.15.32:6816/3066156116 10 ==== osd_op_reply(10 602.00000000 [setxattr (30)] v99'421 uv421 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f17f8069040 con 0x55597148fb50 2024-08-03T20:56:43.513 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:43.511+0000 7f18123b7f40 1 -- 172.21.15.32:0/117654999 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:11 12.10 12:0bd6d154:::602.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99) -- 0x555971556d00 con 0x55597148fb50 2024-08-03T20:56:43.514 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:43.513+0000 7f180ac00640 1 -- 172.21.15.32:0/117654999 <== osd.0 v2:172.21.15.32:6816/3066156116 11 ==== osd_op_reply(11 602.00000000 [setxattr (62)] v99'422 uv422 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f17f8069040 con 0x55597148fb50 2024-08-03T20:56:43.514 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:43.513+0000 7f18123b7f40 1 -- 172.21.15.32:0/117654999 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:12 12.10 12:0bd6d154:::602.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99) -- 0x555971556d00 con 0x55597148fb50 2024-08-03T20:56:43.516 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:43.514+0000 7f180ac00640 1 -- 172.21.15.32:0/117654999 <== osd.0 v2:172.21.15.32:6816/3066156116 12 ==== osd_op_reply(12 602.00000000 [omap-set-header] v99'423 uv423 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f17f8069040 con 0x55597148fb50 2024-08-03T20:56:43.516 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:43.515+0000 7f18123b7f40 1 -- 172.21.15.32:0/117654999 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:13 12.10 12:0cf6ad0f:::10000000000.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99) -- 0x555971556d00 con 0x55597148fb50 2024-08-03T20:56:43.518 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:43.516+0000 7f180ac00640 1 -- 172.21.15.32:0/117654999 <== osd.0 v2:172.21.15.32:6816/3066156116 13 ==== osd_op_reply(13 10000000000.00000000 [create] v99'424 uv399 ondisk = -17 ((17) File exists)) ==== 164+0+0 (crc 0 0 0) 0x7f17f8069040 con 0x55597148fb50 2024-08-03T20:56:43.518 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:43.516+0000 7f18123b7f40 1 -- 172.21.15.32:0/117654999 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:14 12.10 12:0cf6ad0f:::10000000000.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e99) -- 0x555971556d00 con 0x55597148fb50 2024-08-03T20:56:43.519 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:43.518+0000 7f180ac00640 1 -- 172.21.15.32:0/117654999 <== osd.0 v2:172.21.15.32:6816/3066156116 14 ==== osd_op_reply(14 10000000000.00000000 [delete] v99'425 uv425 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f17f8069040 con 0x55597148fb50 2024-08-03T20:56:43.519 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:43.518+0000 7f18123b7f40 1 -- 172.21.15.32:0/117654999 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:15 12.10 12:0cf6ad0f:::10000000000.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99) -- 0x555971556d00 con 0x55597148fb50 2024-08-03T20:56:43.520 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::10000000000.00000000:head# 2024-08-03T20:56:43.521 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:43.519+0000 7f180ac00640 1 -- 172.21.15.32:0/117654999 <== osd.0 v2:172.21.15.32:6816/3066156116 15 ==== osd_op_reply(15 10000000000.00000000 [create] v99'426 uv426 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f17f8069040 con 0x55597148fb50 2024-08-03T20:56:43.521 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:43.519+0000 7f18123b7f40 1 -- 172.21.15.32:0/117654999 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:16 12.10 12:0cf6ad0f:::10000000000.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99) -- 0x555971556d00 con 0x55597148fb50 2024-08-03T20:56:43.522 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:43.520+0000 7f180ac00640 1 -- 172.21.15.32:0/117654999 <== osd.0 v2:172.21.15.32:6816/3066156116 16 ==== osd_op_reply(16 10000000000.00000000 [setxattr (30)] v99'427 uv427 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f17f8069040 con 0x55597148fb50 2024-08-03T20:56:43.522 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:43.521+0000 7f18123b7f40 1 -- 172.21.15.32:0/117654999 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:17 12.10 12:0cf6ad0f:::10000000000.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99) -- 0x555971556d00 con 0x55597148fb50 2024-08-03T20:56:43.523 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:43.522+0000 7f180ac00640 1 -- 172.21.15.32:0/117654999 <== osd.0 v2:172.21.15.32:6816/3066156116 17 ==== osd_op_reply(17 10000000000.00000000 [setxattr (62)] v99'428 uv428 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f17f8069040 con 0x55597148fb50 2024-08-03T20:56:43.523 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:43.522+0000 7f18123b7f40 1 -- 172.21.15.32:0/117654999 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:18 12.10 12:0cf6ad0f:::10000000000.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99) -- 0x555971556d00 con 0x55597148fb50 2024-08-03T20:56:43.525 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:43.524+0000 7f180ac00640 1 -- 172.21.15.32:0/117654999 <== osd.0 v2:172.21.15.32:6816/3066156116 18 ==== osd_op_reply(18 10000000000.00000000 [omap-set-header] v99'429 uv429 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f17f8069040 con 0x55597148fb50 2024-08-03T20:56:43.525 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:43.524+0000 7f18123b7f40 1 -- 172.21.15.32:0/117654999 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:19 12.10 12:0cf6ad0f:::10000000000.00000000:head [omap-set-vals in=593b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99) -- 0x555971556d00 con 0x55597148fb50 2024-08-03T20:56:43.526 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:43.525+0000 7f180ac00640 1 -- 172.21.15.32:0/117654999 <== osd.0 v2:172.21.15.32:6816/3066156116 19 ==== osd_op_reply(19 10000000000.00000000 [omap-set-vals] v99'430 uv430 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f17f8069040 con 0x55597148fb50 2024-08-03T20:56:43.526 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:43.525+0000 7f18123b7f40 1 -- 172.21.15.32:0/117654999 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:20 12.10 12:0d82a743:::600.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99) -- 0x555971556d00 con 0x55597148fb50 2024-08-03T20:56:43.528 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:43.527+0000 7f180ac00640 1 -- 172.21.15.32:0/117654999 <== osd.0 v2:172.21.15.32:6816/3066156116 20 ==== osd_op_reply(20 600.00000000 [create] v99'431 uv405 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f17f8069040 con 0x55597148fb50 2024-08-03T20:56:43.528 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:43.527+0000 7f18123b7f40 1 -- 172.21.15.32:0/117654999 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:21 12.10 12:0d82a743:::600.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e99) -- 0x555971556d00 con 0x55597148fb50 2024-08-03T20:56:43.530 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:43.528+0000 7f180ac00640 1 -- 172.21.15.32:0/117654999 <== osd.0 v2:172.21.15.32:6816/3066156116 21 ==== osd_op_reply(21 600.00000000 [delete] v99'432 uv432 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f17f8069040 con 0x55597148fb50 2024-08-03T20:56:43.530 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:43.528+0000 7f18123b7f40 1 -- 172.21.15.32:0/117654999 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:22 12.10 12:0d82a743:::600.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99) -- 0x555971556d00 con 0x55597148fb50 2024-08-03T20:56:43.531 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::600.00000000:head# 2024-08-03T20:56:43.532 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:43.530+0000 7f180ac00640 1 -- 172.21.15.32:0/117654999 <== osd.0 v2:172.21.15.32:6816/3066156116 22 ==== osd_op_reply(22 600.00000000 [create] v99'433 uv433 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f17f8069040 con 0x55597148fb50 2024-08-03T20:56:43.532 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:43.530+0000 7f18123b7f40 1 -- 172.21.15.32:0/117654999 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:23 12.10 12:0d82a743:::600.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99) -- 0x555971556d00 con 0x55597148fb50 2024-08-03T20:56:43.532 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:43.531+0000 7f180ac00640 1 -- 172.21.15.32:0/117654999 <== osd.0 v2:172.21.15.32:6816/3066156116 23 ==== osd_op_reply(23 600.00000000 [setxattr (30)] v99'434 uv434 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f17f8069040 con 0x55597148fb50 2024-08-03T20:56:43.533 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:43.531+0000 7f18123b7f40 1 -- 172.21.15.32:0/117654999 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:24 12.10 12:0d82a743:::600.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99) -- 0x555971556d00 con 0x55597148fb50 2024-08-03T20:56:43.534 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:43.533+0000 7f180ac00640 1 -- 172.21.15.32:0/117654999 <== osd.0 v2:172.21.15.32:6816/3066156116 24 ==== osd_op_reply(24 600.00000000 [setxattr (62)] v99'435 uv435 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f17f8069040 con 0x55597148fb50 2024-08-03T20:56:43.534 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:43.533+0000 7f18123b7f40 1 -- 172.21.15.32:0/117654999 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:25 12.10 12:0d82a743:::600.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99) -- 0x555971556d00 con 0x55597148fb50 2024-08-03T20:56:43.536 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:43.534+0000 7f180ac00640 1 -- 172.21.15.32:0/117654999 <== osd.0 v2:172.21.15.32:6816/3066156116 25 ==== osd_op_reply(25 600.00000000 [omap-set-header] v99'436 uv436 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f17f8069040 con 0x55597148fb50 2024-08-03T20:56:43.536 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:43.534+0000 7f18123b7f40 1 -- 172.21.15.32:0/117654999 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:26 12.10 12:0d89b25e:::603.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99) -- 0x555971556d00 con 0x55597148fb50 2024-08-03T20:56:43.537 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:43.536+0000 7f180ac00640 1 -- 172.21.15.32:0/117654999 <== osd.0 v2:172.21.15.32:6816/3066156116 26 ==== osd_op_reply(26 603.00000000 [create] v99'437 uv411 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f17f8069040 con 0x55597148fb50 2024-08-03T20:56:43.537 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:43.536+0000 7f18123b7f40 1 -- 172.21.15.32:0/117654999 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:27 12.10 12:0d89b25e:::603.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e99) -- 0x555971556d00 con 0x55597148fb50 2024-08-03T20:56:43.539 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:43.537+0000 7f180ac00640 1 -- 172.21.15.32:0/117654999 <== osd.0 v2:172.21.15.32:6816/3066156116 27 ==== osd_op_reply(27 603.00000000 [delete] v99'438 uv438 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f17f8069040 con 0x55597148fb50 2024-08-03T20:56:43.539 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:43.537+0000 7f18123b7f40 1 -- 172.21.15.32:0/117654999 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:28 12.10 12:0d89b25e:::603.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99) -- 0x555971556d00 con 0x55597148fb50 2024-08-03T20:56:43.540 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::603.00000000:head# 2024-08-03T20:56:43.540 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:43.539+0000 7f180ac00640 1 -- 172.21.15.32:0/117654999 <== osd.0 v2:172.21.15.32:6816/3066156116 28 ==== osd_op_reply(28 603.00000000 [create] v99'439 uv439 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f17f8069040 con 0x55597148fb50 2024-08-03T20:56:43.540 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:43.539+0000 7f18123b7f40 1 -- 172.21.15.32:0/117654999 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:29 12.10 12:0d89b25e:::603.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99) -- 0x555971556d00 con 0x55597148fb50 2024-08-03T20:56:43.541 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:43.540+0000 7f180ac00640 1 -- 172.21.15.32:0/117654999 <== osd.0 v2:172.21.15.32:6816/3066156116 29 ==== osd_op_reply(29 603.00000000 [setxattr (30)] v99'440 uv440 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f17f8069040 con 0x55597148fb50 2024-08-03T20:56:43.541 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:43.540+0000 7f18123b7f40 1 -- 172.21.15.32:0/117654999 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:30 12.10 12:0d89b25e:::603.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99) -- 0x555971556d00 con 0x55597148fb50 2024-08-03T20:56:43.543 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:43.542+0000 7f180ac00640 1 -- 172.21.15.32:0/117654999 <== osd.0 v2:172.21.15.32:6816/3066156116 30 ==== osd_op_reply(30 603.00000000 [setxattr (62)] v99'441 uv441 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f17f8069040 con 0x55597148fb50 2024-08-03T20:56:43.543 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:43.542+0000 7f18123b7f40 1 -- 172.21.15.32:0/117654999 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:31 12.10 12:0d89b25e:::603.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99) -- 0x555971556d00 con 0x55597148fb50 2024-08-03T20:56:43.545 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:43.543+0000 7f180ac00640 1 -- 172.21.15.32:0/117654999 <== osd.0 v2:172.21.15.32:6816/3066156116 31 ==== osd_op_reply(31 603.00000000 [omap-set-header] v99'442 uv442 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f17f8069040 con 0x55597148fb50 2024-08-03T20:56:43.545 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:43.543+0000 7f18123b7f40 1 --2- 172.21.15.32:0/117654999 >> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] conn(0x555971557210 0x55597155a0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:56:43.545 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:43.543+0000 7f18123b7f40 1 -- 172.21.15.32:0/117654999 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:32 12.14 12:287034c4:::1.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99) -- 0x55597155a600 con 0x555971557210 2024-08-03T20:56:43.545 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:43.544+0000 7f180b600640 1 --2- 172.21.15.32:0/117654999 >> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] conn(0x555971557210 0x55597155a0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:56:43.546 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:43.544+0000 7f180b600640 1 --2- 172.21.15.32:0/117654999 >> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] conn(0x555971557210 0x55597155a0c0 crc :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:56:43.548 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:43.546+0000 7f180b600640 1 -- 172.21.15.32:0/117654999 <== osd.5 v2:172.21.15.118:6810/2724891770 1 ==== osd_op_reply(32 1.00000000.inode [create] v99'308 uv291 ondisk = -17 ((17) File exists)) ==== 160+0+0 (crc 0 0 0) 0x7f17fc01b040 con 0x555971557210 2024-08-03T20:56:43.548 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:43.546+0000 7f18123b7f40 1 -- 172.21.15.32:0/117654999 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:33 12.14 12:287034c4:::1.00000000.inode:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e99) -- 0x55597155a9a0 con 0x555971557210 2024-08-03T20:56:43.549 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:43.548+0000 7f180b600640 1 -- 172.21.15.32:0/117654999 <== osd.5 v2:172.21.15.118:6810/2724891770 2 ==== osd_op_reply(33 1.00000000.inode [delete] v99'309 uv309 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f17fc01b040 con 0x555971557210 2024-08-03T20:56:43.550 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:43.548+0000 7f18123b7f40 1 -- 172.21.15.32:0/117654999 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:34 12.14 12:287034c4:::1.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99) -- 0x55597155ad40 con 0x555971557210 2024-08-03T20:56:43.551 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::1.00000000.inode:head# 2024-08-03T20:56:43.552 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:43.550+0000 7f180b600640 1 -- 172.21.15.32:0/117654999 <== osd.5 v2:172.21.15.118:6810/2724891770 3 ==== osd_op_reply(34 1.00000000.inode [create] v99'310 uv310 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f17fc01b040 con 0x555971557210 2024-08-03T20:56:43.552 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:43.550+0000 7f18123b7f40 1 -- 172.21.15.32:0/117654999 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:35 12.14 12:287034c4:::1.00000000.inode:head [write 0~582 in=582b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99) -- 0x55597155b0e0 con 0x555971557210 2024-08-03T20:56:43.553 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:43.552+0000 7f180b600640 1 -- 172.21.15.32:0/117654999 <== osd.5 v2:172.21.15.118:6810/2724891770 4 ==== osd_op_reply(35 1.00000000.inode [write 0~582] v99'311 uv311 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f17fc01b040 con 0x555971557210 2024-08-03T20:56:43.554 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:43.552+0000 7f18123b7f40 1 -- 172.21.15.32:0/117654999 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:36 12.14 12:287034c4:::1.00000000.inode:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99) -- 0x55597155b480 con 0x555971557210 2024-08-03T20:56:43.555 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:43.554+0000 7f180b600640 1 -- 172.21.15.32:0/117654999 <== osd.5 v2:172.21.15.118:6810/2724891770 5 ==== osd_op_reply(36 1.00000000.inode [omap-set-header] v99'312 uv312 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f17fc01b040 con 0x555971557210 2024-08-03T20:56:43.555 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:43.554+0000 7f18123b7f40 1 -- 172.21.15.32:0/117654999 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:37 12.14 12:292cf221:::200.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99) -- 0x55597155b820 con 0x555971557210 2024-08-03T20:56:43.557 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:43.556+0000 7f180b600640 1 -- 172.21.15.32:0/117654999 <== osd.5 v2:172.21.15.118:6810/2724891770 6 ==== osd_op_reply(37 200.00000000 [create] v99'313 uv296 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f17fc01b040 con 0x555971557210 2024-08-03T20:56:43.557 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:43.556+0000 7f18123b7f40 1 -- 172.21.15.32:0/117654999 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:38 12.14 12:292cf221:::200.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e99) -- 0x55597155bbc0 con 0x555971557210 2024-08-03T20:56:43.559 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:43.558+0000 7f180b600640 1 -- 172.21.15.32:0/117654999 <== osd.5 v2:172.21.15.118:6810/2724891770 7 ==== osd_op_reply(38 200.00000000 [delete] v99'314 uv314 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f17fc01b040 con 0x555971557210 2024-08-03T20:56:43.559 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:43.558+0000 7f18123b7f40 1 -- 172.21.15.32:0/117654999 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:39 12.14 12:292cf221:::200.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99) -- 0x55597155bf60 con 0x555971557210 2024-08-03T20:56:43.561 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::200.00000000:head# 2024-08-03T20:56:43.562 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:43.560+0000 7f180b600640 1 -- 172.21.15.32:0/117654999 <== osd.5 v2:172.21.15.118:6810/2724891770 8 ==== osd_op_reply(39 200.00000000 [create] v99'315 uv315 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f17fc01b040 con 0x555971557210 2024-08-03T20:56:43.562 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:43.560+0000 7f18123b7f40 1 -- 172.21.15.32:0/117654999 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:40 12.14 12:292cf221:::200.00000000:head [write 0~90 in=90b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99) -- 0x55597155bf60 con 0x555971557210 2024-08-03T20:56:43.563 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:43.562+0000 7f180b600640 1 -- 172.21.15.32:0/117654999 <== osd.5 v2:172.21.15.118:6810/2724891770 9 ==== osd_op_reply(40 200.00000000 [write 0~90] v99'316 uv316 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f17fc01b040 con 0x555971557210 2024-08-03T20:56:43.563 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:43.562+0000 7f18123b7f40 1 -- 172.21.15.32:0/117654999 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:41 12.14 12:292cf221:::200.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99) -- 0x55597155bf60 con 0x555971557210 2024-08-03T20:56:43.565 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:43.564+0000 7f180b600640 1 -- 172.21.15.32:0/117654999 <== osd.5 v2:172.21.15.118:6810/2724891770 10 ==== osd_op_reply(41 200.00000000 [omap-set-header] v99'317 uv317 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f17fc01b040 con 0x555971557210 2024-08-03T20:56:43.565 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:43.564+0000 7f18123b7f40 1 -- 172.21.15.32:0/117654999 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:42 12.14 12:2e2fa760:::200.00000001:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99) -- 0x55597155bf60 con 0x555971557210 2024-08-03T20:56:43.567 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:43.566+0000 7f180b600640 1 -- 172.21.15.32:0/117654999 <== osd.5 v2:172.21.15.118:6810/2724891770 11 ==== osd_op_reply(42 200.00000001 [create] v99'318 uv301 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f17fc01b040 con 0x555971557210 2024-08-03T20:56:43.567 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:43.566+0000 7f18123b7f40 1 -- 172.21.15.32:0/117654999 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:43 12.14 12:2e2fa760:::200.00000001:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e99) -- 0x55597155bf60 con 0x555971557210 2024-08-03T20:56:43.569 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:43.568+0000 7f180b600640 1 -- 172.21.15.32:0/117654999 <== osd.5 v2:172.21.15.118:6810/2724891770 12 ==== osd_op_reply(43 200.00000001 [delete] v99'319 uv319 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f17fc01b040 con 0x555971557210 2024-08-03T20:56:43.569 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:43.568+0000 7f18123b7f40 1 -- 172.21.15.32:0/117654999 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:44 12.14 12:2e2fa760:::200.00000001:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99) -- 0x55597155bf60 con 0x555971557210 2024-08-03T20:56:43.570 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::200.00000001:head# 2024-08-03T20:56:43.571 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:43.569+0000 7f180b600640 1 -- 172.21.15.32:0/117654999 <== osd.5 v2:172.21.15.118:6810/2724891770 13 ==== osd_op_reply(44 200.00000001 [create] v99'320 uv320 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f17fc01b040 con 0x555971557210 2024-08-03T20:56:43.571 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:43.569+0000 7f18123b7f40 1 -- 172.21.15.32:0/117654999 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:45 12.14 12:2e2fa760:::200.00000001:head [write 0~18118 in=18118b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99) -- 0x555971560650 con 0x555971557210 2024-08-03T20:56:43.573 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:43.571+0000 7f180b600640 1 -- 172.21.15.32:0/117654999 <== osd.5 v2:172.21.15.118:6810/2724891770 14 ==== osd_op_reply(45 200.00000001 [write 0~18118] v99'321 uv321 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f17fc01b040 con 0x555971557210 2024-08-03T20:56:43.573 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:43.571+0000 7f18123b7f40 1 -- 172.21.15.32:0/117654999 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:46 12.14 12:2e2fa760:::200.00000001:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99) -- 0x555971560650 con 0x555971557210 2024-08-03T20:56:43.574 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:43.573+0000 7f180b600640 1 -- 172.21.15.32:0/117654999 <== osd.5 v2:172.21.15.118:6810/2724891770 15 ==== osd_op_reply(46 200.00000001 [omap-set-header] v99'322 uv322 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f17fc01b040 con 0x555971557210 2024-08-03T20:56:43.574 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:43.573+0000 7f18123b7f40 1 -- 172.21.15.32:0/117654999 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:47 12.14 12:2f55791f:::606.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99) -- 0x55597155bf60 con 0x555971557210 2024-08-03T20:56:43.576 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:43.575+0000 7f180b600640 1 -- 172.21.15.32:0/117654999 <== osd.5 v2:172.21.15.118:6810/2724891770 16 ==== osd_op_reply(47 606.00000000 [create] v99'323 uv307 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f17fc01b040 con 0x555971557210 2024-08-03T20:56:43.576 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:43.575+0000 7f18123b7f40 1 -- 172.21.15.32:0/117654999 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:48 12.14 12:2f55791f:::606.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e99) -- 0x55597155bf60 con 0x555971557210 2024-08-03T20:56:43.578 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:43.576+0000 7f180b600640 1 -- 172.21.15.32:0/117654999 <== osd.5 v2:172.21.15.118:6810/2724891770 17 ==== osd_op_reply(48 606.00000000 [delete] v99'324 uv324 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f17fc01b040 con 0x555971557210 2024-08-03T20:56:43.578 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:43.576+0000 7f18123b7f40 1 -- 172.21.15.32:0/117654999 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:49 12.14 12:2f55791f:::606.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99) -- 0x55597155bf60 con 0x555971557210 2024-08-03T20:56:43.579 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::606.00000000:head# 2024-08-03T20:56:43.579 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:43.578+0000 7f180b600640 1 -- 172.21.15.32:0/117654999 <== osd.5 v2:172.21.15.118:6810/2724891770 18 ==== osd_op_reply(49 606.00000000 [create] v99'325 uv325 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f17fc01b040 con 0x555971557210 2024-08-03T20:56:43.580 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:43.578+0000 7f18123b7f40 1 -- 172.21.15.32:0/117654999 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:50 12.14 12:2f55791f:::606.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99) -- 0x55597155cf70 con 0x555971557210 2024-08-03T20:56:43.581 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:43.580+0000 7f180b600640 1 -- 172.21.15.32:0/117654999 <== osd.5 v2:172.21.15.118:6810/2724891770 19 ==== osd_op_reply(50 606.00000000 [setxattr (30)] v99'326 uv326 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f17fc01b040 con 0x555971557210 2024-08-03T20:56:43.581 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:43.580+0000 7f18123b7f40 1 -- 172.21.15.32:0/117654999 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:51 12.14 12:2f55791f:::606.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99) -- 0x55597155cf70 con 0x555971557210 2024-08-03T20:56:43.583 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:43.581+0000 7f180b600640 1 -- 172.21.15.32:0/117654999 <== osd.5 v2:172.21.15.118:6810/2724891770 20 ==== osd_op_reply(51 606.00000000 [setxattr (62)] v99'327 uv327 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f17fc01b040 con 0x555971557210 2024-08-03T20:56:43.583 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:43.581+0000 7f18123b7f40 1 -- 172.21.15.32:0/117654999 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:52 12.14 12:2f55791f:::606.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99) -- 0x55597155bf60 con 0x555971557210 2024-08-03T20:56:43.584 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:43.583+0000 7f180b600640 1 -- 172.21.15.32:0/117654999 <== osd.5 v2:172.21.15.118:6810/2724891770 21 ==== osd_op_reply(52 606.00000000 [omap-set-header] v99'328 uv328 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f17fc01b040 con 0x555971557210 2024-08-03T20:56:43.584 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:43.583+0000 7f18123b7f40 1 -- 172.21.15.32:0/117654999 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:53 12.1c 12:3ed09add:::607.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99) -- 0x55597155bf60 con 0x55597148fb50 2024-08-03T20:56:43.586 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:43.584+0000 7f180ac00640 1 -- 172.21.15.32:0/117654999 <== osd.0 v2:172.21.15.32:6816/3066156116 32 ==== osd_op_reply(53 607.00000000 [create] v99'81 uv80 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f17f8069040 con 0x55597148fb50 2024-08-03T20:56:43.586 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:43.584+0000 7f18123b7f40 1 -- 172.21.15.32:0/117654999 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:54 12.1c 12:3ed09add:::607.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e99) -- 0x55597155bf60 con 0x55597148fb50 2024-08-03T20:56:43.587 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:43.586+0000 7f180ac00640 1 -- 172.21.15.32:0/117654999 <== osd.0 v2:172.21.15.32:6816/3066156116 33 ==== osd_op_reply(54 607.00000000 [delete] v99'82 uv82 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f17f8069040 con 0x55597148fb50 2024-08-03T20:56:43.587 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:43.586+0000 7f18123b7f40 1 -- 172.21.15.32:0/117654999 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:55 12.1c 12:3ed09add:::607.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99) -- 0x55597155bf60 con 0x55597148fb50 2024-08-03T20:56:43.588 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:43.587+0000 7f180ac00640 1 -- 172.21.15.32:0/117654999 <== osd.0 v2:172.21.15.32:6816/3066156116 34 ==== osd_op_reply(55 607.00000000 [create] v99'83 uv83 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f17f8069040 con 0x55597148fb50 2024-08-03T20:56:43.588 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::607.00000000:head# 2024-08-03T20:56:43.589 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:43.587+0000 7f18123b7f40 1 -- 172.21.15.32:0/117654999 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:56 12.1c 12:3ed09add:::607.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99) -- 0x55597155cf70 con 0x55597148fb50 2024-08-03T20:56:43.590 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:43.588+0000 7f180ac00640 1 -- 172.21.15.32:0/117654999 <== osd.0 v2:172.21.15.32:6816/3066156116 35 ==== osd_op_reply(56 607.00000000 [setxattr (30)] v99'84 uv84 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f17f8069040 con 0x55597148fb50 2024-08-03T20:56:43.590 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:43.588+0000 7f18123b7f40 1 -- 172.21.15.32:0/117654999 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:57 12.1c 12:3ed09add:::607.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99) -- 0x55597155cf70 con 0x55597148fb50 2024-08-03T20:56:43.591 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:43.590+0000 7f180ac00640 1 -- 172.21.15.32:0/117654999 <== osd.0 v2:172.21.15.32:6816/3066156116 36 ==== osd_op_reply(57 607.00000000 [setxattr (62)] v99'85 uv85 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f17f8069040 con 0x55597148fb50 2024-08-03T20:56:43.591 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:43.590+0000 7f18123b7f40 1 -- 172.21.15.32:0/117654999 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:58 12.1c 12:3ed09add:::607.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99) -- 0x55597155bf60 con 0x55597148fb50 2024-08-03T20:56:43.593 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:43.591+0000 7f180ac00640 1 -- 172.21.15.32:0/117654999 <== osd.0 v2:172.21.15.32:6816/3066156116 37 ==== osd_op_reply(58 607.00000000 [omap-set-header] v99'86 uv86 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f17f8069040 con 0x55597148fb50 2024-08-03T20:56:43.593 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:43.591+0000 7f18123b7f40 1 --2- 172.21.15.32:0/117654999 >> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] conn(0x55597155c470 0x55597155e890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:56:43.593 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:43.591+0000 7f18123b7f40 1 -- 172.21.15.32:0/117654999 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:59 12.6 12:654134d2:::mds0_openfiles.0:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99) -- 0x55597155edd0 con 0x55597155c470 2024-08-03T20:56:43.593 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:43.591+0000 7f180ac00640 1 --2- 172.21.15.32:0/117654999 >> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] conn(0x55597155c470 0x55597155e890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:56:43.593 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:43.592+0000 7f180ac00640 1 --2- 172.21.15.32:0/117654999 >> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] conn(0x55597155c470 0x55597155e890 crc :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:56:43.595 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:43.593+0000 7f180ac00640 1 -- 172.21.15.32:0/117654999 <== osd.1 v2:172.21.15.32:6824/1551241759 1 ==== osd_op_reply(59 mds0_openfiles.0 [create] v99'70 uv69 ondisk = -17 ((17) File exists)) ==== 160+0+0 (crc 0 0 0) 0x7f17f8069040 con 0x55597155c470 2024-08-03T20:56:43.595 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:43.593+0000 7f18123b7f40 1 -- 172.21.15.32:0/117654999 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:60 12.6 12:654134d2:::mds0_openfiles.0:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e99) -- 0x55597155f1a0 con 0x55597155c470 2024-08-03T20:56:43.596 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:43.594+0000 7f180ac00640 1 -- 172.21.15.32:0/117654999 <== osd.1 v2:172.21.15.32:6824/1551241759 2 ==== osd_op_reply(60 mds0_openfiles.0 [delete] v99'71 uv71 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f17f8069040 con 0x55597155c470 2024-08-03T20:56:43.596 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:43.594+0000 7f18123b7f40 1 -- 172.21.15.32:0/117654999 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:61 12.6 12:654134d2:::mds0_openfiles.0:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99) -- 0x55597155f1a0 con 0x55597155c470 2024-08-03T20:56:43.597 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:43.596+0000 7f180ac00640 1 -- 172.21.15.32:0/117654999 <== osd.1 v2:172.21.15.32:6824/1551241759 3 ==== osd_op_reply(61 mds0_openfiles.0 [create] v99'72 uv72 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f17f8069040 con 0x55597155c470 2024-08-03T20:56:43.597 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds0_openfiles.0:head# 2024-08-03T20:56:43.597 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:43.596+0000 7f18123b7f40 1 -- 172.21.15.32:0/117654999 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:62 12.6 12:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99) -- 0x55597155f1a0 con 0x55597155c470 2024-08-03T20:56:43.599 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:43.597+0000 7f180ac00640 1 -- 172.21.15.32:0/117654999 <== osd.1 v2:172.21.15.32:6824/1551241759 4 ==== osd_op_reply(62 mds0_openfiles.0 [omap-set-header] v99'73 uv73 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f17f8069040 con 0x55597155c470 2024-08-03T20:56:43.599 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:43.597+0000 7f18123b7f40 1 -- 172.21.15.32:0/117654999 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:63 12.e 12:717a0223:::608.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99) -- 0x55597155f110 con 0x55597148fb50 2024-08-03T20:56:43.600 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:43.599+0000 7f180ac00640 1 -- 172.21.15.32:0/117654999 <== osd.0 v2:172.21.15.32:6816/3066156116 38 ==== osd_op_reply(63 608.00000000 [create] v99'81 uv80 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f17f8069040 con 0x55597148fb50 2024-08-03T20:56:43.600 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:43.599+0000 7f18123b7f40 1 -- 172.21.15.32:0/117654999 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:64 12.e 12:717a0223:::608.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e99) -- 0x55597155f110 con 0x55597148fb50 2024-08-03T20:56:43.602 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:43.600+0000 7f180ac00640 1 -- 172.21.15.32:0/117654999 <== osd.0 v2:172.21.15.32:6816/3066156116 39 ==== osd_op_reply(64 608.00000000 [delete] v99'82 uv82 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f17f8069040 con 0x55597148fb50 2024-08-03T20:56:43.602 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:43.600+0000 7f18123b7f40 1 -- 172.21.15.32:0/117654999 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:65 12.e 12:717a0223:::608.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99) -- 0x55597155f050 con 0x55597148fb50 2024-08-03T20:56:43.603 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::608.00000000:head# 2024-08-03T20:56:43.603 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:43.602+0000 7f180ac00640 1 -- 172.21.15.32:0/117654999 <== osd.0 v2:172.21.15.32:6816/3066156116 40 ==== osd_op_reply(65 608.00000000 [create] v99'83 uv83 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f17f8069040 con 0x55597148fb50 2024-08-03T20:56:43.603 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:43.602+0000 7f18123b7f40 1 -- 172.21.15.32:0/117654999 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:66 12.e 12:717a0223:::608.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99) -- 0x55597155f050 con 0x55597148fb50 2024-08-03T20:56:43.604 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:43.603+0000 7f180ac00640 1 -- 172.21.15.32:0/117654999 <== osd.0 v2:172.21.15.32:6816/3066156116 41 ==== osd_op_reply(66 608.00000000 [setxattr (30)] v99'84 uv84 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f17f8069040 con 0x55597148fb50 2024-08-03T20:56:43.604 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:43.603+0000 7f18123b7f40 1 -- 172.21.15.32:0/117654999 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:67 12.e 12:717a0223:::608.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99) -- 0x55597155f050 con 0x55597148fb50 2024-08-03T20:56:43.606 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:43.604+0000 7f180ac00640 1 -- 172.21.15.32:0/117654999 <== osd.0 v2:172.21.15.32:6816/3066156116 42 ==== osd_op_reply(67 608.00000000 [setxattr (62)] v99'85 uv85 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f17f8069040 con 0x55597148fb50 2024-08-03T20:56:43.606 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:43.604+0000 7f18123b7f40 1 -- 172.21.15.32:0/117654999 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:68 12.e 12:717a0223:::608.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99) -- 0x55597155f050 con 0x55597148fb50 2024-08-03T20:56:43.611 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:43.606+0000 7f180ac00640 1 -- 172.21.15.32:0/117654999 <== osd.0 v2:172.21.15.32:6816/3066156116 43 ==== osd_op_reply(68 608.00000000 [omap-set-header] v99'86 uv86 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f17f8069040 con 0x55597148fb50 2024-08-03T20:56:43.611 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:43.606+0000 7f18123b7f40 1 --2- 172.21.15.32:0/117654999 >> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] conn(0x55597155f970 0x55597155fd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:56:43.611 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:43.606+0000 7f18123b7f40 1 -- 172.21.15.32:0/117654999 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:69 12.15 12:a93a17c2:::604.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99) -- 0x5559715624b0 con 0x55597155f970 2024-08-03T20:56:43.611 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:43.606+0000 7f180c000640 1 --2- 172.21.15.32:0/117654999 >> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] conn(0x55597155f970 0x55597155fd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:56:43.611 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:43.607+0000 7f180c000640 1 --2- 172.21.15.32:0/117654999 >> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] conn(0x55597155f970 0x55597155fd80 crc :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:56:43.611 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:43.609+0000 7f180c000640 1 -- 172.21.15.32:0/117654999 <== osd.6 v2:172.21.15.118:6825/801947386 1 ==== osd_op_reply(69 604.00000000 [create] v99'149 uv143 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f180401a030 con 0x55597155f970 2024-08-03T20:56:43.611 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:43.609+0000 7f18123b7f40 1 -- 172.21.15.32:0/117654999 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:70 12.15 12:a93a17c2:::604.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e99) -- 0x555971562850 con 0x55597155f970 2024-08-03T20:56:43.612 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:43.610+0000 7f180c000640 1 -- 172.21.15.32:0/117654999 <== osd.6 v2:172.21.15.118:6825/801947386 2 ==== osd_op_reply(70 604.00000000 [delete] v99'150 uv150 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f180401a030 con 0x55597155f970 2024-08-03T20:56:43.612 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:43.610+0000 7f18123b7f40 1 -- 172.21.15.32:0/117654999 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:71 12.15 12:a93a17c2:::604.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99) -- 0x555971562bf0 con 0x55597155f970 2024-08-03T20:56:43.613 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::604.00000000:head# 2024-08-03T20:56:43.613 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:43.612+0000 7f180c000640 1 -- 172.21.15.32:0/117654999 <== osd.6 v2:172.21.15.118:6825/801947386 3 ==== osd_op_reply(71 604.00000000 [create] v99'151 uv151 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f180401a030 con 0x55597155f970 2024-08-03T20:56:43.613 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:43.612+0000 7f18123b7f40 1 -- 172.21.15.32:0/117654999 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:72 12.15 12:a93a17c2:::604.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99) -- 0x55597153ea20 con 0x55597155f970 2024-08-03T20:56:43.615 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:43.614+0000 7f180c000640 1 -- 172.21.15.32:0/117654999 <== osd.6 v2:172.21.15.118:6825/801947386 4 ==== osd_op_reply(72 604.00000000 [setxattr (30)] v99'152 uv152 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f180401a030 con 0x55597155f970 2024-08-03T20:56:43.615 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:43.614+0000 7f18123b7f40 1 -- 172.21.15.32:0/117654999 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:73 12.15 12:a93a17c2:::604.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99) -- 0x555971564560 con 0x55597155f970 2024-08-03T20:56:43.616 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:43.615+0000 7f180c000640 1 -- 172.21.15.32:0/117654999 <== osd.6 v2:172.21.15.118:6825/801947386 5 ==== osd_op_reply(73 604.00000000 [setxattr (62)] v99'153 uv153 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f180401a030 con 0x55597155f970 2024-08-03T20:56:43.616 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:43.615+0000 7f18123b7f40 1 -- 172.21.15.32:0/117654999 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:74 12.15 12:a93a17c2:::604.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99) -- 0x555971563490 con 0x55597155f970 2024-08-03T20:56:43.618 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:43.616+0000 7f180c000640 1 -- 172.21.15.32:0/117654999 <== osd.6 v2:172.21.15.118:6825/801947386 6 ==== osd_op_reply(74 604.00000000 [omap-set-header] v99'154 uv154 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f180401a030 con 0x55597155f970 2024-08-03T20:56:43.618 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:43.616+0000 7f18123b7f40 1 -- 172.21.15.32:0/117654999 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:75 12.15 12:aa448500:::500.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99) -- 0x555971563880 con 0x55597155f970 2024-08-03T20:56:43.619 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:43.618+0000 7f180c000640 1 -- 172.21.15.32:0/117654999 <== osd.6 v2:172.21.15.118:6825/801947386 7 ==== osd_op_reply(75 500.00000000 [create] v99'155 uv148 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f180401a030 con 0x55597155f970 2024-08-03T20:56:43.619 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:43.618+0000 7f18123b7f40 1 -- 172.21.15.32:0/117654999 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:76 12.15 12:aa448500:::500.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e99) -- 0x555971563c70 con 0x55597155f970 2024-08-03T20:56:43.621 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:43.619+0000 7f180c000640 1 -- 172.21.15.32:0/117654999 <== osd.6 v2:172.21.15.118:6825/801947386 8 ==== osd_op_reply(76 500.00000000 [delete] v99'156 uv156 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f180401a030 con 0x55597155f970 2024-08-03T20:56:43.621 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:43.620+0000 7f18123b7f40 1 -- 172.21.15.32:0/117654999 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:77 12.15 12:aa448500:::500.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99) -- 0x555971564060 con 0x55597155f970 2024-08-03T20:56:43.622 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::500.00000000:head# 2024-08-03T20:56:43.623 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:43.621+0000 7f180c000640 1 -- 172.21.15.32:0/117654999 <== osd.6 v2:172.21.15.118:6825/801947386 9 ==== osd_op_reply(77 500.00000000 [create] v99'157 uv157 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f180401a030 con 0x55597155f970 2024-08-03T20:56:43.623 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:43.621+0000 7f18123b7f40 1 -- 172.21.15.32:0/117654999 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:78 12.15 12:aa448500:::500.00000000:head [write 0~90 in=90b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99) -- 0x555971564060 con 0x55597155f970 2024-08-03T20:56:43.624 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:43.622+0000 7f180c000640 1 -- 172.21.15.32:0/117654999 <== osd.6 v2:172.21.15.118:6825/801947386 10 ==== osd_op_reply(78 500.00000000 [write 0~90] v99'158 uv158 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f180401a030 con 0x55597155f970 2024-08-03T20:56:43.624 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:43.622+0000 7f18123b7f40 1 -- 172.21.15.32:0/117654999 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:79 12.15 12:aa448500:::500.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99) -- 0x555971564010 con 0x55597155f970 2024-08-03T20:56:43.626 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:43.624+0000 7f180c000640 1 -- 172.21.15.32:0/117654999 <== osd.6 v2:172.21.15.118:6825/801947386 11 ==== osd_op_reply(79 500.00000000 [omap-set-header] v99'159 uv159 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f180401a030 con 0x55597155f970 2024-08-03T20:56:43.626 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:43.624+0000 7f18123b7f40 1 --2- 172.21.15.32:0/117654999 >> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] conn(0x555971564ea0 0x5559715672c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:56:43.626 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:43.624+0000 7f18123b7f40 1 -- 172.21.15.32:0/117654999 --> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] -- osd_op(unknown.0.0:80 12.d 12:b50e409b:::mds_snaptable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99) -- 0x555971567800 con 0x555971564ea0 2024-08-03T20:56:43.626 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:43.625+0000 7f180b600640 1 --2- 172.21.15.32:0/117654999 >> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] conn(0x555971564ea0 0x5559715672c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:56:43.626 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:43.625+0000 7f180b600640 1 --2- 172.21.15.32:0/117654999 >> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] conn(0x555971564ea0 0x5559715672c0 crc :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:56:43.628 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:43.626+0000 7f180b600640 1 -- 172.21.15.32:0/117654999 <== osd.7 v2:172.21.15.118:6818/653000048 1 ==== osd_op_reply(80 mds_snaptable [create] v99'66 uv65 ondisk = -17 ((17) File exists)) ==== 157+0+0 (crc 0 0 0) 0x7f17fc01b040 con 0x555971564ea0 2024-08-03T20:56:43.628 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:43.627+0000 7f18123b7f40 1 -- 172.21.15.32:0/117654999 --> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] -- osd_op(unknown.0.0:81 12.d 12:b50e409b:::mds_snaptable:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e99) -- 0x555971567840 con 0x555971564ea0 2024-08-03T20:56:43.629 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:43.628+0000 7f180b600640 1 -- 172.21.15.32:0/117654999 <== osd.7 v2:172.21.15.118:6818/653000048 2 ==== osd_op_reply(81 mds_snaptable [delete] v99'67 uv67 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f17fc01b040 con 0x555971564ea0 2024-08-03T20:56:43.629 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:43.628+0000 7f18123b7f40 1 -- 172.21.15.32:0/117654999 --> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] -- osd_op(unknown.0.0:82 12.d 12:b50e409b:::mds_snaptable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99) -- 0x555971567c60 con 0x555971564ea0 2024-08-03T20:56:43.630 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds_snaptable:head# 2024-08-03T20:56:43.631 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:43.629+0000 7f180b600640 1 -- 172.21.15.32:0/117654999 <== osd.7 v2:172.21.15.118:6818/653000048 3 ==== osd_op_reply(82 mds_snaptable [create] v99'68 uv68 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f17fc01b040 con 0x555971564ea0 2024-08-03T20:56:43.631 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:43.629+0000 7f18123b7f40 1 -- 172.21.15.32:0/117654999 --> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] -- osd_op(unknown.0.0:83 12.d 12:b50e409b:::mds_snaptable:head [write 0~70 in=70b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99) -- 0x555971567c60 con 0x555971564ea0 2024-08-03T20:56:43.632 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:43.630+0000 7f180b600640 1 -- 172.21.15.32:0/117654999 <== osd.7 v2:172.21.15.118:6818/653000048 4 ==== osd_op_reply(83 mds_snaptable [write 0~70] v99'69 uv69 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f17fc01b040 con 0x555971564ea0 2024-08-03T20:56:43.632 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:43.631+0000 7f18123b7f40 1 -- 172.21.15.32:0/117654999 --> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] -- osd_op(unknown.0.0:84 12.d 12:b50e409b:::mds_snaptable:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99) -- 0x555971567c60 con 0x555971564ea0 2024-08-03T20:56:43.633 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:43.632+0000 7f180b600640 1 -- 172.21.15.32:0/117654999 <== osd.7 v2:172.21.15.118:6818/653000048 5 ==== osd_op_reply(84 mds_snaptable [omap-set-header] v99'70 uv70 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f17fc01b040 con 0x555971564ea0 2024-08-03T20:56:43.633 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:43.632+0000 7f18123b7f40 1 -- 172.21.15.32:0/117654999 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:85 12.1d 12:b871830b:::605.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99) -- 0x555971567c60 con 0x55597148fb50 2024-08-03T20:56:43.635 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:43.633+0000 7f180ac00640 1 -- 172.21.15.32:0/117654999 <== osd.0 v2:172.21.15.32:6816/3066156116 44 ==== osd_op_reply(85 605.00000000 [create] v99'81 uv80 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f17f8069040 con 0x55597148fb50 2024-08-03T20:56:43.635 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:43.633+0000 7f18123b7f40 1 -- 172.21.15.32:0/117654999 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:86 12.1d 12:b871830b:::605.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e99) -- 0x555971567c60 con 0x55597148fb50 2024-08-03T20:56:43.636 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:43.635+0000 7f180ac00640 1 -- 172.21.15.32:0/117654999 <== osd.0 v2:172.21.15.32:6816/3066156116 45 ==== osd_op_reply(86 605.00000000 [delete] v99'82 uv82 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f17f8069040 con 0x55597148fb50 2024-08-03T20:56:43.636 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:43.635+0000 7f18123b7f40 1 -- 172.21.15.32:0/117654999 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:87 12.1d 12:b871830b:::605.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99) -- 0x555971567c60 con 0x55597148fb50 2024-08-03T20:56:43.638 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::605.00000000:head# 2024-08-03T20:56:43.638 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:43.636+0000 7f180ac00640 1 -- 172.21.15.32:0/117654999 <== osd.0 v2:172.21.15.32:6816/3066156116 46 ==== osd_op_reply(87 605.00000000 [create] v99'83 uv83 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f17f8069040 con 0x55597148fb50 2024-08-03T20:56:43.638 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:43.636+0000 7f18123b7f40 1 -- 172.21.15.32:0/117654999 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:88 12.1d 12:b871830b:::605.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99) -- 0x555971568c70 con 0x55597148fb50 2024-08-03T20:56:43.639 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:43.638+0000 7f180ac00640 1 -- 172.21.15.32:0/117654999 <== osd.0 v2:172.21.15.32:6816/3066156116 47 ==== osd_op_reply(88 605.00000000 [setxattr (30)] v99'84 uv84 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f17f8069040 con 0x55597148fb50 2024-08-03T20:56:43.639 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:43.638+0000 7f18123b7f40 1 -- 172.21.15.32:0/117654999 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:89 12.1d 12:b871830b:::605.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99) -- 0x555971568c70 con 0x55597148fb50 2024-08-03T20:56:43.640 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:43.639+0000 7f180ac00640 1 -- 172.21.15.32:0/117654999 <== osd.0 v2:172.21.15.32:6816/3066156116 48 ==== osd_op_reply(89 605.00000000 [setxattr (62)] v99'85 uv85 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f17f8069040 con 0x55597148fb50 2024-08-03T20:56:43.641 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:43.639+0000 7f18123b7f40 1 -- 172.21.15.32:0/117654999 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:90 12.1d 12:b871830b:::605.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99) -- 0x555971567c60 con 0x55597148fb50 2024-08-03T20:56:43.642 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:43.641+0000 7f180ac00640 1 -- 172.21.15.32:0/117654999 <== osd.0 v2:172.21.15.32:6816/3066156116 49 ==== osd_op_reply(90 605.00000000 [omap-set-header] v99'86 uv86 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f17f8069040 con 0x55597148fb50 2024-08-03T20:56:43.642 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:43.641+0000 7f18123b7f40 1 -- 172.21.15.32:0/117654999 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:91 12.13 12:c91d4a1d:::mds0_inotable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99) -- 0x555971567c60 con 0x55597155c470 2024-08-03T20:56:43.643 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:43.642+0000 7f180ac00640 1 -- 172.21.15.32:0/117654999 <== osd.1 v2:172.21.15.32:6824/1551241759 5 ==== osd_op_reply(91 mds0_inotable [create] v99'132 uv126 ondisk = -17 ((17) File exists)) ==== 157+0+0 (crc 0 0 0) 0x7f17f8069040 con 0x55597155c470 2024-08-03T20:56:43.643 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:43.642+0000 7f18123b7f40 1 -- 172.21.15.32:0/117654999 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:92 12.13 12:c91d4a1d:::mds0_inotable:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e99) -- 0x555971567c60 con 0x55597155c470 2024-08-03T20:56:43.645 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:43.643+0000 7f180ac00640 1 -- 172.21.15.32:0/117654999 <== osd.1 v2:172.21.15.32:6824/1551241759 6 ==== osd_op_reply(92 mds0_inotable [delete] v99'133 uv133 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f17f8069040 con 0x55597155c470 2024-08-03T20:56:43.645 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:43.643+0000 7f18123b7f40 1 -- 172.21.15.32:0/117654999 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:93 12.13 12:c91d4a1d:::mds0_inotable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99) -- 0x555971567c60 con 0x55597155c470 2024-08-03T20:56:43.646 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds0_inotable:head# 2024-08-03T20:56:43.646 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:43.644+0000 7f180ac00640 1 -- 172.21.15.32:0/117654999 <== osd.1 v2:172.21.15.32:6824/1551241759 7 ==== osd_op_reply(93 mds0_inotable [create] v99'134 uv134 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f17f8069040 con 0x55597155c470 2024-08-03T20:56:43.646 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:43.644+0000 7f18123b7f40 1 -- 172.21.15.32:0/117654999 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:94 12.13 12:c91d4a1d:::mds0_inotable:head [write 0~34 in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99) -- 0x555971563080 con 0x55597155c470 2024-08-03T20:56:43.647 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:43.646+0000 7f180ac00640 1 -- 172.21.15.32:0/117654999 <== osd.1 v2:172.21.15.32:6824/1551241759 8 ==== osd_op_reply(94 mds0_inotable [write 0~34] v99'135 uv135 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f17f8069040 con 0x55597155c470 2024-08-03T20:56:43.647 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:43.646+0000 7f18123b7f40 1 -- 172.21.15.32:0/117654999 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:95 12.13 12:c91d4a1d:::mds0_inotable:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99) -- 0x555971563080 con 0x55597155c470 2024-08-03T20:56:43.648 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:43.647+0000 7f180ac00640 1 -- 172.21.15.32:0/117654999 <== osd.1 v2:172.21.15.32:6824/1551241759 9 ==== osd_op_reply(95 mds0_inotable [omap-set-header] v99'136 uv136 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f17f8069040 con 0x55597155c470 2024-08-03T20:56:43.649 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:43.647+0000 7f18123b7f40 1 -- 172.21.15.32:0/117654999 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:96 12.13 12:cd5a64a3:::100.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99) -- 0x555971563080 con 0x55597155c470 2024-08-03T20:56:43.650 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:43.648+0000 7f180ac00640 1 -- 172.21.15.32:0/117654999 <== osd.1 v2:172.21.15.32:6824/1551241759 10 ==== osd_op_reply(96 100.00000000 [create] v99'137 uv131 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f17f8069040 con 0x55597155c470 2024-08-03T20:56:43.650 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:43.648+0000 7f18123b7f40 1 -- 172.21.15.32:0/117654999 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:97 12.13 12:cd5a64a3:::100.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e99) -- 0x555971563080 con 0x55597155c470 2024-08-03T20:56:43.651 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:43.650+0000 7f180ac00640 1 -- 172.21.15.32:0/117654999 <== osd.1 v2:172.21.15.32:6824/1551241759 11 ==== osd_op_reply(97 100.00000000 [delete] v99'138 uv138 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f17f8069040 con 0x55597155c470 2024-08-03T20:56:43.651 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:43.650+0000 7f18123b7f40 1 -- 172.21.15.32:0/117654999 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:98 12.13 12:cd5a64a3:::100.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99) -- 0x555971563080 con 0x55597155c470 2024-08-03T20:56:43.652 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::100.00000000:head# 2024-08-03T20:56:43.652 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:43.651+0000 7f180ac00640 1 -- 172.21.15.32:0/117654999 <== osd.1 v2:172.21.15.32:6824/1551241759 12 ==== osd_op_reply(98 100.00000000 [create] v99'139 uv139 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f17f8069040 con 0x55597155c470 2024-08-03T20:56:43.652 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:43.651+0000 7f18123b7f40 1 -- 172.21.15.32:0/117654999 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:99 12.13 12:cd5a64a3:::100.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99) -- 0x555971563080 con 0x55597155c470 2024-08-03T20:56:43.654 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:43.652+0000 7f180ac00640 1 -- 172.21.15.32:0/117654999 <== osd.1 v2:172.21.15.32:6824/1551241759 13 ==== osd_op_reply(99 100.00000000 [omap-set-header] v99'140 uv140 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f17f8069040 con 0x55597155c470 2024-08-03T20:56:43.654 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:43.652+0000 7f18123b7f40 1 -- 172.21.15.32:0/117654999 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:100 12.13 12:cd5a64a3:::100.00000000:head [omap-set-vals in=5194b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99) -- 0x55597156ab20 con 0x55597155c470 2024-08-03T20:56:43.655 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:43.654+0000 7f180ac00640 1 -- 172.21.15.32:0/117654999 <== osd.1 v2:172.21.15.32:6824/1551241759 14 ==== osd_op_reply(100 100.00000000 [omap-set-vals] v99'141 uv141 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f17f8069040 con 0x55597155c470 2024-08-03T20:56:43.655 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:43.654+0000 7f18123b7f40 1 --2- 172.21.15.32:0/117654999 >> [v2:172.21.15.32:6800/9088303,v1:172.21.15.32:6801/9088303] conn(0x55597156ac90 0x55597156b190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:56:43.655 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:43.654+0000 7f18123b7f40 1 -- 172.21.15.32:0/117654999 --> [v2:172.21.15.32:6800/9088303,v1:172.21.15.32:6801/9088303] -- osd_op(unknown.0.0:101 12.b 12:d0630e4c:::mds0_sessionmap:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99) -- 0x55597156a180 con 0x55597156ac90 2024-08-03T20:56:43.655 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:43.654+0000 7f180ac00640 1 --2- 172.21.15.32:0/117654999 >> [v2:172.21.15.32:6800/9088303,v1:172.21.15.32:6801/9088303] conn(0x55597156ac90 0x55597156b190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:56:43.656 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:43.654+0000 7f180ac00640 1 --2- 172.21.15.32:0/117654999 >> [v2:172.21.15.32:6800/9088303,v1:172.21.15.32:6801/9088303] conn(0x55597156ac90 0x55597156b190 crc :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:56:43.657 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:43.656+0000 7f180ac00640 1 -- 172.21.15.32:0/117654999 <== osd.3 v2:172.21.15.32:6800/9088303 1 ==== osd_op_reply(101 mds0_sessionmap [create] v99'61 uv60 ondisk = -17 ((17) File exists)) ==== 159+0+0 (crc 0 0 0) 0x7f17f8069040 con 0x55597156ac90 2024-08-03T20:56:43.657 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:43.656+0000 7f18123b7f40 1 -- 172.21.15.32:0/117654999 --> [v2:172.21.15.32:6800/9088303,v1:172.21.15.32:6801/9088303] -- osd_op(unknown.0.0:102 12.b 12:d0630e4c:::mds0_sessionmap:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e99) -- 0x55597156a180 con 0x55597156ac90 2024-08-03T20:56:43.658 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:43.657+0000 7f180ac00640 1 -- 172.21.15.32:0/117654999 <== osd.3 v2:172.21.15.32:6800/9088303 2 ==== osd_op_reply(102 mds0_sessionmap [delete] v99'62 uv62 ondisk = 0) ==== 159+0+0 (crc 0 0 0) 0x7f17f8069040 con 0x55597156ac90 2024-08-03T20:56:43.659 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:43.657+0000 7f18123b7f40 1 -- 172.21.15.32:0/117654999 --> [v2:172.21.15.32:6800/9088303,v1:172.21.15.32:6801/9088303] -- osd_op(unknown.0.0:103 12.b 12:d0630e4c:::mds0_sessionmap:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99) -- 0x55597156a180 con 0x55597156ac90 2024-08-03T20:56:43.660 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:43.659+0000 7f180ac00640 1 -- 172.21.15.32:0/117654999 <== osd.3 v2:172.21.15.32:6800/9088303 3 ==== osd_op_reply(103 mds0_sessionmap [create] v99'63 uv63 ondisk = 0) ==== 159+0+0 (crc 0 0 0) 0x7f17f8069040 con 0x55597156ac90 2024-08-03T20:56:43.660 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds0_sessionmap:head# 2024-08-03T20:56:43.661 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:43.659+0000 7f18123b7f40 1 -- 172.21.15.32:0/117654999 --> [v2:172.21.15.32:6800/9088303,v1:172.21.15.32:6801/9088303] -- osd_op(unknown.0.0:104 12.b 12:d0630e4c:::mds0_sessionmap:head [omap-set-header in=14b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99) -- 0x55597156a180 con 0x55597156ac90 2024-08-03T20:56:43.662 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:43.660+0000 7f180ac00640 1 -- 172.21.15.32:0/117654999 <== osd.3 v2:172.21.15.32:6800/9088303 4 ==== osd_op_reply(104 mds0_sessionmap [omap-set-header] v99'64 uv64 ondisk = 0) ==== 159+0+0 (crc 0 0 0) 0x7f17f8069040 con 0x55597156ac90 2024-08-03T20:56:43.662 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:43.660+0000 7f18123b7f40 1 -- 172.21.15.32:0/117654999 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:105 12.7 12:e0b41d06:::609.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99) -- 0x55597156a180 con 0x55597155c470 2024-08-03T20:56:43.663 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:43.662+0000 7f180ac00640 1 -- 172.21.15.32:0/117654999 <== osd.1 v2:172.21.15.32:6824/1551241759 15 ==== osd_op_reply(105 609.00000000 [create] v99'81 uv80 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f17f8069040 con 0x55597155c470 2024-08-03T20:56:43.663 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:43.662+0000 7f18123b7f40 1 -- 172.21.15.32:0/117654999 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:106 12.7 12:e0b41d06:::609.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e99) -- 0x55597156a180 con 0x55597155c470 2024-08-03T20:56:43.664 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:43.663+0000 7f180ac00640 1 -- 172.21.15.32:0/117654999 <== osd.1 v2:172.21.15.32:6824/1551241759 16 ==== osd_op_reply(106 609.00000000 [delete] v99'82 uv82 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f17f8069040 con 0x55597155c470 2024-08-03T20:56:43.664 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:43.663+0000 7f18123b7f40 1 -- 172.21.15.32:0/117654999 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:107 12.7 12:e0b41d06:::609.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99) -- 0x55597156a180 con 0x55597155c470 2024-08-03T20:56:43.666 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:43.664+0000 7f180ac00640 1 -- 172.21.15.32:0/117654999 <== osd.1 v2:172.21.15.32:6824/1551241759 17 ==== osd_op_reply(107 609.00000000 [create] v99'83 uv83 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f17f8069040 con 0x55597155c470 2024-08-03T20:56:43.666 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::609.00000000:head# 2024-08-03T20:56:43.666 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:43.664+0000 7f18123b7f40 1 -- 172.21.15.32:0/117654999 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:108 12.7 12:e0b41d06:::609.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99) -- 0x55597156a180 con 0x55597155c470 2024-08-03T20:56:43.667 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:43.666+0000 7f180ac00640 1 -- 172.21.15.32:0/117654999 <== osd.1 v2:172.21.15.32:6824/1551241759 18 ==== osd_op_reply(108 609.00000000 [setxattr (30)] v99'84 uv84 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f17f8069040 con 0x55597155c470 2024-08-03T20:56:43.667 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:43.666+0000 7f18123b7f40 1 -- 172.21.15.32:0/117654999 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:109 12.7 12:e0b41d06:::609.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99) -- 0x55597156a180 con 0x55597155c470 2024-08-03T20:56:43.668 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:43.667+0000 7f180ac00640 1 -- 172.21.15.32:0/117654999 <== osd.1 v2:172.21.15.32:6824/1551241759 19 ==== osd_op_reply(109 609.00000000 [setxattr (62)] v99'85 uv85 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f17f8069040 con 0x55597155c470 2024-08-03T20:56:43.668 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:43.667+0000 7f18123b7f40 1 -- 172.21.15.32:0/117654999 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:110 12.7 12:e0b41d06:::609.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99) -- 0x55597156a180 con 0x55597155c470 2024-08-03T20:56:43.670 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:43.669+0000 7f180ac00640 1 -- 172.21.15.32:0/117654999 <== osd.1 v2:172.21.15.32:6824/1551241759 20 ==== osd_op_reply(110 609.00000000 [omap-set-header] v99'86 uv86 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f17f8069040 con 0x55597155c470 2024-08-03T20:56:43.670 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:43.669+0000 7f18123b7f40 1 -- 172.21.15.32:0/117654999 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:111 12.f 12:f1f69726:::400.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99) -- 0x55597156a180 con 0x55597155f970 2024-08-03T20:56:43.672 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:43.670+0000 7f180c000640 1 -- 172.21.15.32:0/117654999 <== osd.6 v2:172.21.15.118:6825/801947386 12 ==== osd_op_reply(111 400.00000000 [create] v99'69 uv68 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f180401a030 con 0x55597155f970 2024-08-03T20:56:43.672 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:43.671+0000 7f18123b7f40 1 -- 172.21.15.32:0/117654999 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:112 12.f 12:f1f69726:::400.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e99) -- 0x55597156a180 con 0x55597155f970 2024-08-03T20:56:43.674 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:43.672+0000 7f180c000640 1 -- 172.21.15.32:0/117654999 <== osd.6 v2:172.21.15.118:6825/801947386 13 ==== osd_op_reply(112 400.00000000 [delete] v99'70 uv70 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f180401a030 con 0x55597155f970 2024-08-03T20:56:43.674 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:43.672+0000 7f18123b7f40 1 -- 172.21.15.32:0/117654999 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:113 12.f 12:f1f69726:::400.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99) -- 0x55597156a180 con 0x55597155f970 2024-08-03T20:56:43.675 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::400.00000000:head# 2024-08-03T20:56:43.675 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:43.674+0000 7f180c000640 1 -- 172.21.15.32:0/117654999 <== osd.6 v2:172.21.15.118:6825/801947386 14 ==== osd_op_reply(113 400.00000000 [create] v99'71 uv71 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f180401a030 con 0x55597155f970 2024-08-03T20:56:43.676 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:43.674+0000 7f18123b7f40 1 -- 172.21.15.32:0/117654999 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:114 12.f 12:f1f69726:::400.00000000:head [write 0~22 in=22b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99) -- 0x55597156a180 con 0x55597155f970 2024-08-03T20:56:43.677 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:43.675+0000 7f180c000640 1 -- 172.21.15.32:0/117654999 <== osd.6 v2:172.21.15.118:6825/801947386 15 ==== osd_op_reply(114 400.00000000 [write 0~22] v99'72 uv72 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f180401a030 con 0x55597155f970 2024-08-03T20:56:43.677 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:43.675+0000 7f18123b7f40 1 -- 172.21.15.32:0/117654999 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:115 12.f 12:f1f69726:::400.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99) -- 0x55597156a180 con 0x55597155f970 2024-08-03T20:56:43.678 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:43.677+0000 7f180c000640 1 -- 172.21.15.32:0/117654999 <== osd.6 v2:172.21.15.118:6825/801947386 16 ==== osd_op_reply(115 400.00000000 [omap-set-header] v99'73 uv73 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f180401a030 con 0x55597155f970 2024-08-03T20:56:43.678 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:43.677+0000 7f18123b7f40 1 -- 172.21.15.32:0/117654999 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:116 12.1f 12:fe07bba1:::100.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99) -- 0x55597156a180 con 0x55597155f970 2024-08-03T20:56:43.680 INFO:teuthology.orchestra.run.smithi032.stdout:Write #-9223372036854775808:00000000:::100.00000000.inode:head# 2024-08-03T20:56:43.680 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:43.679+0000 7f180c000640 1 -- 172.21.15.32:0/117654999 <== osd.6 v2:172.21.15.118:6825/801947386 17 ==== osd_op_reply(116 100.00000000.inode [create] v99'170 uv170 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7f180401a030 con 0x55597155f970 2024-08-03T20:56:43.680 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:43.679+0000 7f18123b7f40 1 -- 172.21.15.32:0/117654999 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:117 12.1f 12:fe07bba1:::100.00000000.inode:head [write 0~582 in=582b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99) -- 0x55597156a180 con 0x55597155f970 2024-08-03T20:56:43.682 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:43.681+0000 7f180c000640 1 -- 172.21.15.32:0/117654999 <== osd.6 v2:172.21.15.118:6825/801947386 18 ==== osd_op_reply(117 100.00000000.inode [write 0~582] v99'171 uv171 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7f180401a030 con 0x55597155f970 2024-08-03T20:56:43.682 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:43.681+0000 7f18123b7f40 1 -- 172.21.15.32:0/117654999 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:118 12.1f 12:fe07bba1:::100.00000000.inode:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99) -- 0x55597156a180 con 0x55597155f970 2024-08-03T20:56:43.684 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:43.682+0000 7f180c000640 1 -- 172.21.15.32:0/117654999 <== osd.6 v2:172.21.15.118:6825/801947386 19 ==== osd_op_reply(118 100.00000000.inode [omap-set-header] v99'172 uv172 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7f180401a030 con 0x55597155f970 2024-08-03T20:56:43.684 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:43.682+0000 7f18123b7f40 1 -- 172.21.15.32:0/117654999 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:119 12.1f 12:ff5b34d6:::1.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99) -- 0x55597156a180 con 0x55597155f970 2024-08-03T20:56:43.685 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:43.684+0000 7f180c000640 1 -- 172.21.15.32:0/117654999 <== osd.6 v2:172.21.15.118:6825/801947386 20 ==== osd_op_reply(119 1.00000000 [create] v99'173 uv168 ondisk = -17 ((17) File exists)) ==== 154+0+0 (crc 0 0 0) 0x7f180401a030 con 0x55597155f970 2024-08-03T20:56:43.685 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:43.684+0000 7f18123b7f40 1 -- 172.21.15.32:0/117654999 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:120 12.1f 12:ff5b34d6:::1.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e99) -- 0x55597156a180 con 0x55597155f970 2024-08-03T20:56:43.687 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:43.686+0000 7f180c000640 1 -- 172.21.15.32:0/117654999 <== osd.6 v2:172.21.15.118:6825/801947386 21 ==== osd_op_reply(120 1.00000000 [delete] v99'174 uv174 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7f180401a030 con 0x55597155f970 2024-08-03T20:56:43.687 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:43.686+0000 7f18123b7f40 1 -- 172.21.15.32:0/117654999 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:121 12.1f 12:ff5b34d6:::1.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99) -- 0x55597156a180 con 0x55597155f970 2024-08-03T20:56:43.688 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:43.687+0000 7f180c000640 1 -- 172.21.15.32:0/117654999 <== osd.6 v2:172.21.15.118:6825/801947386 22 ==== osd_op_reply(121 1.00000000 [create] v99'175 uv175 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7f180401a030 con 0x55597155f970 2024-08-03T20:56:43.689 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::1.00000000:head# 2024-08-03T20:56:43.689 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:43.687+0000 7f18123b7f40 1 -- 172.21.15.32:0/117654999 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:122 12.1f 12:ff5b34d6:::1.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99) -- 0x55597156a180 con 0x55597155f970 2024-08-03T20:56:43.690 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:43.689+0000 7f180c000640 1 -- 172.21.15.32:0/117654999 <== osd.6 v2:172.21.15.118:6825/801947386 23 ==== osd_op_reply(122 1.00000000 [setxattr (30)] v99'176 uv176 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7f180401a030 con 0x55597155f970 2024-08-03T20:56:43.690 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:43.689+0000 7f18123b7f40 1 -- 172.21.15.32:0/117654999 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:123 12.1f 12:ff5b34d6:::1.00000000:head [setxattr parent (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99) -- 0x55597156a180 con 0x55597155f970 2024-08-03T20:56:43.692 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:43.690+0000 7f180c000640 1 -- 172.21.15.32:0/117654999 <== osd.6 v2:172.21.15.118:6825/801947386 24 ==== osd_op_reply(123 1.00000000 [setxattr (30)] v99'177 uv177 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7f180401a030 con 0x55597155f970 2024-08-03T20:56:43.692 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:43.690+0000 7f18123b7f40 1 -- 172.21.15.32:0/117654999 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:124 12.1f 12:ff5b34d6:::1.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99) -- 0x55597156a180 con 0x55597155f970 2024-08-03T20:56:43.693 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:43.692+0000 7f180c000640 1 -- 172.21.15.32:0/117654999 <== osd.6 v2:172.21.15.118:6825/801947386 25 ==== osd_op_reply(124 1.00000000 [omap-set-header] v99'178 uv178 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7f180401a030 con 0x55597155f970 2024-08-03T20:56:43.693 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:43.692+0000 7f18123b7f40 1 -- 172.21.15.32:0/117654999 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:125 12.1f 12:ff5b34d6:::1.00000000:head [omap-set-vals in=584b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99) -- 0x55597156bb80 con 0x55597155f970 2024-08-03T20:56:43.696 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:43.693+0000 7f180c000640 1 -- 172.21.15.32:0/117654999 <== osd.6 v2:172.21.15.118:6825/801947386 26 ==== osd_op_reply(125 1.00000000 [omap-set-vals] v99'179 uv179 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7f180401a030 con 0x55597155f970 2024-08-03T20:56:43.696 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:43.693+0000 7f18123b7f40 1 -- 172.21.15.32:0/117654999 >> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] conn(0x55597148fb50 msgr2=0x555971555940 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:56:43.696 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:43.693+0000 7f18123b7f40 1 --2- 172.21.15.32:0/117654999 >> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] conn(0x55597148fb50 0x555971555940 crc :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:43.696 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:43.693+0000 7f18123b7f40 1 -- 172.21.15.32:0/117654999 >> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] conn(0x55597155c470 msgr2=0x55597155e890 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:56:43.696 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:43.693+0000 7f18123b7f40 1 --2- 172.21.15.32:0/117654999 >> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] conn(0x55597155c470 0x55597155e890 crc :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:43.696 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:43.693+0000 7f18123b7f40 1 -- 172.21.15.32:0/117654999 >> [v2:172.21.15.32:6800/9088303,v1:172.21.15.32:6801/9088303] conn(0x55597156ac90 msgr2=0x55597156b190 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:56:43.696 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:43.693+0000 7f18123b7f40 1 --2- 172.21.15.32:0/117654999 >> [v2:172.21.15.32:6800/9088303,v1:172.21.15.32:6801/9088303] conn(0x55597156ac90 0x55597156b190 crc :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:43.696 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:43.693+0000 7f18123b7f40 1 -- 172.21.15.32:0/117654999 >> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] conn(0x555971557210 msgr2=0x55597155a0c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:56:43.697 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:43.693+0000 7f18123b7f40 1 --2- 172.21.15.32:0/117654999 >> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] conn(0x555971557210 0x55597155a0c0 crc :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:43.697 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:43.693+0000 7f180c000640 1 -- 172.21.15.32:0/117654999 reap_dead start 2024-08-03T20:56:43.697 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:43.693+0000 7f18123b7f40 1 -- 172.21.15.32:0/117654999 >> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] conn(0x55597155f970 msgr2=0x55597155fd80 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:56:43.697 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:43.693+0000 7f18123b7f40 1 --2- 172.21.15.32:0/117654999 >> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] conn(0x55597155f970 0x55597155fd80 crc :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:43.697 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:43.693+0000 7f180c000640 1 -- 172.21.15.32:0/117654999 reap_dead start 2024-08-03T20:56:43.697 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:43.693+0000 7f18123b7f40 1 -- 172.21.15.32:0/117654999 >> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] conn(0x555971564ea0 msgr2=0x5559715672c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:56:43.697 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:43.693+0000 7f18123b7f40 1 --2- 172.21.15.32:0/117654999 >> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] conn(0x555971564ea0 0x5559715672c0 crc :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:43.697 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:43.694+0000 7f18123b7f40 1 -- 172.21.15.32:0/117654999 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f17ec0759f0 msgr2=0x7f17ec077f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:56:43.697 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:43.694+0000 7f18123b7f40 1 --2- 172.21.15.32:0/117654999 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f17ec0759f0 0x7f17ec077f00 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7f17fc000c00 tx=0x7f17fc003850 comp rx=0 tx=0).stop 2024-08-03T20:56:43.697 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:43.694+0000 7f18123b7f40 1 -- 172.21.15.32:0/117654999 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5559714aa330 msgr2=0x55597148f610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:56:43.697 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:43.694+0000 7f18123b7f40 1 --2- 172.21.15.32:0/117654999 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5559714aa330 0x55597148f610 secure :-1 s=READY pgs=1417 cs=0 l=1 rev1=1 crypto rx=0x7f180400d410 tx=0x7f180400d8d0 comp rx=0 tx=0).stop 2024-08-03T20:56:43.697 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:43.694+0000 7f18123b7f40 1 -- 172.21.15.32:0/117654999 shutdown_connections 2024-08-03T20:56:43.697 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:43.694+0000 7f18123b7f40 1 --2- 172.21.15.32:0/117654999 >> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] conn(0x555971564ea0 0x5559715672c0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:43.698 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:43.694+0000 7f18123b7f40 1 --2- 172.21.15.32:0/117654999 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f17ec0759f0 0x7f17ec077f00 unknown :-1 s=CLOSED pgs=579 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:43.698 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:43.694+0000 7f18123b7f40 1 --2- 172.21.15.32:0/117654999 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5559714aa330 0x55597148f610 unknown :-1 s=CLOSED pgs=1417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:43.698 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:43.694+0000 7f18123b7f40 1 -- 172.21.15.32:0/117654999 >> 172.21.15.32:0/117654999 conn(0x5559712fcac0 msgr2=0x5559714b2280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:56:43.698 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:43.694+0000 7f18123b7f40 1 -- 172.21.15.32:0/117654999 shutdown_connections 2024-08-03T20:56:43.698 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:43.694+0000 7f18123b7f40 1 -- 172.21.15.32:0/117654999 wait complete. 2024-08-03T20:56:43.710 DEBUG:teuthology.orchestra.run.smithi032:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata rm 1.00000000 2024-08-03T20:56:43.894 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:43.892+0000 7f4b68578f40 1 Processor -- start 2024-08-03T20:56:43.894 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:43.893+0000 7f4b68578f40 1 -- start start 2024-08-03T20:56:43.897 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:43.894+0000 7f4b68578f40 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55a4a1324b40 0x55a4a13b5580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:56:43.897 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:43.894+0000 7f4b68578f40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55a4a1397be0 0x55a4a143e910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:56:43.897 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:43.894+0000 7f4b68578f40 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a4a13b5ac0 0x55a4a14479f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:56:43.897 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:43.894+0000 7f4b68578f40 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55a4a122de30 con 0x55a4a13b5ac0 2024-08-03T20:56:43.897 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:43.894+0000 7f4b68578f40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x55a4a128bbf0 con 0x55a4a1397be0 2024-08-03T20:56:43.897 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:43.894+0000 7f4b68578f40 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x55a4a13b6f10 con 0x55a4a1324b40 2024-08-03T20:56:43.898 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:43.894+0000 7f4b62000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a4a13b5ac0 0x55a4a14479f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:56:43.898 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:43.894+0000 7f4b61600640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55a4a1324b40 0x55a4a13b5580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:56:43.898 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:43.894+0000 7f4b61600640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55a4a1324b40 0x55a4a13b5580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3301/0 says I am v2:172.21.15.32:39464/0 (socket says 172.21.15.32:39464) 2024-08-03T20:56:43.898 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:43.894+0000 7f4b61600640 1 -- 172.21.15.32:0/3269613913 learned_addr learned my addr 172.21.15.32:0/3269613913 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:56:43.898 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:43.894+0000 7f4b62000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a4a13b5ac0 0x55a4a14479f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:50072/0 (socket says 172.21.15.32:50072) 2024-08-03T20:56:43.898 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:43.894+0000 7f4b60c00640 1 --2- 172.21.15.32:0/3269613913 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55a4a1397be0 0x55a4a143e910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:56:43.898 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:43.895+0000 7f4b61600640 1 -- 172.21.15.32:0/3269613913 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55a4a1397be0 msgr2=0x55a4a143e910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:56:43.898 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:43.895+0000 7f4b61600640 1 --2- 172.21.15.32:0/3269613913 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55a4a1397be0 0x55a4a143e910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:43.898 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:43.895+0000 7f4b61600640 1 -- 172.21.15.32:0/3269613913 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a4a13b5ac0 msgr2=0x55a4a14479f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:56:43.898 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:43.895+0000 7f4b61600640 1 --2- 172.21.15.32:0/3269613913 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a4a13b5ac0 0x55a4a14479f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:43.898 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:43.895+0000 7f4b61600640 1 -- 172.21.15.32:0/3269613913 --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a4a1395d20 con 0x55a4a1324b40 2024-08-03T20:56:43.899 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:43.895+0000 7f4b61600640 1 --2- 172.21.15.32:0/3269613913 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55a4a1324b40 0x55a4a13b5580 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f4b54001d50 tx=0x7f4b5400b5b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1e949317de382df5 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:56:43.899 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:43.895+0000 7f4b5b400640 1 -- 172.21.15.32:0/3269613913 <== mon.2 v2:172.21.15.118:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b5400bd80 con 0x55a4a1324b40 2024-08-03T20:56:43.899 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:43.895+0000 7f4b5b400640 1 -- 172.21.15.32:0/3269613913 <== mon.2 v2:172.21.15.118:3301/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f4b5400f400 con 0x55a4a1324b40 2024-08-03T20:56:43.899 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:43.895+0000 7f4b5b400640 1 -- 172.21.15.32:0/3269613913 <== mon.2 v2:172.21.15.118:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b5400f850 con 0x55a4a1324b40 2024-08-03T20:56:43.900 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:43.896+0000 7f4b68578f40 1 -- 172.21.15.32:0/3269613913 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55a4a1324b40 msgr2=0x55a4a13b5580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:56:43.900 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:43.896+0000 7f4b68578f40 1 --2- 172.21.15.32:0/3269613913 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55a4a1324b40 0x55a4a13b5580 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f4b54001d50 tx=0x7f4b5400b5b0 comp rx=0 tx=0).stop 2024-08-03T20:56:43.900 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:43.896+0000 7f4b68578f40 1 -- 172.21.15.32:0/3269613913 shutdown_connections 2024-08-03T20:56:43.900 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:43.896+0000 7f4b68578f40 1 --2- 172.21.15.32:0/3269613913 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55a4a1397be0 0x55a4a143e910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:43.900 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:43.896+0000 7f4b68578f40 1 --2- 172.21.15.32:0/3269613913 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a4a13b5ac0 0x55a4a14479f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:43.901 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:43.896+0000 7f4b68578f40 1 --2- 172.21.15.32:0/3269613913 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55a4a1324b40 0x55a4a13b5580 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:43.901 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:43.896+0000 7f4b68578f40 1 -- 172.21.15.32:0/3269613913 >> 172.21.15.32:0/3269613913 conn(0x55a4a11ffac0 msgr2=0x55a4a1395190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:56:43.901 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:43.896+0000 7f4b68578f40 1 -- 172.21.15.32:0/3269613913 shutdown_connections 2024-08-03T20:56:43.901 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:43.896+0000 7f4b68578f40 1 -- 172.21.15.32:0/3269613913 wait complete. 2024-08-03T20:56:43.901 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:43.897+0000 7f4b68578f40 1 Processor -- start 2024-08-03T20:56:43.901 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:43.897+0000 7f4b68578f40 1 -- start start 2024-08-03T20:56:43.901 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:43.899+0000 7f4b68578f40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55a4a1324b40 0x55a4a144d370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:56:43.901 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:43.899+0000 7f4b68578f40 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a4a1397be0 0x55a4a13b64c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:56:43.901 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:43.899+0000 7f4b68578f40 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55a4a13b5ac0 0x55a4a1392930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:56:43.901 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:43.899+0000 7f4b68578f40 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55a4a138de60 con 0x55a4a1397be0 2024-08-03T20:56:43.902 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:43.899+0000 7f4b68578f40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x55a4a13b6060 con 0x55a4a1324b40 2024-08-03T20:56:43.902 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:43.899+0000 7f4b68578f40 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x55a4a1395a40 con 0x55a4a13b5ac0 2024-08-03T20:56:43.902 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:43.899+0000 7f4b60c00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a4a1397be0 0x55a4a13b64c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:56:43.902 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:43.899+0000 7f4b60c00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a4a1397be0 0x55a4a13b64c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:50086/0 (socket says 172.21.15.32:50086) 2024-08-03T20:56:43.902 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:43.899+0000 7f4b60c00640 1 -- 172.21.15.32:0/1973349299 learned_addr learned my addr 172.21.15.32:0/1973349299 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:56:43.903 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:43.899+0000 7f4b61600640 1 --2- 172.21.15.32:0/1973349299 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55a4a1324b40 0x55a4a144d370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:56:43.903 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:43.899+0000 7f4b62000640 1 --2- 172.21.15.32:0/1973349299 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55a4a13b5ac0 0x55a4a1392930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:56:43.903 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:43.899+0000 7f4b60c00640 1 -- 172.21.15.32:0/1973349299 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55a4a13b5ac0 msgr2=0x55a4a1392930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:56:43.903 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:43.899+0000 7f4b60c00640 1 --2- 172.21.15.32:0/1973349299 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55a4a13b5ac0 0x55a4a1392930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:43.903 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:43.899+0000 7f4b60c00640 1 -- 172.21.15.32:0/1973349299 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55a4a1324b40 msgr2=0x55a4a144d370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:56:43.903 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:43.899+0000 7f4b60c00640 1 --2- 172.21.15.32:0/1973349299 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55a4a1324b40 0x55a4a144d370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:43.904 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:43.899+0000 7f4b60c00640 1 -- 172.21.15.32:0/1973349299 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a4a1328170 con 0x55a4a1397be0 2024-08-03T20:56:43.904 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:43.900+0000 7f4b60c00640 1 --2- 172.21.15.32:0/1973349299 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a4a1397be0 0x55a4a13b64c0 secure :-1 s=READY pgs=1418 cs=0 l=1 rev1=1 crypto rx=0x7f4b50046600 tx=0x7f4b5004d940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:56:43.904 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:43.900+0000 7f4b5a000640 1 -- 172.21.15.32:0/1973349299 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b50058190 con 0x55a4a1397be0 2024-08-03T20:56:43.904 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:43.900+0000 7f4b68578f40 1 -- 172.21.15.32:0/1973349299 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x55a4a1328450 con 0x55a4a1397be0 2024-08-03T20:56:43.904 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:43.900+0000 7f4b5a000640 1 -- 172.21.15.32:0/1973349299 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f4b50058510 con 0x55a4a1397be0 2024-08-03T20:56:43.904 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:43.900+0000 7f4b5a000640 1 -- 172.21.15.32:0/1973349299 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b50058940 con 0x55a4a1397be0 2024-08-03T20:56:43.904 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:43.900+0000 7f4b68578f40 1 -- 172.21.15.32:0/1973349299 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a4a144d9b0 con 0x55a4a1397be0 2024-08-03T20:56:43.904 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:43.902+0000 7f4b5a000640 1 -- 172.21.15.32:0/1973349299 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f4b50058ae0 con 0x55a4a1397be0 2024-08-03T20:56:43.905 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:43.903+0000 7f4b5a000640 1 --2- 172.21.15.32:0/1973349299 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f4b440759f0 0x7f4b44077f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:56:43.905 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:43.903+0000 7f4b5a000640 1 -- 172.21.15.32:0/1973349299 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 7058+0+0 (secure 0 0 0) 0x7f4b500dbbe0 con 0x55a4a1397be0 2024-08-03T20:56:43.905 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:43.904+0000 7f4b68578f40 1 --2- 172.21.15.32:0/1973349299 >> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] conn(0x55a4a1382820 0x55a4a13b7170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:56:43.905 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:43.904+0000 7f4b61600640 1 --2- 172.21.15.32:0/1973349299 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f4b440759f0 0x7f4b44077f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:56:43.905 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:43.904+0000 7f4b68578f40 1 -- 172.21.15.32:0/1973349299 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:1 12.1f 12:ff5b34d6:::1.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e99) -- 0x55a4a13b76b0 con 0x55a4a1382820 2024-08-03T20:56:43.905 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:43.904+0000 7f4b62000640 1 --2- 172.21.15.32:0/1973349299 >> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] conn(0x55a4a1382820 0x55a4a13b7170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:56:43.906 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:43.905+0000 7f4b61600640 1 --2- 172.21.15.32:0/1973349299 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f4b440759f0 0x7f4b44077f00 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7f4b54001c70 tx=0x7f4b54000ed0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:56:43.906 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:43.905+0000 7f4b62000640 1 --2- 172.21.15.32:0/1973349299 >> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] conn(0x55a4a1382820 0x55a4a13b7170 crc :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:56:43.908 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:43.906+0000 7f4b62000640 1 -- 172.21.15.32:0/1973349299 <== osd.6 v2:172.21.15.118:6825/801947386 1 ==== osd_op_reply(1 1.00000000 [delete] v99'180 uv180 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7f4b5c003040 con 0x55a4a1382820 2024-08-03T20:56:43.908 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:43.907+0000 7f4b68578f40 1 -- 172.21.15.32:0/1973349299 >> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] conn(0x55a4a1382820 msgr2=0x55a4a13b7170 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:56:43.909 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:43.907+0000 7f4b68578f40 1 --2- 172.21.15.32:0/1973349299 >> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] conn(0x55a4a1382820 0x55a4a13b7170 crc :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:43.909 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:43.907+0000 7f4b68578f40 1 -- 172.21.15.32:0/1973349299 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f4b440759f0 msgr2=0x7f4b44077f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:56:43.909 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:43.907+0000 7f4b68578f40 1 --2- 172.21.15.32:0/1973349299 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f4b440759f0 0x7f4b44077f00 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7f4b54001c70 tx=0x7f4b54000ed0 comp rx=0 tx=0).stop 2024-08-03T20:56:43.909 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:43.907+0000 7f4b68578f40 1 -- 172.21.15.32:0/1973349299 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a4a1397be0 msgr2=0x55a4a13b64c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:56:43.909 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:43.907+0000 7f4b68578f40 1 --2- 172.21.15.32:0/1973349299 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a4a1397be0 0x55a4a13b64c0 secure :-1 s=READY pgs=1418 cs=0 l=1 rev1=1 crypto rx=0x7f4b50046600 tx=0x7f4b5004d940 comp rx=0 tx=0).stop 2024-08-03T20:56:43.909 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:43.907+0000 7f4b62000640 1 -- 172.21.15.32:0/1973349299 reap_dead start 2024-08-03T20:56:43.909 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:43.907+0000 7f4b68578f40 1 -- 172.21.15.32:0/1973349299 shutdown_connections 2024-08-03T20:56:43.909 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:43.907+0000 7f4b68578f40 1 -- 172.21.15.32:0/1973349299 >> 172.21.15.32:0/1973349299 conn(0x55a4a11ffac0 msgr2=0x55a4a12df830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:56:43.909 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:43.907+0000 7f4b68578f40 1 -- 172.21.15.32:0/1973349299 shutdown_connections 2024-08-03T20:56:43.909 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:43.907+0000 7f4b68578f40 1 -- 172.21.15.32:0/1973349299 wait complete. 2024-08-03T20:56:43.919 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs joinable true 2024-08-03T20:56:44.151 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:44.149+0000 7f49ad200640 1 -- 172.21.15.32:0/3127028589 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f49a80d0d70 msgr2=0x7f49a80d29a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:56:44.151 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:44.149+0000 7f49ad200640 1 --2- 172.21.15.32:0/3127028589 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f49a80d0d70 0x7f49a80d29a0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f4998009860 tx=0x7f499802a5c0 comp rx=0 tx=0).stop 2024-08-03T20:56:44.151 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:44.149+0000 7f49ad200640 1 -- 172.21.15.32:0/3127028589 shutdown_connections 2024-08-03T20:56:44.151 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:44.149+0000 7f49ad200640 1 --2- 172.21.15.32:0/3127028589 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f49a80d0d70 0x7f49a80d29a0 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:44.151 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:44.149+0000 7f49ad200640 1 -- 172.21.15.32:0/3127028589 >> 172.21.15.32:0/3127028589 conn(0x7f49a80d1e80 msgr2=0x7f49a80d2290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:56:44.151 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:44.149+0000 7f49ad200640 1 -- 172.21.15.32:0/3127028589 shutdown_connections 2024-08-03T20:56:44.151 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:44.149+0000 7f49ad200640 1 -- 172.21.15.32:0/3127028589 wait complete. 2024-08-03T20:56:44.152 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:44.151+0000 7f49ad200640 1 Processor -- start 2024-08-03T20:56:44.152 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:44.151+0000 7f49ad200640 1 -- start start 2024-08-03T20:56:44.155 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:44.153+0000 7f49ad200640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f49a80b76e0 0x7f49a81f85f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:56:44.155 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:44.153+0000 7f49ad200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f49a80d0d70 0x7f49a81f8b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:56:44.155 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:44.153+0000 7f49ad200640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f49a81f91b0 0x7f49a815b4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:56:44.155 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:44.153+0000 7f49ad200640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f49a80afe40 con 0x7f49a80d0d70 2024-08-03T20:56:44.155 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:44.153+0000 7f49ad200640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f49a80afcc0 con 0x7f49a81f91b0 2024-08-03T20:56:44.155 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:44.153+0000 7f49ad200640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7f49a80affc0 con 0x7f49a80b76e0 2024-08-03T20:56:44.155 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:44.153+0000 7f499fe00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f49a80d0d70 0x7f49a81f8b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:56:44.156 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:44.153+0000 7f499fe00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f49a80d0d70 0x7f49a81f8b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:50098/0 (socket says 172.21.15.32:50098) 2024-08-03T20:56:44.156 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:44.153+0000 7f499fe00640 1 -- 172.21.15.32:0/2609596103 learned_addr learned my addr 172.21.15.32:0/2609596103 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:56:44.156 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:44.153+0000 7f49a7e00640 1 --2- 172.21.15.32:0/2609596103 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f49a81f91b0 0x7f49a815b4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:56:44.156 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:44.153+0000 7f49a7400640 1 --2- 172.21.15.32:0/2609596103 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f49a80b76e0 0x7f49a81f85f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:56:44.156 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:44.154+0000 7f499fe00640 1 -- 172.21.15.32:0/2609596103 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f49a80b76e0 msgr2=0x7f49a81f85f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:56:44.156 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:44.154+0000 7f499fe00640 1 --2- 172.21.15.32:0/2609596103 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f49a80b76e0 0x7f49a81f85f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:44.156 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:44.154+0000 7f499fe00640 1 -- 172.21.15.32:0/2609596103 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f49a81f91b0 msgr2=0x7f49a815b4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:56:44.156 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:44.154+0000 7f499fe00640 1 --2- 172.21.15.32:0/2609596103 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f49a81f91b0 0x7f49a815b4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:44.156 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:44.154+0000 7f499fe00640 1 -- 172.21.15.32:0/2609596103 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f49a815c640 con 0x7f49a80d0d70 2024-08-03T20:56:44.156 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:44.154+0000 7f499fe00640 1 --2- 172.21.15.32:0/2609596103 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f49a80d0d70 0x7f49a81f8b30 secure :-1 s=READY pgs=1420 cs=0 l=1 rev1=1 crypto rx=0x7f499000c3c0 tx=0x7f499000c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:56:44.157 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:44.154+0000 7f499f400640 1 -- 172.21.15.32:0/2609596103 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4990017070 con 0x7f49a80d0d70 2024-08-03T20:56:44.158 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:44.154+0000 7f499f400640 1 -- 172.21.15.32:0/2609596103 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f49900053c0 con 0x7f49a80d0d70 2024-08-03T20:56:44.158 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:44.154+0000 7f499f400640 1 -- 172.21.15.32:0/2609596103 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4990003850 con 0x7f49a80d0d70 2024-08-03T20:56:44.158 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:44.154+0000 7f49ad200640 1 -- 172.21.15.32:0/2609596103 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f49a80b1c90 con 0x7f49a80d0d70 2024-08-03T20:56:44.158 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:44.154+0000 7f49ad200640 1 -- 172.21.15.32:0/2609596103 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f49a80b2250 con 0x7f49a80d0d70 2024-08-03T20:56:44.158 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:44.155+0000 7f49ad200640 1 -- 172.21.15.32:0/2609596103 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f49a815c3b0 con 0x7f49a80d0d70 2024-08-03T20:56:44.159 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:44.157+0000 7f499f400640 1 -- 172.21.15.32:0/2609596103 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f499001a070 con 0x7f49a80d0d70 2024-08-03T20:56:44.159 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:44.157+0000 7f499f400640 1 --2- 172.21.15.32:0/2609596103 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f4984077010 0x7f4984079520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:56:44.159 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:44.158+0000 7f499f400640 1 -- 172.21.15.32:0/2609596103 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 7058+0+0 (secure 0 0 0) 0x7f49900981e0 con 0x7f49a80d0d70 2024-08-03T20:56:44.159 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:44.158+0000 7f49a7400640 1 --2- 172.21.15.32:0/2609596103 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f4984077010 0x7f4984079520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:56:44.160 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:44.159+0000 7f49a7400640 1 --2- 172.21.15.32:0/2609596103 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f4984077010 0x7f4984079520 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7f4998034000 tx=0x7f4998004730 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:56:44.163 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:44.162+0000 7f499f400640 1 -- 172.21.15.32:0/2609596103 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7f49900607b0 con 0x7f49a80d0d70 2024-08-03T20:56:44.411 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:44.410+0000 7f49ad200640 1 -- 172.21.15.32:0/2609596103 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "joinable", "val": "true"} v 0) -- 0x7f49a80d29a0 con 0x7f49a80d0d70 2024-08-03T20:56:44.797 INFO:tasks.ceph.mds.a.smithi032.stderr: -51> 2024-08-03T20:56:40.043+0000 7fccc0400640 -1 mds.0.cache.ino(0x100) No data while reading inode 0x100 2024-08-03T20:56:44.797 INFO:tasks.ceph.mds.a.smithi032.stderr: -41> 2024-08-03T20:56:40.043+0000 7fccc0400640 -1 log_channel(cluster) log [ERR] : Error loading MDS rank 0: (2) No such file or directory 2024-08-03T20:56:44.909 INFO:tasks.ceph.mds.a.smithi032.stdout:starting mds.a at 2024-08-03T20:56:45.034 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:45.032+0000 7f499f400640 1 -- 172.21.15.32:0/2609596103 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "joinable", "val": "true"}]=0 cephfs marked joinable; MDS may join as newly active. v176) ==== 162+0+0 (secure 0 0 0) 0x7f4990069040 con 0x7f49a80d0d70 2024-08-03T20:56:45.034 INFO:teuthology.orchestra.run.smithi032.stderr:cephfs marked joinable; MDS may join as newly active. 2024-08-03T20:56:45.037 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:45.035+0000 7f49ad200640 1 -- 172.21.15.32:0/2609596103 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f4984077010 msgr2=0x7f4984079520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:56:45.037 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:45.035+0000 7f49ad200640 1 --2- 172.21.15.32:0/2609596103 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f4984077010 0x7f4984079520 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7f4998034000 tx=0x7f4998004730 comp rx=0 tx=0).stop 2024-08-03T20:56:45.037 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:45.035+0000 7f49ad200640 1 -- 172.21.15.32:0/2609596103 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f49a80d0d70 msgr2=0x7f49a81f8b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:56:45.037 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:45.035+0000 7f49ad200640 1 --2- 172.21.15.32:0/2609596103 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f49a80d0d70 0x7f49a81f8b30 secure :-1 s=READY pgs=1420 cs=0 l=1 rev1=1 crypto rx=0x7f499000c3c0 tx=0x7f499000c880 comp rx=0 tx=0).stop 2024-08-03T20:56:45.037 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:45.035+0000 7f49ad200640 1 -- 172.21.15.32:0/2609596103 shutdown_connections 2024-08-03T20:56:45.037 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:45.035+0000 7f49ad200640 1 --2- 172.21.15.32:0/2609596103 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f4984077010 0x7f4984079520 unknown :-1 s=CLOSED pgs=581 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:45.037 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:45.035+0000 7f49ad200640 1 --2- 172.21.15.32:0/2609596103 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f49a81f91b0 0x7f49a815b4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:45.037 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:45.035+0000 7f49ad200640 1 --2- 172.21.15.32:0/2609596103 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f49a80d0d70 0x7f49a81f8b30 unknown :-1 s=CLOSED pgs=1420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:45.037 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:45.035+0000 7f49ad200640 1 --2- 172.21.15.32:0/2609596103 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f49a80b76e0 0x7f49a81f85f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:45.037 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:45.035+0000 7f49ad200640 1 -- 172.21.15.32:0/2609596103 >> 172.21.15.32:0/2609596103 conn(0x7f49a80d1e80 msgr2=0x7f49a815e140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:56:45.037 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:45.035+0000 7f49ad200640 1 -- 172.21.15.32:0/2609596103 shutdown_connections 2024-08-03T20:56:45.038 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:45.035+0000 7f49ad200640 1 -- 172.21.15.32:0/2609596103 wait complete. 2024-08-03T20:56:45.063 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-08-03T20:56:45.292 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:45.290+0000 7fd2e8000640 1 -- 172.21.15.32:0/334764008 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd2e0073e60 msgr2=0x7fd2e0074270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:56:45.292 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:45.290+0000 7fd2e8000640 1 --2- 172.21.15.32:0/334764008 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd2e0073e60 0x7fd2e0074270 secure :-1 s=READY pgs=1424 cs=0 l=1 rev1=1 crypto rx=0x7fd2cc005fd0 tx=0x7fd2cc02a540 comp rx=0 tx=0).stop 2024-08-03T20:56:45.292 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:45.290+0000 7fd2e8000640 1 -- 172.21.15.32:0/334764008 shutdown_connections 2024-08-03T20:56:45.292 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:45.290+0000 7fd2e8000640 1 --2- 172.21.15.32:0/334764008 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd2e0073e60 0x7fd2e0074270 unknown :-1 s=CLOSED pgs=1424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:45.293 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:45.290+0000 7fd2e8000640 1 --2- 172.21.15.32:0/334764008 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fd2e006e930 0x7fd2e0144f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:45.293 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:45.290+0000 7fd2e8000640 1 -- 172.21.15.32:0/334764008 >> 172.21.15.32:0/334764008 conn(0x7fd2e00b1bd0 msgr2=0x7fd2e00b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:56:45.293 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:45.290+0000 7fd2e8000640 1 -- 172.21.15.32:0/334764008 shutdown_connections 2024-08-03T20:56:45.293 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:45.290+0000 7fd2e8000640 1 -- 172.21.15.32:0/334764008 wait complete. 2024-08-03T20:56:45.294 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:45.293+0000 7fd2e8000640 1 Processor -- start 2024-08-03T20:56:45.294 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:45.293+0000 7fd2e8000640 1 -- start start 2024-08-03T20:56:45.296 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:45.294+0000 7fd2e8000640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fd2e006e930 0x7fd2e01418f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:56:45.296 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:45.294+0000 7fd2e8000640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fd2e0073e60 0x7fd2e0143400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:56:45.296 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:45.294+0000 7fd2e8000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd2e00747b0 0x7fd2e015ec40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:56:45.296 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:45.294+0000 7fd2e8000640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fd2e00cf9c0 con 0x7fd2e00747b0 2024-08-03T20:56:45.296 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:45.294+0000 7fd2e8000640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7fd2e00cf840 con 0x7fd2e0073e60 2024-08-03T20:56:45.296 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:45.294+0000 7fd2e8000640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7fd2e00ccff0 con 0x7fd2e006e930 2024-08-03T20:56:45.296 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:45.294+0000 7fd2e6000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd2e00747b0 0x7fd2e015ec40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:56:45.297 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:45.294+0000 7fd2e5600640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fd2e006e930 0x7fd2e01418f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:56:45.297 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:45.295+0000 7fd2e6000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd2e00747b0 0x7fd2e015ec40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:50132/0 (socket says 172.21.15.32:50132) 2024-08-03T20:56:45.297 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:45.295+0000 7fd2e6000640 1 -- 172.21.15.32:0/2794459261 learned_addr learned my addr 172.21.15.32:0/2794459261 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:56:45.297 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:45.295+0000 7fd2e4c00640 1 --2- 172.21.15.32:0/2794459261 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fd2e0073e60 0x7fd2e0143400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:56:45.298 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:45.295+0000 7fd2e6000640 1 -- 172.21.15.32:0/2794459261 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fd2e006e930 msgr2=0x7fd2e01418f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:56:45.298 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:45.295+0000 7fd2e6000640 1 --2- 172.21.15.32:0/2794459261 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fd2e006e930 0x7fd2e01418f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:45.298 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:45.295+0000 7fd2e6000640 1 -- 172.21.15.32:0/2794459261 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fd2e0073e60 msgr2=0x7fd2e0143400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:56:45.298 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:45.295+0000 7fd2e6000640 1 --2- 172.21.15.32:0/2794459261 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fd2e0073e60 0x7fd2e0143400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:45.299 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:45.295+0000 7fd2e6000640 1 -- 172.21.15.32:0/2794459261 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd2e0143ec0 con 0x7fd2e00747b0 2024-08-03T20:56:45.299 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:45.295+0000 7fd2e5600640 1 --2- 172.21.15.32:0/2794459261 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fd2e006e930 0x7fd2e01418f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-03T20:56:45.299 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:45.295+0000 7fd2e6000640 1 --2- 172.21.15.32:0/2794459261 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd2e00747b0 0x7fd2e015ec40 secure :-1 s=READY pgs=1425 cs=0 l=1 rev1=1 crypto rx=0x7fd2d800c700 tx=0x7fd2d800cbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:56:45.299 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:45.296+0000 7fd2dd600640 1 -- 172.21.15.32:0/2794459261 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2d8003b70 con 0x7fd2e00747b0 2024-08-03T20:56:45.299 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:45.296+0000 7fd2dd600640 1 -- 172.21.15.32:0/2794459261 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fd2d8003d10 con 0x7fd2e00747b0 2024-08-03T20:56:45.299 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:45.296+0000 7fd2dd600640 1 -- 172.21.15.32:0/2794459261 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2d800aab0 con 0x7fd2e00747b0 2024-08-03T20:56:45.299 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:45.296+0000 7fd2e8000640 1 -- 172.21.15.32:0/2794459261 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd2e01441a0 con 0x7fd2e00747b0 2024-08-03T20:56:45.299 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:45.296+0000 7fd2e8000640 1 -- 172.21.15.32:0/2794459261 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd2e0144760 con 0x7fd2e00747b0 2024-08-03T20:56:45.299 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:45.297+0000 7fd2e8000640 1 -- 172.21.15.32:0/2794459261 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd2e0143c10 con 0x7fd2e00747b0 2024-08-03T20:56:45.299 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:45.298+0000 7fd2dd600640 1 -- 172.21.15.32:0/2794459261 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7fd2d800ac50 con 0x7fd2e00747b0 2024-08-03T20:56:45.300 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:45.299+0000 7fd2dd600640 1 --2- 172.21.15.32:0/2794459261 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fd2c0077040 0x7fd2c0079550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:56:45.300 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:45.299+0000 7fd2dd600640 1 -- 172.21.15.32:0/2794459261 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 7058+0+0 (secure 0 0 0) 0x7fd2d8098b70 con 0x7fd2e00747b0 2024-08-03T20:56:45.301 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:45.300+0000 7fd2e5600640 1 --2- 172.21.15.32:0/2794459261 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fd2c0077040 0x7fd2c0079550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:56:45.302 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:45.300+0000 7fd2e5600640 1 --2- 172.21.15.32:0/2794459261 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fd2c0077040 0x7fd2c0079550 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7fd2d4009f20 tx=0x7fd2d4005f90 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:56:45.305 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:45.304+0000 7fd2dd600640 1 -- 172.21.15.32:0/2794459261 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7fd2d80611c0 con 0x7fd2e00747b0 2024-08-03T20:56:45.549 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:45.547+0000 7fd2e8000640 1 -- 172.21.15.32:0/2794459261 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd2e00cd510 con 0x7fd2e00747b0 2024-08-03T20:56:45.550 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:45.548+0000 7fd2dd600640 1 -- 172.21.15.32:0/2794459261 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 176 v176) ==== 95+0+5225 (secure 0 0 0) 0x7fd2d8065710 con 0x7fd2e00747b0 2024-08-03T20:56:45.550 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-03T20:56:45.550 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":176,"btime":"2024-08-03T20:56:45:032166+0000","default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6439,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6835/531080160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6834","nonce":531080160},{"type":"v1","addr":"172.21.15.118:6835","nonce":531080160}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":84},{"gid":6454,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6837/2796882319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6836","nonce":2796882319},{"type":"v1","addr":"172.21.15.118:6837","nonce":2796882319}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":86},{"gid":6813,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6834/2686933890","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":2686933890},{"type":"v1","addr":"172.21.15.32:6834","nonce":2686933890}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":85}],"filesystems":[{"mdsmap":{"epoch":176,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-08-03T20:54:26.950518+0000","modified":"2024-08-03T20:56:45.032150+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":99,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8415},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8415":{"gid":8415,"name":"a","rank":0,"incarnation":176,"state":"up:replay","state_seq":1,"addr":"172.21.15.32:6835/2242533881","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6833","nonce":2242533881},{"type":"v1","addr":"172.21.15.32:6835","nonce":2242533881}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":6}]} 2024-08-03T20:56:45.550 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 176 2024-08-03T20:56:45.552 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:45.550+0000 7fd2e8000640 1 -- 172.21.15.32:0/2794459261 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fd2c0077040 msgr2=0x7fd2c0079550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:56:45.552 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:45.550+0000 7fd2e8000640 1 --2- 172.21.15.32:0/2794459261 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fd2c0077040 0x7fd2c0079550 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7fd2d4009f20 tx=0x7fd2d4005f90 comp rx=0 tx=0).stop 2024-08-03T20:56:45.552 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:45.550+0000 7fd2e8000640 1 -- 172.21.15.32:0/2794459261 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd2e00747b0 msgr2=0x7fd2e015ec40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:56:45.553 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:45.550+0000 7fd2e8000640 1 --2- 172.21.15.32:0/2794459261 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd2e00747b0 0x7fd2e015ec40 secure :-1 s=READY pgs=1425 cs=0 l=1 rev1=1 crypto rx=0x7fd2d800c700 tx=0x7fd2d800cbc0 comp rx=0 tx=0).stop 2024-08-03T20:56:45.553 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:45.550+0000 7fd2e8000640 1 -- 172.21.15.32:0/2794459261 shutdown_connections 2024-08-03T20:56:45.553 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:45.550+0000 7fd2e8000640 1 --2- 172.21.15.32:0/2794459261 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fd2c0077040 0x7fd2c0079550 unknown :-1 s=CLOSED pgs=583 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:45.553 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:45.550+0000 7fd2e8000640 1 --2- 172.21.15.32:0/2794459261 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fd2e0073e60 0x7fd2e0143400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:45.553 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:45.551+0000 7fd2e8000640 1 --2- 172.21.15.32:0/2794459261 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd2e00747b0 0x7fd2e015ec40 unknown :-1 s=CLOSED pgs=1425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:45.553 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:45.551+0000 7fd2e8000640 1 --2- 172.21.15.32:0/2794459261 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fd2e006e930 0x7fd2e01418f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:45.553 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:45.551+0000 7fd2e8000640 1 -- 172.21.15.32:0/2794459261 >> 172.21.15.32:0/2794459261 conn(0x7fd2e00b1bd0 msgr2=0x7fd2e00b07b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:56:45.553 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:45.551+0000 7fd2e8000640 1 -- 172.21.15.32:0/2794459261 shutdown_connections 2024-08-03T20:56:45.553 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:45.551+0000 7fd2e8000640 1 -- 172.21.15.32:0/2794459261 wait complete. 2024-08-03T20:56:45.575 DEBUG:tasks.ceph_test_case:wait_until_true: waiting (timeout=60 retry_count=0)... 2024-08-03T20:56:48.047 INFO:tasks.ceph.mds.a.smithi032.stderr:2024-08-03T20:56:48.045+0000 7f0047a00640 -1 log_channel(cluster) log [ERR] : dir 0x1 object missing on disk; some files may be lost () 2024-08-03T20:56:48.047 INFO:tasks.ceph.mds.a.smithi032.stderr:2024-08-03T20:56:48.045+0000 7f0047a00640 -1 mds.0.damage notify_dirfrag Damage to fragment * of ino 0x1 is fatal because it is a system directory for this rank 2024-08-03T20:56:48.048 INFO:tasks.ceph.mon.a.smithi032.stderr:2024-08-03T20:56:48.047+0000 7f4e6d600640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-08-03T20:56:48.048 INFO:tasks.ceph.mon.a.smithi032.stderr:2024-08-03T20:56:48.047+0000 7f4e6d600640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 mds daemon damaged (MDS_DAMAGE) 2024-08-03T20:56:50.044 INFO:tasks.ceph.mds.a.smithi032.stderr:2024-08-03T20:56:50.043+0000 7f004ca00640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-08-03T20:56:50.576 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-08-03T20:56:50.816 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:50.814+0000 7f8f4c000640 1 -- 172.21.15.32:0/3675072536 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8f440b4de0 msgr2=0x7f8f441515c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:56:50.816 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:50.814+0000 7f8f4c000640 1 --2- 172.21.15.32:0/3675072536 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8f440b4de0 0x7f8f441515c0 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f8f38009830 tx=0x7f8f3802a410 comp rx=0 tx=0).stop 2024-08-03T20:56:50.817 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:50.814+0000 7f8f4c000640 1 -- 172.21.15.32:0/3675072536 shutdown_connections 2024-08-03T20:56:50.817 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:50.814+0000 7f8f4c000640 1 --2- 172.21.15.32:0/3675072536 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f8f440733f0 0x7f8f44073800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:50.817 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:50.814+0000 7f8f4c000640 1 --2- 172.21.15.32:0/3675072536 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8f440b4de0 0x7f8f441515c0 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:50.817 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:50.814+0000 7f8f4c000640 1 -- 172.21.15.32:0/3675072536 >> 172.21.15.32:0/3675072536 conn(0x7f8f440b1bd0 msgr2=0x7f8f440b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:56:50.817 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:50.814+0000 7f8f4c000640 1 -- 172.21.15.32:0/3675072536 shutdown_connections 2024-08-03T20:56:50.817 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:50.814+0000 7f8f4c000640 1 -- 172.21.15.32:0/3675072536 wait complete. 2024-08-03T20:56:50.818 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:50.817+0000 7f8f4c000640 1 Processor -- start 2024-08-03T20:56:50.818 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:50.817+0000 7f8f4c000640 1 -- start start 2024-08-03T20:56:50.821 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:50.818+0000 7f8f4c000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8f440733f0 0x7f8f441462e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:56:50.821 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:50.818+0000 7f8f4c000640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8f440b4de0 0x7f8f4414f3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:56:50.822 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:50.818+0000 7f8f4c000640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f8f4415e170 0x7f8f441543c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:56:50.822 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:50.818+0000 7f8f4c000640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f8f4406e930 con 0x7f8f440733f0 2024-08-03T20:56:50.822 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:50.818+0000 7f8f4c000640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f8f4406eab0 con 0x7f8f440b4de0 2024-08-03T20:56:50.822 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:50.818+0000 7f8f4c000640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7f8f4406ec30 con 0x7f8f4415e170 2024-08-03T20:56:50.822 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:50.818+0000 7f8f49600640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8f440733f0 0x7f8f441462e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:56:50.822 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:50.818+0000 7f8f49600640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8f440733f0 0x7f8f441462e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:39862/0 (socket says 172.21.15.32:39862) 2024-08-03T20:56:50.822 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:50.818+0000 7f8f49600640 1 -- 172.21.15.32:0/1653087851 learned_addr learned my addr 172.21.15.32:0/1653087851 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:56:50.822 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:50.819+0000 7f8f48c00640 1 --2- 172.21.15.32:0/1653087851 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8f440b4de0 0x7f8f4414f3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:56:50.822 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:50.819+0000 7f8f4a000640 1 --2- 172.21.15.32:0/1653087851 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f8f4415e170 0x7f8f441543c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:56:50.822 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:50.819+0000 7f8f49600640 1 -- 172.21.15.32:0/1653087851 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f8f4415e170 msgr2=0x7f8f441543c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:56:50.822 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:50.819+0000 7f8f49600640 1 --2- 172.21.15.32:0/1653087851 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f8f4415e170 0x7f8f441543c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:50.822 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:50.819+0000 7f8f49600640 1 -- 172.21.15.32:0/1653087851 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8f440b4de0 msgr2=0x7f8f4414f3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:56:50.823 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:50.819+0000 7f8f49600640 1 --2- 172.21.15.32:0/1653087851 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8f440b4de0 0x7f8f4414f3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:50.823 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:50.819+0000 7f8f49600640 1 -- 172.21.15.32:0/1653087851 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8f440cd590 con 0x7f8f440733f0 2024-08-03T20:56:50.823 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:50.819+0000 7f8f49600640 1 --2- 172.21.15.32:0/1653087851 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8f440733f0 0x7f8f441462e0 secure :-1 s=READY pgs=1427 cs=0 l=1 rev1=1 crypto rx=0x7f8f38009950 tx=0x7f8f38003b60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:56:50.823 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:50.819+0000 7f8f41600640 1 -- 172.21.15.32:0/1653087851 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f3803f030 con 0x7f8f440733f0 2024-08-03T20:56:50.823 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:50.819+0000 7f8f41600640 1 -- 172.21.15.32:0/1653087851 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f8f38004510 con 0x7f8f440733f0 2024-08-03T20:56:50.823 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:50.819+0000 7f8f41600640 1 -- 172.21.15.32:0/1653087851 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f38037040 con 0x7f8f440733f0 2024-08-03T20:56:50.823 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:50.820+0000 7f8f4c000640 1 -- 172.21.15.32:0/1653087851 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8f440cd870 con 0x7f8f440733f0 2024-08-03T20:56:50.823 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:50.820+0000 7f8f4c000640 1 -- 172.21.15.32:0/1653087851 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8f440cde30 con 0x7f8f440733f0 2024-08-03T20:56:50.824 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:50.821+0000 7f8f4c000640 1 -- 172.21.15.32:0/1653087851 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8f44073800 con 0x7f8f440733f0 2024-08-03T20:56:50.824 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:50.822+0000 7f8f41600640 1 -- 172.21.15.32:0/1653087851 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f8f38042070 con 0x7f8f440733f0 2024-08-03T20:56:50.824 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:50.823+0000 7f8f41600640 1 --2- 172.21.15.32:0/1653087851 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f8f24077010 0x7f8f24079520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:56:50.824 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:50.823+0000 7f8f41600640 1 -- 172.21.15.32:0/1653087851 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 7144+0+0 (secure 0 0 0) 0x7f8f380b8ae0 con 0x7f8f440733f0 2024-08-03T20:56:50.825 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:50.824+0000 7f8f48c00640 1 --2- 172.21.15.32:0/1653087851 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f8f24077010 0x7f8f24079520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:56:50.825 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:50.824+0000 7f8f48c00640 1 --2- 172.21.15.32:0/1653087851 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f8f24077010 0x7f8f24079520 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7f8f4415ce50 tx=0x7f8f30005f90 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:56:50.829 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:50.828+0000 7f8f41600640 1 -- 172.21.15.32:0/1653087851 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7f8f38049070 con 0x7f8f440733f0 2024-08-03T20:56:51.071 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:51.070+0000 7f8f4c000640 1 -- 172.21.15.32:0/1653087851 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f8f44154f40 con 0x7f8f440733f0 2024-08-03T20:56:51.072 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:51.070+0000 7f8f41600640 1 -- 172.21.15.32:0/1653087851 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 180 v180) ==== 95+0+4391 (secure 0 0 0) 0x7f8f380817d0 con 0x7f8f440733f0 2024-08-03T20:56:51.072 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-03T20:56:51.072 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":180,"btime":"2024-08-03T20:56:48:048380+0000","default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6439,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6835/531080160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6834","nonce":531080160},{"type":"v1","addr":"172.21.15.118:6835","nonce":531080160}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":84},{"gid":6454,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6837/2796882319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6836","nonce":2796882319},{"type":"v1","addr":"172.21.15.118:6837","nonce":2796882319}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":86},{"gid":6813,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6834/2686933890","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":2686933890},{"type":"v1","addr":"172.21.15.32:6834","nonce":2686933890}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":85}],"filesystems":[{"mdsmap":{"epoch":180,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-08-03T20:54:26.950518+0000","modified":"2024-08-03T20:56:48.048375+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":100,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":6}]} 2024-08-03T20:56:51.072 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 180 2024-08-03T20:56:51.074 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:51.073+0000 7f8f4c000640 1 -- 172.21.15.32:0/1653087851 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f8f24077010 msgr2=0x7f8f24079520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:56:51.075 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:51.073+0000 7f8f4c000640 1 --2- 172.21.15.32:0/1653087851 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f8f24077010 0x7f8f24079520 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7f8f4415ce50 tx=0x7f8f30005f90 comp rx=0 tx=0).stop 2024-08-03T20:56:51.075 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:51.073+0000 7f8f4c000640 1 -- 172.21.15.32:0/1653087851 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8f440733f0 msgr2=0x7f8f441462e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:56:51.075 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:51.073+0000 7f8f4c000640 1 --2- 172.21.15.32:0/1653087851 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8f440733f0 0x7f8f441462e0 secure :-1 s=READY pgs=1427 cs=0 l=1 rev1=1 crypto rx=0x7f8f38009950 tx=0x7f8f38003b60 comp rx=0 tx=0).stop 2024-08-03T20:56:51.075 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:51.073+0000 7f8f4c000640 1 -- 172.21.15.32:0/1653087851 shutdown_connections 2024-08-03T20:56:51.075 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:51.073+0000 7f8f4c000640 1 --2- 172.21.15.32:0/1653087851 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f8f24077010 0x7f8f24079520 unknown :-1 s=CLOSED pgs=584 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:51.075 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:51.073+0000 7f8f4c000640 1 --2- 172.21.15.32:0/1653087851 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8f440b4de0 0x7f8f4414f3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:51.075 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:51.073+0000 7f8f4c000640 1 --2- 172.21.15.32:0/1653087851 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f8f4415e170 0x7f8f441543c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:51.075 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:51.073+0000 7f8f4c000640 1 --2- 172.21.15.32:0/1653087851 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8f440733f0 0x7f8f441462e0 unknown :-1 s=CLOSED pgs=1427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:51.075 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:51.073+0000 7f8f4c000640 1 -- 172.21.15.32:0/1653087851 >> 172.21.15.32:0/1653087851 conn(0x7f8f440b1bd0 msgr2=0x7f8f44149bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:56:51.075 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:51.073+0000 7f8f4c000640 1 -- 172.21.15.32:0/1653087851 shutdown_connections 2024-08-03T20:56:51.075 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:51.073+0000 7f8f4c000640 1 -- 172.21.15.32:0/1653087851 wait complete. 2024-08-03T20:56:51.101 DEBUG:tasks.ceph_test_case:wait_until_true: success in 5s and 0 retries 2024-08-03T20:56:51.101 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-08-03T20:56:51.331 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:51.328+0000 7f145a800640 1 -- 172.21.15.32:0/1075627368 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f14540acd80 msgr2=0x7f14540ad190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:56:51.331 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:51.328+0000 7f145a800640 1 --2- 172.21.15.32:0/1075627368 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f14540acd80 0x7f14540ad190 secure :-1 s=READY pgs=1429 cs=0 l=1 rev1=1 crypto rx=0x7f1440009830 tx=0x7f144002a470 comp rx=0 tx=0).stop 2024-08-03T20:56:51.331 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:51.328+0000 7f145a800640 1 -- 172.21.15.32:0/1075627368 shutdown_connections 2024-08-03T20:56:51.331 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:51.328+0000 7f145a800640 1 --2- 172.21.15.32:0/1075627368 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f14540acd80 0x7f14540ad190 unknown :-1 s=CLOSED pgs=1429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:51.331 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:51.328+0000 7f145a800640 1 --2- 172.21.15.32:0/1075627368 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f14540af020 0x7f14540742b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:51.331 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:51.328+0000 7f145a800640 1 -- 172.21.15.32:0/1075627368 >> 172.21.15.32:0/1075627368 conn(0x7f14540b2120 msgr2=0x7f14540cfd60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:56:51.332 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:51.328+0000 7f145a800640 1 -- 172.21.15.32:0/1075627368 shutdown_connections 2024-08-03T20:56:51.332 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:51.328+0000 7f145a800640 1 -- 172.21.15.32:0/1075627368 wait complete. 2024-08-03T20:56:51.332 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:51.330+0000 7f145a800640 1 Processor -- start 2024-08-03T20:56:51.333 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:51.330+0000 7f145a800640 1 -- start start 2024-08-03T20:56:51.336 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:51.332+0000 7f145a800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f145406f440 0x7f1454155350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:56:51.336 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:51.332+0000 7f145a800640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f14540acd80 0x7f14540cddb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:56:51.336 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:51.332+0000 7f145a800640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f14540af020 0x7f14540ce2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:56:51.336 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:51.332+0000 7f145a800640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f1454160260 con 0x7f145406f440 2024-08-03T20:56:51.336 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:51.332+0000 7f145a800640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f14541600e0 con 0x7f14540acd80 2024-08-03T20:56:51.336 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:51.332+0000 7f145a800640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7f14541603e0 con 0x7f14540af020 2024-08-03T20:56:51.336 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:51.332+0000 7f1459400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f145406f440 0x7f1454155350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:56:51.337 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:51.332+0000 7f1459400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f145406f440 0x7f1454155350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:39874/0 (socket says 172.21.15.32:39874) 2024-08-03T20:56:51.337 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:51.332+0000 7f1459400640 1 -- 172.21.15.32:0/3110039355 learned_addr learned my addr 172.21.15.32:0/3110039355 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:56:51.337 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:51.332+0000 7f1458a00640 1 --2- 172.21.15.32:0/3110039355 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f14540acd80 0x7f14540cddb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:56:51.337 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:51.332+0000 7f1459e00640 1 --2- 172.21.15.32:0/3110039355 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f14540af020 0x7f14540ce2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:56:51.337 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:51.332+0000 7f1459400640 1 -- 172.21.15.32:0/3110039355 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f14540af020 msgr2=0x7f14540ce2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:56:51.337 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:51.333+0000 7f1459400640 1 --2- 172.21.15.32:0/3110039355 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f14540af020 0x7f14540ce2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:51.337 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:51.333+0000 7f1459400640 1 -- 172.21.15.32:0/3110039355 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f14540acd80 msgr2=0x7f14540cddb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:56:51.337 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:51.333+0000 7f1459400640 1 --2- 172.21.15.32:0/3110039355 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f14540acd80 0x7f14540cddb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:51.337 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:51.333+0000 7f1459400640 1 -- 172.21.15.32:0/3110039355 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f145409aaf0 con 0x7f145406f440 2024-08-03T20:56:51.337 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:51.333+0000 7f1459400640 1 --2- 172.21.15.32:0/3110039355 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f145406f440 0x7f1454155350 secure :-1 s=READY pgs=1430 cs=0 l=1 rev1=1 crypto rx=0x7f144800d740 tx=0x7f144800dc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:56:51.337 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:51.333+0000 7f1451600640 1 -- 172.21.15.32:0/3110039355 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1448017070 con 0x7f145406f440 2024-08-03T20:56:51.337 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:51.333+0000 7f1451600640 1 -- 172.21.15.32:0/3110039355 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f1448009d20 con 0x7f145406f440 2024-08-03T20:56:51.338 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:51.333+0000 7f145a800640 1 -- 172.21.15.32:0/3110039355 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f145409ad60 con 0x7f145406f440 2024-08-03T20:56:51.338 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:51.333+0000 7f1451600640 1 -- 172.21.15.32:0/3110039355 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14480106c0 con 0x7f145406f440 2024-08-03T20:56:51.338 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:51.333+0000 7f145a800640 1 -- 172.21.15.32:0/3110039355 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f145409b320 con 0x7f145406f440 2024-08-03T20:56:51.338 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:51.334+0000 7f145a800640 1 -- 172.21.15.32:0/3110039355 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f14540cc410 con 0x7f145406f440 2024-08-03T20:56:51.338 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:51.336+0000 7f1451600640 1 -- 172.21.15.32:0/3110039355 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f144801a070 con 0x7f145406f440 2024-08-03T20:56:51.339 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:51.336+0000 7f1451600640 1 --2- 172.21.15.32:0/3110039355 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f1434077010 0x7f1434079520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:56:51.339 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:51.337+0000 7f1451600640 1 -- 172.21.15.32:0/3110039355 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 7144+0+0 (secure 0 0 0) 0x7f1448098ff0 con 0x7f145406f440 2024-08-03T20:56:51.339 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:51.337+0000 7f1458a00640 1 --2- 172.21.15.32:0/3110039355 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f1434077010 0x7f1434079520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:56:51.339 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:51.338+0000 7f1458a00640 1 --2- 172.21.15.32:0/3110039355 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f1434077010 0x7f1434079520 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7f1440034000 tx=0x7f1440004730 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:56:51.342 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:51.341+0000 7f1451600640 1 -- 172.21.15.32:0/3110039355 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7f14540cc410 con 0x7f145406f440 2024-08-03T20:56:51.594 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:51.592+0000 7f145a800640 1 -- 172.21.15.32:0/3110039355 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f14540ad190 con 0x7f145406f440 2024-08-03T20:56:51.595 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:51.593+0000 7f1451600640 1 -- 172.21.15.32:0/3110039355 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 180 v180) ==== 95+0+4391 (secure 0 0 0) 0x7f1448061bf0 con 0x7f145406f440 2024-08-03T20:56:51.595 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-03T20:56:51.595 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":180,"btime":"2024-08-03T20:56:48:048380+0000","default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6439,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6835/531080160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6834","nonce":531080160},{"type":"v1","addr":"172.21.15.118:6835","nonce":531080160}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":84},{"gid":6454,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6837/2796882319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6836","nonce":2796882319},{"type":"v1","addr":"172.21.15.118:6837","nonce":2796882319}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":86},{"gid":6813,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6834/2686933890","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":2686933890},{"type":"v1","addr":"172.21.15.32:6834","nonce":2686933890}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":85}],"filesystems":[{"mdsmap":{"epoch":180,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-08-03T20:54:26.950518+0000","modified":"2024-08-03T20:56:48.048375+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":100,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":6}]} 2024-08-03T20:56:51.596 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 180 2024-08-03T20:56:51.597 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:51.595+0000 7f145a800640 1 -- 172.21.15.32:0/3110039355 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f1434077010 msgr2=0x7f1434079520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:56:51.597 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:51.595+0000 7f145a800640 1 --2- 172.21.15.32:0/3110039355 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f1434077010 0x7f1434079520 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7f1440034000 tx=0x7f1440004730 comp rx=0 tx=0).stop 2024-08-03T20:56:51.598 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:51.595+0000 7f145a800640 1 -- 172.21.15.32:0/3110039355 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f145406f440 msgr2=0x7f1454155350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:56:51.598 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:51.595+0000 7f145a800640 1 --2- 172.21.15.32:0/3110039355 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f145406f440 0x7f1454155350 secure :-1 s=READY pgs=1430 cs=0 l=1 rev1=1 crypto rx=0x7f144800d740 tx=0x7f144800dc00 comp rx=0 tx=0).stop 2024-08-03T20:56:51.598 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:51.595+0000 7f145a800640 1 -- 172.21.15.32:0/3110039355 shutdown_connections 2024-08-03T20:56:51.598 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:51.595+0000 7f145a800640 1 --2- 172.21.15.32:0/3110039355 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f1434077010 0x7f1434079520 unknown :-1 s=CLOSED pgs=585 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:51.598 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:51.595+0000 7f145a800640 1 --2- 172.21.15.32:0/3110039355 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f14540acd80 0x7f14540cddb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:51.599 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:51.595+0000 7f145a800640 1 --2- 172.21.15.32:0/3110039355 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f14540af020 0x7f14540ce2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:51.599 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:51.595+0000 7f145a800640 1 --2- 172.21.15.32:0/3110039355 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f145406f440 0x7f1454155350 unknown :-1 s=CLOSED pgs=1430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:51.599 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:51.595+0000 7f145a800640 1 -- 172.21.15.32:0/3110039355 >> 172.21.15.32:0/3110039355 conn(0x7f14540b2120 msgr2=0x7f1454160810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:56:51.599 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:51.595+0000 7f145a800640 1 -- 172.21.15.32:0/3110039355 shutdown_connections 2024-08-03T20:56:51.599 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:51.595+0000 7f145a800640 1 -- 172.21.15.32:0/3110039355 wait complete. 2024-08-03T20:56:51.623 DEBUG:tasks.ceph_test_case:wait_until_true: success in 0s and 0 retries 2024-08-03T20:56:51.623 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-08-03T20:56:51.853 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:51.851+0000 7f4a3a600640 1 -- 172.21.15.32:0/1726668214 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4a340afa60 msgr2=0x7f4a34160120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:56:51.853 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:51.851+0000 7f4a3a600640 1 --2- 172.21.15.32:0/1726668214 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4a340afa60 0x7f4a34160120 secure :-1 s=READY pgs=1431 cs=0 l=1 rev1=1 crypto rx=0x7f4a34074fb0 tx=0x7f4a2c02a980 comp rx=0 tx=0).stop 2024-08-03T20:56:51.853 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:51.851+0000 7f4a3a600640 1 -- 172.21.15.32:0/1726668214 shutdown_connections 2024-08-03T20:56:51.853 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:51.851+0000 7f4a3a600640 1 --2- 172.21.15.32:0/1726668214 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4a340afa60 0x7f4a34160120 unknown :-1 s=CLOSED pgs=1431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:51.853 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:51.851+0000 7f4a3a600640 1 --2- 172.21.15.32:0/1726668214 >> v2:172.21.15.118:3300/0 conn(0x7f4a34153880 0x7f4a340af520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:51.854 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:51.851+0000 7f4a3a600640 1 --2- 172.21.15.32:0/1726668214 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f4a3406f110 0x7f4a34153340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:51.854 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:51.851+0000 7f4a3a600640 1 -- 172.21.15.32:0/1726668214 >> 172.21.15.32:0/1726668214 conn(0x7f4a340cd1e0 msgr2=0x7f4a340cd5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:56:51.854 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:51.851+0000 7f4a3a600640 1 -- 172.21.15.32:0/1726668214 shutdown_connections 2024-08-03T20:56:51.854 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:51.851+0000 7f4a3a600640 1 -- 172.21.15.32:0/1726668214 wait complete. 2024-08-03T20:56:51.854 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:51.853+0000 7f4a3a600640 1 Processor -- start 2024-08-03T20:56:51.854 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:51.853+0000 7f4a3a600640 1 -- start start 2024-08-03T20:56:51.857 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:51.855+0000 7f4a3a600640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4a3406f110 0x7f4a34156680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:56:51.857 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:51.855+0000 7f4a3a600640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f4a340afa60 0x7f4a3414fc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:56:51.857 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:51.855+0000 7f4a3a600640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f4a34153880 0x7f4a3415dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:56:51.857 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:51.855+0000 7f4a3a600640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f4a341609f0 con 0x7f4a3406f110 2024-08-03T20:56:51.857 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:51.855+0000 7f4a3a600640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f4a34160870 con 0x7f4a34153880 2024-08-03T20:56:51.857 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:51.855+0000 7f4a3a600640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7f4a34160b70 con 0x7f4a340afa60 2024-08-03T20:56:51.857 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:51.855+0000 7f4a33400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4a3406f110 0x7f4a34156680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:56:51.858 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:51.855+0000 7f4a33400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4a3406f110 0x7f4a34156680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:39886/0 (socket says 172.21.15.32:39886) 2024-08-03T20:56:51.858 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:51.855+0000 7f4a33400640 1 -- 172.21.15.32:0/1352400706 learned_addr learned my addr 172.21.15.32:0/1352400706 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:56:51.858 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:51.855+0000 7f4a32a00640 1 --2- 172.21.15.32:0/1352400706 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f4a340afa60 0x7f4a3414fc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:56:51.858 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:51.855+0000 7f4a33e00640 1 --2- 172.21.15.32:0/1352400706 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f4a34153880 0x7f4a3415dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:56:51.858 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:51.855+0000 7f4a33400640 1 -- 172.21.15.32:0/1352400706 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f4a340afa60 msgr2=0x7f4a3414fc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:56:51.858 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:51.855+0000 7f4a33400640 1 --2- 172.21.15.32:0/1352400706 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f4a340afa60 0x7f4a3414fc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:51.858 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:51.855+0000 7f4a33400640 1 -- 172.21.15.32:0/1352400706 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f4a34153880 msgr2=0x7f4a3415dcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:56:51.858 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:51.855+0000 7f4a33400640 1 --2- 172.21.15.32:0/1352400706 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f4a34153880 0x7f4a3415dcd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:51.858 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:51.855+0000 7f4a33400640 1 -- 172.21.15.32:0/1352400706 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4a34157f20 con 0x7f4a3406f110 2024-08-03T20:56:51.858 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:51.856+0000 7f4a33e00640 1 --2- 172.21.15.32:0/1352400706 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f4a34153880 0x7f4a3415dcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-03T20:56:51.858 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:51.856+0000 7f4a33400640 1 --2- 172.21.15.32:0/1352400706 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4a3406f110 0x7f4a34156680 secure :-1 s=READY pgs=1432 cs=0 l=1 rev1=1 crypto rx=0x7f4a2400d720 tx=0x7f4a2400dbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:56:51.859 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:51.856+0000 7f4a2b400640 1 -- 172.21.15.32:0/1352400706 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a24017070 con 0x7f4a3406f110 2024-08-03T20:56:51.859 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:51.856+0000 7f4a2b400640 1 -- 172.21.15.32:0/1352400706 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f4a24009ab0 con 0x7f4a3406f110 2024-08-03T20:56:51.859 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:51.856+0000 7f4a3a600640 1 -- 172.21.15.32:0/1352400706 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4a340ca7b0 con 0x7f4a3406f110 2024-08-03T20:56:51.859 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:51.856+0000 7f4a2b400640 1 -- 172.21.15.32:0/1352400706 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a24010460 con 0x7f4a3406f110 2024-08-03T20:56:51.859 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:51.856+0000 7f4a3a600640 1 -- 172.21.15.32:0/1352400706 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4a34155e80 con 0x7f4a3406f110 2024-08-03T20:56:51.859 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:51.857+0000 7f4a3a600640 1 -- 172.21.15.32:0/1352400706 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4a34160120 con 0x7f4a3406f110 2024-08-03T20:56:51.860 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:51.859+0000 7f4a2b400640 1 -- 172.21.15.32:0/1352400706 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f4a2401a070 con 0x7f4a3406f110 2024-08-03T20:56:51.861 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:51.859+0000 7f4a2b400640 1 --2- 172.21.15.32:0/1352400706 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f4a14077010 0x7f4a14079520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:56:51.861 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:51.860+0000 7f4a2b400640 1 -- 172.21.15.32:0/1352400706 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 7144+0+0 (secure 0 0 0) 0x7f4a24098ed0 con 0x7f4a3406f110 2024-08-03T20:56:51.861 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:51.860+0000 7f4a32a00640 1 --2- 172.21.15.32:0/1352400706 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f4a14077010 0x7f4a14079520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:56:51.862 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:51.861+0000 7f4a32a00640 1 --2- 172.21.15.32:0/1352400706 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f4a14077010 0x7f4a14079520 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7f4a340b66b0 tx=0x7f4a2000b2e0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:56:51.866 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:51.864+0000 7f4a2b400640 1 -- 172.21.15.32:0/1352400706 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7f4a24061440 con 0x7f4a3406f110 2024-08-03T20:56:52.112 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:52.110+0000 7f4a3a600640 1 -- 172.21.15.32:0/1352400706 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f4a34076510 con 0x7f4a3406f110 2024-08-03T20:56:52.112 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:52.111+0000 7f4a2b400640 1 -- 172.21.15.32:0/1352400706 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 180 v180) ==== 95+0+4391 (secure 0 0 0) 0x7f4a24065990 con 0x7f4a3406f110 2024-08-03T20:56:52.112 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-03T20:56:52.113 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":180,"btime":"2024-08-03T20:56:48:048380+0000","default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6439,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6835/531080160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6834","nonce":531080160},{"type":"v1","addr":"172.21.15.118:6835","nonce":531080160}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":84},{"gid":6454,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6837/2796882319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6836","nonce":2796882319},{"type":"v1","addr":"172.21.15.118:6837","nonce":2796882319}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":86},{"gid":6813,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6834/2686933890","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":2686933890},{"type":"v1","addr":"172.21.15.32:6834","nonce":2686933890}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":85}],"filesystems":[{"mdsmap":{"epoch":180,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-08-03T20:54:26.950518+0000","modified":"2024-08-03T20:56:48.048375+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":100,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":6}]} 2024-08-03T20:56:52.113 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 180 2024-08-03T20:56:52.115 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:52.113+0000 7f4a3a600640 1 -- 172.21.15.32:0/1352400706 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f4a14077010 msgr2=0x7f4a14079520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:56:52.115 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:52.113+0000 7f4a3a600640 1 --2- 172.21.15.32:0/1352400706 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f4a14077010 0x7f4a14079520 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7f4a340b66b0 tx=0x7f4a2000b2e0 comp rx=0 tx=0).stop 2024-08-03T20:56:52.115 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:52.113+0000 7f4a3a600640 1 -- 172.21.15.32:0/1352400706 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4a3406f110 msgr2=0x7f4a34156680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:56:52.115 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:52.113+0000 7f4a3a600640 1 --2- 172.21.15.32:0/1352400706 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4a3406f110 0x7f4a34156680 secure :-1 s=READY pgs=1432 cs=0 l=1 rev1=1 crypto rx=0x7f4a2400d720 tx=0x7f4a2400dbe0 comp rx=0 tx=0).stop 2024-08-03T20:56:52.115 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:52.113+0000 7f4a3a600640 1 -- 172.21.15.32:0/1352400706 shutdown_connections 2024-08-03T20:56:52.115 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:52.113+0000 7f4a3a600640 1 --2- 172.21.15.32:0/1352400706 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f4a14077010 0x7f4a14079520 unknown :-1 s=CLOSED pgs=586 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:52.115 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:52.113+0000 7f4a3a600640 1 --2- 172.21.15.32:0/1352400706 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f4a34153880 0x7f4a3415dcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:52.115 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:52.113+0000 7f4a3a600640 1 --2- 172.21.15.32:0/1352400706 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f4a340afa60 0x7f4a3414fc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:52.115 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:52.113+0000 7f4a3a600640 1 --2- 172.21.15.32:0/1352400706 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4a3406f110 0x7f4a34156680 unknown :-1 s=CLOSED pgs=1432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:52.115 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:52.113+0000 7f4a3a600640 1 -- 172.21.15.32:0/1352400706 >> 172.21.15.32:0/1352400706 conn(0x7f4a340cd1e0 msgr2=0x7f4a340b56b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:56:52.116 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:52.113+0000 7f4a3a600640 1 -- 172.21.15.32:0/1352400706 shutdown_connections 2024-08-03T20:56:52.116 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:52.113+0000 7f4a3a600640 1 -- 172.21.15.32:0/1352400706 wait complete. 2024-08-03T20:56:52.137 INFO:tasks.cephfs.test_damage:Result: Mutation 'Delete 1.00000000' led to DAMAGED state 2024-08-03T20:56:52.137 INFO:tasks.cephfs.test_damage:Applying mutation 'Corrupt 1.00000000.inode' 2024-08-03T20:56:52.137 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-08-03T20:56:52.137 DEBUG:teuthology.orchestra.run.smithi032:> dd if=/proc/self/mounts of=/dev/stdout 2024-08-03T20:56:52.172 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053453,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6499208k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 none /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249604k,nr_inodes=812401,mode=700,uid=1000,gid=1316 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6499208k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-08-03T20:56:52.173 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi032.front.sepia.ceph.com 2024-08-03T20:56:52.173 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-08-03T20:56:52.173 DEBUG:teuthology.orchestra.run.smithi032:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-08-03T20:56:52.248 INFO:teuthology.orchestra.run.smithi032.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.0': No such file or directory 2024-08-03T20:56:52.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-03T20:56:52.249 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-08-03T20:56:52.521 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:52.518+0000 7f593b200640 1 -- 172.21.15.32:0/3161122442 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f593c0ceef0 msgr2=0x7f593c0b72e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:56:52.521 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:52.518+0000 7f593b200640 1 --2- 172.21.15.32:0/3161122442 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f593c0ceef0 0x7f593c0b72e0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f5924009830 tx=0x7f592402a480 comp rx=0 tx=0).stop 2024-08-03T20:56:52.522 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:52.518+0000 7f593b200640 1 -- 172.21.15.32:0/3161122442 shutdown_connections 2024-08-03T20:56:52.522 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:52.518+0000 7f593b200640 1 --2- 172.21.15.32:0/3161122442 >> v2:172.21.15.118:3300/0 conn(0x7f593c0d29b0 0x7f593c0b1f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:52.522 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:52.518+0000 7f593b200640 1 --2- 172.21.15.32:0/3161122442 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f593c0ceef0 0x7f593c0b72e0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:52.522 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:52.518+0000 7f593b200640 1 -- 172.21.15.32:0/3161122442 >> 172.21.15.32:0/3161122442 conn(0x7f593c0d1e90 msgr2=0x7f593c0d22a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:56:52.522 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:52.519+0000 7f593b200640 1 -- 172.21.15.32:0/3161122442 shutdown_connections 2024-08-03T20:56:52.522 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:52.519+0000 7f593b200640 1 -- 172.21.15.32:0/3161122442 wait complete. 2024-08-03T20:56:52.526 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:52.521+0000 7f593b200640 1 Processor -- start 2024-08-03T20:56:52.526 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:52.521+0000 7f593b200640 1 -- start start 2024-08-03T20:56:52.526 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:52.522+0000 7f593b200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f593c0b1300 0x7f593c06daf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:56:52.527 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:52.522+0000 7f593b200640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f593c0d29b0 0x7f593c1f5830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:56:52.527 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:52.522+0000 7f593b200640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f593c0ca000 0x7f593c0ca410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:56:52.527 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:52.522+0000 7f593b200640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f593c0b29f0 con 0x7f593c0b1300 2024-08-03T20:56:52.527 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:52.522+0000 7f593b200640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f593c0b2870 con 0x7f593c0ca000 2024-08-03T20:56:52.527 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:52.522+0000 7f593b200640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7f593c0b2b70 con 0x7f593c0d29b0 2024-08-03T20:56:52.527 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:52.523+0000 7f5939e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f593c0b1300 0x7f593c06daf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:56:52.528 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:52.523+0000 7f5933e00640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f593c0d29b0 0x7f593c1f5830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:56:52.528 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:52.523+0000 7f593a800640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f593c0ca000 0x7f593c0ca410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:56:52.528 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:52.523+0000 7f5933e00640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f593c0d29b0 0x7f593c1f5830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3301/0 says I am v2:172.21.15.32:56576/0 (socket says 172.21.15.32:56576) 2024-08-03T20:56:52.528 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:52.523+0000 7f593a800640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f593c0ca000 0x7f593c0ca410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.32:44800/0 (socket says 172.21.15.32:44800) 2024-08-03T20:56:52.528 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:52.523+0000 7f5939e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f593c0b1300 0x7f593c06daf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:39888/0 (socket says 172.21.15.32:39888) 2024-08-03T20:56:52.528 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:52.523+0000 7f5933e00640 1 -- 172.21.15.32:0/1566174899 learned_addr learned my addr 172.21.15.32:0/1566174899 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:56:52.528 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:52.523+0000 7f5939e00640 1 -- 172.21.15.32:0/1566174899 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f593c0d29b0 msgr2=0x7f593c1f5830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:56:52.528 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:52.523+0000 7f5939e00640 1 --2- 172.21.15.32:0/1566174899 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f593c0d29b0 0x7f593c1f5830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:52.528 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:52.523+0000 7f5939e00640 1 -- 172.21.15.32:0/1566174899 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f593c0ca000 msgr2=0x7f593c0ca410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:56:52.528 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:52.523+0000 7f5939e00640 1 --2- 172.21.15.32:0/1566174899 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f593c0ca000 0x7f593c0ca410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:52.529 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:52.523+0000 7f5939e00640 1 -- 172.21.15.32:0/1566174899 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f593c0ca950 con 0x7f593c0b1300 2024-08-03T20:56:52.529 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:52.523+0000 7f593a800640 1 --2- 172.21.15.32:0/1566174899 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f593c0ca000 0x7f593c0ca410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-03T20:56:52.529 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:52.524+0000 7f5939e00640 1 --2- 172.21.15.32:0/1566174899 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f593c0b1300 0x7f593c06daf0 secure :-1 s=READY pgs=1433 cs=0 l=1 rev1=1 crypto rx=0x7f592c00c3f0 tx=0x7f592c00c7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:56:52.529 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:52.524+0000 7f5932000640 1 -- 172.21.15.32:0/1566174899 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f592c011790 con 0x7f593c0b1300 2024-08-03T20:56:52.529 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:52.524+0000 7f5932000640 1 -- 172.21.15.32:0/1566174899 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f592c017070 con 0x7f593c0b1300 2024-08-03T20:56:52.529 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:52.524+0000 7f5932000640 1 -- 172.21.15.32:0/1566174899 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f592c011ce0 con 0x7f593c0b1300 2024-08-03T20:56:52.529 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:52.524+0000 7f593b200640 1 -- 172.21.15.32:0/1566174899 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f593c06e4d0 con 0x7f593c0b1300 2024-08-03T20:56:52.529 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:52.524+0000 7f593b200640 1 -- 172.21.15.32:0/1566174899 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f593c06ea90 con 0x7f593c0b1300 2024-08-03T20:56:52.530 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:52.525+0000 7f593b200640 1 -- 172.21.15.32:0/1566174899 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f593c0b3100 con 0x7f593c0b1300 2024-08-03T20:56:52.530 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:52.526+0000 7f5932000640 1 -- 172.21.15.32:0/1566174899 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f592c01c460 con 0x7f593c0b1300 2024-08-03T20:56:52.530 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:52.527+0000 7f5932000640 1 --2- 172.21.15.32:0/1566174899 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f5918077040 0x7f5918079550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:56:52.530 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:52.527+0000 7f5932000640 1 -- 172.21.15.32:0/1566174899 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 7144+0+0 (secure 0 0 0) 0x7f592c098ab0 con 0x7f593c0b1300 2024-08-03T20:56:52.530 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:52.528+0000 7f5933e00640 1 --2- 172.21.15.32:0/1566174899 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f5918077040 0x7f5918079550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:56:52.531 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:52.529+0000 7f5933e00640 1 --2- 172.21.15.32:0/1566174899 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f5918077040 0x7f5918079550 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7f593c06f650 tx=0x7f5924004730 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:56:52.533 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:52.531+0000 7f5932000640 1 -- 172.21.15.32:0/1566174899 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7f593c0ca950 con 0x7f593c0b1300 2024-08-03T20:56:52.780 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:52.779+0000 7f593b200640 1 -- 172.21.15.32:0/1566174899 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7f593c0b0120 con 0x7f593c0b1300 2024-08-03T20:56:52.801 INFO:tasks.ceph.mds.a.smithi032.stderr: -52> 2024-08-03T20:56:48.045+0000 7f0047a00640 -1 log_channel(cluster) log [ERR] : dir 0x1 object missing on disk; some files may be lost () 2024-08-03T20:56:52.801 INFO:tasks.ceph.mds.a.smithi032.stderr: -50> 2024-08-03T20:56:48.045+0000 7f0047a00640 -1 mds.0.damage notify_dirfrag Damage to fragment * of ino 0x1 is fatal because it is a system directory for this rank 2024-08-03T20:56:52.801 INFO:tasks.ceph.mds.a.smithi032.stderr: -27> 2024-08-03T20:56:50.043+0000 7f004ca00640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-08-03T20:56:52.916 INFO:tasks.ceph.mds.a.smithi032.stdout:starting mds.a at 2024-08-03T20:56:53.064 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:53.062+0000 7f5932000640 1 -- 172.21.15.32:0/1566174899 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v181) ==== 161+0+0 (secure 0 0 0) 0x7f592c061670 con 0x7f593c0b1300 2024-08-03T20:56:53.064 INFO:teuthology.orchestra.run.smithi032.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-08-03T20:56:53.066 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:53.065+0000 7f593b200640 1 -- 172.21.15.32:0/1566174899 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f5918077040 msgr2=0x7f5918079550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:56:53.067 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:53.065+0000 7f593b200640 1 --2- 172.21.15.32:0/1566174899 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f5918077040 0x7f5918079550 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7f593c06f650 tx=0x7f5924004730 comp rx=0 tx=0).stop 2024-08-03T20:56:53.067 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:53.065+0000 7f593b200640 1 -- 172.21.15.32:0/1566174899 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f593c0b1300 msgr2=0x7f593c06daf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:56:53.067 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:53.065+0000 7f593b200640 1 --2- 172.21.15.32:0/1566174899 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f593c0b1300 0x7f593c06daf0 secure :-1 s=READY pgs=1433 cs=0 l=1 rev1=1 crypto rx=0x7f592c00c3f0 tx=0x7f592c00c7f0 comp rx=0 tx=0).stop 2024-08-03T20:56:53.067 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:53.065+0000 7f593b200640 1 -- 172.21.15.32:0/1566174899 shutdown_connections 2024-08-03T20:56:53.067 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:53.065+0000 7f593b200640 1 --2- 172.21.15.32:0/1566174899 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f5918077040 0x7f5918079550 unknown :-1 s=CLOSED pgs=587 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:53.067 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:53.065+0000 7f593b200640 1 --2- 172.21.15.32:0/1566174899 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f593c0ca000 0x7f593c0ca410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:53.067 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:53.065+0000 7f593b200640 1 --2- 172.21.15.32:0/1566174899 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f593c0d29b0 0x7f593c1f5830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:53.067 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:53.065+0000 7f593b200640 1 --2- 172.21.15.32:0/1566174899 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f593c0b1300 0x7f593c06daf0 unknown :-1 s=CLOSED pgs=1433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:53.067 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:53.065+0000 7f593b200640 1 -- 172.21.15.32:0/1566174899 >> 172.21.15.32:0/1566174899 conn(0x7f593c0d1e90 msgr2=0x7f593c0b7230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:56:53.067 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:53.065+0000 7f593b200640 1 -- 172.21.15.32:0/1566174899 shutdown_connections 2024-08-03T20:56:53.067 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:53.065+0000 7f593b200640 1 -- 172.21.15.32:0/1566174899 wait complete. 2024-08-03T20:56:53.091 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mds repaired 0 2024-08-03T20:56:53.322 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:53.320+0000 7f68e3e00640 1 -- 172.21.15.32:0/3617721507 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f68e406d9f0 msgr2=0x7f68e406de00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:56:53.322 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:53.320+0000 7f68e3e00640 1 --2- 172.21.15.32:0/3617721507 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f68e406d9f0 0x7f68e406de00 secure :-1 s=READY pgs=1438 cs=0 l=1 rev1=1 crypto rx=0x7f68e415c7b0 tx=0x7f68dc02a790 comp rx=0 tx=0).stop 2024-08-03T20:56:53.322 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:53.320+0000 7f68e3e00640 1 -- 172.21.15.32:0/3617721507 shutdown_connections 2024-08-03T20:56:53.322 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:53.320+0000 7f68e3e00640 1 --2- 172.21.15.32:0/3617721507 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f68e406d9f0 0x7f68e406de00 unknown :-1 s=CLOSED pgs=1438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:53.322 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:53.320+0000 7f68e3e00640 1 --2- 172.21.15.32:0/3617721507 >> v2:172.21.15.118:3300/0 conn(0x7f68e40cd860 0x7f68e4153270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:53.322 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:53.320+0000 7f68e3e00640 1 -- 172.21.15.32:0/3617721507 >> 172.21.15.32:0/3617721507 conn(0x7f68e4072e70 msgr2=0x7f68e4073280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:56:53.322 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:53.320+0000 7f68e3e00640 1 -- 172.21.15.32:0/3617721507 shutdown_connections 2024-08-03T20:56:53.323 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:53.320+0000 7f68e3e00640 1 -- 172.21.15.32:0/3617721507 wait complete. 2024-08-03T20:56:53.324 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:53.322+0000 7f68e3e00640 1 Processor -- start 2024-08-03T20:56:53.324 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:53.323+0000 7f68e3e00640 1 -- start start 2024-08-03T20:56:53.326 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:53.324+0000 7f68e3e00640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f68e406d9f0 0x7f68e415b950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:56:53.326 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:53.324+0000 7f68e3e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f68e4073990 0x7f68e4145ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:56:53.326 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:53.324+0000 7f68e3e00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f68e40cd860 0x7f68e40b0ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:56:53.326 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:53.324+0000 7f68e3e00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f68e40cf6b0 con 0x7f68e4073990 2024-08-03T20:56:53.326 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:53.324+0000 7f68e3e00640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f68e40cf530 con 0x7f68e40cd860 2024-08-03T20:56:53.326 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:53.324+0000 7f68e3e00640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7f68e40cf830 con 0x7f68e406d9f0 2024-08-03T20:56:53.327 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:53.324+0000 7f68da000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f68e4073990 0x7f68e4145ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:56:53.327 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:53.324+0000 7f68da000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f68e4073990 0x7f68e4145ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:39922/0 (socket says 172.21.15.32:39922) 2024-08-03T20:56:53.327 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:53.324+0000 7f68da000640 1 -- 172.21.15.32:0/3033702700 learned_addr learned my addr 172.21.15.32:0/3033702700 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:56:53.327 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:53.325+0000 7f68e3400640 1 --2- 172.21.15.32:0/3033702700 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f68e40cd860 0x7f68e40b0ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:56:53.327 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:53.325+0000 7f68e2a00640 1 --2- 172.21.15.32:0/3033702700 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f68e406d9f0 0x7f68e415b950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:56:53.327 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:53.325+0000 7f68da000640 1 -- 172.21.15.32:0/3033702700 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f68e406d9f0 msgr2=0x7f68e415b950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:56:53.327 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:53.325+0000 7f68da000640 1 --2- 172.21.15.32:0/3033702700 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f68e406d9f0 0x7f68e415b950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:53.327 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:53.325+0000 7f68da000640 1 -- 172.21.15.32:0/3033702700 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f68e40cd860 msgr2=0x7f68e40b0ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:56:53.327 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:53.325+0000 7f68da000640 1 --2- 172.21.15.32:0/3033702700 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f68e40cd860 0x7f68e40b0ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:53.327 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:53.325+0000 7f68da000640 1 -- 172.21.15.32:0/3033702700 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f68e41f9130 con 0x7f68e4073990 2024-08-03T20:56:53.328 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:53.325+0000 7f68da000640 1 --2- 172.21.15.32:0/3033702700 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f68e4073990 0x7f68e4145ef0 secure :-1 s=READY pgs=1439 cs=0 l=1 rev1=1 crypto rx=0x7f68cc00e7f0 tx=0x7f68cc00ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:56:53.328 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:53.325+0000 7f68dbe00640 1 -- 172.21.15.32:0/3033702700 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68cc00eea0 con 0x7f68e4073990 2024-08-03T20:56:53.328 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:53.325+0000 7f68dbe00640 1 -- 172.21.15.32:0/3033702700 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f68cc00ccc0 con 0x7f68e4073990 2024-08-03T20:56:53.328 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:53.325+0000 7f68dbe00640 1 -- 172.21.15.32:0/3033702700 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68cc004280 con 0x7f68e4073990 2024-08-03T20:56:53.329 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:53.325+0000 7f68e3e00640 1 -- 172.21.15.32:0/3033702700 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f68e41f9410 con 0x7f68e4073990 2024-08-03T20:56:53.329 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:53.326+0000 7f68e3e00640 1 -- 172.21.15.32:0/3033702700 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f68e41f99d0 con 0x7f68e4073990 2024-08-03T20:56:53.329 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:53.327+0000 7f68e3e00640 1 -- 172.21.15.32:0/3033702700 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f68e41f8ea0 con 0x7f68e4073990 2024-08-03T20:56:53.329 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:53.328+0000 7f68dbe00640 1 -- 172.21.15.32:0/3033702700 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f68cc0035f0 con 0x7f68e4073990 2024-08-03T20:56:53.330 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:53.329+0000 7f68dbe00640 1 --2- 172.21.15.32:0/3033702700 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f68c0077040 0x7f68c0079550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:56:53.330 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:53.329+0000 7f68dbe00640 1 -- 172.21.15.32:0/3033702700 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 7144+0+0 (secure 0 0 0) 0x7f68cc099670 con 0x7f68e4073990 2024-08-03T20:56:53.331 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:53.329+0000 7f68e2a00640 1 --2- 172.21.15.32:0/3033702700 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f68c0077040 0x7f68c0079550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:56:53.331 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:53.330+0000 7f68e2a00640 1 --2- 172.21.15.32:0/3033702700 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f68c0077040 0x7f68c0079550 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7f68d40013f0 tx=0x7f68d4009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:56:53.334 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:53.333+0000 7f68dbe00640 1 -- 172.21.15.32:0/3033702700 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7f68cc0622c0 con 0x7f68e4073990 2024-08-03T20:56:53.614 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:53.613+0000 7f68e3e00640 1 -- 172.21.15.32:0/3033702700 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "mds repaired", "role": "0"} v 0) -- 0x7f68e40cfd50 con 0x7f68e4073990 2024-08-03T20:56:54.071 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.070+0000 7f68dbe00640 1 -- 172.21.15.32:0/3033702700 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "mds repaired", "role": "0"}]=0 repaired: restoring rank 6:0 v182) ==== 101+0+0 (secure 0 0 0) 0x7f68cc061bc0 con 0x7f68e4073990 2024-08-03T20:56:54.072 INFO:teuthology.orchestra.run.smithi032.stderr:repaired: restoring rank 6:0 2024-08-03T20:56:54.074 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.073+0000 7f68e3e00640 1 -- 172.21.15.32:0/3033702700 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f68c0077040 msgr2=0x7f68c0079550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:56:54.074 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.073+0000 7f68e3e00640 1 --2- 172.21.15.32:0/3033702700 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f68c0077040 0x7f68c0079550 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7f68d40013f0 tx=0x7f68d4009040 comp rx=0 tx=0).stop 2024-08-03T20:56:54.075 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.073+0000 7f68e3e00640 1 -- 172.21.15.32:0/3033702700 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f68e4073990 msgr2=0x7f68e4145ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:56:54.075 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.073+0000 7f68e3e00640 1 --2- 172.21.15.32:0/3033702700 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f68e4073990 0x7f68e4145ef0 secure :-1 s=READY pgs=1439 cs=0 l=1 rev1=1 crypto rx=0x7f68cc00e7f0 tx=0x7f68cc00ecb0 comp rx=0 tx=0).stop 2024-08-03T20:56:54.075 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.073+0000 7f68e3e00640 1 -- 172.21.15.32:0/3033702700 shutdown_connections 2024-08-03T20:56:54.075 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.073+0000 7f68e3e00640 1 --2- 172.21.15.32:0/3033702700 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f68c0077040 0x7f68c0079550 unknown :-1 s=CLOSED pgs=589 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:54.075 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.073+0000 7f68e3e00640 1 --2- 172.21.15.32:0/3033702700 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f68e40cd860 0x7f68e40b0ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:54.075 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.073+0000 7f68e3e00640 1 --2- 172.21.15.32:0/3033702700 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f68e4073990 0x7f68e4145ef0 unknown :-1 s=CLOSED pgs=1439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:54.075 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.073+0000 7f68e3e00640 1 --2- 172.21.15.32:0/3033702700 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f68e406d9f0 0x7f68e415b950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:54.075 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.073+0000 7f68e3e00640 1 -- 172.21.15.32:0/3033702700 >> 172.21.15.32:0/3033702700 conn(0x7f68e4072e70 msgr2=0x7f68e4073280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:56:54.075 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.073+0000 7f68e3e00640 1 -- 172.21.15.32:0/3033702700 shutdown_connections 2024-08-03T20:56:54.075 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.073+0000 7f68e3e00640 1 -- 172.21.15.32:0/3033702700 wait complete. 2024-08-03T20:56:54.102 DEBUG:teuthology.orchestra.run.smithi032:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata import - 2024-08-03T20:56:54.245 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.243+0000 7fdac5e0cf40 1 Processor -- start 2024-08-03T20:56:54.245 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.243+0000 7fdac5e0cf40 1 -- start start 2024-08-03T20:56:54.247 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.244+0000 7fdac5e0cf40 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55f49016f360 0x55f49016c990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:56:54.247 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.244+0000 7fdac5e0cf40 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55f490170ea0 0x55f4900fcfa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:56:54.247 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.244+0000 7fdac5e0cf40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55f49018ccb0 0x55f490189160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:56:54.247 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.244+0000 7fdac5e0cf40 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55f49018d0c0 con 0x55f49016f360 2024-08-03T20:56:54.247 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.244+0000 7fdac5e0cf40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x55f490006e30 con 0x55f49018ccb0 2024-08-03T20:56:54.248 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.244+0000 7fdac5e0cf40 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x55f4900fda20 con 0x55f490170ea0 2024-08-03T20:56:54.248 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.245+0000 7fdabe200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55f49016f360 0x55f49016c990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:56:54.248 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.245+0000 7fdabe200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55f49016f360 0x55f49016c990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:39946/0 (socket says 172.21.15.32:39946) 2024-08-03T20:56:54.248 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.245+0000 7fdabe200640 1 -- 172.21.15.32:0/645451293 learned_addr learned my addr 172.21.15.32:0/645451293 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:56:54.248 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.245+0000 7fdabe200640 1 -- 172.21.15.32:0/645451293 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55f490170ea0 msgr2=0x55f4900fcfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:56:54.248 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.245+0000 7fdabec00640 1 --2- 172.21.15.32:0/645451293 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55f49018ccb0 0x55f490189160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:56:54.248 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.245+0000 7fdabd800640 1 --2- 172.21.15.32:0/645451293 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55f490170ea0 0x55f4900fcfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:56:54.248 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.245+0000 7fdabe200640 1 --2- 172.21.15.32:0/645451293 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55f490170ea0 0x55f4900fcfa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:54.248 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.245+0000 7fdabe200640 1 -- 172.21.15.32:0/645451293 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55f49018ccb0 msgr2=0x55f490189160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:56:54.248 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.245+0000 7fdabe200640 1 --2- 172.21.15.32:0/645451293 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55f49018ccb0 0x55f490189160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:54.248 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.245+0000 7fdabe200640 1 -- 172.21.15.32:0/645451293 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f48ffdff10 con 0x55f49016f360 2024-08-03T20:56:54.248 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.245+0000 7fdabec00640 1 --2- 172.21.15.32:0/645451293 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55f49018ccb0 0x55f490189160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-03T20:56:54.249 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.245+0000 7fdabe200640 1 --2- 172.21.15.32:0/645451293 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55f49016f360 0x55f49016c990 secure :-1 s=READY pgs=1441 cs=0 l=1 rev1=1 crypto rx=0x7fdab0025e00 tx=0x7fdab0002fb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=448bb675cec22b5f server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:56:54.249 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.246+0000 7fdab7e00640 1 -- 172.21.15.32:0/645451293 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdab004daf0 con 0x55f49016f360 2024-08-03T20:56:54.249 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.246+0000 7fdab7e00640 1 -- 172.21.15.32:0/645451293 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fdab004dc90 con 0x55f49016f360 2024-08-03T20:56:54.249 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.246+0000 7fdab7e00640 1 -- 172.21.15.32:0/645451293 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdab00566d0 con 0x55f49016f360 2024-08-03T20:56:54.250 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.246+0000 7fdac5e0cf40 1 -- 172.21.15.32:0/645451293 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55f49016f360 msgr2=0x55f49016c990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:56:54.250 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.246+0000 7fdac5e0cf40 1 --2- 172.21.15.32:0/645451293 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55f49016f360 0x55f49016c990 secure :-1 s=READY pgs=1441 cs=0 l=1 rev1=1 crypto rx=0x7fdab0025e00 tx=0x7fdab0002fb0 comp rx=0 tx=0).stop 2024-08-03T20:56:54.250 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.246+0000 7fdac5e0cf40 1 -- 172.21.15.32:0/645451293 shutdown_connections 2024-08-03T20:56:54.250 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.246+0000 7fdac5e0cf40 1 --2- 172.21.15.32:0/645451293 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55f49018ccb0 0x55f490189160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:54.250 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.246+0000 7fdac5e0cf40 1 --2- 172.21.15.32:0/645451293 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55f490170ea0 0x55f4900fcfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:54.250 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.246+0000 7fdac5e0cf40 1 --2- 172.21.15.32:0/645451293 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55f49016f360 0x55f49016c990 unknown :-1 s=CLOSED pgs=1441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:54.250 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.246+0000 7fdac5e0cf40 1 -- 172.21.15.32:0/645451293 >> 172.21.15.32:0/645451293 conn(0x55f48ffd8ac0 msgr2=0x55f4900ff910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:56:54.250 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.247+0000 7fdac5e0cf40 1 -- 172.21.15.32:0/645451293 shutdown_connections 2024-08-03T20:56:54.251 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.247+0000 7fdac5e0cf40 1 -- 172.21.15.32:0/645451293 wait complete. 2024-08-03T20:56:54.251 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.248+0000 7fdac5e0cf40 1 Processor -- start 2024-08-03T20:56:54.251 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.248+0000 7fdac5e0cf40 1 -- start start 2024-08-03T20:56:54.252 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.249+0000 7fdac5e0cf40 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55f49016f360 0x55f49016d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:56:54.252 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.249+0000 7fdac5e0cf40 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55f490170ea0 0x55f4900f7980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:56:54.253 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.249+0000 7fdac5e0cf40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55f49018ccb0 0x55f4900f9860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:56:54.253 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.249+0000 7fdac5e0cf40 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55f49016e3d0 con 0x55f490170ea0 2024-08-03T20:56:54.253 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.249+0000 7fdac5e0cf40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x55f49016e990 con 0x55f49018ccb0 2024-08-03T20:56:54.253 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.249+0000 7fdac5e0cf40 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x55f4901857b0 con 0x55f49016f360 2024-08-03T20:56:54.253 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.249+0000 7fdabd800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55f490170ea0 0x55f4900f7980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:56:54.253 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.249+0000 7fdabd800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55f490170ea0 0x55f4900f7980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:39948/0 (socket says 172.21.15.32:39948) 2024-08-03T20:56:54.253 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.249+0000 7fdabd800640 1 -- 172.21.15.32:0/828062416 learned_addr learned my addr 172.21.15.32:0/828062416 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:56:54.253 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.250+0000 7fdabe200640 1 --2- 172.21.15.32:0/828062416 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55f49016f360 0x55f49016d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:56:54.253 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.250+0000 7fdabec00640 1 --2- 172.21.15.32:0/828062416 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55f49018ccb0 0x55f4900f9860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:56:54.253 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.250+0000 7fdabd800640 1 -- 172.21.15.32:0/828062416 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55f49016f360 msgr2=0x55f49016d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:56:54.253 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.250+0000 7fdabd800640 1 --2- 172.21.15.32:0/828062416 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55f49016f360 0x55f49016d6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:54.253 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.250+0000 7fdabd800640 1 -- 172.21.15.32:0/828062416 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55f49018ccb0 msgr2=0x55f4900f9860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:56:54.253 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.250+0000 7fdabd800640 1 --2- 172.21.15.32:0/828062416 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55f49018ccb0 0x55f4900f9860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:54.253 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.250+0000 7fdabd800640 1 -- 172.21.15.32:0/828062416 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f4901868b0 con 0x55f490170ea0 2024-08-03T20:56:54.254 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.250+0000 7fdabd800640 1 --2- 172.21.15.32:0/828062416 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55f490170ea0 0x55f4900f7980 secure :-1 s=READY pgs=1442 cs=0 l=1 rev1=1 crypto rx=0x7fdaac00a6e0 tx=0x7fdaac00aba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:56:54.254 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.250+0000 7fdab6a00640 1 -- 172.21.15.32:0/828062416 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdaac0099a0 con 0x55f490170ea0 2024-08-03T20:56:54.254 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.251+0000 7fdab6a00640 1 -- 172.21.15.32:0/828062416 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fdaac009b40 con 0x55f490170ea0 2024-08-03T20:56:54.254 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.251+0000 7fdab6a00640 1 -- 172.21.15.32:0/828062416 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdaac016460 con 0x55f490170ea0 2024-08-03T20:56:54.254 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.251+0000 7fdac5e0cf40 1 -- 172.21.15.32:0/828062416 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x55f49016c8b0 con 0x55f490170ea0 2024-08-03T20:56:54.254 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.251+0000 7fdac5e0cf40 1 -- 172.21.15.32:0/828062416 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f490181490 con 0x55f490170ea0 2024-08-03T20:56:54.254 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.253+0000 7fdab6a00640 1 -- 172.21.15.32:0/828062416 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7fdaac016600 con 0x55f490170ea0 2024-08-03T20:56:54.255 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.254+0000 7fdab6a00640 1 --2- 172.21.15.32:0/828062416 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fdaa00759f0 0x7fdaa0077f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:56:54.255 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.254+0000 7fdab6a00640 1 -- 172.21.15.32:0/828062416 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 7144+0+0 (secure 0 0 0) 0x7fdaac098210 con 0x55f490170ea0 2024-08-03T20:56:54.256 INFO:teuthology.orchestra.run.smithi032.stdout:Importing pool 2024-08-03T20:56:54.257 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.254+0000 7fdac5e0cf40 1 --2- 172.21.15.32:0/828062416 >> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] conn(0x55f490160be0 0x55f490232330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:56:54.257 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.254+0000 7fdac5e0cf40 1 -- 172.21.15.32:0/828062416 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:1 12.10 12:09401a81:::601.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100) -- 0x55f490160ff0 con 0x55f490160be0 2024-08-03T20:56:54.257 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.254+0000 7fdabe200640 1 --2- 172.21.15.32:0/828062416 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fdaa00759f0 0x7fdaa0077f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:56:54.257 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.254+0000 7fdabec00640 1 --2- 172.21.15.32:0/828062416 >> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] conn(0x55f490160be0 0x55f490232330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:56:54.257 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.255+0000 7fdabec00640 1 --2- 172.21.15.32:0/828062416 >> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] conn(0x55f490160be0 0x55f490232330 crc :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:56:54.257 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.255+0000 7fdabe200640 1 --2- 172.21.15.32:0/828062416 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fdaa00759f0 0x7fdaa0077f00 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7fdab0062000 tx=0x7fdab0004940 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:56:54.258 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.257+0000 7fdabec00640 1 -- 172.21.15.32:0/828062416 <== osd.0 v2:172.21.15.32:6816/3066156116 1 ==== osd_op_reply(1 601.00000000 [create] v100'443 uv417 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fdab8003040 con 0x55f490160be0 2024-08-03T20:56:54.259 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.257+0000 7fdac5e0cf40 1 -- 172.21.15.32:0/828062416 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:2 12.10 12:09401a81:::601.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e100) -- 0x55f4900f9da0 con 0x55f490160be0 2024-08-03T20:56:54.260 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.259+0000 7fdabec00640 1 -- 172.21.15.32:0/828062416 <== osd.0 v2:172.21.15.32:6816/3066156116 2 ==== osd_op_reply(2 601.00000000 [delete] v100'444 uv444 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fdab8003040 con 0x55f490160be0 2024-08-03T20:56:54.261 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.259+0000 7fdac5e0cf40 1 -- 172.21.15.32:0/828062416 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:3 12.10 12:09401a81:::601.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100) -- 0x55f490232870 con 0x55f490160be0 2024-08-03T20:56:54.262 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.261+0000 7fdabec00640 1 -- 172.21.15.32:0/828062416 <== osd.0 v2:172.21.15.32:6816/3066156116 3 ==== osd_op_reply(3 601.00000000 [create] v100'445 uv445 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fdab8003040 con 0x55f490160be0 2024-08-03T20:56:54.262 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::601.00000000:head# 2024-08-03T20:56:54.263 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.261+0000 7fdac5e0cf40 1 -- 172.21.15.32:0/828062416 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:4 12.10 12:09401a81:::601.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100) -- 0x55f490233c20 con 0x55f490160be0 2024-08-03T20:56:54.263 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.262+0000 7fdabec00640 1 -- 172.21.15.32:0/828062416 <== osd.0 v2:172.21.15.32:6816/3066156116 4 ==== osd_op_reply(4 601.00000000 [setxattr (30)] v100'446 uv446 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fdab8003040 con 0x55f490160be0 2024-08-03T20:56:54.264 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.262+0000 7fdac5e0cf40 1 -- 172.21.15.32:0/828062416 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:5 12.10 12:09401a81:::601.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100) -- 0x55f490233fc0 con 0x55f490160be0 2024-08-03T20:56:54.265 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.264+0000 7fdabec00640 1 -- 172.21.15.32:0/828062416 <== osd.0 v2:172.21.15.32:6816/3066156116 5 ==== osd_op_reply(5 601.00000000 [setxattr (62)] v100'447 uv447 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fdab8003040 con 0x55f490160be0 2024-08-03T20:56:54.265 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.264+0000 7fdac5e0cf40 1 -- 172.21.15.32:0/828062416 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:6 12.10 12:09401a81:::601.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100) -- 0x55f490232c10 con 0x55f490160be0 2024-08-03T20:56:54.267 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.265+0000 7fdabec00640 1 -- 172.21.15.32:0/828062416 <== osd.0 v2:172.21.15.32:6816/3066156116 6 ==== osd_op_reply(6 601.00000000 [omap-set-header] v100'448 uv448 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fdab8003040 con 0x55f490160be0 2024-08-03T20:56:54.267 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.265+0000 7fdac5e0cf40 1 -- 172.21.15.32:0/828062416 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:7 12.10 12:0bd6d154:::602.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100) -- 0x55f490233000 con 0x55f490160be0 2024-08-03T20:56:54.268 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.267+0000 7fdabec00640 1 -- 172.21.15.32:0/828062416 <== osd.0 v2:172.21.15.32:6816/3066156116 7 ==== osd_op_reply(7 602.00000000 [create] v100'449 uv423 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fdab8003040 con 0x55f490160be0 2024-08-03T20:56:54.268 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.267+0000 7fdac5e0cf40 1 -- 172.21.15.32:0/828062416 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:8 12.10 12:0bd6d154:::602.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e100) -- 0x55f4902333f0 con 0x55f490160be0 2024-08-03T20:56:54.270 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.269+0000 7fdabec00640 1 -- 172.21.15.32:0/828062416 <== osd.0 v2:172.21.15.32:6816/3066156116 8 ==== osd_op_reply(8 602.00000000 [delete] v100'450 uv450 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fdab8003040 con 0x55f490160be0 2024-08-03T20:56:54.270 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.269+0000 7fdac5e0cf40 1 -- 172.21.15.32:0/828062416 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:9 12.10 12:0bd6d154:::602.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100) -- 0x55f4902333f0 con 0x55f490160be0 2024-08-03T20:56:54.272 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::602.00000000:head# 2024-08-03T20:56:54.272 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.270+0000 7fdabec00640 1 -- 172.21.15.32:0/828062416 <== osd.0 v2:172.21.15.32:6816/3066156116 9 ==== osd_op_reply(9 602.00000000 [create] v100'451 uv451 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fdab8003040 con 0x55f490160be0 2024-08-03T20:56:54.272 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.270+0000 7fdac5e0cf40 1 -- 172.21.15.32:0/828062416 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:10 12.10 12:0bd6d154:::602.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100) -- 0x55f4902333a0 con 0x55f490160be0 2024-08-03T20:56:54.273 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.272+0000 7fdabec00640 1 -- 172.21.15.32:0/828062416 <== osd.0 v2:172.21.15.32:6816/3066156116 10 ==== osd_op_reply(10 602.00000000 [setxattr (30)] v100'452 uv452 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fdab8003040 con 0x55f490160be0 2024-08-03T20:56:54.273 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.272+0000 7fdac5e0cf40 1 -- 172.21.15.32:0/828062416 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:11 12.10 12:0bd6d154:::602.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100) -- 0x55f4902333a0 con 0x55f490160be0 2024-08-03T20:56:54.274 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.273+0000 7fdabec00640 1 -- 172.21.15.32:0/828062416 <== osd.0 v2:172.21.15.32:6816/3066156116 11 ==== osd_op_reply(11 602.00000000 [setxattr (62)] v100'453 uv453 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fdab8003040 con 0x55f490160be0 2024-08-03T20:56:54.275 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.273+0000 7fdac5e0cf40 1 -- 172.21.15.32:0/828062416 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:12 12.10 12:0bd6d154:::602.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100) -- 0x55f4902333a0 con 0x55f490160be0 2024-08-03T20:56:54.276 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.275+0000 7fdabec00640 1 -- 172.21.15.32:0/828062416 <== osd.0 v2:172.21.15.32:6816/3066156116 12 ==== osd_op_reply(12 602.00000000 [omap-set-header] v100'454 uv454 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fdab8003040 con 0x55f490160be0 2024-08-03T20:56:54.276 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.275+0000 7fdac5e0cf40 1 -- 172.21.15.32:0/828062416 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:13 12.10 12:0cf6ad0f:::10000000000.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100) -- 0x55f4902333a0 con 0x55f490160be0 2024-08-03T20:56:54.278 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.276+0000 7fdabec00640 1 -- 172.21.15.32:0/828062416 <== osd.0 v2:172.21.15.32:6816/3066156116 13 ==== osd_op_reply(13 10000000000.00000000 [create] v100'455 uv430 ondisk = -17 ((17) File exists)) ==== 164+0+0 (crc 0 0 0) 0x7fdab8003040 con 0x55f490160be0 2024-08-03T20:56:54.278 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.276+0000 7fdac5e0cf40 1 -- 172.21.15.32:0/828062416 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:14 12.10 12:0cf6ad0f:::10000000000.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e100) -- 0x55f4902333a0 con 0x55f490160be0 2024-08-03T20:56:54.279 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.278+0000 7fdabec00640 1 -- 172.21.15.32:0/828062416 <== osd.0 v2:172.21.15.32:6816/3066156116 14 ==== osd_op_reply(14 10000000000.00000000 [delete] v100'456 uv456 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7fdab8003040 con 0x55f490160be0 2024-08-03T20:56:54.279 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.278+0000 7fdac5e0cf40 1 -- 172.21.15.32:0/828062416 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:15 12.10 12:0cf6ad0f:::10000000000.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100) -- 0x55f4902333a0 con 0x55f490160be0 2024-08-03T20:56:54.280 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.279+0000 7fdabec00640 1 -- 172.21.15.32:0/828062416 <== osd.0 v2:172.21.15.32:6816/3066156116 15 ==== osd_op_reply(15 10000000000.00000000 [create] v100'457 uv457 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7fdab8003040 con 0x55f490160be0 2024-08-03T20:56:54.281 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::10000000000.00000000:head# 2024-08-03T20:56:54.281 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.279+0000 7fdac5e0cf40 1 -- 172.21.15.32:0/828062416 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:16 12.10 12:0cf6ad0f:::10000000000.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100) -- 0x55f4902333a0 con 0x55f490160be0 2024-08-03T20:56:54.282 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.281+0000 7fdabec00640 1 -- 172.21.15.32:0/828062416 <== osd.0 v2:172.21.15.32:6816/3066156116 16 ==== osd_op_reply(16 10000000000.00000000 [setxattr (30)] v100'458 uv458 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7fdab8003040 con 0x55f490160be0 2024-08-03T20:56:54.282 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.281+0000 7fdac5e0cf40 1 -- 172.21.15.32:0/828062416 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:17 12.10 12:0cf6ad0f:::10000000000.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100) -- 0x55f4902333a0 con 0x55f490160be0 2024-08-03T20:56:54.284 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.282+0000 7fdabec00640 1 -- 172.21.15.32:0/828062416 <== osd.0 v2:172.21.15.32:6816/3066156116 17 ==== osd_op_reply(17 10000000000.00000000 [setxattr (62)] v100'459 uv459 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7fdab8003040 con 0x55f490160be0 2024-08-03T20:56:54.284 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.282+0000 7fdac5e0cf40 1 -- 172.21.15.32:0/828062416 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:18 12.10 12:0cf6ad0f:::10000000000.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100) -- 0x55f4902333a0 con 0x55f490160be0 2024-08-03T20:56:54.285 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.284+0000 7fdabec00640 1 -- 172.21.15.32:0/828062416 <== osd.0 v2:172.21.15.32:6816/3066156116 18 ==== osd_op_reply(18 10000000000.00000000 [omap-set-header] v100'460 uv460 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7fdab8003040 con 0x55f490160be0 2024-08-03T20:56:54.285 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.284+0000 7fdac5e0cf40 1 -- 172.21.15.32:0/828062416 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:19 12.10 12:0cf6ad0f:::10000000000.00000000:head [omap-set-vals in=593b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100) -- 0x55f4902333a0 con 0x55f490160be0 2024-08-03T20:56:54.287 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.285+0000 7fdabec00640 1 -- 172.21.15.32:0/828062416 <== osd.0 v2:172.21.15.32:6816/3066156116 19 ==== osd_op_reply(19 10000000000.00000000 [omap-set-vals] v100'461 uv461 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7fdab8003040 con 0x55f490160be0 2024-08-03T20:56:54.287 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.286+0000 7fdac5e0cf40 1 -- 172.21.15.32:0/828062416 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:20 12.10 12:0d82a743:::600.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100) -- 0x55f4902333a0 con 0x55f490160be0 2024-08-03T20:56:54.288 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.287+0000 7fdabec00640 1 -- 172.21.15.32:0/828062416 <== osd.0 v2:172.21.15.32:6816/3066156116 20 ==== osd_op_reply(20 600.00000000 [create] v100'462 uv436 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fdab8003040 con 0x55f490160be0 2024-08-03T20:56:54.288 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.287+0000 7fdac5e0cf40 1 -- 172.21.15.32:0/828062416 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:21 12.10 12:0d82a743:::600.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e100) -- 0x55f4902333a0 con 0x55f490160be0 2024-08-03T20:56:54.290 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.289+0000 7fdabec00640 1 -- 172.21.15.32:0/828062416 <== osd.0 v2:172.21.15.32:6816/3066156116 21 ==== osd_op_reply(21 600.00000000 [delete] v100'463 uv463 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fdab8003040 con 0x55f490160be0 2024-08-03T20:56:54.290 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.289+0000 7fdac5e0cf40 1 -- 172.21.15.32:0/828062416 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:22 12.10 12:0d82a743:::600.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100) -- 0x55f4902333a0 con 0x55f490160be0 2024-08-03T20:56:54.291 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::600.00000000:head# 2024-08-03T20:56:54.292 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.290+0000 7fdabec00640 1 -- 172.21.15.32:0/828062416 <== osd.0 v2:172.21.15.32:6816/3066156116 22 ==== osd_op_reply(22 600.00000000 [create] v100'464 uv464 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fdab8003040 con 0x55f490160be0 2024-08-03T20:56:54.292 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.290+0000 7fdac5e0cf40 1 -- 172.21.15.32:0/828062416 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:23 12.10 12:0d82a743:::600.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100) -- 0x55f4902333a0 con 0x55f490160be0 2024-08-03T20:56:54.293 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.292+0000 7fdabec00640 1 -- 172.21.15.32:0/828062416 <== osd.0 v2:172.21.15.32:6816/3066156116 23 ==== osd_op_reply(23 600.00000000 [setxattr (30)] v100'465 uv465 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fdab8003040 con 0x55f490160be0 2024-08-03T20:56:54.293 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.292+0000 7fdac5e0cf40 1 -- 172.21.15.32:0/828062416 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:24 12.10 12:0d82a743:::600.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100) -- 0x55f4902333a0 con 0x55f490160be0 2024-08-03T20:56:54.294 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.293+0000 7fdabec00640 1 -- 172.21.15.32:0/828062416 <== osd.0 v2:172.21.15.32:6816/3066156116 24 ==== osd_op_reply(24 600.00000000 [setxattr (62)] v100'466 uv466 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fdab8003040 con 0x55f490160be0 2024-08-03T20:56:54.295 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.293+0000 7fdac5e0cf40 1 -- 172.21.15.32:0/828062416 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:25 12.10 12:0d82a743:::600.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100) -- 0x55f4902333a0 con 0x55f490160be0 2024-08-03T20:56:54.296 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.295+0000 7fdabec00640 1 -- 172.21.15.32:0/828062416 <== osd.0 v2:172.21.15.32:6816/3066156116 25 ==== osd_op_reply(25 600.00000000 [omap-set-header] v100'467 uv467 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fdab8003040 con 0x55f490160be0 2024-08-03T20:56:54.296 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.295+0000 7fdac5e0cf40 1 -- 172.21.15.32:0/828062416 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:26 12.10 12:0d89b25e:::603.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100) -- 0x55f4902333a0 con 0x55f490160be0 2024-08-03T20:56:54.297 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.296+0000 7fdabec00640 1 -- 172.21.15.32:0/828062416 <== osd.0 v2:172.21.15.32:6816/3066156116 26 ==== osd_op_reply(26 603.00000000 [create] v100'468 uv442 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fdab8003040 con 0x55f490160be0 2024-08-03T20:56:54.298 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.296+0000 7fdac5e0cf40 1 -- 172.21.15.32:0/828062416 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:27 12.10 12:0d89b25e:::603.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e100) -- 0x55f4902333a0 con 0x55f490160be0 2024-08-03T20:56:54.299 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.298+0000 7fdabec00640 1 -- 172.21.15.32:0/828062416 <== osd.0 v2:172.21.15.32:6816/3066156116 27 ==== osd_op_reply(27 603.00000000 [delete] v100'469 uv469 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fdab8003040 con 0x55f490160be0 2024-08-03T20:56:54.299 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.298+0000 7fdac5e0cf40 1 -- 172.21.15.32:0/828062416 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:28 12.10 12:0d89b25e:::603.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100) -- 0x55f4902333a0 con 0x55f490160be0 2024-08-03T20:56:54.300 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.299+0000 7fdabec00640 1 -- 172.21.15.32:0/828062416 <== osd.0 v2:172.21.15.32:6816/3066156116 28 ==== osd_op_reply(28 603.00000000 [create] v100'470 uv470 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fdab8003040 con 0x55f490160be0 2024-08-03T20:56:54.301 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::603.00000000:head# 2024-08-03T20:56:54.301 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.299+0000 7fdac5e0cf40 1 -- 172.21.15.32:0/828062416 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:29 12.10 12:0d89b25e:::603.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100) -- 0x55f4902333a0 con 0x55f490160be0 2024-08-03T20:56:54.302 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.301+0000 7fdabec00640 1 -- 172.21.15.32:0/828062416 <== osd.0 v2:172.21.15.32:6816/3066156116 29 ==== osd_op_reply(29 603.00000000 [setxattr (30)] v100'471 uv471 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fdab8003040 con 0x55f490160be0 2024-08-03T20:56:54.302 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.301+0000 7fdac5e0cf40 1 -- 172.21.15.32:0/828062416 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:30 12.10 12:0d89b25e:::603.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100) -- 0x55f4902333a0 con 0x55f490160be0 2024-08-03T20:56:54.303 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.302+0000 7fdabec00640 1 -- 172.21.15.32:0/828062416 <== osd.0 v2:172.21.15.32:6816/3066156116 30 ==== osd_op_reply(30 603.00000000 [setxattr (62)] v100'472 uv472 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fdab8003040 con 0x55f490160be0 2024-08-03T20:56:54.303 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.302+0000 7fdac5e0cf40 1 -- 172.21.15.32:0/828062416 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:31 12.10 12:0d89b25e:::603.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100) -- 0x55f4902333a0 con 0x55f490160be0 2024-08-03T20:56:54.305 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.303+0000 7fdabec00640 1 -- 172.21.15.32:0/828062416 <== osd.0 v2:172.21.15.32:6816/3066156116 31 ==== osd_op_reply(31 603.00000000 [omap-set-header] v100'473 uv473 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fdab8003040 con 0x55f490160be0 2024-08-03T20:56:54.305 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.304+0000 7fdac5e0cf40 1 --2- 172.21.15.32:0/828062416 >> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] conn(0x55f4902337b0 0x55f490236880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:56:54.305 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.304+0000 7fdac5e0cf40 1 -- 172.21.15.32:0/828062416 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:32 12.14 12:287034c4:::1.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100) -- 0x55f4902333a0 con 0x55f4902337b0 2024-08-03T20:56:54.305 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.304+0000 7fdabe200640 1 --2- 172.21.15.32:0/828062416 >> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] conn(0x55f4902337b0 0x55f490236880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:56:54.306 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.305+0000 7fdabe200640 1 --2- 172.21.15.32:0/828062416 >> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] conn(0x55f4902337b0 0x55f490236880 crc :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:56:54.308 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.307+0000 7fdabe200640 1 -- 172.21.15.32:0/828062416 <== osd.5 v2:172.21.15.118:6810/2724891770 1 ==== osd_op_reply(32 1.00000000.inode [create] v100'330 uv312 ondisk = -17 ((17) File exists)) ==== 160+0+0 (crc 0 0 0) 0x7fdab005f030 con 0x55f4902337b0 2024-08-03T20:56:54.308 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.307+0000 7fdac5e0cf40 1 -- 172.21.15.32:0/828062416 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:33 12.14 12:287034c4:::1.00000000.inode:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e100) -- 0x55f490236dc0 con 0x55f4902337b0 2024-08-03T20:56:54.310 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.309+0000 7fdabe200640 1 -- 172.21.15.32:0/828062416 <== osd.5 v2:172.21.15.118:6810/2724891770 2 ==== osd_op_reply(33 1.00000000.inode [delete] v100'331 uv331 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7fdab005f030 con 0x55f4902337b0 2024-08-03T20:56:54.310 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.309+0000 7fdac5e0cf40 1 -- 172.21.15.32:0/828062416 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:34 12.14 12:287034c4:::1.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100) -- 0x55f490237160 con 0x55f4902337b0 2024-08-03T20:56:54.311 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::1.00000000.inode:head# 2024-08-03T20:56:54.312 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.310+0000 7fdabe200640 1 -- 172.21.15.32:0/828062416 <== osd.5 v2:172.21.15.118:6810/2724891770 3 ==== osd_op_reply(34 1.00000000.inode [create] v100'332 uv332 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7fdab005f030 con 0x55f4902337b0 2024-08-03T20:56:54.312 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.310+0000 7fdac5e0cf40 1 -- 172.21.15.32:0/828062416 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:35 12.14 12:287034c4:::1.00000000.inode:head [write 0~582 in=582b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100) -- 0x55f490237500 con 0x55f4902337b0 2024-08-03T20:56:54.313 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.312+0000 7fdabe200640 1 -- 172.21.15.32:0/828062416 <== osd.5 v2:172.21.15.118:6810/2724891770 4 ==== osd_op_reply(35 1.00000000.inode [write 0~582] v100'333 uv333 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7fdab005f030 con 0x55f4902337b0 2024-08-03T20:56:54.313 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.312+0000 7fdac5e0cf40 1 -- 172.21.15.32:0/828062416 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:36 12.14 12:287034c4:::1.00000000.inode:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100) -- 0x55f4902378a0 con 0x55f4902337b0 2024-08-03T20:56:54.315 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.314+0000 7fdabe200640 1 -- 172.21.15.32:0/828062416 <== osd.5 v2:172.21.15.118:6810/2724891770 5 ==== osd_op_reply(36 1.00000000.inode [omap-set-header] v100'334 uv334 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7fdab005f030 con 0x55f4902337b0 2024-08-03T20:56:54.315 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.314+0000 7fdac5e0cf40 1 -- 172.21.15.32:0/828062416 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:37 12.14 12:292cf221:::200.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100) -- 0x55f490237c40 con 0x55f4902337b0 2024-08-03T20:56:54.316 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.315+0000 7fdabe200640 1 -- 172.21.15.32:0/828062416 <== osd.5 v2:172.21.15.118:6810/2724891770 6 ==== osd_op_reply(37 200.00000000 [create] v100'335 uv317 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fdab005f030 con 0x55f4902337b0 2024-08-03T20:56:54.317 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.315+0000 7fdac5e0cf40 1 -- 172.21.15.32:0/828062416 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:38 12.14 12:292cf221:::200.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e100) -- 0x55f490237fe0 con 0x55f4902337b0 2024-08-03T20:56:54.318 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.317+0000 7fdabe200640 1 -- 172.21.15.32:0/828062416 <== osd.5 v2:172.21.15.118:6810/2724891770 7 ==== osd_op_reply(38 200.00000000 [delete] v100'336 uv336 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fdab005f030 con 0x55f4902337b0 2024-08-03T20:56:54.318 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.317+0000 7fdac5e0cf40 1 -- 172.21.15.32:0/828062416 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:39 12.14 12:292cf221:::200.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100) -- 0x55f490237fe0 con 0x55f4902337b0 2024-08-03T20:56:54.320 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::200.00000000:head# 2024-08-03T20:56:54.320 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.319+0000 7fdabe200640 1 -- 172.21.15.32:0/828062416 <== osd.5 v2:172.21.15.118:6810/2724891770 8 ==== osd_op_reply(39 200.00000000 [create] v100'337 uv337 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fdab005f030 con 0x55f4902337b0 2024-08-03T20:56:54.320 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.319+0000 7fdac5e0cf40 1 -- 172.21.15.32:0/828062416 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:40 12.14 12:292cf221:::200.00000000:head [write 0~90 in=90b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100) -- 0x55f490237fe0 con 0x55f4902337b0 2024-08-03T20:56:54.321 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.320+0000 7fdabe200640 1 -- 172.21.15.32:0/828062416 <== osd.5 v2:172.21.15.118:6810/2724891770 9 ==== osd_op_reply(40 200.00000000 [write 0~90] v100'338 uv338 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fdab005f030 con 0x55f4902337b0 2024-08-03T20:56:54.322 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.320+0000 7fdac5e0cf40 1 -- 172.21.15.32:0/828062416 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:41 12.14 12:292cf221:::200.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100) -- 0x55f490237fe0 con 0x55f4902337b0 2024-08-03T20:56:54.323 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.322+0000 7fdabe200640 1 -- 172.21.15.32:0/828062416 <== osd.5 v2:172.21.15.118:6810/2724891770 10 ==== osd_op_reply(41 200.00000000 [omap-set-header] v100'339 uv339 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fdab005f030 con 0x55f4902337b0 2024-08-03T20:56:54.323 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.322+0000 7fdac5e0cf40 1 -- 172.21.15.32:0/828062416 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:42 12.14 12:2e2fa760:::200.00000001:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100) -- 0x55f490237fe0 con 0x55f4902337b0 2024-08-03T20:56:54.324 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.323+0000 7fdabe200640 1 -- 172.21.15.32:0/828062416 <== osd.5 v2:172.21.15.118:6810/2724891770 11 ==== osd_op_reply(42 200.00000001 [create] v100'340 uv322 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fdab005f030 con 0x55f4902337b0 2024-08-03T20:56:54.325 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.323+0000 7fdac5e0cf40 1 -- 172.21.15.32:0/828062416 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:43 12.14 12:2e2fa760:::200.00000001:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e100) -- 0x55f490237fe0 con 0x55f4902337b0 2024-08-03T20:56:54.326 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.325+0000 7fdabe200640 1 -- 172.21.15.32:0/828062416 <== osd.5 v2:172.21.15.118:6810/2724891770 12 ==== osd_op_reply(43 200.00000001 [delete] v100'341 uv341 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fdab005f030 con 0x55f4902337b0 2024-08-03T20:56:54.326 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.325+0000 7fdac5e0cf40 1 -- 172.21.15.32:0/828062416 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:44 12.14 12:2e2fa760:::200.00000001:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100) -- 0x55f490237fe0 con 0x55f4902337b0 2024-08-03T20:56:54.328 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::200.00000001:head# 2024-08-03T20:56:54.328 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.326+0000 7fdabe200640 1 -- 172.21.15.32:0/828062416 <== osd.5 v2:172.21.15.118:6810/2724891770 13 ==== osd_op_reply(44 200.00000001 [create] v100'342 uv342 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fdab005f030 con 0x55f4902337b0 2024-08-03T20:56:54.328 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.327+0000 7fdac5e0cf40 1 -- 172.21.15.32:0/828062416 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:45 12.14 12:2e2fa760:::200.00000001:head [write 0~18118 in=18118b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100) -- 0x55f49023c6d0 con 0x55f4902337b0 2024-08-03T20:56:54.330 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.329+0000 7fdabe200640 1 -- 172.21.15.32:0/828062416 <== osd.5 v2:172.21.15.118:6810/2724891770 14 ==== osd_op_reply(45 200.00000001 [write 0~18118] v100'343 uv343 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fdab005f030 con 0x55f4902337b0 2024-08-03T20:56:54.330 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.329+0000 7fdac5e0cf40 1 -- 172.21.15.32:0/828062416 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:46 12.14 12:2e2fa760:::200.00000001:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100) -- 0x55f49023c6d0 con 0x55f4902337b0 2024-08-03T20:56:54.332 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.331+0000 7fdabe200640 1 -- 172.21.15.32:0/828062416 <== osd.5 v2:172.21.15.118:6810/2724891770 15 ==== osd_op_reply(46 200.00000001 [omap-set-header] v100'344 uv344 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fdab005f030 con 0x55f4902337b0 2024-08-03T20:56:54.332 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.331+0000 7fdac5e0cf40 1 -- 172.21.15.32:0/828062416 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:47 12.14 12:2f55791f:::606.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100) -- 0x55f490237fe0 con 0x55f4902337b0 2024-08-03T20:56:54.334 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.332+0000 7fdabe200640 1 -- 172.21.15.32:0/828062416 <== osd.5 v2:172.21.15.118:6810/2724891770 16 ==== osd_op_reply(47 606.00000000 [create] v100'345 uv328 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fdab005f030 con 0x55f4902337b0 2024-08-03T20:56:54.334 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.332+0000 7fdac5e0cf40 1 -- 172.21.15.32:0/828062416 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:48 12.14 12:2f55791f:::606.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e100) -- 0x55f490237fe0 con 0x55f4902337b0 2024-08-03T20:56:54.335 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.334+0000 7fdabe200640 1 -- 172.21.15.32:0/828062416 <== osd.5 v2:172.21.15.118:6810/2724891770 17 ==== osd_op_reply(48 606.00000000 [delete] v100'346 uv346 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fdab005f030 con 0x55f4902337b0 2024-08-03T20:56:54.335 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.334+0000 7fdac5e0cf40 1 -- 172.21.15.32:0/828062416 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:49 12.14 12:2f55791f:::606.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100) -- 0x55f490237fe0 con 0x55f4902337b0 2024-08-03T20:56:54.337 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::606.00000000:head# 2024-08-03T20:56:54.337 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.335+0000 7fdabe200640 1 -- 172.21.15.32:0/828062416 <== osd.5 v2:172.21.15.118:6810/2724891770 18 ==== osd_op_reply(49 606.00000000 [create] v100'347 uv347 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fdab005f030 con 0x55f4902337b0 2024-08-03T20:56:54.337 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.335+0000 7fdac5e0cf40 1 -- 172.21.15.32:0/828062416 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:50 12.14 12:2f55791f:::606.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100) -- 0x55f490238ff0 con 0x55f4902337b0 2024-08-03T20:56:54.339 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.337+0000 7fdabe200640 1 -- 172.21.15.32:0/828062416 <== osd.5 v2:172.21.15.118:6810/2724891770 19 ==== osd_op_reply(50 606.00000000 [setxattr (30)] v100'348 uv348 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fdab005f030 con 0x55f4902337b0 2024-08-03T20:56:54.339 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.337+0000 7fdac5e0cf40 1 -- 172.21.15.32:0/828062416 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:51 12.14 12:2f55791f:::606.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100) -- 0x55f490238ff0 con 0x55f4902337b0 2024-08-03T20:56:54.340 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.339+0000 7fdabe200640 1 -- 172.21.15.32:0/828062416 <== osd.5 v2:172.21.15.118:6810/2724891770 20 ==== osd_op_reply(51 606.00000000 [setxattr (62)] v100'349 uv349 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fdab005f030 con 0x55f4902337b0 2024-08-03T20:56:54.340 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.339+0000 7fdac5e0cf40 1 -- 172.21.15.32:0/828062416 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:52 12.14 12:2f55791f:::606.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100) -- 0x55f490237fe0 con 0x55f4902337b0 2024-08-03T20:56:54.341 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.340+0000 7fdabe200640 1 -- 172.21.15.32:0/828062416 <== osd.5 v2:172.21.15.118:6810/2724891770 21 ==== osd_op_reply(52 606.00000000 [omap-set-header] v100'350 uv350 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fdab005f030 con 0x55f4902337b0 2024-08-03T20:56:54.341 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.340+0000 7fdac5e0cf40 1 -- 172.21.15.32:0/828062416 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:53 12.1c 12:3ed09add:::607.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100) -- 0x55f490237fe0 con 0x55f490160be0 2024-08-03T20:56:54.343 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.342+0000 7fdabec00640 1 -- 172.21.15.32:0/828062416 <== osd.0 v2:172.21.15.32:6816/3066156116 32 ==== osd_op_reply(53 607.00000000 [create] v100'87 uv86 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fdab8003040 con 0x55f490160be0 2024-08-03T20:56:54.343 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.342+0000 7fdac5e0cf40 1 -- 172.21.15.32:0/828062416 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:54 12.1c 12:3ed09add:::607.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e100) -- 0x55f490237fe0 con 0x55f490160be0 2024-08-03T20:56:54.344 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.343+0000 7fdabec00640 1 -- 172.21.15.32:0/828062416 <== osd.0 v2:172.21.15.32:6816/3066156116 33 ==== osd_op_reply(54 607.00000000 [delete] v100'88 uv88 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fdab8003040 con 0x55f490160be0 2024-08-03T20:56:54.345 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.343+0000 7fdac5e0cf40 1 -- 172.21.15.32:0/828062416 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:55 12.1c 12:3ed09add:::607.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100) -- 0x55f490237fe0 con 0x55f490160be0 2024-08-03T20:56:54.346 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::607.00000000:head# 2024-08-03T20:56:54.346 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.344+0000 7fdabec00640 1 -- 172.21.15.32:0/828062416 <== osd.0 v2:172.21.15.32:6816/3066156116 34 ==== osd_op_reply(55 607.00000000 [create] v100'89 uv89 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fdab8003040 con 0x55f490160be0 2024-08-03T20:56:54.346 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.344+0000 7fdac5e0cf40 1 -- 172.21.15.32:0/828062416 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:56 12.1c 12:3ed09add:::607.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100) -- 0x55f490238ff0 con 0x55f490160be0 2024-08-03T20:56:54.347 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.346+0000 7fdabec00640 1 -- 172.21.15.32:0/828062416 <== osd.0 v2:172.21.15.32:6816/3066156116 35 ==== osd_op_reply(56 607.00000000 [setxattr (30)] v100'90 uv90 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fdab8003040 con 0x55f490160be0 2024-08-03T20:56:54.347 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.346+0000 7fdac5e0cf40 1 -- 172.21.15.32:0/828062416 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:57 12.1c 12:3ed09add:::607.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100) -- 0x55f490238ff0 con 0x55f490160be0 2024-08-03T20:56:54.348 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.347+0000 7fdabec00640 1 -- 172.21.15.32:0/828062416 <== osd.0 v2:172.21.15.32:6816/3066156116 36 ==== osd_op_reply(57 607.00000000 [setxattr (62)] v100'91 uv91 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fdab8003040 con 0x55f490160be0 2024-08-03T20:56:54.349 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.347+0000 7fdac5e0cf40 1 -- 172.21.15.32:0/828062416 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:58 12.1c 12:3ed09add:::607.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100) -- 0x55f490237fe0 con 0x55f490160be0 2024-08-03T20:56:54.350 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.349+0000 7fdabec00640 1 -- 172.21.15.32:0/828062416 <== osd.0 v2:172.21.15.32:6816/3066156116 37 ==== osd_op_reply(58 607.00000000 [omap-set-header] v100'92 uv92 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fdab8003040 con 0x55f490160be0 2024-08-03T20:56:54.350 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.349+0000 7fdac5e0cf40 1 --2- 172.21.15.32:0/828062416 >> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] conn(0x55f490238660 0x55f49023aa80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:56:54.350 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.349+0000 7fdac5e0cf40 1 -- 172.21.15.32:0/828062416 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:59 12.6 12:654134d2:::mds0_openfiles.0:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100) -- 0x55f49023b0c0 con 0x55f490238660 2024-08-03T20:56:54.350 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.349+0000 7fdabd800640 1 --2- 172.21.15.32:0/828062416 >> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] conn(0x55f490238660 0x55f49023aa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:56:54.351 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.349+0000 7fdabd800640 1 --2- 172.21.15.32:0/828062416 >> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] conn(0x55f490238660 0x55f49023aa80 crc :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:56:54.352 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.351+0000 7fdabd800640 1 -- 172.21.15.32:0/828062416 <== osd.1 v2:172.21.15.32:6824/1551241759 1 ==== osd_op_reply(59 mds0_openfiles.0 [create] v100'74 uv73 ondisk = -17 ((17) File exists)) ==== 160+0+0 (crc 0 0 0) 0x7fdaac05cb30 con 0x55f490238660 2024-08-03T20:56:54.352 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.351+0000 7fdac5e0cf40 1 -- 172.21.15.32:0/828062416 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:60 12.6 12:654134d2:::mds0_openfiles.0:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e100) -- 0x55f49023b5d0 con 0x55f490238660 2024-08-03T20:56:54.353 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.352+0000 7fdabd800640 1 -- 172.21.15.32:0/828062416 <== osd.1 v2:172.21.15.32:6824/1551241759 2 ==== osd_op_reply(60 mds0_openfiles.0 [delete] v100'75 uv75 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7fdaac05cb30 con 0x55f490238660 2024-08-03T20:56:54.353 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.352+0000 7fdac5e0cf40 1 -- 172.21.15.32:0/828062416 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:61 12.6 12:654134d2:::mds0_openfiles.0:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100) -- 0x55f49023bad0 con 0x55f490238660 2024-08-03T20:56:54.354 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.353+0000 7fdabd800640 1 -- 172.21.15.32:0/828062416 <== osd.1 v2:172.21.15.32:6824/1551241759 3 ==== osd_op_reply(61 mds0_openfiles.0 [create] v100'76 uv76 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7fdaac05cb30 con 0x55f490238660 2024-08-03T20:56:54.355 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds0_openfiles.0:head# 2024-08-03T20:56:54.355 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.353+0000 7fdac5e0cf40 1 -- 172.21.15.32:0/828062416 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:62 12.6 12:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100) -- 0x55f490185070 con 0x55f490238660 2024-08-03T20:56:54.356 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.354+0000 7fdabd800640 1 -- 172.21.15.32:0/828062416 <== osd.1 v2:172.21.15.32:6824/1551241759 4 ==== osd_op_reply(62 mds0_openfiles.0 [omap-set-header] v100'77 uv77 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7fdaac05cb30 con 0x55f490238660 2024-08-03T20:56:54.356 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.354+0000 7fdac5e0cf40 1 -- 172.21.15.32:0/828062416 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:63 12.e 12:717a0223:::608.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100) -- 0x55f49023c400 con 0x55f490160be0 2024-08-03T20:56:54.358 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.356+0000 7fdabec00640 1 -- 172.21.15.32:0/828062416 <== osd.0 v2:172.21.15.32:6816/3066156116 38 ==== osd_op_reply(63 608.00000000 [create] v100'87 uv86 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fdab8003040 con 0x55f490160be0 2024-08-03T20:56:54.358 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.356+0000 7fdac5e0cf40 1 -- 172.21.15.32:0/828062416 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:64 12.e 12:717a0223:::608.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e100) -- 0x55f49023c400 con 0x55f490160be0 2024-08-03T20:56:54.359 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.358+0000 7fdabec00640 1 -- 172.21.15.32:0/828062416 <== osd.0 v2:172.21.15.32:6816/3066156116 39 ==== osd_op_reply(64 608.00000000 [delete] v100'88 uv88 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fdab8003040 con 0x55f490160be0 2024-08-03T20:56:54.359 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.358+0000 7fdac5e0cf40 1 -- 172.21.15.32:0/828062416 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:65 12.e 12:717a0223:::608.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100) -- 0x55f49023c400 con 0x55f490160be0 2024-08-03T20:56:54.361 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::608.00000000:head# 2024-08-03T20:56:54.361 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.359+0000 7fdabec00640 1 -- 172.21.15.32:0/828062416 <== osd.0 v2:172.21.15.32:6816/3066156116 40 ==== osd_op_reply(65 608.00000000 [create] v100'89 uv89 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fdab8003040 con 0x55f490160be0 2024-08-03T20:56:54.361 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.359+0000 7fdac5e0cf40 1 -- 172.21.15.32:0/828062416 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:66 12.e 12:717a0223:::608.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100) -- 0x55f49023c400 con 0x55f490160be0 2024-08-03T20:56:54.362 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.361+0000 7fdabec00640 1 -- 172.21.15.32:0/828062416 <== osd.0 v2:172.21.15.32:6816/3066156116 41 ==== osd_op_reply(66 608.00000000 [setxattr (30)] v100'90 uv90 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fdab8003040 con 0x55f490160be0 2024-08-03T20:56:54.362 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.361+0000 7fdac5e0cf40 1 -- 172.21.15.32:0/828062416 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:67 12.e 12:717a0223:::608.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100) -- 0x55f49023c400 con 0x55f490160be0 2024-08-03T20:56:54.364 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.362+0000 7fdabec00640 1 -- 172.21.15.32:0/828062416 <== osd.0 v2:172.21.15.32:6816/3066156116 42 ==== osd_op_reply(67 608.00000000 [setxattr (62)] v100'91 uv91 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fdab8003040 con 0x55f490160be0 2024-08-03T20:56:54.364 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.362+0000 7fdac5e0cf40 1 -- 172.21.15.32:0/828062416 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:68 12.e 12:717a0223:::608.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100) -- 0x55f49023c400 con 0x55f490160be0 2024-08-03T20:56:54.365 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.364+0000 7fdabec00640 1 -- 172.21.15.32:0/828062416 <== osd.0 v2:172.21.15.32:6816/3066156116 43 ==== osd_op_reply(68 608.00000000 [omap-set-header] v100'92 uv92 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fdab8003040 con 0x55f490160be0 2024-08-03T20:56:54.365 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.364+0000 7fdac5e0cf40 1 --2- 172.21.15.32:0/828062416 >> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] conn(0x55f49023cd00 0x55f49023f120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:56:54.366 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.364+0000 7fdac5e0cf40 1 -- 172.21.15.32:0/828062416 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:69 12.15 12:a93a17c2:::604.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100) -- 0x55f49023f660 con 0x55f49023cd00 2024-08-03T20:56:54.366 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.364+0000 7fdabec00640 1 --2- 172.21.15.32:0/828062416 >> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] conn(0x55f49023cd00 0x55f49023f120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:56:54.366 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.365+0000 7fdabec00640 1 --2- 172.21.15.32:0/828062416 >> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] conn(0x55f49023cd00 0x55f49023f120 crc :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:56:54.368 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.366+0000 7fdabec00640 1 -- 172.21.15.32:0/828062416 <== osd.6 v2:172.21.15.118:6825/801947386 1 ==== osd_op_reply(69 604.00000000 [create] v100'160 uv154 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fdab8003040 con 0x55f49023cd00 2024-08-03T20:56:54.368 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.367+0000 7fdac5e0cf40 1 -- 172.21.15.32:0/828062416 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:70 12.15 12:a93a17c2:::604.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e100) -- 0x55f49023f660 con 0x55f49023cd00 2024-08-03T20:56:54.369 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.368+0000 7fdabec00640 1 -- 172.21.15.32:0/828062416 <== osd.6 v2:172.21.15.118:6825/801947386 2 ==== osd_op_reply(70 604.00000000 [delete] v100'161 uv161 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fdab8003040 con 0x55f49023cd00 2024-08-03T20:56:54.369 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.368+0000 7fdac5e0cf40 1 -- 172.21.15.32:0/828062416 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:71 12.15 12:a93a17c2:::604.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100) -- 0x55f49023fa80 con 0x55f49023cd00 2024-08-03T20:56:54.370 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.369+0000 7fdabec00640 1 -- 172.21.15.32:0/828062416 <== osd.6 v2:172.21.15.118:6825/801947386 3 ==== osd_op_reply(71 604.00000000 [create] v100'162 uv162 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fdab8003040 con 0x55f49023cd00 2024-08-03T20:56:54.371 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::604.00000000:head# 2024-08-03T20:56:54.371 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.369+0000 7fdac5e0cf40 1 -- 172.21.15.32:0/828062416 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:72 12.15 12:a93a17c2:::604.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100) -- 0x55f490240a90 con 0x55f49023cd00 2024-08-03T20:56:54.372 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.370+0000 7fdabec00640 1 -- 172.21.15.32:0/828062416 <== osd.6 v2:172.21.15.118:6825/801947386 4 ==== osd_op_reply(72 604.00000000 [setxattr (30)] v100'163 uv163 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fdab8003040 con 0x55f49023cd00 2024-08-03T20:56:54.372 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.370+0000 7fdac5e0cf40 1 -- 172.21.15.32:0/828062416 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:73 12.15 12:a93a17c2:::604.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100) -- 0x55f490240a90 con 0x55f49023cd00 2024-08-03T20:56:54.373 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.372+0000 7fdabec00640 1 -- 172.21.15.32:0/828062416 <== osd.6 v2:172.21.15.118:6825/801947386 5 ==== osd_op_reply(73 604.00000000 [setxattr (62)] v100'164 uv164 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fdab8003040 con 0x55f49023cd00 2024-08-03T20:56:54.373 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.372+0000 7fdac5e0cf40 1 -- 172.21.15.32:0/828062416 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:74 12.15 12:a93a17c2:::604.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100) -- 0x55f49023fa80 con 0x55f49023cd00 2024-08-03T20:56:54.374 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.373+0000 7fdabec00640 1 -- 172.21.15.32:0/828062416 <== osd.6 v2:172.21.15.118:6825/801947386 6 ==== osd_op_reply(74 604.00000000 [omap-set-header] v100'165 uv165 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fdab8003040 con 0x55f49023cd00 2024-08-03T20:56:54.374 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.373+0000 7fdac5e0cf40 1 -- 172.21.15.32:0/828062416 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:75 12.15 12:aa448500:::500.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100) -- 0x55f49023fa80 con 0x55f49023cd00 2024-08-03T20:56:54.376 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.375+0000 7fdabec00640 1 -- 172.21.15.32:0/828062416 <== osd.6 v2:172.21.15.118:6825/801947386 7 ==== osd_op_reply(75 500.00000000 [create] v100'166 uv159 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fdab8003040 con 0x55f49023cd00 2024-08-03T20:56:54.376 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.375+0000 7fdac5e0cf40 1 -- 172.21.15.32:0/828062416 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:76 12.15 12:aa448500:::500.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e100) -- 0x55f49023fa80 con 0x55f49023cd00 2024-08-03T20:56:54.377 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.376+0000 7fdabec00640 1 -- 172.21.15.32:0/828062416 <== osd.6 v2:172.21.15.118:6825/801947386 8 ==== osd_op_reply(76 500.00000000 [delete] v100'167 uv167 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fdab8003040 con 0x55f49023cd00 2024-08-03T20:56:54.377 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.376+0000 7fdac5e0cf40 1 -- 172.21.15.32:0/828062416 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:77 12.15 12:aa448500:::500.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100) -- 0x55f49023fa80 con 0x55f49023cd00 2024-08-03T20:56:54.378 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.377+0000 7fdabec00640 1 -- 172.21.15.32:0/828062416 <== osd.6 v2:172.21.15.118:6825/801947386 9 ==== osd_op_reply(77 500.00000000 [create] v100'168 uv168 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fdab8003040 con 0x55f49023cd00 2024-08-03T20:56:54.379 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::500.00000000:head# 2024-08-03T20:56:54.379 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.377+0000 7fdac5e0cf40 1 -- 172.21.15.32:0/828062416 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:78 12.15 12:aa448500:::500.00000000:head [write 0~90 in=90b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100) -- 0x55f49023fa80 con 0x55f49023cd00 2024-08-03T20:56:54.380 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.379+0000 7fdabec00640 1 -- 172.21.15.32:0/828062416 <== osd.6 v2:172.21.15.118:6825/801947386 10 ==== osd_op_reply(78 500.00000000 [write 0~90] v100'169 uv169 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fdab8003040 con 0x55f49023cd00 2024-08-03T20:56:54.380 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.379+0000 7fdac5e0cf40 1 -- 172.21.15.32:0/828062416 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:79 12.15 12:aa448500:::500.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100) -- 0x55f49023fa80 con 0x55f49023cd00 2024-08-03T20:56:54.381 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.380+0000 7fdabec00640 1 -- 172.21.15.32:0/828062416 <== osd.6 v2:172.21.15.118:6825/801947386 11 ==== osd_op_reply(79 500.00000000 [omap-set-header] v100'170 uv170 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fdab8003040 con 0x55f49023cd00 2024-08-03T20:56:54.381 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.380+0000 7fdac5e0cf40 1 --2- 172.21.15.32:0/828062416 >> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] conn(0x55f490240100 0x55f490242520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:56:54.382 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.380+0000 7fdac5e0cf40 1 -- 172.21.15.32:0/828062416 --> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] -- osd_op(unknown.0.0:80 12.d 12:b50e409b:::mds_snaptable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100) -- 0x55f490242b80 con 0x55f490240100 2024-08-03T20:56:54.382 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.380+0000 7fdabe200640 1 --2- 172.21.15.32:0/828062416 >> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] conn(0x55f490240100 0x55f490242520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:56:54.382 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.381+0000 7fdabe200640 1 --2- 172.21.15.32:0/828062416 >> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] conn(0x55f490240100 0x55f490242520 crc :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:56:54.384 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.382+0000 7fdabe200640 1 -- 172.21.15.32:0/828062416 <== osd.7 v2:172.21.15.118:6818/653000048 1 ==== osd_op_reply(80 mds_snaptable [create] v100'71 uv70 ondisk = -17 ((17) File exists)) ==== 157+0+0 (crc 0 0 0) 0x7fdab005f030 con 0x55f490240100 2024-08-03T20:56:54.384 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.382+0000 7fdac5e0cf40 1 -- 172.21.15.32:0/828062416 --> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] -- osd_op(unknown.0.0:81 12.d 12:b50e409b:::mds_snaptable:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e100) -- 0x55f490242f50 con 0x55f490240100 2024-08-03T20:56:54.385 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.384+0000 7fdabe200640 1 -- 172.21.15.32:0/828062416 <== osd.7 v2:172.21.15.118:6818/653000048 2 ==== osd_op_reply(81 mds_snaptable [delete] v100'72 uv72 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7fdab005f030 con 0x55f490240100 2024-08-03T20:56:54.385 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.384+0000 7fdac5e0cf40 1 -- 172.21.15.32:0/828062416 --> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] -- osd_op(unknown.0.0:82 12.d 12:b50e409b:::mds_snaptable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100) -- 0x55f490242f50 con 0x55f490240100 2024-08-03T20:56:54.386 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.385+0000 7fdabe200640 1 -- 172.21.15.32:0/828062416 <== osd.7 v2:172.21.15.118:6818/653000048 3 ==== osd_op_reply(82 mds_snaptable [create] v100'73 uv73 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7fdab005f030 con 0x55f490240100 2024-08-03T20:56:54.386 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds_snaptable:head# 2024-08-03T20:56:54.387 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.385+0000 7fdac5e0cf40 1 -- 172.21.15.32:0/828062416 --> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] -- osd_op(unknown.0.0:83 12.d 12:b50e409b:::mds_snaptable:head [write 0~70 in=70b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100) -- 0x55f490242f50 con 0x55f490240100 2024-08-03T20:56:54.388 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.387+0000 7fdabe200640 1 -- 172.21.15.32:0/828062416 <== osd.7 v2:172.21.15.118:6818/653000048 4 ==== osd_op_reply(83 mds_snaptable [write 0~70] v100'74 uv74 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7fdab005f030 con 0x55f490240100 2024-08-03T20:56:54.388 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.387+0000 7fdac5e0cf40 1 -- 172.21.15.32:0/828062416 --> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] -- osd_op(unknown.0.0:84 12.d 12:b50e409b:::mds_snaptable:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100) -- 0x55f490242f50 con 0x55f490240100 2024-08-03T20:56:54.389 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.388+0000 7fdabe200640 1 -- 172.21.15.32:0/828062416 <== osd.7 v2:172.21.15.118:6818/653000048 5 ==== osd_op_reply(84 mds_snaptable [omap-set-header] v100'75 uv75 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7fdab005f030 con 0x55f490240100 2024-08-03T20:56:54.390 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.388+0000 7fdac5e0cf40 1 -- 172.21.15.32:0/828062416 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:85 12.1d 12:b871830b:::605.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100) -- 0x55f490242f50 con 0x55f490160be0 2024-08-03T20:56:54.391 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.390+0000 7fdabec00640 1 -- 172.21.15.32:0/828062416 <== osd.0 v2:172.21.15.32:6816/3066156116 44 ==== osd_op_reply(85 605.00000000 [create] v100'87 uv86 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fdab8003040 con 0x55f490160be0 2024-08-03T20:56:54.391 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.390+0000 7fdac5e0cf40 1 -- 172.21.15.32:0/828062416 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:86 12.1d 12:b871830b:::605.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e100) -- 0x55f490242f50 con 0x55f490160be0 2024-08-03T20:56:54.392 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.391+0000 7fdabec00640 1 -- 172.21.15.32:0/828062416 <== osd.0 v2:172.21.15.32:6816/3066156116 45 ==== osd_op_reply(86 605.00000000 [delete] v100'88 uv88 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fdab8003040 con 0x55f490160be0 2024-08-03T20:56:54.393 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.391+0000 7fdac5e0cf40 1 -- 172.21.15.32:0/828062416 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:87 12.1d 12:b871830b:::605.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100) -- 0x55f490242f50 con 0x55f490160be0 2024-08-03T20:56:54.394 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::605.00000000:head# 2024-08-03T20:56:54.394 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.392+0000 7fdabec00640 1 -- 172.21.15.32:0/828062416 <== osd.0 v2:172.21.15.32:6816/3066156116 46 ==== osd_op_reply(87 605.00000000 [create] v100'89 uv89 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fdab8003040 con 0x55f490160be0 2024-08-03T20:56:54.394 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.392+0000 7fdac5e0cf40 1 -- 172.21.15.32:0/828062416 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:88 12.1d 12:b871830b:::605.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100) -- 0x55f490243fa0 con 0x55f490160be0 2024-08-03T20:56:54.395 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.394+0000 7fdabec00640 1 -- 172.21.15.32:0/828062416 <== osd.0 v2:172.21.15.32:6816/3066156116 47 ==== osd_op_reply(88 605.00000000 [setxattr (30)] v100'90 uv90 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fdab8003040 con 0x55f490160be0 2024-08-03T20:56:54.395 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.394+0000 7fdac5e0cf40 1 -- 172.21.15.32:0/828062416 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:89 12.1d 12:b871830b:::605.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100) -- 0x55f490243f60 con 0x55f490160be0 2024-08-03T20:56:54.397 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.395+0000 7fdabec00640 1 -- 172.21.15.32:0/828062416 <== osd.0 v2:172.21.15.32:6816/3066156116 48 ==== osd_op_reply(89 605.00000000 [setxattr (62)] v100'91 uv91 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fdab8003040 con 0x55f490160be0 2024-08-03T20:56:54.397 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.395+0000 7fdac5e0cf40 1 -- 172.21.15.32:0/828062416 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:90 12.1d 12:b871830b:::605.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100) -- 0x55f490242f50 con 0x55f490160be0 2024-08-03T20:56:54.398 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.397+0000 7fdabec00640 1 -- 172.21.15.32:0/828062416 <== osd.0 v2:172.21.15.32:6816/3066156116 49 ==== osd_op_reply(90 605.00000000 [omap-set-header] v100'92 uv92 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fdab8003040 con 0x55f490160be0 2024-08-03T20:56:54.398 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.397+0000 7fdac5e0cf40 1 -- 172.21.15.32:0/828062416 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:91 12.13 12:c91d4a1d:::mds0_inotable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100) -- 0x55f490242f50 con 0x55f490238660 2024-08-03T20:56:54.399 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.398+0000 7fdabd800640 1 -- 172.21.15.32:0/828062416 <== osd.1 v2:172.21.15.32:6824/1551241759 5 ==== osd_op_reply(91 mds0_inotable [create] v100'142 uv136 ondisk = -17 ((17) File exists)) ==== 157+0+0 (crc 0 0 0) 0x7fdaac05cb30 con 0x55f490238660 2024-08-03T20:56:54.399 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.398+0000 7fdac5e0cf40 1 -- 172.21.15.32:0/828062416 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:92 12.13 12:c91d4a1d:::mds0_inotable:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e100) -- 0x55f4902432f0 con 0x55f490238660 2024-08-03T20:56:54.401 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.399+0000 7fdabd800640 1 -- 172.21.15.32:0/828062416 <== osd.1 v2:172.21.15.32:6824/1551241759 6 ==== osd_op_reply(92 mds0_inotable [delete] v100'143 uv143 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7fdaac05cb30 con 0x55f490238660 2024-08-03T20:56:54.401 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.399+0000 7fdac5e0cf40 1 -- 172.21.15.32:0/828062416 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:93 12.13 12:c91d4a1d:::mds0_inotable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100) -- 0x55f4902436e0 con 0x55f490238660 2024-08-03T20:56:54.402 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds0_inotable:head# 2024-08-03T20:56:54.402 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.401+0000 7fdabd800640 1 -- 172.21.15.32:0/828062416 <== osd.1 v2:172.21.15.32:6824/1551241759 7 ==== osd_op_reply(93 mds0_inotable [create] v100'144 uv144 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7fdaac05cb30 con 0x55f490238660 2024-08-03T20:56:54.402 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.401+0000 7fdac5e0cf40 1 -- 172.21.15.32:0/828062416 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:94 12.13 12:c91d4a1d:::mds0_inotable:head [write 0~34 in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100) -- 0x55f490243f80 con 0x55f490238660 2024-08-03T20:56:54.403 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.402+0000 7fdabd800640 1 -- 172.21.15.32:0/828062416 <== osd.1 v2:172.21.15.32:6824/1551241759 8 ==== osd_op_reply(94 mds0_inotable [write 0~34] v100'145 uv145 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7fdaac05cb30 con 0x55f490238660 2024-08-03T20:56:54.403 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.402+0000 7fdac5e0cf40 1 -- 172.21.15.32:0/828062416 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:95 12.13 12:c91d4a1d:::mds0_inotable:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100) -- 0x55f490243f80 con 0x55f490238660 2024-08-03T20:56:54.404 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.403+0000 7fdabd800640 1 -- 172.21.15.32:0/828062416 <== osd.1 v2:172.21.15.32:6824/1551241759 9 ==== osd_op_reply(95 mds0_inotable [omap-set-header] v100'146 uv146 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7fdaac05cb30 con 0x55f490238660 2024-08-03T20:56:54.404 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.403+0000 7fdac5e0cf40 1 -- 172.21.15.32:0/828062416 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:96 12.13 12:cd5a64a3:::100.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100) -- 0x55f49023bfd0 con 0x55f490238660 2024-08-03T20:56:54.406 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.404+0000 7fdabd800640 1 -- 172.21.15.32:0/828062416 <== osd.1 v2:172.21.15.32:6824/1551241759 10 ==== osd_op_reply(96 100.00000000 [create] v100'147 uv141 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fdaac05cb30 con 0x55f490238660 2024-08-03T20:56:54.406 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.404+0000 7fdac5e0cf40 1 -- 172.21.15.32:0/828062416 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:97 12.13 12:cd5a64a3:::100.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e100) -- 0x55f49023bf80 con 0x55f490238660 2024-08-03T20:56:54.407 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.406+0000 7fdabd800640 1 -- 172.21.15.32:0/828062416 <== osd.1 v2:172.21.15.32:6824/1551241759 11 ==== osd_op_reply(97 100.00000000 [delete] v100'148 uv148 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fdaac05cb30 con 0x55f490238660 2024-08-03T20:56:54.407 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.406+0000 7fdac5e0cf40 1 -- 172.21.15.32:0/828062416 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:98 12.13 12:cd5a64a3:::100.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100) -- 0x55f49023bf80 con 0x55f490238660 2024-08-03T20:56:54.408 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::100.00000000:head# 2024-08-03T20:56:54.409 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.407+0000 7fdabd800640 1 -- 172.21.15.32:0/828062416 <== osd.1 v2:172.21.15.32:6824/1551241759 12 ==== osd_op_reply(98 100.00000000 [create] v100'149 uv149 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fdaac05cb30 con 0x55f490238660 2024-08-03T20:56:54.409 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.407+0000 7fdac5e0cf40 1 -- 172.21.15.32:0/828062416 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:99 12.13 12:cd5a64a3:::100.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100) -- 0x55f49023bf80 con 0x55f490238660 2024-08-03T20:56:54.410 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.408+0000 7fdabd800640 1 -- 172.21.15.32:0/828062416 <== osd.1 v2:172.21.15.32:6824/1551241759 13 ==== osd_op_reply(99 100.00000000 [omap-set-header] v100'150 uv150 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fdaac05cb30 con 0x55f490238660 2024-08-03T20:56:54.410 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.409+0000 7fdac5e0cf40 1 -- 172.21.15.32:0/828062416 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:100 12.13 12:cd5a64a3:::100.00000000:head [omap-set-vals in=5194b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100) -- 0x55f490243fb0 con 0x55f490238660 2024-08-03T20:56:54.412 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.410+0000 7fdabd800640 1 -- 172.21.15.32:0/828062416 <== osd.1 v2:172.21.15.32:6824/1551241759 14 ==== osd_op_reply(100 100.00000000 [omap-set-vals] v100'151 uv151 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fdaac05cb30 con 0x55f490238660 2024-08-03T20:56:54.412 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.410+0000 7fdac5e0cf40 1 --2- 172.21.15.32:0/828062416 >> [v2:172.21.15.32:6800/9088303,v1:172.21.15.32:6801/9088303] conn(0x55f490244ad0 0x55f4902470d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:56:54.412 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.410+0000 7fdac5e0cf40 1 -- 172.21.15.32:0/828062416 --> [v2:172.21.15.32:6800/9088303,v1:172.21.15.32:6801/9088303] -- osd_op(unknown.0.0:101 12.b 12:d0630e4c:::mds0_sessionmap:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100) -- 0x55f490247610 con 0x55f490244ad0 2024-08-03T20:56:54.412 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.410+0000 7fdabd800640 1 --2- 172.21.15.32:0/828062416 >> [v2:172.21.15.32:6800/9088303,v1:172.21.15.32:6801/9088303] conn(0x55f490244ad0 0x55f4902470d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:56:54.412 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.411+0000 7fdabd800640 1 --2- 172.21.15.32:0/828062416 >> [v2:172.21.15.32:6800/9088303,v1:172.21.15.32:6801/9088303] conn(0x55f490244ad0 0x55f4902470d0 crc :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:56:54.414 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.412+0000 7fdabd800640 1 -- 172.21.15.32:0/828062416 <== osd.3 v2:172.21.15.32:6800/9088303 1 ==== osd_op_reply(101 mds0_sessionmap [create] v100'66 uv64 ondisk = -17 ((17) File exists)) ==== 159+0+0 (crc 0 0 0) 0x7fdaac05cb30 con 0x55f490244ad0 2024-08-03T20:56:54.414 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.412+0000 7fdac5e0cf40 1 -- 172.21.15.32:0/828062416 --> [v2:172.21.15.32:6800/9088303,v1:172.21.15.32:6801/9088303] -- osd_op(unknown.0.0:102 12.b 12:d0630e4c:::mds0_sessionmap:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e100) -- 0x55f490247610 con 0x55f490244ad0 2024-08-03T20:56:54.415 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.414+0000 7fdabd800640 1 -- 172.21.15.32:0/828062416 <== osd.3 v2:172.21.15.32:6800/9088303 2 ==== osd_op_reply(102 mds0_sessionmap [delete] v100'67 uv67 ondisk = 0) ==== 159+0+0 (crc 0 0 0) 0x7fdaac05cb30 con 0x55f490244ad0 2024-08-03T20:56:54.415 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.414+0000 7fdac5e0cf40 1 -- 172.21.15.32:0/828062416 --> [v2:172.21.15.32:6800/9088303,v1:172.21.15.32:6801/9088303] -- osd_op(unknown.0.0:103 12.b 12:d0630e4c:::mds0_sessionmap:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100) -- 0x55f490247610 con 0x55f490244ad0 2024-08-03T20:56:54.417 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds0_sessionmap:head# 2024-08-03T20:56:54.417 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.415+0000 7fdabd800640 1 -- 172.21.15.32:0/828062416 <== osd.3 v2:172.21.15.32:6800/9088303 3 ==== osd_op_reply(103 mds0_sessionmap [create] v100'68 uv68 ondisk = 0) ==== 159+0+0 (crc 0 0 0) 0x7fdaac05cb30 con 0x55f490244ad0 2024-08-03T20:56:54.417 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.415+0000 7fdac5e0cf40 1 -- 172.21.15.32:0/828062416 --> [v2:172.21.15.32:6800/9088303,v1:172.21.15.32:6801/9088303] -- osd_op(unknown.0.0:104 12.b 12:d0630e4c:::mds0_sessionmap:head [omap-set-header in=14b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100) -- 0x55f490247610 con 0x55f490244ad0 2024-08-03T20:56:54.418 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.417+0000 7fdabd800640 1 -- 172.21.15.32:0/828062416 <== osd.3 v2:172.21.15.32:6800/9088303 4 ==== osd_op_reply(104 mds0_sessionmap [omap-set-header] v100'69 uv69 ondisk = 0) ==== 159+0+0 (crc 0 0 0) 0x7fdaac05cb30 con 0x55f490244ad0 2024-08-03T20:56:54.418 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.417+0000 7fdac5e0cf40 1 -- 172.21.15.32:0/828062416 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:105 12.7 12:e0b41d06:::609.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100) -- 0x55f490247610 con 0x55f490238660 2024-08-03T20:56:54.420 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.418+0000 7fdabd800640 1 -- 172.21.15.32:0/828062416 <== osd.1 v2:172.21.15.32:6824/1551241759 15 ==== osd_op_reply(105 609.00000000 [create] v100'87 uv86 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fdaac05cb30 con 0x55f490238660 2024-08-03T20:56:54.420 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.418+0000 7fdac5e0cf40 1 -- 172.21.15.32:0/828062416 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:106 12.7 12:e0b41d06:::609.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e100) -- 0x55f490247610 con 0x55f490238660 2024-08-03T20:56:54.421 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.420+0000 7fdabd800640 1 -- 172.21.15.32:0/828062416 <== osd.1 v2:172.21.15.32:6824/1551241759 16 ==== osd_op_reply(106 609.00000000 [delete] v100'88 uv88 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fdaac05cb30 con 0x55f490238660 2024-08-03T20:56:54.421 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.420+0000 7fdac5e0cf40 1 -- 172.21.15.32:0/828062416 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:107 12.7 12:e0b41d06:::609.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100) -- 0x55f490247610 con 0x55f490238660 2024-08-03T20:56:54.422 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::609.00000000:head# 2024-08-03T20:56:54.423 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.421+0000 7fdabd800640 1 -- 172.21.15.32:0/828062416 <== osd.1 v2:172.21.15.32:6824/1551241759 17 ==== osd_op_reply(107 609.00000000 [create] v100'89 uv89 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fdaac05cb30 con 0x55f490238660 2024-08-03T20:56:54.423 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.421+0000 7fdac5e0cf40 1 -- 172.21.15.32:0/828062416 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:108 12.7 12:e0b41d06:::609.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100) -- 0x55f490248620 con 0x55f490238660 2024-08-03T20:56:54.424 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.422+0000 7fdabd800640 1 -- 172.21.15.32:0/828062416 <== osd.1 v2:172.21.15.32:6824/1551241759 18 ==== osd_op_reply(108 609.00000000 [setxattr (30)] v100'90 uv90 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fdaac05cb30 con 0x55f490238660 2024-08-03T20:56:54.424 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.422+0000 7fdac5e0cf40 1 -- 172.21.15.32:0/828062416 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:109 12.7 12:e0b41d06:::609.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100) -- 0x55f490248620 con 0x55f490238660 2024-08-03T20:56:54.426 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.424+0000 7fdabd800640 1 -- 172.21.15.32:0/828062416 <== osd.1 v2:172.21.15.32:6824/1551241759 19 ==== osd_op_reply(109 609.00000000 [setxattr (62)] v100'91 uv91 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fdaac05cb30 con 0x55f490238660 2024-08-03T20:56:54.426 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.424+0000 7fdac5e0cf40 1 -- 172.21.15.32:0/828062416 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:110 12.7 12:e0b41d06:::609.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100) -- 0x55f490247610 con 0x55f490238660 2024-08-03T20:56:54.427 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.426+0000 7fdabd800640 1 -- 172.21.15.32:0/828062416 <== osd.1 v2:172.21.15.32:6824/1551241759 20 ==== osd_op_reply(110 609.00000000 [omap-set-header] v100'92 uv92 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fdaac05cb30 con 0x55f490238660 2024-08-03T20:56:54.427 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.426+0000 7fdac5e0cf40 1 -- 172.21.15.32:0/828062416 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:111 12.f 12:f1f69726:::400.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100) -- 0x55f490247610 con 0x55f49023cd00 2024-08-03T20:56:54.428 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.427+0000 7fdabec00640 1 -- 172.21.15.32:0/828062416 <== osd.6 v2:172.21.15.118:6825/801947386 12 ==== osd_op_reply(111 400.00000000 [create] v100'74 uv73 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fdab8003040 con 0x55f49023cd00 2024-08-03T20:56:54.428 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.427+0000 7fdac5e0cf40 1 -- 172.21.15.32:0/828062416 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:112 12.f 12:f1f69726:::400.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e100) -- 0x55f490247610 con 0x55f49023cd00 2024-08-03T20:56:54.430 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.429+0000 7fdabec00640 1 -- 172.21.15.32:0/828062416 <== osd.6 v2:172.21.15.118:6825/801947386 13 ==== osd_op_reply(112 400.00000000 [delete] v100'75 uv75 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fdab8003040 con 0x55f49023cd00 2024-08-03T20:56:54.430 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.429+0000 7fdac5e0cf40 1 -- 172.21.15.32:0/828062416 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:113 12.f 12:f1f69726:::400.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100) -- 0x55f490247610 con 0x55f49023cd00 2024-08-03T20:56:54.431 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::400.00000000:head# 2024-08-03T20:56:54.432 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.430+0000 7fdabec00640 1 -- 172.21.15.32:0/828062416 <== osd.6 v2:172.21.15.118:6825/801947386 14 ==== osd_op_reply(113 400.00000000 [create] v100'76 uv76 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fdab8003040 con 0x55f49023cd00 2024-08-03T20:56:54.432 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.430+0000 7fdac5e0cf40 1 -- 172.21.15.32:0/828062416 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:114 12.f 12:f1f69726:::400.00000000:head [write 0~22 in=22b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100) -- 0x55f490247610 con 0x55f49023cd00 2024-08-03T20:56:54.433 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.432+0000 7fdabec00640 1 -- 172.21.15.32:0/828062416 <== osd.6 v2:172.21.15.118:6825/801947386 15 ==== osd_op_reply(114 400.00000000 [write 0~22] v100'77 uv77 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fdab8003040 con 0x55f49023cd00 2024-08-03T20:56:54.433 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.432+0000 7fdac5e0cf40 1 -- 172.21.15.32:0/828062416 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:115 12.f 12:f1f69726:::400.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100) -- 0x55f490247610 con 0x55f49023cd00 2024-08-03T20:56:54.435 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.434+0000 7fdabec00640 1 -- 172.21.15.32:0/828062416 <== osd.6 v2:172.21.15.118:6825/801947386 16 ==== osd_op_reply(115 400.00000000 [omap-set-header] v100'78 uv78 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fdab8003040 con 0x55f49023cd00 2024-08-03T20:56:54.435 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.434+0000 7fdac5e0cf40 1 -- 172.21.15.32:0/828062416 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:116 12.1f 12:fe07bba1:::100.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100) -- 0x55f490247610 con 0x55f49023cd00 2024-08-03T20:56:54.436 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.435+0000 7fdabec00640 1 -- 172.21.15.32:0/828062416 <== osd.6 v2:172.21.15.118:6825/801947386 17 ==== osd_op_reply(116 100.00000000.inode [create] v100'182 uv172 ondisk = -17 ((17) File exists)) ==== 162+0+0 (crc 0 0 0) 0x7fdab8003040 con 0x55f49023cd00 2024-08-03T20:56:54.437 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.435+0000 7fdac5e0cf40 1 -- 172.21.15.32:0/828062416 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:117 12.1f 12:fe07bba1:::100.00000000.inode:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e100) -- 0x55f490247610 con 0x55f49023cd00 2024-08-03T20:56:54.438 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.437+0000 7fdabec00640 1 -- 172.21.15.32:0/828062416 <== osd.6 v2:172.21.15.118:6825/801947386 18 ==== osd_op_reply(117 100.00000000.inode [delete] v100'183 uv183 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7fdab8003040 con 0x55f49023cd00 2024-08-03T20:56:54.438 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.437+0000 7fdac5e0cf40 1 -- 172.21.15.32:0/828062416 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:118 12.1f 12:fe07bba1:::100.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100) -- 0x55f490247610 con 0x55f49023cd00 2024-08-03T20:56:54.439 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.438+0000 7fdabec00640 1 -- 172.21.15.32:0/828062416 <== osd.6 v2:172.21.15.118:6825/801947386 19 ==== osd_op_reply(118 100.00000000.inode [create] v100'184 uv184 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7fdab8003040 con 0x55f49023cd00 2024-08-03T20:56:54.440 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::100.00000000.inode:head# 2024-08-03T20:56:54.440 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.438+0000 7fdac5e0cf40 1 -- 172.21.15.32:0/828062416 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:119 12.1f 12:fe07bba1:::100.00000000.inode:head [write 0~582 in=582b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100) -- 0x55f490247610 con 0x55f49023cd00 2024-08-03T20:56:54.441 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.440+0000 7fdabec00640 1 -- 172.21.15.32:0/828062416 <== osd.6 v2:172.21.15.118:6825/801947386 20 ==== osd_op_reply(119 100.00000000.inode [write 0~582] v100'185 uv185 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7fdab8003040 con 0x55f49023cd00 2024-08-03T20:56:54.441 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.440+0000 7fdac5e0cf40 1 -- 172.21.15.32:0/828062416 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:120 12.1f 12:fe07bba1:::100.00000000.inode:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100) -- 0x55f490247610 con 0x55f49023cd00 2024-08-03T20:56:54.443 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.441+0000 7fdabec00640 1 -- 172.21.15.32:0/828062416 <== osd.6 v2:172.21.15.118:6825/801947386 21 ==== osd_op_reply(120 100.00000000.inode [omap-set-header] v100'186 uv186 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7fdab8003040 con 0x55f49023cd00 2024-08-03T20:56:54.443 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.441+0000 7fdac5e0cf40 1 -- 172.21.15.32:0/828062416 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:121 12.1f 12:ff5b34d6:::1.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100) -- 0x55f490247610 con 0x55f49023cd00 2024-08-03T20:56:54.444 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.443+0000 7fdabec00640 1 -- 172.21.15.32:0/828062416 <== osd.6 v2:172.21.15.118:6825/801947386 22 ==== osd_op_reply(121 1.00000000 [create] v100'187 uv187 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7fdab8003040 con 0x55f49023cd00 2024-08-03T20:56:54.445 INFO:teuthology.orchestra.run.smithi032.stdout:Write #-9223372036854775808:00000000:::1.00000000:head# 2024-08-03T20:56:54.445 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.443+0000 7fdac5e0cf40 1 -- 172.21.15.32:0/828062416 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:122 12.1f 12:ff5b34d6:::1.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100) -- 0x55f490247610 con 0x55f49023cd00 2024-08-03T20:56:54.446 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.444+0000 7fdabec00640 1 -- 172.21.15.32:0/828062416 <== osd.6 v2:172.21.15.118:6825/801947386 23 ==== osd_op_reply(122 1.00000000 [setxattr (30)] v100'188 uv188 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7fdab8003040 con 0x55f49023cd00 2024-08-03T20:56:54.446 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.445+0000 7fdac5e0cf40 1 -- 172.21.15.32:0/828062416 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:123 12.1f 12:ff5b34d6:::1.00000000:head [setxattr parent (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100) -- 0x55f490248620 con 0x55f49023cd00 2024-08-03T20:56:54.447 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.446+0000 7fdabec00640 1 -- 172.21.15.32:0/828062416 <== osd.6 v2:172.21.15.118:6825/801947386 24 ==== osd_op_reply(123 1.00000000 [setxattr (30)] v100'189 uv189 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7fdab8003040 con 0x55f49023cd00 2024-08-03T20:56:54.448 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.446+0000 7fdac5e0cf40 1 -- 172.21.15.32:0/828062416 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:124 12.1f 12:ff5b34d6:::1.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100) -- 0x55f490247610 con 0x55f49023cd00 2024-08-03T20:56:54.449 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.447+0000 7fdabec00640 1 -- 172.21.15.32:0/828062416 <== osd.6 v2:172.21.15.118:6825/801947386 25 ==== osd_op_reply(124 1.00000000 [omap-set-header] v100'190 uv190 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7fdab8003040 con 0x55f49023cd00 2024-08-03T20:56:54.449 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.448+0000 7fdac5e0cf40 1 -- 172.21.15.32:0/828062416 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:125 12.1f 12:ff5b34d6:::1.00000000:head [omap-set-vals in=584b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100) -- 0x55f490247e80 con 0x55f49023cd00 2024-08-03T20:56:54.451 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.449+0000 7fdabec00640 1 -- 172.21.15.32:0/828062416 <== osd.6 v2:172.21.15.118:6825/801947386 26 ==== osd_op_reply(125 1.00000000 [omap-set-vals] v100'191 uv191 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7fdab8003040 con 0x55f49023cd00 2024-08-03T20:56:54.452 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.449+0000 7fdac5e0cf40 1 -- 172.21.15.32:0/828062416 >> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] conn(0x55f490160be0 msgr2=0x55f490232330 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:56:54.452 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.449+0000 7fdac5e0cf40 1 --2- 172.21.15.32:0/828062416 >> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] conn(0x55f490160be0 0x55f490232330 crc :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:54.452 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.449+0000 7fdac5e0cf40 1 -- 172.21.15.32:0/828062416 >> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] conn(0x55f490238660 msgr2=0x55f49023aa80 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:56:54.452 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.449+0000 7fdac5e0cf40 1 --2- 172.21.15.32:0/828062416 >> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] conn(0x55f490238660 0x55f49023aa80 crc :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:54.452 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.449+0000 7fdac5e0cf40 1 -- 172.21.15.32:0/828062416 >> [v2:172.21.15.32:6800/9088303,v1:172.21.15.32:6801/9088303] conn(0x55f490244ad0 msgr2=0x55f4902470d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:56:54.452 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.449+0000 7fdac5e0cf40 1 --2- 172.21.15.32:0/828062416 >> [v2:172.21.15.32:6800/9088303,v1:172.21.15.32:6801/9088303] conn(0x55f490244ad0 0x55f4902470d0 crc :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:54.452 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.449+0000 7fdac5e0cf40 1 -- 172.21.15.32:0/828062416 >> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] conn(0x55f4902337b0 msgr2=0x55f490236880 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:56:54.452 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.449+0000 7fdac5e0cf40 1 --2- 172.21.15.32:0/828062416 >> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] conn(0x55f4902337b0 0x55f490236880 crc :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:54.452 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.449+0000 7fdac5e0cf40 1 -- 172.21.15.32:0/828062416 >> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] conn(0x55f49023cd00 msgr2=0x55f49023f120 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:56:54.452 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.449+0000 7fdac5e0cf40 1 --2- 172.21.15.32:0/828062416 >> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] conn(0x55f49023cd00 0x55f49023f120 crc :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:54.452 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.449+0000 7fdabec00640 1 -- 172.21.15.32:0/828062416 reap_dead start 2024-08-03T20:56:54.452 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.449+0000 7fdac5e0cf40 1 -- 172.21.15.32:0/828062416 >> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] conn(0x55f490240100 msgr2=0x55f490242520 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:56:54.452 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.449+0000 7fdac5e0cf40 1 --2- 172.21.15.32:0/828062416 >> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] conn(0x55f490240100 0x55f490242520 crc :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:54.453 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.449+0000 7fdac5e0cf40 1 -- 172.21.15.32:0/828062416 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fdaa00759f0 msgr2=0x7fdaa0077f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:56:54.453 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.449+0000 7fdac5e0cf40 1 --2- 172.21.15.32:0/828062416 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fdaa00759f0 0x7fdaa0077f00 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7fdab0062000 tx=0x7fdab0004940 comp rx=0 tx=0).stop 2024-08-03T20:56:54.453 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.449+0000 7fdac5e0cf40 1 -- 172.21.15.32:0/828062416 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55f490170ea0 msgr2=0x55f4900f7980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:56:54.453 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.449+0000 7fdac5e0cf40 1 --2- 172.21.15.32:0/828062416 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55f490170ea0 0x55f4900f7980 secure :-1 s=READY pgs=1442 cs=0 l=1 rev1=1 crypto rx=0x7fdaac00a6e0 tx=0x7fdaac00aba0 comp rx=0 tx=0).stop 2024-08-03T20:56:54.453 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.450+0000 7fdac5e0cf40 1 -- 172.21.15.32:0/828062416 shutdown_connections 2024-08-03T20:56:54.453 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.450+0000 7fdac5e0cf40 1 --2- 172.21.15.32:0/828062416 >> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] conn(0x55f490240100 0x55f490242520 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:54.453 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.450+0000 7fdac5e0cf40 1 --2- 172.21.15.32:0/828062416 >> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] conn(0x55f49023cd00 0x55f49023f120 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:54.453 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.450+0000 7fdac5e0cf40 1 --2- 172.21.15.32:0/828062416 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fdaa00759f0 0x7fdaa0077f00 unknown :-1 s=CLOSED pgs=590 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:54.453 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.450+0000 7fdac5e0cf40 1 --2- 172.21.15.32:0/828062416 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55f490170ea0 0x55f4900f7980 unknown :-1 s=CLOSED pgs=1442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:54.453 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.450+0000 7fdac5e0cf40 1 -- 172.21.15.32:0/828062416 >> 172.21.15.32:0/828062416 conn(0x55f48ffd8ac0 msgr2=0x55f490189160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:56:54.453 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.450+0000 7fdac5e0cf40 1 -- 172.21.15.32:0/828062416 shutdown_connections 2024-08-03T20:56:54.453 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.450+0000 7fdac5e0cf40 1 -- 172.21.15.32:0/828062416 wait complete. 2024-08-03T20:56:54.467 DEBUG:teuthology.orchestra.run.smithi032:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata put 1.00000000.inode - 2024-08-03T20:56:54.654 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.650+0000 7fa389463f40 1 Processor -- start 2024-08-03T20:56:54.654 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.651+0000 7fa389463f40 1 -- start start 2024-08-03T20:56:54.655 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.652+0000 7fa389463f40 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x558cdcc87360 0x558cdcca2f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:56:54.655 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.652+0000 7fa389463f40 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558cdcca4cb0 0x558cdcc195b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:56:54.655 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.652+0000 7fa389463f40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x558cdcca53f0 0x558cdcc19af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:56:54.655 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.652+0000 7fa389463f40 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x558cdcb7cbf0 con 0x558cdcca4cb0 2024-08-03T20:56:54.655 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.652+0000 7fa389463f40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x558cdcb896b0 con 0x558cdcca53f0 2024-08-03T20:56:54.655 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.652+0000 7fa389463f40 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x558cdcb1ee30 con 0x558cdcc87360 2024-08-03T20:56:54.655 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.652+0000 7fa381c00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558cdcca4cb0 0x558cdcc195b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:56:54.655 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.652+0000 7fa381c00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558cdcca4cb0 0x558cdcc195b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:39966/0 (socket says 172.21.15.32:39966) 2024-08-03T20:56:54.655 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.652+0000 7fa381c00640 1 -- 172.21.15.32:0/3276410852 learned_addr learned my addr 172.21.15.32:0/3276410852 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:56:54.655 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.652+0000 7fa383000640 1 --2- 172.21.15.32:0/3276410852 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x558cdcca53f0 0x558cdcc19af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:56:54.655 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.652+0000 7fa382600640 1 --2- 172.21.15.32:0/3276410852 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x558cdcc87360 0x558cdcca2f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:56:54.655 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.653+0000 7fa381c00640 1 -- 172.21.15.32:0/3276410852 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x558cdcc87360 msgr2=0x558cdcca2f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:56:54.655 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.653+0000 7fa381c00640 1 --2- 172.21.15.32:0/3276410852 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x558cdcc87360 0x558cdcca2f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:54.656 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.653+0000 7fa381c00640 1 -- 172.21.15.32:0/3276410852 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x558cdcca53f0 msgr2=0x558cdcc19af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:56:54.656 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.653+0000 7fa381c00640 1 --2- 172.21.15.32:0/3276410852 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x558cdcca53f0 0x558cdcc19af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:54.656 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.653+0000 7fa381c00640 1 -- 172.21.15.32:0/3276410852 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558cdcc17aa0 con 0x558cdcca4cb0 2024-08-03T20:56:54.656 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.653+0000 7fa381c00640 1 --2- 172.21.15.32:0/3276410852 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558cdcca4cb0 0x558cdcc195b0 secure :-1 s=READY pgs=1445 cs=0 l=1 rev1=1 crypto rx=0x7fa370001df0 tx=0x7fa37000b590 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6c1d125220c8f8d8 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:56:54.657 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.653+0000 7fa37be00640 1 -- 172.21.15.32:0/3276410852 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa370016070 con 0x558cdcca4cb0 2024-08-03T20:56:54.657 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.653+0000 7fa37be00640 1 -- 172.21.15.32:0/3276410852 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fa37000f690 con 0x558cdcca4cb0 2024-08-03T20:56:54.658 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.653+0000 7fa37be00640 1 -- 172.21.15.32:0/3276410852 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa37000fae0 con 0x558cdcca4cb0 2024-08-03T20:56:54.658 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.654+0000 7fa389463f40 1 -- 172.21.15.32:0/3276410852 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558cdcca4cb0 msgr2=0x558cdcc195b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:56:54.658 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.654+0000 7fa389463f40 1 --2- 172.21.15.32:0/3276410852 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558cdcca4cb0 0x558cdcc195b0 secure :-1 s=READY pgs=1445 cs=0 l=1 rev1=1 crypto rx=0x7fa370001df0 tx=0x7fa37000b590 comp rx=0 tx=0).stop 2024-08-03T20:56:54.658 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.654+0000 7fa389463f40 1 -- 172.21.15.32:0/3276410852 shutdown_connections 2024-08-03T20:56:54.658 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.654+0000 7fa389463f40 1 --2- 172.21.15.32:0/3276410852 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x558cdcca53f0 0x558cdcc19af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:54.658 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.654+0000 7fa389463f40 1 --2- 172.21.15.32:0/3276410852 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558cdcca4cb0 0x558cdcc195b0 unknown :-1 s=CLOSED pgs=1445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:54.658 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.654+0000 7fa389463f40 1 --2- 172.21.15.32:0/3276410852 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x558cdcc87360 0x558cdcca2f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:54.658 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.654+0000 7fa389463f40 1 -- 172.21.15.32:0/3276410852 >> 172.21.15.32:0/3276410852 conn(0x558cdcaf0ac0 msgr2=0x558cdcca11f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:56:54.658 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.654+0000 7fa389463f40 1 -- 172.21.15.32:0/3276410852 shutdown_connections 2024-08-03T20:56:54.659 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.654+0000 7fa389463f40 1 -- 172.21.15.32:0/3276410852 wait complete. 2024-08-03T20:56:54.659 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.655+0000 7fa389463f40 1 Processor -- start 2024-08-03T20:56:54.659 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.655+0000 7fa389463f40 1 -- start start 2024-08-03T20:56:54.660 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.657+0000 7fa389463f40 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558cdcc87360 0x558cdcc80020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:56:54.660 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.657+0000 7fa389463f40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x558cdcca4cb0 0x558cdcc11400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:56:54.660 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.657+0000 7fa389463f40 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x558cdcca53f0 0x558cdcc11940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:56:54.661 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.657+0000 7fa389463f40 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x558cdcc9dbf0 con 0x558cdcc87360 2024-08-03T20:56:54.661 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.657+0000 7fa389463f40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x558cdcc9e8f0 con 0x558cdcca4cb0 2024-08-03T20:56:54.661 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.657+0000 7fa389463f40 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x558cdcc17800 con 0x558cdcca53f0 2024-08-03T20:56:54.661 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.657+0000 7fa382600640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558cdcc87360 0x558cdcc80020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:56:54.661 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.657+0000 7fa382600640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558cdcc87360 0x558cdcc80020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:39974/0 (socket says 172.21.15.32:39974) 2024-08-03T20:56:54.661 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.657+0000 7fa382600640 1 -- 172.21.15.32:0/910362889 learned_addr learned my addr 172.21.15.32:0/910362889 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:56:54.661 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.657+0000 7fa383000640 1 --2- 172.21.15.32:0/910362889 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x558cdcca53f0 0x558cdcc11940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:56:54.661 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.657+0000 7fa381c00640 1 --2- 172.21.15.32:0/910362889 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x558cdcca4cb0 0x558cdcc11400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:56:54.661 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.657+0000 7fa382600640 1 -- 172.21.15.32:0/910362889 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x558cdcca53f0 msgr2=0x558cdcc11940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:56:54.661 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.658+0000 7fa382600640 1 --2- 172.21.15.32:0/910362889 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x558cdcca53f0 0x558cdcc11940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:54.661 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.658+0000 7fa382600640 1 -- 172.21.15.32:0/910362889 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x558cdcca4cb0 msgr2=0x558cdcc11400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:56:54.661 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.658+0000 7fa382600640 1 --2- 172.21.15.32:0/910362889 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x558cdcca4cb0 0x558cdcc11400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:54.661 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.658+0000 7fa382600640 1 -- 172.21.15.32:0/910362889 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558cdcc82220 con 0x558cdcc87360 2024-08-03T20:56:54.662 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.658+0000 7fa383000640 1 --2- 172.21.15.32:0/910362889 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x558cdcca53f0 0x558cdcc11940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-03T20:56:54.662 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.658+0000 7fa381c00640 1 --2- 172.21.15.32:0/910362889 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x558cdcca4cb0 0x558cdcc11400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-03T20:56:54.662 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.658+0000 7fa382600640 1 --2- 172.21.15.32:0/910362889 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558cdcc87360 0x558cdcc80020 secure :-1 s=READY pgs=1446 cs=0 l=1 rev1=1 crypto rx=0x7fa374046660 tx=0x7fa374056bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:56:54.662 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.658+0000 7fa37aa00640 1 -- 172.21.15.32:0/910362889 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa374068070 con 0x558cdcc87360 2024-08-03T20:56:54.662 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.658+0000 7fa37aa00640 1 -- 172.21.15.32:0/910362889 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fa374056db0 con 0x558cdcc87360 2024-08-03T20:56:54.662 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.658+0000 7fa37aa00640 1 -- 172.21.15.32:0/910362889 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa374004580 con 0x558cdcc87360 2024-08-03T20:56:54.662 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.658+0000 7fa389463f40 1 -- 172.21.15.32:0/910362889 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x558cdcc808a0 con 0x558cdcc87360 2024-08-03T20:56:54.662 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.658+0000 7fa389463f40 1 -- 172.21.15.32:0/910362889 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558cdcc189e0 con 0x558cdcc87360 2024-08-03T20:56:54.663 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.661+0000 7fa37aa00640 1 -- 172.21.15.32:0/910362889 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7fa37406b070 con 0x558cdcc87360 2024-08-03T20:56:54.663 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.661+0000 7fa37aa00640 1 --2- 172.21.15.32:0/910362889 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fa364075a20 0x7fa364077f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:56:54.663 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.662+0000 7fa37aa00640 1 -- 172.21.15.32:0/910362889 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 7144+0+0 (secure 0 0 0) 0x7fa3740dbf60 con 0x558cdcc87360 2024-08-03T20:56:54.663 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.662+0000 7fa389463f40 1 --2- 172.21.15.32:0/910362889 >> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] conn(0x558cdcc84380 0x558cdcc159e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:56:54.663 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.662+0000 7fa381c00640 1 --2- 172.21.15.32:0/910362889 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fa364075a20 0x7fa364077f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:56:54.663 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.662+0000 7fa389463f40 1 -- 172.21.15.32:0/910362889 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:1 12.14 12:287034c4:::1.00000000.inode:head [writefull 0~80 in=80b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100) -- 0x558cdcc82700 con 0x558cdcc84380 2024-08-03T20:56:54.664 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.662+0000 7fa383000640 1 --2- 172.21.15.32:0/910362889 >> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] conn(0x558cdcc84380 0x558cdcc159e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:56:54.664 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.663+0000 7fa381c00640 1 --2- 172.21.15.32:0/910362889 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fa364075a20 0x7fa364077f30 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7fa370001df0 tx=0x7fa370003040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:56:54.664 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.663+0000 7fa383000640 1 --2- 172.21.15.32:0/910362889 >> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] conn(0x558cdcc84380 0x558cdcc159e0 crc :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:56:54.667 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.665+0000 7fa383000640 1 -- 172.21.15.32:0/910362889 <== osd.5 v2:172.21.15.118:6810/2724891770 1 ==== osd_op_reply(1 1.00000000.inode [writefull 0~80] v100'351 uv351 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7fa37c0025f0 con 0x558cdcc84380 2024-08-03T20:56:54.667 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.665+0000 7fa389463f40 1 -- 172.21.15.32:0/910362889 >> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] conn(0x558cdcc84380 msgr2=0x558cdcc159e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:56:54.667 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.665+0000 7fa389463f40 1 --2- 172.21.15.32:0/910362889 >> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] conn(0x558cdcc84380 0x558cdcc159e0 crc :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:54.667 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.665+0000 7fa389463f40 1 -- 172.21.15.32:0/910362889 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fa364075a20 msgr2=0x7fa364077f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:56:54.667 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.665+0000 7fa389463f40 1 --2- 172.21.15.32:0/910362889 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fa364075a20 0x7fa364077f30 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7fa370001df0 tx=0x7fa370003040 comp rx=0 tx=0).stop 2024-08-03T20:56:54.667 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.665+0000 7fa389463f40 1 -- 172.21.15.32:0/910362889 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558cdcc87360 msgr2=0x558cdcc80020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:56:54.667 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.665+0000 7fa389463f40 1 --2- 172.21.15.32:0/910362889 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558cdcc87360 0x558cdcc80020 secure :-1 s=READY pgs=1446 cs=0 l=1 rev1=1 crypto rx=0x7fa374046660 tx=0x7fa374056bc0 comp rx=0 tx=0).stop 2024-08-03T20:56:54.667 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.665+0000 7fa383000640 1 -- 172.21.15.32:0/910362889 reap_dead start 2024-08-03T20:56:54.668 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.665+0000 7fa389463f40 1 -- 172.21.15.32:0/910362889 shutdown_connections 2024-08-03T20:56:54.668 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.665+0000 7fa389463f40 1 -- 172.21.15.32:0/910362889 >> 172.21.15.32:0/910362889 conn(0x558cdcaf0ac0 msgr2=0x558cdcca11d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:56:54.668 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.665+0000 7fa389463f40 1 -- 172.21.15.32:0/910362889 shutdown_connections 2024-08-03T20:56:54.668 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.665+0000 7fa389463f40 1 -- 172.21.15.32:0/910362889 wait complete. 2024-08-03T20:56:54.677 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs joinable true 2024-08-03T20:56:54.910 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.909+0000 7fbd0c600640 1 -- 172.21.15.32:0/2248755296 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fbd040b3e80 msgr2=0x7fbd040b4290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:56:54.911 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.909+0000 7fbd0c600640 1 --2- 172.21.15.32:0/2248755296 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fbd040b3e80 0x7fbd040b4290 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7fbcfc005fd0 tx=0x7fbcfc02a8b0 comp rx=0 tx=0).stop 2024-08-03T20:56:54.911 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.909+0000 7fbd0c600640 1 -- 172.21.15.32:0/2248755296 shutdown_connections 2024-08-03T20:56:54.911 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.909+0000 7fbd0c600640 1 --2- 172.21.15.32:0/2248755296 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fbd040b3e80 0x7fbd040b4290 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:54.911 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.909+0000 7fbd0c600640 1 --2- 172.21.15.32:0/2248755296 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fbd0406e950 0x7fbd0406ed60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:54.911 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.909+0000 7fbd0c600640 1 -- 172.21.15.32:0/2248755296 >> 172.21.15.32:0/2248755296 conn(0x7fbd040b1bd0 msgr2=0x7fbd040b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:56:54.911 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.909+0000 7fbd0c600640 1 -- 172.21.15.32:0/2248755296 shutdown_connections 2024-08-03T20:56:54.911 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.909+0000 7fbd0c600640 1 -- 172.21.15.32:0/2248755296 wait complete. 2024-08-03T20:56:54.912 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.911+0000 7fbd0c600640 1 Processor -- start 2024-08-03T20:56:54.912 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.911+0000 7fbd0c600640 1 -- start start 2024-08-03T20:56:54.915 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.913+0000 7fbd0c600640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fbd0406e950 0x7fbd04151a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:56:54.915 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.913+0000 7fbd0c600640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbd040b3e80 0x7fbd0414f810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:56:54.915 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.913+0000 7fbd0c600640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fbd0415d2b0 0x7fbd04148e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:56:54.915 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.913+0000 7fbd0c600640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fbd040cd010 con 0x7fbd040b3e80 2024-08-03T20:56:54.915 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.913+0000 7fbd0c600640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7fbd040cce90 con 0x7fbd0415d2b0 2024-08-03T20:56:54.915 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.913+0000 7fbd0c600640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7fbd040cd190 con 0x7fbd0406e950 2024-08-03T20:56:54.915 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.913+0000 7fbd09200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbd040b3e80 0x7fbd0414f810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:56:54.915 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.913+0000 7fbd09200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbd040b3e80 0x7fbd0414f810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:39984/0 (socket says 172.21.15.32:39984) 2024-08-03T20:56:54.915 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.913+0000 7fbd09200640 1 -- 172.21.15.32:0/1514086255 learned_addr learned my addr 172.21.15.32:0/1514086255 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:56:54.915 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.913+0000 7fbd0a600640 1 --2- 172.21.15.32:0/1514086255 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fbd0415d2b0 0x7fbd04148e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:56:54.915 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.913+0000 7fbd09c00640 1 --2- 172.21.15.32:0/1514086255 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fbd0406e950 0x7fbd04151a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:56:54.916 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.913+0000 7fbd09200640 1 -- 172.21.15.32:0/1514086255 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fbd0406e950 msgr2=0x7fbd04151a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:56:54.916 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.913+0000 7fbd09200640 1 --2- 172.21.15.32:0/1514086255 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fbd0406e950 0x7fbd04151a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:54.916 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.913+0000 7fbd09200640 1 -- 172.21.15.32:0/1514086255 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fbd0415d2b0 msgr2=0x7fbd04148e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:56:54.916 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.913+0000 7fbd09200640 1 --2- 172.21.15.32:0/1514086255 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fbd0415d2b0 0x7fbd04148e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:54.916 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.913+0000 7fbd09200640 1 -- 172.21.15.32:0/1514086255 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbd0414f050 con 0x7fbd040b3e80 2024-08-03T20:56:54.917 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.914+0000 7fbd09200640 1 --2- 172.21.15.32:0/1514086255 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbd040b3e80 0x7fbd0414f810 secure :-1 s=READY pgs=1448 cs=0 l=1 rev1=1 crypto rx=0x7fbcf000e790 tx=0x7fbcf000ec50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:56:54.917 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.914+0000 7fbd01600640 1 -- 172.21.15.32:0/1514086255 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbcf000eea0 con 0x7fbd040b3e80 2024-08-03T20:56:54.917 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.914+0000 7fbd01600640 1 -- 172.21.15.32:0/1514086255 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fbcf000cdb0 con 0x7fbd040b3e80 2024-08-03T20:56:54.917 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.914+0000 7fbd0c600640 1 -- 172.21.15.32:0/1514086255 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbd041f7960 con 0x7fbd040b3e80 2024-08-03T20:56:54.917 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.914+0000 7fbd01600640 1 -- 172.21.15.32:0/1514086255 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbcf0010690 con 0x7fbd040b3e80 2024-08-03T20:56:54.917 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.914+0000 7fbd0c600640 1 -- 172.21.15.32:0/1514086255 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbd04151f60 con 0x7fbd040b3e80 2024-08-03T20:56:54.917 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.915+0000 7fbd0c600640 1 -- 172.21.15.32:0/1514086255 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbd0414ee80 con 0x7fbd040b3e80 2024-08-03T20:56:54.918 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.917+0000 7fbd01600640 1 -- 172.21.15.32:0/1514086255 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7fbcf0010830 con 0x7fbd040b3e80 2024-08-03T20:56:54.919 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.918+0000 7fbd01600640 1 --2- 172.21.15.32:0/1514086255 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fbce4077010 0x7fbce4079520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:56:54.919 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.918+0000 7fbd01600640 1 -- 172.21.15.32:0/1514086255 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 7144+0+0 (secure 0 0 0) 0x7fbcf00999e0 con 0x7fbd040b3e80 2024-08-03T20:56:54.919 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.918+0000 7fbd09c00640 1 --2- 172.21.15.32:0/1514086255 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fbce4077010 0x7fbce4079520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:56:54.920 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.919+0000 7fbd09c00640 1 --2- 172.21.15.32:0/1514086255 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fbce4077010 0x7fbce4079520 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7fbcf8007fd0 tx=0x7fbcf8001370 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:56:54.922 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:54.921+0000 7fbd01600640 1 -- 172.21.15.32:0/1514086255 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7fbcf0061fe0 con 0x7fbd040b3e80 2024-08-03T20:56:55.168 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:55.166+0000 7fbd0c600640 1 -- 172.21.15.32:0/1514086255 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "joinable", "val": "true"} v 0) -- 0x7fbd0415be10 con 0x7fbd040b3e80 2024-08-03T20:56:56.080 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:56.079+0000 7fbd01600640 1 -- 172.21.15.32:0/1514086255 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "joinable", "val": "true"}]=0 cephfs marked joinable; MDS may join as newly active. v183) ==== 162+0+0 (secure 0 0 0) 0x7fbcf0066530 con 0x7fbd040b3e80 2024-08-03T20:56:56.080 INFO:teuthology.orchestra.run.smithi032.stderr:cephfs marked joinable; MDS may join as newly active. 2024-08-03T20:56:56.085 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:56.083+0000 7fbd0c600640 1 -- 172.21.15.32:0/1514086255 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fbce4077010 msgr2=0x7fbce4079520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:56:56.085 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:56.083+0000 7fbd0c600640 1 --2- 172.21.15.32:0/1514086255 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fbce4077010 0x7fbce4079520 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7fbcf8007fd0 tx=0x7fbcf8001370 comp rx=0 tx=0).stop 2024-08-03T20:56:56.085 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:56.083+0000 7fbd0c600640 1 -- 172.21.15.32:0/1514086255 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbd040b3e80 msgr2=0x7fbd0414f810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:56:56.085 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:56.083+0000 7fbd0c600640 1 --2- 172.21.15.32:0/1514086255 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbd040b3e80 0x7fbd0414f810 secure :-1 s=READY pgs=1448 cs=0 l=1 rev1=1 crypto rx=0x7fbcf000e790 tx=0x7fbcf000ec50 comp rx=0 tx=0).stop 2024-08-03T20:56:56.085 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:56.083+0000 7fbd0c600640 1 -- 172.21.15.32:0/1514086255 shutdown_connections 2024-08-03T20:56:56.086 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:56.083+0000 7fbd0c600640 1 --2- 172.21.15.32:0/1514086255 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fbce4077010 0x7fbce4079520 unknown :-1 s=CLOSED pgs=592 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:56.086 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:56.083+0000 7fbd0c600640 1 --2- 172.21.15.32:0/1514086255 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fbd0415d2b0 0x7fbd04148e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:56.086 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:56.083+0000 7fbd0c600640 1 --2- 172.21.15.32:0/1514086255 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbd040b3e80 0x7fbd0414f810 unknown :-1 s=CLOSED pgs=1448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:56.086 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:56.083+0000 7fbd0c600640 1 --2- 172.21.15.32:0/1514086255 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fbd0406e950 0x7fbd04151a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:56.086 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:56.083+0000 7fbd0c600640 1 -- 172.21.15.32:0/1514086255 >> 172.21.15.32:0/1514086255 conn(0x7fbd040b1bd0 msgr2=0x7fbd040b0320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:56:56.086 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:56.084+0000 7fbd0c600640 1 -- 172.21.15.32:0/1514086255 shutdown_connections 2024-08-03T20:56:56.086 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:56.084+0000 7fbd0c600640 1 -- 172.21.15.32:0/1514086255 wait complete. 2024-08-03T20:56:56.100 INFO:tasks.ceph.mds.a.smithi032.stderr:2024-08-03T20:56:56.099+0000 7f5e76a00640 -1 mds.0.cache.ino(0x1) Corrupt inode 0x1: End of buffer [buffer:2] 2024-08-03T20:56:56.101 INFO:tasks.ceph.mds.a.smithi032.stderr:2024-08-03T20:56:56.099+0000 7f5e76a00640 -1 log_channel(cluster) log [ERR] : Error loading MDS rank 0: (22) Invalid argument 2024-08-03T20:56:56.101 INFO:tasks.ceph.mon.a.smithi032.stderr:2024-08-03T20:56:56.100+0000 7f4e6d600640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-08-03T20:56:56.101 INFO:tasks.ceph.mon.a.smithi032.stderr:2024-08-03T20:56:56.100+0000 7f4e6d600640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 mds daemon damaged (MDS_DAMAGE) 2024-08-03T20:56:56.119 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-08-03T20:56:56.351 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:56.349+0000 7f383b200640 1 -- 172.21.15.32:0/4225090652 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3834144f90 msgr2=0x7f3834074570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:56:56.351 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:56.349+0000 7f383b200640 1 --2- 172.21.15.32:0/4225090652 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3834144f90 0x7f3834074570 secure :-1 s=READY pgs=1450 cs=0 l=1 rev1=1 crypto rx=0x7f3834072d80 tx=0x7f381c02a540 comp rx=0 tx=0).stop 2024-08-03T20:56:56.351 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:56.349+0000 7f383b200640 1 -- 172.21.15.32:0/4225090652 shutdown_connections 2024-08-03T20:56:56.351 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:56.349+0000 7f383b200640 1 --2- 172.21.15.32:0/4225090652 >> v2:172.21.15.118:3300/0 conn(0x7f383415fff0 0x7f3834160400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:56.351 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:56.349+0000 7f383b200640 1 --2- 172.21.15.32:0/4225090652 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3834144f90 0x7f3834074570 unknown :-1 s=CLOSED pgs=1450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:56.351 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:56.349+0000 7f383b200640 1 -- 172.21.15.32:0/4225090652 >> 172.21.15.32:0/4225090652 conn(0x7f38340b1bd0 msgr2=0x7f38340b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:56:56.351 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:56.349+0000 7f383b200640 1 -- 172.21.15.32:0/4225090652 shutdown_connections 2024-08-03T20:56:56.351 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:56.349+0000 7f383b200640 1 -- 172.21.15.32:0/4225090652 wait complete. 2024-08-03T20:56:56.353 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:56.351+0000 7f383b200640 1 Processor -- start 2024-08-03T20:56:56.353 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:56.352+0000 7f383b200640 1 -- start start 2024-08-03T20:56:56.356 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:56.353+0000 7f383b200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f383406e930 0x7f3834140cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:56:56.356 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:56.353+0000 7f383b200640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f3834144f90 0x7f38341f4af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:56:56.356 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:56.353+0000 7f383b200640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f383415fff0 0x7f38340add10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:56:56.356 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:56.353+0000 7f383b200640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f38340cf840 con 0x7f383406e930 2024-08-03T20:56:56.356 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:56.353+0000 7f383b200640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f3834145f30 con 0x7f3834144f90 2024-08-03T20:56:56.356 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:56.353+0000 7f383b200640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7f38340cf9c0 con 0x7f383415fff0 2024-08-03T20:56:56.356 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:56.353+0000 7f3833e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f383406e930 0x7f3834140cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:56:56.357 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:56.353+0000 7f3833e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f383406e930 0x7f3834140cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:39994/0 (socket says 172.21.15.32:39994) 2024-08-03T20:56:56.357 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:56.353+0000 7f3833e00640 1 -- 172.21.15.32:0/206210973 learned_addr learned my addr 172.21.15.32:0/206210973 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:56:56.357 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:56.353+0000 7f3839200640 1 --2- 172.21.15.32:0/206210973 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f383415fff0 0x7f38340add10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:56:56.357 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:56.353+0000 7f382b400640 1 --2- 172.21.15.32:0/206210973 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f3834144f90 0x7f38341f4af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:56:56.357 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:56.354+0000 7f3833e00640 1 -- 172.21.15.32:0/206210973 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f383415fff0 msgr2=0x7f38340add10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:56:56.357 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:56.354+0000 7f3833e00640 1 --2- 172.21.15.32:0/206210973 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f383415fff0 0x7f38340add10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:56.357 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:56.354+0000 7f3833e00640 1 -- 172.21.15.32:0/206210973 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f3834144f90 msgr2=0x7f38341f4af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:56:56.357 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:56.354+0000 7f3833e00640 1 --2- 172.21.15.32:0/206210973 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f3834144f90 0x7f38341f4af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:56.357 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:56.354+0000 7f3833e00640 1 -- 172.21.15.32:0/206210973 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f38341f8a50 con 0x7f383406e930 2024-08-03T20:56:56.357 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:56.354+0000 7f382b400640 1 --2- 172.21.15.32:0/206210973 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f3834144f90 0x7f38341f4af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-03T20:56:56.357 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:56.354+0000 7f3833e00640 1 --2- 172.21.15.32:0/206210973 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f383406e930 0x7f3834140cc0 secure :-1 s=READY pgs=1451 cs=0 l=1 rev1=1 crypto rx=0x7f382400c3a0 tx=0x7f382400c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:56:56.357 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:56.354+0000 7f3830c00640 1 -- 172.21.15.32:0/206210973 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3824011790 con 0x7f383406e930 2024-08-03T20:56:56.358 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:56.354+0000 7f3830c00640 1 -- 172.21.15.32:0/206210973 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f3824017070 con 0x7f383406e930 2024-08-03T20:56:56.358 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:56.354+0000 7f383b200640 1 -- 172.21.15.32:0/206210973 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f38341f8d30 con 0x7f383406e930 2024-08-03T20:56:56.358 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:56.354+0000 7f3830c00640 1 -- 172.21.15.32:0/206210973 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3824011cc0 con 0x7f383406e930 2024-08-03T20:56:56.358 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:56.354+0000 7f383b200640 1 -- 172.21.15.32:0/206210973 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f38341f92f0 con 0x7f383406e930 2024-08-03T20:56:56.358 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:56.355+0000 7f383b200640 1 -- 172.21.15.32:0/206210973 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3834160c30 con 0x7f383406e930 2024-08-03T20:56:56.358 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:56.357+0000 7f3830c00640 1 -- 172.21.15.32:0/206210973 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f382401c460 con 0x7f383406e930 2024-08-03T20:56:56.359 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:56.358+0000 7f3830c00640 1 --2- 172.21.15.32:0/206210973 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f3810077010 0x7f3810079520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:56:56.359 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:56.358+0000 7f3830c00640 1 -- 172.21.15.32:0/206210973 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(101..101 src has 1..101) ==== 7230+0+0 (secure 0 0 0) 0x7f3824098cb0 con 0x7f383406e930 2024-08-03T20:56:56.360 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:56.358+0000 7f382b400640 1 --2- 172.21.15.32:0/206210973 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f3810077010 0x7f3810079520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:56:56.360 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:56.359+0000 7f382b400640 1 --2- 172.21.15.32:0/206210973 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f3810077010 0x7f3810079520 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7f381c034000 tx=0x7f381c004730 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:56:56.363 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:56.362+0000 7f3830c00640 1 -- 172.21.15.32:0/206210973 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7f38240618b0 con 0x7f383406e930 2024-08-03T20:56:56.609 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:56.608+0000 7f383b200640 1 -- 172.21.15.32:0/206210973 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f38340703d0 con 0x7f383406e930 2024-08-03T20:56:56.610 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:56.608+0000 7f3830c00640 1 -- 172.21.15.32:0/206210973 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 185 v185) ==== 95+0+4391 (secure 0 0 0) 0x7f38240611b0 con 0x7f383406e930 2024-08-03T20:56:56.610 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-03T20:56:56.610 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":185,"btime":"2024-08-03T20:56:56:101583+0000","default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6439,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6835/531080160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6834","nonce":531080160},{"type":"v1","addr":"172.21.15.118:6835","nonce":531080160}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":84},{"gid":6454,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6837/2796882319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6836","nonce":2796882319},{"type":"v1","addr":"172.21.15.118:6837","nonce":2796882319}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":86},{"gid":6813,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6834/2686933890","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":2686933890},{"type":"v1","addr":"172.21.15.32:6834","nonce":2686933890}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":85}],"filesystems":[{"mdsmap":{"epoch":185,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-08-03T20:54:26.950518+0000","modified":"2024-08-03T20:56:56.101510+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":101,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":6}]} 2024-08-03T20:56:56.610 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 185 2024-08-03T20:56:56.613 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:56.611+0000 7f383b200640 1 -- 172.21.15.32:0/206210973 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f3810077010 msgr2=0x7f3810079520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:56:56.613 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:56.611+0000 7f383b200640 1 --2- 172.21.15.32:0/206210973 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f3810077010 0x7f3810079520 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7f381c034000 tx=0x7f381c004730 comp rx=0 tx=0).stop 2024-08-03T20:56:56.613 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:56.611+0000 7f383b200640 1 -- 172.21.15.32:0/206210973 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f383406e930 msgr2=0x7f3834140cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:56:56.613 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:56.611+0000 7f383b200640 1 --2- 172.21.15.32:0/206210973 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f383406e930 0x7f3834140cc0 secure :-1 s=READY pgs=1451 cs=0 l=1 rev1=1 crypto rx=0x7f382400c3a0 tx=0x7f382400c860 comp rx=0 tx=0).stop 2024-08-03T20:56:56.613 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:56.611+0000 7f383b200640 1 -- 172.21.15.32:0/206210973 shutdown_connections 2024-08-03T20:56:56.613 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:56.611+0000 7f383b200640 1 --2- 172.21.15.32:0/206210973 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f3810077010 0x7f3810079520 unknown :-1 s=CLOSED pgs=593 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:56.613 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:56.611+0000 7f383b200640 1 --2- 172.21.15.32:0/206210973 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f3834144f90 0x7f38341f4af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:56.613 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:56.611+0000 7f383b200640 1 --2- 172.21.15.32:0/206210973 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f383415fff0 0x7f38340add10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:56.613 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:56.611+0000 7f383b200640 1 --2- 172.21.15.32:0/206210973 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f383406e930 0x7f3834140cc0 unknown :-1 s=CLOSED pgs=1451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:56.613 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:56.611+0000 7f383b200640 1 -- 172.21.15.32:0/206210973 >> 172.21.15.32:0/206210973 conn(0x7f38340b1bd0 msgr2=0x7f38340b2fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:56:56.613 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:56.611+0000 7f383b200640 1 -- 172.21.15.32:0/206210973 shutdown_connections 2024-08-03T20:56:56.613 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:56.611+0000 7f383b200640 1 -- 172.21.15.32:0/206210973 wait complete. 2024-08-03T20:56:56.642 DEBUG:tasks.ceph_test_case:wait_until_true: success in 0s and 0 retries 2024-08-03T20:56:56.642 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-08-03T20:56:56.871 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:56.869+0000 7f76a9e00640 1 -- 172.21.15.32:0/1217937368 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f76a406e930 msgr2=0x7f76a4144f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:56:56.872 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:56.869+0000 7f76a9e00640 1 --2- 172.21.15.32:0/1217937368 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f76a406e930 0x7f76a4144f90 secure :-1 s=READY pgs=1452 cs=0 l=1 rev1=1 crypto rx=0x7f76a4072d80 tx=0x7f769402a560 comp rx=0 tx=0).stop 2024-08-03T20:56:56.872 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:56.869+0000 7f76a9e00640 1 -- 172.21.15.32:0/1217937368 shutdown_connections 2024-08-03T20:56:56.872 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:56.869+0000 7f76a9e00640 1 --2- 172.21.15.32:0/1217937368 >> v2:172.21.15.118:3300/0 conn(0x7f76a415fff0 0x7f76a4160400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:56.872 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:56.869+0000 7f76a9e00640 1 --2- 172.21.15.32:0/1217937368 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f76a406e930 0x7f76a4144f90 unknown :-1 s=CLOSED pgs=1452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:56.872 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:56.869+0000 7f76a9e00640 1 -- 172.21.15.32:0/1217937368 >> 172.21.15.32:0/1217937368 conn(0x7f76a40b1bd0 msgr2=0x7f76a40b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:56:56.872 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:56.869+0000 7f76a9e00640 1 -- 172.21.15.32:0/1217937368 shutdown_connections 2024-08-03T20:56:56.872 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:56.869+0000 7f76a9e00640 1 -- 172.21.15.32:0/1217937368 wait complete. 2024-08-03T20:56:56.873 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:56.871+0000 7f76a9e00640 1 Processor -- start 2024-08-03T20:56:56.873 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:56.871+0000 7f76a9e00640 1 -- start start 2024-08-03T20:56:56.875 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:56.873+0000 7f76a9e00640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f76a406e930 0x7f76a41f8980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:56:56.875 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:56.873+0000 7f76a9e00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f76a4073e60 0x7f76a41f8ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:56:56.875 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:56.873+0000 7f76a9e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f76a415fff0 0x7f76a413d680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:56:56.875 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:56.873+0000 7f76a9e00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f76a40cf840 con 0x7f76a415fff0 2024-08-03T20:56:56.875 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:56.873+0000 7f76a9e00640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f76a4145f30 con 0x7f76a4073e60 2024-08-03T20:56:56.875 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:56.873+0000 7f76a9e00640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7f76a40cf9c0 con 0x7f76a406e930 2024-08-03T20:56:56.875 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:56.873+0000 7f76a3400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f76a415fff0 0x7f76a413d680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:56:56.876 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:56.873+0000 7f76a3400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f76a415fff0 0x7f76a413d680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:40004/0 (socket says 172.21.15.32:40004) 2024-08-03T20:56:56.876 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:56.873+0000 7f76a3400640 1 -- 172.21.15.32:0/3827967442 learned_addr learned my addr 172.21.15.32:0/3827967442 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:56:56.876 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:56.873+0000 7f769a000640 1 --2- 172.21.15.32:0/3827967442 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f76a4073e60 0x7f76a41f8ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:56:56.876 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:56.873+0000 7f76a2a00640 1 --2- 172.21.15.32:0/3827967442 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f76a406e930 0x7f76a41f8980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:56:56.876 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:56.874+0000 7f76a3400640 1 -- 172.21.15.32:0/3827967442 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f76a406e930 msgr2=0x7f76a41f8980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:56:56.876 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:56.874+0000 7f76a3400640 1 --2- 172.21.15.32:0/3827967442 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f76a406e930 0x7f76a41f8980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:56.876 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:56.874+0000 7f76a3400640 1 -- 172.21.15.32:0/3827967442 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f76a4073e60 msgr2=0x7f76a41f8ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:56:56.876 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:56.874+0000 7f76a3400640 1 --2- 172.21.15.32:0/3827967442 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f76a4073e60 0x7f76a41f8ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:56.876 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:56.874+0000 7f76a3400640 1 -- 172.21.15.32:0/3827967442 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f76a415de80 con 0x7f76a415fff0 2024-08-03T20:56:56.876 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:56.874+0000 7f76a2a00640 1 --2- 172.21.15.32:0/3827967442 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f76a406e930 0x7f76a41f8980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-03T20:56:56.876 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:56.874+0000 7f769a000640 1 --2- 172.21.15.32:0/3827967442 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f76a4073e60 0x7f76a41f8ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-03T20:56:56.876 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:56.874+0000 7f76a3400640 1 --2- 172.21.15.32:0/3827967442 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f76a415fff0 0x7f76a413d680 secure :-1 s=READY pgs=1453 cs=0 l=1 rev1=1 crypto rx=0x7f769c00cad0 tx=0x7f769c00cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:56:56.877 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:56.874+0000 7f769be00640 1 -- 172.21.15.32:0/3827967442 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f769c00ae20 con 0x7f76a415fff0 2024-08-03T20:56:56.877 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:56.874+0000 7f769be00640 1 -- 172.21.15.32:0/3827967442 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f769c00f450 con 0x7f76a415fff0 2024-08-03T20:56:56.877 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:56.874+0000 7f769be00640 1 -- 172.21.15.32:0/3827967442 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f769c00f8c0 con 0x7f76a415fff0 2024-08-03T20:56:56.877 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:56.874+0000 7f76a9e00640 1 -- 172.21.15.32:0/3827967442 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f76a4144000 con 0x7f76a415fff0 2024-08-03T20:56:56.877 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:56.874+0000 7f76a9e00640 1 -- 172.21.15.32:0/3827967442 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f76a41f9590 con 0x7f76a415fff0 2024-08-03T20:56:56.878 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:56.875+0000 7f76a9e00640 1 -- 172.21.15.32:0/3827967442 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f76a415e050 con 0x7f76a415fff0 2024-08-03T20:56:56.878 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:56.877+0000 7f769be00640 1 -- 172.21.15.32:0/3827967442 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f769c00fa60 con 0x7f76a415fff0 2024-08-03T20:56:56.879 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:56.877+0000 7f769be00640 1 --2- 172.21.15.32:0/3827967442 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f7680077010 0x7f7680079520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:56:56.879 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:56.878+0000 7f769be00640 1 -- 172.21.15.32:0/3827967442 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(101..101 src has 1..101) ==== 7230+0+0 (secure 0 0 0) 0x7f769c099aa0 con 0x7f76a415fff0 2024-08-03T20:56:56.879 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:56.878+0000 7f76a2a00640 1 --2- 172.21.15.32:0/3827967442 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f7680077010 0x7f7680079520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:56:56.880 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:56.879+0000 7f76a2a00640 1 --2- 172.21.15.32:0/3827967442 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f7680077010 0x7f7680079520 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7f7694033000 tx=0x7f7694004730 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:56:56.883 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:56.882+0000 7f769be00640 1 -- 172.21.15.32:0/3827967442 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7f769c014030 con 0x7f76a415fff0 2024-08-03T20:56:57.134 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:57.130+0000 7f76a9e00640 1 -- 172.21.15.32:0/3827967442 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f76a41441d0 con 0x7f76a415fff0 2024-08-03T20:56:57.135 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:57.131+0000 7f769be00640 1 -- 172.21.15.32:0/3827967442 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 185 v185) ==== 95+0+4391 (secure 0 0 0) 0x7f769c0626a0 con 0x7f76a415fff0 2024-08-03T20:56:57.135 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-03T20:56:57.135 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":185,"btime":"2024-08-03T20:56:56:101583+0000","default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6439,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6835/531080160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6834","nonce":531080160},{"type":"v1","addr":"172.21.15.118:6835","nonce":531080160}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":84},{"gid":6454,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6837/2796882319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6836","nonce":2796882319},{"type":"v1","addr":"172.21.15.118:6837","nonce":2796882319}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":86},{"gid":6813,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6834/2686933890","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":2686933890},{"type":"v1","addr":"172.21.15.32:6834","nonce":2686933890}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":85}],"filesystems":[{"mdsmap":{"epoch":185,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-08-03T20:54:26.950518+0000","modified":"2024-08-03T20:56:56.101510+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":101,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":6}]} 2024-08-03T20:56:57.136 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 185 2024-08-03T20:56:57.136 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:57.133+0000 7f76a9e00640 1 -- 172.21.15.32:0/3827967442 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f7680077010 msgr2=0x7f7680079520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:56:57.136 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:57.133+0000 7f76a9e00640 1 --2- 172.21.15.32:0/3827967442 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f7680077010 0x7f7680079520 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7f7694033000 tx=0x7f7694004730 comp rx=0 tx=0).stop 2024-08-03T20:56:57.136 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:57.133+0000 7f76a9e00640 1 -- 172.21.15.32:0/3827967442 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f76a415fff0 msgr2=0x7f76a413d680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:56:57.136 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:57.133+0000 7f76a9e00640 1 --2- 172.21.15.32:0/3827967442 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f76a415fff0 0x7f76a413d680 secure :-1 s=READY pgs=1453 cs=0 l=1 rev1=1 crypto rx=0x7f769c00cad0 tx=0x7f769c00cf90 comp rx=0 tx=0).stop 2024-08-03T20:56:57.136 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:57.133+0000 7f76a9e00640 1 -- 172.21.15.32:0/3827967442 shutdown_connections 2024-08-03T20:56:57.137 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:57.133+0000 7f76a9e00640 1 --2- 172.21.15.32:0/3827967442 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f7680077010 0x7f7680079520 unknown :-1 s=CLOSED pgs=594 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:57.137 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:57.133+0000 7f76a9e00640 1 --2- 172.21.15.32:0/3827967442 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f76a4073e60 0x7f76a41f8ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:57.137 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:57.133+0000 7f76a9e00640 1 --2- 172.21.15.32:0/3827967442 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f76a415fff0 0x7f76a413d680 unknown :-1 s=CLOSED pgs=1453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:57.137 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:57.133+0000 7f76a9e00640 1 --2- 172.21.15.32:0/3827967442 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f76a406e930 0x7f76a41f8980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:57.137 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:57.133+0000 7f76a9e00640 1 -- 172.21.15.32:0/3827967442 >> 172.21.15.32:0/3827967442 conn(0x7f76a40b1bd0 msgr2=0x7f76a40b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:56:57.137 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:57.133+0000 7f76a9e00640 1 -- 172.21.15.32:0/3827967442 shutdown_connections 2024-08-03T20:56:57.137 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:57.133+0000 7f76a9e00640 1 -- 172.21.15.32:0/3827967442 wait complete. 2024-08-03T20:56:57.160 DEBUG:tasks.ceph_test_case:wait_until_true: success in 0s and 0 retries 2024-08-03T20:56:57.161 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-08-03T20:56:57.389 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:57.387+0000 7fa953000640 1 -- 172.21.15.32:0/4225118115 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa94c15fff0 msgr2=0x7fa94c160400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:56:57.389 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:57.387+0000 7fa953000640 1 --2- 172.21.15.32:0/4225118115 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa94c15fff0 0x7fa94c160400 secure :-1 s=READY pgs=1454 cs=0 l=1 rev1=1 crypto rx=0x7fa94c071700 tx=0x7fa94402aac0 comp rx=0 tx=0).stop 2024-08-03T20:56:57.389 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:57.387+0000 7fa953000640 1 -- 172.21.15.32:0/4225118115 shutdown_connections 2024-08-03T20:56:57.389 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:57.387+0000 7fa953000640 1 --2- 172.21.15.32:0/4225118115 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa94c15fff0 0x7fa94c160400 unknown :-1 s=CLOSED pgs=1454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:57.389 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:57.387+0000 7fa953000640 1 --2- 172.21.15.32:0/4225118115 >> v2:172.21.15.118:3300/0 conn(0x7fa94c06e930 0x7fa94c144f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:57.389 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:57.387+0000 7fa953000640 1 -- 172.21.15.32:0/4225118115 >> 172.21.15.32:0/4225118115 conn(0x7fa94c0b1bd0 msgr2=0x7fa94c0b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:56:57.389 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:57.387+0000 7fa953000640 1 -- 172.21.15.32:0/4225118115 shutdown_connections 2024-08-03T20:56:57.390 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:57.387+0000 7fa953000640 1 -- 172.21.15.32:0/4225118115 wait complete. 2024-08-03T20:56:57.391 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:57.389+0000 7fa953000640 1 Processor -- start 2024-08-03T20:56:57.391 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:57.390+0000 7fa953000640 1 -- start start 2024-08-03T20:56:57.394 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:57.391+0000 7fa953000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa94c06e930 0x7fa94c13dbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:56:57.394 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:57.391+0000 7fa953000640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fa94c073e60 0x7fa94c1e7020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:56:57.394 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:57.391+0000 7fa953000640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa94c15fff0 0x7fa94c1e7560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:56:57.394 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:57.391+0000 7fa953000640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fa94c0cf840 con 0x7fa94c06e930 2024-08-03T20:56:57.394 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:57.391+0000 7fa953000640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7fa94c145f30 con 0x7fa94c15fff0 2024-08-03T20:56:57.394 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:57.391+0000 7fa953000640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7fa94c0cf9c0 con 0x7fa94c073e60 2024-08-03T20:56:57.394 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:57.391+0000 7fa94be00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa94c06e930 0x7fa94c13dbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:56:57.394 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:57.391+0000 7fa94be00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa94c06e930 0x7fa94c13dbf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:40018/0 (socket says 172.21.15.32:40018) 2024-08-03T20:56:57.395 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:57.391+0000 7fa94be00640 1 -- 172.21.15.32:0/612006879 learned_addr learned my addr 172.21.15.32:0/612006879 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:56:57.395 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:57.392+0000 7fa943400640 1 --2- 172.21.15.32:0/612006879 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fa94c073e60 0x7fa94c1e7020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:56:57.395 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:57.392+0000 7fa951000640 1 --2- 172.21.15.32:0/612006879 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa94c15fff0 0x7fa94c1e7560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:56:57.395 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:57.392+0000 7fa94be00640 1 -- 172.21.15.32:0/612006879 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fa94c073e60 msgr2=0x7fa94c1e7020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:56:57.395 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:57.392+0000 7fa94be00640 1 --2- 172.21.15.32:0/612006879 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fa94c073e60 0x7fa94c1e7020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:57.395 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:57.392+0000 7fa94be00640 1 -- 172.21.15.32:0/612006879 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa94c15fff0 msgr2=0x7fa94c1e7560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:56:57.395 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:57.392+0000 7fa94be00640 1 --2- 172.21.15.32:0/612006879 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa94c15fff0 0x7fa94c1e7560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:57.395 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:57.392+0000 7fa94be00640 1 -- 172.21.15.32:0/612006879 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa94c1f3090 con 0x7fa94c06e930 2024-08-03T20:56:57.395 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:57.392+0000 7fa943400640 1 --2- 172.21.15.32:0/612006879 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fa94c073e60 0x7fa94c1e7020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-03T20:56:57.395 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:57.392+0000 7fa951000640 1 --2- 172.21.15.32:0/612006879 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa94c15fff0 0x7fa94c1e7560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-03T20:56:57.395 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:57.392+0000 7fa94be00640 1 --2- 172.21.15.32:0/612006879 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa94c06e930 0x7fa94c13dbf0 secure :-1 s=READY pgs=1455 cs=0 l=1 rev1=1 crypto rx=0x7fa93c00d7a0 tx=0x7fa93c00dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:56:57.395 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:57.392+0000 7fa948c00640 1 -- 172.21.15.32:0/612006879 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa93c017070 con 0x7fa94c06e930 2024-08-03T20:56:57.395 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:57.392+0000 7fa948c00640 1 -- 172.21.15.32:0/612006879 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fa93c009d20 con 0x7fa94c06e930 2024-08-03T20:56:57.395 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:57.392+0000 7fa948c00640 1 -- 172.21.15.32:0/612006879 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa93c004280 con 0x7fa94c06e930 2024-08-03T20:56:57.396 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:57.392+0000 7fa953000640 1 -- 172.21.15.32:0/612006879 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa94c13e130 con 0x7fa94c06e930 2024-08-03T20:56:57.396 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:57.393+0000 7fa953000640 1 -- 172.21.15.32:0/612006879 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa94c143f00 con 0x7fa94c06e930 2024-08-03T20:56:57.396 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:57.394+0000 7fa953000640 1 -- 172.21.15.32:0/612006879 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa94c160be0 con 0x7fa94c06e930 2024-08-03T20:56:57.396 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:57.395+0000 7fa948c00640 1 -- 172.21.15.32:0/612006879 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7fa93c005020 con 0x7fa94c06e930 2024-08-03T20:56:57.397 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:57.396+0000 7fa948c00640 1 --2- 172.21.15.32:0/612006879 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fa928077040 0x7fa928079550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:56:57.397 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:57.396+0000 7fa948c00640 1 -- 172.21.15.32:0/612006879 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(101..101 src has 1..101) ==== 7230+0+0 (secure 0 0 0) 0x7fa93c099300 con 0x7fa94c06e930 2024-08-03T20:56:57.397 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:57.396+0000 7fa943400640 1 --2- 172.21.15.32:0/612006879 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fa928077040 0x7fa928079550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:56:57.398 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:57.397+0000 7fa943400640 1 --2- 172.21.15.32:0/612006879 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fa928077040 0x7fa928079550 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7fa94c1406a0 tx=0x7fa934009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:56:57.401 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:57.400+0000 7fa948c00640 1 -- 172.21.15.32:0/612006879 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7fa93c020020 con 0x7fa94c06e930 2024-08-03T20:56:57.645 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:57.644+0000 7fa953000640 1 -- 172.21.15.32:0/612006879 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa94c074480 con 0x7fa94c06e930 2024-08-03T20:56:57.646 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:57.644+0000 7fa948c00640 1 -- 172.21.15.32:0/612006879 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 185 v185) ==== 95+0+4391 (secure 0 0 0) 0x7fa93c061f00 con 0x7fa94c06e930 2024-08-03T20:56:57.646 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-03T20:56:57.646 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":185,"btime":"2024-08-03T20:56:56:101583+0000","default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6439,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6835/531080160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6834","nonce":531080160},{"type":"v1","addr":"172.21.15.118:6835","nonce":531080160}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":84},{"gid":6454,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6837/2796882319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6836","nonce":2796882319},{"type":"v1","addr":"172.21.15.118:6837","nonce":2796882319}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":86},{"gid":6813,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6834/2686933890","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":2686933890},{"type":"v1","addr":"172.21.15.32:6834","nonce":2686933890}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":85}],"filesystems":[{"mdsmap":{"epoch":185,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-08-03T20:54:26.950518+0000","modified":"2024-08-03T20:56:56.101510+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":101,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":6}]} 2024-08-03T20:56:57.647 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 185 2024-08-03T20:56:57.648 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:57.647+0000 7fa953000640 1 -- 172.21.15.32:0/612006879 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fa928077040 msgr2=0x7fa928079550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:56:57.649 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:57.647+0000 7fa953000640 1 --2- 172.21.15.32:0/612006879 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fa928077040 0x7fa928079550 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7fa94c1406a0 tx=0x7fa934009040 comp rx=0 tx=0).stop 2024-08-03T20:56:57.649 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:57.647+0000 7fa953000640 1 -- 172.21.15.32:0/612006879 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa94c06e930 msgr2=0x7fa94c13dbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:56:57.649 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:57.647+0000 7fa953000640 1 --2- 172.21.15.32:0/612006879 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa94c06e930 0x7fa94c13dbf0 secure :-1 s=READY pgs=1455 cs=0 l=1 rev1=1 crypto rx=0x7fa93c00d7a0 tx=0x7fa93c00dc60 comp rx=0 tx=0).stop 2024-08-03T20:56:57.649 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:57.647+0000 7fa953000640 1 -- 172.21.15.32:0/612006879 shutdown_connections 2024-08-03T20:56:57.649 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:57.647+0000 7fa953000640 1 --2- 172.21.15.32:0/612006879 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fa928077040 0x7fa928079550 unknown :-1 s=CLOSED pgs=595 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:57.649 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:57.647+0000 7fa953000640 1 --2- 172.21.15.32:0/612006879 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa94c15fff0 0x7fa94c1e7560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:57.649 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:57.647+0000 7fa953000640 1 --2- 172.21.15.32:0/612006879 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fa94c073e60 0x7fa94c1e7020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:57.649 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:57.647+0000 7fa953000640 1 --2- 172.21.15.32:0/612006879 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa94c06e930 0x7fa94c13dbf0 unknown :-1 s=CLOSED pgs=1455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:57.649 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:57.647+0000 7fa953000640 1 -- 172.21.15.32:0/612006879 >> 172.21.15.32:0/612006879 conn(0x7fa94c0b1bd0 msgr2=0x7fa94c0b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:56:57.649 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:57.647+0000 7fa953000640 1 -- 172.21.15.32:0/612006879 shutdown_connections 2024-08-03T20:56:57.649 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:57.647+0000 7fa953000640 1 -- 172.21.15.32:0/612006879 wait complete. 2024-08-03T20:56:57.674 INFO:tasks.cephfs.test_damage:Result: Mutation 'Corrupt 1.00000000.inode' led to DAMAGED state 2024-08-03T20:56:57.674 INFO:tasks.cephfs.test_damage:Applying mutation 'Corrupt 200.00000000' 2024-08-03T20:56:57.674 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-08-03T20:56:57.675 DEBUG:teuthology.orchestra.run.smithi032:> dd if=/proc/self/mounts of=/dev/stdout 2024-08-03T20:56:57.711 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053453,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6499208k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 none /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249604k,nr_inodes=812401,mode=700,uid=1000,gid=1316 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6499208k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-08-03T20:56:57.712 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi032.front.sepia.ceph.com 2024-08-03T20:56:57.712 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-08-03T20:56:57.712 DEBUG:teuthology.orchestra.run.smithi032:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-08-03T20:56:57.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-03T20:56:57.786 INFO:teuthology.orchestra.run.smithi032.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.0': No such file or directory 2024-08-03T20:56:57.787 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-08-03T20:56:58.056 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:58.054+0000 7f7e6a200640 1 -- 172.21.15.32:0/2699027458 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7e6406fb50 msgr2=0x7f7e640ad190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:56:58.057 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:58.054+0000 7f7e6a200640 1 --2- 172.21.15.32:0/2699027458 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7e6406fb50 0x7f7e640ad190 secure :-1 s=READY pgs=1457 cs=0 l=1 rev1=1 crypto rx=0x7f7e5c005fd0 tx=0x7f7e5c02a8a0 comp rx=0 tx=0).stop 2024-08-03T20:56:58.057 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:58.054+0000 7f7e6a200640 1 -- 172.21.15.32:0/2699027458 shutdown_connections 2024-08-03T20:56:58.057 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:58.054+0000 7f7e6a200640 1 --2- 172.21.15.32:0/2699027458 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7e6406fb50 0x7f7e640ad190 unknown :-1 s=CLOSED pgs=1457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:58.057 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:58.054+0000 7f7e6a200640 1 -- 172.21.15.32:0/2699027458 >> 172.21.15.32:0/2699027458 conn(0x7f7e640b2120 msgr2=0x7f7e640cfd60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:56:58.057 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:58.054+0000 7f7e6a200640 1 -- 172.21.15.32:0/2699027458 shutdown_connections 2024-08-03T20:56:58.057 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:58.054+0000 7f7e6a200640 1 -- 172.21.15.32:0/2699027458 wait complete. 2024-08-03T20:56:58.058 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:58.056+0000 7f7e6a200640 1 Processor -- start 2024-08-03T20:56:58.059 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:58.056+0000 7f7e6a200640 1 -- start start 2024-08-03T20:56:58.063 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:58.058+0000 7f7e6a200640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f7e6406fb50 0x7f7e640b2c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:56:58.063 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:58.058+0000 7f7e6a200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7e640acd80 0x7f7e641555d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:56:58.063 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:58.058+0000 7f7e6a200640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f7e640af020 0x7f7e640c9260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:56:58.063 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:58.058+0000 7f7e6a200640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f7e6406efb0 con 0x7f7e640acd80 2024-08-03T20:56:58.063 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:58.058+0000 7f7e6a200640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f7e640725f0 con 0x7f7e640af020 2024-08-03T20:56:58.063 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:58.058+0000 7f7e6a200640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7f7e64168430 con 0x7f7e6406fb50 2024-08-03T20:56:58.064 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:58.058+0000 7f7e63e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7e640acd80 0x7f7e641555d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:56:58.064 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:58.058+0000 7f7e63e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7e640acd80 0x7f7e641555d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:40040/0 (socket says 172.21.15.32:40040) 2024-08-03T20:56:58.064 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:58.058+0000 7f7e63e00640 1 -- 172.21.15.32:0/986292631 learned_addr learned my addr 172.21.15.32:0/986292631 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:56:58.064 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:58.058+0000 7f7e68e00640 1 --2- 172.21.15.32:0/986292631 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f7e6406fb50 0x7f7e640b2c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:56:58.064 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:58.058+0000 7f7e69800640 1 --2- 172.21.15.32:0/986292631 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f7e640af020 0x7f7e640c9260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:56:58.064 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:58.059+0000 7f7e63e00640 1 -- 172.21.15.32:0/986292631 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f7e6406fb50 msgr2=0x7f7e640b2c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:56:58.065 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:58.059+0000 7f7e63e00640 1 --2- 172.21.15.32:0/986292631 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f7e6406fb50 0x7f7e640b2c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:58.065 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:58.059+0000 7f7e63e00640 1 -- 172.21.15.32:0/986292631 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f7e640af020 msgr2=0x7f7e640c9260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:56:58.065 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:58.059+0000 7f7e63e00640 1 --2- 172.21.15.32:0/986292631 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f7e640af020 0x7f7e640c9260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:58.065 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:58.059+0000 7f7e63e00640 1 -- 172.21.15.32:0/986292631 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7e641f8c60 con 0x7f7e640acd80 2024-08-03T20:56:58.065 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:58.059+0000 7f7e63e00640 1 --2- 172.21.15.32:0/986292631 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7e640acd80 0x7f7e641555d0 secure :-1 s=READY pgs=1458 cs=0 l=1 rev1=1 crypto rx=0x7f7e5000c430 tx=0x7f7e5000c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:56:58.065 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:58.059+0000 7f7e60c00640 1 -- 172.21.15.32:0/986292631 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e50011790 con 0x7f7e640acd80 2024-08-03T20:56:58.066 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:58.059+0000 7f7e60c00640 1 -- 172.21.15.32:0/986292631 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f7e50017070 con 0x7f7e640acd80 2024-08-03T20:56:58.066 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:58.059+0000 7f7e60c00640 1 -- 172.21.15.32:0/986292631 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e50011ca0 con 0x7f7e640acd80 2024-08-03T20:56:58.066 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:58.059+0000 7f7e6a200640 1 -- 172.21.15.32:0/986292631 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7e641f8f40 con 0x7f7e640acd80 2024-08-03T20:56:58.066 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:58.059+0000 7f7e6a200640 1 -- 172.21.15.32:0/986292631 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7e641f9500 con 0x7f7e640acd80 2024-08-03T20:56:58.066 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:58.061+0000 7f7e6a200640 1 -- 172.21.15.32:0/986292631 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7e640ad190 con 0x7f7e640acd80 2024-08-03T20:56:58.068 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:58.062+0000 7f7e60c00640 1 -- 172.21.15.32:0/986292631 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f7e5001c460 con 0x7f7e640acd80 2024-08-03T20:56:58.068 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:58.063+0000 7f7e60c00640 1 --2- 172.21.15.32:0/986292631 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f7e44077040 0x7f7e44079550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:56:58.068 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:58.063+0000 7f7e60c00640 1 -- 172.21.15.32:0/986292631 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(101..101 src has 1..101) ==== 7230+0+0 (secure 0 0 0) 0x7f7e50098bc0 con 0x7f7e640acd80 2024-08-03T20:56:58.068 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:58.063+0000 7f7e68e00640 1 --2- 172.21.15.32:0/986292631 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f7e44077040 0x7f7e44079550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:56:58.069 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:58.064+0000 7f7e68e00640 1 --2- 172.21.15.32:0/986292631 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f7e44077040 0x7f7e44079550 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7f7e58001720 tx=0x7f7e58009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:56:58.069 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:58.067+0000 7f7e60c00640 1 -- 172.21.15.32:0/986292631 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7f7e50061170 con 0x7f7e640acd80 2024-08-03T20:56:58.311 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:58.309+0000 7f7e6a200640 1 -- 172.21.15.32:0/986292631 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7f7e640d0930 con 0x7f7e640acd80 2024-08-03T20:56:59.114 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:59.112+0000 7f7e60c00640 1 -- 172.21.15.32:0/986292631 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v186) ==== 161+0+0 (secure 0 0 0) 0x7f7e500656c0 con 0x7f7e640acd80 2024-08-03T20:56:59.114 INFO:teuthology.orchestra.run.smithi032.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-08-03T20:56:59.118 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:59.116+0000 7f7e6a200640 1 -- 172.21.15.32:0/986292631 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f7e44077040 msgr2=0x7f7e44079550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:56:59.118 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:59.116+0000 7f7e6a200640 1 --2- 172.21.15.32:0/986292631 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f7e44077040 0x7f7e44079550 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7f7e58001720 tx=0x7f7e58009040 comp rx=0 tx=0).stop 2024-08-03T20:56:59.118 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:59.116+0000 7f7e6a200640 1 -- 172.21.15.32:0/986292631 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7e640acd80 msgr2=0x7f7e641555d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:56:59.119 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:59.116+0000 7f7e6a200640 1 --2- 172.21.15.32:0/986292631 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7e640acd80 0x7f7e641555d0 secure :-1 s=READY pgs=1458 cs=0 l=1 rev1=1 crypto rx=0x7f7e5000c430 tx=0x7f7e5000c8f0 comp rx=0 tx=0).stop 2024-08-03T20:56:59.119 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:59.116+0000 7f7e6a200640 1 -- 172.21.15.32:0/986292631 shutdown_connections 2024-08-03T20:56:59.119 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:59.116+0000 7f7e6a200640 1 --2- 172.21.15.32:0/986292631 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f7e44077040 0x7f7e44079550 unknown :-1 s=CLOSED pgs=596 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:59.119 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:59.116+0000 7f7e6a200640 1 --2- 172.21.15.32:0/986292631 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f7e640af020 0x7f7e640c9260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:59.119 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:59.116+0000 7f7e6a200640 1 --2- 172.21.15.32:0/986292631 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7e640acd80 0x7f7e641555d0 unknown :-1 s=CLOSED pgs=1458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:59.120 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:59.116+0000 7f7e6a200640 1 --2- 172.21.15.32:0/986292631 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f7e6406fb50 0x7f7e640b2c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:59.120 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:59.116+0000 7f7e6a200640 1 -- 172.21.15.32:0/986292631 >> 172.21.15.32:0/986292631 conn(0x7f7e640b2120 msgr2=0x7f7e640b11c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:56:59.120 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:59.116+0000 7f7e6a200640 1 -- 172.21.15.32:0/986292631 shutdown_connections 2024-08-03T20:56:59.120 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:59.116+0000 7f7e6a200640 1 -- 172.21.15.32:0/986292631 wait complete. 2024-08-03T20:56:59.144 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mds repaired 0 2024-08-03T20:56:59.378 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:59.376+0000 7f7d99200640 1 -- 172.21.15.32:0/745494828 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7d78004d40 msgr2=0x7f7d78005150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:56:59.378 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:59.376+0000 7f7d99200640 1 --2- 172.21.15.32:0/745494828 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7d78004d40 0x7f7d78005150 secure :-1 s=READY pgs=1460 cs=0 l=1 rev1=1 crypto rx=0x7f7d8c005fd0 tx=0x7f7d78005830 comp rx=0 tx=0).stop 2024-08-03T20:56:59.378 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:59.376+0000 7f7d93e00640 1 -- 172.21.15.32:0/745494828 reap_dead start 2024-08-03T20:56:59.378 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:59.376+0000 7f7d99200640 1 -- 172.21.15.32:0/745494828 shutdown_connections 2024-08-03T20:56:59.378 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:59.376+0000 7f7d99200640 1 -- 172.21.15.32:0/745494828 >> 172.21.15.32:0/745494828 conn(0x7f7d940b2120 msgr2=0x7f7d940cfd60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:56:59.378 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:59.376+0000 7f7d99200640 1 -- 172.21.15.32:0/745494828 shutdown_connections 2024-08-03T20:56:59.379 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:59.376+0000 7f7d99200640 1 -- 172.21.15.32:0/745494828 wait complete. 2024-08-03T20:56:59.380 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:59.378+0000 7f7d99200640 1 Processor -- start 2024-08-03T20:56:59.380 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:59.378+0000 7f7d99200640 1 -- start start 2024-08-03T20:56:59.383 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:59.380+0000 7f7d99200640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f7d78004d40 0x7f7d9409baa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:56:59.383 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:59.380+0000 7f7d99200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7d94155aa0 0x7f7d9409bfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:56:59.383 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:59.380+0000 7f7d99200640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f7d9409c520 0x7f7d94155fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:56:59.383 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:59.380+0000 7f7d99200640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f7d94070860 con 0x7f7d94155aa0 2024-08-03T20:56:59.383 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:59.380+0000 7f7d99200640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f7d940706e0 con 0x7f7d9409c520 2024-08-03T20:56:59.384 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:59.380+0000 7f7d99200640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7f7d940ce560 con 0x7f7d78004d40 2024-08-03T20:56:59.384 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:59.380+0000 7f7d8be00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7d94155aa0 0x7f7d9409bfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:56:59.384 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:59.380+0000 7f7d8be00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7d94155aa0 0x7f7d9409bfe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:40050/0 (socket says 172.21.15.32:40050) 2024-08-03T20:56:59.384 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:59.380+0000 7f7d8be00640 1 -- 172.21.15.32:0/894284409 learned_addr learned my addr 172.21.15.32:0/894284409 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:56:59.384 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:59.380+0000 7f7d93400640 1 --2- 172.21.15.32:0/894284409 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f7d78004d40 0x7f7d9409baa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:56:59.384 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:59.380+0000 7f7d93e00640 1 --2- 172.21.15.32:0/894284409 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f7d9409c520 0x7f7d94155fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:56:59.384 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:59.381+0000 7f7d8be00640 1 -- 172.21.15.32:0/894284409 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f7d78004d40 msgr2=0x7f7d9409baa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:56:59.384 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:59.381+0000 7f7d8be00640 1 --2- 172.21.15.32:0/894284409 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f7d78004d40 0x7f7d9409baa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:59.384 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:59.381+0000 7f7d8be00640 1 -- 172.21.15.32:0/894284409 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f7d9409c520 msgr2=0x7f7d94155fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:56:59.384 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:59.381+0000 7f7d8be00640 1 --2- 172.21.15.32:0/894284409 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f7d9409c520 0x7f7d94155fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:56:59.384 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:59.381+0000 7f7d8be00640 1 -- 172.21.15.32:0/894284409 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7d940c8c80 con 0x7f7d94155aa0 2024-08-03T20:56:59.384 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:59.381+0000 7f7d93400640 1 --2- 172.21.15.32:0/894284409 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f7d78004d40 0x7f7d9409baa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-03T20:56:59.384 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:59.381+0000 7f7d8be00640 1 --2- 172.21.15.32:0/894284409 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7d94155aa0 0x7f7d9409bfe0 secure :-1 s=READY pgs=1461 cs=0 l=1 rev1=1 crypto rx=0x7f7d7c006630 tx=0x7f7d7c0039c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:56:59.384 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:59.381+0000 7f7d8b400640 1 -- 172.21.15.32:0/894284409 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d7c015070 con 0x7f7d94155aa0 2024-08-03T20:56:59.384 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:59.381+0000 7f7d8b400640 1 -- 172.21.15.32:0/894284409 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f7d7c00f960 con 0x7f7d94155aa0 2024-08-03T20:56:59.385 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:59.381+0000 7f7d99200640 1 -- 172.21.15.32:0/894284409 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7d940c9120 con 0x7f7d94155aa0 2024-08-03T20:56:59.385 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:59.381+0000 7f7d8b400640 1 -- 172.21.15.32:0/894284409 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d7c00fdb0 con 0x7f7d94155aa0 2024-08-03T20:56:59.385 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:59.381+0000 7f7d99200640 1 -- 172.21.15.32:0/894284409 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7d940c96e0 con 0x7f7d94155aa0 2024-08-03T20:56:59.385 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:59.383+0000 7f7d99200640 1 -- 172.21.15.32:0/894284409 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7d940aefb0 con 0x7f7d94155aa0 2024-08-03T20:56:59.385 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:59.384+0000 7f7d8b400640 1 -- 172.21.15.32:0/894284409 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f7d7c011430 con 0x7f7d94155aa0 2024-08-03T20:56:59.386 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:59.385+0000 7f7d8b400640 1 --2- 172.21.15.32:0/894284409 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f7d70077040 0x7f7d70079550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:56:59.386 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:59.385+0000 7f7d8b400640 1 -- 172.21.15.32:0/894284409 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(101..101 src has 1..101) ==== 7230+0+0 (secure 0 0 0) 0x7f7d7c099c00 con 0x7f7d94155aa0 2024-08-03T20:56:59.386 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:59.385+0000 7f7d93400640 1 --2- 172.21.15.32:0/894284409 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f7d70077040 0x7f7d70079550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:56:59.387 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:59.386+0000 7f7d93400640 1 --2- 172.21.15.32:0/894284409 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f7d70077040 0x7f7d70079550 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7f7d84009cd0 tx=0x7f7d84005f90 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:56:59.390 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:59.389+0000 7f7d8b400640 1 -- 172.21.15.32:0/894284409 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7f7d7c0621c0 con 0x7f7d94155aa0 2024-08-03T20:56:59.678 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:56:59.676+0000 7f7d99200640 1 -- 172.21.15.32:0/894284409 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "mds repaired", "role": "0"} v 0) -- 0x7f7d940c8e50 con 0x7f7d94155aa0 2024-08-03T20:57:00.116 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.114+0000 7f7d8b400640 1 -- 172.21.15.32:0/894284409 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "mds repaired", "role": "0"}]=0 repaired: restoring rank 6:0 v187) ==== 101+0+0 (secure 0 0 0) 0x7f7d7c066710 con 0x7f7d94155aa0 2024-08-03T20:57:00.116 INFO:teuthology.orchestra.run.smithi032.stderr:repaired: restoring rank 6:0 2024-08-03T20:57:00.119 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.118+0000 7f7d99200640 1 -- 172.21.15.32:0/894284409 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f7d70077040 msgr2=0x7f7d70079550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:00.119 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.118+0000 7f7d99200640 1 --2- 172.21.15.32:0/894284409 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f7d70077040 0x7f7d70079550 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7f7d84009cd0 tx=0x7f7d84005f90 comp rx=0 tx=0).stop 2024-08-03T20:57:00.119 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.118+0000 7f7d99200640 1 -- 172.21.15.32:0/894284409 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7d94155aa0 msgr2=0x7f7d9409bfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:00.119 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.118+0000 7f7d99200640 1 --2- 172.21.15.32:0/894284409 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7d94155aa0 0x7f7d9409bfe0 secure :-1 s=READY pgs=1461 cs=0 l=1 rev1=1 crypto rx=0x7f7d7c006630 tx=0x7f7d7c0039c0 comp rx=0 tx=0).stop 2024-08-03T20:57:00.120 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.118+0000 7f7d99200640 1 -- 172.21.15.32:0/894284409 shutdown_connections 2024-08-03T20:57:00.120 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.118+0000 7f7d99200640 1 --2- 172.21.15.32:0/894284409 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f7d70077040 0x7f7d70079550 unknown :-1 s=CLOSED pgs=597 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:00.120 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.118+0000 7f7d99200640 1 --2- 172.21.15.32:0/894284409 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f7d9409c520 0x7f7d94155fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:00.120 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.118+0000 7f7d99200640 1 --2- 172.21.15.32:0/894284409 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7d94155aa0 0x7f7d9409bfe0 unknown :-1 s=CLOSED pgs=1461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:00.120 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.118+0000 7f7d99200640 1 --2- 172.21.15.32:0/894284409 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f7d78004d40 0x7f7d9409baa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:00.120 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.118+0000 7f7d99200640 1 -- 172.21.15.32:0/894284409 >> 172.21.15.32:0/894284409 conn(0x7f7d940b2120 msgr2=0x7f7d940b4c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:57:00.120 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.118+0000 7f7d99200640 1 -- 172.21.15.32:0/894284409 shutdown_connections 2024-08-03T20:57:00.120 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.118+0000 7f7d99200640 1 -- 172.21.15.32:0/894284409 wait complete. 2024-08-03T20:57:00.145 DEBUG:teuthology.orchestra.run.smithi032:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata import - 2024-08-03T20:57:00.287 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.286+0000 7ff509475f40 1 Processor -- start 2024-08-03T20:57:00.288 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.286+0000 7ff509475f40 1 -- start start 2024-08-03T20:57:00.289 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.287+0000 7ff509475f40 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555e5de7c360 0x555e5de0b5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:00.289 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.287+0000 7ff509475f40 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x555e5de99cb0 0x555e5de95950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:00.289 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.287+0000 7ff509475f40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x555e5de9a3f0 0x555e5de97f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:00.289 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.287+0000 7ff509475f40 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x555e5dd13e30 con 0x555e5de7c360 2024-08-03T20:57:00.290 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.287+0000 7ff509475f40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x555e5dd71bf0 con 0x555e5de9a3f0 2024-08-03T20:57:00.290 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.287+0000 7ff509475f40 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x555e5de7d440 con 0x555e5de99cb0 2024-08-03T20:57:00.290 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.288+0000 7ff502600640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555e5de7c360 0x555e5de0b5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:00.290 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.288+0000 7ff502600640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555e5de7c360 0x555e5de0b5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:46892/0 (socket says 172.21.15.32:46892) 2024-08-03T20:57:00.290 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.288+0000 7ff502600640 1 -- 172.21.15.32:0/3484029976 learned_addr learned my addr 172.21.15.32:0/3484029976 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:57:00.290 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.288+0000 7ff501c00640 1 --2- 172.21.15.32:0/3484029976 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x555e5de99cb0 0x555e5de95950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:00.290 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.288+0000 7ff503000640 1 --2- 172.21.15.32:0/3484029976 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x555e5de9a3f0 0x555e5de97f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:00.292 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.288+0000 7ff502600640 1 -- 172.21.15.32:0/3484029976 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x555e5de99cb0 msgr2=0x555e5de95950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:57:00.292 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.288+0000 7ff502600640 1 --2- 172.21.15.32:0/3484029976 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x555e5de99cb0 0x555e5de95950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:00.292 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.288+0000 7ff502600640 1 -- 172.21.15.32:0/3484029976 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x555e5de9a3f0 msgr2=0x555e5de97f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:57:00.292 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.288+0000 7ff502600640 1 --2- 172.21.15.32:0/3484029976 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x555e5de9a3f0 0x555e5de97f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:00.292 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.288+0000 7ff502600640 1 -- 172.21.15.32:0/3484029976 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555e5de95e90 con 0x555e5de7c360 2024-08-03T20:57:00.292 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.289+0000 7ff502600640 1 --2- 172.21.15.32:0/3484029976 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555e5de7c360 0x555e5de0b5f0 secure :-1 s=READY pgs=1464 cs=0 l=1 rev1=1 crypto rx=0x7ff4f4000c00 tx=0x7ff4f4002fb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5fda87890887f32b server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:57:00.292 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.289+0000 7ff4fbe00640 1 -- 172.21.15.32:0/3484029976 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4f404dcd0 con 0x555e5de7c360 2024-08-03T20:57:00.292 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.289+0000 7ff4fbe00640 1 -- 172.21.15.32:0/3484029976 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7ff4f4056470 con 0x555e5de7c360 2024-08-03T20:57:00.292 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.289+0000 7ff4fbe00640 1 -- 172.21.15.32:0/3484029976 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4f4056880 con 0x555e5de7c360 2024-08-03T20:57:00.292 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.289+0000 7ff509475f40 1 -- 172.21.15.32:0/3484029976 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555e5de7c360 msgr2=0x555e5de0b5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:00.292 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.289+0000 7ff509475f40 1 --2- 172.21.15.32:0/3484029976 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555e5de7c360 0x555e5de0b5f0 secure :-1 s=READY pgs=1464 cs=0 l=1 rev1=1 crypto rx=0x7ff4f4000c00 tx=0x7ff4f4002fb0 comp rx=0 tx=0).stop 2024-08-03T20:57:00.293 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.290+0000 7ff509475f40 1 -- 172.21.15.32:0/3484029976 shutdown_connections 2024-08-03T20:57:00.293 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.290+0000 7ff509475f40 1 --2- 172.21.15.32:0/3484029976 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x555e5de9a3f0 0x555e5de97f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:00.293 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.290+0000 7ff509475f40 1 --2- 172.21.15.32:0/3484029976 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x555e5de99cb0 0x555e5de95950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:00.293 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.290+0000 7ff509475f40 1 --2- 172.21.15.32:0/3484029976 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555e5de7c360 0x555e5de0b5f0 unknown :-1 s=CLOSED pgs=1464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:00.293 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.290+0000 7ff509475f40 1 -- 172.21.15.32:0/3484029976 >> 172.21.15.32:0/3484029976 conn(0x555e5dce5ac0 msgr2=0x555e5de98ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:57:00.293 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.290+0000 7ff509475f40 1 -- 172.21.15.32:0/3484029976 shutdown_connections 2024-08-03T20:57:00.293 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.290+0000 7ff509475f40 1 -- 172.21.15.32:0/3484029976 wait complete. 2024-08-03T20:57:00.295 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.291+0000 7ff509475f40 1 Processor -- start 2024-08-03T20:57:00.295 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.291+0000 7ff509475f40 1 -- start start 2024-08-03T20:57:00.296 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.292+0000 7ff509475f40 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555e5de7c360 0x555e5de7a650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:00.296 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.292+0000 7ff509475f40 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x555e5de99cb0 0x555e5de0b260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:00.296 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.292+0000 7ff509475f40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x555e5de9a3f0 0x555e5de0b7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:00.296 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.292+0000 7ff509475f40 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x555e5de7b9e0 con 0x555e5de7c360 2024-08-03T20:57:00.296 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.292+0000 7ff509475f40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x555e5ddc5be0 con 0x555e5de9a3f0 2024-08-03T20:57:00.296 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.292+0000 7ff509475f40 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x555e5de0ab00 con 0x555e5de99cb0 2024-08-03T20:57:00.296 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.293+0000 7ff502600640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555e5de7c360 0x555e5de7a650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:00.296 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.293+0000 7ff502600640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555e5de7c360 0x555e5de7a650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:46904/0 (socket says 172.21.15.32:46904) 2024-08-03T20:57:00.296 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.293+0000 7ff502600640 1 -- 172.21.15.32:0/565639949 learned_addr learned my addr 172.21.15.32:0/565639949 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:57:00.296 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.293+0000 7ff501c00640 1 --2- 172.21.15.32:0/565639949 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x555e5de99cb0 0x555e5de0b260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:00.296 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.293+0000 7ff503000640 1 --2- 172.21.15.32:0/565639949 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x555e5de9a3f0 0x555e5de0b7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:00.296 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.293+0000 7ff502600640 1 -- 172.21.15.32:0/565639949 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x555e5de99cb0 msgr2=0x555e5de0b260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:00.296 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.293+0000 7ff502600640 1 --2- 172.21.15.32:0/565639949 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x555e5de99cb0 0x555e5de0b260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:00.297 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.293+0000 7ff502600640 1 -- 172.21.15.32:0/565639949 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x555e5de9a3f0 msgr2=0x555e5de0b7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:00.297 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.293+0000 7ff502600640 1 --2- 172.21.15.32:0/565639949 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x555e5de9a3f0 0x555e5de0b7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:00.297 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.293+0000 7ff502600640 1 -- 172.21.15.32:0/565639949 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555e5de06320 con 0x555e5de7c360 2024-08-03T20:57:00.297 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.293+0000 7ff502600640 1 --2- 172.21.15.32:0/565639949 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555e5de7c360 0x555e5de7a650 secure :-1 s=READY pgs=1465 cs=0 l=1 rev1=1 crypto rx=0x7ff4f4025db0 tx=0x7ff4f404d640 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:57:00.297 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.294+0000 7ff4faa00640 1 -- 172.21.15.32:0/565639949 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4f4056530 con 0x555e5de7c360 2024-08-03T20:57:00.297 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.294+0000 7ff4faa00640 1 -- 172.21.15.32:0/565639949 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7ff4f40567c0 con 0x555e5de7c360 2024-08-03T20:57:00.297 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.294+0000 7ff509475f40 1 -- 172.21.15.32:0/565639949 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x555e5df32080 con 0x555e5de7c360 2024-08-03T20:57:00.297 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.294+0000 7ff4faa00640 1 -- 172.21.15.32:0/565639949 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4f4056a10 con 0x555e5de7c360 2024-08-03T20:57:00.297 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.294+0000 7ff509475f40 1 -- 172.21.15.32:0/565639949 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555e5de98690 con 0x555e5de7c360 2024-08-03T20:57:00.298 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.296+0000 7ff4faa00640 1 -- 172.21.15.32:0/565639949 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7ff4f4072020 con 0x555e5de7c360 2024-08-03T20:57:00.298 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.297+0000 7ff4faa00640 1 --2- 172.21.15.32:0/565639949 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7ff4e40759f0 0x7ff4e4077f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:00.298 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.297+0000 7ff4faa00640 1 -- 172.21.15.32:0/565639949 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(101..101 src has 1..101) ==== 7230+0+0 (secure 0 0 0) 0x7ff4f40dce70 con 0x555e5de7c360 2024-08-03T20:57:00.299 INFO:teuthology.orchestra.run.smithi032.stdout:Importing pool 2024-08-03T20:57:00.300 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.297+0000 7ff501c00640 1 --2- 172.21.15.32:0/565639949 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7ff4e40759f0 0x7ff4e4077f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:00.300 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.297+0000 7ff509475f40 1 --2- 172.21.15.32:0/565639949 >> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] conn(0x555e5de94b60 0x555e5de04dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:00.300 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.297+0000 7ff509475f40 1 -- 172.21.15.32:0/565639949 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:1 12.10 12:09401a81:::601.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101) -- 0x555e5de919b0 con 0x555e5de94b60 2024-08-03T20:57:00.300 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.298+0000 7ff503000640 1 --2- 172.21.15.32:0/565639949 >> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] conn(0x555e5de94b60 0x555e5de04dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:00.300 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.298+0000 7ff501c00640 1 --2- 172.21.15.32:0/565639949 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7ff4e40759f0 0x7ff4e4077f00 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x555e5ddc4af0 tx=0x7ff4f0001690 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:57:00.300 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.298+0000 7ff503000640 1 --2- 172.21.15.32:0/565639949 >> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] conn(0x555e5de94b60 0x555e5de04dc0 crc :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:57:00.302 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.301+0000 7ff503000640 1 -- 172.21.15.32:0/565639949 <== osd.0 v2:172.21.15.32:6816/3066156116 1 ==== osd_op_reply(1 601.00000000 [create] v101'474 uv448 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7ff4fc003040 con 0x555e5de94b60 2024-08-03T20:57:00.302 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.301+0000 7ff509475f40 1 -- 172.21.15.32:0/565639949 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:2 12.10 12:09401a81:::601.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e101) -- 0x555e5df3e7a0 con 0x555e5de94b60 2024-08-03T20:57:00.304 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.303+0000 7ff503000640 1 -- 172.21.15.32:0/565639949 <== osd.0 v2:172.21.15.32:6816/3066156116 2 ==== osd_op_reply(2 601.00000000 [delete] v101'475 uv475 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7ff4fc003040 con 0x555e5de94b60 2024-08-03T20:57:00.304 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.303+0000 7ff509475f40 1 -- 172.21.15.32:0/565639949 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:3 12.10 12:09401a81:::601.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101) -- 0x555e5df3eb40 con 0x555e5de94b60 2024-08-03T20:57:00.305 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::601.00000000:head# 2024-08-03T20:57:00.305 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.304+0000 7ff503000640 1 -- 172.21.15.32:0/565639949 <== osd.0 v2:172.21.15.32:6816/3066156116 3 ==== osd_op_reply(3 601.00000000 [create] v101'476 uv476 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7ff4fc003040 con 0x555e5de94b60 2024-08-03T20:57:00.306 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.304+0000 7ff509475f40 1 -- 172.21.15.32:0/565639949 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:4 12.10 12:09401a81:::601.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101) -- 0x555e5df3fef0 con 0x555e5de94b60 2024-08-03T20:57:00.307 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.306+0000 7ff503000640 1 -- 172.21.15.32:0/565639949 <== osd.0 v2:172.21.15.32:6816/3066156116 4 ==== osd_op_reply(4 601.00000000 [setxattr (30)] v101'477 uv477 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7ff4fc003040 con 0x555e5de94b60 2024-08-03T20:57:00.307 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.306+0000 7ff509475f40 1 -- 172.21.15.32:0/565639949 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:5 12.10 12:09401a81:::601.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101) -- 0x555e5df40290 con 0x555e5de94b60 2024-08-03T20:57:00.309 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.307+0000 7ff503000640 1 -- 172.21.15.32:0/565639949 <== osd.0 v2:172.21.15.32:6816/3066156116 5 ==== osd_op_reply(5 601.00000000 [setxattr (62)] v101'478 uv478 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7ff4fc003040 con 0x555e5de94b60 2024-08-03T20:57:00.309 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.307+0000 7ff509475f40 1 -- 172.21.15.32:0/565639949 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:6 12.10 12:09401a81:::601.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101) -- 0x555e5df3eee0 con 0x555e5de94b60 2024-08-03T20:57:00.310 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.309+0000 7ff503000640 1 -- 172.21.15.32:0/565639949 <== osd.0 v2:172.21.15.32:6816/3066156116 6 ==== osd_op_reply(6 601.00000000 [omap-set-header] v101'479 uv479 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7ff4fc003040 con 0x555e5de94b60 2024-08-03T20:57:00.310 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.309+0000 7ff509475f40 1 -- 172.21.15.32:0/565639949 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:7 12.10 12:0bd6d154:::602.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101) -- 0x555e5df3f280 con 0x555e5de94b60 2024-08-03T20:57:00.312 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.311+0000 7ff503000640 1 -- 172.21.15.32:0/565639949 <== osd.0 v2:172.21.15.32:6816/3066156116 7 ==== osd_op_reply(7 602.00000000 [create] v101'480 uv454 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7ff4fc003040 con 0x555e5de94b60 2024-08-03T20:57:00.312 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.311+0000 7ff509475f40 1 -- 172.21.15.32:0/565639949 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:8 12.10 12:0bd6d154:::602.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e101) -- 0x555e5df3f670 con 0x555e5de94b60 2024-08-03T20:57:00.314 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.312+0000 7ff503000640 1 -- 172.21.15.32:0/565639949 <== osd.0 v2:172.21.15.32:6816/3066156116 8 ==== osd_op_reply(8 602.00000000 [delete] v101'481 uv481 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7ff4fc003040 con 0x555e5de94b60 2024-08-03T20:57:00.314 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.312+0000 7ff509475f40 1 -- 172.21.15.32:0/565639949 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:9 12.10 12:0bd6d154:::602.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101) -- 0x555e5df3f670 con 0x555e5de94b60 2024-08-03T20:57:00.315 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::602.00000000:head# 2024-08-03T20:57:00.316 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.314+0000 7ff503000640 1 -- 172.21.15.32:0/565639949 <== osd.0 v2:172.21.15.32:6816/3066156116 9 ==== osd_op_reply(9 602.00000000 [create] v101'482 uv482 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7ff4fc003040 con 0x555e5de94b60 2024-08-03T20:57:00.316 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.314+0000 7ff509475f40 1 -- 172.21.15.32:0/565639949 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:10 12.10 12:0bd6d154:::602.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101) -- 0x555e5df3f620 con 0x555e5de94b60 2024-08-03T20:57:00.317 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.316+0000 7ff503000640 1 -- 172.21.15.32:0/565639949 <== osd.0 v2:172.21.15.32:6816/3066156116 10 ==== osd_op_reply(10 602.00000000 [setxattr (30)] v101'483 uv483 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7ff4fc003040 con 0x555e5de94b60 2024-08-03T20:57:00.317 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.316+0000 7ff509475f40 1 -- 172.21.15.32:0/565639949 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:11 12.10 12:0bd6d154:::602.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101) -- 0x555e5df3f620 con 0x555e5de94b60 2024-08-03T20:57:00.318 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.317+0000 7ff503000640 1 -- 172.21.15.32:0/565639949 <== osd.0 v2:172.21.15.32:6816/3066156116 11 ==== osd_op_reply(11 602.00000000 [setxattr (62)] v101'484 uv484 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7ff4fc003040 con 0x555e5de94b60 2024-08-03T20:57:00.319 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.317+0000 7ff509475f40 1 -- 172.21.15.32:0/565639949 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:12 12.10 12:0bd6d154:::602.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101) -- 0x555e5df3f620 con 0x555e5de94b60 2024-08-03T20:57:00.320 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.319+0000 7ff503000640 1 -- 172.21.15.32:0/565639949 <== osd.0 v2:172.21.15.32:6816/3066156116 12 ==== osd_op_reply(12 602.00000000 [omap-set-header] v101'485 uv485 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7ff4fc003040 con 0x555e5de94b60 2024-08-03T20:57:00.320 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.319+0000 7ff509475f40 1 -- 172.21.15.32:0/565639949 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:13 12.10 12:0cf6ad0f:::10000000000.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101) -- 0x555e5df3f620 con 0x555e5de94b60 2024-08-03T20:57:00.322 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.320+0000 7ff503000640 1 -- 172.21.15.32:0/565639949 <== osd.0 v2:172.21.15.32:6816/3066156116 13 ==== osd_op_reply(13 10000000000.00000000 [create] v101'486 uv461 ondisk = -17 ((17) File exists)) ==== 164+0+0 (crc 0 0 0) 0x7ff4fc003040 con 0x555e5de94b60 2024-08-03T20:57:00.322 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.321+0000 7ff509475f40 1 -- 172.21.15.32:0/565639949 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:14 12.10 12:0cf6ad0f:::10000000000.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e101) -- 0x555e5df3f620 con 0x555e5de94b60 2024-08-03T20:57:00.324 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.322+0000 7ff503000640 1 -- 172.21.15.32:0/565639949 <== osd.0 v2:172.21.15.32:6816/3066156116 14 ==== osd_op_reply(14 10000000000.00000000 [delete] v101'487 uv487 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7ff4fc003040 con 0x555e5de94b60 2024-08-03T20:57:00.324 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.322+0000 7ff509475f40 1 -- 172.21.15.32:0/565639949 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:15 12.10 12:0cf6ad0f:::10000000000.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101) -- 0x555e5df3f620 con 0x555e5de94b60 2024-08-03T20:57:00.325 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::10000000000.00000000:head# 2024-08-03T20:57:00.326 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.324+0000 7ff503000640 1 -- 172.21.15.32:0/565639949 <== osd.0 v2:172.21.15.32:6816/3066156116 15 ==== osd_op_reply(15 10000000000.00000000 [create] v101'488 uv488 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7ff4fc003040 con 0x555e5de94b60 2024-08-03T20:57:00.326 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.324+0000 7ff509475f40 1 -- 172.21.15.32:0/565639949 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:16 12.10 12:0cf6ad0f:::10000000000.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101) -- 0x555e5df3f620 con 0x555e5de94b60 2024-08-03T20:57:00.327 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.326+0000 7ff503000640 1 -- 172.21.15.32:0/565639949 <== osd.0 v2:172.21.15.32:6816/3066156116 16 ==== osd_op_reply(16 10000000000.00000000 [setxattr (30)] v101'489 uv489 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7ff4fc003040 con 0x555e5de94b60 2024-08-03T20:57:00.327 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.326+0000 7ff509475f40 1 -- 172.21.15.32:0/565639949 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:17 12.10 12:0cf6ad0f:::10000000000.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101) -- 0x555e5df3f620 con 0x555e5de94b60 2024-08-03T20:57:00.329 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.327+0000 7ff503000640 1 -- 172.21.15.32:0/565639949 <== osd.0 v2:172.21.15.32:6816/3066156116 17 ==== osd_op_reply(17 10000000000.00000000 [setxattr (62)] v101'490 uv490 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7ff4fc003040 con 0x555e5de94b60 2024-08-03T20:57:00.329 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.327+0000 7ff509475f40 1 -- 172.21.15.32:0/565639949 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:18 12.10 12:0cf6ad0f:::10000000000.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101) -- 0x555e5df3f620 con 0x555e5de94b60 2024-08-03T20:57:00.330 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.329+0000 7ff503000640 1 -- 172.21.15.32:0/565639949 <== osd.0 v2:172.21.15.32:6816/3066156116 18 ==== osd_op_reply(18 10000000000.00000000 [omap-set-header] v101'491 uv491 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7ff4fc003040 con 0x555e5de94b60 2024-08-03T20:57:00.331 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.329+0000 7ff509475f40 1 -- 172.21.15.32:0/565639949 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:19 12.10 12:0cf6ad0f:::10000000000.00000000:head [omap-set-vals in=593b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101) -- 0x555e5df3fa30 con 0x555e5de94b60 2024-08-03T20:57:00.332 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.331+0000 7ff503000640 1 -- 172.21.15.32:0/565639949 <== osd.0 v2:172.21.15.32:6816/3066156116 19 ==== osd_op_reply(19 10000000000.00000000 [omap-set-vals] v101'492 uv492 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7ff4fc003040 con 0x555e5de94b60 2024-08-03T20:57:00.332 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.331+0000 7ff509475f40 1 -- 172.21.15.32:0/565639949 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:20 12.10 12:0d82a743:::600.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101) -- 0x555e5df3fa30 con 0x555e5de94b60 2024-08-03T20:57:00.334 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.332+0000 7ff503000640 1 -- 172.21.15.32:0/565639949 <== osd.0 v2:172.21.15.32:6816/3066156116 20 ==== osd_op_reply(20 600.00000000 [create] v101'493 uv467 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7ff4fc003040 con 0x555e5de94b60 2024-08-03T20:57:00.334 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.332+0000 7ff509475f40 1 -- 172.21.15.32:0/565639949 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:21 12.10 12:0d82a743:::600.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e101) -- 0x555e5df3fa30 con 0x555e5de94b60 2024-08-03T20:57:00.335 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.334+0000 7ff503000640 1 -- 172.21.15.32:0/565639949 <== osd.0 v2:172.21.15.32:6816/3066156116 21 ==== osd_op_reply(21 600.00000000 [delete] v101'494 uv494 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7ff4fc003040 con 0x555e5de94b60 2024-08-03T20:57:00.335 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.334+0000 7ff509475f40 1 -- 172.21.15.32:0/565639949 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:22 12.10 12:0d82a743:::600.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101) -- 0x555e5df3fa30 con 0x555e5de94b60 2024-08-03T20:57:00.337 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::600.00000000:head# 2024-08-03T20:57:00.337 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.335+0000 7ff503000640 1 -- 172.21.15.32:0/565639949 <== osd.0 v2:172.21.15.32:6816/3066156116 22 ==== osd_op_reply(22 600.00000000 [create] v101'495 uv495 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7ff4fc003040 con 0x555e5de94b60 2024-08-03T20:57:00.337 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.335+0000 7ff509475f40 1 -- 172.21.15.32:0/565639949 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:23 12.10 12:0d82a743:::600.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101) -- 0x555e5df3fa30 con 0x555e5de94b60 2024-08-03T20:57:00.338 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.337+0000 7ff503000640 1 -- 172.21.15.32:0/565639949 <== osd.0 v2:172.21.15.32:6816/3066156116 23 ==== osd_op_reply(23 600.00000000 [setxattr (30)] v101'496 uv496 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7ff4fc003040 con 0x555e5de94b60 2024-08-03T20:57:00.339 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.337+0000 7ff509475f40 1 -- 172.21.15.32:0/565639949 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:24 12.10 12:0d82a743:::600.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101) -- 0x555e5df3fa30 con 0x555e5de94b60 2024-08-03T20:57:00.340 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.338+0000 7ff503000640 1 -- 172.21.15.32:0/565639949 <== osd.0 v2:172.21.15.32:6816/3066156116 24 ==== osd_op_reply(24 600.00000000 [setxattr (62)] v101'497 uv497 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7ff4fc003040 con 0x555e5de94b60 2024-08-03T20:57:00.340 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.338+0000 7ff509475f40 1 -- 172.21.15.32:0/565639949 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:25 12.10 12:0d82a743:::600.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101) -- 0x555e5df3fa30 con 0x555e5de94b60 2024-08-03T20:57:00.341 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.340+0000 7ff503000640 1 -- 172.21.15.32:0/565639949 <== osd.0 v2:172.21.15.32:6816/3066156116 25 ==== osd_op_reply(25 600.00000000 [omap-set-header] v101'498 uv498 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7ff4fc003040 con 0x555e5de94b60 2024-08-03T20:57:00.341 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.340+0000 7ff509475f40 1 -- 172.21.15.32:0/565639949 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:26 12.10 12:0d89b25e:::603.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101) -- 0x555e5df3fa30 con 0x555e5de94b60 2024-08-03T20:57:00.343 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.342+0000 7ff503000640 1 -- 172.21.15.32:0/565639949 <== osd.0 v2:172.21.15.32:6816/3066156116 26 ==== osd_op_reply(26 603.00000000 [create] v101'499 uv473 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7ff4fc003040 con 0x555e5de94b60 2024-08-03T20:57:00.343 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.342+0000 7ff509475f40 1 -- 172.21.15.32:0/565639949 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:27 12.10 12:0d89b25e:::603.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e101) -- 0x555e5df3fa30 con 0x555e5de94b60 2024-08-03T20:57:00.344 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.343+0000 7ff503000640 1 -- 172.21.15.32:0/565639949 <== osd.0 v2:172.21.15.32:6816/3066156116 27 ==== osd_op_reply(27 603.00000000 [delete] v101'500 uv500 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7ff4fc003040 con 0x555e5de94b60 2024-08-03T20:57:00.345 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.343+0000 7ff509475f40 1 -- 172.21.15.32:0/565639949 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:28 12.10 12:0d89b25e:::603.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101) -- 0x555e5df3fa30 con 0x555e5de94b60 2024-08-03T20:57:00.346 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::603.00000000:head# 2024-08-03T20:57:00.346 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.345+0000 7ff503000640 1 -- 172.21.15.32:0/565639949 <== osd.0 v2:172.21.15.32:6816/3066156116 28 ==== osd_op_reply(28 603.00000000 [create] v101'501 uv501 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7ff4fc003040 con 0x555e5de94b60 2024-08-03T20:57:00.346 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.345+0000 7ff509475f40 1 -- 172.21.15.32:0/565639949 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:29 12.10 12:0d89b25e:::603.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101) -- 0x555e5df3fa30 con 0x555e5de94b60 2024-08-03T20:57:00.348 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.346+0000 7ff503000640 1 -- 172.21.15.32:0/565639949 <== osd.0 v2:172.21.15.32:6816/3066156116 29 ==== osd_op_reply(29 603.00000000 [setxattr (30)] v101'502 uv502 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7ff4fc003040 con 0x555e5de94b60 2024-08-03T20:57:00.348 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.346+0000 7ff509475f40 1 -- 172.21.15.32:0/565639949 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:30 12.10 12:0d89b25e:::603.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101) -- 0x555e5df3fa30 con 0x555e5de94b60 2024-08-03T20:57:00.349 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.348+0000 7ff503000640 1 -- 172.21.15.32:0/565639949 <== osd.0 v2:172.21.15.32:6816/3066156116 30 ==== osd_op_reply(30 603.00000000 [setxattr (62)] v101'503 uv503 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7ff4fc003040 con 0x555e5de94b60 2024-08-03T20:57:00.349 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.348+0000 7ff509475f40 1 -- 172.21.15.32:0/565639949 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:31 12.10 12:0d89b25e:::603.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101) -- 0x555e5df3fa30 con 0x555e5de94b60 2024-08-03T20:57:00.351 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.349+0000 7ff503000640 1 -- 172.21.15.32:0/565639949 <== osd.0 v2:172.21.15.32:6816/3066156116 31 ==== osd_op_reply(31 603.00000000 [omap-set-header] v101'504 uv504 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7ff4fc003040 con 0x555e5de94b60 2024-08-03T20:57:00.351 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.349+0000 7ff509475f40 1 --2- 172.21.15.32:0/565639949 >> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] conn(0x555e5df40b40 0x555e5df42f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:00.351 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.349+0000 7ff509475f40 1 -- 172.21.15.32:0/565639949 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:32 12.14 12:287034c4:::1.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101) -- 0x555e5df3fa30 con 0x555e5df40b40 2024-08-03T20:57:00.351 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.350+0000 7ff502600640 1 --2- 172.21.15.32:0/565639949 >> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] conn(0x555e5df40b40 0x555e5df42f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:00.352 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.350+0000 7ff502600640 1 --2- 172.21.15.32:0/565639949 >> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] conn(0x555e5df40b40 0x555e5df42f60 crc :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:57:00.354 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.352+0000 7ff502600640 1 -- 172.21.15.32:0/565639949 <== osd.5 v2:172.21.15.118:6810/2724891770 1 ==== osd_op_reply(32 1.00000000.inode [create] v101'352 uv351 ondisk = -17 ((17) File exists)) ==== 160+0+0 (crc 0 0 0) 0x7ff4f405f030 con 0x555e5df40b40 2024-08-03T20:57:00.354 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.352+0000 7ff509475f40 1 -- 172.21.15.32:0/565639949 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:33 12.14 12:287034c4:::1.00000000.inode:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e101) -- 0x555e5df434a0 con 0x555e5df40b40 2024-08-03T20:57:00.355 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.354+0000 7ff502600640 1 -- 172.21.15.32:0/565639949 <== osd.5 v2:172.21.15.118:6810/2724891770 2 ==== osd_op_reply(33 1.00000000.inode [delete] v101'353 uv353 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7ff4f405f030 con 0x555e5df40b40 2024-08-03T20:57:00.355 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.354+0000 7ff509475f40 1 -- 172.21.15.32:0/565639949 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:34 12.14 12:287034c4:::1.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101) -- 0x555e5df43840 con 0x555e5df40b40 2024-08-03T20:57:00.357 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.355+0000 7ff502600640 1 -- 172.21.15.32:0/565639949 <== osd.5 v2:172.21.15.118:6810/2724891770 3 ==== osd_op_reply(34 1.00000000.inode [create] v101'354 uv354 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7ff4f405f030 con 0x555e5df40b40 2024-08-03T20:57:00.357 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::1.00000000.inode:head# 2024-08-03T20:57:00.357 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.355+0000 7ff509475f40 1 -- 172.21.15.32:0/565639949 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:35 12.14 12:287034c4:::1.00000000.inode:head [write 0~582 in=582b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101) -- 0x555e5df43be0 con 0x555e5df40b40 2024-08-03T20:57:00.358 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.357+0000 7ff502600640 1 -- 172.21.15.32:0/565639949 <== osd.5 v2:172.21.15.118:6810/2724891770 4 ==== osd_op_reply(35 1.00000000.inode [write 0~582] v101'355 uv355 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7ff4f405f030 con 0x555e5df40b40 2024-08-03T20:57:00.358 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.357+0000 7ff509475f40 1 -- 172.21.15.32:0/565639949 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:36 12.14 12:287034c4:::1.00000000.inode:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101) -- 0x555e5df43f80 con 0x555e5df40b40 2024-08-03T20:57:00.359 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.358+0000 7ff502600640 1 -- 172.21.15.32:0/565639949 <== osd.5 v2:172.21.15.118:6810/2724891770 5 ==== osd_op_reply(36 1.00000000.inode [omap-set-header] v101'356 uv356 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7ff4f405f030 con 0x555e5df40b40 2024-08-03T20:57:00.359 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.358+0000 7ff509475f40 1 -- 172.21.15.32:0/565639949 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:37 12.14 12:292cf221:::200.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101) -- 0x555e5df44320 con 0x555e5df40b40 2024-08-03T20:57:00.361 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.359+0000 7ff502600640 1 -- 172.21.15.32:0/565639949 <== osd.5 v2:172.21.15.118:6810/2724891770 6 ==== osd_op_reply(37 200.00000000 [create] v101'357 uv339 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7ff4f405f030 con 0x555e5df40b40 2024-08-03T20:57:00.361 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.360+0000 7ff509475f40 1 -- 172.21.15.32:0/565639949 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:38 12.14 12:292cf221:::200.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e101) -- 0x555e5df44320 con 0x555e5df40b40 2024-08-03T20:57:00.362 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.361+0000 7ff502600640 1 -- 172.21.15.32:0/565639949 <== osd.5 v2:172.21.15.118:6810/2724891770 7 ==== osd_op_reply(38 200.00000000 [delete] v101'358 uv358 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7ff4f405f030 con 0x555e5df40b40 2024-08-03T20:57:00.362 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.361+0000 7ff509475f40 1 -- 172.21.15.32:0/565639949 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:39 12.14 12:292cf221:::200.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101) -- 0x555e5df44320 con 0x555e5df40b40 2024-08-03T20:57:00.364 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.362+0000 7ff502600640 1 -- 172.21.15.32:0/565639949 <== osd.5 v2:172.21.15.118:6810/2724891770 8 ==== osd_op_reply(39 200.00000000 [create] v101'359 uv359 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7ff4f405f030 con 0x555e5df40b40 2024-08-03T20:57:00.364 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::200.00000000:head# 2024-08-03T20:57:00.364 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.362+0000 7ff509475f40 1 -- 172.21.15.32:0/565639949 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:40 12.14 12:292cf221:::200.00000000:head [write 0~90 in=90b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101) -- 0x555e5df44320 con 0x555e5df40b40 2024-08-03T20:57:00.365 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.364+0000 7ff502600640 1 -- 172.21.15.32:0/565639949 <== osd.5 v2:172.21.15.118:6810/2724891770 9 ==== osd_op_reply(40 200.00000000 [write 0~90] v101'360 uv360 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7ff4f405f030 con 0x555e5df40b40 2024-08-03T20:57:00.365 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.364+0000 7ff509475f40 1 -- 172.21.15.32:0/565639949 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:41 12.14 12:292cf221:::200.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101) -- 0x555e5df44320 con 0x555e5df40b40 2024-08-03T20:57:00.367 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.366+0000 7ff502600640 1 -- 172.21.15.32:0/565639949 <== osd.5 v2:172.21.15.118:6810/2724891770 10 ==== osd_op_reply(41 200.00000000 [omap-set-header] v101'361 uv361 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7ff4f405f030 con 0x555e5df40b40 2024-08-03T20:57:00.367 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.366+0000 7ff509475f40 1 -- 172.21.15.32:0/565639949 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:42 12.14 12:2e2fa760:::200.00000001:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101) -- 0x555e5df44320 con 0x555e5df40b40 2024-08-03T20:57:00.369 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.367+0000 7ff502600640 1 -- 172.21.15.32:0/565639949 <== osd.5 v2:172.21.15.118:6810/2724891770 11 ==== osd_op_reply(42 200.00000001 [create] v101'362 uv344 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7ff4f405f030 con 0x555e5df40b40 2024-08-03T20:57:00.369 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.367+0000 7ff509475f40 1 -- 172.21.15.32:0/565639949 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:43 12.14 12:2e2fa760:::200.00000001:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e101) -- 0x555e5df44320 con 0x555e5df40b40 2024-08-03T20:57:00.370 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.369+0000 7ff502600640 1 -- 172.21.15.32:0/565639949 <== osd.5 v2:172.21.15.118:6810/2724891770 12 ==== osd_op_reply(43 200.00000001 [delete] v101'363 uv363 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7ff4f405f030 con 0x555e5df40b40 2024-08-03T20:57:00.370 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.369+0000 7ff509475f40 1 -- 172.21.15.32:0/565639949 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:44 12.14 12:2e2fa760:::200.00000001:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101) -- 0x555e5df44320 con 0x555e5df40b40 2024-08-03T20:57:00.371 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.370+0000 7ff502600640 1 -- 172.21.15.32:0/565639949 <== osd.5 v2:172.21.15.118:6810/2724891770 13 ==== osd_op_reply(44 200.00000001 [create] v101'364 uv364 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7ff4f405f030 con 0x555e5df40b40 2024-08-03T20:57:00.372 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::200.00000001:head# 2024-08-03T20:57:00.372 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.370+0000 7ff509475f40 1 -- 172.21.15.32:0/565639949 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:45 12.14 12:2e2fa760:::200.00000001:head [write 0~18118 in=18118b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101) -- 0x555e5df48a10 con 0x555e5df40b40 2024-08-03T20:57:00.373 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.372+0000 7ff502600640 1 -- 172.21.15.32:0/565639949 <== osd.5 v2:172.21.15.118:6810/2724891770 14 ==== osd_op_reply(45 200.00000001 [write 0~18118] v101'365 uv365 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7ff4f405f030 con 0x555e5df40b40 2024-08-03T20:57:00.373 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.372+0000 7ff509475f40 1 -- 172.21.15.32:0/565639949 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:46 12.14 12:2e2fa760:::200.00000001:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101) -- 0x555e5df48a10 con 0x555e5df40b40 2024-08-03T20:57:00.375 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.374+0000 7ff502600640 1 -- 172.21.15.32:0/565639949 <== osd.5 v2:172.21.15.118:6810/2724891770 15 ==== osd_op_reply(46 200.00000001 [omap-set-header] v101'366 uv366 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7ff4f405f030 con 0x555e5df40b40 2024-08-03T20:57:00.375 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.374+0000 7ff509475f40 1 -- 172.21.15.32:0/565639949 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:47 12.14 12:2f55791f:::606.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101) -- 0x555e5df44320 con 0x555e5df40b40 2024-08-03T20:57:00.377 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.375+0000 7ff502600640 1 -- 172.21.15.32:0/565639949 <== osd.5 v2:172.21.15.118:6810/2724891770 16 ==== osd_op_reply(47 606.00000000 [create] v101'367 uv350 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7ff4f405f030 con 0x555e5df40b40 2024-08-03T20:57:00.377 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.375+0000 7ff509475f40 1 -- 172.21.15.32:0/565639949 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:48 12.14 12:2f55791f:::606.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e101) -- 0x555e5df44320 con 0x555e5df40b40 2024-08-03T20:57:00.378 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.377+0000 7ff502600640 1 -- 172.21.15.32:0/565639949 <== osd.5 v2:172.21.15.118:6810/2724891770 17 ==== osd_op_reply(48 606.00000000 [delete] v101'368 uv368 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7ff4f405f030 con 0x555e5df40b40 2024-08-03T20:57:00.378 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.377+0000 7ff509475f40 1 -- 172.21.15.32:0/565639949 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:49 12.14 12:2f55791f:::606.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101) -- 0x555e5df44320 con 0x555e5df40b40 2024-08-03T20:57:00.380 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.378+0000 7ff502600640 1 -- 172.21.15.32:0/565639949 <== osd.5 v2:172.21.15.118:6810/2724891770 18 ==== osd_op_reply(49 606.00000000 [create] v101'369 uv369 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7ff4f405f030 con 0x555e5df40b40 2024-08-03T20:57:00.380 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::606.00000000:head# 2024-08-03T20:57:00.380 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.378+0000 7ff509475f40 1 -- 172.21.15.32:0/565639949 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:50 12.14 12:2f55791f:::606.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101) -- 0x555e5df45330 con 0x555e5df40b40 2024-08-03T20:57:00.381 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.380+0000 7ff502600640 1 -- 172.21.15.32:0/565639949 <== osd.5 v2:172.21.15.118:6810/2724891770 19 ==== osd_op_reply(50 606.00000000 [setxattr (30)] v101'370 uv370 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7ff4f405f030 con 0x555e5df40b40 2024-08-03T20:57:00.381 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.380+0000 7ff509475f40 1 -- 172.21.15.32:0/565639949 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:51 12.14 12:2f55791f:::606.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101) -- 0x555e5df45330 con 0x555e5df40b40 2024-08-03T20:57:00.383 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.381+0000 7ff502600640 1 -- 172.21.15.32:0/565639949 <== osd.5 v2:172.21.15.118:6810/2724891770 20 ==== osd_op_reply(51 606.00000000 [setxattr (62)] v101'371 uv371 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7ff4f405f030 con 0x555e5df40b40 2024-08-03T20:57:00.383 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.381+0000 7ff509475f40 1 -- 172.21.15.32:0/565639949 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:52 12.14 12:2f55791f:::606.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101) -- 0x555e5df44320 con 0x555e5df40b40 2024-08-03T20:57:00.384 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.383+0000 7ff502600640 1 -- 172.21.15.32:0/565639949 <== osd.5 v2:172.21.15.118:6810/2724891770 21 ==== osd_op_reply(52 606.00000000 [omap-set-header] v101'372 uv372 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7ff4f405f030 con 0x555e5df40b40 2024-08-03T20:57:00.384 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.383+0000 7ff509475f40 1 -- 172.21.15.32:0/565639949 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:53 12.1c 12:3ed09add:::607.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101) -- 0x555e5df44320 con 0x555e5de94b60 2024-08-03T20:57:00.386 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.384+0000 7ff503000640 1 -- 172.21.15.32:0/565639949 <== osd.0 v2:172.21.15.32:6816/3066156116 32 ==== osd_op_reply(53 607.00000000 [create] v101'93 uv92 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7ff4fc003040 con 0x555e5de94b60 2024-08-03T20:57:00.386 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.384+0000 7ff509475f40 1 -- 172.21.15.32:0/565639949 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:54 12.1c 12:3ed09add:::607.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e101) -- 0x555e5df44320 con 0x555e5de94b60 2024-08-03T20:57:00.387 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.385+0000 7ff503000640 1 -- 172.21.15.32:0/565639949 <== osd.0 v2:172.21.15.32:6816/3066156116 33 ==== osd_op_reply(54 607.00000000 [delete] v101'94 uv94 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7ff4fc003040 con 0x555e5de94b60 2024-08-03T20:57:00.387 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.385+0000 7ff509475f40 1 -- 172.21.15.32:0/565639949 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:55 12.1c 12:3ed09add:::607.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101) -- 0x555e5df44320 con 0x555e5de94b60 2024-08-03T20:57:00.388 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::607.00000000:head# 2024-08-03T20:57:00.389 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.387+0000 7ff503000640 1 -- 172.21.15.32:0/565639949 <== osd.0 v2:172.21.15.32:6816/3066156116 34 ==== osd_op_reply(55 607.00000000 [create] v101'95 uv95 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7ff4fc003040 con 0x555e5de94b60 2024-08-03T20:57:00.389 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.387+0000 7ff509475f40 1 -- 172.21.15.32:0/565639949 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:56 12.1c 12:3ed09add:::607.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101) -- 0x555e5df45330 con 0x555e5de94b60 2024-08-03T20:57:00.389 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.388+0000 7ff503000640 1 -- 172.21.15.32:0/565639949 <== osd.0 v2:172.21.15.32:6816/3066156116 35 ==== osd_op_reply(56 607.00000000 [setxattr (30)] v101'96 uv96 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7ff4fc003040 con 0x555e5de94b60 2024-08-03T20:57:00.390 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.388+0000 7ff509475f40 1 -- 172.21.15.32:0/565639949 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:57 12.1c 12:3ed09add:::607.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101) -- 0x555e5df45330 con 0x555e5de94b60 2024-08-03T20:57:00.391 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.389+0000 7ff503000640 1 -- 172.21.15.32:0/565639949 <== osd.0 v2:172.21.15.32:6816/3066156116 36 ==== osd_op_reply(57 607.00000000 [setxattr (62)] v101'97 uv97 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7ff4fc003040 con 0x555e5de94b60 2024-08-03T20:57:00.391 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.389+0000 7ff509475f40 1 -- 172.21.15.32:0/565639949 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:58 12.1c 12:3ed09add:::607.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101) -- 0x555e5df44320 con 0x555e5de94b60 2024-08-03T20:57:00.392 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.391+0000 7ff503000640 1 -- 172.21.15.32:0/565639949 <== osd.0 v2:172.21.15.32:6816/3066156116 37 ==== osd_op_reply(58 607.00000000 [omap-set-header] v101'98 uv98 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7ff4fc003040 con 0x555e5de94b60 2024-08-03T20:57:00.392 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.391+0000 7ff509475f40 1 --2- 172.21.15.32:0/565639949 >> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] conn(0x555e5df449a0 0x555e5df46e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:00.392 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.391+0000 7ff509475f40 1 -- 172.21.15.32:0/565639949 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:59 12.6 12:654134d2:::mds0_openfiles.0:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101) -- 0x555e5df47490 con 0x555e5df449a0 2024-08-03T20:57:00.393 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.391+0000 7ff501c00640 1 --2- 172.21.15.32:0/565639949 >> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] conn(0x555e5df449a0 0x555e5df46e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:00.393 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.392+0000 7ff501c00640 1 --2- 172.21.15.32:0/565639949 >> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] conn(0x555e5df449a0 0x555e5df46e00 crc :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:57:00.394 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.393+0000 7ff501c00640 1 -- 172.21.15.32:0/565639949 <== osd.1 v2:172.21.15.32:6824/1551241759 1 ==== osd_op_reply(59 mds0_openfiles.0 [create] v101'78 uv77 ondisk = -17 ((17) File exists)) ==== 160+0+0 (crc 0 0 0) 0x7ff4f0012490 con 0x555e5df449a0 2024-08-03T20:57:00.395 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.393+0000 7ff509475f40 1 -- 172.21.15.32:0/565639949 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:60 12.6 12:654134d2:::mds0_openfiles.0:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e101) -- 0x555e5df47d30 con 0x555e5df449a0 2024-08-03T20:57:00.396 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.394+0000 7ff501c00640 1 -- 172.21.15.32:0/565639949 <== osd.1 v2:172.21.15.32:6824/1551241759 2 ==== osd_op_reply(60 mds0_openfiles.0 [delete] v101'79 uv79 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7ff4f0012490 con 0x555e5df449a0 2024-08-03T20:57:00.396 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.394+0000 7ff509475f40 1 -- 172.21.15.32:0/565639949 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:61 12.6 12:654134d2:::mds0_openfiles.0:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101) -- 0x555e5df480d0 con 0x555e5df449a0 2024-08-03T20:57:00.397 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.395+0000 7ff501c00640 1 -- 172.21.15.32:0/565639949 <== osd.1 v2:172.21.15.32:6824/1551241759 3 ==== osd_op_reply(61 mds0_openfiles.0 [create] v101'80 uv80 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7ff4f0012490 con 0x555e5df449a0 2024-08-03T20:57:00.397 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds0_openfiles.0:head# 2024-08-03T20:57:00.397 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.395+0000 7ff509475f40 1 -- 172.21.15.32:0/565639949 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:62 12.6 12:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101) -- 0x555e5df48470 con 0x555e5df449a0 2024-08-03T20:57:00.398 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.397+0000 7ff501c00640 1 -- 172.21.15.32:0/565639949 <== osd.1 v2:172.21.15.32:6824/1551241759 4 ==== osd_op_reply(62 mds0_openfiles.0 [omap-set-header] v101'81 uv81 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7ff4f0012490 con 0x555e5df449a0 2024-08-03T20:57:00.398 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.397+0000 7ff509475f40 1 -- 172.21.15.32:0/565639949 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:63 12.e 12:717a0223:::608.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101) -- 0x555e5df47900 con 0x555e5de94b60 2024-08-03T20:57:00.400 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.398+0000 7ff503000640 1 -- 172.21.15.32:0/565639949 <== osd.0 v2:172.21.15.32:6816/3066156116 38 ==== osd_op_reply(63 608.00000000 [create] v101'93 uv92 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7ff4fc003040 con 0x555e5de94b60 2024-08-03T20:57:00.400 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.398+0000 7ff509475f40 1 -- 172.21.15.32:0/565639949 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:64 12.e 12:717a0223:::608.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e101) -- 0x555e5df47900 con 0x555e5de94b60 2024-08-03T20:57:00.402 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.400+0000 7ff503000640 1 -- 172.21.15.32:0/565639949 <== osd.0 v2:172.21.15.32:6816/3066156116 39 ==== osd_op_reply(64 608.00000000 [delete] v101'94 uv94 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7ff4fc003040 con 0x555e5de94b60 2024-08-03T20:57:00.402 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.400+0000 7ff509475f40 1 -- 172.21.15.32:0/565639949 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:65 12.e 12:717a0223:::608.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101) -- 0x555e5df47850 con 0x555e5de94b60 2024-08-03T20:57:00.403 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::608.00000000:head# 2024-08-03T20:57:00.403 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.402+0000 7ff503000640 1 -- 172.21.15.32:0/565639949 <== osd.0 v2:172.21.15.32:6816/3066156116 40 ==== osd_op_reply(65 608.00000000 [create] v101'95 uv95 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7ff4fc003040 con 0x555e5de94b60 2024-08-03T20:57:00.403 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.402+0000 7ff509475f40 1 -- 172.21.15.32:0/565639949 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:66 12.e 12:717a0223:::608.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101) -- 0x555e5df49870 con 0x555e5de94b60 2024-08-03T20:57:00.404 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.403+0000 7ff503000640 1 -- 172.21.15.32:0/565639949 <== osd.0 v2:172.21.15.32:6816/3066156116 41 ==== osd_op_reply(66 608.00000000 [setxattr (30)] v101'96 uv96 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7ff4fc003040 con 0x555e5de94b60 2024-08-03T20:57:00.404 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.403+0000 7ff509475f40 1 -- 172.21.15.32:0/565639949 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:67 12.e 12:717a0223:::608.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101) -- 0x555e5df49870 con 0x555e5de94b60 2024-08-03T20:57:00.406 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.404+0000 7ff503000640 1 -- 172.21.15.32:0/565639949 <== osd.0 v2:172.21.15.32:6816/3066156116 42 ==== osd_op_reply(67 608.00000000 [setxattr (62)] v101'97 uv97 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7ff4fc003040 con 0x555e5de94b60 2024-08-03T20:57:00.406 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.404+0000 7ff509475f40 1 -- 172.21.15.32:0/565639949 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:68 12.e 12:717a0223:::608.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101) -- 0x555e5df48860 con 0x555e5de94b60 2024-08-03T20:57:00.407 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.406+0000 7ff503000640 1 -- 172.21.15.32:0/565639949 <== osd.0 v2:172.21.15.32:6816/3066156116 43 ==== osd_op_reply(68 608.00000000 [omap-set-header] v101'98 uv98 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7ff4fc003040 con 0x555e5de94b60 2024-08-03T20:57:00.408 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.406+0000 7ff509475f40 1 --2- 172.21.15.32:0/565639949 >> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] conn(0x555e5df48ee0 0x555e5df49310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:00.408 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.406+0000 7ff509475f40 1 -- 172.21.15.32:0/565639949 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:69 12.15 12:a93a17c2:::604.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101) -- 0x555e5df4bc90 con 0x555e5df48ee0 2024-08-03T20:57:00.408 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.406+0000 7ff503000640 1 --2- 172.21.15.32:0/565639949 >> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] conn(0x555e5df48ee0 0x555e5df49310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:00.408 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.407+0000 7ff503000640 1 --2- 172.21.15.32:0/565639949 >> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] conn(0x555e5df48ee0 0x555e5df49310 crc :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:57:00.410 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.408+0000 7ff503000640 1 -- 172.21.15.32:0/565639949 <== osd.6 v2:172.21.15.118:6825/801947386 1 ==== osd_op_reply(69 604.00000000 [create] v101'171 uv165 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7ff4fc003040 con 0x555e5df48ee0 2024-08-03T20:57:00.410 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.408+0000 7ff509475f40 1 -- 172.21.15.32:0/565639949 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:70 12.15 12:a93a17c2:::604.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e101) -- 0x555e5df4bc90 con 0x555e5df48ee0 2024-08-03T20:57:00.411 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.410+0000 7ff503000640 1 -- 172.21.15.32:0/565639949 <== osd.6 v2:172.21.15.118:6825/801947386 2 ==== osd_op_reply(70 604.00000000 [delete] v101'172 uv172 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7ff4fc003040 con 0x555e5df48ee0 2024-08-03T20:57:00.411 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.410+0000 7ff509475f40 1 -- 172.21.15.32:0/565639949 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:71 12.15 12:a93a17c2:::604.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101) -- 0x555e5df4bc90 con 0x555e5df48ee0 2024-08-03T20:57:00.413 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::604.00000000:head# 2024-08-03T20:57:00.414 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.411+0000 7ff503000640 1 -- 172.21.15.32:0/565639949 <== osd.6 v2:172.21.15.118:6825/801947386 3 ==== osd_op_reply(71 604.00000000 [create] v101'173 uv173 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7ff4fc003040 con 0x555e5df48ee0 2024-08-03T20:57:00.414 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.411+0000 7ff509475f40 1 -- 172.21.15.32:0/565639949 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:72 12.15 12:a93a17c2:::604.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101) -- 0x555e5df47900 con 0x555e5df48ee0 2024-08-03T20:57:00.414 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.412+0000 7ff503000640 1 -- 172.21.15.32:0/565639949 <== osd.6 v2:172.21.15.118:6825/801947386 4 ==== osd_op_reply(72 604.00000000 [setxattr (30)] v101'174 uv174 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7ff4fc003040 con 0x555e5df48ee0 2024-08-03T20:57:00.419 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.412+0000 7ff509475f40 1 -- 172.21.15.32:0/565639949 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:73 12.15 12:a93a17c2:::604.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101) -- 0x555e5df478c0 con 0x555e5df48ee0 2024-08-03T20:57:00.420 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.414+0000 7ff503000640 1 -- 172.21.15.32:0/565639949 <== osd.6 v2:172.21.15.118:6825/801947386 5 ==== osd_op_reply(73 604.00000000 [setxattr (62)] v101'175 uv175 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7ff4fc003040 con 0x555e5df48ee0 2024-08-03T20:57:00.421 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.414+0000 7ff509475f40 1 -- 172.21.15.32:0/565639949 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:74 12.15 12:a93a17c2:::604.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101) -- 0x555e5df478c0 con 0x555e5df48ee0 2024-08-03T20:57:00.421 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.415+0000 7ff503000640 1 -- 172.21.15.32:0/565639949 <== osd.6 v2:172.21.15.118:6825/801947386 6 ==== osd_op_reply(74 604.00000000 [omap-set-header] v101'176 uv176 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7ff4fc003040 con 0x555e5df48ee0 2024-08-03T20:57:00.421 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.415+0000 7ff509475f40 1 -- 172.21.15.32:0/565639949 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:75 12.15 12:aa448500:::500.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101) -- 0x555e5df478a0 con 0x555e5df48ee0 2024-08-03T20:57:00.421 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.416+0000 7ff503000640 1 -- 172.21.15.32:0/565639949 <== osd.6 v2:172.21.15.118:6825/801947386 7 ==== osd_op_reply(75 500.00000000 [create] v101'177 uv170 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7ff4fc003040 con 0x555e5df48ee0 2024-08-03T20:57:00.421 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.416+0000 7ff509475f40 1 -- 172.21.15.32:0/565639949 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:76 12.15 12:aa448500:::500.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e101) -- 0x555e5df478a0 con 0x555e5df48ee0 2024-08-03T20:57:00.421 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.418+0000 7ff503000640 1 -- 172.21.15.32:0/565639949 <== osd.6 v2:172.21.15.118:6825/801947386 8 ==== osd_op_reply(76 500.00000000 [delete] v101'178 uv178 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7ff4fc003040 con 0x555e5df48ee0 2024-08-03T20:57:00.421 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.418+0000 7ff509475f40 1 -- 172.21.15.32:0/565639949 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:77 12.15 12:aa448500:::500.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101) -- 0x555e5df478a0 con 0x555e5df48ee0 2024-08-03T20:57:00.421 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::500.00000000:head# 2024-08-03T20:57:00.422 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.419+0000 7ff503000640 1 -- 172.21.15.32:0/565639949 <== osd.6 v2:172.21.15.118:6825/801947386 9 ==== osd_op_reply(77 500.00000000 [create] v101'179 uv179 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7ff4fc003040 con 0x555e5df48ee0 2024-08-03T20:57:00.422 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.419+0000 7ff509475f40 1 -- 172.21.15.32:0/565639949 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:78 12.15 12:aa448500:::500.00000000:head [write 0~90 in=90b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101) -- 0x555e5df478a0 con 0x555e5df48ee0 2024-08-03T20:57:00.422 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.421+0000 7ff503000640 1 -- 172.21.15.32:0/565639949 <== osd.6 v2:172.21.15.118:6825/801947386 10 ==== osd_op_reply(78 500.00000000 [write 0~90] v101'180 uv180 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7ff4fc003040 con 0x555e5df48ee0 2024-08-03T20:57:00.423 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.421+0000 7ff509475f40 1 -- 172.21.15.32:0/565639949 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:79 12.15 12:aa448500:::500.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101) -- 0x555e5df478a0 con 0x555e5df48ee0 2024-08-03T20:57:00.424 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.422+0000 7ff503000640 1 -- 172.21.15.32:0/565639949 <== osd.6 v2:172.21.15.118:6825/801947386 11 ==== osd_op_reply(79 500.00000000 [omap-set-header] v101'181 uv181 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7ff4fc003040 con 0x555e5df48ee0 2024-08-03T20:57:00.424 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.422+0000 7ff509475f40 1 --2- 172.21.15.32:0/565639949 >> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] conn(0x555e5df4c6a0 0x555e5df4eac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:00.424 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.422+0000 7ff509475f40 1 -- 172.21.15.32:0/565639949 --> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] -- osd_op(unknown.0.0:80 12.d 12:b50e409b:::mds_snaptable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101) -- 0x555e5df4f000 con 0x555e5df4c6a0 2024-08-03T20:57:00.424 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.423+0000 7ff502600640 1 --2- 172.21.15.32:0/565639949 >> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] conn(0x555e5df4c6a0 0x555e5df4eac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:00.425 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.423+0000 7ff502600640 1 --2- 172.21.15.32:0/565639949 >> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] conn(0x555e5df4c6a0 0x555e5df4eac0 crc :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:57:00.426 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.425+0000 7ff502600640 1 -- 172.21.15.32:0/565639949 <== osd.7 v2:172.21.15.118:6818/653000048 1 ==== osd_op_reply(80 mds_snaptable [create] v101'76 uv75 ondisk = -17 ((17) File exists)) ==== 157+0+0 (crc 0 0 0) 0x7ff4f405f030 con 0x555e5df4c6a0 2024-08-03T20:57:00.426 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.425+0000 7ff509475f40 1 -- 172.21.15.32:0/565639949 --> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] -- osd_op(unknown.0.0:81 12.d 12:b50e409b:::mds_snaptable:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e101) -- 0x555e5df4f0c0 con 0x555e5df4c6a0 2024-08-03T20:57:00.427 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.426+0000 7ff502600640 1 -- 172.21.15.32:0/565639949 <== osd.7 v2:172.21.15.118:6818/653000048 2 ==== osd_op_reply(81 mds_snaptable [delete] v101'77 uv77 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7ff4f405f030 con 0x555e5df4c6a0 2024-08-03T20:57:00.427 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.426+0000 7ff509475f40 1 -- 172.21.15.32:0/565639949 --> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] -- osd_op(unknown.0.0:82 12.d 12:b50e409b:::mds_snaptable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101) -- 0x555e5df4f4e0 con 0x555e5df4c6a0 2024-08-03T20:57:00.429 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds_snaptable:head# 2024-08-03T20:57:00.429 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.427+0000 7ff502600640 1 -- 172.21.15.32:0/565639949 <== osd.7 v2:172.21.15.118:6818/653000048 3 ==== osd_op_reply(82 mds_snaptable [create] v101'78 uv78 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7ff4f405f030 con 0x555e5df4c6a0 2024-08-03T20:57:00.429 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.427+0000 7ff509475f40 1 -- 172.21.15.32:0/565639949 --> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] -- osd_op(unknown.0.0:83 12.d 12:b50e409b:::mds_snaptable:head [write 0~70 in=70b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101) -- 0x555e5df4f4e0 con 0x555e5df4c6a0 2024-08-03T20:57:00.430 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.429+0000 7ff502600640 1 -- 172.21.15.32:0/565639949 <== osd.7 v2:172.21.15.118:6818/653000048 4 ==== osd_op_reply(83 mds_snaptable [write 0~70] v101'79 uv79 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7ff4f405f030 con 0x555e5df4c6a0 2024-08-03T20:57:00.430 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.429+0000 7ff509475f40 1 -- 172.21.15.32:0/565639949 --> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] -- osd_op(unknown.0.0:84 12.d 12:b50e409b:::mds_snaptable:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101) -- 0x555e5df4f4e0 con 0x555e5df4c6a0 2024-08-03T20:57:00.431 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.430+0000 7ff502600640 1 -- 172.21.15.32:0/565639949 <== osd.7 v2:172.21.15.118:6818/653000048 5 ==== osd_op_reply(84 mds_snaptable [omap-set-header] v101'80 uv80 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7ff4f405f030 con 0x555e5df4c6a0 2024-08-03T20:57:00.431 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.430+0000 7ff509475f40 1 -- 172.21.15.32:0/565639949 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:85 12.1d 12:b871830b:::605.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101) -- 0x555e5df4f4e0 con 0x555e5de94b60 2024-08-03T20:57:00.433 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.431+0000 7ff503000640 1 -- 172.21.15.32:0/565639949 <== osd.0 v2:172.21.15.32:6816/3066156116 44 ==== osd_op_reply(85 605.00000000 [create] v101'93 uv92 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7ff4fc003040 con 0x555e5de94b60 2024-08-03T20:57:00.433 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.431+0000 7ff509475f40 1 -- 172.21.15.32:0/565639949 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:86 12.1d 12:b871830b:::605.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e101) -- 0x555e5df4f4e0 con 0x555e5de94b60 2024-08-03T20:57:00.434 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.433+0000 7ff503000640 1 -- 172.21.15.32:0/565639949 <== osd.0 v2:172.21.15.32:6816/3066156116 45 ==== osd_op_reply(86 605.00000000 [delete] v101'94 uv94 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7ff4fc003040 con 0x555e5de94b60 2024-08-03T20:57:00.434 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.433+0000 7ff509475f40 1 -- 172.21.15.32:0/565639949 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:87 12.1d 12:b871830b:::605.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101) -- 0x555e5df4f4e0 con 0x555e5de94b60 2024-08-03T20:57:00.435 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::605.00000000:head# 2024-08-03T20:57:00.436 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.434+0000 7ff503000640 1 -- 172.21.15.32:0/565639949 <== osd.0 v2:172.21.15.32:6816/3066156116 46 ==== osd_op_reply(87 605.00000000 [create] v101'95 uv95 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7ff4fc003040 con 0x555e5de94b60 2024-08-03T20:57:00.436 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.434+0000 7ff509475f40 1 -- 172.21.15.32:0/565639949 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:88 12.1d 12:b871830b:::605.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101) -- 0x555e5df504f0 con 0x555e5de94b60 2024-08-03T20:57:00.437 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.435+0000 7ff503000640 1 -- 172.21.15.32:0/565639949 <== osd.0 v2:172.21.15.32:6816/3066156116 47 ==== osd_op_reply(88 605.00000000 [setxattr (30)] v101'96 uv96 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7ff4fc003040 con 0x555e5de94b60 2024-08-03T20:57:00.437 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.435+0000 7ff509475f40 1 -- 172.21.15.32:0/565639949 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:89 12.1d 12:b871830b:::605.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101) -- 0x555e5df504f0 con 0x555e5de94b60 2024-08-03T20:57:00.438 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.437+0000 7ff503000640 1 -- 172.21.15.32:0/565639949 <== osd.0 v2:172.21.15.32:6816/3066156116 48 ==== osd_op_reply(89 605.00000000 [setxattr (62)] v101'97 uv97 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7ff4fc003040 con 0x555e5de94b60 2024-08-03T20:57:00.438 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.437+0000 7ff509475f40 1 -- 172.21.15.32:0/565639949 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:90 12.1d 12:b871830b:::605.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101) -- 0x555e5df4f4e0 con 0x555e5de94b60 2024-08-03T20:57:00.439 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.438+0000 7ff503000640 1 -- 172.21.15.32:0/565639949 <== osd.0 v2:172.21.15.32:6816/3066156116 49 ==== osd_op_reply(90 605.00000000 [omap-set-header] v101'98 uv98 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7ff4fc003040 con 0x555e5de94b60 2024-08-03T20:57:00.439 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.438+0000 7ff509475f40 1 -- 172.21.15.32:0/565639949 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:91 12.13 12:c91d4a1d:::mds0_inotable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101) -- 0x555e5df4f4e0 con 0x555e5df449a0 2024-08-03T20:57:00.441 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.439+0000 7ff501c00640 1 -- 172.21.15.32:0/565639949 <== osd.1 v2:172.21.15.32:6824/1551241759 5 ==== osd_op_reply(91 mds0_inotable [create] v101'152 uv146 ondisk = -17 ((17) File exists)) ==== 157+0+0 (crc 0 0 0) 0x7ff4f0012490 con 0x555e5df449a0 2024-08-03T20:57:00.441 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.439+0000 7ff509475f40 1 -- 172.21.15.32:0/565639949 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:92 12.13 12:c91d4a1d:::mds0_inotable:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e101) -- 0x555e5df4f880 con 0x555e5df449a0 2024-08-03T20:57:00.442 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.440+0000 7ff501c00640 1 -- 172.21.15.32:0/565639949 <== osd.1 v2:172.21.15.32:6824/1551241759 6 ==== osd_op_reply(92 mds0_inotable [delete] v101'153 uv153 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7ff4f0012490 con 0x555e5df449a0 2024-08-03T20:57:00.442 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.440+0000 7ff509475f40 1 -- 172.21.15.32:0/565639949 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:93 12.13 12:c91d4a1d:::mds0_inotable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101) -- 0x555e5df4fc20 con 0x555e5df449a0 2024-08-03T20:57:00.443 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.442+0000 7ff501c00640 1 -- 172.21.15.32:0/565639949 <== osd.1 v2:172.21.15.32:6824/1551241759 7 ==== osd_op_reply(93 mds0_inotable [create] v101'154 uv154 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7ff4f0012490 con 0x555e5df449a0 2024-08-03T20:57:00.443 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds0_inotable:head# 2024-08-03T20:57:00.443 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.442+0000 7ff509475f40 1 -- 172.21.15.32:0/565639949 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:94 12.13 12:c91d4a1d:::mds0_inotable:head [write 0~34 in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101) -- 0x555e5df4bd30 con 0x555e5df449a0 2024-08-03T20:57:00.444 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.443+0000 7ff501c00640 1 -- 172.21.15.32:0/565639949 <== osd.1 v2:172.21.15.32:6824/1551241759 8 ==== osd_op_reply(94 mds0_inotable [write 0~34] v101'155 uv155 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7ff4f0012490 con 0x555e5df449a0 2024-08-03T20:57:00.444 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.443+0000 7ff509475f40 1 -- 172.21.15.32:0/565639949 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:95 12.13 12:c91d4a1d:::mds0_inotable:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101) -- 0x555e5df4bd30 con 0x555e5df449a0 2024-08-03T20:57:00.446 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.444+0000 7ff501c00640 1 -- 172.21.15.32:0/565639949 <== osd.1 v2:172.21.15.32:6824/1551241759 9 ==== osd_op_reply(95 mds0_inotable [omap-set-header] v101'156 uv156 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7ff4f0012490 con 0x555e5df449a0 2024-08-03T20:57:00.446 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.444+0000 7ff509475f40 1 -- 172.21.15.32:0/565639949 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:96 12.13 12:cd5a64a3:::100.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101) -- 0x555e5df4bd30 con 0x555e5df449a0 2024-08-03T20:57:00.447 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.445+0000 7ff501c00640 1 -- 172.21.15.32:0/565639949 <== osd.1 v2:172.21.15.32:6824/1551241759 10 ==== osd_op_reply(96 100.00000000 [create] v101'157 uv151 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7ff4f0012490 con 0x555e5df449a0 2024-08-03T20:57:00.447 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.445+0000 7ff509475f40 1 -- 172.21.15.32:0/565639949 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:97 12.13 12:cd5a64a3:::100.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e101) -- 0x555e5df4bd30 con 0x555e5df449a0 2024-08-03T20:57:00.448 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.447+0000 7ff501c00640 1 -- 172.21.15.32:0/565639949 <== osd.1 v2:172.21.15.32:6824/1551241759 11 ==== osd_op_reply(97 100.00000000 [delete] v101'158 uv158 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7ff4f0012490 con 0x555e5df449a0 2024-08-03T20:57:00.448 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.447+0000 7ff509475f40 1 -- 172.21.15.32:0/565639949 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:98 12.13 12:cd5a64a3:::100.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101) -- 0x555e5df4bd30 con 0x555e5df449a0 2024-08-03T20:57:00.449 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::100.00000000:head# 2024-08-03T20:57:00.450 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.448+0000 7ff501c00640 1 -- 172.21.15.32:0/565639949 <== osd.1 v2:172.21.15.32:6824/1551241759 12 ==== osd_op_reply(98 100.00000000 [create] v101'159 uv159 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7ff4f0012490 con 0x555e5df449a0 2024-08-03T20:57:00.450 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.448+0000 7ff509475f40 1 -- 172.21.15.32:0/565639949 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:99 12.13 12:cd5a64a3:::100.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101) -- 0x555e5df4bd30 con 0x555e5df449a0 2024-08-03T20:57:00.451 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.449+0000 7ff501c00640 1 -- 172.21.15.32:0/565639949 <== osd.1 v2:172.21.15.32:6824/1551241759 13 ==== osd_op_reply(99 100.00000000 [omap-set-header] v101'160 uv160 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7ff4f0012490 con 0x555e5df449a0 2024-08-03T20:57:00.451 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.449+0000 7ff509475f40 1 -- 172.21.15.32:0/565639949 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:100 12.13 12:cd5a64a3:::100.00000000:head [omap-set-vals in=5194b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101) -- 0x555e5df52e50 con 0x555e5df449a0 2024-08-03T20:57:00.452 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.451+0000 7ff501c00640 1 -- 172.21.15.32:0/565639949 <== osd.1 v2:172.21.15.32:6824/1551241759 14 ==== osd_op_reply(100 100.00000000 [omap-set-vals] v101'161 uv161 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7ff4f0012490 con 0x555e5df449a0 2024-08-03T20:57:00.453 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.451+0000 7ff509475f40 1 --2- 172.21.15.32:0/565639949 >> [v2:172.21.15.32:6800/9088303,v1:172.21.15.32:6801/9088303] conn(0x555e5df504d0 0x555e5df53510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:00.453 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.451+0000 7ff509475f40 1 -- 172.21.15.32:0/565639949 --> [v2:172.21.15.32:6800/9088303,v1:172.21.15.32:6801/9088303] -- osd_op(unknown.0.0:101 12.b 12:d0630e4c:::mds0_sessionmap:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101) -- 0x555e5df528f0 con 0x555e5df504d0 2024-08-03T20:57:00.453 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.451+0000 7ff501c00640 1 --2- 172.21.15.32:0/565639949 >> [v2:172.21.15.32:6800/9088303,v1:172.21.15.32:6801/9088303] conn(0x555e5df504d0 0x555e5df53510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:00.453 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.452+0000 7ff501c00640 1 --2- 172.21.15.32:0/565639949 >> [v2:172.21.15.32:6800/9088303,v1:172.21.15.32:6801/9088303] conn(0x555e5df504d0 0x555e5df53510 crc :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:57:00.455 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.453+0000 7ff501c00640 1 -- 172.21.15.32:0/565639949 <== osd.3 v2:172.21.15.32:6800/9088303 1 ==== osd_op_reply(101 mds0_sessionmap [create] v101'70 uv69 ondisk = -17 ((17) File exists)) ==== 159+0+0 (crc 0 0 0) 0x7ff4f0012490 con 0x555e5df504d0 2024-08-03T20:57:00.455 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.453+0000 7ff509475f40 1 -- 172.21.15.32:0/565639949 --> [v2:172.21.15.32:6800/9088303,v1:172.21.15.32:6801/9088303] -- osd_op(unknown.0.0:102 12.b 12:d0630e4c:::mds0_sessionmap:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e101) -- 0x555e5df52da0 con 0x555e5df504d0 2024-08-03T20:57:00.456 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.455+0000 7ff501c00640 1 -- 172.21.15.32:0/565639949 <== osd.3 v2:172.21.15.32:6800/9088303 2 ==== osd_op_reply(102 mds0_sessionmap [delete] v101'71 uv71 ondisk = 0) ==== 159+0+0 (crc 0 0 0) 0x7ff4f0012490 con 0x555e5df504d0 2024-08-03T20:57:00.456 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.455+0000 7ff509475f40 1 -- 172.21.15.32:0/565639949 --> [v2:172.21.15.32:6800/9088303,v1:172.21.15.32:6801/9088303] -- osd_op(unknown.0.0:103 12.b 12:d0630e4c:::mds0_sessionmap:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101) -- 0x555e5df52da0 con 0x555e5df504d0 2024-08-03T20:57:00.457 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds0_sessionmap:head# 2024-08-03T20:57:00.457 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.456+0000 7ff501c00640 1 -- 172.21.15.32:0/565639949 <== osd.3 v2:172.21.15.32:6800/9088303 3 ==== osd_op_reply(103 mds0_sessionmap [create] v101'72 uv72 ondisk = 0) ==== 159+0+0 (crc 0 0 0) 0x7ff4f0012490 con 0x555e5df504d0 2024-08-03T20:57:00.458 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.456+0000 7ff509475f40 1 -- 172.21.15.32:0/565639949 --> [v2:172.21.15.32:6800/9088303,v1:172.21.15.32:6801/9088303] -- osd_op(unknown.0.0:104 12.b 12:d0630e4c:::mds0_sessionmap:head [omap-set-header in=14b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101) -- 0x555e5df52da0 con 0x555e5df504d0 2024-08-03T20:57:00.458 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.457+0000 7ff501c00640 1 -- 172.21.15.32:0/565639949 <== osd.3 v2:172.21.15.32:6800/9088303 4 ==== osd_op_reply(104 mds0_sessionmap [omap-set-header] v101'73 uv73 ondisk = 0) ==== 159+0+0 (crc 0 0 0) 0x7ff4f0012490 con 0x555e5df504d0 2024-08-03T20:57:00.459 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.457+0000 7ff509475f40 1 -- 172.21.15.32:0/565639949 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:105 12.7 12:e0b41d06:::609.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101) -- 0x555e5df52da0 con 0x555e5df449a0 2024-08-03T20:57:00.460 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.459+0000 7ff501c00640 1 -- 172.21.15.32:0/565639949 <== osd.1 v2:172.21.15.32:6824/1551241759 15 ==== osd_op_reply(105 609.00000000 [create] v101'93 uv92 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7ff4f0012490 con 0x555e5df449a0 2024-08-03T20:57:00.460 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.459+0000 7ff509475f40 1 -- 172.21.15.32:0/565639949 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:106 12.7 12:e0b41d06:::609.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e101) -- 0x555e5df52da0 con 0x555e5df449a0 2024-08-03T20:57:00.462 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.460+0000 7ff501c00640 1 -- 172.21.15.32:0/565639949 <== osd.1 v2:172.21.15.32:6824/1551241759 16 ==== osd_op_reply(106 609.00000000 [delete] v101'94 uv94 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7ff4f0012490 con 0x555e5df449a0 2024-08-03T20:57:00.462 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.460+0000 7ff509475f40 1 -- 172.21.15.32:0/565639949 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:107 12.7 12:e0b41d06:::609.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101) -- 0x555e5df52da0 con 0x555e5df449a0 2024-08-03T20:57:00.463 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.462+0000 7ff501c00640 1 -- 172.21.15.32:0/565639949 <== osd.1 v2:172.21.15.32:6824/1551241759 17 ==== osd_op_reply(107 609.00000000 [create] v101'95 uv95 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7ff4f0012490 con 0x555e5df449a0 2024-08-03T20:57:00.463 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::609.00000000:head# 2024-08-03T20:57:00.464 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.462+0000 7ff509475f40 1 -- 172.21.15.32:0/565639949 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:108 12.7 12:e0b41d06:::609.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101) -- 0x555e5df52da0 con 0x555e5df449a0 2024-08-03T20:57:00.464 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.463+0000 7ff501c00640 1 -- 172.21.15.32:0/565639949 <== osd.1 v2:172.21.15.32:6824/1551241759 18 ==== osd_op_reply(108 609.00000000 [setxattr (30)] v101'96 uv96 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7ff4f0012490 con 0x555e5df449a0 2024-08-03T20:57:00.464 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.463+0000 7ff509475f40 1 -- 172.21.15.32:0/565639949 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:109 12.7 12:e0b41d06:::609.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101) -- 0x555e5df52da0 con 0x555e5df449a0 2024-08-03T20:57:00.466 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.464+0000 7ff501c00640 1 -- 172.21.15.32:0/565639949 <== osd.1 v2:172.21.15.32:6824/1551241759 19 ==== osd_op_reply(109 609.00000000 [setxattr (62)] v101'97 uv97 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7ff4f0012490 con 0x555e5df449a0 2024-08-03T20:57:00.466 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.464+0000 7ff509475f40 1 -- 172.21.15.32:0/565639949 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:110 12.7 12:e0b41d06:::609.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101) -- 0x555e5df52da0 con 0x555e5df449a0 2024-08-03T20:57:00.467 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.466+0000 7ff501c00640 1 -- 172.21.15.32:0/565639949 <== osd.1 v2:172.21.15.32:6824/1551241759 20 ==== osd_op_reply(110 609.00000000 [omap-set-header] v101'98 uv98 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7ff4f0012490 con 0x555e5df449a0 2024-08-03T20:57:00.467 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.466+0000 7ff509475f40 1 -- 172.21.15.32:0/565639949 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:111 12.f 12:f1f69726:::400.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101) -- 0x555e5df52da0 con 0x555e5df48ee0 2024-08-03T20:57:00.469 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.467+0000 7ff503000640 1 -- 172.21.15.32:0/565639949 <== osd.6 v2:172.21.15.118:6825/801947386 12 ==== osd_op_reply(111 400.00000000 [create] v101'79 uv78 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7ff4fc003040 con 0x555e5df48ee0 2024-08-03T20:57:00.469 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.468+0000 7ff509475f40 1 -- 172.21.15.32:0/565639949 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:112 12.f 12:f1f69726:::400.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e101) -- 0x555e5df52da0 con 0x555e5df48ee0 2024-08-03T20:57:00.471 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.469+0000 7ff503000640 1 -- 172.21.15.32:0/565639949 <== osd.6 v2:172.21.15.118:6825/801947386 13 ==== osd_op_reply(112 400.00000000 [delete] v101'80 uv80 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7ff4fc003040 con 0x555e5df48ee0 2024-08-03T20:57:00.471 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.469+0000 7ff509475f40 1 -- 172.21.15.32:0/565639949 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:113 12.f 12:f1f69726:::400.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101) -- 0x555e5df52da0 con 0x555e5df48ee0 2024-08-03T20:57:00.472 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.471+0000 7ff503000640 1 -- 172.21.15.32:0/565639949 <== osd.6 v2:172.21.15.118:6825/801947386 14 ==== osd_op_reply(113 400.00000000 [create] v101'81 uv81 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7ff4fc003040 con 0x555e5df48ee0 2024-08-03T20:57:00.473 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::400.00000000:head# 2024-08-03T20:57:00.473 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.471+0000 7ff509475f40 1 -- 172.21.15.32:0/565639949 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:114 12.f 12:f1f69726:::400.00000000:head [write 0~22 in=22b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101) -- 0x555e5df53a50 con 0x555e5df48ee0 2024-08-03T20:57:00.474 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.472+0000 7ff503000640 1 -- 172.21.15.32:0/565639949 <== osd.6 v2:172.21.15.118:6825/801947386 15 ==== osd_op_reply(114 400.00000000 [write 0~22] v101'82 uv82 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7ff4fc003040 con 0x555e5df48ee0 2024-08-03T20:57:00.474 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.473+0000 7ff509475f40 1 -- 172.21.15.32:0/565639949 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:115 12.f 12:f1f69726:::400.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101) -- 0x555e5df53a50 con 0x555e5df48ee0 2024-08-03T20:57:00.475 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.474+0000 7ff503000640 1 -- 172.21.15.32:0/565639949 <== osd.6 v2:172.21.15.118:6825/801947386 16 ==== osd_op_reply(115 400.00000000 [omap-set-header] v101'83 uv83 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7ff4fc003040 con 0x555e5df48ee0 2024-08-03T20:57:00.476 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.474+0000 7ff509475f40 1 -- 172.21.15.32:0/565639949 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:116 12.1f 12:fe07bba1:::100.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101) -- 0x555e5df53a50 con 0x555e5df48ee0 2024-08-03T20:57:00.477 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.476+0000 7ff503000640 1 -- 172.21.15.32:0/565639949 <== osd.6 v2:172.21.15.118:6825/801947386 17 ==== osd_op_reply(116 100.00000000.inode [create] v101'192 uv186 ondisk = -17 ((17) File exists)) ==== 162+0+0 (crc 0 0 0) 0x7ff4fc003040 con 0x555e5df48ee0 2024-08-03T20:57:00.477 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.476+0000 7ff509475f40 1 -- 172.21.15.32:0/565639949 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:117 12.1f 12:fe07bba1:::100.00000000.inode:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e101) -- 0x555e5df53a50 con 0x555e5df48ee0 2024-08-03T20:57:00.479 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.477+0000 7ff503000640 1 -- 172.21.15.32:0/565639949 <== osd.6 v2:172.21.15.118:6825/801947386 18 ==== osd_op_reply(117 100.00000000.inode [delete] v101'193 uv193 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7ff4fc003040 con 0x555e5df48ee0 2024-08-03T20:57:00.479 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.477+0000 7ff509475f40 1 -- 172.21.15.32:0/565639949 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:118 12.1f 12:fe07bba1:::100.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101) -- 0x555e5df53a50 con 0x555e5df48ee0 2024-08-03T20:57:00.481 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.479+0000 7ff503000640 1 -- 172.21.15.32:0/565639949 <== osd.6 v2:172.21.15.118:6825/801947386 19 ==== osd_op_reply(118 100.00000000.inode [create] v101'194 uv194 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7ff4fc003040 con 0x555e5df48ee0 2024-08-03T20:57:00.481 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::100.00000000.inode:head# 2024-08-03T20:57:00.481 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.480+0000 7ff509475f40 1 -- 172.21.15.32:0/565639949 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:119 12.1f 12:fe07bba1:::100.00000000.inode:head [write 0~582 in=582b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101) -- 0x555e5df53a50 con 0x555e5df48ee0 2024-08-03T20:57:00.483 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.482+0000 7ff503000640 1 -- 172.21.15.32:0/565639949 <== osd.6 v2:172.21.15.118:6825/801947386 20 ==== osd_op_reply(119 100.00000000.inode [write 0~582] v101'195 uv195 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7ff4fc003040 con 0x555e5df48ee0 2024-08-03T20:57:00.483 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.482+0000 7ff509475f40 1 -- 172.21.15.32:0/565639949 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:120 12.1f 12:fe07bba1:::100.00000000.inode:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101) -- 0x555e5df53a50 con 0x555e5df48ee0 2024-08-03T20:57:00.485 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.483+0000 7ff503000640 1 -- 172.21.15.32:0/565639949 <== osd.6 v2:172.21.15.118:6825/801947386 21 ==== osd_op_reply(120 100.00000000.inode [omap-set-header] v101'196 uv196 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7ff4fc003040 con 0x555e5df48ee0 2024-08-03T20:57:00.485 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.483+0000 7ff509475f40 1 -- 172.21.15.32:0/565639949 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:121 12.1f 12:ff5b34d6:::1.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101) -- 0x555e5df53a50 con 0x555e5df48ee0 2024-08-03T20:57:00.486 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.485+0000 7ff503000640 1 -- 172.21.15.32:0/565639949 <== osd.6 v2:172.21.15.118:6825/801947386 22 ==== osd_op_reply(121 1.00000000 [create] v101'197 uv191 ondisk = -17 ((17) File exists)) ==== 154+0+0 (crc 0 0 0) 0x7ff4fc003040 con 0x555e5df48ee0 2024-08-03T20:57:00.486 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.485+0000 7ff509475f40 1 -- 172.21.15.32:0/565639949 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:122 12.1f 12:ff5b34d6:::1.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e101) -- 0x555e5df53a50 con 0x555e5df48ee0 2024-08-03T20:57:00.488 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.486+0000 7ff503000640 1 -- 172.21.15.32:0/565639949 <== osd.6 v2:172.21.15.118:6825/801947386 23 ==== osd_op_reply(122 1.00000000 [delete] v101'198 uv198 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7ff4fc003040 con 0x555e5df48ee0 2024-08-03T20:57:00.488 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.486+0000 7ff509475f40 1 -- 172.21.15.32:0/565639949 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:123 12.1f 12:ff5b34d6:::1.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101) -- 0x555e5df53a50 con 0x555e5df48ee0 2024-08-03T20:57:00.489 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.488+0000 7ff503000640 1 -- 172.21.15.32:0/565639949 <== osd.6 v2:172.21.15.118:6825/801947386 24 ==== osd_op_reply(123 1.00000000 [create] v101'199 uv199 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7ff4fc003040 con 0x555e5df48ee0 2024-08-03T20:57:00.489 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::1.00000000:head# 2024-08-03T20:57:00.489 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.488+0000 7ff509475f40 1 -- 172.21.15.32:0/565639949 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:124 12.1f 12:ff5b34d6:::1.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101) -- 0x555e5df54a60 con 0x555e5df48ee0 2024-08-03T20:57:00.490 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.489+0000 7ff503000640 1 -- 172.21.15.32:0/565639949 <== osd.6 v2:172.21.15.118:6825/801947386 25 ==== osd_op_reply(124 1.00000000 [setxattr (30)] v101'200 uv200 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7ff4fc003040 con 0x555e5df48ee0 2024-08-03T20:57:00.490 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.489+0000 7ff509475f40 1 -- 172.21.15.32:0/565639949 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:125 12.1f 12:ff5b34d6:::1.00000000:head [setxattr parent (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101) -- 0x555e5df54a60 con 0x555e5df48ee0 2024-08-03T20:57:00.491 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.490+0000 7ff503000640 1 -- 172.21.15.32:0/565639949 <== osd.6 v2:172.21.15.118:6825/801947386 26 ==== osd_op_reply(125 1.00000000 [setxattr (30)] v101'201 uv201 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7ff4fc003040 con 0x555e5df48ee0 2024-08-03T20:57:00.492 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.490+0000 7ff509475f40 1 -- 172.21.15.32:0/565639949 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:126 12.1f 12:ff5b34d6:::1.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101) -- 0x555e5df53a50 con 0x555e5df48ee0 2024-08-03T20:57:00.493 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.491+0000 7ff503000640 1 -- 172.21.15.32:0/565639949 <== osd.6 v2:172.21.15.118:6825/801947386 27 ==== osd_op_reply(126 1.00000000 [omap-set-header] v101'202 uv202 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7ff4fc003040 con 0x555e5df48ee0 2024-08-03T20:57:00.493 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.491+0000 7ff509475f40 1 -- 172.21.15.32:0/565639949 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:127 12.1f 12:ff5b34d6:::1.00000000:head [omap-set-vals in=584b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101) -- 0x555e5df541b0 con 0x555e5df48ee0 2024-08-03T20:57:00.495 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.493+0000 7ff503000640 1 -- 172.21.15.32:0/565639949 <== osd.6 v2:172.21.15.118:6825/801947386 28 ==== osd_op_reply(127 1.00000000 [omap-set-vals] v101'203 uv203 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7ff4fc003040 con 0x555e5df48ee0 2024-08-03T20:57:00.495 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.493+0000 7ff509475f40 1 -- 172.21.15.32:0/565639949 >> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] conn(0x555e5de94b60 msgr2=0x555e5de04dc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:00.495 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.493+0000 7ff509475f40 1 --2- 172.21.15.32:0/565639949 >> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] conn(0x555e5de94b60 0x555e5de04dc0 crc :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:00.495 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.493+0000 7ff509475f40 1 -- 172.21.15.32:0/565639949 >> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] conn(0x555e5df449a0 msgr2=0x555e5df46e00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:00.495 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.493+0000 7ff509475f40 1 --2- 172.21.15.32:0/565639949 >> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] conn(0x555e5df449a0 0x555e5df46e00 crc :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:00.495 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.493+0000 7ff509475f40 1 -- 172.21.15.32:0/565639949 >> [v2:172.21.15.32:6800/9088303,v1:172.21.15.32:6801/9088303] conn(0x555e5df504d0 msgr2=0x555e5df53510 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:00.495 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.493+0000 7ff509475f40 1 --2- 172.21.15.32:0/565639949 >> [v2:172.21.15.32:6800/9088303,v1:172.21.15.32:6801/9088303] conn(0x555e5df504d0 0x555e5df53510 crc :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:00.495 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.493+0000 7ff509475f40 1 -- 172.21.15.32:0/565639949 >> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] conn(0x555e5df40b40 msgr2=0x555e5df42f60 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:00.496 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.493+0000 7ff509475f40 1 --2- 172.21.15.32:0/565639949 >> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] conn(0x555e5df40b40 0x555e5df42f60 crc :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:00.496 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.493+0000 7ff503000640 1 -- 172.21.15.32:0/565639949 reap_dead start 2024-08-03T20:57:00.496 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.493+0000 7ff509475f40 1 -- 172.21.15.32:0/565639949 >> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] conn(0x555e5df48ee0 msgr2=0x555e5df49310 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:00.496 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.493+0000 7ff509475f40 1 --2- 172.21.15.32:0/565639949 >> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] conn(0x555e5df48ee0 0x555e5df49310 crc :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:00.496 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.493+0000 7ff509475f40 1 -- 172.21.15.32:0/565639949 >> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] conn(0x555e5df4c6a0 msgr2=0x555e5df4eac0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:00.496 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.493+0000 7ff509475f40 1 --2- 172.21.15.32:0/565639949 >> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] conn(0x555e5df4c6a0 0x555e5df4eac0 crc :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:00.496 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.493+0000 7ff509475f40 1 -- 172.21.15.32:0/565639949 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7ff4e40759f0 msgr2=0x7ff4e4077f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:00.496 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.493+0000 7ff509475f40 1 --2- 172.21.15.32:0/565639949 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7ff4e40759f0 0x7ff4e4077f00 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x555e5ddc4af0 tx=0x7ff4f0001690 comp rx=0 tx=0).stop 2024-08-03T20:57:00.496 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.493+0000 7ff509475f40 1 -- 172.21.15.32:0/565639949 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555e5de7c360 msgr2=0x555e5de7a650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:00.496 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.493+0000 7ff509475f40 1 --2- 172.21.15.32:0/565639949 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555e5de7c360 0x555e5de7a650 secure :-1 s=READY pgs=1465 cs=0 l=1 rev1=1 crypto rx=0x7ff4f4025db0 tx=0x7ff4f404d640 comp rx=0 tx=0).stop 2024-08-03T20:57:00.496 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.493+0000 7ff509475f40 1 -- 172.21.15.32:0/565639949 shutdown_connections 2024-08-03T20:57:00.496 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.493+0000 7ff509475f40 1 --2- 172.21.15.32:0/565639949 >> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] conn(0x555e5df4c6a0 0x555e5df4eac0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:00.496 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.493+0000 7ff509475f40 1 --2- 172.21.15.32:0/565639949 >> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] conn(0x555e5df48ee0 0x555e5df49310 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:00.497 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.493+0000 7ff509475f40 1 --2- 172.21.15.32:0/565639949 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7ff4e40759f0 0x7ff4e4077f00 unknown :-1 s=CLOSED pgs=598 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:00.497 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.493+0000 7ff509475f40 1 --2- 172.21.15.32:0/565639949 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555e5de7c360 0x555e5de7a650 unknown :-1 s=CLOSED pgs=1465 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:00.497 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.493+0000 7ff509475f40 1 -- 172.21.15.32:0/565639949 >> 172.21.15.32:0/565639949 conn(0x555e5dce5ac0 msgr2=0x555e5de0c090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:57:00.497 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.493+0000 7ff509475f40 1 -- 172.21.15.32:0/565639949 shutdown_connections 2024-08-03T20:57:00.497 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.493+0000 7ff509475f40 1 -- 172.21.15.32:0/565639949 wait complete. 2024-08-03T20:57:00.507 DEBUG:teuthology.orchestra.run.smithi032:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata put 200.00000000 - 2024-08-03T20:57:00.692 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.690+0000 7fab13982f40 1 Processor -- start 2024-08-03T20:57:00.692 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.690+0000 7fab13982f40 1 -- start start 2024-08-03T20:57:00.694 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.691+0000 7fab13982f40 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x556a9b278360 0x556a9b292250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:00.694 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.691+0000 7fab13982f40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x556a9b279ea0 0x556a9b290af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:00.694 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.691+0000 7fab13982f40 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x556a9b295cb0 0x556a9b293810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:00.694 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.691+0000 7fab13982f40 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x556a9b2960c0 con 0x556a9b295cb0 2024-08-03T20:57:00.695 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.691+0000 7fab13982f40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x556a9b10fe30 con 0x556a9b279ea0 2024-08-03T20:57:00.695 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.691+0000 7fab13982f40 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x556a9b200020 con 0x556a9b278360 2024-08-03T20:57:00.695 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.692+0000 7fab0d400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x556a9b295cb0 0x556a9b293810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:00.695 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.692+0000 7fab0d400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x556a9b295cb0 0x556a9b293810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:46918/0 (socket says 172.21.15.32:46918) 2024-08-03T20:57:00.695 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.692+0000 7fab0d400640 1 -- 172.21.15.32:0/2505124240 learned_addr learned my addr 172.21.15.32:0/2505124240 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:57:00.695 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.692+0000 7fab07e00640 1 --2- 172.21.15.32:0/2505124240 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x556a9b279ea0 0x556a9b290af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:00.695 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.692+0000 7fab0ca00640 1 --2- 172.21.15.32:0/2505124240 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x556a9b278360 0x556a9b292250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:00.695 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.692+0000 7fab0d400640 1 -- 172.21.15.32:0/2505124240 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x556a9b278360 msgr2=0x556a9b292250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:57:00.695 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.692+0000 7fab0d400640 1 --2- 172.21.15.32:0/2505124240 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x556a9b278360 0x556a9b292250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:00.695 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.692+0000 7fab0d400640 1 -- 172.21.15.32:0/2505124240 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x556a9b279ea0 msgr2=0x556a9b290af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:57:00.695 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.692+0000 7fab0d400640 1 --2- 172.21.15.32:0/2505124240 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x556a9b279ea0 0x556a9b290af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:00.695 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.692+0000 7fab0d400640 1 -- 172.21.15.32:0/2505124240 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556a9b0e8f10 con 0x556a9b295cb0 2024-08-03T20:57:00.695 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.692+0000 7fab07e00640 1 --2- 172.21.15.32:0/2505124240 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x556a9b279ea0 0x556a9b290af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-03T20:57:00.695 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.692+0000 7fab0d400640 1 --2- 172.21.15.32:0/2505124240 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x556a9b295cb0 0x556a9b293810 secure :-1 s=READY pgs=1467 cs=0 l=1 rev1=1 crypto rx=0x7fab0800b650 tx=0x7fab0800bb10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=96b93717d8bd9b16 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:57:00.696 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.693+0000 7fab06a00640 1 -- 172.21.15.32:0/2505124240 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab08009a30 con 0x556a9b295cb0 2024-08-03T20:57:00.696 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.693+0000 7fab06a00640 1 -- 172.21.15.32:0/2505124240 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fab08009ce0 con 0x556a9b295cb0 2024-08-03T20:57:00.696 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.693+0000 7fab06a00640 1 -- 172.21.15.32:0/2505124240 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab0800e6b0 con 0x556a9b295cb0 2024-08-03T20:57:00.697 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.693+0000 7fab13982f40 1 -- 172.21.15.32:0/2505124240 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x556a9b295cb0 msgr2=0x556a9b293810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:00.697 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.693+0000 7fab13982f40 1 --2- 172.21.15.32:0/2505124240 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x556a9b295cb0 0x556a9b293810 secure :-1 s=READY pgs=1467 cs=0 l=1 rev1=1 crypto rx=0x7fab0800b650 tx=0x7fab0800bb10 comp rx=0 tx=0).stop 2024-08-03T20:57:00.697 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.693+0000 7fab13982f40 1 -- 172.21.15.32:0/2505124240 shutdown_connections 2024-08-03T20:57:00.697 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.693+0000 7fab13982f40 1 --2- 172.21.15.32:0/2505124240 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x556a9b279ea0 0x556a9b290af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:00.697 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.693+0000 7fab13982f40 1 --2- 172.21.15.32:0/2505124240 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x556a9b295cb0 0x556a9b293810 unknown :-1 s=CLOSED pgs=1467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:00.697 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.693+0000 7fab13982f40 1 --2- 172.21.15.32:0/2505124240 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x556a9b278360 0x556a9b292250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:00.697 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.693+0000 7fab13982f40 1 -- 172.21.15.32:0/2505124240 >> 172.21.15.32:0/2505124240 conn(0x556a9b0e1ac0 msgr2=0x556a9b291710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:57:00.697 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.693+0000 7fab13982f40 1 -- 172.21.15.32:0/2505124240 shutdown_connections 2024-08-03T20:57:00.698 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.693+0000 7fab13982f40 1 -- 172.21.15.32:0/2505124240 wait complete. 2024-08-03T20:57:00.698 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.694+0000 7fab13982f40 1 Processor -- start 2024-08-03T20:57:00.698 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.695+0000 7fab13982f40 1 -- start start 2024-08-03T20:57:00.700 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.696+0000 7fab13982f40 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x556a9b278360 0x556a9b274390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:00.700 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.696+0000 7fab13982f40 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x556a9b279ea0 0x556a9b32f930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:00.700 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.696+0000 7fab13982f40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x556a9b295cb0 0x556a9b32fe70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:00.700 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.696+0000 7fab13982f40 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x556a9b208480 con 0x556a9b278360 2024-08-03T20:57:00.700 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.696+0000 7fab13982f40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x556a9b277210 con 0x556a9b295cb0 2024-08-03T20:57:00.700 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.696+0000 7fab13982f40 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x556a9b3303b0 con 0x556a9b279ea0 2024-08-03T20:57:00.701 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.696+0000 7fab0ca00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x556a9b278360 0x556a9b274390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:00.701 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.696+0000 7fab07e00640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x556a9b279ea0 0x556a9b32f930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:00.701 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.696+0000 7fab07e00640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x556a9b279ea0 0x556a9b32f930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3301/0 says I am v2:172.21.15.32:45966/0 (socket says 172.21.15.32:45966) 2024-08-03T20:57:00.701 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.696+0000 7fab0ca00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x556a9b278360 0x556a9b274390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:46932/0 (socket says 172.21.15.32:46932) 2024-08-03T20:57:00.701 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.696+0000 7fab07e00640 1 -- 172.21.15.32:0/2656376674 learned_addr learned my addr 172.21.15.32:0/2656376674 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:57:00.701 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.696+0000 7fab0d400640 1 --2- 172.21.15.32:0/2656376674 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x556a9b295cb0 0x556a9b32fe70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:00.701 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.697+0000 7fab0ca00640 1 -- 172.21.15.32:0/2656376674 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x556a9b279ea0 msgr2=0x556a9b32f930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:00.701 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.697+0000 7fab0ca00640 1 --2- 172.21.15.32:0/2656376674 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x556a9b279ea0 0x556a9b32f930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:00.701 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.697+0000 7fab0ca00640 1 -- 172.21.15.32:0/2656376674 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x556a9b295cb0 msgr2=0x556a9b32fe70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:00.701 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.697+0000 7fab0ca00640 1 --2- 172.21.15.32:0/2656376674 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x556a9b295cb0 0x556a9b32fe70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:00.701 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.697+0000 7fab0ca00640 1 -- 172.21.15.32:0/2656376674 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556a9b330600 con 0x556a9b278360 2024-08-03T20:57:00.701 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.697+0000 7fab07e00640 1 --2- 172.21.15.32:0/2656376674 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x556a9b279ea0 0x556a9b32f930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-03T20:57:00.702 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.697+0000 7fab0d400640 1 --2- 172.21.15.32:0/2656376674 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x556a9b295cb0 0x556a9b32fe70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-03T20:57:00.702 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.697+0000 7fab0ca00640 1 --2- 172.21.15.32:0/2656376674 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x556a9b278360 0x556a9b274390 secure :-1 s=READY pgs=1468 cs=0 l=1 rev1=1 crypto rx=0x556a9b298780 tx=0x7fab00004a10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:57:00.702 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.697+0000 7fab05600640 1 -- 172.21.15.32:0/2656376674 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab0000e320 con 0x556a9b278360 2024-08-03T20:57:00.702 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.698+0000 7fab05600640 1 -- 172.21.15.32:0/2656376674 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fab0000e4e0 con 0x556a9b278360 2024-08-03T20:57:00.702 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.698+0000 7fab13982f40 1 -- 172.21.15.32:0/2656376674 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x556a9b322330 con 0x556a9b278360 2024-08-03T20:57:00.702 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.698+0000 7fab05600640 1 -- 172.21.15.32:0/2656376674 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab0000ca80 con 0x556a9b278360 2024-08-03T20:57:00.702 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.698+0000 7fab13982f40 1 -- 172.21.15.32:0/2656376674 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556a9b274b30 con 0x556a9b278360 2024-08-03T20:57:00.703 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.700+0000 7fab05600640 1 -- 172.21.15.32:0/2656376674 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7fab0000cc20 con 0x556a9b278360 2024-08-03T20:57:00.703 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.701+0000 7fab05600640 1 --2- 172.21.15.32:0/2656376674 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7faaec075a20 0x7faaec077f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:00.704 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.701+0000 7fab05600640 1 -- 172.21.15.32:0/2656376674 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(101..101 src has 1..101) ==== 7230+0+0 (secure 0 0 0) 0x7fab0009b790 con 0x556a9b278360 2024-08-03T20:57:00.704 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.701+0000 7fab13982f40 1 --2- 172.21.15.32:0/2656376674 >> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] conn(0x556a9b290000 0x556a9b31f740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:00.704 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.701+0000 7fab13982f40 1 -- 172.21.15.32:0/2656376674 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:1 12.14 12:292cf221:::200.00000000:head [writefull 0~80 in=80b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101) -- 0x556a9b31fc80 con 0x556a9b290000 2024-08-03T20:57:00.704 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.701+0000 7fab07e00640 1 --2- 172.21.15.32:0/2656376674 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7faaec075a20 0x7faaec077f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:00.704 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.702+0000 7fab0d400640 1 --2- 172.21.15.32:0/2656376674 >> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] conn(0x556a9b290000 0x556a9b31f740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:00.704 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.702+0000 7fab07e00640 1 --2- 172.21.15.32:0/2656376674 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7faaec075a20 0x7faaec077f30 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7faaf804efd0 tx=0x7faaf8003020 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:57:00.704 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.702+0000 7fab0d400640 1 --2- 172.21.15.32:0/2656376674 >> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] conn(0x556a9b290000 0x556a9b31f740 crc :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:57:00.706 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.704+0000 7fab0d400640 1 -- 172.21.15.32:0/2656376674 <== osd.5 v2:172.21.15.118:6810/2724891770 1 ==== osd_op_reply(1 200.00000000 [writefull 0~80] v101'373 uv373 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fab08009830 con 0x556a9b290000 2024-08-03T20:57:00.706 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.704+0000 7fab13982f40 1 -- 172.21.15.32:0/2656376674 >> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] conn(0x556a9b290000 msgr2=0x556a9b31f740 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:00.706 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.704+0000 7fab13982f40 1 --2- 172.21.15.32:0/2656376674 >> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] conn(0x556a9b290000 0x556a9b31f740 crc :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:00.707 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.704+0000 7fab13982f40 1 -- 172.21.15.32:0/2656376674 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7faaec075a20 msgr2=0x7faaec077f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:00.707 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.704+0000 7fab13982f40 1 --2- 172.21.15.32:0/2656376674 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7faaec075a20 0x7faaec077f30 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7faaf804efd0 tx=0x7faaf8003020 comp rx=0 tx=0).stop 2024-08-03T20:57:00.707 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.704+0000 7fab13982f40 1 -- 172.21.15.32:0/2656376674 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x556a9b278360 msgr2=0x556a9b274390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:00.707 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.704+0000 7fab13982f40 1 --2- 172.21.15.32:0/2656376674 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x556a9b278360 0x556a9b274390 secure :-1 s=READY pgs=1468 cs=0 l=1 rev1=1 crypto rx=0x556a9b298780 tx=0x7fab00004a10 comp rx=0 tx=0).stop 2024-08-03T20:57:00.707 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.704+0000 7fab0d400640 1 -- 172.21.15.32:0/2656376674 reap_dead start 2024-08-03T20:57:00.707 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.704+0000 7fab13982f40 1 -- 172.21.15.32:0/2656376674 shutdown_connections 2024-08-03T20:57:00.707 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.704+0000 7fab13982f40 1 -- 172.21.15.32:0/2656376674 >> 172.21.15.32:0/2656376674 conn(0x556a9b0e1ac0 msgr2=0x556a9b28f8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:57:00.707 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.705+0000 7fab13982f40 1 -- 172.21.15.32:0/2656376674 shutdown_connections 2024-08-03T20:57:00.707 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.705+0000 7fab13982f40 1 -- 172.21.15.32:0/2656376674 wait complete. 2024-08-03T20:57:00.721 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs joinable true 2024-08-03T20:57:00.853 INFO:tasks.ceph.mds.a.smithi032.stderr: -40> 2024-08-03T20:56:56.099+0000 7f5e76a00640 -1 mds.0.cache.ino(0x1) Corrupt inode 0x1: End of buffer [buffer:2] 2024-08-03T20:57:00.853 INFO:tasks.ceph.mds.a.smithi032.stderr: -38> 2024-08-03T20:56:56.099+0000 7f5e76a00640 -1 log_channel(cluster) log [ERR] : Error loading MDS rank 0: (22) Invalid argument 2024-08-03T20:57:00.953 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.951+0000 7f72ebe00640 1 -- 172.21.15.32:0/3469576201 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f72e40cdc70 msgr2=0x7f72e4166de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:00.954 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.951+0000 7f72ebe00640 1 --2- 172.21.15.32:0/3469576201 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f72e40cdc70 0x7f72e4166de0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f72dc005fd0 tx=0x7f72dc02aa10 comp rx=0 tx=0).stop 2024-08-03T20:57:00.954 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.951+0000 7f72ebe00640 1 -- 172.21.15.32:0/3469576201 shutdown_connections 2024-08-03T20:57:00.954 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.951+0000 7f72ebe00640 1 --2- 172.21.15.32:0/3469576201 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f72e40cdc70 0x7f72e4166de0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:00.954 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.951+0000 7f72ebe00640 1 --2- 172.21.15.32:0/3469576201 >> v2:172.21.15.118:3300/0 conn(0x7f72e409b040 0x7f72e40cd730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:00.954 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.951+0000 7f72ebe00640 1 -- 172.21.15.32:0/3469576201 >> 172.21.15.32:0/3469576201 conn(0x7f72e40b3c00 msgr2=0x7f72e40b4010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:57:00.954 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.952+0000 7f72ebe00640 1 -- 172.21.15.32:0/3469576201 shutdown_connections 2024-08-03T20:57:00.954 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.952+0000 7f72ebe00640 1 -- 172.21.15.32:0/3469576201 wait complete. 2024-08-03T20:57:00.955 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.954+0000 7f72ebe00640 1 Processor -- start 2024-08-03T20:57:00.956 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.954+0000 7f72ebe00640 1 -- start start 2024-08-03T20:57:00.958 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.956+0000 7f72ebe00640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f72e409ac30 0x7f72e40c91a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:00.958 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.956+0000 7f72ebe00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f72e409b040 0x7f72e41f5e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:00.959 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.956+0000 7f72ebe00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f72e40cdc70 0x7f72e41f31d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:00.959 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.956+0000 7f72ebe00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f72e40c9020 con 0x7f72e40cdc70 2024-08-03T20:57:00.959 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.956+0000 7f72ebe00640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f72e40c8ea0 con 0x7f72e409b040 2024-08-03T20:57:00.959 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.956+0000 7f72ebe00640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7f72e40b7a20 con 0x7f72e409ac30 2024-08-03T20:57:00.959 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.956+0000 7f72e9e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f72e40cdc70 0x7f72e41f31d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:00.959 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.956+0000 7f72e9e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f72e40cdc70 0x7f72e41f31d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:46936/0 (socket says 172.21.15.32:46936) 2024-08-03T20:57:00.959 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.956+0000 7f72e9e00640 1 -- 172.21.15.32:0/1938629128 learned_addr learned my addr 172.21.15.32:0/1938629128 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:57:00.959 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.956+0000 7f72e8a00640 1 --2- 172.21.15.32:0/1938629128 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f72e409b040 0x7f72e41f5e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:00.959 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.956+0000 7f72e9400640 1 --2- 172.21.15.32:0/1938629128 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f72e409ac30 0x7f72e40c91a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:00.959 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.957+0000 7f72e8a00640 1 -- 172.21.15.32:0/1938629128 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f72e409ac30 msgr2=0x7f72e40c91a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:00.959 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.957+0000 7f72e8a00640 1 --2- 172.21.15.32:0/1938629128 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f72e409ac30 0x7f72e40c91a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:00.959 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.957+0000 7f72e8a00640 1 -- 172.21.15.32:0/1938629128 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f72e40cdc70 msgr2=0x7f72e41f31d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:00.960 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.957+0000 7f72e8a00640 1 --2- 172.21.15.32:0/1938629128 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f72e40cdc70 0x7f72e41f31d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:00.960 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.957+0000 7f72e8a00640 1 -- 172.21.15.32:0/1938629128 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f72e41f68d0 con 0x7f72e409b040 2024-08-03T20:57:00.960 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.957+0000 7f72e9e00640 1 --2- 172.21.15.32:0/1938629128 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f72e40cdc70 0x7f72e41f31d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-03T20:57:00.960 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.957+0000 7f72e8a00640 1 --2- 172.21.15.32:0/1938629128 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f72e409b040 0x7f72e41f5e10 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7f72d000e6d0 tx=0x7f72d000eb90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:57:00.961 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.958+0000 7f72e1600640 1 -- 172.21.15.32:0/1938629128 <== mon.1 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72d000cab0 con 0x7f72e409b040 2024-08-03T20:57:00.961 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.958+0000 7f72e1600640 1 -- 172.21.15.32:0/1938629128 <== mon.1 v2:172.21.15.118:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f72d000cd40 con 0x7f72e409b040 2024-08-03T20:57:00.961 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.958+0000 7f72ebe00640 1 -- 172.21.15.32:0/1938629128 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f72e41f6bb0 con 0x7f72e409b040 2024-08-03T20:57:00.961 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.958+0000 7f72e1600640 1 -- 172.21.15.32:0/1938629128 <== mon.1 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72d0010690 con 0x7f72e409b040 2024-08-03T20:57:00.961 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.958+0000 7f72ebe00640 1 -- 172.21.15.32:0/1938629128 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f72e40af2d0 con 0x7f72e409b040 2024-08-03T20:57:00.961 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.959+0000 7f72ebe00640 1 -- 172.21.15.32:0/1938629128 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f72e41f6640 con 0x7f72e409b040 2024-08-03T20:57:00.961 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.960+0000 7f72e1600640 1 -- 172.21.15.32:0/1938629128 <== mon.1 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f72d0010830 con 0x7f72e409b040 2024-08-03T20:57:00.963 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.961+0000 7f72e1600640 1 --2- 172.21.15.32:0/1938629128 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f72c4077040 0x7f72c4079550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:00.963 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.961+0000 7f72e1600640 1 -- 172.21.15.32:0/1938629128 <== mon.1 v2:172.21.15.118:3300/0 5 ==== osd_map(101..101 src has 1..101) ==== 7230+0+0 (secure 0 0 0) 0x7f72d0099790 con 0x7f72e409b040 2024-08-03T20:57:00.963 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.962+0000 7f72e9400640 1 --2- 172.21.15.32:0/1938629128 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f72c4077040 0x7f72c4079550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:00.964 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.962+0000 7f72e9400640 1 --2- 172.21.15.32:0/1938629128 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f72c4077040 0x7f72c4079550 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7f72d8001720 tx=0x7f72d8009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:57:00.968 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:00.967+0000 7f72e1600640 1 -- 172.21.15.32:0/1938629128 <== mon.1 v2:172.21.15.118:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7f72d0061cc0 con 0x7f72e409b040 2024-08-03T20:57:00.969 INFO:tasks.ceph.mds.a.smithi032.stdout:starting mds.a at 2024-08-03T20:57:01.218 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:01.216+0000 7f72ebe00640 1 -- 172.21.15.32:0/1938629128 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "joinable", "val": "true"} v 0) -- 0x7f72e40b50b0 con 0x7f72e409b040 2024-08-03T20:57:02.139 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:02.137+0000 7f72e1600640 1 -- 172.21.15.32:0/1938629128 <== mon.1 v2:172.21.15.118:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "joinable", "val": "true"}]=0 cephfs marked joinable; MDS may join as newly active. v189) ==== 162+0+0 (secure 0 0 0) 0x7f72e40b50b0 con 0x7f72e409b040 2024-08-03T20:57:02.139 INFO:teuthology.orchestra.run.smithi032.stderr:cephfs marked joinable; MDS may join as newly active. 2024-08-03T20:57:02.143 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:02.141+0000 7f72ebe00640 1 -- 172.21.15.32:0/1938629128 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f72c4077040 msgr2=0x7f72c4079550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:02.143 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:02.141+0000 7f72ebe00640 1 --2- 172.21.15.32:0/1938629128 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f72c4077040 0x7f72c4079550 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7f72d8001720 tx=0x7f72d8009040 comp rx=0 tx=0).stop 2024-08-03T20:57:02.143 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:02.141+0000 7f72ebe00640 1 -- 172.21.15.32:0/1938629128 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f72e409b040 msgr2=0x7f72e41f5e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:02.144 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:02.141+0000 7f72ebe00640 1 --2- 172.21.15.32:0/1938629128 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f72e409b040 0x7f72e41f5e10 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7f72d000e6d0 tx=0x7f72d000eb90 comp rx=0 tx=0).stop 2024-08-03T20:57:02.144 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:02.142+0000 7f72ebe00640 1 -- 172.21.15.32:0/1938629128 shutdown_connections 2024-08-03T20:57:02.144 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:02.142+0000 7f72ebe00640 1 --2- 172.21.15.32:0/1938629128 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f72c4077040 0x7f72c4079550 unknown :-1 s=CLOSED pgs=600 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:02.144 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:02.142+0000 7f72ebe00640 1 --2- 172.21.15.32:0/1938629128 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f72e409b040 0x7f72e41f5e10 unknown :-1 s=CLOSED pgs=456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:02.144 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:02.142+0000 7f72ebe00640 1 --2- 172.21.15.32:0/1938629128 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f72e40cdc70 0x7f72e41f31d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:02.144 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:02.142+0000 7f72ebe00640 1 --2- 172.21.15.32:0/1938629128 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f72e409ac30 0x7f72e40c91a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:02.144 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:02.142+0000 7f72ebe00640 1 -- 172.21.15.32:0/1938629128 >> 172.21.15.32:0/1938629128 conn(0x7f72e40b3c00 msgr2=0x7f72e40cfce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:57:02.144 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:02.142+0000 7f72ebe00640 1 -- 172.21.15.32:0/1938629128 shutdown_connections 2024-08-03T20:57:02.144 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:02.142+0000 7f72ebe00640 1 -- 172.21.15.32:0/1938629128 wait complete. 2024-08-03T20:57:02.149 INFO:tasks.ceph.mds.a.smithi032.stderr:2024-08-03T20:57:02.148+0000 7fe9b1400640 -1 log_channel(cluster) log [ERR] : Error recovering journal 0x200: (22) Invalid argument 2024-08-03T20:57:02.150 INFO:tasks.ceph.mon.a.smithi032.stderr:2024-08-03T20:57:02.149+0000 7f4e6d600640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-08-03T20:57:02.150 INFO:tasks.ceph.mon.a.smithi032.stderr:2024-08-03T20:57:02.149+0000 7f4e6d600640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 mds daemon damaged (MDS_DAMAGE) 2024-08-03T20:57:02.172 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-08-03T20:57:02.406 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:02.404+0000 7fcddc200640 1 -- 172.21.15.32:0/3848724760 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcdbc004800 msgr2=0x7fcdbc004c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:02.406 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:02.404+0000 7fcddc200640 1 --2- 172.21.15.32:0/3848724760 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcdbc004800 0x7fcdbc004c50 secure :-1 s=READY pgs=1472 cs=0 l=1 rev1=1 crypto rx=0x7fcdc0001830 tx=0x7fcdbc005830 comp rx=0 tx=0).stop 2024-08-03T20:57:02.406 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:02.404+0000 7fcdda200640 1 -- 172.21.15.32:0/3848724760 reap_dead start 2024-08-03T20:57:02.407 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:02.404+0000 7fcddc200640 1 -- 172.21.15.32:0/3848724760 shutdown_connections 2024-08-03T20:57:02.407 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:02.404+0000 7fcddc200640 1 -- 172.21.15.32:0/3848724760 >> 172.21.15.32:0/3848724760 conn(0x7fcdd40b1bd0 msgr2=0x7fcdd40b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:57:02.407 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:02.404+0000 7fcddc200640 1 -- 172.21.15.32:0/3848724760 shutdown_connections 2024-08-03T20:57:02.407 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:02.404+0000 7fcddc200640 1 -- 172.21.15.32:0/3848724760 wait complete. 2024-08-03T20:57:02.408 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:02.407+0000 7fcddc200640 1 Processor -- start 2024-08-03T20:57:02.408 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:02.407+0000 7fcddc200640 1 -- start start 2024-08-03T20:57:02.411 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:02.408+0000 7fcddc200640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fcdbc004800 0x7fcdd41434e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:02.411 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:02.408+0000 7fcddc200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcdd41f4ce0 0x7fcdd41f82d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:02.411 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:02.408+0000 7fcddc200640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fcdd41f50f0 0x7fcdd41f8810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:02.412 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:02.408+0000 7fcddc200640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fcdd40ce800 con 0x7fcdd41f4ce0 2024-08-03T20:57:02.412 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:02.408+0000 7fcddc200640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7fcdd40ce680 con 0x7fcdbc004800 2024-08-03T20:57:02.412 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:02.408+0000 7fcddc200640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7fcdd40ce980 con 0x7fcdd41f50f0 2024-08-03T20:57:02.412 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:02.408+0000 7fcdd8e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcdd41f4ce0 0x7fcdd41f82d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:02.412 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:02.409+0000 7fcdd8e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcdd41f4ce0 0x7fcdd41f82d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:46960/0 (socket says 172.21.15.32:46960) 2024-08-03T20:57:02.412 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:02.409+0000 7fcdd8e00640 1 -- 172.21.15.32:0/3332480493 learned_addr learned my addr 172.21.15.32:0/3332480493 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:57:02.412 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:02.409+0000 7fcdd9800640 1 --2- 172.21.15.32:0/3332480493 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fcdbc004800 0x7fcdd41434e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:02.412 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:02.409+0000 7fcdda200640 1 --2- 172.21.15.32:0/3332480493 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fcdd41f50f0 0x7fcdd41f8810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:02.412 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:02.409+0000 7fcdd8e00640 1 -- 172.21.15.32:0/3332480493 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fcdd41f50f0 msgr2=0x7fcdd41f8810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:02.412 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:02.409+0000 7fcdd8e00640 1 --2- 172.21.15.32:0/3332480493 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fcdd41f50f0 0x7fcdd41f8810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:02.412 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:02.409+0000 7fcdd8e00640 1 -- 172.21.15.32:0/3332480493 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fcdbc004800 msgr2=0x7fcdd41434e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:02.412 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:02.409+0000 7fcdd8e00640 1 --2- 172.21.15.32:0/3332480493 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fcdbc004800 0x7fcdd41434e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:02.412 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:02.409+0000 7fcdd8e00640 1 -- 172.21.15.32:0/3332480493 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcdd4074260 con 0x7fcdd41f4ce0 2024-08-03T20:57:02.412 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:02.409+0000 7fcdd8e00640 1 --2- 172.21.15.32:0/3332480493 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcdd41f4ce0 0x7fcdd41f82d0 secure :-1 s=READY pgs=1473 cs=0 l=1 rev1=1 crypto rx=0x7fcdc0001920 tx=0x7fcdc0006150 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:57:02.413 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:02.409+0000 7fcdd1600640 1 -- 172.21.15.32:0/3332480493 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcdc0043070 con 0x7fcdd41f4ce0 2024-08-03T20:57:02.413 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:02.410+0000 7fcdd1600640 1 -- 172.21.15.32:0/3332480493 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fcdc0003af0 con 0x7fcdd41f4ce0 2024-08-03T20:57:02.413 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:02.410+0000 7fcddc200640 1 -- 172.21.15.32:0/3332480493 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcdd41f57f0 con 0x7fcdd41f4ce0 2024-08-03T20:57:02.413 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:02.410+0000 7fcdd1600640 1 -- 172.21.15.32:0/3332480493 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcdc0033480 con 0x7fcdd41f4ce0 2024-08-03T20:57:02.413 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:02.410+0000 7fcddc200640 1 -- 172.21.15.32:0/3332480493 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcdd40729e0 con 0x7fcdd41f4ce0 2024-08-03T20:57:02.414 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:02.411+0000 7fcddc200640 1 -- 172.21.15.32:0/3332480493 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcdd40727d0 con 0x7fcdd41f4ce0 2024-08-03T20:57:02.414 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:02.412+0000 7fcdd1600640 1 -- 172.21.15.32:0/3332480493 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7fcdc0046070 con 0x7fcdd41f4ce0 2024-08-03T20:57:02.415 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:02.413+0000 7fcdd1600640 1 --2- 172.21.15.32:0/3332480493 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fcdb4077010 0x7fcdb4079520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:02.415 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:02.413+0000 7fcdd1600640 1 -- 172.21.15.32:0/3332480493 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(102..102 src has 1..102) ==== 7316+0+0 (secure 0 0 0) 0x7fcdc00b8df0 con 0x7fcdd41f4ce0 2024-08-03T20:57:02.415 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:02.414+0000 7fcdd9800640 1 --2- 172.21.15.32:0/3332480493 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fcdb4077010 0x7fcdb4079520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:02.416 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:02.415+0000 7fcdd9800640 1 --2- 172.21.15.32:0/3332480493 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fcdb4077010 0x7fcdb4079520 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7fcdc8009bb0 tx=0x7fcdc8005f90 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:57:02.419 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:02.418+0000 7fcdd1600640 1 -- 172.21.15.32:0/3332480493 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7fcdc0081400 con 0x7fcdd41f4ce0 2024-08-03T20:57:02.663 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:02.662+0000 7fcddc200640 1 -- 172.21.15.32:0/3332480493 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fcdd41f5550 con 0x7fcdd41f4ce0 2024-08-03T20:57:02.664 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:02.663+0000 7fcdd1600640 1 -- 172.21.15.32:0/3332480493 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 191 v191) ==== 95+0+4391 (secure 0 0 0) 0x7fcdc0085950 con 0x7fcdd41f4ce0 2024-08-03T20:57:02.664 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-03T20:57:02.664 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":191,"btime":"2024-08-03T20:57:02:150543+0000","default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6439,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6835/531080160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6834","nonce":531080160},{"type":"v1","addr":"172.21.15.118:6835","nonce":531080160}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":84},{"gid":6454,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6837/2796882319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6836","nonce":2796882319},{"type":"v1","addr":"172.21.15.118:6837","nonce":2796882319}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":86},{"gid":6813,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6834/2686933890","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":2686933890},{"type":"v1","addr":"172.21.15.32:6834","nonce":2686933890}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":85}],"filesystems":[{"mdsmap":{"epoch":191,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-08-03T20:54:26.950518+0000","modified":"2024-08-03T20:57:02.150443+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":102,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":6}]} 2024-08-03T20:57:02.665 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 191 2024-08-03T20:57:02.667 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:02.665+0000 7fcddc200640 1 -- 172.21.15.32:0/3332480493 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fcdb4077010 msgr2=0x7fcdb4079520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:02.667 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:02.665+0000 7fcddc200640 1 --2- 172.21.15.32:0/3332480493 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fcdb4077010 0x7fcdb4079520 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7fcdc8009bb0 tx=0x7fcdc8005f90 comp rx=0 tx=0).stop 2024-08-03T20:57:02.667 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:02.665+0000 7fcddc200640 1 -- 172.21.15.32:0/3332480493 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcdd41f4ce0 msgr2=0x7fcdd41f82d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:02.667 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:02.665+0000 7fcddc200640 1 --2- 172.21.15.32:0/3332480493 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcdd41f4ce0 0x7fcdd41f82d0 secure :-1 s=READY pgs=1473 cs=0 l=1 rev1=1 crypto rx=0x7fcdc0001920 tx=0x7fcdc0006150 comp rx=0 tx=0).stop 2024-08-03T20:57:02.667 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:02.665+0000 7fcddc200640 1 -- 172.21.15.32:0/3332480493 shutdown_connections 2024-08-03T20:57:02.667 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:02.665+0000 7fcddc200640 1 --2- 172.21.15.32:0/3332480493 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fcdb4077010 0x7fcdb4079520 unknown :-1 s=CLOSED pgs=602 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:02.667 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:02.665+0000 7fcddc200640 1 --2- 172.21.15.32:0/3332480493 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fcdd41f50f0 0x7fcdd41f8810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:02.667 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:02.665+0000 7fcddc200640 1 --2- 172.21.15.32:0/3332480493 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcdd41f4ce0 0x7fcdd41f82d0 unknown :-1 s=CLOSED pgs=1473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:02.667 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:02.665+0000 7fcddc200640 1 --2- 172.21.15.32:0/3332480493 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fcdbc004800 0x7fcdd41434e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:02.667 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:02.665+0000 7fcddc200640 1 -- 172.21.15.32:0/3332480493 >> 172.21.15.32:0/3332480493 conn(0x7fcdd40b1bd0 msgr2=0x7fcdd4159530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:57:02.667 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:02.665+0000 7fcddc200640 1 -- 172.21.15.32:0/3332480493 shutdown_connections 2024-08-03T20:57:02.668 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:02.665+0000 7fcddc200640 1 -- 172.21.15.32:0/3332480493 wait complete. 2024-08-03T20:57:02.691 DEBUG:tasks.ceph_test_case:wait_until_true: success in 0s and 0 retries 2024-08-03T20:57:02.691 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-08-03T20:57:02.920 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:02.918+0000 7fd7b8600640 1 -- 172.21.15.32:0/3044768821 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd7b006e930 msgr2=0x7fd7b0144f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:02.920 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:02.918+0000 7fd7b8600640 1 --2- 172.21.15.32:0/3044768821 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd7b006e930 0x7fd7b0144f90 secure :-1 s=READY pgs=1475 cs=0 l=1 rev1=1 crypto rx=0x7fd7a4009830 tx=0x7fd7a402a5c0 comp rx=0 tx=0).stop 2024-08-03T20:57:02.920 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:02.918+0000 7fd7b8600640 1 -- 172.21.15.32:0/3044768821 shutdown_connections 2024-08-03T20:57:02.920 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:02.918+0000 7fd7b8600640 1 --2- 172.21.15.32:0/3044768821 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd7b006e930 0x7fd7b0144f90 unknown :-1 s=CLOSED pgs=1475 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:02.920 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:02.918+0000 7fd7b8600640 1 -- 172.21.15.32:0/3044768821 >> 172.21.15.32:0/3044768821 conn(0x7fd7b00b1bd0 msgr2=0x7fd7b00b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:57:02.920 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:02.919+0000 7fd7b8600640 1 -- 172.21.15.32:0/3044768821 shutdown_connections 2024-08-03T20:57:02.921 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:02.919+0000 7fd7b8600640 1 -- 172.21.15.32:0/3044768821 wait complete. 2024-08-03T20:57:02.922 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:02.921+0000 7fd7b8600640 1 Processor -- start 2024-08-03T20:57:02.922 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:02.921+0000 7fd7b8600640 1 -- start start 2024-08-03T20:57:02.925 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:02.922+0000 7fd7b8600640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd7b006e930 0x7fd7b0141ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:02.925 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:02.922+0000 7fd7b8600640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fd7b0073e60 0x7fd7b015bce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:02.925 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:02.922+0000 7fd7b8600640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fd7b015fff0 0x7fd7b015e370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:02.925 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:02.922+0000 7fd7b8600640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fd7b00cf840 con 0x7fd7b006e930 2024-08-03T20:57:02.926 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:02.922+0000 7fd7b8600640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7fd7b0145f30 con 0x7fd7b0073e60 2024-08-03T20:57:02.926 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:02.922+0000 7fd7b8600640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7fd7b00cf9c0 con 0x7fd7b015fff0 2024-08-03T20:57:02.926 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:02.923+0000 7fd7b5c00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd7b006e930 0x7fd7b0141ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:02.926 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:02.923+0000 7fd7b5c00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd7b006e930 0x7fd7b0141ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:46990/0 (socket says 172.21.15.32:46990) 2024-08-03T20:57:02.926 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:02.923+0000 7fd7b5c00640 1 -- 172.21.15.32:0/4201023214 learned_addr learned my addr 172.21.15.32:0/4201023214 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:57:02.926 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:02.923+0000 7fd7b5200640 1 --2- 172.21.15.32:0/4201023214 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fd7b0073e60 0x7fd7b015bce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:02.926 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:02.923+0000 7fd7b6600640 1 --2- 172.21.15.32:0/4201023214 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fd7b015fff0 0x7fd7b015e370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:02.926 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:02.923+0000 7fd7b5c00640 1 -- 172.21.15.32:0/4201023214 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fd7b015fff0 msgr2=0x7fd7b015e370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:02.926 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:02.923+0000 7fd7b5c00640 1 --2- 172.21.15.32:0/4201023214 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fd7b015fff0 0x7fd7b015e370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:02.926 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:02.923+0000 7fd7b5c00640 1 -- 172.21.15.32:0/4201023214 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fd7b0073e60 msgr2=0x7fd7b015bce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:02.926 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:02.923+0000 7fd7b5c00640 1 --2- 172.21.15.32:0/4201023214 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fd7b0073e60 0x7fd7b015bce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:02.926 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:02.923+0000 7fd7b5c00640 1 -- 172.21.15.32:0/4201023214 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd7b015c220 con 0x7fd7b006e930 2024-08-03T20:57:02.927 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:02.924+0000 7fd7b5c00640 1 --2- 172.21.15.32:0/4201023214 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd7b006e930 0x7fd7b0141ef0 secure :-1 s=READY pgs=1476 cs=0 l=1 rev1=1 crypto rx=0x7fd7a402abb0 tx=0x7fd7a4033ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:57:02.927 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:02.924+0000 7fd7ad600640 1 -- 172.21.15.32:0/4201023214 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7a4031b50 con 0x7fd7b006e930 2024-08-03T20:57:02.927 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:02.924+0000 7fd7ad600640 1 -- 172.21.15.32:0/4201023214 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fd7a4031de0 con 0x7fd7b006e930 2024-08-03T20:57:02.927 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:02.924+0000 7fd7b8600640 1 -- 172.21.15.32:0/4201023214 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd7b01e6980 con 0x7fd7b006e930 2024-08-03T20:57:02.927 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:02.924+0000 7fd7ad600640 1 -- 172.21.15.32:0/4201023214 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7a4005280 con 0x7fd7b006e930 2024-08-03T20:57:02.927 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:02.924+0000 7fd7b8600640 1 -- 172.21.15.32:0/4201023214 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd7b01e5d40 con 0x7fd7b006e930 2024-08-03T20:57:02.927 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:02.925+0000 7fd7b8600640 1 -- 172.21.15.32:0/4201023214 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd7b0160e20 con 0x7fd7b006e930 2024-08-03T20:57:02.928 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:02.926+0000 7fd7ad600640 1 -- 172.21.15.32:0/4201023214 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7fd7a4005420 con 0x7fd7b006e930 2024-08-03T20:57:02.929 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:02.927+0000 7fd7ad600640 1 --2- 172.21.15.32:0/4201023214 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fd790077010 0x7fd790079520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:02.929 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:02.928+0000 7fd7ad600640 1 -- 172.21.15.32:0/4201023214 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(102..102 src has 1..102) ==== 7316+0+0 (secure 0 0 0) 0x7fd7a40b9f00 con 0x7fd7b006e930 2024-08-03T20:57:02.930 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:02.928+0000 7fd7b5200640 1 --2- 172.21.15.32:0/4201023214 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fd790077010 0x7fd790079520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:02.930 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:02.929+0000 7fd7b5200640 1 --2- 172.21.15.32:0/4201023214 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fd790077010 0x7fd790079520 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7fd7b01f9e30 tx=0x7fd79c007cc0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:57:02.933 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:02.932+0000 7fd7ad600640 1 -- 172.21.15.32:0/4201023214 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7fd7b015c220 con 0x7fd7b006e930 2024-08-03T20:57:03.176 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:03.174+0000 7fd7b8600640 1 -- 172.21.15.32:0/4201023214 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd7b0069090 con 0x7fd7b006e930 2024-08-03T20:57:03.176 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:03.175+0000 7fd7ad600640 1 -- 172.21.15.32:0/4201023214 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 191 v191) ==== 95+0+4391 (secure 0 0 0) 0x7fd7a403f030 con 0x7fd7b006e930 2024-08-03T20:57:03.177 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-03T20:57:03.177 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":191,"btime":"2024-08-03T20:57:02:150543+0000","default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6439,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6835/531080160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6834","nonce":531080160},{"type":"v1","addr":"172.21.15.118:6835","nonce":531080160}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":84},{"gid":6454,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6837/2796882319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6836","nonce":2796882319},{"type":"v1","addr":"172.21.15.118:6837","nonce":2796882319}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":86},{"gid":6813,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6834/2686933890","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":2686933890},{"type":"v1","addr":"172.21.15.32:6834","nonce":2686933890}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":85}],"filesystems":[{"mdsmap":{"epoch":191,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-08-03T20:54:26.950518+0000","modified":"2024-08-03T20:57:02.150443+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":102,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":6}]} 2024-08-03T20:57:03.177 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 191 2024-08-03T20:57:03.179 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:03.177+0000 7fd7b8600640 1 -- 172.21.15.32:0/4201023214 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fd790077010 msgr2=0x7fd790079520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:03.179 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:03.177+0000 7fd7b8600640 1 --2- 172.21.15.32:0/4201023214 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fd790077010 0x7fd790079520 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7fd7b01f9e30 tx=0x7fd79c007cc0 comp rx=0 tx=0).stop 2024-08-03T20:57:03.179 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:03.177+0000 7fd7b8600640 1 -- 172.21.15.32:0/4201023214 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd7b006e930 msgr2=0x7fd7b0141ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:03.179 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:03.177+0000 7fd7b8600640 1 --2- 172.21.15.32:0/4201023214 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd7b006e930 0x7fd7b0141ef0 secure :-1 s=READY pgs=1476 cs=0 l=1 rev1=1 crypto rx=0x7fd7a402abb0 tx=0x7fd7a4033ba0 comp rx=0 tx=0).stop 2024-08-03T20:57:03.179 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:03.177+0000 7fd7b8600640 1 -- 172.21.15.32:0/4201023214 shutdown_connections 2024-08-03T20:57:03.179 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:03.177+0000 7fd7b8600640 1 --2- 172.21.15.32:0/4201023214 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fd790077010 0x7fd790079520 unknown :-1 s=CLOSED pgs=603 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:03.179 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:03.177+0000 7fd7b8600640 1 --2- 172.21.15.32:0/4201023214 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fd7b0073e60 0x7fd7b015bce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:03.179 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:03.177+0000 7fd7b8600640 1 --2- 172.21.15.32:0/4201023214 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fd7b015fff0 0x7fd7b015e370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:03.179 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:03.177+0000 7fd7b8600640 1 --2- 172.21.15.32:0/4201023214 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd7b006e930 0x7fd7b0141ef0 unknown :-1 s=CLOSED pgs=1476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:03.180 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:03.177+0000 7fd7b8600640 1 -- 172.21.15.32:0/4201023214 >> 172.21.15.32:0/4201023214 conn(0x7fd7b00b1bd0 msgr2=0x7fd7b0074270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:57:03.180 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:03.177+0000 7fd7b8600640 1 -- 172.21.15.32:0/4201023214 shutdown_connections 2024-08-03T20:57:03.180 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:03.178+0000 7fd7b8600640 1 -- 172.21.15.32:0/4201023214 wait complete. 2024-08-03T20:57:03.205 DEBUG:tasks.ceph_test_case:wait_until_true: success in 0s and 0 retries 2024-08-03T20:57:03.205 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-08-03T20:57:03.433 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:03.431+0000 7f0c4ba00640 1 -- 172.21.15.32:0/3097149499 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0c440b4ce0 msgr2=0x7f0c4415ef00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:03.433 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:03.431+0000 7f0c4ba00640 1 --2- 172.21.15.32:0/3097149499 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0c440b4ce0 0x7f0c4415ef00 secure :-1 s=READY pgs=1477 cs=0 l=1 rev1=1 crypto rx=0x7f0c4415ee40 tx=0x7f0c2c02a560 comp rx=0 tx=0).stop 2024-08-03T20:57:03.434 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:03.431+0000 7f0c4ba00640 1 -- 172.21.15.32:0/3097149499 shutdown_connections 2024-08-03T20:57:03.434 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:03.431+0000 7f0c4ba00640 1 --2- 172.21.15.32:0/3097149499 >> v2:172.21.15.118:3300/0 conn(0x7f0c4415f440 0x7f0c4415b910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:03.434 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:03.431+0000 7f0c4ba00640 1 --2- 172.21.15.32:0/3097149499 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0c440b4ce0 0x7f0c4415ef00 unknown :-1 s=CLOSED pgs=1477 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:03.434 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:03.431+0000 7f0c4ba00640 1 --2- 172.21.15.32:0/3097149499 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f0c440b38b0 0x7f0c441523f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:03.434 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:03.431+0000 7f0c4ba00640 1 -- 172.21.15.32:0/3097149499 >> 172.21.15.32:0/3097149499 conn(0x7f0c440b1bd0 msgr2=0x7f0c440b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:57:03.434 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:03.431+0000 7f0c4ba00640 1 -- 172.21.15.32:0/3097149499 shutdown_connections 2024-08-03T20:57:03.435 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:03.432+0000 7f0c4ba00640 1 -- 172.21.15.32:0/3097149499 wait complete. 2024-08-03T20:57:03.435 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:03.434+0000 7f0c4ba00640 1 Processor -- start 2024-08-03T20:57:03.436 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:03.434+0000 7f0c4ba00640 1 -- start start 2024-08-03T20:57:03.439 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:03.435+0000 7f0c4ba00640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f0c440b38b0 0x7f0c44151c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:03.439 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:03.435+0000 7f0c4ba00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0c440b4ce0 0x7f0c44152190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:03.439 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:03.435+0000 7f0c4ba00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f0c4415f440 0x7f0c4414f6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:03.440 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:03.435+0000 7f0c4ba00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f0c4415c1e0 con 0x7f0c440b4ce0 2024-08-03T20:57:03.440 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:03.435+0000 7f0c4ba00640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f0c4415c060 con 0x7f0c4415f440 2024-08-03T20:57:03.440 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:03.435+0000 7f0c4ba00640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7f0c4415c360 con 0x7f0c440b38b0 2024-08-03T20:57:03.440 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:03.436+0000 7f0c3be00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0c440b4ce0 0x7f0c44152190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:03.440 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:03.436+0000 7f0c3be00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0c440b4ce0 0x7f0c44152190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:47018/0 (socket says 172.21.15.32:47018) 2024-08-03T20:57:03.440 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:03.436+0000 7f0c3be00640 1 -- 172.21.15.32:0/3745213432 learned_addr learned my addr 172.21.15.32:0/3745213432 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:57:03.440 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:03.436+0000 7f0c49000640 1 --2- 172.21.15.32:0/3745213432 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f0c440b38b0 0x7f0c44151c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:03.440 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:03.436+0000 7f0c49a00640 1 --2- 172.21.15.32:0/3745213432 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f0c4415f440 0x7f0c4414f6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:03.440 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:03.436+0000 7f0c3be00640 1 -- 172.21.15.32:0/3745213432 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f0c440b38b0 msgr2=0x7f0c44151c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:03.440 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:03.436+0000 7f0c3be00640 1 --2- 172.21.15.32:0/3745213432 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f0c440b38b0 0x7f0c44151c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:03.440 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:03.436+0000 7f0c3be00640 1 -- 172.21.15.32:0/3745213432 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f0c4415f440 msgr2=0x7f0c4414f6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:03.440 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:03.436+0000 7f0c3be00640 1 --2- 172.21.15.32:0/3745213432 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f0c4415f440 0x7f0c4414f6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:03.440 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:03.436+0000 7f0c3be00640 1 -- 172.21.15.32:0/3745213432 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0c441517f0 con 0x7f0c440b4ce0 2024-08-03T20:57:03.441 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:03.436+0000 7f0c49000640 1 --2- 172.21.15.32:0/3745213432 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f0c440b38b0 0x7f0c44151c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-03T20:57:03.441 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:03.436+0000 7f0c49a00640 1 --2- 172.21.15.32:0/3745213432 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f0c4415f440 0x7f0c4414f6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-03T20:57:03.441 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:03.437+0000 7f0c3be00640 1 --2- 172.21.15.32:0/3745213432 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0c440b4ce0 0x7f0c44152190 secure :-1 s=READY pgs=1478 cs=0 l=1 rev1=1 crypto rx=0x7f0c2c034040 tx=0x7f0c2c0039e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:57:03.441 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:03.437+0000 7f0c41600640 1 -- 172.21.15.32:0/3745213432 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c2c03e030 con 0x7f0c440b4ce0 2024-08-03T20:57:03.441 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:03.437+0000 7f0c41600640 1 -- 172.21.15.32:0/3745213432 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f0c2c0047a0 con 0x7f0c440b4ce0 2024-08-03T20:57:03.441 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:03.437+0000 7f0c4ba00640 1 -- 172.21.15.32:0/3745213432 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0c44166020 con 0x7f0c440b4ce0 2024-08-03T20:57:03.441 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:03.437+0000 7f0c41600640 1 -- 172.21.15.32:0/3745213432 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c2c035040 con 0x7f0c440b4ce0 2024-08-03T20:57:03.441 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:03.437+0000 7f0c4ba00640 1 -- 172.21.15.32:0/3745213432 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0c441665e0 con 0x7f0c440b4ce0 2024-08-03T20:57:03.442 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:03.438+0000 7f0c4ba00640 1 -- 172.21.15.32:0/3745213432 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0c4415ef00 con 0x7f0c440b4ce0 2024-08-03T20:57:03.442 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:03.439+0000 7f0c41600640 1 -- 172.21.15.32:0/3745213432 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f0c2c041030 con 0x7f0c440b4ce0 2024-08-03T20:57:03.442 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:03.440+0000 7f0c41600640 1 --2- 172.21.15.32:0/3745213432 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f0c20077040 0x7f0c20079550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:03.442 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:03.440+0000 7f0c41600640 1 -- 172.21.15.32:0/3745213432 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(102..102 src has 1..102) ==== 7316+0+0 (secure 0 0 0) 0x7f0c2c0b9800 con 0x7f0c440b4ce0 2024-08-03T20:57:03.442 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:03.441+0000 7f0c49000640 1 --2- 172.21.15.32:0/3745213432 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f0c20077040 0x7f0c20079550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:03.443 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:03.442+0000 7f0c49000640 1 --2- 172.21.15.32:0/3745213432 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f0c20077040 0x7f0c20079550 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7f0c3400bb40 tx=0x7f0c34005e70 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:57:03.446 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:03.445+0000 7f0c41600640 1 -- 172.21.15.32:0/3745213432 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7f0c2c082290 con 0x7f0c440b4ce0 2024-08-03T20:57:03.691 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:03.690+0000 7f0c4ba00640 1 -- 172.21.15.32:0/3745213432 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f0c440cefa0 con 0x7f0c440b4ce0 2024-08-03T20:57:03.692 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:03.690+0000 7f0c41600640 1 -- 172.21.15.32:0/3745213432 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 191 v191) ==== 95+0+4391 (secure 0 0 0) 0x7f0c2c081b90 con 0x7f0c440b4ce0 2024-08-03T20:57:03.692 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-03T20:57:03.692 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":191,"btime":"2024-08-03T20:57:02:150543+0000","default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6439,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6835/531080160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6834","nonce":531080160},{"type":"v1","addr":"172.21.15.118:6835","nonce":531080160}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":84},{"gid":6454,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6837/2796882319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6836","nonce":2796882319},{"type":"v1","addr":"172.21.15.118:6837","nonce":2796882319}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":86},{"gid":6813,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6834/2686933890","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":2686933890},{"type":"v1","addr":"172.21.15.32:6834","nonce":2686933890}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":85}],"filesystems":[{"mdsmap":{"epoch":191,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-08-03T20:54:26.950518+0000","modified":"2024-08-03T20:57:02.150443+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":102,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":6}]} 2024-08-03T20:57:03.693 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 191 2024-08-03T20:57:03.694 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:03.693+0000 7f0c4ba00640 1 -- 172.21.15.32:0/3745213432 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f0c20077040 msgr2=0x7f0c20079550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:03.695 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:03.693+0000 7f0c4ba00640 1 --2- 172.21.15.32:0/3745213432 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f0c20077040 0x7f0c20079550 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7f0c3400bb40 tx=0x7f0c34005e70 comp rx=0 tx=0).stop 2024-08-03T20:57:03.695 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:03.693+0000 7f0c4ba00640 1 -- 172.21.15.32:0/3745213432 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0c440b4ce0 msgr2=0x7f0c44152190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:03.695 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:03.693+0000 7f0c4ba00640 1 --2- 172.21.15.32:0/3745213432 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0c440b4ce0 0x7f0c44152190 secure :-1 s=READY pgs=1478 cs=0 l=1 rev1=1 crypto rx=0x7f0c2c034040 tx=0x7f0c2c0039e0 comp rx=0 tx=0).stop 2024-08-03T20:57:03.695 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:03.693+0000 7f0c4ba00640 1 -- 172.21.15.32:0/3745213432 shutdown_connections 2024-08-03T20:57:03.695 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:03.693+0000 7f0c4ba00640 1 --2- 172.21.15.32:0/3745213432 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f0c20077040 0x7f0c20079550 unknown :-1 s=CLOSED pgs=604 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:03.695 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:03.693+0000 7f0c4ba00640 1 --2- 172.21.15.32:0/3745213432 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f0c4415f440 0x7f0c4414f6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:03.695 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:03.693+0000 7f0c4ba00640 1 --2- 172.21.15.32:0/3745213432 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0c440b4ce0 0x7f0c44152190 unknown :-1 s=CLOSED pgs=1478 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:03.696 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:03.693+0000 7f0c4ba00640 1 --2- 172.21.15.32:0/3745213432 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f0c440b38b0 0x7f0c44151c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:03.696 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:03.693+0000 7f0c4ba00640 1 -- 172.21.15.32:0/3745213432 >> 172.21.15.32:0/3745213432 conn(0x7f0c440b1bd0 msgr2=0x7f0c4415a440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:57:03.696 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:03.693+0000 7f0c4ba00640 1 -- 172.21.15.32:0/3745213432 shutdown_connections 2024-08-03T20:57:03.696 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:03.693+0000 7f0c4ba00640 1 -- 172.21.15.32:0/3745213432 wait complete. 2024-08-03T20:57:03.717 INFO:tasks.cephfs.test_damage:Result: Mutation 'Corrupt 200.00000000' led to DAMAGED state 2024-08-03T20:57:03.717 INFO:tasks.cephfs.test_damage:Applying mutation 'Corrupt 200.00000001' 2024-08-03T20:57:03.717 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-08-03T20:57:03.717 DEBUG:teuthology.orchestra.run.smithi032:> dd if=/proc/self/mounts of=/dev/stdout 2024-08-03T20:57:03.753 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053453,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6499208k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 none /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249604k,nr_inodes=812401,mode=700,uid=1000,gid=1316 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6499208k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-08-03T20:57:03.753 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi032.front.sepia.ceph.com 2024-08-03T20:57:03.753 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-08-03T20:57:03.753 DEBUG:teuthology.orchestra.run.smithi032:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-08-03T20:57:03.828 INFO:teuthology.orchestra.run.smithi032.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.0': No such file or directory 2024-08-03T20:57:03.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-03T20:57:03.829 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-08-03T20:57:04.099 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:04.097+0000 7fe0f4e00640 1 -- 172.21.15.32:0/2910672391 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe0d4004d40 msgr2=0x7fe0d4005190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:04.099 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:04.097+0000 7fe0f4e00640 1 --2- 172.21.15.32:0/2910672391 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe0d4004d40 0x7fe0d4005190 secure :-1 s=READY pgs=1480 cs=0 l=1 rev1=1 crypto rx=0x7fe0f00d0a30 tx=0x7fe0d4005830 comp rx=0 tx=0).stop 2024-08-03T20:57:04.100 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:04.097+0000 7fe0efe00640 1 -- 172.21.15.32:0/2910672391 reap_dead start 2024-08-03T20:57:04.100 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:04.097+0000 7fe0f4e00640 1 -- 172.21.15.32:0/2910672391 shutdown_connections 2024-08-03T20:57:04.100 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:04.097+0000 7fe0f4e00640 1 -- 172.21.15.32:0/2910672391 >> 172.21.15.32:0/2910672391 conn(0x7fe0f008d7a0 msgr2=0x7fe0f009d3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:57:04.100 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:04.097+0000 7fe0f4e00640 1 -- 172.21.15.32:0/2910672391 shutdown_connections 2024-08-03T20:57:04.100 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:04.097+0000 7fe0f4e00640 1 -- 172.21.15.32:0/2910672391 wait complete. 2024-08-03T20:57:04.101 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:04.100+0000 7fe0f4e00640 1 Processor -- start 2024-08-03T20:57:04.101 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:04.100+0000 7fe0f4e00640 1 -- start start 2024-08-03T20:57:04.103 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:04.101+0000 7fe0f4e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe0d4004d40 0x7fe0f0152a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:04.106 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:04.101+0000 7fe0f4e00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fe0f015c630 0x7fe0f015afc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:04.106 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:04.101+0000 7fe0f4e00640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fe0f00b7fa0 0x7fe0f00b83b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:04.106 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:04.101+0000 7fe0f4e00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fe0f00b4220 con 0x7fe0d4004d40 2024-08-03T20:57:04.107 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:04.101+0000 7fe0f4e00640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7fe0f00b40a0 con 0x7fe0f015c630 2024-08-03T20:57:04.107 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:04.101+0000 7fe0f4e00640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7fe0f00d3f80 con 0x7fe0f00b7fa0 2024-08-03T20:57:04.107 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:04.102+0000 7fe0ef400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe0d4004d40 0x7fe0f0152a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:04.107 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:04.102+0000 7fe0ef400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe0d4004d40 0x7fe0f0152a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:47026/0 (socket says 172.21.15.32:47026) 2024-08-03T20:57:04.107 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:04.102+0000 7fe0ef400640 1 -- 172.21.15.32:0/3256670727 learned_addr learned my addr 172.21.15.32:0/3256670727 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:57:04.107 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:04.102+0000 7fe0e7e00640 1 --2- 172.21.15.32:0/3256670727 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fe0f015c630 0x7fe0f015afc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:04.107 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:04.102+0000 7fe0efe00640 1 --2- 172.21.15.32:0/3256670727 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fe0f00b7fa0 0x7fe0f00b83b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:04.108 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:04.102+0000 7fe0ef400640 1 -- 172.21.15.32:0/3256670727 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fe0f00b7fa0 msgr2=0x7fe0f00b83b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:04.108 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:04.102+0000 7fe0ef400640 1 --2- 172.21.15.32:0/3256670727 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fe0f00b7fa0 0x7fe0f00b83b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:04.108 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:04.102+0000 7fe0ef400640 1 -- 172.21.15.32:0/3256670727 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fe0f015c630 msgr2=0x7fe0f015afc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:04.108 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:04.102+0000 7fe0ef400640 1 --2- 172.21.15.32:0/3256670727 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fe0f015c630 0x7fe0f015afc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:04.108 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:04.102+0000 7fe0ef400640 1 -- 172.21.15.32:0/3256670727 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe0f00b88f0 con 0x7fe0d4004d40 2024-08-03T20:57:04.108 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:04.102+0000 7fe0efe00640 1 --2- 172.21.15.32:0/3256670727 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fe0f00b7fa0 0x7fe0f00b83b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-03T20:57:04.109 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:04.102+0000 7fe0ef400640 1 --2- 172.21.15.32:0/3256670727 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe0d4004d40 0x7fe0f0152a00 secure :-1 s=READY pgs=1481 cs=0 l=1 rev1=1 crypto rx=0x7fe0e0004e00 tx=0x7fe0e0001730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:57:04.110 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:04.103+0000 7fe0e7400640 1 -- 172.21.15.32:0/3256670727 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe0e0015020 con 0x7fe0d4004d40 2024-08-03T20:57:04.110 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:04.103+0000 7fe0e7400640 1 -- 172.21.15.32:0/3256670727 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fe0e0011960 con 0x7fe0d4004d40 2024-08-03T20:57:04.110 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:04.103+0000 7fe0f4e00640 1 -- 172.21.15.32:0/3256670727 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe0f015bb30 con 0x7fe0d4004d40 2024-08-03T20:57:04.110 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:04.103+0000 7fe0e7400640 1 -- 172.21.15.32:0/3256670727 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe0e0011db0 con 0x7fe0d4004d40 2024-08-03T20:57:04.110 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:04.103+0000 7fe0f4e00640 1 -- 172.21.15.32:0/3256670727 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe0f00a0930 con 0x7fe0d4004d40 2024-08-03T20:57:04.111 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:04.104+0000 7fe0f4e00640 1 -- 172.21.15.32:0/3256670727 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe0f00d4900 con 0x7fe0d4004d40 2024-08-03T20:57:04.111 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:04.105+0000 7fe0e7400640 1 -- 172.21.15.32:0/3256670727 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7fe0e0018020 con 0x7fe0d4004d40 2024-08-03T20:57:04.111 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:04.106+0000 7fe0e7400640 1 --2- 172.21.15.32:0/3256670727 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fe0cc077010 0x7fe0cc079520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:04.111 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:04.106+0000 7fe0e7400640 1 -- 172.21.15.32:0/3256670727 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(102..102 src has 1..102) ==== 7316+0+0 (secure 0 0 0) 0x7fe0e0099900 con 0x7fe0d4004d40 2024-08-03T20:57:04.111 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:04.107+0000 7fe0e7e00640 1 --2- 172.21.15.32:0/3256670727 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fe0cc077010 0x7fe0cc079520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:04.111 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:04.107+0000 7fe0e7e00640 1 --2- 172.21.15.32:0/3256670727 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fe0cc077010 0x7fe0cc079520 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7fe0f00d11e0 tx=0x7fe0d8005f90 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:57:04.112 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:04.111+0000 7fe0e7400640 1 -- 172.21.15.32:0/3256670727 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7fe0e001f020 con 0x7fe0d4004d40 2024-08-03T20:57:04.361 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:04.359+0000 7fe0f4e00640 1 -- 172.21.15.32:0/3256670727 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7fe0f00008d0 con 0x7fe0d4004d40 2024-08-03T20:57:05.162 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:05.161+0000 7fe0e7400640 1 -- 172.21.15.32:0/3256670727 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v192) ==== 161+0+0 (secure 0 0 0) 0x7fe0f00b88f0 con 0x7fe0d4004d40 2024-08-03T20:57:05.163 INFO:teuthology.orchestra.run.smithi032.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-08-03T20:57:05.166 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:05.164+0000 7fe0f4e00640 1 -- 172.21.15.32:0/3256670727 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fe0cc077010 msgr2=0x7fe0cc079520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:05.166 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:05.164+0000 7fe0f4e00640 1 --2- 172.21.15.32:0/3256670727 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fe0cc077010 0x7fe0cc079520 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7fe0f00d11e0 tx=0x7fe0d8005f90 comp rx=0 tx=0).stop 2024-08-03T20:57:05.166 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:05.164+0000 7fe0f4e00640 1 -- 172.21.15.32:0/3256670727 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe0d4004d40 msgr2=0x7fe0f0152a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:05.166 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:05.164+0000 7fe0f4e00640 1 --2- 172.21.15.32:0/3256670727 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe0d4004d40 0x7fe0f0152a00 secure :-1 s=READY pgs=1481 cs=0 l=1 rev1=1 crypto rx=0x7fe0e0004e00 tx=0x7fe0e0001730 comp rx=0 tx=0).stop 2024-08-03T20:57:05.166 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:05.164+0000 7fe0f4e00640 1 -- 172.21.15.32:0/3256670727 shutdown_connections 2024-08-03T20:57:05.167 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:05.164+0000 7fe0f4e00640 1 --2- 172.21.15.32:0/3256670727 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fe0cc077010 0x7fe0cc079520 unknown :-1 s=CLOSED pgs=605 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:05.167 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:05.164+0000 7fe0f4e00640 1 --2- 172.21.15.32:0/3256670727 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fe0f015c630 0x7fe0f015afc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:05.167 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:05.164+0000 7fe0f4e00640 1 --2- 172.21.15.32:0/3256670727 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fe0f00b7fa0 0x7fe0f00b83b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:05.167 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:05.164+0000 7fe0f4e00640 1 --2- 172.21.15.32:0/3256670727 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe0d4004d40 0x7fe0f0152a00 unknown :-1 s=CLOSED pgs=1481 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:05.167 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:05.164+0000 7fe0f4e00640 1 -- 172.21.15.32:0/3256670727 >> 172.21.15.32:0/3256670727 conn(0x7fe0f008d7a0 msgr2=0x7fe0f00d1cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:57:05.167 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:05.164+0000 7fe0f4e00640 1 -- 172.21.15.32:0/3256670727 shutdown_connections 2024-08-03T20:57:05.167 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:05.164+0000 7fe0f4e00640 1 -- 172.21.15.32:0/3256670727 wait complete. 2024-08-03T20:57:05.193 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mds repaired 0 2024-08-03T20:57:05.428 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:05.426+0000 7f059b000640 1 -- 172.21.15.32:0/1772395989 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f0594144f90 msgr2=0x7f0594074570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:05.428 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:05.426+0000 7f059b000640 1 --2- 172.21.15.32:0/1772395989 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f0594144f90 0x7f0594074570 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f057c009830 tx=0x7f057c02a510 comp rx=0 tx=0).stop 2024-08-03T20:57:05.428 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:05.426+0000 7f059b000640 1 -- 172.21.15.32:0/1772395989 shutdown_connections 2024-08-03T20:57:05.428 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:05.426+0000 7f059b000640 1 --2- 172.21.15.32:0/1772395989 >> v2:172.21.15.118:3300/0 conn(0x7f059415fff0 0x7f0594160400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:05.428 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:05.426+0000 7f059b000640 1 --2- 172.21.15.32:0/1772395989 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f0594144f90 0x7f0594074570 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:05.428 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:05.426+0000 7f059b000640 1 -- 172.21.15.32:0/1772395989 >> 172.21.15.32:0/1772395989 conn(0x7f05940b1bd0 msgr2=0x7f05940b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:57:05.428 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:05.426+0000 7f059b000640 1 -- 172.21.15.32:0/1772395989 shutdown_connections 2024-08-03T20:57:05.429 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:05.426+0000 7f059b000640 1 -- 172.21.15.32:0/1772395989 wait complete. 2024-08-03T20:57:05.430 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:05.428+0000 7f059b000640 1 Processor -- start 2024-08-03T20:57:05.430 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:05.428+0000 7f059b000640 1 -- start start 2024-08-03T20:57:05.432 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:05.430+0000 7f059b000640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f059406e930 0x7f059415b580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:05.432 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:05.430+0000 7f059b000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0594144f90 0x7f059413f380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:05.432 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:05.430+0000 7f059b000640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f059415fff0 0x7f059413f8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:05.432 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:05.430+0000 7f059b000640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f05940cf840 con 0x7f0594144f90 2024-08-03T20:57:05.432 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:05.430+0000 7f059b000640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f0594145f30 con 0x7f059415fff0 2024-08-03T20:57:05.432 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:05.430+0000 7f059b000640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7f05940cf9c0 con 0x7f059406e930 2024-08-03T20:57:05.432 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:05.430+0000 7f058be00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0594144f90 0x7f059413f380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:05.432 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:05.430+0000 7f058be00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0594144f90 0x7f059413f380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:47036/0 (socket says 172.21.15.32:47036) 2024-08-03T20:57:05.432 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:05.430+0000 7f058be00640 1 -- 172.21.15.32:0/2273570151 learned_addr learned my addr 172.21.15.32:0/2273570151 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:57:05.432 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:05.430+0000 7f0593e00640 1 --2- 172.21.15.32:0/2273570151 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f059406e930 0x7f059415b580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:05.432 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:05.430+0000 7f0599000640 1 --2- 172.21.15.32:0/2273570151 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f059415fff0 0x7f059413f8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:05.433 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:05.431+0000 7f058be00640 1 -- 172.21.15.32:0/2273570151 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f059406e930 msgr2=0x7f059415b580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:05.434 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:05.431+0000 7f058be00640 1 --2- 172.21.15.32:0/2273570151 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f059406e930 0x7f059415b580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:05.434 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:05.431+0000 7f058be00640 1 -- 172.21.15.32:0/2273570151 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f059415fff0 msgr2=0x7f059413f8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:05.434 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:05.431+0000 7f058be00640 1 --2- 172.21.15.32:0/2273570151 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f059415fff0 0x7f059413f8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:05.434 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:05.431+0000 7f058be00640 1 -- 172.21.15.32:0/2273570151 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f05941f5b50 con 0x7f0594144f90 2024-08-03T20:57:05.434 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:05.431+0000 7f0599000640 1 --2- 172.21.15.32:0/2273570151 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f059415fff0 0x7f059413f8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-03T20:57:05.434 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:05.431+0000 7f058be00640 1 --2- 172.21.15.32:0/2273570151 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0594144f90 0x7f059413f380 secure :-1 s=READY pgs=1482 cs=0 l=1 rev1=1 crypto rx=0x7f057c009950 tx=0x7f057c033c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:57:05.434 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:05.431+0000 7f0590c00640 1 -- 172.21.15.32:0/2273570151 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f057c005c90 con 0x7f0594144f90 2024-08-03T20:57:05.434 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:05.431+0000 7f0590c00640 1 -- 172.21.15.32:0/2273570151 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f057c031e50 con 0x7f0594144f90 2024-08-03T20:57:05.434 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:05.431+0000 7f059b000640 1 -- 172.21.15.32:0/2273570151 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f05941f5e30 con 0x7f0594144f90 2024-08-03T20:57:05.434 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:05.431+0000 7f0590c00640 1 -- 172.21.15.32:0/2273570151 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f057c0332f0 con 0x7f0594144f90 2024-08-03T20:57:05.434 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:05.431+0000 7f059b000640 1 -- 172.21.15.32:0/2273570151 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f05941f63f0 con 0x7f0594144f90 2024-08-03T20:57:05.435 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:05.432+0000 7f059b000640 1 -- 172.21.15.32:0/2273570151 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0594160bc0 con 0x7f0594144f90 2024-08-03T20:57:05.435 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:05.434+0000 7f0590c00640 1 -- 172.21.15.32:0/2273570151 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f057c033490 con 0x7f0594144f90 2024-08-03T20:57:05.436 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:05.435+0000 7f0590c00640 1 --2- 172.21.15.32:0/2273570151 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f0570077010 0x7f0570079520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:05.436 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:05.435+0000 7f0590c00640 1 -- 172.21.15.32:0/2273570151 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(102..102 src has 1..102) ==== 7316+0+0 (secure 0 0 0) 0x7f057c0ba280 con 0x7f0594144f90 2024-08-03T20:57:05.436 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:05.435+0000 7f0593e00640 1 --2- 172.21.15.32:0/2273570151 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f0570077010 0x7f0570079520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:05.437 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:05.436+0000 7f0593e00640 1 --2- 172.21.15.32:0/2273570151 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f0570077010 0x7f0570079520 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7f0584007fd0 tx=0x7f0584009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:57:05.440 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:05.439+0000 7f0590c00640 1 -- 172.21.15.32:0/2273570151 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7f057c03f030 con 0x7f0594144f90 2024-08-03T20:57:05.726 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:05.725+0000 7f059b000640 1 -- 172.21.15.32:0/2273570151 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "mds repaired", "role": "0"} v 0) -- 0x7f05940afcd0 con 0x7f0594144f90 2024-08-03T20:57:06.164 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.162+0000 7f0590c00640 1 -- 172.21.15.32:0/2273570151 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "mds repaired", "role": "0"}]=0 repaired: restoring rank 6:0 v193) ==== 101+0+0 (secure 0 0 0) 0x7f057c082da0 con 0x7f0594144f90 2024-08-03T20:57:06.164 INFO:teuthology.orchestra.run.smithi032.stderr:repaired: restoring rank 6:0 2024-08-03T20:57:06.167 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.165+0000 7f059b000640 1 -- 172.21.15.32:0/2273570151 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f0570077010 msgr2=0x7f0570079520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:06.167 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.165+0000 7f059b000640 1 --2- 172.21.15.32:0/2273570151 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f0570077010 0x7f0570079520 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7f0584007fd0 tx=0x7f0584009040 comp rx=0 tx=0).stop 2024-08-03T20:57:06.167 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.165+0000 7f059b000640 1 -- 172.21.15.32:0/2273570151 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0594144f90 msgr2=0x7f059413f380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:06.167 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.165+0000 7f059b000640 1 --2- 172.21.15.32:0/2273570151 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0594144f90 0x7f059413f380 secure :-1 s=READY pgs=1482 cs=0 l=1 rev1=1 crypto rx=0x7f057c009950 tx=0x7f057c033c20 comp rx=0 tx=0).stop 2024-08-03T20:57:06.167 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.165+0000 7f059b000640 1 -- 172.21.15.32:0/2273570151 shutdown_connections 2024-08-03T20:57:06.167 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.165+0000 7f059b000640 1 --2- 172.21.15.32:0/2273570151 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f0570077010 0x7f0570079520 unknown :-1 s=CLOSED pgs=606 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:06.167 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.165+0000 7f059b000640 1 --2- 172.21.15.32:0/2273570151 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f059415fff0 0x7f059413f8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:06.167 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.165+0000 7f059b000640 1 --2- 172.21.15.32:0/2273570151 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0594144f90 0x7f059413f380 unknown :-1 s=CLOSED pgs=1482 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:06.167 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.165+0000 7f059b000640 1 --2- 172.21.15.32:0/2273570151 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f059406e930 0x7f059415b580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:06.167 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.165+0000 7f059b000640 1 -- 172.21.15.32:0/2273570151 >> 172.21.15.32:0/2273570151 conn(0x7f05940b1bd0 msgr2=0x7f05940b4750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:57:06.167 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.165+0000 7f059b000640 1 -- 172.21.15.32:0/2273570151 shutdown_connections 2024-08-03T20:57:06.167 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.165+0000 7f059b000640 1 -- 172.21.15.32:0/2273570151 wait complete. 2024-08-03T20:57:06.190 DEBUG:teuthology.orchestra.run.smithi032:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata import - 2024-08-03T20:57:06.332 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.330+0000 7fd6de0eef40 1 Processor -- start 2024-08-03T20:57:06.332 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.330+0000 7fd6de0eef40 1 -- start start 2024-08-03T20:57:06.333 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.332+0000 7fd6de0eef40 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5653c1c59360 0x5653c1c73e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:06.333 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.332+0000 7fd6de0eef40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5653c1c76cb0 0x5653c1beb430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:06.333 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.332+0000 7fd6de0eef40 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x5653c1c773f0 0x5653c1beb970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:06.333 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.332+0000 7fd6de0eef40 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x5653c1af0e30 con 0x5653c1c59360 2024-08-03T20:57:06.333 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.332+0000 7fd6de0eef40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x5653c1b4ebf0 con 0x5653c1c76cb0 2024-08-03T20:57:06.334 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.332+0000 7fd6de0eef40 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x5653c1c5a440 con 0x5653c1c773f0 2024-08-03T20:57:06.334 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.332+0000 7fd6d7200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5653c1c59360 0x5653c1c73e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:06.334 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.332+0000 7fd6d7200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5653c1c59360 0x5653c1c73e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:47052/0 (socket says 172.21.15.32:47052) 2024-08-03T20:57:06.334 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.332+0000 7fd6d7200640 1 -- 172.21.15.32:0/1276092193 learned_addr learned my addr 172.21.15.32:0/1276092193 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:57:06.336 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.332+0000 7fd6d7200640 1 -- 172.21.15.32:0/1276092193 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x5653c1c773f0 msgr2=0x5653c1beb970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:57:06.336 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.332+0000 7fd6d6800640 1 --2- 172.21.15.32:0/1276092193 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5653c1c76cb0 0x5653c1beb430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:06.336 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.332+0000 7fd6d7c00640 1 --2- 172.21.15.32:0/1276092193 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x5653c1c773f0 0x5653c1beb970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:06.336 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.332+0000 7fd6d7200640 1 --2- 172.21.15.32:0/1276092193 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x5653c1c773f0 0x5653c1beb970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:06.337 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.332+0000 7fd6d7200640 1 -- 172.21.15.32:0/1276092193 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5653c1c76cb0 msgr2=0x5653c1beb430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:57:06.337 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.332+0000 7fd6d7200640 1 --2- 172.21.15.32:0/1276092193 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5653c1c76cb0 0x5653c1beb430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:06.337 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.332+0000 7fd6d7200640 1 -- 172.21.15.32:0/1276092193 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5653c1c6e960 con 0x5653c1c59360 2024-08-03T20:57:06.337 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.332+0000 7fd6d6800640 1 --2- 172.21.15.32:0/1276092193 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5653c1c76cb0 0x5653c1beb430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-03T20:57:06.337 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.332+0000 7fd6d7c00640 1 --2- 172.21.15.32:0/1276092193 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x5653c1c773f0 0x5653c1beb970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-03T20:57:06.337 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.333+0000 7fd6d7200640 1 --2- 172.21.15.32:0/1276092193 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5653c1c59360 0x5653c1c73e30 secure :-1 s=READY pgs=1485 cs=0 l=1 rev1=1 crypto rx=0x7fd6c8004e90 tx=0x7fd6c8002fb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=28f44ef1db05139d server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:57:06.337 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.333+0000 7fd6d5400640 1 -- 172.21.15.32:0/1276092193 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6c804d820 con 0x5653c1c59360 2024-08-03T20:57:06.337 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.333+0000 7fd6d5400640 1 -- 172.21.15.32:0/1276092193 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fd6c804dcd0 con 0x5653c1c59360 2024-08-03T20:57:06.337 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.333+0000 7fd6d5400640 1 -- 172.21.15.32:0/1276092193 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6c80566d0 con 0x5653c1c59360 2024-08-03T20:57:06.337 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.333+0000 7fd6de0eef40 1 -- 172.21.15.32:0/1276092193 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5653c1c59360 msgr2=0x5653c1c73e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:06.337 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.333+0000 7fd6de0eef40 1 --2- 172.21.15.32:0/1276092193 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5653c1c59360 0x5653c1c73e30 secure :-1 s=READY pgs=1485 cs=0 l=1 rev1=1 crypto rx=0x7fd6c8004e90 tx=0x7fd6c8002fb0 comp rx=0 tx=0).stop 2024-08-03T20:57:06.337 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.334+0000 7fd6de0eef40 1 -- 172.21.15.32:0/1276092193 shutdown_connections 2024-08-03T20:57:06.337 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.334+0000 7fd6de0eef40 1 --2- 172.21.15.32:0/1276092193 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5653c1c76cb0 0x5653c1beb430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:06.338 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.334+0000 7fd6de0eef40 1 --2- 172.21.15.32:0/1276092193 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x5653c1c773f0 0x5653c1beb970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:06.338 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.334+0000 7fd6de0eef40 1 --2- 172.21.15.32:0/1276092193 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5653c1c59360 0x5653c1c73e30 unknown :-1 s=CLOSED pgs=1485 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:06.338 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.334+0000 7fd6de0eef40 1 -- 172.21.15.32:0/1276092193 >> 172.21.15.32:0/1276092193 conn(0x5653c1ac2ac0 msgr2=0x5653c1be38b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:57:06.338 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.334+0000 7fd6de0eef40 1 -- 172.21.15.32:0/1276092193 shutdown_connections 2024-08-03T20:57:06.338 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.334+0000 7fd6de0eef40 1 -- 172.21.15.32:0/1276092193 wait complete. 2024-08-03T20:57:06.338 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.335+0000 7fd6de0eef40 1 Processor -- start 2024-08-03T20:57:06.338 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.335+0000 7fd6de0eef40 1 -- start start 2024-08-03T20:57:06.340 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.336+0000 7fd6de0eef40 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x5653c1c59360 0x5653c1c4e670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:06.340 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.336+0000 7fd6de0eef40 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5653c1c76cb0 0x5653c1be59d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:06.340 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.336+0000 7fd6de0eef40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5653c1c773f0 0x5653c1be5f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:06.340 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.336+0000 7fd6de0eef40 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x5653c1c6f3e0 con 0x5653c1c76cb0 2024-08-03T20:57:06.340 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.336+0000 7fd6de0eef40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x5653c1c6f630 con 0x5653c1c773f0 2024-08-03T20:57:06.340 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.336+0000 7fd6de0eef40 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x5653c1d09a20 con 0x5653c1c59360 2024-08-03T20:57:06.340 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.336+0000 7fd6d6800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5653c1c76cb0 0x5653c1be59d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:06.340 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.336+0000 7fd6d6800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5653c1c76cb0 0x5653c1be59d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:47056/0 (socket says 172.21.15.32:47056) 2024-08-03T20:57:06.340 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.337+0000 7fd6d6800640 1 -- 172.21.15.32:0/2953897175 learned_addr learned my addr 172.21.15.32:0/2953897175 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:57:06.340 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.337+0000 7fd6d7200640 1 --2- 172.21.15.32:0/2953897175 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x5653c1c59360 0x5653c1c4e670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:06.340 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.337+0000 7fd6d7c00640 1 --2- 172.21.15.32:0/2953897175 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5653c1c773f0 0x5653c1be5f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:06.340 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.337+0000 7fd6d6800640 1 -- 172.21.15.32:0/2953897175 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x5653c1c59360 msgr2=0x5653c1c4e670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:06.341 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.337+0000 7fd6d6800640 1 --2- 172.21.15.32:0/2953897175 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x5653c1c59360 0x5653c1c4e670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:06.341 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.337+0000 7fd6d6800640 1 -- 172.21.15.32:0/2953897175 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5653c1c773f0 msgr2=0x5653c1be5f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:06.341 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.337+0000 7fd6d6800640 1 --2- 172.21.15.32:0/2953897175 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5653c1c773f0 0x5653c1be5f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:06.341 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.337+0000 7fd6d6800640 1 -- 172.21.15.32:0/2953897175 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5653c1beb660 con 0x5653c1c76cb0 2024-08-03T20:57:06.341 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.337+0000 7fd6d7c00640 1 --2- 172.21.15.32:0/2953897175 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5653c1c773f0 0x5653c1be5f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-03T20:57:06.341 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.337+0000 7fd6d7200640 1 --2- 172.21.15.32:0/2953897175 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x5653c1c59360 0x5653c1c4e670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-03T20:57:06.341 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.337+0000 7fd6d6800640 1 --2- 172.21.15.32:0/2953897175 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5653c1c76cb0 0x5653c1be59d0 secure :-1 s=READY pgs=1486 cs=0 l=1 rev1=1 crypto rx=0x7fd6c4001e30 tx=0x7fd6c400a720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:57:06.341 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.337+0000 7fd6cfe00640 1 -- 172.21.15.32:0/2953897175 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6c400e800 con 0x5653c1c76cb0 2024-08-03T20:57:06.341 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.337+0000 7fd6cfe00640 1 -- 172.21.15.32:0/2953897175 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fd6c4006050 con 0x5653c1c76cb0 2024-08-03T20:57:06.341 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.337+0000 7fd6de0eef40 1 -- 172.21.15.32:0/2953897175 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x5653c1c523a0 con 0x5653c1c76cb0 2024-08-03T20:57:06.342 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.337+0000 7fd6cfe00640 1 -- 172.21.15.32:0/2953897175 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6c400ed30 con 0x5653c1c76cb0 2024-08-03T20:57:06.342 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.338+0000 7fd6de0eef40 1 -- 172.21.15.32:0/2953897175 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5653c1d09760 con 0x5653c1c76cb0 2024-08-03T20:57:06.342 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.340+0000 7fd6cfe00640 1 -- 172.21.15.32:0/2953897175 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7fd6c4016450 con 0x5653c1c76cb0 2024-08-03T20:57:06.342 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.341+0000 7fd6cfe00640 1 --2- 172.21.15.32:0/2953897175 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fd6b80759f0 0x7fd6b8077f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:06.342 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.341+0000 7fd6cfe00640 1 -- 172.21.15.32:0/2953897175 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(102..102 src has 1..102) ==== 7316+0+0 (secure 0 0 0) 0x7fd6c4097e60 con 0x5653c1c76cb0 2024-08-03T20:57:06.343 INFO:teuthology.orchestra.run.smithi032.stdout:Importing pool 2024-08-03T20:57:06.343 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.341+0000 7fd6de0eef40 1 --2- 172.21.15.32:0/2953897175 >> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] conn(0x5653c1be8580 0x5653c1d1b740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:06.344 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.341+0000 7fd6de0eef40 1 -- 172.21.15.32:0/2953897175 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:1 12.10 12:09401a81:::601.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102) -- 0x5653c1c526a0 con 0x5653c1be8580 2024-08-03T20:57:06.344 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.341+0000 7fd6d7200640 1 --2- 172.21.15.32:0/2953897175 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fd6b80759f0 0x7fd6b8077f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:06.344 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.341+0000 7fd6d7c00640 1 --2- 172.21.15.32:0/2953897175 >> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] conn(0x5653c1be8580 0x5653c1d1b740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:06.344 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.342+0000 7fd6d7c00640 1 --2- 172.21.15.32:0/2953897175 >> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] conn(0x5653c1be8580 0x5653c1d1b740 crc :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:57:06.344 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.342+0000 7fd6d7200640 1 --2- 172.21.15.32:0/2953897175 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fd6b80759f0 0x7fd6b8077f00 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7fd6c8000c00 tx=0x7fd6c8004880 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:57:06.345 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.344+0000 7fd6d7c00640 1 -- 172.21.15.32:0/2953897175 <== osd.0 v2:172.21.15.32:6816/3066156116 1 ==== osd_op_reply(1 601.00000000 [create] v102'505 uv479 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fd6d000b040 con 0x5653c1be8580 2024-08-03T20:57:06.346 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.344+0000 7fd6de0eef40 1 -- 172.21.15.32:0/2953897175 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:2 12.10 12:09401a81:::601.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e102) -- 0x5653c1be8990 con 0x5653c1be8580 2024-08-03T20:57:06.347 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.346+0000 7fd6d7c00640 1 -- 172.21.15.32:0/2953897175 <== osd.0 v2:172.21.15.32:6816/3066156116 2 ==== osd_op_reply(2 601.00000000 [delete] v102'506 uv506 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fd6d000b040 con 0x5653c1be8580 2024-08-03T20:57:06.347 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.346+0000 7fd6de0eef40 1 -- 172.21.15.32:0/2953897175 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:3 12.10 12:09401a81:::601.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102) -- 0x5653c1d1bc80 con 0x5653c1be8580 2024-08-03T20:57:06.349 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::601.00000000:head# 2024-08-03T20:57:06.349 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.348+0000 7fd6d7c00640 1 -- 172.21.15.32:0/2953897175 <== osd.0 v2:172.21.15.32:6816/3066156116 3 ==== osd_op_reply(3 601.00000000 [create] v102'507 uv507 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fd6d000b040 con 0x5653c1be8580 2024-08-03T20:57:06.350 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.348+0000 7fd6de0eef40 1 -- 172.21.15.32:0/2953897175 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:4 12.10 12:09401a81:::601.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102) -- 0x5653c1d1d030 con 0x5653c1be8580 2024-08-03T20:57:06.351 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.349+0000 7fd6d7c00640 1 -- 172.21.15.32:0/2953897175 <== osd.0 v2:172.21.15.32:6816/3066156116 4 ==== osd_op_reply(4 601.00000000 [setxattr (30)] v102'508 uv508 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fd6d000b040 con 0x5653c1be8580 2024-08-03T20:57:06.351 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.349+0000 7fd6de0eef40 1 -- 172.21.15.32:0/2953897175 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:5 12.10 12:09401a81:::601.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102) -- 0x5653c1d1d3d0 con 0x5653c1be8580 2024-08-03T20:57:06.352 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.351+0000 7fd6d7c00640 1 -- 172.21.15.32:0/2953897175 <== osd.0 v2:172.21.15.32:6816/3066156116 5 ==== osd_op_reply(5 601.00000000 [setxattr (62)] v102'509 uv509 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fd6d000b040 con 0x5653c1be8580 2024-08-03T20:57:06.353 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.351+0000 7fd6de0eef40 1 -- 172.21.15.32:0/2953897175 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:6 12.10 12:09401a81:::601.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102) -- 0x5653c1d1c020 con 0x5653c1be8580 2024-08-03T20:57:06.354 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.353+0000 7fd6d7c00640 1 -- 172.21.15.32:0/2953897175 <== osd.0 v2:172.21.15.32:6816/3066156116 6 ==== osd_op_reply(6 601.00000000 [omap-set-header] v102'510 uv510 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fd6d000b040 con 0x5653c1be8580 2024-08-03T20:57:06.354 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.353+0000 7fd6de0eef40 1 -- 172.21.15.32:0/2953897175 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:7 12.10 12:0bd6d154:::602.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102) -- 0x5653c1d1c3c0 con 0x5653c1be8580 2024-08-03T20:57:06.356 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.355+0000 7fd6d7c00640 1 -- 172.21.15.32:0/2953897175 <== osd.0 v2:172.21.15.32:6816/3066156116 7 ==== osd_op_reply(7 602.00000000 [create] v102'511 uv485 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fd6d000b040 con 0x5653c1be8580 2024-08-03T20:57:06.356 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.355+0000 7fd6de0eef40 1 -- 172.21.15.32:0/2953897175 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:8 12.10 12:0bd6d154:::602.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e102) -- 0x5653c1d1c3c0 con 0x5653c1be8580 2024-08-03T20:57:06.358 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.356+0000 7fd6d7c00640 1 -- 172.21.15.32:0/2953897175 <== osd.0 v2:172.21.15.32:6816/3066156116 8 ==== osd_op_reply(8 602.00000000 [delete] v102'512 uv512 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fd6d000b040 con 0x5653c1be8580 2024-08-03T20:57:06.358 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.356+0000 7fd6de0eef40 1 -- 172.21.15.32:0/2953897175 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:9 12.10 12:0bd6d154:::602.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102) -- 0x5653c1d1c3c0 con 0x5653c1be8580 2024-08-03T20:57:06.359 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.358+0000 7fd6d7c00640 1 -- 172.21.15.32:0/2953897175 <== osd.0 v2:172.21.15.32:6816/3066156116 9 ==== osd_op_reply(9 602.00000000 [create] v102'513 uv513 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fd6d000b040 con 0x5653c1be8580 2024-08-03T20:57:06.359 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::602.00000000:head# 2024-08-03T20:57:06.360 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.358+0000 7fd6de0eef40 1 -- 172.21.15.32:0/2953897175 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:10 12.10 12:0bd6d154:::602.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102) -- 0x5653c1d1c3c0 con 0x5653c1be8580 2024-08-03T20:57:06.361 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.360+0000 7fd6d7c00640 1 -- 172.21.15.32:0/2953897175 <== osd.0 v2:172.21.15.32:6816/3066156116 10 ==== osd_op_reply(10 602.00000000 [setxattr (30)] v102'514 uv514 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fd6d000b040 con 0x5653c1be8580 2024-08-03T20:57:06.361 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.360+0000 7fd6de0eef40 1 -- 172.21.15.32:0/2953897175 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:11 12.10 12:0bd6d154:::602.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102) -- 0x5653c1d1c3c0 con 0x5653c1be8580 2024-08-03T20:57:06.363 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.361+0000 7fd6d7c00640 1 -- 172.21.15.32:0/2953897175 <== osd.0 v2:172.21.15.32:6816/3066156116 11 ==== osd_op_reply(11 602.00000000 [setxattr (62)] v102'515 uv515 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fd6d000b040 con 0x5653c1be8580 2024-08-03T20:57:06.363 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.361+0000 7fd6de0eef40 1 -- 172.21.15.32:0/2953897175 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:12 12.10 12:0bd6d154:::602.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102) -- 0x5653c1d1c3c0 con 0x5653c1be8580 2024-08-03T20:57:06.366 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.363+0000 7fd6d7c00640 1 -- 172.21.15.32:0/2953897175 <== osd.0 v2:172.21.15.32:6816/3066156116 12 ==== osd_op_reply(12 602.00000000 [omap-set-header] v102'516 uv516 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fd6d000b040 con 0x5653c1be8580 2024-08-03T20:57:06.366 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.363+0000 7fd6de0eef40 1 -- 172.21.15.32:0/2953897175 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:13 12.10 12:0cf6ad0f:::10000000000.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102) -- 0x5653c1d1c3c0 con 0x5653c1be8580 2024-08-03T20:57:06.366 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.365+0000 7fd6d7c00640 1 -- 172.21.15.32:0/2953897175 <== osd.0 v2:172.21.15.32:6816/3066156116 13 ==== osd_op_reply(13 10000000000.00000000 [create] v102'517 uv492 ondisk = -17 ((17) File exists)) ==== 164+0+0 (crc 0 0 0) 0x7fd6d000b040 con 0x5653c1be8580 2024-08-03T20:57:06.366 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.365+0000 7fd6de0eef40 1 -- 172.21.15.32:0/2953897175 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:14 12.10 12:0cf6ad0f:::10000000000.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e102) -- 0x5653c1d1c3c0 con 0x5653c1be8580 2024-08-03T20:57:06.368 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.366+0000 7fd6d7c00640 1 -- 172.21.15.32:0/2953897175 <== osd.0 v2:172.21.15.32:6816/3066156116 14 ==== osd_op_reply(14 10000000000.00000000 [delete] v102'518 uv518 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7fd6d000b040 con 0x5653c1be8580 2024-08-03T20:57:06.368 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.367+0000 7fd6de0eef40 1 -- 172.21.15.32:0/2953897175 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:15 12.10 12:0cf6ad0f:::10000000000.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102) -- 0x5653c1d1c3c0 con 0x5653c1be8580 2024-08-03T20:57:06.370 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.368+0000 7fd6d7c00640 1 -- 172.21.15.32:0/2953897175 <== osd.0 v2:172.21.15.32:6816/3066156116 15 ==== osd_op_reply(15 10000000000.00000000 [create] v102'519 uv519 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7fd6d000b040 con 0x5653c1be8580 2024-08-03T20:57:06.370 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::10000000000.00000000:head# 2024-08-03T20:57:06.370 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.368+0000 7fd6de0eef40 1 -- 172.21.15.32:0/2953897175 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:16 12.10 12:0cf6ad0f:::10000000000.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102) -- 0x5653c1d1c3c0 con 0x5653c1be8580 2024-08-03T20:57:06.371 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.370+0000 7fd6d7c00640 1 -- 172.21.15.32:0/2953897175 <== osd.0 v2:172.21.15.32:6816/3066156116 16 ==== osd_op_reply(16 10000000000.00000000 [setxattr (30)] v102'520 uv520 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7fd6d000b040 con 0x5653c1be8580 2024-08-03T20:57:06.371 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.370+0000 7fd6de0eef40 1 -- 172.21.15.32:0/2953897175 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:17 12.10 12:0cf6ad0f:::10000000000.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102) -- 0x5653c1d1c3c0 con 0x5653c1be8580 2024-08-03T20:57:06.373 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.371+0000 7fd6d7c00640 1 -- 172.21.15.32:0/2953897175 <== osd.0 v2:172.21.15.32:6816/3066156116 17 ==== osd_op_reply(17 10000000000.00000000 [setxattr (62)] v102'521 uv521 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7fd6d000b040 con 0x5653c1be8580 2024-08-03T20:57:06.373 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.371+0000 7fd6de0eef40 1 -- 172.21.15.32:0/2953897175 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:18 12.10 12:0cf6ad0f:::10000000000.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102) -- 0x5653c1d1c3c0 con 0x5653c1be8580 2024-08-03T20:57:06.374 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.373+0000 7fd6d7c00640 1 -- 172.21.15.32:0/2953897175 <== osd.0 v2:172.21.15.32:6816/3066156116 18 ==== osd_op_reply(18 10000000000.00000000 [omap-set-header] v102'522 uv522 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7fd6d000b040 con 0x5653c1be8580 2024-08-03T20:57:06.374 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.373+0000 7fd6de0eef40 1 -- 172.21.15.32:0/2953897175 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:19 12.10 12:0cf6ad0f:::10000000000.00000000:head [omap-set-vals in=593b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102) -- 0x5653c1d1c7a0 con 0x5653c1be8580 2024-08-03T20:57:06.376 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.374+0000 7fd6d7c00640 1 -- 172.21.15.32:0/2953897175 <== osd.0 v2:172.21.15.32:6816/3066156116 19 ==== osd_op_reply(19 10000000000.00000000 [omap-set-vals] v102'523 uv523 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7fd6d000b040 con 0x5653c1be8580 2024-08-03T20:57:06.376 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.375+0000 7fd6de0eef40 1 -- 172.21.15.32:0/2953897175 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:20 12.10 12:0d82a743:::600.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102) -- 0x5653c1d1c7a0 con 0x5653c1be8580 2024-08-03T20:57:06.377 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.376+0000 7fd6d7c00640 1 -- 172.21.15.32:0/2953897175 <== osd.0 v2:172.21.15.32:6816/3066156116 20 ==== osd_op_reply(20 600.00000000 [create] v102'524 uv498 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fd6d000b040 con 0x5653c1be8580 2024-08-03T20:57:06.378 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.376+0000 7fd6de0eef40 1 -- 172.21.15.32:0/2953897175 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:21 12.10 12:0d82a743:::600.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e102) -- 0x5653c1d1cc10 con 0x5653c1be8580 2024-08-03T20:57:06.379 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.378+0000 7fd6d7c00640 1 -- 172.21.15.32:0/2953897175 <== osd.0 v2:172.21.15.32:6816/3066156116 21 ==== osd_op_reply(21 600.00000000 [delete] v102'525 uv525 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fd6d000b040 con 0x5653c1be8580 2024-08-03T20:57:06.379 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.378+0000 7fd6de0eef40 1 -- 172.21.15.32:0/2953897175 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:22 12.10 12:0d82a743:::600.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102) -- 0x5653c1d1c780 con 0x5653c1be8580 2024-08-03T20:57:06.381 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.380+0000 7fd6d7c00640 1 -- 172.21.15.32:0/2953897175 <== osd.0 v2:172.21.15.32:6816/3066156116 22 ==== osd_op_reply(22 600.00000000 [create] v102'526 uv526 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fd6d000b040 con 0x5653c1be8580 2024-08-03T20:57:06.381 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::600.00000000:head# 2024-08-03T20:57:06.381 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.380+0000 7fd6de0eef40 1 -- 172.21.15.32:0/2953897175 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:23 12.10 12:0d82a743:::600.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102) -- 0x5653c1d1c780 con 0x5653c1be8580 2024-08-03T20:57:06.382 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.381+0000 7fd6d7c00640 1 -- 172.21.15.32:0/2953897175 <== osd.0 v2:172.21.15.32:6816/3066156116 23 ==== osd_op_reply(23 600.00000000 [setxattr (30)] v102'527 uv527 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fd6d000b040 con 0x5653c1be8580 2024-08-03T20:57:06.382 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.381+0000 7fd6de0eef40 1 -- 172.21.15.32:0/2953897175 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:24 12.10 12:0d82a743:::600.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102) -- 0x5653c1d1cb80 con 0x5653c1be8580 2024-08-03T20:57:06.384 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.382+0000 7fd6d7c00640 1 -- 172.21.15.32:0/2953897175 <== osd.0 v2:172.21.15.32:6816/3066156116 24 ==== osd_op_reply(24 600.00000000 [setxattr (62)] v102'528 uv528 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fd6d000b040 con 0x5653c1be8580 2024-08-03T20:57:06.384 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.383+0000 7fd6de0eef40 1 -- 172.21.15.32:0/2953897175 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:25 12.10 12:0d82a743:::600.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102) -- 0x5653c1d1cb80 con 0x5653c1be8580 2024-08-03T20:57:06.386 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.384+0000 7fd6d7c00640 1 -- 172.21.15.32:0/2953897175 <== osd.0 v2:172.21.15.32:6816/3066156116 25 ==== osd_op_reply(25 600.00000000 [omap-set-header] v102'529 uv529 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fd6d000b040 con 0x5653c1be8580 2024-08-03T20:57:06.386 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.384+0000 7fd6de0eef40 1 -- 172.21.15.32:0/2953897175 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:26 12.10 12:0d89b25e:::603.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102) -- 0x5653c1d1cb80 con 0x5653c1be8580 2024-08-03T20:57:06.387 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.386+0000 7fd6d7c00640 1 -- 172.21.15.32:0/2953897175 <== osd.0 v2:172.21.15.32:6816/3066156116 26 ==== osd_op_reply(26 603.00000000 [create] v102'530 uv504 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fd6d000b040 con 0x5653c1be8580 2024-08-03T20:57:06.387 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.386+0000 7fd6de0eef40 1 -- 172.21.15.32:0/2953897175 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:27 12.10 12:0d89b25e:::603.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e102) -- 0x5653c1d1cb80 con 0x5653c1be8580 2024-08-03T20:57:06.389 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.388+0000 7fd6d7c00640 1 -- 172.21.15.32:0/2953897175 <== osd.0 v2:172.21.15.32:6816/3066156116 27 ==== osd_op_reply(27 603.00000000 [delete] v102'531 uv531 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fd6d000b040 con 0x5653c1be8580 2024-08-03T20:57:06.389 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.388+0000 7fd6de0eef40 1 -- 172.21.15.32:0/2953897175 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:28 12.10 12:0d89b25e:::603.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102) -- 0x5653c1d1cb80 con 0x5653c1be8580 2024-08-03T20:57:06.390 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.389+0000 7fd6d7c00640 1 -- 172.21.15.32:0/2953897175 <== osd.0 v2:172.21.15.32:6816/3066156116 28 ==== osd_op_reply(28 603.00000000 [create] v102'532 uv532 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fd6d000b040 con 0x5653c1be8580 2024-08-03T20:57:06.391 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::603.00000000:head# 2024-08-03T20:57:06.391 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.389+0000 7fd6de0eef40 1 -- 172.21.15.32:0/2953897175 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:29 12.10 12:0d89b25e:::603.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102) -- 0x5653c1d1cb80 con 0x5653c1be8580 2024-08-03T20:57:06.392 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.391+0000 7fd6d7c00640 1 -- 172.21.15.32:0/2953897175 <== osd.0 v2:172.21.15.32:6816/3066156116 29 ==== osd_op_reply(29 603.00000000 [setxattr (30)] v102'533 uv533 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fd6d000b040 con 0x5653c1be8580 2024-08-03T20:57:06.393 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.391+0000 7fd6de0eef40 1 -- 172.21.15.32:0/2953897175 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:30 12.10 12:0d89b25e:::603.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102) -- 0x5653c1d1c680 con 0x5653c1be8580 2024-08-03T20:57:06.394 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.393+0000 7fd6d7c00640 1 -- 172.21.15.32:0/2953897175 <== osd.0 v2:172.21.15.32:6816/3066156116 30 ==== osd_op_reply(30 603.00000000 [setxattr (62)] v102'534 uv534 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fd6d000b040 con 0x5653c1be8580 2024-08-03T20:57:06.394 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.393+0000 7fd6de0eef40 1 -- 172.21.15.32:0/2953897175 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:31 12.10 12:0d89b25e:::603.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102) -- 0x5653c1d1c680 con 0x5653c1be8580 2024-08-03T20:57:06.396 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.394+0000 7fd6d7c00640 1 -- 172.21.15.32:0/2953897175 <== osd.0 v2:172.21.15.32:6816/3066156116 31 ==== osd_op_reply(31 603.00000000 [omap-set-header] v102'535 uv535 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fd6d000b040 con 0x5653c1be8580 2024-08-03T20:57:06.396 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.394+0000 7fd6de0eef40 1 --2- 172.21.15.32:0/2953897175 >> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] conn(0x5653c1d1d770 0x5653c1d1fb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:06.396 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.394+0000 7fd6de0eef40 1 -- 172.21.15.32:0/2953897175 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:32 12.14 12:287034c4:::1.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102) -- 0x5653c1d200d0 con 0x5653c1d1d770 2024-08-03T20:57:06.396 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.395+0000 7fd6d7200640 1 --2- 172.21.15.32:0/2953897175 >> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] conn(0x5653c1d1d770 0x5653c1d1fb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:06.397 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.395+0000 7fd6d7200640 1 --2- 172.21.15.32:0/2953897175 >> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] conn(0x5653c1d1d770 0x5653c1d1fb90 crc :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:57:06.399 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.398+0000 7fd6d7200640 1 -- 172.21.15.32:0/2953897175 <== osd.5 v2:172.21.15.118:6810/2724891770 1 ==== osd_op_reply(32 1.00000000.inode [create] v102'374 uv356 ondisk = -17 ((17) File exists)) ==== 160+0+0 (crc 0 0 0) 0x7fd6c805f030 con 0x5653c1d1d770 2024-08-03T20:57:06.399 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.398+0000 7fd6de0eef40 1 -- 172.21.15.32:0/2953897175 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:33 12.14 12:287034c4:::1.00000000.inode:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e102) -- 0x5653c1d20470 con 0x5653c1d1d770 2024-08-03T20:57:06.400 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.399+0000 7fd6d7200640 1 -- 172.21.15.32:0/2953897175 <== osd.5 v2:172.21.15.118:6810/2724891770 2 ==== osd_op_reply(33 1.00000000.inode [delete] v102'375 uv375 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7fd6c805f030 con 0x5653c1d1d770 2024-08-03T20:57:06.400 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.399+0000 7fd6de0eef40 1 -- 172.21.15.32:0/2953897175 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:34 12.14 12:287034c4:::1.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102) -- 0x5653c1d20810 con 0x5653c1d1d770 2024-08-03T20:57:06.402 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::1.00000000.inode:head# 2024-08-03T20:57:06.402 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.400+0000 7fd6d7200640 1 -- 172.21.15.32:0/2953897175 <== osd.5 v2:172.21.15.118:6810/2724891770 3 ==== osd_op_reply(34 1.00000000.inode [create] v102'376 uv376 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7fd6c805f030 con 0x5653c1d1d770 2024-08-03T20:57:06.402 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.400+0000 7fd6de0eef40 1 -- 172.21.15.32:0/2953897175 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:35 12.14 12:287034c4:::1.00000000.inode:head [write 0~582 in=582b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102) -- 0x5653c1d20bb0 con 0x5653c1d1d770 2024-08-03T20:57:06.403 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.402+0000 7fd6d7200640 1 -- 172.21.15.32:0/2953897175 <== osd.5 v2:172.21.15.118:6810/2724891770 4 ==== osd_op_reply(35 1.00000000.inode [write 0~582] v102'377 uv377 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7fd6c805f030 con 0x5653c1d1d770 2024-08-03T20:57:06.404 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.402+0000 7fd6de0eef40 1 -- 172.21.15.32:0/2953897175 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:36 12.14 12:287034c4:::1.00000000.inode:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102) -- 0x5653c1d20f50 con 0x5653c1d1d770 2024-08-03T20:57:06.405 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.403+0000 7fd6d7200640 1 -- 172.21.15.32:0/2953897175 <== osd.5 v2:172.21.15.118:6810/2724891770 5 ==== osd_op_reply(36 1.00000000.inode [omap-set-header] v102'378 uv378 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7fd6c805f030 con 0x5653c1d1d770 2024-08-03T20:57:06.405 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.403+0000 7fd6de0eef40 1 -- 172.21.15.32:0/2953897175 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:37 12.14 12:292cf221:::200.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102) -- 0x5653c1d212f0 con 0x5653c1d1d770 2024-08-03T20:57:06.406 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.405+0000 7fd6d7200640 1 -- 172.21.15.32:0/2953897175 <== osd.5 v2:172.21.15.118:6810/2724891770 6 ==== osd_op_reply(37 200.00000000 [create] v102'379 uv373 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fd6c805f030 con 0x5653c1d1d770 2024-08-03T20:57:06.406 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.405+0000 7fd6de0eef40 1 -- 172.21.15.32:0/2953897175 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:38 12.14 12:292cf221:::200.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e102) -- 0x5653c1d212f0 con 0x5653c1d1d770 2024-08-03T20:57:06.408 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.406+0000 7fd6d7200640 1 -- 172.21.15.32:0/2953897175 <== osd.5 v2:172.21.15.118:6810/2724891770 7 ==== osd_op_reply(38 200.00000000 [delete] v102'380 uv380 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fd6c805f030 con 0x5653c1d1d770 2024-08-03T20:57:06.408 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.406+0000 7fd6de0eef40 1 -- 172.21.15.32:0/2953897175 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:39 12.14 12:292cf221:::200.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102) -- 0x5653c1d212f0 con 0x5653c1d1d770 2024-08-03T20:57:06.409 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::200.00000000:head# 2024-08-03T20:57:06.410 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.408+0000 7fd6d7200640 1 -- 172.21.15.32:0/2953897175 <== osd.5 v2:172.21.15.118:6810/2724891770 8 ==== osd_op_reply(39 200.00000000 [create] v102'381 uv381 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fd6c805f030 con 0x5653c1d1d770 2024-08-03T20:57:06.410 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.408+0000 7fd6de0eef40 1 -- 172.21.15.32:0/2953897175 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:40 12.14 12:292cf221:::200.00000000:head [write 0~90 in=90b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102) -- 0x5653c1d212f0 con 0x5653c1d1d770 2024-08-03T20:57:06.411 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.409+0000 7fd6d7200640 1 -- 172.21.15.32:0/2953897175 <== osd.5 v2:172.21.15.118:6810/2724891770 9 ==== osd_op_reply(40 200.00000000 [write 0~90] v102'382 uv382 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fd6c805f030 con 0x5653c1d1d770 2024-08-03T20:57:06.411 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.409+0000 7fd6de0eef40 1 -- 172.21.15.32:0/2953897175 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:41 12.14 12:292cf221:::200.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102) -- 0x5653c1d212f0 con 0x5653c1d1d770 2024-08-03T20:57:06.412 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.411+0000 7fd6d7200640 1 -- 172.21.15.32:0/2953897175 <== osd.5 v2:172.21.15.118:6810/2724891770 10 ==== osd_op_reply(41 200.00000000 [omap-set-header] v102'383 uv383 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fd6c805f030 con 0x5653c1d1d770 2024-08-03T20:57:06.412 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.411+0000 7fd6de0eef40 1 -- 172.21.15.32:0/2953897175 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:42 12.14 12:2e2fa760:::200.00000001:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102) -- 0x5653c1d212f0 con 0x5653c1d1d770 2024-08-03T20:57:06.413 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.412+0000 7fd6d7200640 1 -- 172.21.15.32:0/2953897175 <== osd.5 v2:172.21.15.118:6810/2724891770 11 ==== osd_op_reply(42 200.00000001 [create] v102'384 uv366 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fd6c805f030 con 0x5653c1d1d770 2024-08-03T20:57:06.414 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.412+0000 7fd6de0eef40 1 -- 172.21.15.32:0/2953897175 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:43 12.14 12:2e2fa760:::200.00000001:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e102) -- 0x5653c1d212f0 con 0x5653c1d1d770 2024-08-03T20:57:06.415 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.414+0000 7fd6d7200640 1 -- 172.21.15.32:0/2953897175 <== osd.5 v2:172.21.15.118:6810/2724891770 12 ==== osd_op_reply(43 200.00000001 [delete] v102'385 uv385 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fd6c805f030 con 0x5653c1d1d770 2024-08-03T20:57:06.415 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.414+0000 7fd6de0eef40 1 -- 172.21.15.32:0/2953897175 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:44 12.14 12:2e2fa760:::200.00000001:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102) -- 0x5653c1d212f0 con 0x5653c1d1d770 2024-08-03T20:57:06.416 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::200.00000001:head# 2024-08-03T20:57:06.417 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.415+0000 7fd6d7200640 1 -- 172.21.15.32:0/2953897175 <== osd.5 v2:172.21.15.118:6810/2724891770 13 ==== osd_op_reply(44 200.00000001 [create] v102'386 uv386 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fd6c805f030 con 0x5653c1d1d770 2024-08-03T20:57:06.417 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.415+0000 7fd6de0eef40 1 -- 172.21.15.32:0/2953897175 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:45 12.14 12:2e2fa760:::200.00000001:head [write 0~18118 in=18118b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102) -- 0x5653c1d259e0 con 0x5653c1d1d770 2024-08-03T20:57:06.419 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.417+0000 7fd6d7200640 1 -- 172.21.15.32:0/2953897175 <== osd.5 v2:172.21.15.118:6810/2724891770 14 ==== osd_op_reply(45 200.00000001 [write 0~18118] v102'387 uv387 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fd6c805f030 con 0x5653c1d1d770 2024-08-03T20:57:06.419 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.417+0000 7fd6de0eef40 1 -- 172.21.15.32:0/2953897175 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:46 12.14 12:2e2fa760:::200.00000001:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102) -- 0x5653c1d259e0 con 0x5653c1d1d770 2024-08-03T20:57:06.420 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.419+0000 7fd6d7200640 1 -- 172.21.15.32:0/2953897175 <== osd.5 v2:172.21.15.118:6810/2724891770 15 ==== osd_op_reply(46 200.00000001 [omap-set-header] v102'388 uv388 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fd6c805f030 con 0x5653c1d1d770 2024-08-03T20:57:06.420 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.419+0000 7fd6de0eef40 1 -- 172.21.15.32:0/2953897175 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:47 12.14 12:2f55791f:::606.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102) -- 0x5653c1d212f0 con 0x5653c1d1d770 2024-08-03T20:57:06.422 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.420+0000 7fd6d7200640 1 -- 172.21.15.32:0/2953897175 <== osd.5 v2:172.21.15.118:6810/2724891770 16 ==== osd_op_reply(47 606.00000000 [create] v102'389 uv372 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fd6c805f030 con 0x5653c1d1d770 2024-08-03T20:57:06.422 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.420+0000 7fd6de0eef40 1 -- 172.21.15.32:0/2953897175 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:48 12.14 12:2f55791f:::606.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e102) -- 0x5653c1d212f0 con 0x5653c1d1d770 2024-08-03T20:57:06.423 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.422+0000 7fd6d7200640 1 -- 172.21.15.32:0/2953897175 <== osd.5 v2:172.21.15.118:6810/2724891770 17 ==== osd_op_reply(48 606.00000000 [delete] v102'390 uv390 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fd6c805f030 con 0x5653c1d1d770 2024-08-03T20:57:06.423 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.422+0000 7fd6de0eef40 1 -- 172.21.15.32:0/2953897175 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:49 12.14 12:2f55791f:::606.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102) -- 0x5653c1d212f0 con 0x5653c1d1d770 2024-08-03T20:57:06.424 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.423+0000 7fd6d7200640 1 -- 172.21.15.32:0/2953897175 <== osd.5 v2:172.21.15.118:6810/2724891770 18 ==== osd_op_reply(49 606.00000000 [create] v102'391 uv391 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fd6c805f030 con 0x5653c1d1d770 2024-08-03T20:57:06.424 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::606.00000000:head# 2024-08-03T20:57:06.425 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.423+0000 7fd6de0eef40 1 -- 172.21.15.32:0/2953897175 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:50 12.14 12:2f55791f:::606.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102) -- 0x5653c1d22300 con 0x5653c1d1d770 2024-08-03T20:57:06.426 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.425+0000 7fd6d7200640 1 -- 172.21.15.32:0/2953897175 <== osd.5 v2:172.21.15.118:6810/2724891770 19 ==== osd_op_reply(50 606.00000000 [setxattr (30)] v102'392 uv392 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fd6c805f030 con 0x5653c1d1d770 2024-08-03T20:57:06.426 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.425+0000 7fd6de0eef40 1 -- 172.21.15.32:0/2953897175 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:51 12.14 12:2f55791f:::606.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102) -- 0x5653c1d22300 con 0x5653c1d1d770 2024-08-03T20:57:06.427 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.426+0000 7fd6d7200640 1 -- 172.21.15.32:0/2953897175 <== osd.5 v2:172.21.15.118:6810/2724891770 20 ==== osd_op_reply(51 606.00000000 [setxattr (62)] v102'393 uv393 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fd6c805f030 con 0x5653c1d1d770 2024-08-03T20:57:06.427 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.426+0000 7fd6de0eef40 1 -- 172.21.15.32:0/2953897175 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:52 12.14 12:2f55791f:::606.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102) -- 0x5653c1d212f0 con 0x5653c1d1d770 2024-08-03T20:57:06.429 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.427+0000 7fd6d7200640 1 -- 172.21.15.32:0/2953897175 <== osd.5 v2:172.21.15.118:6810/2724891770 21 ==== osd_op_reply(52 606.00000000 [omap-set-header] v102'394 uv394 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fd6c805f030 con 0x5653c1d1d770 2024-08-03T20:57:06.429 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.427+0000 7fd6de0eef40 1 -- 172.21.15.32:0/2953897175 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:53 12.1c 12:3ed09add:::607.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102) -- 0x5653c1d212f0 con 0x5653c1be8580 2024-08-03T20:57:06.430 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.429+0000 7fd6d7c00640 1 -- 172.21.15.32:0/2953897175 <== osd.0 v2:172.21.15.32:6816/3066156116 32 ==== osd_op_reply(53 607.00000000 [create] v102'99 uv98 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fd6d000b040 con 0x5653c1be8580 2024-08-03T20:57:06.430 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.429+0000 7fd6de0eef40 1 -- 172.21.15.32:0/2953897175 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:54 12.1c 12:3ed09add:::607.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e102) -- 0x5653c1d212f0 con 0x5653c1be8580 2024-08-03T20:57:06.432 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.430+0000 7fd6d7c00640 1 -- 172.21.15.32:0/2953897175 <== osd.0 v2:172.21.15.32:6816/3066156116 33 ==== osd_op_reply(54 607.00000000 [delete] v102'100 uv100 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fd6d000b040 con 0x5653c1be8580 2024-08-03T20:57:06.432 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.430+0000 7fd6de0eef40 1 -- 172.21.15.32:0/2953897175 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:55 12.1c 12:3ed09add:::607.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102) -- 0x5653c1d212f0 con 0x5653c1be8580 2024-08-03T20:57:06.433 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.432+0000 7fd6d7c00640 1 -- 172.21.15.32:0/2953897175 <== osd.0 v2:172.21.15.32:6816/3066156116 34 ==== osd_op_reply(55 607.00000000 [create] v102'101 uv101 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fd6d000b040 con 0x5653c1be8580 2024-08-03T20:57:06.433 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::607.00000000:head# 2024-08-03T20:57:06.433 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.432+0000 7fd6de0eef40 1 -- 172.21.15.32:0/2953897175 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:56 12.1c 12:3ed09add:::607.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102) -- 0x5653c1d22300 con 0x5653c1be8580 2024-08-03T20:57:06.434 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.433+0000 7fd6d7c00640 1 -- 172.21.15.32:0/2953897175 <== osd.0 v2:172.21.15.32:6816/3066156116 35 ==== osd_op_reply(56 607.00000000 [setxattr (30)] v102'102 uv102 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fd6d000b040 con 0x5653c1be8580 2024-08-03T20:57:06.434 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.433+0000 7fd6de0eef40 1 -- 172.21.15.32:0/2953897175 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:57 12.1c 12:3ed09add:::607.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102) -- 0x5653c1d22300 con 0x5653c1be8580 2024-08-03T20:57:06.435 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.434+0000 7fd6d7c00640 1 -- 172.21.15.32:0/2953897175 <== osd.0 v2:172.21.15.32:6816/3066156116 36 ==== osd_op_reply(57 607.00000000 [setxattr (62)] v102'103 uv103 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fd6d000b040 con 0x5653c1be8580 2024-08-03T20:57:06.435 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.434+0000 7fd6de0eef40 1 -- 172.21.15.32:0/2953897175 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:58 12.1c 12:3ed09add:::607.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102) -- 0x5653c1d212f0 con 0x5653c1be8580 2024-08-03T20:57:06.437 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.435+0000 7fd6d7c00640 1 -- 172.21.15.32:0/2953897175 <== osd.0 v2:172.21.15.32:6816/3066156116 37 ==== osd_op_reply(58 607.00000000 [omap-set-header] v102'104 uv104 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fd6d000b040 con 0x5653c1be8580 2024-08-03T20:57:06.437 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.435+0000 7fd6de0eef40 1 --2- 172.21.15.32:0/2953897175 >> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] conn(0x5653c1d21970 0x5653c1d23d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:06.437 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.435+0000 7fd6de0eef40 1 -- 172.21.15.32:0/2953897175 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:59 12.6 12:654134d2:::mds0_openfiles.0:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102) -- 0x5653c1d24340 con 0x5653c1d21970 2024-08-03T20:57:06.437 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.436+0000 7fd6d6800640 1 --2- 172.21.15.32:0/2953897175 >> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] conn(0x5653c1d21970 0x5653c1d23d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:06.438 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.436+0000 7fd6d6800640 1 --2- 172.21.15.32:0/2953897175 >> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] conn(0x5653c1d21970 0x5653c1d23d90 crc :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:57:06.439 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.437+0000 7fd6d6800640 1 -- 172.21.15.32:0/2953897175 <== osd.1 v2:172.21.15.32:6824/1551241759 1 ==== osd_op_reply(59 mds0_openfiles.0 [create] v102'82 uv81 ondisk = -17 ((17) File exists)) ==== 160+0+0 (crc 0 0 0) 0x7fd6c405c6d0 con 0x5653c1d21970 2024-08-03T20:57:06.439 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.437+0000 7fd6de0eef40 1 -- 172.21.15.32:0/2953897175 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:60 12.6 12:654134d2:::mds0_openfiles.0:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e102) -- 0x5653c1d24850 con 0x5653c1d21970 2024-08-03T20:57:06.440 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.439+0000 7fd6d6800640 1 -- 172.21.15.32:0/2953897175 <== osd.1 v2:172.21.15.32:6824/1551241759 2 ==== osd_op_reply(60 mds0_openfiles.0 [delete] v102'83 uv83 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7fd6c405c6d0 con 0x5653c1d21970 2024-08-03T20:57:06.440 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.439+0000 7fd6de0eef40 1 -- 172.21.15.32:0/2953897175 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:61 12.6 12:654134d2:::mds0_openfiles.0:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102) -- 0x5653c1d24d50 con 0x5653c1d21970 2024-08-03T20:57:06.441 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.440+0000 7fd6d6800640 1 -- 172.21.15.32:0/2953897175 <== osd.1 v2:172.21.15.32:6824/1551241759 3 ==== osd_op_reply(61 mds0_openfiles.0 [create] v102'84 uv84 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7fd6c405c6d0 con 0x5653c1d21970 2024-08-03T20:57:06.441 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds0_openfiles.0:head# 2024-08-03T20:57:06.442 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.440+0000 7fd6de0eef40 1 -- 172.21.15.32:0/2953897175 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:62 12.6 12:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102) -- 0x5653c1be7ca0 con 0x5653c1d21970 2024-08-03T20:57:06.443 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.442+0000 7fd6d6800640 1 -- 172.21.15.32:0/2953897175 <== osd.1 v2:172.21.15.32:6824/1551241759 4 ==== osd_op_reply(62 mds0_openfiles.0 [omap-set-header] v102'85 uv85 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7fd6c405c6d0 con 0x5653c1d21970 2024-08-03T20:57:06.443 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.442+0000 7fd6de0eef40 1 -- 172.21.15.32:0/2953897175 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:63 12.e 12:717a0223:::608.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102) -- 0x5653c1d25680 con 0x5653c1be8580 2024-08-03T20:57:06.444 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.443+0000 7fd6d7c00640 1 -- 172.21.15.32:0/2953897175 <== osd.0 v2:172.21.15.32:6816/3066156116 38 ==== osd_op_reply(63 608.00000000 [create] v102'99 uv98 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fd6d000b040 con 0x5653c1be8580 2024-08-03T20:57:06.445 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.443+0000 7fd6de0eef40 1 -- 172.21.15.32:0/2953897175 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:64 12.e 12:717a0223:::608.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e102) -- 0x5653c1d25680 con 0x5653c1be8580 2024-08-03T20:57:06.446 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.445+0000 7fd6d7c00640 1 -- 172.21.15.32:0/2953897175 <== osd.0 v2:172.21.15.32:6816/3066156116 39 ==== osd_op_reply(64 608.00000000 [delete] v102'100 uv100 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fd6d000b040 con 0x5653c1be8580 2024-08-03T20:57:06.446 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.445+0000 7fd6de0eef40 1 -- 172.21.15.32:0/2953897175 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:65 12.e 12:717a0223:::608.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102) -- 0x5653c1d25680 con 0x5653c1be8580 2024-08-03T20:57:06.448 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.447+0000 7fd6d7c00640 1 -- 172.21.15.32:0/2953897175 <== osd.0 v2:172.21.15.32:6816/3066156116 40 ==== osd_op_reply(65 608.00000000 [create] v102'101 uv101 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fd6d000b040 con 0x5653c1be8580 2024-08-03T20:57:06.448 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::608.00000000:head# 2024-08-03T20:57:06.448 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.447+0000 7fd6de0eef40 1 -- 172.21.15.32:0/2953897175 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:66 12.e 12:717a0223:::608.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102) -- 0x5653c1d25680 con 0x5653c1be8580 2024-08-03T20:57:06.449 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.448+0000 7fd6d7c00640 1 -- 172.21.15.32:0/2953897175 <== osd.0 v2:172.21.15.32:6816/3066156116 41 ==== osd_op_reply(66 608.00000000 [setxattr (30)] v102'102 uv102 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fd6d000b040 con 0x5653c1be8580 2024-08-03T20:57:06.449 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.448+0000 7fd6de0eef40 1 -- 172.21.15.32:0/2953897175 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:67 12.e 12:717a0223:::608.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102) -- 0x5653c1d25680 con 0x5653c1be8580 2024-08-03T20:57:06.450 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.449+0000 7fd6d7c00640 1 -- 172.21.15.32:0/2953897175 <== osd.0 v2:172.21.15.32:6816/3066156116 42 ==== osd_op_reply(67 608.00000000 [setxattr (62)] v102'103 uv103 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fd6d000b040 con 0x5653c1be8580 2024-08-03T20:57:06.451 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.449+0000 7fd6de0eef40 1 -- 172.21.15.32:0/2953897175 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:68 12.e 12:717a0223:::608.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102) -- 0x5653c1d256c0 con 0x5653c1be8580 2024-08-03T20:57:06.452 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.451+0000 7fd6d7c00640 1 -- 172.21.15.32:0/2953897175 <== osd.0 v2:172.21.15.32:6816/3066156116 43 ==== osd_op_reply(68 608.00000000 [omap-set-header] v102'104 uv104 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fd6d000b040 con 0x5653c1be8580 2024-08-03T20:57:06.452 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.451+0000 7fd6de0eef40 1 --2- 172.21.15.32:0/2953897175 >> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] conn(0x5653c1d25fc0 0x5653c1d263d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:06.453 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.451+0000 7fd6de0eef40 1 -- 172.21.15.32:0/2953897175 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:69 12.15 12:a93a17c2:::604.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102) -- 0x5653c1d28af0 con 0x5653c1d25fc0 2024-08-03T20:57:06.453 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.451+0000 7fd6d7c00640 1 --2- 172.21.15.32:0/2953897175 >> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] conn(0x5653c1d25fc0 0x5653c1d263d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:06.453 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.452+0000 7fd6d7c00640 1 --2- 172.21.15.32:0/2953897175 >> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] conn(0x5653c1d25fc0 0x5653c1d263d0 crc :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:57:06.454 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.453+0000 7fd6d7c00640 1 -- 172.21.15.32:0/2953897175 <== osd.6 v2:172.21.15.118:6825/801947386 1 ==== osd_op_reply(69 604.00000000 [create] v102'182 uv176 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fd6d000b040 con 0x5653c1d25fc0 2024-08-03T20:57:06.454 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.453+0000 7fd6de0eef40 1 -- 172.21.15.32:0/2953897175 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:70 12.15 12:a93a17c2:::604.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e102) -- 0x5653c1d28af0 con 0x5653c1d25fc0 2024-08-03T20:57:06.456 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.454+0000 7fd6d7c00640 1 -- 172.21.15.32:0/2953897175 <== osd.6 v2:172.21.15.118:6825/801947386 2 ==== osd_op_reply(70 604.00000000 [delete] v102'183 uv183 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fd6d000b040 con 0x5653c1d25fc0 2024-08-03T20:57:06.456 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.454+0000 7fd6de0eef40 1 -- 172.21.15.32:0/2953897175 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:71 12.15 12:a93a17c2:::604.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102) -- 0x5653c1d28af0 con 0x5653c1d25fc0 2024-08-03T20:57:06.457 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::604.00000000:head# 2024-08-03T20:57:06.457 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.456+0000 7fd6d7c00640 1 -- 172.21.15.32:0/2953897175 <== osd.6 v2:172.21.15.118:6825/801947386 3 ==== osd_op_reply(71 604.00000000 [create] v102'184 uv184 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fd6d000b040 con 0x5653c1d25fc0 2024-08-03T20:57:06.457 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.456+0000 7fd6de0eef40 1 -- 172.21.15.32:0/2953897175 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:72 12.15 12:a93a17c2:::604.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102) -- 0x5653c1d29b60 con 0x5653c1d25fc0 2024-08-03T20:57:06.458 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.457+0000 7fd6d7c00640 1 -- 172.21.15.32:0/2953897175 <== osd.6 v2:172.21.15.118:6825/801947386 4 ==== osd_op_reply(72 604.00000000 [setxattr (30)] v102'185 uv185 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fd6d000b040 con 0x5653c1d25fc0 2024-08-03T20:57:06.458 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.457+0000 7fd6de0eef40 1 -- 172.21.15.32:0/2953897175 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:73 12.15 12:a93a17c2:::604.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102) -- 0x5653c1d29b20 con 0x5653c1d25fc0 2024-08-03T20:57:06.460 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.458+0000 7fd6d7c00640 1 -- 172.21.15.32:0/2953897175 <== osd.6 v2:172.21.15.118:6825/801947386 5 ==== osd_op_reply(73 604.00000000 [setxattr (62)] v102'186 uv186 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fd6d000b040 con 0x5653c1d25fc0 2024-08-03T20:57:06.460 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.458+0000 7fd6de0eef40 1 -- 172.21.15.32:0/2953897175 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:74 12.15 12:a93a17c2:::604.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102) -- 0x5653c1d29b20 con 0x5653c1d25fc0 2024-08-03T20:57:06.461 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.460+0000 7fd6d7c00640 1 -- 172.21.15.32:0/2953897175 <== osd.6 v2:172.21.15.118:6825/801947386 6 ==== osd_op_reply(74 604.00000000 [omap-set-header] v102'187 uv187 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fd6d000b040 con 0x5653c1d25fc0 2024-08-03T20:57:06.461 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.460+0000 7fd6de0eef40 1 -- 172.21.15.32:0/2953897175 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:75 12.15 12:aa448500:::500.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102) -- 0x5653c1d29b20 con 0x5653c1d25fc0 2024-08-03T20:57:06.462 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.461+0000 7fd6d7c00640 1 -- 172.21.15.32:0/2953897175 <== osd.6 v2:172.21.15.118:6825/801947386 7 ==== osd_op_reply(75 500.00000000 [create] v102'188 uv181 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fd6d000b040 con 0x5653c1d25fc0 2024-08-03T20:57:06.462 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.461+0000 7fd6de0eef40 1 -- 172.21.15.32:0/2953897175 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:76 12.15 12:aa448500:::500.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e102) -- 0x5653c1d29b20 con 0x5653c1d25fc0 2024-08-03T20:57:06.463 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.462+0000 7fd6d7c00640 1 -- 172.21.15.32:0/2953897175 <== osd.6 v2:172.21.15.118:6825/801947386 8 ==== osd_op_reply(76 500.00000000 [delete] v102'189 uv189 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fd6d000b040 con 0x5653c1d25fc0 2024-08-03T20:57:06.464 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.462+0000 7fd6de0eef40 1 -- 172.21.15.32:0/2953897175 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:77 12.15 12:aa448500:::500.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102) -- 0x5653c1d28b40 con 0x5653c1d25fc0 2024-08-03T20:57:06.465 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::500.00000000:head# 2024-08-03T20:57:06.465 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.464+0000 7fd6d7c00640 1 -- 172.21.15.32:0/2953897175 <== osd.6 v2:172.21.15.118:6825/801947386 9 ==== osd_op_reply(77 500.00000000 [create] v102'190 uv190 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fd6d000b040 con 0x5653c1d25fc0 2024-08-03T20:57:06.465 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.464+0000 7fd6de0eef40 1 -- 172.21.15.32:0/2953897175 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:78 12.15 12:aa448500:::500.00000000:head [write 0~90 in=90b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102) -- 0x5653c1d28b40 con 0x5653c1d25fc0 2024-08-03T20:57:06.466 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.465+0000 7fd6d7c00640 1 -- 172.21.15.32:0/2953897175 <== osd.6 v2:172.21.15.118:6825/801947386 10 ==== osd_op_reply(78 500.00000000 [write 0~90] v102'191 uv191 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fd6d000b040 con 0x5653c1d25fc0 2024-08-03T20:57:06.466 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.465+0000 7fd6de0eef40 1 -- 172.21.15.32:0/2953897175 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:79 12.15 12:aa448500:::500.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102) -- 0x5653c1d28af0 con 0x5653c1d25fc0 2024-08-03T20:57:06.468 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.466+0000 7fd6d7c00640 1 -- 172.21.15.32:0/2953897175 <== osd.6 v2:172.21.15.118:6825/801947386 11 ==== osd_op_reply(79 500.00000000 [omap-set-header] v102'192 uv192 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fd6d000b040 con 0x5653c1d25fc0 2024-08-03T20:57:06.468 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.466+0000 7fd6de0eef40 1 --2- 172.21.15.32:0/2953897175 >> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] conn(0x5653c1d29440 0x5653c1d29850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:06.468 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.466+0000 7fd6de0eef40 1 -- 172.21.15.32:0/2953897175 --> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] -- osd_op(unknown.0.0:80 12.d 12:b50e409b:::mds_snaptable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102) -- 0x5653c1d2bf70 con 0x5653c1d29440 2024-08-03T20:57:06.468 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.467+0000 7fd6d7200640 1 --2- 172.21.15.32:0/2953897175 >> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] conn(0x5653c1d29440 0x5653c1d29850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:06.469 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.468+0000 7fd6d7200640 1 --2- 172.21.15.32:0/2953897175 >> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] conn(0x5653c1d29440 0x5653c1d29850 crc :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:57:06.470 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.469+0000 7fd6d7200640 1 -- 172.21.15.32:0/2953897175 <== osd.7 v2:172.21.15.118:6818/653000048 1 ==== osd_op_reply(80 mds_snaptable [create] v102'81 uv80 ondisk = -17 ((17) File exists)) ==== 157+0+0 (crc 0 0 0) 0x7fd6c805f030 con 0x5653c1d29440 2024-08-03T20:57:06.470 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.469+0000 7fd6de0eef40 1 -- 172.21.15.32:0/2953897175 --> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] -- osd_op(unknown.0.0:81 12.d 12:b50e409b:::mds_snaptable:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e102) -- 0x5653c1d2bf70 con 0x5653c1d29440 2024-08-03T20:57:06.471 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.470+0000 7fd6d7200640 1 -- 172.21.15.32:0/2953897175 <== osd.7 v2:172.21.15.118:6818/653000048 2 ==== osd_op_reply(81 mds_snaptable [delete] v102'82 uv82 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7fd6c805f030 con 0x5653c1d29440 2024-08-03T20:57:06.471 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.470+0000 7fd6de0eef40 1 -- 172.21.15.32:0/2953897175 --> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] -- osd_op(unknown.0.0:82 12.d 12:b50e409b:::mds_snaptable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102) -- 0x5653c1d2c360 con 0x5653c1d29440 2024-08-03T20:57:06.473 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds_snaptable:head# 2024-08-03T20:57:06.473 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.471+0000 7fd6d7200640 1 -- 172.21.15.32:0/2953897175 <== osd.7 v2:172.21.15.118:6818/653000048 3 ==== osd_op_reply(82 mds_snaptable [create] v102'83 uv83 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7fd6c805f030 con 0x5653c1d29440 2024-08-03T20:57:06.473 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.471+0000 7fd6de0eef40 1 -- 172.21.15.32:0/2953897175 --> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] -- osd_op(unknown.0.0:83 12.d 12:b50e409b:::mds_snaptable:head [write 0~70 in=70b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102) -- 0x5653c1d2c360 con 0x5653c1d29440 2024-08-03T20:57:06.474 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.473+0000 7fd6d7200640 1 -- 172.21.15.32:0/2953897175 <== osd.7 v2:172.21.15.118:6818/653000048 4 ==== osd_op_reply(83 mds_snaptable [write 0~70] v102'84 uv84 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7fd6c805f030 con 0x5653c1d29440 2024-08-03T20:57:06.474 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.473+0000 7fd6de0eef40 1 -- 172.21.15.32:0/2953897175 --> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] -- osd_op(unknown.0.0:84 12.d 12:b50e409b:::mds_snaptable:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102) -- 0x5653c1d2c360 con 0x5653c1d29440 2024-08-03T20:57:06.475 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.474+0000 7fd6d7200640 1 -- 172.21.15.32:0/2953897175 <== osd.7 v2:172.21.15.118:6818/653000048 5 ==== osd_op_reply(84 mds_snaptable [omap-set-header] v102'85 uv85 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7fd6c805f030 con 0x5653c1d29440 2024-08-03T20:57:06.476 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.474+0000 7fd6de0eef40 1 -- 172.21.15.32:0/2953897175 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:85 12.1d 12:b871830b:::605.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102) -- 0x5653c1d2c360 con 0x5653c1be8580 2024-08-03T20:57:06.477 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.476+0000 7fd6d7c00640 1 -- 172.21.15.32:0/2953897175 <== osd.0 v2:172.21.15.32:6816/3066156116 44 ==== osd_op_reply(85 605.00000000 [create] v102'99 uv98 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fd6d000b040 con 0x5653c1be8580 2024-08-03T20:57:06.477 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.476+0000 7fd6de0eef40 1 -- 172.21.15.32:0/2953897175 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:86 12.1d 12:b871830b:::605.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e102) -- 0x5653c1d2c360 con 0x5653c1be8580 2024-08-03T20:57:06.478 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.477+0000 7fd6d7c00640 1 -- 172.21.15.32:0/2953897175 <== osd.0 v2:172.21.15.32:6816/3066156116 45 ==== osd_op_reply(86 605.00000000 [delete] v102'100 uv100 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fd6d000b040 con 0x5653c1be8580 2024-08-03T20:57:06.478 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.477+0000 7fd6de0eef40 1 -- 172.21.15.32:0/2953897175 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:87 12.1d 12:b871830b:::605.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102) -- 0x5653c1d2c360 con 0x5653c1be8580 2024-08-03T20:57:06.479 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.478+0000 7fd6d7c00640 1 -- 172.21.15.32:0/2953897175 <== osd.0 v2:172.21.15.32:6816/3066156116 46 ==== osd_op_reply(87 605.00000000 [create] v102'101 uv101 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fd6d000b040 con 0x5653c1be8580 2024-08-03T20:57:06.479 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::605.00000000:head# 2024-08-03T20:57:06.480 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.478+0000 7fd6de0eef40 1 -- 172.21.15.32:0/2953897175 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:88 12.1d 12:b871830b:::605.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102) -- 0x5653c1d2d370 con 0x5653c1be8580 2024-08-03T20:57:06.481 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.479+0000 7fd6d7c00640 1 -- 172.21.15.32:0/2953897175 <== osd.0 v2:172.21.15.32:6816/3066156116 47 ==== osd_op_reply(88 605.00000000 [setxattr (30)] v102'102 uv102 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fd6d000b040 con 0x5653c1be8580 2024-08-03T20:57:06.481 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.479+0000 7fd6de0eef40 1 -- 172.21.15.32:0/2953897175 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:89 12.1d 12:b871830b:::605.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102) -- 0x5653c1d2d370 con 0x5653c1be8580 2024-08-03T20:57:06.482 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.481+0000 7fd6d7c00640 1 -- 172.21.15.32:0/2953897175 <== osd.0 v2:172.21.15.32:6816/3066156116 48 ==== osd_op_reply(89 605.00000000 [setxattr (62)] v102'103 uv103 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fd6d000b040 con 0x5653c1be8580 2024-08-03T20:57:06.482 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.481+0000 7fd6de0eef40 1 -- 172.21.15.32:0/2953897175 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:90 12.1d 12:b871830b:::605.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102) -- 0x5653c1d2c360 con 0x5653c1be8580 2024-08-03T20:57:06.483 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.482+0000 7fd6d7c00640 1 -- 172.21.15.32:0/2953897175 <== osd.0 v2:172.21.15.32:6816/3066156116 49 ==== osd_op_reply(90 605.00000000 [omap-set-header] v102'104 uv104 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fd6d000b040 con 0x5653c1be8580 2024-08-03T20:57:06.484 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.482+0000 7fd6de0eef40 1 -- 172.21.15.32:0/2953897175 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:91 12.13 12:c91d4a1d:::mds0_inotable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102) -- 0x5653c1d2c360 con 0x5653c1d21970 2024-08-03T20:57:06.485 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.483+0000 7fd6d6800640 1 -- 172.21.15.32:0/2953897175 <== osd.1 v2:172.21.15.32:6824/1551241759 5 ==== osd_op_reply(91 mds0_inotable [create] v102'162 uv156 ondisk = -17 ((17) File exists)) ==== 157+0+0 (crc 0 0 0) 0x7fd6c405c6d0 con 0x5653c1d21970 2024-08-03T20:57:06.485 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.483+0000 7fd6de0eef40 1 -- 172.21.15.32:0/2953897175 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:92 12.13 12:c91d4a1d:::mds0_inotable:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e102) -- 0x5653c1d2c700 con 0x5653c1d21970 2024-08-03T20:57:06.486 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.484+0000 7fd6d6800640 1 -- 172.21.15.32:0/2953897175 <== osd.1 v2:172.21.15.32:6824/1551241759 6 ==== osd_op_reply(92 mds0_inotable [delete] v102'163 uv163 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7fd6c405c6d0 con 0x5653c1d21970 2024-08-03T20:57:06.486 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.484+0000 7fd6de0eef40 1 -- 172.21.15.32:0/2953897175 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:93 12.13 12:c91d4a1d:::mds0_inotable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102) -- 0x5653c1d2caa0 con 0x5653c1d21970 2024-08-03T20:57:06.487 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds0_inotable:head# 2024-08-03T20:57:06.487 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.486+0000 7fd6d6800640 1 -- 172.21.15.32:0/2953897175 <== osd.1 v2:172.21.15.32:6824/1551241759 7 ==== osd_op_reply(93 mds0_inotable [create] v102'164 uv164 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7fd6c405c6d0 con 0x5653c1d21970 2024-08-03T20:57:06.487 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.486+0000 7fd6de0eef40 1 -- 172.21.15.32:0/2953897175 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:94 12.13 12:c91d4a1d:::mds0_inotable:head [write 0~34 in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102) -- 0x5653c1d2ce40 con 0x5653c1d21970 2024-08-03T20:57:06.488 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.487+0000 7fd6d6800640 1 -- 172.21.15.32:0/2953897175 <== osd.1 v2:172.21.15.32:6824/1551241759 8 ==== osd_op_reply(94 mds0_inotable [write 0~34] v102'165 uv165 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7fd6c405c6d0 con 0x5653c1d21970 2024-08-03T20:57:06.489 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.487+0000 7fd6de0eef40 1 -- 172.21.15.32:0/2953897175 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:95 12.13 12:c91d4a1d:::mds0_inotable:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102) -- 0x5653c1d2ce40 con 0x5653c1d21970 2024-08-03T20:57:06.490 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.488+0000 7fd6d6800640 1 -- 172.21.15.32:0/2953897175 <== osd.1 v2:172.21.15.32:6824/1551241759 9 ==== osd_op_reply(95 mds0_inotable [omap-set-header] v102'166 uv166 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7fd6c405c6d0 con 0x5653c1d21970 2024-08-03T20:57:06.490 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.488+0000 7fd6de0eef40 1 -- 172.21.15.32:0/2953897175 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:96 12.13 12:cd5a64a3:::100.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102) -- 0x5653c1d2ce40 con 0x5653c1d21970 2024-08-03T20:57:06.491 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.489+0000 7fd6d6800640 1 -- 172.21.15.32:0/2953897175 <== osd.1 v2:172.21.15.32:6824/1551241759 10 ==== osd_op_reply(96 100.00000000 [create] v102'167 uv161 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fd6c405c6d0 con 0x5653c1d21970 2024-08-03T20:57:06.491 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.490+0000 7fd6de0eef40 1 -- 172.21.15.32:0/2953897175 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:97 12.13 12:cd5a64a3:::100.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e102) -- 0x5653c1d2ce40 con 0x5653c1d21970 2024-08-03T20:57:06.492 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.491+0000 7fd6d6800640 1 -- 172.21.15.32:0/2953897175 <== osd.1 v2:172.21.15.32:6824/1551241759 11 ==== osd_op_reply(97 100.00000000 [delete] v102'168 uv168 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fd6c405c6d0 con 0x5653c1d21970 2024-08-03T20:57:06.492 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.491+0000 7fd6de0eef40 1 -- 172.21.15.32:0/2953897175 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:98 12.13 12:cd5a64a3:::100.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102) -- 0x5653c1d2ce40 con 0x5653c1d21970 2024-08-03T20:57:06.493 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.492+0000 7fd6d6800640 1 -- 172.21.15.32:0/2953897175 <== osd.1 v2:172.21.15.32:6824/1551241759 12 ==== osd_op_reply(98 100.00000000 [create] v102'169 uv169 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fd6c405c6d0 con 0x5653c1d21970 2024-08-03T20:57:06.494 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::100.00000000:head# 2024-08-03T20:57:06.494 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.492+0000 7fd6de0eef40 1 -- 172.21.15.32:0/2953897175 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:99 12.13 12:cd5a64a3:::100.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102) -- 0x5653c1d2ce40 con 0x5653c1d21970 2024-08-03T20:57:06.495 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.493+0000 7fd6d6800640 1 -- 172.21.15.32:0/2953897175 <== osd.1 v2:172.21.15.32:6824/1551241759 13 ==== osd_op_reply(99 100.00000000 [omap-set-header] v102'170 uv170 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fd6c405c6d0 con 0x5653c1d21970 2024-08-03T20:57:06.495 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.493+0000 7fd6de0eef40 1 -- 172.21.15.32:0/2953897175 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:100 12.13 12:cd5a64a3:::100.00000000:head [omap-set-vals in=5194b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102) -- 0x5653c1d2fc20 con 0x5653c1d21970 2024-08-03T20:57:06.496 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.495+0000 7fd6d6800640 1 -- 172.21.15.32:0/2953897175 <== osd.1 v2:172.21.15.32:6824/1551241759 14 ==== osd_op_reply(100 100.00000000 [omap-set-vals] v102'171 uv171 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fd6c405c6d0 con 0x5653c1d21970 2024-08-03T20:57:06.496 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.495+0000 7fd6de0eef40 1 --2- 172.21.15.32:0/2953897175 >> [v2:172.21.15.32:6800/9088303,v1:172.21.15.32:6801/9088303] conn(0x5653c1d2d900 0x5653c1d302c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:06.496 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.495+0000 7fd6de0eef40 1 -- 172.21.15.32:0/2953897175 --> [v2:172.21.15.32:6800/9088303,v1:172.21.15.32:6801/9088303] -- osd_op(unknown.0.0:101 12.b 12:d0630e4c:::mds0_sessionmap:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102) -- 0x5653c1d2fd20 con 0x5653c1d2d900 2024-08-03T20:57:06.497 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.495+0000 7fd6d6800640 1 --2- 172.21.15.32:0/2953897175 >> [v2:172.21.15.32:6800/9088303,v1:172.21.15.32:6801/9088303] conn(0x5653c1d2d900 0x5653c1d302c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:06.497 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.496+0000 7fd6d6800640 1 --2- 172.21.15.32:0/2953897175 >> [v2:172.21.15.32:6800/9088303,v1:172.21.15.32:6801/9088303] conn(0x5653c1d2d900 0x5653c1d302c0 crc :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:57:06.498 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.497+0000 7fd6d6800640 1 -- 172.21.15.32:0/2953897175 <== osd.3 v2:172.21.15.32:6800/9088303 1 ==== osd_op_reply(101 mds0_sessionmap [create] v102'74 uv73 ondisk = -17 ((17) File exists)) ==== 159+0+0 (crc 0 0 0) 0x7fd6c405c6d0 con 0x5653c1d2d900 2024-08-03T20:57:06.499 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.497+0000 7fd6de0eef40 1 -- 172.21.15.32:0/2953897175 --> [v2:172.21.15.32:6800/9088303,v1:172.21.15.32:6801/9088303] -- osd_op(unknown.0.0:102 12.b 12:d0630e4c:::mds0_sessionmap:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e102) -- 0x5653c1d2fd20 con 0x5653c1d2d900 2024-08-03T20:57:06.500 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.498+0000 7fd6d6800640 1 -- 172.21.15.32:0/2953897175 <== osd.3 v2:172.21.15.32:6800/9088303 2 ==== osd_op_reply(102 mds0_sessionmap [delete] v102'75 uv75 ondisk = 0) ==== 159+0+0 (crc 0 0 0) 0x7fd6c405c6d0 con 0x5653c1d2d900 2024-08-03T20:57:06.500 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.499+0000 7fd6de0eef40 1 -- 172.21.15.32:0/2953897175 --> [v2:172.21.15.32:6800/9088303,v1:172.21.15.32:6801/9088303] -- osd_op(unknown.0.0:103 12.b 12:d0630e4c:::mds0_sessionmap:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102) -- 0x5653c1d2fd20 con 0x5653c1d2d900 2024-08-03T20:57:06.501 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.500+0000 7fd6d6800640 1 -- 172.21.15.32:0/2953897175 <== osd.3 v2:172.21.15.32:6800/9088303 3 ==== osd_op_reply(103 mds0_sessionmap [create] v102'76 uv76 ondisk = 0) ==== 159+0+0 (crc 0 0 0) 0x7fd6c405c6d0 con 0x5653c1d2d900 2024-08-03T20:57:06.501 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds0_sessionmap:head# 2024-08-03T20:57:06.501 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.500+0000 7fd6de0eef40 1 -- 172.21.15.32:0/2953897175 --> [v2:172.21.15.32:6800/9088303,v1:172.21.15.32:6801/9088303] -- osd_op(unknown.0.0:104 12.b 12:d0630e4c:::mds0_sessionmap:head [omap-set-header in=14b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102) -- 0x5653c1d2fd20 con 0x5653c1d2d900 2024-08-03T20:57:06.502 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.501+0000 7fd6d6800640 1 -- 172.21.15.32:0/2953897175 <== osd.3 v2:172.21.15.32:6800/9088303 4 ==== osd_op_reply(104 mds0_sessionmap [omap-set-header] v102'77 uv77 ondisk = 0) ==== 159+0+0 (crc 0 0 0) 0x7fd6c405c6d0 con 0x5653c1d2d900 2024-08-03T20:57:06.503 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.501+0000 7fd6de0eef40 1 -- 172.21.15.32:0/2953897175 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:105 12.7 12:e0b41d06:::609.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102) -- 0x5653c1d2fd20 con 0x5653c1d21970 2024-08-03T20:57:06.504 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.503+0000 7fd6d6800640 1 -- 172.21.15.32:0/2953897175 <== osd.1 v2:172.21.15.32:6824/1551241759 15 ==== osd_op_reply(105 609.00000000 [create] v102'99 uv98 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fd6c405c6d0 con 0x5653c1d21970 2024-08-03T20:57:06.504 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.503+0000 7fd6de0eef40 1 -- 172.21.15.32:0/2953897175 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:106 12.7 12:e0b41d06:::609.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e102) -- 0x5653c1d2fd20 con 0x5653c1d21970 2024-08-03T20:57:06.506 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.505+0000 7fd6d6800640 1 -- 172.21.15.32:0/2953897175 <== osd.1 v2:172.21.15.32:6824/1551241759 16 ==== osd_op_reply(106 609.00000000 [delete] v102'100 uv100 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fd6c405c6d0 con 0x5653c1d21970 2024-08-03T20:57:06.506 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.505+0000 7fd6de0eef40 1 -- 172.21.15.32:0/2953897175 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:107 12.7 12:e0b41d06:::609.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102) -- 0x5653c1d2fd20 con 0x5653c1d21970 2024-08-03T20:57:06.507 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.506+0000 7fd6d6800640 1 -- 172.21.15.32:0/2953897175 <== osd.1 v2:172.21.15.32:6824/1551241759 17 ==== osd_op_reply(107 609.00000000 [create] v102'101 uv101 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fd6c405c6d0 con 0x5653c1d21970 2024-08-03T20:57:06.507 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::609.00000000:head# 2024-08-03T20:57:06.508 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.506+0000 7fd6de0eef40 1 -- 172.21.15.32:0/2953897175 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:108 12.7 12:e0b41d06:::609.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102) -- 0x5653c1d2fd20 con 0x5653c1d21970 2024-08-03T20:57:06.508 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.507+0000 7fd6d6800640 1 -- 172.21.15.32:0/2953897175 <== osd.1 v2:172.21.15.32:6824/1551241759 18 ==== osd_op_reply(108 609.00000000 [setxattr (30)] v102'102 uv102 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fd6c405c6d0 con 0x5653c1d21970 2024-08-03T20:57:06.509 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.507+0000 7fd6de0eef40 1 -- 172.21.15.32:0/2953897175 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:109 12.7 12:e0b41d06:::609.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102) -- 0x5653c1d2fd20 con 0x5653c1d21970 2024-08-03T20:57:06.510 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.509+0000 7fd6d6800640 1 -- 172.21.15.32:0/2953897175 <== osd.1 v2:172.21.15.32:6824/1551241759 19 ==== osd_op_reply(109 609.00000000 [setxattr (62)] v102'103 uv103 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fd6c405c6d0 con 0x5653c1d21970 2024-08-03T20:57:06.510 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.509+0000 7fd6de0eef40 1 -- 172.21.15.32:0/2953897175 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:110 12.7 12:e0b41d06:::609.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102) -- 0x5653c1d2fd20 con 0x5653c1d21970 2024-08-03T20:57:06.511 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.510+0000 7fd6d6800640 1 -- 172.21.15.32:0/2953897175 <== osd.1 v2:172.21.15.32:6824/1551241759 20 ==== osd_op_reply(110 609.00000000 [omap-set-header] v102'104 uv104 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fd6c405c6d0 con 0x5653c1d21970 2024-08-03T20:57:06.511 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.510+0000 7fd6de0eef40 1 -- 172.21.15.32:0/2953897175 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:111 12.f 12:f1f69726:::400.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102) -- 0x5653c1d2fd20 con 0x5653c1d25fc0 2024-08-03T20:57:06.513 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.511+0000 7fd6d7c00640 1 -- 172.21.15.32:0/2953897175 <== osd.6 v2:172.21.15.118:6825/801947386 12 ==== osd_op_reply(111 400.00000000 [create] v102'84 uv83 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fd6d000b040 con 0x5653c1d25fc0 2024-08-03T20:57:06.513 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.512+0000 7fd6de0eef40 1 -- 172.21.15.32:0/2953897175 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:112 12.f 12:f1f69726:::400.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e102) -- 0x5653c1d2fd20 con 0x5653c1d25fc0 2024-08-03T20:57:06.514 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.513+0000 7fd6d7c00640 1 -- 172.21.15.32:0/2953897175 <== osd.6 v2:172.21.15.118:6825/801947386 13 ==== osd_op_reply(112 400.00000000 [delete] v102'85 uv85 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fd6d000b040 con 0x5653c1d25fc0 2024-08-03T20:57:06.514 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.513+0000 7fd6de0eef40 1 -- 172.21.15.32:0/2953897175 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:113 12.f 12:f1f69726:::400.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102) -- 0x5653c1d2fd20 con 0x5653c1d25fc0 2024-08-03T20:57:06.516 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.515+0000 7fd6d7c00640 1 -- 172.21.15.32:0/2953897175 <== osd.6 v2:172.21.15.118:6825/801947386 14 ==== osd_op_reply(113 400.00000000 [create] v102'86 uv86 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fd6d000b040 con 0x5653c1d25fc0 2024-08-03T20:57:06.516 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::400.00000000:head# 2024-08-03T20:57:06.516 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.515+0000 7fd6de0eef40 1 -- 172.21.15.32:0/2953897175 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:114 12.f 12:f1f69726:::400.00000000:head [write 0~22 in=22b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102) -- 0x5653c1d2fd20 con 0x5653c1d25fc0 2024-08-03T20:57:06.517 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.516+0000 7fd6d7c00640 1 -- 172.21.15.32:0/2953897175 <== osd.6 v2:172.21.15.118:6825/801947386 15 ==== osd_op_reply(114 400.00000000 [write 0~22] v102'87 uv87 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fd6d000b040 con 0x5653c1d25fc0 2024-08-03T20:57:06.517 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.516+0000 7fd6de0eef40 1 -- 172.21.15.32:0/2953897175 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:115 12.f 12:f1f69726:::400.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102) -- 0x5653c1d2fd20 con 0x5653c1d25fc0 2024-08-03T20:57:06.519 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.517+0000 7fd6d7c00640 1 -- 172.21.15.32:0/2953897175 <== osd.6 v2:172.21.15.118:6825/801947386 16 ==== osd_op_reply(115 400.00000000 [omap-set-header] v102'88 uv88 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fd6d000b040 con 0x5653c1d25fc0 2024-08-03T20:57:06.519 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.517+0000 7fd6de0eef40 1 -- 172.21.15.32:0/2953897175 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:116 12.1f 12:fe07bba1:::100.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102) -- 0x5653c1d2fd20 con 0x5653c1d25fc0 2024-08-03T20:57:06.520 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.519+0000 7fd6d7c00640 1 -- 172.21.15.32:0/2953897175 <== osd.6 v2:172.21.15.118:6825/801947386 17 ==== osd_op_reply(116 100.00000000.inode [create] v102'204 uv196 ondisk = -17 ((17) File exists)) ==== 162+0+0 (crc 0 0 0) 0x7fd6d000b040 con 0x5653c1d25fc0 2024-08-03T20:57:06.520 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.519+0000 7fd6de0eef40 1 -- 172.21.15.32:0/2953897175 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:117 12.1f 12:fe07bba1:::100.00000000.inode:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e102) -- 0x5653c1d2fd20 con 0x5653c1d25fc0 2024-08-03T20:57:06.522 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.521+0000 7fd6d7c00640 1 -- 172.21.15.32:0/2953897175 <== osd.6 v2:172.21.15.118:6825/801947386 18 ==== osd_op_reply(117 100.00000000.inode [delete] v102'205 uv205 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7fd6d000b040 con 0x5653c1d25fc0 2024-08-03T20:57:06.522 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.521+0000 7fd6de0eef40 1 -- 172.21.15.32:0/2953897175 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:118 12.1f 12:fe07bba1:::100.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102) -- 0x5653c1d2fd20 con 0x5653c1d25fc0 2024-08-03T20:57:06.523 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::100.00000000.inode:head# 2024-08-03T20:57:06.524 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.522+0000 7fd6d7c00640 1 -- 172.21.15.32:0/2953897175 <== osd.6 v2:172.21.15.118:6825/801947386 19 ==== osd_op_reply(118 100.00000000.inode [create] v102'206 uv206 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7fd6d000b040 con 0x5653c1d25fc0 2024-08-03T20:57:06.524 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.522+0000 7fd6de0eef40 1 -- 172.21.15.32:0/2953897175 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:119 12.1f 12:fe07bba1:::100.00000000.inode:head [write 0~582 in=582b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102) -- 0x5653c1d2fd20 con 0x5653c1d25fc0 2024-08-03T20:57:06.525 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.524+0000 7fd6d7c00640 1 -- 172.21.15.32:0/2953897175 <== osd.6 v2:172.21.15.118:6825/801947386 20 ==== osd_op_reply(119 100.00000000.inode [write 0~582] v102'207 uv207 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7fd6d000b040 con 0x5653c1d25fc0 2024-08-03T20:57:06.525 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.524+0000 7fd6de0eef40 1 -- 172.21.15.32:0/2953897175 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:120 12.1f 12:fe07bba1:::100.00000000.inode:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102) -- 0x5653c1d2fd20 con 0x5653c1d25fc0 2024-08-03T20:57:06.526 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.525+0000 7fd6d7c00640 1 -- 172.21.15.32:0/2953897175 <== osd.6 v2:172.21.15.118:6825/801947386 21 ==== osd_op_reply(120 100.00000000.inode [omap-set-header] v102'208 uv208 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7fd6d000b040 con 0x5653c1d25fc0 2024-08-03T20:57:06.526 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.525+0000 7fd6de0eef40 1 -- 172.21.15.32:0/2953897175 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:121 12.1f 12:ff5b34d6:::1.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102) -- 0x5653c1d2fd20 con 0x5653c1d25fc0 2024-08-03T20:57:06.528 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.526+0000 7fd6d7c00640 1 -- 172.21.15.32:0/2953897175 <== osd.6 v2:172.21.15.118:6825/801947386 22 ==== osd_op_reply(121 1.00000000 [create] v102'209 uv203 ondisk = -17 ((17) File exists)) ==== 154+0+0 (crc 0 0 0) 0x7fd6d000b040 con 0x5653c1d25fc0 2024-08-03T20:57:06.528 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.526+0000 7fd6de0eef40 1 -- 172.21.15.32:0/2953897175 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:122 12.1f 12:ff5b34d6:::1.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e102) -- 0x5653c1d2fd20 con 0x5653c1d25fc0 2024-08-03T20:57:06.529 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.528+0000 7fd6d7c00640 1 -- 172.21.15.32:0/2953897175 <== osd.6 v2:172.21.15.118:6825/801947386 23 ==== osd_op_reply(122 1.00000000 [delete] v102'210 uv210 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7fd6d000b040 con 0x5653c1d25fc0 2024-08-03T20:57:06.529 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.528+0000 7fd6de0eef40 1 -- 172.21.15.32:0/2953897175 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:123 12.1f 12:ff5b34d6:::1.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102) -- 0x5653c1d2fd20 con 0x5653c1d25fc0 2024-08-03T20:57:06.531 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::1.00000000:head# 2024-08-03T20:57:06.531 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.529+0000 7fd6d7c00640 1 -- 172.21.15.32:0/2953897175 <== osd.6 v2:172.21.15.118:6825/801947386 24 ==== osd_op_reply(123 1.00000000 [create] v102'211 uv211 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7fd6d000b040 con 0x5653c1d25fc0 2024-08-03T20:57:06.531 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.529+0000 7fd6de0eef40 1 -- 172.21.15.32:0/2953897175 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:124 12.1f 12:ff5b34d6:::1.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102) -- 0x5653c1d31860 con 0x5653c1d25fc0 2024-08-03T20:57:06.532 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.531+0000 7fd6d7c00640 1 -- 172.21.15.32:0/2953897175 <== osd.6 v2:172.21.15.118:6825/801947386 25 ==== osd_op_reply(124 1.00000000 [setxattr (30)] v102'212 uv212 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7fd6d000b040 con 0x5653c1d25fc0 2024-08-03T20:57:06.532 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.531+0000 7fd6de0eef40 1 -- 172.21.15.32:0/2953897175 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:125 12.1f 12:ff5b34d6:::1.00000000:head [setxattr parent (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102) -- 0x5653c1d31860 con 0x5653c1d25fc0 2024-08-03T20:57:06.534 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.532+0000 7fd6d7c00640 1 -- 172.21.15.32:0/2953897175 <== osd.6 v2:172.21.15.118:6825/801947386 26 ==== osd_op_reply(125 1.00000000 [setxattr (30)] v102'213 uv213 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7fd6d000b040 con 0x5653c1d25fc0 2024-08-03T20:57:06.534 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.533+0000 7fd6de0eef40 1 -- 172.21.15.32:0/2953897175 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:126 12.1f 12:ff5b34d6:::1.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102) -- 0x5653c1d30800 con 0x5653c1d25fc0 2024-08-03T20:57:06.535 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.534+0000 7fd6d7c00640 1 -- 172.21.15.32:0/2953897175 <== osd.6 v2:172.21.15.118:6825/801947386 27 ==== osd_op_reply(126 1.00000000 [omap-set-header] v102'214 uv214 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7fd6d000b040 con 0x5653c1d25fc0 2024-08-03T20:57:06.535 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.534+0000 7fd6de0eef40 1 -- 172.21.15.32:0/2953897175 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:127 12.1f 12:ff5b34d6:::1.00000000:head [omap-set-vals in=584b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102) -- 0x5653c1d30cb0 con 0x5653c1d25fc0 2024-08-03T20:57:06.538 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.535+0000 7fd6d7c00640 1 -- 172.21.15.32:0/2953897175 <== osd.6 v2:172.21.15.118:6825/801947386 28 ==== osd_op_reply(127 1.00000000 [omap-set-vals] v102'215 uv215 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7fd6d000b040 con 0x5653c1d25fc0 2024-08-03T20:57:06.538 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.536+0000 7fd6de0eef40 1 -- 172.21.15.32:0/2953897175 >> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] conn(0x5653c1be8580 msgr2=0x5653c1d1b740 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:06.538 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.536+0000 7fd6de0eef40 1 --2- 172.21.15.32:0/2953897175 >> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] conn(0x5653c1be8580 0x5653c1d1b740 crc :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:06.538 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.536+0000 7fd6de0eef40 1 -- 172.21.15.32:0/2953897175 >> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] conn(0x5653c1d21970 msgr2=0x5653c1d23d90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:06.538 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.536+0000 7fd6de0eef40 1 --2- 172.21.15.32:0/2953897175 >> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] conn(0x5653c1d21970 0x5653c1d23d90 crc :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:06.538 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.536+0000 7fd6de0eef40 1 -- 172.21.15.32:0/2953897175 >> [v2:172.21.15.32:6800/9088303,v1:172.21.15.32:6801/9088303] conn(0x5653c1d2d900 msgr2=0x5653c1d302c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:06.539 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.536+0000 7fd6de0eef40 1 --2- 172.21.15.32:0/2953897175 >> [v2:172.21.15.32:6800/9088303,v1:172.21.15.32:6801/9088303] conn(0x5653c1d2d900 0x5653c1d302c0 crc :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:06.539 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.536+0000 7fd6de0eef40 1 -- 172.21.15.32:0/2953897175 >> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] conn(0x5653c1d1d770 msgr2=0x5653c1d1fb90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:06.539 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.536+0000 7fd6de0eef40 1 --2- 172.21.15.32:0/2953897175 >> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] conn(0x5653c1d1d770 0x5653c1d1fb90 crc :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:06.539 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.536+0000 7fd6de0eef40 1 -- 172.21.15.32:0/2953897175 >> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] conn(0x5653c1d25fc0 msgr2=0x5653c1d263d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:06.539 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.536+0000 7fd6de0eef40 1 --2- 172.21.15.32:0/2953897175 >> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] conn(0x5653c1d25fc0 0x5653c1d263d0 crc :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:06.539 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.536+0000 7fd6de0eef40 1 -- 172.21.15.32:0/2953897175 >> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] conn(0x5653c1d29440 msgr2=0x5653c1d29850 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:06.539 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.536+0000 7fd6de0eef40 1 --2- 172.21.15.32:0/2953897175 >> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] conn(0x5653c1d29440 0x5653c1d29850 crc :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:06.539 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.536+0000 7fd6d7c00640 1 -- 172.21.15.32:0/2953897175 reap_dead start 2024-08-03T20:57:06.539 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.536+0000 7fd6d7c00640 1 -- 172.21.15.32:0/2953897175 reap_dead start 2024-08-03T20:57:06.539 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.536+0000 7fd6d7c00640 1 -- 172.21.15.32:0/2953897175 reap_dead start 2024-08-03T20:57:06.539 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.536+0000 7fd6de0eef40 1 -- 172.21.15.32:0/2953897175 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fd6b80759f0 msgr2=0x7fd6b8077f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:06.539 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.536+0000 7fd6de0eef40 1 --2- 172.21.15.32:0/2953897175 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fd6b80759f0 0x7fd6b8077f00 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7fd6c8000c00 tx=0x7fd6c8004880 comp rx=0 tx=0).stop 2024-08-03T20:57:06.539 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.536+0000 7fd6de0eef40 1 -- 172.21.15.32:0/2953897175 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5653c1c76cb0 msgr2=0x5653c1be59d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:06.539 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.536+0000 7fd6de0eef40 1 --2- 172.21.15.32:0/2953897175 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5653c1c76cb0 0x5653c1be59d0 secure :-1 s=READY pgs=1486 cs=0 l=1 rev1=1 crypto rx=0x7fd6c4001e30 tx=0x7fd6c400a720 comp rx=0 tx=0).stop 2024-08-03T20:57:06.540 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.536+0000 7fd6de0eef40 1 -- 172.21.15.32:0/2953897175 shutdown_connections 2024-08-03T20:57:06.540 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.536+0000 7fd6de0eef40 1 --2- 172.21.15.32:0/2953897175 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fd6b80759f0 0x7fd6b8077f00 unknown :-1 s=CLOSED pgs=607 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:06.540 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.536+0000 7fd6de0eef40 1 --2- 172.21.15.32:0/2953897175 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5653c1c76cb0 0x5653c1be59d0 unknown :-1 s=CLOSED pgs=1486 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:06.540 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.536+0000 7fd6de0eef40 1 -- 172.21.15.32:0/2953897175 >> 172.21.15.32:0/2953897175 conn(0x5653c1ac2ac0 msgr2=0x5653c1be11a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:57:06.540 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.536+0000 7fd6de0eef40 1 -- 172.21.15.32:0/2953897175 shutdown_connections 2024-08-03T20:57:06.540 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.536+0000 7fd6de0eef40 1 -- 172.21.15.32:0/2953897175 wait complete. 2024-08-03T20:57:06.552 DEBUG:teuthology.orchestra.run.smithi032:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata put 200.00000001 - 2024-08-03T20:57:06.735 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.734+0000 7fc43bd2ff40 1 Processor -- start 2024-08-03T20:57:06.736 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.734+0000 7fc43bd2ff40 1 -- start start 2024-08-03T20:57:06.737 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.735+0000 7fc43bd2ff40 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55cfa6062360 0x55cfa5ff2cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:06.737 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.735+0000 7fc43bd2ff40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55cfa6063ea0 0x55cfa607e220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:06.737 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.735+0000 7fc43bd2ff40 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55cfa607fcb0 0x55cfa607e760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:06.737 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.735+0000 7fc43bd2ff40 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55cfa5ef9e30 con 0x55cfa6062360 2024-08-03T20:57:06.737 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.735+0000 7fc43bd2ff40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x55cfa5f57bf0 con 0x55cfa6063ea0 2024-08-03T20:57:06.737 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.735+0000 7fc43bd2ff40 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x55cfa60631b0 con 0x55cfa607fcb0 2024-08-03T20:57:06.737 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.736+0000 7fc434400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55cfa6062360 0x55cfa5ff2cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:06.738 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.736+0000 7fc434400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55cfa6062360 0x55cfa5ff2cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:47066/0 (socket says 172.21.15.32:47066) 2024-08-03T20:57:06.738 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.736+0000 7fc434400640 1 -- 172.21.15.32:0/2101419182 learned_addr learned my addr 172.21.15.32:0/2101419182 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:57:06.739 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.736+0000 7fc433a00640 1 --2- 172.21.15.32:0/2101419182 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55cfa6063ea0 0x55cfa607e220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:06.739 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.736+0000 7fc434e00640 1 --2- 172.21.15.32:0/2101419182 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55cfa607fcb0 0x55cfa607e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:06.739 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.736+0000 7fc434400640 1 -- 172.21.15.32:0/2101419182 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55cfa607fcb0 msgr2=0x55cfa607e760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:57:06.740 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.736+0000 7fc434400640 1 --2- 172.21.15.32:0/2101419182 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55cfa607fcb0 0x55cfa607e760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:06.740 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.736+0000 7fc434400640 1 -- 172.21.15.32:0/2101419182 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55cfa6063ea0 msgr2=0x55cfa607e220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:57:06.740 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.736+0000 7fc434400640 1 --2- 172.21.15.32:0/2101419182 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55cfa6063ea0 0x55cfa607e220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:06.740 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.736+0000 7fc434400640 1 -- 172.21.15.32:0/2101419182 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cfa5ff3200 con 0x55cfa6062360 2024-08-03T20:57:06.740 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.736+0000 7fc434400640 1 --2- 172.21.15.32:0/2101419182 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55cfa6062360 0x55cfa5ff2cc0 secure :-1 s=READY pgs=1489 cs=0 l=1 rev1=1 crypto rx=0x7fc42400c550 tx=0x7fc42400c950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=33538fd798c392bd server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:57:06.740 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.737+0000 7fc432600640 1 -- 172.21.15.32:0/2101419182 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc42400a960 con 0x55cfa6062360 2024-08-03T20:57:06.740 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.737+0000 7fc432600640 1 -- 172.21.15.32:0/2101419182 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fc42400ab00 con 0x55cfa6062360 2024-08-03T20:57:06.740 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.737+0000 7fc432600640 1 -- 172.21.15.32:0/2101419182 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc424015440 con 0x55cfa6062360 2024-08-03T20:57:06.740 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.737+0000 7fc43bd2ff40 1 -- 172.21.15.32:0/2101419182 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55cfa6062360 msgr2=0x55cfa5ff2cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:06.740 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.737+0000 7fc43bd2ff40 1 --2- 172.21.15.32:0/2101419182 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55cfa6062360 0x55cfa5ff2cc0 secure :-1 s=READY pgs=1489 cs=0 l=1 rev1=1 crypto rx=0x7fc42400c550 tx=0x7fc42400c950 comp rx=0 tx=0).stop 2024-08-03T20:57:06.740 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.737+0000 7fc43bd2ff40 1 -- 172.21.15.32:0/2101419182 shutdown_connections 2024-08-03T20:57:06.740 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.737+0000 7fc43bd2ff40 1 --2- 172.21.15.32:0/2101419182 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55cfa6063ea0 0x55cfa607e220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:06.741 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.737+0000 7fc43bd2ff40 1 --2- 172.21.15.32:0/2101419182 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55cfa607fcb0 0x55cfa607e760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:06.741 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.737+0000 7fc43bd2ff40 1 --2- 172.21.15.32:0/2101419182 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55cfa6062360 0x55cfa5ff2cc0 unknown :-1 s=CLOSED pgs=1489 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:06.741 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.737+0000 7fc43bd2ff40 1 -- 172.21.15.32:0/2101419182 >> 172.21.15.32:0/2101419182 conn(0x55cfa5ecbac0 msgr2=0x55cfa607be60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:57:06.741 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.737+0000 7fc43bd2ff40 1 -- 172.21.15.32:0/2101419182 shutdown_connections 2024-08-03T20:57:06.741 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.737+0000 7fc43bd2ff40 1 -- 172.21.15.32:0/2101419182 wait complete. 2024-08-03T20:57:06.743 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.738+0000 7fc43bd2ff40 1 Processor -- start 2024-08-03T20:57:06.743 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.739+0000 7fc43bd2ff40 1 -- start start 2024-08-03T20:57:06.743 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.740+0000 7fc43bd2ff40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55cfa6062360 0x55cfa6060e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:06.743 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.740+0000 7fc43bd2ff40 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55cfa6063ea0 0x55cfa6061360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:06.743 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.740+0000 7fc43bd2ff40 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55cfa607fcb0 0x55cfa605c6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:06.743 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.740+0000 7fc43bd2ff40 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55cfa5ff33d0 con 0x55cfa607fcb0 2024-08-03T20:57:06.743 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.740+0000 7fc43bd2ff40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x55cfa607bb10 con 0x55cfa6062360 2024-08-03T20:57:06.743 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.740+0000 7fc43bd2ff40 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x55cfa5ff4fa0 con 0x55cfa6063ea0 2024-08-03T20:57:06.744 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.740+0000 7fc434e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55cfa607fcb0 0x55cfa605c6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:06.744 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.740+0000 7fc434e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55cfa607fcb0 0x55cfa605c6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:47068/0 (socket says 172.21.15.32:47068) 2024-08-03T20:57:06.744 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.740+0000 7fc434e00640 1 -- 172.21.15.32:0/1177467939 learned_addr learned my addr 172.21.15.32:0/1177467939 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:57:06.744 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.740+0000 7fc434400640 1 --2- 172.21.15.32:0/1177467939 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55cfa6062360 0x55cfa6060e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:06.744 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.740+0000 7fc433a00640 1 --2- 172.21.15.32:0/1177467939 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55cfa6063ea0 0x55cfa6061360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:06.744 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.741+0000 7fc434e00640 1 -- 172.21.15.32:0/1177467939 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55cfa6063ea0 msgr2=0x55cfa6061360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:06.744 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.741+0000 7fc434e00640 1 --2- 172.21.15.32:0/1177467939 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55cfa6063ea0 0x55cfa6061360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:06.744 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.741+0000 7fc434e00640 1 -- 172.21.15.32:0/1177467939 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55cfa6062360 msgr2=0x55cfa6060e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:06.744 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.741+0000 7fc434e00640 1 --2- 172.21.15.32:0/1177467939 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55cfa6062360 0x55cfa6060e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:06.744 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.741+0000 7fc434e00640 1 -- 172.21.15.32:0/1177467939 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cfa5fecab0 con 0x55cfa607fcb0 2024-08-03T20:57:06.744 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.741+0000 7fc434e00640 1 --2- 172.21.15.32:0/1177467939 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55cfa607fcb0 0x55cfa605c6e0 secure :-1 s=READY pgs=1490 cs=0 l=1 rev1=1 crypto rx=0x7fc42c006f50 tx=0x7fc42c003220 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:57:06.744 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.741+0000 7fc431200640 1 -- 172.21.15.32:0/1177467939 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc42c013480 con 0x55cfa607fcb0 2024-08-03T20:57:06.744 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.741+0000 7fc431200640 1 -- 172.21.15.32:0/1177467939 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fc42c018070 con 0x55cfa607fcb0 2024-08-03T20:57:06.745 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.741+0000 7fc43bd2ff40 1 -- 172.21.15.32:0/1177467939 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x55cfa5fecd90 con 0x55cfa607fcb0 2024-08-03T20:57:06.745 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.741+0000 7fc431200640 1 -- 172.21.15.32:0/1177467939 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc42c0139b0 con 0x55cfa607fcb0 2024-08-03T20:57:06.745 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.742+0000 7fc43bd2ff40 1 -- 172.21.15.32:0/1177467939 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cfa5feaad0 con 0x55cfa607fcb0 2024-08-03T20:57:06.745 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.744+0000 7fc431200640 1 -- 172.21.15.32:0/1177467939 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7fc42c013b50 con 0x55cfa607fcb0 2024-08-03T20:57:06.746 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.745+0000 7fc431200640 1 --2- 172.21.15.32:0/1177467939 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fc4140759f0 0x7fc414077f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:06.746 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.745+0000 7fc431200640 1 -- 172.21.15.32:0/1177467939 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(102..102 src has 1..102) ==== 7316+0+0 (secure 0 0 0) 0x7fc42c097860 con 0x55cfa607fcb0 2024-08-03T20:57:06.746 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.745+0000 7fc434400640 1 --2- 172.21.15.32:0/1177467939 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fc4140759f0 0x7fc414077f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:06.747 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.745+0000 7fc43bd2ff40 1 --2- 172.21.15.32:0/1177467939 >> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] conn(0x55cfa5ff3ca0 0x55cfa605fcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:06.747 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.745+0000 7fc43bd2ff40 1 -- 172.21.15.32:0/1177467939 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:1 12.14 12:2e2fa760:::200.00000001:head [writefull 0~80 in=80b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102) -- 0x55cfa6060210 con 0x55cfa5ff3ca0 2024-08-03T20:57:06.747 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.746+0000 7fc433a00640 1 --2- 172.21.15.32:0/1177467939 >> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] conn(0x55cfa5ff3ca0 0x55cfa605fcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:06.747 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.746+0000 7fc434400640 1 --2- 172.21.15.32:0/1177467939 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fc4140759f0 0x7fc414077f00 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7fc42400f040 tx=0x7fc424017040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:57:06.748 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.746+0000 7fc433a00640 1 --2- 172.21.15.32:0/1177467939 >> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] conn(0x55cfa5ff3ca0 0x55cfa605fcd0 crc :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:57:06.750 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.748+0000 7fc433a00640 1 -- 172.21.15.32:0/1177467939 <== osd.5 v2:172.21.15.118:6810/2724891770 1 ==== osd_op_reply(1 200.00000001 [writefull 0~80] v102'395 uv395 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fc42004dab0 con 0x55cfa5ff3ca0 2024-08-03T20:57:06.750 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.748+0000 7fc43bd2ff40 1 -- 172.21.15.32:0/1177467939 >> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] conn(0x55cfa5ff3ca0 msgr2=0x55cfa605fcd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:06.750 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.748+0000 7fc43bd2ff40 1 --2- 172.21.15.32:0/1177467939 >> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] conn(0x55cfa5ff3ca0 0x55cfa605fcd0 crc :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:06.750 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.748+0000 7fc43bd2ff40 1 -- 172.21.15.32:0/1177467939 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fc4140759f0 msgr2=0x7fc414077f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:06.750 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.748+0000 7fc43bd2ff40 1 --2- 172.21.15.32:0/1177467939 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fc4140759f0 0x7fc414077f00 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7fc42400f040 tx=0x7fc424017040 comp rx=0 tx=0).stop 2024-08-03T20:57:06.750 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.748+0000 7fc43bd2ff40 1 -- 172.21.15.32:0/1177467939 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55cfa607fcb0 msgr2=0x55cfa605c6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:06.751 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.748+0000 7fc43bd2ff40 1 --2- 172.21.15.32:0/1177467939 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55cfa607fcb0 0x55cfa605c6e0 secure :-1 s=READY pgs=1490 cs=0 l=1 rev1=1 crypto rx=0x7fc42c006f50 tx=0x7fc42c003220 comp rx=0 tx=0).stop 2024-08-03T20:57:06.751 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.748+0000 7fc434e00640 1 -- 172.21.15.32:0/1177467939 reap_dead start 2024-08-03T20:57:06.751 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.748+0000 7fc43bd2ff40 1 -- 172.21.15.32:0/1177467939 shutdown_connections 2024-08-03T20:57:06.751 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.748+0000 7fc43bd2ff40 1 -- 172.21.15.32:0/1177467939 >> 172.21.15.32:0/1177467939 conn(0x55cfa5ecbac0 msgr2=0x55cfa60814d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:57:06.751 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.748+0000 7fc43bd2ff40 1 -- 172.21.15.32:0/1177467939 shutdown_connections 2024-08-03T20:57:06.751 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.748+0000 7fc43bd2ff40 1 -- 172.21.15.32:0/1177467939 wait complete. 2024-08-03T20:57:06.764 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs joinable true 2024-08-03T20:57:06.902 INFO:tasks.ceph.mds.a.smithi032.stderr: -35> 2024-08-03T20:57:02.148+0000 7fe9b1400640 -1 log_channel(cluster) log [ERR] : Error recovering journal 0x200: (22) Invalid argument 2024-08-03T20:57:06.998 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.996+0000 7f6b86200640 1 -- 172.21.15.32:0/4015011595 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6b800cecc0 msgr2=0x7f6b800b4f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:06.998 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.996+0000 7f6b86200640 1 --2- 172.21.15.32:0/4015011595 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6b800cecc0 0x7f6b800b4f10 secure :-1 s=READY pgs=1492 cs=0 l=1 rev1=1 crypto rx=0x7f6b70005fd0 tx=0x7f6b7002a5d0 comp rx=0 tx=0).stop 2024-08-03T20:57:06.998 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.996+0000 7f6b86200640 1 -- 172.21.15.32:0/4015011595 shutdown_connections 2024-08-03T20:57:06.998 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.996+0000 7f6b86200640 1 --2- 172.21.15.32:0/4015011595 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f6b801606f0 0x7f6b801450f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:06.998 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.996+0000 7f6b86200640 1 --2- 172.21.15.32:0/4015011595 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6b800cecc0 0x7f6b800b4f10 unknown :-1 s=CLOSED pgs=1492 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:06.998 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.996+0000 7f6b86200640 1 -- 172.21.15.32:0/4015011595 >> 172.21.15.32:0/4015011595 conn(0x7f6b800b1bd0 msgr2=0x7f6b800b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:57:06.998 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.996+0000 7f6b86200640 1 -- 172.21.15.32:0/4015011595 shutdown_connections 2024-08-03T20:57:06.999 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.996+0000 7f6b86200640 1 -- 172.21.15.32:0/4015011595 wait complete. 2024-08-03T20:57:07.000 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.998+0000 7f6b86200640 1 Processor -- start 2024-08-03T20:57:07.000 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:06.999+0000 7f6b86200640 1 -- start start 2024-08-03T20:57:07.003 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:07.000+0000 7f6b86200640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f6b800cd210 0x7f6b800c9250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:07.003 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:07.000+0000 7f6b86200640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f6b800cecc0 0x7f6b8013aa30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:07.003 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:07.000+0000 7f6b86200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6b800c9980 0x7f6b800c9e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:07.003 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:07.000+0000 7f6b86200640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f6b80145ad0 con 0x7f6b800c9980 2024-08-03T20:57:07.003 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:07.000+0000 7f6b86200640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f6b80145950 con 0x7f6b800cecc0 2024-08-03T20:57:07.003 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:07.000+0000 7f6b86200640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7f6b80145c50 con 0x7f6b800cd210 2024-08-03T20:57:07.003 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:07.000+0000 7f6b7f400640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f6b800cd210 0x7f6b800c9250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:07.003 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:07.000+0000 7f6b7fe00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6b800c9980 0x7f6b800c9e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:07.003 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:07.001+0000 7f6b7f400640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f6b800cd210 0x7f6b800c9250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3301/0 says I am v2:172.21.15.32:46150/0 (socket says 172.21.15.32:46150) 2024-08-03T20:57:07.003 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:07.001+0000 7f6b7f400640 1 -- 172.21.15.32:0/1695449921 learned_addr learned my addr 172.21.15.32:0/1695449921 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:57:07.003 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:07.001+0000 7f6b7ea00640 1 --2- 172.21.15.32:0/1695449921 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f6b800cecc0 0x7f6b8013aa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:07.003 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:07.001+0000 7f6b7f400640 1 -- 172.21.15.32:0/1695449921 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f6b800cecc0 msgr2=0x7f6b8013aa30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:07.004 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:07.001+0000 7f6b7f400640 1 --2- 172.21.15.32:0/1695449921 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f6b800cecc0 0x7f6b8013aa30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:07.004 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:07.001+0000 7f6b7f400640 1 -- 172.21.15.32:0/1695449921 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6b800c9980 msgr2=0x7f6b800c9e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:07.004 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:07.001+0000 7f6b7f400640 1 --2- 172.21.15.32:0/1695449921 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6b800c9980 0x7f6b800c9e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:07.004 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:07.001+0000 7f6b7f400640 1 -- 172.21.15.32:0/1695449921 --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6b8015f050 con 0x7f6b800cd210 2024-08-03T20:57:07.004 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:07.001+0000 7f6b7fe00640 1 --2- 172.21.15.32:0/1695449921 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6b800c9980 0x7f6b800c9e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-03T20:57:07.004 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:07.001+0000 7f6b7f400640 1 --2- 172.21.15.32:0/1695449921 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f6b800cd210 0x7f6b800c9250 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f6b70034040 tx=0x7f6b700035f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:57:07.005 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:07.002+0000 7f6b77400640 1 -- 172.21.15.32:0/1695449921 <== mon.2 v2:172.21.15.118:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b7003e030 con 0x7f6b800cd210 2024-08-03T20:57:07.005 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:07.002+0000 7f6b77400640 1 -- 172.21.15.32:0/1695449921 <== mon.2 v2:172.21.15.118:3301/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f6b70003ae0 con 0x7f6b800cd210 2024-08-03T20:57:07.005 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:07.002+0000 7f6b86200640 1 -- 172.21.15.32:0/1695449921 --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6b8015f330 con 0x7f6b800cd210 2024-08-03T20:57:07.005 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:07.002+0000 7f6b77400640 1 -- 172.21.15.32:0/1695449921 <== mon.2 v2:172.21.15.118:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b70035040 con 0x7f6b800cd210 2024-08-03T20:57:07.005 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:07.002+0000 7f6b86200640 1 -- 172.21.15.32:0/1695449921 --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f6b8015f8f0 con 0x7f6b800cd210 2024-08-03T20:57:07.005 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:07.003+0000 7f6b86200640 1 -- 172.21.15.32:0/1695449921 --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6b80070190 con 0x7f6b800cd210 2024-08-03T20:57:07.006 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:07.004+0000 7f6b77400640 1 -- 172.21.15.32:0/1695449921 <== mon.2 v2:172.21.15.118:3301/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f6b70041030 con 0x7f6b800cd210 2024-08-03T20:57:07.007 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:07.005+0000 7f6b77400640 1 --2- 172.21.15.32:0/1695449921 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f6b60077010 0x7f6b60079520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:07.007 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:07.006+0000 7f6b77400640 1 -- 172.21.15.32:0/1695449921 <== mon.2 v2:172.21.15.118:3301/0 5 ==== osd_map(102..102 src has 1..102) ==== 7316+0+0 (secure 0 0 0) 0x7f6b700b9350 con 0x7f6b800cd210 2024-08-03T20:57:07.007 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:07.006+0000 7f6b7ea00640 1 --2- 172.21.15.32:0/1695449921 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f6b60077010 0x7f6b60079520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:07.008 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:07.006+0000 7f6b7ea00640 1 --2- 172.21.15.32:0/1695449921 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f6b60077010 0x7f6b60079520 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7f6b8006f820 tx=0x7f6b6c009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:57:07.011 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:07.010+0000 7f6b77400640 1 -- 172.21.15.32:0/1695449921 <== mon.2 v2:172.21.15.118:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7f6b80070190 con 0x7f6b800cd210 2024-08-03T20:57:07.018 INFO:tasks.ceph.mds.a.smithi032.stdout:starting mds.a at 2024-08-03T20:57:07.263 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:07.261+0000 7f6b86200640 1 -- 172.21.15.32:0/1695449921 --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "joinable", "val": "true"} v 0) -- 0x7f6b800b4f10 con 0x7f6b800cd210 2024-08-03T20:57:08.177 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:08.175+0000 7f6b77400640 1 -- 172.21.15.32:0/1695449921 <== mon.2 v2:172.21.15.118:3301/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "joinable", "val": "true"}]=0 cephfs marked joinable; MDS may join as newly active. v195) ==== 162+0+0 (secure 0 0 0) 0x7f6b700818a0 con 0x7f6b800cd210 2024-08-03T20:57:08.177 INFO:teuthology.orchestra.run.smithi032.stderr:cephfs marked joinable; MDS may join as newly active. 2024-08-03T20:57:08.181 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:08.179+0000 7f6b86200640 1 -- 172.21.15.32:0/1695449921 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f6b60077010 msgr2=0x7f6b60079520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:08.181 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:08.179+0000 7f6b86200640 1 --2- 172.21.15.32:0/1695449921 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f6b60077010 0x7f6b60079520 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7f6b8006f820 tx=0x7f6b6c009040 comp rx=0 tx=0).stop 2024-08-03T20:57:08.181 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:08.179+0000 7f6b86200640 1 -- 172.21.15.32:0/1695449921 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f6b800cd210 msgr2=0x7f6b800c9250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:08.181 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:08.179+0000 7f6b86200640 1 --2- 172.21.15.32:0/1695449921 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f6b800cd210 0x7f6b800c9250 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f6b70034040 tx=0x7f6b700035f0 comp rx=0 tx=0).stop 2024-08-03T20:57:08.181 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:08.179+0000 7f6b86200640 1 -- 172.21.15.32:0/1695449921 shutdown_connections 2024-08-03T20:57:08.181 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:08.179+0000 7f6b86200640 1 --2- 172.21.15.32:0/1695449921 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f6b60077010 0x7f6b60079520 unknown :-1 s=CLOSED pgs=609 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:08.181 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:08.179+0000 7f6b86200640 1 --2- 172.21.15.32:0/1695449921 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f6b800cecc0 0x7f6b8013aa30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:08.181 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:08.179+0000 7f6b86200640 1 --2- 172.21.15.32:0/1695449921 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6b800c9980 0x7f6b800c9e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:08.181 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:08.179+0000 7f6b86200640 1 --2- 172.21.15.32:0/1695449921 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f6b800cd210 0x7f6b800c9250 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:08.182 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:08.179+0000 7f6b86200640 1 -- 172.21.15.32:0/1695449921 >> 172.21.15.32:0/1695449921 conn(0x7f6b800b1bd0 msgr2=0x7f6b800d0360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:57:08.182 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:08.179+0000 7f6b86200640 1 -- 172.21.15.32:0/1695449921 shutdown_connections 2024-08-03T20:57:08.182 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:08.179+0000 7f6b86200640 1 -- 172.21.15.32:0/1695449921 wait complete. 2024-08-03T20:57:08.190 INFO:tasks.ceph.mds.a.smithi032.stderr:2024-08-03T20:57:08.188+0000 7f4c6be00640 -1 log_channel(cluster) log [ERR] : missing journal object 2024-08-03T20:57:08.190 INFO:tasks.ceph.mon.a.smithi032.stderr:2024-08-03T20:57:08.189+0000 7f4e6d600640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-08-03T20:57:08.191 INFO:tasks.ceph.mon.a.smithi032.stderr:2024-08-03T20:57:08.189+0000 7f4e6d600640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 mds daemon damaged (MDS_DAMAGE) 2024-08-03T20:57:08.214 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-08-03T20:57:08.417 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-08-03T20:57:08.422 DEBUG:teuthology.orchestra.run.smithi118:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-08-03T20:57:08.448 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:08.446+0000 7f8d89000640 1 -- 172.21.15.32:0/908326408 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8d8406e930 msgr2=0x7f8d84144f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:08.448 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:08.446+0000 7f8d89000640 1 --2- 172.21.15.32:0/908326408 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8d8406e930 0x7f8d84144f90 secure :-1 s=READY pgs=1497 cs=0 l=1 rev1=1 crypto rx=0x7f8d840afaf0 tx=0x7f8d7402a4a0 comp rx=0 tx=0).stop 2024-08-03T20:57:08.448 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:08.446+0000 7f8d89000640 1 -- 172.21.15.32:0/908326408 shutdown_connections 2024-08-03T20:57:08.448 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:08.446+0000 7f8d89000640 1 --2- 172.21.15.32:0/908326408 >> v2:172.21.15.118:3300/0 conn(0x7f8d84073e60 0x7f8d84074270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:08.448 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:08.446+0000 7f8d89000640 1 --2- 172.21.15.32:0/908326408 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8d8406e930 0x7f8d84144f90 unknown :-1 s=CLOSED pgs=1497 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:08.448 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:08.446+0000 7f8d89000640 1 -- 172.21.15.32:0/908326408 >> 172.21.15.32:0/908326408 conn(0x7f8d840b1bd0 msgr2=0x7f8d840b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:57:08.449 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:08.446+0000 7f8d89000640 1 -- 172.21.15.32:0/908326408 shutdown_connections 2024-08-03T20:57:08.449 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:08.446+0000 7f8d89000640 1 -- 172.21.15.32:0/908326408 wait complete. 2024-08-03T20:57:08.450 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:08.448+0000 7f8d89000640 1 Processor -- start 2024-08-03T20:57:08.450 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:08.449+0000 7f8d89000640 1 -- start start 2024-08-03T20:57:08.452 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:08.450+0000 7f8d89000640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8d8406e930 0x7f8d84158de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:08.452 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:08.450+0000 7f8d89000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8d84073e60 0x7f8d8413d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:08.452 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:08.450+0000 7f8d89000640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f8d840747b0 0x7f8d84144080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:08.452 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:08.450+0000 7f8d89000640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f8d840cf9c0 con 0x7f8d84073e60 2024-08-03T20:57:08.453 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:08.450+0000 7f8d89000640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f8d840cf840 con 0x7f8d8406e930 2024-08-03T20:57:08.453 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:08.450+0000 7f8d89000640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7f8d840ccff0 con 0x7f8d840747b0 2024-08-03T20:57:08.453 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:08.450+0000 7f8d81e00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8d8406e930 0x7f8d84158de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:08.453 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:08.450+0000 7f8d7be00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8d84073e60 0x7f8d8413d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:08.453 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:08.450+0000 7f8d81e00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8d8406e930 0x7f8d84158de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.32:46978/0 (socket says 172.21.15.32:46978) 2024-08-03T20:57:08.453 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:08.450+0000 7f8d81e00640 1 -- 172.21.15.32:0/852702625 learned_addr learned my addr 172.21.15.32:0/852702625 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:57:08.453 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:08.451+0000 7f8d82800640 1 --2- 172.21.15.32:0/852702625 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f8d840747b0 0x7f8d84144080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:08.453 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:08.451+0000 7f8d81e00640 1 -- 172.21.15.32:0/852702625 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f8d840747b0 msgr2=0x7f8d84144080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:08.453 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:08.451+0000 7f8d81e00640 1 --2- 172.21.15.32:0/852702625 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f8d840747b0 0x7f8d84144080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:08.453 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:08.451+0000 7f8d81e00640 1 -- 172.21.15.32:0/852702625 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8d84073e60 msgr2=0x7f8d8413d9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:08.453 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:08.451+0000 7f8d81e00640 1 --2- 172.21.15.32:0/852702625 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8d84073e60 0x7f8d8413d9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:08.453 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:08.451+0000 7f8d81e00640 1 -- 172.21.15.32:0/852702625 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8d8415a6c0 con 0x7f8d8406e930 2024-08-03T20:57:08.454 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:08.451+0000 7f8d81e00640 1 --2- 172.21.15.32:0/852702625 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8d8406e930 0x7f8d84158de0 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7f8d74009920 tx=0x7f8d74033c20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:57:08.454 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:08.452+0000 7f8d7a000640 1 -- 172.21.15.32:0/852702625 <== mon.1 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d74031d60 con 0x7f8d8406e930 2024-08-03T20:57:08.455 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:08.452+0000 7f8d7a000640 1 -- 172.21.15.32:0/852702625 <== mon.1 v2:172.21.15.118:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f8d74033e10 con 0x7f8d8406e930 2024-08-03T20:57:08.455 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:08.452+0000 7f8d7a000640 1 -- 172.21.15.32:0/852702625 <== mon.1 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d74005260 con 0x7f8d8406e930 2024-08-03T20:57:08.455 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:08.452+0000 7f8d89000640 1 -- 172.21.15.32:0/852702625 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8d84158450 con 0x7f8d8406e930 2024-08-03T20:57:08.455 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:08.452+0000 7f8d89000640 1 -- 172.21.15.32:0/852702625 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8d84158a10 con 0x7f8d8406e930 2024-08-03T20:57:08.455 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:08.453+0000 7f8d89000640 1 -- 172.21.15.32:0/852702625 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8d840cd580 con 0x7f8d8406e930 2024-08-03T20:57:08.456 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:08.454+0000 7f8d7a000640 1 -- 172.21.15.32:0/852702625 <== mon.1 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f8d74005400 con 0x7f8d8406e930 2024-08-03T20:57:08.457 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:08.455+0000 7f8d7a000640 1 --2- 172.21.15.32:0/852702625 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f8d60077010 0x7f8d60079520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:08.457 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:08.455+0000 7f8d7a000640 1 -- 172.21.15.32:0/852702625 <== mon.1 v2:172.21.15.118:3300/0 5 ==== osd_map(103..103 src has 1..103) ==== 7402+0+0 (secure 0 0 0) 0x7f8d740ba080 con 0x7f8d8406e930 2024-08-03T20:57:08.457 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:08.456+0000 7f8d7be00640 1 --2- 172.21.15.32:0/852702625 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f8d60077010 0x7f8d60079520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:08.458 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:08.457+0000 7f8d7be00640 1 --2- 172.21.15.32:0/852702625 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f8d60077010 0x7f8d60079520 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7f8d841401d0 tx=0x7f8d6c005f90 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:57:08.462 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:08.461+0000 7f8d7a000640 1 -- 172.21.15.32:0/852702625 <== mon.1 v2:172.21.15.118:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7f8d7403f030 con 0x7f8d8406e930 2024-08-03T20:57:08.708 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:08.707+0000 7f8d89000640 1 -- 172.21.15.32:0/852702625 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f8d840b12a0 con 0x7f8d8406e930 2024-08-03T20:57:08.709 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:08.708+0000 7f8d7a000640 1 -- 172.21.15.32:0/852702625 <== mon.1 v2:172.21.15.118:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 197 v197) ==== 95+0+4391 (secure 0 0 0) 0x7f8d74082580 con 0x7f8d8406e930 2024-08-03T20:57:08.709 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-03T20:57:08.709 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":197,"btime":"2024-08-03T20:57:08:191032+0000","default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6439,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6835/531080160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6834","nonce":531080160},{"type":"v1","addr":"172.21.15.118:6835","nonce":531080160}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":84},{"gid":6454,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6837/2796882319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6836","nonce":2796882319},{"type":"v1","addr":"172.21.15.118:6837","nonce":2796882319}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":86},{"gid":6813,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6834/2686933890","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":2686933890},{"type":"v1","addr":"172.21.15.32:6834","nonce":2686933890}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":85}],"filesystems":[{"mdsmap":{"epoch":197,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-08-03T20:54:26.950518+0000","modified":"2024-08-03T20:57:08.190957+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":103,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":6}]} 2024-08-03T20:57:08.710 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 197 2024-08-03T20:57:08.712 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:08.710+0000 7f8d89000640 1 -- 172.21.15.32:0/852702625 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f8d60077010 msgr2=0x7f8d60079520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:08.712 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:08.710+0000 7f8d89000640 1 --2- 172.21.15.32:0/852702625 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f8d60077010 0x7f8d60079520 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7f8d841401d0 tx=0x7f8d6c005f90 comp rx=0 tx=0).stop 2024-08-03T20:57:08.712 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:08.710+0000 7f8d89000640 1 -- 172.21.15.32:0/852702625 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8d8406e930 msgr2=0x7f8d84158de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:08.712 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:08.710+0000 7f8d89000640 1 --2- 172.21.15.32:0/852702625 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8d8406e930 0x7f8d84158de0 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7f8d74009920 tx=0x7f8d74033c20 comp rx=0 tx=0).stop 2024-08-03T20:57:08.713 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:08.710+0000 7f8d89000640 1 -- 172.21.15.32:0/852702625 shutdown_connections 2024-08-03T20:57:08.713 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:08.710+0000 7f8d89000640 1 --2- 172.21.15.32:0/852702625 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f8d60077010 0x7f8d60079520 unknown :-1 s=CLOSED pgs=611 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:08.713 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:08.710+0000 7f8d89000640 1 --2- 172.21.15.32:0/852702625 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f8d840747b0 0x7f8d84144080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:08.713 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:08.710+0000 7f8d89000640 1 --2- 172.21.15.32:0/852702625 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8d84073e60 0x7f8d8413d9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:08.713 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:08.710+0000 7f8d89000640 1 --2- 172.21.15.32:0/852702625 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8d8406e930 0x7f8d84158de0 unknown :-1 s=CLOSED pgs=463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:08.713 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:08.710+0000 7f8d89000640 1 -- 172.21.15.32:0/852702625 >> 172.21.15.32:0/852702625 conn(0x7f8d840b1bd0 msgr2=0x7f8d840b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:57:08.713 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:08.711+0000 7f8d89000640 1 -- 172.21.15.32:0/852702625 shutdown_connections 2024-08-03T20:57:08.713 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:08.711+0000 7f8d89000640 1 -- 172.21.15.32:0/852702625 wait complete. 2024-08-03T20:57:08.735 DEBUG:tasks.ceph_test_case:wait_until_true: success in 0s and 0 retries 2024-08-03T20:57:08.736 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-08-03T20:57:08.965 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:08.963+0000 7fdd1ae00640 1 -- 172.21.15.32:0/77900193 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdcf8003a00 msgr2=0x7fdcf8003e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:08.965 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:08.963+0000 7fdd1ae00640 1 --2- 172.21.15.32:0/77900193 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdcf8003a00 0x7fdcf8003e70 secure :-1 s=READY pgs=1499 cs=0 l=1 rev1=1 crypto rx=0x7fdd04009b50 tx=0x7fdcf8005830 comp rx=0 tx=0).stop 2024-08-03T20:57:08.965 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:08.963+0000 7fdd18e00640 1 -- 172.21.15.32:0/77900193 reap_dead start 2024-08-03T20:57:08.965 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:08.963+0000 7fdd1ae00640 1 -- 172.21.15.32:0/77900193 shutdown_connections 2024-08-03T20:57:08.965 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:08.963+0000 7fdd1ae00640 1 -- 172.21.15.32:0/77900193 >> 172.21.15.32:0/77900193 conn(0x7fdd140b1bd0 msgr2=0x7fdd140b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:57:08.965 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:08.963+0000 7fdd1ae00640 1 -- 172.21.15.32:0/77900193 shutdown_connections 2024-08-03T20:57:08.965 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:08.963+0000 7fdd1ae00640 1 -- 172.21.15.32:0/77900193 wait complete. 2024-08-03T20:57:08.967 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:08.965+0000 7fdd1ae00640 1 Processor -- start 2024-08-03T20:57:08.967 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:08.966+0000 7fdd1ae00640 1 -- start start 2024-08-03T20:57:08.969 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:08.967+0000 7fdd1ae00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fdcf8003a00 0x7fdd1413c9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:08.969 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:08.967+0000 7fdd1ae00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdd14143390 0x7fdd141f51a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:08.970 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:08.967+0000 7fdd1ae00640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fdd141f56e0 0x7fdd140b43b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:08.970 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:08.967+0000 7fdd1ae00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fdd140cf9c0 con 0x7fdd14143390 2024-08-03T20:57:08.970 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:08.967+0000 7fdd1ae00640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7fdd140cf840 con 0x7fdcf8003a00 2024-08-03T20:57:08.970 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:08.967+0000 7fdd1ae00640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7fdd14144f90 con 0x7fdd141f56e0 2024-08-03T20:57:08.970 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:08.967+0000 7fdd0be00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdd14143390 0x7fdd141f51a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:08.970 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:08.967+0000 7fdd0be00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdd14143390 0x7fdd141f51a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:47168/0 (socket says 172.21.15.32:47168) 2024-08-03T20:57:08.970 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:08.967+0000 7fdd0be00640 1 -- 172.21.15.32:0/3597496720 learned_addr learned my addr 172.21.15.32:0/3597496720 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:57:08.970 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:08.968+0000 7fdd13e00640 1 --2- 172.21.15.32:0/3597496720 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fdcf8003a00 0x7fdd1413c9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:08.970 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:08.968+0000 7fdd18e00640 1 --2- 172.21.15.32:0/3597496720 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fdd141f56e0 0x7fdd140b43b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:08.971 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:08.968+0000 7fdd0be00640 1 -- 172.21.15.32:0/3597496720 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fdd141f56e0 msgr2=0x7fdd140b43b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:08.971 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:08.968+0000 7fdd0be00640 1 --2- 172.21.15.32:0/3597496720 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fdd141f56e0 0x7fdd140b43b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:08.971 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:08.968+0000 7fdd0be00640 1 -- 172.21.15.32:0/3597496720 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fdcf8003a00 msgr2=0x7fdd1413c9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:08.971 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:08.968+0000 7fdd0be00640 1 --2- 172.21.15.32:0/3597496720 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fdcf8003a00 0x7fdd1413c9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:08.971 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:08.968+0000 7fdd0be00640 1 -- 172.21.15.32:0/3597496720 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdd141f5f40 con 0x7fdd14143390 2024-08-03T20:57:08.971 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:08.968+0000 7fdd0be00640 1 --2- 172.21.15.32:0/3597496720 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdd14143390 0x7fdd141f51a0 secure :-1 s=READY pgs=1500 cs=0 l=1 rev1=1 crypto rx=0x7fdcfc011a50 tx=0x7fdcfc011f10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:57:08.972 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:08.968+0000 7fdd10c00640 1 -- 172.21.15.32:0/3597496720 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdcfc01f070 con 0x7fdd14143390 2024-08-03T20:57:08.972 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:08.969+0000 7fdd10c00640 1 -- 172.21.15.32:0/3597496720 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fdcfc00fd90 con 0x7fdd14143390 2024-08-03T20:57:08.972 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:08.969+0000 7fdd10c00640 1 -- 172.21.15.32:0/3597496720 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdcfc012690 con 0x7fdd14143390 2024-08-03T20:57:08.972 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:08.969+0000 7fdd1ae00640 1 -- 172.21.15.32:0/3597496720 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdd141425d0 con 0x7fdd14143390 2024-08-03T20:57:08.972 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:08.969+0000 7fdd1ae00640 1 -- 172.21.15.32:0/3597496720 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdd14142b90 con 0x7fdd14143390 2024-08-03T20:57:08.972 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:08.970+0000 7fdd1ae00640 1 -- 172.21.15.32:0/3597496720 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdd141f5cb0 con 0x7fdd14143390 2024-08-03T20:57:08.972 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:08.971+0000 7fdd10c00640 1 -- 172.21.15.32:0/3597496720 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7fdcfc022070 con 0x7fdd14143390 2024-08-03T20:57:08.973 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:08.972+0000 7fdd10c00640 1 --2- 172.21.15.32:0/3597496720 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fdcf0077040 0x7fdcf0079550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:08.973 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:08.972+0000 7fdd10c00640 1 -- 172.21.15.32:0/3597496720 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(103..103 src has 1..103) ==== 7402+0+0 (secure 0 0 0) 0x7fdcfc0999b0 con 0x7fdd14143390 2024-08-03T20:57:08.974 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:08.972+0000 7fdd13e00640 1 --2- 172.21.15.32:0/3597496720 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fdcf0077040 0x7fdcf0079550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:08.975 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:08.973+0000 7fdd13e00640 1 --2- 172.21.15.32:0/3597496720 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fdcf0077040 0x7fdcf0079550 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7fdd04035000 tx=0x7fdd0402d040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:57:08.978 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:08.976+0000 7fdd10c00640 1 -- 172.21.15.32:0/3597496720 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7fdcfc061e20 con 0x7fdd14143390 2024-08-03T20:57:09.222 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:09.221+0000 7fdd1ae00640 1 -- 172.21.15.32:0/3597496720 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fdd1406a430 con 0x7fdd14143390 2024-08-03T20:57:09.223 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:09.221+0000 7fdd10c00640 1 -- 172.21.15.32:0/3597496720 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 197 v197) ==== 95+0+4391 (secure 0 0 0) 0x7fdd1406a430 con 0x7fdd14143390 2024-08-03T20:57:09.223 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-03T20:57:09.223 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":197,"btime":"2024-08-03T20:57:08:191032+0000","default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6439,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6835/531080160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6834","nonce":531080160},{"type":"v1","addr":"172.21.15.118:6835","nonce":531080160}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":84},{"gid":6454,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6837/2796882319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6836","nonce":2796882319},{"type":"v1","addr":"172.21.15.118:6837","nonce":2796882319}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":86},{"gid":6813,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6834/2686933890","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":2686933890},{"type":"v1","addr":"172.21.15.32:6834","nonce":2686933890}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":85}],"filesystems":[{"mdsmap":{"epoch":197,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-08-03T20:54:26.950518+0000","modified":"2024-08-03T20:57:08.190957+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":103,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":6}]} 2024-08-03T20:57:09.223 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 197 2024-08-03T20:57:09.225 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:09.224+0000 7fdd1ae00640 1 -- 172.21.15.32:0/3597496720 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fdcf0077040 msgr2=0x7fdcf0079550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:09.225 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:09.224+0000 7fdd1ae00640 1 --2- 172.21.15.32:0/3597496720 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fdcf0077040 0x7fdcf0079550 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7fdd04035000 tx=0x7fdd0402d040 comp rx=0 tx=0).stop 2024-08-03T20:57:09.226 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:09.224+0000 7fdd1ae00640 1 -- 172.21.15.32:0/3597496720 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdd14143390 msgr2=0x7fdd141f51a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:09.226 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:09.224+0000 7fdd1ae00640 1 --2- 172.21.15.32:0/3597496720 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdd14143390 0x7fdd141f51a0 secure :-1 s=READY pgs=1500 cs=0 l=1 rev1=1 crypto rx=0x7fdcfc011a50 tx=0x7fdcfc011f10 comp rx=0 tx=0).stop 2024-08-03T20:57:09.226 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:09.224+0000 7fdd1ae00640 1 -- 172.21.15.32:0/3597496720 shutdown_connections 2024-08-03T20:57:09.226 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:09.224+0000 7fdd1ae00640 1 --2- 172.21.15.32:0/3597496720 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fdcf0077040 0x7fdcf0079550 unknown :-1 s=CLOSED pgs=612 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:09.226 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:09.224+0000 7fdd1ae00640 1 --2- 172.21.15.32:0/3597496720 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fdd141f56e0 0x7fdd140b43b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:09.226 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:09.224+0000 7fdd1ae00640 1 --2- 172.21.15.32:0/3597496720 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdd14143390 0x7fdd141f51a0 unknown :-1 s=CLOSED pgs=1500 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:09.226 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:09.224+0000 7fdd1ae00640 1 --2- 172.21.15.32:0/3597496720 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fdcf8003a00 0x7fdd1413c9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:09.226 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:09.224+0000 7fdd1ae00640 1 -- 172.21.15.32:0/3597496720 >> 172.21.15.32:0/3597496720 conn(0x7fdd140b1bd0 msgr2=0x7fdd140b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:57:09.226 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:09.224+0000 7fdd1ae00640 1 -- 172.21.15.32:0/3597496720 shutdown_connections 2024-08-03T20:57:09.226 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:09.224+0000 7fdd1ae00640 1 -- 172.21.15.32:0/3597496720 wait complete. 2024-08-03T20:57:09.247 DEBUG:tasks.ceph_test_case:wait_until_true: success in 0s and 0 retries 2024-08-03T20:57:09.247 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-08-03T20:57:09.474 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:09.472+0000 7fa637000640 1 -- 172.21.15.32:0/210556461 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa6300733f0 msgr2=0x7fa630073800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:09.475 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:09.473+0000 7fa637000640 1 --2- 172.21.15.32:0/210556461 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa6300733f0 0x7fa630073800 secure :-1 s=READY pgs=1502 cs=0 l=1 rev1=1 crypto rx=0x7fa61c0092b0 tx=0x7fa61c02a3d0 comp rx=0 tx=0).stop 2024-08-03T20:57:09.475 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:09.473+0000 7fa637000640 1 -- 172.21.15.32:0/210556461 shutdown_connections 2024-08-03T20:57:09.475 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:09.473+0000 7fa637000640 1 --2- 172.21.15.32:0/210556461 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa6300733f0 0x7fa630073800 unknown :-1 s=CLOSED pgs=1502 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:09.475 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:09.473+0000 7fa637000640 1 --2- 172.21.15.32:0/210556461 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fa6300b4de0 0x7fa6301515c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:09.475 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:09.473+0000 7fa637000640 1 -- 172.21.15.32:0/210556461 >> 172.21.15.32:0/210556461 conn(0x7fa6300b1bd0 msgr2=0x7fa6300b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:57:09.475 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:09.473+0000 7fa637000640 1 -- 172.21.15.32:0/210556461 shutdown_connections 2024-08-03T20:57:09.475 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:09.473+0000 7fa637000640 1 -- 172.21.15.32:0/210556461 wait complete. 2024-08-03T20:57:09.476 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:09.476+0000 7fa637000640 1 Processor -- start 2024-08-03T20:57:09.476 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:09.476+0000 7fa637000640 1 -- start start 2024-08-03T20:57:09.479 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:09.478+0000 7fa637000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa6300733f0 0x7fa630154680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:09.479 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:09.478+0000 7fa637000640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa6300b4de0 0x7fa6301f8b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:09.479 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:09.478+0000 7fa637000640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fa63015e170 0x7fa6300a94b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:09.479 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:09.478+0000 7fa637000640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fa63006e930 con 0x7fa6300733f0 2024-08-03T20:57:09.479 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:09.478+0000 7fa637000640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7fa630070ad0 con 0x7fa6300b4de0 2024-08-03T20:57:09.479 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:09.478+0000 7fa637000640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7fa63006eab0 con 0x7fa63015e170 2024-08-03T20:57:09.479 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:09.478+0000 7fa62fe00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa6300733f0 0x7fa630154680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:09.480 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:09.478+0000 7fa62fe00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa6300733f0 0x7fa630154680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:47182/0 (socket says 172.21.15.32:47182) 2024-08-03T20:57:09.480 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:09.478+0000 7fa62fe00640 1 -- 172.21.15.32:0/1618564977 learned_addr learned my addr 172.21.15.32:0/1618564977 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:57:09.480 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:09.478+0000 7fa635000640 1 --2- 172.21.15.32:0/1618564977 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fa63015e170 0x7fa6300a94b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:09.480 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:09.478+0000 7fa62f400640 1 --2- 172.21.15.32:0/1618564977 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa6300b4de0 0x7fa6301f8b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:09.480 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:09.478+0000 7fa62fe00640 1 -- 172.21.15.32:0/1618564977 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fa63015e170 msgr2=0x7fa6300a94b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:09.480 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:09.478+0000 7fa62fe00640 1 --2- 172.21.15.32:0/1618564977 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fa63015e170 0x7fa6300a94b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:09.480 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:09.478+0000 7fa62fe00640 1 -- 172.21.15.32:0/1618564977 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa6300b4de0 msgr2=0x7fa6301f8b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:09.480 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:09.478+0000 7fa62fe00640 1 --2- 172.21.15.32:0/1618564977 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa6300b4de0 0x7fa6301f8b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:09.480 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:09.478+0000 7fa62fe00640 1 -- 172.21.15.32:0/1618564977 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa6301542e0 con 0x7fa6300733f0 2024-08-03T20:57:09.480 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:09.478+0000 7fa62fe00640 1 --2- 172.21.15.32:0/1618564977 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa6300733f0 0x7fa630154680 secure :-1 s=READY pgs=1503 cs=0 l=1 rev1=1 crypto rx=0x7fa62000ea90 tx=0x7fa62000ef50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:57:09.481 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:09.478+0000 7fa627e00640 1 -- 172.21.15.32:0/1618564977 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa62001b070 con 0x7fa6300733f0 2024-08-03T20:57:09.481 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:09.478+0000 7fa627e00640 1 -- 172.21.15.32:0/1618564977 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fa62000cd70 con 0x7fa6300733f0 2024-08-03T20:57:09.481 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:09.478+0000 7fa627e00640 1 -- 172.21.15.32:0/1618564977 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa620016670 con 0x7fa6300733f0 2024-08-03T20:57:09.481 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:09.478+0000 7fa637000640 1 -- 172.21.15.32:0/1618564977 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa630152850 con 0x7fa6300733f0 2024-08-03T20:57:09.481 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:09.478+0000 7fa637000640 1 -- 172.21.15.32:0/1618564977 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa630154bc0 con 0x7fa6300733f0 2024-08-03T20:57:09.481 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:09.479+0000 7fa637000640 1 -- 172.21.15.32:0/1618564977 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa630073800 con 0x7fa6300733f0 2024-08-03T20:57:09.482 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:09.481+0000 7fa627e00640 1 -- 172.21.15.32:0/1618564977 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7fa62001e070 con 0x7fa6300733f0 2024-08-03T20:57:09.483 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:09.481+0000 7fa627e00640 1 --2- 172.21.15.32:0/1618564977 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fa610077010 0x7fa610079520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:09.483 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:09.482+0000 7fa627e00640 1 -- 172.21.15.32:0/1618564977 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(103..103 src has 1..103) ==== 7402+0+0 (secure 0 0 0) 0x7fa620099250 con 0x7fa6300733f0 2024-08-03T20:57:09.483 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:09.482+0000 7fa62f400640 1 --2- 172.21.15.32:0/1618564977 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fa610077010 0x7fa610079520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:09.484 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:09.484+0000 7fa62f400640 1 --2- 172.21.15.32:0/1618564977 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fa610077010 0x7fa610079520 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7fa61c034000 tx=0x7fa61c004730 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:57:09.487 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:09.486+0000 7fa627e00640 1 -- 172.21.15.32:0/1618564977 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7fa6200616c0 con 0x7fa6300733f0 2024-08-03T20:57:09.730 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:09.730+0000 7fa637000640 1 -- 172.21.15.32:0/1618564977 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa630072a70 con 0x7fa6300733f0 2024-08-03T20:57:09.731 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:09.730+0000 7fa627e00640 1 -- 172.21.15.32:0/1618564977 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 197 v197) ==== 95+0+4391 (secure 0 0 0) 0x7fa620065c10 con 0x7fa6300733f0 2024-08-03T20:57:09.731 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-03T20:57:09.731 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":197,"btime":"2024-08-03T20:57:08:191032+0000","default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6439,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6835/531080160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6834","nonce":531080160},{"type":"v1","addr":"172.21.15.118:6835","nonce":531080160}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":84},{"gid":6454,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6837/2796882319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6836","nonce":2796882319},{"type":"v1","addr":"172.21.15.118:6837","nonce":2796882319}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":86},{"gid":6813,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6834/2686933890","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":2686933890},{"type":"v1","addr":"172.21.15.32:6834","nonce":2686933890}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":85}],"filesystems":[{"mdsmap":{"epoch":197,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-08-03T20:54:26.950518+0000","modified":"2024-08-03T20:57:08.190957+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":103,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":6}]} 2024-08-03T20:57:09.731 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 197 2024-08-03T20:57:09.733 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:09.732+0000 7fa637000640 1 -- 172.21.15.32:0/1618564977 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fa610077010 msgr2=0x7fa610079520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:09.733 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:09.732+0000 7fa637000640 1 --2- 172.21.15.32:0/1618564977 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fa610077010 0x7fa610079520 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7fa61c034000 tx=0x7fa61c004730 comp rx=0 tx=0).stop 2024-08-03T20:57:09.733 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:09.732+0000 7fa637000640 1 -- 172.21.15.32:0/1618564977 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa6300733f0 msgr2=0x7fa630154680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:09.733 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:09.732+0000 7fa637000640 1 --2- 172.21.15.32:0/1618564977 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa6300733f0 0x7fa630154680 secure :-1 s=READY pgs=1503 cs=0 l=1 rev1=1 crypto rx=0x7fa62000ea90 tx=0x7fa62000ef50 comp rx=0 tx=0).stop 2024-08-03T20:57:09.733 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:09.732+0000 7fa637000640 1 -- 172.21.15.32:0/1618564977 shutdown_connections 2024-08-03T20:57:09.733 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:09.732+0000 7fa637000640 1 --2- 172.21.15.32:0/1618564977 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fa610077010 0x7fa610079520 unknown :-1 s=CLOSED pgs=613 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:09.733 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:09.732+0000 7fa637000640 1 --2- 172.21.15.32:0/1618564977 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa6300b4de0 0x7fa6301f8b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:09.733 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:09.732+0000 7fa637000640 1 --2- 172.21.15.32:0/1618564977 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fa63015e170 0x7fa6300a94b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:09.734 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:09.732+0000 7fa637000640 1 --2- 172.21.15.32:0/1618564977 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa6300733f0 0x7fa630154680 unknown :-1 s=CLOSED pgs=1503 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:09.734 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:09.732+0000 7fa637000640 1 -- 172.21.15.32:0/1618564977 >> 172.21.15.32:0/1618564977 conn(0x7fa6300b1bd0 msgr2=0x7fa63015cf40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:57:09.734 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:09.733+0000 7fa637000640 1 -- 172.21.15.32:0/1618564977 shutdown_connections 2024-08-03T20:57:09.734 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:09.733+0000 7fa637000640 1 -- 172.21.15.32:0/1618564977 wait complete. 2024-08-03T20:57:09.757 INFO:tasks.cephfs.test_damage:Result: Mutation 'Corrupt 200.00000001' led to DAMAGED state 2024-08-03T20:57:09.757 INFO:tasks.cephfs.test_damage:Applying mutation 'Corrupt 500.00000000' 2024-08-03T20:57:09.758 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-08-03T20:57:09.758 DEBUG:teuthology.orchestra.run.smithi032:> dd if=/proc/self/mounts of=/dev/stdout 2024-08-03T20:57:09.794 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053453,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6499208k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 none /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249604k,nr_inodes=812401,mode=700,uid=1000,gid=1316 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6499208k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-08-03T20:57:09.794 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi032.front.sepia.ceph.com 2024-08-03T20:57:09.794 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-08-03T20:57:09.794 DEBUG:teuthology.orchestra.run.smithi032:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-08-03T20:57:09.869 INFO:teuthology.orchestra.run.smithi032.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.0': No such file or directory 2024-08-03T20:57:09.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-03T20:57:09.870 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-08-03T20:57:10.140 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:10.139+0000 7f62baa00640 1 -- 172.21.15.32:0/381728846 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f62b4073990 msgr2=0x7f62b40cd860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:10.140 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:10.139+0000 7f62baa00640 1 --2- 172.21.15.32:0/381728846 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f62b4073990 0x7f62b40cd860 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7f62a8005fd0 tx=0x7f62a802a540 comp rx=0 tx=0).stop 2024-08-03T20:57:10.140 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:10.139+0000 7f62baa00640 1 -- 172.21.15.32:0/381728846 shutdown_connections 2024-08-03T20:57:10.140 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:10.139+0000 7f62baa00640 1 --2- 172.21.15.32:0/381728846 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f62b406d9f0 0x7f62b406de00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:10.140 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:10.139+0000 7f62baa00640 1 --2- 172.21.15.32:0/381728846 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f62b4073990 0x7f62b40cd860 unknown :-1 s=CLOSED pgs=465 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:10.141 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:10.139+0000 7f62baa00640 1 -- 172.21.15.32:0/381728846 >> 172.21.15.32:0/381728846 conn(0x7f62b4072e70 msgr2=0x7f62b4073280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:57:10.141 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:10.139+0000 7f62baa00640 1 -- 172.21.15.32:0/381728846 shutdown_connections 2024-08-03T20:57:10.141 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:10.139+0000 7f62baa00640 1 -- 172.21.15.32:0/381728846 wait complete. 2024-08-03T20:57:10.142 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:10.142+0000 7f62baa00640 1 Processor -- start 2024-08-03T20:57:10.142 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:10.142+0000 7f62baa00640 1 -- start start 2024-08-03T20:57:10.144 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:10.143+0000 7f62baa00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f62b406d9f0 0x7f62b40b1270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:10.144 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:10.143+0000 7f62baa00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f62b4073990 0x7f62b414ffb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:10.144 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:10.143+0000 7f62baa00640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f62b4153270 0x7f62b4155aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:10.144 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:10.143+0000 7f62baa00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f62b40cf7a0 con 0x7f62b406d9f0 2024-08-03T20:57:10.145 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:10.143+0000 7f62baa00640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f62b40cf620 con 0x7f62b4073990 2024-08-03T20:57:10.145 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:10.143+0000 7f62baa00640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7f62b40cf920 con 0x7f62b4153270 2024-08-03T20:57:10.145 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:10.143+0000 7f62b9600640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f62b406d9f0 0x7f62b40b1270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:10.145 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:10.144+0000 7f62b9600640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f62b406d9f0 0x7f62b40b1270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:33866/0 (socket says 172.21.15.32:33866) 2024-08-03T20:57:10.145 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:10.144+0000 7f62b9600640 1 -- 172.21.15.32:0/1970021830 learned_addr learned my addr 172.21.15.32:0/1970021830 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:57:10.145 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:10.144+0000 7f62ba000640 1 --2- 172.21.15.32:0/1970021830 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f62b4153270 0x7f62b4155aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:10.145 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:10.144+0000 7f62b8c00640 1 --2- 172.21.15.32:0/1970021830 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f62b4073990 0x7f62b414ffb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:10.147 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:10.144+0000 7f62b9600640 1 -- 172.21.15.32:0/1970021830 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f62b4153270 msgr2=0x7f62b4155aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:10.147 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:10.144+0000 7f62b9600640 1 --2- 172.21.15.32:0/1970021830 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f62b4153270 0x7f62b4155aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:10.147 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:10.144+0000 7f62b9600640 1 -- 172.21.15.32:0/1970021830 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f62b4073990 msgr2=0x7f62b414ffb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:10.147 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:10.144+0000 7f62b9600640 1 --2- 172.21.15.32:0/1970021830 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f62b4073990 0x7f62b414ffb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:10.147 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:10.144+0000 7f62b9600640 1 -- 172.21.15.32:0/1970021830 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f62b41f3010 con 0x7f62b406d9f0 2024-08-03T20:57:10.147 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:10.144+0000 7f62b9600640 1 --2- 172.21.15.32:0/1970021830 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f62b406d9f0 0x7f62b40b1270 secure :-1 s=READY pgs=1504 cs=0 l=1 rev1=1 crypto rx=0x7f62a8034040 tx=0x7f62a80038a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:57:10.148 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:10.145+0000 7f62b1600640 1 -- 172.21.15.32:0/1970021830 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62a803e030 con 0x7f62b406d9f0 2024-08-03T20:57:10.148 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:10.145+0000 7f62baa00640 1 -- 172.21.15.32:0/1970021830 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f62b41f32f0 con 0x7f62b406d9f0 2024-08-03T20:57:10.148 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:10.145+0000 7f62b1600640 1 -- 172.21.15.32:0/1970021830 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f62a8003ca0 con 0x7f62b406d9f0 2024-08-03T20:57:10.148 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:10.145+0000 7f62b1600640 1 -- 172.21.15.32:0/1970021830 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62a8035040 con 0x7f62b406d9f0 2024-08-03T20:57:10.148 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:10.145+0000 7f62baa00640 1 -- 172.21.15.32:0/1970021830 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f62b41f38b0 con 0x7f62b406d9f0 2024-08-03T20:57:10.148 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:10.146+0000 7f62baa00640 1 -- 172.21.15.32:0/1970021830 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f62b40cfe30 con 0x7f62b406d9f0 2024-08-03T20:57:10.150 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:10.147+0000 7f62b1600640 1 -- 172.21.15.32:0/1970021830 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f62a8041030 con 0x7f62b406d9f0 2024-08-03T20:57:10.150 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:10.148+0000 7f62b1600640 1 --2- 172.21.15.32:0/1970021830 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f6294077010 0x7f6294079520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:10.150 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:10.148+0000 7f62b1600640 1 -- 172.21.15.32:0/1970021830 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(103..103 src has 1..103) ==== 7402+0+0 (secure 0 0 0) 0x7f62a80b9600 con 0x7f62b406d9f0 2024-08-03T20:57:10.151 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:10.149+0000 7f62b8c00640 1 --2- 172.21.15.32:0/1970021830 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f6294077010 0x7f6294079520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:10.151 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:10.149+0000 7f62b8c00640 1 --2- 172.21.15.32:0/1970021830 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f6294077010 0x7f6294079520 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7f62b415c4a0 tx=0x7f62a0009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:57:10.153 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:10.153+0000 7f62b1600640 1 -- 172.21.15.32:0/1970021830 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7f62a80819f0 con 0x7f62b406d9f0 2024-08-03T20:57:10.397 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:10.397+0000 7f62baa00640 1 -- 172.21.15.32:0/1970021830 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7f62b415ca50 con 0x7f62b406d9f0 2024-08-03T20:57:11.204 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:11.203+0000 7f62b1600640 1 -- 172.21.15.32:0/1970021830 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v198) ==== 161+0+0 (secure 0 0 0) 0x7f62a8085f40 con 0x7f62b406d9f0 2024-08-03T20:57:11.204 INFO:teuthology.orchestra.run.smithi032.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-08-03T20:57:11.207 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:11.206+0000 7f62baa00640 1 -- 172.21.15.32:0/1970021830 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f6294077010 msgr2=0x7f6294079520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:11.207 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:11.206+0000 7f62baa00640 1 --2- 172.21.15.32:0/1970021830 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f6294077010 0x7f6294079520 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7f62b415c4a0 tx=0x7f62a0009040 comp rx=0 tx=0).stop 2024-08-03T20:57:11.208 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:11.206+0000 7f62baa00640 1 -- 172.21.15.32:0/1970021830 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f62b406d9f0 msgr2=0x7f62b40b1270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:11.208 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:11.206+0000 7f62baa00640 1 --2- 172.21.15.32:0/1970021830 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f62b406d9f0 0x7f62b40b1270 secure :-1 s=READY pgs=1504 cs=0 l=1 rev1=1 crypto rx=0x7f62a8034040 tx=0x7f62a80038a0 comp rx=0 tx=0).stop 2024-08-03T20:57:11.208 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:11.207+0000 7f62baa00640 1 -- 172.21.15.32:0/1970021830 shutdown_connections 2024-08-03T20:57:11.208 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:11.207+0000 7f62baa00640 1 --2- 172.21.15.32:0/1970021830 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f6294077010 0x7f6294079520 unknown :-1 s=CLOSED pgs=614 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:11.208 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:11.207+0000 7f62baa00640 1 --2- 172.21.15.32:0/1970021830 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f62b4073990 0x7f62b414ffb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:11.208 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:11.207+0000 7f62baa00640 1 --2- 172.21.15.32:0/1970021830 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f62b4153270 0x7f62b4155aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:11.208 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:11.207+0000 7f62baa00640 1 --2- 172.21.15.32:0/1970021830 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f62b406d9f0 0x7f62b40b1270 unknown :-1 s=CLOSED pgs=1504 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:11.208 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:11.207+0000 7f62baa00640 1 -- 172.21.15.32:0/1970021830 >> 172.21.15.32:0/1970021830 conn(0x7f62b4072e70 msgr2=0x7f62b40af8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:57:11.208 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:11.207+0000 7f62baa00640 1 -- 172.21.15.32:0/1970021830 shutdown_connections 2024-08-03T20:57:11.208 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:11.207+0000 7f62baa00640 1 -- 172.21.15.32:0/1970021830 wait complete. 2024-08-03T20:57:11.238 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mds repaired 0 2024-08-03T20:57:11.472 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:11.471+0000 7faba2a00640 1 -- 172.21.15.32:0/3656972280 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fab9c073e60 msgr2=0x7fab9c074270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:11.473 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:11.471+0000 7faba2a00640 1 --2- 172.21.15.32:0/3656972280 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fab9c073e60 0x7fab9c074270 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7fab88009830 tx=0x7fab8802a480 comp rx=0 tx=0).stop 2024-08-03T20:57:11.473 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:11.471+0000 7faba2a00640 1 -- 172.21.15.32:0/3656972280 shutdown_connections 2024-08-03T20:57:11.473 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:11.471+0000 7faba2a00640 1 --2- 172.21.15.32:0/3656972280 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fab9c073e60 0x7fab9c074270 unknown :-1 s=CLOSED pgs=467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:11.473 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:11.471+0000 7faba2a00640 1 --2- 172.21.15.32:0/3656972280 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fab9c06e930 0x7fab9c144f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:11.473 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:11.471+0000 7faba2a00640 1 -- 172.21.15.32:0/3656972280 >> 172.21.15.32:0/3656972280 conn(0x7fab9c0b1bd0 msgr2=0x7fab9c0b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:57:11.473 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:11.471+0000 7faba2a00640 1 -- 172.21.15.32:0/3656972280 shutdown_connections 2024-08-03T20:57:11.473 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:11.471+0000 7faba2a00640 1 -- 172.21.15.32:0/3656972280 wait complete. 2024-08-03T20:57:11.474 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:11.473+0000 7faba2a00640 1 Processor -- start 2024-08-03T20:57:11.474 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:11.473+0000 7faba2a00640 1 -- start start 2024-08-03T20:57:11.475 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:11.475+0000 7faba2a00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fab9c06e930 0x7fab9c139d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:11.476 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:11.475+0000 7faba2a00640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fab9c073e60 0x7fab9c15ca70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:11.476 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:11.475+0000 7faba2a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fab9c0747b0 0x7fab9c15eca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:11.476 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:11.475+0000 7faba2a00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fab9c0cf9c0 con 0x7fab9c0747b0 2024-08-03T20:57:11.476 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:11.475+0000 7faba2a00640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7fab9c0cf840 con 0x7fab9c06e930 2024-08-03T20:57:11.476 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:11.475+0000 7faba2a00640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7fab9c0ccff0 con 0x7fab9c073e60 2024-08-03T20:57:11.476 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:11.475+0000 7faba0a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fab9c0747b0 0x7fab9c15eca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:11.476 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:11.475+0000 7faba0a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fab9c0747b0 0x7fab9c15eca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:33876/0 (socket says 172.21.15.32:33876) 2024-08-03T20:57:11.476 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:11.475+0000 7faba0a00640 1 -- 172.21.15.32:0/1761884491 learned_addr learned my addr 172.21.15.32:0/1761884491 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:57:11.476 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:11.475+0000 7fab9b400640 1 --2- 172.21.15.32:0/1761884491 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fab9c073e60 0x7fab9c15ca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:11.476 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:11.475+0000 7fab9be00640 1 --2- 172.21.15.32:0/1761884491 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fab9c06e930 0x7fab9c139d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:11.477 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:11.476+0000 7faba0a00640 1 -- 172.21.15.32:0/1761884491 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fab9c073e60 msgr2=0x7fab9c15ca70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:11.477 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:11.476+0000 7faba0a00640 1 --2- 172.21.15.32:0/1761884491 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fab9c073e60 0x7fab9c15ca70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:11.478 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:11.476+0000 7faba0a00640 1 -- 172.21.15.32:0/1761884491 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fab9c06e930 msgr2=0x7fab9c139d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:11.478 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:11.476+0000 7faba0a00640 1 --2- 172.21.15.32:0/1761884491 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fab9c06e930 0x7fab9c139d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:11.478 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:11.476+0000 7faba0a00640 1 -- 172.21.15.32:0/1761884491 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fab9c1535d0 con 0x7fab9c0747b0 2024-08-03T20:57:11.478 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:11.476+0000 7fab9be00640 1 --2- 172.21.15.32:0/1761884491 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fab9c06e930 0x7fab9c139d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-03T20:57:11.478 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:11.476+0000 7faba0a00640 1 --2- 172.21.15.32:0/1761884491 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fab9c0747b0 0x7fab9c15eca0 secure :-1 s=READY pgs=1505 cs=0 l=1 rev1=1 crypto rx=0x7fab9400c700 tx=0x7fab9400cbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:57:11.478 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:11.476+0000 7fab93e00640 1 -- 172.21.15.32:0/1761884491 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab94003b70 con 0x7fab9c0747b0 2024-08-03T20:57:11.478 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:11.476+0000 7fab93e00640 1 -- 172.21.15.32:0/1761884491 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fab94003d10 con 0x7fab9c0747b0 2024-08-03T20:57:11.478 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:11.476+0000 7faba2a00640 1 -- 172.21.15.32:0/1761884491 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fab9c1538b0 con 0x7fab9c0747b0 2024-08-03T20:57:11.478 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:11.476+0000 7fab93e00640 1 -- 172.21.15.32:0/1761884491 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab94005330 con 0x7fab9c0747b0 2024-08-03T20:57:11.479 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:11.476+0000 7faba2a00640 1 -- 172.21.15.32:0/1761884491 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fab9c153e70 con 0x7fab9c0747b0 2024-08-03T20:57:11.479 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:11.477+0000 7faba2a00640 1 -- 172.21.15.32:0/1761884491 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fab9c0cd510 con 0x7fab9c0747b0 2024-08-03T20:57:11.479 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:11.479+0000 7fab93e00640 1 -- 172.21.15.32:0/1761884491 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7fab940054d0 con 0x7fab9c0747b0 2024-08-03T20:57:11.480 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:11.480+0000 7fab93e00640 1 --2- 172.21.15.32:0/1761884491 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fab7c077040 0x7fab7c079550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:11.480 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:11.480+0000 7fab93e00640 1 -- 172.21.15.32:0/1761884491 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(103..103 src has 1..103) ==== 7402+0+0 (secure 0 0 0) 0x7fab94098c30 con 0x7fab9c0747b0 2024-08-03T20:57:11.480 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:11.480+0000 7fab9be00640 1 --2- 172.21.15.32:0/1761884491 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fab7c077040 0x7fab7c079550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:11.481 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:11.481+0000 7fab9be00640 1 --2- 172.21.15.32:0/1761884491 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fab7c077040 0x7fab7c079550 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7fab8c005fd0 tx=0x7fab8c009260 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:57:11.484 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:11.484+0000 7fab93e00640 1 -- 172.21.15.32:0/1761884491 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7fab94061820 con 0x7fab9c0747b0 2024-08-03T20:57:11.767 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:11.766+0000 7faba2a00640 1 -- 172.21.15.32:0/1761884491 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "mds repaired", "role": "0"} v 0) -- 0x7fab9c1451a0 con 0x7fab9c0747b0 2024-08-03T20:57:12.206 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:12.205+0000 7fab93e00640 1 -- 172.21.15.32:0/1761884491 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "mds repaired", "role": "0"}]=0 repaired: restoring rank 6:0 v199) ==== 101+0+0 (secure 0 0 0) 0x7fab94061120 con 0x7fab9c0747b0 2024-08-03T20:57:12.206 INFO:teuthology.orchestra.run.smithi032.stderr:repaired: restoring rank 6:0 2024-08-03T20:57:12.208 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:12.208+0000 7faba2a00640 1 -- 172.21.15.32:0/1761884491 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fab7c077040 msgr2=0x7fab7c079550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:12.209 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:12.208+0000 7faba2a00640 1 --2- 172.21.15.32:0/1761884491 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fab7c077040 0x7fab7c079550 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7fab8c005fd0 tx=0x7fab8c009260 comp rx=0 tx=0).stop 2024-08-03T20:57:12.209 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:12.208+0000 7faba2a00640 1 -- 172.21.15.32:0/1761884491 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fab9c0747b0 msgr2=0x7fab9c15eca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:12.209 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:12.208+0000 7faba2a00640 1 --2- 172.21.15.32:0/1761884491 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fab9c0747b0 0x7fab9c15eca0 secure :-1 s=READY pgs=1505 cs=0 l=1 rev1=1 crypto rx=0x7fab9400c700 tx=0x7fab9400cbc0 comp rx=0 tx=0).stop 2024-08-03T20:57:12.209 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:12.208+0000 7faba2a00640 1 -- 172.21.15.32:0/1761884491 shutdown_connections 2024-08-03T20:57:12.209 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:12.208+0000 7faba2a00640 1 --2- 172.21.15.32:0/1761884491 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fab7c077040 0x7fab7c079550 unknown :-1 s=CLOSED pgs=615 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:12.209 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:12.208+0000 7faba2a00640 1 --2- 172.21.15.32:0/1761884491 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fab9c0747b0 0x7fab9c15eca0 unknown :-1 s=CLOSED pgs=1505 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:12.209 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:12.208+0000 7faba2a00640 1 --2- 172.21.15.32:0/1761884491 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fab9c073e60 0x7fab9c15ca70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:12.209 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:12.208+0000 7faba2a00640 1 --2- 172.21.15.32:0/1761884491 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fab9c06e930 0x7fab9c139d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:12.209 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:12.208+0000 7faba2a00640 1 -- 172.21.15.32:0/1761884491 >> 172.21.15.32:0/1761884491 conn(0x7fab9c0b1bd0 msgr2=0x7fab9c15ff80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:57:12.209 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:12.208+0000 7faba2a00640 1 -- 172.21.15.32:0/1761884491 shutdown_connections 2024-08-03T20:57:12.209 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:12.208+0000 7faba2a00640 1 -- 172.21.15.32:0/1761884491 wait complete. 2024-08-03T20:57:12.236 DEBUG:teuthology.orchestra.run.smithi032:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata import - 2024-08-03T20:57:12.380 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:12.379+0000 7f9b99ffef40 1 Processor -- start 2024-08-03T20:57:12.380 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:12.379+0000 7f9b99ffef40 1 -- start start 2024-08-03T20:57:12.382 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:12.381+0000 7f9b99ffef40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f9b78002340 0x55a43714c3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:12.382 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:12.381+0000 7f9b99ffef40 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55a4370c3b30 0x55a4370c58e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:12.382 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:12.381+0000 7f9b99ffef40 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a43714de30 0x55a4370c4260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:12.383 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:12.381+0000 7f9b99ffef40 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55a437028bf0 con 0x55a43714de30 2024-08-03T20:57:12.383 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:12.381+0000 7f9b99ffef40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x55a4370356b0 con 0x7f9b78002340 2024-08-03T20:57:12.383 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:12.381+0000 7f9b99ffef40 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x55a436fcae30 con 0x55a4370c3b30 2024-08-03T20:57:12.383 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:12.381+0000 7f9b93c00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a43714de30 0x55a4370c4260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:12.383 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:12.381+0000 7f9b93c00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a43714de30 0x55a4370c4260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:33894/0 (socket says 172.21.15.32:33894) 2024-08-03T20:57:12.383 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:12.381+0000 7f9b93c00640 1 -- 172.21.15.32:0/206176298 learned_addr learned my addr 172.21.15.32:0/206176298 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:57:12.383 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:12.381+0000 7f9b92800640 1 --2- 172.21.15.32:0/206176298 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55a4370c3b30 0x55a4370c58e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:12.383 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:12.381+0000 7f9b93200640 1 --2- 172.21.15.32:0/206176298 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f9b78002340 0x55a43714c3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:12.383 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:12.381+0000 7f9b93c00640 1 -- 172.21.15.32:0/206176298 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55a4370c3b30 msgr2=0x55a4370c58e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:57:12.383 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:12.381+0000 7f9b93c00640 1 --2- 172.21.15.32:0/206176298 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55a4370c3b30 0x55a4370c58e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:12.383 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:12.381+0000 7f9b93c00640 1 -- 172.21.15.32:0/206176298 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f9b78002340 msgr2=0x55a43714c3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:57:12.383 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:12.381+0000 7f9b93c00640 1 --2- 172.21.15.32:0/206176298 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f9b78002340 0x55a43714c3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:12.383 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:12.381+0000 7f9b93c00640 1 -- 172.21.15.32:0/206176298 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a4370c56d0 con 0x55a43714de30 2024-08-03T20:57:12.384 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:12.381+0000 7f9b92800640 1 --2- 172.21.15.32:0/206176298 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55a4370c3b30 0x55a4370c58e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-03T20:57:12.384 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:12.381+0000 7f9b93200640 1 --2- 172.21.15.32:0/206176298 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f9b78002340 0x55a43714c3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-03T20:57:12.384 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:12.382+0000 7f9b93c00640 1 --2- 172.21.15.32:0/206176298 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a43714de30 0x55a4370c4260 secure :-1 s=READY pgs=1507 cs=0 l=1 rev1=1 crypto rx=0x7f9b7800ab40 tx=0x7f9b8c010c80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=317113ed27e2bf85 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:57:12.384 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:12.382+0000 7f9b91400640 1 -- 172.21.15.32:0/206176298 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b8c0102f0 con 0x55a43714de30 2024-08-03T20:57:12.384 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:12.382+0000 7f9b91400640 1 -- 172.21.15.32:0/206176298 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f9b8c00b930 con 0x55a43714de30 2024-08-03T20:57:12.384 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:12.382+0000 7f9b91400640 1 -- 172.21.15.32:0/206176298 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b8c00bd80 con 0x55a43714de30 2024-08-03T20:57:12.385 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:12.382+0000 7f9b99ffef40 1 -- 172.21.15.32:0/206176298 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a43714de30 msgr2=0x55a4370c4260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:12.385 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:12.382+0000 7f9b99ffef40 1 --2- 172.21.15.32:0/206176298 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a43714de30 0x55a4370c4260 secure :-1 s=READY pgs=1507 cs=0 l=1 rev1=1 crypto rx=0x7f9b7800ab40 tx=0x7f9b8c010c80 comp rx=0 tx=0).stop 2024-08-03T20:57:12.385 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:12.383+0000 7f9b99ffef40 1 -- 172.21.15.32:0/206176298 shutdown_connections 2024-08-03T20:57:12.385 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:12.383+0000 7f9b99ffef40 1 --2- 172.21.15.32:0/206176298 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a43714de30 0x55a4370c4260 unknown :-1 s=CLOSED pgs=1507 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:12.385 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:12.383+0000 7f9b99ffef40 1 --2- 172.21.15.32:0/206176298 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55a4370c3b30 0x55a4370c58e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:12.385 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:12.383+0000 7f9b99ffef40 1 --2- 172.21.15.32:0/206176298 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f9b78002340 0x55a43714c3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:12.385 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:12.383+0000 7f9b99ffef40 1 -- 172.21.15.32:0/206176298 >> 172.21.15.32:0/206176298 conn(0x55a436f9cac0 msgr2=0x55a4371529b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:57:12.385 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:12.383+0000 7f9b99ffef40 1 -- 172.21.15.32:0/206176298 shutdown_connections 2024-08-03T20:57:12.385 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:12.383+0000 7f9b99ffef40 1 -- 172.21.15.32:0/206176298 wait complete. 2024-08-03T20:57:12.386 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:12.384+0000 7f9b99ffef40 1 Processor -- start 2024-08-03T20:57:12.386 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:12.384+0000 7f9b99ffef40 1 -- start start 2024-08-03T20:57:12.387 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:12.385+0000 7f9b99ffef40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55a4370c3b30 0x55a4371ddfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:12.388 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:12.385+0000 7f9b99ffef40 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a43714de30 0x55a43707ada0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:12.388 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:12.385+0000 7f9b99ffef40 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f9b78002340 0x55a43707b2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:12.388 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:12.385+0000 7f9b99ffef40 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55a43714fc40 con 0x55a43714de30 2024-08-03T20:57:12.388 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:12.385+0000 7f9b99ffef40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x55a43714ac90 con 0x55a4370c3b30 2024-08-03T20:57:12.388 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:12.385+0000 7f9b99ffef40 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x55a4371dd6b0 con 0x7f9b78002340 2024-08-03T20:57:12.388 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:12.385+0000 7f9b92800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a43714de30 0x55a43707ada0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:12.388 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:12.386+0000 7f9b92800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a43714de30 0x55a43707ada0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:33908/0 (socket says 172.21.15.32:33908) 2024-08-03T20:57:12.388 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:12.386+0000 7f9b92800640 1 -- 172.21.15.32:0/3821133758 learned_addr learned my addr 172.21.15.32:0/3821133758 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:57:12.388 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:12.386+0000 7f9b93c00640 1 --2- 172.21.15.32:0/3821133758 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f9b78002340 0x55a43707b2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:12.388 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:12.386+0000 7f9b93200640 1 --2- 172.21.15.32:0/3821133758 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55a4370c3b30 0x55a4371ddfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:12.388 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:12.386+0000 7f9b92800640 1 -- 172.21.15.32:0/3821133758 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f9b78002340 msgr2=0x55a43707b2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:12.388 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:12.386+0000 7f9b92800640 1 --2- 172.21.15.32:0/3821133758 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f9b78002340 0x55a43707b2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:12.388 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:12.386+0000 7f9b92800640 1 -- 172.21.15.32:0/3821133758 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55a4370c3b30 msgr2=0x55a4371ddfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:12.389 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:12.386+0000 7f9b92800640 1 --2- 172.21.15.32:0/3821133758 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55a4370c3b30 0x55a4371ddfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:12.389 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:12.386+0000 7f9b92800640 1 -- 172.21.15.32:0/3821133758 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a4370bfcb0 con 0x55a43714de30 2024-08-03T20:57:12.389 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:12.386+0000 7f9b92800640 1 --2- 172.21.15.32:0/3821133758 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a43714de30 0x55a43707ada0 secure :-1 s=READY pgs=1508 cs=0 l=1 rev1=1 crypto rx=0x7f9b80002d10 tx=0x7f9b8000c6b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:57:12.389 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:12.387+0000 7f9b8be00640 1 -- 172.21.15.32:0/3821133758 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b80010640 con 0x55a43714de30 2024-08-03T20:57:12.389 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:12.387+0000 7f9b8be00640 1 -- 172.21.15.32:0/3821133758 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f9b800108d0 con 0x55a43714de30 2024-08-03T20:57:12.389 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:12.387+0000 7f9b8be00640 1 -- 172.21.15.32:0/3821133758 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b80010ca0 con 0x55a43714de30 2024-08-03T20:57:12.389 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:12.387+0000 7f9b99ffef40 1 -- 172.21.15.32:0/3821133758 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x55a4370c5ab0 con 0x55a43714de30 2024-08-03T20:57:12.389 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:12.387+0000 7f9b99ffef40 1 -- 172.21.15.32:0/3821133758 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a4371de4f0 con 0x55a43714de30 2024-08-03T20:57:12.389 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:12.389+0000 7f9b8be00640 1 -- 172.21.15.32:0/3821133758 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f9b80018400 con 0x55a43714de30 2024-08-03T20:57:12.390 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:12.390+0000 7f9b8be00640 1 --2- 172.21.15.32:0/3821133758 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f9b740759f0 0x7f9b74077f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:12.390 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:12.390+0000 7f9b8be00640 1 -- 172.21.15.32:0/3821133758 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(103..103 src has 1..103) ==== 7402+0+0 (secure 0 0 0) 0x7f9b800980f0 con 0x55a43714de30 2024-08-03T20:57:12.391 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:12.390+0000 7f9b93200640 1 --2- 172.21.15.32:0/3821133758 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f9b740759f0 0x7f9b74077f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:12.391 INFO:teuthology.orchestra.run.smithi032.stdout:Importing pool 2024-08-03T20:57:12.392 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:12.391+0000 7f9b99ffef40 1 --2- 172.21.15.32:0/3821133758 >> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] conn(0x55a4371298a0 0x55a4371dcde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:12.392 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:12.391+0000 7f9b99ffef40 1 -- 172.21.15.32:0/3821133758 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:1 12.10 12:09401a81:::601.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e103) -- 0x55a43707bcb0 con 0x55a4371298a0 2024-08-03T20:57:12.392 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:12.391+0000 7f9b93c00640 1 --2- 172.21.15.32:0/3821133758 >> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] conn(0x55a4371298a0 0x55a4371dcde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:12.392 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:12.391+0000 7f9b93200640 1 --2- 172.21.15.32:0/3821133758 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f9b740759f0 0x7f9b74077f00 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7f9b84056000 tx=0x7f9b84001a60 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:57:12.392 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:12.392+0000 7f9b93c00640 1 --2- 172.21.15.32:0/3821133758 >> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] conn(0x55a4371298a0 0x55a4371dcde0 crc :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:57:12.394 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:12.394+0000 7f9b93c00640 1 -- 172.21.15.32:0/3821133758 <== osd.0 v2:172.21.15.32:6816/3066156116 1 ==== osd_op_reply(1 601.00000000 [create] v103'536 uv510 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f9b8c003040 con 0x55a4371298a0 2024-08-03T20:57:12.394 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:12.394+0000 7f9b99ffef40 1 -- 172.21.15.32:0/3821133758 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:2 12.10 12:09401a81:::601.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e103) -- 0x55a4371f6560 con 0x55a4371298a0 2024-08-03T20:57:12.396 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:12.396+0000 7f9b93c00640 1 -- 172.21.15.32:0/3821133758 <== osd.0 v2:172.21.15.32:6816/3066156116 2 ==== osd_op_reply(2 601.00000000 [delete] v103'537 uv537 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f9b8c003040 con 0x55a4371298a0 2024-08-03T20:57:12.396 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:12.396+0000 7f9b99ffef40 1 -- 172.21.15.32:0/3821133758 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:3 12.10 12:09401a81:::601.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e103) -- 0x55a4371f6900 con 0x55a4371298a0 2024-08-03T20:57:12.397 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::601.00000000:head# 2024-08-03T20:57:12.398 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:12.397+0000 7f9b93c00640 1 -- 172.21.15.32:0/3821133758 <== osd.0 v2:172.21.15.32:6816/3066156116 3 ==== osd_op_reply(3 601.00000000 [create] v103'538 uv538 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f9b8c003040 con 0x55a4371298a0 2024-08-03T20:57:12.398 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:12.397+0000 7f9b99ffef40 1 -- 172.21.15.32:0/3821133758 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:4 12.10 12:09401a81:::601.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e103) -- 0x55a4371f7cb0 con 0x55a4371298a0 2024-08-03T20:57:12.399 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:12.399+0000 7f9b93c00640 1 -- 172.21.15.32:0/3821133758 <== osd.0 v2:172.21.15.32:6816/3066156116 4 ==== osd_op_reply(4 601.00000000 [setxattr (30)] v103'539 uv539 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f9b8c003040 con 0x55a4371298a0 2024-08-03T20:57:12.399 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:12.399+0000 7f9b99ffef40 1 -- 172.21.15.32:0/3821133758 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:5 12.10 12:09401a81:::601.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e103) -- 0x55a4371f8050 con 0x55a4371298a0 2024-08-03T20:57:12.401 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:12.401+0000 7f9b93c00640 1 -- 172.21.15.32:0/3821133758 <== osd.0 v2:172.21.15.32:6816/3066156116 5 ==== osd_op_reply(5 601.00000000 [setxattr (62)] v103'540 uv540 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f9b8c003040 con 0x55a4371298a0 2024-08-03T20:57:12.401 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:12.401+0000 7f9b99ffef40 1 -- 172.21.15.32:0/3821133758 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:6 12.10 12:09401a81:::601.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e103) -- 0x55a4371f6ca0 con 0x55a4371298a0 2024-08-03T20:57:12.403 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:12.402+0000 7f9b93c00640 1 -- 172.21.15.32:0/3821133758 <== osd.0 v2:172.21.15.32:6816/3066156116 6 ==== osd_op_reply(6 601.00000000 [omap-set-header] v103'541 uv541 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f9b8c003040 con 0x55a4371298a0 2024-08-03T20:57:12.403 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:12.402+0000 7f9b99ffef40 1 -- 172.21.15.32:0/3821133758 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:7 12.10 12:0bd6d154:::602.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e103) -- 0x55a4371f7040 con 0x55a4371298a0 2024-08-03T20:57:12.404 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:12.404+0000 7f9b93c00640 1 -- 172.21.15.32:0/3821133758 <== osd.0 v2:172.21.15.32:6816/3066156116 7 ==== osd_op_reply(7 602.00000000 [create] v103'542 uv516 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f9b8c003040 con 0x55a4371298a0 2024-08-03T20:57:12.404 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:12.404+0000 7f9b99ffef40 1 -- 172.21.15.32:0/3821133758 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:8 12.10 12:0bd6d154:::602.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e103) -- 0x55a4371f7040 con 0x55a4371298a0 2024-08-03T20:57:12.406 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:12.406+0000 7f9b93c00640 1 -- 172.21.15.32:0/3821133758 <== osd.0 v2:172.21.15.32:6816/3066156116 8 ==== osd_op_reply(8 602.00000000 [delete] v103'543 uv543 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f9b8c003040 con 0x55a4371298a0 2024-08-03T20:57:12.406 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:12.406+0000 7f9b99ffef40 1 -- 172.21.15.32:0/3821133758 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:9 12.10 12:0bd6d154:::602.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e103) -- 0x55a4371f7040 con 0x55a4371298a0 2024-08-03T20:57:12.408 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::602.00000000:head# 2024-08-03T20:57:12.408 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:12.407+0000 7f9b93c00640 1 -- 172.21.15.32:0/3821133758 <== osd.0 v2:172.21.15.32:6816/3066156116 9 ==== osd_op_reply(9 602.00000000 [create] v103'544 uv544 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f9b8c003040 con 0x55a4371298a0 2024-08-03T20:57:12.408 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:12.407+0000 7f9b99ffef40 1 -- 172.21.15.32:0/3821133758 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:10 12.10 12:0bd6d154:::602.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e103) -- 0x55a4371f7040 con 0x55a4371298a0 2024-08-03T20:57:12.409 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:12.409+0000 7f9b93c00640 1 -- 172.21.15.32:0/3821133758 <== osd.0 v2:172.21.15.32:6816/3066156116 10 ==== osd_op_reply(10 602.00000000 [setxattr (30)] v103'545 uv545 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f9b8c003040 con 0x55a4371298a0 2024-08-03T20:57:12.409 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:12.409+0000 7f9b99ffef40 1 -- 172.21.15.32:0/3821133758 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:11 12.10 12:0bd6d154:::602.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e103) -- 0x55a4371f7040 con 0x55a4371298a0 2024-08-03T20:57:12.410 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:12.410+0000 7f9b93c00640 1 -- 172.21.15.32:0/3821133758 <== osd.0 v2:172.21.15.32:6816/3066156116 11 ==== osd_op_reply(11 602.00000000 [setxattr (62)] v103'546 uv546 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f9b8c003040 con 0x55a4371298a0 2024-08-03T20:57:12.411 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:12.410+0000 7f9b99ffef40 1 -- 172.21.15.32:0/3821133758 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:12 12.10 12:0bd6d154:::602.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e103) -- 0x55a4371f7040 con 0x55a4371298a0 2024-08-03T20:57:12.412 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:12.412+0000 7f9b93c00640 1 -- 172.21.15.32:0/3821133758 <== osd.0 v2:172.21.15.32:6816/3066156116 12 ==== osd_op_reply(12 602.00000000 [omap-set-header] v103'547 uv547 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f9b8c003040 con 0x55a4371298a0 2024-08-03T20:57:12.412 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:12.412+0000 7f9b99ffef40 1 -- 172.21.15.32:0/3821133758 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:13 12.10 12:0cf6ad0f:::10000000000.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e103) -- 0x55a4371f7040 con 0x55a4371298a0 2024-08-03T20:57:12.414 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:12.413+0000 7f9b93c00640 1 -- 172.21.15.32:0/3821133758 <== osd.0 v2:172.21.15.32:6816/3066156116 13 ==== osd_op_reply(13 10000000000.00000000 [create] v103'548 uv523 ondisk = -17 ((17) File exists)) ==== 164+0+0 (crc 0 0 0) 0x7f9b8c003040 con 0x55a4371298a0 2024-08-03T20:57:12.414 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:12.413+0000 7f9b99ffef40 1 -- 172.21.15.32:0/3821133758 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:14 12.10 12:0cf6ad0f:::10000000000.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e103) -- 0x55a4371f7040 con 0x55a4371298a0 2024-08-03T20:57:12.415 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:12.415+0000 7f9b93c00640 1 -- 172.21.15.32:0/3821133758 <== osd.0 v2:172.21.15.32:6816/3066156116 14 ==== osd_op_reply(14 10000000000.00000000 [delete] v103'549 uv549 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f9b8c003040 con 0x55a4371298a0 2024-08-03T20:57:12.415 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:12.415+0000 7f9b99ffef40 1 -- 172.21.15.32:0/3821133758 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:15 12.10 12:0cf6ad0f:::10000000000.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e103) -- 0x55a4371f7040 con 0x55a4371298a0 2024-08-03T20:57:12.417 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::10000000000.00000000:head# 2024-08-03T20:57:12.417 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:12.417+0000 7f9b93c00640 1 -- 172.21.15.32:0/3821133758 <== osd.0 v2:172.21.15.32:6816/3066156116 15 ==== osd_op_reply(15 10000000000.00000000 [create] v103'550 uv550 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f9b8c003040 con 0x55a4371298a0 2024-08-03T20:57:12.417 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:12.417+0000 7f9b99ffef40 1 -- 172.21.15.32:0/3821133758 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:16 12.10 12:0cf6ad0f:::10000000000.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e103) -- 0x55a4371f7040 con 0x55a4371298a0 2024-08-03T20:57:12.419 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:12.418+0000 7f9b93c00640 1 -- 172.21.15.32:0/3821133758 <== osd.0 v2:172.21.15.32:6816/3066156116 16 ==== osd_op_reply(16 10000000000.00000000 [setxattr (30)] v103'551 uv551 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f9b8c003040 con 0x55a4371298a0 2024-08-03T20:57:12.419 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:12.418+0000 7f9b99ffef40 1 -- 172.21.15.32:0/3821133758 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:17 12.10 12:0cf6ad0f:::10000000000.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e103) -- 0x55a4371f7040 con 0x55a4371298a0 2024-08-03T20:57:12.420 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:12.420+0000 7f9b93c00640 1 -- 172.21.15.32:0/3821133758 <== osd.0 v2:172.21.15.32:6816/3066156116 17 ==== osd_op_reply(17 10000000000.00000000 [setxattr (62)] v103'552 uv552 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f9b8c003040 con 0x55a4371298a0 2024-08-03T20:57:12.420 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:12.420+0000 7f9b99ffef40 1 -- 172.21.15.32:0/3821133758 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:18 12.10 12:0cf6ad0f:::10000000000.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e103) -- 0x55a4371f7040 con 0x55a4371298a0 2024-08-03T20:57:12.422 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:12.421+0000 7f9b93c00640 1 -- 172.21.15.32:0/3821133758 <== osd.0 v2:172.21.15.32:6816/3066156116 18 ==== osd_op_reply(18 10000000000.00000000 [omap-set-header] v103'553 uv553 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f9b8c003040 con 0x55a4371298a0 2024-08-03T20:57:12.422 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:12.421+0000 7f9b99ffef40 1 -- 172.21.15.32:0/3821133758 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:19 12.10 12:0cf6ad0f:::10000000000.00000000:head [omap-set-vals in=593b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e103) -- 0x55a4371f7420 con 0x55a4371298a0 2024-08-03T20:57:12.423 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:12.423+0000 7f9b93c00640 1 -- 172.21.15.32:0/3821133758 <== osd.0 v2:172.21.15.32:6816/3066156116 19 ==== osd_op_reply(19 10000000000.00000000 [omap-set-vals] v103'554 uv554 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f9b8c003040 con 0x55a4371298a0 2024-08-03T20:57:12.424 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:12.423+0000 7f9b99ffef40 1 -- 172.21.15.32:0/3821133758 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:20 12.10 12:0d82a743:::600.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e103) -- 0x55a4371f7420 con 0x55a4371298a0 2024-08-03T20:57:12.425 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:12.425+0000 7f9b93c00640 1 -- 172.21.15.32:0/3821133758 <== osd.0 v2:172.21.15.32:6816/3066156116 20 ==== osd_op_reply(20 600.00000000 [create] v103'555 uv529 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f9b8c003040 con 0x55a4371298a0 2024-08-03T20:57:12.425 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:12.425+0000 7f9b99ffef40 1 -- 172.21.15.32:0/3821133758 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:21 12.10 12:0d82a743:::600.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e103) -- 0x55a4371f7400 con 0x55a4371298a0 2024-08-03T20:57:12.427 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:12.426+0000 7f9b93c00640 1 -- 172.21.15.32:0/3821133758 <== osd.0 v2:172.21.15.32:6816/3066156116 21 ==== osd_op_reply(21 600.00000000 [delete] v103'556 uv556 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f9b8c003040 con 0x55a4371298a0 2024-08-03T20:57:12.427 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:12.426+0000 7f9b99ffef40 1 -- 172.21.15.32:0/3821133758 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:22 12.10 12:0d82a743:::600.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e103) -- 0x55a4371f7400 con 0x55a4371298a0 2024-08-03T20:57:12.428 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::600.00000000:head# 2024-08-03T20:57:12.429 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:12.428+0000 7f9b93c00640 1 -- 172.21.15.32:0/3821133758 <== osd.0 v2:172.21.15.32:6816/3066156116 22 ==== osd_op_reply(22 600.00000000 [create] v103'557 uv557 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f9b8c003040 con 0x55a4371298a0 2024-08-03T20:57:12.429 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:12.428+0000 7f9b99ffef40 1 -- 172.21.15.32:0/3821133758 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:23 12.10 12:0d82a743:::600.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e103) -- 0x55a4371f7400 con 0x55a4371298a0 2024-08-03T20:57:12.430 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:12.429+0000 7f9b93c00640 1 -- 172.21.15.32:0/3821133758 <== osd.0 v2:172.21.15.32:6816/3066156116 23 ==== osd_op_reply(23 600.00000000 [setxattr (30)] v103'558 uv558 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f9b8c003040 con 0x55a4371298a0 2024-08-03T20:57:12.430 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:12.430+0000 7f9b99ffef40 1 -- 172.21.15.32:0/3821133758 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:24 12.10 12:0d82a743:::600.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e103) -- 0x55a4371f7300 con 0x55a4371298a0 2024-08-03T20:57:12.431 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:12.431+0000 7f9b93c00640 1 -- 172.21.15.32:0/3821133758 <== osd.0 v2:172.21.15.32:6816/3066156116 24 ==== osd_op_reply(24 600.00000000 [setxattr (62)] v103'559 uv559 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f9b8c003040 con 0x55a4371298a0 2024-08-03T20:57:12.431 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:12.431+0000 7f9b99ffef40 1 -- 172.21.15.32:0/3821133758 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:25 12.10 12:0d82a743:::600.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e103) -- 0x55a4371f7300 con 0x55a4371298a0 2024-08-03T20:57:12.433 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:12.432+0000 7f9b93c00640 1 -- 172.21.15.32:0/3821133758 <== osd.0 v2:172.21.15.32:6816/3066156116 25 ==== osd_op_reply(25 600.00000000 [omap-set-header] v103'560 uv560 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f9b8c003040 con 0x55a4371298a0 2024-08-03T20:57:12.433 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:12.432+0000 7f9b99ffef40 1 -- 172.21.15.32:0/3821133758 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:26 12.10 12:0d89b25e:::603.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e103) -- 0x55a4371f7300 con 0x55a4371298a0 2024-08-03T20:57:12.434 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:12.434+0000 7f9b93c00640 1 -- 172.21.15.32:0/3821133758 <== osd.0 v2:172.21.15.32:6816/3066156116 26 ==== osd_op_reply(26 603.00000000 [create] v103'561 uv535 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f9b8c003040 con 0x55a4371298a0 2024-08-03T20:57:12.434 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:12.434+0000 7f9b99ffef40 1 -- 172.21.15.32:0/3821133758 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:27 12.10 12:0d89b25e:::603.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e103) -- 0x55a4371f7300 con 0x55a4371298a0 2024-08-03T20:57:12.436 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:12.436+0000 7f9b93c00640 1 -- 172.21.15.32:0/3821133758 <== osd.0 v2:172.21.15.32:6816/3066156116 27 ==== osd_op_reply(27 603.00000000 [delete] v103'562 uv562 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f9b8c003040 con 0x55a4371298a0 2024-08-03T20:57:12.436 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:12.436+0000 7f9b99ffef40 1 -- 172.21.15.32:0/3821133758 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:28 12.10 12:0d89b25e:::603.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e103) -- 0x55a4371f7300 con 0x55a4371298a0 2024-08-03T20:57:12.438 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::603.00000000:head# 2024-08-03T20:57:12.438 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:12.437+0000 7f9b93c00640 1 -- 172.21.15.32:0/3821133758 <== osd.0 v2:172.21.15.32:6816/3066156116 28 ==== osd_op_reply(28 603.00000000 [create] v103'563 uv563 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f9b8c003040 con 0x55a4371298a0 2024-08-03T20:57:12.438 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:12.437+0000 7f9b99ffef40 1 -- 172.21.15.32:0/3821133758 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:29 12.10 12:0d89b25e:::603.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e103) -- 0x55a4371f7300 con 0x55a4371298a0 2024-08-03T20:57:12.439 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:12.439+0000 7f9b93c00640 1 -- 172.21.15.32:0/3821133758 <== osd.0 v2:172.21.15.32:6816/3066156116 29 ==== osd_op_reply(29 603.00000000 [setxattr (30)] v103'564 uv564 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f9b8c003040 con 0x55a4371298a0 2024-08-03T20:57:12.439 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:12.439+0000 7f9b99ffef40 1 -- 172.21.15.32:0/3821133758 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:30 12.10 12:0d89b25e:::603.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e103) -- 0x55a4371f7300 con 0x55a4371298a0 2024-08-03T20:57:12.441 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:12.441+0000 7f9b93c00640 1 -- 172.21.15.32:0/3821133758 <== osd.0 v2:172.21.15.32:6816/3066156116 30 ==== osd_op_reply(30 603.00000000 [setxattr (62)] v103'565 uv565 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f9b8c003040 con 0x55a4371298a0 2024-08-03T20:57:12.441 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:12.441+0000 7f9b99ffef40 1 -- 172.21.15.32:0/3821133758 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:31 12.10 12:0d89b25e:::603.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e103) -- 0x55a4371f7300 con 0x55a4371298a0 2024-08-03T20:57:12.442 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:12.442+0000 7f9b93c00640 1 -- 172.21.15.32:0/3821133758 <== osd.0 v2:172.21.15.32:6816/3066156116 31 ==== osd_op_reply(31 603.00000000 [omap-set-header] v103'566 uv566 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f9b8c003040 con 0x55a4371298a0 2024-08-03T20:57:12.443 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:12.442+0000 7f9b99ffef40 1 --2- 172.21.15.32:0/3821133758 >> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] conn(0x55a4371f83f0 0x55a4371fa810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:12.443 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:12.442+0000 7f9b99ffef40 1 -- 172.21.15.32:0/3821133758 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:32 12.14 12:287034c4:::1.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e103) -- 0x55a4371f7300 con 0x55a4371f83f0 2024-08-03T20:57:12.443 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:12.442+0000 7f9b93200640 1 --2- 172.21.15.32:0/3821133758 >> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] conn(0x55a4371f83f0 0x55a4371fa810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:12.443 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:12.443+0000 7f9b93200640 1 --2- 172.21.15.32:0/3821133758 >> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] conn(0x55a4371f83f0 0x55a4371fa810 crc :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:57:12.445 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:12.445+0000 7f9b93200640 1 -- 172.21.15.32:0/3821133758 <== osd.5 v2:172.21.15.118:6810/2724891770 1 ==== osd_op_reply(32 1.00000000.inode [create] v103'396 uv378 ondisk = -17 ((17) File exists)) ==== 160+0+0 (crc 0 0 0) 0x7f9b8404d980 con 0x55a4371f83f0 2024-08-03T20:57:12.445 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:12.445+0000 7f9b99ffef40 1 -- 172.21.15.32:0/3821133758 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:33 12.14 12:287034c4:::1.00000000.inode:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e103) -- 0x55a4371fad50 con 0x55a4371f83f0 2024-08-03T20:57:12.447 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:12.446+0000 7f9b93200640 1 -- 172.21.15.32:0/3821133758 <== osd.5 v2:172.21.15.118:6810/2724891770 2 ==== osd_op_reply(33 1.00000000.inode [delete] v103'397 uv397 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f9b8404d980 con 0x55a4371f83f0 2024-08-03T20:57:12.447 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:12.446+0000 7f9b99ffef40 1 -- 172.21.15.32:0/3821133758 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:34 12.14 12:287034c4:::1.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e103) -- 0x55a4371fb0f0 con 0x55a4371f83f0 2024-08-03T20:57:12.448 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:12.448+0000 7f9b93200640 1 -- 172.21.15.32:0/3821133758 <== osd.5 v2:172.21.15.118:6810/2724891770 3 ==== osd_op_reply(34 1.00000000.inode [create] v103'398 uv398 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f9b8404d980 con 0x55a4371f83f0 2024-08-03T20:57:12.448 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::1.00000000.inode:head# 2024-08-03T20:57:12.449 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:12.448+0000 7f9b99ffef40 1 -- 172.21.15.32:0/3821133758 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:35 12.14 12:287034c4:::1.00000000.inode:head [write 0~582 in=582b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e103) -- 0x55a4371fb490 con 0x55a4371f83f0 2024-08-03T20:57:12.450 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:12.450+0000 7f9b93200640 1 -- 172.21.15.32:0/3821133758 <== osd.5 v2:172.21.15.118:6810/2724891770 4 ==== osd_op_reply(35 1.00000000.inode [write 0~582] v103'399 uv399 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f9b8404d980 con 0x55a4371f83f0 2024-08-03T20:57:12.450 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:12.450+0000 7f9b99ffef40 1 -- 172.21.15.32:0/3821133758 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:36 12.14 12:287034c4:::1.00000000.inode:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e103) -- 0x55a4371fb830 con 0x55a4371f83f0 2024-08-03T20:57:12.452 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:12.451+0000 7f9b93200640 1 -- 172.21.15.32:0/3821133758 <== osd.5 v2:172.21.15.118:6810/2724891770 5 ==== osd_op_reply(36 1.00000000.inode [omap-set-header] v103'400 uv400 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f9b8404d980 con 0x55a4371f83f0 2024-08-03T20:57:12.452 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:12.451+0000 7f9b99ffef40 1 -- 172.21.15.32:0/3821133758 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:37 12.14 12:292cf221:::200.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e103) -- 0x55a4371fbbd0 con 0x55a4371f83f0 2024-08-03T20:57:12.454 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:12.453+0000 7f9b93200640 1 -- 172.21.15.32:0/3821133758 <== osd.5 v2:172.21.15.118:6810/2724891770 6 ==== osd_op_reply(37 200.00000000 [create] v103'401 uv383 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f9b8404d980 con 0x55a4371f83f0 2024-08-03T20:57:12.454 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:12.453+0000 7f9b99ffef40 1 -- 172.21.15.32:0/3821133758 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:38 12.14 12:292cf221:::200.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e103) -- 0x55a4371fbf70 con 0x55a4371f83f0 2024-08-03T20:57:12.455 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:12.454+0000 7f9b93200640 1 -- 172.21.15.32:0/3821133758 <== osd.5 v2:172.21.15.118:6810/2724891770 7 ==== osd_op_reply(38 200.00000000 [delete] v103'402 uv402 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f9b8404d980 con 0x55a4371f83f0 2024-08-03T20:57:12.455 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:12.454+0000 7f9b99ffef40 1 -- 172.21.15.32:0/3821133758 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:39 12.14 12:292cf221:::200.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e103) -- 0x55a4371fc310 con 0x55a4371f83f0 2024-08-03T20:57:12.456 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:12.455+0000 7f9b93200640 1 -- 172.21.15.32:0/3821133758 <== osd.5 v2:172.21.15.118:6810/2724891770 8 ==== osd_op_reply(39 200.00000000 [create] v103'403 uv403 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f9b8404d980 con 0x55a4371f83f0 2024-08-03T20:57:12.456 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::200.00000000:head# 2024-08-03T20:57:12.456 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:12.456+0000 7f9b99ffef40 1 -- 172.21.15.32:0/3821133758 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:40 12.14 12:292cf221:::200.00000000:head [write 0~90 in=90b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e103) -- 0x55a4371fc310 con 0x55a4371f83f0 2024-08-03T20:57:12.457 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:12.457+0000 7f9b93200640 1 -- 172.21.15.32:0/3821133758 <== osd.5 v2:172.21.15.118:6810/2724891770 9 ==== osd_op_reply(40 200.00000000 [write 0~90] v103'404 uv404 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f9b8404d980 con 0x55a4371f83f0 2024-08-03T20:57:12.458 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:12.457+0000 7f9b99ffef40 1 -- 172.21.15.32:0/3821133758 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:41 12.14 12:292cf221:::200.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e103) -- 0x55a4371fc310 con 0x55a4371f83f0 2024-08-03T20:57:12.459 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:12.458+0000 7f9b93200640 1 -- 172.21.15.32:0/3821133758 <== osd.5 v2:172.21.15.118:6810/2724891770 10 ==== osd_op_reply(41 200.00000000 [omap-set-header] v103'405 uv405 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f9b8404d980 con 0x55a4371f83f0 2024-08-03T20:57:12.459 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:12.458+0000 7f9b99ffef40 1 -- 172.21.15.32:0/3821133758 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:42 12.14 12:2e2fa760:::200.00000001:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e103) -- 0x55a4371fc310 con 0x55a4371f83f0 2024-08-03T20:57:12.460 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:12.460+0000 7f9b93200640 1 -- 172.21.15.32:0/3821133758 <== osd.5 v2:172.21.15.118:6810/2724891770 11 ==== osd_op_reply(42 200.00000001 [create] v103'406 uv395 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f9b8404d980 con 0x55a4371f83f0 2024-08-03T20:57:12.460 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:12.460+0000 7f9b99ffef40 1 -- 172.21.15.32:0/3821133758 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:43 12.14 12:2e2fa760:::200.00000001:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e103) -- 0x55a4371fc310 con 0x55a4371f83f0 2024-08-03T20:57:12.461 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:12.461+0000 7f9b93200640 1 -- 172.21.15.32:0/3821133758 <== osd.5 v2:172.21.15.118:6810/2724891770 12 ==== osd_op_reply(43 200.00000001 [delete] v103'407 uv407 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f9b8404d980 con 0x55a4371f83f0 2024-08-03T20:57:12.461 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:12.461+0000 7f9b99ffef40 1 -- 172.21.15.32:0/3821133758 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:44 12.14 12:2e2fa760:::200.00000001:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e103) -- 0x55a4371fc310 con 0x55a4371f83f0 2024-08-03T20:57:12.463 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::200.00000001:head# 2024-08-03T20:57:12.463 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:12.462+0000 7f9b93200640 1 -- 172.21.15.32:0/3821133758 <== osd.5 v2:172.21.15.118:6810/2724891770 13 ==== osd_op_reply(44 200.00000001 [create] v103'408 uv408 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f9b8404d980 con 0x55a4371f83f0 2024-08-03T20:57:12.463 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:12.462+0000 7f9b99ffef40 1 -- 172.21.15.32:0/3821133758 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:45 12.14 12:2e2fa760:::200.00000001:head [write 0~18118 in=18118b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e103) -- 0x55a437200a00 con 0x55a4371f83f0 2024-08-03T20:57:12.465 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:12.465+0000 7f9b93200640 1 -- 172.21.15.32:0/3821133758 <== osd.5 v2:172.21.15.118:6810/2724891770 14 ==== osd_op_reply(45 200.00000001 [write 0~18118] v103'409 uv409 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f9b8404d980 con 0x55a4371f83f0 2024-08-03T20:57:12.465 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:12.465+0000 7f9b99ffef40 1 -- 172.21.15.32:0/3821133758 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:46 12.14 12:2e2fa760:::200.00000001:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e103) -- 0x55a437200a00 con 0x55a4371f83f0 2024-08-03T20:57:12.466 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:12.466+0000 7f9b93200640 1 -- 172.21.15.32:0/3821133758 <== osd.5 v2:172.21.15.118:6810/2724891770 15 ==== osd_op_reply(46 200.00000001 [omap-set-header] v103'410 uv410 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f9b8404d980 con 0x55a4371f83f0 2024-08-03T20:57:12.466 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:12.466+0000 7f9b99ffef40 1 -- 172.21.15.32:0/3821133758 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:47 12.14 12:2f55791f:::606.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e103) -- 0x55a4371fc310 con 0x55a4371f83f0 2024-08-03T20:57:12.468 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:12.467+0000 7f9b93200640 1 -- 172.21.15.32:0/3821133758 <== osd.5 v2:172.21.15.118:6810/2724891770 16 ==== osd_op_reply(47 606.00000000 [create] v103'411 uv394 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f9b8404d980 con 0x55a4371f83f0 2024-08-03T20:57:12.468 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:12.467+0000 7f9b99ffef40 1 -- 172.21.15.32:0/3821133758 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:48 12.14 12:2f55791f:::606.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e103) -- 0x55a4371fc310 con 0x55a4371f83f0 2024-08-03T20:57:12.469 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:12.469+0000 7f9b93200640 1 -- 172.21.15.32:0/3821133758 <== osd.5 v2:172.21.15.118:6810/2724891770 17 ==== osd_op_reply(48 606.00000000 [delete] v103'412 uv412 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f9b8404d980 con 0x55a4371f83f0 2024-08-03T20:57:12.469 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:12.469+0000 7f9b99ffef40 1 -- 172.21.15.32:0/3821133758 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:49 12.14 12:2f55791f:::606.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e103) -- 0x55a4371fc310 con 0x55a4371f83f0 2024-08-03T20:57:12.471 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::606.00000000:head# 2024-08-03T20:57:12.471 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:12.471+0000 7f9b93200640 1 -- 172.21.15.32:0/3821133758 <== osd.5 v2:172.21.15.118:6810/2724891770 18 ==== osd_op_reply(49 606.00000000 [create] v103'413 uv413 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f9b8404d980 con 0x55a4371f83f0 2024-08-03T20:57:12.471 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:12.471+0000 7f9b99ffef40 1 -- 172.21.15.32:0/3821133758 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:50 12.14 12:2f55791f:::606.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e103) -- 0x55a4371fd320 con 0x55a4371f83f0 2024-08-03T20:57:12.472 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:12.472+0000 7f9b93200640 1 -- 172.21.15.32:0/3821133758 <== osd.5 v2:172.21.15.118:6810/2724891770 19 ==== osd_op_reply(50 606.00000000 [setxattr (30)] v103'414 uv414 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f9b8404d980 con 0x55a4371f83f0 2024-08-03T20:57:12.472 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:12.472+0000 7f9b99ffef40 1 -- 172.21.15.32:0/3821133758 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:51 12.14 12:2f55791f:::606.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e103) -- 0x55a4371fd320 con 0x55a4371f83f0 2024-08-03T20:57:12.474 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:12.473+0000 7f9b93200640 1 -- 172.21.15.32:0/3821133758 <== osd.5 v2:172.21.15.118:6810/2724891770 20 ==== osd_op_reply(51 606.00000000 [setxattr (62)] v103'415 uv415 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f9b8404d980 con 0x55a4371f83f0 2024-08-03T20:57:12.474 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:12.473+0000 7f9b99ffef40 1 -- 172.21.15.32:0/3821133758 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:52 12.14 12:2f55791f:::606.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e103) -- 0x55a4371fc310 con 0x55a4371f83f0 2024-08-03T20:57:12.475 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:12.475+0000 7f9b93200640 1 -- 172.21.15.32:0/3821133758 <== osd.5 v2:172.21.15.118:6810/2724891770 21 ==== osd_op_reply(52 606.00000000 [omap-set-header] v103'416 uv416 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f9b8404d980 con 0x55a4371f83f0 2024-08-03T20:57:12.475 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:12.475+0000 7f9b99ffef40 1 -- 172.21.15.32:0/3821133758 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:53 12.1c 12:3ed09add:::607.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e103) -- 0x55a4371fc310 con 0x55a4371298a0 2024-08-03T20:57:12.476 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:12.476+0000 7f9b93c00640 1 -- 172.21.15.32:0/3821133758 <== osd.0 v2:172.21.15.32:6816/3066156116 32 ==== osd_op_reply(53 607.00000000 [create] v103'105 uv104 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f9b8c003040 con 0x55a4371298a0 2024-08-03T20:57:12.476 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:12.476+0000 7f9b99ffef40 1 -- 172.21.15.32:0/3821133758 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:54 12.1c 12:3ed09add:::607.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e103) -- 0x55a4371fc310 con 0x55a4371298a0 2024-08-03T20:57:12.478 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:12.477+0000 7f9b93c00640 1 -- 172.21.15.32:0/3821133758 <== osd.0 v2:172.21.15.32:6816/3066156116 33 ==== osd_op_reply(54 607.00000000 [delete] v103'106 uv106 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f9b8c003040 con 0x55a4371298a0 2024-08-03T20:57:12.478 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:12.477+0000 7f9b99ffef40 1 -- 172.21.15.32:0/3821133758 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:55 12.1c 12:3ed09add:::607.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e103) -- 0x55a4371fc310 con 0x55a4371298a0 2024-08-03T20:57:12.479 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:12.479+0000 7f9b93c00640 1 -- 172.21.15.32:0/3821133758 <== osd.0 v2:172.21.15.32:6816/3066156116 34 ==== osd_op_reply(55 607.00000000 [create] v103'107 uv107 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f9b8c003040 con 0x55a4371298a0 2024-08-03T20:57:12.479 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::607.00000000:head# 2024-08-03T20:57:12.479 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:12.479+0000 7f9b99ffef40 1 -- 172.21.15.32:0/3821133758 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:56 12.1c 12:3ed09add:::607.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e103) -- 0x55a4371fd320 con 0x55a4371298a0 2024-08-03T20:57:12.480 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:12.480+0000 7f9b93c00640 1 -- 172.21.15.32:0/3821133758 <== osd.0 v2:172.21.15.32:6816/3066156116 35 ==== osd_op_reply(56 607.00000000 [setxattr (30)] v103'108 uv108 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f9b8c003040 con 0x55a4371298a0 2024-08-03T20:57:12.480 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:12.480+0000 7f9b99ffef40 1 -- 172.21.15.32:0/3821133758 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:57 12.1c 12:3ed09add:::607.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e103) -- 0x55a4371fd320 con 0x55a4371298a0 2024-08-03T20:57:12.482 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:12.481+0000 7f9b93c00640 1 -- 172.21.15.32:0/3821133758 <== osd.0 v2:172.21.15.32:6816/3066156116 36 ==== osd_op_reply(57 607.00000000 [setxattr (62)] v103'109 uv109 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f9b8c003040 con 0x55a4371298a0 2024-08-03T20:57:12.482 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:12.481+0000 7f9b99ffef40 1 -- 172.21.15.32:0/3821133758 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:58 12.1c 12:3ed09add:::607.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e103) -- 0x55a4371fc310 con 0x55a4371298a0 2024-08-03T20:57:12.483 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:12.482+0000 7f9b93c00640 1 -- 172.21.15.32:0/3821133758 <== osd.0 v2:172.21.15.32:6816/3066156116 37 ==== osd_op_reply(58 607.00000000 [omap-set-header] v103'110 uv110 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f9b8c003040 con 0x55a4371298a0 2024-08-03T20:57:12.483 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:12.482+0000 7f9b99ffef40 1 --2- 172.21.15.32:0/3821133758 >> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] conn(0x55a4371fc990 0x55a4371fcda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:12.483 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:12.482+0000 7f9b99ffef40 1 -- 172.21.15.32:0/3821133758 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:59 12.6 12:654134d2:::mds0_openfiles.0:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e103) -- 0x55a4371ff6f0 con 0x55a4371fc990 2024-08-03T20:57:12.483 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:12.483+0000 7f9b92800640 1 --2- 172.21.15.32:0/3821133758 >> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] conn(0x55a4371fc990 0x55a4371fcda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:12.484 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:12.483+0000 7f9b92800640 1 --2- 172.21.15.32:0/3821133758 >> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] conn(0x55a4371fc990 0x55a4371fcda0 crc :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:57:12.485 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:12.484+0000 7f9b92800640 1 -- 172.21.15.32:0/3821133758 <== osd.1 v2:172.21.15.32:6824/1551241759 1 ==== osd_op_reply(59 mds0_openfiles.0 [create] v103'86 uv85 ondisk = -17 ((17) File exists)) ==== 160+0+0 (crc 0 0 0) 0x7f9b8000b040 con 0x55a4371fc990 2024-08-03T20:57:12.485 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:12.484+0000 7f9b99ffef40 1 -- 172.21.15.32:0/3821133758 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:60 12.6 12:654134d2:::mds0_openfiles.0:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e103) -- 0x55a4371ffa90 con 0x55a4371fc990 2024-08-03T20:57:12.486 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:12.486+0000 7f9b92800640 1 -- 172.21.15.32:0/3821133758 <== osd.1 v2:172.21.15.32:6824/1551241759 2 ==== osd_op_reply(60 mds0_openfiles.0 [delete] v103'87 uv87 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f9b8000b040 con 0x55a4371fc990 2024-08-03T20:57:12.486 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:12.486+0000 7f9b99ffef40 1 -- 172.21.15.32:0/3821133758 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:61 12.6 12:654134d2:::mds0_openfiles.0:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e103) -- 0x55a4371ffeb0 con 0x55a4371fc990 2024-08-03T20:57:12.487 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds0_openfiles.0:head# 2024-08-03T20:57:12.488 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:12.487+0000 7f9b92800640 1 -- 172.21.15.32:0/3821133758 <== osd.1 v2:172.21.15.32:6824/1551241759 3 ==== osd_op_reply(61 mds0_openfiles.0 [create] v103'88 uv88 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f9b8000b040 con 0x55a4371fc990 2024-08-03T20:57:12.488 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:12.487+0000 7f9b99ffef40 1 -- 172.21.15.32:0/3821133758 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:62 12.6 12:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e103) -- 0x55a437129430 con 0x55a4371fc990 2024-08-03T20:57:12.488 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:12.488+0000 7f9b92800640 1 -- 172.21.15.32:0/3821133758 <== osd.1 v2:172.21.15.32:6824/1551241759 4 ==== osd_op_reply(62 mds0_openfiles.0 [omap-set-header] v103'89 uv89 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f9b8000b040 con 0x55a4371fc990 2024-08-03T20:57:12.489 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:12.488+0000 7f9b99ffef40 1 -- 172.21.15.32:0/3821133758 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:63 12.e 12:717a0223:::608.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e103) -- 0x55a4372007e0 con 0x55a4371298a0 2024-08-03T20:57:12.490 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:12.490+0000 7f9b93c00640 1 -- 172.21.15.32:0/3821133758 <== osd.0 v2:172.21.15.32:6816/3066156116 38 ==== osd_op_reply(63 608.00000000 [create] v103'105 uv104 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f9b8c003040 con 0x55a4371298a0 2024-08-03T20:57:12.490 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:12.490+0000 7f9b99ffef40 1 -- 172.21.15.32:0/3821133758 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:64 12.e 12:717a0223:::608.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e103) -- 0x55a4372007e0 con 0x55a4371298a0 2024-08-03T20:57:12.492 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:12.491+0000 7f9b93c00640 1 -- 172.21.15.32:0/3821133758 <== osd.0 v2:172.21.15.32:6816/3066156116 39 ==== osd_op_reply(64 608.00000000 [delete] v103'106 uv106 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f9b8c003040 con 0x55a4371298a0 2024-08-03T20:57:12.492 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:12.492+0000 7f9b99ffef40 1 -- 172.21.15.32:0/3821133758 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:65 12.e 12:717a0223:::608.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e103) -- 0x55a4372007e0 con 0x55a4371298a0 2024-08-03T20:57:12.493 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::608.00000000:head# 2024-08-03T20:57:12.493 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:12.493+0000 7f9b93c00640 1 -- 172.21.15.32:0/3821133758 <== osd.0 v2:172.21.15.32:6816/3066156116 40 ==== osd_op_reply(65 608.00000000 [create] v103'107 uv107 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f9b8c003040 con 0x55a4371298a0 2024-08-03T20:57:12.494 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:12.493+0000 7f9b99ffef40 1 -- 172.21.15.32:0/3821133758 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:66 12.e 12:717a0223:::608.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e103) -- 0x55a4372017f0 con 0x55a4371298a0 2024-08-03T20:57:12.495 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:12.495+0000 7f9b93c00640 1 -- 172.21.15.32:0/3821133758 <== osd.0 v2:172.21.15.32:6816/3066156116 41 ==== osd_op_reply(66 608.00000000 [setxattr (30)] v103'108 uv108 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f9b8c003040 con 0x55a4371298a0 2024-08-03T20:57:12.495 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:12.495+0000 7f9b99ffef40 1 -- 172.21.15.32:0/3821133758 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:67 12.e 12:717a0223:::608.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e103) -- 0x55a4372017f0 con 0x55a4371298a0 2024-08-03T20:57:12.496 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:12.496+0000 7f9b93c00640 1 -- 172.21.15.32:0/3821133758 <== osd.0 v2:172.21.15.32:6816/3066156116 42 ==== osd_op_reply(67 608.00000000 [setxattr (62)] v103'109 uv109 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f9b8c003040 con 0x55a4371298a0 2024-08-03T20:57:12.496 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:12.496+0000 7f9b99ffef40 1 -- 172.21.15.32:0/3821133758 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:68 12.e 12:717a0223:::608.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e103) -- 0x55a4372007e0 con 0x55a4371298a0 2024-08-03T20:57:12.498 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:12.498+0000 7f9b93c00640 1 -- 172.21.15.32:0/3821133758 <== osd.0 v2:172.21.15.32:6816/3066156116 43 ==== osd_op_reply(68 608.00000000 [omap-set-header] v103'110 uv110 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f9b8c003040 con 0x55a4371298a0 2024-08-03T20:57:12.498 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:12.498+0000 7f9b99ffef40 1 --2- 172.21.15.32:0/3821133758 >> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] conn(0x55a437200eb0 0x55a4372012c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:12.498 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:12.498+0000 7f9b99ffef40 1 -- 172.21.15.32:0/3821133758 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:69 12.15 12:a93a17c2:::604.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e103) -- 0x55a437203c00 con 0x55a437200eb0 2024-08-03T20:57:12.499 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:12.498+0000 7f9b93c00640 1 --2- 172.21.15.32:0/3821133758 >> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] conn(0x55a437200eb0 0x55a4372012c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:12.499 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:12.499+0000 7f9b93c00640 1 --2- 172.21.15.32:0/3821133758 >> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] conn(0x55a437200eb0 0x55a4372012c0 crc :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:57:12.501 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:12.500+0000 7f9b93c00640 1 -- 172.21.15.32:0/3821133758 <== osd.6 v2:172.21.15.118:6825/801947386 1 ==== osd_op_reply(69 604.00000000 [create] v103'193 uv187 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f9b8c003040 con 0x55a437200eb0 2024-08-03T20:57:12.501 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:12.500+0000 7f9b99ffef40 1 -- 172.21.15.32:0/3821133758 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:70 12.15 12:a93a17c2:::604.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e103) -- 0x55a437203c00 con 0x55a437200eb0 2024-08-03T20:57:12.502 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:12.502+0000 7f9b93c00640 1 -- 172.21.15.32:0/3821133758 <== osd.6 v2:172.21.15.118:6825/801947386 2 ==== osd_op_reply(70 604.00000000 [delete] v103'194 uv194 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f9b8c003040 con 0x55a437200eb0 2024-08-03T20:57:12.502 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:12.502+0000 7f9b99ffef40 1 -- 172.21.15.32:0/3821133758 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:71 12.15 12:a93a17c2:::604.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e103) -- 0x55a437203c00 con 0x55a437200eb0 2024-08-03T20:57:12.503 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::604.00000000:head# 2024-08-03T20:57:12.504 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:12.503+0000 7f9b93c00640 1 -- 172.21.15.32:0/3821133758 <== osd.6 v2:172.21.15.118:6825/801947386 3 ==== osd_op_reply(71 604.00000000 [create] v103'195 uv195 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f9b8c003040 con 0x55a437200eb0 2024-08-03T20:57:12.504 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:12.503+0000 7f9b99ffef40 1 -- 172.21.15.32:0/3821133758 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:72 12.15 12:a93a17c2:::604.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e103) -- 0x55a437205000 con 0x55a437200eb0 2024-08-03T20:57:12.505 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:12.505+0000 7f9b93c00640 1 -- 172.21.15.32:0/3821133758 <== osd.6 v2:172.21.15.118:6825/801947386 4 ==== osd_op_reply(72 604.00000000 [setxattr (30)] v103'196 uv196 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f9b8c003040 con 0x55a437200eb0 2024-08-03T20:57:12.505 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:12.505+0000 7f9b99ffef40 1 -- 172.21.15.32:0/3821133758 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:73 12.15 12:a93a17c2:::604.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e103) -- 0x55a437205000 con 0x55a437200eb0 2024-08-03T20:57:12.506 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:12.506+0000 7f9b93c00640 1 -- 172.21.15.32:0/3821133758 <== osd.6 v2:172.21.15.118:6825/801947386 5 ==== osd_op_reply(73 604.00000000 [setxattr (62)] v103'197 uv197 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f9b8c003040 con 0x55a437200eb0 2024-08-03T20:57:12.507 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:12.506+0000 7f9b99ffef40 1 -- 172.21.15.32:0/3821133758 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:74 12.15 12:a93a17c2:::604.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e103) -- 0x55a437203ff0 con 0x55a437200eb0 2024-08-03T20:57:12.508 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:12.508+0000 7f9b93c00640 1 -- 172.21.15.32:0/3821133758 <== osd.6 v2:172.21.15.118:6825/801947386 6 ==== osd_op_reply(74 604.00000000 [omap-set-header] v103'198 uv198 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f9b8c003040 con 0x55a437200eb0 2024-08-03T20:57:12.508 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:12.508+0000 7f9b99ffef40 1 -- 172.21.15.32:0/3821133758 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:75 12.15 12:aa448500:::500.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e103) -- 0x55a437203ff0 con 0x55a437200eb0 2024-08-03T20:57:12.509 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:12.509+0000 7f9b93c00640 1 -- 172.21.15.32:0/3821133758 <== osd.6 v2:172.21.15.118:6825/801947386 7 ==== osd_op_reply(75 500.00000000 [create] v103'199 uv192 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f9b8c003040 con 0x55a437200eb0 2024-08-03T20:57:12.509 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:12.509+0000 7f9b99ffef40 1 -- 172.21.15.32:0/3821133758 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:76 12.15 12:aa448500:::500.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e103) -- 0x55a437203ff0 con 0x55a437200eb0 2024-08-03T20:57:12.511 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:12.510+0000 7f9b93c00640 1 -- 172.21.15.32:0/3821133758 <== osd.6 v2:172.21.15.118:6825/801947386 8 ==== osd_op_reply(76 500.00000000 [delete] v103'200 uv200 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f9b8c003040 con 0x55a437200eb0 2024-08-03T20:57:12.511 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:12.510+0000 7f9b99ffef40 1 -- 172.21.15.32:0/3821133758 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:77 12.15 12:aa448500:::500.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e103) -- 0x55a437203ff0 con 0x55a437200eb0 2024-08-03T20:57:12.512 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::500.00000000:head# 2024-08-03T20:57:12.512 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:12.512+0000 7f9b93c00640 1 -- 172.21.15.32:0/3821133758 <== osd.6 v2:172.21.15.118:6825/801947386 9 ==== osd_op_reply(77 500.00000000 [create] v103'201 uv201 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f9b8c003040 con 0x55a437200eb0 2024-08-03T20:57:12.512 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:12.512+0000 7f9b99ffef40 1 -- 172.21.15.32:0/3821133758 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:78 12.15 12:aa448500:::500.00000000:head [write 0~90 in=90b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e103) -- 0x55a437203ff0 con 0x55a437200eb0 2024-08-03T20:57:12.514 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:12.513+0000 7f9b93c00640 1 -- 172.21.15.32:0/3821133758 <== osd.6 v2:172.21.15.118:6825/801947386 10 ==== osd_op_reply(78 500.00000000 [write 0~90] v103'202 uv202 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f9b8c003040 con 0x55a437200eb0 2024-08-03T20:57:12.514 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:12.513+0000 7f9b99ffef40 1 -- 172.21.15.32:0/3821133758 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:79 12.15 12:aa448500:::500.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e103) -- 0x55a437203ff0 con 0x55a437200eb0 2024-08-03T20:57:12.518 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:12.515+0000 7f9b93c00640 1 -- 172.21.15.32:0/3821133758 <== osd.6 v2:172.21.15.118:6825/801947386 11 ==== osd_op_reply(79 500.00000000 [omap-set-header] v103'203 uv203 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f9b8c003040 con 0x55a437200eb0 2024-08-03T20:57:12.518 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:12.515+0000 7f9b99ffef40 1 --2- 172.21.15.32:0/3821133758 >> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] conn(0x55a437204500 0x55a437206920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:12.518 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:12.515+0000 7f9b99ffef40 1 -- 172.21.15.32:0/3821133758 --> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] -- osd_op(unknown.0.0:80 12.d 12:b50e409b:::mds_snaptable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e103) -- 0x55a437206f60 con 0x55a437204500 2024-08-03T20:57:12.518 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:12.515+0000 7f9b93200640 1 --2- 172.21.15.32:0/3821133758 >> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] conn(0x55a437204500 0x55a437206920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:12.518 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:12.516+0000 7f9b93200640 1 --2- 172.21.15.32:0/3821133758 >> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] conn(0x55a437204500 0x55a437206920 crc :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:57:12.519 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:12.518+0000 7f9b93200640 1 -- 172.21.15.32:0/3821133758 <== osd.7 v2:172.21.15.118:6818/653000048 1 ==== osd_op_reply(80 mds_snaptable [create] v103'86 uv85 ondisk = -17 ((17) File exists)) ==== 157+0+0 (crc 0 0 0) 0x7f9b8404d980 con 0x55a437204500 2024-08-03T20:57:12.519 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:12.518+0000 7f9b99ffef40 1 -- 172.21.15.32:0/3821133758 --> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] -- osd_op(unknown.0.0:81 12.d 12:b50e409b:::mds_snaptable:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e103) -- 0x55a437207330 con 0x55a437204500 2024-08-03T20:57:12.519 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:12.519+0000 7f9b93200640 1 -- 172.21.15.32:0/3821133758 <== osd.7 v2:172.21.15.118:6818/653000048 2 ==== osd_op_reply(81 mds_snaptable [delete] v103'87 uv87 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f9b8404d980 con 0x55a437204500 2024-08-03T20:57:12.520 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:12.519+0000 7f9b99ffef40 1 -- 172.21.15.32:0/3821133758 --> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] -- osd_op(unknown.0.0:82 12.d 12:b50e409b:::mds_snaptable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e103) -- 0x55a437207330 con 0x55a437204500 2024-08-03T20:57:12.521 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds_snaptable:head# 2024-08-03T20:57:12.521 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:12.521+0000 7f9b93200640 1 -- 172.21.15.32:0/3821133758 <== osd.7 v2:172.21.15.118:6818/653000048 3 ==== osd_op_reply(82 mds_snaptable [create] v103'88 uv88 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f9b8404d980 con 0x55a437204500 2024-08-03T20:57:12.521 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:12.521+0000 7f9b99ffef40 1 -- 172.21.15.32:0/3821133758 --> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] -- osd_op(unknown.0.0:83 12.d 12:b50e409b:::mds_snaptable:head [write 0~70 in=70b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e103) -- 0x55a437207330 con 0x55a437204500 2024-08-03T20:57:12.522 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:12.522+0000 7f9b93200640 1 -- 172.21.15.32:0/3821133758 <== osd.7 v2:172.21.15.118:6818/653000048 4 ==== osd_op_reply(83 mds_snaptable [write 0~70] v103'89 uv89 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f9b8404d980 con 0x55a437204500 2024-08-03T20:57:12.523 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:12.522+0000 7f9b99ffef40 1 -- 172.21.15.32:0/3821133758 --> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] -- osd_op(unknown.0.0:84 12.d 12:b50e409b:::mds_snaptable:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e103) -- 0x55a437207330 con 0x55a437204500 2024-08-03T20:57:12.524 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:12.523+0000 7f9b93200640 1 -- 172.21.15.32:0/3821133758 <== osd.7 v2:172.21.15.118:6818/653000048 5 ==== osd_op_reply(84 mds_snaptable [omap-set-header] v103'90 uv90 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f9b8404d980 con 0x55a437204500 2024-08-03T20:57:12.524 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:12.523+0000 7f9b99ffef40 1 -- 172.21.15.32:0/3821133758 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:85 12.1d 12:b871830b:::605.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e103) -- 0x55a437207330 con 0x55a4371298a0 2024-08-03T20:57:12.525 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:12.525+0000 7f9b93c00640 1 -- 172.21.15.32:0/3821133758 <== osd.0 v2:172.21.15.32:6816/3066156116 44 ==== osd_op_reply(85 605.00000000 [create] v103'105 uv104 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f9b8c003040 con 0x55a4371298a0 2024-08-03T20:57:12.525 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:12.525+0000 7f9b99ffef40 1 -- 172.21.15.32:0/3821133758 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:86 12.1d 12:b871830b:::605.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e103) -- 0x55a437207330 con 0x55a4371298a0 2024-08-03T20:57:12.526 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:12.526+0000 7f9b93c00640 1 -- 172.21.15.32:0/3821133758 <== osd.0 v2:172.21.15.32:6816/3066156116 45 ==== osd_op_reply(86 605.00000000 [delete] v103'106 uv106 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f9b8c003040 con 0x55a4371298a0 2024-08-03T20:57:12.526 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:12.526+0000 7f9b99ffef40 1 -- 172.21.15.32:0/3821133758 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:87 12.1d 12:b871830b:::605.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e103) -- 0x55a437207330 con 0x55a4371298a0 2024-08-03T20:57:12.527 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:12.527+0000 7f9b93c00640 1 -- 172.21.15.32:0/3821133758 <== osd.0 v2:172.21.15.32:6816/3066156116 46 ==== osd_op_reply(87 605.00000000 [create] v103'107 uv107 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f9b8c003040 con 0x55a4371298a0 2024-08-03T20:57:12.528 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::605.00000000:head# 2024-08-03T20:57:12.528 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:12.527+0000 7f9b99ffef40 1 -- 172.21.15.32:0/3821133758 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:88 12.1d 12:b871830b:::605.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e103) -- 0x55a437207330 con 0x55a4371298a0 2024-08-03T20:57:12.529 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:12.529+0000 7f9b93c00640 1 -- 172.21.15.32:0/3821133758 <== osd.0 v2:172.21.15.32:6816/3066156116 47 ==== osd_op_reply(88 605.00000000 [setxattr (30)] v103'108 uv108 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f9b8c003040 con 0x55a4371298a0 2024-08-03T20:57:12.529 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:12.529+0000 7f9b99ffef40 1 -- 172.21.15.32:0/3821133758 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:89 12.1d 12:b871830b:::605.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e103) -- 0x55a437207330 con 0x55a4371298a0 2024-08-03T20:57:12.530 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:12.530+0000 7f9b93c00640 1 -- 172.21.15.32:0/3821133758 <== osd.0 v2:172.21.15.32:6816/3066156116 48 ==== osd_op_reply(89 605.00000000 [setxattr (62)] v103'109 uv109 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f9b8c003040 con 0x55a4371298a0 2024-08-03T20:57:12.530 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:12.530+0000 7f9b99ffef40 1 -- 172.21.15.32:0/3821133758 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:90 12.1d 12:b871830b:::605.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e103) -- 0x55a437207330 con 0x55a4371298a0 2024-08-03T20:57:12.531 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:12.531+0000 7f9b93c00640 1 -- 172.21.15.32:0/3821133758 <== osd.0 v2:172.21.15.32:6816/3066156116 49 ==== osd_op_reply(90 605.00000000 [omap-set-header] v103'110 uv110 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f9b8c003040 con 0x55a4371298a0 2024-08-03T20:57:12.532 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:12.531+0000 7f9b99ffef40 1 -- 172.21.15.32:0/3821133758 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:91 12.13 12:c91d4a1d:::mds0_inotable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e103) -- 0x55a437207330 con 0x55a4371fc990 2024-08-03T20:57:12.533 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:12.533+0000 7f9b92800640 1 -- 172.21.15.32:0/3821133758 <== osd.1 v2:172.21.15.32:6824/1551241759 5 ==== osd_op_reply(91 mds0_inotable [create] v103'172 uv166 ondisk = -17 ((17) File exists)) ==== 157+0+0 (crc 0 0 0) 0x7f9b8000b040 con 0x55a4371fc990 2024-08-03T20:57:12.533 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:12.533+0000 7f9b99ffef40 1 -- 172.21.15.32:0/3821133758 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:92 12.13 12:c91d4a1d:::mds0_inotable:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e103) -- 0x55a437207720 con 0x55a4371fc990 2024-08-03T20:57:12.534 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:12.534+0000 7f9b92800640 1 -- 172.21.15.32:0/3821133758 <== osd.1 v2:172.21.15.32:6824/1551241759 6 ==== osd_op_reply(92 mds0_inotable [delete] v103'173 uv173 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f9b8000b040 con 0x55a4371fc990 2024-08-03T20:57:12.534 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:12.534+0000 7f9b99ffef40 1 -- 172.21.15.32:0/3821133758 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:93 12.13 12:c91d4a1d:::mds0_inotable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e103) -- 0x55a437207b10 con 0x55a4371fc990 2024-08-03T20:57:12.536 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:12.535+0000 7f9b92800640 1 -- 172.21.15.32:0/3821133758 <== osd.1 v2:172.21.15.32:6824/1551241759 7 ==== osd_op_reply(93 mds0_inotable [create] v103'174 uv174 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f9b8000b040 con 0x55a4371fc990 2024-08-03T20:57:12.536 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds0_inotable:head# 2024-08-03T20:57:12.536 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:12.535+0000 7f9b99ffef40 1 -- 172.21.15.32:0/3821133758 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:94 12.13 12:c91d4a1d:::mds0_inotable:head [write 0~34 in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e103) -- 0x55a4372003a0 con 0x55a4371fc990 2024-08-03T20:57:12.537 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:12.537+0000 7f9b92800640 1 -- 172.21.15.32:0/3821133758 <== osd.1 v2:172.21.15.32:6824/1551241759 8 ==== osd_op_reply(94 mds0_inotable [write 0~34] v103'175 uv175 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f9b8000b040 con 0x55a4371fc990 2024-08-03T20:57:12.537 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:12.537+0000 7f9b99ffef40 1 -- 172.21.15.32:0/3821133758 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:95 12.13 12:c91d4a1d:::mds0_inotable:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e103) -- 0x55a4372003a0 con 0x55a4371fc990 2024-08-03T20:57:12.538 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:12.538+0000 7f9b92800640 1 -- 172.21.15.32:0/3821133758 <== osd.1 v2:172.21.15.32:6824/1551241759 9 ==== osd_op_reply(95 mds0_inotable [omap-set-header] v103'176 uv176 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f9b8000b040 con 0x55a4371fc990 2024-08-03T20:57:12.539 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:12.538+0000 7f9b99ffef40 1 -- 172.21.15.32:0/3821133758 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:96 12.13 12:cd5a64a3:::100.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e103) -- 0x55a437200380 con 0x55a4371fc990 2024-08-03T20:57:12.539 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:12.539+0000 7f9b92800640 1 -- 172.21.15.32:0/3821133758 <== osd.1 v2:172.21.15.32:6824/1551241759 10 ==== osd_op_reply(96 100.00000000 [create] v103'177 uv171 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f9b8000b040 con 0x55a4371fc990 2024-08-03T20:57:12.540 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:12.539+0000 7f9b99ffef40 1 -- 172.21.15.32:0/3821133758 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:97 12.13 12:cd5a64a3:::100.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e103) -- 0x55a437200380 con 0x55a4371fc990 2024-08-03T20:57:12.541 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:12.541+0000 7f9b92800640 1 -- 172.21.15.32:0/3821133758 <== osd.1 v2:172.21.15.32:6824/1551241759 11 ==== osd_op_reply(97 100.00000000 [delete] v103'178 uv178 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f9b8000b040 con 0x55a4371fc990 2024-08-03T20:57:12.541 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:12.541+0000 7f9b99ffef40 1 -- 172.21.15.32:0/3821133758 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:98 12.13 12:cd5a64a3:::100.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e103) -- 0x55a437200380 con 0x55a4371fc990 2024-08-03T20:57:12.542 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:12.542+0000 7f9b92800640 1 -- 172.21.15.32:0/3821133758 <== osd.1 v2:172.21.15.32:6824/1551241759 12 ==== osd_op_reply(98 100.00000000 [create] v103'179 uv179 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f9b8000b040 con 0x55a4371fc990 2024-08-03T20:57:12.542 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::100.00000000:head# 2024-08-03T20:57:12.543 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:12.542+0000 7f9b99ffef40 1 -- 172.21.15.32:0/3821133758 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:99 12.13 12:cd5a64a3:::100.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e103) -- 0x55a437200380 con 0x55a4371fc990 2024-08-03T20:57:12.544 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:12.543+0000 7f9b92800640 1 -- 172.21.15.32:0/3821133758 <== osd.1 v2:172.21.15.32:6824/1551241759 13 ==== osd_op_reply(99 100.00000000 [omap-set-header] v103'180 uv180 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f9b8000b040 con 0x55a4371fc990 2024-08-03T20:57:12.544 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:12.543+0000 7f9b99ffef40 1 -- 172.21.15.32:0/3821133758 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:100 12.13 12:cd5a64a3:::100.00000000:head [omap-set-vals in=5194b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e103) -- 0x55a43720b0b0 con 0x55a4371fc990 2024-08-03T20:57:12.545 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:12.545+0000 7f9b92800640 1 -- 172.21.15.32:0/3821133758 <== osd.1 v2:172.21.15.32:6824/1551241759 14 ==== osd_op_reply(100 100.00000000 [omap-set-vals] v103'181 uv181 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f9b8000b040 con 0x55a4371fc990 2024-08-03T20:57:12.545 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:12.545+0000 7f9b99ffef40 1 --2- 172.21.15.32:0/3821133758 >> [v2:172.21.15.32:6800/9088303,v1:172.21.15.32:6801/9088303] conn(0x55a437208c60 0x55a43720b080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:12.545 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:12.545+0000 7f9b99ffef40 1 -- 172.21.15.32:0/3821133758 --> [v2:172.21.15.32:6800/9088303,v1:172.21.15.32:6801/9088303] -- osd_op(unknown.0.0:101 12.b 12:d0630e4c:::mds0_sessionmap:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e103) -- 0x55a43720b5c0 con 0x55a437208c60 2024-08-03T20:57:12.546 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:12.545+0000 7f9b92800640 1 --2- 172.21.15.32:0/3821133758 >> [v2:172.21.15.32:6800/9088303,v1:172.21.15.32:6801/9088303] conn(0x55a437208c60 0x55a43720b080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:12.546 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:12.546+0000 7f9b92800640 1 --2- 172.21.15.32:0/3821133758 >> [v2:172.21.15.32:6800/9088303,v1:172.21.15.32:6801/9088303] conn(0x55a437208c60 0x55a43720b080 crc :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:57:12.547 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:12.547+0000 7f9b92800640 1 -- 172.21.15.32:0/3821133758 <== osd.3 v2:172.21.15.32:6800/9088303 1 ==== osd_op_reply(101 mds0_sessionmap [create] v103'78 uv77 ondisk = -17 ((17) File exists)) ==== 159+0+0 (crc 0 0 0) 0x7f9b8000b040 con 0x55a437208c60 2024-08-03T20:57:12.547 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:12.547+0000 7f9b99ffef40 1 -- 172.21.15.32:0/3821133758 --> [v2:172.21.15.32:6800/9088303,v1:172.21.15.32:6801/9088303] -- osd_op(unknown.0.0:102 12.b 12:d0630e4c:::mds0_sessionmap:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e103) -- 0x55a43720b5c0 con 0x55a437208c60 2024-08-03T20:57:12.549 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:12.548+0000 7f9b92800640 1 -- 172.21.15.32:0/3821133758 <== osd.3 v2:172.21.15.32:6800/9088303 2 ==== osd_op_reply(102 mds0_sessionmap [delete] v103'79 uv79 ondisk = 0) ==== 159+0+0 (crc 0 0 0) 0x7f9b8000b040 con 0x55a437208c60 2024-08-03T20:57:12.549 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:12.548+0000 7f9b99ffef40 1 -- 172.21.15.32:0/3821133758 --> [v2:172.21.15.32:6800/9088303,v1:172.21.15.32:6801/9088303] -- osd_op(unknown.0.0:103 12.b 12:d0630e4c:::mds0_sessionmap:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e103) -- 0x55a43720b5c0 con 0x55a437208c60 2024-08-03T20:57:12.550 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:12.550+0000 7f9b92800640 1 -- 172.21.15.32:0/3821133758 <== osd.3 v2:172.21.15.32:6800/9088303 3 ==== osd_op_reply(103 mds0_sessionmap [create] v103'80 uv80 ondisk = 0) ==== 159+0+0 (crc 0 0 0) 0x7f9b8000b040 con 0x55a437208c60 2024-08-03T20:57:12.550 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds0_sessionmap:head# 2024-08-03T20:57:12.550 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:12.550+0000 7f9b99ffef40 1 -- 172.21.15.32:0/3821133758 --> [v2:172.21.15.32:6800/9088303,v1:172.21.15.32:6801/9088303] -- osd_op(unknown.0.0:104 12.b 12:d0630e4c:::mds0_sessionmap:head [omap-set-header in=14b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e103) -- 0x55a43720b5c0 con 0x55a437208c60 2024-08-03T20:57:12.552 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:12.551+0000 7f9b92800640 1 -- 172.21.15.32:0/3821133758 <== osd.3 v2:172.21.15.32:6800/9088303 4 ==== osd_op_reply(104 mds0_sessionmap [omap-set-header] v103'81 uv81 ondisk = 0) ==== 159+0+0 (crc 0 0 0) 0x7f9b8000b040 con 0x55a437208c60 2024-08-03T20:57:12.552 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:12.551+0000 7f9b99ffef40 1 -- 172.21.15.32:0/3821133758 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:105 12.7 12:e0b41d06:::609.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e103) -- 0x55a43720b5c0 con 0x55a4371fc990 2024-08-03T20:57:12.553 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:12.553+0000 7f9b92800640 1 -- 172.21.15.32:0/3821133758 <== osd.1 v2:172.21.15.32:6824/1551241759 15 ==== osd_op_reply(105 609.00000000 [create] v103'105 uv104 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f9b8000b040 con 0x55a4371fc990 2024-08-03T20:57:12.553 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:12.553+0000 7f9b99ffef40 1 -- 172.21.15.32:0/3821133758 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:106 12.7 12:e0b41d06:::609.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e103) -- 0x55a43720b5c0 con 0x55a4371fc990 2024-08-03T20:57:12.554 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:12.554+0000 7f9b92800640 1 -- 172.21.15.32:0/3821133758 <== osd.1 v2:172.21.15.32:6824/1551241759 16 ==== osd_op_reply(106 609.00000000 [delete] v103'106 uv106 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f9b8000b040 con 0x55a4371fc990 2024-08-03T20:57:12.554 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:12.554+0000 7f9b99ffef40 1 -- 172.21.15.32:0/3821133758 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:107 12.7 12:e0b41d06:::609.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e103) -- 0x55a43720b5c0 con 0x55a4371fc990 2024-08-03T20:57:12.555 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:12.555+0000 7f9b92800640 1 -- 172.21.15.32:0/3821133758 <== osd.1 v2:172.21.15.32:6824/1551241759 17 ==== osd_op_reply(107 609.00000000 [create] v103'107 uv107 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f9b8000b040 con 0x55a4371fc990 2024-08-03T20:57:12.556 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::609.00000000:head# 2024-08-03T20:57:12.556 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:12.555+0000 7f9b99ffef40 1 -- 172.21.15.32:0/3821133758 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:108 12.7 12:e0b41d06:::609.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e103) -- 0x55a43720c5d0 con 0x55a4371fc990 2024-08-03T20:57:12.557 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:12.557+0000 7f9b92800640 1 -- 172.21.15.32:0/3821133758 <== osd.1 v2:172.21.15.32:6824/1551241759 18 ==== osd_op_reply(108 609.00000000 [setxattr (30)] v103'108 uv108 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f9b8000b040 con 0x55a4371fc990 2024-08-03T20:57:12.557 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:12.557+0000 7f9b99ffef40 1 -- 172.21.15.32:0/3821133758 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:109 12.7 12:e0b41d06:::609.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e103) -- 0x55a43720c5d0 con 0x55a4371fc990 2024-08-03T20:57:12.558 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:12.558+0000 7f9b92800640 1 -- 172.21.15.32:0/3821133758 <== osd.1 v2:172.21.15.32:6824/1551241759 19 ==== osd_op_reply(109 609.00000000 [setxattr (62)] v103'109 uv109 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f9b8000b040 con 0x55a4371fc990 2024-08-03T20:57:12.558 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:12.558+0000 7f9b99ffef40 1 -- 172.21.15.32:0/3821133758 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:110 12.7 12:e0b41d06:::609.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e103) -- 0x55a43720b5c0 con 0x55a4371fc990 2024-08-03T20:57:12.560 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:12.559+0000 7f9b92800640 1 -- 172.21.15.32:0/3821133758 <== osd.1 v2:172.21.15.32:6824/1551241759 20 ==== osd_op_reply(110 609.00000000 [omap-set-header] v103'110 uv110 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f9b8000b040 con 0x55a4371fc990 2024-08-03T20:57:12.560 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:12.559+0000 7f9b99ffef40 1 -- 172.21.15.32:0/3821133758 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:111 12.f 12:f1f69726:::400.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e103) -- 0x55a43720b5c0 con 0x55a437200eb0 2024-08-03T20:57:12.561 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:12.561+0000 7f9b93c00640 1 -- 172.21.15.32:0/3821133758 <== osd.6 v2:172.21.15.118:6825/801947386 12 ==== osd_op_reply(111 400.00000000 [create] v103'89 uv88 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f9b8c003040 con 0x55a437200eb0 2024-08-03T20:57:12.561 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:12.561+0000 7f9b99ffef40 1 -- 172.21.15.32:0/3821133758 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:112 12.f 12:f1f69726:::400.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e103) -- 0x55a43720b5c0 con 0x55a437200eb0 2024-08-03T20:57:12.562 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:12.562+0000 7f9b93c00640 1 -- 172.21.15.32:0/3821133758 <== osd.6 v2:172.21.15.118:6825/801947386 13 ==== osd_op_reply(112 400.00000000 [delete] v103'90 uv90 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f9b8c003040 con 0x55a437200eb0 2024-08-03T20:57:12.563 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:12.562+0000 7f9b99ffef40 1 -- 172.21.15.32:0/3821133758 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:113 12.f 12:f1f69726:::400.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e103) -- 0x55a43720b5c0 con 0x55a437200eb0 2024-08-03T20:57:12.564 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:12.564+0000 7f9b93c00640 1 -- 172.21.15.32:0/3821133758 <== osd.6 v2:172.21.15.118:6825/801947386 14 ==== osd_op_reply(113 400.00000000 [create] v103'91 uv91 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f9b8c003040 con 0x55a437200eb0 2024-08-03T20:57:12.564 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::400.00000000:head# 2024-08-03T20:57:12.564 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:12.564+0000 7f9b99ffef40 1 -- 172.21.15.32:0/3821133758 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:114 12.f 12:f1f69726:::400.00000000:head [write 0~22 in=22b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e103) -- 0x55a437207ef0 con 0x55a437200eb0 2024-08-03T20:57:12.566 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:12.565+0000 7f9b93c00640 1 -- 172.21.15.32:0/3821133758 <== osd.6 v2:172.21.15.118:6825/801947386 15 ==== osd_op_reply(114 400.00000000 [write 0~22] v103'92 uv92 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f9b8c003040 con 0x55a437200eb0 2024-08-03T20:57:12.566 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:12.566+0000 7f9b99ffef40 1 -- 172.21.15.32:0/3821133758 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:115 12.f 12:f1f69726:::400.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e103) -- 0x55a437207ef0 con 0x55a437200eb0 2024-08-03T20:57:12.568 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:12.567+0000 7f9b93c00640 1 -- 172.21.15.32:0/3821133758 <== osd.6 v2:172.21.15.118:6825/801947386 16 ==== osd_op_reply(115 400.00000000 [omap-set-header] v103'93 uv93 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f9b8c003040 con 0x55a437200eb0 2024-08-03T20:57:12.568 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:12.567+0000 7f9b99ffef40 1 -- 172.21.15.32:0/3821133758 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:116 12.1f 12:fe07bba1:::100.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e103) -- 0x55a437207ef0 con 0x55a437200eb0 2024-08-03T20:57:12.569 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:12.569+0000 7f9b93c00640 1 -- 172.21.15.32:0/3821133758 <== osd.6 v2:172.21.15.118:6825/801947386 17 ==== osd_op_reply(116 100.00000000.inode [create] v103'216 uv208 ondisk = -17 ((17) File exists)) ==== 162+0+0 (crc 0 0 0) 0x7f9b8c003040 con 0x55a437200eb0 2024-08-03T20:57:12.570 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:12.569+0000 7f9b99ffef40 1 -- 172.21.15.32:0/3821133758 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:117 12.1f 12:fe07bba1:::100.00000000.inode:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e103) -- 0x55a437207ef0 con 0x55a437200eb0 2024-08-03T20:57:12.571 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:12.570+0000 7f9b93c00640 1 -- 172.21.15.32:0/3821133758 <== osd.6 v2:172.21.15.118:6825/801947386 18 ==== osd_op_reply(117 100.00000000.inode [delete] v103'217 uv217 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7f9b8c003040 con 0x55a437200eb0 2024-08-03T20:57:12.571 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:12.570+0000 7f9b99ffef40 1 -- 172.21.15.32:0/3821133758 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:118 12.1f 12:fe07bba1:::100.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e103) -- 0x55a437207ed0 con 0x55a437200eb0 2024-08-03T20:57:12.572 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::100.00000000.inode:head# 2024-08-03T20:57:12.573 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:12.572+0000 7f9b93c00640 1 -- 172.21.15.32:0/3821133758 <== osd.6 v2:172.21.15.118:6825/801947386 19 ==== osd_op_reply(118 100.00000000.inode [create] v103'218 uv218 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7f9b8c003040 con 0x55a437200eb0 2024-08-03T20:57:12.573 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:12.572+0000 7f9b99ffef40 1 -- 172.21.15.32:0/3821133758 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:119 12.1f 12:fe07bba1:::100.00000000.inode:head [write 0~582 in=582b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e103) -- 0x55a437207ed0 con 0x55a437200eb0 2024-08-03T20:57:12.574 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:12.573+0000 7f9b93c00640 1 -- 172.21.15.32:0/3821133758 <== osd.6 v2:172.21.15.118:6825/801947386 20 ==== osd_op_reply(119 100.00000000.inode [write 0~582] v103'219 uv219 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7f9b8c003040 con 0x55a437200eb0 2024-08-03T20:57:12.574 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:12.573+0000 7f9b99ffef40 1 -- 172.21.15.32:0/3821133758 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:120 12.1f 12:fe07bba1:::100.00000000.inode:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e103) -- 0x55a437207ed0 con 0x55a437200eb0 2024-08-03T20:57:12.575 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:12.575+0000 7f9b93c00640 1 -- 172.21.15.32:0/3821133758 <== osd.6 v2:172.21.15.118:6825/801947386 21 ==== osd_op_reply(120 100.00000000.inode [omap-set-header] v103'220 uv220 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7f9b8c003040 con 0x55a437200eb0 2024-08-03T20:57:12.575 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:12.575+0000 7f9b99ffef40 1 -- 172.21.15.32:0/3821133758 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:121 12.1f 12:ff5b34d6:::1.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e103) -- 0x55a437207ed0 con 0x55a437200eb0 2024-08-03T20:57:12.577 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:12.576+0000 7f9b93c00640 1 -- 172.21.15.32:0/3821133758 <== osd.6 v2:172.21.15.118:6825/801947386 22 ==== osd_op_reply(121 1.00000000 [create] v103'221 uv215 ondisk = -17 ((17) File exists)) ==== 154+0+0 (crc 0 0 0) 0x7f9b8c003040 con 0x55a437200eb0 2024-08-03T20:57:12.577 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:12.576+0000 7f9b99ffef40 1 -- 172.21.15.32:0/3821133758 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:122 12.1f 12:ff5b34d6:::1.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e103) -- 0x55a437207ed0 con 0x55a437200eb0 2024-08-03T20:57:12.578 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:12.578+0000 7f9b93c00640 1 -- 172.21.15.32:0/3821133758 <== osd.6 v2:172.21.15.118:6825/801947386 23 ==== osd_op_reply(122 1.00000000 [delete] v103'222 uv222 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7f9b8c003040 con 0x55a437200eb0 2024-08-03T20:57:12.579 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:12.578+0000 7f9b99ffef40 1 -- 172.21.15.32:0/3821133758 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:123 12.1f 12:ff5b34d6:::1.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e103) -- 0x55a437207ed0 con 0x55a437200eb0 2024-08-03T20:57:12.579 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::1.00000000:head# 2024-08-03T20:57:12.580 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:12.579+0000 7f9b93c00640 1 -- 172.21.15.32:0/3821133758 <== osd.6 v2:172.21.15.118:6825/801947386 24 ==== osd_op_reply(123 1.00000000 [create] v103'223 uv223 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7f9b8c003040 con 0x55a437200eb0 2024-08-03T20:57:12.580 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:12.579+0000 7f9b99ffef40 1 -- 172.21.15.32:0/3821133758 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:124 12.1f 12:ff5b34d6:::1.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e103) -- 0x55a437207ed0 con 0x55a437200eb0 2024-08-03T20:57:12.581 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:12.581+0000 7f9b93c00640 1 -- 172.21.15.32:0/3821133758 <== osd.6 v2:172.21.15.118:6825/801947386 25 ==== osd_op_reply(124 1.00000000 [setxattr (30)] v103'224 uv224 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7f9b8c003040 con 0x55a437200eb0 2024-08-03T20:57:12.581 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:12.581+0000 7f9b99ffef40 1 -- 172.21.15.32:0/3821133758 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:125 12.1f 12:ff5b34d6:::1.00000000:head [setxattr parent (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e103) -- 0x55a437207eb0 con 0x55a437200eb0 2024-08-03T20:57:12.582 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:12.582+0000 7f9b93c00640 1 -- 172.21.15.32:0/3821133758 <== osd.6 v2:172.21.15.118:6825/801947386 26 ==== osd_op_reply(125 1.00000000 [setxattr (30)] v103'225 uv225 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7f9b8c003040 con 0x55a437200eb0 2024-08-03T20:57:12.582 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:12.582+0000 7f9b99ffef40 1 -- 172.21.15.32:0/3821133758 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:126 12.1f 12:ff5b34d6:::1.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e103) -- 0x55a437207eb0 con 0x55a437200eb0 2024-08-03T20:57:12.583 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:12.583+0000 7f9b93c00640 1 -- 172.21.15.32:0/3821133758 <== osd.6 v2:172.21.15.118:6825/801947386 27 ==== osd_op_reply(126 1.00000000 [omap-set-header] v103'226 uv226 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7f9b8c003040 con 0x55a437200eb0 2024-08-03T20:57:12.584 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:12.583+0000 7f9b99ffef40 1 -- 172.21.15.32:0/3821133758 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:127 12.1f 12:ff5b34d6:::1.00000000:head [omap-set-vals in=584b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e103) -- 0x55a43720bd20 con 0x55a437200eb0 2024-08-03T20:57:12.586 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:12.584+0000 7f9b93c00640 1 -- 172.21.15.32:0/3821133758 <== osd.6 v2:172.21.15.118:6825/801947386 28 ==== osd_op_reply(127 1.00000000 [omap-set-vals] v103'227 uv227 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7f9b8c003040 con 0x55a437200eb0 2024-08-03T20:57:12.586 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:12.584+0000 7f9b99ffef40 1 -- 172.21.15.32:0/3821133758 >> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] conn(0x55a4371298a0 msgr2=0x55a4371dcde0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:12.586 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:12.585+0000 7f9b99ffef40 1 --2- 172.21.15.32:0/3821133758 >> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] conn(0x55a4371298a0 0x55a4371dcde0 crc :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:12.586 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:12.585+0000 7f9b99ffef40 1 -- 172.21.15.32:0/3821133758 >> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] conn(0x55a4371fc990 msgr2=0x55a4371fcda0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:12.586 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:12.585+0000 7f9b99ffef40 1 --2- 172.21.15.32:0/3821133758 >> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] conn(0x55a4371fc990 0x55a4371fcda0 crc :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:12.586 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:12.585+0000 7f9b99ffef40 1 -- 172.21.15.32:0/3821133758 >> [v2:172.21.15.32:6800/9088303,v1:172.21.15.32:6801/9088303] conn(0x55a437208c60 msgr2=0x55a43720b080 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:12.586 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:12.585+0000 7f9b99ffef40 1 --2- 172.21.15.32:0/3821133758 >> [v2:172.21.15.32:6800/9088303,v1:172.21.15.32:6801/9088303] conn(0x55a437208c60 0x55a43720b080 crc :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:12.586 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:12.585+0000 7f9b99ffef40 1 -- 172.21.15.32:0/3821133758 >> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] conn(0x55a4371f83f0 msgr2=0x55a4371fa810 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:12.586 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:12.585+0000 7f9b99ffef40 1 --2- 172.21.15.32:0/3821133758 >> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] conn(0x55a4371f83f0 0x55a4371fa810 crc :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:12.587 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:12.585+0000 7f9b99ffef40 1 -- 172.21.15.32:0/3821133758 >> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] conn(0x55a437200eb0 msgr2=0x55a4372012c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:12.587 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:12.585+0000 7f9b99ffef40 1 --2- 172.21.15.32:0/3821133758 >> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] conn(0x55a437200eb0 0x55a4372012c0 crc :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:12.587 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:12.585+0000 7f9b99ffef40 1 -- 172.21.15.32:0/3821133758 >> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] conn(0x55a437204500 msgr2=0x55a437206920 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:12.587 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:12.585+0000 7f9b99ffef40 1 --2- 172.21.15.32:0/3821133758 >> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] conn(0x55a437204500 0x55a437206920 crc :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:12.587 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:12.585+0000 7f9b93c00640 1 -- 172.21.15.32:0/3821133758 reap_dead start 2024-08-03T20:57:12.587 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:12.585+0000 7f9b93c00640 1 -- 172.21.15.32:0/3821133758 reap_dead start 2024-08-03T20:57:12.587 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:12.585+0000 7f9b93c00640 1 -- 172.21.15.32:0/3821133758 reap_dead start 2024-08-03T20:57:12.587 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:12.585+0000 7f9b99ffef40 1 -- 172.21.15.32:0/3821133758 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f9b740759f0 msgr2=0x7f9b74077f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:12.587 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:12.585+0000 7f9b99ffef40 1 --2- 172.21.15.32:0/3821133758 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f9b740759f0 0x7f9b74077f00 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7f9b84056000 tx=0x7f9b84001a60 comp rx=0 tx=0).stop 2024-08-03T20:57:12.587 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:12.585+0000 7f9b99ffef40 1 -- 172.21.15.32:0/3821133758 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a43714de30 msgr2=0x55a43707ada0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:12.587 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:12.585+0000 7f9b99ffef40 1 --2- 172.21.15.32:0/3821133758 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a43714de30 0x55a43707ada0 secure :-1 s=READY pgs=1508 cs=0 l=1 rev1=1 crypto rx=0x7f9b80002d10 tx=0x7f9b8000c6b0 comp rx=0 tx=0).stop 2024-08-03T20:57:12.587 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:12.585+0000 7f9b99ffef40 1 -- 172.21.15.32:0/3821133758 shutdown_connections 2024-08-03T20:57:12.588 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:12.585+0000 7f9b99ffef40 1 --2- 172.21.15.32:0/3821133758 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f9b740759f0 0x7f9b74077f00 unknown :-1 s=CLOSED pgs=616 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:12.588 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:12.585+0000 7f9b99ffef40 1 --2- 172.21.15.32:0/3821133758 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a43714de30 0x55a43707ada0 unknown :-1 s=CLOSED pgs=1508 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:12.588 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:12.585+0000 7f9b99ffef40 1 -- 172.21.15.32:0/3821133758 >> 172.21.15.32:0/3821133758 conn(0x55a436f9cac0 msgr2=0x55a43714cef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:57:12.588 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:12.585+0000 7f9b99ffef40 1 -- 172.21.15.32:0/3821133758 shutdown_connections 2024-08-03T20:57:12.588 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:12.585+0000 7f9b99ffef40 1 -- 172.21.15.32:0/3821133758 wait complete. 2024-08-03T20:57:12.597 DEBUG:teuthology.orchestra.run.smithi032:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata put 500.00000000 - 2024-08-03T20:57:12.782 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:12.781+0000 7f6c04492f40 1 Processor -- start 2024-08-03T20:57:12.782 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:12.781+0000 7f6c04492f40 1 -- start start 2024-08-03T20:57:12.784 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:12.783+0000 7f6c04492f40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55f5d628f8e0 0x55f5d6305e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:12.784 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:12.783+0000 7f6c04492f40 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55f5d6312ba0 0x55f5d6273280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:12.784 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:12.783+0000 7f6c04492f40 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55f5d6324700 0x55f5d62737c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:12.784 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:12.783+0000 7f6c04492f40 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55f5d61f2bf0 con 0x55f5d6312ba0 2024-08-03T20:57:12.784 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:12.783+0000 7f6c04492f40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x55f5d61ff6b0 con 0x55f5d628f8e0 2024-08-03T20:57:12.784 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:12.783+0000 7f6c04492f40 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x55f5d6194e30 con 0x55f5d6324700 2024-08-03T20:57:12.784 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:12.783+0000 7f6bf7e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55f5d6312ba0 0x55f5d6273280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:12.784 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:12.783+0000 7f6bf7e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55f5d6312ba0 0x55f5d6273280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:33916/0 (socket says 172.21.15.32:33916) 2024-08-03T20:57:12.784 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:12.783+0000 7f6bf7e00640 1 -- 172.21.15.32:0/2163236286 learned_addr learned my addr 172.21.15.32:0/2163236286 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:57:12.784 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:12.783+0000 7f6bfe000640 1 --2- 172.21.15.32:0/2163236286 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55f5d6324700 0x55f5d62737c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:12.784 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:12.783+0000 7f6bfd600640 1 --2- 172.21.15.32:0/2163236286 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55f5d628f8e0 0x55f5d6305e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:12.786 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:12.784+0000 7f6bf7e00640 1 -- 172.21.15.32:0/2163236286 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55f5d6324700 msgr2=0x55f5d62737c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:57:12.786 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:12.784+0000 7f6bf7e00640 1 --2- 172.21.15.32:0/2163236286 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55f5d6324700 0x55f5d62737c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:12.786 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:12.784+0000 7f6bf7e00640 1 -- 172.21.15.32:0/2163236286 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55f5d628f8e0 msgr2=0x55f5d6305e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:57:12.786 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:12.784+0000 7f6bf7e00640 1 --2- 172.21.15.32:0/2163236286 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55f5d628f8e0 0x55f5d6305e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:12.786 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:12.784+0000 7f6bf7e00640 1 -- 172.21.15.32:0/2163236286 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f5d628e040 con 0x55f5d6312ba0 2024-08-03T20:57:12.787 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:12.784+0000 7f6bf7e00640 1 --2- 172.21.15.32:0/2163236286 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55f5d6312ba0 0x55f5d6273280 secure :-1 s=READY pgs=1510 cs=0 l=1 rev1=1 crypto rx=0x7f6be8001d40 tx=0x7f6be800b5b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7c9fb091bd8753e server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:57:12.787 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:12.784+0000 7f6bf7400640 1 -- 172.21.15.32:0/2163236286 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6be800bd80 con 0x55f5d6312ba0 2024-08-03T20:57:12.787 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:12.784+0000 7f6bf7400640 1 -- 172.21.15.32:0/2163236286 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f6be800f400 con 0x55f5d6312ba0 2024-08-03T20:57:12.787 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:12.784+0000 7f6bf7400640 1 -- 172.21.15.32:0/2163236286 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6be800f850 con 0x55f5d6312ba0 2024-08-03T20:57:12.787 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:12.785+0000 7f6c04492f40 1 -- 172.21.15.32:0/2163236286 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55f5d6312ba0 msgr2=0x55f5d6273280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:12.787 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:12.785+0000 7f6c04492f40 1 --2- 172.21.15.32:0/2163236286 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55f5d6312ba0 0x55f5d6273280 secure :-1 s=READY pgs=1510 cs=0 l=1 rev1=1 crypto rx=0x7f6be8001d40 tx=0x7f6be800b5b0 comp rx=0 tx=0).stop 2024-08-03T20:57:12.788 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:12.785+0000 7f6c04492f40 1 -- 172.21.15.32:0/2163236286 shutdown_connections 2024-08-03T20:57:12.788 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:12.785+0000 7f6c04492f40 1 --2- 172.21.15.32:0/2163236286 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55f5d6324700 0x55f5d62737c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:12.788 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:12.785+0000 7f6c04492f40 1 --2- 172.21.15.32:0/2163236286 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55f5d6312ba0 0x55f5d6273280 unknown :-1 s=CLOSED pgs=1510 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:12.788 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:12.785+0000 7f6c04492f40 1 --2- 172.21.15.32:0/2163236286 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55f5d628f8e0 0x55f5d6305e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:12.788 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:12.785+0000 7f6c04492f40 1 -- 172.21.15.32:0/2163236286 >> 172.21.15.32:0/2163236286 conn(0x55f5d6166ac0 msgr2=0x55f5d6311ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:57:12.788 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:12.785+0000 7f6c04492f40 1 -- 172.21.15.32:0/2163236286 shutdown_connections 2024-08-03T20:57:12.788 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:12.785+0000 7f6c04492f40 1 -- 172.21.15.32:0/2163236286 wait complete. 2024-08-03T20:57:12.791 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:12.786+0000 7f6c04492f40 1 Processor -- start 2024-08-03T20:57:12.791 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:12.786+0000 7f6c04492f40 1 -- start start 2024-08-03T20:57:12.791 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:12.788+0000 7f6c04492f40 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55f5d628f8e0 0x55f5d6301b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:12.791 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:12.788+0000 7f6c04492f40 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55f5d6312ba0 0x55f5d63020c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:12.792 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:12.788+0000 7f6c04492f40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55f5d6324700 0x55f5d6302600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:12.792 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:12.788+0000 7f6c04492f40 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55f5d628b250 con 0x55f5d628f8e0 2024-08-03T20:57:12.792 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:12.788+0000 7f6c04492f40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x55f5d6274980 con 0x55f5d6324700 2024-08-03T20:57:12.792 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:12.788+0000 7f6c04492f40 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x55f5d630f540 con 0x55f5d6312ba0 2024-08-03T20:57:12.792 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:12.788+0000 7f6bfd600640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55f5d628f8e0 0x55f5d6301b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:12.792 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:12.788+0000 7f6bfd600640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55f5d628f8e0 0x55f5d6301b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:33930/0 (socket says 172.21.15.32:33930) 2024-08-03T20:57:12.792 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:12.788+0000 7f6bfd600640 1 -- 172.21.15.32:0/3436405654 learned_addr learned my addr 172.21.15.32:0/3436405654 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:57:12.792 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:12.788+0000 7f6bfe000640 1 --2- 172.21.15.32:0/3436405654 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55f5d6324700 0x55f5d6302600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:12.792 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:12.788+0000 7f6bf7e00640 1 --2- 172.21.15.32:0/3436405654 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55f5d6312ba0 0x55f5d63020c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:12.792 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:12.788+0000 7f6bfd600640 1 -- 172.21.15.32:0/3436405654 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55f5d6312ba0 msgr2=0x55f5d63020c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:12.792 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:12.788+0000 7f6bfd600640 1 --2- 172.21.15.32:0/3436405654 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55f5d6312ba0 0x55f5d63020c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:12.793 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:12.788+0000 7f6bfd600640 1 -- 172.21.15.32:0/3436405654 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55f5d6324700 msgr2=0x55f5d6302600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:12.793 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:12.788+0000 7f6bfd600640 1 --2- 172.21.15.32:0/3436405654 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55f5d6324700 0x55f5d6302600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:12.793 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:12.788+0000 7f6bfd600640 1 -- 172.21.15.32:0/3436405654 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f5d63b48a0 con 0x55f5d628f8e0 2024-08-03T20:57:12.793 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:12.788+0000 7f6bfe000640 1 --2- 172.21.15.32:0/3436405654 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55f5d6324700 0x55f5d6302600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-03T20:57:12.793 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:12.788+0000 7f6bf7e00640 1 --2- 172.21.15.32:0/3436405654 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55f5d6312ba0 0x55f5d63020c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-03T20:57:12.793 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:12.789+0000 7f6bfd600640 1 --2- 172.21.15.32:0/3436405654 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55f5d628f8e0 0x55f5d6301b80 secure :-1 s=READY pgs=1511 cs=0 l=1 rev1=1 crypto rx=0x7f6bf000efd0 tx=0x7f6bf000c7a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:57:12.794 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:12.789+0000 7f6bf6000640 1 -- 172.21.15.32:0/3436405654 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6bf000f810 con 0x55f5d628f8e0 2024-08-03T20:57:12.794 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:12.789+0000 7f6bf6000640 1 -- 172.21.15.32:0/3436405654 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f6bf0013070 con 0x55f5d628f8e0 2024-08-03T20:57:12.794 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:12.789+0000 7f6c04492f40 1 -- 172.21.15.32:0/3436405654 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x55f5d6311400 con 0x55f5d628f8e0 2024-08-03T20:57:12.794 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:12.789+0000 7f6bf6000640 1 -- 172.21.15.32:0/3436405654 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6bf000fd40 con 0x55f5d628f8e0 2024-08-03T20:57:12.794 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:12.789+0000 7f6c04492f40 1 -- 172.21.15.32:0/3436405654 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f5d630f790 con 0x55f5d628f8e0 2024-08-03T20:57:12.796 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:12.791+0000 7f6bf6000640 1 -- 172.21.15.32:0/3436405654 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f6bf0018470 con 0x55f5d628f8e0 2024-08-03T20:57:12.796 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:12.792+0000 7f6bf6000640 1 --2- 172.21.15.32:0/3436405654 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f6bdc0759f0 0x7f6bdc077f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:12.796 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:12.792+0000 7f6bf6000640 1 -- 172.21.15.32:0/3436405654 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(103..103 src has 1..103) ==== 7402+0+0 (secure 0 0 0) 0x7f6bf0098750 con 0x55f5d628f8e0 2024-08-03T20:57:12.797 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:12.792+0000 7f6bf7e00640 1 --2- 172.21.15.32:0/3436405654 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f6bdc0759f0 0x7f6bdc077f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:12.797 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:12.792+0000 7f6c04492f40 1 --2- 172.21.15.32:0/3436405654 >> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] conn(0x55f5d63011a0 0x55f5d6289920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:12.797 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:12.792+0000 7f6c04492f40 1 -- 172.21.15.32:0/3436405654 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:1 12.15 12:aa448500:::500.00000000:head [writefull 0~80 in=80b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e103) -- 0x55f5d6289e60 con 0x55f5d63011a0 2024-08-03T20:57:12.797 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:12.793+0000 7f6bfe000640 1 --2- 172.21.15.32:0/3436405654 >> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] conn(0x55f5d63011a0 0x55f5d6289920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:12.797 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:12.793+0000 7f6bf7e00640 1 --2- 172.21.15.32:0/3436405654 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f6bdc0759f0 0x7f6bdc077f00 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7f6be8001d40 tx=0x7f6be8011040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:57:12.797 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:12.794+0000 7f6bfe000640 1 --2- 172.21.15.32:0/3436405654 >> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] conn(0x55f5d63011a0 0x55f5d6289920 crc :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:57:12.797 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:12.795+0000 7f6bfe000640 1 -- 172.21.15.32:0/3436405654 <== osd.6 v2:172.21.15.118:6825/801947386 1 ==== osd_op_reply(1 500.00000000 [writefull 0~80] v103'204 uv204 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f6bf805e040 con 0x55f5d63011a0 2024-08-03T20:57:12.797 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:12.796+0000 7f6c04492f40 1 -- 172.21.15.32:0/3436405654 >> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] conn(0x55f5d63011a0 msgr2=0x55f5d6289920 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:12.797 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:12.796+0000 7f6c04492f40 1 --2- 172.21.15.32:0/3436405654 >> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] conn(0x55f5d63011a0 0x55f5d6289920 crc :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:12.797 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:12.796+0000 7f6c04492f40 1 -- 172.21.15.32:0/3436405654 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f6bdc0759f0 msgr2=0x7f6bdc077f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:12.798 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:12.796+0000 7f6c04492f40 1 --2- 172.21.15.32:0/3436405654 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f6bdc0759f0 0x7f6bdc077f00 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7f6be8001d40 tx=0x7f6be8011040 comp rx=0 tx=0).stop 2024-08-03T20:57:12.798 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:12.796+0000 7f6c04492f40 1 -- 172.21.15.32:0/3436405654 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55f5d628f8e0 msgr2=0x55f5d6301b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:12.798 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:12.796+0000 7f6c04492f40 1 --2- 172.21.15.32:0/3436405654 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55f5d628f8e0 0x55f5d6301b80 secure :-1 s=READY pgs=1511 cs=0 l=1 rev1=1 crypto rx=0x7f6bf000efd0 tx=0x7f6bf000c7a0 comp rx=0 tx=0).stop 2024-08-03T20:57:12.798 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:12.796+0000 7f6bfe000640 1 -- 172.21.15.32:0/3436405654 reap_dead start 2024-08-03T20:57:12.798 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:12.796+0000 7f6c04492f40 1 -- 172.21.15.32:0/3436405654 shutdown_connections 2024-08-03T20:57:12.798 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:12.796+0000 7f6c04492f40 1 -- 172.21.15.32:0/3436405654 >> 172.21.15.32:0/3436405654 conn(0x55f5d6166ac0 msgr2=0x55f5d6311c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:57:12.798 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:12.796+0000 7f6c04492f40 1 -- 172.21.15.32:0/3436405654 shutdown_connections 2024-08-03T20:57:12.798 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:12.796+0000 7f6c04492f40 1 -- 172.21.15.32:0/3436405654 wait complete. 2024-08-03T20:57:12.811 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs joinable true 2024-08-03T20:57:12.942 INFO:tasks.ceph.mds.a.smithi032.stderr: -36> 2024-08-03T20:57:08.188+0000 7f4c6be00640 -1 log_channel(cluster) log [ERR] : missing journal object 2024-08-03T20:57:13.046 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:13.045+0000 7fc05a000640 1 -- 172.21.15.32:0/3953485868 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fc054145c50 msgr2=0x7fc05406e950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:13.046 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:13.045+0000 7fc05a000640 1 --2- 172.21.15.32:0/3953485868 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fc054145c50 0x7fc05406e950 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fc044009860 tx=0x7fc04402a5a0 comp rx=0 tx=0).stop 2024-08-03T20:57:13.046 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:13.046+0000 7fc05a000640 1 -- 172.21.15.32:0/3953485868 shutdown_connections 2024-08-03T20:57:13.047 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:13.046+0000 7fc05a000640 1 --2- 172.21.15.32:0/3953485868 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fc054145c50 0x7fc05406e950 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:13.047 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:13.046+0000 7fc05a000640 1 -- 172.21.15.32:0/3953485868 >> 172.21.15.32:0/3953485868 conn(0x7fc0540b1bd0 msgr2=0x7fc0540b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:57:13.047 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:13.046+0000 7fc05a000640 1 -- 172.21.15.32:0/3953485868 shutdown_connections 2024-08-03T20:57:13.047 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:13.046+0000 7fc05a000640 1 -- 172.21.15.32:0/3953485868 wait complete. 2024-08-03T20:57:13.048 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:13.048+0000 7fc05a000640 1 Processor -- start 2024-08-03T20:57:13.048 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:13.048+0000 7fc05a000640 1 -- start start 2024-08-03T20:57:13.051 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:13.050+0000 7fc05a000640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc0540cccf0 0x7fc05413e7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:13.051 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:13.050+0000 7fc05a000640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fc0540ced30 0x7fc0540c4e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:13.051 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:13.050+0000 7fc05a000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc054145c50 0x7fc0541597b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:13.052 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:13.050+0000 7fc05a000640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fc0540b4230 con 0x7fc054145c50 2024-08-03T20:57:13.052 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:13.050+0000 7fc05a000640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7fc0540b40b0 con 0x7fc0540cccf0 2024-08-03T20:57:13.052 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:13.050+0000 7fc05a000640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7fc0540b43b0 con 0x7fc0540ced30 2024-08-03T20:57:13.052 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:13.050+0000 7fc053e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc054145c50 0x7fc0541597b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:13.052 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:13.050+0000 7fc053e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc054145c50 0x7fc0541597b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:33932/0 (socket says 172.21.15.32:33932) 2024-08-03T20:57:13.052 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:13.050+0000 7fc053e00640 1 -- 172.21.15.32:0/1609690982 learned_addr learned my addr 172.21.15.32:0/1609690982 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:57:13.052 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:13.050+0000 7fc052a00640 1 --2- 172.21.15.32:0/1609690982 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fc0540ced30 0x7fc0540c4e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:13.052 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:13.050+0000 7fc053400640 1 --2- 172.21.15.32:0/1609690982 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc0540cccf0 0x7fc05413e7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:13.052 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:13.051+0000 7fc053e00640 1 -- 172.21.15.32:0/1609690982 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fc0540ced30 msgr2=0x7fc0540c4e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:13.052 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:13.051+0000 7fc053e00640 1 --2- 172.21.15.32:0/1609690982 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fc0540ced30 0x7fc0540c4e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:13.052 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:13.051+0000 7fc053e00640 1 -- 172.21.15.32:0/1609690982 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc0540cccf0 msgr2=0x7fc05413e7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:13.052 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:13.051+0000 7fc053e00640 1 --2- 172.21.15.32:0/1609690982 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc0540cccf0 0x7fc05413e7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:13.052 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:13.051+0000 7fc053e00640 1 -- 172.21.15.32:0/1609690982 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc0540c5380 con 0x7fc054145c50 2024-08-03T20:57:13.053 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:13.051+0000 7fc053e00640 1 --2- 172.21.15.32:0/1609690982 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc054145c50 0x7fc0541597b0 secure :-1 s=READY pgs=1514 cs=0 l=1 rev1=1 crypto rx=0x7fc04c00cb10 tx=0x7fc04c00a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:57:13.053 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:13.051+0000 7fc04b400640 1 -- 172.21.15.32:0/1609690982 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc04c016020 con 0x7fc054145c50 2024-08-03T20:57:13.053 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:13.051+0000 7fc04b400640 1 -- 172.21.15.32:0/1609690982 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fc04c00ad80 con 0x7fc054145c50 2024-08-03T20:57:13.054 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:13.051+0000 7fc04b400640 1 -- 172.21.15.32:0/1609690982 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc04c012680 con 0x7fc054145c50 2024-08-03T20:57:13.054 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:13.051+0000 7fc05a000640 1 -- 172.21.15.32:0/1609690982 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc0540c5890 con 0x7fc054145c50 2024-08-03T20:57:13.054 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:13.051+0000 7fc05a000640 1 -- 172.21.15.32:0/1609690982 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc0540c6390 con 0x7fc054145c50 2024-08-03T20:57:13.054 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:13.053+0000 7fc05a000640 1 -- 172.21.15.32:0/1609690982 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc0540c5db0 con 0x7fc054145c50 2024-08-03T20:57:13.054 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:13.054+0000 7fc04b400640 1 -- 172.21.15.32:0/1609690982 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7fc04c019020 con 0x7fc054145c50 2024-08-03T20:57:13.055 INFO:tasks.ceph.mds.a.smithi032.stdout:starting mds.a at 2024-08-03T20:57:13.056 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:13.055+0000 7fc04b400640 1 --2- 172.21.15.32:0/1609690982 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fc034077010 0x7fc034079520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:13.056 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:13.055+0000 7fc04b400640 1 -- 172.21.15.32:0/1609690982 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(103..103 src has 1..103) ==== 7402+0+0 (secure 0 0 0) 0x7fc04c099530 con 0x7fc054145c50 2024-08-03T20:57:13.056 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:13.055+0000 7fc053400640 1 --2- 172.21.15.32:0/1609690982 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fc034077010 0x7fc034079520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:13.056 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:13.056+0000 7fc053400640 1 --2- 172.21.15.32:0/1609690982 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fc034077010 0x7fc034079520 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7fc044034000 tx=0x7fc044004730 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:57:13.060 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:13.059+0000 7fc04b400640 1 -- 172.21.15.32:0/1609690982 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7fc0540c5db0 con 0x7fc054145c50 2024-08-03T20:57:13.310 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:13.309+0000 7fc05a000640 1 -- 172.21.15.32:0/1609690982 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "joinable", "val": "true"} v 0) -- 0x7fc0540c5620 con 0x7fc054145c50 2024-08-03T20:57:14.223 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:14.223+0000 7fc04b400640 1 -- 172.21.15.32:0/1609690982 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "joinable", "val": "true"}]=0 cephfs marked joinable; MDS may join as newly active. v201) ==== 162+0+0 (secure 0 0 0) 0x7fc04c020020 con 0x7fc054145c50 2024-08-03T20:57:14.224 INFO:teuthology.orchestra.run.smithi032.stderr:cephfs marked joinable; MDS may join as newly active. 2024-08-03T20:57:14.227 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:14.226+0000 7fc05a000640 1 -- 172.21.15.32:0/1609690982 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fc034077010 msgr2=0x7fc034079520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:14.227 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:14.226+0000 7fc05a000640 1 --2- 172.21.15.32:0/1609690982 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fc034077010 0x7fc034079520 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7fc044034000 tx=0x7fc044004730 comp rx=0 tx=0).stop 2024-08-03T20:57:14.227 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:14.226+0000 7fc05a000640 1 -- 172.21.15.32:0/1609690982 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc054145c50 msgr2=0x7fc0541597b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:14.228 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:14.226+0000 7fc05a000640 1 --2- 172.21.15.32:0/1609690982 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc054145c50 0x7fc0541597b0 secure :-1 s=READY pgs=1514 cs=0 l=1 rev1=1 crypto rx=0x7fc04c00cb10 tx=0x7fc04c00a060 comp rx=0 tx=0).stop 2024-08-03T20:57:14.228 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:14.226+0000 7fc05a000640 1 -- 172.21.15.32:0/1609690982 shutdown_connections 2024-08-03T20:57:14.228 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:14.226+0000 7fc05a000640 1 --2- 172.21.15.32:0/1609690982 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fc034077010 0x7fc034079520 unknown :-1 s=CLOSED pgs=618 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:14.228 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:14.226+0000 7fc05a000640 1 --2- 172.21.15.32:0/1609690982 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc054145c50 0x7fc0541597b0 unknown :-1 s=CLOSED pgs=1514 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:14.228 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:14.226+0000 7fc05a000640 1 --2- 172.21.15.32:0/1609690982 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fc0540ced30 0x7fc0540c4e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:14.228 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:14.226+0000 7fc05a000640 1 --2- 172.21.15.32:0/1609690982 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc0540cccf0 0x7fc05413e7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:14.228 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:14.226+0000 7fc05a000640 1 -- 172.21.15.32:0/1609690982 >> 172.21.15.32:0/1609690982 conn(0x7fc0540b1bd0 msgr2=0x7fc05415b470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:57:14.228 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:14.226+0000 7fc05a000640 1 -- 172.21.15.32:0/1609690982 shutdown_connections 2024-08-03T20:57:14.228 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:14.226+0000 7fc05a000640 1 -- 172.21.15.32:0/1609690982 wait complete. 2024-08-03T20:57:14.237 INFO:tasks.ceph.mds.a.smithi032.stderr:2024-08-03T20:57:14.236+0000 7ff12d200640 -1 mds.0.purge_queue operator(): Error -22 loading Journaler 2024-08-03T20:57:14.237 INFO:tasks.ceph.mds.a.smithi032.stderr:2024-08-03T20:57:14.236+0000 7ff12d200640 -1 mds.0.202 unhandled write error (22) Invalid argument, force readonly... 2024-08-03T20:57:14.262 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-08-03T20:57:14.494 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:14.494+0000 7f8fed200640 1 -- 172.21.15.32:0/64945455 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8fe80b93d0 msgr2=0x7f8fe80b3240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:14.495 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:14.494+0000 7f8fed200640 1 --2- 172.21.15.32:0/64945455 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8fe80b93d0 0x7f8fe80b3240 secure :-1 s=READY pgs=1516 cs=0 l=1 rev1=1 crypto rx=0x7f8fd0005fd0 tx=0x7f8fd002a580 comp rx=0 tx=0).stop 2024-08-03T20:57:14.495 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:14.494+0000 7f8fed200640 1 -- 172.21.15.32:0/64945455 shutdown_connections 2024-08-03T20:57:14.495 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:14.494+0000 7f8fed200640 1 --2- 172.21.15.32:0/64945455 >> v2:172.21.15.118:3300/0 conn(0x7f8fe80cf6f0 0x7f8fe80cfb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:14.495 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:14.494+0000 7f8fed200640 1 --2- 172.21.15.32:0/64945455 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8fe80b93d0 0x7f8fe80b3240 unknown :-1 s=CLOSED pgs=1516 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:14.495 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:14.494+0000 7f8fed200640 1 -- 172.21.15.32:0/64945455 >> 172.21.15.32:0/64945455 conn(0x7f8fe80d4080 msgr2=0x7f8fe80d4490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:57:14.495 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:14.494+0000 7f8fed200640 1 -- 172.21.15.32:0/64945455 shutdown_connections 2024-08-03T20:57:14.495 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:14.494+0000 7f8fed200640 1 -- 172.21.15.32:0/64945455 wait complete. 2024-08-03T20:57:14.496 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:14.496+0000 7f8fed200640 1 Processor -- start 2024-08-03T20:57:14.496 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:14.496+0000 7f8fed200640 1 -- start start 2024-08-03T20:57:14.499 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:14.498+0000 7f8fed200640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8fe80b93d0 0x7f8fe80a1980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:14.499 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:14.498+0000 7f8fed200640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f8fe80cf6f0 0x7f8fe809edc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:14.499 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:14.498+0000 7f8fed200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8fe80d4ba0 0x7f8fe8154490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:14.499 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:14.498+0000 7f8fed200640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f8fe80d08f0 con 0x7f8fe80d4ba0 2024-08-03T20:57:14.499 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:14.498+0000 7f8fed200640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f8fe80d0770 con 0x7f8fe80b93d0 2024-08-03T20:57:14.499 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:14.498+0000 7f8fed200640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7f8fe80d0a70 con 0x7f8fe80cf6f0 2024-08-03T20:57:14.499 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:14.498+0000 7f8fe7e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8fe80d4ba0 0x7f8fe8154490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:14.499 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:14.498+0000 7f8fe7e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8fe80d4ba0 0x7f8fe8154490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:33956/0 (socket says 172.21.15.32:33956) 2024-08-03T20:57:14.499 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:14.498+0000 7f8fe7e00640 1 -- 172.21.15.32:0/1395714345 learned_addr learned my addr 172.21.15.32:0/1395714345 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:57:14.499 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:14.498+0000 7f8fdfe00640 1 --2- 172.21.15.32:0/1395714345 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f8fe80cf6f0 0x7f8fe809edc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:14.499 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:14.498+0000 7f8fe7400640 1 --2- 172.21.15.32:0/1395714345 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8fe80b93d0 0x7f8fe80a1980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:14.500 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:14.498+0000 7f8fe7e00640 1 -- 172.21.15.32:0/1395714345 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f8fe80cf6f0 msgr2=0x7f8fe809edc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:14.500 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:14.498+0000 7f8fe7e00640 1 --2- 172.21.15.32:0/1395714345 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f8fe80cf6f0 0x7f8fe809edc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:14.500 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:14.498+0000 7f8fe7e00640 1 -- 172.21.15.32:0/1395714345 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8fe80b93d0 msgr2=0x7f8fe80a1980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:14.500 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:14.498+0000 7f8fe7e00640 1 --2- 172.21.15.32:0/1395714345 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8fe80b93d0 0x7f8fe80a1980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:14.500 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:14.498+0000 7f8fe7e00640 1 -- 172.21.15.32:0/1395714345 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8fe809f300 con 0x7f8fe80d4ba0 2024-08-03T20:57:14.500 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:14.498+0000 7f8fe7400640 1 --2- 172.21.15.32:0/1395714345 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8fe80b93d0 0x7f8fe80a1980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-03T20:57:14.501 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:14.499+0000 7f8fe7e00640 1 --2- 172.21.15.32:0/1395714345 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8fe80d4ba0 0x7f8fe8154490 secure :-1 s=READY pgs=1517 cs=0 l=1 rev1=1 crypto rx=0x7f8fe000e960 tx=0x7f8fe000ee20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:57:14.501 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:14.499+0000 7f8fdf400640 1 -- 172.21.15.32:0/1395714345 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8fe0004d20 con 0x7f8fe80d4ba0 2024-08-03T20:57:14.501 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:14.499+0000 7f8fdf400640 1 -- 172.21.15.32:0/1395714345 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f8fe000cb20 con 0x7f8fe80d4ba0 2024-08-03T20:57:14.501 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:14.499+0000 7f8fdf400640 1 -- 172.21.15.32:0/1395714345 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8fe0005020 con 0x7f8fe80d4ba0 2024-08-03T20:57:14.501 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:14.499+0000 7f8fed200640 1 -- 172.21.15.32:0/1395714345 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8fe80a1480 con 0x7f8fe80d4ba0 2024-08-03T20:57:14.501 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:14.499+0000 7f8fed200640 1 -- 172.21.15.32:0/1395714345 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8fe80c8010 con 0x7f8fe80d4ba0 2024-08-03T20:57:14.501 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:14.500+0000 7f8fed200640 1 -- 172.21.15.32:0/1395714345 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8fe80a11f0 con 0x7f8fe80d4ba0 2024-08-03T20:57:14.502 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:14.502+0000 7f8fdf400640 1 -- 172.21.15.32:0/1395714345 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f8fe00051c0 con 0x7f8fe80d4ba0 2024-08-03T20:57:14.503 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:14.502+0000 7f8fdf400640 1 --2- 172.21.15.32:0/1395714345 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f8fc4077040 0x7f8fc4079550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:14.503 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:14.503+0000 7f8fdf400640 1 -- 172.21.15.32:0/1395714345 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(103..103 src has 1..103) ==== 7402+0+0 (secure 0 0 0) 0x7f8fe0099780 con 0x7f8fe80d4ba0 2024-08-03T20:57:14.503 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:14.503+0000 7f8fe7400640 1 --2- 172.21.15.32:0/1395714345 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f8fc4077040 0x7f8fc4079550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:14.504 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:14.504+0000 7f8fe7400640 1 --2- 172.21.15.32:0/1395714345 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f8fc4077040 0x7f8fc4079550 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7f8fd8006fd0 tx=0x7f8fd8008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:57:14.507 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:14.507+0000 7f8fdf400640 1 -- 172.21.15.32:0/1395714345 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7f8fe0016030 con 0x7f8fe80d4ba0 2024-08-03T20:57:14.747 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:14.747+0000 7f8fed200640 1 -- 172.21.15.32:0/1395714345 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f8fe80d0f70 con 0x7f8fe80d4ba0 2024-08-03T20:57:14.748 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:14.748+0000 7f8fdf400640 1 -- 172.21.15.32:0/1395714345 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 202 v202) ==== 95+0+5226 (secure 0 0 0) 0x7f8fe0062360 con 0x7f8fe80d4ba0 2024-08-03T20:57:14.748 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-03T20:57:14.748 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":202,"btime":"2024-08-03T20:57:14:227034+0000","default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6439,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6835/531080160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6834","nonce":531080160},{"type":"v1","addr":"172.21.15.118:6835","nonce":531080160}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":84},{"gid":6454,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6837/2796882319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6836","nonce":2796882319},{"type":"v1","addr":"172.21.15.118:6837","nonce":2796882319}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":86},{"gid":6813,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6834/2686933890","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":2686933890},{"type":"v1","addr":"172.21.15.32:6834","nonce":2686933890}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":85}],"filesystems":[{"mdsmap":{"epoch":202,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-08-03T20:54:26.950518+0000","modified":"2024-08-03T20:57:14.227017+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":103,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8685},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8685":{"gid":8685,"name":"a","rank":0,"incarnation":202,"state":"up:replay","state_seq":1,"addr":"172.21.15.32:6835/2345472481","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6833","nonce":2345472481},{"type":"v1","addr":"172.21.15.32:6835","nonce":2345472481}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":6}]} 2024-08-03T20:57:14.749 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 202 2024-08-03T20:57:14.751 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:14.750+0000 7f8fed200640 1 -- 172.21.15.32:0/1395714345 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f8fc4077040 msgr2=0x7f8fc4079550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:14.751 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:14.750+0000 7f8fed200640 1 --2- 172.21.15.32:0/1395714345 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f8fc4077040 0x7f8fc4079550 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7f8fd8006fd0 tx=0x7f8fd8008040 comp rx=0 tx=0).stop 2024-08-03T20:57:14.751 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:14.750+0000 7f8fed200640 1 -- 172.21.15.32:0/1395714345 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8fe80d4ba0 msgr2=0x7f8fe8154490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:14.751 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:14.750+0000 7f8fed200640 1 --2- 172.21.15.32:0/1395714345 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8fe80d4ba0 0x7f8fe8154490 secure :-1 s=READY pgs=1517 cs=0 l=1 rev1=1 crypto rx=0x7f8fe000e960 tx=0x7f8fe000ee20 comp rx=0 tx=0).stop 2024-08-03T20:57:14.751 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:14.750+0000 7f8fed200640 1 -- 172.21.15.32:0/1395714345 shutdown_connections 2024-08-03T20:57:14.751 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:14.750+0000 7f8fed200640 1 --2- 172.21.15.32:0/1395714345 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f8fc4077040 0x7f8fc4079550 unknown :-1 s=CLOSED pgs=620 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:14.751 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:14.750+0000 7f8fed200640 1 --2- 172.21.15.32:0/1395714345 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8fe80d4ba0 0x7f8fe8154490 unknown :-1 s=CLOSED pgs=1517 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:14.751 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:14.750+0000 7f8fed200640 1 --2- 172.21.15.32:0/1395714345 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f8fe80cf6f0 0x7f8fe809edc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:14.751 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:14.750+0000 7f8fed200640 1 --2- 172.21.15.32:0/1395714345 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8fe80b93d0 0x7f8fe80a1980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:14.751 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:14.750+0000 7f8fed200640 1 -- 172.21.15.32:0/1395714345 >> 172.21.15.32:0/1395714345 conn(0x7f8fe80d4080 msgr2=0x7f8fe8161550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:57:14.751 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:14.750+0000 7f8fed200640 1 -- 172.21.15.32:0/1395714345 shutdown_connections 2024-08-03T20:57:14.752 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:14.750+0000 7f8fed200640 1 -- 172.21.15.32:0/1395714345 wait complete. 2024-08-03T20:57:15.176 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-08-03T20:57:15.410 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:15.409+0000 7fb947200640 1 -- 172.21.15.32:0/50111719 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb9400cffe0 msgr2=0x7fb9400b3e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:15.410 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:15.409+0000 7fb947200640 1 --2- 172.21.15.32:0/50111719 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb9400cffe0 0x7fb9400b3e90 secure :-1 s=READY pgs=1518 cs=0 l=1 rev1=1 crypto rx=0x7fb930005fd0 tx=0x7fb93002a5d0 comp rx=0 tx=0).stop 2024-08-03T20:57:15.410 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:15.409+0000 7fb947200640 1 -- 172.21.15.32:0/50111719 shutdown_connections 2024-08-03T20:57:15.410 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:15.409+0000 7fb947200640 1 --2- 172.21.15.32:0/50111719 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb9400cffe0 0x7fb9400b3e90 unknown :-1 s=CLOSED pgs=1518 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:15.410 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:15.409+0000 7fb947200640 1 --2- 172.21.15.32:0/50111719 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fb9400d26e0 0x7fb940161b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:15.410 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:15.409+0000 7fb947200640 1 -- 172.21.15.32:0/50111719 >> 172.21.15.32:0/50111719 conn(0x7fb94008d7a0 msgr2=0x7fb94009d3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:57:15.411 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:15.409+0000 7fb947200640 1 -- 172.21.15.32:0/50111719 shutdown_connections 2024-08-03T20:57:15.411 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:15.409+0000 7fb947200640 1 -- 172.21.15.32:0/50111719 wait complete. 2024-08-03T20:57:15.412 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:15.411+0000 7fb947200640 1 Processor -- start 2024-08-03T20:57:15.412 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:15.412+0000 7fb947200640 1 -- start start 2024-08-03T20:57:15.415 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:15.413+0000 7fb947200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb9400b43d0 0x7fb9400c7c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:15.415 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:15.413+0000 7fb947200640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb9400cffe0 0x7fb9400b7fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:15.415 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:15.413+0000 7fb947200640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fb9400d26e0 0x7fb9400b84e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:15.415 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:15.413+0000 7fb947200640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fb9400d4310 con 0x7fb9400b43d0 2024-08-03T20:57:15.415 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:15.413+0000 7fb947200640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7fb9400d4190 con 0x7fb9400cffe0 2024-08-03T20:57:15.415 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:15.413+0000 7fb947200640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7fb9400d4490 con 0x7fb9400d26e0 2024-08-03T20:57:15.415 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:15.413+0000 7fb945e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb9400b43d0 0x7fb9400c7c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:15.415 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:15.413+0000 7fb945e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb9400b43d0 0x7fb9400c7c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:33974/0 (socket says 172.21.15.32:33974) 2024-08-03T20:57:15.416 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:15.413+0000 7fb945e00640 1 -- 172.21.15.32:0/1723818105 learned_addr learned my addr 172.21.15.32:0/1723818105 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:57:15.416 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:15.413+0000 7fb945400640 1 --2- 172.21.15.32:0/1723818105 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb9400cffe0 0x7fb9400b7fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:15.416 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:15.413+0000 7fb946800640 1 --2- 172.21.15.32:0/1723818105 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fb9400d26e0 0x7fb9400b84e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:15.416 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:15.414+0000 7fb945e00640 1 -- 172.21.15.32:0/1723818105 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fb9400d26e0 msgr2=0x7fb9400b84e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:15.416 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:15.414+0000 7fb945e00640 1 --2- 172.21.15.32:0/1723818105 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fb9400d26e0 0x7fb9400b84e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:15.416 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:15.414+0000 7fb945e00640 1 -- 172.21.15.32:0/1723818105 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb9400cffe0 msgr2=0x7fb9400b7fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:15.416 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:15.414+0000 7fb945e00640 1 --2- 172.21.15.32:0/1723818105 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb9400cffe0 0x7fb9400b7fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:15.416 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:15.414+0000 7fb945e00640 1 -- 172.21.15.32:0/1723818105 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb94014f9d0 con 0x7fb9400b43d0 2024-08-03T20:57:15.416 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:15.414+0000 7fb945400640 1 --2- 172.21.15.32:0/1723818105 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb9400cffe0 0x7fb9400b7fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-03T20:57:15.416 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:15.414+0000 7fb945e00640 1 --2- 172.21.15.32:0/1723818105 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb9400b43d0 0x7fb9400c7c30 secure :-1 s=READY pgs=1519 cs=0 l=1 rev1=1 crypto rx=0x7fb93400e790 tx=0x7fb93400ec50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:57:15.416 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:15.414+0000 7fb93a000640 1 -- 172.21.15.32:0/1723818105 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb93400eea0 con 0x7fb9400b43d0 2024-08-03T20:57:15.416 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:15.414+0000 7fb93a000640 1 -- 172.21.15.32:0/1723818105 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fb93400cdb0 con 0x7fb9400b43d0 2024-08-03T20:57:15.417 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:15.414+0000 7fb93a000640 1 -- 172.21.15.32:0/1723818105 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb934010690 con 0x7fb9400b43d0 2024-08-03T20:57:15.417 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:15.414+0000 7fb947200640 1 -- 172.21.15.32:0/1723818105 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb94014fcb0 con 0x7fb9400b43d0 2024-08-03T20:57:15.417 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:15.415+0000 7fb947200640 1 -- 172.21.15.32:0/1723818105 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb940150270 con 0x7fb9400b43d0 2024-08-03T20:57:15.417 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:15.416+0000 7fb947200640 1 -- 172.21.15.32:0/1723818105 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb9400d4940 con 0x7fb9400b43d0 2024-08-03T20:57:15.417 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:15.417+0000 7fb93a000640 1 -- 172.21.15.32:0/1723818105 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7fb934010830 con 0x7fb9400b43d0 2024-08-03T20:57:15.419 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:15.418+0000 7fb93a000640 1 --2- 172.21.15.32:0/1723818105 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fb924077010 0x7fb924079520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:15.419 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:15.419+0000 7fb93a000640 1 -- 172.21.15.32:0/1723818105 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(103..103 src has 1..103) ==== 7402+0+0 (secure 0 0 0) 0x7fb934099850 con 0x7fb9400b43d0 2024-08-03T20:57:15.419 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:15.419+0000 7fb945400640 1 --2- 172.21.15.32:0/1723818105 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fb924077010 0x7fb924079520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:15.420 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:15.420+0000 7fb945400640 1 --2- 172.21.15.32:0/1723818105 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fb924077010 0x7fb924079520 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7fb930033000 tx=0x7fb930004730 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:57:15.423 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:15.423+0000 7fb93a000640 1 -- 172.21.15.32:0/1723818105 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7fb934061d90 con 0x7fb9400b43d0 2024-08-03T20:57:15.669 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:15.668+0000 7fb947200640 1 -- 172.21.15.32:0/1723818105 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb94015fb90 con 0x7fb9400b43d0 2024-08-03T20:57:15.669 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:15.669+0000 7fb93a000640 1 -- 172.21.15.32:0/1723818105 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 203 v203) ==== 95+0+5229 (secure 0 0 0) 0x7fb9340662e0 con 0x7fb9400b43d0 2024-08-03T20:57:15.670 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-03T20:57:15.670 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":203,"btime":"2024-08-03T20:57:15:229294+0000","default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6439,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6835/531080160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6834","nonce":531080160},{"type":"v1","addr":"172.21.15.118:6835","nonce":531080160}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":84},{"gid":6454,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6837/2796882319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6836","nonce":2796882319},{"type":"v1","addr":"172.21.15.118:6837","nonce":2796882319}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":86},{"gid":6813,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6834/2686933890","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":2686933890},{"type":"v1","addr":"172.21.15.32:6834","nonce":2686933890}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":85}],"filesystems":[{"mdsmap":{"epoch":203,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-08-03T20:54:26.950518+0000","modified":"2024-08-03T20:57:14.243811+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":103,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8685},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8685":{"gid":8685,"name":"a","rank":0,"incarnation":202,"state":"up:reconnect","state_seq":2,"addr":"172.21.15.32:6835/2345472481","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6833","nonce":2345472481},{"type":"v1","addr":"172.21.15.32:6835","nonce":2345472481}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":6}]} 2024-08-03T20:57:15.670 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 203 2024-08-03T20:57:15.672 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:15.671+0000 7fb947200640 1 -- 172.21.15.32:0/1723818105 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fb924077010 msgr2=0x7fb924079520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:15.672 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:15.671+0000 7fb947200640 1 --2- 172.21.15.32:0/1723818105 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fb924077010 0x7fb924079520 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7fb930033000 tx=0x7fb930004730 comp rx=0 tx=0).stop 2024-08-03T20:57:15.672 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:15.671+0000 7fb947200640 1 -- 172.21.15.32:0/1723818105 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb9400b43d0 msgr2=0x7fb9400c7c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:15.672 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:15.671+0000 7fb947200640 1 --2- 172.21.15.32:0/1723818105 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb9400b43d0 0x7fb9400c7c30 secure :-1 s=READY pgs=1519 cs=0 l=1 rev1=1 crypto rx=0x7fb93400e790 tx=0x7fb93400ec50 comp rx=0 tx=0).stop 2024-08-03T20:57:15.672 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:15.671+0000 7fb947200640 1 -- 172.21.15.32:0/1723818105 shutdown_connections 2024-08-03T20:57:15.672 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:15.671+0000 7fb947200640 1 --2- 172.21.15.32:0/1723818105 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fb924077010 0x7fb924079520 unknown :-1 s=CLOSED pgs=621 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:15.672 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:15.671+0000 7fb947200640 1 --2- 172.21.15.32:0/1723818105 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb9400cffe0 0x7fb9400b7fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:15.672 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:15.671+0000 7fb947200640 1 --2- 172.21.15.32:0/1723818105 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fb9400d26e0 0x7fb9400b84e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:15.673 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:15.671+0000 7fb947200640 1 --2- 172.21.15.32:0/1723818105 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb9400b43d0 0x7fb9400c7c30 unknown :-1 s=CLOSED pgs=1519 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:15.673 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:15.671+0000 7fb947200640 1 -- 172.21.15.32:0/1723818105 >> 172.21.15.32:0/1723818105 conn(0x7fb94008d7a0 msgr2=0x7fb9401602f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:57:15.673 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:15.671+0000 7fb947200640 1 -- 172.21.15.32:0/1723818105 shutdown_connections 2024-08-03T20:57:15.673 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:15.671+0000 7fb947200640 1 -- 172.21.15.32:0/1723818105 wait complete. 2024-08-03T20:57:15.698 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 203, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-08-03T20:54:26.950518+0000', 'modified': '2024-08-03T20:57:14.243811+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 103, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 8685}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8685': {'gid': 8685, 'name': 'a', 'rank': 0, 'incarnation': 202, 'state': 'up:reconnect', 'state_seq': 2, 'addr': '172.21.15.32:6835/2345472481', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6833', 'nonce': 2345472481}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 2345472481}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [13], 'metadata_pool': 12, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 0, 'qdb_cluster': []} 2024-08-03T20:57:15.698 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8685:up:reconnect 2024-08-03T20:57:15.698 DEBUG:tasks.ceph_test_case:wait_until_true: waiting (timeout=60 retry_count=0)... 2024-08-03T20:57:18.224 INFO:tasks.ceph.mds.a.smithi032.stderr:2024-08-03T20:57:18.223+0000 7ff130e00640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-08-03T20:57:20.699 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-08-03T20:57:20.940 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:20.940+0000 7f943dc00640 1 -- 172.21.15.32:0/656973425 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f941c004e60 msgr2=0x7f941c0052b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:20.941 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:20.940+0000 7f943dc00640 1 --2- 172.21.15.32:0/656973425 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f941c004e60 0x7f941c0052b0 secure :-1 s=READY pgs=1521 cs=0 l=1 rev1=1 crypto rx=0x7f9430001d90 tx=0x7f941c005830 comp rx=0 tx=0).stop 2024-08-03T20:57:20.941 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:20.940+0000 7f943d200640 1 -- 172.21.15.32:0/656973425 reap_dead start 2024-08-03T20:57:20.941 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:20.940+0000 7f943dc00640 1 -- 172.21.15.32:0/656973425 shutdown_connections 2024-08-03T20:57:20.941 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:20.940+0000 7f943dc00640 1 -- 172.21.15.32:0/656973425 >> 172.21.15.32:0/656973425 conn(0x7f94380d4090 msgr2=0x7f94380d44a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:57:20.941 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:20.940+0000 7f943dc00640 1 -- 172.21.15.32:0/656973425 shutdown_connections 2024-08-03T20:57:20.941 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:20.940+0000 7f943dc00640 1 -- 172.21.15.32:0/656973425 wait complete. 2024-08-03T20:57:20.942 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:20.942+0000 7f943dc00640 1 Processor -- start 2024-08-03T20:57:20.943 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:20.942+0000 7f943dc00640 1 -- start start 2024-08-03T20:57:20.948 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:20.944+0000 7f943dc00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f941c004e60 0x7f94381deb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:20.948 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:20.944+0000 7f943dc00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f94380d0a60 0x7f94380d0e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:20.948 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:20.944+0000 7f943dc00640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f94381e5fa0 0x7f94381e63b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:20.948 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:20.944+0000 7f943dc00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f94380cf8d0 con 0x7f94380d0a60 2024-08-03T20:57:20.949 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:20.944+0000 7f943dc00640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f94380cf750 con 0x7f941c004e60 2024-08-03T20:57:20.949 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:20.944+0000 7f943dc00640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7f94380cfa50 con 0x7f94381e5fa0 2024-08-03T20:57:20.949 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:20.944+0000 7f942f400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f94380d0a60 0x7f94380d0e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:20.949 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:20.944+0000 7f942f400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f94380d0a60 0x7f94380d0e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:59418/0 (socket says 172.21.15.32:59418) 2024-08-03T20:57:20.949 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:20.944+0000 7f942f400640 1 -- 172.21.15.32:0/2060486627 learned_addr learned my addr 172.21.15.32:0/2060486627 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:57:20.949 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:20.944+0000 7f9437e00640 1 --2- 172.21.15.32:0/2060486627 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f941c004e60 0x7f94381deb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:20.950 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:20.944+0000 7f943d200640 1 --2- 172.21.15.32:0/2060486627 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f94381e5fa0 0x7f94381e63b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:20.950 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:20.944+0000 7f942f400640 1 -- 172.21.15.32:0/2060486627 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f94381e5fa0 msgr2=0x7f94381e63b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:20.950 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:20.944+0000 7f942f400640 1 --2- 172.21.15.32:0/2060486627 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f94381e5fa0 0x7f94381e63b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:20.950 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:20.944+0000 7f942f400640 1 -- 172.21.15.32:0/2060486627 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f941c004e60 msgr2=0x7f94381deb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:20.950 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:20.944+0000 7f942f400640 1 --2- 172.21.15.32:0/2060486627 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f941c004e60 0x7f94381deb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:20.950 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:20.945+0000 7f942f400640 1 -- 172.21.15.32:0/2060486627 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f94381e68f0 con 0x7f94380d0a60 2024-08-03T20:57:20.951 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:20.945+0000 7f942f400640 1 --2- 172.21.15.32:0/2060486627 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f94380d0a60 0x7f94380d0e70 secure :-1 s=READY pgs=1522 cs=0 l=1 rev1=1 crypto rx=0x7f942000e630 tx=0x7f942000eaf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:57:20.951 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:20.945+0000 7f9434c00640 1 -- 172.21.15.32:0/2060486627 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9420015020 con 0x7f94380d0a60 2024-08-03T20:57:20.951 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:20.945+0000 7f9434c00640 1 -- 172.21.15.32:0/2060486627 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f942000ca90 con 0x7f94380d0a60 2024-08-03T20:57:20.951 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:20.945+0000 7f9434c00640 1 -- 172.21.15.32:0/2060486627 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9420011460 con 0x7f94380d0a60 2024-08-03T20:57:20.951 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:20.945+0000 7f943dc00640 1 -- 172.21.15.32:0/2060486627 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f94381f8df0 con 0x7f94380d0a60 2024-08-03T20:57:20.951 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:20.945+0000 7f943dc00640 1 -- 172.21.15.32:0/2060486627 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f94381f93b0 con 0x7f94380d0a60 2024-08-03T20:57:20.952 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:20.946+0000 7f943dc00640 1 -- 172.21.15.32:0/2060486627 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f94380b8d90 con 0x7f94380d0a60 2024-08-03T20:57:20.952 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:20.948+0000 7f9434c00640 1 -- 172.21.15.32:0/2060486627 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f9420018020 con 0x7f94380d0a60 2024-08-03T20:57:20.953 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:20.949+0000 7f9434c00640 1 --2- 172.21.15.32:0/2060486627 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f9414077040 0x7f9414079550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:20.953 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:20.949+0000 7f9434c00640 1 -- 172.21.15.32:0/2060486627 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(103..103 src has 1..103) ==== 7402+0+0 (secure 0 0 0) 0x7f9420099600 con 0x7f94380d0a60 2024-08-03T20:57:20.953 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:20.949+0000 7f9437e00640 1 --2- 172.21.15.32:0/2060486627 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f9414077040 0x7f9414079550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:20.953 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:20.950+0000 7f9437e00640 1 --2- 172.21.15.32:0/2060486627 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f9414077040 0x7f9414079550 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7f9428003640 tx=0x7f942800b260 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:57:20.953 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:20.953+0000 7f9434c00640 1 -- 172.21.15.32:0/2060486627 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7f942001f020 con 0x7f94380d0a60 2024-08-03T20:57:21.196 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:21.196+0000 7f943dc00640 1 -- 172.21.15.32:0/2060486627 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f94380d1490 con 0x7f94380d0a60 2024-08-03T20:57:21.197 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:21.196+0000 7f9434c00640 1 -- 172.21.15.32:0/2060486627 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 206 v206) ==== 95+0+5233 (secure 0 0 0) 0x7f94380d1490 con 0x7f94380d0a60 2024-08-03T20:57:21.197 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-03T20:57:21.197 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":206,"btime":"2024-08-03T20:57:20:251720+0000","default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6439,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6835/531080160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6834","nonce":531080160},{"type":"v1","addr":"172.21.15.118:6835","nonce":531080160}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":84},{"gid":6454,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6837/2796882319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6836","nonce":2796882319},{"type":"v1","addr":"172.21.15.118:6837","nonce":2796882319}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":86},{"gid":6813,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6834/2686933890","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":2686933890},{"type":"v1","addr":"172.21.15.32:6834","nonce":2686933890}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":85}],"filesystems":[{"mdsmap":{"epoch":206,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-08-03T20:54:26.950518+0000","modified":"2024-08-03T20:57:20.242553+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":103,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8685},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8685":{"gid":8685,"name":"a","rank":0,"incarnation":202,"state":"up:active","state_seq":5,"addr":"172.21.15.32:6835/2345472481","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6833","nonce":2345472481},{"type":"v1","addr":"172.21.15.32:6835","nonce":2345472481}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8685,"qdb_cluster":[8685]},"id":6}]} 2024-08-03T20:57:21.197 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 206 2024-08-03T20:57:21.199 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:21.199+0000 7f943dc00640 1 -- 172.21.15.32:0/2060486627 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f9414077040 msgr2=0x7f9414079550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:21.200 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:21.199+0000 7f943dc00640 1 --2- 172.21.15.32:0/2060486627 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f9414077040 0x7f9414079550 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7f9428003640 tx=0x7f942800b260 comp rx=0 tx=0).stop 2024-08-03T20:57:21.200 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:21.199+0000 7f943dc00640 1 -- 172.21.15.32:0/2060486627 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f94380d0a60 msgr2=0x7f94380d0e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:21.200 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:21.199+0000 7f943dc00640 1 --2- 172.21.15.32:0/2060486627 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f94380d0a60 0x7f94380d0e70 secure :-1 s=READY pgs=1522 cs=0 l=1 rev1=1 crypto rx=0x7f942000e630 tx=0x7f942000eaf0 comp rx=0 tx=0).stop 2024-08-03T20:57:21.200 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:21.199+0000 7f943dc00640 1 -- 172.21.15.32:0/2060486627 shutdown_connections 2024-08-03T20:57:21.200 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:21.199+0000 7f943dc00640 1 --2- 172.21.15.32:0/2060486627 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f9414077040 0x7f9414079550 unknown :-1 s=CLOSED pgs=622 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:21.200 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:21.199+0000 7f943dc00640 1 --2- 172.21.15.32:0/2060486627 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f94381e5fa0 0x7f94381e63b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:21.200 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:21.199+0000 7f943dc00640 1 --2- 172.21.15.32:0/2060486627 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f94380d0a60 0x7f94380d0e70 unknown :-1 s=CLOSED pgs=1522 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:21.200 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:21.199+0000 7f943dc00640 1 --2- 172.21.15.32:0/2060486627 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f941c004e60 0x7f94381deb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:21.200 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:21.199+0000 7f943dc00640 1 -- 172.21.15.32:0/2060486627 >> 172.21.15.32:0/2060486627 conn(0x7f94380d4090 msgr2=0x7f94380b1520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:57:21.200 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:21.199+0000 7f943dc00640 1 -- 172.21.15.32:0/2060486627 shutdown_connections 2024-08-03T20:57:21.200 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:21.199+0000 7f943dc00640 1 -- 172.21.15.32:0/2060486627 wait complete. 2024-08-03T20:57:21.624 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-08-03T20:57:21.854 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:21.853+0000 7fc012200640 1 -- 172.21.15.32:0/2117060303 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc00c0b38b0 msgr2=0x7fc00c1523f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:21.855 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:21.853+0000 7fc012200640 1 --2- 172.21.15.32:0/2117060303 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc00c0b38b0 0x7fc00c1523f0 secure :-1 s=READY pgs=1523 cs=0 l=1 rev1=1 crypto rx=0x7fc00c0cd3f0 tx=0x7fbffc02a4d0 comp rx=0 tx=0).stop 2024-08-03T20:57:21.855 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:21.854+0000 7fc012200640 1 -- 172.21.15.32:0/2117060303 shutdown_connections 2024-08-03T20:57:21.855 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:21.854+0000 7fc012200640 1 --2- 172.21.15.32:0/2117060303 >> v2:172.21.15.118:3300/0 conn(0x7fc00c15f610 0x7fc00c0cdf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:21.855 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:21.854+0000 7fc012200640 1 --2- 172.21.15.32:0/2117060303 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc00c0b38b0 0x7fc00c1523f0 unknown :-1 s=CLOSED pgs=1523 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:21.855 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:21.854+0000 7fc012200640 1 -- 172.21.15.32:0/2117060303 >> 172.21.15.32:0/2117060303 conn(0x7fc00c0b1bd0 msgr2=0x7fc00c0b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:57:21.855 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:21.854+0000 7fc012200640 1 -- 172.21.15.32:0/2117060303 shutdown_connections 2024-08-03T20:57:21.855 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:21.854+0000 7fc012200640 1 -- 172.21.15.32:0/2117060303 wait complete. 2024-08-03T20:57:21.856 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:21.856+0000 7fc012200640 1 Processor -- start 2024-08-03T20:57:21.856 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:21.856+0000 7fc012200640 1 -- start start 2024-08-03T20:57:21.859 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:21.857+0000 7fc012200640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fc00c0b38b0 0x7fc00c0b0710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:21.859 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:21.857+0000 7fc012200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc00c0b4ce0 0x7fc00c0b0c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:21.859 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:21.857+0000 7fc012200640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc00c14d680 0x7fc00c1f6db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:21.859 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:21.857+0000 7fc012200640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fc00c0cea30 con 0x7fc00c0b4ce0 2024-08-03T20:57:21.859 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:21.858+0000 7fc012200640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7fc00c0ce8b0 con 0x7fc00c14d680 2024-08-03T20:57:21.859 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:21.858+0000 7fc012200640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7fc00c0cebb0 con 0x7fc00c0b38b0 2024-08-03T20:57:21.859 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:21.858+0000 7fc00aa00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc00c0b4ce0 0x7fc00c0b0c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:21.859 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:21.858+0000 7fc00aa00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc00c0b4ce0 0x7fc00c0b0c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:59428/0 (socket says 172.21.15.32:59428) 2024-08-03T20:57:21.859 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:21.858+0000 7fc00aa00640 1 -- 172.21.15.32:0/3958712221 learned_addr learned my addr 172.21.15.32:0/3958712221 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:57:21.860 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:21.858+0000 7fc00b400640 1 --2- 172.21.15.32:0/3958712221 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fc00c0b38b0 0x7fc00c0b0710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:21.860 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:21.858+0000 7fc00be00640 1 --2- 172.21.15.32:0/3958712221 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc00c14d680 0x7fc00c1f6db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:21.860 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:21.858+0000 7fc00aa00640 1 -- 172.21.15.32:0/3958712221 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fc00c0b38b0 msgr2=0x7fc00c0b0710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:21.860 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:21.858+0000 7fc00aa00640 1 --2- 172.21.15.32:0/3958712221 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fc00c0b38b0 0x7fc00c0b0710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:21.860 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:21.858+0000 7fc00aa00640 1 -- 172.21.15.32:0/3958712221 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc00c14d680 msgr2=0x7fc00c1f6db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:21.860 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:21.858+0000 7fc00aa00640 1 --2- 172.21.15.32:0/3958712221 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc00c14d680 0x7fc00c1f6db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:21.860 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:21.858+0000 7fc00aa00640 1 -- 172.21.15.32:0/3958712221 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc00c1e3e10 con 0x7fc00c0b4ce0 2024-08-03T20:57:21.860 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:21.858+0000 7fc00aa00640 1 --2- 172.21.15.32:0/3958712221 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc00c0b4ce0 0x7fc00c0b0c50 secure :-1 s=READY pgs=1524 cs=0 l=1 rev1=1 crypto rx=0x7fbff800b3f0 tx=0x7fbff800b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:57:21.861 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:21.859+0000 7fc003400640 1 -- 172.21.15.32:0/3958712221 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbff8015020 con 0x7fc00c0b4ce0 2024-08-03T20:57:21.861 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:21.859+0000 7fc003400640 1 -- 172.21.15.32:0/3958712221 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fbff8011920 con 0x7fc00c0b4ce0 2024-08-03T20:57:21.861 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:21.859+0000 7fc012200640 1 -- 172.21.15.32:0/3958712221 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc00c1e40f0 con 0x7fc00c0b4ce0 2024-08-03T20:57:21.861 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:21.859+0000 7fc003400640 1 -- 172.21.15.32:0/3958712221 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbff8011d70 con 0x7fc00c0b4ce0 2024-08-03T20:57:21.861 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:21.859+0000 7fc012200640 1 -- 172.21.15.32:0/3958712221 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc00c1e46b0 con 0x7fc00c0b4ce0 2024-08-03T20:57:21.861 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:21.860+0000 7fc012200640 1 -- 172.21.15.32:0/3958712221 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc00c0cf0f0 con 0x7fc00c0b4ce0 2024-08-03T20:57:21.862 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:21.861+0000 7fc003400640 1 -- 172.21.15.32:0/3958712221 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7fbff8018020 con 0x7fc00c0b4ce0 2024-08-03T20:57:21.862 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:21.862+0000 7fc003400640 1 --2- 172.21.15.32:0/3958712221 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fbfec077040 0x7fbfec079550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:21.863 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:21.862+0000 7fc003400640 1 -- 172.21.15.32:0/3958712221 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(103..103 src has 1..103) ==== 7402+0+0 (secure 0 0 0) 0x7fbff8098b90 con 0x7fc00c0b4ce0 2024-08-03T20:57:21.863 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:21.863+0000 7fc00b400640 1 --2- 172.21.15.32:0/3958712221 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fbfec077040 0x7fbfec079550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:21.864 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:21.864+0000 7fc00b400640 1 --2- 172.21.15.32:0/3958712221 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fbfec077040 0x7fbfec079550 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7fbffc033000 tx=0x7fbffc004730 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:57:21.867 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:21.866+0000 7fc003400640 1 -- 172.21.15.32:0/3958712221 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7fbff801f030 con 0x7fc00c0b4ce0 2024-08-03T20:57:22.108 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:22.107+0000 7fc012200640 1 -- 172.21.15.32:0/3958712221 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc00c15e960 con 0x7fc00c0b4ce0 2024-08-03T20:57:22.108 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:22.108+0000 7fc003400640 1 -- 172.21.15.32:0/3958712221 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 206 v206) ==== 95+0+5233 (secure 0 0 0) 0x7fbff80616e0 con 0x7fc00c0b4ce0 2024-08-03T20:57:22.109 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-03T20:57:22.109 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":206,"btime":"2024-08-03T20:57:20:251720+0000","default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6439,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6835/531080160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6834","nonce":531080160},{"type":"v1","addr":"172.21.15.118:6835","nonce":531080160}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":84},{"gid":6454,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6837/2796882319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6836","nonce":2796882319},{"type":"v1","addr":"172.21.15.118:6837","nonce":2796882319}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":86},{"gid":6813,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6834/2686933890","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":2686933890},{"type":"v1","addr":"172.21.15.32:6834","nonce":2686933890}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":85}],"filesystems":[{"mdsmap":{"epoch":206,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-08-03T20:54:26.950518+0000","modified":"2024-08-03T20:57:20.242553+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":103,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8685},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8685":{"gid":8685,"name":"a","rank":0,"incarnation":202,"state":"up:active","state_seq":5,"addr":"172.21.15.32:6835/2345472481","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6833","nonce":2345472481},{"type":"v1","addr":"172.21.15.32:6835","nonce":2345472481}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8685,"qdb_cluster":[8685]},"id":6}]} 2024-08-03T20:57:22.109 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 206 2024-08-03T20:57:22.111 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:22.110+0000 7fc012200640 1 -- 172.21.15.32:0/3958712221 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fbfec077040 msgr2=0x7fbfec079550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:22.111 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:22.110+0000 7fc012200640 1 --2- 172.21.15.32:0/3958712221 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fbfec077040 0x7fbfec079550 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7fbffc033000 tx=0x7fbffc004730 comp rx=0 tx=0).stop 2024-08-03T20:57:22.111 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:22.110+0000 7fc012200640 1 -- 172.21.15.32:0/3958712221 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc00c0b4ce0 msgr2=0x7fc00c0b0c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:22.111 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:22.110+0000 7fc012200640 1 --2- 172.21.15.32:0/3958712221 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc00c0b4ce0 0x7fc00c0b0c50 secure :-1 s=READY pgs=1524 cs=0 l=1 rev1=1 crypto rx=0x7fbff800b3f0 tx=0x7fbff800b8b0 comp rx=0 tx=0).stop 2024-08-03T20:57:22.111 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:22.110+0000 7fc012200640 1 -- 172.21.15.32:0/3958712221 shutdown_connections 2024-08-03T20:57:22.111 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:22.110+0000 7fc012200640 1 --2- 172.21.15.32:0/3958712221 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fbfec077040 0x7fbfec079550 unknown :-1 s=CLOSED pgs=623 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:22.111 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:22.110+0000 7fc012200640 1 --2- 172.21.15.32:0/3958712221 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc00c14d680 0x7fc00c1f6db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:22.111 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:22.110+0000 7fc012200640 1 --2- 172.21.15.32:0/3958712221 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc00c0b4ce0 0x7fc00c0b0c50 unknown :-1 s=CLOSED pgs=1524 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:22.112 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:22.110+0000 7fc012200640 1 --2- 172.21.15.32:0/3958712221 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fc00c0b38b0 0x7fc00c0b0710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:22.112 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:22.110+0000 7fc012200640 1 -- 172.21.15.32:0/3958712221 >> 172.21.15.32:0/3958712221 conn(0x7fc00c0b1bd0 msgr2=0x7fc00c06f1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:57:22.112 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:22.110+0000 7fc012200640 1 -- 172.21.15.32:0/3958712221 shutdown_connections 2024-08-03T20:57:22.112 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:22.110+0000 7fc012200640 1 -- 172.21.15.32:0/3958712221 wait complete. 2024-08-03T20:57:22.136 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 206, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-08-03T20:54:26.950518+0000', 'modified': '2024-08-03T20:57:20.242553+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 103, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 8685}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8685': {'gid': 8685, 'name': 'a', 'rank': 0, 'incarnation': 202, 'state': 'up:active', 'state_seq': 5, 'addr': '172.21.15.32:6835/2345472481', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6833', 'nonce': 2345472481}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 2345472481}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [13], 'metadata_pool': 12, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8685, 'qdb_cluster': [8685]} 2024-08-03T20:57:22.136 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-08-03T20:57:22.136 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2024-08-03T20:57:22.367 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:22.366+0000 7fe1d6200640 1 -- 172.21.15.32:0/2748202381 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fe1d0144f90 msgr2=0x7fe1d0074570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:22.367 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:22.366+0000 7fe1d6200640 1 --2- 172.21.15.32:0/2748202381 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fe1d0144f90 0x7fe1d0074570 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fe1b8009830 tx=0x7fe1b802a510 comp rx=0 tx=0).stop 2024-08-03T20:57:22.367 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:22.366+0000 7fe1d6200640 1 -- 172.21.15.32:0/2748202381 shutdown_connections 2024-08-03T20:57:22.367 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:22.366+0000 7fe1d6200640 1 --2- 172.21.15.32:0/2748202381 >> v2:172.21.15.118:3300/0 conn(0x7fe1d015fff0 0x7fe1d0160400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:22.367 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:22.366+0000 7fe1d6200640 1 --2- 172.21.15.32:0/2748202381 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fe1d0144f90 0x7fe1d0074570 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:22.367 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:22.366+0000 7fe1d6200640 1 -- 172.21.15.32:0/2748202381 >> 172.21.15.32:0/2748202381 conn(0x7fe1d00b1bd0 msgr2=0x7fe1d00b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:57:22.367 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:22.366+0000 7fe1d6200640 1 -- 172.21.15.32:0/2748202381 shutdown_connections 2024-08-03T20:57:22.367 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:22.366+0000 7fe1d6200640 1 -- 172.21.15.32:0/2748202381 wait complete. 2024-08-03T20:57:22.369 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:22.368+0000 7fe1d6200640 1 Processor -- start 2024-08-03T20:57:22.369 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:22.368+0000 7fe1d6200640 1 -- start start 2024-08-03T20:57:22.372 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:22.370+0000 7fe1d6200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe1d006e930 0x7fe1d015f1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:22.373 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:22.370+0000 7fe1d6200640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fe1d0144f90 0x7fe1d013e140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:22.373 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:22.370+0000 7fe1d6200640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fe1d015fff0 0x7fe1d01f4270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:22.373 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:22.370+0000 7fe1d6200640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fe1d00cf840 con 0x7fe1d006e930 2024-08-03T20:57:22.373 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:22.370+0000 7fe1d6200640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7fe1d0145f30 con 0x7fe1d015fff0 2024-08-03T20:57:22.374 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:22.370+0000 7fe1d6200640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7fe1d00cf9c0 con 0x7fe1d0144f90 2024-08-03T20:57:22.374 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:22.370+0000 7fe1cf400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe1d006e930 0x7fe1d015f1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:22.374 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:22.370+0000 7fe1cf400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe1d006e930 0x7fe1d015f1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:59442/0 (socket says 172.21.15.32:59442) 2024-08-03T20:57:22.374 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:22.370+0000 7fe1cf400640 1 -- 172.21.15.32:0/1138684261 learned_addr learned my addr 172.21.15.32:0/1138684261 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:57:22.374 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:22.370+0000 7fe1cfe00640 1 --2- 172.21.15.32:0/1138684261 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fe1d015fff0 0x7fe1d01f4270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:22.374 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:22.370+0000 7fe1c7e00640 1 --2- 172.21.15.32:0/1138684261 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fe1d0144f90 0x7fe1d013e140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:22.374 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:22.371+0000 7fe1cf400640 1 -- 172.21.15.32:0/1138684261 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fe1d0144f90 msgr2=0x7fe1d013e140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:22.374 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:22.371+0000 7fe1cf400640 1 --2- 172.21.15.32:0/1138684261 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fe1d0144f90 0x7fe1d013e140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:22.374 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:22.371+0000 7fe1cf400640 1 -- 172.21.15.32:0/1138684261 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fe1d015fff0 msgr2=0x7fe1d01f4270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:22.374 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:22.371+0000 7fe1cf400640 1 --2- 172.21.15.32:0/1138684261 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fe1d015fff0 0x7fe1d01f4270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:22.374 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:22.371+0000 7fe1cf400640 1 -- 172.21.15.32:0/1138684261 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe1d015e110 con 0x7fe1d006e930 2024-08-03T20:57:22.374 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:22.371+0000 7fe1cf400640 1 --2- 172.21.15.32:0/1138684261 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe1d006e930 0x7fe1d015f1b0 secure :-1 s=READY pgs=1525 cs=0 l=1 rev1=1 crypto rx=0x7fe1c000c3a0 tx=0x7fe1c000c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:57:22.374 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:22.371+0000 7fe1c7400640 1 -- 172.21.15.32:0/1138684261 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1c0017070 con 0x7fe1d006e930 2024-08-03T20:57:22.375 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:22.371+0000 7fe1c7400640 1 -- 172.21.15.32:0/1138684261 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fe1c00053c0 con 0x7fe1d006e930 2024-08-03T20:57:22.375 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:22.371+0000 7fe1c7400640 1 -- 172.21.15.32:0/1138684261 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1c0003850 con 0x7fe1d006e930 2024-08-03T20:57:22.375 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:22.371+0000 7fe1d6200640 1 -- 172.21.15.32:0/1138684261 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe1d015e3f0 con 0x7fe1d006e930 2024-08-03T20:57:22.375 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:22.372+0000 7fe1d6200640 1 -- 172.21.15.32:0/1138684261 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe1d015e9b0 con 0x7fe1d006e930 2024-08-03T20:57:22.376 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:22.374+0000 7fe1c7400640 1 -- 172.21.15.32:0/1138684261 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7fe1c001a070 con 0x7fe1d006e930 2024-08-03T20:57:22.377 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:22.375+0000 7fe1c7400640 1 --2- 172.21.15.32:0/1138684261 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fe1ac077010 0x7fe1ac079520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:22.377 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:22.375+0000 7fe1c7400640 1 -- 172.21.15.32:0/1138684261 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(103..103 src has 1..103) ==== 7402+0+0 (secure 0 0 0) 0x7fe1c00983d0 con 0x7fe1d006e930 2024-08-03T20:57:22.377 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:22.375+0000 7fe1c7e00640 1 --2- 172.21.15.32:0/1138684261 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fe1ac077010 0x7fe1ac079520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:22.377 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:22.375+0000 7fe1dd1d4740 1 Processor -- start 2024-08-03T20:57:22.377 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:22.376+0000 7fe1dd1d4740 1 -- start start 2024-08-03T20:57:22.377 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:22.376+0000 7fe1c7e00640 1 --2- 172.21.15.32:0/1138684261 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fe1ac077010 0x7fe1ac079520 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7fe1b8034000 tx=0x7fe1b8004730 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:57:22.379 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:22.377+0000 7fe1dd1d4740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55d9c7361270 0x55d9c77502a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:22.379 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:22.377+0000 7fe1dd1d4740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55d9c73383b0 0x55d9c77250f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:22.379 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:22.377+0000 7fe1dd1d4740 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55d9c7340590 0x55d9c7725630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:22.379 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:22.377+0000 7fe1dd1d4740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55d9c7679a60 con 0x55d9c73383b0 2024-08-03T20:57:22.379 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:22.377+0000 7fe1dd1d4740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x55d9c7668f60 con 0x55d9c7361270 2024-08-03T20:57:22.379 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:22.377+0000 7fe1dd1d4740 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x55d9c766cfa0 con 0x55d9c7340590 2024-08-03T20:57:22.379 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:22.377+0000 7fe1c7e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55d9c73383b0 0x55d9c77250f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:22.379 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:22.377+0000 7fe1c7e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55d9c73383b0 0x55d9c77250f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:59448/0 (socket says 172.21.15.32:59448) 2024-08-03T20:57:22.379 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:22.377+0000 7fe1c7e00640 1 -- 172.21.15.32:0/2047268606 learned_addr learned my addr 172.21.15.32:0/2047268606 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:57:22.380 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:22.377+0000 7fe1cf400640 1 --2- 172.21.15.32:0/2047268606 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55d9c7361270 0x55d9c77502a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:22.380 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:22.378+0000 7fe1cfe00640 1 --2- 172.21.15.32:0/2047268606 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55d9c7340590 0x55d9c7725630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:22.380 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:22.378+0000 7fe1c7e00640 1 -- 172.21.15.32:0/2047268606 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55d9c7340590 msgr2=0x55d9c7725630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:57:22.380 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:22.378+0000 7fe1c7e00640 1 --2- 172.21.15.32:0/2047268606 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55d9c7340590 0x55d9c7725630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:22.380 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:22.378+0000 7fe1c7e00640 1 -- 172.21.15.32:0/2047268606 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55d9c7361270 msgr2=0x55d9c77502a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:57:22.380 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:22.378+0000 7fe1c7e00640 1 --2- 172.21.15.32:0/2047268606 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55d9c7361270 0x55d9c77502a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:22.380 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:22.378+0000 7fe1c7e00640 1 -- 172.21.15.32:0/2047268606 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d9c73cc850 con 0x55d9c73383b0 2024-08-03T20:57:22.380 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:22.378+0000 7fe1c7e00640 1 --2- 172.21.15.32:0/2047268606 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55d9c73383b0 0x55d9c77250f0 secure :-1 s=READY pgs=1526 cs=0 l=1 rev1=1 crypto rx=0x7fe1b8009920 tx=0x7fe1b803e070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c692f5e49a525dba server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:57:22.380 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:22.378+0000 7fe1bf400640 1 -- 172.21.15.32:0/2047268606 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1b8003d70 con 0x55d9c73383b0 2024-08-03T20:57:22.380 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:22.378+0000 7fe1bf400640 1 -- 172.21.15.32:0/2047268606 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fe1b8047420 con 0x55d9c73383b0 2024-08-03T20:57:22.380 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:22.378+0000 7fe1bf400640 1 -- 172.21.15.32:0/2047268606 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1b8047830 con 0x55d9c73383b0 2024-08-03T20:57:22.382 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:22.379+0000 7fe1dd1d4740 1 -- 172.21.15.32:0/2047268606 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55d9c73383b0 msgr2=0x55d9c77250f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:22.382 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:22.379+0000 7fe1dd1d4740 1 --2- 172.21.15.32:0/2047268606 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55d9c73383b0 0x55d9c77250f0 secure :-1 s=READY pgs=1526 cs=0 l=1 rev1=1 crypto rx=0x7fe1b8009920 tx=0x7fe1b803e070 comp rx=0 tx=0).stop 2024-08-03T20:57:22.382 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:22.379+0000 7fe1dd1d4740 1 -- 172.21.15.32:0/2047268606 shutdown_connections 2024-08-03T20:57:22.382 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:22.379+0000 7fe1dd1d4740 1 --2- 172.21.15.32:0/2047268606 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55d9c7340590 0x55d9c7725630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:22.382 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:22.379+0000 7fe1dd1d4740 1 --2- 172.21.15.32:0/2047268606 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55d9c73383b0 0x55d9c77250f0 unknown :-1 s=CLOSED pgs=1526 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:22.382 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:22.379+0000 7fe1dd1d4740 1 --2- 172.21.15.32:0/2047268606 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55d9c7361270 0x55d9c77502a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:22.382 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:22.379+0000 7fe1dd1d4740 1 -- 172.21.15.32:0/2047268606 >> 172.21.15.32:0/2047268606 conn(0x55d9c73553c0 msgr2=0x55d9c770b590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:57:22.382 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:22.379+0000 7fe1dd1d4740 1 -- 172.21.15.32:0/2047268606 shutdown_connections 2024-08-03T20:57:22.382 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:22.379+0000 7fe1dd1d4740 1 -- 172.21.15.32:0/2047268606 wait complete. 2024-08-03T20:57:22.383 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:22.380+0000 7fe1dd1d4740 1 -- start start 2024-08-03T20:57:22.383 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:22.381+0000 7fe1dd1d4740 1 Processor -- start 2024-08-03T20:57:22.386 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:22.382+0000 7fe1dd1d4740 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55d9c73383b0 0x55d9c7745ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:22.386 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:22.382+0000 7fe1dd1d4740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55d9c7340590 0x55d9c7747100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:22.387 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:22.382+0000 7fe1dd1d4740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55d9c7361270 0x55d9c7747640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:22.387 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:22.382+0000 7fe1dd1d4740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55d9c73db620 con 0x55d9c7340590 2024-08-03T20:57:22.387 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:22.382+0000 7fe1dd1d4740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x55d9c7606160 con 0x55d9c7361270 2024-08-03T20:57:22.387 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:22.382+0000 7fe1dd1d4740 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x55d9c7735620 con 0x55d9c73383b0 2024-08-03T20:57:22.387 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:22.382+0000 7fe1dd1d4740 10 client.0 _ll_register_callbacks cb 0x55d9c76156b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-03T20:57:22.387 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:22.382+0000 7fe1c7e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55d9c7340590 0x55d9c7747100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:22.387 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:22.382+0000 7fe1c7e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55d9c7340590 0x55d9c7747100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:59452/0 (socket says 172.21.15.32:59452) 2024-08-03T20:57:22.387 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:22.382+0000 7fe1c7e00640 1 -- 172.21.15.32:0/740340679 learned_addr learned my addr 172.21.15.32:0/740340679 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:57:22.387 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:22.382+0000 7fe1cf400640 1 --2- 172.21.15.32:0/740340679 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55d9c73383b0 0x55d9c7745ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:22.387 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:22.383+0000 7fe1cfe00640 1 --2- 172.21.15.32:0/740340679 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55d9c7361270 0x55d9c7747640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:22.387 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:22.383+0000 7fe1c7e00640 1 -- 172.21.15.32:0/740340679 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55d9c73383b0 msgr2=0x55d9c7745ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:57:22.387 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:22.383+0000 7fe1c7e00640 1 --2- 172.21.15.32:0/740340679 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55d9c73383b0 0x55d9c7745ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:22.388 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:22.383+0000 7fe1c7e00640 1 -- 172.21.15.32:0/740340679 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55d9c7361270 msgr2=0x55d9c7747640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:57:22.388 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:22.383+0000 7fe1c7e00640 1 --2- 172.21.15.32:0/740340679 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55d9c7361270 0x55d9c7747640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:22.388 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:22.383+0000 7fe1c7e00640 1 -- 172.21.15.32:0/740340679 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d9c74436b0 con 0x55d9c7340590 2024-08-03T20:57:22.388 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:22.383+0000 7fe1c7e00640 1 --2- 172.21.15.32:0/740340679 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55d9c7340590 0x55d9c7747100 secure :-1 s=READY pgs=1527 cs=0 l=1 rev1=1 crypto rx=0x7fe1b8009780 tx=0x7fe1b8048890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a222c2ce27be6d0c server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:57:22.388 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:22.383+0000 7fe1b3e00640 10 client.0 ms_handle_connect on v2:172.21.15.32:3300/0 2024-08-03T20:57:22.388 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:22.383+0000 7fe1b3e00640 1 -- 172.21.15.32:0/740340679 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1b8048d40 con 0x55d9c7340590 2024-08-03T20:57:22.388 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:22.384+0000 7fe1dd1d4740 1 -- 172.21.15.32:0/740340679 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe1b8047420 con 0x55d9c7340590 2024-08-03T20:57:22.388 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:22.384+0000 7fe1b3e00640 1 -- 172.21.15.32:0/740340679 <== mon.0 v2:172.21.15.32:3300/0 2 ==== osd_map(103..103 src has 1..103) ==== 7402+0+0 (secure 0 0 0) 0x7fe1b8005020 con 0x55d9c7340590 2024-08-03T20:57:22.388 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:22.384+0000 7fe1b3e00640 1 -- 172.21.15.32:0/740340679 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_get_version_reply(handle=1 version=206) ==== 24+0+0 (secure 0 0 0) 0x7fe1b8063050 con 0x55d9c7340590 2024-08-03T20:57:22.388 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:22.384+0000 7fe1dd1d4740 10 client.8718 fetch_fsmap learned FSMap version 206 2024-08-03T20:57:22.389 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:22.384+0000 7fe1dd1d4740 1 -- 172.21.15.32:0/740340679 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,fsmap=206,monmap=2+}) -- 0x55d9c7735420 con 0x55d9c7340590 2024-08-03T20:57:22.389 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:22.384+0000 7fe1b3e00640 1 -- 172.21.15.32:0/740340679 <== mon.0 v2:172.21.15.32:3300/0 4 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fe1b8047420 con 0x55d9c7340590 2024-08-03T20:57:22.389 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:22.384+0000 7fe1b3e00640 1 -- 172.21.15.32:0/740340679 <== mon.0 v2:172.21.15.32:3300/0 5 ==== fsmap(e 206) ==== 3502+0+0 (secure 0 0 0) 0x7fe1b8066020 con 0x55d9c7340590 2024-08-03T20:57:22.389 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:22.384+0000 7fe1dd1d4740 10 client.8718 fetch_fsmap finished waiting for FSMap version 206 2024-08-03T20:57:22.389 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:22.384+0000 7fe1dd1d4740 10 client.8718 resolve_mds: resolved name 'a' to daemon mds.a 2024-08-03T20:57:22.389 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:22.384+0000 7fe1dd1d4740 20 client.8718 populate_metadata read hostname 'smithi032' 2024-08-03T20:57:22.389 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:22.384+0000 7fe1dd1d4740 1 --2- 172.21.15.32:0/740340679 >> [v2:172.21.15.32:6833/2345472481,v1:172.21.15.32:6835/2345472481] conn(0x55d9c731fe40 0x55d9c7734360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:22.389 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:22.384+0000 7fe1dd1d4740 4 client.8718 mds_command: new command op to 8685 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-08-03T20:57:22.389 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:22.384+0000 7fe1dd1d4740 1 -- 172.21.15.32:0/740340679 --> [v2:172.21.15.32:6833/2345472481,v1:172.21.15.32:6835/2345472481] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d9c767b9b0 con 0x55d9c731fe40 2024-08-03T20:57:22.389 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:22.385+0000 7fe1cf400640 1 --2- 172.21.15.32:0/740340679 >> [v2:172.21.15.32:6833/2345472481,v1:172.21.15.32:6835/2345472481] conn(0x55d9c731fe40 0x55d9c7734360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:22.389 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:22.385+0000 7fe1cf400640 1 --2- 172.21.15.32:0/740340679 >> [v2:172.21.15.32:6833/2345472481,v1:172.21.15.32:6835/2345472481] conn(0x55d9c731fe40 0x55d9c7734360 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3e0b439d32dbe6f8 server_cookie=8ff489383ee9b164 in_seq=0 out_seq=0 2024-08-03T20:57:22.390 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:22.385+0000 7fe1b3e00640 10 client.8718 ms_handle_connect on v2:172.21.15.32:6833/2345472481 2024-08-03T20:57:22.391 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:22.386+0000 7fe1b3e00640 1 -- 172.21.15.32:0/740340679 <== mds.0 v2:172.21.15.32:6833/2345472481 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55d9c767b9b0 con 0x55d9c731fe40 2024-08-03T20:57:22.391 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:22.386+0000 7fe1b3e00640 10 client.8718 handle_command_reply: tid=0 2024-08-03T20:57:22.391 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:22.386+0000 7fe1dd1d4740 1 client.8718 shutdown 2024-08-03T20:57:22.391 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:22.386+0000 7fe1dd1d4740 1 -- 172.21.15.32:0/740340679 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55d9c7340590 msgr2=0x55d9c7747100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:22.391 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:22.386+0000 7fe1dd1d4740 1 --2- 172.21.15.32:0/740340679 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55d9c7340590 0x55d9c7747100 secure :-1 s=READY pgs=1527 cs=0 l=1 rev1=1 crypto rx=0x7fe1b8009780 tx=0x7fe1b8048890 comp rx=0 tx=0).stop 2024-08-03T20:57:22.391 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:22.386+0000 7fe1dd1d4740 1 -- 172.21.15.32:0/740340679 shutdown_connections 2024-08-03T20:57:22.392 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:22.386+0000 7fe1dd1d4740 1 --2- 172.21.15.32:0/740340679 >> [v2:172.21.15.32:6833/2345472481,v1:172.21.15.32:6835/2345472481] conn(0x55d9c731fe40 0x55d9c7734360 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:22.392 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:22.386+0000 7fe1dd1d4740 1 --2- 172.21.15.32:0/740340679 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55d9c7361270 0x55d9c7747640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:22.392 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:22.386+0000 7fe1dd1d4740 1 --2- 172.21.15.32:0/740340679 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55d9c7340590 0x55d9c7747100 unknown :-1 s=CLOSED pgs=1527 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:22.392 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:22.386+0000 7fe1dd1d4740 1 --2- 172.21.15.32:0/740340679 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55d9c73383b0 0x55d9c7745ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:22.392 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:22.386+0000 7fe1b3e00640 0 client.8718 ms_handle_reset on v2:172.21.15.32:6833/2345472481 2024-08-03T20:57:22.392 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:22.386+0000 7fe1dd1d4740 1 -- 172.21.15.32:0/740340679 >> 172.21.15.32:0/740340679 conn(0x55d9c73553c0 msgr2=0x55d9c77062b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:57:22.393 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:22.387+0000 7fe1dd1d4740 1 -- 172.21.15.32:0/740340679 shutdown_connections 2024-08-03T20:57:22.393 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:22.387+0000 7fe1dd1d4740 1 -- 172.21.15.32:0/740340679 wait complete. 2024-08-03T20:57:22.393 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:22.387+0000 7fe1dd1d4740 20 client.8718 trim_cache size 0 max 16384 2024-08-03T20:57:22.407 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:22.407+0000 7fe1dd1d4740 1 Processor -- start 2024-08-03T20:57:22.407 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:22.407+0000 7fe1dd1d4740 1 -- start start 2024-08-03T20:57:22.410 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:22.408+0000 7fe1dd1d4740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55d9c73383b0 0x55d9c7747090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:22.410 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:22.408+0000 7fe1dd1d4740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55d9c7340590 0x55d9c77475d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:22.410 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:22.408+0000 7fe1dd1d4740 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55d9c7361270 0x55d9c7747c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:22.410 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:22.408+0000 7fe1dd1d4740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fe1d01fda80 con 0x55d9c73383b0 2024-08-03T20:57:22.410 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:22.408+0000 7fe1dd1d4740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7fe1d00b4f50 con 0x55d9c7340590 2024-08-03T20:57:22.410 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:22.408+0000 7fe1dd1d4740 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7fe1d006fd50 con 0x55d9c7361270 2024-08-03T20:57:22.410 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:22.408+0000 7fe1cf400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55d9c73383b0 0x55d9c7747090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:22.410 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:22.408+0000 7fe1cf400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55d9c73383b0 0x55d9c7747090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:59466/0 (socket says 172.21.15.32:59466) 2024-08-03T20:57:22.411 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:22.409+0000 7fe1cf400640 1 -- 172.21.15.32:0/1624040395 learned_addr learned my addr 172.21.15.32:0/1624040395 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:57:22.411 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:22.409+0000 7fe1c7e00640 1 --2- 172.21.15.32:0/1624040395 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55d9c7340590 0x55d9c77475d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:22.411 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:22.409+0000 7fe1cfe00640 1 --2- 172.21.15.32:0/1624040395 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55d9c7361270 0x55d9c7747c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:22.411 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:22.409+0000 7fe1cf400640 1 -- 172.21.15.32:0/1624040395 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55d9c7361270 msgr2=0x55d9c7747c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:57:22.411 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:22.409+0000 7fe1cf400640 1 --2- 172.21.15.32:0/1624040395 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55d9c7361270 0x55d9c7747c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:22.411 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:22.409+0000 7fe1cf400640 1 -- 172.21.15.32:0/1624040395 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55d9c7340590 msgr2=0x55d9c77475d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:57:22.411 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:22.409+0000 7fe1cf400640 1 --2- 172.21.15.32:0/1624040395 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55d9c7340590 0x55d9c77475d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:22.411 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:22.409+0000 7fe1cf400640 1 -- 172.21.15.32:0/1624040395 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d9c77080f0 con 0x55d9c73383b0 2024-08-03T20:57:22.411 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:22.409+0000 7fe1c7e00640 1 --2- 172.21.15.32:0/1624040395 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55d9c7340590 0x55d9c77475d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-03T20:57:22.411 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:22.409+0000 7fe1cfe00640 1 --2- 172.21.15.32:0/1624040395 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55d9c7361270 0x55d9c7747c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-03T20:57:22.411 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:22.409+0000 7fe1cf400640 1 --2- 172.21.15.32:0/1624040395 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55d9c73383b0 0x55d9c7747090 secure :-1 s=READY pgs=1528 cs=0 l=1 rev1=1 crypto rx=0x7fe1c00607d0 tx=0x7fe1c0000ed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9a74c02be4187ab2 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:57:22.411 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:22.409+0000 7fe1bea00640 1 -- 172.21.15.32:0/1624040395 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1c0069840 con 0x55d9c73383b0 2024-08-03T20:57:22.411 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:22.410+0000 7fe1bea00640 1 -- 172.21.15.32:0/1624040395 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fe1c00962d0 con 0x55d9c73383b0 2024-08-03T20:57:22.412 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:22.410+0000 7fe1bea00640 1 -- 172.21.15.32:0/1624040395 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1c00820f0 con 0x55d9c73383b0 2024-08-03T20:57:22.413 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:22.410+0000 7fe1dd1d4740 1 -- 172.21.15.32:0/1624040395 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55d9c73383b0 msgr2=0x55d9c7747090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:22.413 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:22.410+0000 7fe1dd1d4740 1 --2- 172.21.15.32:0/1624040395 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55d9c73383b0 0x55d9c7747090 secure :-1 s=READY pgs=1528 cs=0 l=1 rev1=1 crypto rx=0x7fe1c00607d0 tx=0x7fe1c0000ed0 comp rx=0 tx=0).stop 2024-08-03T20:57:22.413 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:22.410+0000 7fe1dd1d4740 1 -- 172.21.15.32:0/1624040395 shutdown_connections 2024-08-03T20:57:22.413 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:22.410+0000 7fe1dd1d4740 1 --2- 172.21.15.32:0/1624040395 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55d9c7340590 0x55d9c77475d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:22.413 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:22.410+0000 7fe1dd1d4740 1 --2- 172.21.15.32:0/1624040395 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55d9c7361270 0x55d9c7747c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:22.413 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:22.410+0000 7fe1dd1d4740 1 --2- 172.21.15.32:0/1624040395 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55d9c73383b0 0x55d9c7747090 unknown :-1 s=CLOSED pgs=1528 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:22.413 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:22.410+0000 7fe1dd1d4740 1 -- 172.21.15.32:0/1624040395 >> 172.21.15.32:0/1624040395 conn(0x55d9c73553c0 msgr2=0x55d9c7762490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:57:22.413 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:22.410+0000 7fe1dd1d4740 1 -- 172.21.15.32:0/1624040395 shutdown_connections 2024-08-03T20:57:22.413 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:22.410+0000 7fe1dd1d4740 1 -- 172.21.15.32:0/1624040395 wait complete. 2024-08-03T20:57:22.413 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:22.411+0000 7fe1dd1d4740 1 -- start start 2024-08-03T20:57:22.414 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:22.412+0000 7fe1dd1d4740 1 Processor -- start 2024-08-03T20:57:22.416 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:22.413+0000 7fe1dd1d4740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55d9c73383b0 0x55d9c7732720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:22.416 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:22.413+0000 7fe1dd1d4740 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55d9c7340590 0x55d9c77ccaa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:22.416 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:22.413+0000 7fe1dd1d4740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55d9c7361270 0x55d9c774b6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:22.417 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:22.413+0000 7fe1dd1d4740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55d9c774c4d0 con 0x55d9c7361270 2024-08-03T20:57:22.417 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:22.413+0000 7fe1dd1d4740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x55d9c7746020 con 0x55d9c73383b0 2024-08-03T20:57:22.417 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:22.413+0000 7fe1dd1d4740 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x55d9c7749650 con 0x55d9c7340590 2024-08-03T20:57:22.417 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:22.413+0000 7fe1dd1d4740 10 client.0 _ll_register_callbacks cb 0x7fe1b8004400 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-03T20:57:22.417 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:22.413+0000 7fe1cfe00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55d9c7361270 0x55d9c774b6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:22.417 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:22.413+0000 7fe1cf400640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55d9c73383b0 0x55d9c7732720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:22.417 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:22.414+0000 7fe1cfe00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55d9c7361270 0x55d9c774b6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:59474/0 (socket says 172.21.15.32:59474) 2024-08-03T20:57:22.417 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:22.414+0000 7fe1cfe00640 1 -- 172.21.15.32:0/2932755733 learned_addr learned my addr 172.21.15.32:0/2932755733 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:57:22.417 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:22.414+0000 7fe1cf400640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55d9c73383b0 0x55d9c7732720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.32:58670/0 (socket says 172.21.15.32:58670) 2024-08-03T20:57:22.418 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:22.414+0000 7fe1c7e00640 1 --2- 172.21.15.32:0/2932755733 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55d9c7340590 0x55d9c77ccaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:22.418 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:22.414+0000 7fe1cfe00640 1 -- 172.21.15.32:0/2932755733 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55d9c7340590 msgr2=0x55d9c77ccaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:57:22.418 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:22.414+0000 7fe1cfe00640 1 --2- 172.21.15.32:0/2932755733 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55d9c7340590 0x55d9c77ccaa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:22.418 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:22.414+0000 7fe1cfe00640 1 -- 172.21.15.32:0/2932755733 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55d9c73383b0 msgr2=0x55d9c7732720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:57:22.418 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:22.414+0000 7fe1cfe00640 1 --2- 172.21.15.32:0/2932755733 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55d9c73383b0 0x55d9c7732720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:22.418 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:22.414+0000 7fe1cfe00640 1 -- 172.21.15.32:0/2932755733 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d9c770c730 con 0x55d9c7361270 2024-08-03T20:57:22.418 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:22.414+0000 7fe1cfe00640 1 --2- 172.21.15.32:0/2932755733 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55d9c7361270 0x55d9c774b6b0 secure :-1 s=READY pgs=1529 cs=0 l=1 rev1=1 crypto rx=0x7fe1c800eec0 tx=0x7fe1c800d7b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a5909e93dcf60827 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:57:22.418 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:22.414+0000 7fe1b3e00640 10 client.0 ms_handle_connect on v2:172.21.15.32:3300/0 2024-08-03T20:57:22.418 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:22.414+0000 7fe1b3e00640 1 -- 172.21.15.32:0/2932755733 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1c801e020 con 0x55d9c7361270 2024-08-03T20:57:22.418 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:22.415+0000 7fe1b3e00640 1 -- 172.21.15.32:0/2932755733 <== mon.0 v2:172.21.15.32:3300/0 2 ==== osd_map(103..103 src has 1..103) ==== 7402+0+0 (secure 0 0 0) 0x7fe1c8003b70 con 0x55d9c7361270 2024-08-03T20:57:22.418 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:22.415+0000 7fe1dd1d4740 1 -- 172.21.15.32:0/2932755733 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe1c00962d0 con 0x55d9c7361270 2024-08-03T20:57:22.418 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:22.415+0000 7fe1b3e00640 1 -- 172.21.15.32:0/2932755733 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_get_version_reply(handle=1 version=206) ==== 24+0+0 (secure 0 0 0) 0x7fe1c8003d70 con 0x55d9c7361270 2024-08-03T20:57:22.418 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:22.415+0000 7fe1dd1d4740 10 client.8724 fetch_fsmap learned FSMap version 206 2024-08-03T20:57:22.419 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:22.415+0000 7fe1dd1d4740 1 -- 172.21.15.32:0/2932755733 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,fsmap=206,monmap=2+}) -- 0x55d9c7732020 con 0x55d9c7361270 2024-08-03T20:57:22.419 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:22.415+0000 7fe1b3e00640 1 -- 172.21.15.32:0/2932755733 <== mon.0 v2:172.21.15.32:3300/0 4 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fe1c00962d0 con 0x55d9c7361270 2024-08-03T20:57:22.419 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:22.415+0000 7fe1b3e00640 1 -- 172.21.15.32:0/2932755733 <== mon.0 v2:172.21.15.32:3300/0 5 ==== fsmap(e 206) ==== 3502+0+0 (secure 0 0 0) 0x7fe1c8017070 con 0x55d9c7361270 2024-08-03T20:57:22.419 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:22.415+0000 7fe1dd1d4740 10 client.8724 fetch_fsmap finished waiting for FSMap version 206 2024-08-03T20:57:22.419 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:22.415+0000 7fe1dd1d4740 10 client.8724 resolve_mds: resolved name 'a' to daemon mds.a 2024-08-03T20:57:22.419 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:22.415+0000 7fe1dd1d4740 20 client.8724 populate_metadata read hostname 'smithi032' 2024-08-03T20:57:22.419 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:22.415+0000 7fe1dd1d4740 1 --2- 172.21.15.32:0/2932755733 >> [v2:172.21.15.32:6833/2345472481,v1:172.21.15.32:6835/2345472481] conn(0x55d9c77d1010 0x55d9c7747090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:22.420 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:22.415+0000 7fe1dd1d4740 4 client.8724 mds_command: new command op to 8685 tid=0 multi_id=0 [{"prefix": "status"}] 2024-08-03T20:57:22.420 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:22.415+0000 7fe1dd1d4740 1 -- 172.21.15.32:0/2932755733 --> [v2:172.21.15.32:6833/2345472481,v1:172.21.15.32:6835/2345472481] -- command(tid 0: {"prefix": "status"}) -- 0x55d9c7677450 con 0x55d9c77d1010 2024-08-03T20:57:22.420 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:22.416+0000 7fe1cf400640 1 --2- 172.21.15.32:0/2932755733 >> [v2:172.21.15.32:6833/2345472481,v1:172.21.15.32:6835/2345472481] conn(0x55d9c77d1010 0x55d9c7747090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:22.420 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:22.416+0000 7fe1b3e00640 10 client.8724 ms_handle_connect on v2:172.21.15.32:6833/2345472481 2024-08-03T20:57:22.420 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:22.416+0000 7fe1cf400640 1 --2- 172.21.15.32:0/2932755733 >> [v2:172.21.15.32:6833/2345472481,v1:172.21.15.32:6835/2345472481] conn(0x55d9c77d1010 0x55d9c7747090 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=abcd679ac566bf93 server_cookie=e8cac6568e1f596d in_seq=0 out_seq=0 2024-08-03T20:57:22.421 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:22.416+0000 7fe1b3e00640 1 -- 172.21.15.32:0/2932755733 <== mds.0 v2:172.21.15.32:6833/2345472481 1 ==== command_reply(tid 0: 0 ) ==== 8+0+334 (crc 0 0 0) 0x55d9c7677450 con 0x55d9c77d1010 2024-08-03T20:57:22.421 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:22.416+0000 7fe1b3e00640 10 client.8724 handle_command_reply: tid=0 2024-08-03T20:57:22.422 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:22.416+0000 7fe1dd1d4740 1 client.8724 shutdown 2024-08-03T20:57:22.422 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:22.417+0000 7fe1dd1d4740 1 -- 172.21.15.32:0/2932755733 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55d9c7361270 msgr2=0x55d9c774b6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:22.422 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:22.417+0000 7fe1dd1d4740 1 --2- 172.21.15.32:0/2932755733 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55d9c7361270 0x55d9c774b6b0 secure :-1 s=READY pgs=1529 cs=0 l=1 rev1=1 crypto rx=0x7fe1c800eec0 tx=0x7fe1c800d7b0 comp rx=0 tx=0).stop 2024-08-03T20:57:22.422 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:22.417+0000 7fe1dd1d4740 1 -- 172.21.15.32:0/2932755733 shutdown_connections 2024-08-03T20:57:22.422 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:22.417+0000 7fe1dd1d4740 1 --2- 172.21.15.32:0/2932755733 >> [v2:172.21.15.32:6833/2345472481,v1:172.21.15.32:6835/2345472481] conn(0x55d9c77d1010 0x55d9c7747090 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:22.422 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:22.417+0000 7fe1dd1d4740 1 --2- 172.21.15.32:0/2932755733 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55d9c7361270 0x55d9c774b6b0 unknown :-1 s=CLOSED pgs=1529 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:22.422 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:22.417+0000 7fe1dd1d4740 1 --2- 172.21.15.32:0/2932755733 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55d9c7340590 0x55d9c77ccaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:22.422 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:22.417+0000 7fe1dd1d4740 1 --2- 172.21.15.32:0/2932755733 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55d9c73383b0 0x55d9c7732720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:22.422 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:22.417+0000 7fe1b3e00640 0 client.8724 ms_handle_reset on v2:172.21.15.32:6833/2345472481 2024-08-03T20:57:22.422 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:22.417+0000 7fe1dd1d4740 1 -- 172.21.15.32:0/2932755733 >> 172.21.15.32:0/2932755733 conn(0x55d9c73553c0 msgr2=0x55d9c7700550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:57:22.423 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:22.417+0000 7fe1dd1d4740 1 -- 172.21.15.32:0/2932755733 shutdown_connections 2024-08-03T20:57:22.423 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:22.417+0000 7fe1dd1d4740 1 -- 172.21.15.32:0/2932755733 wait complete. 2024-08-03T20:57:22.423 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:22.417+0000 7fe1dd1d4740 20 client.8724 trim_cache size 0 max 16384 2024-08-03T20:57:22.423 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-08-03T20:57:22.423 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_fsid": "40ab141a-01d7-49b9-ae3c-2edec9876e95", 2024-08-03T20:57:22.423 INFO:teuthology.orchestra.run.smithi032.stdout: "whoami": 0, 2024-08-03T20:57:22.423 INFO:teuthology.orchestra.run.smithi032.stdout: "id": 8685, 2024-08-03T20:57:22.423 INFO:teuthology.orchestra.run.smithi032.stdout: "want_state": "up:active", 2024-08-03T20:57:22.423 INFO:teuthology.orchestra.run.smithi032.stdout: "state": "up:active", 2024-08-03T20:57:22.424 INFO:teuthology.orchestra.run.smithi032.stdout: "fs_name": "cephfs", 2024-08-03T20:57:22.424 INFO:teuthology.orchestra.run.smithi032.stdout: "rank_uptime": 8.1870987020000001, 2024-08-03T20:57:22.424 INFO:teuthology.orchestra.run.smithi032.stdout: "mdsmap_epoch": 206, 2024-08-03T20:57:22.424 INFO:teuthology.orchestra.run.smithi032.stdout: "osdmap_epoch": 103, 2024-08-03T20:57:22.424 INFO:teuthology.orchestra.run.smithi032.stdout: "osdmap_epoch_barrier": 103, 2024-08-03T20:57:22.424 INFO:teuthology.orchestra.run.smithi032.stdout: "uptime": 9.3579160859999995 2024-08-03T20:57:22.424 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-08-03T20:57:22.425 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:22.418+0000 7fe1d6200640 1 -- 172.21.15.32:0/1138684261 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fe1ac077010 msgr2=0x7fe1ac079520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:22.425 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:22.418+0000 7fe1d6200640 1 --2- 172.21.15.32:0/1138684261 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fe1ac077010 0x7fe1ac079520 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7fe1b8034000 tx=0x7fe1b8004730 comp rx=0 tx=0).stop 2024-08-03T20:57:22.425 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:22.418+0000 7fe1d6200640 1 -- 172.21.15.32:0/1138684261 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe1d006e930 msgr2=0x7fe1d015f1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:22.425 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:22.418+0000 7fe1d6200640 1 --2- 172.21.15.32:0/1138684261 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe1d006e930 0x7fe1d015f1b0 secure :-1 s=READY pgs=1525 cs=0 l=1 rev1=1 crypto rx=0x7fe1c000c3a0 tx=0x7fe1c000c860 comp rx=0 tx=0).stop 2024-08-03T20:57:22.425 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:22.418+0000 7fe1d6200640 1 -- 172.21.15.32:0/1138684261 shutdown_connections 2024-08-03T20:57:22.425 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:22.418+0000 7fe1d6200640 1 --2- 172.21.15.32:0/1138684261 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fe1ac077010 0x7fe1ac079520 unknown :-1 s=CLOSED pgs=624 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:22.425 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:22.418+0000 7fe1d6200640 1 --2- 172.21.15.32:0/1138684261 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fe1d015fff0 0x7fe1d01f4270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:22.425 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:22.418+0000 7fe1d6200640 1 --2- 172.21.15.32:0/1138684261 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fe1d0144f90 0x7fe1d013e140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:22.425 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:22.418+0000 7fe1d6200640 1 --2- 172.21.15.32:0/1138684261 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe1d006e930 0x7fe1d015f1b0 unknown :-1 s=CLOSED pgs=1525 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:22.426 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:22.418+0000 7fe1d6200640 1 -- 172.21.15.32:0/1138684261 >> 172.21.15.32:0/1138684261 conn(0x7fe1d00b1bd0 msgr2=0x7fe1d00b09a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:57:22.426 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:22.418+0000 7fe1d6200640 1 -- 172.21.15.32:0/1138684261 shutdown_connections 2024-08-03T20:57:22.426 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:22.418+0000 7fe1d6200640 1 -- 172.21.15.32:0/1138684261 wait complete. 2024-08-03T20:57:22.443 DEBUG:tasks.ceph_test_case:wait_until_true: success in 5s and 0 retries 2024-08-03T20:57:22.444 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-08-03T20:57:22.673 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:22.672+0000 7f2f51000640 1 -- 172.21.15.32:0/3708280785 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f2f4c0ade80 msgr2=0x7f2f4c0ae290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:22.673 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:22.672+0000 7f2f51000640 1 --2- 172.21.15.32:0/3708280785 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f2f4c0ade80 0x7f2f4c0ae290 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f2f44005fd0 tx=0x7f2f4402a8a0 comp rx=0 tx=0).stop 2024-08-03T20:57:22.673 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:22.672+0000 7f2f51000640 1 -- 172.21.15.32:0/3708280785 shutdown_connections 2024-08-03T20:57:22.673 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:22.672+0000 7f2f51000640 1 --2- 172.21.15.32:0/3708280785 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f2f4c0ade80 0x7f2f4c0ae290 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:22.673 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:22.672+0000 7f2f51000640 1 --2- 172.21.15.32:0/3708280785 >> v2:172.21.15.118:3300/0 conn(0x7f2f4c06f720 0x7f2f4c06fb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:22.673 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:22.672+0000 7f2f51000640 1 -- 172.21.15.32:0/3708280785 >> 172.21.15.32:0/3708280785 conn(0x7f2f4c0b1bd0 msgr2=0x7f2f4c0b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:57:22.674 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:22.672+0000 7f2f51000640 1 -- 172.21.15.32:0/3708280785 shutdown_connections 2024-08-03T20:57:22.674 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:22.672+0000 7f2f51000640 1 -- 172.21.15.32:0/3708280785 wait complete. 2024-08-03T20:57:22.675 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:22.674+0000 7f2f51000640 1 Processor -- start 2024-08-03T20:57:22.675 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:22.674+0000 7f2f51000640 1 -- start start 2024-08-03T20:57:22.676 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:22.676+0000 7f2f51000640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f2f4c06f720 0x7f2f4c147a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:22.677 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:22.676+0000 7f2f51000640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f2f4c0ade80 0x7f2f4c157560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:22.677 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:22.676+0000 7f2f51000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2f4c0b39a0 0x7f2f4c157aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:22.677 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:22.676+0000 7f2f51000640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f2f4c0aeb60 con 0x7f2f4c0b39a0 2024-08-03T20:57:22.677 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:22.676+0000 7f2f51000640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f2f4c0ae9e0 con 0x7f2f4c0ade80 2024-08-03T20:57:22.677 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:22.676+0000 7f2f51000640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7f2f4c0aece0 con 0x7f2f4c06f720 2024-08-03T20:57:22.677 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:22.676+0000 7f2f4a800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2f4c0b39a0 0x7f2f4c157aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:22.677 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:22.676+0000 7f2f4a800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2f4c0b39a0 0x7f2f4c157aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:59482/0 (socket says 172.21.15.32:59482) 2024-08-03T20:57:22.677 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:22.676+0000 7f2f4a800640 1 -- 172.21.15.32:0/1963527884 learned_addr learned my addr 172.21.15.32:0/1963527884 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:57:22.679 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:22.676+0000 7f2f49e00640 1 --2- 172.21.15.32:0/1963527884 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f2f4c06f720 0x7f2f4c147a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:22.679 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:22.676+0000 7f2f49400640 1 --2- 172.21.15.32:0/1963527884 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f2f4c0ade80 0x7f2f4c157560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:22.679 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:22.677+0000 7f2f4a800640 1 -- 172.21.15.32:0/1963527884 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f2f4c06f720 msgr2=0x7f2f4c147a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:22.679 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:22.677+0000 7f2f4a800640 1 --2- 172.21.15.32:0/1963527884 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f2f4c06f720 0x7f2f4c147a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:22.679 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:22.677+0000 7f2f4a800640 1 -- 172.21.15.32:0/1963527884 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f2f4c0ade80 msgr2=0x7f2f4c157560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:22.679 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:22.677+0000 7f2f4a800640 1 --2- 172.21.15.32:0/1963527884 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f2f4c0ade80 0x7f2f4c157560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:22.679 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:22.677+0000 7f2f4a800640 1 -- 172.21.15.32:0/1963527884 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2f4c1601d0 con 0x7f2f4c0b39a0 2024-08-03T20:57:22.679 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:22.677+0000 7f2f49400640 1 --2- 172.21.15.32:0/1963527884 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f2f4c0ade80 0x7f2f4c157560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-03T20:57:22.679 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:22.677+0000 7f2f4a800640 1 --2- 172.21.15.32:0/1963527884 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2f4c0b39a0 0x7f2f4c157aa0 secure :-1 s=READY pgs=1530 cs=0 l=1 rev1=1 crypto rx=0x7f2f4402ae90 tx=0x7f2f44032d30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:57:22.679 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:22.677+0000 7f2f42000640 1 -- 172.21.15.32:0/1963527884 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f44041070 con 0x7f2f4c0b39a0 2024-08-03T20:57:22.679 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:22.677+0000 7f2f42000640 1 -- 172.21.15.32:0/1963527884 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f2f44004450 con 0x7f2f4c0b39a0 2024-08-03T20:57:22.679 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:22.677+0000 7f2f42000640 1 -- 172.21.15.32:0/1963527884 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f44031430 con 0x7f2f4c0b39a0 2024-08-03T20:57:22.680 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:22.677+0000 7f2f51000640 1 -- 172.21.15.32:0/1963527884 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2f4c1604b0 con 0x7f2f4c0b39a0 2024-08-03T20:57:22.680 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:22.678+0000 7f2f51000640 1 -- 172.21.15.32:0/1963527884 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2f4c1585e0 con 0x7f2f4c0b39a0 2024-08-03T20:57:22.680 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:22.679+0000 7f2f51000640 1 -- 172.21.15.32:0/1963527884 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2f4c158840 con 0x7f2f4c0b39a0 2024-08-03T20:57:22.680 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:22.680+0000 7f2f42000640 1 -- 172.21.15.32:0/1963527884 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f2f44044070 con 0x7f2f4c0b39a0 2024-08-03T20:57:22.681 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:22.681+0000 7f2f42000640 1 --2- 172.21.15.32:0/1963527884 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f2f2c077010 0x7f2f2c079520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:22.681 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:22.681+0000 7f2f42000640 1 -- 172.21.15.32:0/1963527884 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(103..103 src has 1..103) ==== 7402+0+0 (secure 0 0 0) 0x7f2f4402e020 con 0x7f2f4c0b39a0 2024-08-03T20:57:22.682 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:22.681+0000 7f2f49e00640 1 --2- 172.21.15.32:0/1963527884 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f2f2c077010 0x7f2f2c079520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:22.682 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:22.682+0000 7f2f49e00640 1 --2- 172.21.15.32:0/1963527884 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f2f2c077010 0x7f2f2c079520 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7f2f3c009530 tx=0x7f2f3c009260 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:57:22.686 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:22.686+0000 7f2f42000640 1 -- 172.21.15.32:0/1963527884 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7f2f4404b020 con 0x7f2f4c0b39a0 2024-08-03T20:57:22.930 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:22.930+0000 7f2f51000640 1 -- 172.21.15.32:0/1963527884 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f2f4c06fb30 con 0x7f2f4c0b39a0 2024-08-03T20:57:22.931 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:22.930+0000 7f2f42000640 1 -- 172.21.15.32:0/1963527884 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 206 v206) ==== 95+0+5233 (secure 0 0 0) 0x7f2f44082d60 con 0x7f2f4c0b39a0 2024-08-03T20:57:22.931 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-03T20:57:22.931 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":206,"btime":"2024-08-03T20:57:20:251720+0000","default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6439,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6835/531080160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6834","nonce":531080160},{"type":"v1","addr":"172.21.15.118:6835","nonce":531080160}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":84},{"gid":6454,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6837/2796882319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6836","nonce":2796882319},{"type":"v1","addr":"172.21.15.118:6837","nonce":2796882319}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":86},{"gid":6813,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6834/2686933890","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":2686933890},{"type":"v1","addr":"172.21.15.32:6834","nonce":2686933890}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":85}],"filesystems":[{"mdsmap":{"epoch":206,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-08-03T20:54:26.950518+0000","modified":"2024-08-03T20:57:20.242553+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":103,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8685},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8685":{"gid":8685,"name":"a","rank":0,"incarnation":202,"state":"up:active","state_seq":5,"addr":"172.21.15.32:6835/2345472481","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6833","nonce":2345472481},{"type":"v1","addr":"172.21.15.32:6835","nonce":2345472481}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8685,"qdb_cluster":[8685]},"id":6}]} 2024-08-03T20:57:22.931 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 206 2024-08-03T20:57:22.933 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:22.933+0000 7f2f51000640 1 -- 172.21.15.32:0/1963527884 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f2f2c077010 msgr2=0x7f2f2c079520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:22.934 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:22.933+0000 7f2f51000640 1 --2- 172.21.15.32:0/1963527884 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f2f2c077010 0x7f2f2c079520 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7f2f3c009530 tx=0x7f2f3c009260 comp rx=0 tx=0).stop 2024-08-03T20:57:22.934 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:22.933+0000 7f2f51000640 1 -- 172.21.15.32:0/1963527884 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2f4c0b39a0 msgr2=0x7f2f4c157aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:22.934 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:22.933+0000 7f2f51000640 1 --2- 172.21.15.32:0/1963527884 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2f4c0b39a0 0x7f2f4c157aa0 secure :-1 s=READY pgs=1530 cs=0 l=1 rev1=1 crypto rx=0x7f2f4402ae90 tx=0x7f2f44032d30 comp rx=0 tx=0).stop 2024-08-03T20:57:22.934 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:22.933+0000 7f2f51000640 1 -- 172.21.15.32:0/1963527884 shutdown_connections 2024-08-03T20:57:22.934 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:22.933+0000 7f2f51000640 1 --2- 172.21.15.32:0/1963527884 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f2f2c077010 0x7f2f2c079520 unknown :-1 s=CLOSED pgs=625 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:22.934 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:22.933+0000 7f2f51000640 1 --2- 172.21.15.32:0/1963527884 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f2f4c0ade80 0x7f2f4c157560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:22.934 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:22.933+0000 7f2f51000640 1 --2- 172.21.15.32:0/1963527884 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2f4c0b39a0 0x7f2f4c157aa0 unknown :-1 s=CLOSED pgs=1530 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:22.934 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:22.933+0000 7f2f51000640 1 --2- 172.21.15.32:0/1963527884 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f2f4c06f720 0x7f2f4c147a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:22.934 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:22.933+0000 7f2f51000640 1 -- 172.21.15.32:0/1963527884 >> 172.21.15.32:0/1963527884 conn(0x7f2f4c0b1bd0 msgr2=0x7f2f4c0b14c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:57:22.934 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:22.933+0000 7f2f51000640 1 -- 172.21.15.32:0/1963527884 shutdown_connections 2024-08-03T20:57:22.934 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:22.933+0000 7f2f51000640 1 -- 172.21.15.32:0/1963527884 wait complete. 2024-08-03T20:57:22.958 INFO:tasks.cephfs.test_damage:Mutation 'Corrupt 500.00000000' did not prevent MDS startup, attempting ls... 2024-08-03T20:57:22.958 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-08-03T20:57:23.186 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:23.185+0000 7f5593c00640 1 -- 172.21.15.32:0/4177150238 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f558c06e930 msgr2=0x7f558c144f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:23.187 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:23.185+0000 7f5593c00640 1 --2- 172.21.15.32:0/4177150238 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f558c06e930 0x7f558c144f90 secure :-1 s=READY pgs=1531 cs=0 l=1 rev1=1 crypto rx=0x7f558c0b2f80 tx=0x7f558002a560 comp rx=0 tx=0).stop 2024-08-03T20:57:23.187 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:23.185+0000 7f5593c00640 1 -- 172.21.15.32:0/4177150238 shutdown_connections 2024-08-03T20:57:23.187 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:23.185+0000 7f5593c00640 1 --2- 172.21.15.32:0/4177150238 >> v2:172.21.15.118:3300/0 conn(0x7f558c0747b0 0x7f558c0cc8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:23.187 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:23.185+0000 7f5593c00640 1 --2- 172.21.15.32:0/4177150238 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f558c073e60 0x7f558c074270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:23.187 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:23.185+0000 7f5593c00640 1 --2- 172.21.15.32:0/4177150238 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f558c06e930 0x7f558c144f90 unknown :-1 s=CLOSED pgs=1531 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:23.187 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:23.185+0000 7f5593c00640 1 -- 172.21.15.32:0/4177150238 >> 172.21.15.32:0/4177150238 conn(0x7f558c0b1bd0 msgr2=0x7f558c0b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:57:23.188 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:23.185+0000 7f5593c00640 1 -- 172.21.15.32:0/4177150238 shutdown_connections 2024-08-03T20:57:23.188 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:23.185+0000 7f5593c00640 1 -- 172.21.15.32:0/4177150238 wait complete. 2024-08-03T20:57:23.188 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:23.187+0000 7f5593c00640 1 Processor -- start 2024-08-03T20:57:23.188 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:23.187+0000 7f5593c00640 1 -- start start 2024-08-03T20:57:23.192 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:23.189+0000 7f5593c00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f558c06e930 0x7f558c0cb4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:23.192 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:23.189+0000 7f5593c00640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f558c073e60 0x7f558c0abfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:23.192 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:23.189+0000 7f5593c00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f558c0747b0 0x7f558c0ac520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:23.193 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:23.189+0000 7f5593c00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f558c0cf9c0 con 0x7f558c0747b0 2024-08-03T20:57:23.193 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:23.189+0000 7f5593c00640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f558c0cf840 con 0x7f558c06e930 2024-08-03T20:57:23.193 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:23.189+0000 7f5593c00640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7f558c0cce20 con 0x7f558c073e60 2024-08-03T20:57:23.193 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:23.189+0000 7f5591c00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f558c0747b0 0x7f558c0ac520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:23.193 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:23.189+0000 7f5591c00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f558c0747b0 0x7f558c0ac520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:59496/0 (socket says 172.21.15.32:59496) 2024-08-03T20:57:23.193 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:23.189+0000 7f5591c00640 1 -- 172.21.15.32:0/326014815 learned_addr learned my addr 172.21.15.32:0/326014815 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:57:23.193 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:23.189+0000 7f5591200640 1 --2- 172.21.15.32:0/326014815 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f558c06e930 0x7f558c0cb4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:23.193 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:23.189+0000 7f558be00640 1 --2- 172.21.15.32:0/326014815 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f558c073e60 0x7f558c0abfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:23.194 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:23.190+0000 7f5591c00640 1 -- 172.21.15.32:0/326014815 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f558c073e60 msgr2=0x7f558c0abfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:23.194 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:23.190+0000 7f5591c00640 1 --2- 172.21.15.32:0/326014815 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f558c073e60 0x7f558c0abfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:23.194 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:23.190+0000 7f5591c00640 1 -- 172.21.15.32:0/326014815 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f558c06e930 msgr2=0x7f558c0cb4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:23.194 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:23.190+0000 7f5591c00640 1 --2- 172.21.15.32:0/326014815 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f558c06e930 0x7f558c0cb4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:23.194 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:23.190+0000 7f5591c00640 1 -- 172.21.15.32:0/326014815 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f558c0cba30 con 0x7f558c0747b0 2024-08-03T20:57:23.194 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:23.190+0000 7f5591c00640 1 --2- 172.21.15.32:0/326014815 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f558c0747b0 0x7f558c0ac520 secure :-1 s=READY pgs=1532 cs=0 l=1 rev1=1 crypto rx=0x7f558400e9c0 tx=0x7f558400ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:57:23.195 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:23.190+0000 7f5588c00640 1 -- 172.21.15.32:0/326014815 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5584004d20 con 0x7f558c0747b0 2024-08-03T20:57:23.195 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:23.190+0000 7f5588c00640 1 -- 172.21.15.32:0/326014815 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f558400cb20 con 0x7f558c0747b0 2024-08-03T20:57:23.195 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:23.190+0000 7f5593c00640 1 -- 172.21.15.32:0/326014815 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f558c0c4f70 con 0x7f558c0747b0 2024-08-03T20:57:23.195 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:23.190+0000 7f5588c00640 1 -- 172.21.15.32:0/326014815 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5584005020 con 0x7f558c0747b0 2024-08-03T20:57:23.196 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:23.191+0000 7f5593c00640 1 -- 172.21.15.32:0/326014815 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f558c0c5530 con 0x7f558c0747b0 2024-08-03T20:57:23.196 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:23.192+0000 7f5593c00640 1 -- 172.21.15.32:0/326014815 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f558c074270 con 0x7f558c0747b0 2024-08-03T20:57:23.197 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:23.193+0000 7f5588c00640 1 -- 172.21.15.32:0/326014815 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f55840051c0 con 0x7f558c0747b0 2024-08-03T20:57:23.197 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:23.194+0000 7f5588c00640 1 --2- 172.21.15.32:0/326014815 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f556c077010 0x7f556c079520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:23.197 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:23.194+0000 7f5588c00640 1 -- 172.21.15.32:0/326014815 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(103..103 src has 1..103) ==== 7402+0+0 (secure 0 0 0) 0x7f5584099670 con 0x7f558c0747b0 2024-08-03T20:57:23.197 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:23.194+0000 7f5591200640 1 --2- 172.21.15.32:0/326014815 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f556c077010 0x7f556c079520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:23.197 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:23.195+0000 7f5591200640 1 --2- 172.21.15.32:0/326014815 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f556c077010 0x7f556c079520 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7f5580033000 tx=0x7f5580004730 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:57:23.199 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:23.198+0000 7f5588c00640 1 -- 172.21.15.32:0/326014815 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7f5584016030 con 0x7f558c0747b0 2024-08-03T20:57:23.445 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:23.444+0000 7f5593c00640 1 -- 172.21.15.32:0/326014815 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f558c141000 con 0x7f558c0747b0 2024-08-03T20:57:23.445 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:23.445+0000 7f5588c00640 1 -- 172.21.15.32:0/326014815 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v206) ==== 53+0+73 (secure 0 0 0) 0x7f5584061c00 con 0x7f558c0747b0 2024-08-03T20:57:23.446 INFO:teuthology.orchestra.run.smithi032.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-08-03T20:57:23.448 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:23.447+0000 7f5593c00640 1 -- 172.21.15.32:0/326014815 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f556c077010 msgr2=0x7f556c079520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:23.449 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:23.447+0000 7f5593c00640 1 --2- 172.21.15.32:0/326014815 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f556c077010 0x7f556c079520 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7f5580033000 tx=0x7f5580004730 comp rx=0 tx=0).stop 2024-08-03T20:57:23.449 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:23.447+0000 7f5593c00640 1 -- 172.21.15.32:0/326014815 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f558c0747b0 msgr2=0x7f558c0ac520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:23.449 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:23.447+0000 7f5593c00640 1 --2- 172.21.15.32:0/326014815 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f558c0747b0 0x7f558c0ac520 secure :-1 s=READY pgs=1532 cs=0 l=1 rev1=1 crypto rx=0x7f558400e9c0 tx=0x7f558400ee80 comp rx=0 tx=0).stop 2024-08-03T20:57:23.449 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:23.448+0000 7f5593c00640 1 -- 172.21.15.32:0/326014815 shutdown_connections 2024-08-03T20:57:23.449 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:23.448+0000 7f5593c00640 1 --2- 172.21.15.32:0/326014815 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f556c077010 0x7f556c079520 unknown :-1 s=CLOSED pgs=626 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:23.449 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:23.448+0000 7f5593c00640 1 --2- 172.21.15.32:0/326014815 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f558c0747b0 0x7f558c0ac520 unknown :-1 s=CLOSED pgs=1532 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:23.449 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:23.448+0000 7f5593c00640 1 --2- 172.21.15.32:0/326014815 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f558c073e60 0x7f558c0abfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:23.449 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:23.448+0000 7f5593c00640 1 --2- 172.21.15.32:0/326014815 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f558c06e930 0x7f558c0cb4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:23.449 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:23.448+0000 7f5593c00640 1 -- 172.21.15.32:0/326014815 >> 172.21.15.32:0/326014815 conn(0x7f558c0b1bd0 msgr2=0x7f558c0b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:57:23.449 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:23.448+0000 7f5593c00640 1 -- 172.21.15.32:0/326014815 shutdown_connections 2024-08-03T20:57:23.449 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:23.448+0000 7f5593c00640 1 -- 172.21.15.32:0/326014815 wait complete. 2024-08-03T20:57:23.472 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-08-03T20:57:23.473 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-08-03T20:57:23.473 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi032.front.sepia.ceph.com 2024-08-03T20:57:23.473 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-08-03T20:57:23.473 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-08-03T20:57:23.473 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-08-03T20:57:23.473 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2024-08-03T20:57:23.473 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-08-03T20:57:23.473 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-08-03T20:57:23.473 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-08-03T20:57:23.474 DEBUG:teuthology.orchestra.run.smithi032:> ip addr 2024-08-03T20:57:23.509 INFO:teuthology.orchestra.run.smithi032.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-08-03T20:57:23.509 INFO:teuthology.orchestra.run.smithi032.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-08-03T20:57:23.509 INFO:teuthology.orchestra.run.smithi032.stdout: inet 127.0.0.1/8 scope host lo 2024-08-03T20:57:23.509 INFO:teuthology.orchestra.run.smithi032.stdout: valid_lft forever preferred_lft forever 2024-08-03T20:57:23.510 INFO:teuthology.orchestra.run.smithi032.stdout: inet6 ::1/128 scope host 2024-08-03T20:57:23.510 INFO:teuthology.orchestra.run.smithi032.stdout: valid_lft forever preferred_lft forever 2024-08-03T20:57:23.510 INFO:teuthology.orchestra.run.smithi032.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-08-03T20:57:23.510 INFO:teuthology.orchestra.run.smithi032.stdout: link/ether 0c:c4:7a:6b:fd:0c brd ff:ff:ff:ff:ff:ff 2024-08-03T20:57:23.510 INFO:teuthology.orchestra.run.smithi032.stdout: altname enp7s0f0 2024-08-03T20:57:23.510 INFO:teuthology.orchestra.run.smithi032.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-08-03T20:57:23.510 INFO:teuthology.orchestra.run.smithi032.stdout: link/ether 0c:c4:7a:6b:fd:0d brd ff:ff:ff:ff:ff:ff 2024-08-03T20:57:23.510 INFO:teuthology.orchestra.run.smithi032.stdout: altname enp7s0f1 2024-08-03T20:57:23.510 INFO:teuthology.orchestra.run.smithi032.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-08-03T20:57:23.511 INFO:teuthology.orchestra.run.smithi032.stdout: link/ether ac:1f:6b:f8:26:02 brd ff:ff:ff:ff:ff:ff 2024-08-03T20:57:23.511 INFO:teuthology.orchestra.run.smithi032.stdout: altname enp1s0f0 2024-08-03T20:57:23.511 INFO:teuthology.orchestra.run.smithi032.stdout: inet 172.21.15.32/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-08-03T20:57:23.511 INFO:teuthology.orchestra.run.smithi032.stdout: valid_lft 41913sec preferred_lft 41913sec 2024-08-03T20:57:23.511 INFO:teuthology.orchestra.run.smithi032.stdout: inet6 fe80::ae1f:6bff:fef8:2602/64 scope link noprefixroute 2024-08-03T20:57:23.511 INFO:teuthology.orchestra.run.smithi032.stdout: valid_lft forever preferred_lft forever 2024-08-03T20:57:23.511 INFO:teuthology.orchestra.run.smithi032.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-08-03T20:57:23.511 INFO:teuthology.orchestra.run.smithi032.stdout: link/ether ac:1f:6b:f8:26:03 brd ff:ff:ff:ff:ff:ff 2024-08-03T20:57:23.511 INFO:teuthology.orchestra.run.smithi032.stdout: altname enp1s0f1 2024-08-03T20:57:23.511 INFO:teuthology.orchestra.run.smithi032.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-08-03T20:57:23.511 INFO:teuthology.orchestra.run.smithi032.stdout: link/ether 0a:23:2b:ca:2a:93 brd ff:ff:ff:ff:ff:ff 2024-08-03T20:57:23.511 INFO:teuthology.orchestra.run.smithi032.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-08-03T20:57:23.511 INFO:teuthology.orchestra.run.smithi032.stdout: valid_lft forever preferred_lft forever 2024-08-03T20:57:23.512 INFO:teuthology.orchestra.run.smithi032.stdout: inet6 fe80::d804:7cff:fed9:7ba4/64 scope link 2024-08-03T20:57:23.512 INFO:teuthology.orchestra.run.smithi032.stdout: valid_lft forever preferred_lft forever 2024-08-03T20:57:23.512 INFO:teuthology.orchestra.run.smithi032.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-08-03T20:57:23.512 INFO:teuthology.orchestra.run.smithi032.stdout: link/ether 22:86:28:30:4e:62 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-08-03T20:57:23.512 INFO:teuthology.orchestra.run.smithi032.stdout: inet6 fe80::2086:28ff:fe30:4e62/64 scope link 2024-08-03T20:57:23.512 INFO:teuthology.orchestra.run.smithi032.stdout: valid_lft forever preferred_lft forever 2024-08-03T20:57:23.512 INFO:teuthology.orchestra.run.smithi032.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-08-03T20:57:23.512 INFO:teuthology.orchestra.run.smithi032.stdout: link/ether 0a:23:2b:ca:2a:93 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-08-03T20:57:23.512 INFO:teuthology.orchestra.run.smithi032.stdout: inet6 fe80::823:2bff:feca:2a93/64 scope link 2024-08-03T20:57:23.512 INFO:teuthology.orchestra.run.smithi032.stdout: valid_lft forever preferred_lft forever 2024-08-03T20:57:23.513 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-08-03T20:57:23.513 DEBUG:teuthology.orchestra.run.smithi032:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-08-03T20:57:23.603 INFO:teuthology.orchestra.run.smithi032.stdout:1 2024-08-03T20:57:23.606 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-08-03T20:57:23.606 DEBUG:teuthology.orchestra.run.smithi032:> route 2024-08-03T20:57:23.642 INFO:teuthology.orchestra.run.smithi032.stdout:Kernel IP routing table 2024-08-03T20:57:23.642 INFO:teuthology.orchestra.run.smithi032.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-08-03T20:57:23.642 INFO:teuthology.orchestra.run.smithi032.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-08-03T20:57:23.643 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-08-03T20:57:23.643 INFO:teuthology.orchestra.run.smithi032.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-08-03T20:57:23.643 DEBUG:teuthology.orchestra.run.smithi032:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-08-03T20:57:23.643 DEBUG:teuthology.orchestra.run.smithi032:> set -e 2024-08-03T20:57:23.643 DEBUG:teuthology.orchestra.run.smithi032:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-08-03T20:57:23.643 DEBUG:teuthology.orchestra.run.smithi032:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-08-03T20:57:23.643 DEBUG:teuthology.orchestra.run.smithi032:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-08-03T20:57:23.643 DEBUG:teuthology.orchestra.run.smithi032:> ') 2024-08-03T20:57:23.763 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:23 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-08-03T20:57:23.834 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:23 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-08-03T20:57:23.841 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-08-03T20:57:23.842 DEBUG:teuthology.orchestra.run.smithi032:> ip netns list 2024-08-03T20:57:23.878 INFO:teuthology.orchestra.run.smithi032.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-08-03T20:57:23.878 INFO:teuthology.orchestra.run.smithi032.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-08-03T20:57:23.878 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-08-03T20:57:23.878 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-08-03T20:57:23.879 DEBUG:teuthology.orchestra.run.smithi032:> sudo ip link set brx.0 up 2024-08-03T20:57:23.975 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-08-03T20:57:23.975 DEBUG:teuthology.orchestra.run.smithi032:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-08-03T20:57:24.010 INFO:teuthology.orchestra.run.smithi032.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-08-03T20:57:24.011 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-08-03T20:57:24.011 DEBUG:teuthology.orchestra.run.smithi032:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-08-03T20:57:24.086 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2024-08-03T20:57:24.086 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-08-03T20:57:24.086 DEBUG:teuthology.orchestra.run.smithi032:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-08-03T20:57:24.283 INFO:teuthology.orchestra.run.smithi032.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-08-03T20:57:24.283 INFO:teuthology.orchestra.run.smithi032.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-08-03T20:57:24.283 INFO:teuthology.orchestra.run.smithi032.stdout:invalid new device string format 2024-08-03T20:57:24.303 INFO:teuthology.orchestra.run.smithi032.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-08-03T20:57:24.303 INFO:teuthology.orchestra.run.smithi032.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-08-03T20:57:24.303 INFO:teuthology.orchestra.run.smithi032.stdout:invalid new device string format 2024-08-03T20:57:24.303 INFO:teuthology.orchestra.run.smithi032.stdout:mount.ceph: resolved to: "172.21.15.32:3300,172.21.15.118:3300,172.21.15.118:3301" 2024-08-03T20:57:24.303 INFO:teuthology.orchestra.run.smithi032.stdout:mount.ceph: switching to using v1 address with old syntax 2024-08-03T20:57:24.336 INFO:teuthology.orchestra.run.smithi032.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-08-03T20:57:24.336 INFO:teuthology.orchestra.run.smithi032.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-08-03T20:57:24.336 INFO:teuthology.orchestra.run.smithi032.stdout:invalid new device string format 2024-08-03T20:57:24.336 INFO:teuthology.orchestra.run.smithi032.stdout:mount.ceph: resolved to: "172.21.15.32:3300,172.21.15.118:3300,172.21.15.118:3301" 2024-08-03T20:57:24.336 INFO:teuthology.orchestra.run.smithi032.stdout:mount.ceph: switching to using v1 address with old syntax 2024-08-03T20:57:24.336 INFO:teuthology.orchestra.run.smithi032.stdout:mount.ceph: resolved to: "172.21.15.32:6789,172.21.15.118:6789,172.21.15.118:6790" 2024-08-03T20:57:24.336 INFO:teuthology.orchestra.run.smithi032.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.32:6789,172.21.15.118:6789,172.21.15.118:6790:/ 2024-08-03T20:57:24.336 INFO:teuthology.orchestra.run.smithi032.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,key=0,fsid=40ab141a-01d7-49b9-ae3c-2edec9876e95" will pass to kernel 2024-08-03T20:57:24.338 INFO:teuthology.orchestra.run.smithi032.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2024-08-03T20:57:24.338 INFO:teuthology.orchestra.run.smithi032.stdout: You just mounted a file system that supports labels which does not 2024-08-03T20:57:24.338 INFO:teuthology.orchestra.run.smithi032.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-08-03T20:57:24.338 INFO:teuthology.orchestra.run.smithi032.stdout: applications will generate AVC messages and not be allowed access to 2024-08-03T20:57:24.338 INFO:teuthology.orchestra.run.smithi032.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-08-03T20:57:24.341 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-08-03T20:57:24.341 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-08-03T20:57:24.341 DEBUG:teuthology.orchestra.run.smithi032:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-08-03T20:57:24.397 DEBUG:teuthology.orchestra.run.smithi032:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-08-03T20:57:24.476 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:24 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-08-03T20:57:24.479 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:24 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-08-03T20:57:24.488 INFO:teuthology.orchestra.run.smithi032.stdout:client7331 2024-08-03T20:57:24.488 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-08-03T20:57:24.488 DEBUG:teuthology.orchestra.run.smithi032:> dd if=/proc/self/mounts of=/dev/stdout 2024-08-03T20:57:24.523 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-08-03T20:57:24.523 DEBUG:teuthology.orchestra.run.smithi032:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls -R .) 2024-08-03T20:57:24.569 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-08-03T20:57:24.569 DEBUG:teuthology.orchestra.run.smithi032:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir foo) 2024-08-03T20:57:24.602 INFO:teuthology.orchestra.run.smithi032.stdout:.: 2024-08-03T20:57:24.602 INFO:teuthology.orchestra.run.smithi032.stdout:subdir 2024-08-03T20:57:24.602 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-03T20:57:24.602 INFO:teuthology.orchestra.run.smithi032.stdout:./subdir: 2024-08-03T20:57:24.602 INFO:teuthology.orchestra.run.smithi032.stdout:sixmegs 2024-08-03T20:57:24.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-03T20:57:24.637 INFO:teuthology.orchestra.run.smithi032.stderr:mkdir: cannot create directory ‘foo’: Read-only file system 2024-08-03T20:57:24.637 INFO:tasks.cephfs.test_damage:mkdir: cannot create directory ‘foo’: Read-only file system 2024-08-03T20:57:24.637 INFO:tasks.cephfs.test_damage:Applying mutation 'Corrupt mds_snaptable' 2024-08-03T20:57:24.637 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-08-03T20:57:24.638 DEBUG:teuthology.orchestra.run.smithi032:> dd if=/proc/self/mounts of=/dev/stdout 2024-08-03T20:57:24.712 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-08-03T20:57:24.712 DEBUG:teuthology.orchestra.run.smithi032:> dd if=/proc/self/mounts of=/dev/stdout 2024-08-03T20:57:24.788 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-08-03T20:57:25.057 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:25.056+0000 7f7be7400640 1 -- 172.21.15.32:0/21279633 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7bc8004800 msgr2=0x7f7bc8004c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:25.058 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:25.056+0000 7f7be7400640 1 --2- 172.21.15.32:0/21279633 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7bc8004800 0x7f7bc8004c10 secure :-1 s=READY pgs=1534 cs=0 l=1 rev1=1 crypto rx=0x7f7be0072880 tx=0x7f7bc8005830 comp rx=0 tx=0).stop 2024-08-03T20:57:25.058 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:25.056+0000 7f7be7400640 1 -- 172.21.15.32:0/21279633 shutdown_connections 2024-08-03T20:57:25.058 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:25.056+0000 7f7be7400640 1 --2- 172.21.15.32:0/21279633 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7bc8004800 0x7f7bc8004c10 unknown :-1 s=CLOSED pgs=1534 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:25.058 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:25.056+0000 7f7be7400640 1 -- 172.21.15.32:0/21279633 >> 172.21.15.32:0/21279633 conn(0x7f7be00b1bd0 msgr2=0x7f7be00b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:57:25.058 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:25.056+0000 7f7be7400640 1 -- 172.21.15.32:0/21279633 shutdown_connections 2024-08-03T20:57:25.058 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:25.056+0000 7f7be7400640 1 -- 172.21.15.32:0/21279633 wait complete. 2024-08-03T20:57:25.059 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:25.059+0000 7f7be7400640 1 Processor -- start 2024-08-03T20:57:25.059 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:25.059+0000 7f7be7400640 1 -- start start 2024-08-03T20:57:25.063 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:25.060+0000 7f7be7400640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f7bc8004800 0x7f7be00b4ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:25.063 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:25.060+0000 7f7be7400640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f7be00c9250 0x7f7be013e930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:25.063 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:25.060+0000 7f7be7400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7be0141a60 0x7f7be0141e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:25.063 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:25.060+0000 7f7be7400640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f7be0145110 con 0x7f7be0141a60 2024-08-03T20:57:25.063 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:25.060+0000 7f7be7400640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f7be0144f90 con 0x7f7bc8004800 2024-08-03T20:57:25.063 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:25.060+0000 7f7be7400640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7f7be0145290 con 0x7f7be00c9250 2024-08-03T20:57:25.063 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:25.060+0000 7f7be5400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7be0141a60 0x7f7be0141e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:25.063 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:25.061+0000 7f7be5400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7be0141a60 0x7f7be0141e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:59522/0 (socket says 172.21.15.32:59522) 2024-08-03T20:57:25.063 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:25.061+0000 7f7be5400640 1 -- 172.21.15.32:0/415649205 learned_addr learned my addr 172.21.15.32:0/415649205 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:57:25.064 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:25.061+0000 7f7bdfe00640 1 --2- 172.21.15.32:0/415649205 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f7be00c9250 0x7f7be013e930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:25.064 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:25.061+0000 7f7be4a00640 1 --2- 172.21.15.32:0/415649205 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f7bc8004800 0x7f7be00b4ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:25.064 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:25.061+0000 7f7be5400640 1 -- 172.21.15.32:0/415649205 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f7be00c9250 msgr2=0x7f7be013e930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:25.064 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:25.061+0000 7f7be5400640 1 --2- 172.21.15.32:0/415649205 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f7be00c9250 0x7f7be013e930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:25.064 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:25.061+0000 7f7be5400640 1 -- 172.21.15.32:0/415649205 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f7bc8004800 msgr2=0x7f7be00b4ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:25.064 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:25.061+0000 7f7be5400640 1 --2- 172.21.15.32:0/415649205 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f7bc8004800 0x7f7be00b4ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:25.064 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:25.061+0000 7f7be5400640 1 -- 172.21.15.32:0/415649205 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7be00b4ff0 con 0x7f7be0141a60 2024-08-03T20:57:25.064 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:25.061+0000 7f7bdfe00640 1 --2- 172.21.15.32:0/415649205 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f7be00c9250 0x7f7be013e930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-03T20:57:25.064 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:25.061+0000 7f7be5400640 1 --2- 172.21.15.32:0/415649205 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7be0141a60 0x7f7be0141e70 secure :-1 s=READY pgs=1535 cs=0 l=1 rev1=1 crypto rx=0x7f7be0072db0 tx=0x7f7bd800ec50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:57:25.064 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:25.062+0000 7f7bdcc00640 1 -- 172.21.15.32:0/415649205 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7bd800ca10 con 0x7f7be0141a60 2024-08-03T20:57:25.064 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:25.062+0000 7f7bdcc00640 1 -- 172.21.15.32:0/415649205 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f7bd8018070 con 0x7f7be0141a60 2024-08-03T20:57:25.064 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:25.062+0000 7f7bdcc00640 1 -- 172.21.15.32:0/415649205 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7bd8011460 con 0x7f7be0141a60 2024-08-03T20:57:25.064 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:25.062+0000 7f7be7400640 1 -- 172.21.15.32:0/415649205 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7be01423b0 con 0x7f7be0141a60 2024-08-03T20:57:25.065 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:25.062+0000 7f7be7400640 1 -- 172.21.15.32:0/415649205 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7be01f5ca0 con 0x7f7be0141a60 2024-08-03T20:57:25.065 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:25.063+0000 7f7be7400640 1 -- 172.21.15.32:0/415649205 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7be0074350 con 0x7f7be0141a60 2024-08-03T20:57:25.065 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:25.064+0000 7f7bdcc00640 1 -- 172.21.15.32:0/415649205 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f7bd8011600 con 0x7f7be0141a60 2024-08-03T20:57:25.066 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:25.065+0000 7f7bdcc00640 1 --2- 172.21.15.32:0/415649205 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f7bc0077010 0x7f7bc0079520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:25.066 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:25.065+0000 7f7bdcc00640 1 -- 172.21.15.32:0/415649205 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(103..103 src has 1..103) ==== 7402+0+0 (secure 0 0 0) 0x7f7bd8099400 con 0x7f7be0141a60 2024-08-03T20:57:25.066 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:25.066+0000 7f7be4a00640 1 --2- 172.21.15.32:0/415649205 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f7bc0077010 0x7f7bc0079520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:25.067 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:25.067+0000 7f7be4a00640 1 --2- 172.21.15.32:0/415649205 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f7bc0077010 0x7f7bc0079520 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7f7bd4009cd0 tx=0x7f7bd4005f90 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:57:25.070 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:25.070+0000 7f7bdcc00640 1 -- 172.21.15.32:0/415649205 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7f7be01423b0 con 0x7f7be0141a60 2024-08-03T20:57:25.291 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:25.291+0000 7f7be7400640 1 -- 172.21.15.32:0/415649205 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f7be015f790 con 0x7f7be0141a60 2024-08-03T20:57:25.292 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:25.291+0000 7f7bdcc00640 1 -- 172.21.15.32:0/415649205 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 36 entries v103) ==== 81+0+2192 (secure 0 0 0) 0x7f7bd8061ec0 con 0x7f7be0141a60 2024-08-03T20:57:25.292 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6835/1885034124 2024-08-04T20:57:08.190707+0000 2024-08-03T20:57:25.292 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6833/1885034124 2024-08-04T20:57:08.190707+0000 2024-08-03T20:57:25.292 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6835/2362145329 2024-08-04T20:57:02.150199+0000 2024-08-03T20:57:25.292 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6833/2362145329 2024-08-04T20:57:02.150199+0000 2024-08-03T20:57:25.292 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6835/739874257 2024-08-04T20:56:56.101271+0000 2024-08-03T20:57:25.292 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6833/739874257 2024-08-04T20:56:56.101271+0000 2024-08-03T20:57:25.293 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6835/2242533881 2024-08-04T20:56:48.048099+0000 2024-08-03T20:57:25.293 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6833/1172050551 2024-08-04T20:55:26.214113+0000 2024-08-03T20:57:25.293 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6833/2806168786 2024-08-04T20:56:37.279160+0000 2024-08-03T20:57:25.293 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6835/2292690833 2024-08-04T20:55:20.021329+0000 2024-08-03T20:57:25.293 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6835/3431300225 2024-08-04T20:56:21.088927+0000 2024-08-03T20:57:25.293 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6835/2806168786 2024-08-04T20:56:37.279160+0000 2024-08-03T20:57:25.293 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6835/756683988 2024-08-04T20:55:14.119228+0000 2024-08-03T20:57:25.293 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6833/756683988 2024-08-04T20:55:14.119228+0000 2024-08-03T20:57:25.293 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6835/1016338829 2024-08-04T20:55:06.071485+0000 2024-08-03T20:57:25.293 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6835/1172050551 2024-08-04T20:55:26.214113+0000 2024-08-03T20:57:25.293 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6833/2194902489 2024-08-04T20:55:00.359126+0000 2024-08-03T20:57:25.293 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6833/3431300225 2024-08-04T20:56:21.088927+0000 2024-08-03T20:57:25.293 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6833/2242533881 2024-08-04T20:56:48.048099+0000 2024-08-03T20:57:25.293 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6833/1707454554 2024-08-04T20:54:38.409411+0000 2024-08-03T20:57:25.294 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6835/3191654276 2024-08-04T20:55:42.433364+0000 2024-08-03T20:57:25.294 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6833/2803901497 2024-08-04T20:55:55.181144+0000 2024-08-03T20:57:25.294 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6835/2194902489 2024-08-04T20:55:00.359126+0000 2024-08-03T20:57:25.294 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6835/2803901497 2024-08-04T20:55:55.181144+0000 2024-08-03T20:57:25.294 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6833/2292690833 2024-08-04T20:55:20.021329+0000 2024-08-03T20:57:25.294 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6835/1707454554 2024-08-04T20:54:38.409411+0000 2024-08-03T20:57:25.294 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6835/4172198298 2024-08-04T20:56:04.256182+0000 2024-08-03T20:57:25.294 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6835/3771304550 2024-08-04T20:55:58.212752+0000 2024-08-03T20:57:25.294 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6833/3191654276 2024-08-04T20:55:42.433364+0000 2024-08-03T20:57:25.294 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6835/2178506907 2024-08-04T20:56:40.045549+0000 2024-08-03T20:57:25.295 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6833/3771304550 2024-08-04T20:55:58.212752+0000 2024-08-03T20:57:25.295 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6833/4172198298 2024-08-04T20:56:04.256182+0000 2024-08-03T20:57:25.295 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6833/2278473729 2024-08-04T20:56:13.041693+0000 2024-08-03T20:57:25.295 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6833/1016338829 2024-08-04T20:55:06.071485+0000 2024-08-03T20:57:25.295 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6835/2278473729 2024-08-04T20:56:13.041693+0000 2024-08-03T20:57:25.295 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6833/2178506907 2024-08-04T20:56:40.045549+0000 2024-08-03T20:57:25.296 INFO:teuthology.orchestra.run.smithi032.stderr:listed 36 entries 2024-08-03T20:57:25.296 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:25.294+0000 7f7be7400640 1 -- 172.21.15.32:0/415649205 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f7bc0077010 msgr2=0x7f7bc0079520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:25.296 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:25.294+0000 7f7be7400640 1 --2- 172.21.15.32:0/415649205 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f7bc0077010 0x7f7bc0079520 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7f7bd4009cd0 tx=0x7f7bd4005f90 comp rx=0 tx=0).stop 2024-08-03T20:57:25.296 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:25.294+0000 7f7be7400640 1 -- 172.21.15.32:0/415649205 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7be0141a60 msgr2=0x7f7be0141e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:25.296 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:25.294+0000 7f7be7400640 1 --2- 172.21.15.32:0/415649205 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7be0141a60 0x7f7be0141e70 secure :-1 s=READY pgs=1535 cs=0 l=1 rev1=1 crypto rx=0x7f7be0072db0 tx=0x7f7bd800ec50 comp rx=0 tx=0).stop 2024-08-03T20:57:25.296 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:25.294+0000 7f7be7400640 1 -- 172.21.15.32:0/415649205 shutdown_connections 2024-08-03T20:57:25.296 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:25.294+0000 7f7be7400640 1 --2- 172.21.15.32:0/415649205 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f7bc0077010 0x7f7bc0079520 unknown :-1 s=CLOSED pgs=627 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:25.296 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:25.294+0000 7f7be7400640 1 --2- 172.21.15.32:0/415649205 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7be0141a60 0x7f7be0141e70 unknown :-1 s=CLOSED pgs=1535 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:25.297 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:25.294+0000 7f7be7400640 1 --2- 172.21.15.32:0/415649205 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f7be00c9250 0x7f7be013e930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:25.297 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:25.294+0000 7f7be7400640 1 --2- 172.21.15.32:0/415649205 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f7bc8004800 0x7f7be00b4ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:25.297 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:25.294+0000 7f7be7400640 1 -- 172.21.15.32:0/415649205 >> 172.21.15.32:0/415649205 conn(0x7f7be00b1bd0 msgr2=0x7f7be01e5c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:57:25.297 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:25.294+0000 7f7be7400640 1 -- 172.21.15.32:0/415649205 shutdown_connections 2024-08-03T20:57:25.297 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:25.294+0000 7f7be7400640 1 -- 172.21.15.32:0/415649205 wait complete. 2024-08-03T20:57:25.319 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-08-03T20:57:25.319 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-08-03T20:57:25.319 DEBUG:teuthology.orchestra.run.smithi032:> sudo umount /home/ubuntu/cephtest/mnt.0 -f 2024-08-03T20:57:25.399 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi032.front.sepia.ceph.com 2024-08-03T20:57:25.400 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-08-03T20:57:25.400 DEBUG:teuthology.orchestra.run.smithi032:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-08-03T20:57:25.437 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-08-03T20:57:25.706 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:25.705+0000 7f1768400640 1 -- 172.21.15.32:0/1057128282 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1748003a00 msgr2=0x7f1748003e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:25.706 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:25.705+0000 7f1768400640 1 --2- 172.21.15.32:0/1057128282 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1748003a00 0x7f1748003e70 secure :-1 s=READY pgs=1537 cs=0 l=1 rev1=1 crypto rx=0x7f1754009d00 tx=0x7f1748005830 comp rx=0 tx=0).stop 2024-08-03T20:57:25.706 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:25.705+0000 7f1766400640 1 -- 172.21.15.32:0/1057128282 reap_dead start 2024-08-03T20:57:25.706 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:25.705+0000 7f1768400640 1 -- 172.21.15.32:0/1057128282 shutdown_connections 2024-08-03T20:57:25.707 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:25.705+0000 7f1768400640 1 -- 172.21.15.32:0/1057128282 >> 172.21.15.32:0/1057128282 conn(0x7f17600b1bd0 msgr2=0x7f17600b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:57:25.707 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:25.705+0000 7f1768400640 1 -- 172.21.15.32:0/1057128282 shutdown_connections 2024-08-03T20:57:25.707 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:25.705+0000 7f1768400640 1 -- 172.21.15.32:0/1057128282 wait complete. 2024-08-03T20:57:25.708 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:25.708+0000 7f1768400640 1 Processor -- start 2024-08-03T20:57:25.708 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:25.708+0000 7f1768400640 1 -- start start 2024-08-03T20:57:25.711 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:25.709+0000 7f1768400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1748003a00 0x7f17600c9250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:25.711 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:25.709+0000 7f1768400640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f17600b3b30 0x7f176015a910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:25.711 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:25.709+0000 7f1768400640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f17601f54a0 0x7f17601f58b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:25.711 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:25.709+0000 7f1768400640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f17600cf840 con 0x7f1748003a00 2024-08-03T20:57:25.711 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:25.709+0000 7f1768400640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f1760145f30 con 0x7f17600b3b30 2024-08-03T20:57:25.711 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:25.709+0000 7f1768400640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7f17600cf9c0 con 0x7f17601f54a0 2024-08-03T20:57:25.712 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:25.710+0000 7f1765a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1748003a00 0x7f17600c9250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:25.712 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:25.710+0000 7f1765a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1748003a00 0x7f17600c9250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:59532/0 (socket says 172.21.15.32:59532) 2024-08-03T20:57:25.712 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:25.710+0000 7f1765a00640 1 -- 172.21.15.32:0/3839726773 learned_addr learned my addr 172.21.15.32:0/3839726773 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:57:25.712 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:25.710+0000 7f1766400640 1 --2- 172.21.15.32:0/3839726773 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f17601f54a0 0x7f17601f58b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:25.712 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:25.710+0000 7f1765000640 1 --2- 172.21.15.32:0/3839726773 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f17600b3b30 0x7f176015a910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:25.712 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:25.710+0000 7f1765a00640 1 -- 172.21.15.32:0/3839726773 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f17601f54a0 msgr2=0x7f17601f58b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:25.712 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:25.710+0000 7f1765a00640 1 --2- 172.21.15.32:0/3839726773 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f17601f54a0 0x7f17601f58b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:25.712 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:25.710+0000 7f1765a00640 1 -- 172.21.15.32:0/3839726773 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f17600b3b30 msgr2=0x7f176015a910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:25.712 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:25.710+0000 7f1765a00640 1 --2- 172.21.15.32:0/3839726773 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f17600b3b30 0x7f176015a910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:25.712 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:25.710+0000 7f1765a00640 1 -- 172.21.15.32:0/3839726773 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f17601f8a60 con 0x7f1748003a00 2024-08-03T20:57:25.712 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:25.710+0000 7f1765000640 1 --2- 172.21.15.32:0/3839726773 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f17600b3b30 0x7f176015a910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-03T20:57:25.712 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:25.710+0000 7f1765a00640 1 --2- 172.21.15.32:0/3839726773 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1748003a00 0x7f17600c9250 secure :-1 s=READY pgs=1538 cs=0 l=1 rev1=1 crypto rx=0x7f1754036040 tx=0x7f1754003820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:57:25.712 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:25.711+0000 7f175d600640 1 -- 172.21.15.32:0/3839726773 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1754040030 con 0x7f1748003a00 2024-08-03T20:57:25.713 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:25.711+0000 7f175d600640 1 -- 172.21.15.32:0/3839726773 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f175402ba80 con 0x7f1748003a00 2024-08-03T20:57:25.713 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:25.711+0000 7f175d600640 1 -- 172.21.15.32:0/3839726773 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1754038040 con 0x7f1748003a00 2024-08-03T20:57:25.713 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:25.711+0000 7f1768400640 1 -- 172.21.15.32:0/3839726773 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f17601f8d40 con 0x7f1748003a00 2024-08-03T20:57:25.713 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:25.711+0000 7f1768400640 1 -- 172.21.15.32:0/3839726773 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f176015b160 con 0x7f1748003a00 2024-08-03T20:57:25.713 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:25.712+0000 7f1768400640 1 -- 172.21.15.32:0/3839726773 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1760144f90 con 0x7f1748003a00 2024-08-03T20:57:25.714 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:25.713+0000 7f175d600640 1 -- 172.21.15.32:0/3839726773 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f1754043070 con 0x7f1748003a00 2024-08-03T20:57:25.715 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:25.714+0000 7f175d600640 1 --2- 172.21.15.32:0/3839726773 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f1740077010 0x7f1740079520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:25.715 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:25.714+0000 7f175d600640 1 -- 172.21.15.32:0/3839726773 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(103..103 src has 1..103) ==== 7402+0+0 (secure 0 0 0) 0x7f17540ba0a0 con 0x7f1748003a00 2024-08-03T20:57:25.715 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:25.715+0000 7f1765000640 1 --2- 172.21.15.32:0/3839726773 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f1740077010 0x7f1740079520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:25.716 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:25.716+0000 7f1765000640 1 --2- 172.21.15.32:0/3839726773 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f1740077010 0x7f1740079520 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7f1760138320 tx=0x7f174c000f10 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:57:25.720 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:25.720+0000 7f175d600640 1 -- 172.21.15.32:0/3839726773 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7f175404a070 con 0x7f1748003a00 2024-08-03T20:57:25.968 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:25.967+0000 7f1768400640 1 -- 172.21.15.32:0/3839726773 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7f1760074680 con 0x7f1748003a00 2024-08-03T20:57:25.968 INFO:tasks.ceph.mon.a.smithi032.stderr:2024-08-03T20:57:25.968+0000 7f4e6d600640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-08-03T20:57:25.970 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:25.970+0000 7f175d600640 1 -- 172.21.15.32:0/3839726773 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v207) ==== 161+0+0 (secure 0 0 0) 0x7f175402bc50 con 0x7f1748003a00 2024-08-03T20:57:25.971 INFO:teuthology.orchestra.run.smithi032.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-08-03T20:57:25.973 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:25.973+0000 7f1768400640 1 -- 172.21.15.32:0/3839726773 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f1740077010 msgr2=0x7f1740079520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:25.974 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:25.973+0000 7f1768400640 1 --2- 172.21.15.32:0/3839726773 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f1740077010 0x7f1740079520 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7f1760138320 tx=0x7f174c000f10 comp rx=0 tx=0).stop 2024-08-03T20:57:25.974 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:25.973+0000 7f1768400640 1 -- 172.21.15.32:0/3839726773 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1748003a00 msgr2=0x7f17600c9250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:25.974 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:25.973+0000 7f1768400640 1 --2- 172.21.15.32:0/3839726773 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1748003a00 0x7f17600c9250 secure :-1 s=READY pgs=1538 cs=0 l=1 rev1=1 crypto rx=0x7f1754036040 tx=0x7f1754003820 comp rx=0 tx=0).stop 2024-08-03T20:57:25.974 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:25.973+0000 7f1768400640 1 -- 172.21.15.32:0/3839726773 shutdown_connections 2024-08-03T20:57:25.974 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:25.973+0000 7f1768400640 1 --2- 172.21.15.32:0/3839726773 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f1740077010 0x7f1740079520 unknown :-1 s=CLOSED pgs=628 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:25.974 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:25.973+0000 7f1768400640 1 --2- 172.21.15.32:0/3839726773 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f17600b3b30 0x7f176015a910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:25.974 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:25.973+0000 7f1768400640 1 --2- 172.21.15.32:0/3839726773 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f17601f54a0 0x7f17601f58b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:25.974 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:25.973+0000 7f1768400640 1 --2- 172.21.15.32:0/3839726773 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1748003a00 0x7f17600c9250 unknown :-1 s=CLOSED pgs=1538 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:25.974 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:25.973+0000 7f1768400640 1 -- 172.21.15.32:0/3839726773 >> 172.21.15.32:0/3839726773 conn(0x7f17600b1bd0 msgr2=0x7f17600b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:57:25.974 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:25.973+0000 7f1768400640 1 -- 172.21.15.32:0/3839726773 shutdown_connections 2024-08-03T20:57:25.974 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:25.973+0000 7f1768400640 1 -- 172.21.15.32:0/3839726773 wait complete. 2024-08-03T20:57:25.975 INFO:tasks.ceph.mds.a.smithi032.stderr: -1910> 2024-08-03T20:57:14.236+0000 7ff12d200640 -1 mds.0.purge_queue operator(): Error -22 loading Journaler 2024-08-03T20:57:25.975 INFO:tasks.ceph.mds.a.smithi032.stderr: -1907> 2024-08-03T20:57:14.236+0000 7ff12d200640 -1 mds.0.202 unhandled write error (22) Invalid argument, force readonly... 2024-08-03T20:57:25.977 INFO:tasks.ceph.mds.a.smithi032.stderr: -764> 2024-08-03T20:57:18.223+0000 7ff130e00640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-08-03T20:57:25.997 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mds repaired 0 2024-08-03T20:57:26.096 INFO:tasks.ceph.mds.a.smithi032.stdout:starting mds.a at 2024-08-03T20:57:26.231 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:26.230+0000 7ff97d000640 1 -- 172.21.15.32:0/965805070 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff9780cf6f0 msgr2=0x7ff9780cfb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:26.231 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:26.230+0000 7ff97d000640 1 --2- 172.21.15.32:0/965805070 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff9780cf6f0 0x7ff9780cfb00 secure :-1 s=READY pgs=1543 cs=0 l=1 rev1=1 crypto rx=0x7ff970005fd0 tx=0x7ff97002a900 comp rx=0 tx=0).stop 2024-08-03T20:57:26.231 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:26.230+0000 7ff97d000640 1 -- 172.21.15.32:0/965805070 shutdown_connections 2024-08-03T20:57:26.231 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:26.230+0000 7ff97d000640 1 --2- 172.21.15.32:0/965805070 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff9780cf6f0 0x7ff9780cfb00 unknown :-1 s=CLOSED pgs=1543 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:26.231 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:26.230+0000 7ff97d000640 1 --2- 172.21.15.32:0/965805070 >> v2:172.21.15.118:3300/0 conn(0x7ff9780d4ba0 0x7ff9780b3240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:26.231 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:26.230+0000 7ff97d000640 1 -- 172.21.15.32:0/965805070 >> 172.21.15.32:0/965805070 conn(0x7ff9780d4080 msgr2=0x7ff9780d4490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:57:26.231 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:26.230+0000 7ff97d000640 1 -- 172.21.15.32:0/965805070 shutdown_connections 2024-08-03T20:57:26.231 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:26.230+0000 7ff97d000640 1 -- 172.21.15.32:0/965805070 wait complete. 2024-08-03T20:57:26.233 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:26.232+0000 7ff97d000640 1 Processor -- start 2024-08-03T20:57:26.233 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:26.232+0000 7ff97d000640 1 -- start start 2024-08-03T20:57:26.235 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:26.234+0000 7ff97d000640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ff9780b93d0 0x7ff978152470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:26.235 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:26.234+0000 7ff97d000640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7ff9780cf6f0 0x7ff97815fd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:26.235 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:26.234+0000 7ff97d000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff9780d4ba0 0x7ff9781f5640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:26.235 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:26.234+0000 7ff97d000640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7ff9780d08f0 con 0x7ff9780d4ba0 2024-08-03T20:57:26.236 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:26.234+0000 7ff97d000640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7ff9780d0770 con 0x7ff9780b93d0 2024-08-03T20:57:26.236 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:26.234+0000 7ff97d000640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7ff9780d0a70 con 0x7ff9780cf6f0 2024-08-03T20:57:26.236 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:26.234+0000 7ff977e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff9780d4ba0 0x7ff9781f5640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:26.236 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:26.234+0000 7ff977e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff9780d4ba0 0x7ff9781f5640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:59568/0 (socket says 172.21.15.32:59568) 2024-08-03T20:57:26.236 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:26.234+0000 7ff977e00640 1 -- 172.21.15.32:0/3829581991 learned_addr learned my addr 172.21.15.32:0/3829581991 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:57:26.236 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:26.235+0000 7ff977400640 1 --2- 172.21.15.32:0/3829581991 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ff9780b93d0 0x7ff978152470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:26.236 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:26.235+0000 7ff976a00640 1 --2- 172.21.15.32:0/3829581991 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7ff9780cf6f0 0x7ff97815fd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:26.236 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:26.235+0000 7ff977e00640 1 -- 172.21.15.32:0/3829581991 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7ff9780cf6f0 msgr2=0x7ff97815fd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:26.236 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:26.235+0000 7ff977e00640 1 --2- 172.21.15.32:0/3829581991 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7ff9780cf6f0 0x7ff97815fd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:26.236 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:26.235+0000 7ff977e00640 1 -- 172.21.15.32:0/3829581991 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ff9780b93d0 msgr2=0x7ff978152470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:26.236 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:26.235+0000 7ff977e00640 1 --2- 172.21.15.32:0/3829581991 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ff9780b93d0 0x7ff978152470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:26.236 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:26.235+0000 7ff977e00640 1 -- 172.21.15.32:0/3829581991 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff978160280 con 0x7ff9780d4ba0 2024-08-03T20:57:26.237 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:26.235+0000 7ff977e00640 1 --2- 172.21.15.32:0/3829581991 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff9780d4ba0 0x7ff9781f5640 secure :-1 s=READY pgs=1544 cs=0 l=1 rev1=1 crypto rx=0x7ff970006010 tx=0x7ff970003ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:57:26.237 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:26.235+0000 7ff96f400640 1 -- 172.21.15.32:0/3829581991 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff970031ba0 con 0x7ff9780d4ba0 2024-08-03T20:57:26.237 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:26.235+0000 7ff96f400640 1 -- 172.21.15.32:0/3829581991 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7ff970031e30 con 0x7ff9780d4ba0 2024-08-03T20:57:26.238 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:26.235+0000 7ff96f400640 1 -- 172.21.15.32:0/3829581991 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9700332d0 con 0x7ff9780d4ba0 2024-08-03T20:57:26.238 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:26.235+0000 7ff97d000640 1 -- 172.21.15.32:0/3829581991 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff97809f4e0 con 0x7ff9780d4ba0 2024-08-03T20:57:26.238 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:26.236+0000 7ff97d000640 1 -- 172.21.15.32:0/3829581991 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff97809f200 con 0x7ff9780d4ba0 2024-08-03T20:57:26.238 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:26.237+0000 7ff97d000640 1 -- 172.21.15.32:0/3829581991 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff9780d0f70 con 0x7ff9780d4ba0 2024-08-03T20:57:26.238 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:26.238+0000 7ff96f400640 1 -- 172.21.15.32:0/3829581991 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7ff970033470 con 0x7ff9780d4ba0 2024-08-03T20:57:26.239 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:26.239+0000 7ff96f400640 1 --2- 172.21.15.32:0/3829581991 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7ff958077010 0x7ff958079520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:26.239 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:26.239+0000 7ff96f400640 1 -- 172.21.15.32:0/3829581991 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(104..104 src has 1..104) ==== 7488+0+0 (secure 0 0 0) 0x7ff9700ba6f0 con 0x7ff9780d4ba0 2024-08-03T20:57:26.240 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:26.239+0000 7ff977400640 1 --2- 172.21.15.32:0/3829581991 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7ff958077010 0x7ff958079520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:26.240 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:26.240+0000 7ff977400640 1 --2- 172.21.15.32:0/3829581991 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7ff958077010 0x7ff958079520 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7ff968009b30 tx=0x7ff968005f90 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:57:26.244 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:26.243+0000 7ff96f400640 1 -- 172.21.15.32:0/3829581991 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7ff97003f030 con 0x7ff9780d4ba0 2024-08-03T20:57:26.529 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:26.529+0000 7ff97d000640 1 -- 172.21.15.32:0/3829581991 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "mds repaired", "role": "0"} v 0) -- 0x7ff9780d1630 con 0x7ff9780d4ba0 2024-08-03T20:57:26.973 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:26.972+0000 7ff96f400640 1 -- 172.21.15.32:0/3829581991 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "mds repaired", "role": "0"}]=0 nothing to do: rank is not damaged v208) ==== 107+0+0 (secure 0 0 0) 0x7ff970083210 con 0x7ff9780d4ba0 2024-08-03T20:57:26.973 INFO:teuthology.orchestra.run.smithi032.stderr:nothing to do: rank is not damaged 2024-08-03T20:57:26.976 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:26.975+0000 7ff97d000640 1 -- 172.21.15.32:0/3829581991 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7ff958077010 msgr2=0x7ff958079520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:26.976 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:26.975+0000 7ff97d000640 1 --2- 172.21.15.32:0/3829581991 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7ff958077010 0x7ff958079520 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7ff968009b30 tx=0x7ff968005f90 comp rx=0 tx=0).stop 2024-08-03T20:57:26.976 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:26.975+0000 7ff97d000640 1 -- 172.21.15.32:0/3829581991 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff9780d4ba0 msgr2=0x7ff9781f5640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:26.976 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:26.975+0000 7ff97d000640 1 --2- 172.21.15.32:0/3829581991 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff9780d4ba0 0x7ff9781f5640 secure :-1 s=READY pgs=1544 cs=0 l=1 rev1=1 crypto rx=0x7ff970006010 tx=0x7ff970003ba0 comp rx=0 tx=0).stop 2024-08-03T20:57:26.976 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:26.975+0000 7ff97d000640 1 -- 172.21.15.32:0/3829581991 shutdown_connections 2024-08-03T20:57:26.976 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:26.975+0000 7ff97d000640 1 --2- 172.21.15.32:0/3829581991 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7ff958077010 0x7ff958079520 unknown :-1 s=CLOSED pgs=629 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:26.977 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:26.975+0000 7ff97d000640 1 --2- 172.21.15.32:0/3829581991 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff9780d4ba0 0x7ff9781f5640 unknown :-1 s=CLOSED pgs=1544 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:26.977 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:26.975+0000 7ff97d000640 1 --2- 172.21.15.32:0/3829581991 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7ff9780cf6f0 0x7ff97815fd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:26.977 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:26.975+0000 7ff97d000640 1 --2- 172.21.15.32:0/3829581991 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ff9780b93d0 0x7ff978152470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:26.977 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:26.975+0000 7ff97d000640 1 -- 172.21.15.32:0/3829581991 >> 172.21.15.32:0/3829581991 conn(0x7ff9780d4080 msgr2=0x7ff9780b6ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:57:26.977 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:26.976+0000 7ff97d000640 1 -- 172.21.15.32:0/3829581991 shutdown_connections 2024-08-03T20:57:26.977 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:26.976+0000 7ff97d000640 1 -- 172.21.15.32:0/3829581991 wait complete. 2024-08-03T20:57:27.007 DEBUG:teuthology.orchestra.run.smithi032:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata import - 2024-08-03T20:57:27.152 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.151+0000 7fe7779edf40 1 Processor -- start 2024-08-03T20:57:27.152 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.151+0000 7fe7779edf40 1 -- start start 2024-08-03T20:57:27.154 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.152+0000 7fe7779edf40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x556ea15cc360 0x556ea15c6e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:27.154 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.152+0000 7fe7779edf40 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x556ea15cdea0 0x556ea15e7810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:27.154 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.153+0000 7fe7779edf40 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x556ea15e9cb0 0x556ea15e7d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:27.154 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.153+0000 7fe7779edf40 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x556ea1463e30 con 0x556ea15e9cb0 2024-08-03T20:57:27.154 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.153+0000 7fe7779edf40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x556ea14c1bf0 con 0x556ea15cc360 2024-08-03T20:57:27.154 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.153+0000 7fe7779edf40 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x556ea15cd1b0 con 0x556ea15cdea0 2024-08-03T20:57:27.154 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.153+0000 7fe771600640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x556ea15e9cb0 0x556ea15e7d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:27.154 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.153+0000 7fe771600640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x556ea15e9cb0 0x556ea15e7d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:59586/0 (socket says 172.21.15.32:59586) 2024-08-03T20:57:27.155 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.153+0000 7fe771600640 1 -- 172.21.15.32:0/2565964049 learned_addr learned my addr 172.21.15.32:0/2565964049 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:57:27.155 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.153+0000 7fe76be00640 1 --2- 172.21.15.32:0/2565964049 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x556ea15cdea0 0x556ea15e7810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:27.155 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.153+0000 7fe770c00640 1 --2- 172.21.15.32:0/2565964049 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x556ea15cc360 0x556ea15c6e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:27.155 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.153+0000 7fe771600640 1 -- 172.21.15.32:0/2565964049 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x556ea15cdea0 msgr2=0x556ea15e7810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:57:27.155 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.153+0000 7fe771600640 1 --2- 172.21.15.32:0/2565964049 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x556ea15cdea0 0x556ea15e7810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:27.155 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.153+0000 7fe771600640 1 -- 172.21.15.32:0/2565964049 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x556ea15cc360 msgr2=0x556ea15c6e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:57:27.155 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.153+0000 7fe771600640 1 --2- 172.21.15.32:0/2565964049 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x556ea15cc360 0x556ea15c6e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:27.155 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.153+0000 7fe771600640 1 -- 172.21.15.32:0/2565964049 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556ea15e2c20 con 0x556ea15e9cb0 2024-08-03T20:57:27.155 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.153+0000 7fe770c00640 1 --2- 172.21.15.32:0/2565964049 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x556ea15cc360 0x556ea15c6e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-03T20:57:27.155 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.154+0000 7fe771600640 1 --2- 172.21.15.32:0/2565964049 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x556ea15e9cb0 0x556ea15e7d50 secure :-1 s=READY pgs=1546 cs=0 l=1 rev1=1 crypto rx=0x7fe76c001510 tx=0x7fe76c0019d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b6fb3629f157ef06 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:57:27.157 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.154+0000 7fe76aa00640 1 -- 172.21.15.32:0/2565964049 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe76c016070 con 0x556ea15e9cb0 2024-08-03T20:57:27.157 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.154+0000 7fe76aa00640 1 -- 172.21.15.32:0/2565964049 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fe76c00f800 con 0x556ea15e9cb0 2024-08-03T20:57:27.157 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.154+0000 7fe76aa00640 1 -- 172.21.15.32:0/2565964049 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe76c00fc30 con 0x556ea15e9cb0 2024-08-03T20:57:27.157 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.154+0000 7fe7779edf40 1 -- 172.21.15.32:0/2565964049 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x556ea15e9cb0 msgr2=0x556ea15e7d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:27.158 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.155+0000 7fe7779edf40 1 --2- 172.21.15.32:0/2565964049 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x556ea15e9cb0 0x556ea15e7d50 secure :-1 s=READY pgs=1546 cs=0 l=1 rev1=1 crypto rx=0x7fe76c001510 tx=0x7fe76c0019d0 comp rx=0 tx=0).stop 2024-08-03T20:57:27.158 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.155+0000 7fe7779edf40 1 -- 172.21.15.32:0/2565964049 shutdown_connections 2024-08-03T20:57:27.158 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.155+0000 7fe7779edf40 1 --2- 172.21.15.32:0/2565964049 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x556ea15e9cb0 0x556ea15e7d50 unknown :-1 s=CLOSED pgs=1546 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:27.158 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.155+0000 7fe7779edf40 1 --2- 172.21.15.32:0/2565964049 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x556ea15cdea0 0x556ea15e7810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:27.158 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.155+0000 7fe7779edf40 1 --2- 172.21.15.32:0/2565964049 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x556ea15cc360 0x556ea15c6e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:27.158 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.155+0000 7fe7779edf40 1 -- 172.21.15.32:0/2565964049 >> 172.21.15.32:0/2565964049 conn(0x556ea1435ac0 msgr2=0x556ea15c9600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:57:27.159 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.155+0000 7fe7779edf40 1 -- 172.21.15.32:0/2565964049 shutdown_connections 2024-08-03T20:57:27.159 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.155+0000 7fe7779edf40 1 -- 172.21.15.32:0/2565964049 wait complete. 2024-08-03T20:57:27.159 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.156+0000 7fe7779edf40 1 Processor -- start 2024-08-03T20:57:27.159 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.156+0000 7fe7779edf40 1 -- start start 2024-08-03T20:57:27.161 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.157+0000 7fe7779edf40 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x556ea15cc360 0x556ea16860b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:27.161 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.157+0000 7fe7779edf40 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x556ea15cdea0 0x556ea15bce20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:27.161 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.157+0000 7fe7779edf40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x556ea15e9cb0 0x556ea15c8730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:27.161 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.157+0000 7fe7779edf40 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x556ea15c9e90 con 0x556ea15cc360 2024-08-03T20:57:27.161 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.157+0000 7fe7779edf40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x556ea155c310 con 0x556ea15e9cb0 2024-08-03T20:57:27.161 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.157+0000 7fe7779edf40 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x556ea155d070 con 0x556ea15cdea0 2024-08-03T20:57:27.161 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.158+0000 7fe770c00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x556ea15cc360 0x556ea16860b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:27.161 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.158+0000 7fe770c00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x556ea15cc360 0x556ea16860b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:59592/0 (socket says 172.21.15.32:59592) 2024-08-03T20:57:27.161 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.158+0000 7fe770c00640 1 -- 172.21.15.32:0/4119417304 learned_addr learned my addr 172.21.15.32:0/4119417304 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:57:27.161 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.158+0000 7fe76be00640 1 --2- 172.21.15.32:0/4119417304 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x556ea15cdea0 0x556ea15bce20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:27.161 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.158+0000 7fe771600640 1 --2- 172.21.15.32:0/4119417304 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x556ea15e9cb0 0x556ea15c8730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:27.161 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.158+0000 7fe770c00640 1 -- 172.21.15.32:0/4119417304 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x556ea15cdea0 msgr2=0x556ea15bce20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:27.161 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.158+0000 7fe770c00640 1 --2- 172.21.15.32:0/4119417304 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x556ea15cdea0 0x556ea15bce20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:27.162 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.158+0000 7fe770c00640 1 -- 172.21.15.32:0/4119417304 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x556ea15e9cb0 msgr2=0x556ea15c8730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:27.162 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.158+0000 7fe770c00640 1 --2- 172.21.15.32:0/4119417304 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x556ea15e9cb0 0x556ea15c8730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:27.162 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.158+0000 7fe770c00640 1 -- 172.21.15.32:0/4119417304 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556ea1513e40 con 0x556ea15cc360 2024-08-03T20:57:27.162 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.158+0000 7fe771600640 1 --2- 172.21.15.32:0/4119417304 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x556ea15e9cb0 0x556ea15c8730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-03T20:57:27.162 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.158+0000 7fe770c00640 1 --2- 172.21.15.32:0/4119417304 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x556ea15cc360 0x556ea16860b0 secure :-1 s=READY pgs=1547 cs=0 l=1 rev1=1 crypto rx=0x7fe764025e80 tx=0x7fe76404d6e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:57:27.162 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.159+0000 7fe769600640 1 -- 172.21.15.32:0/4119417304 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe764056670 con 0x556ea15cc360 2024-08-03T20:57:27.162 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.159+0000 7fe769600640 1 -- 172.21.15.32:0/4119417304 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fe764066070 con 0x556ea15cc360 2024-08-03T20:57:27.162 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.159+0000 7fe769600640 1 -- 172.21.15.32:0/4119417304 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe764056a90 con 0x556ea15cc360 2024-08-03T20:57:27.162 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.159+0000 7fe7779edf40 1 -- 172.21.15.32:0/4119417304 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x556ea1514120 con 0x556ea15cc360 2024-08-03T20:57:27.162 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.159+0000 7fe7779edf40 1 -- 172.21.15.32:0/4119417304 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556ea15146e0 con 0x556ea15cc360 2024-08-03T20:57:27.163 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.161+0000 7fe769600640 1 -- 172.21.15.32:0/4119417304 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7fe764056c30 con 0x556ea15cc360 2024-08-03T20:57:27.163 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.162+0000 7fe769600640 1 --2- 172.21.15.32:0/4119417304 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fe7500759f0 0x7fe750077f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:27.163 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.162+0000 7fe769600640 1 -- 172.21.15.32:0/4119417304 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(104..104 src has 1..104) ==== 7488+0+0 (secure 0 0 0) 0x7fe7640dc280 con 0x556ea15cc360 2024-08-03T20:57:27.164 INFO:teuthology.orchestra.run.smithi032.stdout:Importing pool 2024-08-03T20:57:27.164 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.162+0000 7fe7779edf40 1 --2- 172.21.15.32:0/4119417304 >> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] conn(0x556ea15c8c70 0x556ea1514ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:27.164 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.162+0000 7fe7779edf40 1 -- 172.21.15.32:0/4119417304 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:1 12.10 12:09401a81:::601.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104) -- 0x556ea1515420 con 0x556ea15c8c70 2024-08-03T20:57:27.165 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.163+0000 7fe76be00640 1 --2- 172.21.15.32:0/4119417304 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fe7500759f0 0x7fe750077f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:27.165 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.163+0000 7fe771600640 1 --2- 172.21.15.32:0/4119417304 >> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] conn(0x556ea15c8c70 0x556ea1514ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:27.165 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.163+0000 7fe76be00640 1 --2- 172.21.15.32:0/4119417304 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fe7500759f0 0x7fe750077f00 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x556ea15ca480 tx=0x7fe75c004880 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:57:27.165 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.163+0000 7fe771600640 1 --2- 172.21.15.32:0/4119417304 >> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] conn(0x556ea15c8c70 0x556ea1514ee0 crc :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:57:27.166 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.165+0000 7fe771600640 1 -- 172.21.15.32:0/4119417304 <== osd.0 v2:172.21.15.32:6816/3066156116 1 ==== osd_op_reply(1 601.00000000 [create] v104'567 uv541 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fe76c019040 con 0x556ea15c8c70 2024-08-03T20:57:27.166 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.165+0000 7fe7779edf40 1 -- 172.21.15.32:0/4119417304 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:2 12.10 12:09401a81:::601.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e104) -- 0x556ea15157f0 con 0x556ea15c8c70 2024-08-03T20:57:27.167 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.167+0000 7fe771600640 1 -- 172.21.15.32:0/4119417304 <== osd.0 v2:172.21.15.32:6816/3066156116 2 ==== osd_op_reply(2 601.00000000 [delete] v104'568 uv568 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe76c019040 con 0x556ea15c8c70 2024-08-03T20:57:27.168 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.167+0000 7fe7779edf40 1 -- 172.21.15.32:0/4119417304 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:3 12.10 12:09401a81:::601.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104) -- 0x556ea1515bc0 con 0x556ea15c8c70 2024-08-03T20:57:27.169 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.169+0000 7fe771600640 1 -- 172.21.15.32:0/4119417304 <== osd.0 v2:172.21.15.32:6816/3066156116 3 ==== osd_op_reply(3 601.00000000 [create] v104'569 uv569 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe76c019040 con 0x556ea15c8c70 2024-08-03T20:57:27.169 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::601.00000000:head# 2024-08-03T20:57:27.169 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.169+0000 7fe7779edf40 1 -- 172.21.15.32:0/4119417304 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:4 12.10 12:09401a81:::601.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104) -- 0x556ea168f980 con 0x556ea15c8c70 2024-08-03T20:57:27.171 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.170+0000 7fe771600640 1 -- 172.21.15.32:0/4119417304 <== osd.0 v2:172.21.15.32:6816/3066156116 4 ==== osd_op_reply(4 601.00000000 [setxattr (30)] v104'570 uv570 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe76c019040 con 0x556ea15c8c70 2024-08-03T20:57:27.171 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.170+0000 7fe7779edf40 1 -- 172.21.15.32:0/4119417304 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:5 12.10 12:09401a81:::601.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104) -- 0x556ea168fd20 con 0x556ea15c8c70 2024-08-03T20:57:27.172 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.172+0000 7fe771600640 1 -- 172.21.15.32:0/4119417304 <== osd.0 v2:172.21.15.32:6816/3066156116 5 ==== osd_op_reply(5 601.00000000 [setxattr (62)] v104'571 uv571 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe76c019040 con 0x556ea15c8c70 2024-08-03T20:57:27.172 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.172+0000 7fe7779edf40 1 -- 172.21.15.32:0/4119417304 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:6 12.10 12:09401a81:::601.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104) -- 0x556ea168e970 con 0x556ea15c8c70 2024-08-03T20:57:27.174 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.174+0000 7fe771600640 1 -- 172.21.15.32:0/4119417304 <== osd.0 v2:172.21.15.32:6816/3066156116 6 ==== osd_op_reply(6 601.00000000 [omap-set-header] v104'572 uv572 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe76c019040 con 0x556ea15c8c70 2024-08-03T20:57:27.174 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.174+0000 7fe7779edf40 1 -- 172.21.15.32:0/4119417304 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:7 12.10 12:0bd6d154:::602.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104) -- 0x556ea168ed60 con 0x556ea15c8c70 2024-08-03T20:57:27.175 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.175+0000 7fe771600640 1 -- 172.21.15.32:0/4119417304 <== osd.0 v2:172.21.15.32:6816/3066156116 7 ==== osd_op_reply(7 602.00000000 [create] v104'573 uv547 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fe76c019040 con 0x556ea15c8c70 2024-08-03T20:57:27.176 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.175+0000 7fe7779edf40 1 -- 172.21.15.32:0/4119417304 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:8 12.10 12:0bd6d154:::602.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e104) -- 0x556ea168f100 con 0x556ea15c8c70 2024-08-03T20:57:27.178 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.177+0000 7fe771600640 1 -- 172.21.15.32:0/4119417304 <== osd.0 v2:172.21.15.32:6816/3066156116 8 ==== osd_op_reply(8 602.00000000 [delete] v104'574 uv574 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe76c019040 con 0x556ea15c8c70 2024-08-03T20:57:27.178 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.177+0000 7fe7779edf40 1 -- 172.21.15.32:0/4119417304 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:9 12.10 12:0bd6d154:::602.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104) -- 0x556ea168f100 con 0x556ea15c8c70 2024-08-03T20:57:27.179 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.179+0000 7fe771600640 1 -- 172.21.15.32:0/4119417304 <== osd.0 v2:172.21.15.32:6816/3066156116 9 ==== osd_op_reply(9 602.00000000 [create] v104'575 uv575 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe76c019040 con 0x556ea15c8c70 2024-08-03T20:57:27.179 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::602.00000000:head# 2024-08-03T20:57:27.180 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.179+0000 7fe7779edf40 1 -- 172.21.15.32:0/4119417304 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:10 12.10 12:0bd6d154:::602.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104) -- 0x556ea168f100 con 0x556ea15c8c70 2024-08-03T20:57:27.181 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.180+0000 7fe771600640 1 -- 172.21.15.32:0/4119417304 <== osd.0 v2:172.21.15.32:6816/3066156116 10 ==== osd_op_reply(10 602.00000000 [setxattr (30)] v104'576 uv576 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe76c019040 con 0x556ea15c8c70 2024-08-03T20:57:27.181 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.180+0000 7fe7779edf40 1 -- 172.21.15.32:0/4119417304 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:11 12.10 12:0bd6d154:::602.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104) -- 0x556ea168f100 con 0x556ea15c8c70 2024-08-03T20:57:27.182 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.182+0000 7fe771600640 1 -- 172.21.15.32:0/4119417304 <== osd.0 v2:172.21.15.32:6816/3066156116 11 ==== osd_op_reply(11 602.00000000 [setxattr (62)] v104'577 uv577 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe76c019040 con 0x556ea15c8c70 2024-08-03T20:57:27.182 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.182+0000 7fe7779edf40 1 -- 172.21.15.32:0/4119417304 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:12 12.10 12:0bd6d154:::602.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104) -- 0x556ea168f100 con 0x556ea15c8c70 2024-08-03T20:57:27.184 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.184+0000 7fe771600640 1 -- 172.21.15.32:0/4119417304 <== osd.0 v2:172.21.15.32:6816/3066156116 12 ==== osd_op_reply(12 602.00000000 [omap-set-header] v104'578 uv578 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe76c019040 con 0x556ea15c8c70 2024-08-03T20:57:27.184 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.184+0000 7fe7779edf40 1 -- 172.21.15.32:0/4119417304 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:13 12.10 12:0cf6ad0f:::10000000000.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104) -- 0x556ea168f100 con 0x556ea15c8c70 2024-08-03T20:57:27.186 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.185+0000 7fe771600640 1 -- 172.21.15.32:0/4119417304 <== osd.0 v2:172.21.15.32:6816/3066156116 13 ==== osd_op_reply(13 10000000000.00000000 [create] v104'579 uv554 ondisk = -17 ((17) File exists)) ==== 164+0+0 (crc 0 0 0) 0x7fe76c019040 con 0x556ea15c8c70 2024-08-03T20:57:27.186 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.186+0000 7fe7779edf40 1 -- 172.21.15.32:0/4119417304 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:14 12.10 12:0cf6ad0f:::10000000000.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e104) -- 0x556ea168f100 con 0x556ea15c8c70 2024-08-03T20:57:27.187 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.187+0000 7fe771600640 1 -- 172.21.15.32:0/4119417304 <== osd.0 v2:172.21.15.32:6816/3066156116 14 ==== osd_op_reply(14 10000000000.00000000 [delete] v104'580 uv580 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7fe76c019040 con 0x556ea15c8c70 2024-08-03T20:57:27.188 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.187+0000 7fe7779edf40 1 -- 172.21.15.32:0/4119417304 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:15 12.10 12:0cf6ad0f:::10000000000.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104) -- 0x556ea168f100 con 0x556ea15c8c70 2024-08-03T20:57:27.189 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.189+0000 7fe771600640 1 -- 172.21.15.32:0/4119417304 <== osd.0 v2:172.21.15.32:6816/3066156116 15 ==== osd_op_reply(15 10000000000.00000000 [create] v104'581 uv581 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7fe76c019040 con 0x556ea15c8c70 2024-08-03T20:57:27.189 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::10000000000.00000000:head# 2024-08-03T20:57:27.190 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.189+0000 7fe7779edf40 1 -- 172.21.15.32:0/4119417304 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:16 12.10 12:0cf6ad0f:::10000000000.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104) -- 0x556ea168f100 con 0x556ea15c8c70 2024-08-03T20:57:27.191 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.190+0000 7fe771600640 1 -- 172.21.15.32:0/4119417304 <== osd.0 v2:172.21.15.32:6816/3066156116 16 ==== osd_op_reply(16 10000000000.00000000 [setxattr (30)] v104'582 uv582 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7fe76c019040 con 0x556ea15c8c70 2024-08-03T20:57:27.191 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.190+0000 7fe7779edf40 1 -- 172.21.15.32:0/4119417304 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:17 12.10 12:0cf6ad0f:::10000000000.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104) -- 0x556ea168f100 con 0x556ea15c8c70 2024-08-03T20:57:27.192 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.192+0000 7fe771600640 1 -- 172.21.15.32:0/4119417304 <== osd.0 v2:172.21.15.32:6816/3066156116 17 ==== osd_op_reply(17 10000000000.00000000 [setxattr (62)] v104'583 uv583 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7fe76c019040 con 0x556ea15c8c70 2024-08-03T20:57:27.192 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.192+0000 7fe7779edf40 1 -- 172.21.15.32:0/4119417304 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:18 12.10 12:0cf6ad0f:::10000000000.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104) -- 0x556ea168f100 con 0x556ea15c8c70 2024-08-03T20:57:27.194 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.193+0000 7fe771600640 1 -- 172.21.15.32:0/4119417304 <== osd.0 v2:172.21.15.32:6816/3066156116 18 ==== osd_op_reply(18 10000000000.00000000 [omap-set-header] v104'584 uv584 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7fe76c019040 con 0x556ea15c8c70 2024-08-03T20:57:27.194 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.193+0000 7fe7779edf40 1 -- 172.21.15.32:0/4119417304 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:19 12.10 12:0cf6ad0f:::10000000000.00000000:head [omap-set-vals in=593b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104) -- 0x556ea16900c0 con 0x556ea15c8c70 2024-08-03T20:57:27.196 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.195+0000 7fe771600640 1 -- 172.21.15.32:0/4119417304 <== osd.0 v2:172.21.15.32:6816/3066156116 19 ==== osd_op_reply(19 10000000000.00000000 [omap-set-vals] v104'585 uv585 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7fe76c019040 con 0x556ea15c8c70 2024-08-03T20:57:27.196 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.195+0000 7fe7779edf40 1 -- 172.21.15.32:0/4119417304 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:20 12.10 12:0d82a743:::600.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104) -- 0x556ea16900c0 con 0x556ea15c8c70 2024-08-03T20:57:27.197 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.197+0000 7fe771600640 1 -- 172.21.15.32:0/4119417304 <== osd.0 v2:172.21.15.32:6816/3066156116 20 ==== osd_op_reply(20 600.00000000 [create] v104'586 uv560 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fe76c019040 con 0x556ea15c8c70 2024-08-03T20:57:27.197 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.197+0000 7fe7779edf40 1 -- 172.21.15.32:0/4119417304 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:21 12.10 12:0d82a743:::600.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e104) -- 0x556ea16900c0 con 0x556ea15c8c70 2024-08-03T20:57:27.199 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.198+0000 7fe771600640 1 -- 172.21.15.32:0/4119417304 <== osd.0 v2:172.21.15.32:6816/3066156116 21 ==== osd_op_reply(21 600.00000000 [delete] v104'587 uv587 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe76c019040 con 0x556ea15c8c70 2024-08-03T20:57:27.199 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.198+0000 7fe7779edf40 1 -- 172.21.15.32:0/4119417304 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:22 12.10 12:0d82a743:::600.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104) -- 0x556ea16900c0 con 0x556ea15c8c70 2024-08-03T20:57:27.200 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.200+0000 7fe771600640 1 -- 172.21.15.32:0/4119417304 <== osd.0 v2:172.21.15.32:6816/3066156116 22 ==== osd_op_reply(22 600.00000000 [create] v104'588 uv588 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe76c019040 con 0x556ea15c8c70 2024-08-03T20:57:27.200 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::600.00000000:head# 2024-08-03T20:57:27.201 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.200+0000 7fe7779edf40 1 -- 172.21.15.32:0/4119417304 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:23 12.10 12:0d82a743:::600.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104) -- 0x556ea16910d0 con 0x556ea15c8c70 2024-08-03T20:57:27.202 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.201+0000 7fe771600640 1 -- 172.21.15.32:0/4119417304 <== osd.0 v2:172.21.15.32:6816/3066156116 23 ==== osd_op_reply(23 600.00000000 [setxattr (30)] v104'589 uv589 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe76c019040 con 0x556ea15c8c70 2024-08-03T20:57:27.202 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.201+0000 7fe7779edf40 1 -- 172.21.15.32:0/4119417304 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:24 12.10 12:0d82a743:::600.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104) -- 0x556ea16910d0 con 0x556ea15c8c70 2024-08-03T20:57:27.203 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.203+0000 7fe771600640 1 -- 172.21.15.32:0/4119417304 <== osd.0 v2:172.21.15.32:6816/3066156116 24 ==== osd_op_reply(24 600.00000000 [setxattr (62)] v104'590 uv590 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe76c019040 con 0x556ea15c8c70 2024-08-03T20:57:27.203 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.203+0000 7fe7779edf40 1 -- 172.21.15.32:0/4119417304 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:25 12.10 12:0d82a743:::600.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104) -- 0x556ea16900c0 con 0x556ea15c8c70 2024-08-03T20:57:27.205 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.204+0000 7fe771600640 1 -- 172.21.15.32:0/4119417304 <== osd.0 v2:172.21.15.32:6816/3066156116 25 ==== osd_op_reply(25 600.00000000 [omap-set-header] v104'591 uv591 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe76c019040 con 0x556ea15c8c70 2024-08-03T20:57:27.205 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.204+0000 7fe7779edf40 1 -- 172.21.15.32:0/4119417304 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:26 12.10 12:0d89b25e:::603.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104) -- 0x556ea16900c0 con 0x556ea15c8c70 2024-08-03T20:57:27.210 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.206+0000 7fe771600640 1 -- 172.21.15.32:0/4119417304 <== osd.0 v2:172.21.15.32:6816/3066156116 26 ==== osd_op_reply(26 603.00000000 [create] v104'592 uv566 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fe76c019040 con 0x556ea15c8c70 2024-08-03T20:57:27.210 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.206+0000 7fe7779edf40 1 -- 172.21.15.32:0/4119417304 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:27 12.10 12:0d89b25e:::603.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e104) -- 0x556ea16900c0 con 0x556ea15c8c70 2024-08-03T20:57:27.210 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.208+0000 7fe771600640 1 -- 172.21.15.32:0/4119417304 <== osd.0 v2:172.21.15.32:6816/3066156116 27 ==== osd_op_reply(27 603.00000000 [delete] v104'593 uv593 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe76c019040 con 0x556ea15c8c70 2024-08-03T20:57:27.210 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.208+0000 7fe7779edf40 1 -- 172.21.15.32:0/4119417304 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:28 12.10 12:0d89b25e:::603.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104) -- 0x556ea16900c0 con 0x556ea15c8c70 2024-08-03T20:57:27.210 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.209+0000 7fe771600640 1 -- 172.21.15.32:0/4119417304 <== osd.0 v2:172.21.15.32:6816/3066156116 28 ==== osd_op_reply(28 603.00000000 [create] v104'594 uv594 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe76c019040 con 0x556ea15c8c70 2024-08-03T20:57:27.211 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::603.00000000:head# 2024-08-03T20:57:27.211 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.209+0000 7fe7779edf40 1 -- 172.21.15.32:0/4119417304 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:29 12.10 12:0d89b25e:::603.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104) -- 0x556ea16910d0 con 0x556ea15c8c70 2024-08-03T20:57:27.211 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.211+0000 7fe771600640 1 -- 172.21.15.32:0/4119417304 <== osd.0 v2:172.21.15.32:6816/3066156116 29 ==== osd_op_reply(29 603.00000000 [setxattr (30)] v104'595 uv595 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe76c019040 con 0x556ea15c8c70 2024-08-03T20:57:27.211 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.211+0000 7fe7779edf40 1 -- 172.21.15.32:0/4119417304 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:30 12.10 12:0d89b25e:::603.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104) -- 0x556ea16910d0 con 0x556ea15c8c70 2024-08-03T20:57:27.212 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.212+0000 7fe771600640 1 -- 172.21.15.32:0/4119417304 <== osd.0 v2:172.21.15.32:6816/3066156116 30 ==== osd_op_reply(30 603.00000000 [setxattr (62)] v104'596 uv596 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe76c019040 con 0x556ea15c8c70 2024-08-03T20:57:27.213 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.212+0000 7fe7779edf40 1 -- 172.21.15.32:0/4119417304 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:31 12.10 12:0d89b25e:::603.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104) -- 0x556ea16900c0 con 0x556ea15c8c70 2024-08-03T20:57:27.214 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.214+0000 7fe771600640 1 -- 172.21.15.32:0/4119417304 <== osd.0 v2:172.21.15.32:6816/3066156116 31 ==== osd_op_reply(31 603.00000000 [omap-set-header] v104'597 uv597 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe76c019040 con 0x556ea15c8c70 2024-08-03T20:57:27.214 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.214+0000 7fe7779edf40 1 --2- 172.21.15.32:0/4119417304 >> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] conn(0x556ea16905d0 0x556ea16929f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:27.214 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.214+0000 7fe7779edf40 1 -- 172.21.15.32:0/4119417304 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:32 12.14 12:287034c4:::1.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104) -- 0x556ea1692f30 con 0x556ea16905d0 2024-08-03T20:57:27.215 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.214+0000 7fe770c00640 1 --2- 172.21.15.32:0/4119417304 >> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] conn(0x556ea16905d0 0x556ea16929f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:27.215 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.215+0000 7fe770c00640 1 --2- 172.21.15.32:0/4119417304 >> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] conn(0x556ea16905d0 0x556ea16929f0 crc :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:57:27.217 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.217+0000 7fe770c00640 1 -- 172.21.15.32:0/4119417304 <== osd.5 v2:172.21.15.118:6810/2724891770 1 ==== osd_op_reply(32 1.00000000.inode [create] v104'421 uv400 ondisk = -17 ((17) File exists)) ==== 160+0+0 (crc 0 0 0) 0x7fe7640a09c0 con 0x556ea16905d0 2024-08-03T20:57:27.217 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.217+0000 7fe7779edf40 1 -- 172.21.15.32:0/4119417304 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:33 12.14 12:287034c4:::1.00000000.inode:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e104) -- 0x556ea16932d0 con 0x556ea16905d0 2024-08-03T20:57:27.219 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.218+0000 7fe770c00640 1 -- 172.21.15.32:0/4119417304 <== osd.5 v2:172.21.15.118:6810/2724891770 2 ==== osd_op_reply(33 1.00000000.inode [delete] v104'422 uv422 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7fe7640a09c0 con 0x556ea16905d0 2024-08-03T20:57:27.219 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.218+0000 7fe7779edf40 1 -- 172.21.15.32:0/4119417304 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:34 12.14 12:287034c4:::1.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104) -- 0x556ea1693670 con 0x556ea16905d0 2024-08-03T20:57:27.220 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.220+0000 7fe770c00640 1 -- 172.21.15.32:0/4119417304 <== osd.5 v2:172.21.15.118:6810/2724891770 3 ==== osd_op_reply(34 1.00000000.inode [create] v104'423 uv423 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7fe7640a09c0 con 0x556ea16905d0 2024-08-03T20:57:27.221 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::1.00000000.inode:head# 2024-08-03T20:57:27.221 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.220+0000 7fe7779edf40 1 -- 172.21.15.32:0/4119417304 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:35 12.14 12:287034c4:::1.00000000.inode:head [write 0~582 in=582b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104) -- 0x556ea1693a10 con 0x556ea16905d0 2024-08-03T20:57:27.222 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.222+0000 7fe770c00640 1 -- 172.21.15.32:0/4119417304 <== osd.5 v2:172.21.15.118:6810/2724891770 4 ==== osd_op_reply(35 1.00000000.inode [write 0~582] v104'424 uv424 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7fe7640a09c0 con 0x556ea16905d0 2024-08-03T20:57:27.222 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.222+0000 7fe7779edf40 1 -- 172.21.15.32:0/4119417304 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:36 12.14 12:287034c4:::1.00000000.inode:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104) -- 0x556ea1693db0 con 0x556ea16905d0 2024-08-03T20:57:27.224 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.223+0000 7fe770c00640 1 -- 172.21.15.32:0/4119417304 <== osd.5 v2:172.21.15.118:6810/2724891770 5 ==== osd_op_reply(36 1.00000000.inode [omap-set-header] v104'425 uv425 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7fe7640a09c0 con 0x556ea16905d0 2024-08-03T20:57:27.224 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.223+0000 7fe7779edf40 1 -- 172.21.15.32:0/4119417304 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:37 12.14 12:292cf221:::200.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104) -- 0x556ea1694150 con 0x556ea16905d0 2024-08-03T20:57:27.225 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.225+0000 7fe770c00640 1 -- 172.21.15.32:0/4119417304 <== osd.5 v2:172.21.15.118:6810/2724891770 6 ==== osd_op_reply(37 200.00000000 [create] v104'426 uv419 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fe7640a09c0 con 0x556ea16905d0 2024-08-03T20:57:27.225 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.225+0000 7fe7779edf40 1 -- 172.21.15.32:0/4119417304 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:38 12.14 12:292cf221:::200.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e104) -- 0x556ea1694150 con 0x556ea16905d0 2024-08-03T20:57:27.227 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.226+0000 7fe770c00640 1 -- 172.21.15.32:0/4119417304 <== osd.5 v2:172.21.15.118:6810/2724891770 7 ==== osd_op_reply(38 200.00000000 [delete] v104'427 uv427 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe7640a09c0 con 0x556ea16905d0 2024-08-03T20:57:27.227 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.226+0000 7fe7779edf40 1 -- 172.21.15.32:0/4119417304 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:39 12.14 12:292cf221:::200.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104) -- 0x556ea1694150 con 0x556ea16905d0 2024-08-03T20:57:27.228 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.228+0000 7fe770c00640 1 -- 172.21.15.32:0/4119417304 <== osd.5 v2:172.21.15.118:6810/2724891770 8 ==== osd_op_reply(39 200.00000000 [create] v104'428 uv428 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe7640a09c0 con 0x556ea16905d0 2024-08-03T20:57:27.228 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::200.00000000:head# 2024-08-03T20:57:27.228 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.228+0000 7fe7779edf40 1 -- 172.21.15.32:0/4119417304 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:40 12.14 12:292cf221:::200.00000000:head [write 0~90 in=90b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104) -- 0x556ea1694590 con 0x556ea16905d0 2024-08-03T20:57:27.230 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.229+0000 7fe770c00640 1 -- 172.21.15.32:0/4119417304 <== osd.5 v2:172.21.15.118:6810/2724891770 9 ==== osd_op_reply(40 200.00000000 [write 0~90] v104'429 uv429 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe7640a09c0 con 0x556ea16905d0 2024-08-03T20:57:27.230 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.229+0000 7fe7779edf40 1 -- 172.21.15.32:0/4119417304 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:41 12.14 12:292cf221:::200.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104) -- 0x556ea1694590 con 0x556ea16905d0 2024-08-03T20:57:27.231 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.231+0000 7fe770c00640 1 -- 172.21.15.32:0/4119417304 <== osd.5 v2:172.21.15.118:6810/2724891770 10 ==== osd_op_reply(41 200.00000000 [omap-set-header] v104'430 uv430 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe7640a09c0 con 0x556ea16905d0 2024-08-03T20:57:27.231 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.231+0000 7fe7779edf40 1 -- 172.21.15.32:0/4119417304 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:42 12.14 12:2e2fa760:::200.00000001:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104) -- 0x556ea1694280 con 0x556ea16905d0 2024-08-03T20:57:27.233 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.232+0000 7fe770c00640 1 -- 172.21.15.32:0/4119417304 <== osd.5 v2:172.21.15.118:6810/2724891770 11 ==== osd_op_reply(42 200.00000001 [create] v104'431 uv420 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fe7640a09c0 con 0x556ea16905d0 2024-08-03T20:57:27.233 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.232+0000 7fe7779edf40 1 -- 172.21.15.32:0/4119417304 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:43 12.14 12:2e2fa760:::200.00000001:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e104) -- 0x556ea1694230 con 0x556ea16905d0 2024-08-03T20:57:27.234 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.234+0000 7fe770c00640 1 -- 172.21.15.32:0/4119417304 <== osd.5 v2:172.21.15.118:6810/2724891770 12 ==== osd_op_reply(43 200.00000001 [delete] v104'432 uv432 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe7640a09c0 con 0x556ea16905d0 2024-08-03T20:57:27.234 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.234+0000 7fe7779edf40 1 -- 172.21.15.32:0/4119417304 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:44 12.14 12:2e2fa760:::200.00000001:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104) -- 0x556ea1694230 con 0x556ea16905d0 2024-08-03T20:57:27.235 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.235+0000 7fe770c00640 1 -- 172.21.15.32:0/4119417304 <== osd.5 v2:172.21.15.118:6810/2724891770 13 ==== osd_op_reply(44 200.00000001 [create] v104'433 uv433 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe7640a09c0 con 0x556ea16905d0 2024-08-03T20:57:27.236 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::200.00000001:head# 2024-08-03T20:57:27.236 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.235+0000 7fe7779edf40 1 -- 172.21.15.32:0/4119417304 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:45 12.14 12:2e2fa760:::200.00000001:head [write 0~18118 in=18118b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104) -- 0x556ea1694230 con 0x556ea16905d0 2024-08-03T20:57:27.237 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.237+0000 7fe770c00640 1 -- 172.21.15.32:0/4119417304 <== osd.5 v2:172.21.15.118:6810/2724891770 14 ==== osd_op_reply(45 200.00000001 [write 0~18118] v104'434 uv434 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe7640a09c0 con 0x556ea16905d0 2024-08-03T20:57:27.237 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.237+0000 7fe7779edf40 1 -- 172.21.15.32:0/4119417304 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:46 12.14 12:2e2fa760:::200.00000001:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104) -- 0x556ea1694230 con 0x556ea16905d0 2024-08-03T20:57:27.239 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.238+0000 7fe770c00640 1 -- 172.21.15.32:0/4119417304 <== osd.5 v2:172.21.15.118:6810/2724891770 15 ==== osd_op_reply(46 200.00000001 [omap-set-header] v104'435 uv435 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe7640a09c0 con 0x556ea16905d0 2024-08-03T20:57:27.239 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.239+0000 7fe7779edf40 1 -- 172.21.15.32:0/4119417304 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:47 12.14 12:2f55791f:::606.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104) -- 0x556ea1694230 con 0x556ea16905d0 2024-08-03T20:57:27.240 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.240+0000 7fe770c00640 1 -- 172.21.15.32:0/4119417304 <== osd.5 v2:172.21.15.118:6810/2724891770 16 ==== osd_op_reply(47 606.00000000 [create] v104'436 uv416 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fe7640a09c0 con 0x556ea16905d0 2024-08-03T20:57:27.241 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.240+0000 7fe7779edf40 1 -- 172.21.15.32:0/4119417304 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:48 12.14 12:2f55791f:::606.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e104) -- 0x556ea1694230 con 0x556ea16905d0 2024-08-03T20:57:27.242 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.241+0000 7fe770c00640 1 -- 172.21.15.32:0/4119417304 <== osd.5 v2:172.21.15.118:6810/2724891770 17 ==== osd_op_reply(48 606.00000000 [delete] v104'437 uv437 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe7640a09c0 con 0x556ea16905d0 2024-08-03T20:57:27.242 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.241+0000 7fe7779edf40 1 -- 172.21.15.32:0/4119417304 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:49 12.14 12:2f55791f:::606.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104) -- 0x556ea1694230 con 0x556ea16905d0 2024-08-03T20:57:27.243 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.243+0000 7fe770c00640 1 -- 172.21.15.32:0/4119417304 <== osd.5 v2:172.21.15.118:6810/2724891770 18 ==== osd_op_reply(49 606.00000000 [create] v104'438 uv438 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe7640a09c0 con 0x556ea16905d0 2024-08-03T20:57:27.243 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::606.00000000:head# 2024-08-03T20:57:27.244 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.243+0000 7fe7779edf40 1 -- 172.21.15.32:0/4119417304 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:50 12.14 12:2f55791f:::606.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104) -- 0x556ea1694230 con 0x556ea16905d0 2024-08-03T20:57:27.245 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.244+0000 7fe770c00640 1 -- 172.21.15.32:0/4119417304 <== osd.5 v2:172.21.15.118:6810/2724891770 19 ==== osd_op_reply(50 606.00000000 [setxattr (30)] v104'439 uv439 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe7640a09c0 con 0x556ea16905d0 2024-08-03T20:57:27.245 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.244+0000 7fe7779edf40 1 -- 172.21.15.32:0/4119417304 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:51 12.14 12:2f55791f:::606.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104) -- 0x556ea1694230 con 0x556ea16905d0 2024-08-03T20:57:27.246 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.246+0000 7fe770c00640 1 -- 172.21.15.32:0/4119417304 <== osd.5 v2:172.21.15.118:6810/2724891770 20 ==== osd_op_reply(51 606.00000000 [setxattr (62)] v104'440 uv440 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe7640a09c0 con 0x556ea16905d0 2024-08-03T20:57:27.246 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.246+0000 7fe7779edf40 1 -- 172.21.15.32:0/4119417304 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:52 12.14 12:2f55791f:::606.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104) -- 0x556ea1694230 con 0x556ea16905d0 2024-08-03T20:57:27.248 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.247+0000 7fe770c00640 1 -- 172.21.15.32:0/4119417304 <== osd.5 v2:172.21.15.118:6810/2724891770 21 ==== osd_op_reply(52 606.00000000 [omap-set-header] v104'441 uv441 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe7640a09c0 con 0x556ea16905d0 2024-08-03T20:57:27.248 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.247+0000 7fe7779edf40 1 -- 172.21.15.32:0/4119417304 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:53 12.1c 12:3ed09add:::607.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104) -- 0x556ea1694230 con 0x556ea15c8c70 2024-08-03T20:57:27.249 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.249+0000 7fe771600640 1 -- 172.21.15.32:0/4119417304 <== osd.0 v2:172.21.15.32:6816/3066156116 32 ==== osd_op_reply(53 607.00000000 [create] v104'111 uv110 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fe76c019040 con 0x556ea15c8c70 2024-08-03T20:57:27.249 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.249+0000 7fe7779edf40 1 -- 172.21.15.32:0/4119417304 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:54 12.1c 12:3ed09add:::607.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e104) -- 0x556ea1694230 con 0x556ea15c8c70 2024-08-03T20:57:27.250 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.250+0000 7fe771600640 1 -- 172.21.15.32:0/4119417304 <== osd.0 v2:172.21.15.32:6816/3066156116 33 ==== osd_op_reply(54 607.00000000 [delete] v104'112 uv112 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe76c019040 con 0x556ea15c8c70 2024-08-03T20:57:27.251 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.250+0000 7fe7779edf40 1 -- 172.21.15.32:0/4119417304 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:55 12.1c 12:3ed09add:::607.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104) -- 0x556ea1694230 con 0x556ea15c8c70 2024-08-03T20:57:27.252 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.251+0000 7fe771600640 1 -- 172.21.15.32:0/4119417304 <== osd.0 v2:172.21.15.32:6816/3066156116 34 ==== osd_op_reply(55 607.00000000 [create] v104'113 uv113 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe76c019040 con 0x556ea15c8c70 2024-08-03T20:57:27.252 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::607.00000000:head# 2024-08-03T20:57:27.252 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.251+0000 7fe7779edf40 1 -- 172.21.15.32:0/4119417304 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:56 12.1c 12:3ed09add:::607.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104) -- 0x556ea1694230 con 0x556ea15c8c70 2024-08-03T20:57:27.253 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.253+0000 7fe771600640 1 -- 172.21.15.32:0/4119417304 <== osd.0 v2:172.21.15.32:6816/3066156116 35 ==== osd_op_reply(56 607.00000000 [setxattr (30)] v104'114 uv114 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe76c019040 con 0x556ea15c8c70 2024-08-03T20:57:27.253 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.253+0000 7fe7779edf40 1 -- 172.21.15.32:0/4119417304 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:57 12.1c 12:3ed09add:::607.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104) -- 0x556ea1695240 con 0x556ea15c8c70 2024-08-03T20:57:27.254 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.254+0000 7fe771600640 1 -- 172.21.15.32:0/4119417304 <== osd.0 v2:172.21.15.32:6816/3066156116 36 ==== osd_op_reply(57 607.00000000 [setxattr (62)] v104'115 uv115 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe76c019040 con 0x556ea15c8c70 2024-08-03T20:57:27.254 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.254+0000 7fe7779edf40 1 -- 172.21.15.32:0/4119417304 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:58 12.1c 12:3ed09add:::607.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104) -- 0x556ea1694230 con 0x556ea15c8c70 2024-08-03T20:57:27.256 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.255+0000 7fe771600640 1 -- 172.21.15.32:0/4119417304 <== osd.0 v2:172.21.15.32:6816/3066156116 37 ==== osd_op_reply(58 607.00000000 [omap-set-header] v104'116 uv116 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe76c019040 con 0x556ea15c8c70 2024-08-03T20:57:27.256 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.255+0000 7fe7779edf40 1 --2- 172.21.15.32:0/4119417304 >> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] conn(0x556ea16948b0 0x556ea1694cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:27.256 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.255+0000 7fe7779edf40 1 -- 172.21.15.32:0/4119417304 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:59 12.6 12:654134d2:::mds0_openfiles.0:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104) -- 0x556ea1697810 con 0x556ea16948b0 2024-08-03T20:57:27.256 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.256+0000 7fe76be00640 1 --2- 172.21.15.32:0/4119417304 >> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] conn(0x556ea16948b0 0x556ea1694cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:27.257 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.256+0000 7fe76be00640 1 --2- 172.21.15.32:0/4119417304 >> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] conn(0x556ea16948b0 0x556ea1694cc0 crc :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:57:27.258 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.258+0000 7fe76be00640 1 -- 172.21.15.32:0/4119417304 <== osd.1 v2:172.21.15.32:6824/1551241759 1 ==== osd_op_reply(59 mds0_openfiles.0 [create] v104'90 uv89 ondisk = -17 ((17) File exists)) ==== 160+0+0 (crc 0 0 0) 0x7fe75c01f040 con 0x556ea16948b0 2024-08-03T20:57:27.258 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.258+0000 7fe7779edf40 1 -- 172.21.15.32:0/4119417304 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:60 12.6 12:654134d2:::mds0_openfiles.0:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e104) -- 0x556ea1697bb0 con 0x556ea16948b0 2024-08-03T20:57:27.259 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.259+0000 7fe76be00640 1 -- 172.21.15.32:0/4119417304 <== osd.1 v2:172.21.15.32:6824/1551241759 2 ==== osd_op_reply(60 mds0_openfiles.0 [delete] v104'91 uv91 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7fe75c01f040 con 0x556ea16948b0 2024-08-03T20:57:27.260 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.259+0000 7fe7779edf40 1 -- 172.21.15.32:0/4119417304 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:61 12.6 12:654134d2:::mds0_openfiles.0:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104) -- 0x556ea1697ff0 con 0x556ea16948b0 2024-08-03T20:57:27.261 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds0_openfiles.0:head# 2024-08-03T20:57:27.261 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.260+0000 7fe76be00640 1 -- 172.21.15.32:0/4119417304 <== osd.1 v2:172.21.15.32:6824/1551241759 3 ==== osd_op_reply(61 mds0_openfiles.0 [create] v104'92 uv92 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7fe75c01f040 con 0x556ea16948b0 2024-08-03T20:57:27.261 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.260+0000 7fe7779edf40 1 -- 172.21.15.32:0/4119417304 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:62 12.6 12:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104) -- 0x556ea168f4a0 con 0x556ea16948b0 2024-08-03T20:57:27.262 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.262+0000 7fe76be00640 1 -- 172.21.15.32:0/4119417304 <== osd.1 v2:172.21.15.32:6824/1551241759 4 ==== osd_op_reply(62 mds0_openfiles.0 [omap-set-header] v104'93 uv93 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7fe75c01f040 con 0x556ea16948b0 2024-08-03T20:57:27.262 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.262+0000 7fe7779edf40 1 -- 172.21.15.32:0/4119417304 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:63 12.e 12:717a0223:::608.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104) -- 0x556ea1698980 con 0x556ea15c8c70 2024-08-03T20:57:27.264 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.263+0000 7fe771600640 1 -- 172.21.15.32:0/4119417304 <== osd.0 v2:172.21.15.32:6816/3066156116 38 ==== osd_op_reply(63 608.00000000 [create] v104'111 uv110 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fe76c019040 con 0x556ea15c8c70 2024-08-03T20:57:27.264 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.263+0000 7fe7779edf40 1 -- 172.21.15.32:0/4119417304 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:64 12.e 12:717a0223:::608.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e104) -- 0x556ea1698980 con 0x556ea15c8c70 2024-08-03T20:57:27.265 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.265+0000 7fe771600640 1 -- 172.21.15.32:0/4119417304 <== osd.0 v2:172.21.15.32:6816/3066156116 39 ==== osd_op_reply(64 608.00000000 [delete] v104'112 uv112 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe76c019040 con 0x556ea15c8c70 2024-08-03T20:57:27.266 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.265+0000 7fe7779edf40 1 -- 172.21.15.32:0/4119417304 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:65 12.e 12:717a0223:::608.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104) -- 0x556ea1698980 con 0x556ea15c8c70 2024-08-03T20:57:27.267 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::608.00000000:head# 2024-08-03T20:57:27.267 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.267+0000 7fe771600640 1 -- 172.21.15.32:0/4119417304 <== osd.0 v2:172.21.15.32:6816/3066156116 40 ==== osd_op_reply(65 608.00000000 [create] v104'113 uv113 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe76c019040 con 0x556ea15c8c70 2024-08-03T20:57:27.267 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.267+0000 7fe7779edf40 1 -- 172.21.15.32:0/4119417304 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:66 12.e 12:717a0223:::608.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104) -- 0x556ea1699990 con 0x556ea15c8c70 2024-08-03T20:57:27.268 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.268+0000 7fe771600640 1 -- 172.21.15.32:0/4119417304 <== osd.0 v2:172.21.15.32:6816/3066156116 41 ==== osd_op_reply(66 608.00000000 [setxattr (30)] v104'114 uv114 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe76c019040 con 0x556ea15c8c70 2024-08-03T20:57:27.268 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.268+0000 7fe7779edf40 1 -- 172.21.15.32:0/4119417304 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:67 12.e 12:717a0223:::608.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104) -- 0x556ea1699990 con 0x556ea15c8c70 2024-08-03T20:57:27.270 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.270+0000 7fe771600640 1 -- 172.21.15.32:0/4119417304 <== osd.0 v2:172.21.15.32:6816/3066156116 42 ==== osd_op_reply(67 608.00000000 [setxattr (62)] v104'115 uv115 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe76c019040 con 0x556ea15c8c70 2024-08-03T20:57:27.270 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.270+0000 7fe7779edf40 1 -- 172.21.15.32:0/4119417304 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:68 12.e 12:717a0223:::608.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104) -- 0x556ea1698980 con 0x556ea15c8c70 2024-08-03T20:57:27.271 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.271+0000 7fe771600640 1 -- 172.21.15.32:0/4119417304 <== osd.0 v2:172.21.15.32:6816/3066156116 43 ==== osd_op_reply(68 608.00000000 [omap-set-header] v104'116 uv116 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe76c019040 con 0x556ea15c8c70 2024-08-03T20:57:27.272 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.271+0000 7fe7779edf40 1 --2- 172.21.15.32:0/4119417304 >> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] conn(0x556ea16993c0 0x556ea16997d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:27.272 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.271+0000 7fe7779edf40 1 -- 172.21.15.32:0/4119417304 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:69 12.15 12:a93a17c2:::604.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104) -- 0x556ea169bde0 con 0x556ea16993c0 2024-08-03T20:57:27.272 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.272+0000 7fe771600640 1 --2- 172.21.15.32:0/4119417304 >> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] conn(0x556ea16993c0 0x556ea16997d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:27.272 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.272+0000 7fe771600640 1 --2- 172.21.15.32:0/4119417304 >> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] conn(0x556ea16993c0 0x556ea16997d0 crc :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:57:27.274 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.274+0000 7fe771600640 1 -- 172.21.15.32:0/4119417304 <== osd.6 v2:172.21.15.118:6825/801947386 1 ==== osd_op_reply(69 604.00000000 [create] v104'205 uv198 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fe76c019040 con 0x556ea16993c0 2024-08-03T20:57:27.274 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.274+0000 7fe7779edf40 1 -- 172.21.15.32:0/4119417304 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:70 12.15 12:a93a17c2:::604.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e104) -- 0x556ea169bde0 con 0x556ea16993c0 2024-08-03T20:57:27.275 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.275+0000 7fe771600640 1 -- 172.21.15.32:0/4119417304 <== osd.6 v2:172.21.15.118:6825/801947386 2 ==== osd_op_reply(70 604.00000000 [delete] v104'206 uv206 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe76c019040 con 0x556ea16993c0 2024-08-03T20:57:27.276 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.275+0000 7fe7779edf40 1 -- 172.21.15.32:0/4119417304 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:71 12.15 12:a93a17c2:::604.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104) -- 0x556ea169bde0 con 0x556ea16993c0 2024-08-03T20:57:27.277 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::604.00000000:head# 2024-08-03T20:57:27.277 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.276+0000 7fe771600640 1 -- 172.21.15.32:0/4119417304 <== osd.6 v2:172.21.15.118:6825/801947386 3 ==== osd_op_reply(71 604.00000000 [create] v104'207 uv207 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe76c019040 con 0x556ea16993c0 2024-08-03T20:57:27.277 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.276+0000 7fe7779edf40 1 -- 172.21.15.32:0/4119417304 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:72 12.15 12:a93a17c2:::604.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104) -- 0x556ea169cdf0 con 0x556ea16993c0 2024-08-03T20:57:27.278 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.278+0000 7fe771600640 1 -- 172.21.15.32:0/4119417304 <== osd.6 v2:172.21.15.118:6825/801947386 4 ==== osd_op_reply(72 604.00000000 [setxattr (30)] v104'208 uv208 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe76c019040 con 0x556ea16993c0 2024-08-03T20:57:27.278 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.278+0000 7fe7779edf40 1 -- 172.21.15.32:0/4119417304 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:73 12.15 12:a93a17c2:::604.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104) -- 0x556ea169cdf0 con 0x556ea16993c0 2024-08-03T20:57:27.279 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.279+0000 7fe771600640 1 -- 172.21.15.32:0/4119417304 <== osd.6 v2:172.21.15.118:6825/801947386 5 ==== osd_op_reply(73 604.00000000 [setxattr (62)] v104'209 uv209 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe76c019040 con 0x556ea16993c0 2024-08-03T20:57:27.279 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.279+0000 7fe7779edf40 1 -- 172.21.15.32:0/4119417304 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:74 12.15 12:a93a17c2:::604.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104) -- 0x556ea169bde0 con 0x556ea16993c0 2024-08-03T20:57:27.281 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.280+0000 7fe771600640 1 -- 172.21.15.32:0/4119417304 <== osd.6 v2:172.21.15.118:6825/801947386 6 ==== osd_op_reply(74 604.00000000 [omap-set-header] v104'210 uv210 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe76c019040 con 0x556ea16993c0 2024-08-03T20:57:27.281 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.280+0000 7fe7779edf40 1 -- 172.21.15.32:0/4119417304 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:75 12.15 12:aa448500:::500.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104) -- 0x556ea169bde0 con 0x556ea16993c0 2024-08-03T20:57:27.282 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.282+0000 7fe771600640 1 -- 172.21.15.32:0/4119417304 <== osd.6 v2:172.21.15.118:6825/801947386 7 ==== osd_op_reply(75 500.00000000 [create] v104'211 uv204 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fe76c019040 con 0x556ea16993c0 2024-08-03T20:57:27.282 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.282+0000 7fe7779edf40 1 -- 172.21.15.32:0/4119417304 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:76 12.15 12:aa448500:::500.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e104) -- 0x556ea169bde0 con 0x556ea16993c0 2024-08-03T20:57:27.283 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.283+0000 7fe771600640 1 -- 172.21.15.32:0/4119417304 <== osd.6 v2:172.21.15.118:6825/801947386 8 ==== osd_op_reply(76 500.00000000 [delete] v104'212 uv212 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe76c019040 con 0x556ea16993c0 2024-08-03T20:57:27.284 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.283+0000 7fe7779edf40 1 -- 172.21.15.32:0/4119417304 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:77 12.15 12:aa448500:::500.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104) -- 0x556ea169bde0 con 0x556ea16993c0 2024-08-03T20:57:27.285 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::500.00000000:head# 2024-08-03T20:57:27.285 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.284+0000 7fe771600640 1 -- 172.21.15.32:0/4119417304 <== osd.6 v2:172.21.15.118:6825/801947386 9 ==== osd_op_reply(77 500.00000000 [create] v104'213 uv213 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe76c019040 con 0x556ea16993c0 2024-08-03T20:57:27.285 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.285+0000 7fe7779edf40 1 -- 172.21.15.32:0/4119417304 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:78 12.15 12:aa448500:::500.00000000:head [write 0~90 in=90b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104) -- 0x556ea169bde0 con 0x556ea16993c0 2024-08-03T20:57:27.286 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.286+0000 7fe771600640 1 -- 172.21.15.32:0/4119417304 <== osd.6 v2:172.21.15.118:6825/801947386 10 ==== osd_op_reply(78 500.00000000 [write 0~90] v104'214 uv214 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe76c019040 con 0x556ea16993c0 2024-08-03T20:57:27.286 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.286+0000 7fe7779edf40 1 -- 172.21.15.32:0/4119417304 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:79 12.15 12:aa448500:::500.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104) -- 0x556ea169bde0 con 0x556ea16993c0 2024-08-03T20:57:27.288 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.287+0000 7fe771600640 1 -- 172.21.15.32:0/4119417304 <== osd.6 v2:172.21.15.118:6825/801947386 11 ==== osd_op_reply(79 500.00000000 [omap-set-header] v104'215 uv215 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe76c019040 con 0x556ea16993c0 2024-08-03T20:57:27.288 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.287+0000 7fe7779edf40 1 --2- 172.21.15.32:0/4119417304 >> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] conn(0x556ea169c4b0 0x556ea169e8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:27.288 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.287+0000 7fe7779edf40 1 -- 172.21.15.32:0/4119417304 --> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] -- osd_op(unknown.0.0:80 12.d 12:b50e409b:::mds_snaptable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104) -- 0x556ea169f040 con 0x556ea169c4b0 2024-08-03T20:57:27.288 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.288+0000 7fe770c00640 1 --2- 172.21.15.32:0/4119417304 >> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] conn(0x556ea169c4b0 0x556ea169e8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:27.288 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.288+0000 7fe770c00640 1 --2- 172.21.15.32:0/4119417304 >> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] conn(0x556ea169c4b0 0x556ea169e8f0 crc :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:57:27.290 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.290+0000 7fe770c00640 1 -- 172.21.15.32:0/4119417304 <== osd.7 v2:172.21.15.118:6818/653000048 1 ==== osd_op_reply(80 mds_snaptable [create] v104'91 uv90 ondisk = -17 ((17) File exists)) ==== 157+0+0 (crc 0 0 0) 0x7fe7640a09c0 con 0x556ea169c4b0 2024-08-03T20:57:27.290 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.290+0000 7fe7779edf40 1 -- 172.21.15.32:0/4119417304 --> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] -- osd_op(unknown.0.0:81 12.d 12:b50e409b:::mds_snaptable:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e104) -- 0x556ea169f410 con 0x556ea169c4b0 2024-08-03T20:57:27.292 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.291+0000 7fe770c00640 1 -- 172.21.15.32:0/4119417304 <== osd.7 v2:172.21.15.118:6818/653000048 2 ==== osd_op_reply(81 mds_snaptable [delete] v104'92 uv92 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7fe7640a09c0 con 0x556ea169c4b0 2024-08-03T20:57:27.292 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.291+0000 7fe7779edf40 1 -- 172.21.15.32:0/4119417304 --> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] -- osd_op(unknown.0.0:82 12.d 12:b50e409b:::mds_snaptable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104) -- 0x556ea169f410 con 0x556ea169c4b0 2024-08-03T20:57:27.293 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.292+0000 7fe770c00640 1 -- 172.21.15.32:0/4119417304 <== osd.7 v2:172.21.15.118:6818/653000048 3 ==== osd_op_reply(82 mds_snaptable [create] v104'93 uv93 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7fe7640a09c0 con 0x556ea169c4b0 2024-08-03T20:57:27.293 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds_snaptable:head# 2024-08-03T20:57:27.293 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.293+0000 7fe7779edf40 1 -- 172.21.15.32:0/4119417304 --> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] -- osd_op(unknown.0.0:83 12.d 12:b50e409b:::mds_snaptable:head [write 0~70 in=70b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104) -- 0x556ea169f410 con 0x556ea169c4b0 2024-08-03T20:57:27.294 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.294+0000 7fe770c00640 1 -- 172.21.15.32:0/4119417304 <== osd.7 v2:172.21.15.118:6818/653000048 4 ==== osd_op_reply(83 mds_snaptable [write 0~70] v104'94 uv94 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7fe7640a09c0 con 0x556ea169c4b0 2024-08-03T20:57:27.294 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.294+0000 7fe7779edf40 1 -- 172.21.15.32:0/4119417304 --> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] -- osd_op(unknown.0.0:84 12.d 12:b50e409b:::mds_snaptable:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104) -- 0x556ea169f410 con 0x556ea169c4b0 2024-08-03T20:57:27.296 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.295+0000 7fe770c00640 1 -- 172.21.15.32:0/4119417304 <== osd.7 v2:172.21.15.118:6818/653000048 5 ==== osd_op_reply(84 mds_snaptable [omap-set-header] v104'95 uv95 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7fe7640a09c0 con 0x556ea169c4b0 2024-08-03T20:57:27.296 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.295+0000 7fe7779edf40 1 -- 172.21.15.32:0/4119417304 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:85 12.1d 12:b871830b:::605.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104) -- 0x556ea169f410 con 0x556ea15c8c70 2024-08-03T20:57:27.297 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.297+0000 7fe771600640 1 -- 172.21.15.32:0/4119417304 <== osd.0 v2:172.21.15.32:6816/3066156116 44 ==== osd_op_reply(85 605.00000000 [create] v104'111 uv110 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fe76c019040 con 0x556ea15c8c70 2024-08-03T20:57:27.297 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.297+0000 7fe7779edf40 1 -- 172.21.15.32:0/4119417304 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:86 12.1d 12:b871830b:::605.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e104) -- 0x556ea169f410 con 0x556ea15c8c70 2024-08-03T20:57:27.299 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.298+0000 7fe771600640 1 -- 172.21.15.32:0/4119417304 <== osd.0 v2:172.21.15.32:6816/3066156116 45 ==== osd_op_reply(86 605.00000000 [delete] v104'112 uv112 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe76c019040 con 0x556ea15c8c70 2024-08-03T20:57:27.299 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.298+0000 7fe7779edf40 1 -- 172.21.15.32:0/4119417304 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:87 12.1d 12:b871830b:::605.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104) -- 0x556ea169f410 con 0x556ea15c8c70 2024-08-03T20:57:27.300 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.299+0000 7fe771600640 1 -- 172.21.15.32:0/4119417304 <== osd.0 v2:172.21.15.32:6816/3066156116 46 ==== osd_op_reply(87 605.00000000 [create] v104'113 uv113 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe76c019040 con 0x556ea15c8c70 2024-08-03T20:57:27.300 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::605.00000000:head# 2024-08-03T20:57:27.300 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.299+0000 7fe7779edf40 1 -- 172.21.15.32:0/4119417304 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:88 12.1d 12:b871830b:::605.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104) -- 0x556ea16a0460 con 0x556ea15c8c70 2024-08-03T20:57:27.301 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.301+0000 7fe771600640 1 -- 172.21.15.32:0/4119417304 <== osd.0 v2:172.21.15.32:6816/3066156116 47 ==== osd_op_reply(88 605.00000000 [setxattr (30)] v104'114 uv114 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe76c019040 con 0x556ea15c8c70 2024-08-03T20:57:27.301 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.301+0000 7fe7779edf40 1 -- 172.21.15.32:0/4119417304 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:89 12.1d 12:b871830b:::605.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104) -- 0x556ea16a0420 con 0x556ea15c8c70 2024-08-03T20:57:27.302 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.302+0000 7fe771600640 1 -- 172.21.15.32:0/4119417304 <== osd.0 v2:172.21.15.32:6816/3066156116 48 ==== osd_op_reply(89 605.00000000 [setxattr (62)] v104'115 uv115 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe76c019040 con 0x556ea15c8c70 2024-08-03T20:57:27.303 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.302+0000 7fe7779edf40 1 -- 172.21.15.32:0/4119417304 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:90 12.1d 12:b871830b:::605.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104) -- 0x556ea169f410 con 0x556ea15c8c70 2024-08-03T20:57:27.304 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.303+0000 7fe771600640 1 -- 172.21.15.32:0/4119417304 <== osd.0 v2:172.21.15.32:6816/3066156116 49 ==== osd_op_reply(90 605.00000000 [omap-set-header] v104'116 uv116 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe76c019040 con 0x556ea15c8c70 2024-08-03T20:57:27.304 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.303+0000 7fe7779edf40 1 -- 172.21.15.32:0/4119417304 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:91 12.13 12:c91d4a1d:::mds0_inotable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104) -- 0x556ea169f410 con 0x556ea16948b0 2024-08-03T20:57:27.305 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.305+0000 7fe76be00640 1 -- 172.21.15.32:0/4119417304 <== osd.1 v2:172.21.15.32:6824/1551241759 5 ==== osd_op_reply(91 mds0_inotable [create] v104'182 uv176 ondisk = -17 ((17) File exists)) ==== 157+0+0 (crc 0 0 0) 0x7fe75c01f040 con 0x556ea16948b0 2024-08-03T20:57:27.305 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.305+0000 7fe7779edf40 1 -- 172.21.15.32:0/4119417304 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:92 12.13 12:c91d4a1d:::mds0_inotable:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e104) -- 0x556ea169f7b0 con 0x556ea16948b0 2024-08-03T20:57:27.306 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.306+0000 7fe76be00640 1 -- 172.21.15.32:0/4119417304 <== osd.1 v2:172.21.15.32:6824/1551241759 6 ==== osd_op_reply(92 mds0_inotable [delete] v104'183 uv183 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7fe75c01f040 con 0x556ea16948b0 2024-08-03T20:57:27.307 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.306+0000 7fe7779edf40 1 -- 172.21.15.32:0/4119417304 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:93 12.13 12:c91d4a1d:::mds0_inotable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104) -- 0x556ea169fba0 con 0x556ea16948b0 2024-08-03T20:57:27.308 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.307+0000 7fe76be00640 1 -- 172.21.15.32:0/4119417304 <== osd.1 v2:172.21.15.32:6824/1551241759 7 ==== osd_op_reply(93 mds0_inotable [create] v104'184 uv184 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7fe75c01f040 con 0x556ea16948b0 2024-08-03T20:57:27.308 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds0_inotable:head# 2024-08-03T20:57:27.308 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.307+0000 7fe7779edf40 1 -- 172.21.15.32:0/4119417304 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:94 12.13 12:c91d4a1d:::mds0_inotable:head [write 0~34 in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104) -- 0x556ea16984f0 con 0x556ea16948b0 2024-08-03T20:57:27.309 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.309+0000 7fe76be00640 1 -- 172.21.15.32:0/4119417304 <== osd.1 v2:172.21.15.32:6824/1551241759 8 ==== osd_op_reply(94 mds0_inotable [write 0~34] v104'185 uv185 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7fe75c01f040 con 0x556ea16948b0 2024-08-03T20:57:27.309 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.309+0000 7fe7779edf40 1 -- 172.21.15.32:0/4119417304 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:95 12.13 12:c91d4a1d:::mds0_inotable:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104) -- 0x556ea16984f0 con 0x556ea16948b0 2024-08-03T20:57:27.310 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.310+0000 7fe76be00640 1 -- 172.21.15.32:0/4119417304 <== osd.1 v2:172.21.15.32:6824/1551241759 9 ==== osd_op_reply(95 mds0_inotable [omap-set-header] v104'186 uv186 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7fe75c01f040 con 0x556ea16948b0 2024-08-03T20:57:27.311 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.310+0000 7fe7779edf40 1 -- 172.21.15.32:0/4119417304 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:96 12.13 12:cd5a64a3:::100.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104) -- 0x556ea16984f0 con 0x556ea16948b0 2024-08-03T20:57:27.312 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.312+0000 7fe76be00640 1 -- 172.21.15.32:0/4119417304 <== osd.1 v2:172.21.15.32:6824/1551241759 10 ==== osd_op_reply(96 100.00000000 [create] v104'187 uv181 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fe75c01f040 con 0x556ea16948b0 2024-08-03T20:57:27.312 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.312+0000 7fe7779edf40 1 -- 172.21.15.32:0/4119417304 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:97 12.13 12:cd5a64a3:::100.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e104) -- 0x556ea16a03f0 con 0x556ea16948b0 2024-08-03T20:57:27.313 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.313+0000 7fe76be00640 1 -- 172.21.15.32:0/4119417304 <== osd.1 v2:172.21.15.32:6824/1551241759 11 ==== osd_op_reply(97 100.00000000 [delete] v104'188 uv188 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe75c01f040 con 0x556ea16948b0 2024-08-03T20:57:27.313 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.313+0000 7fe7779edf40 1 -- 172.21.15.32:0/4119417304 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:98 12.13 12:cd5a64a3:::100.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104) -- 0x556ea16a03f0 con 0x556ea16948b0 2024-08-03T20:57:27.314 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.314+0000 7fe76be00640 1 -- 172.21.15.32:0/4119417304 <== osd.1 v2:172.21.15.32:6824/1551241759 12 ==== osd_op_reply(98 100.00000000 [create] v104'189 uv189 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe75c01f040 con 0x556ea16948b0 2024-08-03T20:57:27.315 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::100.00000000:head# 2024-08-03T20:57:27.315 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.314+0000 7fe7779edf40 1 -- 172.21.15.32:0/4119417304 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:99 12.13 12:cd5a64a3:::100.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104) -- 0x556ea16a03f0 con 0x556ea16948b0 2024-08-03T20:57:27.316 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.315+0000 7fe76be00640 1 -- 172.21.15.32:0/4119417304 <== osd.1 v2:172.21.15.32:6824/1551241759 13 ==== osd_op_reply(99 100.00000000 [omap-set-header] v104'190 uv190 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe75c01f040 con 0x556ea16948b0 2024-08-03T20:57:27.316 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.316+0000 7fe7779edf40 1 -- 172.21.15.32:0/4119417304 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:100 12.13 12:cd5a64a3:::100.00000000:head [omap-set-vals in=5194b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104) -- 0x556ea16a03f0 con 0x556ea16948b0 2024-08-03T20:57:27.317 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.317+0000 7fe76be00640 1 -- 172.21.15.32:0/4119417304 <== osd.1 v2:172.21.15.32:6824/1551241759 14 ==== osd_op_reply(100 100.00000000 [omap-set-vals] v104'191 uv191 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe75c01f040 con 0x556ea16948b0 2024-08-03T20:57:27.317 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.317+0000 7fe7779edf40 1 --2- 172.21.15.32:0/4119417304 >> [v2:172.21.15.32:6800/9088303,v1:172.21.15.32:6801/9088303] conn(0x556ea16a0da0 0x556ea16a34d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:27.317 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.317+0000 7fe7779edf40 1 -- 172.21.15.32:0/4119417304 --> [v2:172.21.15.32:6800/9088303,v1:172.21.15.32:6801/9088303] -- osd_op(unknown.0.0:101 12.b 12:d0630e4c:::mds0_sessionmap:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104) -- 0x556ea16a3a10 con 0x556ea16a0da0 2024-08-03T20:57:27.318 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.317+0000 7fe76be00640 1 --2- 172.21.15.32:0/4119417304 >> [v2:172.21.15.32:6800/9088303,v1:172.21.15.32:6801/9088303] conn(0x556ea16a0da0 0x556ea16a34d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:27.318 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.318+0000 7fe76be00640 1 --2- 172.21.15.32:0/4119417304 >> [v2:172.21.15.32:6800/9088303,v1:172.21.15.32:6801/9088303] conn(0x556ea16a0da0 0x556ea16a34d0 crc :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:57:27.320 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.319+0000 7fe76be00640 1 -- 172.21.15.32:0/4119417304 <== osd.3 v2:172.21.15.32:6800/9088303 1 ==== osd_op_reply(101 mds0_sessionmap [create] v104'83 uv81 ondisk = -17 ((17) File exists)) ==== 159+0+0 (crc 0 0 0) 0x7fe75c01f040 con 0x556ea16a0da0 2024-08-03T20:57:27.320 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.319+0000 7fe7779edf40 1 -- 172.21.15.32:0/4119417304 --> [v2:172.21.15.32:6800/9088303,v1:172.21.15.32:6801/9088303] -- osd_op(unknown.0.0:102 12.b 12:d0630e4c:::mds0_sessionmap:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e104) -- 0x556ea16a3a10 con 0x556ea16a0da0 2024-08-03T20:57:27.321 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.321+0000 7fe76be00640 1 -- 172.21.15.32:0/4119417304 <== osd.3 v2:172.21.15.32:6800/9088303 2 ==== osd_op_reply(102 mds0_sessionmap [delete] v104'84 uv84 ondisk = 0) ==== 159+0+0 (crc 0 0 0) 0x7fe75c01f040 con 0x556ea16a0da0 2024-08-03T20:57:27.321 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.321+0000 7fe7779edf40 1 -- 172.21.15.32:0/4119417304 --> [v2:172.21.15.32:6800/9088303,v1:172.21.15.32:6801/9088303] -- osd_op(unknown.0.0:103 12.b 12:d0630e4c:::mds0_sessionmap:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104) -- 0x556ea16a3a10 con 0x556ea16a0da0 2024-08-03T20:57:27.322 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds0_sessionmap:head# 2024-08-03T20:57:27.323 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.322+0000 7fe76be00640 1 -- 172.21.15.32:0/4119417304 <== osd.3 v2:172.21.15.32:6800/9088303 3 ==== osd_op_reply(103 mds0_sessionmap [create] v104'85 uv85 ondisk = 0) ==== 159+0+0 (crc 0 0 0) 0x7fe75c01f040 con 0x556ea16a0da0 2024-08-03T20:57:27.323 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.322+0000 7fe7779edf40 1 -- 172.21.15.32:0/4119417304 --> [v2:172.21.15.32:6800/9088303,v1:172.21.15.32:6801/9088303] -- osd_op(unknown.0.0:104 12.b 12:d0630e4c:::mds0_sessionmap:head [omap-set-header in=14b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104) -- 0x556ea16a3a10 con 0x556ea16a0da0 2024-08-03T20:57:27.324 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.323+0000 7fe76be00640 1 -- 172.21.15.32:0/4119417304 <== osd.3 v2:172.21.15.32:6800/9088303 4 ==== osd_op_reply(104 mds0_sessionmap [omap-set-header] v104'86 uv86 ondisk = 0) ==== 159+0+0 (crc 0 0 0) 0x7fe75c01f040 con 0x556ea16a0da0 2024-08-03T20:57:27.324 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.323+0000 7fe7779edf40 1 -- 172.21.15.32:0/4119417304 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:105 12.7 12:e0b41d06:::609.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104) -- 0x556ea16a3a10 con 0x556ea16948b0 2024-08-03T20:57:27.325 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.325+0000 7fe76be00640 1 -- 172.21.15.32:0/4119417304 <== osd.1 v2:172.21.15.32:6824/1551241759 15 ==== osd_op_reply(105 609.00000000 [create] v104'111 uv110 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fe75c01f040 con 0x556ea16948b0 2024-08-03T20:57:27.325 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.325+0000 7fe7779edf40 1 -- 172.21.15.32:0/4119417304 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:106 12.7 12:e0b41d06:::609.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e104) -- 0x556ea16a3a10 con 0x556ea16948b0 2024-08-03T20:57:27.326 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.326+0000 7fe76be00640 1 -- 172.21.15.32:0/4119417304 <== osd.1 v2:172.21.15.32:6824/1551241759 16 ==== osd_op_reply(106 609.00000000 [delete] v104'112 uv112 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe75c01f040 con 0x556ea16948b0 2024-08-03T20:57:27.326 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.326+0000 7fe7779edf40 1 -- 172.21.15.32:0/4119417304 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:107 12.7 12:e0b41d06:::609.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104) -- 0x556ea16a3a10 con 0x556ea16948b0 2024-08-03T20:57:27.328 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.327+0000 7fe76be00640 1 -- 172.21.15.32:0/4119417304 <== osd.1 v2:172.21.15.32:6824/1551241759 17 ==== osd_op_reply(107 609.00000000 [create] v104'113 uv113 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe75c01f040 con 0x556ea16948b0 2024-08-03T20:57:27.328 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::609.00000000:head# 2024-08-03T20:57:27.328 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.328+0000 7fe7779edf40 1 -- 172.21.15.32:0/4119417304 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:108 12.7 12:e0b41d06:::609.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104) -- 0x556ea16a4a20 con 0x556ea16948b0 2024-08-03T20:57:27.329 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.329+0000 7fe76be00640 1 -- 172.21.15.32:0/4119417304 <== osd.1 v2:172.21.15.32:6824/1551241759 18 ==== osd_op_reply(108 609.00000000 [setxattr (30)] v104'114 uv114 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe75c01f040 con 0x556ea16948b0 2024-08-03T20:57:27.329 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.329+0000 7fe7779edf40 1 -- 172.21.15.32:0/4119417304 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:109 12.7 12:e0b41d06:::609.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104) -- 0x556ea16a4a20 con 0x556ea16948b0 2024-08-03T20:57:27.330 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.330+0000 7fe76be00640 1 -- 172.21.15.32:0/4119417304 <== osd.1 v2:172.21.15.32:6824/1551241759 19 ==== osd_op_reply(109 609.00000000 [setxattr (62)] v104'115 uv115 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe75c01f040 con 0x556ea16948b0 2024-08-03T20:57:27.331 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.330+0000 7fe7779edf40 1 -- 172.21.15.32:0/4119417304 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:110 12.7 12:e0b41d06:::609.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104) -- 0x556ea16a3a10 con 0x556ea16948b0 2024-08-03T20:57:27.332 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.332+0000 7fe76be00640 1 -- 172.21.15.32:0/4119417304 <== osd.1 v2:172.21.15.32:6824/1551241759 20 ==== osd_op_reply(110 609.00000000 [omap-set-header] v104'116 uv116 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe75c01f040 con 0x556ea16948b0 2024-08-03T20:57:27.332 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.332+0000 7fe7779edf40 1 -- 172.21.15.32:0/4119417304 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:111 12.f 12:f1f69726:::400.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104) -- 0x556ea16a3a10 con 0x556ea16993c0 2024-08-03T20:57:27.334 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.333+0000 7fe771600640 1 -- 172.21.15.32:0/4119417304 <== osd.6 v2:172.21.15.118:6825/801947386 12 ==== osd_op_reply(111 400.00000000 [create] v104'94 uv93 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fe76c019040 con 0x556ea16993c0 2024-08-03T20:57:27.334 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.333+0000 7fe7779edf40 1 -- 172.21.15.32:0/4119417304 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:112 12.f 12:f1f69726:::400.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e104) -- 0x556ea16a3a10 con 0x556ea16993c0 2024-08-03T20:57:27.335 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.335+0000 7fe771600640 1 -- 172.21.15.32:0/4119417304 <== osd.6 v2:172.21.15.118:6825/801947386 13 ==== osd_op_reply(112 400.00000000 [delete] v104'95 uv95 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe76c019040 con 0x556ea16993c0 2024-08-03T20:57:27.335 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.335+0000 7fe7779edf40 1 -- 172.21.15.32:0/4119417304 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:113 12.f 12:f1f69726:::400.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104) -- 0x556ea16a3a10 con 0x556ea16993c0 2024-08-03T20:57:27.337 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.336+0000 7fe771600640 1 -- 172.21.15.32:0/4119417304 <== osd.6 v2:172.21.15.118:6825/801947386 14 ==== osd_op_reply(113 400.00000000 [create] v104'96 uv96 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe76c019040 con 0x556ea16993c0 2024-08-03T20:57:27.337 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::400.00000000:head# 2024-08-03T20:57:27.337 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.337+0000 7fe7779edf40 1 -- 172.21.15.32:0/4119417304 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:114 12.f 12:f1f69726:::400.00000000:head [write 0~22 in=22b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104) -- 0x556ea16a3a10 con 0x556ea16993c0 2024-08-03T20:57:27.339 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.338+0000 7fe771600640 1 -- 172.21.15.32:0/4119417304 <== osd.6 v2:172.21.15.118:6825/801947386 15 ==== osd_op_reply(114 400.00000000 [write 0~22] v104'97 uv97 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe76c019040 con 0x556ea16993c0 2024-08-03T20:57:27.339 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.338+0000 7fe7779edf40 1 -- 172.21.15.32:0/4119417304 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:115 12.f 12:f1f69726:::400.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104) -- 0x556ea16a3a10 con 0x556ea16993c0 2024-08-03T20:57:27.340 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.340+0000 7fe771600640 1 -- 172.21.15.32:0/4119417304 <== osd.6 v2:172.21.15.118:6825/801947386 16 ==== osd_op_reply(115 400.00000000 [omap-set-header] v104'98 uv98 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe76c019040 con 0x556ea16993c0 2024-08-03T20:57:27.340 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.340+0000 7fe7779edf40 1 -- 172.21.15.32:0/4119417304 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:116 12.1f 12:fe07bba1:::100.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104) -- 0x556ea16a3a10 con 0x556ea16993c0 2024-08-03T20:57:27.342 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.341+0000 7fe771600640 1 -- 172.21.15.32:0/4119417304 <== osd.6 v2:172.21.15.118:6825/801947386 17 ==== osd_op_reply(116 100.00000000.inode [create] v104'229 uv220 ondisk = -17 ((17) File exists)) ==== 162+0+0 (crc 0 0 0) 0x7fe76c019040 con 0x556ea16993c0 2024-08-03T20:57:27.342 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.341+0000 7fe7779edf40 1 -- 172.21.15.32:0/4119417304 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:117 12.1f 12:fe07bba1:::100.00000000.inode:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e104) -- 0x556ea16a3a10 con 0x556ea16993c0 2024-08-03T20:57:27.343 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.343+0000 7fe771600640 1 -- 172.21.15.32:0/4119417304 <== osd.6 v2:172.21.15.118:6825/801947386 18 ==== osd_op_reply(117 100.00000000.inode [delete] v104'230 uv230 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7fe76c019040 con 0x556ea16993c0 2024-08-03T20:57:27.343 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.343+0000 7fe7779edf40 1 -- 172.21.15.32:0/4119417304 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:118 12.1f 12:fe07bba1:::100.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104) -- 0x556ea16a3a10 con 0x556ea16993c0 2024-08-03T20:57:27.344 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.344+0000 7fe771600640 1 -- 172.21.15.32:0/4119417304 <== osd.6 v2:172.21.15.118:6825/801947386 19 ==== osd_op_reply(118 100.00000000.inode [create] v104'231 uv231 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7fe76c019040 con 0x556ea16993c0 2024-08-03T20:57:27.345 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::100.00000000.inode:head# 2024-08-03T20:57:27.345 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.344+0000 7fe7779edf40 1 -- 172.21.15.32:0/4119417304 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:119 12.1f 12:fe07bba1:::100.00000000.inode:head [write 0~582 in=582b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104) -- 0x556ea16a3a10 con 0x556ea16993c0 2024-08-03T20:57:27.346 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.346+0000 7fe771600640 1 -- 172.21.15.32:0/4119417304 <== osd.6 v2:172.21.15.118:6825/801947386 20 ==== osd_op_reply(119 100.00000000.inode [write 0~582] v104'232 uv232 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7fe76c019040 con 0x556ea16993c0 2024-08-03T20:57:27.346 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.346+0000 7fe7779edf40 1 -- 172.21.15.32:0/4119417304 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:120 12.1f 12:fe07bba1:::100.00000000.inode:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104) -- 0x556ea16a3a10 con 0x556ea16993c0 2024-08-03T20:57:27.347 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.347+0000 7fe771600640 1 -- 172.21.15.32:0/4119417304 <== osd.6 v2:172.21.15.118:6825/801947386 21 ==== osd_op_reply(120 100.00000000.inode [omap-set-header] v104'233 uv233 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7fe76c019040 con 0x556ea16993c0 2024-08-03T20:57:27.348 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.347+0000 7fe7779edf40 1 -- 172.21.15.32:0/4119417304 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:121 12.1f 12:ff5b34d6:::1.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104) -- 0x556ea16a3a10 con 0x556ea16993c0 2024-08-03T20:57:27.349 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.348+0000 7fe771600640 1 -- 172.21.15.32:0/4119417304 <== osd.6 v2:172.21.15.118:6825/801947386 22 ==== osd_op_reply(121 1.00000000 [create] v104'234 uv227 ondisk = -17 ((17) File exists)) ==== 154+0+0 (crc 0 0 0) 0x7fe76c019040 con 0x556ea16993c0 2024-08-03T20:57:27.349 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.349+0000 7fe7779edf40 1 -- 172.21.15.32:0/4119417304 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:122 12.1f 12:ff5b34d6:::1.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e104) -- 0x556ea16a3a10 con 0x556ea16993c0 2024-08-03T20:57:27.350 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.350+0000 7fe771600640 1 -- 172.21.15.32:0/4119417304 <== osd.6 v2:172.21.15.118:6825/801947386 23 ==== osd_op_reply(122 1.00000000 [delete] v104'235 uv235 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7fe76c019040 con 0x556ea16993c0 2024-08-03T20:57:27.350 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.350+0000 7fe7779edf40 1 -- 172.21.15.32:0/4119417304 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:123 12.1f 12:ff5b34d6:::1.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104) -- 0x556ea16a3a10 con 0x556ea16993c0 2024-08-03T20:57:27.351 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.351+0000 7fe771600640 1 -- 172.21.15.32:0/4119417304 <== osd.6 v2:172.21.15.118:6825/801947386 24 ==== osd_op_reply(123 1.00000000 [create] v104'236 uv236 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7fe76c019040 con 0x556ea16993c0 2024-08-03T20:57:27.352 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::1.00000000:head# 2024-08-03T20:57:27.352 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.351+0000 7fe7779edf40 1 -- 172.21.15.32:0/4119417304 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:124 12.1f 12:ff5b34d6:::1.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104) -- 0x556ea16a4a20 con 0x556ea16993c0 2024-08-03T20:57:27.353 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.353+0000 7fe771600640 1 -- 172.21.15.32:0/4119417304 <== osd.6 v2:172.21.15.118:6825/801947386 25 ==== osd_op_reply(124 1.00000000 [setxattr (30)] v104'237 uv237 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7fe76c019040 con 0x556ea16993c0 2024-08-03T20:57:27.353 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.353+0000 7fe7779edf40 1 -- 172.21.15.32:0/4119417304 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:125 12.1f 12:ff5b34d6:::1.00000000:head [setxattr parent (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104) -- 0x556ea16a4a20 con 0x556ea16993c0 2024-08-03T20:57:27.354 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.354+0000 7fe771600640 1 -- 172.21.15.32:0/4119417304 <== osd.6 v2:172.21.15.118:6825/801947386 26 ==== osd_op_reply(125 1.00000000 [setxattr (30)] v104'238 uv238 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7fe76c019040 con 0x556ea16993c0 2024-08-03T20:57:27.355 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.354+0000 7fe7779edf40 1 -- 172.21.15.32:0/4119417304 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:126 12.1f 12:ff5b34d6:::1.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104) -- 0x556ea16a3a10 con 0x556ea16993c0 2024-08-03T20:57:27.356 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.356+0000 7fe771600640 1 -- 172.21.15.32:0/4119417304 <== osd.6 v2:172.21.15.118:6825/801947386 27 ==== osd_op_reply(126 1.00000000 [omap-set-header] v104'239 uv239 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7fe76c019040 con 0x556ea16993c0 2024-08-03T20:57:27.356 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.356+0000 7fe7779edf40 1 -- 172.21.15.32:0/4119417304 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:127 12.1f 12:ff5b34d6:::1.00000000:head [omap-set-vals in=584b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104) -- 0x556ea16a4170 con 0x556ea16993c0 2024-08-03T20:57:27.359 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.357+0000 7fe771600640 1 -- 172.21.15.32:0/4119417304 <== osd.6 v2:172.21.15.118:6825/801947386 28 ==== osd_op_reply(127 1.00000000 [omap-set-vals] v104'240 uv240 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7fe76c019040 con 0x556ea16993c0 2024-08-03T20:57:27.359 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.358+0000 7fe7779edf40 1 -- 172.21.15.32:0/4119417304 >> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] conn(0x556ea15c8c70 msgr2=0x556ea1514ee0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:27.359 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.358+0000 7fe7779edf40 1 --2- 172.21.15.32:0/4119417304 >> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] conn(0x556ea15c8c70 0x556ea1514ee0 crc :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:27.359 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.358+0000 7fe7779edf40 1 -- 172.21.15.32:0/4119417304 >> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] conn(0x556ea16948b0 msgr2=0x556ea1694cc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:27.359 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.358+0000 7fe7779edf40 1 --2- 172.21.15.32:0/4119417304 >> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] conn(0x556ea16948b0 0x556ea1694cc0 crc :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:27.359 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.358+0000 7fe7779edf40 1 -- 172.21.15.32:0/4119417304 >> [v2:172.21.15.32:6800/9088303,v1:172.21.15.32:6801/9088303] conn(0x556ea16a0da0 msgr2=0x556ea16a34d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:27.359 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.358+0000 7fe7779edf40 1 --2- 172.21.15.32:0/4119417304 >> [v2:172.21.15.32:6800/9088303,v1:172.21.15.32:6801/9088303] conn(0x556ea16a0da0 0x556ea16a34d0 crc :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:27.359 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.358+0000 7fe7779edf40 1 -- 172.21.15.32:0/4119417304 >> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] conn(0x556ea16905d0 msgr2=0x556ea16929f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:27.360 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.358+0000 7fe7779edf40 1 --2- 172.21.15.32:0/4119417304 >> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] conn(0x556ea16905d0 0x556ea16929f0 crc :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:27.360 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.358+0000 7fe7779edf40 1 -- 172.21.15.32:0/4119417304 >> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] conn(0x556ea16993c0 msgr2=0x556ea16997d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:27.360 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.358+0000 7fe7779edf40 1 --2- 172.21.15.32:0/4119417304 >> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] conn(0x556ea16993c0 0x556ea16997d0 crc :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:27.360 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.358+0000 7fe7779edf40 1 -- 172.21.15.32:0/4119417304 >> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] conn(0x556ea169c4b0 msgr2=0x556ea169e8f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:27.360 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.358+0000 7fe7779edf40 1 --2- 172.21.15.32:0/4119417304 >> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] conn(0x556ea169c4b0 0x556ea169e8f0 crc :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:27.360 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.358+0000 7fe771600640 1 -- 172.21.15.32:0/4119417304 reap_dead start 2024-08-03T20:57:27.360 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.358+0000 7fe771600640 1 -- 172.21.15.32:0/4119417304 reap_dead start 2024-08-03T20:57:27.360 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.358+0000 7fe771600640 1 -- 172.21.15.32:0/4119417304 reap_dead start 2024-08-03T20:57:27.360 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.358+0000 7fe7779edf40 1 -- 172.21.15.32:0/4119417304 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fe7500759f0 msgr2=0x7fe750077f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:27.360 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.358+0000 7fe7779edf40 1 --2- 172.21.15.32:0/4119417304 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fe7500759f0 0x7fe750077f00 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x556ea15ca480 tx=0x7fe75c004880 comp rx=0 tx=0).stop 2024-08-03T20:57:27.360 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.358+0000 7fe7779edf40 1 -- 172.21.15.32:0/4119417304 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x556ea15cc360 msgr2=0x556ea16860b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:27.360 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.358+0000 7fe7779edf40 1 --2- 172.21.15.32:0/4119417304 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x556ea15cc360 0x556ea16860b0 secure :-1 s=READY pgs=1547 cs=0 l=1 rev1=1 crypto rx=0x7fe764025e80 tx=0x7fe76404d6e0 comp rx=0 tx=0).stop 2024-08-03T20:57:27.361 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.358+0000 7fe7779edf40 1 -- 172.21.15.32:0/4119417304 shutdown_connections 2024-08-03T20:57:27.361 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.358+0000 7fe7779edf40 1 --2- 172.21.15.32:0/4119417304 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fe7500759f0 0x7fe750077f00 unknown :-1 s=CLOSED pgs=631 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:27.361 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.358+0000 7fe7779edf40 1 --2- 172.21.15.32:0/4119417304 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x556ea15cc360 0x556ea16860b0 unknown :-1 s=CLOSED pgs=1547 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:27.361 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.358+0000 7fe7779edf40 1 -- 172.21.15.32:0/4119417304 >> 172.21.15.32:0/4119417304 conn(0x556ea1435ac0 msgr2=0x556ea15e61d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:57:27.361 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.358+0000 7fe7779edf40 1 -- 172.21.15.32:0/4119417304 shutdown_connections 2024-08-03T20:57:27.361 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.358+0000 7fe7779edf40 1 -- 172.21.15.32:0/4119417304 wait complete. 2024-08-03T20:57:27.374 DEBUG:teuthology.orchestra.run.smithi032:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata put mds_snaptable - 2024-08-03T20:57:27.517 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.516+0000 7f5196d1df40 1 Processor -- start 2024-08-03T20:57:27.517 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.516+0000 7f5196d1df40 1 -- start start 2024-08-03T20:57:27.519 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.517+0000 7f5196d1df40 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5634ae42a360 0x5634ae4422c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:27.519 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.517+0000 7f5196d1df40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5634ae441330 0x5634ae422630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:27.519 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.517+0000 7f5196d1df40 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x5634ae4483f0 0x5634ae3b3210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:27.519 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.517+0000 7f5196d1df40 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x5634ae31fbf0 con 0x5634ae42a360 2024-08-03T20:57:27.519 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.517+0000 7f5196d1df40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x5634ae32c6b0 con 0x5634ae441330 2024-08-03T20:57:27.520 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.517+0000 7f5196d1df40 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x5634ae2c1e30 con 0x5634ae4483f0 2024-08-03T20:57:27.520 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.518+0000 7f518fe00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5634ae42a360 0x5634ae4422c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:27.520 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.518+0000 7f518fe00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5634ae42a360 0x5634ae4422c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:59606/0 (socket says 172.21.15.32:59606) 2024-08-03T20:57:27.520 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.518+0000 7f518fe00640 1 -- 172.21.15.32:0/3697853238 learned_addr learned my addr 172.21.15.32:0/3697853238 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:57:27.520 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.518+0000 7f518f400640 1 --2- 172.21.15.32:0/3697853238 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5634ae441330 0x5634ae422630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:27.520 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.518+0000 7f5190800640 1 --2- 172.21.15.32:0/3697853238 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x5634ae4483f0 0x5634ae3b3210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:27.520 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.518+0000 7f518fe00640 1 -- 172.21.15.32:0/3697853238 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x5634ae4483f0 msgr2=0x5634ae3b3210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:57:27.520 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.518+0000 7f518fe00640 1 --2- 172.21.15.32:0/3697853238 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x5634ae4483f0 0x5634ae3b3210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:27.520 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.518+0000 7f518fe00640 1 -- 172.21.15.32:0/3697853238 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5634ae441330 msgr2=0x5634ae422630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:57:27.520 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.518+0000 7f518fe00640 1 --2- 172.21.15.32:0/3697853238 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5634ae441330 0x5634ae422630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:27.520 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.518+0000 7f518fe00640 1 -- 172.21.15.32:0/3697853238 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5634ae444340 con 0x5634ae42a360 2024-08-03T20:57:27.521 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.518+0000 7f5190800640 1 --2- 172.21.15.32:0/3697853238 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x5634ae4483f0 0x5634ae3b3210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-03T20:57:27.521 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.518+0000 7f518fe00640 1 --2- 172.21.15.32:0/3697853238 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5634ae42a360 0x5634ae4422c0 secure :-1 s=READY pgs=1550 cs=0 l=1 rev1=1 crypto rx=0x7f5180001e10 tx=0x7f518000b730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2d113ad2d815ab85 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:57:27.521 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.519+0000 7f518e000640 1 -- 172.21.15.32:0/3697853238 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f518000e830 con 0x5634ae42a360 2024-08-03T20:57:27.521 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.519+0000 7f518e000640 1 -- 172.21.15.32:0/3697853238 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f5180006050 con 0x5634ae42a360 2024-08-03T20:57:27.521 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.519+0000 7f518e000640 1 -- 172.21.15.32:0/3697853238 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f518000ed60 con 0x5634ae42a360 2024-08-03T20:57:27.522 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.519+0000 7f5196d1df40 1 -- 172.21.15.32:0/3697853238 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5634ae42a360 msgr2=0x5634ae4422c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:27.522 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.519+0000 7f5196d1df40 1 --2- 172.21.15.32:0/3697853238 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5634ae42a360 0x5634ae4422c0 secure :-1 s=READY pgs=1550 cs=0 l=1 rev1=1 crypto rx=0x7f5180001e10 tx=0x7f518000b730 comp rx=0 tx=0).stop 2024-08-03T20:57:27.522 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.519+0000 7f5196d1df40 1 -- 172.21.15.32:0/3697853238 shutdown_connections 2024-08-03T20:57:27.522 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.519+0000 7f5196d1df40 1 --2- 172.21.15.32:0/3697853238 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5634ae441330 0x5634ae422630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:27.522 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.519+0000 7f5196d1df40 1 --2- 172.21.15.32:0/3697853238 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x5634ae4483f0 0x5634ae3b3210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:27.522 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.519+0000 7f5196d1df40 1 --2- 172.21.15.32:0/3697853238 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5634ae42a360 0x5634ae4422c0 unknown :-1 s=CLOSED pgs=1550 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:27.522 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.519+0000 7f5196d1df40 1 -- 172.21.15.32:0/3697853238 >> 172.21.15.32:0/3697853238 conn(0x5634ae293ac0 msgr2=0x5634ae4418d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:57:27.523 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.519+0000 7f5196d1df40 1 -- 172.21.15.32:0/3697853238 shutdown_connections 2024-08-03T20:57:27.523 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.519+0000 7f5196d1df40 1 -- 172.21.15.32:0/3697853238 wait complete. 2024-08-03T20:57:27.523 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.520+0000 7f5196d1df40 1 Processor -- start 2024-08-03T20:57:27.523 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.521+0000 7f5196d1df40 1 -- start start 2024-08-03T20:57:27.525 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.522+0000 7f5196d1df40 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5634ae42a360 0x5634ae421b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:27.525 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.522+0000 7f5196d1df40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5634ae441330 0x5634ae422630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:27.525 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.522+0000 7f5196d1df40 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x5634ae4483f0 0x5634ae3bb4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:27.525 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.522+0000 7f5196d1df40 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x5634ae4d10c0 con 0x5634ae42a360 2024-08-03T20:57:27.525 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.522+0000 7f5196d1df40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x5634ae423190 con 0x5634ae441330 2024-08-03T20:57:27.525 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.522+0000 7f5196d1df40 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x5634ae424d90 con 0x5634ae4483f0 2024-08-03T20:57:27.525 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.522+0000 7f518fe00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5634ae42a360 0x5634ae421b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:27.525 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.522+0000 7f518fe00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5634ae42a360 0x5634ae421b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:59608/0 (socket says 172.21.15.32:59608) 2024-08-03T20:57:27.525 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.522+0000 7f518fe00640 1 -- 172.21.15.32:0/1509180203 learned_addr learned my addr 172.21.15.32:0/1509180203 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:57:27.525 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.522+0000 7f518f400640 1 --2- 172.21.15.32:0/1509180203 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5634ae441330 0x5634ae422630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:27.525 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.522+0000 7f5190800640 1 --2- 172.21.15.32:0/1509180203 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x5634ae4483f0 0x5634ae3bb4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:27.525 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.523+0000 7f518fe00640 1 -- 172.21.15.32:0/1509180203 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x5634ae4483f0 msgr2=0x5634ae3bb4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:27.526 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.523+0000 7f518fe00640 1 --2- 172.21.15.32:0/1509180203 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x5634ae4483f0 0x5634ae3bb4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:27.526 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.523+0000 7f518fe00640 1 -- 172.21.15.32:0/1509180203 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5634ae441330 msgr2=0x5634ae422630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:27.526 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.523+0000 7f518fe00640 1 --2- 172.21.15.32:0/1509180203 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5634ae441330 0x5634ae422630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:27.526 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.523+0000 7f518fe00640 1 -- 172.21.15.32:0/1509180203 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5634ae3b2600 con 0x5634ae42a360 2024-08-03T20:57:27.526 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.523+0000 7f518fe00640 1 --2- 172.21.15.32:0/1509180203 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5634ae42a360 0x5634ae421b50 secure :-1 s=READY pgs=1551 cs=0 l=1 rev1=1 crypto rx=0x7f5180001bf0 tx=0x7f518000fad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:57:27.526 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.523+0000 7f518cc00640 1 -- 172.21.15.32:0/1509180203 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5180025070 con 0x5634ae42a360 2024-08-03T20:57:27.526 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.523+0000 7f518cc00640 1 -- 172.21.15.32:0/1509180203 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f5180020410 con 0x5634ae42a360 2024-08-03T20:57:27.526 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.523+0000 7f5196d1df40 1 -- 172.21.15.32:0/1509180203 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x5634ae373960 con 0x5634ae42a360 2024-08-03T20:57:27.526 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.523+0000 7f518cc00640 1 -- 172.21.15.32:0/1509180203 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5180020860 con 0x5634ae42a360 2024-08-03T20:57:27.526 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.523+0000 7f5196d1df40 1 -- 172.21.15.32:0/1509180203 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5634ae424fe0 con 0x5634ae42a360 2024-08-03T20:57:27.527 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.525+0000 7f518cc00640 1 -- 172.21.15.32:0/1509180203 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f51800163f0 con 0x5634ae42a360 2024-08-03T20:57:27.528 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.526+0000 7f518cc00640 1 --2- 172.21.15.32:0/1509180203 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f51700759f0 0x7f5170077f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:27.528 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.526+0000 7f518cc00640 1 -- 172.21.15.32:0/1509180203 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(104..104 src has 1..104) ==== 7488+0+0 (secure 0 0 0) 0x7f5180098b70 con 0x5634ae42a360 2024-08-03T20:57:27.528 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.527+0000 7f5196d1df40 1 --2- 172.21.15.32:0/1509180203 >> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] conn(0x5634ae3bb9f0 0x5634ae4d0950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:27.528 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.527+0000 7f5196d1df40 1 -- 172.21.15.32:0/1509180203 --> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] -- osd_op(unknown.0.0:1 12.d 12:b50e409b:::mds_snaptable:head [writefull 0~80 in=80b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104) -- 0x5634ae3b3850 con 0x5634ae3bb9f0 2024-08-03T20:57:27.528 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.527+0000 7f518f400640 1 --2- 172.21.15.32:0/1509180203 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f51700759f0 0x7f5170077f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:27.528 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.527+0000 7f5190800640 1 --2- 172.21.15.32:0/1509180203 >> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] conn(0x5634ae3bb9f0 0x5634ae4d0950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:27.528 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.528+0000 7f518f400640 1 --2- 172.21.15.32:0/1509180203 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f51700759f0 0x7f5170077f00 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x5634ae448c50 tx=0x7f517c003830 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:57:27.529 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.528+0000 7f5190800640 1 --2- 172.21.15.32:0/1509180203 >> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] conn(0x5634ae3bb9f0 0x5634ae4d0950 crc :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:57:27.531 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.530+0000 7f5190800640 1 -- 172.21.15.32:0/1509180203 <== osd.7 v2:172.21.15.118:6818/653000048 1 ==== osd_op_reply(1 mds_snaptable [writefull 0~80] v104'96 uv96 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f518800b040 con 0x5634ae3bb9f0 2024-08-03T20:57:27.531 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.530+0000 7f5196d1df40 1 -- 172.21.15.32:0/1509180203 >> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] conn(0x5634ae3bb9f0 msgr2=0x5634ae4d0950 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:27.531 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.530+0000 7f5196d1df40 1 --2- 172.21.15.32:0/1509180203 >> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] conn(0x5634ae3bb9f0 0x5634ae4d0950 crc :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:27.531 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.530+0000 7f5196d1df40 1 -- 172.21.15.32:0/1509180203 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f51700759f0 msgr2=0x7f5170077f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:27.531 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.530+0000 7f5196d1df40 1 --2- 172.21.15.32:0/1509180203 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f51700759f0 0x7f5170077f00 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x5634ae448c50 tx=0x7f517c003830 comp rx=0 tx=0).stop 2024-08-03T20:57:27.531 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.530+0000 7f5196d1df40 1 -- 172.21.15.32:0/1509180203 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5634ae42a360 msgr2=0x5634ae421b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:27.531 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.530+0000 7f5196d1df40 1 --2- 172.21.15.32:0/1509180203 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5634ae42a360 0x5634ae421b50 secure :-1 s=READY pgs=1551 cs=0 l=1 rev1=1 crypto rx=0x7f5180001bf0 tx=0x7f518000fad0 comp rx=0 tx=0).stop 2024-08-03T20:57:27.531 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.530+0000 7f5190800640 1 -- 172.21.15.32:0/1509180203 reap_dead start 2024-08-03T20:57:27.531 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.530+0000 7f5196d1df40 1 -- 172.21.15.32:0/1509180203 shutdown_connections 2024-08-03T20:57:27.531 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.530+0000 7f5196d1df40 1 -- 172.21.15.32:0/1509180203 >> 172.21.15.32:0/1509180203 conn(0x5634ae293ac0 msgr2=0x5634ae4418d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:57:27.531 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.530+0000 7f5196d1df40 1 -- 172.21.15.32:0/1509180203 shutdown_connections 2024-08-03T20:57:27.532 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.530+0000 7f5196d1df40 1 -- 172.21.15.32:0/1509180203 wait complete. 2024-08-03T20:57:27.542 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs joinable true 2024-08-03T20:57:27.774 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.772+0000 7f10e3e00640 1 -- 172.21.15.32:0/2512869402 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f10e4159ff0 msgr2=0x7f10e415fe10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:27.774 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.772+0000 7f10e3e00640 1 --2- 172.21.15.32:0/2512869402 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f10e4159ff0 0x7f10e415fe10 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f10cc009860 tx=0x7f10cc02a5a0 comp rx=0 tx=0).stop 2024-08-03T20:57:27.774 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.773+0000 7f10e3e00640 1 -- 172.21.15.32:0/2512869402 shutdown_connections 2024-08-03T20:57:27.774 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.773+0000 7f10e3e00640 1 --2- 172.21.15.32:0/2512869402 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f10e4159ff0 0x7f10e415fe10 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:27.774 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.773+0000 7f10e3e00640 1 -- 172.21.15.32:0/2512869402 >> 172.21.15.32:0/2512869402 conn(0x7f10e40cfd80 msgr2=0x7f10e40d0190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:57:27.774 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.773+0000 7f10e3e00640 1 -- 172.21.15.32:0/2512869402 shutdown_connections 2024-08-03T20:57:27.774 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.773+0000 7f10e3e00640 1 -- 172.21.15.32:0/2512869402 wait complete. 2024-08-03T20:57:27.775 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.775+0000 7f10e3e00640 1 Processor -- start 2024-08-03T20:57:27.776 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.775+0000 7f10e3e00640 1 -- start start 2024-08-03T20:57:27.779 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.777+0000 7f10e3e00640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f10e4159ff0 0x7f10e415c7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:27.779 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.777+0000 7f10e3e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f10e4160350 0x7f10e415b000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:27.779 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.777+0000 7f10e3e00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f10e41609d0 0x7f10e415b540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:27.779 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.777+0000 7f10e3e00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f10e406f0d0 con 0x7f10e4160350 2024-08-03T20:57:27.779 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.777+0000 7f10e3e00640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f10e406ef50 con 0x7f10e41609d0 2024-08-03T20:57:27.779 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.777+0000 7f10e3e00640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7f10e406f250 con 0x7f10e4159ff0 2024-08-03T20:57:27.780 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.777+0000 7f10da000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f10e4160350 0x7f10e415b000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:27.780 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.777+0000 7f10da000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f10e4160350 0x7f10e415b000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:59618/0 (socket says 172.21.15.32:59618) 2024-08-03T20:57:27.780 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.777+0000 7f10da000640 1 -- 172.21.15.32:0/1714058403 learned_addr learned my addr 172.21.15.32:0/1714058403 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:57:27.780 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.777+0000 7f10e3400640 1 --2- 172.21.15.32:0/1714058403 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f10e41609d0 0x7f10e415b540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:27.780 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.777+0000 7f10e2a00640 1 --2- 172.21.15.32:0/1714058403 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f10e4159ff0 0x7f10e415c7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:27.780 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.778+0000 7f10da000640 1 -- 172.21.15.32:0/1714058403 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f10e4159ff0 msgr2=0x7f10e415c7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:27.780 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.778+0000 7f10da000640 1 --2- 172.21.15.32:0/1714058403 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f10e4159ff0 0x7f10e415c7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:27.780 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.778+0000 7f10da000640 1 -- 172.21.15.32:0/1714058403 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f10e41609d0 msgr2=0x7f10e415b540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:27.781 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.778+0000 7f10da000640 1 --2- 172.21.15.32:0/1714058403 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f10e41609d0 0x7f10e415b540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:27.781 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.778+0000 7f10da000640 1 -- 172.21.15.32:0/1714058403 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f10e415f2c0 con 0x7f10e4160350 2024-08-03T20:57:27.781 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.778+0000 7f10da000640 1 --2- 172.21.15.32:0/1714058403 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f10e4160350 0x7f10e415b000 secure :-1 s=READY pgs=1553 cs=0 l=1 rev1=1 crypto rx=0x7f10cc02ab90 tx=0x7f10cc033c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:57:27.781 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.778+0000 7f10db400640 1 -- 172.21.15.32:0/1714058403 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10cc005c90 con 0x7f10e4160350 2024-08-03T20:57:27.781 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.778+0000 7f10db400640 1 -- 172.21.15.32:0/1714058403 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f10cc031e50 con 0x7f10e4160350 2024-08-03T20:57:27.781 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.778+0000 7f10db400640 1 -- 172.21.15.32:0/1714058403 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10cc005260 con 0x7f10e4160350 2024-08-03T20:57:27.781 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.778+0000 7f10e3e00640 1 -- 172.21.15.32:0/1714058403 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f10e415d360 con 0x7f10e4160350 2024-08-03T20:57:27.781 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.778+0000 7f10e3e00640 1 -- 172.21.15.32:0/1714058403 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f10e415e7e0 con 0x7f10e4160350 2024-08-03T20:57:27.782 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.779+0000 7f10e3e00640 1 -- 172.21.15.32:0/1714058403 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f10e415fe10 con 0x7f10e4160350 2024-08-03T20:57:27.782 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.781+0000 7f10db400640 1 -- 172.21.15.32:0/1714058403 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f10cc005400 con 0x7f10e4160350 2024-08-03T20:57:27.782 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.781+0000 7f10db400640 1 --2- 172.21.15.32:0/1714058403 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f10c0077010 0x7f10c0079520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:27.782 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.782+0000 7f10db400640 1 -- 172.21.15.32:0/1714058403 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(104..104 src has 1..104) ==== 7488+0+0 (secure 0 0 0) 0x7f10cc0b9f50 con 0x7f10e4160350 2024-08-03T20:57:27.783 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.782+0000 7f10e2a00640 1 --2- 172.21.15.32:0/1714058403 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f10c0077010 0x7f10c0079520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:27.783 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.783+0000 7f10e2a00640 1 --2- 172.21.15.32:0/1714058403 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f10c0077010 0x7f10c0079520 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7f10d4001e90 tx=0x7f10d4001e20 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:57:27.786 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:27.786+0000 7f10db400640 1 -- 172.21.15.32:0/1714058403 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7f10cc03f030 con 0x7f10e4160350 2024-08-03T20:57:28.037 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:28.036+0000 7f10e3e00640 1 -- 172.21.15.32:0/1714058403 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "joinable", "val": "true"} v 0) -- 0x7f10e40d09e0 con 0x7f10e4160350 2024-08-03T20:57:28.342 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:28.342+0000 7f10db400640 1 -- 172.21.15.32:0/1714058403 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "joinable", "val": "true"}]=0 cephfs marked joinable; MDS may join as newly active. v209) ==== 162+0+0 (secure 0 0 0) 0x7f10cc082420 con 0x7f10e4160350 2024-08-03T20:57:28.343 INFO:teuthology.orchestra.run.smithi032.stderr:cephfs marked joinable; MDS may join as newly active. 2024-08-03T20:57:28.346 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:28.345+0000 7f10e3e00640 1 -- 172.21.15.32:0/1714058403 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f10c0077010 msgr2=0x7f10c0079520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:28.346 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:28.345+0000 7f10e3e00640 1 --2- 172.21.15.32:0/1714058403 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f10c0077010 0x7f10c0079520 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7f10d4001e90 tx=0x7f10d4001e20 comp rx=0 tx=0).stop 2024-08-03T20:57:28.346 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:28.345+0000 7f10e3e00640 1 -- 172.21.15.32:0/1714058403 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f10e4160350 msgr2=0x7f10e415b000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:28.347 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:28.345+0000 7f10e3e00640 1 --2- 172.21.15.32:0/1714058403 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f10e4160350 0x7f10e415b000 secure :-1 s=READY pgs=1553 cs=0 l=1 rev1=1 crypto rx=0x7f10cc02ab90 tx=0x7f10cc033c20 comp rx=0 tx=0).stop 2024-08-03T20:57:28.347 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:28.345+0000 7f10e3e00640 1 -- 172.21.15.32:0/1714058403 shutdown_connections 2024-08-03T20:57:28.347 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:28.346+0000 7f10e3e00640 1 --2- 172.21.15.32:0/1714058403 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f10c0077010 0x7f10c0079520 unknown :-1 s=CLOSED pgs=633 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:28.347 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:28.346+0000 7f10e3e00640 1 --2- 172.21.15.32:0/1714058403 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f10e41609d0 0x7f10e415b540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:28.347 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:28.346+0000 7f10e3e00640 1 --2- 172.21.15.32:0/1714058403 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f10e4160350 0x7f10e415b000 unknown :-1 s=CLOSED pgs=1553 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:28.347 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:28.346+0000 7f10e3e00640 1 --2- 172.21.15.32:0/1714058403 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f10e4159ff0 0x7f10e415c7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:28.347 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:28.346+0000 7f10e3e00640 1 -- 172.21.15.32:0/1714058403 >> 172.21.15.32:0/1714058403 conn(0x7f10e40cfd80 msgr2=0x7f10e4156e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:57:28.347 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:28.346+0000 7f10e3e00640 1 -- 172.21.15.32:0/1714058403 shutdown_connections 2024-08-03T20:57:28.347 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:28.346+0000 7f10e3e00640 1 -- 172.21.15.32:0/1714058403 wait complete. 2024-08-03T20:57:28.354 INFO:tasks.ceph.mds.a.smithi032.stderr:2024-08-03T20:57:28.353+0000 7f489b600640 -1 log_channel(cluster) log [ERR] : error decoding table object 'mds_snaptable': Decoder at 'virtual void SnapServer::decode_server_state(ceph::buffer::v15_2_0::list::const_iterator&)' v=5 cannot decode v=100 minimal_decoder=101: Malformed input [buffer:3] 2024-08-03T20:57:28.354 INFO:tasks.ceph.mon.a.smithi032.stderr:2024-08-03T20:57:28.354+0000 7f4e6d600640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-08-03T20:57:28.355 INFO:tasks.ceph.mon.a.smithi032.stderr:2024-08-03T20:57:28.354+0000 7f4e6d600640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 mds daemon damaged (MDS_DAMAGE) 2024-08-03T20:57:28.371 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-08-03T20:57:28.600 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:28.599+0000 7efffa200640 1 -- 172.21.15.32:0/3356256892 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efff40b38b0 msgr2=0x7efff41523f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:28.601 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:28.599+0000 7efffa200640 1 --2- 172.21.15.32:0/3356256892 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efff40b38b0 0x7efff41523f0 secure :-1 s=READY pgs=1555 cs=0 l=1 rev1=1 crypto rx=0x7effe4009830 tx=0x7effe402a510 comp rx=0 tx=0).stop 2024-08-03T20:57:28.601 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:28.600+0000 7efffa200640 1 -- 172.21.15.32:0/3356256892 shutdown_connections 2024-08-03T20:57:28.601 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:28.600+0000 7efffa200640 1 --2- 172.21.15.32:0/3356256892 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7efff40b4ce0 0x7efff415ef00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:28.601 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:28.600+0000 7efffa200640 1 --2- 172.21.15.32:0/3356256892 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efff40b38b0 0x7efff41523f0 unknown :-1 s=CLOSED pgs=1555 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:28.601 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:28.600+0000 7efffa200640 1 -- 172.21.15.32:0/3356256892 >> 172.21.15.32:0/3356256892 conn(0x7efff40b1bd0 msgr2=0x7efff40b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:57:28.601 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:28.600+0000 7efffa200640 1 -- 172.21.15.32:0/3356256892 shutdown_connections 2024-08-03T20:57:28.601 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:28.600+0000 7efffa200640 1 -- 172.21.15.32:0/3356256892 wait complete. 2024-08-03T20:57:28.602 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:28.602+0000 7efffa200640 1 Processor -- start 2024-08-03T20:57:28.602 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:28.602+0000 7efffa200640 1 -- start start 2024-08-03T20:57:28.606 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:28.604+0000 7efffa200640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7efff40b38b0 0x7efff40cb940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:28.606 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:28.604+0000 7efffa200640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7efff40b4ce0 0x7efff4144010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:28.606 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:28.604+0000 7efffa200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efff415f440 0x7efff4144550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:28.606 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:28.604+0000 7efffa200640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7efff415c020 con 0x7efff415f440 2024-08-03T20:57:28.606 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:28.604+0000 7efffa200640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7efff40cf240 con 0x7efff40b38b0 2024-08-03T20:57:28.606 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:28.604+0000 7efffa200640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7efff415c1a0 con 0x7efff40b4ce0 2024-08-03T20:57:28.606 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:28.604+0000 7efff3e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efff415f440 0x7efff4144550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:28.606 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:28.604+0000 7efff3e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efff415f440 0x7efff4144550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:59630/0 (socket says 172.21.15.32:59630) 2024-08-03T20:57:28.606 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:28.604+0000 7efff3e00640 1 -- 172.21.15.32:0/2633973625 learned_addr learned my addr 172.21.15.32:0/2633973625 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:57:28.606 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:28.604+0000 7efff3400640 1 --2- 172.21.15.32:0/2633973625 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7efff40b38b0 0x7efff40cb940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:28.607 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:28.604+0000 7effebe00640 1 --2- 172.21.15.32:0/2633973625 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7efff40b4ce0 0x7efff4144010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:28.607 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:28.604+0000 7efff3e00640 1 -- 172.21.15.32:0/2633973625 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7efff40b4ce0 msgr2=0x7efff4144010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:28.607 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:28.604+0000 7efff3e00640 1 --2- 172.21.15.32:0/2633973625 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7efff40b4ce0 0x7efff4144010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:28.607 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:28.604+0000 7efff3e00640 1 -- 172.21.15.32:0/2633973625 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7efff40b38b0 msgr2=0x7efff40cb940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:28.607 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:28.604+0000 7efff3e00640 1 --2- 172.21.15.32:0/2633973625 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7efff40b38b0 0x7efff40cb940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:28.607 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:28.604+0000 7efff3e00640 1 -- 172.21.15.32:0/2633973625 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efff4158730 con 0x7efff415f440 2024-08-03T20:57:28.607 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:28.604+0000 7efff3400640 1 --2- 172.21.15.32:0/2633973625 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7efff40b38b0 0x7efff40cb940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-03T20:57:28.607 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:28.605+0000 7efff3e00640 1 --2- 172.21.15.32:0/2633973625 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efff415f440 0x7efff4144550 secure :-1 s=READY pgs=1556 cs=0 l=1 rev1=1 crypto rx=0x7effec00c760 tx=0x7effec00cb60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:57:28.607 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:28.605+0000 7effeb400640 1 -- 172.21.15.32:0/2633973625 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7effec003b70 con 0x7efff415f440 2024-08-03T20:57:28.608 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:28.605+0000 7effeb400640 1 -- 172.21.15.32:0/2633973625 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7effec003d10 con 0x7efff415f440 2024-08-03T20:57:28.608 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:28.605+0000 7effeb400640 1 -- 172.21.15.32:0/2633973625 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7effec0052d0 con 0x7efff415f440 2024-08-03T20:57:28.608 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:28.605+0000 7efffa200640 1 -- 172.21.15.32:0/2633973625 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efff4158a10 con 0x7efff415f440 2024-08-03T20:57:28.608 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:28.605+0000 7efffa200640 1 -- 172.21.15.32:0/2633973625 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efff4143810 con 0x7efff415f440 2024-08-03T20:57:28.608 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:28.606+0000 7efffa200640 1 -- 172.21.15.32:0/2633973625 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efff41584a0 con 0x7efff415f440 2024-08-03T20:57:28.608 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:28.607+0000 7effeb400640 1 -- 172.21.15.32:0/2633973625 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7effec005470 con 0x7efff415f440 2024-08-03T20:57:28.609 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:28.609+0000 7effeb400640 1 --2- 172.21.15.32:0/2633973625 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7effd0077010 0x7effd0079520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:28.609 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:28.609+0000 7effeb400640 1 -- 172.21.15.32:0/2633973625 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(105..105 src has 1..105) ==== 7574+0+0 (secure 0 0 0) 0x7effec098f80 con 0x7efff415f440 2024-08-03T20:57:28.610 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:28.609+0000 7efff3400640 1 --2- 172.21.15.32:0/2633973625 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7effd0077010 0x7effd0079520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:28.610 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:28.610+0000 7efff3400640 1 --2- 172.21.15.32:0/2633973625 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7effd0077010 0x7effd0079520 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7effe4034000 tx=0x7effe4004730 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:57:28.614 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:28.614+0000 7effeb400640 1 -- 172.21.15.32:0/2633973625 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7efff415f850 con 0x7efff415f440 2024-08-03T20:57:28.859 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:28.857+0000 7efffa200640 1 -- 172.21.15.32:0/2633973625 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7efff415c650 con 0x7efff415f440 2024-08-03T20:57:28.859 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:28.858+0000 7effeb400640 1 -- 172.21.15.32:0/2633973625 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 211 v211) ==== 95+0+4391 (secure 0 0 0) 0x7effec061990 con 0x7efff415f440 2024-08-03T20:57:28.860 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-03T20:57:28.860 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":211,"btime":"2024-08-03T20:57:28:354846+0000","default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6439,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6835/531080160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6834","nonce":531080160},{"type":"v1","addr":"172.21.15.118:6835","nonce":531080160}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":84},{"gid":6454,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6837/2796882319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6836","nonce":2796882319},{"type":"v1","addr":"172.21.15.118:6837","nonce":2796882319}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":86},{"gid":6813,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6834/2686933890","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":2686933890},{"type":"v1","addr":"172.21.15.32:6834","nonce":2686933890}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":85}],"filesystems":[{"mdsmap":{"epoch":211,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-08-03T20:54:26.950518+0000","modified":"2024-08-03T20:57:28.354785+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":105,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":6}]} 2024-08-03T20:57:28.860 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 211 2024-08-03T20:57:28.862 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:28.860+0000 7efffa200640 1 -- 172.21.15.32:0/2633973625 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7effd0077010 msgr2=0x7effd0079520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:28.862 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:28.860+0000 7efffa200640 1 --2- 172.21.15.32:0/2633973625 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7effd0077010 0x7effd0079520 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7effe4034000 tx=0x7effe4004730 comp rx=0 tx=0).stop 2024-08-03T20:57:28.862 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:28.860+0000 7efffa200640 1 -- 172.21.15.32:0/2633973625 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efff415f440 msgr2=0x7efff4144550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:28.862 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:28.860+0000 7efffa200640 1 --2- 172.21.15.32:0/2633973625 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efff415f440 0x7efff4144550 secure :-1 s=READY pgs=1556 cs=0 l=1 rev1=1 crypto rx=0x7effec00c760 tx=0x7effec00cb60 comp rx=0 tx=0).stop 2024-08-03T20:57:28.862 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:28.860+0000 7efffa200640 1 -- 172.21.15.32:0/2633973625 shutdown_connections 2024-08-03T20:57:28.862 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:28.860+0000 7efffa200640 1 --2- 172.21.15.32:0/2633973625 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7effd0077010 0x7effd0079520 unknown :-1 s=CLOSED pgs=634 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:28.862 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:28.860+0000 7efffa200640 1 --2- 172.21.15.32:0/2633973625 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efff415f440 0x7efff4144550 unknown :-1 s=CLOSED pgs=1556 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:28.863 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:28.860+0000 7efffa200640 1 --2- 172.21.15.32:0/2633973625 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7efff40b4ce0 0x7efff4144010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:28.863 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:28.860+0000 7efffa200640 1 --2- 172.21.15.32:0/2633973625 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7efff40b38b0 0x7efff40cb940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:28.863 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:28.860+0000 7efffa200640 1 -- 172.21.15.32:0/2633973625 >> 172.21.15.32:0/2633973625 conn(0x7efff40b1bd0 msgr2=0x7efff415b910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:57:28.863 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:28.860+0000 7efffa200640 1 -- 172.21.15.32:0/2633973625 shutdown_connections 2024-08-03T20:57:28.863 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:28.860+0000 7efffa200640 1 -- 172.21.15.32:0/2633973625 wait complete. 2024-08-03T20:57:28.884 DEBUG:tasks.ceph_test_case:wait_until_true: success in 0s and 0 retries 2024-08-03T20:57:28.884 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-08-03T20:57:29.112 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:29.110+0000 7f7216000640 1 -- 172.21.15.32:0/3983578315 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f72100d29a0 msgr2=0x7f72100d1c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:29.112 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:29.110+0000 7f7216000640 1 --2- 172.21.15.32:0/3983578315 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f72100d29a0 0x7f72100d1c60 secure :-1 s=READY pgs=1558 cs=0 l=1 rev1=1 crypto rx=0x7f71fc009830 tx=0x7f71fc02a480 comp rx=0 tx=0).stop 2024-08-03T20:57:29.112 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:29.110+0000 7f7216000640 1 -- 172.21.15.32:0/3983578315 shutdown_connections 2024-08-03T20:57:29.112 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:29.110+0000 7f7216000640 1 --2- 172.21.15.32:0/3983578315 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f72100cec30 0x7f72100cf040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:29.113 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:29.110+0000 7f7216000640 1 --2- 172.21.15.32:0/3983578315 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f72100d29a0 0x7f72100d1c60 unknown :-1 s=CLOSED pgs=1558 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:29.113 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:29.110+0000 7f7216000640 1 -- 172.21.15.32:0/3983578315 >> 172.21.15.32:0/3983578315 conn(0x7f72100d4090 msgr2=0x7f72100d44a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:57:29.113 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:29.110+0000 7f7216000640 1 -- 172.21.15.32:0/3983578315 shutdown_connections 2024-08-03T20:57:29.113 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:29.110+0000 7f7216000640 1 -- 172.21.15.32:0/3983578315 wait complete. 2024-08-03T20:57:29.113 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:29.112+0000 7f7216000640 1 Processor -- start 2024-08-03T20:57:29.114 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:29.112+0000 7f7216000640 1 -- start start 2024-08-03T20:57:29.117 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:29.114+0000 7f7216000640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f72100cec30 0x7f72101522e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:29.117 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:29.114+0000 7f7216000640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f72100d4bb0 0x7f72100cde00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:29.117 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:29.114+0000 7f7216000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f72100ce340 0x7f72100b84b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:29.117 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:29.114+0000 7f7216000640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f72100cfad0 con 0x7f72100ce340 2024-08-03T20:57:29.117 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:29.114+0000 7f7216000640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f72100cf950 con 0x7f72100d4bb0 2024-08-03T20:57:29.117 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:29.114+0000 7f7216000640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7f72100cfc50 con 0x7f72100cec30 2024-08-03T20:57:29.117 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:29.114+0000 7f7215600640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f72100ce340 0x7f72100b84b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:29.117 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:29.114+0000 7f7215600640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f72100ce340 0x7f72100b84b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:59654/0 (socket says 172.21.15.32:59654) 2024-08-03T20:57:29.117 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:29.114+0000 7f7215600640 1 -- 172.21.15.32:0/4140147903 learned_addr learned my addr 172.21.15.32:0/4140147903 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:57:29.117 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:29.114+0000 7f7214c00640 1 --2- 172.21.15.32:0/4140147903 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f72100cec30 0x7f72101522e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:29.117 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:29.115+0000 7f720fe00640 1 --2- 172.21.15.32:0/4140147903 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f72100d4bb0 0x7f72100cde00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:29.117 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:29.115+0000 7f7215600640 1 -- 172.21.15.32:0/4140147903 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f72100cec30 msgr2=0x7f72101522e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:29.118 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:29.115+0000 7f7215600640 1 --2- 172.21.15.32:0/4140147903 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f72100cec30 0x7f72101522e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:29.118 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:29.115+0000 7f7215600640 1 -- 172.21.15.32:0/4140147903 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f72100d4bb0 msgr2=0x7f72100cde00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:29.118 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:29.115+0000 7f7215600640 1 --2- 172.21.15.32:0/4140147903 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f72100d4bb0 0x7f72100cde00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:29.118 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:29.115+0000 7f7215600640 1 -- 172.21.15.32:0/4140147903 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7210161be0 con 0x7f72100ce340 2024-08-03T20:57:29.118 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:29.115+0000 7f720fe00640 1 --2- 172.21.15.32:0/4140147903 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f72100d4bb0 0x7f72100cde00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-03T20:57:29.118 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:29.115+0000 7f7215600640 1 --2- 172.21.15.32:0/4140147903 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f72100ce340 0x7f72100b84b0 secure :-1 s=READY pgs=1559 cs=0 l=1 rev1=1 crypto rx=0x7f720800ea80 tx=0x7f720800ef40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:57:29.118 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:29.115+0000 7f720cc00640 1 -- 172.21.15.32:0/4140147903 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f720800ce40 con 0x7f72100ce340 2024-08-03T20:57:29.118 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:29.115+0000 7f720cc00640 1 -- 172.21.15.32:0/4140147903 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f7208004b20 con 0x7f72100ce340 2024-08-03T20:57:29.118 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:29.115+0000 7f720cc00640 1 -- 172.21.15.32:0/4140147903 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7208005020 con 0x7f72100ce340 2024-08-03T20:57:29.118 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:29.115+0000 7f7216000640 1 -- 172.21.15.32:0/4140147903 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7210163130 con 0x7f72100ce340 2024-08-03T20:57:29.118 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:29.115+0000 7f7216000640 1 -- 172.21.15.32:0/4140147903 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f72101636f0 con 0x7f72100ce340 2024-08-03T20:57:29.119 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:29.116+0000 7f7216000640 1 -- 172.21.15.32:0/4140147903 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f72100cf040 con 0x7f72100ce340 2024-08-03T20:57:29.119 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:29.118+0000 7f720cc00640 1 -- 172.21.15.32:0/4140147903 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f72080051c0 con 0x7f72100ce340 2024-08-03T20:57:29.120 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:29.119+0000 7f720cc00640 1 --2- 172.21.15.32:0/4140147903 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f71f0077040 0x7f71f0079550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:29.120 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:29.119+0000 7f720cc00640 1 -- 172.21.15.32:0/4140147903 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(105..105 src has 1..105) ==== 7574+0+0 (secure 0 0 0) 0x7f7208099c00 con 0x7f72100ce340 2024-08-03T20:57:29.120 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:29.119+0000 7f7214c00640 1 --2- 172.21.15.32:0/4140147903 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f71f0077040 0x7f71f0079550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:29.121 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:29.120+0000 7f7214c00640 1 --2- 172.21.15.32:0/4140147903 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f71f0077040 0x7f71f0079550 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7f7204001610 tx=0x7f7204009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:57:29.124 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:29.123+0000 7f720cc00640 1 -- 172.21.15.32:0/4140147903 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7f72080626a0 con 0x7f72100ce340 2024-08-03T20:57:29.374 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:29.372+0000 7f7216000640 1 -- 172.21.15.32:0/4140147903 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f72100cac30 con 0x7f72100ce340 2024-08-03T20:57:29.375 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:29.373+0000 7f720cc00640 1 -- 172.21.15.32:0/4140147903 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 211 v211) ==== 95+0+4391 (secure 0 0 0) 0x7f7208061fa0 con 0x7f72100ce340 2024-08-03T20:57:29.375 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-03T20:57:29.375 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":211,"btime":"2024-08-03T20:57:28:354846+0000","default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6439,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6835/531080160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6834","nonce":531080160},{"type":"v1","addr":"172.21.15.118:6835","nonce":531080160}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":84},{"gid":6454,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6837/2796882319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6836","nonce":2796882319},{"type":"v1","addr":"172.21.15.118:6837","nonce":2796882319}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":86},{"gid":6813,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6834/2686933890","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":2686933890},{"type":"v1","addr":"172.21.15.32:6834","nonce":2686933890}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":85}],"filesystems":[{"mdsmap":{"epoch":211,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-08-03T20:54:26.950518+0000","modified":"2024-08-03T20:57:28.354785+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":105,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":6}]} 2024-08-03T20:57:29.375 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 211 2024-08-03T20:57:29.377 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:29.375+0000 7f7216000640 1 -- 172.21.15.32:0/4140147903 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f71f0077040 msgr2=0x7f71f0079550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:29.377 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:29.375+0000 7f7216000640 1 --2- 172.21.15.32:0/4140147903 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f71f0077040 0x7f71f0079550 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7f7204001610 tx=0x7f7204009040 comp rx=0 tx=0).stop 2024-08-03T20:57:29.377 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:29.375+0000 7f7216000640 1 -- 172.21.15.32:0/4140147903 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f72100ce340 msgr2=0x7f72100b84b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:29.377 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:29.375+0000 7f7216000640 1 --2- 172.21.15.32:0/4140147903 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f72100ce340 0x7f72100b84b0 secure :-1 s=READY pgs=1559 cs=0 l=1 rev1=1 crypto rx=0x7f720800ea80 tx=0x7f720800ef40 comp rx=0 tx=0).stop 2024-08-03T20:57:29.378 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:29.375+0000 7f7216000640 1 -- 172.21.15.32:0/4140147903 shutdown_connections 2024-08-03T20:57:29.378 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:29.375+0000 7f7216000640 1 --2- 172.21.15.32:0/4140147903 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f71f0077040 0x7f71f0079550 unknown :-1 s=CLOSED pgs=635 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:29.378 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:29.376+0000 7f7216000640 1 --2- 172.21.15.32:0/4140147903 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f72100d4bb0 0x7f72100cde00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:29.378 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:29.376+0000 7f7216000640 1 --2- 172.21.15.32:0/4140147903 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f72100ce340 0x7f72100b84b0 unknown :-1 s=CLOSED pgs=1559 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:29.378 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:29.376+0000 7f7216000640 1 --2- 172.21.15.32:0/4140147903 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f72100cec30 0x7f72101522e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:29.378 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:29.376+0000 7f7216000640 1 -- 172.21.15.32:0/4140147903 >> 172.21.15.32:0/4140147903 conn(0x7f72100d4090 msgr2=0x7f721006e640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:57:29.378 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:29.376+0000 7f7216000640 1 -- 172.21.15.32:0/4140147903 shutdown_connections 2024-08-03T20:57:29.378 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:29.376+0000 7f7216000640 1 -- 172.21.15.32:0/4140147903 wait complete. 2024-08-03T20:57:29.399 DEBUG:tasks.ceph_test_case:wait_until_true: success in 0s and 0 retries 2024-08-03T20:57:29.400 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-08-03T20:57:29.628 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:29.626+0000 7f3ab5a00640 1 -- 172.21.15.32:0/2362948426 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3ab0073e60 msgr2=0x7f3ab0074270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:29.628 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:29.626+0000 7f3ab5a00640 1 --2- 172.21.15.32:0/2362948426 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3ab0073e60 0x7f3ab0074270 secure :-1 s=READY pgs=1560 cs=0 l=1 rev1=1 crypto rx=0x7f3ab0071700 tx=0x7f3a9802a560 comp rx=0 tx=0).stop 2024-08-03T20:57:29.628 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:29.626+0000 7f3ab5a00640 1 -- 172.21.15.32:0/2362948426 shutdown_connections 2024-08-03T20:57:29.629 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:29.626+0000 7f3ab5a00640 1 --2- 172.21.15.32:0/2362948426 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f3ab00747b0 0x7f3ab00cc8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:29.629 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:29.626+0000 7f3ab5a00640 1 --2- 172.21.15.32:0/2362948426 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3ab0073e60 0x7f3ab0074270 unknown :-1 s=CLOSED pgs=1560 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:29.629 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:29.626+0000 7f3ab5a00640 1 --2- 172.21.15.32:0/2362948426 >> v2:172.21.15.118:3300/0 conn(0x7f3ab006e930 0x7f3ab0144f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:29.629 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:29.626+0000 7f3ab5a00640 1 -- 172.21.15.32:0/2362948426 >> 172.21.15.32:0/2362948426 conn(0x7f3ab00b1bd0 msgr2=0x7f3ab00b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:57:29.629 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:29.626+0000 7f3ab5a00640 1 -- 172.21.15.32:0/2362948426 shutdown_connections 2024-08-03T20:57:29.629 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:29.627+0000 7f3ab5a00640 1 -- 172.21.15.32:0/2362948426 wait complete. 2024-08-03T20:57:29.630 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:29.629+0000 7f3ab5a00640 1 Processor -- start 2024-08-03T20:57:29.630 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:29.629+0000 7f3ab5a00640 1 -- start start 2024-08-03T20:57:29.633 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:29.630+0000 7f3ab5a00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f3ab006e930 0x7f3ab015bd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:29.634 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:29.630+0000 7f3ab5a00640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f3ab0073e60 0x7f3ab01f5800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:29.634 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:29.630+0000 7f3ab5a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3ab00747b0 0x7f3ab00c5c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:29.634 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:29.630+0000 7f3ab5a00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f3ab00cf9c0 con 0x7f3ab00747b0 2024-08-03T20:57:29.634 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:29.630+0000 7f3ab5a00640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f3ab00cf840 con 0x7f3ab006e930 2024-08-03T20:57:29.634 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:29.630+0000 7f3ab5a00640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7f3ab00cce20 con 0x7f3ab0073e60 2024-08-03T20:57:29.634 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:29.631+0000 7f3aaf400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3ab00747b0 0x7f3ab00c5c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:29.634 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:29.631+0000 7f3aaf400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3ab00747b0 0x7f3ab00c5c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:59664/0 (socket says 172.21.15.32:59664) 2024-08-03T20:57:29.635 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:29.631+0000 7f3aaf400640 1 -- 172.21.15.32:0/2532961517 learned_addr learned my addr 172.21.15.32:0/2532961517 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:57:29.635 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:29.631+0000 7f3aaea00640 1 --2- 172.21.15.32:0/2532961517 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f3ab006e930 0x7f3ab015bd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:29.635 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:29.631+0000 7f3aa7e00640 1 --2- 172.21.15.32:0/2532961517 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f3ab0073e60 0x7f3ab01f5800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:29.635 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:29.631+0000 7f3aaf400640 1 -- 172.21.15.32:0/2532961517 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f3ab0073e60 msgr2=0x7f3ab01f5800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:29.635 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:29.631+0000 7f3aaf400640 1 --2- 172.21.15.32:0/2532961517 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f3ab0073e60 0x7f3ab01f5800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:29.635 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:29.631+0000 7f3aaf400640 1 -- 172.21.15.32:0/2532961517 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f3ab006e930 msgr2=0x7f3ab015bd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:29.635 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:29.631+0000 7f3aaf400640 1 --2- 172.21.15.32:0/2532961517 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f3ab006e930 0x7f3ab015bd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:29.635 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:29.631+0000 7f3aaf400640 1 -- 172.21.15.32:0/2532961517 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3ab01f68a0 con 0x7f3ab00747b0 2024-08-03T20:57:29.635 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:29.631+0000 7f3aaf400640 1 --2- 172.21.15.32:0/2532961517 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3ab00747b0 0x7f3ab00c5c30 secure :-1 s=READY pgs=1561 cs=0 l=1 rev1=1 crypto rx=0x7f3aa800eab0 tx=0x7f3aa800ef70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:57:29.636 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:29.632+0000 7f3aa6a00640 1 -- 172.21.15.32:0/2532961517 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3aa800cea0 con 0x7f3ab00747b0 2024-08-03T20:57:29.636 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:29.632+0000 7f3aa6a00640 1 -- 172.21.15.32:0/2532961517 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f3aa8004b20 con 0x7f3ab00747b0 2024-08-03T20:57:29.636 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:29.632+0000 7f3aa6a00640 1 -- 172.21.15.32:0/2532961517 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3aa8005020 con 0x7f3ab00747b0 2024-08-03T20:57:29.636 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:29.632+0000 7f3ab5a00640 1 -- 172.21.15.32:0/2532961517 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3ab01f6b10 con 0x7f3ab00747b0 2024-08-03T20:57:29.636 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:29.632+0000 7f3ab5a00640 1 -- 172.21.15.32:0/2532961517 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3ab01f70d0 con 0x7f3ab00747b0 2024-08-03T20:57:29.637 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:29.633+0000 7f3ab5a00640 1 -- 172.21.15.32:0/2532961517 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3ab006a430 con 0x7f3ab00747b0 2024-08-03T20:57:29.637 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:29.634+0000 7f3aa6a00640 1 -- 172.21.15.32:0/2532961517 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f3aa80051c0 con 0x7f3ab00747b0 2024-08-03T20:57:29.637 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:29.635+0000 7f3aa6a00640 1 --2- 172.21.15.32:0/2532961517 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f3a8c077040 0x7f3a8c079550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:29.638 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:29.635+0000 7f3aa6a00640 1 -- 172.21.15.32:0/2532961517 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(105..105 src has 1..105) ==== 7574+0+0 (secure 0 0 0) 0x7f3aa8099af0 con 0x7f3ab00747b0 2024-08-03T20:57:29.638 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:29.636+0000 7f3aaea00640 1 --2- 172.21.15.32:0/2532961517 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f3a8c077040 0x7f3a8c079550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:29.638 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:29.636+0000 7f3aaea00640 1 --2- 172.21.15.32:0/2532961517 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f3a8c077040 0x7f3a8c079550 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7f3aa0009b30 tx=0x7f3aa0005f90 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:57:29.640 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:29.639+0000 7f3aa6a00640 1 -- 172.21.15.32:0/2532961517 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7f3ab006a430 con 0x7f3ab00747b0 2024-08-03T20:57:29.881 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:29.879+0000 7f3ab5a00640 1 -- 172.21.15.32:0/2532961517 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f3ab0074270 con 0x7f3ab00747b0 2024-08-03T20:57:29.881 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:29.880+0000 7f3aa6a00640 1 -- 172.21.15.32:0/2532961517 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 211 v211) ==== 95+0+4391 (secure 0 0 0) 0x7f3aa8061eb0 con 0x7f3ab00747b0 2024-08-03T20:57:29.882 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-03T20:57:29.882 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":211,"btime":"2024-08-03T20:57:28:354846+0000","default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6439,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6835/531080160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6834","nonce":531080160},{"type":"v1","addr":"172.21.15.118:6835","nonce":531080160}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":84},{"gid":6454,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6837/2796882319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6836","nonce":2796882319},{"type":"v1","addr":"172.21.15.118:6837","nonce":2796882319}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":86},{"gid":6813,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6834/2686933890","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":2686933890},{"type":"v1","addr":"172.21.15.32:6834","nonce":2686933890}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":85}],"filesystems":[{"mdsmap":{"epoch":211,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-08-03T20:54:26.950518+0000","modified":"2024-08-03T20:57:28.354785+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":105,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":6}]} 2024-08-03T20:57:29.882 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 211 2024-08-03T20:57:29.884 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:29.882+0000 7f3ab5a00640 1 -- 172.21.15.32:0/2532961517 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f3a8c077040 msgr2=0x7f3a8c079550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:29.884 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:29.882+0000 7f3ab5a00640 1 --2- 172.21.15.32:0/2532961517 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f3a8c077040 0x7f3a8c079550 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7f3aa0009b30 tx=0x7f3aa0005f90 comp rx=0 tx=0).stop 2024-08-03T20:57:29.884 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:29.882+0000 7f3ab5a00640 1 -- 172.21.15.32:0/2532961517 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3ab00747b0 msgr2=0x7f3ab00c5c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:29.884 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:29.882+0000 7f3ab5a00640 1 --2- 172.21.15.32:0/2532961517 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3ab00747b0 0x7f3ab00c5c30 secure :-1 s=READY pgs=1561 cs=0 l=1 rev1=1 crypto rx=0x7f3aa800eab0 tx=0x7f3aa800ef70 comp rx=0 tx=0).stop 2024-08-03T20:57:29.884 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:29.882+0000 7f3ab5a00640 1 -- 172.21.15.32:0/2532961517 shutdown_connections 2024-08-03T20:57:29.885 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:29.882+0000 7f3ab5a00640 1 --2- 172.21.15.32:0/2532961517 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f3a8c077040 0x7f3a8c079550 unknown :-1 s=CLOSED pgs=636 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:29.885 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:29.882+0000 7f3ab5a00640 1 --2- 172.21.15.32:0/2532961517 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3ab00747b0 0x7f3ab00c5c30 unknown :-1 s=CLOSED pgs=1561 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:29.885 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:29.882+0000 7f3ab5a00640 1 --2- 172.21.15.32:0/2532961517 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f3ab0073e60 0x7f3ab01f5800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:29.885 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:29.882+0000 7f3ab5a00640 1 --2- 172.21.15.32:0/2532961517 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f3ab006e930 0x7f3ab015bd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:29.885 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:29.882+0000 7f3ab5a00640 1 -- 172.21.15.32:0/2532961517 >> 172.21.15.32:0/2532961517 conn(0x7f3ab00b1bd0 msgr2=0x7f3ab00b3950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:57:29.885 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:29.883+0000 7f3ab5a00640 1 -- 172.21.15.32:0/2532961517 shutdown_connections 2024-08-03T20:57:29.885 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:29.883+0000 7f3ab5a00640 1 -- 172.21.15.32:0/2532961517 wait complete. 2024-08-03T20:57:29.906 INFO:tasks.cephfs.test_damage:Result: Mutation 'Corrupt mds_snaptable' led to DAMAGED state 2024-08-03T20:57:29.906 INFO:tasks.cephfs.test_damage:Applying mutation 'Corrupt mds0_inotable' 2024-08-03T20:57:29.906 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-08-03T20:57:29.906 DEBUG:teuthology.orchestra.run.smithi032:> dd if=/proc/self/mounts of=/dev/stdout 2024-08-03T20:57:29.942 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053453,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6499208k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 none /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249604k,nr_inodes=812401,mode=700,uid=1000,gid=1316 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6499208k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-08-03T20:57:29.942 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi032.front.sepia.ceph.com 2024-08-03T20:57:29.942 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-08-03T20:57:29.942 DEBUG:teuthology.orchestra.run.smithi032:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-08-03T20:57:30.017 INFO:teuthology.orchestra.run.smithi032.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.0': No such file or directory 2024-08-03T20:57:30.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-03T20:57:30.019 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-08-03T20:57:30.289 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:30.287+0000 7f8cbb200640 1 -- 172.21.15.32:0/2513988902 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8cbc0af020 msgr2=0x7f8cbc0742b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:30.289 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:30.287+0000 7f8cbb200640 1 --2- 172.21.15.32:0/2513988902 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8cbc0af020 0x7f8cbc0742b0 secure :-1 s=READY pgs=1562 cs=0 l=1 rev1=1 crypto rx=0x7f8cbc160f00 tx=0x7f8cac02a4d0 comp rx=0 tx=0).stop 2024-08-03T20:57:30.289 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:30.287+0000 7f8cbb200640 1 -- 172.21.15.32:0/2513988902 shutdown_connections 2024-08-03T20:57:30.289 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:30.287+0000 7f8cbb200640 1 --2- 172.21.15.32:0/2513988902 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f8cbc06f440 0x7f8cbc06f850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:30.289 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:30.287+0000 7f8cbb200640 1 --2- 172.21.15.32:0/2513988902 >> v2:172.21.15.118:3300/0 conn(0x7f8cbc0acd80 0x7f8cbc0ad190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:30.289 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:30.287+0000 7f8cbb200640 1 --2- 172.21.15.32:0/2513988902 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8cbc0af020 0x7f8cbc0742b0 unknown :-1 s=CLOSED pgs=1562 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:30.289 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:30.287+0000 7f8cbb200640 1 -- 172.21.15.32:0/2513988902 >> 172.21.15.32:0/2513988902 conn(0x7f8cbc0b2120 msgr2=0x7f8cbc0cfd60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:57:30.289 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:30.287+0000 7f8cbb200640 1 -- 172.21.15.32:0/2513988902 shutdown_connections 2024-08-03T20:57:30.289 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:30.287+0000 7f8cbb200640 1 -- 172.21.15.32:0/2513988902 wait complete. 2024-08-03T20:57:30.290 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:30.289+0000 7f8cbb200640 1 Processor -- start 2024-08-03T20:57:30.290 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:30.289+0000 7f8cbb200640 1 -- start start 2024-08-03T20:57:30.293 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:30.291+0000 7f8cbb200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8cbc06f440 0x7f8cbc155990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:30.293 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:30.291+0000 7f8cbb200640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8cbc0acd80 0x7f8cbc15c790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:30.293 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:30.291+0000 7f8cbb200640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f8cbc0af020 0x7f8cbc15ccd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:30.293 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:30.291+0000 7f8cbb200640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f8cbc15ffd0 con 0x7f8cbc06f440 2024-08-03T20:57:30.293 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:30.291+0000 7f8cbb200640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f8cbc15fe50 con 0x7f8cbc0acd80 2024-08-03T20:57:30.293 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:30.291+0000 7f8cbb200640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7f8cbc160150 con 0x7f8cbc0af020 2024-08-03T20:57:30.293 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:30.291+0000 7f8cb9e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8cbc06f440 0x7f8cbc155990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:30.293 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:30.291+0000 7f8cb9e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8cbc06f440 0x7f8cbc155990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:33782/0 (socket says 172.21.15.32:33782) 2024-08-03T20:57:30.294 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:30.291+0000 7f8cb9e00640 1 -- 172.21.15.32:0/2599665 learned_addr learned my addr 172.21.15.32:0/2599665 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:57:30.294 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:30.291+0000 7f8cba800640 1 --2- 172.21.15.32:0/2599665 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f8cbc0af020 0x7f8cbc15ccd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:30.294 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:30.291+0000 7f8cb3e00640 1 --2- 172.21.15.32:0/2599665 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8cbc0acd80 0x7f8cbc15c790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:30.294 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:30.291+0000 7f8cb9e00640 1 -- 172.21.15.32:0/2599665 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f8cbc0af020 msgr2=0x7f8cbc15ccd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:30.294 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:30.291+0000 7f8cb9e00640 1 --2- 172.21.15.32:0/2599665 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f8cbc0af020 0x7f8cbc15ccd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:30.294 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:30.291+0000 7f8cb9e00640 1 -- 172.21.15.32:0/2599665 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8cbc0acd80 msgr2=0x7f8cbc15c790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:30.294 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:30.291+0000 7f8cb9e00640 1 --2- 172.21.15.32:0/2599665 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8cbc0acd80 0x7f8cbc15c790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:30.294 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:30.291+0000 7f8cb9e00640 1 -- 172.21.15.32:0/2599665 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8cbc0b3cb0 con 0x7f8cbc06f440 2024-08-03T20:57:30.294 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:30.292+0000 7f8cb9e00640 1 --2- 172.21.15.32:0/2599665 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8cbc06f440 0x7f8cbc155990 secure :-1 s=READY pgs=1563 cs=0 l=1 rev1=1 crypto rx=0x7f8cac0099c0 tx=0x7f8cac0038a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:57:30.295 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:30.292+0000 7f8cb2000640 1 -- 172.21.15.32:0/2599665 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8cac03e030 con 0x7f8cbc06f440 2024-08-03T20:57:30.295 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:30.292+0000 7f8cb2000640 1 -- 172.21.15.32:0/2599665 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f8cac003db0 con 0x7f8cbc06f440 2024-08-03T20:57:30.295 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:30.292+0000 7f8cb2000640 1 -- 172.21.15.32:0/2599665 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8cac035040 con 0x7f8cbc06f440 2024-08-03T20:57:30.295 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:30.292+0000 7f8cbb200640 1 -- 172.21.15.32:0/2599665 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8cbc1f3740 con 0x7f8cbc06f440 2024-08-03T20:57:30.295 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:30.292+0000 7f8cbb200640 1 -- 172.21.15.32:0/2599665 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8cbc1f3d00 con 0x7f8cbc06f440 2024-08-03T20:57:30.296 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:30.293+0000 7f8cbb200640 1 -- 172.21.15.32:0/2599665 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8cbc1606e0 con 0x7f8cbc06f440 2024-08-03T20:57:30.296 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:30.294+0000 7f8cb2000640 1 -- 172.21.15.32:0/2599665 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f8cac041030 con 0x7f8cbc06f440 2024-08-03T20:57:30.297 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:30.295+0000 7f8cb2000640 1 --2- 172.21.15.32:0/2599665 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f8c98077010 0x7f8c98079520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:30.297 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:30.295+0000 7f8cb2000640 1 -- 172.21.15.32:0/2599665 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(105..105 src has 1..105) ==== 7574+0+0 (secure 0 0 0) 0x7f8cac0b9aa0 con 0x7f8cbc06f440 2024-08-03T20:57:30.297 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:30.296+0000 7f8cb3e00640 1 --2- 172.21.15.32:0/2599665 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f8c98077010 0x7f8c98079520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:30.298 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:30.297+0000 7f8cb3e00640 1 --2- 172.21.15.32:0/2599665 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f8c98077010 0x7f8c98079520 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7f8cbc0c3ca0 tx=0x7f8ca4005e30 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:57:30.301 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:30.300+0000 7f8cb2000640 1 -- 172.21.15.32:0/2599665 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7f8cac082540 con 0x7f8cbc06f440 2024-08-03T20:57:30.549 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:30.547+0000 7f8cbb200640 1 -- 172.21.15.32:0/2599665 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7f8cbc0d0930 con 0x7f8cbc06f440 2024-08-03T20:57:31.369 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:31.367+0000 7f8cb2000640 1 -- 172.21.15.32:0/2599665 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v212) ==== 161+0+0 (secure 0 0 0) 0x7f8cac081e40 con 0x7f8cbc06f440 2024-08-03T20:57:31.369 INFO:teuthology.orchestra.run.smithi032.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-08-03T20:57:31.372 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:31.370+0000 7f8cbb200640 1 -- 172.21.15.32:0/2599665 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f8c98077010 msgr2=0x7f8c98079520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:31.372 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:31.370+0000 7f8cbb200640 1 --2- 172.21.15.32:0/2599665 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f8c98077010 0x7f8c98079520 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7f8cbc0c3ca0 tx=0x7f8ca4005e30 comp rx=0 tx=0).stop 2024-08-03T20:57:31.372 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:31.370+0000 7f8cbb200640 1 -- 172.21.15.32:0/2599665 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8cbc06f440 msgr2=0x7f8cbc155990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:31.372 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:31.370+0000 7f8cbb200640 1 --2- 172.21.15.32:0/2599665 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8cbc06f440 0x7f8cbc155990 secure :-1 s=READY pgs=1563 cs=0 l=1 rev1=1 crypto rx=0x7f8cac0099c0 tx=0x7f8cac0038a0 comp rx=0 tx=0).stop 2024-08-03T20:57:31.372 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:31.370+0000 7f8cbb200640 1 -- 172.21.15.32:0/2599665 shutdown_connections 2024-08-03T20:57:31.372 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:31.370+0000 7f8cbb200640 1 --2- 172.21.15.32:0/2599665 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f8c98077010 0x7f8c98079520 unknown :-1 s=CLOSED pgs=637 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:31.373 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:31.370+0000 7f8cbb200640 1 --2- 172.21.15.32:0/2599665 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8cbc0acd80 0x7f8cbc15c790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:31.373 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:31.370+0000 7f8cbb200640 1 --2- 172.21.15.32:0/2599665 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f8cbc0af020 0x7f8cbc15ccd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:31.373 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:31.370+0000 7f8cbb200640 1 --2- 172.21.15.32:0/2599665 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8cbc06f440 0x7f8cbc155990 unknown :-1 s=CLOSED pgs=1563 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:31.373 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:31.370+0000 7f8cbb200640 1 -- 172.21.15.32:0/2599665 >> 172.21.15.32:0/2599665 conn(0x7f8cbc0b2120 msgr2=0x7f8cbc0cd1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:57:31.373 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:31.371+0000 7f8cbb200640 1 -- 172.21.15.32:0/2599665 shutdown_connections 2024-08-03T20:57:31.373 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:31.371+0000 7f8cbb200640 1 -- 172.21.15.32:0/2599665 wait complete. 2024-08-03T20:57:31.400 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mds repaired 0 2024-08-03T20:57:31.636 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:31.634+0000 7fe6e1a00640 1 -- 172.21.15.32:0/2385421372 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe6dc15b790 msgr2=0x7fe6dc15f490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:31.636 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:31.634+0000 7fe6e1a00640 1 --2- 172.21.15.32:0/2385421372 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe6dc15b790 0x7fe6dc15f490 secure :-1 s=READY pgs=1565 cs=0 l=1 rev1=1 crypto rx=0x7fe6c4009830 tx=0x7fe6c402a480 comp rx=0 tx=0).stop 2024-08-03T20:57:31.636 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:31.634+0000 7fe6e1a00640 1 -- 172.21.15.32:0/2385421372 shutdown_connections 2024-08-03T20:57:31.636 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:31.634+0000 7fe6e1a00640 1 --2- 172.21.15.32:0/2385421372 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe6dc15b790 0x7fe6dc15f490 unknown :-1 s=CLOSED pgs=1565 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:31.636 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:31.634+0000 7fe6e1a00640 1 --2- 172.21.15.32:0/2385421372 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fe6dc071150 0x7fe6dc070590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:31.636 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:31.634+0000 7fe6e1a00640 1 -- 172.21.15.32:0/2385421372 >> 172.21.15.32:0/2385421372 conn(0x7fe6dc0d0dd0 msgr2=0x7fe6dc0d11e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:57:31.636 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:31.634+0000 7fe6e1a00640 1 -- 172.21.15.32:0/2385421372 shutdown_connections 2024-08-03T20:57:31.637 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:31.634+0000 7fe6e1a00640 1 -- 172.21.15.32:0/2385421372 wait complete. 2024-08-03T20:57:31.638 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:31.636+0000 7fe6e1a00640 1 Processor -- start 2024-08-03T20:57:31.638 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:31.636+0000 7fe6e1a00640 1 -- start start 2024-08-03T20:57:31.639 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:31.638+0000 7fe6e1a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe6dc071150 0x7fe6dc1554b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:31.640 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:31.638+0000 7fe6e1a00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fe6dc15b790 0x7fe6dc1530e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:31.640 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:31.638+0000 7fe6e1a00640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fe6dc15f9d0 0x7fe6dc0b29b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:31.640 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:31.638+0000 7fe6e1a00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fe6dc0736b0 con 0x7fe6dc071150 2024-08-03T20:57:31.640 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:31.638+0000 7fe6e1a00640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7fe6dc15bd40 con 0x7fe6dc15b790 2024-08-03T20:57:31.640 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:31.638+0000 7fe6e1a00640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7fe6dc073830 con 0x7fe6dc15f9d0 2024-08-03T20:57:31.640 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:31.638+0000 7fe6dbe00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe6dc071150 0x7fe6dc1554b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:31.640 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:31.638+0000 7fe6dbe00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe6dc071150 0x7fe6dc1554b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:33806/0 (socket says 172.21.15.32:33806) 2024-08-03T20:57:31.640 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:31.638+0000 7fe6dbe00640 1 -- 172.21.15.32:0/1776712620 learned_addr learned my addr 172.21.15.32:0/1776712620 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:57:31.642 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:31.639+0000 7fe6d3e00640 1 --2- 172.21.15.32:0/1776712620 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fe6dc15b790 0x7fe6dc1530e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:31.642 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:31.638+0000 7fe6e1000640 1 --2- 172.21.15.32:0/1776712620 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fe6dc15f9d0 0x7fe6dc0b29b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:31.642 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:31.639+0000 7fe6dbe00640 1 -- 172.21.15.32:0/1776712620 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fe6dc15f9d0 msgr2=0x7fe6dc0b29b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:31.642 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:31.639+0000 7fe6dbe00640 1 --2- 172.21.15.32:0/1776712620 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fe6dc15f9d0 0x7fe6dc0b29b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:31.642 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:31.639+0000 7fe6dbe00640 1 -- 172.21.15.32:0/1776712620 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fe6dc15b790 msgr2=0x7fe6dc1530e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:31.642 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:31.639+0000 7fe6dbe00640 1 --2- 172.21.15.32:0/1776712620 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fe6dc15b790 0x7fe6dc1530e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:31.642 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:31.639+0000 7fe6dbe00640 1 -- 172.21.15.32:0/1776712620 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe6dc155260 con 0x7fe6dc071150 2024-08-03T20:57:31.642 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:31.639+0000 7fe6d3e00640 1 --2- 172.21.15.32:0/1776712620 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fe6dc15b790 0x7fe6dc1530e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-03T20:57:31.642 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:31.639+0000 7fe6e1000640 1 --2- 172.21.15.32:0/1776712620 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fe6dc15f9d0 0x7fe6dc0b29b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-03T20:57:31.642 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:31.639+0000 7fe6dbe00640 1 --2- 172.21.15.32:0/1776712620 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe6dc071150 0x7fe6dc1554b0 secure :-1 s=READY pgs=1566 cs=0 l=1 rev1=1 crypto rx=0x7fe6cc00d750 tx=0x7fe6cc00dc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:57:31.643 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:31.639+0000 7fe6d8c00640 1 -- 172.21.15.32:0/1776712620 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6cc017070 con 0x7fe6dc071150 2024-08-03T20:57:31.643 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:31.639+0000 7fe6d8c00640 1 -- 172.21.15.32:0/1776712620 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fe6cc004310 con 0x7fe6dc071150 2024-08-03T20:57:31.643 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:31.639+0000 7fe6d8c00640 1 -- 172.21.15.32:0/1776712620 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6cc00de60 con 0x7fe6dc071150 2024-08-03T20:57:31.643 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:31.639+0000 7fe6e1a00640 1 -- 172.21.15.32:0/1776712620 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe6dc154090 con 0x7fe6dc071150 2024-08-03T20:57:31.643 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:31.639+0000 7fe6e1a00640 1 -- 172.21.15.32:0/1776712620 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe6dc154650 con 0x7fe6dc071150 2024-08-03T20:57:31.643 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:31.640+0000 7fe6e1a00640 1 -- 172.21.15.32:0/1776712620 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe6dc073d50 con 0x7fe6dc071150 2024-08-03T20:57:31.643 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:31.642+0000 7fe6d8c00640 1 -- 172.21.15.32:0/1776712620 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7fe6cc009c30 con 0x7fe6dc071150 2024-08-03T20:57:31.644 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:31.643+0000 7fe6d8c00640 1 --2- 172.21.15.32:0/1776712620 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fe6b8077010 0x7fe6b8079520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:31.644 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:31.643+0000 7fe6d8c00640 1 -- 172.21.15.32:0/1776712620 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(105..105 src has 1..105) ==== 7574+0+0 (secure 0 0 0) 0x7fe6cc0993e0 con 0x7fe6dc071150 2024-08-03T20:57:31.644 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:31.643+0000 7fe6d3e00640 1 --2- 172.21.15.32:0/1776712620 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fe6b8077010 0x7fe6b8079520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:31.645 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:31.644+0000 7fe6d3e00640 1 --2- 172.21.15.32:0/1776712620 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fe6b8077010 0x7fe6b8079520 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7fe6c4034000 tx=0x7fe6c4004730 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:57:31.648 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:31.647+0000 7fe6d8c00640 1 -- 172.21.15.32:0/1776712620 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7fe6cc020020 con 0x7fe6dc071150 2024-08-03T20:57:31.930 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:31.929+0000 7fe6e1a00640 1 -- 172.21.15.32:0/1776712620 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "mds repaired", "role": "0"} v 0) -- 0x7fe6dc0b6d50 con 0x7fe6dc071150 2024-08-03T20:57:32.376 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:32.374+0000 7fe6d8c00640 1 -- 172.21.15.32:0/1776712620 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "mds repaired", "role": "0"}]=0 repaired: restoring rank 6:0 v213) ==== 101+0+0 (secure 0 0 0) 0x7fe6dc155260 con 0x7fe6dc071150 2024-08-03T20:57:32.376 INFO:teuthology.orchestra.run.smithi032.stderr:repaired: restoring rank 6:0 2024-08-03T20:57:32.379 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:32.377+0000 7fe6e1a00640 1 -- 172.21.15.32:0/1776712620 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fe6b8077010 msgr2=0x7fe6b8079520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:32.379 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:32.377+0000 7fe6e1a00640 1 --2- 172.21.15.32:0/1776712620 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fe6b8077010 0x7fe6b8079520 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7fe6c4034000 tx=0x7fe6c4004730 comp rx=0 tx=0).stop 2024-08-03T20:57:32.379 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:32.377+0000 7fe6e1a00640 1 -- 172.21.15.32:0/1776712620 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe6dc071150 msgr2=0x7fe6dc1554b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:32.379 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:32.377+0000 7fe6e1a00640 1 --2- 172.21.15.32:0/1776712620 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe6dc071150 0x7fe6dc1554b0 secure :-1 s=READY pgs=1566 cs=0 l=1 rev1=1 crypto rx=0x7fe6cc00d750 tx=0x7fe6cc00dc10 comp rx=0 tx=0).stop 2024-08-03T20:57:32.379 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:32.377+0000 7fe6e1a00640 1 -- 172.21.15.32:0/1776712620 shutdown_connections 2024-08-03T20:57:32.379 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:32.377+0000 7fe6e1a00640 1 --2- 172.21.15.32:0/1776712620 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fe6b8077010 0x7fe6b8079520 unknown :-1 s=CLOSED pgs=638 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:32.379 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:32.377+0000 7fe6e1a00640 1 --2- 172.21.15.32:0/1776712620 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fe6dc15b790 0x7fe6dc1530e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:32.380 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:32.377+0000 7fe6e1a00640 1 --2- 172.21.15.32:0/1776712620 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fe6dc15f9d0 0x7fe6dc0b29b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:32.380 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:32.377+0000 7fe6e1a00640 1 --2- 172.21.15.32:0/1776712620 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe6dc071150 0x7fe6dc1554b0 unknown :-1 s=CLOSED pgs=1566 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:32.380 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:32.377+0000 7fe6e1a00640 1 -- 172.21.15.32:0/1776712620 >> 172.21.15.32:0/1776712620 conn(0x7fe6dc0d0dd0 msgr2=0x7fe6dc0b5e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:57:32.380 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:32.378+0000 7fe6e1a00640 1 -- 172.21.15.32:0/1776712620 shutdown_connections 2024-08-03T20:57:32.380 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:32.378+0000 7fe6e1a00640 1 -- 172.21.15.32:0/1776712620 wait complete. 2024-08-03T20:57:32.408 DEBUG:teuthology.orchestra.run.smithi032:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata import - 2024-08-03T20:57:32.551 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:32.550+0000 7fd3acb25f40 1 Processor -- start 2024-08-03T20:57:32.552 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:32.550+0000 7fd3acb25f40 1 -- start start 2024-08-03T20:57:32.554 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:32.551+0000 7fd3acb25f40 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55620b32cda0 0x55620b3fb9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:32.554 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:32.551+0000 7fd3acb25f40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55620b3722f0 0x55620b3e9460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:32.554 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:32.551+0000 7fd3acb25f40 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55620b3fbf60 0x55620b405fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:32.554 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:32.551+0000 7fd3acb25f40 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55620b27ce30 con 0x55620b32cda0 2024-08-03T20:57:32.554 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:32.551+0000 7fd3acb25f40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x55620b2dabf0 con 0x55620b3722f0 2024-08-03T20:57:32.554 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:32.551+0000 7fd3acb25f40 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x55620b32e610 con 0x55620b3fbf60 2024-08-03T20:57:32.555 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:32.552+0000 7fd3a5200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55620b32cda0 0x55620b3fb9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:32.555 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:32.552+0000 7fd3a5200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55620b32cda0 0x55620b3fb9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:33834/0 (socket says 172.21.15.32:33834) 2024-08-03T20:57:32.555 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:32.552+0000 7fd3a5200640 1 -- 172.21.15.32:0/3138371128 learned_addr learned my addr 172.21.15.32:0/3138371128 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:57:32.555 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:32.552+0000 7fd3a5c00640 1 --2- 172.21.15.32:0/3138371128 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55620b3fbf60 0x55620b405fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:32.555 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:32.552+0000 7fd39fe00640 1 --2- 172.21.15.32:0/3138371128 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55620b3722f0 0x55620b3e9460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:32.555 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:32.552+0000 7fd3a5200640 1 -- 172.21.15.32:0/3138371128 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55620b3fbf60 msgr2=0x55620b405fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:57:32.555 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:32.552+0000 7fd3a5200640 1 --2- 172.21.15.32:0/3138371128 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55620b3fbf60 0x55620b405fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:32.555 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:32.552+0000 7fd3a5200640 1 -- 172.21.15.32:0/3138371128 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55620b3722f0 msgr2=0x55620b3e9460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:57:32.555 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:32.552+0000 7fd3a5200640 1 --2- 172.21.15.32:0/3138371128 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55620b3722f0 0x55620b3e9460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:32.555 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:32.552+0000 7fd3a5200640 1 -- 172.21.15.32:0/3138371128 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55620b406f20 con 0x55620b32cda0 2024-08-03T20:57:32.555 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:32.552+0000 7fd3a5200640 1 --2- 172.21.15.32:0/3138371128 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55620b32cda0 0x55620b3fb9a0 secure :-1 s=READY pgs=1568 cs=0 l=1 rev1=1 crypto rx=0x7fd39800b600 tx=0x7fd39800bac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=139201265ecb5e5b server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:57:32.555 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:32.553+0000 7fd39ea00640 1 -- 172.21.15.32:0/3138371128 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd39800a890 con 0x55620b32cda0 2024-08-03T20:57:32.555 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:32.553+0000 7fd39ea00640 1 -- 172.21.15.32:0/3138371128 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fd39800aa30 con 0x55620b32cda0 2024-08-03T20:57:32.556 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:32.553+0000 7fd39ea00640 1 -- 172.21.15.32:0/3138371128 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd39800ae60 con 0x55620b32cda0 2024-08-03T20:57:32.556 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:32.553+0000 7fd3acb25f40 1 -- 172.21.15.32:0/3138371128 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55620b32cda0 msgr2=0x55620b3fb9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:32.557 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:32.553+0000 7fd3acb25f40 1 --2- 172.21.15.32:0/3138371128 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55620b32cda0 0x55620b3fb9a0 secure :-1 s=READY pgs=1568 cs=0 l=1 rev1=1 crypto rx=0x7fd39800b600 tx=0x7fd39800bac0 comp rx=0 tx=0).stop 2024-08-03T20:57:32.557 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:32.553+0000 7fd3acb25f40 1 -- 172.21.15.32:0/3138371128 shutdown_connections 2024-08-03T20:57:32.557 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:32.553+0000 7fd3acb25f40 1 --2- 172.21.15.32:0/3138371128 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55620b3722f0 0x55620b3e9460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:32.557 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:32.553+0000 7fd3acb25f40 1 --2- 172.21.15.32:0/3138371128 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55620b3fbf60 0x55620b405fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:32.557 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:32.553+0000 7fd3acb25f40 1 --2- 172.21.15.32:0/3138371128 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55620b32cda0 0x55620b3fb9a0 unknown :-1 s=CLOSED pgs=1568 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:32.557 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:32.553+0000 7fd3acb25f40 1 -- 172.21.15.32:0/3138371128 >> 172.21.15.32:0/3138371128 conn(0x55620b24eac0 msgr2=0x55620b3faba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:57:32.557 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:32.553+0000 7fd3acb25f40 1 -- 172.21.15.32:0/3138371128 shutdown_connections 2024-08-03T20:57:32.557 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:32.553+0000 7fd3acb25f40 1 -- 172.21.15.32:0/3138371128 wait complete. 2024-08-03T20:57:32.557 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:32.554+0000 7fd3acb25f40 1 Processor -- start 2024-08-03T20:57:32.557 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:32.555+0000 7fd3acb25f40 1 -- start start 2024-08-03T20:57:32.559 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:32.556+0000 7fd3acb25f40 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55620b32cda0 0x55620b3fd460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:32.559 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:32.556+0000 7fd3acb25f40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55620b3722f0 0x55620b374d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:32.559 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:32.556+0000 7fd3acb25f40 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55620b3fbf60 0x55620b373aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:32.559 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:32.556+0000 7fd3acb25f40 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55620b3e83e0 con 0x55620b3fbf60 2024-08-03T20:57:32.560 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:32.556+0000 7fd3acb25f40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x55620b3e99a0 con 0x55620b3722f0 2024-08-03T20:57:32.560 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:32.556+0000 7fd3acb25f40 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x55620b3de9d0 con 0x55620b32cda0 2024-08-03T20:57:32.560 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:32.556+0000 7fd3a5c00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55620b3fbf60 0x55620b373aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:32.560 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:32.556+0000 7fd3a5c00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55620b3fbf60 0x55620b373aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:33838/0 (socket says 172.21.15.32:33838) 2024-08-03T20:57:32.560 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:32.556+0000 7fd3a5c00640 1 -- 172.21.15.32:0/3831060674 learned_addr learned my addr 172.21.15.32:0/3831060674 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:57:32.560 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:32.556+0000 7fd3a5200640 1 --2- 172.21.15.32:0/3831060674 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55620b32cda0 0x55620b3fd460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:32.560 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:32.556+0000 7fd39fe00640 1 --2- 172.21.15.32:0/3831060674 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55620b3722f0 0x55620b374d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:32.560 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:32.557+0000 7fd3a5c00640 1 -- 172.21.15.32:0/3831060674 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55620b32cda0 msgr2=0x55620b3fd460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:32.560 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:32.557+0000 7fd3a5c00640 1 --2- 172.21.15.32:0/3831060674 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55620b32cda0 0x55620b3fd460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:32.560 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:32.557+0000 7fd3a5c00640 1 -- 172.21.15.32:0/3831060674 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55620b3722f0 msgr2=0x55620b374d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:32.560 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:32.557+0000 7fd3a5c00640 1 --2- 172.21.15.32:0/3831060674 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55620b3722f0 0x55620b374d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:32.560 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:32.557+0000 7fd3a5c00640 1 -- 172.21.15.32:0/3831060674 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55620b3752d0 con 0x55620b3fbf60 2024-08-03T20:57:32.560 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:32.557+0000 7fd3a5c00640 1 --2- 172.21.15.32:0/3831060674 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55620b3fbf60 0x55620b373aa0 secure :-1 s=READY pgs=1569 cs=0 l=1 rev1=1 crypto rx=0x7fd3a0025dd0 tx=0x7fd3a0004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:57:32.560 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:32.557+0000 7fd39d600640 1 -- 172.21.15.32:0/3831060674 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3a0064030 con 0x55620b3fbf60 2024-08-03T20:57:32.561 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:32.557+0000 7fd39d600640 1 -- 172.21.15.32:0/3831060674 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fd3a0004020 con 0x55620b3fbf60 2024-08-03T20:57:32.561 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:32.557+0000 7fd39d600640 1 -- 172.21.15.32:0/3831060674 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3a0004250 con 0x55620b3fbf60 2024-08-03T20:57:32.561 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:32.557+0000 7fd3acb25f40 1 -- 172.21.15.32:0/3831060674 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x55620b3e92d0 con 0x55620b3fbf60 2024-08-03T20:57:32.561 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:32.558+0000 7fd3acb25f40 1 -- 172.21.15.32:0/3831060674 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55620b3dc230 con 0x55620b3fbf60 2024-08-03T20:57:32.561 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:32.560+0000 7fd39d600640 1 -- 172.21.15.32:0/3831060674 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7fd3a005e040 con 0x55620b3fbf60 2024-08-03T20:57:32.562 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:32.560+0000 7fd39d600640 1 --2- 172.21.15.32:0/3831060674 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fd3840759f0 0x7fd384077f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:32.562 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:32.561+0000 7fd39d600640 1 -- 172.21.15.32:0/3831060674 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(105..105 src has 1..105) ==== 7574+0+0 (secure 0 0 0) 0x7fd3a00dbfe0 con 0x55620b3fbf60 2024-08-03T20:57:32.562 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:32.561+0000 7fd3a5200640 1 --2- 172.21.15.32:0/3831060674 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fd3840759f0 0x7fd384077f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:32.563 INFO:teuthology.orchestra.run.smithi032.stdout:Importing pool 2024-08-03T20:57:32.563 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:32.561+0000 7fd3acb25f40 1 --2- 172.21.15.32:0/3831060674 >> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] conn(0x55620b49e7c0 0x55620b49ebd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:32.563 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:32.561+0000 7fd3acb25f40 1 -- 172.21.15.32:0/3831060674 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:1 12.10 12:09401a81:::601.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105) -- 0x55620b3dcf40 con 0x55620b49e7c0 2024-08-03T20:57:32.564 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:32.561+0000 7fd39fe00640 1 --2- 172.21.15.32:0/3831060674 >> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] conn(0x55620b49e7c0 0x55620b49ebd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:32.564 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:32.562+0000 7fd3a5200640 1 --2- 172.21.15.32:0/3831060674 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fd3840759f0 0x7fd384077f00 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7fd398002040 tx=0x7fd398018040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:57:32.564 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:32.562+0000 7fd39fe00640 1 --2- 172.21.15.32:0/3831060674 >> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] conn(0x55620b49e7c0 0x55620b49ebd0 crc :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:57:32.565 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:32.564+0000 7fd39fe00640 1 -- 172.21.15.32:0/3831060674 <== osd.0 v2:172.21.15.32:6816/3066156116 1 ==== osd_op_reply(1 601.00000000 [create] v105'598 uv572 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fd39000e040 con 0x55620b49e7c0 2024-08-03T20:57:32.566 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:32.564+0000 7fd3acb25f40 1 -- 172.21.15.32:0/3831060674 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:2 12.10 12:09401a81:::601.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e105) -- 0x55620b373fe0 con 0x55620b49e7c0 2024-08-03T20:57:32.568 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:32.566+0000 7fd39fe00640 1 -- 172.21.15.32:0/3831060674 <== osd.0 v2:172.21.15.32:6816/3066156116 2 ==== osd_op_reply(2 601.00000000 [delete] v105'599 uv599 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fd39000e040 con 0x55620b49e7c0 2024-08-03T20:57:32.568 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:32.566+0000 7fd3acb25f40 1 -- 172.21.15.32:0/3831060674 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:3 12.10 12:09401a81:::601.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105) -- 0x55620b4a7950 con 0x55620b49e7c0 2024-08-03T20:57:32.569 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::601.00000000:head# 2024-08-03T20:57:32.570 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:32.568+0000 7fd39fe00640 1 -- 172.21.15.32:0/3831060674 <== osd.0 v2:172.21.15.32:6816/3066156116 3 ==== osd_op_reply(3 601.00000000 [create] v105'600 uv600 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fd39000e040 con 0x55620b49e7c0 2024-08-03T20:57:32.570 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:32.568+0000 7fd3acb25f40 1 -- 172.21.15.32:0/3831060674 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:4 12.10 12:09401a81:::601.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105) -- 0x55620b4a8d00 con 0x55620b49e7c0 2024-08-03T20:57:32.571 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:32.570+0000 7fd39fe00640 1 -- 172.21.15.32:0/3831060674 <== osd.0 v2:172.21.15.32:6816/3066156116 4 ==== osd_op_reply(4 601.00000000 [setxattr (30)] v105'601 uv601 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fd39000e040 con 0x55620b49e7c0 2024-08-03T20:57:32.571 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:32.570+0000 7fd3acb25f40 1 -- 172.21.15.32:0/3831060674 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:5 12.10 12:09401a81:::601.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105) -- 0x55620b4a90a0 con 0x55620b49e7c0 2024-08-03T20:57:32.573 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:32.571+0000 7fd39fe00640 1 -- 172.21.15.32:0/3831060674 <== osd.0 v2:172.21.15.32:6816/3066156116 5 ==== osd_op_reply(5 601.00000000 [setxattr (62)] v105'602 uv602 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fd39000e040 con 0x55620b49e7c0 2024-08-03T20:57:32.573 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:32.571+0000 7fd3acb25f40 1 -- 172.21.15.32:0/3831060674 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:6 12.10 12:09401a81:::601.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105) -- 0x55620b4a7cf0 con 0x55620b49e7c0 2024-08-03T20:57:32.575 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:32.573+0000 7fd39fe00640 1 -- 172.21.15.32:0/3831060674 <== osd.0 v2:172.21.15.32:6816/3066156116 6 ==== osd_op_reply(6 601.00000000 [omap-set-header] v105'603 uv603 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fd39000e040 con 0x55620b49e7c0 2024-08-03T20:57:32.575 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:32.574+0000 7fd3acb25f40 1 -- 172.21.15.32:0/3831060674 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:7 12.10 12:0bd6d154:::602.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105) -- 0x55620b4a80e0 con 0x55620b49e7c0 2024-08-03T20:57:32.577 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:32.576+0000 7fd39fe00640 1 -- 172.21.15.32:0/3831060674 <== osd.0 v2:172.21.15.32:6816/3066156116 7 ==== osd_op_reply(7 602.00000000 [create] v105'604 uv578 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fd39000e040 con 0x55620b49e7c0 2024-08-03T20:57:32.577 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:32.576+0000 7fd3acb25f40 1 -- 172.21.15.32:0/3831060674 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:8 12.10 12:0bd6d154:::602.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e105) -- 0x55620b4a84d0 con 0x55620b49e7c0 2024-08-03T20:57:32.579 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:32.577+0000 7fd39fe00640 1 -- 172.21.15.32:0/3831060674 <== osd.0 v2:172.21.15.32:6816/3066156116 8 ==== osd_op_reply(8 602.00000000 [delete] v105'605 uv605 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fd39000e040 con 0x55620b49e7c0 2024-08-03T20:57:32.579 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:32.577+0000 7fd3acb25f40 1 -- 172.21.15.32:0/3831060674 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:9 12.10 12:0bd6d154:::602.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105) -- 0x55620b4a84d0 con 0x55620b49e7c0 2024-08-03T20:57:32.580 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::602.00000000:head# 2024-08-03T20:57:32.581 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:32.579+0000 7fd39fe00640 1 -- 172.21.15.32:0/3831060674 <== osd.0 v2:172.21.15.32:6816/3066156116 9 ==== osd_op_reply(9 602.00000000 [create] v105'606 uv606 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fd39000e040 con 0x55620b49e7c0 2024-08-03T20:57:32.581 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:32.579+0000 7fd3acb25f40 1 -- 172.21.15.32:0/3831060674 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:10 12.10 12:0bd6d154:::602.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105) -- 0x55620b4a8480 con 0x55620b49e7c0 2024-08-03T20:57:32.582 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:32.581+0000 7fd39fe00640 1 -- 172.21.15.32:0/3831060674 <== osd.0 v2:172.21.15.32:6816/3066156116 10 ==== osd_op_reply(10 602.00000000 [setxattr (30)] v105'607 uv607 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fd39000e040 con 0x55620b49e7c0 2024-08-03T20:57:32.582 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:32.581+0000 7fd3acb25f40 1 -- 172.21.15.32:0/3831060674 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:11 12.10 12:0bd6d154:::602.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105) -- 0x55620b4a8480 con 0x55620b49e7c0 2024-08-03T20:57:32.584 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:32.583+0000 7fd39fe00640 1 -- 172.21.15.32:0/3831060674 <== osd.0 v2:172.21.15.32:6816/3066156116 11 ==== osd_op_reply(11 602.00000000 [setxattr (62)] v105'608 uv608 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fd39000e040 con 0x55620b49e7c0 2024-08-03T20:57:32.584 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:32.583+0000 7fd3acb25f40 1 -- 172.21.15.32:0/3831060674 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:12 12.10 12:0bd6d154:::602.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105) -- 0x55620b4a8480 con 0x55620b49e7c0 2024-08-03T20:57:32.586 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:32.585+0000 7fd39fe00640 1 -- 172.21.15.32:0/3831060674 <== osd.0 v2:172.21.15.32:6816/3066156116 12 ==== osd_op_reply(12 602.00000000 [omap-set-header] v105'609 uv609 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fd39000e040 con 0x55620b49e7c0 2024-08-03T20:57:32.586 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:32.585+0000 7fd3acb25f40 1 -- 172.21.15.32:0/3831060674 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:13 12.10 12:0cf6ad0f:::10000000000.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105) -- 0x55620b4a8480 con 0x55620b49e7c0 2024-08-03T20:57:32.588 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:32.586+0000 7fd39fe00640 1 -- 172.21.15.32:0/3831060674 <== osd.0 v2:172.21.15.32:6816/3066156116 13 ==== osd_op_reply(13 10000000000.00000000 [create] v105'610 uv585 ondisk = -17 ((17) File exists)) ==== 164+0+0 (crc 0 0 0) 0x7fd39000e040 con 0x55620b49e7c0 2024-08-03T20:57:32.588 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:32.587+0000 7fd3acb25f40 1 -- 172.21.15.32:0/3831060674 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:14 12.10 12:0cf6ad0f:::10000000000.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e105) -- 0x55620b4a8480 con 0x55620b49e7c0 2024-08-03T20:57:32.589 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:32.588+0000 7fd39fe00640 1 -- 172.21.15.32:0/3831060674 <== osd.0 v2:172.21.15.32:6816/3066156116 14 ==== osd_op_reply(14 10000000000.00000000 [delete] v105'611 uv611 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7fd39000e040 con 0x55620b49e7c0 2024-08-03T20:57:32.589 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:32.588+0000 7fd3acb25f40 1 -- 172.21.15.32:0/3831060674 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:15 12.10 12:0cf6ad0f:::10000000000.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105) -- 0x55620b4a8480 con 0x55620b49e7c0 2024-08-03T20:57:32.591 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::10000000000.00000000:head# 2024-08-03T20:57:32.591 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:32.590+0000 7fd39fe00640 1 -- 172.21.15.32:0/3831060674 <== osd.0 v2:172.21.15.32:6816/3066156116 15 ==== osd_op_reply(15 10000000000.00000000 [create] v105'612 uv612 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7fd39000e040 con 0x55620b49e7c0 2024-08-03T20:57:32.591 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:32.590+0000 7fd3acb25f40 1 -- 172.21.15.32:0/3831060674 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:16 12.10 12:0cf6ad0f:::10000000000.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105) -- 0x55620b4a8480 con 0x55620b49e7c0 2024-08-03T20:57:32.593 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:32.591+0000 7fd39fe00640 1 -- 172.21.15.32:0/3831060674 <== osd.0 v2:172.21.15.32:6816/3066156116 16 ==== osd_op_reply(16 10000000000.00000000 [setxattr (30)] v105'613 uv613 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7fd39000e040 con 0x55620b49e7c0 2024-08-03T20:57:32.593 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:32.591+0000 7fd3acb25f40 1 -- 172.21.15.32:0/3831060674 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:17 12.10 12:0cf6ad0f:::10000000000.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105) -- 0x55620b4a8480 con 0x55620b49e7c0 2024-08-03T20:57:32.594 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:32.593+0000 7fd39fe00640 1 -- 172.21.15.32:0/3831060674 <== osd.0 v2:172.21.15.32:6816/3066156116 17 ==== osd_op_reply(17 10000000000.00000000 [setxattr (62)] v105'614 uv614 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7fd39000e040 con 0x55620b49e7c0 2024-08-03T20:57:32.594 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:32.593+0000 7fd3acb25f40 1 -- 172.21.15.32:0/3831060674 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:18 12.10 12:0cf6ad0f:::10000000000.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105) -- 0x55620b4a8480 con 0x55620b49e7c0 2024-08-03T20:57:32.596 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:32.594+0000 7fd39fe00640 1 -- 172.21.15.32:0/3831060674 <== osd.0 v2:172.21.15.32:6816/3066156116 18 ==== osd_op_reply(18 10000000000.00000000 [omap-set-header] v105'615 uv615 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7fd39000e040 con 0x55620b49e7c0 2024-08-03T20:57:32.596 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:32.594+0000 7fd3acb25f40 1 -- 172.21.15.32:0/3831060674 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:19 12.10 12:0cf6ad0f:::10000000000.00000000:head [omap-set-vals in=593b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105) -- 0x55620b4a88d0 con 0x55620b49e7c0 2024-08-03T20:57:32.597 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:32.596+0000 7fd39fe00640 1 -- 172.21.15.32:0/3831060674 <== osd.0 v2:172.21.15.32:6816/3066156116 19 ==== osd_op_reply(19 10000000000.00000000 [omap-set-vals] v105'616 uv616 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7fd39000e040 con 0x55620b49e7c0 2024-08-03T20:57:32.598 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:32.596+0000 7fd3acb25f40 1 -- 172.21.15.32:0/3831060674 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:20 12.10 12:0d82a743:::600.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105) -- 0x55620b4a8880 con 0x55620b49e7c0 2024-08-03T20:57:32.599 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:32.598+0000 7fd39fe00640 1 -- 172.21.15.32:0/3831060674 <== osd.0 v2:172.21.15.32:6816/3066156116 20 ==== osd_op_reply(20 600.00000000 [create] v105'617 uv591 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fd39000e040 con 0x55620b49e7c0 2024-08-03T20:57:32.599 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:32.598+0000 7fd3acb25f40 1 -- 172.21.15.32:0/3831060674 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:21 12.10 12:0d82a743:::600.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e105) -- 0x55620b4a8880 con 0x55620b49e7c0 2024-08-03T20:57:32.601 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:32.599+0000 7fd39fe00640 1 -- 172.21.15.32:0/3831060674 <== osd.0 v2:172.21.15.32:6816/3066156116 21 ==== osd_op_reply(21 600.00000000 [delete] v105'618 uv618 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fd39000e040 con 0x55620b49e7c0 2024-08-03T20:57:32.601 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:32.599+0000 7fd3acb25f40 1 -- 172.21.15.32:0/3831060674 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:22 12.10 12:0d82a743:::600.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105) -- 0x55620b4a8880 con 0x55620b49e7c0 2024-08-03T20:57:32.602 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::600.00000000:head# 2024-08-03T20:57:32.603 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:32.601+0000 7fd39fe00640 1 -- 172.21.15.32:0/3831060674 <== osd.0 v2:172.21.15.32:6816/3066156116 22 ==== osd_op_reply(22 600.00000000 [create] v105'619 uv619 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fd39000e040 con 0x55620b49e7c0 2024-08-03T20:57:32.603 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:32.601+0000 7fd3acb25f40 1 -- 172.21.15.32:0/3831060674 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:23 12.10 12:0d82a743:::600.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105) -- 0x55620b49f110 con 0x55620b49e7c0 2024-08-03T20:57:32.604 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:32.603+0000 7fd39fe00640 1 -- 172.21.15.32:0/3831060674 <== osd.0 v2:172.21.15.32:6816/3066156116 23 ==== osd_op_reply(23 600.00000000 [setxattr (30)] v105'620 uv620 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fd39000e040 con 0x55620b49e7c0 2024-08-03T20:57:32.604 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:32.603+0000 7fd3acb25f40 1 -- 172.21.15.32:0/3831060674 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:24 12.10 12:0d82a743:::600.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105) -- 0x55620b49f520 con 0x55620b49e7c0 2024-08-03T20:57:32.606 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:32.604+0000 7fd39fe00640 1 -- 172.21.15.32:0/3831060674 <== osd.0 v2:172.21.15.32:6816/3066156116 24 ==== osd_op_reply(24 600.00000000 [setxattr (62)] v105'621 uv621 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fd39000e040 con 0x55620b49e7c0 2024-08-03T20:57:32.606 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:32.605+0000 7fd3acb25f40 1 -- 172.21.15.32:0/3831060674 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:25 12.10 12:0d82a743:::600.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105) -- 0x55620b49f110 con 0x55620b49e7c0 2024-08-03T20:57:32.607 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:32.606+0000 7fd39fe00640 1 -- 172.21.15.32:0/3831060674 <== osd.0 v2:172.21.15.32:6816/3066156116 25 ==== osd_op_reply(25 600.00000000 [omap-set-header] v105'622 uv622 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fd39000e040 con 0x55620b49e7c0 2024-08-03T20:57:32.608 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:32.606+0000 7fd3acb25f40 1 -- 172.21.15.32:0/3831060674 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:26 12.10 12:0d89b25e:::603.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105) -- 0x55620b49f110 con 0x55620b49e7c0 2024-08-03T20:57:32.609 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:32.608+0000 7fd39fe00640 1 -- 172.21.15.32:0/3831060674 <== osd.0 v2:172.21.15.32:6816/3066156116 26 ==== osd_op_reply(26 603.00000000 [create] v105'623 uv597 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fd39000e040 con 0x55620b49e7c0 2024-08-03T20:57:32.609 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:32.608+0000 7fd3acb25f40 1 -- 172.21.15.32:0/3831060674 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:27 12.10 12:0d89b25e:::603.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e105) -- 0x55620b49f110 con 0x55620b49e7c0 2024-08-03T20:57:32.611 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:32.609+0000 7fd39fe00640 1 -- 172.21.15.32:0/3831060674 <== osd.0 v2:172.21.15.32:6816/3066156116 27 ==== osd_op_reply(27 603.00000000 [delete] v105'624 uv624 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fd39000e040 con 0x55620b49e7c0 2024-08-03T20:57:32.611 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:32.609+0000 7fd3acb25f40 1 -- 172.21.15.32:0/3831060674 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:28 12.10 12:0d89b25e:::603.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105) -- 0x55620b49f110 con 0x55620b49e7c0 2024-08-03T20:57:32.612 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::603.00000000:head# 2024-08-03T20:57:32.612 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:32.611+0000 7fd39fe00640 1 -- 172.21.15.32:0/3831060674 <== osd.0 v2:172.21.15.32:6816/3066156116 28 ==== osd_op_reply(28 603.00000000 [create] v105'625 uv625 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fd39000e040 con 0x55620b49e7c0 2024-08-03T20:57:32.613 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:32.611+0000 7fd3acb25f40 1 -- 172.21.15.32:0/3831060674 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:29 12.10 12:0d89b25e:::603.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105) -- 0x55620b49f110 con 0x55620b49e7c0 2024-08-03T20:57:32.614 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:32.612+0000 7fd39fe00640 1 -- 172.21.15.32:0/3831060674 <== osd.0 v2:172.21.15.32:6816/3066156116 29 ==== osd_op_reply(29 603.00000000 [setxattr (30)] v105'626 uv626 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fd39000e040 con 0x55620b49e7c0 2024-08-03T20:57:32.614 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:32.612+0000 7fd3acb25f40 1 -- 172.21.15.32:0/3831060674 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:30 12.10 12:0d89b25e:::603.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105) -- 0x55620b49f110 con 0x55620b49e7c0 2024-08-03T20:57:32.615 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:32.614+0000 7fd39fe00640 1 -- 172.21.15.32:0/3831060674 <== osd.0 v2:172.21.15.32:6816/3066156116 30 ==== osd_op_reply(30 603.00000000 [setxattr (62)] v105'627 uv627 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fd39000e040 con 0x55620b49e7c0 2024-08-03T20:57:32.615 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:32.614+0000 7fd3acb25f40 1 -- 172.21.15.32:0/3831060674 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:31 12.10 12:0d89b25e:::603.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105) -- 0x55620b49f110 con 0x55620b49e7c0 2024-08-03T20:57:32.617 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:32.615+0000 7fd39fe00640 1 -- 172.21.15.32:0/3831060674 <== osd.0 v2:172.21.15.32:6816/3066156116 31 ==== osd_op_reply(31 603.00000000 [omap-set-header] v105'628 uv628 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fd39000e040 con 0x55620b49e7c0 2024-08-03T20:57:32.617 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:32.615+0000 7fd3acb25f40 1 --2- 172.21.15.32:0/3831060674 >> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] conn(0x55620b4a9950 0x55620b4abd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:32.617 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:32.615+0000 7fd3acb25f40 1 -- 172.21.15.32:0/3831060674 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:32 12.14 12:287034c4:::1.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105) -- 0x55620b49f500 con 0x55620b4a9950 2024-08-03T20:57:32.617 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:32.616+0000 7fd3a5200640 1 --2- 172.21.15.32:0/3831060674 >> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] conn(0x55620b4a9950 0x55620b4abd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:32.618 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:32.617+0000 7fd3a5200640 1 --2- 172.21.15.32:0/3831060674 >> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] conn(0x55620b4a9950 0x55620b4abd70 crc :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:57:32.620 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:32.618+0000 7fd3a5200640 1 -- 172.21.15.32:0/3831060674 <== osd.5 v2:172.21.15.118:6810/2724891770 1 ==== osd_op_reply(32 1.00000000.inode [create] v105'442 uv425 ondisk = -17 ((17) File exists)) ==== 160+0+0 (crc 0 0 0) 0x7fd39800ada0 con 0x55620b4a9950 2024-08-03T20:57:32.620 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:32.619+0000 7fd3acb25f40 1 -- 172.21.15.32:0/3831060674 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:33 12.14 12:287034c4:::1.00000000.inode:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e105) -- 0x55620b4ac2b0 con 0x55620b4a9950 2024-08-03T20:57:32.622 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:32.621+0000 7fd3a5200640 1 -- 172.21.15.32:0/3831060674 <== osd.5 v2:172.21.15.118:6810/2724891770 2 ==== osd_op_reply(33 1.00000000.inode [delete] v105'443 uv443 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7fd398023420 con 0x55620b4a9950 2024-08-03T20:57:32.622 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:32.621+0000 7fd3acb25f40 1 -- 172.21.15.32:0/3831060674 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:34 12.14 12:287034c4:::1.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105) -- 0x55620b4ac650 con 0x55620b4a9950 2024-08-03T20:57:32.623 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:32.622+0000 7fd3a5200640 1 -- 172.21.15.32:0/3831060674 <== osd.5 v2:172.21.15.118:6810/2724891770 3 ==== osd_op_reply(34 1.00000000.inode [create] v105'444 uv444 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7fd398023420 con 0x55620b4a9950 2024-08-03T20:57:32.623 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::1.00000000.inode:head# 2024-08-03T20:57:32.624 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:32.622+0000 7fd3acb25f40 1 -- 172.21.15.32:0/3831060674 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:35 12.14 12:287034c4:::1.00000000.inode:head [write 0~582 in=582b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105) -- 0x55620b4a8800 con 0x55620b4a9950 2024-08-03T20:57:32.625 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:32.624+0000 7fd3a5200640 1 -- 172.21.15.32:0/3831060674 <== osd.5 v2:172.21.15.118:6810/2724891770 4 ==== osd_op_reply(35 1.00000000.inode [write 0~582] v105'445 uv445 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7fd398023420 con 0x55620b4a9950 2024-08-03T20:57:32.625 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:32.624+0000 7fd3acb25f40 1 -- 172.21.15.32:0/3831060674 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:36 12.14 12:287034c4:::1.00000000.inode:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105) -- 0x55620b4acef0 con 0x55620b4a9950 2024-08-03T20:57:32.627 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:32.625+0000 7fd3a5200640 1 -- 172.21.15.32:0/3831060674 <== osd.5 v2:172.21.15.118:6810/2724891770 5 ==== osd_op_reply(36 1.00000000.inode [omap-set-header] v105'446 uv446 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7fd398023420 con 0x55620b4a9950 2024-08-03T20:57:32.628 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:32.626+0000 7fd3acb25f40 1 -- 172.21.15.32:0/3831060674 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:37 12.14 12:292cf221:::200.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105) -- 0x55620b4ad2e0 con 0x55620b4a9950 2024-08-03T20:57:32.628 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:32.627+0000 7fd3a5200640 1 -- 172.21.15.32:0/3831060674 <== osd.5 v2:172.21.15.118:6810/2724891770 6 ==== osd_op_reply(37 200.00000000 [create] v105'447 uv430 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fd398023420 con 0x55620b4a9950 2024-08-03T20:57:32.628 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:32.627+0000 7fd3acb25f40 1 -- 172.21.15.32:0/3831060674 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:38 12.14 12:292cf221:::200.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e105) -- 0x55620b4ad6d0 con 0x55620b4a9950 2024-08-03T20:57:32.630 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:32.628+0000 7fd3a5200640 1 -- 172.21.15.32:0/3831060674 <== osd.5 v2:172.21.15.118:6810/2724891770 7 ==== osd_op_reply(38 200.00000000 [delete] v105'448 uv448 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fd398023420 con 0x55620b4a9950 2024-08-03T20:57:32.630 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:32.629+0000 7fd3acb25f40 1 -- 172.21.15.32:0/3831060674 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:39 12.14 12:292cf221:::200.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105) -- 0x55620b4ad6d0 con 0x55620b4a9950 2024-08-03T20:57:32.631 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:32.630+0000 7fd3a5200640 1 -- 172.21.15.32:0/3831060674 <== osd.5 v2:172.21.15.118:6810/2724891770 8 ==== osd_op_reply(39 200.00000000 [create] v105'449 uv449 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fd398023420 con 0x55620b4a9950 2024-08-03T20:57:32.631 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::200.00000000:head# 2024-08-03T20:57:32.632 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:32.630+0000 7fd3acb25f40 1 -- 172.21.15.32:0/3831060674 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:40 12.14 12:292cf221:::200.00000000:head [write 0~90 in=90b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105) -- 0x55620b4adac0 con 0x55620b4a9950 2024-08-03T20:57:32.633 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:32.632+0000 7fd3a5200640 1 -- 172.21.15.32:0/3831060674 <== osd.5 v2:172.21.15.118:6810/2724891770 9 ==== osd_op_reply(40 200.00000000 [write 0~90] v105'450 uv450 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fd398023420 con 0x55620b4a9950 2024-08-03T20:57:32.633 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:32.632+0000 7fd3acb25f40 1 -- 172.21.15.32:0/3831060674 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:41 12.14 12:292cf221:::200.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105) -- 0x55620b4ad800 con 0x55620b4a9950 2024-08-03T20:57:32.634 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:32.633+0000 7fd3a5200640 1 -- 172.21.15.32:0/3831060674 <== osd.5 v2:172.21.15.118:6810/2724891770 10 ==== osd_op_reply(41 200.00000000 [omap-set-header] v105'451 uv451 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fd398023420 con 0x55620b4a9950 2024-08-03T20:57:32.634 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:32.633+0000 7fd3acb25f40 1 -- 172.21.15.32:0/3831060674 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:42 12.14 12:2e2fa760:::200.00000001:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105) -- 0x55620b4ad800 con 0x55620b4a9950 2024-08-03T20:57:32.636 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:32.634+0000 7fd3a5200640 1 -- 172.21.15.32:0/3831060674 <== osd.5 v2:172.21.15.118:6810/2724891770 11 ==== osd_op_reply(42 200.00000001 [create] v105'452 uv435 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fd398023420 con 0x55620b4a9950 2024-08-03T20:57:32.636 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:32.635+0000 7fd3acb25f40 1 -- 172.21.15.32:0/3831060674 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:43 12.14 12:2e2fa760:::200.00000001:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e105) -- 0x55620b4ad7b0 con 0x55620b4a9950 2024-08-03T20:57:32.637 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:32.636+0000 7fd3a5200640 1 -- 172.21.15.32:0/3831060674 <== osd.5 v2:172.21.15.118:6810/2724891770 12 ==== osd_op_reply(43 200.00000001 [delete] v105'453 uv453 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fd398023420 con 0x55620b4a9950 2024-08-03T20:57:32.637 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:32.636+0000 7fd3acb25f40 1 -- 172.21.15.32:0/3831060674 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:44 12.14 12:2e2fa760:::200.00000001:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105) -- 0x55620b4ad7b0 con 0x55620b4a9950 2024-08-03T20:57:32.639 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::200.00000001:head# 2024-08-03T20:57:32.639 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:32.638+0000 7fd3a5200640 1 -- 172.21.15.32:0/3831060674 <== osd.5 v2:172.21.15.118:6810/2724891770 13 ==== osd_op_reply(44 200.00000001 [create] v105'454 uv454 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fd398023420 con 0x55620b4a9950 2024-08-03T20:57:32.639 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:32.638+0000 7fd3acb25f40 1 -- 172.21.15.32:0/3831060674 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:45 12.14 12:2e2fa760:::200.00000001:head [write 0~18118 in=18118b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105) -- 0x55620b4ad7b0 con 0x55620b4a9950 2024-08-03T20:57:32.641 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:32.639+0000 7fd3a5200640 1 -- 172.21.15.32:0/3831060674 <== osd.5 v2:172.21.15.118:6810/2724891770 14 ==== osd_op_reply(45 200.00000001 [write 0~18118] v105'455 uv455 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fd398023420 con 0x55620b4a9950 2024-08-03T20:57:32.641 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:32.639+0000 7fd3acb25f40 1 -- 172.21.15.32:0/3831060674 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:46 12.14 12:2e2fa760:::200.00000001:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105) -- 0x55620b4ad7b0 con 0x55620b4a9950 2024-08-03T20:57:32.642 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:32.641+0000 7fd3a5200640 1 -- 172.21.15.32:0/3831060674 <== osd.5 v2:172.21.15.118:6810/2724891770 15 ==== osd_op_reply(46 200.00000001 [omap-set-header] v105'456 uv456 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fd398023420 con 0x55620b4a9950 2024-08-03T20:57:32.642 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:32.641+0000 7fd3acb25f40 1 -- 172.21.15.32:0/3831060674 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:47 12.14 12:2f55791f:::606.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105) -- 0x55620b4ad7b0 con 0x55620b4a9950 2024-08-03T20:57:32.644 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:32.642+0000 7fd3a5200640 1 -- 172.21.15.32:0/3831060674 <== osd.5 v2:172.21.15.118:6810/2724891770 16 ==== osd_op_reply(47 606.00000000 [create] v105'457 uv441 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fd398023420 con 0x55620b4a9950 2024-08-03T20:57:32.644 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:32.642+0000 7fd3acb25f40 1 -- 172.21.15.32:0/3831060674 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:48 12.14 12:2f55791f:::606.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e105) -- 0x55620b4ad7b0 con 0x55620b4a9950 2024-08-03T20:57:32.645 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:32.644+0000 7fd3a5200640 1 -- 172.21.15.32:0/3831060674 <== osd.5 v2:172.21.15.118:6810/2724891770 17 ==== osd_op_reply(48 606.00000000 [delete] v105'458 uv458 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fd398023420 con 0x55620b4a9950 2024-08-03T20:57:32.645 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:32.644+0000 7fd3acb25f40 1 -- 172.21.15.32:0/3831060674 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:49 12.14 12:2f55791f:::606.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105) -- 0x55620b4ad7b0 con 0x55620b4a9950 2024-08-03T20:57:32.646 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::606.00000000:head# 2024-08-03T20:57:32.646 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:32.645+0000 7fd3a5200640 1 -- 172.21.15.32:0/3831060674 <== osd.5 v2:172.21.15.118:6810/2724891770 18 ==== osd_op_reply(49 606.00000000 [create] v105'459 uv459 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fd398023420 con 0x55620b4a9950 2024-08-03T20:57:32.647 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:32.645+0000 7fd3acb25f40 1 -- 172.21.15.32:0/3831060674 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:50 12.14 12:2f55791f:::606.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105) -- 0x55620b4ad7b0 con 0x55620b4a9950 2024-08-03T20:57:32.647 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:32.646+0000 7fd3a5200640 1 -- 172.21.15.32:0/3831060674 <== osd.5 v2:172.21.15.118:6810/2724891770 19 ==== osd_op_reply(50 606.00000000 [setxattr (30)] v105'460 uv460 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fd398023420 con 0x55620b4a9950 2024-08-03T20:57:32.648 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:32.646+0000 7fd3acb25f40 1 -- 172.21.15.32:0/3831060674 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:51 12.14 12:2f55791f:::606.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105) -- 0x55620b4ad7b0 con 0x55620b4a9950 2024-08-03T20:57:32.649 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:32.647+0000 7fd3a5200640 1 -- 172.21.15.32:0/3831060674 <== osd.5 v2:172.21.15.118:6810/2724891770 20 ==== osd_op_reply(51 606.00000000 [setxattr (62)] v105'461 uv461 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fd398023420 con 0x55620b4a9950 2024-08-03T20:57:32.649 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:32.647+0000 7fd3acb25f40 1 -- 172.21.15.32:0/3831060674 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:52 12.14 12:2f55791f:::606.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105) -- 0x55620b4ad7b0 con 0x55620b4a9950 2024-08-03T20:57:32.650 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:32.649+0000 7fd3a5200640 1 -- 172.21.15.32:0/3831060674 <== osd.5 v2:172.21.15.118:6810/2724891770 21 ==== osd_op_reply(52 606.00000000 [omap-set-header] v105'462 uv462 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fd398023420 con 0x55620b4a9950 2024-08-03T20:57:32.650 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:32.649+0000 7fd3acb25f40 1 -- 172.21.15.32:0/3831060674 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:53 12.1c 12:3ed09add:::607.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105) -- 0x55620b4ad7b0 con 0x55620b49e7c0 2024-08-03T20:57:32.652 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:32.650+0000 7fd39fe00640 1 -- 172.21.15.32:0/3831060674 <== osd.0 v2:172.21.15.32:6816/3066156116 32 ==== osd_op_reply(53 607.00000000 [create] v105'117 uv116 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fd39000e040 con 0x55620b49e7c0 2024-08-03T20:57:32.652 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:32.650+0000 7fd3acb25f40 1 -- 172.21.15.32:0/3831060674 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:54 12.1c 12:3ed09add:::607.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e105) -- 0x55620b4ad7b0 con 0x55620b49e7c0 2024-08-03T20:57:32.653 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:32.652+0000 7fd39fe00640 1 -- 172.21.15.32:0/3831060674 <== osd.0 v2:172.21.15.32:6816/3066156116 33 ==== osd_op_reply(54 607.00000000 [delete] v105'118 uv118 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fd39000e040 con 0x55620b49e7c0 2024-08-03T20:57:32.653 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:32.652+0000 7fd3acb25f40 1 -- 172.21.15.32:0/3831060674 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:55 12.1c 12:3ed09add:::607.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105) -- 0x55620b4ad7b0 con 0x55620b49e7c0 2024-08-03T20:57:32.654 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::607.00000000:head# 2024-08-03T20:57:32.655 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:32.653+0000 7fd39fe00640 1 -- 172.21.15.32:0/3831060674 <== osd.0 v2:172.21.15.32:6816/3066156116 34 ==== osd_op_reply(55 607.00000000 [create] v105'119 uv119 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fd39000e040 con 0x55620b49e7c0 2024-08-03T20:57:32.655 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:32.653+0000 7fd3acb25f40 1 -- 172.21.15.32:0/3831060674 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:56 12.1c 12:3ed09add:::607.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105) -- 0x55620b4ad7b0 con 0x55620b49e7c0 2024-08-03T20:57:32.656 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:32.654+0000 7fd39fe00640 1 -- 172.21.15.32:0/3831060674 <== osd.0 v2:172.21.15.32:6816/3066156116 35 ==== osd_op_reply(56 607.00000000 [setxattr (30)] v105'120 uv120 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fd39000e040 con 0x55620b49e7c0 2024-08-03T20:57:32.656 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:32.654+0000 7fd3acb25f40 1 -- 172.21.15.32:0/3831060674 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:57 12.1c 12:3ed09add:::607.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105) -- 0x55620b4ad7b0 con 0x55620b49e7c0 2024-08-03T20:57:32.657 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:32.655+0000 7fd39fe00640 1 -- 172.21.15.32:0/3831060674 <== osd.0 v2:172.21.15.32:6816/3066156116 36 ==== osd_op_reply(57 607.00000000 [setxattr (62)] v105'121 uv121 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fd39000e040 con 0x55620b49e7c0 2024-08-03T20:57:32.657 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:32.656+0000 7fd3acb25f40 1 -- 172.21.15.32:0/3831060674 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:58 12.1c 12:3ed09add:::607.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105) -- 0x55620b4ad7b0 con 0x55620b49e7c0 2024-08-03T20:57:32.658 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:32.657+0000 7fd39fe00640 1 -- 172.21.15.32:0/3831060674 <== osd.0 v2:172.21.15.32:6816/3066156116 37 ==== osd_op_reply(58 607.00000000 [omap-set-header] v105'122 uv122 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fd39000e040 con 0x55620b49e7c0 2024-08-03T20:57:32.658 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:32.657+0000 7fd3acb25f40 1 --2- 172.21.15.32:0/3831060674 >> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] conn(0x55620b4ae0b0 0x55620b4ae4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:32.658 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:32.657+0000 7fd3acb25f40 1 -- 172.21.15.32:0/3831060674 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:59 12.6 12:654134d2:::mds0_openfiles.0:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105) -- 0x55620b4b0c10 con 0x55620b4ae0b0 2024-08-03T20:57:32.659 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:32.657+0000 7fd39fe00640 1 --2- 172.21.15.32:0/3831060674 >> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] conn(0x55620b4ae0b0 0x55620b4ae4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:32.659 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:32.658+0000 7fd39fe00640 1 --2- 172.21.15.32:0/3831060674 >> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] conn(0x55620b4ae0b0 0x55620b4ae4c0 crc :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:57:32.660 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:32.659+0000 7fd39fe00640 1 -- 172.21.15.32:0/3831060674 <== osd.1 v2:172.21.15.32:6824/1551241759 1 ==== osd_op_reply(59 mds0_openfiles.0 [create] v105'94 uv93 ondisk = -17 ((17) File exists)) ==== 160+0+0 (crc 0 0 0) 0x7fd39000e040 con 0x55620b4ae0b0 2024-08-03T20:57:32.661 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:32.659+0000 7fd3acb25f40 1 -- 172.21.15.32:0/3831060674 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:60 12.6 12:654134d2:::mds0_openfiles.0:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e105) -- 0x55620b4b0c10 con 0x55620b4ae0b0 2024-08-03T20:57:32.662 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:32.660+0000 7fd39fe00640 1 -- 172.21.15.32:0/3831060674 <== osd.1 v2:172.21.15.32:6824/1551241759 2 ==== osd_op_reply(60 mds0_openfiles.0 [delete] v105'95 uv95 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7fd39000e040 con 0x55620b4ae0b0 2024-08-03T20:57:32.662 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:32.660+0000 7fd3acb25f40 1 -- 172.21.15.32:0/3831060674 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:61 12.6 12:654134d2:::mds0_openfiles.0:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105) -- 0x55620b4b0c90 con 0x55620b4ae0b0 2024-08-03T20:57:32.663 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds0_openfiles.0:head# 2024-08-03T20:57:32.663 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:32.661+0000 7fd39fe00640 1 -- 172.21.15.32:0/3831060674 <== osd.1 v2:172.21.15.32:6824/1551241759 3 ==== osd_op_reply(61 mds0_openfiles.0 [create] v105'96 uv96 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7fd39000e040 con 0x55620b4ae0b0 2024-08-03T20:57:32.663 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:32.661+0000 7fd3acb25f40 1 -- 172.21.15.32:0/3831060674 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:62 12.6 12:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105) -- 0x55620b4b0c90 con 0x55620b4ae0b0 2024-08-03T20:57:32.664 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:32.663+0000 7fd39fe00640 1 -- 172.21.15.32:0/3831060674 <== osd.1 v2:172.21.15.32:6824/1551241759 4 ==== osd_op_reply(62 mds0_openfiles.0 [omap-set-header] v105'97 uv97 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7fd39000e040 con 0x55620b4ae0b0 2024-08-03T20:57:32.664 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:32.663+0000 7fd3acb25f40 1 -- 172.21.15.32:0/3831060674 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:63 12.e 12:717a0223:::608.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105) -- 0x55620b4b0c90 con 0x55620b49e7c0 2024-08-03T20:57:32.666 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:32.665+0000 7fd39fe00640 1 -- 172.21.15.32:0/3831060674 <== osd.0 v2:172.21.15.32:6816/3066156116 38 ==== osd_op_reply(63 608.00000000 [create] v105'117 uv116 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fd39000e040 con 0x55620b49e7c0 2024-08-03T20:57:32.666 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:32.665+0000 7fd3acb25f40 1 -- 172.21.15.32:0/3831060674 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:64 12.e 12:717a0223:::608.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e105) -- 0x55620b4b0c90 con 0x55620b49e7c0 2024-08-03T20:57:32.668 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:32.667+0000 7fd39fe00640 1 -- 172.21.15.32:0/3831060674 <== osd.0 v2:172.21.15.32:6816/3066156116 39 ==== osd_op_reply(64 608.00000000 [delete] v105'118 uv118 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fd39000e040 con 0x55620b49e7c0 2024-08-03T20:57:32.668 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:32.667+0000 7fd3acb25f40 1 -- 172.21.15.32:0/3831060674 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:65 12.e 12:717a0223:::608.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105) -- 0x55620b4b0c90 con 0x55620b49e7c0 2024-08-03T20:57:32.669 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:32.668+0000 7fd39fe00640 1 -- 172.21.15.32:0/3831060674 <== osd.0 v2:172.21.15.32:6816/3066156116 40 ==== osd_op_reply(65 608.00000000 [create] v105'119 uv119 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fd39000e040 con 0x55620b49e7c0 2024-08-03T20:57:32.670 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::608.00000000:head# 2024-08-03T20:57:32.670 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:32.668+0000 7fd3acb25f40 1 -- 172.21.15.32:0/3831060674 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:66 12.e 12:717a0223:::608.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105) -- 0x55620b4b0c90 con 0x55620b49e7c0 2024-08-03T20:57:32.671 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:32.670+0000 7fd39fe00640 1 -- 172.21.15.32:0/3831060674 <== osd.0 v2:172.21.15.32:6816/3066156116 41 ==== osd_op_reply(66 608.00000000 [setxattr (30)] v105'120 uv120 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fd39000e040 con 0x55620b49e7c0 2024-08-03T20:57:32.672 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:32.670+0000 7fd3acb25f40 1 -- 172.21.15.32:0/3831060674 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:67 12.e 12:717a0223:::608.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105) -- 0x55620b4b2130 con 0x55620b49e7c0 2024-08-03T20:57:32.673 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:32.671+0000 7fd39fe00640 1 -- 172.21.15.32:0/3831060674 <== osd.0 v2:172.21.15.32:6816/3066156116 42 ==== osd_op_reply(67 608.00000000 [setxattr (62)] v105'121 uv121 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fd39000e040 con 0x55620b49e7c0 2024-08-03T20:57:32.673 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:32.672+0000 7fd3acb25f40 1 -- 172.21.15.32:0/3831060674 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:68 12.e 12:717a0223:::608.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105) -- 0x55620b4b0c90 con 0x55620b49e7c0 2024-08-03T20:57:32.675 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:32.673+0000 7fd39fe00640 1 -- 172.21.15.32:0/3831060674 <== osd.0 v2:172.21.15.32:6816/3066156116 43 ==== osd_op_reply(68 608.00000000 [omap-set-header] v105'122 uv122 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fd39000e040 con 0x55620b49e7c0 2024-08-03T20:57:32.675 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:32.673+0000 7fd3acb25f40 1 --2- 172.21.15.32:0/3831060674 >> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] conn(0x55620b4b1590 0x55620b4b19a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:32.675 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:32.673+0000 7fd3acb25f40 1 -- 172.21.15.32:0/3831060674 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:69 12.15 12:a93a17c2:::604.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105) -- 0x55620b4b40f0 con 0x55620b4b1590 2024-08-03T20:57:32.675 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:32.674+0000 7fd3a5c00640 1 --2- 172.21.15.32:0/3831060674 >> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] conn(0x55620b4b1590 0x55620b4b19a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:32.675 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:32.674+0000 7fd3a5c00640 1 --2- 172.21.15.32:0/3831060674 >> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] conn(0x55620b4b1590 0x55620b4b19a0 crc :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:57:32.677 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:32.675+0000 7fd3a5c00640 1 -- 172.21.15.32:0/3831060674 <== osd.6 v2:172.21.15.118:6825/801947386 1 ==== osd_op_reply(69 604.00000000 [create] v105'216 uv210 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fd3a0070070 con 0x55620b4b1590 2024-08-03T20:57:32.677 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:32.675+0000 7fd3acb25f40 1 -- 172.21.15.32:0/3831060674 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:70 12.15 12:a93a17c2:::604.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e105) -- 0x55620b4b4490 con 0x55620b4b1590 2024-08-03T20:57:32.678 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:32.677+0000 7fd3a5c00640 1 -- 172.21.15.32:0/3831060674 <== osd.6 v2:172.21.15.118:6825/801947386 2 ==== osd_op_reply(70 604.00000000 [delete] v105'217 uv217 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fd3a0070070 con 0x55620b4b1590 2024-08-03T20:57:32.678 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:32.677+0000 7fd3acb25f40 1 -- 172.21.15.32:0/3831060674 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:71 12.15 12:a93a17c2:::604.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105) -- 0x55620b4b48b0 con 0x55620b4b1590 2024-08-03T20:57:32.679 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::604.00000000:head# 2024-08-03T20:57:32.680 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:32.678+0000 7fd3a5c00640 1 -- 172.21.15.32:0/3831060674 <== osd.6 v2:172.21.15.118:6825/801947386 3 ==== osd_op_reply(71 604.00000000 [create] v105'218 uv218 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fd3a0070070 con 0x55620b4b1590 2024-08-03T20:57:32.680 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:32.678+0000 7fd3acb25f40 1 -- 172.21.15.32:0/3831060674 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:72 12.15 12:a93a17c2:::604.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105) -- 0x55620b4acaf0 con 0x55620b4b1590 2024-08-03T20:57:32.680 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:32.679+0000 7fd3a5c00640 1 -- 172.21.15.32:0/3831060674 <== osd.6 v2:172.21.15.118:6825/801947386 4 ==== osd_op_reply(72 604.00000000 [setxattr (30)] v105'219 uv219 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fd3a0070070 con 0x55620b4b1590 2024-08-03T20:57:32.681 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:32.679+0000 7fd3acb25f40 1 -- 172.21.15.32:0/3831060674 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:73 12.15 12:a93a17c2:::604.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105) -- 0x55620b4b6240 con 0x55620b4b1590 2024-08-03T20:57:32.682 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:32.680+0000 7fd3a5c00640 1 -- 172.21.15.32:0/3831060674 <== osd.6 v2:172.21.15.118:6825/801947386 5 ==== osd_op_reply(73 604.00000000 [setxattr (62)] v105'220 uv220 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fd3a0070070 con 0x55620b4b1590 2024-08-03T20:57:32.682 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:32.680+0000 7fd3acb25f40 1 -- 172.21.15.32:0/3831060674 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:74 12.15 12:a93a17c2:::604.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105) -- 0x55620b4b5190 con 0x55620b4b1590 2024-08-03T20:57:32.683 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:32.682+0000 7fd3a5c00640 1 -- 172.21.15.32:0/3831060674 <== osd.6 v2:172.21.15.118:6825/801947386 6 ==== osd_op_reply(74 604.00000000 [omap-set-header] v105'221 uv221 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fd3a0070070 con 0x55620b4b1590 2024-08-03T20:57:32.683 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:32.682+0000 7fd3acb25f40 1 -- 172.21.15.32:0/3831060674 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:75 12.15 12:aa448500:::500.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105) -- 0x55620b4b55a0 con 0x55620b4b1590 2024-08-03T20:57:32.684 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:32.683+0000 7fd3a5c00640 1 -- 172.21.15.32:0/3831060674 <== osd.6 v2:172.21.15.118:6825/801947386 7 ==== osd_op_reply(75 500.00000000 [create] v105'222 uv215 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fd3a0070070 con 0x55620b4b1590 2024-08-03T20:57:32.684 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:32.683+0000 7fd3acb25f40 1 -- 172.21.15.32:0/3831060674 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:76 12.15 12:aa448500:::500.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e105) -- 0x55620b4b5940 con 0x55620b4b1590 2024-08-03T20:57:32.686 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:32.684+0000 7fd3a5c00640 1 -- 172.21.15.32:0/3831060674 <== osd.6 v2:172.21.15.118:6825/801947386 8 ==== osd_op_reply(76 500.00000000 [delete] v105'223 uv223 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fd3a0070070 con 0x55620b4b1590 2024-08-03T20:57:32.686 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:32.684+0000 7fd3acb25f40 1 -- 172.21.15.32:0/3831060674 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:77 12.15 12:aa448500:::500.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105) -- 0x55620b4b5940 con 0x55620b4b1590 2024-08-03T20:57:32.687 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::500.00000000:head# 2024-08-03T20:57:32.687 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:32.685+0000 7fd3a5c00640 1 -- 172.21.15.32:0/3831060674 <== osd.6 v2:172.21.15.118:6825/801947386 9 ==== osd_op_reply(77 500.00000000 [create] v105'224 uv224 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fd3a0070070 con 0x55620b4b1590 2024-08-03T20:57:32.687 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:32.685+0000 7fd3acb25f40 1 -- 172.21.15.32:0/3831060674 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:78 12.15 12:aa448500:::500.00000000:head [write 0~90 in=90b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105) -- 0x55620b4b5940 con 0x55620b4b1590 2024-08-03T20:57:32.688 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:32.687+0000 7fd3a5c00640 1 -- 172.21.15.32:0/3831060674 <== osd.6 v2:172.21.15.118:6825/801947386 10 ==== osd_op_reply(78 500.00000000 [write 0~90] v105'225 uv225 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fd3a0070070 con 0x55620b4b1590 2024-08-03T20:57:32.688 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:32.687+0000 7fd3acb25f40 1 -- 172.21.15.32:0/3831060674 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:79 12.15 12:aa448500:::500.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105) -- 0x55620b4b5940 con 0x55620b4b1590 2024-08-03T20:57:32.689 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:32.688+0000 7fd3a5c00640 1 -- 172.21.15.32:0/3831060674 <== osd.6 v2:172.21.15.118:6825/801947386 11 ==== osd_op_reply(79 500.00000000 [omap-set-header] v105'226 uv226 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fd3a0070070 con 0x55620b4b1590 2024-08-03T20:57:32.689 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:32.688+0000 7fd3acb25f40 1 --2- 172.21.15.32:0/3831060674 >> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] conn(0x55620b4b5d30 0x55620b4b8b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:32.689 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:32.688+0000 7fd3acb25f40 1 -- 172.21.15.32:0/3831060674 --> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] -- osd_op(unknown.0.0:80 12.d 12:b50e409b:::mds_snaptable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105) -- 0x55620b4b9090 con 0x55620b4b5d30 2024-08-03T20:57:32.690 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:32.688+0000 7fd3a5200640 1 --2- 172.21.15.32:0/3831060674 >> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] conn(0x55620b4b5d30 0x55620b4b8b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:32.690 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:32.689+0000 7fd3a5200640 1 --2- 172.21.15.32:0/3831060674 >> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] conn(0x55620b4b5d30 0x55620b4b8b50 crc :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:57:32.692 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:32.691+0000 7fd3a5200640 1 -- 172.21.15.32:0/3831060674 <== osd.7 v2:172.21.15.118:6818/653000048 1 ==== osd_op_reply(80 mds_snaptable [create] v105'97 uv96 ondisk = -17 ((17) File exists)) ==== 157+0+0 (crc 0 0 0) 0x7fd398023420 con 0x55620b4b5d30 2024-08-03T20:57:32.692 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:32.691+0000 7fd3acb25f40 1 -- 172.21.15.32:0/3831060674 --> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] -- osd_op(unknown.0.0:81 12.d 12:b50e409b:::mds_snaptable:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e105) -- 0x55620b4b9090 con 0x55620b4b5d30 2024-08-03T20:57:32.693 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:32.692+0000 7fd3a5200640 1 -- 172.21.15.32:0/3831060674 <== osd.7 v2:172.21.15.118:6818/653000048 2 ==== osd_op_reply(81 mds_snaptable [delete] v105'98 uv98 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7fd398023420 con 0x55620b4b5d30 2024-08-03T20:57:32.693 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:32.692+0000 7fd3acb25f40 1 -- 172.21.15.32:0/3831060674 --> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] -- osd_op(unknown.0.0:82 12.d 12:b50e409b:::mds_snaptable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105) -- 0x55620b4b94e0 con 0x55620b4b5d30 2024-08-03T20:57:32.694 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds_snaptable:head# 2024-08-03T20:57:32.695 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:32.693+0000 7fd3a5200640 1 -- 172.21.15.32:0/3831060674 <== osd.7 v2:172.21.15.118:6818/653000048 3 ==== osd_op_reply(82 mds_snaptable [create] v105'99 uv99 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7fd398023420 con 0x55620b4b5d30 2024-08-03T20:57:32.695 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:32.693+0000 7fd3acb25f40 1 -- 172.21.15.32:0/3831060674 --> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] -- osd_op(unknown.0.0:83 12.d 12:b50e409b:::mds_snaptable:head [write 0~70 in=70b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105) -- 0x55620b4b94e0 con 0x55620b4b5d30 2024-08-03T20:57:32.696 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:32.695+0000 7fd3a5200640 1 -- 172.21.15.32:0/3831060674 <== osd.7 v2:172.21.15.118:6818/653000048 4 ==== osd_op_reply(83 mds_snaptable [write 0~70] v105'100 uv100 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7fd398023420 con 0x55620b4b5d30 2024-08-03T20:57:32.696 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:32.695+0000 7fd3acb25f40 1 -- 172.21.15.32:0/3831060674 --> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] -- osd_op(unknown.0.0:84 12.d 12:b50e409b:::mds_snaptable:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105) -- 0x55620b4b94e0 con 0x55620b4b5d30 2024-08-03T20:57:32.698 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:32.696+0000 7fd3a5200640 1 -- 172.21.15.32:0/3831060674 <== osd.7 v2:172.21.15.118:6818/653000048 5 ==== osd_op_reply(84 mds_snaptable [omap-set-header] v105'101 uv101 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7fd398023420 con 0x55620b4b5d30 2024-08-03T20:57:32.698 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:32.696+0000 7fd3acb25f40 1 -- 172.21.15.32:0/3831060674 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:85 12.1d 12:b871830b:::605.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105) -- 0x55620b4b94e0 con 0x55620b49e7c0 2024-08-03T20:57:32.699 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:32.698+0000 7fd39fe00640 1 -- 172.21.15.32:0/3831060674 <== osd.0 v2:172.21.15.32:6816/3066156116 44 ==== osd_op_reply(85 605.00000000 [create] v105'117 uv116 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fd39000e040 con 0x55620b49e7c0 2024-08-03T20:57:32.699 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:32.698+0000 7fd3acb25f40 1 -- 172.21.15.32:0/3831060674 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:86 12.1d 12:b871830b:::605.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e105) -- 0x55620b4b94e0 con 0x55620b49e7c0 2024-08-03T20:57:32.700 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:32.699+0000 7fd39fe00640 1 -- 172.21.15.32:0/3831060674 <== osd.0 v2:172.21.15.32:6816/3066156116 45 ==== osd_op_reply(86 605.00000000 [delete] v105'118 uv118 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fd39000e040 con 0x55620b49e7c0 2024-08-03T20:57:32.700 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:32.699+0000 7fd3acb25f40 1 -- 172.21.15.32:0/3831060674 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:87 12.1d 12:b871830b:::605.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105) -- 0x55620b4b94e0 con 0x55620b49e7c0 2024-08-03T20:57:32.702 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::605.00000000:head# 2024-08-03T20:57:32.702 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:32.700+0000 7fd39fe00640 1 -- 172.21.15.32:0/3831060674 <== osd.0 v2:172.21.15.32:6816/3066156116 46 ==== osd_op_reply(87 605.00000000 [create] v105'119 uv119 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fd39000e040 con 0x55620b49e7c0 2024-08-03T20:57:32.702 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:32.700+0000 7fd3acb25f40 1 -- 172.21.15.32:0/3831060674 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:88 12.1d 12:b871830b:::605.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105) -- 0x55620b4ba4f0 con 0x55620b49e7c0 2024-08-03T20:57:32.703 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:32.702+0000 7fd39fe00640 1 -- 172.21.15.32:0/3831060674 <== osd.0 v2:172.21.15.32:6816/3066156116 47 ==== osd_op_reply(88 605.00000000 [setxattr (30)] v105'120 uv120 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fd39000e040 con 0x55620b49e7c0 2024-08-03T20:57:32.703 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:32.702+0000 7fd3acb25f40 1 -- 172.21.15.32:0/3831060674 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:89 12.1d 12:b871830b:::605.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105) -- 0x55620b4ba4f0 con 0x55620b49e7c0 2024-08-03T20:57:32.704 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:32.703+0000 7fd39fe00640 1 -- 172.21.15.32:0/3831060674 <== osd.0 v2:172.21.15.32:6816/3066156116 48 ==== osd_op_reply(89 605.00000000 [setxattr (62)] v105'121 uv121 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fd39000e040 con 0x55620b49e7c0 2024-08-03T20:57:32.704 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:32.703+0000 7fd3acb25f40 1 -- 172.21.15.32:0/3831060674 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:90 12.1d 12:b871830b:::605.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105) -- 0x55620b4b94e0 con 0x55620b49e7c0 2024-08-03T20:57:32.706 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:32.704+0000 7fd39fe00640 1 -- 172.21.15.32:0/3831060674 <== osd.0 v2:172.21.15.32:6816/3066156116 49 ==== osd_op_reply(90 605.00000000 [omap-set-header] v105'122 uv122 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fd39000e040 con 0x55620b49e7c0 2024-08-03T20:57:32.706 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:32.704+0000 7fd3acb25f40 1 -- 172.21.15.32:0/3831060674 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:91 12.13 12:c91d4a1d:::mds0_inotable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105) -- 0x55620b4b94e0 con 0x55620b4ae0b0 2024-08-03T20:57:32.707 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:32.706+0000 7fd39fe00640 1 -- 172.21.15.32:0/3831060674 <== osd.1 v2:172.21.15.32:6824/1551241759 5 ==== osd_op_reply(91 mds0_inotable [create] v105'192 uv186 ondisk = -17 ((17) File exists)) ==== 157+0+0 (crc 0 0 0) 0x7fd39000e040 con 0x55620b4ae0b0 2024-08-03T20:57:32.707 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:32.706+0000 7fd3acb25f40 1 -- 172.21.15.32:0/3831060674 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:92 12.13 12:c91d4a1d:::mds0_inotable:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e105) -- 0x55620b4b94e0 con 0x55620b4ae0b0 2024-08-03T20:57:32.708 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:32.707+0000 7fd39fe00640 1 -- 172.21.15.32:0/3831060674 <== osd.1 v2:172.21.15.32:6824/1551241759 6 ==== osd_op_reply(92 mds0_inotable [delete] v105'193 uv193 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7fd39000e040 con 0x55620b4ae0b0 2024-08-03T20:57:32.708 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:32.707+0000 7fd3acb25f40 1 -- 172.21.15.32:0/3831060674 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:93 12.13 12:c91d4a1d:::mds0_inotable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105) -- 0x55620b4b94e0 con 0x55620b4ae0b0 2024-08-03T20:57:32.709 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds0_inotable:head# 2024-08-03T20:57:32.710 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:32.708+0000 7fd39fe00640 1 -- 172.21.15.32:0/3831060674 <== osd.1 v2:172.21.15.32:6824/1551241759 7 ==== osd_op_reply(93 mds0_inotable [create] v105'194 uv194 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7fd39000e040 con 0x55620b4ae0b0 2024-08-03T20:57:32.710 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:32.708+0000 7fd3acb25f40 1 -- 172.21.15.32:0/3831060674 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:94 12.13 12:c91d4a1d:::mds0_inotable:head [write 0~34 in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105) -- 0x55620b4b4d40 con 0x55620b4ae0b0 2024-08-03T20:57:32.711 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:32.710+0000 7fd39fe00640 1 -- 172.21.15.32:0/3831060674 <== osd.1 v2:172.21.15.32:6824/1551241759 8 ==== osd_op_reply(94 mds0_inotable [write 0~34] v105'195 uv195 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7fd39000e040 con 0x55620b4ae0b0 2024-08-03T20:57:32.711 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:32.710+0000 7fd3acb25f40 1 -- 172.21.15.32:0/3831060674 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:95 12.13 12:c91d4a1d:::mds0_inotable:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105) -- 0x55620b4b4d40 con 0x55620b4ae0b0 2024-08-03T20:57:32.712 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:32.711+0000 7fd39fe00640 1 -- 172.21.15.32:0/3831060674 <== osd.1 v2:172.21.15.32:6824/1551241759 9 ==== osd_op_reply(95 mds0_inotable [omap-set-header] v105'196 uv196 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7fd39000e040 con 0x55620b4ae0b0 2024-08-03T20:57:32.712 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:32.711+0000 7fd3acb25f40 1 -- 172.21.15.32:0/3831060674 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:96 12.13 12:cd5a64a3:::100.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105) -- 0x55620b4b4d40 con 0x55620b4ae0b0 2024-08-03T20:57:32.713 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:32.712+0000 7fd39fe00640 1 -- 172.21.15.32:0/3831060674 <== osd.1 v2:172.21.15.32:6824/1551241759 10 ==== osd_op_reply(96 100.00000000 [create] v105'197 uv191 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fd39000e040 con 0x55620b4ae0b0 2024-08-03T20:57:32.714 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:32.712+0000 7fd3acb25f40 1 -- 172.21.15.32:0/3831060674 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:97 12.13 12:cd5a64a3:::100.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e105) -- 0x55620b4b4d40 con 0x55620b4ae0b0 2024-08-03T20:57:32.715 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:32.713+0000 7fd39fe00640 1 -- 172.21.15.32:0/3831060674 <== osd.1 v2:172.21.15.32:6824/1551241759 11 ==== osd_op_reply(97 100.00000000 [delete] v105'198 uv198 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fd39000e040 con 0x55620b4ae0b0 2024-08-03T20:57:32.715 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:32.713+0000 7fd3acb25f40 1 -- 172.21.15.32:0/3831060674 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:98 12.13 12:cd5a64a3:::100.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105) -- 0x55620b4b4d40 con 0x55620b4ae0b0 2024-08-03T20:57:32.716 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::100.00000000:head# 2024-08-03T20:57:32.716 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:32.715+0000 7fd39fe00640 1 -- 172.21.15.32:0/3831060674 <== osd.1 v2:172.21.15.32:6824/1551241759 12 ==== osd_op_reply(98 100.00000000 [create] v105'199 uv199 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fd39000e040 con 0x55620b4ae0b0 2024-08-03T20:57:32.716 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:32.715+0000 7fd3acb25f40 1 -- 172.21.15.32:0/3831060674 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:99 12.13 12:cd5a64a3:::100.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105) -- 0x55620b4b4d40 con 0x55620b4ae0b0 2024-08-03T20:57:32.717 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:32.716+0000 7fd39fe00640 1 -- 172.21.15.32:0/3831060674 <== osd.1 v2:172.21.15.32:6824/1551241759 13 ==== osd_op_reply(99 100.00000000 [omap-set-header] v105'200 uv200 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fd39000e040 con 0x55620b4ae0b0 2024-08-03T20:57:32.717 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:32.716+0000 7fd3acb25f40 1 -- 172.21.15.32:0/3831060674 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:100 12.13 12:cd5a64a3:::100.00000000:head [omap-set-vals in=5194b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105) -- 0x55620b4bc370 con 0x55620b4ae0b0 2024-08-03T20:57:32.719 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:32.717+0000 7fd39fe00640 1 -- 172.21.15.32:0/3831060674 <== osd.1 v2:172.21.15.32:6824/1551241759 14 ==== osd_op_reply(100 100.00000000 [omap-set-vals] v105'201 uv201 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fd39000e040 con 0x55620b4ae0b0 2024-08-03T20:57:32.719 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:32.717+0000 7fd3acb25f40 1 --2- 172.21.15.32:0/3831060674 >> [v2:172.21.15.32:6800/9088303,v1:172.21.15.32:6801/9088303] conn(0x55620b4b99f0 0x55620b4bca30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:32.719 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:32.717+0000 7fd3acb25f40 1 -- 172.21.15.32:0/3831060674 --> [v2:172.21.15.32:6800/9088303,v1:172.21.15.32:6801/9088303] -- osd_op(unknown.0.0:101 12.b 12:d0630e4c:::mds0_sessionmap:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105) -- 0x55620b4bbe10 con 0x55620b4b99f0 2024-08-03T20:57:32.719 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:32.718+0000 7fd39fe00640 1 --2- 172.21.15.32:0/3831060674 >> [v2:172.21.15.32:6800/9088303,v1:172.21.15.32:6801/9088303] conn(0x55620b4b99f0 0x55620b4bca30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:32.720 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:32.718+0000 7fd39fe00640 1 --2- 172.21.15.32:0/3831060674 >> [v2:172.21.15.32:6800/9088303,v1:172.21.15.32:6801/9088303] conn(0x55620b4b99f0 0x55620b4bca30 crc :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:57:32.721 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:32.720+0000 7fd39fe00640 1 -- 172.21.15.32:0/3831060674 <== osd.3 v2:172.21.15.32:6800/9088303 1 ==== osd_op_reply(101 mds0_sessionmap [create] v105'87 uv86 ondisk = -17 ((17) File exists)) ==== 159+0+0 (crc 0 0 0) 0x7fd39000e040 con 0x55620b4b99f0 2024-08-03T20:57:32.721 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:32.720+0000 7fd3acb25f40 1 -- 172.21.15.32:0/3831060674 --> [v2:172.21.15.32:6800/9088303,v1:172.21.15.32:6801/9088303] -- osd_op(unknown.0.0:102 12.b 12:d0630e4c:::mds0_sessionmap:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e105) -- 0x55620b4bbe10 con 0x55620b4b99f0 2024-08-03T20:57:32.722 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:32.721+0000 7fd39fe00640 1 -- 172.21.15.32:0/3831060674 <== osd.3 v2:172.21.15.32:6800/9088303 2 ==== osd_op_reply(102 mds0_sessionmap [delete] v105'88 uv88 ondisk = 0) ==== 159+0+0 (crc 0 0 0) 0x7fd39000e040 con 0x55620b4b99f0 2024-08-03T20:57:32.723 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:32.721+0000 7fd3acb25f40 1 -- 172.21.15.32:0/3831060674 --> [v2:172.21.15.32:6800/9088303,v1:172.21.15.32:6801/9088303] -- osd_op(unknown.0.0:103 12.b 12:d0630e4c:::mds0_sessionmap:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105) -- 0x55620b4bbe10 con 0x55620b4b99f0 2024-08-03T20:57:32.724 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:32.722+0000 7fd39fe00640 1 -- 172.21.15.32:0/3831060674 <== osd.3 v2:172.21.15.32:6800/9088303 3 ==== osd_op_reply(103 mds0_sessionmap [create] v105'89 uv89 ondisk = 0) ==== 159+0+0 (crc 0 0 0) 0x7fd39000e040 con 0x55620b4b99f0 2024-08-03T20:57:32.724 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds0_sessionmap:head# 2024-08-03T20:57:32.724 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:32.722+0000 7fd3acb25f40 1 -- 172.21.15.32:0/3831060674 --> [v2:172.21.15.32:6800/9088303,v1:172.21.15.32:6801/9088303] -- osd_op(unknown.0.0:104 12.b 12:d0630e4c:::mds0_sessionmap:head [omap-set-header in=14b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105) -- 0x55620b4bbe10 con 0x55620b4b99f0 2024-08-03T20:57:32.725 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:32.724+0000 7fd39fe00640 1 -- 172.21.15.32:0/3831060674 <== osd.3 v2:172.21.15.32:6800/9088303 4 ==== osd_op_reply(104 mds0_sessionmap [omap-set-header] v105'90 uv90 ondisk = 0) ==== 159+0+0 (crc 0 0 0) 0x7fd39000e040 con 0x55620b4b99f0 2024-08-03T20:57:32.725 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:32.724+0000 7fd3acb25f40 1 -- 172.21.15.32:0/3831060674 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:105 12.7 12:e0b41d06:::609.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105) -- 0x55620b4bbe10 con 0x55620b4ae0b0 2024-08-03T20:57:32.730 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:32.726+0000 7fd39fe00640 1 -- 172.21.15.32:0/3831060674 <== osd.1 v2:172.21.15.32:6824/1551241759 15 ==== osd_op_reply(105 609.00000000 [create] v105'117 uv116 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fd39000e040 con 0x55620b4ae0b0 2024-08-03T20:57:32.731 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:32.726+0000 7fd3acb25f40 1 -- 172.21.15.32:0/3831060674 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:106 12.7 12:e0b41d06:::609.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e105) -- 0x55620b4bbe10 con 0x55620b4ae0b0 2024-08-03T20:57:32.731 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:32.727+0000 7fd39fe00640 1 -- 172.21.15.32:0/3831060674 <== osd.1 v2:172.21.15.32:6824/1551241759 16 ==== osd_op_reply(106 609.00000000 [delete] v105'118 uv118 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fd39000e040 con 0x55620b4ae0b0 2024-08-03T20:57:32.731 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:32.727+0000 7fd3acb25f40 1 -- 172.21.15.32:0/3831060674 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:107 12.7 12:e0b41d06:::609.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105) -- 0x55620b4bbe10 con 0x55620b4ae0b0 2024-08-03T20:57:32.731 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::609.00000000:head# 2024-08-03T20:57:32.732 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:32.729+0000 7fd39fe00640 1 -- 172.21.15.32:0/3831060674 <== osd.1 v2:172.21.15.32:6824/1551241759 17 ==== osd_op_reply(107 609.00000000 [create] v105'119 uv119 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fd39000e040 con 0x55620b4ae0b0 2024-08-03T20:57:32.732 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:32.729+0000 7fd3acb25f40 1 -- 172.21.15.32:0/3831060674 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:108 12.7 12:e0b41d06:::609.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105) -- 0x55620b4bbe10 con 0x55620b4ae0b0 2024-08-03T20:57:32.732 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:32.730+0000 7fd39fe00640 1 -- 172.21.15.32:0/3831060674 <== osd.1 v2:172.21.15.32:6824/1551241759 18 ==== osd_op_reply(108 609.00000000 [setxattr (30)] v105'120 uv120 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fd39000e040 con 0x55620b4ae0b0 2024-08-03T20:57:32.732 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:32.730+0000 7fd3acb25f40 1 -- 172.21.15.32:0/3831060674 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:109 12.7 12:e0b41d06:::609.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105) -- 0x55620b4bbe10 con 0x55620b4ae0b0 2024-08-03T20:57:32.733 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:32.731+0000 7fd39fe00640 1 -- 172.21.15.32:0/3831060674 <== osd.1 v2:172.21.15.32:6824/1551241759 19 ==== osd_op_reply(109 609.00000000 [setxattr (62)] v105'121 uv121 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fd39000e040 con 0x55620b4ae0b0 2024-08-03T20:57:32.733 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:32.731+0000 7fd3acb25f40 1 -- 172.21.15.32:0/3831060674 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:110 12.7 12:e0b41d06:::609.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105) -- 0x55620b4bbe10 con 0x55620b4ae0b0 2024-08-03T20:57:32.734 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:32.733+0000 7fd39fe00640 1 -- 172.21.15.32:0/3831060674 <== osd.1 v2:172.21.15.32:6824/1551241759 20 ==== osd_op_reply(110 609.00000000 [omap-set-header] v105'122 uv122 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fd39000e040 con 0x55620b4ae0b0 2024-08-03T20:57:32.734 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:32.733+0000 7fd3acb25f40 1 -- 172.21.15.32:0/3831060674 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:111 12.f 12:f1f69726:::400.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105) -- 0x55620b4bbe10 con 0x55620b4b1590 2024-08-03T20:57:32.736 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:32.734+0000 7fd3a5c00640 1 -- 172.21.15.32:0/3831060674 <== osd.6 v2:172.21.15.118:6825/801947386 12 ==== osd_op_reply(111 400.00000000 [create] v105'99 uv98 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fd3a0070070 con 0x55620b4b1590 2024-08-03T20:57:32.736 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:32.734+0000 7fd3acb25f40 1 -- 172.21.15.32:0/3831060674 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:112 12.f 12:f1f69726:::400.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e105) -- 0x55620b4bbe10 con 0x55620b4b1590 2024-08-03T20:57:32.737 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:32.736+0000 7fd3a5c00640 1 -- 172.21.15.32:0/3831060674 <== osd.6 v2:172.21.15.118:6825/801947386 13 ==== osd_op_reply(112 400.00000000 [delete] v105'100 uv100 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fd3a0070070 con 0x55620b4b1590 2024-08-03T20:57:32.737 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:32.736+0000 7fd3acb25f40 1 -- 172.21.15.32:0/3831060674 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:113 12.f 12:f1f69726:::400.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105) -- 0x55620b4bbe10 con 0x55620b4b1590 2024-08-03T20:57:32.739 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::400.00000000:head# 2024-08-03T20:57:32.739 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:32.737+0000 7fd3a5c00640 1 -- 172.21.15.32:0/3831060674 <== osd.6 v2:172.21.15.118:6825/801947386 14 ==== osd_op_reply(113 400.00000000 [create] v105'101 uv101 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fd3a0070070 con 0x55620b4b1590 2024-08-03T20:57:32.739 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:32.738+0000 7fd3acb25f40 1 -- 172.21.15.32:0/3831060674 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:114 12.f 12:f1f69726:::400.00000000:head [write 0~22 in=22b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105) -- 0x55620b4bbe10 con 0x55620b4b1590 2024-08-03T20:57:32.741 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:32.739+0000 7fd3a5c00640 1 -- 172.21.15.32:0/3831060674 <== osd.6 v2:172.21.15.118:6825/801947386 15 ==== osd_op_reply(114 400.00000000 [write 0~22] v105'102 uv102 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fd3a0070070 con 0x55620b4b1590 2024-08-03T20:57:32.741 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:32.739+0000 7fd3acb25f40 1 -- 172.21.15.32:0/3831060674 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:115 12.f 12:f1f69726:::400.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105) -- 0x55620b4bbe10 con 0x55620b4b1590 2024-08-03T20:57:32.742 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:32.741+0000 7fd3a5c00640 1 -- 172.21.15.32:0/3831060674 <== osd.6 v2:172.21.15.118:6825/801947386 16 ==== osd_op_reply(115 400.00000000 [omap-set-header] v105'103 uv103 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fd3a0070070 con 0x55620b4b1590 2024-08-03T20:57:32.742 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:32.741+0000 7fd3acb25f40 1 -- 172.21.15.32:0/3831060674 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:116 12.1f 12:fe07bba1:::100.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105) -- 0x55620b4bbe10 con 0x55620b4b1590 2024-08-03T20:57:32.744 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:32.742+0000 7fd3a5c00640 1 -- 172.21.15.32:0/3831060674 <== osd.6 v2:172.21.15.118:6825/801947386 17 ==== osd_op_reply(116 100.00000000.inode [create] v105'241 uv233 ondisk = -17 ((17) File exists)) ==== 162+0+0 (crc 0 0 0) 0x7fd3a0070070 con 0x55620b4b1590 2024-08-03T20:57:32.744 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:32.742+0000 7fd3acb25f40 1 -- 172.21.15.32:0/3831060674 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:117 12.1f 12:fe07bba1:::100.00000000.inode:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e105) -- 0x55620b4bbe10 con 0x55620b4b1590 2024-08-03T20:57:32.745 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:32.744+0000 7fd3a5c00640 1 -- 172.21.15.32:0/3831060674 <== osd.6 v2:172.21.15.118:6825/801947386 18 ==== osd_op_reply(117 100.00000000.inode [delete] v105'242 uv242 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7fd3a0070070 con 0x55620b4b1590 2024-08-03T20:57:32.745 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:32.744+0000 7fd3acb25f40 1 -- 172.21.15.32:0/3831060674 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:118 12.1f 12:fe07bba1:::100.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105) -- 0x55620b4bbe10 con 0x55620b4b1590 2024-08-03T20:57:32.746 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:32.745+0000 7fd3a5c00640 1 -- 172.21.15.32:0/3831060674 <== osd.6 v2:172.21.15.118:6825/801947386 19 ==== osd_op_reply(118 100.00000000.inode [create] v105'243 uv243 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7fd3a0070070 con 0x55620b4b1590 2024-08-03T20:57:32.746 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::100.00000000.inode:head# 2024-08-03T20:57:32.747 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:32.745+0000 7fd3acb25f40 1 -- 172.21.15.32:0/3831060674 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:119 12.1f 12:fe07bba1:::100.00000000.inode:head [write 0~582 in=582b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105) -- 0x55620b4bbe10 con 0x55620b4b1590 2024-08-03T20:57:32.748 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:32.746+0000 7fd3a5c00640 1 -- 172.21.15.32:0/3831060674 <== osd.6 v2:172.21.15.118:6825/801947386 20 ==== osd_op_reply(119 100.00000000.inode [write 0~582] v105'244 uv244 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7fd3a0070070 con 0x55620b4b1590 2024-08-03T20:57:32.748 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:32.746+0000 7fd3acb25f40 1 -- 172.21.15.32:0/3831060674 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:120 12.1f 12:fe07bba1:::100.00000000.inode:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105) -- 0x55620b4bbe10 con 0x55620b4b1590 2024-08-03T20:57:32.749 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:32.748+0000 7fd3a5c00640 1 -- 172.21.15.32:0/3831060674 <== osd.6 v2:172.21.15.118:6825/801947386 21 ==== osd_op_reply(120 100.00000000.inode [omap-set-header] v105'245 uv245 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7fd3a0070070 con 0x55620b4b1590 2024-08-03T20:57:32.749 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:32.748+0000 7fd3acb25f40 1 -- 172.21.15.32:0/3831060674 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:121 12.1f 12:ff5b34d6:::1.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105) -- 0x55620b4bbe10 con 0x55620b4b1590 2024-08-03T20:57:32.750 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:32.749+0000 7fd3a5c00640 1 -- 172.21.15.32:0/3831060674 <== osd.6 v2:172.21.15.118:6825/801947386 22 ==== osd_op_reply(121 1.00000000 [create] v105'246 uv240 ondisk = -17 ((17) File exists)) ==== 154+0+0 (crc 0 0 0) 0x7fd3a0070070 con 0x55620b4b1590 2024-08-03T20:57:32.750 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:32.749+0000 7fd3acb25f40 1 -- 172.21.15.32:0/3831060674 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:122 12.1f 12:ff5b34d6:::1.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e105) -- 0x55620b4bbe10 con 0x55620b4b1590 2024-08-03T20:57:32.752 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:32.750+0000 7fd3a5c00640 1 -- 172.21.15.32:0/3831060674 <== osd.6 v2:172.21.15.118:6825/801947386 23 ==== osd_op_reply(122 1.00000000 [delete] v105'247 uv247 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7fd3a0070070 con 0x55620b4b1590 2024-08-03T20:57:32.752 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:32.750+0000 7fd3acb25f40 1 -- 172.21.15.32:0/3831060674 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:123 12.1f 12:ff5b34d6:::1.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105) -- 0x55620b4bbe10 con 0x55620b4b1590 2024-08-03T20:57:32.753 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:32.752+0000 7fd3a5c00640 1 -- 172.21.15.32:0/3831060674 <== osd.6 v2:172.21.15.118:6825/801947386 24 ==== osd_op_reply(123 1.00000000 [create] v105'248 uv248 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7fd3a0070070 con 0x55620b4b1590 2024-08-03T20:57:32.753 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::1.00000000:head# 2024-08-03T20:57:32.753 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:32.752+0000 7fd3acb25f40 1 -- 172.21.15.32:0/3831060674 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:124 12.1f 12:ff5b34d6:::1.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105) -- 0x55620b4bbe10 con 0x55620b4b1590 2024-08-03T20:57:32.754 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:32.753+0000 7fd3a5c00640 1 -- 172.21.15.32:0/3831060674 <== osd.6 v2:172.21.15.118:6825/801947386 25 ==== osd_op_reply(124 1.00000000 [setxattr (30)] v105'249 uv249 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7fd3a0070070 con 0x55620b4b1590 2024-08-03T20:57:32.754 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:32.753+0000 7fd3acb25f40 1 -- 172.21.15.32:0/3831060674 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:125 12.1f 12:ff5b34d6:::1.00000000:head [setxattr parent (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105) -- 0x55620b4bbe10 con 0x55620b4b1590 2024-08-03T20:57:32.756 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:32.755+0000 7fd3a5c00640 1 -- 172.21.15.32:0/3831060674 <== osd.6 v2:172.21.15.118:6825/801947386 26 ==== osd_op_reply(125 1.00000000 [setxattr (30)] v105'250 uv250 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7fd3a0070070 con 0x55620b4b1590 2024-08-03T20:57:32.756 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:32.755+0000 7fd3acb25f40 1 -- 172.21.15.32:0/3831060674 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:126 12.1f 12:ff5b34d6:::1.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105) -- 0x55620b4bbe10 con 0x55620b4b1590 2024-08-03T20:57:32.757 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:32.756+0000 7fd3a5c00640 1 -- 172.21.15.32:0/3831060674 <== osd.6 v2:172.21.15.118:6825/801947386 27 ==== osd_op_reply(126 1.00000000 [omap-set-header] v105'251 uv251 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7fd3a0070070 con 0x55620b4b1590 2024-08-03T20:57:32.757 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:32.756+0000 7fd3acb25f40 1 -- 172.21.15.32:0/3831060674 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:127 12.1f 12:ff5b34d6:::1.00000000:head [omap-set-vals in=584b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105) -- 0x55620b4bd420 con 0x55620b4b1590 2024-08-03T20:57:32.760 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:32.757+0000 7fd3a5c00640 1 -- 172.21.15.32:0/3831060674 <== osd.6 v2:172.21.15.118:6825/801947386 28 ==== osd_op_reply(127 1.00000000 [omap-set-vals] v105'252 uv252 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7fd3a0070070 con 0x55620b4b1590 2024-08-03T20:57:32.760 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:32.757+0000 7fd3acb25f40 1 -- 172.21.15.32:0/3831060674 >> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] conn(0x55620b49e7c0 msgr2=0x55620b49ebd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:32.760 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:32.757+0000 7fd3acb25f40 1 --2- 172.21.15.32:0/3831060674 >> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] conn(0x55620b49e7c0 0x55620b49ebd0 crc :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:32.760 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:32.757+0000 7fd3acb25f40 1 -- 172.21.15.32:0/3831060674 >> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] conn(0x55620b4ae0b0 msgr2=0x55620b4ae4c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:32.760 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:32.757+0000 7fd3acb25f40 1 --2- 172.21.15.32:0/3831060674 >> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] conn(0x55620b4ae0b0 0x55620b4ae4c0 crc :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:32.760 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:32.757+0000 7fd3acb25f40 1 -- 172.21.15.32:0/3831060674 >> [v2:172.21.15.32:6800/9088303,v1:172.21.15.32:6801/9088303] conn(0x55620b4b99f0 msgr2=0x55620b4bca30 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:32.760 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:32.757+0000 7fd3acb25f40 1 --2- 172.21.15.32:0/3831060674 >> [v2:172.21.15.32:6800/9088303,v1:172.21.15.32:6801/9088303] conn(0x55620b4b99f0 0x55620b4bca30 crc :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:32.760 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:32.757+0000 7fd3acb25f40 1 -- 172.21.15.32:0/3831060674 >> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] conn(0x55620b4a9950 msgr2=0x55620b4abd70 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:32.760 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:32.757+0000 7fd3acb25f40 1 --2- 172.21.15.32:0/3831060674 >> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] conn(0x55620b4a9950 0x55620b4abd70 crc :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:32.760 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:32.757+0000 7fd3a5c00640 1 -- 172.21.15.32:0/3831060674 reap_dead start 2024-08-03T20:57:32.760 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:32.757+0000 7fd3acb25f40 1 -- 172.21.15.32:0/3831060674 >> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] conn(0x55620b4b1590 msgr2=0x55620b4b19a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:32.760 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:32.757+0000 7fd3acb25f40 1 --2- 172.21.15.32:0/3831060674 >> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] conn(0x55620b4b1590 0x55620b4b19a0 crc :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:32.760 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:32.757+0000 7fd3acb25f40 1 -- 172.21.15.32:0/3831060674 >> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] conn(0x55620b4b5d30 msgr2=0x55620b4b8b50 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:32.761 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:32.757+0000 7fd3acb25f40 1 --2- 172.21.15.32:0/3831060674 >> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] conn(0x55620b4b5d30 0x55620b4b8b50 crc :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:32.761 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:32.758+0000 7fd3acb25f40 1 -- 172.21.15.32:0/3831060674 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fd3840759f0 msgr2=0x7fd384077f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:32.761 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:32.758+0000 7fd3acb25f40 1 --2- 172.21.15.32:0/3831060674 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fd3840759f0 0x7fd384077f00 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7fd398002040 tx=0x7fd398018040 comp rx=0 tx=0).stop 2024-08-03T20:57:32.761 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:32.758+0000 7fd3acb25f40 1 -- 172.21.15.32:0/3831060674 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55620b3fbf60 msgr2=0x55620b373aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:32.761 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:32.758+0000 7fd3acb25f40 1 --2- 172.21.15.32:0/3831060674 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55620b3fbf60 0x55620b373aa0 secure :-1 s=READY pgs=1569 cs=0 l=1 rev1=1 crypto rx=0x7fd3a0025dd0 tx=0x7fd3a0004ad0 comp rx=0 tx=0).stop 2024-08-03T20:57:32.761 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:32.758+0000 7fd3a5c00640 1 -- 172.21.15.32:0/3831060674 reap_dead start 2024-08-03T20:57:32.761 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:32.758+0000 7fd3acb25f40 1 -- 172.21.15.32:0/3831060674 shutdown_connections 2024-08-03T20:57:32.761 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:32.758+0000 7fd3acb25f40 1 -- 172.21.15.32:0/3831060674 >> 172.21.15.32:0/3831060674 conn(0x55620b24eac0 msgr2=0x55620b405410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:57:32.761 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:32.758+0000 7fd3acb25f40 1 -- 172.21.15.32:0/3831060674 shutdown_connections 2024-08-03T20:57:32.761 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:32.758+0000 7fd3acb25f40 1 -- 172.21.15.32:0/3831060674 wait complete. 2024-08-03T20:57:32.771 DEBUG:teuthology.orchestra.run.smithi032:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata put mds0_inotable - 2024-08-03T20:57:32.954 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:32.952+0000 7ff2aa6f4f40 1 Processor -- start 2024-08-03T20:57:32.954 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:32.953+0000 7ff2aa6f4f40 1 -- start start 2024-08-03T20:57:32.956 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:32.954+0000 7ff2aa6f4f40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55feb10cb360 0x55feb10c8990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:32.956 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:32.954+0000 7ff2aa6f4f40 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55feb10ccea0 0x55feb1058fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:32.956 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:32.954+0000 7ff2aa6f4f40 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55feb10e8cb0 0x55feb10e5160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:32.956 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:32.954+0000 7ff2aa6f4f40 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55feb10e90c0 con 0x55feb10ccea0 2024-08-03T20:57:32.956 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:32.954+0000 7ff2aa6f4f40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x55feb0f62e30 con 0x55feb10cb360 2024-08-03T20:57:32.956 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:32.954+0000 7ff2aa6f4f40 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x55feb1059a20 con 0x55feb10e8cb0 2024-08-03T20:57:32.956 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:32.954+0000 7ff2a2e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55feb10ccea0 0x55feb1058fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:32.957 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:32.954+0000 7ff2a2e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55feb10ccea0 0x55feb1058fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:33844/0 (socket says 172.21.15.32:33844) 2024-08-03T20:57:32.957 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:32.954+0000 7ff2a2e00640 1 -- 172.21.15.32:0/2695213533 learned_addr learned my addr 172.21.15.32:0/2695213533 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:57:32.957 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:32.954+0000 7ff2a4200640 1 --2- 172.21.15.32:0/2695213533 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55feb10e8cb0 0x55feb10e5160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:32.957 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:32.954+0000 7ff2a3800640 1 --2- 172.21.15.32:0/2695213533 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55feb10cb360 0x55feb10c8990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:32.957 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:32.955+0000 7ff2a2e00640 1 -- 172.21.15.32:0/2695213533 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55feb10e8cb0 msgr2=0x55feb10e5160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:57:32.957 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:32.955+0000 7ff2a2e00640 1 --2- 172.21.15.32:0/2695213533 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55feb10e8cb0 0x55feb10e5160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:32.957 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:32.955+0000 7ff2a2e00640 1 -- 172.21.15.32:0/2695213533 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55feb10cb360 msgr2=0x55feb10c8990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:57:32.957 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:32.955+0000 7ff2a2e00640 1 --2- 172.21.15.32:0/2695213533 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55feb10cb360 0x55feb10c8990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:32.957 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:32.955+0000 7ff2a2e00640 1 -- 172.21.15.32:0/2695213533 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55feb0f3bf10 con 0x55feb10ccea0 2024-08-03T20:57:32.958 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:32.955+0000 7ff2a2e00640 1 --2- 172.21.15.32:0/2695213533 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55feb10ccea0 0x55feb1058fa0 secure :-1 s=READY pgs=1571 cs=0 l=1 rev1=1 crypto rx=0x7ff29000b600 tx=0x7ff29000bac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=693b5ba24b0e1703 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:57:32.959 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:32.955+0000 7ff2a1a00640 1 -- 172.21.15.32:0/2695213533 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff29000a890 con 0x55feb10ccea0 2024-08-03T20:57:32.959 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:32.955+0000 7ff2a1a00640 1 -- 172.21.15.32:0/2695213533 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7ff29000aa30 con 0x55feb10ccea0 2024-08-03T20:57:32.959 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:32.955+0000 7ff2a1a00640 1 -- 172.21.15.32:0/2695213533 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff29000ae40 con 0x55feb10ccea0 2024-08-03T20:57:32.959 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:32.956+0000 7ff2aa6f4f40 1 -- 172.21.15.32:0/2695213533 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55feb10ccea0 msgr2=0x55feb1058fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:32.959 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:32.956+0000 7ff2aa6f4f40 1 --2- 172.21.15.32:0/2695213533 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55feb10ccea0 0x55feb1058fa0 secure :-1 s=READY pgs=1571 cs=0 l=1 rev1=1 crypto rx=0x7ff29000b600 tx=0x7ff29000bac0 comp rx=0 tx=0).stop 2024-08-03T20:57:32.959 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:32.956+0000 7ff2aa6f4f40 1 -- 172.21.15.32:0/2695213533 shutdown_connections 2024-08-03T20:57:32.959 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:32.956+0000 7ff2aa6f4f40 1 --2- 172.21.15.32:0/2695213533 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55feb10e8cb0 0x55feb10e5160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:32.959 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:32.956+0000 7ff2aa6f4f40 1 --2- 172.21.15.32:0/2695213533 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55feb10ccea0 0x55feb1058fa0 unknown :-1 s=CLOSED pgs=1571 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:32.959 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:32.956+0000 7ff2aa6f4f40 1 --2- 172.21.15.32:0/2695213533 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55feb10cb360 0x55feb10c8990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:32.959 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:32.956+0000 7ff2aa6f4f40 1 -- 172.21.15.32:0/2695213533 >> 172.21.15.32:0/2695213533 conn(0x55feb0f34ac0 msgr2=0x55feb105b910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:57:32.959 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:32.956+0000 7ff2aa6f4f40 1 -- 172.21.15.32:0/2695213533 shutdown_connections 2024-08-03T20:57:32.959 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:32.956+0000 7ff2aa6f4f40 1 -- 172.21.15.32:0/2695213533 wait complete. 2024-08-03T20:57:32.960 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:32.957+0000 7ff2aa6f4f40 1 Processor -- start 2024-08-03T20:57:32.960 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:32.957+0000 7ff2aa6f4f40 1 -- start start 2024-08-03T20:57:32.962 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:32.959+0000 7ff2aa6f4f40 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55feb10cb360 0x55feb10c5750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:32.963 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:32.959+0000 7ff2aa6f4f40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55feb10ccea0 0x55feb1181f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:32.963 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:32.959+0000 7ff2aa6f4f40 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55feb10e8cb0 0x55feb1013240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:32.963 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:32.959+0000 7ff2aa6f4f40 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55feb10ca3d0 con 0x55feb10e8cb0 2024-08-03T20:57:32.963 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:32.959+0000 7ff2aa6f4f40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x55feb10ca990 con 0x55feb10ccea0 2024-08-03T20:57:32.963 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:32.959+0000 7ff2aa6f4f40 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x55feb10e17b0 con 0x55feb10cb360 2024-08-03T20:57:32.963 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:32.959+0000 7ff2a4200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55feb10e8cb0 0x55feb1013240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:32.963 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:32.959+0000 7ff2a4200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55feb10e8cb0 0x55feb1013240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:33860/0 (socket says 172.21.15.32:33860) 2024-08-03T20:57:32.963 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:32.959+0000 7ff2a4200640 1 -- 172.21.15.32:0/3466040052 learned_addr learned my addr 172.21.15.32:0/3466040052 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:57:32.963 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:32.959+0000 7ff2a3800640 1 --2- 172.21.15.32:0/3466040052 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55feb10cb360 0x55feb10c5750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:32.963 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:32.959+0000 7ff2a2e00640 1 --2- 172.21.15.32:0/3466040052 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55feb10ccea0 0x55feb1181f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:32.963 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:32.959+0000 7ff2a4200640 1 -- 172.21.15.32:0/3466040052 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55feb10cb360 msgr2=0x55feb10c5750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:32.963 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:32.959+0000 7ff2a4200640 1 --2- 172.21.15.32:0/3466040052 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55feb10cb360 0x55feb10c5750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:32.964 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:32.959+0000 7ff2a4200640 1 -- 172.21.15.32:0/3466040052 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55feb10ccea0 msgr2=0x55feb1181f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:32.964 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:32.959+0000 7ff2a4200640 1 --2- 172.21.15.32:0/3466040052 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55feb10ccea0 0x55feb1181f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:32.964 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:32.959+0000 7ff2a4200640 1 -- 172.21.15.32:0/3466040052 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55feb10c80f0 con 0x55feb10e8cb0 2024-08-03T20:57:32.964 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:32.960+0000 7ff2a4200640 1 --2- 172.21.15.32:0/3466040052 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55feb10e8cb0 0x55feb1013240 secure :-1 s=READY pgs=1572 cs=0 l=1 rev1=1 crypto rx=0x7ff29c010720 tx=0x7ff29c010be0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:57:32.964 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:32.960+0000 7ff29be00640 1 -- 172.21.15.32:0/3466040052 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff29c004220 con 0x55feb10e8cb0 2024-08-03T20:57:32.964 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:32.960+0000 7ff29be00640 1 -- 172.21.15.32:0/3466040052 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7ff29c00ebc0 con 0x55feb10e8cb0 2024-08-03T20:57:32.964 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:32.960+0000 7ff29be00640 1 -- 172.21.15.32:0/3466040052 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff29c003410 con 0x55feb10e8cb0 2024-08-03T20:57:32.964 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:32.960+0000 7ff2aa6f4f40 1 -- 172.21.15.32:0/3466040052 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x55feb1182490 con 0x55feb10e8cb0 2024-08-03T20:57:32.964 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:32.960+0000 7ff2aa6f4f40 1 -- 172.21.15.32:0/3466040052 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55feb1181750 con 0x55feb10e8cb0 2024-08-03T20:57:32.965 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:32.962+0000 7ff29be00640 1 -- 172.21.15.32:0/3466040052 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7ff29c0035b0 con 0x55feb10e8cb0 2024-08-03T20:57:32.965 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:32.963+0000 7ff29be00640 1 --2- 172.21.15.32:0/3466040052 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7ff284075a20 0x7ff284077f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:32.965 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:32.963+0000 7ff29be00640 1 -- 172.21.15.32:0/3466040052 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(105..105 src has 1..105) ==== 7574+0+0 (secure 0 0 0) 0x7ff29c098b60 con 0x55feb10e8cb0 2024-08-03T20:57:32.965 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:32.964+0000 7ff2a3800640 1 --2- 172.21.15.32:0/3466040052 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7ff284075a20 0x7ff284077f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:32.966 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:32.964+0000 7ff2aa6f4f40 1 --2- 172.21.15.32:0/3466040052 >> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] conn(0x55feb10579d0 0x55feb1057de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:32.966 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:32.964+0000 7ff2aa6f4f40 1 -- 172.21.15.32:0/3466040052 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:1 12.13 12:c91d4a1d:::mds0_inotable:head [writefull 0~80 in=80b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105) -- 0x55feb1184ba0 con 0x55feb10579d0 2024-08-03T20:57:32.966 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:32.964+0000 7ff2a2e00640 1 --2- 172.21.15.32:0/3466040052 >> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] conn(0x55feb10579d0 0x55feb1057de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:32.966 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:32.965+0000 7ff2a3800640 1 --2- 172.21.15.32:0/3466040052 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7ff284075a20 0x7ff284077f30 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7ff294055000 tx=0x7ff294003020 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:57:32.966 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:32.965+0000 7ff2a2e00640 1 --2- 172.21.15.32:0/3466040052 >> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] conn(0x55feb10579d0 0x55feb1057de0 crc :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:57:32.968 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:32.966+0000 7ff2a2e00640 1 -- 172.21.15.32:0/3466040052 <== osd.1 v2:172.21.15.32:6824/1551241759 1 ==== osd_op_reply(1 mds0_inotable [writefull 0~80] v105'202 uv202 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7ff29000aab0 con 0x55feb10579d0 2024-08-03T20:57:32.969 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:32.966+0000 7ff2aa6f4f40 1 -- 172.21.15.32:0/3466040052 >> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] conn(0x55feb10579d0 msgr2=0x55feb1057de0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:32.969 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:32.966+0000 7ff2aa6f4f40 1 --2- 172.21.15.32:0/3466040052 >> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] conn(0x55feb10579d0 0x55feb1057de0 crc :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:32.969 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:32.966+0000 7ff2aa6f4f40 1 -- 172.21.15.32:0/3466040052 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7ff284075a20 msgr2=0x7ff284077f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:32.969 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:32.966+0000 7ff2aa6f4f40 1 --2- 172.21.15.32:0/3466040052 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7ff284075a20 0x7ff284077f30 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7ff294055000 tx=0x7ff294003020 comp rx=0 tx=0).stop 2024-08-03T20:57:32.969 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:32.966+0000 7ff2aa6f4f40 1 -- 172.21.15.32:0/3466040052 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55feb10e8cb0 msgr2=0x55feb1013240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:32.969 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:32.966+0000 7ff2aa6f4f40 1 --2- 172.21.15.32:0/3466040052 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55feb10e8cb0 0x55feb1013240 secure :-1 s=READY pgs=1572 cs=0 l=1 rev1=1 crypto rx=0x7ff29c010720 tx=0x7ff29c010be0 comp rx=0 tx=0).stop 2024-08-03T20:57:32.969 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:32.967+0000 7ff2a4200640 1 -- 172.21.15.32:0/3466040052 reap_dead start 2024-08-03T20:57:32.969 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:32.967+0000 7ff2aa6f4f40 1 -- 172.21.15.32:0/3466040052 shutdown_connections 2024-08-03T20:57:32.969 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:32.967+0000 7ff2aa6f4f40 1 -- 172.21.15.32:0/3466040052 >> 172.21.15.32:0/3466040052 conn(0x55feb0f34ac0 msgr2=0x55feb10e5160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:57:32.969 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:32.967+0000 7ff2aa6f4f40 1 -- 172.21.15.32:0/3466040052 shutdown_connections 2024-08-03T20:57:32.969 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:32.967+0000 7ff2aa6f4f40 1 -- 172.21.15.32:0/3466040052 wait complete. 2024-08-03T20:57:32.979 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs joinable true 2024-08-03T20:57:33.106 INFO:tasks.ceph.mds.a.smithi032.stderr: -50> 2024-08-03T20:57:28.353+0000 7f489b600640 -1 log_channel(cluster) log [ERR] : error decoding table object 'mds_snaptable': Decoder at 'virtual void SnapServer::decode_server_state(ceph::buffer::v15_2_0::list::const_iterator&)' v=5 cannot decode v=100 minimal_decoder=101: Malformed input [buffer:3] 2024-08-03T20:57:33.212 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:33.209+0000 7ff5df400640 1 -- 172.21.15.32:0/1250395839 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7ff5d80d0fb0 msgr2=0x7ff5d80b7d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:33.212 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:33.209+0000 7ff5df400640 1 --2- 172.21.15.32:0/1250395839 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7ff5d80d0fb0 0x7ff5d80b7d00 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7ff5d0005fd0 tx=0x7ff5d002ab20 comp rx=0 tx=0).stop 2024-08-03T20:57:33.212 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:33.210+0000 7ff5df400640 1 -- 172.21.15.32:0/1250395839 shutdown_connections 2024-08-03T20:57:33.212 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:33.210+0000 7ff5df400640 1 --2- 172.21.15.32:0/1250395839 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7ff5d80d0fb0 0x7ff5d80b7d00 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:33.212 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:33.210+0000 7ff5df400640 1 --2- 172.21.15.32:0/1250395839 >> v2:172.21.15.118:3300/0 conn(0x7ff5d80cf9c0 0x7ff5d80d0a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:33.212 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:33.210+0000 7ff5df400640 1 -- 172.21.15.32:0/1250395839 >> 172.21.15.32:0/1250395839 conn(0x7ff5d809d660 msgr2=0x7ff5d80d4330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:57:33.212 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:33.210+0000 7ff5df400640 1 -- 172.21.15.32:0/1250395839 shutdown_connections 2024-08-03T20:57:33.212 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:33.210+0000 7ff5df400640 1 -- 172.21.15.32:0/1250395839 wait complete. 2024-08-03T20:57:33.213 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:33.212+0000 7ff5df400640 1 Processor -- start 2024-08-03T20:57:33.214 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:33.212+0000 7ff5df400640 1 -- start start 2024-08-03T20:57:33.216 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:33.214+0000 7ff5df400640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ff5d80b59f0 0x7ff5d815e200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:33.216 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:33.214+0000 7ff5df400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff5d80cf9c0 0x7ff5d8157b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:33.216 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:33.214+0000 7ff5df400640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7ff5d80d0fb0 0x7ff5d815a130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:33.216 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:33.214+0000 7ff5df400640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7ff5d80b87d0 con 0x7ff5d80cf9c0 2024-08-03T20:57:33.216 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:33.214+0000 7ff5df400640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7ff5d80b8650 con 0x7ff5d80b59f0 2024-08-03T20:57:33.216 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:33.214+0000 7ff5df400640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7ff5d80b8950 con 0x7ff5d80d0fb0 2024-08-03T20:57:33.216 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:33.214+0000 7ff5d7e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff5d80cf9c0 0x7ff5d8157b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:33.216 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:33.214+0000 7ff5d7e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff5d80cf9c0 0x7ff5d8157b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:33872/0 (socket says 172.21.15.32:33872) 2024-08-03T20:57:33.217 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:33.214+0000 7ff5d7e00640 1 -- 172.21.15.32:0/2669044454 learned_addr learned my addr 172.21.15.32:0/2669044454 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:57:33.217 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:33.214+0000 7ff5dca00640 1 --2- 172.21.15.32:0/2669044454 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ff5d80b59f0 0x7ff5d815e200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:33.217 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:33.214+0000 7ff5dd400640 1 --2- 172.21.15.32:0/2669044454 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7ff5d80d0fb0 0x7ff5d815a130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:33.217 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:33.215+0000 7ff5d7e00640 1 -- 172.21.15.32:0/2669044454 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7ff5d80d0fb0 msgr2=0x7ff5d815a130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:33.217 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:33.215+0000 7ff5d7e00640 1 --2- 172.21.15.32:0/2669044454 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7ff5d80d0fb0 0x7ff5d815a130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:33.217 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:33.215+0000 7ff5d7e00640 1 -- 172.21.15.32:0/2669044454 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ff5d80b59f0 msgr2=0x7ff5d815e200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:33.217 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:33.215+0000 7ff5d7e00640 1 --2- 172.21.15.32:0/2669044454 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ff5d80b59f0 0x7ff5d815e200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:33.217 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:33.215+0000 7ff5d7e00640 1 -- 172.21.15.32:0/2669044454 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff5d815eb50 con 0x7ff5d80cf9c0 2024-08-03T20:57:33.218 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:33.215+0000 7ff5d7e00640 1 --2- 172.21.15.32:0/2669044454 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff5d80cf9c0 0x7ff5d8157b00 secure :-1 s=READY pgs=1574 cs=0 l=1 rev1=1 crypto rx=0x7ff5c400e790 tx=0x7ff5c400ec50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:57:33.218 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:33.215+0000 7ff5d4c00640 1 -- 172.21.15.32:0/2669044454 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5c400eea0 con 0x7ff5d80cf9c0 2024-08-03T20:57:33.218 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:33.215+0000 7ff5d4c00640 1 -- 172.21.15.32:0/2669044454 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7ff5c400cdb0 con 0x7ff5d80cf9c0 2024-08-03T20:57:33.218 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:33.215+0000 7ff5d4c00640 1 -- 172.21.15.32:0/2669044454 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5c4010690 con 0x7ff5d80cf9c0 2024-08-03T20:57:33.219 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:33.215+0000 7ff5df400640 1 -- 172.21.15.32:0/2669044454 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff5d815c870 con 0x7ff5d80cf9c0 2024-08-03T20:57:33.219 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:33.215+0000 7ff5df400640 1 -- 172.21.15.32:0/2669044454 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff5d815ce30 con 0x7ff5d80cf9c0 2024-08-03T20:57:33.219 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:33.217+0000 7ff5df400640 1 -- 172.21.15.32:0/2669044454 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff5d80b8e00 con 0x7ff5d80cf9c0 2024-08-03T20:57:33.219 INFO:tasks.ceph.mds.a.smithi032.stdout:starting mds.a at 2024-08-03T20:57:33.219 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:33.218+0000 7ff5d4c00640 1 -- 172.21.15.32:0/2669044454 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7ff5c4010830 con 0x7ff5d80cf9c0 2024-08-03T20:57:33.220 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:33.219+0000 7ff5d4c00640 1 --2- 172.21.15.32:0/2669044454 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7ff5b8077040 0x7ff5b8079550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:33.221 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:33.219+0000 7ff5d4c00640 1 -- 172.21.15.32:0/2669044454 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(105..105 src has 1..105) ==== 7574+0+0 (secure 0 0 0) 0x7ff5c40999d0 con 0x7ff5d80cf9c0 2024-08-03T20:57:33.221 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:33.219+0000 7ff5dca00640 1 --2- 172.21.15.32:0/2669044454 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7ff5b8077040 0x7ff5b8079550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:33.221 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:33.220+0000 7ff5dca00640 1 --2- 172.21.15.32:0/2669044454 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7ff5b8077040 0x7ff5b8079550 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7ff5cc001720 tx=0x7ff5cc009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:57:33.225 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:33.223+0000 7ff5d4c00640 1 -- 172.21.15.32:0/2669044454 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7ff5c4061d90 con 0x7ff5d80cf9c0 2024-08-03T20:57:33.472 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:33.470+0000 7ff5df400640 1 -- 172.21.15.32:0/2669044454 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "joinable", "val": "true"} v 0) -- 0x7ff5d81eb460 con 0x7ff5d80cf9c0 2024-08-03T20:57:34.385 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:34.383+0000 7ff5d4c00640 1 -- 172.21.15.32:0/2669044454 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "joinable", "val": "true"}]=0 cephfs marked joinable; MDS may join as newly active. v215) ==== 162+0+0 (secure 0 0 0) 0x7ff5c40662e0 con 0x7ff5d80cf9c0 2024-08-03T20:57:34.385 INFO:teuthology.orchestra.run.smithi032.stderr:cephfs marked joinable; MDS may join as newly active. 2024-08-03T20:57:34.389 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:34.387+0000 7ff5df400640 1 -- 172.21.15.32:0/2669044454 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7ff5b8077040 msgr2=0x7ff5b8079550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:34.389 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:34.387+0000 7ff5df400640 1 --2- 172.21.15.32:0/2669044454 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7ff5b8077040 0x7ff5b8079550 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7ff5cc001720 tx=0x7ff5cc009040 comp rx=0 tx=0).stop 2024-08-03T20:57:34.389 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:34.387+0000 7ff5df400640 1 -- 172.21.15.32:0/2669044454 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff5d80cf9c0 msgr2=0x7ff5d8157b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:34.389 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:34.387+0000 7ff5df400640 1 --2- 172.21.15.32:0/2669044454 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff5d80cf9c0 0x7ff5d8157b00 secure :-1 s=READY pgs=1574 cs=0 l=1 rev1=1 crypto rx=0x7ff5c400e790 tx=0x7ff5c400ec50 comp rx=0 tx=0).stop 2024-08-03T20:57:34.389 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:34.387+0000 7ff5df400640 1 -- 172.21.15.32:0/2669044454 shutdown_connections 2024-08-03T20:57:34.389 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:34.387+0000 7ff5df400640 1 --2- 172.21.15.32:0/2669044454 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7ff5b8077040 0x7ff5b8079550 unknown :-1 s=CLOSED pgs=641 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:34.389 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:34.387+0000 7ff5df400640 1 --2- 172.21.15.32:0/2669044454 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7ff5d80d0fb0 0x7ff5d815a130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:34.389 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:34.387+0000 7ff5df400640 1 --2- 172.21.15.32:0/2669044454 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff5d80cf9c0 0x7ff5d8157b00 unknown :-1 s=CLOSED pgs=1574 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:34.389 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:34.387+0000 7ff5df400640 1 --2- 172.21.15.32:0/2669044454 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ff5d80b59f0 0x7ff5d815e200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:34.390 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:34.387+0000 7ff5df400640 1 -- 172.21.15.32:0/2669044454 >> 172.21.15.32:0/2669044454 conn(0x7ff5d809d660 msgr2=0x7ff5d806d680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:57:34.390 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:34.387+0000 7ff5df400640 1 -- 172.21.15.32:0/2669044454 shutdown_connections 2024-08-03T20:57:34.390 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:34.387+0000 7ff5df400640 1 -- 172.21.15.32:0/2669044454 wait complete. 2024-08-03T20:57:34.396 INFO:tasks.ceph.mds.a.smithi032.stderr:2024-08-03T20:57:34.395+0000 7f0a10000640 -1 log_channel(cluster) log [ERR] : error decoding table object 'mds0_inotable': Decoder at 'virtual void InoTable::decode_state(ceph::buffer::v15_2_0::list::const_iterator&)' v=2 cannot decode v=100 minimal_decoder=101: Malformed input [buffer:3] 2024-08-03T20:57:34.397 INFO:tasks.ceph.mon.a.smithi032.stderr:2024-08-03T20:57:34.396+0000 7f4e6d600640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-08-03T20:57:34.397 INFO:tasks.ceph.mon.a.smithi032.stderr:2024-08-03T20:57:34.396+0000 7f4e6d600640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 mds daemon damaged (MDS_DAMAGE) 2024-08-03T20:57:34.422 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-08-03T20:57:34.654 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:34.652+0000 7f686ce00640 1 -- 172.21.15.32:0/2569928624 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f68680b38b0 msgr2=0x7f68681523f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:34.654 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:34.652+0000 7f686ce00640 1 --2- 172.21.15.32:0/2569928624 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f68680b38b0 0x7f68681523f0 secure :-1 s=READY pgs=1576 cs=0 l=1 rev1=1 crypto rx=0x7f68680cd6d0 tx=0x7f685802a560 comp rx=0 tx=0).stop 2024-08-03T20:57:34.654 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:34.652+0000 7f686ce00640 1 -- 172.21.15.32:0/2569928624 shutdown_connections 2024-08-03T20:57:34.655 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:34.652+0000 7f686ce00640 1 --2- 172.21.15.32:0/2569928624 >> v2:172.21.15.118:3300/0 conn(0x7f68680b4ce0 0x7f686815ef00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:34.655 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:34.652+0000 7f686ce00640 1 --2- 172.21.15.32:0/2569928624 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f68680b38b0 0x7f68681523f0 unknown :-1 s=CLOSED pgs=1576 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:34.655 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:34.652+0000 7f686ce00640 1 -- 172.21.15.32:0/2569928624 >> 172.21.15.32:0/2569928624 conn(0x7f68680b1bd0 msgr2=0x7f68680b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:57:34.655 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:34.652+0000 7f686ce00640 1 -- 172.21.15.32:0/2569928624 shutdown_connections 2024-08-03T20:57:34.655 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:34.652+0000 7f686ce00640 1 -- 172.21.15.32:0/2569928624 wait complete. 2024-08-03T20:57:34.656 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:34.655+0000 7f686ce00640 1 Processor -- start 2024-08-03T20:57:34.656 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:34.655+0000 7f686ce00640 1 -- start start 2024-08-03T20:57:34.659 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:34.656+0000 7f686ce00640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f68680b38b0 0x7f6868150a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:34.659 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:34.656+0000 7f686ce00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f68680b4ce0 0x7f68681f76c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:34.659 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:34.656+0000 7f686ce00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f686815f440 0x7f6868158110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:34.659 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:34.656+0000 7f686ce00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f686815c020 con 0x7f68680b4ce0 2024-08-03T20:57:34.659 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:34.656+0000 7f686ce00640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f68680cf240 con 0x7f686815f440 2024-08-03T20:57:34.659 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:34.656+0000 7f686ce00640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7f686815c1a0 con 0x7f68680b38b0 2024-08-03T20:57:34.659 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:34.656+0000 7f6865400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f68680b4ce0 0x7f68681f76c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:34.660 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:34.657+0000 7f6865400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f68680b4ce0 0x7f68681f76c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:33890/0 (socket says 172.21.15.32:33890) 2024-08-03T20:57:34.660 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:34.657+0000 7f6865e00640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f68680b38b0 0x7f6868150a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:34.660 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:34.657+0000 7f6865400640 1 -- 172.21.15.32:0/1479492615 learned_addr learned my addr 172.21.15.32:0/1479492615 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:57:34.660 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:34.657+0000 7f6866800640 1 --2- 172.21.15.32:0/1479492615 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f686815f440 0x7f6868158110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:34.660 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:34.657+0000 7f6865400640 1 -- 172.21.15.32:0/1479492615 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f68680b38b0 msgr2=0x7f6868150a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:34.660 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:34.657+0000 7f6865400640 1 --2- 172.21.15.32:0/1479492615 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f68680b38b0 0x7f6868150a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:34.660 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:34.657+0000 7f6865400640 1 -- 172.21.15.32:0/1479492615 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f686815f440 msgr2=0x7f6868158110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:34.660 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:34.657+0000 7f6865400640 1 --2- 172.21.15.32:0/1479492615 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f686815f440 0x7f6868158110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:34.660 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:34.657+0000 7f6865400640 1 -- 172.21.15.32:0/1479492615 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f68681f7c00 con 0x7f68680b4ce0 2024-08-03T20:57:34.660 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:34.657+0000 7f6865400640 1 --2- 172.21.15.32:0/1479492615 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f68680b4ce0 0x7f68681f76c0 secure :-1 s=READY pgs=1577 cs=0 l=1 rev1=1 crypto rx=0x7f685400d780 tx=0x7f685400dc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:57:34.660 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:34.657+0000 7f685e000640 1 -- 172.21.15.32:0/1479492615 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6854017070 con 0x7f68680b4ce0 2024-08-03T20:57:34.660 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:34.658+0000 7f685e000640 1 -- 172.21.15.32:0/1479492615 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f685400de90 con 0x7f68680b4ce0 2024-08-03T20:57:34.661 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:34.658+0000 7f685e000640 1 -- 172.21.15.32:0/1479492615 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68540099d0 con 0x7f68680b4ce0 2024-08-03T20:57:34.661 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:34.658+0000 7f686ce00640 1 -- 172.21.15.32:0/1479492615 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f68681f3bb0 con 0x7f68680b4ce0 2024-08-03T20:57:34.661 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:34.658+0000 7f686ce00640 1 -- 172.21.15.32:0/1479492615 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f68681f4170 con 0x7f68680b4ce0 2024-08-03T20:57:34.661 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:34.659+0000 7f686ce00640 1 -- 172.21.15.32:0/1479492615 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f686815be40 con 0x7f68680b4ce0 2024-08-03T20:57:34.661 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:34.660+0000 7f685e000640 1 -- 172.21.15.32:0/1479492615 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f6854009b70 con 0x7f68680b4ce0 2024-08-03T20:57:34.663 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:34.661+0000 7f685e000640 1 --2- 172.21.15.32:0/1479492615 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f6848077040 0x7f6848079550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:34.663 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:34.661+0000 7f685e000640 1 -- 172.21.15.32:0/1479492615 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(106..106 src has 1..106) ==== 7660+0+0 (secure 0 0 0) 0x7f6854099270 con 0x7f68680b4ce0 2024-08-03T20:57:34.663 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:34.662+0000 7f6865e00640 1 --2- 172.21.15.32:0/1479492615 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f6848077040 0x7f6848079550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:34.664 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:34.663+0000 7f6865e00640 1 --2- 172.21.15.32:0/1479492615 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f6848077040 0x7f6848079550 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7f6858033000 tx=0x7f6858004730 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:57:34.667 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:34.666+0000 7f685e000640 1 -- 172.21.15.32:0/1479492615 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7f686815be40 con 0x7f68680b4ce0 2024-08-03T20:57:34.914 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:34.913+0000 7f686ce00640 1 -- 172.21.15.32:0/1479492615 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f686815c660 con 0x7f68680b4ce0 2024-08-03T20:57:34.915 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:34.914+0000 7f685e000640 1 -- 172.21.15.32:0/1479492615 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 217 v217) ==== 95+0+4391 (secure 0 0 0) 0x7f6854020020 con 0x7f68680b4ce0 2024-08-03T20:57:34.915 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-03T20:57:34.915 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":217,"btime":"2024-08-03T20:57:34:397223+0000","default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6439,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6835/531080160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6834","nonce":531080160},{"type":"v1","addr":"172.21.15.118:6835","nonce":531080160}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":84},{"gid":6454,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6837/2796882319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6836","nonce":2796882319},{"type":"v1","addr":"172.21.15.118:6837","nonce":2796882319}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":86},{"gid":6813,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6834/2686933890","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":2686933890},{"type":"v1","addr":"172.21.15.32:6834","nonce":2686933890}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":85}],"filesystems":[{"mdsmap":{"epoch":217,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-08-03T20:54:26.950518+0000","modified":"2024-08-03T20:57:34.397124+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":106,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":6}]} 2024-08-03T20:57:34.916 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 217 2024-08-03T20:57:34.918 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:34.916+0000 7f686ce00640 1 -- 172.21.15.32:0/1479492615 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f6848077040 msgr2=0x7f6848079550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:34.918 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:34.916+0000 7f686ce00640 1 --2- 172.21.15.32:0/1479492615 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f6848077040 0x7f6848079550 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7f6858033000 tx=0x7f6858004730 comp rx=0 tx=0).stop 2024-08-03T20:57:34.918 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:34.916+0000 7f686ce00640 1 -- 172.21.15.32:0/1479492615 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f68680b4ce0 msgr2=0x7f68681f76c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:34.918 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:34.916+0000 7f686ce00640 1 --2- 172.21.15.32:0/1479492615 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f68680b4ce0 0x7f68681f76c0 secure :-1 s=READY pgs=1577 cs=0 l=1 rev1=1 crypto rx=0x7f685400d780 tx=0x7f685400dc40 comp rx=0 tx=0).stop 2024-08-03T20:57:34.918 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:34.916+0000 7f686ce00640 1 -- 172.21.15.32:0/1479492615 shutdown_connections 2024-08-03T20:57:34.918 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:34.916+0000 7f686ce00640 1 --2- 172.21.15.32:0/1479492615 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f6848077040 0x7f6848079550 unknown :-1 s=CLOSED pgs=643 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:34.918 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:34.916+0000 7f686ce00640 1 --2- 172.21.15.32:0/1479492615 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f686815f440 0x7f6868158110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:34.918 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:34.916+0000 7f686ce00640 1 --2- 172.21.15.32:0/1479492615 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f68680b4ce0 0x7f68681f76c0 unknown :-1 s=CLOSED pgs=1577 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:34.918 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:34.916+0000 7f686ce00640 1 --2- 172.21.15.32:0/1479492615 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f68680b38b0 0x7f6868150a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:34.918 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:34.916+0000 7f686ce00640 1 -- 172.21.15.32:0/1479492615 >> 172.21.15.32:0/1479492615 conn(0x7f68680b1bd0 msgr2=0x7f6868151c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:57:34.918 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:34.916+0000 7f686ce00640 1 -- 172.21.15.32:0/1479492615 shutdown_connections 2024-08-03T20:57:34.918 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:34.916+0000 7f686ce00640 1 -- 172.21.15.32:0/1479492615 wait complete. 2024-08-03T20:57:34.941 DEBUG:tasks.ceph_test_case:wait_until_true: success in 0s and 0 retries 2024-08-03T20:57:34.941 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-08-03T20:57:35.169 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:35.167+0000 7f1b2a600640 1 -- 172.21.15.32:0/1281848981 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1b240d4bb0 msgr2=0x7f1b240d1c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:35.169 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:35.167+0000 7f1b2a600640 1 --2- 172.21.15.32:0/1281848981 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1b240d4bb0 0x7f1b240d1c60 secure :-1 s=READY pgs=1579 cs=0 l=1 rev1=1 crypto rx=0x7f1b240d5190 tx=0x7f1b1402a4b0 comp rx=0 tx=0).stop 2024-08-03T20:57:35.169 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:35.167+0000 7f1b2a600640 1 -- 172.21.15.32:0/1281848981 shutdown_connections 2024-08-03T20:57:35.169 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:35.167+0000 7f1b2a600640 1 --2- 172.21.15.32:0/1281848981 >> v2:172.21.15.118:3300/0 conn(0x7f1b240d2290 0x7f1b240d26a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:35.169 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:35.167+0000 7f1b2a600640 1 --2- 172.21.15.32:0/1281848981 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1b240d4bb0 0x7f1b240d1c60 unknown :-1 s=CLOSED pgs=1579 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:35.169 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:35.167+0000 7f1b2a600640 1 -- 172.21.15.32:0/1281848981 >> 172.21.15.32:0/1281848981 conn(0x7f1b240d4090 msgr2=0x7f1b240d44a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:57:35.169 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:35.167+0000 7f1b2a600640 1 -- 172.21.15.32:0/1281848981 shutdown_connections 2024-08-03T20:57:35.169 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:35.167+0000 7f1b2a600640 1 -- 172.21.15.32:0/1281848981 wait complete. 2024-08-03T20:57:35.170 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:35.169+0000 7f1b2a600640 1 Processor -- start 2024-08-03T20:57:35.171 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:35.169+0000 7f1b2a600640 1 -- start start 2024-08-03T20:57:35.174 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:35.171+0000 7f1b2a600640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f1b240d0aa0 0x7f1b2415e840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:35.174 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:35.171+0000 7f1b2a600640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f1b240d2290 0x7f1b24166df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:35.174 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:35.171+0000 7f1b2a600640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1b240d4bb0 0x7f1b24167330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:35.174 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:35.171+0000 7f1b2a600640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f1b240d15c0 con 0x7f1b240d4bb0 2024-08-03T20:57:35.174 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:35.171+0000 7f1b2a600640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f1b240cfe50 con 0x7f1b240d0aa0 2024-08-03T20:57:35.174 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:35.171+0000 7f1b2a600640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7f1b240d1740 con 0x7f1b240d2290 2024-08-03T20:57:35.174 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:35.171+0000 7f1b29c00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1b240d4bb0 0x7f1b24167330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:35.174 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:35.171+0000 7f1b29c00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1b240d4bb0 0x7f1b24167330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:33912/0 (socket says 172.21.15.32:33912) 2024-08-03T20:57:35.174 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:35.171+0000 7f1b29c00640 1 -- 172.21.15.32:0/3683805295 learned_addr learned my addr 172.21.15.32:0/3683805295 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:57:35.174 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:35.171+0000 7f1b29200640 1 --2- 172.21.15.32:0/3683805295 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f1b240d0aa0 0x7f1b2415e840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:35.175 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:35.171+0000 7f1b1be00640 1 --2- 172.21.15.32:0/3683805295 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f1b240d2290 0x7f1b24166df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:35.175 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:35.172+0000 7f1b29c00640 1 -- 172.21.15.32:0/3683805295 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f1b240d2290 msgr2=0x7f1b24166df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:35.175 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:35.172+0000 7f1b29c00640 1 --2- 172.21.15.32:0/3683805295 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f1b240d2290 0x7f1b24166df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:35.175 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:35.172+0000 7f1b29c00640 1 -- 172.21.15.32:0/3683805295 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f1b240d0aa0 msgr2=0x7f1b2415e840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:35.175 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:35.172+0000 7f1b29c00640 1 --2- 172.21.15.32:0/3683805295 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f1b240d0aa0 0x7f1b2415e840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:35.175 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:35.172+0000 7f1b29c00640 1 -- 172.21.15.32:0/3683805295 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1b24165930 con 0x7f1b240d4bb0 2024-08-03T20:57:35.175 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:35.172+0000 7f1b29c00640 1 --2- 172.21.15.32:0/3683805295 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1b240d4bb0 0x7f1b24167330 secure :-1 s=READY pgs=1580 cs=0 l=1 rev1=1 crypto rx=0x7f1b1c00c730 tx=0x7f1b1c00cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:57:35.175 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:35.172+0000 7f1b21600640 1 -- 172.21.15.32:0/3683805295 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b1c017070 con 0x7f1b240d4bb0 2024-08-03T20:57:35.175 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:35.172+0000 7f1b21600640 1 -- 172.21.15.32:0/3683805295 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f1b1c003c60 con 0x7f1b240d4bb0 2024-08-03T20:57:35.175 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:35.172+0000 7f1b21600640 1 -- 172.21.15.32:0/3683805295 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b1c0052d0 con 0x7f1b240d4bb0 2024-08-03T20:57:35.175 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:35.172+0000 7f1b2a600640 1 -- 172.21.15.32:0/3683805295 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1b240b8ef0 con 0x7f1b240d4bb0 2024-08-03T20:57:35.175 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:35.172+0000 7f1b2a600640 1 -- 172.21.15.32:0/3683805295 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1b240b87e0 con 0x7f1b240d4bb0 2024-08-03T20:57:35.176 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:35.174+0000 7f1b2a600640 1 -- 172.21.15.32:0/3683805295 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1b240d26a0 con 0x7f1b240d4bb0 2024-08-03T20:57:35.176 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:35.175+0000 7f1b21600640 1 -- 172.21.15.32:0/3683805295 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f1b1c01a070 con 0x7f1b240d4bb0 2024-08-03T20:57:35.177 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:35.176+0000 7f1b21600640 1 --2- 172.21.15.32:0/3683805295 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f1b00077010 0x7f1b00079520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:35.177 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:35.176+0000 7f1b21600640 1 -- 172.21.15.32:0/3683805295 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(106..106 src has 1..106) ==== 7660+0+0 (secure 0 0 0) 0x7f1b1c098720 con 0x7f1b240d4bb0 2024-08-03T20:57:35.178 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:35.176+0000 7f1b29200640 1 --2- 172.21.15.32:0/3683805295 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f1b00077010 0x7f1b00079520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:35.178 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:35.177+0000 7f1b29200640 1 --2- 172.21.15.32:0/3683805295 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f1b00077010 0x7f1b00079520 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7f1b14034000 tx=0x7f1b14004730 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:57:35.182 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:35.181+0000 7f1b21600640 1 -- 172.21.15.32:0/3683805295 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7f1b240d2be0 con 0x7f1b240d4bb0 2024-08-03T20:57:35.429 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:35.427+0000 7f1b2a600640 1 -- 172.21.15.32:0/3683805295 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f1b240b1a00 con 0x7f1b240d4bb0 2024-08-03T20:57:35.430 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:35.428+0000 7f1b21600640 1 -- 172.21.15.32:0/3683805295 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 217 v217) ==== 95+0+4391 (secure 0 0 0) 0x7f1b1c0693b0 con 0x7f1b240d4bb0 2024-08-03T20:57:35.430 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-03T20:57:35.430 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":217,"btime":"2024-08-03T20:57:34:397223+0000","default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6439,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6835/531080160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6834","nonce":531080160},{"type":"v1","addr":"172.21.15.118:6835","nonce":531080160}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":84},{"gid":6454,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6837/2796882319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6836","nonce":2796882319},{"type":"v1","addr":"172.21.15.118:6837","nonce":2796882319}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":86},{"gid":6813,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6834/2686933890","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":2686933890},{"type":"v1","addr":"172.21.15.32:6834","nonce":2686933890}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":85}],"filesystems":[{"mdsmap":{"epoch":217,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-08-03T20:54:26.950518+0000","modified":"2024-08-03T20:57:34.397124+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":106,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":6}]} 2024-08-03T20:57:35.430 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 217 2024-08-03T20:57:35.432 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:35.431+0000 7f1b2a600640 1 -- 172.21.15.32:0/3683805295 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f1b00077010 msgr2=0x7f1b00079520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:35.432 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:35.431+0000 7f1b2a600640 1 --2- 172.21.15.32:0/3683805295 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f1b00077010 0x7f1b00079520 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7f1b14034000 tx=0x7f1b14004730 comp rx=0 tx=0).stop 2024-08-03T20:57:35.433 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:35.431+0000 7f1b2a600640 1 -- 172.21.15.32:0/3683805295 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1b240d4bb0 msgr2=0x7f1b24167330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:35.433 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:35.431+0000 7f1b2a600640 1 --2- 172.21.15.32:0/3683805295 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1b240d4bb0 0x7f1b24167330 secure :-1 s=READY pgs=1580 cs=0 l=1 rev1=1 crypto rx=0x7f1b1c00c730 tx=0x7f1b1c00cbf0 comp rx=0 tx=0).stop 2024-08-03T20:57:35.433 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:35.431+0000 7f1b2a600640 1 -- 172.21.15.32:0/3683805295 shutdown_connections 2024-08-03T20:57:35.433 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:35.431+0000 7f1b2a600640 1 --2- 172.21.15.32:0/3683805295 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f1b00077010 0x7f1b00079520 unknown :-1 s=CLOSED pgs=644 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:35.433 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:35.431+0000 7f1b2a600640 1 --2- 172.21.15.32:0/3683805295 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1b240d4bb0 0x7f1b24167330 unknown :-1 s=CLOSED pgs=1580 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:35.433 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:35.431+0000 7f1b2a600640 1 --2- 172.21.15.32:0/3683805295 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f1b240d2290 0x7f1b24166df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:35.433 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:35.431+0000 7f1b2a600640 1 --2- 172.21.15.32:0/3683805295 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f1b240d0aa0 0x7f1b2415e840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:35.433 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:35.431+0000 7f1b2a600640 1 -- 172.21.15.32:0/3683805295 >> 172.21.15.32:0/3683805295 conn(0x7f1b240d4090 msgr2=0x7f1b241684e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:57:35.433 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:35.431+0000 7f1b2a600640 1 -- 172.21.15.32:0/3683805295 shutdown_connections 2024-08-03T20:57:35.433 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:35.431+0000 7f1b2a600640 1 -- 172.21.15.32:0/3683805295 wait complete. 2024-08-03T20:57:35.455 DEBUG:tasks.ceph_test_case:wait_until_true: success in 0s and 0 retries 2024-08-03T20:57:35.455 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-08-03T20:57:35.683 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:35.681+0000 7fb26f600640 1 -- 172.21.15.32:0/2348202890 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb24c003c60 msgr2=0x7fb24c0040b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:35.683 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:35.681+0000 7fb26f600640 1 --2- 172.21.15.32:0/2348202890 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb24c003c60 0x7fb24c0040b0 secure :-1 s=READY pgs=1582 cs=0 l=1 rev1=1 crypto rx=0x7fb258006f10 tx=0x7fb24c005830 comp rx=0 tx=0).stop 2024-08-03T20:57:35.684 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:35.681+0000 7fb26d600640 1 -- 172.21.15.32:0/2348202890 reap_dead start 2024-08-03T20:57:35.684 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:35.681+0000 7fb26f600640 1 -- 172.21.15.32:0/2348202890 shutdown_connections 2024-08-03T20:57:35.684 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:35.682+0000 7fb26f600640 1 -- 172.21.15.32:0/2348202890 >> 172.21.15.32:0/2348202890 conn(0x7fb2680b1bd0 msgr2=0x7fb2680b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:57:35.684 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:35.682+0000 7fb26f600640 1 -- 172.21.15.32:0/2348202890 shutdown_connections 2024-08-03T20:57:35.684 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:35.682+0000 7fb26f600640 1 -- 172.21.15.32:0/2348202890 wait complete. 2024-08-03T20:57:35.685 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:35.684+0000 7fb26f600640 1 Processor -- start 2024-08-03T20:57:35.685 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:35.684+0000 7fb26f600640 1 -- start start 2024-08-03T20:57:35.688 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:35.685+0000 7fb26f600640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fb24c003c60 0x7fb2680cba30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:35.688 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:35.685+0000 7fb26f600640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb2680ad8c0 0x7fb2681436b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:35.689 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:35.685+0000 7fb26f600640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb2680adcd0 0x7fb268143bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:35.689 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:35.685+0000 7fb26f600640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fb2680cf840 con 0x7fb2680ad8c0 2024-08-03T20:57:35.689 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:35.685+0000 7fb26f600640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7fb268145f30 con 0x7fb2680adcd0 2024-08-03T20:57:35.689 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:35.685+0000 7fb26f600640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7fb2680cf9c0 con 0x7fb24c003c60 2024-08-03T20:57:35.689 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:35.686+0000 7fb25fe00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb2680ad8c0 0x7fb2681436b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:35.689 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:35.686+0000 7fb26cc00640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fb24c003c60 0x7fb2680cba30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:35.689 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:35.686+0000 7fb25fe00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb2680ad8c0 0x7fb2681436b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:33918/0 (socket says 172.21.15.32:33918) 2024-08-03T20:57:35.689 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:35.686+0000 7fb25fe00640 1 -- 172.21.15.32:0/23498199 learned_addr learned my addr 172.21.15.32:0/23498199 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:57:35.689 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:35.686+0000 7fb26d600640 1 --2- 172.21.15.32:0/23498199 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb2680adcd0 0x7fb268143bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:35.689 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:35.686+0000 7fb25fe00640 1 -- 172.21.15.32:0/23498199 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fb24c003c60 msgr2=0x7fb2680cba30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:35.689 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:35.686+0000 7fb25fe00640 1 --2- 172.21.15.32:0/23498199 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fb24c003c60 0x7fb2680cba30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:35.689 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:35.686+0000 7fb25fe00640 1 -- 172.21.15.32:0/23498199 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb2680adcd0 msgr2=0x7fb268143bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:35.689 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:35.686+0000 7fb25fe00640 1 --2- 172.21.15.32:0/23498199 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb2680adcd0 0x7fb268143bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:35.690 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:35.686+0000 7fb25fe00640 1 -- 172.21.15.32:0/23498199 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb2680ae4a0 con 0x7fb2680ad8c0 2024-08-03T20:57:35.690 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:35.686+0000 7fb25fe00640 1 --2- 172.21.15.32:0/23498199 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb2680ad8c0 0x7fb2681436b0 secure :-1 s=READY pgs=1583 cs=0 l=1 rev1=1 crypto rx=0x7fb2500119a0 tx=0x7fb250011e60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:57:35.690 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:35.687+0000 7fb265600640 1 -- 172.21.15.32:0/23498199 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb25001e070 con 0x7fb2680ad8c0 2024-08-03T20:57:35.690 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:35.687+0000 7fb265600640 1 -- 172.21.15.32:0/23498199 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fb25000fa00 con 0x7fb2680ad8c0 2024-08-03T20:57:35.690 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:35.687+0000 7fb265600640 1 -- 172.21.15.32:0/23498199 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb25000fe10 con 0x7fb2680ad8c0 2024-08-03T20:57:35.690 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:35.687+0000 7fb26f600640 1 -- 172.21.15.32:0/23498199 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb2680ae780 con 0x7fb2680ad8c0 2024-08-03T20:57:35.690 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:35.687+0000 7fb26f600640 1 -- 172.21.15.32:0/23498199 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb2680aed40 con 0x7fb2680ad8c0 2024-08-03T20:57:35.691 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:35.688+0000 7fb26f600640 1 -- 172.21.15.32:0/23498199 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb2680ae210 con 0x7fb2680ad8c0 2024-08-03T20:57:35.691 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:35.689+0000 7fb265600640 1 -- 172.21.15.32:0/23498199 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7fb250021070 con 0x7fb2680ad8c0 2024-08-03T20:57:35.691 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:35.690+0000 7fb265600640 1 --2- 172.21.15.32:0/23498199 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fb244077010 0x7fb244079520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:35.692 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:35.690+0000 7fb265600640 1 -- 172.21.15.32:0/23498199 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(106..106 src has 1..106) ==== 7660+0+0 (secure 0 0 0) 0x7fb250099c30 con 0x7fb2680ad8c0 2024-08-03T20:57:35.692 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:35.691+0000 7fb26cc00640 1 --2- 172.21.15.32:0/23498199 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fb244077010 0x7fb244079520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:35.693 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:35.692+0000 7fb26cc00640 1 --2- 172.21.15.32:0/23498199 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fb244077010 0x7fb244079520 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7fb258033000 tx=0x7fb25802d000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:57:35.696 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:35.695+0000 7fb265600640 1 -- 172.21.15.32:0/23498199 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7fb250062030 con 0x7fb2680ad8c0 2024-08-03T20:57:35.937 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:35.936+0000 7fb26f600640 1 -- 172.21.15.32:0/23498199 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb268074860 con 0x7fb2680ad8c0 2024-08-03T20:57:35.938 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:35.937+0000 7fb265600640 1 -- 172.21.15.32:0/23498199 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 217 v217) ==== 95+0+4391 (secure 0 0 0) 0x7fb250066580 con 0x7fb2680ad8c0 2024-08-03T20:57:35.938 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-03T20:57:35.938 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":217,"btime":"2024-08-03T20:57:34:397223+0000","default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6439,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6835/531080160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6834","nonce":531080160},{"type":"v1","addr":"172.21.15.118:6835","nonce":531080160}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":84},{"gid":6454,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6837/2796882319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6836","nonce":2796882319},{"type":"v1","addr":"172.21.15.118:6837","nonce":2796882319}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":86},{"gid":6813,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6834/2686933890","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":2686933890},{"type":"v1","addr":"172.21.15.32:6834","nonce":2686933890}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":85}],"filesystems":[{"mdsmap":{"epoch":217,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-08-03T20:54:26.950518+0000","modified":"2024-08-03T20:57:34.397124+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":106,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":6}]} 2024-08-03T20:57:35.939 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 217 2024-08-03T20:57:35.940 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:35.939+0000 7fb26f600640 1 -- 172.21.15.32:0/23498199 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fb244077010 msgr2=0x7fb244079520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:35.940 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:35.939+0000 7fb26f600640 1 --2- 172.21.15.32:0/23498199 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fb244077010 0x7fb244079520 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7fb258033000 tx=0x7fb25802d000 comp rx=0 tx=0).stop 2024-08-03T20:57:35.941 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:35.939+0000 7fb26f600640 1 -- 172.21.15.32:0/23498199 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb2680ad8c0 msgr2=0x7fb2681436b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:35.941 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:35.939+0000 7fb26f600640 1 --2- 172.21.15.32:0/23498199 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb2680ad8c0 0x7fb2681436b0 secure :-1 s=READY pgs=1583 cs=0 l=1 rev1=1 crypto rx=0x7fb2500119a0 tx=0x7fb250011e60 comp rx=0 tx=0).stop 2024-08-03T20:57:35.941 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:35.939+0000 7fb26f600640 1 -- 172.21.15.32:0/23498199 shutdown_connections 2024-08-03T20:57:35.941 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:35.939+0000 7fb26f600640 1 --2- 172.21.15.32:0/23498199 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fb244077010 0x7fb244079520 unknown :-1 s=CLOSED pgs=645 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:35.941 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:35.939+0000 7fb26f600640 1 --2- 172.21.15.32:0/23498199 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb2680adcd0 0x7fb268143bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:35.941 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:35.939+0000 7fb26f600640 1 --2- 172.21.15.32:0/23498199 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb2680ad8c0 0x7fb2681436b0 unknown :-1 s=CLOSED pgs=1583 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:35.941 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:35.939+0000 7fb26f600640 1 --2- 172.21.15.32:0/23498199 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fb24c003c60 0x7fb2680cba30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:35.941 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:35.939+0000 7fb26f600640 1 -- 172.21.15.32:0/23498199 >> 172.21.15.32:0/23498199 conn(0x7fb2680b1bd0 msgr2=0x7fb2680afb40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:57:35.941 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:35.939+0000 7fb26f600640 1 -- 172.21.15.32:0/23498199 shutdown_connections 2024-08-03T20:57:35.941 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:35.939+0000 7fb26f600640 1 -- 172.21.15.32:0/23498199 wait complete. 2024-08-03T20:57:35.964 INFO:tasks.cephfs.test_damage:Result: Mutation 'Corrupt mds0_inotable' led to DAMAGED state 2024-08-03T20:57:35.964 INFO:tasks.cephfs.test_damage:Applying mutation 'Corrupt 400.00000000' 2024-08-03T20:57:35.964 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-08-03T20:57:35.964 DEBUG:teuthology.orchestra.run.smithi032:> dd if=/proc/self/mounts of=/dev/stdout 2024-08-03T20:57:36.000 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053453,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6499208k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 none /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249604k,nr_inodes=812401,mode=700,uid=1000,gid=1316 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6499208k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-08-03T20:57:36.000 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi032.front.sepia.ceph.com 2024-08-03T20:57:36.000 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-08-03T20:57:36.000 DEBUG:teuthology.orchestra.run.smithi032:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-08-03T20:57:36.075 INFO:teuthology.orchestra.run.smithi032.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.0': No such file or directory 2024-08-03T20:57:36.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-03T20:57:36.076 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-08-03T20:57:36.347 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:36.345+0000 7f40eb200640 1 -- 172.21.15.32:0/3961535950 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f40d0005350 msgr2=0x7f40d000b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:36.347 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:36.345+0000 7f40eb200640 1 --2- 172.21.15.32:0/3961535950 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f40d0005350 0x7f40d000b9c0 secure :-1 s=READY pgs=1585 cs=0 l=1 rev1=1 crypto rx=0x7f40ec0d0610 tx=0x7f40d0005830 comp rx=0 tx=0).stop 2024-08-03T20:57:36.347 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:36.345+0000 7f40eb200640 1 -- 172.21.15.32:0/3961535950 shutdown_connections 2024-08-03T20:57:36.347 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:36.345+0000 7f40eb200640 1 --2- 172.21.15.32:0/3961535950 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f40d0005350 0x7f40d000b9c0 unknown :-1 s=CLOSED pgs=1585 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:36.347 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:36.345+0000 7f40eb200640 1 -- 172.21.15.32:0/3961535950 >> 172.21.15.32:0/3961535950 conn(0x7f40ec0b2120 msgr2=0x7f40ec0cfd60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:57:36.347 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:36.345+0000 7f40eb200640 1 -- 172.21.15.32:0/3961535950 shutdown_connections 2024-08-03T20:57:36.348 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:36.345+0000 7f40eb200640 1 -- 172.21.15.32:0/3961535950 wait complete. 2024-08-03T20:57:36.349 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:36.347+0000 7f40eb200640 1 Processor -- start 2024-08-03T20:57:36.349 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:36.348+0000 7f40eb200640 1 -- start start 2024-08-03T20:57:36.352 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:36.349+0000 7f40eb200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f40d0005350 0x7f40ec09bfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:36.352 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:36.349+0000 7f40eb200640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f40ec1f3f50 0x7f40ec1f6850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:36.352 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:36.349+0000 7f40eb200640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f40ec1f6d90 0x7f40ec15ef70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:36.352 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:36.349+0000 7f40eb200640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f40ec0ad190 con 0x7f40d0005350 2024-08-03T20:57:36.352 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:36.349+0000 7f40eb200640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f40ec074520 con 0x7f40ec1f6d90 2024-08-03T20:57:36.352 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:36.349+0000 7f40eb200640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7f40ec0ad310 con 0x7f40ec1f3f50 2024-08-03T20:57:36.352 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:36.349+0000 7f40e9e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f40d0005350 0x7f40ec09bfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:36.352 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:36.349+0000 7f40e9e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f40d0005350 0x7f40ec09bfa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:33936/0 (socket says 172.21.15.32:33936) 2024-08-03T20:57:36.352 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:36.349+0000 7f40e9e00640 1 -- 172.21.15.32:0/1001194546 learned_addr learned my addr 172.21.15.32:0/1001194546 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:57:36.352 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:36.350+0000 7f40ea800640 1 --2- 172.21.15.32:0/1001194546 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f40ec1f6d90 0x7f40ec15ef70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:36.352 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:36.350+0000 7f40e3e00640 1 --2- 172.21.15.32:0/1001194546 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f40ec1f3f50 0x7f40ec1f6850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:36.352 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:36.350+0000 7f40e9e00640 1 -- 172.21.15.32:0/1001194546 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f40ec1f3f50 msgr2=0x7f40ec1f6850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:36.353 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:36.350+0000 7f40e9e00640 1 --2- 172.21.15.32:0/1001194546 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f40ec1f3f50 0x7f40ec1f6850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:36.353 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:36.350+0000 7f40e9e00640 1 -- 172.21.15.32:0/1001194546 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f40ec1f6d90 msgr2=0x7f40ec15ef70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:36.353 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:36.350+0000 7f40e9e00640 1 --2- 172.21.15.32:0/1001194546 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f40ec1f6d90 0x7f40ec15ef70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:36.353 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:36.350+0000 7f40e9e00640 1 -- 172.21.15.32:0/1001194546 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f40ec1f7510 con 0x7f40d0005350 2024-08-03T20:57:36.353 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:36.350+0000 7f40e3e00640 1 --2- 172.21.15.32:0/1001194546 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f40ec1f3f50 0x7f40ec1f6850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-03T20:57:36.353 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:36.350+0000 7f40e9e00640 1 --2- 172.21.15.32:0/1001194546 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f40d0005350 0x7f40ec09bfa0 secure :-1 s=READY pgs=1586 cs=0 l=1 rev1=1 crypto rx=0x7f40dc011ae0 tx=0x7f40dc009e40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:57:36.353 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:36.350+0000 7f40e2000640 1 -- 172.21.15.32:0/1001194546 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40dc01e070 con 0x7f40d0005350 2024-08-03T20:57:36.353 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:36.350+0000 7f40e2000640 1 -- 172.21.15.32:0/1001194546 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f40dc004480 con 0x7f40d0005350 2024-08-03T20:57:36.353 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:36.350+0000 7f40eb200640 1 -- 172.21.15.32:0/1001194546 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f40ec1f3470 con 0x7f40d0005350 2024-08-03T20:57:36.353 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:36.350+0000 7f40e2000640 1 -- 172.21.15.32:0/1001194546 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40dc00fc60 con 0x7f40d0005350 2024-08-03T20:57:36.353 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:36.351+0000 7f40eb200640 1 -- 172.21.15.32:0/1001194546 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f40ec1f3a30 con 0x7f40d0005350 2024-08-03T20:57:36.354 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:36.352+0000 7f40eb200640 1 -- 172.21.15.32:0/1001194546 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f40ec1f3260 con 0x7f40d0005350 2024-08-03T20:57:36.354 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:36.353+0000 7f40e2000640 1 -- 172.21.15.32:0/1001194546 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f40dc021070 con 0x7f40d0005350 2024-08-03T20:57:36.355 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:36.354+0000 7f40e2000640 1 --2- 172.21.15.32:0/1001194546 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f40c8077010 0x7f40c8079520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:36.355 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:36.354+0000 7f40e2000640 1 -- 172.21.15.32:0/1001194546 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(106..106 src has 1..106) ==== 7660+0+0 (secure 0 0 0) 0x7f40dc099af0 con 0x7f40d0005350 2024-08-03T20:57:36.356 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:36.354+0000 7f40e3e00640 1 --2- 172.21.15.32:0/1001194546 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f40c8077010 0x7f40c8079520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:36.356 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:36.355+0000 7f40e3e00640 1 --2- 172.21.15.32:0/1001194546 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f40c8077010 0x7f40c8079520 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7f40d4001860 tx=0x7f40d4001110 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:57:36.360 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:36.358+0000 7f40e2000640 1 -- 172.21.15.32:0/1001194546 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7f40dc061ef0 con 0x7f40d0005350 2024-08-03T20:57:36.608 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:36.606+0000 7f40eb200640 1 -- 172.21.15.32:0/1001194546 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7f40ec06f520 con 0x7f40d0005350 2024-08-03T20:57:37.401 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:37.399+0000 7f40e2000640 1 -- 172.21.15.32:0/1001194546 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v218) ==== 161+0+0 (secure 0 0 0) 0x7f40dc066440 con 0x7f40d0005350 2024-08-03T20:57:37.401 INFO:teuthology.orchestra.run.smithi032.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-08-03T20:57:37.404 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:37.402+0000 7f40eb200640 1 -- 172.21.15.32:0/1001194546 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f40c8077010 msgr2=0x7f40c8079520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:37.404 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:37.402+0000 7f40eb200640 1 --2- 172.21.15.32:0/1001194546 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f40c8077010 0x7f40c8079520 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7f40d4001860 tx=0x7f40d4001110 comp rx=0 tx=0).stop 2024-08-03T20:57:37.404 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:37.402+0000 7f40eb200640 1 -- 172.21.15.32:0/1001194546 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f40d0005350 msgr2=0x7f40ec09bfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:37.404 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:37.402+0000 7f40eb200640 1 --2- 172.21.15.32:0/1001194546 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f40d0005350 0x7f40ec09bfa0 secure :-1 s=READY pgs=1586 cs=0 l=1 rev1=1 crypto rx=0x7f40dc011ae0 tx=0x7f40dc009e40 comp rx=0 tx=0).stop 2024-08-03T20:57:37.404 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:37.402+0000 7f40eb200640 1 -- 172.21.15.32:0/1001194546 shutdown_connections 2024-08-03T20:57:37.404 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:37.402+0000 7f40eb200640 1 --2- 172.21.15.32:0/1001194546 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f40c8077010 0x7f40c8079520 unknown :-1 s=CLOSED pgs=646 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:37.404 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:37.402+0000 7f40eb200640 1 --2- 172.21.15.32:0/1001194546 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f40ec1f6d90 0x7f40ec15ef70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:37.405 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:37.402+0000 7f40eb200640 1 --2- 172.21.15.32:0/1001194546 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f40ec1f3f50 0x7f40ec1f6850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:37.405 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:37.402+0000 7f40eb200640 1 --2- 172.21.15.32:0/1001194546 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f40d0005350 0x7f40ec09bfa0 unknown :-1 s=CLOSED pgs=1586 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:37.405 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:37.402+0000 7f40eb200640 1 -- 172.21.15.32:0/1001194546 >> 172.21.15.32:0/1001194546 conn(0x7f40ec0b2120 msgr2=0x7f40ec0ca370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:57:37.405 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:37.403+0000 7f40eb200640 1 -- 172.21.15.32:0/1001194546 shutdown_connections 2024-08-03T20:57:37.405 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:37.403+0000 7f40eb200640 1 -- 172.21.15.32:0/1001194546 wait complete. 2024-08-03T20:57:37.429 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mds repaired 0 2024-08-03T20:57:37.661 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:37.659+0000 7f80fea00640 1 -- 172.21.15.32:0/2198500542 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f80f815fff0 msgr2=0x7f80f8160400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:37.662 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:37.659+0000 7f80fea00640 1 --2- 172.21.15.32:0/2198500542 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f80f815fff0 0x7f80f8160400 secure :-1 s=READY pgs=1588 cs=0 l=1 rev1=1 crypto rx=0x7f80f0005fd0 tx=0x7f80f002a8b0 comp rx=0 tx=0).stop 2024-08-03T20:57:37.662 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:37.659+0000 7f80fea00640 1 -- 172.21.15.32:0/2198500542 shutdown_connections 2024-08-03T20:57:37.662 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:37.659+0000 7f80fea00640 1 --2- 172.21.15.32:0/2198500542 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f80f815fff0 0x7f80f8160400 unknown :-1 s=CLOSED pgs=1588 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:37.662 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:37.659+0000 7f80fea00640 1 --2- 172.21.15.32:0/2198500542 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f80f806e930 0x7f80f8144f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:37.662 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:37.659+0000 7f80fea00640 1 -- 172.21.15.32:0/2198500542 >> 172.21.15.32:0/2198500542 conn(0x7f80f80b1bd0 msgr2=0x7f80f80b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:57:37.662 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:37.659+0000 7f80fea00640 1 -- 172.21.15.32:0/2198500542 shutdown_connections 2024-08-03T20:57:37.662 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:37.660+0000 7f80fea00640 1 -- 172.21.15.32:0/2198500542 wait complete. 2024-08-03T20:57:37.663 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:37.662+0000 7f80fea00640 1 Processor -- start 2024-08-03T20:57:37.663 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:37.662+0000 7f80fea00640 1 -- start start 2024-08-03T20:57:37.666 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:37.663+0000 7f80fea00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f80f806e930 0x7f80f8156d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:37.666 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:37.663+0000 7f80fea00640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f80f8073e60 0x7f80f80cc540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:37.666 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:37.663+0000 7f80fea00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f80f815fff0 0x7f80f80cca80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:37.666 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:37.663+0000 7f80fea00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f80f80cf840 con 0x7f80f815fff0 2024-08-03T20:57:37.666 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:37.663+0000 7f80fea00640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f80f8145f30 con 0x7f80f806e930 2024-08-03T20:57:37.666 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:37.663+0000 7f80fea00640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7f80f80cf9c0 con 0x7f80f8073e60 2024-08-03T20:57:37.666 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:37.664+0000 7f80fca00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f80f815fff0 0x7f80f80cca80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:37.666 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:37.664+0000 7f80fca00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f80f815fff0 0x7f80f80cca80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:33950/0 (socket says 172.21.15.32:33950) 2024-08-03T20:57:37.666 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:37.664+0000 7f80fca00640 1 -- 172.21.15.32:0/3973345652 learned_addr learned my addr 172.21.15.32:0/3973345652 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:57:37.666 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:37.664+0000 7f80f7e00640 1 --2- 172.21.15.32:0/3973345652 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f80f806e930 0x7f80f8156d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:37.666 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:37.664+0000 7f80ef400640 1 --2- 172.21.15.32:0/3973345652 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f80f8073e60 0x7f80f80cc540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:37.667 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:37.664+0000 7f80fca00640 1 -- 172.21.15.32:0/3973345652 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f80f8073e60 msgr2=0x7f80f80cc540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:37.667 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:37.664+0000 7f80fca00640 1 --2- 172.21.15.32:0/3973345652 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f80f8073e60 0x7f80f80cc540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:37.667 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:37.664+0000 7f80fca00640 1 -- 172.21.15.32:0/3973345652 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f80f806e930 msgr2=0x7f80f8156d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:37.667 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:37.664+0000 7f80fca00640 1 --2- 172.21.15.32:0/3973345652 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f80f806e930 0x7f80f8156d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:37.667 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:37.664+0000 7f80fca00640 1 -- 172.21.15.32:0/3973345652 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f80f80c9430 con 0x7f80f815fff0 2024-08-03T20:57:37.667 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:37.664+0000 7f80fca00640 1 --2- 172.21.15.32:0/3973345652 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f80f815fff0 0x7f80f80cca80 secure :-1 s=READY pgs=1589 cs=0 l=1 rev1=1 crypto rx=0x7f80f002aea0 tx=0x7f80f00316d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:57:37.668 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:37.665+0000 7f80f4c00640 1 -- 172.21.15.32:0/3973345652 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80f0031990 con 0x7f80f815fff0 2024-08-03T20:57:37.668 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:37.665+0000 7f80f4c00640 1 -- 172.21.15.32:0/3973345652 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f80f0031c20 con 0x7f80f815fff0 2024-08-03T20:57:37.668 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:37.665+0000 7f80fea00640 1 -- 172.21.15.32:0/3973345652 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f80f80cb820 con 0x7f80f815fff0 2024-08-03T20:57:37.668 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:37.665+0000 7f80f4c00640 1 -- 172.21.15.32:0/3973345652 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80f00330d0 con 0x7f80f815fff0 2024-08-03T20:57:37.668 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:37.665+0000 7f80fea00640 1 -- 172.21.15.32:0/3973345652 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f80f81596c0 con 0x7f80f815fff0 2024-08-03T20:57:37.668 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:37.666+0000 7f80fea00640 1 -- 172.21.15.32:0/3973345652 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f80f813dae0 con 0x7f80f815fff0 2024-08-03T20:57:37.669 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:37.667+0000 7f80f4c00640 1 -- 172.21.15.32:0/3973345652 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f80f0033270 con 0x7f80f815fff0 2024-08-03T20:57:37.669 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:37.668+0000 7f80f4c00640 1 --2- 172.21.15.32:0/3973345652 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f80d4077030 0x7f80d4079540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:37.669 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:37.668+0000 7f80f4c00640 1 -- 172.21.15.32:0/3973345652 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(106..106 src has 1..106) ==== 7660+0+0 (secure 0 0 0) 0x7f80f00ba550 con 0x7f80f815fff0 2024-08-03T20:57:37.670 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:37.668+0000 7f80f7e00640 1 --2- 172.21.15.32:0/3973345652 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f80d4077030 0x7f80d4079540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:37.670 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:37.669+0000 7f80f7e00640 1 --2- 172.21.15.32:0/3973345652 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f80d4077030 0x7f80d4079540 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7f80e8009ac0 tx=0x7f80e8005f90 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:57:37.674 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:37.672+0000 7f80f4c00640 1 -- 172.21.15.32:0/3973345652 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7f80f003f030 con 0x7f80f815fff0 2024-08-03T20:57:37.960 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:37.958+0000 7f80fea00640 1 -- 172.21.15.32:0/3973345652 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "mds repaired", "role": "0"} v 0) -- 0x7f80f8160bc0 con 0x7f80f815fff0 2024-08-03T20:57:38.407 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:38.405+0000 7f80f4c00640 1 -- 172.21.15.32:0/3973345652 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "mds repaired", "role": "0"}]=0 repaired: restoring rank 6:0 v219) ==== 101+0+0 (secure 0 0 0) 0x7f80f0082f10 con 0x7f80f815fff0 2024-08-03T20:57:38.407 INFO:teuthology.orchestra.run.smithi032.stderr:repaired: restoring rank 6:0 2024-08-03T20:57:38.410 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:38.408+0000 7f80fea00640 1 -- 172.21.15.32:0/3973345652 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f80d4077030 msgr2=0x7f80d4079540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:38.410 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:38.408+0000 7f80fea00640 1 --2- 172.21.15.32:0/3973345652 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f80d4077030 0x7f80d4079540 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7f80e8009ac0 tx=0x7f80e8005f90 comp rx=0 tx=0).stop 2024-08-03T20:57:38.410 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:38.408+0000 7f80fea00640 1 -- 172.21.15.32:0/3973345652 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f80f815fff0 msgr2=0x7f80f80cca80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:38.410 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:38.408+0000 7f80fea00640 1 --2- 172.21.15.32:0/3973345652 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f80f815fff0 0x7f80f80cca80 secure :-1 s=READY pgs=1589 cs=0 l=1 rev1=1 crypto rx=0x7f80f002aea0 tx=0x7f80f00316d0 comp rx=0 tx=0).stop 2024-08-03T20:57:38.410 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:38.408+0000 7f80fea00640 1 -- 172.21.15.32:0/3973345652 shutdown_connections 2024-08-03T20:57:38.410 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:38.408+0000 7f80fea00640 1 --2- 172.21.15.32:0/3973345652 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f80d4077030 0x7f80d4079540 unknown :-1 s=CLOSED pgs=647 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:38.410 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:38.408+0000 7f80fea00640 1 --2- 172.21.15.32:0/3973345652 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f80f815fff0 0x7f80f80cca80 unknown :-1 s=CLOSED pgs=1589 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:38.410 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:38.408+0000 7f80fea00640 1 --2- 172.21.15.32:0/3973345652 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f80f8073e60 0x7f80f80cc540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:38.410 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:38.408+0000 7f80fea00640 1 --2- 172.21.15.32:0/3973345652 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f80f806e930 0x7f80f8156d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:38.410 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:38.408+0000 7f80fea00640 1 -- 172.21.15.32:0/3973345652 >> 172.21.15.32:0/3973345652 conn(0x7f80f80b1bd0 msgr2=0x7f80f8074270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:57:38.410 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:38.408+0000 7f80fea00640 1 -- 172.21.15.32:0/3973345652 shutdown_connections 2024-08-03T20:57:38.411 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:38.408+0000 7f80fea00640 1 -- 172.21.15.32:0/3973345652 wait complete. 2024-08-03T20:57:38.431 DEBUG:teuthology.orchestra.run.smithi032:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata import - 2024-08-03T20:57:38.528 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-08-03T20:57:38.532 DEBUG:teuthology.orchestra.run.smithi118:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-08-03T20:57:38.575 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:38.573+0000 7f6884c1ef40 1 Processor -- start 2024-08-03T20:57:38.575 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:38.574+0000 7f6884c1ef40 1 -- start start 2024-08-03T20:57:38.577 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:38.575+0000 7f6884c1ef40 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55ae6b174360 0x55ae6b18e260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:38.577 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:38.575+0000 7f6884c1ef40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55ae6b175ea0 0x55ae6b193060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:38.577 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:38.575+0000 7f6884c1ef40 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ae6b18b330 0x55ae6b2240c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:38.577 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:38.575+0000 7f6884c1ef40 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55ae6b00be30 con 0x55ae6b18b330 2024-08-03T20:57:38.577 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:38.575+0000 7f6884c1ef40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x55ae6b069bf0 con 0x55ae6b175ea0 2024-08-03T20:57:38.577 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:38.575+0000 7f6884c1ef40 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x55ae6b175440 con 0x55ae6b174360 2024-08-03T20:57:38.577 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:38.575+0000 7f687dc00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ae6b18b330 0x55ae6b2240c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:38.577 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:38.575+0000 7f6877e00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55ae6b175ea0 0x55ae6b193060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:38.578 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:38.575+0000 7f687dc00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ae6b18b330 0x55ae6b2240c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:33964/0 (socket says 172.21.15.32:33964) 2024-08-03T20:57:38.578 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:38.575+0000 7f687dc00640 1 -- 172.21.15.32:0/1082883156 learned_addr learned my addr 172.21.15.32:0/1082883156 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:57:38.578 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:38.575+0000 7f687d200640 1 --2- 172.21.15.32:0/1082883156 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55ae6b174360 0x55ae6b18e260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:38.579 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:38.576+0000 7f687dc00640 1 -- 172.21.15.32:0/1082883156 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55ae6b174360 msgr2=0x55ae6b18e260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:57:38.579 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:38.576+0000 7f687dc00640 1 --2- 172.21.15.32:0/1082883156 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55ae6b174360 0x55ae6b18e260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:38.579 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:38.576+0000 7f687dc00640 1 -- 172.21.15.32:0/1082883156 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55ae6b175ea0 msgr2=0x55ae6b193060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:57:38.579 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:38.576+0000 7f687dc00640 1 --2- 172.21.15.32:0/1082883156 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55ae6b175ea0 0x55ae6b193060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:38.580 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:38.576+0000 7f687dc00640 1 -- 172.21.15.32:0/1082883156 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ae6b175c60 con 0x55ae6b18b330 2024-08-03T20:57:38.580 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:38.576+0000 7f6877e00640 1 --2- 172.21.15.32:0/1082883156 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55ae6b175ea0 0x55ae6b193060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-03T20:57:38.580 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:38.576+0000 7f687dc00640 1 --2- 172.21.15.32:0/1082883156 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ae6b18b330 0x55ae6b2240c0 secure :-1 s=READY pgs=1591 cs=0 l=1 rev1=1 crypto rx=0x7f687800c6a0 tx=0x7f687800cb60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4f469e6a8d7ba233 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:57:38.580 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:38.576+0000 7f6876a00640 1 -- 172.21.15.32:0/1082883156 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6878004020 con 0x55ae6b18b330 2024-08-03T20:57:38.580 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:38.577+0000 7f6876a00640 1 -- 172.21.15.32:0/1082883156 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f68780042b0 con 0x55ae6b18b330 2024-08-03T20:57:38.580 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:38.577+0000 7f6876a00640 1 -- 172.21.15.32:0/1082883156 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f687800aa80 con 0x55ae6b18b330 2024-08-03T20:57:38.580 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:38.577+0000 7f6884c1ef40 1 -- 172.21.15.32:0/1082883156 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ae6b18b330 msgr2=0x55ae6b2240c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:38.580 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:38.577+0000 7f6884c1ef40 1 --2- 172.21.15.32:0/1082883156 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ae6b18b330 0x55ae6b2240c0 secure :-1 s=READY pgs=1591 cs=0 l=1 rev1=1 crypto rx=0x7f687800c6a0 tx=0x7f687800cb60 comp rx=0 tx=0).stop 2024-08-03T20:57:38.580 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:38.577+0000 7f6884c1ef40 1 -- 172.21.15.32:0/1082883156 shutdown_connections 2024-08-03T20:57:38.580 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:38.577+0000 7f6884c1ef40 1 --2- 172.21.15.32:0/1082883156 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55ae6b175ea0 0x55ae6b193060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:38.580 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:38.577+0000 7f6884c1ef40 1 --2- 172.21.15.32:0/1082883156 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ae6b18b330 0x55ae6b2240c0 unknown :-1 s=CLOSED pgs=1591 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:38.580 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:38.577+0000 7f6884c1ef40 1 --2- 172.21.15.32:0/1082883156 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55ae6b174360 0x55ae6b18e260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:38.580 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:38.577+0000 7f6884c1ef40 1 -- 172.21.15.32:0/1082883156 >> 172.21.15.32:0/1082883156 conn(0x55ae6afddac0 msgr2=0x55ae6b18b8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:57:38.580 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:38.578+0000 7f6884c1ef40 1 -- 172.21.15.32:0/1082883156 shutdown_connections 2024-08-03T20:57:38.580 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:38.578+0000 7f6884c1ef40 1 -- 172.21.15.32:0/1082883156 wait complete. 2024-08-03T20:57:38.581 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:38.579+0000 7f6884c1ef40 1 Processor -- start 2024-08-03T20:57:38.581 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:38.579+0000 7f6884c1ef40 1 -- start start 2024-08-03T20:57:38.582 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:38.580+0000 7f6884c1ef40 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55ae6b174360 0x55ae6b22a4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:38.582 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:38.580+0000 7f6884c1ef40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55ae6b175ea0 0x55ae6b191cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:38.582 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:38.580+0000 7f6884c1ef40 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ae6b18b330 0x55ae6b1921f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:38.582 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:38.580+0000 7f6884c1ef40 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55ae6b18cbf0 con 0x55ae6b18b330 2024-08-03T20:57:38.583 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:38.580+0000 7f6884c1ef40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x55ae6b18ca20 con 0x55ae6b175ea0 2024-08-03T20:57:38.583 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:38.580+0000 7f6884c1ef40 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x55ae6b102900 con 0x55ae6b174360 2024-08-03T20:57:38.583 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:38.581+0000 7f687dc00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ae6b18b330 0x55ae6b1921f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:38.583 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:38.581+0000 7f687dc00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ae6b18b330 0x55ae6b1921f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:33968/0 (socket says 172.21.15.32:33968) 2024-08-03T20:57:38.583 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:38.581+0000 7f687dc00640 1 -- 172.21.15.32:0/1550780541 learned_addr learned my addr 172.21.15.32:0/1550780541 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:57:38.583 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:38.581+0000 7f6877e00640 1 --2- 172.21.15.32:0/1550780541 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55ae6b175ea0 0x55ae6b191cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:38.583 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:38.581+0000 7f687d200640 1 --2- 172.21.15.32:0/1550780541 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55ae6b174360 0x55ae6b22a4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:38.584 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:38.581+0000 7f687dc00640 1 -- 172.21.15.32:0/1550780541 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55ae6b174360 msgr2=0x55ae6b22a4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:38.584 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:38.581+0000 7f687dc00640 1 --2- 172.21.15.32:0/1550780541 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55ae6b174360 0x55ae6b22a4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:38.584 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:38.581+0000 7f687dc00640 1 -- 172.21.15.32:0/1550780541 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55ae6b175ea0 msgr2=0x55ae6b191cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:38.584 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:38.581+0000 7f687dc00640 1 --2- 172.21.15.32:0/1550780541 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55ae6b175ea0 0x55ae6b191cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:38.584 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:38.581+0000 7f687dc00640 1 -- 172.21.15.32:0/1550780541 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ae6b22aeb0 con 0x55ae6b18b330 2024-08-03T20:57:38.584 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:38.582+0000 7f687dc00640 1 --2- 172.21.15.32:0/1550780541 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ae6b18b330 0x55ae6b1921f0 secure :-1 s=READY pgs=1592 cs=0 l=1 rev1=1 crypto rx=0x55ae6b16fd70 tx=0x7f687800c370 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:57:38.585 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:38.582+0000 7f6875600640 1 -- 172.21.15.32:0/1550780541 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6878025070 con 0x55ae6b18b330 2024-08-03T20:57:38.585 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:38.582+0000 7f6875600640 1 -- 172.21.15.32:0/1550780541 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f68780042b0 con 0x55ae6b18b330 2024-08-03T20:57:38.585 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:38.582+0000 7f6884c1ef40 1 -- 172.21.15.32:0/1550780541 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x55ae6b22b190 con 0x55ae6b18b330 2024-08-03T20:57:38.585 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:38.582+0000 7f6875600640 1 -- 172.21.15.32:0/1550780541 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f687800f8b0 con 0x55ae6b18b330 2024-08-03T20:57:38.585 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:38.582+0000 7f6884c1ef40 1 -- 172.21.15.32:0/1550780541 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ae6b100e50 con 0x55ae6b18b330 2024-08-03T20:57:38.585 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:38.584+0000 7f6875600640 1 -- 172.21.15.32:0/1550780541 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f6878028070 con 0x55ae6b18b330 2024-08-03T20:57:38.586 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:38.585+0000 7f6875600640 1 --2- 172.21.15.32:0/1550780541 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f685c0759f0 0x7f685c077f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:38.587 INFO:teuthology.orchestra.run.smithi032.stdout:Importing pool 2024-08-03T20:57:38.587 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:38.585+0000 7f6875600640 1 -- 172.21.15.32:0/1550780541 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(106..106 src has 1..106) ==== 7660+0+0 (secure 0 0 0) 0x7f68780992c0 con 0x55ae6b18b330 2024-08-03T20:57:38.588 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:38.586+0000 7f687d200640 1 --2- 172.21.15.32:0/1550780541 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f685c0759f0 0x7f685c077f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:38.588 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:38.586+0000 7f6884c1ef40 1 --2- 172.21.15.32:0/1550780541 >> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] conn(0x55ae6b16f3f0 0x55ae6b101880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:38.588 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:38.586+0000 7f6884c1ef40 1 -- 172.21.15.32:0/1550780541 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:1 12.10 12:09401a81:::601.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106) -- 0x55ae6b16eec0 con 0x55ae6b16f3f0 2024-08-03T20:57:38.588 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:38.586+0000 7f6877e00640 1 --2- 172.21.15.32:0/1550780541 >> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] conn(0x55ae6b16f3f0 0x55ae6b101880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:38.588 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:38.586+0000 7f687d200640 1 --2- 172.21.15.32:0/1550780541 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f685c0759f0 0x7f685c077f00 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x55ae6b194780 tx=0x7f6870001730 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:57:38.588 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:38.587+0000 7f6877e00640 1 --2- 172.21.15.32:0/1550780541 >> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] conn(0x55ae6b16f3f0 0x55ae6b101880 crc :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:57:38.590 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:38.589+0000 7f6877e00640 1 -- 172.21.15.32:0/1550780541 <== osd.0 v2:172.21.15.32:6816/3066156116 1 ==== osd_op_reply(1 601.00000000 [create] v106'629 uv603 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f686804ed50 con 0x55ae6b16f3f0 2024-08-03T20:57:38.590 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:38.589+0000 7f6884c1ef40 1 -- 172.21.15.32:0/1550780541 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:2 12.10 12:09401a81:::601.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) -- 0x55ae6b236c30 con 0x55ae6b16f3f0 2024-08-03T20:57:38.591 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:38.590+0000 7f6877e00640 1 -- 172.21.15.32:0/1550780541 <== osd.0 v2:172.21.15.32:6816/3066156116 2 ==== osd_op_reply(2 601.00000000 [delete] v106'630 uv630 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f686804ed50 con 0x55ae6b16f3f0 2024-08-03T20:57:38.592 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:38.590+0000 7f6884c1ef40 1 -- 172.21.15.32:0/1550780541 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:3 12.10 12:09401a81:::601.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106) -- 0x55ae6b236fd0 con 0x55ae6b16f3f0 2024-08-03T20:57:38.593 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:38.592+0000 7f6877e00640 1 -- 172.21.15.32:0/1550780541 <== osd.0 v2:172.21.15.32:6816/3066156116 3 ==== osd_op_reply(3 601.00000000 [create] v106'631 uv631 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f686804ed50 con 0x55ae6b16f3f0 2024-08-03T20:57:38.593 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::601.00000000:head# 2024-08-03T20:57:38.593 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:38.592+0000 7f6884c1ef40 1 -- 172.21.15.32:0/1550780541 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:4 12.10 12:09401a81:::601.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106) -- 0x55ae6b238380 con 0x55ae6b16f3f0 2024-08-03T20:57:38.594 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:38.593+0000 7f6877e00640 1 -- 172.21.15.32:0/1550780541 <== osd.0 v2:172.21.15.32:6816/3066156116 4 ==== osd_op_reply(4 601.00000000 [setxattr (30)] v106'632 uv632 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f686804ed50 con 0x55ae6b16f3f0 2024-08-03T20:57:38.595 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:38.593+0000 7f6884c1ef40 1 -- 172.21.15.32:0/1550780541 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:5 12.10 12:09401a81:::601.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106) -- 0x55ae6b238720 con 0x55ae6b16f3f0 2024-08-03T20:57:38.596 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:38.595+0000 7f6877e00640 1 -- 172.21.15.32:0/1550780541 <== osd.0 v2:172.21.15.32:6816/3066156116 5 ==== osd_op_reply(5 601.00000000 [setxattr (62)] v106'633 uv633 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f686804ed50 con 0x55ae6b16f3f0 2024-08-03T20:57:38.596 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:38.595+0000 7f6884c1ef40 1 -- 172.21.15.32:0/1550780541 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:6 12.10 12:09401a81:::601.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106) -- 0x55ae6b237370 con 0x55ae6b16f3f0 2024-08-03T20:57:38.597 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:38.596+0000 7f6877e00640 1 -- 172.21.15.32:0/1550780541 <== osd.0 v2:172.21.15.32:6816/3066156116 6 ==== osd_op_reply(6 601.00000000 [omap-set-header] v106'634 uv634 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f686804ed50 con 0x55ae6b16f3f0 2024-08-03T20:57:38.597 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:38.596+0000 7f6884c1ef40 1 -- 172.21.15.32:0/1550780541 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:7 12.10 12:0bd6d154:::602.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106) -- 0x55ae6b237760 con 0x55ae6b16f3f0 2024-08-03T20:57:38.599 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:38.598+0000 7f6877e00640 1 -- 172.21.15.32:0/1550780541 <== osd.0 v2:172.21.15.32:6816/3066156116 7 ==== osd_op_reply(7 602.00000000 [create] v106'635 uv609 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f686804ed50 con 0x55ae6b16f3f0 2024-08-03T20:57:38.599 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:38.598+0000 7f6884c1ef40 1 -- 172.21.15.32:0/1550780541 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:8 12.10 12:0bd6d154:::602.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) -- 0x55ae6b237760 con 0x55ae6b16f3f0 2024-08-03T20:57:38.601 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:38.599+0000 7f6877e00640 1 -- 172.21.15.32:0/1550780541 <== osd.0 v2:172.21.15.32:6816/3066156116 8 ==== osd_op_reply(8 602.00000000 [delete] v106'636 uv636 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f686804ed50 con 0x55ae6b16f3f0 2024-08-03T20:57:38.601 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:38.599+0000 7f6884c1ef40 1 -- 172.21.15.32:0/1550780541 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:9 12.10 12:0bd6d154:::602.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106) -- 0x55ae6b237760 con 0x55ae6b16f3f0 2024-08-03T20:57:38.602 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:38.601+0000 7f6877e00640 1 -- 172.21.15.32:0/1550780541 <== osd.0 v2:172.21.15.32:6816/3066156116 9 ==== osd_op_reply(9 602.00000000 [create] v106'637 uv637 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f686804ed50 con 0x55ae6b16f3f0 2024-08-03T20:57:38.603 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::602.00000000:head# 2024-08-03T20:57:38.603 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:38.601+0000 7f6884c1ef40 1 -- 172.21.15.32:0/1550780541 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:10 12.10 12:0bd6d154:::602.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106) -- 0x55ae6b237760 con 0x55ae6b16f3f0 2024-08-03T20:57:38.604 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:38.602+0000 7f6877e00640 1 -- 172.21.15.32:0/1550780541 <== osd.0 v2:172.21.15.32:6816/3066156116 10 ==== osd_op_reply(10 602.00000000 [setxattr (30)] v106'638 uv638 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f686804ed50 con 0x55ae6b16f3f0 2024-08-03T20:57:38.604 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:38.602+0000 7f6884c1ef40 1 -- 172.21.15.32:0/1550780541 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:11 12.10 12:0bd6d154:::602.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106) -- 0x55ae6b237760 con 0x55ae6b16f3f0 2024-08-03T20:57:38.605 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:38.604+0000 7f6877e00640 1 -- 172.21.15.32:0/1550780541 <== osd.0 v2:172.21.15.32:6816/3066156116 11 ==== osd_op_reply(11 602.00000000 [setxattr (62)] v106'639 uv639 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f686804ed50 con 0x55ae6b16f3f0 2024-08-03T20:57:38.605 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:38.604+0000 7f6884c1ef40 1 -- 172.21.15.32:0/1550780541 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:12 12.10 12:0bd6d154:::602.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106) -- 0x55ae6b237760 con 0x55ae6b16f3f0 2024-08-03T20:57:38.607 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:38.606+0000 7f6877e00640 1 -- 172.21.15.32:0/1550780541 <== osd.0 v2:172.21.15.32:6816/3066156116 12 ==== osd_op_reply(12 602.00000000 [omap-set-header] v106'640 uv640 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f686804ed50 con 0x55ae6b16f3f0 2024-08-03T20:57:38.607 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:38.606+0000 7f6884c1ef40 1 -- 172.21.15.32:0/1550780541 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:13 12.10 12:0cf6ad0f:::10000000000.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106) -- 0x55ae6b237760 con 0x55ae6b16f3f0 2024-08-03T20:57:38.608 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:38.607+0000 7f6877e00640 1 -- 172.21.15.32:0/1550780541 <== osd.0 v2:172.21.15.32:6816/3066156116 13 ==== osd_op_reply(13 10000000000.00000000 [create] v106'641 uv616 ondisk = -17 ((17) File exists)) ==== 164+0+0 (crc 0 0 0) 0x7f686804ed50 con 0x55ae6b16f3f0 2024-08-03T20:57:38.609 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:38.607+0000 7f6884c1ef40 1 -- 172.21.15.32:0/1550780541 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:14 12.10 12:0cf6ad0f:::10000000000.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) -- 0x55ae6b237760 con 0x55ae6b16f3f0 2024-08-03T20:57:38.610 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:38.609+0000 7f6877e00640 1 -- 172.21.15.32:0/1550780541 <== osd.0 v2:172.21.15.32:6816/3066156116 14 ==== osd_op_reply(14 10000000000.00000000 [delete] v106'642 uv642 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f686804ed50 con 0x55ae6b16f3f0 2024-08-03T20:57:38.610 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:38.609+0000 7f6884c1ef40 1 -- 172.21.15.32:0/1550780541 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:15 12.10 12:0cf6ad0f:::10000000000.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106) -- 0x55ae6b237760 con 0x55ae6b16f3f0 2024-08-03T20:57:38.612 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:38.611+0000 7f6877e00640 1 -- 172.21.15.32:0/1550780541 <== osd.0 v2:172.21.15.32:6816/3066156116 15 ==== osd_op_reply(15 10000000000.00000000 [create] v106'643 uv643 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f686804ed50 con 0x55ae6b16f3f0 2024-08-03T20:57:38.613 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::10000000000.00000000:head# 2024-08-03T20:57:38.613 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:38.611+0000 7f6884c1ef40 1 -- 172.21.15.32:0/1550780541 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:16 12.10 12:0cf6ad0f:::10000000000.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106) -- 0x55ae6b237760 con 0x55ae6b16f3f0 2024-08-03T20:57:38.614 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:38.612+0000 7f6877e00640 1 -- 172.21.15.32:0/1550780541 <== osd.0 v2:172.21.15.32:6816/3066156116 16 ==== osd_op_reply(16 10000000000.00000000 [setxattr (30)] v106'644 uv644 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f686804ed50 con 0x55ae6b16f3f0 2024-08-03T20:57:38.614 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:38.612+0000 7f6884c1ef40 1 -- 172.21.15.32:0/1550780541 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:17 12.10 12:0cf6ad0f:::10000000000.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106) -- 0x55ae6b237760 con 0x55ae6b16f3f0 2024-08-03T20:57:38.616 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:38.614+0000 7f6877e00640 1 -- 172.21.15.32:0/1550780541 <== osd.0 v2:172.21.15.32:6816/3066156116 17 ==== osd_op_reply(17 10000000000.00000000 [setxattr (62)] v106'645 uv645 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f686804ed50 con 0x55ae6b16f3f0 2024-08-03T20:57:38.616 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:38.614+0000 7f6884c1ef40 1 -- 172.21.15.32:0/1550780541 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:18 12.10 12:0cf6ad0f:::10000000000.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106) -- 0x55ae6b237760 con 0x55ae6b16f3f0 2024-08-03T20:57:38.617 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:38.615+0000 7f6877e00640 1 -- 172.21.15.32:0/1550780541 <== osd.0 v2:172.21.15.32:6816/3066156116 18 ==== osd_op_reply(18 10000000000.00000000 [omap-set-header] v106'646 uv646 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f686804ed50 con 0x55ae6b16f3f0 2024-08-03T20:57:38.617 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:38.615+0000 7f6884c1ef40 1 -- 172.21.15.32:0/1550780541 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:19 12.10 12:0cf6ad0f:::10000000000.00000000:head [omap-set-vals in=593b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106) -- 0x55ae6b237b60 con 0x55ae6b16f3f0 2024-08-03T20:57:38.618 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:38.617+0000 7f6877e00640 1 -- 172.21.15.32:0/1550780541 <== osd.0 v2:172.21.15.32:6816/3066156116 19 ==== osd_op_reply(19 10000000000.00000000 [omap-set-vals] v106'647 uv647 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f686804ed50 con 0x55ae6b16f3f0 2024-08-03T20:57:38.618 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:38.617+0000 7f6884c1ef40 1 -- 172.21.15.32:0/1550780541 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:20 12.10 12:0d82a743:::600.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106) -- 0x55ae6b237b60 con 0x55ae6b16f3f0 2024-08-03T20:57:38.620 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:38.619+0000 7f6877e00640 1 -- 172.21.15.32:0/1550780541 <== osd.0 v2:172.21.15.32:6816/3066156116 20 ==== osd_op_reply(20 600.00000000 [create] v106'648 uv622 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f686804ed50 con 0x55ae6b16f3f0 2024-08-03T20:57:38.620 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:38.619+0000 7f6884c1ef40 1 -- 172.21.15.32:0/1550780541 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:21 12.10 12:0d82a743:::600.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) -- 0x55ae6b237b60 con 0x55ae6b16f3f0 2024-08-03T20:57:38.622 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:38.620+0000 7f6877e00640 1 -- 172.21.15.32:0/1550780541 <== osd.0 v2:172.21.15.32:6816/3066156116 21 ==== osd_op_reply(21 600.00000000 [delete] v106'649 uv649 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f686804ed50 con 0x55ae6b16f3f0 2024-08-03T20:57:38.622 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:38.620+0000 7f6884c1ef40 1 -- 172.21.15.32:0/1550780541 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:22 12.10 12:0d82a743:::600.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106) -- 0x55ae6b237b60 con 0x55ae6b16f3f0 2024-08-03T20:57:38.623 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::600.00000000:head# 2024-08-03T20:57:38.623 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:38.622+0000 7f6877e00640 1 -- 172.21.15.32:0/1550780541 <== osd.0 v2:172.21.15.32:6816/3066156116 22 ==== osd_op_reply(22 600.00000000 [create] v106'650 uv650 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f686804ed50 con 0x55ae6b16f3f0 2024-08-03T20:57:38.623 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:38.622+0000 7f6884c1ef40 1 -- 172.21.15.32:0/1550780541 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:23 12.10 12:0d82a743:::600.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106) -- 0x55ae6b237b60 con 0x55ae6b16f3f0 2024-08-03T20:57:38.624 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:38.623+0000 7f6877e00640 1 -- 172.21.15.32:0/1550780541 <== osd.0 v2:172.21.15.32:6816/3066156116 23 ==== osd_op_reply(23 600.00000000 [setxattr (30)] v106'651 uv651 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f686804ed50 con 0x55ae6b16f3f0 2024-08-03T20:57:38.624 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:38.623+0000 7f6884c1ef40 1 -- 172.21.15.32:0/1550780541 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:24 12.10 12:0d82a743:::600.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106) -- 0x55ae6b237f90 con 0x55ae6b16f3f0 2024-08-03T20:57:38.625 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:38.624+0000 7f6877e00640 1 -- 172.21.15.32:0/1550780541 <== osd.0 v2:172.21.15.32:6816/3066156116 24 ==== osd_op_reply(24 600.00000000 [setxattr (62)] v106'652 uv652 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f686804ed50 con 0x55ae6b16f3f0 2024-08-03T20:57:38.626 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:38.624+0000 7f6884c1ef40 1 -- 172.21.15.32:0/1550780541 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:25 12.10 12:0d82a743:::600.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106) -- 0x55ae6b237f90 con 0x55ae6b16f3f0 2024-08-03T20:57:38.627 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:38.626+0000 7f6877e00640 1 -- 172.21.15.32:0/1550780541 <== osd.0 v2:172.21.15.32:6816/3066156116 25 ==== osd_op_reply(25 600.00000000 [omap-set-header] v106'653 uv653 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f686804ed50 con 0x55ae6b16f3f0 2024-08-03T20:57:38.627 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:38.626+0000 7f6884c1ef40 1 -- 172.21.15.32:0/1550780541 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:26 12.10 12:0d89b25e:::603.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106) -- 0x55ae6b237f40 con 0x55ae6b16f3f0 2024-08-03T20:57:38.629 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:38.627+0000 7f6877e00640 1 -- 172.21.15.32:0/1550780541 <== osd.0 v2:172.21.15.32:6816/3066156116 26 ==== osd_op_reply(26 603.00000000 [create] v106'654 uv628 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f686804ed50 con 0x55ae6b16f3f0 2024-08-03T20:57:38.629 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:38.627+0000 7f6884c1ef40 1 -- 172.21.15.32:0/1550780541 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:27 12.10 12:0d89b25e:::603.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) -- 0x55ae6b237f40 con 0x55ae6b16f3f0 2024-08-03T20:57:38.630 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:38.628+0000 7f6877e00640 1 -- 172.21.15.32:0/1550780541 <== osd.0 v2:172.21.15.32:6816/3066156116 27 ==== osd_op_reply(27 603.00000000 [delete] v106'655 uv655 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f686804ed50 con 0x55ae6b16f3f0 2024-08-03T20:57:38.630 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:38.629+0000 7f6884c1ef40 1 -- 172.21.15.32:0/1550780541 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:28 12.10 12:0d89b25e:::603.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106) -- 0x55ae6b237f40 con 0x55ae6b16f3f0 2024-08-03T20:57:38.631 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:38.630+0000 7f6877e00640 1 -- 172.21.15.32:0/1550780541 <== osd.0 v2:172.21.15.32:6816/3066156116 28 ==== osd_op_reply(28 603.00000000 [create] v106'656 uv656 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f686804ed50 con 0x55ae6b16f3f0 2024-08-03T20:57:38.632 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::603.00000000:head# 2024-08-03T20:57:38.632 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:38.630+0000 7f6884c1ef40 1 -- 172.21.15.32:0/1550780541 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:29 12.10 12:0d89b25e:::603.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106) -- 0x55ae6b237f40 con 0x55ae6b16f3f0 2024-08-03T20:57:38.632 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:38.631+0000 7f6877e00640 1 -- 172.21.15.32:0/1550780541 <== osd.0 v2:172.21.15.32:6816/3066156116 29 ==== osd_op_reply(29 603.00000000 [setxattr (30)] v106'657 uv657 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f686804ed50 con 0x55ae6b16f3f0 2024-08-03T20:57:38.633 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:38.631+0000 7f6884c1ef40 1 -- 172.21.15.32:0/1550780541 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:30 12.10 12:0d89b25e:::603.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106) -- 0x55ae6b237a20 con 0x55ae6b16f3f0 2024-08-03T20:57:38.634 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:38.633+0000 7f6877e00640 1 -- 172.21.15.32:0/1550780541 <== osd.0 v2:172.21.15.32:6816/3066156116 30 ==== osd_op_reply(30 603.00000000 [setxattr (62)] v106'658 uv658 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f686804ed50 con 0x55ae6b16f3f0 2024-08-03T20:57:38.634 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:38.633+0000 7f6884c1ef40 1 -- 172.21.15.32:0/1550780541 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:31 12.10 12:0d89b25e:::603.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106) -- 0x55ae6b237a20 con 0x55ae6b16f3f0 2024-08-03T20:57:38.636 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:38.634+0000 7f6877e00640 1 -- 172.21.15.32:0/1550780541 <== osd.0 v2:172.21.15.32:6816/3066156116 31 ==== osd_op_reply(31 603.00000000 [omap-set-header] v106'659 uv659 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f686804ed50 con 0x55ae6b16f3f0 2024-08-03T20:57:38.636 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:38.634+0000 7f6884c1ef40 1 --2- 172.21.15.32:0/1550780541 >> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] conn(0x55ae6b237e70 0x55ae6b23afe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:38.636 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:38.634+0000 7f6884c1ef40 1 -- 172.21.15.32:0/1550780541 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:32 12.14 12:287034c4:::1.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106) -- 0x55ae6b237a20 con 0x55ae6b237e70 2024-08-03T20:57:38.636 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:38.635+0000 7f687d200640 1 --2- 172.21.15.32:0/1550780541 >> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] conn(0x55ae6b237e70 0x55ae6b23afe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:38.637 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:38.635+0000 7f687d200640 1 --2- 172.21.15.32:0/1550780541 >> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] conn(0x55ae6b237e70 0x55ae6b23afe0 crc :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:57:38.639 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:38.637+0000 7f687d200640 1 -- 172.21.15.32:0/1550780541 <== osd.5 v2:172.21.15.118:6810/2724891770 1 ==== osd_op_reply(32 1.00000000.inode [create] v106'463 uv446 ondisk = -17 ((17) File exists)) ==== 160+0+0 (crc 0 0 0) 0x7f68700128d0 con 0x55ae6b237e70 2024-08-03T20:57:38.639 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:38.637+0000 7f6884c1ef40 1 -- 172.21.15.32:0/1550780541 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:33 12.14 12:287034c4:::1.00000000.inode:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) -- 0x55ae6b23b520 con 0x55ae6b237e70 2024-08-03T20:57:38.640 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:38.639+0000 7f687d200640 1 -- 172.21.15.32:0/1550780541 <== osd.5 v2:172.21.15.118:6810/2724891770 2 ==== osd_op_reply(33 1.00000000.inode [delete] v106'464 uv464 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f68700128d0 con 0x55ae6b237e70 2024-08-03T20:57:38.640 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:38.639+0000 7f6884c1ef40 1 -- 172.21.15.32:0/1550780541 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:34 12.14 12:287034c4:::1.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106) -- 0x55ae6b23b8c0 con 0x55ae6b237e70 2024-08-03T20:57:38.641 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::1.00000000.inode:head# 2024-08-03T20:57:38.642 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:38.640+0000 7f687d200640 1 -- 172.21.15.32:0/1550780541 <== osd.5 v2:172.21.15.118:6810/2724891770 3 ==== osd_op_reply(34 1.00000000.inode [create] v106'465 uv465 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f68700128d0 con 0x55ae6b237e70 2024-08-03T20:57:38.642 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:38.640+0000 7f6884c1ef40 1 -- 172.21.15.32:0/1550780541 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:35 12.14 12:287034c4:::1.00000000.inode:head [write 0~582 in=582b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106) -- 0x55ae6b23bc60 con 0x55ae6b237e70 2024-08-03T20:57:38.643 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:38.642+0000 7f687d200640 1 -- 172.21.15.32:0/1550780541 <== osd.5 v2:172.21.15.118:6810/2724891770 4 ==== osd_op_reply(35 1.00000000.inode [write 0~582] v106'466 uv466 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f68700128d0 con 0x55ae6b237e70 2024-08-03T20:57:38.643 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:38.642+0000 7f6884c1ef40 1 -- 172.21.15.32:0/1550780541 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:36 12.14 12:287034c4:::1.00000000.inode:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106) -- 0x55ae6b23c000 con 0x55ae6b237e70 2024-08-03T20:57:38.645 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:38.643+0000 7f687d200640 1 -- 172.21.15.32:0/1550780541 <== osd.5 v2:172.21.15.118:6810/2724891770 5 ==== osd_op_reply(36 1.00000000.inode [omap-set-header] v106'467 uv467 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f68700128d0 con 0x55ae6b237e70 2024-08-03T20:57:38.645 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:38.643+0000 7f6884c1ef40 1 -- 172.21.15.32:0/1550780541 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:37 12.14 12:292cf221:::200.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106) -- 0x55ae6b23c3a0 con 0x55ae6b237e70 2024-08-03T20:57:38.646 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:38.645+0000 7f687d200640 1 -- 172.21.15.32:0/1550780541 <== osd.5 v2:172.21.15.118:6810/2724891770 6 ==== osd_op_reply(37 200.00000000 [create] v106'468 uv451 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f68700128d0 con 0x55ae6b237e70 2024-08-03T20:57:38.646 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:38.645+0000 7f6884c1ef40 1 -- 172.21.15.32:0/1550780541 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:38 12.14 12:292cf221:::200.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) -- 0x55ae6b23c740 con 0x55ae6b237e70 2024-08-03T20:57:38.647 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:38.646+0000 7f687d200640 1 -- 172.21.15.32:0/1550780541 <== osd.5 v2:172.21.15.118:6810/2724891770 7 ==== osd_op_reply(38 200.00000000 [delete] v106'469 uv469 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f68700128d0 con 0x55ae6b237e70 2024-08-03T20:57:38.647 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:38.646+0000 7f6884c1ef40 1 -- 172.21.15.32:0/1550780541 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:39 12.14 12:292cf221:::200.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106) -- 0x55ae6b23cae0 con 0x55ae6b237e70 2024-08-03T20:57:38.649 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::200.00000000:head# 2024-08-03T20:57:38.649 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:38.648+0000 7f687d200640 1 -- 172.21.15.32:0/1550780541 <== osd.5 v2:172.21.15.118:6810/2724891770 8 ==== osd_op_reply(39 200.00000000 [create] v106'470 uv470 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f68700128d0 con 0x55ae6b237e70 2024-08-03T20:57:38.649 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:38.648+0000 7f6884c1ef40 1 -- 172.21.15.32:0/1550780541 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:40 12.14 12:292cf221:::200.00000000:head [write 0~90 in=90b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106) -- 0x55ae6b23cae0 con 0x55ae6b237e70 2024-08-03T20:57:38.650 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:38.649+0000 7f687d200640 1 -- 172.21.15.32:0/1550780541 <== osd.5 v2:172.21.15.118:6810/2724891770 9 ==== osd_op_reply(40 200.00000000 [write 0~90] v106'471 uv471 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f68700128d0 con 0x55ae6b237e70 2024-08-03T20:57:38.651 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:38.649+0000 7f6884c1ef40 1 -- 172.21.15.32:0/1550780541 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:41 12.14 12:292cf221:::200.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106) -- 0x55ae6b23cae0 con 0x55ae6b237e70 2024-08-03T20:57:38.652 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:38.650+0000 7f687d200640 1 -- 172.21.15.32:0/1550780541 <== osd.5 v2:172.21.15.118:6810/2724891770 10 ==== osd_op_reply(41 200.00000000 [omap-set-header] v106'472 uv472 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f68700128d0 con 0x55ae6b237e70 2024-08-03T20:57:38.652 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:38.650+0000 7f6884c1ef40 1 -- 172.21.15.32:0/1550780541 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:42 12.14 12:2e2fa760:::200.00000001:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106) -- 0x55ae6b23cae0 con 0x55ae6b237e70 2024-08-03T20:57:38.653 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:38.652+0000 7f687d200640 1 -- 172.21.15.32:0/1550780541 <== osd.5 v2:172.21.15.118:6810/2724891770 11 ==== osd_op_reply(42 200.00000001 [create] v106'473 uv456 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f68700128d0 con 0x55ae6b237e70 2024-08-03T20:57:38.654 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:38.652+0000 7f6884c1ef40 1 -- 172.21.15.32:0/1550780541 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:43 12.14 12:2e2fa760:::200.00000001:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) -- 0x55ae6b23cae0 con 0x55ae6b237e70 2024-08-03T20:57:38.655 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:38.653+0000 7f687d200640 1 -- 172.21.15.32:0/1550780541 <== osd.5 v2:172.21.15.118:6810/2724891770 12 ==== osd_op_reply(43 200.00000001 [delete] v106'474 uv474 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f68700128d0 con 0x55ae6b237e70 2024-08-03T20:57:38.655 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:38.653+0000 7f6884c1ef40 1 -- 172.21.15.32:0/1550780541 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:44 12.14 12:2e2fa760:::200.00000001:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106) -- 0x55ae6b23cae0 con 0x55ae6b237e70 2024-08-03T20:57:38.656 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:38.655+0000 7f687d200640 1 -- 172.21.15.32:0/1550780541 <== osd.5 v2:172.21.15.118:6810/2724891770 13 ==== osd_op_reply(44 200.00000001 [create] v106'475 uv475 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f68700128d0 con 0x55ae6b237e70 2024-08-03T20:57:38.656 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::200.00000001:head# 2024-08-03T20:57:38.656 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:38.655+0000 7f6884c1ef40 1 -- 172.21.15.32:0/1550780541 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:45 12.14 12:2e2fa760:::200.00000001:head [write 0~18118 in=18118b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106) -- 0x55ae6b2411d0 con 0x55ae6b237e70 2024-08-03T20:57:38.658 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:38.656+0000 7f687d200640 1 -- 172.21.15.32:0/1550780541 <== osd.5 v2:172.21.15.118:6810/2724891770 14 ==== osd_op_reply(45 200.00000001 [write 0~18118] v106'476 uv476 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f68700128d0 con 0x55ae6b237e70 2024-08-03T20:57:38.658 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:38.656+0000 7f6884c1ef40 1 -- 172.21.15.32:0/1550780541 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:46 12.14 12:2e2fa760:::200.00000001:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106) -- 0x55ae6b2411d0 con 0x55ae6b237e70 2024-08-03T20:57:38.659 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:38.658+0000 7f687d200640 1 -- 172.21.15.32:0/1550780541 <== osd.5 v2:172.21.15.118:6810/2724891770 15 ==== osd_op_reply(46 200.00000001 [omap-set-header] v106'477 uv477 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f68700128d0 con 0x55ae6b237e70 2024-08-03T20:57:38.659 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:38.658+0000 7f6884c1ef40 1 -- 172.21.15.32:0/1550780541 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:47 12.14 12:2f55791f:::606.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106) -- 0x55ae6b23cae0 con 0x55ae6b237e70 2024-08-03T20:57:38.660 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:38.659+0000 7f687d200640 1 -- 172.21.15.32:0/1550780541 <== osd.5 v2:172.21.15.118:6810/2724891770 16 ==== osd_op_reply(47 606.00000000 [create] v106'478 uv462 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f68700128d0 con 0x55ae6b237e70 2024-08-03T20:57:38.660 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:38.659+0000 7f6884c1ef40 1 -- 172.21.15.32:0/1550780541 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:48 12.14 12:2f55791f:::606.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) -- 0x55ae6b23cae0 con 0x55ae6b237e70 2024-08-03T20:57:38.662 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:38.660+0000 7f687d200640 1 -- 172.21.15.32:0/1550780541 <== osd.5 v2:172.21.15.118:6810/2724891770 17 ==== osd_op_reply(48 606.00000000 [delete] v106'479 uv479 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f68700128d0 con 0x55ae6b237e70 2024-08-03T20:57:38.662 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:38.660+0000 7f6884c1ef40 1 -- 172.21.15.32:0/1550780541 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:49 12.14 12:2f55791f:::606.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106) -- 0x55ae6b23cae0 con 0x55ae6b237e70 2024-08-03T20:57:38.663 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:38.662+0000 7f687d200640 1 -- 172.21.15.32:0/1550780541 <== osd.5 v2:172.21.15.118:6810/2724891770 18 ==== osd_op_reply(49 606.00000000 [create] v106'480 uv480 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f68700128d0 con 0x55ae6b237e70 2024-08-03T20:57:38.663 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::606.00000000:head# 2024-08-03T20:57:38.663 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:38.662+0000 7f6884c1ef40 1 -- 172.21.15.32:0/1550780541 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:50 12.14 12:2f55791f:::606.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106) -- 0x55ae6b23daf0 con 0x55ae6b237e70 2024-08-03T20:57:38.665 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:38.663+0000 7f687d200640 1 -- 172.21.15.32:0/1550780541 <== osd.5 v2:172.21.15.118:6810/2724891770 19 ==== osd_op_reply(50 606.00000000 [setxattr (30)] v106'481 uv481 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f68700128d0 con 0x55ae6b237e70 2024-08-03T20:57:38.665 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:38.663+0000 7f6884c1ef40 1 -- 172.21.15.32:0/1550780541 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:51 12.14 12:2f55791f:::606.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106) -- 0x55ae6b23daf0 con 0x55ae6b237e70 2024-08-03T20:57:38.666 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:38.665+0000 7f687d200640 1 -- 172.21.15.32:0/1550780541 <== osd.5 v2:172.21.15.118:6810/2724891770 20 ==== osd_op_reply(51 606.00000000 [setxattr (62)] v106'482 uv482 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f68700128d0 con 0x55ae6b237e70 2024-08-03T20:57:38.666 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:38.665+0000 7f6884c1ef40 1 -- 172.21.15.32:0/1550780541 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:52 12.14 12:2f55791f:::606.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106) -- 0x55ae6b23cf90 con 0x55ae6b237e70 2024-08-03T20:57:38.667 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:38.666+0000 7f687d200640 1 -- 172.21.15.32:0/1550780541 <== osd.5 v2:172.21.15.118:6810/2724891770 21 ==== osd_op_reply(52 606.00000000 [omap-set-header] v106'483 uv483 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f68700128d0 con 0x55ae6b237e70 2024-08-03T20:57:38.668 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:38.666+0000 7f6884c1ef40 1 -- 172.21.15.32:0/1550780541 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:53 12.1c 12:3ed09add:::607.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106) -- 0x55ae6b23cf90 con 0x55ae6b16f3f0 2024-08-03T20:57:38.669 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:38.668+0000 7f6877e00640 1 -- 172.21.15.32:0/1550780541 <== osd.0 v2:172.21.15.32:6816/3066156116 32 ==== osd_op_reply(53 607.00000000 [create] v106'123 uv122 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f686804ed50 con 0x55ae6b16f3f0 2024-08-03T20:57:38.669 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:38.668+0000 7f6884c1ef40 1 -- 172.21.15.32:0/1550780541 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:54 12.1c 12:3ed09add:::607.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) -- 0x55ae6b23cf90 con 0x55ae6b16f3f0 2024-08-03T20:57:38.670 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:38.669+0000 7f6877e00640 1 -- 172.21.15.32:0/1550780541 <== osd.0 v2:172.21.15.32:6816/3066156116 33 ==== osd_op_reply(54 607.00000000 [delete] v106'124 uv124 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f686804ed50 con 0x55ae6b16f3f0 2024-08-03T20:57:38.671 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:38.669+0000 7f6884c1ef40 1 -- 172.21.15.32:0/1550780541 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:55 12.1c 12:3ed09add:::607.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106) -- 0x55ae6b23cf90 con 0x55ae6b16f3f0 2024-08-03T20:57:38.674 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::607.00000000:head# 2024-08-03T20:57:38.675 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:38.670+0000 7f6877e00640 1 -- 172.21.15.32:0/1550780541 <== osd.0 v2:172.21.15.32:6816/3066156116 34 ==== osd_op_reply(55 607.00000000 [create] v106'125 uv125 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f686804ed50 con 0x55ae6b16f3f0 2024-08-03T20:57:38.675 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:38.670+0000 7f6884c1ef40 1 -- 172.21.15.32:0/1550780541 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:56 12.1c 12:3ed09add:::607.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106) -- 0x55ae6b23dfa0 con 0x55ae6b16f3f0 2024-08-03T20:57:38.675 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:38.672+0000 7f6877e00640 1 -- 172.21.15.32:0/1550780541 <== osd.0 v2:172.21.15.32:6816/3066156116 35 ==== osd_op_reply(56 607.00000000 [setxattr (30)] v106'126 uv126 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f686804ed50 con 0x55ae6b16f3f0 2024-08-03T20:57:38.675 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:38.672+0000 7f6884c1ef40 1 -- 172.21.15.32:0/1550780541 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:57 12.1c 12:3ed09add:::607.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106) -- 0x55ae6b23dfa0 con 0x55ae6b16f3f0 2024-08-03T20:57:38.675 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:38.673+0000 7f6877e00640 1 -- 172.21.15.32:0/1550780541 <== osd.0 v2:172.21.15.32:6816/3066156116 36 ==== osd_op_reply(57 607.00000000 [setxattr (62)] v106'127 uv127 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f686804ed50 con 0x55ae6b16f3f0 2024-08-03T20:57:38.675 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:38.673+0000 7f6884c1ef40 1 -- 172.21.15.32:0/1550780541 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:58 12.1c 12:3ed09add:::607.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106) -- 0x55ae6b23cf90 con 0x55ae6b16f3f0 2024-08-03T20:57:38.676 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:38.674+0000 7f6877e00640 1 -- 172.21.15.32:0/1550780541 <== osd.0 v2:172.21.15.32:6816/3066156116 37 ==== osd_op_reply(58 607.00000000 [omap-set-header] v106'128 uv128 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f686804ed50 con 0x55ae6b16f3f0 2024-08-03T20:57:38.676 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:38.674+0000 7f6884c1ef40 1 --2- 172.21.15.32:0/1550780541 >> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] conn(0x55ae6b23d4a0 0x55ae6b23f8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:38.676 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:38.674+0000 7f6884c1ef40 1 -- 172.21.15.32:0/1550780541 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:59 12.6 12:654134d2:::mds0_openfiles.0:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106) -- 0x55ae6b23fe00 con 0x55ae6b23d4a0 2024-08-03T20:57:38.676 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:38.675+0000 7f6877e00640 1 --2- 172.21.15.32:0/1550780541 >> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] conn(0x55ae6b23d4a0 0x55ae6b23f8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:38.676 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:38.675+0000 7f6877e00640 1 --2- 172.21.15.32:0/1550780541 >> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] conn(0x55ae6b23d4a0 0x55ae6b23f8c0 crc :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:57:38.678 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:38.677+0000 7f6877e00640 1 -- 172.21.15.32:0/1550780541 <== osd.1 v2:172.21.15.32:6824/1551241759 1 ==== osd_op_reply(59 mds0_openfiles.0 [create] v106'98 uv97 ondisk = -17 ((17) File exists)) ==== 160+0+0 (crc 0 0 0) 0x7f686804ed50 con 0x55ae6b23d4a0 2024-08-03T20:57:38.678 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:38.677+0000 7f6884c1ef40 1 -- 172.21.15.32:0/1550780541 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:60 12.6 12:654134d2:::mds0_openfiles.0:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) -- 0x55ae6b23fe00 con 0x55ae6b23d4a0 2024-08-03T20:57:38.679 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:38.678+0000 7f6877e00640 1 -- 172.21.15.32:0/1550780541 <== osd.1 v2:172.21.15.32:6824/1551241759 2 ==== osd_op_reply(60 mds0_openfiles.0 [delete] v106'99 uv99 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f686804ed50 con 0x55ae6b23d4a0 2024-08-03T20:57:38.679 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:38.678+0000 7f6884c1ef40 1 -- 172.21.15.32:0/1550780541 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:61 12.6 12:654134d2:::mds0_openfiles.0:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106) -- 0x55ae6b240220 con 0x55ae6b23d4a0 2024-08-03T20:57:38.680 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:38.679+0000 7f6877e00640 1 -- 172.21.15.32:0/1550780541 <== osd.1 v2:172.21.15.32:6824/1551241759 3 ==== osd_op_reply(61 mds0_openfiles.0 [create] v106'100 uv100 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f686804ed50 con 0x55ae6b23d4a0 2024-08-03T20:57:38.681 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds0_openfiles.0:head# 2024-08-03T20:57:38.681 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:38.679+0000 7f6884c1ef40 1 -- 172.21.15.32:0/1550780541 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:62 12.6 12:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106) -- 0x55ae6b240220 con 0x55ae6b23d4a0 2024-08-03T20:57:38.682 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:38.680+0000 7f6877e00640 1 -- 172.21.15.32:0/1550780541 <== osd.1 v2:172.21.15.32:6824/1551241759 4 ==== osd_op_reply(62 mds0_openfiles.0 [omap-set-header] v106'101 uv101 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f686804ed50 con 0x55ae6b23d4a0 2024-08-03T20:57:38.682 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:38.680+0000 7f6884c1ef40 1 -- 172.21.15.32:0/1550780541 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:63 12.e 12:717a0223:::608.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106) -- 0x55ae6b240220 con 0x55ae6b16f3f0 2024-08-03T20:57:38.683 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:38.682+0000 7f6877e00640 1 -- 172.21.15.32:0/1550780541 <== osd.0 v2:172.21.15.32:6816/3066156116 38 ==== osd_op_reply(63 608.00000000 [create] v106'123 uv122 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f686804ed50 con 0x55ae6b16f3f0 2024-08-03T20:57:38.684 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:38.682+0000 7f6884c1ef40 1 -- 172.21.15.32:0/1550780541 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:64 12.e 12:717a0223:::608.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) -- 0x55ae6b240220 con 0x55ae6b16f3f0 2024-08-03T20:57:38.685 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:38.684+0000 7f6877e00640 1 -- 172.21.15.32:0/1550780541 <== osd.0 v2:172.21.15.32:6816/3066156116 39 ==== osd_op_reply(64 608.00000000 [delete] v106'124 uv124 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f686804ed50 con 0x55ae6b16f3f0 2024-08-03T20:57:38.685 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:38.684+0000 7f6884c1ef40 1 -- 172.21.15.32:0/1550780541 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:65 12.e 12:717a0223:::608.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106) -- 0x55ae6b240220 con 0x55ae6b16f3f0 2024-08-03T20:57:38.686 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:38.685+0000 7f6877e00640 1 -- 172.21.15.32:0/1550780541 <== osd.0 v2:172.21.15.32:6816/3066156116 40 ==== osd_op_reply(65 608.00000000 [create] v106'125 uv125 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f686804ed50 con 0x55ae6b16f3f0 2024-08-03T20:57:38.686 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::608.00000000:head# 2024-08-03T20:57:38.687 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:38.685+0000 7f6884c1ef40 1 -- 172.21.15.32:0/1550780541 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:66 12.e 12:717a0223:::608.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106) -- 0x55ae6b241230 con 0x55ae6b16f3f0 2024-08-03T20:57:38.688 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:38.686+0000 7f6877e00640 1 -- 172.21.15.32:0/1550780541 <== osd.0 v2:172.21.15.32:6816/3066156116 41 ==== osd_op_reply(66 608.00000000 [setxattr (30)] v106'126 uv126 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f686804ed50 con 0x55ae6b16f3f0 2024-08-03T20:57:38.688 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:38.686+0000 7f6884c1ef40 1 -- 172.21.15.32:0/1550780541 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:67 12.e 12:717a0223:::608.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106) -- 0x55ae6b241230 con 0x55ae6b16f3f0 2024-08-03T20:57:38.689 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:38.688+0000 7f6877e00640 1 -- 172.21.15.32:0/1550780541 <== osd.0 v2:172.21.15.32:6816/3066156116 42 ==== osd_op_reply(67 608.00000000 [setxattr (62)] v106'127 uv127 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f686804ed50 con 0x55ae6b16f3f0 2024-08-03T20:57:38.689 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:38.688+0000 7f6884c1ef40 1 -- 172.21.15.32:0/1550780541 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:68 12.e 12:717a0223:::608.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106) -- 0x55ae6b240220 con 0x55ae6b16f3f0 2024-08-03T20:57:38.691 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:38.689+0000 7f6877e00640 1 -- 172.21.15.32:0/1550780541 <== osd.0 v2:172.21.15.32:6816/3066156116 43 ==== osd_op_reply(68 608.00000000 [omap-set-header] v106'128 uv128 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f686804ed50 con 0x55ae6b16f3f0 2024-08-03T20:57:38.691 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:38.689+0000 7f6884c1ef40 1 --2- 172.21.15.32:0/1550780541 >> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] conn(0x55ae6b240730 0x55ae6b242b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:38.691 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:38.689+0000 7f6884c1ef40 1 -- 172.21.15.32:0/1550780541 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:69 12.15 12:a93a17c2:::604.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106) -- 0x55ae6b243160 con 0x55ae6b240730 2024-08-03T20:57:38.691 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:38.690+0000 7f687dc00640 1 --2- 172.21.15.32:0/1550780541 >> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] conn(0x55ae6b240730 0x55ae6b242b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:38.692 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:38.690+0000 7f687dc00640 1 --2- 172.21.15.32:0/1550780541 >> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] conn(0x55ae6b240730 0x55ae6b242b50 crc :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:57:38.693 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:38.692+0000 7f687dc00640 1 -- 172.21.15.32:0/1550780541 <== osd.6 v2:172.21.15.118:6825/801947386 1 ==== osd_op_reply(69 604.00000000 [create] v106'227 uv221 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f6878027030 con 0x55ae6b240730 2024-08-03T20:57:38.693 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:38.692+0000 7f6884c1ef40 1 -- 172.21.15.32:0/1550780541 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:70 12.15 12:a93a17c2:::604.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) -- 0x55ae6b243630 con 0x55ae6b240730 2024-08-03T20:57:38.695 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:38.693+0000 7f687dc00640 1 -- 172.21.15.32:0/1550780541 <== osd.6 v2:172.21.15.118:6825/801947386 2 ==== osd_op_reply(70 604.00000000 [delete] v106'228 uv228 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f6878027030 con 0x55ae6b240730 2024-08-03T20:57:38.695 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:38.693+0000 7f6884c1ef40 1 -- 172.21.15.32:0/1550780541 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:71 12.15 12:a93a17c2:::604.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106) -- 0x55ae6b243b10 con 0x55ae6b240730 2024-08-03T20:57:38.696 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::604.00000000:head# 2024-08-03T20:57:38.696 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:38.695+0000 7f687dc00640 1 -- 172.21.15.32:0/1550780541 <== osd.6 v2:172.21.15.118:6825/801947386 3 ==== osd_op_reply(71 604.00000000 [create] v106'229 uv229 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f6878027030 con 0x55ae6b240730 2024-08-03T20:57:38.696 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:38.695+0000 7f6884c1ef40 1 -- 172.21.15.32:0/1550780541 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:72 12.15 12:a93a17c2:::604.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106) -- 0x55ae6b2453c0 con 0x55ae6b240730 2024-08-03T20:57:38.698 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:38.696+0000 7f687dc00640 1 -- 172.21.15.32:0/1550780541 <== osd.6 v2:172.21.15.118:6825/801947386 4 ==== osd_op_reply(72 604.00000000 [setxattr (30)] v106'230 uv230 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f6878027030 con 0x55ae6b240730 2024-08-03T20:57:38.698 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:38.696+0000 7f6884c1ef40 1 -- 172.21.15.32:0/1550780541 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:73 12.15 12:a93a17c2:::604.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106) -- 0x55ae6b245760 con 0x55ae6b240730 2024-08-03T20:57:38.699 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:38.698+0000 7f687dc00640 1 -- 172.21.15.32:0/1550780541 <== osd.6 v2:172.21.15.118:6825/801947386 5 ==== osd_op_reply(73 604.00000000 [setxattr (62)] v106'231 uv231 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f6878027030 con 0x55ae6b240730 2024-08-03T20:57:38.699 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:38.698+0000 7f6884c1ef40 1 -- 172.21.15.32:0/1550780541 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:74 12.15 12:a93a17c2:::604.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106) -- 0x55ae6b2443b0 con 0x55ae6b240730 2024-08-03T20:57:38.700 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:38.699+0000 7f687dc00640 1 -- 172.21.15.32:0/1550780541 <== osd.6 v2:172.21.15.118:6825/801947386 6 ==== osd_op_reply(74 604.00000000 [omap-set-header] v106'232 uv232 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f6878027030 con 0x55ae6b240730 2024-08-03T20:57:38.701 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:38.699+0000 7f6884c1ef40 1 -- 172.21.15.32:0/1550780541 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:75 12.15 12:aa448500:::500.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106) -- 0x55ae6b2447a0 con 0x55ae6b240730 2024-08-03T20:57:38.702 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:38.700+0000 7f687dc00640 1 -- 172.21.15.32:0/1550780541 <== osd.6 v2:172.21.15.118:6825/801947386 7 ==== osd_op_reply(75 500.00000000 [create] v106'233 uv226 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f6878027030 con 0x55ae6b240730 2024-08-03T20:57:38.702 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:38.701+0000 7f6884c1ef40 1 -- 172.21.15.32:0/1550780541 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:76 12.15 12:aa448500:::500.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) -- 0x55ae6b244b90 con 0x55ae6b240730 2024-08-03T20:57:38.704 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:38.702+0000 7f687dc00640 1 -- 172.21.15.32:0/1550780541 <== osd.6 v2:172.21.15.118:6825/801947386 8 ==== osd_op_reply(76 500.00000000 [delete] v106'234 uv234 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f6878027030 con 0x55ae6b240730 2024-08-03T20:57:38.704 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:38.702+0000 7f6884c1ef40 1 -- 172.21.15.32:0/1550780541 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:77 12.15 12:aa448500:::500.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106) -- 0x55ae6b244b90 con 0x55ae6b240730 2024-08-03T20:57:38.705 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::500.00000000:head# 2024-08-03T20:57:38.705 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:38.704+0000 7f687dc00640 1 -- 172.21.15.32:0/1550780541 <== osd.6 v2:172.21.15.118:6825/801947386 9 ==== osd_op_reply(77 500.00000000 [create] v106'235 uv235 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f6878027030 con 0x55ae6b240730 2024-08-03T20:57:38.705 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:38.704+0000 7f6884c1ef40 1 -- 172.21.15.32:0/1550780541 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:78 12.15 12:aa448500:::500.00000000:head [write 0~90 in=90b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106) -- 0x55ae6b244b40 con 0x55ae6b240730 2024-08-03T20:57:38.707 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:38.705+0000 7f687dc00640 1 -- 172.21.15.32:0/1550780541 <== osd.6 v2:172.21.15.118:6825/801947386 10 ==== osd_op_reply(78 500.00000000 [write 0~90] v106'236 uv236 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f6878027030 con 0x55ae6b240730 2024-08-03T20:57:38.707 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:38.705+0000 7f6884c1ef40 1 -- 172.21.15.32:0/1550780541 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:79 12.15 12:aa448500:::500.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106) -- 0x55ae6b244b40 con 0x55ae6b240730 2024-08-03T20:57:38.708 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:38.707+0000 7f687dc00640 1 -- 172.21.15.32:0/1550780541 <== osd.6 v2:172.21.15.118:6825/801947386 11 ==== osd_op_reply(79 500.00000000 [omap-set-header] v106'237 uv237 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f6878027030 con 0x55ae6b240730 2024-08-03T20:57:38.708 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:38.707+0000 7f6884c1ef40 1 --2- 172.21.15.32:0/1550780541 >> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] conn(0x55ae6b244f80 0x55ae6b248020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:38.708 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:38.707+0000 7f6884c1ef40 1 -- 172.21.15.32:0/1550780541 --> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] -- osd_op(unknown.0.0:80 12.d 12:b50e409b:::mds_snaptable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106) -- 0x55ae6b248560 con 0x55ae6b244f80 2024-08-03T20:57:38.708 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:38.707+0000 7f687d200640 1 --2- 172.21.15.32:0/1550780541 >> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] conn(0x55ae6b244f80 0x55ae6b248020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:38.709 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:38.708+0000 7f687d200640 1 --2- 172.21.15.32:0/1550780541 >> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] conn(0x55ae6b244f80 0x55ae6b248020 crc :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:57:38.710 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:38.709+0000 7f687d200640 1 -- 172.21.15.32:0/1550780541 <== osd.7 v2:172.21.15.118:6818/653000048 1 ==== osd_op_reply(80 mds_snaptable [create] v106'102 uv101 ondisk = -17 ((17) File exists)) ==== 157+0+0 (crc 0 0 0) 0x7f68700128d0 con 0x55ae6b244f80 2024-08-03T20:57:38.711 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:38.709+0000 7f6884c1ef40 1 -- 172.21.15.32:0/1550780541 --> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] -- osd_op(unknown.0.0:81 12.d 12:b50e409b:::mds_snaptable:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) -- 0x55ae6b248560 con 0x55ae6b244f80 2024-08-03T20:57:38.712 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:38.710+0000 7f687d200640 1 -- 172.21.15.32:0/1550780541 <== osd.7 v2:172.21.15.118:6818/653000048 2 ==== osd_op_reply(81 mds_snaptable [delete] v106'103 uv103 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f68700128d0 con 0x55ae6b244f80 2024-08-03T20:57:38.712 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:38.710+0000 7f6884c1ef40 1 -- 172.21.15.32:0/1550780541 --> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] -- osd_op(unknown.0.0:82 12.d 12:b50e409b:::mds_snaptable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106) -- 0x55ae6b2485e0 con 0x55ae6b244f80 2024-08-03T20:57:38.713 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:38.712+0000 7f687d200640 1 -- 172.21.15.32:0/1550780541 <== osd.7 v2:172.21.15.118:6818/653000048 3 ==== osd_op_reply(82 mds_snaptable [create] v106'104 uv104 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f68700128d0 con 0x55ae6b244f80 2024-08-03T20:57:38.714 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds_snaptable:head# 2024-08-03T20:57:38.714 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:38.712+0000 7f6884c1ef40 1 -- 172.21.15.32:0/1550780541 --> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] -- osd_op(unknown.0.0:83 12.d 12:b50e409b:::mds_snaptable:head [write 0~70 in=70b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106) -- 0x55ae6b2489d0 con 0x55ae6b244f80 2024-08-03T20:57:38.714 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:38.713+0000 7f687d200640 1 -- 172.21.15.32:0/1550780541 <== osd.7 v2:172.21.15.118:6818/653000048 4 ==== osd_op_reply(83 mds_snaptable [write 0~70] v106'105 uv105 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f68700128d0 con 0x55ae6b244f80 2024-08-03T20:57:38.715 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:38.713+0000 7f6884c1ef40 1 -- 172.21.15.32:0/1550780541 --> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] -- osd_op(unknown.0.0:84 12.d 12:b50e409b:::mds_snaptable:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106) -- 0x55ae6b2489d0 con 0x55ae6b244f80 2024-08-03T20:57:38.716 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:38.715+0000 7f687d200640 1 -- 172.21.15.32:0/1550780541 <== osd.7 v2:172.21.15.118:6818/653000048 5 ==== osd_op_reply(84 mds_snaptable [omap-set-header] v106'106 uv106 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f68700128d0 con 0x55ae6b244f80 2024-08-03T20:57:38.716 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:38.715+0000 7f6884c1ef40 1 -- 172.21.15.32:0/1550780541 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:85 12.1d 12:b871830b:::605.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106) -- 0x55ae6b2489d0 con 0x55ae6b16f3f0 2024-08-03T20:57:38.717 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:38.716+0000 7f6877e00640 1 -- 172.21.15.32:0/1550780541 <== osd.0 v2:172.21.15.32:6816/3066156116 44 ==== osd_op_reply(85 605.00000000 [create] v106'123 uv122 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f686804ed50 con 0x55ae6b16f3f0 2024-08-03T20:57:38.717 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:38.716+0000 7f6884c1ef40 1 -- 172.21.15.32:0/1550780541 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:86 12.1d 12:b871830b:::605.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) -- 0x55ae6b2489d0 con 0x55ae6b16f3f0 2024-08-03T20:57:38.719 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:38.717+0000 7f6877e00640 1 -- 172.21.15.32:0/1550780541 <== osd.0 v2:172.21.15.32:6816/3066156116 45 ==== osd_op_reply(86 605.00000000 [delete] v106'124 uv124 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f686804ed50 con 0x55ae6b16f3f0 2024-08-03T20:57:38.719 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:38.717+0000 7f6884c1ef40 1 -- 172.21.15.32:0/1550780541 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:87 12.1d 12:b871830b:::605.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106) -- 0x55ae6b2489d0 con 0x55ae6b16f3f0 2024-08-03T20:57:38.720 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::605.00000000:head# 2024-08-03T20:57:38.720 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:38.719+0000 7f6877e00640 1 -- 172.21.15.32:0/1550780541 <== osd.0 v2:172.21.15.32:6816/3066156116 46 ==== osd_op_reply(87 605.00000000 [create] v106'125 uv125 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f686804ed50 con 0x55ae6b16f3f0 2024-08-03T20:57:38.720 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:38.719+0000 7f6884c1ef40 1 -- 172.21.15.32:0/1550780541 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:88 12.1d 12:b871830b:::605.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106) -- 0x55ae6b2499e0 con 0x55ae6b16f3f0 2024-08-03T20:57:38.721 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:38.720+0000 7f6877e00640 1 -- 172.21.15.32:0/1550780541 <== osd.0 v2:172.21.15.32:6816/3066156116 47 ==== osd_op_reply(88 605.00000000 [setxattr (30)] v106'126 uv126 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f686804ed50 con 0x55ae6b16f3f0 2024-08-03T20:57:38.722 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:38.720+0000 7f6884c1ef40 1 -- 172.21.15.32:0/1550780541 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:89 12.1d 12:b871830b:::605.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106) -- 0x55ae6b2499e0 con 0x55ae6b16f3f0 2024-08-03T20:57:38.723 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:38.721+0000 7f6877e00640 1 -- 172.21.15.32:0/1550780541 <== osd.0 v2:172.21.15.32:6816/3066156116 48 ==== osd_op_reply(89 605.00000000 [setxattr (62)] v106'127 uv127 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f686804ed50 con 0x55ae6b16f3f0 2024-08-03T20:57:38.723 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:38.721+0000 7f6884c1ef40 1 -- 172.21.15.32:0/1550780541 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:90 12.1d 12:b871830b:::605.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106) -- 0x55ae6b2489d0 con 0x55ae6b16f3f0 2024-08-03T20:57:38.724 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:38.723+0000 7f6877e00640 1 -- 172.21.15.32:0/1550780541 <== osd.0 v2:172.21.15.32:6816/3066156116 49 ==== osd_op_reply(90 605.00000000 [omap-set-header] v106'128 uv128 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f686804ed50 con 0x55ae6b16f3f0 2024-08-03T20:57:38.724 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:38.723+0000 7f6884c1ef40 1 -- 172.21.15.32:0/1550780541 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:91 12.13 12:c91d4a1d:::mds0_inotable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106) -- 0x55ae6b2489d0 con 0x55ae6b23d4a0 2024-08-03T20:57:38.726 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:38.724+0000 7f6877e00640 1 -- 172.21.15.32:0/1550780541 <== osd.1 v2:172.21.15.32:6824/1551241759 5 ==== osd_op_reply(91 mds0_inotable [create] v106'203 uv202 ondisk = -17 ((17) File exists)) ==== 157+0+0 (crc 0 0 0) 0x7f686804ed50 con 0x55ae6b23d4a0 2024-08-03T20:57:38.726 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:38.724+0000 7f6884c1ef40 1 -- 172.21.15.32:0/1550780541 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:92 12.13 12:c91d4a1d:::mds0_inotable:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) -- 0x55ae6b2489d0 con 0x55ae6b23d4a0 2024-08-03T20:57:38.727 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:38.726+0000 7f6877e00640 1 -- 172.21.15.32:0/1550780541 <== osd.1 v2:172.21.15.32:6824/1551241759 6 ==== osd_op_reply(92 mds0_inotable [delete] v106'204 uv204 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f686804ed50 con 0x55ae6b23d4a0 2024-08-03T20:57:38.727 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:38.726+0000 7f6884c1ef40 1 -- 172.21.15.32:0/1550780541 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:93 12.13 12:c91d4a1d:::mds0_inotable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106) -- 0x55ae6b2489d0 con 0x55ae6b23d4a0 2024-08-03T20:57:38.728 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:38.727+0000 7f6877e00640 1 -- 172.21.15.32:0/1550780541 <== osd.1 v2:172.21.15.32:6824/1551241759 7 ==== osd_op_reply(93 mds0_inotable [create] v106'205 uv205 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f686804ed50 con 0x55ae6b23d4a0 2024-08-03T20:57:38.728 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds0_inotable:head# 2024-08-03T20:57:38.729 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:38.727+0000 7f6884c1ef40 1 -- 172.21.15.32:0/1550780541 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:94 12.13 12:c91d4a1d:::mds0_inotable:head [write 0~34 in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106) -- 0x55ae6b2489d0 con 0x55ae6b23d4a0 2024-08-03T20:57:38.729 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:38.728+0000 7f6877e00640 1 -- 172.21.15.32:0/1550780541 <== osd.1 v2:172.21.15.32:6824/1551241759 8 ==== osd_op_reply(94 mds0_inotable [write 0~34] v106'206 uv206 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f686804ed50 con 0x55ae6b23d4a0 2024-08-03T20:57:38.730 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:38.728+0000 7f6884c1ef40 1 -- 172.21.15.32:0/1550780541 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:95 12.13 12:c91d4a1d:::mds0_inotable:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106) -- 0x55ae6b2489d0 con 0x55ae6b23d4a0 2024-08-03T20:57:38.731 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:38.729+0000 7f6877e00640 1 -- 172.21.15.32:0/1550780541 <== osd.1 v2:172.21.15.32:6824/1551241759 9 ==== osd_op_reply(95 mds0_inotable [omap-set-header] v106'207 uv207 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f686804ed50 con 0x55ae6b23d4a0 2024-08-03T20:57:38.731 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:38.730+0000 7f6884c1ef40 1 -- 172.21.15.32:0/1550780541 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:96 12.13 12:cd5a64a3:::100.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106) -- 0x55ae6b2489d0 con 0x55ae6b23d4a0 2024-08-03T20:57:38.732 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:38.731+0000 7f6877e00640 1 -- 172.21.15.32:0/1550780541 <== osd.1 v2:172.21.15.32:6824/1551241759 10 ==== osd_op_reply(96 100.00000000 [create] v106'208 uv201 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f686804ed50 con 0x55ae6b23d4a0 2024-08-03T20:57:38.732 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:38.731+0000 7f6884c1ef40 1 -- 172.21.15.32:0/1550780541 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:97 12.13 12:cd5a64a3:::100.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) -- 0x55ae6b2489d0 con 0x55ae6b23d4a0 2024-08-03T20:57:38.734 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:38.732+0000 7f6877e00640 1 -- 172.21.15.32:0/1550780541 <== osd.1 v2:172.21.15.32:6824/1551241759 11 ==== osd_op_reply(97 100.00000000 [delete] v106'209 uv209 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f686804ed50 con 0x55ae6b23d4a0 2024-08-03T20:57:38.734 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:38.732+0000 7f6884c1ef40 1 -- 172.21.15.32:0/1550780541 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:98 12.13 12:cd5a64a3:::100.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106) -- 0x55ae6b2489d0 con 0x55ae6b23d4a0 2024-08-03T20:57:38.735 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::100.00000000:head# 2024-08-03T20:57:38.735 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:38.733+0000 7f6877e00640 1 -- 172.21.15.32:0/1550780541 <== osd.1 v2:172.21.15.32:6824/1551241759 12 ==== osd_op_reply(98 100.00000000 [create] v106'210 uv210 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f686804ed50 con 0x55ae6b23d4a0 2024-08-03T20:57:38.735 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:38.733+0000 7f6884c1ef40 1 -- 172.21.15.32:0/1550780541 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:99 12.13 12:cd5a64a3:::100.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106) -- 0x55ae6b2489d0 con 0x55ae6b23d4a0 2024-08-03T20:57:38.736 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:38.735+0000 7f6877e00640 1 -- 172.21.15.32:0/1550780541 <== osd.1 v2:172.21.15.32:6824/1551241759 13 ==== osd_op_reply(99 100.00000000 [omap-set-header] v106'211 uv211 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f686804ed50 con 0x55ae6b23d4a0 2024-08-03T20:57:38.736 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:38.735+0000 7f6884c1ef40 1 -- 172.21.15.32:0/1550780541 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:100 12.13 12:cd5a64a3:::100.00000000:head [omap-set-vals in=5194b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106) -- 0x55ae6b24b750 con 0x55ae6b23d4a0 2024-08-03T20:57:38.738 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:38.736+0000 7f6877e00640 1 -- 172.21.15.32:0/1550780541 <== osd.1 v2:172.21.15.32:6824/1551241759 14 ==== osd_op_reply(100 100.00000000 [omap-set-vals] v106'212 uv212 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f686804ed50 con 0x55ae6b23d4a0 2024-08-03T20:57:38.738 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:38.736+0000 7f6884c1ef40 1 --2- 172.21.15.32:0/1550780541 >> [v2:172.21.15.32:6800/9088303,v1:172.21.15.32:6801/9088303] conn(0x55ae6b248d50 0x55ae6b24b170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:38.738 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:38.736+0000 7f6884c1ef40 1 -- 172.21.15.32:0/1550780541 --> [v2:172.21.15.32:6800/9088303,v1:172.21.15.32:6801/9088303] -- osd_op(unknown.0.0:101 12.b 12:d0630e4c:::mds0_sessionmap:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106) -- 0x55ae6b24be10 con 0x55ae6b248d50 2024-08-03T20:57:38.738 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:38.737+0000 7f6877e00640 1 --2- 172.21.15.32:0/1550780541 >> [v2:172.21.15.32:6800/9088303,v1:172.21.15.32:6801/9088303] conn(0x55ae6b248d50 0x55ae6b24b170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:38.738 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:38.737+0000 7f6877e00640 1 --2- 172.21.15.32:0/1550780541 >> [v2:172.21.15.32:6800/9088303,v1:172.21.15.32:6801/9088303] conn(0x55ae6b248d50 0x55ae6b24b170 crc :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:57:38.740 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:38.739+0000 7f6877e00640 1 -- 172.21.15.32:0/1550780541 <== osd.3 v2:172.21.15.32:6800/9088303 1 ==== osd_op_reply(101 mds0_sessionmap [create] v106'91 uv90 ondisk = -17 ((17) File exists)) ==== 159+0+0 (crc 0 0 0) 0x7f686804ed50 con 0x55ae6b248d50 2024-08-03T20:57:38.740 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:38.739+0000 7f6884c1ef40 1 -- 172.21.15.32:0/1550780541 --> [v2:172.21.15.32:6800/9088303,v1:172.21.15.32:6801/9088303] -- osd_op(unknown.0.0:102 12.b 12:d0630e4c:::mds0_sessionmap:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) -- 0x55ae6b24be10 con 0x55ae6b248d50 2024-08-03T20:57:38.741 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:38.740+0000 7f6877e00640 1 -- 172.21.15.32:0/1550780541 <== osd.3 v2:172.21.15.32:6800/9088303 2 ==== osd_op_reply(102 mds0_sessionmap [delete] v106'92 uv92 ondisk = 0) ==== 159+0+0 (crc 0 0 0) 0x7f686804ed50 con 0x55ae6b248d50 2024-08-03T20:57:38.742 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:38.740+0000 7f6884c1ef40 1 -- 172.21.15.32:0/1550780541 --> [v2:172.21.15.32:6800/9088303,v1:172.21.15.32:6801/9088303] -- osd_op(unknown.0.0:103 12.b 12:d0630e4c:::mds0_sessionmap:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106) -- 0x55ae6b24be10 con 0x55ae6b248d50 2024-08-03T20:57:38.743 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds0_sessionmap:head# 2024-08-03T20:57:38.743 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:38.741+0000 7f6877e00640 1 -- 172.21.15.32:0/1550780541 <== osd.3 v2:172.21.15.32:6800/9088303 3 ==== osd_op_reply(103 mds0_sessionmap [create] v106'93 uv93 ondisk = 0) ==== 159+0+0 (crc 0 0 0) 0x7f686804ed50 con 0x55ae6b248d50 2024-08-03T20:57:38.743 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:38.741+0000 7f6884c1ef40 1 -- 172.21.15.32:0/1550780541 --> [v2:172.21.15.32:6800/9088303,v1:172.21.15.32:6801/9088303] -- osd_op(unknown.0.0:104 12.b 12:d0630e4c:::mds0_sessionmap:head [omap-set-header in=14b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106) -- 0x55ae6b24c2c0 con 0x55ae6b248d50 2024-08-03T20:57:38.744 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:38.743+0000 7f6877e00640 1 -- 172.21.15.32:0/1550780541 <== osd.3 v2:172.21.15.32:6800/9088303 4 ==== osd_op_reply(104 mds0_sessionmap [omap-set-header] v106'94 uv94 ondisk = 0) ==== 159+0+0 (crc 0 0 0) 0x7f686804ed50 con 0x55ae6b248d50 2024-08-03T20:57:38.744 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:38.743+0000 7f6884c1ef40 1 -- 172.21.15.32:0/1550780541 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:105 12.7 12:e0b41d06:::609.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106) -- 0x55ae6b24c2c0 con 0x55ae6b23d4a0 2024-08-03T20:57:38.746 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:38.744+0000 7f6877e00640 1 -- 172.21.15.32:0/1550780541 <== osd.1 v2:172.21.15.32:6824/1551241759 15 ==== osd_op_reply(105 609.00000000 [create] v106'123 uv122 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f686804ed50 con 0x55ae6b23d4a0 2024-08-03T20:57:38.746 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:38.744+0000 7f6884c1ef40 1 -- 172.21.15.32:0/1550780541 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:106 12.7 12:e0b41d06:::609.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) -- 0x55ae6b24c270 con 0x55ae6b23d4a0 2024-08-03T20:57:38.747 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:38.746+0000 7f6877e00640 1 -- 172.21.15.32:0/1550780541 <== osd.1 v2:172.21.15.32:6824/1551241759 16 ==== osd_op_reply(106 609.00000000 [delete] v106'124 uv124 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f686804ed50 con 0x55ae6b23d4a0 2024-08-03T20:57:38.747 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:38.746+0000 7f6884c1ef40 1 -- 172.21.15.32:0/1550780541 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:107 12.7 12:e0b41d06:::609.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106) -- 0x55ae6b24c270 con 0x55ae6b23d4a0 2024-08-03T20:57:38.749 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::609.00000000:head# 2024-08-03T20:57:38.749 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:38.747+0000 7f6877e00640 1 -- 172.21.15.32:0/1550780541 <== osd.1 v2:172.21.15.32:6824/1551241759 17 ==== osd_op_reply(107 609.00000000 [create] v106'125 uv125 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f686804ed50 con 0x55ae6b23d4a0 2024-08-03T20:57:38.749 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:38.747+0000 7f6884c1ef40 1 -- 172.21.15.32:0/1550780541 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:108 12.7 12:e0b41d06:::609.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106) -- 0x55ae6b24d280 con 0x55ae6b23d4a0 2024-08-03T20:57:38.750 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:38.749+0000 7f6877e00640 1 -- 172.21.15.32:0/1550780541 <== osd.1 v2:172.21.15.32:6824/1551241759 18 ==== osd_op_reply(108 609.00000000 [setxattr (30)] v106'126 uv126 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f686804ed50 con 0x55ae6b23d4a0 2024-08-03T20:57:38.750 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:38.749+0000 7f6884c1ef40 1 -- 172.21.15.32:0/1550780541 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:109 12.7 12:e0b41d06:::609.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106) -- 0x55ae6b24d280 con 0x55ae6b23d4a0 2024-08-03T20:57:38.751 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:38.750+0000 7f6877e00640 1 -- 172.21.15.32:0/1550780541 <== osd.1 v2:172.21.15.32:6824/1551241759 19 ==== osd_op_reply(109 609.00000000 [setxattr (62)] v106'127 uv127 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f686804ed50 con 0x55ae6b23d4a0 2024-08-03T20:57:38.752 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:38.750+0000 7f6884c1ef40 1 -- 172.21.15.32:0/1550780541 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:110 12.7 12:e0b41d06:::609.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106) -- 0x55ae6b24c270 con 0x55ae6b23d4a0 2024-08-03T20:57:38.753 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:38.752+0000 7f6877e00640 1 -- 172.21.15.32:0/1550780541 <== osd.1 v2:172.21.15.32:6824/1551241759 20 ==== osd_op_reply(110 609.00000000 [omap-set-header] v106'128 uv128 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f686804ed50 con 0x55ae6b23d4a0 2024-08-03T20:57:38.753 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:38.752+0000 7f6884c1ef40 1 -- 172.21.15.32:0/1550780541 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:111 12.f 12:f1f69726:::400.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106) -- 0x55ae6b24c270 con 0x55ae6b240730 2024-08-03T20:57:38.755 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:38.753+0000 7f687dc00640 1 -- 172.21.15.32:0/1550780541 <== osd.6 v2:172.21.15.118:6825/801947386 12 ==== osd_op_reply(111 400.00000000 [create] v106'104 uv103 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f6878027030 con 0x55ae6b240730 2024-08-03T20:57:38.755 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:38.754+0000 7f6884c1ef40 1 -- 172.21.15.32:0/1550780541 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:112 12.f 12:f1f69726:::400.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) -- 0x55ae6b24c270 con 0x55ae6b240730 2024-08-03T20:57:38.757 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:38.755+0000 7f687dc00640 1 -- 172.21.15.32:0/1550780541 <== osd.6 v2:172.21.15.118:6825/801947386 13 ==== osd_op_reply(112 400.00000000 [delete] v106'105 uv105 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f6878027030 con 0x55ae6b240730 2024-08-03T20:57:38.757 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:38.755+0000 7f6884c1ef40 1 -- 172.21.15.32:0/1550780541 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:113 12.f 12:f1f69726:::400.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106) -- 0x55ae6b24c270 con 0x55ae6b240730 2024-08-03T20:57:38.758 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::400.00000000:head# 2024-08-03T20:57:38.759 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:38.757+0000 7f687dc00640 1 -- 172.21.15.32:0/1550780541 <== osd.6 v2:172.21.15.118:6825/801947386 14 ==== osd_op_reply(113 400.00000000 [create] v106'106 uv106 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f6878027030 con 0x55ae6b240730 2024-08-03T20:57:38.759 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:38.757+0000 7f6884c1ef40 1 -- 172.21.15.32:0/1550780541 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:114 12.f 12:f1f69726:::400.00000000:head [write 0~22 in=22b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106) -- 0x55ae6b24c270 con 0x55ae6b240730 2024-08-03T20:57:38.760 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:38.759+0000 7f687dc00640 1 -- 172.21.15.32:0/1550780541 <== osd.6 v2:172.21.15.118:6825/801947386 15 ==== osd_op_reply(114 400.00000000 [write 0~22] v106'107 uv107 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f6878027030 con 0x55ae6b240730 2024-08-03T20:57:38.760 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:38.759+0000 7f6884c1ef40 1 -- 172.21.15.32:0/1550780541 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:115 12.f 12:f1f69726:::400.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106) -- 0x55ae6b24c270 con 0x55ae6b240730 2024-08-03T20:57:38.761 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:38.760+0000 7f687dc00640 1 -- 172.21.15.32:0/1550780541 <== osd.6 v2:172.21.15.118:6825/801947386 16 ==== osd_op_reply(115 400.00000000 [omap-set-header] v106'108 uv108 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f6878027030 con 0x55ae6b240730 2024-08-03T20:57:38.761 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:38.760+0000 7f6884c1ef40 1 -- 172.21.15.32:0/1550780541 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:116 12.1f 12:fe07bba1:::100.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106) -- 0x55ae6b24c270 con 0x55ae6b240730 2024-08-03T20:57:38.763 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:38.762+0000 7f687dc00640 1 -- 172.21.15.32:0/1550780541 <== osd.6 v2:172.21.15.118:6825/801947386 17 ==== osd_op_reply(116 100.00000000.inode [create] v106'253 uv245 ondisk = -17 ((17) File exists)) ==== 162+0+0 (crc 0 0 0) 0x7f6878027030 con 0x55ae6b240730 2024-08-03T20:57:38.763 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:38.762+0000 7f6884c1ef40 1 -- 172.21.15.32:0/1550780541 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:117 12.1f 12:fe07bba1:::100.00000000.inode:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) -- 0x55ae6b24c270 con 0x55ae6b240730 2024-08-03T20:57:38.765 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:38.763+0000 7f687dc00640 1 -- 172.21.15.32:0/1550780541 <== osd.6 v2:172.21.15.118:6825/801947386 18 ==== osd_op_reply(117 100.00000000.inode [delete] v106'254 uv254 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7f6878027030 con 0x55ae6b240730 2024-08-03T20:57:38.765 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:38.763+0000 7f6884c1ef40 1 -- 172.21.15.32:0/1550780541 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:118 12.1f 12:fe07bba1:::100.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106) -- 0x55ae6b24c270 con 0x55ae6b240730 2024-08-03T20:57:38.766 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:38.765+0000 7f687dc00640 1 -- 172.21.15.32:0/1550780541 <== osd.6 v2:172.21.15.118:6825/801947386 19 ==== osd_op_reply(118 100.00000000.inode [create] v106'255 uv255 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7f6878027030 con 0x55ae6b240730 2024-08-03T20:57:38.766 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::100.00000000.inode:head# 2024-08-03T20:57:38.766 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:38.765+0000 7f6884c1ef40 1 -- 172.21.15.32:0/1550780541 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:119 12.1f 12:fe07bba1:::100.00000000.inode:head [write 0~582 in=582b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106) -- 0x55ae6b24c270 con 0x55ae6b240730 2024-08-03T20:57:38.768 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:38.766+0000 7f687dc00640 1 -- 172.21.15.32:0/1550780541 <== osd.6 v2:172.21.15.118:6825/801947386 20 ==== osd_op_reply(119 100.00000000.inode [write 0~582] v106'256 uv256 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7f6878027030 con 0x55ae6b240730 2024-08-03T20:57:38.768 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:38.766+0000 7f6884c1ef40 1 -- 172.21.15.32:0/1550780541 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:120 12.1f 12:fe07bba1:::100.00000000.inode:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106) -- 0x55ae6b24c270 con 0x55ae6b240730 2024-08-03T20:57:38.769 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:38.768+0000 7f687dc00640 1 -- 172.21.15.32:0/1550780541 <== osd.6 v2:172.21.15.118:6825/801947386 21 ==== osd_op_reply(120 100.00000000.inode [omap-set-header] v106'257 uv257 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7f6878027030 con 0x55ae6b240730 2024-08-03T20:57:38.769 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:38.768+0000 7f6884c1ef40 1 -- 172.21.15.32:0/1550780541 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:121 12.1f 12:ff5b34d6:::1.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106) -- 0x55ae6b24c270 con 0x55ae6b240730 2024-08-03T20:57:38.771 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:38.770+0000 7f687dc00640 1 -- 172.21.15.32:0/1550780541 <== osd.6 v2:172.21.15.118:6825/801947386 22 ==== osd_op_reply(121 1.00000000 [create] v106'258 uv252 ondisk = -17 ((17) File exists)) ==== 154+0+0 (crc 0 0 0) 0x7f6878027030 con 0x55ae6b240730 2024-08-03T20:57:38.771 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:38.770+0000 7f6884c1ef40 1 -- 172.21.15.32:0/1550780541 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:122 12.1f 12:ff5b34d6:::1.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) -- 0x55ae6b24c270 con 0x55ae6b240730 2024-08-03T20:57:38.772 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:38.771+0000 7f687dc00640 1 -- 172.21.15.32:0/1550780541 <== osd.6 v2:172.21.15.118:6825/801947386 23 ==== osd_op_reply(122 1.00000000 [delete] v106'259 uv259 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7f6878027030 con 0x55ae6b240730 2024-08-03T20:57:38.773 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:38.771+0000 7f6884c1ef40 1 -- 172.21.15.32:0/1550780541 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:123 12.1f 12:ff5b34d6:::1.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106) -- 0x55ae6b24c270 con 0x55ae6b240730 2024-08-03T20:57:38.774 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:38.773+0000 7f687dc00640 1 -- 172.21.15.32:0/1550780541 <== osd.6 v2:172.21.15.118:6825/801947386 24 ==== osd_op_reply(123 1.00000000 [create] v106'260 uv260 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7f6878027030 con 0x55ae6b240730 2024-08-03T20:57:38.774 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::1.00000000:head# 2024-08-03T20:57:38.775 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:38.773+0000 7f6884c1ef40 1 -- 172.21.15.32:0/1550780541 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:124 12.1f 12:ff5b34d6:::1.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106) -- 0x55ae6b24d280 con 0x55ae6b240730 2024-08-03T20:57:38.776 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:38.775+0000 7f687dc00640 1 -- 172.21.15.32:0/1550780541 <== osd.6 v2:172.21.15.118:6825/801947386 25 ==== osd_op_reply(124 1.00000000 [setxattr (30)] v106'261 uv261 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7f6878027030 con 0x55ae6b240730 2024-08-03T20:57:38.776 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:38.775+0000 7f6884c1ef40 1 -- 172.21.15.32:0/1550780541 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:125 12.1f 12:ff5b34d6:::1.00000000:head [setxattr parent (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106) -- 0x55ae6b24d280 con 0x55ae6b240730 2024-08-03T20:57:38.777 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:38.776+0000 7f687dc00640 1 -- 172.21.15.32:0/1550780541 <== osd.6 v2:172.21.15.118:6825/801947386 26 ==== osd_op_reply(125 1.00000000 [setxattr (30)] v106'262 uv262 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7f6878027030 con 0x55ae6b240730 2024-08-03T20:57:38.778 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:38.776+0000 7f6884c1ef40 1 -- 172.21.15.32:0/1550780541 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:126 12.1f 12:ff5b34d6:::1.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106) -- 0x55ae6b24c270 con 0x55ae6b240730 2024-08-03T20:57:38.779 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:38.778+0000 7f687dc00640 1 -- 172.21.15.32:0/1550780541 <== osd.6 v2:172.21.15.118:6825/801947386 27 ==== osd_op_reply(126 1.00000000 [omap-set-header] v106'263 uv263 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7f6878027030 con 0x55ae6b240730 2024-08-03T20:57:38.779 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:38.778+0000 7f6884c1ef40 1 -- 172.21.15.32:0/1550780541 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:127 12.1f 12:ff5b34d6:::1.00000000:head [omap-set-vals in=584b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106) -- 0x55ae6b24c7d0 con 0x55ae6b240730 2024-08-03T20:57:38.782 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:38.779+0000 7f687dc00640 1 -- 172.21.15.32:0/1550780541 <== osd.6 v2:172.21.15.118:6825/801947386 28 ==== osd_op_reply(127 1.00000000 [omap-set-vals] v106'264 uv264 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7f6878027030 con 0x55ae6b240730 2024-08-03T20:57:38.782 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:38.779+0000 7f6884c1ef40 1 -- 172.21.15.32:0/1550780541 >> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] conn(0x55ae6b16f3f0 msgr2=0x55ae6b101880 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:38.782 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:38.779+0000 7f6884c1ef40 1 --2- 172.21.15.32:0/1550780541 >> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] conn(0x55ae6b16f3f0 0x55ae6b101880 crc :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:38.782 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:38.779+0000 7f6884c1ef40 1 -- 172.21.15.32:0/1550780541 >> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] conn(0x55ae6b23d4a0 msgr2=0x55ae6b23f8c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:38.782 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:38.779+0000 7f6884c1ef40 1 --2- 172.21.15.32:0/1550780541 >> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] conn(0x55ae6b23d4a0 0x55ae6b23f8c0 crc :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:38.782 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:38.779+0000 7f6884c1ef40 1 -- 172.21.15.32:0/1550780541 >> [v2:172.21.15.32:6800/9088303,v1:172.21.15.32:6801/9088303] conn(0x55ae6b248d50 msgr2=0x55ae6b24b170 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:38.782 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:38.779+0000 7f6884c1ef40 1 --2- 172.21.15.32:0/1550780541 >> [v2:172.21.15.32:6800/9088303,v1:172.21.15.32:6801/9088303] conn(0x55ae6b248d50 0x55ae6b24b170 crc :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:38.782 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:38.779+0000 7f6884c1ef40 1 -- 172.21.15.32:0/1550780541 >> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] conn(0x55ae6b237e70 msgr2=0x55ae6b23afe0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:38.782 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:38.779+0000 7f6884c1ef40 1 --2- 172.21.15.32:0/1550780541 >> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] conn(0x55ae6b237e70 0x55ae6b23afe0 crc :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:38.782 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:38.779+0000 7f687dc00640 1 -- 172.21.15.32:0/1550780541 reap_dead start 2024-08-03T20:57:38.782 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:38.779+0000 7f6884c1ef40 1 -- 172.21.15.32:0/1550780541 >> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] conn(0x55ae6b240730 msgr2=0x55ae6b242b50 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:38.782 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:38.779+0000 7f6884c1ef40 1 --2- 172.21.15.32:0/1550780541 >> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] conn(0x55ae6b240730 0x55ae6b242b50 crc :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:38.783 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:38.779+0000 7f6884c1ef40 1 -- 172.21.15.32:0/1550780541 >> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] conn(0x55ae6b244f80 msgr2=0x55ae6b248020 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:38.783 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:38.779+0000 7f6884c1ef40 1 --2- 172.21.15.32:0/1550780541 >> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] conn(0x55ae6b244f80 0x55ae6b248020 crc :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:38.783 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:38.779+0000 7f6884c1ef40 1 -- 172.21.15.32:0/1550780541 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f685c0759f0 msgr2=0x7f685c077f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:38.783 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:38.779+0000 7f6884c1ef40 1 --2- 172.21.15.32:0/1550780541 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f685c0759f0 0x7f685c077f00 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x55ae6b194780 tx=0x7f6870001730 comp rx=0 tx=0).stop 2024-08-03T20:57:38.783 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:38.779+0000 7f6884c1ef40 1 -- 172.21.15.32:0/1550780541 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ae6b18b330 msgr2=0x55ae6b1921f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:38.783 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:38.779+0000 7f6884c1ef40 1 --2- 172.21.15.32:0/1550780541 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ae6b18b330 0x55ae6b1921f0 secure :-1 s=READY pgs=1592 cs=0 l=1 rev1=1 crypto rx=0x55ae6b16fd70 tx=0x7f687800c370 comp rx=0 tx=0).stop 2024-08-03T20:57:38.783 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:38.779+0000 7f687dc00640 1 -- 172.21.15.32:0/1550780541 reap_dead start 2024-08-03T20:57:38.783 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:38.780+0000 7f6884c1ef40 1 -- 172.21.15.32:0/1550780541 shutdown_connections 2024-08-03T20:57:38.783 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:38.780+0000 7f6884c1ef40 1 -- 172.21.15.32:0/1550780541 >> 172.21.15.32:0/1550780541 conn(0x55ae6afddac0 msgr2=0x55ae6b193060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:57:38.783 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:38.780+0000 7f6884c1ef40 1 -- 172.21.15.32:0/1550780541 shutdown_connections 2024-08-03T20:57:38.783 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:38.780+0000 7f6884c1ef40 1 -- 172.21.15.32:0/1550780541 wait complete. 2024-08-03T20:57:38.794 DEBUG:teuthology.orchestra.run.smithi032:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata put 400.00000000 - 2024-08-03T20:57:38.979 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:38.978+0000 7f27f1e0cf40 1 Processor -- start 2024-08-03T20:57:38.979 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:38.978+0000 7f27f1e0cf40 1 -- start start 2024-08-03T20:57:38.982 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:38.979+0000 7f27f1e0cf40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5566884a7360 0x5566884c5e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:38.982 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:38.979+0000 7f27f1e0cf40 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x5566884a8ea0 0x5566883f0750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:38.982 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:38.979+0000 7f27f1e0cf40 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5566884be330 0x5566884c0470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:38.982 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:38.979+0000 7f27f1e0cf40 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55668833ee30 con 0x5566884be330 2024-08-03T20:57:38.982 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:38.979+0000 7f27f1e0cf40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x55668839cbf0 con 0x5566884a7360 2024-08-03T20:57:38.982 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:38.979+0000 7f27f1e0cf40 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x5566884a8440 con 0x5566884a8ea0 2024-08-03T20:57:38.982 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:38.979+0000 7f27eba00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5566884be330 0x5566884c0470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:38.982 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:38.979+0000 7f27eba00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5566884be330 0x5566884c0470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:33994/0 (socket says 172.21.15.32:33994) 2024-08-03T20:57:38.982 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:38.979+0000 7f27eba00640 1 -- 172.21.15.32:0/425242494 learned_addr learned my addr 172.21.15.32:0/425242494 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:57:38.983 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:38.979+0000 7f27eb000640 1 --2- 172.21.15.32:0/425242494 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5566884a7360 0x5566884c5e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:38.983 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:38.980+0000 7f27ea600640 1 --2- 172.21.15.32:0/425242494 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x5566884a8ea0 0x5566883f0750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:38.983 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:38.980+0000 7f27eba00640 1 -- 172.21.15.32:0/425242494 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x5566884a8ea0 msgr2=0x5566883f0750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:57:38.983 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:38.980+0000 7f27eba00640 1 --2- 172.21.15.32:0/425242494 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x5566884a8ea0 0x5566883f0750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:38.983 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:38.980+0000 7f27eba00640 1 -- 172.21.15.32:0/425242494 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5566884a7360 msgr2=0x5566884c5e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:57:38.983 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:38.980+0000 7f27eba00640 1 --2- 172.21.15.32:0/425242494 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5566884a7360 0x5566884c5e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:38.983 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:38.980+0000 7f27eba00640 1 -- 172.21.15.32:0/425242494 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5566884c7330 con 0x5566884be330 2024-08-03T20:57:38.983 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:38.980+0000 7f27eba00640 1 --2- 172.21.15.32:0/425242494 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5566884be330 0x5566884c0470 secure :-1 s=READY pgs=1594 cs=0 l=1 rev1=1 crypto rx=0x7f27e400b7b0 tx=0x7f27e400bc70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=67275ba075454ff3 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:57:38.983 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:38.980+0000 7f27e9200640 1 -- 172.21.15.32:0/425242494 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27e400db30 con 0x5566884be330 2024-08-03T20:57:38.983 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:38.981+0000 7f27e9200640 1 -- 172.21.15.32:0/425242494 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f27e400ddc0 con 0x5566884be330 2024-08-03T20:57:38.983 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:38.981+0000 7f27e9200640 1 -- 172.21.15.32:0/425242494 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27e400f690 con 0x5566884be330 2024-08-03T20:57:38.985 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:38.981+0000 7f27f1e0cf40 1 -- 172.21.15.32:0/425242494 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5566884be330 msgr2=0x5566884c0470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:38.985 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:38.981+0000 7f27f1e0cf40 1 --2- 172.21.15.32:0/425242494 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5566884be330 0x5566884c0470 secure :-1 s=READY pgs=1594 cs=0 l=1 rev1=1 crypto rx=0x7f27e400b7b0 tx=0x7f27e400bc70 comp rx=0 tx=0).stop 2024-08-03T20:57:38.985 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:38.981+0000 7f27f1e0cf40 1 -- 172.21.15.32:0/425242494 shutdown_connections 2024-08-03T20:57:38.985 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:38.981+0000 7f27f1e0cf40 1 --2- 172.21.15.32:0/425242494 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5566884be330 0x5566884c0470 unknown :-1 s=CLOSED pgs=1594 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:38.985 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:38.981+0000 7f27f1e0cf40 1 --2- 172.21.15.32:0/425242494 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x5566884a8ea0 0x5566883f0750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:38.985 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:38.981+0000 7f27f1e0cf40 1 --2- 172.21.15.32:0/425242494 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5566884a7360 0x5566884c5e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:38.985 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:38.981+0000 7f27f1e0cf40 1 -- 172.21.15.32:0/425242494 >> 172.21.15.32:0/425242494 conn(0x556688310ac0 msgr2=0x5566884c0f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:57:38.985 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:38.981+0000 7f27f1e0cf40 1 -- 172.21.15.32:0/425242494 shutdown_connections 2024-08-03T20:57:38.985 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:38.981+0000 7f27f1e0cf40 1 -- 172.21.15.32:0/425242494 wait complete. 2024-08-03T20:57:38.985 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:38.982+0000 7f27f1e0cf40 1 Processor -- start 2024-08-03T20:57:38.985 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:38.983+0000 7f27f1e0cf40 1 -- start start 2024-08-03T20:57:38.987 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:38.984+0000 7f27f1e0cf40 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x5566884a7360 0x55668855e200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:38.987 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:38.984+0000 7f27f1e0cf40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5566884a8ea0 0x55668849b2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:38.987 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:38.984+0000 7f27f1e0cf40 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5566884be330 0x556688437020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:38.987 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:38.984+0000 7f27f1e0cf40 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x556688439f90 con 0x5566884be330 2024-08-03T20:57:38.987 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:38.984+0000 7f27f1e0cf40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x556688560e70 con 0x5566884a8ea0 2024-08-03T20:57:38.988 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:38.984+0000 7f27f1e0cf40 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x556688561c40 con 0x5566884a7360 2024-08-03T20:57:38.988 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:38.984+0000 7f27eba00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5566884be330 0x556688437020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:38.988 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:38.984+0000 7f27eba00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5566884be330 0x556688437020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:34004/0 (socket says 172.21.15.32:34004) 2024-08-03T20:57:38.988 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:38.984+0000 7f27eba00640 1 -- 172.21.15.32:0/2483044197 learned_addr learned my addr 172.21.15.32:0/2483044197 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:57:38.988 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:38.984+0000 7f27ea600640 1 --2- 172.21.15.32:0/2483044197 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5566884a8ea0 0x55668849b2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:38.988 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:38.985+0000 7f27eb000640 1 --2- 172.21.15.32:0/2483044197 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x5566884a7360 0x55668855e200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:38.988 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:38.985+0000 7f27eba00640 1 -- 172.21.15.32:0/2483044197 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x5566884a7360 msgr2=0x55668855e200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:38.988 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:38.985+0000 7f27eba00640 1 --2- 172.21.15.32:0/2483044197 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x5566884a7360 0x55668855e200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:38.988 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:38.985+0000 7f27eba00640 1 -- 172.21.15.32:0/2483044197 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5566884a8ea0 msgr2=0x55668849b2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:38.988 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:38.985+0000 7f27eba00640 1 --2- 172.21.15.32:0/2483044197 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5566884a8ea0 0x55668849b2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:38.988 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:38.985+0000 7f27eba00640 1 -- 172.21.15.32:0/2483044197 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55668849b7e0 con 0x5566884be330 2024-08-03T20:57:38.988 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:38.985+0000 7f27eba00640 1 --2- 172.21.15.32:0/2483044197 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5566884be330 0x556688437020 secure :-1 s=READY pgs=1595 cs=0 l=1 rev1=1 crypto rx=0x7f27e400d720 tx=0x7f27e400db60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:57:38.989 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:38.985+0000 7f27e3400640 1 -- 172.21.15.32:0/2483044197 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27e4004220 con 0x5566884be330 2024-08-03T20:57:38.989 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:38.985+0000 7f27e3400640 1 -- 172.21.15.32:0/2483044197 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f27e400f730 con 0x5566884be330 2024-08-03T20:57:38.989 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:38.985+0000 7f27f1e0cf40 1 -- 172.21.15.32:0/2483044197 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x5566885610c0 con 0x5566884be330 2024-08-03T20:57:38.989 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:38.985+0000 7f27e3400640 1 -- 172.21.15.32:0/2483044197 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27e400fb60 con 0x5566884be330 2024-08-03T20:57:38.989 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:38.985+0000 7f27f1e0cf40 1 -- 172.21.15.32:0/2483044197 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556688561680 con 0x5566884be330 2024-08-03T20:57:38.989 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:38.987+0000 7f27e3400640 1 -- 172.21.15.32:0/2483044197 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f27e4003040 con 0x5566884be330 2024-08-03T20:57:38.990 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:38.988+0000 7f27e3400640 1 --2- 172.21.15.32:0/2483044197 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f27cc0759f0 0x7f27cc077f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:38.990 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:38.989+0000 7f27e3400640 1 -- 172.21.15.32:0/2483044197 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(106..106 src has 1..106) ==== 7660+0+0 (secure 0 0 0) 0x7f27e40997d0 con 0x5566884be330 2024-08-03T20:57:38.990 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:38.989+0000 7f27eb000640 1 --2- 172.21.15.32:0/2483044197 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f27cc0759f0 0x7f27cc077f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:38.991 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:38.989+0000 7f27f1e0cf40 1 --2- 172.21.15.32:0/2483044197 >> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] conn(0x5566884a66c0 0x556688438a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:38.991 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:38.989+0000 7f27f1e0cf40 1 -- 172.21.15.32:0/2483044197 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:1 12.f 12:f1f69726:::400.00000000:head [writefull 0~80 in=80b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106) -- 0x556688438fc0 con 0x5566884a66c0 2024-08-03T20:57:38.991 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:38.989+0000 7f27ea600640 1 --2- 172.21.15.32:0/2483044197 >> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] conn(0x5566884a66c0 0x556688438a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:38.991 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:38.990+0000 7f27eb000640 1 --2- 172.21.15.32:0/2483044197 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f27cc0759f0 0x7f27cc077f00 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x5566884c7780 tx=0x7f27dc001730 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:57:38.991 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:38.990+0000 7f27ea600640 1 --2- 172.21.15.32:0/2483044197 >> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] conn(0x5566884a66c0 0x556688438a80 crc :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:57:38.993 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:38.992+0000 7f27ea600640 1 -- 172.21.15.32:0/2483044197 <== osd.6 v2:172.21.15.118:6825/801947386 1 ==== osd_op_reply(1 400.00000000 [writefull 0~80] v106'109 uv109 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f27d804db90 con 0x5566884a66c0 2024-08-03T20:57:38.994 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:38.992+0000 7f27f1e0cf40 1 -- 172.21.15.32:0/2483044197 >> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] conn(0x5566884a66c0 msgr2=0x556688438a80 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:38.994 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:38.992+0000 7f27f1e0cf40 1 --2- 172.21.15.32:0/2483044197 >> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] conn(0x5566884a66c0 0x556688438a80 crc :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:38.994 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:38.992+0000 7f27f1e0cf40 1 -- 172.21.15.32:0/2483044197 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f27cc0759f0 msgr2=0x7f27cc077f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:38.994 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:38.992+0000 7f27f1e0cf40 1 --2- 172.21.15.32:0/2483044197 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f27cc0759f0 0x7f27cc077f00 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x5566884c7780 tx=0x7f27dc001730 comp rx=0 tx=0).stop 2024-08-03T20:57:38.994 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:38.992+0000 7f27f1e0cf40 1 -- 172.21.15.32:0/2483044197 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5566884be330 msgr2=0x556688437020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:38.994 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:38.992+0000 7f27f1e0cf40 1 --2- 172.21.15.32:0/2483044197 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5566884be330 0x556688437020 secure :-1 s=READY pgs=1595 cs=0 l=1 rev1=1 crypto rx=0x7f27e400d720 tx=0x7f27e400db60 comp rx=0 tx=0).stop 2024-08-03T20:57:38.994 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:38.992+0000 7f27eba00640 1 -- 172.21.15.32:0/2483044197 reap_dead start 2024-08-03T20:57:38.994 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:38.992+0000 7f27f1e0cf40 1 -- 172.21.15.32:0/2483044197 shutdown_connections 2024-08-03T20:57:38.994 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:38.992+0000 7f27f1e0cf40 1 -- 172.21.15.32:0/2483044197 >> 172.21.15.32:0/2483044197 conn(0x556688310ac0 msgr2=0x5566883f06c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:57:38.994 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:38.992+0000 7f27f1e0cf40 1 -- 172.21.15.32:0/2483044197 shutdown_connections 2024-08-03T20:57:38.994 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:38.992+0000 7f27f1e0cf40 1 -- 172.21.15.32:0/2483044197 wait complete. 2024-08-03T20:57:39.008 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs joinable true 2024-08-03T20:57:39.149 INFO:tasks.ceph.mds.a.smithi032.stderr: -45> 2024-08-03T20:57:34.395+0000 7f0a10000640 -1 log_channel(cluster) log [ERR] : error decoding table object 'mds0_inotable': Decoder at 'virtual void InoTable::decode_state(ceph::buffer::v15_2_0::list::const_iterator&)' v=2 cannot decode v=100 minimal_decoder=101: Malformed input [buffer:3] 2024-08-03T20:57:39.243 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:39.241+0000 7f34ba400640 1 -- 172.21.15.32:0/2091320264 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f34b40ca990 msgr2=0x7f34b4074680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:39.243 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:39.241+0000 7f34ba400640 1 --2- 172.21.15.32:0/2091320264 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f34b40ca990 0x7f34b4074680 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7f34ac009be0 tx=0x7f34ac02a970 comp rx=0 tx=0).stop 2024-08-03T20:57:39.243 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:39.241+0000 7f34ba400640 1 -- 172.21.15.32:0/2091320264 shutdown_connections 2024-08-03T20:57:39.243 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:39.241+0000 7f34ba400640 1 --2- 172.21.15.32:0/2091320264 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f34b40ca990 0x7f34b4074680 unknown :-1 s=CLOSED pgs=493 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:39.243 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:39.241+0000 7f34ba400640 1 -- 172.21.15.32:0/2091320264 >> 172.21.15.32:0/2091320264 conn(0x7f34b40d0340 msgr2=0x7f34b40d0750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:57:39.243 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:39.241+0000 7f34ba400640 1 -- 172.21.15.32:0/2091320264 shutdown_connections 2024-08-03T20:57:39.243 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:39.241+0000 7f34ba400640 1 -- 172.21.15.32:0/2091320264 wait complete. 2024-08-03T20:57:39.245 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:39.243+0000 7f34ba400640 1 Processor -- start 2024-08-03T20:57:39.245 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:39.243+0000 7f34ba400640 1 -- start start 2024-08-03T20:57:39.247 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:39.245+0000 7f34ba400640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f34b4069300 0x7f34b40b5360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:39.247 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:39.245+0000 7f34ba400640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f34b40ca990 0x7f34b41f4fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:39.247 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:39.245+0000 7f34ba400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f34b40d0e60 0x7f34b41f5510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:39.247 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:39.245+0000 7f34ba400640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f34b4167b80 con 0x7f34b40d0e60 2024-08-03T20:57:39.247 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:39.245+0000 7f34ba400640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f34b4167a00 con 0x7f34b4069300 2024-08-03T20:57:39.247 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:39.245+0000 7f34ba400640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7f34b4167d00 con 0x7f34b40ca990 2024-08-03T20:57:39.247 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:39.245+0000 7f34b3e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f34b40d0e60 0x7f34b41f5510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:39.247 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:39.245+0000 7f34b3400640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f34b4069300 0x7f34b40b5360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:39.248 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:39.245+0000 7f34b3e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f34b40d0e60 0x7f34b41f5510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:34010/0 (socket says 172.21.15.32:34010) 2024-08-03T20:57:39.248 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:39.245+0000 7f34b3e00640 1 -- 172.21.15.32:0/674841239 learned_addr learned my addr 172.21.15.32:0/674841239 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:57:39.248 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:39.245+0000 7f34abe00640 1 --2- 172.21.15.32:0/674841239 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f34b40ca990 0x7f34b41f4fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:39.249 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:39.246+0000 7f34b3400640 1 -- 172.21.15.32:0/674841239 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f34b40ca990 msgr2=0x7f34b41f4fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:39.249 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:39.246+0000 7f34b3400640 1 --2- 172.21.15.32:0/674841239 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f34b40ca990 0x7f34b41f4fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:39.249 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:39.246+0000 7f34b3400640 1 -- 172.21.15.32:0/674841239 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f34b40d0e60 msgr2=0x7f34b41f5510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:39.249 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:39.246+0000 7f34b3400640 1 --2- 172.21.15.32:0/674841239 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f34b40d0e60 0x7f34b41f5510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:39.249 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:39.246+0000 7f34b3400640 1 -- 172.21.15.32:0/674841239 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f34b40bda90 con 0x7f34b4069300 2024-08-03T20:57:39.250 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:39.246+0000 7f34b3400640 1 --2- 172.21.15.32:0/674841239 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f34b4069300 0x7f34b40b5360 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7f34a400c3c0 tx=0x7f34a400c880 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:57:39.250 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:39.246+0000 7f34ab400640 1 -- 172.21.15.32:0/674841239 <== mon.1 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34a4017070 con 0x7f34b4069300 2024-08-03T20:57:39.250 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:39.247+0000 7f34ab400640 1 -- 172.21.15.32:0/674841239 <== mon.1 v2:172.21.15.118:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f34a40053c0 con 0x7f34b4069300 2024-08-03T20:57:39.250 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:39.247+0000 7f34ba400640 1 -- 172.21.15.32:0/674841239 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f34b40c3f10 con 0x7f34b4069300 2024-08-03T20:57:39.250 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:39.247+0000 7f34ab400640 1 -- 172.21.15.32:0/674841239 <== mon.1 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34a4003850 con 0x7f34b4069300 2024-08-03T20:57:39.250 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:39.247+0000 7f34ba400640 1 -- 172.21.15.32:0/674841239 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f34b40c6bd0 con 0x7f34b4069300 2024-08-03T20:57:39.250 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:39.248+0000 7f34ba400640 1 -- 172.21.15.32:0/674841239 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f34b415c770 con 0x7f34b4069300 2024-08-03T20:57:39.250 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:39.249+0000 7f34ab400640 1 -- 172.21.15.32:0/674841239 <== mon.1 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f34a401a070 con 0x7f34b4069300 2024-08-03T20:57:39.251 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:39.250+0000 7f34ab400640 1 --2- 172.21.15.32:0/674841239 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f3490077010 0x7f3490079520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:39.252 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:39.250+0000 7f34ab400640 1 -- 172.21.15.32:0/674841239 <== mon.1 v2:172.21.15.118:3300/0 5 ==== osd_map(106..106 src has 1..106) ==== 7660+0+0 (secure 0 0 0) 0x7f34a40984c0 con 0x7f34b4069300 2024-08-03T20:57:39.252 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:39.251+0000 7f34abe00640 1 --2- 172.21.15.32:0/674841239 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f3490077010 0x7f3490079520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:39.252 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:39.251+0000 7f34abe00640 1 --2- 172.21.15.32:0/674841239 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f3490077010 0x7f3490079520 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7f349c007f20 tx=0x7f349c007eb0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:57:39.256 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:39.255+0000 7f34ab400640 1 -- 172.21.15.32:0/674841239 <== mon.1 v2:172.21.15.118:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7f34a4060830 con 0x7f34b4069300 2024-08-03T20:57:39.265 INFO:tasks.ceph.mds.a.smithi032.stdout:starting mds.a at 2024-08-03T20:57:39.511 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:39.509+0000 7f34ba400640 1 -- 172.21.15.32:0/674841239 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "joinable", "val": "true"} v 0) -- 0x7f34b40cd900 con 0x7f34b4069300 2024-08-03T20:57:40.049 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:40.047+0000 7f34ab400640 1 -- 172.21.15.32:0/674841239 <== mon.1 v2:172.21.15.118:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "joinable", "val": "true"}]=0 cephfs marked joinable; MDS may join as newly active. v221) ==== 162+0+0 (secure 0 0 0) 0x7f34a4069150 con 0x7f34b4069300 2024-08-03T20:57:40.049 INFO:teuthology.orchestra.run.smithi032.stderr:cephfs marked joinable; MDS may join as newly active. 2024-08-03T20:57:40.052 INFO:tasks.ceph.mds.a.smithi032.stderr:2024-08-03T20:57:40.050+0000 7f6f2a800640 -1 log_channel(cluster) log [ERR] : failed to read JournalPointer: -22 ((22) Invalid argument) 2024-08-03T20:57:40.053 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:40.051+0000 7f34ba400640 1 -- 172.21.15.32:0/674841239 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f3490077010 msgr2=0x7f3490079520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:40.054 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:40.051+0000 7f34ba400640 1 --2- 172.21.15.32:0/674841239 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f3490077010 0x7f3490079520 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7f349c007f20 tx=0x7f349c007eb0 comp rx=0 tx=0).stop 2024-08-03T20:57:40.054 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:40.051+0000 7f34ba400640 1 -- 172.21.15.32:0/674841239 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f34b4069300 msgr2=0x7f34b40b5360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:40.054 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:40.051+0000 7f34ba400640 1 --2- 172.21.15.32:0/674841239 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f34b4069300 0x7f34b40b5360 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7f34a400c3c0 tx=0x7f34a400c880 comp rx=0 tx=0).stop 2024-08-03T20:57:40.054 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:40.051+0000 7f34ba400640 1 -- 172.21.15.32:0/674841239 shutdown_connections 2024-08-03T20:57:40.054 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:40.051+0000 7f34ba400640 1 --2- 172.21.15.32:0/674841239 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f3490077010 0x7f3490079520 unknown :-1 s=CLOSED pgs=650 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:40.055 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:40.051+0000 7f34ba400640 1 --2- 172.21.15.32:0/674841239 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f34b40d0e60 0x7f34b41f5510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:40.055 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:40.051+0000 7f34ba400640 1 --2- 172.21.15.32:0/674841239 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f34b40ca990 0x7f34b41f4fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:40.055 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:40.051+0000 7f34ba400640 1 --2- 172.21.15.32:0/674841239 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f34b4069300 0x7f34b40b5360 unknown :-1 s=CLOSED pgs=494 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:40.055 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:40.051+0000 7f34ba400640 1 -- 172.21.15.32:0/674841239 >> 172.21.15.32:0/674841239 conn(0x7f34b40d0340 msgr2=0x7f34b415b5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:57:40.055 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:40.051+0000 7f34ba400640 1 -- 172.21.15.32:0/674841239 shutdown_connections 2024-08-03T20:57:40.055 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:40.051+0000 7f34ba400640 1 -- 172.21.15.32:0/674841239 wait complete. 2024-08-03T20:57:40.056 INFO:tasks.ceph.mon.a.smithi032.stderr:2024-08-03T20:57:40.051+0000 7f4e6d600640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-08-03T20:57:40.056 INFO:tasks.ceph.mon.a.smithi032.stderr:2024-08-03T20:57:40.051+0000 7f4e6d600640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 mds daemon damaged (MDS_DAMAGE) 2024-08-03T20:57:40.084 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-08-03T20:57:40.315 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:40.313+0000 7f7cccc00640 1 -- 172.21.15.32:0/4106180179 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f7cc815fff0 msgr2=0x7f7cc8160400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:40.315 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:40.313+0000 7f7cccc00640 1 --2- 172.21.15.32:0/4106180179 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f7cc815fff0 0x7f7cc8160400 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7f7cc0005fd0 tx=0x7f7cc002aaa0 comp rx=0 tx=0).stop 2024-08-03T20:57:40.315 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:40.313+0000 7f7cccc00640 1 -- 172.21.15.32:0/4106180179 shutdown_connections 2024-08-03T20:57:40.315 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:40.313+0000 7f7cccc00640 1 --2- 172.21.15.32:0/4106180179 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f7cc815fff0 0x7f7cc8160400 unknown :-1 s=CLOSED pgs=497 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:40.315 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:40.313+0000 7f7cccc00640 1 --2- 172.21.15.32:0/4106180179 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f7cc8144f90 0x7f7cc8074570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:40.315 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:40.313+0000 7f7cccc00640 1 -- 172.21.15.32:0/4106180179 >> 172.21.15.32:0/4106180179 conn(0x7f7cc80b1bd0 msgr2=0x7f7cc80b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:57:40.315 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:40.313+0000 7f7cccc00640 1 -- 172.21.15.32:0/4106180179 shutdown_connections 2024-08-03T20:57:40.315 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:40.313+0000 7f7cccc00640 1 -- 172.21.15.32:0/4106180179 wait complete. 2024-08-03T20:57:40.317 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:40.315+0000 7f7cccc00640 1 Processor -- start 2024-08-03T20:57:40.317 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:40.315+0000 7f7cccc00640 1 -- start start 2024-08-03T20:57:40.319 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:40.317+0000 7f7cccc00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f7cc806e930 0x7f7cc81f3f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:40.319 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:40.317+0000 7f7cccc00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7cc8144f90 0x7f7cc815bd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:40.319 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:40.317+0000 7f7cccc00640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f7cc815fff0 0x7f7cc813a8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:40.319 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:40.317+0000 7f7cccc00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f7cc80cf840 con 0x7f7cc8144f90 2024-08-03T20:57:40.319 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:40.317+0000 7f7cccc00640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f7cc8145f30 con 0x7f7cc806e930 2024-08-03T20:57:40.319 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:40.317+0000 7f7cccc00640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7f7cc80cf9c0 con 0x7f7cc815fff0 2024-08-03T20:57:40.320 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:40.317+0000 7f7cc5400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7cc8144f90 0x7f7cc815bd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:40.320 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:40.317+0000 7f7cc5400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7cc8144f90 0x7f7cc815bd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:43052/0 (socket says 172.21.15.32:43052) 2024-08-03T20:57:40.320 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:40.317+0000 7f7cc5400640 1 -- 172.21.15.32:0/1884069545 learned_addr learned my addr 172.21.15.32:0/1884069545 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:57:40.320 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:40.317+0000 7f7cc5e00640 1 --2- 172.21.15.32:0/1884069545 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f7cc806e930 0x7f7cc81f3f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:40.320 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:40.317+0000 7f7cc6800640 1 --2- 172.21.15.32:0/1884069545 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f7cc815fff0 0x7f7cc813a8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:40.320 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:40.318+0000 7f7cc5400640 1 -- 172.21.15.32:0/1884069545 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f7cc815fff0 msgr2=0x7f7cc813a8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:40.320 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:40.318+0000 7f7cc5400640 1 --2- 172.21.15.32:0/1884069545 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f7cc815fff0 0x7f7cc813a8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:40.320 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:40.318+0000 7f7cc5400640 1 -- 172.21.15.32:0/1884069545 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f7cc806e930 msgr2=0x7f7cc81f3f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:40.320 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:40.318+0000 7f7cc5400640 1 --2- 172.21.15.32:0/1884069545 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f7cc806e930 0x7f7cc81f3f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:40.320 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:40.318+0000 7f7cc5400640 1 -- 172.21.15.32:0/1884069545 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7cc815c240 con 0x7f7cc8144f90 2024-08-03T20:57:40.322 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:40.318+0000 7f7cc5400640 1 --2- 172.21.15.32:0/1884069545 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7cc8144f90 0x7f7cc815bd00 secure :-1 s=READY pgs=1598 cs=0 l=1 rev1=1 crypto rx=0x7f7cb400e7c0 tx=0x7f7cb400ec80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:57:40.322 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:40.318+0000 7f7cbe000640 1 -- 172.21.15.32:0/1884069545 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7cb400ccc0 con 0x7f7cc8144f90 2024-08-03T20:57:40.322 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:40.318+0000 7f7cbe000640 1 -- 172.21.15.32:0/1884069545 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f7cb4010430 con 0x7f7cc8144f90 2024-08-03T20:57:40.322 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:40.318+0000 7f7cbe000640 1 -- 172.21.15.32:0/1884069545 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7cb4010860 con 0x7f7cc8144f90 2024-08-03T20:57:40.322 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:40.318+0000 7f7cccc00640 1 -- 172.21.15.32:0/1884069545 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7cc813ebf0 con 0x7f7cc8144f90 2024-08-03T20:57:40.322 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:40.319+0000 7f7cccc00640 1 -- 172.21.15.32:0/1884069545 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7cc813a0c0 con 0x7f7cc8144f90 2024-08-03T20:57:40.322 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:40.320+0000 7f7cccc00640 1 -- 172.21.15.32:0/1884069545 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7cc8069090 con 0x7f7cc8144f90 2024-08-03T20:57:40.323 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:40.321+0000 7f7cbe000640 1 -- 172.21.15.32:0/1884069545 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f7cb4010a00 con 0x7f7cc8144f90 2024-08-03T20:57:40.323 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:40.322+0000 7f7cbe000640 1 --2- 172.21.15.32:0/1884069545 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f7ca8077010 0x7f7ca8079520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:40.323 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:40.322+0000 7f7cbe000640 1 -- 172.21.15.32:0/1884069545 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(107..107 src has 1..107) ==== 7746+0+0 (secure 0 0 0) 0x7f7cb4099a60 con 0x7f7cc8144f90 2024-08-03T20:57:40.324 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:40.322+0000 7f7cc5e00640 1 --2- 172.21.15.32:0/1884069545 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f7ca8077010 0x7f7ca8079520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:40.324 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:40.323+0000 7f7cc5e00640 1 --2- 172.21.15.32:0/1884069545 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f7ca8077010 0x7f7ca8079520 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7f7cb8001720 tx=0x7f7cb8009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:57:40.327 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:40.326+0000 7f7cbe000640 1 -- 172.21.15.32:0/1884069545 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7f7cc8069090 con 0x7f7cc8144f90 2024-08-03T20:57:40.572 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:40.571+0000 7f7cccc00640 1 -- 172.21.15.32:0/1884069545 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f7cc806a430 con 0x7f7cc8144f90 2024-08-03T20:57:40.573 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:40.571+0000 7f7cbe000640 1 -- 172.21.15.32:0/1884069545 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 222 v222) ==== 95+0+4391 (secure 0 0 0) 0x7f7cc806a430 con 0x7f7cc8144f90 2024-08-03T20:57:40.573 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-03T20:57:40.573 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":222,"btime":"2024-08-03T20:57:40:053215+0000","default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6439,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6835/531080160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6834","nonce":531080160},{"type":"v1","addr":"172.21.15.118:6835","nonce":531080160}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":84},{"gid":6454,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6837/2796882319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6836","nonce":2796882319},{"type":"v1","addr":"172.21.15.118:6837","nonce":2796882319}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":86},{"gid":6813,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6834/2686933890","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":2686933890},{"type":"v1","addr":"172.21.15.32:6834","nonce":2686933890}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":85}],"filesystems":[{"mdsmap":{"epoch":222,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-08-03T20:54:26.950518+0000","modified":"2024-08-03T20:57:40.053118+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":107,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":6}]} 2024-08-03T20:57:40.573 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 222 2024-08-03T20:57:40.575 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:40.573+0000 7f7cccc00640 1 -- 172.21.15.32:0/1884069545 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f7ca8077010 msgr2=0x7f7ca8079520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:40.576 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:40.573+0000 7f7cccc00640 1 --2- 172.21.15.32:0/1884069545 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f7ca8077010 0x7f7ca8079520 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7f7cb8001720 tx=0x7f7cb8009040 comp rx=0 tx=0).stop 2024-08-03T20:57:40.576 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:40.573+0000 7f7cccc00640 1 -- 172.21.15.32:0/1884069545 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7cc8144f90 msgr2=0x7f7cc815bd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:40.576 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:40.573+0000 7f7cccc00640 1 --2- 172.21.15.32:0/1884069545 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7cc8144f90 0x7f7cc815bd00 secure :-1 s=READY pgs=1598 cs=0 l=1 rev1=1 crypto rx=0x7f7cb400e7c0 tx=0x7f7cb400ec80 comp rx=0 tx=0).stop 2024-08-03T20:57:40.576 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:40.574+0000 7f7cccc00640 1 -- 172.21.15.32:0/1884069545 shutdown_connections 2024-08-03T20:57:40.576 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:40.574+0000 7f7cccc00640 1 --2- 172.21.15.32:0/1884069545 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f7ca8077010 0x7f7ca8079520 unknown :-1 s=CLOSED pgs=652 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:40.576 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:40.574+0000 7f7cccc00640 1 --2- 172.21.15.32:0/1884069545 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f7cc815fff0 0x7f7cc813a8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:40.576 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:40.574+0000 7f7cccc00640 1 --2- 172.21.15.32:0/1884069545 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7cc8144f90 0x7f7cc815bd00 unknown :-1 s=CLOSED pgs=1598 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:40.576 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:40.574+0000 7f7cccc00640 1 --2- 172.21.15.32:0/1884069545 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f7cc806e930 0x7f7cc81f3f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:40.576 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:40.574+0000 7f7cccc00640 1 -- 172.21.15.32:0/1884069545 >> 172.21.15.32:0/1884069545 conn(0x7f7cc80b1bd0 msgr2=0x7f7cc81434f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:57:40.576 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:40.574+0000 7f7cccc00640 1 -- 172.21.15.32:0/1884069545 shutdown_connections 2024-08-03T20:57:40.576 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:40.574+0000 7f7cccc00640 1 -- 172.21.15.32:0/1884069545 wait complete. 2024-08-03T20:57:40.598 DEBUG:tasks.ceph_test_case:wait_until_true: success in 0s and 0 retries 2024-08-03T20:57:40.598 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-08-03T20:57:40.827 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:40.825+0000 7f4f05200640 1 -- 172.21.15.32:0/3920757981 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4f000af020 msgr2=0x7f4f000742b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:40.827 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:40.825+0000 7f4f05200640 1 --2- 172.21.15.32:0/3920757981 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4f000af020 0x7f4f000742b0 secure :-1 s=READY pgs=1599 cs=0 l=1 rev1=1 crypto rx=0x7f4ef0005fd0 tx=0x7f4ef002a4b0 comp rx=0 tx=0).stop 2024-08-03T20:57:40.827 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:40.825+0000 7f4f05200640 1 -- 172.21.15.32:0/3920757981 shutdown_connections 2024-08-03T20:57:40.827 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:40.825+0000 7f4f05200640 1 --2- 172.21.15.32:0/3920757981 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f4f000acd80 0x7f4f000ad190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:40.827 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:40.825+0000 7f4f05200640 1 --2- 172.21.15.32:0/3920757981 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4f000af020 0x7f4f000742b0 unknown :-1 s=CLOSED pgs=1599 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:40.827 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:40.825+0000 7f4f05200640 1 -- 172.21.15.32:0/3920757981 >> 172.21.15.32:0/3920757981 conn(0x7f4f000b2120 msgr2=0x7f4f000cfd60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:57:40.827 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:40.825+0000 7f4f05200640 1 -- 172.21.15.32:0/3920757981 shutdown_connections 2024-08-03T20:57:40.827 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:40.825+0000 7f4f05200640 1 -- 172.21.15.32:0/3920757981 wait complete. 2024-08-03T20:57:40.828 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:40.827+0000 7f4f05200640 1 Processor -- start 2024-08-03T20:57:40.828 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:40.827+0000 7f4f05200640 1 -- start start 2024-08-03T20:57:40.830 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:40.828+0000 7f4f05200640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f4f0006f440 0x7f4f000c4ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:40.831 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:40.828+0000 7f4f05200640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f4f000acd80 0x7f4f00150040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:40.831 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:40.828+0000 7f4f05200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4f000af020 0x7f4f0015ea80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:40.831 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:40.829+0000 7f4f05200640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f4f00160260 con 0x7f4f000af020 2024-08-03T20:57:40.831 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:40.829+0000 7f4f05200640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f4f001600e0 con 0x7f4f0006f440 2024-08-03T20:57:40.831 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:40.829+0000 7f4f05200640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7f4f001603e0 con 0x7f4f000acd80 2024-08-03T20:57:40.831 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:40.829+0000 7f4effe00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4f000af020 0x7f4f0015ea80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:40.831 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:40.829+0000 7f4effe00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4f000af020 0x7f4f0015ea80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:43080/0 (socket says 172.21.15.32:43080) 2024-08-03T20:57:40.831 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:40.829+0000 7f4effe00640 1 -- 172.21.15.32:0/414498091 learned_addr learned my addr 172.21.15.32:0/414498091 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:57:40.831 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:40.829+0000 7f4ef6a00640 1 --2- 172.21.15.32:0/414498091 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f4f000acd80 0x7f4f00150040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:40.831 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:40.829+0000 7f4eff400640 1 --2- 172.21.15.32:0/414498091 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f4f0006f440 0x7f4f000c4ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:40.831 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:40.829+0000 7f4effe00640 1 -- 172.21.15.32:0/414498091 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f4f000acd80 msgr2=0x7f4f00150040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:40.831 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:40.829+0000 7f4effe00640 1 --2- 172.21.15.32:0/414498091 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f4f000acd80 0x7f4f00150040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:40.832 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:40.829+0000 7f4effe00640 1 -- 172.21.15.32:0/414498091 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f4f0006f440 msgr2=0x7f4f000c4ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:40.832 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:40.829+0000 7f4effe00640 1 --2- 172.21.15.32:0/414498091 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f4f0006f440 0x7f4f000c4ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:40.832 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:40.829+0000 7f4effe00640 1 -- 172.21.15.32:0/414498091 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4f00154f00 con 0x7f4f000af020 2024-08-03T20:57:40.832 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:40.829+0000 7f4ef6a00640 1 --2- 172.21.15.32:0/414498091 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f4f000acd80 0x7f4f00150040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-03T20:57:40.833 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:40.830+0000 7f4effe00640 1 --2- 172.21.15.32:0/414498091 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4f000af020 0x7f4f0015ea80 secure :-1 s=READY pgs=1600 cs=0 l=1 rev1=1 crypto rx=0x7f4ef800b960 tx=0x7f4ef800be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:57:40.833 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:40.830+0000 7f4ef7e00640 1 -- 172.21.15.32:0/414498091 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ef8019070 con 0x7f4f000af020 2024-08-03T20:57:40.833 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:40.830+0000 7f4ef7e00640 1 -- 172.21.15.32:0/414498091 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f4ef800cc10 con 0x7f4f000af020 2024-08-03T20:57:40.833 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:40.830+0000 7f4ef7e00640 1 -- 172.21.15.32:0/414498091 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ef8014440 con 0x7f4f000af020 2024-08-03T20:57:40.833 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:40.830+0000 7f4f05200640 1 -- 172.21.15.32:0/414498091 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4f001542f0 con 0x7f4f000af020 2024-08-03T20:57:40.833 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:40.830+0000 7f4f05200640 1 -- 172.21.15.32:0/414498091 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4f001548b0 con 0x7f4f000af020 2024-08-03T20:57:40.833 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:40.831+0000 7f4f05200640 1 -- 172.21.15.32:0/414498091 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4f000ad190 con 0x7f4f000af020 2024-08-03T20:57:40.833 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:40.832+0000 7f4ef7e00640 1 -- 172.21.15.32:0/414498091 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f4ef801c070 con 0x7f4f000af020 2024-08-03T20:57:40.834 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:40.833+0000 7f4ef7e00640 1 --2- 172.21.15.32:0/414498091 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f4edc077010 0x7f4edc079520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:40.835 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:40.833+0000 7f4ef7e00640 1 -- 172.21.15.32:0/414498091 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(107..107 src has 1..107) ==== 7746+0+0 (secure 0 0 0) 0x7f4ef8098e40 con 0x7f4f000af020 2024-08-03T20:57:40.835 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:40.834+0000 7f4eff400640 1 --2- 172.21.15.32:0/414498091 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f4edc077010 0x7f4edc079520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:40.836 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:40.834+0000 7f4eff400640 1 --2- 172.21.15.32:0/414498091 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f4edc077010 0x7f4edc079520 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7f4ef0033000 tx=0x7f4ef0004730 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:57:40.839 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:40.838+0000 7f4ef7e00640 1 -- 172.21.15.32:0/414498091 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7f4ef8061150 con 0x7f4f000af020 2024-08-03T20:57:41.080 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:41.079+0000 7f4f05200640 1 -- 172.21.15.32:0/414498091 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f4f000b3290 con 0x7f4f000af020 2024-08-03T20:57:41.081 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:41.079+0000 7f4ef7e00640 1 -- 172.21.15.32:0/414498091 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 222 v222) ==== 95+0+4391 (secure 0 0 0) 0x7f4ef80656a0 con 0x7f4f000af020 2024-08-03T20:57:41.081 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-03T20:57:41.081 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":222,"btime":"2024-08-03T20:57:40:053215+0000","default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6439,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6835/531080160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6834","nonce":531080160},{"type":"v1","addr":"172.21.15.118:6835","nonce":531080160}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":84},{"gid":6454,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6837/2796882319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6836","nonce":2796882319},{"type":"v1","addr":"172.21.15.118:6837","nonce":2796882319}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":86},{"gid":6813,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6834/2686933890","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":2686933890},{"type":"v1","addr":"172.21.15.32:6834","nonce":2686933890}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":85}],"filesystems":[{"mdsmap":{"epoch":222,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-08-03T20:54:26.950518+0000","modified":"2024-08-03T20:57:40.053118+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":107,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":6}]} 2024-08-03T20:57:41.081 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 222 2024-08-03T20:57:41.083 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:41.081+0000 7f4f05200640 1 -- 172.21.15.32:0/414498091 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f4edc077010 msgr2=0x7f4edc079520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:41.083 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:41.081+0000 7f4f05200640 1 --2- 172.21.15.32:0/414498091 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f4edc077010 0x7f4edc079520 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7f4ef0033000 tx=0x7f4ef0004730 comp rx=0 tx=0).stop 2024-08-03T20:57:41.083 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:41.081+0000 7f4f05200640 1 -- 172.21.15.32:0/414498091 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4f000af020 msgr2=0x7f4f0015ea80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:41.083 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:41.081+0000 7f4f05200640 1 --2- 172.21.15.32:0/414498091 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4f000af020 0x7f4f0015ea80 secure :-1 s=READY pgs=1600 cs=0 l=1 rev1=1 crypto rx=0x7f4ef800b960 tx=0x7f4ef800be20 comp rx=0 tx=0).stop 2024-08-03T20:57:41.084 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:41.082+0000 7f4f05200640 1 -- 172.21.15.32:0/414498091 shutdown_connections 2024-08-03T20:57:41.084 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:41.082+0000 7f4f05200640 1 --2- 172.21.15.32:0/414498091 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f4edc077010 0x7f4edc079520 unknown :-1 s=CLOSED pgs=653 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:41.084 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:41.082+0000 7f4f05200640 1 --2- 172.21.15.32:0/414498091 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4f000af020 0x7f4f0015ea80 unknown :-1 s=CLOSED pgs=1600 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:41.084 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:41.082+0000 7f4f05200640 1 --2- 172.21.15.32:0/414498091 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f4f000acd80 0x7f4f00150040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:41.084 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:41.082+0000 7f4f05200640 1 --2- 172.21.15.32:0/414498091 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f4f0006f440 0x7f4f000c4ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:41.084 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:41.082+0000 7f4f05200640 1 -- 172.21.15.32:0/414498091 >> 172.21.15.32:0/414498091 conn(0x7f4f000b2120 msgr2=0x7f4f000cc7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:57:41.084 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:41.082+0000 7f4f05200640 1 -- 172.21.15.32:0/414498091 shutdown_connections 2024-08-03T20:57:41.084 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:41.082+0000 7f4f05200640 1 -- 172.21.15.32:0/414498091 wait complete. 2024-08-03T20:57:41.109 DEBUG:tasks.ceph_test_case:wait_until_true: success in 0s and 0 retries 2024-08-03T20:57:41.109 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-08-03T20:57:41.339 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:41.337+0000 7fd8c5800640 1 -- 172.21.15.32:0/4033941962 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd8c00b53a0 msgr2=0x7fd8c00b4d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:41.339 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:41.337+0000 7fd8c5800640 1 --2- 172.21.15.32:0/4033941962 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd8c00b53a0 0x7fd8c00b4d70 secure :-1 s=READY pgs=1601 cs=0 l=1 rev1=1 crypto rx=0x7fd8c006e520 tx=0x7fd8b002a4b0 comp rx=0 tx=0).stop 2024-08-03T20:57:41.339 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:41.337+0000 7fd8c5800640 1 -- 172.21.15.32:0/4033941962 shutdown_connections 2024-08-03T20:57:41.339 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:41.337+0000 7fd8c5800640 1 --2- 172.21.15.32:0/4033941962 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fd8c006d9f0 0x7fd8c006de00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:41.339 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:41.337+0000 7fd8c5800640 1 --2- 172.21.15.32:0/4033941962 >> v2:172.21.15.118:3300/0 conn(0x7fd8c015e5c0 0x7fd8c00b3480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:41.339 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:41.337+0000 7fd8c5800640 1 --2- 172.21.15.32:0/4033941962 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd8c00b53a0 0x7fd8c00b4d70 unknown :-1 s=CLOSED pgs=1601 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:41.339 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:41.337+0000 7fd8c5800640 1 -- 172.21.15.32:0/4033941962 >> 172.21.15.32:0/4033941962 conn(0x7fd8c0072e70 msgr2=0x7fd8c0073280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:57:41.339 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:41.337+0000 7fd8c5800640 1 -- 172.21.15.32:0/4033941962 shutdown_connections 2024-08-03T20:57:41.339 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:41.337+0000 7fd8c5800640 1 -- 172.21.15.32:0/4033941962 wait complete. 2024-08-03T20:57:41.340 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:41.339+0000 7fd8c5800640 1 Processor -- start 2024-08-03T20:57:41.341 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:41.339+0000 7fd8c5800640 1 -- start start 2024-08-03T20:57:41.343 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:41.341+0000 7fd8c5800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd8c006d9f0 0x7fd8c0153a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:41.343 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:41.341+0000 7fd8c5800640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fd8c00b53a0 0x7fd8c01f8ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:41.343 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:41.341+0000 7fd8c5800640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fd8c015e5c0 0x7fd8c01f9020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:41.343 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:41.341+0000 7fd8c5800640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fd8c01623e0 con 0x7fd8c006d9f0 2024-08-03T20:57:41.343 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:41.341+0000 7fd8c5800640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7fd8c0162260 con 0x7fd8c015e5c0 2024-08-03T20:57:41.343 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:41.341+0000 7fd8c5800640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7fd8c0162560 con 0x7fd8c00b53a0 2024-08-03T20:57:41.343 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:41.341+0000 7fd8bfe00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd8c006d9f0 0x7fd8c0153a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:41.343 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:41.341+0000 7fd8bfe00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd8c006d9f0 0x7fd8c0153a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:43104/0 (socket says 172.21.15.32:43104) 2024-08-03T20:57:41.343 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:41.341+0000 7fd8bfe00640 1 -- 172.21.15.32:0/969567208 learned_addr learned my addr 172.21.15.32:0/969567208 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:57:41.343 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:41.341+0000 7fd8bfe00640 1 -- 172.21.15.32:0/969567208 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fd8c00b53a0 msgr2=0x7fd8c01f8ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:41.343 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:41.341+0000 7fd8bfe00640 1 --2- 172.21.15.32:0/969567208 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fd8c00b53a0 0x7fd8c01f8ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:41.344 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:41.341+0000 7fd8bfe00640 1 -- 172.21.15.32:0/969567208 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fd8c015e5c0 msgr2=0x7fd8c01f9020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:41.344 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:41.341+0000 7fd8bfe00640 1 --2- 172.21.15.32:0/969567208 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fd8c015e5c0 0x7fd8c01f9020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:41.344 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:41.341+0000 7fd8bfe00640 1 -- 172.21.15.32:0/969567208 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd8c01616c0 con 0x7fd8c006d9f0 2024-08-03T20:57:41.345 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:41.342+0000 7fd8bfe00640 1 --2- 172.21.15.32:0/969567208 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd8c006d9f0 0x7fd8c0153a30 secure :-1 s=READY pgs=1602 cs=0 l=1 rev1=1 crypto rx=0x7fd8b0034040 tx=0x7fd8b0003910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:57:41.345 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:41.342+0000 7fd8bcc00640 1 -- 172.21.15.32:0/969567208 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8b003e030 con 0x7fd8c006d9f0 2024-08-03T20:57:41.345 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:41.342+0000 7fd8bcc00640 1 -- 172.21.15.32:0/969567208 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fd8b0003e00 con 0x7fd8c006d9f0 2024-08-03T20:57:41.345 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:41.342+0000 7fd8bcc00640 1 -- 172.21.15.32:0/969567208 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8b0035040 con 0x7fd8c006d9f0 2024-08-03T20:57:41.345 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:41.342+0000 7fd8c5800640 1 -- 172.21.15.32:0/969567208 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd8c00cf2f0 con 0x7fd8c006d9f0 2024-08-03T20:57:41.345 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:41.342+0000 7fd8c5800640 1 -- 172.21.15.32:0/969567208 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd8c0156e10 con 0x7fd8c006d9f0 2024-08-03T20:57:41.345 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:41.343+0000 7fd8c5800640 1 -- 172.21.15.32:0/969567208 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd8c0050ba0 con 0x7fd8c006d9f0 2024-08-03T20:57:41.346 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:41.344+0000 7fd8bcc00640 1 -- 172.21.15.32:0/969567208 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7fd8b0041030 con 0x7fd8c006d9f0 2024-08-03T20:57:41.347 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:41.345+0000 7fd8bcc00640 1 --2- 172.21.15.32:0/969567208 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fd89c077010 0x7fd89c079520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:41.347 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:41.345+0000 7fd8bcc00640 1 -- 172.21.15.32:0/969567208 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(107..107 src has 1..107) ==== 7746+0+0 (secure 0 0 0) 0x7fd8b00b9840 con 0x7fd8c006d9f0 2024-08-03T20:57:41.347 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:41.346+0000 7fd8b7e00640 1 --2- 172.21.15.32:0/969567208 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fd89c077010 0x7fd89c079520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:41.348 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:41.347+0000 7fd8b7e00640 1 --2- 172.21.15.32:0/969567208 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fd89c077010 0x7fd89c079520 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7fd8c00cffa0 tx=0x7fd8a8005ce0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:57:41.351 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:41.350+0000 7fd8bcc00640 1 -- 172.21.15.32:0/969567208 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7fd8c0050ba0 con 0x7fd8c006d9f0 2024-08-03T20:57:41.596 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:41.595+0000 7fd8c5800640 1 -- 172.21.15.32:0/969567208 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd8c00b4d70 con 0x7fd8c006d9f0 2024-08-03T20:57:41.597 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:41.596+0000 7fd8bcc00640 1 -- 172.21.15.32:0/969567208 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 222 v222) ==== 95+0+4391 (secure 0 0 0) 0x7fd8b00822c0 con 0x7fd8c006d9f0 2024-08-03T20:57:41.597 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-03T20:57:41.597 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":222,"btime":"2024-08-03T20:57:40:053215+0000","default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6439,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6835/531080160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6834","nonce":531080160},{"type":"v1","addr":"172.21.15.118:6835","nonce":531080160}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":84},{"gid":6454,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6837/2796882319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6836","nonce":2796882319},{"type":"v1","addr":"172.21.15.118:6837","nonce":2796882319}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":86},{"gid":6813,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6834/2686933890","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":2686933890},{"type":"v1","addr":"172.21.15.32:6834","nonce":2686933890}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":85}],"filesystems":[{"mdsmap":{"epoch":222,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-08-03T20:54:26.950518+0000","modified":"2024-08-03T20:57:40.053118+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":107,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":6}]} 2024-08-03T20:57:41.598 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 222 2024-08-03T20:57:41.600 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:41.598+0000 7fd8c5800640 1 -- 172.21.15.32:0/969567208 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fd89c077010 msgr2=0x7fd89c079520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:41.600 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:41.598+0000 7fd8c5800640 1 --2- 172.21.15.32:0/969567208 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fd89c077010 0x7fd89c079520 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7fd8c00cffa0 tx=0x7fd8a8005ce0 comp rx=0 tx=0).stop 2024-08-03T20:57:41.600 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:41.598+0000 7fd8c5800640 1 -- 172.21.15.32:0/969567208 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd8c006d9f0 msgr2=0x7fd8c0153a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:41.600 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:41.598+0000 7fd8c5800640 1 --2- 172.21.15.32:0/969567208 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd8c006d9f0 0x7fd8c0153a30 secure :-1 s=READY pgs=1602 cs=0 l=1 rev1=1 crypto rx=0x7fd8b0034040 tx=0x7fd8b0003910 comp rx=0 tx=0).stop 2024-08-03T20:57:41.600 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:41.598+0000 7fd8c5800640 1 -- 172.21.15.32:0/969567208 shutdown_connections 2024-08-03T20:57:41.600 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:41.598+0000 7fd8c5800640 1 --2- 172.21.15.32:0/969567208 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fd89c077010 0x7fd89c079520 unknown :-1 s=CLOSED pgs=654 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:41.600 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:41.598+0000 7fd8c5800640 1 --2- 172.21.15.32:0/969567208 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fd8c015e5c0 0x7fd8c01f9020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:41.600 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:41.598+0000 7fd8c5800640 1 --2- 172.21.15.32:0/969567208 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fd8c00b53a0 0x7fd8c01f8ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:41.600 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:41.598+0000 7fd8c5800640 1 --2- 172.21.15.32:0/969567208 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd8c006d9f0 0x7fd8c0153a30 unknown :-1 s=CLOSED pgs=1602 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:41.600 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:41.598+0000 7fd8c5800640 1 -- 172.21.15.32:0/969567208 >> 172.21.15.32:0/969567208 conn(0x7fd8c0072e70 msgr2=0x7fd8c015a300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:57:41.601 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:41.598+0000 7fd8c5800640 1 -- 172.21.15.32:0/969567208 shutdown_connections 2024-08-03T20:57:41.601 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:41.598+0000 7fd8c5800640 1 -- 172.21.15.32:0/969567208 wait complete. 2024-08-03T20:57:41.622 INFO:tasks.cephfs.test_damage:Result: Mutation 'Corrupt 400.00000000' led to DAMAGED state 2024-08-03T20:57:41.622 INFO:tasks.cephfs.test_damage:Applying mutation 'Corrupt 100.00000000.inode' 2024-08-03T20:57:41.622 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-08-03T20:57:41.622 DEBUG:teuthology.orchestra.run.smithi032:> dd if=/proc/self/mounts of=/dev/stdout 2024-08-03T20:57:41.658 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053453,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6499208k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 none /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249604k,nr_inodes=812401,mode=700,uid=1000,gid=1316 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6499208k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-08-03T20:57:41.658 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi032.front.sepia.ceph.com 2024-08-03T20:57:41.658 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-08-03T20:57:41.658 DEBUG:teuthology.orchestra.run.smithi032:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-08-03T20:57:41.732 INFO:teuthology.orchestra.run.smithi032.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.0': No such file or directory 2024-08-03T20:57:41.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-03T20:57:41.734 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-08-03T20:57:42.005 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:42.003+0000 7fdea0000640 1 -- 172.21.15.32:0/2828330048 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fde80004800 msgr2=0x7fde80004c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:42.005 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:42.003+0000 7fdea0000640 1 --2- 172.21.15.32:0/2828330048 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fde80004800 0x7fde80004c10 secure :-1 s=READY pgs=1604 cs=0 l=1 rev1=1 crypto rx=0x7fde980cc4d0 tx=0x7fde80005830 comp rx=0 tx=0).stop 2024-08-03T20:57:42.005 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:42.003+0000 7fdea0000640 1 -- 172.21.15.32:0/2828330048 shutdown_connections 2024-08-03T20:57:42.005 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:42.003+0000 7fdea0000640 1 --2- 172.21.15.32:0/2828330048 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fde80004800 0x7fde80004c10 unknown :-1 s=CLOSED pgs=1604 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:42.005 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:42.003+0000 7fdea0000640 1 -- 172.21.15.32:0/2828330048 >> 172.21.15.32:0/2828330048 conn(0x7fde980b1bd0 msgr2=0x7fde980b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:57:42.005 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:42.003+0000 7fdea0000640 1 -- 172.21.15.32:0/2828330048 shutdown_connections 2024-08-03T20:57:42.005 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:42.003+0000 7fdea0000640 1 -- 172.21.15.32:0/2828330048 wait complete. 2024-08-03T20:57:42.007 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:42.005+0000 7fdea0000640 1 Processor -- start 2024-08-03T20:57:42.007 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:42.005+0000 7fdea0000640 1 -- start start 2024-08-03T20:57:42.009 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:42.007+0000 7fdea0000640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fde80004800 0x7fde981f5060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:42.009 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:42.007+0000 7fdea0000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fde9815e630 0x7fde9815d530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:42.009 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:42.007+0000 7fdea0000640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fde9815ea40 0x7fde9815da70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:42.009 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:42.007+0000 7fdea0000640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fde98145110 con 0x7fde9815e630 2024-08-03T20:57:42.009 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:42.007+0000 7fdea0000640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7fde98144f90 con 0x7fde9815ea40 2024-08-03T20:57:42.009 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:42.007+0000 7fdea0000640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7fde98145290 con 0x7fde80004800 2024-08-03T20:57:42.010 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:42.007+0000 7fde9cc00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fde9815e630 0x7fde9815d530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:42.010 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:42.007+0000 7fde9cc00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fde9815e630 0x7fde9815d530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:43124/0 (socket says 172.21.15.32:43124) 2024-08-03T20:57:42.010 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:42.007+0000 7fde9cc00640 1 -- 172.21.15.32:0/1652662873 learned_addr learned my addr 172.21.15.32:0/1652662873 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:57:42.010 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:42.007+0000 7fde9d600640 1 --2- 172.21.15.32:0/1652662873 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fde80004800 0x7fde981f5060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:42.010 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:42.007+0000 7fde9e000640 1 --2- 172.21.15.32:0/1652662873 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fde9815ea40 0x7fde9815da70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:42.010 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:42.008+0000 7fde9cc00640 1 -- 172.21.15.32:0/1652662873 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fde80004800 msgr2=0x7fde981f5060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:42.010 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:42.008+0000 7fde9cc00640 1 --2- 172.21.15.32:0/1652662873 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fde80004800 0x7fde981f5060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:42.010 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:42.008+0000 7fde9cc00640 1 -- 172.21.15.32:0/1652662873 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fde9815ea40 msgr2=0x7fde9815da70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:42.010 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:42.008+0000 7fde9cc00640 1 --2- 172.21.15.32:0/1652662873 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fde9815ea40 0x7fde9815da70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:42.010 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:42.008+0000 7fde9cc00640 1 -- 172.21.15.32:0/1652662873 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fde9815ca70 con 0x7fde9815e630 2024-08-03T20:57:42.010 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:42.008+0000 7fde9d600640 1 --2- 172.21.15.32:0/1652662873 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fde80004800 0x7fde981f5060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-03T20:57:42.010 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:42.008+0000 7fde9e000640 1 --2- 172.21.15.32:0/1652662873 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fde9815ea40 0x7fde9815da70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-03T20:57:42.010 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:42.008+0000 7fde9cc00640 1 --2- 172.21.15.32:0/1652662873 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fde9815e630 0x7fde9815d530 secure :-1 s=READY pgs=1605 cs=0 l=1 rev1=1 crypto rx=0x7fde8400c320 tx=0x7fde840064a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:57:42.011 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:42.008+0000 7fde95600640 1 -- 172.21.15.32:0/1652662873 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde84003a00 con 0x7fde9815e630 2024-08-03T20:57:42.011 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:42.008+0000 7fde95600640 1 -- 172.21.15.32:0/1652662873 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fde84003c90 con 0x7fde9815e630 2024-08-03T20:57:42.011 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:42.008+0000 7fde95600640 1 -- 172.21.15.32:0/1652662873 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde8402e6b0 con 0x7fde9815e630 2024-08-03T20:57:42.011 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:42.008+0000 7fdea0000640 1 -- 172.21.15.32:0/1652662873 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fde981f5b90 con 0x7fde9815e630 2024-08-03T20:57:42.012 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:42.008+0000 7fdea0000640 1 -- 172.21.15.32:0/1652662873 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fde9815cd30 con 0x7fde9815e630 2024-08-03T20:57:42.012 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:42.009+0000 7fdea0000640 1 -- 172.21.15.32:0/1652662873 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fde98074350 con 0x7fde9815e630 2024-08-03T20:57:42.012 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:42.011+0000 7fde95600640 1 -- 172.21.15.32:0/1652662873 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7fde84035020 con 0x7fde9815e630 2024-08-03T20:57:42.013 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:42.012+0000 7fde95600640 1 --2- 172.21.15.32:0/1652662873 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fde78077040 0x7fde78079550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:42.013 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:42.012+0000 7fde95600640 1 -- 172.21.15.32:0/1652662873 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(107..107 src has 1..107) ==== 7746+0+0 (secure 0 0 0) 0x7fde840b9670 con 0x7fde9815e630 2024-08-03T20:57:42.013 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:42.012+0000 7fde9d600640 1 --2- 172.21.15.32:0/1652662873 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fde78077040 0x7fde78079550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:42.014 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:42.013+0000 7fde9d600640 1 --2- 172.21.15.32:0/1652662873 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fde78077040 0x7fde78079550 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7fde8c009cd0 tx=0x7fde8c005f90 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:57:42.017 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:42.016+0000 7fde95600640 1 -- 172.21.15.32:0/1652662873 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7fde84047030 con 0x7fde9815e630 2024-08-03T20:57:42.257 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:42.255+0000 7fdea0000640 1 -- 172.21.15.32:0/1652662873 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7fde981f5920 con 0x7fde9815e630 2024-08-03T20:57:42.423 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:42.422+0000 7fde95600640 1 -- 172.21.15.32:0/1652662873 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v223) ==== 161+0+0 (secure 0 0 0) 0x7fde8403e070 con 0x7fde9815e630 2024-08-03T20:57:42.424 INFO:teuthology.orchestra.run.smithi032.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-08-03T20:57:42.426 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:42.424+0000 7fdea0000640 1 -- 172.21.15.32:0/1652662873 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fde78077040 msgr2=0x7fde78079550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:42.426 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:42.424+0000 7fdea0000640 1 --2- 172.21.15.32:0/1652662873 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fde78077040 0x7fde78079550 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7fde8c009cd0 tx=0x7fde8c005f90 comp rx=0 tx=0).stop 2024-08-03T20:57:42.426 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:42.424+0000 7fdea0000640 1 -- 172.21.15.32:0/1652662873 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fde9815e630 msgr2=0x7fde9815d530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:42.426 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:42.424+0000 7fdea0000640 1 --2- 172.21.15.32:0/1652662873 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fde9815e630 0x7fde9815d530 secure :-1 s=READY pgs=1605 cs=0 l=1 rev1=1 crypto rx=0x7fde8400c320 tx=0x7fde840064a0 comp rx=0 tx=0).stop 2024-08-03T20:57:42.427 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:42.424+0000 7fdea0000640 1 -- 172.21.15.32:0/1652662873 shutdown_connections 2024-08-03T20:57:42.427 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:42.424+0000 7fdea0000640 1 --2- 172.21.15.32:0/1652662873 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fde78077040 0x7fde78079550 unknown :-1 s=CLOSED pgs=655 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:42.427 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:42.424+0000 7fdea0000640 1 --2- 172.21.15.32:0/1652662873 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fde9815ea40 0x7fde9815da70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:42.427 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:42.424+0000 7fdea0000640 1 --2- 172.21.15.32:0/1652662873 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fde9815e630 0x7fde9815d530 unknown :-1 s=CLOSED pgs=1605 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:42.427 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:42.424+0000 7fdea0000640 1 --2- 172.21.15.32:0/1652662873 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fde80004800 0x7fde981f5060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:42.427 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:42.424+0000 7fdea0000640 1 -- 172.21.15.32:0/1652662873 >> 172.21.15.32:0/1652662873 conn(0x7fde980b1bd0 msgr2=0x7fde981e5110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:57:42.427 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:42.424+0000 7fdea0000640 1 -- 172.21.15.32:0/1652662873 shutdown_connections 2024-08-03T20:57:42.427 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:42.425+0000 7fdea0000640 1 -- 172.21.15.32:0/1652662873 wait complete. 2024-08-03T20:57:42.451 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mds repaired 0 2024-08-03T20:57:42.680 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:42.677+0000 7f4f89000640 1 -- 172.21.15.32:0/2431321904 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4f840b4ce0 msgr2=0x7f4f8415ef00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:42.680 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:42.678+0000 7f4f89000640 1 --2- 172.21.15.32:0/2431321904 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4f840b4ce0 0x7f4f8415ef00 secure :-1 s=READY pgs=1606 cs=0 l=1 rev1=1 crypto rx=0x7f4f70005fd0 tx=0x7f4f7002a5d0 comp rx=0 tx=0).stop 2024-08-03T20:57:42.680 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:42.678+0000 7f4f89000640 1 -- 172.21.15.32:0/2431321904 shutdown_connections 2024-08-03T20:57:42.680 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:42.678+0000 7f4f89000640 1 --2- 172.21.15.32:0/2431321904 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4f840b4ce0 0x7f4f8415ef00 unknown :-1 s=CLOSED pgs=1606 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:42.680 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:42.678+0000 7f4f89000640 1 --2- 172.21.15.32:0/2431321904 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f4f840b38b0 0x7f4f841523f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:42.680 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:42.678+0000 7f4f89000640 1 -- 172.21.15.32:0/2431321904 >> 172.21.15.32:0/2431321904 conn(0x7f4f840b1bd0 msgr2=0x7f4f840b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:57:42.680 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:42.678+0000 7f4f89000640 1 -- 172.21.15.32:0/2431321904 shutdown_connections 2024-08-03T20:57:42.680 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:42.678+0000 7f4f89000640 1 -- 172.21.15.32:0/2431321904 wait complete. 2024-08-03T20:57:42.681 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:42.680+0000 7f4f89000640 1 Processor -- start 2024-08-03T20:57:42.682 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:42.680+0000 7f4f89000640 1 -- start start 2024-08-03T20:57:42.684 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:42.682+0000 7f4f89000640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f4f840b38b0 0x7f4f8414e210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:42.684 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:42.682+0000 7f4f89000640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f4f840b4ce0 0x7f4f841f90b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:42.684 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:42.682+0000 7f4f89000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4f8415f440 0x7f4f84147e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:42.684 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:42.682+0000 7f4f89000640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f4f8415c020 con 0x7f4f8415f440 2024-08-03T20:57:42.684 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:42.682+0000 7f4f89000640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f4f840cf240 con 0x7f4f840b38b0 2024-08-03T20:57:42.684 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:42.682+0000 7f4f89000640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7f4f8415c1a0 con 0x7f4f840b4ce0 2024-08-03T20:57:42.684 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:42.682+0000 7f4f82800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4f8415f440 0x7f4f84147e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:42.684 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:42.682+0000 7f4f82800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4f8415f440 0x7f4f84147e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:43146/0 (socket says 172.21.15.32:43146) 2024-08-03T20:57:42.684 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:42.682+0000 7f4f82800640 1 -- 172.21.15.32:0/2883049189 learned_addr learned my addr 172.21.15.32:0/2883049189 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:57:42.684 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:42.682+0000 7f4f81e00640 1 --2- 172.21.15.32:0/2883049189 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f4f840b38b0 0x7f4f8414e210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:42.684 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:42.682+0000 7f4f81400640 1 --2- 172.21.15.32:0/2883049189 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f4f840b4ce0 0x7f4f841f90b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:42.685 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:42.683+0000 7f4f82800640 1 -- 172.21.15.32:0/2883049189 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f4f840b4ce0 msgr2=0x7f4f841f90b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:42.685 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:42.683+0000 7f4f82800640 1 --2- 172.21.15.32:0/2883049189 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f4f840b4ce0 0x7f4f841f90b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:42.685 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:42.683+0000 7f4f82800640 1 -- 172.21.15.32:0/2883049189 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f4f840b38b0 msgr2=0x7f4f8414e210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:42.686 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:42.683+0000 7f4f82800640 1 --2- 172.21.15.32:0/2883049189 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f4f840b38b0 0x7f4f8414e210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:42.686 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:42.683+0000 7f4f82800640 1 -- 172.21.15.32:0/2883049189 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4f84150000 con 0x7f4f8415f440 2024-08-03T20:57:42.686 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:42.683+0000 7f4f82800640 1 --2- 172.21.15.32:0/2883049189 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4f8415f440 0x7f4f84147e10 secure :-1 s=READY pgs=1607 cs=0 l=1 rev1=1 crypto rx=0x7f4f7c00c700 tx=0x7f4f7c00cbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:57:42.686 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:42.683+0000 7f4f7a000640 1 -- 172.21.15.32:0/2883049189 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f7c003b70 con 0x7f4f8415f440 2024-08-03T20:57:42.686 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:42.683+0000 7f4f7a000640 1 -- 172.21.15.32:0/2883049189 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f4f7c003d10 con 0x7f4f8415f440 2024-08-03T20:57:42.686 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:42.683+0000 7f4f89000640 1 -- 172.21.15.32:0/2883049189 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4f8414ea10 con 0x7f4f8415f440 2024-08-03T20:57:42.686 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:42.683+0000 7f4f7a000640 1 -- 172.21.15.32:0/2883049189 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f7c00aab0 con 0x7f4f8415f440 2024-08-03T20:57:42.686 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:42.683+0000 7f4f89000640 1 -- 172.21.15.32:0/2883049189 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4f841f95f0 con 0x7f4f8415f440 2024-08-03T20:57:42.686 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:42.684+0000 7f4f89000640 1 -- 172.21.15.32:0/2883049189 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4f8415c650 con 0x7f4f8415f440 2024-08-03T20:57:42.687 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:42.686+0000 7f4f7a000640 1 -- 172.21.15.32:0/2883049189 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f4f7c00ac50 con 0x7f4f8415f440 2024-08-03T20:57:42.688 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:42.687+0000 7f4f7a000640 1 --2- 172.21.15.32:0/2883049189 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f4f64077010 0x7f4f64079520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:42.688 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:42.687+0000 7f4f7a000640 1 -- 172.21.15.32:0/2883049189 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(107..107 src has 1..107) ==== 7746+0+0 (secure 0 0 0) 0x7f4f7c098eb0 con 0x7f4f8415f440 2024-08-03T20:57:42.688 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:42.687+0000 7f4f81e00640 1 --2- 172.21.15.32:0/2883049189 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f4f64077010 0x7f4f64079520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:42.689 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:42.688+0000 7f4f81e00640 1 --2- 172.21.15.32:0/2883049189 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f4f64077010 0x7f4f64079520 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7f4f74005fd0 tx=0x7f4f7400b2e0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:57:42.692 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:42.691+0000 7f4f7a000640 1 -- 172.21.15.32:0/2883049189 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7f4f8415f850 con 0x7f4f8415f440 2024-08-03T20:57:42.982 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:42.980+0000 7f4f89000640 1 -- 172.21.15.32:0/2883049189 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "mds repaired", "role": "0"} v 0) -- 0x7f4f8414d210 con 0x7f4f8415f440 2024-08-03T20:57:43.430 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:43.428+0000 7f4f7a000640 1 -- 172.21.15.32:0/2883049189 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "mds repaired", "role": "0"}]=0 repaired: restoring rank 6:0 v224) ==== 101+0+0 (secure 0 0 0) 0x7f4f7c061250 con 0x7f4f8415f440 2024-08-03T20:57:43.430 INFO:teuthology.orchestra.run.smithi032.stderr:repaired: restoring rank 6:0 2024-08-03T20:57:43.433 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:43.431+0000 7f4f89000640 1 -- 172.21.15.32:0/2883049189 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f4f64077010 msgr2=0x7f4f64079520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:43.433 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:43.431+0000 7f4f89000640 1 --2- 172.21.15.32:0/2883049189 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f4f64077010 0x7f4f64079520 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7f4f74005fd0 tx=0x7f4f7400b2e0 comp rx=0 tx=0).stop 2024-08-03T20:57:43.433 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:43.431+0000 7f4f89000640 1 -- 172.21.15.32:0/2883049189 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4f8415f440 msgr2=0x7f4f84147e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:43.433 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:43.431+0000 7f4f89000640 1 --2- 172.21.15.32:0/2883049189 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4f8415f440 0x7f4f84147e10 secure :-1 s=READY pgs=1607 cs=0 l=1 rev1=1 crypto rx=0x7f4f7c00c700 tx=0x7f4f7c00cbc0 comp rx=0 tx=0).stop 2024-08-03T20:57:43.433 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:43.431+0000 7f4f89000640 1 -- 172.21.15.32:0/2883049189 shutdown_connections 2024-08-03T20:57:43.433 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:43.431+0000 7f4f89000640 1 --2- 172.21.15.32:0/2883049189 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f4f64077010 0x7f4f64079520 unknown :-1 s=CLOSED pgs=656 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:43.434 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:43.431+0000 7f4f89000640 1 --2- 172.21.15.32:0/2883049189 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4f8415f440 0x7f4f84147e10 unknown :-1 s=CLOSED pgs=1607 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:43.434 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:43.431+0000 7f4f89000640 1 --2- 172.21.15.32:0/2883049189 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f4f840b4ce0 0x7f4f841f90b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:43.434 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:43.431+0000 7f4f89000640 1 --2- 172.21.15.32:0/2883049189 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f4f840b38b0 0x7f4f8414e210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:43.434 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:43.431+0000 7f4f89000640 1 -- 172.21.15.32:0/2883049189 >> 172.21.15.32:0/2883049189 conn(0x7f4f840b1bd0 msgr2=0x7f4f8415a3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:57:43.434 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:43.432+0000 7f4f89000640 1 -- 172.21.15.32:0/2883049189 shutdown_connections 2024-08-03T20:57:43.434 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:43.432+0000 7f4f89000640 1 -- 172.21.15.32:0/2883049189 wait complete. 2024-08-03T20:57:43.457 DEBUG:teuthology.orchestra.run.smithi032:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata import - 2024-08-03T20:57:43.601 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:43.599+0000 7f0c6a88ef40 1 Processor -- start 2024-08-03T20:57:43.601 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:43.599+0000 7f0c6a88ef40 1 -- start start 2024-08-03T20:57:43.603 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:43.601+0000 7f0c6a88ef40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55c1307a3b40 0x55c1308c2560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:43.603 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:43.601+0000 7f0c6a88ef40 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55c130816be0 0x55c130817aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:43.603 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:43.601+0000 7f0c6a88ef40 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55c130812c50 0x55c130813060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:43.604 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:43.601+0000 7f0c6a88ef40 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55c1306ace30 con 0x55c130816be0 2024-08-03T20:57:43.604 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:43.601+0000 7f0c6a88ef40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x55c13070abf0 con 0x55c1307a3b40 2024-08-03T20:57:43.604 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:43.601+0000 7f0c6a88ef40 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x55c1308bfdf0 con 0x55c130812c50 2024-08-03T20:57:43.604 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:43.601+0000 7f0c63000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55c130816be0 0x55c130817aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:43.604 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:43.601+0000 7f0c63000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55c130816be0 0x55c130817aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:43172/0 (socket says 172.21.15.32:43172) 2024-08-03T20:57:43.604 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:43.601+0000 7f0c63000640 1 -- 172.21.15.32:0/1118756340 learned_addr learned my addr 172.21.15.32:0/1118756340 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:57:43.604 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:43.601+0000 7f0c63a00640 1 --2- 172.21.15.32:0/1118756340 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55c1307a3b40 0x55c1308c2560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:43.604 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:43.601+0000 7f0c64400640 1 --2- 172.21.15.32:0/1118756340 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55c130812c50 0x55c130813060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:43.604 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:43.601+0000 7f0c63000640 1 -- 172.21.15.32:0/1118756340 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55c130812c50 msgr2=0x55c130813060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:57:43.604 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:43.601+0000 7f0c63000640 1 --2- 172.21.15.32:0/1118756340 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55c130812c50 0x55c130813060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:43.604 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:43.601+0000 7f0c63000640 1 -- 172.21.15.32:0/1118756340 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55c1307a3b40 msgr2=0x55c1308c2560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:57:43.604 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:43.601+0000 7f0c63000640 1 --2- 172.21.15.32:0/1118756340 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55c1307a3b40 0x55c1308c2560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:43.604 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:43.601+0000 7f0c63000640 1 -- 172.21.15.32:0/1118756340 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c1308135a0 con 0x55c130816be0 2024-08-03T20:57:43.604 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:43.602+0000 7f0c63000640 1 --2- 172.21.15.32:0/1118756340 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55c130816be0 0x55c130817aa0 secure :-1 s=READY pgs=1609 cs=0 l=1 rev1=1 crypto rx=0x7f0c50001d50 tx=0x7f0c5000b5b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d866a86f95e12d31 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:57:43.605 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:43.602+0000 7f0c61c00640 1 -- 172.21.15.32:0/1118756340 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c5000bd80 con 0x55c130816be0 2024-08-03T20:57:43.605 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:43.602+0000 7f0c61c00640 1 -- 172.21.15.32:0/1118756340 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f0c5000f400 con 0x55c130816be0 2024-08-03T20:57:43.605 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:43.602+0000 7f0c61c00640 1 -- 172.21.15.32:0/1118756340 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c5000f850 con 0x55c130816be0 2024-08-03T20:57:43.606 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:43.603+0000 7f0c6a88ef40 1 -- 172.21.15.32:0/1118756340 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55c130816be0 msgr2=0x55c130817aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:43.606 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:43.603+0000 7f0c6a88ef40 1 --2- 172.21.15.32:0/1118756340 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55c130816be0 0x55c130817aa0 secure :-1 s=READY pgs=1609 cs=0 l=1 rev1=1 crypto rx=0x7f0c50001d50 tx=0x7f0c5000b5b0 comp rx=0 tx=0).stop 2024-08-03T20:57:43.606 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:43.603+0000 7f0c6a88ef40 1 -- 172.21.15.32:0/1118756340 shutdown_connections 2024-08-03T20:57:43.606 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:43.603+0000 7f0c6a88ef40 1 --2- 172.21.15.32:0/1118756340 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55c130812c50 0x55c130813060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:43.606 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:43.603+0000 7f0c6a88ef40 1 --2- 172.21.15.32:0/1118756340 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55c130816be0 0x55c130817aa0 unknown :-1 s=CLOSED pgs=1609 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:43.606 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:43.603+0000 7f0c6a88ef40 1 --2- 172.21.15.32:0/1118756340 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55c1307a3b40 0x55c1308c2560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:43.606 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:43.603+0000 7f0c6a88ef40 1 -- 172.21.15.32:0/1118756340 >> 172.21.15.32:0/1118756340 conn(0x55c13067eac0 msgr2=0x55c130836d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:57:43.606 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:43.603+0000 7f0c6a88ef40 1 -- 172.21.15.32:0/1118756340 shutdown_connections 2024-08-03T20:57:43.606 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:43.603+0000 7f0c6a88ef40 1 -- 172.21.15.32:0/1118756340 wait complete. 2024-08-03T20:57:43.607 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:43.604+0000 7f0c6a88ef40 1 Processor -- start 2024-08-03T20:57:43.607 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:43.604+0000 7f0c6a88ef40 1 -- start start 2024-08-03T20:57:43.608 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:43.605+0000 7f0c6a88ef40 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55c1307a3b40 0x55c1308108b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:43.608 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:43.605+0000 7f0c6a88ef40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55c130812c50 0x55c130817b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:43.608 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:43.605+0000 7f0c6a88ef40 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55c130816be0 0x55c1308bcde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:43.609 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:43.605+0000 7f0c6a88ef40 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55c130833510 con 0x55c130816be0 2024-08-03T20:57:43.609 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:43.605+0000 7f0c6a88ef40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x55c130832df0 con 0x55c130812c50 2024-08-03T20:57:43.609 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:43.605+0000 7f0c6a88ef40 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x55c130813e30 con 0x55c1307a3b40 2024-08-03T20:57:43.609 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:43.606+0000 7f0c64400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55c130816be0 0x55c1308bcde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:43.609 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:43.606+0000 7f0c64400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55c130816be0 0x55c1308bcde0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:43178/0 (socket says 172.21.15.32:43178) 2024-08-03T20:57:43.609 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:43.606+0000 7f0c64400640 1 -- 172.21.15.32:0/2033997544 learned_addr learned my addr 172.21.15.32:0/2033997544 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:57:43.609 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:43.606+0000 7f0c63a00640 1 --2- 172.21.15.32:0/2033997544 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55c1307a3b40 0x55c1308108b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:43.609 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:43.606+0000 7f0c63000640 1 --2- 172.21.15.32:0/2033997544 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55c130812c50 0x55c130817b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:43.609 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:43.606+0000 7f0c64400640 1 -- 172.21.15.32:0/2033997544 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55c1307a3b40 msgr2=0x55c1308108b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:43.609 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:43.606+0000 7f0c64400640 1 --2- 172.21.15.32:0/2033997544 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55c1307a3b40 0x55c1308108b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:43.609 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:43.606+0000 7f0c64400640 1 -- 172.21.15.32:0/2033997544 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55c130812c50 msgr2=0x55c130817b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:43.609 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:43.606+0000 7f0c64400640 1 --2- 172.21.15.32:0/2033997544 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55c130812c50 0x55c130817b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:43.609 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:43.606+0000 7f0c64400640 1 -- 172.21.15.32:0/2033997544 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c130810f80 con 0x55c130816be0 2024-08-03T20:57:43.609 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:43.606+0000 7f0c64400640 1 --2- 172.21.15.32:0/2033997544 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55c130816be0 0x55c1308bcde0 secure :-1 s=READY pgs=1610 cs=0 l=1 rev1=1 crypto rx=0x7f0c5c025e30 tx=0x7f0c5c04d7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:57:43.609 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:43.607+0000 7f0c5be00640 1 -- 172.21.15.32:0/2033997544 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c5c064030 con 0x55c130816be0 2024-08-03T20:57:43.610 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:43.607+0000 7f0c5be00640 1 -- 172.21.15.32:0/2033997544 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f0c5c004130 con 0x55c130816be0 2024-08-03T20:57:43.610 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:43.607+0000 7f0c5be00640 1 -- 172.21.15.32:0/2033997544 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c5c060440 con 0x55c130816be0 2024-08-03T20:57:43.610 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:43.607+0000 7f0c6a88ef40 1 -- 172.21.15.32:0/2033997544 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x55c130813190 con 0x55c130816be0 2024-08-03T20:57:43.610 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:43.607+0000 7f0c6a88ef40 1 -- 172.21.15.32:0/2033997544 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c130818070 con 0x55c130816be0 2024-08-03T20:57:43.610 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:43.609+0000 7f0c5be00640 1 -- 172.21.15.32:0/2033997544 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f0c5c05e040 con 0x55c130816be0 2024-08-03T20:57:43.611 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:43.610+0000 7f0c5be00640 1 --2- 172.21.15.32:0/2033997544 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f0c440759f0 0x7f0c44077f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:43.612 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:43.610+0000 7f0c5be00640 1 -- 172.21.15.32:0/2033997544 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(107..107 src has 1..107) ==== 7746+0+0 (secure 0 0 0) 0x7f0c5c0dc150 con 0x55c130816be0 2024-08-03T20:57:43.612 INFO:teuthology.orchestra.run.smithi032.stdout:Importing pool 2024-08-03T20:57:43.613 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:43.610+0000 7f0c63a00640 1 --2- 172.21.15.32:0/2033997544 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f0c440759f0 0x7f0c44077f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:43.613 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:43.611+0000 7f0c6a88ef40 1 --2- 172.21.15.32:0/2033997544 >> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] conn(0x55c1307a7d60 0x55c130814190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:43.613 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:43.611+0000 7f0c6a88ef40 1 -- 172.21.15.32:0/2033997544 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:1 12.10 12:09401a81:::601.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107) -- 0x55c1308146d0 con 0x55c1307a7d60 2024-08-03T20:57:43.613 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:43.611+0000 7f0c63000640 1 --2- 172.21.15.32:0/2033997544 >> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] conn(0x55c1307a7d60 0x55c130814190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:43.613 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:43.611+0000 7f0c63a00640 1 --2- 172.21.15.32:0/2033997544 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f0c440759f0 0x7f0c44077f00 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7f0c5400c6c0 tx=0x7f0c54004a20 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:57:43.614 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:43.611+0000 7f0c63000640 1 --2- 172.21.15.32:0/2033997544 >> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] conn(0x55c1307a7d60 0x55c130814190 crc :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:57:43.615 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:43.614+0000 7f0c63000640 1 -- 172.21.15.32:0/2033997544 <== osd.0 v2:172.21.15.32:6816/3066156116 1 ==== osd_op_reply(1 601.00000000 [create] v107'660 uv634 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f0c5000f400 con 0x55c1307a7d60 2024-08-03T20:57:43.615 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:43.614+0000 7f0c6a88ef40 1 -- 172.21.15.32:0/2033997544 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:2 12.10 12:09401a81:::601.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e107) -- 0x55c1308d7d00 con 0x55c1307a7d60 2024-08-03T20:57:43.617 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:43.615+0000 7f0c63000640 1 -- 172.21.15.32:0/2033997544 <== osd.0 v2:172.21.15.32:6816/3066156116 2 ==== osd_op_reply(2 601.00000000 [delete] v107'661 uv661 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f0c5000f400 con 0x55c1307a7d60 2024-08-03T20:57:43.617 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:43.615+0000 7f0c6a88ef40 1 -- 172.21.15.32:0/2033997544 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:3 12.10 12:09401a81:::601.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107) -- 0x55c1308d80a0 con 0x55c1307a7d60 2024-08-03T20:57:43.619 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::601.00000000:head# 2024-08-03T20:57:43.619 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:43.617+0000 7f0c63000640 1 -- 172.21.15.32:0/2033997544 <== osd.0 v2:172.21.15.32:6816/3066156116 3 ==== osd_op_reply(3 601.00000000 [create] v107'662 uv662 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f0c5000f400 con 0x55c1307a7d60 2024-08-03T20:57:43.619 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:43.617+0000 7f0c6a88ef40 1 -- 172.21.15.32:0/2033997544 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:4 12.10 12:09401a81:::601.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107) -- 0x55c1308d9450 con 0x55c1307a7d60 2024-08-03T20:57:43.620 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:43.619+0000 7f0c63000640 1 -- 172.21.15.32:0/2033997544 <== osd.0 v2:172.21.15.32:6816/3066156116 4 ==== osd_op_reply(4 601.00000000 [setxattr (30)] v107'663 uv663 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f0c5000f400 con 0x55c1307a7d60 2024-08-03T20:57:43.620 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:43.619+0000 7f0c6a88ef40 1 -- 172.21.15.32:0/2033997544 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:5 12.10 12:09401a81:::601.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107) -- 0x55c1308d97f0 con 0x55c1307a7d60 2024-08-03T20:57:43.622 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:43.620+0000 7f0c63000640 1 -- 172.21.15.32:0/2033997544 <== osd.0 v2:172.21.15.32:6816/3066156116 5 ==== osd_op_reply(5 601.00000000 [setxattr (62)] v107'664 uv664 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f0c5000f400 con 0x55c1307a7d60 2024-08-03T20:57:43.622 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:43.620+0000 7f0c6a88ef40 1 -- 172.21.15.32:0/2033997544 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:6 12.10 12:09401a81:::601.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107) -- 0x55c1308d8440 con 0x55c1307a7d60 2024-08-03T20:57:43.623 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:43.622+0000 7f0c63000640 1 -- 172.21.15.32:0/2033997544 <== osd.0 v2:172.21.15.32:6816/3066156116 6 ==== osd_op_reply(6 601.00000000 [omap-set-header] v107'665 uv665 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f0c5000f400 con 0x55c1307a7d60 2024-08-03T20:57:43.623 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:43.622+0000 7f0c6a88ef40 1 -- 172.21.15.32:0/2033997544 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:7 12.10 12:0bd6d154:::602.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107) -- 0x55c1308d8830 con 0x55c1307a7d60 2024-08-03T20:57:43.625 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:43.623+0000 7f0c63000640 1 -- 172.21.15.32:0/2033997544 <== osd.0 v2:172.21.15.32:6816/3066156116 7 ==== osd_op_reply(7 602.00000000 [create] v107'666 uv640 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f0c5000f400 con 0x55c1307a7d60 2024-08-03T20:57:43.625 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:43.624+0000 7f0c6a88ef40 1 -- 172.21.15.32:0/2033997544 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:8 12.10 12:0bd6d154:::602.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e107) -- 0x55c1308d8bd0 con 0x55c1307a7d60 2024-08-03T20:57:43.626 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:43.625+0000 7f0c63000640 1 -- 172.21.15.32:0/2033997544 <== osd.0 v2:172.21.15.32:6816/3066156116 8 ==== osd_op_reply(8 602.00000000 [delete] v107'667 uv667 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f0c5000f400 con 0x55c1307a7d60 2024-08-03T20:57:43.626 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:43.625+0000 7f0c6a88ef40 1 -- 172.21.15.32:0/2033997544 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:9 12.10 12:0bd6d154:::602.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107) -- 0x55c1308d8bd0 con 0x55c1307a7d60 2024-08-03T20:57:43.628 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:43.627+0000 7f0c63000640 1 -- 172.21.15.32:0/2033997544 <== osd.0 v2:172.21.15.32:6816/3066156116 9 ==== osd_op_reply(9 602.00000000 [create] v107'668 uv668 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f0c5000f400 con 0x55c1307a7d60 2024-08-03T20:57:43.629 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::602.00000000:head# 2024-08-03T20:57:43.629 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:43.627+0000 7f0c6a88ef40 1 -- 172.21.15.32:0/2033997544 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:10 12.10 12:0bd6d154:::602.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107) -- 0x55c1308d8bd0 con 0x55c1307a7d60 2024-08-03T20:57:43.630 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:43.628+0000 7f0c63000640 1 -- 172.21.15.32:0/2033997544 <== osd.0 v2:172.21.15.32:6816/3066156116 10 ==== osd_op_reply(10 602.00000000 [setxattr (30)] v107'669 uv669 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f0c5000f400 con 0x55c1307a7d60 2024-08-03T20:57:43.630 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:43.629+0000 7f0c6a88ef40 1 -- 172.21.15.32:0/2033997544 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:11 12.10 12:0bd6d154:::602.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107) -- 0x55c1308d8bd0 con 0x55c1307a7d60 2024-08-03T20:57:43.632 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:43.630+0000 7f0c63000640 1 -- 172.21.15.32:0/2033997544 <== osd.0 v2:172.21.15.32:6816/3066156116 11 ==== osd_op_reply(11 602.00000000 [setxattr (62)] v107'670 uv670 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f0c5000f400 con 0x55c1307a7d60 2024-08-03T20:57:43.632 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:43.630+0000 7f0c6a88ef40 1 -- 172.21.15.32:0/2033997544 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:12 12.10 12:0bd6d154:::602.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107) -- 0x55c1308d8bd0 con 0x55c1307a7d60 2024-08-03T20:57:43.633 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:43.632+0000 7f0c63000640 1 -- 172.21.15.32:0/2033997544 <== osd.0 v2:172.21.15.32:6816/3066156116 12 ==== osd_op_reply(12 602.00000000 [omap-set-header] v107'671 uv671 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f0c5000f400 con 0x55c1307a7d60 2024-08-03T20:57:43.634 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:43.632+0000 7f0c6a88ef40 1 -- 172.21.15.32:0/2033997544 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:13 12.10 12:0cf6ad0f:::10000000000.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107) -- 0x55c1308d8bd0 con 0x55c1307a7d60 2024-08-03T20:57:43.635 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:43.634+0000 7f0c63000640 1 -- 172.21.15.32:0/2033997544 <== osd.0 v2:172.21.15.32:6816/3066156116 13 ==== osd_op_reply(13 10000000000.00000000 [create] v107'672 uv647 ondisk = -17 ((17) File exists)) ==== 164+0+0 (crc 0 0 0) 0x7f0c5000f400 con 0x55c1307a7d60 2024-08-03T20:57:43.635 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:43.634+0000 7f0c6a88ef40 1 -- 172.21.15.32:0/2033997544 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:14 12.10 12:0cf6ad0f:::10000000000.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e107) -- 0x55c1308d8bd0 con 0x55c1307a7d60 2024-08-03T20:57:43.637 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:43.635+0000 7f0c63000640 1 -- 172.21.15.32:0/2033997544 <== osd.0 v2:172.21.15.32:6816/3066156116 14 ==== osd_op_reply(14 10000000000.00000000 [delete] v107'673 uv673 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f0c5000f400 con 0x55c1307a7d60 2024-08-03T20:57:43.637 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:43.635+0000 7f0c6a88ef40 1 -- 172.21.15.32:0/2033997544 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:15 12.10 12:0cf6ad0f:::10000000000.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107) -- 0x55c1308d8bd0 con 0x55c1307a7d60 2024-08-03T20:57:43.638 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::10000000000.00000000:head# 2024-08-03T20:57:43.639 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:43.637+0000 7f0c63000640 1 -- 172.21.15.32:0/2033997544 <== osd.0 v2:172.21.15.32:6816/3066156116 15 ==== osd_op_reply(15 10000000000.00000000 [create] v107'674 uv674 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f0c5000f400 con 0x55c1307a7d60 2024-08-03T20:57:43.639 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:43.637+0000 7f0c6a88ef40 1 -- 172.21.15.32:0/2033997544 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:16 12.10 12:0cf6ad0f:::10000000000.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107) -- 0x55c1308d8bd0 con 0x55c1307a7d60 2024-08-03T20:57:43.640 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:43.638+0000 7f0c63000640 1 -- 172.21.15.32:0/2033997544 <== osd.0 v2:172.21.15.32:6816/3066156116 16 ==== osd_op_reply(16 10000000000.00000000 [setxattr (30)] v107'675 uv675 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f0c5000f400 con 0x55c1307a7d60 2024-08-03T20:57:43.640 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:43.639+0000 7f0c6a88ef40 1 -- 172.21.15.32:0/2033997544 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:17 12.10 12:0cf6ad0f:::10000000000.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107) -- 0x55c1308d8bd0 con 0x55c1307a7d60 2024-08-03T20:57:43.642 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:43.640+0000 7f0c63000640 1 -- 172.21.15.32:0/2033997544 <== osd.0 v2:172.21.15.32:6816/3066156116 17 ==== osd_op_reply(17 10000000000.00000000 [setxattr (62)] v107'676 uv676 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f0c5000f400 con 0x55c1307a7d60 2024-08-03T20:57:43.642 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:43.640+0000 7f0c6a88ef40 1 -- 172.21.15.32:0/2033997544 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:18 12.10 12:0cf6ad0f:::10000000000.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107) -- 0x55c1308d8bd0 con 0x55c1307a7d60 2024-08-03T20:57:43.643 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:43.642+0000 7f0c63000640 1 -- 172.21.15.32:0/2033997544 <== osd.0 v2:172.21.15.32:6816/3066156116 18 ==== osd_op_reply(18 10000000000.00000000 [omap-set-header] v107'677 uv677 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f0c5000f400 con 0x55c1307a7d60 2024-08-03T20:57:43.643 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:43.642+0000 7f0c6a88ef40 1 -- 172.21.15.32:0/2033997544 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:19 12.10 12:0cf6ad0f:::10000000000.00000000:head [omap-set-vals in=593b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107) -- 0x55c1308d9b90 con 0x55c1307a7d60 2024-08-03T20:57:43.645 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:43.643+0000 7f0c63000640 1 -- 172.21.15.32:0/2033997544 <== osd.0 v2:172.21.15.32:6816/3066156116 19 ==== osd_op_reply(19 10000000000.00000000 [omap-set-vals] v107'678 uv678 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f0c5000f400 con 0x55c1307a7d60 2024-08-03T20:57:43.645 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:43.644+0000 7f0c6a88ef40 1 -- 172.21.15.32:0/2033997544 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:20 12.10 12:0d82a743:::600.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107) -- 0x55c1308d9b90 con 0x55c1307a7d60 2024-08-03T20:57:43.646 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:43.645+0000 7f0c63000640 1 -- 172.21.15.32:0/2033997544 <== osd.0 v2:172.21.15.32:6816/3066156116 20 ==== osd_op_reply(20 600.00000000 [create] v107'679 uv653 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f0c5000f400 con 0x55c1307a7d60 2024-08-03T20:57:43.647 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:43.645+0000 7f0c6a88ef40 1 -- 172.21.15.32:0/2033997544 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:21 12.10 12:0d82a743:::600.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e107) -- 0x55c1308d9b90 con 0x55c1307a7d60 2024-08-03T20:57:43.648 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:43.647+0000 7f0c63000640 1 -- 172.21.15.32:0/2033997544 <== osd.0 v2:172.21.15.32:6816/3066156116 21 ==== osd_op_reply(21 600.00000000 [delete] v107'680 uv680 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f0c5000f400 con 0x55c1307a7d60 2024-08-03T20:57:43.648 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:43.647+0000 7f0c6a88ef40 1 -- 172.21.15.32:0/2033997544 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:22 12.10 12:0d82a743:::600.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107) -- 0x55c1308d9b90 con 0x55c1307a7d60 2024-08-03T20:57:43.650 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::600.00000000:head# 2024-08-03T20:57:43.650 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:43.648+0000 7f0c63000640 1 -- 172.21.15.32:0/2033997544 <== osd.0 v2:172.21.15.32:6816/3066156116 22 ==== osd_op_reply(22 600.00000000 [create] v107'681 uv681 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f0c5000f400 con 0x55c1307a7d60 2024-08-03T20:57:43.650 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:43.648+0000 7f0c6a88ef40 1 -- 172.21.15.32:0/2033997544 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:23 12.10 12:0d82a743:::600.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107) -- 0x55c1308daba0 con 0x55c1307a7d60 2024-08-03T20:57:43.651 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:43.650+0000 7f0c63000640 1 -- 172.21.15.32:0/2033997544 <== osd.0 v2:172.21.15.32:6816/3066156116 23 ==== osd_op_reply(23 600.00000000 [setxattr (30)] v107'682 uv682 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f0c5000f400 con 0x55c1307a7d60 2024-08-03T20:57:43.651 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:43.650+0000 7f0c6a88ef40 1 -- 172.21.15.32:0/2033997544 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:24 12.10 12:0d82a743:::600.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107) -- 0x55c1308daba0 con 0x55c1307a7d60 2024-08-03T20:57:43.653 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:43.651+0000 7f0c63000640 1 -- 172.21.15.32:0/2033997544 <== osd.0 v2:172.21.15.32:6816/3066156116 24 ==== osd_op_reply(24 600.00000000 [setxattr (62)] v107'683 uv683 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f0c5000f400 con 0x55c1307a7d60 2024-08-03T20:57:43.653 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:43.651+0000 7f0c6a88ef40 1 -- 172.21.15.32:0/2033997544 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:25 12.10 12:0d82a743:::600.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107) -- 0x55c1308d9b90 con 0x55c1307a7d60 2024-08-03T20:57:43.654 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:43.653+0000 7f0c63000640 1 -- 172.21.15.32:0/2033997544 <== osd.0 v2:172.21.15.32:6816/3066156116 25 ==== osd_op_reply(25 600.00000000 [omap-set-header] v107'684 uv684 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f0c5000f400 con 0x55c1307a7d60 2024-08-03T20:57:43.654 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:43.653+0000 7f0c6a88ef40 1 -- 172.21.15.32:0/2033997544 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:26 12.10 12:0d89b25e:::603.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107) -- 0x55c1308d9b90 con 0x55c1307a7d60 2024-08-03T20:57:43.656 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:43.654+0000 7f0c63000640 1 -- 172.21.15.32:0/2033997544 <== osd.0 v2:172.21.15.32:6816/3066156116 26 ==== osd_op_reply(26 603.00000000 [create] v107'685 uv659 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f0c5000f400 con 0x55c1307a7d60 2024-08-03T20:57:43.656 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:43.655+0000 7f0c6a88ef40 1 -- 172.21.15.32:0/2033997544 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:27 12.10 12:0d89b25e:::603.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e107) -- 0x55c1308d9b90 con 0x55c1307a7d60 2024-08-03T20:57:43.657 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:43.656+0000 7f0c63000640 1 -- 172.21.15.32:0/2033997544 <== osd.0 v2:172.21.15.32:6816/3066156116 27 ==== osd_op_reply(27 603.00000000 [delete] v107'686 uv686 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f0c5000f400 con 0x55c1307a7d60 2024-08-03T20:57:43.658 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:43.656+0000 7f0c6a88ef40 1 -- 172.21.15.32:0/2033997544 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:28 12.10 12:0d89b25e:::603.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107) -- 0x55c1308d9b90 con 0x55c1307a7d60 2024-08-03T20:57:43.659 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::603.00000000:head# 2024-08-03T20:57:43.659 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:43.658+0000 7f0c63000640 1 -- 172.21.15.32:0/2033997544 <== osd.0 v2:172.21.15.32:6816/3066156116 28 ==== osd_op_reply(28 603.00000000 [create] v107'687 uv687 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f0c5000f400 con 0x55c1307a7d60 2024-08-03T20:57:43.659 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:43.658+0000 7f0c6a88ef40 1 -- 172.21.15.32:0/2033997544 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:29 12.10 12:0d89b25e:::603.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107) -- 0x55c1308daba0 con 0x55c1307a7d60 2024-08-03T20:57:43.660 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:43.659+0000 7f0c63000640 1 -- 172.21.15.32:0/2033997544 <== osd.0 v2:172.21.15.32:6816/3066156116 29 ==== osd_op_reply(29 603.00000000 [setxattr (30)] v107'688 uv688 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f0c5000f400 con 0x55c1307a7d60 2024-08-03T20:57:43.661 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:43.659+0000 7f0c6a88ef40 1 -- 172.21.15.32:0/2033997544 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:30 12.10 12:0d89b25e:::603.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107) -- 0x55c1308daba0 con 0x55c1307a7d60 2024-08-03T20:57:43.662 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:43.661+0000 7f0c63000640 1 -- 172.21.15.32:0/2033997544 <== osd.0 v2:172.21.15.32:6816/3066156116 30 ==== osd_op_reply(30 603.00000000 [setxattr (62)] v107'689 uv689 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f0c5000f400 con 0x55c1307a7d60 2024-08-03T20:57:43.662 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:43.661+0000 7f0c6a88ef40 1 -- 172.21.15.32:0/2033997544 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:31 12.10 12:0d89b25e:::603.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107) -- 0x55c1308d9b90 con 0x55c1307a7d60 2024-08-03T20:57:43.664 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:43.662+0000 7f0c63000640 1 -- 172.21.15.32:0/2033997544 <== osd.0 v2:172.21.15.32:6816/3066156116 31 ==== osd_op_reply(31 603.00000000 [omap-set-header] v107'690 uv690 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f0c5000f400 con 0x55c1307a7d60 2024-08-03T20:57:43.665 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:43.662+0000 7f0c6a88ef40 1 --2- 172.21.15.32:0/2033997544 >> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] conn(0x55c1308da0a0 0x55c1308dc4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:43.665 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:43.662+0000 7f0c6a88ef40 1 -- 172.21.15.32:0/2033997544 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:32 12.14 12:287034c4:::1.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107) -- 0x55c1308dca00 con 0x55c1308da0a0 2024-08-03T20:57:43.665 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:43.663+0000 7f0c63a00640 1 --2- 172.21.15.32:0/2033997544 >> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] conn(0x55c1308da0a0 0x55c1308dc4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:43.666 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:43.663+0000 7f0c63a00640 1 --2- 172.21.15.32:0/2033997544 >> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] conn(0x55c1308da0a0 0x55c1308dc4c0 crc :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:57:43.667 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:43.666+0000 7f0c63a00640 1 -- 172.21.15.32:0/2033997544 <== osd.5 v2:172.21.15.118:6810/2724891770 1 ==== osd_op_reply(32 1.00000000.inode [create] v107'484 uv467 ondisk = -17 ((17) File exists)) ==== 160+0+0 (crc 0 0 0) 0x7f0c5400a460 con 0x55c1308da0a0 2024-08-03T20:57:43.668 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:43.666+0000 7f0c6a88ef40 1 -- 172.21.15.32:0/2033997544 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:33 12.14 12:287034c4:::1.00000000.inode:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e107) -- 0x55c1308dcda0 con 0x55c1308da0a0 2024-08-03T20:57:43.669 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:43.667+0000 7f0c63a00640 1 -- 172.21.15.32:0/2033997544 <== osd.5 v2:172.21.15.118:6810/2724891770 2 ==== osd_op_reply(33 1.00000000.inode [delete] v107'485 uv485 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f0c5400a460 con 0x55c1308da0a0 2024-08-03T20:57:43.669 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:43.667+0000 7f0c6a88ef40 1 -- 172.21.15.32:0/2033997544 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:34 12.14 12:287034c4:::1.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107) -- 0x55c1308dd140 con 0x55c1308da0a0 2024-08-03T20:57:43.670 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:43.669+0000 7f0c63a00640 1 -- 172.21.15.32:0/2033997544 <== osd.5 v2:172.21.15.118:6810/2724891770 3 ==== osd_op_reply(34 1.00000000.inode [create] v107'486 uv486 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f0c5400a460 con 0x55c1308da0a0 2024-08-03T20:57:43.671 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::1.00000000.inode:head# 2024-08-03T20:57:43.671 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:43.669+0000 7f0c6a88ef40 1 -- 172.21.15.32:0/2033997544 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:35 12.14 12:287034c4:::1.00000000.inode:head [write 0~582 in=582b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107) -- 0x55c1308dd4e0 con 0x55c1308da0a0 2024-08-03T20:57:43.672 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:43.670+0000 7f0c63a00640 1 -- 172.21.15.32:0/2033997544 <== osd.5 v2:172.21.15.118:6810/2724891770 4 ==== osd_op_reply(35 1.00000000.inode [write 0~582] v107'487 uv487 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f0c5400a460 con 0x55c1308da0a0 2024-08-03T20:57:43.672 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:43.670+0000 7f0c6a88ef40 1 -- 172.21.15.32:0/2033997544 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:36 12.14 12:287034c4:::1.00000000.inode:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107) -- 0x55c1308dd880 con 0x55c1308da0a0 2024-08-03T20:57:43.673 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:43.672+0000 7f0c63a00640 1 -- 172.21.15.32:0/2033997544 <== osd.5 v2:172.21.15.118:6810/2724891770 5 ==== osd_op_reply(36 1.00000000.inode [omap-set-header] v107'488 uv488 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f0c5400a460 con 0x55c1308da0a0 2024-08-03T20:57:43.674 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:43.672+0000 7f0c6a88ef40 1 -- 172.21.15.32:0/2033997544 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:37 12.14 12:292cf221:::200.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107) -- 0x55c1308ddc20 con 0x55c1308da0a0 2024-08-03T20:57:43.675 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:43.673+0000 7f0c63a00640 1 -- 172.21.15.32:0/2033997544 <== osd.5 v2:172.21.15.118:6810/2724891770 6 ==== osd_op_reply(37 200.00000000 [create] v107'489 uv472 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f0c5400a460 con 0x55c1308da0a0 2024-08-03T20:57:43.675 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:43.673+0000 7f0c6a88ef40 1 -- 172.21.15.32:0/2033997544 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:38 12.14 12:292cf221:::200.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e107) -- 0x55c1308ddfc0 con 0x55c1308da0a0 2024-08-03T20:57:43.677 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:43.675+0000 7f0c63a00640 1 -- 172.21.15.32:0/2033997544 <== osd.5 v2:172.21.15.118:6810/2724891770 7 ==== osd_op_reply(38 200.00000000 [delete] v107'490 uv490 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f0c5400a460 con 0x55c1308da0a0 2024-08-03T20:57:43.677 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:43.675+0000 7f0c6a88ef40 1 -- 172.21.15.32:0/2033997544 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:39 12.14 12:292cf221:::200.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107) -- 0x55c1308de360 con 0x55c1308da0a0 2024-08-03T20:57:43.678 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::200.00000000:head# 2024-08-03T20:57:43.678 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:43.676+0000 7f0c63a00640 1 -- 172.21.15.32:0/2033997544 <== osd.5 v2:172.21.15.118:6810/2724891770 8 ==== osd_op_reply(39 200.00000000 [create] v107'491 uv491 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f0c5400a460 con 0x55c1308da0a0 2024-08-03T20:57:43.678 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:43.677+0000 7f0c6a88ef40 1 -- 172.21.15.32:0/2033997544 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:40 12.14 12:292cf221:::200.00000000:head [write 0~90 in=90b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107) -- 0x55c1308de360 con 0x55c1308da0a0 2024-08-03T20:57:43.679 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:43.678+0000 7f0c63a00640 1 -- 172.21.15.32:0/2033997544 <== osd.5 v2:172.21.15.118:6810/2724891770 9 ==== osd_op_reply(40 200.00000000 [write 0~90] v107'492 uv492 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f0c5400a460 con 0x55c1308da0a0 2024-08-03T20:57:43.680 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:43.678+0000 7f0c6a88ef40 1 -- 172.21.15.32:0/2033997544 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:41 12.14 12:292cf221:::200.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107) -- 0x55c1308de360 con 0x55c1308da0a0 2024-08-03T20:57:43.681 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:43.680+0000 7f0c63a00640 1 -- 172.21.15.32:0/2033997544 <== osd.5 v2:172.21.15.118:6810/2724891770 10 ==== osd_op_reply(41 200.00000000 [omap-set-header] v107'493 uv493 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f0c5400a460 con 0x55c1308da0a0 2024-08-03T20:57:43.681 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:43.680+0000 7f0c6a88ef40 1 -- 172.21.15.32:0/2033997544 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:42 12.14 12:2e2fa760:::200.00000001:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107) -- 0x55c1308de360 con 0x55c1308da0a0 2024-08-03T20:57:43.683 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:43.681+0000 7f0c63a00640 1 -- 172.21.15.32:0/2033997544 <== osd.5 v2:172.21.15.118:6810/2724891770 11 ==== osd_op_reply(42 200.00000001 [create] v107'494 uv477 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f0c5400a460 con 0x55c1308da0a0 2024-08-03T20:57:43.683 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:43.681+0000 7f0c6a88ef40 1 -- 172.21.15.32:0/2033997544 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:43 12.14 12:2e2fa760:::200.00000001:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e107) -- 0x55c1308de360 con 0x55c1308da0a0 2024-08-03T20:57:43.684 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:43.683+0000 7f0c63a00640 1 -- 172.21.15.32:0/2033997544 <== osd.5 v2:172.21.15.118:6810/2724891770 12 ==== osd_op_reply(43 200.00000001 [delete] v107'495 uv495 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f0c5400a460 con 0x55c1308da0a0 2024-08-03T20:57:43.684 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:43.683+0000 7f0c6a88ef40 1 -- 172.21.15.32:0/2033997544 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:44 12.14 12:2e2fa760:::200.00000001:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107) -- 0x55c1308de360 con 0x55c1308da0a0 2024-08-03T20:57:43.686 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:43.684+0000 7f0c63a00640 1 -- 172.21.15.32:0/2033997544 <== osd.5 v2:172.21.15.118:6810/2724891770 13 ==== osd_op_reply(44 200.00000001 [create] v107'496 uv496 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f0c5400a460 con 0x55c1308da0a0 2024-08-03T20:57:43.686 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::200.00000001:head# 2024-08-03T20:57:43.686 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:43.684+0000 7f0c6a88ef40 1 -- 172.21.15.32:0/2033997544 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:45 12.14 12:2e2fa760:::200.00000001:head [write 0~18118 in=18118b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107) -- 0x55c1308e2a50 con 0x55c1308da0a0 2024-08-03T20:57:43.688 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:43.686+0000 7f0c63a00640 1 -- 172.21.15.32:0/2033997544 <== osd.5 v2:172.21.15.118:6810/2724891770 14 ==== osd_op_reply(45 200.00000001 [write 0~18118] v107'497 uv497 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f0c5400a460 con 0x55c1308da0a0 2024-08-03T20:57:43.688 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:43.686+0000 7f0c6a88ef40 1 -- 172.21.15.32:0/2033997544 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:46 12.14 12:2e2fa760:::200.00000001:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107) -- 0x55c1308e2a50 con 0x55c1308da0a0 2024-08-03T20:57:43.689 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:43.688+0000 7f0c63a00640 1 -- 172.21.15.32:0/2033997544 <== osd.5 v2:172.21.15.118:6810/2724891770 15 ==== osd_op_reply(46 200.00000001 [omap-set-header] v107'498 uv498 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f0c5400a460 con 0x55c1308da0a0 2024-08-03T20:57:43.689 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:43.688+0000 7f0c6a88ef40 1 -- 172.21.15.32:0/2033997544 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:47 12.14 12:2f55791f:::606.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107) -- 0x55c1308de360 con 0x55c1308da0a0 2024-08-03T20:57:43.691 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:43.689+0000 7f0c63a00640 1 -- 172.21.15.32:0/2033997544 <== osd.5 v2:172.21.15.118:6810/2724891770 16 ==== osd_op_reply(47 606.00000000 [create] v107'499 uv483 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f0c5400a460 con 0x55c1308da0a0 2024-08-03T20:57:43.691 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:43.689+0000 7f0c6a88ef40 1 -- 172.21.15.32:0/2033997544 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:48 12.14 12:2f55791f:::606.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e107) -- 0x55c1308de360 con 0x55c1308da0a0 2024-08-03T20:57:43.692 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:43.691+0000 7f0c63a00640 1 -- 172.21.15.32:0/2033997544 <== osd.5 v2:172.21.15.118:6810/2724891770 17 ==== osd_op_reply(48 606.00000000 [delete] v107'500 uv500 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f0c5400a460 con 0x55c1308da0a0 2024-08-03T20:57:43.692 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:43.691+0000 7f0c6a88ef40 1 -- 172.21.15.32:0/2033997544 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:49 12.14 12:2f55791f:::606.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107) -- 0x55c1308de360 con 0x55c1308da0a0 2024-08-03T20:57:43.693 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:43.692+0000 7f0c63a00640 1 -- 172.21.15.32:0/2033997544 <== osd.5 v2:172.21.15.118:6810/2724891770 18 ==== osd_op_reply(49 606.00000000 [create] v107'501 uv501 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f0c5400a460 con 0x55c1308da0a0 2024-08-03T20:57:43.694 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::606.00000000:head# 2024-08-03T20:57:43.694 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:43.692+0000 7f0c6a88ef40 1 -- 172.21.15.32:0/2033997544 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:50 12.14 12:2f55791f:::606.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107) -- 0x55c1308df370 con 0x55c1308da0a0 2024-08-03T20:57:43.695 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:43.694+0000 7f0c63a00640 1 -- 172.21.15.32:0/2033997544 <== osd.5 v2:172.21.15.118:6810/2724891770 19 ==== osd_op_reply(50 606.00000000 [setxattr (30)] v107'502 uv502 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f0c5400a460 con 0x55c1308da0a0 2024-08-03T20:57:43.695 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:43.694+0000 7f0c6a88ef40 1 -- 172.21.15.32:0/2033997544 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:51 12.14 12:2f55791f:::606.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107) -- 0x55c1308df370 con 0x55c1308da0a0 2024-08-03T20:57:43.697 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:43.695+0000 7f0c63a00640 1 -- 172.21.15.32:0/2033997544 <== osd.5 v2:172.21.15.118:6810/2724891770 20 ==== osd_op_reply(51 606.00000000 [setxattr (62)] v107'503 uv503 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f0c5400a460 con 0x55c1308da0a0 2024-08-03T20:57:43.697 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:43.695+0000 7f0c6a88ef40 1 -- 172.21.15.32:0/2033997544 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:52 12.14 12:2f55791f:::606.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107) -- 0x55c1308de360 con 0x55c1308da0a0 2024-08-03T20:57:43.698 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:43.697+0000 7f0c63a00640 1 -- 172.21.15.32:0/2033997544 <== osd.5 v2:172.21.15.118:6810/2724891770 21 ==== osd_op_reply(52 606.00000000 [omap-set-header] v107'504 uv504 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f0c5400a460 con 0x55c1308da0a0 2024-08-03T20:57:43.698 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:43.697+0000 7f0c6a88ef40 1 -- 172.21.15.32:0/2033997544 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:53 12.1c 12:3ed09add:::607.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107) -- 0x55c1308de360 con 0x55c1307a7d60 2024-08-03T20:57:43.700 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:43.698+0000 7f0c63000640 1 -- 172.21.15.32:0/2033997544 <== osd.0 v2:172.21.15.32:6816/3066156116 32 ==== osd_op_reply(53 607.00000000 [create] v107'129 uv128 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f0c5000f400 con 0x55c1307a7d60 2024-08-03T20:57:43.700 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:43.698+0000 7f0c6a88ef40 1 -- 172.21.15.32:0/2033997544 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:54 12.1c 12:3ed09add:::607.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e107) -- 0x55c1308de360 con 0x55c1307a7d60 2024-08-03T20:57:43.701 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:43.700+0000 7f0c63000640 1 -- 172.21.15.32:0/2033997544 <== osd.0 v2:172.21.15.32:6816/3066156116 33 ==== osd_op_reply(54 607.00000000 [delete] v107'130 uv130 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f0c5000f400 con 0x55c1307a7d60 2024-08-03T20:57:43.701 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:43.700+0000 7f0c6a88ef40 1 -- 172.21.15.32:0/2033997544 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:55 12.1c 12:3ed09add:::607.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107) -- 0x55c1308de360 con 0x55c1307a7d60 2024-08-03T20:57:43.702 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:43.701+0000 7f0c63000640 1 -- 172.21.15.32:0/2033997544 <== osd.0 v2:172.21.15.32:6816/3066156116 34 ==== osd_op_reply(55 607.00000000 [create] v107'131 uv131 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f0c5000f400 con 0x55c1307a7d60 2024-08-03T20:57:43.703 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::607.00000000:head# 2024-08-03T20:57:43.703 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:43.701+0000 7f0c6a88ef40 1 -- 172.21.15.32:0/2033997544 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:56 12.1c 12:3ed09add:::607.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107) -- 0x55c1308df370 con 0x55c1307a7d60 2024-08-03T20:57:43.704 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:43.702+0000 7f0c63000640 1 -- 172.21.15.32:0/2033997544 <== osd.0 v2:172.21.15.32:6816/3066156116 35 ==== osd_op_reply(56 607.00000000 [setxattr (30)] v107'132 uv132 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f0c5000f400 con 0x55c1307a7d60 2024-08-03T20:57:43.704 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:43.702+0000 7f0c6a88ef40 1 -- 172.21.15.32:0/2033997544 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:57 12.1c 12:3ed09add:::607.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107) -- 0x55c1308df370 con 0x55c1307a7d60 2024-08-03T20:57:43.705 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:43.704+0000 7f0c63000640 1 -- 172.21.15.32:0/2033997544 <== osd.0 v2:172.21.15.32:6816/3066156116 36 ==== osd_op_reply(57 607.00000000 [setxattr (62)] v107'133 uv133 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f0c5000f400 con 0x55c1307a7d60 2024-08-03T20:57:43.705 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:43.704+0000 7f0c6a88ef40 1 -- 172.21.15.32:0/2033997544 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:58 12.1c 12:3ed09add:::607.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107) -- 0x55c1308de360 con 0x55c1307a7d60 2024-08-03T20:57:43.706 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:43.705+0000 7f0c63000640 1 -- 172.21.15.32:0/2033997544 <== osd.0 v2:172.21.15.32:6816/3066156116 37 ==== osd_op_reply(58 607.00000000 [omap-set-header] v107'134 uv134 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f0c5000f400 con 0x55c1307a7d60 2024-08-03T20:57:43.707 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:43.705+0000 7f0c6a88ef40 1 --2- 172.21.15.32:0/2033997544 >> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] conn(0x55c1308de870 0x55c1308e0c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:43.707 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:43.705+0000 7f0c6a88ef40 1 -- 172.21.15.32:0/2033997544 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:59 12.6 12:654134d2:::mds0_openfiles.0:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107) -- 0x55c1308e11d0 con 0x55c1308de870 2024-08-03T20:57:43.707 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:43.705+0000 7f0c63000640 1 --2- 172.21.15.32:0/2033997544 >> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] conn(0x55c1308de870 0x55c1308e0c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:43.707 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:43.706+0000 7f0c63000640 1 --2- 172.21.15.32:0/2033997544 >> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] conn(0x55c1308de870 0x55c1308e0c90 crc :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:57:43.709 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:43.707+0000 7f0c63000640 1 -- 172.21.15.32:0/2033997544 <== osd.1 v2:172.21.15.32:6824/1551241759 1 ==== osd_op_reply(59 mds0_openfiles.0 [create] v107'102 uv101 ondisk = -17 ((17) File exists)) ==== 160+0+0 (crc 0 0 0) 0x7f0c5000f400 con 0x55c1308de870 2024-08-03T20:57:43.709 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:43.707+0000 7f0c6a88ef40 1 -- 172.21.15.32:0/2033997544 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:60 12.6 12:654134d2:::mds0_openfiles.0:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e107) -- 0x55c1308e11d0 con 0x55c1308de870 2024-08-03T20:57:43.710 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:43.709+0000 7f0c63000640 1 -- 172.21.15.32:0/2033997544 <== osd.1 v2:172.21.15.32:6824/1551241759 2 ==== osd_op_reply(60 mds0_openfiles.0 [delete] v107'103 uv103 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f0c5000f400 con 0x55c1308de870 2024-08-03T20:57:43.710 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:43.709+0000 7f0c6a88ef40 1 -- 172.21.15.32:0/2033997544 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:61 12.6 12:654134d2:::mds0_openfiles.0:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107) -- 0x55c1308e15f0 con 0x55c1308de870 2024-08-03T20:57:43.711 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:43.710+0000 7f0c63000640 1 -- 172.21.15.32:0/2033997544 <== osd.1 v2:172.21.15.32:6824/1551241759 3 ==== osd_op_reply(61 mds0_openfiles.0 [create] v107'104 uv104 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f0c5000f400 con 0x55c1308de870 2024-08-03T20:57:43.711 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds0_openfiles.0:head# 2024-08-03T20:57:43.712 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:43.710+0000 7f0c6a88ef40 1 -- 172.21.15.32:0/2033997544 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:62 12.6 12:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107) -- 0x55c1308e15f0 con 0x55c1308de870 2024-08-03T20:57:43.712 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:43.711+0000 7f0c63000640 1 -- 172.21.15.32:0/2033997544 <== osd.1 v2:172.21.15.32:6824/1551241759 4 ==== osd_op_reply(62 mds0_openfiles.0 [omap-set-header] v107'105 uv105 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f0c5000f400 con 0x55c1308de870 2024-08-03T20:57:43.712 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:43.711+0000 7f0c6a88ef40 1 -- 172.21.15.32:0/2033997544 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:63 12.e 12:717a0223:::608.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107) -- 0x55c1308e15f0 con 0x55c1307a7d60 2024-08-03T20:57:43.714 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:43.713+0000 7f0c63000640 1 -- 172.21.15.32:0/2033997544 <== osd.0 v2:172.21.15.32:6816/3066156116 38 ==== osd_op_reply(63 608.00000000 [create] v107'129 uv128 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f0c5000f400 con 0x55c1307a7d60 2024-08-03T20:57:43.714 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:43.713+0000 7f0c6a88ef40 1 -- 172.21.15.32:0/2033997544 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:64 12.e 12:717a0223:::608.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e107) -- 0x55c1308e15f0 con 0x55c1307a7d60 2024-08-03T20:57:43.715 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:43.714+0000 7f0c63000640 1 -- 172.21.15.32:0/2033997544 <== osd.0 v2:172.21.15.32:6816/3066156116 39 ==== osd_op_reply(64 608.00000000 [delete] v107'130 uv130 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f0c5000f400 con 0x55c1307a7d60 2024-08-03T20:57:43.716 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:43.714+0000 7f0c6a88ef40 1 -- 172.21.15.32:0/2033997544 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:65 12.e 12:717a0223:::608.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107) -- 0x55c1308e15f0 con 0x55c1307a7d60 2024-08-03T20:57:43.717 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::608.00000000:head# 2024-08-03T20:57:43.717 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:43.716+0000 7f0c63000640 1 -- 172.21.15.32:0/2033997544 <== osd.0 v2:172.21.15.32:6816/3066156116 40 ==== osd_op_reply(65 608.00000000 [create] v107'131 uv131 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f0c5000f400 con 0x55c1307a7d60 2024-08-03T20:57:43.717 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:43.716+0000 7f0c6a88ef40 1 -- 172.21.15.32:0/2033997544 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:66 12.e 12:717a0223:::608.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107) -- 0x55c1308e2600 con 0x55c1307a7d60 2024-08-03T20:57:43.719 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:43.717+0000 7f0c63000640 1 -- 172.21.15.32:0/2033997544 <== osd.0 v2:172.21.15.32:6816/3066156116 41 ==== osd_op_reply(66 608.00000000 [setxattr (30)] v107'132 uv132 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f0c5000f400 con 0x55c1307a7d60 2024-08-03T20:57:43.719 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:43.717+0000 7f0c6a88ef40 1 -- 172.21.15.32:0/2033997544 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:67 12.e 12:717a0223:::608.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107) -- 0x55c1308e2600 con 0x55c1307a7d60 2024-08-03T20:57:43.720 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:43.719+0000 7f0c63000640 1 -- 172.21.15.32:0/2033997544 <== osd.0 v2:172.21.15.32:6816/3066156116 42 ==== osd_op_reply(67 608.00000000 [setxattr (62)] v107'133 uv133 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f0c5000f400 con 0x55c1307a7d60 2024-08-03T20:57:43.720 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:43.719+0000 7f0c6a88ef40 1 -- 172.21.15.32:0/2033997544 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:68 12.e 12:717a0223:::608.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107) -- 0x55c1308e15f0 con 0x55c1307a7d60 2024-08-03T20:57:43.722 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:43.720+0000 7f0c63000640 1 -- 172.21.15.32:0/2033997544 <== osd.0 v2:172.21.15.32:6816/3066156116 43 ==== osd_op_reply(68 608.00000000 [omap-set-header] v107'134 uv134 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f0c5000f400 con 0x55c1307a7d60 2024-08-03T20:57:43.722 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:43.720+0000 7f0c6a88ef40 1 --2- 172.21.15.32:0/2033997544 >> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] conn(0x55c1308e1c70 0x55c1308e4090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:43.722 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:43.720+0000 7f0c6a88ef40 1 -- 172.21.15.32:0/2033997544 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:69 12.15 12:a93a17c2:::604.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107) -- 0x55c1308e45d0 con 0x55c1308e1c70 2024-08-03T20:57:43.722 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:43.721+0000 7f0c64400640 1 --2- 172.21.15.32:0/2033997544 >> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] conn(0x55c1308e1c70 0x55c1308e4090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:43.723 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:43.721+0000 7f0c64400640 1 --2- 172.21.15.32:0/2033997544 >> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] conn(0x55c1308e1c70 0x55c1308e4090 crc :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:57:43.724 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:43.723+0000 7f0c64400640 1 -- 172.21.15.32:0/2033997544 <== osd.6 v2:172.21.15.118:6825/801947386 1 ==== osd_op_reply(69 604.00000000 [create] v107'238 uv232 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f0c5c070070 con 0x55c1308e1c70 2024-08-03T20:57:43.724 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:43.723+0000 7f0c6a88ef40 1 -- 172.21.15.32:0/2033997544 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:70 12.15 12:a93a17c2:::604.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e107) -- 0x55c1308e4a40 con 0x55c1308e1c70 2024-08-03T20:57:43.726 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:43.724+0000 7f0c64400640 1 -- 172.21.15.32:0/2033997544 <== osd.6 v2:172.21.15.118:6825/801947386 2 ==== osd_op_reply(70 604.00000000 [delete] v107'239 uv239 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f0c5c070070 con 0x55c1308e1c70 2024-08-03T20:57:43.726 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:43.724+0000 7f0c6a88ef40 1 -- 172.21.15.32:0/2033997544 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:71 12.15 12:a93a17c2:::604.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107) -- 0x55c1308e4f20 con 0x55c1308e1c70 2024-08-03T20:57:43.727 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:43.726+0000 7f0c64400640 1 -- 172.21.15.32:0/2033997544 <== osd.6 v2:172.21.15.118:6825/801947386 3 ==== osd_op_reply(71 604.00000000 [create] v107'240 uv240 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f0c5c070070 con 0x55c1308e1c70 2024-08-03T20:57:43.728 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::604.00000000:head# 2024-08-03T20:57:43.728 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:43.726+0000 7f0c6a88ef40 1 -- 172.21.15.32:0/2033997544 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:72 12.15 12:a93a17c2:::604.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107) -- 0x55c1308e67d0 con 0x55c1308e1c70 2024-08-03T20:57:43.729 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:43.727+0000 7f0c64400640 1 -- 172.21.15.32:0/2033997544 <== osd.6 v2:172.21.15.118:6825/801947386 4 ==== osd_op_reply(72 604.00000000 [setxattr (30)] v107'241 uv241 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f0c5c070070 con 0x55c1308e1c70 2024-08-03T20:57:43.729 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:43.727+0000 7f0c6a88ef40 1 -- 172.21.15.32:0/2033997544 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:73 12.15 12:a93a17c2:::604.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107) -- 0x55c1308e6b70 con 0x55c1308e1c70 2024-08-03T20:57:43.730 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:43.729+0000 7f0c64400640 1 -- 172.21.15.32:0/2033997544 <== osd.6 v2:172.21.15.118:6825/801947386 5 ==== osd_op_reply(73 604.00000000 [setxattr (62)] v107'242 uv242 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f0c5c070070 con 0x55c1308e1c70 2024-08-03T20:57:43.730 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:43.729+0000 7f0c6a88ef40 1 -- 172.21.15.32:0/2033997544 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:74 12.15 12:a93a17c2:::604.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107) -- 0x55c1308e57c0 con 0x55c1308e1c70 2024-08-03T20:57:43.732 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:43.731+0000 7f0c64400640 1 -- 172.21.15.32:0/2033997544 <== osd.6 v2:172.21.15.118:6825/801947386 6 ==== osd_op_reply(74 604.00000000 [omap-set-header] v107'243 uv243 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f0c5c070070 con 0x55c1308e1c70 2024-08-03T20:57:43.732 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:43.731+0000 7f0c6a88ef40 1 -- 172.21.15.32:0/2033997544 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:75 12.15 12:aa448500:::500.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107) -- 0x55c1308e5bb0 con 0x55c1308e1c70 2024-08-03T20:57:43.734 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:43.732+0000 7f0c64400640 1 -- 172.21.15.32:0/2033997544 <== osd.6 v2:172.21.15.118:6825/801947386 7 ==== osd_op_reply(75 500.00000000 [create] v107'244 uv237 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f0c5c070070 con 0x55c1308e1c70 2024-08-03T20:57:43.734 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:43.732+0000 7f0c6a88ef40 1 -- 172.21.15.32:0/2033997544 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:76 12.15 12:aa448500:::500.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e107) -- 0x55c1308e5bb0 con 0x55c1308e1c70 2024-08-03T20:57:43.735 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:43.734+0000 7f0c64400640 1 -- 172.21.15.32:0/2033997544 <== osd.6 v2:172.21.15.118:6825/801947386 8 ==== osd_op_reply(76 500.00000000 [delete] v107'245 uv245 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f0c5c070070 con 0x55c1308e1c70 2024-08-03T20:57:43.735 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:43.734+0000 7f0c6a88ef40 1 -- 172.21.15.32:0/2033997544 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:77 12.15 12:aa448500:::500.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107) -- 0x55c1308e5bb0 con 0x55c1308e1c70 2024-08-03T20:57:43.737 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:43.736+0000 7f0c64400640 1 -- 172.21.15.32:0/2033997544 <== osd.6 v2:172.21.15.118:6825/801947386 9 ==== osd_op_reply(77 500.00000000 [create] v107'246 uv246 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f0c5c070070 con 0x55c1308e1c70 2024-08-03T20:57:43.737 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::500.00000000:head# 2024-08-03T20:57:43.737 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:43.736+0000 7f0c6a88ef40 1 -- 172.21.15.32:0/2033997544 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:78 12.15 12:aa448500:::500.00000000:head [write 0~90 in=90b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107) -- 0x55c1308e5bb0 con 0x55c1308e1c70 2024-08-03T20:57:43.738 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:43.737+0000 7f0c64400640 1 -- 172.21.15.32:0/2033997544 <== osd.6 v2:172.21.15.118:6825/801947386 10 ==== osd_op_reply(78 500.00000000 [write 0~90] v107'247 uv247 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f0c5c070070 con 0x55c1308e1c70 2024-08-03T20:57:43.739 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:43.737+0000 7f0c6a88ef40 1 -- 172.21.15.32:0/2033997544 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:79 12.15 12:aa448500:::500.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107) -- 0x55c1308e5bb0 con 0x55c1308e1c70 2024-08-03T20:57:43.740 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:43.739+0000 7f0c64400640 1 -- 172.21.15.32:0/2033997544 <== osd.6 v2:172.21.15.118:6825/801947386 11 ==== osd_op_reply(79 500.00000000 [omap-set-header] v107'248 uv248 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f0c5c070070 con 0x55c1308e1c70 2024-08-03T20:57:43.740 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:43.739+0000 7f0c6a88ef40 1 --2- 172.21.15.32:0/2033997544 >> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] conn(0x55c1308e6f10 0x55c1308e9330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:43.740 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:43.739+0000 7f0c6a88ef40 1 -- 172.21.15.32:0/2033997544 --> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] -- osd_op(unknown.0.0:80 12.d 12:b50e409b:::mds_snaptable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107) -- 0x55c1308e5bb0 con 0x55c1308e6f10 2024-08-03T20:57:43.740 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:43.739+0000 7f0c63a00640 1 --2- 172.21.15.32:0/2033997544 >> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] conn(0x55c1308e6f10 0x55c1308e9330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:43.741 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:43.740+0000 7f0c63a00640 1 --2- 172.21.15.32:0/2033997544 >> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] conn(0x55c1308e6f10 0x55c1308e9330 crc :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:57:43.743 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:43.741+0000 7f0c63a00640 1 -- 172.21.15.32:0/2033997544 <== osd.7 v2:172.21.15.118:6818/653000048 1 ==== osd_op_reply(80 mds_snaptable [create] v107'107 uv106 ondisk = -17 ((17) File exists)) ==== 157+0+0 (crc 0 0 0) 0x7f0c5400a460 con 0x55c1308e6f10 2024-08-03T20:57:43.743 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:43.741+0000 7f0c6a88ef40 1 -- 172.21.15.32:0/2033997544 --> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] -- osd_op(unknown.0.0:81 12.d 12:b50e409b:::mds_snaptable:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e107) -- 0x55c1308e5bb0 con 0x55c1308e6f10 2024-08-03T20:57:43.744 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:43.743+0000 7f0c63a00640 1 -- 172.21.15.32:0/2033997544 <== osd.7 v2:172.21.15.118:6818/653000048 2 ==== osd_op_reply(81 mds_snaptable [delete] v107'108 uv108 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f0c5400a460 con 0x55c1308e6f10 2024-08-03T20:57:43.744 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:43.743+0000 7f0c6a88ef40 1 -- 172.21.15.32:0/2033997544 --> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] -- osd_op(unknown.0.0:82 12.d 12:b50e409b:::mds_snaptable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107) -- 0x55c1308e9870 con 0x55c1308e6f10 2024-08-03T20:57:43.745 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:43.744+0000 7f0c63a00640 1 -- 172.21.15.32:0/2033997544 <== osd.7 v2:172.21.15.118:6818/653000048 3 ==== osd_op_reply(82 mds_snaptable [create] v107'109 uv109 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f0c5400a460 con 0x55c1308e6f10 2024-08-03T20:57:43.746 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds_snaptable:head# 2024-08-03T20:57:43.746 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:43.744+0000 7f0c6a88ef40 1 -- 172.21.15.32:0/2033997544 --> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] -- osd_op(unknown.0.0:83 12.d 12:b50e409b:::mds_snaptable:head [write 0~70 in=70b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107) -- 0x55c1308e9870 con 0x55c1308e6f10 2024-08-03T20:57:43.747 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:43.746+0000 7f0c63a00640 1 -- 172.21.15.32:0/2033997544 <== osd.7 v2:172.21.15.118:6818/653000048 4 ==== osd_op_reply(83 mds_snaptable [write 0~70] v107'110 uv110 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f0c5400a460 con 0x55c1308e6f10 2024-08-03T20:57:43.747 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:43.746+0000 7f0c6a88ef40 1 -- 172.21.15.32:0/2033997544 --> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] -- osd_op(unknown.0.0:84 12.d 12:b50e409b:::mds_snaptable:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107) -- 0x55c1308e9870 con 0x55c1308e6f10 2024-08-03T20:57:43.748 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:43.747+0000 7f0c63a00640 1 -- 172.21.15.32:0/2033997544 <== osd.7 v2:172.21.15.118:6818/653000048 5 ==== osd_op_reply(84 mds_snaptable [omap-set-header] v107'111 uv111 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f0c5400a460 con 0x55c1308e6f10 2024-08-03T20:57:43.748 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:43.747+0000 7f0c6a88ef40 1 -- 172.21.15.32:0/2033997544 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:85 12.1d 12:b871830b:::605.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107) -- 0x55c1308e9870 con 0x55c1307a7d60 2024-08-03T20:57:43.750 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:43.748+0000 7f0c63000640 1 -- 172.21.15.32:0/2033997544 <== osd.0 v2:172.21.15.32:6816/3066156116 44 ==== osd_op_reply(85 605.00000000 [create] v107'129 uv128 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f0c5000f400 con 0x55c1307a7d60 2024-08-03T20:57:43.750 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:43.748+0000 7f0c6a88ef40 1 -- 172.21.15.32:0/2033997544 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:86 12.1d 12:b871830b:::605.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e107) -- 0x55c1308e9870 con 0x55c1307a7d60 2024-08-03T20:57:43.751 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:43.750+0000 7f0c63000640 1 -- 172.21.15.32:0/2033997544 <== osd.0 v2:172.21.15.32:6816/3066156116 45 ==== osd_op_reply(86 605.00000000 [delete] v107'130 uv130 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f0c5000f400 con 0x55c1307a7d60 2024-08-03T20:57:43.751 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:43.750+0000 7f0c6a88ef40 1 -- 172.21.15.32:0/2033997544 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:87 12.1d 12:b871830b:::605.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107) -- 0x55c1308e9870 con 0x55c1307a7d60 2024-08-03T20:57:43.752 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:43.751+0000 7f0c63000640 1 -- 172.21.15.32:0/2033997544 <== osd.0 v2:172.21.15.32:6816/3066156116 46 ==== osd_op_reply(87 605.00000000 [create] v107'131 uv131 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f0c5000f400 con 0x55c1307a7d60 2024-08-03T20:57:43.753 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::605.00000000:head# 2024-08-03T20:57:43.753 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:43.751+0000 7f0c6a88ef40 1 -- 172.21.15.32:0/2033997544 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:88 12.1d 12:b871830b:::605.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107) -- 0x55c1308ea880 con 0x55c1307a7d60 2024-08-03T20:57:43.754 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:43.752+0000 7f0c63000640 1 -- 172.21.15.32:0/2033997544 <== osd.0 v2:172.21.15.32:6816/3066156116 47 ==== osd_op_reply(88 605.00000000 [setxattr (30)] v107'132 uv132 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f0c5000f400 con 0x55c1307a7d60 2024-08-03T20:57:43.754 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:43.752+0000 7f0c6a88ef40 1 -- 172.21.15.32:0/2033997544 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:89 12.1d 12:b871830b:::605.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107) -- 0x55c1308ea880 con 0x55c1307a7d60 2024-08-03T20:57:43.755 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:43.754+0000 7f0c63000640 1 -- 172.21.15.32:0/2033997544 <== osd.0 v2:172.21.15.32:6816/3066156116 48 ==== osd_op_reply(89 605.00000000 [setxattr (62)] v107'133 uv133 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f0c5000f400 con 0x55c1307a7d60 2024-08-03T20:57:43.755 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:43.754+0000 7f0c6a88ef40 1 -- 172.21.15.32:0/2033997544 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:90 12.1d 12:b871830b:::605.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107) -- 0x55c1308e9870 con 0x55c1307a7d60 2024-08-03T20:57:43.756 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:43.755+0000 7f0c63000640 1 -- 172.21.15.32:0/2033997544 <== osd.0 v2:172.21.15.32:6816/3066156116 49 ==== osd_op_reply(90 605.00000000 [omap-set-header] v107'134 uv134 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f0c5000f400 con 0x55c1307a7d60 2024-08-03T20:57:43.756 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:43.755+0000 7f0c6a88ef40 1 -- 172.21.15.32:0/2033997544 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:91 12.13 12:c91d4a1d:::mds0_inotable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107) -- 0x55c1308e9870 con 0x55c1308de870 2024-08-03T20:57:43.758 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:43.757+0000 7f0c63000640 1 -- 172.21.15.32:0/2033997544 <== osd.1 v2:172.21.15.32:6824/1551241759 5 ==== osd_op_reply(91 mds0_inotable [create] v107'213 uv207 ondisk = -17 ((17) File exists)) ==== 157+0+0 (crc 0 0 0) 0x7f0c5000f400 con 0x55c1308de870 2024-08-03T20:57:43.758 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:43.757+0000 7f0c6a88ef40 1 -- 172.21.15.32:0/2033997544 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:92 12.13 12:c91d4a1d:::mds0_inotable:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e107) -- 0x55c1308e9870 con 0x55c1308de870 2024-08-03T20:57:43.759 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:43.758+0000 7f0c63000640 1 -- 172.21.15.32:0/2033997544 <== osd.1 v2:172.21.15.32:6824/1551241759 6 ==== osd_op_reply(92 mds0_inotable [delete] v107'214 uv214 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f0c5000f400 con 0x55c1308de870 2024-08-03T20:57:43.759 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:43.758+0000 7f0c6a88ef40 1 -- 172.21.15.32:0/2033997544 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:93 12.13 12:c91d4a1d:::mds0_inotable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107) -- 0x55c1308e9870 con 0x55c1308de870 2024-08-03T20:57:43.760 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds0_inotable:head# 2024-08-03T20:57:43.761 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:43.759+0000 7f0c63000640 1 -- 172.21.15.32:0/2033997544 <== osd.1 v2:172.21.15.32:6824/1551241759 7 ==== osd_op_reply(93 mds0_inotable [create] v107'215 uv215 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f0c5000f400 con 0x55c1308de870 2024-08-03T20:57:43.761 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:43.759+0000 7f0c6a88ef40 1 -- 172.21.15.32:0/2033997544 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:94 12.13 12:c91d4a1d:::mds0_inotable:head [write 0~34 in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107) -- 0x55c1308e53b0 con 0x55c1308de870 2024-08-03T20:57:43.762 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:43.761+0000 7f0c63000640 1 -- 172.21.15.32:0/2033997544 <== osd.1 v2:172.21.15.32:6824/1551241759 8 ==== osd_op_reply(94 mds0_inotable [write 0~34] v107'216 uv216 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f0c5000f400 con 0x55c1308de870 2024-08-03T20:57:43.762 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:43.761+0000 7f0c6a88ef40 1 -- 172.21.15.32:0/2033997544 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:95 12.13 12:c91d4a1d:::mds0_inotable:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107) -- 0x55c1308e53b0 con 0x55c1308de870 2024-08-03T20:57:43.763 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:43.762+0000 7f0c63000640 1 -- 172.21.15.32:0/2033997544 <== osd.1 v2:172.21.15.32:6824/1551241759 9 ==== osd_op_reply(95 mds0_inotable [omap-set-header] v107'217 uv217 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f0c5000f400 con 0x55c1308de870 2024-08-03T20:57:43.763 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:43.762+0000 7f0c6a88ef40 1 -- 172.21.15.32:0/2033997544 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:96 12.13 12:cd5a64a3:::100.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107) -- 0x55c1308e53b0 con 0x55c1308de870 2024-08-03T20:57:43.764 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:43.763+0000 7f0c63000640 1 -- 172.21.15.32:0/2033997544 <== osd.1 v2:172.21.15.32:6824/1551241759 10 ==== osd_op_reply(96 100.00000000 [create] v107'218 uv212 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f0c5000f400 con 0x55c1308de870 2024-08-03T20:57:43.765 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:43.763+0000 7f0c6a88ef40 1 -- 172.21.15.32:0/2033997544 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:97 12.13 12:cd5a64a3:::100.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e107) -- 0x55c1308e53b0 con 0x55c1308de870 2024-08-03T20:57:43.766 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:43.764+0000 7f0c63000640 1 -- 172.21.15.32:0/2033997544 <== osd.1 v2:172.21.15.32:6824/1551241759 11 ==== osd_op_reply(97 100.00000000 [delete] v107'219 uv219 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f0c5000f400 con 0x55c1308de870 2024-08-03T20:57:43.766 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:43.764+0000 7f0c6a88ef40 1 -- 172.21.15.32:0/2033997544 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:98 12.13 12:cd5a64a3:::100.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107) -- 0x55c1308e53b0 con 0x55c1308de870 2024-08-03T20:57:43.767 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:43.766+0000 7f0c63000640 1 -- 172.21.15.32:0/2033997544 <== osd.1 v2:172.21.15.32:6824/1551241759 12 ==== osd_op_reply(98 100.00000000 [create] v107'220 uv220 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f0c5000f400 con 0x55c1308de870 2024-08-03T20:57:43.767 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::100.00000000:head# 2024-08-03T20:57:43.768 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:43.766+0000 7f0c6a88ef40 1 -- 172.21.15.32:0/2033997544 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:99 12.13 12:cd5a64a3:::100.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107) -- 0x55c1308e53b0 con 0x55c1308de870 2024-08-03T20:57:43.768 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:43.767+0000 7f0c63000640 1 -- 172.21.15.32:0/2033997544 <== osd.1 v2:172.21.15.32:6824/1551241759 13 ==== osd_op_reply(99 100.00000000 [omap-set-header] v107'221 uv221 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f0c5000f400 con 0x55c1308de870 2024-08-03T20:57:43.768 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:43.767+0000 7f0c6a88ef40 1 -- 172.21.15.32:0/2033997544 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:100 12.13 12:cd5a64a3:::100.00000000:head [omap-set-vals in=5194b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107) -- 0x55c1308ec680 con 0x55c1308de870 2024-08-03T20:57:43.770 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:43.768+0000 7f0c63000640 1 -- 172.21.15.32:0/2033997544 <== osd.1 v2:172.21.15.32:6824/1551241759 14 ==== osd_op_reply(100 100.00000000 [omap-set-vals] v107'222 uv222 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f0c5000f400 con 0x55c1308de870 2024-08-03T20:57:43.770 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:43.768+0000 7f0c6a88ef40 1 --2- 172.21.15.32:0/2033997544 >> [v2:172.21.15.32:6800/9088303,v1:172.21.15.32:6801/9088303] conn(0x55c1308ea3a0 0x55c1308ecd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:43.770 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:43.768+0000 7f0c6a88ef40 1 -- 172.21.15.32:0/2033997544 --> [v2:172.21.15.32:6800/9088303,v1:172.21.15.32:6801/9088303] -- osd_op(unknown.0.0:101 12.b 12:d0630e4c:::mds0_sessionmap:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107) -- 0x55c1308ec7c0 con 0x55c1308ea3a0 2024-08-03T20:57:43.770 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:43.769+0000 7f0c63000640 1 --2- 172.21.15.32:0/2033997544 >> [v2:172.21.15.32:6800/9088303,v1:172.21.15.32:6801/9088303] conn(0x55c1308ea3a0 0x55c1308ecd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:43.771 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:43.769+0000 7f0c63000640 1 --2- 172.21.15.32:0/2033997544 >> [v2:172.21.15.32:6800/9088303,v1:172.21.15.32:6801/9088303] conn(0x55c1308ea3a0 0x55c1308ecd00 crc :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:57:43.772 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:43.771+0000 7f0c63000640 1 -- 172.21.15.32:0/2033997544 <== osd.3 v2:172.21.15.32:6800/9088303 1 ==== osd_op_reply(101 mds0_sessionmap [create] v107'95 uv94 ondisk = -17 ((17) File exists)) ==== 159+0+0 (crc 0 0 0) 0x7f0c5000f400 con 0x55c1308ea3a0 2024-08-03T20:57:43.772 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:43.771+0000 7f0c6a88ef40 1 -- 172.21.15.32:0/2033997544 --> [v2:172.21.15.32:6800/9088303,v1:172.21.15.32:6801/9088303] -- osd_op(unknown.0.0:102 12.b 12:d0630e4c:::mds0_sessionmap:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e107) -- 0x55c1308ec7c0 con 0x55c1308ea3a0 2024-08-03T20:57:43.773 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:43.772+0000 7f0c63000640 1 -- 172.21.15.32:0/2033997544 <== osd.3 v2:172.21.15.32:6800/9088303 2 ==== osd_op_reply(102 mds0_sessionmap [delete] v107'96 uv96 ondisk = 0) ==== 159+0+0 (crc 0 0 0) 0x7f0c5000f400 con 0x55c1308ea3a0 2024-08-03T20:57:43.773 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:43.772+0000 7f0c6a88ef40 1 -- 172.21.15.32:0/2033997544 --> [v2:172.21.15.32:6800/9088303,v1:172.21.15.32:6801/9088303] -- osd_op(unknown.0.0:103 12.b 12:d0630e4c:::mds0_sessionmap:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107) -- 0x55c1308ec7c0 con 0x55c1308ea3a0 2024-08-03T20:57:43.774 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds0_sessionmap:head# 2024-08-03T20:57:43.775 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:43.773+0000 7f0c63000640 1 -- 172.21.15.32:0/2033997544 <== osd.3 v2:172.21.15.32:6800/9088303 3 ==== osd_op_reply(103 mds0_sessionmap [create] v107'97 uv97 ondisk = 0) ==== 159+0+0 (crc 0 0 0) 0x7f0c5000f400 con 0x55c1308ea3a0 2024-08-03T20:57:43.775 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:43.773+0000 7f0c6a88ef40 1 -- 172.21.15.32:0/2033997544 --> [v2:172.21.15.32:6800/9088303,v1:172.21.15.32:6801/9088303] -- osd_op(unknown.0.0:104 12.b 12:d0630e4c:::mds0_sessionmap:head [omap-set-header in=14b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107) -- 0x55c1308ec7c0 con 0x55c1308ea3a0 2024-08-03T20:57:43.776 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:43.775+0000 7f0c63000640 1 -- 172.21.15.32:0/2033997544 <== osd.3 v2:172.21.15.32:6800/9088303 4 ==== osd_op_reply(104 mds0_sessionmap [omap-set-header] v107'98 uv98 ondisk = 0) ==== 159+0+0 (crc 0 0 0) 0x7f0c5000f400 con 0x55c1308ea3a0 2024-08-03T20:57:43.776 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:43.775+0000 7f0c6a88ef40 1 -- 172.21.15.32:0/2033997544 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:105 12.7 12:e0b41d06:::609.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107) -- 0x55c1308ec7c0 con 0x55c1308de870 2024-08-03T20:57:43.777 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:43.776+0000 7f0c63000640 1 -- 172.21.15.32:0/2033997544 <== osd.1 v2:172.21.15.32:6824/1551241759 15 ==== osd_op_reply(105 609.00000000 [create] v107'129 uv128 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f0c5000f400 con 0x55c1308de870 2024-08-03T20:57:43.778 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:43.776+0000 7f0c6a88ef40 1 -- 172.21.15.32:0/2033997544 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:106 12.7 12:e0b41d06:::609.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e107) -- 0x55c1308ec7c0 con 0x55c1308de870 2024-08-03T20:57:43.779 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:43.778+0000 7f0c63000640 1 -- 172.21.15.32:0/2033997544 <== osd.1 v2:172.21.15.32:6824/1551241759 16 ==== osd_op_reply(106 609.00000000 [delete] v107'130 uv130 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f0c5000f400 con 0x55c1308de870 2024-08-03T20:57:43.779 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:43.778+0000 7f0c6a88ef40 1 -- 172.21.15.32:0/2033997544 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:107 12.7 12:e0b41d06:::609.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107) -- 0x55c1308ec7c0 con 0x55c1308de870 2024-08-03T20:57:43.780 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::609.00000000:head# 2024-08-03T20:57:43.780 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:43.779+0000 7f0c63000640 1 -- 172.21.15.32:0/2033997544 <== osd.1 v2:172.21.15.32:6824/1551241759 17 ==== osd_op_reply(107 609.00000000 [create] v107'131 uv131 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f0c5000f400 con 0x55c1308de870 2024-08-03T20:57:43.781 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:43.779+0000 7f0c6a88ef40 1 -- 172.21.15.32:0/2033997544 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:108 12.7 12:e0b41d06:::609.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107) -- 0x55c1308ec7c0 con 0x55c1308de870 2024-08-03T20:57:43.781 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:43.780+0000 7f0c63000640 1 -- 172.21.15.32:0/2033997544 <== osd.1 v2:172.21.15.32:6824/1551241759 18 ==== osd_op_reply(108 609.00000000 [setxattr (30)] v107'132 uv132 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f0c5000f400 con 0x55c1308de870 2024-08-03T20:57:43.782 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:43.780+0000 7f0c6a88ef40 1 -- 172.21.15.32:0/2033997544 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:109 12.7 12:e0b41d06:::609.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107) -- 0x55c1308ec7c0 con 0x55c1308de870 2024-08-03T20:57:43.783 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:43.782+0000 7f0c63000640 1 -- 172.21.15.32:0/2033997544 <== osd.1 v2:172.21.15.32:6824/1551241759 19 ==== osd_op_reply(109 609.00000000 [setxattr (62)] v107'133 uv133 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f0c5000f400 con 0x55c1308de870 2024-08-03T20:57:43.783 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:43.782+0000 7f0c6a88ef40 1 -- 172.21.15.32:0/2033997544 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:110 12.7 12:e0b41d06:::609.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107) -- 0x55c1308ec7c0 con 0x55c1308de870 2024-08-03T20:57:43.784 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:43.783+0000 7f0c63000640 1 -- 172.21.15.32:0/2033997544 <== osd.1 v2:172.21.15.32:6824/1551241759 20 ==== osd_op_reply(110 609.00000000 [omap-set-header] v107'134 uv134 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f0c5000f400 con 0x55c1308de870 2024-08-03T20:57:43.784 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:43.783+0000 7f0c6a88ef40 1 -- 172.21.15.32:0/2033997544 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:111 12.f 12:f1f69726:::400.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107) -- 0x55c1308ec7c0 con 0x55c1308e1c70 2024-08-03T20:57:43.786 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:43.784+0000 7f0c64400640 1 -- 172.21.15.32:0/2033997544 <== osd.6 v2:172.21.15.118:6825/801947386 12 ==== osd_op_reply(111 400.00000000 [create] v107'110 uv109 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f0c5c070070 con 0x55c1308e1c70 2024-08-03T20:57:43.786 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:43.784+0000 7f0c6a88ef40 1 -- 172.21.15.32:0/2033997544 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:112 12.f 12:f1f69726:::400.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e107) -- 0x55c1308ec7c0 con 0x55c1308e1c70 2024-08-03T20:57:43.787 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:43.786+0000 7f0c64400640 1 -- 172.21.15.32:0/2033997544 <== osd.6 v2:172.21.15.118:6825/801947386 13 ==== osd_op_reply(112 400.00000000 [delete] v107'111 uv111 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f0c5c070070 con 0x55c1308e1c70 2024-08-03T20:57:43.788 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:43.786+0000 7f0c6a88ef40 1 -- 172.21.15.32:0/2033997544 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:113 12.f 12:f1f69726:::400.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107) -- 0x55c1308ec7c0 con 0x55c1308e1c70 2024-08-03T20:57:43.789 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::400.00000000:head# 2024-08-03T20:57:43.789 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:43.787+0000 7f0c64400640 1 -- 172.21.15.32:0/2033997544 <== osd.6 v2:172.21.15.118:6825/801947386 14 ==== osd_op_reply(113 400.00000000 [create] v107'112 uv112 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f0c5c070070 con 0x55c1308e1c70 2024-08-03T20:57:43.789 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:43.788+0000 7f0c6a88ef40 1 -- 172.21.15.32:0/2033997544 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:114 12.f 12:f1f69726:::400.00000000:head [write 0~22 in=22b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107) -- 0x55c1308ec7c0 con 0x55c1308e1c70 2024-08-03T20:57:43.791 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:43.789+0000 7f0c64400640 1 -- 172.21.15.32:0/2033997544 <== osd.6 v2:172.21.15.118:6825/801947386 15 ==== osd_op_reply(114 400.00000000 [write 0~22] v107'113 uv113 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f0c5c070070 con 0x55c1308e1c70 2024-08-03T20:57:43.791 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:43.789+0000 7f0c6a88ef40 1 -- 172.21.15.32:0/2033997544 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:115 12.f 12:f1f69726:::400.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107) -- 0x55c1308ec7c0 con 0x55c1308e1c70 2024-08-03T20:57:43.792 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:43.791+0000 7f0c64400640 1 -- 172.21.15.32:0/2033997544 <== osd.6 v2:172.21.15.118:6825/801947386 16 ==== osd_op_reply(115 400.00000000 [omap-set-header] v107'114 uv114 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f0c5c070070 con 0x55c1308e1c70 2024-08-03T20:57:43.793 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:43.791+0000 7f0c6a88ef40 1 -- 172.21.15.32:0/2033997544 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:116 12.1f 12:fe07bba1:::100.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107) -- 0x55c1308ec7c0 con 0x55c1308e1c70 2024-08-03T20:57:43.794 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:43.793+0000 7f0c64400640 1 -- 172.21.15.32:0/2033997544 <== osd.6 v2:172.21.15.118:6825/801947386 17 ==== osd_op_reply(116 100.00000000.inode [create] v107'265 uv257 ondisk = -17 ((17) File exists)) ==== 162+0+0 (crc 0 0 0) 0x7f0c5c070070 con 0x55c1308e1c70 2024-08-03T20:57:43.794 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:43.793+0000 7f0c6a88ef40 1 -- 172.21.15.32:0/2033997544 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:117 12.1f 12:fe07bba1:::100.00000000.inode:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e107) -- 0x55c1308ec7c0 con 0x55c1308e1c70 2024-08-03T20:57:43.795 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:43.794+0000 7f0c64400640 1 -- 172.21.15.32:0/2033997544 <== osd.6 v2:172.21.15.118:6825/801947386 18 ==== osd_op_reply(117 100.00000000.inode [delete] v107'266 uv266 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7f0c5c070070 con 0x55c1308e1c70 2024-08-03T20:57:43.796 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:43.794+0000 7f0c6a88ef40 1 -- 172.21.15.32:0/2033997544 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:118 12.1f 12:fe07bba1:::100.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107) -- 0x55c1308ec7c0 con 0x55c1308e1c70 2024-08-03T20:57:43.797 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::100.00000000.inode:head# 2024-08-03T20:57:43.798 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:43.796+0000 7f0c64400640 1 -- 172.21.15.32:0/2033997544 <== osd.6 v2:172.21.15.118:6825/801947386 19 ==== osd_op_reply(118 100.00000000.inode [create] v107'267 uv267 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7f0c5c070070 con 0x55c1308e1c70 2024-08-03T20:57:43.798 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:43.796+0000 7f0c6a88ef40 1 -- 172.21.15.32:0/2033997544 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:119 12.1f 12:fe07bba1:::100.00000000.inode:head [write 0~582 in=582b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107) -- 0x55c1308ec7c0 con 0x55c1308e1c70 2024-08-03T20:57:43.799 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:43.797+0000 7f0c64400640 1 -- 172.21.15.32:0/2033997544 <== osd.6 v2:172.21.15.118:6825/801947386 20 ==== osd_op_reply(119 100.00000000.inode [write 0~582] v107'268 uv268 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7f0c5c070070 con 0x55c1308e1c70 2024-08-03T20:57:43.800 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:43.797+0000 7f0c6a88ef40 1 -- 172.21.15.32:0/2033997544 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:120 12.1f 12:fe07bba1:::100.00000000.inode:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107) -- 0x55c1308ec7c0 con 0x55c1308e1c70 2024-08-03T20:57:43.800 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:43.799+0000 7f0c64400640 1 -- 172.21.15.32:0/2033997544 <== osd.6 v2:172.21.15.118:6825/801947386 21 ==== osd_op_reply(120 100.00000000.inode [omap-set-header] v107'269 uv269 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7f0c5c070070 con 0x55c1308e1c70 2024-08-03T20:57:43.800 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:43.799+0000 7f0c6a88ef40 1 -- 172.21.15.32:0/2033997544 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:121 12.1f 12:ff5b34d6:::1.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107) -- 0x55c1308ec7c0 con 0x55c1308e1c70 2024-08-03T20:57:43.801 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:43.800+0000 7f0c64400640 1 -- 172.21.15.32:0/2033997544 <== osd.6 v2:172.21.15.118:6825/801947386 22 ==== osd_op_reply(121 1.00000000 [create] v107'270 uv264 ondisk = -17 ((17) File exists)) ==== 154+0+0 (crc 0 0 0) 0x7f0c5c070070 con 0x55c1308e1c70 2024-08-03T20:57:43.801 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:43.800+0000 7f0c6a88ef40 1 -- 172.21.15.32:0/2033997544 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:122 12.1f 12:ff5b34d6:::1.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e107) -- 0x55c1308ec7c0 con 0x55c1308e1c70 2024-08-03T20:57:43.803 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:43.801+0000 7f0c64400640 1 -- 172.21.15.32:0/2033997544 <== osd.6 v2:172.21.15.118:6825/801947386 23 ==== osd_op_reply(122 1.00000000 [delete] v107'271 uv271 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7f0c5c070070 con 0x55c1308e1c70 2024-08-03T20:57:43.803 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:43.801+0000 7f0c6a88ef40 1 -- 172.21.15.32:0/2033997544 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:123 12.1f 12:ff5b34d6:::1.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107) -- 0x55c1308ec7c0 con 0x55c1308e1c70 2024-08-03T20:57:43.804 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::1.00000000:head# 2024-08-03T20:57:43.805 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:43.803+0000 7f0c64400640 1 -- 172.21.15.32:0/2033997544 <== osd.6 v2:172.21.15.118:6825/801947386 24 ==== osd_op_reply(123 1.00000000 [create] v107'272 uv272 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7f0c5c070070 con 0x55c1308e1c70 2024-08-03T20:57:43.805 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:43.803+0000 7f0c6a88ef40 1 -- 172.21.15.32:0/2033997544 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:124 12.1f 12:ff5b34d6:::1.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107) -- 0x55c1308ec7c0 con 0x55c1308e1c70 2024-08-03T20:57:43.806 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:43.804+0000 7f0c64400640 1 -- 172.21.15.32:0/2033997544 <== osd.6 v2:172.21.15.118:6825/801947386 25 ==== osd_op_reply(124 1.00000000 [setxattr (30)] v107'273 uv273 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7f0c5c070070 con 0x55c1308e1c70 2024-08-03T20:57:43.806 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:43.804+0000 7f0c6a88ef40 1 -- 172.21.15.32:0/2033997544 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:125 12.1f 12:ff5b34d6:::1.00000000:head [setxattr parent (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107) -- 0x55c1308ec7c0 con 0x55c1308e1c70 2024-08-03T20:57:43.807 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:43.805+0000 7f0c64400640 1 -- 172.21.15.32:0/2033997544 <== osd.6 v2:172.21.15.118:6825/801947386 26 ==== osd_op_reply(125 1.00000000 [setxattr (30)] v107'274 uv274 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7f0c5c070070 con 0x55c1308e1c70 2024-08-03T20:57:43.807 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:43.805+0000 7f0c6a88ef40 1 -- 172.21.15.32:0/2033997544 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:126 12.1f 12:ff5b34d6:::1.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107) -- 0x55c1308ec7c0 con 0x55c1308e1c70 2024-08-03T20:57:43.808 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:43.807+0000 7f0c64400640 1 -- 172.21.15.32:0/2033997544 <== osd.6 v2:172.21.15.118:6825/801947386 27 ==== osd_op_reply(126 1.00000000 [omap-set-header] v107'275 uv275 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7f0c5c070070 con 0x55c1308e1c70 2024-08-03T20:57:43.808 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:43.807+0000 7f0c6a88ef40 1 -- 172.21.15.32:0/2033997544 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:127 12.1f 12:ff5b34d6:::1.00000000:head [omap-set-vals in=584b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107) -- 0x55c1308ed6f0 con 0x55c1308e1c70 2024-08-03T20:57:43.811 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:43.808+0000 7f0c64400640 1 -- 172.21.15.32:0/2033997544 <== osd.6 v2:172.21.15.118:6825/801947386 28 ==== osd_op_reply(127 1.00000000 [omap-set-vals] v107'276 uv276 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7f0c5c070070 con 0x55c1308e1c70 2024-08-03T20:57:43.811 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:43.808+0000 7f0c6a88ef40 1 -- 172.21.15.32:0/2033997544 >> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] conn(0x55c1307a7d60 msgr2=0x55c130814190 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:43.811 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:43.808+0000 7f0c6a88ef40 1 --2- 172.21.15.32:0/2033997544 >> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] conn(0x55c1307a7d60 0x55c130814190 crc :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:43.811 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:43.808+0000 7f0c6a88ef40 1 -- 172.21.15.32:0/2033997544 >> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] conn(0x55c1308de870 msgr2=0x55c1308e0c90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:43.811 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:43.808+0000 7f0c6a88ef40 1 --2- 172.21.15.32:0/2033997544 >> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] conn(0x55c1308de870 0x55c1308e0c90 crc :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:43.811 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:43.808+0000 7f0c6a88ef40 1 -- 172.21.15.32:0/2033997544 >> [v2:172.21.15.32:6800/9088303,v1:172.21.15.32:6801/9088303] conn(0x55c1308ea3a0 msgr2=0x55c1308ecd00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:43.811 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:43.808+0000 7f0c6a88ef40 1 --2- 172.21.15.32:0/2033997544 >> [v2:172.21.15.32:6800/9088303,v1:172.21.15.32:6801/9088303] conn(0x55c1308ea3a0 0x55c1308ecd00 crc :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:43.811 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:43.808+0000 7f0c6a88ef40 1 -- 172.21.15.32:0/2033997544 >> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] conn(0x55c1308da0a0 msgr2=0x55c1308dc4c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:43.812 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:43.808+0000 7f0c6a88ef40 1 --2- 172.21.15.32:0/2033997544 >> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] conn(0x55c1308da0a0 0x55c1308dc4c0 crc :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:43.812 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:43.808+0000 7f0c64400640 1 -- 172.21.15.32:0/2033997544 reap_dead start 2024-08-03T20:57:43.812 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:43.808+0000 7f0c6a88ef40 1 -- 172.21.15.32:0/2033997544 >> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] conn(0x55c1308e1c70 msgr2=0x55c1308e4090 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:43.812 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:43.808+0000 7f0c6a88ef40 1 --2- 172.21.15.32:0/2033997544 >> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] conn(0x55c1308e1c70 0x55c1308e4090 crc :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:43.812 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:43.808+0000 7f0c64400640 1 -- 172.21.15.32:0/2033997544 reap_dead start 2024-08-03T20:57:43.812 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:43.808+0000 7f0c6a88ef40 1 -- 172.21.15.32:0/2033997544 >> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] conn(0x55c1308e6f10 msgr2=0x55c1308e9330 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:43.812 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:43.808+0000 7f0c6a88ef40 1 --2- 172.21.15.32:0/2033997544 >> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] conn(0x55c1308e6f10 0x55c1308e9330 crc :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:43.812 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:43.809+0000 7f0c6a88ef40 1 -- 172.21.15.32:0/2033997544 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f0c440759f0 msgr2=0x7f0c44077f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:43.812 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:43.809+0000 7f0c6a88ef40 1 --2- 172.21.15.32:0/2033997544 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f0c440759f0 0x7f0c44077f00 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7f0c5400c6c0 tx=0x7f0c54004a20 comp rx=0 tx=0).stop 2024-08-03T20:57:43.812 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:43.809+0000 7f0c6a88ef40 1 -- 172.21.15.32:0/2033997544 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55c130816be0 msgr2=0x55c1308bcde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:43.812 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:43.809+0000 7f0c6a88ef40 1 --2- 172.21.15.32:0/2033997544 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55c130816be0 0x55c1308bcde0 secure :-1 s=READY pgs=1610 cs=0 l=1 rev1=1 crypto rx=0x7f0c5c025e30 tx=0x7f0c5c04d7f0 comp rx=0 tx=0).stop 2024-08-03T20:57:43.812 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:43.809+0000 7f0c6a88ef40 1 -- 172.21.15.32:0/2033997544 shutdown_connections 2024-08-03T20:57:43.812 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:43.809+0000 7f0c6a88ef40 1 --2- 172.21.15.32:0/2033997544 >> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] conn(0x55c1308e6f10 0x55c1308e9330 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:43.813 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:43.809+0000 7f0c6a88ef40 1 --2- 172.21.15.32:0/2033997544 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f0c440759f0 0x7f0c44077f00 unknown :-1 s=CLOSED pgs=657 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:43.813 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:43.809+0000 7f0c6a88ef40 1 --2- 172.21.15.32:0/2033997544 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55c130816be0 0x55c1308bcde0 unknown :-1 s=CLOSED pgs=1610 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:43.813 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:43.809+0000 7f0c6a88ef40 1 -- 172.21.15.32:0/2033997544 >> 172.21.15.32:0/2033997544 conn(0x55c13067eac0 msgr2=0x55c130836cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:57:43.813 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:43.809+0000 7f0c6a88ef40 1 -- 172.21.15.32:0/2033997544 shutdown_connections 2024-08-03T20:57:43.813 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:43.809+0000 7f0c6a88ef40 1 -- 172.21.15.32:0/2033997544 wait complete. 2024-08-03T20:57:43.823 DEBUG:teuthology.orchestra.run.smithi032:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata put 100.00000000.inode - 2024-08-03T20:57:43.967 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:43.966+0000 7f3c035c5f40 1 Processor -- start 2024-08-03T20:57:43.967 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:43.966+0000 7f3c035c5f40 1 -- start start 2024-08-03T20:57:43.969 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:43.967+0000 7f3c035c5f40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x560ae34e0360 0x560ae34d84a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:43.969 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:43.967+0000 7f3c035c5f40 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x560ae34e1ea0 0x560ae34f8c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:43.969 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:43.967+0000 7f3c035c5f40 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x560ae34f7330 0x560ae34f9180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:43.969 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:43.967+0000 7f3c035c5f40 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x560ae3377e30 con 0x560ae34e1ea0 2024-08-03T20:57:43.970 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:43.967+0000 7f3c035c5f40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x560ae33d5bf0 con 0x560ae34e0360 2024-08-03T20:57:43.970 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:43.967+0000 7f3c035c5f40 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x560ae34e1440 con 0x560ae34f7330 2024-08-03T20:57:43.970 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:43.967+0000 7f3bfbe00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x560ae34e1ea0 0x560ae34f8c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:43.970 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:43.967+0000 7f3bfd200640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x560ae34f7330 0x560ae34f9180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:43.970 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:43.967+0000 7f3bfc800640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x560ae34e0360 0x560ae34d84a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:43.970 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:43.967+0000 7f3bfd200640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x560ae34f7330 0x560ae34f9180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3301/0 says I am v2:172.21.15.32:33584/0 (socket says 172.21.15.32:33584) 2024-08-03T20:57:43.970 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:43.967+0000 7f3bfd200640 1 -- 172.21.15.32:0/90138466 learned_addr learned my addr 172.21.15.32:0/90138466 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:57:43.970 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:43.967+0000 7f3bfbe00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x560ae34e1ea0 0x560ae34f8c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:43192/0 (socket says 172.21.15.32:43192) 2024-08-03T20:57:43.970 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:43.968+0000 7f3bfd200640 1 -- 172.21.15.32:0/90138466 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x560ae34e0360 msgr2=0x560ae34d84a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:57:43.970 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:43.968+0000 7f3bfd200640 1 --2- 172.21.15.32:0/90138466 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x560ae34e0360 0x560ae34d84a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:43.970 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:43.968+0000 7f3bfd200640 1 -- 172.21.15.32:0/90138466 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x560ae34e1ea0 msgr2=0x560ae34f8c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:57:43.970 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:43.968+0000 7f3bfd200640 1 --2- 172.21.15.32:0/90138466 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x560ae34e1ea0 0x560ae34f8c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:43.971 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:43.968+0000 7f3bfd200640 1 -- 172.21.15.32:0/90138466 --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560ae34fc910 con 0x560ae34f7330 2024-08-03T20:57:43.971 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:43.968+0000 7f3bfbe00640 1 --2- 172.21.15.32:0/90138466 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x560ae34e1ea0 0x560ae34f8c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-03T20:57:43.972 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:43.968+0000 7f3bfd200640 1 --2- 172.21.15.32:0/90138466 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x560ae34f7330 0x560ae34f9180 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f3bf40464b0 tx=0x7f3bf4003330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=83a57631b951d163 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:57:43.972 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:43.969+0000 7f3bfaa00640 1 -- 172.21.15.32:0/90138466 <== mon.2 v2:172.21.15.118:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3bf4064020 con 0x560ae34f7330 2024-08-03T20:57:43.972 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:43.969+0000 7f3bfaa00640 1 -- 172.21.15.32:0/90138466 <== mon.2 v2:172.21.15.118:3301/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f3bf404daa0 con 0x560ae34f7330 2024-08-03T20:57:43.972 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:43.969+0000 7f3bfaa00640 1 -- 172.21.15.32:0/90138466 <== mon.2 v2:172.21.15.118:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3bf404d810 con 0x560ae34f7330 2024-08-03T20:57:43.972 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:43.969+0000 7f3c035c5f40 1 -- 172.21.15.32:0/90138466 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x560ae34f7330 msgr2=0x560ae34f9180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:43.972 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:43.969+0000 7f3c035c5f40 1 --2- 172.21.15.32:0/90138466 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x560ae34f7330 0x560ae34f9180 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f3bf40464b0 tx=0x7f3bf4003330 comp rx=0 tx=0).stop 2024-08-03T20:57:43.972 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:43.969+0000 7f3c035c5f40 1 -- 172.21.15.32:0/90138466 shutdown_connections 2024-08-03T20:57:43.972 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:43.969+0000 7f3c035c5f40 1 --2- 172.21.15.32:0/90138466 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x560ae34f7330 0x560ae34f9180 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:43.973 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:43.969+0000 7f3c035c5f40 1 --2- 172.21.15.32:0/90138466 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x560ae34e1ea0 0x560ae34f8c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:43.973 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:43.969+0000 7f3c035c5f40 1 --2- 172.21.15.32:0/90138466 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x560ae34e0360 0x560ae34d84a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:43.973 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:43.969+0000 7f3c035c5f40 1 -- 172.21.15.32:0/90138466 >> 172.21.15.32:0/90138466 conn(0x560ae3349ac0 msgr2=0x560ae34f9e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:57:43.973 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:43.969+0000 7f3c035c5f40 1 -- 172.21.15.32:0/90138466 shutdown_connections 2024-08-03T20:57:43.973 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:43.969+0000 7f3c035c5f40 1 -- 172.21.15.32:0/90138466 wait complete. 2024-08-03T20:57:43.973 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:43.970+0000 7f3c035c5f40 1 Processor -- start 2024-08-03T20:57:43.973 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:43.971+0000 7f3c035c5f40 1 -- start start 2024-08-03T20:57:43.976 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:43.972+0000 7f3c035c5f40 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x560ae34e0360 0x560ae34fdea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:43.977 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:43.972+0000 7f3c035c5f40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x560ae34e1ea0 0x560ae346d1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:43.977 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:43.972+0000 7f3c035c5f40 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x560ae34f7330 0x560ae346d720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:43.977 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:43.972+0000 7f3c035c5f40 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x560ae3429bf0 con 0x560ae34f7330 2024-08-03T20:57:43.977 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:43.972+0000 7f3c035c5f40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x560ae3472c70 con 0x560ae34e1ea0 2024-08-03T20:57:43.977 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:43.972+0000 7f3c035c5f40 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x560ae34d87d0 con 0x560ae34e0360 2024-08-03T20:57:43.978 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:43.972+0000 7f3bfd200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x560ae34f7330 0x560ae346d720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:43.978 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:43.972+0000 7f3bfc800640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x560ae34e0360 0x560ae34fdea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:43.978 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:43.972+0000 7f3bfbe00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x560ae34e1ea0 0x560ae346d1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:43.978 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:43.972+0000 7f3bfc800640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x560ae34e0360 0x560ae34fdea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3301/0 says I am v2:172.21.15.32:33596/0 (socket says 172.21.15.32:33596) 2024-08-03T20:57:43.978 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:43.972+0000 7f3bfc800640 1 -- 172.21.15.32:0/2288404836 learned_addr learned my addr 172.21.15.32:0/2288404836 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:57:43.978 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:43.973+0000 7f3bfc800640 1 -- 172.21.15.32:0/2288404836 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x560ae34e1ea0 msgr2=0x560ae346d1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:43.978 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:43.973+0000 7f3bfc800640 1 --2- 172.21.15.32:0/2288404836 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x560ae34e1ea0 0x560ae346d1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:43.978 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:43.973+0000 7f3bfc800640 1 -- 172.21.15.32:0/2288404836 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x560ae34f7330 msgr2=0x560ae346d720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:43.978 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:43.973+0000 7f3bfc800640 1 --2- 172.21.15.32:0/2288404836 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x560ae34f7330 0x560ae346d720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:43.979 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:43.973+0000 7f3bfc800640 1 -- 172.21.15.32:0/2288404836 --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560ae34287f0 con 0x560ae34e0360 2024-08-03T20:57:43.979 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:43.973+0000 7f3bfd200640 1 --2- 172.21.15.32:0/2288404836 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x560ae34f7330 0x560ae346d720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-03T20:57:43.979 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:43.973+0000 7f3bfc800640 1 --2- 172.21.15.32:0/2288404836 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x560ae34e0360 0x560ae34fdea0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f3bf000ed90 tx=0x7f3bf000c600 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:57:43.979 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:43.973+0000 7f3bf9600640 1 -- 172.21.15.32:0/2288404836 <== mon.2 v2:172.21.15.118:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3bf0001ad0 con 0x560ae34e0360 2024-08-03T20:57:43.979 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:43.973+0000 7f3bf9600640 1 -- 172.21.15.32:0/2288404836 <== mon.2 v2:172.21.15.118:3301/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f3bf0001d60 con 0x560ae34e0360 2024-08-03T20:57:43.979 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:43.973+0000 7f3bf9600640 1 -- 172.21.15.32:0/2288404836 <== mon.2 v2:172.21.15.118:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3bf00022b0 con 0x560ae34e0360 2024-08-03T20:57:43.979 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:43.974+0000 7f3c035c5f40 1 -- 172.21.15.32:0/2288404836 --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x560ae346ab10 con 0x560ae34e0360 2024-08-03T20:57:43.979 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:43.974+0000 7f3c035c5f40 1 -- 172.21.15.32:0/2288404836 --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_subscribe({osdmap=0}) -- 0x560ae34d84a0 con 0x560ae34e0360 2024-08-03T20:57:43.981 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:43.976+0000 7f3bf9600640 1 -- 172.21.15.32:0/2288404836 <== mon.2 v2:172.21.15.118:3301/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f3bf0002450 con 0x560ae34e0360 2024-08-03T20:57:43.981 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:43.977+0000 7f3bf9600640 1 --2- 172.21.15.32:0/2288404836 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f3bdc0759f0 0x7f3bdc077f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:43.981 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:43.977+0000 7f3bf9600640 1 -- 172.21.15.32:0/2288404836 <== mon.2 v2:172.21.15.118:3301/0 5 ==== osd_map(107..107 src has 1..107) ==== 7746+0+0 (secure 0 0 0) 0x7f3bf0098080 con 0x560ae34e0360 2024-08-03T20:57:43.981 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:43.977+0000 7f3bfbe00640 1 --2- 172.21.15.32:0/2288404836 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f3bdc0759f0 0x7f3bdc077f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:43.981 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:43.977+0000 7f3c035c5f40 1 --2- 172.21.15.32:0/2288404836 >> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] conn(0x560ae34dbf70 0x560ae3469580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:43.982 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:43.977+0000 7f3c035c5f40 1 -- 172.21.15.32:0/2288404836 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:1 12.1f 12:fe07bba1:::100.00000000.inode:head [writefull 0~80 in=80b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107) -- 0x560ae34fe3e0 con 0x560ae34dbf70 2024-08-03T20:57:43.982 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:43.978+0000 7f3bfd200640 1 --2- 172.21.15.32:0/2288404836 >> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] conn(0x560ae34dbf70 0x560ae3469580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:43.982 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:43.978+0000 7f3bfbe00640 1 --2- 172.21.15.32:0/2288404836 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f3bdc0759f0 0x7f3bdc077f00 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x560ae3428e60 tx=0x7f3be800d040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:57:43.982 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:43.978+0000 7f3bfd200640 1 --2- 172.21.15.32:0/2288404836 >> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] conn(0x560ae34dbf70 0x560ae3469580 crc :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:57:43.984 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:43.980+0000 7f3bfd200640 1 -- 172.21.15.32:0/2288404836 <== osd.6 v2:172.21.15.118:6825/801947386 1 ==== osd_op_reply(1 100.00000000.inode [writefull 0~80] v107'277 uv277 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7f3bf400b040 con 0x560ae34dbf70 2024-08-03T20:57:43.984 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:43.980+0000 7f3c035c5f40 1 -- 172.21.15.32:0/2288404836 >> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] conn(0x560ae34dbf70 msgr2=0x560ae3469580 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:43.984 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:43.980+0000 7f3c035c5f40 1 --2- 172.21.15.32:0/2288404836 >> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] conn(0x560ae34dbf70 0x560ae3469580 crc :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:43.984 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:43.980+0000 7f3c035c5f40 1 -- 172.21.15.32:0/2288404836 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f3bdc0759f0 msgr2=0x7f3bdc077f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:43.984 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:43.980+0000 7f3c035c5f40 1 --2- 172.21.15.32:0/2288404836 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f3bdc0759f0 0x7f3bdc077f00 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x560ae3428e60 tx=0x7f3be800d040 comp rx=0 tx=0).stop 2024-08-03T20:57:43.984 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:43.980+0000 7f3c035c5f40 1 -- 172.21.15.32:0/2288404836 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x560ae34e0360 msgr2=0x560ae34fdea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:43.984 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:43.980+0000 7f3c035c5f40 1 --2- 172.21.15.32:0/2288404836 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x560ae34e0360 0x560ae34fdea0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f3bf000ed90 tx=0x7f3bf000c600 comp rx=0 tx=0).stop 2024-08-03T20:57:43.984 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:43.980+0000 7f3bfd200640 1 -- 172.21.15.32:0/2288404836 reap_dead start 2024-08-03T20:57:43.984 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:43.980+0000 7f3c035c5f40 1 -- 172.21.15.32:0/2288404836 shutdown_connections 2024-08-03T20:57:43.984 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:43.980+0000 7f3c035c5f40 1 -- 172.21.15.32:0/2288404836 >> 172.21.15.32:0/2288404836 conn(0x560ae3349ac0 msgr2=0x560ae3428eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:57:43.984 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:43.981+0000 7f3c035c5f40 1 -- 172.21.15.32:0/2288404836 shutdown_connections 2024-08-03T20:57:43.984 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:43.981+0000 7f3c035c5f40 1 -- 172.21.15.32:0/2288404836 wait complete. 2024-08-03T20:57:43.996 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs joinable true 2024-08-03T20:57:44.226 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:44.224+0000 7fdb98800640 1 -- 172.21.15.32:0/1072844595 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdb901606f0 msgr2=0x7fdb901450f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:44.227 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:44.224+0000 7fdb98800640 1 --2- 172.21.15.32:0/1072844595 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdb901606f0 0x7fdb901450f0 secure :-1 s=READY pgs=1612 cs=0 l=1 rev1=1 crypto rx=0x7fdb8c005fd0 tx=0x7fdb8c02a8b0 comp rx=0 tx=0).stop 2024-08-03T20:57:44.227 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:44.225+0000 7fdb98800640 1 -- 172.21.15.32:0/1072844595 shutdown_connections 2024-08-03T20:57:44.227 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:44.225+0000 7fdb98800640 1 --2- 172.21.15.32:0/1072844595 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdb901606f0 0x7fdb901450f0 unknown :-1 s=CLOSED pgs=1612 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:44.227 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:44.225+0000 7fdb98800640 1 --2- 172.21.15.32:0/1072844595 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fdb900cecc0 0x7fdb900b4f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:44.227 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:44.225+0000 7fdb98800640 1 -- 172.21.15.32:0/1072844595 >> 172.21.15.32:0/1072844595 conn(0x7fdb900b1bd0 msgr2=0x7fdb900b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:57:44.227 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:44.225+0000 7fdb98800640 1 -- 172.21.15.32:0/1072844595 shutdown_connections 2024-08-03T20:57:44.227 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:44.225+0000 7fdb98800640 1 -- 172.21.15.32:0/1072844595 wait complete. 2024-08-03T20:57:44.228 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:44.227+0000 7fdb98800640 1 Processor -- start 2024-08-03T20:57:44.228 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:44.227+0000 7fdb98800640 1 -- start start 2024-08-03T20:57:44.232 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:44.229+0000 7fdb98800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdb900cd210 0x7fdb900c9d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:44.232 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:44.229+0000 7fdb98800640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fdb900cecc0 0x7fdb901f85b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:44.232 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:44.229+0000 7fdb98800640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fdb900a94b0 0x7fdb901f8af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:44.232 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:44.229+0000 7fdb98800640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fdb90145a30 con 0x7fdb900cd210 2024-08-03T20:57:44.232 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:44.229+0000 7fdb98800640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7fdb901458b0 con 0x7fdb900a94b0 2024-08-03T20:57:44.232 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:44.229+0000 7fdb98800640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7fdb90145bb0 con 0x7fdb900cecc0 2024-08-03T20:57:44.232 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:44.229+0000 7fdb95e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdb900cd210 0x7fdb900c9d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:44.232 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:44.229+0000 7fdb95e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdb900cd210 0x7fdb900c9d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:43230/0 (socket says 172.21.15.32:43230) 2024-08-03T20:57:44.232 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:44.229+0000 7fdb95e00640 1 -- 172.21.15.32:0/626421897 learned_addr learned my addr 172.21.15.32:0/626421897 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:57:44.232 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:44.229+0000 7fdb95400640 1 --2- 172.21.15.32:0/626421897 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fdb900cecc0 0x7fdb901f85b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:44.232 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:44.229+0000 7fdb96800640 1 --2- 172.21.15.32:0/626421897 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fdb900a94b0 0x7fdb901f8af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:44.233 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:44.229+0000 7fdb95e00640 1 -- 172.21.15.32:0/626421897 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fdb900cecc0 msgr2=0x7fdb901f85b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:44.233 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:44.229+0000 7fdb95e00640 1 --2- 172.21.15.32:0/626421897 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fdb900cecc0 0x7fdb901f85b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:44.233 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:44.229+0000 7fdb95e00640 1 -- 172.21.15.32:0/626421897 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fdb900a94b0 msgr2=0x7fdb901f8af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:44.233 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:44.229+0000 7fdb95e00640 1 --2- 172.21.15.32:0/626421897 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fdb900a94b0 0x7fdb901f8af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:44.233 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:44.229+0000 7fdb95e00640 1 -- 172.21.15.32:0/626421897 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdb90158cb0 con 0x7fdb900cd210 2024-08-03T20:57:44.233 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:44.230+0000 7fdb95e00640 1 --2- 172.21.15.32:0/626421897 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdb900cd210 0x7fdb900c9d20 secure :-1 s=READY pgs=1613 cs=0 l=1 rev1=1 crypto rx=0x7fdb8400d740 tx=0x7fdb8400dc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:57:44.233 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:44.230+0000 7fdb8a000640 1 -- 172.21.15.32:0/626421897 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb84017070 con 0x7fdb900cd210 2024-08-03T20:57:44.233 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:44.230+0000 7fdb8a000640 1 -- 172.21.15.32:0/626421897 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fdb84009d20 con 0x7fdb900cd210 2024-08-03T20:57:44.233 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:44.230+0000 7fdb8a000640 1 -- 172.21.15.32:0/626421897 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb840106c0 con 0x7fdb900cd210 2024-08-03T20:57:44.233 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:44.230+0000 7fdb98800640 1 -- 172.21.15.32:0/626421897 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdb90158f90 con 0x7fdb900cd210 2024-08-03T20:57:44.233 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:44.230+0000 7fdb98800640 1 -- 172.21.15.32:0/626421897 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdb90159550 con 0x7fdb900cd210 2024-08-03T20:57:44.234 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:44.231+0000 7fdb98800640 1 -- 172.21.15.32:0/626421897 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdb900b4f10 con 0x7fdb900cd210 2024-08-03T20:57:44.234 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:44.232+0000 7fdb8a000640 1 -- 172.21.15.32:0/626421897 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7fdb8401a070 con 0x7fdb900cd210 2024-08-03T20:57:44.235 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:44.233+0000 7fdb8a000640 1 --2- 172.21.15.32:0/626421897 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fdb74077010 0x7fdb74079520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:44.235 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:44.233+0000 7fdb8a000640 1 -- 172.21.15.32:0/626421897 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(107..107 src has 1..107) ==== 7746+0+0 (secure 0 0 0) 0x7fdb840991e0 con 0x7fdb900cd210 2024-08-03T20:57:44.235 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:44.234+0000 7fdb95400640 1 --2- 172.21.15.32:0/626421897 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fdb74077010 0x7fdb74079520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:44.236 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:44.235+0000 7fdb95400640 1 --2- 172.21.15.32:0/626421897 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fdb74077010 0x7fdb74079520 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7fdb80001150 tx=0x7fdb800010e0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:57:44.239 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:44.238+0000 7fdb8a000640 1 -- 172.21.15.32:0/626421897 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7fdb840614f0 con 0x7fdb900cd210 2024-08-03T20:57:44.487 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:44.485+0000 7fdb98800640 1 -- 172.21.15.32:0/626421897 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "joinable", "val": "true"} v 0) -- 0x7fdb900740b0 con 0x7fdb900cd210 2024-08-03T20:57:44.804 INFO:tasks.ceph.mds.a.smithi032.stderr: -45> 2024-08-03T20:57:40.050+0000 7f6f2a800640 -1 log_channel(cluster) log [ERR] : failed to read JournalPointer: -22 ((22) Invalid argument) 2024-08-03T20:57:44.918 INFO:tasks.ceph.mds.a.smithi032.stdout:starting mds.a at 2024-08-03T20:57:45.046 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:45.045+0000 7fdb8a000640 1 -- 172.21.15.32:0/626421897 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "joinable", "val": "true"}]=0 cephfs marked joinable; MDS may join as newly active. v225) ==== 162+0+0 (secure 0 0 0) 0x7fdb84065a40 con 0x7fdb900cd210 2024-08-03T20:57:45.047 INFO:teuthology.orchestra.run.smithi032.stderr:cephfs marked joinable; MDS may join as newly active. 2024-08-03T20:57:45.050 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:45.048+0000 7fdb98800640 1 -- 172.21.15.32:0/626421897 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fdb74077010 msgr2=0x7fdb74079520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:45.050 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:45.048+0000 7fdb98800640 1 --2- 172.21.15.32:0/626421897 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fdb74077010 0x7fdb74079520 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7fdb80001150 tx=0x7fdb800010e0 comp rx=0 tx=0).stop 2024-08-03T20:57:45.050 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:45.048+0000 7fdb98800640 1 -- 172.21.15.32:0/626421897 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdb900cd210 msgr2=0x7fdb900c9d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:45.050 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:45.048+0000 7fdb98800640 1 --2- 172.21.15.32:0/626421897 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdb900cd210 0x7fdb900c9d20 secure :-1 s=READY pgs=1613 cs=0 l=1 rev1=1 crypto rx=0x7fdb8400d740 tx=0x7fdb8400dc00 comp rx=0 tx=0).stop 2024-08-03T20:57:45.050 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:45.048+0000 7fdb98800640 1 -- 172.21.15.32:0/626421897 shutdown_connections 2024-08-03T20:57:45.050 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:45.048+0000 7fdb98800640 1 --2- 172.21.15.32:0/626421897 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fdb74077010 0x7fdb74079520 unknown :-1 s=CLOSED pgs=659 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:45.050 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:45.048+0000 7fdb98800640 1 --2- 172.21.15.32:0/626421897 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fdb900a94b0 0x7fdb901f8af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:45.050 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:45.048+0000 7fdb98800640 1 --2- 172.21.15.32:0/626421897 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fdb900cecc0 0x7fdb901f85b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:45.051 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:45.048+0000 7fdb98800640 1 --2- 172.21.15.32:0/626421897 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdb900cd210 0x7fdb900c9d20 unknown :-1 s=CLOSED pgs=1613 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:45.051 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:45.048+0000 7fdb98800640 1 -- 172.21.15.32:0/626421897 >> 172.21.15.32:0/626421897 conn(0x7fdb900b1bd0 msgr2=0x7fdb90073240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:57:45.051 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:45.048+0000 7fdb98800640 1 -- 172.21.15.32:0/626421897 shutdown_connections 2024-08-03T20:57:45.051 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:45.048+0000 7fdb98800640 1 -- 172.21.15.32:0/626421897 wait complete. 2024-08-03T20:57:45.061 INFO:tasks.ceph.mds.a.smithi032.stderr:2024-08-03T20:57:45.060+0000 7fac75200640 -1 mds.0.cache.ino(0x100) Corrupt inode 0x100: End of buffer [buffer:2] 2024-08-03T20:57:45.061 INFO:tasks.ceph.mds.a.smithi032.stderr:2024-08-03T20:57:45.060+0000 7fac75200640 -1 log_channel(cluster) log [ERR] : Error loading MDS rank 0: (22) Invalid argument 2024-08-03T20:57:45.062 INFO:tasks.ceph.mon.a.smithi032.stderr:2024-08-03T20:57:45.061+0000 7f4e6d600640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-08-03T20:57:45.062 INFO:tasks.ceph.mon.a.smithi032.stderr:2024-08-03T20:57:45.061+0000 7f4e6d600640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 mds daemon damaged (MDS_DAMAGE) 2024-08-03T20:57:45.081 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-08-03T20:57:45.309 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:45.308+0000 7f4bf4200640 1 -- 172.21.15.32:0/284834315 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4bec15fff0 msgr2=0x7f4bec160400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:45.309 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:45.308+0000 7f4bf4200640 1 --2- 172.21.15.32:0/284834315 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4bec15fff0 0x7f4bec160400 secure :-1 s=READY pgs=1618 cs=0 l=1 rev1=1 crypto rx=0x7f4be4005fd0 tx=0x7f4be402a8b0 comp rx=0 tx=0).stop 2024-08-03T20:57:45.310 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:45.308+0000 7f4bf4200640 1 -- 172.21.15.32:0/284834315 shutdown_connections 2024-08-03T20:57:45.310 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:45.308+0000 7f4bf4200640 1 --2- 172.21.15.32:0/284834315 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4bec15fff0 0x7f4bec160400 unknown :-1 s=CLOSED pgs=1618 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:45.310 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:45.308+0000 7f4bf4200640 1 --2- 172.21.15.32:0/284834315 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f4bec06e930 0x7f4bec144f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:45.310 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:45.308+0000 7f4bf4200640 1 -- 172.21.15.32:0/284834315 >> 172.21.15.32:0/284834315 conn(0x7f4bec0b1bd0 msgr2=0x7f4bec0b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:57:45.310 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:45.308+0000 7f4bf4200640 1 -- 172.21.15.32:0/284834315 shutdown_connections 2024-08-03T20:57:45.310 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:45.308+0000 7f4bf4200640 1 -- 172.21.15.32:0/284834315 wait complete. 2024-08-03T20:57:45.311 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:45.310+0000 7f4bf4200640 1 Processor -- start 2024-08-03T20:57:45.311 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:45.310+0000 7f4bf4200640 1 -- start start 2024-08-03T20:57:45.315 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:45.311+0000 7f4bf4200640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f4bec06e930 0x7f4bec143350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:45.315 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:45.311+0000 7f4bf4200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4bec073e60 0x7f4bec15bb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:45.315 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:45.311+0000 7f4bf4200640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f4bec15fff0 0x7f4bec15f200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:45.315 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:45.311+0000 7f4bf4200640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f4bec0cf840 con 0x7f4bec073e60 2024-08-03T20:57:45.315 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:45.312+0000 7f4bf4200640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f4bec145f30 con 0x7f4bec06e930 2024-08-03T20:57:45.315 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:45.312+0000 7f4bf4200640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7f4bec0cf9c0 con 0x7f4bec15fff0 2024-08-03T20:57:45.315 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:45.312+0000 7f4bf0e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4bec073e60 0x7f4bec15bb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:45.316 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:45.312+0000 7f4bf0e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4bec073e60 0x7f4bec15bb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:43260/0 (socket says 172.21.15.32:43260) 2024-08-03T20:57:45.316 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:45.312+0000 7f4bf0e00640 1 -- 172.21.15.32:0/1673538473 learned_addr learned my addr 172.21.15.32:0/1673538473 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:57:45.316 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:45.312+0000 7f4bf2200640 1 --2- 172.21.15.32:0/1673538473 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f4bec15fff0 0x7f4bec15f200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:45.316 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:45.312+0000 7f4bf1800640 1 --2- 172.21.15.32:0/1673538473 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f4bec06e930 0x7f4bec143350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:45.316 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:45.312+0000 7f4bf0e00640 1 -- 172.21.15.32:0/1673538473 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f4bec15fff0 msgr2=0x7f4bec15f200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:45.316 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:45.312+0000 7f4bf0e00640 1 --2- 172.21.15.32:0/1673538473 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f4bec15fff0 0x7f4bec15f200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:45.316 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:45.312+0000 7f4bf0e00640 1 -- 172.21.15.32:0/1673538473 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f4bec06e930 msgr2=0x7f4bec143350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:45.316 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:45.312+0000 7f4bf0e00640 1 --2- 172.21.15.32:0/1673538473 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f4bec06e930 0x7f4bec143350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:45.316 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:45.312+0000 7f4bf0e00640 1 -- 172.21.15.32:0/1673538473 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4bec143ae0 con 0x7f4bec073e60 2024-08-03T20:57:45.316 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:45.312+0000 7f4bf1800640 1 --2- 172.21.15.32:0/1673538473 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f4bec06e930 0x7f4bec143350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-03T20:57:45.316 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:45.313+0000 7f4bf0e00640 1 --2- 172.21.15.32:0/1673538473 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4bec073e60 0x7f4bec15bb70 secure :-1 s=READY pgs=1619 cs=0 l=1 rev1=1 crypto rx=0x7f4bd800c430 tx=0x7f4bd800c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:57:45.317 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:45.313+0000 7f4be9600640 1 -- 172.21.15.32:0/1673538473 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4bd8011790 con 0x7f4bec073e60 2024-08-03T20:57:45.317 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:45.313+0000 7f4be9600640 1 -- 172.21.15.32:0/1673538473 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f4bd8017070 con 0x7f4bec073e60 2024-08-03T20:57:45.317 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:45.313+0000 7f4be9600640 1 -- 172.21.15.32:0/1673538473 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4bd8011ce0 con 0x7f4bec073e60 2024-08-03T20:57:45.317 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:45.313+0000 7f4bf4200640 1 -- 172.21.15.32:0/1673538473 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4bec1f5610 con 0x7f4bec073e60 2024-08-03T20:57:45.317 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:45.313+0000 7f4bf4200640 1 -- 172.21.15.32:0/1673538473 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4bec1f58b0 con 0x7f4bec073e60 2024-08-03T20:57:45.318 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:45.314+0000 7f4bf4200640 1 -- 172.21.15.32:0/1673538473 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4bec160bc0 con 0x7f4bec073e60 2024-08-03T20:57:45.318 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:45.315+0000 7f4be9600640 1 -- 172.21.15.32:0/1673538473 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f4bd801c460 con 0x7f4bec073e60 2024-08-03T20:57:45.318 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:45.316+0000 7f4be9600640 1 --2- 172.21.15.32:0/1673538473 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f4bcc077040 0x7f4bcc079550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:45.318 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:45.316+0000 7f4be9600640 1 -- 172.21.15.32:0/1673538473 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(108..108 src has 1..108) ==== 7832+0+0 (secure 0 0 0) 0x7f4bd8098fb0 con 0x7f4bec073e60 2024-08-03T20:57:45.318 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:45.317+0000 7f4bf1800640 1 --2- 172.21.15.32:0/1673538473 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f4bcc077040 0x7f4bcc079550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:45.319 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:45.317+0000 7f4bf1800640 1 --2- 172.21.15.32:0/1673538473 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f4bcc077040 0x7f4bcc079550 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7f4be0009b80 tx=0x7f4be0005f90 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:57:45.322 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:45.321+0000 7f4be9600640 1 -- 172.21.15.32:0/1673538473 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7f4bec1f5610 con 0x7f4bec073e60 2024-08-03T20:57:45.566 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:45.565+0000 7f4bf4200640 1 -- 172.21.15.32:0/1673538473 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f4bec070880 con 0x7f4bec073e60 2024-08-03T20:57:45.567 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:45.565+0000 7f4be9600640 1 -- 172.21.15.32:0/1673538473 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 226 v226) ==== 95+0+4391 (secure 0 0 0) 0x7f4bd80618c0 con 0x7f4bec073e60 2024-08-03T20:57:45.568 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-03T20:57:45.568 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":226,"btime":"2024-08-03T20:57:45:062715+0000","default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6439,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6835/531080160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6834","nonce":531080160},{"type":"v1","addr":"172.21.15.118:6835","nonce":531080160}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":84},{"gid":6454,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6837/2796882319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6836","nonce":2796882319},{"type":"v1","addr":"172.21.15.118:6837","nonce":2796882319}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":86},{"gid":6813,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6834/2686933890","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":2686933890},{"type":"v1","addr":"172.21.15.32:6834","nonce":2686933890}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":85}],"filesystems":[{"mdsmap":{"epoch":226,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-08-03T20:54:26.950518+0000","modified":"2024-08-03T20:57:45.062650+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":108,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":6}]} 2024-08-03T20:57:45.568 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 226 2024-08-03T20:57:45.569 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:45.567+0000 7f4bf4200640 1 -- 172.21.15.32:0/1673538473 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f4bcc077040 msgr2=0x7f4bcc079550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:45.569 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:45.568+0000 7f4bf4200640 1 --2- 172.21.15.32:0/1673538473 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f4bcc077040 0x7f4bcc079550 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7f4be0009b80 tx=0x7f4be0005f90 comp rx=0 tx=0).stop 2024-08-03T20:57:45.570 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:45.568+0000 7f4bf4200640 1 -- 172.21.15.32:0/1673538473 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4bec073e60 msgr2=0x7f4bec15bb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:45.570 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:45.568+0000 7f4bf4200640 1 --2- 172.21.15.32:0/1673538473 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4bec073e60 0x7f4bec15bb70 secure :-1 s=READY pgs=1619 cs=0 l=1 rev1=1 crypto rx=0x7f4bd800c430 tx=0x7f4bd800c8f0 comp rx=0 tx=0).stop 2024-08-03T20:57:45.570 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:45.568+0000 7f4bf4200640 1 -- 172.21.15.32:0/1673538473 shutdown_connections 2024-08-03T20:57:45.570 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:45.568+0000 7f4bf4200640 1 --2- 172.21.15.32:0/1673538473 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f4bcc077040 0x7f4bcc079550 unknown :-1 s=CLOSED pgs=661 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:45.570 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:45.568+0000 7f4bf4200640 1 --2- 172.21.15.32:0/1673538473 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f4bec15fff0 0x7f4bec15f200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:45.570 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:45.568+0000 7f4bf4200640 1 --2- 172.21.15.32:0/1673538473 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4bec073e60 0x7f4bec15bb70 unknown :-1 s=CLOSED pgs=1619 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:45.570 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:45.568+0000 7f4bf4200640 1 --2- 172.21.15.32:0/1673538473 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f4bec06e930 0x7f4bec143350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:45.570 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:45.568+0000 7f4bf4200640 1 -- 172.21.15.32:0/1673538473 >> 172.21.15.32:0/1673538473 conn(0x7f4bec0b1bd0 msgr2=0x7f4bec0d08d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:57:45.570 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:45.568+0000 7f4bf4200640 1 -- 172.21.15.32:0/1673538473 shutdown_connections 2024-08-03T20:57:45.570 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:45.568+0000 7f4bf4200640 1 -- 172.21.15.32:0/1673538473 wait complete. 2024-08-03T20:57:45.595 DEBUG:tasks.ceph_test_case:wait_until_true: success in 0s and 0 retries 2024-08-03T20:57:45.595 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-08-03T20:57:45.824 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:45.822+0000 7fe7dc600640 1 -- 172.21.15.32:0/1006477235 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe7d4073e60 msgr2=0x7fe7d4074270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:45.825 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:45.822+0000 7fe7dc600640 1 --2- 172.21.15.32:0/1006477235 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe7d4073e60 0x7fe7d4074270 secure :-1 s=READY pgs=1620 cs=0 l=1 rev1=1 crypto rx=0x7fe7c0005fd0 tx=0x7fe7c002a540 comp rx=0 tx=0).stop 2024-08-03T20:57:45.825 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:45.823+0000 7fe7dc600640 1 -- 172.21.15.32:0/1006477235 shutdown_connections 2024-08-03T20:57:45.825 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:45.823+0000 7fe7dc600640 1 --2- 172.21.15.32:0/1006477235 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe7d4073e60 0x7fe7d4074270 unknown :-1 s=CLOSED pgs=1620 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:45.825 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:45.823+0000 7fe7dc600640 1 --2- 172.21.15.32:0/1006477235 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fe7d406e930 0x7fe7d4144f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:45.825 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:45.823+0000 7fe7dc600640 1 -- 172.21.15.32:0/1006477235 >> 172.21.15.32:0/1006477235 conn(0x7fe7d40b1bd0 msgr2=0x7fe7d40b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:57:45.825 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:45.823+0000 7fe7dc600640 1 -- 172.21.15.32:0/1006477235 shutdown_connections 2024-08-03T20:57:45.825 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:45.823+0000 7fe7dc600640 1 -- 172.21.15.32:0/1006477235 wait complete. 2024-08-03T20:57:45.826 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:45.825+0000 7fe7dc600640 1 Processor -- start 2024-08-03T20:57:45.826 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:45.825+0000 7fe7dc600640 1 -- start start 2024-08-03T20:57:45.830 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:45.827+0000 7fe7dc600640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fe7d406e930 0x7fe7d41f6e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:45.830 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:45.827+0000 7fe7dc600640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fe7d4073e60 0x7fe7d412e560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:45.830 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:45.827+0000 7fe7dc600640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe7d40747b0 0x7fe7d4133e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:45.830 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:45.827+0000 7fe7dc600640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fe7d40cf9c0 con 0x7fe7d40747b0 2024-08-03T20:57:45.830 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:45.827+0000 7fe7dc600640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7fe7d40cf840 con 0x7fe7d406e930 2024-08-03T20:57:45.830 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:45.827+0000 7fe7dc600640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7fe7d40ccff0 con 0x7fe7d4073e60 2024-08-03T20:57:45.830 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:45.827+0000 7fe7da600640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe7d40747b0 0x7fe7d4133e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:45.830 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:45.827+0000 7fe7da600640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe7d40747b0 0x7fe7d4133e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:43276/0 (socket says 172.21.15.32:43276) 2024-08-03T20:57:45.830 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:45.827+0000 7fe7da600640 1 -- 172.21.15.32:0/3985013823 learned_addr learned my addr 172.21.15.32:0/3985013823 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:57:45.831 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:45.827+0000 7fe7d9c00640 1 --2- 172.21.15.32:0/3985013823 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fe7d406e930 0x7fe7d41f6e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:45.831 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:45.827+0000 7fe7d9200640 1 --2- 172.21.15.32:0/3985013823 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fe7d4073e60 0x7fe7d412e560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:45.831 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:45.828+0000 7fe7da600640 1 -- 172.21.15.32:0/3985013823 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fe7d4073e60 msgr2=0x7fe7d412e560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:45.831 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:45.828+0000 7fe7da600640 1 --2- 172.21.15.32:0/3985013823 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fe7d4073e60 0x7fe7d412e560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:45.831 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:45.828+0000 7fe7da600640 1 -- 172.21.15.32:0/3985013823 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fe7d406e930 msgr2=0x7fe7d41f6e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:45.831 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:45.828+0000 7fe7da600640 1 --2- 172.21.15.32:0/3985013823 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fe7d406e930 0x7fe7d41f6e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:45.831 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:45.828+0000 7fe7da600640 1 -- 172.21.15.32:0/3985013823 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe7d40b19a0 con 0x7fe7d40747b0 2024-08-03T20:57:45.831 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:45.828+0000 7fe7d9c00640 1 --2- 172.21.15.32:0/3985013823 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fe7d406e930 0x7fe7d41f6e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-03T20:57:45.831 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:45.828+0000 7fe7da600640 1 --2- 172.21.15.32:0/3985013823 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe7d40747b0 0x7fe7d4133e80 secure :-1 s=READY pgs=1621 cs=0 l=1 rev1=1 crypto rx=0x7fe7cc002230 tx=0x7fe7cc00cb90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:57:45.831 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:45.828+0000 7fe7d1600640 1 -- 172.21.15.32:0/3985013823 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7cc003b70 con 0x7fe7d40747b0 2024-08-03T20:57:45.831 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:45.828+0000 7fe7d1600640 1 -- 172.21.15.32:0/3985013823 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fe7cc003d10 con 0x7fe7d40747b0 2024-08-03T20:57:45.831 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:45.828+0000 7fe7d1600640 1 -- 172.21.15.32:0/3985013823 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7cc005330 con 0x7fe7d40747b0 2024-08-03T20:57:45.831 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:45.828+0000 7fe7dc600640 1 -- 172.21.15.32:0/3985013823 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe7d41f73a0 con 0x7fe7d40747b0 2024-08-03T20:57:45.832 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:45.828+0000 7fe7dc600640 1 -- 172.21.15.32:0/3985013823 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe7d41343c0 con 0x7fe7d40747b0 2024-08-03T20:57:45.832 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:45.829+0000 7fe7dc600640 1 -- 172.21.15.32:0/3985013823 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe7d4157820 con 0x7fe7d40747b0 2024-08-03T20:57:45.832 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:45.831+0000 7fe7d1600640 1 -- 172.21.15.32:0/3985013823 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7fe7cc0056e0 con 0x7fe7d40747b0 2024-08-03T20:57:45.834 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:45.832+0000 7fe7d1600640 1 --2- 172.21.15.32:0/3985013823 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fe7b4077010 0x7fe7b4079520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:45.834 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:45.832+0000 7fe7d1600640 1 -- 172.21.15.32:0/3985013823 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(108..108 src has 1..108) ==== 7832+0+0 (secure 0 0 0) 0x7fe7cc099250 con 0x7fe7d40747b0 2024-08-03T20:57:45.834 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:45.833+0000 7fe7d9c00640 1 --2- 172.21.15.32:0/3985013823 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fe7b4077010 0x7fe7b4079520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:45.835 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:45.834+0000 7fe7d9c00640 1 --2- 172.21.15.32:0/3985013823 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fe7b4077010 0x7fe7b4079520 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7fe7c8005fd0 tx=0x7fe7c800f360 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:57:45.838 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:45.837+0000 7fe7d1600640 1 -- 172.21.15.32:0/3985013823 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7fe7d41f73a0 con 0x7fe7d40747b0 2024-08-03T20:57:46.088 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:46.086+0000 7fe7dc600640 1 -- 172.21.15.32:0/3985013823 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe7d40cd510 con 0x7fe7d40747b0 2024-08-03T20:57:46.088 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:46.087+0000 7fe7d1600640 1 -- 172.21.15.32:0/3985013823 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 226 v226) ==== 95+0+4391 (secure 0 0 0) 0x7fe7d40b19a0 con 0x7fe7d40747b0 2024-08-03T20:57:46.089 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-03T20:57:46.089 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":226,"btime":"2024-08-03T20:57:45:062715+0000","default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6439,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6835/531080160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6834","nonce":531080160},{"type":"v1","addr":"172.21.15.118:6835","nonce":531080160}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":84},{"gid":6454,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6837/2796882319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6836","nonce":2796882319},{"type":"v1","addr":"172.21.15.118:6837","nonce":2796882319}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":86},{"gid":6813,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6834/2686933890","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":2686933890},{"type":"v1","addr":"172.21.15.32:6834","nonce":2686933890}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":85}],"filesystems":[{"mdsmap":{"epoch":226,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-08-03T20:54:26.950518+0000","modified":"2024-08-03T20:57:45.062650+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":108,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":6}]} 2024-08-03T20:57:46.089 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 226 2024-08-03T20:57:46.091 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:46.090+0000 7fe7dc600640 1 -- 172.21.15.32:0/3985013823 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fe7b4077010 msgr2=0x7fe7b4079520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:46.092 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:46.090+0000 7fe7dc600640 1 --2- 172.21.15.32:0/3985013823 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fe7b4077010 0x7fe7b4079520 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7fe7c8005fd0 tx=0x7fe7c800f360 comp rx=0 tx=0).stop 2024-08-03T20:57:46.092 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:46.090+0000 7fe7dc600640 1 -- 172.21.15.32:0/3985013823 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe7d40747b0 msgr2=0x7fe7d4133e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:46.092 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:46.090+0000 7fe7dc600640 1 --2- 172.21.15.32:0/3985013823 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe7d40747b0 0x7fe7d4133e80 secure :-1 s=READY pgs=1621 cs=0 l=1 rev1=1 crypto rx=0x7fe7cc002230 tx=0x7fe7cc00cb90 comp rx=0 tx=0).stop 2024-08-03T20:57:46.092 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:46.090+0000 7fe7dc600640 1 -- 172.21.15.32:0/3985013823 shutdown_connections 2024-08-03T20:57:46.092 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:46.090+0000 7fe7dc600640 1 --2- 172.21.15.32:0/3985013823 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fe7b4077010 0x7fe7b4079520 unknown :-1 s=CLOSED pgs=662 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:46.092 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:46.090+0000 7fe7dc600640 1 --2- 172.21.15.32:0/3985013823 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe7d40747b0 0x7fe7d4133e80 unknown :-1 s=CLOSED pgs=1621 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:46.092 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:46.090+0000 7fe7dc600640 1 --2- 172.21.15.32:0/3985013823 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fe7d4073e60 0x7fe7d412e560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:46.092 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:46.090+0000 7fe7dc600640 1 --2- 172.21.15.32:0/3985013823 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fe7d406e930 0x7fe7d41f6e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:46.092 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:46.090+0000 7fe7dc600640 1 -- 172.21.15.32:0/3985013823 >> 172.21.15.32:0/3985013823 conn(0x7fe7d40b1bd0 msgr2=0x7fe7d415ff80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:57:46.092 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:46.090+0000 7fe7dc600640 1 -- 172.21.15.32:0/3985013823 shutdown_connections 2024-08-03T20:57:46.092 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:46.090+0000 7fe7dc600640 1 -- 172.21.15.32:0/3985013823 wait complete. 2024-08-03T20:57:46.118 DEBUG:tasks.ceph_test_case:wait_until_true: success in 0s and 0 retries 2024-08-03T20:57:46.119 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-08-03T20:57:46.348 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:46.346+0000 7fe1e5e00640 1 -- 172.21.15.32:0/3368747345 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fe1e00d4bb0 msgr2=0x7fe1e00d1c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:46.348 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:46.346+0000 7fe1e5e00640 1 --2- 172.21.15.32:0/3368747345 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fe1e00d4bb0 0x7fe1e00d1c60 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fe1d0009830 tx=0x7fe1d002a480 comp rx=0 tx=0).stop 2024-08-03T20:57:46.348 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:46.346+0000 7fe1e5e00640 1 -- 172.21.15.32:0/3368747345 shutdown_connections 2024-08-03T20:57:46.348 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:46.346+0000 7fe1e5e00640 1 --2- 172.21.15.32:0/3368747345 >> v2:172.21.15.118:3300/0 conn(0x7fe1e00d2290 0x7fe1e00d26a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:46.349 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:46.346+0000 7fe1e5e00640 1 --2- 172.21.15.32:0/3368747345 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fe1e00d4bb0 0x7fe1e00d1c60 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:46.349 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:46.346+0000 7fe1e5e00640 1 -- 172.21.15.32:0/3368747345 >> 172.21.15.32:0/3368747345 conn(0x7fe1e00d4090 msgr2=0x7fe1e00d44a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:57:46.349 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:46.347+0000 7fe1e5e00640 1 -- 172.21.15.32:0/3368747345 shutdown_connections 2024-08-03T20:57:46.349 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:46.347+0000 7fe1e5e00640 1 -- 172.21.15.32:0/3368747345 wait complete. 2024-08-03T20:57:46.350 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:46.349+0000 7fe1e5e00640 1 Processor -- start 2024-08-03T20:57:46.350 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:46.349+0000 7fe1e5e00640 1 -- start start 2024-08-03T20:57:46.353 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:46.350+0000 7fe1e5e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe1e00d0aa0 0x7fe1e01f6250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:46.353 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:46.350+0000 7fe1e5e00640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fe1e00d2290 0x7fe1e00b6ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:46.353 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:46.350+0000 7fe1e5e00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fe1e00d4bb0 0x7fe1e015cd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:46.353 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:46.350+0000 7fe1e5e00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fe1e00d15c0 con 0x7fe1e00d0aa0 2024-08-03T20:57:46.353 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:46.350+0000 7fe1e5e00640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7fe1e00cfe90 con 0x7fe1e00d4bb0 2024-08-03T20:57:46.354 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:46.350+0000 7fe1e5e00640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7fe1e00d1740 con 0x7fe1e00d2290 2024-08-03T20:57:46.354 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:46.350+0000 7fe1e4a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe1e00d0aa0 0x7fe1e01f6250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:46.354 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:46.351+0000 7fe1e4a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe1e00d0aa0 0x7fe1e01f6250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:43282/0 (socket says 172.21.15.32:43282) 2024-08-03T20:57:46.354 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:46.351+0000 7fe1e4a00640 1 -- 172.21.15.32:0/3604138824 learned_addr learned my addr 172.21.15.32:0/3604138824 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:57:46.354 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:46.351+0000 7fe1d7e00640 1 --2- 172.21.15.32:0/3604138824 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fe1e00d2290 0x7fe1e00b6ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:46.354 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:46.351+0000 7fe1e5400640 1 --2- 172.21.15.32:0/3604138824 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fe1e00d4bb0 0x7fe1e015cd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:46.354 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:46.351+0000 7fe1e4a00640 1 -- 172.21.15.32:0/3604138824 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fe1e00d2290 msgr2=0x7fe1e00b6ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:46.354 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:46.351+0000 7fe1e4a00640 1 --2- 172.21.15.32:0/3604138824 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fe1e00d2290 0x7fe1e00b6ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:46.354 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:46.351+0000 7fe1e4a00640 1 -- 172.21.15.32:0/3604138824 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fe1e00d4bb0 msgr2=0x7fe1e015cd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:46.354 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:46.351+0000 7fe1e4a00640 1 --2- 172.21.15.32:0/3604138824 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fe1e00d4bb0 0x7fe1e015cd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:46.354 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:46.351+0000 7fe1e4a00640 1 -- 172.21.15.32:0/3604138824 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe1e01f57e0 con 0x7fe1e00d0aa0 2024-08-03T20:57:46.354 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:46.351+0000 7fe1e4a00640 1 --2- 172.21.15.32:0/3604138824 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe1e00d0aa0 0x7fe1e01f6250 secure :-1 s=READY pgs=1622 cs=0 l=1 rev1=1 crypto rx=0x7fe1d002aa70 tx=0x7fe1d0033c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:57:46.354 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:46.352+0000 7fe1dd600640 1 -- 172.21.15.32:0/3604138824 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1d0031d60 con 0x7fe1e00d0aa0 2024-08-03T20:57:46.355 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:46.352+0000 7fe1dd600640 1 -- 172.21.15.32:0/3604138824 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fe1d0033e10 con 0x7fe1e00d0aa0 2024-08-03T20:57:46.355 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:46.352+0000 7fe1dd600640 1 -- 172.21.15.32:0/3604138824 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1d0005260 con 0x7fe1e00d0aa0 2024-08-03T20:57:46.355 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:46.352+0000 7fe1e5e00640 1 -- 172.21.15.32:0/3604138824 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe1e00b75f0 con 0x7fe1e00d0aa0 2024-08-03T20:57:46.355 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:46.352+0000 7fe1e5e00640 1 -- 172.21.15.32:0/3604138824 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe1e015d8d0 con 0x7fe1e00d0aa0 2024-08-03T20:57:46.355 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:46.353+0000 7fe1e5e00640 1 -- 172.21.15.32:0/3604138824 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe1e015db30 con 0x7fe1e00d0aa0 2024-08-03T20:57:46.356 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:46.354+0000 7fe1dd600640 1 -- 172.21.15.32:0/3604138824 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7fe1d0005400 con 0x7fe1e00d0aa0 2024-08-03T20:57:46.357 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:46.356+0000 7fe1dd600640 1 --2- 172.21.15.32:0/3604138824 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fe1bc077010 0x7fe1bc079520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:46.357 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:46.356+0000 7fe1dd600640 1 -- 172.21.15.32:0/3604138824 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(108..108 src has 1..108) ==== 7832+0+0 (secure 0 0 0) 0x7fe1d00ba1e0 con 0x7fe1e00d0aa0 2024-08-03T20:57:46.358 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:46.356+0000 7fe1d7e00640 1 --2- 172.21.15.32:0/3604138824 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fe1bc077010 0x7fe1bc079520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:46.358 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:46.357+0000 7fe1d7e00640 1 --2- 172.21.15.32:0/3604138824 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fe1bc077010 0x7fe1bc079520 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7fe1e00cae20 tx=0x7fe1c8009260 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:57:46.362 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:46.360+0000 7fe1dd600640 1 -- 172.21.15.32:0/3604138824 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7fe1d003f030 con 0x7fe1e00d0aa0 2024-08-03T20:57:46.616 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:46.615+0000 7fe1e5e00640 1 -- 172.21.15.32:0/3604138824 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe1e008c100 con 0x7fe1e00d0aa0 2024-08-03T20:57:46.617 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:46.615+0000 7fe1dd600640 1 -- 172.21.15.32:0/3604138824 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 226 v226) ==== 95+0+4391 (secure 0 0 0) 0x7fe1d00824a0 con 0x7fe1e00d0aa0 2024-08-03T20:57:46.617 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-03T20:57:46.617 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":226,"btime":"2024-08-03T20:57:45:062715+0000","default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6439,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6835/531080160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6834","nonce":531080160},{"type":"v1","addr":"172.21.15.118:6835","nonce":531080160}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":84},{"gid":6454,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6837/2796882319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6836","nonce":2796882319},{"type":"v1","addr":"172.21.15.118:6837","nonce":2796882319}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":86},{"gid":6813,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6834/2686933890","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":2686933890},{"type":"v1","addr":"172.21.15.32:6834","nonce":2686933890}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":85}],"filesystems":[{"mdsmap":{"epoch":226,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-08-03T20:54:26.950518+0000","modified":"2024-08-03T20:57:45.062650+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":108,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":6}]} 2024-08-03T20:57:46.617 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 226 2024-08-03T20:57:46.619 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:46.617+0000 7fe1e5e00640 1 -- 172.21.15.32:0/3604138824 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fe1bc077010 msgr2=0x7fe1bc079520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:46.620 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:46.617+0000 7fe1e5e00640 1 --2- 172.21.15.32:0/3604138824 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fe1bc077010 0x7fe1bc079520 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7fe1e00cae20 tx=0x7fe1c8009260 comp rx=0 tx=0).stop 2024-08-03T20:57:46.620 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:46.617+0000 7fe1e5e00640 1 -- 172.21.15.32:0/3604138824 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe1e00d0aa0 msgr2=0x7fe1e01f6250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:46.620 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:46.617+0000 7fe1e5e00640 1 --2- 172.21.15.32:0/3604138824 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe1e00d0aa0 0x7fe1e01f6250 secure :-1 s=READY pgs=1622 cs=0 l=1 rev1=1 crypto rx=0x7fe1d002aa70 tx=0x7fe1d0033c20 comp rx=0 tx=0).stop 2024-08-03T20:57:46.620 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:46.618+0000 7fe1e5e00640 1 -- 172.21.15.32:0/3604138824 shutdown_connections 2024-08-03T20:57:46.620 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:46.618+0000 7fe1e5e00640 1 --2- 172.21.15.32:0/3604138824 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fe1bc077010 0x7fe1bc079520 unknown :-1 s=CLOSED pgs=663 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:46.620 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:46.618+0000 7fe1e5e00640 1 --2- 172.21.15.32:0/3604138824 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fe1e00d4bb0 0x7fe1e015cd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:46.620 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:46.618+0000 7fe1e5e00640 1 --2- 172.21.15.32:0/3604138824 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fe1e00d2290 0x7fe1e00b6ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:46.620 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:46.618+0000 7fe1e5e00640 1 --2- 172.21.15.32:0/3604138824 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe1e00d0aa0 0x7fe1e01f6250 unknown :-1 s=CLOSED pgs=1622 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:46.620 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:46.618+0000 7fe1e5e00640 1 -- 172.21.15.32:0/3604138824 >> 172.21.15.32:0/3604138824 conn(0x7fe1e00d4090 msgr2=0x7fe1e0166510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:57:46.620 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:46.618+0000 7fe1e5e00640 1 -- 172.21.15.32:0/3604138824 shutdown_connections 2024-08-03T20:57:46.620 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:46.618+0000 7fe1e5e00640 1 -- 172.21.15.32:0/3604138824 wait complete. 2024-08-03T20:57:46.642 INFO:tasks.cephfs.test_damage:Result: Mutation 'Corrupt 100.00000000.inode' led to DAMAGED state 2024-08-03T20:57:46.642 INFO:tasks.cephfs.test_damage:Applying mutation 'Truncate 1.00000000.inode' 2024-08-03T20:57:46.642 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-08-03T20:57:46.642 DEBUG:teuthology.orchestra.run.smithi032:> dd if=/proc/self/mounts of=/dev/stdout 2024-08-03T20:57:46.678 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053453,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6499208k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 none /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249604k,nr_inodes=812401,mode=700,uid=1000,gid=1316 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6499208k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-08-03T20:57:46.678 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi032.front.sepia.ceph.com 2024-08-03T20:57:46.679 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-08-03T20:57:46.679 DEBUG:teuthology.orchestra.run.smithi032:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-08-03T20:57:46.752 INFO:teuthology.orchestra.run.smithi032.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.0': No such file or directory 2024-08-03T20:57:46.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-03T20:57:46.753 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-08-03T20:57:47.023 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:47.021+0000 7fac8d800640 1 -- 172.21.15.32:0/3526430470 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fac6c003a30 msgr2=0x7fac6c003ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:47.023 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:47.021+0000 7fac8d800640 1 --2- 172.21.15.32:0/3526430470 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fac6c003a30 0x7fac6c003ea0 secure :-1 s=READY pgs=1624 cs=0 l=1 rev1=1 crypto rx=0x7fac78009b50 tx=0x7fac6c005830 comp rx=0 tx=0).stop 2024-08-03T20:57:47.023 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:47.021+0000 7fac87400640 1 -- 172.21.15.32:0/3526430470 reap_dead start 2024-08-03T20:57:47.023 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:47.021+0000 7fac8d800640 1 -- 172.21.15.32:0/3526430470 shutdown_connections 2024-08-03T20:57:47.023 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:47.021+0000 7fac8d800640 1 -- 172.21.15.32:0/3526430470 >> 172.21.15.32:0/3526430470 conn(0x7fac880b1bd0 msgr2=0x7fac880b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:57:47.024 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:47.021+0000 7fac8d800640 1 -- 172.21.15.32:0/3526430470 shutdown_connections 2024-08-03T20:57:47.024 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:47.021+0000 7fac8d800640 1 -- 172.21.15.32:0/3526430470 wait complete. 2024-08-03T20:57:47.025 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:47.024+0000 7fac8d800640 1 Processor -- start 2024-08-03T20:57:47.025 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:47.024+0000 7fac8d800640 1 -- start start 2024-08-03T20:57:47.028 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:47.025+0000 7fac8d800640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fac6c003a30 0x7fac881377f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:47.028 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:47.025+0000 7fac8d800640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fac8813a370 0x7fac881f8630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:47.028 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:47.025+0000 7fac8d800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fac8813a780 0x7fac881f8b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:47.028 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:47.025+0000 7fac8d800640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fac88145eb0 con 0x7fac8813a780 2024-08-03T20:57:47.028 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:47.025+0000 7fac8d800640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7fac880cfa50 con 0x7fac8813a370 2024-08-03T20:57:47.029 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:47.025+0000 7fac8d800640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7fac88160b10 con 0x7fac6c003a30 2024-08-03T20:57:47.029 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:47.026+0000 7fac87400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fac8813a780 0x7fac881f8b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:47.029 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:47.026+0000 7fac87400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fac8813a780 0x7fac881f8b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:43300/0 (socket says 172.21.15.32:43300) 2024-08-03T20:57:47.029 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:47.026+0000 7fac87400640 1 -- 172.21.15.32:0/2081461766 learned_addr learned my addr 172.21.15.32:0/2081461766 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:57:47.029 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:47.026+0000 7fac86a00640 1 --2- 172.21.15.32:0/2081461766 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fac6c003a30 0x7fac881377f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:47.029 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:47.026+0000 7fac7e000640 1 --2- 172.21.15.32:0/2081461766 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fac8813a370 0x7fac881f8630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:47.029 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:47.026+0000 7fac87400640 1 -- 172.21.15.32:0/2081461766 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fac6c003a30 msgr2=0x7fac881377f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:47.029 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:47.026+0000 7fac87400640 1 --2- 172.21.15.32:0/2081461766 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fac6c003a30 0x7fac881377f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:47.029 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:47.026+0000 7fac87400640 1 -- 172.21.15.32:0/2081461766 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fac8813a370 msgr2=0x7fac881f8630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:47.030 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:47.026+0000 7fac87400640 1 --2- 172.21.15.32:0/2081461766 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fac8813a370 0x7fac881f8630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:47.030 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:47.026+0000 7fac87400640 1 -- 172.21.15.32:0/2081461766 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fac880b5010 con 0x7fac8813a780 2024-08-03T20:57:47.030 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:47.026+0000 7fac86a00640 1 --2- 172.21.15.32:0/2081461766 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fac6c003a30 0x7fac881377f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-03T20:57:47.030 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:47.026+0000 7fac87400640 1 --2- 172.21.15.32:0/2081461766 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fac8813a780 0x7fac881f8b70 secure :-1 s=READY pgs=1625 cs=0 l=1 rev1=1 crypto rx=0x7fac88072860 tx=0x7fac80001c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:57:47.030 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:47.027+0000 7fac7f400640 1 -- 172.21.15.32:0/2081461766 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac80011900 con 0x7fac8813a780 2024-08-03T20:57:47.030 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:47.027+0000 7fac7f400640 1 -- 172.21.15.32:0/2081461766 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fac80018070 con 0x7fac8813a780 2024-08-03T20:57:47.030 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:47.027+0000 7fac8d800640 1 -- 172.21.15.32:0/2081461766 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fac88137eb0 con 0x7fac8813a780 2024-08-03T20:57:47.030 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:47.027+0000 7fac7f400640 1 -- 172.21.15.32:0/2081461766 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac80011d60 con 0x7fac8813a780 2024-08-03T20:57:47.030 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:47.027+0000 7fac8d800640 1 -- 172.21.15.32:0/2081461766 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fac88138470 con 0x7fac8813a780 2024-08-03T20:57:47.031 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:47.028+0000 7fac8d800640 1 -- 172.21.15.32:0/2081461766 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fac88144f90 con 0x7fac8813a780 2024-08-03T20:57:47.031 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:47.029+0000 7fac7f400640 1 -- 172.21.15.32:0/2081461766 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7fac8001d460 con 0x7fac8813a780 2024-08-03T20:57:47.032 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:47.031+0000 7fac7f400640 1 --2- 172.21.15.32:0/2081461766 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fac64077010 0x7fac64079520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:47.032 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:47.031+0000 7fac7f400640 1 -- 172.21.15.32:0/2081461766 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(108..108 src has 1..108) ==== 7832+0+0 (secure 0 0 0) 0x7fac800999e0 con 0x7fac8813a780 2024-08-03T20:57:47.033 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:47.031+0000 7fac86a00640 1 --2- 172.21.15.32:0/2081461766 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fac64077010 0x7fac64079520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:47.034 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:47.032+0000 7fac86a00640 1 --2- 172.21.15.32:0/2081461766 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fac64077010 0x7fac64079520 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7fac78035000 tx=0x7fac7802d040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:57:47.036 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:47.035+0000 7fac7f400640 1 -- 172.21.15.32:0/2081461766 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7fac80062380 con 0x7fac8813a780 2024-08-03T20:57:47.281 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:47.279+0000 7fac8d800640 1 -- 172.21.15.32:0/2081461766 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7fac88160610 con 0x7fac8813a780 2024-08-03T20:57:47.442 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:47.440+0000 7fac7f400640 1 -- 172.21.15.32:0/2081461766 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v227) ==== 161+0+0 (secure 0 0 0) 0x7fac80061c80 con 0x7fac8813a780 2024-08-03T20:57:47.442 INFO:teuthology.orchestra.run.smithi032.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-08-03T20:57:47.444 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:47.443+0000 7fac8d800640 1 -- 172.21.15.32:0/2081461766 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fac64077010 msgr2=0x7fac64079520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:47.445 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:47.443+0000 7fac8d800640 1 --2- 172.21.15.32:0/2081461766 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fac64077010 0x7fac64079520 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7fac78035000 tx=0x7fac7802d040 comp rx=0 tx=0).stop 2024-08-03T20:57:47.445 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:47.443+0000 7fac8d800640 1 -- 172.21.15.32:0/2081461766 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fac8813a780 msgr2=0x7fac881f8b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:47.445 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:47.443+0000 7fac8d800640 1 --2- 172.21.15.32:0/2081461766 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fac8813a780 0x7fac881f8b70 secure :-1 s=READY pgs=1625 cs=0 l=1 rev1=1 crypto rx=0x7fac88072860 tx=0x7fac80001c10 comp rx=0 tx=0).stop 2024-08-03T20:57:47.445 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:47.443+0000 7fac8d800640 1 -- 172.21.15.32:0/2081461766 shutdown_connections 2024-08-03T20:57:47.445 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:47.443+0000 7fac8d800640 1 --2- 172.21.15.32:0/2081461766 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fac64077010 0x7fac64079520 unknown :-1 s=CLOSED pgs=664 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:47.445 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:47.443+0000 7fac8d800640 1 --2- 172.21.15.32:0/2081461766 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fac8813a370 0x7fac881f8630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:47.445 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:47.443+0000 7fac8d800640 1 --2- 172.21.15.32:0/2081461766 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fac8813a780 0x7fac881f8b70 unknown :-1 s=CLOSED pgs=1625 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:47.445 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:47.443+0000 7fac8d800640 1 --2- 172.21.15.32:0/2081461766 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fac6c003a30 0x7fac881377f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:47.445 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:47.443+0000 7fac8d800640 1 -- 172.21.15.32:0/2081461766 >> 172.21.15.32:0/2081461766 conn(0x7fac880b1bd0 msgr2=0x7fac880b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:57:47.445 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:47.443+0000 7fac8d800640 1 -- 172.21.15.32:0/2081461766 shutdown_connections 2024-08-03T20:57:47.445 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:47.443+0000 7fac8d800640 1 -- 172.21.15.32:0/2081461766 wait complete. 2024-08-03T20:57:47.470 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mds repaired 0 2024-08-03T20:57:47.698 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:47.696+0000 7fd9e4400640 1 -- 172.21.15.32:0/3556639844 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd9dc165160 msgr2=0x7fd9dc167580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:47.698 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:47.696+0000 7fd9e4400640 1 --2- 172.21.15.32:0/3556639844 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd9dc165160 0x7fd9dc167580 secure :-1 s=READY pgs=1627 cs=0 l=1 rev1=1 crypto rx=0x7fd9d4005fd0 tx=0x7fd9d402a820 comp rx=0 tx=0).stop 2024-08-03T20:57:47.698 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:47.696+0000 7fd9e4400640 1 -- 172.21.15.32:0/3556639844 shutdown_connections 2024-08-03T20:57:47.699 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:47.696+0000 7fd9e4400640 1 --2- 172.21.15.32:0/3556639844 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd9dc165160 0x7fd9dc167580 unknown :-1 s=CLOSED pgs=1627 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:47.699 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:47.696+0000 7fd9e4400640 1 -- 172.21.15.32:0/3556639844 >> 172.21.15.32:0/3556639844 conn(0x7fd9dc0b1bd0 msgr2=0x7fd9dc0b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:57:47.699 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:47.697+0000 7fd9e4400640 1 -- 172.21.15.32:0/3556639844 shutdown_connections 2024-08-03T20:57:47.699 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:47.697+0000 7fd9e4400640 1 -- 172.21.15.32:0/3556639844 wait complete. 2024-08-03T20:57:47.700 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:47.699+0000 7fd9e4400640 1 Processor -- start 2024-08-03T20:57:47.700 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:47.699+0000 7fd9e4400640 1 -- start start 2024-08-03T20:57:47.703 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:47.700+0000 7fd9e4400640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fd9dc06e930 0x7fd9dc15b8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:47.703 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:47.700+0000 7fd9e4400640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fd9dc144f90 0x7fd9dc141830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:47.703 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:47.700+0000 7fd9e4400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd9dc165160 0x7fd9dc141d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:47.703 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:47.700+0000 7fd9e4400640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fd9dc145ec0 con 0x7fd9dc165160 2024-08-03T20:57:47.703 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:47.700+0000 7fd9e4400640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7fd9dc0cfa50 con 0x7fd9dc144f90 2024-08-03T20:57:47.703 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:47.700+0000 7fd9e4400640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7fd9dc167ac0 con 0x7fd9dc06e930 2024-08-03T20:57:47.703 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:47.701+0000 7fd9e2400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd9dc165160 0x7fd9dc141d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:47.703 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:47.701+0000 7fd9e2400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd9dc165160 0x7fd9dc141d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:43310/0 (socket says 172.21.15.32:43310) 2024-08-03T20:57:47.703 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:47.701+0000 7fd9e2400640 1 -- 172.21.15.32:0/4274559220 learned_addr learned my addr 172.21.15.32:0/4274559220 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:57:47.703 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:47.701+0000 7fd9e1000640 1 --2- 172.21.15.32:0/4274559220 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fd9dc144f90 0x7fd9dc141830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:47.703 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:47.701+0000 7fd9e1a00640 1 --2- 172.21.15.32:0/4274559220 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fd9dc06e930 0x7fd9dc15b8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:47.703 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:47.701+0000 7fd9e2400640 1 -- 172.21.15.32:0/4274559220 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fd9dc06e930 msgr2=0x7fd9dc15b8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:47.704 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:47.701+0000 7fd9e2400640 1 --2- 172.21.15.32:0/4274559220 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fd9dc06e930 0x7fd9dc15b8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:47.704 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:47.701+0000 7fd9e2400640 1 -- 172.21.15.32:0/4274559220 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fd9dc144f90 msgr2=0x7fd9dc141830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:47.704 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:47.701+0000 7fd9e2400640 1 --2- 172.21.15.32:0/4274559220 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fd9dc144f90 0x7fd9dc141830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:47.704 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:47.701+0000 7fd9e2400640 1 -- 172.21.15.32:0/4274559220 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd9dc1f6180 con 0x7fd9dc165160 2024-08-03T20:57:47.704 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:47.701+0000 7fd9e2400640 1 --2- 172.21.15.32:0/4274559220 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd9dc165160 0x7fd9dc141d70 secure :-1 s=READY pgs=1628 cs=0 l=1 rev1=1 crypto rx=0x7fd9d4009b30 tx=0x7fd9d4032d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:57:47.705 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:47.702+0000 7fd9d9600640 1 -- 172.21.15.32:0/4274559220 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9d4037040 con 0x7fd9dc165160 2024-08-03T20:57:47.705 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:47.702+0000 7fd9d9600640 1 -- 172.21.15.32:0/4274559220 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fd9d4042070 con 0x7fd9dc165160 2024-08-03T20:57:47.705 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:47.702+0000 7fd9d9600640 1 -- 172.21.15.32:0/4274559220 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9d40042a0 con 0x7fd9dc165160 2024-08-03T20:57:47.705 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:47.702+0000 7fd9e4400640 1 -- 172.21.15.32:0/4274559220 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd9dc1392d0 con 0x7fd9dc165160 2024-08-03T20:57:47.705 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:47.702+0000 7fd9e4400640 1 -- 172.21.15.32:0/4274559220 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd9dc139890 con 0x7fd9dc165160 2024-08-03T20:57:47.705 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:47.703+0000 7fd9e4400640 1 -- 172.21.15.32:0/4274559220 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd9dc073e60 con 0x7fd9dc165160 2024-08-03T20:57:47.706 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:47.704+0000 7fd9d9600640 1 -- 172.21.15.32:0/4274559220 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7fd9d4004440 con 0x7fd9dc165160 2024-08-03T20:57:47.706 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:47.705+0000 7fd9d9600640 1 --2- 172.21.15.32:0/4274559220 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fd9bc077040 0x7fd9bc079550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:47.706 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:47.705+0000 7fd9d9600640 1 -- 172.21.15.32:0/4274559220 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(108..108 src has 1..108) ==== 7832+0+0 (secure 0 0 0) 0x7fd9d40b9f00 con 0x7fd9dc165160 2024-08-03T20:57:47.707 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:47.705+0000 7fd9e1a00640 1 --2- 172.21.15.32:0/4274559220 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fd9bc077040 0x7fd9bc079550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:47.708 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:47.706+0000 7fd9e1a00640 1 --2- 172.21.15.32:0/4274559220 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fd9bc077040 0x7fd9bc079550 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7fd9d00015e0 tx=0x7fd9d0009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:57:47.710 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:47.709+0000 7fd9d9600640 1 -- 172.21.15.32:0/4274559220 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7fd9d4082930 con 0x7fd9dc165160 2024-08-03T20:57:47.991 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:47.989+0000 7fd9e4400640 1 -- 172.21.15.32:0/4274559220 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "mds repaired", "role": "0"} v 0) -- 0x7fd9dc1458b0 con 0x7fd9dc165160 2024-08-03T20:57:48.453 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:48.451+0000 7fd9d9600640 1 -- 172.21.15.32:0/4274559220 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "mds repaired", "role": "0"}]=0 repaired: restoring rank 6:0 v228) ==== 101+0+0 (secure 0 0 0) 0x7fd9d4082230 con 0x7fd9dc165160 2024-08-03T20:57:48.453 INFO:teuthology.orchestra.run.smithi032.stderr:repaired: restoring rank 6:0 2024-08-03T20:57:48.456 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:48.454+0000 7fd9e4400640 1 -- 172.21.15.32:0/4274559220 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fd9bc077040 msgr2=0x7fd9bc079550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:48.456 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:48.454+0000 7fd9e4400640 1 --2- 172.21.15.32:0/4274559220 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fd9bc077040 0x7fd9bc079550 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7fd9d00015e0 tx=0x7fd9d0009040 comp rx=0 tx=0).stop 2024-08-03T20:57:48.456 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:48.454+0000 7fd9e4400640 1 -- 172.21.15.32:0/4274559220 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd9dc165160 msgr2=0x7fd9dc141d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:48.456 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:48.454+0000 7fd9e4400640 1 --2- 172.21.15.32:0/4274559220 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd9dc165160 0x7fd9dc141d70 secure :-1 s=READY pgs=1628 cs=0 l=1 rev1=1 crypto rx=0x7fd9d4009b30 tx=0x7fd9d4032d10 comp rx=0 tx=0).stop 2024-08-03T20:57:48.456 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:48.454+0000 7fd9e4400640 1 -- 172.21.15.32:0/4274559220 shutdown_connections 2024-08-03T20:57:48.456 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:48.454+0000 7fd9e4400640 1 --2- 172.21.15.32:0/4274559220 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fd9bc077040 0x7fd9bc079550 unknown :-1 s=CLOSED pgs=665 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:48.456 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:48.454+0000 7fd9e4400640 1 --2- 172.21.15.32:0/4274559220 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fd9dc144f90 0x7fd9dc141830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:48.456 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:48.454+0000 7fd9e4400640 1 --2- 172.21.15.32:0/4274559220 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd9dc165160 0x7fd9dc141d70 unknown :-1 s=CLOSED pgs=1628 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:48.456 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:48.454+0000 7fd9e4400640 1 --2- 172.21.15.32:0/4274559220 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fd9dc06e930 0x7fd9dc15b8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:48.457 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:48.454+0000 7fd9e4400640 1 -- 172.21.15.32:0/4274559220 >> 172.21.15.32:0/4274559220 conn(0x7fd9dc0b1bd0 msgr2=0x7fd9dc166ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:57:48.457 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:48.454+0000 7fd9e4400640 1 -- 172.21.15.32:0/4274559220 shutdown_connections 2024-08-03T20:57:48.457 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:48.454+0000 7fd9e4400640 1 -- 172.21.15.32:0/4274559220 wait complete. 2024-08-03T20:57:48.480 DEBUG:teuthology.orchestra.run.smithi032:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata import - 2024-08-03T20:57:48.624 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:48.622+0000 7f99cc6e9f40 1 Processor -- start 2024-08-03T20:57:48.624 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:48.622+0000 7f99cc6e9f40 1 -- start start 2024-08-03T20:57:48.626 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:48.623+0000 7f99cc6e9f40 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55f2c7db3360 0x55f2c7dcceb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:48.626 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:48.623+0000 7f99cc6e9f40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55f2c7db4ea0 0x55f2c7d44060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:48.626 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:48.623+0000 7f99cc6e9f40 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55f2c7dd0cb0 0x55f2c7d445a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:48.626 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:48.623+0000 7f99cc6e9f40 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55f2c7db41b0 con 0x55f2c7dd0cb0 2024-08-03T20:57:48.626 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:48.623+0000 7f99cc6e9f40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x55f2c7c4ae30 con 0x55f2c7db4ea0 2024-08-03T20:57:48.626 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:48.623+0000 7f99cc6e9f40 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x55f2c7dd10c0 con 0x55f2c7db3360 2024-08-03T20:57:48.626 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:48.623+0000 7f99c6200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55f2c7dd0cb0 0x55f2c7d445a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:48.626 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:48.623+0000 7f99c6200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55f2c7dd0cb0 0x55f2c7d445a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:43328/0 (socket says 172.21.15.32:43328) 2024-08-03T20:57:48.627 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:48.624+0000 7f99c6200640 1 -- 172.21.15.32:0/374927425 learned_addr learned my addr 172.21.15.32:0/374927425 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:57:48.627 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:48.624+0000 7f99c4e00640 1 --2- 172.21.15.32:0/374927425 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55f2c7db4ea0 0x55f2c7d44060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:48.627 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:48.624+0000 7f99c5800640 1 --2- 172.21.15.32:0/374927425 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55f2c7db3360 0x55f2c7dcceb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:48.627 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:48.624+0000 7f99c6200640 1 -- 172.21.15.32:0/374927425 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55f2c7db3360 msgr2=0x55f2c7dcceb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:57:48.627 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:48.624+0000 7f99c6200640 1 --2- 172.21.15.32:0/374927425 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55f2c7db3360 0x55f2c7dcceb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:48.627 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:48.624+0000 7f99c6200640 1 -- 172.21.15.32:0/374927425 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55f2c7db4ea0 msgr2=0x55f2c7d44060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:57:48.627 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:48.624+0000 7f99c6200640 1 --2- 172.21.15.32:0/374927425 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55f2c7db4ea0 0x55f2c7d44060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:48.627 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:48.624+0000 7f99c6200640 1 -- 172.21.15.32:0/374927425 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f2c7c23f10 con 0x55f2c7dd0cb0 2024-08-03T20:57:48.627 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:48.624+0000 7f99c4e00640 1 --2- 172.21.15.32:0/374927425 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55f2c7db4ea0 0x55f2c7d44060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-03T20:57:48.627 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:48.624+0000 7f99c6200640 1 --2- 172.21.15.32:0/374927425 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55f2c7dd0cb0 0x55f2c7d445a0 secure :-1 s=READY pgs=1630 cs=0 l=1 rev1=1 crypto rx=0x7f99c0057040 tx=0x7f99c0003330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=62603cfdaf0906ee server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:57:48.627 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:48.624+0000 7f99bf400640 1 -- 172.21.15.32:0/374927425 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99c0064030 con 0x55f2c7dd0cb0 2024-08-03T20:57:48.627 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:48.624+0000 7f99bf400640 1 -- 172.21.15.32:0/374927425 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f99c0004020 con 0x55f2c7dd0cb0 2024-08-03T20:57:48.627 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:48.624+0000 7f99bf400640 1 -- 172.21.15.32:0/374927425 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99c0004270 con 0x55f2c7dd0cb0 2024-08-03T20:57:48.629 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:48.625+0000 7f99cc6e9f40 1 -- 172.21.15.32:0/374927425 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55f2c7dd0cb0 msgr2=0x55f2c7d445a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:48.629 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:48.625+0000 7f99cc6e9f40 1 --2- 172.21.15.32:0/374927425 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55f2c7dd0cb0 0x55f2c7d445a0 secure :-1 s=READY pgs=1630 cs=0 l=1 rev1=1 crypto rx=0x7f99c0057040 tx=0x7f99c0003330 comp rx=0 tx=0).stop 2024-08-03T20:57:48.629 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:48.625+0000 7f99cc6e9f40 1 -- 172.21.15.32:0/374927425 shutdown_connections 2024-08-03T20:57:48.629 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:48.625+0000 7f99cc6e9f40 1 --2- 172.21.15.32:0/374927425 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55f2c7db4ea0 0x55f2c7d44060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:48.629 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:48.625+0000 7f99cc6e9f40 1 --2- 172.21.15.32:0/374927425 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55f2c7dd0cb0 0x55f2c7d445a0 unknown :-1 s=CLOSED pgs=1630 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:48.629 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:48.625+0000 7f99cc6e9f40 1 --2- 172.21.15.32:0/374927425 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55f2c7db3360 0x55f2c7dcceb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:48.629 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:48.625+0000 7f99cc6e9f40 1 -- 172.21.15.32:0/374927425 >> 172.21.15.32:0/374927425 conn(0x55f2c7c1cac0 msgr2=0x55f2c7e5dbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:57:48.629 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:48.625+0000 7f99cc6e9f40 1 -- 172.21.15.32:0/374927425 shutdown_connections 2024-08-03T20:57:48.629 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:48.625+0000 7f99cc6e9f40 1 -- 172.21.15.32:0/374927425 wait complete. 2024-08-03T20:57:48.630 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:48.626+0000 7f99cc6e9f40 1 Processor -- start 2024-08-03T20:57:48.630 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:48.627+0000 7f99cc6e9f40 1 -- start start 2024-08-03T20:57:48.631 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:48.628+0000 7f99cc6e9f40 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55f2c7db3360 0x55f2c7dcb8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:48.631 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:48.628+0000 7f99cc6e9f40 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55f2c7db4ea0 0x55f2c7da9130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:48.632 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:48.628+0000 7f99cc6e9f40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55f2c7dd0cb0 0x55f2c7d3d1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:48.632 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:48.628+0000 7f99cc6e9f40 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55f2c7d45a80 con 0x55f2c7db4ea0 2024-08-03T20:57:48.632 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:48.628+0000 7f99cc6e9f40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x55f2c7da98b0 con 0x55f2c7dd0cb0 2024-08-03T20:57:48.632 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:48.628+0000 7f99cc6e9f40 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x55f2c7da9cd0 con 0x55f2c7db3360 2024-08-03T20:57:48.632 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:48.628+0000 7f99c4e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55f2c7db4ea0 0x55f2c7da9130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:48.632 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:48.628+0000 7f99c4e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55f2c7db4ea0 0x55f2c7da9130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:43338/0 (socket says 172.21.15.32:43338) 2024-08-03T20:57:48.632 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:48.628+0000 7f99c4e00640 1 -- 172.21.15.32:0/2171694035 learned_addr learned my addr 172.21.15.32:0/2171694035 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:57:48.632 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:48.628+0000 7f99c6200640 1 --2- 172.21.15.32:0/2171694035 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55f2c7dd0cb0 0x55f2c7d3d1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:48.632 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:48.629+0000 7f99c5800640 1 --2- 172.21.15.32:0/2171694035 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55f2c7db3360 0x55f2c7dcb8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:48.632 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:48.629+0000 7f99c4e00640 1 -- 172.21.15.32:0/2171694035 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55f2c7db3360 msgr2=0x55f2c7dcb8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:48.632 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:48.629+0000 7f99c4e00640 1 --2- 172.21.15.32:0/2171694035 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55f2c7db3360 0x55f2c7dcb8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:48.632 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:48.629+0000 7f99c4e00640 1 -- 172.21.15.32:0/2171694035 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55f2c7dd0cb0 msgr2=0x55f2c7d3d1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:48.633 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:48.629+0000 7f99c4e00640 1 --2- 172.21.15.32:0/2171694035 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55f2c7dd0cb0 0x55f2c7d3d1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:48.633 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:48.629+0000 7f99c4e00640 1 -- 172.21.15.32:0/2171694035 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f2c7da9670 con 0x55f2c7db4ea0 2024-08-03T20:57:48.633 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:48.629+0000 7f99c4e00640 1 --2- 172.21.15.32:0/2171694035 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55f2c7db4ea0 0x55f2c7da9130 secure :-1 s=READY pgs=1631 cs=0 l=1 rev1=1 crypto rx=0x7f99b400a6b0 tx=0x7f99b400ab70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:57:48.633 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:48.629+0000 7f99be000640 1 -- 172.21.15.32:0/2171694035 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99b4008990 con 0x55f2c7db4ea0 2024-08-03T20:57:48.633 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:48.629+0000 7f99be000640 1 -- 172.21.15.32:0/2171694035 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f99b4008b30 con 0x55f2c7db4ea0 2024-08-03T20:57:48.633 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:48.629+0000 7f99cc6e9f40 1 -- 172.21.15.32:0/2171694035 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x55f2c7d42dc0 con 0x55f2c7db4ea0 2024-08-03T20:57:48.633 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:48.629+0000 7f99be000640 1 -- 172.21.15.32:0/2171694035 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99b4016420 con 0x55f2c7db4ea0 2024-08-03T20:57:48.633 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:48.629+0000 7f99cc6e9f40 1 -- 172.21.15.32:0/2171694035 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f2c7d42ae0 con 0x55f2c7db4ea0 2024-08-03T20:57:48.634 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:48.632+0000 7f99be000640 1 -- 172.21.15.32:0/2171694035 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f99b401f440 con 0x55f2c7db4ea0 2024-08-03T20:57:48.634 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:48.632+0000 7f99be000640 1 --2- 172.21.15.32:0/2171694035 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f99a8075a20 0x7f99a8077f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:48.634 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:48.633+0000 7f99be000640 1 -- 172.21.15.32:0/2171694035 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(108..108 src has 1..108) ==== 7832+0+0 (secure 0 0 0) 0x7f99b40980b0 con 0x55f2c7db4ea0 2024-08-03T20:57:48.635 INFO:teuthology.orchestra.run.smithi032.stdout:Importing pool 2024-08-03T20:57:48.635 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:48.633+0000 7f99cc6e9f40 1 --2- 172.21.15.32:0/2171694035 >> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] conn(0x55f2c7cfc050 0x55f2c7cfc460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:48.636 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:48.633+0000 7f99c5800640 1 --2- 172.21.15.32:0/2171694035 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f99a8075a20 0x7f99a8077f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:48.636 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:48.633+0000 7f99cc6e9f40 1 -- 172.21.15.32:0/2171694035 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:1 12.10 12:09401a81:::601.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108) -- 0x55f2c7cfc9a0 con 0x55f2c7cfc050 2024-08-03T20:57:48.636 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:48.633+0000 7f99c6200640 1 --2- 172.21.15.32:0/2171694035 >> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] conn(0x55f2c7cfc050 0x55f2c7cfc460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:48.637 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:48.634+0000 7f99c5800640 1 --2- 172.21.15.32:0/2171694035 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f99a8075a20 0x7f99a8077f30 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x55f2c7dd3780 tx=0x7f99b8004af0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:57:48.637 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:48.634+0000 7f99c6200640 1 --2- 172.21.15.32:0/2171694035 >> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] conn(0x55f2c7cfc050 0x55f2c7cfc460 crc :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:57:48.638 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:48.636+0000 7f99c6200640 1 -- 172.21.15.32:0/2171694035 <== osd.0 v2:172.21.15.32:6816/3066156116 1 ==== osd_op_reply(1 601.00000000 [create] v108'691 uv665 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f99c000b040 con 0x55f2c7cfc050 2024-08-03T20:57:48.638 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:48.637+0000 7f99cc6e9f40 1 -- 172.21.15.32:0/2171694035 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:2 12.10 12:09401a81:::601.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e108) -- 0x55f2c7e75b80 con 0x55f2c7cfc050 2024-08-03T20:57:48.639 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:48.638+0000 7f99c6200640 1 -- 172.21.15.32:0/2171694035 <== osd.0 v2:172.21.15.32:6816/3066156116 2 ==== osd_op_reply(2 601.00000000 [delete] v108'692 uv692 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f99c000b040 con 0x55f2c7cfc050 2024-08-03T20:57:48.640 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:48.638+0000 7f99cc6e9f40 1 -- 172.21.15.32:0/2171694035 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:3 12.10 12:09401a81:::601.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108) -- 0x55f2c7e75f20 con 0x55f2c7cfc050 2024-08-03T20:57:48.641 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:48.640+0000 7f99c6200640 1 -- 172.21.15.32:0/2171694035 <== osd.0 v2:172.21.15.32:6816/3066156116 3 ==== osd_op_reply(3 601.00000000 [create] v108'693 uv693 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f99c000b040 con 0x55f2c7cfc050 2024-08-03T20:57:48.641 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::601.00000000:head# 2024-08-03T20:57:48.641 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:48.640+0000 7f99cc6e9f40 1 -- 172.21.15.32:0/2171694035 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:4 12.10 12:09401a81:::601.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108) -- 0x55f2c7e772d0 con 0x55f2c7cfc050 2024-08-03T20:57:48.642 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:48.641+0000 7f99c6200640 1 -- 172.21.15.32:0/2171694035 <== osd.0 v2:172.21.15.32:6816/3066156116 4 ==== osd_op_reply(4 601.00000000 [setxattr (30)] v108'694 uv694 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f99c000b040 con 0x55f2c7cfc050 2024-08-03T20:57:48.643 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:48.641+0000 7f99cc6e9f40 1 -- 172.21.15.32:0/2171694035 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:5 12.10 12:09401a81:::601.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108) -- 0x55f2c7e77670 con 0x55f2c7cfc050 2024-08-03T20:57:48.644 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:48.643+0000 7f99c6200640 1 -- 172.21.15.32:0/2171694035 <== osd.0 v2:172.21.15.32:6816/3066156116 5 ==== osd_op_reply(5 601.00000000 [setxattr (62)] v108'695 uv695 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f99c000b040 con 0x55f2c7cfc050 2024-08-03T20:57:48.644 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:48.643+0000 7f99cc6e9f40 1 -- 172.21.15.32:0/2171694035 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:6 12.10 12:09401a81:::601.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108) -- 0x55f2c7e762c0 con 0x55f2c7cfc050 2024-08-03T20:57:48.646 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:48.644+0000 7f99c6200640 1 -- 172.21.15.32:0/2171694035 <== osd.0 v2:172.21.15.32:6816/3066156116 6 ==== osd_op_reply(6 601.00000000 [omap-set-header] v108'696 uv696 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f99c000b040 con 0x55f2c7cfc050 2024-08-03T20:57:48.646 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:48.644+0000 7f99cc6e9f40 1 -- 172.21.15.32:0/2171694035 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:7 12.10 12:0bd6d154:::602.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108) -- 0x55f2c7e766b0 con 0x55f2c7cfc050 2024-08-03T20:57:48.647 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:48.646+0000 7f99c6200640 1 -- 172.21.15.32:0/2171694035 <== osd.0 v2:172.21.15.32:6816/3066156116 7 ==== osd_op_reply(7 602.00000000 [create] v108'697 uv671 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f99c000b040 con 0x55f2c7cfc050 2024-08-03T20:57:48.648 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:48.646+0000 7f99cc6e9f40 1 -- 172.21.15.32:0/2171694035 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:8 12.10 12:0bd6d154:::602.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e108) -- 0x55f2c7e76a50 con 0x55f2c7cfc050 2024-08-03T20:57:48.649 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:48.648+0000 7f99c6200640 1 -- 172.21.15.32:0/2171694035 <== osd.0 v2:172.21.15.32:6816/3066156116 8 ==== osd_op_reply(8 602.00000000 [delete] v108'698 uv698 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f99c000b040 con 0x55f2c7cfc050 2024-08-03T20:57:48.650 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:48.648+0000 7f99cc6e9f40 1 -- 172.21.15.32:0/2171694035 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:9 12.10 12:0bd6d154:::602.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108) -- 0x55f2c7e76a50 con 0x55f2c7cfc050 2024-08-03T20:57:48.651 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:48.649+0000 7f99c6200640 1 -- 172.21.15.32:0/2171694035 <== osd.0 v2:172.21.15.32:6816/3066156116 9 ==== osd_op_reply(9 602.00000000 [create] v108'699 uv699 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f99c000b040 con 0x55f2c7cfc050 2024-08-03T20:57:48.651 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::602.00000000:head# 2024-08-03T20:57:48.651 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:48.649+0000 7f99cc6e9f40 1 -- 172.21.15.32:0/2171694035 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:10 12.10 12:0bd6d154:::602.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108) -- 0x55f2c7e76a50 con 0x55f2c7cfc050 2024-08-03T20:57:48.652 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:48.651+0000 7f99c6200640 1 -- 172.21.15.32:0/2171694035 <== osd.0 v2:172.21.15.32:6816/3066156116 10 ==== osd_op_reply(10 602.00000000 [setxattr (30)] v108'700 uv700 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f99c000b040 con 0x55f2c7cfc050 2024-08-03T20:57:48.653 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:48.651+0000 7f99cc6e9f40 1 -- 172.21.15.32:0/2171694035 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:11 12.10 12:0bd6d154:::602.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108) -- 0x55f2c7e76a50 con 0x55f2c7cfc050 2024-08-03T20:57:48.654 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:48.652+0000 7f99c6200640 1 -- 172.21.15.32:0/2171694035 <== osd.0 v2:172.21.15.32:6816/3066156116 11 ==== osd_op_reply(11 602.00000000 [setxattr (62)] v108'701 uv701 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f99c000b040 con 0x55f2c7cfc050 2024-08-03T20:57:48.654 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:48.652+0000 7f99cc6e9f40 1 -- 172.21.15.32:0/2171694035 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:12 12.10 12:0bd6d154:::602.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108) -- 0x55f2c7e76a50 con 0x55f2c7cfc050 2024-08-03T20:57:48.655 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:48.654+0000 7f99c6200640 1 -- 172.21.15.32:0/2171694035 <== osd.0 v2:172.21.15.32:6816/3066156116 12 ==== osd_op_reply(12 602.00000000 [omap-set-header] v108'702 uv702 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f99c000b040 con 0x55f2c7cfc050 2024-08-03T20:57:48.655 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:48.654+0000 7f99cc6e9f40 1 -- 172.21.15.32:0/2171694035 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:13 12.10 12:0cf6ad0f:::10000000000.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108) -- 0x55f2c7e76a50 con 0x55f2c7cfc050 2024-08-03T20:57:48.657 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:48.656+0000 7f99c6200640 1 -- 172.21.15.32:0/2171694035 <== osd.0 v2:172.21.15.32:6816/3066156116 13 ==== osd_op_reply(13 10000000000.00000000 [create] v108'703 uv678 ondisk = -17 ((17) File exists)) ==== 164+0+0 (crc 0 0 0) 0x7f99c000b040 con 0x55f2c7cfc050 2024-08-03T20:57:48.657 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:48.656+0000 7f99cc6e9f40 1 -- 172.21.15.32:0/2171694035 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:14 12.10 12:0cf6ad0f:::10000000000.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e108) -- 0x55f2c7e76a50 con 0x55f2c7cfc050 2024-08-03T20:57:48.659 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:48.657+0000 7f99c6200640 1 -- 172.21.15.32:0/2171694035 <== osd.0 v2:172.21.15.32:6816/3066156116 14 ==== osd_op_reply(14 10000000000.00000000 [delete] v108'704 uv704 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f99c000b040 con 0x55f2c7cfc050 2024-08-03T20:57:48.659 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:48.657+0000 7f99cc6e9f40 1 -- 172.21.15.32:0/2171694035 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:15 12.10 12:0cf6ad0f:::10000000000.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108) -- 0x55f2c7e76a50 con 0x55f2c7cfc050 2024-08-03T20:57:48.660 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:48.659+0000 7f99c6200640 1 -- 172.21.15.32:0/2171694035 <== osd.0 v2:172.21.15.32:6816/3066156116 15 ==== osd_op_reply(15 10000000000.00000000 [create] v108'705 uv705 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f99c000b040 con 0x55f2c7cfc050 2024-08-03T20:57:48.660 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::10000000000.00000000:head# 2024-08-03T20:57:48.661 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:48.659+0000 7f99cc6e9f40 1 -- 172.21.15.32:0/2171694035 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:16 12.10 12:0cf6ad0f:::10000000000.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108) -- 0x55f2c7e76a50 con 0x55f2c7cfc050 2024-08-03T20:57:48.662 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:48.661+0000 7f99c6200640 1 -- 172.21.15.32:0/2171694035 <== osd.0 v2:172.21.15.32:6816/3066156116 16 ==== osd_op_reply(16 10000000000.00000000 [setxattr (30)] v108'706 uv706 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f99c000b040 con 0x55f2c7cfc050 2024-08-03T20:57:48.662 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:48.661+0000 7f99cc6e9f40 1 -- 172.21.15.32:0/2171694035 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:17 12.10 12:0cf6ad0f:::10000000000.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108) -- 0x55f2c7e76a50 con 0x55f2c7cfc050 2024-08-03T20:57:48.664 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:48.662+0000 7f99c6200640 1 -- 172.21.15.32:0/2171694035 <== osd.0 v2:172.21.15.32:6816/3066156116 17 ==== osd_op_reply(17 10000000000.00000000 [setxattr (62)] v108'707 uv707 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f99c000b040 con 0x55f2c7cfc050 2024-08-03T20:57:48.664 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:48.662+0000 7f99cc6e9f40 1 -- 172.21.15.32:0/2171694035 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:18 12.10 12:0cf6ad0f:::10000000000.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108) -- 0x55f2c7e76a50 con 0x55f2c7cfc050 2024-08-03T20:57:48.665 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:48.664+0000 7f99c6200640 1 -- 172.21.15.32:0/2171694035 <== osd.0 v2:172.21.15.32:6816/3066156116 18 ==== osd_op_reply(18 10000000000.00000000 [omap-set-header] v108'708 uv708 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f99c000b040 con 0x55f2c7cfc050 2024-08-03T20:57:48.665 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:48.664+0000 7f99cc6e9f40 1 -- 172.21.15.32:0/2171694035 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:19 12.10 12:0cf6ad0f:::10000000000.00000000:head [omap-set-vals in=593b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108) -- 0x55f2c7e77a10 con 0x55f2c7cfc050 2024-08-03T20:57:48.666 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:48.665+0000 7f99c6200640 1 -- 172.21.15.32:0/2171694035 <== osd.0 v2:172.21.15.32:6816/3066156116 19 ==== osd_op_reply(19 10000000000.00000000 [omap-set-vals] v108'709 uv709 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f99c000b040 con 0x55f2c7cfc050 2024-08-03T20:57:48.667 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:48.665+0000 7f99cc6e9f40 1 -- 172.21.15.32:0/2171694035 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:20 12.10 12:0d82a743:::600.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108) -- 0x55f2c7e77a10 con 0x55f2c7cfc050 2024-08-03T20:57:48.668 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:48.667+0000 7f99c6200640 1 -- 172.21.15.32:0/2171694035 <== osd.0 v2:172.21.15.32:6816/3066156116 20 ==== osd_op_reply(20 600.00000000 [create] v108'710 uv684 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f99c000b040 con 0x55f2c7cfc050 2024-08-03T20:57:48.668 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:48.667+0000 7f99cc6e9f40 1 -- 172.21.15.32:0/2171694035 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:21 12.10 12:0d82a743:::600.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e108) -- 0x55f2c7e77a10 con 0x55f2c7cfc050 2024-08-03T20:57:48.670 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:48.668+0000 7f99c6200640 1 -- 172.21.15.32:0/2171694035 <== osd.0 v2:172.21.15.32:6816/3066156116 21 ==== osd_op_reply(21 600.00000000 [delete] v108'711 uv711 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f99c000b040 con 0x55f2c7cfc050 2024-08-03T20:57:48.670 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:48.668+0000 7f99cc6e9f40 1 -- 172.21.15.32:0/2171694035 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:22 12.10 12:0d82a743:::600.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108) -- 0x55f2c7e77a10 con 0x55f2c7cfc050 2024-08-03T20:57:48.671 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:48.670+0000 7f99c6200640 1 -- 172.21.15.32:0/2171694035 <== osd.0 v2:172.21.15.32:6816/3066156116 22 ==== osd_op_reply(22 600.00000000 [create] v108'712 uv712 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f99c000b040 con 0x55f2c7cfc050 2024-08-03T20:57:48.672 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::600.00000000:head# 2024-08-03T20:57:48.672 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:48.670+0000 7f99cc6e9f40 1 -- 172.21.15.32:0/2171694035 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:23 12.10 12:0d82a743:::600.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108) -- 0x55f2c7e78a20 con 0x55f2c7cfc050 2024-08-03T20:57:48.673 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:48.671+0000 7f99c6200640 1 -- 172.21.15.32:0/2171694035 <== osd.0 v2:172.21.15.32:6816/3066156116 23 ==== osd_op_reply(23 600.00000000 [setxattr (30)] v108'713 uv713 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f99c000b040 con 0x55f2c7cfc050 2024-08-03T20:57:48.673 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:48.672+0000 7f99cc6e9f40 1 -- 172.21.15.32:0/2171694035 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:24 12.10 12:0d82a743:::600.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108) -- 0x55f2c7e78a20 con 0x55f2c7cfc050 2024-08-03T20:57:48.674 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:48.673+0000 7f99c6200640 1 -- 172.21.15.32:0/2171694035 <== osd.0 v2:172.21.15.32:6816/3066156116 24 ==== osd_op_reply(24 600.00000000 [setxattr (62)] v108'714 uv714 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f99c000b040 con 0x55f2c7cfc050 2024-08-03T20:57:48.674 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:48.673+0000 7f99cc6e9f40 1 -- 172.21.15.32:0/2171694035 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:25 12.10 12:0d82a743:::600.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108) -- 0x55f2c7e77a10 con 0x55f2c7cfc050 2024-08-03T20:57:48.677 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:48.674+0000 7f99c6200640 1 -- 172.21.15.32:0/2171694035 <== osd.0 v2:172.21.15.32:6816/3066156116 25 ==== osd_op_reply(25 600.00000000 [omap-set-header] v108'715 uv715 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f99c000b040 con 0x55f2c7cfc050 2024-08-03T20:57:48.677 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:48.675+0000 7f99cc6e9f40 1 -- 172.21.15.32:0/2171694035 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:26 12.10 12:0d89b25e:::603.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108) -- 0x55f2c7e77a10 con 0x55f2c7cfc050 2024-08-03T20:57:48.678 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:48.676+0000 7f99c6200640 1 -- 172.21.15.32:0/2171694035 <== osd.0 v2:172.21.15.32:6816/3066156116 26 ==== osd_op_reply(26 603.00000000 [create] v108'716 uv690 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f99c000b040 con 0x55f2c7cfc050 2024-08-03T20:57:48.678 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:48.676+0000 7f99cc6e9f40 1 -- 172.21.15.32:0/2171694035 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:27 12.10 12:0d89b25e:::603.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e108) -- 0x55f2c7e77a10 con 0x55f2c7cfc050 2024-08-03T20:57:48.679 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:48.678+0000 7f99c6200640 1 -- 172.21.15.32:0/2171694035 <== osd.0 v2:172.21.15.32:6816/3066156116 27 ==== osd_op_reply(27 603.00000000 [delete] v108'717 uv717 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f99c000b040 con 0x55f2c7cfc050 2024-08-03T20:57:48.679 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:48.678+0000 7f99cc6e9f40 1 -- 172.21.15.32:0/2171694035 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:28 12.10 12:0d89b25e:::603.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108) -- 0x55f2c7e77a10 con 0x55f2c7cfc050 2024-08-03T20:57:48.680 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:48.679+0000 7f99c6200640 1 -- 172.21.15.32:0/2171694035 <== osd.0 v2:172.21.15.32:6816/3066156116 28 ==== osd_op_reply(28 603.00000000 [create] v108'718 uv718 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f99c000b040 con 0x55f2c7cfc050 2024-08-03T20:57:48.680 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::603.00000000:head# 2024-08-03T20:57:48.681 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:48.679+0000 7f99cc6e9f40 1 -- 172.21.15.32:0/2171694035 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:29 12.10 12:0d89b25e:::603.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108) -- 0x55f2c7e78a20 con 0x55f2c7cfc050 2024-08-03T20:57:48.682 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:48.680+0000 7f99c6200640 1 -- 172.21.15.32:0/2171694035 <== osd.0 v2:172.21.15.32:6816/3066156116 29 ==== osd_op_reply(29 603.00000000 [setxattr (30)] v108'719 uv719 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f99c000b040 con 0x55f2c7cfc050 2024-08-03T20:57:48.682 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:48.681+0000 7f99cc6e9f40 1 -- 172.21.15.32:0/2171694035 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:30 12.10 12:0d89b25e:::603.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108) -- 0x55f2c7e78a20 con 0x55f2c7cfc050 2024-08-03T20:57:48.683 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:48.682+0000 7f99c6200640 1 -- 172.21.15.32:0/2171694035 <== osd.0 v2:172.21.15.32:6816/3066156116 30 ==== osd_op_reply(30 603.00000000 [setxattr (62)] v108'720 uv720 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f99c000b040 con 0x55f2c7cfc050 2024-08-03T20:57:48.683 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:48.682+0000 7f99cc6e9f40 1 -- 172.21.15.32:0/2171694035 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:31 12.10 12:0d89b25e:::603.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108) -- 0x55f2c7e77a10 con 0x55f2c7cfc050 2024-08-03T20:57:48.684 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:48.683+0000 7f99c6200640 1 -- 172.21.15.32:0/2171694035 <== osd.0 v2:172.21.15.32:6816/3066156116 31 ==== osd_op_reply(31 603.00000000 [omap-set-header] v108'721 uv721 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f99c000b040 con 0x55f2c7cfc050 2024-08-03T20:57:48.685 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:48.683+0000 7f99cc6e9f40 1 --2- 172.21.15.32:0/2171694035 >> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] conn(0x55f2c7e77f20 0x55f2c7e7a340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:48.685 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:48.683+0000 7f99cc6e9f40 1 -- 172.21.15.32:0/2171694035 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:32 12.14 12:287034c4:::1.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108) -- 0x55f2c7e7a880 con 0x55f2c7e77f20 2024-08-03T20:57:48.685 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:48.684+0000 7f99c5800640 1 --2- 172.21.15.32:0/2171694035 >> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] conn(0x55f2c7e77f20 0x55f2c7e7a340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:48.685 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:48.684+0000 7f99c5800640 1 --2- 172.21.15.32:0/2171694035 >> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] conn(0x55f2c7e77f20 0x55f2c7e7a340 crc :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:57:48.687 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:48.686+0000 7f99c5800640 1 -- 172.21.15.32:0/2171694035 <== osd.5 v2:172.21.15.118:6810/2724891770 1 ==== osd_op_reply(32 1.00000000.inode [create] v108'505 uv488 ondisk = -17 ((17) File exists)) ==== 160+0+0 (crc 0 0 0) 0x7f99b800a5e0 con 0x55f2c7e77f20 2024-08-03T20:57:48.687 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:48.686+0000 7f99cc6e9f40 1 -- 172.21.15.32:0/2171694035 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:33 12.14 12:287034c4:::1.00000000.inode:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e108) -- 0x55f2c7e7ac20 con 0x55f2c7e77f20 2024-08-03T20:57:48.689 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:48.688+0000 7f99c5800640 1 -- 172.21.15.32:0/2171694035 <== osd.5 v2:172.21.15.118:6810/2724891770 2 ==== osd_op_reply(33 1.00000000.inode [delete] v108'506 uv506 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f99b800a5e0 con 0x55f2c7e77f20 2024-08-03T20:57:48.689 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:48.688+0000 7f99cc6e9f40 1 -- 172.21.15.32:0/2171694035 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:34 12.14 12:287034c4:::1.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108) -- 0x55f2c7e7afc0 con 0x55f2c7e77f20 2024-08-03T20:57:48.691 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:48.689+0000 7f99c5800640 1 -- 172.21.15.32:0/2171694035 <== osd.5 v2:172.21.15.118:6810/2724891770 3 ==== osd_op_reply(34 1.00000000.inode [create] v108'507 uv507 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f99b800a5e0 con 0x55f2c7e77f20 2024-08-03T20:57:48.691 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::1.00000000.inode:head# 2024-08-03T20:57:48.691 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:48.689+0000 7f99cc6e9f40 1 -- 172.21.15.32:0/2171694035 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:35 12.14 12:287034c4:::1.00000000.inode:head [write 0~582 in=582b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108) -- 0x55f2c7e7b360 con 0x55f2c7e77f20 2024-08-03T20:57:48.692 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:48.691+0000 7f99c5800640 1 -- 172.21.15.32:0/2171694035 <== osd.5 v2:172.21.15.118:6810/2724891770 4 ==== osd_op_reply(35 1.00000000.inode [write 0~582] v108'508 uv508 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f99b800a5e0 con 0x55f2c7e77f20 2024-08-03T20:57:48.692 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:48.691+0000 7f99cc6e9f40 1 -- 172.21.15.32:0/2171694035 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:36 12.14 12:287034c4:::1.00000000.inode:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108) -- 0x55f2c7e7b700 con 0x55f2c7e77f20 2024-08-03T20:57:48.694 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:48.692+0000 7f99c5800640 1 -- 172.21.15.32:0/2171694035 <== osd.5 v2:172.21.15.118:6810/2724891770 5 ==== osd_op_reply(36 1.00000000.inode [omap-set-header] v108'509 uv509 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f99b800a5e0 con 0x55f2c7e77f20 2024-08-03T20:57:48.694 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:48.692+0000 7f99cc6e9f40 1 -- 172.21.15.32:0/2171694035 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:37 12.14 12:292cf221:::200.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108) -- 0x55f2c7e7baa0 con 0x55f2c7e77f20 2024-08-03T20:57:48.695 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:48.694+0000 7f99c5800640 1 -- 172.21.15.32:0/2171694035 <== osd.5 v2:172.21.15.118:6810/2724891770 6 ==== osd_op_reply(37 200.00000000 [create] v108'510 uv493 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f99b800a5e0 con 0x55f2c7e77f20 2024-08-03T20:57:48.695 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:48.694+0000 7f99cc6e9f40 1 -- 172.21.15.32:0/2171694035 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:38 12.14 12:292cf221:::200.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e108) -- 0x55f2c7e7be40 con 0x55f2c7e77f20 2024-08-03T20:57:48.697 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:48.695+0000 7f99c5800640 1 -- 172.21.15.32:0/2171694035 <== osd.5 v2:172.21.15.118:6810/2724891770 7 ==== osd_op_reply(38 200.00000000 [delete] v108'511 uv511 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f99b800a5e0 con 0x55f2c7e77f20 2024-08-03T20:57:48.697 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:48.695+0000 7f99cc6e9f40 1 -- 172.21.15.32:0/2171694035 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:39 12.14 12:292cf221:::200.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108) -- 0x55f2c7e7c1e0 con 0x55f2c7e77f20 2024-08-03T20:57:48.698 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::200.00000000:head# 2024-08-03T20:57:48.698 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:48.697+0000 7f99c5800640 1 -- 172.21.15.32:0/2171694035 <== osd.5 v2:172.21.15.118:6810/2724891770 8 ==== osd_op_reply(39 200.00000000 [create] v108'512 uv512 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f99b800a5e0 con 0x55f2c7e77f20 2024-08-03T20:57:48.699 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:48.697+0000 7f99cc6e9f40 1 -- 172.21.15.32:0/2171694035 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:40 12.14 12:292cf221:::200.00000000:head [write 0~90 in=90b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108) -- 0x55f2c7e7c1e0 con 0x55f2c7e77f20 2024-08-03T20:57:48.700 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:48.699+0000 7f99c5800640 1 -- 172.21.15.32:0/2171694035 <== osd.5 v2:172.21.15.118:6810/2724891770 9 ==== osd_op_reply(40 200.00000000 [write 0~90] v108'513 uv513 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f99b800a5e0 con 0x55f2c7e77f20 2024-08-03T20:57:48.700 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:48.699+0000 7f99cc6e9f40 1 -- 172.21.15.32:0/2171694035 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:41 12.14 12:292cf221:::200.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108) -- 0x55f2c7e7c1e0 con 0x55f2c7e77f20 2024-08-03T20:57:48.701 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:48.700+0000 7f99c5800640 1 -- 172.21.15.32:0/2171694035 <== osd.5 v2:172.21.15.118:6810/2724891770 10 ==== osd_op_reply(41 200.00000000 [omap-set-header] v108'514 uv514 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f99b800a5e0 con 0x55f2c7e77f20 2024-08-03T20:57:48.702 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:48.700+0000 7f99cc6e9f40 1 -- 172.21.15.32:0/2171694035 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:42 12.14 12:2e2fa760:::200.00000001:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108) -- 0x55f2c7e7c1e0 con 0x55f2c7e77f20 2024-08-03T20:57:48.705 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:48.702+0000 7f99c5800640 1 -- 172.21.15.32:0/2171694035 <== osd.5 v2:172.21.15.118:6810/2724891770 11 ==== osd_op_reply(42 200.00000001 [create] v108'515 uv498 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f99b800a5e0 con 0x55f2c7e77f20 2024-08-03T20:57:48.706 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:48.702+0000 7f99cc6e9f40 1 -- 172.21.15.32:0/2171694035 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:43 12.14 12:2e2fa760:::200.00000001:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e108) -- 0x55f2c7e7c1e0 con 0x55f2c7e77f20 2024-08-03T20:57:48.706 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:48.703+0000 7f99c5800640 1 -- 172.21.15.32:0/2171694035 <== osd.5 v2:172.21.15.118:6810/2724891770 12 ==== osd_op_reply(43 200.00000001 [delete] v108'516 uv516 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f99b800a5e0 con 0x55f2c7e77f20 2024-08-03T20:57:48.706 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:48.703+0000 7f99cc6e9f40 1 -- 172.21.15.32:0/2171694035 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:44 12.14 12:2e2fa760:::200.00000001:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108) -- 0x55f2c7e7c1e0 con 0x55f2c7e77f20 2024-08-03T20:57:48.706 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:48.705+0000 7f99c5800640 1 -- 172.21.15.32:0/2171694035 <== osd.5 v2:172.21.15.118:6810/2724891770 13 ==== osd_op_reply(44 200.00000001 [create] v108'517 uv517 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f99b800a5e0 con 0x55f2c7e77f20 2024-08-03T20:57:48.707 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::200.00000001:head# 2024-08-03T20:57:48.707 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:48.705+0000 7f99cc6e9f40 1 -- 172.21.15.32:0/2171694035 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:45 12.14 12:2e2fa760:::200.00000001:head [write 0~18118 in=18118b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108) -- 0x55f2c7e808d0 con 0x55f2c7e77f20 2024-08-03T20:57:48.708 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:48.706+0000 7f99c5800640 1 -- 172.21.15.32:0/2171694035 <== osd.5 v2:172.21.15.118:6810/2724891770 14 ==== osd_op_reply(45 200.00000001 [write 0~18118] v108'518 uv518 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f99b800a5e0 con 0x55f2c7e77f20 2024-08-03T20:57:48.708 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:48.707+0000 7f99cc6e9f40 1 -- 172.21.15.32:0/2171694035 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:46 12.14 12:2e2fa760:::200.00000001:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108) -- 0x55f2c7e808d0 con 0x55f2c7e77f20 2024-08-03T20:57:48.709 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:48.708+0000 7f99c5800640 1 -- 172.21.15.32:0/2171694035 <== osd.5 v2:172.21.15.118:6810/2724891770 15 ==== osd_op_reply(46 200.00000001 [omap-set-header] v108'519 uv519 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f99b800a5e0 con 0x55f2c7e77f20 2024-08-03T20:57:48.709 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:48.708+0000 7f99cc6e9f40 1 -- 172.21.15.32:0/2171694035 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:47 12.14 12:2f55791f:::606.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108) -- 0x55f2c7e7c1e0 con 0x55f2c7e77f20 2024-08-03T20:57:48.711 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:48.710+0000 7f99c5800640 1 -- 172.21.15.32:0/2171694035 <== osd.5 v2:172.21.15.118:6810/2724891770 16 ==== osd_op_reply(47 606.00000000 [create] v108'520 uv504 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f99b800a5e0 con 0x55f2c7e77f20 2024-08-03T20:57:48.711 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:48.710+0000 7f99cc6e9f40 1 -- 172.21.15.32:0/2171694035 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:48 12.14 12:2f55791f:::606.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e108) -- 0x55f2c7e7c1e0 con 0x55f2c7e77f20 2024-08-03T20:57:48.713 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:48.711+0000 7f99c5800640 1 -- 172.21.15.32:0/2171694035 <== osd.5 v2:172.21.15.118:6810/2724891770 17 ==== osd_op_reply(48 606.00000000 [delete] v108'521 uv521 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f99b800a5e0 con 0x55f2c7e77f20 2024-08-03T20:57:48.713 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:48.711+0000 7f99cc6e9f40 1 -- 172.21.15.32:0/2171694035 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:49 12.14 12:2f55791f:::606.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108) -- 0x55f2c7e7c1e0 con 0x55f2c7e77f20 2024-08-03T20:57:48.714 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::606.00000000:head# 2024-08-03T20:57:48.714 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:48.713+0000 7f99c5800640 1 -- 172.21.15.32:0/2171694035 <== osd.5 v2:172.21.15.118:6810/2724891770 18 ==== osd_op_reply(49 606.00000000 [create] v108'522 uv522 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f99b800a5e0 con 0x55f2c7e77f20 2024-08-03T20:57:48.714 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:48.713+0000 7f99cc6e9f40 1 -- 172.21.15.32:0/2171694035 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:50 12.14 12:2f55791f:::606.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108) -- 0x55f2c7e7d1f0 con 0x55f2c7e77f20 2024-08-03T20:57:48.716 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:48.714+0000 7f99c5800640 1 -- 172.21.15.32:0/2171694035 <== osd.5 v2:172.21.15.118:6810/2724891770 19 ==== osd_op_reply(50 606.00000000 [setxattr (30)] v108'523 uv523 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f99b800a5e0 con 0x55f2c7e77f20 2024-08-03T20:57:48.716 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:48.714+0000 7f99cc6e9f40 1 -- 172.21.15.32:0/2171694035 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:51 12.14 12:2f55791f:::606.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108) -- 0x55f2c7e7d1f0 con 0x55f2c7e77f20 2024-08-03T20:57:48.717 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:48.716+0000 7f99c5800640 1 -- 172.21.15.32:0/2171694035 <== osd.5 v2:172.21.15.118:6810/2724891770 20 ==== osd_op_reply(51 606.00000000 [setxattr (62)] v108'524 uv524 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f99b800a5e0 con 0x55f2c7e77f20 2024-08-03T20:57:48.717 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:48.716+0000 7f99cc6e9f40 1 -- 172.21.15.32:0/2171694035 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:52 12.14 12:2f55791f:::606.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108) -- 0x55f2c7e7c1e0 con 0x55f2c7e77f20 2024-08-03T20:57:48.719 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:48.717+0000 7f99c5800640 1 -- 172.21.15.32:0/2171694035 <== osd.5 v2:172.21.15.118:6810/2724891770 21 ==== osd_op_reply(52 606.00000000 [omap-set-header] v108'525 uv525 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f99b800a5e0 con 0x55f2c7e77f20 2024-08-03T20:57:48.719 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:48.717+0000 7f99cc6e9f40 1 -- 172.21.15.32:0/2171694035 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:53 12.1c 12:3ed09add:::607.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108) -- 0x55f2c7e7c1e0 con 0x55f2c7cfc050 2024-08-03T20:57:48.720 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:48.719+0000 7f99c6200640 1 -- 172.21.15.32:0/2171694035 <== osd.0 v2:172.21.15.32:6816/3066156116 32 ==== osd_op_reply(53 607.00000000 [create] v108'135 uv134 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f99c000b040 con 0x55f2c7cfc050 2024-08-03T20:57:48.720 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:48.719+0000 7f99cc6e9f40 1 -- 172.21.15.32:0/2171694035 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:54 12.1c 12:3ed09add:::607.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e108) -- 0x55f2c7e7c1e0 con 0x55f2c7cfc050 2024-08-03T20:57:48.721 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:48.720+0000 7f99c6200640 1 -- 172.21.15.32:0/2171694035 <== osd.0 v2:172.21.15.32:6816/3066156116 33 ==== osd_op_reply(54 607.00000000 [delete] v108'136 uv136 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f99c000b040 con 0x55f2c7cfc050 2024-08-03T20:57:48.721 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:48.720+0000 7f99cc6e9f40 1 -- 172.21.15.32:0/2171694035 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:55 12.1c 12:3ed09add:::607.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108) -- 0x55f2c7e7c1e0 con 0x55f2c7cfc050 2024-08-03T20:57:48.722 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:48.721+0000 7f99c6200640 1 -- 172.21.15.32:0/2171694035 <== osd.0 v2:172.21.15.32:6816/3066156116 34 ==== osd_op_reply(55 607.00000000 [create] v108'137 uv137 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f99c000b040 con 0x55f2c7cfc050 2024-08-03T20:57:48.723 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::607.00000000:head# 2024-08-03T20:57:48.723 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:48.721+0000 7f99cc6e9f40 1 -- 172.21.15.32:0/2171694035 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:56 12.1c 12:3ed09add:::607.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108) -- 0x55f2c7e7d1f0 con 0x55f2c7cfc050 2024-08-03T20:57:48.724 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:48.723+0000 7f99c6200640 1 -- 172.21.15.32:0/2171694035 <== osd.0 v2:172.21.15.32:6816/3066156116 35 ==== osd_op_reply(56 607.00000000 [setxattr (30)] v108'138 uv138 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f99c000b040 con 0x55f2c7cfc050 2024-08-03T20:57:48.724 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:48.723+0000 7f99cc6e9f40 1 -- 172.21.15.32:0/2171694035 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:57 12.1c 12:3ed09add:::607.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108) -- 0x55f2c7e7d1f0 con 0x55f2c7cfc050 2024-08-03T20:57:48.725 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:48.724+0000 7f99c6200640 1 -- 172.21.15.32:0/2171694035 <== osd.0 v2:172.21.15.32:6816/3066156116 36 ==== osd_op_reply(57 607.00000000 [setxattr (62)] v108'139 uv139 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f99c000b040 con 0x55f2c7cfc050 2024-08-03T20:57:48.725 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:48.724+0000 7f99cc6e9f40 1 -- 172.21.15.32:0/2171694035 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:58 12.1c 12:3ed09add:::607.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108) -- 0x55f2c7e7c1e0 con 0x55f2c7cfc050 2024-08-03T20:57:48.727 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:48.725+0000 7f99c6200640 1 -- 172.21.15.32:0/2171694035 <== osd.0 v2:172.21.15.32:6816/3066156116 37 ==== osd_op_reply(58 607.00000000 [omap-set-header] v108'140 uv140 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f99c000b040 con 0x55f2c7cfc050 2024-08-03T20:57:48.727 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:48.725+0000 7f99cc6e9f40 1 --2- 172.21.15.32:0/2171694035 >> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] conn(0x55f2c7e7c6f0 0x55f2c7e7eb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:48.727 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:48.725+0000 7f99cc6e9f40 1 -- 172.21.15.32:0/2171694035 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:59 12.6 12:654134d2:::mds0_openfiles.0:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108) -- 0x55f2c7e7f1c0 con 0x55f2c7e7c6f0 2024-08-03T20:57:48.727 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:48.726+0000 7f99c4e00640 1 --2- 172.21.15.32:0/2171694035 >> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] conn(0x55f2c7e7c6f0 0x55f2c7e7eb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:48.727 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:48.726+0000 7f99c4e00640 1 --2- 172.21.15.32:0/2171694035 >> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] conn(0x55f2c7e7c6f0 0x55f2c7e7eb10 crc :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:57:48.729 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:48.727+0000 7f99c4e00640 1 -- 172.21.15.32:0/2171694035 <== osd.1 v2:172.21.15.32:6824/1551241759 1 ==== osd_op_reply(59 mds0_openfiles.0 [create] v108'106 uv105 ondisk = -17 ((17) File exists)) ==== 160+0+0 (crc 0 0 0) 0x7f99b401c020 con 0x55f2c7e7c6f0 2024-08-03T20:57:48.729 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:48.727+0000 7f99cc6e9f40 1 -- 172.21.15.32:0/2171694035 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:60 12.6 12:654134d2:::mds0_openfiles.0:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e108) -- 0x55f2c7e7f6d0 con 0x55f2c7e7c6f0 2024-08-03T20:57:48.730 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:48.729+0000 7f99c4e00640 1 -- 172.21.15.32:0/2171694035 <== osd.1 v2:172.21.15.32:6824/1551241759 2 ==== osd_op_reply(60 mds0_openfiles.0 [delete] v108'107 uv107 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f99b401c020 con 0x55f2c7e7c6f0 2024-08-03T20:57:48.730 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:48.729+0000 7f99cc6e9f40 1 -- 172.21.15.32:0/2171694035 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:61 12.6 12:654134d2:::mds0_openfiles.0:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108) -- 0x55f2c7e7fb90 con 0x55f2c7e7c6f0 2024-08-03T20:57:48.731 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:48.730+0000 7f99c4e00640 1 -- 172.21.15.32:0/2171694035 <== osd.1 v2:172.21.15.32:6824/1551241759 3 ==== osd_op_reply(61 mds0_openfiles.0 [create] v108'108 uv108 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f99b401c020 con 0x55f2c7e7c6f0 2024-08-03T20:57:48.731 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds0_openfiles.0:head# 2024-08-03T20:57:48.732 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:48.730+0000 7f99cc6e9f40 1 -- 172.21.15.32:0/2171694035 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:62 12.6 12:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108) -- 0x55f2c7d40df0 con 0x55f2c7e7c6f0 2024-08-03T20:57:48.733 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:48.731+0000 7f99c4e00640 1 -- 172.21.15.32:0/2171694035 <== osd.1 v2:172.21.15.32:6824/1551241759 4 ==== osd_op_reply(62 mds0_openfiles.0 [omap-set-header] v108'109 uv109 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f99b401c020 con 0x55f2c7e7c6f0 2024-08-03T20:57:48.733 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:48.731+0000 7f99cc6e9f40 1 -- 172.21.15.32:0/2171694035 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:63 12.e 12:717a0223:::608.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108) -- 0x55f2c7e804f0 con 0x55f2c7cfc050 2024-08-03T20:57:48.734 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:48.733+0000 7f99c6200640 1 -- 172.21.15.32:0/2171694035 <== osd.0 v2:172.21.15.32:6816/3066156116 38 ==== osd_op_reply(63 608.00000000 [create] v108'135 uv134 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f99c000b040 con 0x55f2c7cfc050 2024-08-03T20:57:48.735 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:48.733+0000 7f99cc6e9f40 1 -- 172.21.15.32:0/2171694035 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:64 12.e 12:717a0223:::608.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e108) -- 0x55f2c7e804f0 con 0x55f2c7cfc050 2024-08-03T20:57:48.736 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:48.735+0000 7f99c6200640 1 -- 172.21.15.32:0/2171694035 <== osd.0 v2:172.21.15.32:6816/3066156116 39 ==== osd_op_reply(64 608.00000000 [delete] v108'136 uv136 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f99c000b040 con 0x55f2c7cfc050 2024-08-03T20:57:48.736 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:48.735+0000 7f99cc6e9f40 1 -- 172.21.15.32:0/2171694035 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:65 12.e 12:717a0223:::608.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108) -- 0x55f2c7e804f0 con 0x55f2c7cfc050 2024-08-03T20:57:48.738 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:48.736+0000 7f99c6200640 1 -- 172.21.15.32:0/2171694035 <== osd.0 v2:172.21.15.32:6816/3066156116 40 ==== osd_op_reply(65 608.00000000 [create] v108'137 uv137 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f99c000b040 con 0x55f2c7cfc050 2024-08-03T20:57:48.738 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::608.00000000:head# 2024-08-03T20:57:48.738 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:48.736+0000 7f99cc6e9f40 1 -- 172.21.15.32:0/2171694035 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:66 12.e 12:717a0223:::608.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108) -- 0x55f2c7e804f0 con 0x55f2c7cfc050 2024-08-03T20:57:48.739 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:48.738+0000 7f99c6200640 1 -- 172.21.15.32:0/2171694035 <== osd.0 v2:172.21.15.32:6816/3066156116 41 ==== osd_op_reply(66 608.00000000 [setxattr (30)] v108'138 uv138 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f99c000b040 con 0x55f2c7cfc050 2024-08-03T20:57:48.739 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:48.738+0000 7f99cc6e9f40 1 -- 172.21.15.32:0/2171694035 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:67 12.e 12:717a0223:::608.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108) -- 0x55f2c7e804f0 con 0x55f2c7cfc050 2024-08-03T20:57:48.741 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:48.739+0000 7f99c6200640 1 -- 172.21.15.32:0/2171694035 <== osd.0 v2:172.21.15.32:6816/3066156116 42 ==== osd_op_reply(67 608.00000000 [setxattr (62)] v108'139 uv139 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f99c000b040 con 0x55f2c7cfc050 2024-08-03T20:57:48.741 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:48.739+0000 7f99cc6e9f40 1 -- 172.21.15.32:0/2171694035 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:68 12.e 12:717a0223:::608.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108) -- 0x55f2c7e804f0 con 0x55f2c7cfc050 2024-08-03T20:57:48.742 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:48.741+0000 7f99c6200640 1 -- 172.21.15.32:0/2171694035 <== osd.0 v2:172.21.15.32:6816/3066156116 43 ==== osd_op_reply(68 608.00000000 [omap-set-header] v108'140 uv140 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f99c000b040 con 0x55f2c7cfc050 2024-08-03T20:57:48.743 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:48.741+0000 7f99cc6e9f40 1 --2- 172.21.15.32:0/2171694035 >> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] conn(0x55f2c7e80e10 0x55f2c7e81220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:48.743 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:48.741+0000 7f99cc6e9f40 1 -- 172.21.15.32:0/2171694035 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:69 12.15 12:a93a17c2:::604.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108) -- 0x55f2c7e83940 con 0x55f2c7e80e10 2024-08-03T20:57:48.743 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:48.741+0000 7f99c6200640 1 --2- 172.21.15.32:0/2171694035 >> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] conn(0x55f2c7e80e10 0x55f2c7e81220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:48.743 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:48.742+0000 7f99c6200640 1 --2- 172.21.15.32:0/2171694035 >> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] conn(0x55f2c7e80e10 0x55f2c7e81220 crc :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:57:48.745 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:48.743+0000 7f99c6200640 1 -- 172.21.15.32:0/2171694035 <== osd.6 v2:172.21.15.118:6825/801947386 1 ==== osd_op_reply(69 604.00000000 [create] v108'249 uv243 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f99c000b040 con 0x55f2c7e80e10 2024-08-03T20:57:48.745 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:48.744+0000 7f99cc6e9f40 1 -- 172.21.15.32:0/2171694035 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:70 12.15 12:a93a17c2:::604.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e108) -- 0x55f2c7e83940 con 0x55f2c7e80e10 2024-08-03T20:57:48.746 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:48.745+0000 7f99c6200640 1 -- 172.21.15.32:0/2171694035 <== osd.6 v2:172.21.15.118:6825/801947386 2 ==== osd_op_reply(70 604.00000000 [delete] v108'250 uv250 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f99c000b040 con 0x55f2c7e80e10 2024-08-03T20:57:48.747 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:48.745+0000 7f99cc6e9f40 1 -- 172.21.15.32:0/2171694035 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:71 12.15 12:a93a17c2:::604.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108) -- 0x55f2c7e83940 con 0x55f2c7e80e10 2024-08-03T20:57:48.748 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::604.00000000:head# 2024-08-03T20:57:48.748 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:48.746+0000 7f99c6200640 1 -- 172.21.15.32:0/2171694035 <== osd.6 v2:172.21.15.118:6825/801947386 3 ==== osd_op_reply(71 604.00000000 [create] v108'251 uv251 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f99c000b040 con 0x55f2c7e80e10 2024-08-03T20:57:48.748 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:48.746+0000 7f99cc6e9f40 1 -- 172.21.15.32:0/2171694035 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:72 12.15 12:a93a17c2:::604.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108) -- 0x55f2c7e849d0 con 0x55f2c7e80e10 2024-08-03T20:57:48.749 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:48.748+0000 7f99c6200640 1 -- 172.21.15.32:0/2171694035 <== osd.6 v2:172.21.15.118:6825/801947386 4 ==== osd_op_reply(72 604.00000000 [setxattr (30)] v108'252 uv252 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f99c000b040 con 0x55f2c7e80e10 2024-08-03T20:57:48.749 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:48.748+0000 7f99cc6e9f40 1 -- 172.21.15.32:0/2171694035 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:73 12.15 12:a93a17c2:::604.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108) -- 0x55f2c7e84990 con 0x55f2c7e80e10 2024-08-03T20:57:48.751 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:48.749+0000 7f99c6200640 1 -- 172.21.15.32:0/2171694035 <== osd.6 v2:172.21.15.118:6825/801947386 5 ==== osd_op_reply(73 604.00000000 [setxattr (62)] v108'253 uv253 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f99c000b040 con 0x55f2c7e80e10 2024-08-03T20:57:48.751 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:48.749+0000 7f99cc6e9f40 1 -- 172.21.15.32:0/2171694035 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:74 12.15 12:a93a17c2:::604.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108) -- 0x55f2c7e84990 con 0x55f2c7e80e10 2024-08-03T20:57:48.752 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:48.751+0000 7f99c6200640 1 -- 172.21.15.32:0/2171694035 <== osd.6 v2:172.21.15.118:6825/801947386 6 ==== osd_op_reply(74 604.00000000 [omap-set-header] v108'254 uv254 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f99c000b040 con 0x55f2c7e80e10 2024-08-03T20:57:48.752 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:48.751+0000 7f99cc6e9f40 1 -- 172.21.15.32:0/2171694035 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:75 12.15 12:aa448500:::500.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108) -- 0x55f2c7e84970 con 0x55f2c7e80e10 2024-08-03T20:57:48.753 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:48.752+0000 7f99c6200640 1 -- 172.21.15.32:0/2171694035 <== osd.6 v2:172.21.15.118:6825/801947386 7 ==== osd_op_reply(75 500.00000000 [create] v108'255 uv248 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f99c000b040 con 0x55f2c7e80e10 2024-08-03T20:57:48.754 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:48.752+0000 7f99cc6e9f40 1 -- 172.21.15.32:0/2171694035 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:76 12.15 12:aa448500:::500.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e108) -- 0x55f2c7e84970 con 0x55f2c7e80e10 2024-08-03T20:57:48.755 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:48.754+0000 7f99c6200640 1 -- 172.21.15.32:0/2171694035 <== osd.6 v2:172.21.15.118:6825/801947386 8 ==== osd_op_reply(76 500.00000000 [delete] v108'256 uv256 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f99c000b040 con 0x55f2c7e80e10 2024-08-03T20:57:48.755 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:48.754+0000 7f99cc6e9f40 1 -- 172.21.15.32:0/2171694035 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:77 12.15 12:aa448500:::500.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108) -- 0x55f2c7e83940 con 0x55f2c7e80e10 2024-08-03T20:57:48.757 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::500.00000000:head# 2024-08-03T20:57:48.757 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:48.755+0000 7f99c6200640 1 -- 172.21.15.32:0/2171694035 <== osd.6 v2:172.21.15.118:6825/801947386 9 ==== osd_op_reply(77 500.00000000 [create] v108'257 uv257 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f99c000b040 con 0x55f2c7e80e10 2024-08-03T20:57:48.757 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:48.755+0000 7f99cc6e9f40 1 -- 172.21.15.32:0/2171694035 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:78 12.15 12:aa448500:::500.00000000:head [write 0~90 in=90b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108) -- 0x55f2c7e83940 con 0x55f2c7e80e10 2024-08-03T20:57:48.758 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:48.757+0000 7f99c6200640 1 -- 172.21.15.32:0/2171694035 <== osd.6 v2:172.21.15.118:6825/801947386 10 ==== osd_op_reply(78 500.00000000 [write 0~90] v108'258 uv258 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f99c000b040 con 0x55f2c7e80e10 2024-08-03T20:57:48.758 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:48.757+0000 7f99cc6e9f40 1 -- 172.21.15.32:0/2171694035 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:79 12.15 12:aa448500:::500.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108) -- 0x55f2c7e83940 con 0x55f2c7e80e10 2024-08-03T20:57:48.759 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:48.758+0000 7f99c6200640 1 -- 172.21.15.32:0/2171694035 <== osd.6 v2:172.21.15.118:6825/801947386 11 ==== osd_op_reply(79 500.00000000 [omap-set-header] v108'259 uv259 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f99c000b040 con 0x55f2c7e80e10 2024-08-03T20:57:48.760 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:48.758+0000 7f99cc6e9f40 1 --2- 172.21.15.32:0/2171694035 >> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] conn(0x55f2c7e84210 0x55f2c7e86e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:48.760 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:48.758+0000 7f99cc6e9f40 1 -- 172.21.15.32:0/2171694035 --> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] -- osd_op(unknown.0.0:80 12.d 12:b50e409b:::mds_snaptable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108) -- 0x55f2c7e83940 con 0x55f2c7e84210 2024-08-03T20:57:48.760 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:48.759+0000 7f99c5800640 1 --2- 172.21.15.32:0/2171694035 >> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] conn(0x55f2c7e84210 0x55f2c7e86e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:48.760 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:48.759+0000 7f99c5800640 1 --2- 172.21.15.32:0/2171694035 >> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] conn(0x55f2c7e84210 0x55f2c7e86e60 crc :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:57:48.762 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:48.761+0000 7f99c5800640 1 -- 172.21.15.32:0/2171694035 <== osd.7 v2:172.21.15.118:6818/653000048 1 ==== osd_op_reply(80 mds_snaptable [create] v108'112 uv111 ondisk = -17 ((17) File exists)) ==== 157+0+0 (crc 0 0 0) 0x7f99b800a5e0 con 0x55f2c7e84210 2024-08-03T20:57:48.762 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:48.761+0000 7f99cc6e9f40 1 -- 172.21.15.32:0/2171694035 --> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] -- osd_op(unknown.0.0:81 12.d 12:b50e409b:::mds_snaptable:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e108) -- 0x55f2c7e83940 con 0x55f2c7e84210 2024-08-03T20:57:48.763 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:48.762+0000 7f99c5800640 1 -- 172.21.15.32:0/2171694035 <== osd.7 v2:172.21.15.118:6818/653000048 2 ==== osd_op_reply(81 mds_snaptable [delete] v108'113 uv113 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f99b800a5e0 con 0x55f2c7e84210 2024-08-03T20:57:48.763 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:48.762+0000 7f99cc6e9f40 1 -- 172.21.15.32:0/2171694035 --> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] -- osd_op(unknown.0.0:82 12.d 12:b50e409b:::mds_snaptable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108) -- 0x55f2c7e83940 con 0x55f2c7e84210 2024-08-03T20:57:48.764 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:48.763+0000 7f99c5800640 1 -- 172.21.15.32:0/2171694035 <== osd.7 v2:172.21.15.118:6818/653000048 3 ==== osd_op_reply(82 mds_snaptable [create] v108'114 uv114 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f99b800a5e0 con 0x55f2c7e84210 2024-08-03T20:57:48.765 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds_snaptable:head# 2024-08-03T20:57:48.765 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:48.763+0000 7f99cc6e9f40 1 -- 172.21.15.32:0/2171694035 --> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] -- osd_op(unknown.0.0:83 12.d 12:b50e409b:::mds_snaptable:head [write 0~70 in=70b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108) -- 0x55f2c7e873a0 con 0x55f2c7e84210 2024-08-03T20:57:48.766 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:48.765+0000 7f99c5800640 1 -- 172.21.15.32:0/2171694035 <== osd.7 v2:172.21.15.118:6818/653000048 4 ==== osd_op_reply(83 mds_snaptable [write 0~70] v108'115 uv115 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f99b800a5e0 con 0x55f2c7e84210 2024-08-03T20:57:48.766 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:48.765+0000 7f99cc6e9f40 1 -- 172.21.15.32:0/2171694035 --> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] -- osd_op(unknown.0.0:84 12.d 12:b50e409b:::mds_snaptable:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108) -- 0x55f2c7e873a0 con 0x55f2c7e84210 2024-08-03T20:57:48.767 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:48.766+0000 7f99c5800640 1 -- 172.21.15.32:0/2171694035 <== osd.7 v2:172.21.15.118:6818/653000048 5 ==== osd_op_reply(84 mds_snaptable [omap-set-header] v108'116 uv116 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f99b800a5e0 con 0x55f2c7e84210 2024-08-03T20:57:48.768 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:48.766+0000 7f99cc6e9f40 1 -- 172.21.15.32:0/2171694035 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:85 12.1d 12:b871830b:::605.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108) -- 0x55f2c7e873a0 con 0x55f2c7cfc050 2024-08-03T20:57:48.769 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:48.768+0000 7f99c6200640 1 -- 172.21.15.32:0/2171694035 <== osd.0 v2:172.21.15.32:6816/3066156116 44 ==== osd_op_reply(85 605.00000000 [create] v108'135 uv134 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f99c000b040 con 0x55f2c7cfc050 2024-08-03T20:57:48.769 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:48.768+0000 7f99cc6e9f40 1 -- 172.21.15.32:0/2171694035 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:86 12.1d 12:b871830b:::605.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e108) -- 0x55f2c7e873a0 con 0x55f2c7cfc050 2024-08-03T20:57:48.770 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:48.769+0000 7f99c6200640 1 -- 172.21.15.32:0/2171694035 <== osd.0 v2:172.21.15.32:6816/3066156116 45 ==== osd_op_reply(86 605.00000000 [delete] v108'136 uv136 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f99c000b040 con 0x55f2c7cfc050 2024-08-03T20:57:48.770 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:48.769+0000 7f99cc6e9f40 1 -- 172.21.15.32:0/2171694035 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:87 12.1d 12:b871830b:::605.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108) -- 0x55f2c7e873a0 con 0x55f2c7cfc050 2024-08-03T20:57:48.771 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::605.00000000:head# 2024-08-03T20:57:48.772 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:48.770+0000 7f99c6200640 1 -- 172.21.15.32:0/2171694035 <== osd.0 v2:172.21.15.32:6816/3066156116 46 ==== osd_op_reply(87 605.00000000 [create] v108'137 uv137 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f99c000b040 con 0x55f2c7cfc050 2024-08-03T20:57:48.772 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:48.770+0000 7f99cc6e9f40 1 -- 172.21.15.32:0/2171694035 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:88 12.1d 12:b871830b:::605.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108) -- 0x55f2c7e883b0 con 0x55f2c7cfc050 2024-08-03T20:57:48.773 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:48.771+0000 7f99c6200640 1 -- 172.21.15.32:0/2171694035 <== osd.0 v2:172.21.15.32:6816/3066156116 47 ==== osd_op_reply(88 605.00000000 [setxattr (30)] v108'138 uv138 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f99c000b040 con 0x55f2c7cfc050 2024-08-03T20:57:48.773 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:48.771+0000 7f99cc6e9f40 1 -- 172.21.15.32:0/2171694035 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:89 12.1d 12:b871830b:::605.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108) -- 0x55f2c7e883b0 con 0x55f2c7cfc050 2024-08-03T20:57:48.774 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:48.773+0000 7f99c6200640 1 -- 172.21.15.32:0/2171694035 <== osd.0 v2:172.21.15.32:6816/3066156116 48 ==== osd_op_reply(89 605.00000000 [setxattr (62)] v108'139 uv139 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f99c000b040 con 0x55f2c7cfc050 2024-08-03T20:57:48.774 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:48.773+0000 7f99cc6e9f40 1 -- 172.21.15.32:0/2171694035 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:90 12.1d 12:b871830b:::605.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108) -- 0x55f2c7e873a0 con 0x55f2c7cfc050 2024-08-03T20:57:48.775 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:48.774+0000 7f99c6200640 1 -- 172.21.15.32:0/2171694035 <== osd.0 v2:172.21.15.32:6816/3066156116 49 ==== osd_op_reply(90 605.00000000 [omap-set-header] v108'140 uv140 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f99c000b040 con 0x55f2c7cfc050 2024-08-03T20:57:48.775 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:48.774+0000 7f99cc6e9f40 1 -- 172.21.15.32:0/2171694035 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:91 12.13 12:c91d4a1d:::mds0_inotable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108) -- 0x55f2c7e873a0 con 0x55f2c7e7c6f0 2024-08-03T20:57:48.777 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:48.775+0000 7f99c4e00640 1 -- 172.21.15.32:0/2171694035 <== osd.1 v2:172.21.15.32:6824/1551241759 5 ==== osd_op_reply(91 mds0_inotable [create] v108'223 uv217 ondisk = -17 ((17) File exists)) ==== 157+0+0 (crc 0 0 0) 0x7f99b401c020 con 0x55f2c7e7c6f0 2024-08-03T20:57:48.777 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:48.775+0000 7f99cc6e9f40 1 -- 172.21.15.32:0/2171694035 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:92 12.13 12:c91d4a1d:::mds0_inotable:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e108) -- 0x55f2c7e87740 con 0x55f2c7e7c6f0 2024-08-03T20:57:48.778 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:48.777+0000 7f99c4e00640 1 -- 172.21.15.32:0/2171694035 <== osd.1 v2:172.21.15.32:6824/1551241759 6 ==== osd_op_reply(92 mds0_inotable [delete] v108'224 uv224 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f99b401c020 con 0x55f2c7e7c6f0 2024-08-03T20:57:48.778 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:48.777+0000 7f99cc6e9f40 1 -- 172.21.15.32:0/2171694035 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:93 12.13 12:c91d4a1d:::mds0_inotable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108) -- 0x55f2c7e87ae0 con 0x55f2c7e7c6f0 2024-08-03T20:57:48.779 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds0_inotable:head# 2024-08-03T20:57:48.780 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:48.778+0000 7f99c4e00640 1 -- 172.21.15.32:0/2171694035 <== osd.1 v2:172.21.15.32:6824/1551241759 7 ==== osd_op_reply(93 mds0_inotable [create] v108'225 uv225 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f99b401c020 con 0x55f2c7e7c6f0 2024-08-03T20:57:48.780 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:48.778+0000 7f99cc6e9f40 1 -- 172.21.15.32:0/2171694035 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:94 12.13 12:c91d4a1d:::mds0_inotable:head [write 0~34 in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108) -- 0x55f2c7e87ae0 con 0x55f2c7e7c6f0 2024-08-03T20:57:48.781 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:48.779+0000 7f99c4e00640 1 -- 172.21.15.32:0/2171694035 <== osd.1 v2:172.21.15.32:6824/1551241759 8 ==== osd_op_reply(94 mds0_inotable [write 0~34] v108'226 uv226 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f99b401c020 con 0x55f2c7e7c6f0 2024-08-03T20:57:48.781 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:48.779+0000 7f99cc6e9f40 1 -- 172.21.15.32:0/2171694035 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:95 12.13 12:c91d4a1d:::mds0_inotable:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108) -- 0x55f2c7e87ae0 con 0x55f2c7e7c6f0 2024-08-03T20:57:48.782 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:48.781+0000 7f99c4e00640 1 -- 172.21.15.32:0/2171694035 <== osd.1 v2:172.21.15.32:6824/1551241759 9 ==== osd_op_reply(95 mds0_inotable [omap-set-header] v108'227 uv227 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f99b401c020 con 0x55f2c7e7c6f0 2024-08-03T20:57:48.782 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:48.781+0000 7f99cc6e9f40 1 -- 172.21.15.32:0/2171694035 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:96 12.13 12:cd5a64a3:::100.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108) -- 0x55f2c7e87ae0 con 0x55f2c7e7c6f0 2024-08-03T20:57:48.783 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:48.782+0000 7f99c4e00640 1 -- 172.21.15.32:0/2171694035 <== osd.1 v2:172.21.15.32:6824/1551241759 10 ==== osd_op_reply(96 100.00000000 [create] v108'228 uv222 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f99b401c020 con 0x55f2c7e7c6f0 2024-08-03T20:57:48.783 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:48.782+0000 7f99cc6e9f40 1 -- 172.21.15.32:0/2171694035 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:97 12.13 12:cd5a64a3:::100.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e108) -- 0x55f2c7e87ae0 con 0x55f2c7e7c6f0 2024-08-03T20:57:48.785 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:48.783+0000 7f99c4e00640 1 -- 172.21.15.32:0/2171694035 <== osd.1 v2:172.21.15.32:6824/1551241759 11 ==== osd_op_reply(97 100.00000000 [delete] v108'229 uv229 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f99b401c020 con 0x55f2c7e7c6f0 2024-08-03T20:57:48.785 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:48.783+0000 7f99cc6e9f40 1 -- 172.21.15.32:0/2171694035 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:98 12.13 12:cd5a64a3:::100.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108) -- 0x55f2c7e87ae0 con 0x55f2c7e7c6f0 2024-08-03T20:57:48.786 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:48.785+0000 7f99c4e00640 1 -- 172.21.15.32:0/2171694035 <== osd.1 v2:172.21.15.32:6824/1551241759 12 ==== osd_op_reply(98 100.00000000 [create] v108'230 uv230 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f99b401c020 con 0x55f2c7e7c6f0 2024-08-03T20:57:48.786 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::100.00000000:head# 2024-08-03T20:57:48.786 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:48.785+0000 7f99cc6e9f40 1 -- 172.21.15.32:0/2171694035 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:99 12.13 12:cd5a64a3:::100.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108) -- 0x55f2c7e87ae0 con 0x55f2c7e7c6f0 2024-08-03T20:57:48.787 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:48.786+0000 7f99c4e00640 1 -- 172.21.15.32:0/2171694035 <== osd.1 v2:172.21.15.32:6824/1551241759 13 ==== osd_op_reply(99 100.00000000 [omap-set-header] v108'231 uv231 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f99b401c020 con 0x55f2c7e7c6f0 2024-08-03T20:57:48.787 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:48.786+0000 7f99cc6e9f40 1 -- 172.21.15.32:0/2171694035 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:100 12.13 12:cd5a64a3:::100.00000000:head [omap-set-vals in=5194b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108) -- 0x55f2c7e8a870 con 0x55f2c7e7c6f0 2024-08-03T20:57:48.789 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:48.787+0000 7f99c4e00640 1 -- 172.21.15.32:0/2171694035 <== osd.1 v2:172.21.15.32:6824/1551241759 14 ==== osd_op_reply(100 100.00000000 [omap-set-vals] v108'232 uv232 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f99b401c020 con 0x55f2c7e7c6f0 2024-08-03T20:57:48.789 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:48.787+0000 7f99cc6e9f40 1 --2- 172.21.15.32:0/2171694035 >> [v2:172.21.15.32:6800/9088303,v1:172.21.15.32:6801/9088303] conn(0x55f2c7e88550 0x55f2c7e8a970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:48.789 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:48.787+0000 7f99cc6e9f40 1 -- 172.21.15.32:0/2171694035 --> [v2:172.21.15.32:6800/9088303,v1:172.21.15.32:6801/9088303] -- osd_op(unknown.0.0:101 12.b 12:d0630e4c:::mds0_sessionmap:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108) -- 0x55f2c7e8af30 con 0x55f2c7e88550 2024-08-03T20:57:48.789 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:48.787+0000 7f99c4e00640 1 --2- 172.21.15.32:0/2171694035 >> [v2:172.21.15.32:6800/9088303,v1:172.21.15.32:6801/9088303] conn(0x55f2c7e88550 0x55f2c7e8a970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:48.790 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:48.788+0000 7f99c4e00640 1 --2- 172.21.15.32:0/2171694035 >> [v2:172.21.15.32:6800/9088303,v1:172.21.15.32:6801/9088303] conn(0x55f2c7e88550 0x55f2c7e8a970 crc :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:57:48.791 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:48.789+0000 7f99c4e00640 1 -- 172.21.15.32:0/2171694035 <== osd.3 v2:172.21.15.32:6800/9088303 1 ==== osd_op_reply(101 mds0_sessionmap [create] v108'99 uv98 ondisk = -17 ((17) File exists)) ==== 159+0+0 (crc 0 0 0) 0x7f99b401c020 con 0x55f2c7e88550 2024-08-03T20:57:48.791 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:48.789+0000 7f99cc6e9f40 1 -- 172.21.15.32:0/2171694035 --> [v2:172.21.15.32:6800/9088303,v1:172.21.15.32:6801/9088303] -- osd_op(unknown.0.0:102 12.b 12:d0630e4c:::mds0_sessionmap:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e108) -- 0x55f2c7e8af30 con 0x55f2c7e88550 2024-08-03T20:57:48.792 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:48.791+0000 7f99c4e00640 1 -- 172.21.15.32:0/2171694035 <== osd.3 v2:172.21.15.32:6800/9088303 2 ==== osd_op_reply(102 mds0_sessionmap [delete] v108'100 uv100 ondisk = 0) ==== 159+0+0 (crc 0 0 0) 0x7f99b401c020 con 0x55f2c7e88550 2024-08-03T20:57:48.792 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:48.791+0000 7f99cc6e9f40 1 -- 172.21.15.32:0/2171694035 --> [v2:172.21.15.32:6800/9088303,v1:172.21.15.32:6801/9088303] -- osd_op(unknown.0.0:103 12.b 12:d0630e4c:::mds0_sessionmap:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108) -- 0x55f2c7e8af30 con 0x55f2c7e88550 2024-08-03T20:57:48.793 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:48.792+0000 7f99c4e00640 1 -- 172.21.15.32:0/2171694035 <== osd.3 v2:172.21.15.32:6800/9088303 3 ==== osd_op_reply(103 mds0_sessionmap [create] v108'101 uv101 ondisk = 0) ==== 159+0+0 (crc 0 0 0) 0x7f99b401c020 con 0x55f2c7e88550 2024-08-03T20:57:48.794 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds0_sessionmap:head# 2024-08-03T20:57:48.794 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:48.792+0000 7f99cc6e9f40 1 -- 172.21.15.32:0/2171694035 --> [v2:172.21.15.32:6800/9088303,v1:172.21.15.32:6801/9088303] -- osd_op(unknown.0.0:104 12.b 12:d0630e4c:::mds0_sessionmap:head [omap-set-header in=14b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108) -- 0x55f2c7e8afb0 con 0x55f2c7e88550 2024-08-03T20:57:48.795 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:48.793+0000 7f99c4e00640 1 -- 172.21.15.32:0/2171694035 <== osd.3 v2:172.21.15.32:6800/9088303 4 ==== osd_op_reply(104 mds0_sessionmap [omap-set-header] v108'102 uv102 ondisk = 0) ==== 159+0+0 (crc 0 0 0) 0x7f99b401c020 con 0x55f2c7e88550 2024-08-03T20:57:48.795 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:48.793+0000 7f99cc6e9f40 1 -- 172.21.15.32:0/2171694035 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:105 12.7 12:e0b41d06:::609.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108) -- 0x55f2c7e80050 con 0x55f2c7e7c6f0 2024-08-03T20:57:48.796 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:48.795+0000 7f99c4e00640 1 -- 172.21.15.32:0/2171694035 <== osd.1 v2:172.21.15.32:6824/1551241759 15 ==== osd_op_reply(105 609.00000000 [create] v108'135 uv134 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f99b401c020 con 0x55f2c7e7c6f0 2024-08-03T20:57:48.796 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:48.795+0000 7f99cc6e9f40 1 -- 172.21.15.32:0/2171694035 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:106 12.7 12:e0b41d06:::609.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e108) -- 0x55f2c7e80050 con 0x55f2c7e7c6f0 2024-08-03T20:57:48.797 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:48.796+0000 7f99c4e00640 1 -- 172.21.15.32:0/2171694035 <== osd.1 v2:172.21.15.32:6824/1551241759 16 ==== osd_op_reply(106 609.00000000 [delete] v108'136 uv136 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f99b401c020 con 0x55f2c7e7c6f0 2024-08-03T20:57:48.798 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:48.796+0000 7f99cc6e9f40 1 -- 172.21.15.32:0/2171694035 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:107 12.7 12:e0b41d06:::609.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108) -- 0x55f2c7e80050 con 0x55f2c7e7c6f0 2024-08-03T20:57:48.799 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::609.00000000:head# 2024-08-03T20:57:48.799 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:48.798+0000 7f99c4e00640 1 -- 172.21.15.32:0/2171694035 <== osd.1 v2:172.21.15.32:6824/1551241759 17 ==== osd_op_reply(107 609.00000000 [create] v108'137 uv137 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f99b401c020 con 0x55f2c7e7c6f0 2024-08-03T20:57:48.799 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:48.798+0000 7f99cc6e9f40 1 -- 172.21.15.32:0/2171694035 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:108 12.7 12:e0b41d06:::609.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108) -- 0x55f2c7e8c3a0 con 0x55f2c7e7c6f0 2024-08-03T20:57:48.800 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:48.799+0000 7f99c4e00640 1 -- 172.21.15.32:0/2171694035 <== osd.1 v2:172.21.15.32:6824/1551241759 18 ==== osd_op_reply(108 609.00000000 [setxattr (30)] v108'138 uv138 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f99b401c020 con 0x55f2c7e7c6f0 2024-08-03T20:57:48.800 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:48.799+0000 7f99cc6e9f40 1 -- 172.21.15.32:0/2171694035 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:109 12.7 12:e0b41d06:::609.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108) -- 0x55f2c7e8c3a0 con 0x55f2c7e7c6f0 2024-08-03T20:57:48.802 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:48.800+0000 7f99c4e00640 1 -- 172.21.15.32:0/2171694035 <== osd.1 v2:172.21.15.32:6824/1551241759 19 ==== osd_op_reply(109 609.00000000 [setxattr (62)] v108'139 uv139 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f99b401c020 con 0x55f2c7e7c6f0 2024-08-03T20:57:48.802 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:48.800+0000 7f99cc6e9f40 1 -- 172.21.15.32:0/2171694035 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:110 12.7 12:e0b41d06:::609.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108) -- 0x55f2c7e8b390 con 0x55f2c7e7c6f0 2024-08-03T20:57:48.803 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:48.802+0000 7f99c4e00640 1 -- 172.21.15.32:0/2171694035 <== osd.1 v2:172.21.15.32:6824/1551241759 20 ==== osd_op_reply(110 609.00000000 [omap-set-header] v108'140 uv140 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f99b401c020 con 0x55f2c7e7c6f0 2024-08-03T20:57:48.803 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:48.802+0000 7f99cc6e9f40 1 -- 172.21.15.32:0/2171694035 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:111 12.f 12:f1f69726:::400.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108) -- 0x55f2c7e8b390 con 0x55f2c7e80e10 2024-08-03T20:57:48.805 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:48.803+0000 7f99c6200640 1 -- 172.21.15.32:0/2171694035 <== osd.6 v2:172.21.15.118:6825/801947386 12 ==== osd_op_reply(111 400.00000000 [create] v108'115 uv114 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f99c000b040 con 0x55f2c7e80e10 2024-08-03T20:57:48.805 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:48.804+0000 7f99cc6e9f40 1 -- 172.21.15.32:0/2171694035 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:112 12.f 12:f1f69726:::400.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e108) -- 0x55f2c7e8b390 con 0x55f2c7e80e10 2024-08-03T20:57:48.806 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:48.805+0000 7f99c6200640 1 -- 172.21.15.32:0/2171694035 <== osd.6 v2:172.21.15.118:6825/801947386 13 ==== osd_op_reply(112 400.00000000 [delete] v108'116 uv116 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f99c000b040 con 0x55f2c7e80e10 2024-08-03T20:57:48.807 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:48.805+0000 7f99cc6e9f40 1 -- 172.21.15.32:0/2171694035 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:113 12.f 12:f1f69726:::400.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108) -- 0x55f2c7e8b390 con 0x55f2c7e80e10 2024-08-03T20:57:48.808 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:48.806+0000 7f99c6200640 1 -- 172.21.15.32:0/2171694035 <== osd.6 v2:172.21.15.118:6825/801947386 14 ==== osd_op_reply(113 400.00000000 [create] v108'117 uv117 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f99c000b040 con 0x55f2c7e80e10 2024-08-03T20:57:48.808 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::400.00000000:head# 2024-08-03T20:57:48.808 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:48.807+0000 7f99cc6e9f40 1 -- 172.21.15.32:0/2171694035 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:114 12.f 12:f1f69726:::400.00000000:head [write 0~22 in=22b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108) -- 0x55f2c7e8b390 con 0x55f2c7e80e10 2024-08-03T20:57:48.810 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:48.808+0000 7f99c6200640 1 -- 172.21.15.32:0/2171694035 <== osd.6 v2:172.21.15.118:6825/801947386 15 ==== osd_op_reply(114 400.00000000 [write 0~22] v108'118 uv118 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f99c000b040 con 0x55f2c7e80e10 2024-08-03T20:57:48.810 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:48.808+0000 7f99cc6e9f40 1 -- 172.21.15.32:0/2171694035 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:115 12.f 12:f1f69726:::400.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108) -- 0x55f2c7e8b390 con 0x55f2c7e80e10 2024-08-03T20:57:48.811 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:48.810+0000 7f99c6200640 1 -- 172.21.15.32:0/2171694035 <== osd.6 v2:172.21.15.118:6825/801947386 16 ==== osd_op_reply(115 400.00000000 [omap-set-header] v108'119 uv119 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f99c000b040 con 0x55f2c7e80e10 2024-08-03T20:57:48.811 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:48.810+0000 7f99cc6e9f40 1 -- 172.21.15.32:0/2171694035 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:116 12.1f 12:fe07bba1:::100.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108) -- 0x55f2c7e8b390 con 0x55f2c7e80e10 2024-08-03T20:57:48.813 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:48.812+0000 7f99c6200640 1 -- 172.21.15.32:0/2171694035 <== osd.6 v2:172.21.15.118:6825/801947386 17 ==== osd_op_reply(116 100.00000000.inode [create] v108'278 uv277 ondisk = -17 ((17) File exists)) ==== 162+0+0 (crc 0 0 0) 0x7f99c000b040 con 0x55f2c7e80e10 2024-08-03T20:57:48.813 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:48.812+0000 7f99cc6e9f40 1 -- 172.21.15.32:0/2171694035 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:117 12.1f 12:fe07bba1:::100.00000000.inode:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e108) -- 0x55f2c7e8b390 con 0x55f2c7e80e10 2024-08-03T20:57:48.814 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:48.813+0000 7f99c6200640 1 -- 172.21.15.32:0/2171694035 <== osd.6 v2:172.21.15.118:6825/801947386 18 ==== osd_op_reply(117 100.00000000.inode [delete] v108'279 uv279 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7f99c000b040 con 0x55f2c7e80e10 2024-08-03T20:57:48.814 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:48.813+0000 7f99cc6e9f40 1 -- 172.21.15.32:0/2171694035 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:118 12.1f 12:fe07bba1:::100.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108) -- 0x55f2c7e8b390 con 0x55f2c7e80e10 2024-08-03T20:57:48.816 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:48.814+0000 7f99c6200640 1 -- 172.21.15.32:0/2171694035 <== osd.6 v2:172.21.15.118:6825/801947386 19 ==== osd_op_reply(118 100.00000000.inode [create] v108'280 uv280 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7f99c000b040 con 0x55f2c7e80e10 2024-08-03T20:57:48.816 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::100.00000000.inode:head# 2024-08-03T20:57:48.816 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:48.814+0000 7f99cc6e9f40 1 -- 172.21.15.32:0/2171694035 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:119 12.1f 12:fe07bba1:::100.00000000.inode:head [write 0~582 in=582b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108) -- 0x55f2c7e8b390 con 0x55f2c7e80e10 2024-08-03T20:57:48.818 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:48.816+0000 7f99c6200640 1 -- 172.21.15.32:0/2171694035 <== osd.6 v2:172.21.15.118:6825/801947386 20 ==== osd_op_reply(119 100.00000000.inode [write 0~582] v108'281 uv281 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7f99c000b040 con 0x55f2c7e80e10 2024-08-03T20:57:48.818 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:48.816+0000 7f99cc6e9f40 1 -- 172.21.15.32:0/2171694035 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:120 12.1f 12:fe07bba1:::100.00000000.inode:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108) -- 0x55f2c7e8b390 con 0x55f2c7e80e10 2024-08-03T20:57:48.819 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:48.818+0000 7f99c6200640 1 -- 172.21.15.32:0/2171694035 <== osd.6 v2:172.21.15.118:6825/801947386 21 ==== osd_op_reply(120 100.00000000.inode [omap-set-header] v108'282 uv282 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7f99c000b040 con 0x55f2c7e80e10 2024-08-03T20:57:48.819 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:48.818+0000 7f99cc6e9f40 1 -- 172.21.15.32:0/2171694035 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:121 12.1f 12:ff5b34d6:::1.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108) -- 0x55f2c7e8b390 con 0x55f2c7e80e10 2024-08-03T20:57:48.821 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:48.819+0000 7f99c6200640 1 -- 172.21.15.32:0/2171694035 <== osd.6 v2:172.21.15.118:6825/801947386 22 ==== osd_op_reply(121 1.00000000 [create] v108'283 uv276 ondisk = -17 ((17) File exists)) ==== 154+0+0 (crc 0 0 0) 0x7f99c000b040 con 0x55f2c7e80e10 2024-08-03T20:57:48.821 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:48.819+0000 7f99cc6e9f40 1 -- 172.21.15.32:0/2171694035 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:122 12.1f 12:ff5b34d6:::1.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e108) -- 0x55f2c7e8b390 con 0x55f2c7e80e10 2024-08-03T20:57:48.822 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:48.821+0000 7f99c6200640 1 -- 172.21.15.32:0/2171694035 <== osd.6 v2:172.21.15.118:6825/801947386 23 ==== osd_op_reply(122 1.00000000 [delete] v108'284 uv284 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7f99c000b040 con 0x55f2c7e80e10 2024-08-03T20:57:48.822 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:48.821+0000 7f99cc6e9f40 1 -- 172.21.15.32:0/2171694035 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:123 12.1f 12:ff5b34d6:::1.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108) -- 0x55f2c7e8b390 con 0x55f2c7e80e10 2024-08-03T20:57:48.823 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:48.822+0000 7f99c6200640 1 -- 172.21.15.32:0/2171694035 <== osd.6 v2:172.21.15.118:6825/801947386 24 ==== osd_op_reply(123 1.00000000 [create] v108'285 uv285 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7f99c000b040 con 0x55f2c7e80e10 2024-08-03T20:57:48.824 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::1.00000000:head# 2024-08-03T20:57:48.824 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:48.822+0000 7f99cc6e9f40 1 -- 172.21.15.32:0/2171694035 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:124 12.1f 12:ff5b34d6:::1.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108) -- 0x55f2c7e8c3a0 con 0x55f2c7e80e10 2024-08-03T20:57:48.825 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:48.823+0000 7f99c6200640 1 -- 172.21.15.32:0/2171694035 <== osd.6 v2:172.21.15.118:6825/801947386 25 ==== osd_op_reply(124 1.00000000 [setxattr (30)] v108'286 uv286 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7f99c000b040 con 0x55f2c7e80e10 2024-08-03T20:57:48.825 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:48.824+0000 7f99cc6e9f40 1 -- 172.21.15.32:0/2171694035 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:125 12.1f 12:ff5b34d6:::1.00000000:head [setxattr parent (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108) -- 0x55f2c7e8c3a0 con 0x55f2c7e80e10 2024-08-03T20:57:48.827 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:48.825+0000 7f99c6200640 1 -- 172.21.15.32:0/2171694035 <== osd.6 v2:172.21.15.118:6825/801947386 26 ==== osd_op_reply(125 1.00000000 [setxattr (30)] v108'287 uv287 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7f99c000b040 con 0x55f2c7e80e10 2024-08-03T20:57:48.827 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:48.825+0000 7f99cc6e9f40 1 -- 172.21.15.32:0/2171694035 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:126 12.1f 12:ff5b34d6:::1.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108) -- 0x55f2c7e8b390 con 0x55f2c7e80e10 2024-08-03T20:57:48.828 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:48.826+0000 7f99c6200640 1 -- 172.21.15.32:0/2171694035 <== osd.6 v2:172.21.15.118:6825/801947386 27 ==== osd_op_reply(126 1.00000000 [omap-set-header] v108'288 uv288 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7f99c000b040 con 0x55f2c7e80e10 2024-08-03T20:57:48.828 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:48.826+0000 7f99cc6e9f40 1 -- 172.21.15.32:0/2171694035 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:127 12.1f 12:ff5b34d6:::1.00000000:head [omap-set-vals in=584b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108) -- 0x55f2c7e8b8f0 con 0x55f2c7e80e10 2024-08-03T20:57:48.831 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:48.828+0000 7f99c6200640 1 -- 172.21.15.32:0/2171694035 <== osd.6 v2:172.21.15.118:6825/801947386 28 ==== osd_op_reply(127 1.00000000 [omap-set-vals] v108'289 uv289 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7f99c000b040 con 0x55f2c7e80e10 2024-08-03T20:57:48.831 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:48.828+0000 7f99cc6e9f40 1 -- 172.21.15.32:0/2171694035 >> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] conn(0x55f2c7cfc050 msgr2=0x55f2c7cfc460 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:48.831 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:48.828+0000 7f99cc6e9f40 1 --2- 172.21.15.32:0/2171694035 >> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] conn(0x55f2c7cfc050 0x55f2c7cfc460 crc :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:48.831 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:48.828+0000 7f99cc6e9f40 1 -- 172.21.15.32:0/2171694035 >> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] conn(0x55f2c7e7c6f0 msgr2=0x55f2c7e7eb10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:48.831 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:48.828+0000 7f99cc6e9f40 1 --2- 172.21.15.32:0/2171694035 >> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] conn(0x55f2c7e7c6f0 0x55f2c7e7eb10 crc :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:48.831 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:48.828+0000 7f99cc6e9f40 1 -- 172.21.15.32:0/2171694035 >> [v2:172.21.15.32:6800/9088303,v1:172.21.15.32:6801/9088303] conn(0x55f2c7e88550 msgr2=0x55f2c7e8a970 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:48.831 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:48.828+0000 7f99cc6e9f40 1 --2- 172.21.15.32:0/2171694035 >> [v2:172.21.15.32:6800/9088303,v1:172.21.15.32:6801/9088303] conn(0x55f2c7e88550 0x55f2c7e8a970 crc :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:48.831 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:48.828+0000 7f99cc6e9f40 1 -- 172.21.15.32:0/2171694035 >> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] conn(0x55f2c7e77f20 msgr2=0x55f2c7e7a340 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:48.832 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:48.828+0000 7f99cc6e9f40 1 --2- 172.21.15.32:0/2171694035 >> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] conn(0x55f2c7e77f20 0x55f2c7e7a340 crc :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:48.832 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:48.828+0000 7f99cc6e9f40 1 -- 172.21.15.32:0/2171694035 >> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] conn(0x55f2c7e80e10 msgr2=0x55f2c7e81220 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:48.832 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:48.828+0000 7f99cc6e9f40 1 --2- 172.21.15.32:0/2171694035 >> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] conn(0x55f2c7e80e10 0x55f2c7e81220 crc :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:48.832 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:48.828+0000 7f99cc6e9f40 1 -- 172.21.15.32:0/2171694035 >> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] conn(0x55f2c7e84210 msgr2=0x55f2c7e86e60 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:48.832 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:48.828+0000 7f99cc6e9f40 1 --2- 172.21.15.32:0/2171694035 >> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] conn(0x55f2c7e84210 0x55f2c7e86e60 crc :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:48.832 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:48.828+0000 7f99c6200640 1 -- 172.21.15.32:0/2171694035 reap_dead start 2024-08-03T20:57:48.832 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:48.828+0000 7f99c6200640 1 -- 172.21.15.32:0/2171694035 reap_dead start 2024-08-03T20:57:48.832 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:48.828+0000 7f99c6200640 1 -- 172.21.15.32:0/2171694035 reap_dead start 2024-08-03T20:57:48.832 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:48.828+0000 7f99cc6e9f40 1 -- 172.21.15.32:0/2171694035 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f99a8075a20 msgr2=0x7f99a8077f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:48.832 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:48.828+0000 7f99cc6e9f40 1 --2- 172.21.15.32:0/2171694035 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f99a8075a20 0x7f99a8077f30 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x55f2c7dd3780 tx=0x7f99b8004af0 comp rx=0 tx=0).stop 2024-08-03T20:57:48.832 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:48.828+0000 7f99cc6e9f40 1 -- 172.21.15.32:0/2171694035 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55f2c7db4ea0 msgr2=0x55f2c7da9130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:48.832 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:48.828+0000 7f99cc6e9f40 1 --2- 172.21.15.32:0/2171694035 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55f2c7db4ea0 0x55f2c7da9130 secure :-1 s=READY pgs=1631 cs=0 l=1 rev1=1 crypto rx=0x7f99b400a6b0 tx=0x7f99b400ab70 comp rx=0 tx=0).stop 2024-08-03T20:57:48.832 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:48.828+0000 7f99cc6e9f40 1 -- 172.21.15.32:0/2171694035 shutdown_connections 2024-08-03T20:57:48.832 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:48.828+0000 7f99cc6e9f40 1 --2- 172.21.15.32:0/2171694035 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f99a8075a20 0x7f99a8077f30 unknown :-1 s=CLOSED pgs=666 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:48.833 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:48.828+0000 7f99cc6e9f40 1 --2- 172.21.15.32:0/2171694035 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55f2c7db4ea0 0x55f2c7da9130 unknown :-1 s=CLOSED pgs=1631 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:48.833 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:48.829+0000 7f99cc6e9f40 1 -- 172.21.15.32:0/2171694035 >> 172.21.15.32:0/2171694035 conn(0x55f2c7c1cac0 msgr2=0x55f2c7e5d7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:57:48.833 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:48.829+0000 7f99cc6e9f40 1 -- 172.21.15.32:0/2171694035 shutdown_connections 2024-08-03T20:57:48.833 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:48.829+0000 7f99cc6e9f40 1 -- 172.21.15.32:0/2171694035 wait complete. 2024-08-03T20:57:48.841 DEBUG:teuthology.orchestra.run.smithi032:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata truncate 1.00000000.inode 0 2024-08-03T20:57:49.024 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:49.023+0000 7fdc6e292f40 1 Processor -- start 2024-08-03T20:57:49.024 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:49.023+0000 7fdc6e292f40 1 -- start start 2024-08-03T20:57:49.026 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:49.024+0000 7fdc6e292f40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5578648be360 0x5578648b7e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:49.027 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:49.024+0000 7fdc6e292f40 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5578648dbcb0 0x55786484e020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:49.027 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:49.024+0000 7fdc6e292f40 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x5578648dc3f0 0x55786484e560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:49.027 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:49.024+0000 7fdc6e292f40 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x557864755e30 con 0x5578648dbcb0 2024-08-03T20:57:49.027 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:49.024+0000 7fdc6e292f40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x5578647b3bf0 con 0x5578648be360 2024-08-03T20:57:49.027 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:49.024+0000 7fdc6e292f40 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x5578648bf440 con 0x5578648dc3f0 2024-08-03T20:57:49.027 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:49.024+0000 7fdc66a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5578648dbcb0 0x55786484e020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:49.027 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:49.024+0000 7fdc66a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5578648dbcb0 0x55786484e020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:43342/0 (socket says 172.21.15.32:43342) 2024-08-03T20:57:49.027 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:49.025+0000 7fdc66a00640 1 -- 172.21.15.32:0/4067068824 learned_addr learned my addr 172.21.15.32:0/4067068824 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:57:49.027 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:49.025+0000 7fdc67400640 1 --2- 172.21.15.32:0/4067068824 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5578648be360 0x5578648b7e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:49.027 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:49.025+0000 7fdc67e00640 1 --2- 172.21.15.32:0/4067068824 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x5578648dc3f0 0x55786484e560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:49.027 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:49.025+0000 7fdc66a00640 1 -- 172.21.15.32:0/4067068824 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x5578648dc3f0 msgr2=0x55786484e560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:57:49.027 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:49.025+0000 7fdc66a00640 1 --2- 172.21.15.32:0/4067068824 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x5578648dc3f0 0x55786484e560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:49.027 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:49.025+0000 7fdc66a00640 1 -- 172.21.15.32:0/4067068824 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5578648be360 msgr2=0x5578648b7e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:57:49.028 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:49.025+0000 7fdc66a00640 1 --2- 172.21.15.32:0/4067068824 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5578648be360 0x5578648b7e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:49.028 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:49.025+0000 7fdc66a00640 1 -- 172.21.15.32:0/4067068824 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5578648bd370 con 0x5578648dbcb0 2024-08-03T20:57:49.028 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:49.025+0000 7fdc66a00640 1 --2- 172.21.15.32:0/4067068824 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5578648dbcb0 0x55786484e020 secure :-1 s=READY pgs=1632 cs=0 l=1 rev1=1 crypto rx=0x7fdc54001e60 tx=0x7fdc5400b730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fc92e5aa2e007800 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:57:49.029 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:49.026+0000 7fdc65600640 1 -- 172.21.15.32:0/4067068824 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc5400e830 con 0x5578648dbcb0 2024-08-03T20:57:49.029 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:49.026+0000 7fdc65600640 1 -- 172.21.15.32:0/4067068824 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fdc54006050 con 0x5578648dbcb0 2024-08-03T20:57:49.029 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:49.026+0000 7fdc65600640 1 -- 172.21.15.32:0/4067068824 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc5400ed80 con 0x5578648dbcb0 2024-08-03T20:57:49.029 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:49.026+0000 7fdc6e292f40 1 -- 172.21.15.32:0/4067068824 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5578648dbcb0 msgr2=0x55786484e020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:49.029 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:49.026+0000 7fdc6e292f40 1 --2- 172.21.15.32:0/4067068824 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5578648dbcb0 0x55786484e020 secure :-1 s=READY pgs=1632 cs=0 l=1 rev1=1 crypto rx=0x7fdc54001e60 tx=0x7fdc5400b730 comp rx=0 tx=0).stop 2024-08-03T20:57:49.029 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:49.026+0000 7fdc6e292f40 1 -- 172.21.15.32:0/4067068824 shutdown_connections 2024-08-03T20:57:49.029 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:49.026+0000 7fdc6e292f40 1 --2- 172.21.15.32:0/4067068824 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x5578648dc3f0 0x55786484e560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:49.029 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:49.026+0000 7fdc6e292f40 1 --2- 172.21.15.32:0/4067068824 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5578648dbcb0 0x55786484e020 unknown :-1 s=CLOSED pgs=1632 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:49.030 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:49.026+0000 7fdc6e292f40 1 --2- 172.21.15.32:0/4067068824 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5578648be360 0x5578648b7e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:49.030 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:49.026+0000 7fdc6e292f40 1 -- 172.21.15.32:0/4067068824 >> 172.21.15.32:0/4067068824 conn(0x557864727ac0 msgr2=0x5578648d9d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:57:49.030 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:49.026+0000 7fdc6e292f40 1 -- 172.21.15.32:0/4067068824 shutdown_connections 2024-08-03T20:57:49.030 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:49.026+0000 7fdc6e292f40 1 -- 172.21.15.32:0/4067068824 wait complete. 2024-08-03T20:57:49.030 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:49.027+0000 7fdc6e292f40 1 Processor -- start 2024-08-03T20:57:49.030 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:49.028+0000 7fdc6e292f40 1 -- start start 2024-08-03T20:57:49.031 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:49.029+0000 7fdc6e292f40 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x5578648be360 0x55786484ff80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:49.032 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:49.029+0000 7fdc6e292f40 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5578648dbcb0 0x557864977e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:49.032 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:49.029+0000 7fdc6e292f40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5578648dc3f0 0x557864978380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:49.032 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:49.029+0000 7fdc6e292f40 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x5578648bcd40 con 0x5578648dbcb0 2024-08-03T20:57:49.032 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:49.029+0000 7fdc6e292f40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x5578648bc9d0 con 0x5578648dc3f0 2024-08-03T20:57:49.032 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:49.029+0000 7fdc6e292f40 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x55786484ab00 con 0x5578648be360 2024-08-03T20:57:49.032 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:49.029+0000 7fdc66a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5578648dbcb0 0x557864977e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:49.032 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:49.029+0000 7fdc66a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5578648dbcb0 0x557864977e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:43348/0 (socket says 172.21.15.32:43348) 2024-08-03T20:57:49.032 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:49.029+0000 7fdc66a00640 1 -- 172.21.15.32:0/1892102577 learned_addr learned my addr 172.21.15.32:0/1892102577 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:57:49.032 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:49.030+0000 7fdc67e00640 1 --2- 172.21.15.32:0/1892102577 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5578648dc3f0 0x557864978380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:49.032 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:49.030+0000 7fdc67400640 1 --2- 172.21.15.32:0/1892102577 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x5578648be360 0x55786484ff80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:49.032 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:49.030+0000 7fdc66a00640 1 -- 172.21.15.32:0/1892102577 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x5578648be360 msgr2=0x55786484ff80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:49.032 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:49.030+0000 7fdc66a00640 1 --2- 172.21.15.32:0/1892102577 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x5578648be360 0x55786484ff80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:49.032 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:49.030+0000 7fdc66a00640 1 -- 172.21.15.32:0/1892102577 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5578648dc3f0 msgr2=0x557864978380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:49.033 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:49.030+0000 7fdc66a00640 1 --2- 172.21.15.32:0/1892102577 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5578648dc3f0 0x557864978380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:49.033 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:49.030+0000 7fdc66a00640 1 -- 172.21.15.32:0/1892102577 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5578648bb680 con 0x5578648dbcb0 2024-08-03T20:57:49.033 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:49.030+0000 7fdc66a00640 1 --2- 172.21.15.32:0/1892102577 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5578648dbcb0 0x557864977e40 secure :-1 s=READY pgs=1633 cs=0 l=1 rev1=1 crypto rx=0x5578648da700 tx=0x7fdc5400e8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:57:49.033 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:49.030+0000 7fdc5fe00640 1 -- 172.21.15.32:0/1892102577 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc54024070 con 0x5578648dbcb0 2024-08-03T20:57:49.034 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:49.031+0000 7fdc5fe00640 1 -- 172.21.15.32:0/1892102577 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fdc54006050 con 0x5578648dbcb0 2024-08-03T20:57:49.034 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:49.031+0000 7fdc5fe00640 1 -- 172.21.15.32:0/1892102577 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc5401f410 con 0x5578648dbcb0 2024-08-03T20:57:49.034 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:49.031+0000 7fdc6e292f40 1 -- 172.21.15.32:0/1892102577 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x5578648da040 con 0x5578648dbcb0 2024-08-03T20:57:49.034 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:49.031+0000 7fdc6e292f40 1 -- 172.21.15.32:0/1892102577 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55786484cef0 con 0x5578648dbcb0 2024-08-03T20:57:49.034 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:49.033+0000 7fdc5fe00640 1 -- 172.21.15.32:0/1892102577 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7fdc540163f0 con 0x5578648dbcb0 2024-08-03T20:57:49.035 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:49.034+0000 7fdc5fe00640 1 --2- 172.21.15.32:0/1892102577 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fdc48075a20 0x7fdc48077f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:49.035 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:49.034+0000 7fdc5fe00640 1 -- 172.21.15.32:0/1892102577 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(108..108 src has 1..108) ==== 7832+0+0 (secure 0 0 0) 0x7fdc54098a30 con 0x5578648dbcb0 2024-08-03T20:57:49.036 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:49.034+0000 7fdc67400640 1 --2- 172.21.15.32:0/1892102577 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fdc48075a20 0x7fdc48077f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:49.036 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:49.034+0000 7fdc6e292f40 1 --2- 172.21.15.32:0/1892102577 >> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] conn(0x5578648504c0 0x5578648b9d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:49.036 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:49.034+0000 7fdc6e292f40 1 -- 172.21.15.32:0/1892102577 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:1 12.14 12:287034c4:::1.00000000.inode:head [truncate 0] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108) -- 0x5578648ba270 con 0x5578648504c0 2024-08-03T20:57:49.036 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:49.035+0000 7fdc67e00640 1 --2- 172.21.15.32:0/1892102577 >> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] conn(0x5578648504c0 0x5578648b9d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:49.036 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:49.035+0000 7fdc67400640 1 --2- 172.21.15.32:0/1892102577 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fdc48075a20 0x7fdc48077f30 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7fdc58055000 tx=0x7fdc58003020 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:57:49.037 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:49.035+0000 7fdc67e00640 1 --2- 172.21.15.32:0/1892102577 >> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] conn(0x5578648504c0 0x5578648b9d30 crc :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:57:49.039 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:49.037+0000 7fdc67e00640 1 -- 172.21.15.32:0/1892102577 <== osd.5 v2:172.21.15.118:6810/2724891770 1 ==== osd_op_reply(1 1.00000000.inode [truncate 0] v108'526 uv526 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7fdc6000b040 con 0x5578648504c0 2024-08-03T20:57:49.039 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:49.037+0000 7fdc6e292f40 1 -- 172.21.15.32:0/1892102577 >> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] conn(0x5578648504c0 msgr2=0x5578648b9d30 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:49.039 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:49.037+0000 7fdc6e292f40 1 --2- 172.21.15.32:0/1892102577 >> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] conn(0x5578648504c0 0x5578648b9d30 crc :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:49.039 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:49.037+0000 7fdc6e292f40 1 -- 172.21.15.32:0/1892102577 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fdc48075a20 msgr2=0x7fdc48077f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:49.040 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:49.037+0000 7fdc6e292f40 1 --2- 172.21.15.32:0/1892102577 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fdc48075a20 0x7fdc48077f30 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7fdc58055000 tx=0x7fdc58003020 comp rx=0 tx=0).stop 2024-08-03T20:57:49.040 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:49.037+0000 7fdc6e292f40 1 -- 172.21.15.32:0/1892102577 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5578648dbcb0 msgr2=0x557864977e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:49.040 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:49.037+0000 7fdc6e292f40 1 --2- 172.21.15.32:0/1892102577 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5578648dbcb0 0x557864977e40 secure :-1 s=READY pgs=1633 cs=0 l=1 rev1=1 crypto rx=0x5578648da700 tx=0x7fdc5400e8d0 comp rx=0 tx=0).stop 2024-08-03T20:57:49.040 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:49.037+0000 7fdc67e00640 1 -- 172.21.15.32:0/1892102577 reap_dead start 2024-08-03T20:57:49.040 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:49.038+0000 7fdc6e292f40 1 -- 172.21.15.32:0/1892102577 shutdown_connections 2024-08-03T20:57:49.040 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:49.038+0000 7fdc6e292f40 1 -- 172.21.15.32:0/1892102577 >> 172.21.15.32:0/1892102577 conn(0x557864727ac0 msgr2=0x55786484ee00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:57:49.040 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:49.038+0000 7fdc6e292f40 1 -- 172.21.15.32:0/1892102577 shutdown_connections 2024-08-03T20:57:49.040 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:49.038+0000 7fdc6e292f40 1 -- 172.21.15.32:0/1892102577 wait complete. 2024-08-03T20:57:49.050 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs joinable true 2024-08-03T20:57:49.282 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:49.280+0000 7f3d90200640 1 -- 172.21.15.32:0/3690865980 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3d880ced30 msgr2=0x7f3d880cf140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:49.282 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:49.280+0000 7f3d90200640 1 --2- 172.21.15.32:0/3690865980 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3d880ced30 0x7f3d880cf140 secure :-1 s=READY pgs=1635 cs=0 l=1 rev1=1 crypto rx=0x7f3d88074a30 tx=0x7f3d8002a820 comp rx=0 tx=0).stop 2024-08-03T20:57:49.282 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:49.280+0000 7f3d90200640 1 -- 172.21.15.32:0/3690865980 shutdown_connections 2024-08-03T20:57:49.282 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:49.280+0000 7f3d90200640 1 --2- 172.21.15.32:0/3690865980 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3d880ced30 0x7f3d880cf140 unknown :-1 s=CLOSED pgs=1635 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:49.282 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:49.280+0000 7f3d90200640 1 --2- 172.21.15.32:0/3690865980 >> v2:172.21.15.118:3300/0 conn(0x7f3d88145c50 0x7f3d8806e950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:49.282 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:49.280+0000 7f3d90200640 1 -- 172.21.15.32:0/3690865980 >> 172.21.15.32:0/3690865980 conn(0x7f3d880b1bd0 msgr2=0x7f3d880b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:57:49.282 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:49.280+0000 7f3d90200640 1 -- 172.21.15.32:0/3690865980 shutdown_connections 2024-08-03T20:57:49.282 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:49.280+0000 7f3d90200640 1 -- 172.21.15.32:0/3690865980 wait complete. 2024-08-03T20:57:49.283 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:49.282+0000 7f3d90200640 1 Processor -- start 2024-08-03T20:57:49.283 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:49.282+0000 7f3d90200640 1 -- start start 2024-08-03T20:57:49.287 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:49.284+0000 7f3d90200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3d880cccf0 0x7f3d8813ba90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:49.287 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:49.284+0000 7f3d90200640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f3d880ced30 0x7f3d88159720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:49.287 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:49.284+0000 7f3d90200640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f3d88145c50 0x7f3d88144580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:49.287 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:49.284+0000 7f3d90200640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f3d880b4070 con 0x7f3d880cccf0 2024-08-03T20:57:49.287 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:49.284+0000 7f3d90200640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f3d880b3ef0 con 0x7f3d880ced30 2024-08-03T20:57:49.287 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:49.284+0000 7f3d90200640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7f3d880b41f0 con 0x7f3d88145c50 2024-08-03T20:57:49.287 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:49.284+0000 7f3d8d800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3d880cccf0 0x7f3d8813ba90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:49.287 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:49.284+0000 7f3d8d800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3d880cccf0 0x7f3d8813ba90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:43362/0 (socket says 172.21.15.32:43362) 2024-08-03T20:57:49.287 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:49.284+0000 7f3d8d800640 1 -- 172.21.15.32:0/3666505808 learned_addr learned my addr 172.21.15.32:0/3666505808 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:57:49.287 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:49.284+0000 7f3d8e200640 1 --2- 172.21.15.32:0/3666505808 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f3d88145c50 0x7f3d88144580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:49.287 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:49.284+0000 7f3d8ce00640 1 --2- 172.21.15.32:0/3666505808 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f3d880ced30 0x7f3d88159720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:49.288 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:49.284+0000 7f3d8d800640 1 -- 172.21.15.32:0/3666505808 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f3d88145c50 msgr2=0x7f3d88144580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:49.288 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:49.284+0000 7f3d8d800640 1 --2- 172.21.15.32:0/3666505808 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f3d88145c50 0x7f3d88144580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:49.288 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:49.284+0000 7f3d8d800640 1 -- 172.21.15.32:0/3666505808 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f3d880ced30 msgr2=0x7f3d88159720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:49.288 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:49.284+0000 7f3d8d800640 1 --2- 172.21.15.32:0/3666505808 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f3d880ced30 0x7f3d88159720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:49.288 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:49.284+0000 7f3d8d800640 1 -- 172.21.15.32:0/3666505808 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3d881f8d80 con 0x7f3d880cccf0 2024-08-03T20:57:49.288 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:49.285+0000 7f3d8d800640 1 --2- 172.21.15.32:0/3666505808 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3d880cccf0 0x7f3d8813ba90 secure :-1 s=READY pgs=1636 cs=0 l=1 rev1=1 crypto rx=0x7f3d7c00d770 tx=0x7f3d7c00dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:57:49.288 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:49.285+0000 7f3d85600640 1 -- 172.21.15.32:0/3666505808 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d7c017070 con 0x7f3d880cccf0 2024-08-03T20:57:49.288 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:49.285+0000 7f3d85600640 1 -- 172.21.15.32:0/3666505808 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f3d7c009d20 con 0x7f3d880cccf0 2024-08-03T20:57:49.288 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:49.285+0000 7f3d85600640 1 -- 172.21.15.32:0/3666505808 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d7c0106c0 con 0x7f3d880cccf0 2024-08-03T20:57:49.288 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:49.285+0000 7f3d90200640 1 -- 172.21.15.32:0/3666505808 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3d881f9060 con 0x7f3d880cccf0 2024-08-03T20:57:49.288 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:49.285+0000 7f3d90200640 1 -- 172.21.15.32:0/3666505808 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3d881f9620 con 0x7f3d880cccf0 2024-08-03T20:57:49.289 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:49.286+0000 7f3d90200640 1 -- 172.21.15.32:0/3666505808 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3d880b4710 con 0x7f3d880cccf0 2024-08-03T20:57:49.289 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:49.287+0000 7f3d85600640 1 -- 172.21.15.32:0/3666505808 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f3d7c01a070 con 0x7f3d880cccf0 2024-08-03T20:57:49.290 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:49.288+0000 7f3d85600640 1 --2- 172.21.15.32:0/3666505808 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f3d68077040 0x7f3d68079550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:49.290 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:49.289+0000 7f3d85600640 1 -- 172.21.15.32:0/3666505808 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(108..108 src has 1..108) ==== 7832+0+0 (secure 0 0 0) 0x7f3d7c099220 con 0x7f3d880cccf0 2024-08-03T20:57:49.290 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:49.289+0000 7f3d8ce00640 1 --2- 172.21.15.32:0/3666505808 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f3d68077040 0x7f3d68079550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:49.291 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:49.290+0000 7f3d8ce00640 1 --2- 172.21.15.32:0/3666505808 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f3d68077040 0x7f3d68079550 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7f3d74007fd0 tx=0x7f3d74001390 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:57:49.294 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:49.293+0000 7f3d85600640 1 -- 172.21.15.32:0/3666505808 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7f3d7c0614e0 con 0x7f3d880cccf0 2024-08-03T20:57:49.546 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:49.545+0000 7f3d90200640 1 -- 172.21.15.32:0/3666505808 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "joinable", "val": "true"} v 0) -- 0x7f3d880ce6f0 con 0x7f3d880cccf0 2024-08-03T20:57:49.814 INFO:tasks.ceph.mds.a.smithi032.stderr: -34> 2024-08-03T20:57:45.060+0000 7fac75200640 -1 mds.0.cache.ino(0x100) Corrupt inode 0x100: End of buffer [buffer:2] 2024-08-03T20:57:49.814 INFO:tasks.ceph.mds.a.smithi032.stderr: -24> 2024-08-03T20:57:45.060+0000 7fac75200640 -1 log_channel(cluster) log [ERR] : Error loading MDS rank 0: (22) Invalid argument 2024-08-03T20:57:49.925 INFO:tasks.ceph.mds.a.smithi032.stdout:starting mds.a at 2024-08-03T20:57:50.048 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:50.046+0000 7f3d85600640 1 -- 172.21.15.32:0/3666505808 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "joinable", "val": "true"}]=0 cephfs marked joinable; MDS may join as newly active. v229) ==== 162+0+0 (secure 0 0 0) 0x7f3d7c065a30 con 0x7f3d880cccf0 2024-08-03T20:57:50.048 INFO:teuthology.orchestra.run.smithi032.stderr:cephfs marked joinable; MDS may join as newly active. 2024-08-03T20:57:50.050 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:50.049+0000 7f3d90200640 1 -- 172.21.15.32:0/3666505808 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f3d68077040 msgr2=0x7f3d68079550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:50.051 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:50.049+0000 7f3d90200640 1 --2- 172.21.15.32:0/3666505808 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f3d68077040 0x7f3d68079550 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7f3d74007fd0 tx=0x7f3d74001390 comp rx=0 tx=0).stop 2024-08-03T20:57:50.051 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:50.049+0000 7f3d90200640 1 -- 172.21.15.32:0/3666505808 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3d880cccf0 msgr2=0x7f3d8813ba90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:50.051 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:50.049+0000 7f3d90200640 1 --2- 172.21.15.32:0/3666505808 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3d880cccf0 0x7f3d8813ba90 secure :-1 s=READY pgs=1636 cs=0 l=1 rev1=1 crypto rx=0x7f3d7c00d770 tx=0x7f3d7c00dc30 comp rx=0 tx=0).stop 2024-08-03T20:57:50.051 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:50.049+0000 7f3d90200640 1 -- 172.21.15.32:0/3666505808 shutdown_connections 2024-08-03T20:57:50.051 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:50.049+0000 7f3d90200640 1 --2- 172.21.15.32:0/3666505808 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f3d68077040 0x7f3d68079550 unknown :-1 s=CLOSED pgs=668 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:50.051 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:50.049+0000 7f3d90200640 1 --2- 172.21.15.32:0/3666505808 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f3d880ced30 0x7f3d88159720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:50.051 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:50.049+0000 7f3d90200640 1 --2- 172.21.15.32:0/3666505808 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f3d88145c50 0x7f3d88144580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:50.051 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:50.049+0000 7f3d90200640 1 --2- 172.21.15.32:0/3666505808 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3d880cccf0 0x7f3d8813ba90 unknown :-1 s=CLOSED pgs=1636 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:50.051 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:50.049+0000 7f3d90200640 1 -- 172.21.15.32:0/3666505808 >> 172.21.15.32:0/3666505808 conn(0x7f3d880b1bd0 msgr2=0x7f3d88142b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:57:50.051 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:50.049+0000 7f3d90200640 1 -- 172.21.15.32:0/3666505808 shutdown_connections 2024-08-03T20:57:50.051 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:50.049+0000 7f3d90200640 1 -- 172.21.15.32:0/3666505808 wait complete. 2024-08-03T20:57:50.063 INFO:tasks.ceph.mds.a.smithi032.stderr:2024-08-03T20:57:50.062+0000 7f31e2a00640 -1 mds.0.cache.ino(0x1) No data while reading inode 0x1 2024-08-03T20:57:50.063 INFO:tasks.ceph.mds.a.smithi032.stderr:2024-08-03T20:57:50.062+0000 7f31e2a00640 -1 log_channel(cluster) log [ERR] : Error loading MDS rank 0: (2) No such file or directory 2024-08-03T20:57:50.064 INFO:tasks.ceph.mon.a.smithi032.stderr:2024-08-03T20:57:50.062+0000 7f4e6d600640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-08-03T20:57:50.064 INFO:tasks.ceph.mon.a.smithi032.stderr:2024-08-03T20:57:50.062+0000 7f4e6d600640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 mds daemon damaged (MDS_DAMAGE) 2024-08-03T20:57:50.077 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-08-03T20:57:50.309 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:50.307+0000 7f218a000640 1 -- 172.21.15.32:0/2583515734 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2168005270 msgr2=0x7f216800b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:50.309 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:50.307+0000 7f218a000640 1 --2- 172.21.15.32:0/2583515734 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2168005270 0x7f216800b9c0 secure :-1 s=READY pgs=1640 cs=0 l=1 rev1=1 crypto rx=0x7f217c0013d0 tx=0x7f2168005830 comp rx=0 tx=0).stop 2024-08-03T20:57:50.309 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:50.307+0000 7f218a000640 1 -- 172.21.15.32:0/2583515734 shutdown_connections 2024-08-03T20:57:50.309 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:50.307+0000 7f218a000640 1 --2- 172.21.15.32:0/2583515734 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2168005270 0x7f216800b9c0 unknown :-1 s=CLOSED pgs=1640 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:50.309 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:50.307+0000 7f218a000640 1 -- 172.21.15.32:0/2583515734 >> 172.21.15.32:0/2583515734 conn(0x7f21840b1bd0 msgr2=0x7f21840b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:57:50.309 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:50.307+0000 7f218a000640 1 -- 172.21.15.32:0/2583515734 shutdown_connections 2024-08-03T20:57:50.309 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:50.307+0000 7f218a000640 1 -- 172.21.15.32:0/2583515734 wait complete. 2024-08-03T20:57:50.311 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:50.309+0000 7f218a000640 1 Processor -- start 2024-08-03T20:57:50.311 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:50.310+0000 7f218a000640 1 -- start start 2024-08-03T20:57:50.313 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:50.311+0000 7f218a000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2168005270 0x7f218413eaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:50.314 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:50.311+0000 7f218a000640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f21841e7b00 0x7f21840ae1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:50.314 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:50.311+0000 7f218a000640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f21841e7f10 0x7f21840ae6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:50.314 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:50.311+0000 7f218a000640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f21840cf840 con 0x7f2168005270 2024-08-03T20:57:50.314 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:50.311+0000 7f218a000640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f2184145f30 con 0x7f21841e7b00 2024-08-03T20:57:50.314 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:50.311+0000 7f218a000640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7f21840cf9c0 con 0x7f21841e7f10 2024-08-03T20:57:50.314 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:50.311+0000 7f2183400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2168005270 0x7f218413eaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:50.314 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:50.311+0000 7f2183400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2168005270 0x7f218413eaa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:51182/0 (socket says 172.21.15.32:51182) 2024-08-03T20:57:50.314 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:50.311+0000 7f2183400640 1 -- 172.21.15.32:0/2830795333 learned_addr learned my addr 172.21.15.32:0/2830795333 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:57:50.314 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:50.312+0000 7f217aa00640 1 --2- 172.21.15.32:0/2830795333 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f21841e7b00 0x7f21840ae1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:50.314 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:50.312+0000 7f2183e00640 1 --2- 172.21.15.32:0/2830795333 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f21841e7f10 0x7f21840ae6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:50.314 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:50.312+0000 7f2183400640 1 -- 172.21.15.32:0/2830795333 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f21841e7f10 msgr2=0x7f21840ae6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:50.314 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:50.312+0000 7f2183400640 1 --2- 172.21.15.32:0/2830795333 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f21841e7f10 0x7f21840ae6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:50.314 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:50.312+0000 7f2183400640 1 -- 172.21.15.32:0/2830795333 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f21841e7b00 msgr2=0x7f21840ae1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:50.315 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:50.312+0000 7f2183400640 1 --2- 172.21.15.32:0/2830795333 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f21841e7b00 0x7f21840ae1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:50.315 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:50.312+0000 7f2183400640 1 -- 172.21.15.32:0/2830795333 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f21841f97f0 con 0x7f2168005270 2024-08-03T20:57:50.315 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:50.312+0000 7f2183400640 1 --2- 172.21.15.32:0/2830795333 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2168005270 0x7f218413eaa0 secure :-1 s=READY pgs=1641 cs=0 l=1 rev1=1 crypto rx=0x7f2174003ed0 tx=0x7f2174001630 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:57:50.315 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:50.312+0000 7f217be00640 1 -- 172.21.15.32:0/2830795333 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2174015070 con 0x7f2168005270 2024-08-03T20:57:50.316 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:50.313+0000 7f217be00640 1 -- 172.21.15.32:0/2830795333 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f2174011840 con 0x7f2168005270 2024-08-03T20:57:50.316 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:50.313+0000 7f217be00640 1 -- 172.21.15.32:0/2830795333 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2174011c70 con 0x7f2168005270 2024-08-03T20:57:50.316 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:50.313+0000 7f218a000640 1 -- 172.21.15.32:0/2830795333 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f21841e86e0 con 0x7f2168005270 2024-08-03T20:57:50.316 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:50.313+0000 7f218a000640 1 -- 172.21.15.32:0/2830795333 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f21841f59d0 con 0x7f2168005270 2024-08-03T20:57:50.316 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:50.314+0000 7f218a000640 1 -- 172.21.15.32:0/2830795333 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f21840740e0 con 0x7f2168005270 2024-08-03T20:57:50.316 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:50.315+0000 7f217be00640 1 -- 172.21.15.32:0/2830795333 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f2174018070 con 0x7f2168005270 2024-08-03T20:57:50.317 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:50.316+0000 7f217be00640 1 --2- 172.21.15.32:0/2830795333 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f2160077040 0x7f2160079550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:50.318 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:50.316+0000 7f217be00640 1 -- 172.21.15.32:0/2830795333 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(109..109 src has 1..109) ==== 7918+0+0 (secure 0 0 0) 0x7f2174099870 con 0x7f2168005270 2024-08-03T20:57:50.318 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:50.317+0000 7f217aa00640 1 --2- 172.21.15.32:0/2830795333 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f2160077040 0x7f2160079550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:50.318 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:50.317+0000 7f217aa00640 1 --2- 172.21.15.32:0/2830795333 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f2160077040 0x7f2160079550 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7f218415de40 tx=0x7f216c005f90 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:57:50.322 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:50.321+0000 7f217be00640 1 -- 172.21.15.32:0/2830795333 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7f2174061be0 con 0x7f2168005270 2024-08-03T20:57:50.567 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:50.565+0000 7f218a000640 1 -- 172.21.15.32:0/2830795333 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f21840cf220 con 0x7f2168005270 2024-08-03T20:57:50.567 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:50.566+0000 7f217be00640 1 -- 172.21.15.32:0/2830795333 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 230 v230) ==== 95+0+4391 (secure 0 0 0) 0x7f2174066130 con 0x7f2168005270 2024-08-03T20:57:50.567 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-03T20:57:50.568 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":230,"btime":"2024-08-03T20:57:50:064229+0000","default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6439,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6835/531080160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6834","nonce":531080160},{"type":"v1","addr":"172.21.15.118:6835","nonce":531080160}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":84},{"gid":6454,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6837/2796882319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6836","nonce":2796882319},{"type":"v1","addr":"172.21.15.118:6837","nonce":2796882319}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":86},{"gid":6813,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6834/2686933890","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":2686933890},{"type":"v1","addr":"172.21.15.32:6834","nonce":2686933890}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":85}],"filesystems":[{"mdsmap":{"epoch":230,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-08-03T20:54:26.950518+0000","modified":"2024-08-03T20:57:50.064165+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":109,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":6}]} 2024-08-03T20:57:50.568 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 230 2024-08-03T20:57:50.570 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:50.568+0000 7f218a000640 1 -- 172.21.15.32:0/2830795333 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f2160077040 msgr2=0x7f2160079550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:50.570 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:50.568+0000 7f218a000640 1 --2- 172.21.15.32:0/2830795333 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f2160077040 0x7f2160079550 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7f218415de40 tx=0x7f216c005f90 comp rx=0 tx=0).stop 2024-08-03T20:57:50.570 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:50.568+0000 7f218a000640 1 -- 172.21.15.32:0/2830795333 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2168005270 msgr2=0x7f218413eaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:50.570 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:50.568+0000 7f218a000640 1 --2- 172.21.15.32:0/2830795333 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2168005270 0x7f218413eaa0 secure :-1 s=READY pgs=1641 cs=0 l=1 rev1=1 crypto rx=0x7f2174003ed0 tx=0x7f2174001630 comp rx=0 tx=0).stop 2024-08-03T20:57:50.570 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:50.568+0000 7f218a000640 1 -- 172.21.15.32:0/2830795333 shutdown_connections 2024-08-03T20:57:50.570 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:50.568+0000 7f218a000640 1 --2- 172.21.15.32:0/2830795333 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f2160077040 0x7f2160079550 unknown :-1 s=CLOSED pgs=670 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:50.570 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:50.568+0000 7f218a000640 1 --2- 172.21.15.32:0/2830795333 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f21841e7b00 0x7f21840ae1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:50.570 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:50.568+0000 7f218a000640 1 --2- 172.21.15.32:0/2830795333 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f21841e7f10 0x7f21840ae6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:50.570 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:50.568+0000 7f218a000640 1 --2- 172.21.15.32:0/2830795333 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2168005270 0x7f218413eaa0 unknown :-1 s=CLOSED pgs=1641 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:50.570 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:50.568+0000 7f218a000640 1 -- 172.21.15.32:0/2830795333 >> 172.21.15.32:0/2830795333 conn(0x7f21840b1bd0 msgr2=0x7f21840b1220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:57:50.570 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:50.568+0000 7f218a000640 1 -- 172.21.15.32:0/2830795333 shutdown_connections 2024-08-03T20:57:50.571 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:50.568+0000 7f218a000640 1 -- 172.21.15.32:0/2830795333 wait complete. 2024-08-03T20:57:50.593 DEBUG:tasks.ceph_test_case:wait_until_true: success in 0s and 0 retries 2024-08-03T20:57:50.593 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-08-03T20:57:50.821 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:50.819+0000 7fa29fe00640 1 -- 172.21.15.32:0/234424296 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa298144f90 msgr2=0x7fa298074570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:50.821 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:50.819+0000 7fa29fe00640 1 --2- 172.21.15.32:0/234424296 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa298144f90 0x7fa298074570 secure :-1 s=READY pgs=1642 cs=0 l=1 rev1=1 crypto rx=0x7fa284005fd0 tx=0x7fa28402a530 comp rx=0 tx=0).stop 2024-08-03T20:57:50.822 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:50.819+0000 7fa29fe00640 1 -- 172.21.15.32:0/234424296 shutdown_connections 2024-08-03T20:57:50.822 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:50.819+0000 7fa29fe00640 1 --2- 172.21.15.32:0/234424296 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fa29815fff0 0x7fa298160400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:50.822 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:50.819+0000 7fa29fe00640 1 --2- 172.21.15.32:0/234424296 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa298144f90 0x7fa298074570 unknown :-1 s=CLOSED pgs=1642 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:50.822 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:50.819+0000 7fa29fe00640 1 -- 172.21.15.32:0/234424296 >> 172.21.15.32:0/234424296 conn(0x7fa2980b1bd0 msgr2=0x7fa2980b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:57:50.822 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:50.820+0000 7fa29fe00640 1 -- 172.21.15.32:0/234424296 shutdown_connections 2024-08-03T20:57:50.822 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:50.820+0000 7fa29fe00640 1 -- 172.21.15.32:0/234424296 wait complete. 2024-08-03T20:57:50.823 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:50.822+0000 7fa29fe00640 1 Processor -- start 2024-08-03T20:57:50.823 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:50.822+0000 7fa29fe00640 1 -- start start 2024-08-03T20:57:50.825 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:50.823+0000 7fa29fe00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa29806e930 0x7fa2981f3bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:50.826 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:50.823+0000 7fa29fe00640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fa298144f90 0x7fa298156e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:50.826 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:50.823+0000 7fa29fe00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa29815fff0 0x7fa298157480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:50.826 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:50.823+0000 7fa29fe00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fa2980cf840 con 0x7fa29815fff0 2024-08-03T20:57:50.826 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:50.823+0000 7fa29fe00640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7fa298145f30 con 0x7fa29806e930 2024-08-03T20:57:50.826 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:50.823+0000 7fa29fe00640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7fa2980cf9c0 con 0x7fa298144f90 2024-08-03T20:57:50.826 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:50.824+0000 7fa29de00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa29815fff0 0x7fa298157480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:50.826 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:50.824+0000 7fa29d400640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa29806e930 0x7fa2981f3bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:50.826 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:50.824+0000 7fa29ca00640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fa298144f90 0x7fa298156e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:50.826 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:50.824+0000 7fa29d400640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa29806e930 0x7fa2981f3bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.32:32834/0 (socket says 172.21.15.32:32834) 2024-08-03T20:57:50.826 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:50.824+0000 7fa29d400640 1 -- 172.21.15.32:0/1903861628 learned_addr learned my addr 172.21.15.32:0/1903861628 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:57:50.826 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:50.824+0000 7fa29d400640 1 -- 172.21.15.32:0/1903861628 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fa298144f90 msgr2=0x7fa298156e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:50.827 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:50.824+0000 7fa29d400640 1 --2- 172.21.15.32:0/1903861628 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fa298144f90 0x7fa298156e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:50.827 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:50.824+0000 7fa29d400640 1 -- 172.21.15.32:0/1903861628 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa29815fff0 msgr2=0x7fa298157480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:50.827 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:50.824+0000 7fa29d400640 1 --2- 172.21.15.32:0/1903861628 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa29815fff0 0x7fa298157480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:50.827 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:50.824+0000 7fa29d400640 1 -- 172.21.15.32:0/1903861628 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa2981579c0 con 0x7fa29806e930 2024-08-03T20:57:50.828 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:50.825+0000 7fa29d400640 1 --2- 172.21.15.32:0/1903861628 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa29806e930 0x7fa2981f3bd0 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7fa28c00b3f0 tx=0x7fa28c00b8b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:57:50.828 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:50.825+0000 7fa295600640 1 -- 172.21.15.32:0/1903861628 <== mon.1 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa28c015020 con 0x7fa29806e930 2024-08-03T20:57:50.828 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:50.825+0000 7fa295600640 1 -- 172.21.15.32:0/1903861628 <== mon.1 v2:172.21.15.118:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fa28c011920 con 0x7fa29806e930 2024-08-03T20:57:50.828 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:50.825+0000 7fa29fe00640 1 -- 172.21.15.32:0/1903861628 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa298142640 con 0x7fa29806e930 2024-08-03T20:57:50.828 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:50.825+0000 7fa295600640 1 -- 172.21.15.32:0/1903861628 <== mon.1 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa28c011d70 con 0x7fa29806e930 2024-08-03T20:57:50.828 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:50.825+0000 7fa29fe00640 1 -- 172.21.15.32:0/1903861628 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa298142c00 con 0x7fa29806e930 2024-08-03T20:57:50.828 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:50.826+0000 7fa29fe00640 1 -- 172.21.15.32:0/1903861628 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa298160cc0 con 0x7fa29806e930 2024-08-03T20:57:50.829 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:50.828+0000 7fa295600640 1 -- 172.21.15.32:0/1903861628 <== mon.1 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7fa28c018020 con 0x7fa29806e930 2024-08-03T20:57:50.830 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:50.828+0000 7fa295600640 1 --2- 172.21.15.32:0/1903861628 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fa278077010 0x7fa278079520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:50.830 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:50.829+0000 7fa295600640 1 -- 172.21.15.32:0/1903861628 <== mon.1 v2:172.21.15.118:3300/0 5 ==== osd_map(109..109 src has 1..109) ==== 7918+0+0 (secure 0 0 0) 0x7fa28c098f60 con 0x7fa29806e930 2024-08-03T20:57:50.830 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:50.829+0000 7fa29ca00640 1 --2- 172.21.15.32:0/1903861628 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fa278077010 0x7fa278079520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:50.831 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:50.830+0000 7fa29ca00640 1 --2- 172.21.15.32:0/1903861628 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fa278077010 0x7fa278079520 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7fa284033000 tx=0x7fa284004730 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:57:50.833 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:50.832+0000 7fa295600640 1 -- 172.21.15.32:0/1903861628 <== mon.1 v2:172.21.15.118:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7fa28c01f030 con 0x7fa29806e930 2024-08-03T20:57:51.076 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:51.074+0000 7fa29fe00640 1 -- 172.21.15.32:0/1903861628 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa2981459c0 con 0x7fa29806e930 2024-08-03T20:57:51.076 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:51.075+0000 7fa295600640 1 -- 172.21.15.32:0/1903861628 <== mon.1 v2:172.21.15.118:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 230 v230) ==== 95+0+4391 (secure 0 0 0) 0x7fa28c061260 con 0x7fa29806e930 2024-08-03T20:57:51.076 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-03T20:57:51.077 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":230,"btime":"2024-08-03T20:57:50:064229+0000","default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6439,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6835/531080160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6834","nonce":531080160},{"type":"v1","addr":"172.21.15.118:6835","nonce":531080160}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":84},{"gid":6454,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6837/2796882319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6836","nonce":2796882319},{"type":"v1","addr":"172.21.15.118:6837","nonce":2796882319}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":86},{"gid":6813,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6834/2686933890","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":2686933890},{"type":"v1","addr":"172.21.15.32:6834","nonce":2686933890}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":85}],"filesystems":[{"mdsmap":{"epoch":230,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-08-03T20:54:26.950518+0000","modified":"2024-08-03T20:57:50.064165+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":109,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":6}]} 2024-08-03T20:57:51.077 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 230 2024-08-03T20:57:51.079 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:51.077+0000 7fa29fe00640 1 -- 172.21.15.32:0/1903861628 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fa278077010 msgr2=0x7fa278079520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:51.079 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:51.077+0000 7fa29fe00640 1 --2- 172.21.15.32:0/1903861628 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fa278077010 0x7fa278079520 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7fa284033000 tx=0x7fa284004730 comp rx=0 tx=0).stop 2024-08-03T20:57:51.079 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:51.077+0000 7fa29fe00640 1 -- 172.21.15.32:0/1903861628 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa29806e930 msgr2=0x7fa2981f3bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:51.079 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:51.077+0000 7fa29fe00640 1 --2- 172.21.15.32:0/1903861628 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa29806e930 0x7fa2981f3bd0 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7fa28c00b3f0 tx=0x7fa28c00b8b0 comp rx=0 tx=0).stop 2024-08-03T20:57:51.079 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:51.077+0000 7fa29fe00640 1 -- 172.21.15.32:0/1903861628 shutdown_connections 2024-08-03T20:57:51.079 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:51.077+0000 7fa29fe00640 1 --2- 172.21.15.32:0/1903861628 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fa278077010 0x7fa278079520 unknown :-1 s=CLOSED pgs=671 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:51.079 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:51.077+0000 7fa29fe00640 1 --2- 172.21.15.32:0/1903861628 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa29815fff0 0x7fa298157480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:51.079 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:51.077+0000 7fa29fe00640 1 --2- 172.21.15.32:0/1903861628 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fa298144f90 0x7fa298156e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:51.080 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:51.077+0000 7fa29fe00640 1 --2- 172.21.15.32:0/1903861628 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa29806e930 0x7fa2981f3bd0 unknown :-1 s=CLOSED pgs=511 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:51.080 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:51.077+0000 7fa29fe00640 1 -- 172.21.15.32:0/1903861628 >> 172.21.15.32:0/1903861628 conn(0x7fa2980b1bd0 msgr2=0x7fa29806f930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:57:51.080 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:51.077+0000 7fa29fe00640 1 -- 172.21.15.32:0/1903861628 shutdown_connections 2024-08-03T20:57:51.080 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:51.077+0000 7fa29fe00640 1 -- 172.21.15.32:0/1903861628 wait complete. 2024-08-03T20:57:51.102 DEBUG:tasks.ceph_test_case:wait_until_true: success in 0s and 0 retries 2024-08-03T20:57:51.102 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-08-03T20:57:51.331 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:51.329+0000 7f68c2000640 1 -- 172.21.15.32:0/2823499288 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f68bc15b790 msgr2=0x7f68bc15f490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:51.331 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:51.329+0000 7f68c2000640 1 --2- 172.21.15.32:0/2823499288 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f68bc15b790 0x7f68bc15f490 secure :-1 s=READY pgs=1644 cs=0 l=1 rev1=1 crypto rx=0x7f68a8009830 tx=0x7f68a802a390 comp rx=0 tx=0).stop 2024-08-03T20:57:51.331 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:51.329+0000 7f68c2000640 1 -- 172.21.15.32:0/2823499288 shutdown_connections 2024-08-03T20:57:51.331 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:51.329+0000 7f68c2000640 1 --2- 172.21.15.32:0/2823499288 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f68bc15b790 0x7f68bc15f490 unknown :-1 s=CLOSED pgs=1644 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:51.331 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:51.329+0000 7f68c2000640 1 -- 172.21.15.32:0/2823499288 >> 172.21.15.32:0/2823499288 conn(0x7f68bc0d0dd0 msgr2=0x7f68bc0d11e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:57:51.331 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:51.329+0000 7f68c2000640 1 -- 172.21.15.32:0/2823499288 shutdown_connections 2024-08-03T20:57:51.331 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:51.329+0000 7f68c2000640 1 -- 172.21.15.32:0/2823499288 wait complete. 2024-08-03T20:57:51.332 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:51.331+0000 7f68c2000640 1 Processor -- start 2024-08-03T20:57:51.332 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:51.331+0000 7f68c2000640 1 -- start start 2024-08-03T20:57:51.334 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:51.333+0000 7f68c2000640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f68bc071150 0x7f68bc1512c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:51.334 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:51.333+0000 7f68c2000640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f68bc15b790 0x7f68bc0b4610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:51.334 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:51.333+0000 7f68c2000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f68bc15f9d0 0x7f68bc0b4b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:51.335 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:51.333+0000 7f68c2000640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f68bc073830 con 0x7f68bc15f9d0 2024-08-03T20:57:51.335 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:51.333+0000 7f68c2000640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f68bc0736b0 con 0x7f68bc15b790 2024-08-03T20:57:51.335 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:51.333+0000 7f68c2000640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7f68bc0739b0 con 0x7f68bc071150 2024-08-03T20:57:51.335 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:51.333+0000 7f68c1600640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f68bc15f9d0 0x7f68bc0b4b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:51.335 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:51.333+0000 7f68c1600640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f68bc15f9d0 0x7f68bc0b4b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:51222/0 (socket says 172.21.15.32:51222) 2024-08-03T20:57:51.335 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:51.333+0000 7f68c1600640 1 -- 172.21.15.32:0/4208790822 learned_addr learned my addr 172.21.15.32:0/4208790822 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:57:51.336 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:51.333+0000 7f68c1600640 1 -- 172.21.15.32:0/4208790822 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f68bc071150 msgr2=0x7f68bc1512c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:51.336 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:51.333+0000 7f68c1600640 1 --2- 172.21.15.32:0/4208790822 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f68bc071150 0x7f68bc1512c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:51.336 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:51.333+0000 7f68c1600640 1 -- 172.21.15.32:0/4208790822 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f68bc15b790 msgr2=0x7f68bc0b4610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:51.336 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:51.333+0000 7f68bbe00640 1 --2- 172.21.15.32:0/4208790822 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f68bc15b790 0x7f68bc0b4610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:51.337 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:51.333+0000 7f68c1600640 1 --2- 172.21.15.32:0/4208790822 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f68bc15b790 0x7f68bc0b4610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:51.337 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:51.333+0000 7f68c1600640 1 -- 172.21.15.32:0/4208790822 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f68bc0ca830 con 0x7f68bc15f9d0 2024-08-03T20:57:51.337 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:51.333+0000 7f68bbe00640 1 --2- 172.21.15.32:0/4208790822 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f68bc15b790 0x7f68bc0b4610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-03T20:57:51.337 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:51.334+0000 7f68c1600640 1 --2- 172.21.15.32:0/4208790822 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f68bc15f9d0 0x7f68bc0b4b50 secure :-1 s=READY pgs=1645 cs=0 l=1 rev1=1 crypto rx=0x7f68b400c840 tx=0x7f68b400cd00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:57:51.337 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:51.334+0000 7f68b8c00640 1 -- 172.21.15.32:0/4208790822 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68b4017070 con 0x7f68bc15f9d0 2024-08-03T20:57:51.337 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:51.334+0000 7f68b8c00640 1 -- 172.21.15.32:0/4208790822 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f68b4005070 con 0x7f68bc15f9d0 2024-08-03T20:57:51.337 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:51.334+0000 7f68c2000640 1 -- 172.21.15.32:0/4208790822 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f68bc0b32c0 con 0x7f68bc15f9d0 2024-08-03T20:57:51.337 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:51.334+0000 7f68b8c00640 1 -- 172.21.15.32:0/4208790822 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68b40054e0 con 0x7f68bc15f9d0 2024-08-03T20:57:51.337 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:51.334+0000 7f68c2000640 1 -- 172.21.15.32:0/4208790822 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f68bc0ca310 con 0x7f68bc15f9d0 2024-08-03T20:57:51.338 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:51.335+0000 7f68c2000640 1 -- 172.21.15.32:0/4208790822 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f68bc073e50 con 0x7f68bc15f9d0 2024-08-03T20:57:51.338 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:51.336+0000 7f68b8c00640 1 -- 172.21.15.32:0/4208790822 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f68b401a070 con 0x7f68bc15f9d0 2024-08-03T20:57:51.339 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:51.338+0000 7f68b8c00640 1 --2- 172.21.15.32:0/4208790822 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f689c077040 0x7f689c079550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:51.339 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:51.338+0000 7f68b8c00640 1 -- 172.21.15.32:0/4208790822 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(109..109 src has 1..109) ==== 7918+0+0 (secure 0 0 0) 0x7f68b4098980 con 0x7f68bc15f9d0 2024-08-03T20:57:51.340 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:51.338+0000 7f68c0c00640 1 --2- 172.21.15.32:0/4208790822 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f689c077040 0x7f689c079550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:51.340 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:51.339+0000 7f68c0c00640 1 --2- 172.21.15.32:0/4208790822 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f689c077040 0x7f689c079550 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7f68b0001510 tx=0x7f68b0007930 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:57:51.344 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:51.343+0000 7f68b8c00640 1 -- 172.21.15.32:0/4208790822 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7f68b40612d0 con 0x7f68bc15f9d0 2024-08-03T20:57:51.588 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:51.586+0000 7f68c2000640 1 -- 172.21.15.32:0/4208790822 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f68bc0b6d50 con 0x7f68bc15f9d0 2024-08-03T20:57:51.588 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:51.587+0000 7f68b8c00640 1 -- 172.21.15.32:0/4208790822 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 230 v230) ==== 95+0+4391 (secure 0 0 0) 0x7f68b4060bd0 con 0x7f68bc15f9d0 2024-08-03T20:57:51.588 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-03T20:57:51.589 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":230,"btime":"2024-08-03T20:57:50:064229+0000","default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6439,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6835/531080160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6834","nonce":531080160},{"type":"v1","addr":"172.21.15.118:6835","nonce":531080160}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":84},{"gid":6454,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6837/2796882319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6836","nonce":2796882319},{"type":"v1","addr":"172.21.15.118:6837","nonce":2796882319}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":86},{"gid":6813,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6834/2686933890","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":2686933890},{"type":"v1","addr":"172.21.15.32:6834","nonce":2686933890}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":85}],"filesystems":[{"mdsmap":{"epoch":230,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-08-03T20:54:26.950518+0000","modified":"2024-08-03T20:57:50.064165+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":109,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":6}]} 2024-08-03T20:57:51.589 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 230 2024-08-03T20:57:51.591 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:51.589+0000 7f68c2000640 1 -- 172.21.15.32:0/4208790822 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f689c077040 msgr2=0x7f689c079550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:51.591 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:51.589+0000 7f68c2000640 1 --2- 172.21.15.32:0/4208790822 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f689c077040 0x7f689c079550 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7f68b0001510 tx=0x7f68b0007930 comp rx=0 tx=0).stop 2024-08-03T20:57:51.591 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:51.589+0000 7f68c2000640 1 -- 172.21.15.32:0/4208790822 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f68bc15f9d0 msgr2=0x7f68bc0b4b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:51.591 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:51.589+0000 7f68c2000640 1 --2- 172.21.15.32:0/4208790822 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f68bc15f9d0 0x7f68bc0b4b50 secure :-1 s=READY pgs=1645 cs=0 l=1 rev1=1 crypto rx=0x7f68b400c840 tx=0x7f68b400cd00 comp rx=0 tx=0).stop 2024-08-03T20:57:51.591 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:51.589+0000 7f68c2000640 1 -- 172.21.15.32:0/4208790822 shutdown_connections 2024-08-03T20:57:51.591 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:51.589+0000 7f68c2000640 1 --2- 172.21.15.32:0/4208790822 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f689c077040 0x7f689c079550 unknown :-1 s=CLOSED pgs=672 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:51.591 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:51.589+0000 7f68c2000640 1 --2- 172.21.15.32:0/4208790822 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f68bc15b790 0x7f68bc0b4610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:51.591 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:51.589+0000 7f68c2000640 1 --2- 172.21.15.32:0/4208790822 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f68bc15f9d0 0x7f68bc0b4b50 unknown :-1 s=CLOSED pgs=1645 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:51.592 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:51.589+0000 7f68c2000640 1 --2- 172.21.15.32:0/4208790822 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f68bc071150 0x7f68bc1512c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:51.592 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:51.589+0000 7f68c2000640 1 -- 172.21.15.32:0/4208790822 >> 172.21.15.32:0/4208790822 conn(0x7f68bc0d0dd0 msgr2=0x7f68bc0cc060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:57:51.592 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:51.589+0000 7f68c2000640 1 -- 172.21.15.32:0/4208790822 shutdown_connections 2024-08-03T20:57:51.592 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:51.589+0000 7f68c2000640 1 -- 172.21.15.32:0/4208790822 wait complete. 2024-08-03T20:57:51.613 INFO:tasks.cephfs.test_damage:Result: Mutation 'Truncate 1.00000000.inode' led to DAMAGED state 2024-08-03T20:57:51.613 INFO:tasks.cephfs.test_damage:Applying mutation 'Truncate 200.00000000' 2024-08-03T20:57:51.613 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-08-03T20:57:51.614 DEBUG:teuthology.orchestra.run.smithi032:> dd if=/proc/self/mounts of=/dev/stdout 2024-08-03T20:57:51.649 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053453,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6499208k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 none /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249604k,nr_inodes=812401,mode=700,uid=1000,gid=1316 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6499208k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-08-03T20:57:51.650 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi032.front.sepia.ceph.com 2024-08-03T20:57:51.650 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-08-03T20:57:51.650 DEBUG:teuthology.orchestra.run.smithi032:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-08-03T20:57:51.725 INFO:teuthology.orchestra.run.smithi032.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.0': No such file or directory 2024-08-03T20:57:51.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-03T20:57:51.726 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-08-03T20:57:51.996 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:51.994+0000 7f51f2000640 1 -- 172.21.15.32:0/1094929819 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f51ec15f610 msgr2=0x7f51ec1523f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:51.996 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:51.994+0000 7f51f2000640 1 --2- 172.21.15.32:0/1094929819 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f51ec15f610 0x7f51ec1523f0 secure :-1 s=READY pgs=1647 cs=0 l=1 rev1=1 crypto rx=0x7f51d4009830 tx=0x7f51d402a530 comp rx=0 tx=0).stop 2024-08-03T20:57:51.996 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:51.994+0000 7f51f2000640 1 -- 172.21.15.32:0/1094929819 shutdown_connections 2024-08-03T20:57:51.996 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:51.994+0000 7f51f2000640 1 --2- 172.21.15.32:0/1094929819 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f51ec15f610 0x7f51ec1523f0 unknown :-1 s=CLOSED pgs=1647 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:51.996 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:51.994+0000 7f51f2000640 1 -- 172.21.15.32:0/1094929819 >> 172.21.15.32:0/1094929819 conn(0x7f51ec0b1bd0 msgr2=0x7f51ec0b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:57:51.996 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:51.994+0000 7f51f2000640 1 -- 172.21.15.32:0/1094929819 shutdown_connections 2024-08-03T20:57:51.996 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:51.994+0000 7f51f2000640 1 -- 172.21.15.32:0/1094929819 wait complete. 2024-08-03T20:57:51.998 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:51.996+0000 7f51f2000640 1 Processor -- start 2024-08-03T20:57:51.998 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:51.996+0000 7f51f2000640 1 -- start start 2024-08-03T20:57:52.001 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:51.998+0000 7f51f2000640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f51ec0b38b0 0x7f51ec152a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:52.001 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:51.998+0000 7f51f2000640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f51ec0b4ce0 0x7f51ec0af140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:52.001 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:51.998+0000 7f51f2000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f51ec1f2d30 0x7f51ec1548a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:52.001 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:51.998+0000 7f51f2000640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f51ec0cec50 con 0x7f51ec1f2d30 2024-08-03T20:57:52.001 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:51.998+0000 7f51f2000640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f51ec0cead0 con 0x7f51ec0b4ce0 2024-08-03T20:57:52.001 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:51.998+0000 7f51f2000640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7f51ec0cedd0 con 0x7f51ec0b38b0 2024-08-03T20:57:52.001 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:51.998+0000 7f51ebe00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f51ec1f2d30 0x7f51ec1548a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:52.001 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:51.998+0000 7f51ebe00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f51ec1f2d30 0x7f51ec1548a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:51248/0 (socket says 172.21.15.32:51248) 2024-08-03T20:57:52.001 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:51.998+0000 7f51ebe00640 1 -- 172.21.15.32:0/285212646 learned_addr learned my addr 172.21.15.32:0/285212646 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:57:52.002 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:51.998+0000 7f51eb400640 1 --2- 172.21.15.32:0/285212646 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f51ec0b38b0 0x7f51ec152a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:52.002 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:51.998+0000 7f51e2a00640 1 --2- 172.21.15.32:0/285212646 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f51ec0b4ce0 0x7f51ec0af140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:52.002 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:51.999+0000 7f51ebe00640 1 -- 172.21.15.32:0/285212646 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f51ec0b38b0 msgr2=0x7f51ec152a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:52.002 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:51.999+0000 7f51ebe00640 1 --2- 172.21.15.32:0/285212646 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f51ec0b38b0 0x7f51ec152a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:52.002 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:51.999+0000 7f51ebe00640 1 -- 172.21.15.32:0/285212646 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f51ec0b4ce0 msgr2=0x7f51ec0af140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:52.002 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:51.999+0000 7f51ebe00640 1 --2- 172.21.15.32:0/285212646 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f51ec0b4ce0 0x7f51ec0af140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:52.002 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:51.999+0000 7f51ebe00640 1 -- 172.21.15.32:0/285212646 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f51ec1f34d0 con 0x7f51ec1f2d30 2024-08-03T20:57:52.002 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:51.999+0000 7f51ebe00640 1 --2- 172.21.15.32:0/285212646 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f51ec1f2d30 0x7f51ec1548a0 secure :-1 s=READY pgs=1648 cs=0 l=1 rev1=1 crypto rx=0x7f51e400c730 tx=0x7f51e400cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:57:52.002 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:51.999+0000 7f51e3e00640 1 -- 172.21.15.32:0/285212646 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51e4003b70 con 0x7f51ec1f2d30 2024-08-03T20:57:52.002 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:51.999+0000 7f51e3e00640 1 -- 172.21.15.32:0/285212646 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f51e4003d10 con 0x7f51ec1f2d30 2024-08-03T20:57:52.002 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:51.999+0000 7f51e3e00640 1 -- 172.21.15.32:0/285212646 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51e400aab0 con 0x7f51ec1f2d30 2024-08-03T20:57:52.002 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:51.999+0000 7f51f2000640 1 -- 172.21.15.32:0/285212646 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f51ec1f37b0 con 0x7f51ec1f2d30 2024-08-03T20:57:52.003 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:51.999+0000 7f51f2000640 1 -- 172.21.15.32:0/285212646 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f51ec0af680 con 0x7f51ec1f2d30 2024-08-03T20:57:52.003 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:52.001+0000 7f51f2000640 1 -- 172.21.15.32:0/285212646 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f51ec1f3240 con 0x7f51ec1f2d30 2024-08-03T20:57:52.003 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:52.002+0000 7f51e3e00640 1 -- 172.21.15.32:0/285212646 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f51e400ac50 con 0x7f51ec1f2d30 2024-08-03T20:57:52.004 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:52.003+0000 7f51e3e00640 1 --2- 172.21.15.32:0/285212646 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f51c8077010 0x7f51c8079520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:52.004 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:52.003+0000 7f51e3e00640 1 -- 172.21.15.32:0/285212646 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(109..109 src has 1..109) ==== 7918+0+0 (secure 0 0 0) 0x7f51e4098f80 con 0x7f51ec1f2d30 2024-08-03T20:57:52.005 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:52.004+0000 7f51eb400640 1 --2- 172.21.15.32:0/285212646 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f51c8077010 0x7f51c8079520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:52.006 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:52.004+0000 7f51eb400640 1 --2- 172.21.15.32:0/285212646 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f51c8077010 0x7f51c8079520 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7f51dc0046b0 tx=0x7f51dc007cc0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:57:52.009 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:52.008+0000 7f51e3e00640 1 -- 172.21.15.32:0/285212646 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7f51e4061280 con 0x7f51ec1f2d30 2024-08-03T20:57:52.253 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:52.251+0000 7f51f2000640 1 -- 172.21.15.32:0/285212646 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7f51ec1523f0 con 0x7f51ec1f2d30 2024-08-03T20:57:52.466 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:52.464+0000 7f51e3e00640 1 -- 172.21.15.32:0/285212646 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v231) ==== 161+0+0 (secure 0 0 0) 0x7f51e40657d0 con 0x7f51ec1f2d30 2024-08-03T20:57:52.466 INFO:teuthology.orchestra.run.smithi032.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-08-03T20:57:52.468 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:52.467+0000 7f51f2000640 1 -- 172.21.15.32:0/285212646 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f51c8077010 msgr2=0x7f51c8079520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:52.468 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:52.467+0000 7f51f2000640 1 --2- 172.21.15.32:0/285212646 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f51c8077010 0x7f51c8079520 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7f51dc0046b0 tx=0x7f51dc007cc0 comp rx=0 tx=0).stop 2024-08-03T20:57:52.468 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:52.467+0000 7f51f2000640 1 -- 172.21.15.32:0/285212646 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f51ec1f2d30 msgr2=0x7f51ec1548a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:52.469 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:52.467+0000 7f51f2000640 1 --2- 172.21.15.32:0/285212646 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f51ec1f2d30 0x7f51ec1548a0 secure :-1 s=READY pgs=1648 cs=0 l=1 rev1=1 crypto rx=0x7f51e400c730 tx=0x7f51e400cbf0 comp rx=0 tx=0).stop 2024-08-03T20:57:52.469 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:52.467+0000 7f51f2000640 1 -- 172.21.15.32:0/285212646 shutdown_connections 2024-08-03T20:57:52.469 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:52.467+0000 7f51f2000640 1 --2- 172.21.15.32:0/285212646 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f51c8077010 0x7f51c8079520 unknown :-1 s=CLOSED pgs=673 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:52.469 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:52.467+0000 7f51f2000640 1 --2- 172.21.15.32:0/285212646 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f51ec0b4ce0 0x7f51ec0af140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:52.469 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:52.467+0000 7f51f2000640 1 --2- 172.21.15.32:0/285212646 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f51ec1f2d30 0x7f51ec1548a0 unknown :-1 s=CLOSED pgs=1648 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:52.469 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:52.467+0000 7f51f2000640 1 --2- 172.21.15.32:0/285212646 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f51ec0b38b0 0x7f51ec152a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:52.469 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:52.467+0000 7f51f2000640 1 -- 172.21.15.32:0/285212646 >> 172.21.15.32:0/285212646 conn(0x7f51ec0b1bd0 msgr2=0x7f51ec151c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:57:52.469 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:52.467+0000 7f51f2000640 1 -- 172.21.15.32:0/285212646 shutdown_connections 2024-08-03T20:57:52.469 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:52.467+0000 7f51f2000640 1 -- 172.21.15.32:0/285212646 wait complete. 2024-08-03T20:57:52.495 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mds repaired 0 2024-08-03T20:57:52.725 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:52.723+0000 7faa89c00640 1 -- 172.21.15.32:0/919689069 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faa84073e60 msgr2=0x7faa84074270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:52.725 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:52.723+0000 7faa89c00640 1 --2- 172.21.15.32:0/919689069 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faa84073e60 0x7faa84074270 secure :-1 s=READY pgs=1649 cs=0 l=1 rev1=1 crypto rx=0x7faa84071700 tx=0x7faa7002a560 comp rx=0 tx=0).stop 2024-08-03T20:57:52.726 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:52.724+0000 7faa89c00640 1 -- 172.21.15.32:0/919689069 shutdown_connections 2024-08-03T20:57:52.726 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:52.724+0000 7faa89c00640 1 --2- 172.21.15.32:0/919689069 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7faa840747b0 0x7faa840cc8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:52.726 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:52.724+0000 7faa89c00640 1 --2- 172.21.15.32:0/919689069 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faa84073e60 0x7faa84074270 unknown :-1 s=CLOSED pgs=1649 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:52.726 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:52.724+0000 7faa89c00640 1 --2- 172.21.15.32:0/919689069 >> v2:172.21.15.118:3300/0 conn(0x7faa8406e930 0x7faa84144f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:52.726 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:52.724+0000 7faa89c00640 1 -- 172.21.15.32:0/919689069 >> 172.21.15.32:0/919689069 conn(0x7faa840b1bd0 msgr2=0x7faa840b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:57:52.726 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:52.724+0000 7faa89c00640 1 -- 172.21.15.32:0/919689069 shutdown_connections 2024-08-03T20:57:52.726 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:52.724+0000 7faa89c00640 1 -- 172.21.15.32:0/919689069 wait complete. 2024-08-03T20:57:52.727 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:52.726+0000 7faa89c00640 1 Processor -- start 2024-08-03T20:57:52.727 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:52.726+0000 7faa89c00640 1 -- start start 2024-08-03T20:57:52.729 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:52.727+0000 7faa89c00640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7faa8406e930 0x7faa841f7140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:52.729 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:52.727+0000 7faa89c00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7faa84073e60 0x7faa84143b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:52.729 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:52.727+0000 7faa89c00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faa840747b0 0x7faa8413ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:52.730 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:52.727+0000 7faa89c00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7faa840cf9c0 con 0x7faa840747b0 2024-08-03T20:57:52.730 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:52.727+0000 7faa89c00640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7faa840cf840 con 0x7faa84073e60 2024-08-03T20:57:52.730 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:52.727+0000 7faa89c00640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7faa840cce20 con 0x7faa8406e930 2024-08-03T20:57:52.730 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:52.728+0000 7faa83400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faa840747b0 0x7faa8413ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:52.730 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:52.728+0000 7faa83400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faa840747b0 0x7faa8413ee40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:51266/0 (socket says 172.21.15.32:51266) 2024-08-03T20:57:52.730 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:52.728+0000 7faa83400640 1 -- 172.21.15.32:0/25249881 learned_addr learned my addr 172.21.15.32:0/25249881 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:57:52.730 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:52.728+0000 7faa82a00640 1 --2- 172.21.15.32:0/25249881 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7faa8406e930 0x7faa841f7140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:52.730 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:52.728+0000 7faa82000640 1 --2- 172.21.15.32:0/25249881 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7faa84073e60 0x7faa84143b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:52.731 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:52.728+0000 7faa83400640 1 -- 172.21.15.32:0/25249881 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7faa8406e930 msgr2=0x7faa841f7140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:52.731 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:52.728+0000 7faa83400640 1 --2- 172.21.15.32:0/25249881 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7faa8406e930 0x7faa841f7140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:52.731 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:52.728+0000 7faa83400640 1 -- 172.21.15.32:0/25249881 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7faa84073e60 msgr2=0x7faa84143b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:52.731 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:52.728+0000 7faa83400640 1 --2- 172.21.15.32:0/25249881 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7faa84073e60 0x7faa84143b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:52.731 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:52.728+0000 7faa83400640 1 -- 172.21.15.32:0/25249881 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faa8412c490 con 0x7faa840747b0 2024-08-03T20:57:52.732 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:52.729+0000 7faa83400640 1 --2- 172.21.15.32:0/25249881 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faa840747b0 0x7faa8413ee40 secure :-1 s=READY pgs=1650 cs=0 l=1 rev1=1 crypto rx=0x7faa7c00e960 tx=0x7faa7c00ee20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:57:52.732 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:52.729+0000 7faa7aa00640 1 -- 172.21.15.32:0/25249881 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa7c004d20 con 0x7faa840747b0 2024-08-03T20:57:52.732 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:52.729+0000 7faa7aa00640 1 -- 172.21.15.32:0/25249881 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7faa7c00cb20 con 0x7faa840747b0 2024-08-03T20:57:52.732 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:52.729+0000 7faa7aa00640 1 -- 172.21.15.32:0/25249881 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa7c005020 con 0x7faa840747b0 2024-08-03T20:57:52.732 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:52.729+0000 7faa89c00640 1 -- 172.21.15.32:0/25249881 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faa8412e870 con 0x7faa840747b0 2024-08-03T20:57:52.732 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:52.729+0000 7faa89c00640 1 -- 172.21.15.32:0/25249881 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faa8412ee30 con 0x7faa840747b0 2024-08-03T20:57:52.732 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:52.730+0000 7faa89c00640 1 -- 172.21.15.32:0/25249881 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faa8415fa20 con 0x7faa840747b0 2024-08-03T20:57:52.733 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:52.731+0000 7faa7aa00640 1 -- 172.21.15.32:0/25249881 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7faa7c0051c0 con 0x7faa840747b0 2024-08-03T20:57:52.734 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:52.732+0000 7faa7aa00640 1 --2- 172.21.15.32:0/25249881 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7faa64077010 0x7faa64079520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:52.734 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:52.733+0000 7faa7aa00640 1 -- 172.21.15.32:0/25249881 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(109..109 src has 1..109) ==== 7918+0+0 (secure 0 0 0) 0x7faa7c099a10 con 0x7faa840747b0 2024-08-03T20:57:52.734 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:52.733+0000 7faa82a00640 1 --2- 172.21.15.32:0/25249881 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7faa64077010 0x7faa64079520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:52.735 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:52.734+0000 7faa82a00640 1 --2- 172.21.15.32:0/25249881 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7faa64077010 0x7faa64079520 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7faa7400bb50 tx=0x7faa7400b2e0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:57:52.738 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:52.737+0000 7faa7aa00640 1 -- 172.21.15.32:0/25249881 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7faa8415fa20 con 0x7faa840747b0 2024-08-03T20:57:53.029 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:53.027+0000 7faa89c00640 1 -- 172.21.15.32:0/25249881 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "mds repaired", "role": "0"} v 0) -- 0x7faa84074270 con 0x7faa840747b0 2024-08-03T20:57:53.473 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:53.471+0000 7faa7aa00640 1 -- 172.21.15.32:0/25249881 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "mds repaired", "role": "0"}]=0 repaired: restoring rank 6:0 v232) ==== 101+0+0 (secure 0 0 0) 0x7faa7c016030 con 0x7faa840747b0 2024-08-03T20:57:53.473 INFO:teuthology.orchestra.run.smithi032.stderr:repaired: restoring rank 6:0 2024-08-03T20:57:53.476 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:53.474+0000 7faa89c00640 1 -- 172.21.15.32:0/25249881 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7faa64077010 msgr2=0x7faa64079520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:53.476 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:53.474+0000 7faa89c00640 1 --2- 172.21.15.32:0/25249881 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7faa64077010 0x7faa64079520 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7faa7400bb50 tx=0x7faa7400b2e0 comp rx=0 tx=0).stop 2024-08-03T20:57:53.476 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:53.474+0000 7faa89c00640 1 -- 172.21.15.32:0/25249881 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faa840747b0 msgr2=0x7faa8413ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:53.476 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:53.474+0000 7faa89c00640 1 --2- 172.21.15.32:0/25249881 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faa840747b0 0x7faa8413ee40 secure :-1 s=READY pgs=1650 cs=0 l=1 rev1=1 crypto rx=0x7faa7c00e960 tx=0x7faa7c00ee20 comp rx=0 tx=0).stop 2024-08-03T20:57:53.476 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:53.474+0000 7faa89c00640 1 -- 172.21.15.32:0/25249881 shutdown_connections 2024-08-03T20:57:53.476 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:53.474+0000 7faa89c00640 1 --2- 172.21.15.32:0/25249881 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7faa64077010 0x7faa64079520 unknown :-1 s=CLOSED pgs=674 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:53.476 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:53.474+0000 7faa89c00640 1 --2- 172.21.15.32:0/25249881 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7faa84073e60 0x7faa84143b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:53.476 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:53.474+0000 7faa89c00640 1 --2- 172.21.15.32:0/25249881 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faa840747b0 0x7faa8413ee40 unknown :-1 s=CLOSED pgs=1650 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:53.476 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:53.474+0000 7faa89c00640 1 --2- 172.21.15.32:0/25249881 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7faa8406e930 0x7faa841f7140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:53.476 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:53.474+0000 7faa89c00640 1 -- 172.21.15.32:0/25249881 >> 172.21.15.32:0/25249881 conn(0x7faa840b1bd0 msgr2=0x7faa8406f930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:57:53.476 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:53.474+0000 7faa89c00640 1 -- 172.21.15.32:0/25249881 shutdown_connections 2024-08-03T20:57:53.477 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:53.474+0000 7faa89c00640 1 -- 172.21.15.32:0/25249881 wait complete. 2024-08-03T20:57:53.503 DEBUG:teuthology.orchestra.run.smithi032:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata import - 2024-08-03T20:57:53.647 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:53.645+0000 7f6b9371cf40 1 Processor -- start 2024-08-03T20:57:53.647 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:53.645+0000 7f6b9371cf40 1 -- start start 2024-08-03T20:57:53.649 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:53.646+0000 7f6b9371cf40 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55e50201f360 0x55e501f68750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:53.649 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:53.646+0000 7f6b9371cf40 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55e502020ea0 0x55e502037da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:53.649 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:53.646+0000 7f6b9371cf40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55e502036330 0x55e501f679b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:53.649 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:53.646+0000 7f6b9371cf40 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55e501eb6e30 con 0x55e50201f360 2024-08-03T20:57:53.649 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:53.646+0000 7f6b9371cf40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x55e501f14bf0 con 0x55e502036330 2024-08-03T20:57:53.650 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:53.646+0000 7f6b9371cf40 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x55e502020440 con 0x55e502020ea0 2024-08-03T20:57:53.650 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:53.647+0000 7f6b8be00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55e50201f360 0x55e501f68750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:53.650 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:53.647+0000 7f6b8be00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55e50201f360 0x55e501f68750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:51280/0 (socket says 172.21.15.32:51280) 2024-08-03T20:57:53.650 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:53.647+0000 7f6b8be00640 1 -- 172.21.15.32:0/398268542 learned_addr learned my addr 172.21.15.32:0/398268542 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:57:53.650 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:53.647+0000 7f6b8c800640 1 --2- 172.21.15.32:0/398268542 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55e502036330 0x55e501f679b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:53.650 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:53.647+0000 7f6b8b400640 1 --2- 172.21.15.32:0/398268542 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55e502020ea0 0x55e502037da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:53.650 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:53.647+0000 7f6b8be00640 1 -- 172.21.15.32:0/398268542 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55e502020ea0 msgr2=0x55e502037da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:57:53.650 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:53.647+0000 7f6b8be00640 1 --2- 172.21.15.32:0/398268542 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55e502020ea0 0x55e502037da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:53.650 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:53.647+0000 7f6b8be00640 1 -- 172.21.15.32:0/398268542 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55e502036330 msgr2=0x55e501f679b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:57:53.650 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:53.647+0000 7f6b8be00640 1 --2- 172.21.15.32:0/398268542 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55e502036330 0x55e501f679b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:53.650 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:53.647+0000 7f6b8be00640 1 -- 172.21.15.32:0/398268542 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e502037980 con 0x55e50201f360 2024-08-03T20:57:53.650 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:53.648+0000 7f6b8be00640 1 --2- 172.21.15.32:0/398268542 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55e50201f360 0x55e501f68750 secure :-1 s=READY pgs=1652 cs=0 l=1 rev1=1 crypto rx=0x7f6b7c001e10 tx=0x7f6b7c00b730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=49bdf19303875fbb server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:57:53.650 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:53.648+0000 7f6b8a000640 1 -- 172.21.15.32:0/398268542 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b7c00e830 con 0x55e50201f360 2024-08-03T20:57:53.651 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:53.648+0000 7f6b8a000640 1 -- 172.21.15.32:0/398268542 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f6b7c006050 con 0x55e50201f360 2024-08-03T20:57:53.651 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:53.648+0000 7f6b8a000640 1 -- 172.21.15.32:0/398268542 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b7c00ed60 con 0x55e50201f360 2024-08-03T20:57:53.652 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:53.648+0000 7f6b9371cf40 1 -- 172.21.15.32:0/398268542 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55e50201f360 msgr2=0x55e501f68750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:53.652 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:53.648+0000 7f6b9371cf40 1 --2- 172.21.15.32:0/398268542 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55e50201f360 0x55e501f68750 secure :-1 s=READY pgs=1652 cs=0 l=1 rev1=1 crypto rx=0x7f6b7c001e10 tx=0x7f6b7c00b730 comp rx=0 tx=0).stop 2024-08-03T20:57:53.652 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:53.648+0000 7f6b9371cf40 1 -- 172.21.15.32:0/398268542 shutdown_connections 2024-08-03T20:57:53.652 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:53.648+0000 7f6b9371cf40 1 --2- 172.21.15.32:0/398268542 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55e502036330 0x55e501f679b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:53.652 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:53.648+0000 7f6b9371cf40 1 --2- 172.21.15.32:0/398268542 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55e502020ea0 0x55e502037da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:53.652 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:53.648+0000 7f6b9371cf40 1 --2- 172.21.15.32:0/398268542 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55e50201f360 0x55e501f68750 unknown :-1 s=CLOSED pgs=1652 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:53.652 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:53.648+0000 7f6b9371cf40 1 -- 172.21.15.32:0/398268542 >> 172.21.15.32:0/398268542 conn(0x55e501e88ac0 msgr2=0x55e502038e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:57:53.652 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:53.649+0000 7f6b9371cf40 1 -- 172.21.15.32:0/398268542 shutdown_connections 2024-08-03T20:57:53.652 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:53.649+0000 7f6b9371cf40 1 -- 172.21.15.32:0/398268542 wait complete. 2024-08-03T20:57:53.652 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:53.650+0000 7f6b9371cf40 1 Processor -- start 2024-08-03T20:57:53.653 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:53.650+0000 7f6b9371cf40 1 -- start start 2024-08-03T20:57:53.654 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:53.651+0000 7f6b9371cf40 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55e50201f360 0x55e501f68750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:53.654 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:53.651+0000 7f6b9371cf40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55e502020ea0 0x55e5020cdca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:53.654 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:53.651+0000 7f6b9371cf40 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55e502036330 0x55e5020ce1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:53.654 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:53.651+0000 7f6b9371cf40 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55e501fb1fb0 con 0x55e502036330 2024-08-03T20:57:53.655 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:53.651+0000 7f6b9371cf40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x55e502037280 con 0x55e502020ea0 2024-08-03T20:57:53.655 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:53.651+0000 7f6b9371cf40 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x55e501f67f10 con 0x55e50201f360 2024-08-03T20:57:53.655 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:53.651+0000 7f6b8c800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55e502036330 0x55e5020ce1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:53.655 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:53.651+0000 7f6b8c800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55e502036330 0x55e5020ce1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:51296/0 (socket says 172.21.15.32:51296) 2024-08-03T20:57:53.655 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:53.651+0000 7f6b8c800640 1 -- 172.21.15.32:0/3144827011 learned_addr learned my addr 172.21.15.32:0/3144827011 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:57:53.655 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:53.652+0000 7f6b8be00640 1 --2- 172.21.15.32:0/3144827011 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55e50201f360 0x55e501f68750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:53.655 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:53.652+0000 7f6b8b400640 1 --2- 172.21.15.32:0/3144827011 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55e502020ea0 0x55e5020cdca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:53.655 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:53.652+0000 7f6b8c800640 1 -- 172.21.15.32:0/3144827011 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55e50201f360 msgr2=0x55e501f68750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:53.655 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:53.652+0000 7f6b8c800640 1 --2- 172.21.15.32:0/3144827011 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55e50201f360 0x55e501f68750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:53.655 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:53.652+0000 7f6b8c800640 1 -- 172.21.15.32:0/3144827011 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55e502020ea0 msgr2=0x55e5020cdca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:53.655 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:53.652+0000 7f6b8c800640 1 --2- 172.21.15.32:0/3144827011 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55e502020ea0 0x55e5020cdca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:53.655 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:53.652+0000 7f6b8c800640 1 -- 172.21.15.32:0/3144827011 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e501f68c90 con 0x55e502036330 2024-08-03T20:57:53.655 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:53.652+0000 7f6b8b400640 1 --2- 172.21.15.32:0/3144827011 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55e502020ea0 0x55e5020cdca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-03T20:57:53.656 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:53.652+0000 7f6b8c800640 1 --2- 172.21.15.32:0/3144827011 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55e502036330 0x55e5020ce1e0 secure :-1 s=READY pgs=1653 cs=0 l=1 rev1=1 crypto rx=0x7f6b840463f0 tx=0x7f6b840044c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:57:53.656 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:53.652+0000 7f6b88c00640 1 -- 172.21.15.32:0/3144827011 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b84064030 con 0x55e502036330 2024-08-03T20:57:53.656 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:53.653+0000 7f6b88c00640 1 -- 172.21.15.32:0/3144827011 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f6b8404d640 con 0x55e502036330 2024-08-03T20:57:53.656 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:53.653+0000 7f6b88c00640 1 -- 172.21.15.32:0/3144827011 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b84060440 con 0x55e502036330 2024-08-03T20:57:53.656 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:53.653+0000 7f6b9371cf40 1 -- 172.21.15.32:0/3144827011 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x55e5020d56b0 con 0x55e502036330 2024-08-03T20:57:53.656 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:53.653+0000 7f6b9371cf40 1 -- 172.21.15.32:0/3144827011 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e5020d5c70 con 0x55e502036330 2024-08-03T20:57:53.656 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:53.655+0000 7f6b88c00640 1 -- 172.21.15.32:0/3144827011 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f6b84067030 con 0x55e502036330 2024-08-03T20:57:53.657 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:53.656+0000 7f6b88c00640 1 --2- 172.21.15.32:0/3144827011 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f6b6c0759f0 0x7f6b6c077f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:53.657 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:53.656+0000 7f6b88c00640 1 -- 172.21.15.32:0/3144827011 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(109..109 src has 1..109) ==== 7918+0+0 (secure 0 0 0) 0x7f6b840dc1a0 con 0x55e502036330 2024-08-03T20:57:53.658 INFO:teuthology.orchestra.run.smithi032.stdout:Importing pool 2024-08-03T20:57:53.658 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:53.656+0000 7f6b9371cf40 1 --2- 172.21.15.32:0/3144827011 >> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] conn(0x55e501fb02d0 0x55e5020e1850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:53.658 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:53.656+0000 7f6b9371cf40 1 -- 172.21.15.32:0/3144827011 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:1 12.10 12:09401a81:::601.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109) -- 0x55e501fb06e0 con 0x55e501fb02d0 2024-08-03T20:57:53.658 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:53.656+0000 7f6b8be00640 1 --2- 172.21.15.32:0/3144827011 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f6b6c0759f0 0x7f6b6c077f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:53.658 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:53.657+0000 7f6b8b400640 1 --2- 172.21.15.32:0/3144827011 >> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] conn(0x55e501fb02d0 0x55e5020e1850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:53.659 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:53.657+0000 7f6b8b400640 1 --2- 172.21.15.32:0/3144827011 >> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] conn(0x55e501fb02d0 0x55e5020e1850 crc :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:57:53.659 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:53.657+0000 7f6b8be00640 1 --2- 172.21.15.32:0/3144827011 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f6b6c0759f0 0x7f6b6c077f00 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7f6b7c0011f0 tx=0x7f6b7c018040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:57:53.661 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:53.660+0000 7f6b8b400640 1 -- 172.21.15.32:0/3144827011 <== osd.0 v2:172.21.15.32:6816/3066156116 1 ==== osd_op_reply(1 601.00000000 [create] v109'722 uv696 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f6b7800d720 con 0x55e501fb02d0 2024-08-03T20:57:53.661 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:53.660+0000 7f6b9371cf40 1 -- 172.21.15.32:0/3144827011 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:2 12.10 12:09401a81:::601.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e109) -- 0x55e5020e1d90 con 0x55e501fb02d0 2024-08-03T20:57:53.663 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:53.661+0000 7f6b8b400640 1 -- 172.21.15.32:0/3144827011 <== osd.0 v2:172.21.15.32:6816/3066156116 2 ==== osd_op_reply(2 601.00000000 [delete] v109'723 uv723 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f6b7800d720 con 0x55e501fb02d0 2024-08-03T20:57:53.663 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:53.661+0000 7f6b9371cf40 1 -- 172.21.15.32:0/3144827011 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:3 12.10 12:09401a81:::601.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109) -- 0x55e501f67920 con 0x55e501fb02d0 2024-08-03T20:57:53.664 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:53.663+0000 7f6b8b400640 1 -- 172.21.15.32:0/3144827011 <== osd.0 v2:172.21.15.32:6816/3066156116 3 ==== osd_op_reply(3 601.00000000 [create] v109'724 uv724 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f6b7800d720 con 0x55e501fb02d0 2024-08-03T20:57:53.664 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::601.00000000:head# 2024-08-03T20:57:53.665 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:53.663+0000 7f6b9371cf40 1 -- 172.21.15.32:0/3144827011 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:4 12.10 12:09401a81:::601.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109) -- 0x55e5020e3140 con 0x55e501fb02d0 2024-08-03T20:57:53.666 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:53.665+0000 7f6b8b400640 1 -- 172.21.15.32:0/3144827011 <== osd.0 v2:172.21.15.32:6816/3066156116 4 ==== osd_op_reply(4 601.00000000 [setxattr (30)] v109'725 uv725 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f6b7800d720 con 0x55e501fb02d0 2024-08-03T20:57:53.666 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:53.665+0000 7f6b9371cf40 1 -- 172.21.15.32:0/3144827011 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:5 12.10 12:09401a81:::601.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109) -- 0x55e5020e34e0 con 0x55e501fb02d0 2024-08-03T20:57:53.668 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:53.666+0000 7f6b8b400640 1 -- 172.21.15.32:0/3144827011 <== osd.0 v2:172.21.15.32:6816/3066156116 5 ==== osd_op_reply(5 601.00000000 [setxattr (62)] v109'726 uv726 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f6b7800d720 con 0x55e501fb02d0 2024-08-03T20:57:53.668 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:53.666+0000 7f6b9371cf40 1 -- 172.21.15.32:0/3144827011 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:6 12.10 12:09401a81:::601.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109) -- 0x55e5020e2130 con 0x55e501fb02d0 2024-08-03T20:57:53.669 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:53.668+0000 7f6b8b400640 1 -- 172.21.15.32:0/3144827011 <== osd.0 v2:172.21.15.32:6816/3066156116 6 ==== osd_op_reply(6 601.00000000 [omap-set-header] v109'727 uv727 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f6b7800d720 con 0x55e501fb02d0 2024-08-03T20:57:53.669 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:53.668+0000 7f6b9371cf40 1 -- 172.21.15.32:0/3144827011 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:7 12.10 12:0bd6d154:::602.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109) -- 0x55e5020e24d0 con 0x55e501fb02d0 2024-08-03T20:57:53.671 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:53.670+0000 7f6b8b400640 1 -- 172.21.15.32:0/3144827011 <== osd.0 v2:172.21.15.32:6816/3066156116 7 ==== osd_op_reply(7 602.00000000 [create] v109'728 uv702 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f6b7800d720 con 0x55e501fb02d0 2024-08-03T20:57:53.671 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:53.670+0000 7f6b9371cf40 1 -- 172.21.15.32:0/3144827011 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:8 12.10 12:0bd6d154:::602.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e109) -- 0x55e5020e24d0 con 0x55e501fb02d0 2024-08-03T20:57:53.673 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:53.671+0000 7f6b8b400640 1 -- 172.21.15.32:0/3144827011 <== osd.0 v2:172.21.15.32:6816/3066156116 8 ==== osd_op_reply(8 602.00000000 [delete] v109'729 uv729 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f6b7800d720 con 0x55e501fb02d0 2024-08-03T20:57:53.673 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:53.671+0000 7f6b9371cf40 1 -- 172.21.15.32:0/3144827011 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:9 12.10 12:0bd6d154:::602.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109) -- 0x55e5020e24d0 con 0x55e501fb02d0 2024-08-03T20:57:53.674 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:53.673+0000 7f6b8b400640 1 -- 172.21.15.32:0/3144827011 <== osd.0 v2:172.21.15.32:6816/3066156116 9 ==== osd_op_reply(9 602.00000000 [create] v109'730 uv730 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f6b7800d720 con 0x55e501fb02d0 2024-08-03T20:57:53.674 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::602.00000000:head# 2024-08-03T20:57:53.675 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:53.673+0000 7f6b9371cf40 1 -- 172.21.15.32:0/3144827011 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:10 12.10 12:0bd6d154:::602.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109) -- 0x55e5020e24d0 con 0x55e501fb02d0 2024-08-03T20:57:53.676 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:53.674+0000 7f6b8b400640 1 -- 172.21.15.32:0/3144827011 <== osd.0 v2:172.21.15.32:6816/3066156116 10 ==== osd_op_reply(10 602.00000000 [setxattr (30)] v109'731 uv731 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f6b7800d720 con 0x55e501fb02d0 2024-08-03T20:57:53.676 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:53.674+0000 7f6b9371cf40 1 -- 172.21.15.32:0/3144827011 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:11 12.10 12:0bd6d154:::602.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109) -- 0x55e5020e24d0 con 0x55e501fb02d0 2024-08-03T20:57:53.677 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:53.676+0000 7f6b8b400640 1 -- 172.21.15.32:0/3144827011 <== osd.0 v2:172.21.15.32:6816/3066156116 11 ==== osd_op_reply(11 602.00000000 [setxattr (62)] v109'732 uv732 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f6b7800d720 con 0x55e501fb02d0 2024-08-03T20:57:53.678 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:53.676+0000 7f6b9371cf40 1 -- 172.21.15.32:0/3144827011 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:12 12.10 12:0bd6d154:::602.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109) -- 0x55e5020e24d0 con 0x55e501fb02d0 2024-08-03T20:57:53.679 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:53.678+0000 7f6b8b400640 1 -- 172.21.15.32:0/3144827011 <== osd.0 v2:172.21.15.32:6816/3066156116 12 ==== osd_op_reply(12 602.00000000 [omap-set-header] v109'733 uv733 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f6b7800d720 con 0x55e501fb02d0 2024-08-03T20:57:53.679 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:53.678+0000 7f6b9371cf40 1 -- 172.21.15.32:0/3144827011 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:13 12.10 12:0cf6ad0f:::10000000000.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109) -- 0x55e5020e24d0 con 0x55e501fb02d0 2024-08-03T20:57:53.681 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:53.680+0000 7f6b8b400640 1 -- 172.21.15.32:0/3144827011 <== osd.0 v2:172.21.15.32:6816/3066156116 13 ==== osd_op_reply(13 10000000000.00000000 [create] v109'734 uv709 ondisk = -17 ((17) File exists)) ==== 164+0+0 (crc 0 0 0) 0x7f6b7800d720 con 0x55e501fb02d0 2024-08-03T20:57:53.681 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:53.680+0000 7f6b9371cf40 1 -- 172.21.15.32:0/3144827011 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:14 12.10 12:0cf6ad0f:::10000000000.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e109) -- 0x55e5020e24d0 con 0x55e501fb02d0 2024-08-03T20:57:53.683 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:53.681+0000 7f6b8b400640 1 -- 172.21.15.32:0/3144827011 <== osd.0 v2:172.21.15.32:6816/3066156116 14 ==== osd_op_reply(14 10000000000.00000000 [delete] v109'735 uv735 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f6b7800d720 con 0x55e501fb02d0 2024-08-03T20:57:53.683 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:53.682+0000 7f6b9371cf40 1 -- 172.21.15.32:0/3144827011 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:15 12.10 12:0cf6ad0f:::10000000000.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109) -- 0x55e5020e24d0 con 0x55e501fb02d0 2024-08-03T20:57:53.684 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:53.683+0000 7f6b8b400640 1 -- 172.21.15.32:0/3144827011 <== osd.0 v2:172.21.15.32:6816/3066156116 15 ==== osd_op_reply(15 10000000000.00000000 [create] v109'736 uv736 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f6b7800d720 con 0x55e501fb02d0 2024-08-03T20:57:53.685 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::10000000000.00000000:head# 2024-08-03T20:57:53.685 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:53.683+0000 7f6b9371cf40 1 -- 172.21.15.32:0/3144827011 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:16 12.10 12:0cf6ad0f:::10000000000.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109) -- 0x55e5020e24d0 con 0x55e501fb02d0 2024-08-03T20:57:53.686 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:53.685+0000 7f6b8b400640 1 -- 172.21.15.32:0/3144827011 <== osd.0 v2:172.21.15.32:6816/3066156116 16 ==== osd_op_reply(16 10000000000.00000000 [setxattr (30)] v109'737 uv737 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f6b7800d720 con 0x55e501fb02d0 2024-08-03T20:57:53.686 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:53.685+0000 7f6b9371cf40 1 -- 172.21.15.32:0/3144827011 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:17 12.10 12:0cf6ad0f:::10000000000.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109) -- 0x55e5020e24d0 con 0x55e501fb02d0 2024-08-03T20:57:53.687 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:53.686+0000 7f6b8b400640 1 -- 172.21.15.32:0/3144827011 <== osd.0 v2:172.21.15.32:6816/3066156116 17 ==== osd_op_reply(17 10000000000.00000000 [setxattr (62)] v109'738 uv738 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f6b7800d720 con 0x55e501fb02d0 2024-08-03T20:57:53.688 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:53.686+0000 7f6b9371cf40 1 -- 172.21.15.32:0/3144827011 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:18 12.10 12:0cf6ad0f:::10000000000.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109) -- 0x55e5020e24d0 con 0x55e501fb02d0 2024-08-03T20:57:53.689 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:53.688+0000 7f6b8b400640 1 -- 172.21.15.32:0/3144827011 <== osd.0 v2:172.21.15.32:6816/3066156116 18 ==== osd_op_reply(18 10000000000.00000000 [omap-set-header] v109'739 uv739 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f6b7800d720 con 0x55e501fb02d0 2024-08-03T20:57:53.689 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:53.688+0000 7f6b9371cf40 1 -- 172.21.15.32:0/3144827011 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:19 12.10 12:0cf6ad0f:::10000000000.00000000:head [omap-set-vals in=593b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109) -- 0x55e5020e24d0 con 0x55e501fb02d0 2024-08-03T20:57:53.691 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:53.689+0000 7f6b8b400640 1 -- 172.21.15.32:0/3144827011 <== osd.0 v2:172.21.15.32:6816/3066156116 19 ==== osd_op_reply(19 10000000000.00000000 [omap-set-vals] v109'740 uv740 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f6b7800d720 con 0x55e501fb02d0 2024-08-03T20:57:53.691 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:53.689+0000 7f6b9371cf40 1 -- 172.21.15.32:0/3144827011 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:20 12.10 12:0d82a743:::600.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109) -- 0x55e5020e24d0 con 0x55e501fb02d0 2024-08-03T20:57:53.692 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:53.691+0000 7f6b8b400640 1 -- 172.21.15.32:0/3144827011 <== osd.0 v2:172.21.15.32:6816/3066156116 20 ==== osd_op_reply(20 600.00000000 [create] v109'741 uv715 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f6b7800d720 con 0x55e501fb02d0 2024-08-03T20:57:53.692 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:53.691+0000 7f6b9371cf40 1 -- 172.21.15.32:0/3144827011 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:21 12.10 12:0d82a743:::600.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e109) -- 0x55e5020e24d0 con 0x55e501fb02d0 2024-08-03T20:57:53.694 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:53.693+0000 7f6b8b400640 1 -- 172.21.15.32:0/3144827011 <== osd.0 v2:172.21.15.32:6816/3066156116 21 ==== osd_op_reply(21 600.00000000 [delete] v109'742 uv742 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f6b7800d720 con 0x55e501fb02d0 2024-08-03T20:57:53.694 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:53.693+0000 7f6b9371cf40 1 -- 172.21.15.32:0/3144827011 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:22 12.10 12:0d82a743:::600.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109) -- 0x55e5020e24d0 con 0x55e501fb02d0 2024-08-03T20:57:53.696 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:53.694+0000 7f6b8b400640 1 -- 172.21.15.32:0/3144827011 <== osd.0 v2:172.21.15.32:6816/3066156116 22 ==== osd_op_reply(22 600.00000000 [create] v109'743 uv743 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f6b7800d720 con 0x55e501fb02d0 2024-08-03T20:57:53.696 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::600.00000000:head# 2024-08-03T20:57:53.696 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:53.694+0000 7f6b9371cf40 1 -- 172.21.15.32:0/3144827011 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:23 12.10 12:0d82a743:::600.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109) -- 0x55e5020e24d0 con 0x55e501fb02d0 2024-08-03T20:57:53.697 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:53.696+0000 7f6b8b400640 1 -- 172.21.15.32:0/3144827011 <== osd.0 v2:172.21.15.32:6816/3066156116 23 ==== osd_op_reply(23 600.00000000 [setxattr (30)] v109'744 uv744 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f6b7800d720 con 0x55e501fb02d0 2024-08-03T20:57:53.697 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:53.696+0000 7f6b9371cf40 1 -- 172.21.15.32:0/3144827011 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:24 12.10 12:0d82a743:::600.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109) -- 0x55e5020e24d0 con 0x55e501fb02d0 2024-08-03T20:57:53.699 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:53.697+0000 7f6b8b400640 1 -- 172.21.15.32:0/3144827011 <== osd.0 v2:172.21.15.32:6816/3066156116 24 ==== osd_op_reply(24 600.00000000 [setxattr (62)] v109'745 uv745 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f6b7800d720 con 0x55e501fb02d0 2024-08-03T20:57:53.699 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:53.697+0000 7f6b9371cf40 1 -- 172.21.15.32:0/3144827011 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:25 12.10 12:0d82a743:::600.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109) -- 0x55e5020e24d0 con 0x55e501fb02d0 2024-08-03T20:57:53.700 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:53.699+0000 7f6b8b400640 1 -- 172.21.15.32:0/3144827011 <== osd.0 v2:172.21.15.32:6816/3066156116 25 ==== osd_op_reply(25 600.00000000 [omap-set-header] v109'746 uv746 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f6b7800d720 con 0x55e501fb02d0 2024-08-03T20:57:53.701 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:53.699+0000 7f6b9371cf40 1 -- 172.21.15.32:0/3144827011 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:26 12.10 12:0d89b25e:::603.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109) -- 0x55e5020e24d0 con 0x55e501fb02d0 2024-08-03T20:57:53.702 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:53.701+0000 7f6b8b400640 1 -- 172.21.15.32:0/3144827011 <== osd.0 v2:172.21.15.32:6816/3066156116 26 ==== osd_op_reply(26 603.00000000 [create] v109'747 uv721 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f6b7800d720 con 0x55e501fb02d0 2024-08-03T20:57:53.702 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:53.701+0000 7f6b9371cf40 1 -- 172.21.15.32:0/3144827011 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:27 12.10 12:0d89b25e:::603.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e109) -- 0x55e5020e24d0 con 0x55e501fb02d0 2024-08-03T20:57:53.704 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:53.702+0000 7f6b8b400640 1 -- 172.21.15.32:0/3144827011 <== osd.0 v2:172.21.15.32:6816/3066156116 27 ==== osd_op_reply(27 603.00000000 [delete] v109'748 uv748 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f6b7800d720 con 0x55e501fb02d0 2024-08-03T20:57:53.704 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:53.702+0000 7f6b9371cf40 1 -- 172.21.15.32:0/3144827011 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:28 12.10 12:0d89b25e:::603.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109) -- 0x55e5020e24d0 con 0x55e501fb02d0 2024-08-03T20:57:53.705 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:53.704+0000 7f6b8b400640 1 -- 172.21.15.32:0/3144827011 <== osd.0 v2:172.21.15.32:6816/3066156116 28 ==== osd_op_reply(28 603.00000000 [create] v109'749 uv749 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f6b7800d720 con 0x55e501fb02d0 2024-08-03T20:57:53.706 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::603.00000000:head# 2024-08-03T20:57:53.706 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:53.704+0000 7f6b9371cf40 1 -- 172.21.15.32:0/3144827011 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:29 12.10 12:0d89b25e:::603.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109) -- 0x55e5020e24d0 con 0x55e501fb02d0 2024-08-03T20:57:53.707 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:53.706+0000 7f6b8b400640 1 -- 172.21.15.32:0/3144827011 <== osd.0 v2:172.21.15.32:6816/3066156116 29 ==== osd_op_reply(29 603.00000000 [setxattr (30)] v109'750 uv750 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f6b7800d720 con 0x55e501fb02d0 2024-08-03T20:57:53.707 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:53.706+0000 7f6b9371cf40 1 -- 172.21.15.32:0/3144827011 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:30 12.10 12:0d89b25e:::603.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109) -- 0x55e5020e24d0 con 0x55e501fb02d0 2024-08-03T20:57:53.708 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:53.707+0000 7f6b8b400640 1 -- 172.21.15.32:0/3144827011 <== osd.0 v2:172.21.15.32:6816/3066156116 30 ==== osd_op_reply(30 603.00000000 [setxattr (62)] v109'751 uv751 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f6b7800d720 con 0x55e501fb02d0 2024-08-03T20:57:53.709 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:53.707+0000 7f6b9371cf40 1 -- 172.21.15.32:0/3144827011 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:31 12.10 12:0d89b25e:::603.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109) -- 0x55e5020e24d0 con 0x55e501fb02d0 2024-08-03T20:57:53.710 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:53.709+0000 7f6b8b400640 1 -- 172.21.15.32:0/3144827011 <== osd.0 v2:172.21.15.32:6816/3066156116 31 ==== osd_op_reply(31 603.00000000 [omap-set-header] v109'752 uv752 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f6b7800d720 con 0x55e501fb02d0 2024-08-03T20:57:53.710 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:53.709+0000 7f6b9371cf40 1 --2- 172.21.15.32:0/3144827011 >> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] conn(0x55e5020e29e0 0x55e5020e5890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:53.710 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:53.709+0000 7f6b9371cf40 1 -- 172.21.15.32:0/3144827011 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:32 12.14 12:287034c4:::1.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109) -- 0x55e5020e5dd0 con 0x55e5020e29e0 2024-08-03T20:57:53.711 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:53.709+0000 7f6b8be00640 1 --2- 172.21.15.32:0/3144827011 >> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] conn(0x55e5020e29e0 0x55e5020e5890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:53.711 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:53.710+0000 7f6b8be00640 1 --2- 172.21.15.32:0/3144827011 >> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] conn(0x55e5020e29e0 0x55e5020e5890 crc :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:57:53.713 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:53.712+0000 7f6b8be00640 1 -- 172.21.15.32:0/3144827011 <== osd.5 v2:172.21.15.118:6810/2724891770 1 ==== osd_op_reply(32 1.00000000.inode [create] v109'527 uv526 ondisk = -17 ((17) File exists)) ==== 160+0+0 (crc 0 0 0) 0x7f6b7c014070 con 0x55e5020e29e0 2024-08-03T20:57:53.713 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:53.712+0000 7f6b9371cf40 1 -- 172.21.15.32:0/3144827011 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:33 12.14 12:287034c4:::1.00000000.inode:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e109) -- 0x55e5020e6170 con 0x55e5020e29e0 2024-08-03T20:57:53.715 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:53.714+0000 7f6b8be00640 1 -- 172.21.15.32:0/3144827011 <== osd.5 v2:172.21.15.118:6810/2724891770 2 ==== osd_op_reply(33 1.00000000.inode [delete] v109'528 uv528 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f6b7c014070 con 0x55e5020e29e0 2024-08-03T20:57:53.716 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:53.714+0000 7f6b9371cf40 1 -- 172.21.15.32:0/3144827011 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:34 12.14 12:287034c4:::1.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109) -- 0x55e5020e6510 con 0x55e5020e29e0 2024-08-03T20:57:53.717 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::1.00000000.inode:head# 2024-08-03T20:57:53.717 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:53.715+0000 7f6b8be00640 1 -- 172.21.15.32:0/3144827011 <== osd.5 v2:172.21.15.118:6810/2724891770 3 ==== osd_op_reply(34 1.00000000.inode [create] v109'529 uv529 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f6b7c014070 con 0x55e5020e29e0 2024-08-03T20:57:53.717 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:53.715+0000 7f6b9371cf40 1 -- 172.21.15.32:0/3144827011 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:35 12.14 12:287034c4:::1.00000000.inode:head [write 0~582 in=582b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109) -- 0x55e5020e68b0 con 0x55e5020e29e0 2024-08-03T20:57:53.718 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:53.717+0000 7f6b8be00640 1 -- 172.21.15.32:0/3144827011 <== osd.5 v2:172.21.15.118:6810/2724891770 4 ==== osd_op_reply(35 1.00000000.inode [write 0~582] v109'530 uv530 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f6b7c014070 con 0x55e5020e29e0 2024-08-03T20:57:53.718 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:53.717+0000 7f6b9371cf40 1 -- 172.21.15.32:0/3144827011 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:36 12.14 12:287034c4:::1.00000000.inode:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109) -- 0x55e5020e6c50 con 0x55e5020e29e0 2024-08-03T20:57:53.720 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:53.718+0000 7f6b8be00640 1 -- 172.21.15.32:0/3144827011 <== osd.5 v2:172.21.15.118:6810/2724891770 5 ==== osd_op_reply(36 1.00000000.inode [omap-set-header] v109'531 uv531 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f6b7c014070 con 0x55e5020e29e0 2024-08-03T20:57:53.720 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:53.718+0000 7f6b9371cf40 1 -- 172.21.15.32:0/3144827011 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:37 12.14 12:292cf221:::200.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109) -- 0x55e5020e6ff0 con 0x55e5020e29e0 2024-08-03T20:57:53.721 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:53.720+0000 7f6b8be00640 1 -- 172.21.15.32:0/3144827011 <== osd.5 v2:172.21.15.118:6810/2724891770 6 ==== osd_op_reply(37 200.00000000 [create] v109'532 uv514 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f6b7c014070 con 0x55e5020e29e0 2024-08-03T20:57:53.721 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:53.720+0000 7f6b9371cf40 1 -- 172.21.15.32:0/3144827011 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:38 12.14 12:292cf221:::200.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e109) -- 0x55e5020e7390 con 0x55e5020e29e0 2024-08-03T20:57:53.723 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:53.722+0000 7f6b8be00640 1 -- 172.21.15.32:0/3144827011 <== osd.5 v2:172.21.15.118:6810/2724891770 7 ==== osd_op_reply(38 200.00000000 [delete] v109'533 uv533 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f6b7c014070 con 0x55e5020e29e0 2024-08-03T20:57:53.723 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:53.722+0000 7f6b9371cf40 1 -- 172.21.15.32:0/3144827011 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:39 12.14 12:292cf221:::200.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109) -- 0x55e5020e7390 con 0x55e5020e29e0 2024-08-03T20:57:53.724 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:53.723+0000 7f6b8be00640 1 -- 172.21.15.32:0/3144827011 <== osd.5 v2:172.21.15.118:6810/2724891770 8 ==== osd_op_reply(39 200.00000000 [create] v109'534 uv534 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f6b7c014070 con 0x55e5020e29e0 2024-08-03T20:57:53.724 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::200.00000000:head# 2024-08-03T20:57:53.725 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:53.723+0000 7f6b9371cf40 1 -- 172.21.15.32:0/3144827011 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:40 12.14 12:292cf221:::200.00000000:head [write 0~90 in=90b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109) -- 0x55e5020e7390 con 0x55e5020e29e0 2024-08-03T20:57:53.726 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:53.724+0000 7f6b8be00640 1 -- 172.21.15.32:0/3144827011 <== osd.5 v2:172.21.15.118:6810/2724891770 9 ==== osd_op_reply(40 200.00000000 [write 0~90] v109'535 uv535 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f6b7c014070 con 0x55e5020e29e0 2024-08-03T20:57:53.726 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:53.724+0000 7f6b9371cf40 1 -- 172.21.15.32:0/3144827011 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:41 12.14 12:292cf221:::200.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109) -- 0x55e5020e7390 con 0x55e5020e29e0 2024-08-03T20:57:53.727 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:53.726+0000 7f6b8be00640 1 -- 172.21.15.32:0/3144827011 <== osd.5 v2:172.21.15.118:6810/2724891770 10 ==== osd_op_reply(41 200.00000000 [omap-set-header] v109'536 uv536 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f6b7c014070 con 0x55e5020e29e0 2024-08-03T20:57:53.727 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:53.726+0000 7f6b9371cf40 1 -- 172.21.15.32:0/3144827011 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:42 12.14 12:2e2fa760:::200.00000001:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109) -- 0x55e5020e7390 con 0x55e5020e29e0 2024-08-03T20:57:53.729 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:53.727+0000 7f6b8be00640 1 -- 172.21.15.32:0/3144827011 <== osd.5 v2:172.21.15.118:6810/2724891770 11 ==== osd_op_reply(42 200.00000001 [create] v109'537 uv519 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f6b7c014070 con 0x55e5020e29e0 2024-08-03T20:57:53.729 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:53.727+0000 7f6b9371cf40 1 -- 172.21.15.32:0/3144827011 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:43 12.14 12:2e2fa760:::200.00000001:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e109) -- 0x55e5020e7390 con 0x55e5020e29e0 2024-08-03T20:57:53.730 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:53.729+0000 7f6b8be00640 1 -- 172.21.15.32:0/3144827011 <== osd.5 v2:172.21.15.118:6810/2724891770 12 ==== osd_op_reply(43 200.00000001 [delete] v109'538 uv538 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f6b7c014070 con 0x55e5020e29e0 2024-08-03T20:57:53.730 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:53.729+0000 7f6b9371cf40 1 -- 172.21.15.32:0/3144827011 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:44 12.14 12:2e2fa760:::200.00000001:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109) -- 0x55e5020e7390 con 0x55e5020e29e0 2024-08-03T20:57:53.732 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::200.00000001:head# 2024-08-03T20:57:53.732 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:53.730+0000 7f6b8be00640 1 -- 172.21.15.32:0/3144827011 <== osd.5 v2:172.21.15.118:6810/2724891770 13 ==== osd_op_reply(44 200.00000001 [create] v109'539 uv539 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f6b7c014070 con 0x55e5020e29e0 2024-08-03T20:57:53.732 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:53.730+0000 7f6b9371cf40 1 -- 172.21.15.32:0/3144827011 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:45 12.14 12:2e2fa760:::200.00000001:head [write 0~18118 in=18118b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109) -- 0x55e5020e7390 con 0x55e5020e29e0 2024-08-03T20:57:53.734 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:53.732+0000 7f6b8be00640 1 -- 172.21.15.32:0/3144827011 <== osd.5 v2:172.21.15.118:6810/2724891770 14 ==== osd_op_reply(45 200.00000001 [write 0~18118] v109'540 uv540 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f6b7c014070 con 0x55e5020e29e0 2024-08-03T20:57:53.734 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:53.732+0000 7f6b9371cf40 1 -- 172.21.15.32:0/3144827011 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:46 12.14 12:2e2fa760:::200.00000001:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109) -- 0x55e5020e7390 con 0x55e5020e29e0 2024-08-03T20:57:53.735 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:53.734+0000 7f6b8be00640 1 -- 172.21.15.32:0/3144827011 <== osd.5 v2:172.21.15.118:6810/2724891770 15 ==== osd_op_reply(46 200.00000001 [omap-set-header] v109'541 uv541 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f6b7c014070 con 0x55e5020e29e0 2024-08-03T20:57:53.735 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:53.734+0000 7f6b9371cf40 1 -- 172.21.15.32:0/3144827011 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:47 12.14 12:2f55791f:::606.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109) -- 0x55e5020e7390 con 0x55e5020e29e0 2024-08-03T20:57:53.736 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:53.735+0000 7f6b8be00640 1 -- 172.21.15.32:0/3144827011 <== osd.5 v2:172.21.15.118:6810/2724891770 16 ==== osd_op_reply(47 606.00000000 [create] v109'542 uv525 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f6b7c014070 con 0x55e5020e29e0 2024-08-03T20:57:53.737 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:53.735+0000 7f6b9371cf40 1 -- 172.21.15.32:0/3144827011 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:48 12.14 12:2f55791f:::606.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e109) -- 0x55e5020e7390 con 0x55e5020e29e0 2024-08-03T20:57:53.738 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:53.736+0000 7f6b8be00640 1 -- 172.21.15.32:0/3144827011 <== osd.5 v2:172.21.15.118:6810/2724891770 17 ==== osd_op_reply(48 606.00000000 [delete] v109'543 uv543 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f6b7c014070 con 0x55e5020e29e0 2024-08-03T20:57:53.738 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:53.736+0000 7f6b9371cf40 1 -- 172.21.15.32:0/3144827011 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:49 12.14 12:2f55791f:::606.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109) -- 0x55e5020e7390 con 0x55e5020e29e0 2024-08-03T20:57:53.739 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::606.00000000:head# 2024-08-03T20:57:53.739 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:53.738+0000 7f6b8be00640 1 -- 172.21.15.32:0/3144827011 <== osd.5 v2:172.21.15.118:6810/2724891770 18 ==== osd_op_reply(49 606.00000000 [create] v109'544 uv544 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f6b7c014070 con 0x55e5020e29e0 2024-08-03T20:57:53.740 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:53.738+0000 7f6b9371cf40 1 -- 172.21.15.32:0/3144827011 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:50 12.14 12:2f55791f:::606.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109) -- 0x55e5020e7390 con 0x55e5020e29e0 2024-08-03T20:57:53.741 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:53.739+0000 7f6b8be00640 1 -- 172.21.15.32:0/3144827011 <== osd.5 v2:172.21.15.118:6810/2724891770 19 ==== osd_op_reply(50 606.00000000 [setxattr (30)] v109'545 uv545 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f6b7c014070 con 0x55e5020e29e0 2024-08-03T20:57:53.741 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:53.739+0000 7f6b9371cf40 1 -- 172.21.15.32:0/3144827011 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:51 12.14 12:2f55791f:::606.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109) -- 0x55e5020e7390 con 0x55e5020e29e0 2024-08-03T20:57:53.742 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:53.741+0000 7f6b8be00640 1 -- 172.21.15.32:0/3144827011 <== osd.5 v2:172.21.15.118:6810/2724891770 20 ==== osd_op_reply(51 606.00000000 [setxattr (62)] v109'546 uv546 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f6b7c014070 con 0x55e5020e29e0 2024-08-03T20:57:53.742 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:53.741+0000 7f6b9371cf40 1 -- 172.21.15.32:0/3144827011 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:52 12.14 12:2f55791f:::606.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109) -- 0x55e5020e7390 con 0x55e5020e29e0 2024-08-03T20:57:53.744 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:53.742+0000 7f6b8be00640 1 -- 172.21.15.32:0/3144827011 <== osd.5 v2:172.21.15.118:6810/2724891770 21 ==== osd_op_reply(52 606.00000000 [omap-set-header] v109'547 uv547 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f6b7c014070 con 0x55e5020e29e0 2024-08-03T20:57:53.744 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:53.742+0000 7f6b9371cf40 1 -- 172.21.15.32:0/3144827011 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:53 12.1c 12:3ed09add:::607.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109) -- 0x55e5020e7390 con 0x55e501fb02d0 2024-08-03T20:57:53.745 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:53.744+0000 7f6b8b400640 1 -- 172.21.15.32:0/3144827011 <== osd.0 v2:172.21.15.32:6816/3066156116 32 ==== osd_op_reply(53 607.00000000 [create] v109'141 uv140 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f6b7800d720 con 0x55e501fb02d0 2024-08-03T20:57:53.745 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:53.744+0000 7f6b9371cf40 1 -- 172.21.15.32:0/3144827011 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:54 12.1c 12:3ed09add:::607.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e109) -- 0x55e5020e7390 con 0x55e501fb02d0 2024-08-03T20:57:53.746 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:53.745+0000 7f6b8b400640 1 -- 172.21.15.32:0/3144827011 <== osd.0 v2:172.21.15.32:6816/3066156116 33 ==== osd_op_reply(54 607.00000000 [delete] v109'142 uv142 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f6b7800d720 con 0x55e501fb02d0 2024-08-03T20:57:53.747 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:53.745+0000 7f6b9371cf40 1 -- 172.21.15.32:0/3144827011 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:55 12.1c 12:3ed09add:::607.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109) -- 0x55e5020e7390 con 0x55e501fb02d0 2024-08-03T20:57:53.748 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::607.00000000:head# 2024-08-03T20:57:53.748 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:53.747+0000 7f6b8b400640 1 -- 172.21.15.32:0/3144827011 <== osd.0 v2:172.21.15.32:6816/3066156116 34 ==== osd_op_reply(55 607.00000000 [create] v109'143 uv143 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f6b7800d720 con 0x55e501fb02d0 2024-08-03T20:57:53.748 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:53.747+0000 7f6b9371cf40 1 -- 172.21.15.32:0/3144827011 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:56 12.1c 12:3ed09add:::607.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109) -- 0x55e5020e83a0 con 0x55e501fb02d0 2024-08-03T20:57:53.749 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:53.748+0000 7f6b8b400640 1 -- 172.21.15.32:0/3144827011 <== osd.0 v2:172.21.15.32:6816/3066156116 35 ==== osd_op_reply(56 607.00000000 [setxattr (30)] v109'144 uv144 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f6b7800d720 con 0x55e501fb02d0 2024-08-03T20:57:53.749 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:53.748+0000 7f6b9371cf40 1 -- 172.21.15.32:0/3144827011 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:57 12.1c 12:3ed09add:::607.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109) -- 0x55e5020e83a0 con 0x55e501fb02d0 2024-08-03T20:57:53.751 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:53.749+0000 7f6b8b400640 1 -- 172.21.15.32:0/3144827011 <== osd.0 v2:172.21.15.32:6816/3066156116 36 ==== osd_op_reply(57 607.00000000 [setxattr (62)] v109'145 uv145 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f6b7800d720 con 0x55e501fb02d0 2024-08-03T20:57:53.751 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:53.749+0000 7f6b9371cf40 1 -- 172.21.15.32:0/3144827011 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:58 12.1c 12:3ed09add:::607.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109) -- 0x55e5020e7390 con 0x55e501fb02d0 2024-08-03T20:57:53.752 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:53.750+0000 7f6b8b400640 1 -- 172.21.15.32:0/3144827011 <== osd.0 v2:172.21.15.32:6816/3066156116 37 ==== osd_op_reply(58 607.00000000 [omap-set-header] v109'146 uv146 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f6b7800d720 con 0x55e501fb02d0 2024-08-03T20:57:53.752 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:53.750+0000 7f6b9371cf40 1 --2- 172.21.15.32:0/3144827011 >> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] conn(0x55e5020e7c60 0x55e5020e8070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:53.752 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:53.750+0000 7f6b9371cf40 1 -- 172.21.15.32:0/3144827011 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:59 12.6 12:654134d2:::mds0_openfiles.0:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109) -- 0x55e5020ea7e0 con 0x55e5020e7c60 2024-08-03T20:57:53.752 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:53.751+0000 7f6b8b400640 1 --2- 172.21.15.32:0/3144827011 >> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] conn(0x55e5020e7c60 0x55e5020e8070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:53.753 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:53.751+0000 7f6b8b400640 1 --2- 172.21.15.32:0/3144827011 >> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] conn(0x55e5020e7c60 0x55e5020e8070 crc :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:57:53.754 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:53.753+0000 7f6b8b400640 1 -- 172.21.15.32:0/3144827011 <== osd.1 v2:172.21.15.32:6824/1551241759 1 ==== osd_op_reply(59 mds0_openfiles.0 [create] v109'110 uv109 ondisk = -17 ((17) File exists)) ==== 160+0+0 (crc 0 0 0) 0x7f6b7800d720 con 0x55e5020e7c60 2024-08-03T20:57:53.754 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:53.753+0000 7f6b9371cf40 1 -- 172.21.15.32:0/3144827011 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:60 12.6 12:654134d2:::mds0_openfiles.0:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e109) -- 0x55e5020ea7e0 con 0x55e5020e7c60 2024-08-03T20:57:53.755 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:53.754+0000 7f6b8b400640 1 -- 172.21.15.32:0/3144827011 <== osd.1 v2:172.21.15.32:6824/1551241759 2 ==== osd_op_reply(60 mds0_openfiles.0 [delete] v109'111 uv111 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f6b7800d720 con 0x55e5020e7c60 2024-08-03T20:57:53.756 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:53.754+0000 7f6b9371cf40 1 -- 172.21.15.32:0/3144827011 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:61 12.6 12:654134d2:::mds0_openfiles.0:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109) -- 0x55e5020ea7e0 con 0x55e5020e7c60 2024-08-03T20:57:53.756 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds0_openfiles.0:head# 2024-08-03T20:57:53.757 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:53.755+0000 7f6b8b400640 1 -- 172.21.15.32:0/3144827011 <== osd.1 v2:172.21.15.32:6824/1551241759 3 ==== osd_op_reply(61 mds0_openfiles.0 [create] v109'112 uv112 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f6b7800d720 con 0x55e5020e7c60 2024-08-03T20:57:53.757 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:53.755+0000 7f6b9371cf40 1 -- 172.21.15.32:0/3144827011 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:62 12.6 12:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109) -- 0x55e5020eac00 con 0x55e5020e7c60 2024-08-03T20:57:53.758 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:53.756+0000 7f6b8b400640 1 -- 172.21.15.32:0/3144827011 <== osd.1 v2:172.21.15.32:6824/1551241759 4 ==== osd_op_reply(62 mds0_openfiles.0 [omap-set-header] v109'113 uv113 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f6b7800d720 con 0x55e5020e7c60 2024-08-03T20:57:53.758 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:53.756+0000 7f6b9371cf40 1 -- 172.21.15.32:0/3144827011 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:63 12.e 12:717a0223:::608.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109) -- 0x55e5020eac00 con 0x55e501fb02d0 2024-08-03T20:57:53.759 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:53.758+0000 7f6b8b400640 1 -- 172.21.15.32:0/3144827011 <== osd.0 v2:172.21.15.32:6816/3066156116 38 ==== osd_op_reply(63 608.00000000 [create] v109'141 uv140 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f6b7800d720 con 0x55e501fb02d0 2024-08-03T20:57:53.759 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:53.758+0000 7f6b9371cf40 1 -- 172.21.15.32:0/3144827011 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:64 12.e 12:717a0223:::608.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e109) -- 0x55e5020eac00 con 0x55e501fb02d0 2024-08-03T20:57:53.761 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:53.759+0000 7f6b8b400640 1 -- 172.21.15.32:0/3144827011 <== osd.0 v2:172.21.15.32:6816/3066156116 39 ==== osd_op_reply(64 608.00000000 [delete] v109'142 uv142 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f6b7800d720 con 0x55e501fb02d0 2024-08-03T20:57:53.761 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:53.759+0000 7f6b9371cf40 1 -- 172.21.15.32:0/3144827011 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:65 12.e 12:717a0223:::608.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109) -- 0x55e5020eac00 con 0x55e501fb02d0 2024-08-03T20:57:53.762 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::608.00000000:head# 2024-08-03T20:57:53.762 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:53.761+0000 7f6b8b400640 1 -- 172.21.15.32:0/3144827011 <== osd.0 v2:172.21.15.32:6816/3066156116 40 ==== osd_op_reply(65 608.00000000 [create] v109'143 uv143 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f6b7800d720 con 0x55e501fb02d0 2024-08-03T20:57:53.763 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:53.761+0000 7f6b9371cf40 1 -- 172.21.15.32:0/3144827011 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:66 12.e 12:717a0223:::608.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109) -- 0x55e5020ebc10 con 0x55e501fb02d0 2024-08-03T20:57:53.763 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:53.762+0000 7f6b8b400640 1 -- 172.21.15.32:0/3144827011 <== osd.0 v2:172.21.15.32:6816/3066156116 41 ==== osd_op_reply(66 608.00000000 [setxattr (30)] v109'144 uv144 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f6b7800d720 con 0x55e501fb02d0 2024-08-03T20:57:53.764 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:53.762+0000 7f6b9371cf40 1 -- 172.21.15.32:0/3144827011 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:67 12.e 12:717a0223:::608.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109) -- 0x55e5020ebc10 con 0x55e501fb02d0 2024-08-03T20:57:53.765 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:53.764+0000 7f6b8b400640 1 -- 172.21.15.32:0/3144827011 <== osd.0 v2:172.21.15.32:6816/3066156116 42 ==== osd_op_reply(67 608.00000000 [setxattr (62)] v109'145 uv145 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f6b7800d720 con 0x55e501fb02d0 2024-08-03T20:57:53.765 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:53.764+0000 7f6b9371cf40 1 -- 172.21.15.32:0/3144827011 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:68 12.e 12:717a0223:::608.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109) -- 0x55e5020eac00 con 0x55e501fb02d0 2024-08-03T20:57:53.766 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:53.765+0000 7f6b8b400640 1 -- 172.21.15.32:0/3144827011 <== osd.0 v2:172.21.15.32:6816/3066156116 43 ==== osd_op_reply(68 608.00000000 [omap-set-header] v109'146 uv146 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f6b7800d720 con 0x55e501fb02d0 2024-08-03T20:57:53.766 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:53.765+0000 7f6b9371cf40 1 --2- 172.21.15.32:0/3144827011 >> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] conn(0x55e5020eb110 0x55e5020ed530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:53.766 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:53.765+0000 7f6b9371cf40 1 -- 172.21.15.32:0/3144827011 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:69 12.15 12:a93a17c2:::604.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109) -- 0x55e5020edab0 con 0x55e5020eb110 2024-08-03T20:57:53.767 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:53.765+0000 7f6b8c800640 1 --2- 172.21.15.32:0/3144827011 >> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] conn(0x55e5020eb110 0x55e5020ed530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:53.767 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:53.766+0000 7f6b8c800640 1 --2- 172.21.15.32:0/3144827011 >> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] conn(0x55e5020eb110 0x55e5020ed530 crc :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:57:53.769 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:53.768+0000 7f6b8c800640 1 -- 172.21.15.32:0/3144827011 <== osd.6 v2:172.21.15.118:6825/801947386 1 ==== osd_op_reply(69 604.00000000 [create] v109'260 uv254 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f6b840abd70 con 0x55e5020eb110 2024-08-03T20:57:53.769 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:53.768+0000 7f6b9371cf40 1 -- 172.21.15.32:0/3144827011 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:70 12.15 12:a93a17c2:::604.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e109) -- 0x55e5020edf60 con 0x55e5020eb110 2024-08-03T20:57:53.770 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:53.769+0000 7f6b8c800640 1 -- 172.21.15.32:0/3144827011 <== osd.6 v2:172.21.15.118:6825/801947386 2 ==== osd_op_reply(70 604.00000000 [delete] v109'261 uv261 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f6b840abd70 con 0x55e5020eb110 2024-08-03T20:57:53.770 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:53.769+0000 7f6b9371cf40 1 -- 172.21.15.32:0/3144827011 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:71 12.15 12:a93a17c2:::604.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109) -- 0x55e5020ee440 con 0x55e5020eb110 2024-08-03T20:57:53.772 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::604.00000000:head# 2024-08-03T20:57:53.772 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:53.771+0000 7f6b8c800640 1 -- 172.21.15.32:0/3144827011 <== osd.6 v2:172.21.15.118:6825/801947386 3 ==== osd_op_reply(71 604.00000000 [create] v109'262 uv262 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f6b840abd70 con 0x55e5020eb110 2024-08-03T20:57:53.772 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:53.771+0000 7f6b9371cf40 1 -- 172.21.15.32:0/3144827011 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:72 12.15 12:a93a17c2:::604.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109) -- 0x55e5020efcf0 con 0x55e5020eb110 2024-08-03T20:57:53.773 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:53.772+0000 7f6b8c800640 1 -- 172.21.15.32:0/3144827011 <== osd.6 v2:172.21.15.118:6825/801947386 4 ==== osd_op_reply(72 604.00000000 [setxattr (30)] v109'263 uv263 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f6b840abd70 con 0x55e5020eb110 2024-08-03T20:57:53.774 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:53.772+0000 7f6b9371cf40 1 -- 172.21.15.32:0/3144827011 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:73 12.15 12:a93a17c2:::604.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109) -- 0x55e5020f0090 con 0x55e5020eb110 2024-08-03T20:57:53.774 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:53.773+0000 7f6b8c800640 1 -- 172.21.15.32:0/3144827011 <== osd.6 v2:172.21.15.118:6825/801947386 5 ==== osd_op_reply(73 604.00000000 [setxattr (62)] v109'264 uv264 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f6b840abd70 con 0x55e5020eb110 2024-08-03T20:57:53.775 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:53.773+0000 7f6b9371cf40 1 -- 172.21.15.32:0/3144827011 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:74 12.15 12:a93a17c2:::604.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109) -- 0x55e5020eece0 con 0x55e5020eb110 2024-08-03T20:57:53.776 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:53.775+0000 7f6b8c800640 1 -- 172.21.15.32:0/3144827011 <== osd.6 v2:172.21.15.118:6825/801947386 6 ==== osd_op_reply(74 604.00000000 [omap-set-header] v109'265 uv265 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f6b840abd70 con 0x55e5020eb110 2024-08-03T20:57:53.776 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:53.775+0000 7f6b9371cf40 1 -- 172.21.15.32:0/3144827011 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:75 12.15 12:aa448500:::500.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109) -- 0x55e5020ef0f0 con 0x55e5020eb110 2024-08-03T20:57:53.777 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:53.776+0000 7f6b8c800640 1 -- 172.21.15.32:0/3144827011 <== osd.6 v2:172.21.15.118:6825/801947386 7 ==== osd_op_reply(75 500.00000000 [create] v109'266 uv259 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f6b840abd70 con 0x55e5020eb110 2024-08-03T20:57:53.778 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:53.776+0000 7f6b9371cf40 1 -- 172.21.15.32:0/3144827011 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:76 12.15 12:aa448500:::500.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e109) -- 0x55e5020ef4e0 con 0x55e5020eb110 2024-08-03T20:57:53.779 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:53.778+0000 7f6b8c800640 1 -- 172.21.15.32:0/3144827011 <== osd.6 v2:172.21.15.118:6825/801947386 8 ==== osd_op_reply(76 500.00000000 [delete] v109'267 uv267 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f6b840abd70 con 0x55e5020eb110 2024-08-03T20:57:53.779 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:53.778+0000 7f6b9371cf40 1 -- 172.21.15.32:0/3144827011 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:77 12.15 12:aa448500:::500.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109) -- 0x55e5020ef4e0 con 0x55e5020eb110 2024-08-03T20:57:53.780 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::500.00000000:head# 2024-08-03T20:57:53.781 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:53.779+0000 7f6b8c800640 1 -- 172.21.15.32:0/3144827011 <== osd.6 v2:172.21.15.118:6825/801947386 9 ==== osd_op_reply(77 500.00000000 [create] v109'268 uv268 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f6b840abd70 con 0x55e5020eb110 2024-08-03T20:57:53.781 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:53.779+0000 7f6b9371cf40 1 -- 172.21.15.32:0/3144827011 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:78 12.15 12:aa448500:::500.00000000:head [write 0~90 in=90b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109) -- 0x55e5020ef490 con 0x55e5020eb110 2024-08-03T20:57:53.782 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:53.781+0000 7f6b8c800640 1 -- 172.21.15.32:0/3144827011 <== osd.6 v2:172.21.15.118:6825/801947386 10 ==== osd_op_reply(78 500.00000000 [write 0~90] v109'269 uv269 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f6b840abd70 con 0x55e5020eb110 2024-08-03T20:57:53.782 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:53.781+0000 7f6b9371cf40 1 -- 172.21.15.32:0/3144827011 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:79 12.15 12:aa448500:::500.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109) -- 0x55e5020ef490 con 0x55e5020eb110 2024-08-03T20:57:53.784 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:53.782+0000 7f6b8c800640 1 -- 172.21.15.32:0/3144827011 <== osd.6 v2:172.21.15.118:6825/801947386 11 ==== osd_op_reply(79 500.00000000 [omap-set-header] v109'270 uv270 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f6b840abd70 con 0x55e5020eb110 2024-08-03T20:57:53.784 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:53.782+0000 7f6b9371cf40 1 --2- 172.21.15.32:0/3144827011 >> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] conn(0x55e5020ef850 0x55e5020f2950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:53.784 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:53.782+0000 7f6b9371cf40 1 -- 172.21.15.32:0/3144827011 --> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] -- osd_op(unknown.0.0:80 12.d 12:b50e409b:::mds_snaptable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109) -- 0x55e5020f2e90 con 0x55e5020ef850 2024-08-03T20:57:53.784 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:53.783+0000 7f6b8be00640 1 --2- 172.21.15.32:0/3144827011 >> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] conn(0x55e5020ef850 0x55e5020f2950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:53.784 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:53.783+0000 7f6b8be00640 1 --2- 172.21.15.32:0/3144827011 >> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] conn(0x55e5020ef850 0x55e5020f2950 crc :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:57:53.786 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:53.785+0000 7f6b8be00640 1 -- 172.21.15.32:0/3144827011 <== osd.7 v2:172.21.15.118:6818/653000048 1 ==== osd_op_reply(80 mds_snaptable [create] v109'117 uv116 ondisk = -17 ((17) File exists)) ==== 157+0+0 (crc 0 0 0) 0x7f6b7c014070 con 0x55e5020ef850 2024-08-03T20:57:53.786 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:53.785+0000 7f6b9371cf40 1 -- 172.21.15.32:0/3144827011 --> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] -- osd_op(unknown.0.0:81 12.d 12:b50e409b:::mds_snaptable:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e109) -- 0x55e5020f2e90 con 0x55e5020ef850 2024-08-03T20:57:53.787 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:53.786+0000 7f6b8be00640 1 -- 172.21.15.32:0/3144827011 <== osd.7 v2:172.21.15.118:6818/653000048 2 ==== osd_op_reply(81 mds_snaptable [delete] v109'118 uv118 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f6b7c014070 con 0x55e5020ef850 2024-08-03T20:57:53.787 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:53.786+0000 7f6b9371cf40 1 -- 172.21.15.32:0/3144827011 --> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] -- osd_op(unknown.0.0:82 12.d 12:b50e409b:::mds_snaptable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109) -- 0x55e5020f2e90 con 0x55e5020ef850 2024-08-03T20:57:53.789 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds_snaptable:head# 2024-08-03T20:57:53.789 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:53.787+0000 7f6b8be00640 1 -- 172.21.15.32:0/3144827011 <== osd.7 v2:172.21.15.118:6818/653000048 3 ==== osd_op_reply(82 mds_snaptable [create] v109'119 uv119 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f6b7c014070 con 0x55e5020ef850 2024-08-03T20:57:53.789 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:53.787+0000 7f6b9371cf40 1 -- 172.21.15.32:0/3144827011 --> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] -- osd_op(unknown.0.0:83 12.d 12:b50e409b:::mds_snaptable:head [write 0~70 in=70b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109) -- 0x55e5020f3280 con 0x55e5020ef850 2024-08-03T20:57:53.790 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:53.789+0000 7f6b8be00640 1 -- 172.21.15.32:0/3144827011 <== osd.7 v2:172.21.15.118:6818/653000048 4 ==== osd_op_reply(83 mds_snaptable [write 0~70] v109'120 uv120 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f6b7c014070 con 0x55e5020ef850 2024-08-03T20:57:53.790 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:53.789+0000 7f6b9371cf40 1 -- 172.21.15.32:0/3144827011 --> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] -- osd_op(unknown.0.0:84 12.d 12:b50e409b:::mds_snaptable:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109) -- 0x55e5020f3280 con 0x55e5020ef850 2024-08-03T20:57:53.792 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:53.790+0000 7f6b8be00640 1 -- 172.21.15.32:0/3144827011 <== osd.7 v2:172.21.15.118:6818/653000048 5 ==== osd_op_reply(84 mds_snaptable [omap-set-header] v109'121 uv121 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f6b7c014070 con 0x55e5020ef850 2024-08-03T20:57:53.792 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:53.790+0000 7f6b9371cf40 1 -- 172.21.15.32:0/3144827011 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:85 12.1d 12:b871830b:::605.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109) -- 0x55e5020f3280 con 0x55e501fb02d0 2024-08-03T20:57:53.793 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:53.792+0000 7f6b8b400640 1 -- 172.21.15.32:0/3144827011 <== osd.0 v2:172.21.15.32:6816/3066156116 44 ==== osd_op_reply(85 605.00000000 [create] v109'141 uv140 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f6b7800d720 con 0x55e501fb02d0 2024-08-03T20:57:53.793 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:53.792+0000 7f6b9371cf40 1 -- 172.21.15.32:0/3144827011 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:86 12.1d 12:b871830b:::605.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e109) -- 0x55e5020f3280 con 0x55e501fb02d0 2024-08-03T20:57:53.794 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:53.793+0000 7f6b8b400640 1 -- 172.21.15.32:0/3144827011 <== osd.0 v2:172.21.15.32:6816/3066156116 45 ==== osd_op_reply(86 605.00000000 [delete] v109'142 uv142 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f6b7800d720 con 0x55e501fb02d0 2024-08-03T20:57:53.794 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:53.793+0000 7f6b9371cf40 1 -- 172.21.15.32:0/3144827011 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:87 12.1d 12:b871830b:::605.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109) -- 0x55e5020f3280 con 0x55e501fb02d0 2024-08-03T20:57:53.795 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:53.794+0000 7f6b8b400640 1 -- 172.21.15.32:0/3144827011 <== osd.0 v2:172.21.15.32:6816/3066156116 46 ==== osd_op_reply(87 605.00000000 [create] v109'143 uv143 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f6b7800d720 con 0x55e501fb02d0 2024-08-03T20:57:53.796 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::605.00000000:head# 2024-08-03T20:57:53.796 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:53.794+0000 7f6b9371cf40 1 -- 172.21.15.32:0/3144827011 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:88 12.1d 12:b871830b:::605.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109) -- 0x55e5020f4290 con 0x55e501fb02d0 2024-08-03T20:57:53.797 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:53.796+0000 7f6b8b400640 1 -- 172.21.15.32:0/3144827011 <== osd.0 v2:172.21.15.32:6816/3066156116 47 ==== osd_op_reply(88 605.00000000 [setxattr (30)] v109'144 uv144 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f6b7800d720 con 0x55e501fb02d0 2024-08-03T20:57:53.797 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:53.796+0000 7f6b9371cf40 1 -- 172.21.15.32:0/3144827011 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:89 12.1d 12:b871830b:::605.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109) -- 0x55e5020f4290 con 0x55e501fb02d0 2024-08-03T20:57:53.798 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:53.797+0000 7f6b8b400640 1 -- 172.21.15.32:0/3144827011 <== osd.0 v2:172.21.15.32:6816/3066156116 48 ==== osd_op_reply(89 605.00000000 [setxattr (62)] v109'145 uv145 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f6b7800d720 con 0x55e501fb02d0 2024-08-03T20:57:53.798 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:53.797+0000 7f6b9371cf40 1 -- 172.21.15.32:0/3144827011 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:90 12.1d 12:b871830b:::605.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109) -- 0x55e5020f3280 con 0x55e501fb02d0 2024-08-03T20:57:53.800 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:53.798+0000 7f6b8b400640 1 -- 172.21.15.32:0/3144827011 <== osd.0 v2:172.21.15.32:6816/3066156116 49 ==== osd_op_reply(90 605.00000000 [omap-set-header] v109'146 uv146 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f6b7800d720 con 0x55e501fb02d0 2024-08-03T20:57:53.800 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:53.798+0000 7f6b9371cf40 1 -- 172.21.15.32:0/3144827011 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:91 12.13 12:c91d4a1d:::mds0_inotable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109) -- 0x55e5020f3280 con 0x55e5020e7c60 2024-08-03T20:57:53.801 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:53.799+0000 7f6b8b400640 1 -- 172.21.15.32:0/3144827011 <== osd.1 v2:172.21.15.32:6824/1551241759 5 ==== osd_op_reply(91 mds0_inotable [create] v109'233 uv227 ondisk = -17 ((17) File exists)) ==== 157+0+0 (crc 0 0 0) 0x7f6b7800d720 con 0x55e5020e7c60 2024-08-03T20:57:53.801 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:53.799+0000 7f6b9371cf40 1 -- 172.21.15.32:0/3144827011 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:92 12.13 12:c91d4a1d:::mds0_inotable:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e109) -- 0x55e5020f3280 con 0x55e5020e7c60 2024-08-03T20:57:53.802 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:53.801+0000 7f6b8b400640 1 -- 172.21.15.32:0/3144827011 <== osd.1 v2:172.21.15.32:6824/1551241759 6 ==== osd_op_reply(92 mds0_inotable [delete] v109'234 uv234 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f6b7800d720 con 0x55e5020e7c60 2024-08-03T20:57:53.802 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:53.801+0000 7f6b9371cf40 1 -- 172.21.15.32:0/3144827011 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:93 12.13 12:c91d4a1d:::mds0_inotable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109) -- 0x55e5020f3280 con 0x55e5020e7c60 2024-08-03T20:57:53.803 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:53.802+0000 7f6b8b400640 1 -- 172.21.15.32:0/3144827011 <== osd.1 v2:172.21.15.32:6824/1551241759 7 ==== osd_op_reply(93 mds0_inotable [create] v109'235 uv235 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f6b7800d720 con 0x55e5020e7c60 2024-08-03T20:57:53.804 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds0_inotable:head# 2024-08-03T20:57:53.804 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:53.802+0000 7f6b9371cf40 1 -- 172.21.15.32:0/3144827011 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:94 12.13 12:c91d4a1d:::mds0_inotable:head [write 0~34 in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109) -- 0x55e5020f3280 con 0x55e5020e7c60 2024-08-03T20:57:53.805 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:53.804+0000 7f6b8b400640 1 -- 172.21.15.32:0/3144827011 <== osd.1 v2:172.21.15.32:6824/1551241759 8 ==== osd_op_reply(94 mds0_inotable [write 0~34] v109'236 uv236 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f6b7800d720 con 0x55e5020e7c60 2024-08-03T20:57:53.805 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:53.804+0000 7f6b9371cf40 1 -- 172.21.15.32:0/3144827011 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:95 12.13 12:c91d4a1d:::mds0_inotable:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109) -- 0x55e5020f3280 con 0x55e5020e7c60 2024-08-03T20:57:53.806 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:53.805+0000 7f6b8b400640 1 -- 172.21.15.32:0/3144827011 <== osd.1 v2:172.21.15.32:6824/1551241759 9 ==== osd_op_reply(95 mds0_inotable [omap-set-header] v109'237 uv237 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f6b7800d720 con 0x55e5020e7c60 2024-08-03T20:57:53.806 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:53.805+0000 7f6b9371cf40 1 -- 172.21.15.32:0/3144827011 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:96 12.13 12:cd5a64a3:::100.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109) -- 0x55e5020f3280 con 0x55e5020e7c60 2024-08-03T20:57:53.807 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:53.806+0000 7f6b8b400640 1 -- 172.21.15.32:0/3144827011 <== osd.1 v2:172.21.15.32:6824/1551241759 10 ==== osd_op_reply(96 100.00000000 [create] v109'238 uv232 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f6b7800d720 con 0x55e5020e7c60 2024-08-03T20:57:53.807 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:53.806+0000 7f6b9371cf40 1 -- 172.21.15.32:0/3144827011 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:97 12.13 12:cd5a64a3:::100.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e109) -- 0x55e5020f3280 con 0x55e5020e7c60 2024-08-03T20:57:53.809 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:53.807+0000 7f6b8b400640 1 -- 172.21.15.32:0/3144827011 <== osd.1 v2:172.21.15.32:6824/1551241759 11 ==== osd_op_reply(97 100.00000000 [delete] v109'239 uv239 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f6b7800d720 con 0x55e5020e7c60 2024-08-03T20:57:53.809 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:53.808+0000 7f6b9371cf40 1 -- 172.21.15.32:0/3144827011 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:98 12.13 12:cd5a64a3:::100.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109) -- 0x55e5020f3280 con 0x55e5020e7c60 2024-08-03T20:57:53.810 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::100.00000000:head# 2024-08-03T20:57:53.810 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:53.809+0000 7f6b8b400640 1 -- 172.21.15.32:0/3144827011 <== osd.1 v2:172.21.15.32:6824/1551241759 12 ==== osd_op_reply(98 100.00000000 [create] v109'240 uv240 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f6b7800d720 con 0x55e5020e7c60 2024-08-03T20:57:53.810 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:53.809+0000 7f6b9371cf40 1 -- 172.21.15.32:0/3144827011 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:99 12.13 12:cd5a64a3:::100.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109) -- 0x55e5020f3280 con 0x55e5020e7c60 2024-08-03T20:57:53.811 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:53.810+0000 7f6b8b400640 1 -- 172.21.15.32:0/3144827011 <== osd.1 v2:172.21.15.32:6824/1551241759 13 ==== osd_op_reply(99 100.00000000 [omap-set-header] v109'241 uv241 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f6b7800d720 con 0x55e5020e7c60 2024-08-03T20:57:53.812 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:53.810+0000 7f6b9371cf40 1 -- 172.21.15.32:0/3144827011 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:100 12.13 12:cd5a64a3:::100.00000000:head [omap-set-vals in=5194b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109) -- 0x55e5020f6090 con 0x55e5020e7c60 2024-08-03T20:57:53.813 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:53.812+0000 7f6b8b400640 1 -- 172.21.15.32:0/3144827011 <== osd.1 v2:172.21.15.32:6824/1551241759 14 ==== osd_op_reply(100 100.00000000 [omap-set-vals] v109'242 uv242 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f6b7800d720 con 0x55e5020e7c60 2024-08-03T20:57:53.813 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:53.812+0000 7f6b9371cf40 1 --2- 172.21.15.32:0/3144827011 >> [v2:172.21.15.32:6800/9088303,v1:172.21.15.32:6801/9088303] conn(0x55e5020f3db0 0x55e5020f6690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:53.813 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:53.812+0000 7f6b9371cf40 1 -- 172.21.15.32:0/3144827011 --> [v2:172.21.15.32:6800/9088303,v1:172.21.15.32:6801/9088303] -- osd_op(unknown.0.0:101 12.b 12:d0630e4c:::mds0_sessionmap:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109) -- 0x55e5020f61d0 con 0x55e5020f3db0 2024-08-03T20:57:53.813 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:53.812+0000 7f6b8b400640 1 --2- 172.21.15.32:0/3144827011 >> [v2:172.21.15.32:6800/9088303,v1:172.21.15.32:6801/9088303] conn(0x55e5020f3db0 0x55e5020f6690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:53.814 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:53.812+0000 7f6b8b400640 1 --2- 172.21.15.32:0/3144827011 >> [v2:172.21.15.32:6800/9088303,v1:172.21.15.32:6801/9088303] conn(0x55e5020f3db0 0x55e5020f6690 crc :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:57:53.815 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:53.814+0000 7f6b8b400640 1 -- 172.21.15.32:0/3144827011 <== osd.3 v2:172.21.15.32:6800/9088303 1 ==== osd_op_reply(101 mds0_sessionmap [create] v109'103 uv102 ondisk = -17 ((17) File exists)) ==== 159+0+0 (crc 0 0 0) 0x7f6b7800d720 con 0x55e5020f3db0 2024-08-03T20:57:53.815 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:53.814+0000 7f6b9371cf40 1 -- 172.21.15.32:0/3144827011 --> [v2:172.21.15.32:6800/9088303,v1:172.21.15.32:6801/9088303] -- osd_op(unknown.0.0:102 12.b 12:d0630e4c:::mds0_sessionmap:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e109) -- 0x55e5020f61d0 con 0x55e5020f3db0 2024-08-03T20:57:53.816 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:53.815+0000 7f6b8b400640 1 -- 172.21.15.32:0/3144827011 <== osd.3 v2:172.21.15.32:6800/9088303 2 ==== osd_op_reply(102 mds0_sessionmap [delete] v109'104 uv104 ondisk = 0) ==== 159+0+0 (crc 0 0 0) 0x7f6b7800d720 con 0x55e5020f3db0 2024-08-03T20:57:53.817 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:53.815+0000 7f6b9371cf40 1 -- 172.21.15.32:0/3144827011 --> [v2:172.21.15.32:6800/9088303,v1:172.21.15.32:6801/9088303] -- osd_op(unknown.0.0:103 12.b 12:d0630e4c:::mds0_sessionmap:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109) -- 0x55e5020f61d0 con 0x55e5020f3db0 2024-08-03T20:57:53.818 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:53.816+0000 7f6b8b400640 1 -- 172.21.15.32:0/3144827011 <== osd.3 v2:172.21.15.32:6800/9088303 3 ==== osd_op_reply(103 mds0_sessionmap [create] v109'105 uv105 ondisk = 0) ==== 159+0+0 (crc 0 0 0) 0x7f6b7800d720 con 0x55e5020f3db0 2024-08-03T20:57:53.818 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds0_sessionmap:head# 2024-08-03T20:57:53.818 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:53.816+0000 7f6b9371cf40 1 -- 172.21.15.32:0/3144827011 --> [v2:172.21.15.32:6800/9088303,v1:172.21.15.32:6801/9088303] -- osd_op(unknown.0.0:104 12.b 12:d0630e4c:::mds0_sessionmap:head [omap-set-header in=14b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109) -- 0x55e5020f61d0 con 0x55e5020f3db0 2024-08-03T20:57:53.819 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:53.818+0000 7f6b8b400640 1 -- 172.21.15.32:0/3144827011 <== osd.3 v2:172.21.15.32:6800/9088303 4 ==== osd_op_reply(104 mds0_sessionmap [omap-set-header] v109'106 uv106 ondisk = 0) ==== 159+0+0 (crc 0 0 0) 0x7f6b7800d720 con 0x55e5020f3db0 2024-08-03T20:57:53.819 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:53.818+0000 7f6b9371cf40 1 -- 172.21.15.32:0/3144827011 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:105 12.7 12:e0b41d06:::609.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109) -- 0x55e5020f61d0 con 0x55e5020e7c60 2024-08-03T20:57:53.820 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:53.819+0000 7f6b8b400640 1 -- 172.21.15.32:0/3144827011 <== osd.1 v2:172.21.15.32:6824/1551241759 15 ==== osd_op_reply(105 609.00000000 [create] v109'141 uv140 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f6b7800d720 con 0x55e5020e7c60 2024-08-03T20:57:53.821 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:53.819+0000 7f6b9371cf40 1 -- 172.21.15.32:0/3144827011 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:106 12.7 12:e0b41d06:::609.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e109) -- 0x55e5020f61d0 con 0x55e5020e7c60 2024-08-03T20:57:53.822 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:53.821+0000 7f6b8b400640 1 -- 172.21.15.32:0/3144827011 <== osd.1 v2:172.21.15.32:6824/1551241759 16 ==== osd_op_reply(106 609.00000000 [delete] v109'142 uv142 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f6b7800d720 con 0x55e5020e7c60 2024-08-03T20:57:53.822 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:53.821+0000 7f6b9371cf40 1 -- 172.21.15.32:0/3144827011 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:107 12.7 12:e0b41d06:::609.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109) -- 0x55e5020f61d0 con 0x55e5020e7c60 2024-08-03T20:57:53.823 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::609.00000000:head# 2024-08-03T20:57:53.824 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:53.822+0000 7f6b8b400640 1 -- 172.21.15.32:0/3144827011 <== osd.1 v2:172.21.15.32:6824/1551241759 17 ==== osd_op_reply(107 609.00000000 [create] v109'143 uv143 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f6b7800d720 con 0x55e5020e7c60 2024-08-03T20:57:53.824 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:53.822+0000 7f6b9371cf40 1 -- 172.21.15.32:0/3144827011 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:108 12.7 12:e0b41d06:::609.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109) -- 0x55e5020f61d0 con 0x55e5020e7c60 2024-08-03T20:57:53.825 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:53.824+0000 7f6b8b400640 1 -- 172.21.15.32:0/3144827011 <== osd.1 v2:172.21.15.32:6824/1551241759 18 ==== osd_op_reply(108 609.00000000 [setxattr (30)] v109'144 uv144 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f6b7800d720 con 0x55e5020e7c60 2024-08-03T20:57:53.825 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:53.824+0000 7f6b9371cf40 1 -- 172.21.15.32:0/3144827011 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:109 12.7 12:e0b41d06:::609.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109) -- 0x55e5020f61d0 con 0x55e5020e7c60 2024-08-03T20:57:53.826 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:53.825+0000 7f6b8b400640 1 -- 172.21.15.32:0/3144827011 <== osd.1 v2:172.21.15.32:6824/1551241759 19 ==== osd_op_reply(109 609.00000000 [setxattr (62)] v109'145 uv145 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f6b7800d720 con 0x55e5020e7c60 2024-08-03T20:57:53.826 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:53.825+0000 7f6b9371cf40 1 -- 172.21.15.32:0/3144827011 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:110 12.7 12:e0b41d06:::609.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109) -- 0x55e5020f61d0 con 0x55e5020e7c60 2024-08-03T20:57:53.828 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:53.827+0000 7f6b8b400640 1 -- 172.21.15.32:0/3144827011 <== osd.1 v2:172.21.15.32:6824/1551241759 20 ==== osd_op_reply(110 609.00000000 [omap-set-header] v109'146 uv146 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f6b7800d720 con 0x55e5020e7c60 2024-08-03T20:57:53.828 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:53.827+0000 7f6b9371cf40 1 -- 172.21.15.32:0/3144827011 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:111 12.f 12:f1f69726:::400.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109) -- 0x55e5020f61d0 con 0x55e5020eb110 2024-08-03T20:57:53.830 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:53.828+0000 7f6b8c800640 1 -- 172.21.15.32:0/3144827011 <== osd.6 v2:172.21.15.118:6825/801947386 12 ==== osd_op_reply(111 400.00000000 [create] v109'120 uv119 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f6b840abd70 con 0x55e5020eb110 2024-08-03T20:57:53.830 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:53.828+0000 7f6b9371cf40 1 -- 172.21.15.32:0/3144827011 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:112 12.f 12:f1f69726:::400.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e109) -- 0x55e5020f61d0 con 0x55e5020eb110 2024-08-03T20:57:53.831 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:53.830+0000 7f6b8c800640 1 -- 172.21.15.32:0/3144827011 <== osd.6 v2:172.21.15.118:6825/801947386 13 ==== osd_op_reply(112 400.00000000 [delete] v109'121 uv121 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f6b840abd70 con 0x55e5020eb110 2024-08-03T20:57:53.831 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:53.830+0000 7f6b9371cf40 1 -- 172.21.15.32:0/3144827011 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:113 12.f 12:f1f69726:::400.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109) -- 0x55e5020f61d0 con 0x55e5020eb110 2024-08-03T20:57:53.832 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::400.00000000:head# 2024-08-03T20:57:53.833 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:53.831+0000 7f6b8c800640 1 -- 172.21.15.32:0/3144827011 <== osd.6 v2:172.21.15.118:6825/801947386 14 ==== osd_op_reply(113 400.00000000 [create] v109'122 uv122 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f6b840abd70 con 0x55e5020eb110 2024-08-03T20:57:53.833 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:53.831+0000 7f6b9371cf40 1 -- 172.21.15.32:0/3144827011 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:114 12.f 12:f1f69726:::400.00000000:head [write 0~22 in=22b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109) -- 0x55e5020f6bd0 con 0x55e5020eb110 2024-08-03T20:57:53.834 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:53.832+0000 7f6b8c800640 1 -- 172.21.15.32:0/3144827011 <== osd.6 v2:172.21.15.118:6825/801947386 15 ==== osd_op_reply(114 400.00000000 [write 0~22] v109'123 uv123 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f6b840abd70 con 0x55e5020eb110 2024-08-03T20:57:53.834 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:53.833+0000 7f6b9371cf40 1 -- 172.21.15.32:0/3144827011 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:115 12.f 12:f1f69726:::400.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109) -- 0x55e5020f6bd0 con 0x55e5020eb110 2024-08-03T20:57:53.835 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:53.834+0000 7f6b8c800640 1 -- 172.21.15.32:0/3144827011 <== osd.6 v2:172.21.15.118:6825/801947386 16 ==== osd_op_reply(115 400.00000000 [omap-set-header] v109'124 uv124 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f6b840abd70 con 0x55e5020eb110 2024-08-03T20:57:53.835 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:53.834+0000 7f6b9371cf40 1 -- 172.21.15.32:0/3144827011 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:116 12.1f 12:fe07bba1:::100.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109) -- 0x55e5020f6bd0 con 0x55e5020eb110 2024-08-03T20:57:53.837 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:53.836+0000 7f6b8c800640 1 -- 172.21.15.32:0/3144827011 <== osd.6 v2:172.21.15.118:6825/801947386 17 ==== osd_op_reply(116 100.00000000.inode [create] v109'290 uv282 ondisk = -17 ((17) File exists)) ==== 162+0+0 (crc 0 0 0) 0x7f6b840abd70 con 0x55e5020eb110 2024-08-03T20:57:53.837 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:53.836+0000 7f6b9371cf40 1 -- 172.21.15.32:0/3144827011 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:117 12.1f 12:fe07bba1:::100.00000000.inode:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e109) -- 0x55e5020f6bd0 con 0x55e5020eb110 2024-08-03T20:57:53.839 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:53.838+0000 7f6b8c800640 1 -- 172.21.15.32:0/3144827011 <== osd.6 v2:172.21.15.118:6825/801947386 18 ==== osd_op_reply(117 100.00000000.inode [delete] v109'291 uv291 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7f6b840abd70 con 0x55e5020eb110 2024-08-03T20:57:53.839 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:53.838+0000 7f6b9371cf40 1 -- 172.21.15.32:0/3144827011 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:118 12.1f 12:fe07bba1:::100.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109) -- 0x55e5020f6bd0 con 0x55e5020eb110 2024-08-03T20:57:53.841 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:53.839+0000 7f6b8c800640 1 -- 172.21.15.32:0/3144827011 <== osd.6 v2:172.21.15.118:6825/801947386 19 ==== osd_op_reply(118 100.00000000.inode [create] v109'292 uv292 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7f6b840abd70 con 0x55e5020eb110 2024-08-03T20:57:53.841 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::100.00000000.inode:head# 2024-08-03T20:57:53.841 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:53.839+0000 7f6b9371cf40 1 -- 172.21.15.32:0/3144827011 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:119 12.1f 12:fe07bba1:::100.00000000.inode:head [write 0~582 in=582b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109) -- 0x55e5020f6bd0 con 0x55e5020eb110 2024-08-03T20:57:53.842 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:53.841+0000 7f6b8c800640 1 -- 172.21.15.32:0/3144827011 <== osd.6 v2:172.21.15.118:6825/801947386 20 ==== osd_op_reply(119 100.00000000.inode [write 0~582] v109'293 uv293 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7f6b840abd70 con 0x55e5020eb110 2024-08-03T20:57:53.842 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:53.841+0000 7f6b9371cf40 1 -- 172.21.15.32:0/3144827011 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:120 12.1f 12:fe07bba1:::100.00000000.inode:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109) -- 0x55e5020f6bd0 con 0x55e5020eb110 2024-08-03T20:57:53.844 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:53.842+0000 7f6b8c800640 1 -- 172.21.15.32:0/3144827011 <== osd.6 v2:172.21.15.118:6825/801947386 21 ==== osd_op_reply(120 100.00000000.inode [omap-set-header] v109'294 uv294 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7f6b840abd70 con 0x55e5020eb110 2024-08-03T20:57:53.844 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:53.842+0000 7f6b9371cf40 1 -- 172.21.15.32:0/3144827011 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:121 12.1f 12:ff5b34d6:::1.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109) -- 0x55e5020f6bd0 con 0x55e5020eb110 2024-08-03T20:57:53.845 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:53.844+0000 7f6b8c800640 1 -- 172.21.15.32:0/3144827011 <== osd.6 v2:172.21.15.118:6825/801947386 22 ==== osd_op_reply(121 1.00000000 [create] v109'295 uv289 ondisk = -17 ((17) File exists)) ==== 154+0+0 (crc 0 0 0) 0x7f6b840abd70 con 0x55e5020eb110 2024-08-03T20:57:53.845 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:53.844+0000 7f6b9371cf40 1 -- 172.21.15.32:0/3144827011 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:122 12.1f 12:ff5b34d6:::1.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e109) -- 0x55e5020f6bd0 con 0x55e5020eb110 2024-08-03T20:57:53.847 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:53.845+0000 7f6b8c800640 1 -- 172.21.15.32:0/3144827011 <== osd.6 v2:172.21.15.118:6825/801947386 23 ==== osd_op_reply(122 1.00000000 [delete] v109'296 uv296 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7f6b840abd70 con 0x55e5020eb110 2024-08-03T20:57:53.847 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:53.845+0000 7f6b9371cf40 1 -- 172.21.15.32:0/3144827011 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:123 12.1f 12:ff5b34d6:::1.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109) -- 0x55e5020f6bd0 con 0x55e5020eb110 2024-08-03T20:57:53.848 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:53.847+0000 7f6b8c800640 1 -- 172.21.15.32:0/3144827011 <== osd.6 v2:172.21.15.118:6825/801947386 24 ==== osd_op_reply(123 1.00000000 [create] v109'297 uv297 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7f6b840abd70 con 0x55e5020eb110 2024-08-03T20:57:53.849 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::1.00000000:head# 2024-08-03T20:57:53.849 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:53.847+0000 7f6b9371cf40 1 -- 172.21.15.32:0/3144827011 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:124 12.1f 12:ff5b34d6:::1.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109) -- 0x55e5020f7be0 con 0x55e5020eb110 2024-08-03T20:57:53.850 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:53.849+0000 7f6b8c800640 1 -- 172.21.15.32:0/3144827011 <== osd.6 v2:172.21.15.118:6825/801947386 25 ==== osd_op_reply(124 1.00000000 [setxattr (30)] v109'298 uv298 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7f6b840abd70 con 0x55e5020eb110 2024-08-03T20:57:53.850 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:53.849+0000 7f6b9371cf40 1 -- 172.21.15.32:0/3144827011 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:125 12.1f 12:ff5b34d6:::1.00000000:head [setxattr parent (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109) -- 0x55e5020f7be0 con 0x55e5020eb110 2024-08-03T20:57:53.852 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:53.850+0000 7f6b8c800640 1 -- 172.21.15.32:0/3144827011 <== osd.6 v2:172.21.15.118:6825/801947386 26 ==== osd_op_reply(125 1.00000000 [setxattr (30)] v109'299 uv299 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7f6b840abd70 con 0x55e5020eb110 2024-08-03T20:57:53.852 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:53.850+0000 7f6b9371cf40 1 -- 172.21.15.32:0/3144827011 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:126 12.1f 12:ff5b34d6:::1.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109) -- 0x55e5020f6bd0 con 0x55e5020eb110 2024-08-03T20:57:53.853 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:53.852+0000 7f6b8c800640 1 -- 172.21.15.32:0/3144827011 <== osd.6 v2:172.21.15.118:6825/801947386 27 ==== osd_op_reply(126 1.00000000 [omap-set-header] v109'300 uv300 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7f6b840abd70 con 0x55e5020eb110 2024-08-03T20:57:53.853 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:53.852+0000 7f6b9371cf40 1 -- 172.21.15.32:0/3144827011 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:127 12.1f 12:ff5b34d6:::1.00000000:head [omap-set-vals in=584b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109) -- 0x55e5020f7080 con 0x55e5020eb110 2024-08-03T20:57:53.856 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:53.853+0000 7f6b8c800640 1 -- 172.21.15.32:0/3144827011 <== osd.6 v2:172.21.15.118:6825/801947386 28 ==== osd_op_reply(127 1.00000000 [omap-set-vals] v109'301 uv301 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7f6b840abd70 con 0x55e5020eb110 2024-08-03T20:57:53.856 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:53.853+0000 7f6b9371cf40 1 -- 172.21.15.32:0/3144827011 >> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] conn(0x55e501fb02d0 msgr2=0x55e5020e1850 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:53.856 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:53.853+0000 7f6b9371cf40 1 --2- 172.21.15.32:0/3144827011 >> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] conn(0x55e501fb02d0 0x55e5020e1850 crc :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:53.856 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:53.853+0000 7f6b9371cf40 1 -- 172.21.15.32:0/3144827011 >> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] conn(0x55e5020e7c60 msgr2=0x55e5020e8070 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:53.856 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:53.853+0000 7f6b9371cf40 1 --2- 172.21.15.32:0/3144827011 >> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] conn(0x55e5020e7c60 0x55e5020e8070 crc :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:53.856 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:53.853+0000 7f6b9371cf40 1 -- 172.21.15.32:0/3144827011 >> [v2:172.21.15.32:6800/9088303,v1:172.21.15.32:6801/9088303] conn(0x55e5020f3db0 msgr2=0x55e5020f6690 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:53.856 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:53.853+0000 7f6b9371cf40 1 --2- 172.21.15.32:0/3144827011 >> [v2:172.21.15.32:6800/9088303,v1:172.21.15.32:6801/9088303] conn(0x55e5020f3db0 0x55e5020f6690 crc :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:53.856 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:53.853+0000 7f6b9371cf40 1 -- 172.21.15.32:0/3144827011 >> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] conn(0x55e5020e29e0 msgr2=0x55e5020e5890 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:53.856 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:53.853+0000 7f6b8c800640 1 -- 172.21.15.32:0/3144827011 reap_dead start 2024-08-03T20:57:53.856 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:53.853+0000 7f6b9371cf40 1 --2- 172.21.15.32:0/3144827011 >> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] conn(0x55e5020e29e0 0x55e5020e5890 crc :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:53.856 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:53.853+0000 7f6b9371cf40 1 -- 172.21.15.32:0/3144827011 >> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] conn(0x55e5020eb110 msgr2=0x55e5020ed530 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:53.856 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:53.853+0000 7f6b9371cf40 1 --2- 172.21.15.32:0/3144827011 >> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] conn(0x55e5020eb110 0x55e5020ed530 crc :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:53.857 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:53.853+0000 7f6b9371cf40 1 -- 172.21.15.32:0/3144827011 >> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] conn(0x55e5020ef850 msgr2=0x55e5020f2950 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:53.857 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:53.853+0000 7f6b9371cf40 1 --2- 172.21.15.32:0/3144827011 >> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] conn(0x55e5020ef850 0x55e5020f2950 crc :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:53.857 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:53.854+0000 7f6b9371cf40 1 -- 172.21.15.32:0/3144827011 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f6b6c0759f0 msgr2=0x7f6b6c077f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:53.857 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:53.854+0000 7f6b9371cf40 1 --2- 172.21.15.32:0/3144827011 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f6b6c0759f0 0x7f6b6c077f00 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7f6b7c0011f0 tx=0x7f6b7c018040 comp rx=0 tx=0).stop 2024-08-03T20:57:53.857 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:53.854+0000 7f6b9371cf40 1 -- 172.21.15.32:0/3144827011 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55e502036330 msgr2=0x55e5020ce1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:53.857 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:53.854+0000 7f6b9371cf40 1 --2- 172.21.15.32:0/3144827011 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55e502036330 0x55e5020ce1e0 secure :-1 s=READY pgs=1653 cs=0 l=1 rev1=1 crypto rx=0x7f6b840463f0 tx=0x7f6b840044c0 comp rx=0 tx=0).stop 2024-08-03T20:57:53.857 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:53.854+0000 7f6b8c800640 1 -- 172.21.15.32:0/3144827011 reap_dead start 2024-08-03T20:57:53.857 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:53.854+0000 7f6b9371cf40 1 -- 172.21.15.32:0/3144827011 shutdown_connections 2024-08-03T20:57:53.857 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:53.854+0000 7f6b9371cf40 1 -- 172.21.15.32:0/3144827011 >> 172.21.15.32:0/3144827011 conn(0x55e501e88ac0 msgr2=0x55e502037da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:57:53.857 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:53.854+0000 7f6b9371cf40 1 -- 172.21.15.32:0/3144827011 shutdown_connections 2024-08-03T20:57:53.857 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:53.854+0000 7f6b9371cf40 1 -- 172.21.15.32:0/3144827011 wait complete. 2024-08-03T20:57:53.870 DEBUG:teuthology.orchestra.run.smithi032:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata truncate 200.00000000 0 2024-08-03T20:57:54.013 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:54.012+0000 7fae0ee0bf40 1 Processor -- start 2024-08-03T20:57:54.014 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:54.012+0000 7fae0ee0bf40 1 -- start start 2024-08-03T20:57:54.016 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:54.013+0000 7fae0ee0bf40 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55f09c5e9b40 0x55f09c677760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:54.016 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:54.013+0000 7fae0ee0bf40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55f09c65cbe0 0x55f09c656520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:54.016 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:54.013+0000 7fae0ee0bf40 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55f09c678930 0x55f09c678d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:54.016 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:54.013+0000 7fae0ee0bf40 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55f09c550bf0 con 0x55f09c678930 2024-08-03T20:57:54.016 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:54.013+0000 7fae0ee0bf40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x55f09c55d6b0 con 0x55f09c65cbe0 2024-08-03T20:57:54.016 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:54.013+0000 7fae0ee0bf40 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x55f09c4f2e30 con 0x55f09c5e9b40 2024-08-03T20:57:54.016 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:54.013+0000 7fae07c00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55f09c678930 0x55f09c678d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:54.016 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:54.013+0000 7fae07c00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55f09c678930 0x55f09c678d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:51306/0 (socket says 172.21.15.32:51306) 2024-08-03T20:57:54.017 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:54.014+0000 7fae07c00640 1 -- 172.21.15.32:0/236242259 learned_addr learned my addr 172.21.15.32:0/236242259 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:57:54.017 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:54.014+0000 7fae06800640 1 --2- 172.21.15.32:0/236242259 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55f09c65cbe0 0x55f09c656520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:54.017 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:54.014+0000 7fae07200640 1 --2- 172.21.15.32:0/236242259 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55f09c5e9b40 0x55f09c677760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:54.017 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:54.014+0000 7fae07c00640 1 -- 172.21.15.32:0/236242259 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55f09c5e9b40 msgr2=0x55f09c677760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:57:54.017 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:54.014+0000 7fae07c00640 1 --2- 172.21.15.32:0/236242259 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55f09c5e9b40 0x55f09c677760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:54.017 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:54.014+0000 7fae07c00640 1 -- 172.21.15.32:0/236242259 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55f09c65cbe0 msgr2=0x55f09c656520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:57:54.017 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:54.014+0000 7fae07c00640 1 --2- 172.21.15.32:0/236242259 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55f09c65cbe0 0x55f09c656520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:54.017 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:54.014+0000 7fae07c00640 1 -- 172.21.15.32:0/236242259 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f09c659a60 con 0x55f09c678930 2024-08-03T20:57:54.017 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:54.014+0000 7fae06800640 1 --2- 172.21.15.32:0/236242259 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55f09c65cbe0 0x55f09c656520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-03T20:57:54.017 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:54.014+0000 7fae07c00640 1 --2- 172.21.15.32:0/236242259 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55f09c678930 0x55f09c678d40 secure :-1 s=READY pgs=1656 cs=0 l=1 rev1=1 crypto rx=0x7fae00025df0 tx=0x7fae00003330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=53622479530a13c0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:57:54.017 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:54.014+0000 7fae05400640 1 -- 172.21.15.32:0/236242259 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae00064020 con 0x55f09c678930 2024-08-03T20:57:54.018 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:54.014+0000 7fae05400640 1 -- 172.21.15.32:0/236242259 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fae0004dbd0 con 0x55f09c678930 2024-08-03T20:57:54.018 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:54.014+0000 7fae05400640 1 -- 172.21.15.32:0/236242259 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae00060470 con 0x55f09c678930 2024-08-03T20:57:54.025 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:54.015+0000 7fae0ee0bf40 1 -- 172.21.15.32:0/236242259 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55f09c678930 msgr2=0x55f09c678d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:54.025 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:54.015+0000 7fae0ee0bf40 1 --2- 172.21.15.32:0/236242259 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55f09c678930 0x55f09c678d40 secure :-1 s=READY pgs=1656 cs=0 l=1 rev1=1 crypto rx=0x7fae00025df0 tx=0x7fae00003330 comp rx=0 tx=0).stop 2024-08-03T20:57:54.025 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:54.015+0000 7fae0ee0bf40 1 -- 172.21.15.32:0/236242259 shutdown_connections 2024-08-03T20:57:54.025 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:54.015+0000 7fae0ee0bf40 1 --2- 172.21.15.32:0/236242259 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55f09c65cbe0 0x55f09c656520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:54.025 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:54.015+0000 7fae0ee0bf40 1 --2- 172.21.15.32:0/236242259 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55f09c678930 0x55f09c678d40 unknown :-1 s=CLOSED pgs=1656 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:54.025 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:54.015+0000 7fae0ee0bf40 1 --2- 172.21.15.32:0/236242259 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55f09c5e9b40 0x55f09c677760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:54.025 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:54.015+0000 7fae0ee0bf40 1 -- 172.21.15.32:0/236242259 >> 172.21.15.32:0/236242259 conn(0x55f09c4c4ac0 msgr2=0x55f09c67cc10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:57:54.025 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:54.015+0000 7fae0ee0bf40 1 -- 172.21.15.32:0/236242259 shutdown_connections 2024-08-03T20:57:54.025 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:54.015+0000 7fae0ee0bf40 1 -- 172.21.15.32:0/236242259 wait complete. 2024-08-03T20:57:54.025 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:54.016+0000 7fae0ee0bf40 1 Processor -- start 2024-08-03T20:57:54.025 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:54.016+0000 7fae0ee0bf40 1 -- start start 2024-08-03T20:57:54.025 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:54.018+0000 7fae0ee0bf40 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55f09c5e9b40 0x55f09c652130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:54.026 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:54.018+0000 7fae0ee0bf40 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55f09c65cbe0 0x55f09c657210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:54.026 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:54.018+0000 7fae0ee0bf40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55f09c678930 0x55f09c65da60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:54.026 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:54.018+0000 7fae0ee0bf40 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55f09c65e5b0 con 0x55f09c5e9b40 2024-08-03T20:57:54.026 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:54.018+0000 7fae0ee0bf40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x55f09c679390 con 0x55f09c678930 2024-08-03T20:57:54.026 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:54.018+0000 7fae0ee0bf40 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x55f09c5a48c0 con 0x55f09c65cbe0 2024-08-03T20:57:54.026 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:54.018+0000 7fae07200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55f09c5e9b40 0x55f09c652130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:54.026 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:54.018+0000 7fae07200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55f09c5e9b40 0x55f09c652130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:51316/0 (socket says 172.21.15.32:51316) 2024-08-03T20:57:54.026 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:54.018+0000 7fae07200640 1 -- 172.21.15.32:0/772595462 learned_addr learned my addr 172.21.15.32:0/772595462 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:57:54.026 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:54.018+0000 7fae07c00640 1 --2- 172.21.15.32:0/772595462 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55f09c678930 0x55f09c65da60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:54.026 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:54.018+0000 7fae06800640 1 --2- 172.21.15.32:0/772595462 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55f09c65cbe0 0x55f09c657210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:54.026 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:54.018+0000 7fae07200640 1 -- 172.21.15.32:0/772595462 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55f09c65cbe0 msgr2=0x55f09c657210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:54.027 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:54.018+0000 7fae07200640 1 --2- 172.21.15.32:0/772595462 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55f09c65cbe0 0x55f09c657210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:54.027 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:54.019+0000 7fae07200640 1 -- 172.21.15.32:0/772595462 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55f09c678930 msgr2=0x55f09c65da60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:54.027 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:54.019+0000 7fae07200640 1 --2- 172.21.15.32:0/772595462 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55f09c678930 0x55f09c65da60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:54.027 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:54.019+0000 7fae07200640 1 -- 172.21.15.32:0/772595462 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f09c652670 con 0x55f09c5e9b40 2024-08-03T20:57:54.027 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:54.019+0000 7fae07c00640 1 --2- 172.21.15.32:0/772595462 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55f09c678930 0x55f09c65da60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-03T20:57:54.027 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:54.019+0000 7fae07200640 1 --2- 172.21.15.32:0/772595462 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55f09c5e9b40 0x55f09c652130 secure :-1 s=READY pgs=1657 cs=0 l=1 rev1=1 crypto rx=0x7fadf800ddb0 tx=0x7fadf800b590 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:57:54.027 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:54.019+0000 7fadffe00640 1 -- 172.21.15.32:0/772595462 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fadf8015020 con 0x55f09c5e9b40 2024-08-03T20:57:54.027 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:54.019+0000 7fadffe00640 1 -- 172.21.15.32:0/772595462 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fadf8011750 con 0x55f09c5e9b40 2024-08-03T20:57:54.027 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:54.019+0000 7fae0ee0bf40 1 -- 172.21.15.32:0/772595462 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x55f09c67d170 con 0x55f09c5e9b40 2024-08-03T20:57:54.027 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:54.019+0000 7fadffe00640 1 -- 172.21.15.32:0/772595462 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fadf8011ba0 con 0x55f09c5e9b40 2024-08-03T20:57:54.027 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:54.019+0000 7fae0ee0bf40 1 -- 172.21.15.32:0/772595462 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f09c656a10 con 0x55f09c5e9b40 2024-08-03T20:57:54.027 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:54.021+0000 7fadffe00640 1 -- 172.21.15.32:0/772595462 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7fadf8018020 con 0x55f09c5e9b40 2024-08-03T20:57:54.027 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:54.022+0000 7fadffe00640 1 --2- 172.21.15.32:0/772595462 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fade8075a20 0x7fade8077f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:54.028 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:54.022+0000 7fadffe00640 1 -- 172.21.15.32:0/772595462 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(109..109 src has 1..109) ==== 7918+0+0 (secure 0 0 0) 0x7fadf8098410 con 0x55f09c5e9b40 2024-08-03T20:57:54.028 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:54.023+0000 7fae06800640 1 --2- 172.21.15.32:0/772595462 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fade8075a20 0x7fade8077f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:54.028 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:54.023+0000 7fae0ee0bf40 1 --2- 172.21.15.32:0/772595462 >> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] conn(0x55f09c64e640 0x55f09c5a3430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:54.028 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:54.023+0000 7fae0ee0bf40 1 -- 172.21.15.32:0/772595462 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:1 12.14 12:292cf221:::200.00000000:head [truncate 0] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109) -- 0x55f09c657750 con 0x55f09c64e640 2024-08-03T20:57:54.028 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:54.023+0000 7fae07c00640 1 --2- 172.21.15.32:0/772595462 >> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] conn(0x55f09c64e640 0x55f09c5a3430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:54.028 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:54.023+0000 7fae06800640 1 --2- 172.21.15.32:0/772595462 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fade8075a20 0x7fade8077f30 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x55f09c718e80 tx=0x7fadf40057c0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:57:54.028 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:54.024+0000 7fae07c00640 1 --2- 172.21.15.32:0/772595462 >> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] conn(0x55f09c64e640 0x55f09c5a3430 crc :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:57:54.029 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:54.025+0000 7fae07c00640 1 -- 172.21.15.32:0/772595462 <== osd.5 v2:172.21.15.118:6810/2724891770 1 ==== osd_op_reply(1 200.00000000 [truncate 0] v109'548 uv548 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fae0000b040 con 0x55f09c64e640 2024-08-03T20:57:54.029 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:54.025+0000 7fae0ee0bf40 1 -- 172.21.15.32:0/772595462 >> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] conn(0x55f09c64e640 msgr2=0x55f09c5a3430 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:54.029 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:54.025+0000 7fae0ee0bf40 1 --2- 172.21.15.32:0/772595462 >> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] conn(0x55f09c64e640 0x55f09c5a3430 crc :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:54.029 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:54.025+0000 7fae0ee0bf40 1 -- 172.21.15.32:0/772595462 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fade8075a20 msgr2=0x7fade8077f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:54.029 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:54.025+0000 7fae0ee0bf40 1 --2- 172.21.15.32:0/772595462 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fade8075a20 0x7fade8077f30 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x55f09c718e80 tx=0x7fadf40057c0 comp rx=0 tx=0).stop 2024-08-03T20:57:54.029 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:54.025+0000 7fae0ee0bf40 1 -- 172.21.15.32:0/772595462 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55f09c5e9b40 msgr2=0x55f09c652130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:54.030 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:54.025+0000 7fae0ee0bf40 1 --2- 172.21.15.32:0/772595462 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55f09c5e9b40 0x55f09c652130 secure :-1 s=READY pgs=1657 cs=0 l=1 rev1=1 crypto rx=0x7fadf800ddb0 tx=0x7fadf800b590 comp rx=0 tx=0).stop 2024-08-03T20:57:54.030 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:54.025+0000 7fae07c00640 1 -- 172.21.15.32:0/772595462 reap_dead start 2024-08-03T20:57:54.030 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:54.025+0000 7fae0ee0bf40 1 -- 172.21.15.32:0/772595462 shutdown_connections 2024-08-03T20:57:54.030 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:54.025+0000 7fae0ee0bf40 1 -- 172.21.15.32:0/772595462 >> 172.21.15.32:0/772595462 conn(0x55f09c4c4ac0 msgr2=0x55f09c677f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:57:54.030 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:54.025+0000 7fae0ee0bf40 1 -- 172.21.15.32:0/772595462 shutdown_connections 2024-08-03T20:57:54.030 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:54.026+0000 7fae0ee0bf40 1 -- 172.21.15.32:0/772595462 wait complete. 2024-08-03T20:57:54.038 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs joinable true 2024-08-03T20:57:54.270 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:54.268+0000 7f089e000640 1 -- 172.21.15.32:0/2481574587 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f0898155720 msgr2=0x7f08980af0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:54.270 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:54.268+0000 7f089e000640 1 --2- 172.21.15.32:0/2481574587 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f0898155720 0x7f08980af0e0 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7f0888009830 tx=0x7f088802a4a0 comp rx=0 tx=0).stop 2024-08-03T20:57:54.270 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:54.268+0000 7f089e000640 1 -- 172.21.15.32:0/2481574587 shutdown_connections 2024-08-03T20:57:54.270 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:54.268+0000 7f089e000640 1 --2- 172.21.15.32:0/2481574587 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f08980af620 0x7f08981621e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:54.270 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:54.268+0000 7f089e000640 1 --2- 172.21.15.32:0/2481574587 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f0898155720 0x7f08980af0e0 unknown :-1 s=CLOSED pgs=515 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:54.270 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:54.268+0000 7f089e000640 1 -- 172.21.15.32:0/2481574587 >> 172.21.15.32:0/2481574587 conn(0x7f08980b1bd0 msgr2=0x7f08980b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:57:54.270 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:54.268+0000 7f089e000640 1 -- 172.21.15.32:0/2481574587 shutdown_connections 2024-08-03T20:57:54.270 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:54.268+0000 7f089e000640 1 -- 172.21.15.32:0/2481574587 wait complete. 2024-08-03T20:57:54.272 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:54.270+0000 7f089e000640 1 Processor -- start 2024-08-03T20:57:54.272 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:54.270+0000 7f089e000640 1 -- start start 2024-08-03T20:57:54.275 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:54.272+0000 7f089e000640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f08980af620 0x7f0898143790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:54.275 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:54.272+0000 7f089e000640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f0898155720 0x7f0898146c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:54.275 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:54.272+0000 7f089e000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0898162720 0x7f0898144ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:54.275 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:54.272+0000 7f089e000640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f08980b4ed0 con 0x7f0898162720 2024-08-03T20:57:54.275 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:54.272+0000 7f089e000640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f08980b4d50 con 0x7f0898155720 2024-08-03T20:57:54.275 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:54.272+0000 7f089e000640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7f08980b5050 con 0x7f08980af620 2024-08-03T20:57:54.275 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:54.272+0000 7f0897e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0898162720 0x7f0898144ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:54.275 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:54.272+0000 7f0897e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0898162720 0x7f0898144ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:51332/0 (socket says 172.21.15.32:51332) 2024-08-03T20:57:54.275 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:54.272+0000 7f0897e00640 1 -- 172.21.15.32:0/3046987510 learned_addr learned my addr 172.21.15.32:0/3046987510 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:57:54.276 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:54.272+0000 7f0896a00640 1 --2- 172.21.15.32:0/3046987510 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f0898155720 0x7f0898146c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:54.276 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:54.272+0000 7f0897400640 1 --2- 172.21.15.32:0/3046987510 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f08980af620 0x7f0898143790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:54.276 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:54.273+0000 7f0897e00640 1 -- 172.21.15.32:0/3046987510 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f08980af620 msgr2=0x7f0898143790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:54.276 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:54.273+0000 7f0897e00640 1 --2- 172.21.15.32:0/3046987510 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f08980af620 0x7f0898143790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:54.276 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:54.273+0000 7f0897e00640 1 -- 172.21.15.32:0/3046987510 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f0898155720 msgr2=0x7f0898146c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:54.276 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:54.273+0000 7f0897e00640 1 --2- 172.21.15.32:0/3046987510 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f0898155720 0x7f0898146c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:54.276 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:54.273+0000 7f0897e00640 1 -- 172.21.15.32:0/3046987510 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0898143e50 con 0x7f0898162720 2024-08-03T20:57:54.276 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:54.273+0000 7f0897e00640 1 --2- 172.21.15.32:0/3046987510 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0898162720 0x7f0898144ef0 secure :-1 s=READY pgs=1659 cs=0 l=1 rev1=1 crypto rx=0x7f089000cb70 tx=0x7f089000a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:57:54.276 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:54.273+0000 7f088f400640 1 -- 172.21.15.32:0/3046987510 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0890016020 con 0x7f0898162720 2024-08-03T20:57:54.276 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:54.273+0000 7f088f400640 1 -- 172.21.15.32:0/3046987510 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f089000ae30 con 0x7f0898162720 2024-08-03T20:57:54.276 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:54.273+0000 7f089e000640 1 -- 172.21.15.32:0/3046987510 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0898144130 con 0x7f0898162720 2024-08-03T20:57:54.277 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:54.273+0000 7f088f400640 1 -- 172.21.15.32:0/3046987510 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0890012660 con 0x7f0898162720 2024-08-03T20:57:54.277 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:54.273+0000 7f089e000640 1 -- 172.21.15.32:0/3046987510 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f08981446f0 con 0x7f0898162720 2024-08-03T20:57:54.277 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:54.274+0000 7f089e000640 1 -- 172.21.15.32:0/3046987510 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f08981621e0 con 0x7f0898162720 2024-08-03T20:57:54.277 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:54.276+0000 7f088f400640 1 -- 172.21.15.32:0/3046987510 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f0890019020 con 0x7f0898162720 2024-08-03T20:57:54.278 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:54.277+0000 7f088f400640 1 --2- 172.21.15.32:0/3046987510 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f0878077040 0x7f0878079550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:54.279 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:54.277+0000 7f088f400640 1 -- 172.21.15.32:0/3046987510 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(109..109 src has 1..109) ==== 7918+0+0 (secure 0 0 0) 0x7f0890099380 con 0x7f0898162720 2024-08-03T20:57:54.279 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:54.278+0000 7f0897400640 1 --2- 172.21.15.32:0/3046987510 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f0878077040 0x7f0878079550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:54.279 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:54.278+0000 7f0897400640 1 --2- 172.21.15.32:0/3046987510 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f0878077040 0x7f0878079550 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7f0888034000 tx=0x7f0888004730 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:57:54.283 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:54.282+0000 7f088f400640 1 -- 172.21.15.32:0/3046987510 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7f0890020070 con 0x7f0898162720 2024-08-03T20:57:54.534 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:54.532+0000 7f089e000640 1 -- 172.21.15.32:0/3046987510 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "joinable", "val": "true"} v 0) -- 0x7f08980b0cf0 con 0x7f0898162720 2024-08-03T20:57:54.903 INFO:tasks.ceph.mds.a.smithi032.stderr: -26> 2024-08-03T20:57:50.062+0000 7f31e2a00640 -1 mds.0.cache.ino(0x1) No data while reading inode 0x1 2024-08-03T20:57:54.904 INFO:tasks.ceph.mds.a.smithi032.stderr: -24> 2024-08-03T20:57:50.062+0000 7f31e2a00640 -1 log_channel(cluster) log [ERR] : Error loading MDS rank 0: (2) No such file or directory 2024-08-03T20:57:54.926 INFO:tasks.ceph.mds.a.smithi032.stdout:starting mds.a at 2024-08-03T20:57:55.049 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:55.047+0000 7f088f400640 1 -- 172.21.15.32:0/3046987510 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "joinable", "val": "true"}]=0 cephfs marked joinable; MDS may join as newly active. v233) ==== 162+0+0 (secure 0 0 0) 0x7f08900615f0 con 0x7f0898162720 2024-08-03T20:57:55.049 INFO:teuthology.orchestra.run.smithi032.stderr:cephfs marked joinable; MDS may join as newly active. 2024-08-03T20:57:55.052 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:55.050+0000 7f089e000640 1 -- 172.21.15.32:0/3046987510 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f0878077040 msgr2=0x7f0878079550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:55.052 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:55.050+0000 7f089e000640 1 --2- 172.21.15.32:0/3046987510 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f0878077040 0x7f0878079550 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7f0888034000 tx=0x7f0888004730 comp rx=0 tx=0).stop 2024-08-03T20:57:55.052 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:55.050+0000 7f089e000640 1 -- 172.21.15.32:0/3046987510 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0898162720 msgr2=0x7f0898144ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:55.052 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:55.050+0000 7f089e000640 1 --2- 172.21.15.32:0/3046987510 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0898162720 0x7f0898144ef0 secure :-1 s=READY pgs=1659 cs=0 l=1 rev1=1 crypto rx=0x7f089000cb70 tx=0x7f089000a430 comp rx=0 tx=0).stop 2024-08-03T20:57:55.052 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:55.050+0000 7f089e000640 1 -- 172.21.15.32:0/3046987510 shutdown_connections 2024-08-03T20:57:55.052 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:55.050+0000 7f089e000640 1 --2- 172.21.15.32:0/3046987510 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f0878077040 0x7f0878079550 unknown :-1 s=CLOSED pgs=677 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:55.052 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:55.050+0000 7f089e000640 1 --2- 172.21.15.32:0/3046987510 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f0898155720 0x7f0898146c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:55.052 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:55.050+0000 7f089e000640 1 --2- 172.21.15.32:0/3046987510 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0898162720 0x7f0898144ef0 unknown :-1 s=CLOSED pgs=1659 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:55.052 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:55.050+0000 7f089e000640 1 --2- 172.21.15.32:0/3046987510 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f08980af620 0x7f0898143790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:55.052 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:55.050+0000 7f089e000640 1 -- 172.21.15.32:0/3046987510 >> 172.21.15.32:0/3046987510 conn(0x7f08980b1bd0 msgr2=0x7f089815f180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:57:55.053 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:55.050+0000 7f089e000640 1 -- 172.21.15.32:0/3046987510 shutdown_connections 2024-08-03T20:57:55.053 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:55.050+0000 7f089e000640 1 -- 172.21.15.32:0/3046987510 wait complete. 2024-08-03T20:57:55.062 INFO:tasks.ceph.mds.a.smithi032.stderr:2024-08-03T20:57:55.060+0000 7f87e3c00640 -1 log_channel(cluster) log [ERR] : Error loading MDS rank 0: (22) Invalid argument 2024-08-03T20:57:55.062 INFO:tasks.ceph.mon.a.smithi032.stderr:2024-08-03T20:57:55.061+0000 7f4e6d600640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-08-03T20:57:55.062 INFO:tasks.ceph.mon.a.smithi032.stderr:2024-08-03T20:57:55.061+0000 7f4e6d600640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 mds daemon damaged (MDS_DAMAGE) 2024-08-03T20:57:55.077 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-08-03T20:57:55.304 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:55.302+0000 7f87f9e00640 1 -- 172.21.15.32:0/4261649967 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f87f406e930 msgr2=0x7f87f4144f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:55.305 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:55.302+0000 7f87f9e00640 1 --2- 172.21.15.32:0/4261649967 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f87f406e930 0x7f87f4144f90 secure :-1 s=READY pgs=1662 cs=0 l=1 rev1=1 crypto rx=0x7f87e4005fd0 tx=0x7f87e402a530 comp rx=0 tx=0).stop 2024-08-03T20:57:55.305 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:55.302+0000 7f87f9e00640 1 -- 172.21.15.32:0/4261649967 shutdown_connections 2024-08-03T20:57:55.305 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:55.302+0000 7f87f9e00640 1 --2- 172.21.15.32:0/4261649967 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f87f415fff0 0x7f87f4160400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:55.305 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:55.302+0000 7f87f9e00640 1 --2- 172.21.15.32:0/4261649967 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f87f406e930 0x7f87f4144f90 unknown :-1 s=CLOSED pgs=1662 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:55.305 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:55.302+0000 7f87f9e00640 1 -- 172.21.15.32:0/4261649967 >> 172.21.15.32:0/4261649967 conn(0x7f87f40b1bd0 msgr2=0x7f87f40b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:57:55.305 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:55.303+0000 7f87f9e00640 1 -- 172.21.15.32:0/4261649967 shutdown_connections 2024-08-03T20:57:55.305 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:55.303+0000 7f87f9e00640 1 -- 172.21.15.32:0/4261649967 wait complete. 2024-08-03T20:57:55.306 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:55.305+0000 7f87f9e00640 1 Processor -- start 2024-08-03T20:57:55.306 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:55.305+0000 7f87f9e00640 1 -- start start 2024-08-03T20:57:55.309 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:55.306+0000 7f87f9e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f87f406e930 0x7f87f415b790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:55.310 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:55.306+0000 7f87f9e00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f87f4073e60 0x7f87f41f8c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:55.310 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:55.306+0000 7f87f9e00640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f87f415fff0 0x7f87f41f91c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:55.310 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:55.306+0000 7f87f9e00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f87f40cf840 con 0x7f87f406e930 2024-08-03T20:57:55.310 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:55.306+0000 7f87f9e00640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f87f4145f30 con 0x7f87f4073e60 2024-08-03T20:57:55.310 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:55.306+0000 7f87f9e00640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7f87f40cf9c0 con 0x7f87f415fff0 2024-08-03T20:57:55.310 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:55.307+0000 7f87f2a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f87f406e930 0x7f87f415b790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:55.310 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:55.307+0000 7f87f2a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f87f406e930 0x7f87f415b790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:51354/0 (socket says 172.21.15.32:51354) 2024-08-03T20:57:55.310 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:55.307+0000 7f87f2a00640 1 -- 172.21.15.32:0/2952932682 learned_addr learned my addr 172.21.15.32:0/2952932682 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:57:55.310 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:55.307+0000 7f87f2000640 1 --2- 172.21.15.32:0/2952932682 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f87f4073e60 0x7f87f41f8c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:55.310 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:55.307+0000 7f87f3400640 1 --2- 172.21.15.32:0/2952932682 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f87f415fff0 0x7f87f41f91c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:55.311 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:55.307+0000 7f87f2a00640 1 -- 172.21.15.32:0/2952932682 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f87f415fff0 msgr2=0x7f87f41f91c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:55.311 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:55.307+0000 7f87f2a00640 1 --2- 172.21.15.32:0/2952932682 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f87f415fff0 0x7f87f41f91c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:55.311 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:55.307+0000 7f87f2a00640 1 -- 172.21.15.32:0/2952932682 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f87f4073e60 msgr2=0x7f87f41f8c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:55.311 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:55.307+0000 7f87f2a00640 1 --2- 172.21.15.32:0/2952932682 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f87f4073e60 0x7f87f41f8c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:55.311 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:55.307+0000 7f87f2a00640 1 -- 172.21.15.32:0/2952932682 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f87f4135c50 con 0x7f87f406e930 2024-08-03T20:57:55.311 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:55.307+0000 7f87f3400640 1 --2- 172.21.15.32:0/2952932682 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f87f415fff0 0x7f87f41f91c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-03T20:57:55.311 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:55.307+0000 7f87f2a00640 1 --2- 172.21.15.32:0/2952932682 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f87f406e930 0x7f87f415b790 secure :-1 s=READY pgs=1663 cs=0 l=1 rev1=1 crypto rx=0x7f87e4034040 tx=0x7f87e40039e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:57:55.311 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:55.308+0000 7f87eb400640 1 -- 172.21.15.32:0/2952932682 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87e403e030 con 0x7f87f406e930 2024-08-03T20:57:55.311 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:55.308+0000 7f87eb400640 1 -- 172.21.15.32:0/2952932682 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f87e40047a0 con 0x7f87f406e930 2024-08-03T20:57:55.311 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:55.308+0000 7f87eb400640 1 -- 172.21.15.32:0/2952932682 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87e4035040 con 0x7f87f406e930 2024-08-03T20:57:55.311 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:55.308+0000 7f87f9e00640 1 -- 172.21.15.32:0/2952932682 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f87f40ae470 con 0x7f87f406e930 2024-08-03T20:57:55.311 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:55.308+0000 7f87f9e00640 1 -- 172.21.15.32:0/2952932682 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f87f40aea30 con 0x7f87f406e930 2024-08-03T20:57:55.312 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:55.309+0000 7f87f9e00640 1 -- 172.21.15.32:0/2952932682 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f87f4160d10 con 0x7f87f406e930 2024-08-03T20:57:55.312 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:55.310+0000 7f87eb400640 1 -- 172.21.15.32:0/2952932682 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f87e4041030 con 0x7f87f406e930 2024-08-03T20:57:55.313 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:55.311+0000 7f87eb400640 1 --2- 172.21.15.32:0/2952932682 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f87d4077040 0x7f87d4079550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:55.313 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:55.311+0000 7f87eb400640 1 -- 172.21.15.32:0/2952932682 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(110..110 src has 1..110) ==== 8004+0+0 (secure 0 0 0) 0x7f87e40b9a80 con 0x7f87f406e930 2024-08-03T20:57:55.313 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:55.312+0000 7f87f2000640 1 --2- 172.21.15.32:0/2952932682 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f87d4077040 0x7f87d4079550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:55.314 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:55.312+0000 7f87f2000640 1 --2- 172.21.15.32:0/2952932682 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f87d4077040 0x7f87d4079550 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7f87f413f360 tx=0x7f87e0009060 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:57:55.317 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:55.316+0000 7f87eb400640 1 -- 172.21.15.32:0/2952932682 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7f87e40bbe20 con 0x7f87f406e930 2024-08-03T20:57:55.565 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:55.564+0000 7f87f9e00640 1 -- 172.21.15.32:0/2952932682 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f87f41f9700 con 0x7f87f406e930 2024-08-03T20:57:55.566 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:55.565+0000 7f87eb400640 1 -- 172.21.15.32:0/2952932682 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 234 v234) ==== 95+0+4391 (secure 0 0 0) 0x7f87e4081c90 con 0x7f87f406e930 2024-08-03T20:57:55.566 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-03T20:57:55.566 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":234,"btime":"2024-08-03T20:57:55:062760+0000","default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6439,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6835/531080160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6834","nonce":531080160},{"type":"v1","addr":"172.21.15.118:6835","nonce":531080160}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":84},{"gid":6454,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6837/2796882319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6836","nonce":2796882319},{"type":"v1","addr":"172.21.15.118:6837","nonce":2796882319}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":86},{"gid":6813,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6834/2686933890","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":2686933890},{"type":"v1","addr":"172.21.15.32:6834","nonce":2686933890}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":85}],"filesystems":[{"mdsmap":{"epoch":234,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-08-03T20:54:26.950518+0000","modified":"2024-08-03T20:57:55.062696+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":110,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":6}]} 2024-08-03T20:57:55.567 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 234 2024-08-03T20:57:55.568 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:55.567+0000 7f87f9e00640 1 -- 172.21.15.32:0/2952932682 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f87d4077040 msgr2=0x7f87d4079550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:55.568 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:55.567+0000 7f87f9e00640 1 --2- 172.21.15.32:0/2952932682 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f87d4077040 0x7f87d4079550 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7f87f413f360 tx=0x7f87e0009060 comp rx=0 tx=0).stop 2024-08-03T20:57:55.569 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:55.567+0000 7f87f9e00640 1 -- 172.21.15.32:0/2952932682 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f87f406e930 msgr2=0x7f87f415b790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:55.569 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:55.567+0000 7f87f9e00640 1 --2- 172.21.15.32:0/2952932682 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f87f406e930 0x7f87f415b790 secure :-1 s=READY pgs=1663 cs=0 l=1 rev1=1 crypto rx=0x7f87e4034040 tx=0x7f87e40039e0 comp rx=0 tx=0).stop 2024-08-03T20:57:55.569 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:55.567+0000 7f87f9e00640 1 -- 172.21.15.32:0/2952932682 shutdown_connections 2024-08-03T20:57:55.569 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:55.567+0000 7f87f9e00640 1 --2- 172.21.15.32:0/2952932682 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f87d4077040 0x7f87d4079550 unknown :-1 s=CLOSED pgs=679 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:55.569 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:55.567+0000 7f87f9e00640 1 --2- 172.21.15.32:0/2952932682 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f87f4073e60 0x7f87f41f8c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:55.569 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:55.567+0000 7f87f9e00640 1 --2- 172.21.15.32:0/2952932682 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f87f415fff0 0x7f87f41f91c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:55.569 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:55.567+0000 7f87f9e00640 1 --2- 172.21.15.32:0/2952932682 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f87f406e930 0x7f87f415b790 unknown :-1 s=CLOSED pgs=1663 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:55.569 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:55.567+0000 7f87f9e00640 1 -- 172.21.15.32:0/2952932682 >> 172.21.15.32:0/2952932682 conn(0x7f87f40b1bd0 msgr2=0x7f87f40b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:57:55.569 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:55.567+0000 7f87f9e00640 1 -- 172.21.15.32:0/2952932682 shutdown_connections 2024-08-03T20:57:55.569 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:55.567+0000 7f87f9e00640 1 -- 172.21.15.32:0/2952932682 wait complete. 2024-08-03T20:57:55.594 DEBUG:tasks.ceph_test_case:wait_until_true: success in 0s and 0 retries 2024-08-03T20:57:55.594 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-08-03T20:57:55.825 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:55.823+0000 7fa12ac00640 1 -- 172.21.15.32:0/182670143 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa124073e60 msgr2=0x7fa124074270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:55.825 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:55.823+0000 7fa12ac00640 1 --2- 172.21.15.32:0/182670143 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa124073e60 0x7fa124074270 secure :-1 s=READY pgs=1665 cs=0 l=1 rev1=1 crypto rx=0x7fa10c009830 tx=0x7fa10c02a480 comp rx=0 tx=0).stop 2024-08-03T20:57:55.825 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:55.823+0000 7fa12ac00640 1 -- 172.21.15.32:0/182670143 shutdown_connections 2024-08-03T20:57:55.825 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:55.823+0000 7fa12ac00640 1 --2- 172.21.15.32:0/182670143 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa124073e60 0x7fa124074270 unknown :-1 s=CLOSED pgs=1665 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:55.825 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:55.823+0000 7fa12ac00640 1 --2- 172.21.15.32:0/182670143 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fa12406e930 0x7fa124144f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:55.825 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:55.823+0000 7fa12ac00640 1 -- 172.21.15.32:0/182670143 >> 172.21.15.32:0/182670143 conn(0x7fa1240b1bd0 msgr2=0x7fa1240b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:57:55.826 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:55.823+0000 7fa12ac00640 1 -- 172.21.15.32:0/182670143 shutdown_connections 2024-08-03T20:57:55.826 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:55.824+0000 7fa12ac00640 1 -- 172.21.15.32:0/182670143 wait complete. 2024-08-03T20:57:55.827 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:55.826+0000 7fa12ac00640 1 Processor -- start 2024-08-03T20:57:55.827 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:55.826+0000 7fa12ac00640 1 -- start start 2024-08-03T20:57:55.829 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:55.827+0000 7fa12ac00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa12406e930 0x7fa124132390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:55.829 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:55.827+0000 7fa12ac00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa124073e60 0x7fa1241f6740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:55.830 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:55.827+0000 7fa12ac00640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fa1240747b0 0x7fa12415ea80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:55.830 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:55.827+0000 7fa12ac00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fa1240cf9c0 con 0x7fa124073e60 2024-08-03T20:57:55.830 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:55.827+0000 7fa12ac00640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7fa1240cf840 con 0x7fa12406e930 2024-08-03T20:57:55.830 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:55.827+0000 7fa12ac00640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7fa1240ccff0 con 0x7fa1240747b0 2024-08-03T20:57:55.830 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:55.828+0000 7fa11be00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa124073e60 0x7fa1241f6740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:55.830 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:55.828+0000 7fa11be00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa124073e60 0x7fa1241f6740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:51378/0 (socket says 172.21.15.32:51378) 2024-08-03T20:57:55.830 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:55.828+0000 7fa11be00640 1 -- 172.21.15.32:0/530841817 learned_addr learned my addr 172.21.15.32:0/530841817 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:57:55.830 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:55.828+0000 7fa123e00640 1 --2- 172.21.15.32:0/530841817 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa12406e930 0x7fa124132390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:55.830 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:55.828+0000 7fa128c00640 1 --2- 172.21.15.32:0/530841817 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fa1240747b0 0x7fa12415ea80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:55.830 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:55.828+0000 7fa11be00640 1 -- 172.21.15.32:0/530841817 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fa1240747b0 msgr2=0x7fa12415ea80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:55.830 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:55.828+0000 7fa11be00640 1 --2- 172.21.15.32:0/530841817 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fa1240747b0 0x7fa12415ea80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:55.830 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:55.828+0000 7fa11be00640 1 -- 172.21.15.32:0/530841817 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa12406e930 msgr2=0x7fa124132390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:55.831 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:55.828+0000 7fa11be00640 1 --2- 172.21.15.32:0/530841817 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa12406e930 0x7fa124132390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:55.831 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:55.828+0000 7fa11be00640 1 -- 172.21.15.32:0/530841817 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa124133a70 con 0x7fa124073e60 2024-08-03T20:57:55.831 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:55.829+0000 7fa11be00640 1 --2- 172.21.15.32:0/530841817 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa124073e60 0x7fa1241f6740 secure :-1 s=READY pgs=1666 cs=0 l=1 rev1=1 crypto rx=0x7fa10c02aa70 tx=0x7fa10c033c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:57:55.832 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:55.829+0000 7fa120c00640 1 -- 172.21.15.32:0/530841817 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa10c005c90 con 0x7fa124073e60 2024-08-03T20:57:55.832 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:55.829+0000 7fa120c00640 1 -- 172.21.15.32:0/530841817 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fa10c031e50 con 0x7fa124073e60 2024-08-03T20:57:55.832 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:55.829+0000 7fa120c00640 1 -- 172.21.15.32:0/530841817 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa10c005260 con 0x7fa124073e60 2024-08-03T20:57:55.832 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:55.829+0000 7fa12ac00640 1 -- 172.21.15.32:0/530841817 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa124132cb0 con 0x7fa124073e60 2024-08-03T20:57:55.832 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:55.829+0000 7fa12ac00640 1 -- 172.21.15.32:0/530841817 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa124133270 con 0x7fa124073e60 2024-08-03T20:57:55.832 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:55.830+0000 7fa12ac00640 1 -- 172.21.15.32:0/530841817 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa1240cd530 con 0x7fa124073e60 2024-08-03T20:57:55.832 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:55.831+0000 7fa120c00640 1 -- 172.21.15.32:0/530841817 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7fa10c005400 con 0x7fa124073e60 2024-08-03T20:57:55.833 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:55.832+0000 7fa120c00640 1 --2- 172.21.15.32:0/530841817 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fa100077040 0x7fa100079550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:55.833 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:55.832+0000 7fa120c00640 1 -- 172.21.15.32:0/530841817 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(110..110 src has 1..110) ==== 8004+0+0 (secure 0 0 0) 0x7fa10c0ba270 con 0x7fa124073e60 2024-08-03T20:57:55.834 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:55.833+0000 7fa123e00640 1 --2- 172.21.15.32:0/530841817 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fa100077040 0x7fa100079550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:55.835 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:55.834+0000 7fa123e00640 1 --2- 172.21.15.32:0/530841817 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fa100077040 0x7fa100079550 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7fa114009b30 tx=0x7fa114005f90 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:57:55.838 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:55.836+0000 7fa120c00640 1 -- 172.21.15.32:0/530841817 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7fa10c03f030 con 0x7fa124073e60 2024-08-03T20:57:56.085 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:56.083+0000 7fa12ac00640 1 -- 172.21.15.32:0/530841817 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa1240cca90 con 0x7fa124073e60 2024-08-03T20:57:56.085 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:56.084+0000 7fa120c00640 1 -- 172.21.15.32:0/530841817 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 234 v234) ==== 95+0+4391 (secure 0 0 0) 0x7fa10c082480 con 0x7fa124073e60 2024-08-03T20:57:56.085 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-03T20:57:56.086 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":234,"btime":"2024-08-03T20:57:55:062760+0000","default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6439,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6835/531080160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6834","nonce":531080160},{"type":"v1","addr":"172.21.15.118:6835","nonce":531080160}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":84},{"gid":6454,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6837/2796882319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6836","nonce":2796882319},{"type":"v1","addr":"172.21.15.118:6837","nonce":2796882319}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":86},{"gid":6813,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6834/2686933890","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":2686933890},{"type":"v1","addr":"172.21.15.32:6834","nonce":2686933890}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":85}],"filesystems":[{"mdsmap":{"epoch":234,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-08-03T20:54:26.950518+0000","modified":"2024-08-03T20:57:55.062696+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":110,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":6}]} 2024-08-03T20:57:56.086 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 234 2024-08-03T20:57:56.088 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:56.086+0000 7fa12ac00640 1 -- 172.21.15.32:0/530841817 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fa100077040 msgr2=0x7fa100079550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:56.088 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:56.086+0000 7fa12ac00640 1 --2- 172.21.15.32:0/530841817 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fa100077040 0x7fa100079550 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7fa114009b30 tx=0x7fa114005f90 comp rx=0 tx=0).stop 2024-08-03T20:57:56.088 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:56.086+0000 7fa12ac00640 1 -- 172.21.15.32:0/530841817 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa124073e60 msgr2=0x7fa1241f6740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:56.088 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:56.086+0000 7fa12ac00640 1 --2- 172.21.15.32:0/530841817 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa124073e60 0x7fa1241f6740 secure :-1 s=READY pgs=1666 cs=0 l=1 rev1=1 crypto rx=0x7fa10c02aa70 tx=0x7fa10c033c20 comp rx=0 tx=0).stop 2024-08-03T20:57:56.088 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:56.086+0000 7fa12ac00640 1 -- 172.21.15.32:0/530841817 shutdown_connections 2024-08-03T20:57:56.088 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:56.086+0000 7fa12ac00640 1 --2- 172.21.15.32:0/530841817 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fa100077040 0x7fa100079550 unknown :-1 s=CLOSED pgs=680 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:56.088 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:56.086+0000 7fa12ac00640 1 --2- 172.21.15.32:0/530841817 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fa1240747b0 0x7fa12415ea80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:56.089 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:56.086+0000 7fa12ac00640 1 --2- 172.21.15.32:0/530841817 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa124073e60 0x7fa1241f6740 unknown :-1 s=CLOSED pgs=1666 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:56.089 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:56.086+0000 7fa12ac00640 1 --2- 172.21.15.32:0/530841817 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa12406e930 0x7fa124132390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:56.089 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:56.086+0000 7fa12ac00640 1 -- 172.21.15.32:0/530841817 >> 172.21.15.32:0/530841817 conn(0x7fa1240b1bd0 msgr2=0x7fa1240b13a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:57:56.089 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:56.086+0000 7fa12ac00640 1 -- 172.21.15.32:0/530841817 shutdown_connections 2024-08-03T20:57:56.089 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:56.086+0000 7fa12ac00640 1 -- 172.21.15.32:0/530841817 wait complete. 2024-08-03T20:57:56.110 DEBUG:tasks.ceph_test_case:wait_until_true: success in 0s and 0 retries 2024-08-03T20:57:56.110 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-08-03T20:57:56.338 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:56.336+0000 7f38d1e00640 1 -- 172.21.15.32:0/3622943732 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f38cc06e930 msgr2=0x7f38cc144f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:56.338 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:56.336+0000 7f38d1e00640 1 --2- 172.21.15.32:0/3622943732 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f38cc06e930 0x7f38cc144f90 secure :-1 s=READY pgs=1668 cs=0 l=1 rev1=1 crypto rx=0x7f38bc009830 tx=0x7f38bc02a480 comp rx=0 tx=0).stop 2024-08-03T20:57:56.338 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:56.336+0000 7f38d1e00640 1 -- 172.21.15.32:0/3622943732 shutdown_connections 2024-08-03T20:57:56.338 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:56.336+0000 7f38d1e00640 1 --2- 172.21.15.32:0/3622943732 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f38cc073e60 0x7f38cc074270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:56.338 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:56.336+0000 7f38d1e00640 1 --2- 172.21.15.32:0/3622943732 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f38cc06e930 0x7f38cc144f90 unknown :-1 s=CLOSED pgs=1668 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:56.339 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:56.336+0000 7f38d1e00640 1 -- 172.21.15.32:0/3622943732 >> 172.21.15.32:0/3622943732 conn(0x7f38cc0b1bd0 msgr2=0x7f38cc0b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:57:56.339 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:56.336+0000 7f38d1e00640 1 -- 172.21.15.32:0/3622943732 shutdown_connections 2024-08-03T20:57:56.339 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:56.336+0000 7f38d1e00640 1 -- 172.21.15.32:0/3622943732 wait complete. 2024-08-03T20:57:56.340 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:56.339+0000 7f38d1e00640 1 Processor -- start 2024-08-03T20:57:56.340 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:56.339+0000 7f38d1e00640 1 -- start start 2024-08-03T20:57:56.343 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:56.340+0000 7f38d1e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f38cc06e930 0x7f38cc1f8c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:56.343 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:56.340+0000 7f38d1e00640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f38cc073e60 0x7f38cc132fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:56.343 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:56.340+0000 7f38d1e00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f38cc0747b0 0x7f38cc132720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:56.343 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:56.340+0000 7f38d1e00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f38cc0cf9c0 con 0x7f38cc06e930 2024-08-03T20:57:56.343 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:56.340+0000 7f38d1e00640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f38cc0cf840 con 0x7f38cc0747b0 2024-08-03T20:57:56.343 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:56.340+0000 7f38d1e00640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7f38cc0ccff0 con 0x7f38cc073e60 2024-08-03T20:57:56.343 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:56.340+0000 7f38caa00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f38cc06e930 0x7f38cc1f8c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:56.343 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:56.341+0000 7f38caa00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f38cc06e930 0x7f38cc1f8c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:51404/0 (socket says 172.21.15.32:51404) 2024-08-03T20:57:56.344 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:56.341+0000 7f38caa00640 1 -- 172.21.15.32:0/1338094619 learned_addr learned my addr 172.21.15.32:0/1338094619 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:57:56.344 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:56.341+0000 7f38c3e00640 1 --2- 172.21.15.32:0/1338094619 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f38cc073e60 0x7f38cc132fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:56.344 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:56.341+0000 7f38cb400640 1 --2- 172.21.15.32:0/1338094619 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f38cc0747b0 0x7f38cc132720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:56.344 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:56.341+0000 7f38caa00640 1 -- 172.21.15.32:0/1338094619 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f38cc073e60 msgr2=0x7f38cc132fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:56.344 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:56.341+0000 7f38caa00640 1 --2- 172.21.15.32:0/1338094619 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f38cc073e60 0x7f38cc132fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:56.344 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:56.341+0000 7f38caa00640 1 -- 172.21.15.32:0/1338094619 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f38cc0747b0 msgr2=0x7f38cc132720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:56.344 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:56.341+0000 7f38caa00640 1 --2- 172.21.15.32:0/1338094619 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f38cc0747b0 0x7f38cc132720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:56.344 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:56.341+0000 7f38caa00640 1 -- 172.21.15.32:0/1338094619 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f38cc133510 con 0x7f38cc06e930 2024-08-03T20:57:56.344 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:56.341+0000 7f38caa00640 1 --2- 172.21.15.32:0/1338094619 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f38cc06e930 0x7f38cc1f8c20 secure :-1 s=READY pgs=1669 cs=0 l=1 rev1=1 crypto rx=0x7f38bc000c00 tx=0x7f38bc033c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:57:56.344 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:56.342+0000 7f38c3400640 1 -- 172.21.15.32:0/1338094619 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38bc031d60 con 0x7f38cc06e930 2024-08-03T20:57:56.344 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:56.342+0000 7f38c3400640 1 -- 172.21.15.32:0/1338094619 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f38bc033e10 con 0x7f38cc06e930 2024-08-03T20:57:56.344 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:56.342+0000 7f38c3400640 1 -- 172.21.15.32:0/1338094619 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38bc033310 con 0x7f38cc06e930 2024-08-03T20:57:56.344 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:56.342+0000 7f38d1e00640 1 -- 172.21.15.32:0/1338094619 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f38cc1f9450 con 0x7f38cc06e930 2024-08-03T20:57:56.344 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:56.342+0000 7f38d1e00640 1 -- 172.21.15.32:0/1338094619 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f38cc1f9a10 con 0x7f38cc06e930 2024-08-03T20:57:56.345 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:56.343+0000 7f38d1e00640 1 -- 172.21.15.32:0/1338094619 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f38cc06a430 con 0x7f38cc06e930 2024-08-03T20:57:56.345 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:56.344+0000 7f38c3400640 1 -- 172.21.15.32:0/1338094619 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f38bc0334b0 con 0x7f38cc06e930 2024-08-03T20:57:56.347 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:56.345+0000 7f38c3400640 1 --2- 172.21.15.32:0/1338094619 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f38a8077040 0x7f38a8079550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:56.347 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:56.346+0000 7f38c3400640 1 -- 172.21.15.32:0/1338094619 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(110..110 src has 1..110) ==== 8004+0+0 (secure 0 0 0) 0x7f38bc0ba5c0 con 0x7f38cc06e930 2024-08-03T20:57:56.347 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:56.346+0000 7f38c3e00640 1 --2- 172.21.15.32:0/1338094619 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f38a8077040 0x7f38a8079550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:56.348 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:56.347+0000 7f38c3e00640 1 --2- 172.21.15.32:0/1338094619 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f38a8077040 0x7f38a8079550 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 crypto rx=0x7f38cc0b51b0 tx=0x7f38b4005f90 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:57:56.351 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:56.350+0000 7f38c3400640 1 -- 172.21.15.32:0/1338094619 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7f38cc06a430 con 0x7f38cc06e930 2024-08-03T20:57:56.597 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:56.595+0000 7f38d1e00640 1 -- 172.21.15.32:0/1338094619 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f38cc0cd530 con 0x7f38cc06e930 2024-08-03T20:57:56.597 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:56.596+0000 7f38c3400640 1 -- 172.21.15.32:0/1338094619 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 234 v234) ==== 95+0+4391 (secure 0 0 0) 0x7f38bc03f030 con 0x7f38cc06e930 2024-08-03T20:57:56.598 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-03T20:57:56.598 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":234,"btime":"2024-08-03T20:57:55:062760+0000","default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6439,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6835/531080160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6834","nonce":531080160},{"type":"v1","addr":"172.21.15.118:6835","nonce":531080160}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":84},{"gid":6454,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6837/2796882319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6836","nonce":2796882319},{"type":"v1","addr":"172.21.15.118:6837","nonce":2796882319}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":86},{"gid":6813,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6834/2686933890","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":2686933890},{"type":"v1","addr":"172.21.15.32:6834","nonce":2686933890}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":85}],"filesystems":[{"mdsmap":{"epoch":234,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-08-03T20:54:26.950518+0000","modified":"2024-08-03T20:57:55.062696+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":110,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":6}]} 2024-08-03T20:57:56.598 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 234 2024-08-03T20:57:56.600 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:56.598+0000 7f38d1e00640 1 -- 172.21.15.32:0/1338094619 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f38a8077040 msgr2=0x7f38a8079550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:56.600 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:56.598+0000 7f38d1e00640 1 --2- 172.21.15.32:0/1338094619 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f38a8077040 0x7f38a8079550 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 crypto rx=0x7f38cc0b51b0 tx=0x7f38b4005f90 comp rx=0 tx=0).stop 2024-08-03T20:57:56.600 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:56.598+0000 7f38d1e00640 1 -- 172.21.15.32:0/1338094619 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f38cc06e930 msgr2=0x7f38cc1f8c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:56.600 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:56.598+0000 7f38d1e00640 1 --2- 172.21.15.32:0/1338094619 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f38cc06e930 0x7f38cc1f8c20 secure :-1 s=READY pgs=1669 cs=0 l=1 rev1=1 crypto rx=0x7f38bc000c00 tx=0x7f38bc033c20 comp rx=0 tx=0).stop 2024-08-03T20:57:56.600 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:56.598+0000 7f38d1e00640 1 -- 172.21.15.32:0/1338094619 shutdown_connections 2024-08-03T20:57:56.600 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:56.598+0000 7f38d1e00640 1 --2- 172.21.15.32:0/1338094619 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f38a8077040 0x7f38a8079550 unknown :-1 s=CLOSED pgs=681 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:56.600 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:56.598+0000 7f38d1e00640 1 --2- 172.21.15.32:0/1338094619 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f38cc0747b0 0x7f38cc132720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:56.600 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:56.598+0000 7f38d1e00640 1 --2- 172.21.15.32:0/1338094619 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f38cc073e60 0x7f38cc132fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:56.600 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:56.598+0000 7f38d1e00640 1 --2- 172.21.15.32:0/1338094619 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f38cc06e930 0x7f38cc1f8c20 unknown :-1 s=CLOSED pgs=1669 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:56.600 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:56.598+0000 7f38d1e00640 1 -- 172.21.15.32:0/1338094619 >> 172.21.15.32:0/1338094619 conn(0x7f38cc0b1bd0 msgr2=0x7f38cc0b0700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:57:56.600 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:56.598+0000 7f38d1e00640 1 -- 172.21.15.32:0/1338094619 shutdown_connections 2024-08-03T20:57:56.600 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:56.598+0000 7f38d1e00640 1 -- 172.21.15.32:0/1338094619 wait complete. 2024-08-03T20:57:56.621 INFO:tasks.cephfs.test_damage:Result: Mutation 'Truncate 200.00000000' led to DAMAGED state 2024-08-03T20:57:56.622 INFO:tasks.cephfs.test_damage:Applying mutation 'Truncate 200.00000001' 2024-08-03T20:57:56.622 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-08-03T20:57:56.622 DEBUG:teuthology.orchestra.run.smithi032:> dd if=/proc/self/mounts of=/dev/stdout 2024-08-03T20:57:56.658 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053453,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6499208k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 none /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249604k,nr_inodes=812401,mode=700,uid=1000,gid=1316 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6499208k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-08-03T20:57:56.658 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi032.front.sepia.ceph.com 2024-08-03T20:57:56.658 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-08-03T20:57:56.658 DEBUG:teuthology.orchestra.run.smithi032:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-08-03T20:57:56.733 INFO:teuthology.orchestra.run.smithi032.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.0': No such file or directory 2024-08-03T20:57:56.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-03T20:57:56.734 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-08-03T20:57:57.004 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:57.002+0000 7f953ce00640 1 -- 172.21.15.32:0/631364042 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f951c004800 msgr2=0x7f951c004c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:57.004 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:57.002+0000 7f953ce00640 1 --2- 172.21.15.32:0/631364042 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f951c004800 0x7f951c004c10 secure :-1 s=READY pgs=1671 cs=0 l=1 rev1=1 crypto rx=0x7f9520009b50 tx=0x7f951c005830 comp rx=0 tx=0).stop 2024-08-03T20:57:57.004 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:57.002+0000 7f9536800640 1 -- 172.21.15.32:0/631364042 reap_dead start 2024-08-03T20:57:57.004 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:57.002+0000 7f953ce00640 1 -- 172.21.15.32:0/631364042 shutdown_connections 2024-08-03T20:57:57.004 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:57.002+0000 7f953ce00640 1 -- 172.21.15.32:0/631364042 >> 172.21.15.32:0/631364042 conn(0x7f95380b1bd0 msgr2=0x7f95380b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:57:57.004 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:57.002+0000 7f953ce00640 1 -- 172.21.15.32:0/631364042 shutdown_connections 2024-08-03T20:57:57.004 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:57.002+0000 7f953ce00640 1 -- 172.21.15.32:0/631364042 wait complete. 2024-08-03T20:57:57.006 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:57.004+0000 7f953ce00640 1 Processor -- start 2024-08-03T20:57:57.006 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:57.005+0000 7f953ce00640 1 -- start start 2024-08-03T20:57:57.008 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:57.006+0000 7f953ce00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f951c004800 0x7f95381f36c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:57.008 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:57.006+0000 7f953ce00640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f95380af350 0x7f953813b1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:57.008 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:57.006+0000 7f953ce00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f953815bb90 0x7f9538142720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:57.008 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:57.006+0000 7f953ce00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f95380cf9c0 con 0x7f951c004800 2024-08-03T20:57:57.008 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:57.006+0000 7f953ce00640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f95380cf840 con 0x7f953815bb90 2024-08-03T20:57:57.008 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:57.006+0000 7f953ce00640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7f9538074270 con 0x7f95380af350 2024-08-03T20:57:57.008 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:57.006+0000 7f9535e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f951c004800 0x7f95381f36c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:57.008 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:57.006+0000 7f9535e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f951c004800 0x7f95381f36c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:51430/0 (socket says 172.21.15.32:51430) 2024-08-03T20:57:57.008 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:57.006+0000 7f9535e00640 1 -- 172.21.15.32:0/3483568888 learned_addr learned my addr 172.21.15.32:0/3483568888 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:57:57.008 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:57.006+0000 7f952fe00640 1 --2- 172.21.15.32:0/3483568888 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f95380af350 0x7f953813b1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:57.008 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:57.006+0000 7f9536800640 1 --2- 172.21.15.32:0/3483568888 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f953815bb90 0x7f9538142720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:57.010 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:57.007+0000 7f9535e00640 1 -- 172.21.15.32:0/3483568888 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f95380af350 msgr2=0x7f953813b1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:57.010 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:57.007+0000 7f9535e00640 1 --2- 172.21.15.32:0/3483568888 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f95380af350 0x7f953813b1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:57.010 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:57.007+0000 7f9535e00640 1 -- 172.21.15.32:0/3483568888 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f953815bb90 msgr2=0x7f9538142720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:57.010 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:57.007+0000 7f9535e00640 1 --2- 172.21.15.32:0/3483568888 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f953815bb90 0x7f9538142720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:57.010 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:57.007+0000 7f9535e00640 1 -- 172.21.15.32:0/3483568888 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9538141680 con 0x7f951c004800 2024-08-03T20:57:57.010 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:57.007+0000 7f9536800640 1 --2- 172.21.15.32:0/3483568888 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f953815bb90 0x7f9538142720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-03T20:57:57.010 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:57.007+0000 7f9535e00640 1 --2- 172.21.15.32:0/3483568888 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f951c004800 0x7f95381f36c0 secure :-1 s=READY pgs=1672 cs=0 l=1 rev1=1 crypto rx=0x7f9528011b10 tx=0x7f9528009f10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:57:57.010 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:57.007+0000 7f952e000640 1 -- 172.21.15.32:0/3483568888 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f952800fce0 con 0x7f951c004800 2024-08-03T20:57:57.010 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:57.007+0000 7f952e000640 1 -- 172.21.15.32:0/3483568888 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f952800fe80 con 0x7f951c004800 2024-08-03T20:57:57.011 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:57.007+0000 7f952e000640 1 -- 172.21.15.32:0/3483568888 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9528004280 con 0x7f951c004800 2024-08-03T20:57:57.011 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:57.007+0000 7f953ce00640 1 -- 172.21.15.32:0/3483568888 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9538141960 con 0x7f951c004800 2024-08-03T20:57:57.011 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:57.008+0000 7f953ce00640 1 -- 172.21.15.32:0/3483568888 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9538141f20 con 0x7f951c004800 2024-08-03T20:57:57.011 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:57.009+0000 7f953ce00640 1 -- 172.21.15.32:0/3483568888 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9538144f90 con 0x7f951c004800 2024-08-03T20:57:57.011 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:57.010+0000 7f952e000640 1 -- 172.21.15.32:0/3483568888 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f95280035f0 con 0x7f951c004800 2024-08-03T20:57:57.012 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:57.011+0000 7f952e000640 1 --2- 172.21.15.32:0/3483568888 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f9514077040 0x7f9514079550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:57.012 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:57.011+0000 7f952e000640 1 -- 172.21.15.32:0/3483568888 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(110..110 src has 1..110) ==== 8004+0+0 (secure 0 0 0) 0x7f9528099cb0 con 0x7f951c004800 2024-08-03T20:57:57.013 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:57.011+0000 7f952fe00640 1 --2- 172.21.15.32:0/3483568888 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f9514077040 0x7f9514079550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:57.014 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:57.012+0000 7f952fe00640 1 --2- 172.21.15.32:0/3483568888 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f9514077040 0x7f9514079550 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7f9520009b50 tx=0x7f952002d040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:57:57.016 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:57.015+0000 7f952e000640 1 -- 172.21.15.32:0/3483568888 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7f9528061ec0 con 0x7f951c004800 2024-08-03T20:57:57.261 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:57.259+0000 7f953ce00640 1 -- 172.21.15.32:0/3483568888 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7f9538160ac0 con 0x7f951c004800 2024-08-03T20:57:57.484 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:57.482+0000 7f952e000640 1 -- 172.21.15.32:0/3483568888 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v235) ==== 161+0+0 (secure 0 0 0) 0x7f9528066410 con 0x7f951c004800 2024-08-03T20:57:57.484 INFO:teuthology.orchestra.run.smithi032.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-08-03T20:57:57.487 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:57.485+0000 7f953ce00640 1 -- 172.21.15.32:0/3483568888 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f9514077040 msgr2=0x7f9514079550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:57.487 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:57.485+0000 7f953ce00640 1 --2- 172.21.15.32:0/3483568888 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f9514077040 0x7f9514079550 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7f9520009b50 tx=0x7f952002d040 comp rx=0 tx=0).stop 2024-08-03T20:57:57.488 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:57.485+0000 7f953ce00640 1 -- 172.21.15.32:0/3483568888 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f951c004800 msgr2=0x7f95381f36c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:57.488 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:57.485+0000 7f953ce00640 1 --2- 172.21.15.32:0/3483568888 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f951c004800 0x7f95381f36c0 secure :-1 s=READY pgs=1672 cs=0 l=1 rev1=1 crypto rx=0x7f9528011b10 tx=0x7f9528009f10 comp rx=0 tx=0).stop 2024-08-03T20:57:57.488 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:57.486+0000 7f953ce00640 1 -- 172.21.15.32:0/3483568888 shutdown_connections 2024-08-03T20:57:57.488 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:57.486+0000 7f953ce00640 1 --2- 172.21.15.32:0/3483568888 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f9514077040 0x7f9514079550 unknown :-1 s=CLOSED pgs=682 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:57.488 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:57.486+0000 7f953ce00640 1 --2- 172.21.15.32:0/3483568888 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f953815bb90 0x7f9538142720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:57.488 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:57.486+0000 7f953ce00640 1 --2- 172.21.15.32:0/3483568888 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f95380af350 0x7f953813b1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:57.488 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:57.486+0000 7f953ce00640 1 --2- 172.21.15.32:0/3483568888 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f951c004800 0x7f95381f36c0 unknown :-1 s=CLOSED pgs=1672 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:57.488 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:57.486+0000 7f953ce00640 1 -- 172.21.15.32:0/3483568888 >> 172.21.15.32:0/3483568888 conn(0x7f95380b1bd0 msgr2=0x7f9538140bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:57:57.488 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:57.486+0000 7f953ce00640 1 -- 172.21.15.32:0/3483568888 shutdown_connections 2024-08-03T20:57:57.488 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:57.486+0000 7f953ce00640 1 -- 172.21.15.32:0/3483568888 wait complete. 2024-08-03T20:57:57.511 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mds repaired 0 2024-08-03T20:57:57.741 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:57.739+0000 7f708ea00640 1 -- 172.21.15.32:0/367086045 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f70880d4bb0 msgr2=0x7f70880d1c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:57.741 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:57.739+0000 7f708ea00640 1 --2- 172.21.15.32:0/367086045 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f70880d4bb0 0x7f70880d1c60 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7f707c005fd0 tx=0x7f707c02a540 comp rx=0 tx=0).stop 2024-08-03T20:57:57.741 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:57.739+0000 7f708ea00640 1 -- 172.21.15.32:0/367086045 shutdown_connections 2024-08-03T20:57:57.741 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:57.739+0000 7f708ea00640 1 --2- 172.21.15.32:0/367086045 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f70880cec30 0x7f70880cf040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:57.741 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:57.739+0000 7f708ea00640 1 --2- 172.21.15.32:0/367086045 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f70880d4bb0 0x7f70880d1c60 unknown :-1 s=CLOSED pgs=519 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:57.741 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:57.739+0000 7f708ea00640 1 -- 172.21.15.32:0/367086045 >> 172.21.15.32:0/367086045 conn(0x7f70880d4090 msgr2=0x7f70880d44a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:57:57.741 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:57.739+0000 7f708ea00640 1 -- 172.21.15.32:0/367086045 shutdown_connections 2024-08-03T20:57:57.741 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:57.739+0000 7f708ea00640 1 -- 172.21.15.32:0/367086045 wait complete. 2024-08-03T20:57:57.743 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:57.742+0000 7f708ea00640 1 Processor -- start 2024-08-03T20:57:57.743 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:57.742+0000 7f708ea00640 1 -- start start 2024-08-03T20:57:57.746 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:57.743+0000 7f708ea00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f70880cec30 0x7f708815e0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:57.746 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:57.743+0000 7f708ea00640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f70880d2290 0x7f7088164b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:57.746 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:57.743+0000 7f708ea00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f70880d4bb0 0x7f7088167c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:57.746 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:57.743+0000 7f708ea00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f70880cfad0 con 0x7f70880d4bb0 2024-08-03T20:57:57.746 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:57.743+0000 7f708ea00640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f70880cf950 con 0x7f70880cec30 2024-08-03T20:57:57.746 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:57.743+0000 7f708ea00640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7f70880cfc50 con 0x7f70880d2290 2024-08-03T20:57:57.746 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:57.744+0000 7f708e000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f70880d4bb0 0x7f7088167c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:57.746 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:57.744+0000 7f708e000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f70880d4bb0 0x7f7088167c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:51438/0 (socket says 172.21.15.32:51438) 2024-08-03T20:57:57.746 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:57.744+0000 7f708e000640 1 -- 172.21.15.32:0/1705778104 learned_addr learned my addr 172.21.15.32:0/1705778104 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:57:57.746 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:57.744+0000 7f708d600640 1 --2- 172.21.15.32:0/1705778104 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f70880cec30 0x7f708815e0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:57.747 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:57.744+0000 7f708cc00640 1 --2- 172.21.15.32:0/1705778104 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f70880d2290 0x7f7088164b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:57.747 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:57.744+0000 7f708e000640 1 -- 172.21.15.32:0/1705778104 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f70880d2290 msgr2=0x7f7088164b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:57.747 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:57.744+0000 7f708e000640 1 --2- 172.21.15.32:0/1705778104 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f70880d2290 0x7f7088164b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:57.747 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:57.744+0000 7f708e000640 1 -- 172.21.15.32:0/1705778104 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f70880cec30 msgr2=0x7f708815e0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:57.747 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:57.744+0000 7f708e000640 1 --2- 172.21.15.32:0/1705778104 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f70880cec30 0x7f708815e0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:57.747 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:57.744+0000 7f708e000640 1 -- 172.21.15.32:0/1705778104 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7088165720 con 0x7f70880d4bb0 2024-08-03T20:57:57.747 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:57.744+0000 7f708cc00640 1 --2- 172.21.15.32:0/1705778104 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f70880d2290 0x7f7088164b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-03T20:57:57.747 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:57.744+0000 7f708e000640 1 --2- 172.21.15.32:0/1705778104 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f70880d4bb0 0x7f7088167c10 secure :-1 s=READY pgs=1673 cs=0 l=1 rev1=1 crypto rx=0x7f708000ea80 tx=0x7f708000ef40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:57:57.748 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:57.745+0000 7f7085600640 1 -- 172.21.15.32:0/1705778104 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7080004b20 con 0x7f70880d4bb0 2024-08-03T20:57:57.748 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:57.745+0000 7f7085600640 1 -- 172.21.15.32:0/1705778104 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f7080004ce0 con 0x7f70880d4bb0 2024-08-03T20:57:57.748 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:57.745+0000 7f708ea00640 1 -- 172.21.15.32:0/1705778104 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7088163640 con 0x7f70880d4bb0 2024-08-03T20:57:57.748 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:57.745+0000 7f7085600640 1 -- 172.21.15.32:0/1705778104 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7080005280 con 0x7f70880d4bb0 2024-08-03T20:57:57.748 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:57.745+0000 7f708ea00640 1 -- 172.21.15.32:0/1705778104 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7088163c00 con 0x7f70880d4bb0 2024-08-03T20:57:57.748 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:57.746+0000 7f708ea00640 1 -- 172.21.15.32:0/1705778104 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f70880cf040 con 0x7f70880d4bb0 2024-08-03T20:57:57.748 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:57.747+0000 7f7085600640 1 -- 172.21.15.32:0/1705778104 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f7080005420 con 0x7f70880d4bb0 2024-08-03T20:57:57.750 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:57.749+0000 7f7085600640 1 --2- 172.21.15.32:0/1705778104 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f7068077040 0x7f7068079550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:57.750 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:57.749+0000 7f7085600640 1 -- 172.21.15.32:0/1705778104 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(110..110 src has 1..110) ==== 8004+0+0 (secure 0 0 0) 0x7f7080099dc0 con 0x7f70880d4bb0 2024-08-03T20:57:57.751 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:57.749+0000 7f708d600640 1 --2- 172.21.15.32:0/1705778104 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f7068077040 0x7f7068079550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:57.752 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:57.750+0000 7f708d600640 1 --2- 172.21.15.32:0/1705778104 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f7068077040 0x7f7068079550 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7f707c033000 tx=0x7f707c004730 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:57:57.755 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:57.753+0000 7f7085600640 1 -- 172.21.15.32:0/1705778104 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7f70800626b0 con 0x7f70880d4bb0 2024-08-03T20:57:58.039 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:58.037+0000 7f708ea00640 1 -- 172.21.15.32:0/1705778104 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "mds repaired", "role": "0"} v 0) -- 0x7f708808c100 con 0x7f70880d4bb0 2024-08-03T20:57:58.497 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:58.496+0000 7f7085600640 1 -- 172.21.15.32:0/1705778104 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "mds repaired", "role": "0"}]=0 repaired: restoring rank 6:0 v236) ==== 101+0+0 (secure 0 0 0) 0x7f7080061fb0 con 0x7f70880d4bb0 2024-08-03T20:57:58.497 INFO:teuthology.orchestra.run.smithi032.stderr:repaired: restoring rank 6:0 2024-08-03T20:57:58.500 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:58.499+0000 7f708ea00640 1 -- 172.21.15.32:0/1705778104 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f7068077040 msgr2=0x7f7068079550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:58.500 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:58.499+0000 7f708ea00640 1 --2- 172.21.15.32:0/1705778104 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f7068077040 0x7f7068079550 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7f707c033000 tx=0x7f707c004730 comp rx=0 tx=0).stop 2024-08-03T20:57:58.500 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:58.499+0000 7f708ea00640 1 -- 172.21.15.32:0/1705778104 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f70880d4bb0 msgr2=0x7f7088167c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:58.501 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:58.499+0000 7f708ea00640 1 --2- 172.21.15.32:0/1705778104 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f70880d4bb0 0x7f7088167c10 secure :-1 s=READY pgs=1673 cs=0 l=1 rev1=1 crypto rx=0x7f708000ea80 tx=0x7f708000ef40 comp rx=0 tx=0).stop 2024-08-03T20:57:58.501 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:58.499+0000 7f708ea00640 1 -- 172.21.15.32:0/1705778104 shutdown_connections 2024-08-03T20:57:58.501 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:58.499+0000 7f708ea00640 1 --2- 172.21.15.32:0/1705778104 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f7068077040 0x7f7068079550 unknown :-1 s=CLOSED pgs=683 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:58.501 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:58.499+0000 7f708ea00640 1 --2- 172.21.15.32:0/1705778104 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f70880d4bb0 0x7f7088167c10 unknown :-1 s=CLOSED pgs=1673 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:58.501 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:58.499+0000 7f708ea00640 1 --2- 172.21.15.32:0/1705778104 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f70880d2290 0x7f7088164b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:58.501 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:58.499+0000 7f708ea00640 1 --2- 172.21.15.32:0/1705778104 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f70880cec30 0x7f708815e0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:58.501 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:58.499+0000 7f708ea00640 1 -- 172.21.15.32:0/1705778104 >> 172.21.15.32:0/1705778104 conn(0x7f70880d4090 msgr2=0x7f708806d5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:57:58.501 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:58.499+0000 7f708ea00640 1 -- 172.21.15.32:0/1705778104 shutdown_connections 2024-08-03T20:57:58.501 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:58.499+0000 7f708ea00640 1 -- 172.21.15.32:0/1705778104 wait complete. 2024-08-03T20:57:58.525 DEBUG:teuthology.orchestra.run.smithi032:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata import - 2024-08-03T20:57:58.668 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:58.666+0000 7f08b431ff40 1 Processor -- start 2024-08-03T20:57:58.668 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:58.667+0000 7f08b431ff40 1 -- start start 2024-08-03T20:57:58.671 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:58.668+0000 7f08b431ff40 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x555a0b753b40 0x555a0b7e28b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:58.671 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:58.668+0000 7f08b431ff40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x555a0b7c6be0 0x555a0b70e5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:58.671 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:58.668+0000 7f08b431ff40 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555a0b70eb20 0x555a0b751680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:58.671 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:58.668+0000 7f08b431ff40 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x555a0b65ce30 con 0x555a0b70eb20 2024-08-03T20:57:58.671 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:58.668+0000 7f08b431ff40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x555a0b6babf0 con 0x555a0b7c6be0 2024-08-03T20:57:58.671 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:58.668+0000 7f08b431ff40 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x555a0b7c7510 con 0x555a0b753b40 2024-08-03T20:57:58.671 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:58.668+0000 7f08ad400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555a0b70eb20 0x555a0b751680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:58.671 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:58.668+0000 7f08ad400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555a0b70eb20 0x555a0b751680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:51458/0 (socket says 172.21.15.32:51458) 2024-08-03T20:57:58.671 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:58.668+0000 7f08ad400640 1 -- 172.21.15.32:0/376348694 learned_addr learned my addr 172.21.15.32:0/376348694 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:57:58.672 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:58.668+0000 7f08a7e00640 1 --2- 172.21.15.32:0/376348694 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x555a0b7c6be0 0x555a0b70e5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:58.672 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:58.668+0000 7f08aca00640 1 --2- 172.21.15.32:0/376348694 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x555a0b753b40 0x555a0b7e28b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:58.672 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:58.670+0000 7f08ad400640 1 -- 172.21.15.32:0/376348694 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x555a0b753b40 msgr2=0x555a0b7e28b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:57:58.672 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:58.670+0000 7f08ad400640 1 --2- 172.21.15.32:0/376348694 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x555a0b753b40 0x555a0b7e28b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:58.672 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:58.670+0000 7f08ad400640 1 -- 172.21.15.32:0/376348694 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x555a0b7c6be0 msgr2=0x555a0b70e5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:57:58.672 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:58.670+0000 7f08ad400640 1 --2- 172.21.15.32:0/376348694 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x555a0b7c6be0 0x555a0b70e5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:58.672 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:58.670+0000 7f08ad400640 1 -- 172.21.15.32:0/376348694 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555a0b7e3790 con 0x555a0b70eb20 2024-08-03T20:57:58.672 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:58.670+0000 7f08ad400640 1 --2- 172.21.15.32:0/376348694 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555a0b70eb20 0x555a0b751680 secure :-1 s=READY pgs=1675 cs=0 l=1 rev1=1 crypto rx=0x7f08a8057040 tx=0x7f08a8003330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9b5585c4aa2ee4d3 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:57:58.672 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:58.670+0000 7f08a6a00640 1 -- 172.21.15.32:0/376348694 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08a8064030 con 0x555a0b70eb20 2024-08-03T20:57:58.672 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:58.670+0000 7f08a6a00640 1 -- 172.21.15.32:0/376348694 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f08a8004020 con 0x555a0b70eb20 2024-08-03T20:57:58.672 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:58.670+0000 7f08a6a00640 1 -- 172.21.15.32:0/376348694 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08a8004270 con 0x555a0b70eb20 2024-08-03T20:57:58.673 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:58.670+0000 7f08b431ff40 1 -- 172.21.15.32:0/376348694 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555a0b70eb20 msgr2=0x555a0b751680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:58.673 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:58.670+0000 7f08b431ff40 1 --2- 172.21.15.32:0/376348694 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555a0b70eb20 0x555a0b751680 secure :-1 s=READY pgs=1675 cs=0 l=1 rev1=1 crypto rx=0x7f08a8057040 tx=0x7f08a8003330 comp rx=0 tx=0).stop 2024-08-03T20:57:58.673 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:58.670+0000 7f08b431ff40 1 -- 172.21.15.32:0/376348694 shutdown_connections 2024-08-03T20:57:58.673 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:58.670+0000 7f08b431ff40 1 --2- 172.21.15.32:0/376348694 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x555a0b7c6be0 0x555a0b70e5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:58.674 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:58.670+0000 7f08b431ff40 1 --2- 172.21.15.32:0/376348694 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555a0b70eb20 0x555a0b751680 unknown :-1 s=CLOSED pgs=1675 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:58.674 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:58.670+0000 7f08b431ff40 1 --2- 172.21.15.32:0/376348694 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x555a0b753b40 0x555a0b7e28b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:58.674 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:58.670+0000 7f08b431ff40 1 -- 172.21.15.32:0/376348694 >> 172.21.15.32:0/376348694 conn(0x555a0b62eac0 msgr2=0x555a0b7e1f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:57:58.674 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:58.670+0000 7f08b431ff40 1 -- 172.21.15.32:0/376348694 shutdown_connections 2024-08-03T20:57:58.674 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:58.670+0000 7f08b431ff40 1 -- 172.21.15.32:0/376348694 wait complete. 2024-08-03T20:57:58.674 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:58.672+0000 7f08b431ff40 1 Processor -- start 2024-08-03T20:57:58.674 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:58.672+0000 7f08b431ff40 1 -- start start 2024-08-03T20:57:58.676 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:58.674+0000 7f08b431ff40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x555a0b70eb20 0x555a0b7c1220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:58.676 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:58.674+0000 7f08b431ff40 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x555a0b753b40 0x555a0b7c4520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:58.676 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:58.674+0000 7f08b431ff40 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555a0b7c6be0 0x555a0b7c55d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:58.676 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:58.674+0000 7f08b431ff40 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x555a0b7e2df0 con 0x555a0b7c6be0 2024-08-03T20:57:58.676 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:58.674+0000 7f08b431ff40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x555a0b7e3a30 con 0x555a0b70eb20 2024-08-03T20:57:58.676 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:58.674+0000 7f08b431ff40 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x555a0b7e7080 con 0x555a0b753b40 2024-08-03T20:57:58.677 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:58.674+0000 7f08ad400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555a0b7c6be0 0x555a0b7c55d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:58.677 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:58.674+0000 7f08ad400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555a0b7c6be0 0x555a0b7c55d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:51474/0 (socket says 172.21.15.32:51474) 2024-08-03T20:57:58.677 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:58.674+0000 7f08ad400640 1 -- 172.21.15.32:0/3980296818 learned_addr learned my addr 172.21.15.32:0/3980296818 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:57:58.677 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:58.674+0000 7f08a7e00640 1 --2- 172.21.15.32:0/3980296818 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x555a0b753b40 0x555a0b7c4520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:58.677 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:58.674+0000 7f08aca00640 1 --2- 172.21.15.32:0/3980296818 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x555a0b70eb20 0x555a0b7c1220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:58.677 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:58.675+0000 7f08ad400640 1 -- 172.21.15.32:0/3980296818 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x555a0b753b40 msgr2=0x555a0b7c4520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:58.677 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:58.675+0000 7f08ad400640 1 --2- 172.21.15.32:0/3980296818 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x555a0b753b40 0x555a0b7c4520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:58.677 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:58.675+0000 7f08ad400640 1 -- 172.21.15.32:0/3980296818 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x555a0b70eb20 msgr2=0x555a0b7c1220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:58.677 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:58.675+0000 7f08ad400640 1 --2- 172.21.15.32:0/3980296818 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x555a0b70eb20 0x555a0b7c1220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:58.677 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:58.675+0000 7f08ad400640 1 -- 172.21.15.32:0/3980296818 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555a0b7c4a60 con 0x555a0b7c6be0 2024-08-03T20:57:58.677 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:58.675+0000 7f08ad400640 1 --2- 172.21.15.32:0/3980296818 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555a0b7c6be0 0x555a0b7c55d0 secure :-1 s=READY pgs=1676 cs=0 l=1 rev1=1 crypto rx=0x7f08a8025dd0 tx=0x7f08a804dc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:57:58.677 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:58.675+0000 7f08a5600640 1 -- 172.21.15.32:0/3980296818 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08a8060670 con 0x555a0b7c6be0 2024-08-03T20:57:58.677 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:58.675+0000 7f08a5600640 1 -- 172.21.15.32:0/3980296818 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f08a8060900 con 0x555a0b7c6be0 2024-08-03T20:57:58.677 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:58.675+0000 7f08b431ff40 1 -- 172.21.15.32:0/3980296818 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x555a0b87f620 con 0x555a0b7c6be0 2024-08-03T20:57:58.678 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:58.675+0000 7f08a5600640 1 -- 172.21.15.32:0/3980296818 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08a8060d10 con 0x555a0b7c6be0 2024-08-03T20:57:58.678 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:58.675+0000 7f08b431ff40 1 -- 172.21.15.32:0/3980296818 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555a0b7e72d0 con 0x555a0b7c6be0 2024-08-03T20:57:58.678 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:58.677+0000 7f08a5600640 1 -- 172.21.15.32:0/3980296818 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f08a8067070 con 0x555a0b7c6be0 2024-08-03T20:57:58.679 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:58.678+0000 7f08a5600640 1 --2- 172.21.15.32:0/3980296818 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f088c0759f0 0x7f088c077f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:58.679 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:58.678+0000 7f08a5600640 1 -- 172.21.15.32:0/3980296818 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(110..110 src has 1..110) ==== 8004+0+0 (secure 0 0 0) 0x7f08a80dd8f0 con 0x555a0b7c6be0 2024-08-03T20:57:58.679 INFO:teuthology.orchestra.run.smithi032.stdout:Importing pool 2024-08-03T20:57:58.680 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:58.679+0000 7f08aca00640 1 --2- 172.21.15.32:0/3980296818 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f088c0759f0 0x7f088c077f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:58.680 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:58.679+0000 7f08b431ff40 1 --2- 172.21.15.32:0/3980296818 >> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] conn(0x555a0b87b680 0x555a0b87ba90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:58.680 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:58.679+0000 7f08b431ff40 1 -- 172.21.15.32:0/3980296818 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:1 12.10 12:09401a81:::601.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110) -- 0x555a0b887950 con 0x555a0b87b680 2024-08-03T20:57:58.680 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:58.679+0000 7f08a7e00640 1 --2- 172.21.15.32:0/3980296818 >> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] conn(0x555a0b87b680 0x555a0b87ba90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:58.680 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:58.679+0000 7f08aca00640 1 --2- 172.21.15.32:0/3980296818 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f088c0759f0 0x7f088c077f00 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 crypto rx=0x7f08a000c720 tx=0x7f08a000e040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:57:58.680 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:58.679+0000 7f08a7e00640 1 --2- 172.21.15.32:0/3980296818 >> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] conn(0x555a0b87b680 0x555a0b87ba90 crc :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:57:58.682 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:58.681+0000 7f08a7e00640 1 -- 172.21.15.32:0/3980296818 <== osd.0 v2:172.21.15.32:6816/3066156116 1 ==== osd_op_reply(1 601.00000000 [create] v110'753 uv727 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f0898011640 con 0x555a0b87b680 2024-08-03T20:57:58.682 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:58.681+0000 7f08b431ff40 1 -- 172.21.15.32:0/3980296818 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:2 12.10 12:09401a81:::601.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e110) -- 0x555a0b887cf0 con 0x555a0b87b680 2024-08-03T20:57:58.684 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:58.684+0000 7f08a7e00640 1 -- 172.21.15.32:0/3980296818 <== osd.0 v2:172.21.15.32:6816/3066156116 2 ==== osd_op_reply(2 601.00000000 [delete] v110'754 uv754 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f0898011640 con 0x555a0b87b680 2024-08-03T20:57:58.684 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:58.684+0000 7f08b431ff40 1 -- 172.21.15.32:0/3980296818 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:3 12.10 12:09401a81:::601.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110) -- 0x555a0b888090 con 0x555a0b87b680 2024-08-03T20:57:58.686 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:58.685+0000 7f08a7e00640 1 -- 172.21.15.32:0/3980296818 <== osd.0 v2:172.21.15.32:6816/3066156116 3 ==== osd_op_reply(3 601.00000000 [create] v110'755 uv755 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f0898011640 con 0x555a0b87b680 2024-08-03T20:57:58.686 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::601.00000000:head# 2024-08-03T20:57:58.686 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:58.685+0000 7f08b431ff40 1 -- 172.21.15.32:0/3980296818 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:4 12.10 12:09401a81:::601.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110) -- 0x555a0b889440 con 0x555a0b87b680 2024-08-03T20:57:58.688 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:58.687+0000 7f08a7e00640 1 -- 172.21.15.32:0/3980296818 <== osd.0 v2:172.21.15.32:6816/3066156116 4 ==== osd_op_reply(4 601.00000000 [setxattr (30)] v110'756 uv756 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f0898011640 con 0x555a0b87b680 2024-08-03T20:57:58.688 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:58.687+0000 7f08b431ff40 1 -- 172.21.15.32:0/3980296818 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:5 12.10 12:09401a81:::601.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110) -- 0x555a0b8897e0 con 0x555a0b87b680 2024-08-03T20:57:58.689 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:58.689+0000 7f08a7e00640 1 -- 172.21.15.32:0/3980296818 <== osd.0 v2:172.21.15.32:6816/3066156116 5 ==== osd_op_reply(5 601.00000000 [setxattr (62)] v110'757 uv757 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f0898011640 con 0x555a0b87b680 2024-08-03T20:57:58.689 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:58.689+0000 7f08b431ff40 1 -- 172.21.15.32:0/3980296818 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:6 12.10 12:09401a81:::601.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110) -- 0x555a0b888430 con 0x555a0b87b680 2024-08-03T20:57:58.691 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:58.689+0000 7f08a7e00640 1 -- 172.21.15.32:0/3980296818 <== osd.0 v2:172.21.15.32:6816/3066156116 6 ==== osd_op_reply(6 601.00000000 [omap-set-header] v110'758 uv758 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f0898011640 con 0x555a0b87b680 2024-08-03T20:57:58.691 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:58.689+0000 7f08b431ff40 1 -- 172.21.15.32:0/3980296818 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:7 12.10 12:0bd6d154:::602.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110) -- 0x555a0b888820 con 0x555a0b87b680 2024-08-03T20:57:58.692 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:58.691+0000 7f08a7e00640 1 -- 172.21.15.32:0/3980296818 <== osd.0 v2:172.21.15.32:6816/3066156116 7 ==== osd_op_reply(7 602.00000000 [create] v110'759 uv733 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f0898011640 con 0x555a0b87b680 2024-08-03T20:57:58.693 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:58.691+0000 7f08b431ff40 1 -- 172.21.15.32:0/3980296818 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:8 12.10 12:0bd6d154:::602.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e110) -- 0x555a0b888c10 con 0x555a0b87b680 2024-08-03T20:57:58.694 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:58.693+0000 7f08a7e00640 1 -- 172.21.15.32:0/3980296818 <== osd.0 v2:172.21.15.32:6816/3066156116 8 ==== osd_op_reply(8 602.00000000 [delete] v110'760 uv760 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f0898011640 con 0x555a0b87b680 2024-08-03T20:57:58.694 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:58.693+0000 7f08b431ff40 1 -- 172.21.15.32:0/3980296818 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:9 12.10 12:0bd6d154:::602.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110) -- 0x555a0b888c10 con 0x555a0b87b680 2024-08-03T20:57:58.695 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:58.694+0000 7f08a7e00640 1 -- 172.21.15.32:0/3980296818 <== osd.0 v2:172.21.15.32:6816/3066156116 9 ==== osd_op_reply(9 602.00000000 [create] v110'761 uv761 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f0898011640 con 0x555a0b87b680 2024-08-03T20:57:58.696 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::602.00000000:head# 2024-08-03T20:57:58.696 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:58.694+0000 7f08b431ff40 1 -- 172.21.15.32:0/3980296818 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:10 12.10 12:0bd6d154:::602.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110) -- 0x555a0b888bc0 con 0x555a0b87b680 2024-08-03T20:57:58.697 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:58.696+0000 7f08a7e00640 1 -- 172.21.15.32:0/3980296818 <== osd.0 v2:172.21.15.32:6816/3066156116 10 ==== osd_op_reply(10 602.00000000 [setxattr (30)] v110'762 uv762 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f0898011640 con 0x555a0b87b680 2024-08-03T20:57:58.697 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:58.696+0000 7f08b431ff40 1 -- 172.21.15.32:0/3980296818 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:11 12.10 12:0bd6d154:::602.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110) -- 0x555a0b888bc0 con 0x555a0b87b680 2024-08-03T20:57:58.698 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:58.697+0000 7f08a7e00640 1 -- 172.21.15.32:0/3980296818 <== osd.0 v2:172.21.15.32:6816/3066156116 11 ==== osd_op_reply(11 602.00000000 [setxattr (62)] v110'763 uv763 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f0898011640 con 0x555a0b87b680 2024-08-03T20:57:58.699 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:58.697+0000 7f08b431ff40 1 -- 172.21.15.32:0/3980296818 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:12 12.10 12:0bd6d154:::602.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110) -- 0x555a0b888bc0 con 0x555a0b87b680 2024-08-03T20:57:58.700 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:58.700+0000 7f08a7e00640 1 -- 172.21.15.32:0/3980296818 <== osd.0 v2:172.21.15.32:6816/3066156116 12 ==== osd_op_reply(12 602.00000000 [omap-set-header] v110'764 uv764 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f0898011640 con 0x555a0b87b680 2024-08-03T20:57:58.700 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:58.700+0000 7f08b431ff40 1 -- 172.21.15.32:0/3980296818 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:13 12.10 12:0cf6ad0f:::10000000000.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110) -- 0x555a0b888bc0 con 0x555a0b87b680 2024-08-03T20:57:58.702 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:58.700+0000 7f08a7e00640 1 -- 172.21.15.32:0/3980296818 <== osd.0 v2:172.21.15.32:6816/3066156116 13 ==== osd_op_reply(13 10000000000.00000000 [create] v110'765 uv740 ondisk = -17 ((17) File exists)) ==== 164+0+0 (crc 0 0 0) 0x7f0898011640 con 0x555a0b87b680 2024-08-03T20:57:58.702 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:58.701+0000 7f08b431ff40 1 -- 172.21.15.32:0/3980296818 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:14 12.10 12:0cf6ad0f:::10000000000.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e110) -- 0x555a0b888bc0 con 0x555a0b87b680 2024-08-03T20:57:58.703 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:58.703+0000 7f08a7e00640 1 -- 172.21.15.32:0/3980296818 <== osd.0 v2:172.21.15.32:6816/3066156116 14 ==== osd_op_reply(14 10000000000.00000000 [delete] v110'766 uv766 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f0898011640 con 0x555a0b87b680 2024-08-03T20:57:58.704 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:58.703+0000 7f08b431ff40 1 -- 172.21.15.32:0/3980296818 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:15 12.10 12:0cf6ad0f:::10000000000.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110) -- 0x555a0b888bc0 con 0x555a0b87b680 2024-08-03T20:57:58.705 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:58.704+0000 7f08a7e00640 1 -- 172.21.15.32:0/3980296818 <== osd.0 v2:172.21.15.32:6816/3066156116 15 ==== osd_op_reply(15 10000000000.00000000 [create] v110'767 uv767 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f0898011640 con 0x555a0b87b680 2024-08-03T20:57:58.705 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::10000000000.00000000:head# 2024-08-03T20:57:58.706 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:58.704+0000 7f08b431ff40 1 -- 172.21.15.32:0/3980296818 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:16 12.10 12:0cf6ad0f:::10000000000.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110) -- 0x555a0b888bc0 con 0x555a0b87b680 2024-08-03T20:57:58.707 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:58.705+0000 7f08a7e00640 1 -- 172.21.15.32:0/3980296818 <== osd.0 v2:172.21.15.32:6816/3066156116 16 ==== osd_op_reply(16 10000000000.00000000 [setxattr (30)] v110'768 uv768 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f0898011640 con 0x555a0b87b680 2024-08-03T20:57:58.707 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:58.705+0000 7f08b431ff40 1 -- 172.21.15.32:0/3980296818 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:17 12.10 12:0cf6ad0f:::10000000000.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110) -- 0x555a0b888bc0 con 0x555a0b87b680 2024-08-03T20:57:58.708 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:58.708+0000 7f08a7e00640 1 -- 172.21.15.32:0/3980296818 <== osd.0 v2:172.21.15.32:6816/3066156116 17 ==== osd_op_reply(17 10000000000.00000000 [setxattr (62)] v110'769 uv769 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f0898011640 con 0x555a0b87b680 2024-08-03T20:57:58.708 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:58.708+0000 7f08b431ff40 1 -- 172.21.15.32:0/3980296818 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:18 12.10 12:0cf6ad0f:::10000000000.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110) -- 0x555a0b888bc0 con 0x555a0b87b680 2024-08-03T20:57:58.710 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:58.708+0000 7f08a7e00640 1 -- 172.21.15.32:0/3980296818 <== osd.0 v2:172.21.15.32:6816/3066156116 18 ==== osd_op_reply(18 10000000000.00000000 [omap-set-header] v110'770 uv770 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f0898011640 con 0x555a0b87b680 2024-08-03T20:57:58.710 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:58.708+0000 7f08b431ff40 1 -- 172.21.15.32:0/3980296818 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:19 12.10 12:0cf6ad0f:::10000000000.00000000:head [omap-set-vals in=593b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110) -- 0x555a0b888bc0 con 0x555a0b87b680 2024-08-03T20:57:58.711 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:58.710+0000 7f08a7e00640 1 -- 172.21.15.32:0/3980296818 <== osd.0 v2:172.21.15.32:6816/3066156116 19 ==== osd_op_reply(19 10000000000.00000000 [omap-set-vals] v110'771 uv771 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f0898011640 con 0x555a0b87b680 2024-08-03T20:57:58.712 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:58.710+0000 7f08b431ff40 1 -- 172.21.15.32:0/3980296818 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:20 12.10 12:0d82a743:::600.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110) -- 0x555a0b888bc0 con 0x555a0b87b680 2024-08-03T20:57:58.713 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:58.713+0000 7f08a7e00640 1 -- 172.21.15.32:0/3980296818 <== osd.0 v2:172.21.15.32:6816/3066156116 20 ==== osd_op_reply(20 600.00000000 [create] v110'772 uv746 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f0898011640 con 0x555a0b87b680 2024-08-03T20:57:58.713 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:58.713+0000 7f08b431ff40 1 -- 172.21.15.32:0/3980296818 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:21 12.10 12:0d82a743:::600.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e110) -- 0x555a0b888bc0 con 0x555a0b87b680 2024-08-03T20:57:58.715 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:58.714+0000 7f08a7e00640 1 -- 172.21.15.32:0/3980296818 <== osd.0 v2:172.21.15.32:6816/3066156116 21 ==== osd_op_reply(21 600.00000000 [delete] v110'773 uv773 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f0898011640 con 0x555a0b87b680 2024-08-03T20:57:58.715 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:58.714+0000 7f08b431ff40 1 -- 172.21.15.32:0/3980296818 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:22 12.10 12:0d82a743:::600.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110) -- 0x555a0b888bc0 con 0x555a0b87b680 2024-08-03T20:57:58.716 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:58.716+0000 7f08a7e00640 1 -- 172.21.15.32:0/3980296818 <== osd.0 v2:172.21.15.32:6816/3066156116 22 ==== osd_op_reply(22 600.00000000 [create] v110'774 uv774 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f0898011640 con 0x555a0b87b680 2024-08-03T20:57:58.717 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::600.00000000:head# 2024-08-03T20:57:58.717 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:58.716+0000 7f08b431ff40 1 -- 172.21.15.32:0/3980296818 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:23 12.10 12:0d82a743:::600.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110) -- 0x555a0b888bc0 con 0x555a0b87b680 2024-08-03T20:57:58.718 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:58.718+0000 7f08a7e00640 1 -- 172.21.15.32:0/3980296818 <== osd.0 v2:172.21.15.32:6816/3066156116 23 ==== osd_op_reply(23 600.00000000 [setxattr (30)] v110'775 uv775 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f0898011640 con 0x555a0b87b680 2024-08-03T20:57:58.718 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:58.718+0000 7f08b431ff40 1 -- 172.21.15.32:0/3980296818 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:24 12.10 12:0d82a743:::600.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110) -- 0x555a0b888bc0 con 0x555a0b87b680 2024-08-03T20:57:58.720 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:58.718+0000 7f08a7e00640 1 -- 172.21.15.32:0/3980296818 <== osd.0 v2:172.21.15.32:6816/3066156116 24 ==== osd_op_reply(24 600.00000000 [setxattr (62)] v110'776 uv776 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f0898011640 con 0x555a0b87b680 2024-08-03T20:57:58.720 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:58.718+0000 7f08b431ff40 1 -- 172.21.15.32:0/3980296818 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:25 12.10 12:0d82a743:::600.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110) -- 0x555a0b888bc0 con 0x555a0b87b680 2024-08-03T20:57:58.721 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:58.721+0000 7f08a7e00640 1 -- 172.21.15.32:0/3980296818 <== osd.0 v2:172.21.15.32:6816/3066156116 25 ==== osd_op_reply(25 600.00000000 [omap-set-header] v110'777 uv777 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f0898011640 con 0x555a0b87b680 2024-08-03T20:57:58.721 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:58.721+0000 7f08b431ff40 1 -- 172.21.15.32:0/3980296818 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:26 12.10 12:0d89b25e:::603.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110) -- 0x555a0b888bc0 con 0x555a0b87b680 2024-08-03T20:57:58.723 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:58.723+0000 7f08a7e00640 1 -- 172.21.15.32:0/3980296818 <== osd.0 v2:172.21.15.32:6816/3066156116 26 ==== osd_op_reply(26 603.00000000 [create] v110'778 uv752 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f0898011640 con 0x555a0b87b680 2024-08-03T20:57:58.723 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:58.723+0000 7f08b431ff40 1 -- 172.21.15.32:0/3980296818 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:27 12.10 12:0d89b25e:::603.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e110) -- 0x555a0b888bc0 con 0x555a0b87b680 2024-08-03T20:57:58.725 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:58.725+0000 7f08a7e00640 1 -- 172.21.15.32:0/3980296818 <== osd.0 v2:172.21.15.32:6816/3066156116 27 ==== osd_op_reply(27 603.00000000 [delete] v110'779 uv779 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f0898011640 con 0x555a0b87b680 2024-08-03T20:57:58.725 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:58.725+0000 7f08b431ff40 1 -- 172.21.15.32:0/3980296818 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:28 12.10 12:0d89b25e:::603.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110) -- 0x555a0b888bc0 con 0x555a0b87b680 2024-08-03T20:57:58.726 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:58.726+0000 7f08a7e00640 1 -- 172.21.15.32:0/3980296818 <== osd.0 v2:172.21.15.32:6816/3066156116 28 ==== osd_op_reply(28 603.00000000 [create] v110'780 uv780 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f0898011640 con 0x555a0b87b680 2024-08-03T20:57:58.727 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::603.00000000:head# 2024-08-03T20:57:58.727 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:58.726+0000 7f08b431ff40 1 -- 172.21.15.32:0/3980296818 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:29 12.10 12:0d89b25e:::603.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110) -- 0x555a0b888bc0 con 0x555a0b87b680 2024-08-03T20:57:58.728 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:58.727+0000 7f08a7e00640 1 -- 172.21.15.32:0/3980296818 <== osd.0 v2:172.21.15.32:6816/3066156116 29 ==== osd_op_reply(29 603.00000000 [setxattr (30)] v110'781 uv781 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f0898011640 con 0x555a0b87b680 2024-08-03T20:57:58.728 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:58.727+0000 7f08b431ff40 1 -- 172.21.15.32:0/3980296818 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:30 12.10 12:0d89b25e:::603.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110) -- 0x555a0b888bc0 con 0x555a0b87b680 2024-08-03T20:57:58.730 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:58.729+0000 7f08a7e00640 1 -- 172.21.15.32:0/3980296818 <== osd.0 v2:172.21.15.32:6816/3066156116 30 ==== osd_op_reply(30 603.00000000 [setxattr (62)] v110'782 uv782 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f0898011640 con 0x555a0b87b680 2024-08-03T20:57:58.730 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:58.729+0000 7f08b431ff40 1 -- 172.21.15.32:0/3980296818 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:31 12.10 12:0d89b25e:::603.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110) -- 0x555a0b888bc0 con 0x555a0b87b680 2024-08-03T20:57:58.731 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:58.731+0000 7f08a7e00640 1 -- 172.21.15.32:0/3980296818 <== osd.0 v2:172.21.15.32:6816/3066156116 31 ==== osd_op_reply(31 603.00000000 [omap-set-header] v110'783 uv783 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f0898011640 con 0x555a0b87b680 2024-08-03T20:57:58.731 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:58.731+0000 7f08b431ff40 1 --2- 172.21.15.32:0/3980296818 >> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] conn(0x555a0b888fa0 0x555a0b88c0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:58.731 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:58.731+0000 7f08b431ff40 1 -- 172.21.15.32:0/3980296818 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:32 12.14 12:287034c4:::1.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110) -- 0x555a0b888bc0 con 0x555a0b888fa0 2024-08-03T20:57:58.732 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:58.731+0000 7f08aca00640 1 --2- 172.21.15.32:0/3980296818 >> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] conn(0x555a0b888fa0 0x555a0b88c0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:58.732 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:58.732+0000 7f08aca00640 1 --2- 172.21.15.32:0/3980296818 >> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] conn(0x555a0b888fa0 0x555a0b88c0a0 crc :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:57:58.734 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:58.734+0000 7f08aca00640 1 -- 172.21.15.32:0/3980296818 <== osd.5 v2:172.21.15.118:6810/2724891770 1 ==== osd_op_reply(32 1.00000000.inode [create] v110'549 uv531 ondisk = -17 ((17) File exists)) ==== 160+0+0 (crc 0 0 0) 0x7f08a000a4e0 con 0x555a0b888fa0 2024-08-03T20:57:58.735 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:58.734+0000 7f08b431ff40 1 -- 172.21.15.32:0/3980296818 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:33 12.14 12:287034c4:::1.00000000.inode:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e110) -- 0x555a0b88c5e0 con 0x555a0b888fa0 2024-08-03T20:57:58.736 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:58.735+0000 7f08aca00640 1 -- 172.21.15.32:0/3980296818 <== osd.5 v2:172.21.15.118:6810/2724891770 2 ==== osd_op_reply(33 1.00000000.inode [delete] v110'550 uv550 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f08a000a4e0 con 0x555a0b888fa0 2024-08-03T20:57:58.736 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:58.735+0000 7f08b431ff40 1 -- 172.21.15.32:0/3980296818 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:34 12.14 12:287034c4:::1.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110) -- 0x555a0b88c980 con 0x555a0b888fa0 2024-08-03T20:57:58.737 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::1.00000000.inode:head# 2024-08-03T20:57:58.737 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:58.737+0000 7f08aca00640 1 -- 172.21.15.32:0/3980296818 <== osd.5 v2:172.21.15.118:6810/2724891770 3 ==== osd_op_reply(34 1.00000000.inode [create] v110'551 uv551 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f08a000a4e0 con 0x555a0b888fa0 2024-08-03T20:57:58.737 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:58.737+0000 7f08b431ff40 1 -- 172.21.15.32:0/3980296818 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:35 12.14 12:287034c4:::1.00000000.inode:head [write 0~582 in=582b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110) -- 0x555a0b88cd20 con 0x555a0b888fa0 2024-08-03T20:57:58.739 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:58.737+0000 7f08aca00640 1 -- 172.21.15.32:0/3980296818 <== osd.5 v2:172.21.15.118:6810/2724891770 4 ==== osd_op_reply(35 1.00000000.inode [write 0~582] v110'552 uv552 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f08a000a4e0 con 0x555a0b888fa0 2024-08-03T20:57:58.739 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:58.737+0000 7f08b431ff40 1 -- 172.21.15.32:0/3980296818 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:36 12.14 12:287034c4:::1.00000000.inode:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110) -- 0x555a0b88d0c0 con 0x555a0b888fa0 2024-08-03T20:57:58.740 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:58.740+0000 7f08aca00640 1 -- 172.21.15.32:0/3980296818 <== osd.5 v2:172.21.15.118:6810/2724891770 5 ==== osd_op_reply(36 1.00000000.inode [omap-set-header] v110'553 uv553 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f08a000a4e0 con 0x555a0b888fa0 2024-08-03T20:57:58.740 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:58.740+0000 7f08b431ff40 1 -- 172.21.15.32:0/3980296818 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:37 12.14 12:292cf221:::200.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110) -- 0x555a0b88d460 con 0x555a0b888fa0 2024-08-03T20:57:58.742 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:58.741+0000 7f08aca00640 1 -- 172.21.15.32:0/3980296818 <== osd.5 v2:172.21.15.118:6810/2724891770 6 ==== osd_op_reply(37 200.00000000 [create] v110'554 uv548 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f08a000a4e0 con 0x555a0b888fa0 2024-08-03T20:57:58.742 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:58.741+0000 7f08b431ff40 1 -- 172.21.15.32:0/3980296818 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:38 12.14 12:292cf221:::200.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e110) -- 0x555a0b88d800 con 0x555a0b888fa0 2024-08-03T20:57:58.743 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:58.743+0000 7f08aca00640 1 -- 172.21.15.32:0/3980296818 <== osd.5 v2:172.21.15.118:6810/2724891770 7 ==== osd_op_reply(38 200.00000000 [delete] v110'555 uv555 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f08a000a4e0 con 0x555a0b888fa0 2024-08-03T20:57:58.743 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:58.743+0000 7f08b431ff40 1 -- 172.21.15.32:0/3980296818 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:39 12.14 12:292cf221:::200.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110) -- 0x555a0b88dba0 con 0x555a0b888fa0 2024-08-03T20:57:58.745 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::200.00000000:head# 2024-08-03T20:57:58.745 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:58.744+0000 7f08aca00640 1 -- 172.21.15.32:0/3980296818 <== osd.5 v2:172.21.15.118:6810/2724891770 8 ==== osd_op_reply(39 200.00000000 [create] v110'556 uv556 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f08a000a4e0 con 0x555a0b888fa0 2024-08-03T20:57:58.745 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:58.744+0000 7f08b431ff40 1 -- 172.21.15.32:0/3980296818 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:40 12.14 12:292cf221:::200.00000000:head [write 0~90 in=90b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110) -- 0x555a0b7c5bf0 con 0x555a0b888fa0 2024-08-03T20:57:58.746 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:58.746+0000 7f08aca00640 1 -- 172.21.15.32:0/3980296818 <== osd.5 v2:172.21.15.118:6810/2724891770 9 ==== osd_op_reply(40 200.00000000 [write 0~90] v110'557 uv557 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f08a000a4e0 con 0x555a0b888fa0 2024-08-03T20:57:58.746 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:58.746+0000 7f08b431ff40 1 -- 172.21.15.32:0/3980296818 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:41 12.14 12:292cf221:::200.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110) -- 0x555a0b7c5bf0 con 0x555a0b888fa0 2024-08-03T20:57:58.748 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:58.747+0000 7f08aca00640 1 -- 172.21.15.32:0/3980296818 <== osd.5 v2:172.21.15.118:6810/2724891770 10 ==== osd_op_reply(41 200.00000000 [omap-set-header] v110'558 uv558 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f08a000a4e0 con 0x555a0b888fa0 2024-08-03T20:57:58.748 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:58.747+0000 7f08b431ff40 1 -- 172.21.15.32:0/3980296818 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:42 12.14 12:2e2fa760:::200.00000001:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110) -- 0x555a0b7c5bf0 con 0x555a0b888fa0 2024-08-03T20:57:58.749 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:58.749+0000 7f08aca00640 1 -- 172.21.15.32:0/3980296818 <== osd.5 v2:172.21.15.118:6810/2724891770 11 ==== osd_op_reply(42 200.00000001 [create] v110'559 uv541 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f08a000a4e0 con 0x555a0b888fa0 2024-08-03T20:57:58.749 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:58.749+0000 7f08b431ff40 1 -- 172.21.15.32:0/3980296818 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:43 12.14 12:2e2fa760:::200.00000001:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e110) -- 0x555a0b7c5bf0 con 0x555a0b888fa0 2024-08-03T20:57:58.751 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:58.751+0000 7f08aca00640 1 -- 172.21.15.32:0/3980296818 <== osd.5 v2:172.21.15.118:6810/2724891770 12 ==== osd_op_reply(43 200.00000001 [delete] v110'560 uv560 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f08a000a4e0 con 0x555a0b888fa0 2024-08-03T20:57:58.751 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:58.751+0000 7f08b431ff40 1 -- 172.21.15.32:0/3980296818 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:44 12.14 12:2e2fa760:::200.00000001:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110) -- 0x555a0b7c5bf0 con 0x555a0b888fa0 2024-08-03T20:57:58.752 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::200.00000001:head# 2024-08-03T20:57:58.753 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:58.752+0000 7f08aca00640 1 -- 172.21.15.32:0/3980296818 <== osd.5 v2:172.21.15.118:6810/2724891770 13 ==== osd_op_reply(44 200.00000001 [create] v110'561 uv561 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f08a000a4e0 con 0x555a0b888fa0 2024-08-03T20:57:58.753 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:58.752+0000 7f08b431ff40 1 -- 172.21.15.32:0/3980296818 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:45 12.14 12:2e2fa760:::200.00000001:head [write 0~18118 in=18118b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110) -- 0x555a0b7c5bf0 con 0x555a0b888fa0 2024-08-03T20:57:58.754 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:58.754+0000 7f08aca00640 1 -- 172.21.15.32:0/3980296818 <== osd.5 v2:172.21.15.118:6810/2724891770 14 ==== osd_op_reply(45 200.00000001 [write 0~18118] v110'562 uv562 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f08a000a4e0 con 0x555a0b888fa0 2024-08-03T20:57:58.754 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:58.754+0000 7f08b431ff40 1 -- 172.21.15.32:0/3980296818 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:46 12.14 12:2e2fa760:::200.00000001:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110) -- 0x555a0b7c5bf0 con 0x555a0b888fa0 2024-08-03T20:57:58.756 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:58.755+0000 7f08aca00640 1 -- 172.21.15.32:0/3980296818 <== osd.5 v2:172.21.15.118:6810/2724891770 15 ==== osd_op_reply(46 200.00000001 [omap-set-header] v110'563 uv563 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f08a000a4e0 con 0x555a0b888fa0 2024-08-03T20:57:58.756 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:58.756+0000 7f08b431ff40 1 -- 172.21.15.32:0/3980296818 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:47 12.14 12:2f55791f:::606.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110) -- 0x555a0b7c5bf0 con 0x555a0b888fa0 2024-08-03T20:57:58.758 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:58.757+0000 7f08aca00640 1 -- 172.21.15.32:0/3980296818 <== osd.5 v2:172.21.15.118:6810/2724891770 16 ==== osd_op_reply(47 606.00000000 [create] v110'564 uv547 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f08a000a4e0 con 0x555a0b888fa0 2024-08-03T20:57:58.758 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:58.757+0000 7f08b431ff40 1 -- 172.21.15.32:0/3980296818 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:48 12.14 12:2f55791f:::606.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e110) -- 0x555a0b7c5bf0 con 0x555a0b888fa0 2024-08-03T20:57:58.759 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:58.759+0000 7f08aca00640 1 -- 172.21.15.32:0/3980296818 <== osd.5 v2:172.21.15.118:6810/2724891770 17 ==== osd_op_reply(48 606.00000000 [delete] v110'565 uv565 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f08a000a4e0 con 0x555a0b888fa0 2024-08-03T20:57:58.759 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:58.759+0000 7f08b431ff40 1 -- 172.21.15.32:0/3980296818 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:49 12.14 12:2f55791f:::606.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110) -- 0x555a0b7c5bf0 con 0x555a0b888fa0 2024-08-03T20:57:58.760 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::606.00000000:head# 2024-08-03T20:57:58.761 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:58.760+0000 7f08aca00640 1 -- 172.21.15.32:0/3980296818 <== osd.5 v2:172.21.15.118:6810/2724891770 18 ==== osd_op_reply(49 606.00000000 [create] v110'566 uv566 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f08a000a4e0 con 0x555a0b888fa0 2024-08-03T20:57:58.761 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:58.760+0000 7f08b431ff40 1 -- 172.21.15.32:0/3980296818 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:50 12.14 12:2f55791f:::606.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110) -- 0x555a0b7c5bf0 con 0x555a0b888fa0 2024-08-03T20:57:58.762 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:58.761+0000 7f08aca00640 1 -- 172.21.15.32:0/3980296818 <== osd.5 v2:172.21.15.118:6810/2724891770 19 ==== osd_op_reply(50 606.00000000 [setxattr (30)] v110'567 uv567 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f08a000a4e0 con 0x555a0b888fa0 2024-08-03T20:57:58.762 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:58.761+0000 7f08b431ff40 1 -- 172.21.15.32:0/3980296818 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:51 12.14 12:2f55791f:::606.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110) -- 0x555a0b7c5bf0 con 0x555a0b888fa0 2024-08-03T20:57:58.763 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:58.763+0000 7f08aca00640 1 -- 172.21.15.32:0/3980296818 <== osd.5 v2:172.21.15.118:6810/2724891770 20 ==== osd_op_reply(51 606.00000000 [setxattr (62)] v110'568 uv568 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f08a000a4e0 con 0x555a0b888fa0 2024-08-03T20:57:58.763 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:58.763+0000 7f08b431ff40 1 -- 172.21.15.32:0/3980296818 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:52 12.14 12:2f55791f:::606.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110) -- 0x555a0b7c5bf0 con 0x555a0b888fa0 2024-08-03T20:57:58.765 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:58.764+0000 7f08aca00640 1 -- 172.21.15.32:0/3980296818 <== osd.5 v2:172.21.15.118:6810/2724891770 21 ==== osd_op_reply(52 606.00000000 [omap-set-header] v110'569 uv569 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f08a000a4e0 con 0x555a0b888fa0 2024-08-03T20:57:58.765 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:58.764+0000 7f08b431ff40 1 -- 172.21.15.32:0/3980296818 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:53 12.1c 12:3ed09add:::607.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110) -- 0x555a0b7c5bf0 con 0x555a0b87b680 2024-08-03T20:57:58.766 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:58.766+0000 7f08a7e00640 1 -- 172.21.15.32:0/3980296818 <== osd.0 v2:172.21.15.32:6816/3066156116 32 ==== osd_op_reply(53 607.00000000 [create] v110'147 uv146 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f0898011640 con 0x555a0b87b680 2024-08-03T20:57:58.766 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:58.766+0000 7f08b431ff40 1 -- 172.21.15.32:0/3980296818 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:54 12.1c 12:3ed09add:::607.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e110) -- 0x555a0b7c5bf0 con 0x555a0b87b680 2024-08-03T20:57:58.768 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:58.767+0000 7f08a7e00640 1 -- 172.21.15.32:0/3980296818 <== osd.0 v2:172.21.15.32:6816/3066156116 33 ==== osd_op_reply(54 607.00000000 [delete] v110'148 uv148 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f0898011640 con 0x555a0b87b680 2024-08-03T20:57:58.768 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:58.767+0000 7f08b431ff40 1 -- 172.21.15.32:0/3980296818 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:55 12.1c 12:3ed09add:::607.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110) -- 0x555a0b7c5bf0 con 0x555a0b87b680 2024-08-03T20:57:58.769 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:58.768+0000 7f08a7e00640 1 -- 172.21.15.32:0/3980296818 <== osd.0 v2:172.21.15.32:6816/3066156116 34 ==== osd_op_reply(55 607.00000000 [create] v110'149 uv149 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f0898011640 con 0x555a0b87b680 2024-08-03T20:57:58.769 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::607.00000000:head# 2024-08-03T20:57:58.769 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:58.769+0000 7f08b431ff40 1 -- 172.21.15.32:0/3980296818 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:56 12.1c 12:3ed09add:::607.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110) -- 0x555a0b7c5bf0 con 0x555a0b87b680 2024-08-03T20:57:58.770 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:58.770+0000 7f08a7e00640 1 -- 172.21.15.32:0/3980296818 <== osd.0 v2:172.21.15.32:6816/3066156116 35 ==== osd_op_reply(56 607.00000000 [setxattr (30)] v110'150 uv150 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f0898011640 con 0x555a0b87b680 2024-08-03T20:57:58.770 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:58.770+0000 7f08b431ff40 1 -- 172.21.15.32:0/3980296818 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:57 12.1c 12:3ed09add:::607.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110) -- 0x555a0b7c5bf0 con 0x555a0b87b680 2024-08-03T20:57:58.771 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:58.771+0000 7f08a7e00640 1 -- 172.21.15.32:0/3980296818 <== osd.0 v2:172.21.15.32:6816/3066156116 36 ==== osd_op_reply(57 607.00000000 [setxattr (62)] v110'151 uv151 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f0898011640 con 0x555a0b87b680 2024-08-03T20:57:58.771 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:58.771+0000 7f08b431ff40 1 -- 172.21.15.32:0/3980296818 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:58 12.1c 12:3ed09add:::607.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110) -- 0x555a0b7c5bf0 con 0x555a0b87b680 2024-08-03T20:57:58.773 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:58.772+0000 7f08a7e00640 1 -- 172.21.15.32:0/3980296818 <== osd.0 v2:172.21.15.32:6816/3066156116 37 ==== osd_op_reply(58 607.00000000 [omap-set-header] v110'152 uv152 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f0898011640 con 0x555a0b87b680 2024-08-03T20:57:58.773 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:58.772+0000 7f08b431ff40 1 --2- 172.21.15.32:0/3980296818 >> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] conn(0x555a0b88e560 0x555a0b890980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:58.773 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:58.772+0000 7f08b431ff40 1 -- 172.21.15.32:0/3980296818 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:59 12.6 12:654134d2:::mds0_openfiles.0:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110) -- 0x555a0b890ec0 con 0x555a0b88e560 2024-08-03T20:57:58.773 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:58.773+0000 7f08a7e00640 1 --2- 172.21.15.32:0/3980296818 >> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] conn(0x555a0b88e560 0x555a0b890980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:58.774 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:58.773+0000 7f08a7e00640 1 --2- 172.21.15.32:0/3980296818 >> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] conn(0x555a0b88e560 0x555a0b890980 crc :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:57:58.775 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:58.775+0000 7f08a7e00640 1 -- 172.21.15.32:0/3980296818 <== osd.1 v2:172.21.15.32:6824/1551241759 1 ==== osd_op_reply(59 mds0_openfiles.0 [create] v110'114 uv113 ondisk = -17 ((17) File exists)) ==== 160+0+0 (crc 0 0 0) 0x7f0898011640 con 0x555a0b88e560 2024-08-03T20:57:58.775 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:58.775+0000 7f08b431ff40 1 -- 172.21.15.32:0/3980296818 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:60 12.6 12:654134d2:::mds0_openfiles.0:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e110) -- 0x555a0b890f60 con 0x555a0b88e560 2024-08-03T20:57:58.777 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:58.776+0000 7f08a7e00640 1 -- 172.21.15.32:0/3980296818 <== osd.1 v2:172.21.15.32:6824/1551241759 2 ==== osd_op_reply(60 mds0_openfiles.0 [delete] v110'115 uv115 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f0898011640 con 0x555a0b88e560 2024-08-03T20:57:58.777 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:58.776+0000 7f08b431ff40 1 -- 172.21.15.32:0/3980296818 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:61 12.6 12:654134d2:::mds0_openfiles.0:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110) -- 0x555a0b8913a0 con 0x555a0b88e560 2024-08-03T20:57:58.778 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:58.777+0000 7f08a7e00640 1 -- 172.21.15.32:0/3980296818 <== osd.1 v2:172.21.15.32:6824/1551241759 3 ==== osd_op_reply(61 mds0_openfiles.0 [create] v110'116 uv116 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f0898011640 con 0x555a0b88e560 2024-08-03T20:57:58.778 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds0_openfiles.0:head# 2024-08-03T20:57:58.778 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:58.777+0000 7f08b431ff40 1 -- 172.21.15.32:0/3980296818 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:62 12.6 12:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110) -- 0x555a0b8913a0 con 0x555a0b88e560 2024-08-03T20:57:58.779 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:58.779+0000 7f08a7e00640 1 -- 172.21.15.32:0/3980296818 <== osd.1 v2:172.21.15.32:6824/1551241759 4 ==== osd_op_reply(62 mds0_openfiles.0 [omap-set-header] v110'117 uv117 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f0898011640 con 0x555a0b88e560 2024-08-03T20:57:58.779 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:58.779+0000 7f08b431ff40 1 -- 172.21.15.32:0/3980296818 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:63 12.e 12:717a0223:::608.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110) -- 0x555a0b8913a0 con 0x555a0b87b680 2024-08-03T20:57:58.781 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:58.781+0000 7f08a7e00640 1 -- 172.21.15.32:0/3980296818 <== osd.0 v2:172.21.15.32:6816/3066156116 38 ==== osd_op_reply(63 608.00000000 [create] v110'147 uv146 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f0898011640 con 0x555a0b87b680 2024-08-03T20:57:58.781 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:58.781+0000 7f08b431ff40 1 -- 172.21.15.32:0/3980296818 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:64 12.e 12:717a0223:::608.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e110) -- 0x555a0b8913a0 con 0x555a0b87b680 2024-08-03T20:57:58.783 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:58.782+0000 7f08a7e00640 1 -- 172.21.15.32:0/3980296818 <== osd.0 v2:172.21.15.32:6816/3066156116 39 ==== osd_op_reply(64 608.00000000 [delete] v110'148 uv148 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f0898011640 con 0x555a0b87b680 2024-08-03T20:57:58.783 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:58.782+0000 7f08b431ff40 1 -- 172.21.15.32:0/3980296818 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:65 12.e 12:717a0223:::608.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110) -- 0x555a0b8913a0 con 0x555a0b87b680 2024-08-03T20:57:58.784 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::608.00000000:head# 2024-08-03T20:57:58.784 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:58.784+0000 7f08a7e00640 1 -- 172.21.15.32:0/3980296818 <== osd.0 v2:172.21.15.32:6816/3066156116 40 ==== osd_op_reply(65 608.00000000 [create] v110'149 uv149 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f0898011640 con 0x555a0b87b680 2024-08-03T20:57:58.785 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:58.784+0000 7f08b431ff40 1 -- 172.21.15.32:0/3980296818 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:66 12.e 12:717a0223:::608.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110) -- 0x555a0b8923b0 con 0x555a0b87b680 2024-08-03T20:57:58.786 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:58.786+0000 7f08a7e00640 1 -- 172.21.15.32:0/3980296818 <== osd.0 v2:172.21.15.32:6816/3066156116 41 ==== osd_op_reply(66 608.00000000 [setxattr (30)] v110'150 uv150 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f0898011640 con 0x555a0b87b680 2024-08-03T20:57:58.786 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:58.786+0000 7f08b431ff40 1 -- 172.21.15.32:0/3980296818 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:67 12.e 12:717a0223:::608.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110) -- 0x555a0b8923b0 con 0x555a0b87b680 2024-08-03T20:57:58.787 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:58.787+0000 7f08a7e00640 1 -- 172.21.15.32:0/3980296818 <== osd.0 v2:172.21.15.32:6816/3066156116 42 ==== osd_op_reply(67 608.00000000 [setxattr (62)] v110'151 uv151 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f0898011640 con 0x555a0b87b680 2024-08-03T20:57:58.788 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:58.787+0000 7f08b431ff40 1 -- 172.21.15.32:0/3980296818 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:68 12.e 12:717a0223:::608.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110) -- 0x555a0b8913a0 con 0x555a0b87b680 2024-08-03T20:57:58.789 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:58.789+0000 7f08a7e00640 1 -- 172.21.15.32:0/3980296818 <== osd.0 v2:172.21.15.32:6816/3066156116 43 ==== osd_op_reply(68 608.00000000 [omap-set-header] v110'152 uv152 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f0898011640 con 0x555a0b87b680 2024-08-03T20:57:58.789 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:58.789+0000 7f08b431ff40 1 --2- 172.21.15.32:0/3980296818 >> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] conn(0x555a0b8918b0 0x555a0b893cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:58.789 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:58.789+0000 7f08b431ff40 1 -- 172.21.15.32:0/3980296818 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:69 12.15 12:a93a17c2:::604.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110) -- 0x555a0b8942a0 con 0x555a0b8918b0 2024-08-03T20:57:58.790 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:58.789+0000 7f08ad400640 1 --2- 172.21.15.32:0/3980296818 >> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] conn(0x555a0b8918b0 0x555a0b893cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:58.790 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:58.790+0000 7f08ad400640 1 --2- 172.21.15.32:0/3980296818 >> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] conn(0x555a0b8918b0 0x555a0b893cd0 crc :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:57:58.792 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:58.791+0000 7f08ad400640 1 -- 172.21.15.32:0/3980296818 <== osd.6 v2:172.21.15.118:6825/801947386 1 ==== osd_op_reply(69 604.00000000 [create] v110'271 uv265 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f08a8070020 con 0x555a0b8918b0 2024-08-03T20:57:58.792 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:58.791+0000 7f08b431ff40 1 -- 172.21.15.32:0/3980296818 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:70 12.15 12:a93a17c2:::604.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e110) -- 0x555a0b894770 con 0x555a0b8918b0 2024-08-03T20:57:58.793 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:58.793+0000 7f08ad400640 1 -- 172.21.15.32:0/3980296818 <== osd.6 v2:172.21.15.118:6825/801947386 2 ==== osd_op_reply(70 604.00000000 [delete] v110'272 uv272 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f08a8070020 con 0x555a0b8918b0 2024-08-03T20:57:58.793 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:58.793+0000 7f08b431ff40 1 -- 172.21.15.32:0/3980296818 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:71 12.15 12:a93a17c2:::604.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110) -- 0x555a0b894c50 con 0x555a0b8918b0 2024-08-03T20:57:58.794 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::604.00000000:head# 2024-08-03T20:57:58.795 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:58.794+0000 7f08ad400640 1 -- 172.21.15.32:0/3980296818 <== osd.6 v2:172.21.15.118:6825/801947386 3 ==== osd_op_reply(71 604.00000000 [create] v110'273 uv273 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f08a8070020 con 0x555a0b8918b0 2024-08-03T20:57:58.795 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:58.794+0000 7f08b431ff40 1 -- 172.21.15.32:0/3980296818 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:72 12.15 12:a93a17c2:::604.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110) -- 0x555a0b896500 con 0x555a0b8918b0 2024-08-03T20:57:58.796 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:58.795+0000 7f08ad400640 1 -- 172.21.15.32:0/3980296818 <== osd.6 v2:172.21.15.118:6825/801947386 4 ==== osd_op_reply(72 604.00000000 [setxattr (30)] v110'274 uv274 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f08a8070020 con 0x555a0b8918b0 2024-08-03T20:57:58.796 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:58.796+0000 7f08b431ff40 1 -- 172.21.15.32:0/3980296818 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:73 12.15 12:a93a17c2:::604.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110) -- 0x555a0b8968a0 con 0x555a0b8918b0 2024-08-03T20:57:58.797 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:58.797+0000 7f08ad400640 1 -- 172.21.15.32:0/3980296818 <== osd.6 v2:172.21.15.118:6825/801947386 5 ==== osd_op_reply(73 604.00000000 [setxattr (62)] v110'275 uv275 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f08a8070020 con 0x555a0b8918b0 2024-08-03T20:57:58.797 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:58.797+0000 7f08b431ff40 1 -- 172.21.15.32:0/3980296818 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:74 12.15 12:a93a17c2:::604.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110) -- 0x555a0b8954f0 con 0x555a0b8918b0 2024-08-03T20:57:58.798 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:58.798+0000 7f08ad400640 1 -- 172.21.15.32:0/3980296818 <== osd.6 v2:172.21.15.118:6825/801947386 6 ==== osd_op_reply(74 604.00000000 [omap-set-header] v110'276 uv276 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f08a8070020 con 0x555a0b8918b0 2024-08-03T20:57:58.799 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:58.798+0000 7f08b431ff40 1 -- 172.21.15.32:0/3980296818 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:75 12.15 12:aa448500:::500.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110) -- 0x555a0b8958e0 con 0x555a0b8918b0 2024-08-03T20:57:58.800 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:58.800+0000 7f08ad400640 1 -- 172.21.15.32:0/3980296818 <== osd.6 v2:172.21.15.118:6825/801947386 7 ==== osd_op_reply(75 500.00000000 [create] v110'277 uv270 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f08a8070020 con 0x555a0b8918b0 2024-08-03T20:57:58.800 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:58.800+0000 7f08b431ff40 1 -- 172.21.15.32:0/3980296818 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:76 12.15 12:aa448500:::500.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e110) -- 0x555a0b8958e0 con 0x555a0b8918b0 2024-08-03T20:57:58.801 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:58.801+0000 7f08ad400640 1 -- 172.21.15.32:0/3980296818 <== osd.6 v2:172.21.15.118:6825/801947386 8 ==== osd_op_reply(76 500.00000000 [delete] v110'278 uv278 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f08a8070020 con 0x555a0b8918b0 2024-08-03T20:57:58.801 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:58.801+0000 7f08b431ff40 1 -- 172.21.15.32:0/3980296818 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:77 12.15 12:aa448500:::500.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110) -- 0x555a0b8958e0 con 0x555a0b8918b0 2024-08-03T20:57:58.803 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::500.00000000:head# 2024-08-03T20:57:58.803 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:58.803+0000 7f08ad400640 1 -- 172.21.15.32:0/3980296818 <== osd.6 v2:172.21.15.118:6825/801947386 9 ==== osd_op_reply(77 500.00000000 [create] v110'279 uv279 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f08a8070020 con 0x555a0b8918b0 2024-08-03T20:57:58.803 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:58.803+0000 7f08b431ff40 1 -- 172.21.15.32:0/3980296818 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:78 12.15 12:aa448500:::500.00000000:head [write 0~90 in=90b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110) -- 0x555a0b8958e0 con 0x555a0b8918b0 2024-08-03T20:57:58.804 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:58.804+0000 7f08ad400640 1 -- 172.21.15.32:0/3980296818 <== osd.6 v2:172.21.15.118:6825/801947386 10 ==== osd_op_reply(78 500.00000000 [write 0~90] v110'280 uv280 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f08a8070020 con 0x555a0b8918b0 2024-08-03T20:57:58.804 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:58.804+0000 7f08b431ff40 1 -- 172.21.15.32:0/3980296818 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:79 12.15 12:aa448500:::500.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110) -- 0x555a0b8958e0 con 0x555a0b8918b0 2024-08-03T20:57:58.806 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:58.805+0000 7f08ad400640 1 -- 172.21.15.32:0/3980296818 <== osd.6 v2:172.21.15.118:6825/801947386 11 ==== osd_op_reply(79 500.00000000 [omap-set-header] v110'281 uv281 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f08a8070020 con 0x555a0b8918b0 2024-08-03T20:57:58.806 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:58.805+0000 7f08b431ff40 1 --2- 172.21.15.32:0/3980296818 >> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] conn(0x555a0b896c40 0x555a0b899060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:58.806 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:58.805+0000 7f08b431ff40 1 -- 172.21.15.32:0/3980296818 --> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] -- osd_op(unknown.0.0:80 12.d 12:b50e409b:::mds_snaptable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110) -- 0x555a0b8958e0 con 0x555a0b896c40 2024-08-03T20:57:58.806 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:58.806+0000 7f08aca00640 1 --2- 172.21.15.32:0/3980296818 >> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] conn(0x555a0b896c40 0x555a0b899060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:58.807 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:58.806+0000 7f08aca00640 1 --2- 172.21.15.32:0/3980296818 >> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] conn(0x555a0b896c40 0x555a0b899060 crc :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:57:58.809 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:58.808+0000 7f08aca00640 1 -- 172.21.15.32:0/3980296818 <== osd.7 v2:172.21.15.118:6818/653000048 1 ==== osd_op_reply(80 mds_snaptable [create] v110'122 uv121 ondisk = -17 ((17) File exists)) ==== 157+0+0 (crc 0 0 0) 0x7f08a000a4e0 con 0x555a0b896c40 2024-08-03T20:57:58.809 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:58.808+0000 7f08b431ff40 1 -- 172.21.15.32:0/3980296818 --> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] -- osd_op(unknown.0.0:81 12.d 12:b50e409b:::mds_snaptable:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e110) -- 0x555a0b8958e0 con 0x555a0b896c40 2024-08-03T20:57:58.810 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:58.810+0000 7f08aca00640 1 -- 172.21.15.32:0/3980296818 <== osd.7 v2:172.21.15.118:6818/653000048 2 ==== osd_op_reply(81 mds_snaptable [delete] v110'123 uv123 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f08a000a4e0 con 0x555a0b896c40 2024-08-03T20:57:58.810 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:58.810+0000 7f08b431ff40 1 -- 172.21.15.32:0/3980296818 --> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] -- osd_op(unknown.0.0:82 12.d 12:b50e409b:::mds_snaptable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110) -- 0x555a0b8995d0 con 0x555a0b896c40 2024-08-03T20:57:58.811 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds_snaptable:head# 2024-08-03T20:57:58.812 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:58.811+0000 7f08aca00640 1 -- 172.21.15.32:0/3980296818 <== osd.7 v2:172.21.15.118:6818/653000048 3 ==== osd_op_reply(82 mds_snaptable [create] v110'124 uv124 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f08a000a4e0 con 0x555a0b896c40 2024-08-03T20:57:58.812 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:58.811+0000 7f08b431ff40 1 -- 172.21.15.32:0/3980296818 --> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] -- osd_op(unknown.0.0:83 12.d 12:b50e409b:::mds_snaptable:head [write 0~70 in=70b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110) -- 0x555a0b8995a0 con 0x555a0b896c40 2024-08-03T20:57:58.813 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:58.812+0000 7f08aca00640 1 -- 172.21.15.32:0/3980296818 <== osd.7 v2:172.21.15.118:6818/653000048 4 ==== osd_op_reply(83 mds_snaptable [write 0~70] v110'125 uv125 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f08a000a4e0 con 0x555a0b896c40 2024-08-03T20:57:58.813 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:58.812+0000 7f08b431ff40 1 -- 172.21.15.32:0/3980296818 --> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] -- osd_op(unknown.0.0:84 12.d 12:b50e409b:::mds_snaptable:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110) -- 0x555a0b8995a0 con 0x555a0b896c40 2024-08-03T20:57:58.814 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:58.814+0000 7f08aca00640 1 -- 172.21.15.32:0/3980296818 <== osd.7 v2:172.21.15.118:6818/653000048 5 ==== osd_op_reply(84 mds_snaptable [omap-set-header] v110'126 uv126 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f08a000a4e0 con 0x555a0b896c40 2024-08-03T20:57:58.814 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:58.814+0000 7f08b431ff40 1 -- 172.21.15.32:0/3980296818 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:85 12.1d 12:b871830b:::605.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110) -- 0x555a0b8995a0 con 0x555a0b87b680 2024-08-03T20:57:58.816 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:58.815+0000 7f08a7e00640 1 -- 172.21.15.32:0/3980296818 <== osd.0 v2:172.21.15.32:6816/3066156116 44 ==== osd_op_reply(85 605.00000000 [create] v110'147 uv146 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f0898011640 con 0x555a0b87b680 2024-08-03T20:57:58.816 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:58.815+0000 7f08b431ff40 1 -- 172.21.15.32:0/3980296818 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:86 12.1d 12:b871830b:::605.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e110) -- 0x555a0b8995a0 con 0x555a0b87b680 2024-08-03T20:57:58.817 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:58.817+0000 7f08a7e00640 1 -- 172.21.15.32:0/3980296818 <== osd.0 v2:172.21.15.32:6816/3066156116 45 ==== osd_op_reply(86 605.00000000 [delete] v110'148 uv148 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f0898011640 con 0x555a0b87b680 2024-08-03T20:57:58.817 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:58.817+0000 7f08b431ff40 1 -- 172.21.15.32:0/3980296818 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:87 12.1d 12:b871830b:::605.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110) -- 0x555a0b8995a0 con 0x555a0b87b680 2024-08-03T20:57:58.818 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:58.818+0000 7f08a7e00640 1 -- 172.21.15.32:0/3980296818 <== osd.0 v2:172.21.15.32:6816/3066156116 46 ==== osd_op_reply(87 605.00000000 [create] v110'149 uv149 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f0898011640 con 0x555a0b87b680 2024-08-03T20:57:58.818 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::605.00000000:head# 2024-08-03T20:57:58.819 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:58.818+0000 7f08b431ff40 1 -- 172.21.15.32:0/3980296818 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:88 12.1d 12:b871830b:::605.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110) -- 0x555a0b89a5b0 con 0x555a0b87b680 2024-08-03T20:57:58.820 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:58.819+0000 7f08a7e00640 1 -- 172.21.15.32:0/3980296818 <== osd.0 v2:172.21.15.32:6816/3066156116 47 ==== osd_op_reply(88 605.00000000 [setxattr (30)] v110'150 uv150 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f0898011640 con 0x555a0b87b680 2024-08-03T20:57:58.820 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:58.819+0000 7f08b431ff40 1 -- 172.21.15.32:0/3980296818 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:89 12.1d 12:b871830b:::605.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110) -- 0x555a0b89a5b0 con 0x555a0b87b680 2024-08-03T20:57:58.821 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:58.820+0000 7f08a7e00640 1 -- 172.21.15.32:0/3980296818 <== osd.0 v2:172.21.15.32:6816/3066156116 48 ==== osd_op_reply(89 605.00000000 [setxattr (62)] v110'151 uv151 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f0898011640 con 0x555a0b87b680 2024-08-03T20:57:58.821 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:58.821+0000 7f08b431ff40 1 -- 172.21.15.32:0/3980296818 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:90 12.1d 12:b871830b:::605.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110) -- 0x555a0b8995a0 con 0x555a0b87b680 2024-08-03T20:57:58.822 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:58.822+0000 7f08a7e00640 1 -- 172.21.15.32:0/3980296818 <== osd.0 v2:172.21.15.32:6816/3066156116 49 ==== osd_op_reply(90 605.00000000 [omap-set-header] v110'152 uv152 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f0898011640 con 0x555a0b87b680 2024-08-03T20:57:58.822 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:58.822+0000 7f08b431ff40 1 -- 172.21.15.32:0/3980296818 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:91 12.13 12:c91d4a1d:::mds0_inotable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110) -- 0x555a0b8995a0 con 0x555a0b88e560 2024-08-03T20:57:58.824 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:58.823+0000 7f08a7e00640 1 -- 172.21.15.32:0/3980296818 <== osd.1 v2:172.21.15.32:6824/1551241759 5 ==== osd_op_reply(91 mds0_inotable [create] v110'243 uv237 ondisk = -17 ((17) File exists)) ==== 157+0+0 (crc 0 0 0) 0x7f0898011640 con 0x555a0b88e560 2024-08-03T20:57:58.824 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:58.824+0000 7f08b431ff40 1 -- 172.21.15.32:0/3980296818 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:92 12.13 12:c91d4a1d:::mds0_inotable:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e110) -- 0x555a0b8995a0 con 0x555a0b88e560 2024-08-03T20:57:58.825 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:58.825+0000 7f08a7e00640 1 -- 172.21.15.32:0/3980296818 <== osd.1 v2:172.21.15.32:6824/1551241759 6 ==== osd_op_reply(92 mds0_inotable [delete] v110'244 uv244 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f0898011640 con 0x555a0b88e560 2024-08-03T20:57:58.825 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:58.825+0000 7f08b431ff40 1 -- 172.21.15.32:0/3980296818 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:93 12.13 12:c91d4a1d:::mds0_inotable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110) -- 0x555a0b8995a0 con 0x555a0b88e560 2024-08-03T20:57:58.826 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds0_inotable:head# 2024-08-03T20:57:58.827 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:58.826+0000 7f08a7e00640 1 -- 172.21.15.32:0/3980296818 <== osd.1 v2:172.21.15.32:6824/1551241759 7 ==== osd_op_reply(93 mds0_inotable [create] v110'245 uv245 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f0898011640 con 0x555a0b88e560 2024-08-03T20:57:58.827 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:58.826+0000 7f08b431ff40 1 -- 172.21.15.32:0/3980296818 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:94 12.13 12:c91d4a1d:::mds0_inotable:head [write 0~34 in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110) -- 0x555a0b8950e0 con 0x555a0b88e560 2024-08-03T20:57:58.828 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:58.828+0000 7f08a7e00640 1 -- 172.21.15.32:0/3980296818 <== osd.1 v2:172.21.15.32:6824/1551241759 8 ==== osd_op_reply(94 mds0_inotable [write 0~34] v110'246 uv246 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f0898011640 con 0x555a0b88e560 2024-08-03T20:57:58.828 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:58.828+0000 7f08b431ff40 1 -- 172.21.15.32:0/3980296818 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:95 12.13 12:c91d4a1d:::mds0_inotable:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110) -- 0x555a0b8950e0 con 0x555a0b88e560 2024-08-03T20:57:58.829 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:58.829+0000 7f08a7e00640 1 -- 172.21.15.32:0/3980296818 <== osd.1 v2:172.21.15.32:6824/1551241759 9 ==== osd_op_reply(95 mds0_inotable [omap-set-header] v110'247 uv247 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f0898011640 con 0x555a0b88e560 2024-08-03T20:57:58.829 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:58.829+0000 7f08b431ff40 1 -- 172.21.15.32:0/3980296818 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:96 12.13 12:cd5a64a3:::100.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110) -- 0x555a0b8950e0 con 0x555a0b88e560 2024-08-03T20:57:58.831 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:58.830+0000 7f08a7e00640 1 -- 172.21.15.32:0/3980296818 <== osd.1 v2:172.21.15.32:6824/1551241759 10 ==== osd_op_reply(96 100.00000000 [create] v110'248 uv242 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f0898011640 con 0x555a0b88e560 2024-08-03T20:57:58.831 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:58.830+0000 7f08b431ff40 1 -- 172.21.15.32:0/3980296818 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:97 12.13 12:cd5a64a3:::100.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e110) -- 0x555a0b8950e0 con 0x555a0b88e560 2024-08-03T20:57:58.832 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:58.832+0000 7f08a7e00640 1 -- 172.21.15.32:0/3980296818 <== osd.1 v2:172.21.15.32:6824/1551241759 11 ==== osd_op_reply(97 100.00000000 [delete] v110'249 uv249 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f0898011640 con 0x555a0b88e560 2024-08-03T20:57:58.832 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:58.832+0000 7f08b431ff40 1 -- 172.21.15.32:0/3980296818 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:98 12.13 12:cd5a64a3:::100.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110) -- 0x555a0b8950e0 con 0x555a0b88e560 2024-08-03T20:57:58.833 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::100.00000000:head# 2024-08-03T20:57:58.833 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:58.833+0000 7f08a7e00640 1 -- 172.21.15.32:0/3980296818 <== osd.1 v2:172.21.15.32:6824/1551241759 12 ==== osd_op_reply(98 100.00000000 [create] v110'250 uv250 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f0898011640 con 0x555a0b88e560 2024-08-03T20:57:58.833 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:58.833+0000 7f08b431ff40 1 -- 172.21.15.32:0/3980296818 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:99 12.13 12:cd5a64a3:::100.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110) -- 0x555a0b8950e0 con 0x555a0b88e560 2024-08-03T20:57:58.834 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:58.834+0000 7f08a7e00640 1 -- 172.21.15.32:0/3980296818 <== osd.1 v2:172.21.15.32:6824/1551241759 13 ==== osd_op_reply(99 100.00000000 [omap-set-header] v110'251 uv251 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f0898011640 con 0x555a0b88e560 2024-08-03T20:57:58.835 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:58.834+0000 7f08b431ff40 1 -- 172.21.15.32:0/3980296818 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:100 12.13 12:cd5a64a3:::100.00000000:head [omap-set-vals in=5194b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110) -- 0x555a0b89c450 con 0x555a0b88e560 2024-08-03T20:57:58.836 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:58.836+0000 7f08a7e00640 1 -- 172.21.15.32:0/3980296818 <== osd.1 v2:172.21.15.32:6824/1551241759 14 ==== osd_op_reply(100 100.00000000 [omap-set-vals] v110'252 uv252 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f0898011640 con 0x555a0b88e560 2024-08-03T20:57:58.836 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:58.836+0000 7f08b431ff40 1 --2- 172.21.15.32:0/3980296818 >> [v2:172.21.15.32:6800/9088303,v1:172.21.15.32:6801/9088303] conn(0x555a0b899ab0 0x555a0b89cb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:58.836 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:58.836+0000 7f08b431ff40 1 -- 172.21.15.32:0/3980296818 --> [v2:172.21.15.32:6800/9088303,v1:172.21.15.32:6801/9088303] -- osd_op(unknown.0.0:101 12.b 12:d0630e4c:::mds0_sessionmap:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110) -- 0x555a0b89bed0 con 0x555a0b899ab0 2024-08-03T20:57:58.836 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:58.836+0000 7f08a7e00640 1 --2- 172.21.15.32:0/3980296818 >> [v2:172.21.15.32:6800/9088303,v1:172.21.15.32:6801/9088303] conn(0x555a0b899ab0 0x555a0b89cb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:58.837 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:58.837+0000 7f08a7e00640 1 --2- 172.21.15.32:0/3980296818 >> [v2:172.21.15.32:6800/9088303,v1:172.21.15.32:6801/9088303] conn(0x555a0b899ab0 0x555a0b89cb10 crc :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:57:58.838 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:58.838+0000 7f08a7e00640 1 -- 172.21.15.32:0/3980296818 <== osd.3 v2:172.21.15.32:6800/9088303 1 ==== osd_op_reply(101 mds0_sessionmap [create] v110'107 uv106 ondisk = -17 ((17) File exists)) ==== 159+0+0 (crc 0 0 0) 0x7f0898011640 con 0x555a0b899ab0 2024-08-03T20:57:58.838 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:58.838+0000 7f08b431ff40 1 -- 172.21.15.32:0/3980296818 --> [v2:172.21.15.32:6800/9088303,v1:172.21.15.32:6801/9088303] -- osd_op(unknown.0.0:102 12.b 12:d0630e4c:::mds0_sessionmap:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e110) -- 0x555a0b89bed0 con 0x555a0b899ab0 2024-08-03T20:57:58.840 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:58.839+0000 7f08a7e00640 1 -- 172.21.15.32:0/3980296818 <== osd.3 v2:172.21.15.32:6800/9088303 2 ==== osd_op_reply(102 mds0_sessionmap [delete] v110'108 uv108 ondisk = 0) ==== 159+0+0 (crc 0 0 0) 0x7f0898011640 con 0x555a0b899ab0 2024-08-03T20:57:58.840 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:58.839+0000 7f08b431ff40 1 -- 172.21.15.32:0/3980296818 --> [v2:172.21.15.32:6800/9088303,v1:172.21.15.32:6801/9088303] -- osd_op(unknown.0.0:103 12.b 12:d0630e4c:::mds0_sessionmap:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110) -- 0x555a0b89bed0 con 0x555a0b899ab0 2024-08-03T20:57:58.841 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds0_sessionmap:head# 2024-08-03T20:57:58.841 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:58.840+0000 7f08a7e00640 1 -- 172.21.15.32:0/3980296818 <== osd.3 v2:172.21.15.32:6800/9088303 3 ==== osd_op_reply(103 mds0_sessionmap [create] v110'109 uv109 ondisk = 0) ==== 159+0+0 (crc 0 0 0) 0x7f0898011640 con 0x555a0b899ab0 2024-08-03T20:57:58.841 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:58.840+0000 7f08b431ff40 1 -- 172.21.15.32:0/3980296818 --> [v2:172.21.15.32:6800/9088303,v1:172.21.15.32:6801/9088303] -- osd_op(unknown.0.0:104 12.b 12:d0630e4c:::mds0_sessionmap:head [omap-set-header in=14b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110) -- 0x555a0b89bed0 con 0x555a0b899ab0 2024-08-03T20:57:58.842 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:58.842+0000 7f08a7e00640 1 -- 172.21.15.32:0/3980296818 <== osd.3 v2:172.21.15.32:6800/9088303 4 ==== osd_op_reply(104 mds0_sessionmap [omap-set-header] v110'110 uv110 ondisk = 0) ==== 159+0+0 (crc 0 0 0) 0x7f0898011640 con 0x555a0b899ab0 2024-08-03T20:57:58.842 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:58.842+0000 7f08b431ff40 1 -- 172.21.15.32:0/3980296818 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:105 12.7 12:e0b41d06:::609.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110) -- 0x555a0b89bed0 con 0x555a0b88e560 2024-08-03T20:57:58.844 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:58.844+0000 7f08a7e00640 1 -- 172.21.15.32:0/3980296818 <== osd.1 v2:172.21.15.32:6824/1551241759 15 ==== osd_op_reply(105 609.00000000 [create] v110'147 uv146 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f0898011640 con 0x555a0b88e560 2024-08-03T20:57:58.844 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:58.844+0000 7f08b431ff40 1 -- 172.21.15.32:0/3980296818 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:106 12.7 12:e0b41d06:::609.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e110) -- 0x555a0b89bed0 con 0x555a0b88e560 2024-08-03T20:57:58.845 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:58.845+0000 7f08a7e00640 1 -- 172.21.15.32:0/3980296818 <== osd.1 v2:172.21.15.32:6824/1551241759 16 ==== osd_op_reply(106 609.00000000 [delete] v110'148 uv148 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f0898011640 con 0x555a0b88e560 2024-08-03T20:57:58.846 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:58.845+0000 7f08b431ff40 1 -- 172.21.15.32:0/3980296818 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:107 12.7 12:e0b41d06:::609.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110) -- 0x555a0b89bed0 con 0x555a0b88e560 2024-08-03T20:57:58.847 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::609.00000000:head# 2024-08-03T20:57:58.847 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:58.847+0000 7f08a7e00640 1 -- 172.21.15.32:0/3980296818 <== osd.1 v2:172.21.15.32:6824/1551241759 17 ==== osd_op_reply(107 609.00000000 [create] v110'149 uv149 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f0898011640 con 0x555a0b88e560 2024-08-03T20:57:58.847 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:58.847+0000 7f08b431ff40 1 -- 172.21.15.32:0/3980296818 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:108 12.7 12:e0b41d06:::609.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110) -- 0x555a0b89bed0 con 0x555a0b88e560 2024-08-03T20:57:58.848 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:58.848+0000 7f08a7e00640 1 -- 172.21.15.32:0/3980296818 <== osd.1 v2:172.21.15.32:6824/1551241759 18 ==== osd_op_reply(108 609.00000000 [setxattr (30)] v110'150 uv150 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f0898011640 con 0x555a0b88e560 2024-08-03T20:57:58.848 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:58.848+0000 7f08b431ff40 1 -- 172.21.15.32:0/3980296818 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:109 12.7 12:e0b41d06:::609.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110) -- 0x555a0b89bed0 con 0x555a0b88e560 2024-08-03T20:57:58.850 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:58.849+0000 7f08a7e00640 1 -- 172.21.15.32:0/3980296818 <== osd.1 v2:172.21.15.32:6824/1551241759 19 ==== osd_op_reply(109 609.00000000 [setxattr (62)] v110'151 uv151 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f0898011640 con 0x555a0b88e560 2024-08-03T20:57:58.850 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:58.849+0000 7f08b431ff40 1 -- 172.21.15.32:0/3980296818 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:110 12.7 12:e0b41d06:::609.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110) -- 0x555a0b89bed0 con 0x555a0b88e560 2024-08-03T20:57:58.851 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:58.851+0000 7f08a7e00640 1 -- 172.21.15.32:0/3980296818 <== osd.1 v2:172.21.15.32:6824/1551241759 20 ==== osd_op_reply(110 609.00000000 [omap-set-header] v110'152 uv152 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f0898011640 con 0x555a0b88e560 2024-08-03T20:57:58.851 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:58.851+0000 7f08b431ff40 1 -- 172.21.15.32:0/3980296818 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:111 12.f 12:f1f69726:::400.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110) -- 0x555a0b89bed0 con 0x555a0b8918b0 2024-08-03T20:57:58.853 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:58.852+0000 7f08ad400640 1 -- 172.21.15.32:0/3980296818 <== osd.6 v2:172.21.15.118:6825/801947386 12 ==== osd_op_reply(111 400.00000000 [create] v110'125 uv124 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f08a8070020 con 0x555a0b8918b0 2024-08-03T20:57:58.853 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:58.852+0000 7f08b431ff40 1 -- 172.21.15.32:0/3980296818 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:112 12.f 12:f1f69726:::400.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e110) -- 0x555a0b89bed0 con 0x555a0b8918b0 2024-08-03T20:57:58.854 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:58.854+0000 7f08ad400640 1 -- 172.21.15.32:0/3980296818 <== osd.6 v2:172.21.15.118:6825/801947386 13 ==== osd_op_reply(112 400.00000000 [delete] v110'126 uv126 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f08a8070020 con 0x555a0b8918b0 2024-08-03T20:57:58.855 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:58.854+0000 7f08b431ff40 1 -- 172.21.15.32:0/3980296818 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:113 12.f 12:f1f69726:::400.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110) -- 0x555a0b89bed0 con 0x555a0b8918b0 2024-08-03T20:57:58.856 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::400.00000000:head# 2024-08-03T20:57:58.857 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:58.856+0000 7f08ad400640 1 -- 172.21.15.32:0/3980296818 <== osd.6 v2:172.21.15.118:6825/801947386 14 ==== osd_op_reply(113 400.00000000 [create] v110'127 uv127 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f08a8070020 con 0x555a0b8918b0 2024-08-03T20:57:58.857 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:58.856+0000 7f08b431ff40 1 -- 172.21.15.32:0/3980296818 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:114 12.f 12:f1f69726:::400.00000000:head [write 0~22 in=22b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110) -- 0x555a0b89bed0 con 0x555a0b8918b0 2024-08-03T20:57:58.859 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:58.858+0000 7f08ad400640 1 -- 172.21.15.32:0/3980296818 <== osd.6 v2:172.21.15.118:6825/801947386 15 ==== osd_op_reply(114 400.00000000 [write 0~22] v110'128 uv128 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f08a8070020 con 0x555a0b8918b0 2024-08-03T20:57:58.859 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:58.858+0000 7f08b431ff40 1 -- 172.21.15.32:0/3980296818 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:115 12.f 12:f1f69726:::400.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110) -- 0x555a0b89bed0 con 0x555a0b8918b0 2024-08-03T20:57:58.860 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:58.860+0000 7f08ad400640 1 -- 172.21.15.32:0/3980296818 <== osd.6 v2:172.21.15.118:6825/801947386 16 ==== osd_op_reply(115 400.00000000 [omap-set-header] v110'129 uv129 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f08a8070020 con 0x555a0b8918b0 2024-08-03T20:57:58.860 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:58.860+0000 7f08b431ff40 1 -- 172.21.15.32:0/3980296818 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:116 12.1f 12:fe07bba1:::100.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110) -- 0x555a0b89bed0 con 0x555a0b8918b0 2024-08-03T20:57:58.862 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:58.861+0000 7f08ad400640 1 -- 172.21.15.32:0/3980296818 <== osd.6 v2:172.21.15.118:6825/801947386 17 ==== osd_op_reply(116 100.00000000.inode [create] v110'302 uv294 ondisk = -17 ((17) File exists)) ==== 162+0+0 (crc 0 0 0) 0x7f08a8070020 con 0x555a0b8918b0 2024-08-03T20:57:58.862 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:58.861+0000 7f08b431ff40 1 -- 172.21.15.32:0/3980296818 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:117 12.1f 12:fe07bba1:::100.00000000.inode:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e110) -- 0x555a0b89bed0 con 0x555a0b8918b0 2024-08-03T20:57:58.863 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:58.863+0000 7f08ad400640 1 -- 172.21.15.32:0/3980296818 <== osd.6 v2:172.21.15.118:6825/801947386 18 ==== osd_op_reply(117 100.00000000.inode [delete] v110'303 uv303 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7f08a8070020 con 0x555a0b8918b0 2024-08-03T20:57:58.863 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:58.863+0000 7f08b431ff40 1 -- 172.21.15.32:0/3980296818 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:118 12.1f 12:fe07bba1:::100.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110) -- 0x555a0b89bed0 con 0x555a0b8918b0 2024-08-03T20:57:58.864 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::100.00000000.inode:head# 2024-08-03T20:57:58.865 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:58.864+0000 7f08ad400640 1 -- 172.21.15.32:0/3980296818 <== osd.6 v2:172.21.15.118:6825/801947386 19 ==== osd_op_reply(118 100.00000000.inode [create] v110'304 uv304 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7f08a8070020 con 0x555a0b8918b0 2024-08-03T20:57:58.865 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:58.864+0000 7f08b431ff40 1 -- 172.21.15.32:0/3980296818 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:119 12.1f 12:fe07bba1:::100.00000000.inode:head [write 0~582 in=582b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110) -- 0x555a0b89d0a0 con 0x555a0b8918b0 2024-08-03T20:57:58.866 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:58.866+0000 7f08ad400640 1 -- 172.21.15.32:0/3980296818 <== osd.6 v2:172.21.15.118:6825/801947386 20 ==== osd_op_reply(119 100.00000000.inode [write 0~582] v110'305 uv305 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7f08a8070020 con 0x555a0b8918b0 2024-08-03T20:57:58.866 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:58.866+0000 7f08b431ff40 1 -- 172.21.15.32:0/3980296818 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:120 12.1f 12:fe07bba1:::100.00000000.inode:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110) -- 0x555a0b89d050 con 0x555a0b8918b0 2024-08-03T20:57:58.867 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:58.867+0000 7f08ad400640 1 -- 172.21.15.32:0/3980296818 <== osd.6 v2:172.21.15.118:6825/801947386 21 ==== osd_op_reply(120 100.00000000.inode [omap-set-header] v110'306 uv306 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7f08a8070020 con 0x555a0b8918b0 2024-08-03T20:57:58.868 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:58.867+0000 7f08b431ff40 1 -- 172.21.15.32:0/3980296818 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:121 12.1f 12:ff5b34d6:::1.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110) -- 0x555a0b89d050 con 0x555a0b8918b0 2024-08-03T20:57:58.869 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:58.869+0000 7f08ad400640 1 -- 172.21.15.32:0/3980296818 <== osd.6 v2:172.21.15.118:6825/801947386 22 ==== osd_op_reply(121 1.00000000 [create] v110'307 uv301 ondisk = -17 ((17) File exists)) ==== 154+0+0 (crc 0 0 0) 0x7f08a8070020 con 0x555a0b8918b0 2024-08-03T20:57:58.869 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:58.869+0000 7f08b431ff40 1 -- 172.21.15.32:0/3980296818 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:122 12.1f 12:ff5b34d6:::1.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e110) -- 0x555a0b89d050 con 0x555a0b8918b0 2024-08-03T20:57:58.870 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:58.870+0000 7f08ad400640 1 -- 172.21.15.32:0/3980296818 <== osd.6 v2:172.21.15.118:6825/801947386 23 ==== osd_op_reply(122 1.00000000 [delete] v110'308 uv308 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7f08a8070020 con 0x555a0b8918b0 2024-08-03T20:57:58.871 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:58.870+0000 7f08b431ff40 1 -- 172.21.15.32:0/3980296818 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:123 12.1f 12:ff5b34d6:::1.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110) -- 0x555a0b89d050 con 0x555a0b8918b0 2024-08-03T20:57:58.872 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::1.00000000:head# 2024-08-03T20:57:58.872 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:58.871+0000 7f08ad400640 1 -- 172.21.15.32:0/3980296818 <== osd.6 v2:172.21.15.118:6825/801947386 24 ==== osd_op_reply(123 1.00000000 [create] v110'309 uv309 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7f08a8070020 con 0x555a0b8918b0 2024-08-03T20:57:58.872 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:58.871+0000 7f08b431ff40 1 -- 172.21.15.32:0/3980296818 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:124 12.1f 12:ff5b34d6:::1.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110) -- 0x555a0b89e060 con 0x555a0b8918b0 2024-08-03T20:57:58.873 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:58.873+0000 7f08ad400640 1 -- 172.21.15.32:0/3980296818 <== osd.6 v2:172.21.15.118:6825/801947386 25 ==== osd_op_reply(124 1.00000000 [setxattr (30)] v110'310 uv310 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7f08a8070020 con 0x555a0b8918b0 2024-08-03T20:57:58.873 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:58.873+0000 7f08b431ff40 1 -- 172.21.15.32:0/3980296818 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:125 12.1f 12:ff5b34d6:::1.00000000:head [setxattr parent (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110) -- 0x555a0b89e060 con 0x555a0b8918b0 2024-08-03T20:57:58.875 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:58.874+0000 7f08ad400640 1 -- 172.21.15.32:0/3980296818 <== osd.6 v2:172.21.15.118:6825/801947386 26 ==== osd_op_reply(125 1.00000000 [setxattr (30)] v110'311 uv311 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7f08a8070020 con 0x555a0b8918b0 2024-08-03T20:57:58.875 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:58.874+0000 7f08b431ff40 1 -- 172.21.15.32:0/3980296818 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:126 12.1f 12:ff5b34d6:::1.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110) -- 0x555a0b89d050 con 0x555a0b8918b0 2024-08-03T20:57:58.876 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:58.876+0000 7f08ad400640 1 -- 172.21.15.32:0/3980296818 <== osd.6 v2:172.21.15.118:6825/801947386 27 ==== osd_op_reply(126 1.00000000 [omap-set-header] v110'312 uv312 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7f08a8070020 con 0x555a0b8918b0 2024-08-03T20:57:58.876 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:58.876+0000 7f08b431ff40 1 -- 172.21.15.32:0/3980296818 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:127 12.1f 12:ff5b34d6:::1.00000000:head [omap-set-vals in=584b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110) -- 0x555a0b89d500 con 0x555a0b8918b0 2024-08-03T20:57:58.878 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:58.877+0000 7f08ad400640 1 -- 172.21.15.32:0/3980296818 <== osd.6 v2:172.21.15.118:6825/801947386 28 ==== osd_op_reply(127 1.00000000 [omap-set-vals] v110'313 uv313 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7f08a8070020 con 0x555a0b8918b0 2024-08-03T20:57:58.879 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:58.877+0000 7f08b431ff40 1 -- 172.21.15.32:0/3980296818 >> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] conn(0x555a0b87b680 msgr2=0x555a0b87ba90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:58.879 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:58.877+0000 7f08b431ff40 1 --2- 172.21.15.32:0/3980296818 >> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] conn(0x555a0b87b680 0x555a0b87ba90 crc :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:58.879 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:58.877+0000 7f08b431ff40 1 -- 172.21.15.32:0/3980296818 >> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] conn(0x555a0b88e560 msgr2=0x555a0b890980 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:58.879 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:58.877+0000 7f08b431ff40 1 --2- 172.21.15.32:0/3980296818 >> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] conn(0x555a0b88e560 0x555a0b890980 crc :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:58.879 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:58.877+0000 7f08b431ff40 1 -- 172.21.15.32:0/3980296818 >> [v2:172.21.15.32:6800/9088303,v1:172.21.15.32:6801/9088303] conn(0x555a0b899ab0 msgr2=0x555a0b89cb10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:58.879 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:58.877+0000 7f08b431ff40 1 --2- 172.21.15.32:0/3980296818 >> [v2:172.21.15.32:6800/9088303,v1:172.21.15.32:6801/9088303] conn(0x555a0b899ab0 0x555a0b89cb10 crc :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:58.879 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:58.877+0000 7f08b431ff40 1 -- 172.21.15.32:0/3980296818 >> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] conn(0x555a0b888fa0 msgr2=0x555a0b88c0a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:58.879 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:58.877+0000 7f08b431ff40 1 --2- 172.21.15.32:0/3980296818 >> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] conn(0x555a0b888fa0 0x555a0b88c0a0 crc :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:58.879 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:58.877+0000 7f08ad400640 1 -- 172.21.15.32:0/3980296818 reap_dead start 2024-08-03T20:57:58.879 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:58.877+0000 7f08b431ff40 1 -- 172.21.15.32:0/3980296818 >> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] conn(0x555a0b8918b0 msgr2=0x555a0b893cd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:58.880 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:58.877+0000 7f08b431ff40 1 --2- 172.21.15.32:0/3980296818 >> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] conn(0x555a0b8918b0 0x555a0b893cd0 crc :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:58.880 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:58.877+0000 7f08ad400640 1 -- 172.21.15.32:0/3980296818 reap_dead start 2024-08-03T20:57:58.880 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:58.877+0000 7f08b431ff40 1 -- 172.21.15.32:0/3980296818 >> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] conn(0x555a0b896c40 msgr2=0x555a0b899060 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:58.880 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:58.877+0000 7f08b431ff40 1 --2- 172.21.15.32:0/3980296818 >> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] conn(0x555a0b896c40 0x555a0b899060 crc :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:58.880 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:58.878+0000 7f08b431ff40 1 -- 172.21.15.32:0/3980296818 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f088c0759f0 msgr2=0x7f088c077f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:58.880 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:58.878+0000 7f08b431ff40 1 --2- 172.21.15.32:0/3980296818 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f088c0759f0 0x7f088c077f00 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 crypto rx=0x7f08a000c720 tx=0x7f08a000e040 comp rx=0 tx=0).stop 2024-08-03T20:57:58.880 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:58.878+0000 7f08b431ff40 1 -- 172.21.15.32:0/3980296818 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555a0b7c6be0 msgr2=0x555a0b7c55d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:58.880 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:58.878+0000 7f08b431ff40 1 --2- 172.21.15.32:0/3980296818 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555a0b7c6be0 0x555a0b7c55d0 secure :-1 s=READY pgs=1676 cs=0 l=1 rev1=1 crypto rx=0x7f08a8025dd0 tx=0x7f08a804dc20 comp rx=0 tx=0).stop 2024-08-03T20:57:58.880 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:58.878+0000 7f08b431ff40 1 -- 172.21.15.32:0/3980296818 shutdown_connections 2024-08-03T20:57:58.880 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:58.878+0000 7f08b431ff40 1 --2- 172.21.15.32:0/3980296818 >> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] conn(0x555a0b896c40 0x555a0b899060 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:58.880 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:58.878+0000 7f08b431ff40 1 --2- 172.21.15.32:0/3980296818 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f088c0759f0 0x7f088c077f00 unknown :-1 s=CLOSED pgs=684 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:58.880 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:58.878+0000 7f08b431ff40 1 --2- 172.21.15.32:0/3980296818 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555a0b7c6be0 0x555a0b7c55d0 unknown :-1 s=CLOSED pgs=1676 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:58.880 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:58.878+0000 7f08b431ff40 1 -- 172.21.15.32:0/3980296818 >> 172.21.15.32:0/3980296818 conn(0x555a0b62eac0 msgr2=0x555a0b7c2010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:57:58.880 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:58.878+0000 7f08b431ff40 1 -- 172.21.15.32:0/3980296818 shutdown_connections 2024-08-03T20:57:58.881 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:58.878+0000 7f08b431ff40 1 -- 172.21.15.32:0/3980296818 wait complete. 2024-08-03T20:57:58.890 DEBUG:teuthology.orchestra.run.smithi032:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata truncate 200.00000001 0 2024-08-03T20:57:59.073 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:59.073+0000 7f658f020f40 1 Processor -- start 2024-08-03T20:57:59.073 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:59.073+0000 7f658f020f40 1 -- start start 2024-08-03T20:57:59.075 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:59.074+0000 7f658f020f40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56136addd2f0 0x56136adda9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:59.075 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:59.074+0000 7f658f020f40 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x56136ae54f30 0x56136addeaa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:59.075 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:59.074+0000 7f658f020f40 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56136ae4c6b0 0x56136addb220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:59.076 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:59.074+0000 7f658f020f40 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x56136ad45bf0 con 0x56136ae4c6b0 2024-08-03T20:57:59.076 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:59.074+0000 7f658f020f40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x56136ad526b0 con 0x56136addd2f0 2024-08-03T20:57:59.076 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:59.074+0000 7f658f020f40 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x56136ace7e30 con 0x56136ae54f30 2024-08-03T20:57:59.076 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:59.074+0000 7f6588000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56136ae4c6b0 0x56136addb220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:59.076 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:59.074+0000 7f6588000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56136ae4c6b0 0x56136addb220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:51486/0 (socket says 172.21.15.32:51486) 2024-08-03T20:57:59.076 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:59.074+0000 7f6588000640 1 -- 172.21.15.32:0/1408529517 learned_addr learned my addr 172.21.15.32:0/1408529517 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:57:59.076 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:59.074+0000 7f6587600640 1 --2- 172.21.15.32:0/1408529517 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56136addd2f0 0x56136adda9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:59.076 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:59.075+0000 7f6586c00640 1 --2- 172.21.15.32:0/1408529517 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x56136ae54f30 0x56136addeaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:59.076 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:59.075+0000 7f6588000640 1 -- 172.21.15.32:0/1408529517 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x56136ae54f30 msgr2=0x56136addeaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:57:59.076 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:59.075+0000 7f6588000640 1 --2- 172.21.15.32:0/1408529517 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x56136ae54f30 0x56136addeaa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:59.076 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:59.075+0000 7f6588000640 1 -- 172.21.15.32:0/1408529517 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56136addd2f0 msgr2=0x56136adda9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:57:59.076 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:59.075+0000 7f6588000640 1 --2- 172.21.15.32:0/1408529517 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56136addd2f0 0x56136adda9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:59.076 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:59.075+0000 7f6588000640 1 -- 172.21.15.32:0/1408529517 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56136ae6f4f0 con 0x56136ae4c6b0 2024-08-03T20:57:59.077 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:59.075+0000 7f6586c00640 1 --2- 172.21.15.32:0/1408529517 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x56136ae54f30 0x56136addeaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-03T20:57:59.077 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:59.075+0000 7f6588000640 1 --2- 172.21.15.32:0/1408529517 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56136ae4c6b0 0x56136addb220 secure :-1 s=READY pgs=1678 cs=0 l=1 rev1=1 crypto rx=0x7f6580025c70 tx=0x7f6580003330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d4e0ba1788333c24 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:57:59.078 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:59.075+0000 7f6585800640 1 -- 172.21.15.32:0/1408529517 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6580064070 con 0x56136ae4c6b0 2024-08-03T20:57:59.078 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:59.075+0000 7f6585800640 1 -- 172.21.15.32:0/1408529517 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f658004d8f0 con 0x56136ae4c6b0 2024-08-03T20:57:59.078 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:59.075+0000 7f6585800640 1 -- 172.21.15.32:0/1408529517 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f658004dd60 con 0x56136ae4c6b0 2024-08-03T20:57:59.078 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:59.076+0000 7f658f020f40 1 -- 172.21.15.32:0/1408529517 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56136ae4c6b0 msgr2=0x56136addb220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:59.078 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:59.076+0000 7f658f020f40 1 --2- 172.21.15.32:0/1408529517 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56136ae4c6b0 0x56136addb220 secure :-1 s=READY pgs=1678 cs=0 l=1 rev1=1 crypto rx=0x7f6580025c70 tx=0x7f6580003330 comp rx=0 tx=0).stop 2024-08-03T20:57:59.078 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:59.076+0000 7f658f020f40 1 -- 172.21.15.32:0/1408529517 shutdown_connections 2024-08-03T20:57:59.078 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:59.076+0000 7f658f020f40 1 --2- 172.21.15.32:0/1408529517 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56136ae4c6b0 0x56136addb220 unknown :-1 s=CLOSED pgs=1678 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:59.079 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:59.076+0000 7f658f020f40 1 --2- 172.21.15.32:0/1408529517 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x56136ae54f30 0x56136addeaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:59.079 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:59.076+0000 7f658f020f40 1 --2- 172.21.15.32:0/1408529517 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56136addd2f0 0x56136adda9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:59.079 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:59.076+0000 7f658f020f40 1 -- 172.21.15.32:0/1408529517 >> 172.21.15.32:0/1408529517 conn(0x56136acb9ac0 msgr2=0x56136ae731b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:57:59.079 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:59.076+0000 7f658f020f40 1 -- 172.21.15.32:0/1408529517 shutdown_connections 2024-08-03T20:57:59.079 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:59.076+0000 7f658f020f40 1 -- 172.21.15.32:0/1408529517 wait complete. 2024-08-03T20:57:59.079 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:59.077+0000 7f658f020f40 1 Processor -- start 2024-08-03T20:57:59.079 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:59.077+0000 7f658f020f40 1 -- start start 2024-08-03T20:57:59.081 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:59.079+0000 7f658f020f40 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x56136addd2f0 0x56136addb3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:59.081 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:59.079+0000 7f658f020f40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56136ae4c6b0 0x56136ade2690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:59.081 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:59.079+0000 7f658f020f40 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56136ae54f30 0x56136ade2bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:59.081 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:59.079+0000 7f658f020f40 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x56136addaf10 con 0x56136ae54f30 2024-08-03T20:57:59.081 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:59.079+0000 7f658f020f40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x56136ae519d0 con 0x56136ae4c6b0 2024-08-03T20:57:59.081 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:59.079+0000 7f658f020f40 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x56136ae48c80 con 0x56136addd2f0 2024-08-03T20:57:59.081 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:59.079+0000 7f6588000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56136ae54f30 0x56136ade2bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:59.081 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:59.079+0000 7f6588000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56136ae54f30 0x56136ade2bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:51500/0 (socket says 172.21.15.32:51500) 2024-08-03T20:57:59.081 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:59.079+0000 7f6588000640 1 -- 172.21.15.32:0/2533858267 learned_addr learned my addr 172.21.15.32:0/2533858267 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:57:59.082 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:59.079+0000 7f6587600640 1 --2- 172.21.15.32:0/2533858267 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x56136addd2f0 0x56136addb3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:59.082 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:59.079+0000 7f6586c00640 1 --2- 172.21.15.32:0/2533858267 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56136ae4c6b0 0x56136ade2690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:59.082 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:59.079+0000 7f6588000640 1 -- 172.21.15.32:0/2533858267 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x56136addd2f0 msgr2=0x56136addb3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:59.082 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:59.079+0000 7f6588000640 1 --2- 172.21.15.32:0/2533858267 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x56136addd2f0 0x56136addb3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:59.082 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:59.079+0000 7f6588000640 1 -- 172.21.15.32:0/2533858267 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56136ae4c6b0 msgr2=0x56136ade2690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:59.082 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:59.079+0000 7f6588000640 1 --2- 172.21.15.32:0/2533858267 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56136ae4c6b0 0x56136ade2690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:59.082 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:59.079+0000 7f6588000640 1 -- 172.21.15.32:0/2533858267 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56136ae4abd0 con 0x56136ae54f30 2024-08-03T20:57:59.082 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:59.079+0000 7f6587600640 1 --2- 172.21.15.32:0/2533858267 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x56136addd2f0 0x56136addb3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-03T20:57:59.082 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:59.080+0000 7f6588000640 1 --2- 172.21.15.32:0/2533858267 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56136ae54f30 0x56136ade2bd0 secure :-1 s=READY pgs=1679 cs=0 l=1 rev1=1 crypto rx=0x7f6580046370 tx=0x7f658004de20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:57:59.082 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:59.080+0000 7f657fe00640 1 -- 172.21.15.32:0/2533858267 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6580064040 con 0x56136ae54f30 2024-08-03T20:57:59.082 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:59.080+0000 7f657fe00640 1 -- 172.21.15.32:0/2533858267 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f658006d030 con 0x56136ae54f30 2024-08-03T20:57:59.082 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:59.080+0000 7f657fe00640 1 -- 172.21.15.32:0/2533858267 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65800699e0 con 0x56136ae54f30 2024-08-03T20:57:59.082 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:59.080+0000 7f658f020f40 1 -- 172.21.15.32:0/2533858267 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x56136ae70680 con 0x56136ae54f30 2024-08-03T20:57:59.083 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:59.080+0000 7f658f020f40 1 -- 172.21.15.32:0/2533858267 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56136ae489c0 con 0x56136ae54f30 2024-08-03T20:57:59.083 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:59.082+0000 7f657fe00640 1 -- 172.21.15.32:0/2533858267 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f6580067070 con 0x56136ae54f30 2024-08-03T20:57:59.084 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:59.083+0000 7f657fe00640 1 --2- 172.21.15.32:0/2533858267 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f65680759f0 0x7f6568077f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:59.084 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:59.083+0000 7f657fe00640 1 -- 172.21.15.32:0/2533858267 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(110..110 src has 1..110) ==== 8004+0+0 (secure 0 0 0) 0x7f65800dd540 con 0x56136ae54f30 2024-08-03T20:57:59.084 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:59.084+0000 7f6587600640 1 --2- 172.21.15.32:0/2533858267 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f65680759f0 0x7f6568077f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:59.085 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:59.084+0000 7f658f020f40 1 --2- 172.21.15.32:0/2533858267 >> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] conn(0x56136addeaa0 0x56136af0a070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:59.085 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:59.084+0000 7f658f020f40 1 -- 172.21.15.32:0/2533858267 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:1 12.14 12:2e2fa760:::200.00000001:head [truncate 0] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110) -- 0x56136addb910 con 0x56136addeaa0 2024-08-03T20:57:59.085 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:59.084+0000 7f6586c00640 1 --2- 172.21.15.32:0/2533858267 >> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] conn(0x56136addeaa0 0x56136af0a070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:59.085 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:59.084+0000 7f6587600640 1 --2- 172.21.15.32:0/2533858267 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f65680759f0 0x7f6568077f00 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7f6578001120 tx=0x7f65780029f0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:57:59.085 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:59.085+0000 7f6586c00640 1 --2- 172.21.15.32:0/2533858267 >> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] conn(0x56136addeaa0 0x56136af0a070 crc :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:57:59.088 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:59.087+0000 7f6586c00640 1 -- 172.21.15.32:0/2533858267 <== osd.5 v2:172.21.15.118:6810/2724891770 1 ==== osd_op_reply(1 200.00000001 [truncate 0] v110'570 uv570 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f657400f7d0 con 0x56136addeaa0 2024-08-03T20:57:59.088 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:59.087+0000 7f658f020f40 1 -- 172.21.15.32:0/2533858267 >> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] conn(0x56136addeaa0 msgr2=0x56136af0a070 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:59.088 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:59.087+0000 7f658f020f40 1 --2- 172.21.15.32:0/2533858267 >> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] conn(0x56136addeaa0 0x56136af0a070 crc :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:59.088 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:59.087+0000 7f658f020f40 1 -- 172.21.15.32:0/2533858267 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f65680759f0 msgr2=0x7f6568077f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:59.088 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:59.087+0000 7f658f020f40 1 --2- 172.21.15.32:0/2533858267 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f65680759f0 0x7f6568077f00 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7f6578001120 tx=0x7f65780029f0 comp rx=0 tx=0).stop 2024-08-03T20:57:59.088 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:59.087+0000 7f658f020f40 1 -- 172.21.15.32:0/2533858267 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56136ae54f30 msgr2=0x56136ade2bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:59.088 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:59.087+0000 7f658f020f40 1 --2- 172.21.15.32:0/2533858267 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56136ae54f30 0x56136ade2bd0 secure :-1 s=READY pgs=1679 cs=0 l=1 rev1=1 crypto rx=0x7f6580046370 tx=0x7f658004de20 comp rx=0 tx=0).stop 2024-08-03T20:57:59.088 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:59.087+0000 7f6588000640 1 -- 172.21.15.32:0/2533858267 reap_dead start 2024-08-03T20:57:59.088 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:59.087+0000 7f658f020f40 1 -- 172.21.15.32:0/2533858267 shutdown_connections 2024-08-03T20:57:59.088 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:59.087+0000 7f658f020f40 1 -- 172.21.15.32:0/2533858267 >> 172.21.15.32:0/2533858267 conn(0x56136acb9ac0 msgr2=0x56136ae727f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:57:59.089 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:59.087+0000 7f658f020f40 1 -- 172.21.15.32:0/2533858267 shutdown_connections 2024-08-03T20:57:59.089 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:59.087+0000 7f658f020f40 1 -- 172.21.15.32:0/2533858267 wait complete. 2024-08-03T20:57:59.099 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs joinable true 2024-08-03T20:57:59.331 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:59.330+0000 7fe67ea00640 1 -- 172.21.15.32:0/2271950058 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fe678160520 msgr2=0x7fe6781671a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:59.331 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:59.330+0000 7fe67ea00640 1 --2- 172.21.15.32:0/2271950058 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fe678160520 0x7fe6781671a0 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7fe6780cb2c0 tx=0x7fe67002a8b0 comp rx=0 tx=0).stop 2024-08-03T20:57:59.331 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:59.330+0000 7fe67ea00640 1 -- 172.21.15.32:0/2271950058 shutdown_connections 2024-08-03T20:57:59.331 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:59.330+0000 7fe67ea00640 1 --2- 172.21.15.32:0/2271950058 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fe678160520 0x7fe6781671a0 unknown :-1 s=CLOSED pgs=524 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:59.331 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:59.330+0000 7fe67ea00640 1 -- 172.21.15.32:0/2271950058 >> 172.21.15.32:0/2271950058 conn(0x7fe6780cfd80 msgr2=0x7fe6780d0190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:57:59.331 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:59.330+0000 7fe67ea00640 1 -- 172.21.15.32:0/2271950058 shutdown_connections 2024-08-03T20:57:59.331 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:59.330+0000 7fe67ea00640 1 -- 172.21.15.32:0/2271950058 wait complete. 2024-08-03T20:57:59.333 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:59.332+0000 7fe67ea00640 1 Processor -- start 2024-08-03T20:57:59.333 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:59.332+0000 7fe67ea00640 1 -- start start 2024-08-03T20:57:59.336 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:59.334+0000 7fe67ea00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe678159ff0 0x7fe678147ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:59.336 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:59.334+0000 7fe67ea00640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fe6781609d0 0x7fe6781555e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:59.336 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:59.334+0000 7fe67ea00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fe678148330 0x7fe678155b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:59.336 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:59.334+0000 7fe67ea00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fe678167a70 con 0x7fe678159ff0 2024-08-03T20:57:59.336 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:59.334+0000 7fe67ea00640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7fe6781678f0 con 0x7fe678148330 2024-08-03T20:57:59.336 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:59.334+0000 7fe67ea00640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7fe678167bf0 con 0x7fe6781609d0 2024-08-03T20:57:59.336 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:59.334+0000 7fe67d600640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe678159ff0 0x7fe678147ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:59.336 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:59.334+0000 7fe67d600640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe678159ff0 0x7fe678147ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:51504/0 (socket says 172.21.15.32:51504) 2024-08-03T20:57:59.336 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:59.334+0000 7fe67d600640 1 -- 172.21.15.32:0/2833171873 learned_addr learned my addr 172.21.15.32:0/2833171873 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:57:59.336 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:59.334+0000 7fe67e000640 1 --2- 172.21.15.32:0/2833171873 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fe678148330 0x7fe678155b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:59.336 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:59.334+0000 7fe67cc00640 1 --2- 172.21.15.32:0/2833171873 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fe6781609d0 0x7fe6781555e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:59.337 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:59.335+0000 7fe67d600640 1 -- 172.21.15.32:0/2833171873 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fe6781609d0 msgr2=0x7fe6781555e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:59.337 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:59.335+0000 7fe67d600640 1 --2- 172.21.15.32:0/2833171873 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fe6781609d0 0x7fe6781555e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:59.337 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:59.335+0000 7fe67d600640 1 -- 172.21.15.32:0/2833171873 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fe678148330 msgr2=0x7fe678155b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:57:59.337 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:59.335+0000 7fe67d600640 1 --2- 172.21.15.32:0/2833171873 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fe678148330 0x7fe678155b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:57:59.337 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:59.335+0000 7fe67d600640 1 -- 172.21.15.32:0/2833171873 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe678150f60 con 0x7fe678159ff0 2024-08-03T20:57:59.337 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:59.335+0000 7fe67cc00640 1 --2- 172.21.15.32:0/2833171873 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fe6781609d0 0x7fe6781555e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-03T20:57:59.337 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:59.335+0000 7fe67d600640 1 --2- 172.21.15.32:0/2833171873 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe678159ff0 0x7fe678147ca0 secure :-1 s=READY pgs=1681 cs=0 l=1 rev1=1 crypto rx=0x7fe66c00c460 tx=0x7fe66c00c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:57:59.337 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:59.335+0000 7fe675600640 1 -- 172.21.15.32:0/2833171873 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe66c0052d0 con 0x7fe678159ff0 2024-08-03T20:57:59.337 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:59.335+0000 7fe675600640 1 -- 172.21.15.32:0/2833171873 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fe66c005490 con 0x7fe678159ff0 2024-08-03T20:57:59.337 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:59.335+0000 7fe675600640 1 -- 172.21.15.32:0/2833171873 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe66c012870 con 0x7fe678159ff0 2024-08-03T20:57:59.337 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:59.335+0000 7fe67ea00640 1 -- 172.21.15.32:0/2833171873 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe678151240 con 0x7fe678159ff0 2024-08-03T20:57:59.337 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:59.335+0000 7fe67ea00640 1 -- 172.21.15.32:0/2833171873 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe67815c240 con 0x7fe678159ff0 2024-08-03T20:57:59.338 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:59.337+0000 7fe67ea00640 1 -- 172.21.15.32:0/2833171873 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe6780cc050 con 0x7fe678159ff0 2024-08-03T20:57:59.338 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:59.338+0000 7fe675600640 1 -- 172.21.15.32:0/2833171873 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7fe66c012a10 con 0x7fe678159ff0 2024-08-03T20:57:59.340 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:59.339+0000 7fe675600640 1 --2- 172.21.15.32:0/2833171873 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fe658077040 0x7fe658079550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:57:59.340 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:59.340+0000 7fe675600640 1 -- 172.21.15.32:0/2833171873 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(110..110 src has 1..110) ==== 8004+0+0 (secure 0 0 0) 0x7fe66c098c40 con 0x7fe678159ff0 2024-08-03T20:57:59.340 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:59.340+0000 7fe67cc00640 1 --2- 172.21.15.32:0/2833171873 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fe658077040 0x7fe658079550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:57:59.341 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:59.341+0000 7fe67cc00640 1 --2- 172.21.15.32:0/2833171873 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fe658077040 0x7fe658079550 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7fe6780ccf00 tx=0x7fe664009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:57:59.345 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:59.344+0000 7fe675600640 1 -- 172.21.15.32:0/2833171873 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7fe66c0615c0 con 0x7fe678159ff0 2024-08-03T20:57:59.588 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:57:59.587+0000 7fe67ea00640 1 -- 172.21.15.32:0/2833171873 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "joinable", "val": "true"} v 0) -- 0x7fe678151410 con 0x7fe678159ff0 2024-08-03T20:57:59.814 INFO:tasks.ceph.mds.a.smithi032.stderr: -24> 2024-08-03T20:57:55.060+0000 7f87e3c00640 -1 log_channel(cluster) log [ERR] : Error loading MDS rank 0: (22) Invalid argument 2024-08-03T20:57:59.925 INFO:tasks.ceph.mds.a.smithi032.stdout:starting mds.a at 2024-08-03T20:58:00.051 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:00.050+0000 7fe675600640 1 -- 172.21.15.32:0/2833171873 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "joinable", "val": "true"}]=0 cephfs marked joinable; MDS may join as newly active. v237) ==== 162+0+0 (secure 0 0 0) 0x7fe678151410 con 0x7fe678159ff0 2024-08-03T20:58:00.051 INFO:teuthology.orchestra.run.smithi032.stderr:cephfs marked joinable; MDS may join as newly active. 2024-08-03T20:58:00.053 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:00.052+0000 7fe67ea00640 1 -- 172.21.15.32:0/2833171873 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fe658077040 msgr2=0x7fe658079550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:58:00.053 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:00.052+0000 7fe67ea00640 1 --2- 172.21.15.32:0/2833171873 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fe658077040 0x7fe658079550 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7fe6780ccf00 tx=0x7fe664009040 comp rx=0 tx=0).stop 2024-08-03T20:58:00.053 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:00.052+0000 7fe67ea00640 1 -- 172.21.15.32:0/2833171873 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe678159ff0 msgr2=0x7fe678147ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:58:00.054 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:00.053+0000 7fe67ea00640 1 --2- 172.21.15.32:0/2833171873 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe678159ff0 0x7fe678147ca0 secure :-1 s=READY pgs=1681 cs=0 l=1 rev1=1 crypto rx=0x7fe66c00c460 tx=0x7fe66c00c920 comp rx=0 tx=0).stop 2024-08-03T20:58:00.054 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:00.053+0000 7fe67ea00640 1 -- 172.21.15.32:0/2833171873 shutdown_connections 2024-08-03T20:58:00.054 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:00.053+0000 7fe67ea00640 1 --2- 172.21.15.32:0/2833171873 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fe658077040 0x7fe658079550 unknown :-1 s=CLOSED pgs=686 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:00.054 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:00.053+0000 7fe67ea00640 1 --2- 172.21.15.32:0/2833171873 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fe678148330 0x7fe678155b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:00.054 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:00.053+0000 7fe67ea00640 1 --2- 172.21.15.32:0/2833171873 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fe6781609d0 0x7fe6781555e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:00.054 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:00.053+0000 7fe67ea00640 1 --2- 172.21.15.32:0/2833171873 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe678159ff0 0x7fe678147ca0 unknown :-1 s=CLOSED pgs=1681 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:00.054 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:00.053+0000 7fe67ea00640 1 -- 172.21.15.32:0/2833171873 >> 172.21.15.32:0/2833171873 conn(0x7fe6780cfd80 msgr2=0x7fe67815e900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:58:00.054 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:00.053+0000 7fe67ea00640 1 -- 172.21.15.32:0/2833171873 shutdown_connections 2024-08-03T20:58:00.054 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:00.053+0000 7fe67ea00640 1 -- 172.21.15.32:0/2833171873 wait complete. 2024-08-03T20:58:00.067 INFO:tasks.ceph.mds.a.smithi032.stderr:2024-08-03T20:58:00.067+0000 7fce10a00640 -1 log_channel(cluster) log [ERR] : Error loading MDS rank 0: (22) Invalid argument 2024-08-03T20:58:00.068 INFO:tasks.ceph.mon.a.smithi032.stderr:2024-08-03T20:58:00.068+0000 7f4e6d600640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-08-03T20:58:00.068 INFO:tasks.ceph.mon.a.smithi032.stderr:2024-08-03T20:58:00.068+0000 7f4e6d600640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 mds daemon damaged (MDS_DAMAGE) 2024-08-03T20:58:00.080 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-08-03T20:58:00.310 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:00.309+0000 7f9fe0200640 1 -- 172.21.15.32:0/973808821 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9fc0003a00 msgr2=0x7f9fc0003e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:58:00.310 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:00.309+0000 7f9fe0200640 1 --2- 172.21.15.32:0/973808821 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9fc0003a00 0x7f9fc0003e70 secure :-1 s=READY pgs=1686 cs=0 l=1 rev1=1 crypto rx=0x7f9fcc00c4a0 tx=0x7f9fc0005830 comp rx=0 tx=0).stop 2024-08-03T20:58:00.310 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:00.309+0000 7f9fde200640 1 -- 172.21.15.32:0/973808821 reap_dead start 2024-08-03T20:58:00.310 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:00.309+0000 7f9fe0200640 1 -- 172.21.15.32:0/973808821 shutdown_connections 2024-08-03T20:58:00.310 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:00.309+0000 7f9fe0200640 1 -- 172.21.15.32:0/973808821 >> 172.21.15.32:0/973808821 conn(0x7f9fd80cd1e0 msgr2=0x7f9fd80cd5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:58:00.310 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:00.309+0000 7f9fe0200640 1 -- 172.21.15.32:0/973808821 shutdown_connections 2024-08-03T20:58:00.311 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:00.309+0000 7f9fe0200640 1 -- 172.21.15.32:0/973808821 wait complete. 2024-08-03T20:58:00.312 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:00.311+0000 7f9fe0200640 1 Processor -- start 2024-08-03T20:58:00.312 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:00.311+0000 7f9fe0200640 1 -- start start 2024-08-03T20:58:00.315 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:00.313+0000 7f9fe0200640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f9fc0003a00 0x7f9fd80c3410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:58:00.315 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:00.313+0000 7f9fe0200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9fd814fad0 0x7f9fd80b5600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:58:00.315 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:00.313+0000 7f9fe0200640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f9fd80b5b40 0x7f9fd81f6a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:58:00.315 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:00.313+0000 7f9fe0200640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f9fd80cfae0 con 0x7f9fd814fad0 2024-08-03T20:58:00.315 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:00.313+0000 7f9fe0200640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f9fd80cf960 con 0x7f9fd80b5b40 2024-08-03T20:58:00.315 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:00.313+0000 7f9fe0200640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7f9fd80cfc60 con 0x7f9fc0003a00 2024-08-03T20:58:00.315 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:00.313+0000 7f9fdce00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9fd814fad0 0x7f9fd80b5600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:58:00.316 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:00.313+0000 7f9fdce00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9fd814fad0 0x7f9fd80b5600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:51002/0 (socket says 172.21.15.32:51002) 2024-08-03T20:58:00.316 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:00.313+0000 7f9fdce00640 1 -- 172.21.15.32:0/3316910856 learned_addr learned my addr 172.21.15.32:0/3316910856 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:58:00.316 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:00.314+0000 7f9fdd800640 1 --2- 172.21.15.32:0/3316910856 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f9fc0003a00 0x7f9fd80c3410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:58:00.316 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:00.314+0000 7f9fde200640 1 --2- 172.21.15.32:0/3316910856 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f9fd80b5b40 0x7f9fd81f6a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:58:00.316 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:00.314+0000 7f9fdce00640 1 -- 172.21.15.32:0/3316910856 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f9fc0003a00 msgr2=0x7f9fd80c3410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:58:00.316 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:00.314+0000 7f9fdce00640 1 --2- 172.21.15.32:0/3316910856 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f9fc0003a00 0x7f9fd80c3410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:00.316 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:00.314+0000 7f9fdce00640 1 -- 172.21.15.32:0/3316910856 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f9fd80b5b40 msgr2=0x7f9fd81f6a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:58:00.316 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:00.314+0000 7f9fdce00640 1 --2- 172.21.15.32:0/3316910856 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f9fd80b5b40 0x7f9fd81f6a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:00.316 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:00.314+0000 7f9fdce00640 1 -- 172.21.15.32:0/3316910856 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9fd80b5f50 con 0x7f9fd814fad0 2024-08-03T20:58:00.316 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:00.314+0000 7f9fdd800640 1 --2- 172.21.15.32:0/3316910856 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f9fc0003a00 0x7f9fd80c3410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-03T20:58:00.316 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:00.314+0000 7f9fde200640 1 --2- 172.21.15.32:0/3316910856 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f9fd80b5b40 0x7f9fd81f6a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-03T20:58:00.316 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:00.314+0000 7f9fdce00640 1 --2- 172.21.15.32:0/3316910856 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9fd814fad0 0x7f9fd80b5600 secure :-1 s=READY pgs=1687 cs=0 l=1 rev1=1 crypto rx=0x7f9fc4011a90 tx=0x7f9fc4011f50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:58:00.317 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:00.314+0000 7f9fd5600640 1 -- 172.21.15.32:0/3316910856 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9fc400fd90 con 0x7f9fd814fad0 2024-08-03T20:58:00.317 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:00.314+0000 7f9fd5600640 1 -- 172.21.15.32:0/3316910856 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f9fc4004020 con 0x7f9fd814fad0 2024-08-03T20:58:00.317 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:00.314+0000 7f9fd5600640 1 -- 172.21.15.32:0/3316910856 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9fc4004430 con 0x7f9fd814fad0 2024-08-03T20:58:00.317 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:00.314+0000 7f9fe0200640 1 -- 172.21.15.32:0/3316910856 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9fd81f3450 con 0x7f9fd814fad0 2024-08-03T20:58:00.317 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:00.314+0000 7f9fe0200640 1 -- 172.21.15.32:0/3316910856 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9fd81f3c30 con 0x7f9fd814fad0 2024-08-03T20:58:00.317 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:00.316+0000 7f9fe0200640 1 -- 172.21.15.32:0/3316910856 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9fd8152a00 con 0x7f9fd814fad0 2024-08-03T20:58:00.318 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:00.317+0000 7f9fd5600640 1 -- 172.21.15.32:0/3316910856 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f9fc40035f0 con 0x7f9fd814fad0 2024-08-03T20:58:00.318 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:00.318+0000 7f9fd5600640 1 --2- 172.21.15.32:0/3316910856 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f9fb8077010 0x7f9fb8079520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:58:00.318 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:00.318+0000 7f9fd5600640 1 -- 172.21.15.32:0/3316910856 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(111..111 src has 1..111) ==== 8090+0+0 (secure 0 0 0) 0x7f9fc4099f00 con 0x7f9fd814fad0 2024-08-03T20:58:00.319 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:00.318+0000 7f9fdd800640 1 --2- 172.21.15.32:0/3316910856 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f9fb8077010 0x7f9fb8079520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:58:00.319 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:00.319+0000 7f9fdd800640 1 --2- 172.21.15.32:0/3316910856 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f9fb8077010 0x7f9fb8079520 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7f9fcc00c4a0 tx=0x7f9fcc038040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:58:00.323 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:00.323+0000 7f9fd5600640 1 -- 172.21.15.32:0/3316910856 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7f9fd80b5f50 con 0x7f9fd814fad0 2024-08-03T20:58:00.568 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:00.567+0000 7f9fe0200640 1 -- 172.21.15.32:0/3316910856 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f9fd81f36e0 con 0x7f9fd814fad0 2024-08-03T20:58:00.569 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:00.568+0000 7f9fd5600640 1 -- 172.21.15.32:0/3316910856 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 238 v238) ==== 95+0+4391 (secure 0 0 0) 0x7f9fc4062710 con 0x7f9fd814fad0 2024-08-03T20:58:00.569 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-03T20:58:00.569 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":238,"btime":"2024-08-03T20:58:00:068381+0000","default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6439,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6835/531080160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6834","nonce":531080160},{"type":"v1","addr":"172.21.15.118:6835","nonce":531080160}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":84},{"gid":6454,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6837/2796882319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6836","nonce":2796882319},{"type":"v1","addr":"172.21.15.118:6837","nonce":2796882319}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":86},{"gid":6813,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6834/2686933890","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":2686933890},{"type":"v1","addr":"172.21.15.32:6834","nonce":2686933890}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":85}],"filesystems":[{"mdsmap":{"epoch":238,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-08-03T20:54:26.950518+0000","modified":"2024-08-03T20:58:00.068318+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":111,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":6}]} 2024-08-03T20:58:00.569 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 238 2024-08-03T20:58:00.571 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:00.570+0000 7f9fe0200640 1 -- 172.21.15.32:0/3316910856 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f9fb8077010 msgr2=0x7f9fb8079520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:58:00.572 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:00.570+0000 7f9fe0200640 1 --2- 172.21.15.32:0/3316910856 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f9fb8077010 0x7f9fb8079520 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7f9fcc00c4a0 tx=0x7f9fcc038040 comp rx=0 tx=0).stop 2024-08-03T20:58:00.572 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:00.570+0000 7f9fe0200640 1 -- 172.21.15.32:0/3316910856 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9fd814fad0 msgr2=0x7f9fd80b5600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:58:00.572 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:00.570+0000 7f9fe0200640 1 --2- 172.21.15.32:0/3316910856 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9fd814fad0 0x7f9fd80b5600 secure :-1 s=READY pgs=1687 cs=0 l=1 rev1=1 crypto rx=0x7f9fc4011a90 tx=0x7f9fc4011f50 comp rx=0 tx=0).stop 2024-08-03T20:58:00.572 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:00.571+0000 7f9fe0200640 1 -- 172.21.15.32:0/3316910856 shutdown_connections 2024-08-03T20:58:00.572 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:00.571+0000 7f9fe0200640 1 --2- 172.21.15.32:0/3316910856 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f9fb8077010 0x7f9fb8079520 unknown :-1 s=CLOSED pgs=688 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:00.572 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:00.571+0000 7f9fe0200640 1 --2- 172.21.15.32:0/3316910856 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f9fd80b5b40 0x7f9fd81f6a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:00.572 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:00.571+0000 7f9fe0200640 1 --2- 172.21.15.32:0/3316910856 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9fd814fad0 0x7f9fd80b5600 unknown :-1 s=CLOSED pgs=1687 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:00.572 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:00.571+0000 7f9fe0200640 1 --2- 172.21.15.32:0/3316910856 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f9fc0003a00 0x7f9fd80c3410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:00.572 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:00.571+0000 7f9fe0200640 1 -- 172.21.15.32:0/3316910856 >> 172.21.15.32:0/3316910856 conn(0x7f9fd80cd1e0 msgr2=0x7f9fd80ca360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:58:00.572 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:00.571+0000 7f9fe0200640 1 -- 172.21.15.32:0/3316910856 shutdown_connections 2024-08-03T20:58:00.572 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:00.571+0000 7f9fe0200640 1 -- 172.21.15.32:0/3316910856 wait complete. 2024-08-03T20:58:00.598 DEBUG:tasks.ceph_test_case:wait_until_true: success in 0s and 0 retries 2024-08-03T20:58:00.599 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-08-03T20:58:00.826 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:00.825+0000 7f4dbf400640 1 -- 172.21.15.32:0/2796415472 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4db806e930 msgr2=0x7f4db8144f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:58:00.826 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:00.825+0000 7f4dbf400640 1 --2- 172.21.15.32:0/2796415472 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4db806e930 0x7f4db8144f90 secure :-1 s=READY pgs=1688 cs=0 l=1 rev1=1 crypto rx=0x7f4dac005fd0 tx=0x7f4dac02a5d0 comp rx=0 tx=0).stop 2024-08-03T20:58:00.827 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:00.825+0000 7f4dbf400640 1 -- 172.21.15.32:0/2796415472 shutdown_connections 2024-08-03T20:58:00.827 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:00.826+0000 7f4dbf400640 1 --2- 172.21.15.32:0/2796415472 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f4db815fff0 0x7f4db8160400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:00.827 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:00.826+0000 7f4dbf400640 1 --2- 172.21.15.32:0/2796415472 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4db806e930 0x7f4db8144f90 unknown :-1 s=CLOSED pgs=1688 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:00.827 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:00.826+0000 7f4dbf400640 1 -- 172.21.15.32:0/2796415472 >> 172.21.15.32:0/2796415472 conn(0x7f4db80b1bd0 msgr2=0x7f4db80b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:58:00.827 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:00.826+0000 7f4dbf400640 1 -- 172.21.15.32:0/2796415472 shutdown_connections 2024-08-03T20:58:00.827 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:00.826+0000 7f4dbf400640 1 -- 172.21.15.32:0/2796415472 wait complete. 2024-08-03T20:58:00.828 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:00.828+0000 7f4dbf400640 1 Processor -- start 2024-08-03T20:58:00.828 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:00.828+0000 7f4dbf400640 1 -- start start 2024-08-03T20:58:00.831 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:00.829+0000 7f4dbf400640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f4db806e930 0x7f4db81f6310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:58:00.831 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:00.829+0000 7f4dbf400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4db8073e60 0x7f4db81f9720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:58:00.831 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:00.829+0000 7f4dbf400640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f4db815fff0 0x7f4db81445b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:58:00.831 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:00.829+0000 7f4dbf400640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f4db80cf840 con 0x7f4db8073e60 2024-08-03T20:58:00.831 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:00.829+0000 7f4dbf400640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f4db8145f30 con 0x7f4db815fff0 2024-08-03T20:58:00.831 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:00.829+0000 7f4dbf400640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7f4db80cf9c0 con 0x7f4db806e930 2024-08-03T20:58:00.831 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:00.830+0000 7f4db7e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4db8073e60 0x7f4db81f9720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:58:00.831 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:00.830+0000 7f4db7e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4db8073e60 0x7f4db81f9720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:51026/0 (socket says 172.21.15.32:51026) 2024-08-03T20:58:00.831 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:00.830+0000 7f4db7e00640 1 -- 172.21.15.32:0/3172872361 learned_addr learned my addr 172.21.15.32:0/3172872361 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:58:00.831 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:00.830+0000 7f4dbca00640 1 --2- 172.21.15.32:0/3172872361 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f4db806e930 0x7f4db81f6310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:58:00.831 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:00.830+0000 7f4dbd400640 1 --2- 172.21.15.32:0/3172872361 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f4db815fff0 0x7f4db81445b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:58:00.832 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:00.830+0000 7f4db7e00640 1 -- 172.21.15.32:0/3172872361 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f4db806e930 msgr2=0x7f4db81f6310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:58:00.832 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:00.830+0000 7f4db7e00640 1 --2- 172.21.15.32:0/3172872361 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f4db806e930 0x7f4db81f6310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:00.832 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:00.830+0000 7f4db7e00640 1 -- 172.21.15.32:0/3172872361 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f4db815fff0 msgr2=0x7f4db81445b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:58:00.832 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:00.830+0000 7f4db7e00640 1 --2- 172.21.15.32:0/3172872361 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f4db815fff0 0x7f4db81445b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:00.832 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:00.830+0000 7f4db7e00640 1 -- 172.21.15.32:0/3172872361 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4db81f9c60 con 0x7f4db8073e60 2024-08-03T20:58:00.832 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:00.830+0000 7f4dbca00640 1 --2- 172.21.15.32:0/3172872361 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f4db806e930 0x7f4db81f6310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-03T20:58:00.833 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:00.831+0000 7f4db7e00640 1 --2- 172.21.15.32:0/3172872361 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4db8073e60 0x7f4db81f9720 secure :-1 s=READY pgs=1689 cs=0 l=1 rev1=1 crypto rx=0x7f4da400c340 tx=0x7f4da400c800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:58:00.833 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:00.831+0000 7f4db4c00640 1 -- 172.21.15.32:0/3172872361 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4da4017070 con 0x7f4db8073e60 2024-08-03T20:58:00.833 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:00.831+0000 7f4db4c00640 1 -- 172.21.15.32:0/3172872361 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f4da40053c0 con 0x7f4db8073e60 2024-08-03T20:58:00.833 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:00.831+0000 7f4dbf400640 1 -- 172.21.15.32:0/3172872361 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4db8142d50 con 0x7f4db8073e60 2024-08-03T20:58:00.833 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:00.831+0000 7f4db4c00640 1 -- 172.21.15.32:0/3172872361 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4da4003850 con 0x7f4db8073e60 2024-08-03T20:58:00.833 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:00.831+0000 7f4dbf400640 1 -- 172.21.15.32:0/3172872361 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4db81f5830 con 0x7f4db8073e60 2024-08-03T20:58:00.833 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:00.832+0000 7f4dbf400640 1 -- 172.21.15.32:0/3172872361 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4db8160c50 con 0x7f4db8073e60 2024-08-03T20:58:00.834 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:00.833+0000 7f4db4c00640 1 -- 172.21.15.32:0/3172872361 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f4da401a070 con 0x7f4db8073e60 2024-08-03T20:58:00.835 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:00.834+0000 7f4db4c00640 1 --2- 172.21.15.32:0/3172872361 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f4d98077040 0x7f4d98079550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:58:00.835 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:00.834+0000 7f4db4c00640 1 -- 172.21.15.32:0/3172872361 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(111..111 src has 1..111) ==== 8090+0+0 (secure 0 0 0) 0x7f4da4098650 con 0x7f4db8073e60 2024-08-03T20:58:00.835 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:00.835+0000 7f4dbca00640 1 --2- 172.21.15.32:0/3172872361 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f4d98077040 0x7f4d98079550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:58:00.836 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:00.836+0000 7f4dbca00640 1 --2- 172.21.15.32:0/3172872361 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f4d98077040 0x7f4d98079550 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x7f4dac033000 tx=0x7f4dac004730 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:58:00.839 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:00.839+0000 7f4db4c00640 1 -- 172.21.15.32:0/3172872361 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7f4db81f9c60 con 0x7f4db8073e60 2024-08-03T20:58:01.086 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:01.086+0000 7f4dbf400640 1 -- 172.21.15.32:0/3172872361 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f4db81f5f50 con 0x7f4db8073e60 2024-08-03T20:58:01.087 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:01.086+0000 7f4db4c00640 1 -- 172.21.15.32:0/3172872361 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 238 v238) ==== 95+0+4391 (secure 0 0 0) 0x7f4da40690b0 con 0x7f4db8073e60 2024-08-03T20:58:01.087 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-03T20:58:01.087 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":238,"btime":"2024-08-03T20:58:00:068381+0000","default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6439,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6835/531080160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6834","nonce":531080160},{"type":"v1","addr":"172.21.15.118:6835","nonce":531080160}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":84},{"gid":6454,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6837/2796882319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6836","nonce":2796882319},{"type":"v1","addr":"172.21.15.118:6837","nonce":2796882319}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":86},{"gid":6813,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6834/2686933890","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":2686933890},{"type":"v1","addr":"172.21.15.32:6834","nonce":2686933890}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":85}],"filesystems":[{"mdsmap":{"epoch":238,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-08-03T20:54:26.950518+0000","modified":"2024-08-03T20:58:00.068318+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":111,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":6}]} 2024-08-03T20:58:01.087 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 238 2024-08-03T20:58:01.090 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:01.089+0000 7f4dbf400640 1 -- 172.21.15.32:0/3172872361 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f4d98077040 msgr2=0x7f4d98079550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:58:01.090 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:01.089+0000 7f4dbf400640 1 --2- 172.21.15.32:0/3172872361 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f4d98077040 0x7f4d98079550 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x7f4dac033000 tx=0x7f4dac004730 comp rx=0 tx=0).stop 2024-08-03T20:58:01.090 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:01.089+0000 7f4dbf400640 1 -- 172.21.15.32:0/3172872361 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4db8073e60 msgr2=0x7f4db81f9720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:58:01.090 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:01.089+0000 7f4dbf400640 1 --2- 172.21.15.32:0/3172872361 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4db8073e60 0x7f4db81f9720 secure :-1 s=READY pgs=1689 cs=0 l=1 rev1=1 crypto rx=0x7f4da400c340 tx=0x7f4da400c800 comp rx=0 tx=0).stop 2024-08-03T20:58:01.090 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:01.089+0000 7f4dbf400640 1 -- 172.21.15.32:0/3172872361 shutdown_connections 2024-08-03T20:58:01.090 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:01.089+0000 7f4dbf400640 1 --2- 172.21.15.32:0/3172872361 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f4d98077040 0x7f4d98079550 unknown :-1 s=CLOSED pgs=689 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:01.090 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:01.089+0000 7f4dbf400640 1 --2- 172.21.15.32:0/3172872361 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f4db815fff0 0x7f4db81445b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:01.090 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:01.089+0000 7f4dbf400640 1 --2- 172.21.15.32:0/3172872361 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4db8073e60 0x7f4db81f9720 unknown :-1 s=CLOSED pgs=1689 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:01.090 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:01.089+0000 7f4dbf400640 1 --2- 172.21.15.32:0/3172872361 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f4db806e930 0x7f4db81f6310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:01.090 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:01.089+0000 7f4dbf400640 1 -- 172.21.15.32:0/3172872361 >> 172.21.15.32:0/3172872361 conn(0x7f4db80b1bd0 msgr2=0x7f4db80b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:58:01.091 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:01.089+0000 7f4dbf400640 1 -- 172.21.15.32:0/3172872361 shutdown_connections 2024-08-03T20:58:01.091 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:01.089+0000 7f4dbf400640 1 -- 172.21.15.32:0/3172872361 wait complete. 2024-08-03T20:58:01.113 DEBUG:tasks.ceph_test_case:wait_until_true: success in 0s and 0 retries 2024-08-03T20:58:01.113 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-08-03T20:58:01.341 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:01.340+0000 7f9e5d200640 1 -- 172.21.15.32:0/1245564000 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9e580b4ce0 msgr2=0x7f9e5815ef00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:58:01.341 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:01.340+0000 7f9e5d200640 1 --2- 172.21.15.32:0/1245564000 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9e580b4ce0 0x7f9e5815ef00 secure :-1 s=READY pgs=1691 cs=0 l=1 rev1=1 crypto rx=0x7f9e40009830 tx=0x7f9e4002a510 comp rx=0 tx=0).stop 2024-08-03T20:58:01.341 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:01.340+0000 7f9e5d200640 1 -- 172.21.15.32:0/1245564000 shutdown_connections 2024-08-03T20:58:01.341 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:01.340+0000 7f9e5d200640 1 --2- 172.21.15.32:0/1245564000 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9e580b4ce0 0x7f9e5815ef00 unknown :-1 s=CLOSED pgs=1691 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:01.341 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:01.340+0000 7f9e5d200640 1 --2- 172.21.15.32:0/1245564000 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f9e580b38b0 0x7f9e581523f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:01.341 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:01.340+0000 7f9e5d200640 1 -- 172.21.15.32:0/1245564000 >> 172.21.15.32:0/1245564000 conn(0x7f9e580b1bd0 msgr2=0x7f9e580b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:58:01.341 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:01.340+0000 7f9e5d200640 1 -- 172.21.15.32:0/1245564000 shutdown_connections 2024-08-03T20:58:01.341 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:01.340+0000 7f9e5d200640 1 -- 172.21.15.32:0/1245564000 wait complete. 2024-08-03T20:58:01.343 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:01.342+0000 7f9e5d200640 1 Processor -- start 2024-08-03T20:58:01.343 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:01.343+0000 7f9e5d200640 1 -- start start 2024-08-03T20:58:01.345 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:01.344+0000 7f9e5d200640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f9e580b38b0 0x7f9e5813f090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:58:01.345 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:01.344+0000 7f9e5d200640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f9e580b4ce0 0x7f9e58141930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:58:01.345 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:01.344+0000 7f9e5d200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9e5815f440 0x7f9e58141e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:58:01.345 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:01.344+0000 7f9e5d200640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f9e5815c020 con 0x7f9e5815f440 2024-08-03T20:58:01.345 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:01.344+0000 7f9e5d200640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f9e580cf240 con 0x7f9e580b4ce0 2024-08-03T20:58:01.345 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:01.344+0000 7f9e5d200640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7f9e5815c1a0 con 0x7f9e580b38b0 2024-08-03T20:58:01.346 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:01.344+0000 7f9e56800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9e5815f440 0x7f9e58141e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:58:01.346 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:01.344+0000 7f9e56800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9e5815f440 0x7f9e58141e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:51042/0 (socket says 172.21.15.32:51042) 2024-08-03T20:58:01.346 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:01.344+0000 7f9e56800640 1 -- 172.21.15.32:0/2577956495 learned_addr learned my addr 172.21.15.32:0/2577956495 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:58:01.346 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:01.344+0000 7f9e55e00640 1 --2- 172.21.15.32:0/2577956495 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f9e580b38b0 0x7f9e5813f090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:58:01.346 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:01.345+0000 7f9e4fe00640 1 --2- 172.21.15.32:0/2577956495 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f9e580b4ce0 0x7f9e58141930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:58:01.346 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:01.345+0000 7f9e56800640 1 -- 172.21.15.32:0/2577956495 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f9e580b38b0 msgr2=0x7f9e5813f090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:58:01.346 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:01.345+0000 7f9e56800640 1 --2- 172.21.15.32:0/2577956495 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f9e580b38b0 0x7f9e5813f090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:01.346 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:01.345+0000 7f9e56800640 1 -- 172.21.15.32:0/2577956495 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f9e580b4ce0 msgr2=0x7f9e58141930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:58:01.346 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:01.345+0000 7f9e56800640 1 --2- 172.21.15.32:0/2577956495 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f9e580b4ce0 0x7f9e58141930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:01.346 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:01.345+0000 7f9e56800640 1 -- 172.21.15.32:0/2577956495 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9e5813f5d0 con 0x7f9e5815f440 2024-08-03T20:58:01.347 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:01.345+0000 7f9e56800640 1 --2- 172.21.15.32:0/2577956495 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9e5815f440 0x7f9e58141e70 secure :-1 s=READY pgs=1692 cs=0 l=1 rev1=1 crypto rx=0x7f9e5000cb40 tx=0x7f9e5000a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:58:01.347 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:01.345+0000 7f9e4e000640 1 -- 172.21.15.32:0/2577956495 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e50016020 con 0x7f9e5815f440 2024-08-03T20:58:01.347 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:01.346+0000 7f9e4e000640 1 -- 172.21.15.32:0/2577956495 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f9e5000ad90 con 0x7f9e5815f440 2024-08-03T20:58:01.347 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:01.346+0000 7f9e5d200640 1 -- 172.21.15.32:0/2577956495 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9e581f6b30 con 0x7f9e5815f440 2024-08-03T20:58:01.348 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:01.346+0000 7f9e4e000640 1 -- 172.21.15.32:0/2577956495 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e50012680 con 0x7f9e5815f440 2024-08-03T20:58:01.348 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:01.346+0000 7f9e5d200640 1 -- 172.21.15.32:0/2577956495 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9e581f70f0 con 0x7f9e5815f440 2024-08-03T20:58:01.348 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:01.347+0000 7f9e5d200640 1 -- 172.21.15.32:0/2577956495 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9e581f68a0 con 0x7f9e5815f440 2024-08-03T20:58:01.348 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:01.348+0000 7f9e4e000640 1 -- 172.21.15.32:0/2577956495 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f9e50019020 con 0x7f9e5815f440 2024-08-03T20:58:01.349 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:01.349+0000 7f9e4e000640 1 --2- 172.21.15.32:0/2577956495 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f9e34077040 0x7f9e34079550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:58:01.349 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:01.349+0000 7f9e4e000640 1 -- 172.21.15.32:0/2577956495 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(111..111 src has 1..111) ==== 8090+0+0 (secure 0 0 0) 0x7f9e50099830 con 0x7f9e5815f440 2024-08-03T20:58:01.350 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:01.349+0000 7f9e55e00640 1 --2- 172.21.15.32:0/2577956495 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f9e34077040 0x7f9e34079550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:58:01.350 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:01.350+0000 7f9e55e00640 1 --2- 172.21.15.32:0/2577956495 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f9e34077040 0x7f9e34079550 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7f9e48009b30 tx=0x7f9e48005f90 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:58:01.353 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:01.353+0000 7f9e4e000640 1 -- 172.21.15.32:0/2577956495 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7f9e50020020 con 0x7f9e5815f440 2024-08-03T20:58:01.596 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:01.595+0000 7f9e5d200640 1 -- 172.21.15.32:0/2577956495 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f9e5815c650 con 0x7f9e5815f440 2024-08-03T20:58:01.596 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:01.596+0000 7f9e4e000640 1 -- 172.21.15.32:0/2577956495 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 238 v238) ==== 95+0+4391 (secure 0 0 0) 0x7f9e5815f850 con 0x7f9e5815f440 2024-08-03T20:58:01.596 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-03T20:58:01.597 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":238,"btime":"2024-08-03T20:58:00:068381+0000","default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6439,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6835/531080160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6834","nonce":531080160},{"type":"v1","addr":"172.21.15.118:6835","nonce":531080160}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":84},{"gid":6454,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6837/2796882319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6836","nonce":2796882319},{"type":"v1","addr":"172.21.15.118:6837","nonce":2796882319}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":86},{"gid":6813,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6834/2686933890","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":2686933890},{"type":"v1","addr":"172.21.15.32:6834","nonce":2686933890}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":85}],"filesystems":[{"mdsmap":{"epoch":238,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-08-03T20:54:26.950518+0000","modified":"2024-08-03T20:58:00.068318+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":111,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":6}]} 2024-08-03T20:58:01.597 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 238 2024-08-03T20:58:01.599 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:01.598+0000 7f9e5d200640 1 -- 172.21.15.32:0/2577956495 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f9e34077040 msgr2=0x7f9e34079550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:58:01.599 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:01.598+0000 7f9e5d200640 1 --2- 172.21.15.32:0/2577956495 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f9e34077040 0x7f9e34079550 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7f9e48009b30 tx=0x7f9e48005f90 comp rx=0 tx=0).stop 2024-08-03T20:58:01.599 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:01.598+0000 7f9e5d200640 1 -- 172.21.15.32:0/2577956495 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9e5815f440 msgr2=0x7f9e58141e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:58:01.599 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:01.598+0000 7f9e5d200640 1 --2- 172.21.15.32:0/2577956495 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9e5815f440 0x7f9e58141e70 secure :-1 s=READY pgs=1692 cs=0 l=1 rev1=1 crypto rx=0x7f9e5000cb40 tx=0x7f9e5000a060 comp rx=0 tx=0).stop 2024-08-03T20:58:01.599 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:01.598+0000 7f9e5d200640 1 -- 172.21.15.32:0/2577956495 shutdown_connections 2024-08-03T20:58:01.599 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:01.598+0000 7f9e5d200640 1 --2- 172.21.15.32:0/2577956495 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f9e34077040 0x7f9e34079550 unknown :-1 s=CLOSED pgs=690 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:01.599 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:01.598+0000 7f9e5d200640 1 --2- 172.21.15.32:0/2577956495 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f9e580b4ce0 0x7f9e58141930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:01.600 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:01.598+0000 7f9e5d200640 1 --2- 172.21.15.32:0/2577956495 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9e5815f440 0x7f9e58141e70 unknown :-1 s=CLOSED pgs=1692 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:01.600 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:01.598+0000 7f9e5d200640 1 --2- 172.21.15.32:0/2577956495 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f9e580b38b0 0x7f9e5813f090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:01.600 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:01.598+0000 7f9e5d200640 1 -- 172.21.15.32:0/2577956495 >> 172.21.15.32:0/2577956495 conn(0x7f9e580b1bd0 msgr2=0x7f9e58159eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:58:01.600 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:01.598+0000 7f9e5d200640 1 -- 172.21.15.32:0/2577956495 shutdown_connections 2024-08-03T20:58:01.600 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:01.598+0000 7f9e5d200640 1 -- 172.21.15.32:0/2577956495 wait complete. 2024-08-03T20:58:01.621 INFO:tasks.cephfs.test_damage:Result: Mutation 'Truncate 200.00000001' led to DAMAGED state 2024-08-03T20:58:01.621 INFO:tasks.cephfs.test_damage:Applying mutation 'Truncate 500.00000000' 2024-08-03T20:58:01.621 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-08-03T20:58:01.621 DEBUG:teuthology.orchestra.run.smithi032:> dd if=/proc/self/mounts of=/dev/stdout 2024-08-03T20:58:01.657 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053453,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6499208k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 none /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249604k,nr_inodes=812401,mode=700,uid=1000,gid=1316 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6499208k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-08-03T20:58:01.657 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi032.front.sepia.ceph.com 2024-08-03T20:58:01.657 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-08-03T20:58:01.657 DEBUG:teuthology.orchestra.run.smithi032:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-08-03T20:58:01.731 INFO:teuthology.orchestra.run.smithi032.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.0': No such file or directory 2024-08-03T20:58:01.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-03T20:58:01.733 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-08-03T20:58:02.002 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:02.001+0000 7f0193000640 1 -- 172.21.15.32:0/426367878 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f018c0d26e0 msgr2=0x7f018c161b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:58:02.002 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:02.001+0000 7f0193000640 1 --2- 172.21.15.32:0/426367878 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f018c0d26e0 0x7f018c161b80 secure :-1 s=READY pgs=1693 cs=0 l=1 rev1=1 crypto rx=0x7f018c0d3730 tx=0x7f018002a5f0 comp rx=0 tx=0).stop 2024-08-03T20:58:02.002 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:02.001+0000 7f0193000640 1 -- 172.21.15.32:0/426367878 shutdown_connections 2024-08-03T20:58:02.002 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:02.001+0000 7f0193000640 1 --2- 172.21.15.32:0/426367878 >> v2:172.21.15.118:3300/0 conn(0x7f018c0cffe0 0x7f018c0b3e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:02.002 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:02.001+0000 7f0193000640 1 --2- 172.21.15.32:0/426367878 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f018c0d26e0 0x7f018c161b80 unknown :-1 s=CLOSED pgs=1693 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:02.002 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:02.001+0000 7f0193000640 1 -- 172.21.15.32:0/426367878 >> 172.21.15.32:0/426367878 conn(0x7f018c08d7a0 msgr2=0x7f018c09d3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:58:02.002 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:02.001+0000 7f0193000640 1 -- 172.21.15.32:0/426367878 shutdown_connections 2024-08-03T20:58:02.002 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:02.001+0000 7f0193000640 1 -- 172.21.15.32:0/426367878 wait complete. 2024-08-03T20:58:02.004 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:02.003+0000 7f0193000640 1 Processor -- start 2024-08-03T20:58:02.004 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:02.003+0000 7f0193000640 1 -- start start 2024-08-03T20:58:02.007 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:02.005+0000 7f0193000640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f018c0b43d0 0x7f018c1557c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:58:02.007 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:02.005+0000 7f0193000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f018c0cffe0 0x7f018c0cb910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:58:02.007 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:02.005+0000 7f0193000640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f018c0d26e0 0x7f018c06d4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:58:02.007 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:02.005+0000 7f0193000640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f018c0d4100 con 0x7f018c0cffe0 2024-08-03T20:58:02.007 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:02.005+0000 7f0193000640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f018c0d3f80 con 0x7f018c0b43d0 2024-08-03T20:58:02.007 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:02.005+0000 7f0193000640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7f018c0d4280 con 0x7f018c0d26e0 2024-08-03T20:58:02.007 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:02.005+0000 7f0191200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f018c0cffe0 0x7f018c0cb910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:58:02.007 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:02.005+0000 7f0191200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f018c0cffe0 0x7f018c0cb910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:51058/0 (socket says 172.21.15.32:51058) 2024-08-03T20:58:02.007 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:02.005+0000 7f0191200640 1 -- 172.21.15.32:0/111188137 learned_addr learned my addr 172.21.15.32:0/111188137 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:58:02.007 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:02.005+0000 7f0191c00640 1 --2- 172.21.15.32:0/111188137 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f018c0b43d0 0x7f018c1557c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:58:02.007 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:02.005+0000 7f0192600640 1 --2- 172.21.15.32:0/111188137 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f018c0d26e0 0x7f018c06d4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:58:02.007 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:02.005+0000 7f0191200640 1 -- 172.21.15.32:0/111188137 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f018c0d26e0 msgr2=0x7f018c06d4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:58:02.008 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:02.005+0000 7f0191200640 1 --2- 172.21.15.32:0/111188137 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f018c0d26e0 0x7f018c06d4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:02.008 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:02.005+0000 7f0191200640 1 -- 172.21.15.32:0/111188137 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f018c0b43d0 msgr2=0x7f018c1557c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:58:02.008 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:02.005+0000 7f0191200640 1 --2- 172.21.15.32:0/111188137 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f018c0b43d0 0x7f018c1557c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:02.008 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:02.005+0000 7f0191200640 1 -- 172.21.15.32:0/111188137 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f018c155d00 con 0x7f018c0cffe0 2024-08-03T20:58:02.008 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:02.005+0000 7f0192600640 1 --2- 172.21.15.32:0/111188137 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f018c0d26e0 0x7f018c06d4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-03T20:58:02.008 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:02.005+0000 7f0191c00640 1 --2- 172.21.15.32:0/111188137 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f018c0b43d0 0x7f018c1557c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-03T20:58:02.008 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:02.006+0000 7f0191200640 1 --2- 172.21.15.32:0/111188137 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f018c0cffe0 0x7f018c0cb910 secure :-1 s=READY pgs=1694 cs=0 l=1 rev1=1 crypto rx=0x7f017800d6d0 tx=0x7f017800db90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:58:02.008 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:02.006+0000 7f0189600640 1 -- 172.21.15.32:0/111188137 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0178017070 con 0x7f018c0cffe0 2024-08-03T20:58:02.008 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:02.006+0000 7f0189600640 1 -- 172.21.15.32:0/111188137 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f0178005330 con 0x7f018c0cffe0 2024-08-03T20:58:02.008 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:02.006+0000 7f0189600640 1 -- 172.21.15.32:0/111188137 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01780097c0 con 0x7f018c0cffe0 2024-08-03T20:58:02.008 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:02.006+0000 7f0193000640 1 -- 172.21.15.32:0/111188137 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f018c0c7fa0 con 0x7f018c0cffe0 2024-08-03T20:58:02.008 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:02.006+0000 7f0193000640 1 -- 172.21.15.32:0/111188137 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f018c0c8560 con 0x7f018c0cffe0 2024-08-03T20:58:02.009 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:02.007+0000 7f0193000640 1 -- 172.21.15.32:0/111188137 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f018c0c7cf0 con 0x7f018c0cffe0 2024-08-03T20:58:02.009 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:02.008+0000 7f0189600640 1 -- 172.21.15.32:0/111188137 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f0178009b70 con 0x7f018c0cffe0 2024-08-03T20:58:02.010 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:02.009+0000 7f0189600640 1 --2- 172.21.15.32:0/111188137 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f016c077040 0x7f016c079550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:58:02.010 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:02.009+0000 7f0189600640 1 -- 172.21.15.32:0/111188137 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(111..111 src has 1..111) ==== 8090+0+0 (secure 0 0 0) 0x7f0178099520 con 0x7f018c0cffe0 2024-08-03T20:58:02.010 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:02.010+0000 7f0191c00640 1 --2- 172.21.15.32:0/111188137 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f016c077040 0x7f016c079550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:58:02.011 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:02.011+0000 7f0191c00640 1 --2- 172.21.15.32:0/111188137 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f016c077040 0x7f016c079550 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7f0180033000 tx=0x7f0180004730 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:58:02.014 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:02.014+0000 7f0189600640 1 -- 172.21.15.32:0/111188137 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7f018c155d00 con 0x7f018c0cffe0 2024-08-03T20:58:02.260 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:02.259+0000 7f0193000640 1 -- 172.21.15.32:0/111188137 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7f018c0d4730 con 0x7f018c0cffe0 2024-08-03T20:58:02.508 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:02.508+0000 7f0189600640 1 -- 172.21.15.32:0/111188137 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v239) ==== 161+0+0 (secure 0 0 0) 0x7f0178061dc0 con 0x7f018c0cffe0 2024-08-03T20:58:02.508 INFO:teuthology.orchestra.run.smithi032.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-08-03T20:58:02.511 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:02.510+0000 7f0193000640 1 -- 172.21.15.32:0/111188137 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f016c077040 msgr2=0x7f016c079550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:58:02.511 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:02.510+0000 7f0193000640 1 --2- 172.21.15.32:0/111188137 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f016c077040 0x7f016c079550 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7f0180033000 tx=0x7f0180004730 comp rx=0 tx=0).stop 2024-08-03T20:58:02.511 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:02.510+0000 7f0193000640 1 -- 172.21.15.32:0/111188137 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f018c0cffe0 msgr2=0x7f018c0cb910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:58:02.511 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:02.510+0000 7f0193000640 1 --2- 172.21.15.32:0/111188137 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f018c0cffe0 0x7f018c0cb910 secure :-1 s=READY pgs=1694 cs=0 l=1 rev1=1 crypto rx=0x7f017800d6d0 tx=0x7f017800db90 comp rx=0 tx=0).stop 2024-08-03T20:58:02.511 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:02.510+0000 7f0193000640 1 -- 172.21.15.32:0/111188137 shutdown_connections 2024-08-03T20:58:02.511 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:02.510+0000 7f0193000640 1 --2- 172.21.15.32:0/111188137 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f016c077040 0x7f016c079550 unknown :-1 s=CLOSED pgs=691 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:02.511 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:02.510+0000 7f0193000640 1 --2- 172.21.15.32:0/111188137 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f018c0d26e0 0x7f018c06d4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:02.512 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:02.510+0000 7f0193000640 1 --2- 172.21.15.32:0/111188137 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f018c0cffe0 0x7f018c0cb910 unknown :-1 s=CLOSED pgs=1694 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:02.512 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:02.510+0000 7f0193000640 1 --2- 172.21.15.32:0/111188137 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f018c0b43d0 0x7f018c1557c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:02.512 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:02.510+0000 7f0193000640 1 -- 172.21.15.32:0/111188137 >> 172.21.15.32:0/111188137 conn(0x7f018c08d7a0 msgr2=0x7f018c06edc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:58:02.512 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:02.511+0000 7f0193000640 1 -- 172.21.15.32:0/111188137 shutdown_connections 2024-08-03T20:58:02.512 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:02.511+0000 7f0193000640 1 -- 172.21.15.32:0/111188137 wait complete. 2024-08-03T20:58:02.537 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mds repaired 0 2024-08-03T20:58:02.767 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:02.766+0000 7ff115c00640 1 -- 172.21.15.32:0/3636252811 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff110165160 msgr2=0x7ff110167580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:58:02.767 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:02.766+0000 7ff115c00640 1 --2- 172.21.15.32:0/3636252811 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff110165160 0x7ff110167580 secure :-1 s=READY pgs=1696 cs=0 l=1 rev1=1 crypto rx=0x7ff108005fd0 tx=0x7ff10802a8b0 comp rx=0 tx=0).stop 2024-08-03T20:58:02.767 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:02.766+0000 7ff115c00640 1 -- 172.21.15.32:0/3636252811 shutdown_connections 2024-08-03T20:58:02.767 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:02.766+0000 7ff115c00640 1 --2- 172.21.15.32:0/3636252811 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff110165160 0x7ff110167580 unknown :-1 s=CLOSED pgs=1696 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:02.767 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:02.766+0000 7ff115c00640 1 -- 172.21.15.32:0/3636252811 >> 172.21.15.32:0/3636252811 conn(0x7ff1100b1bd0 msgr2=0x7ff1100b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:58:02.767 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:02.766+0000 7ff115c00640 1 -- 172.21.15.32:0/3636252811 shutdown_connections 2024-08-03T20:58:02.767 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:02.766+0000 7ff115c00640 1 -- 172.21.15.32:0/3636252811 wait complete. 2024-08-03T20:58:02.768 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:02.768+0000 7ff115c00640 1 Processor -- start 2024-08-03T20:58:02.769 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:02.768+0000 7ff115c00640 1 -- start start 2024-08-03T20:58:02.771 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:02.770+0000 7ff115c00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ff11006e930 0x7ff110143fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:58:02.771 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:02.770+0000 7ff115c00640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7ff110144f90 0x7ff1101568e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:58:02.771 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:02.770+0000 7ff115c00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff110165160 0x7ff1101f6910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:58:02.771 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:02.770+0000 7ff115c00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7ff1100cf9c0 con 0x7ff110165160 2024-08-03T20:58:02.771 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:02.770+0000 7ff115c00640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7ff1100cf840 con 0x7ff11006e930 2024-08-03T20:58:02.771 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:02.770+0000 7ff115c00640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7ff110167ac0 con 0x7ff110144f90 2024-08-03T20:58:02.771 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:02.770+0000 7ff10f400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff110165160 0x7ff1101f6910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:58:02.771 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:02.770+0000 7ff10f400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff110165160 0x7ff1101f6910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:51070/0 (socket says 172.21.15.32:51070) 2024-08-03T20:58:02.772 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:02.770+0000 7ff10f400640 1 -- 172.21.15.32:0/1178996466 learned_addr learned my addr 172.21.15.32:0/1178996466 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:58:02.772 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:02.770+0000 7ff10ea00640 1 --2- 172.21.15.32:0/1178996466 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ff11006e930 0x7ff110143fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:58:02.772 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:02.770+0000 7ff10e000640 1 --2- 172.21.15.32:0/1178996466 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7ff110144f90 0x7ff1101568e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:58:02.772 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:02.771+0000 7ff10f400640 1 -- 172.21.15.32:0/1178996466 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7ff110144f90 msgr2=0x7ff1101568e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:58:02.772 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:02.771+0000 7ff10f400640 1 --2- 172.21.15.32:0/1178996466 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7ff110144f90 0x7ff1101568e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:02.772 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:02.771+0000 7ff10f400640 1 -- 172.21.15.32:0/1178996466 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ff11006e930 msgr2=0x7ff110143fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:58:02.772 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:02.771+0000 7ff10f400640 1 --2- 172.21.15.32:0/1178996466 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ff11006e930 0x7ff110143fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:02.772 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:02.771+0000 7ff10f400640 1 -- 172.21.15.32:0/1178996466 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff110156e20 con 0x7ff110165160 2024-08-03T20:58:02.772 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:02.771+0000 7ff10ea00640 1 --2- 172.21.15.32:0/1178996466 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ff11006e930 0x7ff110143fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-03T20:58:02.773 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:02.771+0000 7ff10f400640 1 --2- 172.21.15.32:0/1178996466 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff110165160 0x7ff1101f6910 secure :-1 s=READY pgs=1697 cs=0 l=1 rev1=1 crypto rx=0x7ff10802aea0 tx=0x7ff1080039f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:58:02.773 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:02.771+0000 7ff106a00640 1 -- 172.21.15.32:0/1178996466 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff108031b90 con 0x7ff110165160 2024-08-03T20:58:02.773 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:02.771+0000 7ff106a00640 1 -- 172.21.15.32:0/1178996466 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7ff108031e20 con 0x7ff110165160 2024-08-03T20:58:02.773 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:02.771+0000 7ff115c00640 1 -- 172.21.15.32:0/1178996466 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff1101414e0 con 0x7ff110165160 2024-08-03T20:58:02.773 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:02.771+0000 7ff106a00640 1 -- 172.21.15.32:0/1178996466 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1080332f0 con 0x7ff110165160 2024-08-03T20:58:02.773 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:02.772+0000 7ff115c00640 1 -- 172.21.15.32:0/1178996466 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff11013bab0 con 0x7ff110165160 2024-08-03T20:58:02.774 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:02.773+0000 7ff115c00640 1 -- 172.21.15.32:0/1178996466 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff110073e60 con 0x7ff110165160 2024-08-03T20:58:02.774 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:02.774+0000 7ff106a00640 1 -- 172.21.15.32:0/1178996466 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7ff108033490 con 0x7ff110165160 2024-08-03T20:58:02.775 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:02.775+0000 7ff106a00640 1 --2- 172.21.15.32:0/1178996466 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7ff0f0077040 0x7ff0f0079550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:58:02.775 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:02.775+0000 7ff106a00640 1 -- 172.21.15.32:0/1178996466 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(111..111 src has 1..111) ==== 8090+0+0 (secure 0 0 0) 0x7ff1080ba780 con 0x7ff110165160 2024-08-03T20:58:02.776 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:02.775+0000 7ff10ea00640 1 --2- 172.21.15.32:0/1178996466 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7ff0f0077040 0x7ff0f0079550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:58:02.776 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:02.776+0000 7ff10ea00640 1 --2- 172.21.15.32:0/1178996466 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7ff0f0077040 0x7ff0f0079550 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7ff100007fd0 tx=0x7ff100009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:58:02.779 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:02.779+0000 7ff106a00640 1 -- 172.21.15.32:0/1178996466 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7ff10803f030 con 0x7ff110165160 2024-08-03T20:58:03.064 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:03.063+0000 7ff115c00640 1 -- 172.21.15.32:0/1178996466 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "mds repaired", "role": "0"} v 0) -- 0x7ff1101444e0 con 0x7ff110165160 2024-08-03T20:58:03.515 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:03.514+0000 7ff106a00640 1 -- 172.21.15.32:0/1178996466 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "mds repaired", "role": "0"}]=0 repaired: restoring rank 6:0 v240) ==== 101+0+0 (secure 0 0 0) 0x7ff1080830b0 con 0x7ff110165160 2024-08-03T20:58:03.515 INFO:teuthology.orchestra.run.smithi032.stderr:repaired: restoring rank 6:0 2024-08-03T20:58:03.518 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:03.517+0000 7ff115c00640 1 -- 172.21.15.32:0/1178996466 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7ff0f0077040 msgr2=0x7ff0f0079550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:58:03.518 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:03.517+0000 7ff115c00640 1 --2- 172.21.15.32:0/1178996466 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7ff0f0077040 0x7ff0f0079550 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7ff100007fd0 tx=0x7ff100009040 comp rx=0 tx=0).stop 2024-08-03T20:58:03.518 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:03.517+0000 7ff115c00640 1 -- 172.21.15.32:0/1178996466 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff110165160 msgr2=0x7ff1101f6910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:58:03.519 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:03.517+0000 7ff115c00640 1 --2- 172.21.15.32:0/1178996466 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff110165160 0x7ff1101f6910 secure :-1 s=READY pgs=1697 cs=0 l=1 rev1=1 crypto rx=0x7ff10802aea0 tx=0x7ff1080039f0 comp rx=0 tx=0).stop 2024-08-03T20:58:03.519 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:03.517+0000 7ff115c00640 1 -- 172.21.15.32:0/1178996466 shutdown_connections 2024-08-03T20:58:03.519 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:03.517+0000 7ff115c00640 1 --2- 172.21.15.32:0/1178996466 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7ff0f0077040 0x7ff0f0079550 unknown :-1 s=CLOSED pgs=692 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:03.519 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:03.517+0000 7ff115c00640 1 --2- 172.21.15.32:0/1178996466 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff110165160 0x7ff1101f6910 unknown :-1 s=CLOSED pgs=1697 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:03.519 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:03.517+0000 7ff115c00640 1 --2- 172.21.15.32:0/1178996466 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7ff110144f90 0x7ff1101568e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:03.519 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:03.517+0000 7ff115c00640 1 --2- 172.21.15.32:0/1178996466 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ff11006e930 0x7ff110143fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:03.519 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:03.517+0000 7ff115c00640 1 -- 172.21.15.32:0/1178996466 >> 172.21.15.32:0/1178996466 conn(0x7ff1100b1bd0 msgr2=0x7ff1101667e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:58:03.519 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:03.517+0000 7ff115c00640 1 -- 172.21.15.32:0/1178996466 shutdown_connections 2024-08-03T20:58:03.519 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:03.518+0000 7ff115c00640 1 -- 172.21.15.32:0/1178996466 wait complete. 2024-08-03T20:58:03.546 DEBUG:teuthology.orchestra.run.smithi032:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata import - 2024-08-03T20:58:03.689 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:03.689+0000 7f48cd1edf40 1 Processor -- start 2024-08-03T20:58:03.689 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:03.689+0000 7f48cd1edf40 1 -- start start 2024-08-03T20:58:03.691 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:03.690+0000 7f48cd1edf40 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x561ee84898e0 0x561ee8482a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:58:03.691 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:03.690+0000 7f48cd1edf40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x561ee850cba0 0x561ee8485f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:58:03.691 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:03.690+0000 7f48cd1edf40 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561ee85195b0 0x561ee8486460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:58:03.691 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:03.690+0000 7f48cd1edf40 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x561ee838ee30 con 0x561ee85195b0 2024-08-03T20:58:03.691 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:03.690+0000 7f48cd1edf40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x561ee83ecbf0 con 0x561ee850cba0 2024-08-03T20:58:03.692 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:03.690+0000 7f48cd1edf40 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x561ee8516670 con 0x561ee84898e0 2024-08-03T20:58:03.692 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:03.690+0000 7f48c6e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561ee85195b0 0x561ee8486460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:58:03.692 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:03.690+0000 7f48c5a00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x561ee850cba0 0x561ee8485f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:58:03.692 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:03.690+0000 7f48c6400640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x561ee84898e0 0x561ee8482a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:58:03.692 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:03.690+0000 7f48c6e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561ee85195b0 0x561ee8486460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:51072/0 (socket says 172.21.15.32:51072) 2024-08-03T20:58:03.692 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:03.690+0000 7f48c6e00640 1 -- 172.21.15.32:0/1702966942 learned_addr learned my addr 172.21.15.32:0/1702966942 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:58:03.692 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:03.691+0000 7f48c6e00640 1 -- 172.21.15.32:0/1702966942 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x561ee84898e0 msgr2=0x561ee8482a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:58:03.692 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:03.691+0000 7f48c6e00640 1 --2- 172.21.15.32:0/1702966942 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x561ee84898e0 0x561ee8482a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:03.692 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:03.691+0000 7f48c6e00640 1 -- 172.21.15.32:0/1702966942 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x561ee850cba0 msgr2=0x561ee8485f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:58:03.692 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:03.691+0000 7f48c6e00640 1 --2- 172.21.15.32:0/1702966942 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x561ee850cba0 0x561ee8485f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:03.692 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:03.691+0000 7f48c6e00640 1 -- 172.21.15.32:0/1702966942 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561ee85a3120 con 0x561ee85195b0 2024-08-03T20:58:03.694 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:03.691+0000 7f48c6e00640 1 --2- 172.21.15.32:0/1702966942 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561ee85195b0 0x561ee8486460 secure :-1 s=READY pgs=1698 cs=0 l=1 rev1=1 crypto rx=0x7f48c000a980 tx=0x7f48c000ae40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a5c0faa9aa11310 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:58:03.694 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:03.691+0000 7f48bfe00640 1 -- 172.21.15.32:0/1702966942 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48c000dbf0 con 0x561ee85195b0 2024-08-03T20:58:03.694 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:03.691+0000 7f48bfe00640 1 -- 172.21.15.32:0/1702966942 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f48c000ddb0 con 0x561ee85195b0 2024-08-03T20:58:03.694 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:03.691+0000 7f48bfe00640 1 -- 172.21.15.32:0/1702966942 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48c000f690 con 0x561ee85195b0 2024-08-03T20:58:03.694 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:03.692+0000 7f48cd1edf40 1 -- 172.21.15.32:0/1702966942 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561ee85195b0 msgr2=0x561ee8486460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:58:03.694 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:03.692+0000 7f48cd1edf40 1 --2- 172.21.15.32:0/1702966942 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561ee85195b0 0x561ee8486460 secure :-1 s=READY pgs=1698 cs=0 l=1 rev1=1 crypto rx=0x7f48c000a980 tx=0x7f48c000ae40 comp rx=0 tx=0).stop 2024-08-03T20:58:03.695 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:03.692+0000 7f48cd1edf40 1 -- 172.21.15.32:0/1702966942 shutdown_connections 2024-08-03T20:58:03.695 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:03.692+0000 7f48cd1edf40 1 --2- 172.21.15.32:0/1702966942 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x561ee850cba0 0x561ee8485f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:03.695 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:03.692+0000 7f48cd1edf40 1 --2- 172.21.15.32:0/1702966942 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561ee85195b0 0x561ee8486460 unknown :-1 s=CLOSED pgs=1698 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:03.695 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:03.692+0000 7f48cd1edf40 1 --2- 172.21.15.32:0/1702966942 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x561ee84898e0 0x561ee8482a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:03.695 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:03.692+0000 7f48cd1edf40 1 -- 172.21.15.32:0/1702966942 >> 172.21.15.32:0/1702966942 conn(0x561ee8360ac0 msgr2=0x561ee843e340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:58:03.695 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:03.692+0000 7f48cd1edf40 1 -- 172.21.15.32:0/1702966942 shutdown_connections 2024-08-03T20:58:03.695 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:03.692+0000 7f48cd1edf40 1 -- 172.21.15.32:0/1702966942 wait complete. 2024-08-03T20:58:03.695 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:03.693+0000 7f48cd1edf40 1 Processor -- start 2024-08-03T20:58:03.695 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:03.693+0000 7f48cd1edf40 1 -- start start 2024-08-03T20:58:03.697 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:03.695+0000 7f48cd1edf40 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x561ee84898e0 0x561ee8504500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:58:03.697 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:03.695+0000 7f48cd1edf40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x561ee850cba0 0x561ee8485f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:58:03.698 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:03.695+0000 7f48cd1edf40 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561ee85195b0 0x561ee8486460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:58:03.698 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:03.695+0000 7f48cd1edf40 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x561ee8487cf0 con 0x561ee85195b0 2024-08-03T20:58:03.698 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:03.695+0000 7f48cd1edf40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x561ee846b7c0 con 0x561ee850cba0 2024-08-03T20:58:03.698 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:03.695+0000 7f48cd1edf40 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x561ee8504ee0 con 0x561ee84898e0 2024-08-03T20:58:03.698 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:03.695+0000 7f48c6e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561ee85195b0 0x561ee8486460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:58:03.698 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:03.695+0000 7f48c6e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561ee85195b0 0x561ee8486460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:51084/0 (socket says 172.21.15.32:51084) 2024-08-03T20:58:03.698 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:03.695+0000 7f48c6e00640 1 -- 172.21.15.32:0/2928670630 learned_addr learned my addr 172.21.15.32:0/2928670630 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:58:03.698 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:03.695+0000 7f48c5a00640 1 --2- 172.21.15.32:0/2928670630 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x561ee850cba0 0x561ee8485f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:58:03.698 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:03.695+0000 7f48c6400640 1 --2- 172.21.15.32:0/2928670630 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x561ee84898e0 0x561ee8504500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:58:03.698 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:03.696+0000 7f48c6e00640 1 -- 172.21.15.32:0/2928670630 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x561ee84898e0 msgr2=0x561ee8504500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:58:03.699 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:03.696+0000 7f48c6e00640 1 --2- 172.21.15.32:0/2928670630 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x561ee84898e0 0x561ee8504500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:03.699 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:03.696+0000 7f48c6e00640 1 -- 172.21.15.32:0/2928670630 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x561ee850cba0 msgr2=0x561ee8485f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:58:03.699 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:03.696+0000 7f48c6e00640 1 --2- 172.21.15.32:0/2928670630 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x561ee850cba0 0x561ee8485f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:03.699 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:03.696+0000 7f48c6e00640 1 -- 172.21.15.32:0/2928670630 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561ee8504a40 con 0x561ee85195b0 2024-08-03T20:58:03.699 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:03.696+0000 7f48c5a00640 1 --2- 172.21.15.32:0/2928670630 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x561ee850cba0 0x561ee8485f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-03T20:58:03.699 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:03.696+0000 7f48c6e00640 1 --2- 172.21.15.32:0/2928670630 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561ee85195b0 0x561ee8486460 secure :-1 s=READY pgs=1699 cs=0 l=1 rev1=1 crypto rx=0x7f48c000d720 tx=0x7f48c000f6e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:58:03.699 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:03.696+0000 7f48bea00640 1 -- 172.21.15.32:0/2928670630 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48c0023030 con 0x561ee85195b0 2024-08-03T20:58:03.699 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:03.696+0000 7f48bea00640 1 -- 172.21.15.32:0/2928670630 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f48c0004130 con 0x561ee85195b0 2024-08-03T20:58:03.699 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:03.696+0000 7f48bea00640 1 -- 172.21.15.32:0/2928670630 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48c000fb10 con 0x561ee85195b0 2024-08-03T20:58:03.699 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:03.696+0000 7f48cd1edf40 1 -- 172.21.15.32:0/2928670630 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x561ee846bba0 con 0x561ee85195b0 2024-08-03T20:58:03.699 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:03.696+0000 7f48cd1edf40 1 -- 172.21.15.32:0/2928670630 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561ee850f7c0 con 0x561ee85195b0 2024-08-03T20:58:03.700 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:03.699+0000 7f48bea00640 1 -- 172.21.15.32:0/2928670630 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f48c0026070 con 0x561ee85195b0 2024-08-03T20:58:03.700 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:03.700+0000 7f48bea00640 1 --2- 172.21.15.32:0/2928670630 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f48a80759f0 0x7f48a8077f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:58:03.700 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:03.700+0000 7f48bea00640 1 -- 172.21.15.32:0/2928670630 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(111..111 src has 1..111) ==== 8090+0+0 (secure 0 0 0) 0x7f48c0099ff0 con 0x561ee85195b0 2024-08-03T20:58:03.701 INFO:teuthology.orchestra.run.smithi032.stdout:Importing pool 2024-08-03T20:58:03.701 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:03.700+0000 7f48cd1edf40 1 --2- 172.21.15.32:0/2928670630 >> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] conn(0x561ee846a990 0x561ee846ada0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:58:03.702 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:03.700+0000 7f48cd1edf40 1 -- 172.21.15.32:0/2928670630 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:1 12.10 12:09401a81:::601.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111) -- 0x561ee8505230 con 0x561ee846a990 2024-08-03T20:58:03.702 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:03.700+0000 7f48c6400640 1 --2- 172.21.15.32:0/2928670630 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f48a80759f0 0x7f48a8077f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:58:03.702 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:03.700+0000 7f48c5a00640 1 --2- 172.21.15.32:0/2928670630 >> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] conn(0x561ee846a990 0x561ee846ada0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:58:03.702 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:03.701+0000 7f48c5a00640 1 --2- 172.21.15.32:0/2928670630 >> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] conn(0x561ee846a990 0x561ee846ada0 crc :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:58:03.702 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:03.701+0000 7f48c6400640 1 --2- 172.21.15.32:0/2928670630 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f48a80759f0 0x7f48a8077f00 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7f48b8010410 tx=0x7f48b8001730 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:58:03.704 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:03.704+0000 7f48c5a00640 1 -- 172.21.15.32:0/2928670630 <== osd.0 v2:172.21.15.32:6816/3066156116 1 ==== osd_op_reply(1 601.00000000 [create] v111'784 uv758 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f48b404ec60 con 0x561ee846a990 2024-08-03T20:58:03.704 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:03.704+0000 7f48cd1edf40 1 -- 172.21.15.32:0/2928670630 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:2 12.10 12:09401a81:::601.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e111) -- 0x561ee846b2e0 con 0x561ee846a990 2024-08-03T20:58:03.706 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:03.705+0000 7f48c5a00640 1 -- 172.21.15.32:0/2928670630 <== osd.0 v2:172.21.15.32:6816/3066156116 2 ==== osd_op_reply(2 601.00000000 [delete] v111'785 uv785 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f48b404ec60 con 0x561ee846a990 2024-08-03T20:58:03.706 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:03.706+0000 7f48cd1edf40 1 -- 172.21.15.32:0/2928670630 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:3 12.10 12:09401a81:::601.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111) -- 0x561ee85b95b0 con 0x561ee846a990 2024-08-03T20:58:03.707 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::601.00000000:head# 2024-08-03T20:58:03.708 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:03.707+0000 7f48c5a00640 1 -- 172.21.15.32:0/2928670630 <== osd.0 v2:172.21.15.32:6816/3066156116 3 ==== osd_op_reply(3 601.00000000 [create] v111'786 uv786 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f48b404ec60 con 0x561ee846a990 2024-08-03T20:58:03.708 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:03.707+0000 7f48cd1edf40 1 -- 172.21.15.32:0/2928670630 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:4 12.10 12:09401a81:::601.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111) -- 0x561ee85ba960 con 0x561ee846a990 2024-08-03T20:58:03.709 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:03.709+0000 7f48c5a00640 1 -- 172.21.15.32:0/2928670630 <== osd.0 v2:172.21.15.32:6816/3066156116 4 ==== osd_op_reply(4 601.00000000 [setxattr (30)] v111'787 uv787 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f48b404ec60 con 0x561ee846a990 2024-08-03T20:58:03.709 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:03.709+0000 7f48cd1edf40 1 -- 172.21.15.32:0/2928670630 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:5 12.10 12:09401a81:::601.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111) -- 0x561ee85bad00 con 0x561ee846a990 2024-08-03T20:58:03.711 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:03.711+0000 7f48c5a00640 1 -- 172.21.15.32:0/2928670630 <== osd.0 v2:172.21.15.32:6816/3066156116 5 ==== osd_op_reply(5 601.00000000 [setxattr (62)] v111'788 uv788 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f48b404ec60 con 0x561ee846a990 2024-08-03T20:58:03.711 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:03.711+0000 7f48cd1edf40 1 -- 172.21.15.32:0/2928670630 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:6 12.10 12:09401a81:::601.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111) -- 0x561ee85b9950 con 0x561ee846a990 2024-08-03T20:58:03.713 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:03.712+0000 7f48c5a00640 1 -- 172.21.15.32:0/2928670630 <== osd.0 v2:172.21.15.32:6816/3066156116 6 ==== osd_op_reply(6 601.00000000 [omap-set-header] v111'789 uv789 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f48b404ec60 con 0x561ee846a990 2024-08-03T20:58:03.713 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:03.712+0000 7f48cd1edf40 1 -- 172.21.15.32:0/2928670630 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:7 12.10 12:0bd6d154:::602.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111) -- 0x561ee85b9d40 con 0x561ee846a990 2024-08-03T20:58:03.715 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:03.714+0000 7f48c5a00640 1 -- 172.21.15.32:0/2928670630 <== osd.0 v2:172.21.15.32:6816/3066156116 7 ==== osd_op_reply(7 602.00000000 [create] v111'790 uv764 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f48b404ec60 con 0x561ee846a990 2024-08-03T20:58:03.715 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:03.714+0000 7f48cd1edf40 1 -- 172.21.15.32:0/2928670630 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:8 12.10 12:0bd6d154:::602.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e111) -- 0x561ee85b9d40 con 0x561ee846a990 2024-08-03T20:58:03.717 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:03.716+0000 7f48c5a00640 1 -- 172.21.15.32:0/2928670630 <== osd.0 v2:172.21.15.32:6816/3066156116 8 ==== osd_op_reply(8 602.00000000 [delete] v111'791 uv791 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f48b404ec60 con 0x561ee846a990 2024-08-03T20:58:03.717 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:03.716+0000 7f48cd1edf40 1 -- 172.21.15.32:0/2928670630 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:9 12.10 12:0bd6d154:::602.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111) -- 0x561ee85b9d40 con 0x561ee846a990 2024-08-03T20:58:03.718 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:03.718+0000 7f48c5a00640 1 -- 172.21.15.32:0/2928670630 <== osd.0 v2:172.21.15.32:6816/3066156116 9 ==== osd_op_reply(9 602.00000000 [create] v111'792 uv792 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f48b404ec60 con 0x561ee846a990 2024-08-03T20:58:03.719 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::602.00000000:head# 2024-08-03T20:58:03.719 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:03.718+0000 7f48cd1edf40 1 -- 172.21.15.32:0/2928670630 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:10 12.10 12:0bd6d154:::602.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111) -- 0x561ee85b9d40 con 0x561ee846a990 2024-08-03T20:58:03.720 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:03.720+0000 7f48c5a00640 1 -- 172.21.15.32:0/2928670630 <== osd.0 v2:172.21.15.32:6816/3066156116 10 ==== osd_op_reply(10 602.00000000 [setxattr (30)] v111'793 uv793 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f48b404ec60 con 0x561ee846a990 2024-08-03T20:58:03.720 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:03.720+0000 7f48cd1edf40 1 -- 172.21.15.32:0/2928670630 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:11 12.10 12:0bd6d154:::602.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111) -- 0x561ee85b9d40 con 0x561ee846a990 2024-08-03T20:58:03.722 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:03.721+0000 7f48c5a00640 1 -- 172.21.15.32:0/2928670630 <== osd.0 v2:172.21.15.32:6816/3066156116 11 ==== osd_op_reply(11 602.00000000 [setxattr (62)] v111'794 uv794 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f48b404ec60 con 0x561ee846a990 2024-08-03T20:58:03.722 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:03.721+0000 7f48cd1edf40 1 -- 172.21.15.32:0/2928670630 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:12 12.10 12:0bd6d154:::602.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111) -- 0x561ee85b9d40 con 0x561ee846a990 2024-08-03T20:58:03.723 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:03.723+0000 7f48c5a00640 1 -- 172.21.15.32:0/2928670630 <== osd.0 v2:172.21.15.32:6816/3066156116 12 ==== osd_op_reply(12 602.00000000 [omap-set-header] v111'795 uv795 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f48b404ec60 con 0x561ee846a990 2024-08-03T20:58:03.724 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:03.723+0000 7f48cd1edf40 1 -- 172.21.15.32:0/2928670630 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:13 12.10 12:0cf6ad0f:::10000000000.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111) -- 0x561ee85b9d40 con 0x561ee846a990 2024-08-03T20:58:03.725 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:03.725+0000 7f48c5a00640 1 -- 172.21.15.32:0/2928670630 <== osd.0 v2:172.21.15.32:6816/3066156116 13 ==== osd_op_reply(13 10000000000.00000000 [create] v111'796 uv771 ondisk = -17 ((17) File exists)) ==== 164+0+0 (crc 0 0 0) 0x7f48b404ec60 con 0x561ee846a990 2024-08-03T20:58:03.726 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:03.725+0000 7f48cd1edf40 1 -- 172.21.15.32:0/2928670630 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:14 12.10 12:0cf6ad0f:::10000000000.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e111) -- 0x561ee85b9d40 con 0x561ee846a990 2024-08-03T20:58:03.727 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:03.727+0000 7f48c5a00640 1 -- 172.21.15.32:0/2928670630 <== osd.0 v2:172.21.15.32:6816/3066156116 14 ==== osd_op_reply(14 10000000000.00000000 [delete] v111'797 uv797 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f48b404ec60 con 0x561ee846a990 2024-08-03T20:58:03.727 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:03.727+0000 7f48cd1edf40 1 -- 172.21.15.32:0/2928670630 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:15 12.10 12:0cf6ad0f:::10000000000.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111) -- 0x561ee85b9d40 con 0x561ee846a990 2024-08-03T20:58:03.729 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:03.728+0000 7f48c5a00640 1 -- 172.21.15.32:0/2928670630 <== osd.0 v2:172.21.15.32:6816/3066156116 15 ==== osd_op_reply(15 10000000000.00000000 [create] v111'798 uv798 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f48b404ec60 con 0x561ee846a990 2024-08-03T20:58:03.729 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::10000000000.00000000:head# 2024-08-03T20:58:03.729 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:03.728+0000 7f48cd1edf40 1 -- 172.21.15.32:0/2928670630 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:16 12.10 12:0cf6ad0f:::10000000000.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111) -- 0x561ee85b9d40 con 0x561ee846a990 2024-08-03T20:58:03.730 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:03.730+0000 7f48c5a00640 1 -- 172.21.15.32:0/2928670630 <== osd.0 v2:172.21.15.32:6816/3066156116 16 ==== osd_op_reply(16 10000000000.00000000 [setxattr (30)] v111'799 uv799 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f48b404ec60 con 0x561ee846a990 2024-08-03T20:58:03.731 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:03.730+0000 7f48cd1edf40 1 -- 172.21.15.32:0/2928670630 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:17 12.10 12:0cf6ad0f:::10000000000.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111) -- 0x561ee85b9d40 con 0x561ee846a990 2024-08-03T20:58:03.732 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:03.732+0000 7f48c5a00640 1 -- 172.21.15.32:0/2928670630 <== osd.0 v2:172.21.15.32:6816/3066156116 17 ==== osd_op_reply(17 10000000000.00000000 [setxattr (62)] v111'800 uv800 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f48b404ec60 con 0x561ee846a990 2024-08-03T20:58:03.732 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:03.732+0000 7f48cd1edf40 1 -- 172.21.15.32:0/2928670630 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:18 12.10 12:0cf6ad0f:::10000000000.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111) -- 0x561ee85b9d40 con 0x561ee846a990 2024-08-03T20:58:03.734 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:03.734+0000 7f48c5a00640 1 -- 172.21.15.32:0/2928670630 <== osd.0 v2:172.21.15.32:6816/3066156116 18 ==== osd_op_reply(18 10000000000.00000000 [omap-set-header] v111'801 uv801 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f48b404ec60 con 0x561ee846a990 2024-08-03T20:58:03.734 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:03.734+0000 7f48cd1edf40 1 -- 172.21.15.32:0/2928670630 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:19 12.10 12:0cf6ad0f:::10000000000.00000000:head [omap-set-vals in=593b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111) -- 0x561ee85bb0a0 con 0x561ee846a990 2024-08-03T20:58:03.736 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:03.735+0000 7f48c5a00640 1 -- 172.21.15.32:0/2928670630 <== osd.0 v2:172.21.15.32:6816/3066156116 19 ==== osd_op_reply(19 10000000000.00000000 [omap-set-vals] v111'802 uv802 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f48b404ec60 con 0x561ee846a990 2024-08-03T20:58:03.736 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:03.735+0000 7f48cd1edf40 1 -- 172.21.15.32:0/2928670630 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:20 12.10 12:0d82a743:::600.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111) -- 0x561ee85ba4b0 con 0x561ee846a990 2024-08-03T20:58:03.737 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:03.737+0000 7f48c5a00640 1 -- 172.21.15.32:0/2928670630 <== osd.0 v2:172.21.15.32:6816/3066156116 20 ==== osd_op_reply(20 600.00000000 [create] v111'803 uv777 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f48b404ec60 con 0x561ee846a990 2024-08-03T20:58:03.737 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:03.737+0000 7f48cd1edf40 1 -- 172.21.15.32:0/2928670630 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:21 12.10 12:0d82a743:::600.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e111) -- 0x561ee85ba4b0 con 0x561ee846a990 2024-08-03T20:58:03.739 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:03.739+0000 7f48c5a00640 1 -- 172.21.15.32:0/2928670630 <== osd.0 v2:172.21.15.32:6816/3066156116 21 ==== osd_op_reply(21 600.00000000 [delete] v111'804 uv804 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f48b404ec60 con 0x561ee846a990 2024-08-03T20:58:03.739 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:03.739+0000 7f48cd1edf40 1 -- 172.21.15.32:0/2928670630 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:22 12.10 12:0d82a743:::600.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111) -- 0x561ee85ba4b0 con 0x561ee846a990 2024-08-03T20:58:03.741 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:03.740+0000 7f48c5a00640 1 -- 172.21.15.32:0/2928670630 <== osd.0 v2:172.21.15.32:6816/3066156116 22 ==== osd_op_reply(22 600.00000000 [create] v111'805 uv805 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f48b404ec60 con 0x561ee846a990 2024-08-03T20:58:03.741 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::600.00000000:head# 2024-08-03T20:58:03.741 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:03.740+0000 7f48cd1edf40 1 -- 172.21.15.32:0/2928670630 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:23 12.10 12:0d82a743:::600.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111) -- 0x561ee85ba4b0 con 0x561ee846a990 2024-08-03T20:58:03.742 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:03.742+0000 7f48c5a00640 1 -- 172.21.15.32:0/2928670630 <== osd.0 v2:172.21.15.32:6816/3066156116 23 ==== osd_op_reply(23 600.00000000 [setxattr (30)] v111'806 uv806 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f48b404ec60 con 0x561ee846a990 2024-08-03T20:58:03.743 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:03.742+0000 7f48cd1edf40 1 -- 172.21.15.32:0/2928670630 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:24 12.10 12:0d82a743:::600.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111) -- 0x561ee85ba4b0 con 0x561ee846a990 2024-08-03T20:58:03.744 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:03.744+0000 7f48c5a00640 1 -- 172.21.15.32:0/2928670630 <== osd.0 v2:172.21.15.32:6816/3066156116 24 ==== osd_op_reply(24 600.00000000 [setxattr (62)] v111'807 uv807 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f48b404ec60 con 0x561ee846a990 2024-08-03T20:58:03.744 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:03.744+0000 7f48cd1edf40 1 -- 172.21.15.32:0/2928670630 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:25 12.10 12:0d82a743:::600.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111) -- 0x561ee85ba4b0 con 0x561ee846a990 2024-08-03T20:58:03.746 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:03.745+0000 7f48c5a00640 1 -- 172.21.15.32:0/2928670630 <== osd.0 v2:172.21.15.32:6816/3066156116 25 ==== osd_op_reply(25 600.00000000 [omap-set-header] v111'808 uv808 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f48b404ec60 con 0x561ee846a990 2024-08-03T20:58:03.746 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:03.746+0000 7f48cd1edf40 1 -- 172.21.15.32:0/2928670630 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:26 12.10 12:0d89b25e:::603.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111) -- 0x561ee85ba4b0 con 0x561ee846a990 2024-08-03T20:58:03.747 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:03.747+0000 7f48c5a00640 1 -- 172.21.15.32:0/2928670630 <== osd.0 v2:172.21.15.32:6816/3066156116 26 ==== osd_op_reply(26 603.00000000 [create] v111'809 uv783 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f48b404ec60 con 0x561ee846a990 2024-08-03T20:58:03.748 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:03.747+0000 7f48cd1edf40 1 -- 172.21.15.32:0/2928670630 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:27 12.10 12:0d89b25e:::603.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e111) -- 0x561ee85ba4b0 con 0x561ee846a990 2024-08-03T20:58:03.749 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:03.749+0000 7f48c5a00640 1 -- 172.21.15.32:0/2928670630 <== osd.0 v2:172.21.15.32:6816/3066156116 27 ==== osd_op_reply(27 603.00000000 [delete] v111'810 uv810 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f48b404ec60 con 0x561ee846a990 2024-08-03T20:58:03.749 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:03.749+0000 7f48cd1edf40 1 -- 172.21.15.32:0/2928670630 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:28 12.10 12:0d89b25e:::603.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111) -- 0x561ee85ba4b0 con 0x561ee846a990 2024-08-03T20:58:03.751 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:03.750+0000 7f48c5a00640 1 -- 172.21.15.32:0/2928670630 <== osd.0 v2:172.21.15.32:6816/3066156116 28 ==== osd_op_reply(28 603.00000000 [create] v111'811 uv811 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f48b404ec60 con 0x561ee846a990 2024-08-03T20:58:03.751 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::603.00000000:head# 2024-08-03T20:58:03.751 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:03.751+0000 7f48cd1edf40 1 -- 172.21.15.32:0/2928670630 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:29 12.10 12:0d89b25e:::603.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111) -- 0x561ee85ba4b0 con 0x561ee846a990 2024-08-03T20:58:03.752 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:03.752+0000 7f48c5a00640 1 -- 172.21.15.32:0/2928670630 <== osd.0 v2:172.21.15.32:6816/3066156116 29 ==== osd_op_reply(29 603.00000000 [setxattr (30)] v111'812 uv812 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f48b404ec60 con 0x561ee846a990 2024-08-03T20:58:03.753 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:03.752+0000 7f48cd1edf40 1 -- 172.21.15.32:0/2928670630 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:30 12.10 12:0d89b25e:::603.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111) -- 0x561ee85ba4b0 con 0x561ee846a990 2024-08-03T20:58:03.755 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:03.754+0000 7f48c5a00640 1 -- 172.21.15.32:0/2928670630 <== osd.0 v2:172.21.15.32:6816/3066156116 30 ==== osd_op_reply(30 603.00000000 [setxattr (62)] v111'813 uv813 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f48b404ec60 con 0x561ee846a990 2024-08-03T20:58:03.755 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:03.754+0000 7f48cd1edf40 1 -- 172.21.15.32:0/2928670630 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:31 12.10 12:0d89b25e:::603.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111) -- 0x561ee85ba4b0 con 0x561ee846a990 2024-08-03T20:58:03.756 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:03.756+0000 7f48c5a00640 1 -- 172.21.15.32:0/2928670630 <== osd.0 v2:172.21.15.32:6816/3066156116 31 ==== osd_op_reply(31 603.00000000 [omap-set-header] v111'814 uv814 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f48b404ec60 con 0x561ee846a990 2024-08-03T20:58:03.756 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:03.756+0000 7f48cd1edf40 1 --2- 172.21.15.32:0/2928670630 >> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] conn(0x561ee85bb5b0 0x561ee85bd9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:58:03.756 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:03.756+0000 7f48cd1edf40 1 -- 172.21.15.32:0/2928670630 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:32 12.14 12:287034c4:::1.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111) -- 0x561ee85ba4b0 con 0x561ee85bb5b0 2024-08-03T20:58:03.757 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:03.756+0000 7f48c6400640 1 --2- 172.21.15.32:0/2928670630 >> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] conn(0x561ee85bb5b0 0x561ee85bd9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:58:03.757 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:03.757+0000 7f48c6400640 1 --2- 172.21.15.32:0/2928670630 >> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] conn(0x561ee85bb5b0 0x561ee85bd9d0 crc :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:58:03.760 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:03.759+0000 7f48c6400640 1 -- 172.21.15.32:0/2928670630 <== osd.5 v2:172.21.15.118:6810/2724891770 1 ==== osd_op_reply(32 1.00000000.inode [create] v111'571 uv553 ondisk = -17 ((17) File exists)) ==== 160+0+0 (crc 0 0 0) 0x7f48b801b040 con 0x561ee85bb5b0 2024-08-03T20:58:03.760 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:03.760+0000 7f48cd1edf40 1 -- 172.21.15.32:0/2928670630 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:33 12.14 12:287034c4:::1.00000000.inode:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e111) -- 0x561ee85bdf10 con 0x561ee85bb5b0 2024-08-03T20:58:03.762 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:03.761+0000 7f48c6400640 1 -- 172.21.15.32:0/2928670630 <== osd.5 v2:172.21.15.118:6810/2724891770 2 ==== osd_op_reply(33 1.00000000.inode [delete] v111'572 uv572 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f48b801b040 con 0x561ee85bb5b0 2024-08-03T20:58:03.762 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:03.761+0000 7f48cd1edf40 1 -- 172.21.15.32:0/2928670630 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:34 12.14 12:287034c4:::1.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111) -- 0x561ee85be2b0 con 0x561ee85bb5b0 2024-08-03T20:58:03.763 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::1.00000000.inode:head# 2024-08-03T20:58:03.764 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:03.763+0000 7f48c6400640 1 -- 172.21.15.32:0/2928670630 <== osd.5 v2:172.21.15.118:6810/2724891770 3 ==== osd_op_reply(34 1.00000000.inode [create] v111'573 uv573 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f48b801b040 con 0x561ee85bb5b0 2024-08-03T20:58:03.764 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:03.763+0000 7f48cd1edf40 1 -- 172.21.15.32:0/2928670630 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:35 12.14 12:287034c4:::1.00000000.inode:head [write 0~582 in=582b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111) -- 0x561ee85be650 con 0x561ee85bb5b0 2024-08-03T20:58:03.765 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:03.765+0000 7f48c6400640 1 -- 172.21.15.32:0/2928670630 <== osd.5 v2:172.21.15.118:6810/2724891770 4 ==== osd_op_reply(35 1.00000000.inode [write 0~582] v111'574 uv574 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f48b801b040 con 0x561ee85bb5b0 2024-08-03T20:58:03.765 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:03.765+0000 7f48cd1edf40 1 -- 172.21.15.32:0/2928670630 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:36 12.14 12:287034c4:::1.00000000.inode:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111) -- 0x561ee85be9f0 con 0x561ee85bb5b0 2024-08-03T20:58:03.766 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:03.766+0000 7f48c6400640 1 -- 172.21.15.32:0/2928670630 <== osd.5 v2:172.21.15.118:6810/2724891770 5 ==== osd_op_reply(36 1.00000000.inode [omap-set-header] v111'575 uv575 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f48b801b040 con 0x561ee85bb5b0 2024-08-03T20:58:03.767 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:03.766+0000 7f48cd1edf40 1 -- 172.21.15.32:0/2928670630 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:37 12.14 12:292cf221:::200.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111) -- 0x561ee85bed90 con 0x561ee85bb5b0 2024-08-03T20:58:03.768 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:03.768+0000 7f48c6400640 1 -- 172.21.15.32:0/2928670630 <== osd.5 v2:172.21.15.118:6810/2724891770 6 ==== osd_op_reply(37 200.00000000 [create] v111'576 uv558 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f48b801b040 con 0x561ee85bb5b0 2024-08-03T20:58:03.768 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:03.768+0000 7f48cd1edf40 1 -- 172.21.15.32:0/2928670630 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:38 12.14 12:292cf221:::200.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e111) -- 0x561ee85bf130 con 0x561ee85bb5b0 2024-08-03T20:58:03.770 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:03.770+0000 7f48c6400640 1 -- 172.21.15.32:0/2928670630 <== osd.5 v2:172.21.15.118:6810/2724891770 7 ==== osd_op_reply(38 200.00000000 [delete] v111'577 uv577 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f48b801b040 con 0x561ee85bb5b0 2024-08-03T20:58:03.770 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:03.770+0000 7f48cd1edf40 1 -- 172.21.15.32:0/2928670630 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:39 12.14 12:292cf221:::200.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111) -- 0x561ee85bf4d0 con 0x561ee85bb5b0 2024-08-03T20:58:03.771 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::200.00000000:head# 2024-08-03T20:58:03.772 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:03.771+0000 7f48c6400640 1 -- 172.21.15.32:0/2928670630 <== osd.5 v2:172.21.15.118:6810/2724891770 8 ==== osd_op_reply(39 200.00000000 [create] v111'578 uv578 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f48b801b040 con 0x561ee85bb5b0 2024-08-03T20:58:03.772 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:03.771+0000 7f48cd1edf40 1 -- 172.21.15.32:0/2928670630 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:40 12.14 12:292cf221:::200.00000000:head [write 0~90 in=90b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111) -- 0x561ee85ba0e0 con 0x561ee85bb5b0 2024-08-03T20:58:03.773 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:03.773+0000 7f48c6400640 1 -- 172.21.15.32:0/2928670630 <== osd.5 v2:172.21.15.118:6810/2724891770 9 ==== osd_op_reply(40 200.00000000 [write 0~90] v111'579 uv579 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f48b801b040 con 0x561ee85bb5b0 2024-08-03T20:58:03.773 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:03.773+0000 7f48cd1edf40 1 -- 172.21.15.32:0/2928670630 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:41 12.14 12:292cf221:::200.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111) -- 0x561ee85ba0e0 con 0x561ee85bb5b0 2024-08-03T20:58:03.775 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:03.775+0000 7f48c6400640 1 -- 172.21.15.32:0/2928670630 <== osd.5 v2:172.21.15.118:6810/2724891770 10 ==== osd_op_reply(41 200.00000000 [omap-set-header] v111'580 uv580 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f48b801b040 con 0x561ee85bb5b0 2024-08-03T20:58:03.775 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:03.775+0000 7f48cd1edf40 1 -- 172.21.15.32:0/2928670630 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:42 12.14 12:2e2fa760:::200.00000001:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111) -- 0x561ee85ba0e0 con 0x561ee85bb5b0 2024-08-03T20:58:03.777 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:03.777+0000 7f48c6400640 1 -- 172.21.15.32:0/2928670630 <== osd.5 v2:172.21.15.118:6810/2724891770 11 ==== osd_op_reply(42 200.00000001 [create] v111'581 uv570 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f48b801b040 con 0x561ee85bb5b0 2024-08-03T20:58:03.777 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:03.777+0000 7f48cd1edf40 1 -- 172.21.15.32:0/2928670630 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:43 12.14 12:2e2fa760:::200.00000001:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e111) -- 0x561ee85ba0e0 con 0x561ee85bb5b0 2024-08-03T20:58:03.779 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:03.779+0000 7f48c6400640 1 -- 172.21.15.32:0/2928670630 <== osd.5 v2:172.21.15.118:6810/2724891770 12 ==== osd_op_reply(43 200.00000001 [delete] v111'582 uv582 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f48b801b040 con 0x561ee85bb5b0 2024-08-03T20:58:03.779 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:03.779+0000 7f48cd1edf40 1 -- 172.21.15.32:0/2928670630 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:44 12.14 12:2e2fa760:::200.00000001:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111) -- 0x561ee85ba0e0 con 0x561ee85bb5b0 2024-08-03T20:58:03.781 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::200.00000001:head# 2024-08-03T20:58:03.781 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:03.780+0000 7f48c6400640 1 -- 172.21.15.32:0/2928670630 <== osd.5 v2:172.21.15.118:6810/2724891770 13 ==== osd_op_reply(44 200.00000001 [create] v111'583 uv583 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f48b801b040 con 0x561ee85bb5b0 2024-08-03T20:58:03.781 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:03.780+0000 7f48cd1edf40 1 -- 172.21.15.32:0/2928670630 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:45 12.14 12:2e2fa760:::200.00000001:head [write 0~18118 in=18118b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111) -- 0x561ee85ba0e0 con 0x561ee85bb5b0 2024-08-03T20:58:03.783 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:03.783+0000 7f48c6400640 1 -- 172.21.15.32:0/2928670630 <== osd.5 v2:172.21.15.118:6810/2724891770 14 ==== osd_op_reply(45 200.00000001 [write 0~18118] v111'584 uv584 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f48b801b040 con 0x561ee85bb5b0 2024-08-03T20:58:03.783 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:03.783+0000 7f48cd1edf40 1 -- 172.21.15.32:0/2928670630 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:46 12.14 12:2e2fa760:::200.00000001:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111) -- 0x561ee85ba0e0 con 0x561ee85bb5b0 2024-08-03T20:58:03.785 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:03.785+0000 7f48c6400640 1 -- 172.21.15.32:0/2928670630 <== osd.5 v2:172.21.15.118:6810/2724891770 15 ==== osd_op_reply(46 200.00000001 [omap-set-header] v111'585 uv585 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f48b801b040 con 0x561ee85bb5b0 2024-08-03T20:58:03.785 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:03.785+0000 7f48cd1edf40 1 -- 172.21.15.32:0/2928670630 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:47 12.14 12:2f55791f:::606.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111) -- 0x561ee85ba0e0 con 0x561ee85bb5b0 2024-08-03T20:58:03.787 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:03.786+0000 7f48c6400640 1 -- 172.21.15.32:0/2928670630 <== osd.5 v2:172.21.15.118:6810/2724891770 16 ==== osd_op_reply(47 606.00000000 [create] v111'586 uv569 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f48b801b040 con 0x561ee85bb5b0 2024-08-03T20:58:03.787 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:03.786+0000 7f48cd1edf40 1 -- 172.21.15.32:0/2928670630 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:48 12.14 12:2f55791f:::606.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e111) -- 0x561ee85ba0e0 con 0x561ee85bb5b0 2024-08-03T20:58:03.789 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:03.788+0000 7f48c6400640 1 -- 172.21.15.32:0/2928670630 <== osd.5 v2:172.21.15.118:6810/2724891770 17 ==== osd_op_reply(48 606.00000000 [delete] v111'587 uv587 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f48b801b040 con 0x561ee85bb5b0 2024-08-03T20:58:03.789 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:03.788+0000 7f48cd1edf40 1 -- 172.21.15.32:0/2928670630 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:49 12.14 12:2f55791f:::606.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111) -- 0x561ee85ba0e0 con 0x561ee85bb5b0 2024-08-03T20:58:03.790 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::606.00000000:head# 2024-08-03T20:58:03.790 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:03.790+0000 7f48c6400640 1 -- 172.21.15.32:0/2928670630 <== osd.5 v2:172.21.15.118:6810/2724891770 18 ==== osd_op_reply(49 606.00000000 [create] v111'588 uv588 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f48b801b040 con 0x561ee85bb5b0 2024-08-03T20:58:03.791 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:03.790+0000 7f48cd1edf40 1 -- 172.21.15.32:0/2928670630 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:50 12.14 12:2f55791f:::606.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111) -- 0x561ee85ba0e0 con 0x561ee85bb5b0 2024-08-03T20:58:03.792 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:03.792+0000 7f48c6400640 1 -- 172.21.15.32:0/2928670630 <== osd.5 v2:172.21.15.118:6810/2724891770 19 ==== osd_op_reply(50 606.00000000 [setxattr (30)] v111'589 uv589 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f48b801b040 con 0x561ee85bb5b0 2024-08-03T20:58:03.792 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:03.792+0000 7f48cd1edf40 1 -- 172.21.15.32:0/2928670630 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:51 12.14 12:2f55791f:::606.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111) -- 0x561ee85ba0e0 con 0x561ee85bb5b0 2024-08-03T20:58:03.794 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:03.793+0000 7f48c6400640 1 -- 172.21.15.32:0/2928670630 <== osd.5 v2:172.21.15.118:6810/2724891770 20 ==== osd_op_reply(51 606.00000000 [setxattr (62)] v111'590 uv590 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f48b801b040 con 0x561ee85bb5b0 2024-08-03T20:58:03.794 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:03.793+0000 7f48cd1edf40 1 -- 172.21.15.32:0/2928670630 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:52 12.14 12:2f55791f:::606.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111) -- 0x561ee85ba0e0 con 0x561ee85bb5b0 2024-08-03T20:58:03.796 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:03.795+0000 7f48c6400640 1 -- 172.21.15.32:0/2928670630 <== osd.5 v2:172.21.15.118:6810/2724891770 21 ==== osd_op_reply(52 606.00000000 [omap-set-header] v111'591 uv591 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f48b801b040 con 0x561ee85bb5b0 2024-08-03T20:58:03.796 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:03.795+0000 7f48cd1edf40 1 -- 172.21.15.32:0/2928670630 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:53 12.1c 12:3ed09add:::607.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111) -- 0x561ee85ba0e0 con 0x561ee846a990 2024-08-03T20:58:03.797 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:03.797+0000 7f48c5a00640 1 -- 172.21.15.32:0/2928670630 <== osd.0 v2:172.21.15.32:6816/3066156116 32 ==== osd_op_reply(53 607.00000000 [create] v111'153 uv152 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f48b404ec60 con 0x561ee846a990 2024-08-03T20:58:03.797 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:03.797+0000 7f48cd1edf40 1 -- 172.21.15.32:0/2928670630 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:54 12.1c 12:3ed09add:::607.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e111) -- 0x561ee85ba0e0 con 0x561ee846a990 2024-08-03T20:58:03.799 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:03.798+0000 7f48c5a00640 1 -- 172.21.15.32:0/2928670630 <== osd.0 v2:172.21.15.32:6816/3066156116 33 ==== osd_op_reply(54 607.00000000 [delete] v111'154 uv154 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f48b404ec60 con 0x561ee846a990 2024-08-03T20:58:03.799 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:03.798+0000 7f48cd1edf40 1 -- 172.21.15.32:0/2928670630 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:55 12.1c 12:3ed09add:::607.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111) -- 0x561ee85ba0e0 con 0x561ee846a990 2024-08-03T20:58:03.800 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::607.00000000:head# 2024-08-03T20:58:03.800 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:03.800+0000 7f48c5a00640 1 -- 172.21.15.32:0/2928670630 <== osd.0 v2:172.21.15.32:6816/3066156116 34 ==== osd_op_reply(55 607.00000000 [create] v111'155 uv155 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f48b404ec60 con 0x561ee846a990 2024-08-03T20:58:03.800 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:03.800+0000 7f48cd1edf40 1 -- 172.21.15.32:0/2928670630 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:56 12.1c 12:3ed09add:::607.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111) -- 0x561ee85ba0e0 con 0x561ee846a990 2024-08-03T20:58:03.801 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:03.801+0000 7f48c5a00640 1 -- 172.21.15.32:0/2928670630 <== osd.0 v2:172.21.15.32:6816/3066156116 35 ==== osd_op_reply(56 607.00000000 [setxattr (30)] v111'156 uv156 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f48b404ec60 con 0x561ee846a990 2024-08-03T20:58:03.802 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:03.801+0000 7f48cd1edf40 1 -- 172.21.15.32:0/2928670630 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:57 12.1c 12:3ed09add:::607.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111) -- 0x561ee85ba0e0 con 0x561ee846a990 2024-08-03T20:58:03.803 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:03.802+0000 7f48c5a00640 1 -- 172.21.15.32:0/2928670630 <== osd.0 v2:172.21.15.32:6816/3066156116 36 ==== osd_op_reply(57 607.00000000 [setxattr (62)] v111'157 uv157 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f48b404ec60 con 0x561ee846a990 2024-08-03T20:58:03.803 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:03.803+0000 7f48cd1edf40 1 -- 172.21.15.32:0/2928670630 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:58 12.1c 12:3ed09add:::607.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111) -- 0x561ee85ba0e0 con 0x561ee846a990 2024-08-03T20:58:03.804 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:03.804+0000 7f48c5a00640 1 -- 172.21.15.32:0/2928670630 <== osd.0 v2:172.21.15.32:6816/3066156116 37 ==== osd_op_reply(58 607.00000000 [omap-set-header] v111'158 uv158 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f48b404ec60 con 0x561ee846a990 2024-08-03T20:58:03.804 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:03.804+0000 7f48cd1edf40 1 --2- 172.21.15.32:0/2928670630 >> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] conn(0x561ee85bfe90 0x561ee85c22b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:58:03.804 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:03.804+0000 7f48cd1edf40 1 -- 172.21.15.32:0/2928670630 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:59 12.6 12:654134d2:::mds0_openfiles.0:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111) -- 0x561ee85c27f0 con 0x561ee85bfe90 2024-08-03T20:58:03.804 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:03.804+0000 7f48c5a00640 1 --2- 172.21.15.32:0/2928670630 >> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] conn(0x561ee85bfe90 0x561ee85c22b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:58:03.805 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:03.805+0000 7f48c5a00640 1 --2- 172.21.15.32:0/2928670630 >> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] conn(0x561ee85bfe90 0x561ee85c22b0 crc :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:58:03.806 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:03.806+0000 7f48c5a00640 1 -- 172.21.15.32:0/2928670630 <== osd.1 v2:172.21.15.32:6824/1551241759 1 ==== osd_op_reply(59 mds0_openfiles.0 [create] v111'118 uv117 ondisk = -17 ((17) File exists)) ==== 160+0+0 (crc 0 0 0) 0x7f48b404ec60 con 0x561ee85bfe90 2024-08-03T20:58:03.807 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:03.806+0000 7f48cd1edf40 1 -- 172.21.15.32:0/2928670630 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:60 12.6 12:654134d2:::mds0_openfiles.0:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e111) -- 0x561ee85c2830 con 0x561ee85bfe90 2024-08-03T20:58:03.808 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:03.807+0000 7f48c5a00640 1 -- 172.21.15.32:0/2928670630 <== osd.1 v2:172.21.15.32:6824/1551241759 2 ==== osd_op_reply(60 mds0_openfiles.0 [delete] v111'119 uv119 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f48b404ec60 con 0x561ee85bfe90 2024-08-03T20:58:03.808 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:03.807+0000 7f48cd1edf40 1 -- 172.21.15.32:0/2928670630 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:61 12.6 12:654134d2:::mds0_openfiles.0:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111) -- 0x561ee85c2c50 con 0x561ee85bfe90 2024-08-03T20:58:03.809 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds0_openfiles.0:head# 2024-08-03T20:58:03.809 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:03.809+0000 7f48c5a00640 1 -- 172.21.15.32:0/2928670630 <== osd.1 v2:172.21.15.32:6824/1551241759 3 ==== osd_op_reply(61 mds0_openfiles.0 [create] v111'120 uv120 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f48b404ec60 con 0x561ee85bfe90 2024-08-03T20:58:03.809 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:03.809+0000 7f48cd1edf40 1 -- 172.21.15.32:0/2928670630 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:62 12.6 12:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111) -- 0x561ee85c2c50 con 0x561ee85bfe90 2024-08-03T20:58:03.810 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:03.810+0000 7f48c5a00640 1 -- 172.21.15.32:0/2928670630 <== osd.1 v2:172.21.15.32:6824/1551241759 4 ==== osd_op_reply(62 mds0_openfiles.0 [omap-set-header] v111'121 uv121 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f48b404ec60 con 0x561ee85bfe90 2024-08-03T20:58:03.810 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:03.810+0000 7f48cd1edf40 1 -- 172.21.15.32:0/2928670630 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:63 12.e 12:717a0223:::608.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111) -- 0x561ee85c2c50 con 0x561ee846a990 2024-08-03T20:58:03.812 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:03.811+0000 7f48c5a00640 1 -- 172.21.15.32:0/2928670630 <== osd.0 v2:172.21.15.32:6816/3066156116 38 ==== osd_op_reply(63 608.00000000 [create] v111'153 uv152 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f48b404ec60 con 0x561ee846a990 2024-08-03T20:58:03.812 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:03.812+0000 7f48cd1edf40 1 -- 172.21.15.32:0/2928670630 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:64 12.e 12:717a0223:::608.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e111) -- 0x561ee85c2c50 con 0x561ee846a990 2024-08-03T20:58:03.813 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:03.813+0000 7f48c5a00640 1 -- 172.21.15.32:0/2928670630 <== osd.0 v2:172.21.15.32:6816/3066156116 39 ==== osd_op_reply(64 608.00000000 [delete] v111'154 uv154 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f48b404ec60 con 0x561ee846a990 2024-08-03T20:58:03.813 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:03.813+0000 7f48cd1edf40 1 -- 172.21.15.32:0/2928670630 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:65 12.e 12:717a0223:::608.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111) -- 0x561ee85c2c50 con 0x561ee846a990 2024-08-03T20:58:03.815 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::608.00000000:head# 2024-08-03T20:58:03.815 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:03.814+0000 7f48c5a00640 1 -- 172.21.15.32:0/2928670630 <== osd.0 v2:172.21.15.32:6816/3066156116 40 ==== osd_op_reply(65 608.00000000 [create] v111'155 uv155 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f48b404ec60 con 0x561ee846a990 2024-08-03T20:58:03.815 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:03.814+0000 7f48cd1edf40 1 -- 172.21.15.32:0/2928670630 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:66 12.e 12:717a0223:::608.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111) -- 0x561ee85c3c60 con 0x561ee846a990 2024-08-03T20:58:03.816 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:03.816+0000 7f48c5a00640 1 -- 172.21.15.32:0/2928670630 <== osd.0 v2:172.21.15.32:6816/3066156116 41 ==== osd_op_reply(66 608.00000000 [setxattr (30)] v111'156 uv156 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f48b404ec60 con 0x561ee846a990 2024-08-03T20:58:03.816 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:03.816+0000 7f48cd1edf40 1 -- 172.21.15.32:0/2928670630 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:67 12.e 12:717a0223:::608.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111) -- 0x561ee85c3c60 con 0x561ee846a990 2024-08-03T20:58:03.818 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:03.817+0000 7f48c5a00640 1 -- 172.21.15.32:0/2928670630 <== osd.0 v2:172.21.15.32:6816/3066156116 42 ==== osd_op_reply(67 608.00000000 [setxattr (62)] v111'157 uv157 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f48b404ec60 con 0x561ee846a990 2024-08-03T20:58:03.818 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:03.817+0000 7f48cd1edf40 1 -- 172.21.15.32:0/2928670630 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:68 12.e 12:717a0223:::608.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111) -- 0x561ee85c2c50 con 0x561ee846a990 2024-08-03T20:58:03.819 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:03.819+0000 7f48c5a00640 1 -- 172.21.15.32:0/2928670630 <== osd.0 v2:172.21.15.32:6816/3066156116 43 ==== osd_op_reply(68 608.00000000 [omap-set-header] v111'158 uv158 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f48b404ec60 con 0x561ee846a990 2024-08-03T20:58:03.820 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:03.819+0000 7f48cd1edf40 1 --2- 172.21.15.32:0/2928670630 >> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] conn(0x561ee85c3160 0x561ee85c5580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:58:03.820 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:03.819+0000 7f48cd1edf40 1 -- 172.21.15.32:0/2928670630 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:69 12.15 12:a93a17c2:::604.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111) -- 0x561ee85c5bc0 con 0x561ee85c3160 2024-08-03T20:58:03.820 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:03.819+0000 7f48c6e00640 1 --2- 172.21.15.32:0/2928670630 >> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] conn(0x561ee85c3160 0x561ee85c5580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:58:03.820 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:03.820+0000 7f48c6e00640 1 --2- 172.21.15.32:0/2928670630 >> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] conn(0x561ee85c3160 0x561ee85c5580 crc :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:58:03.822 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:03.822+0000 7f48c6e00640 1 -- 172.21.15.32:0/2928670630 <== osd.6 v2:172.21.15.118:6825/801947386 1 ==== osd_op_reply(69 604.00000000 [create] v111'282 uv276 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f48c000b040 con 0x561ee85c3160 2024-08-03T20:58:03.822 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:03.822+0000 7f48cd1edf40 1 -- 172.21.15.32:0/2928670630 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:70 12.15 12:a93a17c2:::604.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e111) -- 0x561ee85c60b0 con 0x561ee85c3160 2024-08-03T20:58:03.823 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:03.823+0000 7f48c6e00640 1 -- 172.21.15.32:0/2928670630 <== osd.6 v2:172.21.15.118:6825/801947386 2 ==== osd_op_reply(70 604.00000000 [delete] v111'283 uv283 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f48c000b040 con 0x561ee85c3160 2024-08-03T20:58:03.823 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:03.823+0000 7f48cd1edf40 1 -- 172.21.15.32:0/2928670630 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:71 12.15 12:a93a17c2:::604.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111) -- 0x561ee85c6590 con 0x561ee85c3160 2024-08-03T20:58:03.825 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:03.825+0000 7f48c6e00640 1 -- 172.21.15.32:0/2928670630 <== osd.6 v2:172.21.15.118:6825/801947386 3 ==== osd_op_reply(71 604.00000000 [create] v111'284 uv284 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f48c000b040 con 0x561ee85c3160 2024-08-03T20:58:03.825 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::604.00000000:head# 2024-08-03T20:58:03.825 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:03.825+0000 7f48cd1edf40 1 -- 172.21.15.32:0/2928670630 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:72 12.15 12:a93a17c2:::604.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111) -- 0x561ee85c7e40 con 0x561ee85c3160 2024-08-03T20:58:03.826 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:03.826+0000 7f48c6e00640 1 -- 172.21.15.32:0/2928670630 <== osd.6 v2:172.21.15.118:6825/801947386 4 ==== osd_op_reply(72 604.00000000 [setxattr (30)] v111'285 uv285 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f48c000b040 con 0x561ee85c3160 2024-08-03T20:58:03.827 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:03.826+0000 7f48cd1edf40 1 -- 172.21.15.32:0/2928670630 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:73 12.15 12:a93a17c2:::604.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111) -- 0x561ee85c81e0 con 0x561ee85c3160 2024-08-03T20:58:03.828 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:03.828+0000 7f48c6e00640 1 -- 172.21.15.32:0/2928670630 <== osd.6 v2:172.21.15.118:6825/801947386 5 ==== osd_op_reply(73 604.00000000 [setxattr (62)] v111'286 uv286 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f48c000b040 con 0x561ee85c3160 2024-08-03T20:58:03.828 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:03.828+0000 7f48cd1edf40 1 -- 172.21.15.32:0/2928670630 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:74 12.15 12:a93a17c2:::604.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111) -- 0x561ee85c6e30 con 0x561ee85c3160 2024-08-03T20:58:03.830 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:03.829+0000 7f48c6e00640 1 -- 172.21.15.32:0/2928670630 <== osd.6 v2:172.21.15.118:6825/801947386 6 ==== osd_op_reply(74 604.00000000 [omap-set-header] v111'287 uv287 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f48c000b040 con 0x561ee85c3160 2024-08-03T20:58:03.830 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:03.829+0000 7f48cd1edf40 1 -- 172.21.15.32:0/2928670630 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:75 12.15 12:aa448500:::500.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111) -- 0x561ee85c7240 con 0x561ee85c3160 2024-08-03T20:58:03.831 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:03.831+0000 7f48c6e00640 1 -- 172.21.15.32:0/2928670630 <== osd.6 v2:172.21.15.118:6825/801947386 7 ==== osd_op_reply(75 500.00000000 [create] v111'288 uv281 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f48c000b040 con 0x561ee85c3160 2024-08-03T20:58:03.831 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:03.831+0000 7f48cd1edf40 1 -- 172.21.15.32:0/2928670630 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:76 12.15 12:aa448500:::500.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e111) -- 0x561ee85c7630 con 0x561ee85c3160 2024-08-03T20:58:03.833 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:03.832+0000 7f48c6e00640 1 -- 172.21.15.32:0/2928670630 <== osd.6 v2:172.21.15.118:6825/801947386 8 ==== osd_op_reply(76 500.00000000 [delete] v111'289 uv289 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f48c000b040 con 0x561ee85c3160 2024-08-03T20:58:03.833 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:03.832+0000 7f48cd1edf40 1 -- 172.21.15.32:0/2928670630 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:77 12.15 12:aa448500:::500.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111) -- 0x561ee85c7630 con 0x561ee85c3160 2024-08-03T20:58:03.834 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::500.00000000:head# 2024-08-03T20:58:03.834 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:03.834+0000 7f48c6e00640 1 -- 172.21.15.32:0/2928670630 <== osd.6 v2:172.21.15.118:6825/801947386 9 ==== osd_op_reply(77 500.00000000 [create] v111'290 uv290 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f48c000b040 con 0x561ee85c3160 2024-08-03T20:58:03.835 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:03.834+0000 7f48cd1edf40 1 -- 172.21.15.32:0/2928670630 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:78 12.15 12:aa448500:::500.00000000:head [write 0~90 in=90b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111) -- 0x561ee85c75e0 con 0x561ee85c3160 2024-08-03T20:58:03.836 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:03.836+0000 7f48c6e00640 1 -- 172.21.15.32:0/2928670630 <== osd.6 v2:172.21.15.118:6825/801947386 10 ==== osd_op_reply(78 500.00000000 [write 0~90] v111'291 uv291 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f48c000b040 con 0x561ee85c3160 2024-08-03T20:58:03.836 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:03.836+0000 7f48cd1edf40 1 -- 172.21.15.32:0/2928670630 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:79 12.15 12:aa448500:::500.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111) -- 0x561ee85c75e0 con 0x561ee85c3160 2024-08-03T20:58:03.838 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:03.837+0000 7f48c6e00640 1 -- 172.21.15.32:0/2928670630 <== osd.6 v2:172.21.15.118:6825/801947386 11 ==== osd_op_reply(79 500.00000000 [omap-set-header] v111'292 uv292 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f48c000b040 con 0x561ee85c3160 2024-08-03T20:58:03.838 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:03.837+0000 7f48cd1edf40 1 --2- 172.21.15.32:0/2928670630 >> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] conn(0x561ee85c79a0 0x561ee85caaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:58:03.838 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:03.837+0000 7f48cd1edf40 1 -- 172.21.15.32:0/2928670630 --> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] -- osd_op(unknown.0.0:80 12.d 12:b50e409b:::mds_snaptable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111) -- 0x561ee85cafe0 con 0x561ee85c79a0 2024-08-03T20:58:03.838 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:03.838+0000 7f48c6400640 1 --2- 172.21.15.32:0/2928670630 >> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] conn(0x561ee85c79a0 0x561ee85caaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:58:03.839 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:03.838+0000 7f48c6400640 1 --2- 172.21.15.32:0/2928670630 >> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] conn(0x561ee85c79a0 0x561ee85caaa0 crc :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:58:03.840 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:03.840+0000 7f48c6400640 1 -- 172.21.15.32:0/2928670630 <== osd.7 v2:172.21.15.118:6818/653000048 1 ==== osd_op_reply(80 mds_snaptable [create] v111'127 uv126 ondisk = -17 ((17) File exists)) ==== 157+0+0 (crc 0 0 0) 0x7f48b801b040 con 0x561ee85c79a0 2024-08-03T20:58:03.841 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:03.840+0000 7f48cd1edf40 1 -- 172.21.15.32:0/2928670630 --> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] -- osd_op(unknown.0.0:81 12.d 12:b50e409b:::mds_snaptable:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e111) -- 0x561ee85cafe0 con 0x561ee85c79a0 2024-08-03T20:58:03.842 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:03.842+0000 7f48c6400640 1 -- 172.21.15.32:0/2928670630 <== osd.7 v2:172.21.15.118:6818/653000048 2 ==== osd_op_reply(81 mds_snaptable [delete] v111'128 uv128 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f48b801b040 con 0x561ee85c79a0 2024-08-03T20:58:03.842 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:03.842+0000 7f48cd1edf40 1 -- 172.21.15.32:0/2928670630 --> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] -- osd_op(unknown.0.0:82 12.d 12:b50e409b:::mds_snaptable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111) -- 0x561ee85cafe0 con 0x561ee85c79a0 2024-08-03T20:58:03.843 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds_snaptable:head# 2024-08-03T20:58:03.843 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:03.843+0000 7f48c6400640 1 -- 172.21.15.32:0/2928670630 <== osd.7 v2:172.21.15.118:6818/653000048 3 ==== osd_op_reply(82 mds_snaptable [create] v111'129 uv129 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f48b801b040 con 0x561ee85c79a0 2024-08-03T20:58:03.844 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:03.843+0000 7f48cd1edf40 1 -- 172.21.15.32:0/2928670630 --> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] -- osd_op(unknown.0.0:83 12.d 12:b50e409b:::mds_snaptable:head [write 0~70 in=70b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111) -- 0x561ee85cb3d0 con 0x561ee85c79a0 2024-08-03T20:58:03.845 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:03.844+0000 7f48c6400640 1 -- 172.21.15.32:0/2928670630 <== osd.7 v2:172.21.15.118:6818/653000048 4 ==== osd_op_reply(83 mds_snaptable [write 0~70] v111'130 uv130 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f48b801b040 con 0x561ee85c79a0 2024-08-03T20:58:03.845 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:03.844+0000 7f48cd1edf40 1 -- 172.21.15.32:0/2928670630 --> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] -- osd_op(unknown.0.0:84 12.d 12:b50e409b:::mds_snaptable:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111) -- 0x561ee85cb3d0 con 0x561ee85c79a0 2024-08-03T20:58:03.846 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:03.846+0000 7f48c6400640 1 -- 172.21.15.32:0/2928670630 <== osd.7 v2:172.21.15.118:6818/653000048 5 ==== osd_op_reply(84 mds_snaptable [omap-set-header] v111'131 uv131 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f48b801b040 con 0x561ee85c79a0 2024-08-03T20:58:03.846 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:03.846+0000 7f48cd1edf40 1 -- 172.21.15.32:0/2928670630 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:85 12.1d 12:b871830b:::605.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111) -- 0x561ee85cb3d0 con 0x561ee846a990 2024-08-03T20:58:03.848 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:03.847+0000 7f48c5a00640 1 -- 172.21.15.32:0/2928670630 <== osd.0 v2:172.21.15.32:6816/3066156116 44 ==== osd_op_reply(85 605.00000000 [create] v111'153 uv152 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f48b404ec60 con 0x561ee846a990 2024-08-03T20:58:03.848 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:03.847+0000 7f48cd1edf40 1 -- 172.21.15.32:0/2928670630 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:86 12.1d 12:b871830b:::605.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e111) -- 0x561ee85cb3d0 con 0x561ee846a990 2024-08-03T20:58:03.849 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:03.849+0000 7f48c5a00640 1 -- 172.21.15.32:0/2928670630 <== osd.0 v2:172.21.15.32:6816/3066156116 45 ==== osd_op_reply(86 605.00000000 [delete] v111'154 uv154 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f48b404ec60 con 0x561ee846a990 2024-08-03T20:58:03.849 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:03.849+0000 7f48cd1edf40 1 -- 172.21.15.32:0/2928670630 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:87 12.1d 12:b871830b:::605.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111) -- 0x561ee85cb3d0 con 0x561ee846a990 2024-08-03T20:58:03.850 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::605.00000000:head# 2024-08-03T20:58:03.851 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:03.850+0000 7f48c5a00640 1 -- 172.21.15.32:0/2928670630 <== osd.0 v2:172.21.15.32:6816/3066156116 46 ==== osd_op_reply(87 605.00000000 [create] v111'155 uv155 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f48b404ec60 con 0x561ee846a990 2024-08-03T20:58:03.851 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:03.850+0000 7f48cd1edf40 1 -- 172.21.15.32:0/2928670630 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:88 12.1d 12:b871830b:::605.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111) -- 0x561ee85cc3e0 con 0x561ee846a990 2024-08-03T20:58:03.851 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:03.851+0000 7f48c5a00640 1 -- 172.21.15.32:0/2928670630 <== osd.0 v2:172.21.15.32:6816/3066156116 47 ==== osd_op_reply(88 605.00000000 [setxattr (30)] v111'156 uv156 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f48b404ec60 con 0x561ee846a990 2024-08-03T20:58:03.852 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:03.851+0000 7f48cd1edf40 1 -- 172.21.15.32:0/2928670630 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:89 12.1d 12:b871830b:::605.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111) -- 0x561ee85cc3e0 con 0x561ee846a990 2024-08-03T20:58:03.853 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:03.852+0000 7f48c5a00640 1 -- 172.21.15.32:0/2928670630 <== osd.0 v2:172.21.15.32:6816/3066156116 48 ==== osd_op_reply(89 605.00000000 [setxattr (62)] v111'157 uv157 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f48b404ec60 con 0x561ee846a990 2024-08-03T20:58:03.853 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:03.852+0000 7f48cd1edf40 1 -- 172.21.15.32:0/2928670630 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:90 12.1d 12:b871830b:::605.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111) -- 0x561ee85cb3d0 con 0x561ee846a990 2024-08-03T20:58:03.854 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:03.854+0000 7f48c5a00640 1 -- 172.21.15.32:0/2928670630 <== osd.0 v2:172.21.15.32:6816/3066156116 49 ==== osd_op_reply(90 605.00000000 [omap-set-header] v111'158 uv158 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f48b404ec60 con 0x561ee846a990 2024-08-03T20:58:03.854 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:03.854+0000 7f48cd1edf40 1 -- 172.21.15.32:0/2928670630 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:91 12.13 12:c91d4a1d:::mds0_inotable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111) -- 0x561ee85cb3d0 con 0x561ee85bfe90 2024-08-03T20:58:03.855 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:03.855+0000 7f48c5a00640 1 -- 172.21.15.32:0/2928670630 <== osd.1 v2:172.21.15.32:6824/1551241759 5 ==== osd_op_reply(91 mds0_inotable [create] v111'253 uv247 ondisk = -17 ((17) File exists)) ==== 157+0+0 (crc 0 0 0) 0x7f48b404ec60 con 0x561ee85bfe90 2024-08-03T20:58:03.856 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:03.855+0000 7f48cd1edf40 1 -- 172.21.15.32:0/2928670630 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:92 12.13 12:c91d4a1d:::mds0_inotable:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e111) -- 0x561ee85cb3d0 con 0x561ee85bfe90 2024-08-03T20:58:03.857 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:03.856+0000 7f48c5a00640 1 -- 172.21.15.32:0/2928670630 <== osd.1 v2:172.21.15.32:6824/1551241759 6 ==== osd_op_reply(92 mds0_inotable [delete] v111'254 uv254 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f48b404ec60 con 0x561ee85bfe90 2024-08-03T20:58:03.857 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:03.856+0000 7f48cd1edf40 1 -- 172.21.15.32:0/2928670630 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:93 12.13 12:c91d4a1d:::mds0_inotable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111) -- 0x561ee85cb3d0 con 0x561ee85bfe90 2024-08-03T20:58:03.858 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds0_inotable:head# 2024-08-03T20:58:03.858 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:03.858+0000 7f48c5a00640 1 -- 172.21.15.32:0/2928670630 <== osd.1 v2:172.21.15.32:6824/1551241759 7 ==== osd_op_reply(93 mds0_inotable [create] v111'255 uv255 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f48b404ec60 con 0x561ee85bfe90 2024-08-03T20:58:03.858 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:03.858+0000 7f48cd1edf40 1 -- 172.21.15.32:0/2928670630 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:94 12.13 12:c91d4a1d:::mds0_inotable:head [write 0~34 in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111) -- 0x561ee85cb3d0 con 0x561ee85bfe90 2024-08-03T20:58:03.859 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:03.859+0000 7f48c5a00640 1 -- 172.21.15.32:0/2928670630 <== osd.1 v2:172.21.15.32:6824/1551241759 8 ==== osd_op_reply(94 mds0_inotable [write 0~34] v111'256 uv256 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f48b404ec60 con 0x561ee85bfe90 2024-08-03T20:58:03.859 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:03.859+0000 7f48cd1edf40 1 -- 172.21.15.32:0/2928670630 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:95 12.13 12:c91d4a1d:::mds0_inotable:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111) -- 0x561ee85cb3d0 con 0x561ee85bfe90 2024-08-03T20:58:03.861 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:03.860+0000 7f48c5a00640 1 -- 172.21.15.32:0/2928670630 <== osd.1 v2:172.21.15.32:6824/1551241759 9 ==== osd_op_reply(95 mds0_inotable [omap-set-header] v111'257 uv257 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f48b404ec60 con 0x561ee85bfe90 2024-08-03T20:58:03.861 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:03.860+0000 7f48cd1edf40 1 -- 172.21.15.32:0/2928670630 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:96 12.13 12:cd5a64a3:::100.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111) -- 0x561ee85cb3d0 con 0x561ee85bfe90 2024-08-03T20:58:03.862 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:03.862+0000 7f48c5a00640 1 -- 172.21.15.32:0/2928670630 <== osd.1 v2:172.21.15.32:6824/1551241759 10 ==== osd_op_reply(96 100.00000000 [create] v111'258 uv252 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f48b404ec60 con 0x561ee85bfe90 2024-08-03T20:58:03.862 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:03.862+0000 7f48cd1edf40 1 -- 172.21.15.32:0/2928670630 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:97 12.13 12:cd5a64a3:::100.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e111) -- 0x561ee85cb3d0 con 0x561ee85bfe90 2024-08-03T20:58:03.863 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:03.863+0000 7f48c5a00640 1 -- 172.21.15.32:0/2928670630 <== osd.1 v2:172.21.15.32:6824/1551241759 11 ==== osd_op_reply(97 100.00000000 [delete] v111'259 uv259 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f48b404ec60 con 0x561ee85bfe90 2024-08-03T20:58:03.863 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:03.863+0000 7f48cd1edf40 1 -- 172.21.15.32:0/2928670630 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:98 12.13 12:cd5a64a3:::100.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111) -- 0x561ee85cb3d0 con 0x561ee85bfe90 2024-08-03T20:58:03.865 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:03.864+0000 7f48c5a00640 1 -- 172.21.15.32:0/2928670630 <== osd.1 v2:172.21.15.32:6824/1551241759 12 ==== osd_op_reply(98 100.00000000 [create] v111'260 uv260 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f48b404ec60 con 0x561ee85bfe90 2024-08-03T20:58:03.865 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::100.00000000:head# 2024-08-03T20:58:03.865 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:03.864+0000 7f48cd1edf40 1 -- 172.21.15.32:0/2928670630 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:99 12.13 12:cd5a64a3:::100.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111) -- 0x561ee85cb3d0 con 0x561ee85bfe90 2024-08-03T20:58:03.866 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:03.866+0000 7f48c5a00640 1 -- 172.21.15.32:0/2928670630 <== osd.1 v2:172.21.15.32:6824/1551241759 13 ==== osd_op_reply(99 100.00000000 [omap-set-header] v111'261 uv261 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f48b404ec60 con 0x561ee85bfe90 2024-08-03T20:58:03.866 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:03.866+0000 7f48cd1edf40 1 -- 172.21.15.32:0/2928670630 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:100 12.13 12:cd5a64a3:::100.00000000:head [omap-set-vals in=5194b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111) -- 0x561ee85ce1e0 con 0x561ee85bfe90 2024-08-03T20:58:03.868 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:03.867+0000 7f48c5a00640 1 -- 172.21.15.32:0/2928670630 <== osd.1 v2:172.21.15.32:6824/1551241759 14 ==== osd_op_reply(100 100.00000000 [omap-set-vals] v111'262 uv262 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f48b404ec60 con 0x561ee85bfe90 2024-08-03T20:58:03.868 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:03.867+0000 7f48cd1edf40 1 --2- 172.21.15.32:0/2928670630 >> [v2:172.21.15.32:6800/9088303,v1:172.21.15.32:6801/9088303] conn(0x561ee85cbf00 0x561ee85ce7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:58:03.868 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:03.867+0000 7f48cd1edf40 1 -- 172.21.15.32:0/2928670630 --> [v2:172.21.15.32:6800/9088303,v1:172.21.15.32:6801/9088303] -- osd_op(unknown.0.0:101 12.b 12:d0630e4c:::mds0_sessionmap:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111) -- 0x561ee85ce320 con 0x561ee85cbf00 2024-08-03T20:58:03.868 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:03.867+0000 7f48c5a00640 1 --2- 172.21.15.32:0/2928670630 >> [v2:172.21.15.32:6800/9088303,v1:172.21.15.32:6801/9088303] conn(0x561ee85cbf00 0x561ee85ce7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:58:03.868 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:03.868+0000 7f48c5a00640 1 --2- 172.21.15.32:0/2928670630 >> [v2:172.21.15.32:6800/9088303,v1:172.21.15.32:6801/9088303] conn(0x561ee85cbf00 0x561ee85ce7c0 crc :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:58:03.870 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:03.869+0000 7f48c5a00640 1 -- 172.21.15.32:0/2928670630 <== osd.3 v2:172.21.15.32:6800/9088303 1 ==== osd_op_reply(101 mds0_sessionmap [create] v111'111 uv110 ondisk = -17 ((17) File exists)) ==== 159+0+0 (crc 0 0 0) 0x7f48b404ec60 con 0x561ee85cbf00 2024-08-03T20:58:03.870 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:03.869+0000 7f48cd1edf40 1 -- 172.21.15.32:0/2928670630 --> [v2:172.21.15.32:6800/9088303,v1:172.21.15.32:6801/9088303] -- osd_op(unknown.0.0:102 12.b 12:d0630e4c:::mds0_sessionmap:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e111) -- 0x561ee85ce320 con 0x561ee85cbf00 2024-08-03T20:58:03.871 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:03.871+0000 7f48c5a00640 1 -- 172.21.15.32:0/2928670630 <== osd.3 v2:172.21.15.32:6800/9088303 2 ==== osd_op_reply(102 mds0_sessionmap [delete] v111'112 uv112 ondisk = 0) ==== 159+0+0 (crc 0 0 0) 0x7f48b404ec60 con 0x561ee85cbf00 2024-08-03T20:58:03.871 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:03.871+0000 7f48cd1edf40 1 -- 172.21.15.32:0/2928670630 --> [v2:172.21.15.32:6800/9088303,v1:172.21.15.32:6801/9088303] -- osd_op(unknown.0.0:103 12.b 12:d0630e4c:::mds0_sessionmap:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111) -- 0x561ee85ce320 con 0x561ee85cbf00 2024-08-03T20:58:03.872 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds0_sessionmap:head# 2024-08-03T20:58:03.873 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:03.872+0000 7f48c5a00640 1 -- 172.21.15.32:0/2928670630 <== osd.3 v2:172.21.15.32:6800/9088303 3 ==== osd_op_reply(103 mds0_sessionmap [create] v111'113 uv113 ondisk = 0) ==== 159+0+0 (crc 0 0 0) 0x7f48b404ec60 con 0x561ee85cbf00 2024-08-03T20:58:03.873 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:03.872+0000 7f48cd1edf40 1 -- 172.21.15.32:0/2928670630 --> [v2:172.21.15.32:6800/9088303,v1:172.21.15.32:6801/9088303] -- osd_op(unknown.0.0:104 12.b 12:d0630e4c:::mds0_sessionmap:head [omap-set-header in=14b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111) -- 0x561ee85ce320 con 0x561ee85cbf00 2024-08-03T20:58:03.874 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:03.873+0000 7f48c5a00640 1 -- 172.21.15.32:0/2928670630 <== osd.3 v2:172.21.15.32:6800/9088303 4 ==== osd_op_reply(104 mds0_sessionmap [omap-set-header] v111'114 uv114 ondisk = 0) ==== 159+0+0 (crc 0 0 0) 0x7f48b404ec60 con 0x561ee85cbf00 2024-08-03T20:58:03.874 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:03.874+0000 7f48cd1edf40 1 -- 172.21.15.32:0/2928670630 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:105 12.7 12:e0b41d06:::609.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111) -- 0x561ee85ce320 con 0x561ee85bfe90 2024-08-03T20:58:03.876 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:03.875+0000 7f48c5a00640 1 -- 172.21.15.32:0/2928670630 <== osd.1 v2:172.21.15.32:6824/1551241759 15 ==== osd_op_reply(105 609.00000000 [create] v111'153 uv152 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f48b404ec60 con 0x561ee85bfe90 2024-08-03T20:58:03.876 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:03.875+0000 7f48cd1edf40 1 -- 172.21.15.32:0/2928670630 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:106 12.7 12:e0b41d06:::609.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e111) -- 0x561ee85ce320 con 0x561ee85bfe90 2024-08-03T20:58:03.877 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:03.877+0000 7f48c5a00640 1 -- 172.21.15.32:0/2928670630 <== osd.1 v2:172.21.15.32:6824/1551241759 16 ==== osd_op_reply(106 609.00000000 [delete] v111'154 uv154 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f48b404ec60 con 0x561ee85bfe90 2024-08-03T20:58:03.877 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:03.877+0000 7f48cd1edf40 1 -- 172.21.15.32:0/2928670630 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:107 12.7 12:e0b41d06:::609.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111) -- 0x561ee85ce320 con 0x561ee85bfe90 2024-08-03T20:58:03.878 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::609.00000000:head# 2024-08-03T20:58:03.879 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:03.878+0000 7f48c5a00640 1 -- 172.21.15.32:0/2928670630 <== osd.1 v2:172.21.15.32:6824/1551241759 17 ==== osd_op_reply(107 609.00000000 [create] v111'155 uv155 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f48b404ec60 con 0x561ee85bfe90 2024-08-03T20:58:03.879 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:03.878+0000 7f48cd1edf40 1 -- 172.21.15.32:0/2928670630 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:108 12.7 12:e0b41d06:::609.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111) -- 0x561ee85ce320 con 0x561ee85bfe90 2024-08-03T20:58:03.880 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:03.879+0000 7f48c5a00640 1 -- 172.21.15.32:0/2928670630 <== osd.1 v2:172.21.15.32:6824/1551241759 18 ==== osd_op_reply(108 609.00000000 [setxattr (30)] v111'156 uv156 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f48b404ec60 con 0x561ee85bfe90 2024-08-03T20:58:03.880 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:03.879+0000 7f48cd1edf40 1 -- 172.21.15.32:0/2928670630 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:109 12.7 12:e0b41d06:::609.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111) -- 0x561ee85ce320 con 0x561ee85bfe90 2024-08-03T20:58:03.881 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:03.881+0000 7f48c5a00640 1 -- 172.21.15.32:0/2928670630 <== osd.1 v2:172.21.15.32:6824/1551241759 19 ==== osd_op_reply(109 609.00000000 [setxattr (62)] v111'157 uv157 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f48b404ec60 con 0x561ee85bfe90 2024-08-03T20:58:03.881 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:03.881+0000 7f48cd1edf40 1 -- 172.21.15.32:0/2928670630 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:110 12.7 12:e0b41d06:::609.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111) -- 0x561ee85ce320 con 0x561ee85bfe90 2024-08-03T20:58:03.882 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:03.882+0000 7f48c5a00640 1 -- 172.21.15.32:0/2928670630 <== osd.1 v2:172.21.15.32:6824/1551241759 20 ==== osd_op_reply(110 609.00000000 [omap-set-header] v111'158 uv158 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f48b404ec60 con 0x561ee85bfe90 2024-08-03T20:58:03.883 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:03.882+0000 7f48cd1edf40 1 -- 172.21.15.32:0/2928670630 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:111 12.f 12:f1f69726:::400.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111) -- 0x561ee85ce320 con 0x561ee85c3160 2024-08-03T20:58:03.884 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:03.884+0000 7f48c6e00640 1 -- 172.21.15.32:0/2928670630 <== osd.6 v2:172.21.15.118:6825/801947386 12 ==== osd_op_reply(111 400.00000000 [create] v111'130 uv129 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f48c000b040 con 0x561ee85c3160 2024-08-03T20:58:03.884 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:03.884+0000 7f48cd1edf40 1 -- 172.21.15.32:0/2928670630 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:112 12.f 12:f1f69726:::400.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e111) -- 0x561ee85ce320 con 0x561ee85c3160 2024-08-03T20:58:03.886 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:03.885+0000 7f48c6e00640 1 -- 172.21.15.32:0/2928670630 <== osd.6 v2:172.21.15.118:6825/801947386 13 ==== osd_op_reply(112 400.00000000 [delete] v111'131 uv131 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f48c000b040 con 0x561ee85c3160 2024-08-03T20:58:03.886 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:03.885+0000 7f48cd1edf40 1 -- 172.21.15.32:0/2928670630 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:113 12.f 12:f1f69726:::400.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111) -- 0x561ee85ce320 con 0x561ee85c3160 2024-08-03T20:58:03.887 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:03.887+0000 7f48c6e00640 1 -- 172.21.15.32:0/2928670630 <== osd.6 v2:172.21.15.118:6825/801947386 14 ==== osd_op_reply(113 400.00000000 [create] v111'132 uv132 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f48c000b040 con 0x561ee85c3160 2024-08-03T20:58:03.887 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::400.00000000:head# 2024-08-03T20:58:03.888 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:03.887+0000 7f48cd1edf40 1 -- 172.21.15.32:0/2928670630 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:114 12.f 12:f1f69726:::400.00000000:head [write 0~22 in=22b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111) -- 0x561ee85ced00 con 0x561ee85c3160 2024-08-03T20:58:03.889 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:03.888+0000 7f48c6e00640 1 -- 172.21.15.32:0/2928670630 <== osd.6 v2:172.21.15.118:6825/801947386 15 ==== osd_op_reply(114 400.00000000 [write 0~22] v111'133 uv133 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f48c000b040 con 0x561ee85c3160 2024-08-03T20:58:03.889 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:03.888+0000 7f48cd1edf40 1 -- 172.21.15.32:0/2928670630 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:115 12.f 12:f1f69726:::400.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111) -- 0x561ee85ced00 con 0x561ee85c3160 2024-08-03T20:58:03.890 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:03.890+0000 7f48c6e00640 1 -- 172.21.15.32:0/2928670630 <== osd.6 v2:172.21.15.118:6825/801947386 16 ==== osd_op_reply(115 400.00000000 [omap-set-header] v111'134 uv134 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f48c000b040 con 0x561ee85c3160 2024-08-03T20:58:03.891 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:03.890+0000 7f48cd1edf40 1 -- 172.21.15.32:0/2928670630 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:116 12.1f 12:fe07bba1:::100.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111) -- 0x561ee85ced00 con 0x561ee85c3160 2024-08-03T20:58:03.892 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:03.892+0000 7f48c6e00640 1 -- 172.21.15.32:0/2928670630 <== osd.6 v2:172.21.15.118:6825/801947386 17 ==== osd_op_reply(116 100.00000000.inode [create] v111'314 uv306 ondisk = -17 ((17) File exists)) ==== 162+0+0 (crc 0 0 0) 0x7f48c000b040 con 0x561ee85c3160 2024-08-03T20:58:03.892 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:03.892+0000 7f48cd1edf40 1 -- 172.21.15.32:0/2928670630 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:117 12.1f 12:fe07bba1:::100.00000000.inode:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e111) -- 0x561ee85ced00 con 0x561ee85c3160 2024-08-03T20:58:03.894 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:03.893+0000 7f48c6e00640 1 -- 172.21.15.32:0/2928670630 <== osd.6 v2:172.21.15.118:6825/801947386 18 ==== osd_op_reply(117 100.00000000.inode [delete] v111'315 uv315 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7f48c000b040 con 0x561ee85c3160 2024-08-03T20:58:03.894 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:03.893+0000 7f48cd1edf40 1 -- 172.21.15.32:0/2928670630 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:118 12.1f 12:fe07bba1:::100.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111) -- 0x561ee85ced00 con 0x561ee85c3160 2024-08-03T20:58:03.895 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::100.00000000.inode:head# 2024-08-03T20:58:03.896 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:03.895+0000 7f48c6e00640 1 -- 172.21.15.32:0/2928670630 <== osd.6 v2:172.21.15.118:6825/801947386 19 ==== osd_op_reply(118 100.00000000.inode [create] v111'316 uv316 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7f48c000b040 con 0x561ee85c3160 2024-08-03T20:58:03.896 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:03.895+0000 7f48cd1edf40 1 -- 172.21.15.32:0/2928670630 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:119 12.1f 12:fe07bba1:::100.00000000.inode:head [write 0~582 in=582b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111) -- 0x561ee85ced00 con 0x561ee85c3160 2024-08-03T20:58:03.897 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:03.897+0000 7f48c6e00640 1 -- 172.21.15.32:0/2928670630 <== osd.6 v2:172.21.15.118:6825/801947386 20 ==== osd_op_reply(119 100.00000000.inode [write 0~582] v111'317 uv317 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7f48c000b040 con 0x561ee85c3160 2024-08-03T20:58:03.897 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:03.897+0000 7f48cd1edf40 1 -- 172.21.15.32:0/2928670630 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:120 12.1f 12:fe07bba1:::100.00000000.inode:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111) -- 0x561ee85ced00 con 0x561ee85c3160 2024-08-03T20:58:03.898 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:03.898+0000 7f48c6e00640 1 -- 172.21.15.32:0/2928670630 <== osd.6 v2:172.21.15.118:6825/801947386 21 ==== osd_op_reply(120 100.00000000.inode [omap-set-header] v111'318 uv318 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7f48c000b040 con 0x561ee85c3160 2024-08-03T20:58:03.898 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:03.898+0000 7f48cd1edf40 1 -- 172.21.15.32:0/2928670630 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:121 12.1f 12:ff5b34d6:::1.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111) -- 0x561ee85ced00 con 0x561ee85c3160 2024-08-03T20:58:03.900 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:03.900+0000 7f48c6e00640 1 -- 172.21.15.32:0/2928670630 <== osd.6 v2:172.21.15.118:6825/801947386 22 ==== osd_op_reply(121 1.00000000 [create] v111'319 uv313 ondisk = -17 ((17) File exists)) ==== 154+0+0 (crc 0 0 0) 0x7f48c000b040 con 0x561ee85c3160 2024-08-03T20:58:03.900 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:03.900+0000 7f48cd1edf40 1 -- 172.21.15.32:0/2928670630 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:122 12.1f 12:ff5b34d6:::1.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e111) -- 0x561ee85ced00 con 0x561ee85c3160 2024-08-03T20:58:03.901 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:03.901+0000 7f48c6e00640 1 -- 172.21.15.32:0/2928670630 <== osd.6 v2:172.21.15.118:6825/801947386 23 ==== osd_op_reply(122 1.00000000 [delete] v111'320 uv320 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7f48c000b040 con 0x561ee85c3160 2024-08-03T20:58:03.901 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:03.901+0000 7f48cd1edf40 1 -- 172.21.15.32:0/2928670630 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:123 12.1f 12:ff5b34d6:::1.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111) -- 0x561ee85ced00 con 0x561ee85c3160 2024-08-03T20:58:03.903 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::1.00000000:head# 2024-08-03T20:58:03.903 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:03.902+0000 7f48c6e00640 1 -- 172.21.15.32:0/2928670630 <== osd.6 v2:172.21.15.118:6825/801947386 24 ==== osd_op_reply(123 1.00000000 [create] v111'321 uv321 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7f48c000b040 con 0x561ee85c3160 2024-08-03T20:58:03.903 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:03.902+0000 7f48cd1edf40 1 -- 172.21.15.32:0/2928670630 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:124 12.1f 12:ff5b34d6:::1.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111) -- 0x561ee85cfd10 con 0x561ee85c3160 2024-08-03T20:58:03.904 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:03.904+0000 7f48c6e00640 1 -- 172.21.15.32:0/2928670630 <== osd.6 v2:172.21.15.118:6825/801947386 25 ==== osd_op_reply(124 1.00000000 [setxattr (30)] v111'322 uv322 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7f48c000b040 con 0x561ee85c3160 2024-08-03T20:58:03.904 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:03.904+0000 7f48cd1edf40 1 -- 172.21.15.32:0/2928670630 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:125 12.1f 12:ff5b34d6:::1.00000000:head [setxattr parent (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111) -- 0x561ee85cfd10 con 0x561ee85c3160 2024-08-03T20:58:03.905 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:03.905+0000 7f48c6e00640 1 -- 172.21.15.32:0/2928670630 <== osd.6 v2:172.21.15.118:6825/801947386 26 ==== osd_op_reply(125 1.00000000 [setxattr (30)] v111'323 uv323 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7f48c000b040 con 0x561ee85c3160 2024-08-03T20:58:03.906 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:03.905+0000 7f48cd1edf40 1 -- 172.21.15.32:0/2928670630 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:126 12.1f 12:ff5b34d6:::1.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111) -- 0x561ee85ced00 con 0x561ee85c3160 2024-08-03T20:58:03.907 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:03.907+0000 7f48c6e00640 1 -- 172.21.15.32:0/2928670630 <== osd.6 v2:172.21.15.118:6825/801947386 27 ==== osd_op_reply(126 1.00000000 [omap-set-header] v111'324 uv324 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7f48c000b040 con 0x561ee85c3160 2024-08-03T20:58:03.907 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:03.907+0000 7f48cd1edf40 1 -- 172.21.15.32:0/2928670630 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:127 12.1f 12:ff5b34d6:::1.00000000:head [omap-set-vals in=584b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111) -- 0x561ee85cf460 con 0x561ee85c3160 2024-08-03T20:58:03.909 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:03.908+0000 7f48c6e00640 1 -- 172.21.15.32:0/2928670630 <== osd.6 v2:172.21.15.118:6825/801947386 28 ==== osd_op_reply(127 1.00000000 [omap-set-vals] v111'325 uv325 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7f48c000b040 con 0x561ee85c3160 2024-08-03T20:58:03.910 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:03.909+0000 7f48cd1edf40 1 -- 172.21.15.32:0/2928670630 >> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] conn(0x561ee846a990 msgr2=0x561ee846ada0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:58:03.910 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:03.909+0000 7f48cd1edf40 1 --2- 172.21.15.32:0/2928670630 >> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] conn(0x561ee846a990 0x561ee846ada0 crc :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:03.910 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:03.909+0000 7f48cd1edf40 1 -- 172.21.15.32:0/2928670630 >> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] conn(0x561ee85bfe90 msgr2=0x561ee85c22b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:58:03.910 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:03.909+0000 7f48cd1edf40 1 --2- 172.21.15.32:0/2928670630 >> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] conn(0x561ee85bfe90 0x561ee85c22b0 crc :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:03.910 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:03.909+0000 7f48cd1edf40 1 -- 172.21.15.32:0/2928670630 >> [v2:172.21.15.32:6800/9088303,v1:172.21.15.32:6801/9088303] conn(0x561ee85cbf00 msgr2=0x561ee85ce7c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:58:03.910 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:03.909+0000 7f48cd1edf40 1 --2- 172.21.15.32:0/2928670630 >> [v2:172.21.15.32:6800/9088303,v1:172.21.15.32:6801/9088303] conn(0x561ee85cbf00 0x561ee85ce7c0 crc :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:03.910 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:03.909+0000 7f48cd1edf40 1 -- 172.21.15.32:0/2928670630 >> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] conn(0x561ee85bb5b0 msgr2=0x561ee85bd9d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:58:03.910 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:03.909+0000 7f48cd1edf40 1 --2- 172.21.15.32:0/2928670630 >> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] conn(0x561ee85bb5b0 0x561ee85bd9d0 crc :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:03.910 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:03.909+0000 7f48cd1edf40 1 -- 172.21.15.32:0/2928670630 >> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] conn(0x561ee85c3160 msgr2=0x561ee85c5580 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:58:03.910 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:03.909+0000 7f48cd1edf40 1 --2- 172.21.15.32:0/2928670630 >> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] conn(0x561ee85c3160 0x561ee85c5580 crc :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:03.910 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:03.909+0000 7f48c6e00640 1 -- 172.21.15.32:0/2928670630 reap_dead start 2024-08-03T20:58:03.910 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:03.909+0000 7f48cd1edf40 1 -- 172.21.15.32:0/2928670630 >> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] conn(0x561ee85c79a0 msgr2=0x561ee85caaa0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:58:03.911 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:03.909+0000 7f48cd1edf40 1 --2- 172.21.15.32:0/2928670630 >> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] conn(0x561ee85c79a0 0x561ee85caaa0 crc :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:03.911 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:03.909+0000 7f48cd1edf40 1 -- 172.21.15.32:0/2928670630 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f48a80759f0 msgr2=0x7f48a8077f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:58:03.911 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:03.909+0000 7f48cd1edf40 1 --2- 172.21.15.32:0/2928670630 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f48a80759f0 0x7f48a8077f00 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7f48b8010410 tx=0x7f48b8001730 comp rx=0 tx=0).stop 2024-08-03T20:58:03.911 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:03.909+0000 7f48cd1edf40 1 -- 172.21.15.32:0/2928670630 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561ee85195b0 msgr2=0x561ee8486460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:58:03.911 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:03.909+0000 7f48cd1edf40 1 --2- 172.21.15.32:0/2928670630 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561ee85195b0 0x561ee8486460 secure :-1 s=READY pgs=1699 cs=0 l=1 rev1=1 crypto rx=0x7f48c000d720 tx=0x7f48c000f6e0 comp rx=0 tx=0).stop 2024-08-03T20:58:03.911 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:03.909+0000 7f48cd1edf40 1 -- 172.21.15.32:0/2928670630 shutdown_connections 2024-08-03T20:58:03.911 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:03.909+0000 7f48cd1edf40 1 --2- 172.21.15.32:0/2928670630 >> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] conn(0x561ee85c79a0 0x561ee85caaa0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:03.911 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:03.909+0000 7f48cd1edf40 1 --2- 172.21.15.32:0/2928670630 >> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] conn(0x561ee85c3160 0x561ee85c5580 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:03.911 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:03.909+0000 7f48cd1edf40 1 --2- 172.21.15.32:0/2928670630 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f48a80759f0 0x7f48a8077f00 unknown :-1 s=CLOSED pgs=693 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:03.911 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:03.909+0000 7f48cd1edf40 1 --2- 172.21.15.32:0/2928670630 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561ee85195b0 0x561ee8486460 unknown :-1 s=CLOSED pgs=1699 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:03.911 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:03.909+0000 7f48cd1edf40 1 -- 172.21.15.32:0/2928670630 >> 172.21.15.32:0/2928670630 conn(0x561ee8360ac0 msgr2=0x561ee8485020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:58:03.911 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:03.909+0000 7f48cd1edf40 1 -- 172.21.15.32:0/2928670630 shutdown_connections 2024-08-03T20:58:03.911 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:03.909+0000 7f48cd1edf40 1 -- 172.21.15.32:0/2928670630 wait complete. 2024-08-03T20:58:03.921 DEBUG:teuthology.orchestra.run.smithi032:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata truncate 500.00000000 0 2024-08-03T20:58:04.104 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:04.104+0000 7f924488cf40 1 Processor -- start 2024-08-03T20:58:04.105 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:04.104+0000 7f924488cf40 1 -- start start 2024-08-03T20:58:04.107 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:04.105+0000 7f924488cf40 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55f73ae59360 0x55f73ae53000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:58:04.107 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:04.105+0000 7f924488cf40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55f73ae76cb0 0x55f73ade9020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:58:04.107 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:04.105+0000 7f924488cf40 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55f73ae773f0 0x55f73ade9560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:58:04.107 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:04.105+0000 7f924488cf40 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55f73acf0e30 con 0x55f73ae773f0 2024-08-03T20:58:04.107 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:04.105+0000 7f924488cf40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x55f73ad4ebf0 con 0x55f73ae76cb0 2024-08-03T20:58:04.107 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:04.105+0000 7f924488cf40 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x55f73ae5a440 con 0x55f73ae59360 2024-08-03T20:58:04.107 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:04.106+0000 7f923e400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55f73ae773f0 0x55f73ade9560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:58:04.108 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:04.106+0000 7f923e400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55f73ae773f0 0x55f73ade9560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:51102/0 (socket says 172.21.15.32:51102) 2024-08-03T20:58:04.108 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:04.106+0000 7f923e400640 1 -- 172.21.15.32:0/2772657334 learned_addr learned my addr 172.21.15.32:0/2772657334 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:58:04.108 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:04.106+0000 7f923da00640 1 --2- 172.21.15.32:0/2772657334 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55f73ae59360 0x55f73ae53000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:58:04.108 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:04.106+0000 7f923d000640 1 --2- 172.21.15.32:0/2772657334 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55f73ae76cb0 0x55f73ade9020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:58:04.108 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:04.106+0000 7f923e400640 1 -- 172.21.15.32:0/2772657334 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55f73ae59360 msgr2=0x55f73ae53000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:58:04.108 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:04.106+0000 7f923e400640 1 --2- 172.21.15.32:0/2772657334 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55f73ae59360 0x55f73ae53000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:04.108 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:04.106+0000 7f923e400640 1 -- 172.21.15.32:0/2772657334 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55f73ae76cb0 msgr2=0x55f73ade9020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:58:04.108 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:04.106+0000 7f923e400640 1 --2- 172.21.15.32:0/2772657334 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55f73ae76cb0 0x55f73ade9020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:04.108 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:04.106+0000 7f923e400640 1 -- 172.21.15.32:0/2772657334 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f73ae57600 con 0x55f73ae773f0 2024-08-03T20:58:04.108 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:04.106+0000 7f923da00640 1 --2- 172.21.15.32:0/2772657334 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55f73ae59360 0x55f73ae53000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-03T20:58:04.108 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:04.106+0000 7f923d000640 1 --2- 172.21.15.32:0/2772657334 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55f73ae76cb0 0x55f73ade9020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-03T20:58:04.108 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:04.106+0000 7f923e400640 1 --2- 172.21.15.32:0/2772657334 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55f73ae773f0 0x55f73ade9560 secure :-1 s=READY pgs=1702 cs=0 l=1 rev1=1 crypto rx=0x7f923800b7e0 tx=0x7f923800bca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=839872e78cdffff2 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:58:04.108 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:04.107+0000 7f9237400640 1 -- 172.21.15.32:0/2772657334 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f923800db30 con 0x55f73ae773f0 2024-08-03T20:58:04.108 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:04.107+0000 7f9237400640 1 -- 172.21.15.32:0/2772657334 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f923800ddc0 con 0x55f73ae773f0 2024-08-03T20:58:04.109 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:04.107+0000 7f9237400640 1 -- 172.21.15.32:0/2772657334 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f923800f690 con 0x55f73ae773f0 2024-08-03T20:58:04.110 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:04.107+0000 7f924488cf40 1 -- 172.21.15.32:0/2772657334 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55f73ae773f0 msgr2=0x55f73ade9560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:58:04.110 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:04.107+0000 7f924488cf40 1 --2- 172.21.15.32:0/2772657334 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55f73ae773f0 0x55f73ade9560 secure :-1 s=READY pgs=1702 cs=0 l=1 rev1=1 crypto rx=0x7f923800b7e0 tx=0x7f923800bca0 comp rx=0 tx=0).stop 2024-08-03T20:58:04.110 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:04.107+0000 7f924488cf40 1 -- 172.21.15.32:0/2772657334 shutdown_connections 2024-08-03T20:58:04.110 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:04.107+0000 7f924488cf40 1 --2- 172.21.15.32:0/2772657334 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55f73ae76cb0 0x55f73ade9020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:04.110 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:04.107+0000 7f924488cf40 1 --2- 172.21.15.32:0/2772657334 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55f73ae773f0 0x55f73ade9560 unknown :-1 s=CLOSED pgs=1702 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:04.110 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:04.107+0000 7f924488cf40 1 --2- 172.21.15.32:0/2772657334 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55f73ae59360 0x55f73ae53000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:04.110 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:04.107+0000 7f924488cf40 1 -- 172.21.15.32:0/2772657334 >> 172.21.15.32:0/2772657334 conn(0x55f73acc2ac0 msgr2=0x55f73ae74d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:58:04.110 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:04.107+0000 7f924488cf40 1 -- 172.21.15.32:0/2772657334 shutdown_connections 2024-08-03T20:58:04.110 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:04.107+0000 7f924488cf40 1 -- 172.21.15.32:0/2772657334 wait complete. 2024-08-03T20:58:04.110 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:04.108+0000 7f924488cf40 1 Processor -- start 2024-08-03T20:58:04.110 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:04.109+0000 7f924488cf40 1 -- start start 2024-08-03T20:58:04.112 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:04.110+0000 7f924488cf40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55f73ae59360 0x55f73ade23a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:58:04.112 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:04.110+0000 7f924488cf40 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55f73ae76cb0 0x55f73ade78f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:58:04.112 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:04.110+0000 7f924488cf40 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55f73ae773f0 0x55f73ade7e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:58:04.112 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:04.110+0000 7f924488cf40 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55f73ada2e10 con 0x55f73ae773f0 2024-08-03T20:58:04.112 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:04.110+0000 7f924488cf40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x55f73ae56e70 con 0x55f73ae59360 2024-08-03T20:58:04.112 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:04.110+0000 7f924488cf40 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x55f73ae57150 con 0x55f73ae76cb0 2024-08-03T20:58:04.112 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:04.110+0000 7f923e400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55f73ae773f0 0x55f73ade7e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:58:04.113 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:04.110+0000 7f923e400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55f73ae773f0 0x55f73ade7e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:51108/0 (socket says 172.21.15.32:51108) 2024-08-03T20:58:04.113 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:04.110+0000 7f923e400640 1 -- 172.21.15.32:0/1990021324 learned_addr learned my addr 172.21.15.32:0/1990021324 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:58:04.113 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:04.110+0000 7f923d000640 1 --2- 172.21.15.32:0/1990021324 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55f73ae76cb0 0x55f73ade78f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:58:04.113 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:04.110+0000 7f923da00640 1 --2- 172.21.15.32:0/1990021324 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55f73ae59360 0x55f73ade23a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:58:04.113 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:04.111+0000 7f923e400640 1 -- 172.21.15.32:0/1990021324 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55f73ae76cb0 msgr2=0x55f73ade78f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:58:04.113 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:04.111+0000 7f923e400640 1 --2- 172.21.15.32:0/1990021324 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55f73ae76cb0 0x55f73ade78f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:04.113 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:04.111+0000 7f923e400640 1 -- 172.21.15.32:0/1990021324 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55f73ae59360 msgr2=0x55f73ade23a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:58:04.113 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:04.111+0000 7f923e400640 1 --2- 172.21.15.32:0/1990021324 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55f73ae59360 0x55f73ade23a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:04.113 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:04.111+0000 7f923e400640 1 -- 172.21.15.32:0/1990021324 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f73ade1300 con 0x55f73ae773f0 2024-08-03T20:58:04.113 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:04.111+0000 7f923e400640 1 --2- 172.21.15.32:0/1990021324 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55f73ae773f0 0x55f73ade7e30 secure :-1 s=READY pgs=1703 cs=0 l=1 rev1=1 crypto rx=0x7f9238002000 tx=0x7f923800b580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:58:04.113 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:04.111+0000 7f9236000640 1 -- 172.21.15.32:0/1990021324 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9238004220 con 0x55f73ae773f0 2024-08-03T20:58:04.113 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:04.111+0000 7f9236000640 1 -- 172.21.15.32:0/1990021324 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f92380107e0 con 0x55f73ae773f0 2024-08-03T20:58:04.113 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:04.111+0000 7f924488cf40 1 -- 172.21.15.32:0/1990021324 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x55f73ade15e0 con 0x55f73ae773f0 2024-08-03T20:58:04.113 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:04.111+0000 7f9236000640 1 -- 172.21.15.32:0/1990021324 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9238010c30 con 0x55f73ae773f0 2024-08-03T20:58:04.114 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:04.112+0000 7f924488cf40 1 -- 172.21.15.32:0/1990021324 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f73ade1ba0 con 0x55f73ae773f0 2024-08-03T20:58:04.114 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:04.114+0000 7f9236000640 1 -- 172.21.15.32:0/1990021324 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f9238029060 con 0x55f73ae773f0 2024-08-03T20:58:04.115 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:04.114+0000 7f9236000640 1 --2- 172.21.15.32:0/1990021324 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f9220075a20 0x7f9220077f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:58:04.115 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:04.115+0000 7f9236000640 1 -- 172.21.15.32:0/1990021324 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(111..111 src has 1..111) ==== 8090+0+0 (secure 0 0 0) 0x7f9238099c10 con 0x55f73ae773f0 2024-08-03T20:58:04.115 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:04.115+0000 7f924488cf40 1 --2- 172.21.15.32:0/1990021324 >> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] conn(0x55f73ade28e0 0x55f73ade61a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:58:04.116 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:04.115+0000 7f924488cf40 1 -- 172.21.15.32:0/1990021324 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:1 12.15 12:aa448500:::500.00000000:head [truncate 0] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111) -- 0x55f73ae6f060 con 0x55f73ade28e0 2024-08-03T20:58:04.116 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:04.115+0000 7f923da00640 1 --2- 172.21.15.32:0/1990021324 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f9220075a20 0x7f9220077f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:58:04.116 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:04.115+0000 7f923d000640 1 --2- 172.21.15.32:0/1990021324 >> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] conn(0x55f73ade28e0 0x55f73ade61a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:58:04.116 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:04.116+0000 7f923da00640 1 --2- 172.21.15.32:0/1990021324 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f9220075a20 0x7f9220077f30 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7f9230055000 tx=0x7f9230003020 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:58:04.117 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:04.116+0000 7f923d000640 1 --2- 172.21.15.32:0/1990021324 >> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] conn(0x55f73ade28e0 0x55f73ade61a0 crc :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:58:04.118 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:04.118+0000 7f923d000640 1 -- 172.21.15.32:0/1990021324 <== osd.6 v2:172.21.15.118:6825/801947386 1 ==== osd_op_reply(1 500.00000000 [truncate 0] v111'293 uv293 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f922c010600 con 0x55f73ade28e0 2024-08-03T20:58:04.119 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:04.118+0000 7f924488cf40 1 -- 172.21.15.32:0/1990021324 >> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] conn(0x55f73ade28e0 msgr2=0x55f73ade61a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:58:04.119 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:04.118+0000 7f924488cf40 1 --2- 172.21.15.32:0/1990021324 >> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] conn(0x55f73ade28e0 0x55f73ade61a0 crc :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:04.119 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:04.118+0000 7f924488cf40 1 -- 172.21.15.32:0/1990021324 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f9220075a20 msgr2=0x7f9220077f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:58:04.119 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:04.118+0000 7f924488cf40 1 --2- 172.21.15.32:0/1990021324 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f9220075a20 0x7f9220077f30 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7f9230055000 tx=0x7f9230003020 comp rx=0 tx=0).stop 2024-08-03T20:58:04.119 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:04.118+0000 7f924488cf40 1 -- 172.21.15.32:0/1990021324 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55f73ae773f0 msgr2=0x55f73ade7e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:58:04.119 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:04.118+0000 7f924488cf40 1 --2- 172.21.15.32:0/1990021324 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55f73ae773f0 0x55f73ade7e30 secure :-1 s=READY pgs=1703 cs=0 l=1 rev1=1 crypto rx=0x7f9238002000 tx=0x7f923800b580 comp rx=0 tx=0).stop 2024-08-03T20:58:04.119 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:04.118+0000 7f923e400640 1 -- 172.21.15.32:0/1990021324 reap_dead start 2024-08-03T20:58:04.119 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:04.118+0000 7f924488cf40 1 -- 172.21.15.32:0/1990021324 shutdown_connections 2024-08-03T20:58:04.119 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:04.118+0000 7f924488cf40 1 -- 172.21.15.32:0/1990021324 >> 172.21.15.32:0/1990021324 conn(0x55f73acc2ac0 msgr2=0x55f73ade9020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:58:04.119 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:04.118+0000 7f924488cf40 1 -- 172.21.15.32:0/1990021324 shutdown_connections 2024-08-03T20:58:04.119 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:04.118+0000 7f924488cf40 1 -- 172.21.15.32:0/1990021324 wait complete. 2024-08-03T20:58:04.133 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs joinable true 2024-08-03T20:58:04.365 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:04.365+0000 7fab39800640 1 -- 172.21.15.32:0/3531501582 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fab340ced30 msgr2=0x7fab340cf140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:58:04.366 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:04.365+0000 7fab39800640 1 --2- 172.21.15.32:0/3531501582 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fab340ced30 0x7fab340cf140 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fab2c005fd0 tx=0x7fab2c02aa90 comp rx=0 tx=0).stop 2024-08-03T20:58:04.366 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:04.365+0000 7fab39800640 1 -- 172.21.15.32:0/3531501582 shutdown_connections 2024-08-03T20:58:04.366 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:04.365+0000 7fab39800640 1 --2- 172.21.15.32:0/3531501582 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fab340ced30 0x7fab340cf140 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:04.366 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:04.365+0000 7fab39800640 1 --2- 172.21.15.32:0/3531501582 >> v2:172.21.15.118:3300/0 conn(0x7fab34145c50 0x7fab3406e950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:04.366 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:04.365+0000 7fab39800640 1 -- 172.21.15.32:0/3531501582 >> 172.21.15.32:0/3531501582 conn(0x7fab340b1bd0 msgr2=0x7fab340b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:58:04.366 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:04.365+0000 7fab39800640 1 -- 172.21.15.32:0/3531501582 shutdown_connections 2024-08-03T20:58:04.366 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:04.365+0000 7fab39800640 1 -- 172.21.15.32:0/3531501582 wait complete. 2024-08-03T20:58:04.367 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:04.367+0000 7fab39800640 1 Processor -- start 2024-08-03T20:58:04.368 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:04.367+0000 7fab39800640 1 -- start start 2024-08-03T20:58:04.370 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:04.368+0000 7fab39800640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fab340cccf0 0x7fab3413c7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:58:04.371 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:04.369+0000 7fab39800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fab340ced30 0x7fab341594f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:58:04.371 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:04.369+0000 7fab39800640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fab34145c50 0x7fab34073820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:58:04.371 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:04.369+0000 7fab39800640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fab340b4070 con 0x7fab340ced30 2024-08-03T20:58:04.371 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:04.369+0000 7fab39800640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7fab340b3ef0 con 0x7fab340cccf0 2024-08-03T20:58:04.371 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:04.369+0000 7fab39800640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7fab340b41f0 con 0x7fab34145c50 2024-08-03T20:58:04.371 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:04.369+0000 7fab32000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fab340ced30 0x7fab341594f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:58:04.371 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:04.369+0000 7fab32000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fab340ced30 0x7fab341594f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:51112/0 (socket says 172.21.15.32:51112) 2024-08-03T20:58:04.371 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:04.369+0000 7fab32000640 1 -- 172.21.15.32:0/1453756807 learned_addr learned my addr 172.21.15.32:0/1453756807 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:58:04.371 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:04.369+0000 7fab32a00640 1 --2- 172.21.15.32:0/1453756807 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fab340cccf0 0x7fab3413c7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:58:04.371 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:04.369+0000 7fab33400640 1 --2- 172.21.15.32:0/1453756807 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fab34145c50 0x7fab34073820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:58:04.371 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:04.369+0000 7fab32000640 1 -- 172.21.15.32:0/1453756807 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fab34145c50 msgr2=0x7fab34073820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:58:04.372 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:04.369+0000 7fab32000640 1 --2- 172.21.15.32:0/1453756807 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fab34145c50 0x7fab34073820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:04.372 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:04.369+0000 7fab32000640 1 -- 172.21.15.32:0/1453756807 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fab340cccf0 msgr2=0x7fab3413c7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:58:04.372 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:04.369+0000 7fab32000640 1 --2- 172.21.15.32:0/1453756807 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fab340cccf0 0x7fab3413c7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:04.372 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:04.369+0000 7fab32000640 1 -- 172.21.15.32:0/1453756807 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fab341f5c60 con 0x7fab340ced30 2024-08-03T20:58:04.372 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:04.370+0000 7fab32000640 1 --2- 172.21.15.32:0/1453756807 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fab340ced30 0x7fab341594f0 secure :-1 s=READY pgs=1704 cs=0 l=1 rev1=1 crypto rx=0x7fab2000c370 tx=0x7fab2000c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:58:04.372 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:04.370+0000 7fab2aa00640 1 -- 172.21.15.32:0/1453756807 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab20017070 con 0x7fab340ced30 2024-08-03T20:58:04.372 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:04.370+0000 7fab2aa00640 1 -- 172.21.15.32:0/1453756807 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fab200053c0 con 0x7fab340ced30 2024-08-03T20:58:04.372 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:04.370+0000 7fab2aa00640 1 -- 172.21.15.32:0/1453756807 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab20003850 con 0x7fab340ced30 2024-08-03T20:58:04.372 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:04.370+0000 7fab39800640 1 -- 172.21.15.32:0/1453756807 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fab341f5f40 con 0x7fab340ced30 2024-08-03T20:58:04.372 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:04.370+0000 7fab39800640 1 -- 172.21.15.32:0/1453756807 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fab341f6500 con 0x7fab340ced30 2024-08-03T20:58:04.373 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:04.371+0000 7fab39800640 1 -- 172.21.15.32:0/1453756807 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fab340b46a0 con 0x7fab340ced30 2024-08-03T20:58:04.373 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:04.372+0000 7fab2aa00640 1 -- 172.21.15.32:0/1453756807 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7fab2001a070 con 0x7fab340ced30 2024-08-03T20:58:04.374 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:04.373+0000 7fab2aa00640 1 --2- 172.21.15.32:0/1453756807 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fab14077040 0x7fab14079550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:58:04.374 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:04.373+0000 7fab2aa00640 1 -- 172.21.15.32:0/1453756807 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(111..111 src has 1..111) ==== 8090+0+0 (secure 0 0 0) 0x7fab20098610 con 0x7fab340ced30 2024-08-03T20:58:04.374 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:04.374+0000 7fab32a00640 1 --2- 172.21.15.32:0/1453756807 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fab14077040 0x7fab14079550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:58:04.375 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:04.375+0000 7fab32a00640 1 --2- 172.21.15.32:0/1453756807 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fab14077040 0x7fab14079550 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7fab24009b30 tx=0x7fab24005f90 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:58:04.378 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:04.378+0000 7fab2aa00640 1 -- 172.21.15.32:0/1453756807 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7fab200607d0 con 0x7fab340ced30 2024-08-03T20:58:04.630 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:04.629+0000 7fab39800640 1 -- 172.21.15.32:0/1453756807 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "joinable", "val": "true"} v 0) -- 0x7fab341f59f0 con 0x7fab340ced30 2024-08-03T20:58:04.820 INFO:tasks.ceph.mds.a.smithi032.stderr: -24> 2024-08-03T20:58:00.067+0000 7fce10a00640 -1 log_channel(cluster) log [ERR] : Error loading MDS rank 0: (22) Invalid argument 2024-08-03T20:58:04.935 INFO:tasks.ceph.mds.a.smithi032.stdout:starting mds.a at 2024-08-03T20:58:05.052 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:05.051+0000 7fab2aa00640 1 -- 172.21.15.32:0/1453756807 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "joinable", "val": "true"}]=0 cephfs marked joinable; MDS may join as newly active. v241) ==== 162+0+0 (secure 0 0 0) 0x7fab200690f0 con 0x7fab340ced30 2024-08-03T20:58:05.052 INFO:teuthology.orchestra.run.smithi032.stderr:cephfs marked joinable; MDS may join as newly active. 2024-08-03T20:58:05.055 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:05.054+0000 7fab39800640 1 -- 172.21.15.32:0/1453756807 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fab14077040 msgr2=0x7fab14079550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:58:05.055 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:05.054+0000 7fab39800640 1 --2- 172.21.15.32:0/1453756807 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fab14077040 0x7fab14079550 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7fab24009b30 tx=0x7fab24005f90 comp rx=0 tx=0).stop 2024-08-03T20:58:05.055 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:05.054+0000 7fab39800640 1 -- 172.21.15.32:0/1453756807 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fab340ced30 msgr2=0x7fab341594f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:58:05.055 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:05.054+0000 7fab39800640 1 --2- 172.21.15.32:0/1453756807 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fab340ced30 0x7fab341594f0 secure :-1 s=READY pgs=1704 cs=0 l=1 rev1=1 crypto rx=0x7fab2000c370 tx=0x7fab2000c830 comp rx=0 tx=0).stop 2024-08-03T20:58:05.055 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:05.054+0000 7fab39800640 1 -- 172.21.15.32:0/1453756807 shutdown_connections 2024-08-03T20:58:05.055 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:05.054+0000 7fab39800640 1 --2- 172.21.15.32:0/1453756807 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fab14077040 0x7fab14079550 unknown :-1 s=CLOSED pgs=695 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:05.055 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:05.054+0000 7fab39800640 1 --2- 172.21.15.32:0/1453756807 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fab34145c50 0x7fab34073820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:05.055 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:05.054+0000 7fab39800640 1 --2- 172.21.15.32:0/1453756807 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fab340ced30 0x7fab341594f0 unknown :-1 s=CLOSED pgs=1704 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:05.055 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:05.054+0000 7fab39800640 1 --2- 172.21.15.32:0/1453756807 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fab340cccf0 0x7fab3413c7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:05.055 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:05.054+0000 7fab39800640 1 -- 172.21.15.32:0/1453756807 >> 172.21.15.32:0/1453756807 conn(0x7fab340b1bd0 msgr2=0x7fab340cff60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:58:05.056 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:05.054+0000 7fab39800640 1 -- 172.21.15.32:0/1453756807 shutdown_connections 2024-08-03T20:58:05.056 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:05.054+0000 7fab39800640 1 -- 172.21.15.32:0/1453756807 wait complete. 2024-08-03T20:58:05.484 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-08-03T20:58:05.717 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:05.716+0000 7f4007600640 1 -- 172.21.15.32:0/1858778135 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3fe4004800 msgr2=0x7f3fe4004c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:58:05.717 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:05.716+0000 7f4007600640 1 --2- 172.21.15.32:0/1858778135 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3fe4004800 0x7f3fe4004c10 secure :-1 s=READY pgs=1708 cs=0 l=1 rev1=1 crypto rx=0x7f4000072880 tx=0x7f3fe4005830 comp rx=0 tx=0).stop 2024-08-03T20:58:05.717 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:05.716+0000 7f4007600640 1 -- 172.21.15.32:0/1858778135 shutdown_connections 2024-08-03T20:58:05.717 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:05.716+0000 7f4007600640 1 --2- 172.21.15.32:0/1858778135 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3fe4004800 0x7f3fe4004c10 unknown :-1 s=CLOSED pgs=1708 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:05.718 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:05.716+0000 7f4007600640 1 -- 172.21.15.32:0/1858778135 >> 172.21.15.32:0/1858778135 conn(0x7f40000b1bd0 msgr2=0x7f40000b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:58:05.718 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:05.716+0000 7f4007600640 1 -- 172.21.15.32:0/1858778135 shutdown_connections 2024-08-03T20:58:05.718 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:05.717+0000 7f4007600640 1 -- 172.21.15.32:0/1858778135 wait complete. 2024-08-03T20:58:05.719 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:05.719+0000 7f4007600640 1 Processor -- start 2024-08-03T20:58:05.719 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:05.719+0000 7f4007600640 1 -- start start 2024-08-03T20:58:05.723 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:05.720+0000 7f4007600640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f3fe4004800 0x7f400015eb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:58:05.723 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:05.720+0000 7f4007600640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f40000b1210 0x7f4000140670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:58:05.723 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:05.720+0000 7f4007600640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f40001f86b0 0x7f40001f8ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:58:05.723 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:05.720+0000 7f4007600640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f4000145110 con 0x7f40001f86b0 2024-08-03T20:58:05.723 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:05.720+0000 7f4007600640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f4000144f90 con 0x7f3fe4004800 2024-08-03T20:58:05.723 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:05.720+0000 7f4007600640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7f4000145290 con 0x7f40000b1210 2024-08-03T20:58:05.723 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:05.721+0000 7f4005600640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f40001f86b0 0x7f40001f8ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:58:05.723 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:05.721+0000 7f4005600640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f40001f86b0 0x7f40001f8ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:51134/0 (socket says 172.21.15.32:51134) 2024-08-03T20:58:05.723 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:05.721+0000 7f4005600640 1 -- 172.21.15.32:0/160089173 learned_addr learned my addr 172.21.15.32:0/160089173 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:58:05.723 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:05.721+0000 7f4004c00640 1 --2- 172.21.15.32:0/160089173 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f3fe4004800 0x7f400015eb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:58:05.723 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:05.721+0000 7f3ff7e00640 1 --2- 172.21.15.32:0/160089173 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f40000b1210 0x7f4000140670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:58:05.723 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:05.721+0000 7f4005600640 1 -- 172.21.15.32:0/160089173 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f40000b1210 msgr2=0x7f4000140670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:58:05.724 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:05.721+0000 7f4005600640 1 --2- 172.21.15.32:0/160089173 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f40000b1210 0x7f4000140670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:05.724 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:05.721+0000 7f4005600640 1 -- 172.21.15.32:0/160089173 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f3fe4004800 msgr2=0x7f400015eb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:58:05.724 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:05.721+0000 7f4005600640 1 --2- 172.21.15.32:0/160089173 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f3fe4004800 0x7f400015eb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:05.724 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:05.721+0000 7f4005600640 1 -- 172.21.15.32:0/160089173 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f40001f9000 con 0x7f40001f86b0 2024-08-03T20:58:05.724 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:05.721+0000 7f4005600640 1 --2- 172.21.15.32:0/160089173 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f40001f86b0 0x7f40001f8ac0 secure :-1 s=READY pgs=1709 cs=0 l=1 rev1=1 crypto rx=0x7f4000072db0 tx=0x7f3ff80019b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:58:05.724 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:05.722+0000 7f3ffd600640 1 -- 172.21.15.32:0/160089173 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ff8011870 con 0x7f40001f86b0 2024-08-03T20:58:05.724 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:05.722+0000 7f3ffd600640 1 -- 172.21.15.32:0/160089173 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f3ff8017070 con 0x7f40001f86b0 2024-08-03T20:58:05.724 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:05.722+0000 7f4007600640 1 -- 172.21.15.32:0/160089173 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f40000b17b0 con 0x7f40001f86b0 2024-08-03T20:58:05.724 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:05.722+0000 7f3ffd600640 1 -- 172.21.15.32:0/160089173 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ff8011da0 con 0x7f40001f86b0 2024-08-03T20:58:05.724 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:05.722+0000 7f4007600640 1 -- 172.21.15.32:0/160089173 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f400015cd50 con 0x7f40001f86b0 2024-08-03T20:58:05.725 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:05.723+0000 7f4007600640 1 -- 172.21.15.32:0/160089173 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f40000b2510 con 0x7f40001f86b0 2024-08-03T20:58:05.725 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:05.724+0000 7f3ffd600640 1 -- 172.21.15.32:0/160089173 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f3ff801c460 con 0x7f40001f86b0 2024-08-03T20:58:05.726 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:05.725+0000 7f3ffd600640 1 --2- 172.21.15.32:0/160089173 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f3fdc077040 0x7f3fdc079550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:58:05.726 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:05.726+0000 7f3ffd600640 1 -- 172.21.15.32:0/160089173 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(111..111 src has 1..111) ==== 8090+0+0 (secure 0 0 0) 0x7f3ff80998f0 con 0x7f40001f86b0 2024-08-03T20:58:05.726 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:05.726+0000 7f4004c00640 1 --2- 172.21.15.32:0/160089173 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f3fdc077040 0x7f3fdc079550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:58:05.727 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:05.727+0000 7f4004c00640 1 --2- 172.21.15.32:0/160089173 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f3fdc077040 0x7f3fdc079550 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7f3ff0009cd0 tx=0x7f3ff0005f90 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:58:05.731 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:05.730+0000 7f3ffd600640 1 -- 172.21.15.32:0/160089173 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7f40000b2510 con 0x7f40001f86b0 2024-08-03T20:58:05.972 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:05.972+0000 7f4007600640 1 -- 172.21.15.32:0/160089173 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f4000074350 con 0x7f40001f86b0 2024-08-03T20:58:05.973 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:05.972+0000 7f3ffd600640 1 -- 172.21.15.32:0/160089173 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 241 v241) ==== 95+0+5226 (secure 0 0 0) 0x7f40001f9000 con 0x7f40001f86b0 2024-08-03T20:58:05.973 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-03T20:58:05.973 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":241,"btime":"2024-08-03T20:58:05:050037+0000","default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6439,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6835/531080160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6834","nonce":531080160},{"type":"v1","addr":"172.21.15.118:6835","nonce":531080160}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":84},{"gid":6454,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6837/2796882319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6836","nonce":2796882319},{"type":"v1","addr":"172.21.15.118:6837","nonce":2796882319}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":86},{"gid":6813,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6834/2686933890","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":2686933890},{"type":"v1","addr":"172.21.15.32:6834","nonce":2686933890}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":85}],"filesystems":[{"mdsmap":{"epoch":241,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-08-03T20:54:26.950518+0000","modified":"2024-08-03T20:58:05.049993+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":111,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":9237},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9237":{"gid":9237,"name":"a","rank":0,"incarnation":241,"state":"up:replay","state_seq":1,"addr":"172.21.15.32:6835/3340816283","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6833","nonce":3340816283},{"type":"v1","addr":"172.21.15.32:6835","nonce":3340816283}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":6}]} 2024-08-03T20:58:05.973 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 241 2024-08-03T20:58:05.975 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:05.974+0000 7f4007600640 1 -- 172.21.15.32:0/160089173 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f3fdc077040 msgr2=0x7f3fdc079550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:58:05.975 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:05.974+0000 7f4007600640 1 --2- 172.21.15.32:0/160089173 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f3fdc077040 0x7f3fdc079550 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7f3ff0009cd0 tx=0x7f3ff0005f90 comp rx=0 tx=0).stop 2024-08-03T20:58:05.975 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:05.974+0000 7f4007600640 1 -- 172.21.15.32:0/160089173 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f40001f86b0 msgr2=0x7f40001f8ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:58:05.975 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:05.974+0000 7f4007600640 1 --2- 172.21.15.32:0/160089173 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f40001f86b0 0x7f40001f8ac0 secure :-1 s=READY pgs=1709 cs=0 l=1 rev1=1 crypto rx=0x7f4000072db0 tx=0x7f3ff80019b0 comp rx=0 tx=0).stop 2024-08-03T20:58:05.976 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:05.975+0000 7f4007600640 1 -- 172.21.15.32:0/160089173 shutdown_connections 2024-08-03T20:58:05.976 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:05.975+0000 7f4007600640 1 --2- 172.21.15.32:0/160089173 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f3fdc077040 0x7f3fdc079550 unknown :-1 s=CLOSED pgs=697 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:05.976 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:05.975+0000 7f4007600640 1 --2- 172.21.15.32:0/160089173 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f40001f86b0 0x7f40001f8ac0 unknown :-1 s=CLOSED pgs=1709 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:05.976 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:05.975+0000 7f4007600640 1 --2- 172.21.15.32:0/160089173 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f40000b1210 0x7f4000140670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:05.976 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:05.975+0000 7f4007600640 1 --2- 172.21.15.32:0/160089173 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f3fe4004800 0x7f400015eb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:05.976 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:05.975+0000 7f4007600640 1 -- 172.21.15.32:0/160089173 >> 172.21.15.32:0/160089173 conn(0x7f40000b1bd0 msgr2=0x7f40000cb9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:58:05.976 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:05.975+0000 7f4007600640 1 -- 172.21.15.32:0/160089173 shutdown_connections 2024-08-03T20:58:05.976 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:05.975+0000 7f4007600640 1 -- 172.21.15.32:0/160089173 wait complete. 2024-08-03T20:58:06.001 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 241, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-08-03T20:54:26.950518+0000', 'modified': '2024-08-03T20:58:05.049993+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 111, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 9237}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9237': {'gid': 9237, 'name': 'a', 'rank': 0, 'incarnation': 241, 'state': 'up:replay', 'state_seq': 1, 'addr': '172.21.15.32:6835/3340816283', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6833', 'nonce': 3340816283}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 3340816283}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [13], 'metadata_pool': 12, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 0, 'qdb_cluster': []} 2024-08-03T20:58:06.001 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9237:up:replay 2024-08-03T20:58:06.001 DEBUG:tasks.ceph_test_case:wait_until_true: waiting (timeout=60 retry_count=0)... 2024-08-03T20:58:08.626 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-08-03T20:58:08.631 DEBUG:teuthology.orchestra.run.smithi118:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-08-03T20:58:10.061 INFO:tasks.ceph.mds.a.smithi032.stderr:2024-08-03T20:58:10.060+0000 7f2661800640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-08-03T20:58:11.406 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-08-03T20:58:11.650 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:11.649+0000 7f8130800640 1 -- 172.21.15.32:0/1082248777 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f812815f610 msgr2=0x7f81280cdf70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:58:11.651 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:11.649+0000 7f8130800640 1 --2- 172.21.15.32:0/1082248777 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f812815f610 0x7f81280cdf70 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f8124005fd0 tx=0x7f812402a840 comp rx=0 tx=0).stop 2024-08-03T20:58:11.651 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:11.649+0000 7f8130800640 1 -- 172.21.15.32:0/1082248777 shutdown_connections 2024-08-03T20:58:11.651 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:11.649+0000 7f8130800640 1 --2- 172.21.15.32:0/1082248777 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f812815f610 0x7f81280cdf70 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:11.651 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:11.649+0000 7f8130800640 1 --2- 172.21.15.32:0/1082248777 >> v2:172.21.15.118:3300/0 conn(0x7f81280b38b0 0x7f81281523f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:11.651 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:11.649+0000 7f8130800640 1 -- 172.21.15.32:0/1082248777 >> 172.21.15.32:0/1082248777 conn(0x7f81280b1bd0 msgr2=0x7f81280b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:58:11.651 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:11.649+0000 7f8130800640 1 -- 172.21.15.32:0/1082248777 shutdown_connections 2024-08-03T20:58:11.651 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:11.649+0000 7f8130800640 1 -- 172.21.15.32:0/1082248777 wait complete. 2024-08-03T20:58:11.652 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:11.651+0000 7f8130800640 1 Processor -- start 2024-08-03T20:58:11.652 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:11.651+0000 7f8130800640 1 -- start start 2024-08-03T20:58:11.654 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:11.653+0000 7f8130800640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f81280b38b0 0x7f8128153a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:58:11.654 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:11.653+0000 7f8130800640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f81280b4ce0 0x7f812806f3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:58:11.654 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:11.653+0000 7f8130800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f812806f8e0 0x7f81281485e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:58:11.654 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:11.653+0000 7f8130800640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f81280cea90 con 0x7f812806f8e0 2024-08-03T20:58:11.654 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:11.653+0000 7f8130800640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f81280ce910 con 0x7f81280b38b0 2024-08-03T20:58:11.654 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:11.653+0000 7f8130800640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7f81280cec10 con 0x7f81280b4ce0 2024-08-03T20:58:11.655 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:11.653+0000 7f812e800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f812806f8e0 0x7f81281485e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:58:11.655 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:11.653+0000 7f812e800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f812806f8e0 0x7f81281485e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:35748/0 (socket says 172.21.15.32:35748) 2024-08-03T20:58:11.655 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:11.653+0000 7f812e800640 1 -- 172.21.15.32:0/3257060691 learned_addr learned my addr 172.21.15.32:0/3257060691 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:58:11.655 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:11.653+0000 7f812de00640 1 --2- 172.21.15.32:0/3257060691 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f81280b38b0 0x7f8128153a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:58:11.655 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:11.653+0000 7f812d400640 1 --2- 172.21.15.32:0/3257060691 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f81280b4ce0 0x7f812806f3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:58:11.655 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:11.654+0000 7f812e800640 1 -- 172.21.15.32:0/3257060691 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f81280b4ce0 msgr2=0x7f812806f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:58:11.655 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:11.654+0000 7f812e800640 1 --2- 172.21.15.32:0/3257060691 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f81280b4ce0 0x7f812806f3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:11.655 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:11.654+0000 7f812e800640 1 -- 172.21.15.32:0/3257060691 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f81280b38b0 msgr2=0x7f8128153a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:58:11.655 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:11.654+0000 7f812e800640 1 --2- 172.21.15.32:0/3257060691 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f81280b38b0 0x7f8128153a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:11.655 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:11.654+0000 7f812e800640 1 -- 172.21.15.32:0/3257060691 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8128155970 con 0x7f812806f8e0 2024-08-03T20:58:11.656 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:11.654+0000 7f812e800640 1 --2- 172.21.15.32:0/3257060691 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f812806f8e0 0x7f81281485e0 secure :-1 s=READY pgs=1711 cs=0 l=1 rev1=1 crypto rx=0x7f8124005fd0 tx=0x7f8124032c70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:58:11.656 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:11.654+0000 7f8122000640 1 -- 172.21.15.32:0/3257060691 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8124040070 con 0x7f812806f8e0 2024-08-03T20:58:11.656 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:11.654+0000 7f8122000640 1 -- 172.21.15.32:0/3257060691 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f8124004540 con 0x7f812806f8e0 2024-08-03T20:58:11.656 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:11.654+0000 7f8130800640 1 -- 172.21.15.32:0/3257060691 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8128154120 con 0x7f812806f8e0 2024-08-03T20:58:11.657 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:11.654+0000 7f8122000640 1 -- 172.21.15.32:0/3257060691 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8124003e60 con 0x7f812806f8e0 2024-08-03T20:58:11.657 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:11.654+0000 7f8130800640 1 -- 172.21.15.32:0/3257060691 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f81281546e0 con 0x7f812806f8e0 2024-08-03T20:58:11.657 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:11.656+0000 7f8130800640 1 -- 172.21.15.32:0/3257060691 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f81280cf190 con 0x7f812806f8e0 2024-08-03T20:58:11.657 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:11.657+0000 7f8122000640 1 -- 172.21.15.32:0/3257060691 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f8124043070 con 0x7f812806f8e0 2024-08-03T20:58:11.659 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:11.658+0000 7f8122000640 1 --2- 172.21.15.32:0/3257060691 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f810c077040 0x7f810c079550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:58:11.659 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:11.659+0000 7f8122000640 1 -- 172.21.15.32:0/3257060691 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(111..111 src has 1..111) ==== 8090+0+0 (secure 0 0 0) 0x7f81240ba430 con 0x7f812806f8e0 2024-08-03T20:58:11.659 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:11.659+0000 7f812de00640 1 --2- 172.21.15.32:0/3257060691 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f810c077040 0x7f810c079550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:58:11.660 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:11.660+0000 7f812de00640 1 --2- 172.21.15.32:0/3257060691 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f810c077040 0x7f810c079550 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 crypto rx=0x7f811c009b30 tx=0x7f811c005f90 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:58:11.663 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:11.663+0000 7f8122000640 1 -- 172.21.15.32:0/3257060691 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7f8124082690 con 0x7f812806f8e0 2024-08-03T20:58:11.908 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:11.907+0000 7f8130800640 1 -- 172.21.15.32:0/3257060691 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f812806d9f0 con 0x7f812806f8e0 2024-08-03T20:58:11.908 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:11.908+0000 7f8122000640 1 -- 172.21.15.32:0/3257060691 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 244 v244) ==== 95+0+5233 (secure 0 0 0) 0x7f8124086be0 con 0x7f812806f8e0 2024-08-03T20:58:11.909 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-03T20:58:11.909 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":244,"btime":"2024-08-03T20:58:08:061434+0000","default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6439,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6835/531080160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6834","nonce":531080160},{"type":"v1","addr":"172.21.15.118:6835","nonce":531080160}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":84},{"gid":6454,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6837/2796882319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6836","nonce":2796882319},{"type":"v1","addr":"172.21.15.118:6837","nonce":2796882319}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":86},{"gid":6813,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6834/2686933890","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":2686933890},{"type":"v1","addr":"172.21.15.32:6834","nonce":2686933890}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":85}],"filesystems":[{"mdsmap":{"epoch":244,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-08-03T20:54:26.950518+0000","modified":"2024-08-03T20:58:08.061429+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":111,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":9237},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9237":{"gid":9237,"name":"a","rank":0,"incarnation":241,"state":"up:active","state_seq":4,"addr":"172.21.15.32:6835/3340816283","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6833","nonce":3340816283},{"type":"v1","addr":"172.21.15.32:6835","nonce":3340816283}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9237,"qdb_cluster":[9237]},"id":6}]} 2024-08-03T20:58:11.909 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 244 2024-08-03T20:58:11.911 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:11.910+0000 7f8130800640 1 -- 172.21.15.32:0/3257060691 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f810c077040 msgr2=0x7f810c079550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:58:11.911 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:11.910+0000 7f8130800640 1 --2- 172.21.15.32:0/3257060691 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f810c077040 0x7f810c079550 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 crypto rx=0x7f811c009b30 tx=0x7f811c005f90 comp rx=0 tx=0).stop 2024-08-03T20:58:11.911 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:11.910+0000 7f8130800640 1 -- 172.21.15.32:0/3257060691 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f812806f8e0 msgr2=0x7f81281485e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:58:11.911 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:11.910+0000 7f8130800640 1 --2- 172.21.15.32:0/3257060691 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f812806f8e0 0x7f81281485e0 secure :-1 s=READY pgs=1711 cs=0 l=1 rev1=1 crypto rx=0x7f8124005fd0 tx=0x7f8124032c70 comp rx=0 tx=0).stop 2024-08-03T20:58:11.911 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:11.910+0000 7f8130800640 1 -- 172.21.15.32:0/3257060691 shutdown_connections 2024-08-03T20:58:11.912 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:11.910+0000 7f8130800640 1 --2- 172.21.15.32:0/3257060691 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f810c077040 0x7f810c079550 unknown :-1 s=CLOSED pgs=698 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:11.912 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:11.910+0000 7f8130800640 1 --2- 172.21.15.32:0/3257060691 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f812806f8e0 0x7f81281485e0 unknown :-1 s=CLOSED pgs=1711 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:11.912 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:11.910+0000 7f8130800640 1 --2- 172.21.15.32:0/3257060691 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f81280b4ce0 0x7f812806f3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:11.912 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:11.910+0000 7f8130800640 1 --2- 172.21.15.32:0/3257060691 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f81280b38b0 0x7f8128153a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:11.912 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:11.911+0000 7f8130800640 1 -- 172.21.15.32:0/3257060691 >> 172.21.15.32:0/3257060691 conn(0x7f81280b1bd0 msgr2=0x7f81280afb40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:58:11.912 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:11.911+0000 7f8130800640 1 -- 172.21.15.32:0/3257060691 shutdown_connections 2024-08-03T20:58:11.912 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:11.911+0000 7f8130800640 1 -- 172.21.15.32:0/3257060691 wait complete. 2024-08-03T20:58:11.938 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 244, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-08-03T20:54:26.950518+0000', 'modified': '2024-08-03T20:58:08.061429+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 111, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 9237}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9237': {'gid': 9237, 'name': 'a', 'rank': 0, 'incarnation': 241, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.32:6835/3340816283', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6833', 'nonce': 3340816283}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 3340816283}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [13], 'metadata_pool': 12, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9237, 'qdb_cluster': [9237]} 2024-08-03T20:58:11.938 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-08-03T20:58:11.938 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2024-08-03T20:58:12.169 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:12.168+0000 7f166d400640 1 -- 172.21.15.32:0/1118321439 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f166806e930 msgr2=0x7f1668144f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:58:12.169 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:12.168+0000 7f166d400640 1 --2- 172.21.15.32:0/1118321439 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f166806e930 0x7f1668144f90 secure :-1 s=READY pgs=1713 cs=0 l=1 rev1=1 crypto rx=0x7f1668072d80 tx=0x7f165802a530 comp rx=0 tx=0).stop 2024-08-03T20:58:12.169 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:12.168+0000 7f166d400640 1 -- 172.21.15.32:0/1118321439 shutdown_connections 2024-08-03T20:58:12.169 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:12.168+0000 7f166d400640 1 --2- 172.21.15.32:0/1118321439 >> v2:172.21.15.118:3300/0 conn(0x7f166815fff0 0x7f1668160400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:12.169 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:12.168+0000 7f166d400640 1 --2- 172.21.15.32:0/1118321439 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f166806e930 0x7f1668144f90 unknown :-1 s=CLOSED pgs=1713 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:12.169 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:12.168+0000 7f166d400640 1 -- 172.21.15.32:0/1118321439 >> 172.21.15.32:0/1118321439 conn(0x7f16680b1bd0 msgr2=0x7f16680b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:58:12.170 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:12.168+0000 7f166d400640 1 -- 172.21.15.32:0/1118321439 shutdown_connections 2024-08-03T20:58:12.170 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:12.168+0000 7f166d400640 1 -- 172.21.15.32:0/1118321439 wait complete. 2024-08-03T20:58:12.171 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:12.171+0000 7f166d400640 1 Processor -- start 2024-08-03T20:58:12.171 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:12.171+0000 7f166d400640 1 -- start start 2024-08-03T20:58:12.174 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:12.172+0000 7f166d400640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f166806e930 0x7f166813fee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:58:12.174 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:12.172+0000 7f166d400640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f1668073e60 0x7f166815b8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:58:12.174 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:12.172+0000 7f166d400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f166815fff0 0x7f166815e610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:58:12.174 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:12.172+0000 7f166d400640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f16680cf840 con 0x7f166815fff0 2024-08-03T20:58:12.174 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:12.172+0000 7f166d400640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f1668145f30 con 0x7f1668073e60 2024-08-03T20:58:12.174 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:12.172+0000 7f166d400640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7f16680cf9c0 con 0x7f166806e930 2024-08-03T20:58:12.174 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:12.172+0000 7f1667200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f166815fff0 0x7f166815e610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:58:12.174 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:12.172+0000 7f1667200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f166815fff0 0x7f166815e610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:35770/0 (socket says 172.21.15.32:35770) 2024-08-03T20:58:12.175 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:12.172+0000 7f1667200640 1 -- 172.21.15.32:0/3553513544 learned_addr learned my addr 172.21.15.32:0/3553513544 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:58:12.175 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:12.173+0000 7f165fe00640 1 --2- 172.21.15.32:0/3553513544 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f1668073e60 0x7f166815b8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:58:12.175 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:12.173+0000 7f1666800640 1 --2- 172.21.15.32:0/3553513544 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f166806e930 0x7f166813fee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:58:12.175 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:12.173+0000 7f1667200640 1 -- 172.21.15.32:0/3553513544 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f166806e930 msgr2=0x7f166813fee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:58:12.175 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:12.173+0000 7f1667200640 1 --2- 172.21.15.32:0/3553513544 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f166806e930 0x7f166813fee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:12.175 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:12.173+0000 7f1667200640 1 -- 172.21.15.32:0/3553513544 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f1668073e60 msgr2=0x7f166815b8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:58:12.175 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:12.173+0000 7f1667200640 1 --2- 172.21.15.32:0/3553513544 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f1668073e60 0x7f166815b8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:12.175 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:12.173+0000 7f1667200640 1 -- 172.21.15.32:0/3553513544 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f166815d850 con 0x7f166815fff0 2024-08-03T20:58:12.175 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:12.173+0000 7f1667200640 1 --2- 172.21.15.32:0/3553513544 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f166815fff0 0x7f166815e610 secure :-1 s=READY pgs=1714 cs=0 l=1 rev1=1 crypto rx=0x7f166000da90 tx=0x7f166000df50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:58:12.175 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:12.173+0000 7f165ea00640 1 -- 172.21.15.32:0/3553513544 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1660017070 con 0x7f166815fff0 2024-08-03T20:58:12.175 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:12.174+0000 7f165ea00640 1 -- 172.21.15.32:0/3553513544 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f1660003970 con 0x7f166815fff0 2024-08-03T20:58:12.175 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:12.174+0000 7f165ea00640 1 -- 172.21.15.32:0/3553513544 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1660003dc0 con 0x7f166815fff0 2024-08-03T20:58:12.175 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:12.174+0000 7f166d400640 1 -- 172.21.15.32:0/3553513544 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f166815db30 con 0x7f166815fff0 2024-08-03T20:58:12.176 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:12.174+0000 7f166d400640 1 -- 172.21.15.32:0/3553513544 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f166815f650 con 0x7f166815fff0 2024-08-03T20:58:12.176 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:12.176+0000 7f165ea00640 1 -- 172.21.15.32:0/3553513544 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f1660003f60 con 0x7f166815fff0 2024-08-03T20:58:12.177 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:12.177+0000 7f165ea00640 1 --2- 172.21.15.32:0/3553513544 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f1644077010 0x7f1644079520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:58:12.177 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:12.177+0000 7f165ea00640 1 -- 172.21.15.32:0/3553513544 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(111..111 src has 1..111) ==== 8090+0+0 (secure 0 0 0) 0x7f1660099630 con 0x7f166815fff0 2024-08-03T20:58:12.178 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:12.177+0000 7f1666800640 1 --2- 172.21.15.32:0/3553513544 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f1644077010 0x7f1644079520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:58:12.178 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:12.178+0000 7f16743da740 1 Processor -- start 2024-08-03T20:58:12.178 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:12.178+0000 7f16743da740 1 -- start start 2024-08-03T20:58:12.178 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:12.178+0000 7f1666800640 1 --2- 172.21.15.32:0/3553513544 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f1644077010 0x7f1644079520 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7f1658034000 tx=0x7f1658004730 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:58:12.181 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:12.179+0000 7f16743da740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5643ad95f270 0x5643add4c0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:58:12.181 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:12.179+0000 7f16743da740 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x5643ad9363b0 0x5643add2eda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:58:12.181 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:12.179+0000 7f16743da740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5643ad93e590 0x5643add2f2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:58:12.181 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:12.179+0000 7f16743da740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x5643adc77a60 con 0x5643ad93e590 2024-08-03T20:58:12.181 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:12.179+0000 7f16743da740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x5643adc66f60 con 0x5643ad95f270 2024-08-03T20:58:12.181 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:12.179+0000 7f16743da740 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x5643adc6afa0 con 0x5643ad9363b0 2024-08-03T20:58:12.181 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:12.179+0000 7f1667200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5643ad93e590 0x5643add2f2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:58:12.181 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:12.180+0000 7f1667200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5643ad93e590 0x5643add2f2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:35784/0 (socket says 172.21.15.32:35784) 2024-08-03T20:58:12.181 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:12.180+0000 7f1667200640 1 -- 172.21.15.32:0/2991099599 learned_addr learned my addr 172.21.15.32:0/2991099599 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:58:12.181 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:12.180+0000 7f1666800640 1 --2- 172.21.15.32:0/2991099599 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5643ad95f270 0x5643add4c0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:58:12.182 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:12.180+0000 7f165fe00640 1 --2- 172.21.15.32:0/2991099599 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x5643ad9363b0 0x5643add2eda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:58:12.182 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:12.180+0000 7f1667200640 1 -- 172.21.15.32:0/2991099599 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x5643ad9363b0 msgr2=0x5643add2eda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:58:12.182 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:12.180+0000 7f1667200640 1 --2- 172.21.15.32:0/2991099599 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x5643ad9363b0 0x5643add2eda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:12.182 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:12.180+0000 7f1667200640 1 -- 172.21.15.32:0/2991099599 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5643ad95f270 msgr2=0x5643add4c0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:58:12.182 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:12.180+0000 7f1667200640 1 --2- 172.21.15.32:0/2991099599 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5643ad95f270 0x5643add4c0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:12.182 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:12.180+0000 7f1667200640 1 -- 172.21.15.32:0/2991099599 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5643ad9ca850 con 0x5643ad93e590 2024-08-03T20:58:12.182 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:12.180+0000 7f1667200640 1 --2- 172.21.15.32:0/2991099599 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5643ad93e590 0x5643add2f2e0 secure :-1 s=READY pgs=1715 cs=0 l=1 rev1=1 crypto rx=0x7f1660061780 tx=0x7f1660061300 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1fdb0a2b4b859567 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:58:12.182 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:12.180+0000 7f1657000640 1 -- 172.21.15.32:0/2991099599 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1660020040 con 0x5643ad93e590 2024-08-03T20:58:12.182 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:12.180+0000 7f1657000640 1 -- 172.21.15.32:0/2991099599 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f1660065d40 con 0x5643ad93e590 2024-08-03T20:58:12.182 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:12.180+0000 7f1657000640 1 -- 172.21.15.32:0/2991099599 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1660060cb0 con 0x5643ad93e590 2024-08-03T20:58:12.183 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:12.181+0000 7f16743da740 1 -- 172.21.15.32:0/2991099599 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5643ad93e590 msgr2=0x5643add2f2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:58:12.183 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:12.181+0000 7f16743da740 1 --2- 172.21.15.32:0/2991099599 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5643ad93e590 0x5643add2f2e0 secure :-1 s=READY pgs=1715 cs=0 l=1 rev1=1 crypto rx=0x7f1660061780 tx=0x7f1660061300 comp rx=0 tx=0).stop 2024-08-03T20:58:12.183 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:12.181+0000 7f16743da740 1 -- 172.21.15.32:0/2991099599 shutdown_connections 2024-08-03T20:58:12.183 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:12.181+0000 7f16743da740 1 --2- 172.21.15.32:0/2991099599 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5643ad93e590 0x5643add2f2e0 unknown :-1 s=CLOSED pgs=1715 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:12.184 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:12.181+0000 7f16743da740 1 --2- 172.21.15.32:0/2991099599 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x5643ad9363b0 0x5643add2eda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:12.184 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:12.181+0000 7f16743da740 1 --2- 172.21.15.32:0/2991099599 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5643ad95f270 0x5643add4c0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:12.184 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:12.181+0000 7f16743da740 1 -- 172.21.15.32:0/2991099599 >> 172.21.15.32:0/2991099599 conn(0x5643ad9533c0 msgr2=0x5643add04fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:58:12.184 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:12.181+0000 7f16743da740 1 -- 172.21.15.32:0/2991099599 shutdown_connections 2024-08-03T20:58:12.184 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:12.181+0000 7f16743da740 1 -- 172.21.15.32:0/2991099599 wait complete. 2024-08-03T20:58:12.184 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:12.182+0000 7f16743da740 1 -- start start 2024-08-03T20:58:12.184 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:12.183+0000 7f16743da740 1 Processor -- start 2024-08-03T20:58:12.187 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:12.184+0000 7f16743da740 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x5643ad9363b0 0x5643add0de70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:58:12.187 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:12.184+0000 7f16743da740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5643ad93e590 0x5643add436d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:58:12.187 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:12.184+0000 7f16743da740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5643ad95f270 0x5643add04ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:58:12.187 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:12.184+0000 7f16743da740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x5643ad9d9620 con 0x5643ad93e590 2024-08-03T20:58:12.187 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:12.184+0000 7f16743da740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x5643adc04160 con 0x5643ad95f270 2024-08-03T20:58:12.188 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:12.184+0000 7f16743da740 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x5643add1cf00 con 0x5643ad9363b0 2024-08-03T20:58:12.188 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:12.184+0000 7f16743da740 10 client.0 _ll_register_callbacks cb 0x5643adc136b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-03T20:58:12.188 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:12.185+0000 7f165fe00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5643ad93e590 0x5643add436d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:58:12.188 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:12.185+0000 7f1666800640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x5643ad9363b0 0x5643add0de70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:58:12.188 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:12.185+0000 7f165fe00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5643ad93e590 0x5643add436d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:35794/0 (socket says 172.21.15.32:35794) 2024-08-03T20:58:12.188 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:12.185+0000 7f165fe00640 1 -- 172.21.15.32:0/1902133327 learned_addr learned my addr 172.21.15.32:0/1902133327 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:58:12.188 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:12.185+0000 7f1667200640 1 --2- 172.21.15.32:0/1902133327 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5643ad95f270 0x5643add04ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:58:12.188 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:12.185+0000 7f165fe00640 1 -- 172.21.15.32:0/1902133327 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x5643ad9363b0 msgr2=0x5643add0de70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:58:12.188 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:12.185+0000 7f165fe00640 1 --2- 172.21.15.32:0/1902133327 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x5643ad9363b0 0x5643add0de70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:12.188 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:12.185+0000 7f165fe00640 1 -- 172.21.15.32:0/1902133327 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5643ad95f270 msgr2=0x5643add04ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:58:12.188 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:12.185+0000 7f165fe00640 1 --2- 172.21.15.32:0/1902133327 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5643ad95f270 0x5643add04ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:12.188 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:12.185+0000 7f165fe00640 1 -- 172.21.15.32:0/1902133327 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5643ada416b0 con 0x5643ad93e590 2024-08-03T20:58:12.188 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:12.185+0000 7f165fe00640 1 --2- 172.21.15.32:0/1902133327 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5643ad93e590 0x5643add436d0 secure :-1 s=READY pgs=1716 cs=0 l=1 rev1=1 crypto rx=0x7f16500112b0 tx=0x7f1650011770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=573ecc7d5accb4de server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:58:12.189 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:12.185+0000 7f164b400640 10 client.0 ms_handle_connect on v2:172.21.15.32:3300/0 2024-08-03T20:58:12.189 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:12.186+0000 7f164b400640 1 -- 172.21.15.32:0/1902133327 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1650019070 con 0x5643ad93e590 2024-08-03T20:58:12.189 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:12.186+0000 7f16743da740 1 -- 172.21.15.32:0/1902133327 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1660065d40 con 0x5643ad93e590 2024-08-03T20:58:12.189 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:12.186+0000 7f164b400640 1 -- 172.21.15.32:0/1902133327 <== mon.0 v2:172.21.15.32:3300/0 2 ==== osd_map(111..111 src has 1..111) ==== 8090+0+0 (secure 0 0 0) 0x7f1650005020 con 0x5643ad93e590 2024-08-03T20:58:12.189 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:12.186+0000 7f164b400640 1 -- 172.21.15.32:0/1902133327 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_get_version_reply(handle=1 version=244) ==== 24+0+0 (secure 0 0 0) 0x7f1650024050 con 0x5643ad93e590 2024-08-03T20:58:12.189 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:12.186+0000 7f16743da740 10 client.9264 fetch_fsmap learned FSMap version 244 2024-08-03T20:58:12.189 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:12.186+0000 7f16743da740 1 -- 172.21.15.32:0/1902133327 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,fsmap=244,monmap=2+}) -- 0x5643add47a90 con 0x5643ad93e590 2024-08-03T20:58:12.189 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:12.186+0000 7f164b400640 1 -- 172.21.15.32:0/1902133327 <== mon.0 v2:172.21.15.32:3300/0 4 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f1660065d40 con 0x5643ad93e590 2024-08-03T20:58:12.189 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:12.186+0000 7f164b400640 1 -- 172.21.15.32:0/1902133327 <== mon.0 v2:172.21.15.32:3300/0 5 ==== fsmap(e 244) ==== 3502+0+0 (secure 0 0 0) 0x7f1650005310 con 0x5643ad93e590 2024-08-03T20:58:12.189 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:12.186+0000 7f16743da740 10 client.9264 fetch_fsmap finished waiting for FSMap version 244 2024-08-03T20:58:12.189 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:12.186+0000 7f16743da740 10 client.9264 resolve_mds: resolved name 'a' to daemon mds.a 2024-08-03T20:58:12.189 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:12.187+0000 7f16743da740 20 client.9264 populate_metadata read hostname 'smithi032' 2024-08-03T20:58:12.189 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:12.187+0000 7f16743da740 1 --2- 172.21.15.32:0/1902133327 >> [v2:172.21.15.32:6833/3340816283,v1:172.21.15.32:6835/3340816283] conn(0x5643ad91de40 0x5643add442c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:58:12.190 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:12.187+0000 7f16743da740 4 client.9264 mds_command: new command op to 9237 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-08-03T20:58:12.190 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:12.187+0000 7f16743da740 1 -- 172.21.15.32:0/1902133327 --> [v2:172.21.15.32:6833/3340816283,v1:172.21.15.32:6835/3340816283] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5643adc799b0 con 0x5643ad91de40 2024-08-03T20:58:12.190 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:12.187+0000 7f1666800640 1 --2- 172.21.15.32:0/1902133327 >> [v2:172.21.15.32:6833/3340816283,v1:172.21.15.32:6835/3340816283] conn(0x5643ad91de40 0x5643add442c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:58:12.191 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:12.187+0000 7f1666800640 1 --2- 172.21.15.32:0/1902133327 >> [v2:172.21.15.32:6833/3340816283,v1:172.21.15.32:6835/3340816283] conn(0x5643ad91de40 0x5643add442c0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ff663d5aa4f7256b server_cookie=6c90f86391046ede in_seq=0 out_seq=0 2024-08-03T20:58:12.191 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:12.187+0000 7f164b400640 10 client.9264 ms_handle_connect on v2:172.21.15.32:6833/3340816283 2024-08-03T20:58:12.191 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:12.188+0000 7f164b400640 1 -- 172.21.15.32:0/1902133327 <== mds.0 v2:172.21.15.32:6833/3340816283 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x5643adc799b0 con 0x5643ad91de40 2024-08-03T20:58:12.191 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:12.188+0000 7f164b400640 10 client.9264 handle_command_reply: tid=0 2024-08-03T20:58:12.191 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:12.188+0000 7f16743da740 1 client.9264 shutdown 2024-08-03T20:58:12.191 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:12.189+0000 7f16743da740 1 -- 172.21.15.32:0/1902133327 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5643ad93e590 msgr2=0x5643add436d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:58:12.191 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:12.189+0000 7f16743da740 1 --2- 172.21.15.32:0/1902133327 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5643ad93e590 0x5643add436d0 secure :-1 s=READY pgs=1716 cs=0 l=1 rev1=1 crypto rx=0x7f16500112b0 tx=0x7f1650011770 comp rx=0 tx=0).stop 2024-08-03T20:58:12.191 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:12.189+0000 7f16743da740 1 -- 172.21.15.32:0/1902133327 shutdown_connections 2024-08-03T20:58:12.191 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:12.189+0000 7f16743da740 1 --2- 172.21.15.32:0/1902133327 >> [v2:172.21.15.32:6833/3340816283,v1:172.21.15.32:6835/3340816283] conn(0x5643ad91de40 0x5643add442c0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:12.192 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:12.189+0000 7f16743da740 1 --2- 172.21.15.32:0/1902133327 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5643ad95f270 0x5643add04ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:12.192 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:12.189+0000 7f16743da740 1 --2- 172.21.15.32:0/1902133327 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5643ad93e590 0x5643add436d0 unknown :-1 s=CLOSED pgs=1716 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:12.192 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:12.189+0000 7f164b400640 0 client.9264 ms_handle_reset on v2:172.21.15.32:6833/3340816283 2024-08-03T20:58:12.192 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:12.189+0000 7f16743da740 1 --2- 172.21.15.32:0/1902133327 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x5643ad9363b0 0x5643add0de70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:12.192 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:12.189+0000 7f16743da740 1 -- 172.21.15.32:0/1902133327 >> 172.21.15.32:0/1902133327 conn(0x5643ad9533c0 msgr2=0x5643adcfe390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:58:12.192 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:12.189+0000 7f16743da740 1 -- 172.21.15.32:0/1902133327 shutdown_connections 2024-08-03T20:58:12.192 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:12.189+0000 7f16743da740 1 -- 172.21.15.32:0/1902133327 wait complete. 2024-08-03T20:58:12.192 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:12.189+0000 7f16743da740 20 client.9264 trim_cache size 0 max 16384 2024-08-03T20:58:12.210 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:12.209+0000 7f16743da740 1 Processor -- start 2024-08-03T20:58:12.210 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:12.209+0000 7f16743da740 1 -- start start 2024-08-03T20:58:12.212 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:12.210+0000 7f16743da740 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x5643ad9363b0 0x5643add04680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:58:12.212 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:12.210+0000 7f16743da740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5643ad93e590 0x5643add6cf30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:58:12.212 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:12.210+0000 7f16743da740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5643ad95f270 0x5643add6d470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:58:12.212 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:12.211+0000 7f16743da740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f16682022e0 con 0x5643ad93e590 2024-08-03T20:58:12.212 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:12.211+0000 7f16743da740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f1668201620 con 0x5643ad95f270 2024-08-03T20:58:12.212 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:12.211+0000 7f16743da740 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7f1668201c70 con 0x5643ad9363b0 2024-08-03T20:58:12.212 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:12.211+0000 7f165fe00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5643ad93e590 0x5643add6cf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:58:12.213 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:12.211+0000 7f165fe00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5643ad93e590 0x5643add6cf30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:35802/0 (socket says 172.21.15.32:35802) 2024-08-03T20:58:12.213 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:12.211+0000 7f165fe00640 1 -- 172.21.15.32:0/2445832546 learned_addr learned my addr 172.21.15.32:0/2445832546 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:58:12.213 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:12.211+0000 7f1666800640 1 --2- 172.21.15.32:0/2445832546 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x5643ad9363b0 0x5643add04680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:58:12.213 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:12.211+0000 7f1667200640 1 --2- 172.21.15.32:0/2445832546 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5643ad95f270 0x5643add6d470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:58:12.213 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:12.211+0000 7f165fe00640 1 -- 172.21.15.32:0/2445832546 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x5643ad9363b0 msgr2=0x5643add04680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:58:12.213 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:12.211+0000 7f165fe00640 1 --2- 172.21.15.32:0/2445832546 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x5643ad9363b0 0x5643add04680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:12.213 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:12.211+0000 7f165fe00640 1 -- 172.21.15.32:0/2445832546 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5643ad95f270 msgr2=0x5643add6d470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:58:12.213 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:12.211+0000 7f165fe00640 1 --2- 172.21.15.32:0/2445832546 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5643ad95f270 0x5643add6d470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:12.213 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:12.211+0000 7f165fe00640 1 -- 172.21.15.32:0/2445832546 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5643adcfa030 con 0x5643ad93e590 2024-08-03T20:58:12.213 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:12.211+0000 7f1666800640 1 --2- 172.21.15.32:0/2445832546 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x5643ad9363b0 0x5643add04680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-03T20:58:12.213 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:12.211+0000 7f165fe00640 1 --2- 172.21.15.32:0/2445832546 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5643ad93e590 0x5643add6cf30 secure :-1 s=READY pgs=1717 cs=0 l=1 rev1=1 crypto rx=0x7f1650007b50 tx=0x7f1650007ce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b34bbf6396d8025d server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:58:12.213 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:12.212+0000 7f1656600640 1 -- 172.21.15.32:0/2445832546 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f165002d070 con 0x5643ad93e590 2024-08-03T20:58:12.213 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:12.212+0000 7f1656600640 1 -- 172.21.15.32:0/2445832546 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f1650021070 con 0x5643ad93e590 2024-08-03T20:58:12.213 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:12.212+0000 7f1656600640 1 -- 172.21.15.32:0/2445832546 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1650019040 con 0x5643ad93e590 2024-08-03T20:58:12.214 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:12.212+0000 7f16743da740 1 -- 172.21.15.32:0/2445832546 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5643ad93e590 msgr2=0x5643add6cf30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:58:12.215 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:12.212+0000 7f16743da740 1 --2- 172.21.15.32:0/2445832546 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5643ad93e590 0x5643add6cf30 secure :-1 s=READY pgs=1717 cs=0 l=1 rev1=1 crypto rx=0x7f1650007b50 tx=0x7f1650007ce0 comp rx=0 tx=0).stop 2024-08-03T20:58:12.215 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:12.212+0000 7f16743da740 1 -- 172.21.15.32:0/2445832546 shutdown_connections 2024-08-03T20:58:12.215 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:12.212+0000 7f16743da740 1 --2- 172.21.15.32:0/2445832546 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5643ad95f270 0x5643add6d470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:12.215 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:12.212+0000 7f16743da740 1 --2- 172.21.15.32:0/2445832546 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5643ad93e590 0x5643add6cf30 unknown :-1 s=CLOSED pgs=1717 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:12.215 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:12.212+0000 7f16743da740 1 --2- 172.21.15.32:0/2445832546 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x5643ad9363b0 0x5643add04680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:12.215 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:12.212+0000 7f16743da740 1 -- 172.21.15.32:0/2445832546 >> 172.21.15.32:0/2445832546 conn(0x5643ad9533c0 msgr2=0x5643add432b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:58:12.215 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:12.213+0000 7f16743da740 1 -- 172.21.15.32:0/2445832546 shutdown_connections 2024-08-03T20:58:12.215 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:12.213+0000 7f16743da740 1 -- 172.21.15.32:0/2445832546 wait complete. 2024-08-03T20:58:12.215 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:12.213+0000 7f16743da740 1 -- start start 2024-08-03T20:58:12.215 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:12.214+0000 7f16743da740 1 Processor -- start 2024-08-03T20:58:12.218 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:12.215+0000 7f16743da740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5643ad9363b0 0x5643addad420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:58:12.218 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:12.215+0000 7f16743da740 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x5643ad93e590 0x5643add451e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:58:12.218 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:12.215+0000 7f16743da740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5643ad95f270 0x5643add45720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:58:12.218 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:12.215+0000 7f16743da740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x5643add05010 con 0x5643ad9363b0 2024-08-03T20:58:12.218 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:12.215+0000 7f16743da740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x5643add1cd00 con 0x5643ad95f270 2024-08-03T20:58:12.218 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:12.215+0000 7f16743da740 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x5643add47fb0 con 0x5643ad93e590 2024-08-03T20:58:12.218 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:12.215+0000 7f16743da740 10 client.0 _ll_register_callbacks cb 0x7f1650001b30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-03T20:58:12.219 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:12.215+0000 7f1666800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5643ad9363b0 0x5643addad420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:58:12.219 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:12.215+0000 7f1666800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5643ad9363b0 0x5643addad420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:35814/0 (socket says 172.21.15.32:35814) 2024-08-03T20:58:12.219 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:12.215+0000 7f1666800640 1 -- 172.21.15.32:0/1955468222 learned_addr learned my addr 172.21.15.32:0/1955468222 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:58:12.219 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:12.215+0000 7f1667200640 1 --2- 172.21.15.32:0/1955468222 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5643ad95f270 0x5643add45720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:58:12.219 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:12.215+0000 7f165fe00640 1 --2- 172.21.15.32:0/1955468222 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x5643ad93e590 0x5643add451e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:58:12.219 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:12.216+0000 7f1666800640 1 -- 172.21.15.32:0/1955468222 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x5643ad93e590 msgr2=0x5643add451e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:58:12.219 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:12.216+0000 7f1666800640 1 --2- 172.21.15.32:0/1955468222 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x5643ad93e590 0x5643add451e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:12.219 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:12.216+0000 7f1666800640 1 -- 172.21.15.32:0/1955468222 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5643ad95f270 msgr2=0x5643add45720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:58:12.219 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:12.216+0000 7f1666800640 1 --2- 172.21.15.32:0/1955468222 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5643ad95f270 0x5643add45720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:12.219 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:12.216+0000 7f1666800640 1 -- 172.21.15.32:0/1955468222 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5643add58510 con 0x5643ad9363b0 2024-08-03T20:58:12.219 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:12.216+0000 7f1666800640 1 --2- 172.21.15.32:0/1955468222 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5643ad9363b0 0x5643addad420 secure :-1 s=READY pgs=1718 cs=0 l=1 rev1=1 crypto rx=0x7f16580095f0 tx=0x7f1658005f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ce532b1f331eb816 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:58:12.219 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:12.216+0000 7f164b400640 10 client.0 ms_handle_connect on v2:172.21.15.32:3300/0 2024-08-03T20:58:12.219 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:12.216+0000 7f164b400640 1 -- 172.21.15.32:0/1955468222 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1658057030 con 0x5643ad9363b0 2024-08-03T20:58:12.219 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:12.216+0000 7f164b400640 1 -- 172.21.15.32:0/1955468222 <== mon.0 v2:172.21.15.32:3300/0 2 ==== osd_map(111..111 src has 1..111) ==== 8090+0+0 (secure 0 0 0) 0x7f1658003b00 con 0x5643ad9363b0 2024-08-03T20:58:12.220 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:12.216+0000 7f16743da740 1 -- 172.21.15.32:0/1955468222 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1650021070 con 0x5643ad9363b0 2024-08-03T20:58:12.220 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:12.217+0000 7f164b400640 1 -- 172.21.15.32:0/1955468222 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_get_version_reply(handle=1 version=244) ==== 24+0+0 (secure 0 0 0) 0x7f1658003d00 con 0x5643ad9363b0 2024-08-03T20:58:12.220 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:12.217+0000 7f16743da740 10 client.9270 fetch_fsmap learned FSMap version 244 2024-08-03T20:58:12.220 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:12.217+0000 7f16743da740 1 -- 172.21.15.32:0/1955468222 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,fsmap=244,monmap=2+}) -- 0x5643addad960 con 0x5643ad9363b0 2024-08-03T20:58:12.220 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:12.217+0000 7f164b400640 1 -- 172.21.15.32:0/1955468222 <== mon.0 v2:172.21.15.32:3300/0 4 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f1650021070 con 0x5643ad9363b0 2024-08-03T20:58:12.220 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:12.217+0000 7f164b400640 1 -- 172.21.15.32:0/1955468222 <== mon.0 v2:172.21.15.32:3300/0 5 ==== fsmap(e 244) ==== 3502+0+0 (secure 0 0 0) 0x7f1658043040 con 0x5643ad9363b0 2024-08-03T20:58:12.220 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:12.217+0000 7f16743da740 10 client.9270 fetch_fsmap finished waiting for FSMap version 244 2024-08-03T20:58:12.220 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:12.217+0000 7f16743da740 10 client.9270 resolve_mds: resolved name 'a' to daemon mds.a 2024-08-03T20:58:12.220 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:12.217+0000 7f16743da740 20 client.9270 populate_metadata read hostname 'smithi032' 2024-08-03T20:58:12.220 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:12.217+0000 7f16743da740 1 --2- 172.21.15.32:0/1955468222 >> [v2:172.21.15.32:6833/3340816283,v1:172.21.15.32:6835/3340816283] conn(0x5643addc8220 0x5643add2a510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:58:12.220 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:12.217+0000 7f16743da740 4 client.9270 mds_command: new command op to 9237 tid=0 multi_id=0 [{"prefix": "status"}] 2024-08-03T20:58:12.220 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:12.217+0000 7f16743da740 1 -- 172.21.15.32:0/1955468222 --> [v2:172.21.15.32:6833/3340816283,v1:172.21.15.32:6835/3340816283] -- command(tid 0: {"prefix": "status"}) -- 0x5643adc75450 con 0x5643addc8220 2024-08-03T20:58:12.220 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:12.217+0000 7f165fe00640 1 --2- 172.21.15.32:0/1955468222 >> [v2:172.21.15.32:6833/3340816283,v1:172.21.15.32:6835/3340816283] conn(0x5643addc8220 0x5643add2a510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:58:12.222 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:12.218+0000 7f165fe00640 1 --2- 172.21.15.32:0/1955468222 >> [v2:172.21.15.32:6833/3340816283,v1:172.21.15.32:6835/3340816283] conn(0x5643addc8220 0x5643add2a510 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c9dad7b6d5a81dcb server_cookie=90b99a9c4fc726bc in_seq=0 out_seq=0 2024-08-03T20:58:12.222 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:12.218+0000 7f164b400640 10 client.9270 ms_handle_connect on v2:172.21.15.32:6833/3340816283 2024-08-03T20:58:12.222 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:12.218+0000 7f164b400640 1 -- 172.21.15.32:0/1955468222 <== mds.0 v2:172.21.15.32:6833/3340816283 1 ==== command_reply(tid 0: 0 ) ==== 8+0+334 (crc 0 0 0) 0x5643adc75450 con 0x5643addc8220 2024-08-03T20:58:12.222 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:12.218+0000 7f164b400640 10 client.9270 handle_command_reply: tid=0 2024-08-03T20:58:12.222 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:12.218+0000 7f16743da740 1 client.9270 shutdown 2024-08-03T20:58:12.222 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:12.218+0000 7f16743da740 1 -- 172.21.15.32:0/1955468222 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5643ad9363b0 msgr2=0x5643addad420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:58:12.222 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:12.218+0000 7f16743da740 1 --2- 172.21.15.32:0/1955468222 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5643ad9363b0 0x5643addad420 secure :-1 s=READY pgs=1718 cs=0 l=1 rev1=1 crypto rx=0x7f16580095f0 tx=0x7f1658005f90 comp rx=0 tx=0).stop 2024-08-03T20:58:12.222 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:12.219+0000 7f16743da740 1 -- 172.21.15.32:0/1955468222 shutdown_connections 2024-08-03T20:58:12.222 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:12.219+0000 7f16743da740 1 --2- 172.21.15.32:0/1955468222 >> [v2:172.21.15.32:6833/3340816283,v1:172.21.15.32:6835/3340816283] conn(0x5643addc8220 0x5643add2a510 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:12.222 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:12.219+0000 7f16743da740 1 --2- 172.21.15.32:0/1955468222 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5643ad95f270 0x5643add45720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:12.222 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:12.219+0000 7f16743da740 1 --2- 172.21.15.32:0/1955468222 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x5643ad93e590 0x5643add451e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:12.222 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:12.219+0000 7f16743da740 1 --2- 172.21.15.32:0/1955468222 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5643ad9363b0 0x5643addad420 unknown :-1 s=CLOSED pgs=1718 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:12.222 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:12.219+0000 7f164b400640 0 client.9270 ms_handle_reset on v2:172.21.15.32:6833/3340816283 2024-08-03T20:58:12.223 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:12.219+0000 7f16743da740 1 -- 172.21.15.32:0/1955468222 >> 172.21.15.32:0/1955468222 conn(0x5643ad9533c0 msgr2=0x5643add6cf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:58:12.223 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:12.219+0000 7f16743da740 1 -- 172.21.15.32:0/1955468222 shutdown_connections 2024-08-03T20:58:12.223 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:12.219+0000 7f16743da740 1 -- 172.21.15.32:0/1955468222 wait complete. 2024-08-03T20:58:12.223 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:12.219+0000 7f16743da740 20 client.9270 trim_cache size 0 max 16384 2024-08-03T20:58:12.223 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-08-03T20:58:12.223 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_fsid": "40ab141a-01d7-49b9-ae3c-2edec9876e95", 2024-08-03T20:58:12.223 INFO:teuthology.orchestra.run.smithi032.stdout: "whoami": 0, 2024-08-03T20:58:12.223 INFO:teuthology.orchestra.run.smithi032.stdout: "id": 9237, 2024-08-03T20:58:12.223 INFO:teuthology.orchestra.run.smithi032.stdout: "want_state": "up:active", 2024-08-03T20:58:12.224 INFO:teuthology.orchestra.run.smithi032.stdout: "state": "up:active", 2024-08-03T20:58:12.224 INFO:teuthology.orchestra.run.smithi032.stdout: "fs_name": "cephfs", 2024-08-03T20:58:12.224 INFO:teuthology.orchestra.run.smithi032.stdout: "rank_uptime": 7.1659002699999998, 2024-08-03T20:58:12.224 INFO:teuthology.orchestra.run.smithi032.stdout: "mdsmap_epoch": 244, 2024-08-03T20:58:12.224 INFO:teuthology.orchestra.run.smithi032.stdout: "osdmap_epoch": 111, 2024-08-03T20:58:12.224 INFO:teuthology.orchestra.run.smithi032.stdout: "osdmap_epoch_barrier": 111, 2024-08-03T20:58:12.224 INFO:teuthology.orchestra.run.smithi032.stdout: "uptime": 7.2808674399999997 2024-08-03T20:58:12.224 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-08-03T20:58:12.225 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:12.219+0000 7f166d400640 1 -- 172.21.15.32:0/3553513544 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f1644077010 msgr2=0x7f1644079520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:58:12.225 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:12.219+0000 7f166d400640 1 --2- 172.21.15.32:0/3553513544 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f1644077010 0x7f1644079520 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7f1658034000 tx=0x7f1658004730 comp rx=0 tx=0).stop 2024-08-03T20:58:12.225 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:12.219+0000 7f166d400640 1 -- 172.21.15.32:0/3553513544 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f166815fff0 msgr2=0x7f166815e610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:58:12.225 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:12.219+0000 7f166d400640 1 --2- 172.21.15.32:0/3553513544 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f166815fff0 0x7f166815e610 secure :-1 s=READY pgs=1714 cs=0 l=1 rev1=1 crypto rx=0x7f166000da90 tx=0x7f166000df50 comp rx=0 tx=0).stop 2024-08-03T20:58:12.225 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:12.220+0000 7f166d400640 1 -- 172.21.15.32:0/3553513544 shutdown_connections 2024-08-03T20:58:12.225 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:12.220+0000 7f166d400640 1 --2- 172.21.15.32:0/3553513544 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f1644077010 0x7f1644079520 unknown :-1 s=CLOSED pgs=699 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:12.225 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:12.220+0000 7f166d400640 1 --2- 172.21.15.32:0/3553513544 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f1668073e60 0x7f166815b8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:12.225 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:12.220+0000 7f166d400640 1 --2- 172.21.15.32:0/3553513544 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f166815fff0 0x7f166815e610 unknown :-1 s=CLOSED pgs=1714 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:12.225 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:12.220+0000 7f166d400640 1 --2- 172.21.15.32:0/3553513544 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f166806e930 0x7f166813fee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:12.225 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:12.220+0000 7f166d400640 1 -- 172.21.15.32:0/3553513544 >> 172.21.15.32:0/3553513544 conn(0x7f16680b1bd0 msgr2=0x7f1668074270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:58:12.226 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:12.220+0000 7f166d400640 1 -- 172.21.15.32:0/3553513544 shutdown_connections 2024-08-03T20:58:12.226 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:12.220+0000 7f166d400640 1 -- 172.21.15.32:0/3553513544 wait complete. 2024-08-03T20:58:12.242 DEBUG:tasks.ceph_test_case:wait_until_true: success in 5s and 0 retries 2024-08-03T20:58:12.242 INFO:tasks.cephfs.test_damage:Daemons came up after mutation 'Truncate 500.00000000', proceeding to ls 2024-08-03T20:58:12.243 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-08-03T20:58:12.473 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:12.472+0000 7f1c58a00640 1 -- 172.21.15.32:0/310994381 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f1c541523f0 msgr2=0x7f1c540b4ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:58:12.473 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:12.472+0000 7f1c58a00640 1 --2- 172.21.15.32:0/310994381 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f1c541523f0 0x7f1c540b4ce0 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7f1c4c005fd0 tx=0x7f1c4c02a8b0 comp rx=0 tx=0).stop 2024-08-03T20:58:12.473 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:12.472+0000 7f1c58a00640 1 -- 172.21.15.32:0/310994381 shutdown_connections 2024-08-03T20:58:12.473 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:12.472+0000 7f1c58a00640 1 --2- 172.21.15.32:0/310994381 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f1c541523f0 0x7f1c540b4ce0 unknown :-1 s=CLOSED pgs=534 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:12.473 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:12.472+0000 7f1c58a00640 1 -- 172.21.15.32:0/310994381 >> 172.21.15.32:0/310994381 conn(0x7f1c540b1bd0 msgr2=0x7f1c540b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:58:12.473 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:12.472+0000 7f1c58a00640 1 -- 172.21.15.32:0/310994381 shutdown_connections 2024-08-03T20:58:12.473 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:12.472+0000 7f1c58a00640 1 -- 172.21.15.32:0/310994381 wait complete. 2024-08-03T20:58:12.474 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:12.474+0000 7f1c58a00640 1 Processor -- start 2024-08-03T20:58:12.475 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:12.474+0000 7f1c58a00640 1 -- start start 2024-08-03T20:58:12.477 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:12.476+0000 7f1c58a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1c540b38b0 0x7f1c5414f750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:58:12.477 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:12.476+0000 7f1c58a00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f1c541523f0 0x7f1c54157020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:58:12.477 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:12.476+0000 7f1c58a00640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f1c541f9120 0x7f1c540d0170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:58:12.477 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:12.476+0000 7f1c58a00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f1c540cec60 con 0x7f1c540b38b0 2024-08-03T20:58:12.477 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:12.476+0000 7f1c58a00640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f1c540ceae0 con 0x7f1c541523f0 2024-08-03T20:58:12.477 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:12.476+0000 7f1c58a00640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7f1c540cede0 con 0x7f1c541f9120 2024-08-03T20:58:12.478 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:12.476+0000 7f1c51e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1c540b38b0 0x7f1c5414f750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:58:12.478 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:12.476+0000 7f1c51e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1c540b38b0 0x7f1c5414f750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:35830/0 (socket says 172.21.15.32:35830) 2024-08-03T20:58:12.478 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:12.476+0000 7f1c51e00640 1 -- 172.21.15.32:0/3652321218 learned_addr learned my addr 172.21.15.32:0/3652321218 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:58:12.478 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:12.476+0000 7f1c52800640 1 --2- 172.21.15.32:0/3652321218 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f1c541f9120 0x7f1c540d0170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:58:12.478 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:12.476+0000 7f1c4be00640 1 --2- 172.21.15.32:0/3652321218 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f1c541523f0 0x7f1c54157020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:58:12.478 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:12.477+0000 7f1c51e00640 1 -- 172.21.15.32:0/3652321218 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f1c541f9120 msgr2=0x7f1c540d0170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:58:12.478 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:12.477+0000 7f1c51e00640 1 --2- 172.21.15.32:0/3652321218 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f1c541f9120 0x7f1c540d0170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:12.478 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:12.477+0000 7f1c51e00640 1 -- 172.21.15.32:0/3652321218 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f1c541523f0 msgr2=0x7f1c54157020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:58:12.478 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:12.477+0000 7f1c51e00640 1 --2- 172.21.15.32:0/3652321218 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f1c541523f0 0x7f1c54157020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:12.479 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:12.477+0000 7f1c51e00640 1 -- 172.21.15.32:0/3652321218 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1c54072a20 con 0x7f1c540b38b0 2024-08-03T20:58:12.479 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:12.477+0000 7f1c51e00640 1 --2- 172.21.15.32:0/3652321218 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1c540b38b0 0x7f1c5414f750 secure :-1 s=READY pgs=1720 cs=0 l=1 rev1=1 crypto rx=0x7f1c4400c3d0 tx=0x7f1c4400c890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:58:12.479 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:12.477+0000 7f1c4a000640 1 -- 172.21.15.32:0/3652321218 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c44017070 con 0x7f1c540b38b0 2024-08-03T20:58:12.480 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:12.477+0000 7f1c4a000640 1 -- 172.21.15.32:0/3652321218 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f1c440053c0 con 0x7f1c540b38b0 2024-08-03T20:58:12.480 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:12.477+0000 7f1c58a00640 1 -- 172.21.15.32:0/3652321218 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1c54072d00 con 0x7f1c540b38b0 2024-08-03T20:58:12.480 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:12.477+0000 7f1c4a000640 1 -- 172.21.15.32:0/3652321218 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c44003850 con 0x7f1c540b38b0 2024-08-03T20:58:12.480 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:12.477+0000 7f1c58a00640 1 -- 172.21.15.32:0/3652321218 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1c540730d0 con 0x7f1c540b38b0 2024-08-03T20:58:12.480 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:12.478+0000 7f1c58a00640 1 -- 172.21.15.32:0/3652321218 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1c541580e0 con 0x7f1c540b38b0 2024-08-03T20:58:12.480 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:12.480+0000 7f1c4a000640 1 -- 172.21.15.32:0/3652321218 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f1c4401a070 con 0x7f1c540b38b0 2024-08-03T20:58:12.481 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:12.480+0000 7f1c4a000640 1 --2- 172.21.15.32:0/3652321218 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f1c30077010 0x7f1c30079520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:58:12.481 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:12.481+0000 7f1c4a000640 1 -- 172.21.15.32:0/3652321218 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(111..111 src has 1..111) ==== 8090+0+0 (secure 0 0 0) 0x7f1c440986a0 con 0x7f1c540b38b0 2024-08-03T20:58:12.481 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:12.481+0000 7f1c4be00640 1 --2- 172.21.15.32:0/3652321218 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f1c30077010 0x7f1c30079520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:58:12.482 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:12.482+0000 7f1c4be00640 1 --2- 172.21.15.32:0/3652321218 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f1c30077010 0x7f1c30079520 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7f1c54152ae0 tx=0x7f1c3c009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:58:12.485 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:12.485+0000 7f1c4a000640 1 -- 172.21.15.32:0/3652321218 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7f1c44060f40 con 0x7f1c540b38b0 2024-08-03T20:58:12.732 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:12.731+0000 7f1c58a00640 1 -- 172.21.15.32:0/3652321218 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f1c54073390 con 0x7f1c540b38b0 2024-08-03T20:58:12.732 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:12.732+0000 7f1c4a000640 1 -- 172.21.15.32:0/3652321218 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v244) ==== 53+0+73 (secure 0 0 0) 0x7f1c44069180 con 0x7f1c540b38b0 2024-08-03T20:58:12.733 INFO:teuthology.orchestra.run.smithi032.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-08-03T20:58:12.735 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:12.734+0000 7f1c58a00640 1 -- 172.21.15.32:0/3652321218 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f1c30077010 msgr2=0x7f1c30079520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:58:12.735 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:12.734+0000 7f1c58a00640 1 --2- 172.21.15.32:0/3652321218 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f1c30077010 0x7f1c30079520 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7f1c54152ae0 tx=0x7f1c3c009040 comp rx=0 tx=0).stop 2024-08-03T20:58:12.735 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:12.734+0000 7f1c58a00640 1 -- 172.21.15.32:0/3652321218 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1c540b38b0 msgr2=0x7f1c5414f750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:58:12.735 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:12.734+0000 7f1c58a00640 1 --2- 172.21.15.32:0/3652321218 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1c540b38b0 0x7f1c5414f750 secure :-1 s=READY pgs=1720 cs=0 l=1 rev1=1 crypto rx=0x7f1c4400c3d0 tx=0x7f1c4400c890 comp rx=0 tx=0).stop 2024-08-03T20:58:12.735 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:12.734+0000 7f1c58a00640 1 -- 172.21.15.32:0/3652321218 shutdown_connections 2024-08-03T20:58:12.735 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:12.734+0000 7f1c58a00640 1 --2- 172.21.15.32:0/3652321218 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f1c30077010 0x7f1c30079520 unknown :-1 s=CLOSED pgs=700 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:12.736 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:12.734+0000 7f1c58a00640 1 --2- 172.21.15.32:0/3652321218 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f1c541523f0 0x7f1c54157020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:12.736 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:12.734+0000 7f1c58a00640 1 --2- 172.21.15.32:0/3652321218 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f1c541f9120 0x7f1c540d0170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:12.736 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:12.734+0000 7f1c58a00640 1 --2- 172.21.15.32:0/3652321218 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1c540b38b0 0x7f1c5414f750 unknown :-1 s=CLOSED pgs=1720 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:12.736 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:12.734+0000 7f1c58a00640 1 -- 172.21.15.32:0/3652321218 >> 172.21.15.32:0/3652321218 conn(0x7f1c540b1bd0 msgr2=0x7f1c54068940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:58:12.736 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:12.735+0000 7f1c58a00640 1 -- 172.21.15.32:0/3652321218 shutdown_connections 2024-08-03T20:58:12.736 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:12.735+0000 7f1c58a00640 1 -- 172.21.15.32:0/3652321218 wait complete. 2024-08-03T20:58:12.760 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-08-03T20:58:12.760 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-08-03T20:58:12.760 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi032.front.sepia.ceph.com 2024-08-03T20:58:12.760 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-08-03T20:58:12.760 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-08-03T20:58:12.760 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-08-03T20:58:12.760 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2024-08-03T20:58:12.760 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-08-03T20:58:12.760 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-08-03T20:58:12.761 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-08-03T20:58:12.761 DEBUG:teuthology.orchestra.run.smithi032:> ip addr 2024-08-03T20:58:12.796 INFO:teuthology.orchestra.run.smithi032.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-08-03T20:58:12.796 INFO:teuthology.orchestra.run.smithi032.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-08-03T20:58:12.796 INFO:teuthology.orchestra.run.smithi032.stdout: inet 127.0.0.1/8 scope host lo 2024-08-03T20:58:12.796 INFO:teuthology.orchestra.run.smithi032.stdout: valid_lft forever preferred_lft forever 2024-08-03T20:58:12.796 INFO:teuthology.orchestra.run.smithi032.stdout: inet6 ::1/128 scope host 2024-08-03T20:58:12.797 INFO:teuthology.orchestra.run.smithi032.stdout: valid_lft forever preferred_lft forever 2024-08-03T20:58:12.797 INFO:teuthology.orchestra.run.smithi032.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-08-03T20:58:12.797 INFO:teuthology.orchestra.run.smithi032.stdout: link/ether 0c:c4:7a:6b:fd:0c brd ff:ff:ff:ff:ff:ff 2024-08-03T20:58:12.797 INFO:teuthology.orchestra.run.smithi032.stdout: altname enp7s0f0 2024-08-03T20:58:12.797 INFO:teuthology.orchestra.run.smithi032.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-08-03T20:58:12.797 INFO:teuthology.orchestra.run.smithi032.stdout: link/ether 0c:c4:7a:6b:fd:0d brd ff:ff:ff:ff:ff:ff 2024-08-03T20:58:12.797 INFO:teuthology.orchestra.run.smithi032.stdout: altname enp7s0f1 2024-08-03T20:58:12.797 INFO:teuthology.orchestra.run.smithi032.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-08-03T20:58:12.797 INFO:teuthology.orchestra.run.smithi032.stdout: link/ether ac:1f:6b:f8:26:02 brd ff:ff:ff:ff:ff:ff 2024-08-03T20:58:12.797 INFO:teuthology.orchestra.run.smithi032.stdout: altname enp1s0f0 2024-08-03T20:58:12.797 INFO:teuthology.orchestra.run.smithi032.stdout: inet 172.21.15.32/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-08-03T20:58:12.797 INFO:teuthology.orchestra.run.smithi032.stdout: valid_lft 41863sec preferred_lft 41863sec 2024-08-03T20:58:12.797 INFO:teuthology.orchestra.run.smithi032.stdout: inet6 fe80::ae1f:6bff:fef8:2602/64 scope link noprefixroute 2024-08-03T20:58:12.797 INFO:teuthology.orchestra.run.smithi032.stdout: valid_lft forever preferred_lft forever 2024-08-03T20:58:12.798 INFO:teuthology.orchestra.run.smithi032.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-08-03T20:58:12.798 INFO:teuthology.orchestra.run.smithi032.stdout: link/ether ac:1f:6b:f8:26:03 brd ff:ff:ff:ff:ff:ff 2024-08-03T20:58:12.798 INFO:teuthology.orchestra.run.smithi032.stdout: altname enp1s0f1 2024-08-03T20:58:12.798 INFO:teuthology.orchestra.run.smithi032.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-08-03T20:58:12.798 INFO:teuthology.orchestra.run.smithi032.stdout: link/ether 0a:23:2b:ca:2a:93 brd ff:ff:ff:ff:ff:ff 2024-08-03T20:58:12.798 INFO:teuthology.orchestra.run.smithi032.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-08-03T20:58:12.798 INFO:teuthology.orchestra.run.smithi032.stdout: valid_lft forever preferred_lft forever 2024-08-03T20:58:12.798 INFO:teuthology.orchestra.run.smithi032.stdout: inet6 fe80::d804:7cff:fed9:7ba4/64 scope link 2024-08-03T20:58:12.798 INFO:teuthology.orchestra.run.smithi032.stdout: valid_lft forever preferred_lft forever 2024-08-03T20:58:12.798 INFO:teuthology.orchestra.run.smithi032.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-08-03T20:58:12.798 INFO:teuthology.orchestra.run.smithi032.stdout: link/ether 22:86:28:30:4e:62 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-08-03T20:58:12.798 INFO:teuthology.orchestra.run.smithi032.stdout: inet6 fe80::2086:28ff:fe30:4e62/64 scope link 2024-08-03T20:58:12.799 INFO:teuthology.orchestra.run.smithi032.stdout: valid_lft forever preferred_lft forever 2024-08-03T20:58:12.799 INFO:teuthology.orchestra.run.smithi032.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-08-03T20:58:12.799 INFO:teuthology.orchestra.run.smithi032.stdout: link/ether 0a:23:2b:ca:2a:93 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-08-03T20:58:12.799 INFO:teuthology.orchestra.run.smithi032.stdout: inet6 fe80::823:2bff:feca:2a93/64 scope link 2024-08-03T20:58:12.799 INFO:teuthology.orchestra.run.smithi032.stdout: valid_lft forever preferred_lft forever 2024-08-03T20:58:12.799 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-08-03T20:58:12.799 DEBUG:teuthology.orchestra.run.smithi032:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-08-03T20:58:12.890 INFO:teuthology.orchestra.run.smithi032.stdout:1 2024-08-03T20:58:12.893 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-08-03T20:58:12.893 DEBUG:teuthology.orchestra.run.smithi032:> route 2024-08-03T20:58:12.929 INFO:teuthology.orchestra.run.smithi032.stdout:Kernel IP routing table 2024-08-03T20:58:12.929 INFO:teuthology.orchestra.run.smithi032.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-08-03T20:58:12.929 INFO:teuthology.orchestra.run.smithi032.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-08-03T20:58:12.929 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-08-03T20:58:12.929 INFO:teuthology.orchestra.run.smithi032.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-08-03T20:58:12.930 DEBUG:teuthology.orchestra.run.smithi032:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-08-03T20:58:12.930 DEBUG:teuthology.orchestra.run.smithi032:> set -e 2024-08-03T20:58:12.930 DEBUG:teuthology.orchestra.run.smithi032:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-08-03T20:58:12.930 DEBUG:teuthology.orchestra.run.smithi032:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-08-03T20:58:12.930 DEBUG:teuthology.orchestra.run.smithi032:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-08-03T20:58:12.930 DEBUG:teuthology.orchestra.run.smithi032:> ') 2024-08-03T20:58:13.048 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:13 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-08-03T20:58:13.120 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:13 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-08-03T20:58:13.128 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-08-03T20:58:13.128 DEBUG:teuthology.orchestra.run.smithi032:> ip netns list 2024-08-03T20:58:13.163 INFO:teuthology.orchestra.run.smithi032.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-08-03T20:58:13.164 INFO:teuthology.orchestra.run.smithi032.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-08-03T20:58:13.164 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-08-03T20:58:13.164 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-08-03T20:58:13.164 DEBUG:teuthology.orchestra.run.smithi032:> sudo ip link set brx.0 up 2024-08-03T20:58:13.260 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-08-03T20:58:13.261 DEBUG:teuthology.orchestra.run.smithi032:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-08-03T20:58:13.295 INFO:teuthology.orchestra.run.smithi032.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-08-03T20:58:13.295 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-08-03T20:58:13.296 DEBUG:teuthology.orchestra.run.smithi032:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-08-03T20:58:13.370 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2024-08-03T20:58:13.370 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-08-03T20:58:13.370 DEBUG:teuthology.orchestra.run.smithi032:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-08-03T20:58:13.568 INFO:teuthology.orchestra.run.smithi032.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-08-03T20:58:13.568 INFO:teuthology.orchestra.run.smithi032.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-08-03T20:58:13.568 INFO:teuthology.orchestra.run.smithi032.stdout:invalid new device string format 2024-08-03T20:58:13.589 INFO:teuthology.orchestra.run.smithi032.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-08-03T20:58:13.589 INFO:teuthology.orchestra.run.smithi032.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-08-03T20:58:13.589 INFO:teuthology.orchestra.run.smithi032.stdout:invalid new device string format 2024-08-03T20:58:13.589 INFO:teuthology.orchestra.run.smithi032.stdout:mount.ceph: resolved to: "172.21.15.32:3300,172.21.15.118:3300,172.21.15.118:3301" 2024-08-03T20:58:13.589 INFO:teuthology.orchestra.run.smithi032.stdout:mount.ceph: switching to using v1 address with old syntax 2024-08-03T20:58:13.618 INFO:teuthology.orchestra.run.smithi032.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-08-03T20:58:13.618 INFO:teuthology.orchestra.run.smithi032.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-08-03T20:58:13.618 INFO:teuthology.orchestra.run.smithi032.stdout:invalid new device string format 2024-08-03T20:58:13.618 INFO:teuthology.orchestra.run.smithi032.stdout:mount.ceph: resolved to: "172.21.15.32:3300,172.21.15.118:3300,172.21.15.118:3301" 2024-08-03T20:58:13.618 INFO:teuthology.orchestra.run.smithi032.stdout:mount.ceph: switching to using v1 address with old syntax 2024-08-03T20:58:13.618 INFO:teuthology.orchestra.run.smithi032.stdout:mount.ceph: resolved to: "172.21.15.32:6789,172.21.15.118:6789,172.21.15.118:6790" 2024-08-03T20:58:13.618 INFO:teuthology.orchestra.run.smithi032.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.32:6789,172.21.15.118:6789,172.21.15.118:6790:/ 2024-08-03T20:58:13.618 INFO:teuthology.orchestra.run.smithi032.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,key=0,fsid=40ab141a-01d7-49b9-ae3c-2edec9876e95" will pass to kernel 2024-08-03T20:58:13.620 INFO:teuthology.orchestra.run.smithi032.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2024-08-03T20:58:13.620 INFO:teuthology.orchestra.run.smithi032.stdout: You just mounted a file system that supports labels which does not 2024-08-03T20:58:13.620 INFO:teuthology.orchestra.run.smithi032.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-08-03T20:58:13.620 INFO:teuthology.orchestra.run.smithi032.stdout: applications will generate AVC messages and not be allowed access to 2024-08-03T20:58:13.620 INFO:teuthology.orchestra.run.smithi032.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-08-03T20:58:13.623 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-08-03T20:58:13.623 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-08-03T20:58:13.623 DEBUG:teuthology.orchestra.run.smithi032:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-08-03T20:58:13.678 DEBUG:teuthology.orchestra.run.smithi032:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-08-03T20:58:13.757 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:13 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-08-03T20:58:13.761 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:13 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-08-03T20:58:13.768 INFO:teuthology.orchestra.run.smithi032.stdout:client9279 2024-08-03T20:58:13.769 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-08-03T20:58:13.769 DEBUG:teuthology.orchestra.run.smithi032:> dd if=/proc/self/mounts of=/dev/stdout 2024-08-03T20:58:13.804 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-08-03T20:58:13.804 DEBUG:teuthology.orchestra.run.smithi032:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls -R .) 2024-08-03T20:58:13.848 INFO:teuthology.orchestra.run:waiting for 20 2024-08-03T20:58:13.881 INFO:teuthology.orchestra.run.smithi032.stdout:.: 2024-08-03T20:58:13.881 INFO:teuthology.orchestra.run.smithi032.stdout:subdir 2024-08-03T20:58:13.881 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-03T20:58:13.882 INFO:teuthology.orchestra.run.smithi032.stdout:./subdir: 2024-08-03T20:58:13.882 INFO:teuthology.orchestra.run.smithi032.stdout:sixmegs 2024-08-03T20:58:13.882 INFO:tasks.cephfs.test_damage:Result: Mutation 'Truncate 500.00000000' did not caused DAMAGED state 2024-08-03T20:58:13.883 INFO:tasks.cephfs.test_damage:Applying mutation 'Truncate mds_snaptable' 2024-08-03T20:58:13.883 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-08-03T20:58:13.883 DEBUG:teuthology.orchestra.run.smithi032:> dd if=/proc/self/mounts of=/dev/stdout 2024-08-03T20:58:13.960 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-08-03T20:58:13.960 DEBUG:teuthology.orchestra.run.smithi032:> dd if=/proc/self/mounts of=/dev/stdout 2024-08-03T20:58:14.036 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-08-03T20:58:14.307 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:14.306+0000 7f9690600640 1 -- 172.21.15.32:0/1404382259 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f968806e930 msgr2=0x7f9688144f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:58:14.307 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:14.306+0000 7f9690600640 1 --2- 172.21.15.32:0/1404382259 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f968806e930 0x7f9688144f90 secure :-1 s=READY pgs=1722 cs=0 l=1 rev1=1 crypto rx=0x7f967c005fd0 tx=0x7f967c02a5d0 comp rx=0 tx=0).stop 2024-08-03T20:58:14.307 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:14.306+0000 7f9690600640 1 -- 172.21.15.32:0/1404382259 shutdown_connections 2024-08-03T20:58:14.308 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:14.306+0000 7f9690600640 1 --2- 172.21.15.32:0/1404382259 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f968815fff0 0x7f9688160400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:14.308 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:14.306+0000 7f9690600640 1 --2- 172.21.15.32:0/1404382259 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f968806e930 0x7f9688144f90 unknown :-1 s=CLOSED pgs=1722 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:14.308 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:14.306+0000 7f9690600640 1 -- 172.21.15.32:0/1404382259 >> 172.21.15.32:0/1404382259 conn(0x7f96880b1bd0 msgr2=0x7f96880b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:58:14.308 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:14.306+0000 7f9690600640 1 -- 172.21.15.32:0/1404382259 shutdown_connections 2024-08-03T20:58:14.308 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:14.306+0000 7f9690600640 1 -- 172.21.15.32:0/1404382259 wait complete. 2024-08-03T20:58:14.309 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:14.309+0000 7f9690600640 1 Processor -- start 2024-08-03T20:58:14.309 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:14.309+0000 7f9690600640 1 -- start start 2024-08-03T20:58:14.312 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:14.310+0000 7f9690600640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f968806e930 0x7f968815b240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:58:14.312 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:14.310+0000 7f9690600640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f9688073e60 0x7f96881e5b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:58:14.312 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:14.310+0000 7f9690600640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f968815fff0 0x7f96881e60d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:58:14.312 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:14.310+0000 7f9690600640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f96880cf840 con 0x7f968806e930 2024-08-03T20:58:14.312 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:14.310+0000 7f9690600640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f9688145f30 con 0x7f968815fff0 2024-08-03T20:58:14.313 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:14.310+0000 7f9690600640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7f96880cf9c0 con 0x7f9688073e60 2024-08-03T20:58:14.313 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:14.310+0000 7f968dc00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f968806e930 0x7f968815b240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:58:14.313 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:14.311+0000 7f968dc00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f968806e930 0x7f968815b240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:35852/0 (socket says 172.21.15.32:35852) 2024-08-03T20:58:14.313 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:14.311+0000 7f968dc00640 1 -- 172.21.15.32:0/3786124639 learned_addr learned my addr 172.21.15.32:0/3786124639 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:58:14.313 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:14.311+0000 7f968d200640 1 --2- 172.21.15.32:0/3786124639 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f9688073e60 0x7f96881e5b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:58:14.313 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:14.311+0000 7f968e600640 1 --2- 172.21.15.32:0/3786124639 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f968815fff0 0x7f96881e60d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:58:14.313 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:14.311+0000 7f968dc00640 1 -- 172.21.15.32:0/3786124639 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f9688073e60 msgr2=0x7f96881e5b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:58:14.313 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:14.311+0000 7f968dc00640 1 --2- 172.21.15.32:0/3786124639 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f9688073e60 0x7f96881e5b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:14.313 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:14.311+0000 7f968dc00640 1 -- 172.21.15.32:0/3786124639 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f968815fff0 msgr2=0x7f96881e60d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:58:14.313 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:14.311+0000 7f968dc00640 1 --2- 172.21.15.32:0/3786124639 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f968815fff0 0x7f96881e60d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:14.313 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:14.311+0000 7f968dc00640 1 -- 172.21.15.32:0/3786124639 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f968815da40 con 0x7f968806e930 2024-08-03T20:58:14.313 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:14.311+0000 7f968d200640 1 --2- 172.21.15.32:0/3786124639 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f9688073e60 0x7f96881e5b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-03T20:58:14.314 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:14.311+0000 7f968e600640 1 --2- 172.21.15.32:0/3786124639 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f968815fff0 0x7f96881e60d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-03T20:58:14.314 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:14.311+0000 7f968dc00640 1 --2- 172.21.15.32:0/3786124639 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f968806e930 0x7f968815b240 secure :-1 s=READY pgs=1723 cs=0 l=1 rev1=1 crypto rx=0x7f967c034040 tx=0x7f967c0038a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:58:14.314 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:14.311+0000 7f9685600640 1 -- 172.21.15.32:0/3786124639 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f967c03e030 con 0x7f968806e930 2024-08-03T20:58:14.314 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:14.312+0000 7f9685600640 1 -- 172.21.15.32:0/3786124639 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f967c003ca0 con 0x7f968806e930 2024-08-03T20:58:14.314 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:14.312+0000 7f9690600640 1 -- 172.21.15.32:0/3786124639 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f968815dd20 con 0x7f968806e930 2024-08-03T20:58:14.314 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:14.312+0000 7f9685600640 1 -- 172.21.15.32:0/3786124639 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f967c035040 con 0x7f968806e930 2024-08-03T20:58:14.314 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:14.312+0000 7f9690600640 1 -- 172.21.15.32:0/3786124639 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f968815e2e0 con 0x7f968806e930 2024-08-03T20:58:14.315 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:14.313+0000 7f9690600640 1 -- 172.21.15.32:0/3786124639 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9688160c70 con 0x7f968806e930 2024-08-03T20:58:14.315 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:14.314+0000 7f9685600640 1 -- 172.21.15.32:0/3786124639 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f967c041030 con 0x7f968806e930 2024-08-03T20:58:14.315 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:14.315+0000 7f9685600640 1 --2- 172.21.15.32:0/3786124639 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f9668077010 0x7f9668079520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:58:14.315 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:14.315+0000 7f9685600640 1 -- 172.21.15.32:0/3786124639 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(111..111 src has 1..111) ==== 8090+0+0 (secure 0 0 0) 0x7f967c0b9820 con 0x7f968806e930 2024-08-03T20:58:14.316 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:14.315+0000 7f968d200640 1 --2- 172.21.15.32:0/3786124639 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f9668077010 0x7f9668079520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:58:14.316 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:14.316+0000 7f968d200640 1 --2- 172.21.15.32:0/3786124639 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f9668077010 0x7f9668079520 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 crypto rx=0x7f96880cd170 tx=0x7f9674009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:58:14.320 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:14.319+0000 7f9685600640 1 -- 172.21.15.32:0/3786124639 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7f967c082030 con 0x7f968806e930 2024-08-03T20:58:14.541 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:14.540+0000 7f9690600640 1 -- 172.21.15.32:0/3786124639 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f968806fe60 con 0x7f968806e930 2024-08-03T20:58:14.541 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:14.541+0000 7f9685600640 1 -- 172.21.15.32:0/3786124639 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 52 entries v111) ==== 81+0+3162 (secure 0 0 0) 0x7f967c08a300 con 0x7f968806e930 2024-08-03T20:58:14.542 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6835/675070591 2024-08-04T20:58:00.068114+0000 2024-08-03T20:58:14.542 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6833/675070591 2024-08-04T20:58:00.068114+0000 2024-08-03T20:58:14.542 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6835/2600386707 2024-08-04T20:57:50.063964+0000 2024-08-03T20:58:14.542 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6835/511221267 2024-08-04T20:57:45.062448+0000 2024-08-03T20:58:14.542 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6835/1474984145 2024-08-04T20:57:40.052808+0000 2024-08-03T20:58:14.542 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6835/2283924374 2024-08-04T20:57:34.396782+0000 2024-08-03T20:58:14.542 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6835/2274054575 2024-08-04T20:57:55.062490+0000 2024-08-03T20:58:14.542 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6835/265163622 2024-08-04T20:57:28.354509+0000 2024-08-03T20:58:14.542 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6833/265163622 2024-08-04T20:57:28.354509+0000 2024-08-03T20:58:14.542 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6833/2345472481 2024-08-04T20:57:25.968590+0000 2024-08-03T20:58:14.543 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6835/1885034124 2024-08-04T20:57:08.190707+0000 2024-08-03T20:58:14.543 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6833/511221267 2024-08-04T20:57:45.062448+0000 2024-08-03T20:58:14.543 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6833/1885034124 2024-08-04T20:57:08.190707+0000 2024-08-03T20:58:14.543 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6835/2362145329 2024-08-04T20:57:02.150199+0000 2024-08-03T20:58:14.543 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6833/2362145329 2024-08-04T20:57:02.150199+0000 2024-08-03T20:58:14.543 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6835/739874257 2024-08-04T20:56:56.101271+0000 2024-08-03T20:58:14.543 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6833/739874257 2024-08-04T20:56:56.101271+0000 2024-08-03T20:58:14.543 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6835/2242533881 2024-08-04T20:56:48.048099+0000 2024-08-03T20:58:14.543 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6833/2274054575 2024-08-04T20:57:55.062490+0000 2024-08-03T20:58:14.543 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6833/1172050551 2024-08-04T20:55:26.214113+0000 2024-08-03T20:58:14.543 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6833/2806168786 2024-08-04T20:56:37.279160+0000 2024-08-03T20:58:14.543 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6835/2292690833 2024-08-04T20:55:20.021329+0000 2024-08-03T20:58:14.543 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6835/3431300225 2024-08-04T20:56:21.088927+0000 2024-08-03T20:58:14.543 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6835/2806168786 2024-08-04T20:56:37.279160+0000 2024-08-03T20:58:14.544 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6835/756683988 2024-08-04T20:55:14.119228+0000 2024-08-03T20:58:14.544 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6833/1474984145 2024-08-04T20:57:40.052808+0000 2024-08-03T20:58:14.544 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6833/756683988 2024-08-04T20:55:14.119228+0000 2024-08-03T20:58:14.544 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6835/1016338829 2024-08-04T20:55:06.071485+0000 2024-08-03T20:58:14.544 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6835/1172050551 2024-08-04T20:55:26.214113+0000 2024-08-03T20:58:14.544 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6833/2194902489 2024-08-04T20:55:00.359126+0000 2024-08-03T20:58:14.544 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6833/2283924374 2024-08-04T20:57:34.396782+0000 2024-08-03T20:58:14.544 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6833/3431300225 2024-08-04T20:56:21.088927+0000 2024-08-03T20:58:14.544 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6833/2242533881 2024-08-04T20:56:48.048099+0000 2024-08-03T20:58:14.544 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6833/1707454554 2024-08-04T20:54:38.409411+0000 2024-08-03T20:58:14.544 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6835/3191654276 2024-08-04T20:55:42.433364+0000 2024-08-03T20:58:14.544 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6833/2803901497 2024-08-04T20:55:55.181144+0000 2024-08-03T20:58:14.544 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6835/2194902489 2024-08-04T20:55:00.359126+0000 2024-08-03T20:58:14.544 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6835/2803901497 2024-08-04T20:55:55.181144+0000 2024-08-03T20:58:14.545 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6833/2292690833 2024-08-04T20:55:20.021329+0000 2024-08-03T20:58:14.545 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6835/1707454554 2024-08-04T20:54:38.409411+0000 2024-08-03T20:58:14.545 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6835/4172198298 2024-08-04T20:56:04.256182+0000 2024-08-03T20:58:14.545 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6835/3771304550 2024-08-04T20:55:58.212752+0000 2024-08-03T20:58:14.545 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6833/3191654276 2024-08-04T20:55:42.433364+0000 2024-08-03T20:58:14.545 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6835/2178506907 2024-08-04T20:56:40.045549+0000 2024-08-03T20:58:14.545 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6833/2600386707 2024-08-04T20:57:50.063964+0000 2024-08-03T20:58:14.545 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6833/3771304550 2024-08-04T20:55:58.212752+0000 2024-08-03T20:58:14.545 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6833/4172198298 2024-08-04T20:56:04.256182+0000 2024-08-03T20:58:14.545 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6835/2345472481 2024-08-04T20:57:25.968590+0000 2024-08-03T20:58:14.545 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6833/2278473729 2024-08-04T20:56:13.041693+0000 2024-08-03T20:58:14.545 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6833/1016338829 2024-08-04T20:55:06.071485+0000 2024-08-03T20:58:14.545 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6835/2278473729 2024-08-04T20:56:13.041693+0000 2024-08-03T20:58:14.546 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6833/2178506907 2024-08-04T20:56:40.045549+0000 2024-08-03T20:58:14.546 INFO:teuthology.orchestra.run.smithi032.stderr:listed 52 entries 2024-08-03T20:58:14.547 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:14.543+0000 7f9690600640 1 -- 172.21.15.32:0/3786124639 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f9668077010 msgr2=0x7f9668079520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:58:14.547 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:14.543+0000 7f9690600640 1 --2- 172.21.15.32:0/3786124639 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f9668077010 0x7f9668079520 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 crypto rx=0x7f96880cd170 tx=0x7f9674009040 comp rx=0 tx=0).stop 2024-08-03T20:58:14.547 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:14.543+0000 7f9690600640 1 -- 172.21.15.32:0/3786124639 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f968806e930 msgr2=0x7f968815b240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:58:14.547 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:14.543+0000 7f9690600640 1 --2- 172.21.15.32:0/3786124639 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f968806e930 0x7f968815b240 secure :-1 s=READY pgs=1723 cs=0 l=1 rev1=1 crypto rx=0x7f967c034040 tx=0x7f967c0038a0 comp rx=0 tx=0).stop 2024-08-03T20:58:14.547 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:14.543+0000 7f9690600640 1 -- 172.21.15.32:0/3786124639 shutdown_connections 2024-08-03T20:58:14.547 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:14.543+0000 7f9690600640 1 --2- 172.21.15.32:0/3786124639 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f9668077010 0x7f9668079520 unknown :-1 s=CLOSED pgs=701 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:14.547 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:14.543+0000 7f9690600640 1 --2- 172.21.15.32:0/3786124639 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f968815fff0 0x7f96881e60d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:14.547 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:14.543+0000 7f9690600640 1 --2- 172.21.15.32:0/3786124639 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f9688073e60 0x7f96881e5b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:14.547 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:14.543+0000 7f9690600640 1 --2- 172.21.15.32:0/3786124639 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f968806e930 0x7f968815b240 unknown :-1 s=CLOSED pgs=1723 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:14.547 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:14.544+0000 7f9690600640 1 -- 172.21.15.32:0/3786124639 >> 172.21.15.32:0/3786124639 conn(0x7f96880b1bd0 msgr2=0x7f96880b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:58:14.547 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:14.544+0000 7f9690600640 1 -- 172.21.15.32:0/3786124639 shutdown_connections 2024-08-03T20:58:14.548 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:14.544+0000 7f9690600640 1 -- 172.21.15.32:0/3786124639 wait complete. 2024-08-03T20:58:14.569 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-08-03T20:58:14.569 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-08-03T20:58:14.569 DEBUG:teuthology.orchestra.run.smithi032:> sudo umount /home/ubuntu/cephtest/mnt.0 -f 2024-08-03T20:58:14.649 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi032.front.sepia.ceph.com 2024-08-03T20:58:14.649 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-08-03T20:58:14.650 DEBUG:teuthology.orchestra.run.smithi032:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-08-03T20:58:14.686 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-08-03T20:58:14.956 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:14.955+0000 7f1764e00640 1 -- 172.21.15.32:0/4241561341 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f176015fff0 msgr2=0x7f1760160400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:58:14.956 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:14.955+0000 7f1764e00640 1 --2- 172.21.15.32:0/4241561341 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f176015fff0 0x7f1760160400 secure :-1 s=READY pgs=1724 cs=0 l=1 rev1=1 crypto rx=0x7f17600afaf0 tx=0x7f175802aac0 comp rx=0 tx=0).stop 2024-08-03T20:58:14.956 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:14.955+0000 7f1764e00640 1 -- 172.21.15.32:0/4241561341 shutdown_connections 2024-08-03T20:58:14.956 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:14.955+0000 7f1764e00640 1 --2- 172.21.15.32:0/4241561341 >> v2:172.21.15.118:3300/0 conn(0x7f1760144f90 0x7f1760074570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:14.956 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:14.955+0000 7f1764e00640 1 --2- 172.21.15.32:0/4241561341 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f176015fff0 0x7f1760160400 unknown :-1 s=CLOSED pgs=1724 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:14.956 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:14.955+0000 7f1764e00640 1 -- 172.21.15.32:0/4241561341 >> 172.21.15.32:0/4241561341 conn(0x7f17600b1bd0 msgr2=0x7f17600b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:58:14.957 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:14.955+0000 7f1764e00640 1 -- 172.21.15.32:0/4241561341 shutdown_connections 2024-08-03T20:58:14.957 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:14.955+0000 7f1764e00640 1 -- 172.21.15.32:0/4241561341 wait complete. 2024-08-03T20:58:14.958 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:14.957+0000 7f1764e00640 1 Processor -- start 2024-08-03T20:58:14.958 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:14.958+0000 7f1764e00640 1 -- start start 2024-08-03T20:58:14.961 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:14.959+0000 7f1764e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f176006e930 0x7f17601f97d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:58:14.961 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:14.959+0000 7f1764e00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f1760144f90 0x7f176013f730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:58:14.961 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:14.959+0000 7f1764e00640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f176015fff0 0x7f17600ccd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:58:14.961 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:14.959+0000 7f1764e00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f17600cf840 con 0x7f176006e930 2024-08-03T20:58:14.961 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:14.959+0000 7f1764e00640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f1760145f30 con 0x7f1760144f90 2024-08-03T20:58:14.962 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:14.959+0000 7f1764e00640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7f17600cf9c0 con 0x7f176015fff0 2024-08-03T20:58:14.962 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:14.959+0000 7f175de00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f176006e930 0x7f17601f97d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:58:14.962 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:14.959+0000 7f175de00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f176006e930 0x7f17601f97d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:35878/0 (socket says 172.21.15.32:35878) 2024-08-03T20:58:14.962 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:14.959+0000 7f175de00640 1 -- 172.21.15.32:0/831215645 learned_addr learned my addr 172.21.15.32:0/831215645 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:58:14.962 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:14.959+0000 7f1757e00640 1 --2- 172.21.15.32:0/831215645 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f1760144f90 0x7f176013f730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:58:14.962 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:14.959+0000 7f175e800640 1 --2- 172.21.15.32:0/831215645 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f176015fff0 0x7f17600ccd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:58:14.962 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:14.960+0000 7f175de00640 1 -- 172.21.15.32:0/831215645 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f176015fff0 msgr2=0x7f17600ccd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:58:14.962 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:14.960+0000 7f175de00640 1 --2- 172.21.15.32:0/831215645 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f176015fff0 0x7f17600ccd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:14.962 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:14.960+0000 7f175de00640 1 -- 172.21.15.32:0/831215645 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f1760144f90 msgr2=0x7f176013f730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:58:14.962 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:14.960+0000 7f175de00640 1 --2- 172.21.15.32:0/831215645 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f1760144f90 0x7f176013f730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:14.962 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:14.960+0000 7f175de00640 1 -- 172.21.15.32:0/831215645 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f176015f4e0 con 0x7f176006e930 2024-08-03T20:58:14.962 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:14.960+0000 7f175de00640 1 --2- 172.21.15.32:0/831215645 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f176006e930 0x7f17601f97d0 secure :-1 s=READY pgs=1725 cs=0 l=1 rev1=1 crypto rx=0x7f175000c3d0 tx=0x7f175000c890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:58:14.963 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:14.960+0000 7f1756000640 1 -- 172.21.15.32:0/831215645 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1750017070 con 0x7f176006e930 2024-08-03T20:58:14.963 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:14.960+0000 7f1756000640 1 -- 172.21.15.32:0/831215645 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f17500053c0 con 0x7f176006e930 2024-08-03T20:58:14.963 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:14.960+0000 7f1764e00640 1 -- 172.21.15.32:0/831215645 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f17600ab390 con 0x7f176006e930 2024-08-03T20:58:14.963 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:14.960+0000 7f1756000640 1 -- 172.21.15.32:0/831215645 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1750003850 con 0x7f176006e930 2024-08-03T20:58:14.963 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:14.961+0000 7f1764e00640 1 -- 172.21.15.32:0/831215645 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f17600a9550 con 0x7f176006e930 2024-08-03T20:58:14.963 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:14.961+0000 7f1764e00640 1 -- 172.21.15.32:0/831215645 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f17600ce910 con 0x7f176006e930 2024-08-03T20:58:14.964 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:14.963+0000 7f1756000640 1 -- 172.21.15.32:0/831215645 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f175001a070 con 0x7f176006e930 2024-08-03T20:58:14.964 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:14.964+0000 7f1756000640 1 --2- 172.21.15.32:0/831215645 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f173c077040 0x7f173c079550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:58:14.964 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:14.964+0000 7f1756000640 1 -- 172.21.15.32:0/831215645 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(111..111 src has 1..111) ==== 8090+0+0 (secure 0 0 0) 0x7f17500986a0 con 0x7f176006e930 2024-08-03T20:58:14.965 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:14.965+0000 7f1757e00640 1 --2- 172.21.15.32:0/831215645 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f173c077040 0x7f173c079550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:58:14.966 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:14.965+0000 7f1757e00640 1 --2- 172.21.15.32:0/831215645 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f173c077040 0x7f173c079550 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7f17600cd890 tx=0x7f1748005f90 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:58:14.968 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:14.968+0000 7f1756000640 1 -- 172.21.15.32:0/831215645 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7f17600ce910 con 0x7f176006e930 2024-08-03T20:58:15.214 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:15.213+0000 7f1764e00640 1 -- 172.21.15.32:0/831215645 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7f1760160cc0 con 0x7f176006e930 2024-08-03T20:58:15.214 INFO:tasks.ceph.mon.a.smithi032.stderr:2024-08-03T20:58:15.214+0000 7f4e6d600640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-08-03T20:58:15.216 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:15.216+0000 7f1756000640 1 -- 172.21.15.32:0/831215645 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v245) ==== 161+0+0 (secure 0 0 0) 0x7f1750069180 con 0x7f176006e930 2024-08-03T20:58:15.217 INFO:teuthology.orchestra.run.smithi032.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-08-03T20:58:15.221 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:15.219+0000 7f1764e00640 1 -- 172.21.15.32:0/831215645 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f173c077040 msgr2=0x7f173c079550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:58:15.222 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:15.219+0000 7f1764e00640 1 --2- 172.21.15.32:0/831215645 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f173c077040 0x7f173c079550 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7f17600cd890 tx=0x7f1748005f90 comp rx=0 tx=0).stop 2024-08-03T20:58:15.222 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:15.219+0000 7f1764e00640 1 -- 172.21.15.32:0/831215645 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f176006e930 msgr2=0x7f17601f97d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:58:15.222 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:15.219+0000 7f1764e00640 1 --2- 172.21.15.32:0/831215645 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f176006e930 0x7f17601f97d0 secure :-1 s=READY pgs=1725 cs=0 l=1 rev1=1 crypto rx=0x7f175000c3d0 tx=0x7f175000c890 comp rx=0 tx=0).stop 2024-08-03T20:58:15.222 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:15.219+0000 7f1764e00640 1 -- 172.21.15.32:0/831215645 shutdown_connections 2024-08-03T20:58:15.222 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:15.219+0000 7f1764e00640 1 --2- 172.21.15.32:0/831215645 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f173c077040 0x7f173c079550 unknown :-1 s=CLOSED pgs=702 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:15.222 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:15.219+0000 7f1764e00640 1 --2- 172.21.15.32:0/831215645 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f1760144f90 0x7f176013f730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:15.222 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:15.219+0000 7f1764e00640 1 --2- 172.21.15.32:0/831215645 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f176015fff0 0x7f17600ccd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:15.222 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:15.219+0000 7f1764e00640 1 --2- 172.21.15.32:0/831215645 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f176006e930 0x7f17601f97d0 unknown :-1 s=CLOSED pgs=1725 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:15.222 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:15.219+0000 7f1764e00640 1 -- 172.21.15.32:0/831215645 >> 172.21.15.32:0/831215645 conn(0x7f17600b1bd0 msgr2=0x7f17600b4460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:58:15.222 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:15.220+0000 7f1764e00640 1 -- 172.21.15.32:0/831215645 shutdown_connections 2024-08-03T20:58:15.223 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:15.220+0000 7f1764e00640 1 -- 172.21.15.32:0/831215645 wait complete. 2024-08-03T20:58:15.224 INFO:tasks.ceph.mds.a.smithi032.stderr: -703> 2024-08-03T20:58:10.060+0000 7f2661800640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-08-03T20:58:15.242 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mds repaired 0 2024-08-03T20:58:15.342 INFO:tasks.ceph.mds.a.smithi032.stdout:starting mds.a at 2024-08-03T20:58:15.475 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:15.474+0000 7f438be00640 1 -- 172.21.15.32:0/2520066138 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f438c06f660 msgr2=0x7f438c06fa70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:58:15.476 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:15.474+0000 7f438be00640 1 --2- 172.21.15.32:0/2520066138 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f438c06f660 0x7f438c06fa70 secure :-1 s=READY pgs=1728 cs=0 l=1 rev1=1 crypto rx=0x7f4384005fd0 tx=0x7f438402aa10 comp rx=0 tx=0).stop 2024-08-03T20:58:15.476 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:15.474+0000 7f438be00640 1 -- 172.21.15.32:0/2520066138 shutdown_connections 2024-08-03T20:58:15.476 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:15.474+0000 7f438be00640 1 --2- 172.21.15.32:0/2520066138 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f438c06f660 0x7f438c06fa70 unknown :-1 s=CLOSED pgs=1728 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:15.476 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:15.474+0000 7f438be00640 1 --2- 172.21.15.32:0/2520066138 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f438c071150 0x7f438c070590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:15.476 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:15.474+0000 7f438be00640 1 -- 172.21.15.32:0/2520066138 >> 172.21.15.32:0/2520066138 conn(0x7f438c0d0dd0 msgr2=0x7f438c0d11e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:58:15.476 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:15.474+0000 7f438be00640 1 -- 172.21.15.32:0/2520066138 shutdown_connections 2024-08-03T20:58:15.477 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:15.474+0000 7f438be00640 1 -- 172.21.15.32:0/2520066138 wait complete. 2024-08-03T20:58:15.479 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:15.476+0000 7f438be00640 1 Processor -- start 2024-08-03T20:58:15.479 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:15.476+0000 7f438be00640 1 -- start start 2024-08-03T20:58:15.479 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:15.478+0000 7f438be00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f438c06f660 0x7f438c152700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:58:15.479 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:15.478+0000 7f438be00640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f438c071150 0x7f438c0b3140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:58:15.479 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:15.478+0000 7f438be00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f438c15b790 0x7f438c1596f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:58:15.479 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:15.478+0000 7f438be00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f438c0736b0 con 0x7f438c15b790 2024-08-03T20:58:15.480 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:15.478+0000 7f438be00640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f438c15fce0 con 0x7f438c06f660 2024-08-03T20:58:15.480 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:15.478+0000 7f438be00640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7f438c073830 con 0x7f438c071150 2024-08-03T20:58:15.480 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:15.478+0000 7f438b400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f438c15b790 0x7f438c1596f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:58:15.480 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:15.478+0000 7f438b400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f438c15b790 0x7f438c1596f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:35900/0 (socket says 172.21.15.32:35900) 2024-08-03T20:58:15.480 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:15.478+0000 7f438b400640 1 -- 172.21.15.32:0/3388318076 learned_addr learned my addr 172.21.15.32:0/3388318076 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:58:15.480 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:15.478+0000 7f4383e00640 1 --2- 172.21.15.32:0/3388318076 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f438c071150 0x7f438c0b3140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:58:15.480 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:15.478+0000 7f438aa00640 1 --2- 172.21.15.32:0/3388318076 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f438c06f660 0x7f438c152700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:58:15.480 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:15.479+0000 7f438b400640 1 -- 172.21.15.32:0/3388318076 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f438c071150 msgr2=0x7f438c0b3140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:58:15.480 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:15.479+0000 7f438b400640 1 --2- 172.21.15.32:0/3388318076 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f438c071150 0x7f438c0b3140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:15.480 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:15.479+0000 7f438b400640 1 -- 172.21.15.32:0/3388318076 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f438c06f660 msgr2=0x7f438c152700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:58:15.480 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:15.479+0000 7f438b400640 1 --2- 172.21.15.32:0/3388318076 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f438c06f660 0x7f438c152700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:15.480 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:15.479+0000 7f438b400640 1 -- 172.21.15.32:0/3388318076 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f438c0b3680 con 0x7f438c15b790 2024-08-03T20:58:15.480 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:15.479+0000 7f4383e00640 1 --2- 172.21.15.32:0/3388318076 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f438c071150 0x7f438c0b3140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-03T20:58:15.481 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:15.479+0000 7f438b400640 1 --2- 172.21.15.32:0/3388318076 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f438c15b790 0x7f438c1596f0 secure :-1 s=READY pgs=1729 cs=0 l=1 rev1=1 crypto rx=0x7f4384006010 tx=0x7f4384003f10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:58:15.481 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:15.479+0000 7f4382a00640 1 -- 172.21.15.32:0/3388318076 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4384046070 con 0x7f438c15b790 2024-08-03T20:58:15.482 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:15.479+0000 7f4382a00640 1 -- 172.21.15.32:0/3388318076 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f4384004540 con 0x7f438c15b790 2024-08-03T20:58:15.482 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:15.479+0000 7f438be00640 1 -- 172.21.15.32:0/3388318076 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f438c0c4c30 con 0x7f438c15b790 2024-08-03T20:58:15.482 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:15.479+0000 7f4382a00640 1 -- 172.21.15.32:0/3388318076 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4384033070 con 0x7f438c15b790 2024-08-03T20:58:15.482 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:15.479+0000 7f438be00640 1 -- 172.21.15.32:0/3388318076 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f438c0c51f0 con 0x7f438c15b790 2024-08-03T20:58:15.482 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:15.480+0000 7f438be00640 1 -- 172.21.15.32:0/3388318076 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f438c073ce0 con 0x7f438c15b790 2024-08-03T20:58:15.482 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:15.482+0000 7f4382a00640 1 -- 172.21.15.32:0/3388318076 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f4384049070 con 0x7f438c15b790 2024-08-03T20:58:15.483 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:15.482+0000 7f4382a00640 1 --2- 172.21.15.32:0/3388318076 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f4368077040 0x7f4368079550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:58:15.483 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:15.483+0000 7f4382a00640 1 -- 172.21.15.32:0/3388318076 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(112..112 src has 1..112) ==== 8176+0+0 (secure 0 0 0) 0x7f43840ba2c0 con 0x7f438c15b790 2024-08-03T20:58:15.483 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:15.483+0000 7f438aa00640 1 --2- 172.21.15.32:0/3388318076 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f4368077040 0x7f4368079550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:58:15.484 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:15.484+0000 7f438aa00640 1 --2- 172.21.15.32:0/3388318076 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f4368077040 0x7f4368079550 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 crypto rx=0x7f437c009ac0 tx=0x7f437c005f90 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:58:15.487 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:15.487+0000 7f4382a00640 1 -- 172.21.15.32:0/3388318076 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7f438403e070 con 0x7f438c15b790 2024-08-03T20:58:15.775 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:15.774+0000 7f438be00640 1 -- 172.21.15.32:0/3388318076 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "mds repaired", "role": "0"} v 0) -- 0x7f438c0b6d50 con 0x7f438c15b790 2024-08-03T20:58:16.219 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:16.218+0000 7f4382a00640 1 -- 172.21.15.32:0/3388318076 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "mds repaired", "role": "0"}]=0 nothing to do: rank is not damaged v246) ==== 107+0+0 (secure 0 0 0) 0x7f4384082aa0 con 0x7f438c15b790 2024-08-03T20:58:16.219 INFO:teuthology.orchestra.run.smithi032.stderr:nothing to do: rank is not damaged 2024-08-03T20:58:16.223 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:16.222+0000 7f438be00640 1 -- 172.21.15.32:0/3388318076 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f4368077040 msgr2=0x7f4368079550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:58:16.223 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:16.222+0000 7f438be00640 1 --2- 172.21.15.32:0/3388318076 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f4368077040 0x7f4368079550 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 crypto rx=0x7f437c009ac0 tx=0x7f437c005f90 comp rx=0 tx=0).stop 2024-08-03T20:58:16.223 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:16.222+0000 7f438be00640 1 -- 172.21.15.32:0/3388318076 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f438c15b790 msgr2=0x7f438c1596f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:58:16.223 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:16.222+0000 7f438be00640 1 --2- 172.21.15.32:0/3388318076 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f438c15b790 0x7f438c1596f0 secure :-1 s=READY pgs=1729 cs=0 l=1 rev1=1 crypto rx=0x7f4384006010 tx=0x7f4384003f10 comp rx=0 tx=0).stop 2024-08-03T20:58:16.223 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:16.222+0000 7f438be00640 1 -- 172.21.15.32:0/3388318076 shutdown_connections 2024-08-03T20:58:16.223 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:16.222+0000 7f438be00640 1 --2- 172.21.15.32:0/3388318076 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f4368077040 0x7f4368079550 unknown :-1 s=CLOSED pgs=703 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:16.223 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:16.222+0000 7f438be00640 1 --2- 172.21.15.32:0/3388318076 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f438c15b790 0x7f438c1596f0 unknown :-1 s=CLOSED pgs=1729 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:16.223 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:16.222+0000 7f438be00640 1 --2- 172.21.15.32:0/3388318076 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f438c071150 0x7f438c0b3140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:16.223 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:16.222+0000 7f438be00640 1 --2- 172.21.15.32:0/3388318076 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f438c06f660 0x7f438c152700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:16.223 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:16.222+0000 7f438be00640 1 -- 172.21.15.32:0/3388318076 >> 172.21.15.32:0/3388318076 conn(0x7f438c0d0dd0 msgr2=0x7f438c068940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:58:16.224 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:16.222+0000 7f438be00640 1 -- 172.21.15.32:0/3388318076 shutdown_connections 2024-08-03T20:58:16.224 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:16.222+0000 7f438be00640 1 -- 172.21.15.32:0/3388318076 wait complete. 2024-08-03T20:58:16.247 DEBUG:teuthology.orchestra.run.smithi032:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata import - 2024-08-03T20:58:16.390 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:16.390+0000 7f5d5ed1cf40 1 Processor -- start 2024-08-03T20:58:16.391 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:16.390+0000 7f5d5ed1cf40 1 -- start start 2024-08-03T20:58:16.393 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:16.391+0000 7f5d5ed1cf40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55c16d35cb40 0x55c16d3d08a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:58:16.393 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:16.391+0000 7f5d5ed1cf40 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55c16d3cfbe0 0x55c16d3eaf20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:58:16.393 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:16.391+0000 7f5d5ed1cf40 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55c16d3eb460 0x55c16d3eb870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:58:16.393 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:16.391+0000 7f5d5ed1cf40 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55c16d2c3bf0 con 0x55c16d3eb460 2024-08-03T20:58:16.393 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:16.391+0000 7f5d5ed1cf40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x55c16d2d06b0 con 0x55c16d35cb40 2024-08-03T20:58:16.393 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:16.391+0000 7f5d5ed1cf40 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x55c16d265e30 con 0x55c16d3cfbe0 2024-08-03T20:58:16.393 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:16.392+0000 7f5d57e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55c16d3eb460 0x55c16d3eb870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:58:16.393 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:16.392+0000 7f5d57e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55c16d3eb460 0x55c16d3eb870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:35904/0 (socket says 172.21.15.32:35904) 2024-08-03T20:58:16.393 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:16.392+0000 7f5d57e00640 1 -- 172.21.15.32:0/3518750740 learned_addr learned my addr 172.21.15.32:0/3518750740 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:58:16.394 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:16.392+0000 7f5d56a00640 1 --2- 172.21.15.32:0/3518750740 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55c16d3cfbe0 0x55c16d3eaf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:58:16.394 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:16.392+0000 7f5d57400640 1 --2- 172.21.15.32:0/3518750740 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55c16d35cb40 0x55c16d3d08a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:58:16.394 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:16.392+0000 7f5d57e00640 1 -- 172.21.15.32:0/3518750740 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55c16d3cfbe0 msgr2=0x55c16d3eaf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:58:16.394 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:16.392+0000 7f5d57e00640 1 --2- 172.21.15.32:0/3518750740 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55c16d3cfbe0 0x55c16d3eaf20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:16.394 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:16.392+0000 7f5d57e00640 1 -- 172.21.15.32:0/3518750740 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55c16d35cb40 msgr2=0x55c16d3d08a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:58:16.394 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:16.392+0000 7f5d57e00640 1 --2- 172.21.15.32:0/3518750740 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55c16d35cb40 0x55c16d3d08a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:16.394 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:16.392+0000 7f5d57e00640 1 -- 172.21.15.32:0/3518750740 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c16d3601b0 con 0x55c16d3eb460 2024-08-03T20:58:16.394 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:16.392+0000 7f5d57e00640 1 --2- 172.21.15.32:0/3518750740 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55c16d3eb460 0x55c16d3eb870 secure :-1 s=READY pgs=1731 cs=0 l=1 rev1=1 crypto rx=0x7f5d50006f70 tx=0x7f5d50001a80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=27f6f53f6ce30f26 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:58:16.394 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:16.393+0000 7f5d55600640 1 -- 172.21.15.32:0/3518750740 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d50016070 con 0x55c16d3eb460 2024-08-03T20:58:16.394 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:16.393+0000 7f5d55600640 1 -- 172.21.15.32:0/3518750740 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f5d5000f800 con 0x55c16d3eb460 2024-08-03T20:58:16.394 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:16.393+0000 7f5d55600640 1 -- 172.21.15.32:0/3518750740 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d5000fc90 con 0x55c16d3eb460 2024-08-03T20:58:16.395 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:16.393+0000 7f5d5ed1cf40 1 -- 172.21.15.32:0/3518750740 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55c16d3eb460 msgr2=0x55c16d3eb870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:58:16.395 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:16.393+0000 7f5d5ed1cf40 1 --2- 172.21.15.32:0/3518750740 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55c16d3eb460 0x55c16d3eb870 secure :-1 s=READY pgs=1731 cs=0 l=1 rev1=1 crypto rx=0x7f5d50006f70 tx=0x7f5d50001a80 comp rx=0 tx=0).stop 2024-08-03T20:58:16.396 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:16.393+0000 7f5d5ed1cf40 1 -- 172.21.15.32:0/3518750740 shutdown_connections 2024-08-03T20:58:16.396 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:16.393+0000 7f5d5ed1cf40 1 --2- 172.21.15.32:0/3518750740 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55c16d3eb460 0x55c16d3eb870 unknown :-1 s=CLOSED pgs=1731 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:16.396 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:16.393+0000 7f5d5ed1cf40 1 --2- 172.21.15.32:0/3518750740 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55c16d3cfbe0 0x55c16d3eaf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:16.396 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:16.393+0000 7f5d5ed1cf40 1 --2- 172.21.15.32:0/3518750740 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55c16d35cb40 0x55c16d3d08a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:16.396 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:16.393+0000 7f5d5ed1cf40 1 -- 172.21.15.32:0/3518750740 >> 172.21.15.32:0/3518750740 conn(0x55c16d237ac0 msgr2=0x55c16d3177e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:58:16.396 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:16.393+0000 7f5d5ed1cf40 1 -- 172.21.15.32:0/3518750740 shutdown_connections 2024-08-03T20:58:16.396 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:16.393+0000 7f5d5ed1cf40 1 -- 172.21.15.32:0/3518750740 wait complete. 2024-08-03T20:58:16.396 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:16.394+0000 7f5d5ed1cf40 1 Processor -- start 2024-08-03T20:58:16.396 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:16.395+0000 7f5d5ed1cf40 1 -- start start 2024-08-03T20:58:16.398 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:16.396+0000 7f5d5ed1cf40 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55c16d35cb40 0x55c16d3ea6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:58:16.398 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:16.396+0000 7f5d5ed1cf40 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55c16d3cfbe0 0x55c16d3ca730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:58:16.398 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:16.396+0000 7f5d5ed1cf40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55c16d3eb460 0x55c16d3c7070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:58:16.398 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:16.396+0000 7f5d5ed1cf40 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55c16d3ebdb0 con 0x55c16d3cfbe0 2024-08-03T20:58:16.398 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:16.396+0000 7f5d5ed1cf40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x55c16d3cac70 con 0x55c16d3eb460 2024-08-03T20:58:16.398 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:16.396+0000 7f5d5ed1cf40 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x55c16d3b6920 con 0x55c16d35cb40 2024-08-03T20:58:16.399 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:16.396+0000 7f5d56a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55c16d3cfbe0 0x55c16d3ca730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:58:16.399 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:16.396+0000 7f5d56a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55c16d3cfbe0 0x55c16d3ca730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:35920/0 (socket says 172.21.15.32:35920) 2024-08-03T20:58:16.399 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:16.396+0000 7f5d56a00640 1 -- 172.21.15.32:0/92850940 learned_addr learned my addr 172.21.15.32:0/92850940 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:58:16.399 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:16.396+0000 7f5d57e00640 1 --2- 172.21.15.32:0/92850940 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55c16d3eb460 0x55c16d3c7070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:58:16.399 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:16.396+0000 7f5d57400640 1 --2- 172.21.15.32:0/92850940 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55c16d35cb40 0x55c16d3ea6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:58:16.399 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:16.397+0000 7f5d56a00640 1 -- 172.21.15.32:0/92850940 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55c16d35cb40 msgr2=0x55c16d3ea6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:58:16.399 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:16.397+0000 7f5d56a00640 1 --2- 172.21.15.32:0/92850940 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55c16d35cb40 0x55c16d3ea6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:16.399 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:16.397+0000 7f5d56a00640 1 -- 172.21.15.32:0/92850940 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55c16d3eb460 msgr2=0x55c16d3c7070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:58:16.399 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:16.397+0000 7f5d56a00640 1 --2- 172.21.15.32:0/92850940 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55c16d3eb460 0x55c16d3c7070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:16.399 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:16.397+0000 7f5d56a00640 1 -- 172.21.15.32:0/92850940 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c16d3c75b0 con 0x55c16d3cfbe0 2024-08-03T20:58:16.399 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:16.397+0000 7f5d56a00640 1 --2- 172.21.15.32:0/92850940 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55c16d3cfbe0 0x55c16d3ca730 secure :-1 s=READY pgs=1732 cs=0 l=1 rev1=1 crypto rx=0x7f5d44046560 tx=0x7f5d44004690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:58:16.399 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:16.397+0000 7f5d4fe00640 1 -- 172.21.15.32:0/92850940 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d44056470 con 0x55c16d3cfbe0 2024-08-03T20:58:16.400 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:16.397+0000 7f5d4fe00640 1 -- 172.21.15.32:0/92850940 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f5d44056700 con 0x55c16d3cfbe0 2024-08-03T20:58:16.400 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:16.397+0000 7f5d5ed1cf40 1 -- 172.21.15.32:0/92850940 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x55c16d3d0f80 con 0x55c16d3cfbe0 2024-08-03T20:58:16.400 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:16.398+0000 7f5d4fe00640 1 -- 172.21.15.32:0/92850940 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d44056910 con 0x55c16d3cfbe0 2024-08-03T20:58:16.400 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:16.398+0000 7f5d5ed1cf40 1 -- 172.21.15.32:0/92850940 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c16d3c9c50 con 0x55c16d3cfbe0 2024-08-03T20:58:16.400 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:16.400+0000 7f5d4fe00640 1 -- 172.21.15.32:0/92850940 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f5d44069020 con 0x55c16d3cfbe0 2024-08-03T20:58:16.401 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:16.401+0000 7f5d4fe00640 1 --2- 172.21.15.32:0/92850940 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f5d38075a20 0x7f5d38077f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:58:16.401 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:16.401+0000 7f5d4fe00640 1 -- 172.21.15.32:0/92850940 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(112..112 src has 1..112) ==== 8176+0+0 (secure 0 0 0) 0x7f5d440dbe30 con 0x55c16d3cfbe0 2024-08-03T20:58:16.401 INFO:teuthology.orchestra.run.smithi032.stdout:Importing pool 2024-08-03T20:58:16.402 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:16.401+0000 7f5d57400640 1 --2- 172.21.15.32:0/92850940 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f5d38075a20 0x7f5d38077f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:58:16.402 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:16.401+0000 7f5d5ed1cf40 1 --2- 172.21.15.32:0/92850940 >> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] conn(0x55c16d3b6150 0x55c16d35f4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:58:16.403 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:16.401+0000 7f5d5ed1cf40 1 -- 172.21.15.32:0/92850940 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:1 12.10 12:09401a81:::601.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112) -- 0x55c16d490640 con 0x55c16d3b6150 2024-08-03T20:58:16.403 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:16.402+0000 7f5d57e00640 1 --2- 172.21.15.32:0/92850940 >> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] conn(0x55c16d3b6150 0x55c16d35f4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:58:16.403 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:16.402+0000 7f5d57400640 1 --2- 172.21.15.32:0/92850940 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f5d38075a20 0x7f5d38077f30 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7f5d48000f60 tx=0x7f5d48003050 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:58:16.403 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:16.402+0000 7f5d57e00640 1 --2- 172.21.15.32:0/92850940 >> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] conn(0x55c16d3b6150 0x55c16d35f4a0 crc :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:58:16.405 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:16.404+0000 7f5d57e00640 1 -- 172.21.15.32:0/92850940 <== osd.0 v2:172.21.15.32:6816/3066156116 1 ==== osd_op_reply(1 601.00000000 [create] v112'815 uv789 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f5d50019040 con 0x55c16d3b6150 2024-08-03T20:58:16.405 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:16.405+0000 7f5d5ed1cf40 1 -- 172.21.15.32:0/92850940 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:2 12.10 12:09401a81:::601.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e112) -- 0x55c16d4909e0 con 0x55c16d3b6150 2024-08-03T20:58:16.406 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:16.406+0000 7f5d57e00640 1 -- 172.21.15.32:0/92850940 <== osd.0 v2:172.21.15.32:6816/3066156116 2 ==== osd_op_reply(2 601.00000000 [delete] v112'816 uv816 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f5d50019040 con 0x55c16d3b6150 2024-08-03T20:58:16.407 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:16.406+0000 7f5d5ed1cf40 1 -- 172.21.15.32:0/92850940 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:3 12.10 12:09401a81:::601.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112) -- 0x55c16d490d80 con 0x55c16d3b6150 2024-08-03T20:58:16.408 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::601.00000000:head# 2024-08-03T20:58:16.409 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:16.408+0000 7f5d57e00640 1 -- 172.21.15.32:0/92850940 <== osd.0 v2:172.21.15.32:6816/3066156116 3 ==== osd_op_reply(3 601.00000000 [create] v112'817 uv817 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f5d50019040 con 0x55c16d3b6150 2024-08-03T20:58:16.409 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:16.408+0000 7f5d5ed1cf40 1 -- 172.21.15.32:0/92850940 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:4 12.10 12:09401a81:::601.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112) -- 0x55c16d492130 con 0x55c16d3b6150 2024-08-03T20:58:16.410 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:16.410+0000 7f5d57e00640 1 -- 172.21.15.32:0/92850940 <== osd.0 v2:172.21.15.32:6816/3066156116 4 ==== osd_op_reply(4 601.00000000 [setxattr (30)] v112'818 uv818 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f5d50019040 con 0x55c16d3b6150 2024-08-03T20:58:16.410 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:16.410+0000 7f5d5ed1cf40 1 -- 172.21.15.32:0/92850940 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:5 12.10 12:09401a81:::601.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112) -- 0x55c16d3d09c0 con 0x55c16d3b6150 2024-08-03T20:58:16.412 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:16.411+0000 7f5d57e00640 1 -- 172.21.15.32:0/92850940 <== osd.0 v2:172.21.15.32:6816/3066156116 5 ==== osd_op_reply(5 601.00000000 [setxattr (62)] v112'819 uv819 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f5d50019040 con 0x55c16d3b6150 2024-08-03T20:58:16.412 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:16.411+0000 7f5d5ed1cf40 1 -- 172.21.15.32:0/92850940 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:6 12.10 12:09401a81:::601.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112) -- 0x55c16d491120 con 0x55c16d3b6150 2024-08-03T20:58:16.413 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:16.413+0000 7f5d57e00640 1 -- 172.21.15.32:0/92850940 <== osd.0 v2:172.21.15.32:6816/3066156116 6 ==== osd_op_reply(6 601.00000000 [omap-set-header] v112'820 uv820 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f5d50019040 con 0x55c16d3b6150 2024-08-03T20:58:16.413 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:16.413+0000 7f5d5ed1cf40 1 -- 172.21.15.32:0/92850940 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:7 12.10 12:0bd6d154:::602.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112) -- 0x55c16d4914c0 con 0x55c16d3b6150 2024-08-03T20:58:16.415 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:16.414+0000 7f5d57e00640 1 -- 172.21.15.32:0/92850940 <== osd.0 v2:172.21.15.32:6816/3066156116 7 ==== osd_op_reply(7 602.00000000 [create] v112'821 uv795 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f5d50019040 con 0x55c16d3b6150 2024-08-03T20:58:16.415 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:16.415+0000 7f5d5ed1cf40 1 -- 172.21.15.32:0/92850940 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:8 12.10 12:0bd6d154:::602.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e112) -- 0x55c16d491860 con 0x55c16d3b6150 2024-08-03T20:58:16.417 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:16.416+0000 7f5d57e00640 1 -- 172.21.15.32:0/92850940 <== osd.0 v2:172.21.15.32:6816/3066156116 8 ==== osd_op_reply(8 602.00000000 [delete] v112'822 uv822 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f5d50019040 con 0x55c16d3b6150 2024-08-03T20:58:16.417 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:16.416+0000 7f5d5ed1cf40 1 -- 172.21.15.32:0/92850940 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:9 12.10 12:0bd6d154:::602.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112) -- 0x55c16d491860 con 0x55c16d3b6150 2024-08-03T20:58:16.418 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::602.00000000:head# 2024-08-03T20:58:16.419 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:16.418+0000 7f5d57e00640 1 -- 172.21.15.32:0/92850940 <== osd.0 v2:172.21.15.32:6816/3066156116 9 ==== osd_op_reply(9 602.00000000 [create] v112'823 uv823 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f5d50019040 con 0x55c16d3b6150 2024-08-03T20:58:16.419 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:16.418+0000 7f5d5ed1cf40 1 -- 172.21.15.32:0/92850940 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:10 12.10 12:0bd6d154:::602.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112) -- 0x55c16d491860 con 0x55c16d3b6150 2024-08-03T20:58:16.420 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:16.420+0000 7f5d57e00640 1 -- 172.21.15.32:0/92850940 <== osd.0 v2:172.21.15.32:6816/3066156116 10 ==== osd_op_reply(10 602.00000000 [setxattr (30)] v112'824 uv824 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f5d50019040 con 0x55c16d3b6150 2024-08-03T20:58:16.420 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:16.420+0000 7f5d5ed1cf40 1 -- 172.21.15.32:0/92850940 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:11 12.10 12:0bd6d154:::602.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112) -- 0x55c16d491860 con 0x55c16d3b6150 2024-08-03T20:58:16.421 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:16.421+0000 7f5d57e00640 1 -- 172.21.15.32:0/92850940 <== osd.0 v2:172.21.15.32:6816/3066156116 11 ==== osd_op_reply(11 602.00000000 [setxattr (62)] v112'825 uv825 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f5d50019040 con 0x55c16d3b6150 2024-08-03T20:58:16.422 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:16.421+0000 7f5d5ed1cf40 1 -- 172.21.15.32:0/92850940 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:12 12.10 12:0bd6d154:::602.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112) -- 0x55c16d491860 con 0x55c16d3b6150 2024-08-03T20:58:16.423 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:16.423+0000 7f5d57e00640 1 -- 172.21.15.32:0/92850940 <== osd.0 v2:172.21.15.32:6816/3066156116 12 ==== osd_op_reply(12 602.00000000 [omap-set-header] v112'826 uv826 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f5d50019040 con 0x55c16d3b6150 2024-08-03T20:58:16.423 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:16.423+0000 7f5d5ed1cf40 1 -- 172.21.15.32:0/92850940 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:13 12.10 12:0cf6ad0f:::10000000000.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112) -- 0x55c16d491860 con 0x55c16d3b6150 2024-08-03T20:58:16.425 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:16.424+0000 7f5d57e00640 1 -- 172.21.15.32:0/92850940 <== osd.0 v2:172.21.15.32:6816/3066156116 13 ==== osd_op_reply(13 10000000000.00000000 [create] v112'827 uv802 ondisk = -17 ((17) File exists)) ==== 164+0+0 (crc 0 0 0) 0x7f5d50019040 con 0x55c16d3b6150 2024-08-03T20:58:16.425 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:16.424+0000 7f5d5ed1cf40 1 -- 172.21.15.32:0/92850940 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:14 12.10 12:0cf6ad0f:::10000000000.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e112) -- 0x55c16d491860 con 0x55c16d3b6150 2024-08-03T20:58:16.426 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:16.426+0000 7f5d57e00640 1 -- 172.21.15.32:0/92850940 <== osd.0 v2:172.21.15.32:6816/3066156116 14 ==== osd_op_reply(14 10000000000.00000000 [delete] v112'828 uv828 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f5d50019040 con 0x55c16d3b6150 2024-08-03T20:58:16.426 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:16.426+0000 7f5d5ed1cf40 1 -- 172.21.15.32:0/92850940 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:15 12.10 12:0cf6ad0f:::10000000000.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112) -- 0x55c16d491860 con 0x55c16d3b6150 2024-08-03T20:58:16.428 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::10000000000.00000000:head# 2024-08-03T20:58:16.428 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:16.428+0000 7f5d57e00640 1 -- 172.21.15.32:0/92850940 <== osd.0 v2:172.21.15.32:6816/3066156116 15 ==== osd_op_reply(15 10000000000.00000000 [create] v112'829 uv829 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f5d50019040 con 0x55c16d3b6150 2024-08-03T20:58:16.428 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:16.428+0000 7f5d5ed1cf40 1 -- 172.21.15.32:0/92850940 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:16 12.10 12:0cf6ad0f:::10000000000.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112) -- 0x55c16d491860 con 0x55c16d3b6150 2024-08-03T20:58:16.430 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:16.429+0000 7f5d57e00640 1 -- 172.21.15.32:0/92850940 <== osd.0 v2:172.21.15.32:6816/3066156116 16 ==== osd_op_reply(16 10000000000.00000000 [setxattr (30)] v112'830 uv830 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f5d50019040 con 0x55c16d3b6150 2024-08-03T20:58:16.430 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:16.429+0000 7f5d5ed1cf40 1 -- 172.21.15.32:0/92850940 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:17 12.10 12:0cf6ad0f:::10000000000.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112) -- 0x55c16d491860 con 0x55c16d3b6150 2024-08-03T20:58:16.431 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:16.431+0000 7f5d57e00640 1 -- 172.21.15.32:0/92850940 <== osd.0 v2:172.21.15.32:6816/3066156116 17 ==== osd_op_reply(17 10000000000.00000000 [setxattr (62)] v112'831 uv831 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f5d50019040 con 0x55c16d3b6150 2024-08-03T20:58:16.431 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:16.431+0000 7f5d5ed1cf40 1 -- 172.21.15.32:0/92850940 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:18 12.10 12:0cf6ad0f:::10000000000.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112) -- 0x55c16d491860 con 0x55c16d3b6150 2024-08-03T20:58:16.433 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:16.433+0000 7f5d57e00640 1 -- 172.21.15.32:0/92850940 <== osd.0 v2:172.21.15.32:6816/3066156116 18 ==== osd_op_reply(18 10000000000.00000000 [omap-set-header] v112'832 uv832 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f5d50019040 con 0x55c16d3b6150 2024-08-03T20:58:16.433 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:16.433+0000 7f5d5ed1cf40 1 -- 172.21.15.32:0/92850940 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:19 12.10 12:0cf6ad0f:::10000000000.00000000:head [omap-set-vals in=593b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112) -- 0x55c16d4924d0 con 0x55c16d3b6150 2024-08-03T20:58:16.434 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:16.434+0000 7f5d57e00640 1 -- 172.21.15.32:0/92850940 <== osd.0 v2:172.21.15.32:6816/3066156116 19 ==== osd_op_reply(19 10000000000.00000000 [omap-set-vals] v112'833 uv833 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f5d50019040 con 0x55c16d3b6150 2024-08-03T20:58:16.435 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:16.434+0000 7f5d5ed1cf40 1 -- 172.21.15.32:0/92850940 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:20 12.10 12:0d82a743:::600.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112) -- 0x55c16d4924d0 con 0x55c16d3b6150 2024-08-03T20:58:16.436 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:16.436+0000 7f5d57e00640 1 -- 172.21.15.32:0/92850940 <== osd.0 v2:172.21.15.32:6816/3066156116 20 ==== osd_op_reply(20 600.00000000 [create] v112'834 uv808 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f5d50019040 con 0x55c16d3b6150 2024-08-03T20:58:16.436 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:16.436+0000 7f5d5ed1cf40 1 -- 172.21.15.32:0/92850940 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:21 12.10 12:0d82a743:::600.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e112) -- 0x55c16d4924d0 con 0x55c16d3b6150 2024-08-03T20:58:16.438 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:16.438+0000 7f5d57e00640 1 -- 172.21.15.32:0/92850940 <== osd.0 v2:172.21.15.32:6816/3066156116 21 ==== osd_op_reply(21 600.00000000 [delete] v112'835 uv835 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f5d50019040 con 0x55c16d3b6150 2024-08-03T20:58:16.438 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:16.438+0000 7f5d5ed1cf40 1 -- 172.21.15.32:0/92850940 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:22 12.10 12:0d82a743:::600.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112) -- 0x55c16d4924d0 con 0x55c16d3b6150 2024-08-03T20:58:16.439 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::600.00000000:head# 2024-08-03T20:58:16.440 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:16.439+0000 7f5d57e00640 1 -- 172.21.15.32:0/92850940 <== osd.0 v2:172.21.15.32:6816/3066156116 22 ==== osd_op_reply(22 600.00000000 [create] v112'836 uv836 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f5d50019040 con 0x55c16d3b6150 2024-08-03T20:58:16.440 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:16.439+0000 7f5d5ed1cf40 1 -- 172.21.15.32:0/92850940 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:23 12.10 12:0d82a743:::600.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112) -- 0x55c16d4934e0 con 0x55c16d3b6150 2024-08-03T20:58:16.441 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:16.441+0000 7f5d57e00640 1 -- 172.21.15.32:0/92850940 <== osd.0 v2:172.21.15.32:6816/3066156116 23 ==== osd_op_reply(23 600.00000000 [setxattr (30)] v112'837 uv837 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f5d50019040 con 0x55c16d3b6150 2024-08-03T20:58:16.441 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:16.441+0000 7f5d5ed1cf40 1 -- 172.21.15.32:0/92850940 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:24 12.10 12:0d82a743:::600.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112) -- 0x55c16d4934e0 con 0x55c16d3b6150 2024-08-03T20:58:16.443 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:16.442+0000 7f5d57e00640 1 -- 172.21.15.32:0/92850940 <== osd.0 v2:172.21.15.32:6816/3066156116 24 ==== osd_op_reply(24 600.00000000 [setxattr (62)] v112'838 uv838 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f5d50019040 con 0x55c16d3b6150 2024-08-03T20:58:16.443 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:16.442+0000 7f5d5ed1cf40 1 -- 172.21.15.32:0/92850940 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:25 12.10 12:0d82a743:::600.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112) -- 0x55c16d4924d0 con 0x55c16d3b6150 2024-08-03T20:58:16.444 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:16.444+0000 7f5d57e00640 1 -- 172.21.15.32:0/92850940 <== osd.0 v2:172.21.15.32:6816/3066156116 25 ==== osd_op_reply(25 600.00000000 [omap-set-header] v112'839 uv839 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f5d50019040 con 0x55c16d3b6150 2024-08-03T20:58:16.444 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:16.444+0000 7f5d5ed1cf40 1 -- 172.21.15.32:0/92850940 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:26 12.10 12:0d89b25e:::603.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112) -- 0x55c16d4924d0 con 0x55c16d3b6150 2024-08-03T20:58:16.446 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:16.446+0000 7f5d57e00640 1 -- 172.21.15.32:0/92850940 <== osd.0 v2:172.21.15.32:6816/3066156116 26 ==== osd_op_reply(26 603.00000000 [create] v112'840 uv814 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f5d50019040 con 0x55c16d3b6150 2024-08-03T20:58:16.446 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:16.446+0000 7f5d5ed1cf40 1 -- 172.21.15.32:0/92850940 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:27 12.10 12:0d89b25e:::603.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e112) -- 0x55c16d4924d0 con 0x55c16d3b6150 2024-08-03T20:58:16.448 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:16.447+0000 7f5d57e00640 1 -- 172.21.15.32:0/92850940 <== osd.0 v2:172.21.15.32:6816/3066156116 27 ==== osd_op_reply(27 603.00000000 [delete] v112'841 uv841 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f5d50019040 con 0x55c16d3b6150 2024-08-03T20:58:16.448 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:16.447+0000 7f5d5ed1cf40 1 -- 172.21.15.32:0/92850940 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:28 12.10 12:0d89b25e:::603.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112) -- 0x55c16d4924d0 con 0x55c16d3b6150 2024-08-03T20:58:16.449 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:16.449+0000 7f5d57e00640 1 -- 172.21.15.32:0/92850940 <== osd.0 v2:172.21.15.32:6816/3066156116 28 ==== osd_op_reply(28 603.00000000 [create] v112'842 uv842 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f5d50019040 con 0x55c16d3b6150 2024-08-03T20:58:16.449 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::603.00000000:head# 2024-08-03T20:58:16.450 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:16.449+0000 7f5d5ed1cf40 1 -- 172.21.15.32:0/92850940 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:29 12.10 12:0d89b25e:::603.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112) -- 0x55c16d4934e0 con 0x55c16d3b6150 2024-08-03T20:58:16.451 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:16.450+0000 7f5d57e00640 1 -- 172.21.15.32:0/92850940 <== osd.0 v2:172.21.15.32:6816/3066156116 29 ==== osd_op_reply(29 603.00000000 [setxattr (30)] v112'843 uv843 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f5d50019040 con 0x55c16d3b6150 2024-08-03T20:58:16.451 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:16.450+0000 7f5d5ed1cf40 1 -- 172.21.15.32:0/92850940 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:30 12.10 12:0d89b25e:::603.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112) -- 0x55c16d4934e0 con 0x55c16d3b6150 2024-08-03T20:58:16.452 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:16.452+0000 7f5d57e00640 1 -- 172.21.15.32:0/92850940 <== osd.0 v2:172.21.15.32:6816/3066156116 30 ==== osd_op_reply(30 603.00000000 [setxattr (62)] v112'844 uv844 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f5d50019040 con 0x55c16d3b6150 2024-08-03T20:58:16.453 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:16.452+0000 7f5d5ed1cf40 1 -- 172.21.15.32:0/92850940 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:31 12.10 12:0d89b25e:::603.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112) -- 0x55c16d4924d0 con 0x55c16d3b6150 2024-08-03T20:58:16.454 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:16.454+0000 7f5d57e00640 1 -- 172.21.15.32:0/92850940 <== osd.0 v2:172.21.15.32:6816/3066156116 31 ==== osd_op_reply(31 603.00000000 [omap-set-header] v112'845 uv845 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f5d50019040 con 0x55c16d3b6150 2024-08-03T20:58:16.454 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:16.454+0000 7f5d5ed1cf40 1 --2- 172.21.15.32:0/92850940 >> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] conn(0x55c16d4929e0 0x55c16d494e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:58:16.454 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:16.454+0000 7f5d5ed1cf40 1 -- 172.21.15.32:0/92850940 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:32 12.14 12:287034c4:::1.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112) -- 0x55c16d495340 con 0x55c16d4929e0 2024-08-03T20:58:16.455 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:16.454+0000 7f5d57400640 1 --2- 172.21.15.32:0/92850940 >> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] conn(0x55c16d4929e0 0x55c16d494e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:58:16.455 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:16.455+0000 7f5d57400640 1 --2- 172.21.15.32:0/92850940 >> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] conn(0x55c16d4929e0 0x55c16d494e00 crc :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:58:16.457 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:16.457+0000 7f5d57400640 1 -- 172.21.15.32:0/92850940 <== osd.5 v2:172.21.15.118:6810/2724891770 1 ==== osd_op_reply(32 1.00000000.inode [create] v112'596 uv575 ondisk = -17 ((17) File exists)) ==== 160+0+0 (crc 0 0 0) 0x7f5d4800e040 con 0x55c16d4929e0 2024-08-03T20:58:16.457 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:16.457+0000 7f5d5ed1cf40 1 -- 172.21.15.32:0/92850940 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:33 12.14 12:287034c4:::1.00000000.inode:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e112) -- 0x55c16d4956e0 con 0x55c16d4929e0 2024-08-03T20:58:16.459 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:16.459+0000 7f5d57400640 1 -- 172.21.15.32:0/92850940 <== osd.5 v2:172.21.15.118:6810/2724891770 2 ==== osd_op_reply(33 1.00000000.inode [delete] v112'597 uv597 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f5d4800e040 con 0x55c16d4929e0 2024-08-03T20:58:16.459 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:16.459+0000 7f5d5ed1cf40 1 -- 172.21.15.32:0/92850940 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:34 12.14 12:287034c4:::1.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112) -- 0x55c16d495a80 con 0x55c16d4929e0 2024-08-03T20:58:16.460 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::1.00000000.inode:head# 2024-08-03T20:58:16.461 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:16.460+0000 7f5d57400640 1 -- 172.21.15.32:0/92850940 <== osd.5 v2:172.21.15.118:6810/2724891770 3 ==== osd_op_reply(34 1.00000000.inode [create] v112'598 uv598 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f5d4800e040 con 0x55c16d4929e0 2024-08-03T20:58:16.461 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:16.460+0000 7f5d5ed1cf40 1 -- 172.21.15.32:0/92850940 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:35 12.14 12:287034c4:::1.00000000.inode:head [write 0~582 in=582b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112) -- 0x55c16d495e20 con 0x55c16d4929e0 2024-08-03T20:58:16.462 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:16.462+0000 7f5d57400640 1 -- 172.21.15.32:0/92850940 <== osd.5 v2:172.21.15.118:6810/2724891770 4 ==== osd_op_reply(35 1.00000000.inode [write 0~582] v112'599 uv599 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f5d4800e040 con 0x55c16d4929e0 2024-08-03T20:58:16.462 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:16.462+0000 7f5d5ed1cf40 1 -- 172.21.15.32:0/92850940 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:36 12.14 12:287034c4:::1.00000000.inode:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112) -- 0x55c16d4961c0 con 0x55c16d4929e0 2024-08-03T20:58:16.464 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:16.463+0000 7f5d57400640 1 -- 172.21.15.32:0/92850940 <== osd.5 v2:172.21.15.118:6810/2724891770 5 ==== osd_op_reply(36 1.00000000.inode [omap-set-header] v112'600 uv600 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f5d4800e040 con 0x55c16d4929e0 2024-08-03T20:58:16.464 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:16.463+0000 7f5d5ed1cf40 1 -- 172.21.15.32:0/92850940 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:37 12.14 12:292cf221:::200.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112) -- 0x55c16d496560 con 0x55c16d4929e0 2024-08-03T20:58:16.465 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:16.465+0000 7f5d57400640 1 -- 172.21.15.32:0/92850940 <== osd.5 v2:172.21.15.118:6810/2724891770 6 ==== osd_op_reply(37 200.00000000 [create] v112'601 uv594 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f5d4800e040 con 0x55c16d4929e0 2024-08-03T20:58:16.465 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:16.465+0000 7f5d5ed1cf40 1 -- 172.21.15.32:0/92850940 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:38 12.14 12:292cf221:::200.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e112) -- 0x55c16d496900 con 0x55c16d4929e0 2024-08-03T20:58:16.467 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:16.466+0000 7f5d57400640 1 -- 172.21.15.32:0/92850940 <== osd.5 v2:172.21.15.118:6810/2724891770 7 ==== osd_op_reply(38 200.00000000 [delete] v112'602 uv602 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f5d4800e040 con 0x55c16d4929e0 2024-08-03T20:58:16.467 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:16.466+0000 7f5d5ed1cf40 1 -- 172.21.15.32:0/92850940 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:39 12.14 12:292cf221:::200.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112) -- 0x55c16d496ca0 con 0x55c16d4929e0 2024-08-03T20:58:16.468 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::200.00000000:head# 2024-08-03T20:58:16.469 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:16.468+0000 7f5d57400640 1 -- 172.21.15.32:0/92850940 <== osd.5 v2:172.21.15.118:6810/2724891770 8 ==== osd_op_reply(39 200.00000000 [create] v112'603 uv603 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f5d4800e040 con 0x55c16d4929e0 2024-08-03T20:58:16.469 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:16.468+0000 7f5d5ed1cf40 1 -- 172.21.15.32:0/92850940 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:40 12.14 12:292cf221:::200.00000000:head [write 0~90 in=90b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112) -- 0x55c16d496ca0 con 0x55c16d4929e0 2024-08-03T20:58:16.470 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:16.469+0000 7f5d57400640 1 -- 172.21.15.32:0/92850940 <== osd.5 v2:172.21.15.118:6810/2724891770 9 ==== osd_op_reply(40 200.00000000 [write 0~90] v112'604 uv604 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f5d4800e040 con 0x55c16d4929e0 2024-08-03T20:58:16.470 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:16.470+0000 7f5d5ed1cf40 1 -- 172.21.15.32:0/92850940 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:41 12.14 12:292cf221:::200.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112) -- 0x55c16d496ca0 con 0x55c16d4929e0 2024-08-03T20:58:16.471 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:16.471+0000 7f5d57400640 1 -- 172.21.15.32:0/92850940 <== osd.5 v2:172.21.15.118:6810/2724891770 10 ==== osd_op_reply(41 200.00000000 [omap-set-header] v112'605 uv605 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f5d4800e040 con 0x55c16d4929e0 2024-08-03T20:58:16.471 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:16.471+0000 7f5d5ed1cf40 1 -- 172.21.15.32:0/92850940 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:42 12.14 12:2e2fa760:::200.00000001:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112) -- 0x55c16d496ca0 con 0x55c16d4929e0 2024-08-03T20:58:16.473 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:16.472+0000 7f5d57400640 1 -- 172.21.15.32:0/92850940 <== osd.5 v2:172.21.15.118:6810/2724891770 11 ==== osd_op_reply(42 200.00000001 [create] v112'606 uv595 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f5d4800e040 con 0x55c16d4929e0 2024-08-03T20:58:16.473 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:16.472+0000 7f5d5ed1cf40 1 -- 172.21.15.32:0/92850940 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:43 12.14 12:2e2fa760:::200.00000001:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e112) -- 0x55c16d496ca0 con 0x55c16d4929e0 2024-08-03T20:58:16.474 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:16.474+0000 7f5d57400640 1 -- 172.21.15.32:0/92850940 <== osd.5 v2:172.21.15.118:6810/2724891770 12 ==== osd_op_reply(43 200.00000001 [delete] v112'607 uv607 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f5d4800e040 con 0x55c16d4929e0 2024-08-03T20:58:16.475 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:16.474+0000 7f5d5ed1cf40 1 -- 172.21.15.32:0/92850940 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:44 12.14 12:2e2fa760:::200.00000001:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112) -- 0x55c16d496ca0 con 0x55c16d4929e0 2024-08-03T20:58:16.476 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::200.00000001:head# 2024-08-03T20:58:16.476 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:16.475+0000 7f5d57400640 1 -- 172.21.15.32:0/92850940 <== osd.5 v2:172.21.15.118:6810/2724891770 13 ==== osd_op_reply(44 200.00000001 [create] v112'608 uv608 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f5d4800e040 con 0x55c16d4929e0 2024-08-03T20:58:16.476 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:16.476+0000 7f5d5ed1cf40 1 -- 172.21.15.32:0/92850940 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:45 12.14 12:2e2fa760:::200.00000001:head [write 0~18118 in=18118b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112) -- 0x55c16d49b390 con 0x55c16d4929e0 2024-08-03T20:58:16.478 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:16.478+0000 7f5d57400640 1 -- 172.21.15.32:0/92850940 <== osd.5 v2:172.21.15.118:6810/2724891770 14 ==== osd_op_reply(45 200.00000001 [write 0~18118] v112'609 uv609 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f5d4800e040 con 0x55c16d4929e0 2024-08-03T20:58:16.478 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:16.478+0000 7f5d5ed1cf40 1 -- 172.21.15.32:0/92850940 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:46 12.14 12:2e2fa760:::200.00000001:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112) -- 0x55c16d49b390 con 0x55c16d4929e0 2024-08-03T20:58:16.480 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:16.479+0000 7f5d57400640 1 -- 172.21.15.32:0/92850940 <== osd.5 v2:172.21.15.118:6810/2724891770 15 ==== osd_op_reply(46 200.00000001 [omap-set-header] v112'610 uv610 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f5d4800e040 con 0x55c16d4929e0 2024-08-03T20:58:16.480 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:16.479+0000 7f5d5ed1cf40 1 -- 172.21.15.32:0/92850940 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:47 12.14 12:2f55791f:::606.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112) -- 0x55c16d496ca0 con 0x55c16d4929e0 2024-08-03T20:58:16.481 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:16.481+0000 7f5d57400640 1 -- 172.21.15.32:0/92850940 <== osd.5 v2:172.21.15.118:6810/2724891770 16 ==== osd_op_reply(47 606.00000000 [create] v112'611 uv591 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f5d4800e040 con 0x55c16d4929e0 2024-08-03T20:58:16.481 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:16.481+0000 7f5d5ed1cf40 1 -- 172.21.15.32:0/92850940 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:48 12.14 12:2f55791f:::606.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e112) -- 0x55c16d496ca0 con 0x55c16d4929e0 2024-08-03T20:58:16.482 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:16.482+0000 7f5d57400640 1 -- 172.21.15.32:0/92850940 <== osd.5 v2:172.21.15.118:6810/2724891770 17 ==== osd_op_reply(48 606.00000000 [delete] v112'612 uv612 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f5d4800e040 con 0x55c16d4929e0 2024-08-03T20:58:16.483 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:16.482+0000 7f5d5ed1cf40 1 -- 172.21.15.32:0/92850940 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:49 12.14 12:2f55791f:::606.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112) -- 0x55c16d496ca0 con 0x55c16d4929e0 2024-08-03T20:58:16.484 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::606.00000000:head# 2024-08-03T20:58:16.484 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:16.484+0000 7f5d57400640 1 -- 172.21.15.32:0/92850940 <== osd.5 v2:172.21.15.118:6810/2724891770 18 ==== osd_op_reply(49 606.00000000 [create] v112'613 uv613 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f5d4800e040 con 0x55c16d4929e0 2024-08-03T20:58:16.484 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:16.484+0000 7f5d5ed1cf40 1 -- 172.21.15.32:0/92850940 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:50 12.14 12:2f55791f:::606.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112) -- 0x55c16d497cb0 con 0x55c16d4929e0 2024-08-03T20:58:16.486 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:16.485+0000 7f5d57400640 1 -- 172.21.15.32:0/92850940 <== osd.5 v2:172.21.15.118:6810/2724891770 19 ==== osd_op_reply(50 606.00000000 [setxattr (30)] v112'614 uv614 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f5d4800e040 con 0x55c16d4929e0 2024-08-03T20:58:16.486 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:16.485+0000 7f5d5ed1cf40 1 -- 172.21.15.32:0/92850940 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:51 12.14 12:2f55791f:::606.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112) -- 0x55c16d497cb0 con 0x55c16d4929e0 2024-08-03T20:58:16.487 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:16.487+0000 7f5d57400640 1 -- 172.21.15.32:0/92850940 <== osd.5 v2:172.21.15.118:6810/2724891770 20 ==== osd_op_reply(51 606.00000000 [setxattr (62)] v112'615 uv615 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f5d4800e040 con 0x55c16d4929e0 2024-08-03T20:58:16.487 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:16.487+0000 7f5d5ed1cf40 1 -- 172.21.15.32:0/92850940 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:52 12.14 12:2f55791f:::606.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112) -- 0x55c16d496ca0 con 0x55c16d4929e0 2024-08-03T20:58:16.488 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:16.488+0000 7f5d57400640 1 -- 172.21.15.32:0/92850940 <== osd.5 v2:172.21.15.118:6810/2724891770 21 ==== osd_op_reply(52 606.00000000 [omap-set-header] v112'616 uv616 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f5d4800e040 con 0x55c16d4929e0 2024-08-03T20:58:16.488 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:16.488+0000 7f5d5ed1cf40 1 -- 172.21.15.32:0/92850940 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:53 12.1c 12:3ed09add:::607.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112) -- 0x55c16d496ca0 con 0x55c16d3b6150 2024-08-03T20:58:16.490 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:16.489+0000 7f5d57e00640 1 -- 172.21.15.32:0/92850940 <== osd.0 v2:172.21.15.32:6816/3066156116 32 ==== osd_op_reply(53 607.00000000 [create] v112'159 uv158 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f5d50019040 con 0x55c16d3b6150 2024-08-03T20:58:16.490 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:16.489+0000 7f5d5ed1cf40 1 -- 172.21.15.32:0/92850940 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:54 12.1c 12:3ed09add:::607.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e112) -- 0x55c16d496ca0 con 0x55c16d3b6150 2024-08-03T20:58:16.491 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:16.491+0000 7f5d57e00640 1 -- 172.21.15.32:0/92850940 <== osd.0 v2:172.21.15.32:6816/3066156116 33 ==== osd_op_reply(54 607.00000000 [delete] v112'160 uv160 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f5d50019040 con 0x55c16d3b6150 2024-08-03T20:58:16.491 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:16.491+0000 7f5d5ed1cf40 1 -- 172.21.15.32:0/92850940 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:55 12.1c 12:3ed09add:::607.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112) -- 0x55c16d496ca0 con 0x55c16d3b6150 2024-08-03T20:58:16.492 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:16.492+0000 7f5d57e00640 1 -- 172.21.15.32:0/92850940 <== osd.0 v2:172.21.15.32:6816/3066156116 34 ==== osd_op_reply(55 607.00000000 [create] v112'161 uv161 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f5d50019040 con 0x55c16d3b6150 2024-08-03T20:58:16.493 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::607.00000000:head# 2024-08-03T20:58:16.493 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:16.492+0000 7f5d5ed1cf40 1 -- 172.21.15.32:0/92850940 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:56 12.1c 12:3ed09add:::607.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112) -- 0x55c16d497cb0 con 0x55c16d3b6150 2024-08-03T20:58:16.494 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:16.493+0000 7f5d57e00640 1 -- 172.21.15.32:0/92850940 <== osd.0 v2:172.21.15.32:6816/3066156116 35 ==== osd_op_reply(56 607.00000000 [setxattr (30)] v112'162 uv162 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f5d50019040 con 0x55c16d3b6150 2024-08-03T20:58:16.494 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:16.494+0000 7f5d5ed1cf40 1 -- 172.21.15.32:0/92850940 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:57 12.1c 12:3ed09add:::607.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112) -- 0x55c16d497cb0 con 0x55c16d3b6150 2024-08-03T20:58:16.495 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:16.495+0000 7f5d57e00640 1 -- 172.21.15.32:0/92850940 <== osd.0 v2:172.21.15.32:6816/3066156116 36 ==== osd_op_reply(57 607.00000000 [setxattr (62)] v112'163 uv163 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f5d50019040 con 0x55c16d3b6150 2024-08-03T20:58:16.495 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:16.495+0000 7f5d5ed1cf40 1 -- 172.21.15.32:0/92850940 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:58 12.1c 12:3ed09add:::607.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112) -- 0x55c16d496ca0 con 0x55c16d3b6150 2024-08-03T20:58:16.497 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:16.496+0000 7f5d57e00640 1 -- 172.21.15.32:0/92850940 <== osd.0 v2:172.21.15.32:6816/3066156116 37 ==== osd_op_reply(58 607.00000000 [omap-set-header] v112'164 uv164 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f5d50019040 con 0x55c16d3b6150 2024-08-03T20:58:16.497 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:16.496+0000 7f5d5ed1cf40 1 --2- 172.21.15.32:0/92850940 >> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] conn(0x55c16d4971b0 0x55c16d4995d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:58:16.497 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:16.496+0000 7f5d5ed1cf40 1 -- 172.21.15.32:0/92850940 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:59 12.6 12:654134d2:::mds0_openfiles.0:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112) -- 0x55c16d499c00 con 0x55c16d4971b0 2024-08-03T20:58:16.497 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:16.496+0000 7f5d56a00640 1 --2- 172.21.15.32:0/92850940 >> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] conn(0x55c16d4971b0 0x55c16d4995d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:58:16.497 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:16.497+0000 7f5d56a00640 1 --2- 172.21.15.32:0/92850940 >> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] conn(0x55c16d4971b0 0x55c16d4995d0 crc :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:58:16.499 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:16.498+0000 7f5d56a00640 1 -- 172.21.15.32:0/92850940 <== osd.1 v2:172.21.15.32:6824/1551241759 1 ==== osd_op_reply(59 mds0_openfiles.0 [create] v112'125 uv124 ondisk = -17 ((17) File exists)) ==== 160+0+0 (crc 0 0 0) 0x7f5d440a0300 con 0x55c16d4971b0 2024-08-03T20:58:16.499 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:16.498+0000 7f5d5ed1cf40 1 -- 172.21.15.32:0/92850940 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:60 12.6 12:654134d2:::mds0_openfiles.0:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e112) -- 0x55c16d49a110 con 0x55c16d4971b0 2024-08-03T20:58:16.500 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:16.500+0000 7f5d56a00640 1 -- 172.21.15.32:0/92850940 <== osd.1 v2:172.21.15.32:6824/1551241759 2 ==== osd_op_reply(60 mds0_openfiles.0 [delete] v112'126 uv126 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f5d440a0300 con 0x55c16d4971b0 2024-08-03T20:58:16.500 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:16.500+0000 7f5d5ed1cf40 1 -- 172.21.15.32:0/92850940 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:61 12.6 12:654134d2:::mds0_openfiles.0:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112) -- 0x55c16d49a610 con 0x55c16d4971b0 2024-08-03T20:58:16.501 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds0_openfiles.0:head# 2024-08-03T20:58:16.501 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:16.501+0000 7f5d56a00640 1 -- 172.21.15.32:0/92850940 <== osd.1 v2:172.21.15.32:6824/1551241759 3 ==== osd_op_reply(61 mds0_openfiles.0 [create] v112'127 uv127 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f5d440a0300 con 0x55c16d4971b0 2024-08-03T20:58:16.502 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:16.501+0000 7f5d5ed1cf40 1 -- 172.21.15.32:0/92850940 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:62 12.6 12:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112) -- 0x55c16d491c00 con 0x55c16d4971b0 2024-08-03T20:58:16.503 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:16.502+0000 7f5d56a00640 1 -- 172.21.15.32:0/92850940 <== osd.1 v2:172.21.15.32:6824/1551241759 4 ==== osd_op_reply(62 mds0_openfiles.0 [omap-set-header] v112'128 uv128 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f5d440a0300 con 0x55c16d4971b0 2024-08-03T20:58:16.503 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:16.502+0000 7f5d5ed1cf40 1 -- 172.21.15.32:0/92850940 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:63 12.e 12:717a0223:::608.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112) -- 0x55c16d49af40 con 0x55c16d3b6150 2024-08-03T20:58:16.504 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:16.504+0000 7f5d57e00640 1 -- 172.21.15.32:0/92850940 <== osd.0 v2:172.21.15.32:6816/3066156116 38 ==== osd_op_reply(63 608.00000000 [create] v112'159 uv158 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f5d50019040 con 0x55c16d3b6150 2024-08-03T20:58:16.504 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:16.504+0000 7f5d5ed1cf40 1 -- 172.21.15.32:0/92850940 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:64 12.e 12:717a0223:::608.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e112) -- 0x55c16d49af40 con 0x55c16d3b6150 2024-08-03T20:58:16.506 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:16.505+0000 7f5d57e00640 1 -- 172.21.15.32:0/92850940 <== osd.0 v2:172.21.15.32:6816/3066156116 39 ==== osd_op_reply(64 608.00000000 [delete] v112'160 uv160 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f5d50019040 con 0x55c16d3b6150 2024-08-03T20:58:16.506 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:16.505+0000 7f5d5ed1cf40 1 -- 172.21.15.32:0/92850940 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:65 12.e 12:717a0223:::608.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112) -- 0x55c16d49af40 con 0x55c16d3b6150 2024-08-03T20:58:16.507 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::608.00000000:head# 2024-08-03T20:58:16.507 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:16.507+0000 7f5d57e00640 1 -- 172.21.15.32:0/92850940 <== osd.0 v2:172.21.15.32:6816/3066156116 40 ==== osd_op_reply(65 608.00000000 [create] v112'161 uv161 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f5d50019040 con 0x55c16d3b6150 2024-08-03T20:58:16.507 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:16.507+0000 7f5d5ed1cf40 1 -- 172.21.15.32:0/92850940 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:66 12.e 12:717a0223:::608.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112) -- 0x55c16d49bf50 con 0x55c16d3b6150 2024-08-03T20:58:16.508 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:16.508+0000 7f5d57e00640 1 -- 172.21.15.32:0/92850940 <== osd.0 v2:172.21.15.32:6816/3066156116 41 ==== osd_op_reply(66 608.00000000 [setxattr (30)] v112'162 uv162 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f5d50019040 con 0x55c16d3b6150 2024-08-03T20:58:16.509 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:16.508+0000 7f5d5ed1cf40 1 -- 172.21.15.32:0/92850940 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:67 12.e 12:717a0223:::608.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112) -- 0x55c16d49bf50 con 0x55c16d3b6150 2024-08-03T20:58:16.510 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:16.509+0000 7f5d57e00640 1 -- 172.21.15.32:0/92850940 <== osd.0 v2:172.21.15.32:6816/3066156116 42 ==== osd_op_reply(67 608.00000000 [setxattr (62)] v112'163 uv163 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f5d50019040 con 0x55c16d3b6150 2024-08-03T20:58:16.510 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:16.509+0000 7f5d5ed1cf40 1 -- 172.21.15.32:0/92850940 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:68 12.e 12:717a0223:::608.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112) -- 0x55c16d49af40 con 0x55c16d3b6150 2024-08-03T20:58:16.511 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:16.511+0000 7f5d57e00640 1 -- 172.21.15.32:0/92850940 <== osd.0 v2:172.21.15.32:6816/3066156116 43 ==== osd_op_reply(68 608.00000000 [omap-set-header] v112'164 uv164 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f5d50019040 con 0x55c16d3b6150 2024-08-03T20:58:16.511 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:16.511+0000 7f5d5ed1cf40 1 --2- 172.21.15.32:0/92850940 >> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] conn(0x55c16d49b810 0x55c16d49dc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:58:16.511 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:16.511+0000 7f5d5ed1cf40 1 -- 172.21.15.32:0/92850940 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:69 12.15 12:a93a17c2:::604.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112) -- 0x55c16d49e170 con 0x55c16d49b810 2024-08-03T20:58:16.512 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:16.511+0000 7f5d57e00640 1 --2- 172.21.15.32:0/92850940 >> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] conn(0x55c16d49b810 0x55c16d49dc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:58:16.512 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:16.512+0000 7f5d57e00640 1 --2- 172.21.15.32:0/92850940 >> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] conn(0x55c16d49b810 0x55c16d49dc30 crc :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:58:16.514 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:16.514+0000 7f5d57e00640 1 -- 172.21.15.32:0/92850940 <== osd.6 v2:172.21.15.118:6825/801947386 1 ==== osd_op_reply(69 604.00000000 [create] v112'294 uv287 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f5d50019040 con 0x55c16d49b810 2024-08-03T20:58:16.514 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:16.514+0000 7f5d5ed1cf40 1 -- 172.21.15.32:0/92850940 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:70 12.15 12:a93a17c2:::604.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e112) -- 0x55c16d49e170 con 0x55c16d49b810 2024-08-03T20:58:16.516 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:16.515+0000 7f5d57e00640 1 -- 172.21.15.32:0/92850940 <== osd.6 v2:172.21.15.118:6825/801947386 2 ==== osd_op_reply(70 604.00000000 [delete] v112'295 uv295 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f5d50019040 con 0x55c16d49b810 2024-08-03T20:58:16.516 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:16.515+0000 7f5d5ed1cf40 1 -- 172.21.15.32:0/92850940 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:71 12.15 12:a93a17c2:::604.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112) -- 0x55c16d49e560 con 0x55c16d49b810 2024-08-03T20:58:16.517 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::604.00000000:head# 2024-08-03T20:58:16.518 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:16.517+0000 7f5d57e00640 1 -- 172.21.15.32:0/92850940 <== osd.6 v2:172.21.15.118:6825/801947386 3 ==== osd_op_reply(71 604.00000000 [create] v112'296 uv296 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f5d50019040 con 0x55c16d49b810 2024-08-03T20:58:16.518 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:16.517+0000 7f5d5ed1cf40 1 -- 172.21.15.32:0/92850940 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:72 12.15 12:a93a17c2:::604.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112) -- 0x55c16d49f570 con 0x55c16d49b810 2024-08-03T20:58:16.519 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:16.518+0000 7f5d57e00640 1 -- 172.21.15.32:0/92850940 <== osd.6 v2:172.21.15.118:6825/801947386 4 ==== osd_op_reply(72 604.00000000 [setxattr (30)] v112'297 uv297 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f5d50019040 con 0x55c16d49b810 2024-08-03T20:58:16.519 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:16.519+0000 7f5d5ed1cf40 1 -- 172.21.15.32:0/92850940 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:73 12.15 12:a93a17c2:::604.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112) -- 0x55c16d49f570 con 0x55c16d49b810 2024-08-03T20:58:16.520 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:16.520+0000 7f5d57e00640 1 -- 172.21.15.32:0/92850940 <== osd.6 v2:172.21.15.118:6825/801947386 5 ==== osd_op_reply(73 604.00000000 [setxattr (62)] v112'298 uv298 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f5d50019040 con 0x55c16d49b810 2024-08-03T20:58:16.520 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:16.520+0000 7f5d5ed1cf40 1 -- 172.21.15.32:0/92850940 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:74 12.15 12:a93a17c2:::604.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112) -- 0x55c16d49e560 con 0x55c16d49b810 2024-08-03T20:58:16.521 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:16.521+0000 7f5d57e00640 1 -- 172.21.15.32:0/92850940 <== osd.6 v2:172.21.15.118:6825/801947386 6 ==== osd_op_reply(74 604.00000000 [omap-set-header] v112'299 uv299 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f5d50019040 con 0x55c16d49b810 2024-08-03T20:58:16.522 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:16.521+0000 7f5d5ed1cf40 1 -- 172.21.15.32:0/92850940 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:75 12.15 12:aa448500:::500.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112) -- 0x55c16d49e560 con 0x55c16d49b810 2024-08-03T20:58:16.523 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:16.523+0000 7f5d57e00640 1 -- 172.21.15.32:0/92850940 <== osd.6 v2:172.21.15.118:6825/801947386 7 ==== osd_op_reply(75 500.00000000 [create] v112'300 uv293 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f5d50019040 con 0x55c16d49b810 2024-08-03T20:58:16.523 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:16.523+0000 7f5d5ed1cf40 1 -- 172.21.15.32:0/92850940 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:76 12.15 12:aa448500:::500.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e112) -- 0x55c16d49e560 con 0x55c16d49b810 2024-08-03T20:58:16.525 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:16.524+0000 7f5d57e00640 1 -- 172.21.15.32:0/92850940 <== osd.6 v2:172.21.15.118:6825/801947386 8 ==== osd_op_reply(76 500.00000000 [delete] v112'301 uv301 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f5d50019040 con 0x55c16d49b810 2024-08-03T20:58:16.525 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:16.524+0000 7f5d5ed1cf40 1 -- 172.21.15.32:0/92850940 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:77 12.15 12:aa448500:::500.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112) -- 0x55c16d49e560 con 0x55c16d49b810 2024-08-03T20:58:16.526 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::500.00000000:head# 2024-08-03T20:58:16.526 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:16.526+0000 7f5d57e00640 1 -- 172.21.15.32:0/92850940 <== osd.6 v2:172.21.15.118:6825/801947386 9 ==== osd_op_reply(77 500.00000000 [create] v112'302 uv302 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f5d50019040 con 0x55c16d49b810 2024-08-03T20:58:16.526 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:16.526+0000 7f5d5ed1cf40 1 -- 172.21.15.32:0/92850940 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:78 12.15 12:aa448500:::500.00000000:head [write 0~90 in=90b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112) -- 0x55c16d49e560 con 0x55c16d49b810 2024-08-03T20:58:16.528 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:16.527+0000 7f5d57e00640 1 -- 172.21.15.32:0/92850940 <== osd.6 v2:172.21.15.118:6825/801947386 10 ==== osd_op_reply(78 500.00000000 [write 0~90] v112'303 uv303 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f5d50019040 con 0x55c16d49b810 2024-08-03T20:58:16.528 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:16.527+0000 7f5d5ed1cf40 1 -- 172.21.15.32:0/92850940 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:79 12.15 12:aa448500:::500.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112) -- 0x55c16d49e560 con 0x55c16d49b810 2024-08-03T20:58:16.530 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:16.529+0000 7f5d57e00640 1 -- 172.21.15.32:0/92850940 <== osd.6 v2:172.21.15.118:6825/801947386 11 ==== osd_op_reply(79 500.00000000 [omap-set-header] v112'304 uv304 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f5d50019040 con 0x55c16d49b810 2024-08-03T20:58:16.530 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:16.529+0000 7f5d5ed1cf40 1 --2- 172.21.15.32:0/92850940 >> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] conn(0x55c16d49ea70 0x55c16d4a0ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:58:16.530 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:16.529+0000 7f5d5ed1cf40 1 -- 172.21.15.32:0/92850940 --> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] -- osd_op(unknown.0.0:80 12.d 12:b50e409b:::mds_snaptable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112) -- 0x55c16d4a15f0 con 0x55c16d49ea70 2024-08-03T20:58:16.530 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:16.530+0000 7f5d57400640 1 --2- 172.21.15.32:0/92850940 >> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] conn(0x55c16d49ea70 0x55c16d4a0ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:58:16.531 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:16.530+0000 7f5d57400640 1 --2- 172.21.15.32:0/92850940 >> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] conn(0x55c16d49ea70 0x55c16d4a0ed0 crc :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:58:16.534 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:16.532+0000 7f5d57400640 1 -- 172.21.15.32:0/92850940 <== osd.7 v2:172.21.15.118:6818/653000048 1 ==== osd_op_reply(80 mds_snaptable [create] v112'132 uv131 ondisk = -17 ((17) File exists)) ==== 157+0+0 (crc 0 0 0) 0x7f5d4800e040 con 0x55c16d49ea70 2024-08-03T20:58:16.534 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:16.532+0000 7f5d5ed1cf40 1 -- 172.21.15.32:0/92850940 --> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] -- osd_op(unknown.0.0:81 12.d 12:b50e409b:::mds_snaptable:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e112) -- 0x55c16d4a1630 con 0x55c16d49ea70 2024-08-03T20:58:16.534 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:16.534+0000 7f5d57400640 1 -- 172.21.15.32:0/92850940 <== osd.7 v2:172.21.15.118:6818/653000048 2 ==== osd_op_reply(81 mds_snaptable [delete] v112'133 uv133 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f5d4800e040 con 0x55c16d49ea70 2024-08-03T20:58:16.534 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:16.534+0000 7f5d5ed1cf40 1 -- 172.21.15.32:0/92850940 --> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] -- osd_op(unknown.0.0:82 12.d 12:b50e409b:::mds_snaptable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112) -- 0x55c16d4a1a50 con 0x55c16d49ea70 2024-08-03T20:58:16.535 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds_snaptable:head# 2024-08-03T20:58:16.536 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:16.535+0000 7f5d57400640 1 -- 172.21.15.32:0/92850940 <== osd.7 v2:172.21.15.118:6818/653000048 3 ==== osd_op_reply(82 mds_snaptable [create] v112'134 uv134 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f5d4800e040 con 0x55c16d49ea70 2024-08-03T20:58:16.536 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:16.535+0000 7f5d5ed1cf40 1 -- 172.21.15.32:0/92850940 --> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] -- osd_op(unknown.0.0:83 12.d 12:b50e409b:::mds_snaptable:head [write 0~70 in=70b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112) -- 0x55c16d4a1a50 con 0x55c16d49ea70 2024-08-03T20:58:16.537 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:16.536+0000 7f5d57400640 1 -- 172.21.15.32:0/92850940 <== osd.7 v2:172.21.15.118:6818/653000048 4 ==== osd_op_reply(83 mds_snaptable [write 0~70] v112'135 uv135 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f5d4800e040 con 0x55c16d49ea70 2024-08-03T20:58:16.537 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:16.537+0000 7f5d5ed1cf40 1 -- 172.21.15.32:0/92850940 --> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] -- osd_op(unknown.0.0:84 12.d 12:b50e409b:::mds_snaptable:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112) -- 0x55c16d4a1a50 con 0x55c16d49ea70 2024-08-03T20:58:16.538 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:16.538+0000 7f5d57400640 1 -- 172.21.15.32:0/92850940 <== osd.7 v2:172.21.15.118:6818/653000048 5 ==== osd_op_reply(84 mds_snaptable [omap-set-header] v112'136 uv136 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f5d4800e040 con 0x55c16d49ea70 2024-08-03T20:58:16.538 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:16.538+0000 7f5d5ed1cf40 1 -- 172.21.15.32:0/92850940 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:85 12.1d 12:b871830b:::605.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112) -- 0x55c16d4a1a50 con 0x55c16d3b6150 2024-08-03T20:58:16.540 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:16.539+0000 7f5d57e00640 1 -- 172.21.15.32:0/92850940 <== osd.0 v2:172.21.15.32:6816/3066156116 44 ==== osd_op_reply(85 605.00000000 [create] v112'159 uv158 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f5d50019040 con 0x55c16d3b6150 2024-08-03T20:58:16.540 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:16.539+0000 7f5d5ed1cf40 1 -- 172.21.15.32:0/92850940 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:86 12.1d 12:b871830b:::605.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e112) -- 0x55c16d4a1a50 con 0x55c16d3b6150 2024-08-03T20:58:16.541 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:16.541+0000 7f5d57e00640 1 -- 172.21.15.32:0/92850940 <== osd.0 v2:172.21.15.32:6816/3066156116 45 ==== osd_op_reply(86 605.00000000 [delete] v112'160 uv160 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f5d50019040 con 0x55c16d3b6150 2024-08-03T20:58:16.541 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:16.541+0000 7f5d5ed1cf40 1 -- 172.21.15.32:0/92850940 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:87 12.1d 12:b871830b:::605.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112) -- 0x55c16d4a1a50 con 0x55c16d3b6150 2024-08-03T20:58:16.542 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:16.542+0000 7f5d57e00640 1 -- 172.21.15.32:0/92850940 <== osd.0 v2:172.21.15.32:6816/3066156116 46 ==== osd_op_reply(87 605.00000000 [create] v112'161 uv161 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f5d50019040 con 0x55c16d3b6150 2024-08-03T20:58:16.542 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::605.00000000:head# 2024-08-03T20:58:16.543 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:16.542+0000 7f5d5ed1cf40 1 -- 172.21.15.32:0/92850940 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:88 12.1d 12:b871830b:::605.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112) -- 0x55c16d4a2a60 con 0x55c16d3b6150 2024-08-03T20:58:16.543 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:16.543+0000 7f5d57e00640 1 -- 172.21.15.32:0/92850940 <== osd.0 v2:172.21.15.32:6816/3066156116 47 ==== osd_op_reply(88 605.00000000 [setxattr (30)] v112'162 uv162 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f5d50019040 con 0x55c16d3b6150 2024-08-03T20:58:16.543 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:16.543+0000 7f5d5ed1cf40 1 -- 172.21.15.32:0/92850940 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:89 12.1d 12:b871830b:::605.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112) -- 0x55c16d4a2a60 con 0x55c16d3b6150 2024-08-03T20:58:16.545 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:16.544+0000 7f5d57e00640 1 -- 172.21.15.32:0/92850940 <== osd.0 v2:172.21.15.32:6816/3066156116 48 ==== osd_op_reply(89 605.00000000 [setxattr (62)] v112'163 uv163 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f5d50019040 con 0x55c16d3b6150 2024-08-03T20:58:16.545 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:16.544+0000 7f5d5ed1cf40 1 -- 172.21.15.32:0/92850940 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:90 12.1d 12:b871830b:::605.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112) -- 0x55c16d4a1a50 con 0x55c16d3b6150 2024-08-03T20:58:16.546 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:16.546+0000 7f5d57e00640 1 -- 172.21.15.32:0/92850940 <== osd.0 v2:172.21.15.32:6816/3066156116 49 ==== osd_op_reply(90 605.00000000 [omap-set-header] v112'164 uv164 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f5d50019040 con 0x55c16d3b6150 2024-08-03T20:58:16.546 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:16.546+0000 7f5d5ed1cf40 1 -- 172.21.15.32:0/92850940 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:91 12.13 12:c91d4a1d:::mds0_inotable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112) -- 0x55c16d4a1a50 con 0x55c16d4971b0 2024-08-03T20:58:16.547 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:16.547+0000 7f5d56a00640 1 -- 172.21.15.32:0/92850940 <== osd.1 v2:172.21.15.32:6824/1551241759 5 ==== osd_op_reply(91 mds0_inotable [create] v112'263 uv257 ondisk = -17 ((17) File exists)) ==== 157+0+0 (crc 0 0 0) 0x7f5d440a0300 con 0x55c16d4971b0 2024-08-03T20:58:16.548 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:16.547+0000 7f5d5ed1cf40 1 -- 172.21.15.32:0/92850940 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:92 12.13 12:c91d4a1d:::mds0_inotable:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e112) -- 0x55c16d4a1df0 con 0x55c16d4971b0 2024-08-03T20:58:16.549 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:16.549+0000 7f5d56a00640 1 -- 172.21.15.32:0/92850940 <== osd.1 v2:172.21.15.32:6824/1551241759 6 ==== osd_op_reply(92 mds0_inotable [delete] v112'264 uv264 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f5d440a0300 con 0x55c16d4971b0 2024-08-03T20:58:16.549 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:16.549+0000 7f5d5ed1cf40 1 -- 172.21.15.32:0/92850940 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:93 12.13 12:c91d4a1d:::mds0_inotable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112) -- 0x55c16d4a1df0 con 0x55c16d4971b0 2024-08-03T20:58:16.550 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:16.550+0000 7f5d56a00640 1 -- 172.21.15.32:0/92850940 <== osd.1 v2:172.21.15.32:6824/1551241759 7 ==== osd_op_reply(93 mds0_inotable [create] v112'265 uv265 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f5d440a0300 con 0x55c16d4971b0 2024-08-03T20:58:16.550 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds0_inotable:head# 2024-08-03T20:58:16.551 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:16.550+0000 7f5d5ed1cf40 1 -- 172.21.15.32:0/92850940 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:94 12.13 12:c91d4a1d:::mds0_inotable:head [write 0~34 in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112) -- 0x55c16d49aac0 con 0x55c16d4971b0 2024-08-03T20:58:16.552 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:16.551+0000 7f5d56a00640 1 -- 172.21.15.32:0/92850940 <== osd.1 v2:172.21.15.32:6824/1551241759 8 ==== osd_op_reply(94 mds0_inotable [write 0~34] v112'266 uv266 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f5d440a0300 con 0x55c16d4971b0 2024-08-03T20:58:16.552 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:16.551+0000 7f5d5ed1cf40 1 -- 172.21.15.32:0/92850940 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:95 12.13 12:c91d4a1d:::mds0_inotable:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112) -- 0x55c16d49aac0 con 0x55c16d4971b0 2024-08-03T20:58:16.553 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:16.553+0000 7f5d56a00640 1 -- 172.21.15.32:0/92850940 <== osd.1 v2:172.21.15.32:6824/1551241759 9 ==== osd_op_reply(95 mds0_inotable [omap-set-header] v112'267 uv267 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f5d440a0300 con 0x55c16d4971b0 2024-08-03T20:58:16.553 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:16.553+0000 7f5d5ed1cf40 1 -- 172.21.15.32:0/92850940 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:96 12.13 12:cd5a64a3:::100.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112) -- 0x55c16d49aac0 con 0x55c16d4971b0 2024-08-03T20:58:16.554 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:16.554+0000 7f5d56a00640 1 -- 172.21.15.32:0/92850940 <== osd.1 v2:172.21.15.32:6824/1551241759 10 ==== osd_op_reply(96 100.00000000 [create] v112'268 uv262 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f5d440a0300 con 0x55c16d4971b0 2024-08-03T20:58:16.555 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:16.554+0000 7f5d5ed1cf40 1 -- 172.21.15.32:0/92850940 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:97 12.13 12:cd5a64a3:::100.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e112) -- 0x55c16d49aac0 con 0x55c16d4971b0 2024-08-03T20:58:16.556 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:16.555+0000 7f5d56a00640 1 -- 172.21.15.32:0/92850940 <== osd.1 v2:172.21.15.32:6824/1551241759 11 ==== osd_op_reply(97 100.00000000 [delete] v112'269 uv269 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f5d440a0300 con 0x55c16d4971b0 2024-08-03T20:58:16.556 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:16.555+0000 7f5d5ed1cf40 1 -- 172.21.15.32:0/92850940 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:98 12.13 12:cd5a64a3:::100.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112) -- 0x55c16d49aac0 con 0x55c16d4971b0 2024-08-03T20:58:16.557 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::100.00000000:head# 2024-08-03T20:58:16.557 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:16.557+0000 7f5d56a00640 1 -- 172.21.15.32:0/92850940 <== osd.1 v2:172.21.15.32:6824/1551241759 12 ==== osd_op_reply(98 100.00000000 [create] v112'270 uv270 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f5d440a0300 con 0x55c16d4971b0 2024-08-03T20:58:16.557 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:16.557+0000 7f5d5ed1cf40 1 -- 172.21.15.32:0/92850940 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:99 12.13 12:cd5a64a3:::100.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112) -- 0x55c16d49aac0 con 0x55c16d4971b0 2024-08-03T20:58:16.558 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:16.558+0000 7f5d56a00640 1 -- 172.21.15.32:0/92850940 <== osd.1 v2:172.21.15.32:6824/1551241759 13 ==== osd_op_reply(99 100.00000000 [omap-set-header] v112'271 uv271 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f5d440a0300 con 0x55c16d4971b0 2024-08-03T20:58:16.558 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:16.558+0000 7f5d5ed1cf40 1 -- 172.21.15.32:0/92850940 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:100 12.13 12:cd5a64a3:::100.00000000:head [omap-set-vals in=5194b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112) -- 0x55c16d4a4c40 con 0x55c16d4971b0 2024-08-03T20:58:16.560 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:16.559+0000 7f5d56a00640 1 -- 172.21.15.32:0/92850940 <== osd.1 v2:172.21.15.32:6824/1551241759 14 ==== osd_op_reply(100 100.00000000 [omap-set-vals] v112'272 uv272 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f5d440a0300 con 0x55c16d4971b0 2024-08-03T20:58:16.560 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:16.559+0000 7f5d5ed1cf40 1 --2- 172.21.15.32:0/92850940 >> [v2:172.21.15.32:6800/9088303,v1:172.21.15.32:6801/9088303] conn(0x55c16d4a2920 0x55c16d4a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:58:16.560 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:16.559+0000 7f5d5ed1cf40 1 -- 172.21.15.32:0/92850940 --> [v2:172.21.15.32:6800/9088303,v1:172.21.15.32:6801/9088303] -- osd_op(unknown.0.0:101 12.b 12:d0630e4c:::mds0_sessionmap:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112) -- 0x55c16d4a5300 con 0x55c16d4a2920 2024-08-03T20:58:16.560 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:16.560+0000 7f5d56a00640 1 --2- 172.21.15.32:0/92850940 >> [v2:172.21.15.32:6800/9088303,v1:172.21.15.32:6801/9088303] conn(0x55c16d4a2920 0x55c16d4a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:58:16.561 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:16.560+0000 7f5d56a00640 1 --2- 172.21.15.32:0/92850940 >> [v2:172.21.15.32:6800/9088303,v1:172.21.15.32:6801/9088303] conn(0x55c16d4a2920 0x55c16d4a4d40 crc :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:58:16.562 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:16.562+0000 7f5d56a00640 1 -- 172.21.15.32:0/92850940 <== osd.3 v2:172.21.15.32:6800/9088303 1 ==== osd_op_reply(101 mds0_sessionmap [create] v112'116 uv114 ondisk = -17 ((17) File exists)) ==== 159+0+0 (crc 0 0 0) 0x7f5d440a0300 con 0x55c16d4a2920 2024-08-03T20:58:16.562 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:16.562+0000 7f5d5ed1cf40 1 -- 172.21.15.32:0/92850940 --> [v2:172.21.15.32:6800/9088303,v1:172.21.15.32:6801/9088303] -- osd_op(unknown.0.0:102 12.b 12:d0630e4c:::mds0_sessionmap:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e112) -- 0x55c16d4a5300 con 0x55c16d4a2920 2024-08-03T20:58:16.563 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:16.563+0000 7f5d56a00640 1 -- 172.21.15.32:0/92850940 <== osd.3 v2:172.21.15.32:6800/9088303 2 ==== osd_op_reply(102 mds0_sessionmap [delete] v112'117 uv117 ondisk = 0) ==== 159+0+0 (crc 0 0 0) 0x7f5d440a0300 con 0x55c16d4a2920 2024-08-03T20:58:16.564 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:16.563+0000 7f5d5ed1cf40 1 -- 172.21.15.32:0/92850940 --> [v2:172.21.15.32:6800/9088303,v1:172.21.15.32:6801/9088303] -- osd_op(unknown.0.0:103 12.b 12:d0630e4c:::mds0_sessionmap:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112) -- 0x55c16d4a5300 con 0x55c16d4a2920 2024-08-03T20:58:16.565 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds0_sessionmap:head# 2024-08-03T20:58:16.565 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:16.564+0000 7f5d56a00640 1 -- 172.21.15.32:0/92850940 <== osd.3 v2:172.21.15.32:6800/9088303 3 ==== osd_op_reply(103 mds0_sessionmap [create] v112'118 uv118 ondisk = 0) ==== 159+0+0 (crc 0 0 0) 0x7f5d440a0300 con 0x55c16d4a2920 2024-08-03T20:58:16.565 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:16.564+0000 7f5d5ed1cf40 1 -- 172.21.15.32:0/92850940 --> [v2:172.21.15.32:6800/9088303,v1:172.21.15.32:6801/9088303] -- osd_op(unknown.0.0:104 12.b 12:d0630e4c:::mds0_sessionmap:head [omap-set-header in=14b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112) -- 0x55c16d4a5330 con 0x55c16d4a2920 2024-08-03T20:58:16.566 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:16.566+0000 7f5d56a00640 1 -- 172.21.15.32:0/92850940 <== osd.3 v2:172.21.15.32:6800/9088303 4 ==== osd_op_reply(104 mds0_sessionmap [omap-set-header] v112'119 uv119 ondisk = 0) ==== 159+0+0 (crc 0 0 0) 0x7f5d440a0300 con 0x55c16d4a2920 2024-08-03T20:58:16.567 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:16.566+0000 7f5d5ed1cf40 1 -- 172.21.15.32:0/92850940 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:105 12.7 12:e0b41d06:::609.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112) -- 0x55c16d4a5330 con 0x55c16d4971b0 2024-08-03T20:58:16.568 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:16.567+0000 7f5d56a00640 1 -- 172.21.15.32:0/92850940 <== osd.1 v2:172.21.15.32:6824/1551241759 15 ==== osd_op_reply(105 609.00000000 [create] v112'159 uv158 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f5d440a0300 con 0x55c16d4971b0 2024-08-03T20:58:16.568 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:16.567+0000 7f5d5ed1cf40 1 -- 172.21.15.32:0/92850940 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:106 12.7 12:e0b41d06:::609.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e112) -- 0x55c16d4a5330 con 0x55c16d4971b0 2024-08-03T20:58:16.569 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:16.569+0000 7f5d56a00640 1 -- 172.21.15.32:0/92850940 <== osd.1 v2:172.21.15.32:6824/1551241759 16 ==== osd_op_reply(106 609.00000000 [delete] v112'160 uv160 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f5d440a0300 con 0x55c16d4971b0 2024-08-03T20:58:16.569 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:16.569+0000 7f5d5ed1cf40 1 -- 172.21.15.32:0/92850940 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:107 12.7 12:e0b41d06:::609.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112) -- 0x55c16d4a5280 con 0x55c16d4971b0 2024-08-03T20:58:16.571 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::609.00000000:head# 2024-08-03T20:58:16.571 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:16.570+0000 7f5d56a00640 1 -- 172.21.15.32:0/92850940 <== osd.1 v2:172.21.15.32:6824/1551241759 17 ==== osd_op_reply(107 609.00000000 [create] v112'161 uv161 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f5d440a0300 con 0x55c16d4971b0 2024-08-03T20:58:16.571 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:16.570+0000 7f5d5ed1cf40 1 -- 172.21.15.32:0/92850940 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:108 12.7 12:e0b41d06:::609.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112) -- 0x55c16d4a62f0 con 0x55c16d4971b0 2024-08-03T20:58:16.572 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:16.572+0000 7f5d56a00640 1 -- 172.21.15.32:0/92850940 <== osd.1 v2:172.21.15.32:6824/1551241759 18 ==== osd_op_reply(108 609.00000000 [setxattr (30)] v112'162 uv162 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f5d440a0300 con 0x55c16d4971b0 2024-08-03T20:58:16.572 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:16.572+0000 7f5d5ed1cf40 1 -- 172.21.15.32:0/92850940 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:109 12.7 12:e0b41d06:::609.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112) -- 0x55c16d4a62b0 con 0x55c16d4971b0 2024-08-03T20:58:16.574 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:16.573+0000 7f5d56a00640 1 -- 172.21.15.32:0/92850940 <== osd.1 v2:172.21.15.32:6824/1551241759 19 ==== osd_op_reply(109 609.00000000 [setxattr (62)] v112'163 uv163 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f5d440a0300 con 0x55c16d4971b0 2024-08-03T20:58:16.574 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:16.573+0000 7f5d5ed1cf40 1 -- 172.21.15.32:0/92850940 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:110 12.7 12:e0b41d06:::609.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112) -- 0x55c16d4a62b0 con 0x55c16d4971b0 2024-08-03T20:58:16.575 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:16.575+0000 7f5d56a00640 1 -- 172.21.15.32:0/92850940 <== osd.1 v2:172.21.15.32:6824/1551241759 20 ==== osd_op_reply(110 609.00000000 [omap-set-header] v112'164 uv164 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f5d440a0300 con 0x55c16d4971b0 2024-08-03T20:58:16.575 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:16.575+0000 7f5d5ed1cf40 1 -- 172.21.15.32:0/92850940 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:111 12.f 12:f1f69726:::400.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112) -- 0x55c16d4a62b0 con 0x55c16d49b810 2024-08-03T20:58:16.577 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:16.577+0000 7f5d57e00640 1 -- 172.21.15.32:0/92850940 <== osd.6 v2:172.21.15.118:6825/801947386 12 ==== osd_op_reply(111 400.00000000 [create] v112'135 uv134 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f5d50019040 con 0x55c16d49b810 2024-08-03T20:58:16.577 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:16.577+0000 7f5d5ed1cf40 1 -- 172.21.15.32:0/92850940 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:112 12.f 12:f1f69726:::400.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e112) -- 0x55c16d4a62b0 con 0x55c16d49b810 2024-08-03T20:58:16.578 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:16.578+0000 7f5d57e00640 1 -- 172.21.15.32:0/92850940 <== osd.6 v2:172.21.15.118:6825/801947386 13 ==== osd_op_reply(112 400.00000000 [delete] v112'136 uv136 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f5d50019040 con 0x55c16d49b810 2024-08-03T20:58:16.578 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:16.578+0000 7f5d5ed1cf40 1 -- 172.21.15.32:0/92850940 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:113 12.f 12:f1f69726:::400.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112) -- 0x55c16d4a62b0 con 0x55c16d49b810 2024-08-03T20:58:16.580 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::400.00000000:head# 2024-08-03T20:58:16.580 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:16.579+0000 7f5d57e00640 1 -- 172.21.15.32:0/92850940 <== osd.6 v2:172.21.15.118:6825/801947386 14 ==== osd_op_reply(113 400.00000000 [create] v112'137 uv137 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f5d50019040 con 0x55c16d49b810 2024-08-03T20:58:16.580 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:16.579+0000 7f5d5ed1cf40 1 -- 172.21.15.32:0/92850940 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:114 12.f 12:f1f69726:::400.00000000:head [write 0~22 in=22b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112) -- 0x55c16d4a5280 con 0x55c16d49b810 2024-08-03T20:58:16.581 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:16.581+0000 7f5d57e00640 1 -- 172.21.15.32:0/92850940 <== osd.6 v2:172.21.15.118:6825/801947386 15 ==== osd_op_reply(114 400.00000000 [write 0~22] v112'138 uv138 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f5d50019040 con 0x55c16d49b810 2024-08-03T20:58:16.581 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:16.581+0000 7f5d5ed1cf40 1 -- 172.21.15.32:0/92850940 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:115 12.f 12:f1f69726:::400.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112) -- 0x55c16d4a5280 con 0x55c16d49b810 2024-08-03T20:58:16.583 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:16.582+0000 7f5d57e00640 1 -- 172.21.15.32:0/92850940 <== osd.6 v2:172.21.15.118:6825/801947386 16 ==== osd_op_reply(115 400.00000000 [omap-set-header] v112'139 uv139 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f5d50019040 con 0x55c16d49b810 2024-08-03T20:58:16.583 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:16.583+0000 7f5d5ed1cf40 1 -- 172.21.15.32:0/92850940 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:116 12.1f 12:fe07bba1:::100.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112) -- 0x55c16d4a5280 con 0x55c16d49b810 2024-08-03T20:58:16.585 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:16.584+0000 7f5d57e00640 1 -- 172.21.15.32:0/92850940 <== osd.6 v2:172.21.15.118:6825/801947386 17 ==== osd_op_reply(116 100.00000000.inode [create] v112'327 uv318 ondisk = -17 ((17) File exists)) ==== 162+0+0 (crc 0 0 0) 0x7f5d50019040 con 0x55c16d49b810 2024-08-03T20:58:16.585 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:16.584+0000 7f5d5ed1cf40 1 -- 172.21.15.32:0/92850940 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:117 12.1f 12:fe07bba1:::100.00000000.inode:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e112) -- 0x55c16d4a5280 con 0x55c16d49b810 2024-08-03T20:58:16.586 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:16.586+0000 7f5d57e00640 1 -- 172.21.15.32:0/92850940 <== osd.6 v2:172.21.15.118:6825/801947386 18 ==== osd_op_reply(117 100.00000000.inode [delete] v112'328 uv328 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7f5d50019040 con 0x55c16d49b810 2024-08-03T20:58:16.586 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:16.586+0000 7f5d5ed1cf40 1 -- 172.21.15.32:0/92850940 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:118 12.1f 12:fe07bba1:::100.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112) -- 0x55c16d4a5280 con 0x55c16d49b810 2024-08-03T20:58:16.588 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::100.00000000.inode:head# 2024-08-03T20:58:16.588 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:16.587+0000 7f5d57e00640 1 -- 172.21.15.32:0/92850940 <== osd.6 v2:172.21.15.118:6825/801947386 19 ==== osd_op_reply(118 100.00000000.inode [create] v112'329 uv329 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7f5d50019040 con 0x55c16d49b810 2024-08-03T20:58:16.588 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:16.587+0000 7f5d5ed1cf40 1 -- 172.21.15.32:0/92850940 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:119 12.1f 12:fe07bba1:::100.00000000.inode:head [write 0~582 in=582b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112) -- 0x55c16d4a5280 con 0x55c16d49b810 2024-08-03T20:58:16.589 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:16.589+0000 7f5d57e00640 1 -- 172.21.15.32:0/92850940 <== osd.6 v2:172.21.15.118:6825/801947386 20 ==== osd_op_reply(119 100.00000000.inode [write 0~582] v112'330 uv330 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7f5d50019040 con 0x55c16d49b810 2024-08-03T20:58:16.590 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:16.589+0000 7f5d5ed1cf40 1 -- 172.21.15.32:0/92850940 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:120 12.1f 12:fe07bba1:::100.00000000.inode:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112) -- 0x55c16d4a5280 con 0x55c16d49b810 2024-08-03T20:58:16.591 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:16.590+0000 7f5d57e00640 1 -- 172.21.15.32:0/92850940 <== osd.6 v2:172.21.15.118:6825/801947386 21 ==== osd_op_reply(120 100.00000000.inode [omap-set-header] v112'331 uv331 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7f5d50019040 con 0x55c16d49b810 2024-08-03T20:58:16.591 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:16.590+0000 7f5d5ed1cf40 1 -- 172.21.15.32:0/92850940 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:121 12.1f 12:ff5b34d6:::1.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112) -- 0x55c16d4a5280 con 0x55c16d49b810 2024-08-03T20:58:16.592 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:16.592+0000 7f5d57e00640 1 -- 172.21.15.32:0/92850940 <== osd.6 v2:172.21.15.118:6825/801947386 22 ==== osd_op_reply(121 1.00000000 [create] v112'332 uv325 ondisk = -17 ((17) File exists)) ==== 154+0+0 (crc 0 0 0) 0x7f5d50019040 con 0x55c16d49b810 2024-08-03T20:58:16.592 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:16.592+0000 7f5d5ed1cf40 1 -- 172.21.15.32:0/92850940 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:122 12.1f 12:ff5b34d6:::1.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e112) -- 0x55c16d4a5280 con 0x55c16d49b810 2024-08-03T20:58:16.594 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:16.593+0000 7f5d57e00640 1 -- 172.21.15.32:0/92850940 <== osd.6 v2:172.21.15.118:6825/801947386 23 ==== osd_op_reply(122 1.00000000 [delete] v112'333 uv333 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7f5d50019040 con 0x55c16d49b810 2024-08-03T20:58:16.594 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:16.593+0000 7f5d5ed1cf40 1 -- 172.21.15.32:0/92850940 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:123 12.1f 12:ff5b34d6:::1.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112) -- 0x55c16d4a5280 con 0x55c16d49b810 2024-08-03T20:58:16.595 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::1.00000000:head# 2024-08-03T20:58:16.595 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:16.594+0000 7f5d57e00640 1 -- 172.21.15.32:0/92850940 <== osd.6 v2:172.21.15.118:6825/801947386 24 ==== osd_op_reply(123 1.00000000 [create] v112'334 uv334 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7f5d50019040 con 0x55c16d49b810 2024-08-03T20:58:16.595 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:16.595+0000 7f5d5ed1cf40 1 -- 172.21.15.32:0/92850940 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:124 12.1f 12:ff5b34d6:::1.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112) -- 0x55c16d4a5280 con 0x55c16d49b810 2024-08-03T20:58:16.596 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:16.596+0000 7f5d57e00640 1 -- 172.21.15.32:0/92850940 <== osd.6 v2:172.21.15.118:6825/801947386 25 ==== osd_op_reply(124 1.00000000 [setxattr (30)] v112'335 uv335 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7f5d50019040 con 0x55c16d49b810 2024-08-03T20:58:16.596 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:16.596+0000 7f5d5ed1cf40 1 -- 172.21.15.32:0/92850940 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:125 12.1f 12:ff5b34d6:::1.00000000:head [setxattr parent (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112) -- 0x55c16d4a5280 con 0x55c16d49b810 2024-08-03T20:58:16.598 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:16.598+0000 7f5d57e00640 1 -- 172.21.15.32:0/92850940 <== osd.6 v2:172.21.15.118:6825/801947386 26 ==== osd_op_reply(125 1.00000000 [setxattr (30)] v112'336 uv336 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7f5d50019040 con 0x55c16d49b810 2024-08-03T20:58:16.598 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:16.598+0000 7f5d5ed1cf40 1 -- 172.21.15.32:0/92850940 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:126 12.1f 12:ff5b34d6:::1.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112) -- 0x55c16d4a5280 con 0x55c16d49b810 2024-08-03T20:58:16.599 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:16.599+0000 7f5d57e00640 1 -- 172.21.15.32:0/92850940 <== osd.6 v2:172.21.15.118:6825/801947386 27 ==== osd_op_reply(126 1.00000000 [omap-set-header] v112'337 uv337 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7f5d50019040 con 0x55c16d49b810 2024-08-03T20:58:16.599 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:16.599+0000 7f5d5ed1cf40 1 -- 172.21.15.32:0/92850940 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:127 12.1f 12:ff5b34d6:::1.00000000:head [omap-set-vals in=584b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112) -- 0x55c16d4a5af0 con 0x55c16d49b810 2024-08-03T20:58:16.602 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:16.600+0000 7f5d57e00640 1 -- 172.21.15.32:0/92850940 <== osd.6 v2:172.21.15.118:6825/801947386 28 ==== osd_op_reply(127 1.00000000 [omap-set-vals] v112'338 uv338 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7f5d50019040 con 0x55c16d49b810 2024-08-03T20:58:16.602 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:16.600+0000 7f5d5ed1cf40 1 -- 172.21.15.32:0/92850940 >> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] conn(0x55c16d3b6150 msgr2=0x55c16d35f4a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:58:16.602 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:16.600+0000 7f5d5ed1cf40 1 --2- 172.21.15.32:0/92850940 >> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] conn(0x55c16d3b6150 0x55c16d35f4a0 crc :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:16.602 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:16.600+0000 7f5d5ed1cf40 1 -- 172.21.15.32:0/92850940 >> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] conn(0x55c16d4971b0 msgr2=0x55c16d4995d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:58:16.602 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:16.600+0000 7f5d5ed1cf40 1 --2- 172.21.15.32:0/92850940 >> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] conn(0x55c16d4971b0 0x55c16d4995d0 crc :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:16.602 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:16.600+0000 7f5d5ed1cf40 1 -- 172.21.15.32:0/92850940 >> [v2:172.21.15.32:6800/9088303,v1:172.21.15.32:6801/9088303] conn(0x55c16d4a2920 msgr2=0x55c16d4a4d40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:58:16.602 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:16.600+0000 7f5d5ed1cf40 1 --2- 172.21.15.32:0/92850940 >> [v2:172.21.15.32:6800/9088303,v1:172.21.15.32:6801/9088303] conn(0x55c16d4a2920 0x55c16d4a4d40 crc :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:16.602 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:16.600+0000 7f5d5ed1cf40 1 -- 172.21.15.32:0/92850940 >> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] conn(0x55c16d4929e0 msgr2=0x55c16d494e00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:58:16.602 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:16.600+0000 7f5d5ed1cf40 1 --2- 172.21.15.32:0/92850940 >> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] conn(0x55c16d4929e0 0x55c16d494e00 crc :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:16.602 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:16.601+0000 7f5d5ed1cf40 1 -- 172.21.15.32:0/92850940 >> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] conn(0x55c16d49b810 msgr2=0x55c16d49dc30 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:58:16.602 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:16.601+0000 7f5d5ed1cf40 1 --2- 172.21.15.32:0/92850940 >> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] conn(0x55c16d49b810 0x55c16d49dc30 crc :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:16.603 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:16.601+0000 7f5d5ed1cf40 1 -- 172.21.15.32:0/92850940 >> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] conn(0x55c16d49ea70 msgr2=0x55c16d4a0ed0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:58:16.603 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:16.601+0000 7f5d5ed1cf40 1 --2- 172.21.15.32:0/92850940 >> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] conn(0x55c16d49ea70 0x55c16d4a0ed0 crc :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:16.603 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:16.601+0000 7f5d57e00640 1 -- 172.21.15.32:0/92850940 reap_dead start 2024-08-03T20:58:16.603 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:16.601+0000 7f5d57e00640 1 -- 172.21.15.32:0/92850940 reap_dead start 2024-08-03T20:58:16.603 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:16.601+0000 7f5d57e00640 1 -- 172.21.15.32:0/92850940 reap_dead start 2024-08-03T20:58:16.603 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:16.601+0000 7f5d5ed1cf40 1 -- 172.21.15.32:0/92850940 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f5d38075a20 msgr2=0x7f5d38077f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:58:16.603 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:16.601+0000 7f5d5ed1cf40 1 --2- 172.21.15.32:0/92850940 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f5d38075a20 0x7f5d38077f30 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7f5d48000f60 tx=0x7f5d48003050 comp rx=0 tx=0).stop 2024-08-03T20:58:16.603 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:16.601+0000 7f5d5ed1cf40 1 -- 172.21.15.32:0/92850940 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55c16d3cfbe0 msgr2=0x55c16d3ca730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:58:16.603 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:16.601+0000 7f5d5ed1cf40 1 --2- 172.21.15.32:0/92850940 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55c16d3cfbe0 0x55c16d3ca730 secure :-1 s=READY pgs=1732 cs=0 l=1 rev1=1 crypto rx=0x7f5d44046560 tx=0x7f5d44004690 comp rx=0 tx=0).stop 2024-08-03T20:58:16.603 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:16.601+0000 7f5d5ed1cf40 1 -- 172.21.15.32:0/92850940 shutdown_connections 2024-08-03T20:58:16.603 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:16.601+0000 7f5d5ed1cf40 1 --2- 172.21.15.32:0/92850940 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f5d38075a20 0x7f5d38077f30 unknown :-1 s=CLOSED pgs=705 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:16.603 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:16.601+0000 7f5d5ed1cf40 1 --2- 172.21.15.32:0/92850940 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55c16d3cfbe0 0x55c16d3ca730 unknown :-1 s=CLOSED pgs=1732 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:16.603 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:16.601+0000 7f5d5ed1cf40 1 -- 172.21.15.32:0/92850940 >> 172.21.15.32:0/92850940 conn(0x55c16d237ac0 msgr2=0x55c16d3ef830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:58:16.604 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:16.601+0000 7f5d5ed1cf40 1 -- 172.21.15.32:0/92850940 shutdown_connections 2024-08-03T20:58:16.604 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:16.601+0000 7f5d5ed1cf40 1 -- 172.21.15.32:0/92850940 wait complete. 2024-08-03T20:58:16.613 DEBUG:teuthology.orchestra.run.smithi032:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata truncate mds_snaptable 0 2024-08-03T20:58:16.796 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:16.796+0000 7f1d8720cf40 1 Processor -- start 2024-08-03T20:58:16.797 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:16.796+0000 7f1d8720cf40 1 -- start start 2024-08-03T20:58:16.799 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:16.797+0000 7f1d8720cf40 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5592821d5010 0x55928218a4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:58:16.799 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:16.797+0000 7f1d8720cf40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5592821d6f50 0x5592822eeb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:58:16.799 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:16.797+0000 7f1d8720cf40 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55928225f820 0x5592822ef050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:58:16.799 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:16.797+0000 7f1d8720cf40 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x5592820dbe30 con 0x5592821d5010 2024-08-03T20:58:16.799 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:16.797+0000 7f1d8720cf40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x559282139bf0 con 0x5592821d6f50 2024-08-03T20:58:16.800 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:16.797+0000 7f1d8720cf40 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x5592821d56f0 con 0x55928225f820 2024-08-03T20:58:16.800 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:16.798+0000 7f1d7f800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5592821d5010 0x55928218a4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:58:16.800 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:16.798+0000 7f1d7f800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5592821d5010 0x55928218a4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:35924/0 (socket says 172.21.15.32:35924) 2024-08-03T20:58:16.800 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:16.798+0000 7f1d7f800640 1 -- 172.21.15.32:0/2834042685 learned_addr learned my addr 172.21.15.32:0/2834042685 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:58:16.800 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:16.798+0000 7f1d80200640 1 --2- 172.21.15.32:0/2834042685 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55928225f820 0x5592822ef050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:58:16.800 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:16.798+0000 7f1d7ee00640 1 --2- 172.21.15.32:0/2834042685 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5592821d6f50 0x5592822eeb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:58:16.800 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:16.798+0000 7f1d7f800640 1 -- 172.21.15.32:0/2834042685 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55928225f820 msgr2=0x5592822ef050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:58:16.800 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:16.798+0000 7f1d7f800640 1 --2- 172.21.15.32:0/2834042685 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55928225f820 0x5592822ef050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:16.800 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:16.798+0000 7f1d7f800640 1 -- 172.21.15.32:0/2834042685 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5592821d6f50 msgr2=0x5592822eeb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:58:16.800 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:16.798+0000 7f1d7f800640 1 --2- 172.21.15.32:0/2834042685 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5592821d6f50 0x5592822eeb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:16.800 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:16.798+0000 7f1d7f800640 1 -- 172.21.15.32:0/2834042685 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5592822ef590 con 0x5592821d5010 2024-08-03T20:58:16.800 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:16.798+0000 7f1d7f800640 1 --2- 172.21.15.32:0/2834042685 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5592821d5010 0x55928218a4c0 secure :-1 s=READY pgs=1734 cs=0 l=1 rev1=1 crypto rx=0x7f1d7000c600 tx=0x7f1d7000cac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d45102c8d3db1360 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:58:16.800 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:16.799+0000 7f1d7da00640 1 -- 172.21.15.32:0/2834042685 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d7000ab10 con 0x5592821d5010 2024-08-03T20:58:16.801 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:16.799+0000 7f1d7da00640 1 -- 172.21.15.32:0/2834042685 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f1d7000acb0 con 0x5592821d5010 2024-08-03T20:58:16.801 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:16.799+0000 7f1d7da00640 1 -- 172.21.15.32:0/2834042685 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d700156a0 con 0x5592821d5010 2024-08-03T20:58:16.802 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:16.799+0000 7f1d8720cf40 1 -- 172.21.15.32:0/2834042685 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5592821d5010 msgr2=0x55928218a4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:58:16.802 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:16.799+0000 7f1d8720cf40 1 --2- 172.21.15.32:0/2834042685 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5592821d5010 0x55928218a4c0 secure :-1 s=READY pgs=1734 cs=0 l=1 rev1=1 crypto rx=0x7f1d7000c600 tx=0x7f1d7000cac0 comp rx=0 tx=0).stop 2024-08-03T20:58:16.802 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:16.799+0000 7f1d8720cf40 1 -- 172.21.15.32:0/2834042685 shutdown_connections 2024-08-03T20:58:16.802 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:16.799+0000 7f1d8720cf40 1 --2- 172.21.15.32:0/2834042685 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5592821d6f50 0x5592822eeb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:16.802 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:16.799+0000 7f1d8720cf40 1 --2- 172.21.15.32:0/2834042685 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55928225f820 0x5592822ef050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:16.802 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:16.799+0000 7f1d8720cf40 1 --2- 172.21.15.32:0/2834042685 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5592821d5010 0x55928218a4c0 unknown :-1 s=CLOSED pgs=1734 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:16.802 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:16.799+0000 7f1d8720cf40 1 -- 172.21.15.32:0/2834042685 >> 172.21.15.32:0/2834042685 conn(0x5592820adac0 msgr2=0x559282262430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:58:16.802 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:16.799+0000 7f1d8720cf40 1 -- 172.21.15.32:0/2834042685 shutdown_connections 2024-08-03T20:58:16.802 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:16.799+0000 7f1d8720cf40 1 -- 172.21.15.32:0/2834042685 wait complete. 2024-08-03T20:58:16.802 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:16.800+0000 7f1d8720cf40 1 Processor -- start 2024-08-03T20:58:16.803 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:16.801+0000 7f1d8720cf40 1 -- start start 2024-08-03T20:58:16.804 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:16.802+0000 7f1d8720cf40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5592821d5010 0x559282255710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:58:16.804 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:16.802+0000 7f1d8720cf40 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x5592821d6f50 0x55928218a4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:58:16.804 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:16.802+0000 7f1d8720cf40 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55928225f820 0x55928218aa00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:58:16.804 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:16.802+0000 7f1d8720cf40 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x5592822eac20 con 0x55928225f820 2024-08-03T20:58:16.804 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:16.802+0000 7f1d8720cf40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x5592822574f0 con 0x5592821d5010 2024-08-03T20:58:16.805 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:16.802+0000 7f1d8720cf40 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x559282257a70 con 0x5592821d6f50 2024-08-03T20:58:16.805 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:16.802+0000 7f1d80200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55928225f820 0x55928218aa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:58:16.805 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:16.802+0000 7f1d80200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55928225f820 0x55928218aa00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:35936/0 (socket says 172.21.15.32:35936) 2024-08-03T20:58:16.805 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:16.802+0000 7f1d7ee00640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x5592821d6f50 0x55928218a4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:58:16.805 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:16.802+0000 7f1d80200640 1 -- 172.21.15.32:0/4144819614 learned_addr learned my addr 172.21.15.32:0/4144819614 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:58:16.805 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:16.802+0000 7f1d7f800640 1 --2- 172.21.15.32:0/4144819614 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5592821d5010 0x559282255710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:58:16.805 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:16.803+0000 7f1d80200640 1 -- 172.21.15.32:0/4144819614 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x5592821d6f50 msgr2=0x55928218a4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:58:16.805 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:16.803+0000 7f1d80200640 1 --2- 172.21.15.32:0/4144819614 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x5592821d6f50 0x55928218a4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:16.805 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:16.803+0000 7f1d80200640 1 -- 172.21.15.32:0/4144819614 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5592821d5010 msgr2=0x559282255710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:58:16.805 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:16.803+0000 7f1d80200640 1 --2- 172.21.15.32:0/4144819614 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5592821d5010 0x559282255710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:16.805 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:16.803+0000 7f1d80200640 1 -- 172.21.15.32:0/4144819614 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559282257790 con 0x55928225f820 2024-08-03T20:58:16.805 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:16.803+0000 7f1d7ee00640 1 --2- 172.21.15.32:0/4144819614 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x5592821d6f50 0x55928218a4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-03T20:58:16.805 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:16.803+0000 7f1d80200640 1 --2- 172.21.15.32:0/4144819614 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55928225f820 0x55928218aa00 secure :-1 s=READY pgs=1735 cs=0 l=1 rev1=1 crypto rx=0x7f1d78006fd0 tx=0x7f1d7800d5f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:58:16.805 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:16.803+0000 7f1d77e00640 1 -- 172.21.15.32:0/4144819614 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d7800de20 con 0x55928225f820 2024-08-03T20:58:16.806 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:16.803+0000 7f1d77e00640 1 -- 172.21.15.32:0/4144819614 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f1d78018070 con 0x55928225f820 2024-08-03T20:58:16.806 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:16.803+0000 7f1d77e00640 1 -- 172.21.15.32:0/4144819614 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d78012980 con 0x55928225f820 2024-08-03T20:58:16.806 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:16.803+0000 7f1d8720cf40 1 -- 172.21.15.32:0/4144819614 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x5592822eebb0 con 0x55928225f820 2024-08-03T20:58:16.806 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:16.804+0000 7f1d8720cf40 1 -- 172.21.15.32:0/4144819614 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5592822ea960 con 0x55928225f820 2024-08-03T20:58:16.806 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:16.806+0000 7f1d77e00640 1 -- 172.21.15.32:0/4144819614 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f1d78012b20 con 0x55928225f820 2024-08-03T20:58:16.807 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:16.807+0000 7f1d77e00640 1 --2- 172.21.15.32:0/4144819614 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f1d60075a20 0x7f1d60077f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:58:16.807 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:16.807+0000 7f1d77e00640 1 -- 172.21.15.32:0/4144819614 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(112..112 src has 1..112) ==== 8176+0+0 (secure 0 0 0) 0x7f1d78097f90 con 0x55928225f820 2024-08-03T20:58:16.808 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:16.807+0000 7f1d8720cf40 1 --2- 172.21.15.32:0/4144819614 >> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] conn(0x5592822e9ed0 0x5592822ea2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:58:16.808 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:16.807+0000 7f1d8720cf40 1 -- 172.21.15.32:0/4144819614 --> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] -- osd_op(unknown.0.0:1 12.d 12:b50e409b:::mds_snaptable:head [truncate 0] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112) -- 0x5592822fb060 con 0x5592822e9ed0 2024-08-03T20:58:16.808 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:16.808+0000 7f1d7f800640 1 --2- 172.21.15.32:0/4144819614 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f1d60075a20 0x7f1d60077f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:58:16.808 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:16.808+0000 7f1d7ee00640 1 --2- 172.21.15.32:0/4144819614 >> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] conn(0x5592822e9ed0 0x5592822ea2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:58:16.809 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:16.808+0000 7f1d7f800640 1 --2- 172.21.15.32:0/4144819614 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f1d60075a20 0x7f1d60077f30 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x7f1d70004160 tx=0x7f1d70017040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:58:16.809 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:16.808+0000 7f1d7ee00640 1 --2- 172.21.15.32:0/4144819614 >> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] conn(0x5592822e9ed0 0x5592822ea2e0 crc :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:58:16.811 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:16.810+0000 7f1d7ee00640 1 -- 172.21.15.32:0/4144819614 <== osd.7 v2:172.21.15.118:6818/653000048 1 ==== osd_op_reply(1 mds_snaptable [truncate 0] v112'137 uv137 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f1d6c04eab0 con 0x5592822e9ed0 2024-08-03T20:58:16.811 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:16.810+0000 7f1d8720cf40 1 -- 172.21.15.32:0/4144819614 >> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] conn(0x5592822e9ed0 msgr2=0x5592822ea2e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:58:16.811 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:16.810+0000 7f1d8720cf40 1 --2- 172.21.15.32:0/4144819614 >> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] conn(0x5592822e9ed0 0x5592822ea2e0 crc :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:16.811 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:16.810+0000 7f1d8720cf40 1 -- 172.21.15.32:0/4144819614 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f1d60075a20 msgr2=0x7f1d60077f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:58:16.811 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:16.810+0000 7f1d8720cf40 1 --2- 172.21.15.32:0/4144819614 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f1d60075a20 0x7f1d60077f30 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x7f1d70004160 tx=0x7f1d70017040 comp rx=0 tx=0).stop 2024-08-03T20:58:16.811 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:16.810+0000 7f1d8720cf40 1 -- 172.21.15.32:0/4144819614 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55928225f820 msgr2=0x55928218aa00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:58:16.812 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:16.810+0000 7f1d8720cf40 1 --2- 172.21.15.32:0/4144819614 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55928225f820 0x55928218aa00 secure :-1 s=READY pgs=1735 cs=0 l=1 rev1=1 crypto rx=0x7f1d78006fd0 tx=0x7f1d7800d5f0 comp rx=0 tx=0).stop 2024-08-03T20:58:16.812 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:16.810+0000 7f1d80200640 1 -- 172.21.15.32:0/4144819614 reap_dead start 2024-08-03T20:58:16.812 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:16.810+0000 7f1d8720cf40 1 -- 172.21.15.32:0/4144819614 shutdown_connections 2024-08-03T20:58:16.812 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:16.811+0000 7f1d8720cf40 1 -- 172.21.15.32:0/4144819614 >> 172.21.15.32:0/4144819614 conn(0x5592820adac0 msgr2=0x5592821b7c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:58:16.812 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:16.811+0000 7f1d8720cf40 1 -- 172.21.15.32:0/4144819614 shutdown_connections 2024-08-03T20:58:16.812 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:16.811+0000 7f1d8720cf40 1 -- 172.21.15.32:0/4144819614 wait complete. 2024-08-03T20:58:16.822 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs joinable true 2024-08-03T20:58:17.053 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:17.052+0000 7f81ac800640 1 -- 172.21.15.32:0/3148513603 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f81a409ac30 msgr2=0x7f81a409b040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:58:17.053 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:17.052+0000 7f81ac800640 1 --2- 172.21.15.32:0/3148513603 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f81a409ac30 0x7f81a409b040 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f8198009830 tx=0x7f819802a480 comp rx=0 tx=0).stop 2024-08-03T20:58:17.053 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:17.052+0000 7f81ac800640 1 -- 172.21.15.32:0/3148513603 shutdown_connections 2024-08-03T20:58:17.054 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:17.052+0000 7f81ac800640 1 --2- 172.21.15.32:0/3148513603 >> v2:172.21.15.118:3300/0 conn(0x7f81a40d00f0 0x7f81a4166de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:17.054 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:17.052+0000 7f81ac800640 1 --2- 172.21.15.32:0/3148513603 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f81a409ac30 0x7f81a409b040 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:17.054 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:17.052+0000 7f81ac800640 1 -- 172.21.15.32:0/3148513603 >> 172.21.15.32:0/3148513603 conn(0x7f81a40b3c00 msgr2=0x7f81a40b4010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:58:17.054 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:17.052+0000 7f81ac800640 1 -- 172.21.15.32:0/3148513603 shutdown_connections 2024-08-03T20:58:17.054 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:17.053+0000 7f81ac800640 1 -- 172.21.15.32:0/3148513603 wait complete. 2024-08-03T20:58:17.055 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:17.055+0000 7f81ac800640 1 Processor -- start 2024-08-03T20:58:17.055 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:17.055+0000 7f81ac800640 1 -- start start 2024-08-03T20:58:17.058 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:17.056+0000 7f81ac800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f81a409ac30 0x7f81a41f9760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:58:17.058 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:17.056+0000 7f81ac800640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f81a40cfce0 0x7f81a40c8610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:58:17.058 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:17.056+0000 7f81ac800640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f81a40d00f0 0x7f81a41f4f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:58:17.058 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:17.056+0000 7f81ac800640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f81a40b7ba0 con 0x7f81a409ac30 2024-08-03T20:58:17.058 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:17.056+0000 7f81ac800640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f81a40b7a20 con 0x7f81a40d00f0 2024-08-03T20:58:17.058 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:17.056+0000 7f81ac800640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7f81a40b2ca0 con 0x7f81a40cfce0 2024-08-03T20:58:17.058 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:17.057+0000 7f81a9e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f81a409ac30 0x7f81a41f9760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:58:17.058 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:17.057+0000 7f81a9e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f81a409ac30 0x7f81a41f9760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:35940/0 (socket says 172.21.15.32:35940) 2024-08-03T20:58:17.058 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:17.057+0000 7f81a9e00640 1 -- 172.21.15.32:0/1345706878 learned_addr learned my addr 172.21.15.32:0/1345706878 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:58:17.059 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:17.057+0000 7f81aa800640 1 --2- 172.21.15.32:0/1345706878 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f81a40d00f0 0x7f81a41f4f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:58:17.059 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:17.057+0000 7f81a9400640 1 --2- 172.21.15.32:0/1345706878 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f81a40cfce0 0x7f81a40c8610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:58:17.059 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:17.057+0000 7f81a9e00640 1 -- 172.21.15.32:0/1345706878 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f81a40cfce0 msgr2=0x7f81a40c8610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:58:17.059 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:17.057+0000 7f81a9e00640 1 --2- 172.21.15.32:0/1345706878 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f81a40cfce0 0x7f81a40c8610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:17.059 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:17.057+0000 7f81a9e00640 1 -- 172.21.15.32:0/1345706878 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f81a40d00f0 msgr2=0x7f81a41f4f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:58:17.059 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:17.057+0000 7f81a9e00640 1 --2- 172.21.15.32:0/1345706878 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f81a40d00f0 0x7f81a41f4f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:17.059 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:17.057+0000 7f81a9e00640 1 -- 172.21.15.32:0/1345706878 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f81a41f8ca0 con 0x7f81a409ac30 2024-08-03T20:58:17.060 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:17.058+0000 7f81a9e00640 1 --2- 172.21.15.32:0/1345706878 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f81a409ac30 0x7f81a41f9760 secure :-1 s=READY pgs=1737 cs=0 l=1 rev1=1 crypto rx=0x7f8198000c00 tx=0x7f8198033c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:58:17.060 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:17.058+0000 7f819e000640 1 -- 172.21.15.32:0/1345706878 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8198005c90 con 0x7f81a409ac30 2024-08-03T20:58:17.060 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:17.058+0000 7f819e000640 1 -- 172.21.15.32:0/1345706878 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f8198031e50 con 0x7f81a409ac30 2024-08-03T20:58:17.060 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:17.058+0000 7f819e000640 1 -- 172.21.15.32:0/1345706878 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8198033290 con 0x7f81a409ac30 2024-08-03T20:58:17.060 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:17.058+0000 7f81ac800640 1 -- 172.21.15.32:0/1345706878 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f81a40c5d60 con 0x7f81a409ac30 2024-08-03T20:58:17.060 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:17.058+0000 7f81ac800640 1 -- 172.21.15.32:0/1345706878 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f81a40c64e0 con 0x7f81a409ac30 2024-08-03T20:58:17.060 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:17.059+0000 7f81ac800640 1 -- 172.21.15.32:0/1345706878 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f81a409b040 con 0x7f81a409ac30 2024-08-03T20:58:17.061 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:17.060+0000 7f819e000640 1 -- 172.21.15.32:0/1345706878 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f8198033430 con 0x7f81a409ac30 2024-08-03T20:58:17.061 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:17.061+0000 7f819e000640 1 --2- 172.21.15.32:0/1345706878 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f8188077010 0x7f8188079520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:58:17.061 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:17.061+0000 7f819e000640 1 -- 172.21.15.32:0/1345706878 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(112..112 src has 1..112) ==== 8176+0+0 (secure 0 0 0) 0x7f81980ba500 con 0x7f81a409ac30 2024-08-03T20:58:17.062 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:17.062+0000 7f81a9400640 1 --2- 172.21.15.32:0/1345706878 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f8188077010 0x7f8188079520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:58:17.062 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:17.062+0000 7f81a9400640 1 --2- 172.21.15.32:0/1345706878 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f8188077010 0x7f8188079520 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7f8194007fd0 tx=0x7f8194001390 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:58:17.066 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:17.066+0000 7f819e000640 1 -- 172.21.15.32:0/1345706878 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7f819803f030 con 0x7f81a409ac30 2024-08-03T20:58:17.312 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:17.311+0000 7f81ac800640 1 -- 172.21.15.32:0/1345706878 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "joinable", "val": "true"} v 0) -- 0x7f81a40ce6b0 con 0x7f81a409ac30 2024-08-03T20:58:18.231 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:18.230+0000 7f819e000640 1 -- 172.21.15.32:0/1345706878 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "joinable", "val": "true"}]=0 cephfs marked joinable; MDS may join as newly active. v247) ==== 162+0+0 (secure 0 0 0) 0x7f8198082d70 con 0x7f81a409ac30 2024-08-03T20:58:18.232 INFO:teuthology.orchestra.run.smithi032.stderr:cephfs marked joinable; MDS may join as newly active. 2024-08-03T20:58:18.235 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:18.233+0000 7f81ac800640 1 -- 172.21.15.32:0/1345706878 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f8188077010 msgr2=0x7f8188079520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:58:18.235 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:18.233+0000 7f81ac800640 1 --2- 172.21.15.32:0/1345706878 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f8188077010 0x7f8188079520 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7f8194007fd0 tx=0x7f8194001390 comp rx=0 tx=0).stop 2024-08-03T20:58:18.235 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:18.233+0000 7f81ac800640 1 -- 172.21.15.32:0/1345706878 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f81a409ac30 msgr2=0x7f81a41f9760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:58:18.235 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:18.233+0000 7f81ac800640 1 --2- 172.21.15.32:0/1345706878 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f81a409ac30 0x7f81a41f9760 secure :-1 s=READY pgs=1737 cs=0 l=1 rev1=1 crypto rx=0x7f8198000c00 tx=0x7f8198033c20 comp rx=0 tx=0).stop 2024-08-03T20:58:18.235 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:18.233+0000 7f81ac800640 1 -- 172.21.15.32:0/1345706878 shutdown_connections 2024-08-03T20:58:18.235 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:18.233+0000 7f81ac800640 1 --2- 172.21.15.32:0/1345706878 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f8188077010 0x7f8188079520 unknown :-1 s=CLOSED pgs=707 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:18.235 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:18.233+0000 7f81ac800640 1 --2- 172.21.15.32:0/1345706878 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f81a40d00f0 0x7f81a41f4f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:18.235 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:18.233+0000 7f81ac800640 1 --2- 172.21.15.32:0/1345706878 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f81a40cfce0 0x7f81a40c8610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:18.235 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:18.233+0000 7f81ac800640 1 --2- 172.21.15.32:0/1345706878 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f81a409ac30 0x7f81a41f9760 unknown :-1 s=CLOSED pgs=1737 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:18.235 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:18.233+0000 7f81ac800640 1 -- 172.21.15.32:0/1345706878 >> 172.21.15.32:0/1345706878 conn(0x7f81a40b3c00 msgr2=0x7f81a41e4990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:58:18.235 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:18.233+0000 7f81ac800640 1 -- 172.21.15.32:0/1345706878 shutdown_connections 2024-08-03T20:58:18.235 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:18.233+0000 7f81ac800640 1 -- 172.21.15.32:0/1345706878 wait complete. 2024-08-03T20:58:18.242 INFO:tasks.ceph.mds.a.smithi032.stderr:2024-08-03T20:58:18.241+0000 7efd68000640 -1 log_channel(cluster) log [ERR] : error decoding table object 'mds_snaptable': End of buffer [buffer:2] 2024-08-03T20:58:18.243 INFO:tasks.ceph.mon.a.smithi032.stderr:2024-08-03T20:58:18.242+0000 7f4e6d600640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-08-03T20:58:18.243 INFO:tasks.ceph.mon.a.smithi032.stderr:2024-08-03T20:58:18.242+0000 7f4e6d600640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 mds daemon damaged (MDS_DAMAGE) 2024-08-03T20:58:18.267 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-08-03T20:58:18.500 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:18.498+0000 7facf5000640 1 -- 172.21.15.32:0/3168723407 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7facf00b38b0 msgr2=0x7facf01523f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:58:18.500 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:18.498+0000 7facf5000640 1 --2- 172.21.15.32:0/3168723407 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7facf00b38b0 0x7facf01523f0 secure :-1 s=READY pgs=1739 cs=0 l=1 rev1=1 crypto rx=0x7face0009830 tx=0x7face002a510 comp rx=0 tx=0).stop 2024-08-03T20:58:18.500 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:18.498+0000 7facf5000640 1 -- 172.21.15.32:0/3168723407 shutdown_connections 2024-08-03T20:58:18.500 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:18.498+0000 7facf5000640 1 --2- 172.21.15.32:0/3168723407 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7facf015f610 0x7facf00cdf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:18.500 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:18.498+0000 7facf5000640 1 --2- 172.21.15.32:0/3168723407 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7facf00b38b0 0x7facf01523f0 unknown :-1 s=CLOSED pgs=1739 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:18.501 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:18.498+0000 7facf5000640 1 -- 172.21.15.32:0/3168723407 >> 172.21.15.32:0/3168723407 conn(0x7facf00b1bd0 msgr2=0x7facf00b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:58:18.501 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:18.498+0000 7facf5000640 1 -- 172.21.15.32:0/3168723407 shutdown_connections 2024-08-03T20:58:18.501 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:18.498+0000 7facf5000640 1 -- 172.21.15.32:0/3168723407 wait complete. 2024-08-03T20:58:18.502 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:18.501+0000 7facf5000640 1 Processor -- start 2024-08-03T20:58:18.502 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:18.501+0000 7facf5000640 1 -- start start 2024-08-03T20:58:18.505 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:18.502+0000 7facf5000640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7facf00b38b0 0x7facf014ad90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:58:18.505 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:18.502+0000 7facf5000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7facf00b4ce0 0x7facf01478a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:58:18.505 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:18.502+0000 7facf5000640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7facf0147de0 0x7facf01fc6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:58:18.506 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:18.502+0000 7facf5000640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7facf00cea90 con 0x7facf00b4ce0 2024-08-03T20:58:18.506 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:18.502+0000 7facf5000640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7facf00ce910 con 0x7facf0147de0 2024-08-03T20:58:18.506 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:18.502+0000 7facf5000640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7facf00cec10 con 0x7facf00b38b0 2024-08-03T20:58:18.506 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:18.503+0000 7faced400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7facf00b4ce0 0x7facf01478a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:58:18.506 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:18.503+0000 7faced400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7facf00b4ce0 0x7facf01478a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:35954/0 (socket says 172.21.15.32:35954) 2024-08-03T20:58:18.506 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:18.503+0000 7faced400640 1 -- 172.21.15.32:0/772120173 learned_addr learned my addr 172.21.15.32:0/772120173 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:58:18.506 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:18.503+0000 7facede00640 1 --2- 172.21.15.32:0/772120173 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7facf00b38b0 0x7facf014ad90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:58:18.506 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:18.503+0000 7facee800640 1 --2- 172.21.15.32:0/772120173 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7facf0147de0 0x7facf01fc6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:58:18.506 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:18.503+0000 7faced400640 1 -- 172.21.15.32:0/772120173 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7facf00b38b0 msgr2=0x7facf014ad90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:58:18.506 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:18.503+0000 7faced400640 1 --2- 172.21.15.32:0/772120173 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7facf00b38b0 0x7facf014ad90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:18.506 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:18.503+0000 7faced400640 1 -- 172.21.15.32:0/772120173 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7facf0147de0 msgr2=0x7facf01fc6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:58:18.507 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:18.503+0000 7faced400640 1 --2- 172.21.15.32:0/772120173 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7facf0147de0 0x7facf01fc6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:18.507 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:18.503+0000 7faced400640 1 -- 172.21.15.32:0/772120173 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7facf014b2d0 con 0x7facf00b4ce0 2024-08-03T20:58:18.507 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:18.503+0000 7facede00640 1 --2- 172.21.15.32:0/772120173 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7facf00b38b0 0x7facf014ad90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-03T20:58:18.507 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:18.504+0000 7faced400640 1 --2- 172.21.15.32:0/772120173 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7facf00b4ce0 0x7facf01478a0 secure :-1 s=READY pgs=1740 cs=0 l=1 rev1=1 crypto rx=0x7facdc00c3c0 tx=0x7facdc00c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:58:18.507 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:18.504+0000 7face6000640 1 -- 172.21.15.32:0/772120173 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7facdc017070 con 0x7facf00b4ce0 2024-08-03T20:58:18.507 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:18.504+0000 7face6000640 1 -- 172.21.15.32:0/772120173 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7facdc0035f0 con 0x7facf00b4ce0 2024-08-03T20:58:18.507 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:18.504+0000 7face6000640 1 -- 172.21.15.32:0/772120173 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7facdc003a40 con 0x7facf00b4ce0 2024-08-03T20:58:18.507 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:18.504+0000 7facf5000640 1 -- 172.21.15.32:0/772120173 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7facf014eee0 con 0x7facf00b4ce0 2024-08-03T20:58:18.507 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:18.504+0000 7facf5000640 1 -- 172.21.15.32:0/772120173 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7facf014f4a0 con 0x7facf00b4ce0 2024-08-03T20:58:18.508 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:18.505+0000 7facf5000640 1 -- 172.21.15.32:0/772120173 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7facf00cf130 con 0x7facf00b4ce0 2024-08-03T20:58:18.508 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:18.506+0000 7face6000640 1 -- 172.21.15.32:0/772120173 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7facdc01a070 con 0x7facf00b4ce0 2024-08-03T20:58:18.509 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:18.508+0000 7face6000640 1 --2- 172.21.15.32:0/772120173 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7facd0077010 0x7facd0079520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:58:18.509 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:18.508+0000 7face6000640 1 -- 172.21.15.32:0/772120173 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(113..113 src has 1..113) ==== 8262+0+0 (secure 0 0 0) 0x7facdc098720 con 0x7facf00b4ce0 2024-08-03T20:58:18.510 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:18.508+0000 7facede00640 1 --2- 172.21.15.32:0/772120173 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7facd0077010 0x7facd0079520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:58:18.510 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:18.509+0000 7facede00640 1 --2- 172.21.15.32:0/772120173 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7facd0077010 0x7facd0079520 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7face0034000 tx=0x7face0004730 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:58:18.514 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:18.512+0000 7face6000640 1 -- 172.21.15.32:0/772120173 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7facf014b2d0 con 0x7facf00b4ce0 2024-08-03T20:58:18.759 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:18.757+0000 7facf5000640 1 -- 172.21.15.32:0/772120173 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7facf01522d0 con 0x7facf00b4ce0 2024-08-03T20:58:18.759 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:18.758+0000 7face6000640 1 -- 172.21.15.32:0/772120173 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 249 v249) ==== 95+0+4391 (secure 0 0 0) 0x7facdc060f00 con 0x7facf00b4ce0 2024-08-03T20:58:18.759 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-03T20:58:18.760 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":249,"btime":"2024-08-03T20:58:18:243477+0000","default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6439,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6835/531080160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6834","nonce":531080160},{"type":"v1","addr":"172.21.15.118:6835","nonce":531080160}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":84},{"gid":6454,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6837/2796882319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6836","nonce":2796882319},{"type":"v1","addr":"172.21.15.118:6837","nonce":2796882319}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":86},{"gid":6813,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6834/2686933890","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":2686933890},{"type":"v1","addr":"172.21.15.32:6834","nonce":2686933890}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":85}],"filesystems":[{"mdsmap":{"epoch":249,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-08-03T20:54:26.950518+0000","modified":"2024-08-03T20:58:18.243404+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":113,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":6}]} 2024-08-03T20:58:18.760 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 249 2024-08-03T20:58:18.762 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:18.760+0000 7facf5000640 1 -- 172.21.15.32:0/772120173 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7facd0077010 msgr2=0x7facd0079520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:58:18.762 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:18.760+0000 7facf5000640 1 --2- 172.21.15.32:0/772120173 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7facd0077010 0x7facd0079520 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7face0034000 tx=0x7face0004730 comp rx=0 tx=0).stop 2024-08-03T20:58:18.762 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:18.760+0000 7facf5000640 1 -- 172.21.15.32:0/772120173 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7facf00b4ce0 msgr2=0x7facf01478a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:58:18.762 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:18.760+0000 7facf5000640 1 --2- 172.21.15.32:0/772120173 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7facf00b4ce0 0x7facf01478a0 secure :-1 s=READY pgs=1740 cs=0 l=1 rev1=1 crypto rx=0x7facdc00c3c0 tx=0x7facdc00c880 comp rx=0 tx=0).stop 2024-08-03T20:58:18.762 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:18.760+0000 7facf5000640 1 -- 172.21.15.32:0/772120173 shutdown_connections 2024-08-03T20:58:18.762 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:18.760+0000 7facf5000640 1 --2- 172.21.15.32:0/772120173 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7facd0077010 0x7facd0079520 unknown :-1 s=CLOSED pgs=708 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:18.762 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:18.760+0000 7facf5000640 1 --2- 172.21.15.32:0/772120173 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7facf0147de0 0x7facf01fc6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:18.763 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:18.760+0000 7facf5000640 1 --2- 172.21.15.32:0/772120173 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7facf00b4ce0 0x7facf01478a0 unknown :-1 s=CLOSED pgs=1740 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:18.763 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:18.760+0000 7facf5000640 1 --2- 172.21.15.32:0/772120173 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7facf00b38b0 0x7facf014ad90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:18.763 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:18.760+0000 7facf5000640 1 -- 172.21.15.32:0/772120173 >> 172.21.15.32:0/772120173 conn(0x7facf00b1bd0 msgr2=0x7facf0151240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:58:18.763 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:18.760+0000 7facf5000640 1 -- 172.21.15.32:0/772120173 shutdown_connections 2024-08-03T20:58:18.763 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:18.761+0000 7facf5000640 1 -- 172.21.15.32:0/772120173 wait complete. 2024-08-03T20:58:18.789 DEBUG:tasks.ceph_test_case:wait_until_true: success in 0s and 0 retries 2024-08-03T20:58:18.790 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-08-03T20:58:19.017 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:19.015+0000 7f6e45600640 1 -- 172.21.15.32:0/244122964 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f6e40073e60 msgr2=0x7f6e40074270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:58:19.017 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:19.015+0000 7f6e45600640 1 --2- 172.21.15.32:0/244122964 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f6e40073e60 0x7f6e40074270 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f6e2c005fd0 tx=0x7f6e2c02a540 comp rx=0 tx=0).stop 2024-08-03T20:58:19.018 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:19.015+0000 7f6e45600640 1 -- 172.21.15.32:0/244122964 shutdown_connections 2024-08-03T20:58:19.018 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:19.015+0000 7f6e45600640 1 --2- 172.21.15.32:0/244122964 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f6e40073e60 0x7f6e40074270 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:19.018 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:19.015+0000 7f6e45600640 1 --2- 172.21.15.32:0/244122964 >> v2:172.21.15.118:3300/0 conn(0x7f6e4006e930 0x7f6e40144f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:19.018 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:19.015+0000 7f6e45600640 1 -- 172.21.15.32:0/244122964 >> 172.21.15.32:0/244122964 conn(0x7f6e400b1bd0 msgr2=0x7f6e400b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:58:19.018 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:19.015+0000 7f6e45600640 1 -- 172.21.15.32:0/244122964 shutdown_connections 2024-08-03T20:58:19.018 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:19.015+0000 7f6e45600640 1 -- 172.21.15.32:0/244122964 wait complete. 2024-08-03T20:58:19.019 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:19.017+0000 7f6e45600640 1 Processor -- start 2024-08-03T20:58:19.019 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:19.018+0000 7f6e45600640 1 -- start start 2024-08-03T20:58:19.021 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:19.019+0000 7f6e45600640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f6e4006e930 0x7f6e401444e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:58:19.022 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:19.019+0000 7f6e45600640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f6e40073e60 0x7f6e4006f360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:58:19.022 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:19.019+0000 7f6e45600640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6e400747b0 0x7f6e4006f8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:58:19.022 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:19.019+0000 7f6e45600640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f6e400cf9c0 con 0x7f6e400747b0 2024-08-03T20:58:19.022 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:19.019+0000 7f6e45600640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f6e400cf840 con 0x7f6e4006e930 2024-08-03T20:58:19.022 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:19.019+0000 7f6e45600640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7f6e400ccff0 con 0x7f6e40073e60 2024-08-03T20:58:19.022 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:19.019+0000 7f6e3f400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6e400747b0 0x7f6e4006f8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:58:19.022 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:19.019+0000 7f6e3f400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6e400747b0 0x7f6e4006f8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:35968/0 (socket says 172.21.15.32:35968) 2024-08-03T20:58:19.022 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:19.019+0000 7f6e3f400640 1 -- 172.21.15.32:0/3125263562 learned_addr learned my addr 172.21.15.32:0/3125263562 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:58:19.023 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:19.020+0000 7f6e3e000640 1 --2- 172.21.15.32:0/3125263562 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f6e40073e60 0x7f6e4006f360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:58:19.023 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:19.020+0000 7f6e3ea00640 1 --2- 172.21.15.32:0/3125263562 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f6e4006e930 0x7f6e401444e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:58:19.023 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:19.020+0000 7f6e3f400640 1 -- 172.21.15.32:0/3125263562 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f6e40073e60 msgr2=0x7f6e4006f360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:58:19.023 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:19.020+0000 7f6e3f400640 1 --2- 172.21.15.32:0/3125263562 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f6e40073e60 0x7f6e4006f360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:19.023 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:19.020+0000 7f6e3f400640 1 -- 172.21.15.32:0/3125263562 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f6e4006e930 msgr2=0x7f6e401444e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:58:19.023 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:19.020+0000 7f6e3f400640 1 --2- 172.21.15.32:0/3125263562 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f6e4006e930 0x7f6e401444e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:19.023 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:19.020+0000 7f6e3f400640 1 -- 172.21.15.32:0/3125263562 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6e40141d70 con 0x7f6e400747b0 2024-08-03T20:58:19.023 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:19.020+0000 7f6e3f400640 1 --2- 172.21.15.32:0/3125263562 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6e400747b0 0x7f6e4006f8a0 secure :-1 s=READY pgs=1741 cs=0 l=1 rev1=1 crypto rx=0x7f6e3800c6c0 tx=0x7f6e3800cb80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:58:19.024 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:19.020+0000 7f6e36a00640 1 -- 172.21.15.32:0/3125263562 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e38003b70 con 0x7f6e400747b0 2024-08-03T20:58:19.025 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:19.021+0000 7f6e36a00640 1 -- 172.21.15.32:0/3125263562 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f6e38003d10 con 0x7f6e400747b0 2024-08-03T20:58:19.025 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:19.021+0000 7f6e45600640 1 -- 172.21.15.32:0/3125263562 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6e401f9430 con 0x7f6e400747b0 2024-08-03T20:58:19.025 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:19.021+0000 7f6e36a00640 1 -- 172.21.15.32:0/3125263562 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e38005330 con 0x7f6e400747b0 2024-08-03T20:58:19.025 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:19.021+0000 7f6e45600640 1 -- 172.21.15.32:0/3125263562 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6e401f9710 con 0x7f6e400747b0 2024-08-03T20:58:19.025 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:19.022+0000 7f6e45600640 1 -- 172.21.15.32:0/3125263562 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6e40142250 con 0x7f6e400747b0 2024-08-03T20:58:19.025 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:19.023+0000 7f6e36a00640 1 -- 172.21.15.32:0/3125263562 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f6e380054d0 con 0x7f6e400747b0 2024-08-03T20:58:19.027 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:19.024+0000 7f6e36a00640 1 --2- 172.21.15.32:0/3125263562 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f6e20077010 0x7f6e20079520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:58:19.027 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:19.025+0000 7f6e36a00640 1 -- 172.21.15.32:0/3125263562 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(113..113 src has 1..113) ==== 8262+0+0 (secure 0 0 0) 0x7f6e380990c0 con 0x7f6e400747b0 2024-08-03T20:58:19.027 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:19.025+0000 7f6e3ea00640 1 --2- 172.21.15.32:0/3125263562 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f6e20077010 0x7f6e20079520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:58:19.027 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:19.026+0000 7f6e3ea00640 1 --2- 172.21.15.32:0/3125263562 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f6e20077010 0x7f6e20079520 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x7f6e300095f0 tx=0x7f6e30005f90 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:58:19.030 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:19.029+0000 7f6e36a00640 1 -- 172.21.15.32:0/3125263562 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7f6e40142250 con 0x7f6e400747b0 2024-08-03T20:58:19.280 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:19.279+0000 7f6e45600640 1 -- 172.21.15.32:0/3125263562 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f6e400cd510 con 0x7f6e400747b0 2024-08-03T20:58:19.281 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:19.279+0000 7f6e36a00640 1 -- 172.21.15.32:0/3125263562 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 249 v249) ==== 95+0+4391 (secure 0 0 0) 0x7f6e380611d0 con 0x7f6e400747b0 2024-08-03T20:58:19.281 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-03T20:58:19.281 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":249,"btime":"2024-08-03T20:58:18:243477+0000","default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6439,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6835/531080160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6834","nonce":531080160},{"type":"v1","addr":"172.21.15.118:6835","nonce":531080160}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":84},{"gid":6454,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6837/2796882319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6836","nonce":2796882319},{"type":"v1","addr":"172.21.15.118:6837","nonce":2796882319}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":86},{"gid":6813,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6834/2686933890","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":2686933890},{"type":"v1","addr":"172.21.15.32:6834","nonce":2686933890}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":85}],"filesystems":[{"mdsmap":{"epoch":249,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-08-03T20:54:26.950518+0000","modified":"2024-08-03T20:58:18.243404+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":113,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":6}]} 2024-08-03T20:58:19.281 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 249 2024-08-03T20:58:19.283 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:19.281+0000 7f6e45600640 1 -- 172.21.15.32:0/3125263562 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f6e20077010 msgr2=0x7f6e20079520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:58:19.284 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:19.282+0000 7f6e45600640 1 --2- 172.21.15.32:0/3125263562 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f6e20077010 0x7f6e20079520 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x7f6e300095f0 tx=0x7f6e30005f90 comp rx=0 tx=0).stop 2024-08-03T20:58:19.284 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:19.282+0000 7f6e45600640 1 -- 172.21.15.32:0/3125263562 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6e400747b0 msgr2=0x7f6e4006f8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:58:19.284 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:19.282+0000 7f6e45600640 1 --2- 172.21.15.32:0/3125263562 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6e400747b0 0x7f6e4006f8a0 secure :-1 s=READY pgs=1741 cs=0 l=1 rev1=1 crypto rx=0x7f6e3800c6c0 tx=0x7f6e3800cb80 comp rx=0 tx=0).stop 2024-08-03T20:58:19.284 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:19.282+0000 7f6e45600640 1 -- 172.21.15.32:0/3125263562 shutdown_connections 2024-08-03T20:58:19.284 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:19.282+0000 7f6e45600640 1 --2- 172.21.15.32:0/3125263562 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f6e20077010 0x7f6e20079520 unknown :-1 s=CLOSED pgs=709 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:19.284 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:19.282+0000 7f6e45600640 1 --2- 172.21.15.32:0/3125263562 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6e400747b0 0x7f6e4006f8a0 unknown :-1 s=CLOSED pgs=1741 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:19.284 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:19.282+0000 7f6e45600640 1 --2- 172.21.15.32:0/3125263562 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f6e40073e60 0x7f6e4006f360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:19.284 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:19.282+0000 7f6e45600640 1 --2- 172.21.15.32:0/3125263562 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f6e4006e930 0x7f6e401444e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:19.284 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:19.282+0000 7f6e45600640 1 -- 172.21.15.32:0/3125263562 >> 172.21.15.32:0/3125263562 conn(0x7f6e400b1bd0 msgr2=0x7f6e40140c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:58:19.284 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:19.282+0000 7f6e45600640 1 -- 172.21.15.32:0/3125263562 shutdown_connections 2024-08-03T20:58:19.284 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:19.282+0000 7f6e45600640 1 -- 172.21.15.32:0/3125263562 wait complete. 2024-08-03T20:58:19.308 DEBUG:tasks.ceph_test_case:wait_until_true: success in 0s and 0 retries 2024-08-03T20:58:19.308 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-08-03T20:58:19.538 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:19.536+0000 7f2880e00640 1 -- 172.21.15.32:0/1201099717 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f287c06e930 msgr2=0x7f287c144f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:58:19.538 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:19.536+0000 7f2880e00640 1 --2- 172.21.15.32:0/1201099717 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f287c06e930 0x7f287c144f90 secure :-1 s=READY pgs=1742 cs=0 l=1 rev1=1 crypto rx=0x7f287c0b2f80 tx=0x7f286c02a560 comp rx=0 tx=0).stop 2024-08-03T20:58:19.539 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:19.536+0000 7f2880e00640 1 -- 172.21.15.32:0/1201099717 shutdown_connections 2024-08-03T20:58:19.539 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:19.536+0000 7f2880e00640 1 --2- 172.21.15.32:0/1201099717 >> v2:172.21.15.118:3300/0 conn(0x7f287c0747b0 0x7f287c0cc8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:19.539 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:19.536+0000 7f2880e00640 1 --2- 172.21.15.32:0/1201099717 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f287c073e60 0x7f287c074270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:19.539 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:19.536+0000 7f2880e00640 1 --2- 172.21.15.32:0/1201099717 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f287c06e930 0x7f287c144f90 unknown :-1 s=CLOSED pgs=1742 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:19.539 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:19.536+0000 7f2880e00640 1 -- 172.21.15.32:0/1201099717 >> 172.21.15.32:0/1201099717 conn(0x7f287c0b1bd0 msgr2=0x7f287c0b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:58:19.539 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:19.536+0000 7f2880e00640 1 -- 172.21.15.32:0/1201099717 shutdown_connections 2024-08-03T20:58:19.539 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:19.536+0000 7f2880e00640 1 -- 172.21.15.32:0/1201099717 wait complete. 2024-08-03T20:58:19.540 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:19.538+0000 7f2880e00640 1 Processor -- start 2024-08-03T20:58:19.540 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:19.539+0000 7f2880e00640 1 -- start start 2024-08-03T20:58:19.542 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:19.540+0000 7f2880e00640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f287c06e930 0x7f287c13c780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:58:19.542 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:19.540+0000 7f2880e00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f287c073e60 0x7f287c0b3000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:58:19.542 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:19.540+0000 7f2880e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f287c0747b0 0x7f287c0b3540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:58:19.543 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:19.540+0000 7f2880e00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f287c0cf9c0 con 0x7f287c0747b0 2024-08-03T20:58:19.543 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:19.540+0000 7f2880e00640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f287c0cf840 con 0x7f287c073e60 2024-08-03T20:58:19.543 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:19.540+0000 7f2880e00640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7f287c0cce20 con 0x7f287c06e930 2024-08-03T20:58:19.543 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:19.540+0000 7f287a800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f287c0747b0 0x7f287c0b3540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:58:19.543 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:19.540+0000 7f287a800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f287c0747b0 0x7f287c0b3540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:35984/0 (socket says 172.21.15.32:35984) 2024-08-03T20:58:19.543 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:19.540+0000 7f287a800640 1 -- 172.21.15.32:0/3760558988 learned_addr learned my addr 172.21.15.32:0/3760558988 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:58:19.543 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:19.540+0000 7f2879e00640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f287c06e930 0x7f287c13c780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:58:19.543 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:19.541+0000 7f2871400640 1 --2- 172.21.15.32:0/3760558988 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f287c073e60 0x7f287c0b3000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:58:19.543 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:19.541+0000 7f287a800640 1 -- 172.21.15.32:0/3760558988 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f287c06e930 msgr2=0x7f287c13c780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:58:19.543 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:19.541+0000 7f287a800640 1 --2- 172.21.15.32:0/3760558988 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f287c06e930 0x7f287c13c780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:19.543 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:19.541+0000 7f287a800640 1 -- 172.21.15.32:0/3760558988 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f287c073e60 msgr2=0x7f287c0b3000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:58:19.543 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:19.541+0000 7f287a800640 1 --2- 172.21.15.32:0/3760558988 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f287c073e60 0x7f287c0b3000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:19.543 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:19.541+0000 7f287a800640 1 -- 172.21.15.32:0/3760558988 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f287c0ac0f0 con 0x7f287c0747b0 2024-08-03T20:58:19.544 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:19.541+0000 7f287a800640 1 --2- 172.21.15.32:0/3760558988 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f287c0747b0 0x7f287c0b3540 secure :-1 s=READY pgs=1743 cs=0 l=1 rev1=1 crypto rx=0x7f287400e9c0 tx=0x7f287400ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:58:19.544 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:19.541+0000 7f2872a00640 1 -- 172.21.15.32:0/3760558988 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2874004d20 con 0x7f287c0747b0 2024-08-03T20:58:19.544 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:19.541+0000 7f2872a00640 1 -- 172.21.15.32:0/3760558988 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f287400cac0 con 0x7f287c0747b0 2024-08-03T20:58:19.544 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:19.542+0000 7f2872a00640 1 -- 172.21.15.32:0/3760558988 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2874005020 con 0x7f287c0747b0 2024-08-03T20:58:19.545 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:19.542+0000 7f2880e00640 1 -- 172.21.15.32:0/3760558988 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f287c0ac390 con 0x7f287c0747b0 2024-08-03T20:58:19.545 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:19.542+0000 7f2880e00640 1 -- 172.21.15.32:0/3760558988 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f287c0af470 con 0x7f287c0747b0 2024-08-03T20:58:19.545 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:19.543+0000 7f2880e00640 1 -- 172.21.15.32:0/3760558988 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f287c074270 con 0x7f287c0747b0 2024-08-03T20:58:19.545 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:19.544+0000 7f2872a00640 1 -- 172.21.15.32:0/3760558988 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f28740051c0 con 0x7f287c0747b0 2024-08-03T20:58:19.546 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:19.545+0000 7f2872a00640 1 --2- 172.21.15.32:0/3760558988 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f2858077010 0x7f2858079520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:58:19.546 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:19.545+0000 7f2872a00640 1 -- 172.21.15.32:0/3760558988 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(113..113 src has 1..113) ==== 8262+0+0 (secure 0 0 0) 0x7f2874099960 con 0x7f287c0747b0 2024-08-03T20:58:19.547 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:19.546+0000 7f2879e00640 1 --2- 172.21.15.32:0/3760558988 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f2858077010 0x7f2858079520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:58:19.548 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:19.547+0000 7f2879e00640 1 --2- 172.21.15.32:0/3760558988 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f2858077010 0x7f2858079520 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7f286c033000 tx=0x7f286c004730 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:58:19.551 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:19.550+0000 7f2872a00640 1 -- 172.21.15.32:0/3760558988 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7f2874016030 con 0x7f287c0747b0 2024-08-03T20:58:19.793 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:19.792+0000 7f2880e00640 1 -- 172.21.15.32:0/3760558988 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f287c0acb40 con 0x7f287c0747b0 2024-08-03T20:58:19.794 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:19.793+0000 7f2872a00640 1 -- 172.21.15.32:0/3760558988 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 249 v249) ==== 95+0+4391 (secure 0 0 0) 0x7f287406a410 con 0x7f287c0747b0 2024-08-03T20:58:19.794 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-03T20:58:19.794 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":249,"btime":"2024-08-03T20:58:18:243477+0000","default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6439,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6835/531080160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6834","nonce":531080160},{"type":"v1","addr":"172.21.15.118:6835","nonce":531080160}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":84},{"gid":6454,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6837/2796882319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6836","nonce":2796882319},{"type":"v1","addr":"172.21.15.118:6837","nonce":2796882319}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":86},{"gid":6813,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6834/2686933890","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":2686933890},{"type":"v1","addr":"172.21.15.32:6834","nonce":2686933890}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":85}],"filesystems":[{"mdsmap":{"epoch":249,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-08-03T20:54:26.950518+0000","modified":"2024-08-03T20:58:18.243404+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":113,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":6}]} 2024-08-03T20:58:19.795 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 249 2024-08-03T20:58:19.797 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:19.795+0000 7f2880e00640 1 -- 172.21.15.32:0/3760558988 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f2858077010 msgr2=0x7f2858079520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:58:19.797 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:19.795+0000 7f2880e00640 1 --2- 172.21.15.32:0/3760558988 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f2858077010 0x7f2858079520 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7f286c033000 tx=0x7f286c004730 comp rx=0 tx=0).stop 2024-08-03T20:58:19.797 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:19.795+0000 7f2880e00640 1 -- 172.21.15.32:0/3760558988 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f287c0747b0 msgr2=0x7f287c0b3540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:58:19.797 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:19.795+0000 7f2880e00640 1 --2- 172.21.15.32:0/3760558988 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f287c0747b0 0x7f287c0b3540 secure :-1 s=READY pgs=1743 cs=0 l=1 rev1=1 crypto rx=0x7f287400e9c0 tx=0x7f287400ee80 comp rx=0 tx=0).stop 2024-08-03T20:58:19.797 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:19.795+0000 7f2880e00640 1 -- 172.21.15.32:0/3760558988 shutdown_connections 2024-08-03T20:58:19.797 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:19.795+0000 7f2880e00640 1 --2- 172.21.15.32:0/3760558988 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f2858077010 0x7f2858079520 unknown :-1 s=CLOSED pgs=710 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:19.797 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:19.795+0000 7f2880e00640 1 --2- 172.21.15.32:0/3760558988 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f287c073e60 0x7f287c0b3000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:19.797 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:19.795+0000 7f2880e00640 1 --2- 172.21.15.32:0/3760558988 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f287c0747b0 0x7f287c0b3540 unknown :-1 s=CLOSED pgs=1743 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:19.797 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:19.795+0000 7f2880e00640 1 --2- 172.21.15.32:0/3760558988 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f287c06e930 0x7f287c13c780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:19.797 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:19.795+0000 7f2880e00640 1 -- 172.21.15.32:0/3760558988 >> 172.21.15.32:0/3760558988 conn(0x7f287c0b1bd0 msgr2=0x7f287c143e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:58:19.797 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:19.795+0000 7f2880e00640 1 -- 172.21.15.32:0/3760558988 shutdown_connections 2024-08-03T20:58:19.797 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:19.795+0000 7f2880e00640 1 -- 172.21.15.32:0/3760558988 wait complete. 2024-08-03T20:58:19.823 INFO:tasks.cephfs.test_damage:Result: Mutation 'Truncate mds_snaptable' led to DAMAGED state 2024-08-03T20:58:19.823 INFO:tasks.cephfs.test_damage:Applying mutation 'Truncate mds0_inotable' 2024-08-03T20:58:19.823 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-08-03T20:58:19.823 DEBUG:teuthology.orchestra.run.smithi032:> dd if=/proc/self/mounts of=/dev/stdout 2024-08-03T20:58:19.858 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053453,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6499208k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 none /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249604k,nr_inodes=812401,mode=700,uid=1000,gid=1316 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6499208k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-08-03T20:58:19.859 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi032.front.sepia.ceph.com 2024-08-03T20:58:19.859 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-08-03T20:58:19.859 DEBUG:teuthology.orchestra.run.smithi032:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-08-03T20:58:19.933 INFO:teuthology.orchestra.run.smithi032.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.0': No such file or directory 2024-08-03T20:58:19.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-03T20:58:19.935 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-08-03T20:58:20.206 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:20.204+0000 7f5eda000640 1 -- 172.21.15.32:0/547695803 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5eb8003a00 msgr2=0x7f5eb8003e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:58:20.206 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:20.204+0000 7f5eda000640 1 --2- 172.21.15.32:0/547695803 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5eb8003a00 0x7f5eb8003e70 secure :-1 s=READY pgs=1745 cs=0 l=1 rev1=1 crypto rx=0x7f5ec4009b50 tx=0x7f5eb8005830 comp rx=0 tx=0).stop 2024-08-03T20:58:20.207 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:20.204+0000 7f5ed3e00640 1 -- 172.21.15.32:0/547695803 reap_dead start 2024-08-03T20:58:20.207 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:20.204+0000 7f5eda000640 1 -- 172.21.15.32:0/547695803 shutdown_connections 2024-08-03T20:58:20.207 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:20.205+0000 7f5eda000640 1 -- 172.21.15.32:0/547695803 >> 172.21.15.32:0/547695803 conn(0x7f5ed40b1bd0 msgr2=0x7f5ed40b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:58:20.207 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:20.205+0000 7f5eda000640 1 -- 172.21.15.32:0/547695803 shutdown_connections 2024-08-03T20:58:20.207 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:20.205+0000 7f5eda000640 1 -- 172.21.15.32:0/547695803 wait complete. 2024-08-03T20:58:20.208 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:20.207+0000 7f5eda000640 1 Processor -- start 2024-08-03T20:58:20.209 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:20.207+0000 7f5eda000640 1 -- start start 2024-08-03T20:58:20.211 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:20.208+0000 7f5eda000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5eb8003a00 0x7f5ed40ae5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:58:20.211 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:20.208+0000 7f5eda000640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f5ed4141ed0 0x7f5ed4154b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:58:20.211 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:20.208+0000 7f5eda000640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f5ed40b2fd0 0x7f5ed40b33e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:58:20.212 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:20.209+0000 7f5eda000640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f5ed40cf9c0 con 0x7f5eb8003a00 2024-08-03T20:58:20.212 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:20.209+0000 7f5eda000640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f5ed40cf840 con 0x7f5ed4141ed0 2024-08-03T20:58:20.212 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:20.209+0000 7f5eda000640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7f5ed4074270 con 0x7f5ed40b2fd0 2024-08-03T20:58:20.212 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:20.209+0000 7f5ed3400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5eb8003a00 0x7f5ed40ae5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:58:20.212 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:20.209+0000 7f5ed3400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5eb8003a00 0x7f5ed40ae5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:42524/0 (socket says 172.21.15.32:42524) 2024-08-03T20:58:20.212 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:20.209+0000 7f5ed3400640 1 -- 172.21.15.32:0/2950175296 learned_addr learned my addr 172.21.15.32:0/2950175296 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:58:20.212 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:20.209+0000 7f5ed3e00640 1 --2- 172.21.15.32:0/2950175296 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f5ed40b2fd0 0x7f5ed40b33e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:58:20.212 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:20.209+0000 7f5ecbe00640 1 --2- 172.21.15.32:0/2950175296 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f5ed4141ed0 0x7f5ed4154b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:58:20.212 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:20.209+0000 7f5ed3400640 1 -- 172.21.15.32:0/2950175296 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f5ed40b2fd0 msgr2=0x7f5ed40b33e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:58:20.212 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:20.209+0000 7f5ed3400640 1 --2- 172.21.15.32:0/2950175296 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f5ed40b2fd0 0x7f5ed40b33e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:20.212 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:20.209+0000 7f5ed3400640 1 -- 172.21.15.32:0/2950175296 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f5ed4141ed0 msgr2=0x7f5ed4154b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:58:20.212 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:20.209+0000 7f5ed3400640 1 --2- 172.21.15.32:0/2950175296 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f5ed4141ed0 0x7f5ed4154b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:20.213 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:20.209+0000 7f5ed3400640 1 -- 172.21.15.32:0/2950175296 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5ed41f9690 con 0x7f5eb8003a00 2024-08-03T20:58:20.213 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:20.209+0000 7f5ecbe00640 1 --2- 172.21.15.32:0/2950175296 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f5ed4141ed0 0x7f5ed4154b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-03T20:58:20.213 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:20.209+0000 7f5ed3e00640 1 --2- 172.21.15.32:0/2950175296 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f5ed40b2fd0 0x7f5ed40b33e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-03T20:58:20.213 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:20.210+0000 7f5ed3400640 1 --2- 172.21.15.32:0/2950175296 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5eb8003a00 0x7f5ed40ae5f0 secure :-1 s=READY pgs=1746 cs=0 l=1 rev1=1 crypto rx=0x7f5ec4036040 tx=0x7f5ec4033670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:58:20.213 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:20.210+0000 7f5ecb400640 1 -- 172.21.15.32:0/2950175296 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ec4038040 con 0x7f5eb8003a00 2024-08-03T20:58:20.213 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:20.210+0000 7f5ecb400640 1 -- 172.21.15.32:0/2950175296 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f5ec4004110 con 0x7f5eb8003a00 2024-08-03T20:58:20.213 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:20.210+0000 7f5eda000640 1 -- 172.21.15.32:0/2950175296 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5ed40adb10 con 0x7f5eb8003a00 2024-08-03T20:58:20.213 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:20.210+0000 7f5ecb400640 1 -- 172.21.15.32:0/2950175296 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ec40337e0 con 0x7f5eb8003a00 2024-08-03T20:58:20.213 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:20.210+0000 7f5eda000640 1 -- 172.21.15.32:0/2950175296 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5ed40b3990 con 0x7f5eb8003a00 2024-08-03T20:58:20.214 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:20.211+0000 7f5eda000640 1 -- 172.21.15.32:0/2950175296 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5ed4144f90 con 0x7f5eb8003a00 2024-08-03T20:58:20.214 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:20.212+0000 7f5ecb400640 1 -- 172.21.15.32:0/2950175296 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f5ec4045020 con 0x7f5eb8003a00 2024-08-03T20:58:20.215 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:20.214+0000 7f5ecb400640 1 --2- 172.21.15.32:0/2950175296 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f5eb0077010 0x7f5eb0079520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:58:20.215 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:20.214+0000 7f5ecb400640 1 -- 172.21.15.32:0/2950175296 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(113..113 src has 1..113) ==== 8262+0+0 (secure 0 0 0) 0x7f5ec40b9e60 con 0x7f5eb8003a00 2024-08-03T20:58:20.218 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:20.214+0000 7f5ecbe00640 1 --2- 172.21.15.32:0/2950175296 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f5eb0077010 0x7f5eb0079520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:58:20.218 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:20.215+0000 7f5ecbe00640 1 --2- 172.21.15.32:0/2950175296 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f5eb0077010 0x7f5eb0079520 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 crypto rx=0x7f5ed40cc0a0 tx=0x7f5ebc005f90 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:58:20.220 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:20.219+0000 7f5ecb400640 1 -- 172.21.15.32:0/2950175296 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7f5ec40826d0 con 0x7f5eb8003a00 2024-08-03T20:58:20.467 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:20.466+0000 7f5eda000640 1 -- 172.21.15.32:0/2950175296 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7f5ed40b3bc0 con 0x7f5eb8003a00 2024-08-03T20:58:21.255 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:21.254+0000 7f5ecb400640 1 -- 172.21.15.32:0/2950175296 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v250) ==== 161+0+0 (secure 0 0 0) 0x7f5ec4081fd0 con 0x7f5eb8003a00 2024-08-03T20:58:21.255 INFO:teuthology.orchestra.run.smithi032.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-08-03T20:58:21.259 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:21.257+0000 7f5eda000640 1 -- 172.21.15.32:0/2950175296 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f5eb0077010 msgr2=0x7f5eb0079520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:58:21.259 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:21.257+0000 7f5eda000640 1 --2- 172.21.15.32:0/2950175296 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f5eb0077010 0x7f5eb0079520 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 crypto rx=0x7f5ed40cc0a0 tx=0x7f5ebc005f90 comp rx=0 tx=0).stop 2024-08-03T20:58:21.259 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:21.257+0000 7f5eda000640 1 -- 172.21.15.32:0/2950175296 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5eb8003a00 msgr2=0x7f5ed40ae5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:58:21.259 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:21.257+0000 7f5eda000640 1 --2- 172.21.15.32:0/2950175296 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5eb8003a00 0x7f5ed40ae5f0 secure :-1 s=READY pgs=1746 cs=0 l=1 rev1=1 crypto rx=0x7f5ec4036040 tx=0x7f5ec4033670 comp rx=0 tx=0).stop 2024-08-03T20:58:21.259 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:21.257+0000 7f5eda000640 1 -- 172.21.15.32:0/2950175296 shutdown_connections 2024-08-03T20:58:21.259 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:21.257+0000 7f5eda000640 1 --2- 172.21.15.32:0/2950175296 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f5eb0077010 0x7f5eb0079520 unknown :-1 s=CLOSED pgs=711 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:21.259 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:21.257+0000 7f5eda000640 1 --2- 172.21.15.32:0/2950175296 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f5ed4141ed0 0x7f5ed4154b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:21.259 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:21.257+0000 7f5eda000640 1 --2- 172.21.15.32:0/2950175296 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f5ed40b2fd0 0x7f5ed40b33e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:21.259 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:21.257+0000 7f5eda000640 1 --2- 172.21.15.32:0/2950175296 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5eb8003a00 0x7f5ed40ae5f0 unknown :-1 s=CLOSED pgs=1746 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:21.259 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:21.257+0000 7f5eda000640 1 -- 172.21.15.32:0/2950175296 >> 172.21.15.32:0/2950175296 conn(0x7f5ed40b1bd0 msgr2=0x7f5ed4140c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:58:21.259 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:21.257+0000 7f5eda000640 1 -- 172.21.15.32:0/2950175296 shutdown_connections 2024-08-03T20:58:21.260 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:21.257+0000 7f5eda000640 1 -- 172.21.15.32:0/2950175296 wait complete. 2024-08-03T20:58:21.285 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mds repaired 0 2024-08-03T20:58:21.516 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:21.515+0000 7fa0c4000640 1 -- 172.21.15.32:0/4149683970 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa0a4003c60 msgr2=0x7fa0a40040b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:58:21.517 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:21.515+0000 7fa0c4000640 1 --2- 172.21.15.32:0/4149683970 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa0a4003c60 0x7fa0a40040b0 secure :-1 s=READY pgs=1748 cs=0 l=1 rev1=1 crypto rx=0x7fa0b0001eb0 tx=0x7fa0a4005830 comp rx=0 tx=0).stop 2024-08-03T20:58:21.517 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:21.515+0000 7fa0c2000640 1 -- 172.21.15.32:0/4149683970 reap_dead start 2024-08-03T20:58:21.517 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:21.515+0000 7fa0c4000640 1 -- 172.21.15.32:0/4149683970 shutdown_connections 2024-08-03T20:58:21.517 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:21.515+0000 7fa0c4000640 1 -- 172.21.15.32:0/4149683970 >> 172.21.15.32:0/4149683970 conn(0x7fa0bc0cd1e0 msgr2=0x7fa0bc0cd5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:58:21.517 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:21.515+0000 7fa0c4000640 1 -- 172.21.15.32:0/4149683970 shutdown_connections 2024-08-03T20:58:21.517 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:21.515+0000 7fa0c4000640 1 -- 172.21.15.32:0/4149683970 wait complete. 2024-08-03T20:58:21.518 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:21.517+0000 7fa0c4000640 1 Processor -- start 2024-08-03T20:58:21.519 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:21.517+0000 7fa0c4000640 1 -- start start 2024-08-03T20:58:21.520 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:21.519+0000 7fa0c4000640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa0a4003c60 0x7fa0bc1f88c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:58:21.520 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:21.519+0000 7fa0c4000640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fa0bc1f5900 0x7fa0bc1559d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:58:21.520 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:21.519+0000 7fa0c4000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa0bc155f10 0x7fa0bc0b0800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:58:21.520 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:21.519+0000 7fa0c4000640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fa0bc15d6b0 con 0x7fa0bc155f10 2024-08-03T20:58:21.521 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:21.519+0000 7fa0c4000640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7fa0bc06df90 con 0x7fa0a4003c60 2024-08-03T20:58:21.521 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:21.519+0000 7fa0c4000640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7fa0bc15d830 con 0x7fa0bc1f5900 2024-08-03T20:58:21.521 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:21.519+0000 7fa0c2000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa0bc155f10 0x7fa0bc0b0800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:58:21.521 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:21.519+0000 7fa0c2000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa0bc155f10 0x7fa0bc0b0800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:42544/0 (socket says 172.21.15.32:42544) 2024-08-03T20:58:21.521 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:21.519+0000 7fa0c2000640 1 -- 172.21.15.32:0/3919876570 learned_addr learned my addr 172.21.15.32:0/3919876570 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:58:21.522 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:21.519+0000 7fa0c0c00640 1 --2- 172.21.15.32:0/3919876570 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fa0bc1f5900 0x7fa0bc1559d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:58:21.522 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:21.519+0000 7fa0c1600640 1 --2- 172.21.15.32:0/3919876570 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa0a4003c60 0x7fa0bc1f88c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:58:21.522 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:21.519+0000 7fa0c2000640 1 -- 172.21.15.32:0/3919876570 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fa0bc1f5900 msgr2=0x7fa0bc1559d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:58:21.522 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:21.519+0000 7fa0c2000640 1 --2- 172.21.15.32:0/3919876570 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fa0bc1f5900 0x7fa0bc1559d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:21.523 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:21.519+0000 7fa0c2000640 1 -- 172.21.15.32:0/3919876570 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa0a4003c60 msgr2=0x7fa0bc1f88c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:58:21.523 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:21.519+0000 7fa0c2000640 1 --2- 172.21.15.32:0/3919876570 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa0a4003c60 0x7fa0bc1f88c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:21.523 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:21.519+0000 7fa0c2000640 1 -- 172.21.15.32:0/3919876570 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa0bc156650 con 0x7fa0bc155f10 2024-08-03T20:58:21.523 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:21.520+0000 7fa0c2000640 1 --2- 172.21.15.32:0/3919876570 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa0bc155f10 0x7fa0bc0b0800 secure :-1 s=READY pgs=1749 cs=0 l=1 rev1=1 crypto rx=0x7fa0bc074500 tx=0x7fa0b4001b80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:58:21.523 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:21.520+0000 7fa0b9600640 1 -- 172.21.15.32:0/3919876570 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0b4010890 con 0x7fa0bc155f10 2024-08-03T20:58:21.523 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:21.520+0000 7fa0b9600640 1 -- 172.21.15.32:0/3919876570 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fa0b4017070 con 0x7fa0bc155f10 2024-08-03T20:58:21.523 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:21.520+0000 7fa0b9600640 1 -- 172.21.15.32:0/3919876570 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0b4010de0 con 0x7fa0bc155f10 2024-08-03T20:58:21.523 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:21.520+0000 7fa0c4000640 1 -- 172.21.15.32:0/3919876570 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa0bc156930 con 0x7fa0bc155f10 2024-08-03T20:58:21.523 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:21.520+0000 7fa0c4000640 1 -- 172.21.15.32:0/3919876570 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa0bc15dcb0 con 0x7fa0bc155f10 2024-08-03T20:58:21.523 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:21.522+0000 7fa0c4000640 1 -- 172.21.15.32:0/3919876570 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa0bc0cbad0 con 0x7fa0bc155f10 2024-08-03T20:58:21.524 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:21.523+0000 7fa0b9600640 1 -- 172.21.15.32:0/3919876570 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7fa0b401c460 con 0x7fa0bc155f10 2024-08-03T20:58:21.526 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:21.524+0000 7fa0b9600640 1 --2- 172.21.15.32:0/3919876570 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fa09c077010 0x7fa09c079520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:58:21.526 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:21.524+0000 7fa0b9600640 1 -- 172.21.15.32:0/3919876570 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(113..113 src has 1..113) ==== 8262+0+0 (secure 0 0 0) 0x7fa0b4099c50 con 0x7fa0bc155f10 2024-08-03T20:58:21.526 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:21.525+0000 7fa0c1600640 1 --2- 172.21.15.32:0/3919876570 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fa09c077010 0x7fa09c079520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:58:21.527 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:21.525+0000 7fa0c1600640 1 --2- 172.21.15.32:0/3919876570 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fa09c077010 0x7fa09c079520 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 crypto rx=0x7fa0b0033000 tx=0x7fa0b002d000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:58:21.530 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:21.529+0000 7fa0b9600640 1 -- 172.21.15.32:0/3919876570 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7fa0bc0cbad0 con 0x7fa0bc155f10 2024-08-03T20:58:21.811 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:21.810+0000 7fa0c4000640 1 -- 172.21.15.32:0/3919876570 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "mds repaired", "role": "0"} v 0) -- 0x7fa0bc1563e0 con 0x7fa0bc155f10 2024-08-03T20:58:22.257 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:22.256+0000 7fa0b9600640 1 -- 172.21.15.32:0/3919876570 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "mds repaired", "role": "0"}]=0 repaired: restoring rank 6:0 v251) ==== 101+0+0 (secure 0 0 0) 0x7fa0b409f020 con 0x7fa0bc155f10 2024-08-03T20:58:22.258 INFO:teuthology.orchestra.run.smithi032.stderr:repaired: restoring rank 6:0 2024-08-03T20:58:22.260 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:22.258+0000 7fa0c4000640 1 -- 172.21.15.32:0/3919876570 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fa09c077010 msgr2=0x7fa09c079520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:58:22.260 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:22.258+0000 7fa0c4000640 1 --2- 172.21.15.32:0/3919876570 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fa09c077010 0x7fa09c079520 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 crypto rx=0x7fa0b0033000 tx=0x7fa0b002d000 comp rx=0 tx=0).stop 2024-08-03T20:58:22.260 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:22.258+0000 7fa0c4000640 1 -- 172.21.15.32:0/3919876570 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa0bc155f10 msgr2=0x7fa0bc0b0800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:58:22.261 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:22.258+0000 7fa0c4000640 1 --2- 172.21.15.32:0/3919876570 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa0bc155f10 0x7fa0bc0b0800 secure :-1 s=READY pgs=1749 cs=0 l=1 rev1=1 crypto rx=0x7fa0bc074500 tx=0x7fa0b4001b80 comp rx=0 tx=0).stop 2024-08-03T20:58:22.261 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:22.259+0000 7fa0c4000640 1 -- 172.21.15.32:0/3919876570 shutdown_connections 2024-08-03T20:58:22.261 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:22.259+0000 7fa0c4000640 1 --2- 172.21.15.32:0/3919876570 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fa09c077010 0x7fa09c079520 unknown :-1 s=CLOSED pgs=712 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:22.261 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:22.259+0000 7fa0c4000640 1 --2- 172.21.15.32:0/3919876570 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa0bc155f10 0x7fa0bc0b0800 unknown :-1 s=CLOSED pgs=1749 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:22.261 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:22.259+0000 7fa0c4000640 1 --2- 172.21.15.32:0/3919876570 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fa0bc1f5900 0x7fa0bc1559d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:22.261 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:22.259+0000 7fa0c4000640 1 --2- 172.21.15.32:0/3919876570 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa0a4003c60 0x7fa0bc1f88c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:22.261 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:22.259+0000 7fa0c4000640 1 -- 172.21.15.32:0/3919876570 >> 172.21.15.32:0/3919876570 conn(0x7fa0bc0cd1e0 msgr2=0x7fa0bc0cf630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:58:22.261 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:22.259+0000 7fa0c4000640 1 -- 172.21.15.32:0/3919876570 shutdown_connections 2024-08-03T20:58:22.261 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:22.259+0000 7fa0c4000640 1 -- 172.21.15.32:0/3919876570 wait complete. 2024-08-03T20:58:22.286 DEBUG:teuthology.orchestra.run.smithi032:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata import - 2024-08-03T20:58:22.431 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:22.429+0000 7f71b1af4f40 1 Processor -- start 2024-08-03T20:58:22.431 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:22.429+0000 7f71b1af4f40 1 -- start start 2024-08-03T20:58:22.433 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:22.431+0000 7f71b1af4f40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f71900030f0 0x55b859a89720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:58:22.433 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:22.431+0000 7f71b1af4f40 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b859a89030 0x55b859b11c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:58:22.434 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:22.431+0000 7f71b1af4f40 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55b859b12180 0x55b859b12590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:58:22.434 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:22.431+0000 7f71b1af4f40 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55b8599ecbf0 con 0x55b859a89030 2024-08-03T20:58:22.434 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:22.431+0000 7f71b1af4f40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x55b8599f96b0 con 0x7f71900030f0 2024-08-03T20:58:22.434 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:22.431+0000 7f71b1af4f40 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x55b85998ee30 con 0x55b859b12180 2024-08-03T20:58:22.434 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:22.431+0000 7f71aa200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b859a89030 0x55b859b11c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:58:22.434 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:22.431+0000 7f71aa200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b859a89030 0x55b859b11c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:42574/0 (socket says 172.21.15.32:42574) 2024-08-03T20:58:22.434 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:22.431+0000 7f71aa200640 1 -- 172.21.15.32:0/575557709 learned_addr learned my addr 172.21.15.32:0/575557709 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:58:22.434 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:22.431+0000 7f71ab600640 1 --2- 172.21.15.32:0/575557709 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55b859b12180 0x55b859b12590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:58:22.434 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:22.431+0000 7f71aac00640 1 --2- 172.21.15.32:0/575557709 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f71900030f0 0x55b859a89720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:58:22.434 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:22.431+0000 7f71aa200640 1 -- 172.21.15.32:0/575557709 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55b859b12180 msgr2=0x55b859b12590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:58:22.434 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:22.431+0000 7f71aa200640 1 --2- 172.21.15.32:0/575557709 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55b859b12180 0x55b859b12590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:22.434 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:22.431+0000 7f71aa200640 1 -- 172.21.15.32:0/575557709 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f71900030f0 msgr2=0x55b859a89720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:58:22.435 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:22.431+0000 7f71aa200640 1 --2- 172.21.15.32:0/575557709 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f71900030f0 0x55b859a89720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:22.435 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:22.431+0000 7f71aa200640 1 -- 172.21.15.32:0/575557709 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b859b14a30 con 0x55b859a89030 2024-08-03T20:58:22.435 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:22.432+0000 7f71aa200640 1 --2- 172.21.15.32:0/575557709 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b859a89030 0x55b859b11c40 secure :-1 s=READY pgs=1752 cs=0 l=1 rev1=1 crypto rx=0x7f71980467a0 tx=0x7f71980019b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7f395c4620ce8aa0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:58:22.435 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:22.432+0000 7f71a8e00640 1 -- 172.21.15.32:0/575557709 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7198001570 con 0x55b859a89030 2024-08-03T20:58:22.435 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:22.432+0000 7f71a8e00640 1 -- 172.21.15.32:0/575557709 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f7198001730 con 0x55b859a89030 2024-08-03T20:58:22.435 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:22.432+0000 7f71a8e00640 1 -- 172.21.15.32:0/575557709 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f719804db60 con 0x55b859a89030 2024-08-03T20:58:22.436 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:22.433+0000 7f71b1af4f40 1 -- 172.21.15.32:0/575557709 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b859a89030 msgr2=0x55b859b11c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:58:22.436 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:22.433+0000 7f71b1af4f40 1 --2- 172.21.15.32:0/575557709 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b859a89030 0x55b859b11c40 secure :-1 s=READY pgs=1752 cs=0 l=1 rev1=1 crypto rx=0x7f71980467a0 tx=0x7f71980019b0 comp rx=0 tx=0).stop 2024-08-03T20:58:22.436 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:22.433+0000 7f71b1af4f40 1 -- 172.21.15.32:0/575557709 shutdown_connections 2024-08-03T20:58:22.436 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:22.433+0000 7f71b1af4f40 1 --2- 172.21.15.32:0/575557709 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55b859b12180 0x55b859b12590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:22.436 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:22.433+0000 7f71b1af4f40 1 --2- 172.21.15.32:0/575557709 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b859a89030 0x55b859b11c40 unknown :-1 s=CLOSED pgs=1752 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:22.436 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:22.433+0000 7f71b1af4f40 1 --2- 172.21.15.32:0/575557709 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f71900030f0 0x55b859a89720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:22.437 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:22.433+0000 7f71b1af4f40 1 -- 172.21.15.32:0/575557709 >> 172.21.15.32:0/575557709 conn(0x55b859960ac0 msgr2=0x55b859b11280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:58:22.437 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:22.433+0000 7f71b1af4f40 1 -- 172.21.15.32:0/575557709 shutdown_connections 2024-08-03T20:58:22.437 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:22.433+0000 7f71b1af4f40 1 -- 172.21.15.32:0/575557709 wait complete. 2024-08-03T20:58:22.437 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:22.434+0000 7f71b1af4f40 1 Processor -- start 2024-08-03T20:58:22.437 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:22.434+0000 7f71b1af4f40 1 -- start start 2024-08-03T20:58:22.439 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:22.435+0000 7f71b1af4f40 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55b859a89030 0x55b859aee590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:58:22.439 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:22.435+0000 7f71b1af4f40 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b859b12180 0x55b859baed90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:58:22.439 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:22.435+0000 7f71b1af4f40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f71900030f0 0x55b859baf2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:58:22.439 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:22.435+0000 7f71b1af4f40 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55b859b0e660 con 0x55b859b12180 2024-08-03T20:58:22.439 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:22.435+0000 7f71b1af4f40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x55b859b0ec20 con 0x7f71900030f0 2024-08-03T20:58:22.439 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:22.435+0000 7f71b1af4f40 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x55b859af5290 con 0x55b859a89030 2024-08-03T20:58:22.439 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:22.436+0000 7f71aa200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b859b12180 0x55b859baed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:58:22.440 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:22.436+0000 7f71aa200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b859b12180 0x55b859baed90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:42586/0 (socket says 172.21.15.32:42586) 2024-08-03T20:58:22.440 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:22.436+0000 7f71aa200640 1 -- 172.21.15.32:0/3472799416 learned_addr learned my addr 172.21.15.32:0/3472799416 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:58:22.440 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:22.436+0000 7f71aac00640 1 --2- 172.21.15.32:0/3472799416 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55b859a89030 0x55b859aee590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:58:22.440 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:22.436+0000 7f71ab600640 1 --2- 172.21.15.32:0/3472799416 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f71900030f0 0x55b859baf2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:58:22.440 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:22.436+0000 7f71aa200640 1 -- 172.21.15.32:0/3472799416 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55b859a89030 msgr2=0x55b859aee590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:58:22.440 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:22.436+0000 7f71aa200640 1 --2- 172.21.15.32:0/3472799416 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55b859a89030 0x55b859aee590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:22.441 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:22.436+0000 7f71aa200640 1 -- 172.21.15.32:0/3472799416 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f71900030f0 msgr2=0x55b859baf2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:58:22.441 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:22.436+0000 7f71aa200640 1 --2- 172.21.15.32:0/3472799416 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f71900030f0 0x55b859baf2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:22.441 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:22.436+0000 7f71aa200640 1 -- 172.21.15.32:0/3472799416 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b859baf810 con 0x55b859b12180 2024-08-03T20:58:22.441 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:22.437+0000 7f71aa200640 1 --2- 172.21.15.32:0/3472799416 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b859b12180 0x55b859baed90 secure :-1 s=READY pgs=1753 cs=0 l=1 rev1=1 crypto rx=0x7f7198009aa0 tx=0x7f719804de30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:58:22.441 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:22.437+0000 7f71a3400640 1 -- 172.21.15.32:0/3472799416 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7198071070 con 0x55b859b12180 2024-08-03T20:58:22.441 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:22.437+0000 7f71a3400640 1 -- 172.21.15.32:0/3472799416 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f7198060430 con 0x55b859b12180 2024-08-03T20:58:22.441 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:22.437+0000 7f71a3400640 1 -- 172.21.15.32:0/3472799416 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7198060880 con 0x55b859b12180 2024-08-03T20:58:22.441 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:22.437+0000 7f71b1af4f40 1 -- 172.21.15.32:0/3472799416 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x55b859b115d0 con 0x55b859b12180 2024-08-03T20:58:22.442 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:22.437+0000 7f71b1af4f40 1 -- 172.21.15.32:0/3472799416 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b859aeead0 con 0x55b859b12180 2024-08-03T20:58:22.442 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:22.439+0000 7f71a3400640 1 -- 172.21.15.32:0/3472799416 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f719805e040 con 0x55b859b12180 2024-08-03T20:58:22.443 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:22.440+0000 7f71a3400640 1 --2- 172.21.15.32:0/3472799416 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f718c0759f0 0x7f718c077f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:58:22.443 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:22.440+0000 7f71a3400640 1 -- 172.21.15.32:0/3472799416 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(113..113 src has 1..113) ==== 8262+0+0 (secure 0 0 0) 0x7f71980dc440 con 0x55b859b12180 2024-08-03T20:58:22.443 INFO:teuthology.orchestra.run.smithi032.stdout:Importing pool 2024-08-03T20:58:22.444 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:22.440+0000 7f71b1af4f40 1 --2- 172.21.15.32:0/3472799416 >> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] conn(0x55b859b12600 0x55b859bb7f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:58:22.444 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:22.440+0000 7f71aac00640 1 --2- 172.21.15.32:0/3472799416 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f718c0759f0 0x7f718c077f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:58:22.444 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:22.440+0000 7f71b1af4f40 1 -- 172.21.15.32:0/3472799416 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:1 12.10 12:09401a81:::601.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113) -- 0x55b859af2d90 con 0x55b859b12600 2024-08-03T20:58:22.444 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:22.441+0000 7f71ab600640 1 --2- 172.21.15.32:0/3472799416 >> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] conn(0x55b859b12600 0x55b859bb7f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:58:22.444 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:22.441+0000 7f71ab600640 1 --2- 172.21.15.32:0/3472799416 >> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] conn(0x55b859b12600 0x55b859bb7f20 crc :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:58:22.445 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:22.441+0000 7f71aac00640 1 --2- 172.21.15.32:0/3472799416 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f718c0759f0 0x7f718c077f00 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x55b859a2a7a0 tx=0x7f719c004880 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:58:22.445 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:22.444+0000 7f71ab600640 1 -- 172.21.15.32:0/3472799416 <== osd.0 v2:172.21.15.32:6816/3066156116 1 ==== osd_op_reply(1 601.00000000 [create] v113'846 uv820 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f71a4009c20 con 0x55b859b12600 2024-08-03T20:58:22.445 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:22.444+0000 7f71b1af4f40 1 -- 172.21.15.32:0/3472799416 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:2 12.10 12:09401a81:::601.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e113) -- 0x55b859af1120 con 0x55b859b12600 2024-08-03T20:58:22.447 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:22.446+0000 7f71ab600640 1 -- 172.21.15.32:0/3472799416 <== osd.0 v2:172.21.15.32:6816/3066156116 2 ==== osd_op_reply(2 601.00000000 [delete] v113'847 uv847 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f71a4009c20 con 0x55b859b12600 2024-08-03T20:58:22.447 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:22.446+0000 7f71b1af4f40 1 -- 172.21.15.32:0/3472799416 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:3 12.10 12:09401a81:::601.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113) -- 0x55b859bb8460 con 0x55b859b12600 2024-08-03T20:58:22.449 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::601.00000000:head# 2024-08-03T20:58:22.449 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:22.447+0000 7f71ab600640 1 -- 172.21.15.32:0/3472799416 <== osd.0 v2:172.21.15.32:6816/3066156116 3 ==== osd_op_reply(3 601.00000000 [create] v113'848 uv848 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f71a4009c20 con 0x55b859b12600 2024-08-03T20:58:22.449 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:22.447+0000 7f71b1af4f40 1 -- 172.21.15.32:0/3472799416 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:4 12.10 12:09401a81:::601.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113) -- 0x55b859bb9810 con 0x55b859b12600 2024-08-03T20:58:22.450 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:22.449+0000 7f71ab600640 1 -- 172.21.15.32:0/3472799416 <== osd.0 v2:172.21.15.32:6816/3066156116 4 ==== osd_op_reply(4 601.00000000 [setxattr (30)] v113'849 uv849 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f71a4009c20 con 0x55b859b12600 2024-08-03T20:58:22.451 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:22.449+0000 7f71b1af4f40 1 -- 172.21.15.32:0/3472799416 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:5 12.10 12:09401a81:::601.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113) -- 0x55b859bb9bb0 con 0x55b859b12600 2024-08-03T20:58:22.453 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:22.451+0000 7f71ab600640 1 -- 172.21.15.32:0/3472799416 <== osd.0 v2:172.21.15.32:6816/3066156116 5 ==== osd_op_reply(5 601.00000000 [setxattr (62)] v113'850 uv850 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f71a4009c20 con 0x55b859b12600 2024-08-03T20:58:22.453 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:22.451+0000 7f71b1af4f40 1 -- 172.21.15.32:0/3472799416 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:6 12.10 12:09401a81:::601.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113) -- 0x55b859bb8800 con 0x55b859b12600 2024-08-03T20:58:22.454 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:22.453+0000 7f71ab600640 1 -- 172.21.15.32:0/3472799416 <== osd.0 v2:172.21.15.32:6816/3066156116 6 ==== osd_op_reply(6 601.00000000 [omap-set-header] v113'851 uv851 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f71a4009c20 con 0x55b859b12600 2024-08-03T20:58:22.454 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:22.453+0000 7f71b1af4f40 1 -- 172.21.15.32:0/3472799416 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:7 12.10 12:0bd6d154:::602.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113) -- 0x55b859bb8bf0 con 0x55b859b12600 2024-08-03T20:58:22.456 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:22.454+0000 7f71ab600640 1 -- 172.21.15.32:0/3472799416 <== osd.0 v2:172.21.15.32:6816/3066156116 7 ==== osd_op_reply(7 602.00000000 [create] v113'852 uv826 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f71a4009c20 con 0x55b859b12600 2024-08-03T20:58:22.456 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:22.454+0000 7f71b1af4f40 1 -- 172.21.15.32:0/3472799416 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:8 12.10 12:0bd6d154:::602.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e113) -- 0x55b859bb8f90 con 0x55b859b12600 2024-08-03T20:58:22.458 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:22.456+0000 7f71ab600640 1 -- 172.21.15.32:0/3472799416 <== osd.0 v2:172.21.15.32:6816/3066156116 8 ==== osd_op_reply(8 602.00000000 [delete] v113'853 uv853 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f71a4009c20 con 0x55b859b12600 2024-08-03T20:58:22.458 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:22.456+0000 7f71b1af4f40 1 -- 172.21.15.32:0/3472799416 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:9 12.10 12:0bd6d154:::602.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113) -- 0x55b859bb9330 con 0x55b859b12600 2024-08-03T20:58:22.459 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::602.00000000:head# 2024-08-03T20:58:22.459 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:22.458+0000 7f71ab600640 1 -- 172.21.15.32:0/3472799416 <== osd.0 v2:172.21.15.32:6816/3066156116 9 ==== osd_op_reply(9 602.00000000 [create] v113'854 uv854 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f71a4009c20 con 0x55b859b12600 2024-08-03T20:58:22.459 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:22.458+0000 7f71b1af4f40 1 -- 172.21.15.32:0/3472799416 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:10 12.10 12:0bd6d154:::602.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113) -- 0x55b859bb9330 con 0x55b859b12600 2024-08-03T20:58:22.461 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:22.459+0000 7f71ab600640 1 -- 172.21.15.32:0/3472799416 <== osd.0 v2:172.21.15.32:6816/3066156116 10 ==== osd_op_reply(10 602.00000000 [setxattr (30)] v113'855 uv855 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f71a4009c20 con 0x55b859b12600 2024-08-03T20:58:22.461 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:22.459+0000 7f71b1af4f40 1 -- 172.21.15.32:0/3472799416 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:11 12.10 12:0bd6d154:::602.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113) -- 0x55b859bb9330 con 0x55b859b12600 2024-08-03T20:58:22.462 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:22.461+0000 7f71ab600640 1 -- 172.21.15.32:0/3472799416 <== osd.0 v2:172.21.15.32:6816/3066156116 11 ==== osd_op_reply(11 602.00000000 [setxattr (62)] v113'856 uv856 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f71a4009c20 con 0x55b859b12600 2024-08-03T20:58:22.462 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:22.461+0000 7f71b1af4f40 1 -- 172.21.15.32:0/3472799416 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:12 12.10 12:0bd6d154:::602.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113) -- 0x55b859bb9330 con 0x55b859b12600 2024-08-03T20:58:22.464 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:22.463+0000 7f71ab600640 1 -- 172.21.15.32:0/3472799416 <== osd.0 v2:172.21.15.32:6816/3066156116 12 ==== osd_op_reply(12 602.00000000 [omap-set-header] v113'857 uv857 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f71a4009c20 con 0x55b859b12600 2024-08-03T20:58:22.464 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:22.463+0000 7f71b1af4f40 1 -- 172.21.15.32:0/3472799416 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:13 12.10 12:0cf6ad0f:::10000000000.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113) -- 0x55b859bb9330 con 0x55b859b12600 2024-08-03T20:58:22.466 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:22.464+0000 7f71ab600640 1 -- 172.21.15.32:0/3472799416 <== osd.0 v2:172.21.15.32:6816/3066156116 13 ==== osd_op_reply(13 10000000000.00000000 [create] v113'858 uv833 ondisk = -17 ((17) File exists)) ==== 164+0+0 (crc 0 0 0) 0x7f71a4009c20 con 0x55b859b12600 2024-08-03T20:58:22.466 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:22.464+0000 7f71b1af4f40 1 -- 172.21.15.32:0/3472799416 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:14 12.10 12:0cf6ad0f:::10000000000.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e113) -- 0x55b859bb9330 con 0x55b859b12600 2024-08-03T20:58:22.467 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:22.466+0000 7f71ab600640 1 -- 172.21.15.32:0/3472799416 <== osd.0 v2:172.21.15.32:6816/3066156116 14 ==== osd_op_reply(14 10000000000.00000000 [delete] v113'859 uv859 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f71a4009c20 con 0x55b859b12600 2024-08-03T20:58:22.468 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:22.466+0000 7f71b1af4f40 1 -- 172.21.15.32:0/3472799416 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:15 12.10 12:0cf6ad0f:::10000000000.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113) -- 0x55b859bb9330 con 0x55b859b12600 2024-08-03T20:58:22.469 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::10000000000.00000000:head# 2024-08-03T20:58:22.469 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:22.468+0000 7f71ab600640 1 -- 172.21.15.32:0/3472799416 <== osd.0 v2:172.21.15.32:6816/3066156116 15 ==== osd_op_reply(15 10000000000.00000000 [create] v113'860 uv860 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f71a4009c20 con 0x55b859b12600 2024-08-03T20:58:22.470 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:22.468+0000 7f71b1af4f40 1 -- 172.21.15.32:0/3472799416 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:16 12.10 12:0cf6ad0f:::10000000000.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113) -- 0x55b859bb9330 con 0x55b859b12600 2024-08-03T20:58:22.474 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:22.470+0000 7f71ab600640 1 -- 172.21.15.32:0/3472799416 <== osd.0 v2:172.21.15.32:6816/3066156116 16 ==== osd_op_reply(16 10000000000.00000000 [setxattr (30)] v113'861 uv861 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f71a4009c20 con 0x55b859b12600 2024-08-03T20:58:22.474 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:22.470+0000 7f71b1af4f40 1 -- 172.21.15.32:0/3472799416 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:17 12.10 12:0cf6ad0f:::10000000000.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113) -- 0x55b859bb9330 con 0x55b859b12600 2024-08-03T20:58:22.474 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:22.471+0000 7f71ab600640 1 -- 172.21.15.32:0/3472799416 <== osd.0 v2:172.21.15.32:6816/3066156116 17 ==== osd_op_reply(17 10000000000.00000000 [setxattr (62)] v113'862 uv862 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f71a4009c20 con 0x55b859b12600 2024-08-03T20:58:22.474 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:22.471+0000 7f71b1af4f40 1 -- 172.21.15.32:0/3472799416 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:18 12.10 12:0cf6ad0f:::10000000000.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113) -- 0x55b859bb9330 con 0x55b859b12600 2024-08-03T20:58:22.475 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:22.473+0000 7f71ab600640 1 -- 172.21.15.32:0/3472799416 <== osd.0 v2:172.21.15.32:6816/3066156116 18 ==== osd_op_reply(18 10000000000.00000000 [omap-set-header] v113'863 uv863 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f71a4009c20 con 0x55b859b12600 2024-08-03T20:58:22.475 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:22.473+0000 7f71b1af4f40 1 -- 172.21.15.32:0/3472799416 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:19 12.10 12:0cf6ad0f:::10000000000.00000000:head [omap-set-vals in=593b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113) -- 0x55b859bb9330 con 0x55b859b12600 2024-08-03T20:58:22.476 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:22.475+0000 7f71ab600640 1 -- 172.21.15.32:0/3472799416 <== osd.0 v2:172.21.15.32:6816/3066156116 19 ==== osd_op_reply(19 10000000000.00000000 [omap-set-vals] v113'864 uv864 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f71a4009c20 con 0x55b859b12600 2024-08-03T20:58:22.476 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:22.475+0000 7f71b1af4f40 1 -- 172.21.15.32:0/3472799416 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:20 12.10 12:0d82a743:::600.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113) -- 0x55b859bb9330 con 0x55b859b12600 2024-08-03T20:58:22.478 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:22.476+0000 7f71ab600640 1 -- 172.21.15.32:0/3472799416 <== osd.0 v2:172.21.15.32:6816/3066156116 20 ==== osd_op_reply(20 600.00000000 [create] v113'865 uv839 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f71a4009c20 con 0x55b859b12600 2024-08-03T20:58:22.478 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:22.476+0000 7f71b1af4f40 1 -- 172.21.15.32:0/3472799416 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:21 12.10 12:0d82a743:::600.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e113) -- 0x55b859bb9330 con 0x55b859b12600 2024-08-03T20:58:22.479 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:22.478+0000 7f71ab600640 1 -- 172.21.15.32:0/3472799416 <== osd.0 v2:172.21.15.32:6816/3066156116 21 ==== osd_op_reply(21 600.00000000 [delete] v113'866 uv866 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f71a4009c20 con 0x55b859b12600 2024-08-03T20:58:22.480 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:22.478+0000 7f71b1af4f40 1 -- 172.21.15.32:0/3472799416 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:22 12.10 12:0d82a743:::600.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113) -- 0x55b859bb9330 con 0x55b859b12600 2024-08-03T20:58:22.481 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::600.00000000:head# 2024-08-03T20:58:22.481 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:22.480+0000 7f71ab600640 1 -- 172.21.15.32:0/3472799416 <== osd.0 v2:172.21.15.32:6816/3066156116 22 ==== osd_op_reply(22 600.00000000 [create] v113'867 uv867 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f71a4009c20 con 0x55b859b12600 2024-08-03T20:58:22.481 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:22.480+0000 7f71b1af4f40 1 -- 172.21.15.32:0/3472799416 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:23 12.10 12:0d82a743:::600.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113) -- 0x55b859bb9330 con 0x55b859b12600 2024-08-03T20:58:22.483 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:22.481+0000 7f71ab600640 1 -- 172.21.15.32:0/3472799416 <== osd.0 v2:172.21.15.32:6816/3066156116 23 ==== osd_op_reply(23 600.00000000 [setxattr (30)] v113'868 uv868 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f71a4009c20 con 0x55b859b12600 2024-08-03T20:58:22.483 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:22.481+0000 7f71b1af4f40 1 -- 172.21.15.32:0/3472799416 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:24 12.10 12:0d82a743:::600.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113) -- 0x55b859bb9330 con 0x55b859b12600 2024-08-03T20:58:22.484 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:22.483+0000 7f71ab600640 1 -- 172.21.15.32:0/3472799416 <== osd.0 v2:172.21.15.32:6816/3066156116 24 ==== osd_op_reply(24 600.00000000 [setxattr (62)] v113'869 uv869 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f71a4009c20 con 0x55b859b12600 2024-08-03T20:58:22.484 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:22.483+0000 7f71b1af4f40 1 -- 172.21.15.32:0/3472799416 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:25 12.10 12:0d82a743:::600.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113) -- 0x55b859bb9330 con 0x55b859b12600 2024-08-03T20:58:22.486 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:22.485+0000 7f71ab600640 1 -- 172.21.15.32:0/3472799416 <== osd.0 v2:172.21.15.32:6816/3066156116 25 ==== osd_op_reply(25 600.00000000 [omap-set-header] v113'870 uv870 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f71a4009c20 con 0x55b859b12600 2024-08-03T20:58:22.486 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:22.485+0000 7f71b1af4f40 1 -- 172.21.15.32:0/3472799416 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:26 12.10 12:0d89b25e:::603.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113) -- 0x55b859bb9330 con 0x55b859b12600 2024-08-03T20:58:22.488 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:22.486+0000 7f71ab600640 1 -- 172.21.15.32:0/3472799416 <== osd.0 v2:172.21.15.32:6816/3066156116 26 ==== osd_op_reply(26 603.00000000 [create] v113'871 uv845 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f71a4009c20 con 0x55b859b12600 2024-08-03T20:58:22.488 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:22.486+0000 7f71b1af4f40 1 -- 172.21.15.32:0/3472799416 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:27 12.10 12:0d89b25e:::603.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e113) -- 0x55b859bb9330 con 0x55b859b12600 2024-08-03T20:58:22.489 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:22.488+0000 7f71ab600640 1 -- 172.21.15.32:0/3472799416 <== osd.0 v2:172.21.15.32:6816/3066156116 27 ==== osd_op_reply(27 603.00000000 [delete] v113'872 uv872 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f71a4009c20 con 0x55b859b12600 2024-08-03T20:58:22.489 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:22.488+0000 7f71b1af4f40 1 -- 172.21.15.32:0/3472799416 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:28 12.10 12:0d89b25e:::603.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113) -- 0x55b859bb9330 con 0x55b859b12600 2024-08-03T20:58:22.491 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::603.00000000:head# 2024-08-03T20:58:22.491 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:22.489+0000 7f71ab600640 1 -- 172.21.15.32:0/3472799416 <== osd.0 v2:172.21.15.32:6816/3066156116 28 ==== osd_op_reply(28 603.00000000 [create] v113'873 uv873 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f71a4009c20 con 0x55b859b12600 2024-08-03T20:58:22.491 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:22.489+0000 7f71b1af4f40 1 -- 172.21.15.32:0/3472799416 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:29 12.10 12:0d89b25e:::603.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113) -- 0x55b859bb9330 con 0x55b859b12600 2024-08-03T20:58:22.492 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:22.491+0000 7f71ab600640 1 -- 172.21.15.32:0/3472799416 <== osd.0 v2:172.21.15.32:6816/3066156116 29 ==== osd_op_reply(29 603.00000000 [setxattr (30)] v113'874 uv874 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f71a4009c20 con 0x55b859b12600 2024-08-03T20:58:22.492 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:22.491+0000 7f71b1af4f40 1 -- 172.21.15.32:0/3472799416 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:30 12.10 12:0d89b25e:::603.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113) -- 0x55b859bb9330 con 0x55b859b12600 2024-08-03T20:58:22.494 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:22.493+0000 7f71ab600640 1 -- 172.21.15.32:0/3472799416 <== osd.0 v2:172.21.15.32:6816/3066156116 30 ==== osd_op_reply(30 603.00000000 [setxattr (62)] v113'875 uv875 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f71a4009c20 con 0x55b859b12600 2024-08-03T20:58:22.494 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:22.493+0000 7f71b1af4f40 1 -- 172.21.15.32:0/3472799416 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:31 12.10 12:0d89b25e:::603.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113) -- 0x55b859bb9330 con 0x55b859b12600 2024-08-03T20:58:22.495 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:22.494+0000 7f71ab600640 1 -- 172.21.15.32:0/3472799416 <== osd.0 v2:172.21.15.32:6816/3066156116 31 ==== osd_op_reply(31 603.00000000 [omap-set-header] v113'876 uv876 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f71a4009c20 con 0x55b859b12600 2024-08-03T20:58:22.496 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:22.494+0000 7f71b1af4f40 1 --2- 172.21.15.32:0/3472799416 >> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] conn(0x55b859bba460 0x55b859bbc880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:58:22.496 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:22.494+0000 7f71b1af4f40 1 -- 172.21.15.32:0/3472799416 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:32 12.14 12:287034c4:::1.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113) -- 0x55b859bb9330 con 0x55b859bba460 2024-08-03T20:58:22.496 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:22.495+0000 7f71aac00640 1 --2- 172.21.15.32:0/3472799416 >> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] conn(0x55b859bba460 0x55b859bbc880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:58:22.497 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:22.495+0000 7f71aac00640 1 --2- 172.21.15.32:0/3472799416 >> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] conn(0x55b859bba460 0x55b859bbc880 crc :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:58:22.499 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:22.498+0000 7f71aac00640 1 -- 172.21.15.32:0/3472799416 <== osd.5 v2:172.21.15.118:6810/2724891770 1 ==== osd_op_reply(32 1.00000000.inode [create] v113'617 uv600 ondisk = -17 ((17) File exists)) ==== 160+0+0 (crc 0 0 0) 0x7f719c00c070 con 0x55b859bba460 2024-08-03T20:58:22.499 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:22.498+0000 7f71b1af4f40 1 -- 172.21.15.32:0/3472799416 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:33 12.14 12:287034c4:::1.00000000.inode:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e113) -- 0x55b859bbcdc0 con 0x55b859bba460 2024-08-03T20:58:22.501 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:22.499+0000 7f71aac00640 1 -- 172.21.15.32:0/3472799416 <== osd.5 v2:172.21.15.118:6810/2724891770 2 ==== osd_op_reply(33 1.00000000.inode [delete] v113'618 uv618 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f719c00c070 con 0x55b859bba460 2024-08-03T20:58:22.501 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:22.499+0000 7f71b1af4f40 1 -- 172.21.15.32:0/3472799416 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:34 12.14 12:287034c4:::1.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113) -- 0x55b859bbd160 con 0x55b859bba460 2024-08-03T20:58:22.502 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::1.00000000.inode:head# 2024-08-03T20:58:22.503 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:22.501+0000 7f71aac00640 1 -- 172.21.15.32:0/3472799416 <== osd.5 v2:172.21.15.118:6810/2724891770 3 ==== osd_op_reply(34 1.00000000.inode [create] v113'619 uv619 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f719c00c070 con 0x55b859bba460 2024-08-03T20:58:22.503 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:22.501+0000 7f71b1af4f40 1 -- 172.21.15.32:0/3472799416 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:35 12.14 12:287034c4:::1.00000000.inode:head [write 0~582 in=582b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113) -- 0x55b859bbd500 con 0x55b859bba460 2024-08-03T20:58:22.504 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:22.502+0000 7f71aac00640 1 -- 172.21.15.32:0/3472799416 <== osd.5 v2:172.21.15.118:6810/2724891770 4 ==== osd_op_reply(35 1.00000000.inode [write 0~582] v113'620 uv620 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f719c00c070 con 0x55b859bba460 2024-08-03T20:58:22.504 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:22.503+0000 7f71b1af4f40 1 -- 172.21.15.32:0/3472799416 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:36 12.14 12:287034c4:::1.00000000.inode:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113) -- 0x55b859bbd8a0 con 0x55b859bba460 2024-08-03T20:58:22.505 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:22.504+0000 7f71aac00640 1 -- 172.21.15.32:0/3472799416 <== osd.5 v2:172.21.15.118:6810/2724891770 5 ==== osd_op_reply(36 1.00000000.inode [omap-set-header] v113'621 uv621 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f719c00c070 con 0x55b859bba460 2024-08-03T20:58:22.505 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:22.504+0000 7f71b1af4f40 1 -- 172.21.15.32:0/3472799416 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:37 12.14 12:292cf221:::200.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113) -- 0x55b859bbdc40 con 0x55b859bba460 2024-08-03T20:58:22.507 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:22.505+0000 7f71aac00640 1 -- 172.21.15.32:0/3472799416 <== osd.5 v2:172.21.15.118:6810/2724891770 6 ==== osd_op_reply(37 200.00000000 [create] v113'622 uv605 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f719c00c070 con 0x55b859bba460 2024-08-03T20:58:22.507 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:22.506+0000 7f71b1af4f40 1 -- 172.21.15.32:0/3472799416 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:38 12.14 12:292cf221:::200.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e113) -- 0x55b859bbdfe0 con 0x55b859bba460 2024-08-03T20:58:22.508 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:22.507+0000 7f71aac00640 1 -- 172.21.15.32:0/3472799416 <== osd.5 v2:172.21.15.118:6810/2724891770 7 ==== osd_op_reply(38 200.00000000 [delete] v113'623 uv623 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f719c00c070 con 0x55b859bba460 2024-08-03T20:58:22.509 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:22.507+0000 7f71b1af4f40 1 -- 172.21.15.32:0/3472799416 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:39 12.14 12:292cf221:::200.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113) -- 0x55b859bbe380 con 0x55b859bba460 2024-08-03T20:58:22.510 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::200.00000000:head# 2024-08-03T20:58:22.510 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:22.509+0000 7f71aac00640 1 -- 172.21.15.32:0/3472799416 <== osd.5 v2:172.21.15.118:6810/2724891770 8 ==== osd_op_reply(39 200.00000000 [create] v113'624 uv624 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f719c00c070 con 0x55b859bba460 2024-08-03T20:58:22.510 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:22.509+0000 7f71b1af4f40 1 -- 172.21.15.32:0/3472799416 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:40 12.14 12:292cf221:::200.00000000:head [write 0~90 in=90b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113) -- 0x55b859bbe380 con 0x55b859bba460 2024-08-03T20:58:22.512 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:22.510+0000 7f71aac00640 1 -- 172.21.15.32:0/3472799416 <== osd.5 v2:172.21.15.118:6810/2724891770 9 ==== osd_op_reply(40 200.00000000 [write 0~90] v113'625 uv625 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f719c00c070 con 0x55b859bba460 2024-08-03T20:58:22.512 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:22.510+0000 7f71b1af4f40 1 -- 172.21.15.32:0/3472799416 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:41 12.14 12:292cf221:::200.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113) -- 0x55b859bbe380 con 0x55b859bba460 2024-08-03T20:58:22.513 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:22.512+0000 7f71aac00640 1 -- 172.21.15.32:0/3472799416 <== osd.5 v2:172.21.15.118:6810/2724891770 10 ==== osd_op_reply(41 200.00000000 [omap-set-header] v113'626 uv626 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f719c00c070 con 0x55b859bba460 2024-08-03T20:58:22.513 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:22.512+0000 7f71b1af4f40 1 -- 172.21.15.32:0/3472799416 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:42 12.14 12:2e2fa760:::200.00000001:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113) -- 0x55b859bbe380 con 0x55b859bba460 2024-08-03T20:58:22.515 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:22.514+0000 7f71aac00640 1 -- 172.21.15.32:0/3472799416 <== osd.5 v2:172.21.15.118:6810/2724891770 11 ==== osd_op_reply(42 200.00000001 [create] v113'627 uv610 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f719c00c070 con 0x55b859bba460 2024-08-03T20:58:22.515 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:22.514+0000 7f71b1af4f40 1 -- 172.21.15.32:0/3472799416 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:43 12.14 12:2e2fa760:::200.00000001:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e113) -- 0x55b859bbe380 con 0x55b859bba460 2024-08-03T20:58:22.516 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:22.515+0000 7f71aac00640 1 -- 172.21.15.32:0/3472799416 <== osd.5 v2:172.21.15.118:6810/2724891770 12 ==== osd_op_reply(43 200.00000001 [delete] v113'628 uv628 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f719c00c070 con 0x55b859bba460 2024-08-03T20:58:22.516 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:22.515+0000 7f71b1af4f40 1 -- 172.21.15.32:0/3472799416 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:44 12.14 12:2e2fa760:::200.00000001:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113) -- 0x55b859bbe380 con 0x55b859bba460 2024-08-03T20:58:22.518 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::200.00000001:head# 2024-08-03T20:58:22.518 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:22.516+0000 7f71aac00640 1 -- 172.21.15.32:0/3472799416 <== osd.5 v2:172.21.15.118:6810/2724891770 13 ==== osd_op_reply(44 200.00000001 [create] v113'629 uv629 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f719c00c070 con 0x55b859bba460 2024-08-03T20:58:22.518 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:22.516+0000 7f71b1af4f40 1 -- 172.21.15.32:0/3472799416 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:45 12.14 12:2e2fa760:::200.00000001:head [write 0~18118 in=18118b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113) -- 0x55b859bbe380 con 0x55b859bba460 2024-08-03T20:58:22.520 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:22.518+0000 7f71aac00640 1 -- 172.21.15.32:0/3472799416 <== osd.5 v2:172.21.15.118:6810/2724891770 14 ==== osd_op_reply(45 200.00000001 [write 0~18118] v113'630 uv630 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f719c00c070 con 0x55b859bba460 2024-08-03T20:58:22.520 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:22.518+0000 7f71b1af4f40 1 -- 172.21.15.32:0/3472799416 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:46 12.14 12:2e2fa760:::200.00000001:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113) -- 0x55b859bbe380 con 0x55b859bba460 2024-08-03T20:58:22.521 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:22.520+0000 7f71aac00640 1 -- 172.21.15.32:0/3472799416 <== osd.5 v2:172.21.15.118:6810/2724891770 15 ==== osd_op_reply(46 200.00000001 [omap-set-header] v113'631 uv631 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f719c00c070 con 0x55b859bba460 2024-08-03T20:58:22.521 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:22.520+0000 7f71b1af4f40 1 -- 172.21.15.32:0/3472799416 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:47 12.14 12:2f55791f:::606.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113) -- 0x55b859bbe380 con 0x55b859bba460 2024-08-03T20:58:22.523 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:22.522+0000 7f71aac00640 1 -- 172.21.15.32:0/3472799416 <== osd.5 v2:172.21.15.118:6810/2724891770 16 ==== osd_op_reply(47 606.00000000 [create] v113'632 uv616 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f719c00c070 con 0x55b859bba460 2024-08-03T20:58:22.523 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:22.522+0000 7f71b1af4f40 1 -- 172.21.15.32:0/3472799416 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:48 12.14 12:2f55791f:::606.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e113) -- 0x55b859bbe380 con 0x55b859bba460 2024-08-03T20:58:22.524 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:22.523+0000 7f71aac00640 1 -- 172.21.15.32:0/3472799416 <== osd.5 v2:172.21.15.118:6810/2724891770 17 ==== osd_op_reply(48 606.00000000 [delete] v113'633 uv633 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f719c00c070 con 0x55b859bba460 2024-08-03T20:58:22.524 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:22.523+0000 7f71b1af4f40 1 -- 172.21.15.32:0/3472799416 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:49 12.14 12:2f55791f:::606.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113) -- 0x55b859bbe380 con 0x55b859bba460 2024-08-03T20:58:22.526 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::606.00000000:head# 2024-08-03T20:58:22.526 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:22.525+0000 7f71aac00640 1 -- 172.21.15.32:0/3472799416 <== osd.5 v2:172.21.15.118:6810/2724891770 18 ==== osd_op_reply(49 606.00000000 [create] v113'634 uv634 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f719c00c070 con 0x55b859bba460 2024-08-03T20:58:22.526 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:22.525+0000 7f71b1af4f40 1 -- 172.21.15.32:0/3472799416 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:50 12.14 12:2f55791f:::606.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113) -- 0x55b859bbe3c0 con 0x55b859bba460 2024-08-03T20:58:22.527 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:22.526+0000 7f71aac00640 1 -- 172.21.15.32:0/3472799416 <== osd.5 v2:172.21.15.118:6810/2724891770 19 ==== osd_op_reply(50 606.00000000 [setxattr (30)] v113'635 uv635 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f719c00c070 con 0x55b859bba460 2024-08-03T20:58:22.528 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:22.526+0000 7f71b1af4f40 1 -- 172.21.15.32:0/3472799416 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:51 12.14 12:2f55791f:::606.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113) -- 0x55b859bbf900 con 0x55b859bba460 2024-08-03T20:58:22.529 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:22.528+0000 7f71aac00640 1 -- 172.21.15.32:0/3472799416 <== osd.5 v2:172.21.15.118:6810/2724891770 20 ==== osd_op_reply(51 606.00000000 [setxattr (62)] v113'636 uv636 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f719c00c070 con 0x55b859bba460 2024-08-03T20:58:22.529 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:22.528+0000 7f71b1af4f40 1 -- 172.21.15.32:0/3472799416 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:52 12.14 12:2f55791f:::606.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113) -- 0x55b859bbf900 con 0x55b859bba460 2024-08-03T20:58:22.531 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:22.529+0000 7f71aac00640 1 -- 172.21.15.32:0/3472799416 <== osd.5 v2:172.21.15.118:6810/2724891770 21 ==== osd_op_reply(52 606.00000000 [omap-set-header] v113'637 uv637 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f719c00c070 con 0x55b859bba460 2024-08-03T20:58:22.531 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:22.529+0000 7f71b1af4f40 1 -- 172.21.15.32:0/3472799416 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:53 12.1c 12:3ed09add:::607.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113) -- 0x55b859bbf840 con 0x55b859b12600 2024-08-03T20:58:22.532 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:22.531+0000 7f71ab600640 1 -- 172.21.15.32:0/3472799416 <== osd.0 v2:172.21.15.32:6816/3066156116 32 ==== osd_op_reply(53 607.00000000 [create] v113'165 uv164 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f71a4009c20 con 0x55b859b12600 2024-08-03T20:58:22.532 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:22.531+0000 7f71b1af4f40 1 -- 172.21.15.32:0/3472799416 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:54 12.1c 12:3ed09add:::607.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e113) -- 0x55b859bbf840 con 0x55b859b12600 2024-08-03T20:58:22.534 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:22.532+0000 7f71ab600640 1 -- 172.21.15.32:0/3472799416 <== osd.0 v2:172.21.15.32:6816/3066156116 33 ==== osd_op_reply(54 607.00000000 [delete] v113'166 uv166 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f71a4009c20 con 0x55b859b12600 2024-08-03T20:58:22.534 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:22.532+0000 7f71b1af4f40 1 -- 172.21.15.32:0/3472799416 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:55 12.1c 12:3ed09add:::607.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113) -- 0x55b859bbf840 con 0x55b859b12600 2024-08-03T20:58:22.535 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:22.533+0000 7f71ab600640 1 -- 172.21.15.32:0/3472799416 <== osd.0 v2:172.21.15.32:6816/3066156116 34 ==== osd_op_reply(55 607.00000000 [create] v113'167 uv167 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f71a4009c20 con 0x55b859b12600 2024-08-03T20:58:22.535 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::607.00000000:head# 2024-08-03T20:58:22.535 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:22.533+0000 7f71b1af4f40 1 -- 172.21.15.32:0/3472799416 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:56 12.1c 12:3ed09add:::607.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113) -- 0x55b859bbf840 con 0x55b859b12600 2024-08-03T20:58:22.536 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:22.535+0000 7f71ab600640 1 -- 172.21.15.32:0/3472799416 <== osd.0 v2:172.21.15.32:6816/3066156116 35 ==== osd_op_reply(56 607.00000000 [setxattr (30)] v113'168 uv168 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f71a4009c20 con 0x55b859b12600 2024-08-03T20:58:22.536 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:22.535+0000 7f71b1af4f40 1 -- 172.21.15.32:0/3472799416 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:57 12.1c 12:3ed09add:::607.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113) -- 0x55b859bbf840 con 0x55b859b12600 2024-08-03T20:58:22.537 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:22.536+0000 7f71ab600640 1 -- 172.21.15.32:0/3472799416 <== osd.0 v2:172.21.15.32:6816/3066156116 36 ==== osd_op_reply(57 607.00000000 [setxattr (62)] v113'169 uv169 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f71a4009c20 con 0x55b859b12600 2024-08-03T20:58:22.537 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:22.536+0000 7f71b1af4f40 1 -- 172.21.15.32:0/3472799416 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:58 12.1c 12:3ed09add:::607.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113) -- 0x55b859bbf840 con 0x55b859b12600 2024-08-03T20:58:22.539 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:22.537+0000 7f71ab600640 1 -- 172.21.15.32:0/3472799416 <== osd.0 v2:172.21.15.32:6816/3066156116 37 ==== osd_op_reply(58 607.00000000 [omap-set-header] v113'170 uv170 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f71a4009c20 con 0x55b859b12600 2024-08-03T20:58:22.539 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:22.537+0000 7f71b1af4f40 1 --2- 172.21.15.32:0/3472799416 >> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] conn(0x55b859bbed00 0x55b859bbf110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:58:22.539 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:22.537+0000 7f71b1af4f40 1 -- 172.21.15.32:0/3472799416 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:59 12.6 12:654134d2:::mds0_openfiles.0:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113) -- 0x55b859bc1d00 con 0x55b859bbed00 2024-08-03T20:58:22.539 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:22.538+0000 7f71aa200640 1 --2- 172.21.15.32:0/3472799416 >> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] conn(0x55b859bbed00 0x55b859bbf110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:58:22.539 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:22.538+0000 7f71aa200640 1 --2- 172.21.15.32:0/3472799416 >> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] conn(0x55b859bbed00 0x55b859bbf110 crc :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:58:22.541 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:22.539+0000 7f71aa200640 1 -- 172.21.15.32:0/3472799416 <== osd.1 v2:172.21.15.32:6824/1551241759 1 ==== osd_op_reply(59 mds0_openfiles.0 [create] v113'129 uv128 ondisk = -17 ((17) File exists)) ==== 160+0+0 (crc 0 0 0) 0x7f71980a08f0 con 0x55b859bbed00 2024-08-03T20:58:22.541 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:22.539+0000 7f71b1af4f40 1 -- 172.21.15.32:0/3472799416 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:60 12.6 12:654134d2:::mds0_openfiles.0:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e113) -- 0x55b859bc20a0 con 0x55b859bbed00 2024-08-03T20:58:22.542 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:22.541+0000 7f71aa200640 1 -- 172.21.15.32:0/3472799416 <== osd.1 v2:172.21.15.32:6824/1551241759 2 ==== osd_op_reply(60 mds0_openfiles.0 [delete] v113'130 uv130 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f71980a08f0 con 0x55b859bbed00 2024-08-03T20:58:22.542 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:22.541+0000 7f71b1af4f40 1 -- 172.21.15.32:0/3472799416 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:61 12.6 12:654134d2:::mds0_openfiles.0:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113) -- 0x55b859bc2500 con 0x55b859bbed00 2024-08-03T20:58:22.543 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds0_openfiles.0:head# 2024-08-03T20:58:22.544 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:22.542+0000 7f71aa200640 1 -- 172.21.15.32:0/3472799416 <== osd.1 v2:172.21.15.32:6824/1551241759 3 ==== osd_op_reply(61 mds0_openfiles.0 [create] v113'131 uv131 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f71980a08f0 con 0x55b859bbed00 2024-08-03T20:58:22.544 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:22.542+0000 7f71b1af4f40 1 -- 172.21.15.32:0/3472799416 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:62 12.6 12:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113) -- 0x55b859b11d20 con 0x55b859bbed00 2024-08-03T20:58:22.545 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:22.544+0000 7f71aa200640 1 -- 172.21.15.32:0/3472799416 <== osd.1 v2:172.21.15.32:6824/1551241759 4 ==== osd_op_reply(62 mds0_openfiles.0 [omap-set-header] v113'132 uv132 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f71980a08f0 con 0x55b859bbed00 2024-08-03T20:58:22.545 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:22.544+0000 7f71b1af4f40 1 -- 172.21.15.32:0/3472799416 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:63 12.e 12:717a0223:::608.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113) -- 0x55b859bc2e30 con 0x55b859b12600 2024-08-03T20:58:22.547 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:22.545+0000 7f71ab600640 1 -- 172.21.15.32:0/3472799416 <== osd.0 v2:172.21.15.32:6816/3066156116 38 ==== osd_op_reply(63 608.00000000 [create] v113'165 uv164 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f71a4009c20 con 0x55b859b12600 2024-08-03T20:58:22.547 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:22.545+0000 7f71b1af4f40 1 -- 172.21.15.32:0/3472799416 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:64 12.e 12:717a0223:::608.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e113) -- 0x55b859bc2e30 con 0x55b859b12600 2024-08-03T20:58:22.548 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:22.547+0000 7f71ab600640 1 -- 172.21.15.32:0/3472799416 <== osd.0 v2:172.21.15.32:6816/3066156116 39 ==== osd_op_reply(64 608.00000000 [delete] v113'166 uv166 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f71a4009c20 con 0x55b859b12600 2024-08-03T20:58:22.548 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:22.547+0000 7f71b1af4f40 1 -- 172.21.15.32:0/3472799416 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:65 12.e 12:717a0223:::608.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113) -- 0x55b859bc2e30 con 0x55b859b12600 2024-08-03T20:58:22.549 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::608.00000000:head# 2024-08-03T20:58:22.550 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:22.548+0000 7f71ab600640 1 -- 172.21.15.32:0/3472799416 <== osd.0 v2:172.21.15.32:6816/3066156116 40 ==== osd_op_reply(65 608.00000000 [create] v113'167 uv167 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f71a4009c20 con 0x55b859b12600 2024-08-03T20:58:22.550 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:22.548+0000 7f71b1af4f40 1 -- 172.21.15.32:0/3472799416 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:66 12.e 12:717a0223:::608.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113) -- 0x55b859bc2e30 con 0x55b859b12600 2024-08-03T20:58:22.551 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:22.550+0000 7f71ab600640 1 -- 172.21.15.32:0/3472799416 <== osd.0 v2:172.21.15.32:6816/3066156116 41 ==== osd_op_reply(66 608.00000000 [setxattr (30)] v113'168 uv168 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f71a4009c20 con 0x55b859b12600 2024-08-03T20:58:22.551 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:22.550+0000 7f71b1af4f40 1 -- 172.21.15.32:0/3472799416 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:67 12.e 12:717a0223:::608.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113) -- 0x55b859bc2e30 con 0x55b859b12600 2024-08-03T20:58:22.552 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:22.551+0000 7f71ab600640 1 -- 172.21.15.32:0/3472799416 <== osd.0 v2:172.21.15.32:6816/3066156116 42 ==== osd_op_reply(67 608.00000000 [setxattr (62)] v113'169 uv169 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f71a4009c20 con 0x55b859b12600 2024-08-03T20:58:22.552 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:22.551+0000 7f71b1af4f40 1 -- 172.21.15.32:0/3472799416 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:68 12.e 12:717a0223:::608.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113) -- 0x55b859bc2e30 con 0x55b859b12600 2024-08-03T20:58:22.554 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:22.553+0000 7f71ab600640 1 -- 172.21.15.32:0/3472799416 <== osd.0 v2:172.21.15.32:6816/3066156116 43 ==== osd_op_reply(68 608.00000000 [omap-set-header] v113'170 uv170 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f71a4009c20 con 0x55b859b12600 2024-08-03T20:58:22.554 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:22.553+0000 7f71b1af4f40 1 --2- 172.21.15.32:0/3472799416 >> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] conn(0x55b859bc3730 0x55b859bc5b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:58:22.554 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:22.553+0000 7f71b1af4f40 1 -- 172.21.15.32:0/3472799416 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:69 12.15 12:a93a17c2:::604.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113) -- 0x55b859bc6090 con 0x55b859bc3730 2024-08-03T20:58:22.555 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:22.553+0000 7f71ab600640 1 --2- 172.21.15.32:0/3472799416 >> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] conn(0x55b859bc3730 0x55b859bc5b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:58:22.555 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:22.554+0000 7f71ab600640 1 --2- 172.21.15.32:0/3472799416 >> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] conn(0x55b859bc3730 0x55b859bc5b50 crc :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:58:22.557 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:22.555+0000 7f71ab600640 1 -- 172.21.15.32:0/3472799416 <== osd.6 v2:172.21.15.118:6825/801947386 1 ==== osd_op_reply(69 604.00000000 [create] v113'305 uv299 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f71a4009c20 con 0x55b859bc3730 2024-08-03T20:58:22.557 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:22.555+0000 7f71b1af4f40 1 -- 172.21.15.32:0/3472799416 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:70 12.15 12:a93a17c2:::604.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e113) -- 0x55b859bc6090 con 0x55b859bc3730 2024-08-03T20:58:22.558 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:22.557+0000 7f71ab600640 1 -- 172.21.15.32:0/3472799416 <== osd.6 v2:172.21.15.118:6825/801947386 2 ==== osd_op_reply(70 604.00000000 [delete] v113'306 uv306 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f71a4009c20 con 0x55b859bc3730 2024-08-03T20:58:22.558 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:22.557+0000 7f71b1af4f40 1 -- 172.21.15.32:0/3472799416 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:71 12.15 12:a93a17c2:::604.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113) -- 0x55b859bc64b0 con 0x55b859bc3730 2024-08-03T20:58:22.559 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::604.00000000:head# 2024-08-03T20:58:22.560 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:22.558+0000 7f71ab600640 1 -- 172.21.15.32:0/3472799416 <== osd.6 v2:172.21.15.118:6825/801947386 3 ==== osd_op_reply(71 604.00000000 [create] v113'307 uv307 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f71a4009c20 con 0x55b859bc3730 2024-08-03T20:58:22.560 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:22.558+0000 7f71b1af4f40 1 -- 172.21.15.32:0/3472799416 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:72 12.15 12:a93a17c2:::604.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113) -- 0x55b859bc74c0 con 0x55b859bc3730 2024-08-03T20:58:22.561 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:22.559+0000 7f71ab600640 1 -- 172.21.15.32:0/3472799416 <== osd.6 v2:172.21.15.118:6825/801947386 4 ==== osd_op_reply(72 604.00000000 [setxattr (30)] v113'308 uv308 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f71a4009c20 con 0x55b859bc3730 2024-08-03T20:58:22.561 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:22.559+0000 7f71b1af4f40 1 -- 172.21.15.32:0/3472799416 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:73 12.15 12:a93a17c2:::604.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113) -- 0x55b859bc74c0 con 0x55b859bc3730 2024-08-03T20:58:22.562 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:22.561+0000 7f71ab600640 1 -- 172.21.15.32:0/3472799416 <== osd.6 v2:172.21.15.118:6825/801947386 5 ==== osd_op_reply(73 604.00000000 [setxattr (62)] v113'309 uv309 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f71a4009c20 con 0x55b859bc3730 2024-08-03T20:58:22.562 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:22.561+0000 7f71b1af4f40 1 -- 172.21.15.32:0/3472799416 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:74 12.15 12:a93a17c2:::604.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113) -- 0x55b859bc64b0 con 0x55b859bc3730 2024-08-03T20:58:22.564 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:22.562+0000 7f71ab600640 1 -- 172.21.15.32:0/3472799416 <== osd.6 v2:172.21.15.118:6825/801947386 6 ==== osd_op_reply(74 604.00000000 [omap-set-header] v113'310 uv310 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f71a4009c20 con 0x55b859bc3730 2024-08-03T20:58:22.564 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:22.562+0000 7f71b1af4f40 1 -- 172.21.15.32:0/3472799416 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:75 12.15 12:aa448500:::500.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113) -- 0x55b859bc64b0 con 0x55b859bc3730 2024-08-03T20:58:22.565 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:22.563+0000 7f71ab600640 1 -- 172.21.15.32:0/3472799416 <== osd.6 v2:172.21.15.118:6825/801947386 7 ==== osd_op_reply(75 500.00000000 [create] v113'311 uv304 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f71a4009c20 con 0x55b859bc3730 2024-08-03T20:58:22.565 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:22.563+0000 7f71b1af4f40 1 -- 172.21.15.32:0/3472799416 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:76 12.15 12:aa448500:::500.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e113) -- 0x55b859bc64b0 con 0x55b859bc3730 2024-08-03T20:58:22.566 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:22.565+0000 7f71ab600640 1 -- 172.21.15.32:0/3472799416 <== osd.6 v2:172.21.15.118:6825/801947386 8 ==== osd_op_reply(76 500.00000000 [delete] v113'312 uv312 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f71a4009c20 con 0x55b859bc3730 2024-08-03T20:58:22.566 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:22.565+0000 7f71b1af4f40 1 -- 172.21.15.32:0/3472799416 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:77 12.15 12:aa448500:::500.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113) -- 0x55b859bc64b0 con 0x55b859bc3730 2024-08-03T20:58:22.568 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::500.00000000:head# 2024-08-03T20:58:22.568 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:22.566+0000 7f71ab600640 1 -- 172.21.15.32:0/3472799416 <== osd.6 v2:172.21.15.118:6825/801947386 9 ==== osd_op_reply(77 500.00000000 [create] v113'313 uv313 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f71a4009c20 con 0x55b859bc3730 2024-08-03T20:58:22.568 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:22.566+0000 7f71b1af4f40 1 -- 172.21.15.32:0/3472799416 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:78 12.15 12:aa448500:::500.00000000:head [write 0~90 in=90b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113) -- 0x55b859bc64b0 con 0x55b859bc3730 2024-08-03T20:58:22.569 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:22.568+0000 7f71ab600640 1 -- 172.21.15.32:0/3472799416 <== osd.6 v2:172.21.15.118:6825/801947386 10 ==== osd_op_reply(78 500.00000000 [write 0~90] v113'314 uv314 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f71a4009c20 con 0x55b859bc3730 2024-08-03T20:58:22.569 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:22.568+0000 7f71b1af4f40 1 -- 172.21.15.32:0/3472799416 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:79 12.15 12:aa448500:::500.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113) -- 0x55b859bc64b0 con 0x55b859bc3730 2024-08-03T20:58:22.571 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:22.569+0000 7f71ab600640 1 -- 172.21.15.32:0/3472799416 <== osd.6 v2:172.21.15.118:6825/801947386 11 ==== osd_op_reply(79 500.00000000 [omap-set-header] v113'315 uv315 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f71a4009c20 con 0x55b859bc3730 2024-08-03T20:58:22.571 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:22.569+0000 7f71b1af4f40 1 --2- 172.21.15.32:0/3472799416 >> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] conn(0x55b859bc69c0 0x55b859bc8e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:58:22.571 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:22.569+0000 7f71b1af4f40 1 -- 172.21.15.32:0/3472799416 --> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] -- osd_op(unknown.0.0:80 12.d 12:b50e409b:::mds_snaptable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113) -- 0x55b859bc9520 con 0x55b859bc69c0 2024-08-03T20:58:22.571 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:22.569+0000 7f71aac00640 1 --2- 172.21.15.32:0/3472799416 >> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] conn(0x55b859bc69c0 0x55b859bc8e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:58:22.571 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:22.570+0000 7f71aac00640 1 --2- 172.21.15.32:0/3472799416 >> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] conn(0x55b859bc69c0 0x55b859bc8e20 crc :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:58:22.573 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:22.572+0000 7f71aac00640 1 -- 172.21.15.32:0/3472799416 <== osd.7 v2:172.21.15.118:6818/653000048 1 ==== osd_op_reply(80 mds_snaptable [create] v113'138 uv137 ondisk = -17 ((17) File exists)) ==== 157+0+0 (crc 0 0 0) 0x7f719c00c070 con 0x55b859bc69c0 2024-08-03T20:58:22.573 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:22.572+0000 7f71b1af4f40 1 -- 172.21.15.32:0/3472799416 --> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] -- osd_op(unknown.0.0:81 12.d 12:b50e409b:::mds_snaptable:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e113) -- 0x55b859bc98f0 con 0x55b859bc69c0 2024-08-03T20:58:22.574 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:22.573+0000 7f71aac00640 1 -- 172.21.15.32:0/3472799416 <== osd.7 v2:172.21.15.118:6818/653000048 2 ==== osd_op_reply(81 mds_snaptable [delete] v113'139 uv139 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f719c00c070 con 0x55b859bc69c0 2024-08-03T20:58:22.575 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:22.573+0000 7f71b1af4f40 1 -- 172.21.15.32:0/3472799416 --> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] -- osd_op(unknown.0.0:82 12.d 12:b50e409b:::mds_snaptable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113) -- 0x55b859bc98f0 con 0x55b859bc69c0 2024-08-03T20:58:22.576 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds_snaptable:head# 2024-08-03T20:58:22.577 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:22.575+0000 7f71aac00640 1 -- 172.21.15.32:0/3472799416 <== osd.7 v2:172.21.15.118:6818/653000048 3 ==== osd_op_reply(82 mds_snaptable [create] v113'140 uv140 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f719c00c070 con 0x55b859bc69c0 2024-08-03T20:58:22.577 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:22.575+0000 7f71b1af4f40 1 -- 172.21.15.32:0/3472799416 --> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] -- osd_op(unknown.0.0:83 12.d 12:b50e409b:::mds_snaptable:head [write 0~70 in=70b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113) -- 0x55b859bc98f0 con 0x55b859bc69c0 2024-08-03T20:58:22.578 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:22.576+0000 7f71aac00640 1 -- 172.21.15.32:0/3472799416 <== osd.7 v2:172.21.15.118:6818/653000048 4 ==== osd_op_reply(83 mds_snaptable [write 0~70] v113'141 uv141 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f719c00c070 con 0x55b859bc69c0 2024-08-03T20:58:22.578 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:22.576+0000 7f71b1af4f40 1 -- 172.21.15.32:0/3472799416 --> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] -- osd_op(unknown.0.0:84 12.d 12:b50e409b:::mds_snaptable:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113) -- 0x55b859bc98f0 con 0x55b859bc69c0 2024-08-03T20:58:22.579 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:22.578+0000 7f71aac00640 1 -- 172.21.15.32:0/3472799416 <== osd.7 v2:172.21.15.118:6818/653000048 5 ==== osd_op_reply(84 mds_snaptable [omap-set-header] v113'142 uv142 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f719c00c070 con 0x55b859bc69c0 2024-08-03T20:58:22.579 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:22.578+0000 7f71b1af4f40 1 -- 172.21.15.32:0/3472799416 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:85 12.1d 12:b871830b:::605.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113) -- 0x55b859bc98f0 con 0x55b859b12600 2024-08-03T20:58:22.581 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:22.579+0000 7f71ab600640 1 -- 172.21.15.32:0/3472799416 <== osd.0 v2:172.21.15.32:6816/3066156116 44 ==== osd_op_reply(85 605.00000000 [create] v113'165 uv164 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f71a4009c20 con 0x55b859b12600 2024-08-03T20:58:22.581 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:22.579+0000 7f71b1af4f40 1 -- 172.21.15.32:0/3472799416 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:86 12.1d 12:b871830b:::605.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e113) -- 0x55b859bc98f0 con 0x55b859b12600 2024-08-03T20:58:22.582 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:22.581+0000 7f71ab600640 1 -- 172.21.15.32:0/3472799416 <== osd.0 v2:172.21.15.32:6816/3066156116 45 ==== osd_op_reply(86 605.00000000 [delete] v113'166 uv166 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f71a4009c20 con 0x55b859b12600 2024-08-03T20:58:22.582 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:22.581+0000 7f71b1af4f40 1 -- 172.21.15.32:0/3472799416 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:87 12.1d 12:b871830b:::605.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113) -- 0x55b859bc98f0 con 0x55b859b12600 2024-08-03T20:58:22.583 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::605.00000000:head# 2024-08-03T20:58:22.584 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:22.582+0000 7f71ab600640 1 -- 172.21.15.32:0/3472799416 <== osd.0 v2:172.21.15.32:6816/3066156116 46 ==== osd_op_reply(87 605.00000000 [create] v113'167 uv167 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f71a4009c20 con 0x55b859b12600 2024-08-03T20:58:22.584 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:22.582+0000 7f71b1af4f40 1 -- 172.21.15.32:0/3472799416 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:88 12.1d 12:b871830b:::605.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113) -- 0x55b859bca900 con 0x55b859b12600 2024-08-03T20:58:22.585 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:22.583+0000 7f71ab600640 1 -- 172.21.15.32:0/3472799416 <== osd.0 v2:172.21.15.32:6816/3066156116 47 ==== osd_op_reply(88 605.00000000 [setxattr (30)] v113'168 uv168 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f71a4009c20 con 0x55b859b12600 2024-08-03T20:58:22.585 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:22.583+0000 7f71b1af4f40 1 -- 172.21.15.32:0/3472799416 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:89 12.1d 12:b871830b:::605.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113) -- 0x55b859bca900 con 0x55b859b12600 2024-08-03T20:58:22.586 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:22.585+0000 7f71ab600640 1 -- 172.21.15.32:0/3472799416 <== osd.0 v2:172.21.15.32:6816/3066156116 48 ==== osd_op_reply(89 605.00000000 [setxattr (62)] v113'169 uv169 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f71a4009c20 con 0x55b859b12600 2024-08-03T20:58:22.586 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:22.585+0000 7f71b1af4f40 1 -- 172.21.15.32:0/3472799416 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:90 12.1d 12:b871830b:::605.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113) -- 0x55b859bc98f0 con 0x55b859b12600 2024-08-03T20:58:22.587 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:22.586+0000 7f71ab600640 1 -- 172.21.15.32:0/3472799416 <== osd.0 v2:172.21.15.32:6816/3066156116 49 ==== osd_op_reply(90 605.00000000 [omap-set-header] v113'170 uv170 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f71a4009c20 con 0x55b859b12600 2024-08-03T20:58:22.587 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:22.586+0000 7f71b1af4f40 1 -- 172.21.15.32:0/3472799416 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:91 12.13 12:c91d4a1d:::mds0_inotable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113) -- 0x55b859bc98f0 con 0x55b859bbed00 2024-08-03T20:58:22.589 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:22.587+0000 7f71aa200640 1 -- 172.21.15.32:0/3472799416 <== osd.1 v2:172.21.15.32:6824/1551241759 5 ==== osd_op_reply(91 mds0_inotable [create] v113'273 uv267 ondisk = -17 ((17) File exists)) ==== 157+0+0 (crc 0 0 0) 0x7f71980a08f0 con 0x55b859bbed00 2024-08-03T20:58:22.589 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:22.587+0000 7f71b1af4f40 1 -- 172.21.15.32:0/3472799416 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:92 12.13 12:c91d4a1d:::mds0_inotable:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e113) -- 0x55b859bc9ce0 con 0x55b859bbed00 2024-08-03T20:58:22.590 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:22.589+0000 7f71aa200640 1 -- 172.21.15.32:0/3472799416 <== osd.1 v2:172.21.15.32:6824/1551241759 6 ==== osd_op_reply(92 mds0_inotable [delete] v113'274 uv274 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f71980a08f0 con 0x55b859bbed00 2024-08-03T20:58:22.590 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:22.589+0000 7f71b1af4f40 1 -- 172.21.15.32:0/3472799416 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:93 12.13 12:c91d4a1d:::mds0_inotable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113) -- 0x55b859bca0d0 con 0x55b859bbed00 2024-08-03T20:58:22.591 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds0_inotable:head# 2024-08-03T20:58:22.591 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:22.590+0000 7f71aa200640 1 -- 172.21.15.32:0/3472799416 <== osd.1 v2:172.21.15.32:6824/1551241759 7 ==== osd_op_reply(93 mds0_inotable [create] v113'275 uv275 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f71980a08f0 con 0x55b859bbed00 2024-08-03T20:58:22.592 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:22.590+0000 7f71b1af4f40 1 -- 172.21.15.32:0/3472799416 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:94 12.13 12:c91d4a1d:::mds0_inotable:head [write 0~34 in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113) -- 0x55b859bc29b0 con 0x55b859bbed00 2024-08-03T20:58:22.592 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:22.591+0000 7f71aa200640 1 -- 172.21.15.32:0/3472799416 <== osd.1 v2:172.21.15.32:6824/1551241759 8 ==== osd_op_reply(94 mds0_inotable [write 0~34] v113'276 uv276 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f71980a08f0 con 0x55b859bbed00 2024-08-03T20:58:22.593 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:22.591+0000 7f71b1af4f40 1 -- 172.21.15.32:0/3472799416 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:95 12.13 12:c91d4a1d:::mds0_inotable:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113) -- 0x55b859bc29b0 con 0x55b859bbed00 2024-08-03T20:58:22.594 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:22.592+0000 7f71aa200640 1 -- 172.21.15.32:0/3472799416 <== osd.1 v2:172.21.15.32:6824/1551241759 9 ==== osd_op_reply(95 mds0_inotable [omap-set-header] v113'277 uv277 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f71980a08f0 con 0x55b859bbed00 2024-08-03T20:58:22.594 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:22.593+0000 7f71b1af4f40 1 -- 172.21.15.32:0/3472799416 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:96 12.13 12:cd5a64a3:::100.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113) -- 0x55b859bc29b0 con 0x55b859bbed00 2024-08-03T20:58:22.595 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:22.594+0000 7f71aa200640 1 -- 172.21.15.32:0/3472799416 <== osd.1 v2:172.21.15.32:6824/1551241759 10 ==== osd_op_reply(96 100.00000000 [create] v113'278 uv272 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f71980a08f0 con 0x55b859bbed00 2024-08-03T20:58:22.595 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:22.594+0000 7f71b1af4f40 1 -- 172.21.15.32:0/3472799416 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:97 12.13 12:cd5a64a3:::100.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e113) -- 0x55b859bc29b0 con 0x55b859bbed00 2024-08-03T20:58:22.596 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:22.595+0000 7f71aa200640 1 -- 172.21.15.32:0/3472799416 <== osd.1 v2:172.21.15.32:6824/1551241759 11 ==== osd_op_reply(97 100.00000000 [delete] v113'279 uv279 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f71980a08f0 con 0x55b859bbed00 2024-08-03T20:58:22.597 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:22.595+0000 7f71b1af4f40 1 -- 172.21.15.32:0/3472799416 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:98 12.13 12:cd5a64a3:::100.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113) -- 0x55b859bc29b0 con 0x55b859bbed00 2024-08-03T20:58:22.598 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::100.00000000:head# 2024-08-03T20:58:22.598 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:22.596+0000 7f71aa200640 1 -- 172.21.15.32:0/3472799416 <== osd.1 v2:172.21.15.32:6824/1551241759 12 ==== osd_op_reply(98 100.00000000 [create] v113'280 uv280 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f71980a08f0 con 0x55b859bbed00 2024-08-03T20:58:22.598 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:22.596+0000 7f71b1af4f40 1 -- 172.21.15.32:0/3472799416 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:99 12.13 12:cd5a64a3:::100.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113) -- 0x55b859bc29b0 con 0x55b859bbed00 2024-08-03T20:58:22.599 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:22.598+0000 7f71aa200640 1 -- 172.21.15.32:0/3472799416 <== osd.1 v2:172.21.15.32:6824/1551241759 13 ==== osd_op_reply(99 100.00000000 [omap-set-header] v113'281 uv281 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f71980a08f0 con 0x55b859bbed00 2024-08-03T20:58:22.599 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:22.598+0000 7f71b1af4f40 1 -- 172.21.15.32:0/3472799416 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:100 12.13 12:cd5a64a3:::100.00000000:head [omap-set-vals in=5194b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113) -- 0x55b859bcd640 con 0x55b859bbed00 2024-08-03T20:58:22.601 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:22.599+0000 7f71aa200640 1 -- 172.21.15.32:0/3472799416 <== osd.1 v2:172.21.15.32:6824/1551241759 14 ==== osd_op_reply(100 100.00000000 [omap-set-vals] v113'282 uv282 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f71980a08f0 con 0x55b859bbed00 2024-08-03T20:58:22.601 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:22.599+0000 7f71b1af4f40 1 --2- 172.21.15.32:0/3472799416 >> [v2:172.21.15.32:6800/9088303,v1:172.21.15.32:6801/9088303] conn(0x55b859bcb1f0 0x55b859bcda60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:58:22.601 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:22.599+0000 7f71b1af4f40 1 -- 172.21.15.32:0/3472799416 --> [v2:172.21.15.32:6800/9088303,v1:172.21.15.32:6801/9088303] -- osd_op(unknown.0.0:101 12.b 12:d0630e4c:::mds0_sessionmap:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113) -- 0x55b859bcd610 con 0x55b859bcb1f0 2024-08-03T20:58:22.601 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:22.600+0000 7f71aa200640 1 --2- 172.21.15.32:0/3472799416 >> [v2:172.21.15.32:6800/9088303,v1:172.21.15.32:6801/9088303] conn(0x55b859bcb1f0 0x55b859bcda60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:58:22.601 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:22.600+0000 7f71aa200640 1 --2- 172.21.15.32:0/3472799416 >> [v2:172.21.15.32:6800/9088303,v1:172.21.15.32:6801/9088303] conn(0x55b859bcb1f0 0x55b859bcda60 crc :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:58:22.603 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:22.602+0000 7f71aa200640 1 -- 172.21.15.32:0/3472799416 <== osd.3 v2:172.21.15.32:6800/9088303 1 ==== osd_op_reply(101 mds0_sessionmap [create] v113'120 uv119 ondisk = -17 ((17) File exists)) ==== 159+0+0 (crc 0 0 0) 0x7f71980a08f0 con 0x55b859bcb1f0 2024-08-03T20:58:22.603 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:22.602+0000 7f71b1af4f40 1 -- 172.21.15.32:0/3472799416 --> [v2:172.21.15.32:6800/9088303,v1:172.21.15.32:6801/9088303] -- osd_op(unknown.0.0:102 12.b 12:d0630e4c:::mds0_sessionmap:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e113) -- 0x55b859bcd610 con 0x55b859bcb1f0 2024-08-03T20:58:22.604 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:22.603+0000 7f71aa200640 1 -- 172.21.15.32:0/3472799416 <== osd.3 v2:172.21.15.32:6800/9088303 2 ==== osd_op_reply(102 mds0_sessionmap [delete] v113'121 uv121 ondisk = 0) ==== 159+0+0 (crc 0 0 0) 0x7f71980a08f0 con 0x55b859bcb1f0 2024-08-03T20:58:22.604 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:22.603+0000 7f71b1af4f40 1 -- 172.21.15.32:0/3472799416 --> [v2:172.21.15.32:6800/9088303,v1:172.21.15.32:6801/9088303] -- osd_op(unknown.0.0:103 12.b 12:d0630e4c:::mds0_sessionmap:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113) -- 0x55b859bcd610 con 0x55b859bcb1f0 2024-08-03T20:58:22.605 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds0_sessionmap:head# 2024-08-03T20:58:22.606 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:22.604+0000 7f71aa200640 1 -- 172.21.15.32:0/3472799416 <== osd.3 v2:172.21.15.32:6800/9088303 3 ==== osd_op_reply(103 mds0_sessionmap [create] v113'122 uv122 ondisk = 0) ==== 159+0+0 (crc 0 0 0) 0x7f71980a08f0 con 0x55b859bcb1f0 2024-08-03T20:58:22.606 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:22.604+0000 7f71b1af4f40 1 -- 172.21.15.32:0/3472799416 --> [v2:172.21.15.32:6800/9088303,v1:172.21.15.32:6801/9088303] -- osd_op(unknown.0.0:104 12.b 12:d0630e4c:::mds0_sessionmap:head [omap-set-header in=14b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113) -- 0x55b859bcd610 con 0x55b859bcb1f0 2024-08-03T20:58:22.607 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:22.605+0000 7f71aa200640 1 -- 172.21.15.32:0/3472799416 <== osd.3 v2:172.21.15.32:6800/9088303 4 ==== osd_op_reply(104 mds0_sessionmap [omap-set-header] v113'123 uv123 ondisk = 0) ==== 159+0+0 (crc 0 0 0) 0x7f71980a08f0 con 0x55b859bcb1f0 2024-08-03T20:58:22.607 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:22.605+0000 7f71b1af4f40 1 -- 172.21.15.32:0/3472799416 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:105 12.7 12:e0b41d06:::609.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113) -- 0x55b859bcd610 con 0x55b859bbed00 2024-08-03T20:58:22.609 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:22.607+0000 7f71aa200640 1 -- 172.21.15.32:0/3472799416 <== osd.1 v2:172.21.15.32:6824/1551241759 15 ==== osd_op_reply(105 609.00000000 [create] v113'165 uv164 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f71980a08f0 con 0x55b859bbed00 2024-08-03T20:58:22.609 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:22.607+0000 7f71b1af4f40 1 -- 172.21.15.32:0/3472799416 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:106 12.7 12:e0b41d06:::609.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e113) -- 0x55b859bcd610 con 0x55b859bbed00 2024-08-03T20:58:22.610 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:22.609+0000 7f71aa200640 1 -- 172.21.15.32:0/3472799416 <== osd.1 v2:172.21.15.32:6824/1551241759 16 ==== osd_op_reply(106 609.00000000 [delete] v113'166 uv166 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f71980a08f0 con 0x55b859bbed00 2024-08-03T20:58:22.610 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:22.609+0000 7f71b1af4f40 1 -- 172.21.15.32:0/3472799416 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:107 12.7 12:e0b41d06:::609.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113) -- 0x55b859bcd610 con 0x55b859bbed00 2024-08-03T20:58:22.612 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::609.00000000:head# 2024-08-03T20:58:22.612 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:22.611+0000 7f71aa200640 1 -- 172.21.15.32:0/3472799416 <== osd.1 v2:172.21.15.32:6824/1551241759 17 ==== osd_op_reply(107 609.00000000 [create] v113'167 uv167 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f71980a08f0 con 0x55b859bbed00 2024-08-03T20:58:22.612 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:22.611+0000 7f71b1af4f40 1 -- 172.21.15.32:0/3472799416 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:108 12.7 12:e0b41d06:::609.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113) -- 0x55b859bcd610 con 0x55b859bbed00 2024-08-03T20:58:22.614 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:22.612+0000 7f71aa200640 1 -- 172.21.15.32:0/3472799416 <== osd.1 v2:172.21.15.32:6824/1551241759 18 ==== osd_op_reply(108 609.00000000 [setxattr (30)] v113'168 uv168 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f71980a08f0 con 0x55b859bbed00 2024-08-03T20:58:22.614 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:22.612+0000 7f71b1af4f40 1 -- 172.21.15.32:0/3472799416 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:109 12.7 12:e0b41d06:::609.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113) -- 0x55b859bcd610 con 0x55b859bbed00 2024-08-03T20:58:22.615 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:22.614+0000 7f71aa200640 1 -- 172.21.15.32:0/3472799416 <== osd.1 v2:172.21.15.32:6824/1551241759 19 ==== osd_op_reply(109 609.00000000 [setxattr (62)] v113'169 uv169 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f71980a08f0 con 0x55b859bbed00 2024-08-03T20:58:22.615 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:22.614+0000 7f71b1af4f40 1 -- 172.21.15.32:0/3472799416 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:110 12.7 12:e0b41d06:::609.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113) -- 0x55b859bcd610 con 0x55b859bbed00 2024-08-03T20:58:22.617 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:22.615+0000 7f71aa200640 1 -- 172.21.15.32:0/3472799416 <== osd.1 v2:172.21.15.32:6824/1551241759 20 ==== osd_op_reply(110 609.00000000 [omap-set-header] v113'170 uv170 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f71980a08f0 con 0x55b859bbed00 2024-08-03T20:58:22.617 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:22.615+0000 7f71b1af4f40 1 -- 172.21.15.32:0/3472799416 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:111 12.f 12:f1f69726:::400.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113) -- 0x55b859bcd610 con 0x55b859bc3730 2024-08-03T20:58:22.618 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:22.617+0000 7f71ab600640 1 -- 172.21.15.32:0/3472799416 <== osd.6 v2:172.21.15.118:6825/801947386 12 ==== osd_op_reply(111 400.00000000 [create] v113'140 uv139 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f71a4009c20 con 0x55b859bc3730 2024-08-03T20:58:22.618 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:22.617+0000 7f71b1af4f40 1 -- 172.21.15.32:0/3472799416 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:112 12.f 12:f1f69726:::400.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e113) -- 0x55b859bcd610 con 0x55b859bc3730 2024-08-03T20:58:22.620 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:22.619+0000 7f71ab600640 1 -- 172.21.15.32:0/3472799416 <== osd.6 v2:172.21.15.118:6825/801947386 13 ==== osd_op_reply(112 400.00000000 [delete] v113'141 uv141 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f71a4009c20 con 0x55b859bc3730 2024-08-03T20:58:22.620 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:22.619+0000 7f71b1af4f40 1 -- 172.21.15.32:0/3472799416 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:113 12.f 12:f1f69726:::400.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113) -- 0x55b859bcd610 con 0x55b859bc3730 2024-08-03T20:58:22.621 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:22.620+0000 7f71ab600640 1 -- 172.21.15.32:0/3472799416 <== osd.6 v2:172.21.15.118:6825/801947386 14 ==== osd_op_reply(113 400.00000000 [create] v113'142 uv142 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f71a4009c20 con 0x55b859bc3730 2024-08-03T20:58:22.621 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::400.00000000:head# 2024-08-03T20:58:22.622 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:22.620+0000 7f71b1af4f40 1 -- 172.21.15.32:0/3472799416 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:114 12.f 12:f1f69726:::400.00000000:head [write 0~22 in=22b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113) -- 0x55b859bcd610 con 0x55b859bc3730 2024-08-03T20:58:22.623 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:22.621+0000 7f71ab600640 1 -- 172.21.15.32:0/3472799416 <== osd.6 v2:172.21.15.118:6825/801947386 15 ==== osd_op_reply(114 400.00000000 [write 0~22] v113'143 uv143 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f71a4009c20 con 0x55b859bc3730 2024-08-03T20:58:22.623 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:22.621+0000 7f71b1af4f40 1 -- 172.21.15.32:0/3472799416 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:115 12.f 12:f1f69726:::400.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113) -- 0x55b859bcd610 con 0x55b859bc3730 2024-08-03T20:58:22.624 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:22.623+0000 7f71ab600640 1 -- 172.21.15.32:0/3472799416 <== osd.6 v2:172.21.15.118:6825/801947386 16 ==== osd_op_reply(115 400.00000000 [omap-set-header] v113'144 uv144 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f71a4009c20 con 0x55b859bc3730 2024-08-03T20:58:22.625 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:22.623+0000 7f71b1af4f40 1 -- 172.21.15.32:0/3472799416 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:116 12.1f 12:fe07bba1:::100.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113) -- 0x55b859bcd610 con 0x55b859bc3730 2024-08-03T20:58:22.626 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:22.625+0000 7f71ab600640 1 -- 172.21.15.32:0/3472799416 <== osd.6 v2:172.21.15.118:6825/801947386 17 ==== osd_op_reply(116 100.00000000.inode [create] v113'339 uv331 ondisk = -17 ((17) File exists)) ==== 162+0+0 (crc 0 0 0) 0x7f71a4009c20 con 0x55b859bc3730 2024-08-03T20:58:22.626 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:22.625+0000 7f71b1af4f40 1 -- 172.21.15.32:0/3472799416 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:117 12.1f 12:fe07bba1:::100.00000000.inode:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e113) -- 0x55b859bcd610 con 0x55b859bc3730 2024-08-03T20:58:22.627 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:22.626+0000 7f71ab600640 1 -- 172.21.15.32:0/3472799416 <== osd.6 v2:172.21.15.118:6825/801947386 18 ==== osd_op_reply(117 100.00000000.inode [delete] v113'340 uv340 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7f71a4009c20 con 0x55b859bc3730 2024-08-03T20:58:22.628 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:22.626+0000 7f71b1af4f40 1 -- 172.21.15.32:0/3472799416 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:118 12.1f 12:fe07bba1:::100.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113) -- 0x55b859bcd610 con 0x55b859bc3730 2024-08-03T20:58:22.629 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:22.627+0000 7f71ab600640 1 -- 172.21.15.32:0/3472799416 <== osd.6 v2:172.21.15.118:6825/801947386 19 ==== osd_op_reply(118 100.00000000.inode [create] v113'341 uv341 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7f71a4009c20 con 0x55b859bc3730 2024-08-03T20:58:22.629 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::100.00000000.inode:head# 2024-08-03T20:58:22.630 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:22.627+0000 7f71b1af4f40 1 -- 172.21.15.32:0/3472799416 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:119 12.1f 12:fe07bba1:::100.00000000.inode:head [write 0~582 in=582b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113) -- 0x55b859bcdff0 con 0x55b859bc3730 2024-08-03T20:58:22.631 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:22.629+0000 7f71ab600640 1 -- 172.21.15.32:0/3472799416 <== osd.6 v2:172.21.15.118:6825/801947386 20 ==== osd_op_reply(119 100.00000000.inode [write 0~582] v113'342 uv342 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7f71a4009c20 con 0x55b859bc3730 2024-08-03T20:58:22.631 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:22.629+0000 7f71b1af4f40 1 -- 172.21.15.32:0/3472799416 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:120 12.1f 12:fe07bba1:::100.00000000.inode:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113) -- 0x55b859bcd630 con 0x55b859bc3730 2024-08-03T20:58:22.632 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:22.631+0000 7f71ab600640 1 -- 172.21.15.32:0/3472799416 <== osd.6 v2:172.21.15.118:6825/801947386 21 ==== osd_op_reply(120 100.00000000.inode [omap-set-header] v113'343 uv343 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7f71a4009c20 con 0x55b859bc3730 2024-08-03T20:58:22.632 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:22.631+0000 7f71b1af4f40 1 -- 172.21.15.32:0/3472799416 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:121 12.1f 12:ff5b34d6:::1.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113) -- 0x55b859bcdff0 con 0x55b859bc3730 2024-08-03T20:58:22.633 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:22.632+0000 7f71ab600640 1 -- 172.21.15.32:0/3472799416 <== osd.6 v2:172.21.15.118:6825/801947386 22 ==== osd_op_reply(121 1.00000000 [create] v113'344 uv338 ondisk = -17 ((17) File exists)) ==== 154+0+0 (crc 0 0 0) 0x7f71a4009c20 con 0x55b859bc3730 2024-08-03T20:58:22.634 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:22.632+0000 7f71b1af4f40 1 -- 172.21.15.32:0/3472799416 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:122 12.1f 12:ff5b34d6:::1.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e113) -- 0x55b859bcdff0 con 0x55b859bc3730 2024-08-03T20:58:22.635 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:22.633+0000 7f71ab600640 1 -- 172.21.15.32:0/3472799416 <== osd.6 v2:172.21.15.118:6825/801947386 23 ==== osd_op_reply(122 1.00000000 [delete] v113'345 uv345 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7f71a4009c20 con 0x55b859bc3730 2024-08-03T20:58:22.635 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:22.634+0000 7f71b1af4f40 1 -- 172.21.15.32:0/3472799416 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:123 12.1f 12:ff5b34d6:::1.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113) -- 0x55b859bcdfa0 con 0x55b859bc3730 2024-08-03T20:58:22.636 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:22.635+0000 7f71ab600640 1 -- 172.21.15.32:0/3472799416 <== osd.6 v2:172.21.15.118:6825/801947386 24 ==== osd_op_reply(123 1.00000000 [create] v113'346 uv346 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7f71a4009c20 con 0x55b859bc3730 2024-08-03T20:58:22.637 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::1.00000000:head# 2024-08-03T20:58:22.637 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:22.635+0000 7f71b1af4f40 1 -- 172.21.15.32:0/3472799416 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:124 12.1f 12:ff5b34d6:::1.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113) -- 0x55b859bcefb0 con 0x55b859bc3730 2024-08-03T20:58:22.638 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:22.636+0000 7f71ab600640 1 -- 172.21.15.32:0/3472799416 <== osd.6 v2:172.21.15.118:6825/801947386 25 ==== osd_op_reply(124 1.00000000 [setxattr (30)] v113'347 uv347 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7f71a4009c20 con 0x55b859bc3730 2024-08-03T20:58:22.638 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:22.636+0000 7f71b1af4f40 1 -- 172.21.15.32:0/3472799416 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:125 12.1f 12:ff5b34d6:::1.00000000:head [setxattr parent (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113) -- 0x55b859bcefb0 con 0x55b859bc3730 2024-08-03T20:58:22.639 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:22.638+0000 7f71ab600640 1 -- 172.21.15.32:0/3472799416 <== osd.6 v2:172.21.15.118:6825/801947386 26 ==== osd_op_reply(125 1.00000000 [setxattr (30)] v113'348 uv348 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7f71a4009c20 con 0x55b859bc3730 2024-08-03T20:58:22.639 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:22.638+0000 7f71b1af4f40 1 -- 172.21.15.32:0/3472799416 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:126 12.1f 12:ff5b34d6:::1.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113) -- 0x55b859bcdfa0 con 0x55b859bc3730 2024-08-03T20:58:22.641 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:22.640+0000 7f71ab600640 1 -- 172.21.15.32:0/3472799416 <== osd.6 v2:172.21.15.118:6825/801947386 27 ==== osd_op_reply(126 1.00000000 [omap-set-header] v113'349 uv349 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7f71a4009c20 con 0x55b859bc3730 2024-08-03T20:58:22.641 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:22.640+0000 7f71b1af4f40 1 -- 172.21.15.32:0/3472799416 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:127 12.1f 12:ff5b34d6:::1.00000000:head [omap-set-vals in=584b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113) -- 0x55b859bce700 con 0x55b859bc3730 2024-08-03T20:58:22.644 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:22.641+0000 7f71ab600640 1 -- 172.21.15.32:0/3472799416 <== osd.6 v2:172.21.15.118:6825/801947386 28 ==== osd_op_reply(127 1.00000000 [omap-set-vals] v113'350 uv350 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7f71a4009c20 con 0x55b859bc3730 2024-08-03T20:58:22.644 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:22.641+0000 7f71b1af4f40 1 -- 172.21.15.32:0/3472799416 >> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] conn(0x55b859b12600 msgr2=0x55b859bb7f20 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:58:22.644 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:22.641+0000 7f71b1af4f40 1 --2- 172.21.15.32:0/3472799416 >> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] conn(0x55b859b12600 0x55b859bb7f20 crc :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:22.644 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:22.642+0000 7f71b1af4f40 1 -- 172.21.15.32:0/3472799416 >> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] conn(0x55b859bbed00 msgr2=0x55b859bbf110 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:58:22.644 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:22.642+0000 7f71b1af4f40 1 --2- 172.21.15.32:0/3472799416 >> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] conn(0x55b859bbed00 0x55b859bbf110 crc :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:22.644 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:22.642+0000 7f71b1af4f40 1 -- 172.21.15.32:0/3472799416 >> [v2:172.21.15.32:6800/9088303,v1:172.21.15.32:6801/9088303] conn(0x55b859bcb1f0 msgr2=0x55b859bcda60 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:58:22.644 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:22.642+0000 7f71b1af4f40 1 --2- 172.21.15.32:0/3472799416 >> [v2:172.21.15.32:6800/9088303,v1:172.21.15.32:6801/9088303] conn(0x55b859bcb1f0 0x55b859bcda60 crc :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:22.644 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:22.642+0000 7f71b1af4f40 1 -- 172.21.15.32:0/3472799416 >> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] conn(0x55b859bba460 msgr2=0x55b859bbc880 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:58:22.645 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:22.642+0000 7f71b1af4f40 1 --2- 172.21.15.32:0/3472799416 >> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] conn(0x55b859bba460 0x55b859bbc880 crc :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:22.645 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:22.642+0000 7f71b1af4f40 1 -- 172.21.15.32:0/3472799416 >> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] conn(0x55b859bc3730 msgr2=0x55b859bc5b50 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:58:22.645 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:22.642+0000 7f71b1af4f40 1 --2- 172.21.15.32:0/3472799416 >> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] conn(0x55b859bc3730 0x55b859bc5b50 crc :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:22.645 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:22.642+0000 7f71b1af4f40 1 -- 172.21.15.32:0/3472799416 >> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] conn(0x55b859bc69c0 msgr2=0x55b859bc8e20 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:58:22.645 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:22.642+0000 7f71b1af4f40 1 --2- 172.21.15.32:0/3472799416 >> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] conn(0x55b859bc69c0 0x55b859bc8e20 crc :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:22.645 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:22.642+0000 7f71ab600640 1 -- 172.21.15.32:0/3472799416 reap_dead start 2024-08-03T20:58:22.645 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:22.642+0000 7f71ab600640 1 -- 172.21.15.32:0/3472799416 reap_dead start 2024-08-03T20:58:22.646 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:22.642+0000 7f71ab600640 1 -- 172.21.15.32:0/3472799416 reap_dead start 2024-08-03T20:58:22.646 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:22.642+0000 7f71b1af4f40 1 -- 172.21.15.32:0/3472799416 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f718c0759f0 msgr2=0x7f718c077f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:58:22.646 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:22.642+0000 7f71b1af4f40 1 --2- 172.21.15.32:0/3472799416 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f718c0759f0 0x7f718c077f00 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x55b859a2a7a0 tx=0x7f719c004880 comp rx=0 tx=0).stop 2024-08-03T20:58:22.646 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:22.642+0000 7f71b1af4f40 1 -- 172.21.15.32:0/3472799416 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b859b12180 msgr2=0x55b859baed90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:58:22.646 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:22.642+0000 7f71b1af4f40 1 --2- 172.21.15.32:0/3472799416 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b859b12180 0x55b859baed90 secure :-1 s=READY pgs=1753 cs=0 l=1 rev1=1 crypto rx=0x7f7198009aa0 tx=0x7f719804de30 comp rx=0 tx=0).stop 2024-08-03T20:58:22.646 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:22.642+0000 7f71b1af4f40 1 -- 172.21.15.32:0/3472799416 shutdown_connections 2024-08-03T20:58:22.646 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:22.642+0000 7f71b1af4f40 1 --2- 172.21.15.32:0/3472799416 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f718c0759f0 0x7f718c077f00 unknown :-1 s=CLOSED pgs=713 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:22.647 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:22.642+0000 7f71b1af4f40 1 --2- 172.21.15.32:0/3472799416 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b859b12180 0x55b859baed90 unknown :-1 s=CLOSED pgs=1753 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:22.647 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:22.642+0000 7f71b1af4f40 1 -- 172.21.15.32:0/3472799416 >> 172.21.15.32:0/3472799416 conn(0x55b859960ac0 msgr2=0x55b859b13000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:58:22.647 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:22.642+0000 7f71b1af4f40 1 -- 172.21.15.32:0/3472799416 shutdown_connections 2024-08-03T20:58:22.647 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:22.642+0000 7f71b1af4f40 1 -- 172.21.15.32:0/3472799416 wait complete. 2024-08-03T20:58:22.655 DEBUG:teuthology.orchestra.run.smithi032:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata truncate mds0_inotable 0 2024-08-03T20:58:22.799 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:22.797+0000 7f86cad06f40 1 Processor -- start 2024-08-03T20:58:22.799 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:22.797+0000 7f86cad06f40 1 -- start start 2024-08-03T20:58:22.801 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:22.799+0000 7f86cad06f40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56089d12f360 0x56089d148a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:58:22.801 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:22.799+0000 7f86cad06f40 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56089d130ea0 0x56089d0c0030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:58:22.801 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:22.799+0000 7f86cad06f40 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x56089d146330 0x56089d0c0570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:58:22.801 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:22.799+0000 7f86cad06f40 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x56089d024bf0 con 0x56089d130ea0 2024-08-03T20:58:22.801 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:22.799+0000 7f86cad06f40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x56089d0316b0 con 0x56089d12f360 2024-08-03T20:58:22.801 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:22.799+0000 7f86cad06f40 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x56089cfc6e30 con 0x56089d146330 2024-08-03T20:58:22.801 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:22.799+0000 7f86c3400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56089d130ea0 0x56089d0c0030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:58:22.801 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:22.799+0000 7f86c3400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56089d130ea0 0x56089d0c0030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:42600/0 (socket says 172.21.15.32:42600) 2024-08-03T20:58:22.801 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:22.799+0000 7f86c3400640 1 -- 172.21.15.32:0/1258059489 learned_addr learned my addr 172.21.15.32:0/1258059489 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:58:22.801 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:22.799+0000 7f86c4800640 1 --2- 172.21.15.32:0/1258059489 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x56089d146330 0x56089d0c0570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:58:22.802 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:22.799+0000 7f86c3e00640 1 --2- 172.21.15.32:0/1258059489 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56089d12f360 0x56089d148a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:58:22.802 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:22.799+0000 7f86c3400640 1 -- 172.21.15.32:0/1258059489 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x56089d146330 msgr2=0x56089d0c0570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:58:22.802 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:22.799+0000 7f86c3400640 1 --2- 172.21.15.32:0/1258059489 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x56089d146330 0x56089d0c0570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:22.802 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:22.799+0000 7f86c3400640 1 -- 172.21.15.32:0/1258059489 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56089d12f360 msgr2=0x56089d148a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:58:22.802 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:22.799+0000 7f86c3400640 1 --2- 172.21.15.32:0/1258059489 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56089d12f360 0x56089d148a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:22.802 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:22.799+0000 7f86c3400640 1 -- 172.21.15.32:0/1258059489 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56089d14b4f0 con 0x56089d130ea0 2024-08-03T20:58:22.803 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:22.800+0000 7f86c3400640 1 --2- 172.21.15.32:0/1258059489 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56089d130ea0 0x56089d0c0030 secure :-1 s=READY pgs=1756 cs=0 l=1 rev1=1 crypto rx=0x7f86b0025f60 tx=0x7f86b0002fb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=17d81fb8fcb53096 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:58:22.804 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:22.800+0000 7f86c2000640 1 -- 172.21.15.32:0/1258059489 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f86b0056670 con 0x56089d130ea0 2024-08-03T20:58:22.804 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:22.800+0000 7f86c2000640 1 -- 172.21.15.32:0/1258059489 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f86b0056830 con 0x56089d130ea0 2024-08-03T20:58:22.804 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:22.800+0000 7f86c2000640 1 -- 172.21.15.32:0/1258059489 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f86b0056c80 con 0x56089d130ea0 2024-08-03T20:58:22.804 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:22.801+0000 7f86cad06f40 1 -- 172.21.15.32:0/1258059489 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56089d130ea0 msgr2=0x56089d0c0030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:58:22.804 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:22.801+0000 7f86cad06f40 1 --2- 172.21.15.32:0/1258059489 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56089d130ea0 0x56089d0c0030 secure :-1 s=READY pgs=1756 cs=0 l=1 rev1=1 crypto rx=0x7f86b0025f60 tx=0x7f86b0002fb0 comp rx=0 tx=0).stop 2024-08-03T20:58:22.804 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:22.801+0000 7f86cad06f40 1 -- 172.21.15.32:0/1258059489 shutdown_connections 2024-08-03T20:58:22.804 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:22.801+0000 7f86cad06f40 1 --2- 172.21.15.32:0/1258059489 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x56089d146330 0x56089d0c0570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:22.804 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:22.801+0000 7f86cad06f40 1 --2- 172.21.15.32:0/1258059489 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56089d130ea0 0x56089d0c0030 unknown :-1 s=CLOSED pgs=1756 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:22.804 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:22.801+0000 7f86cad06f40 1 --2- 172.21.15.32:0/1258059489 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56089d12f360 0x56089d148a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:22.804 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:22.801+0000 7f86cad06f40 1 -- 172.21.15.32:0/1258059489 >> 172.21.15.32:0/1258059489 conn(0x56089cf98ac0 msgr2=0x56089d1491e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:58:22.805 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:22.801+0000 7f86cad06f40 1 -- 172.21.15.32:0/1258059489 shutdown_connections 2024-08-03T20:58:22.805 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:22.801+0000 7f86cad06f40 1 -- 172.21.15.32:0/1258059489 wait complete. 2024-08-03T20:58:22.805 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:22.802+0000 7f86cad06f40 1 Processor -- start 2024-08-03T20:58:22.805 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:22.802+0000 7f86cad06f40 1 -- start start 2024-08-03T20:58:22.807 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:22.803+0000 7f86cad06f40 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x56089d12f360 0x56089d0be780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:58:22.807 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:22.803+0000 7f86cad06f40 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56089d130ea0 0x56089d12b050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:58:22.807 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:22.803+0000 7f86cad06f40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56089d146330 0x56089d147990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:58:22.807 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:22.803+0000 7f86cad06f40 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x56089d148590 con 0x56089d130ea0 2024-08-03T20:58:22.807 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:22.803+0000 7f86cad06f40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x56089d0becc0 con 0x56089d146330 2024-08-03T20:58:22.807 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:22.803+0000 7f86cad06f40 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x56089d0b7ca0 con 0x56089d12f360 2024-08-03T20:58:22.807 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:22.804+0000 7f86c3400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56089d130ea0 0x56089d12b050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:58:22.807 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:22.804+0000 7f86c3400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56089d130ea0 0x56089d12b050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:42610/0 (socket says 172.21.15.32:42610) 2024-08-03T20:58:22.807 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:22.804+0000 7f86c3400640 1 -- 172.21.15.32:0/2180024940 learned_addr learned my addr 172.21.15.32:0/2180024940 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:58:22.807 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:22.804+0000 7f86c3e00640 1 --2- 172.21.15.32:0/2180024940 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x56089d12f360 0x56089d0be780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:58:22.807 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:22.804+0000 7f86c4800640 1 --2- 172.21.15.32:0/2180024940 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56089d146330 0x56089d147990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:58:22.808 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:22.804+0000 7f86c3400640 1 -- 172.21.15.32:0/2180024940 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x56089d12f360 msgr2=0x56089d0be780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:58:22.808 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:22.804+0000 7f86c3400640 1 --2- 172.21.15.32:0/2180024940 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x56089d12f360 0x56089d0be780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:22.808 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:22.804+0000 7f86c3400640 1 -- 172.21.15.32:0/2180024940 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56089d146330 msgr2=0x56089d147990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:58:22.808 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:22.804+0000 7f86c3400640 1 --2- 172.21.15.32:0/2180024940 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56089d146330 0x56089d147990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:22.808 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:22.804+0000 7f86c3400640 1 -- 172.21.15.32:0/2180024940 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56089d0ba370 con 0x56089d130ea0 2024-08-03T20:58:22.808 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:22.804+0000 7f86c3e00640 1 --2- 172.21.15.32:0/2180024940 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x56089d12f360 0x56089d0be780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-03T20:58:22.808 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:22.805+0000 7f86c3400640 1 --2- 172.21.15.32:0/2180024940 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56089d130ea0 0x56089d12b050 secure :-1 s=READY pgs=1757 cs=0 l=1 rev1=1 crypto rx=0x7f86b0004f40 tx=0x7f86b0057590 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:58:22.808 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:22.805+0000 7f86c0c00640 1 -- 172.21.15.32:0/2180024940 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f86b0057840 con 0x56089d130ea0 2024-08-03T20:58:22.808 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:22.805+0000 7f86c0c00640 1 -- 172.21.15.32:0/2180024940 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f86b0070070 con 0x56089d130ea0 2024-08-03T20:58:22.808 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:22.805+0000 7f86cad06f40 1 -- 172.21.15.32:0/2180024940 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x56089d0bdce0 con 0x56089d130ea0 2024-08-03T20:58:22.808 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:22.805+0000 7f86c0c00640 1 -- 172.21.15.32:0/2180024940 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f86b0057db0 con 0x56089d130ea0 2024-08-03T20:58:22.808 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:22.805+0000 7f86cad06f40 1 -- 172.21.15.32:0/2180024940 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56089d0ba090 con 0x56089d130ea0 2024-08-03T20:58:22.809 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:22.807+0000 7f86c0c00640 1 -- 172.21.15.32:0/2180024940 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f86b006a070 con 0x56089d130ea0 2024-08-03T20:58:22.809 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:22.808+0000 7f86c0c00640 1 --2- 172.21.15.32:0/2180024940 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f86a40759f0 0x7f86a4077f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:58:22.809 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:22.808+0000 7f86c0c00640 1 -- 172.21.15.32:0/2180024940 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(113..113 src has 1..113) ==== 8262+0+0 (secure 0 0 0) 0x7f86b00dd830 con 0x56089d130ea0 2024-08-03T20:58:22.810 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:22.809+0000 7f86cad06f40 1 --2- 172.21.15.32:0/2180024940 >> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] conn(0x56089d0b74d0 0x56089d14e530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:58:22.810 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:22.809+0000 7f86cad06f40 1 -- 172.21.15.32:0/2180024940 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:1 12.13 12:c91d4a1d:::mds0_inotable:head [truncate 0] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113) -- 0x56089d1e6640 con 0x56089d0b74d0 2024-08-03T20:58:22.810 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:22.809+0000 7f86c3e00640 1 --2- 172.21.15.32:0/2180024940 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f86a40759f0 0x7f86a4077f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:58:22.811 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:22.809+0000 7f86c4800640 1 --2- 172.21.15.32:0/2180024940 >> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] conn(0x56089d0b74d0 0x56089d14e530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:58:22.811 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:22.809+0000 7f86c4800640 1 --2- 172.21.15.32:0/2180024940 >> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] conn(0x56089d0b74d0 0x56089d14e530 crc :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:58:22.811 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:22.810+0000 7f86c3e00640 1 --2- 172.21.15.32:0/2180024940 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f86a40759f0 0x7f86a4077f00 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7f86b4000c00 tx=0x7f86b40016d0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:58:22.814 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:22.811+0000 7f86c4800640 1 -- 172.21.15.32:0/2180024940 <== osd.1 v2:172.21.15.32:6824/1551241759 1 ==== osd_op_reply(1 mds0_inotable [truncate 0] v113'283 uv283 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f86bc0025f0 con 0x56089d0b74d0 2024-08-03T20:58:22.814 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:22.811+0000 7f86cad06f40 1 -- 172.21.15.32:0/2180024940 >> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] conn(0x56089d0b74d0 msgr2=0x56089d14e530 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:58:22.814 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:22.811+0000 7f86cad06f40 1 --2- 172.21.15.32:0/2180024940 >> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] conn(0x56089d0b74d0 0x56089d14e530 crc :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:22.814 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:22.811+0000 7f86cad06f40 1 -- 172.21.15.32:0/2180024940 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f86a40759f0 msgr2=0x7f86a4077f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:58:22.815 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:22.811+0000 7f86cad06f40 1 --2- 172.21.15.32:0/2180024940 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f86a40759f0 0x7f86a4077f00 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7f86b4000c00 tx=0x7f86b40016d0 comp rx=0 tx=0).stop 2024-08-03T20:58:22.815 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:22.811+0000 7f86cad06f40 1 -- 172.21.15.32:0/2180024940 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56089d130ea0 msgr2=0x56089d12b050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:58:22.815 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:22.811+0000 7f86cad06f40 1 --2- 172.21.15.32:0/2180024940 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56089d130ea0 0x56089d12b050 secure :-1 s=READY pgs=1757 cs=0 l=1 rev1=1 crypto rx=0x7f86b0004f40 tx=0x7f86b0057590 comp rx=0 tx=0).stop 2024-08-03T20:58:22.815 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:22.811+0000 7f86c4800640 1 -- 172.21.15.32:0/2180024940 reap_dead start 2024-08-03T20:58:22.815 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:22.811+0000 7f86cad06f40 1 -- 172.21.15.32:0/2180024940 shutdown_connections 2024-08-03T20:58:22.815 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:22.811+0000 7f86cad06f40 1 -- 172.21.15.32:0/2180024940 >> 172.21.15.32:0/2180024940 conn(0x56089cf98ac0 msgr2=0x56089d0c0bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:58:22.816 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:22.811+0000 7f86cad06f40 1 -- 172.21.15.32:0/2180024940 shutdown_connections 2024-08-03T20:58:22.816 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:22.811+0000 7f86cad06f40 1 -- 172.21.15.32:0/2180024940 wait complete. 2024-08-03T20:58:22.824 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs joinable true 2024-08-03T20:58:22.995 INFO:tasks.ceph.mds.a.smithi032.stderr: -51> 2024-08-03T20:58:18.241+0000 7efd68000640 -1 log_channel(cluster) log [ERR] : error decoding table object 'mds_snaptable': End of buffer [buffer:2] 2024-08-03T20:58:23.058 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:23.056+0000 7f9494800640 1 -- 172.21.15.32:0/2214471439 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f948c0cf3b0 msgr2=0x7f948c0cf7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:58:23.059 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:23.056+0000 7f9494800640 1 --2- 172.21.15.32:0/2214471439 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f948c0cf3b0 0x7f948c0cf7c0 secure :-1 s=READY pgs=1758 cs=0 l=1 rev1=1 crypto rx=0x7f948c0afaf0 tx=0x7f947c02a560 comp rx=0 tx=0).stop 2024-08-03T20:58:23.059 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:23.056+0000 7f9494800640 1 -- 172.21.15.32:0/2214471439 shutdown_connections 2024-08-03T20:58:23.059 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:23.056+0000 7f9494800640 1 --2- 172.21.15.32:0/2214471439 >> v2:172.21.15.118:3300/0 conn(0x7f948c15ba50 0x7f948c15be60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:23.059 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:23.056+0000 7f9494800640 1 --2- 172.21.15.32:0/2214471439 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f948c0cf3b0 0x7f948c0cf7c0 unknown :-1 s=CLOSED pgs=1758 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:23.059 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:23.056+0000 7f9494800640 1 --2- 172.21.15.32:0/2214471439 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f948c155920 0x7f948c0ceb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:23.060 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:23.056+0000 7f9494800640 1 -- 172.21.15.32:0/2214471439 >> 172.21.15.32:0/2214471439 conn(0x7f948c0b1bd0 msgr2=0x7f948c0b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:58:23.060 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:23.056+0000 7f9494800640 1 -- 172.21.15.32:0/2214471439 shutdown_connections 2024-08-03T20:58:23.060 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:23.056+0000 7f9494800640 1 -- 172.21.15.32:0/2214471439 wait complete. 2024-08-03T20:58:23.061 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:23.058+0000 7f9494800640 1 Processor -- start 2024-08-03T20:58:23.061 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:23.059+0000 7f9494800640 1 -- start start 2024-08-03T20:58:23.064 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:23.060+0000 7f9494800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f948c0cf3b0 0x7f948c0ca2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:58:23.064 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:23.060+0000 7f9494800640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f948c155920 0x7f948c1f5070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:58:23.064 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:23.060+0000 7f9494800640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f948c15ba50 0x7f948c0b12b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:58:23.064 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:23.060+0000 7f9494800640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f948c0b3260 con 0x7f948c0cf3b0 2024-08-03T20:58:23.064 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:23.060+0000 7f9494800640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f948c0b30e0 con 0x7f948c15ba50 2024-08-03T20:58:23.064 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:23.060+0000 7f9494800640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7f948c0b33e0 con 0x7f948c155920 2024-08-03T20:58:23.065 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:23.060+0000 7f9491e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f948c0cf3b0 0x7f948c0ca2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:58:23.065 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:23.060+0000 7f9491e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f948c0cf3b0 0x7f948c0ca2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:42624/0 (socket says 172.21.15.32:42624) 2024-08-03T20:58:23.065 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:23.060+0000 7f9491e00640 1 -- 172.21.15.32:0/1339303819 learned_addr learned my addr 172.21.15.32:0/1339303819 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:58:23.065 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:23.060+0000 7f9491400640 1 --2- 172.21.15.32:0/1339303819 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f948c155920 0x7f948c1f5070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:58:23.065 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:23.060+0000 7f9492800640 1 --2- 172.21.15.32:0/1339303819 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f948c15ba50 0x7f948c0b12b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:58:23.065 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:23.061+0000 7f9491e00640 1 -- 172.21.15.32:0/1339303819 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f948c155920 msgr2=0x7f948c1f5070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:58:23.065 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:23.061+0000 7f9491e00640 1 --2- 172.21.15.32:0/1339303819 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f948c155920 0x7f948c1f5070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:23.065 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:23.061+0000 7f9491e00640 1 -- 172.21.15.32:0/1339303819 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f948c15ba50 msgr2=0x7f948c0b12b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:58:23.065 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:23.061+0000 7f9491e00640 1 --2- 172.21.15.32:0/1339303819 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f948c15ba50 0x7f948c0b12b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:23.065 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:23.061+0000 7f9491e00640 1 -- 172.21.15.32:0/1339303819 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f948c0cbb90 con 0x7f948c0cf3b0 2024-08-03T20:58:23.065 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:23.061+0000 7f9491e00640 1 --2- 172.21.15.32:0/1339303819 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f948c0cf3b0 0x7f948c0ca2a0 secure :-1 s=READY pgs=1759 cs=0 l=1 rev1=1 crypto rx=0x7f948000d650 tx=0x7f948000db10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:58:23.065 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:23.061+0000 7f9486000640 1 -- 172.21.15.32:0/1339303819 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9480017070 con 0x7f948c0cf3b0 2024-08-03T20:58:23.065 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:23.061+0000 7f9486000640 1 -- 172.21.15.32:0/1339303819 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f9480004310 con 0x7f948c0cf3b0 2024-08-03T20:58:23.066 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:23.061+0000 7f9486000640 1 -- 172.21.15.32:0/1339303819 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9480005280 con 0x7f948c0cf3b0 2024-08-03T20:58:23.066 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:23.061+0000 7f9494800640 1 -- 172.21.15.32:0/1339303819 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f948c0caaf0 con 0x7f948c0cf3b0 2024-08-03T20:58:23.066 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:23.062+0000 7f9494800640 1 -- 172.21.15.32:0/1339303819 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f948c0cb0b0 con 0x7f948c0cf3b0 2024-08-03T20:58:23.066 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:23.063+0000 7f9494800640 1 -- 172.21.15.32:0/1339303819 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f948c0b3900 con 0x7f948c0cf3b0 2024-08-03T20:58:23.067 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:23.064+0000 7f9486000640 1 -- 172.21.15.32:0/1339303819 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f9480005420 con 0x7f948c0cf3b0 2024-08-03T20:58:23.067 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:23.065+0000 7f9486000640 1 --2- 172.21.15.32:0/1339303819 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f9470077040 0x7f9470079550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:58:23.067 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:23.065+0000 7f9486000640 1 -- 172.21.15.32:0/1339303819 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(113..113 src has 1..113) ==== 8262+0+0 (secure 0 0 0) 0x7f94800994c0 con 0x7f948c0cf3b0 2024-08-03T20:58:23.067 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:23.065+0000 7f9491400640 1 --2- 172.21.15.32:0/1339303819 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f9470077040 0x7f9470079550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:58:23.067 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:23.066+0000 7f9491400640 1 --2- 172.21.15.32:0/1339303819 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f9470077040 0x7f9470079550 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x7f947c033000 tx=0x7f947c004730 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:58:23.071 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:23.070+0000 7f9486000640 1 -- 172.21.15.32:0/1339303819 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7f9480020020 con 0x7f948c0cf3b0 2024-08-03T20:58:23.106 INFO:tasks.ceph.mds.a.smithi032.stdout:starting mds.a at 2024-08-03T20:58:23.318 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:23.316+0000 7f9494800640 1 -- 172.21.15.32:0/1339303819 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "joinable", "val": "true"} v 0) -- 0x7f948c0b3f40 con 0x7f948c0cf3b0 2024-08-03T20:58:24.276 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:24.274+0000 7f9486000640 1 -- 172.21.15.32:0/1339303819 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "joinable", "val": "true"}]=0 cephfs marked joinable; MDS may join as newly active. v253) ==== 162+0+0 (secure 0 0 0) 0x7f94800680c0 con 0x7f948c0cf3b0 2024-08-03T20:58:24.276 INFO:teuthology.orchestra.run.smithi032.stderr:cephfs marked joinable; MDS may join as newly active. 2024-08-03T20:58:24.280 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:24.279+0000 7f9494800640 1 -- 172.21.15.32:0/1339303819 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f9470077040 msgr2=0x7f9470079550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:58:24.280 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:24.279+0000 7f9494800640 1 --2- 172.21.15.32:0/1339303819 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f9470077040 0x7f9470079550 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x7f947c033000 tx=0x7f947c004730 comp rx=0 tx=0).stop 2024-08-03T20:58:24.280 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:24.279+0000 7f9494800640 1 -- 172.21.15.32:0/1339303819 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f948c0cf3b0 msgr2=0x7f948c0ca2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:58:24.281 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:24.279+0000 7f9494800640 1 --2- 172.21.15.32:0/1339303819 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f948c0cf3b0 0x7f948c0ca2a0 secure :-1 s=READY pgs=1759 cs=0 l=1 rev1=1 crypto rx=0x7f948000d650 tx=0x7f948000db10 comp rx=0 tx=0).stop 2024-08-03T20:58:24.281 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:24.279+0000 7f9494800640 1 -- 172.21.15.32:0/1339303819 shutdown_connections 2024-08-03T20:58:24.281 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:24.279+0000 7f9494800640 1 --2- 172.21.15.32:0/1339303819 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f9470077040 0x7f9470079550 unknown :-1 s=CLOSED pgs=715 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:24.281 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:24.279+0000 7f9494800640 1 --2- 172.21.15.32:0/1339303819 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f948c15ba50 0x7f948c0b12b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:24.281 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:24.279+0000 7f9494800640 1 --2- 172.21.15.32:0/1339303819 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f948c155920 0x7f948c1f5070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:24.281 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:24.279+0000 7f9494800640 1 --2- 172.21.15.32:0/1339303819 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f948c0cf3b0 0x7f948c0ca2a0 unknown :-1 s=CLOSED pgs=1759 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:24.281 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:24.279+0000 7f9494800640 1 -- 172.21.15.32:0/1339303819 >> 172.21.15.32:0/1339303819 conn(0x7f948c0b1bd0 msgr2=0x7f948c0b4cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:58:24.281 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:24.279+0000 7f9494800640 1 -- 172.21.15.32:0/1339303819 shutdown_connections 2024-08-03T20:58:24.281 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:24.279+0000 7f9494800640 1 -- 172.21.15.32:0/1339303819 wait complete. 2024-08-03T20:58:24.290 INFO:tasks.ceph.mds.a.smithi032.stderr:2024-08-03T20:58:24.288+0000 7f6998200640 -1 log_channel(cluster) log [ERR] : error decoding table object 'mds0_inotable': End of buffer [buffer:2] 2024-08-03T20:58:24.291 INFO:tasks.ceph.mon.a.smithi032.stderr:2024-08-03T20:58:24.289+0000 7f4e6d600640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-08-03T20:58:24.291 INFO:tasks.ceph.mon.a.smithi032.stderr:2024-08-03T20:58:24.289+0000 7f4e6d600640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 mds daemon damaged (MDS_DAMAGE) 2024-08-03T20:58:24.319 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-08-03T20:58:24.554 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:24.552+0000 7f91ef600640 1 -- 172.21.15.32:0/2324757305 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f91e80747b0 msgr2=0x7f91e80cc8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:58:24.554 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:24.552+0000 7f91ef600640 1 --2- 172.21.15.32:0/2324757305 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f91e80747b0 0x7f91e80cc8e0 secure :-1 s=READY pgs=1763 cs=0 l=1 rev1=1 crypto rx=0x7f91e8071700 tx=0x7f91e002aa10 comp rx=0 tx=0).stop 2024-08-03T20:58:24.554 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:24.552+0000 7f91ef600640 1 -- 172.21.15.32:0/2324757305 shutdown_connections 2024-08-03T20:58:24.554 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:24.552+0000 7f91ef600640 1 --2- 172.21.15.32:0/2324757305 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f91e80747b0 0x7f91e80cc8e0 unknown :-1 s=CLOSED pgs=1763 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:24.554 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:24.552+0000 7f91ef600640 1 --2- 172.21.15.32:0/2324757305 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f91e8073e60 0x7f91e8074270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:24.554 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:24.552+0000 7f91ef600640 1 --2- 172.21.15.32:0/2324757305 >> v2:172.21.15.118:3300/0 conn(0x7f91e806e930 0x7f91e8144f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:24.554 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:24.552+0000 7f91ef600640 1 -- 172.21.15.32:0/2324757305 >> 172.21.15.32:0/2324757305 conn(0x7f91e80b1bd0 msgr2=0x7f91e80b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:58:24.554 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:24.552+0000 7f91ef600640 1 -- 172.21.15.32:0/2324757305 shutdown_connections 2024-08-03T20:58:24.555 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:24.552+0000 7f91ef600640 1 -- 172.21.15.32:0/2324757305 wait complete. 2024-08-03T20:58:24.556 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:24.554+0000 7f91ef600640 1 Processor -- start 2024-08-03T20:58:24.556 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:24.554+0000 7f91ef600640 1 -- start start 2024-08-03T20:58:24.559 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:24.556+0000 7f91ef600640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f91e806e930 0x7f91e80cbc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:58:24.559 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:24.556+0000 7f91ef600640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f91e8073e60 0x7f91e81414e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:58:24.559 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:24.556+0000 7f91ef600640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f91e80747b0 0x7f91e8142fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:58:24.559 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:24.556+0000 7f91ef600640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f91e80cf9c0 con 0x7f91e806e930 2024-08-03T20:58:24.559 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:24.556+0000 7f91ef600640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f91e80cf840 con 0x7f91e80747b0 2024-08-03T20:58:24.559 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:24.556+0000 7f91ef600640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7f91e80cce20 con 0x7f91e8073e60 2024-08-03T20:58:24.559 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:24.556+0000 7f91ecc00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f91e806e930 0x7f91e80cbc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:58:24.559 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:24.556+0000 7f91ecc00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f91e806e930 0x7f91e80cbc50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:42652/0 (socket says 172.21.15.32:42652) 2024-08-03T20:58:24.559 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:24.556+0000 7f91ecc00640 1 -- 172.21.15.32:0/339567816 learned_addr learned my addr 172.21.15.32:0/339567816 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:58:24.559 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:24.556+0000 7f91e7e00640 1 --2- 172.21.15.32:0/339567816 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f91e8073e60 0x7f91e81414e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:58:24.559 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:24.556+0000 7f91ed600640 1 --2- 172.21.15.32:0/339567816 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f91e80747b0 0x7f91e8142fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:58:24.560 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:24.557+0000 7f91ecc00640 1 -- 172.21.15.32:0/339567816 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f91e8073e60 msgr2=0x7f91e81414e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:58:24.560 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:24.557+0000 7f91ecc00640 1 --2- 172.21.15.32:0/339567816 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f91e8073e60 0x7f91e81414e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:24.560 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:24.557+0000 7f91ecc00640 1 -- 172.21.15.32:0/339567816 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f91e80747b0 msgr2=0x7f91e8142fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:58:24.560 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:24.557+0000 7f91ecc00640 1 --2- 172.21.15.32:0/339567816 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f91e80747b0 0x7f91e8142fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:24.560 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:24.557+0000 7f91ecc00640 1 -- 172.21.15.32:0/339567816 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f91e81f8d80 con 0x7f91e806e930 2024-08-03T20:58:24.560 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:24.557+0000 7f91ecc00640 1 --2- 172.21.15.32:0/339567816 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f91e806e930 0x7f91e80cbc50 secure :-1 s=READY pgs=1764 cs=0 l=1 rev1=1 crypto rx=0x7f91dc00c970 tx=0x7f91dc00ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:58:24.560 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:24.557+0000 7f91e4c00640 1 -- 172.21.15.32:0/339567816 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91dc007bb0 con 0x7f91e806e930 2024-08-03T20:58:24.560 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:24.557+0000 7f91e4c00640 1 -- 172.21.15.32:0/339567816 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f91dc007e40 con 0x7f91e806e930 2024-08-03T20:58:24.560 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:24.557+0000 7f91e4c00640 1 -- 172.21.15.32:0/339567816 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91dc00f690 con 0x7f91e806e930 2024-08-03T20:58:24.561 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:24.557+0000 7f91ef600640 1 -- 172.21.15.32:0/339567816 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f91e81f9060 con 0x7f91e806e930 2024-08-03T20:58:24.561 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:24.557+0000 7f91ef600640 1 -- 172.21.15.32:0/339567816 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f91e81f9620 con 0x7f91e806e930 2024-08-03T20:58:24.561 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:24.558+0000 7f91ef600640 1 -- 172.21.15.32:0/339567816 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f91e8074270 con 0x7f91e806e930 2024-08-03T20:58:24.561 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:24.560+0000 7f91e4c00640 1 -- 172.21.15.32:0/339567816 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f91dc00f830 con 0x7f91e806e930 2024-08-03T20:58:24.562 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:24.561+0000 7f91e4c00640 1 --2- 172.21.15.32:0/339567816 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f91c8077010 0x7f91c8079520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:58:24.562 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:24.561+0000 7f91e4c00640 1 -- 172.21.15.32:0/339567816 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(114..114 src has 1..114) ==== 8348+0+0 (secure 0 0 0) 0x7f91dc099bd0 con 0x7f91e806e930 2024-08-03T20:58:24.563 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:24.561+0000 7f91e7e00640 1 --2- 172.21.15.32:0/339567816 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f91c8077010 0x7f91c8079520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:58:24.563 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:24.562+0000 7f91e7e00640 1 --2- 172.21.15.32:0/339567816 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f91c8077010 0x7f91c8079520 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7f91e81e6f10 tx=0x7f91d400b2e0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:58:24.567 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:24.565+0000 7f91e4c00640 1 -- 172.21.15.32:0/339567816 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7f91dc014030 con 0x7f91e806e930 2024-08-03T20:58:24.813 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:24.811+0000 7f91ef600640 1 -- 172.21.15.32:0/339567816 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f91e80b2160 con 0x7f91e806e930 2024-08-03T20:58:24.813 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:24.812+0000 7f91e4c00640 1 -- 172.21.15.32:0/339567816 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 255 v255) ==== 95+0+4391 (secure 0 0 0) 0x7f91dc062250 con 0x7f91e806e930 2024-08-03T20:58:24.814 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-03T20:58:24.814 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":255,"btime":"2024-08-03T20:58:24:291062+0000","default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6439,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6835/531080160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6834","nonce":531080160},{"type":"v1","addr":"172.21.15.118:6835","nonce":531080160}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":84},{"gid":6454,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6837/2796882319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6836","nonce":2796882319},{"type":"v1","addr":"172.21.15.118:6837","nonce":2796882319}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":86},{"gid":6813,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6834/2686933890","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":2686933890},{"type":"v1","addr":"172.21.15.32:6834","nonce":2686933890}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":85}],"filesystems":[{"mdsmap":{"epoch":255,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-08-03T20:54:26.950518+0000","modified":"2024-08-03T20:58:24.290976+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":114,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":6}]} 2024-08-03T20:58:24.814 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 255 2024-08-03T20:58:24.816 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:24.814+0000 7f91ef600640 1 -- 172.21.15.32:0/339567816 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f91c8077010 msgr2=0x7f91c8079520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:58:24.816 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:24.814+0000 7f91ef600640 1 --2- 172.21.15.32:0/339567816 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f91c8077010 0x7f91c8079520 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7f91e81e6f10 tx=0x7f91d400b2e0 comp rx=0 tx=0).stop 2024-08-03T20:58:24.817 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:24.815+0000 7f91ef600640 1 -- 172.21.15.32:0/339567816 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f91e806e930 msgr2=0x7f91e80cbc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:58:24.817 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:24.815+0000 7f91ef600640 1 --2- 172.21.15.32:0/339567816 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f91e806e930 0x7f91e80cbc50 secure :-1 s=READY pgs=1764 cs=0 l=1 rev1=1 crypto rx=0x7f91dc00c970 tx=0x7f91dc00ce30 comp rx=0 tx=0).stop 2024-08-03T20:58:24.817 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:24.815+0000 7f91ef600640 1 -- 172.21.15.32:0/339567816 shutdown_connections 2024-08-03T20:58:24.817 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:24.815+0000 7f91ef600640 1 --2- 172.21.15.32:0/339567816 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f91c8077010 0x7f91c8079520 unknown :-1 s=CLOSED pgs=717 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:24.817 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:24.815+0000 7f91ef600640 1 --2- 172.21.15.32:0/339567816 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f91e80747b0 0x7f91e8142fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:24.817 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:24.815+0000 7f91ef600640 1 --2- 172.21.15.32:0/339567816 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f91e8073e60 0x7f91e81414e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:24.817 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:24.815+0000 7f91ef600640 1 --2- 172.21.15.32:0/339567816 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f91e806e930 0x7f91e80cbc50 unknown :-1 s=CLOSED pgs=1764 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:24.817 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:24.815+0000 7f91ef600640 1 -- 172.21.15.32:0/339567816 >> 172.21.15.32:0/339567816 conn(0x7f91e80b1bd0 msgr2=0x7f91e81408a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:58:24.817 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:24.815+0000 7f91ef600640 1 -- 172.21.15.32:0/339567816 shutdown_connections 2024-08-03T20:58:24.817 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:24.815+0000 7f91ef600640 1 -- 172.21.15.32:0/339567816 wait complete. 2024-08-03T20:58:24.843 DEBUG:tasks.ceph_test_case:wait_until_true: success in 0s and 0 retries 2024-08-03T20:58:24.843 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-08-03T20:58:25.071 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:25.069+0000 7f689fa00640 1 -- 172.21.15.32:0/1878237670 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f689815fff0 msgr2=0x7f6898160400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:58:25.071 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:25.069+0000 7f689fa00640 1 --2- 172.21.15.32:0/1878237670 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f689815fff0 0x7f6898160400 secure :-1 s=READY pgs=1766 cs=0 l=1 rev1=1 crypto rx=0x7f6890005fd0 tx=0x7f689002a8b0 comp rx=0 tx=0).stop 2024-08-03T20:58:25.071 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:25.069+0000 7f689fa00640 1 -- 172.21.15.32:0/1878237670 shutdown_connections 2024-08-03T20:58:25.071 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:25.069+0000 7f689fa00640 1 --2- 172.21.15.32:0/1878237670 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f689815fff0 0x7f6898160400 unknown :-1 s=CLOSED pgs=1766 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:25.072 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:25.069+0000 7f689fa00640 1 --2- 172.21.15.32:0/1878237670 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f6898144f90 0x7f6898074570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:25.072 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:25.069+0000 7f689fa00640 1 -- 172.21.15.32:0/1878237670 >> 172.21.15.32:0/1878237670 conn(0x7f68980b1bd0 msgr2=0x7f68980b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:58:25.072 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:25.069+0000 7f689fa00640 1 -- 172.21.15.32:0/1878237670 shutdown_connections 2024-08-03T20:58:25.072 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:25.069+0000 7f689fa00640 1 -- 172.21.15.32:0/1878237670 wait complete. 2024-08-03T20:58:25.073 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:25.071+0000 7f689fa00640 1 Processor -- start 2024-08-03T20:58:25.073 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:25.072+0000 7f689fa00640 1 -- start start 2024-08-03T20:58:25.076 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:25.073+0000 7f689fa00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f689806e930 0x7f6898156e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:58:25.076 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:25.073+0000 7f689fa00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f6898144f90 0x7f68980cb2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:58:25.076 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:25.073+0000 7f689fa00640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f689815fff0 0x7f689815f360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:58:25.076 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:25.073+0000 7f689fa00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f68980cf840 con 0x7f689806e930 2024-08-03T20:58:25.076 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:25.073+0000 7f689fa00640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f6898145f30 con 0x7f6898144f90 2024-08-03T20:58:25.076 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:25.073+0000 7f689fa00640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7f68980cf9c0 con 0x7f689815fff0 2024-08-03T20:58:25.076 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:25.073+0000 7f689d000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f689806e930 0x7f6898156e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:58:25.076 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:25.073+0000 7f689d000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f689806e930 0x7f6898156e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:42658/0 (socket says 172.21.15.32:42658) 2024-08-03T20:58:25.077 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:25.073+0000 7f689d000640 1 -- 172.21.15.32:0/2621484803 learned_addr learned my addr 172.21.15.32:0/2621484803 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:58:25.077 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:25.074+0000 7f689da00640 1 --2- 172.21.15.32:0/2621484803 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f689815fff0 0x7f689815f360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:58:25.077 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:25.074+0000 7f6897e00640 1 --2- 172.21.15.32:0/2621484803 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f6898144f90 0x7f68980cb2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:58:25.077 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:25.074+0000 7f689d000640 1 -- 172.21.15.32:0/2621484803 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f689815fff0 msgr2=0x7f689815f360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:58:25.077 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:25.074+0000 7f689d000640 1 --2- 172.21.15.32:0/2621484803 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f689815fff0 0x7f689815f360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:25.077 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:25.074+0000 7f689d000640 1 -- 172.21.15.32:0/2621484803 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f6898144f90 msgr2=0x7f68980cb2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:58:25.077 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:25.074+0000 7f689d000640 1 --2- 172.21.15.32:0/2621484803 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f6898144f90 0x7f68980cb2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:25.077 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:25.074+0000 7f689d000640 1 -- 172.21.15.32:0/2621484803 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6898157380 con 0x7f689806e930 2024-08-03T20:58:25.077 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:25.074+0000 7f689d000640 1 --2- 172.21.15.32:0/2621484803 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f689806e930 0x7f6898156e40 secure :-1 s=READY pgs=1767 cs=0 l=1 rev1=1 crypto rx=0x7f688c00c3f0 tx=0x7f688c00c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:58:25.077 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:25.074+0000 7f6894c00640 1 -- 172.21.15.32:0/2621484803 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f688c017070 con 0x7f689806e930 2024-08-03T20:58:25.077 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:25.074+0000 7f6894c00640 1 -- 172.21.15.32:0/2621484803 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f688c0035f0 con 0x7f689806e930 2024-08-03T20:58:25.077 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:25.074+0000 7f6894c00640 1 -- 172.21.15.32:0/2621484803 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f688c003a40 con 0x7f689806e930 2024-08-03T20:58:25.077 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:25.074+0000 7f689fa00640 1 -- 172.21.15.32:0/2621484803 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f68980cbe20 con 0x7f689806e930 2024-08-03T20:58:25.078 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:25.075+0000 7f689fa00640 1 -- 172.21.15.32:0/2621484803 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f68981f8f60 con 0x7f689806e930 2024-08-03T20:58:25.078 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:25.076+0000 7f689fa00640 1 -- 172.21.15.32:0/2621484803 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6898160bc0 con 0x7f689806e930 2024-08-03T20:58:25.078 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:25.077+0000 7f6894c00640 1 -- 172.21.15.32:0/2621484803 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f688c01a070 con 0x7f689806e930 2024-08-03T20:58:25.080 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:25.078+0000 7f6894c00640 1 --2- 172.21.15.32:0/2621484803 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f6878077040 0x7f6878079550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:58:25.080 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:25.079+0000 7f6894c00640 1 -- 172.21.15.32:0/2621484803 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(114..114 src has 1..114) ==== 8348+0+0 (secure 0 0 0) 0x7f688c098660 con 0x7f689806e930 2024-08-03T20:58:25.080 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:25.079+0000 7f6897e00640 1 --2- 172.21.15.32:0/2621484803 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f6878077040 0x7f6878079550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:58:25.081 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:25.080+0000 7f6897e00640 1 --2- 172.21.15.32:0/2621484803 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f6878077040 0x7f6878079550 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 crypto rx=0x7f68981f8990 tx=0x7f6884005f90 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:58:25.085 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:25.083+0000 7f6894c00640 1 -- 172.21.15.32:0/2621484803 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7f6898157380 con 0x7f689806e930 2024-08-03T20:58:25.333 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:25.331+0000 7f689fa00640 1 -- 172.21.15.32:0/2621484803 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f68980703d0 con 0x7f689806e930 2024-08-03T20:58:25.333 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:25.332+0000 7f6894c00640 1 -- 172.21.15.32:0/2621484803 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 255 v255) ==== 95+0+4391 (secure 0 0 0) 0x7f688c069090 con 0x7f689806e930 2024-08-03T20:58:25.333 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-03T20:58:25.333 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":255,"btime":"2024-08-03T20:58:24:291062+0000","default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6439,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6835/531080160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6834","nonce":531080160},{"type":"v1","addr":"172.21.15.118:6835","nonce":531080160}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":84},{"gid":6454,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6837/2796882319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6836","nonce":2796882319},{"type":"v1","addr":"172.21.15.118:6837","nonce":2796882319}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":86},{"gid":6813,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6834/2686933890","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":2686933890},{"type":"v1","addr":"172.21.15.32:6834","nonce":2686933890}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":85}],"filesystems":[{"mdsmap":{"epoch":255,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-08-03T20:54:26.950518+0000","modified":"2024-08-03T20:58:24.290976+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":114,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":6}]} 2024-08-03T20:58:25.334 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 255 2024-08-03T20:58:25.336 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:25.334+0000 7f689fa00640 1 -- 172.21.15.32:0/2621484803 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f6878077040 msgr2=0x7f6878079550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:58:25.336 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:25.334+0000 7f689fa00640 1 --2- 172.21.15.32:0/2621484803 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f6878077040 0x7f6878079550 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 crypto rx=0x7f68981f8990 tx=0x7f6884005f90 comp rx=0 tx=0).stop 2024-08-03T20:58:25.336 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:25.334+0000 7f689fa00640 1 -- 172.21.15.32:0/2621484803 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f689806e930 msgr2=0x7f6898156e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:58:25.336 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:25.334+0000 7f689fa00640 1 --2- 172.21.15.32:0/2621484803 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f689806e930 0x7f6898156e40 secure :-1 s=READY pgs=1767 cs=0 l=1 rev1=1 crypto rx=0x7f688c00c3f0 tx=0x7f688c00c8b0 comp rx=0 tx=0).stop 2024-08-03T20:58:25.336 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:25.334+0000 7f689fa00640 1 -- 172.21.15.32:0/2621484803 shutdown_connections 2024-08-03T20:58:25.336 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:25.334+0000 7f689fa00640 1 --2- 172.21.15.32:0/2621484803 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f6878077040 0x7f6878079550 unknown :-1 s=CLOSED pgs=718 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:25.336 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:25.334+0000 7f689fa00640 1 --2- 172.21.15.32:0/2621484803 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f6898144f90 0x7f68980cb2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:25.337 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:25.334+0000 7f689fa00640 1 --2- 172.21.15.32:0/2621484803 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f689815fff0 0x7f689815f360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:25.337 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:25.334+0000 7f689fa00640 1 --2- 172.21.15.32:0/2621484803 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f689806e930 0x7f6898156e40 unknown :-1 s=CLOSED pgs=1767 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:25.337 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:25.334+0000 7f689fa00640 1 -- 172.21.15.32:0/2621484803 >> 172.21.15.32:0/2621484803 conn(0x7f68980b1bd0 msgr2=0x7f68980d0920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:58:25.337 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:25.334+0000 7f689fa00640 1 -- 172.21.15.32:0/2621484803 shutdown_connections 2024-08-03T20:58:25.337 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:25.334+0000 7f689fa00640 1 -- 172.21.15.32:0/2621484803 wait complete. 2024-08-03T20:58:25.360 DEBUG:tasks.ceph_test_case:wait_until_true: success in 0s and 0 retries 2024-08-03T20:58:25.360 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-08-03T20:58:25.589 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:25.587+0000 7f9591e00640 1 -- 172.21.15.32:0/4022895919 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f958c06e930 msgr2=0x7f958c144f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:58:25.589 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:25.587+0000 7f9591e00640 1 --2- 172.21.15.32:0/4022895919 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f958c06e930 0x7f958c144f90 secure :-1 s=READY pgs=1768 cs=0 l=1 rev1=1 crypto rx=0x7f958c0b2f80 tx=0x7f957c02a560 comp rx=0 tx=0).stop 2024-08-03T20:58:25.589 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:25.587+0000 7f9591e00640 1 -- 172.21.15.32:0/4022895919 shutdown_connections 2024-08-03T20:58:25.589 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:25.587+0000 7f9591e00640 1 --2- 172.21.15.32:0/4022895919 >> v2:172.21.15.118:3300/0 conn(0x7f958c0747b0 0x7f958c0cc8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:25.589 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:25.587+0000 7f9591e00640 1 --2- 172.21.15.32:0/4022895919 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f958c073e60 0x7f958c074270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:25.590 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:25.587+0000 7f9591e00640 1 --2- 172.21.15.32:0/4022895919 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f958c06e930 0x7f958c144f90 unknown :-1 s=CLOSED pgs=1768 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:25.590 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:25.587+0000 7f9591e00640 1 -- 172.21.15.32:0/4022895919 >> 172.21.15.32:0/4022895919 conn(0x7f958c0b1bd0 msgr2=0x7f958c0b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:58:25.590 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:25.587+0000 7f9591e00640 1 -- 172.21.15.32:0/4022895919 shutdown_connections 2024-08-03T20:58:25.590 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:25.587+0000 7f9591e00640 1 -- 172.21.15.32:0/4022895919 wait complete. 2024-08-03T20:58:25.591 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:25.590+0000 7f9591e00640 1 Processor -- start 2024-08-03T20:58:25.591 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:25.590+0000 7f9591e00640 1 -- start start 2024-08-03T20:58:25.596 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:25.591+0000 7f9591e00640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f958c06e930 0x7f958c13c880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:58:25.596 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:25.591+0000 7f9591e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f958c073e60 0x7f958c138e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:58:25.596 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:25.592+0000 7f9591e00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f958c0747b0 0x7f958c1393a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:58:25.596 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:25.592+0000 7f9591e00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f958c0cf9c0 con 0x7f958c073e60 2024-08-03T20:58:25.596 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:25.592+0000 7f9591e00640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f958c0cf840 con 0x7f958c0747b0 2024-08-03T20:58:25.597 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:25.592+0000 7f9591e00640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7f958c0cce20 con 0x7f958c06e930 2024-08-03T20:58:25.597 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:25.592+0000 7f9583e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f958c073e60 0x7f958c138e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:58:25.597 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:25.592+0000 7f9583e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f958c073e60 0x7f958c138e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:42672/0 (socket says 172.21.15.32:42672) 2024-08-03T20:58:25.597 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:25.592+0000 7f9583e00640 1 -- 172.21.15.32:0/2795766161 learned_addr learned my addr 172.21.15.32:0/2795766161 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:58:25.597 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:25.592+0000 7f958aa00640 1 --2- 172.21.15.32:0/2795766161 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f958c06e930 0x7f958c13c880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:58:25.597 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:25.592+0000 7f958b400640 1 --2- 172.21.15.32:0/2795766161 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f958c0747b0 0x7f958c1393a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:58:25.597 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:25.592+0000 7f9583e00640 1 -- 172.21.15.32:0/2795766161 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f958c06e930 msgr2=0x7f958c13c880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:58:25.597 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:25.592+0000 7f9583e00640 1 --2- 172.21.15.32:0/2795766161 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f958c06e930 0x7f958c13c880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:25.597 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:25.592+0000 7f9583e00640 1 -- 172.21.15.32:0/2795766161 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f958c0747b0 msgr2=0x7f958c1393a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:58:25.597 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:25.592+0000 7f9583e00640 1 --2- 172.21.15.32:0/2795766161 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f958c0747b0 0x7f958c1393a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:25.597 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:25.592+0000 7f9583e00640 1 -- 172.21.15.32:0/2795766161 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f958c139b70 con 0x7f958c073e60 2024-08-03T20:58:25.597 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:25.592+0000 7f958b400640 1 --2- 172.21.15.32:0/2795766161 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f958c0747b0 0x7f958c1393a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-03T20:58:25.597 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:25.593+0000 7f9583e00640 1 --2- 172.21.15.32:0/2795766161 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f958c073e60 0x7f958c138e60 secure :-1 s=READY pgs=1769 cs=0 l=1 rev1=1 crypto rx=0x7f957400e7c0 tx=0x7f957400ec80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:58:25.598 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:25.593+0000 7f9583400640 1 -- 172.21.15.32:0/2795766161 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f957400ccc0 con 0x7f958c073e60 2024-08-03T20:58:25.598 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:25.593+0000 7f9583400640 1 -- 172.21.15.32:0/2795766161 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f957400ce80 con 0x7f958c073e60 2024-08-03T20:58:25.598 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:25.593+0000 7f9583400640 1 -- 172.21.15.32:0/2795766161 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9574005280 con 0x7f958c073e60 2024-08-03T20:58:25.598 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:25.593+0000 7f9591e00640 1 -- 172.21.15.32:0/2795766161 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f958c0c4fe0 con 0x7f958c073e60 2024-08-03T20:58:25.598 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:25.593+0000 7f9591e00640 1 -- 172.21.15.32:0/2795766161 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f958c0c55a0 con 0x7f958c073e60 2024-08-03T20:58:25.598 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:25.594+0000 7f9591e00640 1 -- 172.21.15.32:0/2795766161 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f958c074270 con 0x7f958c073e60 2024-08-03T20:58:25.599 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:25.595+0000 7f9583400640 1 -- 172.21.15.32:0/2795766161 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f9574005420 con 0x7f958c073e60 2024-08-03T20:58:25.599 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:25.596+0000 7f9583400640 1 --2- 172.21.15.32:0/2795766161 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f9568077010 0x7f9568079520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:58:25.599 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:25.596+0000 7f9583400640 1 -- 172.21.15.32:0/2795766161 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(114..114 src has 1..114) ==== 8348+0+0 (secure 0 0 0) 0x7f9574099c80 con 0x7f958c073e60 2024-08-03T20:58:25.599 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:25.597+0000 7f958aa00640 1 --2- 172.21.15.32:0/2795766161 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f9568077010 0x7f9568079520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:58:25.599 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:25.598+0000 7f958aa00640 1 --2- 172.21.15.32:0/2795766161 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f9568077010 0x7f9568079520 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 crypto rx=0x7f957c033000 tx=0x7f957c004730 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:58:25.602 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:25.601+0000 7f9583400640 1 -- 172.21.15.32:0/2795766161 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7f95740623e0 con 0x7f958c073e60 2024-08-03T20:58:25.846 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:25.844+0000 7f9591e00640 1 -- 172.21.15.32:0/2795766161 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f958c139900 con 0x7f958c073e60 2024-08-03T20:58:25.846 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:25.845+0000 7f9583400640 1 -- 172.21.15.32:0/2795766161 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 255 v255) ==== 95+0+4391 (secure 0 0 0) 0x7f9574061ce0 con 0x7f958c073e60 2024-08-03T20:58:25.847 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-03T20:58:25.847 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":255,"btime":"2024-08-03T20:58:24:291062+0000","default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6439,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6835/531080160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6834","nonce":531080160},{"type":"v1","addr":"172.21.15.118:6835","nonce":531080160}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":84},{"gid":6454,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6837/2796882319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6836","nonce":2796882319},{"type":"v1","addr":"172.21.15.118:6837","nonce":2796882319}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":86},{"gid":6813,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6834/2686933890","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":2686933890},{"type":"v1","addr":"172.21.15.32:6834","nonce":2686933890}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":85}],"filesystems":[{"mdsmap":{"epoch":255,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-08-03T20:54:26.950518+0000","modified":"2024-08-03T20:58:24.290976+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":114,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":6}]} 2024-08-03T20:58:25.847 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 255 2024-08-03T20:58:25.849 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:25.848+0000 7f9591e00640 1 -- 172.21.15.32:0/2795766161 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f9568077010 msgr2=0x7f9568079520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:58:25.850 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:25.848+0000 7f9591e00640 1 --2- 172.21.15.32:0/2795766161 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f9568077010 0x7f9568079520 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 crypto rx=0x7f957c033000 tx=0x7f957c004730 comp rx=0 tx=0).stop 2024-08-03T20:58:25.850 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:25.848+0000 7f9591e00640 1 -- 172.21.15.32:0/2795766161 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f958c073e60 msgr2=0x7f958c138e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:58:25.850 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:25.848+0000 7f9591e00640 1 --2- 172.21.15.32:0/2795766161 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f958c073e60 0x7f958c138e60 secure :-1 s=READY pgs=1769 cs=0 l=1 rev1=1 crypto rx=0x7f957400e7c0 tx=0x7f957400ec80 comp rx=0 tx=0).stop 2024-08-03T20:58:25.850 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:25.848+0000 7f9591e00640 1 -- 172.21.15.32:0/2795766161 shutdown_connections 2024-08-03T20:58:25.850 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:25.848+0000 7f9591e00640 1 --2- 172.21.15.32:0/2795766161 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f9568077010 0x7f9568079520 unknown :-1 s=CLOSED pgs=719 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:25.851 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:25.848+0000 7f9591e00640 1 --2- 172.21.15.32:0/2795766161 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f958c0747b0 0x7f958c1393a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:25.851 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:25.848+0000 7f9591e00640 1 --2- 172.21.15.32:0/2795766161 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f958c073e60 0x7f958c138e60 unknown :-1 s=CLOSED pgs=1769 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:25.851 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:25.848+0000 7f9591e00640 1 --2- 172.21.15.32:0/2795766161 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f958c06e930 0x7f958c13c880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:25.851 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:25.848+0000 7f9591e00640 1 -- 172.21.15.32:0/2795766161 >> 172.21.15.32:0/2795766161 conn(0x7f958c0b1bd0 msgr2=0x7f958c0b13b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:58:25.851 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:25.848+0000 7f9591e00640 1 -- 172.21.15.32:0/2795766161 shutdown_connections 2024-08-03T20:58:25.851 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:25.848+0000 7f9591e00640 1 -- 172.21.15.32:0/2795766161 wait complete. 2024-08-03T20:58:25.871 INFO:tasks.cephfs.test_damage:Result: Mutation 'Truncate mds0_inotable' led to DAMAGED state 2024-08-03T20:58:25.872 INFO:tasks.cephfs.test_damage:Applying mutation 'Truncate 400.00000000' 2024-08-03T20:58:25.872 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-08-03T20:58:25.872 DEBUG:teuthology.orchestra.run.smithi032:> dd if=/proc/self/mounts of=/dev/stdout 2024-08-03T20:58:25.907 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053453,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6499208k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 none /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249604k,nr_inodes=812401,mode=700,uid=1000,gid=1316 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6499208k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-08-03T20:58:25.908 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi032.front.sepia.ceph.com 2024-08-03T20:58:25.908 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-08-03T20:58:25.908 DEBUG:teuthology.orchestra.run.smithi032:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-08-03T20:58:25.982 INFO:teuthology.orchestra.run.smithi032.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.0': No such file or directory 2024-08-03T20:58:25.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-03T20:58:25.984 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-08-03T20:58:26.255 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:26.253+0000 7fbae7e00640 1 -- 172.21.15.32:0/3614336980 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbae815e000 msgr2=0x7fbae815e410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:58:26.255 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:26.253+0000 7fbae7e00640 1 --2- 172.21.15.32:0/3614336980 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbae815e000 0x7fbae815e410 secure :-1 s=READY pgs=1770 cs=0 l=1 rev1=1 crypto rx=0x7fbae8070130 tx=0x7fbae002a6f0 comp rx=0 tx=0).stop 2024-08-03T20:58:26.255 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:26.253+0000 7fbae7e00640 1 -- 172.21.15.32:0/3614336980 shutdown_connections 2024-08-03T20:58:26.255 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:26.253+0000 7fbae7e00640 1 --2- 172.21.15.32:0/3614336980 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbae815e000 0x7fbae815e410 unknown :-1 s=CLOSED pgs=1770 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:26.255 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:26.253+0000 7fbae7e00640 1 --2- 172.21.15.32:0/3614336980 >> v2:172.21.15.118:3300/0 conn(0x7fbae80ca3b0 0x7fbae80ca7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:26.255 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:26.253+0000 7fbae7e00640 1 --2- 172.21.15.32:0/3614336980 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fbae815a310 0x7fbae80aefb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:26.255 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:26.253+0000 7fbae7e00640 1 -- 172.21.15.32:0/3614336980 >> 172.21.15.32:0/3614336980 conn(0x7fbae80b2120 msgr2=0x7fbae80cfd60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:58:26.255 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:26.253+0000 7fbae7e00640 1 -- 172.21.15.32:0/3614336980 shutdown_connections 2024-08-03T20:58:26.255 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:26.253+0000 7fbae7e00640 1 -- 172.21.15.32:0/3614336980 wait complete. 2024-08-03T20:58:26.256 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:26.255+0000 7fbae7e00640 1 Processor -- start 2024-08-03T20:58:26.257 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:26.255+0000 7fbae7e00640 1 -- start start 2024-08-03T20:58:26.259 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:26.257+0000 7fbae7e00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fbae80ca3b0 0x7fbae815c450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:58:26.260 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:26.257+0000 7fbae7e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbae815a310 0x7fbae81ef300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:58:26.260 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:26.257+0000 7fbae7e00640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fbae815e000 0x7fbae80b4cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:58:26.260 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:26.257+0000 7fbae7e00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fbae80af6b0 con 0x7fbae815a310 2024-08-03T20:58:26.260 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:26.257+0000 7fbae7e00640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7fbae80af530 con 0x7fbae80ca3b0 2024-08-03T20:58:26.260 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:26.257+0000 7fbae7e00640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7fbae80706e0 con 0x7fbae815e000 2024-08-03T20:58:26.260 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:26.257+0000 7fbadfe00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbae815a310 0x7fbae81ef300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:58:26.260 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:26.257+0000 7fbadfe00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbae815a310 0x7fbae81ef300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:42690/0 (socket says 172.21.15.32:42690) 2024-08-03T20:58:26.260 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:26.257+0000 7fbadfe00640 1 -- 172.21.15.32:0/3354335055 learned_addr learned my addr 172.21.15.32:0/3354335055 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:58:26.260 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:26.257+0000 7fbae7400640 1 --2- 172.21.15.32:0/3354335055 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fbae815e000 0x7fbae80b4cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:58:26.260 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:26.257+0000 7fbae6a00640 1 --2- 172.21.15.32:0/3354335055 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fbae80ca3b0 0x7fbae815c450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:58:26.260 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:26.257+0000 7fbadfe00640 1 -- 172.21.15.32:0/3354335055 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fbae815e000 msgr2=0x7fbae80b4cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:58:26.260 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:26.257+0000 7fbadfe00640 1 --2- 172.21.15.32:0/3354335055 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fbae815e000 0x7fbae80b4cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:26.261 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:26.257+0000 7fbadfe00640 1 -- 172.21.15.32:0/3354335055 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fbae80ca3b0 msgr2=0x7fbae815c450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:58:26.261 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:26.257+0000 7fbadfe00640 1 --2- 172.21.15.32:0/3354335055 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fbae80ca3b0 0x7fbae815c450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:26.261 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:26.257+0000 7fbadfe00640 1 -- 172.21.15.32:0/3354335055 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbae815b950 con 0x7fbae815a310 2024-08-03T20:58:26.261 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:26.258+0000 7fbadfe00640 1 --2- 172.21.15.32:0/3354335055 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbae815a310 0x7fbae81ef300 secure :-1 s=READY pgs=1771 cs=0 l=1 rev1=1 crypto rx=0x7fbad000dae0 tx=0x7fbad000a250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:58:26.261 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:26.258+0000 7fbadea00640 1 -- 172.21.15.32:0/3354335055 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbad001b070 con 0x7fbae815a310 2024-08-03T20:58:26.261 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:26.258+0000 7fbadea00640 1 -- 172.21.15.32:0/3354335055 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fbad000e450 con 0x7fbae815a310 2024-08-03T20:58:26.261 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:26.258+0000 7fbae7e00640 1 -- 172.21.15.32:0/3354335055 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbae815ab90 con 0x7fbae815a310 2024-08-03T20:58:26.261 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:26.258+0000 7fbadea00640 1 -- 172.21.15.32:0/3354335055 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbad000e840 con 0x7fbae815a310 2024-08-03T20:58:26.261 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:26.258+0000 7fbae7e00640 1 -- 172.21.15.32:0/3354335055 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbae815b150 con 0x7fbae815a310 2024-08-03T20:58:26.261 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:26.259+0000 7fbae7e00640 1 -- 172.21.15.32:0/3354335055 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbae80aefb0 con 0x7fbae815a310 2024-08-03T20:58:26.262 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:26.261+0000 7fbadea00640 1 -- 172.21.15.32:0/3354335055 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7fbad001e070 con 0x7fbae815a310 2024-08-03T20:58:26.263 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:26.262+0000 7fbadea00640 1 --2- 172.21.15.32:0/3354335055 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fbac4077010 0x7fbac4079520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:58:26.263 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:26.262+0000 7fbadea00640 1 -- 172.21.15.32:0/3354335055 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(114..114 src has 1..114) ==== 8348+0+0 (secure 0 0 0) 0x7fbad0099580 con 0x7fbae815a310 2024-08-03T20:58:26.264 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:26.262+0000 7fbae6a00640 1 --2- 172.21.15.32:0/3354335055 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fbac4077010 0x7fbac4079520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:58:26.264 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:26.263+0000 7fbae6a00640 1 --2- 172.21.15.32:0/3354335055 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fbac4077010 0x7fbac4079520 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7fbad8005fd0 tx=0x7fbad8005850 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:58:26.267 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:26.266+0000 7fbadea00640 1 -- 172.21.15.32:0/3354335055 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7fbad00617b0 con 0x7fbae815a310 2024-08-03T20:58:26.518 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:26.517+0000 7fbae7e00640 1 -- 172.21.15.32:0/3354335055 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7fbae8158280 con 0x7fbae815a310 2024-08-03T20:58:27.303 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:27.301+0000 7fbadea00640 1 -- 172.21.15.32:0/3354335055 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v256) ==== 161+0+0 (secure 0 0 0) 0x7fbad0065d00 con 0x7fbae815a310 2024-08-03T20:58:27.303 INFO:teuthology.orchestra.run.smithi032.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-08-03T20:58:27.307 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:27.305+0000 7fbae7e00640 1 -- 172.21.15.32:0/3354335055 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fbac4077010 msgr2=0x7fbac4079520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:58:27.307 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:27.305+0000 7fbae7e00640 1 --2- 172.21.15.32:0/3354335055 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fbac4077010 0x7fbac4079520 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7fbad8005fd0 tx=0x7fbad8005850 comp rx=0 tx=0).stop 2024-08-03T20:58:27.307 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:27.305+0000 7fbae7e00640 1 -- 172.21.15.32:0/3354335055 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbae815a310 msgr2=0x7fbae81ef300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:58:27.307 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:27.305+0000 7fbae7e00640 1 --2- 172.21.15.32:0/3354335055 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbae815a310 0x7fbae81ef300 secure :-1 s=READY pgs=1771 cs=0 l=1 rev1=1 crypto rx=0x7fbad000dae0 tx=0x7fbad000a250 comp rx=0 tx=0).stop 2024-08-03T20:58:27.307 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:27.305+0000 7fbae7e00640 1 -- 172.21.15.32:0/3354335055 shutdown_connections 2024-08-03T20:58:27.307 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:27.305+0000 7fbae7e00640 1 --2- 172.21.15.32:0/3354335055 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fbac4077010 0x7fbac4079520 unknown :-1 s=CLOSED pgs=720 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:27.307 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:27.305+0000 7fbae7e00640 1 --2- 172.21.15.32:0/3354335055 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fbae815e000 0x7fbae80b4cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:27.308 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:27.305+0000 7fbae7e00640 1 --2- 172.21.15.32:0/3354335055 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbae815a310 0x7fbae81ef300 unknown :-1 s=CLOSED pgs=1771 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:27.308 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:27.305+0000 7fbae7e00640 1 --2- 172.21.15.32:0/3354335055 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fbae80ca3b0 0x7fbae815c450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:27.308 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:27.305+0000 7fbae7e00640 1 -- 172.21.15.32:0/3354335055 >> 172.21.15.32:0/3354335055 conn(0x7fbae80b2120 msgr2=0x7fbae80afa40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:58:27.308 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:27.305+0000 7fbae7e00640 1 -- 172.21.15.32:0/3354335055 shutdown_connections 2024-08-03T20:58:27.308 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:27.305+0000 7fbae7e00640 1 -- 172.21.15.32:0/3354335055 wait complete. 2024-08-03T20:58:27.331 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mds repaired 0 2024-08-03T20:58:27.567 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:27.565+0000 7fc622a00640 1 -- 172.21.15.32:0/287091066 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc61c144f90 msgr2=0x7fc61c074570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:58:27.567 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:27.565+0000 7fc622a00640 1 --2- 172.21.15.32:0/287091066 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc61c144f90 0x7fc61c074570 secure :-1 s=READY pgs=1773 cs=0 l=1 rev1=1 crypto rx=0x7fc61c073b50 tx=0x7fc60402a5c0 comp rx=0 tx=0).stop 2024-08-03T20:58:27.567 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:27.565+0000 7fc622a00640 1 -- 172.21.15.32:0/287091066 shutdown_connections 2024-08-03T20:58:27.567 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:27.565+0000 7fc622a00640 1 --2- 172.21.15.32:0/287091066 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc61c144f90 0x7fc61c074570 unknown :-1 s=CLOSED pgs=1773 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:27.568 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:27.565+0000 7fc622a00640 1 -- 172.21.15.32:0/287091066 >> 172.21.15.32:0/287091066 conn(0x7fc61c0b1bd0 msgr2=0x7fc61c0b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:58:27.568 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:27.565+0000 7fc622a00640 1 -- 172.21.15.32:0/287091066 shutdown_connections 2024-08-03T20:58:27.568 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:27.565+0000 7fc622a00640 1 -- 172.21.15.32:0/287091066 wait complete. 2024-08-03T20:58:27.569 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:27.567+0000 7fc622a00640 1 Processor -- start 2024-08-03T20:58:27.569 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:27.568+0000 7fc622a00640 1 -- start start 2024-08-03T20:58:27.571 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:27.569+0000 7fc622a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc61c06e930 0x7fc61c0aeb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:58:27.571 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:27.569+0000 7fc622a00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc61c144f90 0x7fc61c0b4a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:58:27.571 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:27.569+0000 7fc622a00640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fc61c15fff0 0x7fc61c15ebf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:58:27.572 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:27.569+0000 7fc622a00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fc61c0cf840 con 0x7fc61c06e930 2024-08-03T20:58:27.572 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:27.569+0000 7fc622a00640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7fc61c145f30 con 0x7fc61c144f90 2024-08-03T20:58:27.572 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:27.569+0000 7fc622a00640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7fc61c0cf9c0 con 0x7fc61c15fff0 2024-08-03T20:58:27.572 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:27.569+0000 7fc61be00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc61c06e930 0x7fc61c0aeb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:58:27.572 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:27.569+0000 7fc61be00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc61c06e930 0x7fc61c0aeb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:42714/0 (socket says 172.21.15.32:42714) 2024-08-03T20:58:27.572 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:27.569+0000 7fc61be00640 1 -- 172.21.15.32:0/1291445914 learned_addr learned my addr 172.21.15.32:0/1291445914 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:58:27.572 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:27.569+0000 7fc613e00640 1 --2- 172.21.15.32:0/1291445914 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc61c144f90 0x7fc61c0b4a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:58:27.572 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:27.570+0000 7fc620a00640 1 --2- 172.21.15.32:0/1291445914 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fc61c15fff0 0x7fc61c15ebf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:58:27.572 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:27.570+0000 7fc61be00640 1 -- 172.21.15.32:0/1291445914 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fc61c15fff0 msgr2=0x7fc61c15ebf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:58:27.572 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:27.570+0000 7fc61be00640 1 --2- 172.21.15.32:0/1291445914 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fc61c15fff0 0x7fc61c15ebf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:27.572 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:27.570+0000 7fc61be00640 1 -- 172.21.15.32:0/1291445914 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc61c144f90 msgr2=0x7fc61c0b4a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:58:27.572 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:27.570+0000 7fc61be00640 1 --2- 172.21.15.32:0/1291445914 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc61c144f90 0x7fc61c0b4a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:27.572 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:27.570+0000 7fc61be00640 1 -- 172.21.15.32:0/1291445914 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc61c0ad9f0 con 0x7fc61c06e930 2024-08-03T20:58:27.573 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:27.570+0000 7fc61be00640 1 --2- 172.21.15.32:0/1291445914 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc61c06e930 0x7fc61c0aeb50 secure :-1 s=READY pgs=1774 cs=0 l=1 rev1=1 crypto rx=0x7fc60c00c370 tx=0x7fc60c00c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:58:27.573 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:27.570+0000 7fc618c00640 1 -- 172.21.15.32:0/1291445914 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc60c017070 con 0x7fc61c06e930 2024-08-03T20:58:27.574 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:27.570+0000 7fc618c00640 1 -- 172.21.15.32:0/1291445914 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fc60c0035f0 con 0x7fc61c06e930 2024-08-03T20:58:27.574 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:27.571+0000 7fc622a00640 1 -- 172.21.15.32:0/1291445914 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc61c0adcd0 con 0x7fc61c06e930 2024-08-03T20:58:27.574 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:27.571+0000 7fc618c00640 1 -- 172.21.15.32:0/1291445914 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc60c003a40 con 0x7fc61c06e930 2024-08-03T20:58:27.574 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:27.571+0000 7fc622a00640 1 -- 172.21.15.32:0/1291445914 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc61c0ae290 con 0x7fc61c06e930 2024-08-03T20:58:27.574 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:27.572+0000 7fc622a00640 1 -- 172.21.15.32:0/1291445914 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc61c160400 con 0x7fc61c06e930 2024-08-03T20:58:27.574 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:27.573+0000 7fc618c00640 1 -- 172.21.15.32:0/1291445914 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7fc60c01a070 con 0x7fc61c06e930 2024-08-03T20:58:27.575 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:27.574+0000 7fc618c00640 1 --2- 172.21.15.32:0/1291445914 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fc5f8077010 0x7fc5f8079520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:58:27.575 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:27.574+0000 7fc618c00640 1 -- 172.21.15.32:0/1291445914 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(114..114 src has 1..114) ==== 8348+0+0 (secure 0 0 0) 0x7fc60c0986e0 con 0x7fc61c06e930 2024-08-03T20:58:27.576 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:27.574+0000 7fc613e00640 1 --2- 172.21.15.32:0/1291445914 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fc5f8077010 0x7fc5f8079520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:58:27.576 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:27.575+0000 7fc613e00640 1 --2- 172.21.15.32:0/1291445914 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fc5f8077010 0x7fc5f8079520 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 crypto rx=0x7fc604034000 tx=0x7fc604004730 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:58:27.579 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:27.578+0000 7fc618c00640 1 -- 172.21.15.32:0/1291445914 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7fc60c060790 con 0x7fc61c06e930 2024-08-03T20:58:27.872 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:27.870+0000 7fc622a00640 1 -- 172.21.15.32:0/1291445914 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "mds repaired", "role": "0"} v 0) -- 0x7fc61c145770 con 0x7fc61c06e930 2024-08-03T20:58:28.305 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:28.304+0000 7fc618c00640 1 -- 172.21.15.32:0/1291445914 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "mds repaired", "role": "0"}]=0 repaired: restoring rank 6:0 v257) ==== 101+0+0 (secure 0 0 0) 0x7fc60c0690b0 con 0x7fc61c06e930 2024-08-03T20:58:28.305 INFO:teuthology.orchestra.run.smithi032.stderr:repaired: restoring rank 6:0 2024-08-03T20:58:28.308 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:28.307+0000 7fc622a00640 1 -- 172.21.15.32:0/1291445914 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fc5f8077010 msgr2=0x7fc5f8079520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:58:28.308 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:28.307+0000 7fc622a00640 1 --2- 172.21.15.32:0/1291445914 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fc5f8077010 0x7fc5f8079520 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 crypto rx=0x7fc604034000 tx=0x7fc604004730 comp rx=0 tx=0).stop 2024-08-03T20:58:28.308 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:28.307+0000 7fc622a00640 1 -- 172.21.15.32:0/1291445914 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc61c06e930 msgr2=0x7fc61c0aeb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:58:28.309 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:28.307+0000 7fc622a00640 1 --2- 172.21.15.32:0/1291445914 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc61c06e930 0x7fc61c0aeb50 secure :-1 s=READY pgs=1774 cs=0 l=1 rev1=1 crypto rx=0x7fc60c00c370 tx=0x7fc60c00c830 comp rx=0 tx=0).stop 2024-08-03T20:58:28.309 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:28.307+0000 7fc622a00640 1 -- 172.21.15.32:0/1291445914 shutdown_connections 2024-08-03T20:58:28.309 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:28.307+0000 7fc622a00640 1 --2- 172.21.15.32:0/1291445914 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fc5f8077010 0x7fc5f8079520 unknown :-1 s=CLOSED pgs=721 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:28.309 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:28.307+0000 7fc622a00640 1 --2- 172.21.15.32:0/1291445914 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc61c144f90 0x7fc61c0b4a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:28.309 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:28.307+0000 7fc622a00640 1 --2- 172.21.15.32:0/1291445914 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fc61c15fff0 0x7fc61c15ebf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:28.309 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:28.307+0000 7fc622a00640 1 --2- 172.21.15.32:0/1291445914 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc61c06e930 0x7fc61c0aeb50 unknown :-1 s=CLOSED pgs=1774 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:28.309 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:28.307+0000 7fc622a00640 1 -- 172.21.15.32:0/1291445914 >> 172.21.15.32:0/1291445914 conn(0x7fc61c0b1bd0 msgr2=0x7fc61c0b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:58:28.309 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:28.307+0000 7fc622a00640 1 -- 172.21.15.32:0/1291445914 shutdown_connections 2024-08-03T20:58:28.310 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:28.307+0000 7fc622a00640 1 -- 172.21.15.32:0/1291445914 wait complete. 2024-08-03T20:58:28.336 DEBUG:teuthology.orchestra.run.smithi032:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata import - 2024-08-03T20:58:28.479 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:28.478+0000 7f467b20df40 1 Processor -- start 2024-08-03T20:58:28.479 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:28.478+0000 7f467b20df40 1 -- start start 2024-08-03T20:58:28.481 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:28.479+0000 7f467b20df40 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5589528b9360 0x558952849490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:58:28.481 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:28.479+0000 7f467b20df40 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x5589528baea0 0x55895284a320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:58:28.481 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:28.479+0000 7f467b20df40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5589528d6cb0 0x5589528d3c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:58:28.481 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:28.479+0000 7f467b20df40 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x558952750e30 con 0x5589528b9360 2024-08-03T20:58:28.482 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:28.479+0000 7f467b20df40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x5589527aebf0 con 0x5589528d6cb0 2024-08-03T20:58:28.482 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:28.479+0000 7f467b20df40 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x5589528d70c0 con 0x5589528baea0 2024-08-03T20:58:28.482 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:28.479+0000 7f4673800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5589528b9360 0x558952849490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:58:28.482 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:28.479+0000 7f4673800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5589528b9360 0x558952849490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:42728/0 (socket says 172.21.15.32:42728) 2024-08-03T20:58:28.482 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:28.479+0000 7f4673800640 1 -- 172.21.15.32:0/524701808 learned_addr learned my addr 172.21.15.32:0/524701808 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:58:28.482 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:28.480+0000 7f4672e00640 1 --2- 172.21.15.32:0/524701808 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x5589528baea0 0x55895284a320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:58:28.482 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:28.480+0000 7f4674200640 1 --2- 172.21.15.32:0/524701808 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5589528d6cb0 0x5589528d3c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:58:28.482 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:28.480+0000 7f4673800640 1 -- 172.21.15.32:0/524701808 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x5589528baea0 msgr2=0x55895284a320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:58:28.482 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:28.480+0000 7f4673800640 1 --2- 172.21.15.32:0/524701808 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x5589528baea0 0x55895284a320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:28.482 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:28.480+0000 7f4673800640 1 -- 172.21.15.32:0/524701808 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5589528d6cb0 msgr2=0x5589528d3c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:58:28.482 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:28.480+0000 7f4673800640 1 --2- 172.21.15.32:0/524701808 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5589528d6cb0 0x5589528d3c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:28.483 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:28.480+0000 7f4673800640 1 -- 172.21.15.32:0/524701808 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5589528b7fc0 con 0x5589528b9360 2024-08-03T20:58:28.484 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:28.480+0000 7f4673800640 1 --2- 172.21.15.32:0/524701808 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5589528b9360 0x558952849490 secure :-1 s=READY pgs=1775 cs=0 l=1 rev1=1 crypto rx=0x7f4664046560 tx=0x7f4664002fb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c68c134c1260d41e server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:58:28.484 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:28.481+0000 7f4671a00640 1 -- 172.21.15.32:0/524701808 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f466404db50 con 0x5589528b9360 2024-08-03T20:58:28.485 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:28.481+0000 7f4671a00640 1 -- 172.21.15.32:0/524701808 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f466404dcf0 con 0x5589528b9360 2024-08-03T20:58:28.485 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:28.481+0000 7f4671a00640 1 -- 172.21.15.32:0/524701808 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46640566d0 con 0x5589528b9360 2024-08-03T20:58:28.485 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:28.481+0000 7f467b20df40 1 -- 172.21.15.32:0/524701808 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5589528b9360 msgr2=0x558952849490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:58:28.485 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:28.481+0000 7f467b20df40 1 --2- 172.21.15.32:0/524701808 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5589528b9360 0x558952849490 secure :-1 s=READY pgs=1775 cs=0 l=1 rev1=1 crypto rx=0x7f4664046560 tx=0x7f4664002fb0 comp rx=0 tx=0).stop 2024-08-03T20:58:28.485 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:28.481+0000 7f467b20df40 1 -- 172.21.15.32:0/524701808 shutdown_connections 2024-08-03T20:58:28.485 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:28.481+0000 7f467b20df40 1 --2- 172.21.15.32:0/524701808 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5589528d6cb0 0x5589528d3c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:28.485 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:28.481+0000 7f467b20df40 1 --2- 172.21.15.32:0/524701808 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x5589528baea0 0x55895284a320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:28.485 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:28.481+0000 7f467b20df40 1 --2- 172.21.15.32:0/524701808 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5589528b9360 0x558952849490 unknown :-1 s=CLOSED pgs=1775 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:28.485 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:28.481+0000 7f467b20df40 1 -- 172.21.15.32:0/524701808 >> 172.21.15.32:0/524701808 conn(0x558952722ac0 msgr2=0x55895284b810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:58:28.485 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:28.481+0000 7f467b20df40 1 -- 172.21.15.32:0/524701808 shutdown_connections 2024-08-03T20:58:28.485 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:28.481+0000 7f467b20df40 1 -- 172.21.15.32:0/524701808 wait complete. 2024-08-03T20:58:28.486 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:28.482+0000 7f467b20df40 1 Processor -- start 2024-08-03T20:58:28.486 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:28.483+0000 7f467b20df40 1 -- start start 2024-08-03T20:58:28.487 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:28.484+0000 7f467b20df40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5589528b9360 0x5589528cf170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:58:28.488 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:28.484+0000 7f467b20df40 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x5589528baea0 0x558952846f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:58:28.488 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:28.484+0000 7f467b20df40 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5589528d6cb0 0x5589528b6900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:58:28.488 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:28.484+0000 7f467b20df40 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x5589528499d0 con 0x5589528d6cb0 2024-08-03T20:58:28.488 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:28.484+0000 7f467b20df40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x5589528d5f60 con 0x5589528b9360 2024-08-03T20:58:28.488 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:28.484+0000 7f467b20df40 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x5589528b4c10 con 0x5589528baea0 2024-08-03T20:58:28.488 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:28.484+0000 7f4674200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5589528d6cb0 0x5589528b6900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:58:28.488 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:28.484+0000 7f4674200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5589528d6cb0 0x5589528b6900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:42744/0 (socket says 172.21.15.32:42744) 2024-08-03T20:58:28.488 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:28.484+0000 7f4674200640 1 -- 172.21.15.32:0/4009968897 learned_addr learned my addr 172.21.15.32:0/4009968897 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:58:28.488 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:28.485+0000 7f4672e00640 1 --2- 172.21.15.32:0/4009968897 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x5589528baea0 0x558952846f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:58:28.488 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:28.485+0000 7f4673800640 1 --2- 172.21.15.32:0/4009968897 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5589528b9360 0x5589528cf170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:58:28.489 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:28.485+0000 7f4674200640 1 -- 172.21.15.32:0/4009968897 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x5589528baea0 msgr2=0x558952846f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:58:28.489 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:28.485+0000 7f4674200640 1 --2- 172.21.15.32:0/4009968897 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x5589528baea0 0x558952846f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:28.489 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:28.485+0000 7f4674200640 1 -- 172.21.15.32:0/4009968897 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5589528b9360 msgr2=0x5589528cf170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:58:28.489 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:28.485+0000 7f4674200640 1 --2- 172.21.15.32:0/4009968897 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5589528b9360 0x5589528cf170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:28.489 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:28.485+0000 7f4674200640 1 -- 172.21.15.32:0/4009968897 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558952847a40 con 0x5589528d6cb0 2024-08-03T20:58:28.489 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:28.485+0000 7f4674200640 1 --2- 172.21.15.32:0/4009968897 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5589528d6cb0 0x5589528b6900 secure :-1 s=READY pgs=1776 cs=0 l=1 rev1=1 crypto rx=0x7f466c002fa0 tx=0x7f466c00d5f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:58:28.489 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:28.485+0000 7f466be00640 1 -- 172.21.15.32:0/4009968897 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f466c013480 con 0x5589528d6cb0 2024-08-03T20:58:28.489 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:28.486+0000 7f466be00640 1 -- 172.21.15.32:0/4009968897 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f466c018070 con 0x5589528d6cb0 2024-08-03T20:58:28.489 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:28.486+0000 7f466be00640 1 -- 172.21.15.32:0/4009968897 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f466c0139d0 con 0x5589528d6cb0 2024-08-03T20:58:28.489 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:28.486+0000 7f467b20df40 1 -- 172.21.15.32:0/4009968897 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x558952847d20 con 0x5589528d6cb0 2024-08-03T20:58:28.489 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:28.486+0000 7f467b20df40 1 -- 172.21.15.32:0/4009968897 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5589528482e0 con 0x5589528d6cb0 2024-08-03T20:58:28.490 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:28.488+0000 7f466be00640 1 -- 172.21.15.32:0/4009968897 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f466c013b70 con 0x5589528d6cb0 2024-08-03T20:58:28.490 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:28.489+0000 7f466be00640 1 --2- 172.21.15.32:0/4009968897 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f4654075a20 0x7f4654077f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:58:28.490 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:28.489+0000 7f466be00640 1 -- 172.21.15.32:0/4009968897 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(114..114 src has 1..114) ==== 8348+0+0 (secure 0 0 0) 0x7f466c097e70 con 0x5589528d6cb0 2024-08-03T20:58:28.491 INFO:teuthology.orchestra.run.smithi032.stdout:Importing pool 2024-08-03T20:58:28.491 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:28.489+0000 7f4673800640 1 --2- 172.21.15.32:0/4009968897 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f4654075a20 0x7f4654077f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:58:28.492 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:28.490+0000 7f467b20df40 1 --2- 172.21.15.32:0/4009968897 >> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] conn(0x5589528b3de0 0x5589528b41f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:58:28.492 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:28.490+0000 7f467b20df40 1 -- 172.21.15.32:0/4009968897 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:1 12.10 12:09401a81:::601.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e114) -- 0x5589528b4730 con 0x5589528b3de0 2024-08-03T20:58:28.492 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:28.490+0000 7f4672e00640 1 --2- 172.21.15.32:0/4009968897 >> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] conn(0x5589528b3de0 0x5589528b41f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:58:28.492 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:28.490+0000 7f4673800640 1 --2- 172.21.15.32:0/4009968897 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f4654075a20 0x7f4654077f30 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x7f4664062000 tx=0x7f4664004940 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:58:28.492 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:28.491+0000 7f4672e00640 1 --2- 172.21.15.32:0/4009968897 >> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] conn(0x5589528b3de0 0x5589528b41f0 crc :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:58:28.494 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:28.493+0000 7f4672e00640 1 -- 172.21.15.32:0/4009968897 <== osd.0 v2:172.21.15.32:6816/3066156116 1 ==== osd_op_reply(1 601.00000000 [create] v114'877 uv851 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f466000c040 con 0x5589528b3de0 2024-08-03T20:58:28.495 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:28.493+0000 7f467b20df40 1 -- 172.21.15.32:0/4009968897 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:2 12.10 12:09401a81:::601.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e114) -- 0x55895297b990 con 0x5589528b3de0 2024-08-03T20:58:28.496 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:28.495+0000 7f4672e00640 1 -- 172.21.15.32:0/4009968897 <== osd.0 v2:172.21.15.32:6816/3066156116 2 ==== osd_op_reply(2 601.00000000 [delete] v114'878 uv878 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f466000c040 con 0x5589528b3de0 2024-08-03T20:58:28.496 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:28.495+0000 7f467b20df40 1 -- 172.21.15.32:0/4009968897 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:3 12.10 12:09401a81:::601.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e114) -- 0x55895297bd30 con 0x5589528b3de0 2024-08-03T20:58:28.498 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:28.497+0000 7f4672e00640 1 -- 172.21.15.32:0/4009968897 <== osd.0 v2:172.21.15.32:6816/3066156116 3 ==== osd_op_reply(3 601.00000000 [create] v114'879 uv879 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f466000c040 con 0x5589528b3de0 2024-08-03T20:58:28.498 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::601.00000000:head# 2024-08-03T20:58:28.498 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:28.497+0000 7f467b20df40 1 -- 172.21.15.32:0/4009968897 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:4 12.10 12:09401a81:::601.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e114) -- 0x55895297d0e0 con 0x5589528b3de0 2024-08-03T20:58:28.500 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:28.498+0000 7f4672e00640 1 -- 172.21.15.32:0/4009968897 <== osd.0 v2:172.21.15.32:6816/3066156116 4 ==== osd_op_reply(4 601.00000000 [setxattr (30)] v114'880 uv880 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f466000c040 con 0x5589528b3de0 2024-08-03T20:58:28.500 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:28.498+0000 7f467b20df40 1 -- 172.21.15.32:0/4009968897 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:5 12.10 12:09401a81:::601.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e114) -- 0x55895297d480 con 0x5589528b3de0 2024-08-03T20:58:28.501 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:28.500+0000 7f4672e00640 1 -- 172.21.15.32:0/4009968897 <== osd.0 v2:172.21.15.32:6816/3066156116 5 ==== osd_op_reply(5 601.00000000 [setxattr (62)] v114'881 uv881 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f466000c040 con 0x5589528b3de0 2024-08-03T20:58:28.502 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:28.500+0000 7f467b20df40 1 -- 172.21.15.32:0/4009968897 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:6 12.10 12:09401a81:::601.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e114) -- 0x55895297c0d0 con 0x5589528b3de0 2024-08-03T20:58:28.503 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:28.502+0000 7f4672e00640 1 -- 172.21.15.32:0/4009968897 <== osd.0 v2:172.21.15.32:6816/3066156116 6 ==== osd_op_reply(6 601.00000000 [omap-set-header] v114'882 uv882 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f466000c040 con 0x5589528b3de0 2024-08-03T20:58:28.503 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:28.502+0000 7f467b20df40 1 -- 172.21.15.32:0/4009968897 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:7 12.10 12:0bd6d154:::602.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e114) -- 0x55895297c4c0 con 0x5589528b3de0 2024-08-03T20:58:28.505 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:28.504+0000 7f4672e00640 1 -- 172.21.15.32:0/4009968897 <== osd.0 v2:172.21.15.32:6816/3066156116 7 ==== osd_op_reply(7 602.00000000 [create] v114'883 uv857 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f466000c040 con 0x5589528b3de0 2024-08-03T20:58:28.505 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:28.504+0000 7f467b20df40 1 -- 172.21.15.32:0/4009968897 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:8 12.10 12:0bd6d154:::602.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e114) -- 0x55895297c860 con 0x5589528b3de0 2024-08-03T20:58:28.507 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:28.505+0000 7f4672e00640 1 -- 172.21.15.32:0/4009968897 <== osd.0 v2:172.21.15.32:6816/3066156116 8 ==== osd_op_reply(8 602.00000000 [delete] v114'884 uv884 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f466000c040 con 0x5589528b3de0 2024-08-03T20:58:28.507 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:28.505+0000 7f467b20df40 1 -- 172.21.15.32:0/4009968897 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:9 12.10 12:0bd6d154:::602.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e114) -- 0x55895297c860 con 0x5589528b3de0 2024-08-03T20:58:28.508 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::602.00000000:head# 2024-08-03T20:58:28.508 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:28.507+0000 7f4672e00640 1 -- 172.21.15.32:0/4009968897 <== osd.0 v2:172.21.15.32:6816/3066156116 9 ==== osd_op_reply(9 602.00000000 [create] v114'885 uv885 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f466000c040 con 0x5589528b3de0 2024-08-03T20:58:28.509 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:28.507+0000 7f467b20df40 1 -- 172.21.15.32:0/4009968897 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:10 12.10 12:0bd6d154:::602.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e114) -- 0x55895297c860 con 0x5589528b3de0 2024-08-03T20:58:28.510 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:28.509+0000 7f4672e00640 1 -- 172.21.15.32:0/4009968897 <== osd.0 v2:172.21.15.32:6816/3066156116 10 ==== osd_op_reply(10 602.00000000 [setxattr (30)] v114'886 uv886 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f466000c040 con 0x5589528b3de0 2024-08-03T20:58:28.510 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:28.509+0000 7f467b20df40 1 -- 172.21.15.32:0/4009968897 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:11 12.10 12:0bd6d154:::602.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e114) -- 0x55895297c860 con 0x5589528b3de0 2024-08-03T20:58:28.511 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:28.510+0000 7f4672e00640 1 -- 172.21.15.32:0/4009968897 <== osd.0 v2:172.21.15.32:6816/3066156116 11 ==== osd_op_reply(11 602.00000000 [setxattr (62)] v114'887 uv887 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f466000c040 con 0x5589528b3de0 2024-08-03T20:58:28.511 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:28.510+0000 7f467b20df40 1 -- 172.21.15.32:0/4009968897 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:12 12.10 12:0bd6d154:::602.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e114) -- 0x55895297c860 con 0x5589528b3de0 2024-08-03T20:58:28.513 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:28.512+0000 7f4672e00640 1 -- 172.21.15.32:0/4009968897 <== osd.0 v2:172.21.15.32:6816/3066156116 12 ==== osd_op_reply(12 602.00000000 [omap-set-header] v114'888 uv888 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f466000c040 con 0x5589528b3de0 2024-08-03T20:58:28.513 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:28.512+0000 7f467b20df40 1 -- 172.21.15.32:0/4009968897 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:13 12.10 12:0cf6ad0f:::10000000000.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e114) -- 0x55895297c860 con 0x5589528b3de0 2024-08-03T20:58:28.515 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:28.514+0000 7f4672e00640 1 -- 172.21.15.32:0/4009968897 <== osd.0 v2:172.21.15.32:6816/3066156116 13 ==== osd_op_reply(13 10000000000.00000000 [create] v114'889 uv864 ondisk = -17 ((17) File exists)) ==== 164+0+0 (crc 0 0 0) 0x7f466000c040 con 0x5589528b3de0 2024-08-03T20:58:28.515 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:28.514+0000 7f467b20df40 1 -- 172.21.15.32:0/4009968897 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:14 12.10 12:0cf6ad0f:::10000000000.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e114) -- 0x55895297c860 con 0x5589528b3de0 2024-08-03T20:58:28.517 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:28.515+0000 7f4672e00640 1 -- 172.21.15.32:0/4009968897 <== osd.0 v2:172.21.15.32:6816/3066156116 14 ==== osd_op_reply(14 10000000000.00000000 [delete] v114'890 uv890 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f466000c040 con 0x5589528b3de0 2024-08-03T20:58:28.517 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:28.515+0000 7f467b20df40 1 -- 172.21.15.32:0/4009968897 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:15 12.10 12:0cf6ad0f:::10000000000.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e114) -- 0x55895297c860 con 0x5589528b3de0 2024-08-03T20:58:28.518 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:28.517+0000 7f4672e00640 1 -- 172.21.15.32:0/4009968897 <== osd.0 v2:172.21.15.32:6816/3066156116 15 ==== osd_op_reply(15 10000000000.00000000 [create] v114'891 uv891 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f466000c040 con 0x5589528b3de0 2024-08-03T20:58:28.519 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::10000000000.00000000:head# 2024-08-03T20:58:28.519 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:28.517+0000 7f467b20df40 1 -- 172.21.15.32:0/4009968897 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:16 12.10 12:0cf6ad0f:::10000000000.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e114) -- 0x55895297c860 con 0x5589528b3de0 2024-08-03T20:58:28.520 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:28.518+0000 7f4672e00640 1 -- 172.21.15.32:0/4009968897 <== osd.0 v2:172.21.15.32:6816/3066156116 16 ==== osd_op_reply(16 10000000000.00000000 [setxattr (30)] v114'892 uv892 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f466000c040 con 0x5589528b3de0 2024-08-03T20:58:28.520 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:28.519+0000 7f467b20df40 1 -- 172.21.15.32:0/4009968897 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:17 12.10 12:0cf6ad0f:::10000000000.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e114) -- 0x55895297c860 con 0x5589528b3de0 2024-08-03T20:58:28.521 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:28.520+0000 7f4672e00640 1 -- 172.21.15.32:0/4009968897 <== osd.0 v2:172.21.15.32:6816/3066156116 17 ==== osd_op_reply(17 10000000000.00000000 [setxattr (62)] v114'893 uv893 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f466000c040 con 0x5589528b3de0 2024-08-03T20:58:28.522 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:28.520+0000 7f467b20df40 1 -- 172.21.15.32:0/4009968897 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:18 12.10 12:0cf6ad0f:::10000000000.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e114) -- 0x55895297c860 con 0x5589528b3de0 2024-08-03T20:58:28.523 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:28.522+0000 7f4672e00640 1 -- 172.21.15.32:0/4009968897 <== osd.0 v2:172.21.15.32:6816/3066156116 18 ==== osd_op_reply(18 10000000000.00000000 [omap-set-header] v114'894 uv894 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f466000c040 con 0x5589528b3de0 2024-08-03T20:58:28.523 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:28.522+0000 7f467b20df40 1 -- 172.21.15.32:0/4009968897 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:19 12.10 12:0cf6ad0f:::10000000000.00000000:head [omap-set-vals in=593b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e114) -- 0x55895297cc40 con 0x5589528b3de0 2024-08-03T20:58:28.525 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:28.523+0000 7f4672e00640 1 -- 172.21.15.32:0/4009968897 <== osd.0 v2:172.21.15.32:6816/3066156116 19 ==== osd_op_reply(19 10000000000.00000000 [omap-set-vals] v114'895 uv895 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f466000c040 con 0x5589528b3de0 2024-08-03T20:58:28.525 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:28.523+0000 7f467b20df40 1 -- 172.21.15.32:0/4009968897 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:20 12.10 12:0d82a743:::600.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e114) -- 0x55895297cc40 con 0x5589528b3de0 2024-08-03T20:58:28.526 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:28.525+0000 7f4672e00640 1 -- 172.21.15.32:0/4009968897 <== osd.0 v2:172.21.15.32:6816/3066156116 20 ==== osd_op_reply(20 600.00000000 [create] v114'896 uv870 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f466000c040 con 0x5589528b3de0 2024-08-03T20:58:28.527 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:28.525+0000 7f467b20df40 1 -- 172.21.15.32:0/4009968897 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:21 12.10 12:0d82a743:::600.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e114) -- 0x55895297cc20 con 0x5589528b3de0 2024-08-03T20:58:28.528 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:28.527+0000 7f4672e00640 1 -- 172.21.15.32:0/4009968897 <== osd.0 v2:172.21.15.32:6816/3066156116 21 ==== osd_op_reply(21 600.00000000 [delete] v114'897 uv897 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f466000c040 con 0x5589528b3de0 2024-08-03T20:58:28.528 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:28.527+0000 7f467b20df40 1 -- 172.21.15.32:0/4009968897 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:22 12.10 12:0d82a743:::600.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e114) -- 0x55895297cc20 con 0x5589528b3de0 2024-08-03T20:58:28.530 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::600.00000000:head# 2024-08-03T20:58:28.530 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:28.528+0000 7f4672e00640 1 -- 172.21.15.32:0/4009968897 <== osd.0 v2:172.21.15.32:6816/3066156116 22 ==== osd_op_reply(22 600.00000000 [create] v114'898 uv898 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f466000c040 con 0x5589528b3de0 2024-08-03T20:58:28.530 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:28.528+0000 7f467b20df40 1 -- 172.21.15.32:0/4009968897 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:23 12.10 12:0d82a743:::600.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e114) -- 0x55895297cc20 con 0x5589528b3de0 2024-08-03T20:58:28.531 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:28.530+0000 7f4672e00640 1 -- 172.21.15.32:0/4009968897 <== osd.0 v2:172.21.15.32:6816/3066156116 23 ==== osd_op_reply(23 600.00000000 [setxattr (30)] v114'899 uv899 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f466000c040 con 0x5589528b3de0 2024-08-03T20:58:28.531 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:28.530+0000 7f467b20df40 1 -- 172.21.15.32:0/4009968897 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:24 12.10 12:0d82a743:::600.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e114) -- 0x55895297cc20 con 0x5589528b3de0 2024-08-03T20:58:28.533 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:28.531+0000 7f4672e00640 1 -- 172.21.15.32:0/4009968897 <== osd.0 v2:172.21.15.32:6816/3066156116 24 ==== osd_op_reply(24 600.00000000 [setxattr (62)] v114'900 uv900 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f466000c040 con 0x5589528b3de0 2024-08-03T20:58:28.533 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:28.531+0000 7f467b20df40 1 -- 172.21.15.32:0/4009968897 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:25 12.10 12:0d82a743:::600.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e114) -- 0x55895297cc20 con 0x5589528b3de0 2024-08-03T20:58:28.534 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:28.533+0000 7f4672e00640 1 -- 172.21.15.32:0/4009968897 <== osd.0 v2:172.21.15.32:6816/3066156116 25 ==== osd_op_reply(25 600.00000000 [omap-set-header] v114'901 uv901 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f466000c040 con 0x5589528b3de0 2024-08-03T20:58:28.534 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:28.533+0000 7f467b20df40 1 -- 172.21.15.32:0/4009968897 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:26 12.10 12:0d89b25e:::603.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e114) -- 0x55895297cc20 con 0x5589528b3de0 2024-08-03T20:58:28.536 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:28.535+0000 7f4672e00640 1 -- 172.21.15.32:0/4009968897 <== osd.0 v2:172.21.15.32:6816/3066156116 26 ==== osd_op_reply(26 603.00000000 [create] v114'902 uv876 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f466000c040 con 0x5589528b3de0 2024-08-03T20:58:28.536 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:28.535+0000 7f467b20df40 1 -- 172.21.15.32:0/4009968897 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:27 12.10 12:0d89b25e:::603.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e114) -- 0x55895297cc20 con 0x5589528b3de0 2024-08-03T20:58:28.538 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:28.536+0000 7f4672e00640 1 -- 172.21.15.32:0/4009968897 <== osd.0 v2:172.21.15.32:6816/3066156116 27 ==== osd_op_reply(27 603.00000000 [delete] v114'903 uv903 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f466000c040 con 0x5589528b3de0 2024-08-03T20:58:28.538 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:28.536+0000 7f467b20df40 1 -- 172.21.15.32:0/4009968897 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:28 12.10 12:0d89b25e:::603.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e114) -- 0x55895297cc20 con 0x5589528b3de0 2024-08-03T20:58:28.539 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::603.00000000:head# 2024-08-03T20:58:28.540 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:28.538+0000 7f4672e00640 1 -- 172.21.15.32:0/4009968897 <== osd.0 v2:172.21.15.32:6816/3066156116 28 ==== osd_op_reply(28 603.00000000 [create] v114'904 uv904 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f466000c040 con 0x5589528b3de0 2024-08-03T20:58:28.540 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:28.538+0000 7f467b20df40 1 -- 172.21.15.32:0/4009968897 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:29 12.10 12:0d89b25e:::603.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e114) -- 0x55895297cc20 con 0x5589528b3de0 2024-08-03T20:58:28.541 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:28.540+0000 7f4672e00640 1 -- 172.21.15.32:0/4009968897 <== osd.0 v2:172.21.15.32:6816/3066156116 29 ==== osd_op_reply(29 603.00000000 [setxattr (30)] v114'905 uv905 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f466000c040 con 0x5589528b3de0 2024-08-03T20:58:28.541 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:28.540+0000 7f467b20df40 1 -- 172.21.15.32:0/4009968897 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:30 12.10 12:0d89b25e:::603.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e114) -- 0x55895297cb20 con 0x5589528b3de0 2024-08-03T20:58:28.542 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:28.541+0000 7f4672e00640 1 -- 172.21.15.32:0/4009968897 <== osd.0 v2:172.21.15.32:6816/3066156116 30 ==== osd_op_reply(30 603.00000000 [setxattr (62)] v114'906 uv906 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f466000c040 con 0x5589528b3de0 2024-08-03T20:58:28.543 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:28.541+0000 7f467b20df40 1 -- 172.21.15.32:0/4009968897 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:31 12.10 12:0d89b25e:::603.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e114) -- 0x55895297cb20 con 0x5589528b3de0 2024-08-03T20:58:28.544 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:28.543+0000 7f4672e00640 1 -- 172.21.15.32:0/4009968897 <== osd.0 v2:172.21.15.32:6816/3066156116 31 ==== osd_op_reply(31 603.00000000 [omap-set-header] v114'907 uv907 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f466000c040 con 0x5589528b3de0 2024-08-03T20:58:28.544 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:28.543+0000 7f467b20df40 1 --2- 172.21.15.32:0/4009968897 >> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] conn(0x55895297dd30 0x558952980150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:58:28.545 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:28.543+0000 7f467b20df40 1 -- 172.21.15.32:0/4009968897 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:32 12.14 12:287034c4:::1.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e114) -- 0x55895297cb20 con 0x55895297dd30 2024-08-03T20:58:28.545 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:28.543+0000 7f4673800640 1 --2- 172.21.15.32:0/4009968897 >> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] conn(0x55895297dd30 0x558952980150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:58:28.545 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:28.544+0000 7f4673800640 1 --2- 172.21.15.32:0/4009968897 >> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] conn(0x55895297dd30 0x558952980150 crc :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:58:28.548 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:28.546+0000 7f4673800640 1 -- 172.21.15.32:0/4009968897 <== osd.5 v2:172.21.15.118:6810/2724891770 1 ==== osd_op_reply(32 1.00000000.inode [create] v114'638 uv621 ondisk = -17 ((17) File exists)) ==== 160+0+0 (crc 0 0 0) 0x7f466405f030 con 0x55895297dd30 2024-08-03T20:58:28.548 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:28.546+0000 7f467b20df40 1 -- 172.21.15.32:0/4009968897 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:33 12.14 12:287034c4:::1.00000000.inode:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e114) -- 0x558952980690 con 0x55895297dd30 2024-08-03T20:58:28.549 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:28.548+0000 7f4673800640 1 -- 172.21.15.32:0/4009968897 <== osd.5 v2:172.21.15.118:6810/2724891770 2 ==== osd_op_reply(33 1.00000000.inode [delete] v114'639 uv639 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f466405f030 con 0x55895297dd30 2024-08-03T20:58:28.549 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:28.548+0000 7f467b20df40 1 -- 172.21.15.32:0/4009968897 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:34 12.14 12:287034c4:::1.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e114) -- 0x558952980a30 con 0x55895297dd30 2024-08-03T20:58:28.550 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::1.00000000.inode:head# 2024-08-03T20:58:28.551 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:28.549+0000 7f4673800640 1 -- 172.21.15.32:0/4009968897 <== osd.5 v2:172.21.15.118:6810/2724891770 3 ==== osd_op_reply(34 1.00000000.inode [create] v114'640 uv640 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f466405f030 con 0x55895297dd30 2024-08-03T20:58:28.551 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:28.549+0000 7f467b20df40 1 -- 172.21.15.32:0/4009968897 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:35 12.14 12:287034c4:::1.00000000.inode:head [write 0~582 in=582b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e114) -- 0x558952980dd0 con 0x55895297dd30 2024-08-03T20:58:28.552 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:28.551+0000 7f4673800640 1 -- 172.21.15.32:0/4009968897 <== osd.5 v2:172.21.15.118:6810/2724891770 4 ==== osd_op_reply(35 1.00000000.inode [write 0~582] v114'641 uv641 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f466405f030 con 0x55895297dd30 2024-08-03T20:58:28.552 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:28.551+0000 7f467b20df40 1 -- 172.21.15.32:0/4009968897 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:36 12.14 12:287034c4:::1.00000000.inode:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e114) -- 0x558952981170 con 0x55895297dd30 2024-08-03T20:58:28.554 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:28.553+0000 7f4673800640 1 -- 172.21.15.32:0/4009968897 <== osd.5 v2:172.21.15.118:6810/2724891770 5 ==== osd_op_reply(36 1.00000000.inode [omap-set-header] v114'642 uv642 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f466405f030 con 0x55895297dd30 2024-08-03T20:58:28.554 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:28.553+0000 7f467b20df40 1 -- 172.21.15.32:0/4009968897 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:37 12.14 12:292cf221:::200.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e114) -- 0x558952981510 con 0x55895297dd30 2024-08-03T20:58:28.556 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:28.555+0000 7f4673800640 1 -- 172.21.15.32:0/4009968897 <== osd.5 v2:172.21.15.118:6810/2724891770 6 ==== osd_op_reply(37 200.00000000 [create] v114'643 uv626 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f466405f030 con 0x55895297dd30 2024-08-03T20:58:28.556 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:28.555+0000 7f467b20df40 1 -- 172.21.15.32:0/4009968897 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:38 12.14 12:292cf221:::200.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e114) -- 0x5589529818b0 con 0x55895297dd30 2024-08-03T20:58:28.558 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:28.557+0000 7f4673800640 1 -- 172.21.15.32:0/4009968897 <== osd.5 v2:172.21.15.118:6810/2724891770 7 ==== osd_op_reply(38 200.00000000 [delete] v114'644 uv644 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f466405f030 con 0x55895297dd30 2024-08-03T20:58:28.558 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:28.557+0000 7f467b20df40 1 -- 172.21.15.32:0/4009968897 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:39 12.14 12:292cf221:::200.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e114) -- 0x5589529818b0 con 0x55895297dd30 2024-08-03T20:58:28.560 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::200.00000000:head# 2024-08-03T20:58:28.560 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:28.558+0000 7f4673800640 1 -- 172.21.15.32:0/4009968897 <== osd.5 v2:172.21.15.118:6810/2724891770 8 ==== osd_op_reply(39 200.00000000 [create] v114'645 uv645 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f466405f030 con 0x55895297dd30 2024-08-03T20:58:28.560 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:28.558+0000 7f467b20df40 1 -- 172.21.15.32:0/4009968897 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:40 12.14 12:292cf221:::200.00000000:head [write 0~90 in=90b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e114) -- 0x5589529818b0 con 0x55895297dd30 2024-08-03T20:58:28.562 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:28.560+0000 7f4673800640 1 -- 172.21.15.32:0/4009968897 <== osd.5 v2:172.21.15.118:6810/2724891770 9 ==== osd_op_reply(40 200.00000000 [write 0~90] v114'646 uv646 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f466405f030 con 0x55895297dd30 2024-08-03T20:58:28.562 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:28.560+0000 7f467b20df40 1 -- 172.21.15.32:0/4009968897 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:41 12.14 12:292cf221:::200.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e114) -- 0x5589529818b0 con 0x55895297dd30 2024-08-03T20:58:28.563 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:28.562+0000 7f4673800640 1 -- 172.21.15.32:0/4009968897 <== osd.5 v2:172.21.15.118:6810/2724891770 10 ==== osd_op_reply(41 200.00000000 [omap-set-header] v114'647 uv647 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f466405f030 con 0x55895297dd30 2024-08-03T20:58:28.564 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:28.562+0000 7f467b20df40 1 -- 172.21.15.32:0/4009968897 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:42 12.14 12:2e2fa760:::200.00000001:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e114) -- 0x5589529818b0 con 0x55895297dd30 2024-08-03T20:58:28.565 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:28.564+0000 7f4673800640 1 -- 172.21.15.32:0/4009968897 <== osd.5 v2:172.21.15.118:6810/2724891770 11 ==== osd_op_reply(42 200.00000001 [create] v114'648 uv631 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f466405f030 con 0x55895297dd30 2024-08-03T20:58:28.565 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:28.564+0000 7f467b20df40 1 -- 172.21.15.32:0/4009968897 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:43 12.14 12:2e2fa760:::200.00000001:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e114) -- 0x5589529818b0 con 0x55895297dd30 2024-08-03T20:58:28.567 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:28.566+0000 7f4673800640 1 -- 172.21.15.32:0/4009968897 <== osd.5 v2:172.21.15.118:6810/2724891770 12 ==== osd_op_reply(43 200.00000001 [delete] v114'649 uv649 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f466405f030 con 0x55895297dd30 2024-08-03T20:58:28.567 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:28.566+0000 7f467b20df40 1 -- 172.21.15.32:0/4009968897 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:44 12.14 12:2e2fa760:::200.00000001:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e114) -- 0x558952981930 con 0x55895297dd30 2024-08-03T20:58:28.568 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:28.567+0000 7f4673800640 1 -- 172.21.15.32:0/4009968897 <== osd.5 v2:172.21.15.118:6810/2724891770 13 ==== osd_op_reply(44 200.00000001 [create] v114'650 uv650 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f466405f030 con 0x55895297dd30 2024-08-03T20:58:28.569 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::200.00000001:head# 2024-08-03T20:58:28.569 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:28.567+0000 7f467b20df40 1 -- 172.21.15.32:0/4009968897 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:45 12.14 12:2e2fa760:::200.00000001:head [write 0~18118 in=18118b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e114) -- 0x558952986020 con 0x55895297dd30 2024-08-03T20:58:28.570 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:28.569+0000 7f4673800640 1 -- 172.21.15.32:0/4009968897 <== osd.5 v2:172.21.15.118:6810/2724891770 14 ==== osd_op_reply(45 200.00000001 [write 0~18118] v114'651 uv651 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f466405f030 con 0x55895297dd30 2024-08-03T20:58:28.570 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:28.569+0000 7f467b20df40 1 -- 172.21.15.32:0/4009968897 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:46 12.14 12:2e2fa760:::200.00000001:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e114) -- 0x558952986020 con 0x55895297dd30 2024-08-03T20:58:28.572 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:28.571+0000 7f4673800640 1 -- 172.21.15.32:0/4009968897 <== osd.5 v2:172.21.15.118:6810/2724891770 15 ==== osd_op_reply(46 200.00000001 [omap-set-header] v114'652 uv652 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f466405f030 con 0x55895297dd30 2024-08-03T20:58:28.572 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:28.571+0000 7f467b20df40 1 -- 172.21.15.32:0/4009968897 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:47 12.14 12:2f55791f:::606.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e114) -- 0x558952981930 con 0x55895297dd30 2024-08-03T20:58:28.574 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:28.572+0000 7f4673800640 1 -- 172.21.15.32:0/4009968897 <== osd.5 v2:172.21.15.118:6810/2724891770 16 ==== osd_op_reply(47 606.00000000 [create] v114'653 uv637 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f466405f030 con 0x55895297dd30 2024-08-03T20:58:28.574 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:28.572+0000 7f467b20df40 1 -- 172.21.15.32:0/4009968897 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:48 12.14 12:2f55791f:::606.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e114) -- 0x558952981930 con 0x55895297dd30 2024-08-03T20:58:28.575 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:28.574+0000 7f4673800640 1 -- 172.21.15.32:0/4009968897 <== osd.5 v2:172.21.15.118:6810/2724891770 17 ==== osd_op_reply(48 606.00000000 [delete] v114'654 uv654 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f466405f030 con 0x55895297dd30 2024-08-03T20:58:28.575 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:28.574+0000 7f467b20df40 1 -- 172.21.15.32:0/4009968897 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:49 12.14 12:2f55791f:::606.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e114) -- 0x558952981930 con 0x55895297dd30 2024-08-03T20:58:28.577 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:28.576+0000 7f4673800640 1 -- 172.21.15.32:0/4009968897 <== osd.5 v2:172.21.15.118:6810/2724891770 18 ==== osd_op_reply(49 606.00000000 [create] v114'655 uv655 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f466405f030 con 0x55895297dd30 2024-08-03T20:58:28.577 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::606.00000000:head# 2024-08-03T20:58:28.577 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:28.576+0000 7f467b20df40 1 -- 172.21.15.32:0/4009968897 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:50 12.14 12:2f55791f:::606.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e114) -- 0x558952982980 con 0x55895297dd30 2024-08-03T20:58:28.578 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:28.577+0000 7f4673800640 1 -- 172.21.15.32:0/4009968897 <== osd.5 v2:172.21.15.118:6810/2724891770 19 ==== osd_op_reply(50 606.00000000 [setxattr (30)] v114'656 uv656 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f466405f030 con 0x55895297dd30 2024-08-03T20:58:28.579 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:28.577+0000 7f467b20df40 1 -- 172.21.15.32:0/4009968897 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:51 12.14 12:2f55791f:::606.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e114) -- 0x558952982940 con 0x55895297dd30 2024-08-03T20:58:28.581 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:28.579+0000 7f4673800640 1 -- 172.21.15.32:0/4009968897 <== osd.5 v2:172.21.15.118:6810/2724891770 20 ==== osd_op_reply(51 606.00000000 [setxattr (62)] v114'657 uv657 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f466405f030 con 0x55895297dd30 2024-08-03T20:58:28.581 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:28.579+0000 7f467b20df40 1 -- 172.21.15.32:0/4009968897 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:52 12.14 12:2f55791f:::606.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e114) -- 0x558952981930 con 0x55895297dd30 2024-08-03T20:58:28.582 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:28.581+0000 7f4673800640 1 -- 172.21.15.32:0/4009968897 <== osd.5 v2:172.21.15.118:6810/2724891770 21 ==== osd_op_reply(52 606.00000000 [omap-set-header] v114'658 uv658 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f466405f030 con 0x55895297dd30 2024-08-03T20:58:28.582 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:28.581+0000 7f467b20df40 1 -- 172.21.15.32:0/4009968897 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:53 12.1c 12:3ed09add:::607.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e114) -- 0x558952981930 con 0x5589528b3de0 2024-08-03T20:58:28.584 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:28.582+0000 7f4672e00640 1 -- 172.21.15.32:0/4009968897 <== osd.0 v2:172.21.15.32:6816/3066156116 32 ==== osd_op_reply(53 607.00000000 [create] v114'171 uv170 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f466000c040 con 0x5589528b3de0 2024-08-03T20:58:28.584 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:28.582+0000 7f467b20df40 1 -- 172.21.15.32:0/4009968897 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:54 12.1c 12:3ed09add:::607.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e114) -- 0x558952981930 con 0x5589528b3de0 2024-08-03T20:58:28.585 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:28.584+0000 7f4672e00640 1 -- 172.21.15.32:0/4009968897 <== osd.0 v2:172.21.15.32:6816/3066156116 33 ==== osd_op_reply(54 607.00000000 [delete] v114'172 uv172 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f466000c040 con 0x5589528b3de0 2024-08-03T20:58:28.585 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:28.584+0000 7f467b20df40 1 -- 172.21.15.32:0/4009968897 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:55 12.1c 12:3ed09add:::607.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e114) -- 0x558952981930 con 0x5589528b3de0 2024-08-03T20:58:28.586 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::607.00000000:head# 2024-08-03T20:58:28.587 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:28.585+0000 7f4672e00640 1 -- 172.21.15.32:0/4009968897 <== osd.0 v2:172.21.15.32:6816/3066156116 34 ==== osd_op_reply(55 607.00000000 [create] v114'173 uv173 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f466000c040 con 0x5589528b3de0 2024-08-03T20:58:28.587 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:28.585+0000 7f467b20df40 1 -- 172.21.15.32:0/4009968897 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:56 12.1c 12:3ed09add:::607.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e114) -- 0x558952981930 con 0x5589528b3de0 2024-08-03T20:58:28.588 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:28.586+0000 7f4672e00640 1 -- 172.21.15.32:0/4009968897 <== osd.0 v2:172.21.15.32:6816/3066156116 35 ==== osd_op_reply(56 607.00000000 [setxattr (30)] v114'174 uv174 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f466000c040 con 0x5589528b3de0 2024-08-03T20:58:28.588 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:28.586+0000 7f467b20df40 1 -- 172.21.15.32:0/4009968897 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:57 12.1c 12:3ed09add:::607.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e114) -- 0x558952982940 con 0x5589528b3de0 2024-08-03T20:58:28.589 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:28.588+0000 7f4672e00640 1 -- 172.21.15.32:0/4009968897 <== osd.0 v2:172.21.15.32:6816/3066156116 36 ==== osd_op_reply(57 607.00000000 [setxattr (62)] v114'175 uv175 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f466000c040 con 0x5589528b3de0 2024-08-03T20:58:28.589 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:28.588+0000 7f467b20df40 1 -- 172.21.15.32:0/4009968897 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:58 12.1c 12:3ed09add:::607.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e114) -- 0x558952981930 con 0x5589528b3de0 2024-08-03T20:58:28.591 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:28.589+0000 7f4672e00640 1 -- 172.21.15.32:0/4009968897 <== osd.0 v2:172.21.15.32:6816/3066156116 37 ==== osd_op_reply(58 607.00000000 [omap-set-header] v114'176 uv176 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f466000c040 con 0x5589528b3de0 2024-08-03T20:58:28.591 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:28.589+0000 7f467b20df40 1 --2- 172.21.15.32:0/4009968897 >> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] conn(0x558952981fb0 0x558952982460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:58:28.592 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:28.589+0000 7f467b20df40 1 -- 172.21.15.32:0/4009968897 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:59 12.6 12:654134d2:::mds0_openfiles.0:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e114) -- 0x558952984d90 con 0x558952981fb0 2024-08-03T20:58:28.592 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:28.589+0000 7f4672e00640 1 --2- 172.21.15.32:0/4009968897 >> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] conn(0x558952981fb0 0x558952982460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:58:28.592 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:28.590+0000 7f4672e00640 1 --2- 172.21.15.32:0/4009968897 >> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] conn(0x558952981fb0 0x558952982460 crc :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:58:28.593 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:28.591+0000 7f4672e00640 1 -- 172.21.15.32:0/4009968897 <== osd.1 v2:172.21.15.32:6824/1551241759 1 ==== osd_op_reply(59 mds0_openfiles.0 [create] v114'133 uv132 ondisk = -17 ((17) File exists)) ==== 160+0+0 (crc 0 0 0) 0x7f466000c040 con 0x558952981fb0 2024-08-03T20:58:28.593 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:28.591+0000 7f467b20df40 1 -- 172.21.15.32:0/4009968897 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:60 12.6 12:654134d2:::mds0_openfiles.0:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e114) -- 0x558952984d90 con 0x558952981fb0 2024-08-03T20:58:28.594 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:28.593+0000 7f4672e00640 1 -- 172.21.15.32:0/4009968897 <== osd.1 v2:172.21.15.32:6824/1551241759 2 ==== osd_op_reply(60 mds0_openfiles.0 [delete] v114'134 uv134 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f466000c040 con 0x558952981fb0 2024-08-03T20:58:28.594 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:28.593+0000 7f467b20df40 1 -- 172.21.15.32:0/4009968897 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:61 12.6 12:654134d2:::mds0_openfiles.0:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e114) -- 0x558952984e10 con 0x558952981fb0 2024-08-03T20:58:28.595 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:28.594+0000 7f4672e00640 1 -- 172.21.15.32:0/4009968897 <== osd.1 v2:172.21.15.32:6824/1551241759 3 ==== osd_op_reply(61 mds0_openfiles.0 [create] v114'135 uv135 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f466000c040 con 0x558952981fb0 2024-08-03T20:58:28.596 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds0_openfiles.0:head# 2024-08-03T20:58:28.596 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:28.594+0000 7f467b20df40 1 -- 172.21.15.32:0/4009968897 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:62 12.6 12:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e114) -- 0x558952984e70 con 0x558952981fb0 2024-08-03T20:58:28.597 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:28.595+0000 7f4672e00640 1 -- 172.21.15.32:0/4009968897 <== osd.1 v2:172.21.15.32:6824/1551241759 4 ==== osd_op_reply(62 mds0_openfiles.0 [omap-set-header] v114'136 uv136 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f466000c040 con 0x558952981fb0 2024-08-03T20:58:28.597 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:28.595+0000 7f467b20df40 1 -- 172.21.15.32:0/4009968897 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:63 12.e 12:717a0223:::608.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e114) -- 0x558952984e70 con 0x5589528b3de0 2024-08-03T20:58:28.598 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:28.597+0000 7f4672e00640 1 -- 172.21.15.32:0/4009968897 <== osd.0 v2:172.21.15.32:6816/3066156116 38 ==== osd_op_reply(63 608.00000000 [create] v114'171 uv170 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f466000c040 con 0x5589528b3de0 2024-08-03T20:58:28.599 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:28.597+0000 7f467b20df40 1 -- 172.21.15.32:0/4009968897 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:64 12.e 12:717a0223:::608.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e114) -- 0x558952984e70 con 0x5589528b3de0 2024-08-03T20:58:28.600 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:28.599+0000 7f4672e00640 1 -- 172.21.15.32:0/4009968897 <== osd.0 v2:172.21.15.32:6816/3066156116 39 ==== osd_op_reply(64 608.00000000 [delete] v114'172 uv172 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f466000c040 con 0x5589528b3de0 2024-08-03T20:58:28.600 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:28.599+0000 7f467b20df40 1 -- 172.21.15.32:0/4009968897 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:65 12.e 12:717a0223:::608.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e114) -- 0x558952984e50 con 0x5589528b3de0 2024-08-03T20:58:28.601 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:28.600+0000 7f4672e00640 1 -- 172.21.15.32:0/4009968897 <== osd.0 v2:172.21.15.32:6816/3066156116 40 ==== osd_op_reply(65 608.00000000 [create] v114'173 uv173 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f466000c040 con 0x5589528b3de0 2024-08-03T20:58:28.602 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::608.00000000:head# 2024-08-03T20:58:28.602 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:28.600+0000 7f467b20df40 1 -- 172.21.15.32:0/4009968897 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:66 12.e 12:717a0223:::608.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e114) -- 0x558952985e80 con 0x5589528b3de0 2024-08-03T20:58:28.603 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:28.601+0000 7f4672e00640 1 -- 172.21.15.32:0/4009968897 <== osd.0 v2:172.21.15.32:6816/3066156116 41 ==== osd_op_reply(66 608.00000000 [setxattr (30)] v114'174 uv174 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f466000c040 con 0x5589528b3de0 2024-08-03T20:58:28.603 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:28.602+0000 7f467b20df40 1 -- 172.21.15.32:0/4009968897 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:67 12.e 12:717a0223:::608.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e114) -- 0x558952985e40 con 0x5589528b3de0 2024-08-03T20:58:28.604 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:28.603+0000 7f4672e00640 1 -- 172.21.15.32:0/4009968897 <== osd.0 v2:172.21.15.32:6816/3066156116 42 ==== osd_op_reply(67 608.00000000 [setxattr (62)] v114'175 uv175 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f466000c040 con 0x5589528b3de0 2024-08-03T20:58:28.604 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:28.603+0000 7f467b20df40 1 -- 172.21.15.32:0/4009968897 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:68 12.e 12:717a0223:::608.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e114) -- 0x558952985e40 con 0x5589528b3de0 2024-08-03T20:58:28.606 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:28.604+0000 7f4672e00640 1 -- 172.21.15.32:0/4009968897 <== osd.0 v2:172.21.15.32:6816/3066156116 43 ==== osd_op_reply(68 608.00000000 [omap-set-header] v114'176 uv176 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f466000c040 con 0x5589528b3de0 2024-08-03T20:58:28.606 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:28.604+0000 7f467b20df40 1 --2- 172.21.15.32:0/4009968897 >> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] conn(0x5589529852a0 0x5589529856b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:58:28.606 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:28.604+0000 7f467b20df40 1 -- 172.21.15.32:0/4009968897 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:69 12.15 12:a93a17c2:::604.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e114) -- 0x558952988230 con 0x5589529852a0 2024-08-03T20:58:28.606 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:28.605+0000 7f4674200640 1 --2- 172.21.15.32:0/4009968897 >> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] conn(0x5589529852a0 0x5589529856b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:58:28.607 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:28.606+0000 7f4674200640 1 --2- 172.21.15.32:0/4009968897 >> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] conn(0x5589529852a0 0x5589529856b0 crc :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:58:28.609 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:28.607+0000 7f4674200640 1 -- 172.21.15.32:0/4009968897 <== osd.6 v2:172.21.15.118:6825/801947386 1 ==== osd_op_reply(69 604.00000000 [create] v114'316 uv310 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f466c05c390 con 0x5589529852a0 2024-08-03T20:58:28.609 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:28.607+0000 7f467b20df40 1 -- 172.21.15.32:0/4009968897 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:70 12.15 12:a93a17c2:::604.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e114) -- 0x5589529885d0 con 0x5589529852a0 2024-08-03T20:58:28.610 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:28.609+0000 7f4674200640 1 -- 172.21.15.32:0/4009968897 <== osd.6 v2:172.21.15.118:6825/801947386 2 ==== osd_op_reply(70 604.00000000 [delete] v114'317 uv317 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f466c05c390 con 0x5589529852a0 2024-08-03T20:58:28.611 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:28.609+0000 7f467b20df40 1 -- 172.21.15.32:0/4009968897 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:71 12.15 12:a93a17c2:::604.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e114) -- 0x5589529889b0 con 0x5589529852a0 2024-08-03T20:58:28.612 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::604.00000000:head# 2024-08-03T20:58:28.612 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:28.610+0000 7f4674200640 1 -- 172.21.15.32:0/4009968897 <== osd.6 v2:172.21.15.118:6825/801947386 3 ==== osd_op_reply(71 604.00000000 [create] v114'318 uv318 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f466c05c390 con 0x5589529852a0 2024-08-03T20:58:28.612 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:28.610+0000 7f467b20df40 1 -- 172.21.15.32:0/4009968897 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:72 12.15 12:a93a17c2:::604.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e114) -- 0x5589528b6f40 con 0x5589529852a0 2024-08-03T20:58:28.613 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:28.612+0000 7f4674200640 1 -- 172.21.15.32:0/4009968897 <== osd.6 v2:172.21.15.118:6825/801947386 4 ==== osd_op_reply(72 604.00000000 [setxattr (30)] v114'319 uv319 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f466c05c390 con 0x5589529852a0 2024-08-03T20:58:28.613 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:28.612+0000 7f467b20df40 1 -- 172.21.15.32:0/4009968897 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:73 12.15 12:a93a17c2:::604.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e114) -- 0x55895298a340 con 0x5589529852a0 2024-08-03T20:58:28.615 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:28.613+0000 7f4674200640 1 -- 172.21.15.32:0/4009968897 <== osd.6 v2:172.21.15.118:6825/801947386 5 ==== osd_op_reply(73 604.00000000 [setxattr (62)] v114'320 uv320 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f466c05c390 con 0x5589529852a0 2024-08-03T20:58:28.615 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:28.613+0000 7f467b20df40 1 -- 172.21.15.32:0/4009968897 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:74 12.15 12:a93a17c2:::604.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e114) -- 0x558952989250 con 0x5589529852a0 2024-08-03T20:58:28.616 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:28.615+0000 7f4674200640 1 -- 172.21.15.32:0/4009968897 <== osd.6 v2:172.21.15.118:6825/801947386 6 ==== osd_op_reply(74 604.00000000 [omap-set-header] v114'321 uv321 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f466c05c390 con 0x5589529852a0 2024-08-03T20:58:28.616 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:28.615+0000 7f467b20df40 1 -- 172.21.15.32:0/4009968897 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:75 12.15 12:aa448500:::500.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e114) -- 0x558952989640 con 0x5589529852a0 2024-08-03T20:58:28.618 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:28.616+0000 7f4674200640 1 -- 172.21.15.32:0/4009968897 <== osd.6 v2:172.21.15.118:6825/801947386 7 ==== osd_op_reply(75 500.00000000 [create] v114'322 uv315 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f466c05c390 con 0x5589529852a0 2024-08-03T20:58:28.618 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:28.616+0000 7f467b20df40 1 -- 172.21.15.32:0/4009968897 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:76 12.15 12:aa448500:::500.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e114) -- 0x558952989a30 con 0x5589529852a0 2024-08-03T20:58:28.619 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:28.618+0000 7f4674200640 1 -- 172.21.15.32:0/4009968897 <== osd.6 v2:172.21.15.118:6825/801947386 8 ==== osd_op_reply(76 500.00000000 [delete] v114'323 uv323 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f466c05c390 con 0x5589529852a0 2024-08-03T20:58:28.619 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:28.618+0000 7f467b20df40 1 -- 172.21.15.32:0/4009968897 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:77 12.15 12:aa448500:::500.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e114) -- 0x558952989e20 con 0x5589529852a0 2024-08-03T20:58:28.621 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::500.00000000:head# 2024-08-03T20:58:28.621 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:28.619+0000 7f4674200640 1 -- 172.21.15.32:0/4009968897 <== osd.6 v2:172.21.15.118:6825/801947386 9 ==== osd_op_reply(77 500.00000000 [create] v114'324 uv324 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f466c05c390 con 0x5589529852a0 2024-08-03T20:58:28.621 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:28.619+0000 7f467b20df40 1 -- 172.21.15.32:0/4009968897 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:78 12.15 12:aa448500:::500.00000000:head [write 0~90 in=90b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e114) -- 0x558952989e20 con 0x5589529852a0 2024-08-03T20:58:28.622 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:28.621+0000 7f4674200640 1 -- 172.21.15.32:0/4009968897 <== osd.6 v2:172.21.15.118:6825/801947386 10 ==== osd_op_reply(78 500.00000000 [write 0~90] v114'325 uv325 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f466c05c390 con 0x5589529852a0 2024-08-03T20:58:28.622 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:28.621+0000 7f467b20df40 1 -- 172.21.15.32:0/4009968897 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:79 12.15 12:aa448500:::500.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e114) -- 0x558952989dd0 con 0x5589529852a0 2024-08-03T20:58:28.624 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:28.622+0000 7f4674200640 1 -- 172.21.15.32:0/4009968897 <== osd.6 v2:172.21.15.118:6825/801947386 11 ==== osd_op_reply(79 500.00000000 [omap-set-header] v114'326 uv326 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f466c05c390 con 0x5589529852a0 2024-08-03T20:58:28.624 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:28.622+0000 7f467b20df40 1 --2- 172.21.15.32:0/4009968897 >> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] conn(0x55895298ac80 0x55895298d0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:58:28.624 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:28.622+0000 7f467b20df40 1 -- 172.21.15.32:0/4009968897 --> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] -- osd_op(unknown.0.0:80 12.d 12:b50e409b:::mds_snaptable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e114) -- 0x55895298d5e0 con 0x55895298ac80 2024-08-03T20:58:28.624 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:28.623+0000 7f4673800640 1 --2- 172.21.15.32:0/4009968897 >> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] conn(0x55895298ac80 0x55895298d0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:58:28.625 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:28.623+0000 7f4673800640 1 --2- 172.21.15.32:0/4009968897 >> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] conn(0x55895298ac80 0x55895298d0a0 crc :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:58:28.626 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:28.625+0000 7f4673800640 1 -- 172.21.15.32:0/4009968897 <== osd.7 v2:172.21.15.118:6818/653000048 1 ==== osd_op_reply(80 mds_snaptable [create] v114'143 uv142 ondisk = -17 ((17) File exists)) ==== 157+0+0 (crc 0 0 0) 0x7f466405f030 con 0x55895298ac80 2024-08-03T20:58:28.627 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:28.625+0000 7f467b20df40 1 -- 172.21.15.32:0/4009968897 --> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] -- osd_op(unknown.0.0:81 12.d 12:b50e409b:::mds_snaptable:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e114) -- 0x55895298d620 con 0x55895298ac80 2024-08-03T20:58:28.628 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:28.627+0000 7f4673800640 1 -- 172.21.15.32:0/4009968897 <== osd.7 v2:172.21.15.118:6818/653000048 2 ==== osd_op_reply(81 mds_snaptable [delete] v114'144 uv144 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f466405f030 con 0x55895298ac80 2024-08-03T20:58:28.628 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:28.627+0000 7f467b20df40 1 -- 172.21.15.32:0/4009968897 --> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] -- osd_op(unknown.0.0:82 12.d 12:b50e409b:::mds_snaptable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e114) -- 0x55895298da40 con 0x55895298ac80 2024-08-03T20:58:28.629 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:28.628+0000 7f4673800640 1 -- 172.21.15.32:0/4009968897 <== osd.7 v2:172.21.15.118:6818/653000048 3 ==== osd_op_reply(82 mds_snaptable [create] v114'145 uv145 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f466405f030 con 0x55895298ac80 2024-08-03T20:58:28.630 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds_snaptable:head# 2024-08-03T20:58:28.630 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:28.628+0000 7f467b20df40 1 -- 172.21.15.32:0/4009968897 --> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] -- osd_op(unknown.0.0:83 12.d 12:b50e409b:::mds_snaptable:head [write 0~70 in=70b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e114) -- 0x55895298da40 con 0x55895298ac80 2024-08-03T20:58:28.631 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:28.630+0000 7f4673800640 1 -- 172.21.15.32:0/4009968897 <== osd.7 v2:172.21.15.118:6818/653000048 4 ==== osd_op_reply(83 mds_snaptable [write 0~70] v114'146 uv146 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f466405f030 con 0x55895298ac80 2024-08-03T20:58:28.631 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:28.630+0000 7f467b20df40 1 -- 172.21.15.32:0/4009968897 --> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] -- osd_op(unknown.0.0:84 12.d 12:b50e409b:::mds_snaptable:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e114) -- 0x55895298da40 con 0x55895298ac80 2024-08-03T20:58:28.633 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:28.631+0000 7f4673800640 1 -- 172.21.15.32:0/4009968897 <== osd.7 v2:172.21.15.118:6818/653000048 5 ==== osd_op_reply(84 mds_snaptable [omap-set-header] v114'147 uv147 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f466405f030 con 0x55895298ac80 2024-08-03T20:58:28.633 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:28.631+0000 7f467b20df40 1 -- 172.21.15.32:0/4009968897 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:85 12.1d 12:b871830b:::605.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e114) -- 0x55895298da40 con 0x5589528b3de0 2024-08-03T20:58:28.634 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:28.633+0000 7f4672e00640 1 -- 172.21.15.32:0/4009968897 <== osd.0 v2:172.21.15.32:6816/3066156116 44 ==== osd_op_reply(85 605.00000000 [create] v114'171 uv170 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f466000c040 con 0x5589528b3de0 2024-08-03T20:58:28.634 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:28.633+0000 7f467b20df40 1 -- 172.21.15.32:0/4009968897 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:86 12.1d 12:b871830b:::605.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e114) -- 0x55895298da40 con 0x5589528b3de0 2024-08-03T20:58:28.635 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:28.634+0000 7f4672e00640 1 -- 172.21.15.32:0/4009968897 <== osd.0 v2:172.21.15.32:6816/3066156116 45 ==== osd_op_reply(86 605.00000000 [delete] v114'172 uv172 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f466000c040 con 0x5589528b3de0 2024-08-03T20:58:28.635 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:28.634+0000 7f467b20df40 1 -- 172.21.15.32:0/4009968897 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:87 12.1d 12:b871830b:::605.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e114) -- 0x55895298da40 con 0x5589528b3de0 2024-08-03T20:58:28.636 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:28.635+0000 7f4672e00640 1 -- 172.21.15.32:0/4009968897 <== osd.0 v2:172.21.15.32:6816/3066156116 46 ==== osd_op_reply(87 605.00000000 [create] v114'173 uv173 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f466000c040 con 0x5589528b3de0 2024-08-03T20:58:28.637 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::605.00000000:head# 2024-08-03T20:58:28.637 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:28.635+0000 7f467b20df40 1 -- 172.21.15.32:0/4009968897 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:88 12.1d 12:b871830b:::605.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e114) -- 0x55895298ea50 con 0x5589528b3de0 2024-08-03T20:58:28.638 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:28.636+0000 7f4672e00640 1 -- 172.21.15.32:0/4009968897 <== osd.0 v2:172.21.15.32:6816/3066156116 47 ==== osd_op_reply(88 605.00000000 [setxattr (30)] v114'174 uv174 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f466000c040 con 0x5589528b3de0 2024-08-03T20:58:28.638 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:28.637+0000 7f467b20df40 1 -- 172.21.15.32:0/4009968897 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:89 12.1d 12:b871830b:::605.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e114) -- 0x55895298ea50 con 0x5589528b3de0 2024-08-03T20:58:28.639 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:28.638+0000 7f4672e00640 1 -- 172.21.15.32:0/4009968897 <== osd.0 v2:172.21.15.32:6816/3066156116 48 ==== osd_op_reply(89 605.00000000 [setxattr (62)] v114'175 uv175 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f466000c040 con 0x5589528b3de0 2024-08-03T20:58:28.639 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:28.638+0000 7f467b20df40 1 -- 172.21.15.32:0/4009968897 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:90 12.1d 12:b871830b:::605.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e114) -- 0x55895298da40 con 0x5589528b3de0 2024-08-03T20:58:28.640 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:28.639+0000 7f4672e00640 1 -- 172.21.15.32:0/4009968897 <== osd.0 v2:172.21.15.32:6816/3066156116 49 ==== osd_op_reply(90 605.00000000 [omap-set-header] v114'176 uv176 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f466000c040 con 0x5589528b3de0 2024-08-03T20:58:28.640 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:28.639+0000 7f467b20df40 1 -- 172.21.15.32:0/4009968897 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:91 12.13 12:c91d4a1d:::mds0_inotable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e114) -- 0x55895298da40 con 0x558952981fb0 2024-08-03T20:58:28.642 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:28.641+0000 7f4672e00640 1 -- 172.21.15.32:0/4009968897 <== osd.1 v2:172.21.15.32:6824/1551241759 5 ==== osd_op_reply(91 mds0_inotable [create] v114'284 uv283 ondisk = -17 ((17) File exists)) ==== 157+0+0 (crc 0 0 0) 0x7f466000c040 con 0x558952981fb0 2024-08-03T20:58:28.642 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:28.641+0000 7f467b20df40 1 -- 172.21.15.32:0/4009968897 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:92 12.13 12:c91d4a1d:::mds0_inotable:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e114) -- 0x55895298da40 con 0x558952981fb0 2024-08-03T20:58:28.643 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:28.642+0000 7f4672e00640 1 -- 172.21.15.32:0/4009968897 <== osd.1 v2:172.21.15.32:6824/1551241759 6 ==== osd_op_reply(92 mds0_inotable [delete] v114'285 uv285 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f466000c040 con 0x558952981fb0 2024-08-03T20:58:28.643 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:28.642+0000 7f467b20df40 1 -- 172.21.15.32:0/4009968897 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:93 12.13 12:c91d4a1d:::mds0_inotable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e114) -- 0x55895298da40 con 0x558952981fb0 2024-08-03T20:58:28.645 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds0_inotable:head# 2024-08-03T20:58:28.645 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:28.643+0000 7f4672e00640 1 -- 172.21.15.32:0/4009968897 <== osd.1 v2:172.21.15.32:6824/1551241759 7 ==== osd_op_reply(93 mds0_inotable [create] v114'286 uv286 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f466000c040 con 0x558952981fb0 2024-08-03T20:58:28.645 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:28.643+0000 7f467b20df40 1 -- 172.21.15.32:0/4009968897 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:94 12.13 12:c91d4a1d:::mds0_inotable:head [write 0~34 in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e114) -- 0x55895298de50 con 0x558952981fb0 2024-08-03T20:58:28.646 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:28.645+0000 7f4672e00640 1 -- 172.21.15.32:0/4009968897 <== osd.1 v2:172.21.15.32:6824/1551241759 8 ==== osd_op_reply(94 mds0_inotable [write 0~34] v114'287 uv287 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f466000c040 con 0x558952981fb0 2024-08-03T20:58:28.646 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:28.645+0000 7f467b20df40 1 -- 172.21.15.32:0/4009968897 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:95 12.13 12:c91d4a1d:::mds0_inotable:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e114) -- 0x55895298de00 con 0x558952981fb0 2024-08-03T20:58:28.648 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:28.646+0000 7f4672e00640 1 -- 172.21.15.32:0/4009968897 <== osd.1 v2:172.21.15.32:6824/1551241759 9 ==== osd_op_reply(95 mds0_inotable [omap-set-header] v114'288 uv288 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f466000c040 con 0x558952981fb0 2024-08-03T20:58:28.648 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:28.646+0000 7f467b20df40 1 -- 172.21.15.32:0/4009968897 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:96 12.13 12:cd5a64a3:::100.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e114) -- 0x55895298de00 con 0x558952981fb0 2024-08-03T20:58:28.649 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:28.648+0000 7f4672e00640 1 -- 172.21.15.32:0/4009968897 <== osd.1 v2:172.21.15.32:6824/1551241759 10 ==== osd_op_reply(96 100.00000000 [create] v114'289 uv282 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f466000c040 con 0x558952981fb0 2024-08-03T20:58:28.649 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:28.648+0000 7f467b20df40 1 -- 172.21.15.32:0/4009968897 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:97 12.13 12:cd5a64a3:::100.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e114) -- 0x55895298de00 con 0x558952981fb0 2024-08-03T20:58:28.651 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:28.649+0000 7f4672e00640 1 -- 172.21.15.32:0/4009968897 <== osd.1 v2:172.21.15.32:6824/1551241759 11 ==== osd_op_reply(97 100.00000000 [delete] v114'290 uv290 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f466000c040 con 0x558952981fb0 2024-08-03T20:58:28.651 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:28.649+0000 7f467b20df40 1 -- 172.21.15.32:0/4009968897 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:98 12.13 12:cd5a64a3:::100.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e114) -- 0x55895298de00 con 0x558952981fb0 2024-08-03T20:58:28.652 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::100.00000000:head# 2024-08-03T20:58:28.652 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:28.651+0000 7f4672e00640 1 -- 172.21.15.32:0/4009968897 <== osd.1 v2:172.21.15.32:6824/1551241759 12 ==== osd_op_reply(98 100.00000000 [create] v114'291 uv291 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f466000c040 con 0x558952981fb0 2024-08-03T20:58:28.653 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:28.651+0000 7f467b20df40 1 -- 172.21.15.32:0/4009968897 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:99 12.13 12:cd5a64a3:::100.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e114) -- 0x55895298de00 con 0x558952981fb0 2024-08-03T20:58:28.653 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:28.652+0000 7f4672e00640 1 -- 172.21.15.32:0/4009968897 <== osd.1 v2:172.21.15.32:6824/1551241759 13 ==== osd_op_reply(99 100.00000000 [omap-set-header] v114'292 uv292 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f466000c040 con 0x558952981fb0 2024-08-03T20:58:28.654 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:28.652+0000 7f467b20df40 1 -- 172.21.15.32:0/4009968897 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:100 12.13 12:cd5a64a3:::100.00000000:head [omap-set-vals in=5194b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e114) -- 0x558952990c10 con 0x558952981fb0 2024-08-03T20:58:28.655 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:28.654+0000 7f4672e00640 1 -- 172.21.15.32:0/4009968897 <== osd.1 v2:172.21.15.32:6824/1551241759 14 ==== osd_op_reply(100 100.00000000 [omap-set-vals] v114'293 uv293 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f466000c040 con 0x558952981fb0 2024-08-03T20:58:28.655 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:28.654+0000 7f467b20df40 1 --2- 172.21.15.32:0/4009968897 >> [v2:172.21.15.32:6800/9088303,v1:172.21.15.32:6801/9088303] conn(0x55895298e770 0x558952990b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:58:28.656 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:28.654+0000 7f467b20df40 1 -- 172.21.15.32:0/4009968897 --> [v2:172.21.15.32:6800/9088303,v1:172.21.15.32:6801/9088303] -- osd_op(unknown.0.0:101 12.b 12:d0630e4c:::mds0_sessionmap:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e114) -- 0x5589529910d0 con 0x55895298e770 2024-08-03T20:58:28.656 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:28.654+0000 7f4672e00640 1 --2- 172.21.15.32:0/4009968897 >> [v2:172.21.15.32:6800/9088303,v1:172.21.15.32:6801/9088303] conn(0x55895298e770 0x558952990b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:58:28.656 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:28.655+0000 7f4672e00640 1 --2- 172.21.15.32:0/4009968897 >> [v2:172.21.15.32:6800/9088303,v1:172.21.15.32:6801/9088303] conn(0x55895298e770 0x558952990b90 crc :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:58:28.657 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:28.656+0000 7f4672e00640 1 -- 172.21.15.32:0/4009968897 <== osd.3 v2:172.21.15.32:6800/9088303 1 ==== osd_op_reply(101 mds0_sessionmap [create] v114'124 uv123 ondisk = -17 ((17) File exists)) ==== 159+0+0 (crc 0 0 0) 0x7f466000c040 con 0x55895298e770 2024-08-03T20:58:28.657 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:28.656+0000 7f467b20df40 1 -- 172.21.15.32:0/4009968897 --> [v2:172.21.15.32:6800/9088303,v1:172.21.15.32:6801/9088303] -- osd_op(unknown.0.0:102 12.b 12:d0630e4c:::mds0_sessionmap:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e114) -- 0x5589529910d0 con 0x55895298e770 2024-08-03T20:58:28.659 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:28.657+0000 7f4672e00640 1 -- 172.21.15.32:0/4009968897 <== osd.3 v2:172.21.15.32:6800/9088303 2 ==== osd_op_reply(102 mds0_sessionmap [delete] v114'125 uv125 ondisk = 0) ==== 159+0+0 (crc 0 0 0) 0x7f466000c040 con 0x55895298e770 2024-08-03T20:58:28.659 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:28.657+0000 7f467b20df40 1 -- 172.21.15.32:0/4009968897 --> [v2:172.21.15.32:6800/9088303,v1:172.21.15.32:6801/9088303] -- osd_op(unknown.0.0:103 12.b 12:d0630e4c:::mds0_sessionmap:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e114) -- 0x5589529910d0 con 0x55895298e770 2024-08-03T20:58:28.660 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:28.659+0000 7f4672e00640 1 -- 172.21.15.32:0/4009968897 <== osd.3 v2:172.21.15.32:6800/9088303 3 ==== osd_op_reply(103 mds0_sessionmap [create] v114'126 uv126 ondisk = 0) ==== 159+0+0 (crc 0 0 0) 0x7f466000c040 con 0x55895298e770 2024-08-03T20:58:28.660 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds0_sessionmap:head# 2024-08-03T20:58:28.661 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:28.659+0000 7f467b20df40 1 -- 172.21.15.32:0/4009968897 --> [v2:172.21.15.32:6800/9088303,v1:172.21.15.32:6801/9088303] -- osd_op(unknown.0.0:104 12.b 12:d0630e4c:::mds0_sessionmap:head [omap-set-header in=14b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e114) -- 0x5589529910d0 con 0x55895298e770 2024-08-03T20:58:28.661 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:28.660+0000 7f4672e00640 1 -- 172.21.15.32:0/4009968897 <== osd.3 v2:172.21.15.32:6800/9088303 4 ==== osd_op_reply(104 mds0_sessionmap [omap-set-header] v114'127 uv127 ondisk = 0) ==== 159+0+0 (crc 0 0 0) 0x7f466000c040 con 0x55895298e770 2024-08-03T20:58:28.661 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:28.660+0000 7f467b20df40 1 -- 172.21.15.32:0/4009968897 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:105 12.7 12:e0b41d06:::609.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e114) -- 0x5589529910d0 con 0x558952981fb0 2024-08-03T20:58:28.663 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:28.662+0000 7f4672e00640 1 -- 172.21.15.32:0/4009968897 <== osd.1 v2:172.21.15.32:6824/1551241759 15 ==== osd_op_reply(105 609.00000000 [create] v114'171 uv170 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f466000c040 con 0x558952981fb0 2024-08-03T20:58:28.663 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:28.662+0000 7f467b20df40 1 -- 172.21.15.32:0/4009968897 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:106 12.7 12:e0b41d06:::609.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e114) -- 0x5589529910d0 con 0x558952981fb0 2024-08-03T20:58:28.665 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:28.663+0000 7f4672e00640 1 -- 172.21.15.32:0/4009968897 <== osd.1 v2:172.21.15.32:6824/1551241759 16 ==== osd_op_reply(106 609.00000000 [delete] v114'172 uv172 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f466000c040 con 0x558952981fb0 2024-08-03T20:58:28.665 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:28.663+0000 7f467b20df40 1 -- 172.21.15.32:0/4009968897 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:107 12.7 12:e0b41d06:::609.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e114) -- 0x5589529910d0 con 0x558952981fb0 2024-08-03T20:58:28.666 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::609.00000000:head# 2024-08-03T20:58:28.666 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:28.665+0000 7f4672e00640 1 -- 172.21.15.32:0/4009968897 <== osd.1 v2:172.21.15.32:6824/1551241759 17 ==== osd_op_reply(107 609.00000000 [create] v114'173 uv173 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f466000c040 con 0x558952981fb0 2024-08-03T20:58:28.666 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:28.665+0000 7f467b20df40 1 -- 172.21.15.32:0/4009968897 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:108 12.7 12:e0b41d06:::609.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e114) -- 0x5589529920e0 con 0x558952981fb0 2024-08-03T20:58:28.667 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:28.666+0000 7f4672e00640 1 -- 172.21.15.32:0/4009968897 <== osd.1 v2:172.21.15.32:6824/1551241759 18 ==== osd_op_reply(108 609.00000000 [setxattr (30)] v114'174 uv174 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f466000c040 con 0x558952981fb0 2024-08-03T20:58:28.668 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:28.666+0000 7f467b20df40 1 -- 172.21.15.32:0/4009968897 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:109 12.7 12:e0b41d06:::609.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e114) -- 0x5589529920e0 con 0x558952981fb0 2024-08-03T20:58:28.669 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:28.668+0000 7f4672e00640 1 -- 172.21.15.32:0/4009968897 <== osd.1 v2:172.21.15.32:6824/1551241759 19 ==== osd_op_reply(109 609.00000000 [setxattr (62)] v114'175 uv175 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f466000c040 con 0x558952981fb0 2024-08-03T20:58:28.669 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:28.668+0000 7f467b20df40 1 -- 172.21.15.32:0/4009968897 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:110 12.7 12:e0b41d06:::609.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e114) -- 0x5589529910d0 con 0x558952981fb0 2024-08-03T20:58:28.670 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:28.669+0000 7f4672e00640 1 -- 172.21.15.32:0/4009968897 <== osd.1 v2:172.21.15.32:6824/1551241759 20 ==== osd_op_reply(110 609.00000000 [omap-set-header] v114'176 uv176 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f466000c040 con 0x558952981fb0 2024-08-03T20:58:28.670 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:28.669+0000 7f467b20df40 1 -- 172.21.15.32:0/4009968897 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:111 12.f 12:f1f69726:::400.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e114) -- 0x5589529910d0 con 0x5589529852a0 2024-08-03T20:58:28.672 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:28.671+0000 7f4674200640 1 -- 172.21.15.32:0/4009968897 <== osd.6 v2:172.21.15.118:6825/801947386 12 ==== osd_op_reply(111 400.00000000 [create] v114'145 uv144 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f466c05c390 con 0x5589529852a0 2024-08-03T20:58:28.672 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:28.671+0000 7f467b20df40 1 -- 172.21.15.32:0/4009968897 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:112 12.f 12:f1f69726:::400.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e114) -- 0x5589529910d0 con 0x5589529852a0 2024-08-03T20:58:28.673 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:28.672+0000 7f4674200640 1 -- 172.21.15.32:0/4009968897 <== osd.6 v2:172.21.15.118:6825/801947386 13 ==== osd_op_reply(112 400.00000000 [delete] v114'146 uv146 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f466c05c390 con 0x5589529852a0 2024-08-03T20:58:28.673 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:28.672+0000 7f467b20df40 1 -- 172.21.15.32:0/4009968897 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:113 12.f 12:f1f69726:::400.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e114) -- 0x5589529910d0 con 0x5589529852a0 2024-08-03T20:58:28.675 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::400.00000000:head# 2024-08-03T20:58:28.675 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:28.673+0000 7f4674200640 1 -- 172.21.15.32:0/4009968897 <== osd.6 v2:172.21.15.118:6825/801947386 14 ==== osd_op_reply(113 400.00000000 [create] v114'147 uv147 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f466c05c390 con 0x5589529852a0 2024-08-03T20:58:28.675 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:28.673+0000 7f467b20df40 1 -- 172.21.15.32:0/4009968897 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:114 12.f 12:f1f69726:::400.00000000:head [write 0~22 in=22b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e114) -- 0x5589529910d0 con 0x5589529852a0 2024-08-03T20:58:28.676 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:28.675+0000 7f4674200640 1 -- 172.21.15.32:0/4009968897 <== osd.6 v2:172.21.15.118:6825/801947386 15 ==== osd_op_reply(114 400.00000000 [write 0~22] v114'148 uv148 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f466c05c390 con 0x5589529852a0 2024-08-03T20:58:28.677 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:28.675+0000 7f467b20df40 1 -- 172.21.15.32:0/4009968897 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:115 12.f 12:f1f69726:::400.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e114) -- 0x5589529910d0 con 0x5589529852a0 2024-08-03T20:58:28.678 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:28.676+0000 7f4674200640 1 -- 172.21.15.32:0/4009968897 <== osd.6 v2:172.21.15.118:6825/801947386 16 ==== osd_op_reply(115 400.00000000 [omap-set-header] v114'149 uv149 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f466c05c390 con 0x5589529852a0 2024-08-03T20:58:28.678 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:28.676+0000 7f467b20df40 1 -- 172.21.15.32:0/4009968897 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:116 12.1f 12:fe07bba1:::100.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e114) -- 0x5589529910d0 con 0x5589529852a0 2024-08-03T20:58:28.679 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:28.678+0000 7f4674200640 1 -- 172.21.15.32:0/4009968897 <== osd.6 v2:172.21.15.118:6825/801947386 17 ==== osd_op_reply(116 100.00000000.inode [create] v114'351 uv343 ondisk = -17 ((17) File exists)) ==== 162+0+0 (crc 0 0 0) 0x7f466c05c390 con 0x5589529852a0 2024-08-03T20:58:28.680 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:28.678+0000 7f467b20df40 1 -- 172.21.15.32:0/4009968897 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:117 12.1f 12:fe07bba1:::100.00000000.inode:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e114) -- 0x5589529910d0 con 0x5589529852a0 2024-08-03T20:58:28.681 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:28.680+0000 7f4674200640 1 -- 172.21.15.32:0/4009968897 <== osd.6 v2:172.21.15.118:6825/801947386 18 ==== osd_op_reply(117 100.00000000.inode [delete] v114'352 uv352 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7f466c05c390 con 0x5589529852a0 2024-08-03T20:58:28.681 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:28.680+0000 7f467b20df40 1 -- 172.21.15.32:0/4009968897 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:118 12.1f 12:fe07bba1:::100.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e114) -- 0x5589529910d0 con 0x5589529852a0 2024-08-03T20:58:28.682 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:28.681+0000 7f4674200640 1 -- 172.21.15.32:0/4009968897 <== osd.6 v2:172.21.15.118:6825/801947386 19 ==== osd_op_reply(118 100.00000000.inode [create] v114'353 uv353 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7f466c05c390 con 0x5589529852a0 2024-08-03T20:58:28.683 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::100.00000000.inode:head# 2024-08-03T20:58:28.683 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:28.681+0000 7f467b20df40 1 -- 172.21.15.32:0/4009968897 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:119 12.1f 12:fe07bba1:::100.00000000.inode:head [write 0~582 in=582b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e114) -- 0x5589529910d0 con 0x5589529852a0 2024-08-03T20:58:28.684 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:28.683+0000 7f4674200640 1 -- 172.21.15.32:0/4009968897 <== osd.6 v2:172.21.15.118:6825/801947386 20 ==== osd_op_reply(119 100.00000000.inode [write 0~582] v114'354 uv354 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7f466c05c390 con 0x5589529852a0 2024-08-03T20:58:28.684 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:28.683+0000 7f467b20df40 1 -- 172.21.15.32:0/4009968897 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:120 12.1f 12:fe07bba1:::100.00000000.inode:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e114) -- 0x5589529910d0 con 0x5589529852a0 2024-08-03T20:58:28.686 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:28.684+0000 7f4674200640 1 -- 172.21.15.32:0/4009968897 <== osd.6 v2:172.21.15.118:6825/801947386 21 ==== osd_op_reply(120 100.00000000.inode [omap-set-header] v114'355 uv355 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7f466c05c390 con 0x5589529852a0 2024-08-03T20:58:28.686 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:28.684+0000 7f467b20df40 1 -- 172.21.15.32:0/4009968897 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:121 12.1f 12:ff5b34d6:::1.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e114) -- 0x5589529910d0 con 0x5589529852a0 2024-08-03T20:58:28.688 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:28.686+0000 7f4674200640 1 -- 172.21.15.32:0/4009968897 <== osd.6 v2:172.21.15.118:6825/801947386 22 ==== osd_op_reply(121 1.00000000 [create] v114'356 uv350 ondisk = -17 ((17) File exists)) ==== 154+0+0 (crc 0 0 0) 0x7f466c05c390 con 0x5589529852a0 2024-08-03T20:58:28.688 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:28.686+0000 7f467b20df40 1 -- 172.21.15.32:0/4009968897 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:122 12.1f 12:ff5b34d6:::1.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e114) -- 0x5589529910d0 con 0x5589529852a0 2024-08-03T20:58:28.689 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:28.688+0000 7f4674200640 1 -- 172.21.15.32:0/4009968897 <== osd.6 v2:172.21.15.118:6825/801947386 23 ==== osd_op_reply(122 1.00000000 [delete] v114'357 uv357 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7f466c05c390 con 0x5589529852a0 2024-08-03T20:58:28.689 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:28.688+0000 7f467b20df40 1 -- 172.21.15.32:0/4009968897 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:123 12.1f 12:ff5b34d6:::1.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e114) -- 0x5589529910d0 con 0x5589529852a0 2024-08-03T20:58:28.691 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:28.689+0000 7f4674200640 1 -- 172.21.15.32:0/4009968897 <== osd.6 v2:172.21.15.118:6825/801947386 24 ==== osd_op_reply(123 1.00000000 [create] v114'358 uv358 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7f466c05c390 con 0x5589529852a0 2024-08-03T20:58:28.691 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::1.00000000:head# 2024-08-03T20:58:28.691 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:28.689+0000 7f467b20df40 1 -- 172.21.15.32:0/4009968897 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:124 12.1f 12:ff5b34d6:::1.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e114) -- 0x5589529910d0 con 0x5589529852a0 2024-08-03T20:58:28.692 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:28.691+0000 7f4674200640 1 -- 172.21.15.32:0/4009968897 <== osd.6 v2:172.21.15.118:6825/801947386 25 ==== osd_op_reply(124 1.00000000 [setxattr (30)] v114'359 uv359 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7f466c05c390 con 0x5589529852a0 2024-08-03T20:58:28.692 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:28.691+0000 7f467b20df40 1 -- 172.21.15.32:0/4009968897 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:125 12.1f 12:ff5b34d6:::1.00000000:head [setxattr parent (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e114) -- 0x5589529910d0 con 0x5589529852a0 2024-08-03T20:58:28.694 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:28.692+0000 7f4674200640 1 -- 172.21.15.32:0/4009968897 <== osd.6 v2:172.21.15.118:6825/801947386 26 ==== osd_op_reply(125 1.00000000 [setxattr (30)] v114'360 uv360 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7f466c05c390 con 0x5589529852a0 2024-08-03T20:58:28.694 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:28.692+0000 7f467b20df40 1 -- 172.21.15.32:0/4009968897 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:126 12.1f 12:ff5b34d6:::1.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e114) -- 0x5589529910d0 con 0x5589529852a0 2024-08-03T20:58:28.695 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:28.694+0000 7f4674200640 1 -- 172.21.15.32:0/4009968897 <== osd.6 v2:172.21.15.118:6825/801947386 27 ==== osd_op_reply(126 1.00000000 [omap-set-header] v114'361 uv361 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7f466c05c390 con 0x5589529852a0 2024-08-03T20:58:28.695 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:28.694+0000 7f467b20df40 1 -- 172.21.15.32:0/4009968897 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:127 12.1f 12:ff5b34d6:::1.00000000:head [omap-set-vals in=584b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e114) -- 0x558952991940 con 0x5589529852a0 2024-08-03T20:58:28.698 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:28.695+0000 7f4674200640 1 -- 172.21.15.32:0/4009968897 <== osd.6 v2:172.21.15.118:6825/801947386 28 ==== osd_op_reply(127 1.00000000 [omap-set-vals] v114'362 uv362 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7f466c05c390 con 0x5589529852a0 2024-08-03T20:58:28.698 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:28.695+0000 7f467b20df40 1 -- 172.21.15.32:0/4009968897 >> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] conn(0x5589528b3de0 msgr2=0x5589528b41f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:58:28.698 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:28.695+0000 7f467b20df40 1 --2- 172.21.15.32:0/4009968897 >> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] conn(0x5589528b3de0 0x5589528b41f0 crc :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:28.698 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:28.695+0000 7f467b20df40 1 -- 172.21.15.32:0/4009968897 >> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] conn(0x558952981fb0 msgr2=0x558952982460 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:58:28.698 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:28.695+0000 7f467b20df40 1 --2- 172.21.15.32:0/4009968897 >> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] conn(0x558952981fb0 0x558952982460 crc :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:28.698 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:28.695+0000 7f467b20df40 1 -- 172.21.15.32:0/4009968897 >> [v2:172.21.15.32:6800/9088303,v1:172.21.15.32:6801/9088303] conn(0x55895298e770 msgr2=0x558952990b90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:58:28.698 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:28.695+0000 7f467b20df40 1 --2- 172.21.15.32:0/4009968897 >> [v2:172.21.15.32:6800/9088303,v1:172.21.15.32:6801/9088303] conn(0x55895298e770 0x558952990b90 crc :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:28.698 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:28.695+0000 7f467b20df40 1 -- 172.21.15.32:0/4009968897 >> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] conn(0x55895297dd30 msgr2=0x558952980150 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:58:28.698 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:28.695+0000 7f467b20df40 1 --2- 172.21.15.32:0/4009968897 >> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] conn(0x55895297dd30 0x558952980150 crc :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:28.698 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:28.696+0000 7f4674200640 1 -- 172.21.15.32:0/4009968897 reap_dead start 2024-08-03T20:58:28.699 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:28.696+0000 7f467b20df40 1 -- 172.21.15.32:0/4009968897 >> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] conn(0x5589529852a0 msgr2=0x5589529856b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:58:28.699 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:28.696+0000 7f467b20df40 1 --2- 172.21.15.32:0/4009968897 >> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] conn(0x5589529852a0 0x5589529856b0 crc :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:28.699 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:28.696+0000 7f4674200640 1 -- 172.21.15.32:0/4009968897 reap_dead start 2024-08-03T20:58:28.699 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:28.696+0000 7f467b20df40 1 -- 172.21.15.32:0/4009968897 >> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] conn(0x55895298ac80 msgr2=0x55895298d0a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:58:28.699 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:28.696+0000 7f467b20df40 1 --2- 172.21.15.32:0/4009968897 >> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] conn(0x55895298ac80 0x55895298d0a0 crc :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:28.699 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:28.696+0000 7f467b20df40 1 -- 172.21.15.32:0/4009968897 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f4654075a20 msgr2=0x7f4654077f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:58:28.699 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:28.696+0000 7f467b20df40 1 --2- 172.21.15.32:0/4009968897 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f4654075a20 0x7f4654077f30 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x7f4664062000 tx=0x7f4664004940 comp rx=0 tx=0).stop 2024-08-03T20:58:28.699 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:28.696+0000 7f467b20df40 1 -- 172.21.15.32:0/4009968897 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5589528d6cb0 msgr2=0x5589528b6900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:58:28.699 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:28.696+0000 7f467b20df40 1 --2- 172.21.15.32:0/4009968897 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5589528d6cb0 0x5589528b6900 secure :-1 s=READY pgs=1776 cs=0 l=1 rev1=1 crypto rx=0x7f466c002fa0 tx=0x7f466c00d5f0 comp rx=0 tx=0).stop 2024-08-03T20:58:28.699 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:28.696+0000 7f467b20df40 1 -- 172.21.15.32:0/4009968897 shutdown_connections 2024-08-03T20:58:28.699 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:28.696+0000 7f467b20df40 1 --2- 172.21.15.32:0/4009968897 >> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] conn(0x55895298ac80 0x55895298d0a0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:28.699 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:28.696+0000 7f467b20df40 1 --2- 172.21.15.32:0/4009968897 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f4654075a20 0x7f4654077f30 unknown :-1 s=CLOSED pgs=722 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:28.700 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:28.696+0000 7f467b20df40 1 --2- 172.21.15.32:0/4009968897 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5589528d6cb0 0x5589528b6900 unknown :-1 s=CLOSED pgs=1776 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:28.700 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:28.696+0000 7f467b20df40 1 -- 172.21.15.32:0/4009968897 >> 172.21.15.32:0/4009968897 conn(0x558952722ac0 msgr2=0x5589528491e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:58:28.700 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:28.696+0000 7f467b20df40 1 -- 172.21.15.32:0/4009968897 shutdown_connections 2024-08-03T20:58:28.700 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:28.696+0000 7f467b20df40 1 -- 172.21.15.32:0/4009968897 wait complete. 2024-08-03T20:58:28.711 DEBUG:teuthology.orchestra.run.smithi032:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata truncate 400.00000000 0 2024-08-03T20:58:28.856 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:28.855+0000 7f0891287f40 1 Processor -- start 2024-08-03T20:58:28.857 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:28.855+0000 7f0891287f40 1 -- start start 2024-08-03T20:58:28.859 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:28.856+0000 7f0891287f40 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x558c0df01cb0 0x558c0df00b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:58:28.859 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:28.856+0000 7f0891287f40 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558c0df85ba0 0x558c0deb7500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:58:28.859 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:28.856+0000 7f0891287f40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x558c0deffc60 0x558c0dee2da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:58:28.859 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:28.856+0000 7f0891287f40 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x558c0de07e30 con 0x558c0df85ba0 2024-08-03T20:58:28.859 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:28.856+0000 7f0891287f40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x558c0de65bf0 con 0x558c0deffc60 2024-08-03T20:58:28.859 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:28.856+0000 7f0891287f40 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x558c0df8fdf0 con 0x558c0df01cb0 2024-08-03T20:58:28.859 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:28.856+0000 7f0889a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558c0df85ba0 0x558c0deb7500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:58:28.860 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:28.856+0000 7f0889a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558c0df85ba0 0x558c0deb7500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:42768/0 (socket says 172.21.15.32:42768) 2024-08-03T20:58:28.860 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:28.856+0000 7f0889a00640 1 -- 172.21.15.32:0/1274688226 learned_addr learned my addr 172.21.15.32:0/1274688226 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:58:28.860 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:28.857+0000 7f088a400640 1 --2- 172.21.15.32:0/1274688226 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x558c0df01cb0 0x558c0df00b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:58:28.860 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:28.857+0000 7f088ae00640 1 --2- 172.21.15.32:0/1274688226 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x558c0deffc60 0x558c0dee2da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:58:28.860 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:28.857+0000 7f0889a00640 1 -- 172.21.15.32:0/1274688226 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x558c0df01cb0 msgr2=0x558c0df00b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:58:28.860 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:28.857+0000 7f0889a00640 1 --2- 172.21.15.32:0/1274688226 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x558c0df01cb0 0x558c0df00b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:28.860 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:28.857+0000 7f0889a00640 1 -- 172.21.15.32:0/1274688226 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x558c0deffc60 msgr2=0x558c0dee2da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:58:28.860 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:28.857+0000 7f0889a00640 1 --2- 172.21.15.32:0/1274688226 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x558c0deffc60 0x558c0dee2da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:28.860 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:28.857+0000 7f0889a00640 1 -- 172.21.15.32:0/1274688226 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558c0defbaa0 con 0x558c0df85ba0 2024-08-03T20:58:28.861 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:28.857+0000 7f088ae00640 1 --2- 172.21.15.32:0/1274688226 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x558c0deffc60 0x558c0dee2da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-03T20:58:28.861 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:28.857+0000 7f088a400640 1 --2- 172.21.15.32:0/1274688226 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x558c0df01cb0 0x558c0df00b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-03T20:58:28.861 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:28.857+0000 7f0889a00640 1 --2- 172.21.15.32:0/1274688226 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558c0df85ba0 0x558c0deb7500 secure :-1 s=READY pgs=1778 cs=0 l=1 rev1=1 crypto rx=0x7f0878001cf0 tx=0x7f087800b5b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f66f77a2c674cd09 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:58:28.861 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:28.857+0000 7f0883e00640 1 -- 172.21.15.32:0/1274688226 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f087800bd80 con 0x558c0df85ba0 2024-08-03T20:58:28.861 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:28.857+0000 7f0883e00640 1 -- 172.21.15.32:0/1274688226 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f087800f400 con 0x558c0df85ba0 2024-08-03T20:58:28.861 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:28.857+0000 7f0883e00640 1 -- 172.21.15.32:0/1274688226 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f087800f850 con 0x558c0df85ba0 2024-08-03T20:58:28.864 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:28.858+0000 7f0891287f40 1 -- 172.21.15.32:0/1274688226 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558c0df85ba0 msgr2=0x558c0deb7500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:58:28.864 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:28.858+0000 7f0891287f40 1 --2- 172.21.15.32:0/1274688226 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558c0df85ba0 0x558c0deb7500 secure :-1 s=READY pgs=1778 cs=0 l=1 rev1=1 crypto rx=0x7f0878001cf0 tx=0x7f087800b5b0 comp rx=0 tx=0).stop 2024-08-03T20:58:28.864 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:28.858+0000 7f0891287f40 1 -- 172.21.15.32:0/1274688226 shutdown_connections 2024-08-03T20:58:28.864 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:28.858+0000 7f0891287f40 1 --2- 172.21.15.32:0/1274688226 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x558c0deffc60 0x558c0dee2da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:28.864 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:28.858+0000 7f0891287f40 1 --2- 172.21.15.32:0/1274688226 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558c0df85ba0 0x558c0deb7500 unknown :-1 s=CLOSED pgs=1778 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:28.864 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:28.858+0000 7f0891287f40 1 --2- 172.21.15.32:0/1274688226 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x558c0df01cb0 0x558c0df00b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:28.864 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:28.858+0000 7f0891287f40 1 -- 172.21.15.32:0/1274688226 >> 172.21.15.32:0/1274688226 conn(0x558c0ddd9ac0 msgr2=0x558c0df8e740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:58:28.864 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:28.858+0000 7f0891287f40 1 -- 172.21.15.32:0/1274688226 shutdown_connections 2024-08-03T20:58:28.865 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:28.858+0000 7f0891287f40 1 -- 172.21.15.32:0/1274688226 wait complete. 2024-08-03T20:58:28.865 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:28.859+0000 7f0891287f40 1 Processor -- start 2024-08-03T20:58:28.865 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:28.859+0000 7f0891287f40 1 -- start start 2024-08-03T20:58:28.865 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:28.861+0000 7f0891287f40 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x558c0deffc60 0x558c0df7eff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:58:28.865 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:28.861+0000 7f0891287f40 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558c0df01cb0 0x558c0dee3d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:58:28.865 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:28.861+0000 7f0891287f40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x558c0df85ba0 0x558c0dee4280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:58:28.865 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:28.861+0000 7f0891287f40 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x558c0defb850 con 0x558c0df01cb0 2024-08-03T20:58:28.865 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:28.861+0000 7f0891287f40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x558c0defbd80 con 0x558c0df85ba0 2024-08-03T20:58:28.865 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:28.861+0000 7f0891287f40 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x558c0df00640 con 0x558c0deffc60 2024-08-03T20:58:28.865 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:28.861+0000 7f0889a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558c0df01cb0 0x558c0dee3d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:58:28.865 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:28.861+0000 7f0889a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558c0df01cb0 0x558c0dee3d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:42778/0 (socket says 172.21.15.32:42778) 2024-08-03T20:58:28.865 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:28.861+0000 7f0889a00640 1 -- 172.21.15.32:0/2725366122 learned_addr learned my addr 172.21.15.32:0/2725366122 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:58:28.865 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:28.861+0000 7f088ae00640 1 --2- 172.21.15.32:0/2725366122 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x558c0df85ba0 0x558c0dee4280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:58:28.866 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:28.861+0000 7f088a400640 1 --2- 172.21.15.32:0/2725366122 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x558c0deffc60 0x558c0df7eff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:58:28.866 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:28.862+0000 7f0889a00640 1 -- 172.21.15.32:0/2725366122 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x558c0deffc60 msgr2=0x558c0df7eff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:58:28.866 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:28.862+0000 7f0889a00640 1 --2- 172.21.15.32:0/2725366122 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x558c0deffc60 0x558c0df7eff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:28.866 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:28.862+0000 7f0889a00640 1 -- 172.21.15.32:0/2725366122 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x558c0df85ba0 msgr2=0x558c0dee4280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:58:28.866 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:28.862+0000 7f0889a00640 1 --2- 172.21.15.32:0/2725366122 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x558c0df85ba0 0x558c0dee4280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:28.866 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:28.862+0000 7f0889a00640 1 -- 172.21.15.32:0/2725366122 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558c0df821d0 con 0x558c0df01cb0 2024-08-03T20:58:28.866 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:28.862+0000 7f0889a00640 1 --2- 172.21.15.32:0/2725366122 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558c0df01cb0 0x558c0dee3d40 secure :-1 s=READY pgs=1779 cs=0 l=1 rev1=1 crypto rx=0x7f087800bab0 tx=0x7f087800ff30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:58:28.866 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:28.862+0000 7f0882a00640 1 -- 172.21.15.32:0/2725366122 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f087801b430 con 0x558c0df01cb0 2024-08-03T20:58:28.866 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:28.862+0000 7f0882a00640 1 -- 172.21.15.32:0/2725366122 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f0878020020 con 0x558c0df01cb0 2024-08-03T20:58:28.866 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:28.862+0000 7f0891287f40 1 -- 172.21.15.32:0/2725366122 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x558c0df824b0 con 0x558c0df01cb0 2024-08-03T20:58:28.866 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:28.862+0000 7f0882a00640 1 -- 172.21.15.32:0/2725366122 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f087801b760 con 0x558c0df01cb0 2024-08-03T20:58:28.866 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:28.863+0000 7f0891287f40 1 -- 172.21.15.32:0/2725366122 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558c0df84010 con 0x558c0df01cb0 2024-08-03T20:58:28.867 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:28.865+0000 7f0882a00640 1 -- 172.21.15.32:0/2725366122 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f0878017070 con 0x558c0df01cb0 2024-08-03T20:58:28.867 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:28.866+0000 7f0882a00640 1 --2- 172.21.15.32:0/2725366122 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f086c0759f0 0x7f086c077f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:58:28.868 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:28.866+0000 7f0882a00640 1 -- 172.21.15.32:0/2725366122 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(114..114 src has 1..114) ==== 8348+0+0 (secure 0 0 0) 0x7f0878099180 con 0x558c0df01cb0 2024-08-03T20:58:28.868 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:28.866+0000 7f088a400640 1 --2- 172.21.15.32:0/2725366122 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f086c0759f0 0x7f086c077f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:58:28.868 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:28.866+0000 7f0891287f40 1 --2- 172.21.15.32:0/2725366122 >> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] conn(0x558c0df83950 0x558c0df87020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:58:28.868 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:28.866+0000 7f0891287f40 1 -- 172.21.15.32:0/2725366122 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:1 12.f 12:f1f69726:::400.00000000:head [truncate 0] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e114) -- 0x558c0df87560 con 0x558c0df83950 2024-08-03T20:58:28.869 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:28.867+0000 7f088ae00640 1 --2- 172.21.15.32:0/2725366122 >> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] conn(0x558c0df83950 0x558c0df87020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:58:28.869 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:28.867+0000 7f088a400640 1 --2- 172.21.15.32:0/2725366122 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f086c0759f0 0x7f086c077f00 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0x7f087c00c6e0 tx=0x7f087c004a10 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:58:28.869 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:28.867+0000 7f088ae00640 1 --2- 172.21.15.32:0/2725366122 >> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] conn(0x558c0df83950 0x558c0df87020 crc :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:58:28.871 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:28.868+0000 7f088ae00640 1 -- 172.21.15.32:0/2725366122 <== osd.6 v2:172.21.15.118:6825/801947386 1 ==== osd_op_reply(1 400.00000000 [truncate 0] v114'150 uv150 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f088400b040 con 0x558c0df83950 2024-08-03T20:58:28.871 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:28.869+0000 7f0891287f40 1 -- 172.21.15.32:0/2725366122 >> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] conn(0x558c0df83950 msgr2=0x558c0df87020 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:58:28.871 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:28.869+0000 7f0891287f40 1 --2- 172.21.15.32:0/2725366122 >> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] conn(0x558c0df83950 0x558c0df87020 crc :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:28.871 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:28.869+0000 7f0891287f40 1 -- 172.21.15.32:0/2725366122 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f086c0759f0 msgr2=0x7f086c077f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:58:28.871 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:28.869+0000 7f0891287f40 1 --2- 172.21.15.32:0/2725366122 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f086c0759f0 0x7f086c077f00 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0x7f087c00c6e0 tx=0x7f087c004a10 comp rx=0 tx=0).stop 2024-08-03T20:58:28.871 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:28.869+0000 7f0891287f40 1 -- 172.21.15.32:0/2725366122 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558c0df01cb0 msgr2=0x558c0dee3d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:58:28.871 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:28.869+0000 7f0891287f40 1 --2- 172.21.15.32:0/2725366122 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558c0df01cb0 0x558c0dee3d40 secure :-1 s=READY pgs=1779 cs=0 l=1 rev1=1 crypto rx=0x7f087800bab0 tx=0x7f087800ff30 comp rx=0 tx=0).stop 2024-08-03T20:58:28.871 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:28.869+0000 7f088ae00640 1 -- 172.21.15.32:0/2725366122 reap_dead start 2024-08-03T20:58:28.871 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:28.869+0000 7f0891287f40 1 -- 172.21.15.32:0/2725366122 shutdown_connections 2024-08-03T20:58:28.871 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:28.869+0000 7f0891287f40 1 -- 172.21.15.32:0/2725366122 >> 172.21.15.32:0/2725366122 conn(0x558c0ddd9ac0 msgr2=0x558c0defea90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:58:28.872 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:28.869+0000 7f0891287f40 1 -- 172.21.15.32:0/2725366122 shutdown_connections 2024-08-03T20:58:28.872 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:28.869+0000 7f0891287f40 1 -- 172.21.15.32:0/2725366122 wait complete. 2024-08-03T20:58:28.885 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs joinable true 2024-08-03T20:58:29.043 INFO:tasks.ceph.mds.a.smithi032.stderr: -46> 2024-08-03T20:58:24.288+0000 7f6998200640 -1 log_channel(cluster) log [ERR] : error decoding table object 'mds0_inotable': End of buffer [buffer:2] 2024-08-03T20:58:29.118 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:29.116+0000 7f363d200640 1 -- 172.21.15.32:0/3294642143 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f36380cccf0 msgr2=0x7f36380cd100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:58:29.118 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:29.116+0000 7f363d200640 1 --2- 172.21.15.32:0/3294642143 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f36380cccf0 0x7f36380cd100 secure :-1 s=READY pgs=1781 cs=0 l=1 rev1=1 crypto rx=0x7f3624009830 tx=0x7f362402a500 comp rx=0 tx=0).stop 2024-08-03T20:58:29.118 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:29.116+0000 7f363d200640 1 -- 172.21.15.32:0/3294642143 shutdown_connections 2024-08-03T20:58:29.118 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:29.116+0000 7f363d200640 1 --2- 172.21.15.32:0/3294642143 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f36380ced30 0x7f36380cf140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:29.118 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:29.116+0000 7f363d200640 1 --2- 172.21.15.32:0/3294642143 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f36380cccf0 0x7f36380cd100 unknown :-1 s=CLOSED pgs=1781 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:29.119 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:29.116+0000 7f363d200640 1 -- 172.21.15.32:0/3294642143 >> 172.21.15.32:0/3294642143 conn(0x7f36380b1bd0 msgr2=0x7f36380b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:58:29.119 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:29.116+0000 7f363d200640 1 -- 172.21.15.32:0/3294642143 shutdown_connections 2024-08-03T20:58:29.119 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:29.116+0000 7f363d200640 1 -- 172.21.15.32:0/3294642143 wait complete. 2024-08-03T20:58:29.120 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:29.118+0000 7f363d200640 1 Processor -- start 2024-08-03T20:58:29.120 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:29.118+0000 7f363d200640 1 -- start start 2024-08-03T20:58:29.122 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:29.120+0000 7f363d200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f36380cccf0 0x7f3638138b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:58:29.122 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:29.120+0000 7f363d200640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f36380ced30 0x7f363813a610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:58:29.122 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:29.120+0000 7f363d200640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f3638145c50 0x7f363813d3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:58:29.122 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:29.120+0000 7f363d200640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f36380b4070 con 0x7f36380cccf0 2024-08-03T20:58:29.122 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:29.120+0000 7f363d200640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f36380b3ef0 con 0x7f3638145c50 2024-08-03T20:58:29.122 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:29.120+0000 7f363d200640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7f36380b41f0 con 0x7f36380ced30 2024-08-03T20:58:29.122 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:29.120+0000 7f3635e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f36380cccf0 0x7f3638138b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:58:29.122 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:29.120+0000 7f3635e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f36380cccf0 0x7f3638138b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:42804/0 (socket says 172.21.15.32:42804) 2024-08-03T20:58:29.122 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:29.120+0000 7f3635e00640 1 -- 172.21.15.32:0/1480931527 learned_addr learned my addr 172.21.15.32:0/1480931527 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:58:29.122 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:29.121+0000 7f3635400640 1 --2- 172.21.15.32:0/1480931527 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f36380ced30 0x7f363813a610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:58:29.123 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:29.121+0000 7f3636800640 1 --2- 172.21.15.32:0/1480931527 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f3638145c50 0x7f363813d3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:58:29.124 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:29.121+0000 7f3635e00640 1 -- 172.21.15.32:0/1480931527 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f36380ced30 msgr2=0x7f363813a610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:58:29.124 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:29.121+0000 7f3635e00640 1 --2- 172.21.15.32:0/1480931527 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f36380ced30 0x7f363813a610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:29.124 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:29.121+0000 7f3635e00640 1 -- 172.21.15.32:0/1480931527 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f3638145c50 msgr2=0x7f363813d3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:58:29.124 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:29.121+0000 7f3635e00640 1 --2- 172.21.15.32:0/1480931527 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f3638145c50 0x7f363813d3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:29.124 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:29.121+0000 7f3635e00640 1 -- 172.21.15.32:0/1480931527 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f363813ab50 con 0x7f36380cccf0 2024-08-03T20:58:29.125 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:29.121+0000 7f3636800640 1 --2- 172.21.15.32:0/1480931527 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f3638145c50 0x7f363813d3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-03T20:58:29.125 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:29.121+0000 7f3635e00640 1 --2- 172.21.15.32:0/1480931527 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f36380cccf0 0x7f3638138b30 secure :-1 s=READY pgs=1782 cs=0 l=1 rev1=1 crypto rx=0x7f362800c3f0 tx=0x7f362800c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:58:29.125 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:29.121+0000 7f362e000640 1 -- 172.21.15.32:0/1480931527 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3628017070 con 0x7f36380cccf0 2024-08-03T20:58:29.125 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:29.121+0000 7f362e000640 1 -- 172.21.15.32:0/1480931527 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f36280035f0 con 0x7f36380cccf0 2024-08-03T20:58:29.125 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:29.121+0000 7f363d200640 1 -- 172.21.15.32:0/1480931527 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f36381393d0 con 0x7f36380cccf0 2024-08-03T20:58:29.125 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:29.121+0000 7f362e000640 1 -- 172.21.15.32:0/1480931527 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3628003a40 con 0x7f36380cccf0 2024-08-03T20:58:29.125 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:29.122+0000 7f363d200640 1 -- 172.21.15.32:0/1480931527 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3638139990 con 0x7f36380cccf0 2024-08-03T20:58:29.125 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:29.123+0000 7f363d200640 1 -- 172.21.15.32:0/1480931527 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f36380b4700 con 0x7f36380cccf0 2024-08-03T20:58:29.126 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:29.125+0000 7f362e000640 1 -- 172.21.15.32:0/1480931527 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f362801a070 con 0x7f36380cccf0 2024-08-03T20:58:29.127 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:29.126+0000 7f362e000640 1 --2- 172.21.15.32:0/1480931527 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f3618077010 0x7f3618079520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:58:29.127 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:29.126+0000 7f362e000640 1 -- 172.21.15.32:0/1480931527 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(114..114 src has 1..114) ==== 8348+0+0 (secure 0 0 0) 0x7f3628098700 con 0x7f36380cccf0 2024-08-03T20:58:29.128 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:29.126+0000 7f3635400640 1 --2- 172.21.15.32:0/1480931527 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f3618077010 0x7f3618079520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:58:29.129 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:29.127+0000 7f3635400640 1 --2- 172.21.15.32:0/1480931527 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f3618077010 0x7f3618079520 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 crypto rx=0x7f3624034000 tx=0x7f3624004730 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:58:29.132 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:29.131+0000 7f362e000640 1 -- 172.21.15.32:0/1480931527 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7f3628060f20 con 0x7f36380cccf0 2024-08-03T20:58:29.159 INFO:tasks.ceph.mds.a.smithi032.stdout:starting mds.a at 2024-08-03T20:58:29.383 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:29.381+0000 7f363d200640 1 -- 172.21.15.32:0/1480931527 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "joinable", "val": "true"} v 0) -- 0x7f36380d0b50 con 0x7f36380cccf0 2024-08-03T20:58:30.057 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:30.055+0000 7f362e000640 1 -- 172.21.15.32:0/1480931527 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "joinable", "val": "true"}]=0 cephfs marked joinable; MDS may join as newly active. v259) ==== 162+0+0 (secure 0 0 0) 0x7f3628069160 con 0x7f36380cccf0 2024-08-03T20:58:30.057 INFO:teuthology.orchestra.run.smithi032.stderr:cephfs marked joinable; MDS may join as newly active. 2024-08-03T20:58:30.060 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:30.058+0000 7f363d200640 1 -- 172.21.15.32:0/1480931527 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f3618077010 msgr2=0x7f3618079520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:58:30.060 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:30.058+0000 7f363d200640 1 --2- 172.21.15.32:0/1480931527 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f3618077010 0x7f3618079520 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 crypto rx=0x7f3624034000 tx=0x7f3624004730 comp rx=0 tx=0).stop 2024-08-03T20:58:30.060 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:30.058+0000 7f363d200640 1 -- 172.21.15.32:0/1480931527 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f36380cccf0 msgr2=0x7f3638138b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:58:30.060 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:30.058+0000 7f363d200640 1 --2- 172.21.15.32:0/1480931527 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f36380cccf0 0x7f3638138b30 secure :-1 s=READY pgs=1782 cs=0 l=1 rev1=1 crypto rx=0x7f362800c3f0 tx=0x7f362800c8b0 comp rx=0 tx=0).stop 2024-08-03T20:58:30.060 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:30.058+0000 7f363d200640 1 -- 172.21.15.32:0/1480931527 shutdown_connections 2024-08-03T20:58:30.061 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:30.058+0000 7f363d200640 1 --2- 172.21.15.32:0/1480931527 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f3618077010 0x7f3618079520 unknown :-1 s=CLOSED pgs=724 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:30.061 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:30.058+0000 7f363d200640 1 --2- 172.21.15.32:0/1480931527 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f3638145c50 0x7f363813d3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:30.061 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:30.058+0000 7f363d200640 1 --2- 172.21.15.32:0/1480931527 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f36380ced30 0x7f363813a610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:30.061 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:30.058+0000 7f363d200640 1 --2- 172.21.15.32:0/1480931527 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f36380cccf0 0x7f3638138b30 unknown :-1 s=CLOSED pgs=1782 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:30.061 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:30.059+0000 7f363d200640 1 -- 172.21.15.32:0/1480931527 >> 172.21.15.32:0/1480931527 conn(0x7f36380b1bd0 msgr2=0x7f363815f4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:58:30.061 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:30.059+0000 7f363d200640 1 -- 172.21.15.32:0/1480931527 shutdown_connections 2024-08-03T20:58:30.061 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:30.059+0000 7f363d200640 1 -- 172.21.15.32:0/1480931527 wait complete. 2024-08-03T20:58:30.064 INFO:tasks.ceph.mds.a.smithi032.stderr:2024-08-03T20:58:30.062+0000 7f6434200640 -1 log_channel(cluster) log [ERR] : failed to read JournalPointer: -22 ((22) Invalid argument) 2024-08-03T20:58:30.064 INFO:tasks.ceph.mon.a.smithi032.stderr:2024-08-03T20:58:30.063+0000 7f4e6d600640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-08-03T20:58:30.065 INFO:tasks.ceph.mon.a.smithi032.stderr:2024-08-03T20:58:30.063+0000 7f4e6d600640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 mds daemon damaged (MDS_DAMAGE) 2024-08-03T20:58:30.088 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-08-03T20:58:30.323 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:30.320+0000 7f5deb000640 1 -- 172.21.15.32:0/2803810678 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5dc8003a50 msgr2=0x7f5dc8003ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:58:30.323 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:30.320+0000 7f5deb000640 1 --2- 172.21.15.32:0/2803810678 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5dc8003a50 0x7f5dc8003ec0 secure :-1 s=READY pgs=1787 cs=0 l=1 rev1=1 crypto rx=0x7f5dd400c3f0 tx=0x7f5dc8005830 comp rx=0 tx=0).stop 2024-08-03T20:58:30.323 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:30.320+0000 7f5deb000640 1 -- 172.21.15.32:0/2803810678 shutdown_connections 2024-08-03T20:58:30.324 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:30.320+0000 7f5deb000640 1 --2- 172.21.15.32:0/2803810678 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5dc8003a50 0x7f5dc8003ec0 unknown :-1 s=CLOSED pgs=1787 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:30.324 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:30.320+0000 7f5deb000640 1 -- 172.21.15.32:0/2803810678 >> 172.21.15.32:0/2803810678 conn(0x7f5de40b1bd0 msgr2=0x7f5de40b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:58:30.324 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:30.320+0000 7f5deb000640 1 -- 172.21.15.32:0/2803810678 shutdown_connections 2024-08-03T20:58:30.324 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:30.321+0000 7f5deb000640 1 -- 172.21.15.32:0/2803810678 wait complete. 2024-08-03T20:58:30.325 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:30.323+0000 7f5deb000640 1 Processor -- start 2024-08-03T20:58:30.325 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:30.323+0000 7f5deb000640 1 -- start start 2024-08-03T20:58:30.330 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:30.324+0000 7f5deb000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5dc8003a50 0x7f5de41e6650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:58:30.330 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:30.324+0000 7f5deb000640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f5de414f390 0x7f5de40aeeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:58:30.330 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:30.324+0000 7f5deb000640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f5de41e4ee0 0x7f5de41e52f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:58:30.330 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:30.325+0000 7f5deb000640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f5de4070a40 con 0x7f5dc8003a50 2024-08-03T20:58:30.330 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:30.325+0000 7f5deb000640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f5de40708c0 con 0x7f5de414f390 2024-08-03T20:58:30.331 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:30.325+0000 7f5deb000640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7f5de415e170 con 0x7f5de41e4ee0 2024-08-03T20:58:30.331 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:30.325+0000 7f5de3e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5dc8003a50 0x7f5de41e6650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:58:30.331 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:30.325+0000 7f5de3e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5dc8003a50 0x7f5de41e6650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:41484/0 (socket says 172.21.15.32:41484) 2024-08-03T20:58:30.331 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:30.325+0000 7f5de3e00640 1 -- 172.21.15.32:0/1362319414 learned_addr learned my addr 172.21.15.32:0/1362319414 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:58:30.331 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:30.325+0000 7f5ddbe00640 1 --2- 172.21.15.32:0/1362319414 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f5de414f390 0x7f5de40aeeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:58:30.331 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:30.325+0000 7f5de9000640 1 --2- 172.21.15.32:0/1362319414 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f5de41e4ee0 0x7f5de41e52f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:58:30.331 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:30.325+0000 7f5de3e00640 1 -- 172.21.15.32:0/1362319414 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f5de41e4ee0 msgr2=0x7f5de41e52f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:58:30.332 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:30.325+0000 7f5de3e00640 1 --2- 172.21.15.32:0/1362319414 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f5de41e4ee0 0x7f5de41e52f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:30.332 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:30.325+0000 7f5de3e00640 1 -- 172.21.15.32:0/1362319414 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f5de414f390 msgr2=0x7f5de40aeeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:58:30.332 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:30.325+0000 7f5de3e00640 1 --2- 172.21.15.32:0/1362319414 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f5de414f390 0x7f5de40aeeb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:30.332 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:30.325+0000 7f5de3e00640 1 -- 172.21.15.32:0/1362319414 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5de414fc30 con 0x7f5dc8003a50 2024-08-03T20:58:30.332 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:30.326+0000 7f5de3e00640 1 --2- 172.21.15.32:0/1362319414 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5dc8003a50 0x7f5de41e6650 secure :-1 s=READY pgs=1788 cs=0 l=1 rev1=1 crypto rx=0x7f5dd402b2a0 tx=0x7f5dd40013d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:58:30.332 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:30.326+0000 7f5de0c00640 1 -- 172.21.15.32:0/1362319414 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5dd4042070 con 0x7f5dc8003a50 2024-08-03T20:58:30.332 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:30.326+0000 7f5de0c00640 1 -- 172.21.15.32:0/1362319414 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f5dd40035f0 con 0x7f5dc8003a50 2024-08-03T20:58:30.332 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:30.326+0000 7f5deb000640 1 -- 172.21.15.32:0/1362319414 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5de40ade10 con 0x7f5dc8003a50 2024-08-03T20:58:30.333 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:30.326+0000 7f5de0c00640 1 -- 172.21.15.32:0/1362319414 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5dd4039070 con 0x7f5dc8003a50 2024-08-03T20:58:30.333 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:30.326+0000 7f5deb000640 1 -- 172.21.15.32:0/1362319414 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5de40ae3d0 con 0x7f5dc8003a50 2024-08-03T20:58:30.333 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:30.327+0000 7f5deb000640 1 -- 172.21.15.32:0/1362319414 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5de406f020 con 0x7f5dc8003a50 2024-08-03T20:58:30.333 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:30.328+0000 7f5de0c00640 1 -- 172.21.15.32:0/1362319414 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f5dd4003a20 con 0x7f5dc8003a50 2024-08-03T20:58:30.334 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:30.329+0000 7f5de0c00640 1 --2- 172.21.15.32:0/1362319414 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f5dc0077040 0x7f5dc0079550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:58:30.334 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:30.329+0000 7f5de0c00640 1 -- 172.21.15.32:0/1362319414 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(115..115 src has 1..115) ==== 8434+0+0 (secure 0 0 0) 0x7f5dd40b8e90 con 0x7f5dc8003a50 2024-08-03T20:58:30.334 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:30.330+0000 7f5ddbe00640 1 --2- 172.21.15.32:0/1362319414 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f5dc0077040 0x7f5dc0079550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:58:30.334 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:30.330+0000 7f5ddbe00640 1 --2- 172.21.15.32:0/1362319414 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f5dc0077040 0x7f5dc0079550 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7f5de406ea30 tx=0x7f5dcc010040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:58:30.335 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:30.334+0000 7f5de0c00640 1 -- 172.21.15.32:0/1362319414 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7f5de414fc30 con 0x7f5dc8003a50 2024-08-03T20:58:30.586 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:30.584+0000 7f5deb000640 1 -- 172.21.15.32:0/1362319414 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f5de406ed40 con 0x7f5dc8003a50 2024-08-03T20:58:30.586 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:30.585+0000 7f5de0c00640 1 -- 172.21.15.32:0/1362319414 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 260 v260) ==== 95+0+4391 (secure 0 0 0) 0x7f5dd40ba070 con 0x7f5dc8003a50 2024-08-03T20:58:30.587 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-03T20:58:30.587 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":260,"btime":"2024-08-03T20:58:30:065005+0000","default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6439,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6835/531080160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6834","nonce":531080160},{"type":"v1","addr":"172.21.15.118:6835","nonce":531080160}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":84},{"gid":6454,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6837/2796882319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6836","nonce":2796882319},{"type":"v1","addr":"172.21.15.118:6837","nonce":2796882319}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":86},{"gid":6813,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6834/2686933890","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":2686933890},{"type":"v1","addr":"172.21.15.32:6834","nonce":2686933890}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":85}],"filesystems":[{"mdsmap":{"epoch":260,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-08-03T20:54:26.950518+0000","modified":"2024-08-03T20:58:30.064912+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":115,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":6}]} 2024-08-03T20:58:30.587 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 260 2024-08-03T20:58:30.589 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:30.587+0000 7f5deb000640 1 -- 172.21.15.32:0/1362319414 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f5dc0077040 msgr2=0x7f5dc0079550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:58:30.589 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:30.587+0000 7f5deb000640 1 --2- 172.21.15.32:0/1362319414 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f5dc0077040 0x7f5dc0079550 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7f5de406ea30 tx=0x7f5dcc010040 comp rx=0 tx=0).stop 2024-08-03T20:58:30.589 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:30.587+0000 7f5deb000640 1 -- 172.21.15.32:0/1362319414 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5dc8003a50 msgr2=0x7f5de41e6650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:58:30.589 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:30.587+0000 7f5deb000640 1 --2- 172.21.15.32:0/1362319414 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5dc8003a50 0x7f5de41e6650 secure :-1 s=READY pgs=1788 cs=0 l=1 rev1=1 crypto rx=0x7f5dd402b2a0 tx=0x7f5dd40013d0 comp rx=0 tx=0).stop 2024-08-03T20:58:30.590 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:30.588+0000 7f5deb000640 1 -- 172.21.15.32:0/1362319414 shutdown_connections 2024-08-03T20:58:30.590 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:30.588+0000 7f5deb000640 1 --2- 172.21.15.32:0/1362319414 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f5dc0077040 0x7f5dc0079550 unknown :-1 s=CLOSED pgs=726 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:30.590 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:30.588+0000 7f5deb000640 1 --2- 172.21.15.32:0/1362319414 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f5de414f390 0x7f5de40aeeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:30.590 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:30.588+0000 7f5deb000640 1 --2- 172.21.15.32:0/1362319414 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f5de41e4ee0 0x7f5de41e52f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:30.590 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:30.588+0000 7f5deb000640 1 --2- 172.21.15.32:0/1362319414 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5dc8003a50 0x7f5de41e6650 unknown :-1 s=CLOSED pgs=1788 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:30.590 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:30.588+0000 7f5deb000640 1 -- 172.21.15.32:0/1362319414 >> 172.21.15.32:0/1362319414 conn(0x7f5de40b1bd0 msgr2=0x7f5de40afeb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:58:30.590 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:30.588+0000 7f5deb000640 1 -- 172.21.15.32:0/1362319414 shutdown_connections 2024-08-03T20:58:30.590 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:30.588+0000 7f5deb000640 1 -- 172.21.15.32:0/1362319414 wait complete. 2024-08-03T20:58:30.612 DEBUG:tasks.ceph_test_case:wait_until_true: success in 0s and 0 retries 2024-08-03T20:58:30.612 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-08-03T20:58:30.840 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:30.838+0000 7f3d90000640 1 -- 172.21.15.32:0/3710913751 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3d88144f90 msgr2=0x7f3d88074570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:58:30.840 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:30.838+0000 7f3d90000640 1 --2- 172.21.15.32:0/3710913751 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3d88144f90 0x7f3d88074570 secure :-1 s=READY pgs=1790 cs=0 l=1 rev1=1 crypto rx=0x7f3d74009830 tx=0x7f3d7402a510 comp rx=0 tx=0).stop 2024-08-03T20:58:30.841 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:30.838+0000 7f3d90000640 1 -- 172.21.15.32:0/3710913751 shutdown_connections 2024-08-03T20:58:30.841 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:30.838+0000 7f3d90000640 1 --2- 172.21.15.32:0/3710913751 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f3d8815fff0 0x7f3d88160400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:30.841 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:30.838+0000 7f3d90000640 1 --2- 172.21.15.32:0/3710913751 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3d88144f90 0x7f3d88074570 unknown :-1 s=CLOSED pgs=1790 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:30.841 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:30.838+0000 7f3d90000640 1 -- 172.21.15.32:0/3710913751 >> 172.21.15.32:0/3710913751 conn(0x7f3d880b1bd0 msgr2=0x7f3d880b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:58:30.841 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:30.838+0000 7f3d90000640 1 -- 172.21.15.32:0/3710913751 shutdown_connections 2024-08-03T20:58:30.841 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:30.838+0000 7f3d90000640 1 -- 172.21.15.32:0/3710913751 wait complete. 2024-08-03T20:58:30.842 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:30.840+0000 7f3d90000640 1 Processor -- start 2024-08-03T20:58:30.843 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:30.840+0000 7f3d90000640 1 -- start start 2024-08-03T20:58:30.847 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:30.842+0000 7f3d90000640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f3d8806e930 0x7f3d8813da60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:58:30.847 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:30.842+0000 7f3d90000640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f3d88144f90 0x7f3d880c86f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:58:30.847 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:30.842+0000 7f3d90000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3d8815fff0 0x7f3d881f67a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:58:30.847 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:30.842+0000 7f3d90000640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f3d880cf840 con 0x7f3d8815fff0 2024-08-03T20:58:30.848 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:30.842+0000 7f3d90000640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f3d88145f30 con 0x7f3d8806e930 2024-08-03T20:58:30.848 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:30.842+0000 7f3d90000640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7f3d880cf9c0 con 0x7f3d88144f90 2024-08-03T20:58:30.848 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:30.842+0000 7f3d8e000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3d8815fff0 0x7f3d881f67a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:58:30.848 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:30.842+0000 7f3d8e000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3d8815fff0 0x7f3d881f67a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:41492/0 (socket says 172.21.15.32:41492) 2024-08-03T20:58:30.848 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:30.842+0000 7f3d8e000640 1 -- 172.21.15.32:0/2567356100 learned_addr learned my addr 172.21.15.32:0/2567356100 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:58:30.848 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:30.842+0000 7f3d8d600640 1 --2- 172.21.15.32:0/2567356100 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f3d8806e930 0x7f3d8813da60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:58:30.849 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:30.842+0000 7f3d8cc00640 1 --2- 172.21.15.32:0/2567356100 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f3d88144f90 0x7f3d880c86f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:58:30.849 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:30.842+0000 7f3d8e000640 1 -- 172.21.15.32:0/2567356100 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f3d88144f90 msgr2=0x7f3d880c86f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:58:30.849 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:30.842+0000 7f3d8e000640 1 --2- 172.21.15.32:0/2567356100 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f3d88144f90 0x7f3d880c86f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:30.849 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:30.842+0000 7f3d8e000640 1 -- 172.21.15.32:0/2567356100 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f3d8806e930 msgr2=0x7f3d8813da60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:58:30.849 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:30.843+0000 7f3d8e000640 1 --2- 172.21.15.32:0/2567356100 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f3d8806e930 0x7f3d8813da60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:30.850 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:30.843+0000 7f3d8e000640 1 -- 172.21.15.32:0/2567356100 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3d8813dfa0 con 0x7f3d8815fff0 2024-08-03T20:58:30.850 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:30.843+0000 7f3d8e000640 1 --2- 172.21.15.32:0/2567356100 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3d8815fff0 0x7f3d881f67a0 secure :-1 s=READY pgs=1791 cs=0 l=1 rev1=1 crypto rx=0x7f3d8000e960 tx=0x7f3d8000ee20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:58:30.850 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:30.843+0000 7f3d85600640 1 -- 172.21.15.32:0/2567356100 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d80004d20 con 0x7f3d8815fff0 2024-08-03T20:58:30.850 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:30.843+0000 7f3d85600640 1 -- 172.21.15.32:0/2567356100 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f3d8000cac0 con 0x7f3d8815fff0 2024-08-03T20:58:30.850 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:30.843+0000 7f3d90000640 1 -- 172.21.15.32:0/2567356100 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3d88142bd0 con 0x7f3d8815fff0 2024-08-03T20:58:30.850 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:30.843+0000 7f3d85600640 1 -- 172.21.15.32:0/2567356100 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d80005020 con 0x7f3d8815fff0 2024-08-03T20:58:30.851 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:30.843+0000 7f3d90000640 1 -- 172.21.15.32:0/2567356100 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3d88143190 con 0x7f3d8815fff0 2024-08-03T20:58:30.851 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:30.844+0000 7f3d90000640 1 -- 172.21.15.32:0/2567356100 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3d88141290 con 0x7f3d8815fff0 2024-08-03T20:58:30.851 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:30.846+0000 7f3d85600640 1 -- 172.21.15.32:0/2567356100 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f3d800051c0 con 0x7f3d8815fff0 2024-08-03T20:58:30.853 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:30.846+0000 7f3d85600640 1 --2- 172.21.15.32:0/2567356100 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f3d68077010 0x7f3d68079520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:58:30.853 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:30.847+0000 7f3d85600640 1 -- 172.21.15.32:0/2567356100 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(115..115 src has 1..115) ==== 8434+0+0 (secure 0 0 0) 0x7f3d80099b60 con 0x7f3d8815fff0 2024-08-03T20:58:30.853 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:30.847+0000 7f3d8d600640 1 --2- 172.21.15.32:0/2567356100 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f3d68077010 0x7f3d68079520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:58:30.853 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:30.848+0000 7f3d8d600640 1 --2- 172.21.15.32:0/2567356100 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f3d68077010 0x7f3d68079520 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 crypto rx=0x7f3d7c000f60 tx=0x7f3d7c009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:58:30.853 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:30.851+0000 7f3d85600640 1 -- 172.21.15.32:0/2567356100 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7f3d88141290 con 0x7f3d8815fff0 2024-08-03T20:58:31.095 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:31.094+0000 7f3d90000640 1 -- 172.21.15.32:0/2567356100 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f3d88160c50 con 0x7f3d8815fff0 2024-08-03T20:58:31.096 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:31.095+0000 7f3d85600640 1 -- 172.21.15.32:0/2567356100 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 260 v260) ==== 95+0+4391 (secure 0 0 0) 0x7f3d8813dfa0 con 0x7f3d8815fff0 2024-08-03T20:58:31.096 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-03T20:58:31.096 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":260,"btime":"2024-08-03T20:58:30:065005+0000","default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6439,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6835/531080160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6834","nonce":531080160},{"type":"v1","addr":"172.21.15.118:6835","nonce":531080160}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":84},{"gid":6454,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6837/2796882319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6836","nonce":2796882319},{"type":"v1","addr":"172.21.15.118:6837","nonce":2796882319}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":86},{"gid":6813,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6834/2686933890","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":2686933890},{"type":"v1","addr":"172.21.15.32:6834","nonce":2686933890}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":85}],"filesystems":[{"mdsmap":{"epoch":260,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-08-03T20:54:26.950518+0000","modified":"2024-08-03T20:58:30.064912+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":115,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":6}]} 2024-08-03T20:58:31.097 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 260 2024-08-03T20:58:31.099 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:31.097+0000 7f3d90000640 1 -- 172.21.15.32:0/2567356100 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f3d68077010 msgr2=0x7f3d68079520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:58:31.099 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:31.097+0000 7f3d90000640 1 --2- 172.21.15.32:0/2567356100 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f3d68077010 0x7f3d68079520 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 crypto rx=0x7f3d7c000f60 tx=0x7f3d7c009040 comp rx=0 tx=0).stop 2024-08-03T20:58:31.099 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:31.097+0000 7f3d90000640 1 -- 172.21.15.32:0/2567356100 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3d8815fff0 msgr2=0x7f3d881f67a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:58:31.099 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:31.097+0000 7f3d90000640 1 --2- 172.21.15.32:0/2567356100 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3d8815fff0 0x7f3d881f67a0 secure :-1 s=READY pgs=1791 cs=0 l=1 rev1=1 crypto rx=0x7f3d8000e960 tx=0x7f3d8000ee20 comp rx=0 tx=0).stop 2024-08-03T20:58:31.099 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:31.097+0000 7f3d90000640 1 -- 172.21.15.32:0/2567356100 shutdown_connections 2024-08-03T20:58:31.099 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:31.097+0000 7f3d90000640 1 --2- 172.21.15.32:0/2567356100 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f3d68077010 0x7f3d68079520 unknown :-1 s=CLOSED pgs=727 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:31.099 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:31.097+0000 7f3d90000640 1 --2- 172.21.15.32:0/2567356100 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3d8815fff0 0x7f3d881f67a0 unknown :-1 s=CLOSED pgs=1791 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:31.099 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:31.097+0000 7f3d90000640 1 --2- 172.21.15.32:0/2567356100 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f3d88144f90 0x7f3d880c86f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:31.099 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:31.097+0000 7f3d90000640 1 --2- 172.21.15.32:0/2567356100 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f3d8806e930 0x7f3d8813da60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:31.100 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:31.097+0000 7f3d90000640 1 -- 172.21.15.32:0/2567356100 >> 172.21.15.32:0/2567356100 conn(0x7f3d880b1bd0 msgr2=0x7f3d8815d010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:58:31.100 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:31.097+0000 7f3d90000640 1 -- 172.21.15.32:0/2567356100 shutdown_connections 2024-08-03T20:58:31.100 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:31.097+0000 7f3d90000640 1 -- 172.21.15.32:0/2567356100 wait complete. 2024-08-03T20:58:31.125 DEBUG:tasks.ceph_test_case:wait_until_true: success in 0s and 0 retries 2024-08-03T20:58:31.125 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-08-03T20:58:31.355 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:31.353+0000 7fb781600640 1 -- 172.21.15.32:0/2947185686 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb77c15fff0 msgr2=0x7fb77c160400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:58:31.355 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:31.353+0000 7fb781600640 1 --2- 172.21.15.32:0/2947185686 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb77c15fff0 0x7fb77c160400 secure :-1 s=READY pgs=1793 cs=0 l=1 rev1=1 crypto rx=0x7fb774005fd0 tx=0x7fb77402a8b0 comp rx=0 tx=0).stop 2024-08-03T20:58:31.355 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:31.353+0000 7fb781600640 1 -- 172.21.15.32:0/2947185686 shutdown_connections 2024-08-03T20:58:31.355 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:31.353+0000 7fb781600640 1 --2- 172.21.15.32:0/2947185686 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb77c15fff0 0x7fb77c160400 unknown :-1 s=CLOSED pgs=1793 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:31.356 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:31.353+0000 7fb781600640 1 --2- 172.21.15.32:0/2947185686 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fb77c144f90 0x7fb77c074570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:31.356 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:31.353+0000 7fb781600640 1 -- 172.21.15.32:0/2947185686 >> 172.21.15.32:0/2947185686 conn(0x7fb77c0b1bd0 msgr2=0x7fb77c0b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:58:31.356 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:31.353+0000 7fb781600640 1 -- 172.21.15.32:0/2947185686 shutdown_connections 2024-08-03T20:58:31.356 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:31.353+0000 7fb781600640 1 -- 172.21.15.32:0/2947185686 wait complete. 2024-08-03T20:58:31.357 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:31.355+0000 7fb781600640 1 Processor -- start 2024-08-03T20:58:31.357 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:31.356+0000 7fb781600640 1 -- start start 2024-08-03T20:58:31.360 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:31.357+0000 7fb781600640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb77c06e930 0x7fb77c15f960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:58:31.360 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:31.357+0000 7fb781600640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fb77c144f90 0x7fb77c1327b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:58:31.360 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:31.357+0000 7fb781600640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb77c15fff0 0x7fb77c1437a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:58:31.360 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:31.357+0000 7fb781600640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fb77c0cf840 con 0x7fb77c06e930 2024-08-03T20:58:31.360 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:31.357+0000 7fb781600640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7fb77c145f30 con 0x7fb77c15fff0 2024-08-03T20:58:31.361 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:31.357+0000 7fb781600640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7fb77c0cf9c0 con 0x7fb77c144f90 2024-08-03T20:58:31.361 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:31.357+0000 7fb77aa00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb77c06e930 0x7fb77c15f960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:58:31.361 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:31.357+0000 7fb77aa00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb77c06e930 0x7fb77c15f960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:41520/0 (socket says 172.21.15.32:41520) 2024-08-03T20:58:31.361 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:31.357+0000 7fb77aa00640 1 -- 172.21.15.32:0/3327274177 learned_addr learned my addr 172.21.15.32:0/3327274177 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:58:31.361 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:31.357+0000 7fb77a000640 1 --2- 172.21.15.32:0/3327274177 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fb77c144f90 0x7fb77c1327b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:58:31.361 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:31.358+0000 7fb77b400640 1 --2- 172.21.15.32:0/3327274177 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb77c15fff0 0x7fb77c1437a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:58:31.361 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:31.358+0000 7fb77aa00640 1 -- 172.21.15.32:0/3327274177 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fb77c144f90 msgr2=0x7fb77c1327b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:58:31.361 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:31.358+0000 7fb77aa00640 1 --2- 172.21.15.32:0/3327274177 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fb77c144f90 0x7fb77c1327b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:31.362 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:31.358+0000 7fb77aa00640 1 -- 172.21.15.32:0/3327274177 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb77c15fff0 msgr2=0x7fb77c1437a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:58:31.362 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:31.358+0000 7fb77aa00640 1 --2- 172.21.15.32:0/3327274177 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb77c15fff0 0x7fb77c1437a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:31.362 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:31.358+0000 7fb77aa00640 1 -- 172.21.15.32:0/3327274177 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb77c132cf0 con 0x7fb77c06e930 2024-08-03T20:58:31.362 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:31.358+0000 7fb77b400640 1 --2- 172.21.15.32:0/3327274177 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb77c15fff0 0x7fb77c1437a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-03T20:58:31.362 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:31.358+0000 7fb77aa00640 1 --2- 172.21.15.32:0/3327274177 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb77c06e930 0x7fb77c15f960 secure :-1 s=READY pgs=1794 cs=0 l=1 rev1=1 crypto rx=0x7fb76c00c3f0 tx=0x7fb76c00c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:58:31.362 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:31.358+0000 7fb772a00640 1 -- 172.21.15.32:0/3327274177 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb76c017070 con 0x7fb77c06e930 2024-08-03T20:58:31.362 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:31.358+0000 7fb772a00640 1 -- 172.21.15.32:0/3327274177 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fb76c0035f0 con 0x7fb77c06e930 2024-08-03T20:58:31.362 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:31.358+0000 7fb781600640 1 -- 172.21.15.32:0/3327274177 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb77c13dcb0 con 0x7fb77c06e930 2024-08-03T20:58:31.363 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:31.358+0000 7fb772a00640 1 -- 172.21.15.32:0/3327274177 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb76c003a40 con 0x7fb77c06e930 2024-08-03T20:58:31.363 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:31.358+0000 7fb781600640 1 -- 172.21.15.32:0/3327274177 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb77c1598a0 con 0x7fb77c06e930 2024-08-03T20:58:31.364 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:31.359+0000 7fb781600640 1 -- 172.21.15.32:0/3327274177 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb77c160bc0 con 0x7fb77c06e930 2024-08-03T20:58:31.364 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:31.361+0000 7fb772a00640 1 -- 172.21.15.32:0/3327274177 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7fb76c01a070 con 0x7fb77c06e930 2024-08-03T20:58:31.364 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:31.362+0000 7fb772a00640 1 --2- 172.21.15.32:0/3327274177 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fb75c077040 0x7fb75c079550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:58:31.364 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:31.362+0000 7fb772a00640 1 -- 172.21.15.32:0/3327274177 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(115..115 src has 1..115) ==== 8434+0+0 (secure 0 0 0) 0x7fb76c0987d0 con 0x7fb77c06e930 2024-08-03T20:58:31.364 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:31.362+0000 7fb77a000640 1 --2- 172.21.15.32:0/3327274177 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fb75c077040 0x7fb75c079550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:58:31.365 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:31.363+0000 7fb77a000640 1 --2- 172.21.15.32:0/3327274177 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fb75c077040 0x7fb75c079550 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7fb77c143390 tx=0x7fb768005f90 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:58:31.367 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:31.366+0000 7fb772a00640 1 -- 172.21.15.32:0/3327274177 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7fb76c060820 con 0x7fb77c06e930 2024-08-03T20:58:31.609 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:31.608+0000 7fb781600640 1 -- 172.21.15.32:0/3327274177 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb77c0703d0 con 0x7fb77c06e930 2024-08-03T20:58:31.610 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:31.609+0000 7fb772a00640 1 -- 172.21.15.32:0/3327274177 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 260 v260) ==== 95+0+4391 (secure 0 0 0) 0x7fb76c069140 con 0x7fb77c06e930 2024-08-03T20:58:31.610 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-03T20:58:31.610 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":260,"btime":"2024-08-03T20:58:30:065005+0000","default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6439,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6835/531080160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6834","nonce":531080160},{"type":"v1","addr":"172.21.15.118:6835","nonce":531080160}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":84},{"gid":6454,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6837/2796882319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6836","nonce":2796882319},{"type":"v1","addr":"172.21.15.118:6837","nonce":2796882319}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":86},{"gid":6813,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6834/2686933890","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":2686933890},{"type":"v1","addr":"172.21.15.32:6834","nonce":2686933890}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":85}],"filesystems":[{"mdsmap":{"epoch":260,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-08-03T20:54:26.950518+0000","modified":"2024-08-03T20:58:30.064912+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":115,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":6}]} 2024-08-03T20:58:31.611 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 260 2024-08-03T20:58:31.613 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:31.611+0000 7fb781600640 1 -- 172.21.15.32:0/3327274177 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fb75c077040 msgr2=0x7fb75c079550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:58:31.613 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:31.611+0000 7fb781600640 1 --2- 172.21.15.32:0/3327274177 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fb75c077040 0x7fb75c079550 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7fb77c143390 tx=0x7fb768005f90 comp rx=0 tx=0).stop 2024-08-03T20:58:31.613 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:31.611+0000 7fb781600640 1 -- 172.21.15.32:0/3327274177 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb77c06e930 msgr2=0x7fb77c15f960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:58:31.613 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:31.611+0000 7fb781600640 1 --2- 172.21.15.32:0/3327274177 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb77c06e930 0x7fb77c15f960 secure :-1 s=READY pgs=1794 cs=0 l=1 rev1=1 crypto rx=0x7fb76c00c3f0 tx=0x7fb76c00c8b0 comp rx=0 tx=0).stop 2024-08-03T20:58:31.613 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:31.611+0000 7fb781600640 1 -- 172.21.15.32:0/3327274177 shutdown_connections 2024-08-03T20:58:31.613 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:31.611+0000 7fb781600640 1 --2- 172.21.15.32:0/3327274177 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fb75c077040 0x7fb75c079550 unknown :-1 s=CLOSED pgs=728 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:31.613 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:31.611+0000 7fb781600640 1 --2- 172.21.15.32:0/3327274177 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb77c15fff0 0x7fb77c1437a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:31.613 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:31.611+0000 7fb781600640 1 --2- 172.21.15.32:0/3327274177 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fb77c144f90 0x7fb77c1327b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:31.613 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:31.611+0000 7fb781600640 1 --2- 172.21.15.32:0/3327274177 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb77c06e930 0x7fb77c15f960 unknown :-1 s=CLOSED pgs=1794 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:31.614 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:31.611+0000 7fb781600640 1 -- 172.21.15.32:0/3327274177 >> 172.21.15.32:0/3327274177 conn(0x7fb77c0b1bd0 msgr2=0x7fb77c0d0920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:58:31.614 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:31.611+0000 7fb781600640 1 -- 172.21.15.32:0/3327274177 shutdown_connections 2024-08-03T20:58:31.614 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:31.611+0000 7fb781600640 1 -- 172.21.15.32:0/3327274177 wait complete. 2024-08-03T20:58:31.635 INFO:tasks.cephfs.test_damage:Result: Mutation 'Truncate 400.00000000' led to DAMAGED state 2024-08-03T20:58:31.635 INFO:tasks.cephfs.test_damage:Applying mutation 'Truncate 100.00000000.inode' 2024-08-03T20:58:31.635 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-08-03T20:58:31.635 DEBUG:teuthology.orchestra.run.smithi032:> dd if=/proc/self/mounts of=/dev/stdout 2024-08-03T20:58:31.671 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053453,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6499208k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 none /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249604k,nr_inodes=812401,mode=700,uid=1000,gid=1316 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6499208k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-08-03T20:58:31.671 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi032.front.sepia.ceph.com 2024-08-03T20:58:31.671 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-08-03T20:58:31.671 DEBUG:teuthology.orchestra.run.smithi032:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-08-03T20:58:31.746 INFO:teuthology.orchestra.run.smithi032.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.0': No such file or directory 2024-08-03T20:58:31.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-03T20:58:31.747 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-08-03T20:58:32.018 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:32.016+0000 7fb06b800640 1 -- 172.21.15.32:0/556903254 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb06406e930 msgr2=0x7fb064144f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:58:32.019 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:32.016+0000 7fb06b800640 1 --2- 172.21.15.32:0/556903254 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb06406e930 0x7fb064144f90 secure :-1 s=READY pgs=1795 cs=0 l=1 rev1=1 crypto rx=0x7fb058005fd0 tx=0x7fb05802a540 comp rx=0 tx=0).stop 2024-08-03T20:58:32.019 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:32.016+0000 7fb06b800640 1 -- 172.21.15.32:0/556903254 shutdown_connections 2024-08-03T20:58:32.019 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:32.017+0000 7fb06b800640 1 --2- 172.21.15.32:0/556903254 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fb064073e60 0x7fb064074270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:32.019 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:32.017+0000 7fb06b800640 1 --2- 172.21.15.32:0/556903254 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb06406e930 0x7fb064144f90 unknown :-1 s=CLOSED pgs=1795 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:32.019 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:32.017+0000 7fb06b800640 1 -- 172.21.15.32:0/556903254 >> 172.21.15.32:0/556903254 conn(0x7fb0640b1bd0 msgr2=0x7fb0640b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:58:32.019 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:32.017+0000 7fb06b800640 1 -- 172.21.15.32:0/556903254 shutdown_connections 2024-08-03T20:58:32.019 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:32.017+0000 7fb06b800640 1 -- 172.21.15.32:0/556903254 wait complete. 2024-08-03T20:58:32.020 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:32.019+0000 7fb06b800640 1 Processor -- start 2024-08-03T20:58:32.020 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:32.019+0000 7fb06b800640 1 -- start start 2024-08-03T20:58:32.023 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:32.021+0000 7fb06b800640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fb06406e930 0x7fb0640b3950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:58:32.023 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:32.021+0000 7fb06b800640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb064073e60 0x7fb06413d8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:58:32.023 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:32.021+0000 7fb06b800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb0640747b0 0x7fb064142530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:58:32.023 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:32.021+0000 7fb06b800640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fb0640cf9c0 con 0x7fb0640747b0 2024-08-03T20:58:32.023 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:32.021+0000 7fb06b800640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7fb0640cf840 con 0x7fb064073e60 2024-08-03T20:58:32.023 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:32.021+0000 7fb06b800640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7fb0640ccff0 con 0x7fb06406e930 2024-08-03T20:58:32.023 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:32.021+0000 7fb069800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb0640747b0 0x7fb064142530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:58:32.023 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:32.021+0000 7fb069800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb0640747b0 0x7fb064142530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:41542/0 (socket says 172.21.15.32:41542) 2024-08-03T20:58:32.023 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:32.021+0000 7fb069800640 1 -- 172.21.15.32:0/1867633302 learned_addr learned my addr 172.21.15.32:0/1867633302 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:58:32.024 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:32.021+0000 7fb068e00640 1 --2- 172.21.15.32:0/1867633302 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fb06406e930 0x7fb0640b3950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:58:32.024 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:32.021+0000 7fb063e00640 1 --2- 172.21.15.32:0/1867633302 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb064073e60 0x7fb06413d8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:58:32.024 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:32.021+0000 7fb069800640 1 -- 172.21.15.32:0/1867633302 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fb06406e930 msgr2=0x7fb0640b3950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:58:32.024 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:32.021+0000 7fb069800640 1 --2- 172.21.15.32:0/1867633302 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fb06406e930 0x7fb0640b3950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:32.024 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:32.021+0000 7fb069800640 1 -- 172.21.15.32:0/1867633302 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb064073e60 msgr2=0x7fb06413d8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:58:32.024 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:32.021+0000 7fb069800640 1 --2- 172.21.15.32:0/1867633302 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb064073e60 0x7fb06413d8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:32.024 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:32.021+0000 7fb069800640 1 -- 172.21.15.32:0/1867633302 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb0640b4800 con 0x7fb0640747b0 2024-08-03T20:58:32.024 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:32.022+0000 7fb069800640 1 --2- 172.21.15.32:0/1867633302 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb0640747b0 0x7fb064142530 secure :-1 s=READY pgs=1796 cs=0 l=1 rev1=1 crypto rx=0x7fb05c00c700 tx=0x7fb05c00cbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:58:32.025 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:32.022+0000 7fb060c00640 1 -- 172.21.15.32:0/1867633302 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb05c003b70 con 0x7fb0640747b0 2024-08-03T20:58:32.026 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:32.022+0000 7fb060c00640 1 -- 172.21.15.32:0/1867633302 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fb05c003d10 con 0x7fb0640747b0 2024-08-03T20:58:32.026 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:32.022+0000 7fb060c00640 1 -- 172.21.15.32:0/1867633302 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb05c00aab0 con 0x7fb0640747b0 2024-08-03T20:58:32.026 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:32.022+0000 7fb06b800640 1 -- 172.21.15.32:0/1867633302 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb06415be30 con 0x7fb0640747b0 2024-08-03T20:58:32.026 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:32.022+0000 7fb06b800640 1 -- 172.21.15.32:0/1867633302 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb0640b4da0 con 0x7fb0640747b0 2024-08-03T20:58:32.026 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:32.023+0000 7fb06b800640 1 -- 172.21.15.32:0/1867633302 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb0640cd510 con 0x7fb0640747b0 2024-08-03T20:58:32.026 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:32.024+0000 7fb060c00640 1 -- 172.21.15.32:0/1867633302 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7fb05c00ac50 con 0x7fb0640747b0 2024-08-03T20:58:32.027 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:32.025+0000 7fb060c00640 1 --2- 172.21.15.32:0/1867633302 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fb044077040 0x7fb044079550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:58:32.027 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:32.025+0000 7fb060c00640 1 -- 172.21.15.32:0/1867633302 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(115..115 src has 1..115) ==== 8434+0+0 (secure 0 0 0) 0x7fb05c0990b0 con 0x7fb0640747b0 2024-08-03T20:58:32.027 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:32.026+0000 7fb068e00640 1 --2- 172.21.15.32:0/1867633302 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fb044077040 0x7fb044079550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:58:32.028 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:32.026+0000 7fb068e00640 1 --2- 172.21.15.32:0/1867633302 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fb044077040 0x7fb044079550 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7fb058033000 tx=0x7fb058004730 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:58:32.031 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:32.030+0000 7fb060c00640 1 -- 172.21.15.32:0/1867633302 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7fb05c061200 con 0x7fb0640747b0 2024-08-03T20:58:32.271 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:32.270+0000 7fb06b800640 1 -- 172.21.15.32:0/1867633302 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7fb0640b4f70 con 0x7fb0640747b0 2024-08-03T20:58:32.325 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:32.324+0000 7fb060c00640 1 -- 172.21.15.32:0/1867633302 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v261) ==== 161+0+0 (secure 0 0 0) 0x7fb05c065750 con 0x7fb0640747b0 2024-08-03T20:58:32.326 INFO:teuthology.orchestra.run.smithi032.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-08-03T20:58:32.328 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:32.326+0000 7fb06b800640 1 -- 172.21.15.32:0/1867633302 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fb044077040 msgr2=0x7fb044079550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:58:32.328 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:32.326+0000 7fb06b800640 1 --2- 172.21.15.32:0/1867633302 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fb044077040 0x7fb044079550 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7fb058033000 tx=0x7fb058004730 comp rx=0 tx=0).stop 2024-08-03T20:58:32.328 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:32.326+0000 7fb06b800640 1 -- 172.21.15.32:0/1867633302 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb0640747b0 msgr2=0x7fb064142530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:58:32.328 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:32.326+0000 7fb06b800640 1 --2- 172.21.15.32:0/1867633302 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb0640747b0 0x7fb064142530 secure :-1 s=READY pgs=1796 cs=0 l=1 rev1=1 crypto rx=0x7fb05c00c700 tx=0x7fb05c00cbc0 comp rx=0 tx=0).stop 2024-08-03T20:58:32.328 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:32.326+0000 7fb06b800640 1 -- 172.21.15.32:0/1867633302 shutdown_connections 2024-08-03T20:58:32.328 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:32.326+0000 7fb06b800640 1 --2- 172.21.15.32:0/1867633302 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fb044077040 0x7fb044079550 unknown :-1 s=CLOSED pgs=729 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:32.329 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:32.326+0000 7fb06b800640 1 --2- 172.21.15.32:0/1867633302 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb064073e60 0x7fb06413d8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:32.329 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:32.326+0000 7fb06b800640 1 --2- 172.21.15.32:0/1867633302 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb0640747b0 0x7fb064142530 unknown :-1 s=CLOSED pgs=1796 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:32.329 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:32.326+0000 7fb06b800640 1 --2- 172.21.15.32:0/1867633302 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fb06406e930 0x7fb0640b3950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:32.329 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:32.326+0000 7fb06b800640 1 -- 172.21.15.32:0/1867633302 >> 172.21.15.32:0/1867633302 conn(0x7fb0640b1bd0 msgr2=0x7fb0640b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:58:32.329 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:32.326+0000 7fb06b800640 1 -- 172.21.15.32:0/1867633302 shutdown_connections 2024-08-03T20:58:32.329 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:32.326+0000 7fb06b800640 1 -- 172.21.15.32:0/1867633302 wait complete. 2024-08-03T20:58:32.350 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mds repaired 0 2024-08-03T20:58:32.580 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:32.578+0000 7fd4fe600640 1 -- 172.21.15.32:0/3982581962 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd4f806e930 msgr2=0x7fd4f8144f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:58:32.580 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:32.578+0000 7fd4fe600640 1 --2- 172.21.15.32:0/3982581962 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd4f806e930 0x7fd4f8144f90 secure :-1 s=READY pgs=1797 cs=0 l=1 rev1=1 crypto rx=0x7fd4e8005fd0 tx=0x7fd4e802a4a0 comp rx=0 tx=0).stop 2024-08-03T20:58:32.580 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:32.578+0000 7fd4fe600640 1 -- 172.21.15.32:0/3982581962 shutdown_connections 2024-08-03T20:58:32.580 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:32.578+0000 7fd4fe600640 1 --2- 172.21.15.32:0/3982581962 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fd4f8073e60 0x7fd4f8074270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:32.580 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:32.578+0000 7fd4fe600640 1 --2- 172.21.15.32:0/3982581962 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd4f806e930 0x7fd4f8144f90 unknown :-1 s=CLOSED pgs=1797 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:32.580 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:32.578+0000 7fd4fe600640 1 -- 172.21.15.32:0/3982581962 >> 172.21.15.32:0/3982581962 conn(0x7fd4f80b1bd0 msgr2=0x7fd4f80b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:58:32.580 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:32.578+0000 7fd4fe600640 1 -- 172.21.15.32:0/3982581962 shutdown_connections 2024-08-03T20:58:32.581 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:32.578+0000 7fd4fe600640 1 -- 172.21.15.32:0/3982581962 wait complete. 2024-08-03T20:58:32.582 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:32.580+0000 7fd4fe600640 1 Processor -- start 2024-08-03T20:58:32.582 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:32.581+0000 7fd4fe600640 1 -- start start 2024-08-03T20:58:32.585 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:32.582+0000 7fd4fe600640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fd4f806e930 0x7fd4f8143e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:58:32.585 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:32.582+0000 7fd4fe600640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd4f8073e60 0x7fd4f80ae3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:58:32.585 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:32.582+0000 7fd4fe600640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fd4f80747b0 0x7fd4f80ae8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:58:32.585 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:32.582+0000 7fd4fe600640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fd4f80cf9c0 con 0x7fd4f8073e60 2024-08-03T20:58:32.585 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:32.582+0000 7fd4fe600640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7fd4f80cf840 con 0x7fd4f806e930 2024-08-03T20:58:32.585 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:32.582+0000 7fd4fe600640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7fd4f80ccff0 con 0x7fd4f80747b0 2024-08-03T20:58:32.585 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:32.582+0000 7fd4efe00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd4f8073e60 0x7fd4f80ae3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:58:32.585 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:32.582+0000 7fd4efe00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd4f8073e60 0x7fd4f80ae3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:41566/0 (socket says 172.21.15.32:41566) 2024-08-03T20:58:32.586 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:32.582+0000 7fd4efe00640 1 -- 172.21.15.32:0/640640780 learned_addr learned my addr 172.21.15.32:0/640640780 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:58:32.586 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:32.582+0000 7fd4f7400640 1 --2- 172.21.15.32:0/640640780 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fd4f806e930 0x7fd4f8143e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:58:32.586 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:32.582+0000 7fd4f7e00640 1 --2- 172.21.15.32:0/640640780 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fd4f80747b0 0x7fd4f80ae8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:58:32.586 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:32.583+0000 7fd4efe00640 1 -- 172.21.15.32:0/640640780 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fd4f80747b0 msgr2=0x7fd4f80ae8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:58:32.586 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:32.583+0000 7fd4efe00640 1 --2- 172.21.15.32:0/640640780 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fd4f80747b0 0x7fd4f80ae8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:32.586 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:32.583+0000 7fd4efe00640 1 -- 172.21.15.32:0/640640780 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fd4f806e930 msgr2=0x7fd4f8143e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:58:32.586 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:32.583+0000 7fd4efe00640 1 --2- 172.21.15.32:0/640640780 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fd4f806e930 0x7fd4f8143e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:32.586 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:32.583+0000 7fd4efe00640 1 -- 172.21.15.32:0/640640780 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd4f813baf0 con 0x7fd4f8073e60 2024-08-03T20:58:32.586 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:32.583+0000 7fd4f7e00640 1 --2- 172.21.15.32:0/640640780 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fd4f80747b0 0x7fd4f80ae8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-03T20:58:32.586 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:32.583+0000 7fd4efe00640 1 --2- 172.21.15.32:0/640640780 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd4f8073e60 0x7fd4f80ae3b0 secure :-1 s=READY pgs=1798 cs=0 l=1 rev1=1 crypto rx=0x7fd4e000d700 tx=0x7fd4e000dbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:58:32.586 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:32.583+0000 7fd4ef400640 1 -- 172.21.15.32:0/640640780 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd4e0017070 con 0x7fd4f8073e60 2024-08-03T20:58:32.587 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:32.583+0000 7fd4ef400640 1 -- 172.21.15.32:0/640640780 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fd4e0005310 con 0x7fd4f8073e60 2024-08-03T20:58:32.587 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:32.583+0000 7fd4ef400640 1 -- 172.21.15.32:0/640640780 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd4e0009820 con 0x7fd4f8073e60 2024-08-03T20:58:32.587 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:32.583+0000 7fd4fe600640 1 -- 172.21.15.32:0/640640780 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd4f8144350 con 0x7fd4f8073e60 2024-08-03T20:58:32.587 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:32.583+0000 7fd4fe600640 1 -- 172.21.15.32:0/640640780 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd4f80b4c00 con 0x7fd4f8073e60 2024-08-03T20:58:32.587 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:32.585+0000 7fd4fe600640 1 -- 172.21.15.32:0/640640780 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd4f80b4970 con 0x7fd4f8073e60 2024-08-03T20:58:32.587 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:32.586+0000 7fd4ef400640 1 -- 172.21.15.32:0/640640780 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7fd4e00099c0 con 0x7fd4f8073e60 2024-08-03T20:58:32.588 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:32.587+0000 7fd4ef400640 1 --2- 172.21.15.32:0/640640780 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fd4d4077040 0x7fd4d4079550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:58:32.588 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:32.587+0000 7fd4ef400640 1 -- 172.21.15.32:0/640640780 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(115..115 src has 1..115) ==== 8434+0+0 (secure 0 0 0) 0x7fd4e0099460 con 0x7fd4f8073e60 2024-08-03T20:58:32.589 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:32.587+0000 7fd4f7400640 1 --2- 172.21.15.32:0/640640780 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fd4d4077040 0x7fd4d4079550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:58:32.589 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:32.588+0000 7fd4f7400640 1 --2- 172.21.15.32:0/640640780 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fd4d4077040 0x7fd4d4079550 secure :-1 s=READY pgs=730 cs=0 l=1 rev1=1 crypto rx=0x7fd4e8033000 tx=0x7fd4e8004730 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:58:32.592 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:32.591+0000 7fd4ef400640 1 -- 172.21.15.32:0/640640780 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7fd4e0061510 con 0x7fd4f8073e60 2024-08-03T20:58:32.875 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:32.874+0000 7fd4fe600640 1 -- 172.21.15.32:0/640640780 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "mds repaired", "role": "0"} v 0) -- 0x7fd4f80cd570 con 0x7fd4f8073e60 2024-08-03T20:58:33.339 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:33.335+0000 7fd4ef400640 1 -- 172.21.15.32:0/640640780 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "mds repaired", "role": "0"}]=0 repaired: restoring rank 6:0 v262) ==== 101+0+0 (secure 0 0 0) 0x7fd4e0065a60 con 0x7fd4f8073e60 2024-08-03T20:58:33.339 INFO:teuthology.orchestra.run.smithi032.stderr:repaired: restoring rank 6:0 2024-08-03T20:58:33.340 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:33.338+0000 7fd4fe600640 1 -- 172.21.15.32:0/640640780 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fd4d4077040 msgr2=0x7fd4d4079550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:58:33.340 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:33.338+0000 7fd4fe600640 1 --2- 172.21.15.32:0/640640780 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fd4d4077040 0x7fd4d4079550 secure :-1 s=READY pgs=730 cs=0 l=1 rev1=1 crypto rx=0x7fd4e8033000 tx=0x7fd4e8004730 comp rx=0 tx=0).stop 2024-08-03T20:58:33.340 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:33.338+0000 7fd4fe600640 1 -- 172.21.15.32:0/640640780 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd4f8073e60 msgr2=0x7fd4f80ae3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:58:33.340 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:33.338+0000 7fd4fe600640 1 --2- 172.21.15.32:0/640640780 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd4f8073e60 0x7fd4f80ae3b0 secure :-1 s=READY pgs=1798 cs=0 l=1 rev1=1 crypto rx=0x7fd4e000d700 tx=0x7fd4e000dbc0 comp rx=0 tx=0).stop 2024-08-03T20:58:33.340 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:33.338+0000 7fd4fe600640 1 -- 172.21.15.32:0/640640780 shutdown_connections 2024-08-03T20:58:33.340 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:33.338+0000 7fd4fe600640 1 --2- 172.21.15.32:0/640640780 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fd4d4077040 0x7fd4d4079550 unknown :-1 s=CLOSED pgs=730 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:33.340 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:33.338+0000 7fd4fe600640 1 --2- 172.21.15.32:0/640640780 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fd4f80747b0 0x7fd4f80ae8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:33.341 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:33.338+0000 7fd4fe600640 1 --2- 172.21.15.32:0/640640780 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd4f8073e60 0x7fd4f80ae3b0 unknown :-1 s=CLOSED pgs=1798 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:33.341 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:33.338+0000 7fd4fe600640 1 --2- 172.21.15.32:0/640640780 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fd4f806e930 0x7fd4f8143e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:33.341 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:33.338+0000 7fd4fe600640 1 -- 172.21.15.32:0/640640780 >> 172.21.15.32:0/640640780 conn(0x7fd4f80b1bd0 msgr2=0x7fd4f8160890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:58:33.341 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:33.338+0000 7fd4fe600640 1 -- 172.21.15.32:0/640640780 shutdown_connections 2024-08-03T20:58:33.341 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:33.338+0000 7fd4fe600640 1 -- 172.21.15.32:0/640640780 wait complete. 2024-08-03T20:58:33.363 DEBUG:teuthology.orchestra.run.smithi032:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata import - 2024-08-03T20:58:33.505 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:33.503+0000 7f355d25ef40 1 Processor -- start 2024-08-03T20:58:33.505 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:33.503+0000 7f355d25ef40 1 -- start start 2024-08-03T20:58:33.507 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:33.505+0000 7f355d25ef40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x555e4272cb40 0x555e426e75e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:58:33.507 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:33.505+0000 7f355d25ef40 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555e4279fbe0 0x555e427bbaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:58:33.508 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:33.505+0000 7f355d25ef40 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x555e426e7b20 0x555e427bc030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:58:33.508 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:33.505+0000 7f355d25ef40 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x555e42635e30 con 0x555e4279fbe0 2024-08-03T20:58:33.508 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:33.505+0000 7f355d25ef40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x555e42693bf0 con 0x555e4272cb40 2024-08-03T20:58:33.508 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:33.505+0000 7f355d25ef40 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x555e427a0510 con 0x555e426e7b20 2024-08-03T20:58:33.508 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:33.505+0000 7f3555a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555e4279fbe0 0x555e427bbaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:58:33.508 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:33.505+0000 7f3555a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555e4279fbe0 0x555e427bbaf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:41570/0 (socket says 172.21.15.32:41570) 2024-08-03T20:58:33.508 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:33.505+0000 7f3555a00640 1 -- 172.21.15.32:0/4053519628 learned_addr learned my addr 172.21.15.32:0/4053519628 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:58:33.508 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:33.505+0000 7f3556400640 1 --2- 172.21.15.32:0/4053519628 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x555e4272cb40 0x555e426e75e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:58:33.508 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:33.505+0000 7f3556e00640 1 --2- 172.21.15.32:0/4053519628 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x555e426e7b20 0x555e427bc030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:58:33.508 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:33.505+0000 7f3555a00640 1 -- 172.21.15.32:0/4053519628 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x555e426e7b20 msgr2=0x555e427bc030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:58:33.508 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:33.505+0000 7f3555a00640 1 --2- 172.21.15.32:0/4053519628 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x555e426e7b20 0x555e427bc030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:33.508 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:33.505+0000 7f3555a00640 1 -- 172.21.15.32:0/4053519628 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x555e4272cb40 msgr2=0x555e426e75e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:58:33.508 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:33.505+0000 7f3555a00640 1 --2- 172.21.15.32:0/4053519628 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x555e4272cb40 0x555e426e75e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:33.509 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:33.505+0000 7f3555a00640 1 -- 172.21.15.32:0/4053519628 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555e427bfb60 con 0x555e4279fbe0 2024-08-03T20:58:33.509 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:33.505+0000 7f3556e00640 1 --2- 172.21.15.32:0/4053519628 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x555e426e7b20 0x555e427bc030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-03T20:58:33.509 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:33.506+0000 7f3555a00640 1 --2- 172.21.15.32:0/4053519628 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555e4279fbe0 0x555e427bbaf0 secure :-1 s=READY pgs=1800 cs=0 l=1 rev1=1 crypto rx=0x7f3544001d20 tx=0x7f354400b5b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fb88ca9f353e04c1 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:58:33.509 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:33.506+0000 7f354fe00640 1 -- 172.21.15.32:0/4053519628 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f354400bd80 con 0x555e4279fbe0 2024-08-03T20:58:33.509 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:33.506+0000 7f354fe00640 1 -- 172.21.15.32:0/4053519628 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f354400f400 con 0x555e4279fbe0 2024-08-03T20:58:33.509 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:33.506+0000 7f354fe00640 1 -- 172.21.15.32:0/4053519628 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f354400f890 con 0x555e4279fbe0 2024-08-03T20:58:33.510 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:33.506+0000 7f355d25ef40 1 -- 172.21.15.32:0/4053519628 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555e4279fbe0 msgr2=0x555e427bbaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:58:33.510 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:33.506+0000 7f355d25ef40 1 --2- 172.21.15.32:0/4053519628 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555e4279fbe0 0x555e427bbaf0 secure :-1 s=READY pgs=1800 cs=0 l=1 rev1=1 crypto rx=0x7f3544001d20 tx=0x7f354400b5b0 comp rx=0 tx=0).stop 2024-08-03T20:58:33.510 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:33.507+0000 7f355d25ef40 1 -- 172.21.15.32:0/4053519628 shutdown_connections 2024-08-03T20:58:33.510 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:33.507+0000 7f355d25ef40 1 --2- 172.21.15.32:0/4053519628 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x555e426e7b20 0x555e427bc030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:33.510 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:33.507+0000 7f355d25ef40 1 --2- 172.21.15.32:0/4053519628 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555e4279fbe0 0x555e427bbaf0 unknown :-1 s=CLOSED pgs=1800 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:33.510 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:33.507+0000 7f355d25ef40 1 --2- 172.21.15.32:0/4053519628 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x555e4272cb40 0x555e426e75e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:33.510 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:33.507+0000 7f355d25ef40 1 -- 172.21.15.32:0/4053519628 >> 172.21.15.32:0/4053519628 conn(0x555e42607ac0 msgr2=0x555e427baf20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:58:33.511 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:33.507+0000 7f355d25ef40 1 -- 172.21.15.32:0/4053519628 shutdown_connections 2024-08-03T20:58:33.511 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:33.507+0000 7f355d25ef40 1 -- 172.21.15.32:0/4053519628 wait complete. 2024-08-03T20:58:33.511 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:33.508+0000 7f355d25ef40 1 Processor -- start 2024-08-03T20:58:33.511 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:33.508+0000 7f355d25ef40 1 -- start start 2024-08-03T20:58:33.513 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:33.509+0000 7f355d25ef40 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x555e426e7b20 0x555e4279c020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:58:33.513 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:33.509+0000 7f355d25ef40 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555e4272cb40 0x555e42857e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:58:33.513 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:33.509+0000 7f355d25ef40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x555e4279fbe0 0x555e428583b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:58:33.513 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:33.509+0000 7f355d25ef40 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x555e427bc570 con 0x555e4272cb40 2024-08-03T20:58:33.513 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:33.509+0000 7f355d25ef40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x555e4272fe40 con 0x555e4279fbe0 2024-08-03T20:58:33.513 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:33.509+0000 7f355d25ef40 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x555e427ba070 con 0x555e426e7b20 2024-08-03T20:58:33.513 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:33.509+0000 7f3555a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555e4272cb40 0x555e42857e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:58:33.513 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:33.509+0000 7f3555a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555e4272cb40 0x555e42857e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:41576/0 (socket says 172.21.15.32:41576) 2024-08-03T20:58:33.513 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:33.509+0000 7f3555a00640 1 -- 172.21.15.32:0/964533858 learned_addr learned my addr 172.21.15.32:0/964533858 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:58:33.513 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:33.510+0000 7f3556400640 1 --2- 172.21.15.32:0/964533858 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x555e426e7b20 0x555e4279c020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:58:33.513 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:33.510+0000 7f3556e00640 1 --2- 172.21.15.32:0/964533858 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x555e4279fbe0 0x555e428583b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:58:33.513 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:33.510+0000 7f3555a00640 1 -- 172.21.15.32:0/964533858 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x555e426e7b20 msgr2=0x555e4279c020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:58:33.514 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:33.510+0000 7f3555a00640 1 --2- 172.21.15.32:0/964533858 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x555e426e7b20 0x555e4279c020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:33.514 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:33.510+0000 7f3555a00640 1 -- 172.21.15.32:0/964533858 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x555e4279fbe0 msgr2=0x555e428583b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:58:33.514 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:33.510+0000 7f3555a00640 1 --2- 172.21.15.32:0/964533858 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x555e4279fbe0 0x555e428583b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:33.514 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:33.510+0000 7f3555a00640 1 -- 172.21.15.32:0/964533858 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555e427b9dc0 con 0x555e4272cb40 2024-08-03T20:58:33.514 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:33.510+0000 7f3555a00640 1 --2- 172.21.15.32:0/964533858 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555e4272cb40 0x555e42857e70 secure :-1 s=READY pgs=1801 cs=0 l=1 rev1=1 crypto rx=0x555e427beb10 tx=0x7f354400fea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:58:33.514 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:33.510+0000 7f354ea00640 1 -- 172.21.15.32:0/964533858 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f354401b430 con 0x555e4272cb40 2024-08-03T20:58:33.514 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:33.511+0000 7f354ea00640 1 -- 172.21.15.32:0/964533858 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f3544020020 con 0x555e4272cb40 2024-08-03T20:58:33.514 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:33.511+0000 7f354ea00640 1 -- 172.21.15.32:0/964533858 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f354401b740 con 0x555e4272cb40 2024-08-03T20:58:33.514 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:33.511+0000 7f355d25ef40 1 -- 172.21.15.32:0/964533858 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x555e4279a5d0 con 0x555e4272cb40 2024-08-03T20:58:33.514 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:33.511+0000 7f355d25ef40 1 -- 172.21.15.32:0/964533858 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555e42796160 con 0x555e4272cb40 2024-08-03T20:58:33.515 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:33.513+0000 7f354ea00640 1 -- 172.21.15.32:0/964533858 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f3544017070 con 0x555e4272cb40 2024-08-03T20:58:33.515 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:33.514+0000 7f354ea00640 1 --2- 172.21.15.32:0/964533858 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f35380759f0 0x7f3538077f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:58:33.515 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:33.514+0000 7f354ea00640 1 -- 172.21.15.32:0/964533858 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(115..115 src has 1..115) ==== 8434+0+0 (secure 0 0 0) 0x7f3544099470 con 0x555e4272cb40 2024-08-03T20:58:33.515 INFO:teuthology.orchestra.run.smithi032.stdout:Importing pool 2024-08-03T20:58:33.516 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:33.514+0000 7f355d25ef40 1 --2- 172.21.15.32:0/964533858 >> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] conn(0x555e42794f40 0x555e42795350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:58:33.516 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:33.514+0000 7f355d25ef40 1 -- 172.21.15.32:0/964533858 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:1 12.10 12:09401a81:::601.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115) -- 0x555e4279cbd0 con 0x555e42794f40 2024-08-03T20:58:33.516 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:33.514+0000 7f3556400640 1 --2- 172.21.15.32:0/964533858 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f35380759f0 0x7f3538077f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:58:33.517 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:33.514+0000 7f3556e00640 1 --2- 172.21.15.32:0/964533858 >> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] conn(0x555e42794f40 0x555e42795350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:58:33.517 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:33.515+0000 7f3556400640 1 --2- 172.21.15.32:0/964533858 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f35380759f0 0x7f3538077f00 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7f354800b770 tx=0x7f3548004880 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:58:33.517 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:33.515+0000 7f3556e00640 1 --2- 172.21.15.32:0/964533858 >> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] conn(0x555e42794f40 0x555e42795350 crc :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:58:33.519 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:33.518+0000 7f3556e00640 1 -- 172.21.15.32:0/964533858 <== osd.0 v2:172.21.15.32:6816/3066156116 1 ==== osd_op_reply(1 601.00000000 [create] v115'908 uv882 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f355000b040 con 0x555e42794f40 2024-08-03T20:58:33.519 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:33.518+0000 7f355d25ef40 1 -- 172.21.15.32:0/964533858 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:2 12.10 12:09401a81:::601.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e115) -- 0x555e4279c560 con 0x555e42794f40 2024-08-03T20:58:33.521 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:33.520+0000 7f3556e00640 1 -- 172.21.15.32:0/964533858 <== osd.0 v2:172.21.15.32:6816/3066156116 2 ==== osd_op_reply(2 601.00000000 [delete] v115'909 uv909 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f355000b040 con 0x555e42794f40 2024-08-03T20:58:33.521 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:33.520+0000 7f355d25ef40 1 -- 172.21.15.32:0/964533858 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:3 12.10 12:09401a81:::601.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115) -- 0x555e42795890 con 0x555e42794f40 2024-08-03T20:58:33.523 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::601.00000000:head# 2024-08-03T20:58:33.523 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:33.521+0000 7f3556e00640 1 -- 172.21.15.32:0/964533858 <== osd.0 v2:172.21.15.32:6816/3066156116 3 ==== osd_op_reply(3 601.00000000 [create] v115'910 uv910 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f355000b040 con 0x555e42794f40 2024-08-03T20:58:33.523 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:33.521+0000 7f355d25ef40 1 -- 172.21.15.32:0/964533858 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:4 12.10 12:09401a81:::601.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115) -- 0x555e42795c30 con 0x555e42794f40 2024-08-03T20:58:33.524 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:33.523+0000 7f3556e00640 1 -- 172.21.15.32:0/964533858 <== osd.0 v2:172.21.15.32:6816/3066156116 4 ==== osd_op_reply(4 601.00000000 [setxattr (30)] v115'911 uv911 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f355000b040 con 0x555e42794f40 2024-08-03T20:58:33.525 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:33.523+0000 7f355d25ef40 1 -- 172.21.15.32:0/964533858 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:5 12.10 12:09401a81:::601.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115) -- 0x555e42862150 con 0x555e42794f40 2024-08-03T20:58:33.527 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:33.525+0000 7f3556e00640 1 -- 172.21.15.32:0/964533858 <== osd.0 v2:172.21.15.32:6816/3066156116 5 ==== osd_op_reply(5 601.00000000 [setxattr (62)] v115'912 uv912 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f355000b040 con 0x555e42794f40 2024-08-03T20:58:33.527 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:33.525+0000 7f355d25ef40 1 -- 172.21.15.32:0/964533858 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:6 12.10 12:09401a81:::601.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115) -- 0x555e42861140 con 0x555e42794f40 2024-08-03T20:58:33.528 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:33.527+0000 7f3556e00640 1 -- 172.21.15.32:0/964533858 <== osd.0 v2:172.21.15.32:6816/3066156116 6 ==== osd_op_reply(6 601.00000000 [omap-set-header] v115'913 uv913 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f355000b040 con 0x555e42794f40 2024-08-03T20:58:33.528 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:33.527+0000 7f355d25ef40 1 -- 172.21.15.32:0/964533858 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:7 12.10 12:0bd6d154:::602.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115) -- 0x555e428614e0 con 0x555e42794f40 2024-08-03T20:58:33.531 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:33.529+0000 7f3556e00640 1 -- 172.21.15.32:0/964533858 <== osd.0 v2:172.21.15.32:6816/3066156116 7 ==== osd_op_reply(7 602.00000000 [create] v115'914 uv888 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f355000b040 con 0x555e42794f40 2024-08-03T20:58:33.531 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:33.529+0000 7f355d25ef40 1 -- 172.21.15.32:0/964533858 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:8 12.10 12:0bd6d154:::602.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e115) -- 0x555e42861880 con 0x555e42794f40 2024-08-03T20:58:33.532 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:33.531+0000 7f3556e00640 1 -- 172.21.15.32:0/964533858 <== osd.0 v2:172.21.15.32:6816/3066156116 8 ==== osd_op_reply(8 602.00000000 [delete] v115'915 uv915 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f355000b040 con 0x555e42794f40 2024-08-03T20:58:33.533 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:33.531+0000 7f355d25ef40 1 -- 172.21.15.32:0/964533858 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:9 12.10 12:0bd6d154:::602.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115) -- 0x555e42861880 con 0x555e42794f40 2024-08-03T20:58:33.534 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::602.00000000:head# 2024-08-03T20:58:33.539 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:33.533+0000 7f3556e00640 1 -- 172.21.15.32:0/964533858 <== osd.0 v2:172.21.15.32:6816/3066156116 9 ==== osd_op_reply(9 602.00000000 [create] v115'916 uv916 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f355000b040 con 0x555e42794f40 2024-08-03T20:58:33.539 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:33.533+0000 7f355d25ef40 1 -- 172.21.15.32:0/964533858 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:10 12.10 12:0bd6d154:::602.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115) -- 0x555e42861880 con 0x555e42794f40 2024-08-03T20:58:33.539 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:33.535+0000 7f3556e00640 1 -- 172.21.15.32:0/964533858 <== osd.0 v2:172.21.15.32:6816/3066156116 10 ==== osd_op_reply(10 602.00000000 [setxattr (30)] v115'917 uv917 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f355000b040 con 0x555e42794f40 2024-08-03T20:58:33.539 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:33.535+0000 7f355d25ef40 1 -- 172.21.15.32:0/964533858 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:11 12.10 12:0bd6d154:::602.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115) -- 0x555e42861880 con 0x555e42794f40 2024-08-03T20:58:33.539 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:33.536+0000 7f3556e00640 1 -- 172.21.15.32:0/964533858 <== osd.0 v2:172.21.15.32:6816/3066156116 11 ==== osd_op_reply(11 602.00000000 [setxattr (62)] v115'918 uv918 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f355000b040 con 0x555e42794f40 2024-08-03T20:58:33.540 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:33.536+0000 7f355d25ef40 1 -- 172.21.15.32:0/964533858 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:12 12.10 12:0bd6d154:::602.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115) -- 0x555e42861880 con 0x555e42794f40 2024-08-03T20:58:33.540 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:33.538+0000 7f3556e00640 1 -- 172.21.15.32:0/964533858 <== osd.0 v2:172.21.15.32:6816/3066156116 12 ==== osd_op_reply(12 602.00000000 [omap-set-header] v115'919 uv919 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f355000b040 con 0x555e42794f40 2024-08-03T20:58:33.540 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:33.538+0000 7f355d25ef40 1 -- 172.21.15.32:0/964533858 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:13 12.10 12:0cf6ad0f:::10000000000.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115) -- 0x555e42861880 con 0x555e42794f40 2024-08-03T20:58:33.541 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:33.540+0000 7f3556e00640 1 -- 172.21.15.32:0/964533858 <== osd.0 v2:172.21.15.32:6816/3066156116 13 ==== osd_op_reply(13 10000000000.00000000 [create] v115'920 uv895 ondisk = -17 ((17) File exists)) ==== 164+0+0 (crc 0 0 0) 0x7f355000b040 con 0x555e42794f40 2024-08-03T20:58:33.541 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:33.540+0000 7f355d25ef40 1 -- 172.21.15.32:0/964533858 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:14 12.10 12:0cf6ad0f:::10000000000.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e115) -- 0x555e42861880 con 0x555e42794f40 2024-08-03T20:58:33.543 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:33.542+0000 7f3556e00640 1 -- 172.21.15.32:0/964533858 <== osd.0 v2:172.21.15.32:6816/3066156116 14 ==== osd_op_reply(14 10000000000.00000000 [delete] v115'921 uv921 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f355000b040 con 0x555e42794f40 2024-08-03T20:58:33.543 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:33.542+0000 7f355d25ef40 1 -- 172.21.15.32:0/964533858 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:15 12.10 12:0cf6ad0f:::10000000000.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115) -- 0x555e42861880 con 0x555e42794f40 2024-08-03T20:58:33.545 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::10000000000.00000000:head# 2024-08-03T20:58:33.545 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:33.543+0000 7f3556e00640 1 -- 172.21.15.32:0/964533858 <== osd.0 v2:172.21.15.32:6816/3066156116 15 ==== osd_op_reply(15 10000000000.00000000 [create] v115'922 uv922 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f355000b040 con 0x555e42794f40 2024-08-03T20:58:33.545 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:33.543+0000 7f355d25ef40 1 -- 172.21.15.32:0/964533858 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:16 12.10 12:0cf6ad0f:::10000000000.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115) -- 0x555e42861880 con 0x555e42794f40 2024-08-03T20:58:33.547 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:33.545+0000 7f3556e00640 1 -- 172.21.15.32:0/964533858 <== osd.0 v2:172.21.15.32:6816/3066156116 16 ==== osd_op_reply(16 10000000000.00000000 [setxattr (30)] v115'923 uv923 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f355000b040 con 0x555e42794f40 2024-08-03T20:58:33.547 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:33.545+0000 7f355d25ef40 1 -- 172.21.15.32:0/964533858 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:17 12.10 12:0cf6ad0f:::10000000000.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115) -- 0x555e42861880 con 0x555e42794f40 2024-08-03T20:58:33.548 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:33.547+0000 7f3556e00640 1 -- 172.21.15.32:0/964533858 <== osd.0 v2:172.21.15.32:6816/3066156116 17 ==== osd_op_reply(17 10000000000.00000000 [setxattr (62)] v115'924 uv924 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f355000b040 con 0x555e42794f40 2024-08-03T20:58:33.548 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:33.547+0000 7f355d25ef40 1 -- 172.21.15.32:0/964533858 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:18 12.10 12:0cf6ad0f:::10000000000.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115) -- 0x555e42861880 con 0x555e42794f40 2024-08-03T20:58:33.550 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:33.549+0000 7f3556e00640 1 -- 172.21.15.32:0/964533858 <== osd.0 v2:172.21.15.32:6816/3066156116 18 ==== osd_op_reply(18 10000000000.00000000 [omap-set-header] v115'925 uv925 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f355000b040 con 0x555e42794f40 2024-08-03T20:58:33.550 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:33.549+0000 7f355d25ef40 1 -- 172.21.15.32:0/964533858 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:19 12.10 12:0cf6ad0f:::10000000000.00000000:head [omap-set-vals in=593b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115) -- 0x555e428624f0 con 0x555e42794f40 2024-08-03T20:58:33.552 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:33.551+0000 7f3556e00640 1 -- 172.21.15.32:0/964533858 <== osd.0 v2:172.21.15.32:6816/3066156116 19 ==== osd_op_reply(19 10000000000.00000000 [omap-set-vals] v115'926 uv926 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f355000b040 con 0x555e42794f40 2024-08-03T20:58:33.552 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:33.551+0000 7f355d25ef40 1 -- 172.21.15.32:0/964533858 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:20 12.10 12:0d82a743:::600.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115) -- 0x555e428624f0 con 0x555e42794f40 2024-08-03T20:58:33.554 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:33.552+0000 7f3556e00640 1 -- 172.21.15.32:0/964533858 <== osd.0 v2:172.21.15.32:6816/3066156116 20 ==== osd_op_reply(20 600.00000000 [create] v115'927 uv901 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f355000b040 con 0x555e42794f40 2024-08-03T20:58:33.554 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:33.552+0000 7f355d25ef40 1 -- 172.21.15.32:0/964533858 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:21 12.10 12:0d82a743:::600.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e115) -- 0x555e428624f0 con 0x555e42794f40 2024-08-03T20:58:33.555 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:33.554+0000 7f3556e00640 1 -- 172.21.15.32:0/964533858 <== osd.0 v2:172.21.15.32:6816/3066156116 21 ==== osd_op_reply(21 600.00000000 [delete] v115'928 uv928 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f355000b040 con 0x555e42794f40 2024-08-03T20:58:33.555 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:33.554+0000 7f355d25ef40 1 -- 172.21.15.32:0/964533858 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:22 12.10 12:0d82a743:::600.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115) -- 0x555e428624f0 con 0x555e42794f40 2024-08-03T20:58:33.557 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::600.00000000:head# 2024-08-03T20:58:33.557 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:33.556+0000 7f3556e00640 1 -- 172.21.15.32:0/964533858 <== osd.0 v2:172.21.15.32:6816/3066156116 22 ==== osd_op_reply(22 600.00000000 [create] v115'929 uv929 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f355000b040 con 0x555e42794f40 2024-08-03T20:58:33.558 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:33.556+0000 7f355d25ef40 1 -- 172.21.15.32:0/964533858 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:23 12.10 12:0d82a743:::600.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115) -- 0x555e42863500 con 0x555e42794f40 2024-08-03T20:58:33.559 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:33.557+0000 7f3556e00640 1 -- 172.21.15.32:0/964533858 <== osd.0 v2:172.21.15.32:6816/3066156116 23 ==== osd_op_reply(23 600.00000000 [setxattr (30)] v115'930 uv930 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f355000b040 con 0x555e42794f40 2024-08-03T20:58:33.559 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:33.557+0000 7f355d25ef40 1 -- 172.21.15.32:0/964533858 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:24 12.10 12:0d82a743:::600.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115) -- 0x555e42863500 con 0x555e42794f40 2024-08-03T20:58:33.560 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:33.559+0000 7f3556e00640 1 -- 172.21.15.32:0/964533858 <== osd.0 v2:172.21.15.32:6816/3066156116 24 ==== osd_op_reply(24 600.00000000 [setxattr (62)] v115'931 uv931 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f355000b040 con 0x555e42794f40 2024-08-03T20:58:33.560 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:33.559+0000 7f355d25ef40 1 -- 172.21.15.32:0/964533858 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:25 12.10 12:0d82a743:::600.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115) -- 0x555e428624f0 con 0x555e42794f40 2024-08-03T20:58:33.562 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:33.560+0000 7f3556e00640 1 -- 172.21.15.32:0/964533858 <== osd.0 v2:172.21.15.32:6816/3066156116 25 ==== osd_op_reply(25 600.00000000 [omap-set-header] v115'932 uv932 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f355000b040 con 0x555e42794f40 2024-08-03T20:58:33.562 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:33.561+0000 7f355d25ef40 1 -- 172.21.15.32:0/964533858 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:26 12.10 12:0d89b25e:::603.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115) -- 0x555e428624f0 con 0x555e42794f40 2024-08-03T20:58:33.564 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:33.562+0000 7f3556e00640 1 -- 172.21.15.32:0/964533858 <== osd.0 v2:172.21.15.32:6816/3066156116 26 ==== osd_op_reply(26 603.00000000 [create] v115'933 uv907 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f355000b040 con 0x555e42794f40 2024-08-03T20:58:33.564 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:33.562+0000 7f355d25ef40 1 -- 172.21.15.32:0/964533858 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:27 12.10 12:0d89b25e:::603.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e115) -- 0x555e428624f0 con 0x555e42794f40 2024-08-03T20:58:33.565 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:33.564+0000 7f3556e00640 1 -- 172.21.15.32:0/964533858 <== osd.0 v2:172.21.15.32:6816/3066156116 27 ==== osd_op_reply(27 603.00000000 [delete] v115'934 uv934 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f355000b040 con 0x555e42794f40 2024-08-03T20:58:33.565 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:33.564+0000 7f355d25ef40 1 -- 172.21.15.32:0/964533858 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:28 12.10 12:0d89b25e:::603.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115) -- 0x555e428624f0 con 0x555e42794f40 2024-08-03T20:58:33.567 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::603.00000000:head# 2024-08-03T20:58:33.567 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:33.566+0000 7f3556e00640 1 -- 172.21.15.32:0/964533858 <== osd.0 v2:172.21.15.32:6816/3066156116 28 ==== osd_op_reply(28 603.00000000 [create] v115'935 uv935 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f355000b040 con 0x555e42794f40 2024-08-03T20:58:33.567 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:33.566+0000 7f355d25ef40 1 -- 172.21.15.32:0/964533858 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:29 12.10 12:0d89b25e:::603.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115) -- 0x555e42863500 con 0x555e42794f40 2024-08-03T20:58:33.569 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:33.567+0000 7f3556e00640 1 -- 172.21.15.32:0/964533858 <== osd.0 v2:172.21.15.32:6816/3066156116 29 ==== osd_op_reply(29 603.00000000 [setxattr (30)] v115'936 uv936 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f355000b040 con 0x555e42794f40 2024-08-03T20:58:33.569 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:33.567+0000 7f355d25ef40 1 -- 172.21.15.32:0/964533858 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:30 12.10 12:0d89b25e:::603.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115) -- 0x555e42863500 con 0x555e42794f40 2024-08-03T20:58:33.570 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:33.569+0000 7f3556e00640 1 -- 172.21.15.32:0/964533858 <== osd.0 v2:172.21.15.32:6816/3066156116 30 ==== osd_op_reply(30 603.00000000 [setxattr (62)] v115'937 uv937 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f355000b040 con 0x555e42794f40 2024-08-03T20:58:33.571 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:33.569+0000 7f355d25ef40 1 -- 172.21.15.32:0/964533858 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:31 12.10 12:0d89b25e:::603.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115) -- 0x555e428624f0 con 0x555e42794f40 2024-08-03T20:58:33.572 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:33.571+0000 7f3556e00640 1 -- 172.21.15.32:0/964533858 <== osd.0 v2:172.21.15.32:6816/3066156116 31 ==== osd_op_reply(31 603.00000000 [omap-set-header] v115'938 uv938 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f355000b040 con 0x555e42794f40 2024-08-03T20:58:33.572 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:33.571+0000 7f355d25ef40 1 --2- 172.21.15.32:0/964533858 >> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] conn(0x555e42862a00 0x555e42864e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:58:33.572 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:33.571+0000 7f355d25ef40 1 -- 172.21.15.32:0/964533858 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:32 12.14 12:287034c4:::1.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115) -- 0x555e42865360 con 0x555e42862a00 2024-08-03T20:58:33.573 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:33.571+0000 7f3556400640 1 --2- 172.21.15.32:0/964533858 >> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] conn(0x555e42862a00 0x555e42864e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:58:33.573 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:33.572+0000 7f3556400640 1 --2- 172.21.15.32:0/964533858 >> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] conn(0x555e42862a00 0x555e42864e20 crc :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:58:33.576 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:33.574+0000 7f3556400640 1 -- 172.21.15.32:0/964533858 <== osd.5 v2:172.21.15.118:6810/2724891770 1 ==== osd_op_reply(32 1.00000000.inode [create] v115'659 uv642 ondisk = -17 ((17) File exists)) ==== 160+0+0 (crc 0 0 0) 0x7f354800abc0 con 0x555e42862a00 2024-08-03T20:58:33.576 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:33.574+0000 7f355d25ef40 1 -- 172.21.15.32:0/964533858 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:33 12.14 12:287034c4:::1.00000000.inode:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e115) -- 0x555e42865700 con 0x555e42862a00 2024-08-03T20:58:33.577 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:33.576+0000 7f3556400640 1 -- 172.21.15.32:0/964533858 <== osd.5 v2:172.21.15.118:6810/2724891770 2 ==== osd_op_reply(33 1.00000000.inode [delete] v115'660 uv660 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f354800abc0 con 0x555e42862a00 2024-08-03T20:58:33.577 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:33.576+0000 7f355d25ef40 1 -- 172.21.15.32:0/964533858 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:34 12.14 12:287034c4:::1.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115) -- 0x555e42865aa0 con 0x555e42862a00 2024-08-03T20:58:33.579 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::1.00000000.inode:head# 2024-08-03T20:58:33.579 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:33.578+0000 7f3556400640 1 -- 172.21.15.32:0/964533858 <== osd.5 v2:172.21.15.118:6810/2724891770 3 ==== osd_op_reply(34 1.00000000.inode [create] v115'661 uv661 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f354800abc0 con 0x555e42862a00 2024-08-03T20:58:33.579 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:33.578+0000 7f355d25ef40 1 -- 172.21.15.32:0/964533858 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:35 12.14 12:287034c4:::1.00000000.inode:head [write 0~582 in=582b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115) -- 0x555e42865e40 con 0x555e42862a00 2024-08-03T20:58:33.581 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:33.580+0000 7f3556400640 1 -- 172.21.15.32:0/964533858 <== osd.5 v2:172.21.15.118:6810/2724891770 4 ==== osd_op_reply(35 1.00000000.inode [write 0~582] v115'662 uv662 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f354800abc0 con 0x555e42862a00 2024-08-03T20:58:33.581 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:33.580+0000 7f355d25ef40 1 -- 172.21.15.32:0/964533858 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:36 12.14 12:287034c4:::1.00000000.inode:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115) -- 0x555e428661e0 con 0x555e42862a00 2024-08-03T20:58:33.583 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:33.581+0000 7f3556400640 1 -- 172.21.15.32:0/964533858 <== osd.5 v2:172.21.15.118:6810/2724891770 5 ==== osd_op_reply(36 1.00000000.inode [omap-set-header] v115'663 uv663 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f354800abc0 con 0x555e42862a00 2024-08-03T20:58:33.583 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:33.581+0000 7f355d25ef40 1 -- 172.21.15.32:0/964533858 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:37 12.14 12:292cf221:::200.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115) -- 0x555e42866580 con 0x555e42862a00 2024-08-03T20:58:33.585 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:33.584+0000 7f3556400640 1 -- 172.21.15.32:0/964533858 <== osd.5 v2:172.21.15.118:6810/2724891770 6 ==== osd_op_reply(37 200.00000000 [create] v115'664 uv647 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f354800abc0 con 0x555e42862a00 2024-08-03T20:58:33.585 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:33.584+0000 7f355d25ef40 1 -- 172.21.15.32:0/964533858 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:38 12.14 12:292cf221:::200.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e115) -- 0x555e42866920 con 0x555e42862a00 2024-08-03T20:58:33.587 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:33.586+0000 7f3556400640 1 -- 172.21.15.32:0/964533858 <== osd.5 v2:172.21.15.118:6810/2724891770 7 ==== osd_op_reply(38 200.00000000 [delete] v115'665 uv665 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f354800abc0 con 0x555e42862a00 2024-08-03T20:58:33.587 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:33.586+0000 7f355d25ef40 1 -- 172.21.15.32:0/964533858 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:39 12.14 12:292cf221:::200.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115) -- 0x555e42866d10 con 0x555e42862a00 2024-08-03T20:58:33.589 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:33.587+0000 7f3556400640 1 -- 172.21.15.32:0/964533858 <== osd.5 v2:172.21.15.118:6810/2724891770 8 ==== osd_op_reply(39 200.00000000 [create] v115'666 uv666 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f354800abc0 con 0x555e42862a00 2024-08-03T20:58:33.589 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::200.00000000:head# 2024-08-03T20:58:33.589 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:33.587+0000 7f355d25ef40 1 -- 172.21.15.32:0/964533858 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:40 12.14 12:292cf221:::200.00000000:head [write 0~90 in=90b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115) -- 0x555e42867100 con 0x555e42862a00 2024-08-03T20:58:33.591 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:33.589+0000 7f3556400640 1 -- 172.21.15.32:0/964533858 <== osd.5 v2:172.21.15.118:6810/2724891770 9 ==== osd_op_reply(40 200.00000000 [write 0~90] v115'667 uv667 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f354800abc0 con 0x555e42862a00 2024-08-03T20:58:33.591 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:33.589+0000 7f355d25ef40 1 -- 172.21.15.32:0/964533858 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:41 12.14 12:292cf221:::200.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115) -- 0x555e42867100 con 0x555e42862a00 2024-08-03T20:58:33.592 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:33.591+0000 7f3556400640 1 -- 172.21.15.32:0/964533858 <== osd.5 v2:172.21.15.118:6810/2724891770 10 ==== osd_op_reply(41 200.00000000 [omap-set-header] v115'668 uv668 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f354800abc0 con 0x555e42862a00 2024-08-03T20:58:33.592 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:33.591+0000 7f355d25ef40 1 -- 172.21.15.32:0/964533858 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:42 12.14 12:2e2fa760:::200.00000001:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115) -- 0x555e42866e90 con 0x555e42862a00 2024-08-03T20:58:33.594 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:33.593+0000 7f3556400640 1 -- 172.21.15.32:0/964533858 <== osd.5 v2:172.21.15.118:6810/2724891770 11 ==== osd_op_reply(42 200.00000001 [create] v115'669 uv652 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f354800abc0 con 0x555e42862a00 2024-08-03T20:58:33.594 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:33.593+0000 7f355d25ef40 1 -- 172.21.15.32:0/964533858 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:43 12.14 12:2e2fa760:::200.00000001:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e115) -- 0x555e42866e90 con 0x555e42862a00 2024-08-03T20:58:33.596 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:33.595+0000 7f3556400640 1 -- 172.21.15.32:0/964533858 <== osd.5 v2:172.21.15.118:6810/2724891770 12 ==== osd_op_reply(43 200.00000001 [delete] v115'670 uv670 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f354800abc0 con 0x555e42862a00 2024-08-03T20:58:33.596 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:33.595+0000 7f355d25ef40 1 -- 172.21.15.32:0/964533858 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:44 12.14 12:2e2fa760:::200.00000001:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115) -- 0x555e42866e40 con 0x555e42862a00 2024-08-03T20:58:33.597 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:33.596+0000 7f3556400640 1 -- 172.21.15.32:0/964533858 <== osd.5 v2:172.21.15.118:6810/2724891770 13 ==== osd_op_reply(44 200.00000001 [create] v115'671 uv671 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f354800abc0 con 0x555e42862a00 2024-08-03T20:58:33.598 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::200.00000001:head# 2024-08-03T20:58:33.598 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:33.596+0000 7f355d25ef40 1 -- 172.21.15.32:0/964533858 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:45 12.14 12:2e2fa760:::200.00000001:head [write 0~18118 in=18118b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115) -- 0x555e4286b530 con 0x555e42862a00 2024-08-03T20:58:33.600 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:33.598+0000 7f3556400640 1 -- 172.21.15.32:0/964533858 <== osd.5 v2:172.21.15.118:6810/2724891770 14 ==== osd_op_reply(45 200.00000001 [write 0~18118] v115'672 uv672 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f354800abc0 con 0x555e42862a00 2024-08-03T20:58:33.600 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:33.598+0000 7f355d25ef40 1 -- 172.21.15.32:0/964533858 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:46 12.14 12:2e2fa760:::200.00000001:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115) -- 0x555e4286b530 con 0x555e42862a00 2024-08-03T20:58:33.601 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:33.600+0000 7f3556400640 1 -- 172.21.15.32:0/964533858 <== osd.5 v2:172.21.15.118:6810/2724891770 15 ==== osd_op_reply(46 200.00000001 [omap-set-header] v115'673 uv673 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f354800abc0 con 0x555e42862a00 2024-08-03T20:58:33.602 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:33.600+0000 7f355d25ef40 1 -- 172.21.15.32:0/964533858 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:47 12.14 12:2f55791f:::606.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115) -- 0x555e42866e40 con 0x555e42862a00 2024-08-03T20:58:33.603 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:33.602+0000 7f3556400640 1 -- 172.21.15.32:0/964533858 <== osd.5 v2:172.21.15.118:6810/2724891770 16 ==== osd_op_reply(47 606.00000000 [create] v115'674 uv658 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f354800abc0 con 0x555e42862a00 2024-08-03T20:58:33.603 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:33.602+0000 7f355d25ef40 1 -- 172.21.15.32:0/964533858 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:48 12.14 12:2f55791f:::606.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e115) -- 0x555e42866e40 con 0x555e42862a00 2024-08-03T20:58:33.605 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:33.603+0000 7f3556400640 1 -- 172.21.15.32:0/964533858 <== osd.5 v2:172.21.15.118:6810/2724891770 17 ==== osd_op_reply(48 606.00000000 [delete] v115'675 uv675 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f354800abc0 con 0x555e42862a00 2024-08-03T20:58:33.605 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:33.603+0000 7f355d25ef40 1 -- 172.21.15.32:0/964533858 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:49 12.14 12:2f55791f:::606.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115) -- 0x555e42866df0 con 0x555e42862a00 2024-08-03T20:58:33.606 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:33.605+0000 7f3556400640 1 -- 172.21.15.32:0/964533858 <== osd.5 v2:172.21.15.118:6810/2724891770 18 ==== osd_op_reply(49 606.00000000 [create] v115'676 uv676 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f354800abc0 con 0x555e42862a00 2024-08-03T20:58:33.607 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::606.00000000:head# 2024-08-03T20:58:33.607 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:33.605+0000 7f355d25ef40 1 -- 172.21.15.32:0/964533858 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:50 12.14 12:2f55791f:::606.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115) -- 0x555e42866df0 con 0x555e42862a00 2024-08-03T20:58:33.608 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:33.607+0000 7f3556400640 1 -- 172.21.15.32:0/964533858 <== osd.5 v2:172.21.15.118:6810/2724891770 19 ==== osd_op_reply(50 606.00000000 [setxattr (30)] v115'677 uv677 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f354800abc0 con 0x555e42862a00 2024-08-03T20:58:33.608 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:33.607+0000 7f355d25ef40 1 -- 172.21.15.32:0/964533858 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:51 12.14 12:2f55791f:::606.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115) -- 0x555e42866df0 con 0x555e42862a00 2024-08-03T20:58:33.610 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:33.608+0000 7f3556400640 1 -- 172.21.15.32:0/964533858 <== osd.5 v2:172.21.15.118:6810/2724891770 20 ==== osd_op_reply(51 606.00000000 [setxattr (62)] v115'678 uv678 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f354800abc0 con 0x555e42862a00 2024-08-03T20:58:33.610 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:33.609+0000 7f355d25ef40 1 -- 172.21.15.32:0/964533858 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:52 12.14 12:2f55791f:::606.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115) -- 0x555e42866df0 con 0x555e42862a00 2024-08-03T20:58:33.611 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:33.610+0000 7f3556400640 1 -- 172.21.15.32:0/964533858 <== osd.5 v2:172.21.15.118:6810/2724891770 21 ==== osd_op_reply(52 606.00000000 [omap-set-header] v115'679 uv679 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f354800abc0 con 0x555e42862a00 2024-08-03T20:58:33.612 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:33.610+0000 7f355d25ef40 1 -- 172.21.15.32:0/964533858 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:53 12.1c 12:3ed09add:::607.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115) -- 0x555e42866df0 con 0x555e42794f40 2024-08-03T20:58:33.613 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:33.612+0000 7f3556e00640 1 -- 172.21.15.32:0/964533858 <== osd.0 v2:172.21.15.32:6816/3066156116 32 ==== osd_op_reply(53 607.00000000 [create] v115'177 uv176 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f355000b040 con 0x555e42794f40 2024-08-03T20:58:33.613 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:33.612+0000 7f355d25ef40 1 -- 172.21.15.32:0/964533858 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:54 12.1c 12:3ed09add:::607.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e115) -- 0x555e42866df0 con 0x555e42794f40 2024-08-03T20:58:33.614 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:33.613+0000 7f3556e00640 1 -- 172.21.15.32:0/964533858 <== osd.0 v2:172.21.15.32:6816/3066156116 33 ==== osd_op_reply(54 607.00000000 [delete] v115'178 uv178 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f355000b040 con 0x555e42794f40 2024-08-03T20:58:33.614 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:33.613+0000 7f355d25ef40 1 -- 172.21.15.32:0/964533858 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:55 12.1c 12:3ed09add:::607.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115) -- 0x555e42866df0 con 0x555e42794f40 2024-08-03T20:58:33.615 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::607.00000000:head# 2024-08-03T20:58:33.616 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:33.614+0000 7f3556e00640 1 -- 172.21.15.32:0/964533858 <== osd.0 v2:172.21.15.32:6816/3066156116 34 ==== osd_op_reply(55 607.00000000 [create] v115'179 uv179 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f355000b040 con 0x555e42794f40 2024-08-03T20:58:33.616 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:33.614+0000 7f355d25ef40 1 -- 172.21.15.32:0/964533858 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:56 12.1c 12:3ed09add:::607.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115) -- 0x555e42866df0 con 0x555e42794f40 2024-08-03T20:58:33.617 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:33.615+0000 7f3556e00640 1 -- 172.21.15.32:0/964533858 <== osd.0 v2:172.21.15.32:6816/3066156116 35 ==== osd_op_reply(56 607.00000000 [setxattr (30)] v115'180 uv180 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f355000b040 con 0x555e42794f40 2024-08-03T20:58:33.617 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:33.615+0000 7f355d25ef40 1 -- 172.21.15.32:0/964533858 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:57 12.1c 12:3ed09add:::607.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115) -- 0x555e42866df0 con 0x555e42794f40 2024-08-03T20:58:33.618 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:33.617+0000 7f3556e00640 1 -- 172.21.15.32:0/964533858 <== osd.0 v2:172.21.15.32:6816/3066156116 36 ==== osd_op_reply(57 607.00000000 [setxattr (62)] v115'181 uv181 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f355000b040 con 0x555e42794f40 2024-08-03T20:58:33.618 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:33.617+0000 7f355d25ef40 1 -- 172.21.15.32:0/964533858 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:58 12.1c 12:3ed09add:::607.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115) -- 0x555e42866df0 con 0x555e42794f40 2024-08-03T20:58:33.619 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:33.618+0000 7f3556e00640 1 -- 172.21.15.32:0/964533858 <== osd.0 v2:172.21.15.32:6816/3066156116 37 ==== osd_op_reply(58 607.00000000 [omap-set-header] v115'182 uv182 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f355000b040 con 0x555e42794f40 2024-08-03T20:58:33.620 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:33.618+0000 7f355d25ef40 1 --2- 172.21.15.32:0/964533858 >> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] conn(0x555e428676f0 0x555e42867b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:58:33.620 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:33.618+0000 7f355d25ef40 1 -- 172.21.15.32:0/964533858 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:59 12.6 12:654134d2:::mds0_openfiles.0:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115) -- 0x555e4286a390 con 0x555e428676f0 2024-08-03T20:58:33.620 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:33.618+0000 7f3555a00640 1 --2- 172.21.15.32:0/964533858 >> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] conn(0x555e428676f0 0x555e42867b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:58:33.620 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:33.619+0000 7f3555a00640 1 --2- 172.21.15.32:0/964533858 >> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] conn(0x555e428676f0 0x555e42867b00 crc :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:58:33.621 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:33.620+0000 7f3555a00640 1 -- 172.21.15.32:0/964533858 <== osd.1 v2:172.21.15.32:6824/1551241759 1 ==== osd_op_reply(59 mds0_openfiles.0 [create] v115'137 uv136 ondisk = -17 ((17) File exists)) ==== 160+0+0 (crc 0 0 0) 0x7f354402c070 con 0x555e428676f0 2024-08-03T20:58:33.621 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:33.620+0000 7f355d25ef40 1 -- 172.21.15.32:0/964533858 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:60 12.6 12:654134d2:::mds0_openfiles.0:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e115) -- 0x555e4286a730 con 0x555e428676f0 2024-08-03T20:58:33.623 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:33.621+0000 7f3555a00640 1 -- 172.21.15.32:0/964533858 <== osd.1 v2:172.21.15.32:6824/1551241759 2 ==== osd_op_reply(60 mds0_openfiles.0 [delete] v115'138 uv138 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f354402c070 con 0x555e428676f0 2024-08-03T20:58:33.623 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:33.621+0000 7f355d25ef40 1 -- 172.21.15.32:0/964533858 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:61 12.6 12:654134d2:::mds0_openfiles.0:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115) -- 0x555e4286ab70 con 0x555e428676f0 2024-08-03T20:58:33.624 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds0_openfiles.0:head# 2024-08-03T20:58:33.624 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:33.623+0000 7f3555a00640 1 -- 172.21.15.32:0/964533858 <== osd.1 v2:172.21.15.32:6824/1551241759 3 ==== osd_op_reply(61 mds0_openfiles.0 [create] v115'139 uv139 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f354402c070 con 0x555e428676f0 2024-08-03T20:58:33.624 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:33.623+0000 7f355d25ef40 1 -- 172.21.15.32:0/964533858 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:62 12.6 12:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115) -- 0x555e427ba3f0 con 0x555e428676f0 2024-08-03T20:58:33.625 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:33.624+0000 7f3555a00640 1 -- 172.21.15.32:0/964533858 <== osd.1 v2:172.21.15.32:6824/1551241759 4 ==== osd_op_reply(62 mds0_openfiles.0 [omap-set-header] v115'140 uv140 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f354402c070 con 0x555e428676f0 2024-08-03T20:58:33.625 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:33.624+0000 7f355d25ef40 1 -- 172.21.15.32:0/964533858 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:63 12.e 12:717a0223:::608.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115) -- 0x555e4286b4a0 con 0x555e42794f40 2024-08-03T20:58:33.627 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:33.626+0000 7f3556e00640 1 -- 172.21.15.32:0/964533858 <== osd.0 v2:172.21.15.32:6816/3066156116 38 ==== osd_op_reply(63 608.00000000 [create] v115'177 uv176 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f355000b040 con 0x555e42794f40 2024-08-03T20:58:33.627 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:33.626+0000 7f355d25ef40 1 -- 172.21.15.32:0/964533858 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:64 12.e 12:717a0223:::608.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e115) -- 0x555e4286b4a0 con 0x555e42794f40 2024-08-03T20:58:33.628 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:33.627+0000 7f3556e00640 1 -- 172.21.15.32:0/964533858 <== osd.0 v2:172.21.15.32:6816/3066156116 39 ==== osd_op_reply(64 608.00000000 [delete] v115'178 uv178 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f355000b040 con 0x555e42794f40 2024-08-03T20:58:33.628 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:33.627+0000 7f355d25ef40 1 -- 172.21.15.32:0/964533858 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:65 12.e 12:717a0223:::608.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115) -- 0x555e4286b4a0 con 0x555e42794f40 2024-08-03T20:58:33.630 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:33.629+0000 7f3556e00640 1 -- 172.21.15.32:0/964533858 <== osd.0 v2:172.21.15.32:6816/3066156116 40 ==== osd_op_reply(65 608.00000000 [create] v115'179 uv179 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f355000b040 con 0x555e42794f40 2024-08-03T20:58:33.630 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::608.00000000:head# 2024-08-03T20:58:33.630 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:33.629+0000 7f355d25ef40 1 -- 172.21.15.32:0/964533858 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:66 12.e 12:717a0223:::608.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115) -- 0x555e4286b4a0 con 0x555e42794f40 2024-08-03T20:58:33.632 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:33.630+0000 7f3556e00640 1 -- 172.21.15.32:0/964533858 <== osd.0 v2:172.21.15.32:6816/3066156116 41 ==== osd_op_reply(66 608.00000000 [setxattr (30)] v115'180 uv180 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f355000b040 con 0x555e42794f40 2024-08-03T20:58:33.632 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:33.630+0000 7f355d25ef40 1 -- 172.21.15.32:0/964533858 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:67 12.e 12:717a0223:::608.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115) -- 0x555e4286b4a0 con 0x555e42794f40 2024-08-03T20:58:33.633 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:33.632+0000 7f3556e00640 1 -- 172.21.15.32:0/964533858 <== osd.0 v2:172.21.15.32:6816/3066156116 42 ==== osd_op_reply(67 608.00000000 [setxattr (62)] v115'181 uv181 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f355000b040 con 0x555e42794f40 2024-08-03T20:58:33.633 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:33.632+0000 7f355d25ef40 1 -- 172.21.15.32:0/964533858 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:68 12.e 12:717a0223:::608.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115) -- 0x555e4286b4a0 con 0x555e42794f40 2024-08-03T20:58:33.635 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:33.633+0000 7f3556e00640 1 -- 172.21.15.32:0/964533858 <== osd.0 v2:172.21.15.32:6816/3066156116 43 ==== osd_op_reply(68 608.00000000 [omap-set-header] v115'182 uv182 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f355000b040 con 0x555e42794f40 2024-08-03T20:58:33.635 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:33.633+0000 7f355d25ef40 1 --2- 172.21.15.32:0/964533858 >> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] conn(0x555e4286bda0 0x555e4286c1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:58:33.635 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:33.633+0000 7f355d25ef40 1 -- 172.21.15.32:0/964533858 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:69 12.15 12:a93a17c2:::604.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115) -- 0x555e4286e8d0 con 0x555e4286bda0 2024-08-03T20:58:33.635 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:33.634+0000 7f3556e00640 1 --2- 172.21.15.32:0/964533858 >> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] conn(0x555e4286bda0 0x555e4286c1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:58:33.636 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:33.634+0000 7f3556e00640 1 --2- 172.21.15.32:0/964533858 >> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] conn(0x555e4286bda0 0x555e4286c1b0 crc :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:58:33.638 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:33.636+0000 7f3556e00640 1 -- 172.21.15.32:0/964533858 <== osd.6 v2:172.21.15.118:6825/801947386 1 ==== osd_op_reply(69 604.00000000 [create] v115'327 uv321 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f355000b040 con 0x555e4286bda0 2024-08-03T20:58:33.638 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:33.636+0000 7f355d25ef40 1 -- 172.21.15.32:0/964533858 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:70 12.15 12:a93a17c2:::604.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e115) -- 0x555e4286e8d0 con 0x555e4286bda0 2024-08-03T20:58:33.639 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:33.638+0000 7f3556e00640 1 -- 172.21.15.32:0/964533858 <== osd.6 v2:172.21.15.118:6825/801947386 2 ==== osd_op_reply(70 604.00000000 [delete] v115'328 uv328 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f355000b040 con 0x555e4286bda0 2024-08-03T20:58:33.639 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:33.638+0000 7f355d25ef40 1 -- 172.21.15.32:0/964533858 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:71 12.15 12:a93a17c2:::604.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115) -- 0x555e4286e8d0 con 0x555e4286bda0 2024-08-03T20:58:33.640 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::604.00000000:head# 2024-08-03T20:58:33.641 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:33.639+0000 7f3556e00640 1 -- 172.21.15.32:0/964533858 <== osd.6 v2:172.21.15.118:6825/801947386 3 ==== osd_op_reply(71 604.00000000 [create] v115'329 uv329 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f355000b040 con 0x555e4286bda0 2024-08-03T20:58:33.641 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:33.639+0000 7f355d25ef40 1 -- 172.21.15.32:0/964533858 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:72 12.15 12:a93a17c2:::604.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115) -- 0x555e4286aff0 con 0x555e4286bda0 2024-08-03T20:58:33.642 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:33.641+0000 7f3556e00640 1 -- 172.21.15.32:0/964533858 <== osd.6 v2:172.21.15.118:6825/801947386 4 ==== osd_op_reply(72 604.00000000 [setxattr (30)] v115'330 uv330 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f355000b040 con 0x555e4286bda0 2024-08-03T20:58:33.642 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:33.641+0000 7f355d25ef40 1 -- 172.21.15.32:0/964533858 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:73 12.15 12:a93a17c2:::604.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115) -- 0x555e4286aff0 con 0x555e4286bda0 2024-08-03T20:58:33.643 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:33.642+0000 7f3556e00640 1 -- 172.21.15.32:0/964533858 <== osd.6 v2:172.21.15.118:6825/801947386 5 ==== osd_op_reply(73 604.00000000 [setxattr (62)] v115'331 uv331 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f355000b040 con 0x555e4286bda0 2024-08-03T20:58:33.643 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:33.642+0000 7f355d25ef40 1 -- 172.21.15.32:0/964533858 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:74 12.15 12:a93a17c2:::604.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115) -- 0x555e4286aff0 con 0x555e4286bda0 2024-08-03T20:58:33.645 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:33.644+0000 7f3556e00640 1 -- 172.21.15.32:0/964533858 <== osd.6 v2:172.21.15.118:6825/801947386 6 ==== osd_op_reply(74 604.00000000 [omap-set-header] v115'332 uv332 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f355000b040 con 0x555e4286bda0 2024-08-03T20:58:33.645 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:33.644+0000 7f355d25ef40 1 -- 172.21.15.32:0/964533858 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:75 12.15 12:aa448500:::500.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115) -- 0x555e4286afd0 con 0x555e4286bda0 2024-08-03T20:58:33.646 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:33.645+0000 7f3556e00640 1 -- 172.21.15.32:0/964533858 <== osd.6 v2:172.21.15.118:6825/801947386 7 ==== osd_op_reply(75 500.00000000 [create] v115'333 uv326 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f355000b040 con 0x555e4286bda0 2024-08-03T20:58:33.646 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:33.645+0000 7f355d25ef40 1 -- 172.21.15.32:0/964533858 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:76 12.15 12:aa448500:::500.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e115) -- 0x555e4286afd0 con 0x555e4286bda0 2024-08-03T20:58:33.648 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:33.646+0000 7f3556e00640 1 -- 172.21.15.32:0/964533858 <== osd.6 v2:172.21.15.118:6825/801947386 8 ==== osd_op_reply(76 500.00000000 [delete] v115'334 uv334 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f355000b040 con 0x555e4286bda0 2024-08-03T20:58:33.648 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:33.647+0000 7f355d25ef40 1 -- 172.21.15.32:0/964533858 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:77 12.15 12:aa448500:::500.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115) -- 0x555e4286afd0 con 0x555e4286bda0 2024-08-03T20:58:33.649 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:33.648+0000 7f3556e00640 1 -- 172.21.15.32:0/964533858 <== osd.6 v2:172.21.15.118:6825/801947386 9 ==== osd_op_reply(77 500.00000000 [create] v115'335 uv335 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f355000b040 con 0x555e4286bda0 2024-08-03T20:58:33.649 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::500.00000000:head# 2024-08-03T20:58:33.649 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:33.648+0000 7f355d25ef40 1 -- 172.21.15.32:0/964533858 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:78 12.15 12:aa448500:::500.00000000:head [write 0~90 in=90b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115) -- 0x555e4286afd0 con 0x555e4286bda0 2024-08-03T20:58:33.651 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:33.649+0000 7f3556e00640 1 -- 172.21.15.32:0/964533858 <== osd.6 v2:172.21.15.118:6825/801947386 10 ==== osd_op_reply(78 500.00000000 [write 0~90] v115'336 uv336 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f355000b040 con 0x555e4286bda0 2024-08-03T20:58:33.651 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:33.649+0000 7f355d25ef40 1 -- 172.21.15.32:0/964533858 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:79 12.15 12:aa448500:::500.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115) -- 0x555e4286afd0 con 0x555e4286bda0 2024-08-03T20:58:33.653 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:33.651+0000 7f3556e00640 1 -- 172.21.15.32:0/964533858 <== osd.6 v2:172.21.15.118:6825/801947386 11 ==== osd_op_reply(79 500.00000000 [omap-set-header] v115'337 uv337 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f355000b040 con 0x555e4286bda0 2024-08-03T20:58:33.653 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:33.651+0000 7f355d25ef40 1 --2- 172.21.15.32:0/964533858 >> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] conn(0x555e4286f2b0 0x555e428716d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:58:33.653 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:33.651+0000 7f355d25ef40 1 -- 172.21.15.32:0/964533858 --> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] -- osd_op(unknown.0.0:80 12.d 12:b50e409b:::mds_snaptable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115) -- 0x555e42871c10 con 0x555e4286f2b0 2024-08-03T20:58:33.653 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:33.651+0000 7f3556400640 1 --2- 172.21.15.32:0/964533858 >> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] conn(0x555e4286f2b0 0x555e428716d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:58:33.654 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:33.652+0000 7f3556400640 1 --2- 172.21.15.32:0/964533858 >> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] conn(0x555e4286f2b0 0x555e428716d0 crc :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:58:33.655 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:33.654+0000 7f3556400640 1 -- 172.21.15.32:0/964533858 <== osd.7 v2:172.21.15.118:6818/653000048 1 ==== osd_op_reply(80 mds_snaptable [create] v115'148 uv147 ondisk = -17 ((17) File exists)) ==== 157+0+0 (crc 0 0 0) 0x7f354800abc0 con 0x555e4286f2b0 2024-08-03T20:58:33.655 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:33.654+0000 7f355d25ef40 1 -- 172.21.15.32:0/964533858 --> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] -- osd_op(unknown.0.0:81 12.d 12:b50e409b:::mds_snaptable:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e115) -- 0x555e42871cd0 con 0x555e4286f2b0 2024-08-03T20:58:33.657 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:33.655+0000 7f3556400640 1 -- 172.21.15.32:0/964533858 <== osd.7 v2:172.21.15.118:6818/653000048 2 ==== osd_op_reply(81 mds_snaptable [delete] v115'149 uv149 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f354800abc0 con 0x555e4286f2b0 2024-08-03T20:58:33.657 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:33.655+0000 7f355d25ef40 1 -- 172.21.15.32:0/964533858 --> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] -- osd_op(unknown.0.0:82 12.d 12:b50e409b:::mds_snaptable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115) -- 0x555e428720f0 con 0x555e4286f2b0 2024-08-03T20:58:33.658 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:33.657+0000 7f3556400640 1 -- 172.21.15.32:0/964533858 <== osd.7 v2:172.21.15.118:6818/653000048 3 ==== osd_op_reply(82 mds_snaptable [create] v115'150 uv150 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f354800abc0 con 0x555e4286f2b0 2024-08-03T20:58:33.659 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds_snaptable:head# 2024-08-03T20:58:33.659 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:33.657+0000 7f355d25ef40 1 -- 172.21.15.32:0/964533858 --> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] -- osd_op(unknown.0.0:83 12.d 12:b50e409b:::mds_snaptable:head [write 0~70 in=70b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115) -- 0x555e428720f0 con 0x555e4286f2b0 2024-08-03T20:58:33.660 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:33.658+0000 7f3556400640 1 -- 172.21.15.32:0/964533858 <== osd.7 v2:172.21.15.118:6818/653000048 4 ==== osd_op_reply(83 mds_snaptable [write 0~70] v115'151 uv151 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f354800abc0 con 0x555e4286f2b0 2024-08-03T20:58:33.660 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:33.658+0000 7f355d25ef40 1 -- 172.21.15.32:0/964533858 --> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] -- osd_op(unknown.0.0:84 12.d 12:b50e409b:::mds_snaptable:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115) -- 0x555e428720f0 con 0x555e4286f2b0 2024-08-03T20:58:33.661 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:33.660+0000 7f3556400640 1 -- 172.21.15.32:0/964533858 <== osd.7 v2:172.21.15.118:6818/653000048 5 ==== osd_op_reply(84 mds_snaptable [omap-set-header] v115'152 uv152 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f354800abc0 con 0x555e4286f2b0 2024-08-03T20:58:33.661 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:33.660+0000 7f355d25ef40 1 -- 172.21.15.32:0/964533858 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:85 12.1d 12:b871830b:::605.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115) -- 0x555e428720f0 con 0x555e42794f40 2024-08-03T20:58:33.663 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:33.661+0000 7f3556e00640 1 -- 172.21.15.32:0/964533858 <== osd.0 v2:172.21.15.32:6816/3066156116 44 ==== osd_op_reply(85 605.00000000 [create] v115'177 uv176 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f355000b040 con 0x555e42794f40 2024-08-03T20:58:33.663 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:33.661+0000 7f355d25ef40 1 -- 172.21.15.32:0/964533858 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:86 12.1d 12:b871830b:::605.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e115) -- 0x555e428720f0 con 0x555e42794f40 2024-08-03T20:58:33.664 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:33.663+0000 7f3556e00640 1 -- 172.21.15.32:0/964533858 <== osd.0 v2:172.21.15.32:6816/3066156116 45 ==== osd_op_reply(86 605.00000000 [delete] v115'178 uv178 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f355000b040 con 0x555e42794f40 2024-08-03T20:58:33.664 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:33.663+0000 7f355d25ef40 1 -- 172.21.15.32:0/964533858 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:87 12.1d 12:b871830b:::605.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115) -- 0x555e428720f0 con 0x555e42794f40 2024-08-03T20:58:33.665 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:33.664+0000 7f3556e00640 1 -- 172.21.15.32:0/964533858 <== osd.0 v2:172.21.15.32:6816/3066156116 46 ==== osd_op_reply(87 605.00000000 [create] v115'179 uv179 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f355000b040 con 0x555e42794f40 2024-08-03T20:58:33.665 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::605.00000000:head# 2024-08-03T20:58:33.666 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:33.664+0000 7f355d25ef40 1 -- 172.21.15.32:0/964533858 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:88 12.1d 12:b871830b:::605.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115) -- 0x555e42873100 con 0x555e42794f40 2024-08-03T20:58:33.666 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:33.665+0000 7f3556e00640 1 -- 172.21.15.32:0/964533858 <== osd.0 v2:172.21.15.32:6816/3066156116 47 ==== osd_op_reply(88 605.00000000 [setxattr (30)] v115'180 uv180 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f355000b040 con 0x555e42794f40 2024-08-03T20:58:33.667 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:33.665+0000 7f355d25ef40 1 -- 172.21.15.32:0/964533858 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:89 12.1d 12:b871830b:::605.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115) -- 0x555e42873100 con 0x555e42794f40 2024-08-03T20:58:33.668 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:33.666+0000 7f3556e00640 1 -- 172.21.15.32:0/964533858 <== osd.0 v2:172.21.15.32:6816/3066156116 48 ==== osd_op_reply(89 605.00000000 [setxattr (62)] v115'181 uv181 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f355000b040 con 0x555e42794f40 2024-08-03T20:58:33.668 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:33.666+0000 7f355d25ef40 1 -- 172.21.15.32:0/964533858 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:90 12.1d 12:b871830b:::605.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115) -- 0x555e428720f0 con 0x555e42794f40 2024-08-03T20:58:33.669 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:33.668+0000 7f3556e00640 1 -- 172.21.15.32:0/964533858 <== osd.0 v2:172.21.15.32:6816/3066156116 49 ==== osd_op_reply(90 605.00000000 [omap-set-header] v115'182 uv182 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f355000b040 con 0x555e42794f40 2024-08-03T20:58:33.669 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:33.668+0000 7f355d25ef40 1 -- 172.21.15.32:0/964533858 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:91 12.13 12:c91d4a1d:::mds0_inotable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115) -- 0x555e428720f0 con 0x555e428676f0 2024-08-03T20:58:33.670 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:33.669+0000 7f3555a00640 1 -- 172.21.15.32:0/964533858 <== osd.1 v2:172.21.15.32:6824/1551241759 5 ==== osd_op_reply(91 mds0_inotable [create] v115'294 uv288 ondisk = -17 ((17) File exists)) ==== 157+0+0 (crc 0 0 0) 0x7f354402c070 con 0x555e428676f0 2024-08-03T20:58:33.671 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:33.669+0000 7f355d25ef40 1 -- 172.21.15.32:0/964533858 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:92 12.13 12:c91d4a1d:::mds0_inotable:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e115) -- 0x555e42872490 con 0x555e428676f0 2024-08-03T20:58:33.672 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:33.670+0000 7f3555a00640 1 -- 172.21.15.32:0/964533858 <== osd.1 v2:172.21.15.32:6824/1551241759 6 ==== osd_op_reply(92 mds0_inotable [delete] v115'295 uv295 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f354402c070 con 0x555e428676f0 2024-08-03T20:58:33.672 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:33.670+0000 7f355d25ef40 1 -- 172.21.15.32:0/964533858 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:93 12.13 12:c91d4a1d:::mds0_inotable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115) -- 0x555e42872830 con 0x555e428676f0 2024-08-03T20:58:33.673 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds0_inotable:head# 2024-08-03T20:58:33.673 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:33.672+0000 7f3555a00640 1 -- 172.21.15.32:0/964533858 <== osd.1 v2:172.21.15.32:6824/1551241759 7 ==== osd_op_reply(93 mds0_inotable [create] v115'296 uv296 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f354402c070 con 0x555e428676f0 2024-08-03T20:58:33.673 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:33.672+0000 7f355d25ef40 1 -- 172.21.15.32:0/964533858 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:94 12.13 12:c91d4a1d:::mds0_inotable:head [write 0~34 in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115) -- 0x555e42872c20 con 0x555e428676f0 2024-08-03T20:58:33.674 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:33.673+0000 7f3555a00640 1 -- 172.21.15.32:0/964533858 <== osd.1 v2:172.21.15.32:6824/1551241759 8 ==== osd_op_reply(94 mds0_inotable [write 0~34] v115'297 uv297 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f354402c070 con 0x555e428676f0 2024-08-03T20:58:33.675 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:33.673+0000 7f355d25ef40 1 -- 172.21.15.32:0/964533858 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:95 12.13 12:c91d4a1d:::mds0_inotable:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115) -- 0x555e42872c20 con 0x555e428676f0 2024-08-03T20:58:33.676 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:33.674+0000 7f3555a00640 1 -- 172.21.15.32:0/964533858 <== osd.1 v2:172.21.15.32:6824/1551241759 9 ==== osd_op_reply(95 mds0_inotable [omap-set-header] v115'298 uv298 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f354402c070 con 0x555e428676f0 2024-08-03T20:58:33.676 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:33.674+0000 7f355d25ef40 1 -- 172.21.15.32:0/964533858 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:96 12.13 12:cd5a64a3:::100.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115) -- 0x555e42872bd0 con 0x555e428676f0 2024-08-03T20:58:33.677 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:33.676+0000 7f3555a00640 1 -- 172.21.15.32:0/964533858 <== osd.1 v2:172.21.15.32:6824/1551241759 10 ==== osd_op_reply(96 100.00000000 [create] v115'299 uv293 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f354402c070 con 0x555e428676f0 2024-08-03T20:58:33.677 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:33.676+0000 7f355d25ef40 1 -- 172.21.15.32:0/964533858 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:97 12.13 12:cd5a64a3:::100.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e115) -- 0x555e42872bd0 con 0x555e428676f0 2024-08-03T20:58:33.678 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:33.677+0000 7f3555a00640 1 -- 172.21.15.32:0/964533858 <== osd.1 v2:172.21.15.32:6824/1551241759 11 ==== osd_op_reply(97 100.00000000 [delete] v115'300 uv300 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f354402c070 con 0x555e428676f0 2024-08-03T20:58:33.679 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:33.677+0000 7f355d25ef40 1 -- 172.21.15.32:0/964533858 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:98 12.13 12:cd5a64a3:::100.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115) -- 0x555e42872bd0 con 0x555e428676f0 2024-08-03T20:58:33.679 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:33.678+0000 7f3555a00640 1 -- 172.21.15.32:0/964533858 <== osd.1 v2:172.21.15.32:6824/1551241759 12 ==== osd_op_reply(98 100.00000000 [create] v115'301 uv301 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f354402c070 con 0x555e428676f0 2024-08-03T20:58:33.680 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::100.00000000:head# 2024-08-03T20:58:33.680 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:33.678+0000 7f355d25ef40 1 -- 172.21.15.32:0/964533858 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:99 12.13 12:cd5a64a3:::100.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115) -- 0x555e42872bd0 con 0x555e428676f0 2024-08-03T20:58:33.681 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:33.679+0000 7f3555a00640 1 -- 172.21.15.32:0/964533858 <== osd.1 v2:172.21.15.32:6824/1551241759 13 ==== osd_op_reply(99 100.00000000 [omap-set-header] v115'302 uv302 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f354402c070 con 0x555e428676f0 2024-08-03T20:58:33.681 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:33.680+0000 7f355d25ef40 1 -- 172.21.15.32:0/964533858 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:100 12.13 12:cd5a64a3:::100.00000000:head [omap-set-vals in=5194b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115) -- 0x555e428730a0 con 0x555e428676f0 2024-08-03T20:58:33.683 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:33.681+0000 7f3555a00640 1 -- 172.21.15.32:0/964533858 <== osd.1 v2:172.21.15.32:6824/1551241759 14 ==== osd_op_reply(100 100.00000000 [omap-set-vals] v115'303 uv303 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f354402c070 con 0x555e428676f0 2024-08-03T20:58:33.683 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:33.681+0000 7f355d25ef40 1 --2- 172.21.15.32:0/964533858 >> [v2:172.21.15.32:6800/9088303,v1:172.21.15.32:6801/9088303] conn(0x555e42873a00 0x555e42876110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:58:33.683 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:33.681+0000 7f355d25ef40 1 -- 172.21.15.32:0/964533858 --> [v2:172.21.15.32:6800/9088303,v1:172.21.15.32:6801/9088303] -- osd_op(unknown.0.0:101 12.b 12:d0630e4c:::mds0_sessionmap:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115) -- 0x555e42876650 con 0x555e42873a00 2024-08-03T20:58:33.683 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:33.681+0000 7f3555a00640 1 --2- 172.21.15.32:0/964533858 >> [v2:172.21.15.32:6800/9088303,v1:172.21.15.32:6801/9088303] conn(0x555e42873a00 0x555e42876110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:58:33.684 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:33.682+0000 7f3555a00640 1 --2- 172.21.15.32:0/964533858 >> [v2:172.21.15.32:6800/9088303,v1:172.21.15.32:6801/9088303] conn(0x555e42873a00 0x555e42876110 crc :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:58:33.685 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:33.683+0000 7f3555a00640 1 -- 172.21.15.32:0/964533858 <== osd.3 v2:172.21.15.32:6800/9088303 1 ==== osd_op_reply(101 mds0_sessionmap [create] v115'128 uv127 ondisk = -17 ((17) File exists)) ==== 159+0+0 (crc 0 0 0) 0x7f354402c070 con 0x555e42873a00 2024-08-03T20:58:33.685 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:33.683+0000 7f355d25ef40 1 -- 172.21.15.32:0/964533858 --> [v2:172.21.15.32:6800/9088303,v1:172.21.15.32:6801/9088303] -- osd_op(unknown.0.0:102 12.b 12:d0630e4c:::mds0_sessionmap:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e115) -- 0x555e42876650 con 0x555e42873a00 2024-08-03T20:58:33.686 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:33.685+0000 7f3555a00640 1 -- 172.21.15.32:0/964533858 <== osd.3 v2:172.21.15.32:6800/9088303 2 ==== osd_op_reply(102 mds0_sessionmap [delete] v115'129 uv129 ondisk = 0) ==== 159+0+0 (crc 0 0 0) 0x7f354402c070 con 0x555e42873a00 2024-08-03T20:58:33.686 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:33.685+0000 7f355d25ef40 1 -- 172.21.15.32:0/964533858 --> [v2:172.21.15.32:6800/9088303,v1:172.21.15.32:6801/9088303] -- osd_op(unknown.0.0:103 12.b 12:d0630e4c:::mds0_sessionmap:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115) -- 0x555e42876650 con 0x555e42873a00 2024-08-03T20:58:33.687 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds0_sessionmap:head# 2024-08-03T20:58:33.688 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:33.686+0000 7f3555a00640 1 -- 172.21.15.32:0/964533858 <== osd.3 v2:172.21.15.32:6800/9088303 3 ==== osd_op_reply(103 mds0_sessionmap [create] v115'130 uv130 ondisk = 0) ==== 159+0+0 (crc 0 0 0) 0x7f354402c070 con 0x555e42873a00 2024-08-03T20:58:33.688 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:33.686+0000 7f355d25ef40 1 -- 172.21.15.32:0/964533858 --> [v2:172.21.15.32:6800/9088303,v1:172.21.15.32:6801/9088303] -- osd_op(unknown.0.0:104 12.b 12:d0630e4c:::mds0_sessionmap:head [omap-set-header in=14b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115) -- 0x555e42876650 con 0x555e42873a00 2024-08-03T20:58:33.689 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:33.687+0000 7f3555a00640 1 -- 172.21.15.32:0/964533858 <== osd.3 v2:172.21.15.32:6800/9088303 4 ==== osd_op_reply(104 mds0_sessionmap [omap-set-header] v115'131 uv131 ondisk = 0) ==== 159+0+0 (crc 0 0 0) 0x7f354402c070 con 0x555e42873a00 2024-08-03T20:58:33.689 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:33.687+0000 7f355d25ef40 1 -- 172.21.15.32:0/964533858 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:105 12.7 12:e0b41d06:::609.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115) -- 0x555e42876650 con 0x555e428676f0 2024-08-03T20:58:33.690 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:33.689+0000 7f3555a00640 1 -- 172.21.15.32:0/964533858 <== osd.1 v2:172.21.15.32:6824/1551241759 15 ==== osd_op_reply(105 609.00000000 [create] v115'177 uv176 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f354402c070 con 0x555e428676f0 2024-08-03T20:58:33.690 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:33.689+0000 7f355d25ef40 1 -- 172.21.15.32:0/964533858 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:106 12.7 12:e0b41d06:::609.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e115) -- 0x555e42876650 con 0x555e428676f0 2024-08-03T20:58:33.692 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:33.690+0000 7f3555a00640 1 -- 172.21.15.32:0/964533858 <== osd.1 v2:172.21.15.32:6824/1551241759 16 ==== osd_op_reply(106 609.00000000 [delete] v115'178 uv178 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f354402c070 con 0x555e428676f0 2024-08-03T20:58:33.692 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:33.690+0000 7f355d25ef40 1 -- 172.21.15.32:0/964533858 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:107 12.7 12:e0b41d06:::609.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115) -- 0x555e42876650 con 0x555e428676f0 2024-08-03T20:58:33.693 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:33.692+0000 7f3555a00640 1 -- 172.21.15.32:0/964533858 <== osd.1 v2:172.21.15.32:6824/1551241759 17 ==== osd_op_reply(107 609.00000000 [create] v115'179 uv179 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f354402c070 con 0x555e428676f0 2024-08-03T20:58:33.693 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::609.00000000:head# 2024-08-03T20:58:33.693 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:33.692+0000 7f355d25ef40 1 -- 172.21.15.32:0/964533858 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:108 12.7 12:e0b41d06:::609.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115) -- 0x555e42877660 con 0x555e428676f0 2024-08-03T20:58:33.695 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:33.693+0000 7f3555a00640 1 -- 172.21.15.32:0/964533858 <== osd.1 v2:172.21.15.32:6824/1551241759 18 ==== osd_op_reply(108 609.00000000 [setxattr (30)] v115'180 uv180 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f354402c070 con 0x555e428676f0 2024-08-03T20:58:33.695 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:33.693+0000 7f355d25ef40 1 -- 172.21.15.32:0/964533858 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:109 12.7 12:e0b41d06:::609.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115) -- 0x555e42877660 con 0x555e428676f0 2024-08-03T20:58:33.696 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:33.694+0000 7f3555a00640 1 -- 172.21.15.32:0/964533858 <== osd.1 v2:172.21.15.32:6824/1551241759 19 ==== osd_op_reply(109 609.00000000 [setxattr (62)] v115'181 uv181 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f354402c070 con 0x555e428676f0 2024-08-03T20:58:33.696 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:33.694+0000 7f355d25ef40 1 -- 172.21.15.32:0/964533858 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:110 12.7 12:e0b41d06:::609.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115) -- 0x555e42876650 con 0x555e428676f0 2024-08-03T20:58:33.697 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:33.696+0000 7f3555a00640 1 -- 172.21.15.32:0/964533858 <== osd.1 v2:172.21.15.32:6824/1551241759 20 ==== osd_op_reply(110 609.00000000 [omap-set-header] v115'182 uv182 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f354402c070 con 0x555e428676f0 2024-08-03T20:58:33.697 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:33.696+0000 7f355d25ef40 1 -- 172.21.15.32:0/964533858 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:111 12.f 12:f1f69726:::400.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115) -- 0x555e42876650 con 0x555e4286bda0 2024-08-03T20:58:33.699 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:33.697+0000 7f3556e00640 1 -- 172.21.15.32:0/964533858 <== osd.6 v2:172.21.15.118:6825/801947386 12 ==== osd_op_reply(111 400.00000000 [create] v115'151 uv150 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f355000b040 con 0x555e4286bda0 2024-08-03T20:58:33.699 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:33.697+0000 7f355d25ef40 1 -- 172.21.15.32:0/964533858 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:112 12.f 12:f1f69726:::400.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e115) -- 0x555e42876650 con 0x555e4286bda0 2024-08-03T20:58:33.700 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:33.699+0000 7f3556e00640 1 -- 172.21.15.32:0/964533858 <== osd.6 v2:172.21.15.118:6825/801947386 13 ==== osd_op_reply(112 400.00000000 [delete] v115'152 uv152 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f355000b040 con 0x555e4286bda0 2024-08-03T20:58:33.700 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:33.699+0000 7f355d25ef40 1 -- 172.21.15.32:0/964533858 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:113 12.f 12:f1f69726:::400.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115) -- 0x555e42876650 con 0x555e4286bda0 2024-08-03T20:58:33.701 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::400.00000000:head# 2024-08-03T20:58:33.702 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:33.700+0000 7f3556e00640 1 -- 172.21.15.32:0/964533858 <== osd.6 v2:172.21.15.118:6825/801947386 14 ==== osd_op_reply(113 400.00000000 [create] v115'153 uv153 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f355000b040 con 0x555e4286bda0 2024-08-03T20:58:33.702 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:33.700+0000 7f355d25ef40 1 -- 172.21.15.32:0/964533858 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:114 12.f 12:f1f69726:::400.00000000:head [write 0~22 in=22b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115) -- 0x555e42876650 con 0x555e4286bda0 2024-08-03T20:58:33.703 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:33.702+0000 7f3556e00640 1 -- 172.21.15.32:0/964533858 <== osd.6 v2:172.21.15.118:6825/801947386 15 ==== osd_op_reply(114 400.00000000 [write 0~22] v115'154 uv154 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f355000b040 con 0x555e4286bda0 2024-08-03T20:58:33.703 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:33.702+0000 7f355d25ef40 1 -- 172.21.15.32:0/964533858 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:115 12.f 12:f1f69726:::400.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115) -- 0x555e42876650 con 0x555e4286bda0 2024-08-03T20:58:33.705 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:33.703+0000 7f3556e00640 1 -- 172.21.15.32:0/964533858 <== osd.6 v2:172.21.15.118:6825/801947386 16 ==== osd_op_reply(115 400.00000000 [omap-set-header] v115'155 uv155 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f355000b040 con 0x555e4286bda0 2024-08-03T20:58:33.705 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:33.704+0000 7f355d25ef40 1 -- 172.21.15.32:0/964533858 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:116 12.1f 12:fe07bba1:::100.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115) -- 0x555e42876650 con 0x555e4286bda0 2024-08-03T20:58:33.707 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:33.705+0000 7f3556e00640 1 -- 172.21.15.32:0/964533858 <== osd.6 v2:172.21.15.118:6825/801947386 17 ==== osd_op_reply(116 100.00000000.inode [create] v115'363 uv355 ondisk = -17 ((17) File exists)) ==== 162+0+0 (crc 0 0 0) 0x7f355000b040 con 0x555e4286bda0 2024-08-03T20:58:33.707 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:33.705+0000 7f355d25ef40 1 -- 172.21.15.32:0/964533858 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:117 12.1f 12:fe07bba1:::100.00000000.inode:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e115) -- 0x555e42876650 con 0x555e4286bda0 2024-08-03T20:58:33.708 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:33.707+0000 7f3556e00640 1 -- 172.21.15.32:0/964533858 <== osd.6 v2:172.21.15.118:6825/801947386 18 ==== osd_op_reply(117 100.00000000.inode [delete] v115'364 uv364 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7f355000b040 con 0x555e4286bda0 2024-08-03T20:58:33.708 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:33.707+0000 7f355d25ef40 1 -- 172.21.15.32:0/964533858 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:118 12.1f 12:fe07bba1:::100.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115) -- 0x555e42876650 con 0x555e4286bda0 2024-08-03T20:58:33.710 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::100.00000000.inode:head# 2024-08-03T20:58:33.710 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:33.708+0000 7f3556e00640 1 -- 172.21.15.32:0/964533858 <== osd.6 v2:172.21.15.118:6825/801947386 19 ==== osd_op_reply(118 100.00000000.inode [create] v115'365 uv365 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7f355000b040 con 0x555e4286bda0 2024-08-03T20:58:33.710 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:33.708+0000 7f355d25ef40 1 -- 172.21.15.32:0/964533858 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:119 12.1f 12:fe07bba1:::100.00000000.inode:head [write 0~582 in=582b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115) -- 0x555e42876650 con 0x555e4286bda0 2024-08-03T20:58:33.712 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:33.710+0000 7f3556e00640 1 -- 172.21.15.32:0/964533858 <== osd.6 v2:172.21.15.118:6825/801947386 20 ==== osd_op_reply(119 100.00000000.inode [write 0~582] v115'366 uv366 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7f355000b040 con 0x555e4286bda0 2024-08-03T20:58:33.712 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:33.710+0000 7f355d25ef40 1 -- 172.21.15.32:0/964533858 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:120 12.1f 12:fe07bba1:::100.00000000.inode:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115) -- 0x555e42876650 con 0x555e4286bda0 2024-08-03T20:58:33.713 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:33.712+0000 7f3556e00640 1 -- 172.21.15.32:0/964533858 <== osd.6 v2:172.21.15.118:6825/801947386 21 ==== osd_op_reply(120 100.00000000.inode [omap-set-header] v115'367 uv367 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7f355000b040 con 0x555e4286bda0 2024-08-03T20:58:33.713 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:33.712+0000 7f355d25ef40 1 -- 172.21.15.32:0/964533858 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:121 12.1f 12:ff5b34d6:::1.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115) -- 0x555e42876650 con 0x555e4286bda0 2024-08-03T20:58:33.714 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:33.713+0000 7f3556e00640 1 -- 172.21.15.32:0/964533858 <== osd.6 v2:172.21.15.118:6825/801947386 22 ==== osd_op_reply(121 1.00000000 [create] v115'368 uv362 ondisk = -17 ((17) File exists)) ==== 154+0+0 (crc 0 0 0) 0x7f355000b040 con 0x555e4286bda0 2024-08-03T20:58:33.715 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:33.713+0000 7f355d25ef40 1 -- 172.21.15.32:0/964533858 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:122 12.1f 12:ff5b34d6:::1.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e115) -- 0x555e42876650 con 0x555e4286bda0 2024-08-03T20:58:33.716 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:33.715+0000 7f3556e00640 1 -- 172.21.15.32:0/964533858 <== osd.6 v2:172.21.15.118:6825/801947386 23 ==== osd_op_reply(122 1.00000000 [delete] v115'369 uv369 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7f355000b040 con 0x555e4286bda0 2024-08-03T20:58:33.716 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:33.715+0000 7f355d25ef40 1 -- 172.21.15.32:0/964533858 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:123 12.1f 12:ff5b34d6:::1.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115) -- 0x555e42876650 con 0x555e4286bda0 2024-08-03T20:58:33.717 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::1.00000000:head# 2024-08-03T20:58:33.718 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:33.716+0000 7f3556e00640 1 -- 172.21.15.32:0/964533858 <== osd.6 v2:172.21.15.118:6825/801947386 24 ==== osd_op_reply(123 1.00000000 [create] v115'370 uv370 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7f355000b040 con 0x555e4286bda0 2024-08-03T20:58:33.718 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:33.716+0000 7f355d25ef40 1 -- 172.21.15.32:0/964533858 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:124 12.1f 12:ff5b34d6:::1.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115) -- 0x555e42877660 con 0x555e4286bda0 2024-08-03T20:58:33.719 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:33.718+0000 7f3556e00640 1 -- 172.21.15.32:0/964533858 <== osd.6 v2:172.21.15.118:6825/801947386 25 ==== osd_op_reply(124 1.00000000 [setxattr (30)] v115'371 uv371 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7f355000b040 con 0x555e4286bda0 2024-08-03T20:58:33.719 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:33.718+0000 7f355d25ef40 1 -- 172.21.15.32:0/964533858 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:125 12.1f 12:ff5b34d6:::1.00000000:head [setxattr parent (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115) -- 0x555e42877660 con 0x555e4286bda0 2024-08-03T20:58:33.720 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:33.719+0000 7f3556e00640 1 -- 172.21.15.32:0/964533858 <== osd.6 v2:172.21.15.118:6825/801947386 26 ==== osd_op_reply(125 1.00000000 [setxattr (30)] v115'372 uv372 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7f355000b040 con 0x555e4286bda0 2024-08-03T20:58:33.721 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:33.719+0000 7f355d25ef40 1 -- 172.21.15.32:0/964533858 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:126 12.1f 12:ff5b34d6:::1.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115) -- 0x555e42876650 con 0x555e4286bda0 2024-08-03T20:58:33.722 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:33.721+0000 7f3556e00640 1 -- 172.21.15.32:0/964533858 <== osd.6 v2:172.21.15.118:6825/801947386 27 ==== osd_op_reply(126 1.00000000 [omap-set-header] v115'373 uv373 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7f355000b040 con 0x555e4286bda0 2024-08-03T20:58:33.722 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:33.721+0000 7f355d25ef40 1 -- 172.21.15.32:0/964533858 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:127 12.1f 12:ff5b34d6:::1.00000000:head [omap-set-vals in=584b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115) -- 0x555e42876b00 con 0x555e4286bda0 2024-08-03T20:58:33.725 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:33.722+0000 7f3556e00640 1 -- 172.21.15.32:0/964533858 <== osd.6 v2:172.21.15.118:6825/801947386 28 ==== osd_op_reply(127 1.00000000 [omap-set-vals] v115'374 uv374 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7f355000b040 con 0x555e4286bda0 2024-08-03T20:58:33.725 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:33.722+0000 7f355d25ef40 1 -- 172.21.15.32:0/964533858 >> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] conn(0x555e42794f40 msgr2=0x555e42795350 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:58:33.725 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:33.722+0000 7f355d25ef40 1 --2- 172.21.15.32:0/964533858 >> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] conn(0x555e42794f40 0x555e42795350 crc :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:33.725 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:33.722+0000 7f355d25ef40 1 -- 172.21.15.32:0/964533858 >> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] conn(0x555e428676f0 msgr2=0x555e42867b00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:58:33.725 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:33.722+0000 7f355d25ef40 1 --2- 172.21.15.32:0/964533858 >> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] conn(0x555e428676f0 0x555e42867b00 crc :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:33.725 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:33.722+0000 7f355d25ef40 1 -- 172.21.15.32:0/964533858 >> [v2:172.21.15.32:6800/9088303,v1:172.21.15.32:6801/9088303] conn(0x555e42873a00 msgr2=0x555e42876110 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:58:33.725 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:33.722+0000 7f355d25ef40 1 --2- 172.21.15.32:0/964533858 >> [v2:172.21.15.32:6800/9088303,v1:172.21.15.32:6801/9088303] conn(0x555e42873a00 0x555e42876110 crc :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:33.725 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:33.722+0000 7f355d25ef40 1 -- 172.21.15.32:0/964533858 >> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] conn(0x555e42862a00 msgr2=0x555e42864e20 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:58:33.725 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:33.722+0000 7f355d25ef40 1 --2- 172.21.15.32:0/964533858 >> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] conn(0x555e42862a00 0x555e42864e20 crc :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:33.725 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:33.722+0000 7f355d25ef40 1 -- 172.21.15.32:0/964533858 >> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] conn(0x555e4286bda0 msgr2=0x555e4286c1b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:58:33.725 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:33.722+0000 7f355d25ef40 1 --2- 172.21.15.32:0/964533858 >> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] conn(0x555e4286bda0 0x555e4286c1b0 crc :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:33.725 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:33.722+0000 7f355d25ef40 1 -- 172.21.15.32:0/964533858 >> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] conn(0x555e4286f2b0 msgr2=0x555e428716d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:58:33.725 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:33.722+0000 7f355d25ef40 1 --2- 172.21.15.32:0/964533858 >> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] conn(0x555e4286f2b0 0x555e428716d0 crc :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:33.726 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:33.722+0000 7f3556e00640 1 -- 172.21.15.32:0/964533858 reap_dead start 2024-08-03T20:58:33.726 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:33.723+0000 7f3556e00640 1 -- 172.21.15.32:0/964533858 reap_dead start 2024-08-03T20:58:33.726 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:33.723+0000 7f3556e00640 1 -- 172.21.15.32:0/964533858 reap_dead start 2024-08-03T20:58:33.726 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:33.723+0000 7f355d25ef40 1 -- 172.21.15.32:0/964533858 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f35380759f0 msgr2=0x7f3538077f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:58:33.726 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:33.723+0000 7f355d25ef40 1 --2- 172.21.15.32:0/964533858 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f35380759f0 0x7f3538077f00 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7f354800b770 tx=0x7f3548004880 comp rx=0 tx=0).stop 2024-08-03T20:58:33.726 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:33.723+0000 7f355d25ef40 1 -- 172.21.15.32:0/964533858 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555e4272cb40 msgr2=0x555e42857e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:58:33.726 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:33.723+0000 7f355d25ef40 1 --2- 172.21.15.32:0/964533858 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555e4272cb40 0x555e42857e70 secure :-1 s=READY pgs=1801 cs=0 l=1 rev1=1 crypto rx=0x555e427beb10 tx=0x7f354400fea0 comp rx=0 tx=0).stop 2024-08-03T20:58:33.726 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:33.723+0000 7f355d25ef40 1 -- 172.21.15.32:0/964533858 shutdown_connections 2024-08-03T20:58:33.726 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:33.723+0000 7f355d25ef40 1 --2- 172.21.15.32:0/964533858 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f35380759f0 0x7f3538077f00 unknown :-1 s=CLOSED pgs=731 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:33.726 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:33.723+0000 7f355d25ef40 1 --2- 172.21.15.32:0/964533858 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555e4272cb40 0x555e42857e70 unknown :-1 s=CLOSED pgs=1801 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:33.726 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:33.723+0000 7f355d25ef40 1 -- 172.21.15.32:0/964533858 >> 172.21.15.32:0/964533858 conn(0x555e42607ac0 msgr2=0x555e427baf90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:58:33.726 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:33.723+0000 7f355d25ef40 1 -- 172.21.15.32:0/964533858 shutdown_connections 2024-08-03T20:58:33.727 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:33.723+0000 7f355d25ef40 1 -- 172.21.15.32:0/964533858 wait complete. 2024-08-03T20:58:33.738 DEBUG:teuthology.orchestra.run.smithi032:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata truncate 100.00000000.inode 0 2024-08-03T20:58:33.882 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:33.881+0000 7fa98bf71f40 1 Processor -- start 2024-08-03T20:58:33.883 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:33.881+0000 7fa98bf71f40 1 -- start start 2024-08-03T20:58:33.885 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:33.882+0000 7fa98bf71f40 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x56408175ab40 0x564081714f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:58:33.885 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:33.882+0000 7fa98bf71f40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5640817cdbe0 0x5640817154c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:58:33.885 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:33.882+0000 7fa98bf71f40 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x564081878400 0x5640818755d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:58:33.885 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:33.882+0000 7fa98bf71f40 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x564081663e30 con 0x564081878400 2024-08-03T20:58:33.885 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:33.882+0000 7fa98bf71f40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x5640816c1bf0 con 0x5640817cdbe0 2024-08-03T20:58:33.885 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:33.882+0000 7fa98bf71f40 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x564081876cb0 con 0x56408175ab40 2024-08-03T20:58:33.885 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:33.882+0000 7fa985a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x564081878400 0x5640818755d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:58:33.885 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:33.882+0000 7fa985a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x564081878400 0x5640818755d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:41600/0 (socket says 172.21.15.32:41600) 2024-08-03T20:58:33.885 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:33.882+0000 7fa985a00640 1 -- 172.21.15.32:0/1496687359 learned_addr learned my addr 172.21.15.32:0/1496687359 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:58:33.885 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:33.882+0000 7fa97fe00640 1 --2- 172.21.15.32:0/1496687359 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5640817cdbe0 0x5640817154c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:58:33.886 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:33.882+0000 7fa985000640 1 --2- 172.21.15.32:0/1496687359 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x56408175ab40 0x564081714f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:58:33.886 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:33.883+0000 7fa985a00640 1 -- 172.21.15.32:0/1496687359 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x56408175ab40 msgr2=0x564081714f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:58:33.886 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:33.883+0000 7fa985a00640 1 --2- 172.21.15.32:0/1496687359 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x56408175ab40 0x564081714f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:33.886 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:33.883+0000 7fa985a00640 1 -- 172.21.15.32:0/1496687359 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5640817cdbe0 msgr2=0x5640817154c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:58:33.886 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:33.883+0000 7fa985a00640 1 --2- 172.21.15.32:0/1496687359 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5640817cdbe0 0x5640817154c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:33.886 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:33.883+0000 7fa985a00640 1 -- 172.21.15.32:0/1496687359 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5640817c74f0 con 0x564081878400 2024-08-03T20:58:33.886 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:33.883+0000 7fa97fe00640 1 --2- 172.21.15.32:0/1496687359 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5640817cdbe0 0x5640817154c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-03T20:58:33.886 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:33.883+0000 7fa985000640 1 --2- 172.21.15.32:0/1496687359 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x56408175ab40 0x564081714f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-03T20:58:33.886 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:33.883+0000 7fa985a00640 1 --2- 172.21.15.32:0/1496687359 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x564081878400 0x5640818755d0 secure :-1 s=READY pgs=1803 cs=0 l=1 rev1=1 crypto rx=0x7fa98000d790 tx=0x7fa98000dc50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fa54c37bbc8cc8c2 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:58:33.886 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:33.883+0000 7fa97ea00640 1 -- 172.21.15.32:0/1496687359 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa98000dea0 con 0x564081878400 2024-08-03T20:58:33.886 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:33.883+0000 7fa97ea00640 1 -- 172.21.15.32:0/1496687359 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fa98000bbe0 con 0x564081878400 2024-08-03T20:58:33.887 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:33.883+0000 7fa97ea00640 1 -- 172.21.15.32:0/1496687359 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa98000f430 con 0x564081878400 2024-08-03T20:58:33.887 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:33.884+0000 7fa98bf71f40 1 -- 172.21.15.32:0/1496687359 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x564081878400 msgr2=0x5640818755d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:58:33.888 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:33.884+0000 7fa98bf71f40 1 --2- 172.21.15.32:0/1496687359 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x564081878400 0x5640818755d0 secure :-1 s=READY pgs=1803 cs=0 l=1 rev1=1 crypto rx=0x7fa98000d790 tx=0x7fa98000dc50 comp rx=0 tx=0).stop 2024-08-03T20:58:33.888 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:33.884+0000 7fa98bf71f40 1 -- 172.21.15.32:0/1496687359 shutdown_connections 2024-08-03T20:58:33.888 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:33.884+0000 7fa98bf71f40 1 --2- 172.21.15.32:0/1496687359 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5640817cdbe0 0x5640817154c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:33.888 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:33.884+0000 7fa98bf71f40 1 --2- 172.21.15.32:0/1496687359 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x564081878400 0x5640818755d0 unknown :-1 s=CLOSED pgs=1803 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:33.888 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:33.884+0000 7fa98bf71f40 1 --2- 172.21.15.32:0/1496687359 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x56408175ab40 0x564081714f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:33.888 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:33.884+0000 7fa98bf71f40 1 -- 172.21.15.32:0/1496687359 >> 172.21.15.32:0/1496687359 conn(0x564081635ac0 msgr2=0x5640817cb190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:58:33.888 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:33.884+0000 7fa98bf71f40 1 -- 172.21.15.32:0/1496687359 shutdown_connections 2024-08-03T20:58:33.888 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:33.884+0000 7fa98bf71f40 1 -- 172.21.15.32:0/1496687359 wait complete. 2024-08-03T20:58:33.888 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:33.885+0000 7fa98bf71f40 1 Processor -- start 2024-08-03T20:58:33.888 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:33.885+0000 7fa98bf71f40 1 -- start start 2024-08-03T20:58:33.890 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:33.887+0000 7fa98bf71f40 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56408175ab40 0x564081713c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:58:33.890 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:33.887+0000 7fa98bf71f40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5640817cdbe0 0x5640817c23c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:58:33.891 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:33.887+0000 7fa98bf71f40 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x564081878400 0x564081714f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:58:33.891 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:33.887+0000 7fa98bf71f40 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x5640817e9930 con 0x56408175ab40 2024-08-03T20:58:33.891 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:33.887+0000 7fa98bf71f40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x564081878050 con 0x5640817cdbe0 2024-08-03T20:58:33.891 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:33.887+0000 7fa98bf71f40 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x56408175e950 con 0x564081878400 2024-08-03T20:58:33.891 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:33.887+0000 7fa985000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56408175ab40 0x564081713c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:58:33.891 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:33.887+0000 7fa985000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56408175ab40 0x564081713c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:41614/0 (socket says 172.21.15.32:41614) 2024-08-03T20:58:33.891 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:33.887+0000 7fa985000640 1 -- 172.21.15.32:0/1264430707 learned_addr learned my addr 172.21.15.32:0/1264430707 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:58:33.891 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:33.887+0000 7fa97fe00640 1 --2- 172.21.15.32:0/1264430707 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5640817cdbe0 0x5640817c23c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:58:33.891 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:33.887+0000 7fa985a00640 1 --2- 172.21.15.32:0/1264430707 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x564081878400 0x564081714f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:58:33.891 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:33.887+0000 7fa985000640 1 -- 172.21.15.32:0/1264430707 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x564081878400 msgr2=0x564081714f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:58:33.891 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:33.887+0000 7fa985000640 1 --2- 172.21.15.32:0/1264430707 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x564081878400 0x564081714f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:33.892 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:33.888+0000 7fa985000640 1 -- 172.21.15.32:0/1264430707 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5640817cdbe0 msgr2=0x5640817c23c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:58:33.892 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:33.888+0000 7fa985000640 1 --2- 172.21.15.32:0/1264430707 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5640817cdbe0 0x5640817c23c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:33.892 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:33.888+0000 7fa985000640 1 -- 172.21.15.32:0/1264430707 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564081714900 con 0x56408175ab40 2024-08-03T20:58:33.892 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:33.888+0000 7fa985000640 1 --2- 172.21.15.32:0/1264430707 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56408175ab40 0x564081713c50 secure :-1 s=READY pgs=1804 cs=0 l=1 rev1=1 crypto rx=0x7fa97800dfd0 tx=0x7fa97800b5b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:58:33.892 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:33.888+0000 7fa97d600640 1 -- 172.21.15.32:0/1264430707 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa978011480 con 0x56408175ab40 2024-08-03T20:58:33.892 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:33.888+0000 7fa97d600640 1 -- 172.21.15.32:0/1264430707 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fa978016070 con 0x56408175ab40 2024-08-03T20:58:33.892 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:33.888+0000 7fa97d600640 1 -- 172.21.15.32:0/1264430707 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9780118a0 con 0x56408175ab40 2024-08-03T20:58:33.892 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:33.888+0000 7fa98bf71f40 1 -- 172.21.15.32:0/1264430707 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x56408175de70 con 0x56408175ab40 2024-08-03T20:58:33.892 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:33.888+0000 7fa98bf71f40 1 -- 172.21.15.32:0/1264430707 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56408175e430 con 0x56408175ab40 2024-08-03T20:58:33.893 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:33.890+0000 7fa97d600640 1 -- 172.21.15.32:0/1264430707 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7fa978011a40 con 0x56408175ab40 2024-08-03T20:58:33.894 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:33.891+0000 7fa97d600640 1 --2- 172.21.15.32:0/1264430707 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fa9640759f0 0x7fa964077f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:58:33.894 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:33.891+0000 7fa97d600640 1 -- 172.21.15.32:0/1264430707 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(115..115 src has 1..115) ==== 8434+0+0 (secure 0 0 0) 0x7fa978097df0 con 0x56408175ab40 2024-08-03T20:58:33.894 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:33.892+0000 7fa97fe00640 1 --2- 172.21.15.32:0/1264430707 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fa9640759f0 0x7fa964077f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:58:33.894 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:33.892+0000 7fa98bf71f40 1 --2- 172.21.15.32:0/1264430707 >> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] conn(0x5640817eca20 0x5640817ece30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:58:33.894 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:33.892+0000 7fa98bf71f40 1 -- 172.21.15.32:0/1264430707 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:1 12.1f 12:fe07bba1:::100.00000000.inode:head [truncate 0] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115) -- 0x5640817ce8b0 con 0x5640817eca20 2024-08-03T20:58:33.894 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:33.892+0000 7fa985a00640 1 --2- 172.21.15.32:0/1264430707 >> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] conn(0x5640817eca20 0x5640817ece30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:58:33.895 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:33.892+0000 7fa97fe00640 1 --2- 172.21.15.32:0/1264430707 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fa9640759f0 0x7fa964077f00 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7fa970055000 tx=0x7fa970003020 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:58:33.895 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:33.893+0000 7fa985a00640 1 --2- 172.21.15.32:0/1264430707 >> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] conn(0x5640817eca20 0x5640817ece30 crc :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:58:33.897 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:33.895+0000 7fa985a00640 1 -- 172.21.15.32:0/1264430707 <== osd.6 v2:172.21.15.118:6825/801947386 1 ==== osd_op_reply(1 100.00000000.inode [truncate 0] v115'375 uv375 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7fa980015020 con 0x5640817eca20 2024-08-03T20:58:33.897 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:33.895+0000 7fa98bf71f40 1 -- 172.21.15.32:0/1264430707 >> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] conn(0x5640817eca20 msgr2=0x5640817ece30 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:58:33.897 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:33.895+0000 7fa98bf71f40 1 --2- 172.21.15.32:0/1264430707 >> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] conn(0x5640817eca20 0x5640817ece30 crc :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:33.897 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:33.895+0000 7fa98bf71f40 1 -- 172.21.15.32:0/1264430707 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fa9640759f0 msgr2=0x7fa964077f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:58:33.898 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:33.895+0000 7fa98bf71f40 1 --2- 172.21.15.32:0/1264430707 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fa9640759f0 0x7fa964077f00 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7fa970055000 tx=0x7fa970003020 comp rx=0 tx=0).stop 2024-08-03T20:58:33.898 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:33.895+0000 7fa98bf71f40 1 -- 172.21.15.32:0/1264430707 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56408175ab40 msgr2=0x564081713c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:58:33.898 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:33.895+0000 7fa98bf71f40 1 --2- 172.21.15.32:0/1264430707 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56408175ab40 0x564081713c50 secure :-1 s=READY pgs=1804 cs=0 l=1 rev1=1 crypto rx=0x7fa97800dfd0 tx=0x7fa97800b5b0 comp rx=0 tx=0).stop 2024-08-03T20:58:33.898 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:33.895+0000 7fa985a00640 1 -- 172.21.15.32:0/1264430707 reap_dead start 2024-08-03T20:58:33.898 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:33.895+0000 7fa98bf71f40 1 -- 172.21.15.32:0/1264430707 shutdown_connections 2024-08-03T20:58:33.898 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:33.895+0000 7fa98bf71f40 1 -- 172.21.15.32:0/1264430707 >> 172.21.15.32:0/1264430707 conn(0x564081635ac0 msgr2=0x5640817e8f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:58:33.898 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:33.895+0000 7fa98bf71f40 1 -- 172.21.15.32:0/1264430707 shutdown_connections 2024-08-03T20:58:33.898 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:33.895+0000 7fa98bf71f40 1 -- 172.21.15.32:0/1264430707 wait complete. 2024-08-03T20:58:33.911 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs joinable true 2024-08-03T20:58:34.142 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:34.140+0000 7fe6b9200640 1 -- 172.21.15.32:0/4073041721 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe6b40caae0 msgr2=0x7fe6b40cd950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:58:34.143 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:34.140+0000 7fe6b9200640 1 --2- 172.21.15.32:0/4073041721 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe6b40caae0 0x7fe6b40cd950 secure :-1 s=READY pgs=1805 cs=0 l=1 rev1=1 crypto rx=0x7fe6ac005fd0 tx=0x7fe6ac02aa90 comp rx=0 tx=0).stop 2024-08-03T20:58:34.143 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:34.141+0000 7fe6b9200640 1 -- 172.21.15.32:0/4073041721 shutdown_connections 2024-08-03T20:58:34.143 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:34.141+0000 7fe6b9200640 1 --2- 172.21.15.32:0/4073041721 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe6b40caae0 0x7fe6b40cd950 unknown :-1 s=CLOSED pgs=1805 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:34.143 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:34.141+0000 7fe6b9200640 1 --2- 172.21.15.32:0/4073041721 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fe6b40cf6a0 0x7fe6b40742a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:34.143 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:34.141+0000 7fe6b9200640 1 -- 172.21.15.32:0/4073041721 >> 172.21.15.32:0/4073041721 conn(0x7fe6b40cfd80 msgr2=0x7fe6b40d0190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:58:34.143 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:34.141+0000 7fe6b9200640 1 -- 172.21.15.32:0/4073041721 shutdown_connections 2024-08-03T20:58:34.143 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:34.141+0000 7fe6b9200640 1 -- 172.21.15.32:0/4073041721 wait complete. 2024-08-03T20:58:34.144 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:34.143+0000 7fe6b9200640 1 Processor -- start 2024-08-03T20:58:34.144 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:34.143+0000 7fe6b9200640 1 -- start start 2024-08-03T20:58:34.147 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:34.144+0000 7fe6b9200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe6b40700f0 0x7fe6b41f2e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:58:34.147 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:34.144+0000 7fe6b9200640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fe6b40cf6a0 0x7fe6b40c5c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:58:34.147 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:34.144+0000 7fe6b9200640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fe6b40c61a0 0x7fe6b41f4d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:58:34.148 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:34.144+0000 7fe6b9200640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fe6b415b530 con 0x7fe6b40700f0 2024-08-03T20:58:34.148 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:34.144+0000 7fe6b9200640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7fe6b40cb260 con 0x7fe6b40c61a0 2024-08-03T20:58:34.148 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:34.144+0000 7fe6b9200640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7fe6b40cde90 con 0x7fe6b40cf6a0 2024-08-03T20:58:34.148 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:34.145+0000 7fe6b3400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe6b40700f0 0x7fe6b41f2e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:58:34.148 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:34.145+0000 7fe6b3400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe6b40700f0 0x7fe6b41f2e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:41628/0 (socket says 172.21.15.32:41628) 2024-08-03T20:58:34.148 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:34.145+0000 7fe6b3400640 1 -- 172.21.15.32:0/1065458635 learned_addr learned my addr 172.21.15.32:0/1065458635 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:58:34.148 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:34.145+0000 7fe6b2a00640 1 --2- 172.21.15.32:0/1065458635 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fe6b40cf6a0 0x7fe6b40c5c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:58:34.148 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:34.145+0000 7fe6b3e00640 1 --2- 172.21.15.32:0/1065458635 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fe6b40c61a0 0x7fe6b41f4d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:58:34.148 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:34.145+0000 7fe6b3400640 1 -- 172.21.15.32:0/1065458635 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fe6b40cf6a0 msgr2=0x7fe6b40c5c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:58:34.148 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:34.145+0000 7fe6b3400640 1 --2- 172.21.15.32:0/1065458635 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fe6b40cf6a0 0x7fe6b40c5c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:34.148 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:34.145+0000 7fe6b3400640 1 -- 172.21.15.32:0/1065458635 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fe6b40c61a0 msgr2=0x7fe6b41f4d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:58:34.148 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:34.145+0000 7fe6b3400640 1 --2- 172.21.15.32:0/1065458635 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fe6b40c61a0 0x7fe6b41f4d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:34.148 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:34.145+0000 7fe6b3400640 1 -- 172.21.15.32:0/1065458635 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe6b41f39a0 con 0x7fe6b40700f0 2024-08-03T20:58:34.149 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:34.145+0000 7fe6b3400640 1 --2- 172.21.15.32:0/1065458635 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe6b40700f0 0x7fe6b41f2e70 secure :-1 s=READY pgs=1806 cs=0 l=1 rev1=1 crypto rx=0x7fe6a400e7c0 tx=0x7fe6a400ec80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:58:34.149 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:34.146+0000 7fe6ab400640 1 -- 172.21.15.32:0/1065458635 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6a400ccc0 con 0x7fe6b40700f0 2024-08-03T20:58:34.149 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:34.146+0000 7fe6ab400640 1 -- 172.21.15.32:0/1065458635 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fe6a4010430 con 0x7fe6b40700f0 2024-08-03T20:58:34.149 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:34.146+0000 7fe6ab400640 1 -- 172.21.15.32:0/1065458635 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6a4010860 con 0x7fe6b40700f0 2024-08-03T20:58:34.149 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:34.146+0000 7fe6b9200640 1 -- 172.21.15.32:0/1065458635 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe6b41f3c80 con 0x7fe6b40700f0 2024-08-03T20:58:34.149 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:34.146+0000 7fe6b9200640 1 -- 172.21.15.32:0/1065458635 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe6b41f5df0 con 0x7fe6b40700f0 2024-08-03T20:58:34.150 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:34.147+0000 7fe6b9200640 1 -- 172.21.15.32:0/1065458635 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe6b41f6050 con 0x7fe6b40700f0 2024-08-03T20:58:34.150 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:34.148+0000 7fe6ab400640 1 -- 172.21.15.32:0/1065458635 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7fe6a4010a00 con 0x7fe6b40700f0 2024-08-03T20:58:34.151 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:34.149+0000 7fe6ab400640 1 --2- 172.21.15.32:0/1065458635 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fe694077010 0x7fe694079520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:58:34.151 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:34.149+0000 7fe6ab400640 1 -- 172.21.15.32:0/1065458635 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(115..115 src has 1..115) ==== 8434+0+0 (secure 0 0 0) 0x7fe6a4099ce0 con 0x7fe6b40700f0 2024-08-03T20:58:34.151 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:34.150+0000 7fe6b2a00640 1 --2- 172.21.15.32:0/1065458635 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fe694077010 0x7fe694079520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:58:34.152 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:34.150+0000 7fe6b2a00640 1 --2- 172.21.15.32:0/1065458635 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fe694077010 0x7fe694079520 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 crypto rx=0x7fe6b40b5a10 tx=0x7fe6a0009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:58:34.155 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:34.154+0000 7fe6ab400640 1 -- 172.21.15.32:0/1065458635 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7fe6a4061dc0 con 0x7fe6b40700f0 2024-08-03T20:58:34.403 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:34.402+0000 7fe6b9200640 1 -- 172.21.15.32:0/1065458635 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "joinable", "val": "true"} v 0) -- 0x7fe6b41f4360 con 0x7fe6b40700f0 2024-08-03T20:58:34.817 INFO:tasks.ceph.mds.a.smithi032.stderr: -43> 2024-08-03T20:58:30.062+0000 7f6434200640 -1 log_channel(cluster) log [ERR] : failed to read JournalPointer: -22 ((22) Invalid argument) 2024-08-03T20:58:34.929 INFO:tasks.ceph.mds.a.smithi032.stdout:starting mds.a at 2024-08-03T20:58:35.058 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:35.057+0000 7fe6ab400640 1 -- 172.21.15.32:0/1065458635 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "joinable", "val": "true"}]=0 cephfs marked joinable; MDS may join as newly active. v263) ==== 162+0+0 (secure 0 0 0) 0x7fe6a4066310 con 0x7fe6b40700f0 2024-08-03T20:58:35.059 INFO:teuthology.orchestra.run.smithi032.stderr:cephfs marked joinable; MDS may join as newly active. 2024-08-03T20:58:35.061 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:35.059+0000 7fe6b9200640 1 -- 172.21.15.32:0/1065458635 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fe694077010 msgr2=0x7fe694079520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:58:35.061 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:35.059+0000 7fe6b9200640 1 --2- 172.21.15.32:0/1065458635 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fe694077010 0x7fe694079520 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 crypto rx=0x7fe6b40b5a10 tx=0x7fe6a0009040 comp rx=0 tx=0).stop 2024-08-03T20:58:35.062 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:35.059+0000 7fe6b9200640 1 -- 172.21.15.32:0/1065458635 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe6b40700f0 msgr2=0x7fe6b41f2e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:58:35.062 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:35.059+0000 7fe6b9200640 1 --2- 172.21.15.32:0/1065458635 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe6b40700f0 0x7fe6b41f2e70 secure :-1 s=READY pgs=1806 cs=0 l=1 rev1=1 crypto rx=0x7fe6a400e7c0 tx=0x7fe6a400ec80 comp rx=0 tx=0).stop 2024-08-03T20:58:35.062 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:35.060+0000 7fe6b9200640 1 -- 172.21.15.32:0/1065458635 shutdown_connections 2024-08-03T20:58:35.062 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:35.060+0000 7fe6b9200640 1 --2- 172.21.15.32:0/1065458635 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fe694077010 0x7fe694079520 unknown :-1 s=CLOSED pgs=733 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:35.062 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:35.060+0000 7fe6b9200640 1 --2- 172.21.15.32:0/1065458635 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fe6b40c61a0 0x7fe6b41f4d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:35.062 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:35.060+0000 7fe6b9200640 1 --2- 172.21.15.32:0/1065458635 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fe6b40cf6a0 0x7fe6b40c5c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:35.062 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:35.060+0000 7fe6b9200640 1 --2- 172.21.15.32:0/1065458635 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe6b40700f0 0x7fe6b41f2e70 unknown :-1 s=CLOSED pgs=1806 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:35.062 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:35.060+0000 7fe6b9200640 1 -- 172.21.15.32:0/1065458635 >> 172.21.15.32:0/1065458635 conn(0x7fe6b40cfd80 msgr2=0x7fe6b40ca3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:58:35.062 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:35.060+0000 7fe6b9200640 1 -- 172.21.15.32:0/1065458635 shutdown_connections 2024-08-03T20:58:35.062 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:35.060+0000 7fe6b9200640 1 -- 172.21.15.32:0/1065458635 wait complete. 2024-08-03T20:58:35.072 INFO:tasks.ceph.mds.a.smithi032.stderr:2024-08-03T20:58:35.071+0000 7f047c000640 -1 mds.0.cache.ino(0x100) No data while reading inode 0x100 2024-08-03T20:58:35.073 INFO:tasks.ceph.mds.a.smithi032.stderr:2024-08-03T20:58:35.072+0000 7f047c000640 -1 log_channel(cluster) log [ERR] : Error loading MDS rank 0: (2) No such file or directory 2024-08-03T20:58:35.074 INFO:tasks.ceph.mon.a.smithi032.stderr:2024-08-03T20:58:35.072+0000 7f4e6d600640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-08-03T20:58:35.074 INFO:tasks.ceph.mon.a.smithi032.stderr:2024-08-03T20:58:35.072+0000 7f4e6d600640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 mds daemon damaged (MDS_DAMAGE) 2024-08-03T20:58:35.092 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-08-03T20:58:35.323 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:35.321+0000 7f00f2c00640 1 -- 172.21.15.32:0/4273053644 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f00ec0733f0 msgr2=0x7f00ec073800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:58:35.323 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:35.321+0000 7f00f2c00640 1 --2- 172.21.15.32:0/4273053644 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f00ec0733f0 0x7f00ec073800 secure :-1 s=READY pgs=1808 cs=0 l=1 rev1=1 crypto rx=0x7f00d4005fd0 tx=0x7f00d402a4b0 comp rx=0 tx=0).stop 2024-08-03T20:58:35.323 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:35.321+0000 7f00f2c00640 1 -- 172.21.15.32:0/4273053644 shutdown_connections 2024-08-03T20:58:35.324 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:35.321+0000 7f00f2c00640 1 --2- 172.21.15.32:0/4273053644 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f00ec0733f0 0x7f00ec073800 unknown :-1 s=CLOSED pgs=1808 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:35.324 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:35.321+0000 7f00f2c00640 1 --2- 172.21.15.32:0/4273053644 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f00ec0b4de0 0x7f00ec1515c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:35.324 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:35.321+0000 7f00f2c00640 1 -- 172.21.15.32:0/4273053644 >> 172.21.15.32:0/4273053644 conn(0x7f00ec0b1bd0 msgr2=0x7f00ec0b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:58:35.324 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:35.321+0000 7f00f2c00640 1 -- 172.21.15.32:0/4273053644 shutdown_connections 2024-08-03T20:58:35.324 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:35.322+0000 7f00f2c00640 1 -- 172.21.15.32:0/4273053644 wait complete. 2024-08-03T20:58:35.325 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:35.324+0000 7f00f2c00640 1 Processor -- start 2024-08-03T20:58:35.325 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:35.324+0000 7f00f2c00640 1 -- start start 2024-08-03T20:58:35.328 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:35.325+0000 7f00f2c00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f00ec0733f0 0x7f00ec14d740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:58:35.328 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:35.325+0000 7f00f2c00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f00ec0b4de0 0x7f00ec1f59d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:58:35.328 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:35.325+0000 7f00f2c00640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f00ec15e170 0x7f00ec1f5f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:58:35.328 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:35.325+0000 7f00f2c00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f00ec06e930 con 0x7f00ec0b4de0 2024-08-03T20:58:35.328 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:35.325+0000 7f00f2c00640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f00ec070ad0 con 0x7f00ec0733f0 2024-08-03T20:58:35.328 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:35.325+0000 7f00f2c00640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7f00ec06eab0 con 0x7f00ec15e170 2024-08-03T20:58:35.328 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:35.326+0000 7f00e3e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f00ec0b4de0 0x7f00ec1f59d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:58:35.328 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:35.326+0000 7f00ebe00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f00ec0733f0 0x7f00ec14d740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:58:35.328 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:35.326+0000 7f00e3e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f00ec0b4de0 0x7f00ec1f59d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:41662/0 (socket says 172.21.15.32:41662) 2024-08-03T20:58:35.328 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:35.326+0000 7f00e3e00640 1 -- 172.21.15.32:0/2936561935 learned_addr learned my addr 172.21.15.32:0/2936561935 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:58:35.328 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:35.326+0000 7f00f0c00640 1 --2- 172.21.15.32:0/2936561935 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f00ec15e170 0x7f00ec1f5f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:58:35.328 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:35.326+0000 7f00e3e00640 1 -- 172.21.15.32:0/2936561935 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f00ec15e170 msgr2=0x7f00ec1f5f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:58:35.329 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:35.326+0000 7f00e3e00640 1 --2- 172.21.15.32:0/2936561935 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f00ec15e170 0x7f00ec1f5f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:35.329 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:35.326+0000 7f00e3e00640 1 -- 172.21.15.32:0/2936561935 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f00ec0733f0 msgr2=0x7f00ec14d740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:58:35.329 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:35.326+0000 7f00e3e00640 1 --2- 172.21.15.32:0/2936561935 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f00ec0733f0 0x7f00ec14d740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:35.329 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:35.326+0000 7f00e3e00640 1 -- 172.21.15.32:0/2936561935 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f00ec1f4930 con 0x7f00ec0b4de0 2024-08-03T20:58:35.329 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:35.327+0000 7f00e3e00640 1 --2- 172.21.15.32:0/2936561935 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f00ec0b4de0 0x7f00ec1f59d0 secure :-1 s=READY pgs=1809 cs=0 l=1 rev1=1 crypto rx=0x7f00d4006010 tx=0x7f00d40039e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:58:35.330 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:35.327+0000 7f00e8c00640 1 -- 172.21.15.32:0/2936561935 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00d403e030 con 0x7f00ec0b4de0 2024-08-03T20:58:35.330 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:35.327+0000 7f00e8c00640 1 -- 172.21.15.32:0/2936561935 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f00d40047a0 con 0x7f00ec0b4de0 2024-08-03T20:58:35.330 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:35.327+0000 7f00e8c00640 1 -- 172.21.15.32:0/2936561935 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00d4035040 con 0x7f00ec0b4de0 2024-08-03T20:58:35.330 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:35.327+0000 7f00f2c00640 1 -- 172.21.15.32:0/2936561935 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f00ec1f4c10 con 0x7f00ec0b4de0 2024-08-03T20:58:35.330 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:35.327+0000 7f00f2c00640 1 -- 172.21.15.32:0/2936561935 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f00ec1f51d0 con 0x7f00ec0b4de0 2024-08-03T20:58:35.330 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:35.328+0000 7f00f2c00640 1 -- 172.21.15.32:0/2936561935 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f00ec073800 con 0x7f00ec0b4de0 2024-08-03T20:58:35.330 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:35.329+0000 7f00e8c00640 1 -- 172.21.15.32:0/2936561935 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f00d4041030 con 0x7f00ec0b4de0 2024-08-03T20:58:35.332 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:35.331+0000 7f00e8c00640 1 --2- 172.21.15.32:0/2936561935 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f00c8077010 0x7f00c8079520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:58:35.332 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:35.331+0000 7f00e8c00640 1 -- 172.21.15.32:0/2936561935 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(116..116 src has 1..116) ==== 8520+0+0 (secure 0 0 0) 0x7f00d40b9ca0 con 0x7f00ec0b4de0 2024-08-03T20:58:35.332 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:35.331+0000 7f00ebe00640 1 --2- 172.21.15.32:0/2936561935 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f00c8077010 0x7f00c8079520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:58:35.333 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:35.332+0000 7f00ebe00640 1 --2- 172.21.15.32:0/2936561935 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f00c8077010 0x7f00c8079520 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 crypto rx=0x7f00dc005fd0 tx=0x7f00dc00a250 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:58:35.337 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:35.336+0000 7f00e8c00640 1 -- 172.21.15.32:0/2936561935 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7f00ec0b51f0 con 0x7f00ec0b4de0 2024-08-03T20:58:35.586 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:35.585+0000 7f00f2c00640 1 -- 172.21.15.32:0/2936561935 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f00ec1f46c0 con 0x7f00ec0b4de0 2024-08-03T20:58:35.587 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:35.585+0000 7f00e8c00640 1 -- 172.21.15.32:0/2936561935 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 264 v264) ==== 95+0+4391 (secure 0 0 0) 0x7f00d4081ca0 con 0x7f00ec0b4de0 2024-08-03T20:58:35.587 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-03T20:58:35.587 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":264,"btime":"2024-08-03T20:58:35:074113+0000","default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6439,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6835/531080160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6834","nonce":531080160},{"type":"v1","addr":"172.21.15.118:6835","nonce":531080160}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":84},{"gid":6454,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6837/2796882319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6836","nonce":2796882319},{"type":"v1","addr":"172.21.15.118:6837","nonce":2796882319}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":86},{"gid":6813,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6834/2686933890","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":2686933890},{"type":"v1","addr":"172.21.15.32:6834","nonce":2686933890}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":85}],"filesystems":[{"mdsmap":{"epoch":264,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-08-03T20:54:26.950518+0000","modified":"2024-08-03T20:58:35.074047+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":116,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":6}]} 2024-08-03T20:58:35.587 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 264 2024-08-03T20:58:35.589 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:35.587+0000 7f00f2c00640 1 -- 172.21.15.32:0/2936561935 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f00c8077010 msgr2=0x7f00c8079520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:58:35.589 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:35.587+0000 7f00f2c00640 1 --2- 172.21.15.32:0/2936561935 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f00c8077010 0x7f00c8079520 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 crypto rx=0x7f00dc005fd0 tx=0x7f00dc00a250 comp rx=0 tx=0).stop 2024-08-03T20:58:35.589 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:35.587+0000 7f00f2c00640 1 -- 172.21.15.32:0/2936561935 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f00ec0b4de0 msgr2=0x7f00ec1f59d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:58:35.589 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:35.587+0000 7f00f2c00640 1 --2- 172.21.15.32:0/2936561935 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f00ec0b4de0 0x7f00ec1f59d0 secure :-1 s=READY pgs=1809 cs=0 l=1 rev1=1 crypto rx=0x7f00d4006010 tx=0x7f00d40039e0 comp rx=0 tx=0).stop 2024-08-03T20:58:35.589 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:35.588+0000 7f00f2c00640 1 -- 172.21.15.32:0/2936561935 shutdown_connections 2024-08-03T20:58:35.589 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:35.588+0000 7f00f2c00640 1 --2- 172.21.15.32:0/2936561935 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f00c8077010 0x7f00c8079520 unknown :-1 s=CLOSED pgs=735 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:35.589 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:35.588+0000 7f00f2c00640 1 --2- 172.21.15.32:0/2936561935 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f00ec15e170 0x7f00ec1f5f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:35.590 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:35.588+0000 7f00f2c00640 1 --2- 172.21.15.32:0/2936561935 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f00ec0b4de0 0x7f00ec1f59d0 unknown :-1 s=CLOSED pgs=1809 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:35.590 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:35.588+0000 7f00f2c00640 1 --2- 172.21.15.32:0/2936561935 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f00ec0733f0 0x7f00ec14d740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:35.590 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:35.588+0000 7f00f2c00640 1 -- 172.21.15.32:0/2936561935 >> 172.21.15.32:0/2936561935 conn(0x7f00ec0b1bd0 msgr2=0x7f00ec0d0820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:58:35.590 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:35.588+0000 7f00f2c00640 1 -- 172.21.15.32:0/2936561935 shutdown_connections 2024-08-03T20:58:35.590 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:35.588+0000 7f00f2c00640 1 -- 172.21.15.32:0/2936561935 wait complete. 2024-08-03T20:58:35.611 DEBUG:tasks.ceph_test_case:wait_until_true: success in 0s and 0 retries 2024-08-03T20:58:35.611 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-08-03T20:58:35.838 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:35.836+0000 7fda72c00640 1 -- 172.21.15.32:0/3750343440 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fda6c0af720 msgr2=0x7fda6c0cd330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:58:35.838 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:35.836+0000 7fda72c00640 1 --2- 172.21.15.32:0/3750343440 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fda6c0af720 0x7fda6c0cd330 secure :-1 s=READY pgs=1810 cs=0 l=1 rev1=1 crypto rx=0x7fda6c0b37d0 tx=0x7fda5c02a5f0 comp rx=0 tx=0).stop 2024-08-03T20:58:35.838 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:35.836+0000 7fda72c00640 1 -- 172.21.15.32:0/3750343440 shutdown_connections 2024-08-03T20:58:35.838 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:35.836+0000 7fda72c00640 1 --2- 172.21.15.32:0/3750343440 >> v2:172.21.15.118:3300/0 conn(0x7fda6c0727d0 0x7fda6c072be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:35.839 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:35.836+0000 7fda72c00640 1 --2- 172.21.15.32:0/3750343440 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fda6c0af720 0x7fda6c0cd330 unknown :-1 s=CLOSED pgs=1810 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:35.839 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:35.836+0000 7fda72c00640 1 -- 172.21.15.32:0/3750343440 >> 172.21.15.32:0/3750343440 conn(0x7fda6c0b1bd0 msgr2=0x7fda6c0b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:58:35.839 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:35.836+0000 7fda72c00640 1 -- 172.21.15.32:0/3750343440 shutdown_connections 2024-08-03T20:58:35.839 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:35.836+0000 7fda72c00640 1 -- 172.21.15.32:0/3750343440 wait complete. 2024-08-03T20:58:35.840 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:35.839+0000 7fda72c00640 1 Processor -- start 2024-08-03T20:58:35.840 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:35.839+0000 7fda72c00640 1 -- start start 2024-08-03T20:58:35.843 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:35.840+0000 7fda72c00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fda6c0727d0 0x7fda6c1f65d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:58:35.843 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:35.840+0000 7fda72c00640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fda6c0b4f20 0x7fda6c15d7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:58:35.843 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:35.840+0000 7fda72c00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fda6c0b09f0 0x7fda6c1f8d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:58:35.843 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:35.840+0000 7fda72c00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fda6c06ed80 con 0x7fda6c0727d0 2024-08-03T20:58:35.843 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:35.840+0000 7fda72c00640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7fda6c06ec00 con 0x7fda6c0b09f0 2024-08-03T20:58:35.843 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:35.840+0000 7fda72c00640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7fda6c06ef00 con 0x7fda6c0b4f20 2024-08-03T20:58:35.843 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:35.841+0000 7fda6be00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fda6c0727d0 0x7fda6c1f65d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:58:35.843 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:35.841+0000 7fda6be00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fda6c0727d0 0x7fda6c1f65d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:41670/0 (socket says 172.21.15.32:41670) 2024-08-03T20:58:35.843 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:35.841+0000 7fda6be00640 1 -- 172.21.15.32:0/645980210 learned_addr learned my addr 172.21.15.32:0/645980210 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:58:35.843 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:35.841+0000 7fda6be00640 1 -- 172.21.15.32:0/645980210 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fda6c0b4f20 msgr2=0x7fda6c15d7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:58:35.843 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:35.841+0000 7fda6be00640 1 --2- 172.21.15.32:0/645980210 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fda6c0b4f20 0x7fda6c15d7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:35.843 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:35.841+0000 7fda70c00640 1 --2- 172.21.15.32:0/645980210 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fda6c0b09f0 0x7fda6c1f8d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:58:35.843 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:35.841+0000 7fda6be00640 1 -- 172.21.15.32:0/645980210 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fda6c0b09f0 msgr2=0x7fda6c1f8d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:58:35.844 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:35.841+0000 7fda6be00640 1 --2- 172.21.15.32:0/645980210 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fda6c0b09f0 0x7fda6c1f8d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:35.844 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:35.841+0000 7fda6be00640 1 -- 172.21.15.32:0/645980210 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fda6c15dd00 con 0x7fda6c0727d0 2024-08-03T20:58:35.844 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:35.841+0000 7fda70c00640 1 --2- 172.21.15.32:0/645980210 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fda6c0b09f0 0x7fda6c1f8d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-03T20:58:35.844 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:35.841+0000 7fda6be00640 1 --2- 172.21.15.32:0/645980210 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fda6c0727d0 0x7fda6c1f65d0 secure :-1 s=READY pgs=1811 cs=0 l=1 rev1=1 crypto rx=0x7fda5c000c00 tx=0x7fda5c003820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:58:35.844 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:35.842+0000 7fda68c00640 1 -- 172.21.15.32:0/645980210 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda5c03e030 con 0x7fda6c0727d0 2024-08-03T20:58:35.845 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:35.842+0000 7fda68c00640 1 -- 172.21.15.32:0/645980210 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fda5c003d10 con 0x7fda6c0727d0 2024-08-03T20:58:35.845 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:35.842+0000 7fda68c00640 1 -- 172.21.15.32:0/645980210 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda5c035040 con 0x7fda6c0727d0 2024-08-03T20:58:35.845 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:35.842+0000 7fda72c00640 1 -- 172.21.15.32:0/645980210 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fda6c0aff10 con 0x7fda6c0727d0 2024-08-03T20:58:35.845 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:35.842+0000 7fda72c00640 1 -- 172.21.15.32:0/645980210 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fda6c0b04d0 con 0x7fda6c0727d0 2024-08-03T20:58:35.845 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:35.843+0000 7fda72c00640 1 -- 172.21.15.32:0/645980210 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fda6c0b4af0 con 0x7fda6c0727d0 2024-08-03T20:58:35.845 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:35.844+0000 7fda68c00640 1 -- 172.21.15.32:0/645980210 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7fda5c041030 con 0x7fda6c0727d0 2024-08-03T20:58:35.847 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:35.846+0000 7fda68c00640 1 --2- 172.21.15.32:0/645980210 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fda48077010 0x7fda48079520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:58:35.847 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:35.846+0000 7fda68c00640 1 -- 172.21.15.32:0/645980210 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(116..116 src has 1..116) ==== 8520+0+0 (secure 0 0 0) 0x7fda5c0b9ae0 con 0x7fda6c0727d0 2024-08-03T20:58:35.848 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:35.846+0000 7fda63e00640 1 --2- 172.21.15.32:0/645980210 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fda48077010 0x7fda48079520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:58:35.848 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:35.847+0000 7fda63e00640 1 --2- 172.21.15.32:0/645980210 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fda48077010 0x7fda48079520 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 crypto rx=0x7fda54000f60 tx=0x7fda540079b0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:58:35.852 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:35.850+0000 7fda68c00640 1 -- 172.21.15.32:0/645980210 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7fda5c082220 con 0x7fda6c0727d0 2024-08-03T20:58:36.093 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:36.092+0000 7fda72c00640 1 -- 172.21.15.32:0/645980210 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fda6c14ffb0 con 0x7fda6c0727d0 2024-08-03T20:58:36.094 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:36.092+0000 7fda68c00640 1 -- 172.21.15.32:0/645980210 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 264 v264) ==== 95+0+4391 (secure 0 0 0) 0x7fda5c081b20 con 0x7fda6c0727d0 2024-08-03T20:58:36.094 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-03T20:58:36.094 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":264,"btime":"2024-08-03T20:58:35:074113+0000","default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6439,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6835/531080160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6834","nonce":531080160},{"type":"v1","addr":"172.21.15.118:6835","nonce":531080160}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":84},{"gid":6454,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6837/2796882319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6836","nonce":2796882319},{"type":"v1","addr":"172.21.15.118:6837","nonce":2796882319}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":86},{"gid":6813,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6834/2686933890","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":2686933890},{"type":"v1","addr":"172.21.15.32:6834","nonce":2686933890}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":85}],"filesystems":[{"mdsmap":{"epoch":264,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-08-03T20:54:26.950518+0000","modified":"2024-08-03T20:58:35.074047+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":116,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":6}]} 2024-08-03T20:58:36.095 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 264 2024-08-03T20:58:36.096 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:36.095+0000 7fda72c00640 1 -- 172.21.15.32:0/645980210 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fda48077010 msgr2=0x7fda48079520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:58:36.096 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:36.095+0000 7fda72c00640 1 --2- 172.21.15.32:0/645980210 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fda48077010 0x7fda48079520 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 crypto rx=0x7fda54000f60 tx=0x7fda540079b0 comp rx=0 tx=0).stop 2024-08-03T20:58:36.097 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:36.095+0000 7fda72c00640 1 -- 172.21.15.32:0/645980210 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fda6c0727d0 msgr2=0x7fda6c1f65d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:58:36.097 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:36.095+0000 7fda72c00640 1 --2- 172.21.15.32:0/645980210 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fda6c0727d0 0x7fda6c1f65d0 secure :-1 s=READY pgs=1811 cs=0 l=1 rev1=1 crypto rx=0x7fda5c000c00 tx=0x7fda5c003820 comp rx=0 tx=0).stop 2024-08-03T20:58:36.097 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:36.095+0000 7fda72c00640 1 -- 172.21.15.32:0/645980210 shutdown_connections 2024-08-03T20:58:36.097 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:36.095+0000 7fda72c00640 1 --2- 172.21.15.32:0/645980210 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fda48077010 0x7fda48079520 unknown :-1 s=CLOSED pgs=736 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:36.097 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:36.095+0000 7fda72c00640 1 --2- 172.21.15.32:0/645980210 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fda6c0b09f0 0x7fda6c1f8d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:36.097 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:36.095+0000 7fda72c00640 1 --2- 172.21.15.32:0/645980210 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fda6c0b4f20 0x7fda6c15d7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:36.097 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:36.095+0000 7fda72c00640 1 --2- 172.21.15.32:0/645980210 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fda6c0727d0 0x7fda6c1f65d0 unknown :-1 s=CLOSED pgs=1811 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:36.097 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:36.095+0000 7fda72c00640 1 -- 172.21.15.32:0/645980210 >> 172.21.15.32:0/645980210 conn(0x7fda6c0b1bd0 msgr2=0x7fda6c1504d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:58:36.097 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:36.095+0000 7fda72c00640 1 -- 172.21.15.32:0/645980210 shutdown_connections 2024-08-03T20:58:36.097 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:36.095+0000 7fda72c00640 1 -- 172.21.15.32:0/645980210 wait complete. 2024-08-03T20:58:36.124 DEBUG:tasks.ceph_test_case:wait_until_true: success in 0s and 0 retries 2024-08-03T20:58:36.124 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-08-03T20:58:36.353 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:36.351+0000 7fce7fe00640 1 -- 172.21.15.32:0/731099715 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fce78073e60 msgr2=0x7fce78074270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:58:36.353 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:36.351+0000 7fce7fe00640 1 --2- 172.21.15.32:0/731099715 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fce78073e60 0x7fce78074270 secure :-1 s=READY pgs=1812 cs=0 l=1 rev1=1 crypto rx=0x7fce78071700 tx=0x7fce6402a560 comp rx=0 tx=0).stop 2024-08-03T20:58:36.353 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:36.351+0000 7fce7fe00640 1 -- 172.21.15.32:0/731099715 shutdown_connections 2024-08-03T20:58:36.353 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:36.351+0000 7fce7fe00640 1 --2- 172.21.15.32:0/731099715 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fce78073e60 0x7fce78074270 unknown :-1 s=CLOSED pgs=1812 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:36.353 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:36.351+0000 7fce7fe00640 1 --2- 172.21.15.32:0/731099715 >> v2:172.21.15.118:3300/0 conn(0x7fce7806e930 0x7fce78144f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:36.353 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:36.351+0000 7fce7fe00640 1 -- 172.21.15.32:0/731099715 >> 172.21.15.32:0/731099715 conn(0x7fce780b1bd0 msgr2=0x7fce780b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:58:36.353 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:36.351+0000 7fce7fe00640 1 -- 172.21.15.32:0/731099715 shutdown_connections 2024-08-03T20:58:36.353 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:36.351+0000 7fce7fe00640 1 -- 172.21.15.32:0/731099715 wait complete. 2024-08-03T20:58:36.354 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:36.353+0000 7fce7fe00640 1 Processor -- start 2024-08-03T20:58:36.355 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:36.353+0000 7fce7fe00640 1 -- start start 2024-08-03T20:58:36.357 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:36.355+0000 7fce7fe00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fce7806e930 0x7fce781f3400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:58:36.357 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:36.355+0000 7fce7fe00640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fce78073e60 0x7fce780af380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:58:36.357 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:36.355+0000 7fce7fe00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fce780747b0 0x7fce781436b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:58:36.357 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:36.355+0000 7fce7fe00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fce780cf9c0 con 0x7fce780747b0 2024-08-03T20:58:36.357 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:36.355+0000 7fce7fe00640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7fce780cf840 con 0x7fce7806e930 2024-08-03T20:58:36.357 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:36.355+0000 7fce7fe00640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7fce780ccff0 con 0x7fce78073e60 2024-08-03T20:58:36.357 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:36.355+0000 7fce7de00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fce780747b0 0x7fce781436b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:58:36.357 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:36.355+0000 7fce7de00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fce780747b0 0x7fce781436b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:41684/0 (socket says 172.21.15.32:41684) 2024-08-03T20:58:36.358 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:36.355+0000 7fce7de00640 1 -- 172.21.15.32:0/4081522493 learned_addr learned my addr 172.21.15.32:0/4081522493 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:58:36.358 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:36.355+0000 7fce7d400640 1 --2- 172.21.15.32:0/4081522493 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fce7806e930 0x7fce781f3400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:58:36.358 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:36.355+0000 7fce7ca00640 1 --2- 172.21.15.32:0/4081522493 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fce78073e60 0x7fce780af380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:58:36.358 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:36.355+0000 7fce7de00640 1 -- 172.21.15.32:0/4081522493 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fce78073e60 msgr2=0x7fce780af380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:58:36.358 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:36.355+0000 7fce7de00640 1 --2- 172.21.15.32:0/4081522493 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fce78073e60 0x7fce780af380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:36.358 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:36.355+0000 7fce7de00640 1 -- 172.21.15.32:0/4081522493 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fce7806e930 msgr2=0x7fce781f3400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:58:36.358 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:36.355+0000 7fce7de00640 1 --2- 172.21.15.32:0/4081522493 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fce7806e930 0x7fce781f3400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:36.358 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:36.355+0000 7fce7de00640 1 -- 172.21.15.32:0/4081522493 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fce781f3c50 con 0x7fce780747b0 2024-08-03T20:58:36.358 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:36.356+0000 7fce7de00640 1 --2- 172.21.15.32:0/4081522493 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fce780747b0 0x7fce781436b0 secure :-1 s=READY pgs=1813 cs=0 l=1 rev1=1 crypto rx=0x7fce7000c700 tx=0x7fce7000cbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:58:36.359 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:36.356+0000 7fce75600640 1 -- 172.21.15.32:0/4081522493 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce70003b70 con 0x7fce780747b0 2024-08-03T20:58:36.359 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:36.356+0000 7fce75600640 1 -- 172.21.15.32:0/4081522493 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fce70003d10 con 0x7fce780747b0 2024-08-03T20:58:36.359 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:36.356+0000 7fce75600640 1 -- 172.21.15.32:0/4081522493 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce7000aab0 con 0x7fce780747b0 2024-08-03T20:58:36.359 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:36.356+0000 7fce7fe00640 1 -- 172.21.15.32:0/4081522493 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fce781f3f30 con 0x7fce780747b0 2024-08-03T20:58:36.359 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:36.356+0000 7fce7fe00640 1 -- 172.21.15.32:0/4081522493 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fce781f44f0 con 0x7fce780747b0 2024-08-03T20:58:36.359 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:36.357+0000 7fce7fe00640 1 -- 172.21.15.32:0/4081522493 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fce781f39c0 con 0x7fce780747b0 2024-08-03T20:58:36.360 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:36.359+0000 7fce75600640 1 -- 172.21.15.32:0/4081522493 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7fce7000ac50 con 0x7fce780747b0 2024-08-03T20:58:36.361 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:36.359+0000 7fce75600640 1 --2- 172.21.15.32:0/4081522493 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fce58077040 0x7fce58079550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:58:36.361 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:36.360+0000 7fce75600640 1 -- 172.21.15.32:0/4081522493 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(116..116 src has 1..116) ==== 8520+0+0 (secure 0 0 0) 0x7fce700990d0 con 0x7fce780747b0 2024-08-03T20:58:36.361 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:36.360+0000 7fce7d400640 1 --2- 172.21.15.32:0/4081522493 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fce58077040 0x7fce58079550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:58:36.362 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:36.361+0000 7fce7d400640 1 --2- 172.21.15.32:0/4081522493 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fce58077040 0x7fce58079550 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x7fce6c009b30 tx=0x7fce6c005f90 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:58:36.365 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:36.364+0000 7fce75600640 1 -- 172.21.15.32:0/4081522493 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7fce700611c0 con 0x7fce780747b0 2024-08-03T20:58:36.606 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:36.605+0000 7fce7fe00640 1 -- 172.21.15.32:0/4081522493 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fce780ccde0 con 0x7fce780747b0 2024-08-03T20:58:36.607 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:36.605+0000 7fce75600640 1 -- 172.21.15.32:0/4081522493 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 264 v264) ==== 95+0+4391 (secure 0 0 0) 0x7fce780ccde0 con 0x7fce780747b0 2024-08-03T20:58:36.607 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-03T20:58:36.607 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":264,"btime":"2024-08-03T20:58:35:074113+0000","default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6439,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6835/531080160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6834","nonce":531080160},{"type":"v1","addr":"172.21.15.118:6835","nonce":531080160}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":84},{"gid":6454,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6837/2796882319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6836","nonce":2796882319},{"type":"v1","addr":"172.21.15.118:6837","nonce":2796882319}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":86},{"gid":6813,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6834/2686933890","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":2686933890},{"type":"v1","addr":"172.21.15.32:6834","nonce":2686933890}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":85}],"filesystems":[{"mdsmap":{"epoch":264,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-08-03T20:54:26.950518+0000","modified":"2024-08-03T20:58:35.074047+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":116,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":6}]} 2024-08-03T20:58:36.607 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 264 2024-08-03T20:58:36.609 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:36.608+0000 7fce7fe00640 1 -- 172.21.15.32:0/4081522493 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fce58077040 msgr2=0x7fce58079550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:58:36.610 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:36.608+0000 7fce7fe00640 1 --2- 172.21.15.32:0/4081522493 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fce58077040 0x7fce58079550 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x7fce6c009b30 tx=0x7fce6c005f90 comp rx=0 tx=0).stop 2024-08-03T20:58:36.610 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:36.608+0000 7fce7fe00640 1 -- 172.21.15.32:0/4081522493 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fce780747b0 msgr2=0x7fce781436b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:58:36.610 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:36.608+0000 7fce7fe00640 1 --2- 172.21.15.32:0/4081522493 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fce780747b0 0x7fce781436b0 secure :-1 s=READY pgs=1813 cs=0 l=1 rev1=1 crypto rx=0x7fce7000c700 tx=0x7fce7000cbc0 comp rx=0 tx=0).stop 2024-08-03T20:58:36.610 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:36.608+0000 7fce7fe00640 1 -- 172.21.15.32:0/4081522493 shutdown_connections 2024-08-03T20:58:36.610 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:36.608+0000 7fce7fe00640 1 --2- 172.21.15.32:0/4081522493 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fce58077040 0x7fce58079550 unknown :-1 s=CLOSED pgs=737 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:36.610 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:36.608+0000 7fce7fe00640 1 --2- 172.21.15.32:0/4081522493 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fce780747b0 0x7fce781436b0 unknown :-1 s=CLOSED pgs=1813 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:36.610 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:36.608+0000 7fce7fe00640 1 --2- 172.21.15.32:0/4081522493 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fce78073e60 0x7fce780af380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:36.610 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:36.608+0000 7fce7fe00640 1 --2- 172.21.15.32:0/4081522493 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fce7806e930 0x7fce781f3400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:36.610 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:36.608+0000 7fce7fe00640 1 -- 172.21.15.32:0/4081522493 >> 172.21.15.32:0/4081522493 conn(0x7fce780b1bd0 msgr2=0x7fce7806ef70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:58:36.610 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:36.608+0000 7fce7fe00640 1 -- 172.21.15.32:0/4081522493 shutdown_connections 2024-08-03T20:58:36.610 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:36.608+0000 7fce7fe00640 1 -- 172.21.15.32:0/4081522493 wait complete. 2024-08-03T20:58:36.636 INFO:tasks.cephfs.test_damage:Result: Mutation 'Truncate 100.00000000.inode' led to DAMAGED state 2024-08-03T20:58:36.636 INFO:tasks.cephfs.test_damage:Applying mutation 'Corrupt omap key 10000000000.00000000:sixmegs_head' 2024-08-03T20:58:36.636 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-08-03T20:58:36.636 DEBUG:teuthology.orchestra.run.smithi032:> dd if=/proc/self/mounts of=/dev/stdout 2024-08-03T20:58:36.671 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053453,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6499208k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 none /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249604k,nr_inodes=812401,mode=700,uid=1000,gid=1316 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6499208k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-08-03T20:58:36.672 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi032.front.sepia.ceph.com 2024-08-03T20:58:36.672 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-08-03T20:58:36.672 DEBUG:teuthology.orchestra.run.smithi032:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-08-03T20:58:36.747 INFO:teuthology.orchestra.run.smithi032.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.0': No such file or directory 2024-08-03T20:58:36.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-03T20:58:36.748 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-08-03T20:58:37.017 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:37.015+0000 7f2dada00640 1 -- 172.21.15.32:0/1405479317 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2da815f9d0 msgr2=0x7f2da806f660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:58:37.017 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:37.015+0000 7f2dada00640 1 --2- 172.21.15.32:0/1405479317 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2da815f9d0 0x7f2da806f660 secure :-1 s=READY pgs=1815 cs=0 l=1 rev1=1 crypto rx=0x7f2da0005fd0 tx=0x7f2da002a820 comp rx=0 tx=0).stop 2024-08-03T20:58:37.017 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:37.015+0000 7f2dada00640 1 -- 172.21.15.32:0/1405479317 shutdown_connections 2024-08-03T20:58:37.017 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:37.015+0000 7f2dada00640 1 --2- 172.21.15.32:0/1405479317 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2da815f9d0 0x7f2da806f660 unknown :-1 s=CLOSED pgs=1815 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:37.017 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:37.015+0000 7f2dada00640 1 --2- 172.21.15.32:0/1405479317 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f2da815b790 0x7f2da815f490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:37.017 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:37.015+0000 7f2dada00640 1 -- 172.21.15.32:0/1405479317 >> 172.21.15.32:0/1405479317 conn(0x7f2da80d0dd0 msgr2=0x7f2da80d11e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:58:37.017 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:37.015+0000 7f2dada00640 1 -- 172.21.15.32:0/1405479317 shutdown_connections 2024-08-03T20:58:37.017 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:37.015+0000 7f2dada00640 1 -- 172.21.15.32:0/1405479317 wait complete. 2024-08-03T20:58:37.019 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:37.017+0000 7f2dada00640 1 Processor -- start 2024-08-03T20:58:37.019 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:37.018+0000 7f2dada00640 1 -- start start 2024-08-03T20:58:37.021 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:37.019+0000 7f2dada00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2da8071150 0x7f2da81f6520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:58:37.021 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:37.019+0000 7f2dada00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f2da815b790 0x7f2da8151230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:58:37.021 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:37.019+0000 7f2dada00640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f2da815f9d0 0x7f2da81583c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:58:37.022 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:37.019+0000 7f2dada00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f2da806ff30 con 0x7f2da8071150 2024-08-03T20:58:37.022 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:37.019+0000 7f2dada00640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f2da806fdb0 con 0x7f2da815b790 2024-08-03T20:58:37.022 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:37.019+0000 7f2dada00640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7f2da80736b0 con 0x7f2da815f9d0 2024-08-03T20:58:37.022 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:37.019+0000 7f2da7e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2da8071150 0x7f2da81f6520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:58:37.022 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:37.019+0000 7f2da7e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2da8071150 0x7f2da81f6520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:41708/0 (socket says 172.21.15.32:41708) 2024-08-03T20:58:37.022 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:37.019+0000 7f2da7e00640 1 -- 172.21.15.32:0/3493114748 learned_addr learned my addr 172.21.15.32:0/3493114748 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:58:37.022 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:37.019+0000 7f2dad000640 1 --2- 172.21.15.32:0/3493114748 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f2da815f9d0 0x7f2da81583c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:58:37.022 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:37.020+0000 7f2d9f400640 1 --2- 172.21.15.32:0/3493114748 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f2da815b790 0x7f2da8151230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:58:37.022 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:37.020+0000 7f2da7e00640 1 -- 172.21.15.32:0/3493114748 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f2da815f9d0 msgr2=0x7f2da81583c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:58:37.022 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:37.020+0000 7f2da7e00640 1 --2- 172.21.15.32:0/3493114748 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f2da815f9d0 0x7f2da81583c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:37.022 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:37.020+0000 7f2da7e00640 1 -- 172.21.15.32:0/3493114748 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f2da815b790 msgr2=0x7f2da8151230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:58:37.023 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:37.020+0000 7f2da7e00640 1 --2- 172.21.15.32:0/3493114748 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f2da815b790 0x7f2da8151230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:37.023 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:37.020+0000 7f2da7e00640 1 -- 172.21.15.32:0/3493114748 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2da8155ea0 con 0x7f2da8071150 2024-08-03T20:58:37.023 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:37.020+0000 7f2da7e00640 1 --2- 172.21.15.32:0/3493114748 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2da8071150 0x7f2da81f6520 secure :-1 s=READY pgs=1816 cs=0 l=1 rev1=1 crypto rx=0x7f2d9800c430 tx=0x7f2d9800c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:58:37.024 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:37.020+0000 7f2da4c00640 1 -- 172.21.15.32:0/3493114748 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d98011790 con 0x7f2da8071150 2024-08-03T20:58:37.024 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:37.020+0000 7f2da4c00640 1 -- 172.21.15.32:0/3493114748 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f2d98017070 con 0x7f2da8071150 2024-08-03T20:58:37.024 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:37.020+0000 7f2dada00640 1 -- 172.21.15.32:0/3493114748 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2da8156180 con 0x7f2da8071150 2024-08-03T20:58:37.024 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:37.020+0000 7f2da4c00640 1 -- 172.21.15.32:0/3493114748 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d98011ce0 con 0x7f2da8071150 2024-08-03T20:58:37.024 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:37.021+0000 7f2dada00640 1 -- 172.21.15.32:0/3493114748 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2da80c9d90 con 0x7f2da8071150 2024-08-03T20:58:37.024 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:37.022+0000 7f2dada00640 1 -- 172.21.15.32:0/3493114748 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2da81563c0 con 0x7f2da8071150 2024-08-03T20:58:37.024 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:37.023+0000 7f2da4c00640 1 -- 172.21.15.32:0/3493114748 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f2d9801c460 con 0x7f2da8071150 2024-08-03T20:58:37.025 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:37.024+0000 7f2da4c00640 1 --2- 172.21.15.32:0/3493114748 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f2d84077010 0x7f2d84079520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:58:37.025 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:37.024+0000 7f2da4c00640 1 -- 172.21.15.32:0/3493114748 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(116..116 src has 1..116) ==== 8520+0+0 (secure 0 0 0) 0x7f2d98098ff0 con 0x7f2da8071150 2024-08-03T20:58:37.026 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:37.024+0000 7f2d9f400640 1 --2- 172.21.15.32:0/3493114748 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f2d84077010 0x7f2d84079520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:58:37.026 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:37.025+0000 7f2d9f400640 1 --2- 172.21.15.32:0/3493114748 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f2d84077010 0x7f2d84079520 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 crypto rx=0x7f2da8071070 tx=0x7f2d90005f90 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:58:37.030 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:37.028+0000 7f2da4c00640 1 -- 172.21.15.32:0/3493114748 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7f2da81563c0 con 0x7f2da8071150 2024-08-03T20:58:37.273 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:37.271+0000 7f2dada00640 1 -- 172.21.15.32:0/3493114748 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7f2da80c9ff0 con 0x7f2da8071150 2024-08-03T20:58:37.354 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:37.352+0000 7f2da4c00640 1 -- 172.21.15.32:0/3493114748 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v265) ==== 161+0+0 (secure 0 0 0) 0x7f2d98061170 con 0x7f2da8071150 2024-08-03T20:58:37.355 INFO:teuthology.orchestra.run.smithi032.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-08-03T20:58:37.358 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:37.356+0000 7f2dada00640 1 -- 172.21.15.32:0/3493114748 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f2d84077010 msgr2=0x7f2d84079520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:58:37.359 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:37.356+0000 7f2dada00640 1 --2- 172.21.15.32:0/3493114748 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f2d84077010 0x7f2d84079520 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 crypto rx=0x7f2da8071070 tx=0x7f2d90005f90 comp rx=0 tx=0).stop 2024-08-03T20:58:37.359 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:37.356+0000 7f2dada00640 1 -- 172.21.15.32:0/3493114748 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2da8071150 msgr2=0x7f2da81f6520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:58:37.359 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:37.356+0000 7f2dada00640 1 --2- 172.21.15.32:0/3493114748 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2da8071150 0x7f2da81f6520 secure :-1 s=READY pgs=1816 cs=0 l=1 rev1=1 crypto rx=0x7f2d9800c430 tx=0x7f2d9800c8f0 comp rx=0 tx=0).stop 2024-08-03T20:58:37.359 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:37.356+0000 7f2dada00640 1 -- 172.21.15.32:0/3493114748 shutdown_connections 2024-08-03T20:58:37.359 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:37.356+0000 7f2dada00640 1 --2- 172.21.15.32:0/3493114748 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f2d84077010 0x7f2d84079520 unknown :-1 s=CLOSED pgs=738 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:37.359 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:37.356+0000 7f2dada00640 1 --2- 172.21.15.32:0/3493114748 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f2da815b790 0x7f2da8151230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:37.359 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:37.356+0000 7f2dada00640 1 --2- 172.21.15.32:0/3493114748 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f2da815f9d0 0x7f2da81583c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:37.359 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:37.356+0000 7f2dada00640 1 --2- 172.21.15.32:0/3493114748 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2da8071150 0x7f2da81f6520 unknown :-1 s=CLOSED pgs=1816 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:37.359 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:37.356+0000 7f2dada00640 1 -- 172.21.15.32:0/3493114748 >> 172.21.15.32:0/3493114748 conn(0x7f2da80d0dd0 msgr2=0x7f2da80cc060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:58:37.359 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:37.356+0000 7f2dada00640 1 -- 172.21.15.32:0/3493114748 shutdown_connections 2024-08-03T20:58:37.359 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:37.356+0000 7f2dada00640 1 -- 172.21.15.32:0/3493114748 wait complete. 2024-08-03T20:58:37.381 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mds repaired 0 2024-08-03T20:58:37.610 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:37.608+0000 7f5421000640 1 -- 172.21.15.32:0/1926585083 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f541c06fb50 msgr2=0x7f541c0ad190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:58:37.610 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:37.608+0000 7f5421000640 1 --2- 172.21.15.32:0/1926585083 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f541c06fb50 0x7f541c0ad190 secure :-1 s=READY pgs=1818 cs=0 l=1 rev1=1 crypto rx=0x7f5414005fd0 tx=0x7f541402a810 comp rx=0 tx=0).stop 2024-08-03T20:58:37.610 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:37.608+0000 7f5421000640 1 -- 172.21.15.32:0/1926585083 shutdown_connections 2024-08-03T20:58:37.610 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:37.608+0000 7f5421000640 1 --2- 172.21.15.32:0/1926585083 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f541c06fb50 0x7f541c0ad190 unknown :-1 s=CLOSED pgs=1818 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:37.610 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:37.608+0000 7f5421000640 1 --2- 172.21.15.32:0/1926585083 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f541c0af020 0x7f541c0742b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:37.610 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:37.608+0000 7f5421000640 1 -- 172.21.15.32:0/1926585083 >> 172.21.15.32:0/1926585083 conn(0x7f541c0b2120 msgr2=0x7f541c0cfd60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:58:37.610 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:37.608+0000 7f5421000640 1 -- 172.21.15.32:0/1926585083 shutdown_connections 2024-08-03T20:58:37.610 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:37.608+0000 7f5421000640 1 -- 172.21.15.32:0/1926585083 wait complete. 2024-08-03T20:58:37.612 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:37.610+0000 7f5421000640 1 Processor -- start 2024-08-03T20:58:37.612 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:37.611+0000 7f5421000640 1 -- start start 2024-08-03T20:58:37.615 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:37.612+0000 7f5421000640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f541c06fb50 0x7f541c1f3490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:58:37.615 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:37.612+0000 7f5421000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f541c0acd80 0x7f541c14cfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:58:37.615 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:37.612+0000 7f5421000640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f541c0af020 0x7f541c0c4be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:58:37.615 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:37.612+0000 7f5421000640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f541c1602c0 con 0x7f541c0acd80 2024-08-03T20:58:37.615 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:37.612+0000 7f5421000640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f541c160140 con 0x7f541c0af020 2024-08-03T20:58:37.615 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:37.612+0000 7f5421000640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7f541c160440 con 0x7f541c06fb50 2024-08-03T20:58:37.615 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:37.612+0000 7f5413e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f541c0acd80 0x7f541c14cfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:58:37.615 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:37.612+0000 7f5413e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f541c0acd80 0x7f541c14cfa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:41726/0 (socket says 172.21.15.32:41726) 2024-08-03T20:58:37.616 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:37.612+0000 7f5413e00640 1 -- 172.21.15.32:0/90795609 learned_addr learned my addr 172.21.15.32:0/90795609 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:58:37.616 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:37.612+0000 7f541b400640 1 --2- 172.21.15.32:0/90795609 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f541c06fb50 0x7f541c1f3490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:58:37.616 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:37.613+0000 7f541be00640 1 --2- 172.21.15.32:0/90795609 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f541c0af020 0x7f541c0c4be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:58:37.616 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:37.613+0000 7f5413e00640 1 -- 172.21.15.32:0/90795609 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f541c06fb50 msgr2=0x7f541c1f3490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:58:37.616 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:37.613+0000 7f5413e00640 1 --2- 172.21.15.32:0/90795609 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f541c06fb50 0x7f541c1f3490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:37.616 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:37.613+0000 7f5413e00640 1 -- 172.21.15.32:0/90795609 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f541c0af020 msgr2=0x7f541c0c4be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:58:37.616 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:37.613+0000 7f5413e00640 1 --2- 172.21.15.32:0/90795609 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f541c0af020 0x7f541c0c4be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:37.616 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:37.613+0000 7f5413e00640 1 -- 172.21.15.32:0/90795609 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f541c0c3b40 con 0x7f541c0acd80 2024-08-03T20:58:37.616 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:37.613+0000 7f541b400640 1 --2- 172.21.15.32:0/90795609 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f541c06fb50 0x7f541c1f3490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-03T20:58:37.616 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:37.613+0000 7f5413e00640 1 --2- 172.21.15.32:0/90795609 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f541c0acd80 0x7f541c14cfa0 secure :-1 s=READY pgs=1819 cs=0 l=1 rev1=1 crypto rx=0x7f540400c340 tx=0x7f540400c800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:58:37.616 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:37.613+0000 7f5413400640 1 -- 172.21.15.32:0/90795609 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5404011790 con 0x7f541c0acd80 2024-08-03T20:58:37.616 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:37.613+0000 7f5413400640 1 -- 172.21.15.32:0/90795609 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f5404017070 con 0x7f541c0acd80 2024-08-03T20:58:37.616 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:37.614+0000 7f5413400640 1 -- 172.21.15.32:0/90795609 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5404011cc0 con 0x7f541c0acd80 2024-08-03T20:58:37.616 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:37.614+0000 7f5421000640 1 -- 172.21.15.32:0/90795609 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f541c0c3e20 con 0x7f541c0acd80 2024-08-03T20:58:37.617 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:37.614+0000 7f5421000640 1 -- 172.21.15.32:0/90795609 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f541c0c43e0 con 0x7f541c0acd80 2024-08-03T20:58:37.617 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:37.615+0000 7f5421000640 1 -- 172.21.15.32:0/90795609 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f541c0ad190 con 0x7f541c0acd80 2024-08-03T20:58:37.617 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:37.616+0000 7f5413400640 1 -- 172.21.15.32:0/90795609 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f540401c460 con 0x7f541c0acd80 2024-08-03T20:58:37.619 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:37.617+0000 7f5413400640 1 --2- 172.21.15.32:0/90795609 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f53f8077010 0x7f53f8079520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:58:37.619 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:37.618+0000 7f5413400640 1 -- 172.21.15.32:0/90795609 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(116..116 src has 1..116) ==== 8520+0+0 (secure 0 0 0) 0x7f5404099140 con 0x7f541c0acd80 2024-08-03T20:58:37.619 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:37.618+0000 7f541b400640 1 --2- 172.21.15.32:0/90795609 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f53f8077010 0x7f53f8079520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:58:37.620 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:37.619+0000 7f541b400640 1 --2- 172.21.15.32:0/90795609 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f53f8077010 0x7f53f8079520 secure :-1 s=READY pgs=739 cs=0 l=1 rev1=1 crypto rx=0x7f540c005fd0 tx=0x7f540c00f360 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:58:37.624 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:37.622+0000 7f5413400640 1 -- 172.21.15.32:0/90795609 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7f5404061820 con 0x7f541c0acd80 2024-08-03T20:58:37.909 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:37.908+0000 7f5421000640 1 -- 172.21.15.32:0/90795609 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "mds repaired", "role": "0"} v 0) -- 0x7f541c0d0930 con 0x7f541c0acd80 2024-08-03T20:58:38.356 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:38.354+0000 7f5413400640 1 -- 172.21.15.32:0/90795609 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "mds repaired", "role": "0"}]=0 repaired: restoring rank 6:0 v266) ==== 101+0+0 (secure 0 0 0) 0x7f5404061120 con 0x7f541c0acd80 2024-08-03T20:58:38.356 INFO:teuthology.orchestra.run.smithi032.stderr:repaired: restoring rank 6:0 2024-08-03T20:58:38.359 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:38.357+0000 7f5421000640 1 -- 172.21.15.32:0/90795609 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f53f8077010 msgr2=0x7f53f8079520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:58:38.359 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:38.357+0000 7f5421000640 1 --2- 172.21.15.32:0/90795609 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f53f8077010 0x7f53f8079520 secure :-1 s=READY pgs=739 cs=0 l=1 rev1=1 crypto rx=0x7f540c005fd0 tx=0x7f540c00f360 comp rx=0 tx=0).stop 2024-08-03T20:58:38.359 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:38.357+0000 7f5421000640 1 -- 172.21.15.32:0/90795609 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f541c0acd80 msgr2=0x7f541c14cfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:58:38.359 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:38.357+0000 7f5421000640 1 --2- 172.21.15.32:0/90795609 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f541c0acd80 0x7f541c14cfa0 secure :-1 s=READY pgs=1819 cs=0 l=1 rev1=1 crypto rx=0x7f540400c340 tx=0x7f540400c800 comp rx=0 tx=0).stop 2024-08-03T20:58:38.359 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:38.357+0000 7f5421000640 1 -- 172.21.15.32:0/90795609 shutdown_connections 2024-08-03T20:58:38.359 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:38.357+0000 7f5421000640 1 --2- 172.21.15.32:0/90795609 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f53f8077010 0x7f53f8079520 unknown :-1 s=CLOSED pgs=739 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:38.359 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:38.357+0000 7f5421000640 1 --2- 172.21.15.32:0/90795609 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f541c0af020 0x7f541c0c4be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:38.360 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:38.357+0000 7f5421000640 1 --2- 172.21.15.32:0/90795609 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f541c0acd80 0x7f541c14cfa0 unknown :-1 s=CLOSED pgs=1819 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:38.360 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:38.357+0000 7f5421000640 1 --2- 172.21.15.32:0/90795609 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f541c06fb50 0x7f541c1f3490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:38.360 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:38.357+0000 7f5421000640 1 -- 172.21.15.32:0/90795609 >> 172.21.15.32:0/90795609 conn(0x7f541c0b2120 msgr2=0x7f541c0cbca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:58:38.360 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:38.357+0000 7f5421000640 1 -- 172.21.15.32:0/90795609 shutdown_connections 2024-08-03T20:58:38.360 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:38.357+0000 7f5421000640 1 -- 172.21.15.32:0/90795609 wait complete. 2024-08-03T20:58:38.383 DEBUG:teuthology.orchestra.run.smithi032:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata import - 2024-08-03T20:58:38.527 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:38.525+0000 7fa6cd571f40 1 Processor -- start 2024-08-03T20:58:38.527 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:38.525+0000 7fa6cd571f40 1 -- start start 2024-08-03T20:58:38.529 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:38.527+0000 7fa6cd571f40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5651a4dc0360 0x5651a4dd88e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:58:38.529 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:38.527+0000 7fa6cd571f40 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5651a4dc1ea0 0x5651a4ddef80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:58:38.529 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:38.527+0000 7fa6cd571f40 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x5651a4dd7330 0x5651a4ddf4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:58:38.529 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:38.527+0000 7fa6cd571f40 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x5651a4c57e30 con 0x5651a4dc1ea0 2024-08-03T20:58:38.529 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:38.527+0000 7fa6cd571f40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x5651a4cb5bf0 con 0x5651a4dc0360 2024-08-03T20:58:38.529 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:38.527+0000 7fa6cd571f40 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x5651a4dc1440 con 0x5651a4dd7330 2024-08-03T20:58:38.529 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:38.527+0000 7fa6c5c00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5651a4dc1ea0 0x5651a4ddef80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:58:38.529 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:38.527+0000 7fa6c5c00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5651a4dc1ea0 0x5651a4ddef80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:41744/0 (socket says 172.21.15.32:41744) 2024-08-03T20:58:38.530 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:38.527+0000 7fa6c5c00640 1 -- 172.21.15.32:0/3954453961 learned_addr learned my addr 172.21.15.32:0/3954453961 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:58:38.530 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:38.527+0000 7fa6c5c00640 1 -- 172.21.15.32:0/3954453961 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x5651a4dd7330 msgr2=0x5651a4ddf4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:58:38.530 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:38.527+0000 7fa6c5c00640 1 --2- 172.21.15.32:0/3954453961 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x5651a4dd7330 0x5651a4ddf4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:38.530 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:38.527+0000 7fa6c5c00640 1 -- 172.21.15.32:0/3954453961 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5651a4dc0360 msgr2=0x5651a4dd88e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:58:38.530 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:38.527+0000 7fa6c5c00640 1 --2- 172.21.15.32:0/3954453961 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5651a4dc0360 0x5651a4dd88e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:38.530 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:38.527+0000 7fa6c5c00640 1 -- 172.21.15.32:0/3954453961 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5651a4dda490 con 0x5651a4dc1ea0 2024-08-03T20:58:38.530 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:38.527+0000 7fa6c5c00640 1 --2- 172.21.15.32:0/3954453961 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5651a4dc1ea0 0x5651a4ddef80 secure :-1 s=READY pgs=1821 cs=0 l=1 rev1=1 crypto rx=0x7fa6b400c550 tx=0x7fa6b400c950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=51767a092cb33bc2 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:58:38.530 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:38.528+0000 7fa6bfe00640 1 -- 172.21.15.32:0/3954453961 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6b400a960 con 0x5651a4dc1ea0 2024-08-03T20:58:38.530 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:38.528+0000 7fa6bfe00640 1 -- 172.21.15.32:0/3954453961 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fa6b400ab00 con 0x5651a4dc1ea0 2024-08-03T20:58:38.530 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:38.528+0000 7fa6bfe00640 1 -- 172.21.15.32:0/3954453961 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6b4008420 con 0x5651a4dc1ea0 2024-08-03T20:58:38.531 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:38.528+0000 7fa6cd571f40 1 -- 172.21.15.32:0/3954453961 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5651a4dc1ea0 msgr2=0x5651a4ddef80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:58:38.531 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:38.528+0000 7fa6cd571f40 1 --2- 172.21.15.32:0/3954453961 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5651a4dc1ea0 0x5651a4ddef80 secure :-1 s=READY pgs=1821 cs=0 l=1 rev1=1 crypto rx=0x7fa6b400c550 tx=0x7fa6b400c950 comp rx=0 tx=0).stop 2024-08-03T20:58:38.531 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:38.528+0000 7fa6cd571f40 1 -- 172.21.15.32:0/3954453961 shutdown_connections 2024-08-03T20:58:38.532 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:38.528+0000 7fa6cd571f40 1 --2- 172.21.15.32:0/3954453961 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x5651a4dd7330 0x5651a4ddf4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:38.532 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:38.528+0000 7fa6cd571f40 1 --2- 172.21.15.32:0/3954453961 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5651a4dc1ea0 0x5651a4ddef80 unknown :-1 s=CLOSED pgs=1821 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:38.532 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:38.528+0000 7fa6cd571f40 1 --2- 172.21.15.32:0/3954453961 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5651a4dc0360 0x5651a4dd88e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:38.532 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:38.528+0000 7fa6cd571f40 1 -- 172.21.15.32:0/3954453961 >> 172.21.15.32:0/3954453961 conn(0x5651a4c29ac0 msgr2=0x5651a4d4add0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:58:38.532 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:38.528+0000 7fa6cd571f40 1 -- 172.21.15.32:0/3954453961 shutdown_connections 2024-08-03T20:58:38.532 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:38.529+0000 7fa6cd571f40 1 -- 172.21.15.32:0/3954453961 wait complete. 2024-08-03T20:58:38.532 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:38.530+0000 7fa6cd571f40 1 Processor -- start 2024-08-03T20:58:38.532 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:38.530+0000 7fa6cd571f40 1 -- start start 2024-08-03T20:58:38.533 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:38.531+0000 7fa6cd571f40 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x5651a4dc0360 0x5651a4db7600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:58:38.533 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:38.531+0000 7fa6cd571f40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5651a4dc1ea0 0x5651a4ddef80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:58:38.534 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:38.531+0000 7fa6cd571f40 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5651a4dd7330 0x5651a4ddf4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:58:38.534 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:38.531+0000 7fa6cd571f40 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x5651a4d4ac50 con 0x5651a4dd7330 2024-08-03T20:58:38.534 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:38.531+0000 7fa6cd571f40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x5651a4ddfcb0 con 0x5651a4dc1ea0 2024-08-03T20:58:38.534 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:38.531+0000 7fa6cd571f40 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x5651a4e69120 con 0x5651a4dc0360 2024-08-03T20:58:38.534 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:38.531+0000 7fa6c7000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5651a4dd7330 0x5651a4ddf4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:58:38.534 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:38.531+0000 7fa6c7000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5651a4dd7330 0x5651a4ddf4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:41748/0 (socket says 172.21.15.32:41748) 2024-08-03T20:58:38.534 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:38.531+0000 7fa6c7000640 1 -- 172.21.15.32:0/37780910 learned_addr learned my addr 172.21.15.32:0/37780910 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:58:38.534 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:38.532+0000 7fa6c6600640 1 --2- 172.21.15.32:0/37780910 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x5651a4dc0360 0x5651a4db7600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:58:38.534 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:38.532+0000 7fa6c5c00640 1 --2- 172.21.15.32:0/37780910 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5651a4dc1ea0 0x5651a4ddef80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:58:38.534 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:38.532+0000 7fa6c7000640 1 -- 172.21.15.32:0/37780910 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x5651a4dc0360 msgr2=0x5651a4db7600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:58:38.534 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:38.532+0000 7fa6c7000640 1 --2- 172.21.15.32:0/37780910 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x5651a4dc0360 0x5651a4db7600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:38.534 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:38.532+0000 7fa6c7000640 1 -- 172.21.15.32:0/37780910 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5651a4dc1ea0 msgr2=0x5651a4ddef80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:58:38.535 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:38.532+0000 7fa6c7000640 1 --2- 172.21.15.32:0/37780910 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5651a4dc1ea0 0x5651a4ddef80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:38.535 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:38.532+0000 7fa6c7000640 1 -- 172.21.15.32:0/37780910 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5651a4e6a2d0 con 0x5651a4dd7330 2024-08-03T20:58:38.535 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:38.532+0000 7fa6c7000640 1 --2- 172.21.15.32:0/37780910 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5651a4dd7330 0x5651a4ddf4c0 secure :-1 s=READY pgs=1822 cs=0 l=1 rev1=1 crypto rx=0x7fa6c0025db0 tx=0x7fa6c004d610 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:58:38.535 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:38.532+0000 7fa6bea00640 1 -- 172.21.15.32:0/37780910 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6c0064070 con 0x5651a4dd7330 2024-08-03T20:58:38.536 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:38.533+0000 7fa6bea00640 1 -- 172.21.15.32:0/37780910 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fa6c0060430 con 0x5651a4dd7330 2024-08-03T20:58:38.536 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:38.533+0000 7fa6bea00640 1 -- 172.21.15.32:0/37780910 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6c0060860 con 0x5651a4dd7330 2024-08-03T20:58:38.536 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:38.533+0000 7fa6cd571f40 1 -- 172.21.15.32:0/37780910 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x5651a4e69ff0 con 0x5651a4dd7330 2024-08-03T20:58:38.536 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:38.533+0000 7fa6cd571f40 1 -- 172.21.15.32:0/37780910 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5651a4ddc0d0 con 0x5651a4dd7330 2024-08-03T20:58:38.536 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:38.535+0000 7fa6bea00640 1 -- 172.21.15.32:0/37780910 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7fa6c0067070 con 0x5651a4dd7330 2024-08-03T20:58:38.537 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:38.536+0000 7fa6bea00640 1 --2- 172.21.15.32:0/37780910 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fa6a80759f0 0x7fa6a8077f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:58:38.537 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:38.536+0000 7fa6bea00640 1 -- 172.21.15.32:0/37780910 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(116..116 src has 1..116) ==== 8520+0+0 (secure 0 0 0) 0x7fa6c00dc5d0 con 0x5651a4dd7330 2024-08-03T20:58:38.538 INFO:teuthology.orchestra.run.smithi032.stdout:Importing pool 2024-08-03T20:58:38.538 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:38.536+0000 7fa6c6600640 1 --2- 172.21.15.32:0/37780910 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fa6a80759f0 0x7fa6a8077f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:58:38.538 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:38.536+0000 7fa6cd571f40 1 --2- 172.21.15.32:0/37780910 >> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] conn(0x5651a4e682f0 0x5651a4e68700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:58:38.539 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:38.536+0000 7fa6cd571f40 1 -- 172.21.15.32:0/37780910 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:1 12.10 12:09401a81:::601.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116) -- 0x5651a4e808a0 con 0x5651a4e682f0 2024-08-03T20:58:38.539 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:38.537+0000 7fa6c5c00640 1 --2- 172.21.15.32:0/37780910 >> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] conn(0x5651a4e682f0 0x5651a4e68700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:58:38.539 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:38.537+0000 7fa6c6600640 1 --2- 172.21.15.32:0/37780910 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fa6a80759f0 0x7fa6a8077f00 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 crypto rx=0x5651a4de0780 tx=0x7fa6b8001450 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:58:38.539 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:38.537+0000 7fa6c5c00640 1 --2- 172.21.15.32:0/37780910 >> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] conn(0x5651a4e682f0 0x5651a4e68700 crc :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:58:38.541 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:38.540+0000 7fa6c5c00640 1 -- 172.21.15.32:0/37780910 <== osd.0 v2:172.21.15.32:6816/3066156116 1 ==== osd_op_reply(1 601.00000000 [create] v116'939 uv913 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fa6b400ab80 con 0x5651a4e682f0 2024-08-03T20:58:38.541 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:38.540+0000 7fa6cd571f40 1 -- 172.21.15.32:0/37780910 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:2 12.10 12:09401a81:::601.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e116) -- 0x5651a4e80c40 con 0x5651a4e682f0 2024-08-03T20:58:38.543 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:38.541+0000 7fa6c5c00640 1 -- 172.21.15.32:0/37780910 <== osd.0 v2:172.21.15.32:6816/3066156116 2 ==== osd_op_reply(2 601.00000000 [delete] v116'940 uv940 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fa6b400ab80 con 0x5651a4e682f0 2024-08-03T20:58:38.543 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:38.541+0000 7fa6cd571f40 1 -- 172.21.15.32:0/37780910 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:3 12.10 12:09401a81:::601.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116) -- 0x5651a4e80fe0 con 0x5651a4e682f0 2024-08-03T20:58:38.545 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::601.00000000:head# 2024-08-03T20:58:38.545 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:38.543+0000 7fa6c5c00640 1 -- 172.21.15.32:0/37780910 <== osd.0 v2:172.21.15.32:6816/3066156116 3 ==== osd_op_reply(3 601.00000000 [create] v116'941 uv941 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fa6b400ab80 con 0x5651a4e682f0 2024-08-03T20:58:38.545 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:38.543+0000 7fa6cd571f40 1 -- 172.21.15.32:0/37780910 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:4 12.10 12:09401a81:::601.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116) -- 0x5651a4e82390 con 0x5651a4e682f0 2024-08-03T20:58:38.547 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:38.545+0000 7fa6c5c00640 1 -- 172.21.15.32:0/37780910 <== osd.0 v2:172.21.15.32:6816/3066156116 4 ==== osd_op_reply(4 601.00000000 [setxattr (30)] v116'942 uv942 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fa6b400ab80 con 0x5651a4e682f0 2024-08-03T20:58:38.547 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:38.545+0000 7fa6cd571f40 1 -- 172.21.15.32:0/37780910 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:5 12.10 12:09401a81:::601.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116) -- 0x5651a4e68c40 con 0x5651a4e682f0 2024-08-03T20:58:38.548 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:38.547+0000 7fa6c5c00640 1 -- 172.21.15.32:0/37780910 <== osd.0 v2:172.21.15.32:6816/3066156116 5 ==== osd_op_reply(5 601.00000000 [setxattr (62)] v116'943 uv943 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fa6b400ab80 con 0x5651a4e682f0 2024-08-03T20:58:38.548 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:38.547+0000 7fa6cd571f40 1 -- 172.21.15.32:0/37780910 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:6 12.10 12:09401a81:::601.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116) -- 0x5651a4e81380 con 0x5651a4e682f0 2024-08-03T20:58:38.550 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:38.549+0000 7fa6c5c00640 1 -- 172.21.15.32:0/37780910 <== osd.0 v2:172.21.15.32:6816/3066156116 6 ==== osd_op_reply(6 601.00000000 [omap-set-header] v116'944 uv944 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fa6b400ab80 con 0x5651a4e682f0 2024-08-03T20:58:38.550 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:38.549+0000 7fa6cd571f40 1 -- 172.21.15.32:0/37780910 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:7 12.10 12:0bd6d154:::602.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116) -- 0x5651a4e81790 con 0x5651a4e682f0 2024-08-03T20:58:38.552 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:38.551+0000 7fa6c5c00640 1 -- 172.21.15.32:0/37780910 <== osd.0 v2:172.21.15.32:6816/3066156116 7 ==== osd_op_reply(7 602.00000000 [create] v116'945 uv919 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fa6b400ab80 con 0x5651a4e682f0 2024-08-03T20:58:38.552 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:38.551+0000 7fa6cd571f40 1 -- 172.21.15.32:0/37780910 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:8 12.10 12:0bd6d154:::602.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e116) -- 0x5651a4e81b30 con 0x5651a4e682f0 2024-08-03T20:58:38.554 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:38.552+0000 7fa6c5c00640 1 -- 172.21.15.32:0/37780910 <== osd.0 v2:172.21.15.32:6816/3066156116 8 ==== osd_op_reply(8 602.00000000 [delete] v116'946 uv946 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fa6b400ab80 con 0x5651a4e682f0 2024-08-03T20:58:38.554 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:38.552+0000 7fa6cd571f40 1 -- 172.21.15.32:0/37780910 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:9 12.10 12:0bd6d154:::602.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116) -- 0x5651a4e81b30 con 0x5651a4e682f0 2024-08-03T20:58:38.555 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::602.00000000:head# 2024-08-03T20:58:38.555 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:38.554+0000 7fa6c5c00640 1 -- 172.21.15.32:0/37780910 <== osd.0 v2:172.21.15.32:6816/3066156116 9 ==== osd_op_reply(9 602.00000000 [create] v116'947 uv947 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fa6b400ab80 con 0x5651a4e682f0 2024-08-03T20:58:38.556 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:38.554+0000 7fa6cd571f40 1 -- 172.21.15.32:0/37780910 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:10 12.10 12:0bd6d154:::602.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116) -- 0x5651a4e81b30 con 0x5651a4e682f0 2024-08-03T20:58:38.557 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:38.556+0000 7fa6c5c00640 1 -- 172.21.15.32:0/37780910 <== osd.0 v2:172.21.15.32:6816/3066156116 10 ==== osd_op_reply(10 602.00000000 [setxattr (30)] v116'948 uv948 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fa6b400ab80 con 0x5651a4e682f0 2024-08-03T20:58:38.557 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:38.556+0000 7fa6cd571f40 1 -- 172.21.15.32:0/37780910 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:11 12.10 12:0bd6d154:::602.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116) -- 0x5651a4e81b30 con 0x5651a4e682f0 2024-08-03T20:58:38.559 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:38.558+0000 7fa6c5c00640 1 -- 172.21.15.32:0/37780910 <== osd.0 v2:172.21.15.32:6816/3066156116 11 ==== osd_op_reply(11 602.00000000 [setxattr (62)] v116'949 uv949 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fa6b400ab80 con 0x5651a4e682f0 2024-08-03T20:58:38.559 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:38.558+0000 7fa6cd571f40 1 -- 172.21.15.32:0/37780910 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:12 12.10 12:0bd6d154:::602.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116) -- 0x5651a4e81b30 con 0x5651a4e682f0 2024-08-03T20:58:38.561 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:38.559+0000 7fa6c5c00640 1 -- 172.21.15.32:0/37780910 <== osd.0 v2:172.21.15.32:6816/3066156116 12 ==== osd_op_reply(12 602.00000000 [omap-set-header] v116'950 uv950 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fa6b400ab80 con 0x5651a4e682f0 2024-08-03T20:58:38.561 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:38.559+0000 7fa6cd571f40 1 -- 172.21.15.32:0/37780910 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:13 12.10 12:0cf6ad0f:::10000000000.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116) -- 0x5651a4e81b30 con 0x5651a4e682f0 2024-08-03T20:58:38.562 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:38.561+0000 7fa6c5c00640 1 -- 172.21.15.32:0/37780910 <== osd.0 v2:172.21.15.32:6816/3066156116 13 ==== osd_op_reply(13 10000000000.00000000 [create] v116'951 uv926 ondisk = -17 ((17) File exists)) ==== 164+0+0 (crc 0 0 0) 0x7fa6b400ab80 con 0x5651a4e682f0 2024-08-03T20:58:38.562 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:38.561+0000 7fa6cd571f40 1 -- 172.21.15.32:0/37780910 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:14 12.10 12:0cf6ad0f:::10000000000.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e116) -- 0x5651a4e81b30 con 0x5651a4e682f0 2024-08-03T20:58:38.564 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:38.563+0000 7fa6c5c00640 1 -- 172.21.15.32:0/37780910 <== osd.0 v2:172.21.15.32:6816/3066156116 14 ==== osd_op_reply(14 10000000000.00000000 [delete] v116'952 uv952 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7fa6b400ab80 con 0x5651a4e682f0 2024-08-03T20:58:38.564 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:38.563+0000 7fa6cd571f40 1 -- 172.21.15.32:0/37780910 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:15 12.10 12:0cf6ad0f:::10000000000.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116) -- 0x5651a4e81b30 con 0x5651a4e682f0 2024-08-03T20:58:38.566 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::10000000000.00000000:head# 2024-08-03T20:58:38.566 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:38.564+0000 7fa6c5c00640 1 -- 172.21.15.32:0/37780910 <== osd.0 v2:172.21.15.32:6816/3066156116 15 ==== osd_op_reply(15 10000000000.00000000 [create] v116'953 uv953 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7fa6b400ab80 con 0x5651a4e682f0 2024-08-03T20:58:38.566 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:38.564+0000 7fa6cd571f40 1 -- 172.21.15.32:0/37780910 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:16 12.10 12:0cf6ad0f:::10000000000.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116) -- 0x5651a4e81b30 con 0x5651a4e682f0 2024-08-03T20:58:38.567 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:38.566+0000 7fa6c5c00640 1 -- 172.21.15.32:0/37780910 <== osd.0 v2:172.21.15.32:6816/3066156116 16 ==== osd_op_reply(16 10000000000.00000000 [setxattr (30)] v116'954 uv954 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7fa6b400ab80 con 0x5651a4e682f0 2024-08-03T20:58:38.567 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:38.566+0000 7fa6cd571f40 1 -- 172.21.15.32:0/37780910 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:17 12.10 12:0cf6ad0f:::10000000000.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116) -- 0x5651a4e81b30 con 0x5651a4e682f0 2024-08-03T20:58:38.569 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:38.567+0000 7fa6c5c00640 1 -- 172.21.15.32:0/37780910 <== osd.0 v2:172.21.15.32:6816/3066156116 17 ==== osd_op_reply(17 10000000000.00000000 [setxattr (62)] v116'955 uv955 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7fa6b400ab80 con 0x5651a4e682f0 2024-08-03T20:58:38.569 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:38.567+0000 7fa6cd571f40 1 -- 172.21.15.32:0/37780910 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:18 12.10 12:0cf6ad0f:::10000000000.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116) -- 0x5651a4e81b30 con 0x5651a4e682f0 2024-08-03T20:58:38.570 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:38.569+0000 7fa6c5c00640 1 -- 172.21.15.32:0/37780910 <== osd.0 v2:172.21.15.32:6816/3066156116 18 ==== osd_op_reply(18 10000000000.00000000 [omap-set-header] v116'956 uv956 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7fa6b400ab80 con 0x5651a4e682f0 2024-08-03T20:58:38.571 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:38.569+0000 7fa6cd571f40 1 -- 172.21.15.32:0/37780910 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:19 12.10 12:0cf6ad0f:::10000000000.00000000:head [omap-set-vals in=593b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116) -- 0x5651a4e81ef0 con 0x5651a4e682f0 2024-08-03T20:58:38.572 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:38.571+0000 7fa6c5c00640 1 -- 172.21.15.32:0/37780910 <== osd.0 v2:172.21.15.32:6816/3066156116 19 ==== osd_op_reply(19 10000000000.00000000 [omap-set-vals] v116'957 uv957 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7fa6b400ab80 con 0x5651a4e682f0 2024-08-03T20:58:38.572 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:38.571+0000 7fa6cd571f40 1 -- 172.21.15.32:0/37780910 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:20 12.10 12:0d82a743:::600.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116) -- 0x5651a4e81df0 con 0x5651a4e682f0 2024-08-03T20:58:38.574 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:38.572+0000 7fa6c5c00640 1 -- 172.21.15.32:0/37780910 <== osd.0 v2:172.21.15.32:6816/3066156116 20 ==== osd_op_reply(20 600.00000000 [create] v116'958 uv932 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fa6b400ab80 con 0x5651a4e682f0 2024-08-03T20:58:38.574 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:38.573+0000 7fa6cd571f40 1 -- 172.21.15.32:0/37780910 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:21 12.10 12:0d82a743:::600.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e116) -- 0x5651a4e81df0 con 0x5651a4e682f0 2024-08-03T20:58:38.576 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:38.574+0000 7fa6c5c00640 1 -- 172.21.15.32:0/37780910 <== osd.0 v2:172.21.15.32:6816/3066156116 21 ==== osd_op_reply(21 600.00000000 [delete] v116'959 uv959 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fa6b400ab80 con 0x5651a4e682f0 2024-08-03T20:58:38.576 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:38.574+0000 7fa6cd571f40 1 -- 172.21.15.32:0/37780910 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:22 12.10 12:0d82a743:::600.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116) -- 0x5651a4e81df0 con 0x5651a4e682f0 2024-08-03T20:58:38.577 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::600.00000000:head# 2024-08-03T20:58:38.577 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:38.576+0000 7fa6c5c00640 1 -- 172.21.15.32:0/37780910 <== osd.0 v2:172.21.15.32:6816/3066156116 22 ==== osd_op_reply(22 600.00000000 [create] v116'960 uv960 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fa6b400ab80 con 0x5651a4e682f0 2024-08-03T20:58:38.577 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:38.576+0000 7fa6cd571f40 1 -- 172.21.15.32:0/37780910 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:23 12.10 12:0d82a743:::600.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116) -- 0x5651a4e81df0 con 0x5651a4e682f0 2024-08-03T20:58:38.579 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:38.577+0000 7fa6c5c00640 1 -- 172.21.15.32:0/37780910 <== osd.0 v2:172.21.15.32:6816/3066156116 23 ==== osd_op_reply(23 600.00000000 [setxattr (30)] v116'961 uv961 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fa6b400ab80 con 0x5651a4e682f0 2024-08-03T20:58:38.579 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:38.577+0000 7fa6cd571f40 1 -- 172.21.15.32:0/37780910 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:24 12.10 12:0d82a743:::600.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116) -- 0x5651a4e81df0 con 0x5651a4e682f0 2024-08-03T20:58:38.580 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:38.579+0000 7fa6c5c00640 1 -- 172.21.15.32:0/37780910 <== osd.0 v2:172.21.15.32:6816/3066156116 24 ==== osd_op_reply(24 600.00000000 [setxattr (62)] v116'962 uv962 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fa6b400ab80 con 0x5651a4e682f0 2024-08-03T20:58:38.580 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:38.579+0000 7fa6cd571f40 1 -- 172.21.15.32:0/37780910 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:25 12.10 12:0d82a743:::600.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116) -- 0x5651a4e81df0 con 0x5651a4e682f0 2024-08-03T20:58:38.582 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:38.580+0000 7fa6c5c00640 1 -- 172.21.15.32:0/37780910 <== osd.0 v2:172.21.15.32:6816/3066156116 25 ==== osd_op_reply(25 600.00000000 [omap-set-header] v116'963 uv963 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fa6b400ab80 con 0x5651a4e682f0 2024-08-03T20:58:38.582 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:38.580+0000 7fa6cd571f40 1 -- 172.21.15.32:0/37780910 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:26 12.10 12:0d89b25e:::603.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116) -- 0x5651a4e81df0 con 0x5651a4e682f0 2024-08-03T20:58:38.583 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:38.582+0000 7fa6c5c00640 1 -- 172.21.15.32:0/37780910 <== osd.0 v2:172.21.15.32:6816/3066156116 26 ==== osd_op_reply(26 603.00000000 [create] v116'964 uv938 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fa6b400ab80 con 0x5651a4e682f0 2024-08-03T20:58:38.584 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:38.582+0000 7fa6cd571f40 1 -- 172.21.15.32:0/37780910 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:27 12.10 12:0d89b25e:::603.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e116) -- 0x5651a4e81df0 con 0x5651a4e682f0 2024-08-03T20:58:38.585 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:38.584+0000 7fa6c5c00640 1 -- 172.21.15.32:0/37780910 <== osd.0 v2:172.21.15.32:6816/3066156116 27 ==== osd_op_reply(27 603.00000000 [delete] v116'965 uv965 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fa6b400ab80 con 0x5651a4e682f0 2024-08-03T20:58:38.585 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:38.584+0000 7fa6cd571f40 1 -- 172.21.15.32:0/37780910 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:28 12.10 12:0d89b25e:::603.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116) -- 0x5651a4e81df0 con 0x5651a4e682f0 2024-08-03T20:58:38.587 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::603.00000000:head# 2024-08-03T20:58:38.587 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:38.585+0000 7fa6c5c00640 1 -- 172.21.15.32:0/37780910 <== osd.0 v2:172.21.15.32:6816/3066156116 28 ==== osd_op_reply(28 603.00000000 [create] v116'966 uv966 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fa6b400ab80 con 0x5651a4e682f0 2024-08-03T20:58:38.587 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:38.585+0000 7fa6cd571f40 1 -- 172.21.15.32:0/37780910 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:29 12.10 12:0d89b25e:::603.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116) -- 0x5651a4e81df0 con 0x5651a4e682f0 2024-08-03T20:58:38.588 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:38.587+0000 7fa6c5c00640 1 -- 172.21.15.32:0/37780910 <== osd.0 v2:172.21.15.32:6816/3066156116 29 ==== osd_op_reply(29 603.00000000 [setxattr (30)] v116'967 uv967 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fa6b400ab80 con 0x5651a4e682f0 2024-08-03T20:58:38.588 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:38.587+0000 7fa6cd571f40 1 -- 172.21.15.32:0/37780910 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:30 12.10 12:0d89b25e:::603.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116) -- 0x5651a4e81df0 con 0x5651a4e682f0 2024-08-03T20:58:38.590 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:38.588+0000 7fa6c5c00640 1 -- 172.21.15.32:0/37780910 <== osd.0 v2:172.21.15.32:6816/3066156116 30 ==== osd_op_reply(30 603.00000000 [setxattr (62)] v116'968 uv968 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fa6b400ab80 con 0x5651a4e682f0 2024-08-03T20:58:38.590 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:38.588+0000 7fa6cd571f40 1 -- 172.21.15.32:0/37780910 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:31 12.10 12:0d89b25e:::603.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116) -- 0x5651a4e81df0 con 0x5651a4e682f0 2024-08-03T20:58:38.592 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:38.590+0000 7fa6c5c00640 1 -- 172.21.15.32:0/37780910 <== osd.0 v2:172.21.15.32:6816/3066156116 31 ==== osd_op_reply(31 603.00000000 [omap-set-header] v116'969 uv969 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fa6b400ab80 con 0x5651a4e682f0 2024-08-03T20:58:38.592 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:38.590+0000 7fa6cd571f40 1 --2- 172.21.15.32:0/37780910 >> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] conn(0x5651a4e82c40 0x5651a4e85060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:58:38.592 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:38.590+0000 7fa6cd571f40 1 -- 172.21.15.32:0/37780910 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:32 12.14 12:287034c4:::1.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116) -- 0x5651a4e81df0 con 0x5651a4e82c40 2024-08-03T20:58:38.592 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:38.591+0000 7fa6c6600640 1 --2- 172.21.15.32:0/37780910 >> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] conn(0x5651a4e82c40 0x5651a4e85060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:58:38.593 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:38.591+0000 7fa6c6600640 1 --2- 172.21.15.32:0/37780910 >> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] conn(0x5651a4e82c40 0x5651a4e85060 crc :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:58:38.595 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:38.593+0000 7fa6c6600640 1 -- 172.21.15.32:0/37780910 <== osd.5 v2:172.21.15.118:6810/2724891770 1 ==== osd_op_reply(32 1.00000000.inode [create] v116'680 uv663 ondisk = -17 ((17) File exists)) ==== 160+0+0 (crc 0 0 0) 0x7fa6b800e040 con 0x5651a4e82c40 2024-08-03T20:58:38.595 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:38.593+0000 7fa6cd571f40 1 -- 172.21.15.32:0/37780910 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:33 12.14 12:287034c4:::1.00000000.inode:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e116) -- 0x5651a4e855a0 con 0x5651a4e82c40 2024-08-03T20:58:38.597 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:38.595+0000 7fa6c6600640 1 -- 172.21.15.32:0/37780910 <== osd.5 v2:172.21.15.118:6810/2724891770 2 ==== osd_op_reply(33 1.00000000.inode [delete] v116'681 uv681 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7fa6b800e040 con 0x5651a4e82c40 2024-08-03T20:58:38.597 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:38.595+0000 7fa6cd571f40 1 -- 172.21.15.32:0/37780910 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:34 12.14 12:287034c4:::1.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116) -- 0x5651a4e85940 con 0x5651a4e82c40 2024-08-03T20:58:38.598 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::1.00000000.inode:head# 2024-08-03T20:58:38.598 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:38.597+0000 7fa6c6600640 1 -- 172.21.15.32:0/37780910 <== osd.5 v2:172.21.15.118:6810/2724891770 3 ==== osd_op_reply(34 1.00000000.inode [create] v116'682 uv682 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7fa6b800e040 con 0x5651a4e82c40 2024-08-03T20:58:38.598 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:38.597+0000 7fa6cd571f40 1 -- 172.21.15.32:0/37780910 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:35 12.14 12:287034c4:::1.00000000.inode:head [write 0~582 in=582b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116) -- 0x5651a4e85ce0 con 0x5651a4e82c40 2024-08-03T20:58:38.600 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:38.598+0000 7fa6c6600640 1 -- 172.21.15.32:0/37780910 <== osd.5 v2:172.21.15.118:6810/2724891770 4 ==== osd_op_reply(35 1.00000000.inode [write 0~582] v116'683 uv683 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7fa6b800e040 con 0x5651a4e82c40 2024-08-03T20:58:38.600 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:38.598+0000 7fa6cd571f40 1 -- 172.21.15.32:0/37780910 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:36 12.14 12:287034c4:::1.00000000.inode:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116) -- 0x5651a4e86080 con 0x5651a4e82c40 2024-08-03T20:58:38.601 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:38.600+0000 7fa6c6600640 1 -- 172.21.15.32:0/37780910 <== osd.5 v2:172.21.15.118:6810/2724891770 5 ==== osd_op_reply(36 1.00000000.inode [omap-set-header] v116'684 uv684 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7fa6b800e040 con 0x5651a4e82c40 2024-08-03T20:58:38.601 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:38.600+0000 7fa6cd571f40 1 -- 172.21.15.32:0/37780910 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:37 12.14 12:292cf221:::200.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116) -- 0x5651a4e86420 con 0x5651a4e82c40 2024-08-03T20:58:38.603 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:38.601+0000 7fa6c6600640 1 -- 172.21.15.32:0/37780910 <== osd.5 v2:172.21.15.118:6810/2724891770 6 ==== osd_op_reply(37 200.00000000 [create] v116'685 uv668 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fa6b800e040 con 0x5651a4e82c40 2024-08-03T20:58:38.603 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:38.601+0000 7fa6cd571f40 1 -- 172.21.15.32:0/37780910 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:38 12.14 12:292cf221:::200.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e116) -- 0x5651a4e867c0 con 0x5651a4e82c40 2024-08-03T20:58:38.604 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:38.603+0000 7fa6c6600640 1 -- 172.21.15.32:0/37780910 <== osd.5 v2:172.21.15.118:6810/2724891770 7 ==== osd_op_reply(38 200.00000000 [delete] v116'686 uv686 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fa6b800e040 con 0x5651a4e82c40 2024-08-03T20:58:38.604 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:38.603+0000 7fa6cd571f40 1 -- 172.21.15.32:0/37780910 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:39 12.14 12:292cf221:::200.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116) -- 0x5651a4e86b60 con 0x5651a4e82c40 2024-08-03T20:58:38.606 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::200.00000000:head# 2024-08-03T20:58:38.606 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:38.605+0000 7fa6c6600640 1 -- 172.21.15.32:0/37780910 <== osd.5 v2:172.21.15.118:6810/2724891770 8 ==== osd_op_reply(39 200.00000000 [create] v116'687 uv687 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fa6b800e040 con 0x5651a4e82c40 2024-08-03T20:58:38.606 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:38.605+0000 7fa6cd571f40 1 -- 172.21.15.32:0/37780910 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:40 12.14 12:292cf221:::200.00000000:head [write 0~90 in=90b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116) -- 0x5651a4e69870 con 0x5651a4e82c40 2024-08-03T20:58:38.608 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:38.607+0000 7fa6c6600640 1 -- 172.21.15.32:0/37780910 <== osd.5 v2:172.21.15.118:6810/2724891770 9 ==== osd_op_reply(40 200.00000000 [write 0~90] v116'688 uv688 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fa6b800e040 con 0x5651a4e82c40 2024-08-03T20:58:38.608 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:38.607+0000 7fa6cd571f40 1 -- 172.21.15.32:0/37780910 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:41 12.14 12:292cf221:::200.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116) -- 0x5651a4e69870 con 0x5651a4e82c40 2024-08-03T20:58:38.609 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:38.608+0000 7fa6c6600640 1 -- 172.21.15.32:0/37780910 <== osd.5 v2:172.21.15.118:6810/2724891770 10 ==== osd_op_reply(41 200.00000000 [omap-set-header] v116'689 uv689 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fa6b800e040 con 0x5651a4e82c40 2024-08-03T20:58:38.610 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:38.608+0000 7fa6cd571f40 1 -- 172.21.15.32:0/37780910 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:42 12.14 12:2e2fa760:::200.00000001:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116) -- 0x5651a4e69870 con 0x5651a4e82c40 2024-08-03T20:58:38.611 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:38.610+0000 7fa6c6600640 1 -- 172.21.15.32:0/37780910 <== osd.5 v2:172.21.15.118:6810/2724891770 11 ==== osd_op_reply(42 200.00000001 [create] v116'690 uv673 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fa6b800e040 con 0x5651a4e82c40 2024-08-03T20:58:38.612 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:38.610+0000 7fa6cd571f40 1 -- 172.21.15.32:0/37780910 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:43 12.14 12:2e2fa760:::200.00000001:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e116) -- 0x5651a4e69870 con 0x5651a4e82c40 2024-08-03T20:58:38.613 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:38.612+0000 7fa6c6600640 1 -- 172.21.15.32:0/37780910 <== osd.5 v2:172.21.15.118:6810/2724891770 12 ==== osd_op_reply(43 200.00000001 [delete] v116'691 uv691 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fa6b800e040 con 0x5651a4e82c40 2024-08-03T20:58:38.613 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:38.612+0000 7fa6cd571f40 1 -- 172.21.15.32:0/37780910 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:44 12.14 12:2e2fa760:::200.00000001:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116) -- 0x5651a4e69870 con 0x5651a4e82c40 2024-08-03T20:58:38.615 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:38.613+0000 7fa6c6600640 1 -- 172.21.15.32:0/37780910 <== osd.5 v2:172.21.15.118:6810/2724891770 13 ==== osd_op_reply(44 200.00000001 [create] v116'692 uv692 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fa6b800e040 con 0x5651a4e82c40 2024-08-03T20:58:38.615 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::200.00000001:head# 2024-08-03T20:58:38.615 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:38.614+0000 7fa6cd571f40 1 -- 172.21.15.32:0/37780910 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:45 12.14 12:2e2fa760:::200.00000001:head [write 0~18118 in=18118b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116) -- 0x5651a4e69870 con 0x5651a4e82c40 2024-08-03T20:58:38.617 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:38.615+0000 7fa6c6600640 1 -- 172.21.15.32:0/37780910 <== osd.5 v2:172.21.15.118:6810/2724891770 14 ==== osd_op_reply(45 200.00000001 [write 0~18118] v116'693 uv693 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fa6b800e040 con 0x5651a4e82c40 2024-08-03T20:58:38.617 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:38.616+0000 7fa6cd571f40 1 -- 172.21.15.32:0/37780910 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:46 12.14 12:2e2fa760:::200.00000001:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116) -- 0x5651a4e69870 con 0x5651a4e82c40 2024-08-03T20:58:38.618 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:38.617+0000 7fa6c6600640 1 -- 172.21.15.32:0/37780910 <== osd.5 v2:172.21.15.118:6810/2724891770 15 ==== osd_op_reply(46 200.00000001 [omap-set-header] v116'694 uv694 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fa6b800e040 con 0x5651a4e82c40 2024-08-03T20:58:38.618 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:38.617+0000 7fa6cd571f40 1 -- 172.21.15.32:0/37780910 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:47 12.14 12:2f55791f:::606.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116) -- 0x5651a4e69870 con 0x5651a4e82c40 2024-08-03T20:58:38.620 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:38.619+0000 7fa6c6600640 1 -- 172.21.15.32:0/37780910 <== osd.5 v2:172.21.15.118:6810/2724891770 16 ==== osd_op_reply(47 606.00000000 [create] v116'695 uv679 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fa6b800e040 con 0x5651a4e82c40 2024-08-03T20:58:38.620 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:38.619+0000 7fa6cd571f40 1 -- 172.21.15.32:0/37780910 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:48 12.14 12:2f55791f:::606.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e116) -- 0x5651a4e69870 con 0x5651a4e82c40 2024-08-03T20:58:38.622 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:38.621+0000 7fa6c6600640 1 -- 172.21.15.32:0/37780910 <== osd.5 v2:172.21.15.118:6810/2724891770 17 ==== osd_op_reply(48 606.00000000 [delete] v116'696 uv696 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fa6b800e040 con 0x5651a4e82c40 2024-08-03T20:58:38.622 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:38.621+0000 7fa6cd571f40 1 -- 172.21.15.32:0/37780910 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:49 12.14 12:2f55791f:::606.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116) -- 0x5651a4e69870 con 0x5651a4e82c40 2024-08-03T20:58:38.623 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:38.622+0000 7fa6c6600640 1 -- 172.21.15.32:0/37780910 <== osd.5 v2:172.21.15.118:6810/2724891770 18 ==== osd_op_reply(49 606.00000000 [create] v116'697 uv697 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fa6b800e040 con 0x5651a4e82c40 2024-08-03T20:58:38.624 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::606.00000000:head# 2024-08-03T20:58:38.624 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:38.622+0000 7fa6cd571f40 1 -- 172.21.15.32:0/37780910 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:50 12.14 12:2f55791f:::606.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116) -- 0x5651a4e69870 con 0x5651a4e82c40 2024-08-03T20:58:38.625 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:38.624+0000 7fa6c6600640 1 -- 172.21.15.32:0/37780910 <== osd.5 v2:172.21.15.118:6810/2724891770 19 ==== osd_op_reply(50 606.00000000 [setxattr (30)] v116'698 uv698 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fa6b800e040 con 0x5651a4e82c40 2024-08-03T20:58:38.625 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:38.624+0000 7fa6cd571f40 1 -- 172.21.15.32:0/37780910 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:51 12.14 12:2f55791f:::606.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116) -- 0x5651a4e69870 con 0x5651a4e82c40 2024-08-03T20:58:38.627 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:38.625+0000 7fa6c6600640 1 -- 172.21.15.32:0/37780910 <== osd.5 v2:172.21.15.118:6810/2724891770 20 ==== osd_op_reply(51 606.00000000 [setxattr (62)] v116'699 uv699 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fa6b800e040 con 0x5651a4e82c40 2024-08-03T20:58:38.627 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:38.625+0000 7fa6cd571f40 1 -- 172.21.15.32:0/37780910 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:52 12.14 12:2f55791f:::606.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116) -- 0x5651a4e69870 con 0x5651a4e82c40 2024-08-03T20:58:38.628 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:38.627+0000 7fa6c6600640 1 -- 172.21.15.32:0/37780910 <== osd.5 v2:172.21.15.118:6810/2724891770 21 ==== osd_op_reply(52 606.00000000 [omap-set-header] v116'700 uv700 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fa6b800e040 con 0x5651a4e82c40 2024-08-03T20:58:38.629 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:38.627+0000 7fa6cd571f40 1 -- 172.21.15.32:0/37780910 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:53 12.1c 12:3ed09add:::607.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116) -- 0x5651a4e69870 con 0x5651a4e682f0 2024-08-03T20:58:38.630 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:38.629+0000 7fa6c5c00640 1 -- 172.21.15.32:0/37780910 <== osd.0 v2:172.21.15.32:6816/3066156116 32 ==== osd_op_reply(53 607.00000000 [create] v116'183 uv182 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fa6b400ab80 con 0x5651a4e682f0 2024-08-03T20:58:38.630 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:38.629+0000 7fa6cd571f40 1 -- 172.21.15.32:0/37780910 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:54 12.1c 12:3ed09add:::607.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e116) -- 0x5651a4e69870 con 0x5651a4e682f0 2024-08-03T20:58:38.631 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:38.630+0000 7fa6c5c00640 1 -- 172.21.15.32:0/37780910 <== osd.0 v2:172.21.15.32:6816/3066156116 33 ==== osd_op_reply(54 607.00000000 [delete] v116'184 uv184 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fa6b400ab80 con 0x5651a4e682f0 2024-08-03T20:58:38.631 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:38.630+0000 7fa6cd571f40 1 -- 172.21.15.32:0/37780910 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:55 12.1c 12:3ed09add:::607.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116) -- 0x5651a4e69870 con 0x5651a4e682f0 2024-08-03T20:58:38.633 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::607.00000000:head# 2024-08-03T20:58:38.633 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:38.631+0000 7fa6c5c00640 1 -- 172.21.15.32:0/37780910 <== osd.0 v2:172.21.15.32:6816/3066156116 34 ==== osd_op_reply(55 607.00000000 [create] v116'185 uv185 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fa6b400ab80 con 0x5651a4e682f0 2024-08-03T20:58:38.633 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:38.631+0000 7fa6cd571f40 1 -- 172.21.15.32:0/37780910 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:56 12.1c 12:3ed09add:::607.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116) -- 0x5651a4e69870 con 0x5651a4e682f0 2024-08-03T20:58:38.634 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:38.633+0000 7fa6c5c00640 1 -- 172.21.15.32:0/37780910 <== osd.0 v2:172.21.15.32:6816/3066156116 35 ==== osd_op_reply(56 607.00000000 [setxattr (30)] v116'186 uv186 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fa6b400ab80 con 0x5651a4e682f0 2024-08-03T20:58:38.634 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:38.633+0000 7fa6cd571f40 1 -- 172.21.15.32:0/37780910 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:57 12.1c 12:3ed09add:::607.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116) -- 0x5651a4e69870 con 0x5651a4e682f0 2024-08-03T20:58:38.635 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:38.634+0000 7fa6c5c00640 1 -- 172.21.15.32:0/37780910 <== osd.0 v2:172.21.15.32:6816/3066156116 36 ==== osd_op_reply(57 607.00000000 [setxattr (62)] v116'187 uv187 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fa6b400ab80 con 0x5651a4e682f0 2024-08-03T20:58:38.635 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:38.634+0000 7fa6cd571f40 1 -- 172.21.15.32:0/37780910 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:58 12.1c 12:3ed09add:::607.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116) -- 0x5651a4e69870 con 0x5651a4e682f0 2024-08-03T20:58:38.637 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:38.635+0000 7fa6c5c00640 1 -- 172.21.15.32:0/37780910 <== osd.0 v2:172.21.15.32:6816/3066156116 37 ==== osd_op_reply(58 607.00000000 [omap-set-header] v116'188 uv188 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fa6b400ab80 con 0x5651a4e682f0 2024-08-03T20:58:38.637 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:38.635+0000 7fa6cd571f40 1 --2- 172.21.15.32:0/37780910 >> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] conn(0x5651a4e87520 0x5651a4e89940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:58:38.637 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:38.635+0000 7fa6cd571f40 1 -- 172.21.15.32:0/37780910 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:59 12.6 12:654134d2:::mds0_openfiles.0:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116) -- 0x5651a4e89e80 con 0x5651a4e87520 2024-08-03T20:58:38.637 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:38.636+0000 7fa6c5c00640 1 --2- 172.21.15.32:0/37780910 >> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] conn(0x5651a4e87520 0x5651a4e89940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:58:38.638 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:38.636+0000 7fa6c5c00640 1 --2- 172.21.15.32:0/37780910 >> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] conn(0x5651a4e87520 0x5651a4e89940 crc :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:58:38.639 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:38.638+0000 7fa6c5c00640 1 -- 172.21.15.32:0/37780910 <== osd.1 v2:172.21.15.32:6824/1551241759 1 ==== osd_op_reply(59 mds0_openfiles.0 [create] v116'141 uv140 ondisk = -17 ((17) File exists)) ==== 160+0+0 (crc 0 0 0) 0x7fa6b400ab80 con 0x5651a4e87520 2024-08-03T20:58:38.639 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:38.638+0000 7fa6cd571f40 1 -- 172.21.15.32:0/37780910 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:60 12.6 12:654134d2:::mds0_openfiles.0:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e116) -- 0x5651a4e89f00 con 0x5651a4e87520 2024-08-03T20:58:38.640 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:38.639+0000 7fa6c5c00640 1 -- 172.21.15.32:0/37780910 <== osd.1 v2:172.21.15.32:6824/1551241759 2 ==== osd_op_reply(60 mds0_openfiles.0 [delete] v116'142 uv142 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7fa6b400ab80 con 0x5651a4e87520 2024-08-03T20:58:38.640 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:38.639+0000 7fa6cd571f40 1 -- 172.21.15.32:0/37780910 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:61 12.6 12:654134d2:::mds0_openfiles.0:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116) -- 0x5651a4e8a320 con 0x5651a4e87520 2024-08-03T20:58:38.641 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:38.640+0000 7fa6c5c00640 1 -- 172.21.15.32:0/37780910 <== osd.1 v2:172.21.15.32:6824/1551241759 3 ==== osd_op_reply(61 mds0_openfiles.0 [create] v116'143 uv143 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7fa6b400ab80 con 0x5651a4e87520 2024-08-03T20:58:38.642 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds0_openfiles.0:head# 2024-08-03T20:58:38.642 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:38.640+0000 7fa6cd571f40 1 -- 172.21.15.32:0/37780910 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:62 12.6 12:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116) -- 0x5651a4e8a320 con 0x5651a4e87520 2024-08-03T20:58:38.643 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:38.641+0000 7fa6c5c00640 1 -- 172.21.15.32:0/37780910 <== osd.1 v2:172.21.15.32:6824/1551241759 4 ==== osd_op_reply(62 mds0_openfiles.0 [omap-set-header] v116'144 uv144 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7fa6b400ab80 con 0x5651a4e87520 2024-08-03T20:58:38.643 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:38.642+0000 7fa6cd571f40 1 -- 172.21.15.32:0/37780910 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:63 12.e 12:717a0223:::608.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116) -- 0x5651a4e8a320 con 0x5651a4e682f0 2024-08-03T20:58:38.644 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:38.643+0000 7fa6c5c00640 1 -- 172.21.15.32:0/37780910 <== osd.0 v2:172.21.15.32:6816/3066156116 38 ==== osd_op_reply(63 608.00000000 [create] v116'183 uv182 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fa6b400ab80 con 0x5651a4e682f0 2024-08-03T20:58:38.644 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:38.643+0000 7fa6cd571f40 1 -- 172.21.15.32:0/37780910 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:64 12.e 12:717a0223:::608.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e116) -- 0x5651a4e8a320 con 0x5651a4e682f0 2024-08-03T20:58:38.646 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:38.644+0000 7fa6c5c00640 1 -- 172.21.15.32:0/37780910 <== osd.0 v2:172.21.15.32:6816/3066156116 39 ==== osd_op_reply(64 608.00000000 [delete] v116'184 uv184 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fa6b400ab80 con 0x5651a4e682f0 2024-08-03T20:58:38.646 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:38.645+0000 7fa6cd571f40 1 -- 172.21.15.32:0/37780910 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:65 12.e 12:717a0223:::608.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116) -- 0x5651a4e8a320 con 0x5651a4e682f0 2024-08-03T20:58:38.647 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:38.646+0000 7fa6c5c00640 1 -- 172.21.15.32:0/37780910 <== osd.0 v2:172.21.15.32:6816/3066156116 40 ==== osd_op_reply(65 608.00000000 [create] v116'185 uv185 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fa6b400ab80 con 0x5651a4e682f0 2024-08-03T20:58:38.647 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::608.00000000:head# 2024-08-03T20:58:38.647 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:38.646+0000 7fa6cd571f40 1 -- 172.21.15.32:0/37780910 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:66 12.e 12:717a0223:::608.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116) -- 0x5651a4e8b330 con 0x5651a4e682f0 2024-08-03T20:58:38.648 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:38.647+0000 7fa6c5c00640 1 -- 172.21.15.32:0/37780910 <== osd.0 v2:172.21.15.32:6816/3066156116 41 ==== osd_op_reply(66 608.00000000 [setxattr (30)] v116'186 uv186 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fa6b400ab80 con 0x5651a4e682f0 2024-08-03T20:58:38.649 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:38.647+0000 7fa6cd571f40 1 -- 172.21.15.32:0/37780910 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:67 12.e 12:717a0223:::608.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116) -- 0x5651a4e8b330 con 0x5651a4e682f0 2024-08-03T20:58:38.650 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:38.648+0000 7fa6c5c00640 1 -- 172.21.15.32:0/37780910 <== osd.0 v2:172.21.15.32:6816/3066156116 42 ==== osd_op_reply(67 608.00000000 [setxattr (62)] v116'187 uv187 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fa6b400ab80 con 0x5651a4e682f0 2024-08-03T20:58:38.650 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:38.648+0000 7fa6cd571f40 1 -- 172.21.15.32:0/37780910 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:68 12.e 12:717a0223:::608.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116) -- 0x5651a4e8a320 con 0x5651a4e682f0 2024-08-03T20:58:38.651 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:38.650+0000 7fa6c5c00640 1 -- 172.21.15.32:0/37780910 <== osd.0 v2:172.21.15.32:6816/3066156116 43 ==== osd_op_reply(68 608.00000000 [omap-set-header] v116'188 uv188 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fa6b400ab80 con 0x5651a4e682f0 2024-08-03T20:58:38.651 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:38.650+0000 7fa6cd571f40 1 --2- 172.21.15.32:0/37780910 >> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] conn(0x5651a4e8a830 0x5651a4e8cc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:58:38.651 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:38.650+0000 7fa6cd571f40 1 -- 172.21.15.32:0/37780910 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:69 12.15 12:a93a17c2:::604.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116) -- 0x5651a4e8d260 con 0x5651a4e8a830 2024-08-03T20:58:38.652 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:38.650+0000 7fa6c7000640 1 --2- 172.21.15.32:0/37780910 >> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] conn(0x5651a4e8a830 0x5651a4e8cc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:58:38.652 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:38.651+0000 7fa6c7000640 1 --2- 172.21.15.32:0/37780910 >> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] conn(0x5651a4e8a830 0x5651a4e8cc50 crc :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:58:38.654 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:38.652+0000 7fa6c7000640 1 -- 172.21.15.32:0/37780910 <== osd.6 v2:172.21.15.118:6825/801947386 1 ==== osd_op_reply(69 604.00000000 [create] v116'338 uv332 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fa6c0070070 con 0x5651a4e8a830 2024-08-03T20:58:38.654 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:38.652+0000 7fa6cd571f40 1 -- 172.21.15.32:0/37780910 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:70 12.15 12:a93a17c2:::604.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e116) -- 0x5651a4e8d750 con 0x5651a4e8a830 2024-08-03T20:58:38.655 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:38.654+0000 7fa6c7000640 1 -- 172.21.15.32:0/37780910 <== osd.6 v2:172.21.15.118:6825/801947386 2 ==== osd_op_reply(70 604.00000000 [delete] v116'339 uv339 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fa6c0070070 con 0x5651a4e8a830 2024-08-03T20:58:38.655 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:38.654+0000 7fa6cd571f40 1 -- 172.21.15.32:0/37780910 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:71 12.15 12:a93a17c2:::604.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116) -- 0x5651a4e8dc30 con 0x5651a4e8a830 2024-08-03T20:58:38.657 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::604.00000000:head# 2024-08-03T20:58:38.657 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:38.656+0000 7fa6c7000640 1 -- 172.21.15.32:0/37780910 <== osd.6 v2:172.21.15.118:6825/801947386 3 ==== osd_op_reply(71 604.00000000 [create] v116'340 uv340 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fa6c0070070 con 0x5651a4e8a830 2024-08-03T20:58:38.657 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:38.656+0000 7fa6cd571f40 1 -- 172.21.15.32:0/37780910 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:72 12.15 12:a93a17c2:::604.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116) -- 0x5651a4e8f4e0 con 0x5651a4e8a830 2024-08-03T20:58:38.658 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:38.657+0000 7fa6c7000640 1 -- 172.21.15.32:0/37780910 <== osd.6 v2:172.21.15.118:6825/801947386 4 ==== osd_op_reply(72 604.00000000 [setxattr (30)] v116'341 uv341 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fa6c0070070 con 0x5651a4e8a830 2024-08-03T20:58:38.658 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:38.657+0000 7fa6cd571f40 1 -- 172.21.15.32:0/37780910 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:73 12.15 12:a93a17c2:::604.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116) -- 0x5651a4e8f880 con 0x5651a4e8a830 2024-08-03T20:58:38.660 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:38.659+0000 7fa6c7000640 1 -- 172.21.15.32:0/37780910 <== osd.6 v2:172.21.15.118:6825/801947386 5 ==== osd_op_reply(73 604.00000000 [setxattr (62)] v116'342 uv342 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fa6c0070070 con 0x5651a4e8a830 2024-08-03T20:58:38.660 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:38.659+0000 7fa6cd571f40 1 -- 172.21.15.32:0/37780910 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:74 12.15 12:a93a17c2:::604.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116) -- 0x5651a4e8e4d0 con 0x5651a4e8a830 2024-08-03T20:58:38.661 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:38.660+0000 7fa6c7000640 1 -- 172.21.15.32:0/37780910 <== osd.6 v2:172.21.15.118:6825/801947386 6 ==== osd_op_reply(74 604.00000000 [omap-set-header] v116'343 uv343 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fa6c0070070 con 0x5651a4e8a830 2024-08-03T20:58:38.661 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:38.660+0000 7fa6cd571f40 1 -- 172.21.15.32:0/37780910 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:75 12.15 12:aa448500:::500.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116) -- 0x5651a4e8e8e0 con 0x5651a4e8a830 2024-08-03T20:58:38.663 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:38.661+0000 7fa6c7000640 1 -- 172.21.15.32:0/37780910 <== osd.6 v2:172.21.15.118:6825/801947386 7 ==== osd_op_reply(75 500.00000000 [create] v116'344 uv337 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fa6c0070070 con 0x5651a4e8a830 2024-08-03T20:58:38.663 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:38.661+0000 7fa6cd571f40 1 -- 172.21.15.32:0/37780910 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:76 12.15 12:aa448500:::500.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e116) -- 0x5651a4e8ecd0 con 0x5651a4e8a830 2024-08-03T20:58:38.664 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:38.663+0000 7fa6c7000640 1 -- 172.21.15.32:0/37780910 <== osd.6 v2:172.21.15.118:6825/801947386 8 ==== osd_op_reply(76 500.00000000 [delete] v116'345 uv345 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fa6c0070070 con 0x5651a4e8a830 2024-08-03T20:58:38.664 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:38.663+0000 7fa6cd571f40 1 -- 172.21.15.32:0/37780910 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:77 12.15 12:aa448500:::500.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116) -- 0x5651a4e8ecd0 con 0x5651a4e8a830 2024-08-03T20:58:38.665 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:38.664+0000 7fa6c7000640 1 -- 172.21.15.32:0/37780910 <== osd.6 v2:172.21.15.118:6825/801947386 9 ==== osd_op_reply(77 500.00000000 [create] v116'346 uv346 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fa6c0070070 con 0x5651a4e8a830 2024-08-03T20:58:38.665 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::500.00000000:head# 2024-08-03T20:58:38.666 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:38.664+0000 7fa6cd571f40 1 -- 172.21.15.32:0/37780910 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:78 12.15 12:aa448500:::500.00000000:head [write 0~90 in=90b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116) -- 0x5651a4e8ec80 con 0x5651a4e8a830 2024-08-03T20:58:38.670 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:38.666+0000 7fa6c7000640 1 -- 172.21.15.32:0/37780910 <== osd.6 v2:172.21.15.118:6825/801947386 10 ==== osd_op_reply(78 500.00000000 [write 0~90] v116'347 uv347 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fa6c0070070 con 0x5651a4e8a830 2024-08-03T20:58:38.670 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:38.666+0000 7fa6cd571f40 1 -- 172.21.15.32:0/37780910 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:79 12.15 12:aa448500:::500.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116) -- 0x5651a4e8ec80 con 0x5651a4e8a830 2024-08-03T20:58:38.670 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:38.667+0000 7fa6c7000640 1 -- 172.21.15.32:0/37780910 <== osd.6 v2:172.21.15.118:6825/801947386 11 ==== osd_op_reply(79 500.00000000 [omap-set-header] v116'348 uv348 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fa6c0070070 con 0x5651a4e8a830 2024-08-03T20:58:38.671 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:38.667+0000 7fa6cd571f40 1 --2- 172.21.15.32:0/37780910 >> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] conn(0x5651a4e8f040 0x5651a4e92140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:58:38.671 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:38.667+0000 7fa6cd571f40 1 -- 172.21.15.32:0/37780910 --> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] -- osd_op(unknown.0.0:80 12.d 12:b50e409b:::mds_snaptable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116) -- 0x5651a4e92680 con 0x5651a4e8f040 2024-08-03T20:58:38.671 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:38.667+0000 7fa6c6600640 1 --2- 172.21.15.32:0/37780910 >> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] conn(0x5651a4e8f040 0x5651a4e92140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:58:38.671 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:38.668+0000 7fa6c6600640 1 --2- 172.21.15.32:0/37780910 >> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] conn(0x5651a4e8f040 0x5651a4e92140 crc :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:58:38.671 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:38.669+0000 7fa6c6600640 1 -- 172.21.15.32:0/37780910 <== osd.7 v2:172.21.15.118:6818/653000048 1 ==== osd_op_reply(80 mds_snaptable [create] v116'153 uv152 ondisk = -17 ((17) File exists)) ==== 157+0+0 (crc 0 0 0) 0x7fa6b800e040 con 0x5651a4e8f040 2024-08-03T20:58:38.671 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:38.669+0000 7fa6cd571f40 1 -- 172.21.15.32:0/37780910 --> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] -- osd_op(unknown.0.0:81 12.d 12:b50e409b:::mds_snaptable:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e116) -- 0x5651a4e92680 con 0x5651a4e8f040 2024-08-03T20:58:38.672 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:38.671+0000 7fa6c6600640 1 -- 172.21.15.32:0/37780910 <== osd.7 v2:172.21.15.118:6818/653000048 2 ==== osd_op_reply(81 mds_snaptable [delete] v116'154 uv154 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7fa6b800e040 con 0x5651a4e8f040 2024-08-03T20:58:38.672 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:38.671+0000 7fa6cd571f40 1 -- 172.21.15.32:0/37780910 --> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] -- osd_op(unknown.0.0:82 12.d 12:b50e409b:::mds_snaptable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116) -- 0x5651a4e92680 con 0x5651a4e8f040 2024-08-03T20:58:38.673 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:38.672+0000 7fa6c6600640 1 -- 172.21.15.32:0/37780910 <== osd.7 v2:172.21.15.118:6818/653000048 3 ==== osd_op_reply(82 mds_snaptable [create] v116'155 uv155 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7fa6b800e040 con 0x5651a4e8f040 2024-08-03T20:58:38.673 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds_snaptable:head# 2024-08-03T20:58:38.674 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:38.672+0000 7fa6cd571f40 1 -- 172.21.15.32:0/37780910 --> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] -- osd_op(unknown.0.0:83 12.d 12:b50e409b:::mds_snaptable:head [write 0~70 in=70b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116) -- 0x5651a4e8e0a0 con 0x5651a4e8f040 2024-08-03T20:58:38.675 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:38.673+0000 7fa6c6600640 1 -- 172.21.15.32:0/37780910 <== osd.7 v2:172.21.15.118:6818/653000048 4 ==== osd_op_reply(83 mds_snaptable [write 0~70] v116'156 uv156 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7fa6b800e040 con 0x5651a4e8f040 2024-08-03T20:58:38.675 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:38.673+0000 7fa6cd571f40 1 -- 172.21.15.32:0/37780910 --> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] -- osd_op(unknown.0.0:84 12.d 12:b50e409b:::mds_snaptable:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116) -- 0x5651a4e8e060 con 0x5651a4e8f040 2024-08-03T20:58:38.676 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:38.675+0000 7fa6c6600640 1 -- 172.21.15.32:0/37780910 <== osd.7 v2:172.21.15.118:6818/653000048 5 ==== osd_op_reply(84 mds_snaptable [omap-set-header] v116'157 uv157 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7fa6b800e040 con 0x5651a4e8f040 2024-08-03T20:58:38.676 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:38.675+0000 7fa6cd571f40 1 -- 172.21.15.32:0/37780910 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:85 12.1d 12:b871830b:::605.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116) -- 0x5651a4e8e060 con 0x5651a4e682f0 2024-08-03T20:58:38.677 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:38.676+0000 7fa6c5c00640 1 -- 172.21.15.32:0/37780910 <== osd.0 v2:172.21.15.32:6816/3066156116 44 ==== osd_op_reply(85 605.00000000 [create] v116'183 uv182 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fa6b400ab80 con 0x5651a4e682f0 2024-08-03T20:58:38.678 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:38.676+0000 7fa6cd571f40 1 -- 172.21.15.32:0/37780910 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:86 12.1d 12:b871830b:::605.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e116) -- 0x5651a4e8e060 con 0x5651a4e682f0 2024-08-03T20:58:38.679 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:38.677+0000 7fa6c5c00640 1 -- 172.21.15.32:0/37780910 <== osd.0 v2:172.21.15.32:6816/3066156116 45 ==== osd_op_reply(86 605.00000000 [delete] v116'184 uv184 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fa6b400ab80 con 0x5651a4e682f0 2024-08-03T20:58:38.679 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:38.677+0000 7fa6cd571f40 1 -- 172.21.15.32:0/37780910 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:87 12.1d 12:b871830b:::605.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116) -- 0x5651a4e8e060 con 0x5651a4e682f0 2024-08-03T20:58:38.680 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::605.00000000:head# 2024-08-03T20:58:38.680 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:38.679+0000 7fa6c5c00640 1 -- 172.21.15.32:0/37780910 <== osd.0 v2:172.21.15.32:6816/3066156116 46 ==== osd_op_reply(87 605.00000000 [create] v116'185 uv185 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fa6b400ab80 con 0x5651a4e682f0 2024-08-03T20:58:38.680 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:38.679+0000 7fa6cd571f40 1 -- 172.21.15.32:0/37780910 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:88 12.1d 12:b871830b:::605.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116) -- 0x5651a4e8e060 con 0x5651a4e682f0 2024-08-03T20:58:38.681 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:38.680+0000 7fa6c5c00640 1 -- 172.21.15.32:0/37780910 <== osd.0 v2:172.21.15.32:6816/3066156116 47 ==== osd_op_reply(88 605.00000000 [setxattr (30)] v116'186 uv186 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fa6b400ab80 con 0x5651a4e682f0 2024-08-03T20:58:38.682 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:38.680+0000 7fa6cd571f40 1 -- 172.21.15.32:0/37780910 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:89 12.1d 12:b871830b:::605.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116) -- 0x5651a4e8e060 con 0x5651a4e682f0 2024-08-03T20:58:38.683 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:38.682+0000 7fa6c5c00640 1 -- 172.21.15.32:0/37780910 <== osd.0 v2:172.21.15.32:6816/3066156116 48 ==== osd_op_reply(89 605.00000000 [setxattr (62)] v116'187 uv187 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fa6b400ab80 con 0x5651a4e682f0 2024-08-03T20:58:38.683 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:38.682+0000 7fa6cd571f40 1 -- 172.21.15.32:0/37780910 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:90 12.1d 12:b871830b:::605.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116) -- 0x5651a4e8e060 con 0x5651a4e682f0 2024-08-03T20:58:38.684 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:38.683+0000 7fa6c5c00640 1 -- 172.21.15.32:0/37780910 <== osd.0 v2:172.21.15.32:6816/3066156116 49 ==== osd_op_reply(90 605.00000000 [omap-set-header] v116'188 uv188 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fa6b400ab80 con 0x5651a4e682f0 2024-08-03T20:58:38.684 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:38.683+0000 7fa6cd571f40 1 -- 172.21.15.32:0/37780910 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:91 12.13 12:c91d4a1d:::mds0_inotable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116) -- 0x5651a4e8e060 con 0x5651a4e87520 2024-08-03T20:58:38.686 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:38.684+0000 7fa6c5c00640 1 -- 172.21.15.32:0/37780910 <== osd.1 v2:172.21.15.32:6824/1551241759 5 ==== osd_op_reply(91 mds0_inotable [create] v116'304 uv298 ondisk = -17 ((17) File exists)) ==== 157+0+0 (crc 0 0 0) 0x7fa6b400ab80 con 0x5651a4e87520 2024-08-03T20:58:38.686 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:38.684+0000 7fa6cd571f40 1 -- 172.21.15.32:0/37780910 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:92 12.13 12:c91d4a1d:::mds0_inotable:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e116) -- 0x5651a4e8e060 con 0x5651a4e87520 2024-08-03T20:58:38.687 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:38.686+0000 7fa6c5c00640 1 -- 172.21.15.32:0/37780910 <== osd.1 v2:172.21.15.32:6824/1551241759 6 ==== osd_op_reply(92 mds0_inotable [delete] v116'305 uv305 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7fa6b400ab80 con 0x5651a4e87520 2024-08-03T20:58:38.687 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:38.686+0000 7fa6cd571f40 1 -- 172.21.15.32:0/37780910 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:93 12.13 12:c91d4a1d:::mds0_inotable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116) -- 0x5651a4e8e060 con 0x5651a4e87520 2024-08-03T20:58:38.688 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:38.687+0000 7fa6c5c00640 1 -- 172.21.15.32:0/37780910 <== osd.1 v2:172.21.15.32:6824/1551241759 7 ==== osd_op_reply(93 mds0_inotable [create] v116'306 uv306 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7fa6b400ab80 con 0x5651a4e87520 2024-08-03T20:58:38.689 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds0_inotable:head# 2024-08-03T20:58:38.689 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:38.687+0000 7fa6cd571f40 1 -- 172.21.15.32:0/37780910 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:94 12.13 12:c91d4a1d:::mds0_inotable:head [write 0~34 in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116) -- 0x5651a4e92b80 con 0x5651a4e87520 2024-08-03T20:58:38.690 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:38.689+0000 7fa6c5c00640 1 -- 172.21.15.32:0/37780910 <== osd.1 v2:172.21.15.32:6824/1551241759 8 ==== osd_op_reply(94 mds0_inotable [write 0~34] v116'307 uv307 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7fa6b400ab80 con 0x5651a4e87520 2024-08-03T20:58:38.690 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:38.689+0000 7fa6cd571f40 1 -- 172.21.15.32:0/37780910 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:95 12.13 12:c91d4a1d:::mds0_inotable:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116) -- 0x5651a4e92b80 con 0x5651a4e87520 2024-08-03T20:58:38.691 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:38.690+0000 7fa6c5c00640 1 -- 172.21.15.32:0/37780910 <== osd.1 v2:172.21.15.32:6824/1551241759 9 ==== osd_op_reply(95 mds0_inotable [omap-set-header] v116'308 uv308 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7fa6b400ab80 con 0x5651a4e87520 2024-08-03T20:58:38.691 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:38.690+0000 7fa6cd571f40 1 -- 172.21.15.32:0/37780910 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:96 12.13 12:cd5a64a3:::100.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116) -- 0x5651a4e92b80 con 0x5651a4e87520 2024-08-03T20:58:38.692 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:38.691+0000 7fa6c5c00640 1 -- 172.21.15.32:0/37780910 <== osd.1 v2:172.21.15.32:6824/1551241759 10 ==== osd_op_reply(96 100.00000000 [create] v116'309 uv303 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fa6b400ab80 con 0x5651a4e87520 2024-08-03T20:58:38.692 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:38.691+0000 7fa6cd571f40 1 -- 172.21.15.32:0/37780910 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:97 12.13 12:cd5a64a3:::100.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e116) -- 0x5651a4e92b80 con 0x5651a4e87520 2024-08-03T20:58:38.694 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:38.692+0000 7fa6c5c00640 1 -- 172.21.15.32:0/37780910 <== osd.1 v2:172.21.15.32:6824/1551241759 11 ==== osd_op_reply(97 100.00000000 [delete] v116'310 uv310 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fa6b400ab80 con 0x5651a4e87520 2024-08-03T20:58:38.694 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:38.693+0000 7fa6cd571f40 1 -- 172.21.15.32:0/37780910 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:98 12.13 12:cd5a64a3:::100.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116) -- 0x5651a4e92b80 con 0x5651a4e87520 2024-08-03T20:58:38.695 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::100.00000000:head# 2024-08-03T20:58:38.695 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:38.694+0000 7fa6c5c00640 1 -- 172.21.15.32:0/37780910 <== osd.1 v2:172.21.15.32:6824/1551241759 12 ==== osd_op_reply(98 100.00000000 [create] v116'311 uv311 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fa6b400ab80 con 0x5651a4e87520 2024-08-03T20:58:38.695 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:38.694+0000 7fa6cd571f40 1 -- 172.21.15.32:0/37780910 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:99 12.13 12:cd5a64a3:::100.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116) -- 0x5651a4e92b80 con 0x5651a4e87520 2024-08-03T20:58:38.696 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:38.695+0000 7fa6c5c00640 1 -- 172.21.15.32:0/37780910 <== osd.1 v2:172.21.15.32:6824/1551241759 13 ==== osd_op_reply(99 100.00000000 [omap-set-header] v116'312 uv312 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fa6b400ab80 con 0x5651a4e87520 2024-08-03T20:58:38.696 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:38.695+0000 7fa6cd571f40 1 -- 172.21.15.32:0/37780910 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:100 12.13 12:cd5a64a3:::100.00000000:head [omap-set-vals in=5194b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116) -- 0x5651a4e92f40 con 0x5651a4e87520 2024-08-03T20:58:38.698 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:38.697+0000 7fa6c5c00640 1 -- 172.21.15.32:0/37780910 <== osd.1 v2:172.21.15.32:6824/1551241759 14 ==== osd_op_reply(100 100.00000000 [omap-set-vals] v116'313 uv313 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fa6b400ab80 con 0x5651a4e87520 2024-08-03T20:58:38.698 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:38.697+0000 7fa6cd571f40 1 --2- 172.21.15.32:0/37780910 >> [v2:172.21.15.32:6800/9088303,v1:172.21.15.32:6801/9088303] conn(0x5651a4e93770 0x5651a4e95bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:58:38.698 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:38.697+0000 7fa6cd571f40 1 -- 172.21.15.32:0/37780910 --> [v2:172.21.15.32:6800/9088303,v1:172.21.15.32:6801/9088303] -- osd_op(unknown.0.0:101 12.b 12:d0630e4c:::mds0_sessionmap:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116) -- 0x5651a4e96100 con 0x5651a4e93770 2024-08-03T20:58:38.698 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:38.697+0000 7fa6c5c00640 1 --2- 172.21.15.32:0/37780910 >> [v2:172.21.15.32:6800/9088303,v1:172.21.15.32:6801/9088303] conn(0x5651a4e93770 0x5651a4e95bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:58:38.699 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:38.698+0000 7fa6c5c00640 1 --2- 172.21.15.32:0/37780910 >> [v2:172.21.15.32:6800/9088303,v1:172.21.15.32:6801/9088303] conn(0x5651a4e93770 0x5651a4e95bc0 crc :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:58:38.700 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:38.699+0000 7fa6c5c00640 1 -- 172.21.15.32:0/37780910 <== osd.3 v2:172.21.15.32:6800/9088303 1 ==== osd_op_reply(101 mds0_sessionmap [create] v116'132 uv131 ondisk = -17 ((17) File exists)) ==== 159+0+0 (crc 0 0 0) 0x7fa6b400ab80 con 0x5651a4e93770 2024-08-03T20:58:38.700 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:38.699+0000 7fa6cd571f40 1 -- 172.21.15.32:0/37780910 --> [v2:172.21.15.32:6800/9088303,v1:172.21.15.32:6801/9088303] -- osd_op(unknown.0.0:102 12.b 12:d0630e4c:::mds0_sessionmap:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e116) -- 0x5651a4e96100 con 0x5651a4e93770 2024-08-03T20:58:38.702 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:38.700+0000 7fa6c5c00640 1 -- 172.21.15.32:0/37780910 <== osd.3 v2:172.21.15.32:6800/9088303 2 ==== osd_op_reply(102 mds0_sessionmap [delete] v116'133 uv133 ondisk = 0) ==== 159+0+0 (crc 0 0 0) 0x7fa6b400ab80 con 0x5651a4e93770 2024-08-03T20:58:38.702 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:38.700+0000 7fa6cd571f40 1 -- 172.21.15.32:0/37780910 --> [v2:172.21.15.32:6800/9088303,v1:172.21.15.32:6801/9088303] -- osd_op(unknown.0.0:103 12.b 12:d0630e4c:::mds0_sessionmap:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116) -- 0x5651a4e96100 con 0x5651a4e93770 2024-08-03T20:58:38.703 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:38.702+0000 7fa6c5c00640 1 -- 172.21.15.32:0/37780910 <== osd.3 v2:172.21.15.32:6800/9088303 3 ==== osd_op_reply(103 mds0_sessionmap [create] v116'134 uv134 ondisk = 0) ==== 159+0+0 (crc 0 0 0) 0x7fa6b400ab80 con 0x5651a4e93770 2024-08-03T20:58:38.703 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds0_sessionmap:head# 2024-08-03T20:58:38.703 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:38.702+0000 7fa6cd571f40 1 -- 172.21.15.32:0/37780910 --> [v2:172.21.15.32:6800/9088303,v1:172.21.15.32:6801/9088303] -- osd_op(unknown.0.0:104 12.b 12:d0630e4c:::mds0_sessionmap:head [omap-set-header in=14b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116) -- 0x5651a4e96100 con 0x5651a4e93770 2024-08-03T20:58:38.704 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:38.703+0000 7fa6c5c00640 1 -- 172.21.15.32:0/37780910 <== osd.3 v2:172.21.15.32:6800/9088303 4 ==== osd_op_reply(104 mds0_sessionmap [omap-set-header] v116'135 uv135 ondisk = 0) ==== 159+0+0 (crc 0 0 0) 0x7fa6b400ab80 con 0x5651a4e93770 2024-08-03T20:58:38.704 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:38.703+0000 7fa6cd571f40 1 -- 172.21.15.32:0/37780910 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:105 12.7 12:e0b41d06:::609.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116) -- 0x5651a4e96100 con 0x5651a4e87520 2024-08-03T20:58:38.706 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:38.705+0000 7fa6c5c00640 1 -- 172.21.15.32:0/37780910 <== osd.1 v2:172.21.15.32:6824/1551241759 15 ==== osd_op_reply(105 609.00000000 [create] v116'183 uv182 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fa6b400ab80 con 0x5651a4e87520 2024-08-03T20:58:38.706 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:38.705+0000 7fa6cd571f40 1 -- 172.21.15.32:0/37780910 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:106 12.7 12:e0b41d06:::609.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e116) -- 0x5651a4e96100 con 0x5651a4e87520 2024-08-03T20:58:38.707 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:38.706+0000 7fa6c5c00640 1 -- 172.21.15.32:0/37780910 <== osd.1 v2:172.21.15.32:6824/1551241759 16 ==== osd_op_reply(106 609.00000000 [delete] v116'184 uv184 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fa6b400ab80 con 0x5651a4e87520 2024-08-03T20:58:38.708 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:38.706+0000 7fa6cd571f40 1 -- 172.21.15.32:0/37780910 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:107 12.7 12:e0b41d06:::609.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116) -- 0x5651a4e96100 con 0x5651a4e87520 2024-08-03T20:58:38.709 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::609.00000000:head# 2024-08-03T20:58:38.709 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:38.708+0000 7fa6c5c00640 1 -- 172.21.15.32:0/37780910 <== osd.1 v2:172.21.15.32:6824/1551241759 17 ==== osd_op_reply(107 609.00000000 [create] v116'185 uv185 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fa6b400ab80 con 0x5651a4e87520 2024-08-03T20:58:38.709 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:38.708+0000 7fa6cd571f40 1 -- 172.21.15.32:0/37780910 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:108 12.7 12:e0b41d06:::609.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116) -- 0x5651a4e97110 con 0x5651a4e87520 2024-08-03T20:58:38.710 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:38.709+0000 7fa6c5c00640 1 -- 172.21.15.32:0/37780910 <== osd.1 v2:172.21.15.32:6824/1551241759 18 ==== osd_op_reply(108 609.00000000 [setxattr (30)] v116'186 uv186 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fa6b400ab80 con 0x5651a4e87520 2024-08-03T20:58:38.710 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:38.709+0000 7fa6cd571f40 1 -- 172.21.15.32:0/37780910 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:109 12.7 12:e0b41d06:::609.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116) -- 0x5651a4e97110 con 0x5651a4e87520 2024-08-03T20:58:38.712 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:38.710+0000 7fa6c5c00640 1 -- 172.21.15.32:0/37780910 <== osd.1 v2:172.21.15.32:6824/1551241759 19 ==== osd_op_reply(109 609.00000000 [setxattr (62)] v116'187 uv187 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fa6b400ab80 con 0x5651a4e87520 2024-08-03T20:58:38.712 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:38.711+0000 7fa6cd571f40 1 -- 172.21.15.32:0/37780910 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:110 12.7 12:e0b41d06:::609.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116) -- 0x5651a4e96100 con 0x5651a4e87520 2024-08-03T20:58:38.713 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:38.712+0000 7fa6c5c00640 1 -- 172.21.15.32:0/37780910 <== osd.1 v2:172.21.15.32:6824/1551241759 20 ==== osd_op_reply(110 609.00000000 [omap-set-header] v116'188 uv188 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fa6b400ab80 con 0x5651a4e87520 2024-08-03T20:58:38.713 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:38.712+0000 7fa6cd571f40 1 -- 172.21.15.32:0/37780910 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:111 12.f 12:f1f69726:::400.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116) -- 0x5651a4e96100 con 0x5651a4e8a830 2024-08-03T20:58:38.715 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:38.714+0000 7fa6c7000640 1 -- 172.21.15.32:0/37780910 <== osd.6 v2:172.21.15.118:6825/801947386 12 ==== osd_op_reply(111 400.00000000 [create] v116'156 uv155 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fa6c0070070 con 0x5651a4e8a830 2024-08-03T20:58:38.715 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:38.714+0000 7fa6cd571f40 1 -- 172.21.15.32:0/37780910 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:112 12.f 12:f1f69726:::400.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e116) -- 0x5651a4e96100 con 0x5651a4e8a830 2024-08-03T20:58:38.716 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:38.715+0000 7fa6c7000640 1 -- 172.21.15.32:0/37780910 <== osd.6 v2:172.21.15.118:6825/801947386 13 ==== osd_op_reply(112 400.00000000 [delete] v116'157 uv157 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fa6c0070070 con 0x5651a4e8a830 2024-08-03T20:58:38.716 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:38.715+0000 7fa6cd571f40 1 -- 172.21.15.32:0/37780910 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:113 12.f 12:f1f69726:::400.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116) -- 0x5651a4e96100 con 0x5651a4e8a830 2024-08-03T20:58:38.718 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::400.00000000:head# 2024-08-03T20:58:38.718 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:38.717+0000 7fa6c7000640 1 -- 172.21.15.32:0/37780910 <== osd.6 v2:172.21.15.118:6825/801947386 14 ==== osd_op_reply(113 400.00000000 [create] v116'158 uv158 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fa6c0070070 con 0x5651a4e8a830 2024-08-03T20:58:38.718 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:38.717+0000 7fa6cd571f40 1 -- 172.21.15.32:0/37780910 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:114 12.f 12:f1f69726:::400.00000000:head [write 0~22 in=22b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116) -- 0x5651a4e96100 con 0x5651a4e8a830 2024-08-03T20:58:38.720 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:38.718+0000 7fa6c7000640 1 -- 172.21.15.32:0/37780910 <== osd.6 v2:172.21.15.118:6825/801947386 15 ==== osd_op_reply(114 400.00000000 [write 0~22] v116'159 uv159 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fa6c0070070 con 0x5651a4e8a830 2024-08-03T20:58:38.720 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:38.718+0000 7fa6cd571f40 1 -- 172.21.15.32:0/37780910 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:115 12.f 12:f1f69726:::400.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116) -- 0x5651a4e96100 con 0x5651a4e8a830 2024-08-03T20:58:38.721 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:38.720+0000 7fa6c7000640 1 -- 172.21.15.32:0/37780910 <== osd.6 v2:172.21.15.118:6825/801947386 16 ==== osd_op_reply(115 400.00000000 [omap-set-header] v116'160 uv160 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fa6c0070070 con 0x5651a4e8a830 2024-08-03T20:58:38.721 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:38.720+0000 7fa6cd571f40 1 -- 172.21.15.32:0/37780910 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:116 12.1f 12:fe07bba1:::100.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116) -- 0x5651a4e96100 con 0x5651a4e8a830 2024-08-03T20:58:38.723 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:38.722+0000 7fa6c7000640 1 -- 172.21.15.32:0/37780910 <== osd.6 v2:172.21.15.118:6825/801947386 17 ==== osd_op_reply(116 100.00000000.inode [create] v116'376 uv375 ondisk = -17 ((17) File exists)) ==== 162+0+0 (crc 0 0 0) 0x7fa6c0070070 con 0x5651a4e8a830 2024-08-03T20:58:38.723 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:38.722+0000 7fa6cd571f40 1 -- 172.21.15.32:0/37780910 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:117 12.1f 12:fe07bba1:::100.00000000.inode:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e116) -- 0x5651a4e96100 con 0x5651a4e8a830 2024-08-03T20:58:38.724 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:38.723+0000 7fa6c7000640 1 -- 172.21.15.32:0/37780910 <== osd.6 v2:172.21.15.118:6825/801947386 18 ==== osd_op_reply(117 100.00000000.inode [delete] v116'377 uv377 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7fa6c0070070 con 0x5651a4e8a830 2024-08-03T20:58:38.724 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:38.723+0000 7fa6cd571f40 1 -- 172.21.15.32:0/37780910 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:118 12.1f 12:fe07bba1:::100.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116) -- 0x5651a4e96100 con 0x5651a4e8a830 2024-08-03T20:58:38.726 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::100.00000000.inode:head# 2024-08-03T20:58:38.726 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:38.724+0000 7fa6c7000640 1 -- 172.21.15.32:0/37780910 <== osd.6 v2:172.21.15.118:6825/801947386 19 ==== osd_op_reply(118 100.00000000.inode [create] v116'378 uv378 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7fa6c0070070 con 0x5651a4e8a830 2024-08-03T20:58:38.726 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:38.724+0000 7fa6cd571f40 1 -- 172.21.15.32:0/37780910 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:119 12.1f 12:fe07bba1:::100.00000000.inode:head [write 0~582 in=582b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116) -- 0x5651a4e96100 con 0x5651a4e8a830 2024-08-03T20:58:38.727 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:38.726+0000 7fa6c7000640 1 -- 172.21.15.32:0/37780910 <== osd.6 v2:172.21.15.118:6825/801947386 20 ==== osd_op_reply(119 100.00000000.inode [write 0~582] v116'379 uv379 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7fa6c0070070 con 0x5651a4e8a830 2024-08-03T20:58:38.727 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:38.726+0000 7fa6cd571f40 1 -- 172.21.15.32:0/37780910 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:120 12.1f 12:fe07bba1:::100.00000000.inode:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116) -- 0x5651a4e96100 con 0x5651a4e8a830 2024-08-03T20:58:38.729 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:38.727+0000 7fa6c7000640 1 -- 172.21.15.32:0/37780910 <== osd.6 v2:172.21.15.118:6825/801947386 21 ==== osd_op_reply(120 100.00000000.inode [omap-set-header] v116'380 uv380 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7fa6c0070070 con 0x5651a4e8a830 2024-08-03T20:58:38.729 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:38.727+0000 7fa6cd571f40 1 -- 172.21.15.32:0/37780910 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:121 12.1f 12:ff5b34d6:::1.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116) -- 0x5651a4e96100 con 0x5651a4e8a830 2024-08-03T20:58:38.730 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-08-03T20:58:38.731 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:38.729+0000 7fa6c7000640 1 -- 172.21.15.32:0/37780910 <== osd.6 v2:172.21.15.118:6825/801947386 22 ==== osd_op_reply(121 1.00000000 [create] v116'381 uv374 ondisk = -17 ((17) File exists)) ==== 154+0+0 (crc 0 0 0) 0x7fa6c0070070 con 0x5651a4e8a830 2024-08-03T20:58:38.731 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:38.729+0000 7fa6cd571f40 1 -- 172.21.15.32:0/37780910 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:122 12.1f 12:ff5b34d6:::1.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e116) -- 0x5651a4e96100 con 0x5651a4e8a830 2024-08-03T20:58:38.732 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:38.730+0000 7fa6c7000640 1 -- 172.21.15.32:0/37780910 <== osd.6 v2:172.21.15.118:6825/801947386 23 ==== osd_op_reply(122 1.00000000 [delete] v116'382 uv382 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7fa6c0070070 con 0x5651a4e8a830 2024-08-03T20:58:38.732 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:38.730+0000 7fa6cd571f40 1 -- 172.21.15.32:0/37780910 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:123 12.1f 12:ff5b34d6:::1.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116) -- 0x5651a4e96100 con 0x5651a4e8a830 2024-08-03T20:58:38.735 DEBUG:teuthology.orchestra.run.smithi118:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-08-03T20:58:38.736 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::1.00000000:head# 2024-08-03T20:58:38.737 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:38.731+0000 7fa6c7000640 1 -- 172.21.15.32:0/37780910 <== osd.6 v2:172.21.15.118:6825/801947386 24 ==== osd_op_reply(123 1.00000000 [create] v116'383 uv383 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7fa6c0070070 con 0x5651a4e8a830 2024-08-03T20:58:38.737 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:38.731+0000 7fa6cd571f40 1 -- 172.21.15.32:0/37780910 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:124 12.1f 12:ff5b34d6:::1.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116) -- 0x5651a4e97110 con 0x5651a4e8a830 2024-08-03T20:58:38.737 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:38.733+0000 7fa6c7000640 1 -- 172.21.15.32:0/37780910 <== osd.6 v2:172.21.15.118:6825/801947386 25 ==== osd_op_reply(124 1.00000000 [setxattr (30)] v116'384 uv384 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7fa6c0070070 con 0x5651a4e8a830 2024-08-03T20:58:38.737 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:38.733+0000 7fa6cd571f40 1 -- 172.21.15.32:0/37780910 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:125 12.1f 12:ff5b34d6:::1.00000000:head [setxattr parent (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116) -- 0x5651a4e97110 con 0x5651a4e8a830 2024-08-03T20:58:38.737 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:38.734+0000 7fa6c7000640 1 -- 172.21.15.32:0/37780910 <== osd.6 v2:172.21.15.118:6825/801947386 26 ==== osd_op_reply(125 1.00000000 [setxattr (30)] v116'385 uv385 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7fa6c0070070 con 0x5651a4e8a830 2024-08-03T20:58:38.737 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:38.734+0000 7fa6cd571f40 1 -- 172.21.15.32:0/37780910 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:126 12.1f 12:ff5b34d6:::1.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116) -- 0x5651a4e96100 con 0x5651a4e8a830 2024-08-03T20:58:38.737 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:38.736+0000 7fa6c7000640 1 -- 172.21.15.32:0/37780910 <== osd.6 v2:172.21.15.118:6825/801947386 27 ==== osd_op_reply(126 1.00000000 [omap-set-header] v116'386 uv386 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7fa6c0070070 con 0x5651a4e8a830 2024-08-03T20:58:38.738 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:38.736+0000 7fa6cd571f40 1 -- 172.21.15.32:0/37780910 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:127 12.1f 12:ff5b34d6:::1.00000000:head [omap-set-vals in=584b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116) -- 0x5651a4e969c0 con 0x5651a4e8a830 2024-08-03T20:58:38.739 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:38.737+0000 7fa6c7000640 1 -- 172.21.15.32:0/37780910 <== osd.6 v2:172.21.15.118:6825/801947386 28 ==== osd_op_reply(127 1.00000000 [omap-set-vals] v116'387 uv387 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7fa6c0070070 con 0x5651a4e8a830 2024-08-03T20:58:38.739 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:38.737+0000 7fa6cd571f40 1 -- 172.21.15.32:0/37780910 >> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] conn(0x5651a4e682f0 msgr2=0x5651a4e68700 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:58:38.740 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:38.737+0000 7fa6cd571f40 1 --2- 172.21.15.32:0/37780910 >> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] conn(0x5651a4e682f0 0x5651a4e68700 crc :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:38.740 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:38.737+0000 7fa6cd571f40 1 -- 172.21.15.32:0/37780910 >> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] conn(0x5651a4e87520 msgr2=0x5651a4e89940 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:58:38.740 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:38.737+0000 7fa6cd571f40 1 --2- 172.21.15.32:0/37780910 >> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] conn(0x5651a4e87520 0x5651a4e89940 crc :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:38.740 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:38.737+0000 7fa6cd571f40 1 -- 172.21.15.32:0/37780910 >> [v2:172.21.15.32:6800/9088303,v1:172.21.15.32:6801/9088303] conn(0x5651a4e93770 msgr2=0x5651a4e95bc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:58:38.740 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:38.737+0000 7fa6cd571f40 1 --2- 172.21.15.32:0/37780910 >> [v2:172.21.15.32:6800/9088303,v1:172.21.15.32:6801/9088303] conn(0x5651a4e93770 0x5651a4e95bc0 crc :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:38.740 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:38.737+0000 7fa6cd571f40 1 -- 172.21.15.32:0/37780910 >> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] conn(0x5651a4e82c40 msgr2=0x5651a4e85060 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:58:38.740 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:38.737+0000 7fa6cd571f40 1 --2- 172.21.15.32:0/37780910 >> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] conn(0x5651a4e82c40 0x5651a4e85060 crc :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:38.740 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:38.737+0000 7fa6cd571f40 1 -- 172.21.15.32:0/37780910 >> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] conn(0x5651a4e8a830 msgr2=0x5651a4e8cc50 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:58:38.740 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:38.737+0000 7fa6cd571f40 1 --2- 172.21.15.32:0/37780910 >> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] conn(0x5651a4e8a830 0x5651a4e8cc50 crc :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:38.740 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:38.737+0000 7fa6cd571f40 1 -- 172.21.15.32:0/37780910 >> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] conn(0x5651a4e8f040 msgr2=0x5651a4e92140 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:58:38.741 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:38.737+0000 7fa6cd571f40 1 --2- 172.21.15.32:0/37780910 >> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] conn(0x5651a4e8f040 0x5651a4e92140 crc :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:38.741 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:38.737+0000 7fa6c7000640 1 -- 172.21.15.32:0/37780910 reap_dead start 2024-08-03T20:58:38.741 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:38.737+0000 7fa6c7000640 1 -- 172.21.15.32:0/37780910 reap_dead start 2024-08-03T20:58:38.741 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:38.737+0000 7fa6c7000640 1 -- 172.21.15.32:0/37780910 reap_dead start 2024-08-03T20:58:38.741 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:38.737+0000 7fa6cd571f40 1 -- 172.21.15.32:0/37780910 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fa6a80759f0 msgr2=0x7fa6a8077f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:58:38.741 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:38.737+0000 7fa6cd571f40 1 --2- 172.21.15.32:0/37780910 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fa6a80759f0 0x7fa6a8077f00 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 crypto rx=0x5651a4de0780 tx=0x7fa6b8001450 comp rx=0 tx=0).stop 2024-08-03T20:58:38.741 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:38.737+0000 7fa6cd571f40 1 -- 172.21.15.32:0/37780910 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5651a4dd7330 msgr2=0x5651a4ddf4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:58:38.741 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:38.737+0000 7fa6cd571f40 1 --2- 172.21.15.32:0/37780910 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5651a4dd7330 0x5651a4ddf4c0 secure :-1 s=READY pgs=1822 cs=0 l=1 rev1=1 crypto rx=0x7fa6c0025db0 tx=0x7fa6c004d610 comp rx=0 tx=0).stop 2024-08-03T20:58:38.741 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:38.738+0000 7fa6cd571f40 1 -- 172.21.15.32:0/37780910 shutdown_connections 2024-08-03T20:58:38.741 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:38.738+0000 7fa6cd571f40 1 --2- 172.21.15.32:0/37780910 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fa6a80759f0 0x7fa6a8077f00 unknown :-1 s=CLOSED pgs=740 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:38.741 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:38.738+0000 7fa6cd571f40 1 --2- 172.21.15.32:0/37780910 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5651a4dd7330 0x5651a4ddf4c0 unknown :-1 s=CLOSED pgs=1822 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:38.741 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:38.738+0000 7fa6cd571f40 1 -- 172.21.15.32:0/37780910 >> 172.21.15.32:0/37780910 conn(0x5651a4c29ac0 msgr2=0x5651a4d4a250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:58:38.742 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:38.738+0000 7fa6cd571f40 1 -- 172.21.15.32:0/37780910 shutdown_connections 2024-08-03T20:58:38.742 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:38.738+0000 7fa6cd571f40 1 -- 172.21.15.32:0/37780910 wait complete. 2024-08-03T20:58:38.750 DEBUG:teuthology.orchestra.run.smithi032:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata setomapval 10000000000.00000000 sixmegs_head deadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeef 2024-08-03T20:58:38.936 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:38.934+0000 7f8bcc11bf40 1 Processor -- start 2024-08-03T20:58:38.936 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:38.934+0000 7f8bcc11bf40 1 -- start start 2024-08-03T20:58:38.938 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:38.936+0000 7f8bcc11bf40 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55f443b5c740 0x55f443bc6e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:58:38.938 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:38.936+0000 7f8bcc11bf40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55f443bce3f0 0x55f443b11a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:58:38.939 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:38.936+0000 7f8bcc11bf40 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55f443be9d90 0x55f443bc81a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:58:38.939 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:38.936+0000 7f8bcc11bf40 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55f443a61e30 con 0x55f443be9d90 2024-08-03T20:58:38.939 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:38.936+0000 7f8bcc11bf40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x55f443abfbf0 con 0x55f443bce3f0 2024-08-03T20:58:38.939 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:38.936+0000 7f8bcc11bf40 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x55f443b12540 con 0x55f443b5c740 2024-08-03T20:58:38.939 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:38.936+0000 7f8bc5200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55f443be9d90 0x55f443bc81a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:58:38.939 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:38.936+0000 7f8bc5200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55f443be9d90 0x55f443bc81a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:41774/0 (socket says 172.21.15.32:41774) 2024-08-03T20:58:38.939 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:38.936+0000 7f8bc5200640 1 -- 172.21.15.32:0/2143739196 learned_addr learned my addr 172.21.15.32:0/2143739196 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:58:38.939 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:38.936+0000 7f8bc3e00640 1 --2- 172.21.15.32:0/2143739196 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55f443bce3f0 0x55f443b11a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:58:38.939 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:38.936+0000 7f8bc4800640 1 --2- 172.21.15.32:0/2143739196 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55f443b5c740 0x55f443bc6e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:58:38.939 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:38.936+0000 7f8bc5200640 1 -- 172.21.15.32:0/2143739196 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55f443b5c740 msgr2=0x55f443bc6e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:58:38.939 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:38.936+0000 7f8bc5200640 1 --2- 172.21.15.32:0/2143739196 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55f443b5c740 0x55f443bc6e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:38.939 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:38.936+0000 7f8bc5200640 1 -- 172.21.15.32:0/2143739196 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55f443bce3f0 msgr2=0x55f443b11a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:58:38.939 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:38.936+0000 7f8bc5200640 1 --2- 172.21.15.32:0/2143739196 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55f443bce3f0 0x55f443b11a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:38.940 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:38.936+0000 7f8bc5200640 1 -- 172.21.15.32:0/2143739196 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f443be8d10 con 0x55f443be9d90 2024-08-03T20:58:38.940 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:38.936+0000 7f8bc3e00640 1 --2- 172.21.15.32:0/2143739196 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55f443bce3f0 0x55f443b11a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-03T20:58:38.940 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:38.937+0000 7f8bc5200640 1 --2- 172.21.15.32:0/2143739196 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55f443be9d90 0x55f443bc81a0 secure :-1 s=READY pgs=1824 cs=0 l=1 rev1=1 crypto rx=0x7f8bbc00a4e0 tx=0x7f8bbc00a9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9550936de4be4c80 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:58:38.940 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:38.937+0000 7f8bc2a00640 1 -- 172.21.15.32:0/2143739196 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8bbc00f7b0 con 0x55f443be9d90 2024-08-03T20:58:38.940 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:38.937+0000 7f8bc2a00640 1 -- 172.21.15.32:0/2143739196 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f8bbc015070 con 0x55f443be9d90 2024-08-03T20:58:38.940 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:38.937+0000 7f8bc2a00640 1 -- 172.21.15.32:0/2143739196 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8bbc00fd00 con 0x55f443be9d90 2024-08-03T20:58:38.941 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:38.938+0000 7f8bcc11bf40 1 -- 172.21.15.32:0/2143739196 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55f443be9d90 msgr2=0x55f443bc81a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:58:38.941 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:38.938+0000 7f8bcc11bf40 1 --2- 172.21.15.32:0/2143739196 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55f443be9d90 0x55f443bc81a0 secure :-1 s=READY pgs=1824 cs=0 l=1 rev1=1 crypto rx=0x7f8bbc00a4e0 tx=0x7f8bbc00a9a0 comp rx=0 tx=0).stop 2024-08-03T20:58:38.941 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:38.938+0000 7f8bcc11bf40 1 -- 172.21.15.32:0/2143739196 shutdown_connections 2024-08-03T20:58:38.941 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:38.938+0000 7f8bcc11bf40 1 --2- 172.21.15.32:0/2143739196 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55f443bce3f0 0x55f443b11a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:38.941 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:38.938+0000 7f8bcc11bf40 1 --2- 172.21.15.32:0/2143739196 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55f443be9d90 0x55f443bc81a0 unknown :-1 s=CLOSED pgs=1824 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:38.941 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:38.938+0000 7f8bcc11bf40 1 --2- 172.21.15.32:0/2143739196 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55f443b5c740 0x55f443bc6e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:38.941 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:38.938+0000 7f8bcc11bf40 1 -- 172.21.15.32:0/2143739196 >> 172.21.15.32:0/2143739196 conn(0x55f443a33ac0 msgr2=0x55f443be3170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:58:38.941 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:38.938+0000 7f8bcc11bf40 1 -- 172.21.15.32:0/2143739196 shutdown_connections 2024-08-03T20:58:38.942 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:38.938+0000 7f8bcc11bf40 1 -- 172.21.15.32:0/2143739196 wait complete. 2024-08-03T20:58:38.942 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:38.939+0000 7f8bcc11bf40 1 Processor -- start 2024-08-03T20:58:38.942 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:38.939+0000 7f8bcc11bf40 1 -- start start 2024-08-03T20:58:38.944 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:38.940+0000 7f8bcc11bf40 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55f443b5c740 0x55f443bc3c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:58:38.944 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:38.940+0000 7f8bcc11bf40 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55f443bce3f0 0x55f443bcae80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:58:38.944 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:38.941+0000 7f8bcc11bf40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55f443be9d90 0x55f443bc9e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:58:38.944 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:38.941+0000 7f8bcc11bf40 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55f443bc79a0 con 0x55f443b5c740 2024-08-03T20:58:38.944 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:38.941+0000 7f8bcc11bf40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x55f443bc76c0 con 0x55f443be9d90 2024-08-03T20:58:38.944 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:38.941+0000 7f8bcc11bf40 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x55f443bc7c80 con 0x55f443bce3f0 2024-08-03T20:58:38.944 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:38.941+0000 7f8bc4800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55f443b5c740 0x55f443bc3c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:58:38.944 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:38.941+0000 7f8bc4800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55f443b5c740 0x55f443bc3c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:41780/0 (socket says 172.21.15.32:41780) 2024-08-03T20:58:38.944 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:38.941+0000 7f8bc4800640 1 -- 172.21.15.32:0/1302372019 learned_addr learned my addr 172.21.15.32:0/1302372019 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:58:38.944 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:38.941+0000 7f8bc5200640 1 --2- 172.21.15.32:0/1302372019 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55f443be9d90 0x55f443bc9e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:58:38.944 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:38.941+0000 7f8bc3e00640 1 --2- 172.21.15.32:0/1302372019 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55f443bce3f0 0x55f443bcae80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:58:38.945 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:38.941+0000 7f8bc4800640 1 -- 172.21.15.32:0/1302372019 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55f443bce3f0 msgr2=0x55f443bcae80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:58:38.945 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:38.941+0000 7f8bc4800640 1 --2- 172.21.15.32:0/1302372019 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55f443bce3f0 0x55f443bcae80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:38.945 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:38.941+0000 7f8bc4800640 1 -- 172.21.15.32:0/1302372019 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55f443be9d90 msgr2=0x55f443bc9e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:58:38.945 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:38.941+0000 7f8bc4800640 1 --2- 172.21.15.32:0/1302372019 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55f443be9d90 0x55f443bc9e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:38.945 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:38.941+0000 7f8bc4800640 1 -- 172.21.15.32:0/1302372019 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f443bc4140 con 0x55f443b5c740 2024-08-03T20:58:38.945 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:38.941+0000 7f8bc3e00640 1 --2- 172.21.15.32:0/1302372019 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55f443bce3f0 0x55f443bcae80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-03T20:58:38.945 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:38.941+0000 7f8bc5200640 1 --2- 172.21.15.32:0/1302372019 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55f443be9d90 0x55f443bc9e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-03T20:58:38.945 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:38.942+0000 7f8bc4800640 1 --2- 172.21.15.32:0/1302372019 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55f443b5c740 0x55f443bc3c00 secure :-1 s=READY pgs=1825 cs=0 l=1 rev1=1 crypto rx=0x7f8bb800a680 tx=0x7f8bb800ab40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:58:38.945 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:38.942+0000 7f8bc1600640 1 -- 172.21.15.32:0/1302372019 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8bb801a070 con 0x55f443b5c740 2024-08-03T20:58:38.945 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:38.942+0000 7f8bc1600640 1 -- 172.21.15.32:0/1302372019 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f8bb8008990 con 0x55f443b5c740 2024-08-03T20:58:38.945 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:38.942+0000 7f8bcc11bf40 1 -- 172.21.15.32:0/1302372019 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x55f443be26f0 con 0x55f443b5c740 2024-08-03T20:58:38.945 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:38.942+0000 7f8bc1600640 1 -- 172.21.15.32:0/1302372019 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8bb8008de0 con 0x55f443b5c740 2024-08-03T20:58:38.945 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:38.942+0000 7f8bcc11bf40 1 -- 172.21.15.32:0/1302372019 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f443be2cb0 con 0x55f443b5c740 2024-08-03T20:58:38.946 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:38.944+0000 7f8bc1600640 1 -- 172.21.15.32:0/1302372019 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f8bb801d070 con 0x55f443b5c740 2024-08-03T20:58:38.946 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:38.945+0000 7f8bc1600640 1 --2- 172.21.15.32:0/1302372019 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f8ba40759f0 0x7f8ba4077f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:58:38.947 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:38.945+0000 7f8bc1600640 1 -- 172.21.15.32:0/1302372019 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(116..116 src has 1..116) ==== 8520+0+0 (secure 0 0 0) 0x7f8bb8097bd0 con 0x55f443b5c740 2024-08-03T20:58:38.947 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:38.946+0000 7f8bc3e00640 1 --2- 172.21.15.32:0/1302372019 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f8ba40759f0 0x7f8ba4077f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:58:38.947 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:38.946+0000 7f8bcc11bf40 1 --2- 172.21.15.32:0/1302372019 >> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] conn(0x55f443b59ac0 0x55f443b59ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:58:38.947 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:38.946+0000 7f8bcc11bf40 1 -- 172.21.15.32:0/1302372019 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:1 12.10 12:0cf6ad0f:::10000000000.00000000:head [omap-set-vals in=104b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116) -- 0x55f443b14c00 con 0x55f443b59ac0 2024-08-03T20:58:38.947 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:38.946+0000 7f8bc5200640 1 --2- 172.21.15.32:0/1302372019 >> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] conn(0x55f443b59ac0 0x55f443b59ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:58:38.948 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:38.946+0000 7f8bc3e00640 1 --2- 172.21.15.32:0/1302372019 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f8ba40759f0 0x7f8ba4077f00 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 crypto rx=0x7f8bb0055000 tx=0x7f8bb0058040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:58:38.948 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:38.947+0000 7f8bc5200640 1 --2- 172.21.15.32:0/1302372019 >> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] conn(0x55f443b59ac0 0x55f443b59ed0 crc :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:58:38.951 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:38.949+0000 7f8bc5200640 1 -- 172.21.15.32:0/1302372019 <== osd.0 v2:172.21.15.32:6816/3066156116 1 ==== osd_op_reply(1 10000000000.00000000 [omap-set-vals] v116'970 uv970 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f8bbc00e070 con 0x55f443b59ac0 2024-08-03T20:58:38.951 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:38.949+0000 7f8bcc11bf40 1 -- 172.21.15.32:0/1302372019 >> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] conn(0x55f443b59ac0 msgr2=0x55f443b59ed0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:58:38.951 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:38.949+0000 7f8bcc11bf40 1 --2- 172.21.15.32:0/1302372019 >> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] conn(0x55f443b59ac0 0x55f443b59ed0 crc :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:38.951 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:38.949+0000 7f8bcc11bf40 1 -- 172.21.15.32:0/1302372019 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f8ba40759f0 msgr2=0x7f8ba4077f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:58:38.951 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:38.949+0000 7f8bcc11bf40 1 --2- 172.21.15.32:0/1302372019 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f8ba40759f0 0x7f8ba4077f00 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 crypto rx=0x7f8bb0055000 tx=0x7f8bb0058040 comp rx=0 tx=0).stop 2024-08-03T20:58:38.951 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:38.949+0000 7f8bcc11bf40 1 -- 172.21.15.32:0/1302372019 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55f443b5c740 msgr2=0x55f443bc3c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:58:38.951 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:38.949+0000 7f8bcc11bf40 1 --2- 172.21.15.32:0/1302372019 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55f443b5c740 0x55f443bc3c00 secure :-1 s=READY pgs=1825 cs=0 l=1 rev1=1 crypto rx=0x7f8bb800a680 tx=0x7f8bb800ab40 comp rx=0 tx=0).stop 2024-08-03T20:58:38.951 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:38.949+0000 7f8bc5200640 1 -- 172.21.15.32:0/1302372019 reap_dead start 2024-08-03T20:58:38.951 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:38.949+0000 7f8bcc11bf40 1 -- 172.21.15.32:0/1302372019 shutdown_connections 2024-08-03T20:58:38.951 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:38.949+0000 7f8bcc11bf40 1 -- 172.21.15.32:0/1302372019 >> 172.21.15.32:0/1302372019 conn(0x55f443a33ac0 msgr2=0x55f443be3170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:58:38.951 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:38.949+0000 7f8bcc11bf40 1 -- 172.21.15.32:0/1302372019 shutdown_connections 2024-08-03T20:58:38.951 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:38.949+0000 7f8bcc11bf40 1 -- 172.21.15.32:0/1302372019 wait complete. 2024-08-03T20:58:38.961 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs joinable true 2024-08-03T20:58:39.195 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:39.193+0000 7f3843a00640 1 -- 172.21.15.32:0/3631688529 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3824003c10 msgr2=0x7f3824004060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:58:39.195 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:39.193+0000 7f3843a00640 1 --2- 172.21.15.32:0/3631688529 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3824003c10 0x7f3824004060 secure :-1 s=READY pgs=1827 cs=0 l=1 rev1=1 crypto rx=0x7f3830009b50 tx=0x7f3824005830 comp rx=0 tx=0).stop 2024-08-03T20:58:39.195 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:39.193+0000 7f3841a00640 1 -- 172.21.15.32:0/3631688529 reap_dead start 2024-08-03T20:58:39.196 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:39.193+0000 7f3843a00640 1 -- 172.21.15.32:0/3631688529 shutdown_connections 2024-08-03T20:58:39.196 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:39.193+0000 7f3843a00640 1 -- 172.21.15.32:0/3631688529 >> 172.21.15.32:0/3631688529 conn(0x7f383c0b1bd0 msgr2=0x7f383c0b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:58:39.196 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:39.193+0000 7f3843a00640 1 -- 172.21.15.32:0/3631688529 shutdown_connections 2024-08-03T20:58:39.196 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:39.193+0000 7f3843a00640 1 -- 172.21.15.32:0/3631688529 wait complete. 2024-08-03T20:58:39.197 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:39.196+0000 7f3843a00640 1 Processor -- start 2024-08-03T20:58:39.197 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:39.196+0000 7f3843a00640 1 -- start start 2024-08-03T20:58:39.199 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:39.197+0000 7f3843a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3824003c10 0x7f383c1594f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:58:39.200 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:39.197+0000 7f3843a00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f383c144900 0x7f383c144d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:58:39.200 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:39.197+0000 7f3843a00640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f383c0cbee0 0x7f383c0cc2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:58:39.200 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:39.197+0000 7f3843a00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f383c0b3ce0 con 0x7f3824003c10 2024-08-03T20:58:39.200 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:39.197+0000 7f3843a00640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f383c06efa0 con 0x7f383c144900 2024-08-03T20:58:39.200 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:39.197+0000 7f3843a00640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7f383c0b3e60 con 0x7f383c0cbee0 2024-08-03T20:58:39.200 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:39.197+0000 7f3841000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3824003c10 0x7f383c1594f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:58:39.200 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:39.198+0000 7f3841000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3824003c10 0x7f383c1594f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:41800/0 (socket says 172.21.15.32:41800) 2024-08-03T20:58:39.200 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:39.198+0000 7f3841000640 1 -- 172.21.15.32:0/3769677184 learned_addr learned my addr 172.21.15.32:0/3769677184 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:58:39.200 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:39.198+0000 7f383be00640 1 --2- 172.21.15.32:0/3769677184 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f383c144900 0x7f383c144d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:58:39.200 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:39.198+0000 7f3841a00640 1 --2- 172.21.15.32:0/3769677184 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f383c0cbee0 0x7f383c0cc2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:58:39.200 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:39.198+0000 7f3841000640 1 -- 172.21.15.32:0/3769677184 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f383c0cbee0 msgr2=0x7f383c0cc2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:58:39.200 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:39.198+0000 7f3841000640 1 --2- 172.21.15.32:0/3769677184 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f383c0cbee0 0x7f383c0cc2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:39.200 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:39.198+0000 7f3841000640 1 -- 172.21.15.32:0/3769677184 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f383c144900 msgr2=0x7f383c144d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:58:39.201 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:39.198+0000 7f3841000640 1 --2- 172.21.15.32:0/3769677184 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f383c144900 0x7f383c144d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:39.201 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:39.198+0000 7f3841000640 1 -- 172.21.15.32:0/3769677184 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f383c143ff0 con 0x7f3824003c10 2024-08-03T20:58:39.201 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:39.198+0000 7f3841a00640 1 --2- 172.21.15.32:0/3769677184 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f383c0cbee0 0x7f383c0cc2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-03T20:58:39.201 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:39.198+0000 7f3841000640 1 --2- 172.21.15.32:0/3769677184 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3824003c10 0x7f383c1594f0 secure :-1 s=READY pgs=1828 cs=0 l=1 rev1=1 crypto rx=0x7f3830036040 tx=0x7f3830033670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:58:39.202 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:39.198+0000 7f3838c00640 1 -- 172.21.15.32:0/3769677184 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3830038040 con 0x7f3824003c10 2024-08-03T20:58:39.202 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:39.199+0000 7f3838c00640 1 -- 172.21.15.32:0/3769677184 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f3830004020 con 0x7f3824003c10 2024-08-03T20:58:39.202 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:39.199+0000 7f3838c00640 1 -- 172.21.15.32:0/3769677184 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3830004470 con 0x7f3824003c10 2024-08-03T20:58:39.202 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:39.199+0000 7f3843a00640 1 -- 172.21.15.32:0/3769677184 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f383c0cc830 con 0x7f3824003c10 2024-08-03T20:58:39.202 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:39.199+0000 7f3843a00640 1 -- 172.21.15.32:0/3769677184 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f383c0c52c0 con 0x7f3824003c10 2024-08-03T20:58:39.202 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:39.200+0000 7f3843a00640 1 -- 172.21.15.32:0/3769677184 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f383c15e220 con 0x7f3824003c10 2024-08-03T20:58:39.203 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:39.201+0000 7f3838c00640 1 -- 172.21.15.32:0/3769677184 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f3830045020 con 0x7f3824003c10 2024-08-03T20:58:39.203 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:39.202+0000 7f3838c00640 1 --2- 172.21.15.32:0/3769677184 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f381c077040 0x7f381c079550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:58:39.204 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:39.202+0000 7f3838c00640 1 -- 172.21.15.32:0/3769677184 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(116..116 src has 1..116) ==== 8520+0+0 (secure 0 0 0) 0x7f38300b9e60 con 0x7f3824003c10 2024-08-03T20:58:39.204 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:39.203+0000 7f383be00640 1 --2- 172.21.15.32:0/3769677184 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f381c077040 0x7f381c079550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:58:39.205 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:39.203+0000 7f383be00640 1 --2- 172.21.15.32:0/3769677184 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f381c077040 0x7f381c079550 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 crypto rx=0x7f383c1f68e0 tx=0x7f3828006b10 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:58:39.208 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:39.207+0000 7f3838c00640 1 -- 172.21.15.32:0/3769677184 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7f383c15e220 con 0x7f3824003c10 2024-08-03T20:58:39.458 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:39.456+0000 7f3843a00640 1 -- 172.21.15.32:0/3769677184 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "joinable", "val": "true"} v 0) -- 0x7f383c0b43f0 con 0x7f3824003c10 2024-08-03T20:58:39.826 INFO:tasks.ceph.mds.a.smithi032.stderr: -34> 2024-08-03T20:58:35.071+0000 7f047c000640 -1 mds.0.cache.ino(0x100) No data while reading inode 0x100 2024-08-03T20:58:39.827 INFO:tasks.ceph.mds.a.smithi032.stderr: -24> 2024-08-03T20:58:35.072+0000 7f047c000640 -1 log_channel(cluster) log [ERR] : Error loading MDS rank 0: (2) No such file or directory 2024-08-03T20:58:39.941 INFO:tasks.ceph.mds.a.smithi032.stdout:starting mds.a at 2024-08-03T20:58:40.059 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:40.058+0000 7f3838c00640 1 -- 172.21.15.32:0/3769677184 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "joinable", "val": "true"}]=0 cephfs marked joinable; MDS may join as newly active. v267) ==== 162+0+0 (secure 0 0 0) 0x7f383c0cc830 con 0x7f3824003c10 2024-08-03T20:58:40.060 INFO:teuthology.orchestra.run.smithi032.stderr:cephfs marked joinable; MDS may join as newly active. 2024-08-03T20:58:40.062 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:40.060+0000 7f3843a00640 1 -- 172.21.15.32:0/3769677184 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f381c077040 msgr2=0x7f381c079550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:58:40.062 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:40.060+0000 7f3843a00640 1 --2- 172.21.15.32:0/3769677184 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f381c077040 0x7f381c079550 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 crypto rx=0x7f383c1f68e0 tx=0x7f3828006b10 comp rx=0 tx=0).stop 2024-08-03T20:58:40.063 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:40.060+0000 7f3843a00640 1 -- 172.21.15.32:0/3769677184 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3824003c10 msgr2=0x7f383c1594f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:58:40.063 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:40.060+0000 7f3843a00640 1 --2- 172.21.15.32:0/3769677184 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3824003c10 0x7f383c1594f0 secure :-1 s=READY pgs=1828 cs=0 l=1 rev1=1 crypto rx=0x7f3830036040 tx=0x7f3830033670 comp rx=0 tx=0).stop 2024-08-03T20:58:40.063 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:40.061+0000 7f3843a00640 1 -- 172.21.15.32:0/3769677184 shutdown_connections 2024-08-03T20:58:40.063 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:40.061+0000 7f3843a00640 1 --2- 172.21.15.32:0/3769677184 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f381c077040 0x7f381c079550 unknown :-1 s=CLOSED pgs=742 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:40.063 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:40.061+0000 7f3843a00640 1 --2- 172.21.15.32:0/3769677184 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f383c144900 0x7f383c144d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:40.063 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:40.061+0000 7f3843a00640 1 --2- 172.21.15.32:0/3769677184 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f383c0cbee0 0x7f383c0cc2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:40.063 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:40.061+0000 7f3843a00640 1 --2- 172.21.15.32:0/3769677184 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3824003c10 0x7f383c1594f0 unknown :-1 s=CLOSED pgs=1828 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:40.063 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:40.061+0000 7f3843a00640 1 -- 172.21.15.32:0/3769677184 >> 172.21.15.32:0/3769677184 conn(0x7f383c0b1bd0 msgr2=0x7f383c15db10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:58:40.063 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:40.061+0000 7f3843a00640 1 -- 172.21.15.32:0/3769677184 shutdown_connections 2024-08-03T20:58:40.063 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:40.061+0000 7f3843a00640 1 -- 172.21.15.32:0/3769677184 wait complete. 2024-08-03T20:58:40.490 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-08-03T20:58:40.723 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:40.720+0000 7f51df200640 1 -- 172.21.15.32:0/1355544141 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f51d806e930 msgr2=0x7f51d8144f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:58:40.723 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:40.720+0000 7f51df200640 1 --2- 172.21.15.32:0/1355544141 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f51d806e930 0x7f51d8144f90 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f51c8009830 tx=0x7f51c802a4a0 comp rx=0 tx=0).stop 2024-08-03T20:58:40.723 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:40.720+0000 7f51df200640 1 -- 172.21.15.32:0/1355544141 shutdown_connections 2024-08-03T20:58:40.723 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:40.720+0000 7f51df200640 1 --2- 172.21.15.32:0/1355544141 >> v2:172.21.15.118:3300/0 conn(0x7f51d815fff0 0x7f51d8160400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:40.723 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:40.720+0000 7f51df200640 1 --2- 172.21.15.32:0/1355544141 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f51d806e930 0x7f51d8144f90 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:40.723 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:40.720+0000 7f51df200640 1 -- 172.21.15.32:0/1355544141 >> 172.21.15.32:0/1355544141 conn(0x7f51d80b1bd0 msgr2=0x7f51d80b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:58:40.723 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:40.720+0000 7f51df200640 1 -- 172.21.15.32:0/1355544141 shutdown_connections 2024-08-03T20:58:40.723 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:40.720+0000 7f51df200640 1 -- 172.21.15.32:0/1355544141 wait complete. 2024-08-03T20:58:40.724 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:40.722+0000 7f51df200640 1 Processor -- start 2024-08-03T20:58:40.724 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:40.722+0000 7f51df200640 1 -- start start 2024-08-03T20:58:40.726 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:40.724+0000 7f51df200640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f51d806e930 0x7f51d813db80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:58:40.726 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:40.724+0000 7f51df200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f51d8073e60 0x7f51d81f6be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:58:40.727 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:40.724+0000 7f51df200640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f51d815fff0 0x7f51d81f7120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:58:40.727 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:40.724+0000 7f51df200640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f51d80cf840 con 0x7f51d8073e60 2024-08-03T20:58:40.727 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:40.724+0000 7f51df200640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f51d8145f30 con 0x7f51d806e930 2024-08-03T20:58:40.727 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:40.724+0000 7f51df200640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7f51d80cf9c0 con 0x7f51d815fff0 2024-08-03T20:58:40.727 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:40.724+0000 7f51cf400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f51d8073e60 0x7f51d81f6be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:58:40.727 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:40.724+0000 7f51cf400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f51d8073e60 0x7f51d81f6be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:32984/0 (socket says 172.21.15.32:32984) 2024-08-03T20:58:40.727 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:40.724+0000 7f51cf400640 1 -- 172.21.15.32:0/1833610300 learned_addr learned my addr 172.21.15.32:0/1833610300 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:58:40.727 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:40.724+0000 7f51d7e00640 1 --2- 172.21.15.32:0/1833610300 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f51d806e930 0x7f51d813db80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:58:40.727 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:40.724+0000 7f51dd200640 1 --2- 172.21.15.32:0/1833610300 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f51d815fff0 0x7f51d81f7120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:58:40.727 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:40.725+0000 7f51cf400640 1 -- 172.21.15.32:0/1833610300 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f51d815fff0 msgr2=0x7f51d81f7120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:58:40.727 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:40.725+0000 7f51cf400640 1 --2- 172.21.15.32:0/1833610300 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f51d815fff0 0x7f51d81f7120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:40.727 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:40.725+0000 7f51cf400640 1 -- 172.21.15.32:0/1833610300 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f51d806e930 msgr2=0x7f51d813db80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:58:40.727 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:40.725+0000 7f51cf400640 1 --2- 172.21.15.32:0/1833610300 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f51d806e930 0x7f51d813db80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:40.727 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:40.725+0000 7f51cf400640 1 -- 172.21.15.32:0/1833610300 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f51d815f7c0 con 0x7f51d8073e60 2024-08-03T20:58:40.728 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:40.725+0000 7f51cf400640 1 --2- 172.21.15.32:0/1833610300 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f51d8073e60 0x7f51d81f6be0 secure :-1 s=READY pgs=1832 cs=0 l=1 rev1=1 crypto rx=0x7f51c000c500 tx=0x7f51c000c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:58:40.728 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:40.725+0000 7f51d4c00640 1 -- 172.21.15.32:0/1833610300 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51c0016020 con 0x7f51d8073e60 2024-08-03T20:58:40.728 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:40.726+0000 7f51d4c00640 1 -- 172.21.15.32:0/1833610300 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f51c0012940 con 0x7f51d8073e60 2024-08-03T20:58:40.729 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:40.726+0000 7f51d4c00640 1 -- 172.21.15.32:0/1833610300 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51c0012d90 con 0x7f51d8073e60 2024-08-03T20:58:40.729 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:40.726+0000 7f51df200640 1 -- 172.21.15.32:0/1833610300 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f51d815faa0 con 0x7f51d8073e60 2024-08-03T20:58:40.729 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:40.726+0000 7f51df200640 1 -- 172.21.15.32:0/1833610300 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f51d815ce30 con 0x7f51d8073e60 2024-08-03T20:58:40.729 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:40.727+0000 7f51df200640 1 -- 172.21.15.32:0/1833610300 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f51d815f530 con 0x7f51d8073e60 2024-08-03T20:58:40.729 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:40.728+0000 7f51d4c00640 1 -- 172.21.15.32:0/1833610300 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f51c0019020 con 0x7f51d8073e60 2024-08-03T20:58:40.730 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:40.729+0000 7f51d4c00640 1 --2- 172.21.15.32:0/1833610300 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f51b4077010 0x7f51b4079520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:58:40.730 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:40.729+0000 7f51d4c00640 1 -- 172.21.15.32:0/1833610300 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(116..116 src has 1..116) ==== 8520+0+0 (secure 0 0 0) 0x7f51c00994b0 con 0x7f51d8073e60 2024-08-03T20:58:40.731 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:40.729+0000 7f51d7e00640 1 --2- 172.21.15.32:0/1833610300 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f51b4077010 0x7f51b4079520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:58:40.732 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:40.730+0000 7f51d7e00640 1 --2- 172.21.15.32:0/1833610300 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f51b4077010 0x7f51b4079520 secure :-1 s=READY pgs=744 cs=0 l=1 rev1=1 crypto rx=0x7f51c8034000 tx=0x7f51c8004730 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:58:40.735 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:40.734+0000 7f51d4c00640 1 -- 172.21.15.32:0/1833610300 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7f51c0020020 con 0x7f51d8073e60 2024-08-03T20:58:40.979 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:40.978+0000 7f51df200640 1 -- 172.21.15.32:0/1833610300 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f51d80b19f0 con 0x7f51d8073e60 2024-08-03T20:58:40.980 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:40.978+0000 7f51d4c00640 1 -- 172.21.15.32:0/1833610300 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 267 v267) ==== 95+0+5223 (secure 0 0 0) 0x7f51d80b19f0 con 0x7f51d8073e60 2024-08-03T20:58:40.980 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-03T20:58:40.980 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":267,"btime":"2024-08-03T20:58:40:057846+0000","default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6439,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6835/531080160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6834","nonce":531080160},{"type":"v1","addr":"172.21.15.118:6835","nonce":531080160}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":84},{"gid":6454,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6837/2796882319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6836","nonce":2796882319},{"type":"v1","addr":"172.21.15.118:6837","nonce":2796882319}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":86},{"gid":6813,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6834/2686933890","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":2686933890},{"type":"v1","addr":"172.21.15.32:6834","nonce":2686933890}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":85}],"filesystems":[{"mdsmap":{"epoch":267,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-08-03T20:54:26.950518+0000","modified":"2024-08-03T20:58:40.057809+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":116,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":9591},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9591":{"gid":9591,"name":"a","rank":0,"incarnation":267,"state":"up:replay","state_seq":1,"addr":"172.21.15.32:6835/556650445","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6833","nonce":556650445},{"type":"v1","addr":"172.21.15.32:6835","nonce":556650445}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":6}]} 2024-08-03T20:58:40.980 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 267 2024-08-03T20:58:40.982 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:40.980+0000 7f51df200640 1 -- 172.21.15.32:0/1833610300 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f51b4077010 msgr2=0x7f51b4079520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:58:40.982 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:40.980+0000 7f51df200640 1 --2- 172.21.15.32:0/1833610300 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f51b4077010 0x7f51b4079520 secure :-1 s=READY pgs=744 cs=0 l=1 rev1=1 crypto rx=0x7f51c8034000 tx=0x7f51c8004730 comp rx=0 tx=0).stop 2024-08-03T20:58:40.983 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:40.980+0000 7f51df200640 1 -- 172.21.15.32:0/1833610300 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f51d8073e60 msgr2=0x7f51d81f6be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:58:40.983 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:40.980+0000 7f51df200640 1 --2- 172.21.15.32:0/1833610300 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f51d8073e60 0x7f51d81f6be0 secure :-1 s=READY pgs=1832 cs=0 l=1 rev1=1 crypto rx=0x7f51c000c500 tx=0x7f51c000c9c0 comp rx=0 tx=0).stop 2024-08-03T20:58:40.983 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:40.981+0000 7f51df200640 1 -- 172.21.15.32:0/1833610300 shutdown_connections 2024-08-03T20:58:40.983 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:40.981+0000 7f51df200640 1 --2- 172.21.15.32:0/1833610300 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f51b4077010 0x7f51b4079520 unknown :-1 s=CLOSED pgs=744 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:40.983 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:40.981+0000 7f51df200640 1 --2- 172.21.15.32:0/1833610300 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f51d815fff0 0x7f51d81f7120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:40.983 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:40.981+0000 7f51df200640 1 --2- 172.21.15.32:0/1833610300 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f51d8073e60 0x7f51d81f6be0 unknown :-1 s=CLOSED pgs=1832 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:40.983 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:40.981+0000 7f51df200640 1 --2- 172.21.15.32:0/1833610300 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f51d806e930 0x7f51d813db80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:40.983 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:40.981+0000 7f51df200640 1 -- 172.21.15.32:0/1833610300 >> 172.21.15.32:0/1833610300 conn(0x7f51d80b1bd0 msgr2=0x7f51d80b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:58:40.983 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:40.981+0000 7f51df200640 1 -- 172.21.15.32:0/1833610300 shutdown_connections 2024-08-03T20:58:40.983 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:40.981+0000 7f51df200640 1 -- 172.21.15.32:0/1833610300 wait complete. 2024-08-03T20:58:41.005 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 267, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-08-03T20:54:26.950518+0000', 'modified': '2024-08-03T20:58:40.057809+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 116, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 9591}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9591': {'gid': 9591, 'name': 'a', 'rank': 0, 'incarnation': 267, 'state': 'up:replay', 'state_seq': 1, 'addr': '172.21.15.32:6835/556650445', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6833', 'nonce': 556650445}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 556650445}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [13], 'metadata_pool': 12, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 0, 'qdb_cluster': []} 2024-08-03T20:58:41.005 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9591:up:replay 2024-08-03T20:58:41.005 DEBUG:tasks.ceph_test_case:wait_until_true: waiting (timeout=60 retry_count=0)... 2024-08-03T20:58:45.069 INFO:tasks.ceph.mds.a.smithi032.stderr:2024-08-03T20:58:45.067+0000 7ff617400640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-08-03T20:58:46.409 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-08-03T20:58:46.649 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:46.647+0000 7fc493e00640 1 -- 172.21.15.32:0/2485226440 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc4940af020 msgr2=0x7fc4940742b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:58:46.649 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:46.647+0000 7fc493e00640 1 --2- 172.21.15.32:0/2485226440 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc4940af020 0x7fc4940742b0 secure :-1 s=READY pgs=1833 cs=0 l=1 rev1=1 crypto rx=0x7fc484005fd0 tx=0x7fc48402a540 comp rx=0 tx=0).stop 2024-08-03T20:58:46.649 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:46.647+0000 7fc493e00640 1 -- 172.21.15.32:0/2485226440 shutdown_connections 2024-08-03T20:58:46.649 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:46.647+0000 7fc493e00640 1 --2- 172.21.15.32:0/2485226440 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fc4940acd80 0x7fc4940ad190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:46.650 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:46.647+0000 7fc493e00640 1 --2- 172.21.15.32:0/2485226440 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc4940af020 0x7fc4940742b0 unknown :-1 s=CLOSED pgs=1833 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:46.650 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:46.647+0000 7fc493e00640 1 -- 172.21.15.32:0/2485226440 >> 172.21.15.32:0/2485226440 conn(0x7fc4940b2120 msgr2=0x7fc4940cfd60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:58:46.650 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:46.647+0000 7fc493e00640 1 -- 172.21.15.32:0/2485226440 shutdown_connections 2024-08-03T20:58:46.650 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:46.647+0000 7fc493e00640 1 -- 172.21.15.32:0/2485226440 wait complete. 2024-08-03T20:58:46.651 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:46.649+0000 7fc493e00640 1 Processor -- start 2024-08-03T20:58:46.651 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:46.650+0000 7fc493e00640 1 -- start start 2024-08-03T20:58:46.653 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:46.651+0000 7fc493e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc49406f440 0x7fc4940c6e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:58:46.653 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:46.651+0000 7fc493e00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc4940acd80 0x7fc4941f6d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:58:46.653 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:46.651+0000 7fc493e00640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fc4940af020 0x7fc4940c60e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:58:46.653 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:46.651+0000 7fc493e00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fc494160260 con 0x7fc49406f440 2024-08-03T20:58:46.653 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:46.651+0000 7fc493e00640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7fc4941600e0 con 0x7fc4940acd80 2024-08-03T20:58:46.653 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:46.651+0000 7fc493e00640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7fc4941603e0 con 0x7fc4940af020 2024-08-03T20:58:46.653 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:46.651+0000 7fc492a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc49406f440 0x7fc4940c6e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:58:46.653 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:46.651+0000 7fc492a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc49406f440 0x7fc4940c6e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:33002/0 (socket says 172.21.15.32:33002) 2024-08-03T20:58:46.653 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:46.651+0000 7fc492a00640 1 -- 172.21.15.32:0/3986914102 learned_addr learned my addr 172.21.15.32:0/3986914102 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:58:46.655 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:46.652+0000 7fc48be00640 1 --2- 172.21.15.32:0/3986914102 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc4940acd80 0x7fc4941f6d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:58:46.655 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:46.652+0000 7fc492a00640 1 -- 172.21.15.32:0/3986914102 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fc4940af020 msgr2=0x7fc4940c60e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:58:46.655 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:46.652+0000 7fc492a00640 1 --2- 172.21.15.32:0/3986914102 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fc4940af020 0x7fc4940c60e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:46.655 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:46.652+0000 7fc492a00640 1 -- 172.21.15.32:0/3986914102 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc4940acd80 msgr2=0x7fc4941f6d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:58:46.655 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:46.652+0000 7fc492a00640 1 --2- 172.21.15.32:0/3986914102 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc4940acd80 0x7fc4941f6d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:46.655 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:46.652+0000 7fc492a00640 1 -- 172.21.15.32:0/3986914102 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc4940c7350 con 0x7fc49406f440 2024-08-03T20:58:46.655 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:46.652+0000 7fc48be00640 1 --2- 172.21.15.32:0/3986914102 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc4940acd80 0x7fc4941f6d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-03T20:58:46.655 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:46.652+0000 7fc492a00640 1 --2- 172.21.15.32:0/3986914102 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc49406f440 0x7fc4940c6e10 secure :-1 s=READY pgs=1834 cs=0 l=1 rev1=1 crypto rx=0x7fc484034040 tx=0x7fc4840038a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:58:46.655 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:46.652+0000 7fc48aa00640 1 -- 172.21.15.32:0/3986914102 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc48403e030 con 0x7fc49406f440 2024-08-03T20:58:46.655 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:46.652+0000 7fc48aa00640 1 -- 172.21.15.32:0/3986914102 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fc484003d90 con 0x7fc49406f440 2024-08-03T20:58:46.655 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:46.652+0000 7fc493e00640 1 -- 172.21.15.32:0/3986914102 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc4941f7240 con 0x7fc49406f440 2024-08-03T20:58:46.655 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:46.652+0000 7fc48aa00640 1 -- 172.21.15.32:0/3986914102 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc484035040 con 0x7fc49406f440 2024-08-03T20:58:46.656 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:46.652+0000 7fc493e00640 1 -- 172.21.15.32:0/3986914102 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc4941f7510 con 0x7fc49406f440 2024-08-03T20:58:46.656 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:46.653+0000 7fc493e00640 1 -- 172.21.15.32:0/3986914102 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc4940ad190 con 0x7fc49406f440 2024-08-03T20:58:46.656 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:46.655+0000 7fc48aa00640 1 -- 172.21.15.32:0/3986914102 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7fc484041030 con 0x7fc49406f440 2024-08-03T20:58:46.657 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:46.656+0000 7fc48aa00640 1 --2- 172.21.15.32:0/3986914102 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fc470077010 0x7fc470079520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:58:46.657 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:46.656+0000 7fc48aa00640 1 -- 172.21.15.32:0/3986914102 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(116..116 src has 1..116) ==== 8520+0+0 (secure 0 0 0) 0x7fc4840b9b50 con 0x7fc49406f440 2024-08-03T20:58:46.658 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:46.656+0000 7fc48be00640 1 --2- 172.21.15.32:0/3986914102 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fc470077010 0x7fc470079520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:58:46.658 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:46.657+0000 7fc48be00640 1 --2- 172.21.15.32:0/3986914102 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fc470077010 0x7fc470079520 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 crypto rx=0x7fc4940c8440 tx=0x7fc47c005f90 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:58:46.662 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:46.661+0000 7fc48aa00640 1 -- 172.21.15.32:0/3986914102 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7fc484082230 con 0x7fc49406f440 2024-08-03T20:58:46.906 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:46.905+0000 7fc493e00640 1 -- 172.21.15.32:0/3986914102 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc4940d0930 con 0x7fc49406f440 2024-08-03T20:58:46.907 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:46.905+0000 7fc48aa00640 1 -- 172.21.15.32:0/3986914102 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 270 v270) ==== 95+0+5230 (secure 0 0 0) 0x7fc484081b30 con 0x7fc49406f440 2024-08-03T20:58:46.907 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-03T20:58:46.907 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":270,"btime":"2024-08-03T20:58:43:064745+0000","default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6439,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6835/531080160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6834","nonce":531080160},{"type":"v1","addr":"172.21.15.118:6835","nonce":531080160}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":84},{"gid":6454,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6837/2796882319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6836","nonce":2796882319},{"type":"v1","addr":"172.21.15.118:6837","nonce":2796882319}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":86},{"gid":6813,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6834/2686933890","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":2686933890},{"type":"v1","addr":"172.21.15.32:6834","nonce":2686933890}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":85}],"filesystems":[{"mdsmap":{"epoch":270,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-08-03T20:54:26.950518+0000","modified":"2024-08-03T20:58:43.064736+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":116,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":9591},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9591":{"gid":9591,"name":"a","rank":0,"incarnation":267,"state":"up:active","state_seq":4,"addr":"172.21.15.32:6835/556650445","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6833","nonce":556650445},{"type":"v1","addr":"172.21.15.32:6835","nonce":556650445}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9591,"qdb_cluster":[9591]},"id":6}]} 2024-08-03T20:58:46.908 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 270 2024-08-03T20:58:46.910 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:46.908+0000 7fc493e00640 1 -- 172.21.15.32:0/3986914102 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fc470077010 msgr2=0x7fc470079520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:58:46.910 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:46.908+0000 7fc493e00640 1 --2- 172.21.15.32:0/3986914102 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fc470077010 0x7fc470079520 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 crypto rx=0x7fc4940c8440 tx=0x7fc47c005f90 comp rx=0 tx=0).stop 2024-08-03T20:58:46.910 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:46.908+0000 7fc493e00640 1 -- 172.21.15.32:0/3986914102 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc49406f440 msgr2=0x7fc4940c6e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:58:46.910 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:46.908+0000 7fc493e00640 1 --2- 172.21.15.32:0/3986914102 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc49406f440 0x7fc4940c6e10 secure :-1 s=READY pgs=1834 cs=0 l=1 rev1=1 crypto rx=0x7fc484034040 tx=0x7fc4840038a0 comp rx=0 tx=0).stop 2024-08-03T20:58:46.910 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:46.908+0000 7fc493e00640 1 -- 172.21.15.32:0/3986914102 shutdown_connections 2024-08-03T20:58:46.910 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:46.908+0000 7fc493e00640 1 --2- 172.21.15.32:0/3986914102 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fc470077010 0x7fc470079520 unknown :-1 s=CLOSED pgs=745 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:46.910 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:46.908+0000 7fc493e00640 1 --2- 172.21.15.32:0/3986914102 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc4940acd80 0x7fc4941f6d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:46.911 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:46.908+0000 7fc493e00640 1 --2- 172.21.15.32:0/3986914102 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fc4940af020 0x7fc4940c60e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:46.911 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:46.908+0000 7fc493e00640 1 --2- 172.21.15.32:0/3986914102 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc49406f440 0x7fc4940c6e10 unknown :-1 s=CLOSED pgs=1834 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:46.911 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:46.908+0000 7fc493e00640 1 -- 172.21.15.32:0/3986914102 >> 172.21.15.32:0/3986914102 conn(0x7fc4940b2120 msgr2=0x7fc4940ad3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:58:46.911 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:46.908+0000 7fc493e00640 1 -- 172.21.15.32:0/3986914102 shutdown_connections 2024-08-03T20:58:46.911 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:46.908+0000 7fc493e00640 1 -- 172.21.15.32:0/3986914102 wait complete. 2024-08-03T20:58:46.935 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 270, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-08-03T20:54:26.950518+0000', 'modified': '2024-08-03T20:58:43.064736+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 116, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 9591}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9591': {'gid': 9591, 'name': 'a', 'rank': 0, 'incarnation': 267, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.32:6835/556650445', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6833', 'nonce': 556650445}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 556650445}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [13], 'metadata_pool': 12, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9591, 'qdb_cluster': [9591]} 2024-08-03T20:58:46.935 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-08-03T20:58:46.936 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2024-08-03T20:58:47.169 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:47.168+0000 7f9a77e00640 1 -- 172.21.15.32:0/1206615561 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9a78073990 msgr2=0x7f9a780cd860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:58:47.169 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:47.168+0000 7f9a77e00640 1 --2- 172.21.15.32:0/1206615561 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9a78073990 0x7f9a780cd860 secure :-1 s=READY pgs=1836 cs=0 l=1 rev1=1 crypto rx=0x7f9a68005fd0 tx=0x7f9a6802a540 comp rx=0 tx=0).stop 2024-08-03T20:58:47.169 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:47.168+0000 7f9a77e00640 1 -- 172.21.15.32:0/1206615561 shutdown_connections 2024-08-03T20:58:47.169 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:47.168+0000 7f9a77e00640 1 --2- 172.21.15.32:0/1206615561 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f9a7806d9f0 0x7f9a7806de00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:47.169 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:47.168+0000 7f9a77e00640 1 --2- 172.21.15.32:0/1206615561 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9a78073990 0x7f9a780cd860 unknown :-1 s=CLOSED pgs=1836 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:47.169 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:47.168+0000 7f9a77e00640 1 -- 172.21.15.32:0/1206615561 >> 172.21.15.32:0/1206615561 conn(0x7f9a78072e70 msgr2=0x7f9a78073280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:58:47.169 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:47.168+0000 7f9a77e00640 1 -- 172.21.15.32:0/1206615561 shutdown_connections 2024-08-03T20:58:47.169 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:47.168+0000 7f9a77e00640 1 -- 172.21.15.32:0/1206615561 wait complete. 2024-08-03T20:58:47.171 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:47.169+0000 7f9a77e00640 1 Processor -- start 2024-08-03T20:58:47.171 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:47.169+0000 7f9a77e00640 1 -- start start 2024-08-03T20:58:47.174 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:47.171+0000 7f9a77e00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f9a7806d9f0 0x7f9a780b0f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:58:47.174 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:47.171+0000 7f9a77e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9a78073990 0x7f9a78154880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:58:47.174 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:47.171+0000 7f9a77e00640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f9a78153270 0x7f9a7815b240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:58:47.174 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:47.171+0000 7f9a77e00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f9a780cf7a0 con 0x7f9a78073990 2024-08-03T20:58:47.174 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:47.171+0000 7f9a77e00640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f9a780cf620 con 0x7f9a7806d9f0 2024-08-03T20:58:47.174 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:47.171+0000 7f9a77e00640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7f9a780cf920 con 0x7f9a78153270 2024-08-03T20:58:47.174 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:47.171+0000 7f9a6fe00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9a78073990 0x7f9a78154880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:58:47.174 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:47.171+0000 7f9a6fe00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9a78073990 0x7f9a78154880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:33028/0 (socket says 172.21.15.32:33028) 2024-08-03T20:58:47.174 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:47.171+0000 7f9a6fe00640 1 -- 172.21.15.32:0/2700674611 learned_addr learned my addr 172.21.15.32:0/2700674611 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:58:47.175 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:47.172+0000 7f9a76a00640 1 --2- 172.21.15.32:0/2700674611 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f9a7806d9f0 0x7f9a780b0f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:58:47.175 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:47.172+0000 7f9a77400640 1 --2- 172.21.15.32:0/2700674611 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f9a78153270 0x7f9a7815b240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:58:47.175 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:47.172+0000 7f9a6fe00640 1 -- 172.21.15.32:0/2700674611 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f9a78153270 msgr2=0x7f9a7815b240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:58:47.175 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:47.172+0000 7f9a6fe00640 1 --2- 172.21.15.32:0/2700674611 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f9a78153270 0x7f9a7815b240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:47.175 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:47.172+0000 7f9a6fe00640 1 -- 172.21.15.32:0/2700674611 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f9a7806d9f0 msgr2=0x7f9a780b0f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:58:47.175 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:47.172+0000 7f9a6fe00640 1 --2- 172.21.15.32:0/2700674611 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f9a7806d9f0 0x7f9a780b0f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:47.175 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:47.172+0000 7f9a6fe00640 1 -- 172.21.15.32:0/2700674611 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9a780aeae0 con 0x7f9a78073990 2024-08-03T20:58:47.175 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:47.172+0000 7f9a76a00640 1 --2- 172.21.15.32:0/2700674611 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f9a7806d9f0 0x7f9a780b0f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-03T20:58:47.175 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:47.172+0000 7f9a6fe00640 1 --2- 172.21.15.32:0/2700674611 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9a78073990 0x7f9a78154880 secure :-1 s=READY pgs=1837 cs=0 l=1 rev1=1 crypto rx=0x7f9a6000c3d0 tx=0x7f9a6000c890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:58:47.175 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:47.172+0000 7f9a6ea00640 1 -- 172.21.15.32:0/2700674611 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a60017070 con 0x7f9a78073990 2024-08-03T20:58:47.175 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:47.172+0000 7f9a6ea00640 1 -- 172.21.15.32:0/2700674611 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f9a600053c0 con 0x7f9a78073990 2024-08-03T20:58:47.175 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:47.172+0000 7f9a6ea00640 1 -- 172.21.15.32:0/2700674611 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a60003850 con 0x7f9a78073990 2024-08-03T20:58:47.175 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:47.172+0000 7f9a77e00640 1 -- 172.21.15.32:0/2700674611 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9a780aedc0 con 0x7f9a78073990 2024-08-03T20:58:47.176 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:47.173+0000 7f9a77e00640 1 -- 172.21.15.32:0/2700674611 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9a7806f270 con 0x7f9a78073990 2024-08-03T20:58:47.176 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:47.175+0000 7f9a6ea00640 1 -- 172.21.15.32:0/2700674611 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f9a6001a070 con 0x7f9a78073990 2024-08-03T20:58:47.177 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:47.176+0000 7f9a6ea00640 1 --2- 172.21.15.32:0/2700674611 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f9a54077010 0x7f9a54079520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:58:47.178 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:47.176+0000 7f9a6ea00640 1 -- 172.21.15.32:0/2700674611 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(116..116 src has 1..116) ==== 8520+0+0 (secure 0 0 0) 0x7f9a60098830 con 0x7f9a78073990 2024-08-03T20:58:47.178 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:47.177+0000 7f9a849eb740 1 Processor -- start 2024-08-03T20:58:47.178 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:47.177+0000 7f9a76a00640 1 --2- 172.21.15.32:0/2700674611 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f9a54077010 0x7f9a54079520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:58:47.178 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:47.177+0000 7f9a849eb740 1 -- start start 2024-08-03T20:58:47.179 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:47.177+0000 7f9a76a00640 1 --2- 172.21.15.32:0/2700674611 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f9a54077010 0x7f9a54079520 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 crypto rx=0x7f9a68033000 tx=0x7f9a68004730 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:58:47.182 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:47.178+0000 7f9a849eb740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56270a5f0270 0x56270a9db080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:58:47.182 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:47.178+0000 7f9a849eb740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56270a5c73b0 0x56270a9afed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:58:47.182 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:47.178+0000 7f9a849eb740 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x56270a5cf590 0x56270a9b0410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:58:47.183 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:47.178+0000 7f9a849eb740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x56270a908a60 con 0x56270a5f0270 2024-08-03T20:58:47.183 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:47.178+0000 7f9a849eb740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x56270a8f7f60 con 0x56270a5c73b0 2024-08-03T20:58:47.183 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:47.178+0000 7f9a849eb740 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x56270a8fbfa0 con 0x56270a5cf590 2024-08-03T20:58:47.183 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:47.178+0000 7f9a76a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56270a5f0270 0x56270a9db080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:58:47.183 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:47.178+0000 7f9a76a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56270a5f0270 0x56270a9db080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:33040/0 (socket says 172.21.15.32:33040) 2024-08-03T20:58:47.183 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:47.178+0000 7f9a76a00640 1 -- 172.21.15.32:0/3869587963 learned_addr learned my addr 172.21.15.32:0/3869587963 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:58:47.183 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:47.179+0000 7f9a6fe00640 1 --2- 172.21.15.32:0/3869587963 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56270a5c73b0 0x56270a9afed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:58:47.183 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:47.179+0000 7f9a77400640 1 --2- 172.21.15.32:0/3869587963 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x56270a5cf590 0x56270a9b0410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:58:47.183 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:47.179+0000 7f9a76a00640 1 -- 172.21.15.32:0/3869587963 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x56270a5cf590 msgr2=0x56270a9b0410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:58:47.183 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:47.179+0000 7f9a76a00640 1 --2- 172.21.15.32:0/3869587963 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x56270a5cf590 0x56270a9b0410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:47.183 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:47.179+0000 7f9a76a00640 1 -- 172.21.15.32:0/3869587963 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56270a5c73b0 msgr2=0x56270a9afed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:58:47.183 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:47.179+0000 7f9a76a00640 1 --2- 172.21.15.32:0/3869587963 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56270a5c73b0 0x56270a9afed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:47.183 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:47.179+0000 7f9a76a00640 1 -- 172.21.15.32:0/3869587963 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56270a65b850 con 0x56270a5f0270 2024-08-03T20:58:47.184 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:47.179+0000 7f9a77400640 1 --2- 172.21.15.32:0/3869587963 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x56270a5cf590 0x56270a9b0410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-03T20:58:47.184 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:47.179+0000 7f9a76a00640 1 --2- 172.21.15.32:0/3869587963 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56270a5f0270 0x56270a9db080 secure :-1 s=READY pgs=1838 cs=0 l=1 rev1=1 crypto rx=0x7f9a6802ab30 tx=0x7f9a6803c000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9e95adf56f4b57ad server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:58:47.184 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:47.179+0000 7f9a67400640 1 -- 172.21.15.32:0/3869587963 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a68035040 con 0x56270a5f0270 2024-08-03T20:58:47.184 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:47.180+0000 7f9a67400640 1 -- 172.21.15.32:0/3869587963 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f9a6804c070 con 0x56270a5f0270 2024-08-03T20:58:47.184 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:47.180+0000 7f9a67400640 1 -- 172.21.15.32:0/3869587963 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a68003d80 con 0x56270a5f0270 2024-08-03T20:58:47.184 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:47.180+0000 7f9a849eb740 1 -- 172.21.15.32:0/3869587963 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56270a5f0270 msgr2=0x56270a9db080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:58:47.184 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:47.180+0000 7f9a849eb740 1 --2- 172.21.15.32:0/3869587963 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56270a5f0270 0x56270a9db080 secure :-1 s=READY pgs=1838 cs=0 l=1 rev1=1 crypto rx=0x7f9a6802ab30 tx=0x7f9a6803c000 comp rx=0 tx=0).stop 2024-08-03T20:58:47.184 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:47.180+0000 7f9a849eb740 1 -- 172.21.15.32:0/3869587963 shutdown_connections 2024-08-03T20:58:47.184 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:47.180+0000 7f9a849eb740 1 --2- 172.21.15.32:0/3869587963 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56270a5c73b0 0x56270a9afed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:47.184 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:47.180+0000 7f9a849eb740 1 --2- 172.21.15.32:0/3869587963 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x56270a5cf590 0x56270a9b0410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:47.184 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:47.180+0000 7f9a849eb740 1 --2- 172.21.15.32:0/3869587963 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56270a5f0270 0x56270a9db080 unknown :-1 s=CLOSED pgs=1838 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:47.184 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:47.180+0000 7f9a849eb740 1 -- 172.21.15.32:0/3869587963 >> 172.21.15.32:0/3869587963 conn(0x56270a5e43c0 msgr2=0x56270a996370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:58:47.184 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:47.180+0000 7f9a849eb740 1 -- 172.21.15.32:0/3869587963 shutdown_connections 2024-08-03T20:58:47.185 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:47.180+0000 7f9a849eb740 1 -- 172.21.15.32:0/3869587963 wait complete. 2024-08-03T20:58:47.189 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:47.181+0000 7f9a849eb740 1 -- start start 2024-08-03T20:58:47.189 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:47.182+0000 7f9a849eb740 1 Processor -- start 2024-08-03T20:58:47.189 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:47.184+0000 7f9a849eb740 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x56270a5c73b0 0x56270a9d7cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:58:47.189 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:47.184+0000 7f9a849eb740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56270a5cf590 0x56270a9d4000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:58:47.190 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:47.184+0000 7f9a849eb740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56270a5f0270 0x56270a9be360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:58:47.190 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:47.184+0000 7f9a849eb740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x56270a66a620 con 0x56270a5f0270 2024-08-03T20:58:47.190 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:47.184+0000 7f9a849eb740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x56270a895160 con 0x56270a5cf590 2024-08-03T20:58:47.190 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:47.184+0000 7f9a849eb740 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x56270a9bf590 con 0x56270a5c73b0 2024-08-03T20:58:47.190 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:47.184+0000 7f9a849eb740 10 client.0 _ll_register_callbacks cb 0x56270a8a46b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-03T20:58:47.190 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:47.184+0000 7f9a77400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56270a5f0270 0x56270a9be360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:58:47.190 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:47.184+0000 7f9a77400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56270a5f0270 0x56270a9be360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:33042/0 (socket says 172.21.15.32:33042) 2024-08-03T20:58:47.190 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:47.184+0000 7f9a77400640 1 -- 172.21.15.32:0/1073301030 learned_addr learned my addr 172.21.15.32:0/1073301030 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:58:47.190 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:47.184+0000 7f9a6fe00640 1 --2- 172.21.15.32:0/1073301030 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56270a5cf590 0x56270a9d4000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:58:47.190 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:47.184+0000 7f9a76a00640 1 --2- 172.21.15.32:0/1073301030 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x56270a5c73b0 0x56270a9d7cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:58:47.190 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:47.184+0000 7f9a77400640 1 -- 172.21.15.32:0/1073301030 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x56270a5c73b0 msgr2=0x56270a9d7cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:58:47.190 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:47.184+0000 7f9a77400640 1 --2- 172.21.15.32:0/1073301030 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x56270a5c73b0 0x56270a9d7cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:47.190 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:47.184+0000 7f9a77400640 1 -- 172.21.15.32:0/1073301030 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56270a5cf590 msgr2=0x56270a9d4000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:58:47.191 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:47.184+0000 7f9a77400640 1 --2- 172.21.15.32:0/1073301030 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56270a5cf590 0x56270a9d4000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:47.191 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:47.184+0000 7f9a77400640 1 -- 172.21.15.32:0/1073301030 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56270a6d26b0 con 0x56270a5f0270 2024-08-03T20:58:47.191 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:47.184+0000 7f9a77400640 1 --2- 172.21.15.32:0/1073301030 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56270a5f0270 0x56270a9be360 secure :-1 s=READY pgs=1839 cs=0 l=1 rev1=1 crypto rx=0x7f9a700114c0 tx=0x7f9a70011980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=537748a62aa94e32 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:58:47.191 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:47.184+0000 7f9a5be00640 10 client.0 ms_handle_connect on v2:172.21.15.32:3300/0 2024-08-03T20:58:47.191 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:47.185+0000 7f9a5be00640 1 -- 172.21.15.32:0/1073301030 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a70004d20 con 0x56270a5f0270 2024-08-03T20:58:47.191 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:47.185+0000 7f9a849eb740 1 -- 172.21.15.32:0/1073301030 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9a6804c070 con 0x56270a5f0270 2024-08-03T20:58:47.191 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:47.185+0000 7f9a5be00640 1 -- 172.21.15.32:0/1073301030 <== mon.0 v2:172.21.15.32:3300/0 2 ==== osd_map(116..116 src has 1..116) ==== 8520+0+0 (secure 0 0 0) 0x7f9a70012780 con 0x56270a5f0270 2024-08-03T20:58:47.191 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:47.185+0000 7f9a5be00640 1 -- 172.21.15.32:0/1073301030 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_get_version_reply(handle=1 version=270) ==== 24+0+0 (secure 0 0 0) 0x7f9a70012980 con 0x56270a5f0270 2024-08-03T20:58:47.191 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:47.185+0000 7f9a849eb740 10 client.9618 fetch_fsmap learned FSMap version 270 2024-08-03T20:58:47.191 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:47.185+0000 7f9a849eb740 1 -- 172.21.15.32:0/1073301030 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,fsmap=270,monmap=2+}) -- 0x56270a9d3180 con 0x56270a5f0270 2024-08-03T20:58:47.191 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:47.185+0000 7f9a5be00640 1 -- 172.21.15.32:0/1073301030 <== mon.0 v2:172.21.15.32:3300/0 4 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f9a6804c070 con 0x56270a5f0270 2024-08-03T20:58:47.191 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:47.185+0000 7f9a5be00640 1 -- 172.21.15.32:0/1073301030 <== mon.0 v2:172.21.15.32:3300/0 5 ==== fsmap(e 270) ==== 3502+0+0 (secure 0 0 0) 0x7f9a7000f040 con 0x56270a5f0270 2024-08-03T20:58:47.191 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:47.185+0000 7f9a849eb740 10 client.9618 fetch_fsmap finished waiting for FSMap version 270 2024-08-03T20:58:47.192 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:47.185+0000 7f9a849eb740 10 client.9618 resolve_mds: resolved name 'a' to daemon mds.a 2024-08-03T20:58:47.192 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:47.185+0000 7f9a849eb740 20 client.9618 populate_metadata read hostname 'smithi032' 2024-08-03T20:58:47.192 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:47.185+0000 7f9a849eb740 1 --2- 172.21.15.32:0/1073301030 >> [v2:172.21.15.32:6833/556650445,v1:172.21.15.32:6835/556650445] conn(0x56270a5aee40 0x56270a9f8140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:58:47.192 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:47.185+0000 7f9a849eb740 4 client.9618 mds_command: new command op to 9591 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-08-03T20:58:47.192 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:47.185+0000 7f9a849eb740 1 -- 172.21.15.32:0/1073301030 --> [v2:172.21.15.32:6833/556650445,v1:172.21.15.32:6835/556650445] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56270a90a9b0 con 0x56270a5aee40 2024-08-03T20:58:47.192 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:47.186+0000 7f9a76a00640 1 --2- 172.21.15.32:0/1073301030 >> [v2:172.21.15.32:6833/556650445,v1:172.21.15.32:6835/556650445] conn(0x56270a5aee40 0x56270a9f8140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:58:47.192 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:47.186+0000 7f9a76a00640 1 --2- 172.21.15.32:0/1073301030 >> [v2:172.21.15.32:6833/556650445,v1:172.21.15.32:6835/556650445] conn(0x56270a5aee40 0x56270a9f8140 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=da530a8f9a6db80 server_cookie=f7414afa6241abab in_seq=0 out_seq=0 2024-08-03T20:58:47.192 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:47.186+0000 7f9a5be00640 10 client.9618 ms_handle_connect on v2:172.21.15.32:6833/556650445 2024-08-03T20:58:47.192 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:47.187+0000 7f9a5be00640 1 -- 172.21.15.32:0/1073301030 <== mds.0 v2:172.21.15.32:6833/556650445 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x56270a90a9b0 con 0x56270a5aee40 2024-08-03T20:58:47.192 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:47.187+0000 7f9a5be00640 10 client.9618 handle_command_reply: tid=0 2024-08-03T20:58:47.192 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:47.187+0000 7f9a849eb740 1 client.9618 shutdown 2024-08-03T20:58:47.192 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:47.187+0000 7f9a849eb740 1 -- 172.21.15.32:0/1073301030 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56270a5f0270 msgr2=0x56270a9be360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:58:47.192 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:47.188+0000 7f9a849eb740 1 --2- 172.21.15.32:0/1073301030 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56270a5f0270 0x56270a9be360 secure :-1 s=READY pgs=1839 cs=0 l=1 rev1=1 crypto rx=0x7f9a700114c0 tx=0x7f9a70011980 comp rx=0 tx=0).stop 2024-08-03T20:58:47.192 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:47.188+0000 7f9a849eb740 1 -- 172.21.15.32:0/1073301030 shutdown_connections 2024-08-03T20:58:47.193 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:47.188+0000 7f9a849eb740 1 --2- 172.21.15.32:0/1073301030 >> [v2:172.21.15.32:6833/556650445,v1:172.21.15.32:6835/556650445] conn(0x56270a5aee40 0x56270a9f8140 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:47.193 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:47.188+0000 7f9a849eb740 1 --2- 172.21.15.32:0/1073301030 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56270a5cf590 0x56270a9d4000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:47.193 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:47.188+0000 7f9a849eb740 1 --2- 172.21.15.32:0/1073301030 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56270a5f0270 0x56270a9be360 unknown :-1 s=CLOSED pgs=1839 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:47.193 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:47.188+0000 7f9a849eb740 1 --2- 172.21.15.32:0/1073301030 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x56270a5c73b0 0x56270a9d7cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:47.193 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:47.188+0000 7f9a5be00640 0 client.9618 ms_handle_reset on v2:172.21.15.32:6833/556650445 2024-08-03T20:58:47.193 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:47.188+0000 7f9a849eb740 1 -- 172.21.15.32:0/1073301030 >> 172.21.15.32:0/1073301030 conn(0x56270a5e43c0 msgr2=0x56270a991090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:58:47.193 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:47.188+0000 7f9a849eb740 1 -- 172.21.15.32:0/1073301030 shutdown_connections 2024-08-03T20:58:47.193 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:47.188+0000 7f9a849eb740 1 -- 172.21.15.32:0/1073301030 wait complete. 2024-08-03T20:58:47.193 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:47.188+0000 7f9a849eb740 20 client.9618 trim_cache size 0 max 16384 2024-08-03T20:58:47.209 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:47.208+0000 7f9a849eb740 1 Processor -- start 2024-08-03T20:58:47.210 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:47.208+0000 7f9a849eb740 1 -- start start 2024-08-03T20:58:47.212 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:47.209+0000 7f9a849eb740 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x56270a5e43c0 0x56270aa4fbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:58:47.212 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:47.209+0000 7f9a849eb740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56270a5c73b0 0x56270a98be20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:58:47.212 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:47.209+0000 7f9a849eb740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56270a5cf590 0x56270a9bf820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:58:47.212 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:47.209+0000 7f9a849eb740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f9a78202970 con 0x56270a5cf590 2024-08-03T20:58:47.212 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:47.209+0000 7f9a849eb740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f9a78201cd0 con 0x56270a5c73b0 2024-08-03T20:58:47.212 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:47.209+0000 7f9a849eb740 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7f9a78202320 con 0x56270a5e43c0 2024-08-03T20:58:47.212 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:47.210+0000 7f9a77400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56270a5cf590 0x56270a9bf820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:58:47.212 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:47.210+0000 7f9a77400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56270a5cf590 0x56270a9bf820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:33054/0 (socket says 172.21.15.32:33054) 2024-08-03T20:58:47.212 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:47.210+0000 7f9a77400640 1 -- 172.21.15.32:0/2258678408 learned_addr learned my addr 172.21.15.32:0/2258678408 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:58:47.212 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:47.210+0000 7f9a76a00640 1 --2- 172.21.15.32:0/2258678408 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x56270a5e43c0 0x56270aa4fbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:58:47.212 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:47.210+0000 7f9a6fe00640 1 --2- 172.21.15.32:0/2258678408 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56270a5c73b0 0x56270a98be20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:58:47.213 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:47.210+0000 7f9a77400640 1 -- 172.21.15.32:0/2258678408 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x56270a5e43c0 msgr2=0x56270aa4fbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:58:47.213 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:47.210+0000 7f9a77400640 1 --2- 172.21.15.32:0/2258678408 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x56270a5e43c0 0x56270aa4fbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:47.213 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:47.210+0000 7f9a77400640 1 -- 172.21.15.32:0/2258678408 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56270a5c73b0 msgr2=0x56270a98be20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:58:47.213 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:47.210+0000 7f9a77400640 1 --2- 172.21.15.32:0/2258678408 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56270a5c73b0 0x56270a98be20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:47.213 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:47.210+0000 7f9a77400640 1 -- 172.21.15.32:0/2258678408 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56270a9dd9c0 con 0x56270a5cf590 2024-08-03T20:58:47.213 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:47.210+0000 7f9a77400640 1 --2- 172.21.15.32:0/2258678408 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56270a5cf590 0x56270a9bf820 secure :-1 s=READY pgs=1840 cs=0 l=1 rev1=1 crypto rx=0x7f9a70009b70 tx=0x7f9a70009790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=af466360af35dc5f server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:58:47.214 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:47.211+0000 7f9a66a00640 1 -- 172.21.15.32:0/2258678408 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a7002b070 con 0x56270a5cf590 2024-08-03T20:58:47.214 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:47.211+0000 7f9a66a00640 1 -- 172.21.15.32:0/2258678408 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f9a70012780 con 0x56270a5cf590 2024-08-03T20:58:47.214 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:47.211+0000 7f9a66a00640 1 -- 172.21.15.32:0/2258678408 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a70004d80 con 0x56270a5cf590 2024-08-03T20:58:47.214 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:47.213+0000 7f9a849eb740 1 -- 172.21.15.32:0/2258678408 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56270a5cf590 msgr2=0x56270a9bf820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:58:47.214 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:47.213+0000 7f9a849eb740 1 --2- 172.21.15.32:0/2258678408 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56270a5cf590 0x56270a9bf820 secure :-1 s=READY pgs=1840 cs=0 l=1 rev1=1 crypto rx=0x7f9a70009b70 tx=0x7f9a70009790 comp rx=0 tx=0).stop 2024-08-03T20:58:47.214 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:47.213+0000 7f9a849eb740 1 -- 172.21.15.32:0/2258678408 shutdown_connections 2024-08-03T20:58:47.214 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:47.213+0000 7f9a849eb740 1 --2- 172.21.15.32:0/2258678408 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56270a5c73b0 0x56270a98be20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:47.214 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:47.213+0000 7f9a849eb740 1 --2- 172.21.15.32:0/2258678408 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56270a5cf590 0x56270a9bf820 unknown :-1 s=CLOSED pgs=1840 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:47.215 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:47.213+0000 7f9a849eb740 1 --2- 172.21.15.32:0/2258678408 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x56270a5e43c0 0x56270aa4fbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:47.215 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:47.213+0000 7f9a849eb740 1 -- 172.21.15.32:0/2258678408 >> 172.21.15.32:0/2258678408 conn(0x56270a9be490 msgr2=0x56270a991220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:58:47.215 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:47.213+0000 7f9a849eb740 1 -- 172.21.15.32:0/2258678408 shutdown_connections 2024-08-03T20:58:47.215 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:47.213+0000 7f9a849eb740 1 -- 172.21.15.32:0/2258678408 wait complete. 2024-08-03T20:58:47.215 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:47.213+0000 7f9a849eb740 1 -- start start 2024-08-03T20:58:47.215 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:47.213+0000 7f9a849eb740 1 Processor -- start 2024-08-03T20:58:47.217 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:47.215+0000 7f9a849eb740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56270a5c73b0 0x56270a9f4380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:58:47.217 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:47.215+0000 7f9a849eb740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56270a5cf590 0x56270a9d1870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:58:47.217 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:47.215+0000 7f9a849eb740 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x56270a5e43c0 0x56270a9ec7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:58:47.218 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:47.215+0000 7f9a849eb740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x56270a9bed70 con 0x56270a5c73b0 2024-08-03T20:58:47.218 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:47.215+0000 7f9a849eb740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x56270a9bf3f0 con 0x56270a5cf590 2024-08-03T20:58:47.218 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:47.215+0000 7f9a849eb740 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x56270a9edcf0 con 0x56270a5e43c0 2024-08-03T20:58:47.218 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:47.215+0000 7f9a849eb740 10 client.0 _ll_register_callbacks cb 0x7f9a70009c30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-03T20:58:47.218 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:47.215+0000 7f9a76a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56270a5c73b0 0x56270a9f4380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:58:47.218 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:47.215+0000 7f9a76a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56270a5c73b0 0x56270a9f4380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:33064/0 (socket says 172.21.15.32:33064) 2024-08-03T20:58:47.218 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:47.215+0000 7f9a76a00640 1 -- 172.21.15.32:0/1933049195 learned_addr learned my addr 172.21.15.32:0/1933049195 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:58:47.218 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:47.215+0000 7f9a6fe00640 1 --2- 172.21.15.32:0/1933049195 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56270a5cf590 0x56270a9d1870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:58:47.218 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:47.215+0000 7f9a77400640 1 --2- 172.21.15.32:0/1933049195 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x56270a5e43c0 0x56270a9ec7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:58:47.218 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:47.215+0000 7f9a76a00640 1 -- 172.21.15.32:0/1933049195 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x56270a5e43c0 msgr2=0x56270a9ec7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:58:47.218 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:47.215+0000 7f9a76a00640 1 --2- 172.21.15.32:0/1933049195 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x56270a5e43c0 0x56270a9ec7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:47.218 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:47.215+0000 7f9a76a00640 1 -- 172.21.15.32:0/1933049195 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56270a5cf590 msgr2=0x56270a9d1870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:58:47.218 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:47.216+0000 7f9a76a00640 1 --2- 172.21.15.32:0/1933049195 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56270a5cf590 0x56270a9d1870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:47.219 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:47.216+0000 7f9a76a00640 1 -- 172.21.15.32:0/1933049195 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56270a9d1db0 con 0x56270a5c73b0 2024-08-03T20:58:47.219 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:47.216+0000 7f9a76a00640 1 --2- 172.21.15.32:0/1933049195 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56270a5c73b0 0x56270a9f4380 secure :-1 s=READY pgs=1841 cs=0 l=1 rev1=1 crypto rx=0x7f9a68000c00 tx=0x7f9a68004510 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9154442e21419fe9 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:58:47.219 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:47.216+0000 7f9a5be00640 10 client.0 ms_handle_connect on v2:172.21.15.32:3300/0 2024-08-03T20:58:47.219 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:47.216+0000 7f9a5be00640 1 -- 172.21.15.32:0/1933049195 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a68003fd0 con 0x56270a5c73b0 2024-08-03T20:58:47.219 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:47.216+0000 7f9a5be00640 1 -- 172.21.15.32:0/1933049195 <== mon.0 v2:172.21.15.32:3300/0 2 ==== osd_map(116..116 src has 1..116) ==== 8520+0+0 (secure 0 0 0) 0x7f9a6805d450 con 0x56270a5c73b0 2024-08-03T20:58:47.219 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:47.216+0000 7f9a849eb740 1 -- 172.21.15.32:0/1933049195 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9a70012780 con 0x56270a5c73b0 2024-08-03T20:58:47.221 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:47.216+0000 7f9a5be00640 1 -- 172.21.15.32:0/1933049195 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_get_version_reply(handle=1 version=270) ==== 24+0+0 (secure 0 0 0) 0x7f9a6806b050 con 0x56270a5c73b0 2024-08-03T20:58:47.221 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:47.216+0000 7f9a849eb740 10 client.9624 fetch_fsmap learned FSMap version 270 2024-08-03T20:58:47.221 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:47.216+0000 7f9a849eb740 1 -- 172.21.15.32:0/1933049195 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,fsmap=270,monmap=2+}) -- 0x56270a9bcaa0 con 0x56270a5c73b0 2024-08-03T20:58:47.221 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:47.217+0000 7f9a5be00640 1 -- 172.21.15.32:0/1933049195 <== mon.0 v2:172.21.15.32:3300/0 4 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f9a70012780 con 0x56270a5c73b0 2024-08-03T20:58:47.221 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:47.217+0000 7f9a5be00640 1 -- 172.21.15.32:0/1933049195 <== mon.0 v2:172.21.15.32:3300/0 5 ==== fsmap(e 270) ==== 3502+0+0 (secure 0 0 0) 0x7f9a6803a040 con 0x56270a5c73b0 2024-08-03T20:58:47.222 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:47.217+0000 7f9a849eb740 10 client.9624 fetch_fsmap finished waiting for FSMap version 270 2024-08-03T20:58:47.222 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:47.217+0000 7f9a849eb740 10 client.9624 resolve_mds: resolved name 'a' to daemon mds.a 2024-08-03T20:58:47.222 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:47.217+0000 7f9a849eb740 20 client.9624 populate_metadata read hostname 'smithi032' 2024-08-03T20:58:47.222 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:47.217+0000 7f9a849eb740 1 --2- 172.21.15.32:0/1933049195 >> [v2:172.21.15.32:6833/556650445,v1:172.21.15.32:6835/556650445] conn(0x56270a584280 0x56270a9d3910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:58:47.222 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:47.217+0000 7f9a849eb740 4 client.9624 mds_command: new command op to 9591 tid=0 multi_id=0 [{"prefix": "status"}] 2024-08-03T20:58:47.222 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:47.217+0000 7f9a849eb740 1 -- 172.21.15.32:0/1933049195 --> [v2:172.21.15.32:6833/556650445,v1:172.21.15.32:6835/556650445] -- command(tid 0: {"prefix": "status"}) -- 0x56270a906450 con 0x56270a584280 2024-08-03T20:58:47.222 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:47.217+0000 7f9a6fe00640 1 --2- 172.21.15.32:0/1933049195 >> [v2:172.21.15.32:6833/556650445,v1:172.21.15.32:6835/556650445] conn(0x56270a584280 0x56270a9d3910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:58:47.222 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:47.217+0000 7f9a5be00640 10 client.9624 ms_handle_connect on v2:172.21.15.32:6833/556650445 2024-08-03T20:58:47.222 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:47.217+0000 7f9a6fe00640 1 --2- 172.21.15.32:0/1933049195 >> [v2:172.21.15.32:6833/556650445,v1:172.21.15.32:6835/556650445] conn(0x56270a584280 0x56270a9d3910 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=aa7ac05a17ec6413 server_cookie=8c905878ff01d9b4 in_seq=0 out_seq=0 2024-08-03T20:58:47.222 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:47.218+0000 7f9a5be00640 1 -- 172.21.15.32:0/1933049195 <== mds.0 v2:172.21.15.32:6833/556650445 1 ==== command_reply(tid 0: 0 ) ==== 8+0+334 (crc 0 0 0) 0x56270a906450 con 0x56270a584280 2024-08-03T20:58:47.222 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:47.218+0000 7f9a5be00640 10 client.9624 handle_command_reply: tid=0 2024-08-03T20:58:47.222 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:47.218+0000 7f9a849eb740 1 client.9624 shutdown 2024-08-03T20:58:47.222 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:47.218+0000 7f9a849eb740 1 -- 172.21.15.32:0/1933049195 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56270a5c73b0 msgr2=0x56270a9f4380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:58:47.223 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:47.218+0000 7f9a849eb740 1 --2- 172.21.15.32:0/1933049195 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56270a5c73b0 0x56270a9f4380 secure :-1 s=READY pgs=1841 cs=0 l=1 rev1=1 crypto rx=0x7f9a68000c00 tx=0x7f9a68004510 comp rx=0 tx=0).stop 2024-08-03T20:58:47.223 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:47.218+0000 7f9a849eb740 1 -- 172.21.15.32:0/1933049195 shutdown_connections 2024-08-03T20:58:47.223 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:47.218+0000 7f9a849eb740 1 --2- 172.21.15.32:0/1933049195 >> [v2:172.21.15.32:6833/556650445,v1:172.21.15.32:6835/556650445] conn(0x56270a584280 0x56270a9d3910 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:47.223 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:47.218+0000 7f9a849eb740 1 --2- 172.21.15.32:0/1933049195 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56270a5cf590 0x56270a9d1870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:47.223 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:47.218+0000 7f9a849eb740 1 --2- 172.21.15.32:0/1933049195 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x56270a5e43c0 0x56270a9ec7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:47.223 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:47.218+0000 7f9a849eb740 1 --2- 172.21.15.32:0/1933049195 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56270a5c73b0 0x56270a9f4380 unknown :-1 s=CLOSED pgs=1841 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:47.223 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:47.218+0000 7f9a849eb740 1 -- 172.21.15.32:0/1933049195 >> 172.21.15.32:0/1933049195 conn(0x56270a9be490 msgr2=0x56270a9db090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:58:47.223 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:47.218+0000 7f9a5be00640 0 client.9624 ms_handle_reset on v2:172.21.15.32:6833/556650445 2024-08-03T20:58:47.223 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:47.218+0000 7f9a849eb740 1 -- 172.21.15.32:0/1933049195 shutdown_connections 2024-08-03T20:58:47.223 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:47.218+0000 7f9a849eb740 1 -- 172.21.15.32:0/1933049195 wait complete. 2024-08-03T20:58:47.223 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:47.218+0000 7f9a849eb740 20 client.9624 trim_cache size 0 max 16384 2024-08-03T20:58:47.224 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-08-03T20:58:47.224 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_fsid": "40ab141a-01d7-49b9-ae3c-2edec9876e95", 2024-08-03T20:58:47.224 INFO:teuthology.orchestra.run.smithi032.stdout: "whoami": 0, 2024-08-03T20:58:47.224 INFO:teuthology.orchestra.run.smithi032.stdout: "id": 9591, 2024-08-03T20:58:47.224 INFO:teuthology.orchestra.run.smithi032.stdout: "want_state": "up:active", 2024-08-03T20:58:47.224 INFO:teuthology.orchestra.run.smithi032.stdout: "state": "up:active", 2024-08-03T20:58:47.224 INFO:teuthology.orchestra.run.smithi032.stdout: "fs_name": "cephfs", 2024-08-03T20:58:47.224 INFO:teuthology.orchestra.run.smithi032.stdout: "rank_uptime": 7.1590384470000004, 2024-08-03T20:58:47.224 INFO:teuthology.orchestra.run.smithi032.stdout: "mdsmap_epoch": 270, 2024-08-03T20:58:47.224 INFO:teuthology.orchestra.run.smithi032.stdout: "osdmap_epoch": 116, 2024-08-03T20:58:47.224 INFO:teuthology.orchestra.run.smithi032.stdout: "osdmap_epoch_barrier": 116, 2024-08-03T20:58:47.224 INFO:teuthology.orchestra.run.smithi032.stdout: "uptime": 7.2747735670000004 2024-08-03T20:58:47.224 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-08-03T20:58:47.225 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:47.219+0000 7f9a77e00640 1 -- 172.21.15.32:0/2700674611 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f9a54077010 msgr2=0x7f9a54079520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:58:47.225 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:47.219+0000 7f9a77e00640 1 --2- 172.21.15.32:0/2700674611 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f9a54077010 0x7f9a54079520 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 crypto rx=0x7f9a68033000 tx=0x7f9a68004730 comp rx=0 tx=0).stop 2024-08-03T20:58:47.225 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:47.219+0000 7f9a77e00640 1 -- 172.21.15.32:0/2700674611 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9a78073990 msgr2=0x7f9a78154880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:58:47.226 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:47.219+0000 7f9a77e00640 1 --2- 172.21.15.32:0/2700674611 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9a78073990 0x7f9a78154880 secure :-1 s=READY pgs=1837 cs=0 l=1 rev1=1 crypto rx=0x7f9a6000c3d0 tx=0x7f9a6000c890 comp rx=0 tx=0).stop 2024-08-03T20:58:47.226 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:47.219+0000 7f9a77e00640 1 -- 172.21.15.32:0/2700674611 shutdown_connections 2024-08-03T20:58:47.226 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:47.219+0000 7f9a77e00640 1 --2- 172.21.15.32:0/2700674611 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f9a54077010 0x7f9a54079520 unknown :-1 s=CLOSED pgs=746 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:47.226 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:47.219+0000 7f9a77e00640 1 --2- 172.21.15.32:0/2700674611 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f9a78153270 0x7f9a7815b240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:47.226 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:47.219+0000 7f9a77e00640 1 --2- 172.21.15.32:0/2700674611 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9a78073990 0x7f9a78154880 unknown :-1 s=CLOSED pgs=1837 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:47.226 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:47.219+0000 7f9a77e00640 1 --2- 172.21.15.32:0/2700674611 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f9a7806d9f0 0x7f9a780b0f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:47.226 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:47.219+0000 7f9a77e00640 1 -- 172.21.15.32:0/2700674611 >> 172.21.15.32:0/2700674611 conn(0x7f9a78072e70 msgr2=0x7f9a78073280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:58:47.226 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:47.219+0000 7f9a77e00640 1 -- 172.21.15.32:0/2700674611 shutdown_connections 2024-08-03T20:58:47.226 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:47.219+0000 7f9a77e00640 1 -- 172.21.15.32:0/2700674611 wait complete. 2024-08-03T20:58:47.245 DEBUG:tasks.ceph_test_case:wait_until_true: success in 5s and 0 retries 2024-08-03T20:58:47.245 INFO:tasks.cephfs.test_damage:Daemons came up after mutation 'Corrupt omap key 10000000000.00000000:sixmegs_head', proceeding to ls 2024-08-03T20:58:47.246 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-08-03T20:58:47.475 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:47.473+0000 7fbec8800640 1 -- 172.21.15.32:0/3407798725 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbea8003a00 msgr2=0x7fbea8003e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:58:47.475 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:47.473+0000 7fbec8800640 1 --2- 172.21.15.32:0/3407798725 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbea8003a00 0x7fbea8003e70 secure :-1 s=READY pgs=1843 cs=0 l=1 rev1=1 crypto rx=0x7fbeb4009b50 tx=0x7fbea8005830 comp rx=0 tx=0).stop 2024-08-03T20:58:47.475 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:47.473+0000 7fbec6800640 1 -- 172.21.15.32:0/3407798725 reap_dead start 2024-08-03T20:58:47.475 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:47.473+0000 7fbec8800640 1 -- 172.21.15.32:0/3407798725 shutdown_connections 2024-08-03T20:58:47.475 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:47.473+0000 7fbec8800640 1 -- 172.21.15.32:0/3407798725 >> 172.21.15.32:0/3407798725 conn(0x7fbec00b1bd0 msgr2=0x7fbec00b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:58:47.475 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:47.473+0000 7fbec8800640 1 -- 172.21.15.32:0/3407798725 shutdown_connections 2024-08-03T20:58:47.475 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:47.473+0000 7fbec8800640 1 -- 172.21.15.32:0/3407798725 wait complete. 2024-08-03T20:58:47.477 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:47.476+0000 7fbec8800640 1 Processor -- start 2024-08-03T20:58:47.477 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:47.476+0000 7fbec8800640 1 -- start start 2024-08-03T20:58:47.479 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:47.478+0000 7fbec8800640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fbea8003a00 0x7fbec01f7260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:58:47.479 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:47.478+0000 7fbec8800640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fbec012e640 0x7fbec00aedb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:58:47.479 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:47.478+0000 7fbec8800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbec013eb80 0x7fbec013ef90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:58:47.479 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:47.478+0000 7fbec8800640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fbec00cf9c0 con 0x7fbec013eb80 2024-08-03T20:58:47.479 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:47.478+0000 7fbec8800640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7fbec00cf840 con 0x7fbea8003a00 2024-08-03T20:58:47.479 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:47.478+0000 7fbec8800640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7fbec0144f90 con 0x7fbec012e640 2024-08-03T20:58:47.479 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:47.478+0000 7fbec6800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbec013eb80 0x7fbec013ef90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:58:47.480 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:47.478+0000 7fbec6800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbec013eb80 0x7fbec013ef90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:33080/0 (socket says 172.21.15.32:33080) 2024-08-03T20:58:47.480 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:47.478+0000 7fbec6800640 1 -- 172.21.15.32:0/823461218 learned_addr learned my addr 172.21.15.32:0/823461218 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:58:47.480 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:47.478+0000 7fbec5e00640 1 --2- 172.21.15.32:0/823461218 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fbea8003a00 0x7fbec01f7260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:58:47.480 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:47.478+0000 7fbec5400640 1 --2- 172.21.15.32:0/823461218 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fbec012e640 0x7fbec00aedb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:58:47.480 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:47.478+0000 7fbec6800640 1 -- 172.21.15.32:0/823461218 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fbec012e640 msgr2=0x7fbec00aedb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:58:47.480 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:47.478+0000 7fbec6800640 1 --2- 172.21.15.32:0/823461218 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fbec012e640 0x7fbec00aedb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:47.480 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:47.478+0000 7fbec6800640 1 -- 172.21.15.32:0/823461218 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fbea8003a00 msgr2=0x7fbec01f7260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:58:47.480 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:47.478+0000 7fbec6800640 1 --2- 172.21.15.32:0/823461218 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fbea8003a00 0x7fbec01f7260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:47.480 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:47.478+0000 7fbec6800640 1 -- 172.21.15.32:0/823461218 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbec013da40 con 0x7fbec013eb80 2024-08-03T20:58:47.481 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:47.478+0000 7fbec6800640 1 --2- 172.21.15.32:0/823461218 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbec013eb80 0x7fbec013ef90 secure :-1 s=READY pgs=1844 cs=0 l=1 rev1=1 crypto rx=0x7fbebc001390 tx=0x7fbebc00ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:58:47.481 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:47.478+0000 7fbeba000640 1 -- 172.21.15.32:0/823461218 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbebc00cba0 con 0x7fbec013eb80 2024-08-03T20:58:47.481 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:47.478+0000 7fbeba000640 1 -- 172.21.15.32:0/823461218 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fbebc018070 con 0x7fbec013eb80 2024-08-03T20:58:47.481 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:47.478+0000 7fbeba000640 1 -- 172.21.15.32:0/823461218 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbebc0116c0 con 0x7fbec013eb80 2024-08-03T20:58:47.481 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:47.478+0000 7fbec8800640 1 -- 172.21.15.32:0/823461218 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbec01f77a0 con 0x7fbec013eb80 2024-08-03T20:58:47.481 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:47.479+0000 7fbec8800640 1 -- 172.21.15.32:0/823461218 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbec015cee0 con 0x7fbec013eb80 2024-08-03T20:58:47.482 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:47.480+0000 7fbec8800640 1 -- 172.21.15.32:0/823461218 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbec015cc50 con 0x7fbec013eb80 2024-08-03T20:58:47.482 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:47.482+0000 7fbeba000640 1 -- 172.21.15.32:0/823461218 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7fbebc011860 con 0x7fbec013eb80 2024-08-03T20:58:47.484 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:47.483+0000 7fbeba000640 1 --2- 172.21.15.32:0/823461218 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fbea4077040 0x7fbea4079550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:58:47.484 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:47.483+0000 7fbeba000640 1 -- 172.21.15.32:0/823461218 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(116..116 src has 1..116) ==== 8520+0+0 (secure 0 0 0) 0x7fbebc0997f0 con 0x7fbec013eb80 2024-08-03T20:58:47.484 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:47.483+0000 7fbec5e00640 1 --2- 172.21.15.32:0/823461218 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fbea4077040 0x7fbea4079550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:58:47.485 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:47.484+0000 7fbec5e00640 1 --2- 172.21.15.32:0/823461218 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fbea4077040 0x7fbea4079550 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 crypto rx=0x7fbeb4009b50 tx=0x7fbeb402d040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:58:47.488 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:47.488+0000 7fbeba000640 1 -- 172.21.15.32:0/823461218 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7fbec01f77a0 con 0x7fbec013eb80 2024-08-03T20:58:47.742 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:47.741+0000 7fbec8800640 1 -- 172.21.15.32:0/823461218 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7fbec0144610 con 0x7fbec013eb80 2024-08-03T20:58:47.742 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:47.742+0000 7fbeba000640 1 -- 172.21.15.32:0/823461218 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v270) ==== 53+0+73 (secure 0 0 0) 0x7fbebc00ee60 con 0x7fbec013eb80 2024-08-03T20:58:47.743 INFO:teuthology.orchestra.run.smithi032.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-08-03T20:58:47.745 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:47.744+0000 7fbec8800640 1 -- 172.21.15.32:0/823461218 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fbea4077040 msgr2=0x7fbea4079550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:58:47.745 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:47.744+0000 7fbec8800640 1 --2- 172.21.15.32:0/823461218 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fbea4077040 0x7fbea4079550 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 crypto rx=0x7fbeb4009b50 tx=0x7fbeb402d040 comp rx=0 tx=0).stop 2024-08-03T20:58:47.746 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:47.744+0000 7fbec8800640 1 -- 172.21.15.32:0/823461218 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbec013eb80 msgr2=0x7fbec013ef90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:58:47.746 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:47.744+0000 7fbec8800640 1 --2- 172.21.15.32:0/823461218 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbec013eb80 0x7fbec013ef90 secure :-1 s=READY pgs=1844 cs=0 l=1 rev1=1 crypto rx=0x7fbebc001390 tx=0x7fbebc00ed30 comp rx=0 tx=0).stop 2024-08-03T20:58:47.746 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:47.744+0000 7fbec8800640 1 -- 172.21.15.32:0/823461218 shutdown_connections 2024-08-03T20:58:47.746 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:47.744+0000 7fbec8800640 1 --2- 172.21.15.32:0/823461218 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fbea4077040 0x7fbea4079550 unknown :-1 s=CLOSED pgs=747 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:47.746 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:47.744+0000 7fbec8800640 1 --2- 172.21.15.32:0/823461218 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbec013eb80 0x7fbec013ef90 unknown :-1 s=CLOSED pgs=1844 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:47.747 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:47.744+0000 7fbec8800640 1 --2- 172.21.15.32:0/823461218 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fbec012e640 0x7fbec00aedb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:47.747 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:47.744+0000 7fbec8800640 1 --2- 172.21.15.32:0/823461218 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fbea8003a00 0x7fbec01f7260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:47.747 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:47.744+0000 7fbec8800640 1 -- 172.21.15.32:0/823461218 >> 172.21.15.32:0/823461218 conn(0x7fbec00b1bd0 msgr2=0x7fbec00b0be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:58:47.747 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:47.744+0000 7fbec8800640 1 -- 172.21.15.32:0/823461218 shutdown_connections 2024-08-03T20:58:47.747 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:47.744+0000 7fbec8800640 1 -- 172.21.15.32:0/823461218 wait complete. 2024-08-03T20:58:47.770 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-08-03T20:58:47.771 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-08-03T20:58:47.771 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi032.front.sepia.ceph.com 2024-08-03T20:58:47.771 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-08-03T20:58:47.771 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-08-03T20:58:47.771 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-08-03T20:58:47.771 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2024-08-03T20:58:47.771 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-08-03T20:58:47.771 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-08-03T20:58:47.771 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-08-03T20:58:47.772 DEBUG:teuthology.orchestra.run.smithi032:> ip addr 2024-08-03T20:58:47.807 INFO:teuthology.orchestra.run.smithi032.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-08-03T20:58:47.808 INFO:teuthology.orchestra.run.smithi032.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-08-03T20:58:47.808 INFO:teuthology.orchestra.run.smithi032.stdout: inet 127.0.0.1/8 scope host lo 2024-08-03T20:58:47.808 INFO:teuthology.orchestra.run.smithi032.stdout: valid_lft forever preferred_lft forever 2024-08-03T20:58:47.808 INFO:teuthology.orchestra.run.smithi032.stdout: inet6 ::1/128 scope host 2024-08-03T20:58:47.808 INFO:teuthology.orchestra.run.smithi032.stdout: valid_lft forever preferred_lft forever 2024-08-03T20:58:47.808 INFO:teuthology.orchestra.run.smithi032.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-08-03T20:58:47.808 INFO:teuthology.orchestra.run.smithi032.stdout: link/ether 0c:c4:7a:6b:fd:0c brd ff:ff:ff:ff:ff:ff 2024-08-03T20:58:47.808 INFO:teuthology.orchestra.run.smithi032.stdout: altname enp7s0f0 2024-08-03T20:58:47.808 INFO:teuthology.orchestra.run.smithi032.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-08-03T20:58:47.808 INFO:teuthology.orchestra.run.smithi032.stdout: link/ether 0c:c4:7a:6b:fd:0d brd ff:ff:ff:ff:ff:ff 2024-08-03T20:58:47.808 INFO:teuthology.orchestra.run.smithi032.stdout: altname enp7s0f1 2024-08-03T20:58:47.808 INFO:teuthology.orchestra.run.smithi032.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-08-03T20:58:47.808 INFO:teuthology.orchestra.run.smithi032.stdout: link/ether ac:1f:6b:f8:26:02 brd ff:ff:ff:ff:ff:ff 2024-08-03T20:58:47.809 INFO:teuthology.orchestra.run.smithi032.stdout: altname enp1s0f0 2024-08-03T20:58:47.809 INFO:teuthology.orchestra.run.smithi032.stdout: inet 172.21.15.32/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-08-03T20:58:47.809 INFO:teuthology.orchestra.run.smithi032.stdout: valid_lft 41828sec preferred_lft 41828sec 2024-08-03T20:58:47.809 INFO:teuthology.orchestra.run.smithi032.stdout: inet6 fe80::ae1f:6bff:fef8:2602/64 scope link noprefixroute 2024-08-03T20:58:47.809 INFO:teuthology.orchestra.run.smithi032.stdout: valid_lft forever preferred_lft forever 2024-08-03T20:58:47.809 INFO:teuthology.orchestra.run.smithi032.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-08-03T20:58:47.809 INFO:teuthology.orchestra.run.smithi032.stdout: link/ether ac:1f:6b:f8:26:03 brd ff:ff:ff:ff:ff:ff 2024-08-03T20:58:47.809 INFO:teuthology.orchestra.run.smithi032.stdout: altname enp1s0f1 2024-08-03T20:58:47.809 INFO:teuthology.orchestra.run.smithi032.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-08-03T20:58:47.809 INFO:teuthology.orchestra.run.smithi032.stdout: link/ether 0a:23:2b:ca:2a:93 brd ff:ff:ff:ff:ff:ff 2024-08-03T20:58:47.809 INFO:teuthology.orchestra.run.smithi032.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-08-03T20:58:47.809 INFO:teuthology.orchestra.run.smithi032.stdout: valid_lft forever preferred_lft forever 2024-08-03T20:58:47.809 INFO:teuthology.orchestra.run.smithi032.stdout: inet6 fe80::d804:7cff:fed9:7ba4/64 scope link 2024-08-03T20:58:47.810 INFO:teuthology.orchestra.run.smithi032.stdout: valid_lft forever preferred_lft forever 2024-08-03T20:58:47.810 INFO:teuthology.orchestra.run.smithi032.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-08-03T20:58:47.810 INFO:teuthology.orchestra.run.smithi032.stdout: link/ether 22:86:28:30:4e:62 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-08-03T20:58:47.810 INFO:teuthology.orchestra.run.smithi032.stdout: inet6 fe80::2086:28ff:fe30:4e62/64 scope link 2024-08-03T20:58:47.810 INFO:teuthology.orchestra.run.smithi032.stdout: valid_lft forever preferred_lft forever 2024-08-03T20:58:47.810 INFO:teuthology.orchestra.run.smithi032.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-08-03T20:58:47.810 INFO:teuthology.orchestra.run.smithi032.stdout: link/ether 0a:23:2b:ca:2a:93 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-08-03T20:58:47.810 INFO:teuthology.orchestra.run.smithi032.stdout: inet6 fe80::823:2bff:feca:2a93/64 scope link 2024-08-03T20:58:47.810 INFO:teuthology.orchestra.run.smithi032.stdout: valid_lft forever preferred_lft forever 2024-08-03T20:58:47.810 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-08-03T20:58:47.811 DEBUG:teuthology.orchestra.run.smithi032:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-08-03T20:58:47.902 INFO:teuthology.orchestra.run.smithi032.stdout:1 2024-08-03T20:58:47.905 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-08-03T20:58:47.905 DEBUG:teuthology.orchestra.run.smithi032:> route 2024-08-03T20:58:47.941 INFO:teuthology.orchestra.run.smithi032.stdout:Kernel IP routing table 2024-08-03T20:58:47.941 INFO:teuthology.orchestra.run.smithi032.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-08-03T20:58:47.941 INFO:teuthology.orchestra.run.smithi032.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-08-03T20:58:47.941 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-08-03T20:58:47.941 INFO:teuthology.orchestra.run.smithi032.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-08-03T20:58:47.942 DEBUG:teuthology.orchestra.run.smithi032:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-08-03T20:58:47.942 DEBUG:teuthology.orchestra.run.smithi032:> set -e 2024-08-03T20:58:47.942 DEBUG:teuthology.orchestra.run.smithi032:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-08-03T20:58:47.942 DEBUG:teuthology.orchestra.run.smithi032:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-08-03T20:58:47.942 DEBUG:teuthology.orchestra.run.smithi032:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-08-03T20:58:47.942 DEBUG:teuthology.orchestra.run.smithi032:> ') 2024-08-03T20:58:48.061 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:48 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-08-03T20:58:48.132 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:48 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-08-03T20:58:48.141 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-08-03T20:58:48.142 DEBUG:teuthology.orchestra.run.smithi032:> ip netns list 2024-08-03T20:58:48.177 INFO:teuthology.orchestra.run.smithi032.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-08-03T20:58:48.177 INFO:teuthology.orchestra.run.smithi032.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-08-03T20:58:48.177 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-08-03T20:58:48.177 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-08-03T20:58:48.177 DEBUG:teuthology.orchestra.run.smithi032:> sudo ip link set brx.0 up 2024-08-03T20:58:48.274 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-08-03T20:58:48.274 DEBUG:teuthology.orchestra.run.smithi032:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-08-03T20:58:48.309 INFO:teuthology.orchestra.run.smithi032.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-08-03T20:58:48.309 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-08-03T20:58:48.309 DEBUG:teuthology.orchestra.run.smithi032:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-08-03T20:58:48.385 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2024-08-03T20:58:48.385 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-08-03T20:58:48.385 DEBUG:teuthology.orchestra.run.smithi032:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-08-03T20:58:48.580 INFO:teuthology.orchestra.run.smithi032.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-08-03T20:58:48.581 INFO:teuthology.orchestra.run.smithi032.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-08-03T20:58:48.581 INFO:teuthology.orchestra.run.smithi032.stdout:invalid new device string format 2024-08-03T20:58:48.600 INFO:teuthology.orchestra.run.smithi032.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-08-03T20:58:48.601 INFO:teuthology.orchestra.run.smithi032.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-08-03T20:58:48.601 INFO:teuthology.orchestra.run.smithi032.stdout:invalid new device string format 2024-08-03T20:58:48.601 INFO:teuthology.orchestra.run.smithi032.stdout:mount.ceph: resolved to: "172.21.15.32:3300,172.21.15.118:3300,172.21.15.118:3301" 2024-08-03T20:58:48.601 INFO:teuthology.orchestra.run.smithi032.stdout:mount.ceph: switching to using v1 address with old syntax 2024-08-03T20:58:48.632 INFO:teuthology.orchestra.run.smithi032.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-08-03T20:58:48.633 INFO:teuthology.orchestra.run.smithi032.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-08-03T20:58:48.633 INFO:teuthology.orchestra.run.smithi032.stdout:invalid new device string format 2024-08-03T20:58:48.633 INFO:teuthology.orchestra.run.smithi032.stdout:mount.ceph: resolved to: "172.21.15.32:3300,172.21.15.118:3300,172.21.15.118:3301" 2024-08-03T20:58:48.633 INFO:teuthology.orchestra.run.smithi032.stdout:mount.ceph: switching to using v1 address with old syntax 2024-08-03T20:58:48.633 INFO:teuthology.orchestra.run.smithi032.stdout:mount.ceph: resolved to: "172.21.15.32:6789,172.21.15.118:6789,172.21.15.118:6790" 2024-08-03T20:58:48.633 INFO:teuthology.orchestra.run.smithi032.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.32:6789,172.21.15.118:6789,172.21.15.118:6790:/ 2024-08-03T20:58:48.633 INFO:teuthology.orchestra.run.smithi032.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,key=0,fsid=40ab141a-01d7-49b9-ae3c-2edec9876e95" will pass to kernel 2024-08-03T20:58:48.634 INFO:teuthology.orchestra.run.smithi032.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2024-08-03T20:58:48.634 INFO:teuthology.orchestra.run.smithi032.stdout: You just mounted a file system that supports labels which does not 2024-08-03T20:58:48.634 INFO:teuthology.orchestra.run.smithi032.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-08-03T20:58:48.634 INFO:teuthology.orchestra.run.smithi032.stdout: applications will generate AVC messages and not be allowed access to 2024-08-03T20:58:48.634 INFO:teuthology.orchestra.run.smithi032.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-08-03T20:58:48.637 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-08-03T20:58:48.637 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-08-03T20:58:48.638 DEBUG:teuthology.orchestra.run.smithi032:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-08-03T20:58:48.694 DEBUG:teuthology.orchestra.run.smithi032:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-08-03T20:58:48.773 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:48 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-08-03T20:58:48.776 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:48 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-08-03T20:58:48.784 INFO:teuthology.orchestra.run.smithi032.stdout:client9633 2024-08-03T20:58:48.785 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-08-03T20:58:48.785 DEBUG:teuthology.orchestra.run.smithi032:> dd if=/proc/self/mounts of=/dev/stdout 2024-08-03T20:58:48.822 DEBUG:teuthology.orchestra.run.smithi032:> stdin-killer -- python3 -c ' 2024-08-03T20:58:48.822 DEBUG:teuthology.orchestra.run.smithi032:> import os 2024-08-03T20:58:48.822 DEBUG:teuthology.orchestra.run.smithi032:> import stat 2024-08-03T20:58:48.822 DEBUG:teuthology.orchestra.run.smithi032:> import json 2024-08-03T20:58:48.822 DEBUG:teuthology.orchestra.run.smithi032:> import sys 2024-08-03T20:58:48.823 DEBUG:teuthology.orchestra.run.smithi032:> 2024-08-03T20:58:48.823 DEBUG:teuthology.orchestra.run.smithi032:> try: 2024-08-03T20:58:48.823 DEBUG:teuthology.orchestra.run.smithi032:> s = os.stat('"'"'/home/ubuntu/cephtest/mnt.0/./subdir/sixmegs'"'"') 2024-08-03T20:58:48.823 DEBUG:teuthology.orchestra.run.smithi032:> except OSError as e: 2024-08-03T20:58:48.823 DEBUG:teuthology.orchestra.run.smithi032:> sys.exit(e.errno) 2024-08-03T20:58:48.823 DEBUG:teuthology.orchestra.run.smithi032:> 2024-08-03T20:58:48.824 DEBUG:teuthology.orchestra.run.smithi032:> attrs = ["st_mode", "st_ino", "st_dev", "st_nlink", "st_uid", "st_gid", "st_size", "st_atime", "st_mtime", "st_ctime"] 2024-08-03T20:58:48.824 DEBUG:teuthology.orchestra.run.smithi032:> print(json.dumps( 2024-08-03T20:58:48.824 DEBUG:teuthology.orchestra.run.smithi032:> dict([(a, getattr(s, a)) for a in attrs]), 2024-08-03T20:58:48.824 DEBUG:teuthology.orchestra.run.smithi032:> indent=2)) 2024-08-03T20:58:48.824 DEBUG:teuthology.orchestra.run.smithi032:> ' 2024-08-03T20:58:48.867 INFO:teuthology.orchestra.run:waiting for 20 2024-08-03T20:58:48.972 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:48 stdin-killer INFO: command exited with status 5: exiting normally with same code! 2024-08-03T20:58:50.082 INFO:tasks.ceph.mon.a.smithi032.stderr:2024-08-03T20:58:50.081+0000 7f4e70800640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 MDSs report damaged metadata (MDS_DAMAGE) 2024-08-03T20:58:54.969 DEBUG:teuthology.orchestra.run:got remote process result: 5 2024-08-03T20:58:54.969 INFO:tasks.cephfs.test_damage:Result: EIO on client 2024-08-03T20:58:54.969 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-08-03T20:58:55.211 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:55.210+0000 7feb01e00640 1 -- 172.21.15.32:0/2431587006 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7feafc144f90 msgr2=0x7feafc074570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:58:55.212 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:55.210+0000 7feb01e00640 1 --2- 172.21.15.32:0/2431587006 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7feafc144f90 0x7feafc074570 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7feae4009860 tx=0x7feae402a5a0 comp rx=0 tx=0).stop 2024-08-03T20:58:55.212 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:55.211+0000 7feb01e00640 1 -- 172.21.15.32:0/2431587006 shutdown_connections 2024-08-03T20:58:55.212 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:55.211+0000 7feb01e00640 1 --2- 172.21.15.32:0/2431587006 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7feafc144f90 0x7feafc074570 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:55.212 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:55.211+0000 7feb01e00640 1 -- 172.21.15.32:0/2431587006 >> 172.21.15.32:0/2431587006 conn(0x7feafc0b1bd0 msgr2=0x7feafc0b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:58:55.212 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:55.211+0000 7feb01e00640 1 -- 172.21.15.32:0/2431587006 shutdown_connections 2024-08-03T20:58:55.212 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:55.211+0000 7feb01e00640 1 -- 172.21.15.32:0/2431587006 wait complete. 2024-08-03T20:58:55.213 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:55.213+0000 7feb01e00640 1 Processor -- start 2024-08-03T20:58:55.213 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:55.213+0000 7feb01e00640 1 -- start start 2024-08-03T20:58:55.216 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:55.215+0000 7feb01e00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7feafc06e930 0x7feafc154c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:58:55.217 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:55.215+0000 7feb01e00640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7feafc144f90 0x7feafc15f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:58:55.217 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:55.215+0000 7feb01e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7feafc15fff0 0x7feafc15f6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:58:55.217 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:55.215+0000 7feb01e00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7feafc0cf840 con 0x7feafc15fff0 2024-08-03T20:58:55.217 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:55.215+0000 7feb01e00640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7feafc145f30 con 0x7feafc06e930 2024-08-03T20:58:55.217 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:55.215+0000 7feb01e00640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7feafc0cf9c0 con 0x7feafc144f90 2024-08-03T20:58:55.217 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:55.215+0000 7feafb400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7feafc15fff0 0x7feafc15f6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:58:55.217 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:55.215+0000 7feafb400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7feafc15fff0 0x7feafc15f6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:41966/0 (socket says 172.21.15.32:41966) 2024-08-03T20:58:55.217 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:55.215+0000 7feafb400640 1 -- 172.21.15.32:0/1674621416 learned_addr learned my addr 172.21.15.32:0/1674621416 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:58:55.217 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:55.215+0000 7feaf3e00640 1 --2- 172.21.15.32:0/1674621416 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7feafc144f90 0x7feafc15f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:58:55.217 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:55.215+0000 7feafaa00640 1 --2- 172.21.15.32:0/1674621416 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7feafc06e930 0x7feafc154c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:58:55.218 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:55.215+0000 7feafb400640 1 -- 172.21.15.32:0/1674621416 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7feafc144f90 msgr2=0x7feafc15f1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:58:55.218 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:55.215+0000 7feafb400640 1 --2- 172.21.15.32:0/1674621416 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7feafc144f90 0x7feafc15f1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:55.218 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:55.215+0000 7feafb400640 1 -- 172.21.15.32:0/1674621416 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7feafc06e930 msgr2=0x7feafc154c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:58:55.218 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:55.215+0000 7feafb400640 1 --2- 172.21.15.32:0/1674621416 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7feafc06e930 0x7feafc154c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:55.218 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:55.215+0000 7feafb400640 1 -- 172.21.15.32:0/1674621416 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feafc1f9230 con 0x7feafc15fff0 2024-08-03T20:58:55.218 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:55.216+0000 7feafb400640 1 --2- 172.21.15.32:0/1674621416 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7feafc15fff0 0x7feafc15f6e0 secure :-1 s=READY pgs=1847 cs=0 l=1 rev1=1 crypto rx=0x7feaf400c700 tx=0x7feaf400cbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:58:55.218 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:55.216+0000 7feaf3400640 1 -- 172.21.15.32:0/1674621416 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feaf4003b70 con 0x7feafc15fff0 2024-08-03T20:58:55.218 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:55.216+0000 7feaf3400640 1 -- 172.21.15.32:0/1674621416 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7feaf4003d10 con 0x7feafc15fff0 2024-08-03T20:58:55.218 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:55.216+0000 7feb01e00640 1 -- 172.21.15.32:0/1674621416 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7feafc1f9510 con 0x7feafc15fff0 2024-08-03T20:58:55.219 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:55.216+0000 7feaf3400640 1 -- 172.21.15.32:0/1674621416 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feaf400aab0 con 0x7feafc15fff0 2024-08-03T20:58:55.219 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:55.216+0000 7feb01e00640 1 -- 172.21.15.32:0/1674621416 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7feafc13ce40 con 0x7feafc15fff0 2024-08-03T20:58:55.220 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:55.217+0000 7feb01e00640 1 -- 172.21.15.32:0/1674621416 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7feafc160400 con 0x7feafc15fff0 2024-08-03T20:58:55.220 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:55.219+0000 7feaf3400640 1 -- 172.21.15.32:0/1674621416 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7feaf400ac50 con 0x7feafc15fff0 2024-08-03T20:58:55.220 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:55.219+0000 7feaf3400640 1 --2- 172.21.15.32:0/1674621416 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fead8077010 0x7fead8079520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:58:55.220 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:55.220+0000 7feaf3400640 1 -- 172.21.15.32:0/1674621416 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(116..116 src has 1..116) ==== 8520+0+0 (secure 0 0 0) 0x7feaf4099140 con 0x7feafc15fff0 2024-08-03T20:58:55.220 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:55.220+0000 7feafaa00640 1 --2- 172.21.15.32:0/1674621416 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fead8077010 0x7fead8079520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:58:55.221 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:55.221+0000 7feafaa00640 1 --2- 172.21.15.32:0/1674621416 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fead8077010 0x7fead8079520 secure :-1 s=READY pgs=748 cs=0 l=1 rev1=1 crypto rx=0x7feaec0046b0 tx=0x7feaec007cc0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:58:55.225 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:55.224+0000 7feaf3400640 1 -- 172.21.15.32:0/1674621416 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7feaf4061230 con 0x7feafc15fff0 2024-08-03T20:58:55.472 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:55.471+0000 7feb01e00640 1 -- 172.21.15.32:0/1674621416 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7feafc0af2f0 con 0x7feafc15fff0 2024-08-03T20:58:55.472 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:55.472+0000 7feaf3400640 1 -- 172.21.15.32:0/1674621416 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 271 v271) ==== 95+0+5230 (secure 0 0 0) 0x7feaf4065780 con 0x7feafc15fff0 2024-08-03T20:58:55.473 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-03T20:58:55.473 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":271,"btime":"2024-08-03T20:58:50:081974+0000","default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6439,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6835/531080160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6834","nonce":531080160},{"type":"v1","addr":"172.21.15.118:6835","nonce":531080160}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":84},{"gid":6454,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6837/2796882319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6836","nonce":2796882319},{"type":"v1","addr":"172.21.15.118:6837","nonce":2796882319}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":86},{"gid":6813,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6834/2686933890","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":2686933890},{"type":"v1","addr":"172.21.15.32:6834","nonce":2686933890}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":85}],"filesystems":[{"mdsmap":{"epoch":271,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-08-03T20:54:26.950518+0000","modified":"2024-08-03T20:58:50.065919+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":116,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":9591},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9591":{"gid":9591,"name":"a","rank":0,"incarnation":267,"state":"up:active","state_seq":6,"addr":"172.21.15.32:6835/556650445","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6833","nonce":556650445},{"type":"v1","addr":"172.21.15.32:6835","nonce":556650445}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9591,"qdb_cluster":[9591]},"id":6}]} 2024-08-03T20:58:55.473 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 271 2024-08-03T20:58:55.475 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:55.475+0000 7feb01e00640 1 -- 172.21.15.32:0/1674621416 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fead8077010 msgr2=0x7fead8079520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:58:55.476 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:55.475+0000 7feb01e00640 1 --2- 172.21.15.32:0/1674621416 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fead8077010 0x7fead8079520 secure :-1 s=READY pgs=748 cs=0 l=1 rev1=1 crypto rx=0x7feaec0046b0 tx=0x7feaec007cc0 comp rx=0 tx=0).stop 2024-08-03T20:58:55.476 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:55.475+0000 7feb01e00640 1 -- 172.21.15.32:0/1674621416 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7feafc15fff0 msgr2=0x7feafc15f6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:58:55.476 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:55.475+0000 7feb01e00640 1 --2- 172.21.15.32:0/1674621416 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7feafc15fff0 0x7feafc15f6e0 secure :-1 s=READY pgs=1847 cs=0 l=1 rev1=1 crypto rx=0x7feaf400c700 tx=0x7feaf400cbc0 comp rx=0 tx=0).stop 2024-08-03T20:58:55.476 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:55.475+0000 7feb01e00640 1 -- 172.21.15.32:0/1674621416 shutdown_connections 2024-08-03T20:58:55.476 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:55.475+0000 7feb01e00640 1 --2- 172.21.15.32:0/1674621416 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fead8077010 0x7fead8079520 unknown :-1 s=CLOSED pgs=748 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:55.476 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:55.475+0000 7feb01e00640 1 --2- 172.21.15.32:0/1674621416 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7feafc15fff0 0x7feafc15f6e0 unknown :-1 s=CLOSED pgs=1847 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:55.476 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:55.475+0000 7feb01e00640 1 --2- 172.21.15.32:0/1674621416 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7feafc144f90 0x7feafc15f1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:55.476 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:55.475+0000 7feb01e00640 1 --2- 172.21.15.32:0/1674621416 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7feafc06e930 0x7feafc154c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:55.476 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:55.475+0000 7feb01e00640 1 -- 172.21.15.32:0/1674621416 >> 172.21.15.32:0/1674621416 conn(0x7feafc0b1bd0 msgr2=0x7feafc0b0120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:58:55.476 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:55.475+0000 7feb01e00640 1 -- 172.21.15.32:0/1674621416 shutdown_connections 2024-08-03T20:58:55.476 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:55.475+0000 7feb01e00640 1 -- 172.21.15.32:0/1674621416 wait complete. 2024-08-03T20:58:55.499 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a damage ls --format=json-pretty 2024-08-03T20:58:55.729 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:55.728+0000 7f34ac000640 1 -- 172.21.15.32:0/507368731 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f34a406e950 msgr2=0x7f34a406ed60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:58:55.730 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:55.729+0000 7f34ac000640 1 --2- 172.21.15.32:0/507368731 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f34a406e950 0x7f34a406ed60 secure :-1 s=READY pgs=1849 cs=0 l=1 rev1=1 crypto rx=0x7f3490005fd0 tx=0x7f349002a670 comp rx=0 tx=0).stop 2024-08-03T20:58:55.730 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:55.729+0000 7f34ac000640 1 -- 172.21.15.32:0/507368731 shutdown_connections 2024-08-03T20:58:55.730 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:55.729+0000 7f34ac000640 1 --2- 172.21.15.32:0/507368731 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f34a406e950 0x7f34a406ed60 unknown :-1 s=CLOSED pgs=1849 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:55.730 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:55.729+0000 7f34ac000640 1 -- 172.21.15.32:0/507368731 >> 172.21.15.32:0/507368731 conn(0x7f34a40b1bd0 msgr2=0x7f34a40b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:58:55.730 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:55.729+0000 7f34ac000640 1 -- 172.21.15.32:0/507368731 shutdown_connections 2024-08-03T20:58:55.730 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:55.729+0000 7f34ac000640 1 -- 172.21.15.32:0/507368731 wait complete. 2024-08-03T20:58:55.731 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:55.731+0000 7f34ac000640 1 Processor -- start 2024-08-03T20:58:55.732 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:55.731+0000 7f34ac000640 1 -- start start 2024-08-03T20:58:55.734 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:55.733+0000 7f34ac000640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f34a406e950 0x7f34a414bcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:58:55.734 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:55.733+0000 7f34ac000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f34a40b3e80 0x7f34a4072e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:58:55.734 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:55.733+0000 7f34ac000640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f34a415d2b0 0x7f34a4073390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:58:55.734 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:55.733+0000 7f34ac000640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f34a40b4290 con 0x7f34a40b3e80 2024-08-03T20:58:55.734 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:55.733+0000 7f34ac000640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f34a40cd5a0 con 0x7f34a406e950 2024-08-03T20:58:55.734 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:55.733+0000 7f34ac000640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7f34a40b4410 con 0x7f34a415d2b0 2024-08-03T20:58:55.735 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:55.733+0000 7f34a8c00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f34a40b3e80 0x7f34a4072e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:58:55.735 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:55.733+0000 7f34a8c00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f34a40b3e80 0x7f34a4072e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:41998/0 (socket says 172.21.15.32:41998) 2024-08-03T20:58:55.735 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:55.733+0000 7f34a8c00640 1 -- 172.21.15.32:0/1987084263 learned_addr learned my addr 172.21.15.32:0/1987084263 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:58:55.735 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:55.733+0000 7f34aa000640 1 --2- 172.21.15.32:0/1987084263 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f34a415d2b0 0x7f34a4073390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:58:55.735 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:55.733+0000 7f34a9600640 1 --2- 172.21.15.32:0/1987084263 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f34a406e950 0x7f34a414bcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:58:55.735 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:55.733+0000 7f34a8c00640 1 -- 172.21.15.32:0/1987084263 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f34a415d2b0 msgr2=0x7f34a4073390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:58:55.735 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:55.733+0000 7f34a8c00640 1 --2- 172.21.15.32:0/1987084263 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f34a415d2b0 0x7f34a4073390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:55.735 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:55.733+0000 7f34a8c00640 1 -- 172.21.15.32:0/1987084263 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f34a406e950 msgr2=0x7f34a414bcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:58:55.735 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:55.733+0000 7f34a8c00640 1 --2- 172.21.15.32:0/1987084263 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f34a406e950 0x7f34a414bcf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:55.735 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:55.733+0000 7f34a8c00640 1 -- 172.21.15.32:0/1987084263 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f34a41585d0 con 0x7f34a40b3e80 2024-08-03T20:58:55.735 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:55.734+0000 7f34a8c00640 1 --2- 172.21.15.32:0/1987084263 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f34a40b3e80 0x7f34a4072e50 secure :-1 s=READY pgs=1850 cs=0 l=1 rev1=1 crypto rx=0x7f3490009230 tx=0x7f3490003910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:58:55.736 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:55.734+0000 7f34a1600640 1 -- 172.21.15.32:0/1987084263 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f349003e030 con 0x7f34a40b3e80 2024-08-03T20:58:55.736 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:55.734+0000 7f34a1600640 1 -- 172.21.15.32:0/1987084263 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f3490003e00 con 0x7f34a40b3e80 2024-08-03T20:58:55.736 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:55.734+0000 7f34a1600640 1 -- 172.21.15.32:0/1987084263 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3490035040 con 0x7f34a40b3e80 2024-08-03T20:58:55.736 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:55.734+0000 7f34ac000640 1 -- 172.21.15.32:0/1987084263 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f34a41588b0 con 0x7f34a40b3e80 2024-08-03T20:58:55.736 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:55.734+0000 7f34ac000640 1 -- 172.21.15.32:0/1987084263 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f34a4158e70 con 0x7f34a40b3e80 2024-08-03T20:58:55.737 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:55.737+0000 7f34a1600640 1 -- 172.21.15.32:0/1987084263 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f3490041030 con 0x7f34a40b3e80 2024-08-03T20:58:55.738 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:55.738+0000 7f34a1600640 1 --2- 172.21.15.32:0/1987084263 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f3484077010 0x7f3484079520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:58:55.738 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:55.738+0000 7f34a1600640 1 -- 172.21.15.32:0/1987084263 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(116..116 src has 1..116) ==== 8520+0+0 (secure 0 0 0) 0x7f34900b9d30 con 0x7f34a40b3e80 2024-08-03T20:58:55.739 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:55.738+0000 7f34a9600640 1 --2- 172.21.15.32:0/1987084263 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f3484077010 0x7f3484079520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:58:55.739 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:55.738+0000 7f34b303c740 1 Processor -- start 2024-08-03T20:58:55.739 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:55.738+0000 7f34b303c740 1 -- start start 2024-08-03T20:58:55.739 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:55.739+0000 7f34a9600640 1 --2- 172.21.15.32:0/1987084263 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f3484077010 0x7f3484079520 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 crypto rx=0x7f3498000f60 tx=0x7f3498009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:58:55.741 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:55.740+0000 7f34b303c740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5556bf44b350 0x5556bf8387a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:58:55.741 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:55.740+0000 7f34b303c740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5556bf422490 0x5556bf81b470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:58:55.741 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:55.740+0000 7f34b303c740 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x5556bf42a670 0x5556bf81b9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:58:55.742 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:55.740+0000 7f34b303c740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x5556bf764050 con 0x5556bf422490 2024-08-03T20:58:55.742 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:55.740+0000 7f34b303c740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x5556bf753550 con 0x5556bf44b350 2024-08-03T20:58:55.742 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:55.740+0000 7f34b303c740 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x5556bf757590 con 0x5556bf42a670 2024-08-03T20:58:55.742 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:55.740+0000 7f34a8c00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5556bf422490 0x5556bf81b470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:58:55.742 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:55.740+0000 7f34a8c00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5556bf422490 0x5556bf81b470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:42000/0 (socket says 172.21.15.32:42000) 2024-08-03T20:58:55.742 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:55.740+0000 7f34a8c00640 1 -- 172.21.15.32:0/698894251 learned_addr learned my addr 172.21.15.32:0/698894251 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:58:55.742 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:55.740+0000 7f34a9600640 1 --2- 172.21.15.32:0/698894251 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5556bf44b350 0x5556bf8387a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:58:55.742 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:55.740+0000 7f34aa000640 1 --2- 172.21.15.32:0/698894251 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x5556bf42a670 0x5556bf81b9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:58:55.742 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:55.740+0000 7f34a8c00640 1 -- 172.21.15.32:0/698894251 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x5556bf42a670 msgr2=0x5556bf81b9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:58:55.742 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:55.740+0000 7f34a8c00640 1 --2- 172.21.15.32:0/698894251 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x5556bf42a670 0x5556bf81b9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:55.742 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:55.740+0000 7f34a8c00640 1 -- 172.21.15.32:0/698894251 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5556bf44b350 msgr2=0x5556bf8387a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:58:55.742 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:55.740+0000 7f34a8c00640 1 --2- 172.21.15.32:0/698894251 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5556bf44b350 0x5556bf8387a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:55.743 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:55.740+0000 7f34a8c00640 1 -- 172.21.15.32:0/698894251 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5556bf733a10 con 0x5556bf422490 2024-08-03T20:58:55.743 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:55.741+0000 7f34a8c00640 1 --2- 172.21.15.32:0/698894251 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5556bf422490 0x5556bf81b470 secure :-1 s=READY pgs=1851 cs=0 l=1 rev1=1 crypto rx=0x7f349007dd00 tx=0x7f349002a670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=706756df5f5536a6 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:58:55.743 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:55.741+0000 7f3495c00640 1 -- 172.21.15.32:0/698894251 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3490048030 con 0x5556bf422490 2024-08-03T20:58:55.743 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:55.741+0000 7f3495c00640 1 -- 172.21.15.32:0/698894251 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f3490081c80 con 0x5556bf422490 2024-08-03T20:58:55.743 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:55.741+0000 7f3495c00640 1 -- 172.21.15.32:0/698894251 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f349008ab00 con 0x5556bf422490 2024-08-03T20:58:55.744 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:55.742+0000 7f34b303c740 1 -- 172.21.15.32:0/698894251 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5556bf422490 msgr2=0x5556bf81b470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:58:55.744 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:55.742+0000 7f34b303c740 1 --2- 172.21.15.32:0/698894251 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5556bf422490 0x5556bf81b470 secure :-1 s=READY pgs=1851 cs=0 l=1 rev1=1 crypto rx=0x7f349007dd00 tx=0x7f349002a670 comp rx=0 tx=0).stop 2024-08-03T20:58:55.744 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:55.742+0000 7f34b303c740 1 -- 172.21.15.32:0/698894251 shutdown_connections 2024-08-03T20:58:55.744 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:55.742+0000 7f34b303c740 1 --2- 172.21.15.32:0/698894251 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x5556bf42a670 0x5556bf81b9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:55.744 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:55.742+0000 7f34b303c740 1 --2- 172.21.15.32:0/698894251 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5556bf422490 0x5556bf81b470 unknown :-1 s=CLOSED pgs=1851 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:55.744 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:55.742+0000 7f34b303c740 1 --2- 172.21.15.32:0/698894251 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5556bf44b350 0x5556bf8387a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:55.744 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:55.742+0000 7f34b303c740 1 -- 172.21.15.32:0/698894251 >> 172.21.15.32:0/698894251 conn(0x5556bf43f4a0 msgr2=0x5556bf7f1690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:58:55.744 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:55.742+0000 7f34b303c740 1 -- 172.21.15.32:0/698894251 shutdown_connections 2024-08-03T20:58:55.745 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:55.742+0000 7f34b303c740 1 -- 172.21.15.32:0/698894251 wait complete. 2024-08-03T20:58:55.745 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:55.743+0000 7f34b303c740 1 -- start start 2024-08-03T20:58:55.745 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:55.743+0000 7f34b303c740 1 Processor -- start 2024-08-03T20:58:55.747 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:55.745+0000 7f34b303c740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5556bf422490 0x5556bf8091d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:58:55.748 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:55.745+0000 7f34b303c740 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x5556bf42a670 0x5556bf833d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:58:55.748 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:55.745+0000 7f34b303c740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5556bf44b350 0x5556bf834290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:58:55.748 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:55.745+0000 7f34b303c740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x5556bf68bda0 con 0x5556bf422490 2024-08-03T20:58:55.748 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:55.745+0000 7f34b303c740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x5556bf6edcd0 con 0x5556bf44b350 2024-08-03T20:58:55.748 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:55.745+0000 7f34b303c740 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x5556bf846b40 con 0x5556bf42a670 2024-08-03T20:58:55.748 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:55.745+0000 7f34b303c740 10 client.0 _ll_register_callbacks cb 0x5556bf462ba0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-03T20:58:55.748 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:55.745+0000 7f34a9600640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5556bf422490 0x5556bf8091d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:58:55.748 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:55.745+0000 7f34a9600640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5556bf422490 0x5556bf8091d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:42012/0 (socket says 172.21.15.32:42012) 2024-08-03T20:58:55.748 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:55.745+0000 7f34a9600640 1 -- 172.21.15.32:0/420693721 learned_addr learned my addr 172.21.15.32:0/420693721 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:58:55.748 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:55.745+0000 7f34a9600640 1 -- 172.21.15.32:0/420693721 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x5556bf42a670 msgr2=0x5556bf833d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:58:55.748 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:55.745+0000 7f34a9600640 1 --2- 172.21.15.32:0/420693721 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x5556bf42a670 0x5556bf833d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:55.748 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:55.745+0000 7f34a9600640 1 -- 172.21.15.32:0/420693721 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5556bf44b350 msgr2=0x5556bf834290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:58:55.748 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:55.745+0000 7f34aa000640 1 --2- 172.21.15.32:0/420693721 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5556bf44b350 0x5556bf834290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:58:55.749 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:55.745+0000 7f34a9600640 1 --2- 172.21.15.32:0/420693721 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5556bf44b350 0x5556bf834290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:55.749 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:55.745+0000 7f34a9600640 1 -- 172.21.15.32:0/420693721 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5556bf4b6930 con 0x5556bf422490 2024-08-03T20:58:55.749 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:55.745+0000 7f34aa000640 1 --2- 172.21.15.32:0/420693721 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5556bf44b350 0x5556bf834290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-03T20:58:55.749 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:55.746+0000 7f34a9600640 1 --2- 172.21.15.32:0/420693721 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5556bf422490 0x5556bf8091d0 secure :-1 s=READY pgs=1852 cs=0 l=1 rev1=1 crypto rx=0x7f34980039f0 tx=0x7f3498000ed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=678c926add920eb server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:58:55.749 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:55.746+0000 7f348a000640 10 client.0 ms_handle_connect on v2:172.21.15.32:3300/0 2024-08-03T20:58:55.749 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:55.746+0000 7f348a000640 1 -- 172.21.15.32:0/420693721 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3498027060 con 0x5556bf422490 2024-08-03T20:58:55.749 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:55.746+0000 7f348a000640 1 -- 172.21.15.32:0/420693721 <== mon.0 v2:172.21.15.32:3300/0 2 ==== osd_map(116..116 src has 1..116) ==== 8520+0+0 (secure 0 0 0) 0x7f3498010040 con 0x5556bf422490 2024-08-03T20:58:55.749 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:55.746+0000 7f34b303c740 1 -- 172.21.15.32:0/420693721 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3490081c80 con 0x5556bf422490 2024-08-03T20:58:55.749 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:55.746+0000 7f348a000640 1 -- 172.21.15.32:0/420693721 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_get_version_reply(handle=1 version=271) ==== 24+0+0 (secure 0 0 0) 0x7f349800ec40 con 0x5556bf422490 2024-08-03T20:58:55.749 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:55.746+0000 7f34b303c740 10 client.9654 fetch_fsmap learned FSMap version 271 2024-08-03T20:58:55.749 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:55.746+0000 7f34b303c740 1 -- 172.21.15.32:0/420693721 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,fsmap=271,monmap=2+}) -- 0x5556bf52f7c0 con 0x5556bf422490 2024-08-03T20:58:55.749 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:55.746+0000 7f348a000640 1 -- 172.21.15.32:0/420693721 <== mon.0 v2:172.21.15.32:3300/0 4 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f3490081c80 con 0x5556bf422490 2024-08-03T20:58:55.749 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:55.746+0000 7f348a000640 1 -- 172.21.15.32:0/420693721 <== mon.0 v2:172.21.15.32:3300/0 5 ==== fsmap(e 271) ==== 3502+0+0 (secure 0 0 0) 0x7f3498022430 con 0x5556bf422490 2024-08-03T20:58:55.749 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:55.747+0000 7f34b303c740 10 client.9654 fetch_fsmap finished waiting for FSMap version 271 2024-08-03T20:58:55.750 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:55.747+0000 7f34b303c740 10 client.9654 resolve_mds: resolved name 'a' to daemon mds.a 2024-08-03T20:58:55.750 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:55.747+0000 7f34b303c740 20 client.9654 populate_metadata read hostname 'smithi032' 2024-08-03T20:58:55.750 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:55.747+0000 7f34b303c740 1 --2- 172.21.15.32:0/420693721 >> [v2:172.21.15.32:6833/556650445,v1:172.21.15.32:6835/556650445] conn(0x5556bf40a130 0x5556bf830ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:58:55.750 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:55.747+0000 7f34b303c740 4 client.9654 mds_command: new command op to 9591 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-08-03T20:58:55.750 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:55.747+0000 7f34b303c740 1 -- 172.21.15.32:0/420693721 --> [v2:172.21.15.32:6833/556650445,v1:172.21.15.32:6835/556650445] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5556bf765fa0 con 0x5556bf40a130 2024-08-03T20:58:55.750 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:55.747+0000 7f34a8c00640 1 --2- 172.21.15.32:0/420693721 >> [v2:172.21.15.32:6833/556650445,v1:172.21.15.32:6835/556650445] conn(0x5556bf40a130 0x5556bf830ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:58:55.751 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:55.747+0000 7f34a8c00640 1 --2- 172.21.15.32:0/420693721 >> [v2:172.21.15.32:6833/556650445,v1:172.21.15.32:6835/556650445] conn(0x5556bf40a130 0x5556bf830ec0 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=26754a42a2e82978 server_cookie=83a226820695fa25 in_seq=0 out_seq=0 2024-08-03T20:58:55.751 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:55.747+0000 7f348a000640 10 client.9654 ms_handle_connect on v2:172.21.15.32:6833/556650445 2024-08-03T20:58:55.751 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:55.748+0000 7f348a000640 1 -- 172.21.15.32:0/420693721 <== mds.0 v2:172.21.15.32:6833/556650445 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x5556bf765fa0 con 0x5556bf40a130 2024-08-03T20:58:55.751 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:55.748+0000 7f348a000640 10 client.9654 handle_command_reply: tid=0 2024-08-03T20:58:55.752 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:55.748+0000 7f34b303c740 1 client.9654 shutdown 2024-08-03T20:58:55.752 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:55.748+0000 7f34b303c740 1 -- 172.21.15.32:0/420693721 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5556bf422490 msgr2=0x5556bf8091d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:58:55.752 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:55.748+0000 7f34b303c740 1 --2- 172.21.15.32:0/420693721 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5556bf422490 0x5556bf8091d0 secure :-1 s=READY pgs=1852 cs=0 l=1 rev1=1 crypto rx=0x7f34980039f0 tx=0x7f3498000ed0 comp rx=0 tx=0).stop 2024-08-03T20:58:55.752 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:55.749+0000 7f34b303c740 1 -- 172.21.15.32:0/420693721 shutdown_connections 2024-08-03T20:58:55.752 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:55.749+0000 7f34b303c740 1 --2- 172.21.15.32:0/420693721 >> [v2:172.21.15.32:6833/556650445,v1:172.21.15.32:6835/556650445] conn(0x5556bf40a130 0x5556bf830ec0 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:55.752 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:55.749+0000 7f34b303c740 1 --2- 172.21.15.32:0/420693721 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5556bf44b350 0x5556bf834290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:55.752 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:55.749+0000 7f34b303c740 1 --2- 172.21.15.32:0/420693721 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x5556bf42a670 0x5556bf833d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:55.752 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:55.749+0000 7f348a000640 0 client.9654 ms_handle_reset on v2:172.21.15.32:6833/556650445 2024-08-03T20:58:55.752 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:55.749+0000 7f34b303c740 1 --2- 172.21.15.32:0/420693721 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5556bf422490 0x5556bf8091d0 unknown :-1 s=CLOSED pgs=1852 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:55.752 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:55.749+0000 7f34b303c740 1 -- 172.21.15.32:0/420693721 >> 172.21.15.32:0/420693721 conn(0x5556bf43f4a0 msgr2=0x5556bf7f2670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:58:55.752 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:55.749+0000 7f34b303c740 1 -- 172.21.15.32:0/420693721 shutdown_connections 2024-08-03T20:58:55.752 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:55.749+0000 7f34b303c740 1 -- 172.21.15.32:0/420693721 wait complete. 2024-08-03T20:58:55.752 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:55.749+0000 7f34b303c740 20 client.9654 trim_cache size 0 max 16384 2024-08-03T20:58:55.764 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:55.764+0000 7f34b303c740 1 Processor -- start 2024-08-03T20:58:55.765 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:55.764+0000 7f34b303c740 1 -- start start 2024-08-03T20:58:55.767 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:55.765+0000 7f34b303c740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5556bf422490 0x5556bf835a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:58:55.767 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:55.765+0000 7f34b303c740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5556bf42a670 0x5556bf838580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:58:55.767 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:55.765+0000 7f34b303c740 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x5556bf44b350 0x5556bf838ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:58:55.767 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:55.765+0000 7f34b303c740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f34a4206e00 con 0x5556bf42a670 2024-08-03T20:58:55.767 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:55.765+0000 7f34b303c740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f34a4206160 con 0x5556bf422490 2024-08-03T20:58:55.767 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:55.765+0000 7f34b303c740 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7f34a42067b0 con 0x5556bf44b350 2024-08-03T20:58:55.768 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:55.766+0000 7f34a8c00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5556bf42a670 0x5556bf838580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:58:55.768 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:55.766+0000 7f34a8c00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5556bf42a670 0x5556bf838580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:42022/0 (socket says 172.21.15.32:42022) 2024-08-03T20:58:55.768 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:55.766+0000 7f34a8c00640 1 -- 172.21.15.32:0/194179285 learned_addr learned my addr 172.21.15.32:0/194179285 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:58:55.768 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:55.766+0000 7f34a9600640 1 --2- 172.21.15.32:0/194179285 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5556bf422490 0x5556bf835a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:58:55.768 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:55.766+0000 7f34aa000640 1 --2- 172.21.15.32:0/194179285 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x5556bf44b350 0x5556bf838ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:58:55.768 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:55.766+0000 7f34a8c00640 1 -- 172.21.15.32:0/194179285 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x5556bf44b350 msgr2=0x5556bf838ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:58:55.768 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:55.766+0000 7f34a8c00640 1 --2- 172.21.15.32:0/194179285 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x5556bf44b350 0x5556bf838ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:55.768 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:55.766+0000 7f34a8c00640 1 -- 172.21.15.32:0/194179285 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5556bf422490 msgr2=0x5556bf835a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:58:55.768 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:55.766+0000 7f34a8c00640 1 --2- 172.21.15.32:0/194179285 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5556bf422490 0x5556bf835a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:55.768 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:55.766+0000 7f34a8c00640 1 -- 172.21.15.32:0/194179285 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5556bf812e10 con 0x5556bf42a670 2024-08-03T20:58:55.768 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:55.766+0000 7f34a9600640 1 --2- 172.21.15.32:0/194179285 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5556bf422490 0x5556bf835a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-03T20:58:55.768 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:55.766+0000 7f34a8c00640 1 --2- 172.21.15.32:0/194179285 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5556bf42a670 0x5556bf838580 secure :-1 s=READY pgs=1853 cs=0 l=1 rev1=1 crypto rx=0x7f349007dd00 tx=0x7f34900b9040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=54e037334510ad0c server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:58:55.769 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:55.766+0000 7f3495200640 1 -- 172.21.15.32:0/194179285 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34900a0070 con 0x5556bf42a670 2024-08-03T20:58:55.769 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:55.766+0000 7f3495200640 1 -- 172.21.15.32:0/194179285 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f349008ad30 con 0x5556bf42a670 2024-08-03T20:58:55.769 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:55.766+0000 7f3495200640 1 -- 172.21.15.32:0/194179285 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3490048040 con 0x5556bf42a670 2024-08-03T20:58:55.770 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:55.767+0000 7f34b303c740 1 -- 172.21.15.32:0/194179285 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5556bf42a670 msgr2=0x5556bf838580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:58:55.770 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:55.767+0000 7f34b303c740 1 --2- 172.21.15.32:0/194179285 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5556bf42a670 0x5556bf838580 secure :-1 s=READY pgs=1853 cs=0 l=1 rev1=1 crypto rx=0x7f349007dd00 tx=0x7f34900b9040 comp rx=0 tx=0).stop 2024-08-03T20:58:55.770 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:55.767+0000 7f34b303c740 1 -- 172.21.15.32:0/194179285 shutdown_connections 2024-08-03T20:58:55.770 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:55.767+0000 7f34b303c740 1 --2- 172.21.15.32:0/194179285 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x5556bf44b350 0x5556bf838ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:55.770 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:55.767+0000 7f34b303c740 1 --2- 172.21.15.32:0/194179285 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5556bf42a670 0x5556bf838580 unknown :-1 s=CLOSED pgs=1853 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:55.770 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:55.767+0000 7f34b303c740 1 --2- 172.21.15.32:0/194179285 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5556bf422490 0x5556bf835a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:55.770 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:55.767+0000 7f34b303c740 1 -- 172.21.15.32:0/194179285 >> 172.21.15.32:0/194179285 conn(0x5556bf43f4a0 msgr2=0x5556bf7e71a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:58:55.770 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:55.767+0000 7f34b303c740 1 -- 172.21.15.32:0/194179285 shutdown_connections 2024-08-03T20:58:55.771 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:55.767+0000 7f34b303c740 1 -- 172.21.15.32:0/194179285 wait complete. 2024-08-03T20:58:55.771 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:55.768+0000 7f34b303c740 1 -- start start 2024-08-03T20:58:55.771 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:55.768+0000 7f34b303c740 1 Processor -- start 2024-08-03T20:58:55.775 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:55.770+0000 7f34b303c740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5556bf422490 0x5556bf838770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:58:55.775 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:55.770+0000 7f34b303c740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5556bf42a670 0x5556bf8b7170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:58:55.775 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:55.770+0000 7f34b303c740 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x5556bf44b350 0x5556bf8b76b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:58:55.776 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:55.770+0000 7f34b303c740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x5556bf845c70 con 0x5556bf42a670 2024-08-03T20:58:55.776 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:55.770+0000 7f34b303c740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x5556bf809830 con 0x5556bf422490 2024-08-03T20:58:55.776 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:55.770+0000 7f34b303c740 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x5556bf846220 con 0x5556bf44b350 2024-08-03T20:58:55.776 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:55.770+0000 7f34b303c740 10 client.0 _ll_register_callbacks cb 0x7f34980050d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-03T20:58:55.776 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:55.770+0000 7f34a8c00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5556bf42a670 0x5556bf8b7170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:58:55.776 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:55.770+0000 7f34a8c00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5556bf42a670 0x5556bf8b7170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:42038/0 (socket says 172.21.15.32:42038) 2024-08-03T20:58:55.776 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:55.770+0000 7f34a8c00640 1 -- 172.21.15.32:0/3018456388 learned_addr learned my addr 172.21.15.32:0/3018456388 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:58:55.776 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:55.770+0000 7f34a9600640 1 --2- 172.21.15.32:0/3018456388 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5556bf422490 0x5556bf838770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:58:55.776 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:55.770+0000 7f34aa000640 1 --2- 172.21.15.32:0/3018456388 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x5556bf44b350 0x5556bf8b76b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:58:55.776 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:55.771+0000 7f34a8c00640 1 -- 172.21.15.32:0/3018456388 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x5556bf44b350 msgr2=0x5556bf8b76b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:58:55.776 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:55.771+0000 7f34a8c00640 1 --2- 172.21.15.32:0/3018456388 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x5556bf44b350 0x5556bf8b76b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:55.776 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:55.771+0000 7f34a8c00640 1 -- 172.21.15.32:0/3018456388 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5556bf422490 msgr2=0x5556bf838770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:58:55.777 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:55.771+0000 7f34a8c00640 1 --2- 172.21.15.32:0/3018456388 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5556bf422490 0x5556bf838770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:55.777 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:55.771+0000 7f34a8c00640 1 -- 172.21.15.32:0/3018456388 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5556bf7ecb40 con 0x5556bf42a670 2024-08-03T20:58:55.777 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:55.771+0000 7f34a8c00640 1 --2- 172.21.15.32:0/3018456388 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5556bf42a670 0x5556bf8b7170 secure :-1 s=READY pgs=1854 cs=0 l=1 rev1=1 crypto rx=0x7f349009b040 tx=0x7f34900057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b062dc6d136240a0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:58:55.777 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:55.771+0000 7f348a000640 10 client.0 ms_handle_connect on v2:172.21.15.32:3300/0 2024-08-03T20:58:55.777 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:55.771+0000 7f348a000640 1 -- 172.21.15.32:0/3018456388 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34900a0070 con 0x5556bf42a670 2024-08-03T20:58:55.777 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:55.771+0000 7f348a000640 1 -- 172.21.15.32:0/3018456388 <== mon.0 v2:172.21.15.32:3300/0 2 ==== osd_map(116..116 src has 1..116) ==== 8520+0+0 (secure 0 0 0) 0x7f349008aab0 con 0x5556bf42a670 2024-08-03T20:58:55.777 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:55.771+0000 7f34b303c740 1 -- 172.21.15.32:0/3018456388 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f349008ad30 con 0x5556bf42a670 2024-08-03T20:58:55.777 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:55.771+0000 7f348a000640 1 -- 172.21.15.32:0/3018456388 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_get_version_reply(handle=1 version=271) ==== 24+0+0 (secure 0 0 0) 0x7f3490048050 con 0x5556bf42a670 2024-08-03T20:58:55.777 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:55.771+0000 7f34b303c740 10 client.9660 fetch_fsmap learned FSMap version 271 2024-08-03T20:58:55.777 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:55.771+0000 7f34b303c740 1 -- 172.21.15.32:0/3018456388 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,fsmap=271,monmap=2+}) -- 0x5556bf856560 con 0x5556bf42a670 2024-08-03T20:58:55.777 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:55.772+0000 7f348a000640 1 -- 172.21.15.32:0/3018456388 <== mon.0 v2:172.21.15.32:3300/0 4 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f349008ad30 con 0x5556bf42a670 2024-08-03T20:58:55.777 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:55.772+0000 7f348a000640 1 -- 172.21.15.32:0/3018456388 <== mon.0 v2:172.21.15.32:3300/0 5 ==== fsmap(e 271) ==== 3502+0+0 (secure 0 0 0) 0x7f34900b7b70 con 0x5556bf42a670 2024-08-03T20:58:55.777 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:55.772+0000 7f34b303c740 10 client.9660 fetch_fsmap finished waiting for FSMap version 271 2024-08-03T20:58:55.777 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:55.772+0000 7f34b303c740 10 client.9660 resolve_mds: resolved name 'a' to daemon mds.a 2024-08-03T20:58:55.778 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:55.772+0000 7f34b303c740 20 client.9660 populate_metadata read hostname 'smithi032' 2024-08-03T20:58:55.778 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:55.772+0000 7f34b303c740 1 --2- 172.21.15.32:0/3018456388 >> [v2:172.21.15.32:6833/556650445,v1:172.21.15.32:6835/556650445] conn(0x5556bf8b81a0 0x5556bf8b85b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:58:55.778 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:55.772+0000 7f34b303c740 4 client.9660 mds_command: new command op to 9591 tid=0 multi_id=0 [{"prefix": "damage ls", "format": "json-pretty"}] 2024-08-03T20:58:55.778 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:55.772+0000 7f34b303c740 1 -- 172.21.15.32:0/3018456388 --> [v2:172.21.15.32:6833/556650445,v1:172.21.15.32:6835/556650445] -- command(tid 0: {"prefix": "damage ls", "format": "json-pretty"}) -- 0x5556bf761a40 con 0x5556bf8b81a0 2024-08-03T20:58:55.778 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:55.772+0000 7f34a9600640 1 --2- 172.21.15.32:0/3018456388 >> [v2:172.21.15.32:6833/556650445,v1:172.21.15.32:6835/556650445] conn(0x5556bf8b81a0 0x5556bf8b85b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:58:55.778 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:55.773+0000 7f348a000640 10 client.9660 ms_handle_connect on v2:172.21.15.32:6833/556650445 2024-08-03T20:58:55.778 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:55.773+0000 7f34a9600640 1 --2- 172.21.15.32:0/3018456388 >> [v2:172.21.15.32:6833/556650445,v1:172.21.15.32:6835/556650445] conn(0x5556bf8b81a0 0x5556bf8b85b0 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=25fd2130832813c9 server_cookie=33959a2477213c9 in_seq=0 out_seq=0 2024-08-03T20:58:55.778 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:55.773+0000 7f348a000640 1 -- 172.21.15.32:0/3018456388 <== mds.0 v2:172.21.15.32:6833/556650445 1 ==== command_reply(tid 0: 0 ) ==== 8+0+215 (crc 0 0 0) 0x5556bf761a40 con 0x5556bf8b81a0 2024-08-03T20:58:55.778 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:55.773+0000 7f348a000640 10 client.9660 handle_command_reply: tid=0 2024-08-03T20:58:55.778 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:55.773+0000 7f34b303c740 1 client.9660 shutdown 2024-08-03T20:58:55.778 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:55.773+0000 7f34b303c740 1 -- 172.21.15.32:0/3018456388 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5556bf42a670 msgr2=0x5556bf8b7170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:58:55.778 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:55.773+0000 7f34b303c740 1 --2- 172.21.15.32:0/3018456388 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5556bf42a670 0x5556bf8b7170 secure :-1 s=READY pgs=1854 cs=0 l=1 rev1=1 crypto rx=0x7f349009b040 tx=0x7f34900057f0 comp rx=0 tx=0).stop 2024-08-03T20:58:55.779 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:55.773+0000 7f34b303c740 1 -- 172.21.15.32:0/3018456388 shutdown_connections 2024-08-03T20:58:55.779 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:55.773+0000 7f34b303c740 1 --2- 172.21.15.32:0/3018456388 >> [v2:172.21.15.32:6833/556650445,v1:172.21.15.32:6835/556650445] conn(0x5556bf8b81a0 0x5556bf8b85b0 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:55.779 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:55.773+0000 7f34b303c740 1 --2- 172.21.15.32:0/3018456388 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x5556bf44b350 0x5556bf8b76b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:55.779 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:55.773+0000 7f34b303c740 1 --2- 172.21.15.32:0/3018456388 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5556bf42a670 0x5556bf8b7170 unknown :-1 s=CLOSED pgs=1854 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:55.779 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:55.773+0000 7f34b303c740 1 --2- 172.21.15.32:0/3018456388 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5556bf422490 0x5556bf838770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:55.779 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:55.773+0000 7f348a000640 0 client.9660 ms_handle_reset on v2:172.21.15.32:6833/556650445 2024-08-03T20:58:55.779 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:55.773+0000 7f34b303c740 1 -- 172.21.15.32:0/3018456388 >> 172.21.15.32:0/3018456388 conn(0x5556bf43f4a0 msgr2=0x5556bf8126a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:58:55.779 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:55.773+0000 7f34b303c740 1 -- 172.21.15.32:0/3018456388 shutdown_connections 2024-08-03T20:58:55.779 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:55.773+0000 7f34b303c740 1 -- 172.21.15.32:0/3018456388 wait complete. 2024-08-03T20:58:55.779 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:55.774+0000 7f34b303c740 20 client.9660 trim_cache size 0 max 16384 2024-08-03T20:58:55.780 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-03T20:58:55.780 INFO:teuthology.orchestra.run.smithi032.stdout:[ 2024-08-03T20:58:55.780 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-03T20:58:55.780 INFO:teuthology.orchestra.run.smithi032.stdout: "damage_type": "dentry", 2024-08-03T20:58:55.780 INFO:teuthology.orchestra.run.smithi032.stdout: "id": 2411841783, 2024-08-03T20:58:55.780 INFO:teuthology.orchestra.run.smithi032.stdout: "ino": 1099511627776, 2024-08-03T20:58:55.780 INFO:teuthology.orchestra.run.smithi032.stdout: "frag": "*", 2024-08-03T20:58:55.780 INFO:teuthology.orchestra.run.smithi032.stdout: "dname": "sixmegs", 2024-08-03T20:58:55.780 INFO:teuthology.orchestra.run.smithi032.stdout: "snap_id": "head", 2024-08-03T20:58:55.780 INFO:teuthology.orchestra.run.smithi032.stdout: "path": "/subdir/sixmegs" 2024-08-03T20:58:55.780 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-03T20:58:55.780 INFO:teuthology.orchestra.run.smithi032.stdout:] 2024-08-03T20:58:55.781 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:55.774+0000 7f34ac000640 1 -- 172.21.15.32:0/1987084263 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f3484077010 msgr2=0x7f3484079520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:58:55.781 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:55.774+0000 7f34ac000640 1 --2- 172.21.15.32:0/1987084263 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f3484077010 0x7f3484079520 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 crypto rx=0x7f3498000f60 tx=0x7f3498009040 comp rx=0 tx=0).stop 2024-08-03T20:58:55.781 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:55.774+0000 7f34ac000640 1 -- 172.21.15.32:0/1987084263 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f34a40b3e80 msgr2=0x7f34a4072e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:58:55.781 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:55.774+0000 7f34ac000640 1 --2- 172.21.15.32:0/1987084263 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f34a40b3e80 0x7f34a4072e50 secure :-1 s=READY pgs=1850 cs=0 l=1 rev1=1 crypto rx=0x7f3490009230 tx=0x7f3490003910 comp rx=0 tx=0).stop 2024-08-03T20:58:55.782 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:55.774+0000 7f34ac000640 1 -- 172.21.15.32:0/1987084263 shutdown_connections 2024-08-03T20:58:55.782 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:55.774+0000 7f34ac000640 1 --2- 172.21.15.32:0/1987084263 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f3484077010 0x7f3484079520 unknown :-1 s=CLOSED pgs=749 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:55.782 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:55.774+0000 7f34ac000640 1 --2- 172.21.15.32:0/1987084263 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f34a415d2b0 0x7f34a4073390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:55.782 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:55.774+0000 7f34ac000640 1 --2- 172.21.15.32:0/1987084263 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f34a40b3e80 0x7f34a4072e50 unknown :-1 s=CLOSED pgs=1850 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:55.782 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:55.774+0000 7f34ac000640 1 --2- 172.21.15.32:0/1987084263 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f34a406e950 0x7f34a414bcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:55.782 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:55.774+0000 7f34ac000640 1 -- 172.21.15.32:0/1987084263 >> 172.21.15.32:0/1987084263 conn(0x7f34a40b1bd0 msgr2=0x7f34a40d08e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:58:55.782 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:55.774+0000 7f34ac000640 1 -- 172.21.15.32:0/1987084263 shutdown_connections 2024-08-03T20:58:55.782 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:55.774+0000 7f34ac000640 1 -- 172.21.15.32:0/1987084263 wait complete. 2024-08-03T20:58:55.797 INFO:tasks.cephfs.test_damage:Applying mutation 'Corrupt omap key 100.00000000:stray0_head' 2024-08-03T20:58:55.797 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-08-03T20:58:55.797 DEBUG:teuthology.orchestra.run.smithi032:> dd if=/proc/self/mounts of=/dev/stdout 2024-08-03T20:58:55.833 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-08-03T20:58:55.833 DEBUG:teuthology.orchestra.run.smithi032:> dd if=/proc/self/mounts of=/dev/stdout 2024-08-03T20:58:55.909 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-08-03T20:58:56.179 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:56.179+0000 7f9507e00640 1 -- 172.21.15.32:0/2728776867 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f95080b93d0 msgr2=0x7f95080b3240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:58:56.180 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:56.179+0000 7f9507e00640 1 --2- 172.21.15.32:0/2728776867 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f95080b93d0 0x7f95080b3240 secure :-1 s=READY pgs=1856 cs=0 l=1 rev1=1 crypto rx=0x7f94f0005fd0 tx=0x7f94f002a5d0 comp rx=0 tx=0).stop 2024-08-03T20:58:56.180 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:56.179+0000 7f9507e00640 1 -- 172.21.15.32:0/2728776867 shutdown_connections 2024-08-03T20:58:56.180 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:56.179+0000 7f9507e00640 1 --2- 172.21.15.32:0/2728776867 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f95080cf6f0 0x7f95080cfb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:56.180 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:56.179+0000 7f9507e00640 1 --2- 172.21.15.32:0/2728776867 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f95080b93d0 0x7f95080b3240 unknown :-1 s=CLOSED pgs=1856 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:56.180 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:56.179+0000 7f9507e00640 1 -- 172.21.15.32:0/2728776867 >> 172.21.15.32:0/2728776867 conn(0x7f95080d4080 msgr2=0x7f95080d4490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:58:56.180 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:56.179+0000 7f9507e00640 1 -- 172.21.15.32:0/2728776867 shutdown_connections 2024-08-03T20:58:56.180 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:56.179+0000 7f9507e00640 1 -- 172.21.15.32:0/2728776867 wait complete. 2024-08-03T20:58:56.181 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:56.181+0000 7f9507e00640 1 Processor -- start 2024-08-03T20:58:56.182 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:56.181+0000 7f9507e00640 1 -- start start 2024-08-03T20:58:56.183 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:56.183+0000 7f9507e00640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f95080b93d0 0x7f950815bce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:58:56.184 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:56.183+0000 7f9507e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f95080cf6f0 0x7f95080b6030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:58:56.184 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:56.183+0000 7f9507e00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f95080d4ba0 0x7f95080b6570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:58:56.184 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:56.183+0000 7f9507e00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f95080d08f0 con 0x7f95080cf6f0 2024-08-03T20:58:56.184 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:56.183+0000 7f9507e00640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f95080d0770 con 0x7f95080d4ba0 2024-08-03T20:58:56.184 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:56.183+0000 7f9507e00640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7f95080d0a70 con 0x7f95080b93d0 2024-08-03T20:58:56.184 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:56.183+0000 7f94fe000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f95080cf6f0 0x7f95080b6030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:58:56.184 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:56.183+0000 7f94fe000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f95080cf6f0 0x7f95080b6030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:42058/0 (socket says 172.21.15.32:42058) 2024-08-03T20:58:56.184 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:56.183+0000 7f94fe000640 1 -- 172.21.15.32:0/3791311621 learned_addr learned my addr 172.21.15.32:0/3791311621 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:58:56.185 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:56.183+0000 7f9506a00640 1 --2- 172.21.15.32:0/3791311621 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f95080b93d0 0x7f950815bce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:58:56.185 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:56.183+0000 7f9507400640 1 --2- 172.21.15.32:0/3791311621 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f95080d4ba0 0x7f95080b6570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:58:56.185 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:56.184+0000 7f94fe000640 1 -- 172.21.15.32:0/3791311621 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f95080b93d0 msgr2=0x7f950815bce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:58:56.185 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:56.184+0000 7f94fe000640 1 --2- 172.21.15.32:0/3791311621 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f95080b93d0 0x7f950815bce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:56.186 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:56.184+0000 7f94fe000640 1 -- 172.21.15.32:0/3791311621 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f95080d4ba0 msgr2=0x7f95080b6570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:58:56.186 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:56.184+0000 7f94fe000640 1 --2- 172.21.15.32:0/3791311621 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f95080d4ba0 0x7f95080b6570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:56.186 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:56.184+0000 7f94fe000640 1 -- 172.21.15.32:0/3791311621 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f95081e9ff0 con 0x7f95080cf6f0 2024-08-03T20:58:56.186 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:56.184+0000 7f94fe000640 1 --2- 172.21.15.32:0/3791311621 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f95080cf6f0 0x7f95080b6030 secure :-1 s=READY pgs=1857 cs=0 l=1 rev1=1 crypto rx=0x7f94f0034040 tx=0x7f94f00038a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:58:56.186 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:56.184+0000 7f94ff400640 1 -- 172.21.15.32:0/3791311621 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94f003e030 con 0x7f95080cf6f0 2024-08-03T20:58:56.186 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:56.184+0000 7f94ff400640 1 -- 172.21.15.32:0/3791311621 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f94f0003ca0 con 0x7f95080cf6f0 2024-08-03T20:58:56.186 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:56.184+0000 7f94ff400640 1 -- 172.21.15.32:0/3791311621 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94f0035040 con 0x7f95080cf6f0 2024-08-03T20:58:56.186 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:56.184+0000 7f9507e00640 1 -- 172.21.15.32:0/3791311621 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f950815e140 con 0x7f95080cf6f0 2024-08-03T20:58:56.186 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:56.184+0000 7f9507e00640 1 -- 172.21.15.32:0/3791311621 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f95081e8a80 con 0x7f95080cf6f0 2024-08-03T20:58:56.187 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:56.185+0000 7f9507e00640 1 -- 172.21.15.32:0/3791311621 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f95080d0f00 con 0x7f95080cf6f0 2024-08-03T20:58:56.187 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:56.187+0000 7f94ff400640 1 -- 172.21.15.32:0/3791311621 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f94f0041030 con 0x7f95080cf6f0 2024-08-03T20:58:56.188 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:56.187+0000 7f94ff400640 1 --2- 172.21.15.32:0/3791311621 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f94e4077040 0x7f94e4079550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:58:56.188 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:56.188+0000 7f94ff400640 1 -- 172.21.15.32:0/3791311621 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(116..116 src has 1..116) ==== 8520+0+0 (secure 0 0 0) 0x7f94f00b9a00 con 0x7f95080cf6f0 2024-08-03T20:58:56.188 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:56.188+0000 7f9506a00640 1 --2- 172.21.15.32:0/3791311621 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f94e4077040 0x7f94e4079550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:58:56.189 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:56.189+0000 7f9506a00640 1 --2- 172.21.15.32:0/3791311621 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f94e4077040 0x7f94e4079550 secure :-1 s=READY pgs=750 cs=0 l=1 rev1=1 crypto rx=0x7f94f80015e0 tx=0x7f94f8009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:58:56.192 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:56.192+0000 7f94ff400640 1 -- 172.21.15.32:0/3791311621 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7f94f00820e0 con 0x7f95080cf6f0 2024-08-03T20:58:56.407 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:56.407+0000 7f9507e00640 1 -- 172.21.15.32:0/3791311621 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f95080d1630 con 0x7f95080cf6f0 2024-08-03T20:58:56.408 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:56.407+0000 7f94ff400640 1 -- 172.21.15.32:0/3791311621 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 62 entries v116) ==== 81+0+3772 (secure 0 0 0) 0x7f94f008a320 con 0x7f95080cf6f0 2024-08-03T20:58:56.408 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6835/2156596385 2024-08-04T20:58:35.073838+0000 2024-08-03T20:58:56.408 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6833/2156596385 2024-08-04T20:58:35.073838+0000 2024-08-03T20:58:56.408 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6835/3669493978 2024-08-04T20:58:30.064676+0000 2024-08-03T20:58:56.409 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6833/2178506907 2024-08-04T20:56:40.045549+0000 2024-08-03T20:58:56.409 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6835/2278473729 2024-08-04T20:56:13.041693+0000 2024-08-03T20:58:56.409 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6835/2345472481 2024-08-04T20:57:25.968590+0000 2024-08-03T20:58:56.409 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6833/3940515773 2024-08-04T20:58:24.290690+0000 2024-08-03T20:58:56.409 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6833/2600386707 2024-08-04T20:57:50.063964+0000 2024-08-03T20:58:56.409 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6835/2178506907 2024-08-04T20:56:40.045549+0000 2024-08-03T20:58:56.409 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6833/2292690833 2024-08-04T20:55:20.021329+0000 2024-08-03T20:58:56.409 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6835/2803901497 2024-08-04T20:55:55.181144+0000 2024-08-03T20:58:56.409 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6835/2194902489 2024-08-04T20:55:00.359126+0000 2024-08-03T20:58:56.409 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6833/2803901497 2024-08-04T20:55:55.181144+0000 2024-08-03T20:58:56.409 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6835/3191654276 2024-08-04T20:55:42.433364+0000 2024-08-03T20:58:56.409 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6833/2242533881 2024-08-04T20:56:48.048099+0000 2024-08-03T20:58:56.409 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6833/2283924374 2024-08-04T20:57:34.396782+0000 2024-08-03T20:58:56.410 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6835/1172050551 2024-08-04T20:55:26.214113+0000 2024-08-03T20:58:56.410 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6835/1016338829 2024-08-04T20:55:06.071485+0000 2024-08-03T20:58:56.410 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6833/1474984145 2024-08-04T20:57:40.052808+0000 2024-08-03T20:58:56.410 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6833/3669493978 2024-08-04T20:58:30.064676+0000 2024-08-03T20:58:56.410 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6835/756683988 2024-08-04T20:55:14.119228+0000 2024-08-03T20:58:56.410 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6833/2194902489 2024-08-04T20:55:00.359126+0000 2024-08-03T20:58:56.410 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6835/2806168786 2024-08-04T20:56:37.279160+0000 2024-08-03T20:58:56.410 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6833/2278473729 2024-08-04T20:56:13.041693+0000 2024-08-03T20:58:56.410 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6833/3191654276 2024-08-04T20:55:42.433364+0000 2024-08-03T20:58:56.410 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6835/3431300225 2024-08-04T20:56:21.088927+0000 2024-08-03T20:58:56.410 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6833/4172198298 2024-08-04T20:56:04.256182+0000 2024-08-03T20:58:56.411 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6833/3431300225 2024-08-04T20:56:21.088927+0000 2024-08-03T20:58:56.411 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6835/2292690833 2024-08-04T20:55:20.021329+0000 2024-08-03T20:58:56.411 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6833/2806168786 2024-08-04T20:56:37.279160+0000 2024-08-03T20:58:56.411 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6833/1172050551 2024-08-04T20:55:26.214113+0000 2024-08-03T20:58:56.411 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6835/4172198298 2024-08-04T20:56:04.256182+0000 2024-08-03T20:58:56.411 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6835/3409752696 2024-08-04T20:58:18.243140+0000 2024-08-03T20:58:56.411 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6833/2274054575 2024-08-04T20:57:55.062490+0000 2024-08-03T20:58:56.411 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6833/1016338829 2024-08-04T20:55:06.071485+0000 2024-08-03T20:58:56.411 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6833/1707454554 2024-08-04T20:54:38.409411+0000 2024-08-03T20:58:56.411 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6835/2242533881 2024-08-04T20:56:48.048099+0000 2024-08-03T20:58:56.411 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6833/739874257 2024-08-04T20:56:56.101271+0000 2024-08-03T20:58:56.411 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6835/739874257 2024-08-04T20:56:56.101271+0000 2024-08-03T20:58:56.412 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6833/2362145329 2024-08-04T20:57:02.150199+0000 2024-08-03T20:58:56.412 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6833/3771304550 2024-08-04T20:55:58.212752+0000 2024-08-03T20:58:56.412 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6835/2362145329 2024-08-04T20:57:02.150199+0000 2024-08-03T20:58:56.412 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6835/3940515773 2024-08-04T20:58:24.290690+0000 2024-08-03T20:58:56.412 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6833/1885034124 2024-08-04T20:57:08.190707+0000 2024-08-03T20:58:56.412 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6833/511221267 2024-08-04T20:57:45.062448+0000 2024-08-03T20:58:56.412 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6835/3771304550 2024-08-04T20:55:58.212752+0000 2024-08-03T20:58:56.412 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6835/1885034124 2024-08-04T20:57:08.190707+0000 2024-08-03T20:58:56.412 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6833/2345472481 2024-08-04T20:57:25.968590+0000 2024-08-03T20:58:56.412 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6833/265163622 2024-08-04T20:57:28.354509+0000 2024-08-03T20:58:56.412 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6835/265163622 2024-08-04T20:57:28.354509+0000 2024-08-03T20:58:56.412 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6835/2274054575 2024-08-04T20:57:55.062490+0000 2024-08-03T20:58:56.412 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6835/2283924374 2024-08-04T20:57:34.396782+0000 2024-08-03T20:58:56.412 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6833/756683988 2024-08-04T20:55:14.119228+0000 2024-08-03T20:58:56.413 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6835/1474984145 2024-08-04T20:57:40.052808+0000 2024-08-03T20:58:56.413 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6835/511221267 2024-08-04T20:57:45.062448+0000 2024-08-03T20:58:56.413 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6835/1707454554 2024-08-04T20:54:38.409411+0000 2024-08-03T20:58:56.413 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6835/2600386707 2024-08-04T20:57:50.063964+0000 2024-08-03T20:58:56.413 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6833/675070591 2024-08-04T20:58:00.068114+0000 2024-08-03T20:58:56.413 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6833/3340816283 2024-08-04T20:58:15.214542+0000 2024-08-03T20:58:56.413 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6835/3340816283 2024-08-04T20:58:15.214542+0000 2024-08-03T20:58:56.413 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6835/675070591 2024-08-04T20:58:00.068114+0000 2024-08-03T20:58:56.413 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6833/3409752696 2024-08-04T20:58:18.243140+0000 2024-08-03T20:58:56.414 INFO:teuthology.orchestra.run.smithi032.stderr:listed 62 entries 2024-08-03T20:58:56.414 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:56.410+0000 7f9507e00640 1 -- 172.21.15.32:0/3791311621 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f94e4077040 msgr2=0x7f94e4079550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:58:56.414 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:56.410+0000 7f9507e00640 1 --2- 172.21.15.32:0/3791311621 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f94e4077040 0x7f94e4079550 secure :-1 s=READY pgs=750 cs=0 l=1 rev1=1 crypto rx=0x7f94f80015e0 tx=0x7f94f8009040 comp rx=0 tx=0).stop 2024-08-03T20:58:56.414 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:56.410+0000 7f9507e00640 1 -- 172.21.15.32:0/3791311621 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f95080cf6f0 msgr2=0x7f95080b6030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:58:56.414 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:56.410+0000 7f9507e00640 1 --2- 172.21.15.32:0/3791311621 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f95080cf6f0 0x7f95080b6030 secure :-1 s=READY pgs=1857 cs=0 l=1 rev1=1 crypto rx=0x7f94f0034040 tx=0x7f94f00038a0 comp rx=0 tx=0).stop 2024-08-03T20:58:56.415 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:56.410+0000 7f9507e00640 1 -- 172.21.15.32:0/3791311621 shutdown_connections 2024-08-03T20:58:56.415 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:56.410+0000 7f9507e00640 1 --2- 172.21.15.32:0/3791311621 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f94e4077040 0x7f94e4079550 unknown :-1 s=CLOSED pgs=750 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:56.415 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:56.410+0000 7f9507e00640 1 --2- 172.21.15.32:0/3791311621 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f95080d4ba0 0x7f95080b6570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:56.415 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:56.410+0000 7f9507e00640 1 --2- 172.21.15.32:0/3791311621 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f95080cf6f0 0x7f95080b6030 unknown :-1 s=CLOSED pgs=1857 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:56.415 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:56.410+0000 7f9507e00640 1 --2- 172.21.15.32:0/3791311621 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f95080b93d0 0x7f950815bce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:56.415 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:56.410+0000 7f9507e00640 1 -- 172.21.15.32:0/3791311621 >> 172.21.15.32:0/3791311621 conn(0x7f95080d4080 msgr2=0x7f95080b1080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:58:56.415 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:56.410+0000 7f9507e00640 1 -- 172.21.15.32:0/3791311621 shutdown_connections 2024-08-03T20:58:56.415 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:56.410+0000 7f9507e00640 1 -- 172.21.15.32:0/3791311621 wait complete. 2024-08-03T20:58:56.436 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-08-03T20:58:56.437 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-08-03T20:58:56.437 DEBUG:teuthology.orchestra.run.smithi032:> sudo umount /home/ubuntu/cephtest/mnt.0 -f 2024-08-03T20:58:56.497 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi032.front.sepia.ceph.com 2024-08-03T20:58:56.497 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-08-03T20:58:56.497 DEBUG:teuthology.orchestra.run.smithi032:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-08-03T20:58:56.534 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-08-03T20:58:56.804 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:56.803+0000 7f2cf5c00640 1 -- 172.21.15.32:0/2392813807 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2cd4003a00 msgr2=0x7f2cd4003e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:58:56.804 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:56.803+0000 7f2cf5c00640 1 --2- 172.21.15.32:0/2392813807 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2cd4003a00 0x7f2cd4003e70 secure :-1 s=READY pgs=1859 cs=0 l=1 rev1=1 crypto rx=0x7f2ce000c290 tx=0x7f2cd4005830 comp rx=0 tx=0).stop 2024-08-03T20:58:56.804 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:56.803+0000 7f2cef400640 1 -- 172.21.15.32:0/2392813807 reap_dead start 2024-08-03T20:58:56.805 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:56.804+0000 7f2cf5c00640 1 -- 172.21.15.32:0/2392813807 shutdown_connections 2024-08-03T20:58:56.805 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:56.804+0000 7f2cf5c00640 1 -- 172.21.15.32:0/2392813807 >> 172.21.15.32:0/2392813807 conn(0x7f2cf00b1bd0 msgr2=0x7f2cf00b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:58:56.805 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:56.804+0000 7f2cf5c00640 1 -- 172.21.15.32:0/2392813807 shutdown_connections 2024-08-03T20:58:56.805 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:56.804+0000 7f2cf5c00640 1 -- 172.21.15.32:0/2392813807 wait complete. 2024-08-03T20:58:56.806 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:56.806+0000 7f2cf5c00640 1 Processor -- start 2024-08-03T20:58:56.806 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:56.806+0000 7f2cf5c00640 1 -- start start 2024-08-03T20:58:56.808 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:56.807+0000 7f2cf5c00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2cd4003a00 0x7f2cf0143f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:58:56.808 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:56.807+0000 7f2cf5c00640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f2cf015de20 0x7f2cf015bd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:58:56.808 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:56.807+0000 7f2cf5c00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f2cf013f4e0 0x7f2cf013f8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:58:56.808 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:56.807+0000 7f2cf5c00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f2cf00cf840 con 0x7f2cd4003a00 2024-08-03T20:58:56.808 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:56.807+0000 7f2cf5c00640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f2cf0145f30 con 0x7f2cf013f4e0 2024-08-03T20:58:56.808 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:56.807+0000 7f2cf5c00640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7f2cf00cf9c0 con 0x7f2cf015de20 2024-08-03T20:58:56.809 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:56.808+0000 7f2ceea00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2cd4003a00 0x7f2cf0143f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:58:56.809 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:56.808+0000 7f2ceea00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2cd4003a00 0x7f2cf0143f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:42066/0 (socket says 172.21.15.32:42066) 2024-08-03T20:58:56.809 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:56.808+0000 7f2ceea00640 1 -- 172.21.15.32:0/4292355421 learned_addr learned my addr 172.21.15.32:0/4292355421 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:58:56.810 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:56.808+0000 7f2cef400640 1 --2- 172.21.15.32:0/4292355421 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f2cf013f4e0 0x7f2cf013f8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:58:56.810 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:56.808+0000 7f2ce6000640 1 --2- 172.21.15.32:0/4292355421 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f2cf015de20 0x7f2cf015bd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:58:56.810 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:56.808+0000 7f2ceea00640 1 -- 172.21.15.32:0/4292355421 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f2cf015de20 msgr2=0x7f2cf015bd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:58:56.810 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:56.808+0000 7f2ceea00640 1 --2- 172.21.15.32:0/4292355421 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f2cf015de20 0x7f2cf015bd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:56.810 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:56.808+0000 7f2ceea00640 1 -- 172.21.15.32:0/4292355421 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f2cf013f4e0 msgr2=0x7f2cf013f8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:58:56.810 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:56.808+0000 7f2ceea00640 1 --2- 172.21.15.32:0/4292355421 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f2cf013f4e0 0x7f2cf013f8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:56.810 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:56.808+0000 7f2ceea00640 1 -- 172.21.15.32:0/4292355421 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2cf013fe30 con 0x7f2cd4003a00 2024-08-03T20:58:56.810 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:56.808+0000 7f2ce6000640 1 --2- 172.21.15.32:0/4292355421 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f2cf015de20 0x7f2cf015bd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-03T20:58:56.810 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:56.808+0000 7f2cef400640 1 --2- 172.21.15.32:0/4292355421 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f2cf013f4e0 0x7f2cf013f8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-03T20:58:56.811 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:56.809+0000 7f2ceea00640 1 --2- 172.21.15.32:0/4292355421 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2cd4003a00 0x7f2cf0143f70 secure :-1 s=READY pgs=1860 cs=0 l=1 rev1=1 crypto rx=0x7f2ce000c3b0 tx=0x7f2ce0006d70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:58:56.811 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:56.809+0000 7f2ce7400640 1 -- 172.21.15.32:0/4292355421 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ce0031040 con 0x7f2cd4003a00 2024-08-03T20:58:56.811 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:56.809+0000 7f2ce7400640 1 -- 172.21.15.32:0/4292355421 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f2ce00035f0 con 0x7f2cd4003a00 2024-08-03T20:58:56.811 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:56.809+0000 7f2ce7400640 1 -- 172.21.15.32:0/4292355421 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ce0003a00 con 0x7f2cd4003a00 2024-08-03T20:58:56.811 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:56.809+0000 7f2cf5c00640 1 -- 172.21.15.32:0/4292355421 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2cf015d990 con 0x7f2cd4003a00 2024-08-03T20:58:56.811 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:56.809+0000 7f2cf5c00640 1 -- 172.21.15.32:0/4292355421 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2cf01f5ad0 con 0x7f2cd4003a00 2024-08-03T20:58:56.811 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:56.810+0000 7f2cf5c00640 1 -- 172.21.15.32:0/4292355421 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2cf00747a0 con 0x7f2cd4003a00 2024-08-03T20:58:56.812 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:56.811+0000 7f2ce7400640 1 -- 172.21.15.32:0/4292355421 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f2ce0034070 con 0x7f2cd4003a00 2024-08-03T20:58:56.813 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:56.812+0000 7f2ce7400640 1 --2- 172.21.15.32:0/4292355421 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f2ccc077010 0x7f2ccc079520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:58:56.813 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:56.812+0000 7f2ce7400640 1 -- 172.21.15.32:0/4292355421 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(116..116 src has 1..116) ==== 8520+0+0 (secure 0 0 0) 0x7f2ce00b95c0 con 0x7f2cd4003a00 2024-08-03T20:58:56.813 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:56.813+0000 7f2ce6000640 1 --2- 172.21.15.32:0/4292355421 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f2ccc077010 0x7f2ccc079520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:58:56.814 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:56.813+0000 7f2ce6000640 1 --2- 172.21.15.32:0/4292355421 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f2ccc077010 0x7f2ccc079520 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7f2cf006ac30 tx=0x7f2cd8005f90 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:58:56.817 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:56.816+0000 7f2ce7400640 1 -- 172.21.15.32:0/4292355421 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7f2cf013fe30 con 0x7f2cd4003a00 2024-08-03T20:58:57.063 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:57.062+0000 7f2cf5c00640 1 -- 172.21.15.32:0/4292355421 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7f2cf00af8d0 con 0x7f2cd4003a00 2024-08-03T20:58:57.063 INFO:tasks.ceph.mon.a.smithi032.stderr:2024-08-03T20:58:57.063+0000 7f4e6d600640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-08-03T20:58:57.066 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:57.065+0000 7f2ce7400640 1 -- 172.21.15.32:0/4292355421 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v272) ==== 161+0+0 (secure 0 0 0) 0x7f2ce0081c10 con 0x7f2cd4003a00 2024-08-03T20:58:57.066 INFO:teuthology.orchestra.run.smithi032.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-08-03T20:58:57.070 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:57.069+0000 7f2cf5c00640 1 -- 172.21.15.32:0/4292355421 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f2ccc077010 msgr2=0x7f2ccc079520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:58:57.070 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:57.069+0000 7f2cf5c00640 1 --2- 172.21.15.32:0/4292355421 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f2ccc077010 0x7f2ccc079520 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7f2cf006ac30 tx=0x7f2cd8005f90 comp rx=0 tx=0).stop 2024-08-03T20:58:57.070 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:57.069+0000 7f2cf5c00640 1 -- 172.21.15.32:0/4292355421 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2cd4003a00 msgr2=0x7f2cf0143f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:58:57.070 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:57.069+0000 7f2cf5c00640 1 --2- 172.21.15.32:0/4292355421 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2cd4003a00 0x7f2cf0143f70 secure :-1 s=READY pgs=1860 cs=0 l=1 rev1=1 crypto rx=0x7f2ce000c3b0 tx=0x7f2ce0006d70 comp rx=0 tx=0).stop 2024-08-03T20:58:57.071 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:57.069+0000 7f2cf5c00640 1 -- 172.21.15.32:0/4292355421 shutdown_connections 2024-08-03T20:58:57.071 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:57.069+0000 7f2cf5c00640 1 --2- 172.21.15.32:0/4292355421 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f2ccc077010 0x7f2ccc079520 unknown :-1 s=CLOSED pgs=751 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:57.071 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:57.069+0000 7f2cf5c00640 1 --2- 172.21.15.32:0/4292355421 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f2cf013f4e0 0x7f2cf013f8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:57.071 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:57.069+0000 7f2cf5c00640 1 --2- 172.21.15.32:0/4292355421 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f2cf015de20 0x7f2cf015bd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:57.071 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:57.069+0000 7f2cf5c00640 1 --2- 172.21.15.32:0/4292355421 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2cd4003a00 0x7f2cf0143f70 unknown :-1 s=CLOSED pgs=1860 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:57.071 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:57.069+0000 7f2cf5c00640 1 -- 172.21.15.32:0/4292355421 >> 172.21.15.32:0/4292355421 conn(0x7f2cf00b1bd0 msgr2=0x7f2cf00cf0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:58:57.071 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:57.070+0000 7f2cf5c00640 1 -- 172.21.15.32:0/4292355421 shutdown_connections 2024-08-03T20:58:57.071 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:57.070+0000 7f2cf5c00640 1 -- 172.21.15.32:0/4292355421 wait complete. 2024-08-03T20:58:57.074 INFO:tasks.ceph.mds.a.smithi032.stderr: -979> 2024-08-03T20:58:45.067+0000 7ff617400640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-08-03T20:58:57.094 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mds repaired 0 2024-08-03T20:58:57.191 INFO:tasks.ceph.mds.a.smithi032.stdout:starting mds.a at 2024-08-03T20:58:57.330 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:57.329+0000 7fec4b200640 1 -- 172.21.15.32:0/646148588 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fec4c06f440 msgr2=0x7fec4c06f850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:58:57.330 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:57.329+0000 7fec4b200640 1 --2- 172.21.15.32:0/646148588 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fec4c06f440 0x7fec4c06f850 secure :-1 s=READY pgs=1865 cs=0 l=1 rev1=1 crypto rx=0x7fec4c160f00 tx=0x7fec4402a820 comp rx=0 tx=0).stop 2024-08-03T20:58:57.330 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:57.329+0000 7fec4b200640 1 -- 172.21.15.32:0/646148588 shutdown_connections 2024-08-03T20:58:57.330 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:57.329+0000 7fec4b200640 1 --2- 172.21.15.32:0/646148588 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fec4c06f440 0x7fec4c06f850 unknown :-1 s=CLOSED pgs=1865 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:57.330 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:57.329+0000 7fec4b200640 1 --2- 172.21.15.32:0/646148588 >> v2:172.21.15.118:3300/0 conn(0x7fec4c0acd80 0x7fec4c0ad190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:57.330 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:57.329+0000 7fec4b200640 1 -- 172.21.15.32:0/646148588 >> 172.21.15.32:0/646148588 conn(0x7fec4c0b2120 msgr2=0x7fec4c0cfd60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:58:57.330 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:57.329+0000 7fec4b200640 1 -- 172.21.15.32:0/646148588 shutdown_connections 2024-08-03T20:58:57.331 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:57.329+0000 7fec4b200640 1 -- 172.21.15.32:0/646148588 wait complete. 2024-08-03T20:58:57.332 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:57.331+0000 7fec4b200640 1 Processor -- start 2024-08-03T20:58:57.332 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:57.331+0000 7fec4b200640 1 -- start start 2024-08-03T20:58:57.334 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:57.333+0000 7fec4b200640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fec4c06f440 0x7fec4c09d2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:58:57.334 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:57.333+0000 7fec4b200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fec4c0acd80 0x7fec4c153df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:58:57.334 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:57.333+0000 7fec4b200640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fec4c0af020 0x7fec4c15b320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:58:57.335 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:57.333+0000 7fec4b200640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fec4c1601e0 con 0x7fec4c0acd80 2024-08-03T20:58:57.335 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:57.333+0000 7fec4b200640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7fec4c160060 con 0x7fec4c06f440 2024-08-03T20:58:57.335 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:57.333+0000 7fec4b200640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7fec4c160360 con 0x7fec4c0af020 2024-08-03T20:58:57.335 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:57.333+0000 7fec41400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fec4c0acd80 0x7fec4c153df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:58:57.335 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:57.333+0000 7fec41400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fec4c0acd80 0x7fec4c153df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:42104/0 (socket says 172.21.15.32:42104) 2024-08-03T20:58:57.335 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:57.333+0000 7fec41400640 1 -- 172.21.15.32:0/3243548857 learned_addr learned my addr 172.21.15.32:0/3243548857 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:58:57.335 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:57.333+0000 7fec49e00640 1 --2- 172.21.15.32:0/3243548857 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fec4c06f440 0x7fec4c09d2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:58:57.335 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:57.333+0000 7fec4a800640 1 --2- 172.21.15.32:0/3243548857 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fec4c0af020 0x7fec4c15b320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:58:57.335 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:57.334+0000 7fec41400640 1 -- 172.21.15.32:0/3243548857 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fec4c0af020 msgr2=0x7fec4c15b320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:58:57.336 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:57.334+0000 7fec41400640 1 --2- 172.21.15.32:0/3243548857 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fec4c0af020 0x7fec4c15b320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:57.336 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:57.334+0000 7fec41400640 1 -- 172.21.15.32:0/3243548857 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fec4c06f440 msgr2=0x7fec4c09d2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:58:57.336 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:57.334+0000 7fec41400640 1 --2- 172.21.15.32:0/3243548857 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fec4c06f440 0x7fec4c09d2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:57.336 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:57.334+0000 7fec41400640 1 -- 172.21.15.32:0/3243548857 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fec4c0b5c40 con 0x7fec4c0acd80 2024-08-03T20:58:57.336 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:57.334+0000 7fec41400640 1 --2- 172.21.15.32:0/3243548857 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fec4c0acd80 0x7fec4c153df0 secure :-1 s=READY pgs=1866 cs=0 l=1 rev1=1 crypto rx=0x7fec3400e7c0 tx=0x7fec3400ec80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:58:57.337 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:57.334+0000 7fec42a00640 1 -- 172.21.15.32:0/3243548857 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec3400ccc0 con 0x7fec4c0acd80 2024-08-03T20:58:57.337 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:57.334+0000 7fec42a00640 1 -- 172.21.15.32:0/3243548857 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fec34010430 con 0x7fec4c0acd80 2024-08-03T20:58:57.337 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:57.334+0000 7fec42a00640 1 -- 172.21.15.32:0/3243548857 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec34010860 con 0x7fec4c0acd80 2024-08-03T20:58:57.337 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:57.334+0000 7fec4b200640 1 -- 172.21.15.32:0/3243548857 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fec4c09c5e0 con 0x7fec4c0acd80 2024-08-03T20:58:57.337 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:57.335+0000 7fec4b200640 1 -- 172.21.15.32:0/3243548857 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fec4c09cba0 con 0x7fec4c0acd80 2024-08-03T20:58:57.337 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:57.336+0000 7fec4b200640 1 -- 172.21.15.32:0/3243548857 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fec4c0ad190 con 0x7fec4c0acd80 2024-08-03T20:58:57.337 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:57.337+0000 7fec42a00640 1 -- 172.21.15.32:0/3243548857 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7fec34010a00 con 0x7fec4c0acd80 2024-08-03T20:58:57.338 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:57.338+0000 7fec42a00640 1 --2- 172.21.15.32:0/3243548857 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fec28077010 0x7fec28079520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:58:57.338 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:57.338+0000 7fec42a00640 1 -- 172.21.15.32:0/3243548857 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(117..117 src has 1..117) ==== 8606+0+0 (secure 0 0 0) 0x7fec34099e00 con 0x7fec4c0acd80 2024-08-03T20:58:57.339 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:57.338+0000 7fec49e00640 1 --2- 172.21.15.32:0/3243548857 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fec28077010 0x7fec28079520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:58:57.339 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:57.339+0000 7fec49e00640 1 --2- 172.21.15.32:0/3243548857 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fec28077010 0x7fec28079520 secure :-1 s=READY pgs=752 cs=0 l=1 rev1=1 crypto rx=0x7fec3c000f60 tx=0x7fec3c009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:58:57.342 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:57.342+0000 7fec42a00640 1 -- 172.21.15.32:0/3243548857 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7fec34061dc0 con 0x7fec4c0acd80 2024-08-03T20:58:57.630 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:57.629+0000 7fec4b200640 1 -- 172.21.15.32:0/3243548857 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "mds repaired", "role": "0"} v 0) -- 0x7fec4c156860 con 0x7fec4c0acd80 2024-08-03T20:58:58.108 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.108+0000 7fec42a00640 1 -- 172.21.15.32:0/3243548857 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "mds repaired", "role": "0"}]=0 nothing to do: rank is not damaged v273) ==== 107+0+0 (secure 0 0 0) 0x7fec34066310 con 0x7fec4c0acd80 2024-08-03T20:58:58.109 INFO:teuthology.orchestra.run.smithi032.stderr:nothing to do: rank is not damaged 2024-08-03T20:58:58.111 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.111+0000 7fec4b200640 1 -- 172.21.15.32:0/3243548857 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fec28077010 msgr2=0x7fec28079520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:58:58.112 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.111+0000 7fec4b200640 1 --2- 172.21.15.32:0/3243548857 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fec28077010 0x7fec28079520 secure :-1 s=READY pgs=752 cs=0 l=1 rev1=1 crypto rx=0x7fec3c000f60 tx=0x7fec3c009040 comp rx=0 tx=0).stop 2024-08-03T20:58:58.112 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.111+0000 7fec4b200640 1 -- 172.21.15.32:0/3243548857 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fec4c0acd80 msgr2=0x7fec4c153df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:58:58.112 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.111+0000 7fec4b200640 1 --2- 172.21.15.32:0/3243548857 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fec4c0acd80 0x7fec4c153df0 secure :-1 s=READY pgs=1866 cs=0 l=1 rev1=1 crypto rx=0x7fec3400e7c0 tx=0x7fec3400ec80 comp rx=0 tx=0).stop 2024-08-03T20:58:58.112 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.111+0000 7fec4b200640 1 -- 172.21.15.32:0/3243548857 shutdown_connections 2024-08-03T20:58:58.112 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.111+0000 7fec4b200640 1 --2- 172.21.15.32:0/3243548857 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fec28077010 0x7fec28079520 unknown :-1 s=CLOSED pgs=752 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:58.112 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.111+0000 7fec4b200640 1 --2- 172.21.15.32:0/3243548857 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fec4c0af020 0x7fec4c15b320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:58.112 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.111+0000 7fec4b200640 1 --2- 172.21.15.32:0/3243548857 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fec4c0acd80 0x7fec4c153df0 unknown :-1 s=CLOSED pgs=1866 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:58.112 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.111+0000 7fec4b200640 1 --2- 172.21.15.32:0/3243548857 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fec4c06f440 0x7fec4c09d2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:58.112 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.111+0000 7fec4b200640 1 -- 172.21.15.32:0/3243548857 >> 172.21.15.32:0/3243548857 conn(0x7fec4c0b2120 msgr2=0x7fec4c068ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:58:58.112 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.111+0000 7fec4b200640 1 -- 172.21.15.32:0/3243548857 shutdown_connections 2024-08-03T20:58:58.112 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.111+0000 7fec4b200640 1 -- 172.21.15.32:0/3243548857 wait complete. 2024-08-03T20:58:58.144 DEBUG:teuthology.orchestra.run.smithi032:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata import - 2024-08-03T20:58:58.287 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.287+0000 7f52d9fe6f40 1 Processor -- start 2024-08-03T20:58:58.287 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.287+0000 7f52d9fe6f40 1 -- start start 2024-08-03T20:58:58.289 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.288+0000 7f52d9fe6f40 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55f084cbb360 0x55f084cd0c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:58:58.289 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.288+0000 7f52d9fe6f40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55f084cd8cb0 0x55f084cd6d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:58:58.289 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.288+0000 7f52d9fe6f40 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55f084cd93f0 0x55f084cd72a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:58:58.289 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.288+0000 7f52d9fe6f40 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55f084b52e30 con 0x55f084cbb360 2024-08-03T20:58:58.289 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.288+0000 7f52d9fe6f40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x55f084bb0bf0 con 0x55f084cd8cb0 2024-08-03T20:58:58.290 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.288+0000 7f52d9fe6f40 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x55f084cbc440 con 0x55f084cd93f0 2024-08-03T20:58:58.290 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.288+0000 7f52d3200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55f084cbb360 0x55f084cd0c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:58:58.290 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.288+0000 7f52d3200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55f084cbb360 0x55f084cd0c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:42120/0 (socket says 172.21.15.32:42120) 2024-08-03T20:58:58.290 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.288+0000 7f52d3200640 1 -- 172.21.15.32:0/2236123966 learned_addr learned my addr 172.21.15.32:0/2236123966 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:58:58.290 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.289+0000 7f52d3200640 1 -- 172.21.15.32:0/2236123966 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55f084cd93f0 msgr2=0x55f084cd72a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:58:58.290 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.289+0000 7f52d3200640 1 --2- 172.21.15.32:0/2236123966 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55f084cd93f0 0x55f084cd72a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:58.290 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.289+0000 7f52d3200640 1 -- 172.21.15.32:0/2236123966 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55f084cd8cb0 msgr2=0x55f084cd6d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:58:58.290 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.289+0000 7f52d3200640 1 --2- 172.21.15.32:0/2236123966 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55f084cd8cb0 0x55f084cd6d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:58.290 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.289+0000 7f52d3200640 1 -- 172.21.15.32:0/2236123966 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f084c4c330 con 0x55f084cbb360 2024-08-03T20:58:58.290 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.289+0000 7f52d3200640 1 --2- 172.21.15.32:0/2236123966 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55f084cbb360 0x55f084cd0c40 secure :-1 s=READY pgs=1869 cs=0 l=1 rev1=1 crypto rx=0x7f52c4025de0 tx=0x7f52c4002fb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=329b7ec7b2d37360 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:58:58.292 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.289+0000 7f52d1400640 1 -- 172.21.15.32:0/2236123966 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52c404dcd0 con 0x55f084cbb360 2024-08-03T20:58:58.292 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.289+0000 7f52d1400640 1 -- 172.21.15.32:0/2236123966 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f52c4056470 con 0x55f084cbb360 2024-08-03T20:58:58.292 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.289+0000 7f52d1400640 1 -- 172.21.15.32:0/2236123966 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52c40568a0 con 0x55f084cbb360 2024-08-03T20:58:58.292 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.290+0000 7f52d9fe6f40 1 -- 172.21.15.32:0/2236123966 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55f084cbb360 msgr2=0x55f084cd0c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:58:58.292 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.290+0000 7f52d9fe6f40 1 --2- 172.21.15.32:0/2236123966 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55f084cbb360 0x55f084cd0c40 secure :-1 s=READY pgs=1869 cs=0 l=1 rev1=1 crypto rx=0x7f52c4025de0 tx=0x7f52c4002fb0 comp rx=0 tx=0).stop 2024-08-03T20:58:58.292 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.290+0000 7f52d9fe6f40 1 -- 172.21.15.32:0/2236123966 shutdown_connections 2024-08-03T20:58:58.292 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.290+0000 7f52d9fe6f40 1 --2- 172.21.15.32:0/2236123966 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55f084cd8cb0 0x55f084cd6d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:58.292 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.290+0000 7f52d9fe6f40 1 --2- 172.21.15.32:0/2236123966 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55f084cd93f0 0x55f084cd72a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:58.292 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.290+0000 7f52d9fe6f40 1 --2- 172.21.15.32:0/2236123966 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55f084cbb360 0x55f084cd0c40 unknown :-1 s=CLOSED pgs=1869 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:58.292 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.290+0000 7f52d9fe6f40 1 -- 172.21.15.32:0/2236123966 >> 172.21.15.32:0/2236123966 conn(0x55f084b24ac0 msgr2=0x55f084cd7ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:58:58.292 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.290+0000 7f52d9fe6f40 1 -- 172.21.15.32:0/2236123966 shutdown_connections 2024-08-03T20:58:58.292 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.290+0000 7f52d9fe6f40 1 -- 172.21.15.32:0/2236123966 wait complete. 2024-08-03T20:58:58.292 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.291+0000 7f52d9fe6f40 1 Processor -- start 2024-08-03T20:58:58.293 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.291+0000 7f52d9fe6f40 1 -- start start 2024-08-03T20:58:58.294 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.293+0000 7f52d9fe6f40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55f084cbb360 0x55f084cd5440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:58:58.294 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.293+0000 7f52d9fe6f40 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55f084cd8cb0 0x55f084cb2be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:58:58.294 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.293+0000 7f52d9fe6f40 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55f084cd93f0 0x55f084c453d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:58:58.294 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.293+0000 7f52d9fe6f40 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55f084c4b230 con 0x55f084cd8cb0 2024-08-03T20:58:58.294 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.293+0000 7f52d9fe6f40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x55f084c4c5d0 con 0x55f084cbb360 2024-08-03T20:58:58.294 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.293+0000 7f52d9fe6f40 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x55f084c4c8b0 con 0x55f084cd93f0 2024-08-03T20:58:58.294 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.293+0000 7f52d2800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55f084cd8cb0 0x55f084cb2be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:58:58.294 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.293+0000 7f52d3200640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55f084cbb360 0x55f084cd5440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:58:58.294 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.293+0000 7f52d2800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55f084cd8cb0 0x55f084cb2be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:42124/0 (socket says 172.21.15.32:42124) 2024-08-03T20:58:58.294 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.293+0000 7f52d2800640 1 -- 172.21.15.32:0/2665912631 learned_addr learned my addr 172.21.15.32:0/2665912631 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:58:58.295 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.293+0000 7f52d3c00640 1 --2- 172.21.15.32:0/2665912631 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55f084cd93f0 0x55f084c453d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:58:58.295 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.294+0000 7f52d2800640 1 -- 172.21.15.32:0/2665912631 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55f084cd93f0 msgr2=0x55f084c453d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:58:58.296 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.294+0000 7f52d2800640 1 --2- 172.21.15.32:0/2665912631 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55f084cd93f0 0x55f084c453d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:58.296 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.294+0000 7f52d2800640 1 -- 172.21.15.32:0/2665912631 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55f084cbb360 msgr2=0x55f084cd5440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:58:58.296 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.294+0000 7f52d2800640 1 --2- 172.21.15.32:0/2665912631 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55f084cbb360 0x55f084cd5440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:58.296 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.294+0000 7f52d2800640 1 -- 172.21.15.32:0/2665912631 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f084c45910 con 0x55f084cd8cb0 2024-08-03T20:58:58.296 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.294+0000 7f52d2800640 1 --2- 172.21.15.32:0/2665912631 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55f084cd8cb0 0x55f084cb2be0 secure :-1 s=READY pgs=1870 cs=0 l=1 rev1=1 crypto rx=0x7f52c000bd90 tx=0x7f52c00095f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:58:58.296 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.294+0000 7f52cbe00640 1 -- 172.21.15.32:0/2665912631 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52c000e620 con 0x55f084cd8cb0 2024-08-03T20:58:58.296 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.294+0000 7f52cbe00640 1 -- 172.21.15.32:0/2665912631 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f52c0006050 con 0x55f084cd8cb0 2024-08-03T20:58:58.296 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.294+0000 7f52d9fe6f40 1 -- 172.21.15.32:0/2665912631 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x55f084c440a0 con 0x55f084cd8cb0 2024-08-03T20:58:58.296 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.294+0000 7f52cbe00640 1 -- 172.21.15.32:0/2665912631 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52c000ea60 con 0x55f084cd8cb0 2024-08-03T20:58:58.296 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.294+0000 7f52d9fe6f40 1 -- 172.21.15.32:0/2665912631 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f084c44bd0 con 0x55f084cd8cb0 2024-08-03T20:58:58.297 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.297+0000 7f52cbe00640 1 -- 172.21.15.32:0/2665912631 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f52c000ec00 con 0x55f084cd8cb0 2024-08-03T20:58:58.298 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.298+0000 7f52cbe00640 1 --2- 172.21.15.32:0/2665912631 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f52b40759f0 0x7f52b4077f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:58:58.298 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.298+0000 7f52cbe00640 1 -- 172.21.15.32:0/2665912631 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(117..117 src has 1..117) ==== 8606+0+0 (secure 0 0 0) 0x7f52c0097fe0 con 0x55f084cd8cb0 2024-08-03T20:58:58.298 INFO:teuthology.orchestra.run.smithi032.stdout:Importing pool 2024-08-03T20:58:58.299 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.298+0000 7f52d9fe6f40 1 --2- 172.21.15.32:0/2665912631 >> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] conn(0x55f084cd45e0 0x55f084cd1150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:58:58.299 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.298+0000 7f52d9fe6f40 1 -- 172.21.15.32:0/2665912631 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:1 12.10 12:09401a81:::601.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117) -- 0x55f084cb31f0 con 0x55f084cd45e0 2024-08-03T20:58:58.299 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.298+0000 7f52d3200640 1 --2- 172.21.15.32:0/2665912631 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f52b40759f0 0x7f52b4077f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:58:58.299 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.298+0000 7f52d3c00640 1 --2- 172.21.15.32:0/2665912631 >> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] conn(0x55f084cd45e0 0x55f084cd1150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:58:58.300 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.299+0000 7f52d3c00640 1 --2- 172.21.15.32:0/2665912631 >> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] conn(0x55f084cd45e0 0x55f084cd1150 crc :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:58:58.300 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.299+0000 7f52d3200640 1 --2- 172.21.15.32:0/2665912631 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f52b40759f0 0x7f52b4077f00 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 crypto rx=0x7f52c4025de0 tx=0x7f52c4004880 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:58:58.302 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.302+0000 7f52d3c00640 1 -- 172.21.15.32:0/2665912631 <== osd.0 v2:172.21.15.32:6816/3066156116 1 ==== osd_op_reply(1 601.00000000 [create] v117'971 uv944 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f52cc00b040 con 0x55f084cd45e0 2024-08-03T20:58:58.302 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.302+0000 7f52d9fe6f40 1 -- 172.21.15.32:0/2665912631 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:2 12.10 12:09401a81:::601.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e117) -- 0x55f084d7d840 con 0x55f084cd45e0 2024-08-03T20:58:58.304 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.304+0000 7f52d3c00640 1 -- 172.21.15.32:0/2665912631 <== osd.0 v2:172.21.15.32:6816/3066156116 2 ==== osd_op_reply(2 601.00000000 [delete] v117'972 uv972 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f52cc00b040 con 0x55f084cd45e0 2024-08-03T20:58:58.304 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.304+0000 7f52d9fe6f40 1 -- 172.21.15.32:0/2665912631 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:3 12.10 12:09401a81:::601.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117) -- 0x55f084d7dbe0 con 0x55f084cd45e0 2024-08-03T20:58:58.305 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::601.00000000:head# 2024-08-03T20:58:58.306 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.305+0000 7f52d3c00640 1 -- 172.21.15.32:0/2665912631 <== osd.0 v2:172.21.15.32:6816/3066156116 3 ==== osd_op_reply(3 601.00000000 [create] v117'973 uv973 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f52cc00b040 con 0x55f084cd45e0 2024-08-03T20:58:58.306 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.305+0000 7f52d9fe6f40 1 -- 172.21.15.32:0/2665912631 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:4 12.10 12:09401a81:::601.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117) -- 0x55f084d7ef90 con 0x55f084cd45e0 2024-08-03T20:58:58.308 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.307+0000 7f52d3c00640 1 -- 172.21.15.32:0/2665912631 <== osd.0 v2:172.21.15.32:6816/3066156116 4 ==== osd_op_reply(4 601.00000000 [setxattr (30)] v117'974 uv974 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f52cc00b040 con 0x55f084cd45e0 2024-08-03T20:58:58.308 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.307+0000 7f52d9fe6f40 1 -- 172.21.15.32:0/2665912631 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:5 12.10 12:09401a81:::601.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117) -- 0x55f084d7f330 con 0x55f084cd45e0 2024-08-03T20:58:58.309 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.309+0000 7f52d3c00640 1 -- 172.21.15.32:0/2665912631 <== osd.0 v2:172.21.15.32:6816/3066156116 5 ==== osd_op_reply(5 601.00000000 [setxattr (62)] v117'975 uv975 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f52cc00b040 con 0x55f084cd45e0 2024-08-03T20:58:58.309 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.309+0000 7f52d9fe6f40 1 -- 172.21.15.32:0/2665912631 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:6 12.10 12:09401a81:::601.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117) -- 0x55f084d7df80 con 0x55f084cd45e0 2024-08-03T20:58:58.311 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.310+0000 7f52d3c00640 1 -- 172.21.15.32:0/2665912631 <== osd.0 v2:172.21.15.32:6816/3066156116 6 ==== osd_op_reply(6 601.00000000 [omap-set-header] v117'976 uv976 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f52cc00b040 con 0x55f084cd45e0 2024-08-03T20:58:58.311 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.311+0000 7f52d9fe6f40 1 -- 172.21.15.32:0/2665912631 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:7 12.10 12:0bd6d154:::602.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117) -- 0x55f084d7e390 con 0x55f084cd45e0 2024-08-03T20:58:58.313 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.312+0000 7f52d3c00640 1 -- 172.21.15.32:0/2665912631 <== osd.0 v2:172.21.15.32:6816/3066156116 7 ==== osd_op_reply(7 602.00000000 [create] v117'977 uv950 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f52cc00b040 con 0x55f084cd45e0 2024-08-03T20:58:58.313 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.312+0000 7f52d9fe6f40 1 -- 172.21.15.32:0/2665912631 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:8 12.10 12:0bd6d154:::602.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e117) -- 0x55f084d7e780 con 0x55f084cd45e0 2024-08-03T20:58:58.315 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.314+0000 7f52d3c00640 1 -- 172.21.15.32:0/2665912631 <== osd.0 v2:172.21.15.32:6816/3066156116 8 ==== osd_op_reply(8 602.00000000 [delete] v117'978 uv978 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f52cc00b040 con 0x55f084cd45e0 2024-08-03T20:58:58.315 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.314+0000 7f52d9fe6f40 1 -- 172.21.15.32:0/2665912631 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:9 12.10 12:0bd6d154:::602.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117) -- 0x55f084d7e780 con 0x55f084cd45e0 2024-08-03T20:58:58.316 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.316+0000 7f52d3c00640 1 -- 172.21.15.32:0/2665912631 <== osd.0 v2:172.21.15.32:6816/3066156116 9 ==== osd_op_reply(9 602.00000000 [create] v117'979 uv979 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f52cc00b040 con 0x55f084cd45e0 2024-08-03T20:58:58.316 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::602.00000000:head# 2024-08-03T20:58:58.317 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.316+0000 7f52d9fe6f40 1 -- 172.21.15.32:0/2665912631 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:10 12.10 12:0bd6d154:::602.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117) -- 0x55f084d7e730 con 0x55f084cd45e0 2024-08-03T20:58:58.318 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.317+0000 7f52d3c00640 1 -- 172.21.15.32:0/2665912631 <== osd.0 v2:172.21.15.32:6816/3066156116 10 ==== osd_op_reply(10 602.00000000 [setxattr (30)] v117'980 uv980 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f52cc00b040 con 0x55f084cd45e0 2024-08-03T20:58:58.318 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.317+0000 7f52d9fe6f40 1 -- 172.21.15.32:0/2665912631 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:11 12.10 12:0bd6d154:::602.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117) -- 0x55f084d7e730 con 0x55f084cd45e0 2024-08-03T20:58:58.319 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.319+0000 7f52d3c00640 1 -- 172.21.15.32:0/2665912631 <== osd.0 v2:172.21.15.32:6816/3066156116 11 ==== osd_op_reply(11 602.00000000 [setxattr (62)] v117'981 uv981 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f52cc00b040 con 0x55f084cd45e0 2024-08-03T20:58:58.319 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.319+0000 7f52d9fe6f40 1 -- 172.21.15.32:0/2665912631 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:12 12.10 12:0bd6d154:::602.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117) -- 0x55f084d7e730 con 0x55f084cd45e0 2024-08-03T20:58:58.321 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.321+0000 7f52d3c00640 1 -- 172.21.15.32:0/2665912631 <== osd.0 v2:172.21.15.32:6816/3066156116 12 ==== osd_op_reply(12 602.00000000 [omap-set-header] v117'982 uv982 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f52cc00b040 con 0x55f084cd45e0 2024-08-03T20:58:58.321 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.321+0000 7f52d9fe6f40 1 -- 172.21.15.32:0/2665912631 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:13 12.10 12:0cf6ad0f:::10000000000.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117) -- 0x55f084d7e730 con 0x55f084cd45e0 2024-08-03T20:58:58.323 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.323+0000 7f52d3c00640 1 -- 172.21.15.32:0/2665912631 <== osd.0 v2:172.21.15.32:6816/3066156116 13 ==== osd_op_reply(13 10000000000.00000000 [create] v117'983 uv970 ondisk = -17 ((17) File exists)) ==== 164+0+0 (crc 0 0 0) 0x7f52cc00b040 con 0x55f084cd45e0 2024-08-03T20:58:58.323 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.323+0000 7f52d9fe6f40 1 -- 172.21.15.32:0/2665912631 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:14 12.10 12:0cf6ad0f:::10000000000.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e117) -- 0x55f084d7e730 con 0x55f084cd45e0 2024-08-03T20:58:58.325 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.324+0000 7f52d3c00640 1 -- 172.21.15.32:0/2665912631 <== osd.0 v2:172.21.15.32:6816/3066156116 14 ==== osd_op_reply(14 10000000000.00000000 [delete] v117'984 uv984 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f52cc00b040 con 0x55f084cd45e0 2024-08-03T20:58:58.325 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.324+0000 7f52d9fe6f40 1 -- 172.21.15.32:0/2665912631 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:15 12.10 12:0cf6ad0f:::10000000000.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117) -- 0x55f084d7e730 con 0x55f084cd45e0 2024-08-03T20:58:58.326 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.326+0000 7f52d3c00640 1 -- 172.21.15.32:0/2665912631 <== osd.0 v2:172.21.15.32:6816/3066156116 15 ==== osd_op_reply(15 10000000000.00000000 [create] v117'985 uv985 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f52cc00b040 con 0x55f084cd45e0 2024-08-03T20:58:58.326 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::10000000000.00000000:head# 2024-08-03T20:58:58.327 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.326+0000 7f52d9fe6f40 1 -- 172.21.15.32:0/2665912631 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:16 12.10 12:0cf6ad0f:::10000000000.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117) -- 0x55f084d7e730 con 0x55f084cd45e0 2024-08-03T20:58:58.328 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.328+0000 7f52d3c00640 1 -- 172.21.15.32:0/2665912631 <== osd.0 v2:172.21.15.32:6816/3066156116 16 ==== osd_op_reply(16 10000000000.00000000 [setxattr (30)] v117'986 uv986 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f52cc00b040 con 0x55f084cd45e0 2024-08-03T20:58:58.328 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.328+0000 7f52d9fe6f40 1 -- 172.21.15.32:0/2665912631 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:17 12.10 12:0cf6ad0f:::10000000000.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117) -- 0x55f084d7e730 con 0x55f084cd45e0 2024-08-03T20:58:58.329 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.329+0000 7f52d3c00640 1 -- 172.21.15.32:0/2665912631 <== osd.0 v2:172.21.15.32:6816/3066156116 17 ==== osd_op_reply(17 10000000000.00000000 [setxattr (62)] v117'987 uv987 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f52cc00b040 con 0x55f084cd45e0 2024-08-03T20:58:58.330 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.329+0000 7f52d9fe6f40 1 -- 172.21.15.32:0/2665912631 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:18 12.10 12:0cf6ad0f:::10000000000.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117) -- 0x55f084d7ebb0 con 0x55f084cd45e0 2024-08-03T20:58:58.331 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.331+0000 7f52d3c00640 1 -- 172.21.15.32:0/2665912631 <== osd.0 v2:172.21.15.32:6816/3066156116 18 ==== osd_op_reply(18 10000000000.00000000 [omap-set-header] v117'988 uv988 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f52cc00b040 con 0x55f084cd45e0 2024-08-03T20:58:58.331 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.331+0000 7f52d9fe6f40 1 -- 172.21.15.32:0/2665912631 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:19 12.10 12:0cf6ad0f:::10000000000.00000000:head [omap-set-vals in=593b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117) -- 0x55f084d7e730 con 0x55f084cd45e0 2024-08-03T20:58:58.333 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.332+0000 7f52d3c00640 1 -- 172.21.15.32:0/2665912631 <== osd.0 v2:172.21.15.32:6816/3066156116 19 ==== osd_op_reply(19 10000000000.00000000 [omap-set-vals] v117'989 uv989 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f52cc00b040 con 0x55f084cd45e0 2024-08-03T20:58:58.333 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.332+0000 7f52d9fe6f40 1 -- 172.21.15.32:0/2665912631 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:20 12.10 12:0d82a743:::600.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117) -- 0x55f084d7e730 con 0x55f084cd45e0 2024-08-03T20:58:58.334 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.334+0000 7f52d3c00640 1 -- 172.21.15.32:0/2665912631 <== osd.0 v2:172.21.15.32:6816/3066156116 20 ==== osd_op_reply(20 600.00000000 [create] v117'990 uv963 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f52cc00b040 con 0x55f084cd45e0 2024-08-03T20:58:58.335 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.334+0000 7f52d9fe6f40 1 -- 172.21.15.32:0/2665912631 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:21 12.10 12:0d82a743:::600.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e117) -- 0x55f084d7e730 con 0x55f084cd45e0 2024-08-03T20:58:58.336 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.336+0000 7f52d3c00640 1 -- 172.21.15.32:0/2665912631 <== osd.0 v2:172.21.15.32:6816/3066156116 21 ==== osd_op_reply(21 600.00000000 [delete] v117'991 uv991 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f52cc00b040 con 0x55f084cd45e0 2024-08-03T20:58:58.336 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.336+0000 7f52d9fe6f40 1 -- 172.21.15.32:0/2665912631 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:22 12.10 12:0d82a743:::600.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117) -- 0x55f084d7e730 con 0x55f084cd45e0 2024-08-03T20:58:58.338 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::600.00000000:head# 2024-08-03T20:58:58.338 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.337+0000 7f52d3c00640 1 -- 172.21.15.32:0/2665912631 <== osd.0 v2:172.21.15.32:6816/3066156116 22 ==== osd_op_reply(22 600.00000000 [create] v117'992 uv992 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f52cc00b040 con 0x55f084cd45e0 2024-08-03T20:58:58.338 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.337+0000 7f52d9fe6f40 1 -- 172.21.15.32:0/2665912631 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:23 12.10 12:0d82a743:::600.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117) -- 0x55f084d7e730 con 0x55f084cd45e0 2024-08-03T20:58:58.339 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.339+0000 7f52d3c00640 1 -- 172.21.15.32:0/2665912631 <== osd.0 v2:172.21.15.32:6816/3066156116 23 ==== osd_op_reply(23 600.00000000 [setxattr (30)] v117'993 uv993 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f52cc00b040 con 0x55f084cd45e0 2024-08-03T20:58:58.339 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.339+0000 7f52d9fe6f40 1 -- 172.21.15.32:0/2665912631 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:24 12.10 12:0d82a743:::600.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117) -- 0x55f084d7e730 con 0x55f084cd45e0 2024-08-03T20:58:58.341 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.340+0000 7f52d3c00640 1 -- 172.21.15.32:0/2665912631 <== osd.0 v2:172.21.15.32:6816/3066156116 24 ==== osd_op_reply(24 600.00000000 [setxattr (62)] v117'994 uv994 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f52cc00b040 con 0x55f084cd45e0 2024-08-03T20:58:58.341 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.340+0000 7f52d9fe6f40 1 -- 172.21.15.32:0/2665912631 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:25 12.10 12:0d82a743:::600.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117) -- 0x55f084d7e730 con 0x55f084cd45e0 2024-08-03T20:58:58.342 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.342+0000 7f52d3c00640 1 -- 172.21.15.32:0/2665912631 <== osd.0 v2:172.21.15.32:6816/3066156116 25 ==== osd_op_reply(25 600.00000000 [omap-set-header] v117'995 uv995 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f52cc00b040 con 0x55f084cd45e0 2024-08-03T20:58:58.343 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.342+0000 7f52d9fe6f40 1 -- 172.21.15.32:0/2665912631 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:26 12.10 12:0d89b25e:::603.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117) -- 0x55f084d7e730 con 0x55f084cd45e0 2024-08-03T20:58:58.344 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.344+0000 7f52d3c00640 1 -- 172.21.15.32:0/2665912631 <== osd.0 v2:172.21.15.32:6816/3066156116 26 ==== osd_op_reply(26 603.00000000 [create] v117'996 uv969 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f52cc00b040 con 0x55f084cd45e0 2024-08-03T20:58:58.344 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.344+0000 7f52d9fe6f40 1 -- 172.21.15.32:0/2665912631 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:27 12.10 12:0d89b25e:::603.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e117) -- 0x55f084d7e730 con 0x55f084cd45e0 2024-08-03T20:58:58.346 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.345+0000 7f52d3c00640 1 -- 172.21.15.32:0/2665912631 <== osd.0 v2:172.21.15.32:6816/3066156116 27 ==== osd_op_reply(27 603.00000000 [delete] v117'997 uv997 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f52cc00b040 con 0x55f084cd45e0 2024-08-03T20:58:58.346 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.345+0000 7f52d9fe6f40 1 -- 172.21.15.32:0/2665912631 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:28 12.10 12:0d89b25e:::603.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117) -- 0x55f084d7e730 con 0x55f084cd45e0 2024-08-03T20:58:58.347 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::603.00000000:head# 2024-08-03T20:58:58.348 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.347+0000 7f52d3c00640 1 -- 172.21.15.32:0/2665912631 <== osd.0 v2:172.21.15.32:6816/3066156116 28 ==== osd_op_reply(28 603.00000000 [create] v117'998 uv998 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f52cc00b040 con 0x55f084cd45e0 2024-08-03T20:58:58.348 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.347+0000 7f52d9fe6f40 1 -- 172.21.15.32:0/2665912631 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:29 12.10 12:0d89b25e:::603.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117) -- 0x55f084d7e730 con 0x55f084cd45e0 2024-08-03T20:58:58.349 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.349+0000 7f52d3c00640 1 -- 172.21.15.32:0/2665912631 <== osd.0 v2:172.21.15.32:6816/3066156116 29 ==== osd_op_reply(29 603.00000000 [setxattr (30)] v117'999 uv999 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f52cc00b040 con 0x55f084cd45e0 2024-08-03T20:58:58.349 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.349+0000 7f52d9fe6f40 1 -- 172.21.15.32:0/2665912631 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:30 12.10 12:0d89b25e:::603.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117) -- 0x55f084d7e730 con 0x55f084cd45e0 2024-08-03T20:58:58.351 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.350+0000 7f52d3c00640 1 -- 172.21.15.32:0/2665912631 <== osd.0 v2:172.21.15.32:6816/3066156116 30 ==== osd_op_reply(30 603.00000000 [setxattr (62)] v117'1000 uv1000 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f52cc00b040 con 0x55f084cd45e0 2024-08-03T20:58:58.351 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.350+0000 7f52d9fe6f40 1 -- 172.21.15.32:0/2665912631 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:31 12.10 12:0d89b25e:::603.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117) -- 0x55f084d7e730 con 0x55f084cd45e0 2024-08-03T20:58:58.352 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.352+0000 7f52d3c00640 1 -- 172.21.15.32:0/2665912631 <== osd.0 v2:172.21.15.32:6816/3066156116 31 ==== osd_op_reply(31 603.00000000 [omap-set-header] v117'1001 uv1001 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f52cc00b040 con 0x55f084cd45e0 2024-08-03T20:58:58.352 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.352+0000 7f52d9fe6f40 1 --2- 172.21.15.32:0/2665912631 >> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] conn(0x55f084d7fbe0 0x55f084d82000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:58:58.352 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.352+0000 7f52d9fe6f40 1 -- 172.21.15.32:0/2665912631 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:32 12.14 12:287034c4:::1.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117) -- 0x55f084d7e730 con 0x55f084d7fbe0 2024-08-03T20:58:58.353 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.352+0000 7f52d3200640 1 --2- 172.21.15.32:0/2665912631 >> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] conn(0x55f084d7fbe0 0x55f084d82000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:58:58.353 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.353+0000 7f52d3200640 1 --2- 172.21.15.32:0/2665912631 >> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] conn(0x55f084d7fbe0 0x55f084d82000 crc :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:58:58.355 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.355+0000 7f52d3200640 1 -- 172.21.15.32:0/2665912631 <== osd.5 v2:172.21.15.118:6810/2724891770 1 ==== osd_op_reply(32 1.00000000.inode [create] v117'705 uv684 ondisk = -17 ((17) File exists)) ==== 160+0+0 (crc 0 0 0) 0x7f52c405f030 con 0x55f084d7fbe0 2024-08-03T20:58:58.355 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.355+0000 7f52d9fe6f40 1 -- 172.21.15.32:0/2665912631 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:33 12.14 12:287034c4:::1.00000000.inode:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e117) -- 0x55f084d82540 con 0x55f084d7fbe0 2024-08-03T20:58:58.357 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.356+0000 7f52d3200640 1 -- 172.21.15.32:0/2665912631 <== osd.5 v2:172.21.15.118:6810/2724891770 2 ==== osd_op_reply(33 1.00000000.inode [delete] v117'706 uv706 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f52c405f030 con 0x55f084d7fbe0 2024-08-03T20:58:58.357 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.356+0000 7f52d9fe6f40 1 -- 172.21.15.32:0/2665912631 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:34 12.14 12:287034c4:::1.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117) -- 0x55f084d828e0 con 0x55f084d7fbe0 2024-08-03T20:58:58.358 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::1.00000000.inode:head# 2024-08-03T20:58:58.359 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.358+0000 7f52d3200640 1 -- 172.21.15.32:0/2665912631 <== osd.5 v2:172.21.15.118:6810/2724891770 3 ==== osd_op_reply(34 1.00000000.inode [create] v117'707 uv707 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f52c405f030 con 0x55f084d7fbe0 2024-08-03T20:58:58.359 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.358+0000 7f52d9fe6f40 1 -- 172.21.15.32:0/2665912631 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:35 12.14 12:287034c4:::1.00000000.inode:head [write 0~582 in=582b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117) -- 0x55f084d82c80 con 0x55f084d7fbe0 2024-08-03T20:58:58.360 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.360+0000 7f52d3200640 1 -- 172.21.15.32:0/2665912631 <== osd.5 v2:172.21.15.118:6810/2724891770 4 ==== osd_op_reply(35 1.00000000.inode [write 0~582] v117'708 uv708 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f52c405f030 con 0x55f084d7fbe0 2024-08-03T20:58:58.360 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.360+0000 7f52d9fe6f40 1 -- 172.21.15.32:0/2665912631 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:36 12.14 12:287034c4:::1.00000000.inode:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117) -- 0x55f084d83020 con 0x55f084d7fbe0 2024-08-03T20:58:58.361 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.361+0000 7f52d3200640 1 -- 172.21.15.32:0/2665912631 <== osd.5 v2:172.21.15.118:6810/2724891770 5 ==== osd_op_reply(36 1.00000000.inode [omap-set-header] v117'709 uv709 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f52c405f030 con 0x55f084d7fbe0 2024-08-03T20:58:58.361 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.361+0000 7f52d9fe6f40 1 -- 172.21.15.32:0/2665912631 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:37 12.14 12:292cf221:::200.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117) -- 0x55f084d833c0 con 0x55f084d7fbe0 2024-08-03T20:58:58.363 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.362+0000 7f52d3200640 1 -- 172.21.15.32:0/2665912631 <== osd.5 v2:172.21.15.118:6810/2724891770 6 ==== osd_op_reply(37 200.00000000 [create] v117'710 uv703 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f52c405f030 con 0x55f084d7fbe0 2024-08-03T20:58:58.363 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.363+0000 7f52d9fe6f40 1 -- 172.21.15.32:0/2665912631 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:38 12.14 12:292cf221:::200.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e117) -- 0x55f084d833c0 con 0x55f084d7fbe0 2024-08-03T20:58:58.364 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.364+0000 7f52d3200640 1 -- 172.21.15.32:0/2665912631 <== osd.5 v2:172.21.15.118:6810/2724891770 7 ==== osd_op_reply(38 200.00000000 [delete] v117'711 uv711 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f52c405f030 con 0x55f084d7fbe0 2024-08-03T20:58:58.365 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.364+0000 7f52d9fe6f40 1 -- 172.21.15.32:0/2665912631 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:39 12.14 12:292cf221:::200.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117) -- 0x55f084d833c0 con 0x55f084d7fbe0 2024-08-03T20:58:58.366 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::200.00000000:head# 2024-08-03T20:58:58.366 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.365+0000 7f52d3200640 1 -- 172.21.15.32:0/2665912631 <== osd.5 v2:172.21.15.118:6810/2724891770 8 ==== osd_op_reply(39 200.00000000 [create] v117'712 uv712 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f52c405f030 con 0x55f084d7fbe0 2024-08-03T20:58:58.366 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.365+0000 7f52d9fe6f40 1 -- 172.21.15.32:0/2665912631 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:40 12.14 12:292cf221:::200.00000000:head [write 0~90 in=90b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117) -- 0x55f084d833c0 con 0x55f084d7fbe0 2024-08-03T20:58:58.367 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.367+0000 7f52d3200640 1 -- 172.21.15.32:0/2665912631 <== osd.5 v2:172.21.15.118:6810/2724891770 9 ==== osd_op_reply(40 200.00000000 [write 0~90] v117'713 uv713 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f52c405f030 con 0x55f084d7fbe0 2024-08-03T20:58:58.367 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.367+0000 7f52d9fe6f40 1 -- 172.21.15.32:0/2665912631 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:41 12.14 12:292cf221:::200.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117) -- 0x55f084d833c0 con 0x55f084d7fbe0 2024-08-03T20:58:58.368 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.368+0000 7f52d3200640 1 -- 172.21.15.32:0/2665912631 <== osd.5 v2:172.21.15.118:6810/2724891770 10 ==== osd_op_reply(41 200.00000000 [omap-set-header] v117'714 uv714 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f52c405f030 con 0x55f084d7fbe0 2024-08-03T20:58:58.369 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.368+0000 7f52d9fe6f40 1 -- 172.21.15.32:0/2665912631 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:42 12.14 12:2e2fa760:::200.00000001:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117) -- 0x55f084d833c0 con 0x55f084d7fbe0 2024-08-03T20:58:58.370 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.370+0000 7f52d3200640 1 -- 172.21.15.32:0/2665912631 <== osd.5 v2:172.21.15.118:6810/2724891770 11 ==== osd_op_reply(42 200.00000001 [create] v117'715 uv704 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f52c405f030 con 0x55f084d7fbe0 2024-08-03T20:58:58.371 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.370+0000 7f52d9fe6f40 1 -- 172.21.15.32:0/2665912631 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:43 12.14 12:2e2fa760:::200.00000001:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e117) -- 0x55f084d833c0 con 0x55f084d7fbe0 2024-08-03T20:58:58.372 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.372+0000 7f52d3200640 1 -- 172.21.15.32:0/2665912631 <== osd.5 v2:172.21.15.118:6810/2724891770 12 ==== osd_op_reply(43 200.00000001 [delete] v117'716 uv716 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f52c405f030 con 0x55f084d7fbe0 2024-08-03T20:58:58.372 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.372+0000 7f52d9fe6f40 1 -- 172.21.15.32:0/2665912631 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:44 12.14 12:2e2fa760:::200.00000001:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117) -- 0x55f084d833c0 con 0x55f084d7fbe0 2024-08-03T20:58:58.374 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::200.00000001:head# 2024-08-03T20:58:58.374 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.373+0000 7f52d3200640 1 -- 172.21.15.32:0/2665912631 <== osd.5 v2:172.21.15.118:6810/2724891770 13 ==== osd_op_reply(44 200.00000001 [create] v117'717 uv717 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f52c405f030 con 0x55f084d7fbe0 2024-08-03T20:58:58.374 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.374+0000 7f52d9fe6f40 1 -- 172.21.15.32:0/2665912631 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:45 12.14 12:2e2fa760:::200.00000001:head [write 0~18118 in=18118b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117) -- 0x55f084d87ab0 con 0x55f084d7fbe0 2024-08-03T20:58:58.376 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.375+0000 7f52d3200640 1 -- 172.21.15.32:0/2665912631 <== osd.5 v2:172.21.15.118:6810/2724891770 14 ==== osd_op_reply(45 200.00000001 [write 0~18118] v117'718 uv718 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f52c405f030 con 0x55f084d7fbe0 2024-08-03T20:58:58.376 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.375+0000 7f52d9fe6f40 1 -- 172.21.15.32:0/2665912631 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:46 12.14 12:2e2fa760:::200.00000001:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117) -- 0x55f084d87ab0 con 0x55f084d7fbe0 2024-08-03T20:58:58.378 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.377+0000 7f52d3200640 1 -- 172.21.15.32:0/2665912631 <== osd.5 v2:172.21.15.118:6810/2724891770 15 ==== osd_op_reply(46 200.00000001 [omap-set-header] v117'719 uv719 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f52c405f030 con 0x55f084d7fbe0 2024-08-03T20:58:58.378 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.377+0000 7f52d9fe6f40 1 -- 172.21.15.32:0/2665912631 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:47 12.14 12:2f55791f:::606.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117) -- 0x55f084d833c0 con 0x55f084d7fbe0 2024-08-03T20:58:58.379 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.379+0000 7f52d3200640 1 -- 172.21.15.32:0/2665912631 <== osd.5 v2:172.21.15.118:6810/2724891770 16 ==== osd_op_reply(47 606.00000000 [create] v117'720 uv700 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f52c405f030 con 0x55f084d7fbe0 2024-08-03T20:58:58.379 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.379+0000 7f52d9fe6f40 1 -- 172.21.15.32:0/2665912631 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:48 12.14 12:2f55791f:::606.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e117) -- 0x55f084d833c0 con 0x55f084d7fbe0 2024-08-03T20:58:58.381 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.380+0000 7f52d3200640 1 -- 172.21.15.32:0/2665912631 <== osd.5 v2:172.21.15.118:6810/2724891770 17 ==== osd_op_reply(48 606.00000000 [delete] v117'721 uv721 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f52c405f030 con 0x55f084d7fbe0 2024-08-03T20:58:58.381 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.380+0000 7f52d9fe6f40 1 -- 172.21.15.32:0/2665912631 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:49 12.14 12:2f55791f:::606.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117) -- 0x55f084d833c0 con 0x55f084d7fbe0 2024-08-03T20:58:58.382 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::606.00000000:head# 2024-08-03T20:58:58.382 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.382+0000 7f52d3200640 1 -- 172.21.15.32:0/2665912631 <== osd.5 v2:172.21.15.118:6810/2724891770 18 ==== osd_op_reply(49 606.00000000 [create] v117'722 uv722 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f52c405f030 con 0x55f084d7fbe0 2024-08-03T20:58:58.382 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.382+0000 7f52d9fe6f40 1 -- 172.21.15.32:0/2665912631 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:50 12.14 12:2f55791f:::606.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117) -- 0x55f084d843d0 con 0x55f084d7fbe0 2024-08-03T20:58:58.386 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.383+0000 7f52d3200640 1 -- 172.21.15.32:0/2665912631 <== osd.5 v2:172.21.15.118:6810/2724891770 19 ==== osd_op_reply(50 606.00000000 [setxattr (30)] v117'723 uv723 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f52c405f030 con 0x55f084d7fbe0 2024-08-03T20:58:58.387 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.384+0000 7f52d9fe6f40 1 -- 172.21.15.32:0/2665912631 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:51 12.14 12:2f55791f:::606.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117) -- 0x55f084d843d0 con 0x55f084d7fbe0 2024-08-03T20:58:58.387 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.385+0000 7f52d3200640 1 -- 172.21.15.32:0/2665912631 <== osd.5 v2:172.21.15.118:6810/2724891770 20 ==== osd_op_reply(51 606.00000000 [setxattr (62)] v117'724 uv724 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f52c405f030 con 0x55f084d7fbe0 2024-08-03T20:58:58.387 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.385+0000 7f52d9fe6f40 1 -- 172.21.15.32:0/2665912631 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:52 12.14 12:2f55791f:::606.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117) -- 0x55f084d833c0 con 0x55f084d7fbe0 2024-08-03T20:58:58.387 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.387+0000 7f52d3200640 1 -- 172.21.15.32:0/2665912631 <== osd.5 v2:172.21.15.118:6810/2724891770 21 ==== osd_op_reply(52 606.00000000 [omap-set-header] v117'725 uv725 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f52c405f030 con 0x55f084d7fbe0 2024-08-03T20:58:58.387 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.387+0000 7f52d9fe6f40 1 -- 172.21.15.32:0/2665912631 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:53 12.1c 12:3ed09add:::607.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117) -- 0x55f084d833c0 con 0x55f084cd45e0 2024-08-03T20:58:58.389 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.388+0000 7f52d3c00640 1 -- 172.21.15.32:0/2665912631 <== osd.0 v2:172.21.15.32:6816/3066156116 32 ==== osd_op_reply(53 607.00000000 [create] v117'189 uv188 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f52cc00b040 con 0x55f084cd45e0 2024-08-03T20:58:58.389 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.388+0000 7f52d9fe6f40 1 -- 172.21.15.32:0/2665912631 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:54 12.1c 12:3ed09add:::607.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e117) -- 0x55f084d833c0 con 0x55f084cd45e0 2024-08-03T20:58:58.390 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.390+0000 7f52d3c00640 1 -- 172.21.15.32:0/2665912631 <== osd.0 v2:172.21.15.32:6816/3066156116 33 ==== osd_op_reply(54 607.00000000 [delete] v117'190 uv190 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f52cc00b040 con 0x55f084cd45e0 2024-08-03T20:58:58.390 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.390+0000 7f52d9fe6f40 1 -- 172.21.15.32:0/2665912631 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:55 12.1c 12:3ed09add:::607.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117) -- 0x55f084d833c0 con 0x55f084cd45e0 2024-08-03T20:58:58.391 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::607.00000000:head# 2024-08-03T20:58:58.392 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.391+0000 7f52d3c00640 1 -- 172.21.15.32:0/2665912631 <== osd.0 v2:172.21.15.32:6816/3066156116 34 ==== osd_op_reply(55 607.00000000 [create] v117'191 uv191 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f52cc00b040 con 0x55f084cd45e0 2024-08-03T20:58:58.392 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.391+0000 7f52d9fe6f40 1 -- 172.21.15.32:0/2665912631 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:56 12.1c 12:3ed09add:::607.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117) -- 0x55f084d843d0 con 0x55f084cd45e0 2024-08-03T20:58:58.393 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.392+0000 7f52d3c00640 1 -- 172.21.15.32:0/2665912631 <== osd.0 v2:172.21.15.32:6816/3066156116 35 ==== osd_op_reply(56 607.00000000 [setxattr (30)] v117'192 uv192 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f52cc00b040 con 0x55f084cd45e0 2024-08-03T20:58:58.393 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.392+0000 7f52d9fe6f40 1 -- 172.21.15.32:0/2665912631 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:57 12.1c 12:3ed09add:::607.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117) -- 0x55f084d843d0 con 0x55f084cd45e0 2024-08-03T20:58:58.394 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.394+0000 7f52d3c00640 1 -- 172.21.15.32:0/2665912631 <== osd.0 v2:172.21.15.32:6816/3066156116 36 ==== osd_op_reply(57 607.00000000 [setxattr (62)] v117'193 uv193 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f52cc00b040 con 0x55f084cd45e0 2024-08-03T20:58:58.395 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.394+0000 7f52d9fe6f40 1 -- 172.21.15.32:0/2665912631 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:58 12.1c 12:3ed09add:::607.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117) -- 0x55f084d833c0 con 0x55f084cd45e0 2024-08-03T20:58:58.395 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.395+0000 7f52d3c00640 1 -- 172.21.15.32:0/2665912631 <== osd.0 v2:172.21.15.32:6816/3066156116 37 ==== osd_op_reply(58 607.00000000 [omap-set-header] v117'194 uv194 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f52cc00b040 con 0x55f084cd45e0 2024-08-03T20:58:58.396 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.395+0000 7f52d9fe6f40 1 --2- 172.21.15.32:0/2665912631 >> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] conn(0x55f084d838d0 0x55f084d85d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:58:58.396 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.395+0000 7f52d9fe6f40 1 -- 172.21.15.32:0/2665912631 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:59 12.6 12:654134d2:::mds0_openfiles.0:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117) -- 0x55f084d863e0 con 0x55f084d838d0 2024-08-03T20:58:58.396 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.395+0000 7f52d2800640 1 --2- 172.21.15.32:0/2665912631 >> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] conn(0x55f084d838d0 0x55f084d85d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:58:58.396 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.396+0000 7f52d2800640 1 --2- 172.21.15.32:0/2665912631 >> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] conn(0x55f084d838d0 0x55f084d85d10 crc :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:58:58.398 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.397+0000 7f52d2800640 1 -- 172.21.15.32:0/2665912631 <== osd.1 v2:172.21.15.32:6824/1551241759 1 ==== osd_op_reply(59 mds0_openfiles.0 [create] v117'148 uv147 ondisk = -17 ((17) File exists)) ==== 160+0+0 (crc 0 0 0) 0x7f52c005c380 con 0x55f084d838d0 2024-08-03T20:58:58.398 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.398+0000 7f52d9fe6f40 1 -- 172.21.15.32:0/2665912631 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:60 12.6 12:654134d2:::mds0_openfiles.0:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e117) -- 0x55f084d868f0 con 0x55f084d838d0 2024-08-03T20:58:58.399 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.399+0000 7f52d2800640 1 -- 172.21.15.32:0/2665912631 <== osd.1 v2:172.21.15.32:6824/1551241759 2 ==== osd_op_reply(60 mds0_openfiles.0 [delete] v117'149 uv149 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f52c005c380 con 0x55f084d838d0 2024-08-03T20:58:58.399 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.399+0000 7f52d9fe6f40 1 -- 172.21.15.32:0/2665912631 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:61 12.6 12:654134d2:::mds0_openfiles.0:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117) -- 0x55f084d86df0 con 0x55f084d838d0 2024-08-03T20:58:58.400 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds0_openfiles.0:head# 2024-08-03T20:58:58.401 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.400+0000 7f52d2800640 1 -- 172.21.15.32:0/2665912631 <== osd.1 v2:172.21.15.32:6824/1551241759 3 ==== osd_op_reply(61 mds0_openfiles.0 [create] v117'150 uv150 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f52c005c380 con 0x55f084d838d0 2024-08-03T20:58:58.401 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.400+0000 7f52d9fe6f40 1 -- 172.21.15.32:0/2665912631 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:62 12.6 12:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117) -- 0x55f084cd4210 con 0x55f084d838d0 2024-08-03T20:58:58.402 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.401+0000 7f52d2800640 1 -- 172.21.15.32:0/2665912631 <== osd.1 v2:172.21.15.32:6824/1551241759 4 ==== osd_op_reply(62 mds0_openfiles.0 [omap-set-header] v117'151 uv151 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f52c005c380 con 0x55f084d838d0 2024-08-03T20:58:58.402 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.401+0000 7f52d9fe6f40 1 -- 172.21.15.32:0/2665912631 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:63 12.e 12:717a0223:::608.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117) -- 0x55f084d87720 con 0x55f084cd45e0 2024-08-03T20:58:58.403 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.403+0000 7f52d3c00640 1 -- 172.21.15.32:0/2665912631 <== osd.0 v2:172.21.15.32:6816/3066156116 38 ==== osd_op_reply(63 608.00000000 [create] v117'189 uv188 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f52cc00b040 con 0x55f084cd45e0 2024-08-03T20:58:58.403 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.403+0000 7f52d9fe6f40 1 -- 172.21.15.32:0/2665912631 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:64 12.e 12:717a0223:::608.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e117) -- 0x55f084d87720 con 0x55f084cd45e0 2024-08-03T20:58:58.405 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.404+0000 7f52d3c00640 1 -- 172.21.15.32:0/2665912631 <== osd.0 v2:172.21.15.32:6816/3066156116 39 ==== osd_op_reply(64 608.00000000 [delete] v117'190 uv190 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f52cc00b040 con 0x55f084cd45e0 2024-08-03T20:58:58.405 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.404+0000 7f52d9fe6f40 1 -- 172.21.15.32:0/2665912631 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:65 12.e 12:717a0223:::608.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117) -- 0x55f084d87720 con 0x55f084cd45e0 2024-08-03T20:58:58.406 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.406+0000 7f52d3c00640 1 -- 172.21.15.32:0/2665912631 <== osd.0 v2:172.21.15.32:6816/3066156116 40 ==== osd_op_reply(65 608.00000000 [create] v117'191 uv191 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f52cc00b040 con 0x55f084cd45e0 2024-08-03T20:58:58.406 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::608.00000000:head# 2024-08-03T20:58:58.407 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.406+0000 7f52d9fe6f40 1 -- 172.21.15.32:0/2665912631 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:66 12.e 12:717a0223:::608.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117) -- 0x55f084d87720 con 0x55f084cd45e0 2024-08-03T20:58:58.408 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.407+0000 7f52d3c00640 1 -- 172.21.15.32:0/2665912631 <== osd.0 v2:172.21.15.32:6816/3066156116 41 ==== osd_op_reply(66 608.00000000 [setxattr (30)] v117'192 uv192 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f52cc00b040 con 0x55f084cd45e0 2024-08-03T20:58:58.408 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.407+0000 7f52d9fe6f40 1 -- 172.21.15.32:0/2665912631 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:67 12.e 12:717a0223:::608.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117) -- 0x55f084d87720 con 0x55f084cd45e0 2024-08-03T20:58:58.409 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.409+0000 7f52d3c00640 1 -- 172.21.15.32:0/2665912631 <== osd.0 v2:172.21.15.32:6816/3066156116 42 ==== osd_op_reply(67 608.00000000 [setxattr (62)] v117'193 uv193 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f52cc00b040 con 0x55f084cd45e0 2024-08-03T20:58:58.409 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.409+0000 7f52d9fe6f40 1 -- 172.21.15.32:0/2665912631 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:68 12.e 12:717a0223:::608.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117) -- 0x55f084d87720 con 0x55f084cd45e0 2024-08-03T20:58:58.410 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.410+0000 7f52d3c00640 1 -- 172.21.15.32:0/2665912631 <== osd.0 v2:172.21.15.32:6816/3066156116 43 ==== osd_op_reply(68 608.00000000 [omap-set-header] v117'194 uv194 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f52cc00b040 con 0x55f084cd45e0 2024-08-03T20:58:58.410 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.410+0000 7f52d9fe6f40 1 --2- 172.21.15.32:0/2665912631 >> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] conn(0x55f084d88020 0x55f084d88430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:58:58.410 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.410+0000 7f52d9fe6f40 1 -- 172.21.15.32:0/2665912631 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:69 12.15 12:a93a17c2:::604.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117) -- 0x55f084d8ab50 con 0x55f084d88020 2024-08-03T20:58:58.411 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.410+0000 7f52d3c00640 1 --2- 172.21.15.32:0/2665912631 >> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] conn(0x55f084d88020 0x55f084d88430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:58:58.412 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.411+0000 7f52d3c00640 1 --2- 172.21.15.32:0/2665912631 >> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] conn(0x55f084d88020 0x55f084d88430 crc :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:58:58.413 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.413+0000 7f52d3c00640 1 -- 172.21.15.32:0/2665912631 <== osd.6 v2:172.21.15.118:6825/801947386 1 ==== osd_op_reply(69 604.00000000 [create] v117'349 uv343 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f52cc00b040 con 0x55f084d88020 2024-08-03T20:58:58.413 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.413+0000 7f52d9fe6f40 1 -- 172.21.15.32:0/2665912631 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:70 12.15 12:a93a17c2:::604.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e117) -- 0x55f084d8ab50 con 0x55f084d88020 2024-08-03T20:58:58.415 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.415+0000 7f52d3c00640 1 -- 172.21.15.32:0/2665912631 <== osd.6 v2:172.21.15.118:6825/801947386 2 ==== osd_op_reply(70 604.00000000 [delete] v117'350 uv350 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f52cc00b040 con 0x55f084d88020 2024-08-03T20:58:58.415 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.415+0000 7f52d9fe6f40 1 -- 172.21.15.32:0/2665912631 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:71 12.15 12:a93a17c2:::604.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117) -- 0x55f084d8ab50 con 0x55f084d88020 2024-08-03T20:58:58.416 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.416+0000 7f52d3c00640 1 -- 172.21.15.32:0/2665912631 <== osd.6 v2:172.21.15.118:6825/801947386 3 ==== osd_op_reply(71 604.00000000 [create] v117'351 uv351 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f52cc00b040 con 0x55f084d88020 2024-08-03T20:58:58.416 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::604.00000000:head# 2024-08-03T20:58:58.417 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.416+0000 7f52d9fe6f40 1 -- 172.21.15.32:0/2665912631 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:72 12.15 12:a93a17c2:::604.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117) -- 0x55f084d8bf50 con 0x55f084d88020 2024-08-03T20:58:58.418 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.417+0000 7f52d3c00640 1 -- 172.21.15.32:0/2665912631 <== osd.6 v2:172.21.15.118:6825/801947386 4 ==== osd_op_reply(72 604.00000000 [setxattr (30)] v117'352 uv352 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f52cc00b040 con 0x55f084d88020 2024-08-03T20:58:58.418 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.417+0000 7f52d9fe6f40 1 -- 172.21.15.32:0/2665912631 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:73 12.15 12:a93a17c2:::604.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117) -- 0x55f084d8bf50 con 0x55f084d88020 2024-08-03T20:58:58.419 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.418+0000 7f52d3c00640 1 -- 172.21.15.32:0/2665912631 <== osd.6 v2:172.21.15.118:6825/801947386 5 ==== osd_op_reply(73 604.00000000 [setxattr (62)] v117'353 uv353 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f52cc00b040 con 0x55f084d88020 2024-08-03T20:58:58.419 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.419+0000 7f52d9fe6f40 1 -- 172.21.15.32:0/2665912631 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:74 12.15 12:a93a17c2:::604.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117) -- 0x55f084d8af40 con 0x55f084d88020 2024-08-03T20:58:58.420 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.420+0000 7f52d3c00640 1 -- 172.21.15.32:0/2665912631 <== osd.6 v2:172.21.15.118:6825/801947386 6 ==== osd_op_reply(74 604.00000000 [omap-set-header] v117'354 uv354 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f52cc00b040 con 0x55f084d88020 2024-08-03T20:58:58.421 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.420+0000 7f52d9fe6f40 1 -- 172.21.15.32:0/2665912631 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:75 12.15 12:aa448500:::500.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117) -- 0x55f084d8af40 con 0x55f084d88020 2024-08-03T20:58:58.422 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.422+0000 7f52d3c00640 1 -- 172.21.15.32:0/2665912631 <== osd.6 v2:172.21.15.118:6825/801947386 7 ==== osd_op_reply(75 500.00000000 [create] v117'355 uv348 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f52cc00b040 con 0x55f084d88020 2024-08-03T20:58:58.422 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.422+0000 7f52d9fe6f40 1 -- 172.21.15.32:0/2665912631 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:76 12.15 12:aa448500:::500.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e117) -- 0x55f084d8af40 con 0x55f084d88020 2024-08-03T20:58:58.423 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.423+0000 7f52d3c00640 1 -- 172.21.15.32:0/2665912631 <== osd.6 v2:172.21.15.118:6825/801947386 8 ==== osd_op_reply(76 500.00000000 [delete] v117'356 uv356 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f52cc00b040 con 0x55f084d88020 2024-08-03T20:58:58.424 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.423+0000 7f52d9fe6f40 1 -- 172.21.15.32:0/2665912631 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:77 12.15 12:aa448500:::500.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117) -- 0x55f084d8af40 con 0x55f084d88020 2024-08-03T20:58:58.425 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.424+0000 7f52d3c00640 1 -- 172.21.15.32:0/2665912631 <== osd.6 v2:172.21.15.118:6825/801947386 9 ==== osd_op_reply(77 500.00000000 [create] v117'357 uv357 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f52cc00b040 con 0x55f084d88020 2024-08-03T20:58:58.425 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::500.00000000:head# 2024-08-03T20:58:58.425 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.424+0000 7f52d9fe6f40 1 -- 172.21.15.32:0/2665912631 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:78 12.15 12:aa448500:::500.00000000:head [write 0~90 in=90b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117) -- 0x55f084d8af40 con 0x55f084d88020 2024-08-03T20:58:58.426 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.426+0000 7f52d3c00640 1 -- 172.21.15.32:0/2665912631 <== osd.6 v2:172.21.15.118:6825/801947386 10 ==== osd_op_reply(78 500.00000000 [write 0~90] v117'358 uv358 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f52cc00b040 con 0x55f084d88020 2024-08-03T20:58:58.427 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.426+0000 7f52d9fe6f40 1 -- 172.21.15.32:0/2665912631 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:79 12.15 12:aa448500:::500.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117) -- 0x55f084d8af40 con 0x55f084d88020 2024-08-03T20:58:58.428 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.427+0000 7f52d3c00640 1 -- 172.21.15.32:0/2665912631 <== osd.6 v2:172.21.15.118:6825/801947386 11 ==== osd_op_reply(79 500.00000000 [omap-set-header] v117'359 uv359 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f52cc00b040 con 0x55f084d88020 2024-08-03T20:58:58.428 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.428+0000 7f52d9fe6f40 1 --2- 172.21.15.32:0/2665912631 >> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] conn(0x55f084d8b450 0x55f084d8d870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:58:58.428 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.428+0000 7f52d9fe6f40 1 -- 172.21.15.32:0/2665912631 --> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] -- osd_op(unknown.0.0:80 12.d 12:b50e409b:::mds_snaptable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117) -- 0x55f084d8df00 con 0x55f084d8b450 2024-08-03T20:58:58.428 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.428+0000 7f52d3200640 1 --2- 172.21.15.32:0/2665912631 >> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] conn(0x55f084d8b450 0x55f084d8d870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:58:58.429 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.429+0000 7f52d3200640 1 --2- 172.21.15.32:0/2665912631 >> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] conn(0x55f084d8b450 0x55f084d8d870 crc :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:58:58.430 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.430+0000 7f52d3200640 1 -- 172.21.15.32:0/2665912631 <== osd.7 v2:172.21.15.118:6818/653000048 1 ==== osd_op_reply(80 mds_snaptable [create] v117'158 uv157 ondisk = -17 ((17) File exists)) ==== 157+0+0 (crc 0 0 0) 0x7f52c405f030 con 0x55f084d8b450 2024-08-03T20:58:58.430 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.430+0000 7f52d9fe6f40 1 -- 172.21.15.32:0/2665912631 --> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] -- osd_op(unknown.0.0:81 12.d 12:b50e409b:::mds_snaptable:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e117) -- 0x55f084d8e2d0 con 0x55f084d8b450 2024-08-03T20:58:58.432 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.431+0000 7f52d3200640 1 -- 172.21.15.32:0/2665912631 <== osd.7 v2:172.21.15.118:6818/653000048 2 ==== osd_op_reply(81 mds_snaptable [delete] v117'159 uv159 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f52c405f030 con 0x55f084d8b450 2024-08-03T20:58:58.432 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.431+0000 7f52d9fe6f40 1 -- 172.21.15.32:0/2665912631 --> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] -- osd_op(unknown.0.0:82 12.d 12:b50e409b:::mds_snaptable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117) -- 0x55f084d8e2d0 con 0x55f084d8b450 2024-08-03T20:58:58.434 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds_snaptable:head# 2024-08-03T20:58:58.434 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.432+0000 7f52d3200640 1 -- 172.21.15.32:0/2665912631 <== osd.7 v2:172.21.15.118:6818/653000048 3 ==== osd_op_reply(82 mds_snaptable [create] v117'160 uv160 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f52c405f030 con 0x55f084d8b450 2024-08-03T20:58:58.435 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.432+0000 7f52d9fe6f40 1 -- 172.21.15.32:0/2665912631 --> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] -- osd_op(unknown.0.0:83 12.d 12:b50e409b:::mds_snaptable:head [write 0~70 in=70b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117) -- 0x55f084d8e2d0 con 0x55f084d8b450 2024-08-03T20:58:58.435 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.434+0000 7f52d3200640 1 -- 172.21.15.32:0/2665912631 <== osd.7 v2:172.21.15.118:6818/653000048 4 ==== osd_op_reply(83 mds_snaptable [write 0~70] v117'161 uv161 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f52c405f030 con 0x55f084d8b450 2024-08-03T20:58:58.435 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.434+0000 7f52d9fe6f40 1 -- 172.21.15.32:0/2665912631 --> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] -- osd_op(unknown.0.0:84 12.d 12:b50e409b:::mds_snaptable:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117) -- 0x55f084d8e2d0 con 0x55f084d8b450 2024-08-03T20:58:58.436 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.435+0000 7f52d3200640 1 -- 172.21.15.32:0/2665912631 <== osd.7 v2:172.21.15.118:6818/653000048 5 ==== osd_op_reply(84 mds_snaptable [omap-set-header] v117'162 uv162 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f52c405f030 con 0x55f084d8b450 2024-08-03T20:58:58.436 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.435+0000 7f52d9fe6f40 1 -- 172.21.15.32:0/2665912631 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:85 12.1d 12:b871830b:::605.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117) -- 0x55f084d8e2d0 con 0x55f084cd45e0 2024-08-03T20:58:58.437 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.437+0000 7f52d3c00640 1 -- 172.21.15.32:0/2665912631 <== osd.0 v2:172.21.15.32:6816/3066156116 44 ==== osd_op_reply(85 605.00000000 [create] v117'189 uv188 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f52cc00b040 con 0x55f084cd45e0 2024-08-03T20:58:58.437 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.437+0000 7f52d9fe6f40 1 -- 172.21.15.32:0/2665912631 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:86 12.1d 12:b871830b:::605.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e117) -- 0x55f084d8e2d0 con 0x55f084cd45e0 2024-08-03T20:58:58.439 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.438+0000 7f52d3c00640 1 -- 172.21.15.32:0/2665912631 <== osd.0 v2:172.21.15.32:6816/3066156116 45 ==== osd_op_reply(86 605.00000000 [delete] v117'190 uv190 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f52cc00b040 con 0x55f084cd45e0 2024-08-03T20:58:58.439 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.438+0000 7f52d9fe6f40 1 -- 172.21.15.32:0/2665912631 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:87 12.1d 12:b871830b:::605.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117) -- 0x55f084d8e2d0 con 0x55f084cd45e0 2024-08-03T20:58:58.440 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.439+0000 7f52d3c00640 1 -- 172.21.15.32:0/2665912631 <== osd.0 v2:172.21.15.32:6816/3066156116 46 ==== osd_op_reply(87 605.00000000 [create] v117'191 uv191 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f52cc00b040 con 0x55f084cd45e0 2024-08-03T20:58:58.440 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::605.00000000:head# 2024-08-03T20:58:58.440 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.439+0000 7f52d9fe6f40 1 -- 172.21.15.32:0/2665912631 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:88 12.1d 12:b871830b:::605.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117) -- 0x55f084d8f320 con 0x55f084cd45e0 2024-08-03T20:58:58.441 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.441+0000 7f52d3c00640 1 -- 172.21.15.32:0/2665912631 <== osd.0 v2:172.21.15.32:6816/3066156116 47 ==== osd_op_reply(88 605.00000000 [setxattr (30)] v117'192 uv192 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f52cc00b040 con 0x55f084cd45e0 2024-08-03T20:58:58.441 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.441+0000 7f52d9fe6f40 1 -- 172.21.15.32:0/2665912631 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:89 12.1d 12:b871830b:::605.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117) -- 0x55f084d8f2e0 con 0x55f084cd45e0 2024-08-03T20:58:58.442 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.442+0000 7f52d3c00640 1 -- 172.21.15.32:0/2665912631 <== osd.0 v2:172.21.15.32:6816/3066156116 48 ==== osd_op_reply(89 605.00000000 [setxattr (62)] v117'193 uv193 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f52cc00b040 con 0x55f084cd45e0 2024-08-03T20:58:58.442 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.442+0000 7f52d9fe6f40 1 -- 172.21.15.32:0/2665912631 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:90 12.1d 12:b871830b:::605.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117) -- 0x55f084d8e2d0 con 0x55f084cd45e0 2024-08-03T20:58:58.444 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.443+0000 7f52d3c00640 1 -- 172.21.15.32:0/2665912631 <== osd.0 v2:172.21.15.32:6816/3066156116 49 ==== osd_op_reply(90 605.00000000 [omap-set-header] v117'194 uv194 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f52cc00b040 con 0x55f084cd45e0 2024-08-03T20:58:58.444 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.443+0000 7f52d9fe6f40 1 -- 172.21.15.32:0/2665912631 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:91 12.13 12:c91d4a1d:::mds0_inotable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117) -- 0x55f084d8e2d0 con 0x55f084d838d0 2024-08-03T20:58:58.445 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.445+0000 7f52d2800640 1 -- 172.21.15.32:0/2665912631 <== osd.1 v2:172.21.15.32:6824/1551241759 5 ==== osd_op_reply(91 mds0_inotable [create] v117'314 uv308 ondisk = -17 ((17) File exists)) ==== 157+0+0 (crc 0 0 0) 0x7f52c005c380 con 0x55f084d838d0 2024-08-03T20:58:58.445 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.445+0000 7f52d9fe6f40 1 -- 172.21.15.32:0/2665912631 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:92 12.13 12:c91d4a1d:::mds0_inotable:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e117) -- 0x55f084d8e670 con 0x55f084d838d0 2024-08-03T20:58:58.447 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.447+0000 7f52d2800640 1 -- 172.21.15.32:0/2665912631 <== osd.1 v2:172.21.15.32:6824/1551241759 6 ==== osd_op_reply(92 mds0_inotable [delete] v117'315 uv315 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f52c005c380 con 0x55f084d838d0 2024-08-03T20:58:58.448 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.447+0000 7f52d9fe6f40 1 -- 172.21.15.32:0/2665912631 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:93 12.13 12:c91d4a1d:::mds0_inotable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117) -- 0x55f084d8ea60 con 0x55f084d838d0 2024-08-03T20:58:58.449 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.448+0000 7f52d2800640 1 -- 172.21.15.32:0/2665912631 <== osd.1 v2:172.21.15.32:6824/1551241759 7 ==== osd_op_reply(93 mds0_inotable [create] v117'316 uv316 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f52c005c380 con 0x55f084d838d0 2024-08-03T20:58:58.449 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds0_inotable:head# 2024-08-03T20:58:58.449 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.449+0000 7f52d9fe6f40 1 -- 172.21.15.32:0/2665912631 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:94 12.13 12:c91d4a1d:::mds0_inotable:head [write 0~34 in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117) -- 0x55f084d8ee50 con 0x55f084d838d0 2024-08-03T20:58:58.450 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.450+0000 7f52d2800640 1 -- 172.21.15.32:0/2665912631 <== osd.1 v2:172.21.15.32:6824/1551241759 8 ==== osd_op_reply(94 mds0_inotable [write 0~34] v117'317 uv317 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f52c005c380 con 0x55f084d838d0 2024-08-03T20:58:58.450 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.450+0000 7f52d9fe6f40 1 -- 172.21.15.32:0/2665912631 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:95 12.13 12:c91d4a1d:::mds0_inotable:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117) -- 0x55f084d8ee50 con 0x55f084d838d0 2024-08-03T20:58:58.452 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.451+0000 7f52d2800640 1 -- 172.21.15.32:0/2665912631 <== osd.1 v2:172.21.15.32:6824/1551241759 9 ==== osd_op_reply(95 mds0_inotable [omap-set-header] v117'318 uv318 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f52c005c380 con 0x55f084d838d0 2024-08-03T20:58:58.452 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.451+0000 7f52d9fe6f40 1 -- 172.21.15.32:0/2665912631 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:96 12.13 12:cd5a64a3:::100.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117) -- 0x55f084d8f240 con 0x55f084d838d0 2024-08-03T20:58:58.453 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.453+0000 7f52d2800640 1 -- 172.21.15.32:0/2665912631 <== osd.1 v2:172.21.15.32:6824/1551241759 10 ==== osd_op_reply(96 100.00000000 [create] v117'319 uv313 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f52c005c380 con 0x55f084d838d0 2024-08-03T20:58:58.453 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.453+0000 7f52d9fe6f40 1 -- 172.21.15.32:0/2665912631 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:97 12.13 12:cd5a64a3:::100.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e117) -- 0x55f084d8eb00 con 0x55f084d838d0 2024-08-03T20:58:58.454 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.454+0000 7f52d2800640 1 -- 172.21.15.32:0/2665912631 <== osd.1 v2:172.21.15.32:6824/1551241759 11 ==== osd_op_reply(97 100.00000000 [delete] v117'320 uv320 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f52c005c380 con 0x55f084d838d0 2024-08-03T20:58:58.454 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.454+0000 7f52d9fe6f40 1 -- 172.21.15.32:0/2665912631 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:98 12.13 12:cd5a64a3:::100.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117) -- 0x55f084d8eb00 con 0x55f084d838d0 2024-08-03T20:58:58.455 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.455+0000 7f52d2800640 1 -- 172.21.15.32:0/2665912631 <== osd.1 v2:172.21.15.32:6824/1551241759 12 ==== osd_op_reply(98 100.00000000 [create] v117'321 uv321 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f52c005c380 con 0x55f084d838d0 2024-08-03T20:58:58.456 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::100.00000000:head# 2024-08-03T20:58:58.456 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.455+0000 7f52d9fe6f40 1 -- 172.21.15.32:0/2665912631 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:99 12.13 12:cd5a64a3:::100.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117) -- 0x55f084d8eb00 con 0x55f084d838d0 2024-08-03T20:58:58.457 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.457+0000 7f52d2800640 1 -- 172.21.15.32:0/2665912631 <== osd.1 v2:172.21.15.32:6824/1551241759 13 ==== osd_op_reply(99 100.00000000 [omap-set-header] v117'322 uv322 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f52c005c380 con 0x55f084d838d0 2024-08-03T20:58:58.457 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.457+0000 7f52d9fe6f40 1 -- 172.21.15.32:0/2665912631 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:100 12.13 12:cd5a64a3:::100.00000000:head [omap-set-vals in=5194b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117) -- 0x55f084d87220 con 0x55f084d838d0 2024-08-03T20:58:58.459 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.458+0000 7f52d2800640 1 -- 172.21.15.32:0/2665912631 <== osd.1 v2:172.21.15.32:6824/1551241759 14 ==== osd_op_reply(100 100.00000000 [omap-set-vals] v117'323 uv323 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f52c005c380 con 0x55f084d838d0 2024-08-03T20:58:58.459 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.458+0000 7f52d9fe6f40 1 --2- 172.21.15.32:0/2665912631 >> [v2:172.21.15.32:6800/9088303,v1:172.21.15.32:6801/9088303] conn(0x55f084d87220 0x55f084d92100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:58:58.460 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.458+0000 7f52d9fe6f40 1 -- 172.21.15.32:0/2665912631 --> [v2:172.21.15.32:6800/9088303,v1:172.21.15.32:6801/9088303] -- osd_op(unknown.0.0:101 12.b 12:d0630e4c:::mds0_sessionmap:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117) -- 0x55f084d91c70 con 0x55f084d87220 2024-08-03T20:58:58.460 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.459+0000 7f52d2800640 1 --2- 172.21.15.32:0/2665912631 >> [v2:172.21.15.32:6800/9088303,v1:172.21.15.32:6801/9088303] conn(0x55f084d87220 0x55f084d92100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:58:58.460 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.459+0000 7f52d2800640 1 --2- 172.21.15.32:0/2665912631 >> [v2:172.21.15.32:6800/9088303,v1:172.21.15.32:6801/9088303] conn(0x55f084d87220 0x55f084d92100 crc :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:58:58.461 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.461+0000 7f52d2800640 1 -- 172.21.15.32:0/2665912631 <== osd.3 v2:172.21.15.32:6800/9088303 1 ==== osd_op_reply(101 mds0_sessionmap [create] v117'137 uv135 ondisk = -17 ((17) File exists)) ==== 159+0+0 (crc 0 0 0) 0x7f52c005c380 con 0x55f084d87220 2024-08-03T20:58:58.461 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.461+0000 7f52d9fe6f40 1 -- 172.21.15.32:0/2665912631 --> [v2:172.21.15.32:6800/9088303,v1:172.21.15.32:6801/9088303] -- osd_op(unknown.0.0:102 12.b 12:d0630e4c:::mds0_sessionmap:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e117) -- 0x55f084d91c70 con 0x55f084d87220 2024-08-03T20:58:58.463 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.462+0000 7f52d2800640 1 -- 172.21.15.32:0/2665912631 <== osd.3 v2:172.21.15.32:6800/9088303 2 ==== osd_op_reply(102 mds0_sessionmap [delete] v117'138 uv138 ondisk = 0) ==== 159+0+0 (crc 0 0 0) 0x7f52c005c380 con 0x55f084d87220 2024-08-03T20:58:58.463 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.462+0000 7f52d9fe6f40 1 -- 172.21.15.32:0/2665912631 --> [v2:172.21.15.32:6800/9088303,v1:172.21.15.32:6801/9088303] -- osd_op(unknown.0.0:103 12.b 12:d0630e4c:::mds0_sessionmap:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117) -- 0x55f084d91c70 con 0x55f084d87220 2024-08-03T20:58:58.464 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds0_sessionmap:head# 2024-08-03T20:58:58.464 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.463+0000 7f52d2800640 1 -- 172.21.15.32:0/2665912631 <== osd.3 v2:172.21.15.32:6800/9088303 3 ==== osd_op_reply(103 mds0_sessionmap [create] v117'139 uv139 ondisk = 0) ==== 159+0+0 (crc 0 0 0) 0x7f52c005c380 con 0x55f084d87220 2024-08-03T20:58:58.464 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.463+0000 7f52d9fe6f40 1 -- 172.21.15.32:0/2665912631 --> [v2:172.21.15.32:6800/9088303,v1:172.21.15.32:6801/9088303] -- osd_op(unknown.0.0:104 12.b 12:d0630e4c:::mds0_sessionmap:head [omap-set-header in=14b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117) -- 0x55f084d91c70 con 0x55f084d87220 2024-08-03T20:58:58.465 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.465+0000 7f52d2800640 1 -- 172.21.15.32:0/2665912631 <== osd.3 v2:172.21.15.32:6800/9088303 4 ==== osd_op_reply(104 mds0_sessionmap [omap-set-header] v117'140 uv140 ondisk = 0) ==== 159+0+0 (crc 0 0 0) 0x7f52c005c380 con 0x55f084d87220 2024-08-03T20:58:58.465 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.465+0000 7f52d9fe6f40 1 -- 172.21.15.32:0/2665912631 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:105 12.7 12:e0b41d06:::609.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117) -- 0x55f084d91c70 con 0x55f084d838d0 2024-08-03T20:58:58.467 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.466+0000 7f52d2800640 1 -- 172.21.15.32:0/2665912631 <== osd.1 v2:172.21.15.32:6824/1551241759 15 ==== osd_op_reply(105 609.00000000 [create] v117'189 uv188 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f52c005c380 con 0x55f084d838d0 2024-08-03T20:58:58.467 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.467+0000 7f52d9fe6f40 1 -- 172.21.15.32:0/2665912631 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:106 12.7 12:e0b41d06:::609.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e117) -- 0x55f084d91c70 con 0x55f084d838d0 2024-08-03T20:58:58.468 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.468+0000 7f52d2800640 1 -- 172.21.15.32:0/2665912631 <== osd.1 v2:172.21.15.32:6824/1551241759 16 ==== osd_op_reply(106 609.00000000 [delete] v117'190 uv190 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f52c005c380 con 0x55f084d838d0 2024-08-03T20:58:58.468 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.468+0000 7f52d9fe6f40 1 -- 172.21.15.32:0/2665912631 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:107 12.7 12:e0b41d06:::609.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117) -- 0x55f084d91c70 con 0x55f084d838d0 2024-08-03T20:58:58.470 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.469+0000 7f52d2800640 1 -- 172.21.15.32:0/2665912631 <== osd.1 v2:172.21.15.32:6824/1551241759 17 ==== osd_op_reply(107 609.00000000 [create] v117'191 uv191 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f52c005c380 con 0x55f084d838d0 2024-08-03T20:58:58.470 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::609.00000000:head# 2024-08-03T20:58:58.470 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.469+0000 7f52d9fe6f40 1 -- 172.21.15.32:0/2665912631 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:108 12.7 12:e0b41d06:::609.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117) -- 0x55f084d91c70 con 0x55f084d838d0 2024-08-03T20:58:58.471 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.471+0000 7f52d2800640 1 -- 172.21.15.32:0/2665912631 <== osd.1 v2:172.21.15.32:6824/1551241759 18 ==== osd_op_reply(108 609.00000000 [setxattr (30)] v117'192 uv192 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f52c005c380 con 0x55f084d838d0 2024-08-03T20:58:58.471 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.471+0000 7f52d9fe6f40 1 -- 172.21.15.32:0/2665912631 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:109 12.7 12:e0b41d06:::609.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117) -- 0x55f084d91c70 con 0x55f084d838d0 2024-08-03T20:58:58.472 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.472+0000 7f52d2800640 1 -- 172.21.15.32:0/2665912631 <== osd.1 v2:172.21.15.32:6824/1551241759 19 ==== osd_op_reply(109 609.00000000 [setxattr (62)] v117'193 uv193 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f52c005c380 con 0x55f084d838d0 2024-08-03T20:58:58.472 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.472+0000 7f52d9fe6f40 1 -- 172.21.15.32:0/2665912631 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:110 12.7 12:e0b41d06:::609.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117) -- 0x55f084d91c70 con 0x55f084d838d0 2024-08-03T20:58:58.474 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.474+0000 7f52d2800640 1 -- 172.21.15.32:0/2665912631 <== osd.1 v2:172.21.15.32:6824/1551241759 20 ==== osd_op_reply(110 609.00000000 [omap-set-header] v117'194 uv194 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f52c005c380 con 0x55f084d838d0 2024-08-03T20:58:58.474 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.474+0000 7f52d9fe6f40 1 -- 172.21.15.32:0/2665912631 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:111 12.f 12:f1f69726:::400.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117) -- 0x55f084d91c70 con 0x55f084d88020 2024-08-03T20:58:58.476 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.475+0000 7f52d3c00640 1 -- 172.21.15.32:0/2665912631 <== osd.6 v2:172.21.15.118:6825/801947386 12 ==== osd_op_reply(111 400.00000000 [create] v117'161 uv160 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f52cc00b040 con 0x55f084d88020 2024-08-03T20:58:58.476 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.475+0000 7f52d9fe6f40 1 -- 172.21.15.32:0/2665912631 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:112 12.f 12:f1f69726:::400.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e117) -- 0x55f084d91c70 con 0x55f084d88020 2024-08-03T20:58:58.477 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.477+0000 7f52d3c00640 1 -- 172.21.15.32:0/2665912631 <== osd.6 v2:172.21.15.118:6825/801947386 13 ==== osd_op_reply(112 400.00000000 [delete] v117'162 uv162 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f52cc00b040 con 0x55f084d88020 2024-08-03T20:58:58.477 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.477+0000 7f52d9fe6f40 1 -- 172.21.15.32:0/2665912631 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:113 12.f 12:f1f69726:::400.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117) -- 0x55f084d91c70 con 0x55f084d88020 2024-08-03T20:58:58.478 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::400.00000000:head# 2024-08-03T20:58:58.479 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.478+0000 7f52d3c00640 1 -- 172.21.15.32:0/2665912631 <== osd.6 v2:172.21.15.118:6825/801947386 14 ==== osd_op_reply(113 400.00000000 [create] v117'163 uv163 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f52cc00b040 con 0x55f084d88020 2024-08-03T20:58:58.479 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.478+0000 7f52d9fe6f40 1 -- 172.21.15.32:0/2665912631 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:114 12.f 12:f1f69726:::400.00000000:head [write 0~22 in=22b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117) -- 0x55f084d92640 con 0x55f084d88020 2024-08-03T20:58:58.480 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.479+0000 7f52d3c00640 1 -- 172.21.15.32:0/2665912631 <== osd.6 v2:172.21.15.118:6825/801947386 15 ==== osd_op_reply(114 400.00000000 [write 0~22] v117'164 uv164 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f52cc00b040 con 0x55f084d88020 2024-08-03T20:58:58.480 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.479+0000 7f52d9fe6f40 1 -- 172.21.15.32:0/2665912631 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:115 12.f 12:f1f69726:::400.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117) -- 0x55f084d92640 con 0x55f084d88020 2024-08-03T20:58:58.481 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.481+0000 7f52d3c00640 1 -- 172.21.15.32:0/2665912631 <== osd.6 v2:172.21.15.118:6825/801947386 16 ==== osd_op_reply(115 400.00000000 [omap-set-header] v117'165 uv165 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f52cc00b040 con 0x55f084d88020 2024-08-03T20:58:58.481 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.481+0000 7f52d9fe6f40 1 -- 172.21.15.32:0/2665912631 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:116 12.1f 12:fe07bba1:::100.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117) -- 0x55f084d92640 con 0x55f084d88020 2024-08-03T20:58:58.483 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.483+0000 7f52d3c00640 1 -- 172.21.15.32:0/2665912631 <== osd.6 v2:172.21.15.118:6825/801947386 17 ==== osd_op_reply(116 100.00000000.inode [create] v117'389 uv380 ondisk = -17 ((17) File exists)) ==== 162+0+0 (crc 0 0 0) 0x7f52cc00b040 con 0x55f084d88020 2024-08-03T20:58:58.483 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.483+0000 7f52d9fe6f40 1 -- 172.21.15.32:0/2665912631 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:117 12.1f 12:fe07bba1:::100.00000000.inode:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e117) -- 0x55f084d92640 con 0x55f084d88020 2024-08-03T20:58:58.485 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.484+0000 7f52d3c00640 1 -- 172.21.15.32:0/2665912631 <== osd.6 v2:172.21.15.118:6825/801947386 18 ==== osd_op_reply(117 100.00000000.inode [delete] v117'390 uv390 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7f52cc00b040 con 0x55f084d88020 2024-08-03T20:58:58.485 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.484+0000 7f52d9fe6f40 1 -- 172.21.15.32:0/2665912631 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:118 12.1f 12:fe07bba1:::100.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117) -- 0x55f084d92640 con 0x55f084d88020 2024-08-03T20:58:58.487 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::100.00000000.inode:head# 2024-08-03T20:58:58.487 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.486+0000 7f52d3c00640 1 -- 172.21.15.32:0/2665912631 <== osd.6 v2:172.21.15.118:6825/801947386 19 ==== osd_op_reply(118 100.00000000.inode [create] v117'391 uv391 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7f52cc00b040 con 0x55f084d88020 2024-08-03T20:58:58.487 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.486+0000 7f52d9fe6f40 1 -- 172.21.15.32:0/2665912631 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:119 12.1f 12:fe07bba1:::100.00000000.inode:head [write 0~582 in=582b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117) -- 0x55f084d92640 con 0x55f084d88020 2024-08-03T20:58:58.488 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.488+0000 7f52d3c00640 1 -- 172.21.15.32:0/2665912631 <== osd.6 v2:172.21.15.118:6825/801947386 20 ==== osd_op_reply(119 100.00000000.inode [write 0~582] v117'392 uv392 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7f52cc00b040 con 0x55f084d88020 2024-08-03T20:58:58.488 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.488+0000 7f52d9fe6f40 1 -- 172.21.15.32:0/2665912631 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:120 12.1f 12:fe07bba1:::100.00000000.inode:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117) -- 0x55f084d92640 con 0x55f084d88020 2024-08-03T20:58:58.490 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.489+0000 7f52d3c00640 1 -- 172.21.15.32:0/2665912631 <== osd.6 v2:172.21.15.118:6825/801947386 21 ==== osd_op_reply(120 100.00000000.inode [omap-set-header] v117'393 uv393 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7f52cc00b040 con 0x55f084d88020 2024-08-03T20:58:58.490 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.489+0000 7f52d9fe6f40 1 -- 172.21.15.32:0/2665912631 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:121 12.1f 12:ff5b34d6:::1.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117) -- 0x55f084d92640 con 0x55f084d88020 2024-08-03T20:58:58.492 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.491+0000 7f52d3c00640 1 -- 172.21.15.32:0/2665912631 <== osd.6 v2:172.21.15.118:6825/801947386 22 ==== osd_op_reply(121 1.00000000 [create] v117'394 uv387 ondisk = -17 ((17) File exists)) ==== 154+0+0 (crc 0 0 0) 0x7f52cc00b040 con 0x55f084d88020 2024-08-03T20:58:58.492 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.491+0000 7f52d9fe6f40 1 -- 172.21.15.32:0/2665912631 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:122 12.1f 12:ff5b34d6:::1.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e117) -- 0x55f084d92640 con 0x55f084d88020 2024-08-03T20:58:58.493 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.492+0000 7f52d3c00640 1 -- 172.21.15.32:0/2665912631 <== osd.6 v2:172.21.15.118:6825/801947386 23 ==== osd_op_reply(122 1.00000000 [delete] v117'395 uv395 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7f52cc00b040 con 0x55f084d88020 2024-08-03T20:58:58.493 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.492+0000 7f52d9fe6f40 1 -- 172.21.15.32:0/2665912631 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:123 12.1f 12:ff5b34d6:::1.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117) -- 0x55f084d92640 con 0x55f084d88020 2024-08-03T20:58:58.494 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.494+0000 7f52d3c00640 1 -- 172.21.15.32:0/2665912631 <== osd.6 v2:172.21.15.118:6825/801947386 24 ==== osd_op_reply(123 1.00000000 [create] v117'396 uv396 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7f52cc00b040 con 0x55f084d88020 2024-08-03T20:58:58.494 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::1.00000000:head# 2024-08-03T20:58:58.495 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.494+0000 7f52d9fe6f40 1 -- 172.21.15.32:0/2665912631 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:124 12.1f 12:ff5b34d6:::1.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117) -- 0x55f084d93650 con 0x55f084d88020 2024-08-03T20:58:58.496 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.496+0000 7f52d3c00640 1 -- 172.21.15.32:0/2665912631 <== osd.6 v2:172.21.15.118:6825/801947386 25 ==== osd_op_reply(124 1.00000000 [setxattr (30)] v117'397 uv397 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7f52cc00b040 con 0x55f084d88020 2024-08-03T20:58:58.496 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.496+0000 7f52d9fe6f40 1 -- 172.21.15.32:0/2665912631 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:125 12.1f 12:ff5b34d6:::1.00000000:head [setxattr parent (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117) -- 0x55f084d93650 con 0x55f084d88020 2024-08-03T20:58:58.498 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.497+0000 7f52d3c00640 1 -- 172.21.15.32:0/2665912631 <== osd.6 v2:172.21.15.118:6825/801947386 26 ==== osd_op_reply(125 1.00000000 [setxattr (30)] v117'398 uv398 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7f52cc00b040 con 0x55f084d88020 2024-08-03T20:58:58.498 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.497+0000 7f52d9fe6f40 1 -- 172.21.15.32:0/2665912631 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:126 12.1f 12:ff5b34d6:::1.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117) -- 0x55f084d92640 con 0x55f084d88020 2024-08-03T20:58:58.499 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.499+0000 7f52d3c00640 1 -- 172.21.15.32:0/2665912631 <== osd.6 v2:172.21.15.118:6825/801947386 27 ==== osd_op_reply(126 1.00000000 [omap-set-header] v117'399 uv399 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7f52cc00b040 con 0x55f084d88020 2024-08-03T20:58:58.499 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.499+0000 7f52d9fe6f40 1 -- 172.21.15.32:0/2665912631 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:127 12.1f 12:ff5b34d6:::1.00000000:head [omap-set-vals in=584b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117) -- 0x55f084d92da0 con 0x55f084d88020 2024-08-03T20:58:58.502 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.501+0000 7f52d3c00640 1 -- 172.21.15.32:0/2665912631 <== osd.6 v2:172.21.15.118:6825/801947386 28 ==== osd_op_reply(127 1.00000000 [omap-set-vals] v117'400 uv400 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7f52cc00b040 con 0x55f084d88020 2024-08-03T20:58:58.502 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.501+0000 7f52d9fe6f40 1 -- 172.21.15.32:0/2665912631 >> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] conn(0x55f084cd45e0 msgr2=0x55f084cd1150 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:58:58.502 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.501+0000 7f52d9fe6f40 1 --2- 172.21.15.32:0/2665912631 >> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] conn(0x55f084cd45e0 0x55f084cd1150 crc :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:58.502 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.501+0000 7f52d9fe6f40 1 -- 172.21.15.32:0/2665912631 >> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] conn(0x55f084d838d0 msgr2=0x55f084d85d10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:58:58.502 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.501+0000 7f52d9fe6f40 1 --2- 172.21.15.32:0/2665912631 >> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] conn(0x55f084d838d0 0x55f084d85d10 crc :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:58.503 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.501+0000 7f52d9fe6f40 1 -- 172.21.15.32:0/2665912631 >> [v2:172.21.15.32:6800/9088303,v1:172.21.15.32:6801/9088303] conn(0x55f084d87220 msgr2=0x55f084d92100 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:58:58.503 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.501+0000 7f52d9fe6f40 1 --2- 172.21.15.32:0/2665912631 >> [v2:172.21.15.32:6800/9088303,v1:172.21.15.32:6801/9088303] conn(0x55f084d87220 0x55f084d92100 crc :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:58.503 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.501+0000 7f52d9fe6f40 1 -- 172.21.15.32:0/2665912631 >> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] conn(0x55f084d7fbe0 msgr2=0x55f084d82000 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:58:58.503 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.501+0000 7f52d9fe6f40 1 --2- 172.21.15.32:0/2665912631 >> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] conn(0x55f084d7fbe0 0x55f084d82000 crc :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:58.503 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.501+0000 7f52d9fe6f40 1 -- 172.21.15.32:0/2665912631 >> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] conn(0x55f084d88020 msgr2=0x55f084d88430 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:58:58.503 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.501+0000 7f52d9fe6f40 1 --2- 172.21.15.32:0/2665912631 >> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] conn(0x55f084d88020 0x55f084d88430 crc :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:58.503 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.501+0000 7f52d9fe6f40 1 -- 172.21.15.32:0/2665912631 >> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] conn(0x55f084d8b450 msgr2=0x55f084d8d870 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:58:58.503 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.501+0000 7f52d9fe6f40 1 --2- 172.21.15.32:0/2665912631 >> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] conn(0x55f084d8b450 0x55f084d8d870 crc :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:58.503 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.501+0000 7f52d3c00640 1 -- 172.21.15.32:0/2665912631 reap_dead start 2024-08-03T20:58:58.503 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.501+0000 7f52d3c00640 1 -- 172.21.15.32:0/2665912631 reap_dead start 2024-08-03T20:58:58.503 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.501+0000 7f52d3c00640 1 -- 172.21.15.32:0/2665912631 reap_dead start 2024-08-03T20:58:58.503 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.501+0000 7f52d9fe6f40 1 -- 172.21.15.32:0/2665912631 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f52b40759f0 msgr2=0x7f52b4077f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:58:58.503 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.501+0000 7f52d9fe6f40 1 --2- 172.21.15.32:0/2665912631 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f52b40759f0 0x7f52b4077f00 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 crypto rx=0x7f52c4025de0 tx=0x7f52c4004880 comp rx=0 tx=0).stop 2024-08-03T20:58:58.503 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.501+0000 7f52d9fe6f40 1 -- 172.21.15.32:0/2665912631 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55f084cd8cb0 msgr2=0x55f084cb2be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:58:58.504 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.501+0000 7f52d9fe6f40 1 --2- 172.21.15.32:0/2665912631 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55f084cd8cb0 0x55f084cb2be0 secure :-1 s=READY pgs=1870 cs=0 l=1 rev1=1 crypto rx=0x7f52c000bd90 tx=0x7f52c00095f0 comp rx=0 tx=0).stop 2024-08-03T20:58:58.504 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.501+0000 7f52d9fe6f40 1 -- 172.21.15.32:0/2665912631 shutdown_connections 2024-08-03T20:58:58.504 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.501+0000 7f52d9fe6f40 1 --2- 172.21.15.32:0/2665912631 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f52b40759f0 0x7f52b4077f00 unknown :-1 s=CLOSED pgs=754 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:58.504 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.501+0000 7f52d9fe6f40 1 --2- 172.21.15.32:0/2665912631 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55f084cd8cb0 0x55f084cb2be0 unknown :-1 s=CLOSED pgs=1870 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:58.504 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.501+0000 7f52d9fe6f40 1 -- 172.21.15.32:0/2665912631 >> 172.21.15.32:0/2665912631 conn(0x55f084b24ac0 msgr2=0x55f084cd6d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:58:58.504 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.501+0000 7f52d9fe6f40 1 -- 172.21.15.32:0/2665912631 shutdown_connections 2024-08-03T20:58:58.504 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.501+0000 7f52d9fe6f40 1 -- 172.21.15.32:0/2665912631 wait complete. 2024-08-03T20:58:58.517 DEBUG:teuthology.orchestra.run.smithi032:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata setomapval 100.00000000 stray0_head deadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeef 2024-08-03T20:58:58.662 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.662+0000 7fe3d0e2ef40 1 Processor -- start 2024-08-03T20:58:58.662 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.662+0000 7fe3d0e2ef40 1 -- start start 2024-08-03T20:58:58.664 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.663+0000 7fe3d0e2ef40 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x560ca67573f0 0x560ca676b5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:58:58.664 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.663+0000 7fe3d0e2ef40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x560ca6772d90 0x560ca67fec90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:58:58.664 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.663+0000 7fe3d0e2ef40 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x560ca67ff1d0 0x560ca6752f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:58:58.664 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.663+0000 7fe3d0e2ef40 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x560ca6648bf0 con 0x560ca67573f0 2024-08-03T20:58:58.664 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.663+0000 7fe3d0e2ef40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x560ca66556b0 con 0x560ca6772d90 2024-08-03T20:58:58.665 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.663+0000 7fe3d0e2ef40 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x560ca65eae30 con 0x560ca67ff1d0 2024-08-03T20:58:58.665 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.663+0000 7fe3ca000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x560ca67573f0 0x560ca676b5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:58:58.665 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.663+0000 7fe3ca000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x560ca67573f0 0x560ca676b5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:42130/0 (socket says 172.21.15.32:42130) 2024-08-03T20:58:58.665 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.663+0000 7fe3ca000640 1 -- 172.21.15.32:0/33959178 learned_addr learned my addr 172.21.15.32:0/33959178 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:58:58.665 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.663+0000 7fe3c9600640 1 --2- 172.21.15.32:0/33959178 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x560ca6772d90 0x560ca67fec90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:58:58.665 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.663+0000 7fe3caa00640 1 --2- 172.21.15.32:0/33959178 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x560ca67ff1d0 0x560ca6752f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:58:58.665 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.664+0000 7fe3ca000640 1 -- 172.21.15.32:0/33959178 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x560ca67ff1d0 msgr2=0x560ca6752f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:58:58.665 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.664+0000 7fe3ca000640 1 --2- 172.21.15.32:0/33959178 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x560ca67ff1d0 0x560ca6752f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:58.665 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.664+0000 7fe3ca000640 1 -- 172.21.15.32:0/33959178 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x560ca6772d90 msgr2=0x560ca67fec90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:58:58.665 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.664+0000 7fe3ca000640 1 --2- 172.21.15.32:0/33959178 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x560ca6772d90 0x560ca67fec90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:58.665 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.664+0000 7fe3ca000640 1 -- 172.21.15.32:0/33959178 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560ca676c280 con 0x560ca67573f0 2024-08-03T20:58:58.665 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.664+0000 7fe3caa00640 1 --2- 172.21.15.32:0/33959178 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x560ca67ff1d0 0x560ca6752f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-03T20:58:58.665 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.664+0000 7fe3ca000640 1 --2- 172.21.15.32:0/33959178 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x560ca67573f0 0x560ca676b5a0 secure :-1 s=READY pgs=1872 cs=0 l=1 rev1=1 crypto rx=0x7fe3bc001e50 tx=0x7fe3bc00b730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bfafd9bde543fcbc server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:58:58.667 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.664+0000 7fe3c3e00640 1 -- 172.21.15.32:0/33959178 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3bc012070 con 0x560ca67573f0 2024-08-03T20:58:58.667 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.664+0000 7fe3c3e00640 1 -- 172.21.15.32:0/33959178 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fe3bc006050 con 0x560ca67573f0 2024-08-03T20:58:58.668 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.664+0000 7fe3c3e00640 1 -- 172.21.15.32:0/33959178 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3bc00ebe0 con 0x560ca67573f0 2024-08-03T20:58:58.668 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.665+0000 7fe3d0e2ef40 1 -- 172.21.15.32:0/33959178 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x560ca67573f0 msgr2=0x560ca676b5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:58:58.668 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.665+0000 7fe3d0e2ef40 1 --2- 172.21.15.32:0/33959178 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x560ca67573f0 0x560ca676b5a0 secure :-1 s=READY pgs=1872 cs=0 l=1 rev1=1 crypto rx=0x7fe3bc001e50 tx=0x7fe3bc00b730 comp rx=0 tx=0).stop 2024-08-03T20:58:58.668 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.665+0000 7fe3d0e2ef40 1 -- 172.21.15.32:0/33959178 shutdown_connections 2024-08-03T20:58:58.668 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.665+0000 7fe3d0e2ef40 1 --2- 172.21.15.32:0/33959178 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x560ca6772d90 0x560ca67fec90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:58.668 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.665+0000 7fe3d0e2ef40 1 --2- 172.21.15.32:0/33959178 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x560ca67ff1d0 0x560ca6752f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:58.668 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.665+0000 7fe3d0e2ef40 1 --2- 172.21.15.32:0/33959178 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x560ca67573f0 0x560ca676b5a0 unknown :-1 s=CLOSED pgs=1872 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:58.668 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.665+0000 7fe3d0e2ef40 1 -- 172.21.15.32:0/33959178 >> 172.21.15.32:0/33959178 conn(0x560ca65bcac0 msgr2=0x560ca669c560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:58:58.668 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.665+0000 7fe3d0e2ef40 1 -- 172.21.15.32:0/33959178 shutdown_connections 2024-08-03T20:58:58.668 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.665+0000 7fe3d0e2ef40 1 -- 172.21.15.32:0/33959178 wait complete. 2024-08-03T20:58:58.668 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.666+0000 7fe3d0e2ef40 1 Processor -- start 2024-08-03T20:58:58.669 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.666+0000 7fe3d0e2ef40 1 -- start start 2024-08-03T20:58:58.670 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.668+0000 7fe3d0e2ef40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x560ca67573f0 0x560ca6809f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:58:58.670 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.668+0000 7fe3d0e2ef40 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x560ca6772d90 0x560ca674d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:58:58.671 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.668+0000 7fe3d0e2ef40 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x560ca67ff1d0 0x560ca6750190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:58:58.671 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.668+0000 7fe3d0e2ef40 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x560ca6768980 con 0x560ca6772d90 2024-08-03T20:58:58.671 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.668+0000 7fe3d0e2ef40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x560ca6768eb0 con 0x560ca67573f0 2024-08-03T20:58:58.671 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.668+0000 7fe3d0e2ef40 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x560ca680acd0 con 0x560ca67ff1d0 2024-08-03T20:58:58.671 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.668+0000 7fe3c9600640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x560ca6772d90 0x560ca674d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:58:58.671 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.668+0000 7fe3c9600640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x560ca6772d90 0x560ca674d300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:42146/0 (socket says 172.21.15.32:42146) 2024-08-03T20:58:58.672 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.668+0000 7fe3c9600640 1 -- 172.21.15.32:0/215900608 learned_addr learned my addr 172.21.15.32:0/215900608 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:58:58.672 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.668+0000 7fe3caa00640 1 --2- 172.21.15.32:0/215900608 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x560ca67ff1d0 0x560ca6750190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:58:58.672 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.668+0000 7fe3ca000640 1 --2- 172.21.15.32:0/215900608 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x560ca67573f0 0x560ca6809f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:58:58.672 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.668+0000 7fe3c9600640 1 -- 172.21.15.32:0/215900608 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x560ca67ff1d0 msgr2=0x560ca6750190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:58:58.672 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.668+0000 7fe3c9600640 1 --2- 172.21.15.32:0/215900608 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x560ca67ff1d0 0x560ca6750190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:58.672 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.668+0000 7fe3c9600640 1 -- 172.21.15.32:0/215900608 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x560ca67573f0 msgr2=0x560ca6809f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:58:58.672 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.668+0000 7fe3c9600640 1 --2- 172.21.15.32:0/215900608 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x560ca67573f0 0x560ca6809f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:58.672 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.668+0000 7fe3c9600640 1 -- 172.21.15.32:0/215900608 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560ca6753a30 con 0x560ca6772d90 2024-08-03T20:58:58.673 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.669+0000 7fe3c9600640 1 --2- 172.21.15.32:0/215900608 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x560ca6772d90 0x560ca674d300 secure :-1 s=READY pgs=1873 cs=0 l=1 rev1=1 crypto rx=0x7fe3b80465c0 tx=0x7fe3b8004580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:58:58.673 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.669+0000 7fe3c2a00640 1 -- 172.21.15.32:0/215900608 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3b8066030 con 0x560ca6772d90 2024-08-03T20:58:58.673 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.669+0000 7fe3c2a00640 1 -- 172.21.15.32:0/215900608 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fe3b8056470 con 0x560ca6772d90 2024-08-03T20:58:58.673 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.669+0000 7fe3c2a00640 1 -- 172.21.15.32:0/215900608 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3b8056680 con 0x560ca6772d90 2024-08-03T20:58:58.673 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.669+0000 7fe3d0e2ef40 1 -- 172.21.15.32:0/215900608 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x560ca6753d10 con 0x560ca6772d90 2024-08-03T20:58:58.673 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.669+0000 7fe3d0e2ef40 1 -- 172.21.15.32:0/215900608 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560ca6752230 con 0x560ca6772d90 2024-08-03T20:58:58.674 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.671+0000 7fe3c2a00640 1 -- 172.21.15.32:0/215900608 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7fe3b8069070 con 0x560ca6772d90 2024-08-03T20:58:58.674 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.672+0000 7fe3c2a00640 1 --2- 172.21.15.32:0/215900608 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fe3ac0759f0 0x7fe3ac077f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:58:58.674 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.672+0000 7fe3c2a00640 1 -- 172.21.15.32:0/215900608 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(117..117 src has 1..117) ==== 8606+0+0 (secure 0 0 0) 0x7fe3b80dc340 con 0x560ca6772d90 2024-08-03T20:58:58.674 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.673+0000 7fe3ca000640 1 --2- 172.21.15.32:0/215900608 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fe3ac0759f0 0x7fe3ac077f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:58:58.674 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.673+0000 7fe3d0e2ef40 1 --2- 172.21.15.32:0/215900608 >> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] conn(0x560ca680a5d0 0x560ca6752f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:58:58.675 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.673+0000 7fe3d0e2ef40 1 -- 172.21.15.32:0/215900608 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:1 12.13 12:cd5a64a3:::100.00000000:head [omap-set-vals in=103b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117) -- 0x560ca674d960 con 0x560ca680a5d0 2024-08-03T20:58:58.675 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.673+0000 7fe3caa00640 1 --2- 172.21.15.32:0/215900608 >> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] conn(0x560ca680a5d0 0x560ca6752f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:58:58.675 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.673+0000 7fe3ca000640 1 --2- 172.21.15.32:0/215900608 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fe3ac0759f0 0x7fe3ac077f00 secure :-1 s=READY pgs=755 cs=0 l=1 rev1=1 crypto rx=0x7fe3bc001e50 tx=0x7fe3bc01a040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:58:58.675 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.673+0000 7fe3caa00640 1 --2- 172.21.15.32:0/215900608 >> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] conn(0x560ca680a5d0 0x560ca6752f40 crc :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:58:58.676 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.675+0000 7fe3caa00640 1 -- 172.21.15.32:0/215900608 <== osd.1 v2:172.21.15.32:6824/1551241759 1 ==== osd_op_reply(1 100.00000000 [omap-set-vals] v117'324 uv324 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe3c4010610 con 0x560ca680a5d0 2024-08-03T20:58:58.676 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.675+0000 7fe3d0e2ef40 1 -- 172.21.15.32:0/215900608 >> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] conn(0x560ca680a5d0 msgr2=0x560ca6752f40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:58:58.676 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.675+0000 7fe3d0e2ef40 1 --2- 172.21.15.32:0/215900608 >> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] conn(0x560ca680a5d0 0x560ca6752f40 crc :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:58.676 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.675+0000 7fe3d0e2ef40 1 -- 172.21.15.32:0/215900608 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fe3ac0759f0 msgr2=0x7fe3ac077f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:58:58.676 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.675+0000 7fe3d0e2ef40 1 --2- 172.21.15.32:0/215900608 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fe3ac0759f0 0x7fe3ac077f00 secure :-1 s=READY pgs=755 cs=0 l=1 rev1=1 crypto rx=0x7fe3bc001e50 tx=0x7fe3bc01a040 comp rx=0 tx=0).stop 2024-08-03T20:58:58.677 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.675+0000 7fe3d0e2ef40 1 -- 172.21.15.32:0/215900608 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x560ca6772d90 msgr2=0x560ca674d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:58:58.677 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.675+0000 7fe3d0e2ef40 1 --2- 172.21.15.32:0/215900608 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x560ca6772d90 0x560ca674d300 secure :-1 s=READY pgs=1873 cs=0 l=1 rev1=1 crypto rx=0x7fe3b80465c0 tx=0x7fe3b8004580 comp rx=0 tx=0).stop 2024-08-03T20:58:58.677 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.675+0000 7fe3caa00640 1 -- 172.21.15.32:0/215900608 reap_dead start 2024-08-03T20:58:58.677 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.675+0000 7fe3d0e2ef40 1 -- 172.21.15.32:0/215900608 shutdown_connections 2024-08-03T20:58:58.677 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.675+0000 7fe3d0e2ef40 1 -- 172.21.15.32:0/215900608 >> 172.21.15.32:0/215900608 conn(0x560ca65bcac0 msgr2=0x560ca676b6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:58:58.677 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.676+0000 7fe3d0e2ef40 1 -- 172.21.15.32:0/215900608 shutdown_connections 2024-08-03T20:58:58.677 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.676+0000 7fe3d0e2ef40 1 -- 172.21.15.32:0/215900608 wait complete. 2024-08-03T20:58:58.687 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs joinable true 2024-08-03T20:58:58.917 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.916+0000 7f408f000640 1 -- 172.21.15.32:0/2906936336 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f40880cccf0 msgr2=0x7f40880cd100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:58:58.917 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.916+0000 7f408f000640 1 --2- 172.21.15.32:0/2906936336 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f40880cccf0 0x7f40880cd100 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7f4070009830 tx=0x7f407002a500 comp rx=0 tx=0).stop 2024-08-03T20:58:58.918 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.917+0000 7f408f000640 1 -- 172.21.15.32:0/2906936336 shutdown_connections 2024-08-03T20:58:58.918 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.917+0000 7f408f000640 1 --2- 172.21.15.32:0/2906936336 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f40880ced30 0x7f40880cf140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:58.918 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.917+0000 7f408f000640 1 --2- 172.21.15.32:0/2906936336 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f40880cccf0 0x7f40880cd100 unknown :-1 s=CLOSED pgs=569 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:58.918 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.917+0000 7f408f000640 1 -- 172.21.15.32:0/2906936336 >> 172.21.15.32:0/2906936336 conn(0x7f40880b1bd0 msgr2=0x7f40880b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:58:58.918 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.917+0000 7f408f000640 1 -- 172.21.15.32:0/2906936336 shutdown_connections 2024-08-03T20:58:58.918 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.917+0000 7f408f000640 1 -- 172.21.15.32:0/2906936336 wait complete. 2024-08-03T20:58:58.919 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.919+0000 7f408f000640 1 Processor -- start 2024-08-03T20:58:58.919 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.919+0000 7f408f000640 1 -- start start 2024-08-03T20:58:58.922 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.920+0000 7f408f000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f40880cccf0 0x7f408813c1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:58:58.922 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.920+0000 7f408f000640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f40880ced30 0x7f408813a670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:58:58.922 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.920+0000 7f408f000640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f4088145c50 0x7f4088144320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:58:58.922 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.920+0000 7f408f000640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f40880b4070 con 0x7f40880cccf0 2024-08-03T20:58:58.922 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.920+0000 7f408f000640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f40880b3ef0 con 0x7f40880ced30 2024-08-03T20:58:58.922 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.920+0000 7f408f000640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7f40880b41f0 con 0x7f4088145c50 2024-08-03T20:58:58.923 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.921+0000 7f4087e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f40880cccf0 0x7f408813c1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:58:58.923 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.921+0000 7f4087e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f40880cccf0 0x7f408813c1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:42148/0 (socket says 172.21.15.32:42148) 2024-08-03T20:58:58.923 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.921+0000 7f4087e00640 1 -- 172.21.15.32:0/1721498963 learned_addr learned my addr 172.21.15.32:0/1721498963 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:58:58.923 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.921+0000 7f407f400640 1 --2- 172.21.15.32:0/1721498963 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f40880ced30 0x7f408813a670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:58:58.923 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.921+0000 7f408d000640 1 --2- 172.21.15.32:0/1721498963 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f4088145c50 0x7f4088144320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:58:58.923 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.921+0000 7f4087e00640 1 -- 172.21.15.32:0/1721498963 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f4088145c50 msgr2=0x7f4088144320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:58:58.923 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.921+0000 7f4087e00640 1 --2- 172.21.15.32:0/1721498963 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f4088145c50 0x7f4088144320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:58.923 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.921+0000 7f4087e00640 1 -- 172.21.15.32:0/1721498963 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f40880ced30 msgr2=0x7f408813a670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:58:58.923 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.921+0000 7f4087e00640 1 --2- 172.21.15.32:0/1721498963 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f40880ced30 0x7f408813a670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:58:58.923 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.921+0000 7f4087e00640 1 -- 172.21.15.32:0/1721498963 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f408813abb0 con 0x7f40880cccf0 2024-08-03T20:58:58.923 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.921+0000 7f4087e00640 1 --2- 172.21.15.32:0/1721498963 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f40880cccf0 0x7f408813c1e0 secure :-1 s=READY pgs=1875 cs=0 l=1 rev1=1 crypto rx=0x7f407800c460 tx=0x7f407800c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:58:58.923 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.922+0000 7f4084c00640 1 -- 172.21.15.32:0/1721498963 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4078016020 con 0x7f40880cccf0 2024-08-03T20:58:58.923 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.922+0000 7f4084c00640 1 -- 172.21.15.32:0/1721498963 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f4078012940 con 0x7f40880cccf0 2024-08-03T20:58:58.924 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.922+0000 7f4084c00640 1 -- 172.21.15.32:0/1721498963 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4078012d90 con 0x7f40880cccf0 2024-08-03T20:58:58.924 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.922+0000 7f408f000640 1 -- 172.21.15.32:0/1721498963 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f408813c870 con 0x7f40880cccf0 2024-08-03T20:58:58.924 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.922+0000 7f408f000640 1 -- 172.21.15.32:0/1721498963 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f408813cdf0 con 0x7f40880cccf0 2024-08-03T20:58:58.924 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.923+0000 7f408f000640 1 -- 172.21.15.32:0/1721498963 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f40880b46e0 con 0x7f40880cccf0 2024-08-03T20:58:58.924 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.924+0000 7f4084c00640 1 -- 172.21.15.32:0/1721498963 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f4078019020 con 0x7f40880cccf0 2024-08-03T20:58:58.925 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.925+0000 7f4084c00640 1 --2- 172.21.15.32:0/1721498963 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f4064077010 0x7f4064079520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:58:58.925 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.925+0000 7f4084c00640 1 -- 172.21.15.32:0/1721498963 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(117..117 src has 1..117) ==== 8606+0+0 (secure 0 0 0) 0x7f4078098e90 con 0x7f40880cccf0 2024-08-03T20:58:58.926 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.925+0000 7f407f400640 1 --2- 172.21.15.32:0/1721498963 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f4064077010 0x7f4064079520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:58:58.926 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.926+0000 7f407f400640 1 --2- 172.21.15.32:0/1721498963 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f4064077010 0x7f4064079520 secure :-1 s=READY pgs=756 cs=0 l=1 rev1=1 crypto rx=0x7f4070034000 tx=0x7f4070004730 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:58:58.930 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:58.930+0000 7f4084c00640 1 -- 172.21.15.32:0/1721498963 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7f4078020070 con 0x7f40880cccf0 2024-08-03T20:58:59.177 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:58:59.176+0000 7f408f000640 1 -- 172.21.15.32:0/1721498963 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "joinable", "val": "true"} v 0) -- 0x7f4088070690 con 0x7f40880cccf0 2024-08-03T20:59:00.066 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:00.065+0000 7f4084c00640 1 -- 172.21.15.32:0/1721498963 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "joinable", "val": "true"}]=0 cephfs marked joinable; MDS may join as newly active. v274) ==== 162+0+0 (secure 0 0 0) 0x7f407809a020 con 0x7f40880cccf0 2024-08-03T20:59:00.066 INFO:teuthology.orchestra.run.smithi032.stderr:cephfs marked joinable; MDS may join as newly active. 2024-08-03T20:59:00.069 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:00.068+0000 7f408f000640 1 -- 172.21.15.32:0/1721498963 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f4064077010 msgr2=0x7f4064079520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:59:00.069 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:00.068+0000 7f408f000640 1 --2- 172.21.15.32:0/1721498963 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f4064077010 0x7f4064079520 secure :-1 s=READY pgs=756 cs=0 l=1 rev1=1 crypto rx=0x7f4070034000 tx=0x7f4070004730 comp rx=0 tx=0).stop 2024-08-03T20:59:00.069 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:00.068+0000 7f408f000640 1 -- 172.21.15.32:0/1721498963 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f40880cccf0 msgr2=0x7f408813c1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:59:00.069 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:00.068+0000 7f408f000640 1 --2- 172.21.15.32:0/1721498963 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f40880cccf0 0x7f408813c1e0 secure :-1 s=READY pgs=1875 cs=0 l=1 rev1=1 crypto rx=0x7f407800c460 tx=0x7f407800c920 comp rx=0 tx=0).stop 2024-08-03T20:59:00.069 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:00.068+0000 7f408f000640 1 -- 172.21.15.32:0/1721498963 shutdown_connections 2024-08-03T20:59:00.069 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:00.068+0000 7f408f000640 1 --2- 172.21.15.32:0/1721498963 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f4064077010 0x7f4064079520 unknown :-1 s=CLOSED pgs=756 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:59:00.070 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:00.069+0000 7f408f000640 1 --2- 172.21.15.32:0/1721498963 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f40880ced30 0x7f408813a670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:59:00.070 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:00.069+0000 7f408f000640 1 --2- 172.21.15.32:0/1721498963 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f4088145c50 0x7f4088144320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:59:00.070 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:00.069+0000 7f408f000640 1 --2- 172.21.15.32:0/1721498963 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f40880cccf0 0x7f408813c1e0 unknown :-1 s=CLOSED pgs=1875 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:59:00.070 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:00.069+0000 7f408f000640 1 -- 172.21.15.32:0/1721498963 >> 172.21.15.32:0/1721498963 conn(0x7f40880b1bd0 msgr2=0x7f408815b350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:59:00.070 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:00.069+0000 7f408f000640 1 -- 172.21.15.32:0/1721498963 shutdown_connections 2024-08-03T20:59:00.070 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:00.069+0000 7f408f000640 1 -- 172.21.15.32:0/1721498963 wait complete. 2024-08-03T20:59:00.101 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-08-03T20:59:00.335 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:00.334+0000 7fa613200640 1 -- 172.21.15.32:0/1566639802 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa5f8003fd0 msgr2=0x7fa5f8004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:59:00.335 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:00.334+0000 7fa613200640 1 --2- 172.21.15.32:0/1566639802 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa5f8003fd0 0x7fa5f8004800 secure :-1 s=READY pgs=1877 cs=0 l=1 rev1=1 crypto rx=0x7fa5fc009b20 tx=0x7fa5f8005830 comp rx=0 tx=0).stop 2024-08-03T20:59:00.335 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:00.334+0000 7fa612800640 1 -- 172.21.15.32:0/1566639802 reap_dead start 2024-08-03T20:59:00.335 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:00.334+0000 7fa613200640 1 -- 172.21.15.32:0/1566639802 shutdown_connections 2024-08-03T20:59:00.335 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:00.334+0000 7fa613200640 1 -- 172.21.15.32:0/1566639802 >> 172.21.15.32:0/1566639802 conn(0x7fa6140d4080 msgr2=0x7fa6140d4490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:59:00.335 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:00.334+0000 7fa613200640 1 -- 172.21.15.32:0/1566639802 shutdown_connections 2024-08-03T20:59:00.335 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:00.334+0000 7fa613200640 1 -- 172.21.15.32:0/1566639802 wait complete. 2024-08-03T20:59:00.336 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:00.336+0000 7fa613200640 1 Processor -- start 2024-08-03T20:59:00.337 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:00.336+0000 7fa613200640 1 -- start start 2024-08-03T20:59:00.339 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:00.338+0000 7fa613200640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fa5f8003fd0 0x7fa6140a1580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:59:00.339 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:00.338+0000 7fa613200640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa6140c70f0 0x7fa6141f2f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:59:00.340 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:00.338+0000 7fa613200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa6140c7500 0x7fa6141f3480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:59:00.340 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:00.338+0000 7fa613200640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fa6140b33c0 con 0x7fa6140c7500 2024-08-03T20:59:00.340 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:00.338+0000 7fa613200640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7fa6140b3240 con 0x7fa6140c70f0 2024-08-03T20:59:00.340 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:00.338+0000 7fa613200640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7fa6140b3540 con 0x7fa5f8003fd0 2024-08-03T20:59:00.340 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:00.338+0000 7fa612800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa6140c7500 0x7fa6141f3480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:59:00.340 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:00.338+0000 7fa612800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa6140c7500 0x7fa6141f3480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:49344/0 (socket says 172.21.15.32:49344) 2024-08-03T20:59:00.340 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:00.338+0000 7fa612800640 1 -- 172.21.15.32:0/3860573962 learned_addr learned my addr 172.21.15.32:0/3860573962 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:59:00.340 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:00.338+0000 7fa611e00640 1 --2- 172.21.15.32:0/3860573962 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fa5f8003fd0 0x7fa6140a1580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:59:00.340 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:00.338+0000 7fa609400640 1 --2- 172.21.15.32:0/3860573962 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa6140c70f0 0x7fa6141f2f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:59:00.340 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:00.338+0000 7fa612800640 1 -- 172.21.15.32:0/3860573962 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fa5f8003fd0 msgr2=0x7fa6140a1580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:59:00.340 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:00.338+0000 7fa612800640 1 --2- 172.21.15.32:0/3860573962 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fa5f8003fd0 0x7fa6140a1580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:59:00.340 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:00.338+0000 7fa612800640 1 -- 172.21.15.32:0/3860573962 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa6140c70f0 msgr2=0x7fa6141f2f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:59:00.340 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:00.338+0000 7fa612800640 1 --2- 172.21.15.32:0/3860573962 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa6140c70f0 0x7fa6141f2f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:59:00.341 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:00.338+0000 7fa612800640 1 -- 172.21.15.32:0/3860573962 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa6140a1ac0 con 0x7fa6140c7500 2024-08-03T20:59:00.341 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:00.338+0000 7fa609400640 1 --2- 172.21.15.32:0/3860573962 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa6140c70f0 0x7fa6141f2f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-03T20:59:00.341 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:00.339+0000 7fa612800640 1 --2- 172.21.15.32:0/3860573962 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa6140c7500 0x7fa6141f3480 secure :-1 s=READY pgs=1878 cs=0 l=1 rev1=1 crypto rx=0x7fa6140cee20 tx=0x7fa60c006720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:59:00.341 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:00.339+0000 7fa60aa00640 1 -- 172.21.15.32:0/3860573962 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa60c00fa20 con 0x7fa6140c7500 2024-08-03T20:59:00.341 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:00.339+0000 7fa60aa00640 1 -- 172.21.15.32:0/3860573962 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fa60c00fbe0 con 0x7fa6140c7500 2024-08-03T20:59:00.341 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:00.339+0000 7fa613200640 1 -- 172.21.15.32:0/3860573962 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa6141f39c0 con 0x7fa6140c7500 2024-08-03T20:59:00.341 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:00.339+0000 7fa60aa00640 1 -- 172.21.15.32:0/3860573962 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa60c011430 con 0x7fa6140c7500 2024-08-03T20:59:00.341 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:00.339+0000 7fa613200640 1 -- 172.21.15.32:0/3860573962 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa6141f98e0 con 0x7fa6140c7500 2024-08-03T20:59:00.342 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:00.340+0000 7fa613200640 1 -- 172.21.15.32:0/3860573962 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa6140ce460 con 0x7fa6140c7500 2024-08-03T20:59:00.342 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:00.341+0000 7fa60aa00640 1 -- 172.21.15.32:0/3860573962 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7fa60c0115f0 con 0x7fa6140c7500 2024-08-03T20:59:00.343 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:00.342+0000 7fa60aa00640 1 --2- 172.21.15.32:0/3860573962 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fa5f0077010 0x7fa5f0079520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:59:00.343 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:00.342+0000 7fa60aa00640 1 -- 172.21.15.32:0/3860573962 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(117..117 src has 1..117) ==== 8606+0+0 (secure 0 0 0) 0x7fa60c09a060 con 0x7fa6140c7500 2024-08-03T20:59:00.343 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:00.343+0000 7fa611e00640 1 --2- 172.21.15.32:0/3860573962 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fa5f0077010 0x7fa5f0079520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:59:00.344 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:00.344+0000 7fa611e00640 1 --2- 172.21.15.32:0/3860573962 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fa5f0077010 0x7fa5f0079520 secure :-1 s=READY pgs=757 cs=0 l=1 rev1=1 crypto rx=0x7fa604009af0 tx=0x7fa604005f90 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:59:00.347 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:00.347+0000 7fa60aa00640 1 -- 172.21.15.32:0/3860573962 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7fa60c062660 con 0x7fa6140c7500 2024-08-03T20:59:00.593 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:00.592+0000 7fa613200640 1 -- 172.21.15.32:0/3860573962 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa6140d1630 con 0x7fa6140c7500 2024-08-03T20:59:00.593 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:00.593+0000 7fa60aa00640 1 -- 172.21.15.32:0/3860573962 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 274 v274) ==== 95+0+5223 (secure 0 0 0) 0x7fa60c061f60 con 0x7fa6140c7500 2024-08-03T20:59:00.593 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-03T20:59:00.593 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":274,"btime":"2024-08-03T20:59:00:062816+0000","default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6439,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6835/531080160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6834","nonce":531080160},{"type":"v1","addr":"172.21.15.118:6835","nonce":531080160}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":84},{"gid":6454,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6837/2796882319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6836","nonce":2796882319},{"type":"v1","addr":"172.21.15.118:6837","nonce":2796882319}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":86},{"gid":6813,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6834/2686933890","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":2686933890},{"type":"v1","addr":"172.21.15.32:6834","nonce":2686933890}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":85}],"filesystems":[{"mdsmap":{"epoch":274,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-08-03T20:54:26.950518+0000","modified":"2024-08-03T20:59:00.062792+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":117,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":9684},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9684":{"gid":9684,"name":"a","rank":0,"incarnation":274,"state":"up:replay","state_seq":1,"addr":"172.21.15.32:6835/506775245","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6833","nonce":506775245},{"type":"v1","addr":"172.21.15.32:6835","nonce":506775245}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":6}]} 2024-08-03T20:59:00.594 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 274 2024-08-03T20:59:00.596 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:00.595+0000 7fa613200640 1 -- 172.21.15.32:0/3860573962 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fa5f0077010 msgr2=0x7fa5f0079520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:59:00.596 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:00.595+0000 7fa613200640 1 --2- 172.21.15.32:0/3860573962 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fa5f0077010 0x7fa5f0079520 secure :-1 s=READY pgs=757 cs=0 l=1 rev1=1 crypto rx=0x7fa604009af0 tx=0x7fa604005f90 comp rx=0 tx=0).stop 2024-08-03T20:59:00.596 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:00.595+0000 7fa613200640 1 -- 172.21.15.32:0/3860573962 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa6140c7500 msgr2=0x7fa6141f3480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:59:00.596 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:00.595+0000 7fa613200640 1 --2- 172.21.15.32:0/3860573962 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa6140c7500 0x7fa6141f3480 secure :-1 s=READY pgs=1878 cs=0 l=1 rev1=1 crypto rx=0x7fa6140cee20 tx=0x7fa60c006720 comp rx=0 tx=0).stop 2024-08-03T20:59:00.596 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:00.596+0000 7fa613200640 1 -- 172.21.15.32:0/3860573962 shutdown_connections 2024-08-03T20:59:00.596 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:00.596+0000 7fa613200640 1 --2- 172.21.15.32:0/3860573962 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fa5f0077010 0x7fa5f0079520 unknown :-1 s=CLOSED pgs=757 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:59:00.596 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:00.596+0000 7fa613200640 1 --2- 172.21.15.32:0/3860573962 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa6140c70f0 0x7fa6141f2f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:59:00.597 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:00.596+0000 7fa613200640 1 --2- 172.21.15.32:0/3860573962 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa6140c7500 0x7fa6141f3480 unknown :-1 s=CLOSED pgs=1878 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:59:00.597 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:00.596+0000 7fa613200640 1 --2- 172.21.15.32:0/3860573962 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fa5f8003fd0 0x7fa6140a1580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:59:00.597 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:00.596+0000 7fa613200640 1 -- 172.21.15.32:0/3860573962 >> 172.21.15.32:0/3860573962 conn(0x7fa6140d4080 msgr2=0x7fa6140d1c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:59:00.597 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:00.596+0000 7fa613200640 1 -- 172.21.15.32:0/3860573962 shutdown_connections 2024-08-03T20:59:00.597 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:00.596+0000 7fa613200640 1 -- 172.21.15.32:0/3860573962 wait complete. 2024-08-03T20:59:00.619 DEBUG:tasks.ceph_test_case:wait_until_true: waiting (timeout=60 retry_count=0)... 2024-08-03T20:59:03.074 INFO:tasks.ceph.mds.a.smithi032.stderr:2024-08-03T20:59:03.073+0000 7fb139a00640 -1 mds.0.damage notify_dentry Damage to dentries in fragment * of ino 0x100is fatal because it is a system directory for this rank 2024-08-03T20:59:03.076 INFO:tasks.ceph.mon.a.smithi032.stderr:2024-08-03T20:59:03.076+0000 7f4e6d600640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-08-03T20:59:03.076 INFO:tasks.ceph.mon.a.smithi032.stderr:2024-08-03T20:59:03.076+0000 7f4e6d600640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 mds daemon damaged (MDS_DAMAGE) 2024-08-03T20:59:03.118 INFO:tasks.ceph.mds.a.smithi032.stderr:2024-08-03T20:59:03.117+0000 7fb13ea00640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-08-03T20:59:05.620 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-08-03T20:59:05.859 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:05.858+0000 7f3027200640 1 -- 172.21.15.32:0/490652720 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f302006e930 msgr2=0x7f3020144f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:59:05.859 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:05.858+0000 7f3027200640 1 --2- 172.21.15.32:0/490652720 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f302006e930 0x7f3020144f90 secure :-1 s=READY pgs=1880 cs=0 l=1 rev1=1 crypto rx=0x7f30200afaf0 tx=0x7f301002a4a0 comp rx=0 tx=0).stop 2024-08-03T20:59:05.860 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:05.858+0000 7f3027200640 1 -- 172.21.15.32:0/490652720 shutdown_connections 2024-08-03T20:59:05.860 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:05.858+0000 7f3027200640 1 --2- 172.21.15.32:0/490652720 >> v2:172.21.15.118:3300/0 conn(0x7f3020073e60 0x7f3020074270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:59:05.860 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:05.858+0000 7f3027200640 1 --2- 172.21.15.32:0/490652720 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f302006e930 0x7f3020144f90 unknown :-1 s=CLOSED pgs=1880 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:59:05.860 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:05.858+0000 7f3027200640 1 -- 172.21.15.32:0/490652720 >> 172.21.15.32:0/490652720 conn(0x7f30200b1bd0 msgr2=0x7f30200b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:59:05.860 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:05.858+0000 7f3027200640 1 -- 172.21.15.32:0/490652720 shutdown_connections 2024-08-03T20:59:05.860 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:05.858+0000 7f3027200640 1 -- 172.21.15.32:0/490652720 wait complete. 2024-08-03T20:59:05.861 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:05.861+0000 7f3027200640 1 Processor -- start 2024-08-03T20:59:05.861 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:05.861+0000 7f3027200640 1 -- start start 2024-08-03T20:59:05.864 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:05.862+0000 7f3027200640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f302006e930 0x7f302015b7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:59:05.864 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:05.862+0000 7f3027200640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f3020073e60 0x7f3020150a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:59:05.864 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:05.862+0000 7f3027200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f30200747b0 0x7f3020144510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:59:05.864 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:05.862+0000 7f3027200640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f30200cf9c0 con 0x7f30200747b0 2024-08-03T20:59:05.865 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:05.862+0000 7f3027200640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f30200cf840 con 0x7f302006e930 2024-08-03T20:59:05.865 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:05.862+0000 7f3027200640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7f30200ccff0 con 0x7f3020073e60 2024-08-03T20:59:05.865 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:05.863+0000 7f3025200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f30200747b0 0x7f3020144510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:59:05.865 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:05.863+0000 7f301fe00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f302006e930 0x7f302015b7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:59:05.865 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:05.863+0000 7f301fe00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f302006e930 0x7f302015b7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.32:60218/0 (socket says 172.21.15.32:60218) 2024-08-03T20:59:05.865 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:05.863+0000 7f301fe00640 1 -- 172.21.15.32:0/1773529663 learned_addr learned my addr 172.21.15.32:0/1773529663 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:59:05.865 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:05.863+0000 7f3017e00640 1 --2- 172.21.15.32:0/1773529663 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f3020073e60 0x7f3020150a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:59:05.865 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:05.863+0000 7f3025200640 1 -- 172.21.15.32:0/1773529663 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f3020073e60 msgr2=0x7f3020150a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:59:05.865 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:05.863+0000 7f3025200640 1 --2- 172.21.15.32:0/1773529663 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f3020073e60 0x7f3020150a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:59:05.865 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:05.863+0000 7f3025200640 1 -- 172.21.15.32:0/1773529663 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f302006e930 msgr2=0x7f302015b7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:59:05.865 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:05.863+0000 7f3025200640 1 --2- 172.21.15.32:0/1773529663 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f302006e930 0x7f302015b7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:59:05.865 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:05.863+0000 7f3025200640 1 -- 172.21.15.32:0/1773529663 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f30201f8a80 con 0x7f30200747b0 2024-08-03T20:59:05.866 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:05.863+0000 7f301fe00640 1 --2- 172.21.15.32:0/1773529663 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f302006e930 0x7f302015b7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-03T20:59:05.866 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:05.863+0000 7f3025200640 1 --2- 172.21.15.32:0/1773529663 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f30200747b0 0x7f3020144510 secure :-1 s=READY pgs=1881 cs=0 l=1 rev1=1 crypto rx=0x7f301800caa0 tx=0x7f301800cf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:59:05.866 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:05.864+0000 7f301cc00640 1 -- 172.21.15.32:0/1773529663 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3018016020 con 0x7f30200747b0 2024-08-03T20:59:05.866 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:05.864+0000 7f301cc00640 1 -- 172.21.15.32:0/1773529663 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f301800acb0 con 0x7f30200747b0 2024-08-03T20:59:05.866 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:05.864+0000 7f301cc00640 1 -- 172.21.15.32:0/1773529663 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3018012680 con 0x7f30200747b0 2024-08-03T20:59:05.866 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:05.864+0000 7f3027200640 1 -- 172.21.15.32:0/1773529663 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f302015bd00 con 0x7f30200747b0 2024-08-03T20:59:05.866 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:05.864+0000 7f3027200640 1 -- 172.21.15.32:0/1773529663 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f30201f9270 con 0x7f30200747b0 2024-08-03T20:59:05.867 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:05.865+0000 7f3027200640 1 -- 172.21.15.32:0/1773529663 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f30201f8d00 con 0x7f30200747b0 2024-08-03T20:59:05.867 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:05.866+0000 7f301cc00640 1 -- 172.21.15.32:0/1773529663 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f3018019020 con 0x7f30200747b0 2024-08-03T20:59:05.867 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:05.867+0000 7f301cc00640 1 --2- 172.21.15.32:0/1773529663 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f2ffc077010 0x7f2ffc079520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:59:05.868 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:05.867+0000 7f301cc00640 1 -- 172.21.15.32:0/1773529663 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(118..118 src has 1..118) ==== 8692+0+0 (secure 0 0 0) 0x7f3018099a40 con 0x7f30200747b0 2024-08-03T20:59:05.868 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:05.868+0000 7f301fe00640 1 --2- 172.21.15.32:0/1773529663 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f2ffc077010 0x7f2ffc079520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:59:05.869 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:05.868+0000 7f301fe00640 1 --2- 172.21.15.32:0/1773529663 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f2ffc077010 0x7f2ffc079520 secure :-1 s=READY pgs=758 cs=0 l=1 rev1=1 crypto rx=0x7f30100095f0 tx=0x7f3010004730 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:59:05.872 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:05.872+0000 7f301cc00640 1 -- 172.21.15.32:0/1773529663 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7f3018020020 con 0x7f30200747b0 2024-08-03T20:59:06.119 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:06.119+0000 7f3027200640 1 -- 172.21.15.32:0/1773529663 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f30200cd580 con 0x7f30200747b0 2024-08-03T20:59:06.125 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:06.119+0000 7f301cc00640 1 -- 172.21.15.32:0/1773529663 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 278 v278) ==== 95+0+4391 (secure 0 0 0) 0x7f3018061990 con 0x7f30200747b0 2024-08-03T20:59:06.126 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-03T20:59:06.126 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":278,"btime":"2024-08-03T20:59:03:076596+0000","default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6439,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6835/531080160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6834","nonce":531080160},{"type":"v1","addr":"172.21.15.118:6835","nonce":531080160}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":84},{"gid":6454,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6837/2796882319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6836","nonce":2796882319},{"type":"v1","addr":"172.21.15.118:6837","nonce":2796882319}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":86},{"gid":6813,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6834/2686933890","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":2686933890},{"type":"v1","addr":"172.21.15.32:6834","nonce":2686933890}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":85}],"filesystems":[{"mdsmap":{"epoch":278,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-08-03T20:54:26.950518+0000","modified":"2024-08-03T20:59:03.076594+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":118,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":6}]} 2024-08-03T20:59:06.127 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 278 2024-08-03T20:59:06.127 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:06.122+0000 7f3027200640 1 -- 172.21.15.32:0/1773529663 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f2ffc077010 msgr2=0x7f2ffc079520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:59:06.127 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:06.122+0000 7f3027200640 1 --2- 172.21.15.32:0/1773529663 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f2ffc077010 0x7f2ffc079520 secure :-1 s=READY pgs=758 cs=0 l=1 rev1=1 crypto rx=0x7f30100095f0 tx=0x7f3010004730 comp rx=0 tx=0).stop 2024-08-03T20:59:06.127 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:06.122+0000 7f3027200640 1 -- 172.21.15.32:0/1773529663 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f30200747b0 msgr2=0x7f3020144510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:59:06.127 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:06.122+0000 7f3027200640 1 --2- 172.21.15.32:0/1773529663 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f30200747b0 0x7f3020144510 secure :-1 s=READY pgs=1881 cs=0 l=1 rev1=1 crypto rx=0x7f301800caa0 tx=0x7f301800cf60 comp rx=0 tx=0).stop 2024-08-03T20:59:06.127 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:06.122+0000 7f3027200640 1 -- 172.21.15.32:0/1773529663 shutdown_connections 2024-08-03T20:59:06.127 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:06.122+0000 7f3027200640 1 --2- 172.21.15.32:0/1773529663 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f2ffc077010 0x7f2ffc079520 unknown :-1 s=CLOSED pgs=758 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:59:06.127 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:06.122+0000 7f3027200640 1 --2- 172.21.15.32:0/1773529663 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f30200747b0 0x7f3020144510 unknown :-1 s=CLOSED pgs=1881 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:59:06.127 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:06.122+0000 7f3027200640 1 --2- 172.21.15.32:0/1773529663 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f3020073e60 0x7f3020150a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:59:06.128 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:06.122+0000 7f3027200640 1 --2- 172.21.15.32:0/1773529663 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f302006e930 0x7f302015b7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:59:06.128 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:06.122+0000 7f3027200640 1 -- 172.21.15.32:0/1773529663 >> 172.21.15.32:0/1773529663 conn(0x7f30200b1bd0 msgr2=0x7f302006ef70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:59:06.128 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:06.122+0000 7f3027200640 1 -- 172.21.15.32:0/1773529663 shutdown_connections 2024-08-03T20:59:06.128 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:06.122+0000 7f3027200640 1 -- 172.21.15.32:0/1773529663 wait complete. 2024-08-03T20:59:06.149 DEBUG:tasks.ceph_test_case:wait_until_true: success in 5s and 0 retries 2024-08-03T20:59:06.149 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-08-03T20:59:06.377 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:06.376+0000 7f9edbe00640 1 -- 172.21.15.32:0/2306132705 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9ed4073e60 msgr2=0x7f9ed4074270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:59:06.377 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:06.376+0000 7f9edbe00640 1 --2- 172.21.15.32:0/2306132705 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9ed4073e60 0x7f9ed4074270 secure :-1 s=READY pgs=1882 cs=0 l=1 rev1=1 crypto rx=0x7f9ed4071700 tx=0x7f9ec002a560 comp rx=0 tx=0).stop 2024-08-03T20:59:06.377 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:06.376+0000 7f9edbe00640 1 -- 172.21.15.32:0/2306132705 shutdown_connections 2024-08-03T20:59:06.377 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:06.376+0000 7f9edbe00640 1 --2- 172.21.15.32:0/2306132705 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9ed4073e60 0x7f9ed4074270 unknown :-1 s=CLOSED pgs=1882 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:59:06.377 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:06.376+0000 7f9edbe00640 1 --2- 172.21.15.32:0/2306132705 >> v2:172.21.15.118:3300/0 conn(0x7f9ed406e930 0x7f9ed4144f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:59:06.377 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:06.376+0000 7f9edbe00640 1 -- 172.21.15.32:0/2306132705 >> 172.21.15.32:0/2306132705 conn(0x7f9ed40b1bd0 msgr2=0x7f9ed40b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:59:06.377 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:06.376+0000 7f9edbe00640 1 -- 172.21.15.32:0/2306132705 shutdown_connections 2024-08-03T20:59:06.378 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:06.376+0000 7f9edbe00640 1 -- 172.21.15.32:0/2306132705 wait complete. 2024-08-03T20:59:06.379 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:06.378+0000 7f9edbe00640 1 Processor -- start 2024-08-03T20:59:06.379 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:06.379+0000 7f9edbe00640 1 -- start start 2024-08-03T20:59:06.381 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:06.380+0000 7f9edbe00640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f9ed406e930 0x7f9ed4132dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:59:06.381 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:06.380+0000 7f9edbe00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f9ed4073e60 0x7f9ed4133310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:59:06.381 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:06.380+0000 7f9edbe00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9ed40747b0 0x7f9ed4143e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:59:06.381 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:06.380+0000 7f9edbe00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f9ed40cf9c0 con 0x7f9ed40747b0 2024-08-03T20:59:06.381 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:06.380+0000 7f9edbe00640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f9ed40cf840 con 0x7f9ed4073e60 2024-08-03T20:59:06.381 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:06.380+0000 7f9edbe00640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7f9ed40ccff0 con 0x7f9ed406e930 2024-08-03T20:59:06.382 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:06.380+0000 7f9ed9e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9ed40747b0 0x7f9ed4143e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:59:06.382 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:06.381+0000 7f9ed8a00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f9ed4073e60 0x7f9ed4133310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:59:06.382 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:06.381+0000 7f9ed8a00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f9ed4073e60 0x7f9ed4133310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.32:60242/0 (socket says 172.21.15.32:60242) 2024-08-03T20:59:06.382 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:06.381+0000 7f9ed9e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9ed40747b0 0x7f9ed4143e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:49364/0 (socket says 172.21.15.32:49364) 2024-08-03T20:59:06.382 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:06.381+0000 7f9ed8a00640 1 -- 172.21.15.32:0/2736688673 learned_addr learned my addr 172.21.15.32:0/2736688673 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:59:06.382 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:06.381+0000 7f9ed9400640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f9ed406e930 0x7f9ed4132dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:59:06.383 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:06.381+0000 7f9ed8a00640 1 -- 172.21.15.32:0/2736688673 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f9ed406e930 msgr2=0x7f9ed4132dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:59:06.383 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:06.381+0000 7f9ed8a00640 1 --2- 172.21.15.32:0/2736688673 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f9ed406e930 0x7f9ed4132dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:59:06.383 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:06.381+0000 7f9ed8a00640 1 -- 172.21.15.32:0/2736688673 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9ed40747b0 msgr2=0x7f9ed4143e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:59:06.383 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:06.381+0000 7f9ed8a00640 1 --2- 172.21.15.32:0/2736688673 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9ed40747b0 0x7f9ed4143e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:59:06.383 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:06.381+0000 7f9ed8a00640 1 -- 172.21.15.32:0/2736688673 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9ed40c4e40 con 0x7f9ed4073e60 2024-08-03T20:59:06.383 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:06.381+0000 7f9ed9e00640 1 --2- 172.21.15.32:0/2736688673 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9ed40747b0 0x7f9ed4143e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-03T20:59:06.384 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:06.381+0000 7f9ed8a00640 1 --2- 172.21.15.32:0/2736688673 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f9ed4073e60 0x7f9ed4133310 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7f9ec0034040 tx=0x7f9ec0003820 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:59:06.384 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:06.382+0000 7f9ed1600640 1 -- 172.21.15.32:0/2736688673 <== mon.1 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ec003e030 con 0x7f9ed4073e60 2024-08-03T20:59:06.384 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:06.382+0000 7f9ed1600640 1 -- 172.21.15.32:0/2736688673 <== mon.1 v2:172.21.15.118:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f9ec0003d10 con 0x7f9ed4073e60 2024-08-03T20:59:06.384 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:06.382+0000 7f9ed1600640 1 -- 172.21.15.32:0/2736688673 <== mon.1 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ec0035040 con 0x7f9ed4073e60 2024-08-03T20:59:06.384 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:06.382+0000 7f9edbe00640 1 -- 172.21.15.32:0/2736688673 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9ed40c52e0 con 0x7f9ed4073e60 2024-08-03T20:59:06.384 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:06.382+0000 7f9edbe00640 1 -- 172.21.15.32:0/2736688673 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9ed415e8c0 con 0x7f9ed4073e60 2024-08-03T20:59:06.384 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:06.383+0000 7f9edbe00640 1 -- 172.21.15.32:0/2736688673 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9ed406a430 con 0x7f9ed4073e60 2024-08-03T20:59:06.385 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:06.384+0000 7f9ed1600640 1 -- 172.21.15.32:0/2736688673 <== mon.1 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f9ec0041030 con 0x7f9ed4073e60 2024-08-03T20:59:06.386 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:06.385+0000 7f9ed1600640 1 --2- 172.21.15.32:0/2736688673 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f9eb4077010 0x7f9eb4079520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:59:06.386 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:06.386+0000 7f9ed1600640 1 -- 172.21.15.32:0/2736688673 <== mon.1 v2:172.21.15.118:3300/0 5 ==== osd_map(118..118 src has 1..118) ==== 8692+0+0 (secure 0 0 0) 0x7f9ec00b9bb0 con 0x7f9ed4073e60 2024-08-03T20:59:06.386 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:06.386+0000 7f9ed9400640 1 --2- 172.21.15.32:0/2736688673 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f9eb4077010 0x7f9eb4079520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:59:06.387 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:06.387+0000 7f9ed9400640 1 --2- 172.21.15.32:0/2736688673 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f9eb4077010 0x7f9eb4079520 secure :-1 s=READY pgs=759 cs=0 l=1 rev1=1 crypto rx=0x7f9ec8009b30 tx=0x7f9ec8005f90 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:59:06.391 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:06.391+0000 7f9ed1600640 1 -- 172.21.15.32:0/2736688673 <== mon.1 v2:172.21.15.118:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7f9ed406a430 con 0x7f9ed4073e60 2024-08-03T20:59:06.638 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:06.637+0000 7f9edbe00640 1 -- 172.21.15.32:0/2736688673 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f9ed40c5070 con 0x7f9ed4073e60 2024-08-03T20:59:06.639 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:06.638+0000 7f9ed1600640 1 -- 172.21.15.32:0/2736688673 <== mon.1 v2:172.21.15.118:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 278 v278) ==== 95+0+4391 (secure 0 0 0) 0x7f9ec0081b60 con 0x7f9ed4073e60 2024-08-03T20:59:06.640 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-03T20:59:06.640 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":278,"btime":"2024-08-03T20:59:03:076596+0000","default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6439,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6835/531080160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6834","nonce":531080160},{"type":"v1","addr":"172.21.15.118:6835","nonce":531080160}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":84},{"gid":6454,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6837/2796882319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6836","nonce":2796882319},{"type":"v1","addr":"172.21.15.118:6837","nonce":2796882319}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":86},{"gid":6813,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6834/2686933890","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":2686933890},{"type":"v1","addr":"172.21.15.32:6834","nonce":2686933890}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":85}],"filesystems":[{"mdsmap":{"epoch":278,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-08-03T20:54:26.950518+0000","modified":"2024-08-03T20:59:03.076594+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":118,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":6}]} 2024-08-03T20:59:06.640 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 278 2024-08-03T20:59:06.642 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:06.640+0000 7f9edbe00640 1 -- 172.21.15.32:0/2736688673 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f9eb4077010 msgr2=0x7f9eb4079520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:59:06.642 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:06.640+0000 7f9edbe00640 1 --2- 172.21.15.32:0/2736688673 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f9eb4077010 0x7f9eb4079520 secure :-1 s=READY pgs=759 cs=0 l=1 rev1=1 crypto rx=0x7f9ec8009b30 tx=0x7f9ec8005f90 comp rx=0 tx=0).stop 2024-08-03T20:59:06.642 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:06.640+0000 7f9edbe00640 1 -- 172.21.15.32:0/2736688673 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f9ed4073e60 msgr2=0x7f9ed4133310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:59:06.642 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:06.640+0000 7f9edbe00640 1 --2- 172.21.15.32:0/2736688673 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f9ed4073e60 0x7f9ed4133310 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7f9ec0034040 tx=0x7f9ec0003820 comp rx=0 tx=0).stop 2024-08-03T20:59:06.642 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:06.640+0000 7f9edbe00640 1 -- 172.21.15.32:0/2736688673 shutdown_connections 2024-08-03T20:59:06.642 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:06.640+0000 7f9edbe00640 1 --2- 172.21.15.32:0/2736688673 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f9eb4077010 0x7f9eb4079520 unknown :-1 s=CLOSED pgs=759 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:59:06.643 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:06.640+0000 7f9edbe00640 1 --2- 172.21.15.32:0/2736688673 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f9ed4073e60 0x7f9ed4133310 unknown :-1 s=CLOSED pgs=571 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:59:06.643 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:06.640+0000 7f9edbe00640 1 --2- 172.21.15.32:0/2736688673 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9ed40747b0 0x7f9ed4143e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:59:06.643 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:06.640+0000 7f9edbe00640 1 --2- 172.21.15.32:0/2736688673 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f9ed406e930 0x7f9ed4132dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:59:06.643 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:06.640+0000 7f9edbe00640 1 -- 172.21.15.32:0/2736688673 >> 172.21.15.32:0/2736688673 conn(0x7f9ed40b1bd0 msgr2=0x7f9ed40b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:59:06.643 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:06.640+0000 7f9edbe00640 1 -- 172.21.15.32:0/2736688673 shutdown_connections 2024-08-03T20:59:06.643 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:06.640+0000 7f9edbe00640 1 -- 172.21.15.32:0/2736688673 wait complete. 2024-08-03T20:59:06.664 DEBUG:tasks.ceph_test_case:wait_until_true: success in 0s and 0 retries 2024-08-03T20:59:06.664 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-08-03T20:59:06.893 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:06.891+0000 7f9241e00640 1 -- 172.21.15.32:0/3884439703 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9220004d40 msgr2=0x7f9220005190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:59:06.893 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:06.891+0000 7f9241e00640 1 --2- 172.21.15.32:0/3884439703 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9220004d40 0x7f9220005190 secure :-1 s=READY pgs=1884 cs=0 l=1 rev1=1 crypto rx=0x7f9234005fd0 tx=0x7f9220005830 comp rx=0 tx=0).stop 2024-08-03T20:59:06.893 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:06.891+0000 7f9241400640 1 -- 172.21.15.32:0/3884439703 reap_dead start 2024-08-03T20:59:06.893 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:06.891+0000 7f9241e00640 1 -- 172.21.15.32:0/3884439703 shutdown_connections 2024-08-03T20:59:06.894 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:06.891+0000 7f9241e00640 1 -- 172.21.15.32:0/3884439703 >> 172.21.15.32:0/3884439703 conn(0x7f923c0d4090 msgr2=0x7f923c0d44a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:59:06.894 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:06.891+0000 7f9241e00640 1 -- 172.21.15.32:0/3884439703 shutdown_connections 2024-08-03T20:59:06.894 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:06.891+0000 7f9241e00640 1 -- 172.21.15.32:0/3884439703 wait complete. 2024-08-03T20:59:06.895 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:06.894+0000 7f9241e00640 1 Processor -- start 2024-08-03T20:59:06.895 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:06.894+0000 7f9241e00640 1 -- start start 2024-08-03T20:59:06.898 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:06.895+0000 7f9241e00640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f9220004d40 0x7f923c1e29d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:59:06.898 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:06.895+0000 7f9241e00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f923c1dd410 0x7f923c1e7550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:59:06.898 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:06.895+0000 7f9241e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f923c0b7f10 0x7f923c0b8320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:59:06.898 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:06.895+0000 7f9241e00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f923c0cf810 con 0x7f923c0b7f10 2024-08-03T20:59:06.898 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:06.895+0000 7f9241e00640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f923c0cf690 con 0x7f923c1dd410 2024-08-03T20:59:06.898 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:06.895+0000 7f9241e00640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7f923c0cf990 con 0x7f9220004d40 2024-08-03T20:59:06.898 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:06.896+0000 7f9241400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f923c0b7f10 0x7f923c0b8320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:59:06.898 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:06.896+0000 7f9240a00640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f9220004d40 0x7f923c1e29d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:59:06.898 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:06.896+0000 7f9241400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f923c0b7f10 0x7f923c0b8320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:49376/0 (socket says 172.21.15.32:49376) 2024-08-03T20:59:06.898 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:06.896+0000 7f9241400640 1 -- 172.21.15.32:0/2813360114 learned_addr learned my addr 172.21.15.32:0/2813360114 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:59:06.898 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:06.896+0000 7f9233e00640 1 --2- 172.21.15.32:0/2813360114 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f923c1dd410 0x7f923c1e7550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:59:06.898 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:06.896+0000 7f9241400640 1 -- 172.21.15.32:0/2813360114 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f9220004d40 msgr2=0x7f923c1e29d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:59:06.899 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:06.896+0000 7f9241400640 1 --2- 172.21.15.32:0/2813360114 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f9220004d40 0x7f923c1e29d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:59:06.899 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:06.896+0000 7f9241400640 1 -- 172.21.15.32:0/2813360114 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f923c1dd410 msgr2=0x7f923c1e7550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:59:06.899 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:06.896+0000 7f9241400640 1 --2- 172.21.15.32:0/2813360114 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f923c1dd410 0x7f923c1e7550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:59:06.899 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:06.896+0000 7f9241400640 1 -- 172.21.15.32:0/2813360114 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f923c1f94e0 con 0x7f923c0b7f10 2024-08-03T20:59:06.899 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:06.896+0000 7f9241400640 1 --2- 172.21.15.32:0/2813360114 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f923c0b7f10 0x7f923c0b8320 secure :-1 s=READY pgs=1885 cs=0 l=1 rev1=1 crypto rx=0x7f9234002230 tx=0x7f9234033c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:59:06.900 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:06.897+0000 7f9239600640 1 -- 172.21.15.32:0/2813360114 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9234009ab0 con 0x7f923c0b7f10 2024-08-03T20:59:06.901 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:06.897+0000 7f9239600640 1 -- 172.21.15.32:0/2813360114 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f9234042070 con 0x7f923c0b7f10 2024-08-03T20:59:06.901 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:06.897+0000 7f9239600640 1 -- 172.21.15.32:0/2813360114 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f923403c7b0 con 0x7f923c0b7f10 2024-08-03T20:59:06.901 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:06.897+0000 7f9241e00640 1 -- 172.21.15.32:0/2813360114 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f923c0b8860 con 0x7f923c0b7f10 2024-08-03T20:59:06.901 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:06.897+0000 7f9241e00640 1 -- 172.21.15.32:0/2813360114 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f923c1f8960 con 0x7f923c0b7f10 2024-08-03T20:59:06.901 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:06.898+0000 7f9241e00640 1 -- 172.21.15.32:0/2813360114 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f923c1f86d0 con 0x7f923c0b7f10 2024-08-03T20:59:06.901 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:06.899+0000 7f9239600640 1 -- 172.21.15.32:0/2813360114 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f9234047460 con 0x7f923c0b7f10 2024-08-03T20:59:06.903 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:06.900+0000 7f9239600640 1 --2- 172.21.15.32:0/2813360114 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f9218077040 0x7f9218079550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:59:06.903 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:06.900+0000 7f9239600640 1 -- 172.21.15.32:0/2813360114 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(118..118 src has 1..118) ==== 8692+0+0 (secure 0 0 0) 0x7f92340ba9a0 con 0x7f923c0b7f10 2024-08-03T20:59:06.903 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:06.900+0000 7f9240a00640 1 --2- 172.21.15.32:0/2813360114 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f9218077040 0x7f9218079550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:59:06.903 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:06.901+0000 7f9240a00640 1 --2- 172.21.15.32:0/2813360114 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f9218077040 0x7f9218079550 secure :-1 s=READY pgs=760 cs=0 l=1 rev1=1 crypto rx=0x7f922c006fd0 tx=0x7f922c006b10 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:59:06.906 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:06.904+0000 7f9239600640 1 -- 172.21.15.32:0/2813360114 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7f923c0b8860 con 0x7f923c0b7f10 2024-08-03T20:59:07.148 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:07.147+0000 7f9241e00640 1 -- 172.21.15.32:0/2813360114 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f923c0cfe80 con 0x7f923c0b7f10 2024-08-03T20:59:07.150 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:07.148+0000 7f9239600640 1 -- 172.21.15.32:0/2813360114 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 278 v278) ==== 95+0+4391 (secure 0 0 0) 0x7f923408b210 con 0x7f923c0b7f10 2024-08-03T20:59:07.150 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-03T20:59:07.150 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":278,"btime":"2024-08-03T20:59:03:076596+0000","default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6439,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6835/531080160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6834","nonce":531080160},{"type":"v1","addr":"172.21.15.118:6835","nonce":531080160}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":84},{"gid":6454,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6837/2796882319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6836","nonce":2796882319},{"type":"v1","addr":"172.21.15.118:6837","nonce":2796882319}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":86},{"gid":6813,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6834/2686933890","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":2686933890},{"type":"v1","addr":"172.21.15.32:6834","nonce":2686933890}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":85}],"filesystems":[{"mdsmap":{"epoch":278,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-08-03T20:54:26.950518+0000","modified":"2024-08-03T20:59:03.076594+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":118,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":6}]} 2024-08-03T20:59:07.150 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 278 2024-08-03T20:59:07.153 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:07.151+0000 7f9241e00640 1 -- 172.21.15.32:0/2813360114 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f9218077040 msgr2=0x7f9218079550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:59:07.153 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:07.151+0000 7f9241e00640 1 --2- 172.21.15.32:0/2813360114 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f9218077040 0x7f9218079550 secure :-1 s=READY pgs=760 cs=0 l=1 rev1=1 crypto rx=0x7f922c006fd0 tx=0x7f922c006b10 comp rx=0 tx=0).stop 2024-08-03T20:59:07.153 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:07.151+0000 7f9241e00640 1 -- 172.21.15.32:0/2813360114 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f923c0b7f10 msgr2=0x7f923c0b8320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:59:07.153 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:07.151+0000 7f9241e00640 1 --2- 172.21.15.32:0/2813360114 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f923c0b7f10 0x7f923c0b8320 secure :-1 s=READY pgs=1885 cs=0 l=1 rev1=1 crypto rx=0x7f9234002230 tx=0x7f9234033c20 comp rx=0 tx=0).stop 2024-08-03T20:59:07.153 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:07.151+0000 7f9241e00640 1 -- 172.21.15.32:0/2813360114 shutdown_connections 2024-08-03T20:59:07.153 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:07.151+0000 7f9241e00640 1 --2- 172.21.15.32:0/2813360114 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f9218077040 0x7f9218079550 unknown :-1 s=CLOSED pgs=760 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:59:07.153 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:07.151+0000 7f9241e00640 1 --2- 172.21.15.32:0/2813360114 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f923c1dd410 0x7f923c1e7550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:59:07.153 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:07.151+0000 7f9241e00640 1 --2- 172.21.15.32:0/2813360114 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f923c0b7f10 0x7f923c0b8320 unknown :-1 s=CLOSED pgs=1885 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:59:07.153 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:07.151+0000 7f9241e00640 1 --2- 172.21.15.32:0/2813360114 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f9220004d40 0x7f923c1e29d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:59:07.153 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:07.151+0000 7f9241e00640 1 -- 172.21.15.32:0/2813360114 >> 172.21.15.32:0/2813360114 conn(0x7f923c0d4090 msgr2=0x7f923c0d2070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:59:07.153 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:07.151+0000 7f9241e00640 1 -- 172.21.15.32:0/2813360114 shutdown_connections 2024-08-03T20:59:07.153 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:07.151+0000 7f9241e00640 1 -- 172.21.15.32:0/2813360114 wait complete. 2024-08-03T20:59:07.178 INFO:tasks.cephfs.test_damage:Result: Mutation 'Corrupt omap key 100.00000000:stray0_head' led to DAMAGED state 2024-08-03T20:59:07.178 INFO:tasks.cephfs.test_damage:Applying mutation 'Corrupt omap key 1.00000000:subdir_head' 2024-08-03T20:59:07.178 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-08-03T20:59:07.178 DEBUG:teuthology.orchestra.run.smithi032:> dd if=/proc/self/mounts of=/dev/stdout 2024-08-03T20:59:07.214 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053453,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6499208k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 none /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249604k,nr_inodes=812401,mode=700,uid=1000,gid=1316 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6499208k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-08-03T20:59:07.214 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi032.front.sepia.ceph.com 2024-08-03T20:59:07.214 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-08-03T20:59:07.214 DEBUG:teuthology.orchestra.run.smithi032:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-08-03T20:59:07.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-03T20:59:07.290 INFO:teuthology.orchestra.run.smithi032.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.0': No such file or directory 2024-08-03T20:59:07.291 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-08-03T20:59:07.564 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:07.562+0000 7fcf16c00640 1 -- 172.21.15.32:0/1688255659 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcef4005270 msgr2=0x7fcef400b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:59:07.564 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:07.562+0000 7fcf16c00640 1 --2- 172.21.15.32:0/1688255659 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcef4005270 0x7fcef400b9c0 secure :-1 s=READY pgs=1887 cs=0 l=1 rev1=1 crypto rx=0x7fcf08001dc0 tx=0x7fcef4005830 comp rx=0 tx=0).stop 2024-08-03T20:59:07.564 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:07.562+0000 7fcf16c00640 1 -- 172.21.15.32:0/1688255659 shutdown_connections 2024-08-03T20:59:07.564 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:07.562+0000 7fcf16c00640 1 --2- 172.21.15.32:0/1688255659 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcef4005270 0x7fcef400b9c0 unknown :-1 s=CLOSED pgs=1887 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:59:07.565 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:07.562+0000 7fcf16c00640 1 -- 172.21.15.32:0/1688255659 >> 172.21.15.32:0/1688255659 conn(0x7fcf100b1bd0 msgr2=0x7fcf100b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:59:07.565 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:07.563+0000 7fcf16c00640 1 -- 172.21.15.32:0/1688255659 shutdown_connections 2024-08-03T20:59:07.565 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:07.563+0000 7fcf16c00640 1 -- 172.21.15.32:0/1688255659 wait complete. 2024-08-03T20:59:07.566 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:07.565+0000 7fcf16c00640 1 Processor -- start 2024-08-03T20:59:07.567 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:07.565+0000 7fcf16c00640 1 -- start start 2024-08-03T20:59:07.569 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:07.566+0000 7fcf16c00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fcef4005270 0x7fcf1013b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:59:07.569 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:07.566+0000 7fcf16c00640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fcf1015fb30 0x7fcf10156530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:59:07.569 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:07.566+0000 7fcf16c00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcf1013d170 0x7fcf1013d580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:59:07.569 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:07.566+0000 7fcf16c00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fcf100cf840 con 0x7fcf1013d170 2024-08-03T20:59:07.569 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:07.566+0000 7fcf16c00640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7fcf10145f30 con 0x7fcef4005270 2024-08-03T20:59:07.569 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:07.566+0000 7fcf16c00640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7fcf100cf9c0 con 0x7fcf1015fb30 2024-08-03T20:59:07.569 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:07.567+0000 7fcf14c00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcf1013d170 0x7fcf1013d580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:59:07.569 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:07.567+0000 7fcf14c00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcf1013d170 0x7fcf1013d580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:49394/0 (socket says 172.21.15.32:49394) 2024-08-03T20:59:07.569 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:07.567+0000 7fcf14c00640 1 -- 172.21.15.32:0/631978196 learned_addr learned my addr 172.21.15.32:0/631978196 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:59:07.569 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:07.567+0000 7fcf07400640 1 --2- 172.21.15.32:0/631978196 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fcf1015fb30 0x7fcf10156530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:59:07.570 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:07.567+0000 7fcf0fe00640 1 --2- 172.21.15.32:0/631978196 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fcef4005270 0x7fcf1013b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:59:07.570 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:07.567+0000 7fcf14c00640 1 -- 172.21.15.32:0/631978196 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fcf1015fb30 msgr2=0x7fcf10156530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:59:07.570 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:07.567+0000 7fcf14c00640 1 --2- 172.21.15.32:0/631978196 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fcf1015fb30 0x7fcf10156530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:59:07.570 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:07.567+0000 7fcf14c00640 1 -- 172.21.15.32:0/631978196 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fcef4005270 msgr2=0x7fcf1013b750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:59:07.570 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:07.567+0000 7fcf14c00640 1 --2- 172.21.15.32:0/631978196 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fcef4005270 0x7fcf1013b750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:59:07.570 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:07.567+0000 7fcf14c00640 1 -- 172.21.15.32:0/631978196 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcf1013dac0 con 0x7fcf1013d170 2024-08-03T20:59:07.570 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:07.567+0000 7fcf14c00640 1 --2- 172.21.15.32:0/631978196 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcf1013d170 0x7fcf1013d580 secure :-1 s=READY pgs=1888 cs=0 l=1 rev1=1 crypto rx=0x7fcf08001eb0 tx=0x7fcf08006d20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:59:07.571 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:07.568+0000 7fcf0cc00640 1 -- 172.21.15.32:0/631978196 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf08003d80 con 0x7fcf1013d170 2024-08-03T20:59:07.571 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:07.568+0000 7fcf0cc00640 1 -- 172.21.15.32:0/631978196 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fcf08005a90 con 0x7fcf1013d170 2024-08-03T20:59:07.571 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:07.568+0000 7fcf0cc00640 1 -- 172.21.15.32:0/631978196 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf08005ea0 con 0x7fcf1013d170 2024-08-03T20:59:07.571 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:07.568+0000 7fcf16c00640 1 -- 172.21.15.32:0/631978196 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcf1013c050 con 0x7fcf1013d170 2024-08-03T20:59:07.571 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:07.568+0000 7fcf16c00640 1 -- 172.21.15.32:0/631978196 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcf1013c610 con 0x7fcf1013d170 2024-08-03T20:59:07.571 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:07.569+0000 7fcf16c00640 1 -- 172.21.15.32:0/631978196 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcf1013bda0 con 0x7fcf1013d170 2024-08-03T20:59:07.572 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:07.570+0000 7fcf0cc00640 1 -- 172.21.15.32:0/631978196 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7fcf08006040 con 0x7fcf1013d170 2024-08-03T20:59:07.573 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:07.571+0000 7fcf0cc00640 1 --2- 172.21.15.32:0/631978196 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fceec077010 0x7fceec079520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:59:07.573 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:07.571+0000 7fcf0cc00640 1 -- 172.21.15.32:0/631978196 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(118..118 src has 1..118) ==== 8692+0+0 (secure 0 0 0) 0x7fcf080b8d00 con 0x7fcf1013d170 2024-08-03T20:59:07.573 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:07.572+0000 7fcf0fe00640 1 --2- 172.21.15.32:0/631978196 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fceec077010 0x7fceec079520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:59:07.574 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:07.573+0000 7fcf0fe00640 1 --2- 172.21.15.32:0/631978196 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fceec077010 0x7fceec079520 secure :-1 s=READY pgs=761 cs=0 l=1 rev1=1 crypto rx=0x7fcf00001d80 tx=0x7fcf000067a0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:59:07.577 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:07.576+0000 7fcf0cc00640 1 -- 172.21.15.32:0/631978196 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7fcf1013dac0 con 0x7fcf1013d170 2024-08-03T20:59:07.822 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:07.820+0000 7fcf16c00640 1 -- 172.21.15.32:0/631978196 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7fcf100740e0 con 0x7fcf1013d170 2024-08-03T20:59:07.828 INFO:tasks.ceph.mds.a.smithi032.stderr: -53> 2024-08-03T20:59:03.073+0000 7fb139a00640 -1 mds.0.damage notify_dentry Damage to dentries in fragment * of ino 0x100is fatal because it is a system directory for this rank 2024-08-03T20:59:07.828 INFO:tasks.ceph.mds.a.smithi032.stderr: -36> 2024-08-03T20:59:03.117+0000 7fb13ea00640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-08-03T20:59:07.939 INFO:tasks.ceph.mds.a.smithi032.stdout:starting mds.a at 2024-08-03T20:59:08.158 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:08.157+0000 7fcf0cc00640 1 -- 172.21.15.32:0/631978196 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v279) ==== 161+0+0 (secure 0 0 0) 0x7fcf08080d10 con 0x7fcf1013d170 2024-08-03T20:59:08.158 INFO:teuthology.orchestra.run.smithi032.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-08-03T20:59:08.162 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:08.160+0000 7fcf16c00640 1 -- 172.21.15.32:0/631978196 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fceec077010 msgr2=0x7fceec079520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:59:08.163 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:08.160+0000 7fcf16c00640 1 --2- 172.21.15.32:0/631978196 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fceec077010 0x7fceec079520 secure :-1 s=READY pgs=761 cs=0 l=1 rev1=1 crypto rx=0x7fcf00001d80 tx=0x7fcf000067a0 comp rx=0 tx=0).stop 2024-08-03T20:59:08.163 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:08.160+0000 7fcf16c00640 1 -- 172.21.15.32:0/631978196 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcf1013d170 msgr2=0x7fcf1013d580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:59:08.163 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:08.160+0000 7fcf16c00640 1 --2- 172.21.15.32:0/631978196 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcf1013d170 0x7fcf1013d580 secure :-1 s=READY pgs=1888 cs=0 l=1 rev1=1 crypto rx=0x7fcf08001eb0 tx=0x7fcf08006d20 comp rx=0 tx=0).stop 2024-08-03T20:59:08.163 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:08.160+0000 7fcf16c00640 1 -- 172.21.15.32:0/631978196 shutdown_connections 2024-08-03T20:59:08.163 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:08.160+0000 7fcf16c00640 1 --2- 172.21.15.32:0/631978196 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fceec077010 0x7fceec079520 unknown :-1 s=CLOSED pgs=761 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:59:08.163 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:08.160+0000 7fcf16c00640 1 --2- 172.21.15.32:0/631978196 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcf1013d170 0x7fcf1013d580 unknown :-1 s=CLOSED pgs=1888 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:59:08.164 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:08.160+0000 7fcf16c00640 1 --2- 172.21.15.32:0/631978196 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fcf1015fb30 0x7fcf10156530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:59:08.164 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:08.160+0000 7fcf16c00640 1 --2- 172.21.15.32:0/631978196 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fcef4005270 0x7fcf1013b750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:59:08.164 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:08.160+0000 7fcf16c00640 1 -- 172.21.15.32:0/631978196 >> 172.21.15.32:0/631978196 conn(0x7fcf100b1bd0 msgr2=0x7fcf10143a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:59:08.164 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:08.160+0000 7fcf16c00640 1 -- 172.21.15.32:0/631978196 shutdown_connections 2024-08-03T20:59:08.164 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:08.160+0000 7fcf16c00640 1 -- 172.21.15.32:0/631978196 wait complete. 2024-08-03T20:59:08.187 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mds repaired 0 2024-08-03T20:59:08.416 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:08.414+0000 7f097a600640 1 -- 172.21.15.32:0/1659634725 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f097406e930 msgr2=0x7f0974144f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:59:08.417 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:08.414+0000 7f097a600640 1 --2- 172.21.15.32:0/1659634725 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f097406e930 0x7f0974144f90 secure :-1 s=READY pgs=1892 cs=0 l=1 rev1=1 crypto rx=0x7f0964009830 tx=0x7f096402a5b0 comp rx=0 tx=0).stop 2024-08-03T20:59:08.417 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:08.415+0000 7f097a600640 1 -- 172.21.15.32:0/1659634725 shutdown_connections 2024-08-03T20:59:08.417 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:08.415+0000 7f097a600640 1 --2- 172.21.15.32:0/1659634725 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f097406e930 0x7f0974144f90 unknown :-1 s=CLOSED pgs=1892 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:59:08.417 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:08.415+0000 7f097a600640 1 -- 172.21.15.32:0/1659634725 >> 172.21.15.32:0/1659634725 conn(0x7f09740b1bd0 msgr2=0x7f09740b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:59:08.417 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:08.415+0000 7f097a600640 1 -- 172.21.15.32:0/1659634725 shutdown_connections 2024-08-03T20:59:08.417 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:08.415+0000 7f097a600640 1 -- 172.21.15.32:0/1659634725 wait complete. 2024-08-03T20:59:08.418 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:08.417+0000 7f097a600640 1 Processor -- start 2024-08-03T20:59:08.418 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:08.417+0000 7f097a600640 1 -- start start 2024-08-03T20:59:08.420 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:08.418+0000 7f097a600640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f097406e930 0x7f0974141eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:59:08.420 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:08.418+0000 7f097a600640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f0974073e60 0x7f097413a040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:59:08.420 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:08.418+0000 7f097a600640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f097415fff0 0x7f09741f8800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:59:08.420 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:08.418+0000 7f097a600640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f09740cf840 con 0x7f097406e930 2024-08-03T20:59:08.420 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:08.418+0000 7f097a600640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f0974145f30 con 0x7f097415fff0 2024-08-03T20:59:08.420 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:08.418+0000 7f097a600640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7f09740cf9c0 con 0x7f0974073e60 2024-08-03T20:59:08.424 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:08.419+0000 7f0973400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f097406e930 0x7f0974141eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:59:08.424 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:08.419+0000 7f0973400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f097406e930 0x7f0974141eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:49428/0 (socket says 172.21.15.32:49428) 2024-08-03T20:59:08.424 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:08.419+0000 7f0973400640 1 -- 172.21.15.32:0/1237076075 learned_addr learned my addr 172.21.15.32:0/1237076075 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:59:08.424 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:08.419+0000 7f0973400640 1 -- 172.21.15.32:0/1237076075 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f0974073e60 msgr2=0x7f097413a040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:59:08.424 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:08.419+0000 7f0973e00640 1 --2- 172.21.15.32:0/1237076075 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f097415fff0 0x7f09741f8800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:59:08.424 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:08.419+0000 7f0972a00640 1 --2- 172.21.15.32:0/1237076075 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f0974073e60 0x7f097413a040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:59:08.424 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:08.419+0000 7f0973400640 1 --2- 172.21.15.32:0/1237076075 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f0974073e60 0x7f097413a040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:59:08.424 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:08.419+0000 7f0973400640 1 -- 172.21.15.32:0/1237076075 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f097415fff0 msgr2=0x7f09741f8800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:59:08.425 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:08.419+0000 7f0973400640 1 --2- 172.21.15.32:0/1237076075 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f097415fff0 0x7f09741f8800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:59:08.425 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:08.419+0000 7f0973400640 1 -- 172.21.15.32:0/1237076075 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f09741f90c0 con 0x7f097406e930 2024-08-03T20:59:08.425 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:08.419+0000 7f0973e00640 1 --2- 172.21.15.32:0/1237076075 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f097415fff0 0x7f09741f8800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-03T20:59:08.425 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:08.419+0000 7f0972a00640 1 --2- 172.21.15.32:0/1237076075 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f0974073e60 0x7f097413a040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-03T20:59:08.425 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:08.419+0000 7f0973400640 1 --2- 172.21.15.32:0/1237076075 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f097406e930 0x7f0974141eb0 secure :-1 s=READY pgs=1893 cs=0 l=1 rev1=1 crypto rx=0x7f0964000c00 tx=0x7f0964033c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:59:08.425 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:08.420+0000 7f096b400640 1 -- 172.21.15.32:0/1237076075 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0964031d60 con 0x7f097406e930 2024-08-03T20:59:08.425 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:08.420+0000 7f096b400640 1 -- 172.21.15.32:0/1237076075 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f0964033e10 con 0x7f097406e930 2024-08-03T20:59:08.425 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:08.420+0000 7f096b400640 1 -- 172.21.15.32:0/1237076075 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0964033310 con 0x7f097406e930 2024-08-03T20:59:08.425 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:08.420+0000 7f097a600640 1 -- 172.21.15.32:0/1237076075 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f09741f93a0 con 0x7f097406e930 2024-08-03T20:59:08.425 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:08.420+0000 7f097a600640 1 -- 172.21.15.32:0/1237076075 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f097415ec40 con 0x7f097406e930 2024-08-03T20:59:08.425 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:08.421+0000 7f097a600640 1 -- 172.21.15.32:0/1237076075 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0974160db0 con 0x7f097406e930 2024-08-03T20:59:08.425 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:08.422+0000 7f096b400640 1 -- 172.21.15.32:0/1237076075 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f09640334b0 con 0x7f097406e930 2024-08-03T20:59:08.426 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:08.423+0000 7f096b400640 1 --2- 172.21.15.32:0/1237076075 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f0954077010 0x7f0954079520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:59:08.426 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:08.423+0000 7f096b400640 1 -- 172.21.15.32:0/1237076075 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(118..118 src has 1..118) ==== 8692+0+0 (secure 0 0 0) 0x7f09640ba910 con 0x7f097406e930 2024-08-03T20:59:08.426 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:08.424+0000 7f0972a00640 1 --2- 172.21.15.32:0/1237076075 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f0954077010 0x7f0954079520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:59:08.426 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:08.424+0000 7f0972a00640 1 --2- 172.21.15.32:0/1237076075 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f0954077010 0x7f0954079520 secure :-1 s=READY pgs=763 cs=0 l=1 rev1=1 crypto rx=0x7f097415c3d0 tx=0x7f0960009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:59:08.429 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:08.428+0000 7f096b400640 1 -- 172.21.15.32:0/1237076075 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7f096403f030 con 0x7f097406e930 2024-08-03T20:59:08.710 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:08.708+0000 7f097a600640 1 -- 172.21.15.32:0/1237076075 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "mds repaired", "role": "0"} v 0) -- 0x7f09741457e0 con 0x7f097406e930 2024-08-03T20:59:08.828 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-08-03T20:59:08.834 DEBUG:teuthology.orchestra.run.smithi118:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-08-03T20:59:09.164 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.162+0000 7f096b400640 1 -- 172.21.15.32:0/1237076075 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "mds repaired", "role": "0"}]=0 repaired: restoring rank 6:0 v280) ==== 101+0+0 (secure 0 0 0) 0x7f0964082eb0 con 0x7f097406e930 2024-08-03T20:59:09.164 INFO:teuthology.orchestra.run.smithi032.stderr:repaired: restoring rank 6:0 2024-08-03T20:59:09.167 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.165+0000 7f097a600640 1 -- 172.21.15.32:0/1237076075 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f0954077010 msgr2=0x7f0954079520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:59:09.167 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.165+0000 7f097a600640 1 --2- 172.21.15.32:0/1237076075 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f0954077010 0x7f0954079520 secure :-1 s=READY pgs=763 cs=0 l=1 rev1=1 crypto rx=0x7f097415c3d0 tx=0x7f0960009040 comp rx=0 tx=0).stop 2024-08-03T20:59:09.167 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.165+0000 7f097a600640 1 -- 172.21.15.32:0/1237076075 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f097406e930 msgr2=0x7f0974141eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:59:09.167 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.165+0000 7f097a600640 1 --2- 172.21.15.32:0/1237076075 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f097406e930 0x7f0974141eb0 secure :-1 s=READY pgs=1893 cs=0 l=1 rev1=1 crypto rx=0x7f0964000c00 tx=0x7f0964033c20 comp rx=0 tx=0).stop 2024-08-03T20:59:09.167 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.165+0000 7f097a600640 1 -- 172.21.15.32:0/1237076075 shutdown_connections 2024-08-03T20:59:09.168 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.165+0000 7f097a600640 1 --2- 172.21.15.32:0/1237076075 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f0954077010 0x7f0954079520 unknown :-1 s=CLOSED pgs=763 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:59:09.168 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.165+0000 7f097a600640 1 --2- 172.21.15.32:0/1237076075 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f097415fff0 0x7f09741f8800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:59:09.168 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.165+0000 7f097a600640 1 --2- 172.21.15.32:0/1237076075 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f0974073e60 0x7f097413a040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:59:09.168 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.165+0000 7f097a600640 1 --2- 172.21.15.32:0/1237076075 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f097406e930 0x7f0974141eb0 unknown :-1 s=CLOSED pgs=1893 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:59:09.168 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.165+0000 7f097a600640 1 -- 172.21.15.32:0/1237076075 >> 172.21.15.32:0/1237076075 conn(0x7f09740b1bd0 msgr2=0x7f0974160400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:59:09.168 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.165+0000 7f097a600640 1 -- 172.21.15.32:0/1237076075 shutdown_connections 2024-08-03T20:59:09.168 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.165+0000 7f097a600640 1 -- 172.21.15.32:0/1237076075 wait complete. 2024-08-03T20:59:09.195 DEBUG:teuthology.orchestra.run.smithi032:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata import - 2024-08-03T20:59:09.338 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.336+0000 7f7df7828f40 1 Processor -- start 2024-08-03T20:59:09.339 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.336+0000 7f7df7828f40 1 -- start start 2024-08-03T20:59:09.343 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.338+0000 7f7df7828f40 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b2d4511360 0x55b2d44a10b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:59:09.343 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.338+0000 7f7df7828f40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55b2d452ecb0 0x55b2d44a3780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:59:09.343 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.338+0000 7f7df7828f40 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55b2d452f3f0 0x55b2d452a050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:59:09.344 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.338+0000 7f7df7828f40 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55b2d4406bf0 con 0x55b2d4511360 2024-08-03T20:59:09.344 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.338+0000 7f7df7828f40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x55b2d44136b0 con 0x55b2d452ecb0 2024-08-03T20:59:09.344 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.338+0000 7f7df7828f40 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x55b2d43a8e30 con 0x55b2d452f3f0 2024-08-03T20:59:09.344 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.338+0000 7f7df0a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b2d4511360 0x55b2d44a10b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:59:09.344 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.338+0000 7f7df0a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b2d4511360 0x55b2d44a10b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:49436/0 (socket says 172.21.15.32:49436) 2024-08-03T20:59:09.345 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.338+0000 7f7df0a00640 1 -- 172.21.15.32:0/1876270713 learned_addr learned my addr 172.21.15.32:0/1876270713 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:59:09.345 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.338+0000 7f7debe00640 1 --2- 172.21.15.32:0/1876270713 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55b2d452ecb0 0x55b2d44a3780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:59:09.345 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.338+0000 7f7df0a00640 1 -- 172.21.15.32:0/1876270713 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55b2d452f3f0 msgr2=0x55b2d452a050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:59:09.345 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.338+0000 7f7df1400640 1 --2- 172.21.15.32:0/1876270713 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55b2d452f3f0 0x55b2d452a050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:59:09.345 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.339+0000 7f7df0a00640 1 --2- 172.21.15.32:0/1876270713 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55b2d452f3f0 0x55b2d452a050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:59:09.345 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.339+0000 7f7df0a00640 1 -- 172.21.15.32:0/1876270713 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55b2d452ecb0 msgr2=0x55b2d44a3780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:59:09.346 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.339+0000 7f7df0a00640 1 --2- 172.21.15.32:0/1876270713 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55b2d452ecb0 0x55b2d44a3780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:59:09.346 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.339+0000 7f7df0a00640 1 -- 172.21.15.32:0/1876270713 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b2d450b5d0 con 0x55b2d4511360 2024-08-03T20:59:09.346 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.339+0000 7f7debe00640 1 --2- 172.21.15.32:0/1876270713 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55b2d452ecb0 0x55b2d44a3780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-03T20:59:09.346 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.339+0000 7f7df1400640 1 --2- 172.21.15.32:0/1876270713 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55b2d452f3f0 0x55b2d452a050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-03T20:59:09.346 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.339+0000 7f7df0a00640 1 --2- 172.21.15.32:0/1876270713 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b2d4511360 0x55b2d44a10b0 secure :-1 s=READY pgs=1896 cs=0 l=1 rev1=1 crypto rx=0x7f7de4046660 tx=0x7f7de4002fb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8f3d6bb155df5448 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:59:09.347 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.339+0000 7f7deaa00640 1 -- 172.21.15.32:0/1876270713 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7de4066070 con 0x55b2d4511360 2024-08-03T20:59:09.347 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.339+0000 7f7deaa00640 1 -- 172.21.15.32:0/1876270713 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f7de4056760 con 0x55b2d4511360 2024-08-03T20:59:09.347 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.339+0000 7f7deaa00640 1 -- 172.21.15.32:0/1876270713 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7de404d9c0 con 0x55b2d4511360 2024-08-03T20:59:09.347 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.340+0000 7f7df7828f40 1 -- 172.21.15.32:0/1876270713 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b2d4511360 msgr2=0x55b2d44a10b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:59:09.347 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.340+0000 7f7df7828f40 1 --2- 172.21.15.32:0/1876270713 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b2d4511360 0x55b2d44a10b0 secure :-1 s=READY pgs=1896 cs=0 l=1 rev1=1 crypto rx=0x7f7de4046660 tx=0x7f7de4002fb0 comp rx=0 tx=0).stop 2024-08-03T20:59:09.347 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.340+0000 7f7df7828f40 1 -- 172.21.15.32:0/1876270713 shutdown_connections 2024-08-03T20:59:09.348 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.340+0000 7f7df7828f40 1 --2- 172.21.15.32:0/1876270713 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55b2d452ecb0 0x55b2d44a3780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:59:09.348 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.340+0000 7f7df7828f40 1 --2- 172.21.15.32:0/1876270713 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55b2d452f3f0 0x55b2d452a050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:59:09.348 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.340+0000 7f7df7828f40 1 --2- 172.21.15.32:0/1876270713 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b2d4511360 0x55b2d44a10b0 unknown :-1 s=CLOSED pgs=1896 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:59:09.348 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.340+0000 7f7df7828f40 1 -- 172.21.15.32:0/1876270713 >> 172.21.15.32:0/1876270713 conn(0x55b2d437aac0 msgr2=0x55b2d452cd60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:59:09.348 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.340+0000 7f7df7828f40 1 -- 172.21.15.32:0/1876270713 shutdown_connections 2024-08-03T20:59:09.349 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.340+0000 7f7df7828f40 1 -- 172.21.15.32:0/1876270713 wait complete. 2024-08-03T20:59:09.349 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.341+0000 7f7df7828f40 1 Processor -- start 2024-08-03T20:59:09.349 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.341+0000 7f7df7828f40 1 -- start start 2024-08-03T20:59:09.353 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.343+0000 7f7df7828f40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55b2d4511360 0x55b2d45070a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:59:09.353 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.343+0000 7f7df7828f40 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55b2d452ecb0 0x55b2d45c7b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:59:09.353 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.343+0000 7f7df7828f40 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b2d452f3f0 0x55b2d452cd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:59:09.353 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.343+0000 7f7df7828f40 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55b2d44a3220 con 0x55b2d452f3f0 2024-08-03T20:59:09.353 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.343+0000 7f7df7828f40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x55b2d452b700 con 0x55b2d4511360 2024-08-03T20:59:09.354 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.343+0000 7f7df7828f40 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x55b2d4499d40 con 0x55b2d452ecb0 2024-08-03T20:59:09.354 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.343+0000 7f7df1400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b2d452f3f0 0x55b2d452cd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:59:09.354 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.343+0000 7f7df1400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b2d452f3f0 0x55b2d452cd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:49452/0 (socket says 172.21.15.32:49452) 2024-08-03T20:59:09.354 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.343+0000 7f7df1400640 1 -- 172.21.15.32:0/617165089 learned_addr learned my addr 172.21.15.32:0/617165089 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:59:09.354 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.343+0000 7f7df0a00640 1 --2- 172.21.15.32:0/617165089 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55b2d4511360 0x55b2d45070a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:59:09.354 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.343+0000 7f7debe00640 1 --2- 172.21.15.32:0/617165089 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55b2d452ecb0 0x55b2d45c7b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:59:09.354 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.344+0000 7f7df1400640 1 -- 172.21.15.32:0/617165089 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55b2d452ecb0 msgr2=0x55b2d45c7b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:59:09.355 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.344+0000 7f7df1400640 1 --2- 172.21.15.32:0/617165089 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55b2d452ecb0 0x55b2d45c7b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:59:09.355 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.344+0000 7f7df1400640 1 -- 172.21.15.32:0/617165089 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55b2d4511360 msgr2=0x55b2d45070a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:59:09.355 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.344+0000 7f7df1400640 1 --2- 172.21.15.32:0/617165089 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55b2d4511360 0x55b2d45070a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:59:09.355 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.344+0000 7f7df1400640 1 -- 172.21.15.32:0/617165089 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b2d45c7070 con 0x55b2d452f3f0 2024-08-03T20:59:09.355 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.344+0000 7f7df0a00640 1 --2- 172.21.15.32:0/617165089 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55b2d4511360 0x55b2d45070a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-03T20:59:09.355 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.344+0000 7f7df1400640 1 --2- 172.21.15.32:0/617165089 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b2d452f3f0 0x55b2d452cd60 secure :-1 s=READY pgs=1897 cs=0 l=1 rev1=1 crypto rx=0x7f7dec0041a0 tx=0x7f7dec00c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:59:09.355 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.344+0000 7f7de9600640 1 -- 172.21.15.32:0/617165089 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7dec017070 con 0x55b2d452f3f0 2024-08-03T20:59:09.356 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.344+0000 7f7de9600640 1 -- 172.21.15.32:0/617165089 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f7dec012470 con 0x55b2d452f3f0 2024-08-03T20:59:09.356 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.344+0000 7f7df7828f40 1 -- 172.21.15.32:0/617165089 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x55b2d44a3d30 con 0x55b2d452f3f0 2024-08-03T20:59:09.356 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.344+0000 7f7de9600640 1 -- 172.21.15.32:0/617165089 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7dec012860 con 0x55b2d452f3f0 2024-08-03T20:59:09.356 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.344+0000 7f7df7828f40 1 -- 172.21.15.32:0/617165089 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b2d45c7330 con 0x55b2d452f3f0 2024-08-03T20:59:09.356 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.346+0000 7f7de9600640 1 -- 172.21.15.32:0/617165089 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f7dec01a070 con 0x55b2d452f3f0 2024-08-03T20:59:09.356 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.347+0000 7f7de9600640 1 --2- 172.21.15.32:0/617165089 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f7dd0075a20 0x7f7dd0077f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:59:09.357 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.347+0000 7f7de9600640 1 -- 172.21.15.32:0/617165089 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(118..118 src has 1..118) ==== 8692+0+0 (secure 0 0 0) 0x7f7dec0978c0 con 0x55b2d452f3f0 2024-08-03T20:59:09.357 INFO:teuthology.orchestra.run.smithi032.stdout:Importing pool 2024-08-03T20:59:09.360 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.348+0000 7f7df7828f40 1 --2- 172.21.15.32:0/617165089 >> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] conn(0x55b2d4507770 0x55b2d4499020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:59:09.360 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.348+0000 7f7df7828f40 1 -- 172.21.15.32:0/617165089 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:1 12.10 12:09401a81:::601.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e118) -- 0x55b2d4499560 con 0x55b2d4507770 2024-08-03T20:59:09.360 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.348+0000 7f7df0a00640 1 --2- 172.21.15.32:0/617165089 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f7dd0075a20 0x7f7dd0077f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:59:09.360 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.348+0000 7f7debe00640 1 --2- 172.21.15.32:0/617165089 >> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] conn(0x55b2d4507770 0x55b2d4499020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:59:09.360 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.349+0000 7f7debe00640 1 --2- 172.21.15.32:0/617165089 >> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] conn(0x55b2d4507770 0x55b2d4499020 crc :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:59:09.360 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.349+0000 7f7df0a00640 1 --2- 172.21.15.32:0/617165089 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f7dd0075a20 0x7f7dd0077f30 secure :-1 s=READY pgs=764 cs=0 l=1 rev1=1 crypto rx=0x7f7de4046660 tx=0x7f7de4058040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:59:09.360 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.351+0000 7f7debe00640 1 -- 172.21.15.32:0/617165089 <== osd.0 v2:172.21.15.32:6816/3066156116 1 ==== osd_op_reply(1 601.00000000 [create] v118'1002 uv976 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f7ddc00e4b0 con 0x55b2d4507770 2024-08-03T20:59:09.360 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.352+0000 7f7df7828f40 1 -- 172.21.15.32:0/617165089 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:2 12.10 12:09401a81:::601.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e118) -- 0x55b2d4499930 con 0x55b2d4507770 2024-08-03T20:59:09.360 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.353+0000 7f7debe00640 1 -- 172.21.15.32:0/617165089 <== osd.0 v2:172.21.15.32:6816/3066156116 2 ==== osd_op_reply(2 601.00000000 [delete] v118'1003 uv1003 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f7ddc00e4b0 con 0x55b2d4507770 2024-08-03T20:59:09.361 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.353+0000 7f7df7828f40 1 -- 172.21.15.32:0/617165089 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:3 12.10 12:09401a81:::601.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e118) -- 0x55b2d45d37d0 con 0x55b2d4507770 2024-08-03T20:59:09.361 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::601.00000000:head# 2024-08-03T20:59:09.362 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.355+0000 7f7debe00640 1 -- 172.21.15.32:0/617165089 <== osd.0 v2:172.21.15.32:6816/3066156116 3 ==== osd_op_reply(3 601.00000000 [create] v118'1004 uv1004 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f7ddc00e4b0 con 0x55b2d4507770 2024-08-03T20:59:09.362 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.355+0000 7f7df7828f40 1 -- 172.21.15.32:0/617165089 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:4 12.10 12:09401a81:::601.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e118) -- 0x55b2d45d4b80 con 0x55b2d4507770 2024-08-03T20:59:09.362 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.357+0000 7f7debe00640 1 -- 172.21.15.32:0/617165089 <== osd.0 v2:172.21.15.32:6816/3066156116 4 ==== osd_op_reply(4 601.00000000 [setxattr (30)] v118'1005 uv1005 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f7ddc00e4b0 con 0x55b2d4507770 2024-08-03T20:59:09.362 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.357+0000 7f7df7828f40 1 -- 172.21.15.32:0/617165089 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:5 12.10 12:09401a81:::601.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e118) -- 0x55b2d45d4f20 con 0x55b2d4507770 2024-08-03T20:59:09.362 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.359+0000 7f7debe00640 1 -- 172.21.15.32:0/617165089 <== osd.0 v2:172.21.15.32:6816/3066156116 5 ==== osd_op_reply(5 601.00000000 [setxattr (62)] v118'1006 uv1006 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f7ddc00e4b0 con 0x55b2d4507770 2024-08-03T20:59:09.362 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.359+0000 7f7df7828f40 1 -- 172.21.15.32:0/617165089 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:6 12.10 12:09401a81:::601.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e118) -- 0x55b2d45d3b70 con 0x55b2d4507770 2024-08-03T20:59:09.363 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.361+0000 7f7debe00640 1 -- 172.21.15.32:0/617165089 <== osd.0 v2:172.21.15.32:6816/3066156116 6 ==== osd_op_reply(6 601.00000000 [omap-set-header] v118'1007 uv1007 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f7ddc00e4b0 con 0x55b2d4507770 2024-08-03T20:59:09.363 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.361+0000 7f7df7828f40 1 -- 172.21.15.32:0/617165089 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:7 12.10 12:0bd6d154:::602.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e118) -- 0x55b2d45d3f60 con 0x55b2d4507770 2024-08-03T20:59:09.364 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.362+0000 7f7debe00640 1 -- 172.21.15.32:0/617165089 <== osd.0 v2:172.21.15.32:6816/3066156116 7 ==== osd_op_reply(7 602.00000000 [create] v118'1008 uv982 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f7ddc00e4b0 con 0x55b2d4507770 2024-08-03T20:59:09.364 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.362+0000 7f7df7828f40 1 -- 172.21.15.32:0/617165089 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:8 12.10 12:0bd6d154:::602.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e118) -- 0x55b2d45d4350 con 0x55b2d4507770 2024-08-03T20:59:09.365 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.364+0000 7f7debe00640 1 -- 172.21.15.32:0/617165089 <== osd.0 v2:172.21.15.32:6816/3066156116 8 ==== osd_op_reply(8 602.00000000 [delete] v118'1009 uv1009 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f7ddc00e4b0 con 0x55b2d4507770 2024-08-03T20:59:09.366 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.364+0000 7f7df7828f40 1 -- 172.21.15.32:0/617165089 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:9 12.10 12:0bd6d154:::602.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e118) -- 0x55b2d45d4350 con 0x55b2d4507770 2024-08-03T20:59:09.367 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::602.00000000:head# 2024-08-03T20:59:09.367 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.366+0000 7f7debe00640 1 -- 172.21.15.32:0/617165089 <== osd.0 v2:172.21.15.32:6816/3066156116 9 ==== osd_op_reply(9 602.00000000 [create] v118'1010 uv1010 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f7ddc00e4b0 con 0x55b2d4507770 2024-08-03T20:59:09.368 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.366+0000 7f7df7828f40 1 -- 172.21.15.32:0/617165089 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:10 12.10 12:0bd6d154:::602.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e118) -- 0x55b2d45d4350 con 0x55b2d4507770 2024-08-03T20:59:09.369 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.367+0000 7f7debe00640 1 -- 172.21.15.32:0/617165089 <== osd.0 v2:172.21.15.32:6816/3066156116 10 ==== osd_op_reply(10 602.00000000 [setxattr (30)] v118'1011 uv1011 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f7ddc00e4b0 con 0x55b2d4507770 2024-08-03T20:59:09.369 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.367+0000 7f7df7828f40 1 -- 172.21.15.32:0/617165089 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:11 12.10 12:0bd6d154:::602.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e118) -- 0x55b2d45d4350 con 0x55b2d4507770 2024-08-03T20:59:09.371 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.369+0000 7f7debe00640 1 -- 172.21.15.32:0/617165089 <== osd.0 v2:172.21.15.32:6816/3066156116 11 ==== osd_op_reply(11 602.00000000 [setxattr (62)] v118'1012 uv1012 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f7ddc00e4b0 con 0x55b2d4507770 2024-08-03T20:59:09.371 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.369+0000 7f7df7828f40 1 -- 172.21.15.32:0/617165089 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:12 12.10 12:0bd6d154:::602.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e118) -- 0x55b2d45d4350 con 0x55b2d4507770 2024-08-03T20:59:09.372 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.371+0000 7f7debe00640 1 -- 172.21.15.32:0/617165089 <== osd.0 v2:172.21.15.32:6816/3066156116 12 ==== osd_op_reply(12 602.00000000 [omap-set-header] v118'1013 uv1013 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f7ddc00e4b0 con 0x55b2d4507770 2024-08-03T20:59:09.372 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.371+0000 7f7df7828f40 1 -- 172.21.15.32:0/617165089 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:13 12.10 12:0cf6ad0f:::10000000000.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e118) -- 0x55b2d45d4350 con 0x55b2d4507770 2024-08-03T20:59:09.374 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.373+0000 7f7debe00640 1 -- 172.21.15.32:0/617165089 <== osd.0 v2:172.21.15.32:6816/3066156116 13 ==== osd_op_reply(13 10000000000.00000000 [create] v118'1014 uv989 ondisk = -17 ((17) File exists)) ==== 164+0+0 (crc 0 0 0) 0x7f7ddc00e4b0 con 0x55b2d4507770 2024-08-03T20:59:09.374 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.373+0000 7f7df7828f40 1 -- 172.21.15.32:0/617165089 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:14 12.10 12:0cf6ad0f:::10000000000.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e118) -- 0x55b2d45d4350 con 0x55b2d4507770 2024-08-03T20:59:09.376 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.375+0000 7f7debe00640 1 -- 172.21.15.32:0/617165089 <== osd.0 v2:172.21.15.32:6816/3066156116 14 ==== osd_op_reply(14 10000000000.00000000 [delete] v118'1015 uv1015 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f7ddc00e4b0 con 0x55b2d4507770 2024-08-03T20:59:09.376 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.375+0000 7f7df7828f40 1 -- 172.21.15.32:0/617165089 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:15 12.10 12:0cf6ad0f:::10000000000.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e118) -- 0x55b2d45d4350 con 0x55b2d4507770 2024-08-03T20:59:09.377 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::10000000000.00000000:head# 2024-08-03T20:59:09.378 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.376+0000 7f7debe00640 1 -- 172.21.15.32:0/617165089 <== osd.0 v2:172.21.15.32:6816/3066156116 15 ==== osd_op_reply(15 10000000000.00000000 [create] v118'1016 uv1016 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f7ddc00e4b0 con 0x55b2d4507770 2024-08-03T20:59:09.378 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.376+0000 7f7df7828f40 1 -- 172.21.15.32:0/617165089 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:16 12.10 12:0cf6ad0f:::10000000000.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e118) -- 0x55b2d45d4350 con 0x55b2d4507770 2024-08-03T20:59:09.379 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.378+0000 7f7debe00640 1 -- 172.21.15.32:0/617165089 <== osd.0 v2:172.21.15.32:6816/3066156116 16 ==== osd_op_reply(16 10000000000.00000000 [setxattr (30)] v118'1017 uv1017 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f7ddc00e4b0 con 0x55b2d4507770 2024-08-03T20:59:09.379 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.378+0000 7f7df7828f40 1 -- 172.21.15.32:0/617165089 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:17 12.10 12:0cf6ad0f:::10000000000.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e118) -- 0x55b2d45d4350 con 0x55b2d4507770 2024-08-03T20:59:09.381 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.380+0000 7f7debe00640 1 -- 172.21.15.32:0/617165089 <== osd.0 v2:172.21.15.32:6816/3066156116 17 ==== osd_op_reply(17 10000000000.00000000 [setxattr (62)] v118'1018 uv1018 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f7ddc00e4b0 con 0x55b2d4507770 2024-08-03T20:59:09.381 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.380+0000 7f7df7828f40 1 -- 172.21.15.32:0/617165089 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:18 12.10 12:0cf6ad0f:::10000000000.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e118) -- 0x55b2d45d47d0 con 0x55b2d4507770 2024-08-03T20:59:09.383 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.381+0000 7f7debe00640 1 -- 172.21.15.32:0/617165089 <== osd.0 v2:172.21.15.32:6816/3066156116 18 ==== osd_op_reply(18 10000000000.00000000 [omap-set-header] v118'1019 uv1019 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f7ddc00e4b0 con 0x55b2d4507770 2024-08-03T20:59:09.383 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.382+0000 7f7df7828f40 1 -- 172.21.15.32:0/617165089 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:19 12.10 12:0cf6ad0f:::10000000000.00000000:head [omap-set-vals in=593b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e118) -- 0x55b2d45d5770 con 0x55b2d4507770 2024-08-03T20:59:09.385 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.383+0000 7f7debe00640 1 -- 172.21.15.32:0/617165089 <== osd.0 v2:172.21.15.32:6816/3066156116 19 ==== osd_op_reply(19 10000000000.00000000 [omap-set-vals] v118'1020 uv1020 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f7ddc00e4b0 con 0x55b2d4507770 2024-08-03T20:59:09.385 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.383+0000 7f7df7828f40 1 -- 172.21.15.32:0/617165089 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:20 12.10 12:0d82a743:::600.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e118) -- 0x55b2d45d5770 con 0x55b2d4507770 2024-08-03T20:59:09.386 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.385+0000 7f7debe00640 1 -- 172.21.15.32:0/617165089 <== osd.0 v2:172.21.15.32:6816/3066156116 20 ==== osd_op_reply(20 600.00000000 [create] v118'1021 uv995 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f7ddc00e4b0 con 0x55b2d4507770 2024-08-03T20:59:09.387 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.385+0000 7f7df7828f40 1 -- 172.21.15.32:0/617165089 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:21 12.10 12:0d82a743:::600.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e118) -- 0x55b2d45d5770 con 0x55b2d4507770 2024-08-03T20:59:09.388 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.387+0000 7f7debe00640 1 -- 172.21.15.32:0/617165089 <== osd.0 v2:172.21.15.32:6816/3066156116 21 ==== osd_op_reply(21 600.00000000 [delete] v118'1022 uv1022 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f7ddc00e4b0 con 0x55b2d4507770 2024-08-03T20:59:09.389 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.387+0000 7f7df7828f40 1 -- 172.21.15.32:0/617165089 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:22 12.10 12:0d82a743:::600.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e118) -- 0x55b2d45d5770 con 0x55b2d4507770 2024-08-03T20:59:09.390 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.389+0000 7f7debe00640 1 -- 172.21.15.32:0/617165089 <== osd.0 v2:172.21.15.32:6816/3066156116 22 ==== osd_op_reply(22 600.00000000 [create] v118'1023 uv1023 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f7ddc00e4b0 con 0x55b2d4507770 2024-08-03T20:59:09.390 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::600.00000000:head# 2024-08-03T20:59:09.391 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.389+0000 7f7df7828f40 1 -- 172.21.15.32:0/617165089 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:23 12.10 12:0d82a743:::600.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e118) -- 0x55b2d45d6780 con 0x55b2d4507770 2024-08-03T20:59:09.392 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.390+0000 7f7debe00640 1 -- 172.21.15.32:0/617165089 <== osd.0 v2:172.21.15.32:6816/3066156116 23 ==== osd_op_reply(23 600.00000000 [setxattr (30)] v118'1024 uv1024 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f7ddc00e4b0 con 0x55b2d4507770 2024-08-03T20:59:09.392 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.390+0000 7f7df7828f40 1 -- 172.21.15.32:0/617165089 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:24 12.10 12:0d82a743:::600.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e118) -- 0x55b2d45d6780 con 0x55b2d4507770 2024-08-03T20:59:09.393 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.392+0000 7f7debe00640 1 -- 172.21.15.32:0/617165089 <== osd.0 v2:172.21.15.32:6816/3066156116 24 ==== osd_op_reply(24 600.00000000 [setxattr (62)] v118'1025 uv1025 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f7ddc00e4b0 con 0x55b2d4507770 2024-08-03T20:59:09.394 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.392+0000 7f7df7828f40 1 -- 172.21.15.32:0/617165089 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:25 12.10 12:0d82a743:::600.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e118) -- 0x55b2d45d5770 con 0x55b2d4507770 2024-08-03T20:59:09.395 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.394+0000 7f7debe00640 1 -- 172.21.15.32:0/617165089 <== osd.0 v2:172.21.15.32:6816/3066156116 25 ==== osd_op_reply(25 600.00000000 [omap-set-header] v118'1026 uv1026 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f7ddc00e4b0 con 0x55b2d4507770 2024-08-03T20:59:09.396 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.394+0000 7f7df7828f40 1 -- 172.21.15.32:0/617165089 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:26 12.10 12:0d89b25e:::603.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e118) -- 0x55b2d45d5770 con 0x55b2d4507770 2024-08-03T20:59:09.397 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.396+0000 7f7debe00640 1 -- 172.21.15.32:0/617165089 <== osd.0 v2:172.21.15.32:6816/3066156116 26 ==== osd_op_reply(26 603.00000000 [create] v118'1027 uv1001 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f7ddc00e4b0 con 0x55b2d4507770 2024-08-03T20:59:09.397 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.396+0000 7f7df7828f40 1 -- 172.21.15.32:0/617165089 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:27 12.10 12:0d89b25e:::603.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e118) -- 0x55b2d45d5770 con 0x55b2d4507770 2024-08-03T20:59:09.399 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.397+0000 7f7debe00640 1 -- 172.21.15.32:0/617165089 <== osd.0 v2:172.21.15.32:6816/3066156116 27 ==== osd_op_reply(27 603.00000000 [delete] v118'1028 uv1028 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f7ddc00e4b0 con 0x55b2d4507770 2024-08-03T20:59:09.399 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.397+0000 7f7df7828f40 1 -- 172.21.15.32:0/617165089 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:28 12.10 12:0d89b25e:::603.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e118) -- 0x55b2d45d5770 con 0x55b2d4507770 2024-08-03T20:59:09.400 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.399+0000 7f7debe00640 1 -- 172.21.15.32:0/617165089 <== osd.0 v2:172.21.15.32:6816/3066156116 28 ==== osd_op_reply(28 603.00000000 [create] v118'1029 uv1029 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f7ddc00e4b0 con 0x55b2d4507770 2024-08-03T20:59:09.401 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::603.00000000:head# 2024-08-03T20:59:09.401 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.399+0000 7f7df7828f40 1 -- 172.21.15.32:0/617165089 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:29 12.10 12:0d89b25e:::603.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e118) -- 0x55b2d45d6780 con 0x55b2d4507770 2024-08-03T20:59:09.402 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.401+0000 7f7debe00640 1 -- 172.21.15.32:0/617165089 <== osd.0 v2:172.21.15.32:6816/3066156116 29 ==== osd_op_reply(29 603.00000000 [setxattr (30)] v118'1030 uv1030 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f7ddc00e4b0 con 0x55b2d4507770 2024-08-03T20:59:09.403 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.401+0000 7f7df7828f40 1 -- 172.21.15.32:0/617165089 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:30 12.10 12:0d89b25e:::603.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e118) -- 0x55b2d45d6780 con 0x55b2d4507770 2024-08-03T20:59:09.404 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.402+0000 7f7debe00640 1 -- 172.21.15.32:0/617165089 <== osd.0 v2:172.21.15.32:6816/3066156116 30 ==== osd_op_reply(30 603.00000000 [setxattr (62)] v118'1031 uv1031 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f7ddc00e4b0 con 0x55b2d4507770 2024-08-03T20:59:09.404 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.403+0000 7f7df7828f40 1 -- 172.21.15.32:0/617165089 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:31 12.10 12:0d89b25e:::603.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e118) -- 0x55b2d45d5770 con 0x55b2d4507770 2024-08-03T20:59:09.406 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.404+0000 7f7debe00640 1 -- 172.21.15.32:0/617165089 <== osd.0 v2:172.21.15.32:6816/3066156116 31 ==== osd_op_reply(31 603.00000000 [omap-set-header] v118'1032 uv1032 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f7ddc00e4b0 con 0x55b2d4507770 2024-08-03T20:59:09.406 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.404+0000 7f7df7828f40 1 --2- 172.21.15.32:0/617165089 >> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] conn(0x55b2d45d5c80 0x55b2d45d80a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:59:09.406 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.404+0000 7f7df7828f40 1 -- 172.21.15.32:0/617165089 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:32 12.14 12:287034c4:::1.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e118) -- 0x55b2d45d85e0 con 0x55b2d45d5c80 2024-08-03T20:59:09.406 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.405+0000 7f7df0a00640 1 --2- 172.21.15.32:0/617165089 >> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] conn(0x55b2d45d5c80 0x55b2d45d80a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:59:09.407 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.406+0000 7f7df0a00640 1 --2- 172.21.15.32:0/617165089 >> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] conn(0x55b2d45d5c80 0x55b2d45d80a0 crc :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:59:09.409 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.408+0000 7f7df0a00640 1 -- 172.21.15.32:0/617165089 <== osd.5 v2:172.21.15.118:6810/2724891770 1 ==== osd_op_reply(32 1.00000000.inode [create] v118'727 uv709 ondisk = -17 ((17) File exists)) ==== 160+0+0 (crc 0 0 0) 0x7f7de405f070 con 0x55b2d45d5c80 2024-08-03T20:59:09.409 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.408+0000 7f7df7828f40 1 -- 172.21.15.32:0/617165089 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:33 12.14 12:287034c4:::1.00000000.inode:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e118) -- 0x55b2d45d8980 con 0x55b2d45d5c80 2024-08-03T20:59:09.411 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.410+0000 7f7df0a00640 1 -- 172.21.15.32:0/617165089 <== osd.5 v2:172.21.15.118:6810/2724891770 2 ==== osd_op_reply(33 1.00000000.inode [delete] v118'728 uv728 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f7de405f070 con 0x55b2d45d5c80 2024-08-03T20:59:09.411 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.410+0000 7f7df7828f40 1 -- 172.21.15.32:0/617165089 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:34 12.14 12:287034c4:::1.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e118) -- 0x55b2d45d8d20 con 0x55b2d45d5c80 2024-08-03T20:59:09.412 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::1.00000000.inode:head# 2024-08-03T20:59:09.413 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.411+0000 7f7df0a00640 1 -- 172.21.15.32:0/617165089 <== osd.5 v2:172.21.15.118:6810/2724891770 3 ==== osd_op_reply(34 1.00000000.inode [create] v118'729 uv729 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f7de405f070 con 0x55b2d45d5c80 2024-08-03T20:59:09.413 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.411+0000 7f7df7828f40 1 -- 172.21.15.32:0/617165089 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:35 12.14 12:287034c4:::1.00000000.inode:head [write 0~582 in=582b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e118) -- 0x55b2d45d90c0 con 0x55b2d45d5c80 2024-08-03T20:59:09.414 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.413+0000 7f7df0a00640 1 -- 172.21.15.32:0/617165089 <== osd.5 v2:172.21.15.118:6810/2724891770 4 ==== osd_op_reply(35 1.00000000.inode [write 0~582] v118'730 uv730 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f7de405f070 con 0x55b2d45d5c80 2024-08-03T20:59:09.414 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.413+0000 7f7df7828f40 1 -- 172.21.15.32:0/617165089 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:36 12.14 12:287034c4:::1.00000000.inode:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e118) -- 0x55b2d45d9460 con 0x55b2d45d5c80 2024-08-03T20:59:09.416 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.414+0000 7f7df0a00640 1 -- 172.21.15.32:0/617165089 <== osd.5 v2:172.21.15.118:6810/2724891770 5 ==== osd_op_reply(36 1.00000000.inode [omap-set-header] v118'731 uv731 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f7de405f070 con 0x55b2d45d5c80 2024-08-03T20:59:09.416 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.414+0000 7f7df7828f40 1 -- 172.21.15.32:0/617165089 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:37 12.14 12:292cf221:::200.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e118) -- 0x55b2d45d9800 con 0x55b2d45d5c80 2024-08-03T20:59:09.417 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.416+0000 7f7df0a00640 1 -- 172.21.15.32:0/617165089 <== osd.5 v2:172.21.15.118:6810/2724891770 6 ==== osd_op_reply(37 200.00000000 [create] v118'732 uv714 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f7de405f070 con 0x55b2d45d5c80 2024-08-03T20:59:09.418 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.416+0000 7f7df7828f40 1 -- 172.21.15.32:0/617165089 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:38 12.14 12:292cf221:::200.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e118) -- 0x55b2d45d9800 con 0x55b2d45d5c80 2024-08-03T20:59:09.419 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.418+0000 7f7df0a00640 1 -- 172.21.15.32:0/617165089 <== osd.5 v2:172.21.15.118:6810/2724891770 7 ==== osd_op_reply(38 200.00000000 [delete] v118'733 uv733 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f7de405f070 con 0x55b2d45d5c80 2024-08-03T20:59:09.419 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.418+0000 7f7df7828f40 1 -- 172.21.15.32:0/617165089 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:39 12.14 12:292cf221:::200.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e118) -- 0x55b2d45d9800 con 0x55b2d45d5c80 2024-08-03T20:59:09.421 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::200.00000000:head# 2024-08-03T20:59:09.421 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.420+0000 7f7df0a00640 1 -- 172.21.15.32:0/617165089 <== osd.5 v2:172.21.15.118:6810/2724891770 8 ==== osd_op_reply(39 200.00000000 [create] v118'734 uv734 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f7de405f070 con 0x55b2d45d5c80 2024-08-03T20:59:09.422 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.420+0000 7f7df7828f40 1 -- 172.21.15.32:0/617165089 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:40 12.14 12:292cf221:::200.00000000:head [write 0~90 in=90b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e118) -- 0x55b2d45d9800 con 0x55b2d45d5c80 2024-08-03T20:59:09.422 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.421+0000 7f7df0a00640 1 -- 172.21.15.32:0/617165089 <== osd.5 v2:172.21.15.118:6810/2724891770 9 ==== osd_op_reply(40 200.00000000 [write 0~90] v118'735 uv735 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f7de405f070 con 0x55b2d45d5c80 2024-08-03T20:59:09.423 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.421+0000 7f7df7828f40 1 -- 172.21.15.32:0/617165089 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:41 12.14 12:292cf221:::200.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e118) -- 0x55b2d45d9800 con 0x55b2d45d5c80 2024-08-03T20:59:09.424 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.423+0000 7f7df0a00640 1 -- 172.21.15.32:0/617165089 <== osd.5 v2:172.21.15.118:6810/2724891770 10 ==== osd_op_reply(41 200.00000000 [omap-set-header] v118'736 uv736 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f7de405f070 con 0x55b2d45d5c80 2024-08-03T20:59:09.424 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.423+0000 7f7df7828f40 1 -- 172.21.15.32:0/617165089 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:42 12.14 12:2e2fa760:::200.00000001:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e118) -- 0x55b2d45d9800 con 0x55b2d45d5c80 2024-08-03T20:59:09.426 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.425+0000 7f7df0a00640 1 -- 172.21.15.32:0/617165089 <== osd.5 v2:172.21.15.118:6810/2724891770 11 ==== osd_op_reply(42 200.00000001 [create] v118'737 uv719 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f7de405f070 con 0x55b2d45d5c80 2024-08-03T20:59:09.426 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.425+0000 7f7df7828f40 1 -- 172.21.15.32:0/617165089 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:43 12.14 12:2e2fa760:::200.00000001:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e118) -- 0x55b2d45d9800 con 0x55b2d45d5c80 2024-08-03T20:59:09.428 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.427+0000 7f7df0a00640 1 -- 172.21.15.32:0/617165089 <== osd.5 v2:172.21.15.118:6810/2724891770 12 ==== osd_op_reply(43 200.00000001 [delete] v118'738 uv738 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f7de405f070 con 0x55b2d45d5c80 2024-08-03T20:59:09.428 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.427+0000 7f7df7828f40 1 -- 172.21.15.32:0/617165089 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:44 12.14 12:2e2fa760:::200.00000001:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e118) -- 0x55b2d45d9800 con 0x55b2d45d5c80 2024-08-03T20:59:09.430 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::200.00000001:head# 2024-08-03T20:59:09.431 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.428+0000 7f7df0a00640 1 -- 172.21.15.32:0/617165089 <== osd.5 v2:172.21.15.118:6810/2724891770 13 ==== osd_op_reply(44 200.00000001 [create] v118'739 uv739 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f7de405f070 con 0x55b2d45d5c80 2024-08-03T20:59:09.431 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.429+0000 7f7df7828f40 1 -- 172.21.15.32:0/617165089 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:45 12.14 12:2e2fa760:::200.00000001:head [write 0~18118 in=18118b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e118) -- 0x55b2d45ddef0 con 0x55b2d45d5c80 2024-08-03T20:59:09.432 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.430+0000 7f7df0a00640 1 -- 172.21.15.32:0/617165089 <== osd.5 v2:172.21.15.118:6810/2724891770 14 ==== osd_op_reply(45 200.00000001 [write 0~18118] v118'740 uv740 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f7de405f070 con 0x55b2d45d5c80 2024-08-03T20:59:09.432 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.430+0000 7f7df7828f40 1 -- 172.21.15.32:0/617165089 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:46 12.14 12:2e2fa760:::200.00000001:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e118) -- 0x55b2d45ddef0 con 0x55b2d45d5c80 2024-08-03T20:59:09.433 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.432+0000 7f7df0a00640 1 -- 172.21.15.32:0/617165089 <== osd.5 v2:172.21.15.118:6810/2724891770 15 ==== osd_op_reply(46 200.00000001 [omap-set-header] v118'741 uv741 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f7de405f070 con 0x55b2d45d5c80 2024-08-03T20:59:09.434 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.432+0000 7f7df7828f40 1 -- 172.21.15.32:0/617165089 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:47 12.14 12:2f55791f:::606.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e118) -- 0x55b2d45d9800 con 0x55b2d45d5c80 2024-08-03T20:59:09.435 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.434+0000 7f7df0a00640 1 -- 172.21.15.32:0/617165089 <== osd.5 v2:172.21.15.118:6810/2724891770 16 ==== osd_op_reply(47 606.00000000 [create] v118'742 uv725 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f7de405f070 con 0x55b2d45d5c80 2024-08-03T20:59:09.435 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.434+0000 7f7df7828f40 1 -- 172.21.15.32:0/617165089 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:48 12.14 12:2f55791f:::606.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e118) -- 0x55b2d45d9800 con 0x55b2d45d5c80 2024-08-03T20:59:09.437 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.436+0000 7f7df0a00640 1 -- 172.21.15.32:0/617165089 <== osd.5 v2:172.21.15.118:6810/2724891770 17 ==== osd_op_reply(48 606.00000000 [delete] v118'743 uv743 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f7de405f070 con 0x55b2d45d5c80 2024-08-03T20:59:09.437 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.436+0000 7f7df7828f40 1 -- 172.21.15.32:0/617165089 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:49 12.14 12:2f55791f:::606.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e118) -- 0x55b2d45d9800 con 0x55b2d45d5c80 2024-08-03T20:59:09.438 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::606.00000000:head# 2024-08-03T20:59:09.439 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.437+0000 7f7df0a00640 1 -- 172.21.15.32:0/617165089 <== osd.5 v2:172.21.15.118:6810/2724891770 18 ==== osd_op_reply(49 606.00000000 [create] v118'744 uv744 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f7de405f070 con 0x55b2d45d5c80 2024-08-03T20:59:09.439 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.437+0000 7f7df7828f40 1 -- 172.21.15.32:0/617165089 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:50 12.14 12:2f55791f:::606.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e118) -- 0x55b2d45da810 con 0x55b2d45d5c80 2024-08-03T20:59:09.440 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.439+0000 7f7df0a00640 1 -- 172.21.15.32:0/617165089 <== osd.5 v2:172.21.15.118:6810/2724891770 19 ==== osd_op_reply(50 606.00000000 [setxattr (30)] v118'745 uv745 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f7de405f070 con 0x55b2d45d5c80 2024-08-03T20:59:09.440 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.439+0000 7f7df7828f40 1 -- 172.21.15.32:0/617165089 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:51 12.14 12:2f55791f:::606.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e118) -- 0x55b2d45da810 con 0x55b2d45d5c80 2024-08-03T20:59:09.442 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.440+0000 7f7df0a00640 1 -- 172.21.15.32:0/617165089 <== osd.5 v2:172.21.15.118:6810/2724891770 20 ==== osd_op_reply(51 606.00000000 [setxattr (62)] v118'746 uv746 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f7de405f070 con 0x55b2d45d5c80 2024-08-03T20:59:09.442 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.440+0000 7f7df7828f40 1 -- 172.21.15.32:0/617165089 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:52 12.14 12:2f55791f:::606.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e118) -- 0x55b2d45d9800 con 0x55b2d45d5c80 2024-08-03T20:59:09.446 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.442+0000 7f7df0a00640 1 -- 172.21.15.32:0/617165089 <== osd.5 v2:172.21.15.118:6810/2724891770 21 ==== osd_op_reply(52 606.00000000 [omap-set-header] v118'747 uv747 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f7de405f070 con 0x55b2d45d5c80 2024-08-03T20:59:09.446 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.442+0000 7f7df7828f40 1 -- 172.21.15.32:0/617165089 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:53 12.1c 12:3ed09add:::607.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e118) -- 0x55b2d45d9920 con 0x55b2d4507770 2024-08-03T20:59:09.446 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.443+0000 7f7debe00640 1 -- 172.21.15.32:0/617165089 <== osd.0 v2:172.21.15.32:6816/3066156116 32 ==== osd_op_reply(53 607.00000000 [create] v118'195 uv194 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f7ddc00e4b0 con 0x55b2d4507770 2024-08-03T20:59:09.446 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.443+0000 7f7df7828f40 1 -- 172.21.15.32:0/617165089 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:54 12.1c 12:3ed09add:::607.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e118) -- 0x55b2d45d9920 con 0x55b2d4507770 2024-08-03T20:59:09.447 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.445+0000 7f7debe00640 1 -- 172.21.15.32:0/617165089 <== osd.0 v2:172.21.15.32:6816/3066156116 33 ==== osd_op_reply(54 607.00000000 [delete] v118'196 uv196 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f7ddc00e4b0 con 0x55b2d4507770 2024-08-03T20:59:09.447 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.445+0000 7f7df7828f40 1 -- 172.21.15.32:0/617165089 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:55 12.1c 12:3ed09add:::607.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e118) -- 0x55b2d45d9920 con 0x55b2d4507770 2024-08-03T20:59:09.447 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.446+0000 7f7debe00640 1 -- 172.21.15.32:0/617165089 <== osd.0 v2:172.21.15.32:6816/3066156116 34 ==== osd_op_reply(55 607.00000000 [create] v118'197 uv197 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f7ddc00e4b0 con 0x55b2d4507770 2024-08-03T20:59:09.448 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::607.00000000:head# 2024-08-03T20:59:09.448 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.446+0000 7f7df7828f40 1 -- 172.21.15.32:0/617165089 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:56 12.1c 12:3ed09add:::607.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e118) -- 0x55b2d45da930 con 0x55b2d4507770 2024-08-03T20:59:09.449 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.447+0000 7f7debe00640 1 -- 172.21.15.32:0/617165089 <== osd.0 v2:172.21.15.32:6816/3066156116 35 ==== osd_op_reply(56 607.00000000 [setxattr (30)] v118'198 uv198 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f7ddc00e4b0 con 0x55b2d4507770 2024-08-03T20:59:09.449 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.447+0000 7f7df7828f40 1 -- 172.21.15.32:0/617165089 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:57 12.1c 12:3ed09add:::607.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e118) -- 0x55b2d45da930 con 0x55b2d4507770 2024-08-03T20:59:09.450 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.449+0000 7f7debe00640 1 -- 172.21.15.32:0/617165089 <== osd.0 v2:172.21.15.32:6816/3066156116 36 ==== osd_op_reply(57 607.00000000 [setxattr (62)] v118'199 uv199 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f7ddc00e4b0 con 0x55b2d4507770 2024-08-03T20:59:09.450 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.449+0000 7f7df7828f40 1 -- 172.21.15.32:0/617165089 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:58 12.1c 12:3ed09add:::607.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e118) -- 0x55b2d45d9920 con 0x55b2d4507770 2024-08-03T20:59:09.451 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.450+0000 7f7debe00640 1 -- 172.21.15.32:0/617165089 <== osd.0 v2:172.21.15.32:6816/3066156116 37 ==== osd_op_reply(58 607.00000000 [omap-set-header] v118'200 uv200 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f7ddc00e4b0 con 0x55b2d4507770 2024-08-03T20:59:09.452 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.450+0000 7f7df7828f40 1 --2- 172.21.15.32:0/617165089 >> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] conn(0x55b2d45d9fa0 0x55b2d45da3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:59:09.452 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.450+0000 7f7df7828f40 1 -- 172.21.15.32:0/617165089 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:59 12.6 12:654134d2:::mds0_openfiles.0:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e118) -- 0x55b2d45dcd30 con 0x55b2d45d9fa0 2024-08-03T20:59:09.452 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.450+0000 7f7debe00640 1 --2- 172.21.15.32:0/617165089 >> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] conn(0x55b2d45d9fa0 0x55b2d45da3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:59:09.452 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.451+0000 7f7debe00640 1 --2- 172.21.15.32:0/617165089 >> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] conn(0x55b2d45d9fa0 0x55b2d45da3b0 crc :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:59:09.453 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.452+0000 7f7debe00640 1 -- 172.21.15.32:0/617165089 <== osd.1 v2:172.21.15.32:6824/1551241759 1 ==== osd_op_reply(59 mds0_openfiles.0 [create] v118'152 uv151 ondisk = -17 ((17) File exists)) ==== 160+0+0 (crc 0 0 0) 0x7f7ddc00e4b0 con 0x55b2d45d9fa0 2024-08-03T20:59:09.453 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.452+0000 7f7df7828f40 1 -- 172.21.15.32:0/617165089 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:60 12.6 12:654134d2:::mds0_openfiles.0:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e118) -- 0x55b2d45dcd30 con 0x55b2d45d9fa0 2024-08-03T20:59:09.454 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.453+0000 7f7debe00640 1 -- 172.21.15.32:0/617165089 <== osd.1 v2:172.21.15.32:6824/1551241759 2 ==== osd_op_reply(60 mds0_openfiles.0 [delete] v118'153 uv153 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f7ddc00e4b0 con 0x55b2d45d9fa0 2024-08-03T20:59:09.455 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.453+0000 7f7df7828f40 1 -- 172.21.15.32:0/617165089 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:61 12.6 12:654134d2:::mds0_openfiles.0:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e118) -- 0x55b2d45dcd30 con 0x55b2d45d9fa0 2024-08-03T20:59:09.456 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.455+0000 7f7debe00640 1 -- 172.21.15.32:0/617165089 <== osd.1 v2:172.21.15.32:6824/1551241759 3 ==== osd_op_reply(61 mds0_openfiles.0 [create] v118'154 uv154 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f7ddc00e4b0 con 0x55b2d45d9fa0 2024-08-03T20:59:09.456 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds0_openfiles.0:head# 2024-08-03T20:59:09.456 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.455+0000 7f7df7828f40 1 -- 172.21.15.32:0/617165089 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:62 12.6 12:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e118) -- 0x55b2d45dcd30 con 0x55b2d45d9fa0 2024-08-03T20:59:09.457 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.456+0000 7f7debe00640 1 -- 172.21.15.32:0/617165089 <== osd.1 v2:172.21.15.32:6824/1551241759 4 ==== osd_op_reply(62 mds0_openfiles.0 [omap-set-header] v118'155 uv155 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f7ddc00e4b0 con 0x55b2d45d9fa0 2024-08-03T20:59:09.457 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.456+0000 7f7df7828f40 1 -- 172.21.15.32:0/617165089 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:63 12.e 12:717a0223:::608.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e118) -- 0x55b2d45dd150 con 0x55b2d4507770 2024-08-03T20:59:09.459 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.457+0000 7f7debe00640 1 -- 172.21.15.32:0/617165089 <== osd.0 v2:172.21.15.32:6816/3066156116 38 ==== osd_op_reply(63 608.00000000 [create] v118'195 uv194 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f7ddc00e4b0 con 0x55b2d4507770 2024-08-03T20:59:09.459 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.458+0000 7f7df7828f40 1 -- 172.21.15.32:0/617165089 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:64 12.e 12:717a0223:::608.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e118) -- 0x55b2d45dd150 con 0x55b2d4507770 2024-08-03T20:59:09.461 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.459+0000 7f7debe00640 1 -- 172.21.15.32:0/617165089 <== osd.0 v2:172.21.15.32:6816/3066156116 39 ==== osd_op_reply(64 608.00000000 [delete] v118'196 uv196 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f7ddc00e4b0 con 0x55b2d4507770 2024-08-03T20:59:09.461 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.459+0000 7f7df7828f40 1 -- 172.21.15.32:0/617165089 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:65 12.e 12:717a0223:::608.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e118) -- 0x55b2d45dcdb0 con 0x55b2d4507770 2024-08-03T20:59:09.462 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.460+0000 7f7debe00640 1 -- 172.21.15.32:0/617165089 <== osd.0 v2:172.21.15.32:6816/3066156116 40 ==== osd_op_reply(65 608.00000000 [create] v118'197 uv197 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f7ddc00e4b0 con 0x55b2d4507770 2024-08-03T20:59:09.462 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::608.00000000:head# 2024-08-03T20:59:09.462 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.461+0000 7f7df7828f40 1 -- 172.21.15.32:0/617165089 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:66 12.e 12:717a0223:::608.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e118) -- 0x55b2d45dddc0 con 0x55b2d4507770 2024-08-03T20:59:09.463 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.462+0000 7f7debe00640 1 -- 172.21.15.32:0/617165089 <== osd.0 v2:172.21.15.32:6816/3066156116 41 ==== osd_op_reply(66 608.00000000 [setxattr (30)] v118'198 uv198 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f7ddc00e4b0 con 0x55b2d4507770 2024-08-03T20:59:09.464 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.462+0000 7f7df7828f40 1 -- 172.21.15.32:0/617165089 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:67 12.e 12:717a0223:::608.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e118) -- 0x55b2d45ddd40 con 0x55b2d4507770 2024-08-03T20:59:09.465 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.464+0000 7f7debe00640 1 -- 172.21.15.32:0/617165089 <== osd.0 v2:172.21.15.32:6816/3066156116 42 ==== osd_op_reply(67 608.00000000 [setxattr (62)] v118'199 uv199 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f7ddc00e4b0 con 0x55b2d4507770 2024-08-03T20:59:09.465 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.464+0000 7f7df7828f40 1 -- 172.21.15.32:0/617165089 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:68 12.e 12:717a0223:::608.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e118) -- 0x55b2d45dcd30 con 0x55b2d4507770 2024-08-03T20:59:09.466 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.465+0000 7f7debe00640 1 -- 172.21.15.32:0/617165089 <== osd.0 v2:172.21.15.32:6816/3066156116 43 ==== osd_op_reply(68 608.00000000 [omap-set-header] v118'200 uv200 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f7ddc00e4b0 con 0x55b2d4507770 2024-08-03T20:59:09.466 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.465+0000 7f7df7828f40 1 --2- 172.21.15.32:0/617165089 >> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] conn(0x55b2d45dd400 0x55b2d45dd810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:59:09.467 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.465+0000 7f7df7828f40 1 -- 172.21.15.32:0/617165089 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:69 12.15 12:a93a17c2:::604.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e118) -- 0x55b2d45e0190 con 0x55b2d45dd400 2024-08-03T20:59:09.467 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.465+0000 7f7df1400640 1 --2- 172.21.15.32:0/617165089 >> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] conn(0x55b2d45dd400 0x55b2d45dd810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:59:09.467 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.466+0000 7f7df1400640 1 --2- 172.21.15.32:0/617165089 >> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] conn(0x55b2d45dd400 0x55b2d45dd810 crc :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:59:09.469 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.468+0000 7f7df1400640 1 -- 172.21.15.32:0/617165089 <== osd.6 v2:172.21.15.118:6825/801947386 1 ==== osd_op_reply(69 604.00000000 [create] v118'360 uv354 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f7dec019030 con 0x55b2d45dd400 2024-08-03T20:59:09.469 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.468+0000 7f7df7828f40 1 -- 172.21.15.32:0/617165089 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:70 12.15 12:a93a17c2:::604.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e118) -- 0x55b2d45e0530 con 0x55b2d45dd400 2024-08-03T20:59:09.471 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.470+0000 7f7df1400640 1 -- 172.21.15.32:0/617165089 <== osd.6 v2:172.21.15.118:6825/801947386 2 ==== osd_op_reply(70 604.00000000 [delete] v118'361 uv361 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f7dec019030 con 0x55b2d45dd400 2024-08-03T20:59:09.471 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.470+0000 7f7df7828f40 1 -- 172.21.15.32:0/617165089 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:71 12.15 12:a93a17c2:::604.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e118) -- 0x55b2d45e0910 con 0x55b2d45dd400 2024-08-03T20:59:09.472 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::604.00000000:head# 2024-08-03T20:59:09.473 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.471+0000 7f7df1400640 1 -- 172.21.15.32:0/617165089 <== osd.6 v2:172.21.15.118:6825/801947386 3 ==== osd_op_reply(71 604.00000000 [create] v118'362 uv362 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f7dec019030 con 0x55b2d45dd400 2024-08-03T20:59:09.473 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.471+0000 7f7df7828f40 1 -- 172.21.15.32:0/617165089 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:72 12.15 12:a93a17c2:::604.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e118) -- 0x55b2d45e21c0 con 0x55b2d45dd400 2024-08-03T20:59:09.474 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.473+0000 7f7df1400640 1 -- 172.21.15.32:0/617165089 <== osd.6 v2:172.21.15.118:6825/801947386 4 ==== osd_op_reply(72 604.00000000 [setxattr (30)] v118'363 uv363 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f7dec019030 con 0x55b2d45dd400 2024-08-03T20:59:09.474 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.473+0000 7f7df7828f40 1 -- 172.21.15.32:0/617165089 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:73 12.15 12:a93a17c2:::604.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e118) -- 0x55b2d45e2560 con 0x55b2d45dd400 2024-08-03T20:59:09.475 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.474+0000 7f7df1400640 1 -- 172.21.15.32:0/617165089 <== osd.6 v2:172.21.15.118:6825/801947386 5 ==== osd_op_reply(73 604.00000000 [setxattr (62)] v118'364 uv364 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f7dec019030 con 0x55b2d45dd400 2024-08-03T20:59:09.475 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.474+0000 7f7df7828f40 1 -- 172.21.15.32:0/617165089 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:74 12.15 12:a93a17c2:::604.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e118) -- 0x55b2d45e11b0 con 0x55b2d45dd400 2024-08-03T20:59:09.476 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.475+0000 7f7df1400640 1 -- 172.21.15.32:0/617165089 <== osd.6 v2:172.21.15.118:6825/801947386 6 ==== osd_op_reply(74 604.00000000 [omap-set-header] v118'365 uv365 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f7dec019030 con 0x55b2d45dd400 2024-08-03T20:59:09.476 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.475+0000 7f7df7828f40 1 -- 172.21.15.32:0/617165089 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:75 12.15 12:aa448500:::500.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e118) -- 0x55b2d45e15a0 con 0x55b2d45dd400 2024-08-03T20:59:09.478 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.477+0000 7f7df1400640 1 -- 172.21.15.32:0/617165089 <== osd.6 v2:172.21.15.118:6825/801947386 7 ==== osd_op_reply(75 500.00000000 [create] v118'366 uv359 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f7dec019030 con 0x55b2d45dd400 2024-08-03T20:59:09.478 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.477+0000 7f7df7828f40 1 -- 172.21.15.32:0/617165089 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:76 12.15 12:aa448500:::500.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e118) -- 0x55b2d45e15a0 con 0x55b2d45dd400 2024-08-03T20:59:09.479 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.478+0000 7f7df1400640 1 -- 172.21.15.32:0/617165089 <== osd.6 v2:172.21.15.118:6825/801947386 8 ==== osd_op_reply(76 500.00000000 [delete] v118'367 uv367 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f7dec019030 con 0x55b2d45dd400 2024-08-03T20:59:09.480 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.478+0000 7f7df7828f40 1 -- 172.21.15.32:0/617165089 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:77 12.15 12:aa448500:::500.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e118) -- 0x55b2d45e15a0 con 0x55b2d45dd400 2024-08-03T20:59:09.481 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::500.00000000:head# 2024-08-03T20:59:09.481 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.479+0000 7f7df1400640 1 -- 172.21.15.32:0/617165089 <== osd.6 v2:172.21.15.118:6825/801947386 9 ==== osd_op_reply(77 500.00000000 [create] v118'368 uv368 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f7dec019030 con 0x55b2d45dd400 2024-08-03T20:59:09.481 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.479+0000 7f7df7828f40 1 -- 172.21.15.32:0/617165089 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:78 12.15 12:aa448500:::500.00000000:head [write 0~90 in=90b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e118) -- 0x55b2d45e15a0 con 0x55b2d45dd400 2024-08-03T20:59:09.482 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.481+0000 7f7df1400640 1 -- 172.21.15.32:0/617165089 <== osd.6 v2:172.21.15.118:6825/801947386 10 ==== osd_op_reply(78 500.00000000 [write 0~90] v118'369 uv369 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f7dec019030 con 0x55b2d45dd400 2024-08-03T20:59:09.482 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.481+0000 7f7df7828f40 1 -- 172.21.15.32:0/617165089 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:79 12.15 12:aa448500:::500.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e118) -- 0x55b2d45e15a0 con 0x55b2d45dd400 2024-08-03T20:59:09.484 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.483+0000 7f7df1400640 1 -- 172.21.15.32:0/617165089 <== osd.6 v2:172.21.15.118:6825/801947386 11 ==== osd_op_reply(79 500.00000000 [omap-set-header] v118'370 uv370 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f7dec019030 con 0x55b2d45dd400 2024-08-03T20:59:09.485 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.483+0000 7f7df7828f40 1 --2- 172.21.15.32:0/617165089 >> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] conn(0x55b2d45e2900 0x55b2d45e4d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:59:09.485 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.483+0000 7f7df7828f40 1 -- 172.21.15.32:0/617165089 --> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] -- osd_op(unknown.0.0:80 12.d 12:b50e409b:::mds_snaptable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e118) -- 0x55b2d45e15a0 con 0x55b2d45e2900 2024-08-03T20:59:09.485 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.483+0000 7f7df0a00640 1 --2- 172.21.15.32:0/617165089 >> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] conn(0x55b2d45e2900 0x55b2d45e4d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:59:09.485 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.484+0000 7f7df0a00640 1 --2- 172.21.15.32:0/617165089 >> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] conn(0x55b2d45e2900 0x55b2d45e4d20 crc :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:59:09.487 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.486+0000 7f7df0a00640 1 -- 172.21.15.32:0/617165089 <== osd.7 v2:172.21.15.118:6818/653000048 1 ==== osd_op_reply(80 mds_snaptable [create] v118'163 uv162 ondisk = -17 ((17) File exists)) ==== 157+0+0 (crc 0 0 0) 0x7f7de405f070 con 0x55b2d45e2900 2024-08-03T20:59:09.487 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.486+0000 7f7df7828f40 1 -- 172.21.15.32:0/617165089 --> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] -- osd_op(unknown.0.0:81 12.d 12:b50e409b:::mds_snaptable:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e118) -- 0x55b2d45e15a0 con 0x55b2d45e2900 2024-08-03T20:59:09.489 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.487+0000 7f7df0a00640 1 -- 172.21.15.32:0/617165089 <== osd.7 v2:172.21.15.118:6818/653000048 2 ==== osd_op_reply(81 mds_snaptable [delete] v118'164 uv164 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f7de405f070 con 0x55b2d45e2900 2024-08-03T20:59:09.489 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.487+0000 7f7df7828f40 1 -- 172.21.15.32:0/617165089 --> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] -- osd_op(unknown.0.0:82 12.d 12:b50e409b:::mds_snaptable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e118) -- 0x55b2d45e52b0 con 0x55b2d45e2900 2024-08-03T20:59:09.490 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds_snaptable:head# 2024-08-03T20:59:09.490 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.488+0000 7f7df0a00640 1 -- 172.21.15.32:0/617165089 <== osd.7 v2:172.21.15.118:6818/653000048 3 ==== osd_op_reply(82 mds_snaptable [create] v118'165 uv165 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f7de405f070 con 0x55b2d45e2900 2024-08-03T20:59:09.490 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.488+0000 7f7df7828f40 1 -- 172.21.15.32:0/617165089 --> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] -- osd_op(unknown.0.0:83 12.d 12:b50e409b:::mds_snaptable:head [write 0~70 in=70b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e118) -- 0x55b2d45e52b0 con 0x55b2d45e2900 2024-08-03T20:59:09.491 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.490+0000 7f7df0a00640 1 -- 172.21.15.32:0/617165089 <== osd.7 v2:172.21.15.118:6818/653000048 4 ==== osd_op_reply(83 mds_snaptable [write 0~70] v118'166 uv166 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f7de405f070 con 0x55b2d45e2900 2024-08-03T20:59:09.492 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.490+0000 7f7df7828f40 1 -- 172.21.15.32:0/617165089 --> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] -- osd_op(unknown.0.0:84 12.d 12:b50e409b:::mds_snaptable:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e118) -- 0x55b2d45e52b0 con 0x55b2d45e2900 2024-08-03T20:59:09.493 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.492+0000 7f7df0a00640 1 -- 172.21.15.32:0/617165089 <== osd.7 v2:172.21.15.118:6818/653000048 5 ==== osd_op_reply(84 mds_snaptable [omap-set-header] v118'167 uv167 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f7de405f070 con 0x55b2d45e2900 2024-08-03T20:59:09.493 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.492+0000 7f7df7828f40 1 -- 172.21.15.32:0/617165089 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:85 12.1d 12:b871830b:::605.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e118) -- 0x55b2d45e52b0 con 0x55b2d4507770 2024-08-03T20:59:09.495 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.493+0000 7f7debe00640 1 -- 172.21.15.32:0/617165089 <== osd.0 v2:172.21.15.32:6816/3066156116 44 ==== osd_op_reply(85 605.00000000 [create] v118'195 uv194 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f7ddc00e4b0 con 0x55b2d4507770 2024-08-03T20:59:09.495 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.493+0000 7f7df7828f40 1 -- 172.21.15.32:0/617165089 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:86 12.1d 12:b871830b:::605.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e118) -- 0x55b2d45e52b0 con 0x55b2d4507770 2024-08-03T20:59:09.496 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.495+0000 7f7debe00640 1 -- 172.21.15.32:0/617165089 <== osd.0 v2:172.21.15.32:6816/3066156116 45 ==== osd_op_reply(86 605.00000000 [delete] v118'196 uv196 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f7ddc00e4b0 con 0x55b2d4507770 2024-08-03T20:59:09.496 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.495+0000 7f7df7828f40 1 -- 172.21.15.32:0/617165089 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:87 12.1d 12:b871830b:::605.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e118) -- 0x55b2d45e52b0 con 0x55b2d4507770 2024-08-03T20:59:09.497 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.496+0000 7f7debe00640 1 -- 172.21.15.32:0/617165089 <== osd.0 v2:172.21.15.32:6816/3066156116 46 ==== osd_op_reply(87 605.00000000 [create] v118'197 uv197 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f7ddc00e4b0 con 0x55b2d4507770 2024-08-03T20:59:09.497 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::605.00000000:head# 2024-08-03T20:59:09.498 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.496+0000 7f7df7828f40 1 -- 172.21.15.32:0/617165089 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:88 12.1d 12:b871830b:::605.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e118) -- 0x55b2d45e62c0 con 0x55b2d4507770 2024-08-03T20:59:09.499 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.497+0000 7f7debe00640 1 -- 172.21.15.32:0/617165089 <== osd.0 v2:172.21.15.32:6816/3066156116 47 ==== osd_op_reply(88 605.00000000 [setxattr (30)] v118'198 uv198 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f7ddc00e4b0 con 0x55b2d4507770 2024-08-03T20:59:09.499 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.497+0000 7f7df7828f40 1 -- 172.21.15.32:0/617165089 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:89 12.1d 12:b871830b:::605.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e118) -- 0x55b2d45e62c0 con 0x55b2d4507770 2024-08-03T20:59:09.500 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.498+0000 7f7debe00640 1 -- 172.21.15.32:0/617165089 <== osd.0 v2:172.21.15.32:6816/3066156116 48 ==== osd_op_reply(89 605.00000000 [setxattr (62)] v118'199 uv199 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f7ddc00e4b0 con 0x55b2d4507770 2024-08-03T20:59:09.500 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.499+0000 7f7df7828f40 1 -- 172.21.15.32:0/617165089 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:90 12.1d 12:b871830b:::605.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e118) -- 0x55b2d45e52b0 con 0x55b2d4507770 2024-08-03T20:59:09.501 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.500+0000 7f7debe00640 1 -- 172.21.15.32:0/617165089 <== osd.0 v2:172.21.15.32:6816/3066156116 49 ==== osd_op_reply(90 605.00000000 [omap-set-header] v118'200 uv200 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f7ddc00e4b0 con 0x55b2d4507770 2024-08-03T20:59:09.501 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.500+0000 7f7df7828f40 1 -- 172.21.15.32:0/617165089 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:91 12.13 12:c91d4a1d:::mds0_inotable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e118) -- 0x55b2d45e52b0 con 0x55b2d45d9fa0 2024-08-03T20:59:09.503 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.501+0000 7f7debe00640 1 -- 172.21.15.32:0/617165089 <== osd.1 v2:172.21.15.32:6824/1551241759 5 ==== osd_op_reply(91 mds0_inotable [create] v118'325 uv318 ondisk = -17 ((17) File exists)) ==== 157+0+0 (crc 0 0 0) 0x7f7ddc00e4b0 con 0x55b2d45d9fa0 2024-08-03T20:59:09.503 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.501+0000 7f7df7828f40 1 -- 172.21.15.32:0/617165089 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:92 12.13 12:c91d4a1d:::mds0_inotable:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e118) -- 0x55b2d45e52b0 con 0x55b2d45d9fa0 2024-08-03T20:59:09.504 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.503+0000 7f7debe00640 1 -- 172.21.15.32:0/617165089 <== osd.1 v2:172.21.15.32:6824/1551241759 6 ==== osd_op_reply(92 mds0_inotable [delete] v118'326 uv326 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f7ddc00e4b0 con 0x55b2d45d9fa0 2024-08-03T20:59:09.504 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.503+0000 7f7df7828f40 1 -- 172.21.15.32:0/617165089 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:93 12.13 12:c91d4a1d:::mds0_inotable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e118) -- 0x55b2d45e52b0 con 0x55b2d45d9fa0 2024-08-03T20:59:09.505 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.504+0000 7f7debe00640 1 -- 172.21.15.32:0/617165089 <== osd.1 v2:172.21.15.32:6824/1551241759 7 ==== osd_op_reply(93 mds0_inotable [create] v118'327 uv327 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f7ddc00e4b0 con 0x55b2d45d9fa0 2024-08-03T20:59:09.506 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds0_inotable:head# 2024-08-03T20:59:09.506 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.504+0000 7f7df7828f40 1 -- 172.21.15.32:0/617165089 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:94 12.13 12:c91d4a1d:::mds0_inotable:head [write 0~34 in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e118) -- 0x55b2d45e52b0 con 0x55b2d45d9fa0 2024-08-03T20:59:09.507 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.506+0000 7f7debe00640 1 -- 172.21.15.32:0/617165089 <== osd.1 v2:172.21.15.32:6824/1551241759 8 ==== osd_op_reply(94 mds0_inotable [write 0~34] v118'328 uv328 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f7ddc00e4b0 con 0x55b2d45d9fa0 2024-08-03T20:59:09.507 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.506+0000 7f7df7828f40 1 -- 172.21.15.32:0/617165089 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:95 12.13 12:c91d4a1d:::mds0_inotable:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e118) -- 0x55b2d45e52b0 con 0x55b2d45d9fa0 2024-08-03T20:59:09.508 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.507+0000 7f7debe00640 1 -- 172.21.15.32:0/617165089 <== osd.1 v2:172.21.15.32:6824/1551241759 9 ==== osd_op_reply(95 mds0_inotable [omap-set-header] v118'329 uv329 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f7ddc00e4b0 con 0x55b2d45d9fa0 2024-08-03T20:59:09.508 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.507+0000 7f7df7828f40 1 -- 172.21.15.32:0/617165089 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:96 12.13 12:cd5a64a3:::100.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e118) -- 0x55b2d45e52b0 con 0x55b2d45d9fa0 2024-08-03T20:59:09.510 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.508+0000 7f7debe00640 1 -- 172.21.15.32:0/617165089 <== osd.1 v2:172.21.15.32:6824/1551241759 10 ==== osd_op_reply(96 100.00000000 [create] v118'330 uv324 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f7ddc00e4b0 con 0x55b2d45d9fa0 2024-08-03T20:59:09.510 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.508+0000 7f7df7828f40 1 -- 172.21.15.32:0/617165089 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:97 12.13 12:cd5a64a3:::100.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e118) -- 0x55b2d45e52b0 con 0x55b2d45d9fa0 2024-08-03T20:59:09.511 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.510+0000 7f7debe00640 1 -- 172.21.15.32:0/617165089 <== osd.1 v2:172.21.15.32:6824/1551241759 11 ==== osd_op_reply(97 100.00000000 [delete] v118'331 uv331 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f7ddc00e4b0 con 0x55b2d45d9fa0 2024-08-03T20:59:09.511 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.510+0000 7f7df7828f40 1 -- 172.21.15.32:0/617165089 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:98 12.13 12:cd5a64a3:::100.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e118) -- 0x55b2d45e52b0 con 0x55b2d45d9fa0 2024-08-03T20:59:09.512 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.511+0000 7f7debe00640 1 -- 172.21.15.32:0/617165089 <== osd.1 v2:172.21.15.32:6824/1551241759 12 ==== osd_op_reply(98 100.00000000 [create] v118'332 uv332 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f7ddc00e4b0 con 0x55b2d45d9fa0 2024-08-03T20:59:09.512 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::100.00000000:head# 2024-08-03T20:59:09.513 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.511+0000 7f7df7828f40 1 -- 172.21.15.32:0/617165089 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:99 12.13 12:cd5a64a3:::100.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e118) -- 0x55b2d45e52b0 con 0x55b2d45d9fa0 2024-08-03T20:59:09.513 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.512+0000 7f7debe00640 1 -- 172.21.15.32:0/617165089 <== osd.1 v2:172.21.15.32:6824/1551241759 13 ==== osd_op_reply(99 100.00000000 [omap-set-header] v118'333 uv333 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f7ddc00e4b0 con 0x55b2d45d9fa0 2024-08-03T20:59:09.514 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.512+0000 7f7df7828f40 1 -- 172.21.15.32:0/617165089 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:100 12.13 12:cd5a64a3:::100.00000000:head [omap-set-vals in=5194b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e118) -- 0x55b2d45e80c0 con 0x55b2d45d9fa0 2024-08-03T20:59:09.515 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.514+0000 7f7debe00640 1 -- 172.21.15.32:0/617165089 <== osd.1 v2:172.21.15.32:6824/1551241759 14 ==== osd_op_reply(100 100.00000000 [omap-set-vals] v118'334 uv334 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f7ddc00e4b0 con 0x55b2d45d9fa0 2024-08-03T20:59:09.515 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.514+0000 7f7df7828f40 1 --2- 172.21.15.32:0/617165089 >> [v2:172.21.15.32:6800/9088303,v1:172.21.15.32:6801/9088303] conn(0x55b2d45e5de0 0x55b2d45e8720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:59:09.516 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.514+0000 7f7df7828f40 1 -- 172.21.15.32:0/617165089 --> [v2:172.21.15.32:6800/9088303,v1:172.21.15.32:6801/9088303] -- osd_op(unknown.0.0:101 12.b 12:d0630e4c:::mds0_sessionmap:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e118) -- 0x55b2d45e8200 con 0x55b2d45e5de0 2024-08-03T20:59:09.516 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.514+0000 7f7debe00640 1 --2- 172.21.15.32:0/617165089 >> [v2:172.21.15.32:6800/9088303,v1:172.21.15.32:6801/9088303] conn(0x55b2d45e5de0 0x55b2d45e8720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:59:09.516 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.515+0000 7f7debe00640 1 --2- 172.21.15.32:0/617165089 >> [v2:172.21.15.32:6800/9088303,v1:172.21.15.32:6801/9088303] conn(0x55b2d45e5de0 0x55b2d45e8720 crc :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:59:09.517 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.516+0000 7f7debe00640 1 -- 172.21.15.32:0/617165089 <== osd.3 v2:172.21.15.32:6800/9088303 1 ==== osd_op_reply(101 mds0_sessionmap [create] v118'142 uv140 ondisk = -17 ((17) File exists)) ==== 159+0+0 (crc 0 0 0) 0x7f7ddc00e4b0 con 0x55b2d45e5de0 2024-08-03T20:59:09.517 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.516+0000 7f7df7828f40 1 -- 172.21.15.32:0/617165089 --> [v2:172.21.15.32:6800/9088303,v1:172.21.15.32:6801/9088303] -- osd_op(unknown.0.0:102 12.b 12:d0630e4c:::mds0_sessionmap:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e118) -- 0x55b2d45e8200 con 0x55b2d45e5de0 2024-08-03T20:59:09.519 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.517+0000 7f7debe00640 1 -- 172.21.15.32:0/617165089 <== osd.3 v2:172.21.15.32:6800/9088303 2 ==== osd_op_reply(102 mds0_sessionmap [delete] v118'143 uv143 ondisk = 0) ==== 159+0+0 (crc 0 0 0) 0x7f7ddc00e4b0 con 0x55b2d45e5de0 2024-08-03T20:59:09.519 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.518+0000 7f7df7828f40 1 -- 172.21.15.32:0/617165089 --> [v2:172.21.15.32:6800/9088303,v1:172.21.15.32:6801/9088303] -- osd_op(unknown.0.0:103 12.b 12:d0630e4c:::mds0_sessionmap:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e118) -- 0x55b2d45e8200 con 0x55b2d45e5de0 2024-08-03T20:59:09.520 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.519+0000 7f7debe00640 1 -- 172.21.15.32:0/617165089 <== osd.3 v2:172.21.15.32:6800/9088303 3 ==== osd_op_reply(103 mds0_sessionmap [create] v118'144 uv144 ondisk = 0) ==== 159+0+0 (crc 0 0 0) 0x7f7ddc00e4b0 con 0x55b2d45e5de0 2024-08-03T20:59:09.520 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds0_sessionmap:head# 2024-08-03T20:59:09.521 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.519+0000 7f7df7828f40 1 -- 172.21.15.32:0/617165089 --> [v2:172.21.15.32:6800/9088303,v1:172.21.15.32:6801/9088303] -- osd_op(unknown.0.0:104 12.b 12:d0630e4c:::mds0_sessionmap:head [omap-set-header in=14b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e118) -- 0x55b2d45e8200 con 0x55b2d45e5de0 2024-08-03T20:59:09.522 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.520+0000 7f7debe00640 1 -- 172.21.15.32:0/617165089 <== osd.3 v2:172.21.15.32:6800/9088303 4 ==== osd_op_reply(104 mds0_sessionmap [omap-set-header] v118'145 uv145 ondisk = 0) ==== 159+0+0 (crc 0 0 0) 0x7f7ddc00e4b0 con 0x55b2d45e5de0 2024-08-03T20:59:09.522 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.520+0000 7f7df7828f40 1 -- 172.21.15.32:0/617165089 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:105 12.7 12:e0b41d06:::609.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e118) -- 0x55b2d45e8200 con 0x55b2d45d9fa0 2024-08-03T20:59:09.524 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.522+0000 7f7debe00640 1 -- 172.21.15.32:0/617165089 <== osd.1 v2:172.21.15.32:6824/1551241759 15 ==== osd_op_reply(105 609.00000000 [create] v118'195 uv194 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f7ddc00e4b0 con 0x55b2d45d9fa0 2024-08-03T20:59:09.524 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.522+0000 7f7df7828f40 1 -- 172.21.15.32:0/617165089 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:106 12.7 12:e0b41d06:::609.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e118) -- 0x55b2d45e8200 con 0x55b2d45d9fa0 2024-08-03T20:59:09.525 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.524+0000 7f7debe00640 1 -- 172.21.15.32:0/617165089 <== osd.1 v2:172.21.15.32:6824/1551241759 16 ==== osd_op_reply(106 609.00000000 [delete] v118'196 uv196 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f7ddc00e4b0 con 0x55b2d45d9fa0 2024-08-03T20:59:09.525 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.524+0000 7f7df7828f40 1 -- 172.21.15.32:0/617165089 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:107 12.7 12:e0b41d06:::609.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e118) -- 0x55b2d45e8200 con 0x55b2d45d9fa0 2024-08-03T20:59:09.526 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.525+0000 7f7debe00640 1 -- 172.21.15.32:0/617165089 <== osd.1 v2:172.21.15.32:6824/1551241759 17 ==== osd_op_reply(107 609.00000000 [create] v118'197 uv197 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f7ddc00e4b0 con 0x55b2d45d9fa0 2024-08-03T20:59:09.527 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::609.00000000:head# 2024-08-03T20:59:09.527 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.525+0000 7f7df7828f40 1 -- 172.21.15.32:0/617165089 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:108 12.7 12:e0b41d06:::609.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e118) -- 0x55b2d45e8200 con 0x55b2d45d9fa0 2024-08-03T20:59:09.528 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.527+0000 7f7debe00640 1 -- 172.21.15.32:0/617165089 <== osd.1 v2:172.21.15.32:6824/1551241759 18 ==== osd_op_reply(108 609.00000000 [setxattr (30)] v118'198 uv198 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f7ddc00e4b0 con 0x55b2d45d9fa0 2024-08-03T20:59:09.528 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.527+0000 7f7df7828f40 1 -- 172.21.15.32:0/617165089 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:109 12.7 12:e0b41d06:::609.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e118) -- 0x55b2d45e8200 con 0x55b2d45d9fa0 2024-08-03T20:59:09.529 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.528+0000 7f7debe00640 1 -- 172.21.15.32:0/617165089 <== osd.1 v2:172.21.15.32:6824/1551241759 19 ==== osd_op_reply(109 609.00000000 [setxattr (62)] v118'199 uv199 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f7ddc00e4b0 con 0x55b2d45d9fa0 2024-08-03T20:59:09.529 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.528+0000 7f7df7828f40 1 -- 172.21.15.32:0/617165089 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:110 12.7 12:e0b41d06:::609.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e118) -- 0x55b2d45e8200 con 0x55b2d45d9fa0 2024-08-03T20:59:09.531 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.529+0000 7f7debe00640 1 -- 172.21.15.32:0/617165089 <== osd.1 v2:172.21.15.32:6824/1551241759 20 ==== osd_op_reply(110 609.00000000 [omap-set-header] v118'200 uv200 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f7ddc00e4b0 con 0x55b2d45d9fa0 2024-08-03T20:59:09.531 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.529+0000 7f7df7828f40 1 -- 172.21.15.32:0/617165089 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:111 12.f 12:f1f69726:::400.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e118) -- 0x55b2d45e8200 con 0x55b2d45dd400 2024-08-03T20:59:09.532 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.531+0000 7f7df1400640 1 -- 172.21.15.32:0/617165089 <== osd.6 v2:172.21.15.118:6825/801947386 12 ==== osd_op_reply(111 400.00000000 [create] v118'166 uv165 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f7dec019030 con 0x55b2d45dd400 2024-08-03T20:59:09.533 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.531+0000 7f7df7828f40 1 -- 172.21.15.32:0/617165089 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:112 12.f 12:f1f69726:::400.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e118) -- 0x55b2d45e8200 con 0x55b2d45dd400 2024-08-03T20:59:09.534 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.533+0000 7f7df1400640 1 -- 172.21.15.32:0/617165089 <== osd.6 v2:172.21.15.118:6825/801947386 13 ==== osd_op_reply(112 400.00000000 [delete] v118'167 uv167 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f7dec019030 con 0x55b2d45dd400 2024-08-03T20:59:09.534 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.533+0000 7f7df7828f40 1 -- 172.21.15.32:0/617165089 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:113 12.f 12:f1f69726:::400.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e118) -- 0x55b2d45e8200 con 0x55b2d45dd400 2024-08-03T20:59:09.536 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.534+0000 7f7df1400640 1 -- 172.21.15.32:0/617165089 <== osd.6 v2:172.21.15.118:6825/801947386 14 ==== osd_op_reply(113 400.00000000 [create] v118'168 uv168 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f7dec019030 con 0x55b2d45dd400 2024-08-03T20:59:09.536 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::400.00000000:head# 2024-08-03T20:59:09.536 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.534+0000 7f7df7828f40 1 -- 172.21.15.32:0/617165089 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:114 12.f 12:f1f69726:::400.00000000:head [write 0~22 in=22b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e118) -- 0x55b2d45e8200 con 0x55b2d45dd400 2024-08-03T20:59:09.537 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.536+0000 7f7df1400640 1 -- 172.21.15.32:0/617165089 <== osd.6 v2:172.21.15.118:6825/801947386 15 ==== osd_op_reply(114 400.00000000 [write 0~22] v118'169 uv169 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f7dec019030 con 0x55b2d45dd400 2024-08-03T20:59:09.538 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.536+0000 7f7df7828f40 1 -- 172.21.15.32:0/617165089 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:115 12.f 12:f1f69726:::400.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e118) -- 0x55b2d45e8200 con 0x55b2d45dd400 2024-08-03T20:59:09.539 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.538+0000 7f7df1400640 1 -- 172.21.15.32:0/617165089 <== osd.6 v2:172.21.15.118:6825/801947386 16 ==== osd_op_reply(115 400.00000000 [omap-set-header] v118'170 uv170 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f7dec019030 con 0x55b2d45dd400 2024-08-03T20:59:09.539 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.538+0000 7f7df7828f40 1 -- 172.21.15.32:0/617165089 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:116 12.1f 12:fe07bba1:::100.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e118) -- 0x55b2d45e8200 con 0x55b2d45dd400 2024-08-03T20:59:09.541 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.540+0000 7f7df1400640 1 -- 172.21.15.32:0/617165089 <== osd.6 v2:172.21.15.118:6825/801947386 17 ==== osd_op_reply(116 100.00000000.inode [create] v118'402 uv393 ondisk = -17 ((17) File exists)) ==== 162+0+0 (crc 0 0 0) 0x7f7dec019030 con 0x55b2d45dd400 2024-08-03T20:59:09.541 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.540+0000 7f7df7828f40 1 -- 172.21.15.32:0/617165089 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:117 12.1f 12:fe07bba1:::100.00000000.inode:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e118) -- 0x55b2d45e8200 con 0x55b2d45dd400 2024-08-03T20:59:09.543 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.542+0000 7f7df1400640 1 -- 172.21.15.32:0/617165089 <== osd.6 v2:172.21.15.118:6825/801947386 18 ==== osd_op_reply(117 100.00000000.inode [delete] v118'403 uv403 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7f7dec019030 con 0x55b2d45dd400 2024-08-03T20:59:09.543 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.542+0000 7f7df7828f40 1 -- 172.21.15.32:0/617165089 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:118 12.1f 12:fe07bba1:::100.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e118) -- 0x55b2d45e8200 con 0x55b2d45dd400 2024-08-03T20:59:09.545 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.543+0000 7f7df1400640 1 -- 172.21.15.32:0/617165089 <== osd.6 v2:172.21.15.118:6825/801947386 19 ==== osd_op_reply(118 100.00000000.inode [create] v118'404 uv404 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7f7dec019030 con 0x55b2d45dd400 2024-08-03T20:59:09.545 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::100.00000000.inode:head# 2024-08-03T20:59:09.545 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.543+0000 7f7df7828f40 1 -- 172.21.15.32:0/617165089 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:119 12.1f 12:fe07bba1:::100.00000000.inode:head [write 0~582 in=582b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e118) -- 0x55b2d45e8200 con 0x55b2d45dd400 2024-08-03T20:59:09.546 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.545+0000 7f7df1400640 1 -- 172.21.15.32:0/617165089 <== osd.6 v2:172.21.15.118:6825/801947386 20 ==== osd_op_reply(119 100.00000000.inode [write 0~582] v118'405 uv405 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7f7dec019030 con 0x55b2d45dd400 2024-08-03T20:59:09.546 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.545+0000 7f7df7828f40 1 -- 172.21.15.32:0/617165089 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:120 12.1f 12:fe07bba1:::100.00000000.inode:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e118) -- 0x55b2d45e8200 con 0x55b2d45dd400 2024-08-03T20:59:09.548 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.546+0000 7f7df1400640 1 -- 172.21.15.32:0/617165089 <== osd.6 v2:172.21.15.118:6825/801947386 21 ==== osd_op_reply(120 100.00000000.inode [omap-set-header] v118'406 uv406 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7f7dec019030 con 0x55b2d45dd400 2024-08-03T20:59:09.548 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.546+0000 7f7df7828f40 1 -- 172.21.15.32:0/617165089 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:121 12.1f 12:ff5b34d6:::1.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e118) -- 0x55b2d45e8200 con 0x55b2d45dd400 2024-08-03T20:59:09.549 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.548+0000 7f7df1400640 1 -- 172.21.15.32:0/617165089 <== osd.6 v2:172.21.15.118:6825/801947386 22 ==== osd_op_reply(121 1.00000000 [create] v118'407 uv400 ondisk = -17 ((17) File exists)) ==== 154+0+0 (crc 0 0 0) 0x7f7dec019030 con 0x55b2d45dd400 2024-08-03T20:59:09.550 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.548+0000 7f7df7828f40 1 -- 172.21.15.32:0/617165089 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:122 12.1f 12:ff5b34d6:::1.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e118) -- 0x55b2d45e8200 con 0x55b2d45dd400 2024-08-03T20:59:09.551 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.550+0000 7f7df1400640 1 -- 172.21.15.32:0/617165089 <== osd.6 v2:172.21.15.118:6825/801947386 23 ==== osd_op_reply(122 1.00000000 [delete] v118'408 uv408 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7f7dec019030 con 0x55b2d45dd400 2024-08-03T20:59:09.551 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.550+0000 7f7df7828f40 1 -- 172.21.15.32:0/617165089 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:123 12.1f 12:ff5b34d6:::1.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e118) -- 0x55b2d45e8200 con 0x55b2d45dd400 2024-08-03T20:59:09.552 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::1.00000000:head# 2024-08-03T20:59:09.553 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.551+0000 7f7df1400640 1 -- 172.21.15.32:0/617165089 <== osd.6 v2:172.21.15.118:6825/801947386 24 ==== osd_op_reply(123 1.00000000 [create] v118'409 uv409 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7f7dec019030 con 0x55b2d45dd400 2024-08-03T20:59:09.553 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.551+0000 7f7df7828f40 1 -- 172.21.15.32:0/617165089 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:124 12.1f 12:ff5b34d6:::1.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e118) -- 0x55b2d45e8200 con 0x55b2d45dd400 2024-08-03T20:59:09.554 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.552+0000 7f7df1400640 1 -- 172.21.15.32:0/617165089 <== osd.6 v2:172.21.15.118:6825/801947386 25 ==== osd_op_reply(124 1.00000000 [setxattr (30)] v118'410 uv410 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7f7dec019030 con 0x55b2d45dd400 2024-08-03T20:59:09.554 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.552+0000 7f7df7828f40 1 -- 172.21.15.32:0/617165089 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:125 12.1f 12:ff5b34d6:::1.00000000:head [setxattr parent (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e118) -- 0x55b2d45e8200 con 0x55b2d45dd400 2024-08-03T20:59:09.555 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.554+0000 7f7df1400640 1 -- 172.21.15.32:0/617165089 <== osd.6 v2:172.21.15.118:6825/801947386 26 ==== osd_op_reply(125 1.00000000 [setxattr (30)] v118'411 uv411 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7f7dec019030 con 0x55b2d45dd400 2024-08-03T20:59:09.555 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.554+0000 7f7df7828f40 1 -- 172.21.15.32:0/617165089 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:126 12.1f 12:ff5b34d6:::1.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e118) -- 0x55b2d45e8200 con 0x55b2d45dd400 2024-08-03T20:59:09.556 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.555+0000 7f7df1400640 1 -- 172.21.15.32:0/617165089 <== osd.6 v2:172.21.15.118:6825/801947386 27 ==== osd_op_reply(126 1.00000000 [omap-set-header] v118'412 uv412 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7f7dec019030 con 0x55b2d45dd400 2024-08-03T20:59:09.556 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.555+0000 7f7df7828f40 1 -- 172.21.15.32:0/617165089 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:127 12.1f 12:ff5b34d6:::1.00000000:head [omap-set-vals in=584b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e118) -- 0x55b2d45e9160 con 0x55b2d45dd400 2024-08-03T20:59:09.559 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.557+0000 7f7df1400640 1 -- 172.21.15.32:0/617165089 <== osd.6 v2:172.21.15.118:6825/801947386 28 ==== osd_op_reply(127 1.00000000 [omap-set-vals] v118'413 uv413 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7f7dec019030 con 0x55b2d45dd400 2024-08-03T20:59:09.559 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.557+0000 7f7df7828f40 1 -- 172.21.15.32:0/617165089 >> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] conn(0x55b2d4507770 msgr2=0x55b2d4499020 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:59:09.559 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.557+0000 7f7df7828f40 1 --2- 172.21.15.32:0/617165089 >> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] conn(0x55b2d4507770 0x55b2d4499020 crc :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:59:09.559 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.557+0000 7f7df7828f40 1 -- 172.21.15.32:0/617165089 >> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] conn(0x55b2d45d9fa0 msgr2=0x55b2d45da3b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:59:09.559 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.557+0000 7f7df7828f40 1 --2- 172.21.15.32:0/617165089 >> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] conn(0x55b2d45d9fa0 0x55b2d45da3b0 crc :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:59:09.559 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.557+0000 7f7df7828f40 1 -- 172.21.15.32:0/617165089 >> [v2:172.21.15.32:6800/9088303,v1:172.21.15.32:6801/9088303] conn(0x55b2d45e5de0 msgr2=0x55b2d45e8720 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:59:09.560 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.557+0000 7f7df7828f40 1 --2- 172.21.15.32:0/617165089 >> [v2:172.21.15.32:6800/9088303,v1:172.21.15.32:6801/9088303] conn(0x55b2d45e5de0 0x55b2d45e8720 crc :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:59:09.560 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.557+0000 7f7df7828f40 1 -- 172.21.15.32:0/617165089 >> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] conn(0x55b2d45d5c80 msgr2=0x55b2d45d80a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:59:09.560 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.557+0000 7f7df7828f40 1 --2- 172.21.15.32:0/617165089 >> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] conn(0x55b2d45d5c80 0x55b2d45d80a0 crc :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:59:09.560 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.557+0000 7f7df1400640 1 -- 172.21.15.32:0/617165089 reap_dead start 2024-08-03T20:59:09.560 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.557+0000 7f7df7828f40 1 -- 172.21.15.32:0/617165089 >> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] conn(0x55b2d45dd400 msgr2=0x55b2d45dd810 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:59:09.560 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.557+0000 7f7df7828f40 1 --2- 172.21.15.32:0/617165089 >> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] conn(0x55b2d45dd400 0x55b2d45dd810 crc :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:59:09.560 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.557+0000 7f7df1400640 1 -- 172.21.15.32:0/617165089 reap_dead start 2024-08-03T20:59:09.560 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.557+0000 7f7df7828f40 1 -- 172.21.15.32:0/617165089 >> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] conn(0x55b2d45e2900 msgr2=0x55b2d45e4d20 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:59:09.560 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.557+0000 7f7df7828f40 1 --2- 172.21.15.32:0/617165089 >> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] conn(0x55b2d45e2900 0x55b2d45e4d20 crc :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:59:09.560 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.557+0000 7f7df7828f40 1 -- 172.21.15.32:0/617165089 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f7dd0075a20 msgr2=0x7f7dd0077f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:59:09.560 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.557+0000 7f7df7828f40 1 --2- 172.21.15.32:0/617165089 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f7dd0075a20 0x7f7dd0077f30 secure :-1 s=READY pgs=764 cs=0 l=1 rev1=1 crypto rx=0x7f7de4046660 tx=0x7f7de4058040 comp rx=0 tx=0).stop 2024-08-03T20:59:09.561 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.557+0000 7f7df7828f40 1 -- 172.21.15.32:0/617165089 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b2d452f3f0 msgr2=0x55b2d452cd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:59:09.561 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.557+0000 7f7df7828f40 1 --2- 172.21.15.32:0/617165089 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b2d452f3f0 0x55b2d452cd60 secure :-1 s=READY pgs=1897 cs=0 l=1 rev1=1 crypto rx=0x7f7dec0041a0 tx=0x7f7dec00c520 comp rx=0 tx=0).stop 2024-08-03T20:59:09.561 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.557+0000 7f7df7828f40 1 -- 172.21.15.32:0/617165089 shutdown_connections 2024-08-03T20:59:09.561 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.557+0000 7f7df7828f40 1 --2- 172.21.15.32:0/617165089 >> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] conn(0x55b2d45e2900 0x55b2d45e4d20 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:59:09.561 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.557+0000 7f7df7828f40 1 --2- 172.21.15.32:0/617165089 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f7dd0075a20 0x7f7dd0077f30 unknown :-1 s=CLOSED pgs=764 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:59:09.561 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.557+0000 7f7df7828f40 1 --2- 172.21.15.32:0/617165089 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b2d452f3f0 0x55b2d452cd60 unknown :-1 s=CLOSED pgs=1897 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:59:09.561 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.557+0000 7f7df7828f40 1 -- 172.21.15.32:0/617165089 >> 172.21.15.32:0/617165089 conn(0x55b2d437aac0 msgr2=0x55b2d44a1080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:59:09.561 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.557+0000 7f7df7828f40 1 -- 172.21.15.32:0/617165089 shutdown_connections 2024-08-03T20:59:09.561 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.557+0000 7f7df7828f40 1 -- 172.21.15.32:0/617165089 wait complete. 2024-08-03T20:59:09.572 DEBUG:teuthology.orchestra.run.smithi032:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata setomapval 1.00000000 subdir_head deadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeef 2024-08-03T20:59:09.717 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.715+0000 7f4192c41f40 1 Processor -- start 2024-08-03T20:59:09.717 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.715+0000 7f4192c41f40 1 -- start start 2024-08-03T20:59:09.721 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.716+0000 7f4192c41f40 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55816dff6740 0x55816dff6bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:59:09.721 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.716+0000 7f4192c41f40 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55816e0683f0 0x55816e064bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:59:09.721 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.716+0000 7f4192c41f40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55816e083d90 0x55816e065110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:59:09.721 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.716+0000 7f4192c41f40 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55816defbe30 con 0x55816dff6740 2024-08-03T20:59:09.721 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.716+0000 7f4192c41f40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x55816df59bf0 con 0x55816e083d90 2024-08-03T20:59:09.721 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.716+0000 7f4192c41f40 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x55816dff2bd0 con 0x55816e0683f0 2024-08-03T20:59:09.721 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.716+0000 7f418be00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55816dff6740 0x55816dff6bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:59:09.722 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.717+0000 7f418be00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55816dff6740 0x55816dff6bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:49474/0 (socket says 172.21.15.32:49474) 2024-08-03T20:59:09.722 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.717+0000 7f418be00640 1 -- 172.21.15.32:0/2259909994 learned_addr learned my addr 172.21.15.32:0/2259909994 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:59:09.722 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.717+0000 7f418b400640 1 --2- 172.21.15.32:0/2259909994 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55816e0683f0 0x55816e064bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:59:09.722 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.717+0000 7f418c800640 1 --2- 172.21.15.32:0/2259909994 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55816e083d90 0x55816e065110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:59:09.722 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.717+0000 7f418be00640 1 -- 172.21.15.32:0/2259909994 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55816e0683f0 msgr2=0x55816e064bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:59:09.722 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.717+0000 7f418be00640 1 --2- 172.21.15.32:0/2259909994 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55816e0683f0 0x55816e064bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:59:09.722 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.717+0000 7f418be00640 1 -- 172.21.15.32:0/2259909994 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55816e083d90 msgr2=0x55816e065110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:59:09.722 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.717+0000 7f418be00640 1 --2- 172.21.15.32:0/2259909994 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55816e083d90 0x55816e065110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:59:09.722 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.717+0000 7f418be00640 1 -- 172.21.15.32:0/2259909994 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55816dfad650 con 0x55816dff6740 2024-08-03T20:59:09.722 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.717+0000 7f418be00640 1 --2- 172.21.15.32:0/2259909994 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55816dff6740 0x55816dff6bc0 secure :-1 s=READY pgs=1900 cs=0 l=1 rev1=1 crypto rx=0x7f417c000c00 tx=0x7f417c002fb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5dc1db834d5f2548 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:59:09.723 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.718+0000 7f418a000640 1 -- 172.21.15.32:0/2259909994 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f417c04dd50 con 0x55816dff6740 2024-08-03T20:59:09.723 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.718+0000 7f418a000640 1 -- 172.21.15.32:0/2259909994 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f417c056470 con 0x55816dff6740 2024-08-03T20:59:09.723 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.718+0000 7f418a000640 1 -- 172.21.15.32:0/2259909994 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f417c056860 con 0x55816dff6740 2024-08-03T20:59:09.723 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.718+0000 7f4192c41f40 1 -- 172.21.15.32:0/2259909994 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55816dff6740 msgr2=0x55816dff6bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:59:09.723 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.718+0000 7f4192c41f40 1 --2- 172.21.15.32:0/2259909994 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55816dff6740 0x55816dff6bc0 secure :-1 s=READY pgs=1900 cs=0 l=1 rev1=1 crypto rx=0x7f417c000c00 tx=0x7f417c002fb0 comp rx=0 tx=0).stop 2024-08-03T20:59:09.723 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.718+0000 7f4192c41f40 1 -- 172.21.15.32:0/2259909994 shutdown_connections 2024-08-03T20:59:09.723 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.718+0000 7f4192c41f40 1 --2- 172.21.15.32:0/2259909994 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55816e083d90 0x55816e065110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:59:09.723 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.718+0000 7f4192c41f40 1 --2- 172.21.15.32:0/2259909994 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55816e0683f0 0x55816e064bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:59:09.723 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.718+0000 7f4192c41f40 1 --2- 172.21.15.32:0/2259909994 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55816dff6740 0x55816dff6bc0 unknown :-1 s=CLOSED pgs=1900 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:59:09.723 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.718+0000 7f4192c41f40 1 -- 172.21.15.32:0/2259909994 >> 172.21.15.32:0/2259909994 conn(0x55816decdac0 msgr2=0x55816e10d740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:59:09.723 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.718+0000 7f4192c41f40 1 -- 172.21.15.32:0/2259909994 shutdown_connections 2024-08-03T20:59:09.724 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.718+0000 7f4192c41f40 1 -- 172.21.15.32:0/2259909994 wait complete. 2024-08-03T20:59:09.724 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.719+0000 7f4192c41f40 1 Processor -- start 2024-08-03T20:59:09.724 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.720+0000 7f4192c41f40 1 -- start start 2024-08-03T20:59:09.725 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.721+0000 7f4192c41f40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55816dff6740 0x55816e064bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:59:09.726 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.721+0000 7f4192c41f40 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55816e0683f0 0x55816e065110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:59:09.726 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.721+0000 7f4192c41f40 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55816e083d90 0x55816e05d700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:59:09.726 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.721+0000 7f4192c41f40 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55816dfaeb80 con 0x55816e0683f0 2024-08-03T20:59:09.726 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.721+0000 7f4192c41f40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x55816e07c5c0 con 0x55816dff6740 2024-08-03T20:59:09.726 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.721+0000 7f4192c41f40 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x55816dff3ae0 con 0x55816e083d90 2024-08-03T20:59:09.726 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.721+0000 7f418b400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55816e0683f0 0x55816e065110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:59:09.726 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.721+0000 7f418be00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55816dff6740 0x55816e064bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:59:09.726 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.721+0000 7f418b400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55816e0683f0 0x55816e065110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:49486/0 (socket says 172.21.15.32:49486) 2024-08-03T20:59:09.726 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.721+0000 7f418c800640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55816e083d90 0x55816e05d700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:59:09.726 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.721+0000 7f418be00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55816dff6740 0x55816e064bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.32:60346/0 (socket says 172.21.15.32:60346) 2024-08-03T20:59:09.726 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.721+0000 7f418be00640 1 -- 172.21.15.32:0/4040124642 learned_addr learned my addr 172.21.15.32:0/4040124642 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:59:09.727 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.722+0000 7f418b400640 1 -- 172.21.15.32:0/4040124642 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55816e083d90 msgr2=0x55816e05d700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:59:09.727 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.722+0000 7f418b400640 1 --2- 172.21.15.32:0/4040124642 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55816e083d90 0x55816e05d700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:59:09.727 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.722+0000 7f418b400640 1 -- 172.21.15.32:0/4040124642 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55816dff6740 msgr2=0x55816e064bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:59:09.727 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.722+0000 7f418b400640 1 --2- 172.21.15.32:0/4040124642 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55816dff6740 0x55816e064bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:59:09.727 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.722+0000 7f418b400640 1 -- 172.21.15.32:0/4040124642 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55816e111a50 con 0x55816e0683f0 2024-08-03T20:59:09.727 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.722+0000 7f418be00640 1 --2- 172.21.15.32:0/4040124642 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55816dff6740 0x55816e064bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-03T20:59:09.727 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.722+0000 7f418b400640 1 --2- 172.21.15.32:0/4040124642 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55816e0683f0 0x55816e065110 secure :-1 s=READY pgs=1901 cs=0 l=1 rev1=1 crypto rx=0x7f41780084c0 tx=0x7f41780085b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:59:09.727 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.723+0000 7f4188c00640 1 -- 172.21.15.32:0/4040124642 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4178016640 con 0x55816e0683f0 2024-08-03T20:59:09.727 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.723+0000 7f4188c00640 1 -- 172.21.15.32:0/4040124642 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f41780167e0 con 0x55816e0683f0 2024-08-03T20:59:09.727 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.723+0000 7f4188c00640 1 -- 172.21.15.32:0/4040124642 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4178016c10 con 0x55816e0683f0 2024-08-03T20:59:09.727 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.723+0000 7f4192c41f40 1 -- 172.21.15.32:0/4040124642 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x55816e05c940 con 0x55816e0683f0 2024-08-03T20:59:09.728 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.723+0000 7f4192c41f40 1 -- 172.21.15.32:0/4040124642 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55816e05cf00 con 0x55816e0683f0 2024-08-03T20:59:09.729 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.725+0000 7f4188c00640 1 -- 172.21.15.32:0/4040124642 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f417800a440 con 0x55816e0683f0 2024-08-03T20:59:09.729 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.726+0000 7f4188c00640 1 --2- 172.21.15.32:0/4040124642 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f416c0759f0 0x7f416c077f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:59:09.729 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.726+0000 7f4188c00640 1 -- 172.21.15.32:0/4040124642 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(118..118 src has 1..118) ==== 8692+0+0 (secure 0 0 0) 0x7f4178098aa0 con 0x55816e0683f0 2024-08-03T20:59:09.729 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.726+0000 7f4192c41f40 1 --2- 172.21.15.32:0/4040124642 >> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] conn(0x55816e05abd0 0x55816e05e150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:59:09.729 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.726+0000 7f4192c41f40 1 -- 172.21.15.32:0/4040124642 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:1 12.1f 12:ff5b34d6:::1.00000000:head [omap-set-vals in=103b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e118) -- 0x55816e126560 con 0x55816e05abd0 2024-08-03T20:59:09.729 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.726+0000 7f418be00640 1 --2- 172.21.15.32:0/4040124642 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f416c0759f0 0x7f416c077f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:59:09.729 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.727+0000 7f418c800640 1 --2- 172.21.15.32:0/4040124642 >> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] conn(0x55816e05abd0 0x55816e05e150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:59:09.729 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.727+0000 7f418be00640 1 --2- 172.21.15.32:0/4040124642 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f416c0759f0 0x7f416c077f00 secure :-1 s=READY pgs=765 cs=0 l=1 rev1=1 crypto rx=0x7f417c04d820 tx=0x7f417c004880 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:59:09.729 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.727+0000 7f418c800640 1 --2- 172.21.15.32:0/4040124642 >> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] conn(0x55816e05abd0 0x55816e05e150 crc :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:59:09.732 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.729+0000 7f418c800640 1 -- 172.21.15.32:0/4040124642 <== osd.6 v2:172.21.15.118:6825/801947386 1 ==== osd_op_reply(1 1.00000000 [omap-set-vals] v118'414 uv414 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7f4184002740 con 0x55816e05abd0 2024-08-03T20:59:09.732 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.729+0000 7f4192c41f40 1 -- 172.21.15.32:0/4040124642 >> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] conn(0x55816e05abd0 msgr2=0x55816e05e150 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:59:09.732 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.729+0000 7f4192c41f40 1 --2- 172.21.15.32:0/4040124642 >> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] conn(0x55816e05abd0 0x55816e05e150 crc :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:59:09.732 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.729+0000 7f4192c41f40 1 -- 172.21.15.32:0/4040124642 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f416c0759f0 msgr2=0x7f416c077f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:59:09.733 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.729+0000 7f4192c41f40 1 --2- 172.21.15.32:0/4040124642 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f416c0759f0 0x7f416c077f00 secure :-1 s=READY pgs=765 cs=0 l=1 rev1=1 crypto rx=0x7f417c04d820 tx=0x7f417c004880 comp rx=0 tx=0).stop 2024-08-03T20:59:09.733 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.729+0000 7f4192c41f40 1 -- 172.21.15.32:0/4040124642 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55816e0683f0 msgr2=0x55816e065110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:59:09.733 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.729+0000 7f4192c41f40 1 --2- 172.21.15.32:0/4040124642 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55816e0683f0 0x55816e065110 secure :-1 s=READY pgs=1901 cs=0 l=1 rev1=1 crypto rx=0x7f41780084c0 tx=0x7f41780085b0 comp rx=0 tx=0).stop 2024-08-03T20:59:09.733 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.729+0000 7f418c800640 1 -- 172.21.15.32:0/4040124642 reap_dead start 2024-08-03T20:59:09.733 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.729+0000 7f4192c41f40 1 -- 172.21.15.32:0/4040124642 shutdown_connections 2024-08-03T20:59:09.733 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.729+0000 7f4192c41f40 1 -- 172.21.15.32:0/4040124642 >> 172.21.15.32:0/4040124642 conn(0x55816decdac0 msgr2=0x55816dfacbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:59:09.734 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.729+0000 7f4192c41f40 1 -- 172.21.15.32:0/4040124642 shutdown_connections 2024-08-03T20:59:09.734 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.730+0000 7f4192c41f40 1 -- 172.21.15.32:0/4040124642 wait complete. 2024-08-03T20:59:09.745 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs joinable true 2024-08-03T20:59:09.977 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.975+0000 7f9228c00640 1 -- 172.21.15.32:0/3475142953 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f92240d13a0 msgr2=0x7f922409ac30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:59:09.977 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.975+0000 7f9228c00640 1 --2- 172.21.15.32:0/3475142953 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f92240d13a0 0x7f922409ac30 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f920c009830 tx=0x7f920c02a380 comp rx=0 tx=0).stop 2024-08-03T20:59:09.977 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.975+0000 7f9228c00640 1 -- 172.21.15.32:0/3475142953 shutdown_connections 2024-08-03T20:59:09.978 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.975+0000 7f9228c00640 1 --2- 172.21.15.32:0/3475142953 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f92240d13a0 0x7f922409ac30 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:59:09.978 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.975+0000 7f9228c00640 1 --2- 172.21.15.32:0/3475142953 >> v2:172.21.15.118:3300/0 conn(0x7f92240cc760 0x7f92240d0e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:59:09.978 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.975+0000 7f9228c00640 1 -- 172.21.15.32:0/3475142953 >> 172.21.15.32:0/3475142953 conn(0x7f92240b3c00 msgr2=0x7f92240b4010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:59:09.978 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.975+0000 7f9228c00640 1 -- 172.21.15.32:0/3475142953 shutdown_connections 2024-08-03T20:59:09.978 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.975+0000 7f9228c00640 1 -- 172.21.15.32:0/3475142953 wait complete. 2024-08-03T20:59:09.979 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.978+0000 7f9228c00640 1 Processor -- start 2024-08-03T20:59:09.979 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.978+0000 7f9228c00640 1 -- start start 2024-08-03T20:59:09.983 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.979+0000 7f9228c00640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f922409b170 0x7f92240b18d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:59:09.983 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.979+0000 7f9228c00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f92240cc760 0x7f92241f80f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:59:09.983 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.979+0000 7f9228c00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f92240d13a0 0x7f92241f8630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:59:09.983 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.979+0000 7f9228c00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f92240b2fb0 con 0x7f92240d13a0 2024-08-03T20:59:09.983 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.979+0000 7f9228c00640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f92240b2e30 con 0x7f92240cc760 2024-08-03T20:59:09.983 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.979+0000 7f9228c00640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7f92240ccb70 con 0x7f922409b170 2024-08-03T20:59:09.983 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.980+0000 7f9222800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f92240d13a0 0x7f92241f8630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:59:09.984 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.980+0000 7f9221e00640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f922409b170 0x7f92240b18d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:59:09.984 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.980+0000 7f9219400640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f92240cc760 0x7f92241f80f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:59:09.984 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.980+0000 7f9221e00640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f922409b170 0x7f92240b18d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3301/0 says I am v2:172.21.15.32:49732/0 (socket says 172.21.15.32:49732) 2024-08-03T20:59:09.984 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.980+0000 7f9221e00640 1 -- 172.21.15.32:0/877350688 learned_addr learned my addr 172.21.15.32:0/877350688 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:59:09.984 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.980+0000 7f9219400640 1 -- 172.21.15.32:0/877350688 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f922409b170 msgr2=0x7f92240b18d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:59:09.984 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.980+0000 7f9219400640 1 --2- 172.21.15.32:0/877350688 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f922409b170 0x7f92240b18d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:59:09.984 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.980+0000 7f9219400640 1 -- 172.21.15.32:0/877350688 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f92240d13a0 msgr2=0x7f92241f8630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:59:09.984 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.980+0000 7f9219400640 1 --2- 172.21.15.32:0/877350688 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f92240d13a0 0x7f92241f8630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:59:09.984 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.980+0000 7f9219400640 1 -- 172.21.15.32:0/877350688 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f92240cb6e0 con 0x7f92240cc760 2024-08-03T20:59:09.984 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.980+0000 7f9222800640 1 --2- 172.21.15.32:0/877350688 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f92240d13a0 0x7f92241f8630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-03T20:59:09.984 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.981+0000 7f9219400640 1 --2- 172.21.15.32:0/877350688 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f92240cc760 0x7f92241f80f0 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7f920c031000 tx=0x7f920c032e40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:59:09.985 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.981+0000 7f921aa00640 1 -- 172.21.15.32:0/877350688 <== mon.1 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f920c037040 con 0x7f92240cc760 2024-08-03T20:59:09.985 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.981+0000 7f921aa00640 1 -- 172.21.15.32:0/877350688 <== mon.1 v2:172.21.15.118:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f920c004110 con 0x7f92240cc760 2024-08-03T20:59:09.985 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.981+0000 7f921aa00640 1 -- 172.21.15.32:0/877350688 <== mon.1 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f920c005020 con 0x7f92240cc760 2024-08-03T20:59:09.985 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.981+0000 7f9228c00640 1 -- 172.21.15.32:0/877350688 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f92240cb9c0 con 0x7f92240cc760 2024-08-03T20:59:09.985 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.981+0000 7f9228c00640 1 -- 172.21.15.32:0/877350688 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f92240cbf80 con 0x7f92240cc760 2024-08-03T20:59:09.986 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.982+0000 7f9228c00640 1 -- 172.21.15.32:0/877350688 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f922409b5d0 con 0x7f92240cc760 2024-08-03T20:59:09.986 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.983+0000 7f921aa00640 1 -- 172.21.15.32:0/877350688 <== mon.1 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f920c044030 con 0x7f92240cc760 2024-08-03T20:59:09.986 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.984+0000 7f921aa00640 1 --2- 172.21.15.32:0/877350688 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f9200077040 0x7f9200079550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:59:09.987 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.984+0000 7f921aa00640 1 -- 172.21.15.32:0/877350688 <== mon.1 v2:172.21.15.118:3300/0 5 ==== osd_map(118..118 src has 1..118) ==== 8692+0+0 (secure 0 0 0) 0x7f920c0b94c0 con 0x7f92240cc760 2024-08-03T20:59:09.987 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.985+0000 7f9221e00640 1 --2- 172.21.15.32:0/877350688 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f9200077040 0x7f9200079550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:59:09.987 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.985+0000 7f9221e00640 1 --2- 172.21.15.32:0/877350688 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f9200077040 0x7f9200079550 secure :-1 s=READY pgs=766 cs=0 l=1 rev1=1 crypto rx=0x7f9214009bf0 tx=0x7f9214005f90 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:59:09.990 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:09.989+0000 7f921aa00640 1 -- 172.21.15.32:0/877350688 <== mon.1 v2:172.21.15.118:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7f920c04b020 con 0x7f92240cc760 2024-08-03T20:59:10.234 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:10.232+0000 7f9228c00640 1 -- 172.21.15.32:0/877350688 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "joinable", "val": "true"} v 0) -- 0x7f9224068f80 con 0x7f92240cc760 2024-08-03T20:59:11.176 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:11.174+0000 7f921aa00640 1 -- 172.21.15.32:0/877350688 <== mon.1 v2:172.21.15.118:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "joinable", "val": "true"}]=0 cephfs marked joinable; MDS may join as newly active. v281) ==== 162+0+0 (secure 0 0 0) 0x7f920c0814a0 con 0x7f92240cc760 2024-08-03T20:59:11.176 INFO:teuthology.orchestra.run.smithi032.stderr:cephfs marked joinable; MDS may join as newly active. 2024-08-03T20:59:11.179 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:11.177+0000 7f9228c00640 1 -- 172.21.15.32:0/877350688 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f9200077040 msgr2=0x7f9200079550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:59:11.179 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:11.177+0000 7f9228c00640 1 --2- 172.21.15.32:0/877350688 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f9200077040 0x7f9200079550 secure :-1 s=READY pgs=766 cs=0 l=1 rev1=1 crypto rx=0x7f9214009bf0 tx=0x7f9214005f90 comp rx=0 tx=0).stop 2024-08-03T20:59:11.179 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:11.177+0000 7f9228c00640 1 -- 172.21.15.32:0/877350688 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f92240cc760 msgr2=0x7f92241f80f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:59:11.179 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:11.177+0000 7f9228c00640 1 --2- 172.21.15.32:0/877350688 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f92240cc760 0x7f92241f80f0 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7f920c031000 tx=0x7f920c032e40 comp rx=0 tx=0).stop 2024-08-03T20:59:11.179 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:11.177+0000 7f9228c00640 1 -- 172.21.15.32:0/877350688 shutdown_connections 2024-08-03T20:59:11.179 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:11.177+0000 7f9228c00640 1 --2- 172.21.15.32:0/877350688 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f9200077040 0x7f9200079550 unknown :-1 s=CLOSED pgs=766 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:59:11.179 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:11.177+0000 7f9228c00640 1 --2- 172.21.15.32:0/877350688 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f92240cc760 0x7f92241f80f0 unknown :-1 s=CLOSED pgs=576 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:59:11.180 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:11.177+0000 7f9228c00640 1 --2- 172.21.15.32:0/877350688 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f92240d13a0 0x7f92241f8630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:59:11.180 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:11.177+0000 7f9228c00640 1 --2- 172.21.15.32:0/877350688 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f922409b170 0x7f92240b18d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:59:11.180 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:11.177+0000 7f9228c00640 1 -- 172.21.15.32:0/877350688 >> 172.21.15.32:0/877350688 conn(0x7f92240b3c00 msgr2=0x7f92241e8990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:59:11.180 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:11.178+0000 7f9228c00640 1 -- 172.21.15.32:0/877350688 shutdown_connections 2024-08-03T20:59:11.180 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:11.178+0000 7f9228c00640 1 -- 172.21.15.32:0/877350688 wait complete. 2024-08-03T20:59:11.612 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-08-03T20:59:11.848 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:11.846+0000 7fa5a3c00640 1 -- 172.21.15.32:0/1298748369 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa59c0733f0 msgr2=0x7fa59c073800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:59:11.848 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:11.846+0000 7fa5a3c00640 1 --2- 172.21.15.32:0/1298748369 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa59c0733f0 0x7fa59c073800 secure :-1 s=READY pgs=1903 cs=0 l=1 rev1=1 crypto rx=0x7fa59c070500 tx=0x7fa58802a4b0 comp rx=0 tx=0).stop 2024-08-03T20:59:11.848 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:11.846+0000 7fa5a3c00640 1 -- 172.21.15.32:0/1298748369 shutdown_connections 2024-08-03T20:59:11.848 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:11.846+0000 7fa5a3c00640 1 --2- 172.21.15.32:0/1298748369 >> v2:172.21.15.118:3300/0 conn(0x7fa59c15e170 0x7fa59c15e580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:59:11.848 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:11.846+0000 7fa5a3c00640 1 --2- 172.21.15.32:0/1298748369 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa59c0733f0 0x7fa59c073800 unknown :-1 s=CLOSED pgs=1903 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:59:11.848 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:11.846+0000 7fa5a3c00640 1 --2- 172.21.15.32:0/1298748369 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fa59c0b4de0 0x7fa59c1515c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:59:11.848 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:11.846+0000 7fa5a3c00640 1 -- 172.21.15.32:0/1298748369 >> 172.21.15.32:0/1298748369 conn(0x7fa59c0b1bd0 msgr2=0x7fa59c0b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:59:11.849 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:11.846+0000 7fa5a3c00640 1 -- 172.21.15.32:0/1298748369 shutdown_connections 2024-08-03T20:59:11.849 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:11.846+0000 7fa5a3c00640 1 -- 172.21.15.32:0/1298748369 wait complete. 2024-08-03T20:59:11.850 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:11.848+0000 7fa5a3c00640 1 Processor -- start 2024-08-03T20:59:11.850 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:11.848+0000 7fa5a3c00640 1 -- start start 2024-08-03T20:59:11.852 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:11.850+0000 7fa5a3c00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa59c0733f0 0x7fa59c1f5c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:59:11.852 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:11.850+0000 7fa5a3c00640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fa59c0b4de0 0x7fa59c15b530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:59:11.852 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:11.850+0000 7fa5a3c00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa59c15e170 0x7fa59c0cc120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:59:11.852 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:11.850+0000 7fa5a3c00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fa59c06e930 con 0x7fa59c15e170 2024-08-03T20:59:11.852 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:11.850+0000 7fa5a3c00640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7fa59c070ad0 con 0x7fa59c0733f0 2024-08-03T20:59:11.852 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:11.850+0000 7fa5a3c00640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7fa59c06eab0 con 0x7fa59c0b4de0 2024-08-03T20:59:11.853 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:11.850+0000 7fa5a1c00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa59c15e170 0x7fa59c0cc120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:59:11.853 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:11.850+0000 7fa5a1c00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa59c15e170 0x7fa59c0cc120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:46632/0 (socket says 172.21.15.32:46632) 2024-08-03T20:59:11.853 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:11.850+0000 7fa5a1c00640 1 -- 172.21.15.32:0/2641500226 learned_addr learned my addr 172.21.15.32:0/2641500226 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:59:11.853 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:11.850+0000 7fa59be00640 1 --2- 172.21.15.32:0/2641500226 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fa59c0b4de0 0x7fa59c15b530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:59:11.853 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:11.850+0000 7fa5a1200640 1 --2- 172.21.15.32:0/2641500226 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa59c0733f0 0x7fa59c1f5c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:59:11.853 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:11.851+0000 7fa5a1c00640 1 -- 172.21.15.32:0/2641500226 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fa59c0b4de0 msgr2=0x7fa59c15b530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:59:11.853 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:11.851+0000 7fa5a1c00640 1 --2- 172.21.15.32:0/2641500226 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fa59c0b4de0 0x7fa59c15b530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:59:11.853 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:11.851+0000 7fa5a1c00640 1 -- 172.21.15.32:0/2641500226 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa59c0733f0 msgr2=0x7fa59c1f5c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:59:11.853 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:11.851+0000 7fa5a1c00640 1 --2- 172.21.15.32:0/2641500226 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa59c0733f0 0x7fa59c1f5c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:59:11.853 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:11.851+0000 7fa5a1c00640 1 -- 172.21.15.32:0/2641500226 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa59c154440 con 0x7fa59c15e170 2024-08-03T20:59:11.853 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:11.851+0000 7fa59be00640 1 --2- 172.21.15.32:0/2641500226 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fa59c0b4de0 0x7fa59c15b530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-03T20:59:11.853 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:11.851+0000 7fa5a1c00640 1 --2- 172.21.15.32:0/2641500226 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa59c15e170 0x7fa59c0cc120 secure :-1 s=READY pgs=1904 cs=0 l=1 rev1=1 crypto rx=0x7fa59400ddc0 tx=0x7fa59400b730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:59:11.854 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:11.851+0000 7fa598c00640 1 -- 172.21.15.32:0/2641500226 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa594004d20 con 0x7fa59c15e170 2024-08-03T20:59:11.855 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:11.851+0000 7fa598c00640 1 -- 172.21.15.32:0/2641500226 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fa59400be40 con 0x7fa59c15e170 2024-08-03T20:59:11.855 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:11.851+0000 7fa598c00640 1 -- 172.21.15.32:0/2641500226 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa59400e8c0 con 0x7fa59c15e170 2024-08-03T20:59:11.855 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:11.851+0000 7fa5a3c00640 1 -- 172.21.15.32:0/2641500226 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa59c154720 con 0x7fa59c15e170 2024-08-03T20:59:11.855 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:11.851+0000 7fa5a3c00640 1 -- 172.21.15.32:0/2641500226 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa59c15b010 con 0x7fa59c15e170 2024-08-03T20:59:11.855 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:11.853+0000 7fa5a3c00640 1 -- 172.21.15.32:0/2641500226 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa59c1515c0 con 0x7fa59c15e170 2024-08-03T20:59:11.856 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:11.854+0000 7fa598c00640 1 -- 172.21.15.32:0/2641500226 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7fa59400ea60 con 0x7fa59c15e170 2024-08-03T20:59:11.857 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:11.855+0000 7fa598c00640 1 --2- 172.21.15.32:0/2641500226 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fa57c077010 0x7fa57c079520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:59:11.857 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:11.856+0000 7fa598c00640 1 -- 172.21.15.32:0/2641500226 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(118..118 src has 1..118) ==== 8692+0+0 (secure 0 0 0) 0x7fa594099c70 con 0x7fa59c15e170 2024-08-03T20:59:11.857 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:11.856+0000 7fa5a1200640 1 --2- 172.21.15.32:0/2641500226 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fa57c077010 0x7fa57c079520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:59:11.858 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:11.857+0000 7fa5a1200640 1 --2- 172.21.15.32:0/2641500226 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fa57c077010 0x7fa57c079520 secure :-1 s=READY pgs=767 cs=0 l=1 rev1=1 crypto rx=0x7fa590004630 tx=0x7fa59000b2e0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:59:11.862 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:11.860+0000 7fa598c00640 1 -- 172.21.15.32:0/2641500226 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7fa594061ce0 con 0x7fa59c15e170 2024-08-03T20:59:12.107 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:12.105+0000 7fa5a3c00640 1 -- 172.21.15.32:0/2641500226 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa59c15aae0 con 0x7fa59c15e170 2024-08-03T20:59:12.107 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:12.106+0000 7fa598c00640 1 -- 172.21.15.32:0/2641500226 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 282 v282) ==== 95+0+5226 (secure 0 0 0) 0x7fa594066230 con 0x7fa59c15e170 2024-08-03T20:59:12.107 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-03T20:59:12.108 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":282,"btime":"2024-08-03T20:59:11:175327+0000","default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6439,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6835/531080160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6834","nonce":531080160},{"type":"v1","addr":"172.21.15.118:6835","nonce":531080160}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":84},{"gid":6454,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6837/2796882319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6836","nonce":2796882319},{"type":"v1","addr":"172.21.15.118:6837","nonce":2796882319}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":86},{"gid":6813,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6834/2686933890","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":2686933890},{"type":"v1","addr":"172.21.15.32:6834","nonce":2686933890}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":85}],"filesystems":[{"mdsmap":{"epoch":282,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-08-03T20:54:26.950518+0000","modified":"2024-08-03T20:59:11.175309+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":118,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":9759},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9759":{"gid":9759,"name":"a","rank":0,"incarnation":282,"state":"up:replay","state_seq":1,"addr":"172.21.15.32:6835/2173042128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6833","nonce":2173042128},{"type":"v1","addr":"172.21.15.32:6835","nonce":2173042128}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":6}]} 2024-08-03T20:59:12.108 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 282 2024-08-03T20:59:12.110 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:12.108+0000 7fa5a3c00640 1 -- 172.21.15.32:0/2641500226 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fa57c077010 msgr2=0x7fa57c079520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:59:12.110 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:12.108+0000 7fa5a3c00640 1 --2- 172.21.15.32:0/2641500226 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fa57c077010 0x7fa57c079520 secure :-1 s=READY pgs=767 cs=0 l=1 rev1=1 crypto rx=0x7fa590004630 tx=0x7fa59000b2e0 comp rx=0 tx=0).stop 2024-08-03T20:59:12.110 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:12.108+0000 7fa5a3c00640 1 -- 172.21.15.32:0/2641500226 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa59c15e170 msgr2=0x7fa59c0cc120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:59:12.110 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:12.108+0000 7fa5a3c00640 1 --2- 172.21.15.32:0/2641500226 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa59c15e170 0x7fa59c0cc120 secure :-1 s=READY pgs=1904 cs=0 l=1 rev1=1 crypto rx=0x7fa59400ddc0 tx=0x7fa59400b730 comp rx=0 tx=0).stop 2024-08-03T20:59:12.111 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:12.108+0000 7fa5a3c00640 1 -- 172.21.15.32:0/2641500226 shutdown_connections 2024-08-03T20:59:12.111 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:12.108+0000 7fa5a3c00640 1 --2- 172.21.15.32:0/2641500226 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fa57c077010 0x7fa57c079520 unknown :-1 s=CLOSED pgs=767 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:59:12.111 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:12.108+0000 7fa5a3c00640 1 --2- 172.21.15.32:0/2641500226 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa59c15e170 0x7fa59c0cc120 unknown :-1 s=CLOSED pgs=1904 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:59:12.111 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:12.108+0000 7fa5a3c00640 1 --2- 172.21.15.32:0/2641500226 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fa59c0b4de0 0x7fa59c15b530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:59:12.111 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:12.108+0000 7fa5a3c00640 1 --2- 172.21.15.32:0/2641500226 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa59c0733f0 0x7fa59c1f5c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:59:12.111 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:12.108+0000 7fa5a3c00640 1 -- 172.21.15.32:0/2641500226 >> 172.21.15.32:0/2641500226 conn(0x7fa59c0b1bd0 msgr2=0x7fa59c150c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:59:12.111 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:12.108+0000 7fa5a3c00640 1 -- 172.21.15.32:0/2641500226 shutdown_connections 2024-08-03T20:59:12.111 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:12.109+0000 7fa5a3c00640 1 -- 172.21.15.32:0/2641500226 wait complete. 2024-08-03T20:59:12.132 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 282, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-08-03T20:54:26.950518+0000', 'modified': '2024-08-03T20:59:11.175309+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 118, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 9759}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9759': {'gid': 9759, 'name': 'a', 'rank': 0, 'incarnation': 282, 'state': 'up:replay', 'state_seq': 1, 'addr': '172.21.15.32:6835/2173042128', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6833', 'nonce': 2173042128}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 2173042128}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [13], 'metadata_pool': 12, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 0, 'qdb_cluster': []} 2024-08-03T20:59:12.132 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9759:up:replay 2024-08-03T20:59:12.132 DEBUG:tasks.ceph_test_case:wait_until_true: waiting (timeout=60 retry_count=0)... 2024-08-03T20:59:17.537 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-08-03T20:59:17.777 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:17.775+0000 7f8da9e00640 1 -- 172.21.15.32:0/1775013066 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8da40cee10 msgr2=0x7f8da415c0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:59:17.777 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:17.775+0000 7f8da9e00640 1 --2- 172.21.15.32:0/1775013066 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8da40cee10 0x7f8da415c0b0 secure :-1 s=READY pgs=1905 cs=0 l=1 rev1=1 crypto rx=0x7f8da40701e0 tx=0x7f8d9402a560 comp rx=0 tx=0).stop 2024-08-03T20:59:17.777 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:17.775+0000 7f8da9e00640 1 -- 172.21.15.32:0/1775013066 shutdown_connections 2024-08-03T20:59:17.777 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:17.775+0000 7f8da9e00640 1 --2- 172.21.15.32:0/1775013066 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f8da40af450 0x7f8da406f9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:59:17.778 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:17.775+0000 7f8da9e00640 1 --2- 172.21.15.32:0/1775013066 >> v2:172.21.15.118:3300/0 conn(0x7f8da40af040 0x7f8da406f480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:59:17.778 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:17.775+0000 7f8da9e00640 1 --2- 172.21.15.32:0/1775013066 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8da40cee10 0x7f8da415c0b0 unknown :-1 s=CLOSED pgs=1905 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:59:17.778 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:17.775+0000 7f8da9e00640 1 -- 172.21.15.32:0/1775013066 >> 172.21.15.32:0/1775013066 conn(0x7f8da40b2120 msgr2=0x7f8da40cfd60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:59:17.778 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:17.775+0000 7f8da9e00640 1 -- 172.21.15.32:0/1775013066 shutdown_connections 2024-08-03T20:59:17.778 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:17.775+0000 7f8da9e00640 1 -- 172.21.15.32:0/1775013066 wait complete. 2024-08-03T20:59:17.779 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:17.777+0000 7f8da9e00640 1 Processor -- start 2024-08-03T20:59:17.779 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:17.778+0000 7f8da9e00640 1 -- start start 2024-08-03T20:59:17.781 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:17.779+0000 7f8da9e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8da40af040 0x7f8da4147890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:59:17.781 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:17.779+0000 7f8da9e00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8da40af450 0x7f8da4148230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:59:17.781 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:17.779+0000 7f8da9e00640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f8da40cee10 0x7f8da4148770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:59:17.781 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:17.779+0000 7f8da9e00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f8da40ca740 con 0x7f8da40af040 2024-08-03T20:59:17.781 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:17.779+0000 7f8da9e00640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f8da40ca5c0 con 0x7f8da40af450 2024-08-03T20:59:17.781 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:17.779+0000 7f8da9e00640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7f8da40ca8c0 con 0x7f8da40cee10 2024-08-03T20:59:17.781 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:17.779+0000 7f8da8a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8da40af040 0x7f8da4147890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:59:17.782 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:17.779+0000 7f8da8a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8da40af040 0x7f8da4147890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:46660/0 (socket says 172.21.15.32:46660) 2024-08-03T20:59:17.782 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:17.779+0000 7f8da8a00640 1 -- 172.21.15.32:0/1026046044 learned_addr learned my addr 172.21.15.32:0/1026046044 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:59:17.782 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:17.779+0000 7f8d9be00640 1 --2- 172.21.15.32:0/1026046044 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8da40af450 0x7f8da4148230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:59:17.782 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:17.779+0000 7f8da9400640 1 --2- 172.21.15.32:0/1026046044 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f8da40cee10 0x7f8da4148770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:59:17.782 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:17.780+0000 7f8da8a00640 1 -- 172.21.15.32:0/1026046044 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f8da40cee10 msgr2=0x7f8da4148770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:59:17.782 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:17.780+0000 7f8da8a00640 1 --2- 172.21.15.32:0/1026046044 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f8da40cee10 0x7f8da4148770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:59:17.782 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:17.780+0000 7f8da8a00640 1 -- 172.21.15.32:0/1026046044 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8da40af450 msgr2=0x7f8da4148230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:59:17.782 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:17.780+0000 7f8da8a00640 1 --2- 172.21.15.32:0/1026046044 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8da40af450 0x7f8da4148230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:59:17.782 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:17.780+0000 7f8da8a00640 1 -- 172.21.15.32:0/1026046044 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8da409d3a0 con 0x7f8da40af040 2024-08-03T20:59:17.782 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:17.780+0000 7f8da9400640 1 --2- 172.21.15.32:0/1026046044 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f8da40cee10 0x7f8da4148770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-03T20:59:17.783 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:17.780+0000 7f8da8a00640 1 --2- 172.21.15.32:0/1026046044 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8da40af040 0x7f8da4147890 secure :-1 s=READY pgs=1906 cs=0 l=1 rev1=1 crypto rx=0x7f8d94034040 tx=0x7f8d940038a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:59:17.783 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:17.780+0000 7f8da1600640 1 -- 172.21.15.32:0/1026046044 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d9403e030 con 0x7f8da40af040 2024-08-03T20:59:17.783 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:17.780+0000 7f8da1600640 1 -- 172.21.15.32:0/1026046044 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f8d94003ca0 con 0x7f8da40af040 2024-08-03T20:59:17.783 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:17.780+0000 7f8da1600640 1 -- 172.21.15.32:0/1026046044 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d94035040 con 0x7f8da40af040 2024-08-03T20:59:17.784 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:17.780+0000 7f8da9e00640 1 -- 172.21.15.32:0/1026046044 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8da409c5e0 con 0x7f8da40af040 2024-08-03T20:59:17.784 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:17.781+0000 7f8da9e00640 1 -- 172.21.15.32:0/1026046044 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8da409cba0 con 0x7f8da40af040 2024-08-03T20:59:17.784 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:17.782+0000 7f8da9e00640 1 -- 172.21.15.32:0/1026046044 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8da40cae50 con 0x7f8da40af040 2024-08-03T20:59:17.784 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:17.783+0000 7f8da1600640 1 -- 172.21.15.32:0/1026046044 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f8d94041030 con 0x7f8da40af040 2024-08-03T20:59:17.786 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:17.784+0000 7f8da1600640 1 --2- 172.21.15.32:0/1026046044 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f8d80077040 0x7f8d80079550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:59:17.786 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:17.785+0000 7f8da1600640 1 -- 172.21.15.32:0/1026046044 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(118..118 src has 1..118) ==== 8692+0+0 (secure 0 0 0) 0x7f8d940b9aa0 con 0x7f8da40af040 2024-08-03T20:59:17.786 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:17.785+0000 7f8d9be00640 1 --2- 172.21.15.32:0/1026046044 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f8d80077040 0x7f8d80079550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:59:17.787 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:17.786+0000 7f8d9be00640 1 --2- 172.21.15.32:0/1026046044 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f8d80077040 0x7f8d80079550 secure :-1 s=READY pgs=768 cs=0 l=1 rev1=1 crypto rx=0x7f8da415b670 tx=0x7f8d8c005f90 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:59:17.790 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:17.789+0000 7f8da1600640 1 -- 172.21.15.32:0/1026046044 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7f8d94081a80 con 0x7f8da40af040 2024-08-03T20:59:18.033 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:18.032+0000 7f8da9e00640 1 -- 172.21.15.32:0/1026046044 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f8da40ce880 con 0x7f8da40af040 2024-08-03T20:59:18.034 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:18.033+0000 7f8da1600640 1 -- 172.21.15.32:0/1026046044 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 285 v285) ==== 95+0+5233 (secure 0 0 0) 0x7f8d94085fd0 con 0x7f8da40af040 2024-08-03T20:59:18.035 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-03T20:59:18.035 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":285,"btime":"2024-08-03T20:59:14:185763+0000","default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6439,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6835/531080160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6834","nonce":531080160},{"type":"v1","addr":"172.21.15.118:6835","nonce":531080160}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":84},{"gid":6454,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6837/2796882319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6836","nonce":2796882319},{"type":"v1","addr":"172.21.15.118:6837","nonce":2796882319}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":86},{"gid":6813,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6834/2686933890","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":2686933890},{"type":"v1","addr":"172.21.15.32:6834","nonce":2686933890}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":85}],"filesystems":[{"mdsmap":{"epoch":285,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-08-03T20:54:26.950518+0000","modified":"2024-08-03T20:59:14.185750+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":118,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":9759},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9759":{"gid":9759,"name":"a","rank":0,"incarnation":282,"state":"up:active","state_seq":4,"addr":"172.21.15.32:6835/2173042128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6833","nonce":2173042128},{"type":"v1","addr":"172.21.15.32:6835","nonce":2173042128}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9759,"qdb_cluster":[9759]},"id":6}]} 2024-08-03T20:59:18.035 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 285 2024-08-03T20:59:18.037 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:18.035+0000 7f8da9e00640 1 -- 172.21.15.32:0/1026046044 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f8d80077040 msgr2=0x7f8d80079550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:59:18.037 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:18.035+0000 7f8da9e00640 1 --2- 172.21.15.32:0/1026046044 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f8d80077040 0x7f8d80079550 secure :-1 s=READY pgs=768 cs=0 l=1 rev1=1 crypto rx=0x7f8da415b670 tx=0x7f8d8c005f90 comp rx=0 tx=0).stop 2024-08-03T20:59:18.037 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:18.035+0000 7f8da9e00640 1 -- 172.21.15.32:0/1026046044 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8da40af040 msgr2=0x7f8da4147890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:59:18.037 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:18.035+0000 7f8da9e00640 1 --2- 172.21.15.32:0/1026046044 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8da40af040 0x7f8da4147890 secure :-1 s=READY pgs=1906 cs=0 l=1 rev1=1 crypto rx=0x7f8d94034040 tx=0x7f8d940038a0 comp rx=0 tx=0).stop 2024-08-03T20:59:18.037 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:18.035+0000 7f8da9e00640 1 -- 172.21.15.32:0/1026046044 shutdown_connections 2024-08-03T20:59:18.037 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:18.035+0000 7f8da9e00640 1 --2- 172.21.15.32:0/1026046044 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f8d80077040 0x7f8d80079550 unknown :-1 s=CLOSED pgs=768 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:59:18.037 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:18.035+0000 7f8da9e00640 1 --2- 172.21.15.32:0/1026046044 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8da40af450 0x7f8da4148230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:59:18.037 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:18.035+0000 7f8da9e00640 1 --2- 172.21.15.32:0/1026046044 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f8da40cee10 0x7f8da4148770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:59:18.037 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:18.035+0000 7f8da9e00640 1 --2- 172.21.15.32:0/1026046044 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8da40af040 0x7f8da4147890 unknown :-1 s=CLOSED pgs=1906 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:59:18.037 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:18.035+0000 7f8da9e00640 1 -- 172.21.15.32:0/1026046044 >> 172.21.15.32:0/1026046044 conn(0x7f8da40b2120 msgr2=0x7f8da406f510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:59:18.038 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:18.035+0000 7f8da9e00640 1 -- 172.21.15.32:0/1026046044 shutdown_connections 2024-08-03T20:59:18.038 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:18.035+0000 7f8da9e00640 1 -- 172.21.15.32:0/1026046044 wait complete. 2024-08-03T20:59:18.062 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 285, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-08-03T20:54:26.950518+0000', 'modified': '2024-08-03T20:59:14.185750+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 118, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 9759}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9759': {'gid': 9759, 'name': 'a', 'rank': 0, 'incarnation': 282, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.32:6835/2173042128', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6833', 'nonce': 2173042128}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 2173042128}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [13], 'metadata_pool': 12, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9759, 'qdb_cluster': [9759]} 2024-08-03T20:59:18.062 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-08-03T20:59:18.062 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2024-08-03T20:59:18.168 INFO:tasks.ceph.mds.a.smithi032.stderr:2024-08-03T20:59:18.166+0000 7f07b2000640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-08-03T20:59:18.296 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:18.293+0000 7f6bc7600640 1 -- 172.21.15.32:0/374673534 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6bc006e930 msgr2=0x7f6bc0144f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:59:18.296 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:18.293+0000 7f6bc7600640 1 --2- 172.21.15.32:0/374673534 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6bc006e930 0x7f6bc0144f90 secure :-1 s=READY pgs=1907 cs=0 l=1 rev1=1 crypto rx=0x7f6bc0072d80 tx=0x7f6bb002a5f0 comp rx=0 tx=0).stop 2024-08-03T20:59:18.296 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:18.293+0000 7f6bc7600640 1 -- 172.21.15.32:0/374673534 shutdown_connections 2024-08-03T20:59:18.296 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:18.293+0000 7f6bc7600640 1 --2- 172.21.15.32:0/374673534 >> v2:172.21.15.118:3300/0 conn(0x7f6bc015fff0 0x7f6bc0160400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:59:18.296 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:18.293+0000 7f6bc7600640 1 --2- 172.21.15.32:0/374673534 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6bc006e930 0x7f6bc0144f90 unknown :-1 s=CLOSED pgs=1907 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:59:18.296 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:18.293+0000 7f6bc7600640 1 -- 172.21.15.32:0/374673534 >> 172.21.15.32:0/374673534 conn(0x7f6bc00b1bd0 msgr2=0x7f6bc00b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:59:18.296 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:18.293+0000 7f6bc7600640 1 -- 172.21.15.32:0/374673534 shutdown_connections 2024-08-03T20:59:18.296 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:18.293+0000 7f6bc7600640 1 -- 172.21.15.32:0/374673534 wait complete. 2024-08-03T20:59:18.297 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:18.296+0000 7f6bc7600640 1 Processor -- start 2024-08-03T20:59:18.297 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:18.296+0000 7f6bc7600640 1 -- start start 2024-08-03T20:59:18.300 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:18.297+0000 7f6bc7600640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f6bc006e930 0x7f6bc015cd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:59:18.300 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:18.297+0000 7f6bc7600640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6bc0073e60 0x7f6bc01eb510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:59:18.300 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:18.297+0000 7f6bc7600640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f6bc015fff0 0x7f6bc01f8cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:59:18.300 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:18.297+0000 7f6bc7600640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f6bc00cf840 con 0x7f6bc0073e60 2024-08-03T20:59:18.301 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:18.297+0000 7f6bc7600640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f6bc0145f30 con 0x7f6bc006e930 2024-08-03T20:59:18.301 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:18.297+0000 7f6bc7600640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7f6bc00cf9c0 con 0x7f6bc015fff0 2024-08-03T20:59:18.301 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:18.297+0000 7f6bb7e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6bc0073e60 0x7f6bc01eb510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:59:18.301 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:18.297+0000 7f6bb7e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6bc0073e60 0x7f6bc01eb510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:46676/0 (socket says 172.21.15.32:46676) 2024-08-03T20:59:18.301 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:18.297+0000 7f6bb7e00640 1 -- 172.21.15.32:0/2379126739 learned_addr learned my addr 172.21.15.32:0/2379126739 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:59:18.301 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:18.298+0000 7f6bc5600640 1 --2- 172.21.15.32:0/2379126739 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f6bc015fff0 0x7f6bc01f8cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:59:18.301 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:18.298+0000 7f6bc4c00640 1 --2- 172.21.15.32:0/2379126739 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f6bc006e930 0x7f6bc015cd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:59:18.301 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:18.298+0000 7f6bb7e00640 1 -- 172.21.15.32:0/2379126739 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f6bc015fff0 msgr2=0x7f6bc01f8cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:59:18.301 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:18.298+0000 7f6bb7e00640 1 --2- 172.21.15.32:0/2379126739 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f6bc015fff0 0x7f6bc01f8cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:59:18.301 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:18.298+0000 7f6bb7e00640 1 -- 172.21.15.32:0/2379126739 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f6bc006e930 msgr2=0x7f6bc015cd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:59:18.301 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:18.298+0000 7f6bb7e00640 1 --2- 172.21.15.32:0/2379126739 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f6bc006e930 0x7f6bc015cd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:59:18.301 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:18.298+0000 7f6bb7e00640 1 -- 172.21.15.32:0/2379126739 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6bc015d530 con 0x7f6bc0073e60 2024-08-03T20:59:18.301 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:18.298+0000 7f6bc4c00640 1 --2- 172.21.15.32:0/2379126739 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f6bc006e930 0x7f6bc015cd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-03T20:59:18.302 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:18.298+0000 7f6bb7e00640 1 --2- 172.21.15.32:0/2379126739 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6bc0073e60 0x7f6bc01eb510 secure :-1 s=READY pgs=1908 cs=0 l=1 rev1=1 crypto rx=0x7f6ba800c340 tx=0x7f6ba800c800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:59:18.302 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:18.298+0000 7f6bbd600640 1 -- 172.21.15.32:0/2379126739 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ba8017070 con 0x7f6bc0073e60 2024-08-03T20:59:18.302 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:18.298+0000 7f6bbd600640 1 -- 172.21.15.32:0/2379126739 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f6ba80053c0 con 0x7f6bc0073e60 2024-08-03T20:59:18.302 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:18.298+0000 7f6bbd600640 1 -- 172.21.15.32:0/2379126739 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ba8003850 con 0x7f6bc0073e60 2024-08-03T20:59:18.302 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:18.298+0000 7f6bc7600640 1 -- 172.21.15.32:0/2379126739 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6bc015d810 con 0x7f6bc0073e60 2024-08-03T20:59:18.302 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:18.299+0000 7f6bc7600640 1 -- 172.21.15.32:0/2379126739 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6bc015ddd0 con 0x7f6bc0073e60 2024-08-03T20:59:18.303 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:18.301+0000 7f6bbd600640 1 -- 172.21.15.32:0/2379126739 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f6ba801a070 con 0x7f6bc0073e60 2024-08-03T20:59:18.304 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:18.302+0000 7f6bbd600640 1 --2- 172.21.15.32:0/2379126739 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f6b9c077040 0x7f6b9c079550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:59:18.304 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:18.302+0000 7f6bbd600640 1 -- 172.21.15.32:0/2379126739 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(118..118 src has 1..118) ==== 8692+0+0 (secure 0 0 0) 0x7f6ba8098840 con 0x7f6bc0073e60 2024-08-03T20:59:18.304 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:18.302+0000 7f6bce785740 1 Processor -- start 2024-08-03T20:59:18.304 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:18.302+0000 7f6bce785740 1 -- start start 2024-08-03T20:59:18.304 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:18.303+0000 7f6bc4c00640 1 --2- 172.21.15.32:0/2379126739 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f6b9c077040 0x7f6b9c079550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:59:18.308 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:18.303+0000 7f6bc4c00640 1 --2- 172.21.15.32:0/2379126739 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f6b9c077040 0x7f6b9c079550 secure :-1 s=READY pgs=769 cs=0 l=1 rev1=1 crypto rx=0x7f6bb0033000 tx=0x7f6bb0004730 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:59:18.308 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:18.304+0000 7f6bce785740 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x5631f7f32270 0x5631f831ce90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:59:18.308 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:18.304+0000 7f6bce785740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5631f7f093b0 0x5631f82f1ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:59:18.308 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:18.304+0000 7f6bce785740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5631f7f11590 0x5631f82f2220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:59:18.308 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:18.304+0000 7f6bce785740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x5631f824aa60 con 0x5631f7f11590 2024-08-03T20:59:18.308 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:18.304+0000 7f6bce785740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x5631f8239f60 con 0x5631f7f093b0 2024-08-03T20:59:18.308 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:18.304+0000 7f6bce785740 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x5631f823dfa0 con 0x5631f7f32270 2024-08-03T20:59:18.308 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:18.304+0000 7f6bc5600640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5631f7f11590 0x5631f82f2220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:59:18.309 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:18.304+0000 7f6bc5600640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5631f7f11590 0x5631f82f2220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:46688/0 (socket says 172.21.15.32:46688) 2024-08-03T20:59:18.309 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:18.304+0000 7f6bc5600640 1 -- 172.21.15.32:0/3571980957 learned_addr learned my addr 172.21.15.32:0/3571980957 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:59:18.309 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:18.304+0000 7f6bc4c00640 1 --2- 172.21.15.32:0/3571980957 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x5631f7f32270 0x5631f831ce90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:59:18.309 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:18.304+0000 7f6bb7e00640 1 --2- 172.21.15.32:0/3571980957 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5631f7f093b0 0x5631f82f1ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:59:18.309 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:18.305+0000 7f6bc5600640 1 -- 172.21.15.32:0/3571980957 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x5631f7f32270 msgr2=0x5631f831ce90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:59:18.309 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:18.305+0000 7f6bc5600640 1 --2- 172.21.15.32:0/3571980957 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x5631f7f32270 0x5631f831ce90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:59:18.310 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:18.305+0000 7f6bc5600640 1 -- 172.21.15.32:0/3571980957 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5631f7f093b0 msgr2=0x5631f82f1ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:59:18.310 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:18.305+0000 7f6bc5600640 1 --2- 172.21.15.32:0/3571980957 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5631f7f093b0 0x5631f82f1ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:59:18.310 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:18.305+0000 7f6bc5600640 1 -- 172.21.15.32:0/3571980957 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5631f7f9d850 con 0x5631f7f11590 2024-08-03T20:59:18.310 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:18.305+0000 7f6bc4c00640 1 --2- 172.21.15.32:0/3571980957 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x5631f7f32270 0x5631f831ce90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-03T20:59:18.310 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:18.305+0000 7f6bc5600640 1 --2- 172.21.15.32:0/3571980957 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5631f7f11590 0x5631f82f2220 secure :-1 s=READY pgs=1909 cs=0 l=1 rev1=1 crypto rx=0x7f6bb8009f10 tx=0x7f6bb80107a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7933dfa8949f3e server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:59:18.310 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:18.305+0000 7f6bb5200640 1 -- 172.21.15.32:0/3571980957 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6bb8004d20 con 0x5631f7f11590 2024-08-03T20:59:18.310 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:18.305+0000 7f6bb5200640 1 -- 172.21.15.32:0/3571980957 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f6bb8015070 con 0x5631f7f11590 2024-08-03T20:59:18.310 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:18.305+0000 7f6bb5200640 1 -- 172.21.15.32:0/3571980957 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6bb8005280 con 0x5631f7f11590 2024-08-03T20:59:18.310 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:18.306+0000 7f6bce785740 1 -- 172.21.15.32:0/3571980957 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5631f7f11590 msgr2=0x5631f82f2220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:59:18.311 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:18.306+0000 7f6bce785740 1 --2- 172.21.15.32:0/3571980957 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5631f7f11590 0x5631f82f2220 secure :-1 s=READY pgs=1909 cs=0 l=1 rev1=1 crypto rx=0x7f6bb8009f10 tx=0x7f6bb80107a0 comp rx=0 tx=0).stop 2024-08-03T20:59:18.311 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:18.306+0000 7f6bce785740 1 -- 172.21.15.32:0/3571980957 shutdown_connections 2024-08-03T20:59:18.311 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:18.306+0000 7f6bce785740 1 --2- 172.21.15.32:0/3571980957 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5631f7f093b0 0x5631f82f1ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:59:18.311 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:18.306+0000 7f6bce785740 1 --2- 172.21.15.32:0/3571980957 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5631f7f11590 0x5631f82f2220 unknown :-1 s=CLOSED pgs=1909 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:59:18.311 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:18.306+0000 7f6bce785740 1 --2- 172.21.15.32:0/3571980957 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x5631f7f32270 0x5631f831ce90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:59:18.311 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:18.306+0000 7f6bce785740 1 -- 172.21.15.32:0/3571980957 >> 172.21.15.32:0/3571980957 conn(0x5631f7f263c0 msgr2=0x5631f82d8180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:59:18.311 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:18.306+0000 7f6bce785740 1 -- 172.21.15.32:0/3571980957 shutdown_connections 2024-08-03T20:59:18.311 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:18.306+0000 7f6bce785740 1 -- 172.21.15.32:0/3571980957 wait complete. 2024-08-03T20:59:18.316 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:18.307+0000 7f6bce785740 1 -- start start 2024-08-03T20:59:18.317 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:18.307+0000 7f6bce785740 1 Processor -- start 2024-08-03T20:59:18.317 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:18.309+0000 7f6bce785740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5631f7f093b0 0x5631f82d6930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:59:18.317 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:18.309+0000 7f6bce785740 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x5631f7f11590 0x5631f82f0900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:59:18.317 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:18.309+0000 7f6bce785740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5631f7f32270 0x5631f8313e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:59:18.317 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:18.309+0000 7f6bce785740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x5631f7fac620 con 0x5631f7f32270 2024-08-03T20:59:18.317 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:18.309+0000 7f6bce785740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x5631f81d7160 con 0x5631f7f093b0 2024-08-03T20:59:18.317 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:18.309+0000 7f6bce785740 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x5631f83018a0 con 0x5631f7f11590 2024-08-03T20:59:18.317 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:18.309+0000 7f6bce785740 10 client.0 _ll_register_callbacks cb 0x5631f81e66b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-03T20:59:18.317 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:18.309+0000 7f6bc5600640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5631f7f32270 0x5631f8313e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:59:18.317 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:18.309+0000 7f6bc5600640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5631f7f32270 0x5631f8313e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:46702/0 (socket says 172.21.15.32:46702) 2024-08-03T20:59:18.317 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:18.309+0000 7f6bc5600640 1 -- 172.21.15.32:0/3482557408 learned_addr learned my addr 172.21.15.32:0/3482557408 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:59:18.317 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:18.309+0000 7f6bb7e00640 1 --2- 172.21.15.32:0/3482557408 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x5631f7f11590 0x5631f82f0900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:59:18.318 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:18.309+0000 7f6bc4c00640 1 --2- 172.21.15.32:0/3482557408 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5631f7f093b0 0x5631f82d6930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:59:18.318 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:18.310+0000 7f6bc5600640 1 -- 172.21.15.32:0/3482557408 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x5631f7f11590 msgr2=0x5631f82f0900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:59:18.318 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:18.310+0000 7f6bc5600640 1 --2- 172.21.15.32:0/3482557408 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x5631f7f11590 0x5631f82f0900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:59:18.318 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:18.310+0000 7f6bc5600640 1 -- 172.21.15.32:0/3482557408 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5631f7f093b0 msgr2=0x5631f82d6930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:59:18.318 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:18.310+0000 7f6bc5600640 1 --2- 172.21.15.32:0/3482557408 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5631f7f093b0 0x5631f82d6930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:59:18.318 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:18.310+0000 7f6bc5600640 1 -- 172.21.15.32:0/3482557408 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5631f80146b0 con 0x5631f7f32270 2024-08-03T20:59:18.318 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:18.310+0000 7f6bc5600640 1 --2- 172.21.15.32:0/3482557408 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5631f7f32270 0x5631f8313e10 secure :-1 s=READY pgs=1910 cs=0 l=1 rev1=1 crypto rx=0x7f6bb8009370 tx=0x7f6bb801c6b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4c37d0d418002e0f server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:59:18.318 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:18.310+0000 7f6bad600640 10 client.0 ms_handle_connect on v2:172.21.15.32:3300/0 2024-08-03T20:59:18.318 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:18.310+0000 7f6bad600640 1 -- 172.21.15.32:0/3482557408 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6bb801ca60 con 0x5631f7f32270 2024-08-03T20:59:18.318 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:18.310+0000 7f6bce785740 1 -- 172.21.15.32:0/3482557408 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6bb8015070 con 0x5631f7f32270 2024-08-03T20:59:18.318 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:18.310+0000 7f6bad600640 1 -- 172.21.15.32:0/3482557408 <== mon.0 v2:172.21.15.32:3300/0 2 ==== osd_map(118..118 src has 1..118) ==== 8692+0+0 (secure 0 0 0) 0x7f6bb801cc00 con 0x5631f7f32270 2024-08-03T20:59:18.318 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:18.311+0000 7f6bad600640 1 -- 172.21.15.32:0/3482557408 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_get_version_reply(handle=1 version=285) ==== 24+0+0 (secure 0 0 0) 0x7f6bb801ce00 con 0x5631f7f32270 2024-08-03T20:59:18.318 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:18.311+0000 7f6bce785740 10 client.9822 fetch_fsmap learned FSMap version 285 2024-08-03T20:59:18.319 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:18.311+0000 7f6bce785740 1 -- 172.21.15.32:0/3482557408 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,fsmap=285,monmap=2+}) -- 0x5631f8314350 con 0x5631f7f32270 2024-08-03T20:59:18.319 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:18.311+0000 7f6bad600640 1 -- 172.21.15.32:0/3482557408 <== mon.0 v2:172.21.15.32:3300/0 4 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f6bb8015070 con 0x5631f7f32270 2024-08-03T20:59:18.319 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:18.311+0000 7f6bad600640 1 -- 172.21.15.32:0/3482557408 <== mon.0 v2:172.21.15.32:3300/0 5 ==== fsmap(e 285) ==== 3502+0+0 (secure 0 0 0) 0x7f6bb801a070 con 0x5631f7f32270 2024-08-03T20:59:18.319 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:18.311+0000 7f6bce785740 10 client.9822 fetch_fsmap finished waiting for FSMap version 285 2024-08-03T20:59:18.319 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:18.311+0000 7f6bce785740 10 client.9822 resolve_mds: resolved name 'a' to daemon mds.a 2024-08-03T20:59:18.319 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:18.311+0000 7f6bce785740 20 client.9822 populate_metadata read hostname 'smithi032' 2024-08-03T20:59:18.319 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:18.311+0000 7f6bce785740 1 --2- 172.21.15.32:0/3482557408 >> [v2:172.21.15.32:6833/2173042128,v1:172.21.15.32:6835/2173042128] conn(0x5631f7ef0e40 0x5631f832f890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:59:18.319 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:18.311+0000 7f6bce785740 4 client.9822 mds_command: new command op to 9759 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-08-03T20:59:18.319 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:18.311+0000 7f6bce785740 1 -- 172.21.15.32:0/3482557408 --> [v2:172.21.15.32:6833/2173042128,v1:172.21.15.32:6835/2173042128] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5631f824c9b0 con 0x5631f7ef0e40 2024-08-03T20:59:18.319 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:18.311+0000 7f6bc4c00640 1 --2- 172.21.15.32:0/3482557408 >> [v2:172.21.15.32:6833/2173042128,v1:172.21.15.32:6835/2173042128] conn(0x5631f7ef0e40 0x5631f832f890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:59:18.320 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:18.312+0000 7f6bc4c00640 1 --2- 172.21.15.32:0/3482557408 >> [v2:172.21.15.32:6833/2173042128,v1:172.21.15.32:6835/2173042128] conn(0x5631f7ef0e40 0x5631f832f890 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e3e8c4a2fdebfc08 server_cookie=f190f80002d379ec in_seq=0 out_seq=0 2024-08-03T20:59:18.320 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:18.312+0000 7f6bad600640 10 client.9822 ms_handle_connect on v2:172.21.15.32:6833/2173042128 2024-08-03T20:59:18.320 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:18.313+0000 7f6bad600640 1 -- 172.21.15.32:0/3482557408 <== mds.0 v2:172.21.15.32:6833/2173042128 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x5631f824c9b0 con 0x5631f7ef0e40 2024-08-03T20:59:18.320 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:18.313+0000 7f6bad600640 10 client.9822 handle_command_reply: tid=0 2024-08-03T20:59:18.320 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:18.313+0000 7f6bce785740 1 client.9822 shutdown 2024-08-03T20:59:18.320 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:18.313+0000 7f6bce785740 1 -- 172.21.15.32:0/3482557408 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5631f7f32270 msgr2=0x5631f8313e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:59:18.320 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:18.313+0000 7f6bce785740 1 --2- 172.21.15.32:0/3482557408 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5631f7f32270 0x5631f8313e10 secure :-1 s=READY pgs=1910 cs=0 l=1 rev1=1 crypto rx=0x7f6bb8009370 tx=0x7f6bb801c6b0 comp rx=0 tx=0).stop 2024-08-03T20:59:18.320 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:18.313+0000 7f6bce785740 1 -- 172.21.15.32:0/3482557408 shutdown_connections 2024-08-03T20:59:18.320 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:18.313+0000 7f6bce785740 1 --2- 172.21.15.32:0/3482557408 >> [v2:172.21.15.32:6833/2173042128,v1:172.21.15.32:6835/2173042128] conn(0x5631f7ef0e40 0x5631f832f890 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:59:18.321 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:18.313+0000 7f6bce785740 1 --2- 172.21.15.32:0/3482557408 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5631f7f32270 0x5631f8313e10 unknown :-1 s=CLOSED pgs=1910 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:59:18.321 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:18.313+0000 7f6bce785740 1 --2- 172.21.15.32:0/3482557408 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x5631f7f11590 0x5631f82f0900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:59:18.321 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:18.313+0000 7f6bce785740 1 --2- 172.21.15.32:0/3482557408 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5631f7f093b0 0x5631f82d6930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:59:18.321 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:18.313+0000 7f6bad600640 0 client.9822 ms_handle_reset on v2:172.21.15.32:6833/2173042128 2024-08-03T20:59:18.321 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:18.313+0000 7f6bce785740 1 -- 172.21.15.32:0/3482557408 >> 172.21.15.32:0/3482557408 conn(0x5631f7f263c0 msgr2=0x5631f82d2ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:59:18.321 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:18.313+0000 7f6bce785740 1 -- 172.21.15.32:0/3482557408 shutdown_connections 2024-08-03T20:59:18.321 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:18.313+0000 7f6bce785740 1 -- 172.21.15.32:0/3482557408 wait complete. 2024-08-03T20:59:18.321 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:18.314+0000 7f6bce785740 20 client.9822 trim_cache size 0 max 16384 2024-08-03T20:59:18.335 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:18.334+0000 7f6bce785740 1 Processor -- start 2024-08-03T20:59:18.336 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:18.334+0000 7f6bce785740 1 -- start start 2024-08-03T20:59:18.338 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:18.335+0000 7f6bce785740 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x5631f7f093b0 0x5631f82d2b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:59:18.338 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:18.335+0000 7f6bce785740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5631f7f11590 0x5631f838fec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:59:18.338 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:18.335+0000 7f6bce785740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5631f7f32270 0x5631f8390400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:59:18.338 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:18.335+0000 7f6bce785740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f6bc0202d40 con 0x5631f7f32270 2024-08-03T20:59:18.339 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:18.335+0000 7f6bce785740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f6bc02020a0 con 0x5631f7f11590 2024-08-03T20:59:18.339 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:18.335+0000 7f6bce785740 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7f6bc02026f0 con 0x5631f7f093b0 2024-08-03T20:59:18.339 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:18.336+0000 7f6bc5600640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5631f7f32270 0x5631f8390400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:59:18.339 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:18.336+0000 7f6bc5600640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5631f7f32270 0x5631f8390400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:46708/0 (socket says 172.21.15.32:46708) 2024-08-03T20:59:18.339 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:18.336+0000 7f6bc5600640 1 -- 172.21.15.32:0/2139404433 learned_addr learned my addr 172.21.15.32:0/2139404433 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:59:18.339 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:18.336+0000 7f6bc4c00640 1 --2- 172.21.15.32:0/2139404433 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x5631f7f093b0 0x5631f82d2b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:59:18.339 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:18.336+0000 7f6bb7e00640 1 --2- 172.21.15.32:0/2139404433 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5631f7f11590 0x5631f838fec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:59:18.339 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:18.336+0000 7f6bc5600640 1 -- 172.21.15.32:0/2139404433 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x5631f7f093b0 msgr2=0x5631f82d2b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:59:18.339 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:18.336+0000 7f6bc5600640 1 --2- 172.21.15.32:0/2139404433 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x5631f7f093b0 0x5631f82d2b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:59:18.339 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:18.336+0000 7f6bc5600640 1 -- 172.21.15.32:0/2139404433 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5631f7f11590 msgr2=0x5631f838fec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:59:18.340 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:18.336+0000 7f6bc5600640 1 --2- 172.21.15.32:0/2139404433 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5631f7f11590 0x5631f838fec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:59:18.340 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:18.336+0000 7f6bc5600640 1 -- 172.21.15.32:0/2139404433 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5631f8313310 con 0x5631f7f32270 2024-08-03T20:59:18.340 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:18.336+0000 7f6bc5600640 1 --2- 172.21.15.32:0/2139404433 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5631f7f32270 0x5631f8390400 secure :-1 s=READY pgs=1911 cs=0 l=1 rev1=1 crypto rx=0x7f6bb802e040 tx=0x7f6bb80107a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7125adbf7ccef429 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:59:18.340 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:18.337+0000 7f6bafe00640 1 -- 172.21.15.32:0/2139404433 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6bb801ca60 con 0x5631f7f32270 2024-08-03T20:59:18.340 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:18.337+0000 7f6bafe00640 1 -- 172.21.15.32:0/2139404433 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f6bb8020030 con 0x5631f7f32270 2024-08-03T20:59:18.340 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:18.337+0000 7f6bafe00640 1 -- 172.21.15.32:0/2139404433 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6bb8005210 con 0x5631f7f32270 2024-08-03T20:59:18.344 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:18.337+0000 7f6bce785740 1 -- 172.21.15.32:0/2139404433 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5631f7f32270 msgr2=0x5631f8390400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:59:18.344 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:18.337+0000 7f6bce785740 1 --2- 172.21.15.32:0/2139404433 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5631f7f32270 0x5631f8390400 secure :-1 s=READY pgs=1911 cs=0 l=1 rev1=1 crypto rx=0x7f6bb802e040 tx=0x7f6bb80107a0 comp rx=0 tx=0).stop 2024-08-03T20:59:18.344 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:18.337+0000 7f6bce785740 1 -- 172.21.15.32:0/2139404433 shutdown_connections 2024-08-03T20:59:18.344 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:18.337+0000 7f6bce785740 1 --2- 172.21.15.32:0/2139404433 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5631f7f11590 0x5631f838fec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:59:18.344 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:18.337+0000 7f6bce785740 1 --2- 172.21.15.32:0/2139404433 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5631f7f32270 0x5631f8390400 unknown :-1 s=CLOSED pgs=1911 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:59:18.344 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:18.337+0000 7f6bce785740 1 --2- 172.21.15.32:0/2139404433 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x5631f7f093b0 0x5631f82d2b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:59:18.344 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:18.337+0000 7f6bce785740 1 -- 172.21.15.32:0/2139404433 >> 172.21.15.32:0/2139404433 conn(0x5631f7f263c0 msgr2=0x5631f8313c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:59:18.344 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:18.338+0000 7f6bce785740 1 -- 172.21.15.32:0/2139404433 shutdown_connections 2024-08-03T20:59:18.344 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:18.338+0000 7f6bce785740 1 -- 172.21.15.32:0/2139404433 wait complete. 2024-08-03T20:59:18.344 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:18.338+0000 7f6bce785740 1 -- start start 2024-08-03T20:59:18.345 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:18.339+0000 7f6bce785740 1 Processor -- start 2024-08-03T20:59:18.345 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:18.340+0000 7f6bce785740 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x5631f7f093b0 0x5631f838c700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:59:18.345 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:18.340+0000 7f6bce785740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5631f7f11590 0x5631f831c410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:59:18.345 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:18.340+0000 7f6bce785740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5631f7f32270 0x5631f831c950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:59:18.345 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:18.340+0000 7f6bce785740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x5631f8301310 con 0x5631f7f32270 2024-08-03T20:59:18.345 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:18.340+0000 7f6bce785740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x5631f82f0f00 con 0x5631f7f11590 2024-08-03T20:59:18.345 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:18.340+0000 7f6bce785740 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x5631f8316390 con 0x5631f7f093b0 2024-08-03T20:59:18.345 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:18.340+0000 7f6bce785740 10 client.0 _ll_register_callbacks cb 0x7f6ba805c6b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-03T20:59:18.345 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:18.340+0000 7f6bc5600640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5631f7f32270 0x5631f831c950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:59:18.345 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:18.340+0000 7f6bc5600640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5631f7f32270 0x5631f831c950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:46714/0 (socket says 172.21.15.32:46714) 2024-08-03T20:59:18.345 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:18.341+0000 7f6bc5600640 1 -- 172.21.15.32:0/3464342363 learned_addr learned my addr 172.21.15.32:0/3464342363 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:59:18.345 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:18.341+0000 7f6bc4c00640 1 --2- 172.21.15.32:0/3464342363 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x5631f7f093b0 0x5631f838c700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:59:18.345 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:18.341+0000 7f6bb7e00640 1 --2- 172.21.15.32:0/3464342363 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5631f7f11590 0x5631f831c410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:59:18.345 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:18.341+0000 7f6bc5600640 1 -- 172.21.15.32:0/3464342363 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x5631f7f093b0 msgr2=0x5631f838c700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:59:18.346 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:18.341+0000 7f6bc5600640 1 --2- 172.21.15.32:0/3464342363 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x5631f7f093b0 0x5631f838c700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:59:18.346 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:18.341+0000 7f6bc5600640 1 -- 172.21.15.32:0/3464342363 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5631f7f11590 msgr2=0x5631f831c410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:59:18.346 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:18.341+0000 7f6bc5600640 1 --2- 172.21.15.32:0/3464342363 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5631f7f11590 0x5631f831c410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:59:18.346 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:18.341+0000 7f6bc5600640 1 -- 172.21.15.32:0/3464342363 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5631f832b820 con 0x5631f7f32270 2024-08-03T20:59:18.346 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:18.341+0000 7f6bc4c00640 1 --2- 172.21.15.32:0/3464342363 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x5631f7f093b0 0x5631f838c700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-03T20:59:18.346 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:18.341+0000 7f6bc5600640 1 --2- 172.21.15.32:0/3464342363 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5631f7f32270 0x5631f831c950 secure :-1 s=READY pgs=1912 cs=0 l=1 rev1=1 crypto rx=0x7f6bb8000f80 tx=0x7f6bb8036b10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7dd7ff75682ac1cb server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:59:18.346 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:18.341+0000 7f6bad600640 10 client.0 ms_handle_connect on v2:172.21.15.32:3300/0 2024-08-03T20:59:18.346 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:18.341+0000 7f6bad600640 1 -- 172.21.15.32:0/3464342363 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6bb803a070 con 0x5631f7f32270 2024-08-03T20:59:18.347 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:18.342+0000 7f6bad600640 1 -- 172.21.15.32:0/3464342363 <== mon.0 v2:172.21.15.32:3300/0 2 ==== osd_map(118..118 src has 1..118) ==== 8692+0+0 (secure 0 0 0) 0x7f6bb8011440 con 0x5631f7f32270 2024-08-03T20:59:18.347 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:18.342+0000 7f6bce785740 1 -- 172.21.15.32:0/3464342363 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6bb8020030 con 0x5631f7f32270 2024-08-03T20:59:18.347 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:18.342+0000 7f6bad600640 1 -- 172.21.15.32:0/3464342363 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_get_version_reply(handle=1 version=285) ==== 24+0+0 (secure 0 0 0) 0x7f6bb801ca60 con 0x5631f7f32270 2024-08-03T20:59:18.347 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:18.342+0000 7f6bce785740 10 client.9828 fetch_fsmap learned FSMap version 285 2024-08-03T20:59:18.347 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:18.342+0000 7f6bce785740 1 -- 172.21.15.32:0/3464342363 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,fsmap=285,monmap=2+}) -- 0x5631f8336610 con 0x5631f7f32270 2024-08-03T20:59:18.347 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:18.342+0000 7f6bad600640 1 -- 172.21.15.32:0/3464342363 <== mon.0 v2:172.21.15.32:3300/0 4 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f6bb8020030 con 0x5631f7f32270 2024-08-03T20:59:18.347 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:18.342+0000 7f6bad600640 1 -- 172.21.15.32:0/3464342363 <== mon.0 v2:172.21.15.32:3300/0 5 ==== fsmap(e 285) ==== 3502+0+0 (secure 0 0 0) 0x7f6bb8033020 con 0x5631f7f32270 2024-08-03T20:59:18.347 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:18.342+0000 7f6bce785740 10 client.9828 fetch_fsmap finished waiting for FSMap version 285 2024-08-03T20:59:18.348 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:18.342+0000 7f6bce785740 10 client.9828 resolve_mds: resolved name 'a' to daemon mds.a 2024-08-03T20:59:18.348 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:18.342+0000 7f6bce785740 20 client.9828 populate_metadata read hostname 'smithi032' 2024-08-03T20:59:18.348 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:18.342+0000 7f6bce785740 1 --2- 172.21.15.32:0/3464342363 >> [v2:172.21.15.32:6833/2173042128,v1:172.21.15.32:6835/2173042128] conn(0x5631f838de90 0x5631f82d1550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:59:18.348 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:18.342+0000 7f6bce785740 4 client.9828 mds_command: new command op to 9759 tid=0 multi_id=0 [{"prefix": "status"}] 2024-08-03T20:59:18.348 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:18.342+0000 7f6bce785740 1 -- 172.21.15.32:0/3464342363 --> [v2:172.21.15.32:6833/2173042128,v1:172.21.15.32:6835/2173042128] -- command(tid 0: {"prefix": "status"}) -- 0x5631f8248450 con 0x5631f838de90 2024-08-03T20:59:18.348 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:18.343+0000 7f6bc4c00640 1 --2- 172.21.15.32:0/3464342363 >> [v2:172.21.15.32:6833/2173042128,v1:172.21.15.32:6835/2173042128] conn(0x5631f838de90 0x5631f82d1550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:59:18.350 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:18.343+0000 7f6bc4c00640 1 --2- 172.21.15.32:0/3464342363 >> [v2:172.21.15.32:6833/2173042128,v1:172.21.15.32:6835/2173042128] conn(0x5631f838de90 0x5631f82d1550 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=451906de47c0e3a5 server_cookie=7cfbcc227a4071eb in_seq=0 out_seq=0 2024-08-03T20:59:18.350 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:18.343+0000 7f6bad600640 10 client.9828 ms_handle_connect on v2:172.21.15.32:6833/2173042128 2024-08-03T20:59:18.351 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:18.343+0000 7f6bad600640 1 -- 172.21.15.32:0/3464342363 <== mds.0 v2:172.21.15.32:6833/2173042128 1 ==== command_reply(tid 0: 0 ) ==== 8+0+334 (crc 0 0 0) 0x5631f8248450 con 0x5631f838de90 2024-08-03T20:59:18.351 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:18.343+0000 7f6bad600640 10 client.9828 handle_command_reply: tid=0 2024-08-03T20:59:18.351 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:18.343+0000 7f6bce785740 1 client.9828 shutdown 2024-08-03T20:59:18.351 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:18.344+0000 7f6bce785740 1 -- 172.21.15.32:0/3464342363 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5631f7f32270 msgr2=0x5631f831c950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:59:18.351 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:18.344+0000 7f6bce785740 1 --2- 172.21.15.32:0/3464342363 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5631f7f32270 0x5631f831c950 secure :-1 s=READY pgs=1912 cs=0 l=1 rev1=1 crypto rx=0x7f6bb8000f80 tx=0x7f6bb8036b10 comp rx=0 tx=0).stop 2024-08-03T20:59:18.351 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:18.344+0000 7f6bce785740 1 -- 172.21.15.32:0/3464342363 shutdown_connections 2024-08-03T20:59:18.351 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:18.344+0000 7f6bce785740 1 --2- 172.21.15.32:0/3464342363 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5631f7f11590 0x5631f831c410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:59:18.351 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:18.344+0000 7f6bce785740 1 --2- 172.21.15.32:0/3464342363 >> [v2:172.21.15.32:6833/2173042128,v1:172.21.15.32:6835/2173042128] conn(0x5631f838de90 0x5631f82d1550 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:59:18.351 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:18.344+0000 7f6bce785740 1 --2- 172.21.15.32:0/3464342363 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5631f7f32270 0x5631f831c950 unknown :-1 s=CLOSED pgs=1912 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:59:18.351 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:18.344+0000 7f6bce785740 1 --2- 172.21.15.32:0/3464342363 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x5631f7f093b0 0x5631f838c700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:59:18.351 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:18.344+0000 7f6bad600640 0 client.9828 ms_handle_reset on v2:172.21.15.32:6833/2173042128 2024-08-03T20:59:18.352 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:18.344+0000 7f6bce785740 1 -- 172.21.15.32:0/3464342363 >> 172.21.15.32:0/3464342363 conn(0x5631f7f263c0 msgr2=0x5631f832bab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:59:18.352 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:18.344+0000 7f6bce785740 1 -- 172.21.15.32:0/3464342363 shutdown_connections 2024-08-03T20:59:18.352 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:18.344+0000 7f6bce785740 1 -- 172.21.15.32:0/3464342363 wait complete. 2024-08-03T20:59:18.352 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:18.344+0000 7f6bce785740 20 client.9828 trim_cache size 0 max 16384 2024-08-03T20:59:18.352 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-08-03T20:59:18.352 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_fsid": "40ab141a-01d7-49b9-ae3c-2edec9876e95", 2024-08-03T20:59:18.352 INFO:teuthology.orchestra.run.smithi032.stdout: "whoami": 0, 2024-08-03T20:59:18.353 INFO:teuthology.orchestra.run.smithi032.stdout: "id": 9759, 2024-08-03T20:59:18.353 INFO:teuthology.orchestra.run.smithi032.stdout: "want_state": "up:active", 2024-08-03T20:59:18.353 INFO:teuthology.orchestra.run.smithi032.stdout: "state": "up:active", 2024-08-03T20:59:18.353 INFO:teuthology.orchestra.run.smithi032.stdout: "fs_name": "cephfs", 2024-08-03T20:59:18.353 INFO:teuthology.orchestra.run.smithi032.stdout: "rank_uptime": 7.1669568879999996, 2024-08-03T20:59:18.353 INFO:teuthology.orchestra.run.smithi032.stdout: "mdsmap_epoch": 285, 2024-08-03T20:59:18.353 INFO:teuthology.orchestra.run.smithi032.stdout: "osdmap_epoch": 118, 2024-08-03T20:59:18.353 INFO:teuthology.orchestra.run.smithi032.stdout: "osdmap_epoch_barrier": 118, 2024-08-03T20:59:18.353 INFO:teuthology.orchestra.run.smithi032.stdout: "uptime": 10.402298632999999 2024-08-03T20:59:18.353 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-08-03T20:59:18.354 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:18.345+0000 7f6bc7600640 1 -- 172.21.15.32:0/2379126739 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f6b9c077040 msgr2=0x7f6b9c079550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:59:18.354 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:18.345+0000 7f6bc7600640 1 --2- 172.21.15.32:0/2379126739 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f6b9c077040 0x7f6b9c079550 secure :-1 s=READY pgs=769 cs=0 l=1 rev1=1 crypto rx=0x7f6bb0033000 tx=0x7f6bb0004730 comp rx=0 tx=0).stop 2024-08-03T20:59:18.354 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:18.345+0000 7f6bc7600640 1 -- 172.21.15.32:0/2379126739 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6bc0073e60 msgr2=0x7f6bc01eb510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:59:18.354 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:18.345+0000 7f6bc7600640 1 --2- 172.21.15.32:0/2379126739 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6bc0073e60 0x7f6bc01eb510 secure :-1 s=READY pgs=1908 cs=0 l=1 rev1=1 crypto rx=0x7f6ba800c340 tx=0x7f6ba800c800 comp rx=0 tx=0).stop 2024-08-03T20:59:18.355 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:18.345+0000 7f6bc7600640 1 -- 172.21.15.32:0/2379126739 shutdown_connections 2024-08-03T20:59:18.355 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:18.345+0000 7f6bc7600640 1 --2- 172.21.15.32:0/2379126739 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f6b9c077040 0x7f6b9c079550 unknown :-1 s=CLOSED pgs=769 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:59:18.355 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:18.345+0000 7f6bc7600640 1 --2- 172.21.15.32:0/2379126739 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f6bc015fff0 0x7f6bc01f8cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:59:18.355 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:18.345+0000 7f6bc7600640 1 --2- 172.21.15.32:0/2379126739 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6bc0073e60 0x7f6bc01eb510 unknown :-1 s=CLOSED pgs=1908 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:59:18.355 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:18.345+0000 7f6bc7600640 1 --2- 172.21.15.32:0/2379126739 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f6bc006e930 0x7f6bc015cd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:59:18.355 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:18.345+0000 7f6bc7600640 1 -- 172.21.15.32:0/2379126739 >> 172.21.15.32:0/2379126739 conn(0x7f6bc00b1bd0 msgr2=0x7f6bc00b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:59:18.355 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:18.345+0000 7f6bc7600640 1 -- 172.21.15.32:0/2379126739 shutdown_connections 2024-08-03T20:59:18.355 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:18.345+0000 7f6bc7600640 1 -- 172.21.15.32:0/2379126739 wait complete. 2024-08-03T20:59:18.370 DEBUG:tasks.ceph_test_case:wait_until_true: success in 5s and 0 retries 2024-08-03T20:59:18.370 INFO:tasks.cephfs.test_damage:Daemons came up after mutation 'Corrupt omap key 1.00000000:subdir_head', proceeding to ls 2024-08-03T20:59:18.370 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-08-03T20:59:18.600 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:18.597+0000 7fc585c00640 1 -- 172.21.15.32:0/3461971366 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc5800b4ce0 msgr2=0x7fc58015ef00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:59:18.600 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:18.597+0000 7fc585c00640 1 --2- 172.21.15.32:0/3461971366 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc5800b4ce0 0x7fc58015ef00 secure :-1 s=READY pgs=1914 cs=0 l=1 rev1=1 crypto rx=0x7fc56c009830 tx=0x7fc56c02a510 comp rx=0 tx=0).stop 2024-08-03T20:59:18.600 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:18.598+0000 7fc585c00640 1 -- 172.21.15.32:0/3461971366 shutdown_connections 2024-08-03T20:59:18.600 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:18.598+0000 7fc585c00640 1 --2- 172.21.15.32:0/3461971366 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc5800b4ce0 0x7fc58015ef00 unknown :-1 s=CLOSED pgs=1914 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:59:18.600 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:18.598+0000 7fc585c00640 1 --2- 172.21.15.32:0/3461971366 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fc5800b38b0 0x7fc5801523f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:59:18.601 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:18.598+0000 7fc585c00640 1 -- 172.21.15.32:0/3461971366 >> 172.21.15.32:0/3461971366 conn(0x7fc5800b1bd0 msgr2=0x7fc5800b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:59:18.601 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:18.598+0000 7fc585c00640 1 -- 172.21.15.32:0/3461971366 shutdown_connections 2024-08-03T20:59:18.601 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:18.598+0000 7fc585c00640 1 -- 172.21.15.32:0/3461971366 wait complete. 2024-08-03T20:59:18.601 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:18.600+0000 7fc585c00640 1 Processor -- start 2024-08-03T20:59:18.601 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:18.600+0000 7fc585c00640 1 -- start start 2024-08-03T20:59:18.604 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:18.602+0000 7fc585c00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc5800b38b0 0x7fc580155530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:59:18.604 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:18.602+0000 7fc585c00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc5800b4ce0 0x7fc580151dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:59:18.604 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:18.602+0000 7fc585c00640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fc58015f440 0x7fc580072eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:59:18.604 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:18.602+0000 7fc585c00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fc58015c020 con 0x7fc5800b4ce0 2024-08-03T20:59:18.604 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:18.602+0000 7fc585c00640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7fc5800cf240 con 0x7fc5800b38b0 2024-08-03T20:59:18.604 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:18.602+0000 7fc585c00640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7fc58015c1a0 con 0x7fc58015f440 2024-08-03T20:59:18.604 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:18.602+0000 7fc57e000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc5800b4ce0 0x7fc580151dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:59:18.605 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:18.602+0000 7fc57e000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc5800b4ce0 0x7fc580151dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:46742/0 (socket says 172.21.15.32:46742) 2024-08-03T20:59:18.605 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:18.602+0000 7fc57e000640 1 -- 172.21.15.32:0/3010161937 learned_addr learned my addr 172.21.15.32:0/3010161937 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:59:18.605 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:18.602+0000 7fc57ea00640 1 --2- 172.21.15.32:0/3010161937 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc5800b38b0 0x7fc580155530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:59:18.605 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:18.602+0000 7fc57f400640 1 --2- 172.21.15.32:0/3010161937 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fc58015f440 0x7fc580072eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:59:18.605 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:18.602+0000 7fc57e000640 1 -- 172.21.15.32:0/3010161937 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fc58015f440 msgr2=0x7fc580072eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:59:18.605 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:18.602+0000 7fc57e000640 1 --2- 172.21.15.32:0/3010161937 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fc58015f440 0x7fc580072eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:59:18.605 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:18.602+0000 7fc57e000640 1 -- 172.21.15.32:0/3010161937 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc5800b38b0 msgr2=0x7fc580155530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:59:18.605 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:18.602+0000 7fc57e000640 1 --2- 172.21.15.32:0/3010161937 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc5800b38b0 0x7fc580155530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:59:18.605 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:18.602+0000 7fc57e000640 1 -- 172.21.15.32:0/3010161937 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc580155fd0 con 0x7fc5800b4ce0 2024-08-03T20:59:18.605 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:18.603+0000 7fc57e000640 1 --2- 172.21.15.32:0/3010161937 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc5800b4ce0 0x7fc580151dc0 secure :-1 s=READY pgs=1915 cs=0 l=1 rev1=1 crypto rx=0x7fc56c009950 tx=0x7fc56c033c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:59:18.606 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:18.603+0000 7fc576a00640 1 -- 172.21.15.32:0/3010161937 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc56c005c90 con 0x7fc5800b4ce0 2024-08-03T20:59:18.607 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:18.603+0000 7fc576a00640 1 -- 172.21.15.32:0/3010161937 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fc56c031e50 con 0x7fc5800b4ce0 2024-08-03T20:59:18.607 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:18.603+0000 7fc585c00640 1 -- 172.21.15.32:0/3010161937 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc5801f9c10 con 0x7fc5800b4ce0 2024-08-03T20:59:18.607 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:18.603+0000 7fc576a00640 1 -- 172.21.15.32:0/3010161937 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc56c005260 con 0x7fc5800b4ce0 2024-08-03T20:59:18.607 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:18.603+0000 7fc585c00640 1 -- 172.21.15.32:0/3010161937 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc5801fa1d0 con 0x7fc5800b4ce0 2024-08-03T20:59:18.607 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:18.604+0000 7fc585c00640 1 -- 172.21.15.32:0/3010161937 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc5801f9980 con 0x7fc5800b4ce0 2024-08-03T20:59:18.608 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:18.605+0000 7fc576a00640 1 -- 172.21.15.32:0/3010161937 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7fc56c005400 con 0x7fc5800b4ce0 2024-08-03T20:59:18.608 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:18.606+0000 7fc576a00640 1 --2- 172.21.15.32:0/3010161937 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fc560077010 0x7fc560079520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:59:18.608 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:18.606+0000 7fc576a00640 1 -- 172.21.15.32:0/3010161937 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(118..118 src has 1..118) ==== 8692+0+0 (secure 0 0 0) 0x7fc56c0ba530 con 0x7fc5800b4ce0 2024-08-03T20:59:18.608 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:18.607+0000 7fc57ea00640 1 --2- 172.21.15.32:0/3010161937 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fc560077010 0x7fc560079520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:59:18.609 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:18.607+0000 7fc57ea00640 1 --2- 172.21.15.32:0/3010161937 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fc560077010 0x7fc560079520 secure :-1 s=READY pgs=770 cs=0 l=1 rev1=1 crypto rx=0x7fc570009b30 tx=0x7fc570005f90 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:59:18.612 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:18.611+0000 7fc576a00640 1 -- 172.21.15.32:0/3010161937 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7fc56c03f030 con 0x7fc5800b4ce0 2024-08-03T20:59:18.861 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:18.859+0000 7fc585c00640 1 -- 172.21.15.32:0/3010161937 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7fc58015c650 con 0x7fc5800b4ce0 2024-08-03T20:59:18.861 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:18.860+0000 7fc576a00640 1 -- 172.21.15.32:0/3010161937 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v285) ==== 53+0+73 (secure 0 0 0) 0x7fc56c082510 con 0x7fc5800b4ce0 2024-08-03T20:59:18.862 INFO:teuthology.orchestra.run.smithi032.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-08-03T20:59:18.864 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:18.862+0000 7fc585c00640 1 -- 172.21.15.32:0/3010161937 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fc560077010 msgr2=0x7fc560079520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:59:18.864 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:18.862+0000 7fc585c00640 1 --2- 172.21.15.32:0/3010161937 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fc560077010 0x7fc560079520 secure :-1 s=READY pgs=770 cs=0 l=1 rev1=1 crypto rx=0x7fc570009b30 tx=0x7fc570005f90 comp rx=0 tx=0).stop 2024-08-03T20:59:18.864 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:18.862+0000 7fc585c00640 1 -- 172.21.15.32:0/3010161937 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc5800b4ce0 msgr2=0x7fc580151dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:59:18.864 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:18.862+0000 7fc585c00640 1 --2- 172.21.15.32:0/3010161937 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc5800b4ce0 0x7fc580151dc0 secure :-1 s=READY pgs=1915 cs=0 l=1 rev1=1 crypto rx=0x7fc56c009950 tx=0x7fc56c033c20 comp rx=0 tx=0).stop 2024-08-03T20:59:18.864 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:18.862+0000 7fc585c00640 1 -- 172.21.15.32:0/3010161937 shutdown_connections 2024-08-03T20:59:18.864 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:18.862+0000 7fc585c00640 1 --2- 172.21.15.32:0/3010161937 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fc560077010 0x7fc560079520 unknown :-1 s=CLOSED pgs=770 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:59:18.864 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:18.862+0000 7fc585c00640 1 --2- 172.21.15.32:0/3010161937 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fc58015f440 0x7fc580072eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:59:18.864 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:18.862+0000 7fc585c00640 1 --2- 172.21.15.32:0/3010161937 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc5800b4ce0 0x7fc580151dc0 unknown :-1 s=CLOSED pgs=1915 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:59:18.865 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:18.862+0000 7fc585c00640 1 --2- 172.21.15.32:0/3010161937 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc5800b38b0 0x7fc580155530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:59:18.865 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:18.862+0000 7fc585c00640 1 -- 172.21.15.32:0/3010161937 >> 172.21.15.32:0/3010161937 conn(0x7fc5800b1bd0 msgr2=0x7fc5800aea20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:59:18.865 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:18.862+0000 7fc585c00640 1 -- 172.21.15.32:0/3010161937 shutdown_connections 2024-08-03T20:59:18.865 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:18.862+0000 7fc585c00640 1 -- 172.21.15.32:0/3010161937 wait complete. 2024-08-03T20:59:18.890 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-08-03T20:59:18.890 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-08-03T20:59:18.891 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi032.front.sepia.ceph.com 2024-08-03T20:59:18.891 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-08-03T20:59:18.891 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-08-03T20:59:18.891 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-08-03T20:59:18.891 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2024-08-03T20:59:18.891 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-08-03T20:59:18.891 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-08-03T20:59:18.892 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-08-03T20:59:18.892 DEBUG:teuthology.orchestra.run.smithi032:> ip addr 2024-08-03T20:59:18.928 INFO:teuthology.orchestra.run.smithi032.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-08-03T20:59:18.928 INFO:teuthology.orchestra.run.smithi032.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-08-03T20:59:18.928 INFO:teuthology.orchestra.run.smithi032.stdout: inet 127.0.0.1/8 scope host lo 2024-08-03T20:59:18.929 INFO:teuthology.orchestra.run.smithi032.stdout: valid_lft forever preferred_lft forever 2024-08-03T20:59:18.929 INFO:teuthology.orchestra.run.smithi032.stdout: inet6 ::1/128 scope host 2024-08-03T20:59:18.929 INFO:teuthology.orchestra.run.smithi032.stdout: valid_lft forever preferred_lft forever 2024-08-03T20:59:18.929 INFO:teuthology.orchestra.run.smithi032.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-08-03T20:59:18.929 INFO:teuthology.orchestra.run.smithi032.stdout: link/ether 0c:c4:7a:6b:fd:0c brd ff:ff:ff:ff:ff:ff 2024-08-03T20:59:18.929 INFO:teuthology.orchestra.run.smithi032.stdout: altname enp7s0f0 2024-08-03T20:59:18.929 INFO:teuthology.orchestra.run.smithi032.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-08-03T20:59:18.929 INFO:teuthology.orchestra.run.smithi032.stdout: link/ether 0c:c4:7a:6b:fd:0d brd ff:ff:ff:ff:ff:ff 2024-08-03T20:59:18.929 INFO:teuthology.orchestra.run.smithi032.stdout: altname enp7s0f1 2024-08-03T20:59:18.929 INFO:teuthology.orchestra.run.smithi032.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-08-03T20:59:18.929 INFO:teuthology.orchestra.run.smithi032.stdout: link/ether ac:1f:6b:f8:26:02 brd ff:ff:ff:ff:ff:ff 2024-08-03T20:59:18.929 INFO:teuthology.orchestra.run.smithi032.stdout: altname enp1s0f0 2024-08-03T20:59:18.929 INFO:teuthology.orchestra.run.smithi032.stdout: inet 172.21.15.32/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-08-03T20:59:18.930 INFO:teuthology.orchestra.run.smithi032.stdout: valid_lft 41797sec preferred_lft 41797sec 2024-08-03T20:59:18.930 INFO:teuthology.orchestra.run.smithi032.stdout: inet6 fe80::ae1f:6bff:fef8:2602/64 scope link noprefixroute 2024-08-03T20:59:18.930 INFO:teuthology.orchestra.run.smithi032.stdout: valid_lft forever preferred_lft forever 2024-08-03T20:59:18.930 INFO:teuthology.orchestra.run.smithi032.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-08-03T20:59:18.930 INFO:teuthology.orchestra.run.smithi032.stdout: link/ether ac:1f:6b:f8:26:03 brd ff:ff:ff:ff:ff:ff 2024-08-03T20:59:18.930 INFO:teuthology.orchestra.run.smithi032.stdout: altname enp1s0f1 2024-08-03T20:59:18.930 INFO:teuthology.orchestra.run.smithi032.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-08-03T20:59:18.930 INFO:teuthology.orchestra.run.smithi032.stdout: link/ether 0a:23:2b:ca:2a:93 brd ff:ff:ff:ff:ff:ff 2024-08-03T20:59:18.930 INFO:teuthology.orchestra.run.smithi032.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-08-03T20:59:18.930 INFO:teuthology.orchestra.run.smithi032.stdout: valid_lft forever preferred_lft forever 2024-08-03T20:59:18.930 INFO:teuthology.orchestra.run.smithi032.stdout: inet6 fe80::d804:7cff:fed9:7ba4/64 scope link 2024-08-03T20:59:18.930 INFO:teuthology.orchestra.run.smithi032.stdout: valid_lft forever preferred_lft forever 2024-08-03T20:59:18.931 INFO:teuthology.orchestra.run.smithi032.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-08-03T20:59:18.931 INFO:teuthology.orchestra.run.smithi032.stdout: link/ether 22:86:28:30:4e:62 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-08-03T20:59:18.931 INFO:teuthology.orchestra.run.smithi032.stdout: inet6 fe80::2086:28ff:fe30:4e62/64 scope link 2024-08-03T20:59:18.931 INFO:teuthology.orchestra.run.smithi032.stdout: valid_lft forever preferred_lft forever 2024-08-03T20:59:18.931 INFO:teuthology.orchestra.run.smithi032.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-08-03T20:59:18.931 INFO:teuthology.orchestra.run.smithi032.stdout: link/ether 0a:23:2b:ca:2a:93 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-08-03T20:59:18.931 INFO:teuthology.orchestra.run.smithi032.stdout: inet6 fe80::823:2bff:feca:2a93/64 scope link 2024-08-03T20:59:18.931 INFO:teuthology.orchestra.run.smithi032.stdout: valid_lft forever preferred_lft forever 2024-08-03T20:59:18.931 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-08-03T20:59:18.931 DEBUG:teuthology.orchestra.run.smithi032:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-08-03T20:59:19.022 INFO:teuthology.orchestra.run.smithi032.stdout:1 2024-08-03T20:59:19.025 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-08-03T20:59:19.025 DEBUG:teuthology.orchestra.run.smithi032:> route 2024-08-03T20:59:19.062 INFO:teuthology.orchestra.run.smithi032.stdout:Kernel IP routing table 2024-08-03T20:59:19.062 INFO:teuthology.orchestra.run.smithi032.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-08-03T20:59:19.062 INFO:teuthology.orchestra.run.smithi032.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-08-03T20:59:19.062 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-08-03T20:59:19.062 INFO:teuthology.orchestra.run.smithi032.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-08-03T20:59:19.063 DEBUG:teuthology.orchestra.run.smithi032:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-08-03T20:59:19.063 DEBUG:teuthology.orchestra.run.smithi032:> set -e 2024-08-03T20:59:19.063 DEBUG:teuthology.orchestra.run.smithi032:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-08-03T20:59:19.063 DEBUG:teuthology.orchestra.run.smithi032:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-08-03T20:59:19.063 DEBUG:teuthology.orchestra.run.smithi032:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-08-03T20:59:19.063 DEBUG:teuthology.orchestra.run.smithi032:> ') 2024-08-03T20:59:19.181 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:19 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-08-03T20:59:19.253 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:19 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-08-03T20:59:19.261 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-08-03T20:59:19.261 DEBUG:teuthology.orchestra.run.smithi032:> ip netns list 2024-08-03T20:59:19.297 INFO:teuthology.orchestra.run.smithi032.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-08-03T20:59:19.298 INFO:teuthology.orchestra.run.smithi032.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-08-03T20:59:19.298 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-08-03T20:59:19.298 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-08-03T20:59:19.298 DEBUG:teuthology.orchestra.run.smithi032:> sudo ip link set brx.0 up 2024-08-03T20:59:19.394 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-08-03T20:59:19.394 DEBUG:teuthology.orchestra.run.smithi032:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-08-03T20:59:19.429 INFO:teuthology.orchestra.run.smithi032.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-08-03T20:59:19.429 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-08-03T20:59:19.429 DEBUG:teuthology.orchestra.run.smithi032:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-08-03T20:59:19.504 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2024-08-03T20:59:19.505 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-08-03T20:59:19.505 DEBUG:teuthology.orchestra.run.smithi032:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-08-03T20:59:19.707 INFO:teuthology.orchestra.run.smithi032.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-08-03T20:59:19.707 INFO:teuthology.orchestra.run.smithi032.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-08-03T20:59:19.707 INFO:teuthology.orchestra.run.smithi032.stdout:invalid new device string format 2024-08-03T20:59:19.723 INFO:teuthology.orchestra.run.smithi032.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-08-03T20:59:19.724 INFO:teuthology.orchestra.run.smithi032.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-08-03T20:59:19.724 INFO:teuthology.orchestra.run.smithi032.stdout:invalid new device string format 2024-08-03T20:59:19.724 INFO:teuthology.orchestra.run.smithi032.stdout:mount.ceph: resolved to: "172.21.15.32:3300,172.21.15.118:3300,172.21.15.118:3301" 2024-08-03T20:59:19.724 INFO:teuthology.orchestra.run.smithi032.stdout:mount.ceph: switching to using v1 address with old syntax 2024-08-03T20:59:19.756 INFO:teuthology.orchestra.run.smithi032.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-08-03T20:59:19.756 INFO:teuthology.orchestra.run.smithi032.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-08-03T20:59:19.756 INFO:teuthology.orchestra.run.smithi032.stdout:invalid new device string format 2024-08-03T20:59:19.756 INFO:teuthology.orchestra.run.smithi032.stdout:mount.ceph: resolved to: "172.21.15.32:3300,172.21.15.118:3300,172.21.15.118:3301" 2024-08-03T20:59:19.757 INFO:teuthology.orchestra.run.smithi032.stdout:mount.ceph: switching to using v1 address with old syntax 2024-08-03T20:59:19.757 INFO:teuthology.orchestra.run.smithi032.stdout:mount.ceph: resolved to: "172.21.15.32:6789,172.21.15.118:6789,172.21.15.118:6790" 2024-08-03T20:59:19.757 INFO:teuthology.orchestra.run.smithi032.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.32:6789,172.21.15.118:6789,172.21.15.118:6790:/ 2024-08-03T20:59:19.757 INFO:teuthology.orchestra.run.smithi032.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,key=0,fsid=40ab141a-01d7-49b9-ae3c-2edec9876e95" will pass to kernel 2024-08-03T20:59:19.758 INFO:teuthology.orchestra.run.smithi032.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2024-08-03T20:59:19.758 INFO:teuthology.orchestra.run.smithi032.stdout: You just mounted a file system that supports labels which does not 2024-08-03T20:59:19.758 INFO:teuthology.orchestra.run.smithi032.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-08-03T20:59:19.758 INFO:teuthology.orchestra.run.smithi032.stdout: applications will generate AVC messages and not be allowed access to 2024-08-03T20:59:19.758 INFO:teuthology.orchestra.run.smithi032.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-08-03T20:59:19.761 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-08-03T20:59:19.761 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-08-03T20:59:19.761 DEBUG:teuthology.orchestra.run.smithi032:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-08-03T20:59:19.816 DEBUG:teuthology.orchestra.run.smithi032:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-08-03T20:59:19.897 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:19 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-08-03T20:59:19.900 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:19 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-08-03T20:59:19.908 INFO:teuthology.orchestra.run.smithi032.stdout:client9840 2024-08-03T20:59:19.909 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-08-03T20:59:19.909 DEBUG:teuthology.orchestra.run.smithi032:> dd if=/proc/self/mounts of=/dev/stdout 2024-08-03T20:59:19.944 DEBUG:teuthology.orchestra.run.smithi032:> stdin-killer -- python3 -c ' 2024-08-03T20:59:19.944 DEBUG:teuthology.orchestra.run.smithi032:> import os 2024-08-03T20:59:19.944 DEBUG:teuthology.orchestra.run.smithi032:> import stat 2024-08-03T20:59:19.944 DEBUG:teuthology.orchestra.run.smithi032:> import json 2024-08-03T20:59:19.944 DEBUG:teuthology.orchestra.run.smithi032:> import sys 2024-08-03T20:59:19.944 DEBUG:teuthology.orchestra.run.smithi032:> 2024-08-03T20:59:19.944 DEBUG:teuthology.orchestra.run.smithi032:> try: 2024-08-03T20:59:19.944 DEBUG:teuthology.orchestra.run.smithi032:> s = os.stat('"'"'/home/ubuntu/cephtest/mnt.0/./subdir'"'"') 2024-08-03T20:59:19.944 DEBUG:teuthology.orchestra.run.smithi032:> except OSError as e: 2024-08-03T20:59:19.944 DEBUG:teuthology.orchestra.run.smithi032:> sys.exit(e.errno) 2024-08-03T20:59:19.945 DEBUG:teuthology.orchestra.run.smithi032:> 2024-08-03T20:59:19.945 DEBUG:teuthology.orchestra.run.smithi032:> attrs = ["st_mode", "st_ino", "st_dev", "st_nlink", "st_uid", "st_gid", "st_size", "st_atime", "st_mtime", "st_ctime"] 2024-08-03T20:59:19.945 DEBUG:teuthology.orchestra.run.smithi032:> print(json.dumps( 2024-08-03T20:59:19.945 DEBUG:teuthology.orchestra.run.smithi032:> dict([(a, getattr(s, a)) for a in attrs]), 2024-08-03T20:59:19.945 DEBUG:teuthology.orchestra.run.smithi032:> indent=2)) 2024-08-03T20:59:19.945 DEBUG:teuthology.orchestra.run.smithi032:> ' 2024-08-03T20:59:19.988 INFO:teuthology.orchestra.run:waiting for 20 2024-08-03T20:59:20.090 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:20 stdin-killer INFO: command exited with status 5: exiting normally with same code! 2024-08-03T20:59:21.490 INFO:tasks.ceph.mon.a.smithi032.stderr:2024-08-03T20:59:21.489+0000 7f4e70800640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 MDSs report damaged metadata (MDS_DAMAGE) 2024-08-03T20:59:26.090 DEBUG:teuthology.orchestra.run:got remote process result: 5 2024-08-03T20:59:26.090 INFO:tasks.cephfs.test_damage:Result: EIO on client 2024-08-03T20:59:26.090 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-08-03T20:59:26.331 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:26.329+0000 7f2797c00640 1 -- 172.21.15.32:0/4106794357 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2778003c10 msgr2=0x7f2778004060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:59:26.332 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:26.329+0000 7f2797c00640 1 --2- 172.21.15.32:0/4106794357 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2778003c10 0x7f2778004060 secure :-1 s=READY pgs=1918 cs=0 l=1 rev1=1 crypto rx=0x7f2784006fb0 tx=0x7f2778005830 comp rx=0 tx=0).stop 2024-08-03T20:59:26.332 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:26.329+0000 7f2795c00640 1 -- 172.21.15.32:0/4106794357 reap_dead start 2024-08-03T20:59:26.332 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:26.330+0000 7f2797c00640 1 -- 172.21.15.32:0/4106794357 shutdown_connections 2024-08-03T20:59:26.332 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:26.330+0000 7f2797c00640 1 -- 172.21.15.32:0/4106794357 >> 172.21.15.32:0/4106794357 conn(0x7f27900b1bd0 msgr2=0x7f27900b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:59:26.332 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:26.330+0000 7f2797c00640 1 -- 172.21.15.32:0/4106794357 shutdown_connections 2024-08-03T20:59:26.332 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:26.330+0000 7f2797c00640 1 -- 172.21.15.32:0/4106794357 wait complete. 2024-08-03T20:59:26.333 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:26.332+0000 7f2797c00640 1 Processor -- start 2024-08-03T20:59:26.333 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:26.332+0000 7f2797c00640 1 -- start start 2024-08-03T20:59:26.335 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:26.334+0000 7f2797c00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2778003c10 0x7f2790148980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:59:26.336 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:26.334+0000 7f2797c00640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f2790159230 0x7f27900ce150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:59:26.336 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:26.334+0000 7f2797c00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f279014be00 0x7f27900ce690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:59:26.336 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:26.334+0000 7f2797c00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f27900ae9a0 con 0x7f2778003c10 2024-08-03T20:59:26.336 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:26.334+0000 7f2797c00640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f279006fe80 con 0x7f279014be00 2024-08-03T20:59:26.336 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:26.334+0000 7f2797c00640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7f27900aeb20 con 0x7f2790159230 2024-08-03T20:59:26.336 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:26.334+0000 7f2795200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2778003c10 0x7f2790148980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:59:26.336 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:26.334+0000 7f2795200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2778003c10 0x7f2790148980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:41432/0 (socket says 172.21.15.32:41432) 2024-08-03T20:59:26.336 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:26.334+0000 7f2795200640 1 -- 172.21.15.32:0/616989284 learned_addr learned my addr 172.21.15.32:0/616989284 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:59:26.336 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:26.334+0000 7f2795200640 1 -- 172.21.15.32:0/616989284 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f2790159230 msgr2=0x7f27900ce150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:59:26.336 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:26.334+0000 7f2795200640 1 --2- 172.21.15.32:0/616989284 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f2790159230 0x7f27900ce150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:59:26.336 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:26.334+0000 7f2795200640 1 -- 172.21.15.32:0/616989284 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f279014be00 msgr2=0x7f27900ce690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:59:26.336 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:26.334+0000 7f2795200640 1 --2- 172.21.15.32:0/616989284 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f279014be00 0x7f27900ce690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:59:26.336 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:26.334+0000 7f2795200640 1 -- 172.21.15.32:0/616989284 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f27900ccb10 con 0x7f2778003c10 2024-08-03T20:59:26.337 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:26.335+0000 7f2795200640 1 --2- 172.21.15.32:0/616989284 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2778003c10 0x7f2790148980 secure :-1 s=READY pgs=1919 cs=0 l=1 rev1=1 crypto rx=0x7f2784006fb0 tx=0x7f2784006c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:59:26.337 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:26.335+0000 7f278cc00640 1 -- 172.21.15.32:0/616989284 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2784003a00 con 0x7f2778003c10 2024-08-03T20:59:26.338 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:26.335+0000 7f278cc00640 1 -- 172.21.15.32:0/616989284 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f2784003ba0 con 0x7f2778003c10 2024-08-03T20:59:26.338 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:26.335+0000 7f278cc00640 1 -- 172.21.15.32:0/616989284 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2784006540 con 0x7f2778003c10 2024-08-03T20:59:26.338 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:26.335+0000 7f2797c00640 1 -- 172.21.15.32:0/616989284 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f27900d0550 con 0x7f2778003c10 2024-08-03T20:59:26.338 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:26.335+0000 7f2797c00640 1 -- 172.21.15.32:0/616989284 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f27900d0b10 con 0x7f2778003c10 2024-08-03T20:59:26.338 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:26.336+0000 7f2797c00640 1 -- 172.21.15.32:0/616989284 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f27900af0b0 con 0x7f2778003c10 2024-08-03T20:59:26.338 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:26.337+0000 7f278cc00640 1 -- 172.21.15.32:0/616989284 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f2784035020 con 0x7f2778003c10 2024-08-03T20:59:26.340 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:26.338+0000 7f278cc00640 1 --2- 172.21.15.32:0/616989284 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f2770077010 0x7f2770079520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:59:26.340 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:26.338+0000 7f278cc00640 1 -- 172.21.15.32:0/616989284 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(118..118 src has 1..118) ==== 8692+0+0 (secure 0 0 0) 0x7f27840b9fe0 con 0x7f2778003c10 2024-08-03T20:59:26.340 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:26.339+0000 7f278fe00640 1 --2- 172.21.15.32:0/616989284 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f2770077010 0x7f2770079520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:59:26.340 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:26.339+0000 7f278fe00640 1 --2- 172.21.15.32:0/616989284 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f2770077010 0x7f2770079520 secure :-1 s=READY pgs=771 cs=0 l=1 rev1=1 crypto rx=0x7f27900b3ac0 tx=0x7f277c009430 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:59:26.344 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:26.343+0000 7f278cc00640 1 -- 172.21.15.32:0/616989284 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7f27840826e0 con 0x7f2778003c10 2024-08-03T20:59:26.597 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:26.595+0000 7f2797c00640 1 -- 172.21.15.32:0/616989284 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f27900d0d50 con 0x7f2778003c10 2024-08-03T20:59:26.622 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:26.596+0000 7f278cc00640 1 -- 172.21.15.32:0/616989284 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 286 v286) ==== 95+0+5233 (secure 0 0 0) 0x7f2784081fe0 con 0x7f2778003c10 2024-08-03T20:59:26.623 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-03T20:59:26.623 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":286,"btime":"2024-08-03T20:59:21:490368+0000","default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6439,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6835/531080160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6834","nonce":531080160},{"type":"v1","addr":"172.21.15.118:6835","nonce":531080160}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":84},{"gid":6454,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6837/2796882319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6836","nonce":2796882319},{"type":"v1","addr":"172.21.15.118:6837","nonce":2796882319}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":86},{"gid":6813,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6834/2686933890","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":2686933890},{"type":"v1","addr":"172.21.15.32:6834","nonce":2686933890}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":85}],"filesystems":[{"mdsmap":{"epoch":286,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-08-03T20:54:26.950518+0000","modified":"2024-08-03T20:59:21.186082+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":118,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":9759},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9759":{"gid":9759,"name":"a","rank":0,"incarnation":282,"state":"up:active","state_seq":6,"addr":"172.21.15.32:6835/2173042128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6833","nonce":2173042128},{"type":"v1","addr":"172.21.15.32:6835","nonce":2173042128}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9759,"qdb_cluster":[9759]},"id":6}]} 2024-08-03T20:59:26.624 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 286 2024-08-03T20:59:26.624 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:26.598+0000 7f2797c00640 1 -- 172.21.15.32:0/616989284 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f2770077010 msgr2=0x7f2770079520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:59:26.624 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:26.598+0000 7f2797c00640 1 --2- 172.21.15.32:0/616989284 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f2770077010 0x7f2770079520 secure :-1 s=READY pgs=771 cs=0 l=1 rev1=1 crypto rx=0x7f27900b3ac0 tx=0x7f277c009430 comp rx=0 tx=0).stop 2024-08-03T20:59:26.624 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:26.598+0000 7f2797c00640 1 -- 172.21.15.32:0/616989284 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2778003c10 msgr2=0x7f2790148980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:59:26.624 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:26.598+0000 7f2797c00640 1 --2- 172.21.15.32:0/616989284 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2778003c10 0x7f2790148980 secure :-1 s=READY pgs=1919 cs=0 l=1 rev1=1 crypto rx=0x7f2784006fb0 tx=0x7f2784006c40 comp rx=0 tx=0).stop 2024-08-03T20:59:26.624 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:26.598+0000 7f2797c00640 1 -- 172.21.15.32:0/616989284 shutdown_connections 2024-08-03T20:59:26.624 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:26.598+0000 7f2797c00640 1 --2- 172.21.15.32:0/616989284 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f2770077010 0x7f2770079520 unknown :-1 s=CLOSED pgs=771 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:59:26.624 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:26.598+0000 7f2797c00640 1 --2- 172.21.15.32:0/616989284 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f279014be00 0x7f27900ce690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:59:26.624 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:26.598+0000 7f2797c00640 1 --2- 172.21.15.32:0/616989284 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f2790159230 0x7f27900ce150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:59:26.624 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:26.598+0000 7f2797c00640 1 --2- 172.21.15.32:0/616989284 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2778003c10 0x7f2790148980 unknown :-1 s=CLOSED pgs=1919 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:59:26.624 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:26.598+0000 7f2797c00640 1 -- 172.21.15.32:0/616989284 >> 172.21.15.32:0/616989284 conn(0x7f27900b1bd0 msgr2=0x7f279015cc70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:59:26.625 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:26.598+0000 7f2797c00640 1 -- 172.21.15.32:0/616989284 shutdown_connections 2024-08-03T20:59:26.625 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:26.598+0000 7f2797c00640 1 -- 172.21.15.32:0/616989284 wait complete. 2024-08-03T20:59:26.627 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a damage ls --format=json-pretty 2024-08-03T20:59:26.858 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:26.856+0000 7f595d200640 1 -- 172.21.15.32:0/37718752 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f593c003ff0 msgr2=0x7f593c004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:59:26.859 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:26.856+0000 7f595d200640 1 --2- 172.21.15.32:0/37718752 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f593c003ff0 0x7f593c004800 secure :-1 s=READY pgs=1921 cs=0 l=1 rev1=1 crypto rx=0x7f594000c260 tx=0x7f593c005830 comp rx=0 tx=0).stop 2024-08-03T20:59:26.859 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:26.857+0000 7f5956800640 1 -- 172.21.15.32:0/37718752 reap_dead start 2024-08-03T20:59:26.859 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:26.857+0000 7f595d200640 1 -- 172.21.15.32:0/37718752 shutdown_connections 2024-08-03T20:59:26.859 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:26.857+0000 7f595d200640 1 -- 172.21.15.32:0/37718752 >> 172.21.15.32:0/37718752 conn(0x7f59580b1bd0 msgr2=0x7f59580b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:59:26.859 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:26.857+0000 7f595d200640 1 -- 172.21.15.32:0/37718752 shutdown_connections 2024-08-03T20:59:26.859 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:26.857+0000 7f595d200640 1 -- 172.21.15.32:0/37718752 wait complete. 2024-08-03T20:59:26.860 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:26.859+0000 7f595d200640 1 Processor -- start 2024-08-03T20:59:26.861 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:26.859+0000 7f595d200640 1 -- start start 2024-08-03T20:59:26.863 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:26.861+0000 7f595d200640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f593c003ff0 0x7f59581f9230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:59:26.864 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:26.861+0000 7f595d200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f595813f840 0x7f5958159310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:59:26.864 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:26.861+0000 7f595d200640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f595815ede0 0x7f595815f1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:59:26.864 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:26.861+0000 7f595d200640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f59580cf6c0 con 0x7f595813f840 2024-08-03T20:59:26.864 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:26.861+0000 7f595d200640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f59580cf540 con 0x7f593c003ff0 2024-08-03T20:59:26.864 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:26.861+0000 7f595d200640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7f59580b3ce0 con 0x7f595815ede0 2024-08-03T20:59:26.864 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:26.861+0000 7f594fe00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f595813f840 0x7f5958159310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:59:26.864 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:26.861+0000 7f594fe00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f595813f840 0x7f5958159310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:41438/0 (socket says 172.21.15.32:41438) 2024-08-03T20:59:26.864 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:26.861+0000 7f594fe00640 1 -- 172.21.15.32:0/2628783321 learned_addr learned my addr 172.21.15.32:0/2628783321 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:59:26.864 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:26.861+0000 7f5955e00640 1 --2- 172.21.15.32:0/2628783321 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f593c003ff0 0x7f59581f9230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:59:26.864 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:26.861+0000 7f5956800640 1 --2- 172.21.15.32:0/2628783321 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f595815ede0 0x7f595815f1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:59:26.864 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:26.861+0000 7f594fe00640 1 -- 172.21.15.32:0/2628783321 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f595815ede0 msgr2=0x7f595815f1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:59:26.865 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:26.861+0000 7f594fe00640 1 --2- 172.21.15.32:0/2628783321 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f595815ede0 0x7f595815f1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:59:26.865 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:26.861+0000 7f594fe00640 1 -- 172.21.15.32:0/2628783321 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f593c003ff0 msgr2=0x7f59581f9230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:59:26.865 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:26.861+0000 7f594fe00640 1 --2- 172.21.15.32:0/2628783321 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f593c003ff0 0x7f59581f9230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:59:26.865 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:26.861+0000 7f594fe00640 1 -- 172.21.15.32:0/2628783321 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f595815f730 con 0x7f595813f840 2024-08-03T20:59:26.865 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:26.861+0000 7f5955e00640 1 --2- 172.21.15.32:0/2628783321 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f593c003ff0 0x7f59581f9230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-03T20:59:26.865 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:26.861+0000 7f5956800640 1 --2- 172.21.15.32:0/2628783321 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f595815ede0 0x7f595815f1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-03T20:59:26.865 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:26.862+0000 7f594fe00640 1 --2- 172.21.15.32:0/2628783321 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f595813f840 0x7f5958159310 secure :-1 s=READY pgs=1922 cs=0 l=1 rev1=1 crypto rx=0x7f594000c380 tx=0x7f5940001d20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:59:26.865 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:26.862+0000 7f594e000640 1 -- 172.21.15.32:0/2628783321 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5940031040 con 0x7f595813f840 2024-08-03T20:59:26.865 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:26.862+0000 7f594e000640 1 -- 172.21.15.32:0/2628783321 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f59400035f0 con 0x7f595813f840 2024-08-03T20:59:26.865 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:26.862+0000 7f595d200640 1 -- 172.21.15.32:0/2628783321 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f59581e4ba0 con 0x7f595813f840 2024-08-03T20:59:26.865 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:26.862+0000 7f594e000640 1 -- 172.21.15.32:0/2628783321 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5940003a40 con 0x7f595813f840 2024-08-03T20:59:26.865 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:26.862+0000 7f595d200640 1 -- 172.21.15.32:0/2628783321 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5958142cc0 con 0x7f595813f840 2024-08-03T20:59:26.867 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:26.866+0000 7f594e000640 1 -- 172.21.15.32:0/2628783321 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f5940034070 con 0x7f595813f840 2024-08-03T20:59:26.868 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:26.867+0000 7f594e000640 1 --2- 172.21.15.32:0/2628783321 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f5934077010 0x7f5934079520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:59:26.868 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:26.867+0000 7f594e000640 1 -- 172.21.15.32:0/2628783321 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(118..118 src has 1..118) ==== 8692+0+0 (secure 0 0 0) 0x7f59400b94b0 con 0x7f595813f840 2024-08-03T20:59:26.868 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:26.867+0000 7f59643e4740 1 Processor -- start 2024-08-03T20:59:26.869 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:26.867+0000 7f5955e00640 1 --2- 172.21.15.32:0/2628783321 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f5934077010 0x7f5934079520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:59:26.869 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:26.867+0000 7f59643e4740 1 -- start start 2024-08-03T20:59:26.869 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:26.868+0000 7f5955e00640 1 --2- 172.21.15.32:0/2628783321 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f5934077010 0x7f5934079520 secure :-1 s=READY pgs=772 cs=0 l=1 rev1=1 crypto rx=0x7f5948009be0 tx=0x7f5948005f90 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:59:26.871 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:26.868+0000 7f59643e4740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55ca8b9d6350 0x55ca8bdc37a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:59:26.871 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:26.868+0000 7f59643e4740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ca8b9ad490 0x55ca8bda6470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:59:26.871 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:26.868+0000 7f59643e4740 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55ca8b9b5670 0x55ca8bda69b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:59:26.871 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:26.868+0000 7f59643e4740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55ca8bcef050 con 0x55ca8b9ad490 2024-08-03T20:59:26.871 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:26.868+0000 7f59643e4740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x55ca8bcde550 con 0x55ca8b9d6350 2024-08-03T20:59:26.871 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:26.868+0000 7f59643e4740 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x55ca8bce2590 con 0x55ca8b9b5670 2024-08-03T20:59:26.871 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:26.869+0000 7f594fe00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ca8b9ad490 0x55ca8bda6470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:59:26.871 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:26.869+0000 7f594fe00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ca8b9ad490 0x55ca8bda6470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:41454/0 (socket says 172.21.15.32:41454) 2024-08-03T20:59:26.871 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:26.869+0000 7f594fe00640 1 -- 172.21.15.32:0/420120694 learned_addr learned my addr 172.21.15.32:0/420120694 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:59:26.871 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:26.869+0000 7f5955e00640 1 --2- 172.21.15.32:0/420120694 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55ca8b9d6350 0x55ca8bdc37a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:59:26.871 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:26.869+0000 7f5956800640 1 --2- 172.21.15.32:0/420120694 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55ca8b9b5670 0x55ca8bda69b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:59:26.871 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:26.869+0000 7f594fe00640 1 -- 172.21.15.32:0/420120694 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55ca8b9b5670 msgr2=0x55ca8bda69b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:59:26.872 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:26.869+0000 7f594fe00640 1 --2- 172.21.15.32:0/420120694 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55ca8b9b5670 0x55ca8bda69b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:59:26.872 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:26.869+0000 7f594fe00640 1 -- 172.21.15.32:0/420120694 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55ca8b9d6350 msgr2=0x55ca8bdc37a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:59:26.872 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:26.869+0000 7f594fe00640 1 --2- 172.21.15.32:0/420120694 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55ca8b9d6350 0x55ca8bdc37a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:59:26.872 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:26.869+0000 7f594fe00640 1 -- 172.21.15.32:0/420120694 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ca8bcbea10 con 0x55ca8b9ad490 2024-08-03T20:59:26.872 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:26.869+0000 7f594fe00640 1 --2- 172.21.15.32:0/420120694 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ca8b9ad490 0x55ca8bda6470 secure :-1 s=READY pgs=1923 cs=0 l=1 rev1=1 crypto rx=0x7f594007da30 tx=0x7f5940080cb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a7711d3048a4b8d1 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:59:26.873 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:26.870+0000 7f5946a00640 1 -- 172.21.15.32:0/420120694 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5940085a40 con 0x55ca8b9ad490 2024-08-03T20:59:26.873 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:26.870+0000 7f5946a00640 1 -- 172.21.15.32:0/420120694 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f594008a460 con 0x55ca8b9ad490 2024-08-03T20:59:26.873 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:26.870+0000 7f5946a00640 1 -- 172.21.15.32:0/420120694 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5940086e30 con 0x55ca8b9ad490 2024-08-03T20:59:26.873 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:26.870+0000 7f59643e4740 1 -- 172.21.15.32:0/420120694 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ca8b9ad490 msgr2=0x55ca8bda6470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:59:26.873 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:26.870+0000 7f59643e4740 1 --2- 172.21.15.32:0/420120694 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ca8b9ad490 0x55ca8bda6470 secure :-1 s=READY pgs=1923 cs=0 l=1 rev1=1 crypto rx=0x7f594007da30 tx=0x7f5940080cb0 comp rx=0 tx=0).stop 2024-08-03T20:59:26.873 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:26.870+0000 7f59643e4740 1 -- 172.21.15.32:0/420120694 shutdown_connections 2024-08-03T20:59:26.873 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:26.870+0000 7f59643e4740 1 --2- 172.21.15.32:0/420120694 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55ca8b9b5670 0x55ca8bda69b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:59:26.873 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:26.870+0000 7f59643e4740 1 --2- 172.21.15.32:0/420120694 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ca8b9ad490 0x55ca8bda6470 unknown :-1 s=CLOSED pgs=1923 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:59:26.874 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:26.870+0000 7f59643e4740 1 --2- 172.21.15.32:0/420120694 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55ca8b9d6350 0x55ca8bdc37a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:59:26.874 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:26.871+0000 7f59643e4740 1 -- 172.21.15.32:0/420120694 >> 172.21.15.32:0/420120694 conn(0x55ca8b9ca4a0 msgr2=0x55ca8bd7c690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:59:26.874 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:26.871+0000 7f59643e4740 1 -- 172.21.15.32:0/420120694 shutdown_connections 2024-08-03T20:59:26.874 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:26.871+0000 7f59643e4740 1 -- 172.21.15.32:0/420120694 wait complete. 2024-08-03T20:59:26.874 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:26.872+0000 7f59643e4740 1 -- start start 2024-08-03T20:59:26.874 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:26.872+0000 7f59643e4740 1 Processor -- start 2024-08-03T20:59:26.881 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:26.873+0000 7f59643e4740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55ca8b9ad490 0x55ca8bd9e7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:59:26.881 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:26.873+0000 7f59643e4740 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55ca8b9b5670 0x55ca8bdc1d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:59:26.881 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:26.873+0000 7f59643e4740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ca8b9d6350 0x55ca8bdb9020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:59:26.881 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:26.873+0000 7f59643e4740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55ca8bc16da0 con 0x55ca8b9d6350 2024-08-03T20:59:26.881 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:26.873+0000 7f59643e4740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x55ca8bc78cd0 con 0x55ca8b9ad490 2024-08-03T20:59:26.881 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:26.873+0000 7f59643e4740 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x55ca8bcde550 con 0x55ca8b9b5670 2024-08-03T20:59:26.881 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:26.873+0000 7f59643e4740 10 client.0 _ll_register_callbacks cb 0x55ca8b9edba0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-03T20:59:26.882 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:26.874+0000 7f5956800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ca8b9d6350 0x55ca8bdb9020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:59:26.882 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:26.874+0000 7f5956800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ca8b9d6350 0x55ca8bdb9020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:41470/0 (socket says 172.21.15.32:41470) 2024-08-03T20:59:26.882 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:26.874+0000 7f5956800640 1 -- 172.21.15.32:0/135982912 learned_addr learned my addr 172.21.15.32:0/135982912 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:59:26.882 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:26.874+0000 7f594fe00640 1 --2- 172.21.15.32:0/135982912 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55ca8b9b5670 0x55ca8bdc1d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:59:26.882 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:26.874+0000 7f5955e00640 1 --2- 172.21.15.32:0/135982912 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55ca8b9ad490 0x55ca8bd9e7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:59:26.882 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:26.874+0000 7f5956800640 1 -- 172.21.15.32:0/135982912 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55ca8b9b5670 msgr2=0x55ca8bdc1d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:59:26.882 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:26.874+0000 7f5956800640 1 --2- 172.21.15.32:0/135982912 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55ca8b9b5670 0x55ca8bdc1d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:59:26.882 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:26.874+0000 7f5956800640 1 -- 172.21.15.32:0/135982912 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55ca8b9ad490 msgr2=0x55ca8bd9e7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:59:26.882 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:26.874+0000 7f5956800640 1 --2- 172.21.15.32:0/135982912 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55ca8b9ad490 0x55ca8bd9e7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:59:26.882 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:26.874+0000 7f5956800640 1 -- 172.21.15.32:0/135982912 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ca8ba41930 con 0x55ca8b9d6350 2024-08-03T20:59:26.882 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:26.874+0000 7f5955e00640 1 --2- 172.21.15.32:0/135982912 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55ca8b9ad490 0x55ca8bd9e7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-03T20:59:26.882 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:26.874+0000 7f5956800640 1 --2- 172.21.15.32:0/135982912 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ca8b9d6350 0x55ca8bdb9020 secure :-1 s=READY pgs=1924 cs=0 l=1 rev1=1 crypto rx=0x7f5950005fd0 tx=0x7f5950004410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d6cbcdfee1567587 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:59:26.882 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:26.875+0000 7f593b400640 10 client.0 ms_handle_connect on v2:172.21.15.32:3300/0 2024-08-03T20:59:26.883 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:26.875+0000 7f593b400640 1 -- 172.21.15.32:0/135982912 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5950010840 con 0x55ca8b9d6350 2024-08-03T20:59:26.883 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:26.875+0000 7f59643e4740 1 -- 172.21.15.32:0/135982912 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f594008a460 con 0x55ca8b9d6350 2024-08-03T20:59:26.883 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:26.875+0000 7f593b400640 1 -- 172.21.15.32:0/135982912 <== mon.0 v2:172.21.15.32:3300/0 2 ==== osd_map(118..118 src has 1..118) ==== 8692+0+0 (secure 0 0 0) 0x7f59500109e0 con 0x55ca8b9d6350 2024-08-03T20:59:26.883 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:26.875+0000 7f593b400640 1 -- 172.21.15.32:0/135982912 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_get_version_reply(handle=1 version=286) ==== 24+0+0 (secure 0 0 0) 0x7f5950010be0 con 0x55ca8b9d6350 2024-08-03T20:59:26.883 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:26.875+0000 7f59643e4740 10 client.9858 fetch_fsmap learned FSMap version 286 2024-08-03T20:59:26.883 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:26.875+0000 7f59643e4740 1 -- 172.21.15.32:0/135982912 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,fsmap=286,monmap=2+}) -- 0x55ca8baba7c0 con 0x55ca8b9d6350 2024-08-03T20:59:26.883 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:26.875+0000 7f593b400640 1 -- 172.21.15.32:0/135982912 <== mon.0 v2:172.21.15.32:3300/0 4 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f594008a460 con 0x55ca8b9d6350 2024-08-03T20:59:26.883 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:26.875+0000 7f593b400640 1 -- 172.21.15.32:0/135982912 <== mon.0 v2:172.21.15.32:3300/0 5 ==== fsmap(e 286) ==== 3502+0+0 (secure 0 0 0) 0x7f595001c430 con 0x55ca8b9d6350 2024-08-03T20:59:26.883 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:26.875+0000 7f59643e4740 10 client.9858 fetch_fsmap finished waiting for FSMap version 286 2024-08-03T20:59:26.883 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:26.875+0000 7f59643e4740 10 client.9858 resolve_mds: resolved name 'a' to daemon mds.a 2024-08-03T20:59:26.883 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:26.875+0000 7f59643e4740 20 client.9858 populate_metadata read hostname 'smithi032' 2024-08-03T20:59:26.883 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:26.875+0000 7f59643e4740 1 --2- 172.21.15.32:0/135982912 >> [v2:172.21.15.32:6833/2173042128,v1:172.21.15.32:6835/2173042128] conn(0x55ca8b995130 0x55ca8bdd0840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:59:26.883 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:26.875+0000 7f59643e4740 4 client.9858 mds_command: new command op to 9759 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-08-03T20:59:26.884 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:26.875+0000 7f59643e4740 1 -- 172.21.15.32:0/135982912 --> [v2:172.21.15.32:6833/2173042128,v1:172.21.15.32:6835/2173042128] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ca8bcf0fa0 con 0x55ca8b995130 2024-08-03T20:59:26.884 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:26.876+0000 7f5955e00640 1 --2- 172.21.15.32:0/135982912 >> [v2:172.21.15.32:6833/2173042128,v1:172.21.15.32:6835/2173042128] conn(0x55ca8b995130 0x55ca8bdd0840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:59:26.884 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:26.876+0000 7f593b400640 10 client.9858 ms_handle_connect on v2:172.21.15.32:6833/2173042128 2024-08-03T20:59:26.884 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:26.876+0000 7f5955e00640 1 --2- 172.21.15.32:0/135982912 >> [v2:172.21.15.32:6833/2173042128,v1:172.21.15.32:6835/2173042128] conn(0x55ca8b995130 0x55ca8bdd0840 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=57f9a3742ccf3e04 server_cookie=4a7a003ff566257e in_seq=0 out_seq=0 2024-08-03T20:59:26.884 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:26.877+0000 7f593b400640 1 -- 172.21.15.32:0/135982912 <== mds.0 v2:172.21.15.32:6833/2173042128 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55ca8bcf0fa0 con 0x55ca8b995130 2024-08-03T20:59:26.884 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:26.877+0000 7f593b400640 10 client.9858 handle_command_reply: tid=0 2024-08-03T20:59:26.884 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:26.877+0000 7f59643e4740 1 client.9858 shutdown 2024-08-03T20:59:26.884 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:26.877+0000 7f59643e4740 1 -- 172.21.15.32:0/135982912 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ca8b9d6350 msgr2=0x55ca8bdb9020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:59:26.884 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:26.877+0000 7f59643e4740 1 --2- 172.21.15.32:0/135982912 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ca8b9d6350 0x55ca8bdb9020 secure :-1 s=READY pgs=1924 cs=0 l=1 rev1=1 crypto rx=0x7f5950005fd0 tx=0x7f5950004410 comp rx=0 tx=0).stop 2024-08-03T20:59:26.884 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:26.878+0000 7f59643e4740 1 -- 172.21.15.32:0/135982912 shutdown_connections 2024-08-03T20:59:26.884 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:26.878+0000 7f59643e4740 1 --2- 172.21.15.32:0/135982912 >> [v2:172.21.15.32:6833/2173042128,v1:172.21.15.32:6835/2173042128] conn(0x55ca8b995130 0x55ca8bdd0840 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:59:26.884 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:26.878+0000 7f59643e4740 1 --2- 172.21.15.32:0/135982912 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ca8b9d6350 0x55ca8bdb9020 unknown :-1 s=CLOSED pgs=1924 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:59:26.884 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:26.878+0000 7f59643e4740 1 --2- 172.21.15.32:0/135982912 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55ca8b9b5670 0x55ca8bdc1d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:59:26.884 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:26.878+0000 7f593b400640 0 client.9858 ms_handle_reset on v2:172.21.15.32:6833/2173042128 2024-08-03T20:59:26.885 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:26.878+0000 7f59643e4740 1 --2- 172.21.15.32:0/135982912 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55ca8b9ad490 0x55ca8bd9e7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:59:26.885 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:26.878+0000 7f59643e4740 1 -- 172.21.15.32:0/135982912 >> 172.21.15.32:0/135982912 conn(0x55ca8b9ca4a0 msgr2=0x55ca8bda6470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:59:26.885 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:26.878+0000 7f59643e4740 1 -- 172.21.15.32:0/135982912 shutdown_connections 2024-08-03T20:59:26.885 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:26.878+0000 7f59643e4740 1 -- 172.21.15.32:0/135982912 wait complete. 2024-08-03T20:59:26.885 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:26.878+0000 7f59643e4740 20 client.9858 trim_cache size 0 max 16384 2024-08-03T20:59:26.894 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:26.893+0000 7f59643e4740 1 Processor -- start 2024-08-03T20:59:26.894 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:26.893+0000 7f59643e4740 1 -- start start 2024-08-03T20:59:26.896 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:26.894+0000 7f59643e4740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55ca8b9ad490 0x55ca8bdbab70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:59:26.896 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:26.894+0000 7f59643e4740 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55ca8b9b5670 0x55ca8bda0c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:59:26.896 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:26.894+0000 7f59643e4740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ca8b9d6350 0x55ca8bde52f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:59:26.896 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:26.894+0000 7f59643e4740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f59582032d0 con 0x55ca8b9d6350 2024-08-03T20:59:26.896 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:26.894+0000 7f59643e4740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f5958202610 con 0x55ca8b9ad490 2024-08-03T20:59:26.897 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:26.894+0000 7f59643e4740 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7f5958202c60 con 0x55ca8b9b5670 2024-08-03T20:59:26.897 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:26.894+0000 7f5956800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ca8b9d6350 0x55ca8bde52f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:59:26.897 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:26.895+0000 7f5956800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ca8b9d6350 0x55ca8bde52f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:41480/0 (socket says 172.21.15.32:41480) 2024-08-03T20:59:26.897 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:26.895+0000 7f5956800640 1 -- 172.21.15.32:0/1028055464 learned_addr learned my addr 172.21.15.32:0/1028055464 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:59:26.897 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:26.895+0000 7f594fe00640 1 --2- 172.21.15.32:0/1028055464 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55ca8b9b5670 0x55ca8bda0c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:59:26.897 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:26.895+0000 7f5955e00640 1 --2- 172.21.15.32:0/1028055464 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55ca8b9ad490 0x55ca8bdbab70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:59:26.897 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:26.895+0000 7f5956800640 1 -- 172.21.15.32:0/1028055464 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55ca8b9b5670 msgr2=0x55ca8bda0c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:59:26.897 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:26.895+0000 7f5956800640 1 --2- 172.21.15.32:0/1028055464 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55ca8b9b5670 0x55ca8bda0c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:59:26.897 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:26.895+0000 7f5956800640 1 -- 172.21.15.32:0/1028055464 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55ca8b9ad490 msgr2=0x55ca8bdbab70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:59:26.897 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:26.895+0000 7f5956800640 1 --2- 172.21.15.32:0/1028055464 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55ca8b9ad490 0x55ca8bdbab70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:59:26.897 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:26.895+0000 7f5956800640 1 -- 172.21.15.32:0/1028055464 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ca8bdbb4d0 con 0x55ca8b9d6350 2024-08-03T20:59:26.899 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:26.895+0000 7f5956800640 1 --2- 172.21.15.32:0/1028055464 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ca8b9d6350 0x55ca8bde52f0 secure :-1 s=READY pgs=1925 cs=0 l=1 rev1=1 crypto rx=0x7f59500096f0 tx=0x7f59500049c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a9d9bd0f84b25065 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:59:26.899 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:26.895+0000 7f5946000640 1 -- 172.21.15.32:0/1028055464 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f595001c430 con 0x55ca8b9d6350 2024-08-03T20:59:26.899 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:26.895+0000 7f5946000640 1 -- 172.21.15.32:0/1028055464 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f595001c980 con 0x55ca8b9d6350 2024-08-03T20:59:26.899 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:26.895+0000 7f5946000640 1 -- 172.21.15.32:0/1028055464 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5950004d10 con 0x55ca8b9d6350 2024-08-03T20:59:26.899 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:26.896+0000 7f59643e4740 1 -- 172.21.15.32:0/1028055464 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ca8b9d6350 msgr2=0x55ca8bde52f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:59:26.899 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:26.896+0000 7f59643e4740 1 --2- 172.21.15.32:0/1028055464 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ca8b9d6350 0x55ca8bde52f0 secure :-1 s=READY pgs=1925 cs=0 l=1 rev1=1 crypto rx=0x7f59500096f0 tx=0x7f59500049c0 comp rx=0 tx=0).stop 2024-08-03T20:59:26.899 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:26.896+0000 7f59643e4740 1 -- 172.21.15.32:0/1028055464 shutdown_connections 2024-08-03T20:59:26.899 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:26.896+0000 7f59643e4740 1 --2- 172.21.15.32:0/1028055464 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ca8b9d6350 0x55ca8bde52f0 unknown :-1 s=CLOSED pgs=1925 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:59:26.899 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:26.896+0000 7f59643e4740 1 --2- 172.21.15.32:0/1028055464 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55ca8b9b5670 0x55ca8bda0c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:59:26.899 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:26.896+0000 7f59643e4740 1 --2- 172.21.15.32:0/1028055464 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55ca8b9ad490 0x55ca8bdbab70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:59:26.899 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:26.896+0000 7f59643e4740 1 -- 172.21.15.32:0/1028055464 >> 172.21.15.32:0/1028055464 conn(0x55ca8b9ca4a0 msgr2=0x55ca8bdb98f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:59:26.899 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:26.896+0000 7f59643e4740 1 -- 172.21.15.32:0/1028055464 shutdown_connections 2024-08-03T20:59:26.900 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:26.896+0000 7f59643e4740 1 -- 172.21.15.32:0/1028055464 wait complete. 2024-08-03T20:59:26.900 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:26.897+0000 7f59643e4740 1 -- start start 2024-08-03T20:59:26.900 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:26.897+0000 7f59643e4740 1 Processor -- start 2024-08-03T20:59:26.902 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:26.899+0000 7f59643e4740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ca8b9ad490 0x55ca8be3e720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:59:26.903 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:26.899+0000 7f59643e4740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55ca8b9b5670 0x55ca8bd95b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:59:26.903 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:26.899+0000 7f59643e4740 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55ca8b9d6350 0x55ca8bd960c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:59:26.903 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:26.899+0000 7f59643e4740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55ca8bda2eb0 con 0x55ca8b9ad490 2024-08-03T20:59:26.903 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:26.899+0000 7f59643e4740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x55ca8bcef050 con 0x55ca8b9b5670 2024-08-03T20:59:26.903 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:26.899+0000 7f59643e4740 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x55ca8bdd4210 con 0x55ca8b9d6350 2024-08-03T20:59:26.903 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:26.899+0000 7f59643e4740 10 client.0 _ll_register_callbacks cb 0x7f59500093a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-03T20:59:26.903 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:26.899+0000 7f5955e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ca8b9ad490 0x55ca8be3e720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:59:26.903 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:26.899+0000 7f5955e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ca8b9ad490 0x55ca8be3e720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:41494/0 (socket says 172.21.15.32:41494) 2024-08-03T20:59:26.903 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:26.899+0000 7f5955e00640 1 -- 172.21.15.32:0/1237922223 learned_addr learned my addr 172.21.15.32:0/1237922223 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:59:26.903 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:26.899+0000 7f594fe00640 1 --2- 172.21.15.32:0/1237922223 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55ca8b9b5670 0x55ca8bd95b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:59:26.903 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:26.900+0000 7f5956800640 1 --2- 172.21.15.32:0/1237922223 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55ca8b9d6350 0x55ca8bd960c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:59:26.903 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:26.900+0000 7f5955e00640 1 -- 172.21.15.32:0/1237922223 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55ca8b9d6350 msgr2=0x55ca8bd960c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:59:26.903 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:26.900+0000 7f5955e00640 1 --2- 172.21.15.32:0/1237922223 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55ca8b9d6350 0x55ca8bd960c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:59:26.904 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:26.900+0000 7f5955e00640 1 -- 172.21.15.32:0/1237922223 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55ca8b9b5670 msgr2=0x55ca8bd95b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:59:26.904 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:26.900+0000 7f5955e00640 1 --2- 172.21.15.32:0/1237922223 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55ca8b9b5670 0x55ca8bd95b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:59:26.904 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:26.900+0000 7f5955e00640 1 -- 172.21.15.32:0/1237922223 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ca8bdb80b0 con 0x55ca8b9ad490 2024-08-03T20:59:26.904 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:26.900+0000 7f594fe00640 1 --2- 172.21.15.32:0/1237922223 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55ca8b9b5670 0x55ca8bd95b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-03T20:59:26.904 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:26.900+0000 7f5955e00640 1 --2- 172.21.15.32:0/1237922223 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ca8b9ad490 0x55ca8be3e720 secure :-1 s=READY pgs=1926 cs=0 l=1 rev1=1 crypto rx=0x7f5948011990 tx=0x7f59480046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7c48d567b243afcb server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:59:26.904 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:26.900+0000 7f593b400640 10 client.0 ms_handle_connect on v2:172.21.15.32:3300/0 2024-08-03T20:59:26.904 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:26.900+0000 7f593b400640 1 -- 172.21.15.32:0/1237922223 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f594801dd00 con 0x55ca8b9ad490 2024-08-03T20:59:26.904 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:26.900+0000 7f593b400640 1 -- 172.21.15.32:0/1237922223 <== mon.0 v2:172.21.15.32:3300/0 2 ==== osd_map(118..118 src has 1..118) ==== 8692+0+0 (secure 0 0 0) 0x7f594802a480 con 0x55ca8b9ad490 2024-08-03T20:59:26.904 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:26.900+0000 7f59643e4740 1 -- 172.21.15.32:0/1237922223 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f595001c980 con 0x55ca8b9ad490 2024-08-03T20:59:26.904 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:26.900+0000 7f593b400640 1 -- 172.21.15.32:0/1237922223 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_get_version_reply(handle=1 version=286) ==== 24+0+0 (secure 0 0 0) 0x7f594803a050 con 0x55ca8b9ad490 2024-08-03T20:59:26.904 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:26.900+0000 7f59643e4740 10 client.9864 fetch_fsmap learned FSMap version 286 2024-08-03T20:59:26.904 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:26.900+0000 7f59643e4740 1 -- 172.21.15.32:0/1237922223 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,fsmap=286,monmap=2+}) -- 0x55ca8bd969d0 con 0x55ca8b9ad490 2024-08-03T20:59:26.904 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:26.901+0000 7f593b400640 1 -- 172.21.15.32:0/1237922223 <== mon.0 v2:172.21.15.32:3300/0 4 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f595001c980 con 0x55ca8b9ad490 2024-08-03T20:59:26.904 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:26.901+0000 7f593b400640 1 -- 172.21.15.32:0/1237922223 <== mon.0 v2:172.21.15.32:3300/0 5 ==== fsmap(e 286) ==== 3502+0+0 (secure 0 0 0) 0x7f5948031070 con 0x55ca8b9ad490 2024-08-03T20:59:26.905 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:26.901+0000 7f59643e4740 10 client.9864 fetch_fsmap finished waiting for FSMap version 286 2024-08-03T20:59:26.905 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:26.901+0000 7f59643e4740 10 client.9864 resolve_mds: resolved name 'a' to daemon mds.a 2024-08-03T20:59:26.905 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:26.901+0000 7f59643e4740 20 client.9864 populate_metadata read hostname 'smithi032' 2024-08-03T20:59:26.905 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:26.901+0000 7f59643e4740 1 --2- 172.21.15.32:0/1237922223 >> [v2:172.21.15.32:6833/2173042128,v1:172.21.15.32:6835/2173042128] conn(0x55ca8bdd02d0 0x55ca8bd941e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:59:26.905 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:26.901+0000 7f59643e4740 4 client.9864 mds_command: new command op to 9759 tid=0 multi_id=0 [{"prefix": "damage ls", "format": "json-pretty"}] 2024-08-03T20:59:26.905 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:26.901+0000 7f59643e4740 1 -- 172.21.15.32:0/1237922223 --> [v2:172.21.15.32:6833/2173042128,v1:172.21.15.32:6835/2173042128] -- command(tid 0: {"prefix": "damage ls", "format": "json-pretty"}) -- 0x55ca8bceca40 con 0x55ca8bdd02d0 2024-08-03T20:59:26.905 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:26.901+0000 7f594fe00640 1 --2- 172.21.15.32:0/1237922223 >> [v2:172.21.15.32:6833/2173042128,v1:172.21.15.32:6835/2173042128] conn(0x55ca8bdd02d0 0x55ca8bd941e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:59:26.906 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:26.902+0000 7f594fe00640 1 --2- 172.21.15.32:0/1237922223 >> [v2:172.21.15.32:6833/2173042128,v1:172.21.15.32:6835/2173042128] conn(0x55ca8bdd02d0 0x55ca8bd941e0 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a71ece7e1294e309 server_cookie=810aef1e6587c894 in_seq=0 out_seq=0 2024-08-03T20:59:26.906 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:26.902+0000 7f593b400640 10 client.9864 ms_handle_connect on v2:172.21.15.32:6833/2173042128 2024-08-03T20:59:26.906 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:26.902+0000 7f593b400640 1 -- 172.21.15.32:0/1237922223 <== mds.0 v2:172.21.15.32:6833/2173042128 1 ==== command_reply(tid 0: 0 ) ==== 8+0+194 (crc 0 0 0) 0x55ca8bceca40 con 0x55ca8bdd02d0 2024-08-03T20:59:26.906 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:26.902+0000 7f593b400640 10 client.9864 handle_command_reply: tid=0 2024-08-03T20:59:26.906 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:26.902+0000 7f59643e4740 1 client.9864 shutdown 2024-08-03T20:59:26.906 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:26.902+0000 7f59643e4740 1 -- 172.21.15.32:0/1237922223 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ca8b9ad490 msgr2=0x55ca8be3e720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:59:26.907 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:26.902+0000 7f59643e4740 1 --2- 172.21.15.32:0/1237922223 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ca8b9ad490 0x55ca8be3e720 secure :-1 s=READY pgs=1926 cs=0 l=1 rev1=1 crypto rx=0x7f5948011990 tx=0x7f59480046c0 comp rx=0 tx=0).stop 2024-08-03T20:59:26.907 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:26.902+0000 7f59643e4740 1 -- 172.21.15.32:0/1237922223 shutdown_connections 2024-08-03T20:59:26.907 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:26.902+0000 7f59643e4740 1 --2- 172.21.15.32:0/1237922223 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55ca8b9b5670 0x55ca8bd95b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:59:26.907 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:26.902+0000 7f59643e4740 1 --2- 172.21.15.32:0/1237922223 >> [v2:172.21.15.32:6833/2173042128,v1:172.21.15.32:6835/2173042128] conn(0x55ca8bdd02d0 0x55ca8bd941e0 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:59:26.907 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:26.902+0000 7f59643e4740 1 --2- 172.21.15.32:0/1237922223 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55ca8b9d6350 0x55ca8bd960c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:59:26.907 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:26.902+0000 7f59643e4740 1 --2- 172.21.15.32:0/1237922223 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ca8b9ad490 0x55ca8be3e720 unknown :-1 s=CLOSED pgs=1926 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:59:26.907 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:26.902+0000 7f593b400640 0 client.9864 ms_handle_reset on v2:172.21.15.32:6833/2173042128 2024-08-03T20:59:26.907 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:26.902+0000 7f59643e4740 1 -- 172.21.15.32:0/1237922223 >> 172.21.15.32:0/1237922223 conn(0x55ca8b9ca4a0 msgr2=0x55ca8bd77a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:59:26.907 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:26.902+0000 7f59643e4740 1 -- 172.21.15.32:0/1237922223 shutdown_connections 2024-08-03T20:59:26.907 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:26.902+0000 7f59643e4740 1 -- 172.21.15.32:0/1237922223 wait complete. 2024-08-03T20:59:26.907 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:26.903+0000 7f59643e4740 20 client.9864 trim_cache size 0 max 16384 2024-08-03T20:59:26.907 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-03T20:59:26.908 INFO:teuthology.orchestra.run.smithi032.stdout:[ 2024-08-03T20:59:26.908 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-03T20:59:26.908 INFO:teuthology.orchestra.run.smithi032.stdout: "damage_type": "dentry", 2024-08-03T20:59:26.908 INFO:teuthology.orchestra.run.smithi032.stdout: "id": 3390487363, 2024-08-03T20:59:26.908 INFO:teuthology.orchestra.run.smithi032.stdout: "ino": 1, 2024-08-03T20:59:26.908 INFO:teuthology.orchestra.run.smithi032.stdout: "frag": "*", 2024-08-03T20:59:26.908 INFO:teuthology.orchestra.run.smithi032.stdout: "dname": "subdir", 2024-08-03T20:59:26.908 INFO:teuthology.orchestra.run.smithi032.stdout: "snap_id": "head", 2024-08-03T20:59:26.908 INFO:teuthology.orchestra.run.smithi032.stdout: "path": "/subdir" 2024-08-03T20:59:26.908 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-03T20:59:26.908 INFO:teuthology.orchestra.run.smithi032.stdout:] 2024-08-03T20:59:26.909 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:26.903+0000 7f595d200640 1 -- 172.21.15.32:0/2628783321 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f5934077010 msgr2=0x7f5934079520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:59:26.909 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:26.903+0000 7f595d200640 1 --2- 172.21.15.32:0/2628783321 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f5934077010 0x7f5934079520 secure :-1 s=READY pgs=772 cs=0 l=1 rev1=1 crypto rx=0x7f5948009be0 tx=0x7f5948005f90 comp rx=0 tx=0).stop 2024-08-03T20:59:26.909 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:26.903+0000 7f595d200640 1 -- 172.21.15.32:0/2628783321 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f595813f840 msgr2=0x7f5958159310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:59:26.909 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:26.903+0000 7f595d200640 1 --2- 172.21.15.32:0/2628783321 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f595813f840 0x7f5958159310 secure :-1 s=READY pgs=1922 cs=0 l=1 rev1=1 crypto rx=0x7f594000c380 tx=0x7f5940001d20 comp rx=0 tx=0).stop 2024-08-03T20:59:26.909 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:26.903+0000 7f595d200640 1 -- 172.21.15.32:0/2628783321 shutdown_connections 2024-08-03T20:59:26.909 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:26.903+0000 7f595d200640 1 --2- 172.21.15.32:0/2628783321 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f5934077010 0x7f5934079520 unknown :-1 s=CLOSED pgs=772 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:59:26.909 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:26.903+0000 7f595d200640 1 --2- 172.21.15.32:0/2628783321 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f595815ede0 0x7f595815f1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:59:26.909 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:26.903+0000 7f595d200640 1 --2- 172.21.15.32:0/2628783321 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f595813f840 0x7f5958159310 unknown :-1 s=CLOSED pgs=1922 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:59:26.910 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:26.903+0000 7f595d200640 1 --2- 172.21.15.32:0/2628783321 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f593c003ff0 0x7f59581f9230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:59:26.910 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:26.903+0000 7f595d200640 1 -- 172.21.15.32:0/2628783321 >> 172.21.15.32:0/2628783321 conn(0x7f59580b1bd0 msgr2=0x7f595815db10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:59:26.910 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:26.903+0000 7f595d200640 1 -- 172.21.15.32:0/2628783321 shutdown_connections 2024-08-03T20:59:26.910 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:26.903+0000 7f595d200640 1 -- 172.21.15.32:0/2628783321 wait complete. 2024-08-03T20:59:26.930 INFO:tasks.cephfs.test_damage:Applying mutation 'Corrupt omap header on 10000000000.00000000' 2024-08-03T20:59:26.930 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-08-03T20:59:26.930 DEBUG:teuthology.orchestra.run.smithi032:> dd if=/proc/self/mounts of=/dev/stdout 2024-08-03T20:59:26.965 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-08-03T20:59:26.965 DEBUG:teuthology.orchestra.run.smithi032:> dd if=/proc/self/mounts of=/dev/stdout 2024-08-03T20:59:27.041 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-08-03T20:59:27.312 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:27.310+0000 7fe76c400640 1 -- 172.21.15.32:0/165927058 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fe76406e930 msgr2=0x7fe764144f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:59:27.312 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:27.310+0000 7fe76c400640 1 --2- 172.21.15.32:0/165927058 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fe76406e930 0x7fe764144f90 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7fe758009830 tx=0x7fe75802a5c0 comp rx=0 tx=0).stop 2024-08-03T20:59:27.312 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:27.310+0000 7fe76c400640 1 -- 172.21.15.32:0/165927058 shutdown_connections 2024-08-03T20:59:27.312 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:27.310+0000 7fe76c400640 1 --2- 172.21.15.32:0/165927058 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fe76406e930 0x7fe764144f90 unknown :-1 s=CLOSED pgs=580 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:59:27.312 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:27.310+0000 7fe76c400640 1 -- 172.21.15.32:0/165927058 >> 172.21.15.32:0/165927058 conn(0x7fe7640b1bd0 msgr2=0x7fe7640b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:59:27.312 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:27.310+0000 7fe76c400640 1 -- 172.21.15.32:0/165927058 shutdown_connections 2024-08-03T20:59:27.312 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:27.310+0000 7fe76c400640 1 -- 172.21.15.32:0/165927058 wait complete. 2024-08-03T20:59:27.313 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:27.312+0000 7fe76c400640 1 Processor -- start 2024-08-03T20:59:27.314 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:27.312+0000 7fe76c400640 1 -- start start 2024-08-03T20:59:27.315 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:27.314+0000 7fe76c400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe76406e930 0x7fe76413d9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:59:27.316 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:27.314+0000 7fe76c400640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fe764073e60 0x7fe7640b2fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:59:27.316 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:27.314+0000 7fe76c400640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fe76415fff0 0x7fe7640af2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:59:27.316 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:27.314+0000 7fe76c400640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fe7640cf840 con 0x7fe76406e930 2024-08-03T20:59:27.316 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:27.314+0000 7fe76c400640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7fe764145f30 con 0x7fe76415fff0 2024-08-03T20:59:27.316 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:27.314+0000 7fe76c400640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7fe7640cf9c0 con 0x7fe764073e60 2024-08-03T20:59:27.316 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:27.314+0000 7fe769a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe76406e930 0x7fe76413d9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:59:27.316 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:27.314+0000 7fe769a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe76406e930 0x7fe76413d9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:41496/0 (socket says 172.21.15.32:41496) 2024-08-03T20:59:27.316 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:27.314+0000 7fe769a00640 1 -- 172.21.15.32:0/1261285938 learned_addr learned my addr 172.21.15.32:0/1261285938 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:59:27.316 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:27.314+0000 7fe769000640 1 --2- 172.21.15.32:0/1261285938 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fe764073e60 0x7fe7640b2fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:59:27.316 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:27.314+0000 7fe76a400640 1 --2- 172.21.15.32:0/1261285938 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fe76415fff0 0x7fe7640af2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:59:27.317 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:27.314+0000 7fe769a00640 1 -- 172.21.15.32:0/1261285938 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fe764073e60 msgr2=0x7fe7640b2fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:59:27.317 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:27.314+0000 7fe769a00640 1 --2- 172.21.15.32:0/1261285938 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fe764073e60 0x7fe7640b2fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:59:27.317 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:27.314+0000 7fe769a00640 1 -- 172.21.15.32:0/1261285938 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fe76415fff0 msgr2=0x7fe7640af2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:59:27.317 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:27.314+0000 7fe769a00640 1 --2- 172.21.15.32:0/1261285938 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fe76415fff0 0x7fe7640af2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:59:27.318 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:27.315+0000 7fe769a00640 1 -- 172.21.15.32:0/1261285938 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe7640ae240 con 0x7fe76406e930 2024-08-03T20:59:27.318 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:27.315+0000 7fe769a00640 1 --2- 172.21.15.32:0/1261285938 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe76406e930 0x7fe76413d9d0 secure :-1 s=READY pgs=1928 cs=0 l=1 rev1=1 crypto rx=0x7fe75802abb0 tx=0x7fe758033ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:59:27.318 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:27.315+0000 7fe761600640 1 -- 172.21.15.32:0/1261285938 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe758031b50 con 0x7fe76406e930 2024-08-03T20:59:27.318 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:27.315+0000 7fe761600640 1 -- 172.21.15.32:0/1261285938 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fe758031de0 con 0x7fe76406e930 2024-08-03T20:59:27.318 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:27.315+0000 7fe761600640 1 -- 172.21.15.32:0/1261285938 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe758005280 con 0x7fe76406e930 2024-08-03T20:59:27.318 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:27.315+0000 7fe76c400640 1 -- 172.21.15.32:0/1261285938 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe7640ae520 con 0x7fe76406e930 2024-08-03T20:59:27.318 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:27.315+0000 7fe76c400640 1 -- 172.21.15.32:0/1261285938 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe7640aeae0 con 0x7fe76406e930 2024-08-03T20:59:27.318 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:27.316+0000 7fe76c400640 1 -- 172.21.15.32:0/1261285938 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe764070480 con 0x7fe76406e930 2024-08-03T20:59:27.319 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:27.317+0000 7fe761600640 1 -- 172.21.15.32:0/1261285938 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7fe758005420 con 0x7fe76406e930 2024-08-03T20:59:27.320 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:27.318+0000 7fe761600640 1 --2- 172.21.15.32:0/1261285938 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fe744077010 0x7fe744079520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:59:27.320 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:27.318+0000 7fe761600640 1 -- 172.21.15.32:0/1261285938 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(118..118 src has 1..118) ==== 8692+0+0 (secure 0 0 0) 0x7fe7580ba510 con 0x7fe76406e930 2024-08-03T20:59:27.321 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:27.319+0000 7fe769000640 1 --2- 172.21.15.32:0/1261285938 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fe744077010 0x7fe744079520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:59:27.321 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:27.320+0000 7fe769000640 1 --2- 172.21.15.32:0/1261285938 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fe744077010 0x7fe744079520 secure :-1 s=READY pgs=773 cs=0 l=1 rev1=1 crypto rx=0x7fe7500046b0 tx=0x7fe750007cc0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:59:27.324 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:27.323+0000 7fe761600640 1 -- 172.21.15.32:0/1261285938 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7fe75803f030 con 0x7fe76406e930 2024-08-03T20:59:27.543 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:27.541+0000 7fe76c400640 1 -- 172.21.15.32:0/1261285938 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7fe764160e20 con 0x7fe76406e930 2024-08-03T20:59:27.543 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:27.542+0000 7fe761600640 1 -- 172.21.15.32:0/1261285938 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 66 entries v118) ==== 81+0+4012 (secure 0 0 0) 0x7fe758031770 con 0x7fe76406e930 2024-08-03T20:59:27.544 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6835/506775245 2024-08-04T20:59:03.076256+0000 2024-08-03T20:59:27.544 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6833/506775245 2024-08-04T20:59:03.076256+0000 2024-08-03T20:59:27.544 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6833/556650445 2024-08-04T20:58:57.063754+0000 2024-08-03T20:59:27.544 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6835/2156596385 2024-08-04T20:58:35.073838+0000 2024-08-03T20:59:27.544 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6833/2156596385 2024-08-04T20:58:35.073838+0000 2024-08-03T20:59:27.544 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6835/3669493978 2024-08-04T20:58:30.064676+0000 2024-08-03T20:59:27.544 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6833/2178506907 2024-08-04T20:56:40.045549+0000 2024-08-03T20:59:27.544 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6835/556650445 2024-08-04T20:58:57.063754+0000 2024-08-03T20:59:27.545 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6835/2278473729 2024-08-04T20:56:13.041693+0000 2024-08-03T20:59:27.545 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6835/2345472481 2024-08-04T20:57:25.968590+0000 2024-08-03T20:59:27.545 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6833/3940515773 2024-08-04T20:58:24.290690+0000 2024-08-03T20:59:27.545 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6833/2600386707 2024-08-04T20:57:50.063964+0000 2024-08-03T20:59:27.545 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6835/2178506907 2024-08-04T20:56:40.045549+0000 2024-08-03T20:59:27.545 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6833/2292690833 2024-08-04T20:55:20.021329+0000 2024-08-03T20:59:27.545 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6835/2803901497 2024-08-04T20:55:55.181144+0000 2024-08-03T20:59:27.545 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6835/2194902489 2024-08-04T20:55:00.359126+0000 2024-08-03T20:59:27.545 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6833/2803901497 2024-08-04T20:55:55.181144+0000 2024-08-03T20:59:27.545 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6835/3191654276 2024-08-04T20:55:42.433364+0000 2024-08-03T20:59:27.545 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6833/2242533881 2024-08-04T20:56:48.048099+0000 2024-08-03T20:59:27.545 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6833/2283924374 2024-08-04T20:57:34.396782+0000 2024-08-03T20:59:27.545 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6835/1172050551 2024-08-04T20:55:26.214113+0000 2024-08-03T20:59:27.546 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6835/1016338829 2024-08-04T20:55:06.071485+0000 2024-08-03T20:59:27.546 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6833/1474984145 2024-08-04T20:57:40.052808+0000 2024-08-03T20:59:27.546 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6833/3669493978 2024-08-04T20:58:30.064676+0000 2024-08-03T20:59:27.546 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6835/756683988 2024-08-04T20:55:14.119228+0000 2024-08-03T20:59:27.546 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6833/2194902489 2024-08-04T20:55:00.359126+0000 2024-08-03T20:59:27.546 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6835/2806168786 2024-08-04T20:56:37.279160+0000 2024-08-03T20:59:27.546 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6833/2278473729 2024-08-04T20:56:13.041693+0000 2024-08-03T20:59:27.546 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6833/3191654276 2024-08-04T20:55:42.433364+0000 2024-08-03T20:59:27.546 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6835/3431300225 2024-08-04T20:56:21.088927+0000 2024-08-03T20:59:27.546 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6833/4172198298 2024-08-04T20:56:04.256182+0000 2024-08-03T20:59:27.546 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6833/3431300225 2024-08-04T20:56:21.088927+0000 2024-08-03T20:59:27.546 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6835/2292690833 2024-08-04T20:55:20.021329+0000 2024-08-03T20:59:27.546 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6833/2806168786 2024-08-04T20:56:37.279160+0000 2024-08-03T20:59:27.547 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6833/1172050551 2024-08-04T20:55:26.214113+0000 2024-08-03T20:59:27.547 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6835/4172198298 2024-08-04T20:56:04.256182+0000 2024-08-03T20:59:27.547 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6835/3409752696 2024-08-04T20:58:18.243140+0000 2024-08-03T20:59:27.547 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6833/2274054575 2024-08-04T20:57:55.062490+0000 2024-08-03T20:59:27.547 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6833/1016338829 2024-08-04T20:55:06.071485+0000 2024-08-03T20:59:27.547 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6833/1707454554 2024-08-04T20:54:38.409411+0000 2024-08-03T20:59:27.547 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6835/2242533881 2024-08-04T20:56:48.048099+0000 2024-08-03T20:59:27.547 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6833/739874257 2024-08-04T20:56:56.101271+0000 2024-08-03T20:59:27.547 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6835/739874257 2024-08-04T20:56:56.101271+0000 2024-08-03T20:59:27.547 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6833/2362145329 2024-08-04T20:57:02.150199+0000 2024-08-03T20:59:27.547 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6833/3771304550 2024-08-04T20:55:58.212752+0000 2024-08-03T20:59:27.547 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6835/2362145329 2024-08-04T20:57:02.150199+0000 2024-08-03T20:59:27.547 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6835/3940515773 2024-08-04T20:58:24.290690+0000 2024-08-03T20:59:27.547 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6833/1885034124 2024-08-04T20:57:08.190707+0000 2024-08-03T20:59:27.547 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6833/511221267 2024-08-04T20:57:45.062448+0000 2024-08-03T20:59:27.548 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6835/3771304550 2024-08-04T20:55:58.212752+0000 2024-08-03T20:59:27.548 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6835/1885034124 2024-08-04T20:57:08.190707+0000 2024-08-03T20:59:27.548 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6833/2345472481 2024-08-04T20:57:25.968590+0000 2024-08-03T20:59:27.548 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6833/265163622 2024-08-04T20:57:28.354509+0000 2024-08-03T20:59:27.548 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6835/265163622 2024-08-04T20:57:28.354509+0000 2024-08-03T20:59:27.548 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6835/2274054575 2024-08-04T20:57:55.062490+0000 2024-08-03T20:59:27.548 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6835/2283924374 2024-08-04T20:57:34.396782+0000 2024-08-03T20:59:27.548 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6833/756683988 2024-08-04T20:55:14.119228+0000 2024-08-03T20:59:27.548 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6835/1474984145 2024-08-04T20:57:40.052808+0000 2024-08-03T20:59:27.548 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6835/511221267 2024-08-04T20:57:45.062448+0000 2024-08-03T20:59:27.548 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6835/1707454554 2024-08-04T20:54:38.409411+0000 2024-08-03T20:59:27.548 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6835/2600386707 2024-08-04T20:57:50.063964+0000 2024-08-03T20:59:27.548 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6833/675070591 2024-08-04T20:58:00.068114+0000 2024-08-03T20:59:27.548 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6833/3340816283 2024-08-04T20:58:15.214542+0000 2024-08-03T20:59:27.549 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6835/3340816283 2024-08-04T20:58:15.214542+0000 2024-08-03T20:59:27.549 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6835/675070591 2024-08-04T20:58:00.068114+0000 2024-08-03T20:59:27.549 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6833/3409752696 2024-08-04T20:58:18.243140+0000 2024-08-03T20:59:27.550 INFO:teuthology.orchestra.run.smithi032.stderr:listed 66 entries 2024-08-03T20:59:27.550 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:27.544+0000 7fe76c400640 1 -- 172.21.15.32:0/1261285938 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fe744077010 msgr2=0x7fe744079520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:59:27.550 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:27.544+0000 7fe76c400640 1 --2- 172.21.15.32:0/1261285938 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fe744077010 0x7fe744079520 secure :-1 s=READY pgs=773 cs=0 l=1 rev1=1 crypto rx=0x7fe7500046b0 tx=0x7fe750007cc0 comp rx=0 tx=0).stop 2024-08-03T20:59:27.550 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:27.544+0000 7fe76c400640 1 -- 172.21.15.32:0/1261285938 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe76406e930 msgr2=0x7fe76413d9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:59:27.550 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:27.544+0000 7fe76c400640 1 --2- 172.21.15.32:0/1261285938 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe76406e930 0x7fe76413d9d0 secure :-1 s=READY pgs=1928 cs=0 l=1 rev1=1 crypto rx=0x7fe75802abb0 tx=0x7fe758033ba0 comp rx=0 tx=0).stop 2024-08-03T20:59:27.550 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:27.544+0000 7fe76c400640 1 -- 172.21.15.32:0/1261285938 shutdown_connections 2024-08-03T20:59:27.550 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:27.544+0000 7fe76c400640 1 --2- 172.21.15.32:0/1261285938 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fe744077010 0x7fe744079520 unknown :-1 s=CLOSED pgs=773 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:59:27.550 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:27.544+0000 7fe76c400640 1 --2- 172.21.15.32:0/1261285938 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fe76415fff0 0x7fe7640af2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:59:27.550 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:27.544+0000 7fe76c400640 1 --2- 172.21.15.32:0/1261285938 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fe764073e60 0x7fe7640b2fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:59:27.550 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:27.544+0000 7fe76c400640 1 --2- 172.21.15.32:0/1261285938 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe76406e930 0x7fe76413d9d0 unknown :-1 s=CLOSED pgs=1928 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:59:27.550 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:27.544+0000 7fe76c400640 1 -- 172.21.15.32:0/1261285938 >> 172.21.15.32:0/1261285938 conn(0x7fe7640b1bd0 msgr2=0x7fe7640b1490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:59:27.551 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:27.544+0000 7fe76c400640 1 -- 172.21.15.32:0/1261285938 shutdown_connections 2024-08-03T20:59:27.551 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:27.544+0000 7fe76c400640 1 -- 172.21.15.32:0/1261285938 wait complete. 2024-08-03T20:59:27.571 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-08-03T20:59:27.572 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-08-03T20:59:27.572 DEBUG:teuthology.orchestra.run.smithi032:> sudo umount /home/ubuntu/cephtest/mnt.0 -f 2024-08-03T20:59:27.631 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi032.front.sepia.ceph.com 2024-08-03T20:59:27.632 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-08-03T20:59:27.632 DEBUG:teuthology.orchestra.run.smithi032:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-08-03T20:59:27.668 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-08-03T20:59:27.938 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:27.935+0000 7f75e3200640 1 -- 172.21.15.32:0/3463934266 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f75e406d0f0 msgr2=0x7f75e40711f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:59:27.938 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:27.935+0000 7f75e3200640 1 --2- 172.21.15.32:0/3463934266 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f75e406d0f0 0x7f75e40711f0 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7f75cc009830 tx=0x7f75cc02a420 comp rx=0 tx=0).stop 2024-08-03T20:59:27.938 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:27.936+0000 7f75e3200640 1 -- 172.21.15.32:0/3463934266 shutdown_connections 2024-08-03T20:59:27.938 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:27.936+0000 7f75e3200640 1 --2- 172.21.15.32:0/3463934266 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f75e406d0f0 0x7f75e40711f0 unknown :-1 s=CLOSED pgs=582 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:59:27.938 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:27.936+0000 7f75e3200640 1 -- 172.21.15.32:0/3463934266 >> 172.21.15.32:0/3463934266 conn(0x7f75e40d0dd0 msgr2=0x7f75e40d11e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:59:27.938 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:27.936+0000 7f75e3200640 1 -- 172.21.15.32:0/3463934266 shutdown_connections 2024-08-03T20:59:27.938 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:27.936+0000 7f75e3200640 1 -- 172.21.15.32:0/3463934266 wait complete. 2024-08-03T20:59:27.939 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:27.938+0000 7f75e3200640 1 Processor -- start 2024-08-03T20:59:27.939 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:27.938+0000 7f75e3200640 1 -- start start 2024-08-03T20:59:27.942 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:27.940+0000 7f75e3200640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f75e40cb420 0x7f75e41f2c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:59:27.942 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:27.940+0000 7f75e3200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f75e415d310 0x7f75e40c7430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:59:27.942 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:27.940+0000 7f75e3200640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f75e414d880 0x7f75e414cbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:59:27.942 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:27.940+0000 7f75e3200640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f75e4159d60 con 0x7f75e415d310 2024-08-03T20:59:27.942 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:27.940+0000 7f75e3200640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f75e4159be0 con 0x7f75e414d880 2024-08-03T20:59:27.943 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:27.940+0000 7f75e3200640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7f75e4159ee0 con 0x7f75e40cb420 2024-08-03T20:59:27.943 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:27.940+0000 7f75dbe00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f75e415d310 0x7f75e40c7430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:59:27.943 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:27.940+0000 7f75dbe00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f75e415d310 0x7f75e40c7430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:41502/0 (socket says 172.21.15.32:41502) 2024-08-03T20:59:27.943 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:27.940+0000 7f75dbe00640 1 -- 172.21.15.32:0/3413884827 learned_addr learned my addr 172.21.15.32:0/3413884827 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:59:27.943 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:27.940+0000 7f75e1e00640 1 --2- 172.21.15.32:0/3413884827 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f75e40cb420 0x7f75e41f2c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:59:27.943 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:27.940+0000 7f75e2800640 1 --2- 172.21.15.32:0/3413884827 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f75e414d880 0x7f75e414cbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:59:27.943 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:27.940+0000 7f75dbe00640 1 -- 172.21.15.32:0/3413884827 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f75e40cb420 msgr2=0x7f75e41f2c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:59:27.943 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:27.940+0000 7f75dbe00640 1 --2- 172.21.15.32:0/3413884827 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f75e40cb420 0x7f75e41f2c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:59:27.943 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:27.941+0000 7f75dbe00640 1 -- 172.21.15.32:0/3413884827 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f75e414d880 msgr2=0x7f75e414cbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:59:27.943 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:27.941+0000 7f75dbe00640 1 --2- 172.21.15.32:0/3413884827 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f75e414d880 0x7f75e414cbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:59:27.943 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:27.941+0000 7f75dbe00640 1 -- 172.21.15.32:0/3413884827 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f75e40c5970 con 0x7f75e415d310 2024-08-03T20:59:27.943 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:27.941+0000 7f75dbe00640 1 --2- 172.21.15.32:0/3413884827 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f75e415d310 0x7f75e40c7430 secure :-1 s=READY pgs=1930 cs=0 l=1 rev1=1 crypto rx=0x7f75cc009950 tx=0x7f75cc003c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:59:27.944 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:27.941+0000 7f75da000640 1 -- 172.21.15.32:0/3413884827 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75cc03f030 con 0x7f75e415d310 2024-08-03T20:59:27.944 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:27.941+0000 7f75e3200640 1 -- 172.21.15.32:0/3413884827 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f75e40c5c50 con 0x7f75e415d310 2024-08-03T20:59:27.944 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:27.941+0000 7f75da000640 1 -- 172.21.15.32:0/3413884827 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f75cc004510 con 0x7f75e415d310 2024-08-03T20:59:27.945 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:27.941+0000 7f75da000640 1 -- 172.21.15.32:0/3413884827 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75cc037040 con 0x7f75e415d310 2024-08-03T20:59:27.945 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:27.941+0000 7f75e3200640 1 -- 172.21.15.32:0/3413884827 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f75e40c6210 con 0x7f75e415d310 2024-08-03T20:59:27.945 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:27.943+0000 7f75e3200640 1 -- 172.21.15.32:0/3413884827 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f75e40711f0 con 0x7f75e415d310 2024-08-03T20:59:27.945 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:27.944+0000 7f75da000640 1 -- 172.21.15.32:0/3413884827 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f75cc042070 con 0x7f75e415d310 2024-08-03T20:59:27.946 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:27.945+0000 7f75da000640 1 --2- 172.21.15.32:0/3413884827 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f75c0077040 0x7f75c0079550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:59:27.946 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:27.945+0000 7f75da000640 1 -- 172.21.15.32:0/3413884827 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(118..118 src has 1..118) ==== 8692+0+0 (secure 0 0 0) 0x7f75cc0b9240 con 0x7f75e415d310 2024-08-03T20:59:27.946 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:27.945+0000 7f75e1e00640 1 --2- 172.21.15.32:0/3413884827 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f75c0077040 0x7f75c0079550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:59:27.947 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:27.946+0000 7f75e1e00640 1 --2- 172.21.15.32:0/3413884827 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f75c0077040 0x7f75c0079550 secure :-1 s=READY pgs=774 cs=0 l=1 rev1=1 crypto rx=0x7f75d4000f60 tx=0x7f75d4009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:59:27.950 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:27.949+0000 7f75da000640 1 -- 172.21.15.32:0/3413884827 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7f75cc049070 con 0x7f75e415d310 2024-08-03T20:59:28.193 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:28.192+0000 7f75e3200640 1 -- 172.21.15.32:0/3413884827 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7f75e40c44f0 con 0x7f75e415d310 2024-08-03T20:59:28.194 INFO:tasks.ceph.mon.a.smithi032.stderr:2024-08-03T20:59:28.192+0000 7f4e6d600640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-08-03T20:59:28.196 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:28.195+0000 7f75da000640 1 -- 172.21.15.32:0/3413884827 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v287) ==== 161+0+0 (secure 0 0 0) 0x7f75cc081110 con 0x7f75e415d310 2024-08-03T20:59:28.196 INFO:teuthology.orchestra.run.smithi032.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-08-03T20:59:28.200 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:28.198+0000 7f75e3200640 1 -- 172.21.15.32:0/3413884827 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f75c0077040 msgr2=0x7f75c0079550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:59:28.200 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:28.198+0000 7f75e3200640 1 --2- 172.21.15.32:0/3413884827 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f75c0077040 0x7f75c0079550 secure :-1 s=READY pgs=774 cs=0 l=1 rev1=1 crypto rx=0x7f75d4000f60 tx=0x7f75d4009040 comp rx=0 tx=0).stop 2024-08-03T20:59:28.200 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:28.198+0000 7f75e3200640 1 -- 172.21.15.32:0/3413884827 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f75e415d310 msgr2=0x7f75e40c7430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:59:28.200 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:28.198+0000 7f75e3200640 1 --2- 172.21.15.32:0/3413884827 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f75e415d310 0x7f75e40c7430 secure :-1 s=READY pgs=1930 cs=0 l=1 rev1=1 crypto rx=0x7f75cc009950 tx=0x7f75cc003c10 comp rx=0 tx=0).stop 2024-08-03T20:59:28.200 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:28.198+0000 7f75e3200640 1 -- 172.21.15.32:0/3413884827 shutdown_connections 2024-08-03T20:59:28.200 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:28.198+0000 7f75e3200640 1 --2- 172.21.15.32:0/3413884827 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f75c0077040 0x7f75c0079550 unknown :-1 s=CLOSED pgs=774 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:59:28.200 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:28.198+0000 7f75e3200640 1 --2- 172.21.15.32:0/3413884827 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f75e414d880 0x7f75e414cbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:59:28.200 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:28.198+0000 7f75e3200640 1 --2- 172.21.15.32:0/3413884827 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f75e415d310 0x7f75e40c7430 unknown :-1 s=CLOSED pgs=1930 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:59:28.200 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:28.198+0000 7f75e3200640 1 --2- 172.21.15.32:0/3413884827 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f75e40cb420 0x7f75e41f2c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:59:28.200 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:28.198+0000 7f75e3200640 1 -- 172.21.15.32:0/3413884827 >> 172.21.15.32:0/3413884827 conn(0x7f75e40d0dd0 msgr2=0x7f75e40b1f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:59:28.200 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:28.198+0000 7f75e3200640 1 -- 172.21.15.32:0/3413884827 shutdown_connections 2024-08-03T20:59:28.201 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:28.198+0000 7f75e3200640 1 -- 172.21.15.32:0/3413884827 wait complete. 2024-08-03T20:59:28.204 INFO:tasks.ceph.mds.a.smithi032.stderr: -804> 2024-08-03T20:59:18.166+0000 7f07b2000640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-08-03T20:59:28.224 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mds repaired 0 2024-08-03T20:59:28.322 INFO:tasks.ceph.mds.a.smithi032.stdout:starting mds.a at 2024-08-03T20:59:28.458 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:28.455+0000 7fd83be00640 1 -- 172.21.15.32:0/3304344748 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fd834144f90 msgr2=0x7fd834074570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:59:28.458 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:28.455+0000 7fd83be00640 1 --2- 172.21.15.32:0/3304344748 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fd834144f90 0x7fd834074570 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fd820005fd0 tx=0x7fd82002a5d0 comp rx=0 tx=0).stop 2024-08-03T20:59:28.458 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:28.456+0000 7fd83be00640 1 -- 172.21.15.32:0/3304344748 shutdown_connections 2024-08-03T20:59:28.458 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:28.456+0000 7fd83be00640 1 --2- 172.21.15.32:0/3304344748 >> v2:172.21.15.118:3300/0 conn(0x7fd83415fff0 0x7fd834160400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:59:28.458 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:28.456+0000 7fd83be00640 1 --2- 172.21.15.32:0/3304344748 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fd834144f90 0x7fd834074570 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:59:28.458 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:28.456+0000 7fd83be00640 1 -- 172.21.15.32:0/3304344748 >> 172.21.15.32:0/3304344748 conn(0x7fd8340b1bd0 msgr2=0x7fd8340b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:59:28.458 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:28.456+0000 7fd83be00640 1 -- 172.21.15.32:0/3304344748 shutdown_connections 2024-08-03T20:59:28.458 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:28.456+0000 7fd83be00640 1 -- 172.21.15.32:0/3304344748 wait complete. 2024-08-03T20:59:28.459 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:28.458+0000 7fd83be00640 1 Processor -- start 2024-08-03T20:59:28.459 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:28.458+0000 7fd83be00640 1 -- start start 2024-08-03T20:59:28.462 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:28.459+0000 7fd83be00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd83406e930 0x7fd83415c020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:59:28.462 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:28.459+0000 7fd83be00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fd834144f90 0x7fd8341e5ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:59:28.462 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:28.459+0000 7fd83be00640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fd83415fff0 0x7fd834143bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:59:28.462 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:28.459+0000 7fd83be00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fd8340cf840 con 0x7fd83406e930 2024-08-03T20:59:28.462 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:28.459+0000 7fd83be00640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7fd834145f30 con 0x7fd834144f90 2024-08-03T20:59:28.462 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:28.459+0000 7fd83be00640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7fd8340cf9c0 con 0x7fd83415fff0 2024-08-03T20:59:28.463 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:28.460+0000 7fd839400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd83406e930 0x7fd83415c020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:59:28.463 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:28.460+0000 7fd839e00640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fd83415fff0 0x7fd834143bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:59:28.463 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:28.460+0000 7fd838a00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fd834144f90 0x7fd8341e5ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:59:28.463 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:28.460+0000 7fd839400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd83406e930 0x7fd83415c020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:41530/0 (socket says 172.21.15.32:41530) 2024-08-03T20:59:28.463 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:28.460+0000 7fd839400640 1 -- 172.21.15.32:0/4104191938 learned_addr learned my addr 172.21.15.32:0/4104191938 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:59:28.463 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:28.460+0000 7fd839e00640 1 -- 172.21.15.32:0/4104191938 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fd834144f90 msgr2=0x7fd8341e5ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:59:28.463 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:28.460+0000 7fd839e00640 1 --2- 172.21.15.32:0/4104191938 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fd834144f90 0x7fd8341e5ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:59:28.463 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:28.460+0000 7fd839e00640 1 -- 172.21.15.32:0/4104191938 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd83406e930 msgr2=0x7fd83415c020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:59:28.463 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:28.460+0000 7fd839e00640 1 --2- 172.21.15.32:0/4104191938 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd83406e930 0x7fd83415c020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:59:28.463 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:28.460+0000 7fd839e00640 1 -- 172.21.15.32:0/4104191938 --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd834144110 con 0x7fd83415fff0 2024-08-03T20:59:28.463 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:28.461+0000 7fd839e00640 1 --2- 172.21.15.32:0/4104191938 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fd83415fff0 0x7fd834143bd0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fd82c00e9b0 tx=0x7fd82c00ee70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:59:28.464 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:28.461+0000 7fd831600640 1 -- 172.21.15.32:0/4104191938 <== mon.2 v2:172.21.15.118:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd82c00cd80 con 0x7fd83415fff0 2024-08-03T20:59:28.464 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:28.461+0000 7fd831600640 1 -- 172.21.15.32:0/4104191938 <== mon.2 v2:172.21.15.118:3301/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fd82c004b20 con 0x7fd83415fff0 2024-08-03T20:59:28.464 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:28.461+0000 7fd83be00640 1 -- 172.21.15.32:0/4104191938 --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd834142e10 con 0x7fd83415fff0 2024-08-03T20:59:28.464 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:28.461+0000 7fd831600640 1 -- 172.21.15.32:0/4104191938 <== mon.2 v2:172.21.15.118:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd82c005020 con 0x7fd83415fff0 2024-08-03T20:59:28.465 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:28.461+0000 7fd83be00640 1 -- 172.21.15.32:0/4104191938 --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fd8341433d0 con 0x7fd83415fff0 2024-08-03T20:59:28.465 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:28.462+0000 7fd83be00640 1 -- 172.21.15.32:0/4104191938 --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd83406a430 con 0x7fd83415fff0 2024-08-03T20:59:28.465 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:28.464+0000 7fd831600640 1 -- 172.21.15.32:0/4104191938 <== mon.2 v2:172.21.15.118:3301/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7fd82c0051c0 con 0x7fd83415fff0 2024-08-03T20:59:28.466 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:28.465+0000 7fd831600640 1 --2- 172.21.15.32:0/4104191938 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fd814077010 0x7fd814079520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:59:28.466 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:28.465+0000 7fd831600640 1 -- 172.21.15.32:0/4104191938 <== mon.2 v2:172.21.15.118:3301/0 5 ==== osd_map(119..119 src has 1..119) ==== 8778+0+0 (secure 0 0 0) 0x7fd82c09a350 con 0x7fd83415fff0 2024-08-03T20:59:28.466 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:28.465+0000 7fd839400640 1 --2- 172.21.15.32:0/4104191938 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fd814077010 0x7fd814079520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:59:28.467 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:28.466+0000 7fd839400640 1 --2- 172.21.15.32:0/4104191938 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fd814077010 0x7fd814079520 secure :-1 s=READY pgs=775 cs=0 l=1 rev1=1 crypto rx=0x7fd8280017e0 tx=0x7fd828009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:59:28.471 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:28.469+0000 7fd831600640 1 -- 172.21.15.32:0/4104191938 <== mon.2 v2:172.21.15.118:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7fd83406a430 con 0x7fd83415fff0 2024-08-03T20:59:28.756 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:28.754+0000 7fd83be00640 1 -- 172.21.15.32:0/4104191938 --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_command({"prefix": "mds repaired", "role": "0"} v 0) -- 0x7fd834072ac0 con 0x7fd83415fff0 2024-08-03T20:59:29.205 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:29.203+0000 7fd831600640 1 -- 172.21.15.32:0/4104191938 <== mon.2 v2:172.21.15.118:3301/0 7 ==== mon_command_ack([{"prefix": "mds repaired", "role": "0"}]=0 nothing to do: rank is not damaged v288) ==== 107+0+0 (secure 0 0 0) 0x7fd834072ac0 con 0x7fd83415fff0 2024-08-03T20:59:29.205 INFO:teuthology.orchestra.run.smithi032.stderr:nothing to do: rank is not damaged 2024-08-03T20:59:29.208 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:29.206+0000 7fd83be00640 1 -- 172.21.15.32:0/4104191938 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fd814077010 msgr2=0x7fd814079520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:59:29.208 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:29.206+0000 7fd83be00640 1 --2- 172.21.15.32:0/4104191938 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fd814077010 0x7fd814079520 secure :-1 s=READY pgs=775 cs=0 l=1 rev1=1 crypto rx=0x7fd8280017e0 tx=0x7fd828009040 comp rx=0 tx=0).stop 2024-08-03T20:59:29.208 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:29.206+0000 7fd83be00640 1 -- 172.21.15.32:0/4104191938 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fd83415fff0 msgr2=0x7fd834143bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:59:29.208 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:29.206+0000 7fd83be00640 1 --2- 172.21.15.32:0/4104191938 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fd83415fff0 0x7fd834143bd0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fd82c00e9b0 tx=0x7fd82c00ee70 comp rx=0 tx=0).stop 2024-08-03T20:59:29.208 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:29.206+0000 7fd83be00640 1 -- 172.21.15.32:0/4104191938 shutdown_connections 2024-08-03T20:59:29.208 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:29.206+0000 7fd83be00640 1 --2- 172.21.15.32:0/4104191938 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fd814077010 0x7fd814079520 unknown :-1 s=CLOSED pgs=775 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:59:29.208 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:29.206+0000 7fd83be00640 1 --2- 172.21.15.32:0/4104191938 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fd834144f90 0x7fd8341e5ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:59:29.208 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:29.206+0000 7fd83be00640 1 --2- 172.21.15.32:0/4104191938 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fd83415fff0 0x7fd834143bd0 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:59:29.208 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:29.206+0000 7fd83be00640 1 --2- 172.21.15.32:0/4104191938 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd83406e930 0x7fd83415c020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:59:29.208 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:29.206+0000 7fd83be00640 1 -- 172.21.15.32:0/4104191938 >> 172.21.15.32:0/4104191938 conn(0x7fd8340b1bd0 msgr2=0x7fd8340b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:59:29.209 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:29.206+0000 7fd83be00640 1 -- 172.21.15.32:0/4104191938 shutdown_connections 2024-08-03T20:59:29.209 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:29.206+0000 7fd83be00640 1 -- 172.21.15.32:0/4104191938 wait complete. 2024-08-03T20:59:29.231 DEBUG:teuthology.orchestra.run.smithi032:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata import - 2024-08-03T20:59:29.373 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:29.371+0000 7f0401397f40 1 Processor -- start 2024-08-03T20:59:29.373 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:29.371+0000 7f0401397f40 1 -- start start 2024-08-03T20:59:29.375 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:29.373+0000 7f0401397f40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x561ec8fef8e0 0x561ec907cf10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:59:29.376 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:29.373+0000 7f0401397f40 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x561ec9072ba0 0x561ec8fcf430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:59:29.376 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:29.373+0000 7f0401397f40 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561ec907f5b0 0x561ec9074d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:59:29.376 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:29.373+0000 7f0401397f40 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x561ec8ef4e30 con 0x561ec907f5b0 2024-08-03T20:59:29.376 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:29.373+0000 7f0401397f40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x561ec8f52bf0 con 0x561ec8fef8e0 2024-08-03T20:59:29.376 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:29.373+0000 7f0401397f40 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x561ec8feee80 con 0x561ec9072ba0 2024-08-03T20:59:29.376 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:29.373+0000 7f03fb000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561ec907f5b0 0x561ec9074d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:59:29.376 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:29.373+0000 7f03fb000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561ec907f5b0 0x561ec9074d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:41558/0 (socket says 172.21.15.32:41558) 2024-08-03T20:59:29.376 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:29.373+0000 7f03fb000640 1 -- 172.21.15.32:0/1940758120 learned_addr learned my addr 172.21.15.32:0/1940758120 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:59:29.376 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:29.373+0000 7f03fa600640 1 --2- 172.21.15.32:0/1940758120 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x561ec8fef8e0 0x561ec907cf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:59:29.376 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:29.373+0000 7f03f9c00640 1 --2- 172.21.15.32:0/1940758120 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x561ec9072ba0 0x561ec8fcf430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:59:29.376 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:29.374+0000 7f03fb000640 1 -- 172.21.15.32:0/1940758120 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x561ec9072ba0 msgr2=0x561ec8fcf430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:59:29.376 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:29.374+0000 7f03fb000640 1 --2- 172.21.15.32:0/1940758120 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x561ec9072ba0 0x561ec8fcf430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:59:29.377 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:29.374+0000 7f03fb000640 1 -- 172.21.15.32:0/1940758120 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x561ec8fef8e0 msgr2=0x561ec907cf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:59:29.377 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:29.374+0000 7f03fb000640 1 --2- 172.21.15.32:0/1940758120 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x561ec8fef8e0 0x561ec907cf10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:59:29.377 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:29.374+0000 7f03fb000640 1 -- 172.21.15.32:0/1940758120 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561ec8feb3f0 con 0x561ec907f5b0 2024-08-03T20:59:29.377 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:29.374+0000 7f03fb000640 1 --2- 172.21.15.32:0/1940758120 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561ec907f5b0 0x561ec9074d60 secure :-1 s=READY pgs=1934 cs=0 l=1 rev1=1 crypto rx=0x7f03f400c710 tx=0x7f03f400cbd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=145607e6e6e9d449 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:59:29.377 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:29.374+0000 7f03f3e00640 1 -- 172.21.15.32:0/1940758120 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03f4013070 con 0x561ec907f5b0 2024-08-03T20:59:29.377 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:29.374+0000 7f03f3e00640 1 -- 172.21.15.32:0/1940758120 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f03f400a930 con 0x561ec907f5b0 2024-08-03T20:59:29.377 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:29.374+0000 7f03f3e00640 1 -- 172.21.15.32:0/1940758120 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03f400ad80 con 0x561ec907f5b0 2024-08-03T20:59:29.378 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:29.375+0000 7f0401397f40 1 -- 172.21.15.32:0/1940758120 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561ec907f5b0 msgr2=0x561ec9074d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:59:29.378 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:29.375+0000 7f0401397f40 1 --2- 172.21.15.32:0/1940758120 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561ec907f5b0 0x561ec9074d60 secure :-1 s=READY pgs=1934 cs=0 l=1 rev1=1 crypto rx=0x7f03f400c710 tx=0x7f03f400cbd0 comp rx=0 tx=0).stop 2024-08-03T20:59:29.378 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:29.375+0000 7f0401397f40 1 -- 172.21.15.32:0/1940758120 shutdown_connections 2024-08-03T20:59:29.378 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:29.375+0000 7f0401397f40 1 --2- 172.21.15.32:0/1940758120 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561ec907f5b0 0x561ec9074d60 unknown :-1 s=CLOSED pgs=1934 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:59:29.378 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:29.375+0000 7f0401397f40 1 --2- 172.21.15.32:0/1940758120 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x561ec9072ba0 0x561ec8fcf430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:59:29.379 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:29.375+0000 7f0401397f40 1 --2- 172.21.15.32:0/1940758120 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x561ec8fef8e0 0x561ec907cf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:59:29.379 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:29.375+0000 7f0401397f40 1 -- 172.21.15.32:0/1940758120 >> 172.21.15.32:0/1940758120 conn(0x561ec8ec6ac0 msgr2=0x561ec8fea8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:59:29.379 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:29.375+0000 7f0401397f40 1 -- 172.21.15.32:0/1940758120 shutdown_connections 2024-08-03T20:59:29.379 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:29.375+0000 7f0401397f40 1 -- 172.21.15.32:0/1940758120 wait complete. 2024-08-03T20:59:29.379 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:29.376+0000 7f0401397f40 1 Processor -- start 2024-08-03T20:59:29.379 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:29.376+0000 7f0401397f40 1 -- start start 2024-08-03T20:59:29.381 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:29.378+0000 7f0401397f40 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x561ec8fef8e0 0x561ec906a1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:59:29.382 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:29.378+0000 7f0401397f40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x561ec9072ba0 0x561ec9073e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:59:29.382 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:29.378+0000 7f0401397f40 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561ec907f5b0 0x561ec90743b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:59:29.382 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:29.378+0000 7f0401397f40 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x561ec8fd1a80 con 0x561ec907f5b0 2024-08-03T20:59:29.382 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:29.378+0000 7f0401397f40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x561ec8fd1d60 con 0x561ec9072ba0 2024-08-03T20:59:29.382 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:29.378+0000 7f0401397f40 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x561ec8feb0e0 con 0x561ec8fef8e0 2024-08-03T20:59:29.382 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:29.378+0000 7f03fb000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561ec907f5b0 0x561ec90743b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:59:29.382 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:29.378+0000 7f03fb000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561ec907f5b0 0x561ec90743b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:41572/0 (socket says 172.21.15.32:41572) 2024-08-03T20:59:29.382 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:29.378+0000 7f03fb000640 1 -- 172.21.15.32:0/2215333455 learned_addr learned my addr 172.21.15.32:0/2215333455 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:59:29.382 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:29.378+0000 7f03f9c00640 1 --2- 172.21.15.32:0/2215333455 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x561ec9072ba0 0x561ec9073e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:59:29.382 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:29.378+0000 7f03fa600640 1 --2- 172.21.15.32:0/2215333455 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x561ec8fef8e0 0x561ec906a1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:59:29.382 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:29.378+0000 7f03fb000640 1 -- 172.21.15.32:0/2215333455 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x561ec8fef8e0 msgr2=0x561ec906a1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:59:29.382 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:29.378+0000 7f03fb000640 1 --2- 172.21.15.32:0/2215333455 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x561ec8fef8e0 0x561ec906a1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:59:29.383 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:29.378+0000 7f03fb000640 1 -- 172.21.15.32:0/2215333455 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x561ec9072ba0 msgr2=0x561ec9073e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:59:29.383 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:29.378+0000 7f03fb000640 1 --2- 172.21.15.32:0/2215333455 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x561ec9072ba0 0x561ec9073e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:59:29.383 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:29.378+0000 7f03fb000640 1 -- 172.21.15.32:0/2215333455 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561ec91132c0 con 0x561ec907f5b0 2024-08-03T20:59:29.383 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:29.378+0000 7f03fa600640 1 --2- 172.21.15.32:0/2215333455 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x561ec8fef8e0 0x561ec906a1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-03T20:59:29.383 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:29.378+0000 7f03f9c00640 1 --2- 172.21.15.32:0/2215333455 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x561ec9072ba0 0x561ec9073e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-03T20:59:29.383 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:29.379+0000 7f03fb000640 1 --2- 172.21.15.32:0/2215333455 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561ec907f5b0 0x561ec90743b0 secure :-1 s=READY pgs=1935 cs=0 l=1 rev1=1 crypto rx=0x7f03f400b010 tx=0x7f03f40116b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:59:29.383 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:29.379+0000 7f03f2a00640 1 -- 172.21.15.32:0/2215333455 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03f4013040 con 0x561ec907f5b0 2024-08-03T20:59:29.383 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:29.379+0000 7f03f2a00640 1 -- 172.21.15.32:0/2215333455 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f03f4004020 con 0x561ec907f5b0 2024-08-03T20:59:29.383 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:29.379+0000 7f0401397f40 1 -- 172.21.15.32:0/2215333455 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x561ec91135a0 con 0x561ec907f5b0 2024-08-03T20:59:29.383 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:29.379+0000 7f03f2a00640 1 -- 172.21.15.32:0/2215333455 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03f40112e0 con 0x561ec907f5b0 2024-08-03T20:59:29.383 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:29.379+0000 7f0401397f40 1 -- 172.21.15.32:0/2215333455 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561ec906a730 con 0x561ec907f5b0 2024-08-03T20:59:29.384 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:29.381+0000 7f03f2a00640 1 -- 172.21.15.32:0/2215333455 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f03f4016070 con 0x561ec907f5b0 2024-08-03T20:59:29.384 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:29.382+0000 7f03f2a00640 1 --2- 172.21.15.32:0/2215333455 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f03dc0759f0 0x7f03dc077f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:59:29.384 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:29.382+0000 7f03f2a00640 1 -- 172.21.15.32:0/2215333455 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(119..119 src has 1..119) ==== 8778+0+0 (secure 0 0 0) 0x7f03f4099ba0 con 0x561ec907f5b0 2024-08-03T20:59:29.384 INFO:teuthology.orchestra.run.smithi032.stdout:Importing pool 2024-08-03T20:59:29.385 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:29.382+0000 7f03fa600640 1 --2- 172.21.15.32:0/2215333455 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f03dc0759f0 0x7f03dc077f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:59:29.385 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:29.383+0000 7f0401397f40 1 --2- 172.21.15.32:0/2215333455 >> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] conn(0x561ec906b9b0 0x561ec906bdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:59:29.385 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:29.383+0000 7f0401397f40 1 -- 172.21.15.32:0/2215333455 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:1 12.10 12:09401a81:::601.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119) -- 0x561ec906c300 con 0x561ec906b9b0 2024-08-03T20:59:29.385 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:29.383+0000 7f03f9c00640 1 --2- 172.21.15.32:0/2215333455 >> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] conn(0x561ec906b9b0 0x561ec906bdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:59:29.386 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:29.383+0000 7f03fa600640 1 --2- 172.21.15.32:0/2215333455 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f03dc0759f0 0x7f03dc077f00 secure :-1 s=READY pgs=777 cs=0 l=1 rev1=1 crypto rx=0x7f03ec002f50 tx=0x7f03ec002550 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:59:29.386 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:29.384+0000 7f03f9c00640 1 --2- 172.21.15.32:0/2215333455 >> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] conn(0x561ec906b9b0 0x561ec906bdc0 crc :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:59:29.387 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:29.386+0000 7f03f9c00640 1 -- 172.21.15.32:0/2215333455 <== osd.0 v2:172.21.15.32:6816/3066156116 1 ==== osd_op_reply(1 601.00000000 [create] v119'1033 uv1007 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f03e8056470 con 0x561ec906b9b0 2024-08-03T20:59:29.388 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:29.386+0000 7f0401397f40 1 -- 172.21.15.32:0/2215333455 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:2 12.10 12:09401a81:::601.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e119) -- 0x561ec911f6f0 con 0x561ec906b9b0 2024-08-03T20:59:29.390 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:29.389+0000 7f03f9c00640 1 -- 172.21.15.32:0/2215333455 <== osd.0 v2:172.21.15.32:6816/3066156116 2 ==== osd_op_reply(2 601.00000000 [delete] v119'1034 uv1034 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f03e8056470 con 0x561ec906b9b0 2024-08-03T20:59:29.390 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:29.389+0000 7f0401397f40 1 -- 172.21.15.32:0/2215333455 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:3 12.10 12:09401a81:::601.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119) -- 0x561ec911fa90 con 0x561ec906b9b0 2024-08-03T20:59:29.392 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::601.00000000:head# 2024-08-03T20:59:29.392 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:29.390+0000 7f03f9c00640 1 -- 172.21.15.32:0/2215333455 <== osd.0 v2:172.21.15.32:6816/3066156116 3 ==== osd_op_reply(3 601.00000000 [create] v119'1035 uv1035 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f03e8056470 con 0x561ec906b9b0 2024-08-03T20:59:29.392 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:29.390+0000 7f0401397f40 1 -- 172.21.15.32:0/2215333455 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:4 12.10 12:09401a81:::601.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119) -- 0x561ec9120e40 con 0x561ec906b9b0 2024-08-03T20:59:29.394 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:29.392+0000 7f03f9c00640 1 -- 172.21.15.32:0/2215333455 <== osd.0 v2:172.21.15.32:6816/3066156116 4 ==== osd_op_reply(4 601.00000000 [setxattr (30)] v119'1036 uv1036 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f03e8056470 con 0x561ec906b9b0 2024-08-03T20:59:29.394 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:29.392+0000 7f0401397f40 1 -- 172.21.15.32:0/2215333455 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:5 12.10 12:09401a81:::601.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119) -- 0x561ec906c6e0 con 0x561ec906b9b0 2024-08-03T20:59:29.395 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:29.394+0000 7f03f9c00640 1 -- 172.21.15.32:0/2215333455 <== osd.0 v2:172.21.15.32:6816/3066156116 5 ==== osd_op_reply(5 601.00000000 [setxattr (62)] v119'1037 uv1037 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f03e8056470 con 0x561ec906b9b0 2024-08-03T20:59:29.395 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:29.394+0000 7f0401397f40 1 -- 172.21.15.32:0/2215333455 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:6 12.10 12:09401a81:::601.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119) -- 0x561ec911fe30 con 0x561ec906b9b0 2024-08-03T20:59:29.397 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:29.396+0000 7f03f9c00640 1 -- 172.21.15.32:0/2215333455 <== osd.0 v2:172.21.15.32:6816/3066156116 6 ==== osd_op_reply(6 601.00000000 [omap-set-header] v119'1038 uv1038 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f03e8056470 con 0x561ec906b9b0 2024-08-03T20:59:29.397 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:29.396+0000 7f0401397f40 1 -- 172.21.15.32:0/2215333455 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:7 12.10 12:0bd6d154:::602.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119) -- 0x561ec9120240 con 0x561ec906b9b0 2024-08-03T20:59:29.399 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:29.398+0000 7f03f9c00640 1 -- 172.21.15.32:0/2215333455 <== osd.0 v2:172.21.15.32:6816/3066156116 7 ==== osd_op_reply(7 602.00000000 [create] v119'1039 uv1013 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f03e8056470 con 0x561ec906b9b0 2024-08-03T20:59:29.399 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:29.398+0000 7f0401397f40 1 -- 172.21.15.32:0/2215333455 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:8 12.10 12:0bd6d154:::602.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e119) -- 0x561ec9120240 con 0x561ec906b9b0 2024-08-03T20:59:29.401 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:29.399+0000 7f03f9c00640 1 -- 172.21.15.32:0/2215333455 <== osd.0 v2:172.21.15.32:6816/3066156116 8 ==== osd_op_reply(8 602.00000000 [delete] v119'1040 uv1040 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f03e8056470 con 0x561ec906b9b0 2024-08-03T20:59:29.401 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:29.399+0000 7f0401397f40 1 -- 172.21.15.32:0/2215333455 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:9 12.10 12:0bd6d154:::602.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119) -- 0x561ec9120240 con 0x561ec906b9b0 2024-08-03T20:59:29.402 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::602.00000000:head# 2024-08-03T20:59:29.403 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:29.401+0000 7f03f9c00640 1 -- 172.21.15.32:0/2215333455 <== osd.0 v2:172.21.15.32:6816/3066156116 9 ==== osd_op_reply(9 602.00000000 [create] v119'1041 uv1041 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f03e8056470 con 0x561ec906b9b0 2024-08-03T20:59:29.403 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:29.401+0000 7f0401397f40 1 -- 172.21.15.32:0/2215333455 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:10 12.10 12:0bd6d154:::602.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119) -- 0x561ec9120240 con 0x561ec906b9b0 2024-08-03T20:59:29.404 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:29.403+0000 7f03f9c00640 1 -- 172.21.15.32:0/2215333455 <== osd.0 v2:172.21.15.32:6816/3066156116 10 ==== osd_op_reply(10 602.00000000 [setxattr (30)] v119'1042 uv1042 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f03e8056470 con 0x561ec906b9b0 2024-08-03T20:59:29.404 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:29.403+0000 7f0401397f40 1 -- 172.21.15.32:0/2215333455 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:11 12.10 12:0bd6d154:::602.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119) -- 0x561ec9120240 con 0x561ec906b9b0 2024-08-03T20:59:29.406 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:29.405+0000 7f03f9c00640 1 -- 172.21.15.32:0/2215333455 <== osd.0 v2:172.21.15.32:6816/3066156116 11 ==== osd_op_reply(11 602.00000000 [setxattr (62)] v119'1043 uv1043 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f03e8056470 con 0x561ec906b9b0 2024-08-03T20:59:29.406 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:29.405+0000 7f0401397f40 1 -- 172.21.15.32:0/2215333455 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:12 12.10 12:0bd6d154:::602.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119) -- 0x561ec9120240 con 0x561ec906b9b0 2024-08-03T20:59:29.408 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:29.406+0000 7f03f9c00640 1 -- 172.21.15.32:0/2215333455 <== osd.0 v2:172.21.15.32:6816/3066156116 12 ==== osd_op_reply(12 602.00000000 [omap-set-header] v119'1044 uv1044 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f03e8056470 con 0x561ec906b9b0 2024-08-03T20:59:29.408 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:29.407+0000 7f0401397f40 1 -- 172.21.15.32:0/2215333455 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:13 12.10 12:0cf6ad0f:::10000000000.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119) -- 0x561ec9120240 con 0x561ec906b9b0 2024-08-03T20:59:29.410 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:29.408+0000 7f03f9c00640 1 -- 172.21.15.32:0/2215333455 <== osd.0 v2:172.21.15.32:6816/3066156116 13 ==== osd_op_reply(13 10000000000.00000000 [create] v119'1045 uv1020 ondisk = -17 ((17) File exists)) ==== 164+0+0 (crc 0 0 0) 0x7f03e8056470 con 0x561ec906b9b0 2024-08-03T20:59:29.410 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:29.408+0000 7f0401397f40 1 -- 172.21.15.32:0/2215333455 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:14 12.10 12:0cf6ad0f:::10000000000.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e119) -- 0x561ec9120240 con 0x561ec906b9b0 2024-08-03T20:59:29.411 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:29.410+0000 7f03f9c00640 1 -- 172.21.15.32:0/2215333455 <== osd.0 v2:172.21.15.32:6816/3066156116 14 ==== osd_op_reply(14 10000000000.00000000 [delete] v119'1046 uv1046 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f03e8056470 con 0x561ec906b9b0 2024-08-03T20:59:29.412 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:29.410+0000 7f0401397f40 1 -- 172.21.15.32:0/2215333455 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:15 12.10 12:0cf6ad0f:::10000000000.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119) -- 0x561ec9120240 con 0x561ec906b9b0 2024-08-03T20:59:29.413 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::10000000000.00000000:head# 2024-08-03T20:59:29.413 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:29.412+0000 7f03f9c00640 1 -- 172.21.15.32:0/2215333455 <== osd.0 v2:172.21.15.32:6816/3066156116 15 ==== osd_op_reply(15 10000000000.00000000 [create] v119'1047 uv1047 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f03e8056470 con 0x561ec906b9b0 2024-08-03T20:59:29.413 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:29.412+0000 7f0401397f40 1 -- 172.21.15.32:0/2215333455 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:16 12.10 12:0cf6ad0f:::10000000000.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119) -- 0x561ec9120240 con 0x561ec906b9b0 2024-08-03T20:59:29.415 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:29.414+0000 7f03f9c00640 1 -- 172.21.15.32:0/2215333455 <== osd.0 v2:172.21.15.32:6816/3066156116 16 ==== osd_op_reply(16 10000000000.00000000 [setxattr (30)] v119'1048 uv1048 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f03e8056470 con 0x561ec906b9b0 2024-08-03T20:59:29.415 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:29.414+0000 7f0401397f40 1 -- 172.21.15.32:0/2215333455 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:17 12.10 12:0cf6ad0f:::10000000000.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119) -- 0x561ec9120240 con 0x561ec906b9b0 2024-08-03T20:59:29.417 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:29.415+0000 7f03f9c00640 1 -- 172.21.15.32:0/2215333455 <== osd.0 v2:172.21.15.32:6816/3066156116 17 ==== osd_op_reply(17 10000000000.00000000 [setxattr (62)] v119'1049 uv1049 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f03e8056470 con 0x561ec906b9b0 2024-08-03T20:59:29.417 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:29.415+0000 7f0401397f40 1 -- 172.21.15.32:0/2215333455 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:18 12.10 12:0cf6ad0f:::10000000000.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119) -- 0x561ec9120240 con 0x561ec906b9b0 2024-08-03T20:59:29.419 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:29.417+0000 7f03f9c00640 1 -- 172.21.15.32:0/2215333455 <== osd.0 v2:172.21.15.32:6816/3066156116 18 ==== osd_op_reply(18 10000000000.00000000 [omap-set-header] v119'1050 uv1050 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f03e8056470 con 0x561ec906b9b0 2024-08-03T20:59:29.419 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:29.417+0000 7f0401397f40 1 -- 172.21.15.32:0/2215333455 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:19 12.10 12:0cf6ad0f:::10000000000.00000000:head [omap-set-vals in=593b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119) -- 0x561ec91211e0 con 0x561ec906b9b0 2024-08-03T20:59:29.421 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:29.419+0000 7f03f9c00640 1 -- 172.21.15.32:0/2215333455 <== osd.0 v2:172.21.15.32:6816/3066156116 19 ==== osd_op_reply(19 10000000000.00000000 [omap-set-vals] v119'1051 uv1051 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f03e8056470 con 0x561ec906b9b0 2024-08-03T20:59:29.421 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:29.419+0000 7f0401397f40 1 -- 172.21.15.32:0/2215333455 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:20 12.10 12:0d82a743:::600.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119) -- 0x561ec91211e0 con 0x561ec906b9b0 2024-08-03T20:59:29.423 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:29.421+0000 7f03f9c00640 1 -- 172.21.15.32:0/2215333455 <== osd.0 v2:172.21.15.32:6816/3066156116 20 ==== osd_op_reply(20 600.00000000 [create] v119'1052 uv1026 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f03e8056470 con 0x561ec906b9b0 2024-08-03T20:59:29.423 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:29.421+0000 7f0401397f40 1 -- 172.21.15.32:0/2215333455 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:21 12.10 12:0d82a743:::600.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e119) -- 0x561ec9120a00 con 0x561ec906b9b0 2024-08-03T20:59:29.424 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:29.423+0000 7f03f9c00640 1 -- 172.21.15.32:0/2215333455 <== osd.0 v2:172.21.15.32:6816/3066156116 21 ==== osd_op_reply(21 600.00000000 [delete] v119'1053 uv1053 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f03e8056470 con 0x561ec906b9b0 2024-08-03T20:59:29.424 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:29.423+0000 7f0401397f40 1 -- 172.21.15.32:0/2215333455 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:22 12.10 12:0d82a743:::600.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119) -- 0x561ec91209b0 con 0x561ec906b9b0 2024-08-03T20:59:29.426 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::600.00000000:head# 2024-08-03T20:59:29.426 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:29.424+0000 7f03f9c00640 1 -- 172.21.15.32:0/2215333455 <== osd.0 v2:172.21.15.32:6816/3066156116 22 ==== osd_op_reply(22 600.00000000 [create] v119'1054 uv1054 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f03e8056470 con 0x561ec906b9b0 2024-08-03T20:59:29.426 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:29.425+0000 7f0401397f40 1 -- 172.21.15.32:0/2215333455 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:23 12.10 12:0d82a743:::600.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119) -- 0x561ec91209b0 con 0x561ec906b9b0 2024-08-03T20:59:29.428 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:29.426+0000 7f03f9c00640 1 -- 172.21.15.32:0/2215333455 <== osd.0 v2:172.21.15.32:6816/3066156116 23 ==== osd_op_reply(23 600.00000000 [setxattr (30)] v119'1055 uv1055 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f03e8056470 con 0x561ec906b9b0 2024-08-03T20:59:29.428 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:29.426+0000 7f0401397f40 1 -- 172.21.15.32:0/2215333455 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:24 12.10 12:0d82a743:::600.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119) -- 0x561ec91209b0 con 0x561ec906b9b0 2024-08-03T20:59:29.429 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:29.428+0000 7f03f9c00640 1 -- 172.21.15.32:0/2215333455 <== osd.0 v2:172.21.15.32:6816/3066156116 24 ==== osd_op_reply(24 600.00000000 [setxattr (62)] v119'1056 uv1056 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f03e8056470 con 0x561ec906b9b0 2024-08-03T20:59:29.429 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:29.428+0000 7f0401397f40 1 -- 172.21.15.32:0/2215333455 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:25 12.10 12:0d82a743:::600.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119) -- 0x561ec91209b0 con 0x561ec906b9b0 2024-08-03T20:59:29.431 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:29.429+0000 7f03f9c00640 1 -- 172.21.15.32:0/2215333455 <== osd.0 v2:172.21.15.32:6816/3066156116 25 ==== osd_op_reply(25 600.00000000 [omap-set-header] v119'1057 uv1057 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f03e8056470 con 0x561ec906b9b0 2024-08-03T20:59:29.431 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:29.430+0000 7f0401397f40 1 -- 172.21.15.32:0/2215333455 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:26 12.10 12:0d89b25e:::603.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119) -- 0x561ec91209b0 con 0x561ec906b9b0 2024-08-03T20:59:29.433 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:29.431+0000 7f03f9c00640 1 -- 172.21.15.32:0/2215333455 <== osd.0 v2:172.21.15.32:6816/3066156116 26 ==== osd_op_reply(26 603.00000000 [create] v119'1058 uv1032 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f03e8056470 con 0x561ec906b9b0 2024-08-03T20:59:29.433 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:29.431+0000 7f0401397f40 1 -- 172.21.15.32:0/2215333455 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:27 12.10 12:0d89b25e:::603.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e119) -- 0x561ec91209b0 con 0x561ec906b9b0 2024-08-03T20:59:29.435 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:29.433+0000 7f03f9c00640 1 -- 172.21.15.32:0/2215333455 <== osd.0 v2:172.21.15.32:6816/3066156116 27 ==== osd_op_reply(27 603.00000000 [delete] v119'1059 uv1059 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f03e8056470 con 0x561ec906b9b0 2024-08-03T20:59:29.435 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:29.433+0000 7f0401397f40 1 -- 172.21.15.32:0/2215333455 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:28 12.10 12:0d89b25e:::603.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119) -- 0x561ec91209b0 con 0x561ec906b9b0 2024-08-03T20:59:29.436 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::603.00000000:head# 2024-08-03T20:59:29.437 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:29.435+0000 7f03f9c00640 1 -- 172.21.15.32:0/2215333455 <== osd.0 v2:172.21.15.32:6816/3066156116 28 ==== osd_op_reply(28 603.00000000 [create] v119'1060 uv1060 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f03e8056470 con 0x561ec906b9b0 2024-08-03T20:59:29.437 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:29.435+0000 7f0401397f40 1 -- 172.21.15.32:0/2215333455 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:29 12.10 12:0d89b25e:::603.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119) -- 0x561ec91209b0 con 0x561ec906b9b0 2024-08-03T20:59:29.438 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:29.437+0000 7f03f9c00640 1 -- 172.21.15.32:0/2215333455 <== osd.0 v2:172.21.15.32:6816/3066156116 29 ==== osd_op_reply(29 603.00000000 [setxattr (30)] v119'1061 uv1061 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f03e8056470 con 0x561ec906b9b0 2024-08-03T20:59:29.438 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:29.437+0000 7f0401397f40 1 -- 172.21.15.32:0/2215333455 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:30 12.10 12:0d89b25e:::603.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119) -- 0x561ec91209b0 con 0x561ec906b9b0 2024-08-03T20:59:29.440 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:29.438+0000 7f03f9c00640 1 -- 172.21.15.32:0/2215333455 <== osd.0 v2:172.21.15.32:6816/3066156116 30 ==== osd_op_reply(30 603.00000000 [setxattr (62)] v119'1062 uv1062 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f03e8056470 con 0x561ec906b9b0 2024-08-03T20:59:29.440 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:29.438+0000 7f0401397f40 1 -- 172.21.15.32:0/2215333455 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:31 12.10 12:0d89b25e:::603.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119) -- 0x561ec91209b0 con 0x561ec906b9b0 2024-08-03T20:59:29.442 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:29.440+0000 7f03f9c00640 1 -- 172.21.15.32:0/2215333455 <== osd.0 v2:172.21.15.32:6816/3066156116 31 ==== osd_op_reply(31 603.00000000 [omap-set-header] v119'1063 uv1063 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f03e8056470 con 0x561ec906b9b0 2024-08-03T20:59:29.442 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:29.440+0000 7f0401397f40 1 --2- 172.21.15.32:0/2215333455 >> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] conn(0x561ec91216f0 0x561ec9123b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:59:29.442 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:29.440+0000 7f0401397f40 1 -- 172.21.15.32:0/2215333455 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:32 12.14 12:287034c4:::1.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119) -- 0x561ec91209b0 con 0x561ec91216f0 2024-08-03T20:59:29.443 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:29.441+0000 7f03fa600640 1 --2- 172.21.15.32:0/2215333455 >> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] conn(0x561ec91216f0 0x561ec9123b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:59:29.443 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:29.441+0000 7f03fa600640 1 --2- 172.21.15.32:0/2215333455 >> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] conn(0x561ec91216f0 0x561ec9123b10 crc :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:59:29.446 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:29.444+0000 7f03fa600640 1 -- 172.21.15.32:0/2215333455 <== osd.5 v2:172.21.15.118:6810/2724891770 1 ==== osd_op_reply(32 1.00000000.inode [create] v119'752 uv731 ondisk = -17 ((17) File exists)) ==== 160+0+0 (crc 0 0 0) 0x7f03ec011540 con 0x561ec91216f0 2024-08-03T20:59:29.446 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:29.444+0000 7f0401397f40 1 -- 172.21.15.32:0/2215333455 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:33 12.14 12:287034c4:::1.00000000.inode:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e119) -- 0x561ec9124050 con 0x561ec91216f0 2024-08-03T20:59:29.447 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:29.446+0000 7f03fa600640 1 -- 172.21.15.32:0/2215333455 <== osd.5 v2:172.21.15.118:6810/2724891770 2 ==== osd_op_reply(33 1.00000000.inode [delete] v119'753 uv753 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f03ec011540 con 0x561ec91216f0 2024-08-03T20:59:29.448 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:29.446+0000 7f0401397f40 1 -- 172.21.15.32:0/2215333455 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:34 12.14 12:287034c4:::1.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119) -- 0x561ec91243f0 con 0x561ec91216f0 2024-08-03T20:59:29.449 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:29.447+0000 7f03fa600640 1 -- 172.21.15.32:0/2215333455 <== osd.5 v2:172.21.15.118:6810/2724891770 3 ==== osd_op_reply(34 1.00000000.inode [create] v119'754 uv754 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f03ec011540 con 0x561ec91216f0 2024-08-03T20:59:29.449 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::1.00000000.inode:head# 2024-08-03T20:59:29.449 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:29.447+0000 7f0401397f40 1 -- 172.21.15.32:0/2215333455 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:35 12.14 12:287034c4:::1.00000000.inode:head [write 0~582 in=582b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119) -- 0x561ec9124790 con 0x561ec91216f0 2024-08-03T20:59:29.451 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:29.449+0000 7f03fa600640 1 -- 172.21.15.32:0/2215333455 <== osd.5 v2:172.21.15.118:6810/2724891770 4 ==== osd_op_reply(35 1.00000000.inode [write 0~582] v119'755 uv755 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f03ec011540 con 0x561ec91216f0 2024-08-03T20:59:29.451 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:29.449+0000 7f0401397f40 1 -- 172.21.15.32:0/2215333455 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:36 12.14 12:287034c4:::1.00000000.inode:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119) -- 0x561ec9124b30 con 0x561ec91216f0 2024-08-03T20:59:29.453 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:29.451+0000 7f03fa600640 1 -- 172.21.15.32:0/2215333455 <== osd.5 v2:172.21.15.118:6810/2724891770 5 ==== osd_op_reply(36 1.00000000.inode [omap-set-header] v119'756 uv756 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f03ec011540 con 0x561ec91216f0 2024-08-03T20:59:29.453 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:29.451+0000 7f0401397f40 1 -- 172.21.15.32:0/2215333455 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:37 12.14 12:292cf221:::200.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119) -- 0x561ec9124ed0 con 0x561ec91216f0 2024-08-03T20:59:29.455 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:29.453+0000 7f03fa600640 1 -- 172.21.15.32:0/2215333455 <== osd.5 v2:172.21.15.118:6810/2724891770 6 ==== osd_op_reply(37 200.00000000 [create] v119'757 uv750 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f03ec011540 con 0x561ec91216f0 2024-08-03T20:59:29.455 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:29.453+0000 7f0401397f40 1 -- 172.21.15.32:0/2215333455 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:38 12.14 12:292cf221:::200.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e119) -- 0x561ec9125270 con 0x561ec91216f0 2024-08-03T20:59:29.456 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:29.455+0000 7f03fa600640 1 -- 172.21.15.32:0/2215333455 <== osd.5 v2:172.21.15.118:6810/2724891770 7 ==== osd_op_reply(38 200.00000000 [delete] v119'758 uv758 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f03ec011540 con 0x561ec91216f0 2024-08-03T20:59:29.457 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:29.455+0000 7f0401397f40 1 -- 172.21.15.32:0/2215333455 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:39 12.14 12:292cf221:::200.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119) -- 0x561ec9125610 con 0x561ec91216f0 2024-08-03T20:59:29.458 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:29.457+0000 7f03fa600640 1 -- 172.21.15.32:0/2215333455 <== osd.5 v2:172.21.15.118:6810/2724891770 8 ==== osd_op_reply(39 200.00000000 [create] v119'759 uv759 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f03ec011540 con 0x561ec91216f0 2024-08-03T20:59:29.459 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::200.00000000:head# 2024-08-03T20:59:29.459 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:29.457+0000 7f0401397f40 1 -- 172.21.15.32:0/2215333455 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:40 12.14 12:292cf221:::200.00000000:head [write 0~90 in=90b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119) -- 0x561ec9125610 con 0x561ec91216f0 2024-08-03T20:59:29.460 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:29.458+0000 7f03fa600640 1 -- 172.21.15.32:0/2215333455 <== osd.5 v2:172.21.15.118:6810/2724891770 9 ==== osd_op_reply(40 200.00000000 [write 0~90] v119'760 uv760 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f03ec011540 con 0x561ec91216f0 2024-08-03T20:59:29.460 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:29.458+0000 7f0401397f40 1 -- 172.21.15.32:0/2215333455 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:41 12.14 12:292cf221:::200.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119) -- 0x561ec9125610 con 0x561ec91216f0 2024-08-03T20:59:29.461 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:29.460+0000 7f03fa600640 1 -- 172.21.15.32:0/2215333455 <== osd.5 v2:172.21.15.118:6810/2724891770 10 ==== osd_op_reply(41 200.00000000 [omap-set-header] v119'761 uv761 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f03ec011540 con 0x561ec91216f0 2024-08-03T20:59:29.462 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:29.460+0000 7f0401397f40 1 -- 172.21.15.32:0/2215333455 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:42 12.14 12:2e2fa760:::200.00000001:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119) -- 0x561ec9125610 con 0x561ec91216f0 2024-08-03T20:59:29.463 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:29.462+0000 7f03fa600640 1 -- 172.21.15.32:0/2215333455 <== osd.5 v2:172.21.15.118:6810/2724891770 11 ==== osd_op_reply(42 200.00000001 [create] v119'762 uv751 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f03ec011540 con 0x561ec91216f0 2024-08-03T20:59:29.463 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:29.462+0000 7f0401397f40 1 -- 172.21.15.32:0/2215333455 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:43 12.14 12:2e2fa760:::200.00000001:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e119) -- 0x561ec9125610 con 0x561ec91216f0 2024-08-03T20:59:29.465 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:29.464+0000 7f03fa600640 1 -- 172.21.15.32:0/2215333455 <== osd.5 v2:172.21.15.118:6810/2724891770 12 ==== osd_op_reply(43 200.00000001 [delete] v119'763 uv763 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f03ec011540 con 0x561ec91216f0 2024-08-03T20:59:29.465 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:29.464+0000 7f0401397f40 1 -- 172.21.15.32:0/2215333455 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:44 12.14 12:2e2fa760:::200.00000001:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119) -- 0x561ec9125610 con 0x561ec91216f0 2024-08-03T20:59:29.467 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::200.00000001:head# 2024-08-03T20:59:29.467 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:29.465+0000 7f03fa600640 1 -- 172.21.15.32:0/2215333455 <== osd.5 v2:172.21.15.118:6810/2724891770 13 ==== osd_op_reply(44 200.00000001 [create] v119'764 uv764 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f03ec011540 con 0x561ec91216f0 2024-08-03T20:59:29.467 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:29.465+0000 7f0401397f40 1 -- 172.21.15.32:0/2215333455 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:45 12.14 12:2e2fa760:::200.00000001:head [write 0~18118 in=18118b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119) -- 0x561ec9125610 con 0x561ec91216f0 2024-08-03T20:59:29.469 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:29.467+0000 7f03fa600640 1 -- 172.21.15.32:0/2215333455 <== osd.5 v2:172.21.15.118:6810/2724891770 14 ==== osd_op_reply(45 200.00000001 [write 0~18118] v119'765 uv765 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f03ec011540 con 0x561ec91216f0 2024-08-03T20:59:29.469 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:29.467+0000 7f0401397f40 1 -- 172.21.15.32:0/2215333455 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:46 12.14 12:2e2fa760:::200.00000001:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119) -- 0x561ec9125610 con 0x561ec91216f0 2024-08-03T20:59:29.470 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:29.469+0000 7f03fa600640 1 -- 172.21.15.32:0/2215333455 <== osd.5 v2:172.21.15.118:6810/2724891770 15 ==== osd_op_reply(46 200.00000001 [omap-set-header] v119'766 uv766 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f03ec011540 con 0x561ec91216f0 2024-08-03T20:59:29.470 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:29.469+0000 7f0401397f40 1 -- 172.21.15.32:0/2215333455 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:47 12.14 12:2f55791f:::606.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119) -- 0x561ec9125610 con 0x561ec91216f0 2024-08-03T20:59:29.472 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:29.471+0000 7f03fa600640 1 -- 172.21.15.32:0/2215333455 <== osd.5 v2:172.21.15.118:6810/2724891770 16 ==== osd_op_reply(47 606.00000000 [create] v119'767 uv747 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f03ec011540 con 0x561ec91216f0 2024-08-03T20:59:29.472 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:29.471+0000 7f0401397f40 1 -- 172.21.15.32:0/2215333455 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:48 12.14 12:2f55791f:::606.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e119) -- 0x561ec9125610 con 0x561ec91216f0 2024-08-03T20:59:29.474 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:29.472+0000 7f03fa600640 1 -- 172.21.15.32:0/2215333455 <== osd.5 v2:172.21.15.118:6810/2724891770 17 ==== osd_op_reply(48 606.00000000 [delete] v119'768 uv768 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f03ec011540 con 0x561ec91216f0 2024-08-03T20:59:29.475 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:29.473+0000 7f0401397f40 1 -- 172.21.15.32:0/2215333455 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:49 12.14 12:2f55791f:::606.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119) -- 0x561ec9125610 con 0x561ec91216f0 2024-08-03T20:59:29.475 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:29.474+0000 7f03fa600640 1 -- 172.21.15.32:0/2215333455 <== osd.5 v2:172.21.15.118:6810/2724891770 18 ==== osd_op_reply(49 606.00000000 [create] v119'769 uv769 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f03ec011540 con 0x561ec91216f0 2024-08-03T20:59:29.476 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::606.00000000:head# 2024-08-03T20:59:29.476 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:29.474+0000 7f0401397f40 1 -- 172.21.15.32:0/2215333455 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:50 12.14 12:2f55791f:::606.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119) -- 0x561ec9126620 con 0x561ec91216f0 2024-08-03T20:59:29.477 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:29.476+0000 7f03fa600640 1 -- 172.21.15.32:0/2215333455 <== osd.5 v2:172.21.15.118:6810/2724891770 19 ==== osd_op_reply(50 606.00000000 [setxattr (30)] v119'770 uv770 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f03ec011540 con 0x561ec91216f0 2024-08-03T20:59:29.477 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:29.476+0000 7f0401397f40 1 -- 172.21.15.32:0/2215333455 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:51 12.14 12:2f55791f:::606.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119) -- 0x561ec9126620 con 0x561ec91216f0 2024-08-03T20:59:29.478 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:29.477+0000 7f03fa600640 1 -- 172.21.15.32:0/2215333455 <== osd.5 v2:172.21.15.118:6810/2724891770 20 ==== osd_op_reply(51 606.00000000 [setxattr (62)] v119'771 uv771 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f03ec011540 con 0x561ec91216f0 2024-08-03T20:59:29.479 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:29.477+0000 7f0401397f40 1 -- 172.21.15.32:0/2215333455 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:52 12.14 12:2f55791f:::606.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119) -- 0x561ec9125610 con 0x561ec91216f0 2024-08-03T20:59:29.480 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:29.479+0000 7f03fa600640 1 -- 172.21.15.32:0/2215333455 <== osd.5 v2:172.21.15.118:6810/2724891770 21 ==== osd_op_reply(52 606.00000000 [omap-set-header] v119'772 uv772 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f03ec011540 con 0x561ec91216f0 2024-08-03T20:59:29.481 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:29.479+0000 7f0401397f40 1 -- 172.21.15.32:0/2215333455 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:53 12.1c 12:3ed09add:::607.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119) -- 0x561ec9125610 con 0x561ec906b9b0 2024-08-03T20:59:29.482 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:29.480+0000 7f03f9c00640 1 -- 172.21.15.32:0/2215333455 <== osd.0 v2:172.21.15.32:6816/3066156116 32 ==== osd_op_reply(53 607.00000000 [create] v119'201 uv200 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f03e8056470 con 0x561ec906b9b0 2024-08-03T20:59:29.482 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:29.480+0000 7f0401397f40 1 -- 172.21.15.32:0/2215333455 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:54 12.1c 12:3ed09add:::607.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e119) -- 0x561ec9125610 con 0x561ec906b9b0 2024-08-03T20:59:29.483 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:29.482+0000 7f03f9c00640 1 -- 172.21.15.32:0/2215333455 <== osd.0 v2:172.21.15.32:6816/3066156116 33 ==== osd_op_reply(54 607.00000000 [delete] v119'202 uv202 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f03e8056470 con 0x561ec906b9b0 2024-08-03T20:59:29.484 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:29.482+0000 7f0401397f40 1 -- 172.21.15.32:0/2215333455 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:55 12.1c 12:3ed09add:::607.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119) -- 0x561ec9125610 con 0x561ec906b9b0 2024-08-03T20:59:29.484 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::607.00000000:head# 2024-08-03T20:59:29.485 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:29.483+0000 7f03f9c00640 1 -- 172.21.15.32:0/2215333455 <== osd.0 v2:172.21.15.32:6816/3066156116 34 ==== osd_op_reply(55 607.00000000 [create] v119'203 uv203 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f03e8056470 con 0x561ec906b9b0 2024-08-03T20:59:29.485 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:29.483+0000 7f0401397f40 1 -- 172.21.15.32:0/2215333455 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:56 12.1c 12:3ed09add:::607.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119) -- 0x561ec9126620 con 0x561ec906b9b0 2024-08-03T20:59:29.486 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:29.484+0000 7f03f9c00640 1 -- 172.21.15.32:0/2215333455 <== osd.0 v2:172.21.15.32:6816/3066156116 35 ==== osd_op_reply(56 607.00000000 [setxattr (30)] v119'204 uv204 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f03e8056470 con 0x561ec906b9b0 2024-08-03T20:59:29.486 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:29.484+0000 7f0401397f40 1 -- 172.21.15.32:0/2215333455 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:57 12.1c 12:3ed09add:::607.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119) -- 0x561ec9126620 con 0x561ec906b9b0 2024-08-03T20:59:29.487 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:29.486+0000 7f03f9c00640 1 -- 172.21.15.32:0/2215333455 <== osd.0 v2:172.21.15.32:6816/3066156116 36 ==== osd_op_reply(57 607.00000000 [setxattr (62)] v119'205 uv205 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f03e8056470 con 0x561ec906b9b0 2024-08-03T20:59:29.487 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:29.486+0000 7f0401397f40 1 -- 172.21.15.32:0/2215333455 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:58 12.1c 12:3ed09add:::607.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119) -- 0x561ec9125610 con 0x561ec906b9b0 2024-08-03T20:59:29.489 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:29.487+0000 7f03f9c00640 1 -- 172.21.15.32:0/2215333455 <== osd.0 v2:172.21.15.32:6816/3066156116 37 ==== osd_op_reply(58 607.00000000 [omap-set-header] v119'206 uv206 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f03e8056470 con 0x561ec906b9b0 2024-08-03T20:59:29.489 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:29.487+0000 7f0401397f40 1 --2- 172.21.15.32:0/2215333455 >> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] conn(0x561ec9125ee0 0x561ec91262f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:59:29.489 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:29.487+0000 7f0401397f40 1 -- 172.21.15.32:0/2215333455 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:59 12.6 12:654134d2:::mds0_openfiles.0:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119) -- 0x561ec9128a20 con 0x561ec9125ee0 2024-08-03T20:59:29.489 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:29.488+0000 7f03f9c00640 1 --2- 172.21.15.32:0/2215333455 >> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] conn(0x561ec9125ee0 0x561ec91262f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:59:29.489 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:29.488+0000 7f03f9c00640 1 --2- 172.21.15.32:0/2215333455 >> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] conn(0x561ec9125ee0 0x561ec91262f0 crc :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:59:29.493 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:29.492+0000 7f03f9c00640 1 -- 172.21.15.32:0/2215333455 <== osd.1 v2:172.21.15.32:6824/1551241759 1 ==== osd_op_reply(59 mds0_openfiles.0 [create] v119'157 uv156 ondisk = -17 ((17) File exists)) ==== 160+0+0 (crc 0 0 0) 0x7f03e8056470 con 0x561ec9125ee0 2024-08-03T20:59:29.493 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:29.492+0000 7f0401397f40 1 -- 172.21.15.32:0/2215333455 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:60 12.6 12:654134d2:::mds0_openfiles.0:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e119) -- 0x561ec9128a20 con 0x561ec9125ee0 2024-08-03T20:59:29.494 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:29.493+0000 7f03f9c00640 1 -- 172.21.15.32:0/2215333455 <== osd.1 v2:172.21.15.32:6824/1551241759 2 ==== osd_op_reply(60 mds0_openfiles.0 [delete] v119'158 uv158 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f03e8056470 con 0x561ec9125ee0 2024-08-03T20:59:29.495 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:29.493+0000 7f0401397f40 1 -- 172.21.15.32:0/2215333455 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:61 12.6 12:654134d2:::mds0_openfiles.0:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119) -- 0x561ec9128aa0 con 0x561ec9125ee0 2024-08-03T20:59:29.496 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:29.494+0000 7f03f9c00640 1 -- 172.21.15.32:0/2215333455 <== osd.1 v2:172.21.15.32:6824/1551241759 3 ==== osd_op_reply(61 mds0_openfiles.0 [create] v119'159 uv159 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f03e8056470 con 0x561ec9125ee0 2024-08-03T20:59:29.496 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds0_openfiles.0:head# 2024-08-03T20:59:29.497 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:29.494+0000 7f0401397f40 1 -- 172.21.15.32:0/2215333455 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:62 12.6 12:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119) -- 0x561ec9128eb0 con 0x561ec9125ee0 2024-08-03T20:59:29.497 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:29.496+0000 7f03f9c00640 1 -- 172.21.15.32:0/2215333455 <== osd.1 v2:172.21.15.32:6824/1551241759 4 ==== osd_op_reply(62 mds0_openfiles.0 [omap-set-header] v119'160 uv160 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f03e8056470 con 0x561ec9125ee0 2024-08-03T20:59:29.497 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:29.496+0000 7f0401397f40 1 -- 172.21.15.32:0/2215333455 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:63 12.e 12:717a0223:::608.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119) -- 0x561ec9128eb0 con 0x561ec906b9b0 2024-08-03T20:59:29.499 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:29.498+0000 7f03f9c00640 1 -- 172.21.15.32:0/2215333455 <== osd.0 v2:172.21.15.32:6816/3066156116 38 ==== osd_op_reply(63 608.00000000 [create] v119'201 uv200 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f03e8056470 con 0x561ec906b9b0 2024-08-03T20:59:29.499 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:29.498+0000 7f0401397f40 1 -- 172.21.15.32:0/2215333455 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:64 12.e 12:717a0223:::608.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e119) -- 0x561ec9128eb0 con 0x561ec906b9b0 2024-08-03T20:59:29.501 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:29.499+0000 7f03f9c00640 1 -- 172.21.15.32:0/2215333455 <== osd.0 v2:172.21.15.32:6816/3066156116 39 ==== osd_op_reply(64 608.00000000 [delete] v119'202 uv202 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f03e8056470 con 0x561ec906b9b0 2024-08-03T20:59:29.501 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:29.499+0000 7f0401397f40 1 -- 172.21.15.32:0/2215333455 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:65 12.e 12:717a0223:::608.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119) -- 0x561ec9128eb0 con 0x561ec906b9b0 2024-08-03T20:59:29.502 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::608.00000000:head# 2024-08-03T20:59:29.502 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:29.501+0000 7f03f9c00640 1 -- 172.21.15.32:0/2215333455 <== osd.0 v2:172.21.15.32:6816/3066156116 40 ==== osd_op_reply(65 608.00000000 [create] v119'203 uv203 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f03e8056470 con 0x561ec906b9b0 2024-08-03T20:59:29.502 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:29.501+0000 7f0401397f40 1 -- 172.21.15.32:0/2215333455 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:66 12.e 12:717a0223:::608.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119) -- 0x561ec9129ec0 con 0x561ec906b9b0 2024-08-03T20:59:29.504 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:29.502+0000 7f03f9c00640 1 -- 172.21.15.32:0/2215333455 <== osd.0 v2:172.21.15.32:6816/3066156116 41 ==== osd_op_reply(66 608.00000000 [setxattr (30)] v119'204 uv204 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f03e8056470 con 0x561ec906b9b0 2024-08-03T20:59:29.504 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:29.502+0000 7f0401397f40 1 -- 172.21.15.32:0/2215333455 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:67 12.e 12:717a0223:::608.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119) -- 0x561ec9129ec0 con 0x561ec906b9b0 2024-08-03T20:59:29.505 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:29.504+0000 7f03f9c00640 1 -- 172.21.15.32:0/2215333455 <== osd.0 v2:172.21.15.32:6816/3066156116 42 ==== osd_op_reply(67 608.00000000 [setxattr (62)] v119'205 uv205 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f03e8056470 con 0x561ec906b9b0 2024-08-03T20:59:29.505 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:29.504+0000 7f0401397f40 1 -- 172.21.15.32:0/2215333455 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:68 12.e 12:717a0223:::608.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119) -- 0x561ec9128eb0 con 0x561ec906b9b0 2024-08-03T20:59:29.507 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:29.505+0000 7f03f9c00640 1 -- 172.21.15.32:0/2215333455 <== osd.0 v2:172.21.15.32:6816/3066156116 43 ==== osd_op_reply(68 608.00000000 [omap-set-header] v119'206 uv206 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f03e8056470 con 0x561ec906b9b0 2024-08-03T20:59:29.507 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:29.505+0000 7f0401397f40 1 --2- 172.21.15.32:0/2215333455 >> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] conn(0x561ec91293c0 0x561ec912b7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:59:29.507 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:29.505+0000 7f0401397f40 1 -- 172.21.15.32:0/2215333455 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:69 12.15 12:a93a17c2:::604.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119) -- 0x561ec912bd70 con 0x561ec91293c0 2024-08-03T20:59:29.507 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:29.506+0000 7f03fb000640 1 --2- 172.21.15.32:0/2215333455 >> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] conn(0x561ec91293c0 0x561ec912b7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:59:29.507 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:29.506+0000 7f03fb000640 1 --2- 172.21.15.32:0/2215333455 >> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] conn(0x561ec91293c0 0x561ec912b7e0 crc :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:59:29.509 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:29.508+0000 7f03fb000640 1 -- 172.21.15.32:0/2215333455 <== osd.6 v2:172.21.15.118:6825/801947386 1 ==== osd_op_reply(69 604.00000000 [create] v119'371 uv365 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f03f4015030 con 0x561ec91293c0 2024-08-03T20:59:29.509 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:29.508+0000 7f0401397f40 1 -- 172.21.15.32:0/2215333455 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:70 12.15 12:a93a17c2:::604.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e119) -- 0x561ec912c260 con 0x561ec91293c0 2024-08-03T20:59:29.511 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:29.509+0000 7f03fb000640 1 -- 172.21.15.32:0/2215333455 <== osd.6 v2:172.21.15.118:6825/801947386 2 ==== osd_op_reply(70 604.00000000 [delete] v119'372 uv372 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f03f4015030 con 0x561ec91293c0 2024-08-03T20:59:29.511 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:29.509+0000 7f0401397f40 1 -- 172.21.15.32:0/2215333455 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:71 12.15 12:a93a17c2:::604.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119) -- 0x561ec912c740 con 0x561ec91293c0 2024-08-03T20:59:29.513 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:29.511+0000 7f03fb000640 1 -- 172.21.15.32:0/2215333455 <== osd.6 v2:172.21.15.118:6825/801947386 3 ==== osd_op_reply(71 604.00000000 [create] v119'373 uv373 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f03f4015030 con 0x561ec91293c0 2024-08-03T20:59:29.513 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::604.00000000:head# 2024-08-03T20:59:29.513 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:29.511+0000 7f0401397f40 1 -- 172.21.15.32:0/2215333455 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:72 12.15 12:a93a17c2:::604.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119) -- 0x561ec912dff0 con 0x561ec91293c0 2024-08-03T20:59:29.514 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:29.512+0000 7f03fb000640 1 -- 172.21.15.32:0/2215333455 <== osd.6 v2:172.21.15.118:6825/801947386 4 ==== osd_op_reply(72 604.00000000 [setxattr (30)] v119'374 uv374 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f03f4015030 con 0x561ec91293c0 2024-08-03T20:59:29.514 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:29.512+0000 7f0401397f40 1 -- 172.21.15.32:0/2215333455 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:73 12.15 12:a93a17c2:::604.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119) -- 0x561ec912e390 con 0x561ec91293c0 2024-08-03T20:59:29.515 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:29.513+0000 7f03fb000640 1 -- 172.21.15.32:0/2215333455 <== osd.6 v2:172.21.15.118:6825/801947386 5 ==== osd_op_reply(73 604.00000000 [setxattr (62)] v119'375 uv375 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f03f4015030 con 0x561ec91293c0 2024-08-03T20:59:29.515 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:29.513+0000 7f0401397f40 1 -- 172.21.15.32:0/2215333455 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:74 12.15 12:a93a17c2:::604.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119) -- 0x561ec912cfe0 con 0x561ec91293c0 2024-08-03T20:59:29.516 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:29.515+0000 7f03fb000640 1 -- 172.21.15.32:0/2215333455 <== osd.6 v2:172.21.15.118:6825/801947386 6 ==== osd_op_reply(74 604.00000000 [omap-set-header] v119'376 uv376 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f03f4015030 con 0x561ec91293c0 2024-08-03T20:59:29.516 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:29.515+0000 7f0401397f40 1 -- 172.21.15.32:0/2215333455 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:75 12.15 12:aa448500:::500.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119) -- 0x561ec912d3d0 con 0x561ec91293c0 2024-08-03T20:59:29.518 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:29.516+0000 7f03fb000640 1 -- 172.21.15.32:0/2215333455 <== osd.6 v2:172.21.15.118:6825/801947386 7 ==== osd_op_reply(75 500.00000000 [create] v119'377 uv370 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f03f4015030 con 0x561ec91293c0 2024-08-03T20:59:29.518 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:29.516+0000 7f0401397f40 1 -- 172.21.15.32:0/2215333455 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:76 12.15 12:aa448500:::500.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e119) -- 0x561ec912d3d0 con 0x561ec91293c0 2024-08-03T20:59:29.519 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:29.518+0000 7f03fb000640 1 -- 172.21.15.32:0/2215333455 <== osd.6 v2:172.21.15.118:6825/801947386 8 ==== osd_op_reply(76 500.00000000 [delete] v119'378 uv378 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f03f4015030 con 0x561ec91293c0 2024-08-03T20:59:29.519 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:29.518+0000 7f0401397f40 1 -- 172.21.15.32:0/2215333455 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:77 12.15 12:aa448500:::500.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119) -- 0x561ec912d3d0 con 0x561ec91293c0 2024-08-03T20:59:29.521 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:29.519+0000 7f03fb000640 1 -- 172.21.15.32:0/2215333455 <== osd.6 v2:172.21.15.118:6825/801947386 9 ==== osd_op_reply(77 500.00000000 [create] v119'379 uv379 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f03f4015030 con 0x561ec91293c0 2024-08-03T20:59:29.522 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::500.00000000:head# 2024-08-03T20:59:29.522 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:29.519+0000 7f0401397f40 1 -- 172.21.15.32:0/2215333455 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:78 12.15 12:aa448500:::500.00000000:head [write 0~90 in=90b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119) -- 0x561ec912d3d0 con 0x561ec91293c0 2024-08-03T20:59:29.523 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:29.521+0000 7f03fb000640 1 -- 172.21.15.32:0/2215333455 <== osd.6 v2:172.21.15.118:6825/801947386 10 ==== osd_op_reply(78 500.00000000 [write 0~90] v119'380 uv380 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f03f4015030 con 0x561ec91293c0 2024-08-03T20:59:29.523 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:29.521+0000 7f0401397f40 1 -- 172.21.15.32:0/2215333455 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:79 12.15 12:aa448500:::500.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119) -- 0x561ec912d3d0 con 0x561ec91293c0 2024-08-03T20:59:29.524 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:29.522+0000 7f03fb000640 1 -- 172.21.15.32:0/2215333455 <== osd.6 v2:172.21.15.118:6825/801947386 11 ==== osd_op_reply(79 500.00000000 [omap-set-header] v119'381 uv381 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f03f4015030 con 0x561ec91293c0 2024-08-03T20:59:29.524 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:29.522+0000 7f0401397f40 1 --2- 172.21.15.32:0/2215333455 >> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] conn(0x561ec912e730 0x561ec9130b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:59:29.524 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:29.522+0000 7f0401397f40 1 -- 172.21.15.32:0/2215333455 --> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] -- osd_op(unknown.0.0:80 12.d 12:b50e409b:::mds_snaptable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119) -- 0x561ec912d3d0 con 0x561ec912e730 2024-08-03T20:59:29.524 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:29.523+0000 7f03fa600640 1 --2- 172.21.15.32:0/2215333455 >> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] conn(0x561ec912e730 0x561ec9130b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:59:29.525 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:29.523+0000 7f03fa600640 1 --2- 172.21.15.32:0/2215333455 >> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] conn(0x561ec912e730 0x561ec9130b50 crc :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:59:29.526 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:29.525+0000 7f03fa600640 1 -- 172.21.15.32:0/2215333455 <== osd.7 v2:172.21.15.118:6818/653000048 1 ==== osd_op_reply(80 mds_snaptable [create] v119'168 uv167 ondisk = -17 ((17) File exists)) ==== 157+0+0 (crc 0 0 0) 0x7f03ec011540 con 0x561ec912e730 2024-08-03T20:59:29.526 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:29.525+0000 7f0401397f40 1 -- 172.21.15.32:0/2215333455 --> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] -- osd_op(unknown.0.0:81 12.d 12:b50e409b:::mds_snaptable:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e119) -- 0x561ec912d3d0 con 0x561ec912e730 2024-08-03T20:59:29.528 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:29.527+0000 7f03fa600640 1 -- 172.21.15.32:0/2215333455 <== osd.7 v2:172.21.15.118:6818/653000048 2 ==== osd_op_reply(81 mds_snaptable [delete] v119'169 uv169 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f03ec011540 con 0x561ec912e730 2024-08-03T20:59:29.528 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:29.527+0000 7f0401397f40 1 -- 172.21.15.32:0/2215333455 --> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] -- osd_op(unknown.0.0:82 12.d 12:b50e409b:::mds_snaptable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119) -- 0x561ec9131090 con 0x561ec912e730 2024-08-03T20:59:29.529 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:29.528+0000 7f03fa600640 1 -- 172.21.15.32:0/2215333455 <== osd.7 v2:172.21.15.118:6818/653000048 3 ==== osd_op_reply(82 mds_snaptable [create] v119'170 uv170 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f03ec011540 con 0x561ec912e730 2024-08-03T20:59:29.529 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds_snaptable:head# 2024-08-03T20:59:29.530 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:29.528+0000 7f0401397f40 1 -- 172.21.15.32:0/2215333455 --> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] -- osd_op(unknown.0.0:83 12.d 12:b50e409b:::mds_snaptable:head [write 0~70 in=70b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119) -- 0x561ec9131090 con 0x561ec912e730 2024-08-03T20:59:29.531 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:29.529+0000 7f03fa600640 1 -- 172.21.15.32:0/2215333455 <== osd.7 v2:172.21.15.118:6818/653000048 4 ==== osd_op_reply(83 mds_snaptable [write 0~70] v119'171 uv171 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f03ec011540 con 0x561ec912e730 2024-08-03T20:59:29.531 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:29.530+0000 7f0401397f40 1 -- 172.21.15.32:0/2215333455 --> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] -- osd_op(unknown.0.0:84 12.d 12:b50e409b:::mds_snaptable:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119) -- 0x561ec9131090 con 0x561ec912e730 2024-08-03T20:59:29.532 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:29.531+0000 7f03fa600640 1 -- 172.21.15.32:0/2215333455 <== osd.7 v2:172.21.15.118:6818/653000048 5 ==== osd_op_reply(84 mds_snaptable [omap-set-header] v119'172 uv172 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f03ec011540 con 0x561ec912e730 2024-08-03T20:59:29.532 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:29.531+0000 7f0401397f40 1 -- 172.21.15.32:0/2215333455 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:85 12.1d 12:b871830b:::605.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119) -- 0x561ec9131090 con 0x561ec906b9b0 2024-08-03T20:59:29.534 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:29.533+0000 7f03f9c00640 1 -- 172.21.15.32:0/2215333455 <== osd.0 v2:172.21.15.32:6816/3066156116 44 ==== osd_op_reply(85 605.00000000 [create] v119'201 uv200 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f03e8056470 con 0x561ec906b9b0 2024-08-03T20:59:29.534 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:29.533+0000 7f0401397f40 1 -- 172.21.15.32:0/2215333455 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:86 12.1d 12:b871830b:::605.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e119) -- 0x561ec9131090 con 0x561ec906b9b0 2024-08-03T20:59:29.535 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:29.534+0000 7f03f9c00640 1 -- 172.21.15.32:0/2215333455 <== osd.0 v2:172.21.15.32:6816/3066156116 45 ==== osd_op_reply(86 605.00000000 [delete] v119'202 uv202 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f03e8056470 con 0x561ec906b9b0 2024-08-03T20:59:29.535 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:29.534+0000 7f0401397f40 1 -- 172.21.15.32:0/2215333455 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:87 12.1d 12:b871830b:::605.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119) -- 0x561ec9131090 con 0x561ec906b9b0 2024-08-03T20:59:29.536 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::605.00000000:head# 2024-08-03T20:59:29.537 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:29.535+0000 7f03f9c00640 1 -- 172.21.15.32:0/2215333455 <== osd.0 v2:172.21.15.32:6816/3066156116 46 ==== osd_op_reply(87 605.00000000 [create] v119'203 uv203 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f03e8056470 con 0x561ec906b9b0 2024-08-03T20:59:29.537 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:29.535+0000 7f0401397f40 1 -- 172.21.15.32:0/2215333455 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:88 12.1d 12:b871830b:::605.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119) -- 0x561ec91320a0 con 0x561ec906b9b0 2024-08-03T20:59:29.542 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:29.536+0000 7f03f9c00640 1 -- 172.21.15.32:0/2215333455 <== osd.0 v2:172.21.15.32:6816/3066156116 47 ==== osd_op_reply(88 605.00000000 [setxattr (30)] v119'204 uv204 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f03e8056470 con 0x561ec906b9b0 2024-08-03T20:59:29.542 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:29.536+0000 7f0401397f40 1 -- 172.21.15.32:0/2215333455 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:89 12.1d 12:b871830b:::605.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119) -- 0x561ec91320a0 con 0x561ec906b9b0 2024-08-03T20:59:29.542 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:29.538+0000 7f03f9c00640 1 -- 172.21.15.32:0/2215333455 <== osd.0 v2:172.21.15.32:6816/3066156116 48 ==== osd_op_reply(89 605.00000000 [setxattr (62)] v119'205 uv205 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f03e8056470 con 0x561ec906b9b0 2024-08-03T20:59:29.543 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:29.538+0000 7f0401397f40 1 -- 172.21.15.32:0/2215333455 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:90 12.1d 12:b871830b:::605.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119) -- 0x561ec9131090 con 0x561ec906b9b0 2024-08-03T20:59:29.543 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:29.539+0000 7f03f9c00640 1 -- 172.21.15.32:0/2215333455 <== osd.0 v2:172.21.15.32:6816/3066156116 49 ==== osd_op_reply(90 605.00000000 [omap-set-header] v119'206 uv206 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f03e8056470 con 0x561ec906b9b0 2024-08-03T20:59:29.543 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:29.539+0000 7f0401397f40 1 -- 172.21.15.32:0/2215333455 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:91 12.13 12:c91d4a1d:::mds0_inotable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119) -- 0x561ec9131090 con 0x561ec9125ee0 2024-08-03T20:59:29.543 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:29.541+0000 7f03f9c00640 1 -- 172.21.15.32:0/2215333455 <== osd.1 v2:172.21.15.32:6824/1551241759 5 ==== osd_op_reply(91 mds0_inotable [create] v119'335 uv329 ondisk = -17 ((17) File exists)) ==== 157+0+0 (crc 0 0 0) 0x7f03e8056470 con 0x561ec9125ee0 2024-08-03T20:59:29.543 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:29.541+0000 7f0401397f40 1 -- 172.21.15.32:0/2215333455 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:92 12.13 12:c91d4a1d:::mds0_inotable:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e119) -- 0x561ec9131090 con 0x561ec9125ee0 2024-08-03T20:59:29.543 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:29.542+0000 7f03f9c00640 1 -- 172.21.15.32:0/2215333455 <== osd.1 v2:172.21.15.32:6824/1551241759 6 ==== osd_op_reply(92 mds0_inotable [delete] v119'336 uv336 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f03e8056470 con 0x561ec9125ee0 2024-08-03T20:59:29.544 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:29.542+0000 7f0401397f40 1 -- 172.21.15.32:0/2215333455 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:93 12.13 12:c91d4a1d:::mds0_inotable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119) -- 0x561ec9131090 con 0x561ec9125ee0 2024-08-03T20:59:29.545 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds0_inotable:head# 2024-08-03T20:59:29.545 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:29.543+0000 7f03f9c00640 1 -- 172.21.15.32:0/2215333455 <== osd.1 v2:172.21.15.32:6824/1551241759 7 ==== osd_op_reply(93 mds0_inotable [create] v119'337 uv337 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f03e8056470 con 0x561ec9125ee0 2024-08-03T20:59:29.545 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:29.543+0000 7f0401397f40 1 -- 172.21.15.32:0/2215333455 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:94 12.13 12:c91d4a1d:::mds0_inotable:head [write 0~34 in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119) -- 0x561ec9131090 con 0x561ec9125ee0 2024-08-03T20:59:29.546 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:29.545+0000 7f03f9c00640 1 -- 172.21.15.32:0/2215333455 <== osd.1 v2:172.21.15.32:6824/1551241759 8 ==== osd_op_reply(94 mds0_inotable [write 0~34] v119'338 uv338 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f03e8056470 con 0x561ec9125ee0 2024-08-03T20:59:29.546 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:29.545+0000 7f0401397f40 1 -- 172.21.15.32:0/2215333455 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:95 12.13 12:c91d4a1d:::mds0_inotable:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119) -- 0x561ec9131090 con 0x561ec9125ee0 2024-08-03T20:59:29.547 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:29.546+0000 7f03f9c00640 1 -- 172.21.15.32:0/2215333455 <== osd.1 v2:172.21.15.32:6824/1551241759 9 ==== osd_op_reply(95 mds0_inotable [omap-set-header] v119'339 uv339 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f03e8056470 con 0x561ec9125ee0 2024-08-03T20:59:29.548 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:29.546+0000 7f0401397f40 1 -- 172.21.15.32:0/2215333455 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:96 12.13 12:cd5a64a3:::100.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119) -- 0x561ec9131090 con 0x561ec9125ee0 2024-08-03T20:59:29.549 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:29.547+0000 7f03f9c00640 1 -- 172.21.15.32:0/2215333455 <== osd.1 v2:172.21.15.32:6824/1551241759 10 ==== osd_op_reply(96 100.00000000 [create] v119'340 uv334 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f03e8056470 con 0x561ec9125ee0 2024-08-03T20:59:29.549 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:29.547+0000 7f0401397f40 1 -- 172.21.15.32:0/2215333455 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:97 12.13 12:cd5a64a3:::100.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e119) -- 0x561ec9131090 con 0x561ec9125ee0 2024-08-03T20:59:29.550 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:29.549+0000 7f03f9c00640 1 -- 172.21.15.32:0/2215333455 <== osd.1 v2:172.21.15.32:6824/1551241759 11 ==== osd_op_reply(97 100.00000000 [delete] v119'341 uv341 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f03e8056470 con 0x561ec9125ee0 2024-08-03T20:59:29.550 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:29.549+0000 7f0401397f40 1 -- 172.21.15.32:0/2215333455 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:98 12.13 12:cd5a64a3:::100.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119) -- 0x561ec9131090 con 0x561ec9125ee0 2024-08-03T20:59:29.551 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::100.00000000:head# 2024-08-03T20:59:29.552 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:29.550+0000 7f03f9c00640 1 -- 172.21.15.32:0/2215333455 <== osd.1 v2:172.21.15.32:6824/1551241759 12 ==== osd_op_reply(98 100.00000000 [create] v119'342 uv342 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f03e8056470 con 0x561ec9125ee0 2024-08-03T20:59:29.552 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:29.550+0000 7f0401397f40 1 -- 172.21.15.32:0/2215333455 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:99 12.13 12:cd5a64a3:::100.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119) -- 0x561ec9131090 con 0x561ec9125ee0 2024-08-03T20:59:29.553 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:29.551+0000 7f03f9c00640 1 -- 172.21.15.32:0/2215333455 <== osd.1 v2:172.21.15.32:6824/1551241759 13 ==== osd_op_reply(99 100.00000000 [omap-set-header] v119'343 uv343 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f03e8056470 con 0x561ec9125ee0 2024-08-03T20:59:29.553 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:29.551+0000 7f0401397f40 1 -- 172.21.15.32:0/2215333455 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:100 12.13 12:cd5a64a3:::100.00000000:head [omap-set-vals in=5194b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119) -- 0x561ec9133de0 con 0x561ec9125ee0 2024-08-03T20:59:29.555 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:29.553+0000 7f03f9c00640 1 -- 172.21.15.32:0/2215333455 <== osd.1 v2:172.21.15.32:6824/1551241759 14 ==== osd_op_reply(100 100.00000000 [omap-set-vals] v119'344 uv344 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f03e8056470 con 0x561ec9125ee0 2024-08-03T20:59:29.555 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:29.553+0000 7f0401397f40 1 --2- 172.21.15.32:0/2215333455 >> [v2:172.21.15.32:6800/9088303,v1:172.21.15.32:6801/9088303] conn(0x561ec9131b00 0x561ec9134460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:59:29.555 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:29.553+0000 7f0401397f40 1 -- 172.21.15.32:0/2215333455 --> [v2:172.21.15.32:6800/9088303,v1:172.21.15.32:6801/9088303] -- osd_op(unknown.0.0:101 12.b 12:d0630e4c:::mds0_sessionmap:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119) -- 0x561ec9133f20 con 0x561ec9131b00 2024-08-03T20:59:29.555 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:29.553+0000 7f03f9c00640 1 --2- 172.21.15.32:0/2215333455 >> [v2:172.21.15.32:6800/9088303,v1:172.21.15.32:6801/9088303] conn(0x561ec9131b00 0x561ec9134460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:59:29.555 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:29.554+0000 7f03f9c00640 1 --2- 172.21.15.32:0/2215333455 >> [v2:172.21.15.32:6800/9088303,v1:172.21.15.32:6801/9088303] conn(0x561ec9131b00 0x561ec9134460 crc :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:59:29.557 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:29.555+0000 7f03f9c00640 1 -- 172.21.15.32:0/2215333455 <== osd.3 v2:172.21.15.32:6800/9088303 1 ==== osd_op_reply(101 mds0_sessionmap [create] v119'147 uv145 ondisk = -17 ((17) File exists)) ==== 159+0+0 (crc 0 0 0) 0x7f03e8070040 con 0x561ec9131b00 2024-08-03T20:59:29.557 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:29.555+0000 7f0401397f40 1 -- 172.21.15.32:0/2215333455 --> [v2:172.21.15.32:6800/9088303,v1:172.21.15.32:6801/9088303] -- osd_op(unknown.0.0:102 12.b 12:d0630e4c:::mds0_sessionmap:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e119) -- 0x561ec9133f20 con 0x561ec9131b00 2024-08-03T20:59:29.558 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:29.556+0000 7f03f9c00640 1 -- 172.21.15.32:0/2215333455 <== osd.3 v2:172.21.15.32:6800/9088303 2 ==== osd_op_reply(102 mds0_sessionmap [delete] v119'148 uv148 ondisk = 0) ==== 159+0+0 (crc 0 0 0) 0x7f03e8070040 con 0x561ec9131b00 2024-08-03T20:59:29.558 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:29.557+0000 7f0401397f40 1 -- 172.21.15.32:0/2215333455 --> [v2:172.21.15.32:6800/9088303,v1:172.21.15.32:6801/9088303] -- osd_op(unknown.0.0:103 12.b 12:d0630e4c:::mds0_sessionmap:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119) -- 0x561ec9133f20 con 0x561ec9131b00 2024-08-03T20:59:29.559 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds0_sessionmap:head# 2024-08-03T20:59:29.560 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:29.558+0000 7f03f9c00640 1 -- 172.21.15.32:0/2215333455 <== osd.3 v2:172.21.15.32:6800/9088303 3 ==== osd_op_reply(103 mds0_sessionmap [create] v119'149 uv149 ondisk = 0) ==== 159+0+0 (crc 0 0 0) 0x7f03e8070040 con 0x561ec9131b00 2024-08-03T20:59:29.560 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:29.558+0000 7f0401397f40 1 -- 172.21.15.32:0/2215333455 --> [v2:172.21.15.32:6800/9088303,v1:172.21.15.32:6801/9088303] -- osd_op(unknown.0.0:104 12.b 12:d0630e4c:::mds0_sessionmap:head [omap-set-header in=14b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119) -- 0x561ec9133f20 con 0x561ec9131b00 2024-08-03T20:59:29.561 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:29.559+0000 7f03f9c00640 1 -- 172.21.15.32:0/2215333455 <== osd.3 v2:172.21.15.32:6800/9088303 4 ==== osd_op_reply(104 mds0_sessionmap [omap-set-header] v119'150 uv150 ondisk = 0) ==== 159+0+0 (crc 0 0 0) 0x7f03e8070040 con 0x561ec9131b00 2024-08-03T20:59:29.561 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:29.559+0000 7f0401397f40 1 -- 172.21.15.32:0/2215333455 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:105 12.7 12:e0b41d06:::609.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119) -- 0x561ec9133f20 con 0x561ec9125ee0 2024-08-03T20:59:29.562 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:29.561+0000 7f03f9c00640 1 -- 172.21.15.32:0/2215333455 <== osd.1 v2:172.21.15.32:6824/1551241759 15 ==== osd_op_reply(105 609.00000000 [create] v119'201 uv200 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f03e8070040 con 0x561ec9125ee0 2024-08-03T20:59:29.562 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:29.561+0000 7f0401397f40 1 -- 172.21.15.32:0/2215333455 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:106 12.7 12:e0b41d06:::609.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e119) -- 0x561ec9133f20 con 0x561ec9125ee0 2024-08-03T20:59:29.563 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:29.562+0000 7f03f9c00640 1 -- 172.21.15.32:0/2215333455 <== osd.1 v2:172.21.15.32:6824/1551241759 16 ==== osd_op_reply(106 609.00000000 [delete] v119'202 uv202 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f03e8070040 con 0x561ec9125ee0 2024-08-03T20:59:29.564 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:29.562+0000 7f0401397f40 1 -- 172.21.15.32:0/2215333455 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:107 12.7 12:e0b41d06:::609.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119) -- 0x561ec9133f20 con 0x561ec9125ee0 2024-08-03T20:59:29.565 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:29.563+0000 7f03f9c00640 1 -- 172.21.15.32:0/2215333455 <== osd.1 v2:172.21.15.32:6824/1551241759 17 ==== osd_op_reply(107 609.00000000 [create] v119'203 uv203 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f03e8070040 con 0x561ec9125ee0 2024-08-03T20:59:29.565 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::609.00000000:head# 2024-08-03T20:59:29.565 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:29.563+0000 7f0401397f40 1 -- 172.21.15.32:0/2215333455 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:108 12.7 12:e0b41d06:::609.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119) -- 0x561ec9133f20 con 0x561ec9125ee0 2024-08-03T20:59:29.566 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:29.565+0000 7f03f9c00640 1 -- 172.21.15.32:0/2215333455 <== osd.1 v2:172.21.15.32:6824/1551241759 18 ==== osd_op_reply(108 609.00000000 [setxattr (30)] v119'204 uv204 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f03e8070040 con 0x561ec9125ee0 2024-08-03T20:59:29.566 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:29.565+0000 7f0401397f40 1 -- 172.21.15.32:0/2215333455 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:109 12.7 12:e0b41d06:::609.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119) -- 0x561ec9133f20 con 0x561ec9125ee0 2024-08-03T20:59:29.568 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:29.566+0000 7f03f9c00640 1 -- 172.21.15.32:0/2215333455 <== osd.1 v2:172.21.15.32:6824/1551241759 19 ==== osd_op_reply(109 609.00000000 [setxattr (62)] v119'205 uv205 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f03e8070040 con 0x561ec9125ee0 2024-08-03T20:59:29.568 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:29.566+0000 7f0401397f40 1 -- 172.21.15.32:0/2215333455 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:110 12.7 12:e0b41d06:::609.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119) -- 0x561ec9133f20 con 0x561ec9125ee0 2024-08-03T20:59:29.569 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:29.568+0000 7f03f9c00640 1 -- 172.21.15.32:0/2215333455 <== osd.1 v2:172.21.15.32:6824/1551241759 20 ==== osd_op_reply(110 609.00000000 [omap-set-header] v119'206 uv206 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f03e8070040 con 0x561ec9125ee0 2024-08-03T20:59:29.569 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:29.568+0000 7f0401397f40 1 -- 172.21.15.32:0/2215333455 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:111 12.f 12:f1f69726:::400.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119) -- 0x561ec9133f20 con 0x561ec91293c0 2024-08-03T20:59:29.570 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:29.569+0000 7f03fb000640 1 -- 172.21.15.32:0/2215333455 <== osd.6 v2:172.21.15.118:6825/801947386 12 ==== osd_op_reply(111 400.00000000 [create] v119'171 uv170 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f03f4015030 con 0x561ec91293c0 2024-08-03T20:59:29.571 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:29.569+0000 7f0401397f40 1 -- 172.21.15.32:0/2215333455 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:112 12.f 12:f1f69726:::400.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e119) -- 0x561ec9133f20 con 0x561ec91293c0 2024-08-03T20:59:29.572 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:29.571+0000 7f03fb000640 1 -- 172.21.15.32:0/2215333455 <== osd.6 v2:172.21.15.118:6825/801947386 13 ==== osd_op_reply(112 400.00000000 [delete] v119'172 uv172 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f03f4015030 con 0x561ec91293c0 2024-08-03T20:59:29.572 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:29.571+0000 7f0401397f40 1 -- 172.21.15.32:0/2215333455 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:113 12.f 12:f1f69726:::400.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119) -- 0x561ec9133f20 con 0x561ec91293c0 2024-08-03T20:59:29.573 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::400.00000000:head# 2024-08-03T20:59:29.574 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:29.572+0000 7f03fb000640 1 -- 172.21.15.32:0/2215333455 <== osd.6 v2:172.21.15.118:6825/801947386 14 ==== osd_op_reply(113 400.00000000 [create] v119'173 uv173 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f03f4015030 con 0x561ec91293c0 2024-08-03T20:59:29.574 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:29.572+0000 7f0401397f40 1 -- 172.21.15.32:0/2215333455 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:114 12.f 12:f1f69726:::400.00000000:head [write 0~22 in=22b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119) -- 0x561ec9133f20 con 0x561ec91293c0 2024-08-03T20:59:29.575 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:29.574+0000 7f03fb000640 1 -- 172.21.15.32:0/2215333455 <== osd.6 v2:172.21.15.118:6825/801947386 15 ==== osd_op_reply(114 400.00000000 [write 0~22] v119'174 uv174 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f03f4015030 con 0x561ec91293c0 2024-08-03T20:59:29.575 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:29.574+0000 7f0401397f40 1 -- 172.21.15.32:0/2215333455 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:115 12.f 12:f1f69726:::400.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119) -- 0x561ec9133f20 con 0x561ec91293c0 2024-08-03T20:59:29.577 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:29.575+0000 7f03fb000640 1 -- 172.21.15.32:0/2215333455 <== osd.6 v2:172.21.15.118:6825/801947386 16 ==== osd_op_reply(115 400.00000000 [omap-set-header] v119'175 uv175 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f03f4015030 con 0x561ec91293c0 2024-08-03T20:59:29.577 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:29.576+0000 7f0401397f40 1 -- 172.21.15.32:0/2215333455 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:116 12.1f 12:fe07bba1:::100.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119) -- 0x561ec9133f20 con 0x561ec91293c0 2024-08-03T20:59:29.579 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:29.577+0000 7f03fb000640 1 -- 172.21.15.32:0/2215333455 <== osd.6 v2:172.21.15.118:6825/801947386 17 ==== osd_op_reply(116 100.00000000.inode [create] v119'416 uv406 ondisk = -17 ((17) File exists)) ==== 162+0+0 (crc 0 0 0) 0x7f03f4015030 con 0x561ec91293c0 2024-08-03T20:59:29.579 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:29.577+0000 7f0401397f40 1 -- 172.21.15.32:0/2215333455 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:117 12.1f 12:fe07bba1:::100.00000000.inode:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e119) -- 0x561ec9133f20 con 0x561ec91293c0 2024-08-03T20:59:29.580 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:29.579+0000 7f03fb000640 1 -- 172.21.15.32:0/2215333455 <== osd.6 v2:172.21.15.118:6825/801947386 18 ==== osd_op_reply(117 100.00000000.inode [delete] v119'417 uv417 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7f03f4015030 con 0x561ec91293c0 2024-08-03T20:59:29.580 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:29.579+0000 7f0401397f40 1 -- 172.21.15.32:0/2215333455 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:118 12.1f 12:fe07bba1:::100.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119) -- 0x561ec9133f20 con 0x561ec91293c0 2024-08-03T20:59:29.582 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:29.580+0000 7f03fb000640 1 -- 172.21.15.32:0/2215333455 <== osd.6 v2:172.21.15.118:6825/801947386 19 ==== osd_op_reply(118 100.00000000.inode [create] v119'418 uv418 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7f03f4015030 con 0x561ec91293c0 2024-08-03T20:59:29.582 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::100.00000000.inode:head# 2024-08-03T20:59:29.582 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:29.580+0000 7f0401397f40 1 -- 172.21.15.32:0/2215333455 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:119 12.1f 12:fe07bba1:::100.00000000.inode:head [write 0~582 in=582b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119) -- 0x561ec91349a0 con 0x561ec91293c0 2024-08-03T20:59:29.583 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:29.582+0000 7f03fb000640 1 -- 172.21.15.32:0/2215333455 <== osd.6 v2:172.21.15.118:6825/801947386 20 ==== osd_op_reply(119 100.00000000.inode [write 0~582] v119'419 uv419 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7f03f4015030 con 0x561ec91293c0 2024-08-03T20:59:29.583 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:29.582+0000 7f0401397f40 1 -- 172.21.15.32:0/2215333455 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:120 12.1f 12:fe07bba1:::100.00000000.inode:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119) -- 0x561ec91349a0 con 0x561ec91293c0 2024-08-03T20:59:29.585 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:29.583+0000 7f03fb000640 1 -- 172.21.15.32:0/2215333455 <== osd.6 v2:172.21.15.118:6825/801947386 21 ==== osd_op_reply(120 100.00000000.inode [omap-set-header] v119'420 uv420 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7f03f4015030 con 0x561ec91293c0 2024-08-03T20:59:29.585 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:29.583+0000 7f0401397f40 1 -- 172.21.15.32:0/2215333455 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:121 12.1f 12:ff5b34d6:::1.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119) -- 0x561ec91349a0 con 0x561ec91293c0 2024-08-03T20:59:29.587 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:29.585+0000 7f03fb000640 1 -- 172.21.15.32:0/2215333455 <== osd.6 v2:172.21.15.118:6825/801947386 22 ==== osd_op_reply(121 1.00000000 [create] v119'421 uv414 ondisk = -17 ((17) File exists)) ==== 154+0+0 (crc 0 0 0) 0x7f03f4015030 con 0x561ec91293c0 2024-08-03T20:59:29.587 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:29.585+0000 7f0401397f40 1 -- 172.21.15.32:0/2215333455 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:122 12.1f 12:ff5b34d6:::1.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e119) -- 0x561ec91349a0 con 0x561ec91293c0 2024-08-03T20:59:29.588 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:29.586+0000 7f03fb000640 1 -- 172.21.15.32:0/2215333455 <== osd.6 v2:172.21.15.118:6825/801947386 23 ==== osd_op_reply(122 1.00000000 [delete] v119'422 uv422 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7f03f4015030 con 0x561ec91293c0 2024-08-03T20:59:29.588 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:29.586+0000 7f0401397f40 1 -- 172.21.15.32:0/2215333455 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:123 12.1f 12:ff5b34d6:::1.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119) -- 0x561ec91349a0 con 0x561ec91293c0 2024-08-03T20:59:29.589 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::1.00000000:head# 2024-08-03T20:59:29.590 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:29.588+0000 7f03fb000640 1 -- 172.21.15.32:0/2215333455 <== osd.6 v2:172.21.15.118:6825/801947386 24 ==== osd_op_reply(123 1.00000000 [create] v119'423 uv423 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7f03f4015030 con 0x561ec91293c0 2024-08-03T20:59:29.590 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:29.588+0000 7f0401397f40 1 -- 172.21.15.32:0/2215333455 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:124 12.1f 12:ff5b34d6:::1.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119) -- 0x561ec91359b0 con 0x561ec91293c0 2024-08-03T20:59:29.591 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:29.589+0000 7f03fb000640 1 -- 172.21.15.32:0/2215333455 <== osd.6 v2:172.21.15.118:6825/801947386 25 ==== osd_op_reply(124 1.00000000 [setxattr (30)] v119'424 uv424 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7f03f4015030 con 0x561ec91293c0 2024-08-03T20:59:29.591 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:29.589+0000 7f0401397f40 1 -- 172.21.15.32:0/2215333455 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:125 12.1f 12:ff5b34d6:::1.00000000:head [setxattr parent (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119) -- 0x561ec91359b0 con 0x561ec91293c0 2024-08-03T20:59:29.592 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:29.591+0000 7f03fb000640 1 -- 172.21.15.32:0/2215333455 <== osd.6 v2:172.21.15.118:6825/801947386 26 ==== osd_op_reply(125 1.00000000 [setxattr (30)] v119'425 uv425 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7f03f4015030 con 0x561ec91293c0 2024-08-03T20:59:29.592 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:29.591+0000 7f0401397f40 1 -- 172.21.15.32:0/2215333455 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:126 12.1f 12:ff5b34d6:::1.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119) -- 0x561ec91349a0 con 0x561ec91293c0 2024-08-03T20:59:29.593 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:29.592+0000 7f03fb000640 1 -- 172.21.15.32:0/2215333455 <== osd.6 v2:172.21.15.118:6825/801947386 27 ==== osd_op_reply(126 1.00000000 [omap-set-header] v119'426 uv426 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7f03f4015030 con 0x561ec91293c0 2024-08-03T20:59:29.594 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:29.592+0000 7f0401397f40 1 -- 172.21.15.32:0/2215333455 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:127 12.1f 12:ff5b34d6:::1.00000000:head [omap-set-vals in=584b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119) -- 0x561ec9134e50 con 0x561ec91293c0 2024-08-03T20:59:29.596 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:29.593+0000 7f03fb000640 1 -- 172.21.15.32:0/2215333455 <== osd.6 v2:172.21.15.118:6825/801947386 28 ==== osd_op_reply(127 1.00000000 [omap-set-vals] v119'427 uv427 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7f03f4015030 con 0x561ec91293c0 2024-08-03T20:59:29.596 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:29.593+0000 7f0401397f40 1 -- 172.21.15.32:0/2215333455 >> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] conn(0x561ec906b9b0 msgr2=0x561ec906bdc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:59:29.596 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:29.593+0000 7f0401397f40 1 --2- 172.21.15.32:0/2215333455 >> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] conn(0x561ec906b9b0 0x561ec906bdc0 crc :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:59:29.596 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:29.593+0000 7f0401397f40 1 -- 172.21.15.32:0/2215333455 >> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] conn(0x561ec9125ee0 msgr2=0x561ec91262f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:59:29.596 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:29.593+0000 7f0401397f40 1 --2- 172.21.15.32:0/2215333455 >> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] conn(0x561ec9125ee0 0x561ec91262f0 crc :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:59:29.597 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:29.593+0000 7f0401397f40 1 -- 172.21.15.32:0/2215333455 >> [v2:172.21.15.32:6800/9088303,v1:172.21.15.32:6801/9088303] conn(0x561ec9131b00 msgr2=0x561ec9134460 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:59:29.597 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:29.593+0000 7f0401397f40 1 --2- 172.21.15.32:0/2215333455 >> [v2:172.21.15.32:6800/9088303,v1:172.21.15.32:6801/9088303] conn(0x561ec9131b00 0x561ec9134460 crc :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:59:29.597 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:29.594+0000 7f0401397f40 1 -- 172.21.15.32:0/2215333455 >> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] conn(0x561ec91216f0 msgr2=0x561ec9123b10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:59:29.597 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:29.594+0000 7f0401397f40 1 --2- 172.21.15.32:0/2215333455 >> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] conn(0x561ec91216f0 0x561ec9123b10 crc :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:59:29.597 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:29.594+0000 7f03fb000640 1 -- 172.21.15.32:0/2215333455 reap_dead start 2024-08-03T20:59:29.597 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:29.594+0000 7f0401397f40 1 -- 172.21.15.32:0/2215333455 >> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] conn(0x561ec91293c0 msgr2=0x561ec912b7e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:59:29.597 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:29.594+0000 7f0401397f40 1 --2- 172.21.15.32:0/2215333455 >> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] conn(0x561ec91293c0 0x561ec912b7e0 crc :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:59:29.597 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:29.594+0000 7f0401397f40 1 -- 172.21.15.32:0/2215333455 >> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] conn(0x561ec912e730 msgr2=0x561ec9130b50 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:59:29.597 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:29.594+0000 7f0401397f40 1 --2- 172.21.15.32:0/2215333455 >> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] conn(0x561ec912e730 0x561ec9130b50 crc :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:59:29.597 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:29.594+0000 7f0401397f40 1 -- 172.21.15.32:0/2215333455 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f03dc0759f0 msgr2=0x7f03dc077f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:59:29.597 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:29.594+0000 7f0401397f40 1 --2- 172.21.15.32:0/2215333455 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f03dc0759f0 0x7f03dc077f00 secure :-1 s=READY pgs=777 cs=0 l=1 rev1=1 crypto rx=0x7f03ec002f50 tx=0x7f03ec002550 comp rx=0 tx=0).stop 2024-08-03T20:59:29.598 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:29.594+0000 7f0401397f40 1 -- 172.21.15.32:0/2215333455 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561ec907f5b0 msgr2=0x561ec90743b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:59:29.598 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:29.594+0000 7f0401397f40 1 --2- 172.21.15.32:0/2215333455 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561ec907f5b0 0x561ec90743b0 secure :-1 s=READY pgs=1935 cs=0 l=1 rev1=1 crypto rx=0x7f03f400b010 tx=0x7f03f40116b0 comp rx=0 tx=0).stop 2024-08-03T20:59:29.598 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:29.594+0000 7f03fb000640 1 -- 172.21.15.32:0/2215333455 reap_dead start 2024-08-03T20:59:29.598 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:29.594+0000 7f0401397f40 1 -- 172.21.15.32:0/2215333455 shutdown_connections 2024-08-03T20:59:29.598 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:29.594+0000 7f0401397f40 1 -- 172.21.15.32:0/2215333455 >> 172.21.15.32:0/2215333455 conn(0x561ec8ec6ac0 msgr2=0x561ec8fea8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:59:29.598 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:29.594+0000 7f0401397f40 1 -- 172.21.15.32:0/2215333455 shutdown_connections 2024-08-03T20:59:29.598 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:29.594+0000 7f0401397f40 1 -- 172.21.15.32:0/2215333455 wait complete. 2024-08-03T20:59:29.607 DEBUG:teuthology.orchestra.run.smithi032:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata setomapheader 10000000000.00000000 deadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeef 2024-08-03T20:59:29.792 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:29.791+0000 7fa1a5692f40 1 Processor -- start 2024-08-03T20:59:29.792 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:29.791+0000 7fa1a5692f40 1 -- start start 2024-08-03T20:59:29.795 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:29.792+0000 7fa1a5692f40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55ba3864b0e0 0x55ba386d6210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:59:29.795 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:29.792+0000 7fa1a5692f40 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ba386d7840 0x55ba3862e6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:59:29.795 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:29.792+0000 7fa1a5692f40 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55ba386bd370 0x55ba386bd780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:59:29.795 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:29.792+0000 7fa1a5692f40 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55ba38551e30 con 0x55ba386d7840 2024-08-03T20:59:29.795 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:29.792+0000 7fa1a5692f40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x55ba385afbf0 con 0x55ba3864b0e0 2024-08-03T20:59:29.795 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:29.792+0000 7fa1a5692f40 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x55ba38602530 con 0x55ba386bd370 2024-08-03T20:59:29.795 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:29.792+0000 7fa19de00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ba386d7840 0x55ba3862e6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:59:29.795 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:29.792+0000 7fa19de00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ba386d7840 0x55ba3862e6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:55306/0 (socket says 172.21.15.32:55306) 2024-08-03T20:59:29.795 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:29.792+0000 7fa19de00640 1 -- 172.21.15.32:0/1661726213 learned_addr learned my addr 172.21.15.32:0/1661726213 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:59:29.795 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:29.793+0000 7fa19f200640 1 --2- 172.21.15.32:0/1661726213 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55ba386bd370 0x55ba386bd780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:59:29.795 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:29.793+0000 7fa19e800640 1 --2- 172.21.15.32:0/1661726213 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55ba3864b0e0 0x55ba386d6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:59:29.795 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:29.793+0000 7fa19de00640 1 -- 172.21.15.32:0/1661726213 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55ba386bd370 msgr2=0x55ba386bd780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:59:29.796 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:29.793+0000 7fa19de00640 1 --2- 172.21.15.32:0/1661726213 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55ba386bd370 0x55ba386bd780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:59:29.796 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:29.793+0000 7fa19de00640 1 -- 172.21.15.32:0/1661726213 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55ba3864b0e0 msgr2=0x55ba386d6210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:59:29.796 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:29.793+0000 7fa19de00640 1 --2- 172.21.15.32:0/1661726213 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55ba3864b0e0 0x55ba386d6210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:59:29.796 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:29.793+0000 7fa19de00640 1 -- 172.21.15.32:0/1661726213 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ba386b8460 con 0x55ba386d7840 2024-08-03T20:59:29.796 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:29.793+0000 7fa19de00640 1 --2- 172.21.15.32:0/1661726213 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ba386d7840 0x55ba3862e6c0 secure :-1 s=READY pgs=1937 cs=0 l=1 rev1=1 crypto rx=0x7fa18c001de0 tx=0x7fa18c00b670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d5a0cd01096630e8 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:59:29.796 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:29.793+0000 7fa19ca00640 1 -- 172.21.15.32:0/1661726213 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa18c012070 con 0x55ba386d7840 2024-08-03T20:59:29.796 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:29.793+0000 7fa19ca00640 1 -- 172.21.15.32:0/1661726213 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fa18c006050 con 0x55ba386d7840 2024-08-03T20:59:29.796 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:29.793+0000 7fa19ca00640 1 -- 172.21.15.32:0/1661726213 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa18c00ebc0 con 0x55ba386d7840 2024-08-03T20:59:29.797 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:29.794+0000 7fa1a5692f40 1 -- 172.21.15.32:0/1661726213 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ba386d7840 msgr2=0x55ba3862e6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:59:29.797 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:29.794+0000 7fa1a5692f40 1 --2- 172.21.15.32:0/1661726213 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ba386d7840 0x55ba3862e6c0 secure :-1 s=READY pgs=1937 cs=0 l=1 rev1=1 crypto rx=0x7fa18c001de0 tx=0x7fa18c00b670 comp rx=0 tx=0).stop 2024-08-03T20:59:29.798 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:29.794+0000 7fa1a5692f40 1 -- 172.21.15.32:0/1661726213 shutdown_connections 2024-08-03T20:59:29.798 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:29.794+0000 7fa1a5692f40 1 --2- 172.21.15.32:0/1661726213 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55ba386bd370 0x55ba386bd780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:59:29.798 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:29.794+0000 7fa1a5692f40 1 --2- 172.21.15.32:0/1661726213 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ba386d7840 0x55ba3862e6c0 unknown :-1 s=CLOSED pgs=1937 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:59:29.798 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:29.794+0000 7fa1a5692f40 1 --2- 172.21.15.32:0/1661726213 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55ba3864b0e0 0x55ba386d6210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:59:29.798 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:29.794+0000 7fa1a5692f40 1 -- 172.21.15.32:0/1661726213 >> 172.21.15.32:0/1661726213 conn(0x55ba38523ac0 msgr2=0x55ba3862c8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:59:29.798 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:29.794+0000 7fa1a5692f40 1 -- 172.21.15.32:0/1661726213 shutdown_connections 2024-08-03T20:59:29.798 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:29.794+0000 7fa1a5692f40 1 -- 172.21.15.32:0/1661726213 wait complete. 2024-08-03T20:59:29.798 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:29.795+0000 7fa1a5692f40 1 Processor -- start 2024-08-03T20:59:29.798 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:29.795+0000 7fa1a5692f40 1 -- start start 2024-08-03T20:59:29.800 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:29.797+0000 7fa1a5692f40 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55ba3864b0e0 0x55ba38649910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:59:29.800 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:29.797+0000 7fa1a5692f40 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ba386bd370 0x55ba387737c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:59:29.801 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:29.797+0000 7fa1a5692f40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55ba386d7840 0x55ba386cfc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:59:29.801 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:29.797+0000 7fa1a5692f40 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55ba3862de50 con 0x55ba386bd370 2024-08-03T20:59:29.801 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:29.797+0000 7fa1a5692f40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x55ba38603cc0 con 0x55ba386d7840 2024-08-03T20:59:29.801 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:29.797+0000 7fa1a5692f40 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x55ba3862c570 con 0x55ba3864b0e0 2024-08-03T20:59:29.801 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:29.797+0000 7fa19de00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ba386bd370 0x55ba387737c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:59:29.801 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:29.797+0000 7fa19de00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ba386bd370 0x55ba387737c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:55312/0 (socket says 172.21.15.32:55312) 2024-08-03T20:59:29.801 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:29.797+0000 7fa19de00640 1 -- 172.21.15.32:0/659237643 learned_addr learned my addr 172.21.15.32:0/659237643 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:59:29.801 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:29.797+0000 7fa19e800640 1 --2- 172.21.15.32:0/659237643 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55ba3864b0e0 0x55ba38649910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:59:29.801 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:29.797+0000 7fa19f200640 1 --2- 172.21.15.32:0/659237643 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55ba386d7840 0x55ba386cfc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:59:29.802 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:29.798+0000 7fa19de00640 1 -- 172.21.15.32:0/659237643 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55ba3864b0e0 msgr2=0x55ba38649910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:59:29.802 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:29.798+0000 7fa19de00640 1 --2- 172.21.15.32:0/659237643 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55ba3864b0e0 0x55ba38649910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:59:29.802 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:29.798+0000 7fa19de00640 1 -- 172.21.15.32:0/659237643 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55ba386d7840 msgr2=0x55ba386cfc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:59:29.802 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:29.798+0000 7fa19de00640 1 --2- 172.21.15.32:0/659237643 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55ba386d7840 0x55ba386cfc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:59:29.802 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:29.798+0000 7fa19de00640 1 -- 172.21.15.32:0/659237643 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ba386b5dc0 con 0x55ba386bd370 2024-08-03T20:59:29.802 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:29.798+0000 7fa19de00640 1 --2- 172.21.15.32:0/659237643 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ba386bd370 0x55ba387737c0 secure :-1 s=READY pgs=1938 cs=0 l=1 rev1=1 crypto rx=0x55ba386bc280 tx=0x7fa18c010b70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:59:29.802 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:29.798+0000 7fa197400640 1 -- 172.21.15.32:0/659237643 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa18c025070 con 0x55ba386bd370 2024-08-03T20:59:29.802 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:29.798+0000 7fa197400640 1 -- 172.21.15.32:0/659237643 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fa18c00e920 con 0x55ba386bd370 2024-08-03T20:59:29.802 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:29.798+0000 7fa1a5692f40 1 -- 172.21.15.32:0/659237643 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x55ba38649e50 con 0x55ba386bd370 2024-08-03T20:59:29.802 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:29.798+0000 7fa197400640 1 -- 172.21.15.32:0/659237643 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa18c012040 con 0x55ba386bd370 2024-08-03T20:59:29.802 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:29.798+0000 7fa1a5692f40 1 -- 172.21.15.32:0/659237643 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ba386b9550 con 0x55ba386bd370 2024-08-03T20:59:29.804 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:29.800+0000 7fa197400640 1 -- 172.21.15.32:0/659237643 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7fa18c015070 con 0x55ba386bd370 2024-08-03T20:59:29.804 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:29.801+0000 7fa197400640 1 --2- 172.21.15.32:0/659237643 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fa1800759f0 0x7fa180077f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:59:29.804 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:29.801+0000 7fa197400640 1 -- 172.21.15.32:0/659237643 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(119..119 src has 1..119) ==== 8778+0+0 (secure 0 0 0) 0x7fa18c098f10 con 0x55ba386bd370 2024-08-03T20:59:29.804 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:29.802+0000 7fa1a5692f40 1 --2- 172.21.15.32:0/659237643 >> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] conn(0x55ba386b6a90 0x55ba3877c760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:59:29.804 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:29.802+0000 7fa1a5692f40 1 -- 172.21.15.32:0/659237643 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:1 12.10 12:0cf6ad0f:::10000000000.00000000:head [omap-set-header in=80b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119) -- 0x55ba38773d00 con 0x55ba386b6a90 2024-08-03T20:59:29.804 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:29.802+0000 7fa19e800640 1 --2- 172.21.15.32:0/659237643 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fa1800759f0 0x7fa180077f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:59:29.804 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:29.802+0000 7fa19f200640 1 --2- 172.21.15.32:0/659237643 >> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] conn(0x55ba386b6a90 0x55ba3877c760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:59:29.804 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:29.803+0000 7fa19e800640 1 --2- 172.21.15.32:0/659237643 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fa1800759f0 0x7fa180077f00 secure :-1 s=READY pgs=778 cs=0 l=1 rev1=1 crypto rx=0x7fa19000c740 tx=0x7fa190004880 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:59:29.805 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:29.803+0000 7fa19f200640 1 --2- 172.21.15.32:0/659237643 >> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] conn(0x55ba386b6a90 0x55ba3877c760 crc :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:59:29.806 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:29.805+0000 7fa19f200640 1 -- 172.21.15.32:0/659237643 <== osd.0 v2:172.21.15.32:6816/3066156116 1 ==== osd_op_reply(1 10000000000.00000000 [omap-set-header] v119'1064 uv1064 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7fa19800b040 con 0x55ba386b6a90 2024-08-03T20:59:29.807 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:29.805+0000 7fa1a5692f40 1 -- 172.21.15.32:0/659237643 >> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] conn(0x55ba386b6a90 msgr2=0x55ba3877c760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:59:29.807 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:29.805+0000 7fa1a5692f40 1 --2- 172.21.15.32:0/659237643 >> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] conn(0x55ba386b6a90 0x55ba3877c760 crc :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:59:29.807 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:29.805+0000 7fa1a5692f40 1 -- 172.21.15.32:0/659237643 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fa1800759f0 msgr2=0x7fa180077f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:59:29.807 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:29.805+0000 7fa1a5692f40 1 --2- 172.21.15.32:0/659237643 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fa1800759f0 0x7fa180077f00 secure :-1 s=READY pgs=778 cs=0 l=1 rev1=1 crypto rx=0x7fa19000c740 tx=0x7fa190004880 comp rx=0 tx=0).stop 2024-08-03T20:59:29.807 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:29.805+0000 7fa1a5692f40 1 -- 172.21.15.32:0/659237643 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ba386bd370 msgr2=0x55ba387737c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:59:29.807 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:29.805+0000 7fa1a5692f40 1 --2- 172.21.15.32:0/659237643 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ba386bd370 0x55ba387737c0 secure :-1 s=READY pgs=1938 cs=0 l=1 rev1=1 crypto rx=0x55ba386bc280 tx=0x7fa18c010b70 comp rx=0 tx=0).stop 2024-08-03T20:59:29.807 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:29.805+0000 7fa19f200640 1 -- 172.21.15.32:0/659237643 reap_dead start 2024-08-03T20:59:29.807 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:29.805+0000 7fa1a5692f40 1 -- 172.21.15.32:0/659237643 shutdown_connections 2024-08-03T20:59:29.807 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:29.805+0000 7fa1a5692f40 1 -- 172.21.15.32:0/659237643 >> 172.21.15.32:0/659237643 conn(0x55ba38523ac0 msgr2=0x55ba386b7600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:59:29.807 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:29.805+0000 7fa1a5692f40 1 -- 172.21.15.32:0/659237643 shutdown_connections 2024-08-03T20:59:29.807 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:29.805+0000 7fa1a5692f40 1 -- 172.21.15.32:0/659237643 wait complete. 2024-08-03T20:59:29.816 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs joinable true 2024-08-03T20:59:30.048 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:30.046+0000 7f50f2800640 1 -- 172.21.15.32:0/3756016476 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f50ec160350 msgr2=0x7f50ec06e630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:59:30.048 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:30.046+0000 7f50f2800640 1 --2- 172.21.15.32:0/3756016476 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f50ec160350 0x7f50ec06e630 secure :-1 s=READY pgs=1940 cs=0 l=1 rev1=1 crypto rx=0x7f50ec06da30 tx=0x7f50e402a8b0 comp rx=0 tx=0).stop 2024-08-03T20:59:30.048 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:30.046+0000 7f50f2800640 1 -- 172.21.15.32:0/3756016476 shutdown_connections 2024-08-03T20:59:30.048 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:30.046+0000 7f50f2800640 1 --2- 172.21.15.32:0/3756016476 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f50ec160350 0x7f50ec06e630 unknown :-1 s=CLOSED pgs=1940 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:59:30.048 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:30.046+0000 7f50f2800640 1 --2- 172.21.15.32:0/3756016476 >> v2:172.21.15.118:3300/0 conn(0x7f50ec159ff0 0x7f50ec15fe10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:59:30.048 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:30.046+0000 7f50f2800640 1 -- 172.21.15.32:0/3756016476 >> 172.21.15.32:0/3756016476 conn(0x7f50ec0cfd80 msgr2=0x7f50ec0d0190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:59:30.048 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:30.046+0000 7f50f2800640 1 -- 172.21.15.32:0/3756016476 shutdown_connections 2024-08-03T20:59:30.048 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:30.046+0000 7f50f2800640 1 -- 172.21.15.32:0/3756016476 wait complete. 2024-08-03T20:59:30.050 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:30.048+0000 7f50f2800640 1 Processor -- start 2024-08-03T20:59:30.050 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:30.048+0000 7f50f2800640 1 -- start start 2024-08-03T20:59:30.052 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:30.050+0000 7f50f2800640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f50ec159ff0 0x7f50ec0b5260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:59:30.052 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:30.050+0000 7f50f2800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f50ec160350 0x7f50ec15c540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:59:30.052 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:30.050+0000 7f50f2800640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f50ec1609d0 0x7f50ec15ca80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:59:30.052 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:30.050+0000 7f50f2800640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f50ec06ef00 con 0x7f50ec160350 2024-08-03T20:59:30.052 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:30.050+0000 7f50f2800640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f50ec06ed80 con 0x7f50ec159ff0 2024-08-03T20:59:30.052 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:30.050+0000 7f50f2800640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7f50ec06f080 con 0x7f50ec1609d0 2024-08-03T20:59:30.052 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:30.050+0000 7f50f0a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f50ec160350 0x7f50ec15c540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:59:30.052 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:30.050+0000 7f50f0a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f50ec160350 0x7f50ec15c540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:55330/0 (socket says 172.21.15.32:55330) 2024-08-03T20:59:30.052 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:30.050+0000 7f50f0a00640 1 -- 172.21.15.32:0/919441121 learned_addr learned my addr 172.21.15.32:0/919441121 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:59:30.052 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:30.050+0000 7f50f1400640 1 --2- 172.21.15.32:0/919441121 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f50ec159ff0 0x7f50ec0b5260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:59:30.053 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:30.050+0000 7f50f1e00640 1 --2- 172.21.15.32:0/919441121 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f50ec1609d0 0x7f50ec15ca80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:59:30.054 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:30.051+0000 7f50f0a00640 1 -- 172.21.15.32:0/919441121 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f50ec1609d0 msgr2=0x7f50ec15ca80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:59:30.054 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:30.051+0000 7f50f0a00640 1 --2- 172.21.15.32:0/919441121 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f50ec1609d0 0x7f50ec15ca80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:59:30.054 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:30.051+0000 7f50f0a00640 1 -- 172.21.15.32:0/919441121 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f50ec159ff0 msgr2=0x7f50ec0b5260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:59:30.054 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:30.051+0000 7f50f0a00640 1 --2- 172.21.15.32:0/919441121 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f50ec159ff0 0x7f50ec0b5260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:59:30.054 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:30.051+0000 7f50f0a00640 1 -- 172.21.15.32:0/919441121 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f50ec0c68d0 con 0x7f50ec160350 2024-08-03T20:59:30.054 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:30.051+0000 7f50f0a00640 1 --2- 172.21.15.32:0/919441121 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f50ec160350 0x7f50ec15c540 secure :-1 s=READY pgs=1941 cs=0 l=1 rev1=1 crypto rx=0x7f50d800e790 tx=0x7f50d800ec50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:59:30.055 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:30.051+0000 7f50e9600640 1 -- 172.21.15.32:0/919441121 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f50d800eea0 con 0x7f50ec160350 2024-08-03T20:59:30.055 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:30.051+0000 7f50e9600640 1 -- 172.21.15.32:0/919441121 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f50d800cdb0 con 0x7f50ec160350 2024-08-03T20:59:30.055 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:30.051+0000 7f50f2800640 1 -- 172.21.15.32:0/919441121 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f50ec0b57a0 con 0x7f50ec160350 2024-08-03T20:59:30.055 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:30.051+0000 7f50e9600640 1 -- 172.21.15.32:0/919441121 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f50d8010690 con 0x7f50ec160350 2024-08-03T20:59:30.055 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:30.051+0000 7f50f2800640 1 -- 172.21.15.32:0/919441121 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f50ec0c74e0 con 0x7f50ec160350 2024-08-03T20:59:30.055 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:30.052+0000 7f50f2800640 1 -- 172.21.15.32:0/919441121 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f50ec15d680 con 0x7f50ec160350 2024-08-03T20:59:30.055 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:30.054+0000 7f50e9600640 1 -- 172.21.15.32:0/919441121 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f50d8010830 con 0x7f50ec160350 2024-08-03T20:59:30.056 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:30.054+0000 7f50e9600640 1 --2- 172.21.15.32:0/919441121 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f50cc077010 0x7f50cc079520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:59:30.056 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:30.055+0000 7f50e9600640 1 -- 172.21.15.32:0/919441121 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(119..119 src has 1..119) ==== 8778+0+0 (secure 0 0 0) 0x7f50d8099f10 con 0x7f50ec160350 2024-08-03T20:59:30.056 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:30.055+0000 7f50f1400640 1 --2- 172.21.15.32:0/919441121 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f50cc077010 0x7f50cc079520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:59:30.057 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:30.056+0000 7f50f1400640 1 --2- 172.21.15.32:0/919441121 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f50cc077010 0x7f50cc079520 secure :-1 s=READY pgs=779 cs=0 l=1 rev1=1 crypto rx=0x7f50e0000f60 tx=0x7f50e0009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:59:30.061 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:30.059+0000 7f50e9600640 1 -- 172.21.15.32:0/919441121 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7f50ec15d680 con 0x7f50ec160350 2024-08-03T20:59:30.311 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:30.309+0000 7f50f2800640 1 -- 172.21.15.32:0/919441121 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "joinable", "val": "true"} v 0) -- 0x7f50ec0c6f90 con 0x7f50ec160350 2024-08-03T20:59:30.523 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:30.521+0000 7f50e9600640 1 -- 172.21.15.32:0/919441121 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "joinable", "val": "true"}]=0 cephfs marked joinable; MDS may join as newly active. v289) ==== 162+0+0 (secure 0 0 0) 0x7f50d8061e10 con 0x7f50ec160350 2024-08-03T20:59:30.523 INFO:teuthology.orchestra.run.smithi032.stderr:cephfs marked joinable; MDS may join as newly active. 2024-08-03T20:59:30.526 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:30.524+0000 7f50f2800640 1 -- 172.21.15.32:0/919441121 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f50cc077010 msgr2=0x7f50cc079520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:59:30.526 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:30.524+0000 7f50f2800640 1 --2- 172.21.15.32:0/919441121 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f50cc077010 0x7f50cc079520 secure :-1 s=READY pgs=779 cs=0 l=1 rev1=1 crypto rx=0x7f50e0000f60 tx=0x7f50e0009040 comp rx=0 tx=0).stop 2024-08-03T20:59:30.526 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:30.524+0000 7f50f2800640 1 -- 172.21.15.32:0/919441121 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f50ec160350 msgr2=0x7f50ec15c540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:59:30.526 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:30.524+0000 7f50f2800640 1 --2- 172.21.15.32:0/919441121 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f50ec160350 0x7f50ec15c540 secure :-1 s=READY pgs=1941 cs=0 l=1 rev1=1 crypto rx=0x7f50d800e790 tx=0x7f50d800ec50 comp rx=0 tx=0).stop 2024-08-03T20:59:30.526 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:30.524+0000 7f50f2800640 1 -- 172.21.15.32:0/919441121 shutdown_connections 2024-08-03T20:59:30.526 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:30.524+0000 7f50f2800640 1 --2- 172.21.15.32:0/919441121 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f50cc077010 0x7f50cc079520 unknown :-1 s=CLOSED pgs=779 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:59:30.526 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:30.524+0000 7f50f2800640 1 --2- 172.21.15.32:0/919441121 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f50ec1609d0 0x7f50ec15ca80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:59:30.527 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:30.524+0000 7f50f2800640 1 --2- 172.21.15.32:0/919441121 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f50ec160350 0x7f50ec15c540 unknown :-1 s=CLOSED pgs=1941 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:59:30.527 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:30.524+0000 7f50f2800640 1 --2- 172.21.15.32:0/919441121 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f50ec159ff0 0x7f50ec0b5260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:59:30.527 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:30.524+0000 7f50f2800640 1 -- 172.21.15.32:0/919441121 >> 172.21.15.32:0/919441121 conn(0x7f50ec0cfd80 msgr2=0x7f50ec0ad8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:59:30.527 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:30.524+0000 7f50f2800640 1 -- 172.21.15.32:0/919441121 shutdown_connections 2024-08-03T20:59:30.527 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:30.524+0000 7f50f2800640 1 -- 172.21.15.32:0/919441121 wait complete. 2024-08-03T20:59:30.953 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-08-03T20:59:31.183 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:31.181+0000 7ff7a4000640 1 -- 172.21.15.32:0/742841973 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff79c15fff0 msgr2=0x7ff79c160400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:59:31.183 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:31.181+0000 7ff7a4000640 1 --2- 172.21.15.32:0/742841973 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff79c15fff0 0x7ff79c160400 secure :-1 s=READY pgs=1943 cs=0 l=1 rev1=1 crypto rx=0x7ff794005fd0 tx=0x7ff79402a8b0 comp rx=0 tx=0).stop 2024-08-03T20:59:31.183 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:31.181+0000 7ff7a4000640 1 -- 172.21.15.32:0/742841973 shutdown_connections 2024-08-03T20:59:31.184 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:31.181+0000 7ff7a4000640 1 --2- 172.21.15.32:0/742841973 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff79c15fff0 0x7ff79c160400 unknown :-1 s=CLOSED pgs=1943 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:59:31.184 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:31.181+0000 7ff7a4000640 1 --2- 172.21.15.32:0/742841973 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7ff79c06e930 0x7ff79c144f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:59:31.184 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:31.181+0000 7ff7a4000640 1 -- 172.21.15.32:0/742841973 >> 172.21.15.32:0/742841973 conn(0x7ff79c0b1bd0 msgr2=0x7ff79c0b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:59:31.184 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:31.181+0000 7ff7a4000640 1 -- 172.21.15.32:0/742841973 shutdown_connections 2024-08-03T20:59:31.184 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:31.181+0000 7ff7a4000640 1 -- 172.21.15.32:0/742841973 wait complete. 2024-08-03T20:59:31.185 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:31.184+0000 7ff7a4000640 1 Processor -- start 2024-08-03T20:59:31.185 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:31.184+0000 7ff7a4000640 1 -- start start 2024-08-03T20:59:31.187 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:31.185+0000 7ff7a4000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff79c06e930 0x7ff79c0b4a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:59:31.188 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:31.185+0000 7ff7a4000640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7ff79c073e60 0x7ff79c15b580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:59:31.188 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:31.185+0000 7ff7a4000640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ff79c15fff0 0x7ff79c1f84e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:59:31.188 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:31.185+0000 7ff7a4000640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7ff79c0cf840 con 0x7ff79c06e930 2024-08-03T20:59:31.188 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:31.185+0000 7ff7a4000640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7ff79c145f30 con 0x7ff79c15fff0 2024-08-03T20:59:31.188 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:31.185+0000 7ff7a4000640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7ff79c0cf9c0 con 0x7ff79c073e60 2024-08-03T20:59:31.188 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:31.185+0000 7ff7a1600640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff79c06e930 0x7ff79c0b4a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:59:31.188 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:31.185+0000 7ff7a1600640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff79c06e930 0x7ff79c0b4a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:55346/0 (socket says 172.21.15.32:55346) 2024-08-03T20:59:31.188 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:31.186+0000 7ff7a1600640 1 -- 172.21.15.32:0/2323240601 learned_addr learned my addr 172.21.15.32:0/2323240601 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:59:31.188 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:31.186+0000 7ff7a2000640 1 --2- 172.21.15.32:0/2323240601 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ff79c15fff0 0x7ff79c1f84e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:59:31.188 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:31.186+0000 7ff7a0c00640 1 --2- 172.21.15.32:0/2323240601 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7ff79c073e60 0x7ff79c15b580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:59:31.188 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:31.186+0000 7ff7a1600640 1 -- 172.21.15.32:0/2323240601 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7ff79c073e60 msgr2=0x7ff79c15b580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:59:31.189 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:31.186+0000 7ff7a1600640 1 --2- 172.21.15.32:0/2323240601 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7ff79c073e60 0x7ff79c15b580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:59:31.189 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:31.186+0000 7ff7a1600640 1 -- 172.21.15.32:0/2323240601 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ff79c15fff0 msgr2=0x7ff79c1f84e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:59:31.189 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:31.186+0000 7ff7a1600640 1 --2- 172.21.15.32:0/2323240601 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ff79c15fff0 0x7ff79c1f84e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:59:31.189 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:31.186+0000 7ff7a1600640 1 -- 172.21.15.32:0/2323240601 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff79c0b3ef0 con 0x7ff79c06e930 2024-08-03T20:59:31.189 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:31.186+0000 7ff7a0c00640 1 --2- 172.21.15.32:0/2323240601 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7ff79c073e60 0x7ff79c15b580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-03T20:59:31.189 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:31.186+0000 7ff7a1600640 1 --2- 172.21.15.32:0/2323240601 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff79c06e930 0x7ff79c0b4a40 secure :-1 s=READY pgs=1944 cs=0 l=1 rev1=1 crypto rx=0x7ff79000d7a0 tx=0x7ff79000dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:59:31.190 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:31.187+0000 7ff799600640 1 -- 172.21.15.32:0/2323240601 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff790017070 con 0x7ff79c06e930 2024-08-03T20:59:31.190 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:31.187+0000 7ff799600640 1 -- 172.21.15.32:0/2323240601 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7ff790004310 con 0x7ff79c06e930 2024-08-03T20:59:31.190 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:31.187+0000 7ff799600640 1 -- 172.21.15.32:0/2323240601 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff790009c30 con 0x7ff79c06e930 2024-08-03T20:59:31.190 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:31.187+0000 7ff7a4000640 1 -- 172.21.15.32:0/2323240601 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff79c143680 con 0x7ff79c06e930 2024-08-03T20:59:31.190 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:31.187+0000 7ff7a4000640 1 -- 172.21.15.32:0/2323240601 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff79c143c40 con 0x7ff79c06e930 2024-08-03T20:59:31.190 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:31.188+0000 7ff7a4000640 1 -- 172.21.15.32:0/2323240601 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff79c160bc0 con 0x7ff79c06e930 2024-08-03T20:59:31.191 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:31.189+0000 7ff799600640 1 -- 172.21.15.32:0/2323240601 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7ff790005020 con 0x7ff79c06e930 2024-08-03T20:59:31.191 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:31.190+0000 7ff799600640 1 --2- 172.21.15.32:0/2323240601 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7ff77c077010 0x7ff77c079520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:59:31.191 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:31.190+0000 7ff799600640 1 -- 172.21.15.32:0/2323240601 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(119..119 src has 1..119) ==== 8778+0+0 (secure 0 0 0) 0x7ff7900997e0 con 0x7ff79c06e930 2024-08-03T20:59:31.192 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:31.190+0000 7ff7a0c00640 1 --2- 172.21.15.32:0/2323240601 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7ff77c077010 0x7ff77c079520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:59:31.192 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:31.191+0000 7ff7a0c00640 1 --2- 172.21.15.32:0/2323240601 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7ff77c077010 0x7ff77c079520 secure :-1 s=READY pgs=780 cs=0 l=1 rev1=1 crypto rx=0x7ff788000f60 tx=0x7ff788009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:59:31.195 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:31.194+0000 7ff799600640 1 -- 172.21.15.32:0/2323240601 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7ff790020020 con 0x7ff79c06e930 2024-08-03T20:59:31.445 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:31.443+0000 7ff7a4000640 1 -- 172.21.15.32:0/2323240601 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff79c074480 con 0x7ff79c06e930 2024-08-03T20:59:31.445 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:31.444+0000 7ff799600640 1 -- 172.21.15.32:0/2323240601 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 290 v290) ==== 95+0+5226 (secure 0 0 0) 0x7ff790061dc0 con 0x7ff79c06e930 2024-08-03T20:59:31.446 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-03T20:59:31.446 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":290,"btime":"2024-08-03T20:59:30:525737+0000","default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6439,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6835/531080160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6834","nonce":531080160},{"type":"v1","addr":"172.21.15.118:6835","nonce":531080160}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":84},{"gid":6454,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6837/2796882319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6836","nonce":2796882319},{"type":"v1","addr":"172.21.15.118:6837","nonce":2796882319}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":86},{"gid":6813,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6834/2686933890","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":2686933890},{"type":"v1","addr":"172.21.15.32:6834","nonce":2686933890}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":85}],"filesystems":[{"mdsmap":{"epoch":290,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-08-03T20:54:26.950518+0000","modified":"2024-08-03T20:59:30.525723+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":119,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":9882},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9882":{"gid":9882,"name":"a","rank":0,"incarnation":290,"state":"up:replay","state_seq":1,"addr":"172.21.15.32:6835/2106229503","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6833","nonce":2106229503},{"type":"v1","addr":"172.21.15.32:6835","nonce":2106229503}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":6}]} 2024-08-03T20:59:31.447 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 290 2024-08-03T20:59:31.448 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:31.446+0000 7ff7a4000640 1 -- 172.21.15.32:0/2323240601 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7ff77c077010 msgr2=0x7ff77c079520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:59:31.449 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:31.446+0000 7ff7a4000640 1 --2- 172.21.15.32:0/2323240601 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7ff77c077010 0x7ff77c079520 secure :-1 s=READY pgs=780 cs=0 l=1 rev1=1 crypto rx=0x7ff788000f60 tx=0x7ff788009040 comp rx=0 tx=0).stop 2024-08-03T20:59:31.449 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:31.446+0000 7ff7a4000640 1 -- 172.21.15.32:0/2323240601 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff79c06e930 msgr2=0x7ff79c0b4a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:59:31.449 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:31.446+0000 7ff7a4000640 1 --2- 172.21.15.32:0/2323240601 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff79c06e930 0x7ff79c0b4a40 secure :-1 s=READY pgs=1944 cs=0 l=1 rev1=1 crypto rx=0x7ff79000d7a0 tx=0x7ff79000dc60 comp rx=0 tx=0).stop 2024-08-03T20:59:31.449 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:31.446+0000 7ff7a4000640 1 -- 172.21.15.32:0/2323240601 shutdown_connections 2024-08-03T20:59:31.449 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:31.446+0000 7ff7a4000640 1 --2- 172.21.15.32:0/2323240601 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7ff77c077010 0x7ff77c079520 unknown :-1 s=CLOSED pgs=780 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:59:31.450 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:31.446+0000 7ff7a4000640 1 --2- 172.21.15.32:0/2323240601 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ff79c15fff0 0x7ff79c1f84e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:59:31.450 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:31.446+0000 7ff7a4000640 1 --2- 172.21.15.32:0/2323240601 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7ff79c073e60 0x7ff79c15b580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:59:31.450 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:31.446+0000 7ff7a4000640 1 --2- 172.21.15.32:0/2323240601 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff79c06e930 0x7ff79c0b4a40 unknown :-1 s=CLOSED pgs=1944 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:59:31.450 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:31.446+0000 7ff7a4000640 1 -- 172.21.15.32:0/2323240601 >> 172.21.15.32:0/2323240601 conn(0x7ff79c0b1bd0 msgr2=0x7ff79c140cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:59:31.450 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:31.446+0000 7ff7a4000640 1 -- 172.21.15.32:0/2323240601 shutdown_connections 2024-08-03T20:59:31.451 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:31.446+0000 7ff7a4000640 1 -- 172.21.15.32:0/2323240601 wait complete. 2024-08-03T20:59:31.473 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 290, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-08-03T20:54:26.950518+0000', 'modified': '2024-08-03T20:59:30.525723+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 119, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 9882}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9882': {'gid': 9882, 'name': 'a', 'rank': 0, 'incarnation': 290, 'state': 'up:replay', 'state_seq': 1, 'addr': '172.21.15.32:6835/2106229503', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6833', 'nonce': 2106229503}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 2106229503}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [13], 'metadata_pool': 12, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 0, 'qdb_cluster': []} 2024-08-03T20:59:31.473 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9882:up:replay 2024-08-03T20:59:31.473 DEBUG:tasks.ceph_test_case:wait_until_true: waiting (timeout=60 retry_count=0)... 2024-08-03T20:59:34.210 INFO:tasks.ceph.mds.a.smithi032.stderr:2024-08-03T20:59:34.209+0000 7f97e4200640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-08-03T20:59:36.877 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-08-03T20:59:37.115 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:37.114+0000 7f82e4000640 1 -- 172.21.15.32:0/2716411529 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f82dc06e930 msgr2=0x7f82dc144f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:59:37.115 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:37.114+0000 7f82e4000640 1 --2- 172.21.15.32:0/2716411529 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f82dc06e930 0x7f82dc144f90 secure :-1 s=READY pgs=1945 cs=0 l=1 rev1=1 crypto rx=0x7f82dc0b2f80 tx=0x7f82d002a560 comp rx=0 tx=0).stop 2024-08-03T20:59:37.115 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:37.114+0000 7f82e4000640 1 -- 172.21.15.32:0/2716411529 shutdown_connections 2024-08-03T20:59:37.115 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:37.114+0000 7f82e4000640 1 --2- 172.21.15.32:0/2716411529 >> v2:172.21.15.118:3300/0 conn(0x7f82dc0747b0 0x7f82dc0cc8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:59:37.115 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:37.114+0000 7f82e4000640 1 --2- 172.21.15.32:0/2716411529 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f82dc073e60 0x7f82dc074270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:59:37.116 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:37.114+0000 7f82e4000640 1 --2- 172.21.15.32:0/2716411529 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f82dc06e930 0x7f82dc144f90 unknown :-1 s=CLOSED pgs=1945 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:59:37.116 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:37.114+0000 7f82e4000640 1 -- 172.21.15.32:0/2716411529 >> 172.21.15.32:0/2716411529 conn(0x7f82dc0b1bd0 msgr2=0x7f82dc0b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:59:37.116 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:37.114+0000 7f82e4000640 1 -- 172.21.15.32:0/2716411529 shutdown_connections 2024-08-03T20:59:37.116 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:37.114+0000 7f82e4000640 1 -- 172.21.15.32:0/2716411529 wait complete. 2024-08-03T20:59:37.117 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:37.116+0000 7f82e4000640 1 Processor -- start 2024-08-03T20:59:37.117 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:37.117+0000 7f82e4000640 1 -- start start 2024-08-03T20:59:37.122 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:37.118+0000 7f82e4000640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f82dc06e930 0x7f82dc0abf50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:59:37.122 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:37.118+0000 7f82e4000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f82dc073e60 0x7f82dc1f4ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:59:37.122 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:37.118+0000 7f82e4000640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f82dc0747b0 0x7f82dc0ae670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:59:37.122 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:37.118+0000 7f82e4000640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f82dc0cf9c0 con 0x7f82dc073e60 2024-08-03T20:59:37.122 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:37.118+0000 7f82e4000640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f82dc0cf840 con 0x7f82dc0747b0 2024-08-03T20:59:37.122 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:37.118+0000 7f82e4000640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7f82dc0cce20 con 0x7f82dc06e930 2024-08-03T20:59:37.122 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:37.118+0000 7f82e0c00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f82dc073e60 0x7f82dc1f4ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:59:37.124 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:37.118+0000 7f82e0c00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f82dc073e60 0x7f82dc1f4ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:55352/0 (socket says 172.21.15.32:55352) 2024-08-03T20:59:37.124 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:37.118+0000 7f82e0c00640 1 -- 172.21.15.32:0/1110429391 learned_addr learned my addr 172.21.15.32:0/1110429391 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:59:37.124 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:37.118+0000 7f82e2000640 1 --2- 172.21.15.32:0/1110429391 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f82dc0747b0 0x7f82dc0ae670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:59:37.124 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:37.118+0000 7f82e1600640 1 --2- 172.21.15.32:0/1110429391 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f82dc06e930 0x7f82dc0abf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:59:37.124 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:37.119+0000 7f82e0c00640 1 -- 172.21.15.32:0/1110429391 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f82dc06e930 msgr2=0x7f82dc0abf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:59:37.124 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:37.119+0000 7f82e0c00640 1 --2- 172.21.15.32:0/1110429391 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f82dc06e930 0x7f82dc0abf50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:59:37.125 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:37.119+0000 7f82e0c00640 1 -- 172.21.15.32:0/1110429391 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f82dc0747b0 msgr2=0x7f82dc0ae670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:59:37.125 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:37.119+0000 7f82e0c00640 1 --2- 172.21.15.32:0/1110429391 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f82dc0747b0 0x7f82dc0ae670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:59:37.125 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:37.119+0000 7f82e0c00640 1 -- 172.21.15.32:0/1110429391 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f82dc0ac9c0 con 0x7f82dc073e60 2024-08-03T20:59:37.125 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:37.119+0000 7f82e0c00640 1 --2- 172.21.15.32:0/1110429391 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f82dc073e60 0x7f82dc1f4ff0 secure :-1 s=READY pgs=1946 cs=0 l=1 rev1=1 crypto rx=0x7f82c800d6d0 tx=0x7f82c800db90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:59:37.125 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:37.119+0000 7f82d9600640 1 -- 172.21.15.32:0/1110429391 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82c8017070 con 0x7f82dc073e60 2024-08-03T20:59:37.125 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:37.120+0000 7f82d9600640 1 -- 172.21.15.32:0/1110429391 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f82c8005330 con 0x7f82dc073e60 2024-08-03T20:59:37.125 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:37.120+0000 7f82e4000640 1 -- 172.21.15.32:0/1110429391 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f82dc0acec0 con 0x7f82dc073e60 2024-08-03T20:59:37.125 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:37.120+0000 7f82d9600640 1 -- 172.21.15.32:0/1110429391 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82c8009820 con 0x7f82dc073e60 2024-08-03T20:59:37.125 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:37.120+0000 7f82e4000640 1 -- 172.21.15.32:0/1110429391 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f82dc0ca1d0 con 0x7f82dc073e60 2024-08-03T20:59:37.125 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:37.121+0000 7f82e4000640 1 -- 172.21.15.32:0/1110429391 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f82dc0acc30 con 0x7f82dc073e60 2024-08-03T20:59:37.125 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:37.122+0000 7f82d9600640 1 -- 172.21.15.32:0/1110429391 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f82c8009bd0 con 0x7f82dc073e60 2024-08-03T20:59:37.125 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:37.123+0000 7f82d9600640 1 --2- 172.21.15.32:0/1110429391 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f82bc077010 0x7f82bc079520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:59:37.125 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:37.123+0000 7f82d9600640 1 -- 172.21.15.32:0/1110429391 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(119..119 src has 1..119) ==== 8778+0+0 (secure 0 0 0) 0x7f82c8099700 con 0x7f82dc073e60 2024-08-03T20:59:37.126 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:37.124+0000 7f82e1600640 1 --2- 172.21.15.32:0/1110429391 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f82bc077010 0x7f82bc079520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:59:37.126 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:37.125+0000 7f82e1600640 1 --2- 172.21.15.32:0/1110429391 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f82bc077010 0x7f82bc079520 secure :-1 s=READY pgs=781 cs=0 l=1 rev1=1 crypto rx=0x7f82d0033000 tx=0x7f82d0004730 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:59:37.128 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:37.128+0000 7f82d9600640 1 -- 172.21.15.32:0/1110429391 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7f82c8061690 con 0x7f82dc073e60 2024-08-03T20:59:37.377 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:37.376+0000 7f82e4000640 1 -- 172.21.15.32:0/1110429391 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f82dc0c9ce0 con 0x7f82dc073e60 2024-08-03T20:59:37.377 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:37.377+0000 7f82d9600640 1 -- 172.21.15.32:0/1110429391 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 293 v293) ==== 95+0+5233 (secure 0 0 0) 0x7f82c8065be0 con 0x7f82dc073e60 2024-08-03T20:59:37.378 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-03T20:59:37.378 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":293,"btime":"2024-08-03T20:59:33:536830+0000","default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6439,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6835/531080160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6834","nonce":531080160},{"type":"v1","addr":"172.21.15.118:6835","nonce":531080160}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":84},{"gid":6454,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6837/2796882319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6836","nonce":2796882319},{"type":"v1","addr":"172.21.15.118:6837","nonce":2796882319}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":86},{"gid":6813,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6834/2686933890","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":2686933890},{"type":"v1","addr":"172.21.15.32:6834","nonce":2686933890}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":85}],"filesystems":[{"mdsmap":{"epoch":293,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-08-03T20:54:26.950518+0000","modified":"2024-08-03T20:59:33.536820+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":119,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":9882},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9882":{"gid":9882,"name":"a","rank":0,"incarnation":290,"state":"up:active","state_seq":4,"addr":"172.21.15.32:6835/2106229503","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6833","nonce":2106229503},{"type":"v1","addr":"172.21.15.32:6835","nonce":2106229503}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9882,"qdb_cluster":[9882]},"id":6}]} 2024-08-03T20:59:37.378 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 293 2024-08-03T20:59:37.380 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:37.379+0000 7f82e4000640 1 -- 172.21.15.32:0/1110429391 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f82bc077010 msgr2=0x7f82bc079520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:59:37.380 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:37.380+0000 7f82e4000640 1 --2- 172.21.15.32:0/1110429391 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f82bc077010 0x7f82bc079520 secure :-1 s=READY pgs=781 cs=0 l=1 rev1=1 crypto rx=0x7f82d0033000 tx=0x7f82d0004730 comp rx=0 tx=0).stop 2024-08-03T20:59:37.381 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:37.380+0000 7f82e4000640 1 -- 172.21.15.32:0/1110429391 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f82dc073e60 msgr2=0x7f82dc1f4ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:59:37.381 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:37.380+0000 7f82e4000640 1 --2- 172.21.15.32:0/1110429391 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f82dc073e60 0x7f82dc1f4ff0 secure :-1 s=READY pgs=1946 cs=0 l=1 rev1=1 crypto rx=0x7f82c800d6d0 tx=0x7f82c800db90 comp rx=0 tx=0).stop 2024-08-03T20:59:37.381 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:37.380+0000 7f82e4000640 1 -- 172.21.15.32:0/1110429391 shutdown_connections 2024-08-03T20:59:37.381 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:37.380+0000 7f82e4000640 1 --2- 172.21.15.32:0/1110429391 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f82bc077010 0x7f82bc079520 unknown :-1 s=CLOSED pgs=781 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:59:37.381 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:37.380+0000 7f82e4000640 1 --2- 172.21.15.32:0/1110429391 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f82dc0747b0 0x7f82dc0ae670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:59:37.381 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:37.380+0000 7f82e4000640 1 --2- 172.21.15.32:0/1110429391 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f82dc073e60 0x7f82dc1f4ff0 unknown :-1 s=CLOSED pgs=1946 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:59:37.381 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:37.380+0000 7f82e4000640 1 --2- 172.21.15.32:0/1110429391 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f82dc06e930 0x7f82dc0abf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:59:37.381 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:37.380+0000 7f82e4000640 1 -- 172.21.15.32:0/1110429391 >> 172.21.15.32:0/1110429391 conn(0x7f82dc0b1bd0 msgr2=0x7f82dc0b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:59:37.381 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:37.380+0000 7f82e4000640 1 -- 172.21.15.32:0/1110429391 shutdown_connections 2024-08-03T20:59:37.381 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:37.380+0000 7f82e4000640 1 -- 172.21.15.32:0/1110429391 wait complete. 2024-08-03T20:59:37.405 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 293, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-08-03T20:54:26.950518+0000', 'modified': '2024-08-03T20:59:33.536820+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 119, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 9882}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9882': {'gid': 9882, 'name': 'a', 'rank': 0, 'incarnation': 290, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.32:6835/2106229503', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6833', 'nonce': 2106229503}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 2106229503}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [13], 'metadata_pool': 12, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9882, 'qdb_cluster': [9882]} 2024-08-03T20:59:37.405 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-08-03T20:59:37.405 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2024-08-03T20:59:37.635 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:37.634+0000 7f897d600640 1 -- 172.21.15.32:0/649408362 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f89780d0350 msgr2=0x7f89780d0760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:59:37.635 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:37.634+0000 7f897d600640 1 --2- 172.21.15.32:0/649408362 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f89780d0350 0x7f89780d0760 secure :-1 s=READY pgs=1948 cs=0 l=1 rev1=1 crypto rx=0x7f8978074520 tx=0x7f897002a790 comp rx=0 tx=0).stop 2024-08-03T20:59:37.636 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:37.634+0000 7f897d600640 1 -- 172.21.15.32:0/649408362 shutdown_connections 2024-08-03T20:59:37.636 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:37.634+0000 7f897d600640 1 --2- 172.21.15.32:0/649408362 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f89780d0350 0x7f89780d0760 unknown :-1 s=CLOSED pgs=1948 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:59:37.636 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:37.634+0000 7f897d600640 1 --2- 172.21.15.32:0/649408362 >> v2:172.21.15.118:3300/0 conn(0x7f89780d1c30 0x7f897806e8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:59:37.636 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:37.634+0000 7f897d600640 1 -- 172.21.15.32:0/649408362 >> 172.21.15.32:0/649408362 conn(0x7f89780b3bb0 msgr2=0x7f89780b3fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:59:37.636 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:37.635+0000 7f897d600640 1 -- 172.21.15.32:0/649408362 shutdown_connections 2024-08-03T20:59:37.636 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:37.635+0000 7f897d600640 1 -- 172.21.15.32:0/649408362 wait complete. 2024-08-03T20:59:37.637 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:37.637+0000 7f897d600640 1 Processor -- start 2024-08-03T20:59:37.637 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:37.637+0000 7f897d600640 1 -- start start 2024-08-03T20:59:37.640 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:37.638+0000 7f897d600640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f89780b78d0 0x7f89780c9150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:59:37.641 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:37.638+0000 7f897d600640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f89780d0350 0x7f89781f8e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:59:37.641 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:37.639+0000 7f897d600640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f89780d1c30 0x7f89781f9370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:59:37.641 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:37.639+0000 7f897d600640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f89780d0e20 con 0x7f89780d0350 2024-08-03T20:59:37.641 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:37.639+0000 7f897d600640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f89780d0ca0 con 0x7f89780d1c30 2024-08-03T20:59:37.641 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:37.639+0000 7f897d600640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7f89780d0fa0 con 0x7f89780b78d0 2024-08-03T20:59:37.641 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:37.639+0000 7f8976000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f89780d0350 0x7f89781f8e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:59:37.641 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:37.639+0000 7f8976000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f89780d0350 0x7f89781f8e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:55380/0 (socket says 172.21.15.32:55380) 2024-08-03T20:59:37.641 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:37.639+0000 7f8976000640 1 -- 172.21.15.32:0/1824402147 learned_addr learned my addr 172.21.15.32:0/1824402147 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:59:37.641 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:37.639+0000 7f8977400640 1 --2- 172.21.15.32:0/1824402147 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f89780d1c30 0x7f89781f9370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:59:37.641 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:37.639+0000 7f8976a00640 1 --2- 172.21.15.32:0/1824402147 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f89780b78d0 0x7f89780c9150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:59:37.641 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:37.639+0000 7f8976000640 1 -- 172.21.15.32:0/1824402147 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f89780b78d0 msgr2=0x7f89780c9150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:59:37.641 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:37.639+0000 7f8976000640 1 --2- 172.21.15.32:0/1824402147 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f89780b78d0 0x7f89780c9150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:59:37.641 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:37.639+0000 7f8976000640 1 -- 172.21.15.32:0/1824402147 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f89780d1c30 msgr2=0x7f89781f9370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:59:37.642 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:37.639+0000 7f8976000640 1 --2- 172.21.15.32:0/1824402147 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f89780d1c30 0x7f89781f9370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:59:37.642 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:37.639+0000 7f8976000640 1 -- 172.21.15.32:0/1824402147 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f89780cae70 con 0x7f89780d0350 2024-08-03T20:59:37.642 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:37.640+0000 7f8976000640 1 --2- 172.21.15.32:0/1824402147 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f89780d0350 0x7f89781f8e30 secure :-1 s=READY pgs=1949 cs=0 l=1 rev1=1 crypto rx=0x7f896400c430 tx=0x7f896400c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:59:37.642 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:37.640+0000 7f896ea00640 1 -- 172.21.15.32:0/1824402147 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8964011790 con 0x7f89780d0350 2024-08-03T20:59:37.642 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:37.640+0000 7f896ea00640 1 -- 172.21.15.32:0/1824402147 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f8964017070 con 0x7f89780d0350 2024-08-03T20:59:37.642 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:37.640+0000 7f897d600640 1 -- 172.21.15.32:0/1824402147 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f89780cb150 con 0x7f89780d0350 2024-08-03T20:59:37.642 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:37.640+0000 7f896ea00640 1 -- 172.21.15.32:0/1824402147 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8964011cc0 con 0x7f89780d0350 2024-08-03T20:59:37.642 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:37.640+0000 7f897d600640 1 -- 172.21.15.32:0/1824402147 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f89780cb710 con 0x7f89780d0350 2024-08-03T20:59:37.644 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:37.643+0000 7f896ea00640 1 -- 172.21.15.32:0/1824402147 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f896401c460 con 0x7f89780d0350 2024-08-03T20:59:37.645 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:37.644+0000 7f896ea00640 1 --2- 172.21.15.32:0/1824402147 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f8958077010 0x7f8958079520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:59:37.645 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:37.645+0000 7f896ea00640 1 -- 172.21.15.32:0/1824402147 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(119..119 src has 1..119) ==== 8778+0+0 (secure 0 0 0) 0x7f89640991e0 con 0x7f89780d0350 2024-08-03T20:59:37.645 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:37.645+0000 7f898479c740 1 Processor -- start 2024-08-03T20:59:37.645 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:37.645+0000 7f8976a00640 1 --2- 172.21.15.32:0/1824402147 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f8958077010 0x7f8958079520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:59:37.645 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:37.645+0000 7f898479c740 1 -- start start 2024-08-03T20:59:37.646 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:37.646+0000 7f8976a00640 1 --2- 172.21.15.32:0/1824402147 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f8958077010 0x7f8958079520 secure :-1 s=READY pgs=782 cs=0 l=1 rev1=1 crypto rx=0x7f8968009b90 tx=0x7f8968005f30 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:59:37.647 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:37.646+0000 7f898479c740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ca5578e270 0x55ca55b7b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:59:37.647 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:37.646+0000 7f898479c740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55ca557653b0 0x55ca55b5dda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:59:37.647 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:37.646+0000 7f898479c740 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55ca5576d590 0x55ca55b5e2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:59:37.647 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:37.646+0000 7f898479c740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55ca55aa6a60 con 0x55ca5578e270 2024-08-03T20:59:37.647 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:37.646+0000 7f898479c740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x55ca55a95f60 con 0x55ca557653b0 2024-08-03T20:59:37.647 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:37.646+0000 7f898479c740 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x55ca55a99fa0 con 0x55ca5576d590 2024-08-03T20:59:37.647 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:37.647+0000 7f8976a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ca5578e270 0x55ca55b7b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:59:37.647 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:37.647+0000 7f8976a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ca5578e270 0x55ca55b7b0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:55386/0 (socket says 172.21.15.32:55386) 2024-08-03T20:59:37.647 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:37.647+0000 7f8976a00640 1 -- 172.21.15.32:0/2835416567 learned_addr learned my addr 172.21.15.32:0/2835416567 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:59:37.649 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:37.647+0000 7f8976000640 1 --2- 172.21.15.32:0/2835416567 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55ca557653b0 0x55ca55b5dda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:59:37.649 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:37.647+0000 7f8977400640 1 --2- 172.21.15.32:0/2835416567 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55ca5576d590 0x55ca55b5e2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:59:37.649 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:37.647+0000 7f8976a00640 1 -- 172.21.15.32:0/2835416567 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55ca5576d590 msgr2=0x55ca55b5e2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:59:37.649 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:37.647+0000 7f8976a00640 1 --2- 172.21.15.32:0/2835416567 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55ca5576d590 0x55ca55b5e2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:59:37.650 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:37.647+0000 7f8976a00640 1 -- 172.21.15.32:0/2835416567 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55ca557653b0 msgr2=0x55ca55b5dda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:59:37.650 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:37.647+0000 7f8976a00640 1 --2- 172.21.15.32:0/2835416567 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55ca557653b0 0x55ca55b5dda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:59:37.650 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:37.647+0000 7f8976a00640 1 -- 172.21.15.32:0/2835416567 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ca557f9850 con 0x55ca5578e270 2024-08-03T20:59:37.650 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:37.647+0000 7f8976000640 1 --2- 172.21.15.32:0/2835416567 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55ca557653b0 0x55ca55b5dda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-03T20:59:37.650 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:37.647+0000 7f8976a00640 1 --2- 172.21.15.32:0/2835416567 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ca5578e270 0x55ca55b7b0d0 secure :-1 s=READY pgs=1950 cs=0 l=1 rev1=1 crypto rx=0x7f8968012010 tx=0x7f8968014000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=74f450b4d3edb388 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:59:37.650 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:37.647+0000 7f8963000640 1 -- 172.21.15.32:0/2835416567 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f896801f620 con 0x55ca5578e270 2024-08-03T20:59:37.650 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:37.648+0000 7f8963000640 1 -- 172.21.15.32:0/2835416567 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f896801f8b0 con 0x55ca5578e270 2024-08-03T20:59:37.650 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:37.648+0000 7f8963000640 1 -- 172.21.15.32:0/2835416567 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f896801fce0 con 0x55ca5578e270 2024-08-03T20:59:37.650 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:37.648+0000 7f898479c740 1 -- 172.21.15.32:0/2835416567 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ca5578e270 msgr2=0x55ca55b7b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:59:37.650 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:37.648+0000 7f898479c740 1 --2- 172.21.15.32:0/2835416567 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ca5578e270 0x55ca55b7b0d0 secure :-1 s=READY pgs=1950 cs=0 l=1 rev1=1 crypto rx=0x7f8968012010 tx=0x7f8968014000 comp rx=0 tx=0).stop 2024-08-03T20:59:37.650 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:37.648+0000 7f898479c740 1 -- 172.21.15.32:0/2835416567 shutdown_connections 2024-08-03T20:59:37.650 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:37.648+0000 7f898479c740 1 --2- 172.21.15.32:0/2835416567 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55ca557653b0 0x55ca55b5dda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:59:37.650 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:37.648+0000 7f898479c740 1 --2- 172.21.15.32:0/2835416567 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55ca5576d590 0x55ca55b5e2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:59:37.651 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:37.648+0000 7f898479c740 1 --2- 172.21.15.32:0/2835416567 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ca5578e270 0x55ca55b7b0d0 unknown :-1 s=CLOSED pgs=1950 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:59:37.651 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:37.648+0000 7f898479c740 1 -- 172.21.15.32:0/2835416567 >> 172.21.15.32:0/2835416567 conn(0x55ca557823c0 msgr2=0x55ca55b33fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:59:37.651 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:37.648+0000 7f898479c740 1 -- 172.21.15.32:0/2835416567 shutdown_connections 2024-08-03T20:59:37.651 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:37.649+0000 7f898479c740 1 -- 172.21.15.32:0/2835416567 wait complete. 2024-08-03T20:59:37.652 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:37.649+0000 7f898479c740 1 -- start start 2024-08-03T20:59:37.652 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:37.650+0000 7f898479c740 1 Processor -- start 2024-08-03T20:59:37.652 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:37.651+0000 7f898479c740 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55ca557653b0 0x55ca55b88150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:59:37.652 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:37.651+0000 7f898479c740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ca5576d590 0x55ca55b598f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:59:37.653 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:37.651+0000 7f898479c740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55ca5578e270 0x55ca55b562c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:59:37.653 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:37.651+0000 7f898479c740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55ca55808620 con 0x55ca5576d590 2024-08-03T20:59:37.653 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:37.651+0000 7f898479c740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x55ca55a33160 con 0x55ca5578e270 2024-08-03T20:59:37.653 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:37.651+0000 7f898479c740 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x55ca55b5a180 con 0x55ca557653b0 2024-08-03T20:59:37.653 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:37.651+0000 7f898479c740 10 client.0 _ll_register_callbacks cb 0x55ca55a426b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-03T20:59:37.653 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:37.651+0000 7f8976000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ca5576d590 0x55ca55b598f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:59:37.653 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:37.652+0000 7f8976a00640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55ca557653b0 0x55ca55b88150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:59:37.653 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:37.652+0000 7f8977400640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55ca5578e270 0x55ca55b562c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:59:37.653 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:37.652+0000 7f8976000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ca5576d590 0x55ca55b598f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:55400/0 (socket says 172.21.15.32:55400) 2024-08-03T20:59:37.653 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:37.652+0000 7f8976000640 1 -- 172.21.15.32:0/2381810184 learned_addr learned my addr 172.21.15.32:0/2381810184 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:59:37.653 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:37.652+0000 7f8976a00640 1 --2- 172.21.15.32:0/2381810184 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55ca557653b0 0x55ca55b88150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3301/0 says I am v2:172.21.15.32:56866/0 (socket says 172.21.15.32:56866) 2024-08-03T20:59:37.653 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:37.652+0000 7f8976000640 1 -- 172.21.15.32:0/2381810184 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55ca557653b0 msgr2=0x55ca55b88150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:59:37.653 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:37.652+0000 7f8976000640 1 --2- 172.21.15.32:0/2381810184 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55ca557653b0 0x55ca55b88150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:59:37.654 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:37.652+0000 7f8976000640 1 -- 172.21.15.32:0/2381810184 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55ca5578e270 msgr2=0x55ca55b562c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:59:37.654 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:37.652+0000 7f8976000640 1 --2- 172.21.15.32:0/2381810184 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55ca5578e270 0x55ca55b562c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:59:37.654 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:37.652+0000 7f8976000640 1 -- 172.21.15.32:0/2381810184 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ca558706b0 con 0x55ca5576d590 2024-08-03T20:59:37.656 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:37.652+0000 7f8976000640 1 --2- 172.21.15.32:0/2381810184 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ca5576d590 0x55ca55b598f0 secure :-1 s=READY pgs=1951 cs=0 l=1 rev1=1 crypto rx=0x7f896405d6e0 tx=0x7f8964060950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a3d19c261d814a36 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:59:37.656 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:37.652+0000 7f8957400640 10 client.0 ms_handle_connect on v2:172.21.15.32:3300/0 2024-08-03T20:59:37.656 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:37.653+0000 7f8957400640 1 -- 172.21.15.32:0/2381810184 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89640610f0 con 0x55ca5576d590 2024-08-03T20:59:37.657 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:37.653+0000 7f8957400640 1 -- 172.21.15.32:0/2381810184 <== mon.0 v2:172.21.15.32:3300/0 2 ==== osd_map(119..119 src has 1..119) ==== 8778+0+0 (secure 0 0 0) 0x7f8964060560 con 0x55ca5576d590 2024-08-03T20:59:37.657 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:37.653+0000 7f898479c740 1 -- 172.21.15.32:0/2381810184 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f896801f8b0 con 0x55ca5576d590 2024-08-03T20:59:37.657 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:37.653+0000 7f8957400640 1 -- 172.21.15.32:0/2381810184 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_get_version_reply(handle=1 version=293) ==== 24+0+0 (secure 0 0 0) 0x7f89640656f0 con 0x55ca5576d590 2024-08-03T20:59:37.657 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:37.653+0000 7f898479c740 10 client.9942 fetch_fsmap learned FSMap version 293 2024-08-03T20:59:37.657 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:37.653+0000 7f898479c740 1 -- 172.21.15.32:0/2381810184 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,fsmap=293,monmap=2+}) -- 0x55ca55b568b0 con 0x55ca5576d590 2024-08-03T20:59:37.657 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:37.653+0000 7f8957400640 1 -- 172.21.15.32:0/2381810184 <== mon.0 v2:172.21.15.32:3300/0 4 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f896801f8b0 con 0x55ca5576d590 2024-08-03T20:59:37.657 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:37.653+0000 7f8957400640 1 -- 172.21.15.32:0/2381810184 <== mon.0 v2:172.21.15.32:3300/0 5 ==== fsmap(e 293) ==== 3502+0+0 (secure 0 0 0) 0x7f8964065170 con 0x55ca5576d590 2024-08-03T20:59:37.657 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:37.653+0000 7f898479c740 10 client.9942 fetch_fsmap finished waiting for FSMap version 293 2024-08-03T20:59:37.657 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:37.653+0000 7f898479c740 10 client.9942 resolve_mds: resolved name 'a' to daemon mds.a 2024-08-03T20:59:37.657 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:37.653+0000 7f898479c740 20 client.9942 populate_metadata read hostname 'smithi032' 2024-08-03T20:59:37.657 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:37.653+0000 7f898479c740 1 --2- 172.21.15.32:0/2381810184 >> [v2:172.21.15.32:6833/2106229503,v1:172.21.15.32:6835/2106229503] conn(0x55ca5574ce40 0x55ca55b71e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:59:37.657 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:37.653+0000 7f898479c740 4 client.9942 mds_command: new command op to 9882 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-08-03T20:59:37.657 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:37.653+0000 7f898479c740 1 -- 172.21.15.32:0/2381810184 --> [v2:172.21.15.32:6833/2106229503,v1:172.21.15.32:6835/2106229503] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ca55aa89b0 con 0x55ca5574ce40 2024-08-03T20:59:37.658 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:37.654+0000 7f8976a00640 1 --2- 172.21.15.32:0/2381810184 >> [v2:172.21.15.32:6833/2106229503,v1:172.21.15.32:6835/2106229503] conn(0x55ca5574ce40 0x55ca55b71e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:59:37.658 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:37.654+0000 7f8976a00640 1 --2- 172.21.15.32:0/2381810184 >> [v2:172.21.15.32:6833/2106229503,v1:172.21.15.32:6835/2106229503] conn(0x55ca5574ce40 0x55ca55b71e60 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7031560e2d4dc55f server_cookie=ecedfbf344bbf268 in_seq=0 out_seq=0 2024-08-03T20:59:37.658 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:37.654+0000 7f8957400640 10 client.9942 ms_handle_connect on v2:172.21.15.32:6833/2106229503 2024-08-03T20:59:37.658 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:37.655+0000 7f8957400640 1 -- 172.21.15.32:0/2381810184 <== mds.0 v2:172.21.15.32:6833/2106229503 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55ca55aa89b0 con 0x55ca5574ce40 2024-08-03T20:59:37.658 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:37.655+0000 7f8957400640 10 client.9942 handle_command_reply: tid=0 2024-08-03T20:59:37.658 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:37.655+0000 7f898479c740 1 client.9942 shutdown 2024-08-03T20:59:37.658 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:37.655+0000 7f898479c740 1 -- 172.21.15.32:0/2381810184 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ca5576d590 msgr2=0x55ca55b598f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:59:37.658 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:37.655+0000 7f898479c740 1 --2- 172.21.15.32:0/2381810184 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ca5576d590 0x55ca55b598f0 secure :-1 s=READY pgs=1951 cs=0 l=1 rev1=1 crypto rx=0x7f896405d6e0 tx=0x7f8964060950 comp rx=0 tx=0).stop 2024-08-03T20:59:37.658 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:37.655+0000 7f898479c740 1 -- 172.21.15.32:0/2381810184 shutdown_connections 2024-08-03T20:59:37.658 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:37.655+0000 7f898479c740 1 --2- 172.21.15.32:0/2381810184 >> [v2:172.21.15.32:6833/2106229503,v1:172.21.15.32:6835/2106229503] conn(0x55ca5574ce40 0x55ca55b71e60 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:59:37.658 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:37.655+0000 7f898479c740 1 --2- 172.21.15.32:0/2381810184 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55ca5578e270 0x55ca55b562c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:59:37.658 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:37.655+0000 7f898479c740 1 --2- 172.21.15.32:0/2381810184 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ca5576d590 0x55ca55b598f0 unknown :-1 s=CLOSED pgs=1951 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:59:37.658 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:37.655+0000 7f898479c740 1 --2- 172.21.15.32:0/2381810184 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55ca557653b0 0x55ca55b88150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:59:37.659 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:37.655+0000 7f8957400640 0 client.9942 ms_handle_reset on v2:172.21.15.32:6833/2106229503 2024-08-03T20:59:37.659 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:37.655+0000 7f898479c740 1 -- 172.21.15.32:0/2381810184 >> 172.21.15.32:0/2381810184 conn(0x55ca557823c0 msgr2=0x55ca55b4d560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:59:37.659 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:37.656+0000 7f898479c740 1 -- 172.21.15.32:0/2381810184 shutdown_connections 2024-08-03T20:59:37.659 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:37.656+0000 7f898479c740 1 -- 172.21.15.32:0/2381810184 wait complete. 2024-08-03T20:59:37.659 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:37.656+0000 7f898479c740 20 client.9942 trim_cache size 0 max 16384 2024-08-03T20:59:37.676 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:37.676+0000 7f898479c740 1 Processor -- start 2024-08-03T20:59:37.676 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:37.676+0000 7f898479c740 1 -- start start 2024-08-03T20:59:37.679 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:37.677+0000 7f898479c740 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55ca557653b0 0x55ca55b4d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:59:37.679 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:37.677+0000 7f898479c740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ca5576d590 0x55ca55b4d630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:59:37.679 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:37.677+0000 7f898479c740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55ca5578e270 0x55ca55b5dda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:59:37.679 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:37.677+0000 7f898479c740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f8978202b90 con 0x55ca5576d590 2024-08-03T20:59:37.679 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:37.677+0000 7f898479c740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f8978153880 con 0x55ca5578e270 2024-08-03T20:59:37.679 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:37.677+0000 7f898479c740 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7f8978202540 con 0x55ca557653b0 2024-08-03T20:59:37.679 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:37.678+0000 7f8976000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ca5576d590 0x55ca55b4d630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:59:37.679 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:37.678+0000 7f8976000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ca5576d590 0x55ca55b4d630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:55414/0 (socket says 172.21.15.32:55414) 2024-08-03T20:59:37.679 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:37.678+0000 7f8976000640 1 -- 172.21.15.32:0/476559716 learned_addr learned my addr 172.21.15.32:0/476559716 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:59:37.679 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:37.678+0000 7f8976a00640 1 --2- 172.21.15.32:0/476559716 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55ca557653b0 0x55ca55b4d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:59:37.679 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:37.678+0000 7f8977400640 1 --2- 172.21.15.32:0/476559716 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55ca5578e270 0x55ca55b5dda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:59:37.679 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:37.678+0000 7f8976000640 1 -- 172.21.15.32:0/476559716 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55ca557653b0 msgr2=0x55ca55b4d0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:59:37.680 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:37.678+0000 7f8976000640 1 --2- 172.21.15.32:0/476559716 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55ca557653b0 0x55ca55b4d0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:59:37.680 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:37.678+0000 7f8976000640 1 -- 172.21.15.32:0/476559716 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55ca5578e270 msgr2=0x55ca55b5dda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:59:37.680 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:37.678+0000 7f8976000640 1 --2- 172.21.15.32:0/476559716 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55ca5578e270 0x55ca55b5dda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:59:37.680 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:37.678+0000 7f8976000640 1 -- 172.21.15.32:0/476559716 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ca55b347e0 con 0x55ca5576d590 2024-08-03T20:59:37.680 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:37.678+0000 7f8976000640 1 --2- 172.21.15.32:0/476559716 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ca5576d590 0x55ca55b4d630 secure :-1 s=READY pgs=1952 cs=0 l=1 rev1=1 crypto rx=0x7f8964007fd0 tx=0x7f89640969a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=653e62613061c322 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:59:37.681 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:37.679+0000 7f8962600640 1 -- 172.21.15.32:0/476559716 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89640610f0 con 0x55ca5576d590 2024-08-03T20:59:37.681 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:37.679+0000 7f8962600640 1 -- 172.21.15.32:0/476559716 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f896406a1a0 con 0x55ca5576d590 2024-08-03T20:59:37.681 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:37.679+0000 7f8962600640 1 -- 172.21.15.32:0/476559716 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8964065170 con 0x55ca5576d590 2024-08-03T20:59:37.681 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:37.679+0000 7f898479c740 1 -- 172.21.15.32:0/476559716 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ca5576d590 msgr2=0x55ca55b4d630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:59:37.681 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:37.679+0000 7f898479c740 1 --2- 172.21.15.32:0/476559716 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ca5576d590 0x55ca55b4d630 secure :-1 s=READY pgs=1952 cs=0 l=1 rev1=1 crypto rx=0x7f8964007fd0 tx=0x7f89640969a0 comp rx=0 tx=0).stop 2024-08-03T20:59:37.681 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:37.679+0000 7f898479c740 1 -- 172.21.15.32:0/476559716 shutdown_connections 2024-08-03T20:59:37.681 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:37.679+0000 7f898479c740 1 --2- 172.21.15.32:0/476559716 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55ca5578e270 0x55ca55b5dda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:59:37.682 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:37.679+0000 7f898479c740 1 --2- 172.21.15.32:0/476559716 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ca5576d590 0x55ca55b4d630 unknown :-1 s=CLOSED pgs=1952 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:59:37.682 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:37.679+0000 7f898479c740 1 --2- 172.21.15.32:0/476559716 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55ca557653b0 0x55ca55b4d0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:59:37.682 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:37.679+0000 7f898479c740 1 -- 172.21.15.32:0/476559716 >> 172.21.15.32:0/476559716 conn(0x55ca557823c0 msgr2=0x55ca55b335f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:59:37.682 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:37.680+0000 7f898479c740 1 -- 172.21.15.32:0/476559716 shutdown_connections 2024-08-03T20:59:37.682 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:37.680+0000 7f898479c740 1 -- 172.21.15.32:0/476559716 wait complete. 2024-08-03T20:59:37.682 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:37.680+0000 7f898479c740 1 -- start start 2024-08-03T20:59:37.682 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:37.681+0000 7f898479c740 1 Processor -- start 2024-08-03T20:59:37.686 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:37.682+0000 7f898479c740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ca557653b0 0x55ca55b4c310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:59:37.686 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:37.682+0000 7f898479c740 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55ca5576d590 0x55ca55bf58c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:59:37.686 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:37.682+0000 7f898479c740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55ca5578e270 0x55ca55b9bc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:59:37.686 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:37.682+0000 7f898479c740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55ca55b57f10 con 0x55ca557653b0 2024-08-03T20:59:37.686 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:37.682+0000 7f898479c740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x55ca55b57a20 con 0x55ca5578e270 2024-08-03T20:59:37.687 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:37.682+0000 7f898479c740 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x55ca55b5a760 con 0x55ca5576d590 2024-08-03T20:59:37.687 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:37.682+0000 7f898479c740 10 client.0 _ll_register_callbacks cb 0x7f8964060ad0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-03T20:59:37.687 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:37.682+0000 7f8976a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ca557653b0 0x55ca55b4c310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:59:37.687 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:37.682+0000 7f8976a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ca557653b0 0x55ca55b4c310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:55422/0 (socket says 172.21.15.32:55422) 2024-08-03T20:59:37.687 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:37.682+0000 7f8976a00640 1 -- 172.21.15.32:0/471848548 learned_addr learned my addr 172.21.15.32:0/471848548 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:59:37.687 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:37.683+0000 7f8977400640 1 --2- 172.21.15.32:0/471848548 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55ca5578e270 0x55ca55b9bc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:59:37.687 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:37.683+0000 7f8976000640 1 --2- 172.21.15.32:0/471848548 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55ca5576d590 0x55ca55bf58c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:59:37.687 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:37.683+0000 7f8976a00640 1 -- 172.21.15.32:0/471848548 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55ca5576d590 msgr2=0x55ca55bf58c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:59:37.687 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:37.683+0000 7f8976a00640 1 --2- 172.21.15.32:0/471848548 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55ca5576d590 0x55ca55bf58c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:59:37.687 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:37.683+0000 7f8976a00640 1 -- 172.21.15.32:0/471848548 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55ca5578e270 msgr2=0x55ca55b9bc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:59:37.687 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:37.683+0000 7f8976a00640 1 --2- 172.21.15.32:0/471848548 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55ca5578e270 0x55ca55b9bc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:59:37.687 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:37.683+0000 7f8976a00640 1 -- 172.21.15.32:0/471848548 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ca55b54c00 con 0x55ca557653b0 2024-08-03T20:59:37.687 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:37.683+0000 7f8976a00640 1 --2- 172.21.15.32:0/471848548 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ca557653b0 0x55ca55b4c310 secure :-1 s=READY pgs=1953 cs=0 l=1 rev1=1 crypto rx=0x7f8968011fb0 tx=0x7f89680046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b4965ab623e42109 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:59:37.688 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:37.683+0000 7f8957400640 10 client.0 ms_handle_connect on v2:172.21.15.32:3300/0 2024-08-03T20:59:37.688 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:37.683+0000 7f8957400640 1 -- 172.21.15.32:0/471848548 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f896801fcc0 con 0x55ca557653b0 2024-08-03T20:59:37.688 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:37.683+0000 7f8957400640 1 -- 172.21.15.32:0/471848548 <== mon.0 v2:172.21.15.32:3300/0 2 ==== osd_map(119..119 src has 1..119) ==== 8778+0+0 (secure 0 0 0) 0x7f8968005030 con 0x55ca557653b0 2024-08-03T20:59:37.688 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:37.683+0000 7f898479c740 1 -- 172.21.15.32:0/471848548 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f896406a1a0 con 0x55ca557653b0 2024-08-03T20:59:37.688 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:37.684+0000 7f8957400640 1 -- 172.21.15.32:0/471848548 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_get_version_reply(handle=1 version=293) ==== 24+0+0 (secure 0 0 0) 0x7f896801fe60 con 0x55ca557653b0 2024-08-03T20:59:37.688 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:37.684+0000 7f898479c740 10 client.9948 fetch_fsmap learned FSMap version 293 2024-08-03T20:59:37.688 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:37.684+0000 7f898479c740 1 -- 172.21.15.32:0/471848548 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,fsmap=293,monmap=2+}) -- 0x55ca55b9c720 con 0x55ca557653b0 2024-08-03T20:59:37.688 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:37.684+0000 7f8957400640 1 -- 172.21.15.32:0/471848548 <== mon.0 v2:172.21.15.32:3300/0 4 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f896406a1a0 con 0x55ca557653b0 2024-08-03T20:59:37.688 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:37.684+0000 7f8957400640 1 -- 172.21.15.32:0/471848548 <== mon.0 v2:172.21.15.32:3300/0 5 ==== fsmap(e 293) ==== 3502+0+0 (secure 0 0 0) 0x7f8968005230 con 0x55ca557653b0 2024-08-03T20:59:37.688 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:37.684+0000 7f898479c740 10 client.9948 fetch_fsmap finished waiting for FSMap version 293 2024-08-03T20:59:37.688 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:37.684+0000 7f898479c740 10 client.9948 resolve_mds: resolved name 'a' to daemon mds.a 2024-08-03T20:59:37.688 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:37.684+0000 7f898479c740 20 client.9948 populate_metadata read hostname 'smithi032' 2024-08-03T20:59:37.688 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:37.684+0000 7f898479c740 1 --2- 172.21.15.32:0/471848548 >> [v2:172.21.15.32:6833/2106229503,v1:172.21.15.32:6835/2106229503] conn(0x55ca55b54580 0x55ca55bf96c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:59:37.688 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:37.684+0000 7f898479c740 4 client.9948 mds_command: new command op to 9882 tid=0 multi_id=0 [{"prefix": "status"}] 2024-08-03T20:59:37.689 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:37.684+0000 7f898479c740 1 -- 172.21.15.32:0/471848548 --> [v2:172.21.15.32:6833/2106229503,v1:172.21.15.32:6835/2106229503] -- command(tid 0: {"prefix": "status"}) -- 0x55ca55b2ff20 con 0x55ca55b54580 2024-08-03T20:59:37.689 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:37.684+0000 7f8976000640 1 --2- 172.21.15.32:0/471848548 >> [v2:172.21.15.32:6833/2106229503,v1:172.21.15.32:6835/2106229503] conn(0x55ca55b54580 0x55ca55bf96c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:59:37.689 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:37.685+0000 7f8976000640 1 --2- 172.21.15.32:0/471848548 >> [v2:172.21.15.32:6833/2106229503,v1:172.21.15.32:6835/2106229503] conn(0x55ca55b54580 0x55ca55bf96c0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7aff0e50920f1b2d server_cookie=3caf84ce5bb2d0bd in_seq=0 out_seq=0 2024-08-03T20:59:37.689 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:37.685+0000 7f8957400640 10 client.9948 ms_handle_connect on v2:172.21.15.32:6833/2106229503 2024-08-03T20:59:37.689 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:37.685+0000 7f8957400640 1 -- 172.21.15.32:0/471848548 <== mds.0 v2:172.21.15.32:6833/2106229503 1 ==== command_reply(tid 0: 0 ) ==== 8+0+334 (crc 0 0 0) 0x55ca55b2ff20 con 0x55ca55b54580 2024-08-03T20:59:37.689 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:37.685+0000 7f8957400640 10 client.9948 handle_command_reply: tid=0 2024-08-03T20:59:37.689 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:37.685+0000 7f898479c740 1 client.9948 shutdown 2024-08-03T20:59:37.689 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:37.685+0000 7f898479c740 1 -- 172.21.15.32:0/471848548 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ca557653b0 msgr2=0x55ca55b4c310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:59:37.689 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:37.685+0000 7f898479c740 1 --2- 172.21.15.32:0/471848548 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ca557653b0 0x55ca55b4c310 secure :-1 s=READY pgs=1953 cs=0 l=1 rev1=1 crypto rx=0x7f8968011fb0 tx=0x7f89680046c0 comp rx=0 tx=0).stop 2024-08-03T20:59:37.689 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:37.685+0000 7f898479c740 1 -- 172.21.15.32:0/471848548 shutdown_connections 2024-08-03T20:59:37.689 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:37.685+0000 7f898479c740 1 --2- 172.21.15.32:0/471848548 >> [v2:172.21.15.32:6833/2106229503,v1:172.21.15.32:6835/2106229503] conn(0x55ca55b54580 0x55ca55bf96c0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:59:37.689 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:37.685+0000 7f898479c740 1 --2- 172.21.15.32:0/471848548 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55ca5578e270 0x55ca55b9bc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:59:37.689 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:37.685+0000 7f898479c740 1 --2- 172.21.15.32:0/471848548 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55ca5576d590 0x55ca55bf58c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:59:37.690 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:37.685+0000 7f898479c740 1 --2- 172.21.15.32:0/471848548 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ca557653b0 0x55ca55b4c310 unknown :-1 s=CLOSED pgs=1953 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:59:37.690 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:37.685+0000 7f8957400640 0 client.9948 ms_handle_reset on v2:172.21.15.32:6833/2106229503 2024-08-03T20:59:37.690 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:37.685+0000 7f898479c740 1 -- 172.21.15.32:0/471848548 >> 172.21.15.32:0/471848548 conn(0x55ca557823c0 msgr2=0x55ca55b4cf60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:59:37.690 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:37.685+0000 7f898479c740 1 -- 172.21.15.32:0/471848548 shutdown_connections 2024-08-03T20:59:37.690 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:37.685+0000 7f898479c740 1 -- 172.21.15.32:0/471848548 wait complete. 2024-08-03T20:59:37.690 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:37.686+0000 7f898479c740 20 client.9948 trim_cache size 0 max 16384 2024-08-03T20:59:37.690 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-08-03T20:59:37.690 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_fsid": "40ab141a-01d7-49b9-ae3c-2edec9876e95", 2024-08-03T20:59:37.690 INFO:teuthology.orchestra.run.smithi032.stdout: "whoami": 0, 2024-08-03T20:59:37.690 INFO:teuthology.orchestra.run.smithi032.stdout: "id": 9882, 2024-08-03T20:59:37.691 INFO:teuthology.orchestra.run.smithi032.stdout: "want_state": "up:active", 2024-08-03T20:59:37.691 INFO:teuthology.orchestra.run.smithi032.stdout: "state": "up:active", 2024-08-03T20:59:37.691 INFO:teuthology.orchestra.run.smithi032.stdout: "fs_name": "cephfs", 2024-08-03T20:59:37.691 INFO:teuthology.orchestra.run.smithi032.stdout: "rank_uptime": 7.1577160900000001, 2024-08-03T20:59:37.691 INFO:teuthology.orchestra.run.smithi032.stdout: "mdsmap_epoch": 293, 2024-08-03T20:59:37.691 INFO:teuthology.orchestra.run.smithi032.stdout: "osdmap_epoch": 119, 2024-08-03T20:59:37.691 INFO:teuthology.orchestra.run.smithi032.stdout: "osdmap_epoch_barrier": 119, 2024-08-03T20:59:37.691 INFO:teuthology.orchestra.run.smithi032.stdout: "uptime": 9.3596232159999992 2024-08-03T20:59:37.691 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-08-03T20:59:37.692 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:37.686+0000 7f897d600640 1 -- 172.21.15.32:0/1824402147 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f8958077010 msgr2=0x7f8958079520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:59:37.692 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:37.686+0000 7f897d600640 1 --2- 172.21.15.32:0/1824402147 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f8958077010 0x7f8958079520 secure :-1 s=READY pgs=782 cs=0 l=1 rev1=1 crypto rx=0x7f8968009b90 tx=0x7f8968005f30 comp rx=0 tx=0).stop 2024-08-03T20:59:37.692 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:37.686+0000 7f897d600640 1 -- 172.21.15.32:0/1824402147 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f89780d0350 msgr2=0x7f89781f8e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:59:37.692 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:37.686+0000 7f897d600640 1 --2- 172.21.15.32:0/1824402147 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f89780d0350 0x7f89781f8e30 secure :-1 s=READY pgs=1949 cs=0 l=1 rev1=1 crypto rx=0x7f896400c430 tx=0x7f896400c8f0 comp rx=0 tx=0).stop 2024-08-03T20:59:37.692 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:37.686+0000 7f897d600640 1 -- 172.21.15.32:0/1824402147 shutdown_connections 2024-08-03T20:59:37.692 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:37.686+0000 7f897d600640 1 --2- 172.21.15.32:0/1824402147 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f8958077010 0x7f8958079520 unknown :-1 s=CLOSED pgs=782 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:59:37.692 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:37.686+0000 7f897d600640 1 --2- 172.21.15.32:0/1824402147 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f89780d1c30 0x7f89781f9370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:59:37.692 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:37.686+0000 7f897d600640 1 --2- 172.21.15.32:0/1824402147 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f89780d0350 0x7f89781f8e30 unknown :-1 s=CLOSED pgs=1949 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:59:37.692 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:37.686+0000 7f897d600640 1 --2- 172.21.15.32:0/1824402147 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f89780b78d0 0x7f89780c9150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:59:37.693 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:37.686+0000 7f897d600640 1 -- 172.21.15.32:0/1824402147 >> 172.21.15.32:0/1824402147 conn(0x7f89780b3bb0 msgr2=0x7f89780cce20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:59:37.693 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:37.686+0000 7f897d600640 1 -- 172.21.15.32:0/1824402147 shutdown_connections 2024-08-03T20:59:37.693 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:37.686+0000 7f897d600640 1 -- 172.21.15.32:0/1824402147 wait complete. 2024-08-03T20:59:37.713 DEBUG:tasks.ceph_test_case:wait_until_true: success in 5s and 0 retries 2024-08-03T20:59:37.713 INFO:tasks.cephfs.test_damage:Daemons came up after mutation 'Corrupt omap header on 10000000000.00000000', proceeding to ls 2024-08-03T20:59:37.713 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-08-03T20:59:37.943 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:37.942+0000 7f24a9200640 1 -- 172.21.15.32:0/4162911321 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2488004800 msgr2=0x7f2488004c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:59:37.943 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:37.942+0000 7f24a9200640 1 --2- 172.21.15.32:0/4162911321 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2488004800 0x7f2488004c10 secure :-1 s=READY pgs=1955 cs=0 l=1 rev1=1 crypto rx=0x7f248c006fd0 tx=0x7f2488005830 comp rx=0 tx=0).stop 2024-08-03T20:59:37.943 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:37.942+0000 7f24a9200640 1 -- 172.21.15.32:0/4162911321 shutdown_connections 2024-08-03T20:59:37.943 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:37.942+0000 7f24a9200640 1 --2- 172.21.15.32:0/4162911321 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2488004800 0x7f2488004c10 unknown :-1 s=CLOSED pgs=1955 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:59:37.943 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:37.942+0000 7f24a9200640 1 -- 172.21.15.32:0/4162911321 >> 172.21.15.32:0/4162911321 conn(0x7f24a408d7a0 msgr2=0x7f24a409d3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:59:37.944 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:37.942+0000 7f24a9200640 1 -- 172.21.15.32:0/4162911321 shutdown_connections 2024-08-03T20:59:37.944 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:37.942+0000 7f24a9200640 1 -- 172.21.15.32:0/4162911321 wait complete. 2024-08-03T20:59:37.945 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:37.944+0000 7f24a9200640 1 Processor -- start 2024-08-03T20:59:37.945 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:37.945+0000 7f24a9200640 1 -- start start 2024-08-03T20:59:37.947 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:37.946+0000 7f24a9200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2488004800 0x7f24a415dc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:59:37.947 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:37.946+0000 7f24a9200640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f24a415e450 0x7f24a41e8a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:59:37.947 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:37.946+0000 7f24a9200640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f24a40cd8e0 0x7f24a41f8f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:59:37.948 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:37.946+0000 7f24a9200640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f24a40cf920 con 0x7f2488004800 2024-08-03T20:59:37.948 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:37.946+0000 7f24a9200640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f24a4161ef0 con 0x7f24a415e450 2024-08-03T20:59:37.948 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:37.946+0000 7f24a9200640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7f24a40cfaa0 con 0x7f24a40cd8e0 2024-08-03T20:59:37.948 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:37.946+0000 7f24a3400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2488004800 0x7f24a415dc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:59:37.948 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:37.946+0000 7f24a3400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2488004800 0x7f24a415dc70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:55448/0 (socket says 172.21.15.32:55448) 2024-08-03T20:59:37.948 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:37.946+0000 7f24a3400640 1 -- 172.21.15.32:0/1946248391 learned_addr learned my addr 172.21.15.32:0/1946248391 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:59:37.948 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:37.947+0000 7f24a3e00640 1 --2- 172.21.15.32:0/1946248391 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f24a40cd8e0 0x7f24a41f8f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:59:37.948 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:37.947+0000 7f249be00640 1 --2- 172.21.15.32:0/1946248391 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f24a415e450 0x7f24a41e8a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:59:37.948 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:37.947+0000 7f24a3400640 1 -- 172.21.15.32:0/1946248391 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f24a40cd8e0 msgr2=0x7f24a41f8f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:59:37.948 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:37.947+0000 7f24a3400640 1 --2- 172.21.15.32:0/1946248391 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f24a40cd8e0 0x7f24a41f8f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:59:37.948 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:37.947+0000 7f24a3400640 1 -- 172.21.15.32:0/1946248391 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f24a415e450 msgr2=0x7f24a41e8a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:59:37.948 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:37.947+0000 7f24a3400640 1 --2- 172.21.15.32:0/1946248391 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f24a415e450 0x7f24a41e8a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:59:37.948 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:37.947+0000 7f24a3400640 1 -- 172.21.15.32:0/1946248391 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f24a40cbcb0 con 0x7f2488004800 2024-08-03T20:59:37.948 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:37.947+0000 7f24a3e00640 1 --2- 172.21.15.32:0/1946248391 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f24a40cd8e0 0x7f24a41f8f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-03T20:59:37.949 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:37.947+0000 7f249be00640 1 --2- 172.21.15.32:0/1946248391 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f24a415e450 0x7f24a41e8a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-03T20:59:37.949 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:37.947+0000 7f24a3400640 1 --2- 172.21.15.32:0/1946248391 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2488004800 0x7f24a415dc70 secure :-1 s=READY pgs=1956 cs=0 l=1 rev1=1 crypto rx=0x7f2494011a10 tx=0x7f2494011ed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:59:37.950 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:37.947+0000 7f249b400640 1 -- 172.21.15.32:0/1946248391 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f249401e070 con 0x7f2488004800 2024-08-03T20:59:37.950 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:37.948+0000 7f249b400640 1 -- 172.21.15.32:0/1946248391 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f2494004460 con 0x7f2488004800 2024-08-03T20:59:37.950 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:37.948+0000 7f249b400640 1 -- 172.21.15.32:0/1946248391 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f249400fc00 con 0x7f2488004800 2024-08-03T20:59:37.950 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:37.948+0000 7f24a9200640 1 -- 172.21.15.32:0/1946248391 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f24a40cbf90 con 0x7f2488004800 2024-08-03T20:59:37.950 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:37.948+0000 7f24a9200640 1 -- 172.21.15.32:0/1946248391 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f24a40cb630 con 0x7f2488004800 2024-08-03T20:59:37.950 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:37.949+0000 7f24a9200640 1 -- 172.21.15.32:0/1946248391 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f24a40cc1d0 con 0x7f2488004800 2024-08-03T20:59:37.950 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:37.950+0000 7f249b400640 1 -- 172.21.15.32:0/1946248391 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f2494021070 con 0x7f2488004800 2024-08-03T20:59:37.951 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:37.951+0000 7f249b400640 1 --2- 172.21.15.32:0/1946248391 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f2480077010 0x7f2480079520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:59:37.952 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:37.951+0000 7f249b400640 1 -- 172.21.15.32:0/1946248391 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(119..119 src has 1..119) ==== 8778+0+0 (secure 0 0 0) 0x7f2494099ee0 con 0x7f2488004800 2024-08-03T20:59:37.952 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:37.952+0000 7f249be00640 1 --2- 172.21.15.32:0/1946248391 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f2480077010 0x7f2480079520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:59:37.952 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:37.952+0000 7f249be00640 1 --2- 172.21.15.32:0/1946248391 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f2480077010 0x7f2480079520 secure :-1 s=READY pgs=783 cs=0 l=1 rev1=1 crypto rx=0x7f248c033000 tx=0x7f24a40d2e20 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:59:37.956 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:37.956+0000 7f249b400640 1 -- 172.21.15.32:0/1946248391 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7f24a40cc1d0 con 0x7f2488004800 2024-08-03T20:59:38.200 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:38.199+0000 7f24a9200640 1 -- 172.21.15.32:0/1946248391 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f24a40b3e90 con 0x7f2488004800 2024-08-03T20:59:38.200 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:38.200+0000 7f249b400640 1 -- 172.21.15.32:0/1946248391 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v293) ==== 53+0+73 (secure 0 0 0) 0x7f2494061e70 con 0x7f2488004800 2024-08-03T20:59:38.201 INFO:teuthology.orchestra.run.smithi032.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-08-03T20:59:38.203 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:38.202+0000 7f24a9200640 1 -- 172.21.15.32:0/1946248391 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f2480077010 msgr2=0x7f2480079520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:59:38.203 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:38.202+0000 7f24a9200640 1 --2- 172.21.15.32:0/1946248391 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f2480077010 0x7f2480079520 secure :-1 s=READY pgs=783 cs=0 l=1 rev1=1 crypto rx=0x7f248c033000 tx=0x7f24a40d2e20 comp rx=0 tx=0).stop 2024-08-03T20:59:38.203 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:38.202+0000 7f24a9200640 1 -- 172.21.15.32:0/1946248391 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2488004800 msgr2=0x7f24a415dc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:59:38.203 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:38.202+0000 7f24a9200640 1 --2- 172.21.15.32:0/1946248391 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2488004800 0x7f24a415dc70 secure :-1 s=READY pgs=1956 cs=0 l=1 rev1=1 crypto rx=0x7f2494011a10 tx=0x7f2494011ed0 comp rx=0 tx=0).stop 2024-08-03T20:59:38.203 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:38.202+0000 7f24a9200640 1 -- 172.21.15.32:0/1946248391 shutdown_connections 2024-08-03T20:59:38.203 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:38.202+0000 7f24a9200640 1 --2- 172.21.15.32:0/1946248391 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f2480077010 0x7f2480079520 unknown :-1 s=CLOSED pgs=783 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:59:38.203 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:38.202+0000 7f24a9200640 1 --2- 172.21.15.32:0/1946248391 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f24a415e450 0x7f24a41e8a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:59:38.203 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:38.202+0000 7f24a9200640 1 --2- 172.21.15.32:0/1946248391 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f24a40cd8e0 0x7f24a41f8f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:59:38.203 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:38.202+0000 7f24a9200640 1 --2- 172.21.15.32:0/1946248391 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2488004800 0x7f24a415dc70 unknown :-1 s=CLOSED pgs=1956 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:59:38.203 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:38.202+0000 7f24a9200640 1 -- 172.21.15.32:0/1946248391 >> 172.21.15.32:0/1946248391 conn(0x7f24a408d7a0 msgr2=0x7f24a406d6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:59:38.203 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:38.202+0000 7f24a9200640 1 -- 172.21.15.32:0/1946248391 shutdown_connections 2024-08-03T20:59:38.204 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:38.202+0000 7f24a9200640 1 -- 172.21.15.32:0/1946248391 wait complete. 2024-08-03T20:59:38.226 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-08-03T20:59:38.226 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-08-03T20:59:38.226 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi032.front.sepia.ceph.com 2024-08-03T20:59:38.226 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-08-03T20:59:38.226 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-08-03T20:59:38.226 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-08-03T20:59:38.226 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2024-08-03T20:59:38.226 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-08-03T20:59:38.226 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-08-03T20:59:38.227 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-08-03T20:59:38.227 DEBUG:teuthology.orchestra.run.smithi032:> ip addr 2024-08-03T20:59:38.263 INFO:teuthology.orchestra.run.smithi032.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-08-03T20:59:38.263 INFO:teuthology.orchestra.run.smithi032.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-08-03T20:59:38.263 INFO:teuthology.orchestra.run.smithi032.stdout: inet 127.0.0.1/8 scope host lo 2024-08-03T20:59:38.263 INFO:teuthology.orchestra.run.smithi032.stdout: valid_lft forever preferred_lft forever 2024-08-03T20:59:38.263 INFO:teuthology.orchestra.run.smithi032.stdout: inet6 ::1/128 scope host 2024-08-03T20:59:38.263 INFO:teuthology.orchestra.run.smithi032.stdout: valid_lft forever preferred_lft forever 2024-08-03T20:59:38.263 INFO:teuthology.orchestra.run.smithi032.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-08-03T20:59:38.263 INFO:teuthology.orchestra.run.smithi032.stdout: link/ether 0c:c4:7a:6b:fd:0c brd ff:ff:ff:ff:ff:ff 2024-08-03T20:59:38.263 INFO:teuthology.orchestra.run.smithi032.stdout: altname enp7s0f0 2024-08-03T20:59:38.263 INFO:teuthology.orchestra.run.smithi032.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-08-03T20:59:38.263 INFO:teuthology.orchestra.run.smithi032.stdout: link/ether 0c:c4:7a:6b:fd:0d brd ff:ff:ff:ff:ff:ff 2024-08-03T20:59:38.263 INFO:teuthology.orchestra.run.smithi032.stdout: altname enp7s0f1 2024-08-03T20:59:38.264 INFO:teuthology.orchestra.run.smithi032.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-08-03T20:59:38.264 INFO:teuthology.orchestra.run.smithi032.stdout: link/ether ac:1f:6b:f8:26:02 brd ff:ff:ff:ff:ff:ff 2024-08-03T20:59:38.264 INFO:teuthology.orchestra.run.smithi032.stdout: altname enp1s0f0 2024-08-03T20:59:38.264 INFO:teuthology.orchestra.run.smithi032.stdout: inet 172.21.15.32/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-08-03T20:59:38.264 INFO:teuthology.orchestra.run.smithi032.stdout: valid_lft 41778sec preferred_lft 41778sec 2024-08-03T20:59:38.264 INFO:teuthology.orchestra.run.smithi032.stdout: inet6 fe80::ae1f:6bff:fef8:2602/64 scope link noprefixroute 2024-08-03T20:59:38.264 INFO:teuthology.orchestra.run.smithi032.stdout: valid_lft forever preferred_lft forever 2024-08-03T20:59:38.264 INFO:teuthology.orchestra.run.smithi032.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-08-03T20:59:38.264 INFO:teuthology.orchestra.run.smithi032.stdout: link/ether ac:1f:6b:f8:26:03 brd ff:ff:ff:ff:ff:ff 2024-08-03T20:59:38.264 INFO:teuthology.orchestra.run.smithi032.stdout: altname enp1s0f1 2024-08-03T20:59:38.264 INFO:teuthology.orchestra.run.smithi032.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-08-03T20:59:38.264 INFO:teuthology.orchestra.run.smithi032.stdout: link/ether 0a:23:2b:ca:2a:93 brd ff:ff:ff:ff:ff:ff 2024-08-03T20:59:38.264 INFO:teuthology.orchestra.run.smithi032.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-08-03T20:59:38.265 INFO:teuthology.orchestra.run.smithi032.stdout: valid_lft forever preferred_lft forever 2024-08-03T20:59:38.265 INFO:teuthology.orchestra.run.smithi032.stdout: inet6 fe80::d804:7cff:fed9:7ba4/64 scope link 2024-08-03T20:59:38.265 INFO:teuthology.orchestra.run.smithi032.stdout: valid_lft forever preferred_lft forever 2024-08-03T20:59:38.265 INFO:teuthology.orchestra.run.smithi032.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-08-03T20:59:38.265 INFO:teuthology.orchestra.run.smithi032.stdout: link/ether 22:86:28:30:4e:62 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-08-03T20:59:38.265 INFO:teuthology.orchestra.run.smithi032.stdout: inet6 fe80::2086:28ff:fe30:4e62/64 scope link 2024-08-03T20:59:38.265 INFO:teuthology.orchestra.run.smithi032.stdout: valid_lft forever preferred_lft forever 2024-08-03T20:59:38.265 INFO:teuthology.orchestra.run.smithi032.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-08-03T20:59:38.265 INFO:teuthology.orchestra.run.smithi032.stdout: link/ether 0a:23:2b:ca:2a:93 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-08-03T20:59:38.265 INFO:teuthology.orchestra.run.smithi032.stdout: inet6 fe80::823:2bff:feca:2a93/64 scope link 2024-08-03T20:59:38.265 INFO:teuthology.orchestra.run.smithi032.stdout: valid_lft forever preferred_lft forever 2024-08-03T20:59:38.266 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-08-03T20:59:38.266 DEBUG:teuthology.orchestra.run.smithi032:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-08-03T20:59:38.357 INFO:teuthology.orchestra.run.smithi032.stdout:1 2024-08-03T20:59:38.360 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-08-03T20:59:38.360 DEBUG:teuthology.orchestra.run.smithi032:> route 2024-08-03T20:59:38.397 INFO:teuthology.orchestra.run.smithi032.stdout:Kernel IP routing table 2024-08-03T20:59:38.397 INFO:teuthology.orchestra.run.smithi032.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-08-03T20:59:38.397 INFO:teuthology.orchestra.run.smithi032.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-08-03T20:59:38.397 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-08-03T20:59:38.397 INFO:teuthology.orchestra.run.smithi032.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-08-03T20:59:38.397 DEBUG:teuthology.orchestra.run.smithi032:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-08-03T20:59:38.397 DEBUG:teuthology.orchestra.run.smithi032:> set -e 2024-08-03T20:59:38.398 DEBUG:teuthology.orchestra.run.smithi032:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-08-03T20:59:38.398 DEBUG:teuthology.orchestra.run.smithi032:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-08-03T20:59:38.398 DEBUG:teuthology.orchestra.run.smithi032:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-08-03T20:59:38.398 DEBUG:teuthology.orchestra.run.smithi032:> ') 2024-08-03T20:59:38.518 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:38 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-08-03T20:59:38.590 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:38 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-08-03T20:59:38.599 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-08-03T20:59:38.599 DEBUG:teuthology.orchestra.run.smithi032:> ip netns list 2024-08-03T20:59:38.635 INFO:teuthology.orchestra.run.smithi032.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-08-03T20:59:38.635 INFO:teuthology.orchestra.run.smithi032.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-08-03T20:59:38.635 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-08-03T20:59:38.635 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-08-03T20:59:38.635 DEBUG:teuthology.orchestra.run.smithi032:> sudo ip link set brx.0 up 2024-08-03T20:59:38.731 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-08-03T20:59:38.731 DEBUG:teuthology.orchestra.run.smithi032:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-08-03T20:59:38.766 INFO:teuthology.orchestra.run.smithi032.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-08-03T20:59:38.766 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-08-03T20:59:38.767 DEBUG:teuthology.orchestra.run.smithi032:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-08-03T20:59:38.841 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2024-08-03T20:59:38.841 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-08-03T20:59:38.842 DEBUG:teuthology.orchestra.run.smithi032:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-08-03T20:59:38.942 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-08-03T20:59:38.946 DEBUG:teuthology.orchestra.run.smithi118:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-08-03T20:59:39.039 INFO:teuthology.orchestra.run.smithi032.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-08-03T20:59:39.040 INFO:teuthology.orchestra.run.smithi032.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-08-03T20:59:39.040 INFO:teuthology.orchestra.run.smithi032.stdout:invalid new device string format 2024-08-03T20:59:39.061 INFO:teuthology.orchestra.run.smithi032.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-08-03T20:59:39.061 INFO:teuthology.orchestra.run.smithi032.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-08-03T20:59:39.061 INFO:teuthology.orchestra.run.smithi032.stdout:invalid new device string format 2024-08-03T20:59:39.061 INFO:teuthology.orchestra.run.smithi032.stdout:mount.ceph: resolved to: "172.21.15.32:3300,172.21.15.118:3300,172.21.15.118:3301" 2024-08-03T20:59:39.061 INFO:teuthology.orchestra.run.smithi032.stdout:mount.ceph: switching to using v1 address with old syntax 2024-08-03T20:59:39.094 INFO:teuthology.orchestra.run.smithi032.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-08-03T20:59:39.094 INFO:teuthology.orchestra.run.smithi032.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-08-03T20:59:39.094 INFO:teuthology.orchestra.run.smithi032.stdout:invalid new device string format 2024-08-03T20:59:39.094 INFO:teuthology.orchestra.run.smithi032.stdout:mount.ceph: resolved to: "172.21.15.32:3300,172.21.15.118:3300,172.21.15.118:3301" 2024-08-03T20:59:39.094 INFO:teuthology.orchestra.run.smithi032.stdout:mount.ceph: switching to using v1 address with old syntax 2024-08-03T20:59:39.095 INFO:teuthology.orchestra.run.smithi032.stdout:mount.ceph: resolved to: "172.21.15.32:6789,172.21.15.118:6789,172.21.15.118:6790" 2024-08-03T20:59:39.095 INFO:teuthology.orchestra.run.smithi032.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.32:6789,172.21.15.118:6789,172.21.15.118:6790:/ 2024-08-03T20:59:39.095 INFO:teuthology.orchestra.run.smithi032.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,key=0,fsid=40ab141a-01d7-49b9-ae3c-2edec9876e95" will pass to kernel 2024-08-03T20:59:39.096 INFO:teuthology.orchestra.run.smithi032.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2024-08-03T20:59:39.096 INFO:teuthology.orchestra.run.smithi032.stdout: You just mounted a file system that supports labels which does not 2024-08-03T20:59:39.096 INFO:teuthology.orchestra.run.smithi032.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-08-03T20:59:39.096 INFO:teuthology.orchestra.run.smithi032.stdout: applications will generate AVC messages and not be allowed access to 2024-08-03T20:59:39.096 INFO:teuthology.orchestra.run.smithi032.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-08-03T20:59:39.099 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-08-03T20:59:39.099 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-08-03T20:59:39.099 DEBUG:teuthology.orchestra.run.smithi032:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-08-03T20:59:39.195 DEBUG:teuthology.orchestra.run.smithi032:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-08-03T20:59:39.275 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:39 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-08-03T20:59:39.278 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:39 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-08-03T20:59:39.286 INFO:teuthology.orchestra.run.smithi032.stdout:client8183 2024-08-03T20:59:39.286 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-08-03T20:59:39.286 DEBUG:teuthology.orchestra.run.smithi032:> dd if=/proc/self/mounts of=/dev/stdout 2024-08-03T20:59:39.321 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-08-03T20:59:39.322 DEBUG:teuthology.orchestra.run.smithi032:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls -R .) 2024-08-03T20:59:39.365 INFO:teuthology.orchestra.run:waiting for 20 2024-08-03T20:59:39.398 INFO:tasks.ceph.mds.a.smithi032.stderr:2024-08-03T20:59:39.397+0000 7f97df200640 -1 mds.0.cache.dir(0x10000000000) Corrupt fnode in dirfrag 0x10000000000: Decoder at 'void fnode_t::decode(ceph::buffer::v15_2_0::list::const_iterator&)' v=3 cannot decode v=100 minimal_decoder=101: Malformed input [buffer:3] 2024-08-03T20:59:39.398 INFO:teuthology.orchestra.run.smithi032.stdout:.: 2024-08-03T20:59:39.399 INFO:teuthology.orchestra.run.smithi032.stdout:subdir 2024-08-03T20:59:39.399 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-03T20:59:39.399 INFO:teuthology.orchestra.run.smithi032.stdout:./subdir: 2024-08-03T20:59:39.400 INFO:tasks.cephfs.test_damage:Result: Mutation 'Corrupt omap header on 10000000000.00000000' did not caused DAMAGED state 2024-08-03T20:59:39.400 INFO:tasks.cephfs.test_damage:Applying mutation 'Corrupt omap header on 600.00000000' 2024-08-03T20:59:39.400 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-08-03T20:59:39.400 DEBUG:teuthology.orchestra.run.smithi032:> dd if=/proc/self/mounts of=/dev/stdout 2024-08-03T20:59:39.477 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-08-03T20:59:39.478 DEBUG:teuthology.orchestra.run.smithi032:> dd if=/proc/self/mounts of=/dev/stdout 2024-08-03T20:59:39.553 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-08-03T20:59:39.826 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:39.825+0000 7f6c30c00640 1 -- 172.21.15.32:0/3084688350 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6c100099b0 msgr2=0x7f6c1000bdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:59:39.826 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:39.825+0000 7f6c30c00640 1 --2- 172.21.15.32:0/3084688350 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6c100099b0 0x7f6c1000bdd0 secure :-1 s=READY pgs=1958 cs=0 l=1 rev1=1 crypto rx=0x7f6c2c15d9c0 tx=0x7f6c10005830 comp rx=0 tx=0).stop 2024-08-03T20:59:39.826 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:39.825+0000 7f6c2be00640 1 -- 172.21.15.32:0/3084688350 reap_dead start 2024-08-03T20:59:39.826 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:39.825+0000 7f6c30c00640 1 -- 172.21.15.32:0/3084688350 shutdown_connections 2024-08-03T20:59:39.826 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:39.825+0000 7f6c30c00640 1 -- 172.21.15.32:0/3084688350 >> 172.21.15.32:0/3084688350 conn(0x7f6c2c0b2120 msgr2=0x7f6c2c0cfd60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:59:39.826 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:39.825+0000 7f6c30c00640 1 -- 172.21.15.32:0/3084688350 shutdown_connections 2024-08-03T20:59:39.826 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:39.825+0000 7f6c30c00640 1 -- 172.21.15.32:0/3084688350 wait complete. 2024-08-03T20:59:39.827 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:39.827+0000 7f6c30c00640 1 Processor -- start 2024-08-03T20:59:39.827 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:39.827+0000 7f6c30c00640 1 -- start start 2024-08-03T20:59:39.830 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:39.829+0000 7f6c30c00640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f6c100099b0 0x7f6c2c0c9390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:59:39.830 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:39.829+0000 7f6c30c00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6c2c0cd930 0x7f6c2c0cb320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:59:39.830 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:39.829+0000 7f6c30c00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f6c2c0c9b70 0x7f6c2c0cb860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:59:39.830 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:39.829+0000 7f6c30c00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f6c2c0742b0 con 0x7f6c2c0cd930 2024-08-03T20:59:39.830 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:39.829+0000 7f6c30c00640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f6c2c1607f0 con 0x7f6c2c0c9b70 2024-08-03T20:59:39.831 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:39.829+0000 7f6c30c00640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7f6c2c074430 con 0x7f6c100099b0 2024-08-03T20:59:39.831 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:39.829+0000 7f6c22a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6c2c0cd930 0x7f6c2c0cb320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:59:39.831 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:39.829+0000 7f6c2b400640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f6c100099b0 0x7f6c2c0c9390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:59:39.831 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:39.829+0000 7f6c22a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6c2c0cd930 0x7f6c2c0cb320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:33062/0 (socket says 172.21.15.32:33062) 2024-08-03T20:59:39.831 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:39.829+0000 7f6c22a00640 1 -- 172.21.15.32:0/2708743287 learned_addr learned my addr 172.21.15.32:0/2708743287 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:59:39.831 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:39.829+0000 7f6c2be00640 1 --2- 172.21.15.32:0/2708743287 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f6c2c0c9b70 0x7f6c2c0cb860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:59:39.831 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:39.830+0000 7f6c22a00640 1 -- 172.21.15.32:0/2708743287 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f6c100099b0 msgr2=0x7f6c2c0c9390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:59:39.831 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:39.830+0000 7f6c22a00640 1 --2- 172.21.15.32:0/2708743287 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f6c100099b0 0x7f6c2c0c9390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:59:39.831 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:39.830+0000 7f6c22a00640 1 -- 172.21.15.32:0/2708743287 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f6c2c0c9b70 msgr2=0x7f6c2c0cb860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:59:39.831 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:39.830+0000 7f6c22a00640 1 --2- 172.21.15.32:0/2708743287 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f6c2c0c9b70 0x7f6c2c0cb860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:59:39.831 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:39.830+0000 7f6c22a00640 1 -- 172.21.15.32:0/2708743287 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6c2c0c43d0 con 0x7f6c2c0cd930 2024-08-03T20:59:39.832 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:39.830+0000 7f6c22a00640 1 --2- 172.21.15.32:0/2708743287 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6c2c0cd930 0x7f6c2c0cb320 secure :-1 s=READY pgs=1959 cs=0 l=1 rev1=1 crypto rx=0x7f6c14006660 tx=0x7f6c140039c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:59:39.832 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:39.830+0000 7f6c23e00640 1 -- 172.21.15.32:0/2708743287 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c14015070 con 0x7f6c2c0cd930 2024-08-03T20:59:39.832 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:39.830+0000 7f6c23e00640 1 -- 172.21.15.32:0/2708743287 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f6c14011900 con 0x7f6c2c0cd930 2024-08-03T20:59:39.832 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:39.830+0000 7f6c30c00640 1 -- 172.21.15.32:0/2708743287 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6c2c0b5b40 con 0x7f6c2c0cd930 2024-08-03T20:59:39.833 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:39.830+0000 7f6c23e00640 1 -- 172.21.15.32:0/2708743287 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c14011d30 con 0x7f6c2c0cd930 2024-08-03T20:59:39.833 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:39.830+0000 7f6c30c00640 1 -- 172.21.15.32:0/2708743287 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6c2c0b4f30 con 0x7f6c2c0cd930 2024-08-03T20:59:39.833 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:39.831+0000 7f6c30c00640 1 -- 172.21.15.32:0/2708743287 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6c2c0ad3a0 con 0x7f6c2c0cd930 2024-08-03T20:59:39.833 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:39.833+0000 7f6c23e00640 1 -- 172.21.15.32:0/2708743287 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f6c14018070 con 0x7f6c2c0cd930 2024-08-03T20:59:39.834 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:39.834+0000 7f6c23e00640 1 --2- 172.21.15.32:0/2708743287 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f6c08077010 0x7f6c08079520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:59:39.834 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:39.834+0000 7f6c23e00640 1 -- 172.21.15.32:0/2708743287 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(119..119 src has 1..119) ==== 8778+0+0 (secure 0 0 0) 0x7f6c1409a220 con 0x7f6c2c0cd930 2024-08-03T20:59:39.834 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:39.834+0000 7f6c2b400640 1 --2- 172.21.15.32:0/2708743287 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f6c08077010 0x7f6c08079520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:59:39.835 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:39.835+0000 7f6c2b400640 1 --2- 172.21.15.32:0/2708743287 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f6c08077010 0x7f6c08079520 secure :-1 s=READY pgs=784 cs=0 l=1 rev1=1 crypto rx=0x7f6c1c005fd0 tx=0x7f6c1c005dd0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:59:39.838 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:39.838+0000 7f6c23e00640 1 -- 172.21.15.32:0/2708743287 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7f6c140621c0 con 0x7f6c2c0cd930 2024-08-03T20:59:40.054 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:40.053+0000 7f6c30c00640 1 -- 172.21.15.32:0/2708743287 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f6c2c1f8bd0 con 0x7f6c2c0cd930 2024-08-03T20:59:40.055 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:40.054+0000 7f6c23e00640 1 -- 172.21.15.32:0/2708743287 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 68 entries v119) ==== 81+0+4134 (secure 0 0 0) 0x7f6c14066710 con 0x7f6c2c0cd930 2024-08-03T20:59:40.056 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6835/2173042128 2024-08-04T20:59:28.193772+0000 2024-08-03T20:59:40.056 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6833/2173042128 2024-08-04T20:59:28.193772+0000 2024-08-03T20:59:40.056 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6835/506775245 2024-08-04T20:59:03.076256+0000 2024-08-03T20:59:40.056 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6833/506775245 2024-08-04T20:59:03.076256+0000 2024-08-03T20:59:40.056 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6833/556650445 2024-08-04T20:58:57.063754+0000 2024-08-03T20:59:40.057 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6835/2156596385 2024-08-04T20:58:35.073838+0000 2024-08-03T20:59:40.057 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6833/2156596385 2024-08-04T20:58:35.073838+0000 2024-08-03T20:59:40.057 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6835/3669493978 2024-08-04T20:58:30.064676+0000 2024-08-03T20:59:40.057 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6833/2178506907 2024-08-04T20:56:40.045549+0000 2024-08-03T20:59:40.057 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6835/556650445 2024-08-04T20:58:57.063754+0000 2024-08-03T20:59:40.057 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6835/2278473729 2024-08-04T20:56:13.041693+0000 2024-08-03T20:59:40.057 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6835/2345472481 2024-08-04T20:57:25.968590+0000 2024-08-03T20:59:40.058 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6833/3940515773 2024-08-04T20:58:24.290690+0000 2024-08-03T20:59:40.058 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6833/2600386707 2024-08-04T20:57:50.063964+0000 2024-08-03T20:59:40.058 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6835/2178506907 2024-08-04T20:56:40.045549+0000 2024-08-03T20:59:40.058 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6833/2292690833 2024-08-04T20:55:20.021329+0000 2024-08-03T20:59:40.058 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6835/2803901497 2024-08-04T20:55:55.181144+0000 2024-08-03T20:59:40.058 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6835/2194902489 2024-08-04T20:55:00.359126+0000 2024-08-03T20:59:40.058 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6833/2803901497 2024-08-04T20:55:55.181144+0000 2024-08-03T20:59:40.059 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6835/3191654276 2024-08-04T20:55:42.433364+0000 2024-08-03T20:59:40.059 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6833/2242533881 2024-08-04T20:56:48.048099+0000 2024-08-03T20:59:40.059 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6833/2283924374 2024-08-04T20:57:34.396782+0000 2024-08-03T20:59:40.059 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6835/1172050551 2024-08-04T20:55:26.214113+0000 2024-08-03T20:59:40.059 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6835/1016338829 2024-08-04T20:55:06.071485+0000 2024-08-03T20:59:40.059 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6833/1474984145 2024-08-04T20:57:40.052808+0000 2024-08-03T20:59:40.059 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6833/3669493978 2024-08-04T20:58:30.064676+0000 2024-08-03T20:59:40.059 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6835/756683988 2024-08-04T20:55:14.119228+0000 2024-08-03T20:59:40.060 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6833/2194902489 2024-08-04T20:55:00.359126+0000 2024-08-03T20:59:40.060 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6835/2806168786 2024-08-04T20:56:37.279160+0000 2024-08-03T20:59:40.060 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6833/2278473729 2024-08-04T20:56:13.041693+0000 2024-08-03T20:59:40.060 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6833/3191654276 2024-08-04T20:55:42.433364+0000 2024-08-03T20:59:40.060 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6835/3431300225 2024-08-04T20:56:21.088927+0000 2024-08-03T20:59:40.060 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6833/4172198298 2024-08-04T20:56:04.256182+0000 2024-08-03T20:59:40.060 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6833/3431300225 2024-08-04T20:56:21.088927+0000 2024-08-03T20:59:40.061 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6835/2292690833 2024-08-04T20:55:20.021329+0000 2024-08-03T20:59:40.061 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6833/2806168786 2024-08-04T20:56:37.279160+0000 2024-08-03T20:59:40.061 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6833/1172050551 2024-08-04T20:55:26.214113+0000 2024-08-03T20:59:40.061 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6835/4172198298 2024-08-04T20:56:04.256182+0000 2024-08-03T20:59:40.061 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6835/3409752696 2024-08-04T20:58:18.243140+0000 2024-08-03T20:59:40.061 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6833/2274054575 2024-08-04T20:57:55.062490+0000 2024-08-03T20:59:40.061 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6833/1016338829 2024-08-04T20:55:06.071485+0000 2024-08-03T20:59:40.062 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6833/1707454554 2024-08-04T20:54:38.409411+0000 2024-08-03T20:59:40.062 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6835/2242533881 2024-08-04T20:56:48.048099+0000 2024-08-03T20:59:40.062 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6833/739874257 2024-08-04T20:56:56.101271+0000 2024-08-03T20:59:40.062 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6835/739874257 2024-08-04T20:56:56.101271+0000 2024-08-03T20:59:40.062 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6833/2362145329 2024-08-04T20:57:02.150199+0000 2024-08-03T20:59:40.063 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6833/3771304550 2024-08-04T20:55:58.212752+0000 2024-08-03T20:59:40.063 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6835/2362145329 2024-08-04T20:57:02.150199+0000 2024-08-03T20:59:40.063 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6835/3940515773 2024-08-04T20:58:24.290690+0000 2024-08-03T20:59:40.063 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6833/1885034124 2024-08-04T20:57:08.190707+0000 2024-08-03T20:59:40.063 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6833/511221267 2024-08-04T20:57:45.062448+0000 2024-08-03T20:59:40.063 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6835/3771304550 2024-08-04T20:55:58.212752+0000 2024-08-03T20:59:40.064 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6835/1885034124 2024-08-04T20:57:08.190707+0000 2024-08-03T20:59:40.064 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6833/2345472481 2024-08-04T20:57:25.968590+0000 2024-08-03T20:59:40.064 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6833/265163622 2024-08-04T20:57:28.354509+0000 2024-08-03T20:59:40.064 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6835/265163622 2024-08-04T20:57:28.354509+0000 2024-08-03T20:59:40.064 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6835/2274054575 2024-08-04T20:57:55.062490+0000 2024-08-03T20:59:40.064 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6835/2283924374 2024-08-04T20:57:34.396782+0000 2024-08-03T20:59:40.065 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6833/756683988 2024-08-04T20:55:14.119228+0000 2024-08-03T20:59:40.065 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6835/1474984145 2024-08-04T20:57:40.052808+0000 2024-08-03T20:59:40.065 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6835/511221267 2024-08-04T20:57:45.062448+0000 2024-08-03T20:59:40.065 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6835/1707454554 2024-08-04T20:54:38.409411+0000 2024-08-03T20:59:40.065 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6835/2600386707 2024-08-04T20:57:50.063964+0000 2024-08-03T20:59:40.065 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6833/675070591 2024-08-04T20:58:00.068114+0000 2024-08-03T20:59:40.065 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6833/3340816283 2024-08-04T20:58:15.214542+0000 2024-08-03T20:59:40.066 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6835/3340816283 2024-08-04T20:58:15.214542+0000 2024-08-03T20:59:40.066 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6835/675070591 2024-08-04T20:58:00.068114+0000 2024-08-03T20:59:40.066 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6833/3409752696 2024-08-04T20:58:18.243140+0000 2024-08-03T20:59:40.067 INFO:teuthology.orchestra.run.smithi032.stderr:listed 68 entries 2024-08-03T20:59:40.067 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:40.057+0000 7f6c30c00640 1 -- 172.21.15.32:0/2708743287 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f6c08077010 msgr2=0x7f6c08079520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:59:40.067 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:40.057+0000 7f6c30c00640 1 --2- 172.21.15.32:0/2708743287 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f6c08077010 0x7f6c08079520 secure :-1 s=READY pgs=784 cs=0 l=1 rev1=1 crypto rx=0x7f6c1c005fd0 tx=0x7f6c1c005dd0 comp rx=0 tx=0).stop 2024-08-03T20:59:40.067 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:40.057+0000 7f6c30c00640 1 -- 172.21.15.32:0/2708743287 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6c2c0cd930 msgr2=0x7f6c2c0cb320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:59:40.067 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:40.057+0000 7f6c30c00640 1 --2- 172.21.15.32:0/2708743287 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6c2c0cd930 0x7f6c2c0cb320 secure :-1 s=READY pgs=1959 cs=0 l=1 rev1=1 crypto rx=0x7f6c14006660 tx=0x7f6c140039c0 comp rx=0 tx=0).stop 2024-08-03T20:59:40.067 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:40.057+0000 7f6c30c00640 1 -- 172.21.15.32:0/2708743287 shutdown_connections 2024-08-03T20:59:40.067 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:40.057+0000 7f6c30c00640 1 --2- 172.21.15.32:0/2708743287 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f6c08077010 0x7f6c08079520 unknown :-1 s=CLOSED pgs=784 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:59:40.067 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:40.057+0000 7f6c30c00640 1 --2- 172.21.15.32:0/2708743287 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f6c2c0c9b70 0x7f6c2c0cb860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:59:40.068 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:40.057+0000 7f6c30c00640 1 --2- 172.21.15.32:0/2708743287 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6c2c0cd930 0x7f6c2c0cb320 unknown :-1 s=CLOSED pgs=1959 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:59:40.068 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:40.057+0000 7f6c30c00640 1 --2- 172.21.15.32:0/2708743287 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f6c100099b0 0x7f6c2c0c9390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:59:40.068 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:40.057+0000 7f6c30c00640 1 -- 172.21.15.32:0/2708743287 >> 172.21.15.32:0/2708743287 conn(0x7f6c2c0b2120 msgr2=0x7f6c2c0af8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:59:40.068 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:40.057+0000 7f6c30c00640 1 -- 172.21.15.32:0/2708743287 shutdown_connections 2024-08-03T20:59:40.068 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:40.057+0000 7f6c30c00640 1 -- 172.21.15.32:0/2708743287 wait complete. 2024-08-03T20:59:40.082 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-08-03T20:59:40.083 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-08-03T20:59:40.083 DEBUG:teuthology.orchestra.run.smithi032:> sudo umount /home/ubuntu/cephtest/mnt.0 -f 2024-08-03T20:59:40.152 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi032.front.sepia.ceph.com 2024-08-03T20:59:40.153 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-08-03T20:59:40.153 DEBUG:teuthology.orchestra.run.smithi032:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-08-03T20:59:40.189 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-08-03T20:59:40.462 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:40.458+0000 7f6f83e00640 1 -- 172.21.15.32:0/2261732436 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6f840af020 msgr2=0x7f6f840742b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:59:40.462 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:40.458+0000 7f6f83e00640 1 --2- 172.21.15.32:0/2261732436 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6f840af020 0x7f6f840742b0 secure :-1 s=READY pgs=1960 cs=0 l=1 rev1=1 crypto rx=0x7f6f84160f00 tx=0x7f6f7402a560 comp rx=0 tx=0).stop 2024-08-03T20:59:40.463 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:40.459+0000 7f6f83e00640 1 -- 172.21.15.32:0/2261732436 shutdown_connections 2024-08-03T20:59:40.463 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:40.459+0000 7f6f83e00640 1 --2- 172.21.15.32:0/2261732436 >> v2:172.21.15.118:3300/0 conn(0x7f6f840acd80 0x7f6f840ad190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:59:40.463 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:40.459+0000 7f6f83e00640 1 --2- 172.21.15.32:0/2261732436 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6f840af020 0x7f6f840742b0 unknown :-1 s=CLOSED pgs=1960 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:59:40.463 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:40.459+0000 7f6f83e00640 1 -- 172.21.15.32:0/2261732436 >> 172.21.15.32:0/2261732436 conn(0x7f6f840b2120 msgr2=0x7f6f840cfd60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:59:40.463 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:40.459+0000 7f6f83e00640 1 -- 172.21.15.32:0/2261732436 shutdown_connections 2024-08-03T20:59:40.463 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:40.459+0000 7f6f83e00640 1 -- 172.21.15.32:0/2261732436 wait complete. 2024-08-03T20:59:40.463 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:40.461+0000 7f6f83e00640 1 Processor -- start 2024-08-03T20:59:40.463 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:40.461+0000 7f6f83e00640 1 -- start start 2024-08-03T20:59:40.465 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:40.462+0000 7f6f83e00640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f6f8406f440 0x7f6f841f5fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:59:40.465 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:40.463+0000 7f6f83e00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f6f840acd80 0x7f6f840cd930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:59:40.465 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:40.463+0000 7f6f83e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6f840af020 0x7f6f840b39f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:59:40.466 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:40.463+0000 7f6f83e00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f6f841601e0 con 0x7f6f840af020 2024-08-03T20:59:40.466 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:40.463+0000 7f6f83e00640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f6f84160060 con 0x7f6f840acd80 2024-08-03T20:59:40.466 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:40.463+0000 7f6f83e00640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7f6f84160360 con 0x7f6f8406f440 2024-08-03T20:59:40.466 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:40.463+0000 7f6f83400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6f840af020 0x7f6f840b39f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:59:40.466 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:40.463+0000 7f6f83400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6f840af020 0x7f6f840b39f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:33088/0 (socket says 172.21.15.32:33088) 2024-08-03T20:59:40.466 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:40.463+0000 7f6f83400640 1 -- 172.21.15.32:0/1347935454 learned_addr learned my addr 172.21.15.32:0/1347935454 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:59:40.466 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:40.463+0000 7f6f7a000640 1 --2- 172.21.15.32:0/1347935454 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f6f840acd80 0x7f6f840cd930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:59:40.466 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:40.463+0000 7f6f82a00640 1 --2- 172.21.15.32:0/1347935454 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f6f8406f440 0x7f6f841f5fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:59:40.466 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:40.463+0000 7f6f83400640 1 -- 172.21.15.32:0/1347935454 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f6f8406f440 msgr2=0x7f6f841f5fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:59:40.466 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:40.463+0000 7f6f83400640 1 --2- 172.21.15.32:0/1347935454 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f6f8406f440 0x7f6f841f5fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:59:40.466 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:40.463+0000 7f6f83400640 1 -- 172.21.15.32:0/1347935454 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f6f840acd80 msgr2=0x7f6f840cd930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:59:40.466 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:40.463+0000 7f6f83400640 1 --2- 172.21.15.32:0/1347935454 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f6f840acd80 0x7f6f840cd930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:59:40.467 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:40.463+0000 7f6f83400640 1 -- 172.21.15.32:0/1347935454 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6f840b4a20 con 0x7f6f840af020 2024-08-03T20:59:40.467 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:40.464+0000 7f6f83400640 1 --2- 172.21.15.32:0/1347935454 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6f840af020 0x7f6f840b39f0 secure :-1 s=READY pgs=1961 cs=0 l=1 rev1=1 crypto rx=0x7f6f7c00c6b0 tx=0x7f6f7c00cb70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:59:40.467 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:40.464+0000 7f6f7b400640 1 -- 172.21.15.32:0/1347935454 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f7c003b70 con 0x7f6f840af020 2024-08-03T20:59:40.467 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:40.464+0000 7f6f7b400640 1 -- 172.21.15.32:0/1347935454 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f6f7c003d10 con 0x7f6f840af020 2024-08-03T20:59:40.467 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:40.464+0000 7f6f7b400640 1 -- 172.21.15.32:0/1347935454 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f7c005330 con 0x7f6f840af020 2024-08-03T20:59:40.467 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:40.464+0000 7f6f83e00640 1 -- 172.21.15.32:0/1347935454 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6f840b2c30 con 0x7f6f840af020 2024-08-03T20:59:40.467 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:40.464+0000 7f6f83e00640 1 -- 172.21.15.32:0/1347935454 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6f840b31f0 con 0x7f6f840af020 2024-08-03T20:59:40.468 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:40.465+0000 7f6f83e00640 1 -- 172.21.15.32:0/1347935454 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6f840ad190 con 0x7f6f840af020 2024-08-03T20:59:40.468 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:40.466+0000 7f6f7b400640 1 -- 172.21.15.32:0/1347935454 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f6f7c0054d0 con 0x7f6f840af020 2024-08-03T20:59:40.468 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:40.467+0000 7f6f7b400640 1 --2- 172.21.15.32:0/1347935454 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f6f60077010 0x7f6f60079520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:59:40.468 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:40.467+0000 7f6f7b400640 1 -- 172.21.15.32:0/1347935454 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(119..119 src has 1..119) ==== 8778+0+0 (secure 0 0 0) 0x7f6f7c099250 con 0x7f6f840af020 2024-08-03T20:59:40.468 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:40.468+0000 7f6f82a00640 1 --2- 172.21.15.32:0/1347935454 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f6f60077010 0x7f6f60079520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:59:40.469 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:40.469+0000 7f6f82a00640 1 --2- 172.21.15.32:0/1347935454 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f6f60077010 0x7f6f60079520 secure :-1 s=READY pgs=785 cs=0 l=1 rev1=1 crypto rx=0x7f6f74033000 tx=0x7f6f74004730 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:59:40.472 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:40.472+0000 7f6f7b400640 1 -- 172.21.15.32:0/1347935454 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7f6f7c0611f0 con 0x7f6f840af020 2024-08-03T20:59:40.719 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:40.718+0000 7f6f83e00640 1 -- 172.21.15.32:0/1347935454 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7f6f840d0930 con 0x7f6f840af020 2024-08-03T20:59:40.720 INFO:tasks.ceph.mon.a.smithi032.stderr:2024-08-03T20:59:40.719+0000 7f4e6d600640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-08-03T20:59:40.722 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:40.721+0000 7f6f7b400640 1 -- 172.21.15.32:0/1347935454 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v294) ==== 161+0+0 (secure 0 0 0) 0x7f6f7c065740 con 0x7f6f840af020 2024-08-03T20:59:40.722 INFO:teuthology.orchestra.run.smithi032.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-08-03T20:59:40.724 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:40.724+0000 7f6f83e00640 1 -- 172.21.15.32:0/1347935454 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f6f60077010 msgr2=0x7f6f60079520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:59:40.725 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:40.724+0000 7f6f83e00640 1 --2- 172.21.15.32:0/1347935454 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f6f60077010 0x7f6f60079520 secure :-1 s=READY pgs=785 cs=0 l=1 rev1=1 crypto rx=0x7f6f74033000 tx=0x7f6f74004730 comp rx=0 tx=0).stop 2024-08-03T20:59:40.725 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:40.724+0000 7f6f83e00640 1 -- 172.21.15.32:0/1347935454 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6f840af020 msgr2=0x7f6f840b39f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:59:40.725 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:40.724+0000 7f6f83e00640 1 --2- 172.21.15.32:0/1347935454 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6f840af020 0x7f6f840b39f0 secure :-1 s=READY pgs=1961 cs=0 l=1 rev1=1 crypto rx=0x7f6f7c00c6b0 tx=0x7f6f7c00cb70 comp rx=0 tx=0).stop 2024-08-03T20:59:40.725 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:40.724+0000 7f6f83e00640 1 -- 172.21.15.32:0/1347935454 shutdown_connections 2024-08-03T20:59:40.725 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:40.724+0000 7f6f83e00640 1 --2- 172.21.15.32:0/1347935454 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f6f60077010 0x7f6f60079520 unknown :-1 s=CLOSED pgs=785 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:59:40.725 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:40.724+0000 7f6f83e00640 1 --2- 172.21.15.32:0/1347935454 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f6f840acd80 0x7f6f840cd930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:59:40.725 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:40.724+0000 7f6f83e00640 1 --2- 172.21.15.32:0/1347935454 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6f840af020 0x7f6f840b39f0 unknown :-1 s=CLOSED pgs=1961 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:59:40.725 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:40.724+0000 7f6f83e00640 1 --2- 172.21.15.32:0/1347935454 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f6f8406f440 0x7f6f841f5fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:59:40.725 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:40.724+0000 7f6f83e00640 1 -- 172.21.15.32:0/1347935454 >> 172.21.15.32:0/1347935454 conn(0x7f6f840b2120 msgr2=0x7f6f840cc7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:59:40.725 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:40.724+0000 7f6f83e00640 1 -- 172.21.15.32:0/1347935454 shutdown_connections 2024-08-03T20:59:40.725 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:40.724+0000 7f6f83e00640 1 -- 172.21.15.32:0/1347935454 wait complete. 2024-08-03T20:59:40.729 INFO:tasks.ceph.mds.a.smithi032.stderr: -739> 2024-08-03T20:59:34.209+0000 7f97e4200640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-08-03T20:59:40.731 INFO:tasks.ceph.mds.a.smithi032.stderr: -180> 2024-08-03T20:59:39.397+0000 7f97df200640 -1 mds.0.cache.dir(0x10000000000) Corrupt fnode in dirfrag 0x10000000000: Decoder at 'void fnode_t::decode(ceph::buffer::v15_2_0::list::const_iterator&)' v=3 cannot decode v=100 minimal_decoder=101: Malformed input [buffer:3] 2024-08-03T20:59:40.752 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mds repaired 0 2024-08-03T20:59:40.849 INFO:tasks.ceph.mds.a.smithi032.stdout:starting mds.a at 2024-08-03T20:59:40.986 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:40.985+0000 7fbc3fc00640 1 -- 172.21.15.32:0/1419836535 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbc38165160 msgr2=0x7fbc38167580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:59:40.986 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:40.985+0000 7fbc3fc00640 1 --2- 172.21.15.32:0/1419836535 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbc38165160 0x7fbc38167580 secure :-1 s=READY pgs=1964 cs=0 l=1 rev1=1 crypto rx=0x7fbc30005fd0 tx=0x7fbc3002a8b0 comp rx=0 tx=0).stop 2024-08-03T20:59:40.986 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:40.985+0000 7fbc3fc00640 1 -- 172.21.15.32:0/1419836535 shutdown_connections 2024-08-03T20:59:40.986 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:40.985+0000 7fbc3fc00640 1 --2- 172.21.15.32:0/1419836535 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbc38165160 0x7fbc38167580 unknown :-1 s=CLOSED pgs=1964 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:59:40.986 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:40.985+0000 7fbc3fc00640 1 -- 172.21.15.32:0/1419836535 >> 172.21.15.32:0/1419836535 conn(0x7fbc380b1bd0 msgr2=0x7fbc380b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:59:40.986 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:40.985+0000 7fbc3fc00640 1 -- 172.21.15.32:0/1419836535 shutdown_connections 2024-08-03T20:59:40.986 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:40.985+0000 7fbc3fc00640 1 -- 172.21.15.32:0/1419836535 wait complete. 2024-08-03T20:59:40.988 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:40.987+0000 7fbc3fc00640 1 Processor -- start 2024-08-03T20:59:40.988 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:40.987+0000 7fbc3fc00640 1 -- start start 2024-08-03T20:59:40.990 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:40.989+0000 7fbc3fc00640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fbc3806e930 0x7fbc38156aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:59:40.990 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:40.989+0000 7fbc3fc00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fbc38144f90 0x7fbc380b3160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:59:40.990 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:40.989+0000 7fbc3fc00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbc38165160 0x7fbc381593f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:59:40.990 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:40.989+0000 7fbc3fc00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fbc380cf9c0 con 0x7fbc38165160 2024-08-03T20:59:40.990 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:40.989+0000 7fbc3fc00640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7fbc380cf840 con 0x7fbc38144f90 2024-08-03T20:59:40.990 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:40.989+0000 7fbc3fc00640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7fbc38167ac0 con 0x7fbc3806e930 2024-08-03T20:59:40.991 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:40.989+0000 7fbc3dc00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbc38165160 0x7fbc381593f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:59:40.991 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:40.989+0000 7fbc3dc00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbc38165160 0x7fbc381593f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:33106/0 (socket says 172.21.15.32:33106) 2024-08-03T20:59:40.991 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:40.989+0000 7fbc3dc00640 1 -- 172.21.15.32:0/4111314630 learned_addr learned my addr 172.21.15.32:0/4111314630 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:59:40.991 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:40.989+0000 7fbc3d200640 1 --2- 172.21.15.32:0/4111314630 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fbc3806e930 0x7fbc38156aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:59:40.991 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:40.989+0000 7fbc37e00640 1 --2- 172.21.15.32:0/4111314630 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fbc38144f90 0x7fbc380b3160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:59:40.991 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:40.990+0000 7fbc3dc00640 1 -- 172.21.15.32:0/4111314630 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fbc3806e930 msgr2=0x7fbc38156aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:59:40.991 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:40.990+0000 7fbc3dc00640 1 --2- 172.21.15.32:0/4111314630 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fbc3806e930 0x7fbc38156aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:59:40.991 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:40.990+0000 7fbc3dc00640 1 -- 172.21.15.32:0/4111314630 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fbc38144f90 msgr2=0x7fbc380b3160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:59:40.991 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:40.990+0000 7fbc3dc00640 1 --2- 172.21.15.32:0/4111314630 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fbc38144f90 0x7fbc380b3160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:59:40.991 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:40.990+0000 7fbc3dc00640 1 -- 172.21.15.32:0/4111314630 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbc38158720 con 0x7fbc38165160 2024-08-03T20:59:40.992 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:40.990+0000 7fbc3dc00640 1 --2- 172.21.15.32:0/4111314630 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbc38165160 0x7fbc381593f0 secure :-1 s=READY pgs=1965 cs=0 l=1 rev1=1 crypto rx=0x7fbc30009cd0 tx=0x7fbc300039f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:59:40.992 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:40.990+0000 7fbc34c00640 1 -- 172.21.15.32:0/4111314630 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc30031b90 con 0x7fbc38165160 2024-08-03T20:59:40.992 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:40.990+0000 7fbc34c00640 1 -- 172.21.15.32:0/4111314630 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fbc30031e20 con 0x7fbc38165160 2024-08-03T20:59:40.992 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:40.990+0000 7fbc34c00640 1 -- 172.21.15.32:0/4111314630 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc300332f0 con 0x7fbc38165160 2024-08-03T20:59:40.992 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:40.990+0000 7fbc3fc00640 1 -- 172.21.15.32:0/4111314630 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbc38158a00 con 0x7fbc38165160 2024-08-03T20:59:40.993 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:40.991+0000 7fbc3fc00640 1 -- 172.21.15.32:0/4111314630 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbc380b3e60 con 0x7fbc38165160 2024-08-03T20:59:40.993 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:40.992+0000 7fbc3fc00640 1 -- 172.21.15.32:0/4111314630 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbc38073e60 con 0x7fbc38165160 2024-08-03T20:59:40.993 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:40.993+0000 7fbc34c00640 1 -- 172.21.15.32:0/4111314630 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7fbc30033490 con 0x7fbc38165160 2024-08-03T20:59:40.994 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:40.994+0000 7fbc34c00640 1 --2- 172.21.15.32:0/4111314630 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fbc18077010 0x7fbc18079520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:59:40.995 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:40.994+0000 7fbc34c00640 1 -- 172.21.15.32:0/4111314630 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(120..120 src has 1..120) ==== 8864+0+0 (secure 0 0 0) 0x7fbc300ba9c0 con 0x7fbc38165160 2024-08-03T20:59:40.995 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:40.995+0000 7fbc3d200640 1 --2- 172.21.15.32:0/4111314630 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fbc18077010 0x7fbc18079520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:59:40.995 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:40.995+0000 7fbc3d200640 1 --2- 172.21.15.32:0/4111314630 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fbc18077010 0x7fbc18079520 secure :-1 s=READY pgs=786 cs=0 l=1 rev1=1 crypto rx=0x7fbc2c000f60 tx=0x7fbc2c009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:59:40.999 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:40.999+0000 7fbc34c00640 1 -- 172.21.15.32:0/4111314630 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7fbc3003f030 con 0x7fbc38165160 2024-08-03T20:59:41.279 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:41.279+0000 7fbc3fc00640 1 -- 172.21.15.32:0/4111314630 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "mds repaired", "role": "0"} v 0) -- 0x7fbc381584b0 con 0x7fbc38165160 2024-08-03T20:59:41.724 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:41.724+0000 7fbc34c00640 1 -- 172.21.15.32:0/4111314630 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "mds repaired", "role": "0"}]=0 nothing to do: rank is not damaged v295) ==== 107+0+0 (secure 0 0 0) 0x7fbc300317b0 con 0x7fbc38165160 2024-08-03T20:59:41.724 INFO:teuthology.orchestra.run.smithi032.stderr:nothing to do: rank is not damaged 2024-08-03T20:59:41.727 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:41.726+0000 7fbc3fc00640 1 -- 172.21.15.32:0/4111314630 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fbc18077010 msgr2=0x7fbc18079520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:59:41.727 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:41.726+0000 7fbc3fc00640 1 --2- 172.21.15.32:0/4111314630 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fbc18077010 0x7fbc18079520 secure :-1 s=READY pgs=786 cs=0 l=1 rev1=1 crypto rx=0x7fbc2c000f60 tx=0x7fbc2c009040 comp rx=0 tx=0).stop 2024-08-03T20:59:41.727 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:41.726+0000 7fbc3fc00640 1 -- 172.21.15.32:0/4111314630 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbc38165160 msgr2=0x7fbc381593f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:59:41.727 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:41.726+0000 7fbc3fc00640 1 --2- 172.21.15.32:0/4111314630 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbc38165160 0x7fbc381593f0 secure :-1 s=READY pgs=1965 cs=0 l=1 rev1=1 crypto rx=0x7fbc30009cd0 tx=0x7fbc300039f0 comp rx=0 tx=0).stop 2024-08-03T20:59:41.727 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:41.726+0000 7fbc3fc00640 1 -- 172.21.15.32:0/4111314630 shutdown_connections 2024-08-03T20:59:41.727 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:41.726+0000 7fbc3fc00640 1 --2- 172.21.15.32:0/4111314630 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fbc18077010 0x7fbc18079520 unknown :-1 s=CLOSED pgs=786 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:59:41.728 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:41.726+0000 7fbc3fc00640 1 --2- 172.21.15.32:0/4111314630 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fbc38144f90 0x7fbc380b3160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:59:41.728 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:41.726+0000 7fbc3fc00640 1 --2- 172.21.15.32:0/4111314630 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbc38165160 0x7fbc381593f0 unknown :-1 s=CLOSED pgs=1965 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:59:41.728 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:41.726+0000 7fbc3fc00640 1 --2- 172.21.15.32:0/4111314630 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fbc3806e930 0x7fbc38156aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:59:41.728 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:41.726+0000 7fbc3fc00640 1 -- 172.21.15.32:0/4111314630 >> 172.21.15.32:0/4111314630 conn(0x7fbc380b1bd0 msgr2=0x7fbc380cc540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:59:41.728 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:41.726+0000 7fbc3fc00640 1 -- 172.21.15.32:0/4111314630 shutdown_connections 2024-08-03T20:59:41.728 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:41.726+0000 7fbc3fc00640 1 -- 172.21.15.32:0/4111314630 wait complete. 2024-08-03T20:59:41.753 DEBUG:teuthology.orchestra.run.smithi032:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata import - 2024-08-03T20:59:41.896 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:41.895+0000 7fa1e4540f40 1 Processor -- start 2024-08-03T20:59:41.896 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:41.895+0000 7fa1e4540f40 1 -- start start 2024-08-03T20:59:41.898 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:41.896+0000 7fa1e4540f40 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55a74a6f3360 0x55a74a711f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:59:41.898 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:41.897+0000 7fa1e4540f40 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a74a6f4ea0 0x55a74a683c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:59:41.898 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:41.897+0000 7fa1e4540f40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55a74a70a330 0x55a74a684150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:59:41.898 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:41.897+0000 7fa1e4540f40 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55a74a58ae30 con 0x55a74a6f4ea0 2024-08-03T20:59:41.899 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:41.897+0000 7fa1e4540f40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x55a74a5e8bf0 con 0x55a74a70a330 2024-08-03T20:59:41.899 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:41.897+0000 7fa1e4540f40 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x55a74a6f4440 con 0x55a74a6f3360 2024-08-03T20:59:41.899 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:41.897+0000 7fa1dcc00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a74a6f4ea0 0x55a74a683c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:59:41.899 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:41.897+0000 7fa1dd600640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55a74a6f3360 0x55a74a711f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:59:41.899 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:41.897+0000 7fa1dcc00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a74a6f4ea0 0x55a74a683c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:33126/0 (socket says 172.21.15.32:33126) 2024-08-03T20:59:41.899 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:41.897+0000 7fa1dcc00640 1 -- 172.21.15.32:0/2948215424 learned_addr learned my addr 172.21.15.32:0/2948215424 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:59:41.899 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:41.897+0000 7fa1de000640 1 --2- 172.21.15.32:0/2948215424 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55a74a70a330 0x55a74a684150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:59:41.899 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:41.897+0000 7fa1dcc00640 1 -- 172.21.15.32:0/2948215424 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55a74a6f3360 msgr2=0x55a74a711f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:59:41.899 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:41.897+0000 7fa1dcc00640 1 --2- 172.21.15.32:0/2948215424 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55a74a6f3360 0x55a74a711f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:59:41.899 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:41.897+0000 7fa1dcc00640 1 -- 172.21.15.32:0/2948215424 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55a74a70a330 msgr2=0x55a74a684150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:59:41.899 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:41.897+0000 7fa1dcc00640 1 --2- 172.21.15.32:0/2948215424 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55a74a70a330 0x55a74a684150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:59:41.899 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:41.897+0000 7fa1dcc00640 1 -- 172.21.15.32:0/2948215424 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a74a6ef6b0 con 0x55a74a6f4ea0 2024-08-03T20:59:41.900 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:41.897+0000 7fa1de000640 1 --2- 172.21.15.32:0/2948215424 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55a74a70a330 0x55a74a684150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-03T20:59:41.900 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:41.898+0000 7fa1dcc00640 1 --2- 172.21.15.32:0/2948215424 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a74a6f4ea0 0x55a74a683c10 secure :-1 s=READY pgs=1967 cs=0 l=1 rev1=1 crypto rx=0x7fa1cc025e80 tx=0x7fa1cc002fb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a1a559fcd6daeb7c server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:59:41.900 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:41.898+0000 7fa1d7400640 1 -- 172.21.15.32:0/2948215424 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1cc04d8e0 con 0x55a74a6f4ea0 2024-08-03T20:59:41.900 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:41.898+0000 7fa1d7400640 1 -- 172.21.15.32:0/2948215424 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fa1cc04dd90 con 0x55a74a6f4ea0 2024-08-03T20:59:41.900 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:41.898+0000 7fa1d7400640 1 -- 172.21.15.32:0/2948215424 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1cc0566d0 con 0x55a74a6f4ea0 2024-08-03T20:59:41.901 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:41.899+0000 7fa1e4540f40 1 -- 172.21.15.32:0/2948215424 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a74a6f4ea0 msgr2=0x55a74a683c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:59:41.901 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:41.899+0000 7fa1e4540f40 1 --2- 172.21.15.32:0/2948215424 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a74a6f4ea0 0x55a74a683c10 secure :-1 s=READY pgs=1967 cs=0 l=1 rev1=1 crypto rx=0x7fa1cc025e80 tx=0x7fa1cc002fb0 comp rx=0 tx=0).stop 2024-08-03T20:59:41.901 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:41.899+0000 7fa1e4540f40 1 -- 172.21.15.32:0/2948215424 shutdown_connections 2024-08-03T20:59:41.901 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:41.899+0000 7fa1e4540f40 1 --2- 172.21.15.32:0/2948215424 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55a74a70a330 0x55a74a684150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:59:41.901 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:41.899+0000 7fa1e4540f40 1 --2- 172.21.15.32:0/2948215424 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a74a6f4ea0 0x55a74a683c10 unknown :-1 s=CLOSED pgs=1967 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:59:41.901 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:41.899+0000 7fa1e4540f40 1 --2- 172.21.15.32:0/2948215424 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55a74a6f3360 0x55a74a711f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:59:41.901 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:41.899+0000 7fa1e4540f40 1 -- 172.21.15.32:0/2948215424 >> 172.21.15.32:0/2948215424 conn(0x55a74a55cac0 msgr2=0x55a74a70d090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:59:41.901 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:41.899+0000 7fa1e4540f40 1 -- 172.21.15.32:0/2948215424 shutdown_connections 2024-08-03T20:59:41.902 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:41.899+0000 7fa1e4540f40 1 -- 172.21.15.32:0/2948215424 wait complete. 2024-08-03T20:59:41.902 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:41.900+0000 7fa1e4540f40 1 Processor -- start 2024-08-03T20:59:41.902 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:41.900+0000 7fa1e4540f40 1 -- start start 2024-08-03T20:59:41.903 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:41.901+0000 7fa1e4540f40 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a74a6f3360 0x55a74a7a9d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:59:41.904 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:41.901+0000 7fa1e4540f40 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55a74a6f4ea0 0x55a74a6eb640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:59:41.904 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:41.901+0000 7fa1e4540f40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55a74a70a330 0x55a74a63ada0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:59:41.904 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:41.901+0000 7fa1e4540f40 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55a74a70bf40 con 0x55a74a6f3360 2024-08-03T20:59:41.904 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:41.901+0000 7fa1e4540f40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x55a74a685da0 con 0x55a74a70a330 2024-08-03T20:59:41.904 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:41.901+0000 7fa1e4540f40 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x55a74a63bb30 con 0x55a74a6f4ea0 2024-08-03T20:59:41.904 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:41.902+0000 7fa1dd600640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a74a6f3360 0x55a74a7a9d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:59:41.904 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:41.902+0000 7fa1dd600640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a74a6f3360 0x55a74a7a9d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:33138/0 (socket says 172.21.15.32:33138) 2024-08-03T20:59:41.904 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:41.902+0000 7fa1dd600640 1 -- 172.21.15.32:0/2758706093 learned_addr learned my addr 172.21.15.32:0/2758706093 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:59:41.904 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:41.902+0000 7fa1dcc00640 1 --2- 172.21.15.32:0/2758706093 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55a74a6f4ea0 0x55a74a6eb640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:59:41.904 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:41.902+0000 7fa1de000640 1 --2- 172.21.15.32:0/2758706093 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55a74a70a330 0x55a74a63ada0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:59:41.904 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:41.902+0000 7fa1dd600640 1 -- 172.21.15.32:0/2758706093 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55a74a6f4ea0 msgr2=0x55a74a6eb640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:59:41.904 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:41.902+0000 7fa1dd600640 1 --2- 172.21.15.32:0/2758706093 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55a74a6f4ea0 0x55a74a6eb640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:59:41.905 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:41.902+0000 7fa1dd600640 1 -- 172.21.15.32:0/2758706093 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55a74a70a330 msgr2=0x55a74a63ada0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:59:41.905 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:41.902+0000 7fa1dd600640 1 --2- 172.21.15.32:0/2758706093 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55a74a70a330 0x55a74a63ada0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:59:41.905 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:41.902+0000 7fa1dd600640 1 -- 172.21.15.32:0/2758706093 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a74a682820 con 0x55a74a6f3360 2024-08-03T20:59:41.905 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:41.903+0000 7fa1dd600640 1 --2- 172.21.15.32:0/2758706093 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a74a6f3360 0x55a74a7a9d00 secure :-1 s=READY pgs=1968 cs=0 l=1 rev1=1 crypto rx=0x7fa1d0002cc0 tx=0x7fa1d000c620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:59:41.905 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:41.903+0000 7fa1d6000640 1 -- 172.21.15.32:0/2758706093 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1d0015020 con 0x55a74a6f3360 2024-08-03T20:59:41.905 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:41.903+0000 7fa1d6000640 1 -- 172.21.15.32:0/2758706093 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fa1d0011450 con 0x55a74a6f3360 2024-08-03T20:59:41.905 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:41.903+0000 7fa1d6000640 1 -- 172.21.15.32:0/2758706093 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1d0011840 con 0x55a74a6f3360 2024-08-03T20:59:41.905 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:41.903+0000 7fa1e4540f40 1 -- 172.21.15.32:0/2758706093 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x55a74a682b00 con 0x55a74a6f3360 2024-08-03T20:59:41.905 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:41.903+0000 7fa1e4540f40 1 -- 172.21.15.32:0/2758706093 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a74a7a9800 con 0x55a74a6f3360 2024-08-03T20:59:41.906 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:41.905+0000 7fa1d6000640 1 -- 172.21.15.32:0/2758706093 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7fa1d0018020 con 0x55a74a6f3360 2024-08-03T20:59:41.906 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:41.906+0000 7fa1d6000640 1 --2- 172.21.15.32:0/2758706093 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fa1c00759f0 0x7fa1c0077f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:59:41.907 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:41.906+0000 7fa1d6000640 1 -- 172.21.15.32:0/2758706093 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(120..120 src has 1..120) ==== 8864+0+0 (secure 0 0 0) 0x7fa1d00983a0 con 0x55a74a6f3360 2024-08-03T20:59:41.907 INFO:teuthology.orchestra.run.smithi032.stdout:Importing pool 2024-08-03T20:59:41.907 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:41.907+0000 7fa1e4540f40 1 --2- 172.21.15.32:0/2758706093 >> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] conn(0x55a74a63b2e0 0x55a74a70a8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:59:41.908 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:41.907+0000 7fa1e4540f40 1 -- 172.21.15.32:0/2758706093 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:1 12.10 12:09401a81:::601.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120) -- 0x55a74a63b6f0 con 0x55a74a63b2e0 2024-08-03T20:59:41.908 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:41.907+0000 7fa1dcc00640 1 --2- 172.21.15.32:0/2758706093 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fa1c00759f0 0x7fa1c0077f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:59:41.908 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:41.907+0000 7fa1de000640 1 --2- 172.21.15.32:0/2758706093 >> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] conn(0x55a74a63b2e0 0x55a74a70a8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:59:41.908 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:41.908+0000 7fa1de000640 1 --2- 172.21.15.32:0/2758706093 >> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] conn(0x55a74a63b2e0 0x55a74a70a8d0 crc :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:59:41.908 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:41.908+0000 7fa1dcc00640 1 --2- 172.21.15.32:0/2758706093 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fa1c00759f0 0x7fa1c0077f00 secure :-1 s=READY pgs=788 cs=0 l=1 rev1=1 crypto rx=0x7fa1cc062000 tx=0x7fa1cc004940 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:59:41.910 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:41.910+0000 7fa1de000640 1 -- 172.21.15.32:0/2758706093 <== osd.0 v2:172.21.15.32:6816/3066156116 1 ==== osd_op_reply(1 601.00000000 [create] v120'1065 uv1038 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fa1d800b040 con 0x55a74a63b2e0 2024-08-03T20:59:41.910 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:41.910+0000 7fa1e4540f40 1 -- 172.21.15.32:0/2758706093 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:2 12.10 12:09401a81:::601.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e120) -- 0x55a74a70ae10 con 0x55a74a63b2e0 2024-08-03T20:59:41.913 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:41.912+0000 7fa1de000640 1 -- 172.21.15.32:0/2758706093 <== osd.0 v2:172.21.15.32:6816/3066156116 2 ==== osd_op_reply(2 601.00000000 [delete] v120'1066 uv1066 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fa1d800b040 con 0x55a74a63b2e0 2024-08-03T20:59:41.913 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:41.912+0000 7fa1e4540f40 1 -- 172.21.15.32:0/2758706093 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:3 12.10 12:09401a81:::601.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120) -- 0x55a74a7b58a0 con 0x55a74a63b2e0 2024-08-03T20:59:41.914 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::601.00000000:head# 2024-08-03T20:59:41.915 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:41.914+0000 7fa1de000640 1 -- 172.21.15.32:0/2758706093 <== osd.0 v2:172.21.15.32:6816/3066156116 3 ==== osd_op_reply(3 601.00000000 [create] v120'1067 uv1067 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fa1d800b040 con 0x55a74a63b2e0 2024-08-03T20:59:41.915 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:41.914+0000 7fa1e4540f40 1 -- 172.21.15.32:0/2758706093 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:4 12.10 12:09401a81:::601.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120) -- 0x55a74a7b6c50 con 0x55a74a63b2e0 2024-08-03T20:59:41.916 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:41.916+0000 7fa1de000640 1 -- 172.21.15.32:0/2758706093 <== osd.0 v2:172.21.15.32:6816/3066156116 4 ==== osd_op_reply(4 601.00000000 [setxattr (30)] v120'1068 uv1068 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fa1d800b040 con 0x55a74a63b2e0 2024-08-03T20:59:41.916 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:41.916+0000 7fa1e4540f40 1 -- 172.21.15.32:0/2758706093 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:5 12.10 12:09401a81:::601.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120) -- 0x55a74a7b6ff0 con 0x55a74a63b2e0 2024-08-03T20:59:41.918 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:41.917+0000 7fa1de000640 1 -- 172.21.15.32:0/2758706093 <== osd.0 v2:172.21.15.32:6816/3066156116 5 ==== osd_op_reply(5 601.00000000 [setxattr (62)] v120'1069 uv1069 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fa1d800b040 con 0x55a74a63b2e0 2024-08-03T20:59:41.918 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:41.917+0000 7fa1e4540f40 1 -- 172.21.15.32:0/2758706093 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:6 12.10 12:09401a81:::601.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120) -- 0x55a74a7b5c40 con 0x55a74a63b2e0 2024-08-03T20:59:41.920 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:41.919+0000 7fa1de000640 1 -- 172.21.15.32:0/2758706093 <== osd.0 v2:172.21.15.32:6816/3066156116 6 ==== osd_op_reply(6 601.00000000 [omap-set-header] v120'1070 uv1070 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fa1d800b040 con 0x55a74a63b2e0 2024-08-03T20:59:41.920 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:41.919+0000 7fa1e4540f40 1 -- 172.21.15.32:0/2758706093 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:7 12.10 12:0bd6d154:::602.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120) -- 0x55a74a7b6030 con 0x55a74a63b2e0 2024-08-03T20:59:41.922 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:41.921+0000 7fa1de000640 1 -- 172.21.15.32:0/2758706093 <== osd.0 v2:172.21.15.32:6816/3066156116 7 ==== osd_op_reply(7 602.00000000 [create] v120'1071 uv1044 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fa1d800b040 con 0x55a74a63b2e0 2024-08-03T20:59:41.922 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:41.921+0000 7fa1e4540f40 1 -- 172.21.15.32:0/2758706093 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:8 12.10 12:0bd6d154:::602.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e120) -- 0x55a74a7b63d0 con 0x55a74a63b2e0 2024-08-03T20:59:41.923 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:41.923+0000 7fa1de000640 1 -- 172.21.15.32:0/2758706093 <== osd.0 v2:172.21.15.32:6816/3066156116 8 ==== osd_op_reply(8 602.00000000 [delete] v120'1072 uv1072 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fa1d800b040 con 0x55a74a63b2e0 2024-08-03T20:59:41.923 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:41.923+0000 7fa1e4540f40 1 -- 172.21.15.32:0/2758706093 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:9 12.10 12:0bd6d154:::602.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120) -- 0x55a74a7b63d0 con 0x55a74a63b2e0 2024-08-03T20:59:41.925 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:41.925+0000 7fa1de000640 1 -- 172.21.15.32:0/2758706093 <== osd.0 v2:172.21.15.32:6816/3066156116 9 ==== osd_op_reply(9 602.00000000 [create] v120'1073 uv1073 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fa1d800b040 con 0x55a74a63b2e0 2024-08-03T20:59:41.925 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::602.00000000:head# 2024-08-03T20:59:41.926 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:41.925+0000 7fa1e4540f40 1 -- 172.21.15.32:0/2758706093 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:10 12.10 12:0bd6d154:::602.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120) -- 0x55a74a7b63d0 con 0x55a74a63b2e0 2024-08-03T20:59:41.927 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:41.926+0000 7fa1de000640 1 -- 172.21.15.32:0/2758706093 <== osd.0 v2:172.21.15.32:6816/3066156116 10 ==== osd_op_reply(10 602.00000000 [setxattr (30)] v120'1074 uv1074 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fa1d800b040 con 0x55a74a63b2e0 2024-08-03T20:59:41.927 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:41.927+0000 7fa1e4540f40 1 -- 172.21.15.32:0/2758706093 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:11 12.10 12:0bd6d154:::602.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120) -- 0x55a74a7b63d0 con 0x55a74a63b2e0 2024-08-03T20:59:41.929 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:41.928+0000 7fa1de000640 1 -- 172.21.15.32:0/2758706093 <== osd.0 v2:172.21.15.32:6816/3066156116 11 ==== osd_op_reply(11 602.00000000 [setxattr (62)] v120'1075 uv1075 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fa1d800b040 con 0x55a74a63b2e0 2024-08-03T20:59:41.929 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:41.928+0000 7fa1e4540f40 1 -- 172.21.15.32:0/2758706093 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:12 12.10 12:0bd6d154:::602.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120) -- 0x55a74a7b63d0 con 0x55a74a63b2e0 2024-08-03T20:59:41.930 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:41.930+0000 7fa1de000640 1 -- 172.21.15.32:0/2758706093 <== osd.0 v2:172.21.15.32:6816/3066156116 12 ==== osd_op_reply(12 602.00000000 [omap-set-header] v120'1076 uv1076 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fa1d800b040 con 0x55a74a63b2e0 2024-08-03T20:59:41.930 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:41.930+0000 7fa1e4540f40 1 -- 172.21.15.32:0/2758706093 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:13 12.10 12:0cf6ad0f:::10000000000.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120) -- 0x55a74a7b63d0 con 0x55a74a63b2e0 2024-08-03T20:59:41.932 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:41.932+0000 7fa1de000640 1 -- 172.21.15.32:0/2758706093 <== osd.0 v2:172.21.15.32:6816/3066156116 13 ==== osd_op_reply(13 10000000000.00000000 [create] v120'1077 uv1064 ondisk = -17 ((17) File exists)) ==== 164+0+0 (crc 0 0 0) 0x7fa1d800b040 con 0x55a74a63b2e0 2024-08-03T20:59:41.932 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:41.932+0000 7fa1e4540f40 1 -- 172.21.15.32:0/2758706093 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:14 12.10 12:0cf6ad0f:::10000000000.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e120) -- 0x55a74a7b63d0 con 0x55a74a63b2e0 2024-08-03T20:59:41.934 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:41.934+0000 7fa1de000640 1 -- 172.21.15.32:0/2758706093 <== osd.0 v2:172.21.15.32:6816/3066156116 14 ==== osd_op_reply(14 10000000000.00000000 [delete] v120'1078 uv1078 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7fa1d800b040 con 0x55a74a63b2e0 2024-08-03T20:59:41.934 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:41.934+0000 7fa1e4540f40 1 -- 172.21.15.32:0/2758706093 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:15 12.10 12:0cf6ad0f:::10000000000.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120) -- 0x55a74a7b63d0 con 0x55a74a63b2e0 2024-08-03T20:59:41.936 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:41.935+0000 7fa1de000640 1 -- 172.21.15.32:0/2758706093 <== osd.0 v2:172.21.15.32:6816/3066156116 15 ==== osd_op_reply(15 10000000000.00000000 [create] v120'1079 uv1079 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7fa1d800b040 con 0x55a74a63b2e0 2024-08-03T20:59:41.936 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::10000000000.00000000:head# 2024-08-03T20:59:41.936 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:41.935+0000 7fa1e4540f40 1 -- 172.21.15.32:0/2758706093 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:16 12.10 12:0cf6ad0f:::10000000000.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120) -- 0x55a74a7b63d0 con 0x55a74a63b2e0 2024-08-03T20:59:41.937 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:41.937+0000 7fa1de000640 1 -- 172.21.15.32:0/2758706093 <== osd.0 v2:172.21.15.32:6816/3066156116 16 ==== osd_op_reply(16 10000000000.00000000 [setxattr (30)] v120'1080 uv1080 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7fa1d800b040 con 0x55a74a63b2e0 2024-08-03T20:59:41.937 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:41.937+0000 7fa1e4540f40 1 -- 172.21.15.32:0/2758706093 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:17 12.10 12:0cf6ad0f:::10000000000.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120) -- 0x55a74a7b63d0 con 0x55a74a63b2e0 2024-08-03T20:59:41.939 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:41.939+0000 7fa1de000640 1 -- 172.21.15.32:0/2758706093 <== osd.0 v2:172.21.15.32:6816/3066156116 17 ==== osd_op_reply(17 10000000000.00000000 [setxattr (62)] v120'1081 uv1081 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7fa1d800b040 con 0x55a74a63b2e0 2024-08-03T20:59:41.939 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:41.939+0000 7fa1e4540f40 1 -- 172.21.15.32:0/2758706093 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:18 12.10 12:0cf6ad0f:::10000000000.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120) -- 0x55a74a7b63d0 con 0x55a74a63b2e0 2024-08-03T20:59:41.941 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:41.941+0000 7fa1de000640 1 -- 172.21.15.32:0/2758706093 <== osd.0 v2:172.21.15.32:6816/3066156116 18 ==== osd_op_reply(18 10000000000.00000000 [omap-set-header] v120'1082 uv1082 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7fa1d800b040 con 0x55a74a63b2e0 2024-08-03T20:59:41.941 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:41.941+0000 7fa1e4540f40 1 -- 172.21.15.32:0/2758706093 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:19 12.10 12:0cf6ad0f:::10000000000.00000000:head [omap-set-vals in=593b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120) -- 0x55a74a7b7390 con 0x55a74a63b2e0 2024-08-03T20:59:41.942 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:41.942+0000 7fa1de000640 1 -- 172.21.15.32:0/2758706093 <== osd.0 v2:172.21.15.32:6816/3066156116 19 ==== osd_op_reply(19 10000000000.00000000 [omap-set-vals] v120'1083 uv1083 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7fa1d800b040 con 0x55a74a63b2e0 2024-08-03T20:59:41.943 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:41.942+0000 7fa1e4540f40 1 -- 172.21.15.32:0/2758706093 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:20 12.10 12:0d82a743:::600.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120) -- 0x55a74a7b7390 con 0x55a74a63b2e0 2024-08-03T20:59:41.944 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:41.944+0000 7fa1de000640 1 -- 172.21.15.32:0/2758706093 <== osd.0 v2:172.21.15.32:6816/3066156116 20 ==== osd_op_reply(20 600.00000000 [create] v120'1084 uv1057 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fa1d800b040 con 0x55a74a63b2e0 2024-08-03T20:59:41.945 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:41.944+0000 7fa1e4540f40 1 -- 172.21.15.32:0/2758706093 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:21 12.10 12:0d82a743:::600.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e120) -- 0x55a74a7b7390 con 0x55a74a63b2e0 2024-08-03T20:59:41.946 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:41.946+0000 7fa1de000640 1 -- 172.21.15.32:0/2758706093 <== osd.0 v2:172.21.15.32:6816/3066156116 21 ==== osd_op_reply(21 600.00000000 [delete] v120'1085 uv1085 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fa1d800b040 con 0x55a74a63b2e0 2024-08-03T20:59:41.946 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:41.946+0000 7fa1e4540f40 1 -- 172.21.15.32:0/2758706093 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:22 12.10 12:0d82a743:::600.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120) -- 0x55a74a7b7390 con 0x55a74a63b2e0 2024-08-03T20:59:41.948 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:41.948+0000 7fa1de000640 1 -- 172.21.15.32:0/2758706093 <== osd.0 v2:172.21.15.32:6816/3066156116 22 ==== osd_op_reply(22 600.00000000 [create] v120'1086 uv1086 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fa1d800b040 con 0x55a74a63b2e0 2024-08-03T20:59:41.948 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::600.00000000:head# 2024-08-03T20:59:41.948 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:41.948+0000 7fa1e4540f40 1 -- 172.21.15.32:0/2758706093 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:23 12.10 12:0d82a743:::600.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120) -- 0x55a74a7b83a0 con 0x55a74a63b2e0 2024-08-03T20:59:41.949 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:41.949+0000 7fa1de000640 1 -- 172.21.15.32:0/2758706093 <== osd.0 v2:172.21.15.32:6816/3066156116 23 ==== osd_op_reply(23 600.00000000 [setxattr (30)] v120'1087 uv1087 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fa1d800b040 con 0x55a74a63b2e0 2024-08-03T20:59:41.950 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:41.949+0000 7fa1e4540f40 1 -- 172.21.15.32:0/2758706093 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:24 12.10 12:0d82a743:::600.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120) -- 0x55a74a7b83a0 con 0x55a74a63b2e0 2024-08-03T20:59:41.956 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:41.951+0000 7fa1de000640 1 -- 172.21.15.32:0/2758706093 <== osd.0 v2:172.21.15.32:6816/3066156116 24 ==== osd_op_reply(24 600.00000000 [setxattr (62)] v120'1088 uv1088 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fa1d800b040 con 0x55a74a63b2e0 2024-08-03T20:59:41.956 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:41.951+0000 7fa1e4540f40 1 -- 172.21.15.32:0/2758706093 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:25 12.10 12:0d82a743:::600.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120) -- 0x55a74a7b7390 con 0x55a74a63b2e0 2024-08-03T20:59:41.956 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:41.953+0000 7fa1de000640 1 -- 172.21.15.32:0/2758706093 <== osd.0 v2:172.21.15.32:6816/3066156116 25 ==== osd_op_reply(25 600.00000000 [omap-set-header] v120'1089 uv1089 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fa1d800b040 con 0x55a74a63b2e0 2024-08-03T20:59:41.956 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:41.953+0000 7fa1e4540f40 1 -- 172.21.15.32:0/2758706093 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:26 12.10 12:0d89b25e:::603.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120) -- 0x55a74a7b7390 con 0x55a74a63b2e0 2024-08-03T20:59:41.956 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:41.954+0000 7fa1de000640 1 -- 172.21.15.32:0/2758706093 <== osd.0 v2:172.21.15.32:6816/3066156116 26 ==== osd_op_reply(26 603.00000000 [create] v120'1090 uv1063 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fa1d800b040 con 0x55a74a63b2e0 2024-08-03T20:59:41.956 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:41.954+0000 7fa1e4540f40 1 -- 172.21.15.32:0/2758706093 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:27 12.10 12:0d89b25e:::603.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e120) -- 0x55a74a7b7390 con 0x55a74a63b2e0 2024-08-03T20:59:41.957 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:41.956+0000 7fa1de000640 1 -- 172.21.15.32:0/2758706093 <== osd.0 v2:172.21.15.32:6816/3066156116 27 ==== osd_op_reply(27 603.00000000 [delete] v120'1091 uv1091 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fa1d800b040 con 0x55a74a63b2e0 2024-08-03T20:59:41.957 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:41.956+0000 7fa1e4540f40 1 -- 172.21.15.32:0/2758706093 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:28 12.10 12:0d89b25e:::603.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120) -- 0x55a74a7b7390 con 0x55a74a63b2e0 2024-08-03T20:59:41.958 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::603.00000000:head# 2024-08-03T20:59:41.958 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:41.958+0000 7fa1de000640 1 -- 172.21.15.32:0/2758706093 <== osd.0 v2:172.21.15.32:6816/3066156116 28 ==== osd_op_reply(28 603.00000000 [create] v120'1092 uv1092 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fa1d800b040 con 0x55a74a63b2e0 2024-08-03T20:59:41.959 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:41.958+0000 7fa1e4540f40 1 -- 172.21.15.32:0/2758706093 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:29 12.10 12:0d89b25e:::603.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120) -- 0x55a74a7b83a0 con 0x55a74a63b2e0 2024-08-03T20:59:41.960 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:41.959+0000 7fa1de000640 1 -- 172.21.15.32:0/2758706093 <== osd.0 v2:172.21.15.32:6816/3066156116 29 ==== osd_op_reply(29 603.00000000 [setxattr (30)] v120'1093 uv1093 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fa1d800b040 con 0x55a74a63b2e0 2024-08-03T20:59:41.960 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:41.959+0000 7fa1e4540f40 1 -- 172.21.15.32:0/2758706093 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:30 12.10 12:0d89b25e:::603.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120) -- 0x55a74a7b83a0 con 0x55a74a63b2e0 2024-08-03T20:59:41.961 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:41.961+0000 7fa1de000640 1 -- 172.21.15.32:0/2758706093 <== osd.0 v2:172.21.15.32:6816/3066156116 30 ==== osd_op_reply(30 603.00000000 [setxattr (62)] v120'1094 uv1094 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fa1d800b040 con 0x55a74a63b2e0 2024-08-03T20:59:41.962 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:41.961+0000 7fa1e4540f40 1 -- 172.21.15.32:0/2758706093 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:31 12.10 12:0d89b25e:::603.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120) -- 0x55a74a7b7390 con 0x55a74a63b2e0 2024-08-03T20:59:41.963 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:41.963+0000 7fa1de000640 1 -- 172.21.15.32:0/2758706093 <== osd.0 v2:172.21.15.32:6816/3066156116 31 ==== osd_op_reply(31 603.00000000 [omap-set-header] v120'1095 uv1095 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fa1d800b040 con 0x55a74a63b2e0 2024-08-03T20:59:41.963 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:41.963+0000 7fa1e4540f40 1 --2- 172.21.15.32:0/2758706093 >> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] conn(0x55a74a7b78a0 0x55a74a7b9cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:59:41.963 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:41.963+0000 7fa1e4540f40 1 -- 172.21.15.32:0/2758706093 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:32 12.14 12:287034c4:::1.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120) -- 0x55a74a7ba200 con 0x55a74a7b78a0 2024-08-03T20:59:41.964 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:41.963+0000 7fa1dd600640 1 --2- 172.21.15.32:0/2758706093 >> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] conn(0x55a74a7b78a0 0x55a74a7b9cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:59:41.964 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:41.964+0000 7fa1dd600640 1 --2- 172.21.15.32:0/2758706093 >> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] conn(0x55a74a7b78a0 0x55a74a7b9cc0 crc :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:59:41.967 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:41.966+0000 7fa1dd600640 1 -- 172.21.15.32:0/2758706093 <== osd.5 v2:172.21.15.118:6810/2724891770 1 ==== osd_op_reply(32 1.00000000.inode [create] v120'777 uv756 ondisk = -17 ((17) File exists)) ==== 160+0+0 (crc 0 0 0) 0x7fa1d0021070 con 0x55a74a7b78a0 2024-08-03T20:59:41.967 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:41.966+0000 7fa1e4540f40 1 -- 172.21.15.32:0/2758706093 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:33 12.14 12:287034c4:::1.00000000.inode:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e120) -- 0x55a74a7ba5a0 con 0x55a74a7b78a0 2024-08-03T20:59:41.968 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:41.968+0000 7fa1dd600640 1 -- 172.21.15.32:0/2758706093 <== osd.5 v2:172.21.15.118:6810/2724891770 2 ==== osd_op_reply(33 1.00000000.inode [delete] v120'778 uv778 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7fa1d0021070 con 0x55a74a7b78a0 2024-08-03T20:59:41.968 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:41.968+0000 7fa1e4540f40 1 -- 172.21.15.32:0/2758706093 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:34 12.14 12:287034c4:::1.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120) -- 0x55a74a7ba940 con 0x55a74a7b78a0 2024-08-03T20:59:41.970 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:41.969+0000 7fa1dd600640 1 -- 172.21.15.32:0/2758706093 <== osd.5 v2:172.21.15.118:6810/2724891770 3 ==== osd_op_reply(34 1.00000000.inode [create] v120'779 uv779 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7fa1d0021070 con 0x55a74a7b78a0 2024-08-03T20:59:41.970 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::1.00000000.inode:head# 2024-08-03T20:59:41.970 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:41.969+0000 7fa1e4540f40 1 -- 172.21.15.32:0/2758706093 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:35 12.14 12:287034c4:::1.00000000.inode:head [write 0~582 in=582b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120) -- 0x55a74a7bace0 con 0x55a74a7b78a0 2024-08-03T20:59:41.972 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:41.971+0000 7fa1dd600640 1 -- 172.21.15.32:0/2758706093 <== osd.5 v2:172.21.15.118:6810/2724891770 4 ==== osd_op_reply(35 1.00000000.inode [write 0~582] v120'780 uv780 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7fa1d0021070 con 0x55a74a7b78a0 2024-08-03T20:59:41.972 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:41.971+0000 7fa1e4540f40 1 -- 172.21.15.32:0/2758706093 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:36 12.14 12:287034c4:::1.00000000.inode:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120) -- 0x55a74a7bb080 con 0x55a74a7b78a0 2024-08-03T20:59:41.973 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:41.973+0000 7fa1dd600640 1 -- 172.21.15.32:0/2758706093 <== osd.5 v2:172.21.15.118:6810/2724891770 5 ==== osd_op_reply(36 1.00000000.inode [omap-set-header] v120'781 uv781 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7fa1d0021070 con 0x55a74a7b78a0 2024-08-03T20:59:41.973 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:41.973+0000 7fa1e4540f40 1 -- 172.21.15.32:0/2758706093 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:37 12.14 12:292cf221:::200.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120) -- 0x55a74a7bb420 con 0x55a74a7b78a0 2024-08-03T20:59:41.974 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:41.974+0000 7fa1dd600640 1 -- 172.21.15.32:0/2758706093 <== osd.5 v2:172.21.15.118:6810/2724891770 6 ==== osd_op_reply(37 200.00000000 [create] v120'782 uv775 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fa1d0021070 con 0x55a74a7b78a0 2024-08-03T20:59:41.975 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:41.974+0000 7fa1e4540f40 1 -- 172.21.15.32:0/2758706093 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:38 12.14 12:292cf221:::200.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e120) -- 0x55a74a7bb7c0 con 0x55a74a7b78a0 2024-08-03T20:59:41.976 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:41.976+0000 7fa1dd600640 1 -- 172.21.15.32:0/2758706093 <== osd.5 v2:172.21.15.118:6810/2724891770 7 ==== osd_op_reply(38 200.00000000 [delete] v120'783 uv783 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fa1d0021070 con 0x55a74a7b78a0 2024-08-03T20:59:41.977 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:41.976+0000 7fa1e4540f40 1 -- 172.21.15.32:0/2758706093 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:39 12.14 12:292cf221:::200.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120) -- 0x55a74a7bbb60 con 0x55a74a7b78a0 2024-08-03T20:59:41.978 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:41.978+0000 7fa1dd600640 1 -- 172.21.15.32:0/2758706093 <== osd.5 v2:172.21.15.118:6810/2724891770 8 ==== osd_op_reply(39 200.00000000 [create] v120'784 uv784 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fa1d0021070 con 0x55a74a7b78a0 2024-08-03T20:59:41.978 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::200.00000000:head# 2024-08-03T20:59:41.978 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:41.978+0000 7fa1e4540f40 1 -- 172.21.15.32:0/2758706093 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:40 12.14 12:292cf221:::200.00000000:head [write 0~90 in=90b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120) -- 0x55a74a680240 con 0x55a74a7b78a0 2024-08-03T20:59:41.980 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:41.979+0000 7fa1dd600640 1 -- 172.21.15.32:0/2758706093 <== osd.5 v2:172.21.15.118:6810/2724891770 9 ==== osd_op_reply(40 200.00000000 [write 0~90] v120'785 uv785 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fa1d0021070 con 0x55a74a7b78a0 2024-08-03T20:59:41.980 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:41.979+0000 7fa1e4540f40 1 -- 172.21.15.32:0/2758706093 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:41 12.14 12:292cf221:::200.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120) -- 0x55a74a680240 con 0x55a74a7b78a0 2024-08-03T20:59:41.981 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:41.981+0000 7fa1dd600640 1 -- 172.21.15.32:0/2758706093 <== osd.5 v2:172.21.15.118:6810/2724891770 10 ==== osd_op_reply(41 200.00000000 [omap-set-header] v120'786 uv786 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fa1d0021070 con 0x55a74a7b78a0 2024-08-03T20:59:41.981 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:41.981+0000 7fa1e4540f40 1 -- 172.21.15.32:0/2758706093 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:42 12.14 12:2e2fa760:::200.00000001:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120) -- 0x55a74a680240 con 0x55a74a7b78a0 2024-08-03T20:59:41.983 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:41.983+0000 7fa1dd600640 1 -- 172.21.15.32:0/2758706093 <== osd.5 v2:172.21.15.118:6810/2724891770 11 ==== osd_op_reply(42 200.00000001 [create] v120'787 uv776 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fa1d0021070 con 0x55a74a7b78a0 2024-08-03T20:59:41.983 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:41.983+0000 7fa1e4540f40 1 -- 172.21.15.32:0/2758706093 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:43 12.14 12:2e2fa760:::200.00000001:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e120) -- 0x55a74a680240 con 0x55a74a7b78a0 2024-08-03T20:59:41.985 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:41.984+0000 7fa1dd600640 1 -- 172.21.15.32:0/2758706093 <== osd.5 v2:172.21.15.118:6810/2724891770 12 ==== osd_op_reply(43 200.00000001 [delete] v120'788 uv788 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fa1d0021070 con 0x55a74a7b78a0 2024-08-03T20:59:41.985 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:41.984+0000 7fa1e4540f40 1 -- 172.21.15.32:0/2758706093 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:44 12.14 12:2e2fa760:::200.00000001:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120) -- 0x55a74a680240 con 0x55a74a7b78a0 2024-08-03T20:59:41.986 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:41.986+0000 7fa1dd600640 1 -- 172.21.15.32:0/2758706093 <== osd.5 v2:172.21.15.118:6810/2724891770 13 ==== osd_op_reply(44 200.00000001 [create] v120'789 uv789 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fa1d0021070 con 0x55a74a7b78a0 2024-08-03T20:59:41.986 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::200.00000001:head# 2024-08-03T20:59:41.986 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:41.986+0000 7fa1e4540f40 1 -- 172.21.15.32:0/2758706093 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:45 12.14 12:2e2fa760:::200.00000001:head [write 0~18118 in=18118b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120) -- 0x55a74a680240 con 0x55a74a7b78a0 2024-08-03T20:59:41.988 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:41.988+0000 7fa1dd600640 1 -- 172.21.15.32:0/2758706093 <== osd.5 v2:172.21.15.118:6810/2724891770 14 ==== osd_op_reply(45 200.00000001 [write 0~18118] v120'790 uv790 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fa1d0021070 con 0x55a74a7b78a0 2024-08-03T20:59:41.988 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:41.988+0000 7fa1e4540f40 1 -- 172.21.15.32:0/2758706093 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:46 12.14 12:2e2fa760:::200.00000001:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120) -- 0x55a74a680240 con 0x55a74a7b78a0 2024-08-03T20:59:41.990 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:41.990+0000 7fa1dd600640 1 -- 172.21.15.32:0/2758706093 <== osd.5 v2:172.21.15.118:6810/2724891770 15 ==== osd_op_reply(46 200.00000001 [omap-set-header] v120'791 uv791 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fa1d0021070 con 0x55a74a7b78a0 2024-08-03T20:59:41.990 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:41.990+0000 7fa1e4540f40 1 -- 172.21.15.32:0/2758706093 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:47 12.14 12:2f55791f:::606.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120) -- 0x55a74a680240 con 0x55a74a7b78a0 2024-08-03T20:59:41.992 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:41.991+0000 7fa1dd600640 1 -- 172.21.15.32:0/2758706093 <== osd.5 v2:172.21.15.118:6810/2724891770 16 ==== osd_op_reply(47 606.00000000 [create] v120'792 uv772 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fa1d0021070 con 0x55a74a7b78a0 2024-08-03T20:59:41.992 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:41.992+0000 7fa1e4540f40 1 -- 172.21.15.32:0/2758706093 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:48 12.14 12:2f55791f:::606.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e120) -- 0x55a74a680240 con 0x55a74a7b78a0 2024-08-03T20:59:41.993 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:41.993+0000 7fa1dd600640 1 -- 172.21.15.32:0/2758706093 <== osd.5 v2:172.21.15.118:6810/2724891770 17 ==== osd_op_reply(48 606.00000000 [delete] v120'793 uv793 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fa1d0021070 con 0x55a74a7b78a0 2024-08-03T20:59:41.993 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:41.993+0000 7fa1e4540f40 1 -- 172.21.15.32:0/2758706093 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:49 12.14 12:2f55791f:::606.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120) -- 0x55a74a680240 con 0x55a74a7b78a0 2024-08-03T20:59:41.995 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:41.995+0000 7fa1dd600640 1 -- 172.21.15.32:0/2758706093 <== osd.5 v2:172.21.15.118:6810/2724891770 18 ==== osd_op_reply(49 606.00000000 [create] v120'794 uv794 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fa1d0021070 con 0x55a74a7b78a0 2024-08-03T20:59:41.995 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::606.00000000:head# 2024-08-03T20:59:41.996 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:41.995+0000 7fa1e4540f40 1 -- 172.21.15.32:0/2758706093 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:50 12.14 12:2f55791f:::606.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120) -- 0x55a74a680240 con 0x55a74a7b78a0 2024-08-03T20:59:41.997 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:41.996+0000 7fa1dd600640 1 -- 172.21.15.32:0/2758706093 <== osd.5 v2:172.21.15.118:6810/2724891770 19 ==== osd_op_reply(50 606.00000000 [setxattr (30)] v120'795 uv795 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fa1d0021070 con 0x55a74a7b78a0 2024-08-03T20:59:41.997 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:41.996+0000 7fa1e4540f40 1 -- 172.21.15.32:0/2758706093 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:51 12.14 12:2f55791f:::606.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120) -- 0x55a74a680240 con 0x55a74a7b78a0 2024-08-03T20:59:41.998 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:41.998+0000 7fa1dd600640 1 -- 172.21.15.32:0/2758706093 <== osd.5 v2:172.21.15.118:6810/2724891770 20 ==== osd_op_reply(51 606.00000000 [setxattr (62)] v120'796 uv796 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fa1d0021070 con 0x55a74a7b78a0 2024-08-03T20:59:41.998 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:41.998+0000 7fa1e4540f40 1 -- 172.21.15.32:0/2758706093 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:52 12.14 12:2f55791f:::606.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120) -- 0x55a74a680240 con 0x55a74a7b78a0 2024-08-03T20:59:42.000 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:41.999+0000 7fa1dd600640 1 -- 172.21.15.32:0/2758706093 <== osd.5 v2:172.21.15.118:6810/2724891770 21 ==== osd_op_reply(52 606.00000000 [omap-set-header] v120'797 uv797 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fa1d0021070 con 0x55a74a7b78a0 2024-08-03T20:59:42.000 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:41.999+0000 7fa1e4540f40 1 -- 172.21.15.32:0/2758706093 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:53 12.1c 12:3ed09add:::607.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120) -- 0x55a74a680240 con 0x55a74a63b2e0 2024-08-03T20:59:42.001 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:42.001+0000 7fa1de000640 1 -- 172.21.15.32:0/2758706093 <== osd.0 v2:172.21.15.32:6816/3066156116 32 ==== osd_op_reply(53 607.00000000 [create] v120'207 uv206 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fa1d800b040 con 0x55a74a63b2e0 2024-08-03T20:59:42.002 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:42.001+0000 7fa1e4540f40 1 -- 172.21.15.32:0/2758706093 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:54 12.1c 12:3ed09add:::607.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e120) -- 0x55a74a680240 con 0x55a74a63b2e0 2024-08-03T20:59:42.003 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:42.003+0000 7fa1de000640 1 -- 172.21.15.32:0/2758706093 <== osd.0 v2:172.21.15.32:6816/3066156116 33 ==== osd_op_reply(54 607.00000000 [delete] v120'208 uv208 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fa1d800b040 con 0x55a74a63b2e0 2024-08-03T20:59:42.003 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:42.003+0000 7fa1e4540f40 1 -- 172.21.15.32:0/2758706093 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:55 12.1c 12:3ed09add:::607.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120) -- 0x55a74a680240 con 0x55a74a63b2e0 2024-08-03T20:59:42.004 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:42.004+0000 7fa1de000640 1 -- 172.21.15.32:0/2758706093 <== osd.0 v2:172.21.15.32:6816/3066156116 34 ==== osd_op_reply(55 607.00000000 [create] v120'209 uv209 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fa1d800b040 con 0x55a74a63b2e0 2024-08-03T20:59:42.005 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::607.00000000:head# 2024-08-03T20:59:42.005 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:42.004+0000 7fa1e4540f40 1 -- 172.21.15.32:0/2758706093 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:56 12.1c 12:3ed09add:::607.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120) -- 0x55a74a7bd0c0 con 0x55a74a63b2e0 2024-08-03T20:59:42.006 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:42.005+0000 7fa1de000640 1 -- 172.21.15.32:0/2758706093 <== osd.0 v2:172.21.15.32:6816/3066156116 35 ==== osd_op_reply(56 607.00000000 [setxattr (30)] v120'210 uv210 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fa1d800b040 con 0x55a74a63b2e0 2024-08-03T20:59:42.006 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:42.005+0000 7fa1e4540f40 1 -- 172.21.15.32:0/2758706093 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:57 12.1c 12:3ed09add:::607.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120) -- 0x55a74a7bd0c0 con 0x55a74a63b2e0 2024-08-03T20:59:42.007 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:42.007+0000 7fa1de000640 1 -- 172.21.15.32:0/2758706093 <== osd.0 v2:172.21.15.32:6816/3066156116 36 ==== osd_op_reply(57 607.00000000 [setxattr (62)] v120'211 uv211 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fa1d800b040 con 0x55a74a63b2e0 2024-08-03T20:59:42.007 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:42.007+0000 7fa1e4540f40 1 -- 172.21.15.32:0/2758706093 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:58 12.1c 12:3ed09add:::607.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120) -- 0x55a74a7bc030 con 0x55a74a63b2e0 2024-08-03T20:59:42.008 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:42.008+0000 7fa1de000640 1 -- 172.21.15.32:0/2758706093 <== osd.0 v2:172.21.15.32:6816/3066156116 37 ==== osd_op_reply(58 607.00000000 [omap-set-header] v120'212 uv212 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fa1d800b040 con 0x55a74a63b2e0 2024-08-03T20:59:42.009 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:42.008+0000 7fa1e4540f40 1 --2- 172.21.15.32:0/2758706093 >> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] conn(0x55a74a7bc930 0x55a74a7bf0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:59:42.009 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:42.008+0000 7fa1e4540f40 1 -- 172.21.15.32:0/2758706093 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:59 12.6 12:654134d2:::mds0_openfiles.0:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120) -- 0x55a74a7bc030 con 0x55a74a7bc930 2024-08-03T20:59:42.009 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:42.009+0000 7fa1dcc00640 1 --2- 172.21.15.32:0/2758706093 >> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] conn(0x55a74a7bc930 0x55a74a7bf0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:59:42.009 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:42.009+0000 7fa1dcc00640 1 --2- 172.21.15.32:0/2758706093 >> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] conn(0x55a74a7bc930 0x55a74a7bf0d0 crc :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:59:42.013 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:42.013+0000 7fa1dcc00640 1 -- 172.21.15.32:0/2758706093 <== osd.1 v2:172.21.15.32:6824/1551241759 1 ==== osd_op_reply(59 mds0_openfiles.0 [create] v120'164 uv163 ondisk = -17 ((17) File exists)) ==== 160+0+0 (crc 0 0 0) 0x7fa1cc05f030 con 0x55a74a7bc930 2024-08-03T20:59:42.013 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:42.013+0000 7fa1e4540f40 1 -- 172.21.15.32:0/2758706093 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:60 12.6 12:654134d2:::mds0_openfiles.0:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e120) -- 0x55a74a7bf610 con 0x55a74a7bc930 2024-08-03T20:59:42.014 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:42.014+0000 7fa1dcc00640 1 -- 172.21.15.32:0/2758706093 <== osd.1 v2:172.21.15.32:6824/1551241759 2 ==== osd_op_reply(60 mds0_openfiles.0 [delete] v120'165 uv165 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7fa1cc05f030 con 0x55a74a7bc930 2024-08-03T20:59:42.015 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:42.014+0000 7fa1e4540f40 1 -- 172.21.15.32:0/2758706093 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:61 12.6 12:654134d2:::mds0_openfiles.0:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120) -- 0x55a74a7bf9b0 con 0x55a74a7bc930 2024-08-03T20:59:42.016 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:42.015+0000 7fa1dcc00640 1 -- 172.21.15.32:0/2758706093 <== osd.1 v2:172.21.15.32:6824/1551241759 3 ==== osd_op_reply(61 mds0_openfiles.0 [create] v120'166 uv166 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7fa1cc05f030 con 0x55a74a7bc930 2024-08-03T20:59:42.016 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds0_openfiles.0:head# 2024-08-03T20:59:42.016 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:42.016+0000 7fa1e4540f40 1 -- 172.21.15.32:0/2758706093 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:62 12.6 12:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120) -- 0x55a74a7bbc00 con 0x55a74a7bc930 2024-08-03T20:59:42.017 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:42.017+0000 7fa1dcc00640 1 -- 172.21.15.32:0/2758706093 <== osd.1 v2:172.21.15.32:6824/1551241759 4 ==== osd_op_reply(62 mds0_openfiles.0 [omap-set-header] v120'167 uv167 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7fa1cc05f030 con 0x55a74a7bc930 2024-08-03T20:59:42.017 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:42.017+0000 7fa1e4540f40 1 -- 172.21.15.32:0/2758706093 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:63 12.e 12:717a0223:::608.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120) -- 0x55a74a7c0250 con 0x55a74a63b2e0 2024-08-03T20:59:42.019 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:42.018+0000 7fa1de000640 1 -- 172.21.15.32:0/2758706093 <== osd.0 v2:172.21.15.32:6816/3066156116 38 ==== osd_op_reply(63 608.00000000 [create] v120'207 uv206 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fa1d800b040 con 0x55a74a63b2e0 2024-08-03T20:59:42.019 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:42.018+0000 7fa1e4540f40 1 -- 172.21.15.32:0/2758706093 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:64 12.e 12:717a0223:::608.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e120) -- 0x55a74a7c0250 con 0x55a74a63b2e0 2024-08-03T20:59:42.020 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:42.020+0000 7fa1de000640 1 -- 172.21.15.32:0/2758706093 <== osd.0 v2:172.21.15.32:6816/3066156116 39 ==== osd_op_reply(64 608.00000000 [delete] v120'208 uv208 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fa1d800b040 con 0x55a74a63b2e0 2024-08-03T20:59:42.021 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:42.020+0000 7fa1e4540f40 1 -- 172.21.15.32:0/2758706093 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:65 12.e 12:717a0223:::608.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120) -- 0x55a74a7c0250 con 0x55a74a63b2e0 2024-08-03T20:59:42.021 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:42.021+0000 7fa1de000640 1 -- 172.21.15.32:0/2758706093 <== osd.0 v2:172.21.15.32:6816/3066156116 40 ==== osd_op_reply(65 608.00000000 [create] v120'209 uv209 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fa1d800b040 con 0x55a74a63b2e0 2024-08-03T20:59:42.022 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::608.00000000:head# 2024-08-03T20:59:42.022 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:42.021+0000 7fa1e4540f40 1 -- 172.21.15.32:0/2758706093 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:66 12.e 12:717a0223:::608.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120) -- 0x55a74a7c1260 con 0x55a74a63b2e0 2024-08-03T20:59:42.023 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:42.023+0000 7fa1de000640 1 -- 172.21.15.32:0/2758706093 <== osd.0 v2:172.21.15.32:6816/3066156116 41 ==== osd_op_reply(66 608.00000000 [setxattr (30)] v120'210 uv210 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fa1d800b040 con 0x55a74a63b2e0 2024-08-03T20:59:42.023 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:42.023+0000 7fa1e4540f40 1 -- 172.21.15.32:0/2758706093 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:67 12.e 12:717a0223:::608.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120) -- 0x55a74a7c1260 con 0x55a74a63b2e0 2024-08-03T20:59:42.025 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:42.024+0000 7fa1de000640 1 -- 172.21.15.32:0/2758706093 <== osd.0 v2:172.21.15.32:6816/3066156116 42 ==== osd_op_reply(67 608.00000000 [setxattr (62)] v120'211 uv211 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fa1d800b040 con 0x55a74a63b2e0 2024-08-03T20:59:42.025 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:42.024+0000 7fa1e4540f40 1 -- 172.21.15.32:0/2758706093 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:68 12.e 12:717a0223:::608.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120) -- 0x55a74a7c0250 con 0x55a74a63b2e0 2024-08-03T20:59:42.026 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:42.025+0000 7fa1de000640 1 -- 172.21.15.32:0/2758706093 <== osd.0 v2:172.21.15.32:6816/3066156116 43 ==== osd_op_reply(68 608.00000000 [omap-set-header] v120'212 uv212 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fa1d800b040 con 0x55a74a63b2e0 2024-08-03T20:59:42.026 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:42.026+0000 7fa1e4540f40 1 --2- 172.21.15.32:0/2758706093 >> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] conn(0x55a74a7c0920 0x55a74a7c0d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:59:42.026 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:42.026+0000 7fa1e4540f40 1 -- 172.21.15.32:0/2758706093 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:69 12.15 12:a93a17c2:::604.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120) -- 0x55a74a7c36b0 con 0x55a74a7c0920 2024-08-03T20:59:42.026 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:42.026+0000 7fa1de000640 1 --2- 172.21.15.32:0/2758706093 >> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] conn(0x55a74a7c0920 0x55a74a7c0d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:59:42.027 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:42.027+0000 7fa1de000640 1 --2- 172.21.15.32:0/2758706093 >> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] conn(0x55a74a7c0920 0x55a74a7c0d30 crc :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:59:42.029 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:42.029+0000 7fa1de000640 1 -- 172.21.15.32:0/2758706093 <== osd.6 v2:172.21.15.118:6825/801947386 1 ==== osd_op_reply(69 604.00000000 [create] v120'382 uv376 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fa1d800b040 con 0x55a74a7c0920 2024-08-03T20:59:42.029 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:42.029+0000 7fa1e4540f40 1 -- 172.21.15.32:0/2758706093 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:70 12.15 12:a93a17c2:::604.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e120) -- 0x55a74a7c36b0 con 0x55a74a7c0920 2024-08-03T20:59:42.031 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:42.030+0000 7fa1de000640 1 -- 172.21.15.32:0/2758706093 <== osd.6 v2:172.21.15.118:6825/801947386 2 ==== osd_op_reply(70 604.00000000 [delete] v120'383 uv383 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fa1d800b040 con 0x55a74a7c0920 2024-08-03T20:59:42.031 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:42.030+0000 7fa1e4540f40 1 -- 172.21.15.32:0/2758706093 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:71 12.15 12:a93a17c2:::604.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120) -- 0x55a74a7c36b0 con 0x55a74a7c0920 2024-08-03T20:59:42.032 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::604.00000000:head# 2024-08-03T20:59:42.032 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:42.032+0000 7fa1de000640 1 -- 172.21.15.32:0/2758706093 <== osd.6 v2:172.21.15.118:6825/801947386 3 ==== osd_op_reply(71 604.00000000 [create] v120'384 uv384 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fa1d800b040 con 0x55a74a7c0920 2024-08-03T20:59:42.032 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:42.032+0000 7fa1e4540f40 1 -- 172.21.15.32:0/2758706093 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:72 12.15 12:a93a17c2:::604.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120) -- 0x55a74a7c4700 con 0x55a74a7c0920 2024-08-03T20:59:42.033 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:42.033+0000 7fa1de000640 1 -- 172.21.15.32:0/2758706093 <== osd.6 v2:172.21.15.118:6825/801947386 4 ==== osd_op_reply(72 604.00000000 [setxattr (30)] v120'385 uv385 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fa1d800b040 con 0x55a74a7c0920 2024-08-03T20:59:42.034 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:42.033+0000 7fa1e4540f40 1 -- 172.21.15.32:0/2758706093 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:73 12.15 12:a93a17c2:::604.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120) -- 0x55a74a7c4700 con 0x55a74a7c0920 2024-08-03T20:59:42.035 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:42.035+0000 7fa1de000640 1 -- 172.21.15.32:0/2758706093 <== osd.6 v2:172.21.15.118:6825/801947386 5 ==== osd_op_reply(73 604.00000000 [setxattr (62)] v120'386 uv386 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fa1d800b040 con 0x55a74a7c0920 2024-08-03T20:59:42.035 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:42.035+0000 7fa1e4540f40 1 -- 172.21.15.32:0/2758706093 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:74 12.15 12:a93a17c2:::604.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120) -- 0x55a74a7c36b0 con 0x55a74a7c0920 2024-08-03T20:59:42.036 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:42.036+0000 7fa1de000640 1 -- 172.21.15.32:0/2758706093 <== osd.6 v2:172.21.15.118:6825/801947386 6 ==== osd_op_reply(74 604.00000000 [omap-set-header] v120'387 uv387 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fa1d800b040 con 0x55a74a7c0920 2024-08-03T20:59:42.036 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:42.036+0000 7fa1e4540f40 1 -- 172.21.15.32:0/2758706093 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:75 12.15 12:aa448500:::500.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120) -- 0x55a74a7c36b0 con 0x55a74a7c0920 2024-08-03T20:59:42.038 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:42.037+0000 7fa1de000640 1 -- 172.21.15.32:0/2758706093 <== osd.6 v2:172.21.15.118:6825/801947386 7 ==== osd_op_reply(75 500.00000000 [create] v120'388 uv381 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fa1d800b040 con 0x55a74a7c0920 2024-08-03T20:59:42.038 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:42.037+0000 7fa1e4540f40 1 -- 172.21.15.32:0/2758706093 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:76 12.15 12:aa448500:::500.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e120) -- 0x55a74a7c36b0 con 0x55a74a7c0920 2024-08-03T20:59:42.039 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:42.039+0000 7fa1de000640 1 -- 172.21.15.32:0/2758706093 <== osd.6 v2:172.21.15.118:6825/801947386 8 ==== osd_op_reply(76 500.00000000 [delete] v120'389 uv389 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fa1d800b040 con 0x55a74a7c0920 2024-08-03T20:59:42.039 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:42.039+0000 7fa1e4540f40 1 -- 172.21.15.32:0/2758706093 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:77 12.15 12:aa448500:::500.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120) -- 0x55a74a7c36b0 con 0x55a74a7c0920 2024-08-03T20:59:42.041 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:42.040+0000 7fa1de000640 1 -- 172.21.15.32:0/2758706093 <== osd.6 v2:172.21.15.118:6825/801947386 9 ==== osd_op_reply(77 500.00000000 [create] v120'390 uv390 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fa1d800b040 con 0x55a74a7c0920 2024-08-03T20:59:42.041 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::500.00000000:head# 2024-08-03T20:59:42.041 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:42.040+0000 7fa1e4540f40 1 -- 172.21.15.32:0/2758706093 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:78 12.15 12:aa448500:::500.00000000:head [write 0~90 in=90b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120) -- 0x55a74a7c36b0 con 0x55a74a7c0920 2024-08-03T20:59:42.042 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:42.042+0000 7fa1de000640 1 -- 172.21.15.32:0/2758706093 <== osd.6 v2:172.21.15.118:6825/801947386 10 ==== osd_op_reply(78 500.00000000 [write 0~90] v120'391 uv391 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fa1d800b040 con 0x55a74a7c0920 2024-08-03T20:59:42.043 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:42.042+0000 7fa1e4540f40 1 -- 172.21.15.32:0/2758706093 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:79 12.15 12:aa448500:::500.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120) -- 0x55a74a7c36b0 con 0x55a74a7c0920 2024-08-03T20:59:42.044 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:42.044+0000 7fa1de000640 1 -- 172.21.15.32:0/2758706093 <== osd.6 v2:172.21.15.118:6825/801947386 11 ==== osd_op_reply(79 500.00000000 [omap-set-header] v120'392 uv392 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fa1d800b040 con 0x55a74a7c0920 2024-08-03T20:59:42.044 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:42.044+0000 7fa1e4540f40 1 --2- 172.21.15.32:0/2758706093 >> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] conn(0x55a74a7c3fd0 0x55a74a7c43e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:59:42.044 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:42.044+0000 7fa1e4540f40 1 -- 172.21.15.32:0/2758706093 --> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] -- osd_op(unknown.0.0:80 12.d 12:b50e409b:::mds_snaptable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120) -- 0x55a74a7c6b90 con 0x55a74a7c3fd0 2024-08-03T20:59:42.044 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:42.044+0000 7fa1dd600640 1 --2- 172.21.15.32:0/2758706093 >> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] conn(0x55a74a7c3fd0 0x55a74a7c43e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:59:42.045 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:42.045+0000 7fa1dd600640 1 --2- 172.21.15.32:0/2758706093 >> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] conn(0x55a74a7c3fd0 0x55a74a7c43e0 crc :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:59:42.047 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:42.046+0000 7fa1dd600640 1 -- 172.21.15.32:0/2758706093 <== osd.7 v2:172.21.15.118:6818/653000048 1 ==== osd_op_reply(80 mds_snaptable [create] v120'173 uv172 ondisk = -17 ((17) File exists)) ==== 157+0+0 (crc 0 0 0) 0x7fa1d0021070 con 0x55a74a7c3fd0 2024-08-03T20:59:42.047 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:42.047+0000 7fa1e4540f40 1 -- 172.21.15.32:0/2758706093 --> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] -- osd_op(unknown.0.0:81 12.d 12:b50e409b:::mds_snaptable:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e120) -- 0x55a74a7c6b90 con 0x55a74a7c3fd0 2024-08-03T20:59:42.048 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:42.048+0000 7fa1dd600640 1 -- 172.21.15.32:0/2758706093 <== osd.7 v2:172.21.15.118:6818/653000048 2 ==== osd_op_reply(81 mds_snaptable [delete] v120'174 uv174 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7fa1d0021070 con 0x55a74a7c3fd0 2024-08-03T20:59:42.048 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:42.048+0000 7fa1e4540f40 1 -- 172.21.15.32:0/2758706093 --> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] -- osd_op(unknown.0.0:82 12.d 12:b50e409b:::mds_snaptable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120) -- 0x55a74a7c6b90 con 0x55a74a7c3fd0 2024-08-03T20:59:42.049 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds_snaptable:head# 2024-08-03T20:59:42.050 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:42.049+0000 7fa1dd600640 1 -- 172.21.15.32:0/2758706093 <== osd.7 v2:172.21.15.118:6818/653000048 3 ==== osd_op_reply(82 mds_snaptable [create] v120'175 uv175 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7fa1d0021070 con 0x55a74a7c3fd0 2024-08-03T20:59:42.050 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:42.049+0000 7fa1e4540f40 1 -- 172.21.15.32:0/2758706093 --> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] -- osd_op(unknown.0.0:83 12.d 12:b50e409b:::mds_snaptable:head [write 0~70 in=70b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120) -- 0x55a74a7c6f80 con 0x55a74a7c3fd0 2024-08-03T20:59:42.051 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:42.051+0000 7fa1dd600640 1 -- 172.21.15.32:0/2758706093 <== osd.7 v2:172.21.15.118:6818/653000048 4 ==== osd_op_reply(83 mds_snaptable [write 0~70] v120'176 uv176 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7fa1d0021070 con 0x55a74a7c3fd0 2024-08-03T20:59:42.051 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:42.051+0000 7fa1e4540f40 1 -- 172.21.15.32:0/2758706093 --> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] -- osd_op(unknown.0.0:84 12.d 12:b50e409b:::mds_snaptable:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120) -- 0x55a74a7c6f80 con 0x55a74a7c3fd0 2024-08-03T20:59:42.053 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:42.052+0000 7fa1dd600640 1 -- 172.21.15.32:0/2758706093 <== osd.7 v2:172.21.15.118:6818/653000048 5 ==== osd_op_reply(84 mds_snaptable [omap-set-header] v120'177 uv177 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7fa1d0021070 con 0x55a74a7c3fd0 2024-08-03T20:59:42.053 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:42.052+0000 7fa1e4540f40 1 -- 172.21.15.32:0/2758706093 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:85 12.1d 12:b871830b:::605.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120) -- 0x55a74a7c6f80 con 0x55a74a63b2e0 2024-08-03T20:59:42.054 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:42.054+0000 7fa1de000640 1 -- 172.21.15.32:0/2758706093 <== osd.0 v2:172.21.15.32:6816/3066156116 44 ==== osd_op_reply(85 605.00000000 [create] v120'207 uv206 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fa1d800b040 con 0x55a74a63b2e0 2024-08-03T20:59:42.055 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:42.054+0000 7fa1e4540f40 1 -- 172.21.15.32:0/2758706093 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:86 12.1d 12:b871830b:::605.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e120) -- 0x55a74a7c6f80 con 0x55a74a63b2e0 2024-08-03T20:59:42.056 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:42.055+0000 7fa1de000640 1 -- 172.21.15.32:0/2758706093 <== osd.0 v2:172.21.15.32:6816/3066156116 45 ==== osd_op_reply(86 605.00000000 [delete] v120'208 uv208 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fa1d800b040 con 0x55a74a63b2e0 2024-08-03T20:59:42.056 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:42.055+0000 7fa1e4540f40 1 -- 172.21.15.32:0/2758706093 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:87 12.1d 12:b871830b:::605.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120) -- 0x55a74a7c6f80 con 0x55a74a63b2e0 2024-08-03T20:59:42.057 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:42.056+0000 7fa1de000640 1 -- 172.21.15.32:0/2758706093 <== osd.0 v2:172.21.15.32:6816/3066156116 46 ==== osd_op_reply(87 605.00000000 [create] v120'209 uv209 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fa1d800b040 con 0x55a74a63b2e0 2024-08-03T20:59:42.057 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::605.00000000:head# 2024-08-03T20:59:42.057 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:42.056+0000 7fa1e4540f40 1 -- 172.21.15.32:0/2758706093 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:88 12.1d 12:b871830b:::605.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120) -- 0x55a74a7c7f90 con 0x55a74a63b2e0 2024-08-03T20:59:42.058 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:42.058+0000 7fa1de000640 1 -- 172.21.15.32:0/2758706093 <== osd.0 v2:172.21.15.32:6816/3066156116 47 ==== osd_op_reply(88 605.00000000 [setxattr (30)] v120'210 uv210 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fa1d800b040 con 0x55a74a63b2e0 2024-08-03T20:59:42.058 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:42.058+0000 7fa1e4540f40 1 -- 172.21.15.32:0/2758706093 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:89 12.1d 12:b871830b:::605.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120) -- 0x55a74a7c7f90 con 0x55a74a63b2e0 2024-08-03T20:59:42.059 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:42.059+0000 7fa1de000640 1 -- 172.21.15.32:0/2758706093 <== osd.0 v2:172.21.15.32:6816/3066156116 48 ==== osd_op_reply(89 605.00000000 [setxattr (62)] v120'211 uv211 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fa1d800b040 con 0x55a74a63b2e0 2024-08-03T20:59:42.059 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:42.059+0000 7fa1e4540f40 1 -- 172.21.15.32:0/2758706093 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:90 12.1d 12:b871830b:::605.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120) -- 0x55a74a7c6f80 con 0x55a74a63b2e0 2024-08-03T20:59:42.061 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:42.060+0000 7fa1de000640 1 -- 172.21.15.32:0/2758706093 <== osd.0 v2:172.21.15.32:6816/3066156116 49 ==== osd_op_reply(90 605.00000000 [omap-set-header] v120'212 uv212 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fa1d800b040 con 0x55a74a63b2e0 2024-08-03T20:59:42.061 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:42.060+0000 7fa1e4540f40 1 -- 172.21.15.32:0/2758706093 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:91 12.13 12:c91d4a1d:::mds0_inotable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120) -- 0x55a74a7c6f80 con 0x55a74a7bc930 2024-08-03T20:59:42.062 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:42.062+0000 7fa1dcc00640 1 -- 172.21.15.32:0/2758706093 <== osd.1 v2:172.21.15.32:6824/1551241759 5 ==== osd_op_reply(91 mds0_inotable [create] v120'345 uv339 ondisk = -17 ((17) File exists)) ==== 157+0+0 (crc 0 0 0) 0x7fa1cc05f030 con 0x55a74a7bc930 2024-08-03T20:59:42.062 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:42.062+0000 7fa1e4540f40 1 -- 172.21.15.32:0/2758706093 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:92 12.13 12:c91d4a1d:::mds0_inotable:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e120) -- 0x55a74a7c7320 con 0x55a74a7bc930 2024-08-03T20:59:42.063 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:42.063+0000 7fa1dcc00640 1 -- 172.21.15.32:0/2758706093 <== osd.1 v2:172.21.15.32:6824/1551241759 6 ==== osd_op_reply(92 mds0_inotable [delete] v120'346 uv346 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7fa1cc05f030 con 0x55a74a7bc930 2024-08-03T20:59:42.064 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:42.063+0000 7fa1e4540f40 1 -- 172.21.15.32:0/2758706093 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:93 12.13 12:c91d4a1d:::mds0_inotable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120) -- 0x55a74a7c7320 con 0x55a74a7bc930 2024-08-03T20:59:42.065 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds0_inotable:head# 2024-08-03T20:59:42.065 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:42.064+0000 7fa1dcc00640 1 -- 172.21.15.32:0/2758706093 <== osd.1 v2:172.21.15.32:6824/1551241759 7 ==== osd_op_reply(93 mds0_inotable [create] v120'347 uv347 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7fa1cc05f030 con 0x55a74a7bc930 2024-08-03T20:59:42.065 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:42.064+0000 7fa1e4540f40 1 -- 172.21.15.32:0/2758706093 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:94 12.13 12:c91d4a1d:::mds0_inotable:head [write 0~34 in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120) -- 0x55a74a7c7320 con 0x55a74a7bc930 2024-08-03T20:59:42.066 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:42.066+0000 7fa1dcc00640 1 -- 172.21.15.32:0/2758706093 <== osd.1 v2:172.21.15.32:6824/1551241759 8 ==== osd_op_reply(94 mds0_inotable [write 0~34] v120'348 uv348 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7fa1cc05f030 con 0x55a74a7bc930 2024-08-03T20:59:42.066 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:42.066+0000 7fa1e4540f40 1 -- 172.21.15.32:0/2758706093 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:95 12.13 12:c91d4a1d:::mds0_inotable:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120) -- 0x55a74a7c7320 con 0x55a74a7bc930 2024-08-03T20:59:42.067 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:42.067+0000 7fa1dcc00640 1 -- 172.21.15.32:0/2758706093 <== osd.1 v2:172.21.15.32:6824/1551241759 9 ==== osd_op_reply(95 mds0_inotable [omap-set-header] v120'349 uv349 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7fa1cc05f030 con 0x55a74a7bc930 2024-08-03T20:59:42.067 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:42.067+0000 7fa1e4540f40 1 -- 172.21.15.32:0/2758706093 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:96 12.13 12:cd5a64a3:::100.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120) -- 0x55a74a7c7320 con 0x55a74a7bc930 2024-08-03T20:59:42.069 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:42.068+0000 7fa1dcc00640 1 -- 172.21.15.32:0/2758706093 <== osd.1 v2:172.21.15.32:6824/1551241759 10 ==== osd_op_reply(96 100.00000000 [create] v120'350 uv344 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fa1cc05f030 con 0x55a74a7bc930 2024-08-03T20:59:42.069 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:42.069+0000 7fa1e4540f40 1 -- 172.21.15.32:0/2758706093 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:97 12.13 12:cd5a64a3:::100.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e120) -- 0x55a74a7c7320 con 0x55a74a7bc930 2024-08-03T20:59:42.070 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:42.070+0000 7fa1dcc00640 1 -- 172.21.15.32:0/2758706093 <== osd.1 v2:172.21.15.32:6824/1551241759 11 ==== osd_op_reply(97 100.00000000 [delete] v120'351 uv351 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fa1cc05f030 con 0x55a74a7bc930 2024-08-03T20:59:42.070 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:42.070+0000 7fa1e4540f40 1 -- 172.21.15.32:0/2758706093 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:98 12.13 12:cd5a64a3:::100.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120) -- 0x55a74a7c7320 con 0x55a74a7bc930 2024-08-03T20:59:42.072 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::100.00000000:head# 2024-08-03T20:59:42.072 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:42.071+0000 7fa1dcc00640 1 -- 172.21.15.32:0/2758706093 <== osd.1 v2:172.21.15.32:6824/1551241759 12 ==== osd_op_reply(98 100.00000000 [create] v120'352 uv352 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fa1cc05f030 con 0x55a74a7bc930 2024-08-03T20:59:42.072 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:42.071+0000 7fa1e4540f40 1 -- 172.21.15.32:0/2758706093 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:99 12.13 12:cd5a64a3:::100.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120) -- 0x55a74a7c7320 con 0x55a74a7bc930 2024-08-03T20:59:42.073 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:42.073+0000 7fa1dcc00640 1 -- 172.21.15.32:0/2758706093 <== osd.1 v2:172.21.15.32:6824/1551241759 13 ==== osd_op_reply(99 100.00000000 [omap-set-header] v120'353 uv353 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fa1cc05f030 con 0x55a74a7bc930 2024-08-03T20:59:42.073 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:42.073+0000 7fa1e4540f40 1 -- 172.21.15.32:0/2758706093 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:100 12.13 12:cd5a64a3:::100.00000000:head [omap-set-vals in=5194b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120) -- 0x55a74a7ca130 con 0x55a74a7bc930 2024-08-03T20:59:42.075 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:42.074+0000 7fa1dcc00640 1 -- 172.21.15.32:0/2758706093 <== osd.1 v2:172.21.15.32:6824/1551241759 14 ==== osd_op_reply(100 100.00000000 [omap-set-vals] v120'354 uv354 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fa1cc05f030 con 0x55a74a7bc930 2024-08-03T20:59:42.075 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:42.074+0000 7fa1e4540f40 1 --2- 172.21.15.32:0/2758706093 >> [v2:172.21.15.32:6800/9088303,v1:172.21.15.32:6801/9088303] conn(0x55a74a7c7e50 0x55a74a7ca730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:59:42.075 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:42.074+0000 7fa1e4540f40 1 -- 172.21.15.32:0/2758706093 --> [v2:172.21.15.32:6800/9088303,v1:172.21.15.32:6801/9088303] -- osd_op(unknown.0.0:101 12.b 12:d0630e4c:::mds0_sessionmap:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120) -- 0x55a74a7ca270 con 0x55a74a7c7e50 2024-08-03T20:59:42.076 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:42.075+0000 7fa1dcc00640 1 --2- 172.21.15.32:0/2758706093 >> [v2:172.21.15.32:6800/9088303,v1:172.21.15.32:6801/9088303] conn(0x55a74a7c7e50 0x55a74a7ca730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:59:42.076 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:42.075+0000 7fa1dcc00640 1 --2- 172.21.15.32:0/2758706093 >> [v2:172.21.15.32:6800/9088303,v1:172.21.15.32:6801/9088303] conn(0x55a74a7c7e50 0x55a74a7ca730 crc :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:59:42.077 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:42.077+0000 7fa1dcc00640 1 -- 172.21.15.32:0/2758706093 <== osd.3 v2:172.21.15.32:6800/9088303 1 ==== osd_op_reply(101 mds0_sessionmap [create] v120'152 uv150 ondisk = -17 ((17) File exists)) ==== 159+0+0 (crc 0 0 0) 0x7fa1cc05f030 con 0x55a74a7c7e50 2024-08-03T20:59:42.077 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:42.077+0000 7fa1e4540f40 1 -- 172.21.15.32:0/2758706093 --> [v2:172.21.15.32:6800/9088303,v1:172.21.15.32:6801/9088303] -- osd_op(unknown.0.0:102 12.b 12:d0630e4c:::mds0_sessionmap:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e120) -- 0x55a74a7ca270 con 0x55a74a7c7e50 2024-08-03T20:59:42.079 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:42.078+0000 7fa1dcc00640 1 -- 172.21.15.32:0/2758706093 <== osd.3 v2:172.21.15.32:6800/9088303 2 ==== osd_op_reply(102 mds0_sessionmap [delete] v120'153 uv153 ondisk = 0) ==== 159+0+0 (crc 0 0 0) 0x7fa1cc05f030 con 0x55a74a7c7e50 2024-08-03T20:59:42.079 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:42.078+0000 7fa1e4540f40 1 -- 172.21.15.32:0/2758706093 --> [v2:172.21.15.32:6800/9088303,v1:172.21.15.32:6801/9088303] -- osd_op(unknown.0.0:103 12.b 12:d0630e4c:::mds0_sessionmap:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120) -- 0x55a74a7ca270 con 0x55a74a7c7e50 2024-08-03T20:59:42.080 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds0_sessionmap:head# 2024-08-03T20:59:42.080 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:42.079+0000 7fa1dcc00640 1 -- 172.21.15.32:0/2758706093 <== osd.3 v2:172.21.15.32:6800/9088303 3 ==== osd_op_reply(103 mds0_sessionmap [create] v120'154 uv154 ondisk = 0) ==== 159+0+0 (crc 0 0 0) 0x7fa1cc05f030 con 0x55a74a7c7e50 2024-08-03T20:59:42.080 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:42.079+0000 7fa1e4540f40 1 -- 172.21.15.32:0/2758706093 --> [v2:172.21.15.32:6800/9088303,v1:172.21.15.32:6801/9088303] -- osd_op(unknown.0.0:104 12.b 12:d0630e4c:::mds0_sessionmap:head [omap-set-header in=14b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120) -- 0x55a74a7ca270 con 0x55a74a7c7e50 2024-08-03T20:59:42.081 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:42.081+0000 7fa1dcc00640 1 -- 172.21.15.32:0/2758706093 <== osd.3 v2:172.21.15.32:6800/9088303 4 ==== osd_op_reply(104 mds0_sessionmap [omap-set-header] v120'155 uv155 ondisk = 0) ==== 159+0+0 (crc 0 0 0) 0x7fa1cc05f030 con 0x55a74a7c7e50 2024-08-03T20:59:42.081 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:42.081+0000 7fa1e4540f40 1 -- 172.21.15.32:0/2758706093 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:105 12.7 12:e0b41d06:::609.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120) -- 0x55a74a7ca270 con 0x55a74a7bc930 2024-08-03T20:59:42.083 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:42.083+0000 7fa1dcc00640 1 -- 172.21.15.32:0/2758706093 <== osd.1 v2:172.21.15.32:6824/1551241759 15 ==== osd_op_reply(105 609.00000000 [create] v120'207 uv206 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fa1cc05f030 con 0x55a74a7bc930 2024-08-03T20:59:42.083 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:42.083+0000 7fa1e4540f40 1 -- 172.21.15.32:0/2758706093 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:106 12.7 12:e0b41d06:::609.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e120) -- 0x55a74a7ca270 con 0x55a74a7bc930 2024-08-03T20:59:42.084 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:42.084+0000 7fa1dcc00640 1 -- 172.21.15.32:0/2758706093 <== osd.1 v2:172.21.15.32:6824/1551241759 16 ==== osd_op_reply(106 609.00000000 [delete] v120'208 uv208 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fa1cc05f030 con 0x55a74a7bc930 2024-08-03T20:59:42.085 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:42.084+0000 7fa1e4540f40 1 -- 172.21.15.32:0/2758706093 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:107 12.7 12:e0b41d06:::609.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120) -- 0x55a74a7ca270 con 0x55a74a7bc930 2024-08-03T20:59:42.086 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::609.00000000:head# 2024-08-03T20:59:42.086 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:42.085+0000 7fa1dcc00640 1 -- 172.21.15.32:0/2758706093 <== osd.1 v2:172.21.15.32:6824/1551241759 17 ==== osd_op_reply(107 609.00000000 [create] v120'209 uv209 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fa1cc05f030 con 0x55a74a7bc930 2024-08-03T20:59:42.086 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:42.085+0000 7fa1e4540f40 1 -- 172.21.15.32:0/2758706093 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:108 12.7 12:e0b41d06:::609.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120) -- 0x55a74a7ca270 con 0x55a74a7bc930 2024-08-03T20:59:42.087 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:42.087+0000 7fa1dcc00640 1 -- 172.21.15.32:0/2758706093 <== osd.1 v2:172.21.15.32:6824/1551241759 18 ==== osd_op_reply(108 609.00000000 [setxattr (30)] v120'210 uv210 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fa1cc05f030 con 0x55a74a7bc930 2024-08-03T20:59:42.087 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:42.087+0000 7fa1e4540f40 1 -- 172.21.15.32:0/2758706093 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:109 12.7 12:e0b41d06:::609.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120) -- 0x55a74a7ca270 con 0x55a74a7bc930 2024-08-03T20:59:42.089 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:42.088+0000 7fa1dcc00640 1 -- 172.21.15.32:0/2758706093 <== osd.1 v2:172.21.15.32:6824/1551241759 19 ==== osd_op_reply(109 609.00000000 [setxattr (62)] v120'211 uv211 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fa1cc05f030 con 0x55a74a7bc930 2024-08-03T20:59:42.089 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:42.088+0000 7fa1e4540f40 1 -- 172.21.15.32:0/2758706093 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:110 12.7 12:e0b41d06:::609.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120) -- 0x55a74a7ca270 con 0x55a74a7bc930 2024-08-03T20:59:42.090 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:42.090+0000 7fa1dcc00640 1 -- 172.21.15.32:0/2758706093 <== osd.1 v2:172.21.15.32:6824/1551241759 20 ==== osd_op_reply(110 609.00000000 [omap-set-header] v120'212 uv212 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fa1cc05f030 con 0x55a74a7bc930 2024-08-03T20:59:42.090 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:42.090+0000 7fa1e4540f40 1 -- 172.21.15.32:0/2758706093 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:111 12.f 12:f1f69726:::400.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120) -- 0x55a74a7ca270 con 0x55a74a7c0920 2024-08-03T20:59:42.091 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:42.091+0000 7fa1de000640 1 -- 172.21.15.32:0/2758706093 <== osd.6 v2:172.21.15.118:6825/801947386 12 ==== osd_op_reply(111 400.00000000 [create] v120'176 uv175 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fa1d800b040 con 0x55a74a7c0920 2024-08-03T20:59:42.091 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:42.091+0000 7fa1e4540f40 1 -- 172.21.15.32:0/2758706093 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:112 12.f 12:f1f69726:::400.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e120) -- 0x55a74a7ca270 con 0x55a74a7c0920 2024-08-03T20:59:42.093 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:42.093+0000 7fa1de000640 1 -- 172.21.15.32:0/2758706093 <== osd.6 v2:172.21.15.118:6825/801947386 13 ==== osd_op_reply(112 400.00000000 [delete] v120'177 uv177 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fa1d800b040 con 0x55a74a7c0920 2024-08-03T20:59:42.093 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:42.093+0000 7fa1e4540f40 1 -- 172.21.15.32:0/2758706093 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:113 12.f 12:f1f69726:::400.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120) -- 0x55a74a7ca270 con 0x55a74a7c0920 2024-08-03T20:59:42.095 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::400.00000000:head# 2024-08-03T20:59:42.095 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:42.094+0000 7fa1de000640 1 -- 172.21.15.32:0/2758706093 <== osd.6 v2:172.21.15.118:6825/801947386 14 ==== osd_op_reply(113 400.00000000 [create] v120'178 uv178 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fa1d800b040 con 0x55a74a7c0920 2024-08-03T20:59:42.095 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:42.094+0000 7fa1e4540f40 1 -- 172.21.15.32:0/2758706093 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:114 12.f 12:f1f69726:::400.00000000:head [write 0~22 in=22b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120) -- 0x55a74a7ca270 con 0x55a74a7c0920 2024-08-03T20:59:42.096 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:42.096+0000 7fa1de000640 1 -- 172.21.15.32:0/2758706093 <== osd.6 v2:172.21.15.118:6825/801947386 15 ==== osd_op_reply(114 400.00000000 [write 0~22] v120'179 uv179 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fa1d800b040 con 0x55a74a7c0920 2024-08-03T20:59:42.096 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:42.096+0000 7fa1e4540f40 1 -- 172.21.15.32:0/2758706093 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:115 12.f 12:f1f69726:::400.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120) -- 0x55a74a7ca270 con 0x55a74a7c0920 2024-08-03T20:59:42.098 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:42.097+0000 7fa1de000640 1 -- 172.21.15.32:0/2758706093 <== osd.6 v2:172.21.15.118:6825/801947386 16 ==== osd_op_reply(115 400.00000000 [omap-set-header] v120'180 uv180 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fa1d800b040 con 0x55a74a7c0920 2024-08-03T20:59:42.098 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:42.098+0000 7fa1e4540f40 1 -- 172.21.15.32:0/2758706093 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:116 12.1f 12:fe07bba1:::100.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120) -- 0x55a74a7ca270 con 0x55a74a7c0920 2024-08-03T20:59:42.100 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:42.100+0000 7fa1de000640 1 -- 172.21.15.32:0/2758706093 <== osd.6 v2:172.21.15.118:6825/801947386 17 ==== osd_op_reply(116 100.00000000.inode [create] v120'429 uv420 ondisk = -17 ((17) File exists)) ==== 162+0+0 (crc 0 0 0) 0x7fa1d800b040 con 0x55a74a7c0920 2024-08-03T20:59:42.100 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:42.100+0000 7fa1e4540f40 1 -- 172.21.15.32:0/2758706093 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:117 12.1f 12:fe07bba1:::100.00000000.inode:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e120) -- 0x55a74a7ca270 con 0x55a74a7c0920 2024-08-03T20:59:42.101 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:42.101+0000 7fa1de000640 1 -- 172.21.15.32:0/2758706093 <== osd.6 v2:172.21.15.118:6825/801947386 18 ==== osd_op_reply(117 100.00000000.inode [delete] v120'430 uv430 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7fa1d800b040 con 0x55a74a7c0920 2024-08-03T20:59:42.101 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:42.101+0000 7fa1e4540f40 1 -- 172.21.15.32:0/2758706093 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:118 12.1f 12:fe07bba1:::100.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120) -- 0x55a74a7ca270 con 0x55a74a7c0920 2024-08-03T20:59:42.103 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::100.00000000.inode:head# 2024-08-03T20:59:42.103 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:42.102+0000 7fa1de000640 1 -- 172.21.15.32:0/2758706093 <== osd.6 v2:172.21.15.118:6825/801947386 19 ==== osd_op_reply(118 100.00000000.inode [create] v120'431 uv431 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7fa1d800b040 con 0x55a74a7c0920 2024-08-03T20:59:42.103 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:42.102+0000 7fa1e4540f40 1 -- 172.21.15.32:0/2758706093 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:119 12.1f 12:fe07bba1:::100.00000000.inode:head [write 0~582 in=582b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120) -- 0x55a74a7cad30 con 0x55a74a7c0920 2024-08-03T20:59:42.104 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:42.104+0000 7fa1de000640 1 -- 172.21.15.32:0/2758706093 <== osd.6 v2:172.21.15.118:6825/801947386 20 ==== osd_op_reply(119 100.00000000.inode [write 0~582] v120'432 uv432 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7fa1d800b040 con 0x55a74a7c0920 2024-08-03T20:59:42.104 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:42.104+0000 7fa1e4540f40 1 -- 172.21.15.32:0/2758706093 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:120 12.1f 12:fe07bba1:::100.00000000.inode:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120) -- 0x55a74a7cad30 con 0x55a74a7c0920 2024-08-03T20:59:42.106 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:42.105+0000 7fa1de000640 1 -- 172.21.15.32:0/2758706093 <== osd.6 v2:172.21.15.118:6825/801947386 21 ==== osd_op_reply(120 100.00000000.inode [omap-set-header] v120'433 uv433 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7fa1d800b040 con 0x55a74a7c0920 2024-08-03T20:59:42.106 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:42.105+0000 7fa1e4540f40 1 -- 172.21.15.32:0/2758706093 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:121 12.1f 12:ff5b34d6:::1.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120) -- 0x55a74a7cac70 con 0x55a74a7c0920 2024-08-03T20:59:42.107 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:42.107+0000 7fa1de000640 1 -- 172.21.15.32:0/2758706093 <== osd.6 v2:172.21.15.118:6825/801947386 22 ==== osd_op_reply(121 1.00000000 [create] v120'434 uv427 ondisk = -17 ((17) File exists)) ==== 154+0+0 (crc 0 0 0) 0x7fa1d800b040 con 0x55a74a7c0920 2024-08-03T20:59:42.107 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:42.107+0000 7fa1e4540f40 1 -- 172.21.15.32:0/2758706093 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:122 12.1f 12:ff5b34d6:::1.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e120) -- 0x55a74a7cac70 con 0x55a74a7c0920 2024-08-03T20:59:42.108 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:42.108+0000 7fa1de000640 1 -- 172.21.15.32:0/2758706093 <== osd.6 v2:172.21.15.118:6825/801947386 23 ==== osd_op_reply(122 1.00000000 [delete] v120'435 uv435 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7fa1d800b040 con 0x55a74a7c0920 2024-08-03T20:59:42.108 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:42.108+0000 7fa1e4540f40 1 -- 172.21.15.32:0/2758706093 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:123 12.1f 12:ff5b34d6:::1.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120) -- 0x55a74a7cac70 con 0x55a74a7c0920 2024-08-03T20:59:42.110 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:42.109+0000 7fa1de000640 1 -- 172.21.15.32:0/2758706093 <== osd.6 v2:172.21.15.118:6825/801947386 24 ==== osd_op_reply(123 1.00000000 [create] v120'436 uv436 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7fa1d800b040 con 0x55a74a7c0920 2024-08-03T20:59:42.110 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::1.00000000:head# 2024-08-03T20:59:42.110 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:42.109+0000 7fa1e4540f40 1 -- 172.21.15.32:0/2758706093 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:124 12.1f 12:ff5b34d6:::1.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120) -- 0x55a74a7cbc80 con 0x55a74a7c0920 2024-08-03T20:59:42.111 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:42.111+0000 7fa1de000640 1 -- 172.21.15.32:0/2758706093 <== osd.6 v2:172.21.15.118:6825/801947386 25 ==== osd_op_reply(124 1.00000000 [setxattr (30)] v120'437 uv437 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7fa1d800b040 con 0x55a74a7c0920 2024-08-03T20:59:42.111 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:42.111+0000 7fa1e4540f40 1 -- 172.21.15.32:0/2758706093 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:125 12.1f 12:ff5b34d6:::1.00000000:head [setxattr parent (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120) -- 0x55a74a7cbc80 con 0x55a74a7c0920 2024-08-03T20:59:42.112 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:42.112+0000 7fa1de000640 1 -- 172.21.15.32:0/2758706093 <== osd.6 v2:172.21.15.118:6825/801947386 26 ==== osd_op_reply(125 1.00000000 [setxattr (30)] v120'438 uv438 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7fa1d800b040 con 0x55a74a7c0920 2024-08-03T20:59:42.112 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:42.112+0000 7fa1e4540f40 1 -- 172.21.15.32:0/2758706093 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:126 12.1f 12:ff5b34d6:::1.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120) -- 0x55a74a7cac70 con 0x55a74a7c0920 2024-08-03T20:59:42.114 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:42.113+0000 7fa1de000640 1 -- 172.21.15.32:0/2758706093 <== osd.6 v2:172.21.15.118:6825/801947386 27 ==== osd_op_reply(126 1.00000000 [omap-set-header] v120'439 uv439 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7fa1d800b040 con 0x55a74a7c0920 2024-08-03T20:59:42.114 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:42.113+0000 7fa1e4540f40 1 -- 172.21.15.32:0/2758706093 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:127 12.1f 12:ff5b34d6:::1.00000000:head [omap-set-vals in=584b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120) -- 0x55a74a7cb120 con 0x55a74a7c0920 2024-08-03T20:59:42.116 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:42.115+0000 7fa1de000640 1 -- 172.21.15.32:0/2758706093 <== osd.6 v2:172.21.15.118:6825/801947386 28 ==== osd_op_reply(127 1.00000000 [omap-set-vals] v120'440 uv440 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7fa1d800b040 con 0x55a74a7c0920 2024-08-03T20:59:42.116 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:42.115+0000 7fa1e4540f40 1 -- 172.21.15.32:0/2758706093 >> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] conn(0x55a74a63b2e0 msgr2=0x55a74a70a8d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:59:42.116 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:42.115+0000 7fa1e4540f40 1 --2- 172.21.15.32:0/2758706093 >> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] conn(0x55a74a63b2e0 0x55a74a70a8d0 crc :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:59:42.116 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:42.115+0000 7fa1e4540f40 1 -- 172.21.15.32:0/2758706093 >> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] conn(0x55a74a7bc930 msgr2=0x55a74a7bf0d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:59:42.116 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:42.115+0000 7fa1e4540f40 1 --2- 172.21.15.32:0/2758706093 >> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] conn(0x55a74a7bc930 0x55a74a7bf0d0 crc :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:59:42.116 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:42.115+0000 7fa1e4540f40 1 -- 172.21.15.32:0/2758706093 >> [v2:172.21.15.32:6800/9088303,v1:172.21.15.32:6801/9088303] conn(0x55a74a7c7e50 msgr2=0x55a74a7ca730 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:59:42.116 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:42.115+0000 7fa1e4540f40 1 --2- 172.21.15.32:0/2758706093 >> [v2:172.21.15.32:6800/9088303,v1:172.21.15.32:6801/9088303] conn(0x55a74a7c7e50 0x55a74a7ca730 crc :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:59:42.117 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:42.115+0000 7fa1e4540f40 1 -- 172.21.15.32:0/2758706093 >> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] conn(0x55a74a7b78a0 msgr2=0x55a74a7b9cc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:59:42.117 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:42.115+0000 7fa1e4540f40 1 --2- 172.21.15.32:0/2758706093 >> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] conn(0x55a74a7b78a0 0x55a74a7b9cc0 crc :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:59:42.117 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:42.115+0000 7fa1e4540f40 1 -- 172.21.15.32:0/2758706093 >> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] conn(0x55a74a7c0920 msgr2=0x55a74a7c0d30 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:59:42.117 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:42.115+0000 7fa1e4540f40 1 --2- 172.21.15.32:0/2758706093 >> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] conn(0x55a74a7c0920 0x55a74a7c0d30 crc :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:59:42.117 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:42.115+0000 7fa1e4540f40 1 -- 172.21.15.32:0/2758706093 >> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] conn(0x55a74a7c3fd0 msgr2=0x55a74a7c43e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:59:42.117 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:42.115+0000 7fa1e4540f40 1 --2- 172.21.15.32:0/2758706093 >> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] conn(0x55a74a7c3fd0 0x55a74a7c43e0 crc :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:59:42.117 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:42.115+0000 7fa1de000640 1 -- 172.21.15.32:0/2758706093 reap_dead start 2024-08-03T20:59:42.117 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:42.115+0000 7fa1de000640 1 -- 172.21.15.32:0/2758706093 reap_dead start 2024-08-03T20:59:42.117 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:42.115+0000 7fa1de000640 1 -- 172.21.15.32:0/2758706093 reap_dead start 2024-08-03T20:59:42.117 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:42.115+0000 7fa1e4540f40 1 -- 172.21.15.32:0/2758706093 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fa1c00759f0 msgr2=0x7fa1c0077f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:59:42.117 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:42.115+0000 7fa1e4540f40 1 --2- 172.21.15.32:0/2758706093 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fa1c00759f0 0x7fa1c0077f00 secure :-1 s=READY pgs=788 cs=0 l=1 rev1=1 crypto rx=0x7fa1cc062000 tx=0x7fa1cc004940 comp rx=0 tx=0).stop 2024-08-03T20:59:42.117 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:42.115+0000 7fa1e4540f40 1 -- 172.21.15.32:0/2758706093 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a74a6f3360 msgr2=0x55a74a7a9d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:59:42.117 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:42.115+0000 7fa1e4540f40 1 --2- 172.21.15.32:0/2758706093 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a74a6f3360 0x55a74a7a9d00 secure :-1 s=READY pgs=1968 cs=0 l=1 rev1=1 crypto rx=0x7fa1d0002cc0 tx=0x7fa1d000c620 comp rx=0 tx=0).stop 2024-08-03T20:59:42.118 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:42.115+0000 7fa1e4540f40 1 -- 172.21.15.32:0/2758706093 shutdown_connections 2024-08-03T20:59:42.118 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:42.115+0000 7fa1e4540f40 1 --2- 172.21.15.32:0/2758706093 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fa1c00759f0 0x7fa1c0077f00 unknown :-1 s=CLOSED pgs=788 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:59:42.118 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:42.115+0000 7fa1e4540f40 1 --2- 172.21.15.32:0/2758706093 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a74a6f3360 0x55a74a7a9d00 unknown :-1 s=CLOSED pgs=1968 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:59:42.118 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:42.115+0000 7fa1e4540f40 1 -- 172.21.15.32:0/2758706093 >> 172.21.15.32:0/2758706093 conn(0x55a74a55cac0 msgr2=0x55a74a685070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:59:42.118 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:42.115+0000 7fa1e4540f40 1 -- 172.21.15.32:0/2758706093 shutdown_connections 2024-08-03T20:59:42.118 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:42.115+0000 7fa1e4540f40 1 -- 172.21.15.32:0/2758706093 wait complete. 2024-08-03T20:59:42.127 DEBUG:teuthology.orchestra.run.smithi032:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata setomapheader 600.00000000 deadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeef 2024-08-03T20:59:42.312 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:42.312+0000 7fe29b6cdf40 1 Processor -- start 2024-08-03T20:59:42.313 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:42.312+0000 7fe29b6cdf40 1 -- start start 2024-08-03T20:59:42.315 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:42.313+0000 7fe29b6cdf40 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55df9d288360 0x55df9d218490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:59:42.315 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:42.313+0000 7fe29b6cdf40 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55df9d2a5cb0 0x55df9d21a8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:59:42.315 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:42.313+0000 7fe29b6cdf40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55df9d2a63f0 0x55df9d212650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:59:42.315 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:42.313+0000 7fe29b6cdf40 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55df9d11fe30 con 0x55df9d288360 2024-08-03T20:59:42.316 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:42.313+0000 7fe29b6cdf40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x55df9d17dbf0 con 0x55df9d2a63f0 2024-08-03T20:59:42.316 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:42.313+0000 7fe29b6cdf40 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x55df9d289440 con 0x55df9d2a5cb0 2024-08-03T20:59:42.316 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:42.314+0000 7fe294800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55df9d288360 0x55df9d218490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:59:42.316 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:42.314+0000 7fe294800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55df9d288360 0x55df9d218490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:33150/0 (socket says 172.21.15.32:33150) 2024-08-03T20:59:42.316 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:42.314+0000 7fe294800640 1 -- 172.21.15.32:0/2103282015 learned_addr learned my addr 172.21.15.32:0/2103282015 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:59:42.316 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:42.314+0000 7fe293e00640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55df9d2a5cb0 0x55df9d21a8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:59:42.316 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:42.314+0000 7fe295200640 1 --2- 172.21.15.32:0/2103282015 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55df9d2a63f0 0x55df9d212650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:59:42.316 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:42.314+0000 7fe294800640 1 -- 172.21.15.32:0/2103282015 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55df9d2a5cb0 msgr2=0x55df9d21a8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:59:42.316 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:42.314+0000 7fe294800640 1 --2- 172.21.15.32:0/2103282015 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55df9d2a5cb0 0x55df9d21a8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:59:42.316 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:42.314+0000 7fe294800640 1 -- 172.21.15.32:0/2103282015 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55df9d2a63f0 msgr2=0x55df9d212650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:59:42.316 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:42.314+0000 7fe294800640 1 --2- 172.21.15.32:0/2103282015 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55df9d2a63f0 0x55df9d212650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:59:42.316 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:42.314+0000 7fe294800640 1 -- 172.21.15.32:0/2103282015 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55df9d29e710 con 0x55df9d288360 2024-08-03T20:59:42.316 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:42.314+0000 7fe294800640 1 --2- 172.21.15.32:0/2103282015 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55df9d288360 0x55df9d218490 secure :-1 s=READY pgs=1969 cs=0 l=1 rev1=1 crypto rx=0x7fe28800b7c0 tx=0x7fe28800bc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7842f59b870b550d server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:59:42.317 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:42.315+0000 7fe292a00640 1 -- 172.21.15.32:0/2103282015 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe28800abc0 con 0x55df9d288360 2024-08-03T20:59:42.317 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:42.315+0000 7fe292a00640 1 -- 172.21.15.32:0/2103282015 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fe28800ad60 con 0x55df9d288360 2024-08-03T20:59:42.317 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:42.315+0000 7fe292a00640 1 -- 172.21.15.32:0/2103282015 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe288015680 con 0x55df9d288360 2024-08-03T20:59:42.318 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:42.315+0000 7fe29b6cdf40 1 -- 172.21.15.32:0/2103282015 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55df9d288360 msgr2=0x55df9d218490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:59:42.318 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:42.315+0000 7fe29b6cdf40 1 --2- 172.21.15.32:0/2103282015 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55df9d288360 0x55df9d218490 secure :-1 s=READY pgs=1969 cs=0 l=1 rev1=1 crypto rx=0x7fe28800b7c0 tx=0x7fe28800bc80 comp rx=0 tx=0).stop 2024-08-03T20:59:42.318 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:42.315+0000 7fe29b6cdf40 1 -- 172.21.15.32:0/2103282015 shutdown_connections 2024-08-03T20:59:42.318 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:42.315+0000 7fe29b6cdf40 1 --2- 172.21.15.32:0/2103282015 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55df9d2a63f0 0x55df9d212650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:59:42.318 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:42.315+0000 7fe29b6cdf40 1 --2- 172.21.15.32:0/2103282015 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55df9d2a5cb0 0x55df9d21a8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:59:42.319 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:42.315+0000 7fe29b6cdf40 1 --2- 172.21.15.32:0/2103282015 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55df9d288360 0x55df9d218490 unknown :-1 s=CLOSED pgs=1969 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:59:42.319 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:42.315+0000 7fe29b6cdf40 1 -- 172.21.15.32:0/2103282015 >> 172.21.15.32:0/2103282015 conn(0x55df9d0f1ac0 msgr2=0x55df9d2a0f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:59:42.319 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:42.315+0000 7fe29b6cdf40 1 -- 172.21.15.32:0/2103282015 shutdown_connections 2024-08-03T20:59:42.319 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:42.315+0000 7fe29b6cdf40 1 -- 172.21.15.32:0/2103282015 wait complete. 2024-08-03T20:59:42.319 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:42.316+0000 7fe29b6cdf40 1 Processor -- start 2024-08-03T20:59:42.319 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:42.317+0000 7fe29b6cdf40 1 -- start start 2024-08-03T20:59:42.322 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:42.318+0000 7fe29b6cdf40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55df9d288360 0x55df9d1cf3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:59:42.322 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:42.318+0000 7fe29b6cdf40 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55df9d2a5cb0 0x55df9d27e2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:59:42.323 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:42.318+0000 7fe29b6cdf40 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55df9d2a63f0 0x55df9d287040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:59:42.323 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:42.318+0000 7fe29b6cdf40 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55df9d2189d0 con 0x55df9d2a5cb0 2024-08-03T20:59:42.323 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:42.318+0000 7fe29b6cdf40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x55df9d1cfed0 con 0x55df9d288360 2024-08-03T20:59:42.323 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:42.318+0000 7fe29b6cdf40 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x55df9d219ed0 con 0x55df9d2a63f0 2024-08-03T20:59:42.323 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:42.318+0000 7fe293e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55df9d2a5cb0 0x55df9d27e2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:59:42.323 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:42.318+0000 7fe293e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55df9d2a5cb0 0x55df9d27e2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:33156/0 (socket says 172.21.15.32:33156) 2024-08-03T20:59:42.323 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:42.318+0000 7fe293e00640 1 -- 172.21.15.32:0/2790515774 learned_addr learned my addr 172.21.15.32:0/2790515774 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:59:42.323 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:42.319+0000 7fe294800640 1 --2- 172.21.15.32:0/2790515774 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55df9d288360 0x55df9d1cf3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:59:42.323 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:42.319+0000 7fe295200640 1 --2- 172.21.15.32:0/2790515774 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55df9d2a63f0 0x55df9d287040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:59:42.324 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:42.319+0000 7fe293e00640 1 -- 172.21.15.32:0/2790515774 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55df9d2a63f0 msgr2=0x55df9d287040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:59:42.324 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:42.319+0000 7fe293e00640 1 --2- 172.21.15.32:0/2790515774 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55df9d2a63f0 0x55df9d287040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:59:42.324 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:42.319+0000 7fe293e00640 1 -- 172.21.15.32:0/2790515774 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55df9d288360 msgr2=0x55df9d1cf3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:59:42.324 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:42.319+0000 7fe293e00640 1 --2- 172.21.15.32:0/2790515774 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55df9d288360 0x55df9d1cf3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:59:42.324 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:42.319+0000 7fe293e00640 1 -- 172.21.15.32:0/2790515774 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55df9d27c7d0 con 0x55df9d2a5cb0 2024-08-03T20:59:42.324 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:42.319+0000 7fe293e00640 1 --2- 172.21.15.32:0/2790515774 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55df9d2a5cb0 0x55df9d27e2d0 secure :-1 s=READY pgs=1970 cs=0 l=1 rev1=1 crypto rx=0x7fe280002ed0 tx=0x7fe28000c7a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:59:42.324 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:42.319+0000 7fe291600640 1 -- 172.21.15.32:0/2790515774 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe28000f810 con 0x55df9d2a5cb0 2024-08-03T20:59:42.324 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:42.319+0000 7fe291600640 1 -- 172.21.15.32:0/2790515774 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fe280013070 con 0x55df9d2a5cb0 2024-08-03T20:59:42.325 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:42.319+0000 7fe29b6cdf40 1 -- 172.21.15.32:0/2790515774 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x55df9d27ca90 con 0x55df9d2a5cb0 2024-08-03T20:59:42.325 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:42.319+0000 7fe291600640 1 -- 172.21.15.32:0/2790515774 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe28000fd40 con 0x55df9d2a5cb0 2024-08-03T20:59:42.325 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:42.319+0000 7fe29b6cdf40 1 -- 172.21.15.32:0/2790515774 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55df9d2801b0 con 0x55df9d2a5cb0 2024-08-03T20:59:42.325 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:42.322+0000 7fe291600640 1 -- 172.21.15.32:0/2790515774 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7fe280017420 con 0x55df9d2a5cb0 2024-08-03T20:59:42.328 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:42.322+0000 7fe291600640 1 --2- 172.21.15.32:0/2790515774 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fe2740759f0 0x7fe274077f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:59:42.328 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:42.323+0000 7fe291600640 1 -- 172.21.15.32:0/2790515774 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(120..120 src has 1..120) ==== 8864+0+0 (secure 0 0 0) 0x7fe280098770 con 0x55df9d2a5cb0 2024-08-03T20:59:42.328 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:42.323+0000 7fe294800640 1 --2- 172.21.15.32:0/2790515774 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fe2740759f0 0x7fe274077f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:59:42.328 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:42.323+0000 7fe29b6cdf40 1 --2- 172.21.15.32:0/2790515774 >> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] conn(0x55df9d27bb90 0x55df9d34afd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:59:42.328 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:42.323+0000 7fe29b6cdf40 1 -- 172.21.15.32:0/2790515774 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:1 12.10 12:0d82a743:::600.00000000:head [omap-set-header in=80b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120) -- 0x55df9d27bfa0 con 0x55df9d27bb90 2024-08-03T20:59:42.328 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:42.323+0000 7fe295200640 1 --2- 172.21.15.32:0/2790515774 >> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] conn(0x55df9d27bb90 0x55df9d34afd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:59:42.329 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:42.324+0000 7fe294800640 1 --2- 172.21.15.32:0/2790515774 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fe2740759f0 0x7fe274077f00 secure :-1 s=READY pgs=789 cs=0 l=1 rev1=1 crypto rx=0x7fe28800b760 tx=0x7fe288017040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:59:42.329 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:42.324+0000 7fe295200640 1 --2- 172.21.15.32:0/2790515774 >> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] conn(0x55df9d27bb90 0x55df9d34afd0 crc :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:59:42.329 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:42.326+0000 7fe295200640 1 -- 172.21.15.32:0/2790515774 <== osd.0 v2:172.21.15.32:6816/3066156116 1 ==== osd_op_reply(1 600.00000000 [omap-set-header] v120'1096 uv1096 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe28c00b040 con 0x55df9d27bb90 2024-08-03T20:59:42.329 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:42.326+0000 7fe29b6cdf40 1 -- 172.21.15.32:0/2790515774 >> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] conn(0x55df9d27bb90 msgr2=0x55df9d34afd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:59:42.329 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:42.326+0000 7fe29b6cdf40 1 --2- 172.21.15.32:0/2790515774 >> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] conn(0x55df9d27bb90 0x55df9d34afd0 crc :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:59:42.329 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:42.326+0000 7fe29b6cdf40 1 -- 172.21.15.32:0/2790515774 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fe2740759f0 msgr2=0x7fe274077f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:59:42.329 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:42.326+0000 7fe29b6cdf40 1 --2- 172.21.15.32:0/2790515774 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fe2740759f0 0x7fe274077f00 secure :-1 s=READY pgs=789 cs=0 l=1 rev1=1 crypto rx=0x7fe28800b760 tx=0x7fe288017040 comp rx=0 tx=0).stop 2024-08-03T20:59:42.329 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:42.326+0000 7fe29b6cdf40 1 -- 172.21.15.32:0/2790515774 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55df9d2a5cb0 msgr2=0x55df9d27e2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:59:42.330 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:42.326+0000 7fe29b6cdf40 1 --2- 172.21.15.32:0/2790515774 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55df9d2a5cb0 0x55df9d27e2d0 secure :-1 s=READY pgs=1970 cs=0 l=1 rev1=1 crypto rx=0x7fe280002ed0 tx=0x7fe28000c7a0 comp rx=0 tx=0).stop 2024-08-03T20:59:42.330 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:42.326+0000 7fe295200640 1 -- 172.21.15.32:0/2790515774 reap_dead start 2024-08-03T20:59:42.330 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:42.326+0000 7fe29b6cdf40 1 -- 172.21.15.32:0/2790515774 shutdown_connections 2024-08-03T20:59:42.330 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:42.326+0000 7fe29b6cdf40 1 -- 172.21.15.32:0/2790515774 >> 172.21.15.32:0/2790515774 conn(0x55df9d0f1ac0 msgr2=0x55df9d21a8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:59:42.330 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:42.326+0000 7fe29b6cdf40 1 -- 172.21.15.32:0/2790515774 shutdown_connections 2024-08-03T20:59:42.330 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:42.326+0000 7fe29b6cdf40 1 -- 172.21.15.32:0/2790515774 wait complete. 2024-08-03T20:59:42.341 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs joinable true 2024-08-03T20:59:42.572 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:42.571+0000 7f6af6600640 1 -- 172.21.15.32:0/4146928702 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6af00cccf0 msgr2=0x7f6af00cd100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:59:42.573 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:42.571+0000 7f6af6600640 1 --2- 172.21.15.32:0/4146928702 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6af00cccf0 0x7f6af00cd100 secure :-1 s=READY pgs=1971 cs=0 l=1 rev1=1 crypto rx=0x7f6af00cca80 tx=0x7f6adc02a560 comp rx=0 tx=0).stop 2024-08-03T20:59:42.573 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:42.572+0000 7f6af6600640 1 -- 172.21.15.32:0/4146928702 shutdown_connections 2024-08-03T20:59:42.573 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:42.572+0000 7f6af6600640 1 --2- 172.21.15.32:0/4146928702 >> v2:172.21.15.118:3300/0 conn(0x7f6af00ced30 0x7f6af00cf140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:59:42.573 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:42.572+0000 7f6af6600640 1 --2- 172.21.15.32:0/4146928702 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6af00cccf0 0x7f6af00cd100 unknown :-1 s=CLOSED pgs=1971 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:59:42.573 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:42.572+0000 7f6af6600640 1 -- 172.21.15.32:0/4146928702 >> 172.21.15.32:0/4146928702 conn(0x7f6af00b1bd0 msgr2=0x7f6af00b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:59:42.573 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:42.572+0000 7f6af6600640 1 -- 172.21.15.32:0/4146928702 shutdown_connections 2024-08-03T20:59:42.573 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:42.572+0000 7f6af6600640 1 -- 172.21.15.32:0/4146928702 wait complete. 2024-08-03T20:59:42.574 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:42.574+0000 7f6af6600640 1 Processor -- start 2024-08-03T20:59:42.574 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:42.574+0000 7f6af6600640 1 -- start start 2024-08-03T20:59:42.576 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:42.575+0000 7f6af6600640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f6af00cccf0 0x7f6af01538d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:59:42.577 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:42.575+0000 7f6af6600640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6af00ced30 0x7f6af013e6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:59:42.577 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:42.575+0000 7f6af6600640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f6af0145c50 0x7f6af0073840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:59:42.577 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:42.575+0000 7f6af6600640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f6af00b4070 con 0x7f6af00ced30 2024-08-03T20:59:42.577 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:42.575+0000 7f6af6600640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f6af00b3ef0 con 0x7f6af0145c50 2024-08-03T20:59:42.577 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:42.575+0000 7f6af6600640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7f6af00b41f0 con 0x7f6af00cccf0 2024-08-03T20:59:42.577 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:42.576+0000 7f6aeea00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6af00ced30 0x7f6af013e6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:59:42.577 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:42.576+0000 7f6aeea00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6af00ced30 0x7f6af013e6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:33162/0 (socket says 172.21.15.32:33162) 2024-08-03T20:59:42.577 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:42.576+0000 7f6aeea00640 1 -- 172.21.15.32:0/1991312337 learned_addr learned my addr 172.21.15.32:0/1991312337 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:59:42.577 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:42.576+0000 7f6aefe00640 1 --2- 172.21.15.32:0/1991312337 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f6af0145c50 0x7f6af0073840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:59:42.578 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:42.576+0000 7f6aef400640 1 --2- 172.21.15.32:0/1991312337 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f6af00cccf0 0x7f6af01538d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:59:42.578 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:42.576+0000 7f6aeea00640 1 -- 172.21.15.32:0/1991312337 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f6af00cccf0 msgr2=0x7f6af01538d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:59:42.578 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:42.576+0000 7f6aeea00640 1 --2- 172.21.15.32:0/1991312337 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f6af00cccf0 0x7f6af01538d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:59:42.578 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:42.576+0000 7f6aeea00640 1 -- 172.21.15.32:0/1991312337 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f6af0145c50 msgr2=0x7f6af0073840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:59:42.578 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:42.576+0000 7f6aeea00640 1 --2- 172.21.15.32:0/1991312337 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f6af0145c50 0x7f6af0073840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:59:42.578 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:42.576+0000 7f6aeea00640 1 -- 172.21.15.32:0/1991312337 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6af013fe80 con 0x7f6af00ced30 2024-08-03T20:59:42.579 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:42.577+0000 7f6aeea00640 1 --2- 172.21.15.32:0/1991312337 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6af00ced30 0x7f6af013e6c0 secure :-1 s=READY pgs=1972 cs=0 l=1 rev1=1 crypto rx=0x7f6adc034040 tx=0x7f6adc003910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:59:42.579 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:42.577+0000 7f6ae7400640 1 -- 172.21.15.32:0/1991312337 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6adc03e030 con 0x7f6af00ced30 2024-08-03T20:59:42.580 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:42.577+0000 7f6ae7400640 1 -- 172.21.15.32:0/1991312337 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f6adc003e00 con 0x7f6af00ced30 2024-08-03T20:59:42.580 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:42.577+0000 7f6ae7400640 1 -- 172.21.15.32:0/1991312337 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6adc035040 con 0x7f6af00ced30 2024-08-03T20:59:42.580 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:42.577+0000 7f6af6600640 1 -- 172.21.15.32:0/1991312337 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6af0140160 con 0x7f6af00ced30 2024-08-03T20:59:42.580 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:42.577+0000 7f6af6600640 1 -- 172.21.15.32:0/1991312337 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6af0140720 con 0x7f6af00ced30 2024-08-03T20:59:42.580 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:42.578+0000 7f6af6600640 1 -- 172.21.15.32:0/1991312337 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6af0141f40 con 0x7f6af00ced30 2024-08-03T20:59:42.580 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:42.579+0000 7f6ae7400640 1 -- 172.21.15.32:0/1991312337 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f6adc041030 con 0x7f6af00ced30 2024-08-03T20:59:42.580 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:42.580+0000 7f6ae7400640 1 --2- 172.21.15.32:0/1991312337 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f6ad0077010 0x7f6ad0079520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:59:42.581 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:42.580+0000 7f6ae7400640 1 -- 172.21.15.32:0/1991312337 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(120..120 src has 1..120) ==== 8864+0+0 (secure 0 0 0) 0x7f6adc0b9c60 con 0x7f6af00ced30 2024-08-03T20:59:42.581 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:42.581+0000 7f6aef400640 1 --2- 172.21.15.32:0/1991312337 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f6ad0077010 0x7f6ad0079520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:59:42.582 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:42.581+0000 7f6aef400640 1 --2- 172.21.15.32:0/1991312337 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f6ad0077010 0x7f6ad0079520 secure :-1 s=READY pgs=790 cs=0 l=1 rev1=1 crypto rx=0x7f6ae0001720 tx=0x7f6ae0009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:59:42.585 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:42.585+0000 7f6ae7400640 1 -- 172.21.15.32:0/1991312337 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7f6adc081bc0 con 0x7f6af00ced30 2024-08-03T20:59:42.836 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:42.836+0000 7f6af6600640 1 -- 172.21.15.32:0/1991312337 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "joinable", "val": "true"} v 0) -- 0x7f6af00b4710 con 0x7f6af00ced30 2024-08-03T20:59:43.736 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:43.736+0000 7f6ae7400640 1 -- 172.21.15.32:0/1991312337 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "joinable", "val": "true"}]=0 cephfs marked joinable; MDS may join as newly active. v296) ==== 162+0+0 (secure 0 0 0) 0x7f6adc086110 con 0x7f6af00ced30 2024-08-03T20:59:43.737 INFO:teuthology.orchestra.run.smithi032.stderr:cephfs marked joinable; MDS may join as newly active. 2024-08-03T20:59:43.740 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:43.739+0000 7f6af6600640 1 -- 172.21.15.32:0/1991312337 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f6ad0077010 msgr2=0x7f6ad0079520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:59:43.740 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:43.739+0000 7f6af6600640 1 --2- 172.21.15.32:0/1991312337 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f6ad0077010 0x7f6ad0079520 secure :-1 s=READY pgs=790 cs=0 l=1 rev1=1 crypto rx=0x7f6ae0001720 tx=0x7f6ae0009040 comp rx=0 tx=0).stop 2024-08-03T20:59:43.740 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:43.739+0000 7f6af6600640 1 -- 172.21.15.32:0/1991312337 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6af00ced30 msgr2=0x7f6af013e6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:59:43.740 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:43.739+0000 7f6af6600640 1 --2- 172.21.15.32:0/1991312337 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6af00ced30 0x7f6af013e6c0 secure :-1 s=READY pgs=1972 cs=0 l=1 rev1=1 crypto rx=0x7f6adc034040 tx=0x7f6adc003910 comp rx=0 tx=0).stop 2024-08-03T20:59:43.740 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:43.739+0000 7f6af6600640 1 -- 172.21.15.32:0/1991312337 shutdown_connections 2024-08-03T20:59:43.740 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:43.739+0000 7f6af6600640 1 --2- 172.21.15.32:0/1991312337 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f6ad0077010 0x7f6ad0079520 unknown :-1 s=CLOSED pgs=790 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:59:43.740 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:43.739+0000 7f6af6600640 1 --2- 172.21.15.32:0/1991312337 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f6af0145c50 0x7f6af0073840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:59:43.740 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:43.739+0000 7f6af6600640 1 --2- 172.21.15.32:0/1991312337 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6af00ced30 0x7f6af013e6c0 unknown :-1 s=CLOSED pgs=1972 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:59:43.740 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:43.739+0000 7f6af6600640 1 --2- 172.21.15.32:0/1991312337 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f6af00cccf0 0x7f6af01538d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:59:43.740 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:43.739+0000 7f6af6600640 1 -- 172.21.15.32:0/1991312337 >> 172.21.15.32:0/1991312337 conn(0x7f6af00b1bd0 msgr2=0x7f6af015b9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:59:43.741 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:43.739+0000 7f6af6600640 1 -- 172.21.15.32:0/1991312337 shutdown_connections 2024-08-03T20:59:43.741 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:43.739+0000 7f6af6600640 1 -- 172.21.15.32:0/1991312337 wait complete. 2024-08-03T20:59:43.774 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-08-03T20:59:44.006 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:44.005+0000 7fc7d0400640 1 -- 172.21.15.32:0/378490717 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc7c8073e60 msgr2=0x7fc7c8074270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:59:44.006 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:44.005+0000 7fc7d0400640 1 --2- 172.21.15.32:0/378490717 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc7c8073e60 0x7fc7c8074270 secure :-1 s=READY pgs=1973 cs=0 l=1 rev1=1 crypto rx=0x7fc7c8071700 tx=0x7fc7b402a560 comp rx=0 tx=0).stop 2024-08-03T20:59:44.007 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:44.005+0000 7fc7d0400640 1 -- 172.21.15.32:0/378490717 shutdown_connections 2024-08-03T20:59:44.007 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:44.005+0000 7fc7d0400640 1 --2- 172.21.15.32:0/378490717 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fc7c80747b0 0x7fc7c80cc8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:59:44.007 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:44.005+0000 7fc7d0400640 1 --2- 172.21.15.32:0/378490717 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc7c8073e60 0x7fc7c8074270 unknown :-1 s=CLOSED pgs=1973 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:59:44.007 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:44.005+0000 7fc7d0400640 1 --2- 172.21.15.32:0/378490717 >> v2:172.21.15.118:3300/0 conn(0x7fc7c806e930 0x7fc7c8144f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:59:44.007 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:44.005+0000 7fc7d0400640 1 -- 172.21.15.32:0/378490717 >> 172.21.15.32:0/378490717 conn(0x7fc7c80b1bd0 msgr2=0x7fc7c80b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:59:44.007 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:44.005+0000 7fc7d0400640 1 -- 172.21.15.32:0/378490717 shutdown_connections 2024-08-03T20:59:44.007 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:44.005+0000 7fc7d0400640 1 -- 172.21.15.32:0/378490717 wait complete. 2024-08-03T20:59:44.008 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:44.008+0000 7fc7d0400640 1 Processor -- start 2024-08-03T20:59:44.008 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:44.008+0000 7fc7d0400640 1 -- start start 2024-08-03T20:59:44.011 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:44.009+0000 7fc7d0400640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fc7c806e930 0x7fc7c813cfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:59:44.011 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:44.009+0000 7fc7d0400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc7c8073e60 0x7fc7c81f4e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:59:44.011 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:44.009+0000 7fc7d0400640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc7c80747b0 0x7fc7c8143aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:59:44.012 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:44.009+0000 7fc7d0400640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fc7c80cf9c0 con 0x7fc7c8073e60 2024-08-03T20:59:44.012 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:44.009+0000 7fc7d0400640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7fc7c80cf840 con 0x7fc7c80747b0 2024-08-03T20:59:44.012 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:44.009+0000 7fc7d0400640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7fc7c80cce20 con 0x7fc7c806e930 2024-08-03T20:59:44.012 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:44.009+0000 7fc7cd000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc7c8073e60 0x7fc7c81f4e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:59:44.012 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:44.009+0000 7fc7cd000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc7c8073e60 0x7fc7c81f4e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:33170/0 (socket says 172.21.15.32:33170) 2024-08-03T20:59:44.012 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:44.009+0000 7fc7cd000640 1 -- 172.21.15.32:0/1311169601 learned_addr learned my addr 172.21.15.32:0/1311169601 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:59:44.012 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:44.009+0000 7fc7cda00640 1 --2- 172.21.15.32:0/1311169601 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fc7c806e930 0x7fc7c813cfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:59:44.012 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:44.010+0000 7fc7ce400640 1 --2- 172.21.15.32:0/1311169601 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc7c80747b0 0x7fc7c8143aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:59:44.012 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:44.010+0000 7fc7cd000640 1 -- 172.21.15.32:0/1311169601 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fc7c806e930 msgr2=0x7fc7c813cfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:59:44.012 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:44.010+0000 7fc7cd000640 1 --2- 172.21.15.32:0/1311169601 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fc7c806e930 0x7fc7c813cfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:59:44.012 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:44.010+0000 7fc7cd000640 1 -- 172.21.15.32:0/1311169601 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc7c80747b0 msgr2=0x7fc7c8143aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:59:44.013 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:44.010+0000 7fc7cd000640 1 --2- 172.21.15.32:0/1311169601 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc7c80747b0 0x7fc7c8143aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:59:44.013 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:44.010+0000 7fc7cd000640 1 -- 172.21.15.32:0/1311169601 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc7c81f8980 con 0x7fc7c8073e60 2024-08-03T20:59:44.013 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:44.010+0000 7fc7cd000640 1 --2- 172.21.15.32:0/1311169601 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc7c8073e60 0x7fc7c81f4e50 secure :-1 s=READY pgs=1974 cs=0 l=1 rev1=1 crypto rx=0x7fc7b4000c00 tx=0x7fc7b4003910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:59:44.013 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:44.011+0000 7fc7c5600640 1 -- 172.21.15.32:0/1311169601 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7b403e030 con 0x7fc7c8073e60 2024-08-03T20:59:44.013 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:44.011+0000 7fc7c5600640 1 -- 172.21.15.32:0/1311169601 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fc7b4003e00 con 0x7fc7c8073e60 2024-08-03T20:59:44.013 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:44.011+0000 7fc7c5600640 1 -- 172.21.15.32:0/1311169601 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7b4035040 con 0x7fc7c8073e60 2024-08-03T20:59:44.013 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:44.011+0000 7fc7d0400640 1 -- 172.21.15.32:0/1311169601 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc7c81f6230 con 0x7fc7c8073e60 2024-08-03T20:59:44.013 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:44.011+0000 7fc7d0400640 1 -- 172.21.15.32:0/1311169601 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc7c81f8f20 con 0x7fc7c8073e60 2024-08-03T20:59:44.014 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:44.012+0000 7fc7d0400640 1 -- 172.21.15.32:0/1311169601 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc7c8074270 con 0x7fc7c8073e60 2024-08-03T20:59:44.014 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:44.013+0000 7fc7c5600640 1 -- 172.21.15.32:0/1311169601 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7fc7b4041030 con 0x7fc7c8073e60 2024-08-03T20:59:44.016 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:44.014+0000 7fc7c5600640 1 --2- 172.21.15.32:0/1311169601 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fc7a8077010 0x7fc7a8079520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:59:44.016 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:44.014+0000 7fc7c5600640 1 -- 172.21.15.32:0/1311169601 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(120..120 src has 1..120) ==== 8864+0+0 (secure 0 0 0) 0x7fc7b40b9c30 con 0x7fc7c8073e60 2024-08-03T20:59:44.016 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:44.014+0000 7fc7cda00640 1 --2- 172.21.15.32:0/1311169601 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fc7a8077010 0x7fc7a8079520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:59:44.016 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:44.015+0000 7fc7cda00640 1 --2- 172.21.15.32:0/1311169601 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fc7a8077010 0x7fc7a8079520 secure :-1 s=READY pgs=791 cs=0 l=1 rev1=1 crypto rx=0x7fc7bc00bb50 tx=0x7fc7bc00b2e0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:59:44.018 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:44.018+0000 7fc7c5600640 1 -- 172.21.15.32:0/1311169601 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7fc7b4081b90 con 0x7fc7c8073e60 2024-08-03T20:59:44.261 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:44.261+0000 7fc7d0400640 1 -- 172.21.15.32:0/1311169601 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc7c81f5fc0 con 0x7fc7c8073e60 2024-08-03T20:59:44.262 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:44.261+0000 7fc7c5600640 1 -- 172.21.15.32:0/1311169601 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 297 v297) ==== 95+0+5226 (secure 0 0 0) 0x7fc7b408a420 con 0x7fc7c8073e60 2024-08-03T20:59:44.262 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-03T20:59:44.262 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":297,"btime":"2024-08-03T20:59:43:739633+0000","default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6439,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6835/531080160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6834","nonce":531080160},{"type":"v1","addr":"172.21.15.118:6835","nonce":531080160}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":84},{"gid":6454,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6837/2796882319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6836","nonce":2796882319},{"type":"v1","addr":"172.21.15.118:6837","nonce":2796882319}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":86},{"gid":6813,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6834/2686933890","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":2686933890},{"type":"v1","addr":"172.21.15.32:6834","nonce":2686933890}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":85}],"filesystems":[{"mdsmap":{"epoch":297,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-08-03T20:54:26.950518+0000","modified":"2024-08-03T20:59:43.739615+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":120,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":9969},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9969":{"gid":9969,"name":"a","rank":0,"incarnation":297,"state":"up:replay","state_seq":1,"addr":"172.21.15.32:6835/3127944613","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6833","nonce":3127944613},{"type":"v1","addr":"172.21.15.32:6835","nonce":3127944613}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":6}]} 2024-08-03T20:59:44.263 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 297 2024-08-03T20:59:44.264 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:44.264+0000 7fc7d0400640 1 -- 172.21.15.32:0/1311169601 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fc7a8077010 msgr2=0x7fc7a8079520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:59:44.265 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:44.264+0000 7fc7d0400640 1 --2- 172.21.15.32:0/1311169601 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fc7a8077010 0x7fc7a8079520 secure :-1 s=READY pgs=791 cs=0 l=1 rev1=1 crypto rx=0x7fc7bc00bb50 tx=0x7fc7bc00b2e0 comp rx=0 tx=0).stop 2024-08-03T20:59:44.265 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:44.264+0000 7fc7d0400640 1 -- 172.21.15.32:0/1311169601 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc7c8073e60 msgr2=0x7fc7c81f4e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:59:44.265 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:44.264+0000 7fc7d0400640 1 --2- 172.21.15.32:0/1311169601 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc7c8073e60 0x7fc7c81f4e50 secure :-1 s=READY pgs=1974 cs=0 l=1 rev1=1 crypto rx=0x7fc7b4000c00 tx=0x7fc7b4003910 comp rx=0 tx=0).stop 2024-08-03T20:59:44.265 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:44.264+0000 7fc7d0400640 1 -- 172.21.15.32:0/1311169601 shutdown_connections 2024-08-03T20:59:44.265 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:44.264+0000 7fc7d0400640 1 --2- 172.21.15.32:0/1311169601 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fc7a8077010 0x7fc7a8079520 unknown :-1 s=CLOSED pgs=791 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:59:44.265 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:44.264+0000 7fc7d0400640 1 --2- 172.21.15.32:0/1311169601 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc7c80747b0 0x7fc7c8143aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:59:44.265 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:44.264+0000 7fc7d0400640 1 --2- 172.21.15.32:0/1311169601 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc7c8073e60 0x7fc7c81f4e50 unknown :-1 s=CLOSED pgs=1974 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:59:44.265 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:44.264+0000 7fc7d0400640 1 --2- 172.21.15.32:0/1311169601 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fc7c806e930 0x7fc7c813cfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:59:44.266 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:44.264+0000 7fc7d0400640 1 -- 172.21.15.32:0/1311169601 >> 172.21.15.32:0/1311169601 conn(0x7fc7c80b1bd0 msgr2=0x7fc7c815cb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:59:44.266 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:44.264+0000 7fc7d0400640 1 -- 172.21.15.32:0/1311169601 shutdown_connections 2024-08-03T20:59:44.266 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:44.264+0000 7fc7d0400640 1 -- 172.21.15.32:0/1311169601 wait complete. 2024-08-03T20:59:44.290 DEBUG:tasks.ceph_test_case:wait_until_true: waiting (timeout=60 retry_count=0)... 2024-08-03T20:59:46.766 INFO:tasks.ceph.mds.a.smithi032.stderr:2024-08-03T20:59:46.765+0000 7f7556c00640 -1 mds.0.cache.dir(0x600) Corrupt fnode in dirfrag 0x600: Decoder at 'void fnode_t::decode(ceph::buffer::v15_2_0::list::const_iterator&)' v=3 cannot decode v=100 minimal_decoder=101: Malformed input [buffer:3] 2024-08-03T20:59:46.766 INFO:tasks.ceph.mds.a.smithi032.stderr:2024-08-03T20:59:46.766+0000 7f7556c00640 -1 mds.0.damage notify_dirfrag Damage to fragment * of ino 0x600 is fatal because it is a system directory for this rank 2024-08-03T20:59:46.767 INFO:tasks.ceph.mon.a.smithi032.stderr:2024-08-03T20:59:46.766+0000 7f4e6d600640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-08-03T20:59:46.767 INFO:tasks.ceph.mon.a.smithi032.stderr:2024-08-03T20:59:46.766+0000 7f4e6d600640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 mds daemon damaged (MDS_DAMAGE) 2024-08-03T20:59:49.292 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-08-03T20:59:49.533 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:49.532+0000 7ff944c00640 1 -- 172.21.15.32:0/1894463795 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ff9400af020 msgr2=0x7ff9400742b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:59:49.533 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:49.532+0000 7ff944c00640 1 --2- 172.21.15.32:0/1894463795 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ff9400af020 0x7ff9400742b0 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7ff930005fd0 tx=0x7ff93002a540 comp rx=0 tx=0).stop 2024-08-03T20:59:49.533 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:49.532+0000 7ff944c00640 1 -- 172.21.15.32:0/1894463795 shutdown_connections 2024-08-03T20:59:49.533 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:49.532+0000 7ff944c00640 1 --2- 172.21.15.32:0/1894463795 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7ff9400acd80 0x7ff9400ad190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:59:49.533 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:49.532+0000 7ff944c00640 1 --2- 172.21.15.32:0/1894463795 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ff9400af020 0x7ff9400742b0 unknown :-1 s=CLOSED pgs=596 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:59:49.533 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:49.532+0000 7ff944c00640 1 -- 172.21.15.32:0/1894463795 >> 172.21.15.32:0/1894463795 conn(0x7ff9400b2120 msgr2=0x7ff9400cfd60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:59:49.533 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:49.532+0000 7ff944c00640 1 -- 172.21.15.32:0/1894463795 shutdown_connections 2024-08-03T20:59:49.533 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:49.532+0000 7ff944c00640 1 -- 172.21.15.32:0/1894463795 wait complete. 2024-08-03T20:59:49.535 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:49.534+0000 7ff944c00640 1 Processor -- start 2024-08-03T20:59:49.535 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:49.534+0000 7ff944c00640 1 -- start start 2024-08-03T20:59:49.536 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:49.536+0000 7ff944c00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff94006f440 0x7ff94015f240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:59:49.537 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:49.536+0000 7ff944c00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ff9400acd80 0x7ff9400cb9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:59:49.537 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:49.536+0000 7ff944c00640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7ff9400af020 0x7ff9401f8110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:59:49.537 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:49.536+0000 7ff944c00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7ff940160260 con 0x7ff94006f440 2024-08-03T20:59:49.537 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:49.536+0000 7ff944c00640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7ff9401600e0 con 0x7ff9400acd80 2024-08-03T20:59:49.537 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:49.536+0000 7ff944c00640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7ff9401603e0 con 0x7ff9400af020 2024-08-03T20:59:49.537 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:49.536+0000 7ff93f400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff94006f440 0x7ff94015f240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:59:49.537 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:49.536+0000 7ff93f400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff94006f440 0x7ff94015f240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:33176/0 (socket says 172.21.15.32:33176) 2024-08-03T20:59:49.537 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:49.536+0000 7ff93f400640 1 -- 172.21.15.32:0/1351467013 learned_addr learned my addr 172.21.15.32:0/1351467013 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:59:49.538 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:49.536+0000 7ff93fe00640 1 --2- 172.21.15.32:0/1351467013 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7ff9400af020 0x7ff9401f8110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:59:49.538 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:49.537+0000 7ff93ea00640 1 --2- 172.21.15.32:0/1351467013 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ff9400acd80 0x7ff9400cb9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:59:49.539 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:49.537+0000 7ff93f400640 1 -- 172.21.15.32:0/1351467013 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7ff9400af020 msgr2=0x7ff9401f8110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:59:49.539 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:49.537+0000 7ff93f400640 1 --2- 172.21.15.32:0/1351467013 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7ff9400af020 0x7ff9401f8110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:59:49.539 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:49.537+0000 7ff93f400640 1 -- 172.21.15.32:0/1351467013 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ff9400acd80 msgr2=0x7ff9400cb9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:59:49.539 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:49.537+0000 7ff93f400640 1 --2- 172.21.15.32:0/1351467013 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ff9400acd80 0x7ff9400cb9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:59:49.539 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:49.537+0000 7ff93f400640 1 -- 172.21.15.32:0/1351467013 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff9401f5b90 con 0x7ff94006f440 2024-08-03T20:59:49.539 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:49.537+0000 7ff93f400640 1 --2- 172.21.15.32:0/1351467013 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff94006f440 0x7ff94015f240 secure :-1 s=READY pgs=1975 cs=0 l=1 rev1=1 crypto rx=0x7ff930000c00 tx=0x7ff9300038a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:59:49.539 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:49.537+0000 7ff937400640 1 -- 172.21.15.32:0/1351467013 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff93003e030 con 0x7ff94006f440 2024-08-03T20:59:49.539 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:49.537+0000 7ff944c00640 1 -- 172.21.15.32:0/1351467013 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff9401f5e70 con 0x7ff94006f440 2024-08-03T20:59:49.539 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:49.537+0000 7ff937400640 1 -- 172.21.15.32:0/1351467013 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7ff930003db0 con 0x7ff94006f440 2024-08-03T20:59:49.539 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:49.537+0000 7ff937400640 1 -- 172.21.15.32:0/1351467013 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff930035040 con 0x7ff94006f440 2024-08-03T20:59:49.539 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:49.537+0000 7ff944c00640 1 -- 172.21.15.32:0/1351467013 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff9401f6430 con 0x7ff94006f440 2024-08-03T20:59:49.540 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:49.538+0000 7ff944c00640 1 -- 172.21.15.32:0/1351467013 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff9400ad190 con 0x7ff94006f440 2024-08-03T20:59:49.540 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:49.540+0000 7ff937400640 1 -- 172.21.15.32:0/1351467013 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7ff930041030 con 0x7ff94006f440 2024-08-03T20:59:49.541 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:49.541+0000 7ff937400640 1 --2- 172.21.15.32:0/1351467013 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7ff920077010 0x7ff920079520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:59:49.541 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:49.541+0000 7ff937400640 1 -- 172.21.15.32:0/1351467013 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(121..121 src has 1..121) ==== 8950+0+0 (secure 0 0 0) 0x7ff9300b9da0 con 0x7ff94006f440 2024-08-03T20:59:49.542 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:49.541+0000 7ff93ea00640 1 --2- 172.21.15.32:0/1351467013 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7ff920077010 0x7ff920079520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:59:49.542 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:49.542+0000 7ff93ea00640 1 --2- 172.21.15.32:0/1351467013 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7ff920077010 0x7ff920079520 secure :-1 s=READY pgs=792 cs=0 l=1 rev1=1 crypto rx=0x7ff9400c9e10 tx=0x7ff92c005f90 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:59:49.545 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:49.545+0000 7ff937400640 1 -- 172.21.15.32:0/1351467013 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7ff9300821c0 con 0x7ff94006f440 2024-08-03T20:59:49.792 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:49.792+0000 7ff944c00640 1 -- 172.21.15.32:0/1351467013 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff9400b1710 con 0x7ff94006f440 2024-08-03T20:59:49.793 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:49.792+0000 7ff937400640 1 -- 172.21.15.32:0/1351467013 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 301 v301) ==== 95+0+4391 (secure 0 0 0) 0x7ff930081ac0 con 0x7ff94006f440 2024-08-03T20:59:49.793 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-03T20:59:49.793 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":301,"btime":"2024-08-03T20:59:46:767047+0000","default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6439,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6835/531080160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6834","nonce":531080160},{"type":"v1","addr":"172.21.15.118:6835","nonce":531080160}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":84},{"gid":6454,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6837/2796882319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6836","nonce":2796882319},{"type":"v1","addr":"172.21.15.118:6837","nonce":2796882319}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":86},{"gid":6813,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6834/2686933890","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":2686933890},{"type":"v1","addr":"172.21.15.32:6834","nonce":2686933890}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":85}],"filesystems":[{"mdsmap":{"epoch":301,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-08-03T20:54:26.950518+0000","modified":"2024-08-03T20:59:46.767042+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":121,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":6}]} 2024-08-03T20:59:49.793 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 301 2024-08-03T20:59:49.795 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:49.794+0000 7ff944c00640 1 -- 172.21.15.32:0/1351467013 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7ff920077010 msgr2=0x7ff920079520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:59:49.795 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:49.794+0000 7ff944c00640 1 --2- 172.21.15.32:0/1351467013 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7ff920077010 0x7ff920079520 secure :-1 s=READY pgs=792 cs=0 l=1 rev1=1 crypto rx=0x7ff9400c9e10 tx=0x7ff92c005f90 comp rx=0 tx=0).stop 2024-08-03T20:59:49.795 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:49.794+0000 7ff944c00640 1 -- 172.21.15.32:0/1351467013 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff94006f440 msgr2=0x7ff94015f240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:59:49.795 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:49.794+0000 7ff944c00640 1 --2- 172.21.15.32:0/1351467013 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff94006f440 0x7ff94015f240 secure :-1 s=READY pgs=1975 cs=0 l=1 rev1=1 crypto rx=0x7ff930000c00 tx=0x7ff9300038a0 comp rx=0 tx=0).stop 2024-08-03T20:59:49.796 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:49.794+0000 7ff944c00640 1 -- 172.21.15.32:0/1351467013 shutdown_connections 2024-08-03T20:59:49.796 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:49.794+0000 7ff944c00640 1 --2- 172.21.15.32:0/1351467013 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7ff920077010 0x7ff920079520 unknown :-1 s=CLOSED pgs=792 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:59:49.796 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:49.795+0000 7ff944c00640 1 --2- 172.21.15.32:0/1351467013 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ff9400acd80 0x7ff9400cb9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:59:49.796 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:49.795+0000 7ff944c00640 1 --2- 172.21.15.32:0/1351467013 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7ff9400af020 0x7ff9401f8110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:59:49.796 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:49.795+0000 7ff944c00640 1 --2- 172.21.15.32:0/1351467013 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff94006f440 0x7ff94015f240 unknown :-1 s=CLOSED pgs=1975 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:59:49.796 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:49.795+0000 7ff944c00640 1 -- 172.21.15.32:0/1351467013 >> 172.21.15.32:0/1351467013 conn(0x7ff9400b2120 msgr2=0x7ff94015b320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:59:49.796 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:49.795+0000 7ff944c00640 1 -- 172.21.15.32:0/1351467013 shutdown_connections 2024-08-03T20:59:49.796 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:49.795+0000 7ff944c00640 1 -- 172.21.15.32:0/1351467013 wait complete. 2024-08-03T20:59:49.820 DEBUG:tasks.ceph_test_case:wait_until_true: success in 5s and 0 retries 2024-08-03T20:59:49.820 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-08-03T20:59:50.048 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:50.047+0000 7f94c7a00640 1 -- 172.21.15.32:0/353543686 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f94c006e930 msgr2=0x7f94c0144f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:59:50.048 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:50.047+0000 7f94c7a00640 1 --2- 172.21.15.32:0/353543686 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f94c006e930 0x7f94c0144f90 secure :-1 s=READY pgs=1976 cs=0 l=1 rev1=1 crypto rx=0x7f94b0005fd0 tx=0x7f94b002a530 comp rx=0 tx=0).stop 2024-08-03T20:59:50.048 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:50.047+0000 7f94c7a00640 1 -- 172.21.15.32:0/353543686 shutdown_connections 2024-08-03T20:59:50.048 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:50.047+0000 7f94c7a00640 1 --2- 172.21.15.32:0/353543686 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f94c015fff0 0x7f94c0160400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:59:50.048 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:50.047+0000 7f94c7a00640 1 --2- 172.21.15.32:0/353543686 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f94c006e930 0x7f94c0144f90 unknown :-1 s=CLOSED pgs=1976 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:59:50.048 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:50.047+0000 7f94c7a00640 1 -- 172.21.15.32:0/353543686 >> 172.21.15.32:0/353543686 conn(0x7f94c00b1bd0 msgr2=0x7f94c00b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:59:50.048 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:50.047+0000 7f94c7a00640 1 -- 172.21.15.32:0/353543686 shutdown_connections 2024-08-03T20:59:50.048 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:50.047+0000 7f94c7a00640 1 -- 172.21.15.32:0/353543686 wait complete. 2024-08-03T20:59:50.050 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:50.049+0000 7f94c7a00640 1 Processor -- start 2024-08-03T20:59:50.050 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:50.049+0000 7f94c7a00640 1 -- start start 2024-08-03T20:59:50.051 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:50.051+0000 7f94c7a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f94c006e930 0x7f94c015b040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:59:50.052 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:50.051+0000 7f94c7a00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f94c0073e60 0x7f94c01e7220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:59:50.052 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:50.051+0000 7f94c7a00640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f94c015fff0 0x7f94c01e7760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:59:50.052 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:50.051+0000 7f94c7a00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f94c00cf840 con 0x7f94c006e930 2024-08-03T20:59:50.052 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:50.051+0000 7f94c7a00640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f94c0145f30 con 0x7f94c0073e60 2024-08-03T20:59:50.052 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:50.051+0000 7f94c7a00640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7f94c00cf9c0 con 0x7f94c015fff0 2024-08-03T20:59:50.052 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:50.051+0000 7f94c5000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f94c006e930 0x7f94c015b040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:59:50.052 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:50.051+0000 7f94c5000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f94c006e930 0x7f94c015b040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:54084/0 (socket says 172.21.15.32:54084) 2024-08-03T20:59:50.052 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:50.051+0000 7f94c5000640 1 -- 172.21.15.32:0/2502601744 learned_addr learned my addr 172.21.15.32:0/2502601744 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:59:50.053 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:50.051+0000 7f94b7e00640 1 --2- 172.21.15.32:0/2502601744 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f94c0073e60 0x7f94c01e7220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:59:50.054 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:50.051+0000 7f94c5a00640 1 --2- 172.21.15.32:0/2502601744 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f94c015fff0 0x7f94c01e7760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:59:50.054 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:50.052+0000 7f94c5000640 1 -- 172.21.15.32:0/2502601744 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f94c015fff0 msgr2=0x7f94c01e7760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:59:50.054 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:50.052+0000 7f94c5000640 1 --2- 172.21.15.32:0/2502601744 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f94c015fff0 0x7f94c01e7760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:59:50.054 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:50.052+0000 7f94c5000640 1 -- 172.21.15.32:0/2502601744 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f94c0073e60 msgr2=0x7f94c01e7220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:59:50.054 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:50.052+0000 7f94c5000640 1 --2- 172.21.15.32:0/2502601744 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f94c0073e60 0x7f94c01e7220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:59:50.054 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:50.052+0000 7f94c5000640 1 -- 172.21.15.32:0/2502601744 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f94c01e9990 con 0x7f94c006e930 2024-08-03T20:59:50.054 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:50.052+0000 7f94c5000640 1 --2- 172.21.15.32:0/2502601744 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f94c006e930 0x7f94c015b040 secure :-1 s=READY pgs=1977 cs=0 l=1 rev1=1 crypto rx=0x7f94b0006010 tx=0x7f94b00039e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:59:50.054 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:50.052+0000 7f94bd600640 1 -- 172.21.15.32:0/2502601744 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94b003e030 con 0x7f94c006e930 2024-08-03T20:59:50.054 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:50.052+0000 7f94bd600640 1 -- 172.21.15.32:0/2502601744 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f94b00047a0 con 0x7f94c006e930 2024-08-03T20:59:50.054 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:50.052+0000 7f94bd600640 1 -- 172.21.15.32:0/2502601744 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94b0035040 con 0x7f94c006e930 2024-08-03T20:59:50.054 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:50.052+0000 7f94c7a00640 1 -- 172.21.15.32:0/2502601744 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f94c01e96b0 con 0x7f94c006e930 2024-08-03T20:59:50.055 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:50.053+0000 7f94c7a00640 1 -- 172.21.15.32:0/2502601744 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f94c0142270 con 0x7f94c006e930 2024-08-03T20:59:50.055 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:50.054+0000 7f94c7a00640 1 -- 172.21.15.32:0/2502601744 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f94c0160d10 con 0x7f94c006e930 2024-08-03T20:59:50.055 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:50.055+0000 7f94bd600640 1 -- 172.21.15.32:0/2502601744 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f94b0041030 con 0x7f94c006e930 2024-08-03T20:59:50.056 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:50.056+0000 7f94bd600640 1 --2- 172.21.15.32:0/2502601744 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f949c077010 0x7f949c079520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:59:50.056 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:50.056+0000 7f94bd600640 1 -- 172.21.15.32:0/2502601744 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(121..121 src has 1..121) ==== 8950+0+0 (secure 0 0 0) 0x7f94b00b9e70 con 0x7f94c006e930 2024-08-03T20:59:50.057 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:50.056+0000 7f94b7e00640 1 --2- 172.21.15.32:0/2502601744 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f949c077010 0x7f949c079520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:59:50.057 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:50.057+0000 7f94b7e00640 1 --2- 172.21.15.32:0/2502601744 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f949c077010 0x7f949c079520 secure :-1 s=READY pgs=793 cs=0 l=1 rev1=1 crypto rx=0x7f94a8006fd0 tx=0x7f94a8008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:59:50.060 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:50.060+0000 7f94bd600640 1 -- 172.21.15.32:0/2502601744 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7f94b00823a0 con 0x7f94c006e930 2024-08-03T20:59:50.303 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:50.303+0000 7f94c7a00640 1 -- 172.21.15.32:0/2502601744 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f94c015b580 con 0x7f94c006e930 2024-08-03T20:59:50.304 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:50.303+0000 7f94bd600640 1 -- 172.21.15.32:0/2502601744 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 301 v301) ==== 95+0+4391 (secure 0 0 0) 0x7f94b0081ca0 con 0x7f94c006e930 2024-08-03T20:59:50.304 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-03T20:59:50.304 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":301,"btime":"2024-08-03T20:59:46:767047+0000","default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6439,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6835/531080160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6834","nonce":531080160},{"type":"v1","addr":"172.21.15.118:6835","nonce":531080160}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":84},{"gid":6454,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6837/2796882319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6836","nonce":2796882319},{"type":"v1","addr":"172.21.15.118:6837","nonce":2796882319}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":86},{"gid":6813,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6834/2686933890","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":2686933890},{"type":"v1","addr":"172.21.15.32:6834","nonce":2686933890}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":85}],"filesystems":[{"mdsmap":{"epoch":301,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-08-03T20:54:26.950518+0000","modified":"2024-08-03T20:59:46.767042+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":121,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":6}]} 2024-08-03T20:59:50.305 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 301 2024-08-03T20:59:50.307 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:50.306+0000 7f94c7a00640 1 -- 172.21.15.32:0/2502601744 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f949c077010 msgr2=0x7f949c079520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:59:50.307 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:50.306+0000 7f94c7a00640 1 --2- 172.21.15.32:0/2502601744 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f949c077010 0x7f949c079520 secure :-1 s=READY pgs=793 cs=0 l=1 rev1=1 crypto rx=0x7f94a8006fd0 tx=0x7f94a8008040 comp rx=0 tx=0).stop 2024-08-03T20:59:50.307 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:50.306+0000 7f94c7a00640 1 -- 172.21.15.32:0/2502601744 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f94c006e930 msgr2=0x7f94c015b040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:59:50.307 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:50.306+0000 7f94c7a00640 1 --2- 172.21.15.32:0/2502601744 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f94c006e930 0x7f94c015b040 secure :-1 s=READY pgs=1977 cs=0 l=1 rev1=1 crypto rx=0x7f94b0006010 tx=0x7f94b00039e0 comp rx=0 tx=0).stop 2024-08-03T20:59:50.307 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:50.306+0000 7f94c7a00640 1 -- 172.21.15.32:0/2502601744 shutdown_connections 2024-08-03T20:59:50.307 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:50.306+0000 7f94c7a00640 1 --2- 172.21.15.32:0/2502601744 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f949c077010 0x7f949c079520 unknown :-1 s=CLOSED pgs=793 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:59:50.307 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:50.306+0000 7f94c7a00640 1 --2- 172.21.15.32:0/2502601744 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f94c0073e60 0x7f94c01e7220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:59:50.307 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:50.306+0000 7f94c7a00640 1 --2- 172.21.15.32:0/2502601744 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f94c015fff0 0x7f94c01e7760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:59:50.307 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:50.306+0000 7f94c7a00640 1 --2- 172.21.15.32:0/2502601744 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f94c006e930 0x7f94c015b040 unknown :-1 s=CLOSED pgs=1977 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:59:50.307 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:50.306+0000 7f94c7a00640 1 -- 172.21.15.32:0/2502601744 >> 172.21.15.32:0/2502601744 conn(0x7f94c00b1bd0 msgr2=0x7f94c00afb20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:59:50.308 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:50.306+0000 7f94c7a00640 1 -- 172.21.15.32:0/2502601744 shutdown_connections 2024-08-03T20:59:50.308 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:50.306+0000 7f94c7a00640 1 -- 172.21.15.32:0/2502601744 wait complete. 2024-08-03T20:59:50.332 DEBUG:tasks.ceph_test_case:wait_until_true: success in 0s and 0 retries 2024-08-03T20:59:50.332 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-08-03T20:59:50.561 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:50.560+0000 7fdd1da00640 1 -- 172.21.15.32:0/3453463113 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdd18073990 msgr2=0x7fdd180cd860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:59:50.561 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:50.560+0000 7fdd1da00640 1 --2- 172.21.15.32:0/3453463113 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdd18073990 0x7fdd180cd860 secure :-1 s=READY pgs=1978 cs=0 l=1 rev1=1 crypto rx=0x7fdd08005fd0 tx=0x7fdd0802a540 comp rx=0 tx=0).stop 2024-08-03T20:59:50.561 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:50.560+0000 7fdd1da00640 1 -- 172.21.15.32:0/3453463113 shutdown_connections 2024-08-03T20:59:50.562 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:50.560+0000 7fdd1da00640 1 --2- 172.21.15.32:0/3453463113 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fdd18153270 0x7fdd1815d230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:59:50.562 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:50.560+0000 7fdd1da00640 1 --2- 172.21.15.32:0/3453463113 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdd18073990 0x7fdd180cd860 unknown :-1 s=CLOSED pgs=1978 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:59:50.562 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:50.560+0000 7fdd1da00640 1 -- 172.21.15.32:0/3453463113 >> 172.21.15.32:0/3453463113 conn(0x7fdd18072e70 msgr2=0x7fdd18073280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:59:50.562 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:50.560+0000 7fdd1da00640 1 -- 172.21.15.32:0/3453463113 shutdown_connections 2024-08-03T20:59:50.562 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:50.560+0000 7fdd1da00640 1 -- 172.21.15.32:0/3453463113 wait complete. 2024-08-03T20:59:50.563 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:50.563+0000 7fdd1da00640 1 Processor -- start 2024-08-03T20:59:50.563 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:50.563+0000 7fdd1da00640 1 -- start start 2024-08-03T20:59:50.566 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:50.564+0000 7fdd1da00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdd1806d9f0 0x7fdd181471b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:59:50.566 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:50.564+0000 7fdd1da00640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fdd18073990 0x7fdd1815b240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:59:50.566 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:50.564+0000 7fdd1da00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fdd18153270 0x7fdd1815b780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:59:50.566 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:50.564+0000 7fdd1da00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fdd180cf730 con 0x7fdd1806d9f0 2024-08-03T20:59:50.566 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:50.564+0000 7fdd1da00640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7fdd180cf5b0 con 0x7fdd18153270 2024-08-03T20:59:50.566 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:50.564+0000 7fdd1da00640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7fdd180cf8b0 con 0x7fdd18073990 2024-08-03T20:59:50.566 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:50.565+0000 7fdd17e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdd1806d9f0 0x7fdd181471b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:59:50.566 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:50.565+0000 7fdd17e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdd1806d9f0 0x7fdd181471b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:54098/0 (socket says 172.21.15.32:54098) 2024-08-03T20:59:50.566 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:50.565+0000 7fdd17e00640 1 -- 172.21.15.32:0/3502490473 learned_addr learned my addr 172.21.15.32:0/3502490473 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:59:50.566 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:50.565+0000 7fdd17400640 1 --2- 172.21.15.32:0/3502490473 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fdd18073990 0x7fdd1815b240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:59:50.567 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:50.565+0000 7fdd1d000640 1 --2- 172.21.15.32:0/3502490473 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fdd18153270 0x7fdd1815b780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:59:50.567 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:50.565+0000 7fdd17e00640 1 -- 172.21.15.32:0/3502490473 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fdd18073990 msgr2=0x7fdd1815b240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:59:50.567 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:50.565+0000 7fdd17e00640 1 --2- 172.21.15.32:0/3502490473 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fdd18073990 0x7fdd1815b240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:59:50.567 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:50.565+0000 7fdd17e00640 1 -- 172.21.15.32:0/3502490473 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fdd18153270 msgr2=0x7fdd1815b780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:59:50.567 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:50.565+0000 7fdd17e00640 1 --2- 172.21.15.32:0/3502490473 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fdd18153270 0x7fdd1815b780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:59:50.567 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:50.565+0000 7fdd17e00640 1 -- 172.21.15.32:0/3502490473 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdd1806f980 con 0x7fdd1806d9f0 2024-08-03T20:59:50.567 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:50.565+0000 7fdd17e00640 1 --2- 172.21.15.32:0/3502490473 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdd1806d9f0 0x7fdd181471b0 secure :-1 s=READY pgs=1979 cs=0 l=1 rev1=1 crypto rx=0x7fdd08000c00 tx=0x7fdd08003910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:59:50.568 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:50.566+0000 7fdd0fe00640 1 -- 172.21.15.32:0/3502490473 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd0803e030 con 0x7fdd1806d9f0 2024-08-03T20:59:50.568 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:50.566+0000 7fdd0fe00640 1 -- 172.21.15.32:0/3502490473 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fdd08003e00 con 0x7fdd1806d9f0 2024-08-03T20:59:50.568 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:50.566+0000 7fdd1da00640 1 -- 172.21.15.32:0/3502490473 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdd1815e4d0 con 0x7fdd1806d9f0 2024-08-03T20:59:50.568 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:50.566+0000 7fdd0fe00640 1 -- 172.21.15.32:0/3502490473 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd08035040 con 0x7fdd1806d9f0 2024-08-03T20:59:50.568 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:50.566+0000 7fdd1da00640 1 -- 172.21.15.32:0/3502490473 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdd1815e7b0 con 0x7fdd1806d9f0 2024-08-03T20:59:50.568 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:50.567+0000 7fdd1da00640 1 -- 172.21.15.32:0/3502490473 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdd180cfd60 con 0x7fdd1806d9f0 2024-08-03T20:59:50.569 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:50.568+0000 7fdd0fe00640 1 -- 172.21.15.32:0/3502490473 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7fdd08041030 con 0x7fdd1806d9f0 2024-08-03T20:59:50.570 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:50.569+0000 7fdd0fe00640 1 --2- 172.21.15.32:0/3502490473 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fdcf8077010 0x7fdcf8079520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:59:50.570 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:50.569+0000 7fdd0fe00640 1 -- 172.21.15.32:0/3502490473 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(121..121 src has 1..121) ==== 8950+0+0 (secure 0 0 0) 0x7fdd080b9d20 con 0x7fdd1806d9f0 2024-08-03T20:59:50.570 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:50.570+0000 7fdd17400640 1 --2- 172.21.15.32:0/3502490473 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fdcf8077010 0x7fdcf8079520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:59:50.571 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:50.570+0000 7fdd17400640 1 --2- 172.21.15.32:0/3502490473 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fdcf8077010 0x7fdcf8079520 secure :-1 s=READY pgs=794 cs=0 l=1 rev1=1 crypto rx=0x7fdd04005fd0 tx=0x7fdd04005850 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:59:50.574 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:50.574+0000 7fdd0fe00640 1 -- 172.21.15.32:0/3502490473 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7fdd08082250 con 0x7fdd1806d9f0 2024-08-03T20:59:50.818 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:50.817+0000 7fdd1da00640 1 -- 172.21.15.32:0/3502490473 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fdd18073320 con 0x7fdd1806d9f0 2024-08-03T20:59:50.818 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:50.818+0000 7fdd0fe00640 1 -- 172.21.15.32:0/3502490473 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 301 v301) ==== 95+0+4391 (secure 0 0 0) 0x7fdd08081b50 con 0x7fdd1806d9f0 2024-08-03T20:59:50.819 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-03T20:59:50.819 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":301,"btime":"2024-08-03T20:59:46:767047+0000","default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6439,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6835/531080160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6834","nonce":531080160},{"type":"v1","addr":"172.21.15.118:6835","nonce":531080160}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":84},{"gid":6454,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6837/2796882319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6836","nonce":2796882319},{"type":"v1","addr":"172.21.15.118:6837","nonce":2796882319}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":86},{"gid":6813,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6834/2686933890","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":2686933890},{"type":"v1","addr":"172.21.15.32:6834","nonce":2686933890}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":85}],"filesystems":[{"mdsmap":{"epoch":301,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-08-03T20:54:26.950518+0000","modified":"2024-08-03T20:59:46.767042+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":121,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":6}]} 2024-08-03T20:59:50.819 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 301 2024-08-03T20:59:50.821 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:50.820+0000 7fdd1da00640 1 -- 172.21.15.32:0/3502490473 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fdcf8077010 msgr2=0x7fdcf8079520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:59:50.821 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:50.820+0000 7fdd1da00640 1 --2- 172.21.15.32:0/3502490473 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fdcf8077010 0x7fdcf8079520 secure :-1 s=READY pgs=794 cs=0 l=1 rev1=1 crypto rx=0x7fdd04005fd0 tx=0x7fdd04005850 comp rx=0 tx=0).stop 2024-08-03T20:59:50.821 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:50.820+0000 7fdd1da00640 1 -- 172.21.15.32:0/3502490473 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdd1806d9f0 msgr2=0x7fdd181471b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:59:50.822 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:50.820+0000 7fdd1da00640 1 --2- 172.21.15.32:0/3502490473 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdd1806d9f0 0x7fdd181471b0 secure :-1 s=READY pgs=1979 cs=0 l=1 rev1=1 crypto rx=0x7fdd08000c00 tx=0x7fdd08003910 comp rx=0 tx=0).stop 2024-08-03T20:59:50.822 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:50.821+0000 7fdd1da00640 1 -- 172.21.15.32:0/3502490473 shutdown_connections 2024-08-03T20:59:50.822 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:50.821+0000 7fdd1da00640 1 --2- 172.21.15.32:0/3502490473 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fdcf8077010 0x7fdcf8079520 unknown :-1 s=CLOSED pgs=794 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:59:50.822 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:50.821+0000 7fdd1da00640 1 --2- 172.21.15.32:0/3502490473 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fdd18153270 0x7fdd1815b780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:59:50.822 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:50.821+0000 7fdd1da00640 1 --2- 172.21.15.32:0/3502490473 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fdd18073990 0x7fdd1815b240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:59:50.822 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:50.821+0000 7fdd1da00640 1 --2- 172.21.15.32:0/3502490473 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdd1806d9f0 0x7fdd181471b0 unknown :-1 s=CLOSED pgs=1979 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:59:50.822 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:50.821+0000 7fdd1da00640 1 -- 172.21.15.32:0/3502490473 >> 172.21.15.32:0/3502490473 conn(0x7fdd18072e70 msgr2=0x7fdd180b4d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:59:50.822 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:50.821+0000 7fdd1da00640 1 -- 172.21.15.32:0/3502490473 shutdown_connections 2024-08-03T20:59:50.822 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:50.821+0000 7fdd1da00640 1 -- 172.21.15.32:0/3502490473 wait complete. 2024-08-03T20:59:50.846 INFO:tasks.cephfs.test_damage:Result: Mutation 'Corrupt omap header on 600.00000000' led to DAMAGED state 2024-08-03T20:59:50.846 INFO:tasks.cephfs.test_damage:Applying mutation 'Corrupt omap header on mds0_openfiles.0' 2024-08-03T20:59:50.846 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-08-03T20:59:50.846 DEBUG:teuthology.orchestra.run.smithi032:> dd if=/proc/self/mounts of=/dev/stdout 2024-08-03T20:59:50.881 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053453,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6499208k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 none /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249604k,nr_inodes=812401,mode=700,uid=1000,gid=1316 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6499208k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-08-03T20:59:50.882 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi032.front.sepia.ceph.com 2024-08-03T20:59:50.882 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-08-03T20:59:50.882 DEBUG:teuthology.orchestra.run.smithi032:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-08-03T20:59:50.956 INFO:teuthology.orchestra.run.smithi032.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.0': No such file or directory 2024-08-03T20:59:50.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-03T20:59:50.957 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-08-03T20:59:51.226 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:51.225+0000 7fc6cfc00640 1 -- 172.21.15.32:0/2132547305 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc6c8144f90 msgr2=0x7fc6c8074570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:59:51.227 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:51.225+0000 7fc6cfc00640 1 --2- 172.21.15.32:0/2132547305 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc6c8144f90 0x7fc6c8074570 secure :-1 s=READY pgs=1981 cs=0 l=1 rev1=1 crypto rx=0x7fc6c8073b50 tx=0x7fc6b002a5c0 comp rx=0 tx=0).stop 2024-08-03T20:59:51.227 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:51.225+0000 7fc6cfc00640 1 -- 172.21.15.32:0/2132547305 shutdown_connections 2024-08-03T20:59:51.227 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:51.225+0000 7fc6cfc00640 1 --2- 172.21.15.32:0/2132547305 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc6c8144f90 0x7fc6c8074570 unknown :-1 s=CLOSED pgs=1981 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:59:51.227 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:51.225+0000 7fc6cfc00640 1 -- 172.21.15.32:0/2132547305 >> 172.21.15.32:0/2132547305 conn(0x7fc6c80b1bd0 msgr2=0x7fc6c80b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:59:51.227 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:51.225+0000 7fc6cfc00640 1 -- 172.21.15.32:0/2132547305 shutdown_connections 2024-08-03T20:59:51.227 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:51.225+0000 7fc6cfc00640 1 -- 172.21.15.32:0/2132547305 wait complete. 2024-08-03T20:59:51.228 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:51.228+0000 7fc6cfc00640 1 Processor -- start 2024-08-03T20:59:51.228 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:51.228+0000 7fc6cfc00640 1 -- start start 2024-08-03T20:59:51.230 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:51.229+0000 7fc6cfc00640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fc6c806e930 0x7fc6c81f75a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:59:51.231 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:51.229+0000 7fc6cfc00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc6c8144f90 0x7fc6c81f6310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:59:51.231 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:51.229+0000 7fc6cfc00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc6c815fff0 0x7fc6c8143cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:59:51.231 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:51.229+0000 7fc6cfc00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fc6c80cf840 con 0x7fc6c8144f90 2024-08-03T20:59:51.231 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:51.229+0000 7fc6cfc00640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7fc6c8145f30 con 0x7fc6c815fff0 2024-08-03T20:59:51.231 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:51.229+0000 7fc6cfc00640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7fc6c80cf9c0 con 0x7fc6c806e930 2024-08-03T20:59:51.231 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:51.230+0000 7fc6bfe00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc6c8144f90 0x7fc6c81f6310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:59:51.231 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:51.230+0000 7fc6bfe00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc6c8144f90 0x7fc6c81f6310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:54112/0 (socket says 172.21.15.32:54112) 2024-08-03T20:59:51.231 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:51.230+0000 7fc6bfe00640 1 -- 172.21.15.32:0/1958935190 learned_addr learned my addr 172.21.15.32:0/1958935190 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:59:51.231 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:51.230+0000 7fc6cdc00640 1 --2- 172.21.15.32:0/1958935190 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc6c815fff0 0x7fc6c8143cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:59:51.231 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:51.230+0000 7fc6cd200640 1 --2- 172.21.15.32:0/1958935190 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fc6c806e930 0x7fc6c81f75a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:59:51.231 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:51.230+0000 7fc6bfe00640 1 -- 172.21.15.32:0/1958935190 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fc6c806e930 msgr2=0x7fc6c81f75a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:59:51.231 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:51.230+0000 7fc6bfe00640 1 --2- 172.21.15.32:0/1958935190 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fc6c806e930 0x7fc6c81f75a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:59:51.232 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:51.230+0000 7fc6bfe00640 1 -- 172.21.15.32:0/1958935190 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc6c815fff0 msgr2=0x7fc6c8143cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:59:51.232 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:51.230+0000 7fc6bfe00640 1 --2- 172.21.15.32:0/1958935190 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc6c815fff0 0x7fc6c8143cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:59:51.232 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:51.230+0000 7fc6bfe00640 1 -- 172.21.15.32:0/1958935190 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc6c81f7ae0 con 0x7fc6c8144f90 2024-08-03T20:59:51.233 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:51.231+0000 7fc6bfe00640 1 --2- 172.21.15.32:0/1958935190 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc6c8144f90 0x7fc6c81f6310 secure :-1 s=READY pgs=1982 cs=0 l=1 rev1=1 crypto rx=0x7fc6b002abb0 tx=0x7fc6b0033c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:59:51.233 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:51.231+0000 7fc6c5600640 1 -- 172.21.15.32:0/1958935190 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6b0031d60 con 0x7fc6c8144f90 2024-08-03T20:59:51.233 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:51.231+0000 7fc6c5600640 1 -- 172.21.15.32:0/1958935190 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fc6b0033e10 con 0x7fc6c8144f90 2024-08-03T20:59:51.233 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:51.231+0000 7fc6c5600640 1 -- 172.21.15.32:0/1958935190 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6b0005280 con 0x7fc6c8144f90 2024-08-03T20:59:51.233 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:51.231+0000 7fc6cfc00640 1 -- 172.21.15.32:0/1958935190 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc6c80a9900 con 0x7fc6c8144f90 2024-08-03T20:59:51.233 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:51.231+0000 7fc6cfc00640 1 -- 172.21.15.32:0/1958935190 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc6c80a9ec0 con 0x7fc6c8144f90 2024-08-03T20:59:51.233 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:51.232+0000 7fc6cfc00640 1 -- 172.21.15.32:0/1958935190 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc6c80a9670 con 0x7fc6c8144f90 2024-08-03T20:59:51.234 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:51.233+0000 7fc6c5600640 1 -- 172.21.15.32:0/1958935190 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7fc6b0005420 con 0x7fc6c8144f90 2024-08-03T20:59:51.234 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:51.234+0000 7fc6c5600640 1 --2- 172.21.15.32:0/1958935190 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fc6a4077010 0x7fc6a4079520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:59:51.235 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:51.234+0000 7fc6c5600640 1 -- 172.21.15.32:0/1958935190 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(121..121 src has 1..121) ==== 8950+0+0 (secure 0 0 0) 0x7fc6b00ba650 con 0x7fc6c8144f90 2024-08-03T20:59:51.235 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:51.235+0000 7fc6cd200640 1 --2- 172.21.15.32:0/1958935190 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fc6a4077010 0x7fc6a4079520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:59:51.236 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:51.235+0000 7fc6cd200640 1 --2- 172.21.15.32:0/1958935190 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fc6a4077010 0x7fc6a4079520 secure :-1 s=READY pgs=795 cs=0 l=1 rev1=1 crypto rx=0x7fc6b80046b0 tx=0x7fc6b8007cc0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:59:51.239 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:51.239+0000 7fc6c5600640 1 -- 172.21.15.32:0/1958935190 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7fc6c81f7ae0 con 0x7fc6c8144f90 2024-08-03T20:59:51.484 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:51.484+0000 7fc6cfc00640 1 -- 172.21.15.32:0/1958935190 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7fc6c8073070 con 0x7fc6c8144f90 2024-08-03T20:59:51.520 INFO:tasks.ceph.mds.a.smithi032.stderr: -45> 2024-08-03T20:59:46.765+0000 7f7556c00640 -1 mds.0.cache.dir(0x600) Corrupt fnode in dirfrag 0x600: Decoder at 'void fnode_t::decode(ceph::buffer::v15_2_0::list::const_iterator&)' v=3 cannot decode v=100 minimal_decoder=101: Malformed input [buffer:3] 2024-08-03T20:59:51.520 INFO:tasks.ceph.mds.a.smithi032.stderr: -42> 2024-08-03T20:59:46.766+0000 7f7556c00640 -1 mds.0.damage notify_dirfrag Damage to fragment * of ino 0x600 is fatal because it is a system directory for this rank 2024-08-03T20:59:51.630 INFO:tasks.ceph.mds.a.smithi032.stdout:starting mds.a at 2024-08-03T20:59:51.770 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:51.769+0000 7fc6c5600640 1 -- 172.21.15.32:0/1958935190 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v302) ==== 161+0+0 (secure 0 0 0) 0x7fc6c8073070 con 0x7fc6c8144f90 2024-08-03T20:59:51.770 INFO:teuthology.orchestra.run.smithi032.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-08-03T20:59:51.772 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:51.772+0000 7fc6cfc00640 1 -- 172.21.15.32:0/1958935190 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fc6a4077010 msgr2=0x7fc6a4079520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:59:51.773 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:51.772+0000 7fc6cfc00640 1 --2- 172.21.15.32:0/1958935190 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fc6a4077010 0x7fc6a4079520 secure :-1 s=READY pgs=795 cs=0 l=1 rev1=1 crypto rx=0x7fc6b80046b0 tx=0x7fc6b8007cc0 comp rx=0 tx=0).stop 2024-08-03T20:59:51.773 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:51.772+0000 7fc6cfc00640 1 -- 172.21.15.32:0/1958935190 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc6c8144f90 msgr2=0x7fc6c81f6310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:59:51.773 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:51.772+0000 7fc6cfc00640 1 --2- 172.21.15.32:0/1958935190 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc6c8144f90 0x7fc6c81f6310 secure :-1 s=READY pgs=1982 cs=0 l=1 rev1=1 crypto rx=0x7fc6b002abb0 tx=0x7fc6b0033c20 comp rx=0 tx=0).stop 2024-08-03T20:59:51.773 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:51.772+0000 7fc6cfc00640 1 -- 172.21.15.32:0/1958935190 shutdown_connections 2024-08-03T20:59:51.773 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:51.772+0000 7fc6cfc00640 1 --2- 172.21.15.32:0/1958935190 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fc6a4077010 0x7fc6a4079520 unknown :-1 s=CLOSED pgs=795 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:59:51.773 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:51.772+0000 7fc6cfc00640 1 --2- 172.21.15.32:0/1958935190 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc6c815fff0 0x7fc6c8143cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:59:51.773 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:51.772+0000 7fc6cfc00640 1 --2- 172.21.15.32:0/1958935190 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc6c8144f90 0x7fc6c81f6310 unknown :-1 s=CLOSED pgs=1982 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:59:51.773 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:51.772+0000 7fc6cfc00640 1 --2- 172.21.15.32:0/1958935190 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fc6c806e930 0x7fc6c81f75a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:59:51.773 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:51.772+0000 7fc6cfc00640 1 -- 172.21.15.32:0/1958935190 >> 172.21.15.32:0/1958935190 conn(0x7fc6c80b1bd0 msgr2=0x7fc6c80cff40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:59:51.773 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:51.772+0000 7fc6cfc00640 1 -- 172.21.15.32:0/1958935190 shutdown_connections 2024-08-03T20:59:51.773 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:51.772+0000 7fc6cfc00640 1 -- 172.21.15.32:0/1958935190 wait complete. 2024-08-03T20:59:51.797 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mds repaired 0 2024-08-03T20:59:52.026 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:52.025+0000 7f0903200640 1 -- 172.21.15.32:0/3120783945 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f08fc144f90 msgr2=0x7f08fc074570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:59:52.026 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:52.025+0000 7f0903200640 1 --2- 172.21.15.32:0/3120783945 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f08fc144f90 0x7f08fc074570 secure :-1 s=READY pgs=1986 cs=0 l=1 rev1=1 crypto rx=0x7f08fc072d80 tx=0x7f08e402a540 comp rx=0 tx=0).stop 2024-08-03T20:59:52.026 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:52.025+0000 7f0903200640 1 -- 172.21.15.32:0/3120783945 shutdown_connections 2024-08-03T20:59:52.026 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:52.025+0000 7f0903200640 1 --2- 172.21.15.32:0/3120783945 >> v2:172.21.15.118:3300/0 conn(0x7f08fc15fff0 0x7f08fc160400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:59:52.026 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:52.025+0000 7f0903200640 1 --2- 172.21.15.32:0/3120783945 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f08fc144f90 0x7f08fc074570 unknown :-1 s=CLOSED pgs=1986 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:59:52.026 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:52.025+0000 7f0903200640 1 -- 172.21.15.32:0/3120783945 >> 172.21.15.32:0/3120783945 conn(0x7f08fc0b1bd0 msgr2=0x7f08fc0b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:59:52.026 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:52.025+0000 7f0903200640 1 -- 172.21.15.32:0/3120783945 shutdown_connections 2024-08-03T20:59:52.027 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:52.025+0000 7f0903200640 1 -- 172.21.15.32:0/3120783945 wait complete. 2024-08-03T20:59:52.028 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:52.027+0000 7f0903200640 1 Processor -- start 2024-08-03T20:59:52.028 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:52.027+0000 7f0903200640 1 -- start start 2024-08-03T20:59:52.030 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:52.029+0000 7f0903200640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f08fc06e930 0x7f08fc1e5b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:59:52.030 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:52.029+0000 7f0903200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f08fc144f90 0x7f08fc15bc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:59:52.030 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:52.029+0000 7f0903200640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f08fc15fff0 0x7f08fc0aeb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:59:52.030 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:52.029+0000 7f0903200640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f08fc0cf840 con 0x7f08fc144f90 2024-08-03T20:59:52.030 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:52.029+0000 7f0903200640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f08fc145f30 con 0x7f08fc06e930 2024-08-03T20:59:52.030 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:52.029+0000 7f0903200640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7f08fc0cf9c0 con 0x7f08fc15fff0 2024-08-03T20:59:52.030 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:52.029+0000 7f08f3400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f08fc144f90 0x7f08fc15bc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:59:52.031 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:52.029+0000 7f08f3400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f08fc144f90 0x7f08fc15bc80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:54142/0 (socket says 172.21.15.32:54142) 2024-08-03T20:59:52.033 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:52.029+0000 7f08f3400640 1 -- 172.21.15.32:0/2663718457 learned_addr learned my addr 172.21.15.32:0/2663718457 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:59:52.033 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:52.029+0000 7f08fbe00640 1 --2- 172.21.15.32:0/2663718457 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f08fc06e930 0x7f08fc1e5b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:59:52.033 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:52.029+0000 7f0901200640 1 --2- 172.21.15.32:0/2663718457 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f08fc15fff0 0x7f08fc0aeb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:59:52.034 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:52.030+0000 7f08f3400640 1 -- 172.21.15.32:0/2663718457 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f08fc15fff0 msgr2=0x7f08fc0aeb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:59:52.034 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:52.030+0000 7f08f3400640 1 --2- 172.21.15.32:0/2663718457 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f08fc15fff0 0x7f08fc0aeb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:59:52.034 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:52.030+0000 7f08f3400640 1 -- 172.21.15.32:0/2663718457 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f08fc06e930 msgr2=0x7f08fc1e5b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:59:52.034 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:52.030+0000 7f08f3400640 1 --2- 172.21.15.32:0/2663718457 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f08fc06e930 0x7f08fc1e5b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:59:52.034 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:52.030+0000 7f08f3400640 1 -- 172.21.15.32:0/2663718457 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f08fc1f53e0 con 0x7f08fc144f90 2024-08-03T20:59:52.035 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:52.030+0000 7f08f3400640 1 --2- 172.21.15.32:0/2663718457 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f08fc144f90 0x7f08fc15bc80 secure :-1 s=READY pgs=1987 cs=0 l=1 rev1=1 crypto rx=0x7f08e40098f0 tx=0x7f08e4033c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:59:52.035 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:52.030+0000 7f08f8c00640 1 -- 172.21.15.32:0/2663718457 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08e4005c90 con 0x7f08fc144f90 2024-08-03T20:59:52.036 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:52.030+0000 7f08f8c00640 1 -- 172.21.15.32:0/2663718457 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f08e4031e50 con 0x7f08fc144f90 2024-08-03T20:59:52.036 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:52.030+0000 7f08f8c00640 1 -- 172.21.15.32:0/2663718457 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08e4005260 con 0x7f08fc144f90 2024-08-03T20:59:52.036 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:52.030+0000 7f0903200640 1 -- 172.21.15.32:0/2663718457 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f08fc1f56c0 con 0x7f08fc144f90 2024-08-03T20:59:52.036 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:52.031+0000 7f0903200640 1 -- 172.21.15.32:0/2663718457 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f08fc1f5c80 con 0x7f08fc144f90 2024-08-03T20:59:52.036 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:52.031+0000 7f0903200640 1 -- 172.21.15.32:0/2663718457 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f08fc073070 con 0x7f08fc144f90 2024-08-03T20:59:52.036 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:52.033+0000 7f08f8c00640 1 -- 172.21.15.32:0/2663718457 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f08e4005400 con 0x7f08fc144f90 2024-08-03T20:59:52.036 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:52.034+0000 7f08f8c00640 1 --2- 172.21.15.32:0/2663718457 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f08d8077010 0x7f08d8079520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:59:52.036 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:52.034+0000 7f08f8c00640 1 -- 172.21.15.32:0/2663718457 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(121..121 src has 1..121) ==== 8950+0+0 (secure 0 0 0) 0x7f08e40ba600 con 0x7f08fc144f90 2024-08-03T20:59:52.036 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:52.034+0000 7f08fbe00640 1 --2- 172.21.15.32:0/2663718457 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f08d8077010 0x7f08d8079520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:59:52.036 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:52.035+0000 7f08fbe00640 1 --2- 172.21.15.32:0/2663718457 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f08d8077010 0x7f08d8079520 secure :-1 s=READY pgs=797 cs=0 l=1 rev1=1 crypto rx=0x7f08ec007fd0 tx=0x7f08ec001370 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:59:52.038 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:52.038+0000 7f08f8c00640 1 -- 172.21.15.32:0/2663718457 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7f08fc073070 con 0x7f08fc144f90 2024-08-03T20:59:52.318 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:52.318+0000 7f0903200640 1 -- 172.21.15.32:0/2663718457 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "mds repaired", "role": "0"} v 0) -- 0x7f08fc160c30 con 0x7f08fc144f90 2024-08-03T20:59:52.781 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:52.780+0000 7f08f8c00640 1 -- 172.21.15.32:0/2663718457 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "mds repaired", "role": "0"}]=0 repaired: restoring rank 6:0 v303) ==== 101+0+0 (secure 0 0 0) 0x7f08e403f030 con 0x7f08fc144f90 2024-08-03T20:59:52.781 INFO:teuthology.orchestra.run.smithi032.stderr:repaired: restoring rank 6:0 2024-08-03T20:59:52.785 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:52.783+0000 7f0903200640 1 -- 172.21.15.32:0/2663718457 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f08d8077010 msgr2=0x7f08d8079520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:59:52.785 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:52.783+0000 7f0903200640 1 --2- 172.21.15.32:0/2663718457 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f08d8077010 0x7f08d8079520 secure :-1 s=READY pgs=797 cs=0 l=1 rev1=1 crypto rx=0x7f08ec007fd0 tx=0x7f08ec001370 comp rx=0 tx=0).stop 2024-08-03T20:59:52.785 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:52.783+0000 7f0903200640 1 -- 172.21.15.32:0/2663718457 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f08fc144f90 msgr2=0x7f08fc15bc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:59:52.785 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:52.783+0000 7f0903200640 1 --2- 172.21.15.32:0/2663718457 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f08fc144f90 0x7f08fc15bc80 secure :-1 s=READY pgs=1987 cs=0 l=1 rev1=1 crypto rx=0x7f08e40098f0 tx=0x7f08e4033c20 comp rx=0 tx=0).stop 2024-08-03T20:59:52.785 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:52.783+0000 7f0903200640 1 -- 172.21.15.32:0/2663718457 shutdown_connections 2024-08-03T20:59:52.785 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:52.783+0000 7f0903200640 1 --2- 172.21.15.32:0/2663718457 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f08d8077010 0x7f08d8079520 unknown :-1 s=CLOSED pgs=797 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:59:52.785 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:52.783+0000 7f0903200640 1 --2- 172.21.15.32:0/2663718457 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f08fc15fff0 0x7f08fc0aeb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:59:52.785 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:52.783+0000 7f0903200640 1 --2- 172.21.15.32:0/2663718457 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f08fc144f90 0x7f08fc15bc80 unknown :-1 s=CLOSED pgs=1987 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:59:52.785 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:52.783+0000 7f0903200640 1 --2- 172.21.15.32:0/2663718457 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f08fc06e930 0x7f08fc1e5b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:59:52.785 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:52.783+0000 7f0903200640 1 -- 172.21.15.32:0/2663718457 >> 172.21.15.32:0/2663718457 conn(0x7f08fc0b1bd0 msgr2=0x7f08fc06ef70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:59:52.785 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:52.783+0000 7f0903200640 1 -- 172.21.15.32:0/2663718457 shutdown_connections 2024-08-03T20:59:52.785 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:52.783+0000 7f0903200640 1 -- 172.21.15.32:0/2663718457 wait complete. 2024-08-03T20:59:52.807 DEBUG:teuthology.orchestra.run.smithi032:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata import - 2024-08-03T20:59:52.951 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:52.951+0000 7f81c5fedf40 1 Processor -- start 2024-08-03T20:59:52.952 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:52.951+0000 7f81c5fedf40 1 -- start start 2024-08-03T20:59:52.953 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:52.952+0000 7f81c5fedf40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5588fa7dfb40 0x5588fa799160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:59:52.953 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:52.952+0000 7f81c5fedf40 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x5588fa852be0 0x5588fa853940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:59:52.953 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:52.952+0000 7f81c5fedf40 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5588fa7e3650 0x5588fa871680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:59:52.953 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:52.952+0000 7f81c5fedf40 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x5588fa746bf0 con 0x5588fa7e3650 2024-08-03T20:59:52.953 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:52.952+0000 7f81c5fedf40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x5588fa7536b0 con 0x5588fa7dfb40 2024-08-03T20:59:52.953 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:52.952+0000 7f81c5fedf40 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x5588fa6e8e30 con 0x5588fa852be0 2024-08-03T20:59:52.953 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:52.952+0000 7f81bfc00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5588fa7e3650 0x5588fa871680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:59:52.954 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:52.952+0000 7f81bfc00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5588fa7e3650 0x5588fa871680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:54146/0 (socket says 172.21.15.32:54146) 2024-08-03T20:59:52.954 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:52.953+0000 7f81bfc00640 1 -- 172.21.15.32:0/4187305300 learned_addr learned my addr 172.21.15.32:0/4187305300 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:59:52.954 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:52.953+0000 7f81be800640 1 --2- 172.21.15.32:0/4187305300 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x5588fa852be0 0x5588fa853940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:59:52.954 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:52.953+0000 7f81bf200640 1 --2- 172.21.15.32:0/4187305300 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5588fa7dfb40 0x5588fa799160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:59:52.955 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:52.953+0000 7f81bfc00640 1 -- 172.21.15.32:0/4187305300 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x5588fa852be0 msgr2=0x5588fa853940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:59:52.955 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:52.953+0000 7f81bfc00640 1 --2- 172.21.15.32:0/4187305300 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x5588fa852be0 0x5588fa853940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:59:52.955 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:52.953+0000 7f81bfc00640 1 -- 172.21.15.32:0/4187305300 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5588fa7dfb40 msgr2=0x5588fa799160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:59:52.956 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:52.953+0000 7f81bfc00640 1 --2- 172.21.15.32:0/4187305300 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5588fa7dfb40 0x5588fa799160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:59:52.956 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:52.953+0000 7f81bfc00640 1 -- 172.21.15.32:0/4187305300 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5588fa79ad30 con 0x5588fa7e3650 2024-08-03T20:59:52.956 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:52.953+0000 7f81bfc00640 1 --2- 172.21.15.32:0/4187305300 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5588fa7e3650 0x5588fa871680 secure :-1 s=READY pgs=1989 cs=0 l=1 rev1=1 crypto rx=0x7f81b8001620 tx=0x7f81b8001ae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e9334b0aa83dd1e0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:59:52.956 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:52.954+0000 7f81bd400640 1 -- 172.21.15.32:0/4187305300 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f81b8016070 con 0x5588fa7e3650 2024-08-03T20:59:52.956 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:52.954+0000 7f81bd400640 1 -- 172.21.15.32:0/4187305300 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f81b800f800 con 0x5588fa7e3650 2024-08-03T20:59:52.956 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:52.954+0000 7f81bd400640 1 -- 172.21.15.32:0/4187305300 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f81b800fc10 con 0x5588fa7e3650 2024-08-03T20:59:52.956 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:52.954+0000 7f81c5fedf40 1 -- 172.21.15.32:0/4187305300 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5588fa7e3650 msgr2=0x5588fa871680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:59:52.956 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:52.954+0000 7f81c5fedf40 1 --2- 172.21.15.32:0/4187305300 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5588fa7e3650 0x5588fa871680 secure :-1 s=READY pgs=1989 cs=0 l=1 rev1=1 crypto rx=0x7f81b8001620 tx=0x7f81b8001ae0 comp rx=0 tx=0).stop 2024-08-03T20:59:52.956 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:52.954+0000 7f81c5fedf40 1 -- 172.21.15.32:0/4187305300 shutdown_connections 2024-08-03T20:59:52.956 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:52.954+0000 7f81c5fedf40 1 --2- 172.21.15.32:0/4187305300 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5588fa7e3650 0x5588fa871680 unknown :-1 s=CLOSED pgs=1989 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:59:52.956 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:52.954+0000 7f81c5fedf40 1 --2- 172.21.15.32:0/4187305300 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x5588fa852be0 0x5588fa853940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:59:52.956 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:52.954+0000 7f81c5fedf40 1 --2- 172.21.15.32:0/4187305300 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5588fa7dfb40 0x5588fa799160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:59:52.956 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:52.954+0000 7f81c5fedf40 1 -- 172.21.15.32:0/4187305300 >> 172.21.15.32:0/4187305300 conn(0x5588fa6baac0 msgr2=0x5588fa854380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:59:52.957 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:52.954+0000 7f81c5fedf40 1 -- 172.21.15.32:0/4187305300 shutdown_connections 2024-08-03T20:59:52.957 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:52.954+0000 7f81c5fedf40 1 -- 172.21.15.32:0/4187305300 wait complete. 2024-08-03T20:59:52.958 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:52.955+0000 7f81c5fedf40 1 Processor -- start 2024-08-03T20:59:52.958 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:52.956+0000 7f81c5fedf40 1 -- start start 2024-08-03T20:59:52.958 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:52.957+0000 7f81c5fedf40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5588fa7dfb40 0x5588fa84b8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:59:52.958 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:52.957+0000 7f81c5fedf40 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5588fa7e3650 0x5588fa84d070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:59:52.958 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:52.957+0000 7f81c5fedf40 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x5588fa852be0 0x5588fa858100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:59:52.958 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:52.957+0000 7f81c5fedf40 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x5588fa79aa50 con 0x5588fa7e3650 2024-08-03T20:59:52.958 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:52.957+0000 7f81c5fedf40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x5588fa7986a0 con 0x5588fa7dfb40 2024-08-03T20:59:52.958 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:52.957+0000 7f81c5fedf40 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x5588fa84d9e0 con 0x5588fa852be0 2024-08-03T20:59:52.958 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:52.957+0000 7f81be800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5588fa7e3650 0x5588fa84d070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:59:52.959 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:52.957+0000 7f81be800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5588fa7e3650 0x5588fa84d070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:54148/0 (socket says 172.21.15.32:54148) 2024-08-03T20:59:52.959 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:52.957+0000 7f81be800640 1 -- 172.21.15.32:0/2475970920 learned_addr learned my addr 172.21.15.32:0/2475970920 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:59:52.959 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:52.958+0000 7f81bf200640 1 --2- 172.21.15.32:0/2475970920 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5588fa7dfb40 0x5588fa84b8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:59:52.959 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:52.958+0000 7f81bfc00640 1 --2- 172.21.15.32:0/2475970920 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x5588fa852be0 0x5588fa858100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:59:52.960 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:52.958+0000 7f81be800640 1 -- 172.21.15.32:0/2475970920 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x5588fa852be0 msgr2=0x5588fa858100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:59:52.960 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:52.958+0000 7f81be800640 1 --2- 172.21.15.32:0/2475970920 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x5588fa852be0 0x5588fa858100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:59:52.960 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:52.958+0000 7f81be800640 1 -- 172.21.15.32:0/2475970920 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5588fa7dfb40 msgr2=0x5588fa84b8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:59:52.960 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:52.958+0000 7f81be800640 1 --2- 172.21.15.32:0/2475970920 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5588fa7dfb40 0x5588fa84b8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:59:52.960 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:52.958+0000 7f81be800640 1 -- 172.21.15.32:0/2475970920 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5588fa850a20 con 0x5588fa7e3650 2024-08-03T20:59:52.960 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:52.958+0000 7f81be800640 1 --2- 172.21.15.32:0/2475970920 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5588fa7e3650 0x5588fa84d070 secure :-1 s=READY pgs=1990 cs=0 l=1 rev1=1 crypto rx=0x7f81ac0465c0 tx=0x7f81ac04d880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:59:52.960 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:52.959+0000 7f81b7e00640 1 -- 172.21.15.32:0/2475970920 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f81ac066030 con 0x5588fa7e3650 2024-08-03T20:59:52.961 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:52.959+0000 7f81c5fedf40 1 -- 172.21.15.32:0/2475970920 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x5588fa850740 con 0x5588fa7e3650 2024-08-03T20:59:52.961 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:52.959+0000 7f81b7e00640 1 -- 172.21.15.32:0/2475970920 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f81ac056620 con 0x5588fa7e3650 2024-08-03T20:59:52.961 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:52.959+0000 7f81b7e00640 1 -- 172.21.15.32:0/2475970920 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f81ac0567f0 con 0x5588fa7e3650 2024-08-03T20:59:52.961 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:52.959+0000 7f81c5fedf40 1 -- 172.21.15.32:0/2475970920 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5588fa857900 con 0x5588fa7e3650 2024-08-03T20:59:52.961 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:52.961+0000 7f81b7e00640 1 -- 172.21.15.32:0/2475970920 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f81ac069070 con 0x5588fa7e3650 2024-08-03T20:59:52.962 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:52.962+0000 7f81b7e00640 1 --2- 172.21.15.32:0/2475970920 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f81a0075a20 0x7f81a0077f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:59:52.962 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:52.962+0000 7f81b7e00640 1 -- 172.21.15.32:0/2475970920 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(121..121 src has 1..121) ==== 8950+0+0 (secure 0 0 0) 0x7f81ac0dc6b0 con 0x5588fa7e3650 2024-08-03T20:59:52.963 INFO:teuthology.orchestra.run.smithi032.stdout:Importing pool 2024-08-03T20:59:52.963 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:52.962+0000 7f81bf200640 1 --2- 172.21.15.32:0/2475970920 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f81a0075a20 0x7f81a0077f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:59:52.964 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:52.962+0000 7f81c5fedf40 1 --2- 172.21.15.32:0/2475970920 >> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] conn(0x5588fa871b20 0x5588fa913700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:59:52.964 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:52.962+0000 7f81c5fedf40 1 -- 172.21.15.32:0/2475970920 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:1 12.10 12:09401a81:::601.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121) -- 0x5588fa858640 con 0x5588fa871b20 2024-08-03T20:59:52.964 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:52.962+0000 7f81bfc00640 1 --2- 172.21.15.32:0/2475970920 >> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] conn(0x5588fa871b20 0x5588fa913700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:59:52.964 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:52.963+0000 7f81bf200640 1 --2- 172.21.15.32:0/2475970920 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f81a0075a20 0x7f81a0077f30 secure :-1 s=READY pgs=798 cs=0 l=1 rev1=1 crypto rx=0x7f81b0000f60 tx=0x7f81b0001790 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:59:52.964 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:52.963+0000 7f81bfc00640 1 --2- 172.21.15.32:0/2475970920 >> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] conn(0x5588fa871b20 0x5588fa913700 crc :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:59:52.966 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:52.966+0000 7f81bfc00640 1 -- 172.21.15.32:0/2475970920 <== osd.0 v2:172.21.15.32:6816/3066156116 1 ==== osd_op_reply(1 601.00000000 [create] v121'1097 uv1070 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f81b800f070 con 0x5588fa871b20 2024-08-03T20:59:52.966 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:52.966+0000 7f81c5fedf40 1 -- 172.21.15.32:0/2475970920 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:2 12.10 12:09401a81:::601.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e121) -- 0x5588fa913c40 con 0x5588fa871b20 2024-08-03T20:59:52.968 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:52.968+0000 7f81bfc00640 1 -- 172.21.15.32:0/2475970920 <== osd.0 v2:172.21.15.32:6816/3066156116 2 ==== osd_op_reply(2 601.00000000 [delete] v121'1098 uv1098 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f81b800f070 con 0x5588fa871b20 2024-08-03T20:59:52.968 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:52.968+0000 7f81c5fedf40 1 -- 172.21.15.32:0/2475970920 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:3 12.10 12:09401a81:::601.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121) -- 0x5588fa913fe0 con 0x5588fa871b20 2024-08-03T20:59:52.970 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::601.00000000:head# 2024-08-03T20:59:52.970 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:52.969+0000 7f81bfc00640 1 -- 172.21.15.32:0/2475970920 <== osd.0 v2:172.21.15.32:6816/3066156116 3 ==== osd_op_reply(3 601.00000000 [create] v121'1099 uv1099 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f81b800f070 con 0x5588fa871b20 2024-08-03T20:59:52.970 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:52.969+0000 7f81c5fedf40 1 -- 172.21.15.32:0/2475970920 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:4 12.10 12:09401a81:::601.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121) -- 0x5588fa915390 con 0x5588fa871b20 2024-08-03T20:59:52.972 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:52.971+0000 7f81bfc00640 1 -- 172.21.15.32:0/2475970920 <== osd.0 v2:172.21.15.32:6816/3066156116 4 ==== osd_op_reply(4 601.00000000 [setxattr (30)] v121'1100 uv1100 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f81b800f070 con 0x5588fa871b20 2024-08-03T20:59:52.972 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:52.971+0000 7f81c5fedf40 1 -- 172.21.15.32:0/2475970920 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:5 12.10 12:09401a81:::601.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121) -- 0x5588fa915730 con 0x5588fa871b20 2024-08-03T20:59:52.973 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:52.973+0000 7f81bfc00640 1 -- 172.21.15.32:0/2475970920 <== osd.0 v2:172.21.15.32:6816/3066156116 5 ==== osd_op_reply(5 601.00000000 [setxattr (62)] v121'1101 uv1101 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f81b800f070 con 0x5588fa871b20 2024-08-03T20:59:52.974 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:52.973+0000 7f81c5fedf40 1 -- 172.21.15.32:0/2475970920 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:6 12.10 12:09401a81:::601.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121) -- 0x5588fa914380 con 0x5588fa871b20 2024-08-03T20:59:52.976 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:52.975+0000 7f81bfc00640 1 -- 172.21.15.32:0/2475970920 <== osd.0 v2:172.21.15.32:6816/3066156116 6 ==== osd_op_reply(6 601.00000000 [omap-set-header] v121'1102 uv1102 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f81b800f070 con 0x5588fa871b20 2024-08-03T20:59:52.976 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:52.975+0000 7f81c5fedf40 1 -- 172.21.15.32:0/2475970920 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:7 12.10 12:0bd6d154:::602.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121) -- 0x5588fa914720 con 0x5588fa871b20 2024-08-03T20:59:52.977 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:52.977+0000 7f81bfc00640 1 -- 172.21.15.32:0/2475970920 <== osd.0 v2:172.21.15.32:6816/3066156116 7 ==== osd_op_reply(7 602.00000000 [create] v121'1103 uv1076 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f81b800f070 con 0x5588fa871b20 2024-08-03T20:59:52.977 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:52.977+0000 7f81c5fedf40 1 -- 172.21.15.32:0/2475970920 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:8 12.10 12:0bd6d154:::602.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e121) -- 0x5588fa914b10 con 0x5588fa871b20 2024-08-03T20:59:52.979 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:52.979+0000 7f81bfc00640 1 -- 172.21.15.32:0/2475970920 <== osd.0 v2:172.21.15.32:6816/3066156116 8 ==== osd_op_reply(8 602.00000000 [delete] v121'1104 uv1104 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f81b800f070 con 0x5588fa871b20 2024-08-03T20:59:52.979 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:52.979+0000 7f81c5fedf40 1 -- 172.21.15.32:0/2475970920 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:9 12.10 12:0bd6d154:::602.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121) -- 0x5588fa914b10 con 0x5588fa871b20 2024-08-03T20:59:52.981 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::602.00000000:head# 2024-08-03T20:59:52.981 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:52.981+0000 7f81bfc00640 1 -- 172.21.15.32:0/2475970920 <== osd.0 v2:172.21.15.32:6816/3066156116 9 ==== osd_op_reply(9 602.00000000 [create] v121'1105 uv1105 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f81b800f070 con 0x5588fa871b20 2024-08-03T20:59:52.981 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:52.981+0000 7f81c5fedf40 1 -- 172.21.15.32:0/2475970920 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:10 12.10 12:0bd6d154:::602.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121) -- 0x5588fa914ac0 con 0x5588fa871b20 2024-08-03T20:59:52.983 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:52.982+0000 7f81bfc00640 1 -- 172.21.15.32:0/2475970920 <== osd.0 v2:172.21.15.32:6816/3066156116 10 ==== osd_op_reply(10 602.00000000 [setxattr (30)] v121'1106 uv1106 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f81b800f070 con 0x5588fa871b20 2024-08-03T20:59:52.983 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:52.982+0000 7f81c5fedf40 1 -- 172.21.15.32:0/2475970920 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:11 12.10 12:0bd6d154:::602.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121) -- 0x5588fa914ac0 con 0x5588fa871b20 2024-08-03T20:59:52.985 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:52.984+0000 7f81bfc00640 1 -- 172.21.15.32:0/2475970920 <== osd.0 v2:172.21.15.32:6816/3066156116 11 ==== osd_op_reply(11 602.00000000 [setxattr (62)] v121'1107 uv1107 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f81b800f070 con 0x5588fa871b20 2024-08-03T20:59:52.985 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:52.984+0000 7f81c5fedf40 1 -- 172.21.15.32:0/2475970920 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:12 12.10 12:0bd6d154:::602.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121) -- 0x5588fa914ac0 con 0x5588fa871b20 2024-08-03T20:59:52.987 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:52.986+0000 7f81bfc00640 1 -- 172.21.15.32:0/2475970920 <== osd.0 v2:172.21.15.32:6816/3066156116 12 ==== osd_op_reply(12 602.00000000 [omap-set-header] v121'1108 uv1108 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f81b800f070 con 0x5588fa871b20 2024-08-03T20:59:52.987 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:52.986+0000 7f81c5fedf40 1 -- 172.21.15.32:0/2475970920 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:13 12.10 12:0cf6ad0f:::10000000000.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121) -- 0x5588fa914ac0 con 0x5588fa871b20 2024-08-03T20:59:52.989 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:52.988+0000 7f81bfc00640 1 -- 172.21.15.32:0/2475970920 <== osd.0 v2:172.21.15.32:6816/3066156116 13 ==== osd_op_reply(13 10000000000.00000000 [create] v121'1109 uv1083 ondisk = -17 ((17) File exists)) ==== 164+0+0 (crc 0 0 0) 0x7f81b800f070 con 0x5588fa871b20 2024-08-03T20:59:52.989 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:52.988+0000 7f81c5fedf40 1 -- 172.21.15.32:0/2475970920 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:14 12.10 12:0cf6ad0f:::10000000000.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e121) -- 0x5588fa914ac0 con 0x5588fa871b20 2024-08-03T20:59:52.990 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:52.990+0000 7f81bfc00640 1 -- 172.21.15.32:0/2475970920 <== osd.0 v2:172.21.15.32:6816/3066156116 14 ==== osd_op_reply(14 10000000000.00000000 [delete] v121'1110 uv1110 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f81b800f070 con 0x5588fa871b20 2024-08-03T20:59:52.991 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:52.990+0000 7f81c5fedf40 1 -- 172.21.15.32:0/2475970920 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:15 12.10 12:0cf6ad0f:::10000000000.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121) -- 0x5588fa914ac0 con 0x5588fa871b20 2024-08-03T20:59:52.993 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::10000000000.00000000:head# 2024-08-03T20:59:52.993 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:52.992+0000 7f81bfc00640 1 -- 172.21.15.32:0/2475970920 <== osd.0 v2:172.21.15.32:6816/3066156116 15 ==== osd_op_reply(15 10000000000.00000000 [create] v121'1111 uv1111 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f81b800f070 con 0x5588fa871b20 2024-08-03T20:59:52.993 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:52.992+0000 7f81c5fedf40 1 -- 172.21.15.32:0/2475970920 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:16 12.10 12:0cf6ad0f:::10000000000.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121) -- 0x5588fa914ac0 con 0x5588fa871b20 2024-08-03T20:59:52.995 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:52.994+0000 7f81bfc00640 1 -- 172.21.15.32:0/2475970920 <== osd.0 v2:172.21.15.32:6816/3066156116 16 ==== osd_op_reply(16 10000000000.00000000 [setxattr (30)] v121'1112 uv1112 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f81b800f070 con 0x5588fa871b20 2024-08-03T20:59:52.995 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:52.994+0000 7f81c5fedf40 1 -- 172.21.15.32:0/2475970920 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:17 12.10 12:0cf6ad0f:::10000000000.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121) -- 0x5588fa914ac0 con 0x5588fa871b20 2024-08-03T20:59:52.996 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:52.996+0000 7f81bfc00640 1 -- 172.21.15.32:0/2475970920 <== osd.0 v2:172.21.15.32:6816/3066156116 17 ==== osd_op_reply(17 10000000000.00000000 [setxattr (62)] v121'1113 uv1113 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f81b800f070 con 0x5588fa871b20 2024-08-03T20:59:52.997 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:52.996+0000 7f81c5fedf40 1 -- 172.21.15.32:0/2475970920 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:18 12.10 12:0cf6ad0f:::10000000000.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121) -- 0x5588fa914ac0 con 0x5588fa871b20 2024-08-03T20:59:52.998 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:52.998+0000 7f81bfc00640 1 -- 172.21.15.32:0/2475970920 <== osd.0 v2:172.21.15.32:6816/3066156116 18 ==== osd_op_reply(18 10000000000.00000000 [omap-set-header] v121'1114 uv1114 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f81b800f070 con 0x5588fa871b20 2024-08-03T20:59:52.998 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:52.998+0000 7f81c5fedf40 1 -- 172.21.15.32:0/2475970920 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:19 12.10 12:0cf6ad0f:::10000000000.00000000:head [omap-set-vals in=593b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121) -- 0x5588fa914ac0 con 0x5588fa871b20 2024-08-03T20:59:53.000 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:53.000+0000 7f81bfc00640 1 -- 172.21.15.32:0/2475970920 <== osd.0 v2:172.21.15.32:6816/3066156116 19 ==== osd_op_reply(19 10000000000.00000000 [omap-set-vals] v121'1115 uv1115 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f81b800f070 con 0x5588fa871b20 2024-08-03T20:59:53.000 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:53.000+0000 7f81c5fedf40 1 -- 172.21.15.32:0/2475970920 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:20 12.10 12:0d82a743:::600.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121) -- 0x5588fa914ac0 con 0x5588fa871b20 2024-08-03T20:59:53.002 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:53.002+0000 7f81bfc00640 1 -- 172.21.15.32:0/2475970920 <== osd.0 v2:172.21.15.32:6816/3066156116 20 ==== osd_op_reply(20 600.00000000 [create] v121'1116 uv1096 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f81b800f070 con 0x5588fa871b20 2024-08-03T20:59:53.002 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:53.002+0000 7f81c5fedf40 1 -- 172.21.15.32:0/2475970920 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:21 12.10 12:0d82a743:::600.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e121) -- 0x5588fa914ac0 con 0x5588fa871b20 2024-08-03T20:59:53.004 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:53.004+0000 7f81bfc00640 1 -- 172.21.15.32:0/2475970920 <== osd.0 v2:172.21.15.32:6816/3066156116 21 ==== osd_op_reply(21 600.00000000 [delete] v121'1117 uv1117 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f81b800f070 con 0x5588fa871b20 2024-08-03T20:59:53.004 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:53.004+0000 7f81c5fedf40 1 -- 172.21.15.32:0/2475970920 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:22 12.10 12:0d82a743:::600.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121) -- 0x5588fa914ac0 con 0x5588fa871b20 2024-08-03T20:59:53.006 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::600.00000000:head# 2024-08-03T20:59:53.006 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:53.005+0000 7f81bfc00640 1 -- 172.21.15.32:0/2475970920 <== osd.0 v2:172.21.15.32:6816/3066156116 22 ==== osd_op_reply(22 600.00000000 [create] v121'1118 uv1118 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f81b800f070 con 0x5588fa871b20 2024-08-03T20:59:53.006 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:53.005+0000 7f81c5fedf40 1 -- 172.21.15.32:0/2475970920 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:23 12.10 12:0d82a743:::600.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121) -- 0x5588fa914ac0 con 0x5588fa871b20 2024-08-03T20:59:53.007 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:53.007+0000 7f81bfc00640 1 -- 172.21.15.32:0/2475970920 <== osd.0 v2:172.21.15.32:6816/3066156116 23 ==== osd_op_reply(23 600.00000000 [setxattr (30)] v121'1119 uv1119 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f81b800f070 con 0x5588fa871b20 2024-08-03T20:59:53.008 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:53.007+0000 7f81c5fedf40 1 -- 172.21.15.32:0/2475970920 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:24 12.10 12:0d82a743:::600.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121) -- 0x5588fa914ac0 con 0x5588fa871b20 2024-08-03T20:59:53.009 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:53.009+0000 7f81bfc00640 1 -- 172.21.15.32:0/2475970920 <== osd.0 v2:172.21.15.32:6816/3066156116 24 ==== osd_op_reply(24 600.00000000 [setxattr (62)] v121'1120 uv1120 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f81b800f070 con 0x5588fa871b20 2024-08-03T20:59:53.009 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:53.009+0000 7f81c5fedf40 1 -- 172.21.15.32:0/2475970920 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:25 12.10 12:0d82a743:::600.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121) -- 0x5588fa914ac0 con 0x5588fa871b20 2024-08-03T20:59:53.011 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:53.011+0000 7f81bfc00640 1 -- 172.21.15.32:0/2475970920 <== osd.0 v2:172.21.15.32:6816/3066156116 25 ==== osd_op_reply(25 600.00000000 [omap-set-header] v121'1121 uv1121 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f81b800f070 con 0x5588fa871b20 2024-08-03T20:59:53.011 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:53.011+0000 7f81c5fedf40 1 -- 172.21.15.32:0/2475970920 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:26 12.10 12:0d89b25e:::603.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121) -- 0x5588fa914ac0 con 0x5588fa871b20 2024-08-03T20:59:53.013 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:53.013+0000 7f81bfc00640 1 -- 172.21.15.32:0/2475970920 <== osd.0 v2:172.21.15.32:6816/3066156116 26 ==== osd_op_reply(26 603.00000000 [create] v121'1122 uv1095 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f81b800f070 con 0x5588fa871b20 2024-08-03T20:59:53.013 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:53.013+0000 7f81c5fedf40 1 -- 172.21.15.32:0/2475970920 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:27 12.10 12:0d89b25e:::603.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e121) -- 0x5588fa914ac0 con 0x5588fa871b20 2024-08-03T20:59:53.015 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:53.015+0000 7f81bfc00640 1 -- 172.21.15.32:0/2475970920 <== osd.0 v2:172.21.15.32:6816/3066156116 27 ==== osd_op_reply(27 603.00000000 [delete] v121'1123 uv1123 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f81b800f070 con 0x5588fa871b20 2024-08-03T20:59:53.015 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:53.015+0000 7f81c5fedf40 1 -- 172.21.15.32:0/2475970920 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:28 12.10 12:0d89b25e:::603.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121) -- 0x5588fa914ac0 con 0x5588fa871b20 2024-08-03T20:59:53.017 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::603.00000000:head# 2024-08-03T20:59:53.017 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:53.016+0000 7f81bfc00640 1 -- 172.21.15.32:0/2475970920 <== osd.0 v2:172.21.15.32:6816/3066156116 28 ==== osd_op_reply(28 603.00000000 [create] v121'1124 uv1124 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f81b800f070 con 0x5588fa871b20 2024-08-03T20:59:53.017 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:53.016+0000 7f81c5fedf40 1 -- 172.21.15.32:0/2475970920 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:29 12.10 12:0d89b25e:::603.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121) -- 0x5588fa914ac0 con 0x5588fa871b20 2024-08-03T20:59:53.019 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:53.018+0000 7f81bfc00640 1 -- 172.21.15.32:0/2475970920 <== osd.0 v2:172.21.15.32:6816/3066156116 29 ==== osd_op_reply(29 603.00000000 [setxattr (30)] v121'1125 uv1125 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f81b800f070 con 0x5588fa871b20 2024-08-03T20:59:53.019 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:53.018+0000 7f81c5fedf40 1 -- 172.21.15.32:0/2475970920 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:30 12.10 12:0d89b25e:::603.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121) -- 0x5588fa914ac0 con 0x5588fa871b20 2024-08-03T20:59:53.021 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:53.020+0000 7f81bfc00640 1 -- 172.21.15.32:0/2475970920 <== osd.0 v2:172.21.15.32:6816/3066156116 30 ==== osd_op_reply(30 603.00000000 [setxattr (62)] v121'1126 uv1126 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f81b800f070 con 0x5588fa871b20 2024-08-03T20:59:53.021 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:53.020+0000 7f81c5fedf40 1 -- 172.21.15.32:0/2475970920 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:31 12.10 12:0d89b25e:::603.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121) -- 0x5588fa914ac0 con 0x5588fa871b20 2024-08-03T20:59:53.023 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:53.022+0000 7f81bfc00640 1 -- 172.21.15.32:0/2475970920 <== osd.0 v2:172.21.15.32:6816/3066156116 31 ==== osd_op_reply(31 603.00000000 [omap-set-header] v121'1127 uv1127 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f81b800f070 con 0x5588fa871b20 2024-08-03T20:59:53.023 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:53.022+0000 7f81c5fedf40 1 --2- 172.21.15.32:0/2475970920 >> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] conn(0x5588fa915ad0 0x5588fa917ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:59:53.023 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:53.022+0000 7f81c5fedf40 1 -- 172.21.15.32:0/2475970920 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:32 12.14 12:287034c4:::1.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121) -- 0x5588fa914fd0 con 0x5588fa915ad0 2024-08-03T20:59:53.023 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:53.022+0000 7f81bf200640 1 --2- 172.21.15.32:0/2475970920 >> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] conn(0x5588fa915ad0 0x5588fa917ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:59:53.023 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:53.023+0000 7f81bf200640 1 --2- 172.21.15.32:0/2475970920 >> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] conn(0x5588fa915ad0 0x5588fa917ef0 crc :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:59:53.026 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:53.025+0000 7f81bf200640 1 -- 172.21.15.32:0/2475970920 <== osd.5 v2:172.21.15.118:6810/2724891770 1 ==== osd_op_reply(32 1.00000000.inode [create] v121'799 uv781 ondisk = -17 ((17) File exists)) ==== 160+0+0 (crc 0 0 0) 0x7f81b0012950 con 0x5588fa915ad0 2024-08-03T20:59:53.026 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:53.025+0000 7f81c5fedf40 1 -- 172.21.15.32:0/2475970920 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:33 12.14 12:287034c4:::1.00000000.inode:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e121) -- 0x5588fa918430 con 0x5588fa915ad0 2024-08-03T20:59:53.027 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:53.027+0000 7f81bf200640 1 -- 172.21.15.32:0/2475970920 <== osd.5 v2:172.21.15.118:6810/2724891770 2 ==== osd_op_reply(33 1.00000000.inode [delete] v121'800 uv800 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f81b0012950 con 0x5588fa915ad0 2024-08-03T20:59:53.027 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:53.027+0000 7f81c5fedf40 1 -- 172.21.15.32:0/2475970920 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:34 12.14 12:287034c4:::1.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121) -- 0x5588fa9187d0 con 0x5588fa915ad0 2024-08-03T20:59:53.029 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::1.00000000.inode:head# 2024-08-03T20:59:53.029 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:53.028+0000 7f81bf200640 1 -- 172.21.15.32:0/2475970920 <== osd.5 v2:172.21.15.118:6810/2724891770 3 ==== osd_op_reply(34 1.00000000.inode [create] v121'801 uv801 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f81b0012950 con 0x5588fa915ad0 2024-08-03T20:59:53.029 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:53.028+0000 7f81c5fedf40 1 -- 172.21.15.32:0/2475970920 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:35 12.14 12:287034c4:::1.00000000.inode:head [write 0~582 in=582b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121) -- 0x5588fa918b70 con 0x5588fa915ad0 2024-08-03T20:59:53.031 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:53.030+0000 7f81bf200640 1 -- 172.21.15.32:0/2475970920 <== osd.5 v2:172.21.15.118:6810/2724891770 4 ==== osd_op_reply(35 1.00000000.inode [write 0~582] v121'802 uv802 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f81b0012950 con 0x5588fa915ad0 2024-08-03T20:59:53.031 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:53.030+0000 7f81c5fedf40 1 -- 172.21.15.32:0/2475970920 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:36 12.14 12:287034c4:::1.00000000.inode:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121) -- 0x5588fa918f10 con 0x5588fa915ad0 2024-08-03T20:59:53.032 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:53.032+0000 7f81bf200640 1 -- 172.21.15.32:0/2475970920 <== osd.5 v2:172.21.15.118:6810/2724891770 5 ==== osd_op_reply(36 1.00000000.inode [omap-set-header] v121'803 uv803 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f81b0012950 con 0x5588fa915ad0 2024-08-03T20:59:53.032 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:53.032+0000 7f81c5fedf40 1 -- 172.21.15.32:0/2475970920 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:37 12.14 12:292cf221:::200.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121) -- 0x5588fa919300 con 0x5588fa915ad0 2024-08-03T20:59:53.034 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:53.034+0000 7f81bf200640 1 -- 172.21.15.32:0/2475970920 <== osd.5 v2:172.21.15.118:6810/2724891770 6 ==== osd_op_reply(37 200.00000000 [create] v121'804 uv786 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f81b0012950 con 0x5588fa915ad0 2024-08-03T20:59:53.034 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:53.034+0000 7f81c5fedf40 1 -- 172.21.15.32:0/2475970920 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:38 12.14 12:292cf221:::200.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e121) -- 0x5588fa9196f0 con 0x5588fa915ad0 2024-08-03T20:59:53.036 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:53.035+0000 7f81bf200640 1 -- 172.21.15.32:0/2475970920 <== osd.5 v2:172.21.15.118:6810/2724891770 7 ==== osd_op_reply(38 200.00000000 [delete] v121'805 uv805 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f81b0012950 con 0x5588fa915ad0 2024-08-03T20:59:53.036 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:53.035+0000 7f81c5fedf40 1 -- 172.21.15.32:0/2475970920 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:39 12.14 12:292cf221:::200.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121) -- 0x5588fa919ae0 con 0x5588fa915ad0 2024-08-03T20:59:53.037 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::200.00000000:head# 2024-08-03T20:59:53.038 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:53.037+0000 7f81bf200640 1 -- 172.21.15.32:0/2475970920 <== osd.5 v2:172.21.15.118:6810/2724891770 8 ==== osd_op_reply(39 200.00000000 [create] v121'806 uv806 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f81b0012950 con 0x5588fa915ad0 2024-08-03T20:59:53.038 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:53.037+0000 7f81c5fedf40 1 -- 172.21.15.32:0/2475970920 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:40 12.14 12:292cf221:::200.00000000:head [write 0~90 in=90b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121) -- 0x5588fa919ae0 con 0x5588fa915ad0 2024-08-03T20:59:53.039 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:53.039+0000 7f81bf200640 1 -- 172.21.15.32:0/2475970920 <== osd.5 v2:172.21.15.118:6810/2724891770 9 ==== osd_op_reply(40 200.00000000 [write 0~90] v121'807 uv807 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f81b0012950 con 0x5588fa915ad0 2024-08-03T20:59:53.039 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:53.039+0000 7f81c5fedf40 1 -- 172.21.15.32:0/2475970920 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:41 12.14 12:292cf221:::200.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121) -- 0x5588fa919a90 con 0x5588fa915ad0 2024-08-03T20:59:53.041 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:53.040+0000 7f81bf200640 1 -- 172.21.15.32:0/2475970920 <== osd.5 v2:172.21.15.118:6810/2724891770 10 ==== osd_op_reply(41 200.00000000 [omap-set-header] v121'808 uv808 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f81b0012950 con 0x5588fa915ad0 2024-08-03T20:59:53.041 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:53.040+0000 7f81c5fedf40 1 -- 172.21.15.32:0/2475970920 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:42 12.14 12:2e2fa760:::200.00000001:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121) -- 0x5588fa919a90 con 0x5588fa915ad0 2024-08-03T20:59:53.042 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:53.042+0000 7f81bf200640 1 -- 172.21.15.32:0/2475970920 <== osd.5 v2:172.21.15.118:6810/2724891770 11 ==== osd_op_reply(42 200.00000001 [create] v121'809 uv791 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f81b0012950 con 0x5588fa915ad0 2024-08-03T20:59:53.043 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:53.042+0000 7f81c5fedf40 1 -- 172.21.15.32:0/2475970920 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:43 12.14 12:2e2fa760:::200.00000001:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e121) -- 0x5588fa919a90 con 0x5588fa915ad0 2024-08-03T20:59:53.044 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:53.044+0000 7f81bf200640 1 -- 172.21.15.32:0/2475970920 <== osd.5 v2:172.21.15.118:6810/2724891770 12 ==== osd_op_reply(43 200.00000001 [delete] v121'810 uv810 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f81b0012950 con 0x5588fa915ad0 2024-08-03T20:59:53.044 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:53.044+0000 7f81c5fedf40 1 -- 172.21.15.32:0/2475970920 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:44 12.14 12:2e2fa760:::200.00000001:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121) -- 0x5588fa919a90 con 0x5588fa915ad0 2024-08-03T20:59:53.046 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::200.00000001:head# 2024-08-03T20:59:53.046 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:53.046+0000 7f81bf200640 1 -- 172.21.15.32:0/2475970920 <== osd.5 v2:172.21.15.118:6810/2724891770 13 ==== osd_op_reply(44 200.00000001 [create] v121'811 uv811 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f81b0012950 con 0x5588fa915ad0 2024-08-03T20:59:53.046 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:53.046+0000 7f81c5fedf40 1 -- 172.21.15.32:0/2475970920 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:45 12.14 12:2e2fa760:::200.00000001:head [write 0~18118 in=18118b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121) -- 0x5588fa91e1c0 con 0x5588fa915ad0 2024-08-03T20:59:53.048 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:53.048+0000 7f81bf200640 1 -- 172.21.15.32:0/2475970920 <== osd.5 v2:172.21.15.118:6810/2724891770 14 ==== osd_op_reply(45 200.00000001 [write 0~18118] v121'812 uv812 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f81b0012950 con 0x5588fa915ad0 2024-08-03T20:59:53.048 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:53.048+0000 7f81c5fedf40 1 -- 172.21.15.32:0/2475970920 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:46 12.14 12:2e2fa760:::200.00000001:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121) -- 0x5588fa91e180 con 0x5588fa915ad0 2024-08-03T20:59:53.050 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:53.049+0000 7f81bf200640 1 -- 172.21.15.32:0/2475970920 <== osd.5 v2:172.21.15.118:6810/2724891770 15 ==== osd_op_reply(46 200.00000001 [omap-set-header] v121'813 uv813 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f81b0012950 con 0x5588fa915ad0 2024-08-03T20:59:53.050 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:53.049+0000 7f81c5fedf40 1 -- 172.21.15.32:0/2475970920 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:47 12.14 12:2f55791f:::606.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121) -- 0x5588fa919a90 con 0x5588fa915ad0 2024-08-03T20:59:53.051 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:53.051+0000 7f81bf200640 1 -- 172.21.15.32:0/2475970920 <== osd.5 v2:172.21.15.118:6810/2724891770 16 ==== osd_op_reply(47 606.00000000 [create] v121'814 uv797 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f81b0012950 con 0x5588fa915ad0 2024-08-03T20:59:53.052 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:53.051+0000 7f81c5fedf40 1 -- 172.21.15.32:0/2475970920 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:48 12.14 12:2f55791f:::606.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e121) -- 0x5588fa919a90 con 0x5588fa915ad0 2024-08-03T20:59:53.053 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:53.053+0000 7f81bf200640 1 -- 172.21.15.32:0/2475970920 <== osd.5 v2:172.21.15.118:6810/2724891770 17 ==== osd_op_reply(48 606.00000000 [delete] v121'815 uv815 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f81b0012950 con 0x5588fa915ad0 2024-08-03T20:59:53.053 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:53.053+0000 7f81c5fedf40 1 -- 172.21.15.32:0/2475970920 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:49 12.14 12:2f55791f:::606.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121) -- 0x5588fa919a90 con 0x5588fa915ad0 2024-08-03T20:59:53.055 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::606.00000000:head# 2024-08-03T20:59:53.055 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:53.054+0000 7f81bf200640 1 -- 172.21.15.32:0/2475970920 <== osd.5 v2:172.21.15.118:6810/2724891770 18 ==== osd_op_reply(49 606.00000000 [create] v121'816 uv816 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f81b0012950 con 0x5588fa915ad0 2024-08-03T20:59:53.055 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:53.054+0000 7f81c5fedf40 1 -- 172.21.15.32:0/2475970920 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:50 12.14 12:2f55791f:::606.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121) -- 0x5588fa91aee0 con 0x5588fa915ad0 2024-08-03T20:59:53.057 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:53.056+0000 7f81bf200640 1 -- 172.21.15.32:0/2475970920 <== osd.5 v2:172.21.15.118:6810/2724891770 19 ==== osd_op_reply(50 606.00000000 [setxattr (30)] v121'817 uv817 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f81b0012950 con 0x5588fa915ad0 2024-08-03T20:59:53.057 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:53.056+0000 7f81c5fedf40 1 -- 172.21.15.32:0/2475970920 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:51 12.14 12:2f55791f:::606.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121) -- 0x5588fa91abe0 con 0x5588fa915ad0 2024-08-03T20:59:53.058 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:53.058+0000 7f81bf200640 1 -- 172.21.15.32:0/2475970920 <== osd.5 v2:172.21.15.118:6810/2724891770 20 ==== osd_op_reply(51 606.00000000 [setxattr (62)] v121'818 uv818 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f81b0012950 con 0x5588fa915ad0 2024-08-03T20:59:53.058 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:53.058+0000 7f81c5fedf40 1 -- 172.21.15.32:0/2475970920 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:52 12.14 12:2f55791f:::606.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121) -- 0x5588fa919bd0 con 0x5588fa915ad0 2024-08-03T20:59:53.060 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:53.060+0000 7f81bf200640 1 -- 172.21.15.32:0/2475970920 <== osd.5 v2:172.21.15.118:6810/2724891770 21 ==== osd_op_reply(52 606.00000000 [omap-set-header] v121'819 uv819 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f81b0012950 con 0x5588fa915ad0 2024-08-03T20:59:53.060 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:53.060+0000 7f81c5fedf40 1 -- 172.21.15.32:0/2475970920 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:53 12.1c 12:3ed09add:::607.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121) -- 0x5588fa919b80 con 0x5588fa871b20 2024-08-03T20:59:53.062 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:53.061+0000 7f81bfc00640 1 -- 172.21.15.32:0/2475970920 <== osd.0 v2:172.21.15.32:6816/3066156116 32 ==== osd_op_reply(53 607.00000000 [create] v121'213 uv212 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f81b800f070 con 0x5588fa871b20 2024-08-03T20:59:53.062 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:53.061+0000 7f81c5fedf40 1 -- 172.21.15.32:0/2475970920 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:54 12.1c 12:3ed09add:::607.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e121) -- 0x5588fa919b80 con 0x5588fa871b20 2024-08-03T20:59:53.063 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:53.063+0000 7f81bfc00640 1 -- 172.21.15.32:0/2475970920 <== osd.0 v2:172.21.15.32:6816/3066156116 33 ==== osd_op_reply(54 607.00000000 [delete] v121'214 uv214 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f81b800f070 con 0x5588fa871b20 2024-08-03T20:59:53.063 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:53.063+0000 7f81c5fedf40 1 -- 172.21.15.32:0/2475970920 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:55 12.1c 12:3ed09add:::607.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121) -- 0x5588fa919b80 con 0x5588fa871b20 2024-08-03T20:59:53.064 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::607.00000000:head# 2024-08-03T20:59:53.065 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:53.064+0000 7f81bfc00640 1 -- 172.21.15.32:0/2475970920 <== osd.0 v2:172.21.15.32:6816/3066156116 34 ==== osd_op_reply(55 607.00000000 [create] v121'215 uv215 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f81b800f070 con 0x5588fa871b20 2024-08-03T20:59:53.065 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:53.064+0000 7f81c5fedf40 1 -- 172.21.15.32:0/2475970920 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:56 12.1c 12:3ed09add:::607.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121) -- 0x5588fa91abd0 con 0x5588fa871b20 2024-08-03T20:59:53.066 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:53.065+0000 7f81bfc00640 1 -- 172.21.15.32:0/2475970920 <== osd.0 v2:172.21.15.32:6816/3066156116 35 ==== osd_op_reply(56 607.00000000 [setxattr (30)] v121'216 uv216 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f81b800f070 con 0x5588fa871b20 2024-08-03T20:59:53.066 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:53.065+0000 7f81c5fedf40 1 -- 172.21.15.32:0/2475970920 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:57 12.1c 12:3ed09add:::607.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121) -- 0x5588fa91b060 con 0x5588fa871b20 2024-08-03T20:59:53.067 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:53.067+0000 7f81bfc00640 1 -- 172.21.15.32:0/2475970920 <== osd.0 v2:172.21.15.32:6816/3066156116 36 ==== osd_op_reply(57 607.00000000 [setxattr (62)] v121'217 uv217 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f81b800f070 con 0x5588fa871b20 2024-08-03T20:59:53.067 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:53.067+0000 7f81c5fedf40 1 -- 172.21.15.32:0/2475970920 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:58 12.1c 12:3ed09add:::607.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121) -- 0x5588fa919b30 con 0x5588fa871b20 2024-08-03T20:59:53.069 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:53.068+0000 7f81bfc00640 1 -- 172.21.15.32:0/2475970920 <== osd.0 v2:172.21.15.32:6816/3066156116 37 ==== osd_op_reply(58 607.00000000 [omap-set-header] v121'218 uv218 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f81b800f070 con 0x5588fa871b20 2024-08-03T20:59:53.069 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:53.068+0000 7f81c5fedf40 1 --2- 172.21.15.32:0/2475970920 >> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] conn(0x5588fa91a1b0 0x5588fa91d020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:59:53.069 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:53.068+0000 7f81c5fedf40 1 -- 172.21.15.32:0/2475970920 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:59 12.6 12:654134d2:::mds0_openfiles.0:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121) -- 0x5588fa91a7a0 con 0x5588fa91a1b0 2024-08-03T20:59:53.069 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:53.068+0000 7f81be800640 1 --2- 172.21.15.32:0/2475970920 >> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] conn(0x5588fa91a1b0 0x5588fa91d020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:59:53.069 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:53.069+0000 7f81be800640 1 --2- 172.21.15.32:0/2475970920 >> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] conn(0x5588fa91a1b0 0x5588fa91d020 crc :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:59:53.071 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:53.070+0000 7f81be800640 1 -- 172.21.15.32:0/2475970920 <== osd.1 v2:172.21.15.32:6824/1551241759 1 ==== osd_op_reply(59 mds0_openfiles.0 [create] v121'168 uv167 ondisk = -17 ((17) File exists)) ==== 160+0+0 (crc 0 0 0) 0x7f81ac0dc3c0 con 0x5588fa91a1b0 2024-08-03T20:59:53.071 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:53.070+0000 7f81c5fedf40 1 -- 172.21.15.32:0/2475970920 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:60 12.6 12:654134d2:::mds0_openfiles.0:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e121) -- 0x5588fa91d6d0 con 0x5588fa91a1b0 2024-08-03T20:59:53.072 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:53.072+0000 7f81be800640 1 -- 172.21.15.32:0/2475970920 <== osd.1 v2:172.21.15.32:6824/1551241759 2 ==== osd_op_reply(60 mds0_openfiles.0 [delete] v121'169 uv169 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f81ac0dc3c0 con 0x5588fa91a1b0 2024-08-03T20:59:53.072 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:53.072+0000 7f81c5fedf40 1 -- 172.21.15.32:0/2475970920 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:61 12.6 12:654134d2:::mds0_openfiles.0:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121) -- 0x5588fa91dbd0 con 0x5588fa91a1b0 2024-08-03T20:59:53.073 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:53.073+0000 7f81be800640 1 -- 172.21.15.32:0/2475970920 <== osd.1 v2:172.21.15.32:6824/1551241759 3 ==== osd_op_reply(61 mds0_openfiles.0 [create] v121'170 uv170 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f81ac0dc3c0 con 0x5588fa91a1b0 2024-08-03T20:59:53.073 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds0_openfiles.0:head# 2024-08-03T20:59:53.074 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:53.073+0000 7f81c5fedf40 1 -- 172.21.15.32:0/2475970920 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:62 12.6 12:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121) -- 0x5588fa854690 con 0x5588fa91a1b0 2024-08-03T20:59:53.074 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:53.074+0000 7f81be800640 1 -- 172.21.15.32:0/2475970920 <== osd.1 v2:172.21.15.32:6824/1551241759 4 ==== osd_op_reply(62 mds0_openfiles.0 [omap-set-header] v121'171 uv171 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f81ac0dc3c0 con 0x5588fa91a1b0 2024-08-03T20:59:53.075 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:53.074+0000 7f81c5fedf40 1 -- 172.21.15.32:0/2475970920 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:63 12.e 12:717a0223:::608.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121) -- 0x5588fa91e500 con 0x5588fa871b20 2024-08-03T20:59:53.076 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:53.076+0000 7f81bfc00640 1 -- 172.21.15.32:0/2475970920 <== osd.0 v2:172.21.15.32:6816/3066156116 38 ==== osd_op_reply(63 608.00000000 [create] v121'213 uv212 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f81b800f070 con 0x5588fa871b20 2024-08-03T20:59:53.076 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:53.076+0000 7f81c5fedf40 1 -- 172.21.15.32:0/2475970920 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:64 12.e 12:717a0223:::608.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e121) -- 0x5588fa91e500 con 0x5588fa871b20 2024-08-03T20:59:53.078 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:53.077+0000 7f81bfc00640 1 -- 172.21.15.32:0/2475970920 <== osd.0 v2:172.21.15.32:6816/3066156116 39 ==== osd_op_reply(64 608.00000000 [delete] v121'214 uv214 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f81b800f070 con 0x5588fa871b20 2024-08-03T20:59:53.078 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:53.078+0000 7f81c5fedf40 1 -- 172.21.15.32:0/2475970920 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:65 12.e 12:717a0223:::608.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121) -- 0x5588fa91e500 con 0x5588fa871b20 2024-08-03T20:59:53.079 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::608.00000000:head# 2024-08-03T20:59:53.079 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:53.079+0000 7f81bfc00640 1 -- 172.21.15.32:0/2475970920 <== osd.0 v2:172.21.15.32:6816/3066156116 40 ==== osd_op_reply(65 608.00000000 [create] v121'215 uv215 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f81b800f070 con 0x5588fa871b20 2024-08-03T20:59:53.079 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:53.079+0000 7f81c5fedf40 1 -- 172.21.15.32:0/2475970920 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:66 12.e 12:717a0223:::608.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121) -- 0x5588fa91e500 con 0x5588fa871b20 2024-08-03T20:59:53.080 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:53.080+0000 7f81bfc00640 1 -- 172.21.15.32:0/2475970920 <== osd.0 v2:172.21.15.32:6816/3066156116 41 ==== osd_op_reply(66 608.00000000 [setxattr (30)] v121'216 uv216 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f81b800f070 con 0x5588fa871b20 2024-08-03T20:59:53.080 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:53.080+0000 7f81c5fedf40 1 -- 172.21.15.32:0/2475970920 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:67 12.e 12:717a0223:::608.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121) -- 0x5588fa91e500 con 0x5588fa871b20 2024-08-03T20:59:53.082 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:53.081+0000 7f81bfc00640 1 -- 172.21.15.32:0/2475970920 <== osd.0 v2:172.21.15.32:6816/3066156116 42 ==== osd_op_reply(67 608.00000000 [setxattr (62)] v121'217 uv217 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f81b800f070 con 0x5588fa871b20 2024-08-03T20:59:53.082 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:53.082+0000 7f81c5fedf40 1 -- 172.21.15.32:0/2475970920 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:68 12.e 12:717a0223:::608.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121) -- 0x5588fa91e500 con 0x5588fa871b20 2024-08-03T20:59:53.083 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:53.083+0000 7f81bfc00640 1 -- 172.21.15.32:0/2475970920 <== osd.0 v2:172.21.15.32:6816/3066156116 43 ==== osd_op_reply(68 608.00000000 [omap-set-header] v121'218 uv218 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f81b800f070 con 0x5588fa871b20 2024-08-03T20:59:53.083 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:53.083+0000 7f81c5fedf40 1 --2- 172.21.15.32:0/2475970920 >> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] conn(0x5588fa91edd0 0x5588fa9211f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:59:53.084 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:53.083+0000 7f81c5fedf40 1 -- 172.21.15.32:0/2475970920 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:69 12.15 12:a93a17c2:::604.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121) -- 0x5588fa921730 con 0x5588fa91edd0 2024-08-03T20:59:53.084 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:53.083+0000 7f81bfc00640 1 --2- 172.21.15.32:0/2475970920 >> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] conn(0x5588fa91edd0 0x5588fa9211f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:59:53.084 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:53.084+0000 7f81bfc00640 1 --2- 172.21.15.32:0/2475970920 >> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] conn(0x5588fa91edd0 0x5588fa9211f0 crc :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:59:53.086 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:53.086+0000 7f81bfc00640 1 -- 172.21.15.32:0/2475970920 <== osd.6 v2:172.21.15.118:6825/801947386 1 ==== osd_op_reply(69 604.00000000 [create] v121'393 uv387 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f81b800f070 con 0x5588fa91edd0 2024-08-03T20:59:53.086 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:53.086+0000 7f81c5fedf40 1 -- 172.21.15.32:0/2475970920 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:70 12.15 12:a93a17c2:::604.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e121) -- 0x5588fa9217b0 con 0x5588fa91edd0 2024-08-03T20:59:53.088 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:53.087+0000 7f81bfc00640 1 -- 172.21.15.32:0/2475970920 <== osd.6 v2:172.21.15.118:6825/801947386 2 ==== osd_op_reply(70 604.00000000 [delete] v121'394 uv394 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f81b800f070 con 0x5588fa91edd0 2024-08-03T20:59:53.088 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:53.087+0000 7f81c5fedf40 1 -- 172.21.15.32:0/2475970920 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:71 12.15 12:a93a17c2:::604.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121) -- 0x5588fa921ba0 con 0x5588fa91edd0 2024-08-03T20:59:53.089 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::604.00000000:head# 2024-08-03T20:59:53.089 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:53.089+0000 7f81bfc00640 1 -- 172.21.15.32:0/2475970920 <== osd.6 v2:172.21.15.118:6825/801947386 3 ==== osd_op_reply(71 604.00000000 [create] v121'395 uv395 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f81b800f070 con 0x5588fa91edd0 2024-08-03T20:59:53.089 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:53.089+0000 7f81c5fedf40 1 -- 172.21.15.32:0/2475970920 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:72 12.15 12:a93a17c2:::604.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121) -- 0x5588fa922bb0 con 0x5588fa91edd0 2024-08-03T20:59:53.090 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:53.090+0000 7f81bfc00640 1 -- 172.21.15.32:0/2475970920 <== osd.6 v2:172.21.15.118:6825/801947386 4 ==== osd_op_reply(72 604.00000000 [setxattr (30)] v121'396 uv396 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f81b800f070 con 0x5588fa91edd0 2024-08-03T20:59:53.090 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:53.090+0000 7f81c5fedf40 1 -- 172.21.15.32:0/2475970920 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:73 12.15 12:a93a17c2:::604.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121) -- 0x5588fa922bb0 con 0x5588fa91edd0 2024-08-03T20:59:53.092 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:53.091+0000 7f81bfc00640 1 -- 172.21.15.32:0/2475970920 <== osd.6 v2:172.21.15.118:6825/801947386 5 ==== osd_op_reply(73 604.00000000 [setxattr (62)] v121'397 uv397 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f81b800f070 con 0x5588fa91edd0 2024-08-03T20:59:53.092 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:53.092+0000 7f81c5fedf40 1 -- 172.21.15.32:0/2475970920 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:74 12.15 12:a93a17c2:::604.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121) -- 0x5588fa921ba0 con 0x5588fa91edd0 2024-08-03T20:59:53.093 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:53.093+0000 7f81bfc00640 1 -- 172.21.15.32:0/2475970920 <== osd.6 v2:172.21.15.118:6825/801947386 6 ==== osd_op_reply(74 604.00000000 [omap-set-header] v121'398 uv398 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f81b800f070 con 0x5588fa91edd0 2024-08-03T20:59:53.093 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:53.093+0000 7f81c5fedf40 1 -- 172.21.15.32:0/2475970920 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:75 12.15 12:aa448500:::500.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121) -- 0x5588fa921ba0 con 0x5588fa91edd0 2024-08-03T20:59:53.095 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:53.095+0000 7f81bfc00640 1 -- 172.21.15.32:0/2475970920 <== osd.6 v2:172.21.15.118:6825/801947386 7 ==== osd_op_reply(75 500.00000000 [create] v121'399 uv392 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f81b800f070 con 0x5588fa91edd0 2024-08-03T20:59:53.095 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:53.095+0000 7f81c5fedf40 1 -- 172.21.15.32:0/2475970920 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:76 12.15 12:aa448500:::500.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e121) -- 0x5588fa921ba0 con 0x5588fa91edd0 2024-08-03T20:59:53.096 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:53.096+0000 7f81bfc00640 1 -- 172.21.15.32:0/2475970920 <== osd.6 v2:172.21.15.118:6825/801947386 8 ==== osd_op_reply(76 500.00000000 [delete] v121'400 uv400 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f81b800f070 con 0x5588fa91edd0 2024-08-03T20:59:53.096 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:53.096+0000 7f81c5fedf40 1 -- 172.21.15.32:0/2475970920 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:77 12.15 12:aa448500:::500.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121) -- 0x5588fa921ba0 con 0x5588fa91edd0 2024-08-03T20:59:53.098 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::500.00000000:head# 2024-08-03T20:59:53.098 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:53.097+0000 7f81bfc00640 1 -- 172.21.15.32:0/2475970920 <== osd.6 v2:172.21.15.118:6825/801947386 9 ==== osd_op_reply(77 500.00000000 [create] v121'401 uv401 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f81b800f070 con 0x5588fa91edd0 2024-08-03T20:59:53.098 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:53.097+0000 7f81c5fedf40 1 -- 172.21.15.32:0/2475970920 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:78 12.15 12:aa448500:::500.00000000:head [write 0~90 in=90b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121) -- 0x5588fa921ba0 con 0x5588fa91edd0 2024-08-03T20:59:53.101 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:53.099+0000 7f81bfc00640 1 -- 172.21.15.32:0/2475970920 <== osd.6 v2:172.21.15.118:6825/801947386 10 ==== osd_op_reply(78 500.00000000 [write 0~90] v121'402 uv402 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f81b800f070 con 0x5588fa91edd0 2024-08-03T20:59:53.102 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:53.099+0000 7f81c5fedf40 1 -- 172.21.15.32:0/2475970920 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:79 12.15 12:aa448500:::500.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121) -- 0x5588fa921ba0 con 0x5588fa91edd0 2024-08-03T20:59:53.102 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:53.101+0000 7f81bfc00640 1 -- 172.21.15.32:0/2475970920 <== osd.6 v2:172.21.15.118:6825/801947386 11 ==== osd_op_reply(79 500.00000000 [omap-set-header] v121'403 uv403 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f81b800f070 con 0x5588fa91edd0 2024-08-03T20:59:53.102 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:53.101+0000 7f81c5fedf40 1 --2- 172.21.15.32:0/2475970920 >> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] conn(0x5588fa9220b0 0x5588fa9244f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:59:53.102 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:53.101+0000 7f81c5fedf40 1 -- 172.21.15.32:0/2475970920 --> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] -- osd_op(unknown.0.0:80 12.d 12:b50e409b:::mds_snaptable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121) -- 0x5588fa924bf0 con 0x5588fa9220b0 2024-08-03T20:59:53.102 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:53.101+0000 7f81bf200640 1 --2- 172.21.15.32:0/2475970920 >> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] conn(0x5588fa9220b0 0x5588fa9244f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:59:53.102 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:53.102+0000 7f81bf200640 1 --2- 172.21.15.32:0/2475970920 >> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] conn(0x5588fa9220b0 0x5588fa9244f0 crc :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:59:53.104 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:53.103+0000 7f81bf200640 1 -- 172.21.15.32:0/2475970920 <== osd.7 v2:172.21.15.118:6818/653000048 1 ==== osd_op_reply(80 mds_snaptable [create] v121'178 uv177 ondisk = -17 ((17) File exists)) ==== 157+0+0 (crc 0 0 0) 0x7f81b0012950 con 0x5588fa9220b0 2024-08-03T20:59:53.104 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:53.103+0000 7f81c5fedf40 1 -- 172.21.15.32:0/2475970920 --> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] -- osd_op(unknown.0.0:81 12.d 12:b50e409b:::mds_snaptable:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e121) -- 0x5588fa924fc0 con 0x5588fa9220b0 2024-08-03T20:59:53.105 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:53.105+0000 7f81bf200640 1 -- 172.21.15.32:0/2475970920 <== osd.7 v2:172.21.15.118:6818/653000048 2 ==== osd_op_reply(81 mds_snaptable [delete] v121'179 uv179 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f81b0012950 con 0x5588fa9220b0 2024-08-03T20:59:53.105 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:53.105+0000 7f81c5fedf40 1 -- 172.21.15.32:0/2475970920 --> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] -- osd_op(unknown.0.0:82 12.d 12:b50e409b:::mds_snaptable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121) -- 0x5588fa924fc0 con 0x5588fa9220b0 2024-08-03T20:59:53.107 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds_snaptable:head# 2024-08-03T20:59:53.107 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:53.106+0000 7f81bf200640 1 -- 172.21.15.32:0/2475970920 <== osd.7 v2:172.21.15.118:6818/653000048 3 ==== osd_op_reply(82 mds_snaptable [create] v121'180 uv180 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f81b0012950 con 0x5588fa9220b0 2024-08-03T20:59:53.107 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:53.106+0000 7f81c5fedf40 1 -- 172.21.15.32:0/2475970920 --> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] -- osd_op(unknown.0.0:83 12.d 12:b50e409b:::mds_snaptable:head [write 0~70 in=70b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121) -- 0x5588fa924fc0 con 0x5588fa9220b0 2024-08-03T20:59:53.108 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:53.108+0000 7f81bf200640 1 -- 172.21.15.32:0/2475970920 <== osd.7 v2:172.21.15.118:6818/653000048 4 ==== osd_op_reply(83 mds_snaptable [write 0~70] v121'181 uv181 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f81b0012950 con 0x5588fa9220b0 2024-08-03T20:59:53.108 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:53.108+0000 7f81c5fedf40 1 -- 172.21.15.32:0/2475970920 --> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] -- osd_op(unknown.0.0:84 12.d 12:b50e409b:::mds_snaptable:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121) -- 0x5588fa924fc0 con 0x5588fa9220b0 2024-08-03T20:59:53.110 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:53.109+0000 7f81bf200640 1 -- 172.21.15.32:0/2475970920 <== osd.7 v2:172.21.15.118:6818/653000048 5 ==== osd_op_reply(84 mds_snaptable [omap-set-header] v121'182 uv182 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f81b0012950 con 0x5588fa9220b0 2024-08-03T20:59:53.110 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:53.109+0000 7f81c5fedf40 1 -- 172.21.15.32:0/2475970920 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:85 12.1d 12:b871830b:::605.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121) -- 0x5588fa924fc0 con 0x5588fa871b20 2024-08-03T20:59:53.111 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:53.111+0000 7f81bfc00640 1 -- 172.21.15.32:0/2475970920 <== osd.0 v2:172.21.15.32:6816/3066156116 44 ==== osd_op_reply(85 605.00000000 [create] v121'213 uv212 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f81b800f070 con 0x5588fa871b20 2024-08-03T20:59:53.111 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:53.111+0000 7f81c5fedf40 1 -- 172.21.15.32:0/2475970920 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:86 12.1d 12:b871830b:::605.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e121) -- 0x5588fa924fc0 con 0x5588fa871b20 2024-08-03T20:59:53.113 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:53.112+0000 7f81bfc00640 1 -- 172.21.15.32:0/2475970920 <== osd.0 v2:172.21.15.32:6816/3066156116 45 ==== osd_op_reply(86 605.00000000 [delete] v121'214 uv214 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f81b800f070 con 0x5588fa871b20 2024-08-03T20:59:53.113 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:53.112+0000 7f81c5fedf40 1 -- 172.21.15.32:0/2475970920 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:87 12.1d 12:b871830b:::605.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121) -- 0x5588fa924fc0 con 0x5588fa871b20 2024-08-03T20:59:53.114 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::605.00000000:head# 2024-08-03T20:59:53.114 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:53.114+0000 7f81bfc00640 1 -- 172.21.15.32:0/2475970920 <== osd.0 v2:172.21.15.32:6816/3066156116 46 ==== osd_op_reply(87 605.00000000 [create] v121'215 uv215 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f81b800f070 con 0x5588fa871b20 2024-08-03T20:59:53.114 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:53.114+0000 7f81c5fedf40 1 -- 172.21.15.32:0/2475970920 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:88 12.1d 12:b871830b:::605.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121) -- 0x5588fa925fd0 con 0x5588fa871b20 2024-08-03T20:59:53.115 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:53.115+0000 7f81bfc00640 1 -- 172.21.15.32:0/2475970920 <== osd.0 v2:172.21.15.32:6816/3066156116 47 ==== osd_op_reply(88 605.00000000 [setxattr (30)] v121'216 uv216 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f81b800f070 con 0x5588fa871b20 2024-08-03T20:59:53.115 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:53.115+0000 7f81c5fedf40 1 -- 172.21.15.32:0/2475970920 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:89 12.1d 12:b871830b:::605.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121) -- 0x5588fa925fd0 con 0x5588fa871b20 2024-08-03T20:59:53.117 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:53.116+0000 7f81bfc00640 1 -- 172.21.15.32:0/2475970920 <== osd.0 v2:172.21.15.32:6816/3066156116 48 ==== osd_op_reply(89 605.00000000 [setxattr (62)] v121'217 uv217 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f81b800f070 con 0x5588fa871b20 2024-08-03T20:59:53.117 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:53.116+0000 7f81c5fedf40 1 -- 172.21.15.32:0/2475970920 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:90 12.1d 12:b871830b:::605.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121) -- 0x5588fa924fc0 con 0x5588fa871b20 2024-08-03T20:59:53.118 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:53.118+0000 7f81bfc00640 1 -- 172.21.15.32:0/2475970920 <== osd.0 v2:172.21.15.32:6816/3066156116 49 ==== osd_op_reply(90 605.00000000 [omap-set-header] v121'218 uv218 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f81b800f070 con 0x5588fa871b20 2024-08-03T20:59:53.118 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:53.118+0000 7f81c5fedf40 1 -- 172.21.15.32:0/2475970920 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:91 12.13 12:c91d4a1d:::mds0_inotable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121) -- 0x5588fa924fc0 con 0x5588fa91a1b0 2024-08-03T20:59:53.119 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:53.119+0000 7f81be800640 1 -- 172.21.15.32:0/2475970920 <== osd.1 v2:172.21.15.32:6824/1551241759 5 ==== osd_op_reply(91 mds0_inotable [create] v121'355 uv349 ondisk = -17 ((17) File exists)) ==== 157+0+0 (crc 0 0 0) 0x7f81ac0dc3c0 con 0x5588fa91a1b0 2024-08-03T20:59:53.120 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:53.119+0000 7f81c5fedf40 1 -- 172.21.15.32:0/2475970920 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:92 12.13 12:c91d4a1d:::mds0_inotable:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e121) -- 0x5588fa925380 con 0x5588fa91a1b0 2024-08-03T20:59:53.121 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:53.121+0000 7f81be800640 1 -- 172.21.15.32:0/2475970920 <== osd.1 v2:172.21.15.32:6824/1551241759 6 ==== osd_op_reply(92 mds0_inotable [delete] v121'356 uv356 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f81ac0dc3c0 con 0x5588fa91a1b0 2024-08-03T20:59:53.121 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:53.121+0000 7f81c5fedf40 1 -- 172.21.15.32:0/2475970920 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:93 12.13 12:c91d4a1d:::mds0_inotable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121) -- 0x5588fa925770 con 0x5588fa91a1b0 2024-08-03T20:59:53.122 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds0_inotable:head# 2024-08-03T20:59:53.122 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:53.122+0000 7f81be800640 1 -- 172.21.15.32:0/2475970920 <== osd.1 v2:172.21.15.32:6824/1551241759 7 ==== osd_op_reply(93 mds0_inotable [create] v121'357 uv357 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f81ac0dc3c0 con 0x5588fa91a1b0 2024-08-03T20:59:53.123 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:53.122+0000 7f81c5fedf40 1 -- 172.21.15.32:0/2475970920 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:94 12.13 12:c91d4a1d:::mds0_inotable:head [write 0~34 in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121) -- 0x5588fa925b60 con 0x5588fa91a1b0 2024-08-03T20:59:53.124 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:53.123+0000 7f81be800640 1 -- 172.21.15.32:0/2475970920 <== osd.1 v2:172.21.15.32:6824/1551241759 8 ==== osd_op_reply(94 mds0_inotable [write 0~34] v121'358 uv358 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f81ac0dc3c0 con 0x5588fa91a1b0 2024-08-03T20:59:53.124 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:53.123+0000 7f81c5fedf40 1 -- 172.21.15.32:0/2475970920 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:95 12.13 12:c91d4a1d:::mds0_inotable:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121) -- 0x5588fa925b60 con 0x5588fa91a1b0 2024-08-03T20:59:53.125 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:53.125+0000 7f81be800640 1 -- 172.21.15.32:0/2475970920 <== osd.1 v2:172.21.15.32:6824/1551241759 9 ==== osd_op_reply(95 mds0_inotable [omap-set-header] v121'359 uv359 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f81ac0dc3c0 con 0x5588fa91a1b0 2024-08-03T20:59:53.125 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:53.125+0000 7f81c5fedf40 1 -- 172.21.15.32:0/2475970920 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:96 12.13 12:cd5a64a3:::100.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121) -- 0x5588fa925f70 con 0x5588fa91a1b0 2024-08-03T20:59:53.126 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:53.126+0000 7f81be800640 1 -- 172.21.15.32:0/2475970920 <== osd.1 v2:172.21.15.32:6824/1551241759 10 ==== osd_op_reply(96 100.00000000 [create] v121'360 uv354 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f81ac0dc3c0 con 0x5588fa91a1b0 2024-08-03T20:59:53.126 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:53.126+0000 7f81c5fedf40 1 -- 172.21.15.32:0/2475970920 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:97 12.13 12:cd5a64a3:::100.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e121) -- 0x5588fa925810 con 0x5588fa91a1b0 2024-08-03T20:59:53.128 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:53.127+0000 7f81be800640 1 -- 172.21.15.32:0/2475970920 <== osd.1 v2:172.21.15.32:6824/1551241759 11 ==== osd_op_reply(97 100.00000000 [delete] v121'361 uv361 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f81ac0dc3c0 con 0x5588fa91a1b0 2024-08-03T20:59:53.128 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:53.127+0000 7f81c5fedf40 1 -- 172.21.15.32:0/2475970920 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:98 12.13 12:cd5a64a3:::100.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121) -- 0x5588fa925810 con 0x5588fa91a1b0 2024-08-03T20:59:53.129 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::100.00000000:head# 2024-08-03T20:59:53.129 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:53.129+0000 7f81be800640 1 -- 172.21.15.32:0/2475970920 <== osd.1 v2:172.21.15.32:6824/1551241759 12 ==== osd_op_reply(98 100.00000000 [create] v121'362 uv362 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f81ac0dc3c0 con 0x5588fa91a1b0 2024-08-03T20:59:53.129 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:53.129+0000 7f81c5fedf40 1 -- 172.21.15.32:0/2475970920 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:99 12.13 12:cd5a64a3:::100.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121) -- 0x5588fa925810 con 0x5588fa91a1b0 2024-08-03T20:59:53.130 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:53.130+0000 7f81be800640 1 -- 172.21.15.32:0/2475970920 <== osd.1 v2:172.21.15.32:6824/1551241759 13 ==== osd_op_reply(99 100.00000000 [omap-set-header] v121'363 uv363 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f81ac0dc3c0 con 0x5588fa91a1b0 2024-08-03T20:59:53.130 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:53.130+0000 7f81c5fedf40 1 -- 172.21.15.32:0/2475970920 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:100 12.13 12:cd5a64a3:::100.00000000:head [omap-set-vals in=5194b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121) -- 0x5588fa925cd0 con 0x5588fa91a1b0 2024-08-03T20:59:53.132 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:53.131+0000 7f81be800640 1 -- 172.21.15.32:0/2475970920 <== osd.1 v2:172.21.15.32:6824/1551241759 14 ==== osd_op_reply(100 100.00000000 [omap-set-vals] v121'364 uv364 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f81ac0dc3c0 con 0x5588fa91a1b0 2024-08-03T20:59:53.132 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:53.132+0000 7f81c5fedf40 1 --2- 172.21.15.32:0/2475970920 >> [v2:172.21.15.32:6800/9088303,v1:172.21.15.32:6801/9088303] conn(0x5588fa9263b0 0x5588fa9287d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:59:53.132 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:53.132+0000 7f81c5fedf40 1 -- 172.21.15.32:0/2475970920 --> [v2:172.21.15.32:6800/9088303,v1:172.21.15.32:6801/9088303] -- osd_op(unknown.0.0:101 12.b 12:d0630e4c:::mds0_sessionmap:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121) -- 0x5588fa928d10 con 0x5588fa9263b0 2024-08-03T20:59:53.132 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:53.132+0000 7f81be800640 1 --2- 172.21.15.32:0/2475970920 >> [v2:172.21.15.32:6800/9088303,v1:172.21.15.32:6801/9088303] conn(0x5588fa9263b0 0x5588fa9287d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:59:53.133 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:53.132+0000 7f81be800640 1 --2- 172.21.15.32:0/2475970920 >> [v2:172.21.15.32:6800/9088303,v1:172.21.15.32:6801/9088303] conn(0x5588fa9263b0 0x5588fa9287d0 crc :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:59:53.134 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:53.134+0000 7f81be800640 1 -- 172.21.15.32:0/2475970920 <== osd.3 v2:172.21.15.32:6800/9088303 1 ==== osd_op_reply(101 mds0_sessionmap [create] v121'157 uv155 ondisk = -17 ((17) File exists)) ==== 159+0+0 (crc 0 0 0) 0x7f81ac0dc3c0 con 0x5588fa9263b0 2024-08-03T20:59:53.134 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:53.134+0000 7f81c5fedf40 1 -- 172.21.15.32:0/2475970920 --> [v2:172.21.15.32:6800/9088303,v1:172.21.15.32:6801/9088303] -- osd_op(unknown.0.0:102 12.b 12:d0630e4c:::mds0_sessionmap:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e121) -- 0x5588fa928d10 con 0x5588fa9263b0 2024-08-03T20:59:53.135 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:53.135+0000 7f81be800640 1 -- 172.21.15.32:0/2475970920 <== osd.3 v2:172.21.15.32:6800/9088303 2 ==== osd_op_reply(102 mds0_sessionmap [delete] v121'158 uv158 ondisk = 0) ==== 159+0+0 (crc 0 0 0) 0x7f81ac0dc3c0 con 0x5588fa9263b0 2024-08-03T20:59:53.135 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:53.135+0000 7f81c5fedf40 1 -- 172.21.15.32:0/2475970920 --> [v2:172.21.15.32:6800/9088303,v1:172.21.15.32:6801/9088303] -- osd_op(unknown.0.0:103 12.b 12:d0630e4c:::mds0_sessionmap:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121) -- 0x5588fa928d10 con 0x5588fa9263b0 2024-08-03T20:59:53.136 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:53.136+0000 7f81be800640 1 -- 172.21.15.32:0/2475970920 <== osd.3 v2:172.21.15.32:6800/9088303 3 ==== osd_op_reply(103 mds0_sessionmap [create] v121'159 uv159 ondisk = 0) ==== 159+0+0 (crc 0 0 0) 0x7f81ac0dc3c0 con 0x5588fa9263b0 2024-08-03T20:59:53.137 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds0_sessionmap:head# 2024-08-03T20:59:53.137 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:53.136+0000 7f81c5fedf40 1 -- 172.21.15.32:0/2475970920 --> [v2:172.21.15.32:6800/9088303,v1:172.21.15.32:6801/9088303] -- osd_op(unknown.0.0:104 12.b 12:d0630e4c:::mds0_sessionmap:head [omap-set-header in=14b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121) -- 0x5588fa928d10 con 0x5588fa9263b0 2024-08-03T20:59:53.138 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:53.138+0000 7f81be800640 1 -- 172.21.15.32:0/2475970920 <== osd.3 v2:172.21.15.32:6800/9088303 4 ==== osd_op_reply(104 mds0_sessionmap [omap-set-header] v121'160 uv160 ondisk = 0) ==== 159+0+0 (crc 0 0 0) 0x7f81ac0dc3c0 con 0x5588fa9263b0 2024-08-03T20:59:53.138 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:53.138+0000 7f81c5fedf40 1 -- 172.21.15.32:0/2475970920 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:105 12.7 12:e0b41d06:::609.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121) -- 0x5588fa928d10 con 0x5588fa91a1b0 2024-08-03T20:59:53.140 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:53.139+0000 7f81be800640 1 -- 172.21.15.32:0/2475970920 <== osd.1 v2:172.21.15.32:6824/1551241759 15 ==== osd_op_reply(105 609.00000000 [create] v121'213 uv212 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f81ac0dc3c0 con 0x5588fa91a1b0 2024-08-03T20:59:53.140 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:53.139+0000 7f81c5fedf40 1 -- 172.21.15.32:0/2475970920 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:106 12.7 12:e0b41d06:::609.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e121) -- 0x5588fa928d10 con 0x5588fa91a1b0 2024-08-03T20:59:53.141 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:53.141+0000 7f81be800640 1 -- 172.21.15.32:0/2475970920 <== osd.1 v2:172.21.15.32:6824/1551241759 16 ==== osd_op_reply(106 609.00000000 [delete] v121'214 uv214 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f81ac0dc3c0 con 0x5588fa91a1b0 2024-08-03T20:59:53.141 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:53.141+0000 7f81c5fedf40 1 -- 172.21.15.32:0/2475970920 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:107 12.7 12:e0b41d06:::609.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121) -- 0x5588fa928d10 con 0x5588fa91a1b0 2024-08-03T20:59:53.142 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:53.142+0000 7f81be800640 1 -- 172.21.15.32:0/2475970920 <== osd.1 v2:172.21.15.32:6824/1551241759 17 ==== osd_op_reply(107 609.00000000 [create] v121'215 uv215 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f81ac0dc3c0 con 0x5588fa91a1b0 2024-08-03T20:59:53.143 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::609.00000000:head# 2024-08-03T20:59:53.143 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:53.142+0000 7f81c5fedf40 1 -- 172.21.15.32:0/2475970920 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:108 12.7 12:e0b41d06:::609.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121) -- 0x5588fa92a110 con 0x5588fa91a1b0 2024-08-03T20:59:53.144 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:53.143+0000 7f81be800640 1 -- 172.21.15.32:0/2475970920 <== osd.1 v2:172.21.15.32:6824/1551241759 18 ==== osd_op_reply(108 609.00000000 [setxattr (30)] v121'216 uv216 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f81ac0dc3c0 con 0x5588fa91a1b0 2024-08-03T20:59:53.144 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:53.143+0000 7f81c5fedf40 1 -- 172.21.15.32:0/2475970920 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:109 12.7 12:e0b41d06:::609.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121) -- 0x5588fa92a110 con 0x5588fa91a1b0 2024-08-03T20:59:53.145 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:53.145+0000 7f81be800640 1 -- 172.21.15.32:0/2475970920 <== osd.1 v2:172.21.15.32:6824/1551241759 19 ==== osd_op_reply(109 609.00000000 [setxattr (62)] v121'217 uv217 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f81ac0dc3c0 con 0x5588fa91a1b0 2024-08-03T20:59:53.145 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:53.145+0000 7f81c5fedf40 1 -- 172.21.15.32:0/2475970920 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:110 12.7 12:e0b41d06:::609.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121) -- 0x5588fa928d50 con 0x5588fa91a1b0 2024-08-03T20:59:53.147 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:53.146+0000 7f81be800640 1 -- 172.21.15.32:0/2475970920 <== osd.1 v2:172.21.15.32:6824/1551241759 20 ==== osd_op_reply(110 609.00000000 [omap-set-header] v121'218 uv218 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f81ac0dc3c0 con 0x5588fa91a1b0 2024-08-03T20:59:53.147 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:53.146+0000 7f81c5fedf40 1 -- 172.21.15.32:0/2475970920 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:111 12.f 12:f1f69726:::400.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121) -- 0x5588fa928d50 con 0x5588fa91edd0 2024-08-03T20:59:53.148 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:53.148+0000 7f81bfc00640 1 -- 172.21.15.32:0/2475970920 <== osd.6 v2:172.21.15.118:6825/801947386 12 ==== osd_op_reply(111 400.00000000 [create] v121'181 uv180 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f81b800f070 con 0x5588fa91edd0 2024-08-03T20:59:53.148 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:53.148+0000 7f81c5fedf40 1 -- 172.21.15.32:0/2475970920 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:112 12.f 12:f1f69726:::400.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e121) -- 0x5588fa928d50 con 0x5588fa91edd0 2024-08-03T20:59:53.150 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:53.149+0000 7f81bfc00640 1 -- 172.21.15.32:0/2475970920 <== osd.6 v2:172.21.15.118:6825/801947386 13 ==== osd_op_reply(112 400.00000000 [delete] v121'182 uv182 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f81b800f070 con 0x5588fa91edd0 2024-08-03T20:59:53.150 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:53.149+0000 7f81c5fedf40 1 -- 172.21.15.32:0/2475970920 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:113 12.f 12:f1f69726:::400.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121) -- 0x5588fa928d50 con 0x5588fa91edd0 2024-08-03T20:59:53.151 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:53.151+0000 7f81bfc00640 1 -- 172.21.15.32:0/2475970920 <== osd.6 v2:172.21.15.118:6825/801947386 14 ==== osd_op_reply(113 400.00000000 [create] v121'183 uv183 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f81b800f070 con 0x5588fa91edd0 2024-08-03T20:59:53.152 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::400.00000000:head# 2024-08-03T20:59:53.152 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:53.151+0000 7f81c5fedf40 1 -- 172.21.15.32:0/2475970920 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:114 12.f 12:f1f69726:::400.00000000:head [write 0~22 in=22b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121) -- 0x5588fa91e070 con 0x5588fa91edd0 2024-08-03T20:59:53.153 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:53.153+0000 7f81bfc00640 1 -- 172.21.15.32:0/2475970920 <== osd.6 v2:172.21.15.118:6825/801947386 15 ==== osd_op_reply(114 400.00000000 [write 0~22] v121'184 uv184 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f81b800f070 con 0x5588fa91edd0 2024-08-03T20:59:53.153 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:53.153+0000 7f81c5fedf40 1 -- 172.21.15.32:0/2475970920 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:115 12.f 12:f1f69726:::400.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121) -- 0x5588fa91e070 con 0x5588fa91edd0 2024-08-03T20:59:53.155 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:53.154+0000 7f81bfc00640 1 -- 172.21.15.32:0/2475970920 <== osd.6 v2:172.21.15.118:6825/801947386 16 ==== osd_op_reply(115 400.00000000 [omap-set-header] v121'185 uv185 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f81b800f070 con 0x5588fa91edd0 2024-08-03T20:59:53.155 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:53.154+0000 7f81c5fedf40 1 -- 172.21.15.32:0/2475970920 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:116 12.1f 12:fe07bba1:::100.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121) -- 0x5588fa91e070 con 0x5588fa91edd0 2024-08-03T20:59:53.156 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:53.156+0000 7f81bfc00640 1 -- 172.21.15.32:0/2475970920 <== osd.6 v2:172.21.15.118:6825/801947386 17 ==== osd_op_reply(116 100.00000000.inode [create] v121'442 uv433 ondisk = -17 ((17) File exists)) ==== 162+0+0 (crc 0 0 0) 0x7f81b800f070 con 0x5588fa91edd0 2024-08-03T20:59:53.156 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:53.156+0000 7f81c5fedf40 1 -- 172.21.15.32:0/2475970920 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:117 12.1f 12:fe07bba1:::100.00000000.inode:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e121) -- 0x5588fa91e070 con 0x5588fa91edd0 2024-08-03T20:59:53.158 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:53.157+0000 7f81bfc00640 1 -- 172.21.15.32:0/2475970920 <== osd.6 v2:172.21.15.118:6825/801947386 18 ==== osd_op_reply(117 100.00000000.inode [delete] v121'443 uv443 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7f81b800f070 con 0x5588fa91edd0 2024-08-03T20:59:53.158 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:53.157+0000 7f81c5fedf40 1 -- 172.21.15.32:0/2475970920 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:118 12.1f 12:fe07bba1:::100.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121) -- 0x5588fa91e070 con 0x5588fa91edd0 2024-08-03T20:59:53.159 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::100.00000000.inode:head# 2024-08-03T20:59:53.160 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:53.159+0000 7f81bfc00640 1 -- 172.21.15.32:0/2475970920 <== osd.6 v2:172.21.15.118:6825/801947386 19 ==== osd_op_reply(118 100.00000000.inode [create] v121'444 uv444 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7f81b800f070 con 0x5588fa91edd0 2024-08-03T20:59:53.160 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:53.159+0000 7f81c5fedf40 1 -- 172.21.15.32:0/2475970920 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:119 12.1f 12:fe07bba1:::100.00000000.inode:head [write 0~582 in=582b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121) -- 0x5588fa91e070 con 0x5588fa91edd0 2024-08-03T20:59:53.161 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:53.160+0000 7f81bfc00640 1 -- 172.21.15.32:0/2475970920 <== osd.6 v2:172.21.15.118:6825/801947386 20 ==== osd_op_reply(119 100.00000000.inode [write 0~582] v121'445 uv445 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7f81b800f070 con 0x5588fa91edd0 2024-08-03T20:59:53.161 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:53.160+0000 7f81c5fedf40 1 -- 172.21.15.32:0/2475970920 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:120 12.1f 12:fe07bba1:::100.00000000.inode:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121) -- 0x5588fa91e070 con 0x5588fa91edd0 2024-08-03T20:59:53.162 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:53.162+0000 7f81bfc00640 1 -- 172.21.15.32:0/2475970920 <== osd.6 v2:172.21.15.118:6825/801947386 21 ==== osd_op_reply(120 100.00000000.inode [omap-set-header] v121'446 uv446 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7f81b800f070 con 0x5588fa91edd0 2024-08-03T20:59:53.162 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:53.162+0000 7f81c5fedf40 1 -- 172.21.15.32:0/2475970920 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:121 12.1f 12:ff5b34d6:::1.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121) -- 0x5588fa91e070 con 0x5588fa91edd0 2024-08-03T20:59:53.163 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:53.163+0000 7f81bfc00640 1 -- 172.21.15.32:0/2475970920 <== osd.6 v2:172.21.15.118:6825/801947386 22 ==== osd_op_reply(121 1.00000000 [create] v121'447 uv440 ondisk = -17 ((17) File exists)) ==== 154+0+0 (crc 0 0 0) 0x7f81b800f070 con 0x5588fa91edd0 2024-08-03T20:59:53.164 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:53.163+0000 7f81c5fedf40 1 -- 172.21.15.32:0/2475970920 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:122 12.1f 12:ff5b34d6:::1.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e121) -- 0x5588fa91e070 con 0x5588fa91edd0 2024-08-03T20:59:53.165 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:53.164+0000 7f81bfc00640 1 -- 172.21.15.32:0/2475970920 <== osd.6 v2:172.21.15.118:6825/801947386 23 ==== osd_op_reply(122 1.00000000 [delete] v121'448 uv448 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7f81b800f070 con 0x5588fa91edd0 2024-08-03T20:59:53.165 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:53.164+0000 7f81c5fedf40 1 -- 172.21.15.32:0/2475970920 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:123 12.1f 12:ff5b34d6:::1.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121) -- 0x5588fa91e070 con 0x5588fa91edd0 2024-08-03T20:59:53.166 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:53.166+0000 7f81bfc00640 1 -- 172.21.15.32:0/2475970920 <== osd.6 v2:172.21.15.118:6825/801947386 24 ==== osd_op_reply(123 1.00000000 [create] v121'449 uv449 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7f81b800f070 con 0x5588fa91edd0 2024-08-03T20:59:53.167 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::1.00000000:head# 2024-08-03T20:59:53.167 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:53.166+0000 7f81c5fedf40 1 -- 172.21.15.32:0/2475970920 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:124 12.1f 12:ff5b34d6:::1.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121) -- 0x5588fa91e070 con 0x5588fa91edd0 2024-08-03T20:59:53.167 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:53.167+0000 7f81bfc00640 1 -- 172.21.15.32:0/2475970920 <== osd.6 v2:172.21.15.118:6825/801947386 25 ==== osd_op_reply(124 1.00000000 [setxattr (30)] v121'450 uv450 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7f81b800f070 con 0x5588fa91edd0 2024-08-03T20:59:53.168 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:53.167+0000 7f81c5fedf40 1 -- 172.21.15.32:0/2475970920 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:125 12.1f 12:ff5b34d6:::1.00000000:head [setxattr parent (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121) -- 0x5588fa91e070 con 0x5588fa91edd0 2024-08-03T20:59:53.169 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:53.168+0000 7f81bfc00640 1 -- 172.21.15.32:0/2475970920 <== osd.6 v2:172.21.15.118:6825/801947386 26 ==== osd_op_reply(125 1.00000000 [setxattr (30)] v121'451 uv451 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7f81b800f070 con 0x5588fa91edd0 2024-08-03T20:59:53.169 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:53.168+0000 7f81c5fedf40 1 -- 172.21.15.32:0/2475970920 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:126 12.1f 12:ff5b34d6:::1.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121) -- 0x5588fa91e070 con 0x5588fa91edd0 2024-08-03T20:59:53.170 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:53.170+0000 7f81bfc00640 1 -- 172.21.15.32:0/2475970920 <== osd.6 v2:172.21.15.118:6825/801947386 27 ==== osd_op_reply(126 1.00000000 [omap-set-header] v121'452 uv452 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7f81b800f070 con 0x5588fa91edd0 2024-08-03T20:59:53.170 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:53.170+0000 7f81c5fedf40 1 -- 172.21.15.32:0/2475970920 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:127 12.1f 12:ff5b34d6:::1.00000000:head [omap-set-vals in=584b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121) -- 0x5588fa9296e0 con 0x5588fa91edd0 2024-08-03T20:59:53.173 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:53.171+0000 7f81bfc00640 1 -- 172.21.15.32:0/2475970920 <== osd.6 v2:172.21.15.118:6825/801947386 28 ==== osd_op_reply(127 1.00000000 [omap-set-vals] v121'453 uv453 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7f81b800f070 con 0x5588fa91edd0 2024-08-03T20:59:53.173 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:53.171+0000 7f81c5fedf40 1 -- 172.21.15.32:0/2475970920 >> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] conn(0x5588fa871b20 msgr2=0x5588fa913700 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:59:53.174 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:53.171+0000 7f81c5fedf40 1 --2- 172.21.15.32:0/2475970920 >> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] conn(0x5588fa871b20 0x5588fa913700 crc :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:59:53.174 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:53.171+0000 7f81c5fedf40 1 -- 172.21.15.32:0/2475970920 >> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] conn(0x5588fa91a1b0 msgr2=0x5588fa91d020 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:59:53.174 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:53.171+0000 7f81c5fedf40 1 --2- 172.21.15.32:0/2475970920 >> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] conn(0x5588fa91a1b0 0x5588fa91d020 crc :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:59:53.174 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:53.171+0000 7f81c5fedf40 1 -- 172.21.15.32:0/2475970920 >> [v2:172.21.15.32:6800/9088303,v1:172.21.15.32:6801/9088303] conn(0x5588fa9263b0 msgr2=0x5588fa9287d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:59:53.174 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:53.171+0000 7f81c5fedf40 1 --2- 172.21.15.32:0/2475970920 >> [v2:172.21.15.32:6800/9088303,v1:172.21.15.32:6801/9088303] conn(0x5588fa9263b0 0x5588fa9287d0 crc :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:59:53.174 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:53.171+0000 7f81c5fedf40 1 -- 172.21.15.32:0/2475970920 >> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] conn(0x5588fa915ad0 msgr2=0x5588fa917ef0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:59:53.174 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:53.171+0000 7f81c5fedf40 1 --2- 172.21.15.32:0/2475970920 >> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] conn(0x5588fa915ad0 0x5588fa917ef0 crc :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:59:53.174 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:53.171+0000 7f81c5fedf40 1 -- 172.21.15.32:0/2475970920 >> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] conn(0x5588fa91edd0 msgr2=0x5588fa9211f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:59:53.175 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:53.171+0000 7f81c5fedf40 1 --2- 172.21.15.32:0/2475970920 >> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] conn(0x5588fa91edd0 0x5588fa9211f0 crc :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:59:53.175 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:53.171+0000 7f81c5fedf40 1 -- 172.21.15.32:0/2475970920 >> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] conn(0x5588fa9220b0 msgr2=0x5588fa9244f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:59:53.175 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:53.171+0000 7f81c5fedf40 1 --2- 172.21.15.32:0/2475970920 >> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] conn(0x5588fa9220b0 0x5588fa9244f0 crc :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:59:53.175 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:53.171+0000 7f81bfc00640 1 -- 172.21.15.32:0/2475970920 reap_dead start 2024-08-03T20:59:53.175 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:53.171+0000 7f81bfc00640 1 -- 172.21.15.32:0/2475970920 reap_dead start 2024-08-03T20:59:53.175 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:53.171+0000 7f81bfc00640 1 -- 172.21.15.32:0/2475970920 reap_dead start 2024-08-03T20:59:53.175 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:53.171+0000 7f81c5fedf40 1 -- 172.21.15.32:0/2475970920 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f81a0075a20 msgr2=0x7f81a0077f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:59:53.175 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:53.171+0000 7f81c5fedf40 1 --2- 172.21.15.32:0/2475970920 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f81a0075a20 0x7f81a0077f30 secure :-1 s=READY pgs=798 cs=0 l=1 rev1=1 crypto rx=0x7f81b0000f60 tx=0x7f81b0001790 comp rx=0 tx=0).stop 2024-08-03T20:59:53.176 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:53.172+0000 7f81c5fedf40 1 -- 172.21.15.32:0/2475970920 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5588fa7e3650 msgr2=0x5588fa84d070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:59:53.176 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:53.172+0000 7f81c5fedf40 1 --2- 172.21.15.32:0/2475970920 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5588fa7e3650 0x5588fa84d070 secure :-1 s=READY pgs=1990 cs=0 l=1 rev1=1 crypto rx=0x7f81ac0465c0 tx=0x7f81ac04d880 comp rx=0 tx=0).stop 2024-08-03T20:59:53.176 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:53.172+0000 7f81c5fedf40 1 -- 172.21.15.32:0/2475970920 shutdown_connections 2024-08-03T20:59:53.176 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:53.172+0000 7f81c5fedf40 1 --2- 172.21.15.32:0/2475970920 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f81a0075a20 0x7f81a0077f30 unknown :-1 s=CLOSED pgs=798 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:59:53.176 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:53.172+0000 7f81c5fedf40 1 --2- 172.21.15.32:0/2475970920 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5588fa7e3650 0x5588fa84d070 unknown :-1 s=CLOSED pgs=1990 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:59:53.176 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:53.172+0000 7f81c5fedf40 1 -- 172.21.15.32:0/2475970920 >> 172.21.15.32:0/2475970920 conn(0x5588fa6baac0 msgr2=0x5588fa79b700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:59:53.176 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:53.172+0000 7f81c5fedf40 1 -- 172.21.15.32:0/2475970920 shutdown_connections 2024-08-03T20:59:53.176 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:53.172+0000 7f81c5fedf40 1 -- 172.21.15.32:0/2475970920 wait complete. 2024-08-03T20:59:53.184 DEBUG:teuthology.orchestra.run.smithi032:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata setomapheader mds0_openfiles.0 deadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeef 2024-08-03T20:59:53.369 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:53.368+0000 7f2e012fff40 1 Processor -- start 2024-08-03T20:59:53.369 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:53.368+0000 7f2e012fff40 1 -- start start 2024-08-03T20:59:53.372 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:53.370+0000 7f2e012fff40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x560843cf0b40 0x560843ca98f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:59:53.372 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:53.370+0000 7f2e012fff40 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x560843d63be0 0x560843caa3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:59:53.372 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:53.370+0000 7f2e012fff40 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x560843d80510 0x560843e13b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:59:53.372 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:53.370+0000 7f2e012fff40 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x560843bf9e30 con 0x560843d63be0 2024-08-03T20:59:53.372 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:53.370+0000 7f2e012fff40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x560843c57bf0 con 0x560843cf0b40 2024-08-03T20:59:53.372 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:53.370+0000 7f2e012fff40 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x560843d64510 con 0x560843d80510 2024-08-03T20:59:53.372 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:53.370+0000 7f2df9a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x560843d63be0 0x560843caa3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:59:53.372 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:53.370+0000 7f2df9a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x560843d63be0 0x560843caa3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:54170/0 (socket says 172.21.15.32:54170) 2024-08-03T20:59:53.372 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:53.370+0000 7f2df9a00640 1 -- 172.21.15.32:0/1154189080 learned_addr learned my addr 172.21.15.32:0/1154189080 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:59:53.372 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:53.370+0000 7f2dfa400640 1 --2- 172.21.15.32:0/1154189080 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x560843cf0b40 0x560843ca98f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:59:53.372 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:53.370+0000 7f2dfae00640 1 --2- 172.21.15.32:0/1154189080 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x560843d80510 0x560843e13b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:59:53.372 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:53.371+0000 7f2df9a00640 1 -- 172.21.15.32:0/1154189080 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x560843d80510 msgr2=0x560843e13b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:59:53.372 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:53.371+0000 7f2df9a00640 1 --2- 172.21.15.32:0/1154189080 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x560843d80510 0x560843e13b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:59:53.373 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:53.371+0000 7f2df9a00640 1 -- 172.21.15.32:0/1154189080 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x560843cf0b40 msgr2=0x560843ca98f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T20:59:53.373 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:53.371+0000 7f2df9a00640 1 --2- 172.21.15.32:0/1154189080 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x560843cf0b40 0x560843ca98f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:59:53.373 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:53.371+0000 7f2df9a00640 1 -- 172.21.15.32:0/1154189080 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560843caa120 con 0x560843d63be0 2024-08-03T20:59:53.373 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:53.371+0000 7f2df9a00640 1 --2- 172.21.15.32:0/1154189080 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x560843d63be0 0x560843caa3e0 secure :-1 s=READY pgs=1992 cs=0 l=1 rev1=1 crypto rx=0x7f2de800c580 tx=0x7f2de800c980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d062d69ad066c00c server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:59:53.373 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:53.371+0000 7f2df3e00640 1 -- 172.21.15.32:0/1154189080 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2de800a9e0 con 0x560843d63be0 2024-08-03T20:59:53.373 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:53.371+0000 7f2df3e00640 1 -- 172.21.15.32:0/1154189080 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f2de800ab80 con 0x560843d63be0 2024-08-03T20:59:53.373 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:53.371+0000 7f2df3e00640 1 -- 172.21.15.32:0/1154189080 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2de8008420 con 0x560843d63be0 2024-08-03T20:59:53.374 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:53.372+0000 7f2e012fff40 1 -- 172.21.15.32:0/1154189080 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x560843d63be0 msgr2=0x560843caa3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:59:53.374 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:53.372+0000 7f2e012fff40 1 --2- 172.21.15.32:0/1154189080 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x560843d63be0 0x560843caa3e0 secure :-1 s=READY pgs=1992 cs=0 l=1 rev1=1 crypto rx=0x7f2de800c580 tx=0x7f2de800c980 comp rx=0 tx=0).stop 2024-08-03T20:59:53.374 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:53.372+0000 7f2e012fff40 1 -- 172.21.15.32:0/1154189080 shutdown_connections 2024-08-03T20:59:53.375 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:53.372+0000 7f2e012fff40 1 --2- 172.21.15.32:0/1154189080 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x560843d80510 0x560843e13b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:59:53.375 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:53.372+0000 7f2e012fff40 1 --2- 172.21.15.32:0/1154189080 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x560843d63be0 0x560843caa3e0 unknown :-1 s=CLOSED pgs=1992 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:59:53.375 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:53.372+0000 7f2e012fff40 1 --2- 172.21.15.32:0/1154189080 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x560843cf0b40 0x560843ca98f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:59:53.375 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:53.372+0000 7f2e012fff40 1 -- 172.21.15.32:0/1154189080 >> 172.21.15.32:0/1154189080 conn(0x560843bcbac0 msgr2=0x560843d61190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:59:53.375 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:53.372+0000 7f2e012fff40 1 -- 172.21.15.32:0/1154189080 shutdown_connections 2024-08-03T20:59:53.375 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:53.372+0000 7f2e012fff40 1 -- 172.21.15.32:0/1154189080 wait complete. 2024-08-03T20:59:53.375 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:53.373+0000 7f2e012fff40 1 Processor -- start 2024-08-03T20:59:53.375 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:53.373+0000 7f2e012fff40 1 -- start start 2024-08-03T20:59:53.377 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:53.375+0000 7f2e012fff40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x560843cf0b40 0x560843d7e4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:59:53.377 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:53.375+0000 7f2e012fff40 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x560843d63be0 0x560843ca93e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:59:53.377 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:53.375+0000 7f2e012fff40 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x560843d80510 0x560843ca9920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:59:53.377 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:53.375+0000 7f2e012fff40 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x560843d58e70 con 0x560843d80510 2024-08-03T20:59:53.377 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:53.375+0000 7f2e012fff40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x560843d620a0 con 0x560843cf0b40 2024-08-03T20:59:53.377 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:53.375+0000 7f2e012fff40 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x560843e1ada0 con 0x560843d63be0 2024-08-03T20:59:53.377 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:53.375+0000 7f2dfae00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x560843d80510 0x560843ca9920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:59:53.377 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:53.375+0000 7f2dfae00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x560843d80510 0x560843ca9920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:54174/0 (socket says 172.21.15.32:54174) 2024-08-03T20:59:53.378 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:53.375+0000 7f2dfae00640 1 -- 172.21.15.32:0/287241083 learned_addr learned my addr 172.21.15.32:0/287241083 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:59:53.378 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:53.375+0000 7f2df9a00640 1 --2- 172.21.15.32:0/287241083 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x560843d63be0 0x560843ca93e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:59:53.378 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:53.375+0000 7f2dfae00640 1 -- 172.21.15.32:0/287241083 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x560843d63be0 msgr2=0x560843ca93e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:59:53.378 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:53.375+0000 7f2dfa400640 1 --2- 172.21.15.32:0/287241083 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x560843cf0b40 0x560843d7e4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:59:53.378 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:53.375+0000 7f2dfae00640 1 --2- 172.21.15.32:0/287241083 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x560843d63be0 0x560843ca93e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:59:53.378 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:53.375+0000 7f2dfae00640 1 -- 172.21.15.32:0/287241083 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x560843cf0b40 msgr2=0x560843d7e4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:59:53.378 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:53.375+0000 7f2dfae00640 1 --2- 172.21.15.32:0/287241083 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x560843cf0b40 0x560843d7e4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:59:53.378 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:53.375+0000 7f2dfae00640 1 -- 172.21.15.32:0/287241083 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560843d7f5a0 con 0x560843d80510 2024-08-03T20:59:53.379 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:53.375+0000 7f2df9a00640 1 --2- 172.21.15.32:0/287241083 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x560843d63be0 0x560843ca93e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-03T20:59:53.379 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:53.375+0000 7f2dfa400640 1 --2- 172.21.15.32:0/287241083 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x560843cf0b40 0x560843d7e4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-03T20:59:53.379 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:53.376+0000 7f2dfae00640 1 --2- 172.21.15.32:0/287241083 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x560843d80510 0x560843ca9920 secure :-1 s=READY pgs=1993 cs=0 l=1 rev1=1 crypto rx=0x7f2df400d3f0 tx=0x7f2df400d8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:59:53.379 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:53.376+0000 7f2df2a00640 1 -- 172.21.15.32:0/287241083 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2df4006e30 con 0x560843d80510 2024-08-03T20:59:53.379 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:53.376+0000 7f2df2a00640 1 -- 172.21.15.32:0/287241083 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f2df4017070 con 0x560843d80510 2024-08-03T20:59:53.379 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:53.376+0000 7f2e012fff40 1 -- 172.21.15.32:0/287241083 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x560843e191f0 con 0x560843d80510 2024-08-03T20:59:53.379 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:53.376+0000 7f2df2a00640 1 -- 172.21.15.32:0/287241083 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2df40118e0 con 0x560843d80510 2024-08-03T20:59:53.379 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:53.376+0000 7f2e012fff40 1 -- 172.21.15.32:0/287241083 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560843e18c30 con 0x560843d80510 2024-08-03T20:59:53.380 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:53.378+0000 7f2df2a00640 1 -- 172.21.15.32:0/287241083 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f2df4011a80 con 0x560843d80510 2024-08-03T20:59:53.380 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:53.379+0000 7f2df2a00640 1 --2- 172.21.15.32:0/287241083 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f2ddc075a20 0x7f2ddc077f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:59:53.380 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:53.379+0000 7f2df2a00640 1 -- 172.21.15.32:0/287241083 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(121..121 src has 1..121) ==== 8950+0+0 (secure 0 0 0) 0x7f2df4097b40 con 0x560843d80510 2024-08-03T20:59:53.381 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:53.380+0000 7f2dfa400640 1 --2- 172.21.15.32:0/287241083 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f2ddc075a20 0x7f2ddc077f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:59:53.381 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:53.380+0000 7f2e012fff40 1 --2- 172.21.15.32:0/287241083 >> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] conn(0x560843e19f70 0x560843e1a380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:59:53.381 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:53.380+0000 7f2e012fff40 1 -- 172.21.15.32:0/287241083 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:1 12.6 12:654134d2:::mds0_openfiles.0:head [omap-set-header in=80b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121) -- 0x560843e18250 con 0x560843e19f70 2024-08-03T20:59:53.381 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:53.380+0000 7f2df9a00640 1 --2- 172.21.15.32:0/287241083 >> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] conn(0x560843e19f70 0x560843e1a380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:59:53.381 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:53.380+0000 7f2dfa400640 1 --2- 172.21.15.32:0/287241083 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f2ddc075a20 0x7f2ddc077f30 secure :-1 s=READY pgs=799 cs=0 l=1 rev1=1 crypto rx=0x7f2dec055000 tx=0x7f2dec003020 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:59:53.381 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:53.380+0000 7f2df9a00640 1 --2- 172.21.15.32:0/287241083 >> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] conn(0x560843e19f70 0x560843e1a380 crc :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:59:53.383 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:53.382+0000 7f2df9a00640 1 -- 172.21.15.32:0/287241083 <== osd.1 v2:172.21.15.32:6824/1551241759 1 ==== osd_op_reply(1 mds0_openfiles.0 [omap-set-header] v121'172 uv172 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f2de800aa40 con 0x560843e19f70 2024-08-03T20:59:53.383 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:53.382+0000 7f2e012fff40 1 -- 172.21.15.32:0/287241083 >> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] conn(0x560843e19f70 msgr2=0x560843e1a380 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:59:53.383 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:53.382+0000 7f2e012fff40 1 --2- 172.21.15.32:0/287241083 >> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] conn(0x560843e19f70 0x560843e1a380 crc :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:59:53.383 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:53.382+0000 7f2e012fff40 1 -- 172.21.15.32:0/287241083 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f2ddc075a20 msgr2=0x7f2ddc077f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:59:53.383 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:53.382+0000 7f2e012fff40 1 --2- 172.21.15.32:0/287241083 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f2ddc075a20 0x7f2ddc077f30 secure :-1 s=READY pgs=799 cs=0 l=1 rev1=1 crypto rx=0x7f2dec055000 tx=0x7f2dec003020 comp rx=0 tx=0).stop 2024-08-03T20:59:53.383 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:53.382+0000 7f2e012fff40 1 -- 172.21.15.32:0/287241083 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x560843d80510 msgr2=0x560843ca9920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:59:53.384 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:53.382+0000 7f2e012fff40 1 --2- 172.21.15.32:0/287241083 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x560843d80510 0x560843ca9920 secure :-1 s=READY pgs=1993 cs=0 l=1 rev1=1 crypto rx=0x7f2df400d3f0 tx=0x7f2df400d8b0 comp rx=0 tx=0).stop 2024-08-03T20:59:53.384 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:53.382+0000 7f2dfae00640 1 -- 172.21.15.32:0/287241083 reap_dead start 2024-08-03T20:59:53.384 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:53.382+0000 7f2e012fff40 1 -- 172.21.15.32:0/287241083 shutdown_connections 2024-08-03T20:59:53.384 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:53.382+0000 7f2e012fff40 1 -- 172.21.15.32:0/287241083 >> 172.21.15.32:0/287241083 conn(0x560843bcbac0 msgr2=0x560843d80a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:59:53.384 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:53.382+0000 7f2e012fff40 1 -- 172.21.15.32:0/287241083 shutdown_connections 2024-08-03T20:59:53.384 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:53.383+0000 7f2e012fff40 1 -- 172.21.15.32:0/287241083 wait complete. 2024-08-03T20:59:53.393 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs joinable true 2024-08-03T20:59:53.624 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:53.623+0000 7fd43b200640 1 -- 172.21.15.32:0/311763266 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd43c1609d0 msgr2=0x7fd43c0cc050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:59:53.624 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:53.623+0000 7fd43b200640 1 --2- 172.21.15.32:0/311763266 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd43c1609d0 0x7fd43c0cc050 secure :-1 s=READY pgs=1995 cs=0 l=1 rev1=1 crypto rx=0x7fd43c0cb2c0 tx=0x7fd42c02a5c0 comp rx=0 tx=0).stop 2024-08-03T20:59:53.624 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:53.623+0000 7fd43b200640 1 -- 172.21.15.32:0/311763266 shutdown_connections 2024-08-03T20:59:53.624 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:53.623+0000 7fd43b200640 1 --2- 172.21.15.32:0/311763266 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd43c1609d0 0x7fd43c0cc050 unknown :-1 s=CLOSED pgs=1995 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:59:53.624 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:53.623+0000 7fd43b200640 1 -- 172.21.15.32:0/311763266 >> 172.21.15.32:0/311763266 conn(0x7fd43c0cfd80 msgr2=0x7fd43c0d0190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:59:53.625 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:53.623+0000 7fd43b200640 1 -- 172.21.15.32:0/311763266 shutdown_connections 2024-08-03T20:59:53.625 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:53.623+0000 7fd43b200640 1 -- 172.21.15.32:0/311763266 wait complete. 2024-08-03T20:59:53.626 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:53.626+0000 7fd43b200640 1 Processor -- start 2024-08-03T20:59:53.626 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:53.626+0000 7fd43b200640 1 -- start start 2024-08-03T20:59:53.629 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:53.627+0000 7fd43b200640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fd43c159ff0 0x7fd43c157bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:59:53.629 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:53.627+0000 7fd43b200640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fd43c1609d0 0x7fd43c0aec40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:59:53.629 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:53.627+0000 7fd43b200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd43c0b4bc0 0x7fd43c0b4fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:59:53.629 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:53.627+0000 7fd43b200640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fd43c06f0d0 con 0x7fd43c0b4bc0 2024-08-03T20:59:53.629 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:53.627+0000 7fd43b200640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7fd43c06ef50 con 0x7fd43c159ff0 2024-08-03T20:59:53.629 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:53.627+0000 7fd43b200640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7fd43c06f250 con 0x7fd43c1609d0 2024-08-03T20:59:53.629 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:53.628+0000 7fd43a800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd43c0b4bc0 0x7fd43c0b4fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:59:53.629 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:53.628+0000 7fd43a800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd43c0b4bc0 0x7fd43c0b4fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:54198/0 (socket says 172.21.15.32:54198) 2024-08-03T20:59:53.629 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:53.628+0000 7fd43a800640 1 -- 172.21.15.32:0/1070418870 learned_addr learned my addr 172.21.15.32:0/1070418870 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:59:53.629 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:53.628+0000 7fd439e00640 1 --2- 172.21.15.32:0/1070418870 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fd43c159ff0 0x7fd43c157bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:59:53.629 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:53.628+0000 7fd433e00640 1 --2- 172.21.15.32:0/1070418870 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fd43c1609d0 0x7fd43c0aec40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:59:53.629 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:53.628+0000 7fd43a800640 1 -- 172.21.15.32:0/1070418870 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fd43c1609d0 msgr2=0x7fd43c0aec40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:59:53.629 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:53.628+0000 7fd43a800640 1 --2- 172.21.15.32:0/1070418870 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fd43c1609d0 0x7fd43c0aec40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:59:53.630 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:53.628+0000 7fd43a800640 1 -- 172.21.15.32:0/1070418870 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fd43c159ff0 msgr2=0x7fd43c157bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:59:53.630 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:53.628+0000 7fd43a800640 1 --2- 172.21.15.32:0/1070418870 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fd43c159ff0 0x7fd43c157bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:59:53.630 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:53.628+0000 7fd43a800640 1 -- 172.21.15.32:0/1070418870 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd43c0b5510 con 0x7fd43c0b4bc0 2024-08-03T20:59:53.630 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:53.628+0000 7fd43a800640 1 --2- 172.21.15.32:0/1070418870 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd43c0b4bc0 0x7fd43c0b4fd0 secure :-1 s=READY pgs=1996 cs=0 l=1 rev1=1 crypto rx=0x7fd43400cb10 tx=0x7fd43400a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:59:53.630 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:53.629+0000 7fd432000640 1 -- 172.21.15.32:0/1070418870 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd434016020 con 0x7fd43c0b4bc0 2024-08-03T20:59:53.631 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:53.629+0000 7fd432000640 1 -- 172.21.15.32:0/1070418870 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fd43400ad60 con 0x7fd43c0b4bc0 2024-08-03T20:59:53.631 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:53.629+0000 7fd432000640 1 -- 172.21.15.32:0/1070418870 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd434012680 con 0x7fd43c0b4bc0 2024-08-03T20:59:53.631 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:53.629+0000 7fd43b200640 1 -- 172.21.15.32:0/1070418870 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd43c15b1a0 con 0x7fd43c0b4bc0 2024-08-03T20:59:53.631 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:53.629+0000 7fd43b200640 1 -- 172.21.15.32:0/1070418870 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd43c1f5430 con 0x7fd43c0b4bc0 2024-08-03T20:59:53.631 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:53.630+0000 7fd43b200640 1 -- 172.21.15.32:0/1070418870 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd43c0cc050 con 0x7fd43c0b4bc0 2024-08-03T20:59:53.631 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:53.631+0000 7fd432000640 1 -- 172.21.15.32:0/1070418870 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7fd434019020 con 0x7fd43c0b4bc0 2024-08-03T20:59:53.632 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:53.632+0000 7fd432000640 1 --2- 172.21.15.32:0/1070418870 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fd418077040 0x7fd418079550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:59:53.632 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:53.632+0000 7fd432000640 1 -- 172.21.15.32:0/1070418870 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(121..121 src has 1..121) ==== 8950+0+0 (secure 0 0 0) 0x7fd434099870 con 0x7fd43c0b4bc0 2024-08-03T20:59:53.633 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:53.632+0000 7fd439e00640 1 --2- 172.21.15.32:0/1070418870 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fd418077040 0x7fd418079550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:59:53.633 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:53.633+0000 7fd439e00640 1 --2- 172.21.15.32:0/1070418870 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fd418077040 0x7fd418079550 secure :-1 s=READY pgs=800 cs=0 l=1 rev1=1 crypto rx=0x7fd42c034000 tx=0x7fd42c004730 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:59:53.637 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:53.636+0000 7fd432000640 1 -- 172.21.15.32:0/1070418870 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7fd434020030 con 0x7fd43c0b4bc0 2024-08-03T20:59:53.886 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:53.885+0000 7fd43b200640 1 -- 172.21.15.32:0/1070418870 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "joinable", "val": "true"} v 0) -- 0x7fd43c0d09e0 con 0x7fd43c0b4bc0 2024-08-03T20:59:54.790 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:54.789+0000 7fd432000640 1 -- 172.21.15.32:0/1070418870 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "joinable", "val": "true"}]=0 cephfs marked joinable; MDS may join as newly active. v304) ==== 162+0+0 (secure 0 0 0) 0x7fd43c0b5510 con 0x7fd43c0b4bc0 2024-08-03T20:59:54.790 INFO:teuthology.orchestra.run.smithi032.stderr:cephfs marked joinable; MDS may join as newly active. 2024-08-03T20:59:54.793 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:54.792+0000 7fd43b200640 1 -- 172.21.15.32:0/1070418870 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fd418077040 msgr2=0x7fd418079550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:59:54.794 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:54.792+0000 7fd43b200640 1 --2- 172.21.15.32:0/1070418870 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fd418077040 0x7fd418079550 secure :-1 s=READY pgs=800 cs=0 l=1 rev1=1 crypto rx=0x7fd42c034000 tx=0x7fd42c004730 comp rx=0 tx=0).stop 2024-08-03T20:59:54.794 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:54.793+0000 7fd43b200640 1 -- 172.21.15.32:0/1070418870 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd43c0b4bc0 msgr2=0x7fd43c0b4fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:59:54.794 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:54.793+0000 7fd43b200640 1 --2- 172.21.15.32:0/1070418870 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd43c0b4bc0 0x7fd43c0b4fd0 secure :-1 s=READY pgs=1996 cs=0 l=1 rev1=1 crypto rx=0x7fd43400cb10 tx=0x7fd43400a060 comp rx=0 tx=0).stop 2024-08-03T20:59:54.794 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:54.793+0000 7fd43b200640 1 -- 172.21.15.32:0/1070418870 shutdown_connections 2024-08-03T20:59:54.794 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:54.793+0000 7fd43b200640 1 --2- 172.21.15.32:0/1070418870 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fd418077040 0x7fd418079550 unknown :-1 s=CLOSED pgs=800 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:59:54.794 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:54.793+0000 7fd43b200640 1 --2- 172.21.15.32:0/1070418870 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd43c0b4bc0 0x7fd43c0b4fd0 unknown :-1 s=CLOSED pgs=1996 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:59:54.794 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:54.793+0000 7fd43b200640 1 --2- 172.21.15.32:0/1070418870 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fd43c1609d0 0x7fd43c0aec40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:59:54.794 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:54.793+0000 7fd43b200640 1 --2- 172.21.15.32:0/1070418870 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fd43c159ff0 0x7fd43c157bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:59:54.794 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:54.793+0000 7fd43b200640 1 -- 172.21.15.32:0/1070418870 >> 172.21.15.32:0/1070418870 conn(0x7fd43c0cfd80 msgr2=0x7fd43c0725f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:59:54.794 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:54.793+0000 7fd43b200640 1 -- 172.21.15.32:0/1070418870 shutdown_connections 2024-08-03T20:59:54.794 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:54.793+0000 7fd43b200640 1 -- 172.21.15.32:0/1070418870 wait complete. 2024-08-03T20:59:54.802 INFO:tasks.ceph.mds.a.smithi032.stderr:2024-08-03T20:59:54.800+0000 7fc55e800640 -1 mds.0.openfiles _load_finish: corrupted header/values: End of buffer [buffer:2] 2024-08-03T20:59:55.227 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-08-03T20:59:55.470 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:55.468+0000 7fb072c00640 1 -- 172.21.15.32:0/1127607207 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb06c06e930 msgr2=0x7fb06c144f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:59:55.470 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:55.468+0000 7fb072c00640 1 --2- 172.21.15.32:0/1127607207 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb06c06e930 0x7fb06c144f90 secure :-1 s=READY pgs=1997 cs=0 l=1 rev1=1 crypto rx=0x7fb06c0b2f80 tx=0x7fb05c02a560 comp rx=0 tx=0).stop 2024-08-03T20:59:55.470 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:55.468+0000 7fb072c00640 1 -- 172.21.15.32:0/1127607207 shutdown_connections 2024-08-03T20:59:55.470 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:55.468+0000 7fb072c00640 1 --2- 172.21.15.32:0/1127607207 >> v2:172.21.15.118:3300/0 conn(0x7fb06c0747b0 0x7fb06c0cc8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:59:55.470 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:55.468+0000 7fb072c00640 1 --2- 172.21.15.32:0/1127607207 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fb06c073e60 0x7fb06c074270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:59:55.470 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:55.468+0000 7fb072c00640 1 --2- 172.21.15.32:0/1127607207 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb06c06e930 0x7fb06c144f90 unknown :-1 s=CLOSED pgs=1997 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:59:55.471 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:55.468+0000 7fb072c00640 1 -- 172.21.15.32:0/1127607207 >> 172.21.15.32:0/1127607207 conn(0x7fb06c0b1bd0 msgr2=0x7fb06c0b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:59:55.471 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:55.468+0000 7fb072c00640 1 -- 172.21.15.32:0/1127607207 shutdown_connections 2024-08-03T20:59:55.471 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:55.468+0000 7fb072c00640 1 -- 172.21.15.32:0/1127607207 wait complete. 2024-08-03T20:59:55.472 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:55.471+0000 7fb072c00640 1 Processor -- start 2024-08-03T20:59:55.472 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:55.471+0000 7fb072c00640 1 -- start start 2024-08-03T20:59:55.475 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:55.472+0000 7fb072c00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb06c06e930 0x7fb06c0cb4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:59:55.475 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:55.472+0000 7fb072c00640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fb06c073e60 0x7fb06c0abfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:59:55.475 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:55.473+0000 7fb072c00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb06c0747b0 0x7fb06c0ac520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:59:55.475 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:55.473+0000 7fb072c00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fb06c0cf9c0 con 0x7fb06c0747b0 2024-08-03T20:59:55.475 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:55.473+0000 7fb072c00640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7fb06c0cf840 con 0x7fb06c06e930 2024-08-03T20:59:55.475 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:55.473+0000 7fb072c00640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7fb06c0cce20 con 0x7fb06c073e60 2024-08-03T20:59:55.475 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:55.473+0000 7fb070c00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb06c0747b0 0x7fb06c0ac520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:59:55.475 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:55.473+0000 7fb06be00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb06c06e930 0x7fb06c0cb4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:59:55.476 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:55.473+0000 7fb063400640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fb06c073e60 0x7fb06c0abfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:59:55.476 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:55.473+0000 7fb070c00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb06c0747b0 0x7fb06c0ac520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:54220/0 (socket says 172.21.15.32:54220) 2024-08-03T20:59:55.476 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:55.473+0000 7fb070c00640 1 -- 172.21.15.32:0/379352819 learned_addr learned my addr 172.21.15.32:0/379352819 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T20:59:55.476 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:55.473+0000 7fb070c00640 1 -- 172.21.15.32:0/379352819 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fb06c073e60 msgr2=0x7fb06c0abfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:59:55.476 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:55.473+0000 7fb070c00640 1 --2- 172.21.15.32:0/379352819 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fb06c073e60 0x7fb06c0abfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:59:55.476 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:55.473+0000 7fb070c00640 1 -- 172.21.15.32:0/379352819 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb06c06e930 msgr2=0x7fb06c0cb4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:59:55.476 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:55.473+0000 7fb070c00640 1 --2- 172.21.15.32:0/379352819 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb06c06e930 0x7fb06c0cb4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:59:55.476 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:55.473+0000 7fb070c00640 1 -- 172.21.15.32:0/379352819 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb06c0cba30 con 0x7fb06c0747b0 2024-08-03T20:59:55.476 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:55.474+0000 7fb070c00640 1 --2- 172.21.15.32:0/379352819 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb06c0747b0 0x7fb06c0ac520 secure :-1 s=READY pgs=1998 cs=0 l=1 rev1=1 crypto rx=0x7fb06400e9c0 tx=0x7fb06400ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:59:55.477 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:55.474+0000 7fb068c00640 1 -- 172.21.15.32:0/379352819 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb064004d20 con 0x7fb06c0747b0 2024-08-03T20:59:55.477 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:55.474+0000 7fb068c00640 1 -- 172.21.15.32:0/379352819 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fb06400cb20 con 0x7fb06c0747b0 2024-08-03T20:59:55.477 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:55.474+0000 7fb072c00640 1 -- 172.21.15.32:0/379352819 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb06c0c4f70 con 0x7fb06c0747b0 2024-08-03T20:59:55.477 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:55.474+0000 7fb068c00640 1 -- 172.21.15.32:0/379352819 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb064005020 con 0x7fb06c0747b0 2024-08-03T20:59:55.477 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:55.474+0000 7fb072c00640 1 -- 172.21.15.32:0/379352819 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb06c0c5530 con 0x7fb06c0747b0 2024-08-03T20:59:55.477 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:55.475+0000 7fb072c00640 1 -- 172.21.15.32:0/379352819 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb06c074270 con 0x7fb06c0747b0 2024-08-03T20:59:55.478 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:55.476+0000 7fb068c00640 1 -- 172.21.15.32:0/379352819 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7fb0640051c0 con 0x7fb06c0747b0 2024-08-03T20:59:55.479 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:55.477+0000 7fb068c00640 1 --2- 172.21.15.32:0/379352819 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fb048077010 0x7fb048079520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T20:59:55.479 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:55.478+0000 7fb068c00640 1 -- 172.21.15.32:0/379352819 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(121..121 src has 1..121) ==== 8950+0+0 (secure 0 0 0) 0x7fb064099c90 con 0x7fb06c0747b0 2024-08-03T20:59:55.479 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:55.478+0000 7fb06be00640 1 --2- 172.21.15.32:0/379352819 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fb048077010 0x7fb048079520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T20:59:55.480 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:55.479+0000 7fb06be00640 1 --2- 172.21.15.32:0/379352819 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fb048077010 0x7fb048079520 secure :-1 s=READY pgs=801 cs=0 l=1 rev1=1 crypto rx=0x7fb05c033000 tx=0x7fb05c004730 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T20:59:55.483 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:55.482+0000 7fb068c00640 1 -- 172.21.15.32:0/379352819 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7fb064016030 con 0x7fb06c0747b0 2024-08-03T20:59:55.724 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:55.722+0000 7fb072c00640 1 -- 172.21.15.32:0/379352819 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb06c141000 con 0x7fb06c0747b0 2024-08-03T20:59:55.724 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:55.723+0000 7fb068c00640 1 -- 172.21.15.32:0/379352819 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 305 v305) ==== 95+0+5229 (secure 0 0 0) 0x7fb064061c00 con 0x7fb06c0747b0 2024-08-03T20:59:55.725 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-03T20:59:55.725 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":305,"btime":"2024-08-03T20:59:54:792776+0000","default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6439,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6835/531080160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6834","nonce":531080160},{"type":"v1","addr":"172.21.15.118:6835","nonce":531080160}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":84},{"gid":6454,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6837/2796882319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6836","nonce":2796882319},{"type":"v1","addr":"172.21.15.118:6837","nonce":2796882319}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":86},{"gid":6813,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6834/2686933890","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":2686933890},{"type":"v1","addr":"172.21.15.32:6834","nonce":2686933890}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":85}],"filesystems":[{"mdsmap":{"epoch":305,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-08-03T20:54:26.950518+0000","modified":"2024-08-03T20:59:54.792748+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":121,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":10035},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10035":{"gid":10035,"name":"a","rank":0,"incarnation":305,"state":"up:replay","state_seq":1,"addr":"172.21.15.32:6835/2017655932","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6833","nonce":2017655932},{"type":"v1","addr":"172.21.15.32:6835","nonce":2017655932}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":6}]} 2024-08-03T20:59:55.725 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 305 2024-08-03T20:59:55.727 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:55.725+0000 7fb072c00640 1 -- 172.21.15.32:0/379352819 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fb048077010 msgr2=0x7fb048079520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:59:55.727 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:55.725+0000 7fb072c00640 1 --2- 172.21.15.32:0/379352819 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fb048077010 0x7fb048079520 secure :-1 s=READY pgs=801 cs=0 l=1 rev1=1 crypto rx=0x7fb05c033000 tx=0x7fb05c004730 comp rx=0 tx=0).stop 2024-08-03T20:59:55.727 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:55.725+0000 7fb072c00640 1 -- 172.21.15.32:0/379352819 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb06c0747b0 msgr2=0x7fb06c0ac520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T20:59:55.727 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:55.725+0000 7fb072c00640 1 --2- 172.21.15.32:0/379352819 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb06c0747b0 0x7fb06c0ac520 secure :-1 s=READY pgs=1998 cs=0 l=1 rev1=1 crypto rx=0x7fb06400e9c0 tx=0x7fb06400ee80 comp rx=0 tx=0).stop 2024-08-03T20:59:55.728 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:55.725+0000 7fb072c00640 1 -- 172.21.15.32:0/379352819 shutdown_connections 2024-08-03T20:59:55.728 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:55.725+0000 7fb072c00640 1 --2- 172.21.15.32:0/379352819 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fb048077010 0x7fb048079520 unknown :-1 s=CLOSED pgs=801 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:59:55.728 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:55.725+0000 7fb072c00640 1 --2- 172.21.15.32:0/379352819 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb06c0747b0 0x7fb06c0ac520 unknown :-1 s=CLOSED pgs=1998 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:59:55.728 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:55.725+0000 7fb072c00640 1 --2- 172.21.15.32:0/379352819 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fb06c073e60 0x7fb06c0abfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:59:55.728 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:55.725+0000 7fb072c00640 1 --2- 172.21.15.32:0/379352819 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb06c06e930 0x7fb06c0cb4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T20:59:55.728 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:55.725+0000 7fb072c00640 1 -- 172.21.15.32:0/379352819 >> 172.21.15.32:0/379352819 conn(0x7fb06c0b1bd0 msgr2=0x7fb06c0b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T20:59:55.728 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:55.725+0000 7fb072c00640 1 -- 172.21.15.32:0/379352819 shutdown_connections 2024-08-03T20:59:55.728 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T20:59:55.726+0000 7fb072c00640 1 -- 172.21.15.32:0/379352819 wait complete. 2024-08-03T20:59:55.752 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 305, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-08-03T20:54:26.950518+0000', 'modified': '2024-08-03T20:59:54.792748+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 121, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 10035}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_10035': {'gid': 10035, 'name': 'a', 'rank': 0, 'incarnation': 305, 'state': 'up:replay', 'state_seq': 1, 'addr': '172.21.15.32:6835/2017655932', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6833', 'nonce': 2017655932}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 2017655932}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [13], 'metadata_pool': 12, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 0, 'qdb_cluster': []} 2024-08-03T20:59:55.752 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_10035:up:replay 2024-08-03T20:59:55.752 DEBUG:tasks.ceph_test_case:wait_until_true: waiting (timeout=60 retry_count=0)... 2024-08-03T21:00:01.157 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-08-03T21:00:01.397 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:01.395+0000 7fa8a8200640 1 -- 172.21.15.32:0/1456610177 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa8a006e930 msgr2=0x7fa8a0144f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:00:01.397 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:01.395+0000 7fa8a8200640 1 --2- 172.21.15.32:0/1456610177 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa8a006e930 0x7fa8a0144f90 secure :-1 s=READY pgs=2000 cs=0 l=1 rev1=1 crypto rx=0x7fa894009830 tx=0x7fa89402a460 comp rx=0 tx=0).stop 2024-08-03T21:00:01.397 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:01.395+0000 7fa8a8200640 1 -- 172.21.15.32:0/1456610177 shutdown_connections 2024-08-03T21:00:01.397 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:01.395+0000 7fa8a8200640 1 --2- 172.21.15.32:0/1456610177 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa8a006e930 0x7fa8a0144f90 unknown :-1 s=CLOSED pgs=2000 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:01.397 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:01.395+0000 7fa8a8200640 1 -- 172.21.15.32:0/1456610177 >> 172.21.15.32:0/1456610177 conn(0x7fa8a00b1bd0 msgr2=0x7fa8a00b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T21:00:01.398 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:01.395+0000 7fa8a8200640 1 -- 172.21.15.32:0/1456610177 shutdown_connections 2024-08-03T21:00:01.398 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:01.395+0000 7fa8a8200640 1 -- 172.21.15.32:0/1456610177 wait complete. 2024-08-03T21:00:01.399 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:01.397+0000 7fa8a8200640 1 Processor -- start 2024-08-03T21:00:01.400 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:01.398+0000 7fa8a8200640 1 -- start start 2024-08-03T21:00:01.401 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:01.399+0000 7fa8a8200640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa8a006e930 0x7fa8a013dc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:00:01.401 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:01.399+0000 7fa8a8200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa8a0073e60 0x7fa8a00ae890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:00:01.401 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:01.399+0000 7fa8a8200640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fa8a015fff0 0x7fa8a00aedd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:00:01.401 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:01.399+0000 7fa8a8200640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fa8a00cf840 con 0x7fa8a0073e60 2024-08-03T21:00:01.401 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:01.399+0000 7fa8a8200640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7fa8a0145f30 con 0x7fa8a006e930 2024-08-03T21:00:01.401 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:01.399+0000 7fa8a8200640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7fa8a00cf9c0 con 0x7fa8a015fff0 2024-08-03T21:00:01.401 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:01.399+0000 7fa8a4e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa8a0073e60 0x7fa8a00ae890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:00:01.401 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:01.399+0000 7fa8a4e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa8a0073e60 0x7fa8a00ae890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:39542/0 (socket says 172.21.15.32:39542) 2024-08-03T21:00:01.401 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:01.399+0000 7fa8a4e00640 1 -- 172.21.15.32:0/3373734278 learned_addr learned my addr 172.21.15.32:0/3373734278 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T21:00:01.402 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:01.400+0000 7fa8a6200640 1 --2- 172.21.15.32:0/3373734278 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fa8a015fff0 0x7fa8a00aedd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:00:01.403 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:01.400+0000 7fa8a5800640 1 --2- 172.21.15.32:0/3373734278 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa8a006e930 0x7fa8a013dc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:00:01.403 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:01.400+0000 7fa8a4e00640 1 -- 172.21.15.32:0/3373734278 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fa8a015fff0 msgr2=0x7fa8a00aedd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:00:01.403 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:01.400+0000 7fa8a4e00640 1 --2- 172.21.15.32:0/3373734278 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fa8a015fff0 0x7fa8a00aedd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:01.403 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:01.400+0000 7fa8a4e00640 1 -- 172.21.15.32:0/3373734278 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa8a006e930 msgr2=0x7fa8a013dc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:00:01.403 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:01.400+0000 7fa8a4e00640 1 --2- 172.21.15.32:0/3373734278 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa8a006e930 0x7fa8a013dc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:01.403 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:01.400+0000 7fa8a4e00640 1 -- 172.21.15.32:0/3373734278 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa8a01414e0 con 0x7fa8a0073e60 2024-08-03T21:00:01.403 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:01.400+0000 7fa8a4e00640 1 --2- 172.21.15.32:0/3373734278 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa8a0073e60 0x7fa8a00ae890 secure :-1 s=READY pgs=2001 cs=0 l=1 rev1=1 crypto rx=0x7fa88c00c480 tx=0x7fa88c00c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T21:00:01.403 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:01.400+0000 7fa89d600640 1 -- 172.21.15.32:0/3373734278 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa88c016020 con 0x7fa8a0073e60 2024-08-03T21:00:01.403 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:01.400+0000 7fa89d600640 1 -- 172.21.15.32:0/3373734278 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fa88c012920 con 0x7fa8a0073e60 2024-08-03T21:00:01.403 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:01.400+0000 7fa89d600640 1 -- 172.21.15.32:0/3373734278 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa88c012d10 con 0x7fa8a0073e60 2024-08-03T21:00:01.403 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:01.400+0000 7fa8a8200640 1 -- 172.21.15.32:0/3373734278 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa8a01419e0 con 0x7fa8a0073e60 2024-08-03T21:00:01.403 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:01.401+0000 7fa8a8200640 1 -- 172.21.15.32:0/3373734278 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa8a0142140 con 0x7fa8a0073e60 2024-08-03T21:00:01.404 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:01.402+0000 7fa8a8200640 1 -- 172.21.15.32:0/3373734278 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa8a0050cc0 con 0x7fa8a0073e60 2024-08-03T21:00:01.404 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:01.403+0000 7fa89d600640 1 -- 172.21.15.32:0/3373734278 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7fa88c019020 con 0x7fa8a0073e60 2024-08-03T21:00:01.406 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:01.404+0000 7fa89d600640 1 --2- 172.21.15.32:0/3373734278 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fa880077010 0x7fa880079520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:00:01.406 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:01.405+0000 7fa89d600640 1 -- 172.21.15.32:0/3373734278 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(121..121 src has 1..121) ==== 8950+0+0 (secure 0 0 0) 0x7fa88c0993e0 con 0x7fa8a0073e60 2024-08-03T21:00:01.406 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:01.405+0000 7fa8a5800640 1 --2- 172.21.15.32:0/3373734278 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fa880077010 0x7fa880079520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:00:01.407 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:01.406+0000 7fa8a5800640 1 --2- 172.21.15.32:0/3373734278 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fa880077010 0x7fa880079520 secure :-1 s=READY pgs=802 cs=0 l=1 rev1=1 crypto rx=0x7fa894034000 tx=0x7fa89402c040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T21:00:01.410 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:01.409+0000 7fa89d600640 1 -- 172.21.15.32:0/3373734278 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7fa8a0050cc0 con 0x7fa8a0073e60 2024-08-03T21:00:01.654 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:01.652+0000 7fa8a8200640 1 -- 172.21.15.32:0/3373734278 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa8a0160dd0 con 0x7fa8a0073e60 2024-08-03T21:00:01.654 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:01.653+0000 7fa89d600640 1 -- 172.21.15.32:0/3373734278 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 308 v308) ==== 95+0+5238 (secure 0 0 0) 0x7fa88c020030 con 0x7fa8a0073e60 2024-08-03T21:00:01.655 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-03T21:00:01.655 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":308,"btime":"2024-08-03T20:59:57:804485+0000","default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6439,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6835/531080160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6834","nonce":531080160},{"type":"v1","addr":"172.21.15.118:6835","nonce":531080160}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":84},{"gid":6454,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6837/2796882319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6836","nonce":2796882319},{"type":"v1","addr":"172.21.15.118:6837","nonce":2796882319}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":86},{"gid":6813,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6834/2686933890","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":2686933890},{"type":"v1","addr":"172.21.15.32:6834","nonce":2686933890}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":85}],"filesystems":[{"mdsmap":{"epoch":308,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-08-03T20:54:26.950518+0000","modified":"2024-08-03T20:59:57.804479+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":121,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":10035},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10035":{"gid":10035,"name":"a","rank":0,"incarnation":305,"state":"up:active","state_seq":4,"addr":"172.21.15.32:6835/2017655932","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6833","nonce":2017655932},{"type":"v1","addr":"172.21.15.32:6835","nonce":2017655932}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":10035,"qdb_cluster":[10035]},"id":6}]} 2024-08-03T21:00:01.655 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 308 2024-08-03T21:00:01.657 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:01.655+0000 7fa8a8200640 1 -- 172.21.15.32:0/3373734278 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fa880077010 msgr2=0x7fa880079520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:00:01.657 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:01.655+0000 7fa8a8200640 1 --2- 172.21.15.32:0/3373734278 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fa880077010 0x7fa880079520 secure :-1 s=READY pgs=802 cs=0 l=1 rev1=1 crypto rx=0x7fa894034000 tx=0x7fa89402c040 comp rx=0 tx=0).stop 2024-08-03T21:00:01.657 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:01.655+0000 7fa8a8200640 1 -- 172.21.15.32:0/3373734278 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa8a0073e60 msgr2=0x7fa8a00ae890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:00:01.657 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:01.655+0000 7fa8a8200640 1 --2- 172.21.15.32:0/3373734278 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa8a0073e60 0x7fa8a00ae890 secure :-1 s=READY pgs=2001 cs=0 l=1 rev1=1 crypto rx=0x7fa88c00c480 tx=0x7fa88c00c940 comp rx=0 tx=0).stop 2024-08-03T21:00:01.658 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:01.655+0000 7fa8a8200640 1 -- 172.21.15.32:0/3373734278 shutdown_connections 2024-08-03T21:00:01.658 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:01.655+0000 7fa8a8200640 1 --2- 172.21.15.32:0/3373734278 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fa880077010 0x7fa880079520 unknown :-1 s=CLOSED pgs=802 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:01.658 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:01.655+0000 7fa8a8200640 1 --2- 172.21.15.32:0/3373734278 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fa8a015fff0 0x7fa8a00aedd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:01.658 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:01.655+0000 7fa8a8200640 1 --2- 172.21.15.32:0/3373734278 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa8a0073e60 0x7fa8a00ae890 unknown :-1 s=CLOSED pgs=2001 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:01.658 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:01.655+0000 7fa8a8200640 1 --2- 172.21.15.32:0/3373734278 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa8a006e930 0x7fa8a013dc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:01.658 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:01.655+0000 7fa8a8200640 1 -- 172.21.15.32:0/3373734278 >> 172.21.15.32:0/3373734278 conn(0x7fa8a00b1bd0 msgr2=0x7fa8a0074270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T21:00:01.658 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:01.656+0000 7fa8a8200640 1 -- 172.21.15.32:0/3373734278 shutdown_connections 2024-08-03T21:00:01.658 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:01.656+0000 7fa8a8200640 1 -- 172.21.15.32:0/3373734278 wait complete. 2024-08-03T21:00:01.682 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 308, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-08-03T20:54:26.950518+0000', 'modified': '2024-08-03T20:59:57.804479+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 121, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 10035}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_10035': {'gid': 10035, 'name': 'a', 'rank': 0, 'incarnation': 305, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.32:6835/2017655932', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6833', 'nonce': 2017655932}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 2017655932}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [13], 'metadata_pool': 12, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 10035, 'qdb_cluster': [10035]} 2024-08-03T21:00:01.682 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-08-03T21:00:01.683 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2024-08-03T21:00:01.780 INFO:tasks.ceph.mds.a.smithi032.stderr:2024-08-03T21:00:01.778+0000 7fc563800640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-08-03T21:00:01.913 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:01.911+0000 7f362be00640 1 -- 172.21.15.32:0/3264901903 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f362c0af020 msgr2=0x7f362c0742b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:00:01.913 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:01.911+0000 7f362be00640 1 --2- 172.21.15.32:0/3264901903 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f362c0af020 0x7f362c0742b0 secure :-1 s=READY pgs=2003 cs=0 l=1 rev1=1 crypto rx=0x7f362c160f00 tx=0x7f361c02a4a0 comp rx=0 tx=0).stop 2024-08-03T21:00:01.913 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:01.911+0000 7f362be00640 1 -- 172.21.15.32:0/3264901903 shutdown_connections 2024-08-03T21:00:01.913 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:01.911+0000 7f362be00640 1 --2- 172.21.15.32:0/3264901903 >> v2:172.21.15.118:3300/0 conn(0x7f362c0acd80 0x7f362c0ad190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:01.913 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:01.911+0000 7f362be00640 1 --2- 172.21.15.32:0/3264901903 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f362c0af020 0x7f362c0742b0 unknown :-1 s=CLOSED pgs=2003 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:01.913 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:01.911+0000 7f362be00640 1 -- 172.21.15.32:0/3264901903 >> 172.21.15.32:0/3264901903 conn(0x7f362c0b2120 msgr2=0x7f362c0cfd60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T21:00:01.914 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:01.911+0000 7f362be00640 1 -- 172.21.15.32:0/3264901903 shutdown_connections 2024-08-03T21:00:01.914 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:01.911+0000 7f362be00640 1 -- 172.21.15.32:0/3264901903 wait complete. 2024-08-03T21:00:01.915 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:01.913+0000 7f362be00640 1 Processor -- start 2024-08-03T21:00:01.915 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:01.913+0000 7f362be00640 1 -- start start 2024-08-03T21:00:01.918 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:01.915+0000 7f362be00640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f362c06f440 0x7f362c1f59a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:00:01.918 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:01.915+0000 7f362be00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f362c0acd80 0x7f362c1f5ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:00:01.918 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:01.915+0000 7f362be00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f362c0af020 0x7f362c15ea80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:00:01.918 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:01.915+0000 7f362be00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f362c1601e0 con 0x7f362c0acd80 2024-08-03T21:00:01.918 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:01.915+0000 7f362be00640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f362c160060 con 0x7f362c0af020 2024-08-03T21:00:01.918 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:01.915+0000 7f362be00640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7f362c160360 con 0x7f362c06f440 2024-08-03T21:00:01.918 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:01.915+0000 7f3623e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f362c0acd80 0x7f362c1f5ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:00:01.918 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:01.915+0000 7f3623e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f362c0acd80 0x7f362c1f5ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:39572/0 (socket says 172.21.15.32:39572) 2024-08-03T21:00:01.918 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:01.915+0000 7f3623e00640 1 -- 172.21.15.32:0/133691314 learned_addr learned my addr 172.21.15.32:0/133691314 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T21:00:01.918 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:01.915+0000 7f362b400640 1 --2- 172.21.15.32:0/133691314 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f362c0af020 0x7f362c15ea80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:00:01.919 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:01.915+0000 7f362aa00640 1 --2- 172.21.15.32:0/133691314 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f362c06f440 0x7f362c1f59a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:00:01.919 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:01.916+0000 7f3623e00640 1 -- 172.21.15.32:0/133691314 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f362c06f440 msgr2=0x7f362c1f59a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:00:01.919 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:01.916+0000 7f3623e00640 1 --2- 172.21.15.32:0/133691314 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f362c06f440 0x7f362c1f59a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:01.919 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:01.916+0000 7f3623e00640 1 -- 172.21.15.32:0/133691314 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f362c0af020 msgr2=0x7f362c15ea80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:00:01.919 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:01.916+0000 7f3623e00640 1 --2- 172.21.15.32:0/133691314 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f362c0af020 0x7f362c15ea80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:01.919 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:01.916+0000 7f3623e00640 1 -- 172.21.15.32:0/133691314 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f362c0b2cd0 con 0x7f362c0acd80 2024-08-03T21:00:01.919 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:01.916+0000 7f362b400640 1 --2- 172.21.15.32:0/133691314 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f362c0af020 0x7f362c15ea80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-03T21:00:01.919 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:01.916+0000 7f3623e00640 1 --2- 172.21.15.32:0/133691314 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f362c0acd80 0x7f362c1f5ee0 secure :-1 s=READY pgs=2004 cs=0 l=1 rev1=1 crypto rx=0x7f361400e790 tx=0x7f361400ec50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T21:00:01.919 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:01.916+0000 7f3623400640 1 -- 172.21.15.32:0/133691314 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f361400ca60 con 0x7f362c0acd80 2024-08-03T21:00:01.919 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:01.916+0000 7f3623400640 1 -- 172.21.15.32:0/133691314 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f361400ccf0 con 0x7f362c0acd80 2024-08-03T21:00:01.919 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:01.916+0000 7f362be00640 1 -- 172.21.15.32:0/133691314 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f362c0b3250 con 0x7f362c0acd80 2024-08-03T21:00:01.919 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:01.916+0000 7f3623400640 1 -- 172.21.15.32:0/133691314 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3614010690 con 0x7f362c0acd80 2024-08-03T21:00:01.919 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:01.916+0000 7f362be00640 1 -- 172.21.15.32:0/133691314 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f362c1f5480 con 0x7f362c0acd80 2024-08-03T21:00:01.920 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:01.919+0000 7f3623400640 1 -- 172.21.15.32:0/133691314 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f3614010830 con 0x7f362c0acd80 2024-08-03T21:00:01.921 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:01.920+0000 7f3623400640 1 --2- 172.21.15.32:0/133691314 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f3608077010 0x7f3608079520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:00:01.921 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:01.920+0000 7f3623400640 1 -- 172.21.15.32:0/133691314 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(121..121 src has 1..121) ==== 8950+0+0 (secure 0 0 0) 0x7f3614099fb0 con 0x7f362c0acd80 2024-08-03T21:00:01.922 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:01.920+0000 7f362aa00640 1 --2- 172.21.15.32:0/133691314 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f3608077010 0x7f3608079520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:00:01.922 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:01.920+0000 7f3638fa8740 1 Processor -- start 2024-08-03T21:00:01.922 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:01.920+0000 7f3638fa8740 1 -- start start 2024-08-03T21:00:01.922 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:01.921+0000 7f362aa00640 1 --2- 172.21.15.32:0/133691314 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f3608077010 0x7f3608079520 secure :-1 s=READY pgs=803 cs=0 l=1 rev1=1 crypto rx=0x7f361c034000 tx=0x7f361c004730 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T21:00:01.923 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:01.922+0000 7f3638fa8740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55d1b8f36270 0x55d1b9321080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:00:01.924 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:01.922+0000 7f3638fa8740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55d1b8f0d3b0 0x55d1b92f5ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:00:01.924 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:01.922+0000 7f3638fa8740 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55d1b8f15590 0x55d1b92f6410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:00:01.924 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:01.922+0000 7f3638fa8740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55d1b924ea60 con 0x55d1b8f36270 2024-08-03T21:00:01.924 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:01.922+0000 7f3638fa8740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x55d1b923df60 con 0x55d1b8f0d3b0 2024-08-03T21:00:01.924 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:01.922+0000 7f3638fa8740 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x55d1b9241fa0 con 0x55d1b8f15590 2024-08-03T21:00:01.924 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:01.922+0000 7f362aa00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55d1b8f36270 0x55d1b9321080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:00:01.924 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:01.922+0000 7f362aa00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55d1b8f36270 0x55d1b9321080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:39578/0 (socket says 172.21.15.32:39578) 2024-08-03T21:00:01.924 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:01.922+0000 7f362aa00640 1 -- 172.21.15.32:0/1059967858 learned_addr learned my addr 172.21.15.32:0/1059967858 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T21:00:01.925 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:01.923+0000 7f362aa00640 1 -- 172.21.15.32:0/1059967858 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55d1b8f15590 msgr2=0x55d1b92f6410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T21:00:01.926 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:01.923+0000 7f362aa00640 1 --2- 172.21.15.32:0/1059967858 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55d1b8f15590 0x55d1b92f6410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:01.926 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:01.923+0000 7f362aa00640 1 -- 172.21.15.32:0/1059967858 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55d1b8f0d3b0 msgr2=0x55d1b92f5ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T21:00:01.926 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:01.923+0000 7f3623e00640 1 --2- 172.21.15.32:0/1059967858 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55d1b8f0d3b0 0x55d1b92f5ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:00:01.926 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:01.923+0000 7f362aa00640 1 --2- 172.21.15.32:0/1059967858 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55d1b8f0d3b0 0x55d1b92f5ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:01.926 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:01.923+0000 7f362aa00640 1 -- 172.21.15.32:0/1059967858 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d1b8fa1850 con 0x55d1b8f36270 2024-08-03T21:00:01.926 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:01.923+0000 7f3623e00640 1 --2- 172.21.15.32:0/1059967858 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55d1b8f0d3b0 0x55d1b92f5ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-03T21:00:01.926 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:01.923+0000 7f362aa00640 1 --2- 172.21.15.32:0/1059967858 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55d1b8f36270 0x55d1b9321080 secure :-1 s=READY pgs=2005 cs=0 l=1 rev1=1 crypto rx=0x7f361c02aa90 tx=0x7f361c03e070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2d6fe1f6e2f9684e server_cookie=0 in_seq=0 out_seq=0 2024-08-03T21:00:01.926 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:01.923+0000 7f361b400640 1 -- 172.21.15.32:0/1059967858 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f361c003d80 con 0x55d1b8f36270 2024-08-03T21:00:01.926 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:01.923+0000 7f361b400640 1 -- 172.21.15.32:0/1059967858 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f361c047420 con 0x55d1b8f36270 2024-08-03T21:00:01.926 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:01.923+0000 7f361b400640 1 -- 172.21.15.32:0/1059967858 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f361c0477d0 con 0x55d1b8f36270 2024-08-03T21:00:01.926 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:01.924+0000 7f3638fa8740 1 -- 172.21.15.32:0/1059967858 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55d1b8f36270 msgr2=0x55d1b9321080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:00:01.926 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:01.924+0000 7f3638fa8740 1 --2- 172.21.15.32:0/1059967858 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55d1b8f36270 0x55d1b9321080 secure :-1 s=READY pgs=2005 cs=0 l=1 rev1=1 crypto rx=0x7f361c02aa90 tx=0x7f361c03e070 comp rx=0 tx=0).stop 2024-08-03T21:00:01.926 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:01.924+0000 7f3638fa8740 1 -- 172.21.15.32:0/1059967858 shutdown_connections 2024-08-03T21:00:01.927 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:01.924+0000 7f3638fa8740 1 --2- 172.21.15.32:0/1059967858 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55d1b8f0d3b0 0x55d1b92f5ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:01.927 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:01.924+0000 7f3638fa8740 1 --2- 172.21.15.32:0/1059967858 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55d1b8f15590 0x55d1b92f6410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:01.927 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:01.924+0000 7f3638fa8740 1 --2- 172.21.15.32:0/1059967858 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55d1b8f36270 0x55d1b9321080 unknown :-1 s=CLOSED pgs=2005 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:01.927 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:01.924+0000 7f3638fa8740 1 -- 172.21.15.32:0/1059967858 >> 172.21.15.32:0/1059967858 conn(0x55d1b8f2a3c0 msgr2=0x55d1b92dc370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T21:00:01.927 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:01.924+0000 7f3638fa8740 1 -- 172.21.15.32:0/1059967858 shutdown_connections 2024-08-03T21:00:01.927 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:01.924+0000 7f3638fa8740 1 -- 172.21.15.32:0/1059967858 wait complete. 2024-08-03T21:00:01.927 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:01.925+0000 7f3638fa8740 1 -- start start 2024-08-03T21:00:01.927 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:01.925+0000 7f3638fa8740 1 Processor -- start 2024-08-03T21:00:01.929 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:01.927+0000 7f3638fa8740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55d1b8f0d3b0 0x55d1b9305180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:00:01.930 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:01.927+0000 7f3638fa8740 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55d1b8f15590 0x55d1b9303780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:00:01.930 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:01.927+0000 7f3638fa8740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55d1b8f36270 0x55d1b931d5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:00:01.930 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:01.927+0000 7f3638fa8740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55d1b8fb0620 con 0x55d1b8f36270 2024-08-03T21:00:01.930 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:01.927+0000 7f3638fa8740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x55d1b91db160 con 0x55d1b8f0d3b0 2024-08-03T21:00:01.930 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:01.927+0000 7f3638fa8740 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x55d1b9305a40 con 0x55d1b8f15590 2024-08-03T21:00:01.930 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:01.927+0000 7f3638fa8740 10 client.0 _ll_register_callbacks cb 0x55d1b91ea6b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-03T21:00:01.930 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:01.927+0000 7f362b400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55d1b8f36270 0x55d1b931d5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:00:01.930 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:01.927+0000 7f362b400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55d1b8f36270 0x55d1b931d5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:39590/0 (socket says 172.21.15.32:39590) 2024-08-03T21:00:01.930 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:01.927+0000 7f362b400640 1 -- 172.21.15.32:0/3137142714 learned_addr learned my addr 172.21.15.32:0/3137142714 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T21:00:01.930 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:01.928+0000 7f362aa00640 1 --2- 172.21.15.32:0/3137142714 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55d1b8f0d3b0 0x55d1b9305180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:00:01.930 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:01.928+0000 7f362b400640 1 -- 172.21.15.32:0/3137142714 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55d1b8f15590 msgr2=0x55d1b9303780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T21:00:01.930 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:01.928+0000 7f3623e00640 1 --2- 172.21.15.32:0/3137142714 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55d1b8f15590 0x55d1b9303780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:00:01.931 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:01.928+0000 7f362b400640 1 --2- 172.21.15.32:0/3137142714 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55d1b8f15590 0x55d1b9303780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:01.931 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:01.928+0000 7f362b400640 1 -- 172.21.15.32:0/3137142714 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55d1b8f0d3b0 msgr2=0x55d1b9305180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T21:00:01.931 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:01.928+0000 7f362b400640 1 --2- 172.21.15.32:0/3137142714 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55d1b8f0d3b0 0x55d1b9305180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:01.931 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:01.928+0000 7f362b400640 1 -- 172.21.15.32:0/3137142714 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d1b90186b0 con 0x55d1b8f36270 2024-08-03T21:00:01.931 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:01.928+0000 7f362aa00640 1 --2- 172.21.15.32:0/3137142714 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55d1b8f0d3b0 0x55d1b9305180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-03T21:00:01.931 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:01.928+0000 7f3623e00640 1 --2- 172.21.15.32:0/3137142714 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55d1b8f15590 0x55d1b9303780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-03T21:00:01.931 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:01.928+0000 7f362b400640 1 --2- 172.21.15.32:0/3137142714 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55d1b8f36270 0x55d1b931d5e0 secure :-1 s=READY pgs=2006 cs=0 l=1 rev1=1 crypto rx=0x7f362400e7e0 tx=0x7f362400eca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bafe726e42213ea server_cookie=0 in_seq=0 out_seq=0 2024-08-03T21:00:01.931 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:01.928+0000 7f360fe00640 10 client.0 ms_handle_connect on v2:172.21.15.32:3300/0 2024-08-03T21:00:01.931 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:01.928+0000 7f360fe00640 1 -- 172.21.15.32:0/3137142714 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f362400c8d0 con 0x55d1b8f36270 2024-08-03T21:00:01.931 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:01.928+0000 7f3638fa8740 1 -- 172.21.15.32:0/3137142714 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f361c047420 con 0x55d1b8f36270 2024-08-03T21:00:01.931 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:01.928+0000 7f360fe00640 1 -- 172.21.15.32:0/3137142714 <== mon.0 v2:172.21.15.32:3300/0 2 ==== osd_map(121..121 src has 1..121) ==== 8950+0+0 (secure 0 0 0) 0x7f362400ca70 con 0x55d1b8f36270 2024-08-03T21:00:01.933 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:01.929+0000 7f360fe00640 1 -- 172.21.15.32:0/3137142714 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_get_version_reply(handle=1 version=308) ==== 24+0+0 (secure 0 0 0) 0x7f3624023050 con 0x55d1b8f36270 2024-08-03T21:00:01.933 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:01.929+0000 7f3638fa8740 10 client.10101 fetch_fsmap learned FSMap version 308 2024-08-03T21:00:01.933 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:01.929+0000 7f3638fa8740 1 -- 172.21.15.32:0/3137142714 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,fsmap=308,monmap=2+}) -- 0x55d1b9305780 con 0x55d1b8f36270 2024-08-03T21:00:01.934 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:01.929+0000 7f360fe00640 1 -- 172.21.15.32:0/3137142714 <== mon.0 v2:172.21.15.32:3300/0 4 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f361c047420 con 0x55d1b8f36270 2024-08-03T21:00:01.934 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:01.929+0000 7f360fe00640 1 -- 172.21.15.32:0/3137142714 <== mon.0 v2:172.21.15.32:3300/0 5 ==== fsmap(e 308) ==== 3502+0+0 (secure 0 0 0) 0x7f3624014070 con 0x55d1b8f36270 2024-08-03T21:00:01.934 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:01.929+0000 7f3638fa8740 10 client.10101 fetch_fsmap finished waiting for FSMap version 308 2024-08-03T21:00:01.934 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:01.929+0000 7f3638fa8740 10 client.10101 resolve_mds: resolved name 'a' to daemon mds.a 2024-08-03T21:00:01.934 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:01.929+0000 7f3638fa8740 20 client.10101 populate_metadata read hostname 'smithi032' 2024-08-03T21:00:01.934 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:01.929+0000 7f3638fa8740 1 --2- 172.21.15.32:0/3137142714 >> [v2:172.21.15.32:6833/2017655932,v1:172.21.15.32:6835/2017655932] conn(0x55d1b8ef4e40 0x55d1b93304c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:00:01.934 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:01.929+0000 7f3638fa8740 4 client.10101 mds_command: new command op to 10035 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-08-03T21:00:01.934 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:01.929+0000 7f3638fa8740 1 -- 172.21.15.32:0/3137142714 --> [v2:172.21.15.32:6833/2017655932,v1:172.21.15.32:6835/2017655932] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d1b92509b0 con 0x55d1b8ef4e40 2024-08-03T21:00:01.934 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:01.929+0000 7f362aa00640 1 --2- 172.21.15.32:0/3137142714 >> [v2:172.21.15.32:6833/2017655932,v1:172.21.15.32:6835/2017655932] conn(0x55d1b8ef4e40 0x55d1b93304c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:00:01.934 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:01.930+0000 7f362aa00640 1 --2- 172.21.15.32:0/3137142714 >> [v2:172.21.15.32:6833/2017655932,v1:172.21.15.32:6835/2017655932] conn(0x55d1b8ef4e40 0x55d1b93304c0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1a4648020f2a332e server_cookie=a1553ff8c2635a02 in_seq=0 out_seq=0 2024-08-03T21:00:01.934 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:01.930+0000 7f360fe00640 10 client.10101 ms_handle_connect on v2:172.21.15.32:6833/2017655932 2024-08-03T21:00:01.934 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:01.931+0000 7f360fe00640 1 -- 172.21.15.32:0/3137142714 <== mds.0 v2:172.21.15.32:6833/2017655932 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55d1b92509b0 con 0x55d1b8ef4e40 2024-08-03T21:00:01.934 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:01.931+0000 7f360fe00640 10 client.10101 handle_command_reply: tid=0 2024-08-03T21:00:01.935 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:01.931+0000 7f3638fa8740 1 client.10101 shutdown 2024-08-03T21:00:01.935 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:01.931+0000 7f3638fa8740 1 -- 172.21.15.32:0/3137142714 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55d1b8f36270 msgr2=0x55d1b931d5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:00:01.935 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:01.931+0000 7f3638fa8740 1 --2- 172.21.15.32:0/3137142714 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55d1b8f36270 0x55d1b931d5e0 secure :-1 s=READY pgs=2006 cs=0 l=1 rev1=1 crypto rx=0x7f362400e7e0 tx=0x7f362400eca0 comp rx=0 tx=0).stop 2024-08-03T21:00:01.935 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:01.931+0000 7f3638fa8740 1 -- 172.21.15.32:0/3137142714 shutdown_connections 2024-08-03T21:00:01.935 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:01.931+0000 7f3638fa8740 1 --2- 172.21.15.32:0/3137142714 >> [v2:172.21.15.32:6833/2017655932,v1:172.21.15.32:6835/2017655932] conn(0x55d1b8ef4e40 0x55d1b93304c0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:01.935 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:01.931+0000 7f3638fa8740 1 --2- 172.21.15.32:0/3137142714 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55d1b8f36270 0x55d1b931d5e0 unknown :-1 s=CLOSED pgs=2006 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:01.935 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:01.931+0000 7f3638fa8740 1 --2- 172.21.15.32:0/3137142714 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55d1b8f15590 0x55d1b9303780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:01.935 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:01.931+0000 7f3638fa8740 1 --2- 172.21.15.32:0/3137142714 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55d1b8f0d3b0 0x55d1b9305180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:01.935 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:01.931+0000 7f360fe00640 0 client.10101 ms_handle_reset on v2:172.21.15.32:6833/2017655932 2024-08-03T21:00:01.935 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:01.931+0000 7f3638fa8740 1 -- 172.21.15.32:0/3137142714 >> 172.21.15.32:0/3137142714 conn(0x55d1b8f2a3c0 msgr2=0x55d1b9317560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T21:00:01.935 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:01.932+0000 7f3638fa8740 1 -- 172.21.15.32:0/3137142714 shutdown_connections 2024-08-03T21:00:01.935 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:01.932+0000 7f3638fa8740 1 -- 172.21.15.32:0/3137142714 wait complete. 2024-08-03T21:00:01.935 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:01.932+0000 7f3638fa8740 20 client.10101 trim_cache size 0 max 16384 2024-08-03T21:00:01.953 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:01.952+0000 7f3638fa8740 1 Processor -- start 2024-08-03T21:00:01.953 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:01.952+0000 7f3638fa8740 1 -- start start 2024-08-03T21:00:01.955 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:01.953+0000 7f3638fa8740 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55d1b8f0d3b0 0x55d1b93441d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:00:01.955 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:01.953+0000 7f3638fa8740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55d1b8f15590 0x55d1b92f5c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:00:01.955 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:01.953+0000 7f3638fa8740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55d1b8f36270 0x55d1b92f61d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:00:01.955 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:01.953+0000 7f3638fa8740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f362c202be0 con 0x55d1b8f36270 2024-08-03T21:00:01.955 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:01.953+0000 7f3638fa8740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f362c201f40 con 0x55d1b8f15590 2024-08-03T21:00:01.956 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:01.953+0000 7f3638fa8740 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7f362c202590 con 0x55d1b8f0d3b0 2024-08-03T21:00:01.956 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:01.953+0000 7f362b400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55d1b8f36270 0x55d1b92f61d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:00:01.956 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:01.953+0000 7f362b400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55d1b8f36270 0x55d1b92f61d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:39598/0 (socket says 172.21.15.32:39598) 2024-08-03T21:00:01.956 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:01.953+0000 7f362b400640 1 -- 172.21.15.32:0/3495497273 learned_addr learned my addr 172.21.15.32:0/3495497273 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T21:00:01.956 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:01.953+0000 7f362aa00640 1 --2- 172.21.15.32:0/3495497273 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55d1b8f0d3b0 0x55d1b93441d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:00:01.956 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:01.954+0000 7f3623e00640 1 --2- 172.21.15.32:0/3495497273 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55d1b8f15590 0x55d1b92f5c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:00:01.956 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:01.954+0000 7f362b400640 1 -- 172.21.15.32:0/3495497273 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55d1b8f0d3b0 msgr2=0x55d1b93441d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T21:00:01.956 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:01.954+0000 7f362b400640 1 --2- 172.21.15.32:0/3495497273 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55d1b8f0d3b0 0x55d1b93441d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:01.956 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:01.954+0000 7f362b400640 1 -- 172.21.15.32:0/3495497273 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55d1b8f15590 msgr2=0x55d1b92f5c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T21:00:01.956 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:01.954+0000 7f362b400640 1 --2- 172.21.15.32:0/3495497273 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55d1b8f15590 0x55d1b92f5c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:01.956 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:01.954+0000 7f362b400640 1 -- 172.21.15.32:0/3495497273 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d1b9322960 con 0x55d1b8f36270 2024-08-03T21:00:01.956 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:01.954+0000 7f362b400640 1 --2- 172.21.15.32:0/3495497273 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55d1b8f36270 0x55d1b92f61d0 secure :-1 s=READY pgs=2007 cs=0 l=1 rev1=1 crypto rx=0x7f362400ce50 tx=0x7f362400e760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ebc0549e10e913a1 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T21:00:01.958 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:01.954+0000 7f361aa00640 1 -- 172.21.15.32:0/3495497273 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f362400c4b0 con 0x55d1b8f36270 2024-08-03T21:00:01.958 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:01.954+0000 7f361aa00640 1 -- 172.21.15.32:0/3495497273 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f362401a070 con 0x55d1b8f36270 2024-08-03T21:00:01.958 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:01.954+0000 7f361aa00640 1 -- 172.21.15.32:0/3495497273 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f362400caf0 con 0x55d1b8f36270 2024-08-03T21:00:01.958 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:01.955+0000 7f3638fa8740 1 -- 172.21.15.32:0/3495497273 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55d1b8f36270 msgr2=0x55d1b92f61d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:00:01.958 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:01.955+0000 7f3638fa8740 1 --2- 172.21.15.32:0/3495497273 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55d1b8f36270 0x55d1b92f61d0 secure :-1 s=READY pgs=2007 cs=0 l=1 rev1=1 crypto rx=0x7f362400ce50 tx=0x7f362400e760 comp rx=0 tx=0).stop 2024-08-03T21:00:01.958 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:01.955+0000 7f3638fa8740 1 -- 172.21.15.32:0/3495497273 shutdown_connections 2024-08-03T21:00:01.958 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:01.955+0000 7f3638fa8740 1 --2- 172.21.15.32:0/3495497273 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55d1b8f15590 0x55d1b92f5c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:01.958 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:01.955+0000 7f3638fa8740 1 --2- 172.21.15.32:0/3495497273 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55d1b8f36270 0x55d1b92f61d0 unknown :-1 s=CLOSED pgs=2007 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:01.958 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:01.955+0000 7f3638fa8740 1 --2- 172.21.15.32:0/3495497273 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55d1b8f0d3b0 0x55d1b93441d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:01.958 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:01.955+0000 7f3638fa8740 1 -- 172.21.15.32:0/3495497273 >> 172.21.15.32:0/3495497273 conn(0x55d1b8f2a3c0 msgr2=0x55d1b9317560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T21:00:01.958 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:01.955+0000 7f3638fa8740 1 -- 172.21.15.32:0/3495497273 shutdown_connections 2024-08-03T21:00:01.958 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:01.955+0000 7f3638fa8740 1 -- 172.21.15.32:0/3495497273 wait complete. 2024-08-03T21:00:01.958 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:01.956+0000 7f3638fa8740 1 -- start start 2024-08-03T21:00:01.959 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:01.956+0000 7f3638fa8740 1 Processor -- start 2024-08-03T21:00:01.961 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:01.958+0000 7f3638fa8740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55d1b8f0d3b0 0x55d1b92dc1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:00:01.961 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:01.958+0000 7f3638fa8740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55d1b8f15590 0x55d1b92f5e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:00:01.961 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:01.958+0000 7f3638fa8740 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55d1b8f36270 0x55d1b9397670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:00:01.962 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:01.958+0000 7f3638fa8740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55d1b9333e40 con 0x55d1b8f15590 2024-08-03T21:00:01.962 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:01.958+0000 7f3638fa8740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x55d1b93339a0 con 0x55d1b8f0d3b0 2024-08-03T21:00:01.962 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:01.958+0000 7f3638fa8740 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x55d1b92e7b60 con 0x55d1b8f36270 2024-08-03T21:00:01.962 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:01.958+0000 7f3638fa8740 10 client.0 _ll_register_callbacks cb 0x7f362400e390 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-03T21:00:01.962 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:01.958+0000 7f3623e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55d1b8f15590 0x55d1b92f5e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:00:01.962 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:01.958+0000 7f3623e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55d1b8f15590 0x55d1b92f5e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:39612/0 (socket says 172.21.15.32:39612) 2024-08-03T21:00:01.962 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:01.958+0000 7f3623e00640 1 -- 172.21.15.32:0/1853160680 learned_addr learned my addr 172.21.15.32:0/1853160680 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T21:00:01.962 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:01.958+0000 7f362aa00640 1 --2- 172.21.15.32:0/1853160680 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55d1b8f0d3b0 0x55d1b92dc1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:00:01.962 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:01.958+0000 7f362b400640 1 --2- 172.21.15.32:0/1853160680 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55d1b8f36270 0x55d1b9397670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:00:01.962 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:01.958+0000 7f3623e00640 1 -- 172.21.15.32:0/1853160680 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55d1b8f36270 msgr2=0x55d1b9397670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T21:00:01.962 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:01.958+0000 7f3623e00640 1 --2- 172.21.15.32:0/1853160680 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55d1b8f36270 0x55d1b9397670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:01.962 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:01.958+0000 7f3623e00640 1 -- 172.21.15.32:0/1853160680 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55d1b8f0d3b0 msgr2=0x55d1b92dc1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T21:00:01.962 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:01.958+0000 7f3623e00640 1 --2- 172.21.15.32:0/1853160680 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55d1b8f0d3b0 0x55d1b92dc1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:01.963 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:01.958+0000 7f3623e00640 1 -- 172.21.15.32:0/1853160680 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d1b93a2140 con 0x55d1b8f15590 2024-08-03T21:00:01.963 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:01.958+0000 7f362b400640 1 --2- 172.21.15.32:0/1853160680 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55d1b8f36270 0x55d1b9397670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-03T21:00:01.963 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:01.958+0000 7f362aa00640 1 --2- 172.21.15.32:0/1853160680 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55d1b8f0d3b0 0x55d1b92dc1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-03T21:00:01.963 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:01.959+0000 7f3623e00640 1 --2- 172.21.15.32:0/1853160680 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55d1b8f15590 0x55d1b92f5e80 secure :-1 s=READY pgs=2008 cs=0 l=1 rev1=1 crypto rx=0x7f3614061b00 tx=0x7f36140623e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f97a57a6383b5bab server_cookie=0 in_seq=0 out_seq=0 2024-08-03T21:00:01.963 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:01.959+0000 7f360fe00640 10 client.0 ms_handle_connect on v2:172.21.15.32:3300/0 2024-08-03T21:00:01.963 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:01.959+0000 7f360fe00640 1 -- 172.21.15.32:0/1853160680 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3614095060 con 0x55d1b8f15590 2024-08-03T21:00:01.963 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:01.959+0000 7f360fe00640 1 -- 172.21.15.32:0/1853160680 <== mon.0 v2:172.21.15.32:3300/0 2 ==== osd_map(121..121 src has 1..121) ==== 8950+0+0 (secure 0 0 0) 0x7f361407b210 con 0x55d1b8f15590 2024-08-03T21:00:01.963 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:01.959+0000 7f3638fa8740 1 -- 172.21.15.32:0/1853160680 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f362401a070 con 0x55d1b8f15590 2024-08-03T21:00:01.963 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:01.959+0000 7f360fe00640 1 -- 172.21.15.32:0/1853160680 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_get_version_reply(handle=1 version=308) ==== 24+0+0 (secure 0 0 0) 0x7f3614080050 con 0x55d1b8f15590 2024-08-03T21:00:01.963 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:01.959+0000 7f3638fa8740 10 client.10107 fetch_fsmap learned FSMap version 308 2024-08-03T21:00:01.963 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:01.959+0000 7f3638fa8740 1 -- 172.21.15.32:0/1853160680 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,fsmap=308,monmap=2+}) -- 0x55d1b931bb10 con 0x55d1b8f15590 2024-08-03T21:00:01.963 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:01.959+0000 7f360fe00640 1 -- 172.21.15.32:0/1853160680 <== mon.0 v2:172.21.15.32:3300/0 4 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f362401a070 con 0x55d1b8f15590 2024-08-03T21:00:01.963 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:01.960+0000 7f360fe00640 1 -- 172.21.15.32:0/1853160680 <== mon.0 v2:172.21.15.32:3300/0 5 ==== fsmap(e 308) ==== 3502+0+0 (secure 0 0 0) 0x7f361407b410 con 0x55d1b8f15590 2024-08-03T21:00:01.963 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:01.960+0000 7f3638fa8740 10 client.10107 fetch_fsmap finished waiting for FSMap version 308 2024-08-03T21:00:01.964 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:01.960+0000 7f3638fa8740 10 client.10107 resolve_mds: resolved name 'a' to daemon mds.a 2024-08-03T21:00:01.964 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:01.960+0000 7f3638fa8740 20 client.10107 populate_metadata read hostname 'smithi032' 2024-08-03T21:00:01.964 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:01.960+0000 7f3638fa8740 1 --2- 172.21.15.32:0/1853160680 >> [v2:172.21.15.32:6833/2017655932,v1:172.21.15.32:6835/2017655932] conn(0x55d1b939e4e0 0x55d1b92d56a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:00:01.964 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:01.960+0000 7f3638fa8740 4 client.10107 mds_command: new command op to 10035 tid=0 multi_id=0 [{"prefix": "status"}] 2024-08-03T21:00:01.964 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:01.960+0000 7f3638fa8740 1 -- 172.21.15.32:0/1853160680 --> [v2:172.21.15.32:6833/2017655932,v1:172.21.15.32:6835/2017655932] -- command(tid 0: {"prefix": "status"}) -- 0x55d1b924c450 con 0x55d1b939e4e0 2024-08-03T21:00:01.964 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:01.960+0000 7f362aa00640 1 --2- 172.21.15.32:0/1853160680 >> [v2:172.21.15.32:6833/2017655932,v1:172.21.15.32:6835/2017655932] conn(0x55d1b939e4e0 0x55d1b92d56a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:00:01.965 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:01.960+0000 7f362aa00640 1 --2- 172.21.15.32:0/1853160680 >> [v2:172.21.15.32:6833/2017655932,v1:172.21.15.32:6835/2017655932] conn(0x55d1b939e4e0 0x55d1b92d56a0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3b98a214d5105980 server_cookie=2fa7d4c32863fd8c in_seq=0 out_seq=0 2024-08-03T21:00:01.965 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:01.960+0000 7f360fe00640 10 client.10107 ms_handle_connect on v2:172.21.15.32:6833/2017655932 2024-08-03T21:00:01.965 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:01.961+0000 7f360fe00640 1 -- 172.21.15.32:0/1853160680 <== mds.0 v2:172.21.15.32:6833/2017655932 1 ==== command_reply(tid 0: 0 ) ==== 8+0+335 (crc 0 0 0) 0x55d1b924c450 con 0x55d1b939e4e0 2024-08-03T21:00:01.965 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:01.961+0000 7f360fe00640 10 client.10107 handle_command_reply: tid=0 2024-08-03T21:00:01.965 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:01.961+0000 7f3638fa8740 1 client.10107 shutdown 2024-08-03T21:00:01.965 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:01.961+0000 7f3638fa8740 1 -- 172.21.15.32:0/1853160680 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55d1b8f15590 msgr2=0x55d1b92f5e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:00:01.966 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:01.961+0000 7f3638fa8740 1 --2- 172.21.15.32:0/1853160680 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55d1b8f15590 0x55d1b92f5e80 secure :-1 s=READY pgs=2008 cs=0 l=1 rev1=1 crypto rx=0x7f3614061b00 tx=0x7f36140623e0 comp rx=0 tx=0).stop 2024-08-03T21:00:01.966 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:01.961+0000 7f3638fa8740 1 -- 172.21.15.32:0/1853160680 shutdown_connections 2024-08-03T21:00:01.966 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:01.961+0000 7f3638fa8740 1 --2- 172.21.15.32:0/1853160680 >> [v2:172.21.15.32:6833/2017655932,v1:172.21.15.32:6835/2017655932] conn(0x55d1b939e4e0 0x55d1b92d56a0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:01.966 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:01.961+0000 7f3638fa8740 1 --2- 172.21.15.32:0/1853160680 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55d1b8f36270 0x55d1b9397670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:01.966 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:01.961+0000 7f3638fa8740 1 --2- 172.21.15.32:0/1853160680 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55d1b8f15590 0x55d1b92f5e80 unknown :-1 s=CLOSED pgs=2008 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:01.966 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:01.961+0000 7f3638fa8740 1 --2- 172.21.15.32:0/1853160680 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55d1b8f0d3b0 0x55d1b92dc1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:01.966 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:01.961+0000 7f3638fa8740 1 -- 172.21.15.32:0/1853160680 >> 172.21.15.32:0/1853160680 conn(0x55d1b8f2a3c0 msgr2=0x55d1b9317560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T21:00:01.966 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:01.961+0000 7f360fe00640 0 client.10107 ms_handle_reset on v2:172.21.15.32:6833/2017655932 2024-08-03T21:00:01.966 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:01.961+0000 7f3638fa8740 1 -- 172.21.15.32:0/1853160680 shutdown_connections 2024-08-03T21:00:01.966 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:01.961+0000 7f3638fa8740 1 -- 172.21.15.32:0/1853160680 wait complete. 2024-08-03T21:00:01.966 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:01.961+0000 7f3638fa8740 20 client.10107 trim_cache size 0 max 16384 2024-08-03T21:00:01.966 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-08-03T21:00:01.967 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_fsid": "40ab141a-01d7-49b9-ae3c-2edec9876e95", 2024-08-03T21:00:01.967 INFO:teuthology.orchestra.run.smithi032.stdout: "whoami": 0, 2024-08-03T21:00:01.967 INFO:teuthology.orchestra.run.smithi032.stdout: "id": 10035, 2024-08-03T21:00:01.967 INFO:teuthology.orchestra.run.smithi032.stdout: "want_state": "up:active", 2024-08-03T21:00:01.967 INFO:teuthology.orchestra.run.smithi032.stdout: "state": "up:active", 2024-08-03T21:00:01.967 INFO:teuthology.orchestra.run.smithi032.stdout: "fs_name": "cephfs", 2024-08-03T21:00:01.967 INFO:teuthology.orchestra.run.smithi032.stdout: "rank_uptime": 7.1661902209999999, 2024-08-03T21:00:01.967 INFO:teuthology.orchestra.run.smithi032.stdout: "mdsmap_epoch": 308, 2024-08-03T21:00:01.967 INFO:teuthology.orchestra.run.smithi032.stdout: "osdmap_epoch": 121, 2024-08-03T21:00:01.967 INFO:teuthology.orchestra.run.smithi032.stdout: "osdmap_epoch_barrier": 121, 2024-08-03T21:00:01.967 INFO:teuthology.orchestra.run.smithi032.stdout: "uptime": 10.328621677999999 2024-08-03T21:00:01.967 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-08-03T21:00:01.968 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:01.962+0000 7f362be00640 1 -- 172.21.15.32:0/133691314 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f3608077010 msgr2=0x7f3608079520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:00:01.968 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:01.962+0000 7f362be00640 1 --2- 172.21.15.32:0/133691314 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f3608077010 0x7f3608079520 secure :-1 s=READY pgs=803 cs=0 l=1 rev1=1 crypto rx=0x7f361c034000 tx=0x7f361c004730 comp rx=0 tx=0).stop 2024-08-03T21:00:01.968 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:01.962+0000 7f362be00640 1 -- 172.21.15.32:0/133691314 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f362c0acd80 msgr2=0x7f362c1f5ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:00:01.968 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:01.962+0000 7f362be00640 1 --2- 172.21.15.32:0/133691314 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f362c0acd80 0x7f362c1f5ee0 secure :-1 s=READY pgs=2004 cs=0 l=1 rev1=1 crypto rx=0x7f361400e790 tx=0x7f361400ec50 comp rx=0 tx=0).stop 2024-08-03T21:00:01.968 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:01.962+0000 7f362be00640 1 -- 172.21.15.32:0/133691314 shutdown_connections 2024-08-03T21:00:01.968 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:01.962+0000 7f362be00640 1 --2- 172.21.15.32:0/133691314 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f3608077010 0x7f3608079520 unknown :-1 s=CLOSED pgs=803 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:01.968 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:01.962+0000 7f362be00640 1 --2- 172.21.15.32:0/133691314 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f362c0af020 0x7f362c15ea80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:01.969 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:01.962+0000 7f362be00640 1 --2- 172.21.15.32:0/133691314 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f362c0acd80 0x7f362c1f5ee0 unknown :-1 s=CLOSED pgs=2004 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:01.969 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:01.962+0000 7f362be00640 1 --2- 172.21.15.32:0/133691314 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f362c06f440 0x7f362c1f59a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:01.969 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:01.962+0000 7f362be00640 1 -- 172.21.15.32:0/133691314 >> 172.21.15.32:0/133691314 conn(0x7f362c0b2120 msgr2=0x7f362c15bbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T21:00:01.969 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:01.962+0000 7f362be00640 1 -- 172.21.15.32:0/133691314 shutdown_connections 2024-08-03T21:00:01.969 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:01.962+0000 7f362be00640 1 -- 172.21.15.32:0/133691314 wait complete. 2024-08-03T21:00:01.985 DEBUG:tasks.ceph_test_case:wait_until_true: success in 5s and 0 retries 2024-08-03T21:00:01.985 INFO:tasks.cephfs.test_damage:Daemons came up after mutation 'Corrupt omap header on mds0_openfiles.0', proceeding to ls 2024-08-03T21:00:01.986 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-08-03T21:00:02.214 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:02.212+0000 7f17e6400640 1 -- 172.21.15.32:0/2931322786 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f17c4004800 msgr2=0x7f17c4004c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:00:02.214 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:02.212+0000 7f17e6400640 1 --2- 172.21.15.32:0/2931322786 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f17c4004800 0x7f17c4004c10 secure :-1 s=READY pgs=2010 cs=0 l=1 rev1=1 crypto rx=0x7f17c80017d0 tx=0x7f17c4005830 comp rx=0 tx=0).stop 2024-08-03T21:00:02.214 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:02.212+0000 7f17dfe00640 1 -- 172.21.15.32:0/2931322786 reap_dead start 2024-08-03T21:00:02.215 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:02.212+0000 7f17e6400640 1 -- 172.21.15.32:0/2931322786 shutdown_connections 2024-08-03T21:00:02.215 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:02.212+0000 7f17e6400640 1 -- 172.21.15.32:0/2931322786 >> 172.21.15.32:0/2931322786 conn(0x7f17e00b1bd0 msgr2=0x7f17e00b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T21:00:02.215 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:02.213+0000 7f17e6400640 1 -- 172.21.15.32:0/2931322786 shutdown_connections 2024-08-03T21:00:02.215 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:02.213+0000 7f17e6400640 1 -- 172.21.15.32:0/2931322786 wait complete. 2024-08-03T21:00:02.216 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:02.215+0000 7f17e6400640 1 Processor -- start 2024-08-03T21:00:02.216 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:02.215+0000 7f17e6400640 1 -- start start 2024-08-03T21:00:02.218 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:02.216+0000 7f17e6400640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f17c4004800 0x7f17e01f7a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:00:02.218 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:02.216+0000 7f17e6400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f17e00adca0 0x7f17e00ae0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:00:02.218 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:02.216+0000 7f17e6400640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f17e00ae5f0 0x7f17e01f82c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:00:02.218 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:02.216+0000 7f17e6400640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f17e00cf9c0 con 0x7f17e00adca0 2024-08-03T21:00:02.219 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:02.216+0000 7f17e6400640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f17e00cf840 con 0x7f17e00ae5f0 2024-08-03T21:00:02.219 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:02.216+0000 7f17e6400640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7f17e0160b10 con 0x7f17c4004800 2024-08-03T21:00:02.219 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:02.217+0000 7f17d7e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f17e00adca0 0x7f17e00ae0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:00:02.219 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:02.217+0000 7f17df400640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f17c4004800 0x7f17e01f7a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:00:02.219 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:02.217+0000 7f17d7e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f17e00adca0 0x7f17e00ae0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:39634/0 (socket says 172.21.15.32:39634) 2024-08-03T21:00:02.219 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:02.217+0000 7f17df400640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f17c4004800 0x7f17e01f7a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3301/0 says I am v2:172.21.15.32:51342/0 (socket says 172.21.15.32:51342) 2024-08-03T21:00:02.219 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:02.217+0000 7f17d7e00640 1 -- 172.21.15.32:0/3997570642 learned_addr learned my addr 172.21.15.32:0/3997570642 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T21:00:02.219 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:02.217+0000 7f17dfe00640 1 --2- 172.21.15.32:0/3997570642 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f17e00ae5f0 0x7f17e01f82c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:00:02.220 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:02.217+0000 7f17d7e00640 1 -- 172.21.15.32:0/3997570642 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f17c4004800 msgr2=0x7f17e01f7a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:00:02.220 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:02.217+0000 7f17d7e00640 1 --2- 172.21.15.32:0/3997570642 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f17c4004800 0x7f17e01f7a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:02.221 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:02.217+0000 7f17d7e00640 1 -- 172.21.15.32:0/3997570642 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f17e00ae5f0 msgr2=0x7f17e01f82c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:00:02.221 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:02.217+0000 7f17d7e00640 1 --2- 172.21.15.32:0/3997570642 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f17e00ae5f0 0x7f17e01f82c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:02.221 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:02.217+0000 7f17d7e00640 1 -- 172.21.15.32:0/3997570642 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f17e01541f0 con 0x7f17e00adca0 2024-08-03T21:00:02.221 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:02.218+0000 7f17d7e00640 1 --2- 172.21.15.32:0/3997570642 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f17e00adca0 0x7f17e00ae0b0 secure :-1 s=READY pgs=2011 cs=0 l=1 rev1=1 crypto rx=0x7f17c80018c0 tx=0x7f17c80060e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T21:00:02.221 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:02.218+0000 7f17d7400640 1 -- 172.21.15.32:0/3997570642 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17c8043070 con 0x7f17e00adca0 2024-08-03T21:00:02.221 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:02.218+0000 7f17d7400640 1 -- 172.21.15.32:0/3997570642 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f17c8003af0 con 0x7f17e00adca0 2024-08-03T21:00:02.221 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:02.218+0000 7f17e6400640 1 -- 172.21.15.32:0/3997570642 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f17e01544d0 con 0x7f17e00adca0 2024-08-03T21:00:02.221 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:02.218+0000 7f17d7400640 1 -- 172.21.15.32:0/3997570642 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17c80064c0 con 0x7f17e00adca0 2024-08-03T21:00:02.221 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:02.218+0000 7f17e6400640 1 -- 172.21.15.32:0/3997570642 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f17e006fd10 con 0x7f17e00adca0 2024-08-03T21:00:02.222 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:02.219+0000 7f17e6400640 1 -- 172.21.15.32:0/3997570642 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f17e0144f90 con 0x7f17e00adca0 2024-08-03T21:00:02.222 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:02.220+0000 7f17d7400640 1 -- 172.21.15.32:0/3997570642 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f17c8033480 con 0x7f17e00adca0 2024-08-03T21:00:02.223 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:02.221+0000 7f17d7400640 1 --2- 172.21.15.32:0/3997570642 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f17bc077010 0x7f17bc079520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:00:02.223 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:02.221+0000 7f17d7400640 1 -- 172.21.15.32:0/3997570642 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(121..121 src has 1..121) ==== 8950+0+0 (secure 0 0 0) 0x7f17c80b94d0 con 0x7f17e00adca0 2024-08-03T21:00:02.223 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:02.222+0000 7f17df400640 1 --2- 172.21.15.32:0/3997570642 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f17bc077010 0x7f17bc079520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:00:02.224 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:02.223+0000 7f17df400640 1 --2- 172.21.15.32:0/3997570642 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f17bc077010 0x7f17bc079520 secure :-1 s=READY pgs=804 cs=0 l=1 rev1=1 crypto rx=0x7f17d0005eb0 tx=0x7f17d0005f90 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T21:00:02.227 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:02.226+0000 7f17d7400640 1 -- 172.21.15.32:0/3997570642 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7f17c80813f0 con 0x7f17e00adca0 2024-08-03T21:00:02.478 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:02.476+0000 7f17e6400640 1 -- 172.21.15.32:0/3997570642 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f17e00ad8c0 con 0x7f17e00adca0 2024-08-03T21:00:02.479 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:02.477+0000 7f17d7400640 1 -- 172.21.15.32:0/3997570642 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v308) ==== 53+0+73 (secure 0 0 0) 0x7f17c8085940 con 0x7f17e00adca0 2024-08-03T21:00:02.479 INFO:teuthology.orchestra.run.smithi032.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-08-03T21:00:02.481 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:02.479+0000 7f17e6400640 1 -- 172.21.15.32:0/3997570642 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f17bc077010 msgr2=0x7f17bc079520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:00:02.481 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:02.479+0000 7f17e6400640 1 --2- 172.21.15.32:0/3997570642 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f17bc077010 0x7f17bc079520 secure :-1 s=READY pgs=804 cs=0 l=1 rev1=1 crypto rx=0x7f17d0005eb0 tx=0x7f17d0005f90 comp rx=0 tx=0).stop 2024-08-03T21:00:02.481 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:02.479+0000 7f17e6400640 1 -- 172.21.15.32:0/3997570642 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f17e00adca0 msgr2=0x7f17e00ae0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:00:02.482 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:02.479+0000 7f17e6400640 1 --2- 172.21.15.32:0/3997570642 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f17e00adca0 0x7f17e00ae0b0 secure :-1 s=READY pgs=2011 cs=0 l=1 rev1=1 crypto rx=0x7f17c80018c0 tx=0x7f17c80060e0 comp rx=0 tx=0).stop 2024-08-03T21:00:02.482 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:02.480+0000 7f17e6400640 1 -- 172.21.15.32:0/3997570642 shutdown_connections 2024-08-03T21:00:02.482 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:02.480+0000 7f17e6400640 1 --2- 172.21.15.32:0/3997570642 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f17bc077010 0x7f17bc079520 unknown :-1 s=CLOSED pgs=804 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:02.482 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:02.480+0000 7f17e6400640 1 --2- 172.21.15.32:0/3997570642 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f17e00ae5f0 0x7f17e01f82c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:02.482 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:02.480+0000 7f17e6400640 1 --2- 172.21.15.32:0/3997570642 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f17e00adca0 0x7f17e00ae0b0 unknown :-1 s=CLOSED pgs=2011 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:02.482 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:02.480+0000 7f17e6400640 1 --2- 172.21.15.32:0/3997570642 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f17c4004800 0x7f17e01f7a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:02.482 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:02.480+0000 7f17e6400640 1 -- 172.21.15.32:0/3997570642 >> 172.21.15.32:0/3997570642 conn(0x7f17e00b1bd0 msgr2=0x7f17e00b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T21:00:02.482 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:02.480+0000 7f17e6400640 1 -- 172.21.15.32:0/3997570642 shutdown_connections 2024-08-03T21:00:02.482 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:02.480+0000 7f17e6400640 1 -- 172.21.15.32:0/3997570642 wait complete. 2024-08-03T21:00:02.503 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-08-03T21:00:02.503 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-08-03T21:00:02.503 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi032.front.sepia.ceph.com 2024-08-03T21:00:02.503 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-08-03T21:00:02.503 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-08-03T21:00:02.503 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-08-03T21:00:02.503 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2024-08-03T21:00:02.503 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-08-03T21:00:02.503 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-08-03T21:00:02.504 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-08-03T21:00:02.504 DEBUG:teuthology.orchestra.run.smithi032:> ip addr 2024-08-03T21:00:02.539 INFO:teuthology.orchestra.run.smithi032.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-08-03T21:00:02.540 INFO:teuthology.orchestra.run.smithi032.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-08-03T21:00:02.540 INFO:teuthology.orchestra.run.smithi032.stdout: inet 127.0.0.1/8 scope host lo 2024-08-03T21:00:02.540 INFO:teuthology.orchestra.run.smithi032.stdout: valid_lft forever preferred_lft forever 2024-08-03T21:00:02.540 INFO:teuthology.orchestra.run.smithi032.stdout: inet6 ::1/128 scope host 2024-08-03T21:00:02.540 INFO:teuthology.orchestra.run.smithi032.stdout: valid_lft forever preferred_lft forever 2024-08-03T21:00:02.540 INFO:teuthology.orchestra.run.smithi032.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-08-03T21:00:02.540 INFO:teuthology.orchestra.run.smithi032.stdout: link/ether 0c:c4:7a:6b:fd:0c brd ff:ff:ff:ff:ff:ff 2024-08-03T21:00:02.540 INFO:teuthology.orchestra.run.smithi032.stdout: altname enp7s0f0 2024-08-03T21:00:02.540 INFO:teuthology.orchestra.run.smithi032.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-08-03T21:00:02.540 INFO:teuthology.orchestra.run.smithi032.stdout: link/ether 0c:c4:7a:6b:fd:0d brd ff:ff:ff:ff:ff:ff 2024-08-03T21:00:02.540 INFO:teuthology.orchestra.run.smithi032.stdout: altname enp7s0f1 2024-08-03T21:00:02.540 INFO:teuthology.orchestra.run.smithi032.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-08-03T21:00:02.540 INFO:teuthology.orchestra.run.smithi032.stdout: link/ether ac:1f:6b:f8:26:02 brd ff:ff:ff:ff:ff:ff 2024-08-03T21:00:02.540 INFO:teuthology.orchestra.run.smithi032.stdout: altname enp1s0f0 2024-08-03T21:00:02.541 INFO:teuthology.orchestra.run.smithi032.stdout: inet 172.21.15.32/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-08-03T21:00:02.541 INFO:teuthology.orchestra.run.smithi032.stdout: valid_lft 41753sec preferred_lft 41753sec 2024-08-03T21:00:02.541 INFO:teuthology.orchestra.run.smithi032.stdout: inet6 fe80::ae1f:6bff:fef8:2602/64 scope link noprefixroute 2024-08-03T21:00:02.541 INFO:teuthology.orchestra.run.smithi032.stdout: valid_lft forever preferred_lft forever 2024-08-03T21:00:02.541 INFO:teuthology.orchestra.run.smithi032.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-08-03T21:00:02.541 INFO:teuthology.orchestra.run.smithi032.stdout: link/ether ac:1f:6b:f8:26:03 brd ff:ff:ff:ff:ff:ff 2024-08-03T21:00:02.541 INFO:teuthology.orchestra.run.smithi032.stdout: altname enp1s0f1 2024-08-03T21:00:02.541 INFO:teuthology.orchestra.run.smithi032.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-08-03T21:00:02.541 INFO:teuthology.orchestra.run.smithi032.stdout: link/ether 0a:23:2b:ca:2a:93 brd ff:ff:ff:ff:ff:ff 2024-08-03T21:00:02.541 INFO:teuthology.orchestra.run.smithi032.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-08-03T21:00:02.542 INFO:teuthology.orchestra.run.smithi032.stdout: valid_lft forever preferred_lft forever 2024-08-03T21:00:02.542 INFO:teuthology.orchestra.run.smithi032.stdout: inet6 fe80::d804:7cff:fed9:7ba4/64 scope link 2024-08-03T21:00:02.542 INFO:teuthology.orchestra.run.smithi032.stdout: valid_lft forever preferred_lft forever 2024-08-03T21:00:02.542 INFO:teuthology.orchestra.run.smithi032.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-08-03T21:00:02.542 INFO:teuthology.orchestra.run.smithi032.stdout: link/ether 22:86:28:30:4e:62 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-08-03T21:00:02.542 INFO:teuthology.orchestra.run.smithi032.stdout: inet6 fe80::2086:28ff:fe30:4e62/64 scope link 2024-08-03T21:00:02.542 INFO:teuthology.orchestra.run.smithi032.stdout: valid_lft forever preferred_lft forever 2024-08-03T21:00:02.542 INFO:teuthology.orchestra.run.smithi032.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-08-03T21:00:02.542 INFO:teuthology.orchestra.run.smithi032.stdout: link/ether 0a:23:2b:ca:2a:93 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-08-03T21:00:02.542 INFO:teuthology.orchestra.run.smithi032.stdout: inet6 fe80::823:2bff:feca:2a93/64 scope link 2024-08-03T21:00:02.542 INFO:teuthology.orchestra.run.smithi032.stdout: valid_lft forever preferred_lft forever 2024-08-03T21:00:02.543 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-08-03T21:00:02.543 DEBUG:teuthology.orchestra.run.smithi032:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-08-03T21:00:02.632 INFO:teuthology.orchestra.run.smithi032.stdout:1 2024-08-03T21:00:02.636 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-08-03T21:00:02.636 DEBUG:teuthology.orchestra.run.smithi032:> route 2024-08-03T21:00:02.672 INFO:teuthology.orchestra.run.smithi032.stdout:Kernel IP routing table 2024-08-03T21:00:02.672 INFO:teuthology.orchestra.run.smithi032.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-08-03T21:00:02.672 INFO:teuthology.orchestra.run.smithi032.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-08-03T21:00:02.672 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-08-03T21:00:02.672 INFO:teuthology.orchestra.run.smithi032.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-08-03T21:00:02.672 DEBUG:teuthology.orchestra.run.smithi032:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-08-03T21:00:02.673 DEBUG:teuthology.orchestra.run.smithi032:> set -e 2024-08-03T21:00:02.673 DEBUG:teuthology.orchestra.run.smithi032:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-08-03T21:00:02.673 DEBUG:teuthology.orchestra.run.smithi032:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-08-03T21:00:02.673 DEBUG:teuthology.orchestra.run.smithi032:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-08-03T21:00:02.673 DEBUG:teuthology.orchestra.run.smithi032:> ') 2024-08-03T21:00:02.792 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:02 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-08-03T21:00:02.864 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:02 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-08-03T21:00:02.873 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-08-03T21:00:02.873 DEBUG:teuthology.orchestra.run.smithi032:> ip netns list 2024-08-03T21:00:02.908 INFO:teuthology.orchestra.run.smithi032.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-08-03T21:00:02.908 INFO:teuthology.orchestra.run.smithi032.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-08-03T21:00:02.908 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-08-03T21:00:02.908 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-08-03T21:00:02.909 DEBUG:teuthology.orchestra.run.smithi032:> sudo ip link set brx.0 up 2024-08-03T21:00:03.004 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-08-03T21:00:03.004 DEBUG:teuthology.orchestra.run.smithi032:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-08-03T21:00:03.039 INFO:teuthology.orchestra.run.smithi032.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-08-03T21:00:03.039 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-08-03T21:00:03.039 DEBUG:teuthology.orchestra.run.smithi032:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-08-03T21:00:03.114 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2024-08-03T21:00:03.114 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-08-03T21:00:03.114 DEBUG:teuthology.orchestra.run.smithi032:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-08-03T21:00:03.311 INFO:teuthology.orchestra.run.smithi032.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-08-03T21:00:03.311 INFO:teuthology.orchestra.run.smithi032.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-08-03T21:00:03.311 INFO:teuthology.orchestra.run.smithi032.stdout:invalid new device string format 2024-08-03T21:00:03.334 INFO:teuthology.orchestra.run.smithi032.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-08-03T21:00:03.334 INFO:teuthology.orchestra.run.smithi032.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-08-03T21:00:03.334 INFO:teuthology.orchestra.run.smithi032.stdout:invalid new device string format 2024-08-03T21:00:03.334 INFO:teuthology.orchestra.run.smithi032.stdout:mount.ceph: resolved to: "172.21.15.32:3300,172.21.15.118:3300,172.21.15.118:3301" 2024-08-03T21:00:03.334 INFO:teuthology.orchestra.run.smithi032.stdout:mount.ceph: switching to using v1 address with old syntax 2024-08-03T21:00:03.362 INFO:teuthology.orchestra.run.smithi032.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-08-03T21:00:03.362 INFO:teuthology.orchestra.run.smithi032.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-08-03T21:00:03.362 INFO:teuthology.orchestra.run.smithi032.stdout:invalid new device string format 2024-08-03T21:00:03.362 INFO:teuthology.orchestra.run.smithi032.stdout:mount.ceph: resolved to: "172.21.15.32:3300,172.21.15.118:3300,172.21.15.118:3301" 2024-08-03T21:00:03.362 INFO:teuthology.orchestra.run.smithi032.stdout:mount.ceph: switching to using v1 address with old syntax 2024-08-03T21:00:03.362 INFO:teuthology.orchestra.run.smithi032.stdout:mount.ceph: resolved to: "172.21.15.32:6789,172.21.15.118:6789,172.21.15.118:6790" 2024-08-03T21:00:03.362 INFO:teuthology.orchestra.run.smithi032.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.32:6789,172.21.15.118:6789,172.21.15.118:6790:/ 2024-08-03T21:00:03.363 INFO:teuthology.orchestra.run.smithi032.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,key=0,fsid=40ab141a-01d7-49b9-ae3c-2edec9876e95" will pass to kernel 2024-08-03T21:00:03.364 INFO:teuthology.orchestra.run.smithi032.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2024-08-03T21:00:03.364 INFO:teuthology.orchestra.run.smithi032.stdout: You just mounted a file system that supports labels which does not 2024-08-03T21:00:03.364 INFO:teuthology.orchestra.run.smithi032.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-08-03T21:00:03.364 INFO:teuthology.orchestra.run.smithi032.stdout: applications will generate AVC messages and not be allowed access to 2024-08-03T21:00:03.364 INFO:teuthology.orchestra.run.smithi032.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-08-03T21:00:03.367 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-08-03T21:00:03.367 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-08-03T21:00:03.367 DEBUG:teuthology.orchestra.run.smithi032:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-08-03T21:00:03.423 DEBUG:teuthology.orchestra.run.smithi032:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-08-03T21:00:03.506 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:03 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-08-03T21:00:03.509 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:03 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-08-03T21:00:03.517 INFO:teuthology.orchestra.run.smithi032.stdout:client10116 2024-08-03T21:00:03.517 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-08-03T21:00:03.517 DEBUG:teuthology.orchestra.run.smithi032:> dd if=/proc/self/mounts of=/dev/stdout 2024-08-03T21:00:03.553 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-08-03T21:00:03.553 DEBUG:teuthology.orchestra.run.smithi032:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls -R .) 2024-08-03T21:00:03.598 INFO:teuthology.orchestra.run:waiting for 20 2024-08-03T21:00:03.631 INFO:teuthology.orchestra.run.smithi032.stdout:.: 2024-08-03T21:00:03.631 INFO:teuthology.orchestra.run.smithi032.stdout:subdir 2024-08-03T21:00:03.631 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-03T21:00:03.631 INFO:teuthology.orchestra.run.smithi032.stdout:./subdir: 2024-08-03T21:00:03.631 INFO:teuthology.orchestra.run.smithi032.stdout:sixmegs 2024-08-03T21:00:03.632 INFO:tasks.cephfs.test_damage:Result: Mutation 'Corrupt omap header on mds0_openfiles.0' did not caused DAMAGED state 2024-08-03T21:00:03.632 INFO:tasks.cephfs.test_damage:Applying mutation 'Corrupt omap header on 100.00000000' 2024-08-03T21:00:03.632 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-08-03T21:00:03.633 DEBUG:teuthology.orchestra.run.smithi032:> dd if=/proc/self/mounts of=/dev/stdout 2024-08-03T21:00:03.709 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-08-03T21:00:03.709 DEBUG:teuthology.orchestra.run.smithi032:> dd if=/proc/self/mounts of=/dev/stdout 2024-08-03T21:00:03.785 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-08-03T21:00:04.057 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:04.055+0000 7f2450a00640 1 -- 172.21.15.32:0/1513472840 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2430004800 msgr2=0x7f2430004c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:00:04.057 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:04.055+0000 7f2450a00640 1 --2- 172.21.15.32:0/1513472840 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2430004800 0x7f2430004c10 secure :-1 s=READY pgs=2014 cs=0 l=1 rev1=1 crypto rx=0x7f244c0703f0 tx=0x7f2430005830 comp rx=0 tx=0).stop 2024-08-03T21:00:04.057 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:04.055+0000 7f244b200640 1 -- 172.21.15.32:0/1513472840 reap_dead start 2024-08-03T21:00:04.057 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:04.055+0000 7f2450a00640 1 -- 172.21.15.32:0/1513472840 shutdown_connections 2024-08-03T21:00:04.057 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:04.055+0000 7f2450a00640 1 -- 172.21.15.32:0/1513472840 >> 172.21.15.32:0/1513472840 conn(0x7f244c0b2120 msgr2=0x7f244c0cfd60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T21:00:04.057 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:04.055+0000 7f2450a00640 1 -- 172.21.15.32:0/1513472840 shutdown_connections 2024-08-03T21:00:04.058 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:04.055+0000 7f2450a00640 1 -- 172.21.15.32:0/1513472840 wait complete. 2024-08-03T21:00:04.059 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:04.058+0000 7f2450a00640 1 Processor -- start 2024-08-03T21:00:04.059 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:04.058+0000 7f2450a00640 1 -- start start 2024-08-03T21:00:04.062 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:04.059+0000 7f2450a00640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f2430004800 0x7f244c1f50b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:00:04.062 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:04.059+0000 7f2450a00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f244c1f90b0 0x7f244c0b4850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:00:04.063 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:04.059+0000 7f2450a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f244c0b0c00 0x7f244c0b1010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:00:04.063 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:04.059+0000 7f2450a00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f244c074920 con 0x7f244c0b0c00 2024-08-03T21:00:04.063 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:04.059+0000 7f2450a00640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f244c0747a0 con 0x7f244c1f90b0 2024-08-03T21:00:04.063 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:04.059+0000 7f2450a00640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7f244c15fe50 con 0x7f2430004800 2024-08-03T21:00:04.063 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:04.060+0000 7f244b200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f244c0b0c00 0x7f244c0b1010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:00:04.063 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:04.060+0000 7f244b200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f244c0b0c00 0x7f244c0b1010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:39644/0 (socket says 172.21.15.32:39644) 2024-08-03T21:00:04.063 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:04.060+0000 7f244b200640 1 -- 172.21.15.32:0/1530032126 learned_addr learned my addr 172.21.15.32:0/1530032126 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T21:00:04.063 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:04.060+0000 7f244a800640 1 --2- 172.21.15.32:0/1530032126 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f2430004800 0x7f244c1f50b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:00:04.063 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:04.060+0000 7f2443e00640 1 --2- 172.21.15.32:0/1530032126 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f244c1f90b0 0x7f244c0b4850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:00:04.063 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:04.060+0000 7f244b200640 1 -- 172.21.15.32:0/1530032126 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f2430004800 msgr2=0x7f244c1f50b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:00:04.063 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:04.060+0000 7f244b200640 1 --2- 172.21.15.32:0/1530032126 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f2430004800 0x7f244c1f50b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:04.063 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:04.060+0000 7f244b200640 1 -- 172.21.15.32:0/1530032126 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f244c1f90b0 msgr2=0x7f244c0b4850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:00:04.063 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:04.060+0000 7f244b200640 1 --2- 172.21.15.32:0/1530032126 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f244c1f90b0 0x7f244c0b4850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:04.063 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:04.060+0000 7f244b200640 1 -- 172.21.15.32:0/1530032126 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f244c147ac0 con 0x7f244c0b0c00 2024-08-03T21:00:04.064 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:04.060+0000 7f244a800640 1 --2- 172.21.15.32:0/1530032126 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f2430004800 0x7f244c1f50b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-03T21:00:04.064 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:04.060+0000 7f2443e00640 1 --2- 172.21.15.32:0/1530032126 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f244c1f90b0 0x7f244c0b4850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-03T21:00:04.064 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:04.060+0000 7f244b200640 1 --2- 172.21.15.32:0/1530032126 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f244c0b0c00 0x7f244c0b1010 secure :-1 s=READY pgs=2015 cs=0 l=1 rev1=1 crypto rx=0x7f2444005fd0 tx=0x7f244400f790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T21:00:04.064 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:04.061+0000 7f2442a00640 1 -- 172.21.15.32:0/1530032126 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2444003d80 con 0x7f244c0b0c00 2024-08-03T21:00:04.064 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:04.061+0000 7f2442a00640 1 -- 172.21.15.32:0/1530032126 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f2444004b20 con 0x7f244c0b0c00 2024-08-03T21:00:04.064 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:04.061+0000 7f2442a00640 1 -- 172.21.15.32:0/1530032126 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2444004030 con 0x7f244c0b0c00 2024-08-03T21:00:04.064 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:04.061+0000 7f2450a00640 1 -- 172.21.15.32:0/1530032126 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f244c1f94c0 con 0x7f244c0b0c00 2024-08-03T21:00:04.064 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:04.061+0000 7f2450a00640 1 -- 172.21.15.32:0/1530032126 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f244c1f5a50 con 0x7f244c0b0c00 2024-08-03T21:00:04.065 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:04.062+0000 7f2450a00640 1 -- 172.21.15.32:0/1530032126 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f244c0ad190 con 0x7f244c0b0c00 2024-08-03T21:00:04.065 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:04.063+0000 7f2442a00640 1 -- 172.21.15.32:0/1530032126 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f24440041d0 con 0x7f244c0b0c00 2024-08-03T21:00:04.065 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:04.064+0000 7f2442a00640 1 --2- 172.21.15.32:0/1530032126 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f2428077010 0x7f2428079520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:00:04.065 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:04.064+0000 7f2442a00640 1 -- 172.21.15.32:0/1530032126 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(121..121 src has 1..121) ==== 8950+0+0 (secure 0 0 0) 0x7f244409a400 con 0x7f244c0b0c00 2024-08-03T21:00:04.066 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:04.064+0000 7f244a800640 1 --2- 172.21.15.32:0/1530032126 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f2428077010 0x7f2428079520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:00:04.066 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:04.065+0000 7f244a800640 1 --2- 172.21.15.32:0/1530032126 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f2428077010 0x7f2428079520 secure :-1 s=READY pgs=805 cs=0 l=1 rev1=1 crypto rx=0x7f243c009a30 tx=0x7f243c005f90 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T21:00:04.070 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:04.068+0000 7f2442a00640 1 -- 172.21.15.32:0/1530032126 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7f2444020020 con 0x7f244c0b0c00 2024-08-03T21:00:04.287 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:04.285+0000 7f2450a00640 1 -- 172.21.15.32:0/1530032126 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f244c0d0930 con 0x7f244c0b0c00 2024-08-03T21:00:04.288 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:04.286+0000 7f2442a00640 1 -- 172.21.15.32:0/1530032126 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 72 entries v121) ==== 81+0+4378 (secure 0 0 0) 0x7f2444062930 con 0x7f244c0b0c00 2024-08-03T21:00:04.288 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6835/3127944613 2024-08-04T20:59:46.766712+0000 2024-08-03T21:00:04.288 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6835/2106229503 2024-08-04T20:59:40.719664+0000 2024-08-03T21:00:04.288 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6833/2106229503 2024-08-04T20:59:40.719664+0000 2024-08-03T21:00:04.288 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6835/2173042128 2024-08-04T20:59:28.193772+0000 2024-08-03T21:00:04.288 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6833/2173042128 2024-08-04T20:59:28.193772+0000 2024-08-03T21:00:04.288 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6835/506775245 2024-08-04T20:59:03.076256+0000 2024-08-03T21:00:04.288 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6833/506775245 2024-08-04T20:59:03.076256+0000 2024-08-03T21:00:04.288 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6833/556650445 2024-08-04T20:58:57.063754+0000 2024-08-03T21:00:04.289 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6835/2156596385 2024-08-04T20:58:35.073838+0000 2024-08-03T21:00:04.289 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6833/2156596385 2024-08-04T20:58:35.073838+0000 2024-08-03T21:00:04.289 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6835/3669493978 2024-08-04T20:58:30.064676+0000 2024-08-03T21:00:04.289 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6833/2178506907 2024-08-04T20:56:40.045549+0000 2024-08-03T21:00:04.289 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6835/556650445 2024-08-04T20:58:57.063754+0000 2024-08-03T21:00:04.289 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6835/2278473729 2024-08-04T20:56:13.041693+0000 2024-08-03T21:00:04.289 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6835/2345472481 2024-08-04T20:57:25.968590+0000 2024-08-03T21:00:04.289 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6833/3940515773 2024-08-04T20:58:24.290690+0000 2024-08-03T21:00:04.289 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6833/2600386707 2024-08-04T20:57:50.063964+0000 2024-08-03T21:00:04.289 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6835/2178506907 2024-08-04T20:56:40.045549+0000 2024-08-03T21:00:04.289 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6833/2292690833 2024-08-04T20:55:20.021329+0000 2024-08-03T21:00:04.289 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6835/2803901497 2024-08-04T20:55:55.181144+0000 2024-08-03T21:00:04.290 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6835/2194902489 2024-08-04T20:55:00.359126+0000 2024-08-03T21:00:04.290 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6833/3127944613 2024-08-04T20:59:46.766712+0000 2024-08-03T21:00:04.290 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6833/2803901497 2024-08-04T20:55:55.181144+0000 2024-08-03T21:00:04.290 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6835/3191654276 2024-08-04T20:55:42.433364+0000 2024-08-03T21:00:04.290 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6833/2242533881 2024-08-04T20:56:48.048099+0000 2024-08-03T21:00:04.290 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6833/2283924374 2024-08-04T20:57:34.396782+0000 2024-08-03T21:00:04.290 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6835/1172050551 2024-08-04T20:55:26.214113+0000 2024-08-03T21:00:04.290 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6835/1016338829 2024-08-04T20:55:06.071485+0000 2024-08-03T21:00:04.290 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6833/1474984145 2024-08-04T20:57:40.052808+0000 2024-08-03T21:00:04.290 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6833/3669493978 2024-08-04T20:58:30.064676+0000 2024-08-03T21:00:04.290 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6835/756683988 2024-08-04T20:55:14.119228+0000 2024-08-03T21:00:04.291 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6833/2194902489 2024-08-04T20:55:00.359126+0000 2024-08-03T21:00:04.291 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6835/2806168786 2024-08-04T20:56:37.279160+0000 2024-08-03T21:00:04.291 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6833/2278473729 2024-08-04T20:56:13.041693+0000 2024-08-03T21:00:04.291 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6833/3191654276 2024-08-04T20:55:42.433364+0000 2024-08-03T21:00:04.291 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6835/3431300225 2024-08-04T20:56:21.088927+0000 2024-08-03T21:00:04.291 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6833/4172198298 2024-08-04T20:56:04.256182+0000 2024-08-03T21:00:04.291 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6833/3431300225 2024-08-04T20:56:21.088927+0000 2024-08-03T21:00:04.291 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6835/2292690833 2024-08-04T20:55:20.021329+0000 2024-08-03T21:00:04.291 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6833/2806168786 2024-08-04T20:56:37.279160+0000 2024-08-03T21:00:04.291 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6833/1172050551 2024-08-04T20:55:26.214113+0000 2024-08-03T21:00:04.291 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6835/4172198298 2024-08-04T20:56:04.256182+0000 2024-08-03T21:00:04.292 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6835/3409752696 2024-08-04T20:58:18.243140+0000 2024-08-03T21:00:04.292 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6833/2274054575 2024-08-04T20:57:55.062490+0000 2024-08-03T21:00:04.292 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6833/1016338829 2024-08-04T20:55:06.071485+0000 2024-08-03T21:00:04.292 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6833/1707454554 2024-08-04T20:54:38.409411+0000 2024-08-03T21:00:04.292 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6835/2242533881 2024-08-04T20:56:48.048099+0000 2024-08-03T21:00:04.292 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6833/739874257 2024-08-04T20:56:56.101271+0000 2024-08-03T21:00:04.292 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6835/739874257 2024-08-04T20:56:56.101271+0000 2024-08-03T21:00:04.292 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6833/2362145329 2024-08-04T20:57:02.150199+0000 2024-08-03T21:00:04.292 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6833/3771304550 2024-08-04T20:55:58.212752+0000 2024-08-03T21:00:04.293 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6835/2362145329 2024-08-04T20:57:02.150199+0000 2024-08-03T21:00:04.293 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6835/3940515773 2024-08-04T20:58:24.290690+0000 2024-08-03T21:00:04.293 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6833/1885034124 2024-08-04T20:57:08.190707+0000 2024-08-03T21:00:04.293 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6833/511221267 2024-08-04T20:57:45.062448+0000 2024-08-03T21:00:04.293 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6835/3771304550 2024-08-04T20:55:58.212752+0000 2024-08-03T21:00:04.293 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6835/1885034124 2024-08-04T20:57:08.190707+0000 2024-08-03T21:00:04.293 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6833/2345472481 2024-08-04T20:57:25.968590+0000 2024-08-03T21:00:04.293 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6833/265163622 2024-08-04T20:57:28.354509+0000 2024-08-03T21:00:04.293 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6835/265163622 2024-08-04T20:57:28.354509+0000 2024-08-03T21:00:04.293 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6835/2274054575 2024-08-04T20:57:55.062490+0000 2024-08-03T21:00:04.293 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6835/2283924374 2024-08-04T20:57:34.396782+0000 2024-08-03T21:00:04.293 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6833/756683988 2024-08-04T20:55:14.119228+0000 2024-08-03T21:00:04.293 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6835/1474984145 2024-08-04T20:57:40.052808+0000 2024-08-03T21:00:04.294 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6835/511221267 2024-08-04T20:57:45.062448+0000 2024-08-03T21:00:04.294 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6835/1707454554 2024-08-04T20:54:38.409411+0000 2024-08-03T21:00:04.294 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6835/2600386707 2024-08-04T20:57:50.063964+0000 2024-08-03T21:00:04.294 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6833/675070591 2024-08-04T20:58:00.068114+0000 2024-08-03T21:00:04.294 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6833/3340816283 2024-08-04T20:58:15.214542+0000 2024-08-03T21:00:04.294 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6835/3340816283 2024-08-04T20:58:15.214542+0000 2024-08-03T21:00:04.294 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6835/675070591 2024-08-04T20:58:00.068114+0000 2024-08-03T21:00:04.294 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6833/3409752696 2024-08-04T20:58:18.243140+0000 2024-08-03T21:00:04.295 INFO:teuthology.orchestra.run.smithi032.stderr:listed 72 entries 2024-08-03T21:00:04.295 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:04.288+0000 7f2450a00640 1 -- 172.21.15.32:0/1530032126 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f2428077010 msgr2=0x7f2428079520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:00:04.295 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:04.288+0000 7f2450a00640 1 --2- 172.21.15.32:0/1530032126 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f2428077010 0x7f2428079520 secure :-1 s=READY pgs=805 cs=0 l=1 rev1=1 crypto rx=0x7f243c009a30 tx=0x7f243c005f90 comp rx=0 tx=0).stop 2024-08-03T21:00:04.296 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:04.288+0000 7f2450a00640 1 -- 172.21.15.32:0/1530032126 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f244c0b0c00 msgr2=0x7f244c0b1010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:00:04.296 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:04.288+0000 7f2450a00640 1 --2- 172.21.15.32:0/1530032126 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f244c0b0c00 0x7f244c0b1010 secure :-1 s=READY pgs=2015 cs=0 l=1 rev1=1 crypto rx=0x7f2444005fd0 tx=0x7f244400f790 comp rx=0 tx=0).stop 2024-08-03T21:00:04.296 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:04.288+0000 7f2450a00640 1 -- 172.21.15.32:0/1530032126 shutdown_connections 2024-08-03T21:00:04.296 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:04.288+0000 7f2450a00640 1 --2- 172.21.15.32:0/1530032126 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f2428077010 0x7f2428079520 unknown :-1 s=CLOSED pgs=805 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:04.296 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:04.288+0000 7f2450a00640 1 --2- 172.21.15.32:0/1530032126 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f244c1f90b0 0x7f244c0b4850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:04.296 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:04.288+0000 7f2450a00640 1 --2- 172.21.15.32:0/1530032126 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f244c0b0c00 0x7f244c0b1010 unknown :-1 s=CLOSED pgs=2015 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:04.296 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:04.288+0000 7f2450a00640 1 --2- 172.21.15.32:0/1530032126 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f2430004800 0x7f244c1f50b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:04.296 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:04.288+0000 7f2450a00640 1 -- 172.21.15.32:0/1530032126 >> 172.21.15.32:0/1530032126 conn(0x7f244c0b2120 msgr2=0x7f244c0cb7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T21:00:04.296 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:04.289+0000 7f2450a00640 1 -- 172.21.15.32:0/1530032126 shutdown_connections 2024-08-03T21:00:04.296 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:04.289+0000 7f2450a00640 1 -- 172.21.15.32:0/1530032126 wait complete. 2024-08-03T21:00:04.315 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-08-03T21:00:04.315 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-08-03T21:00:04.315 DEBUG:teuthology.orchestra.run.smithi032:> sudo umount /home/ubuntu/cephtest/mnt.0 -f 2024-08-03T21:00:04.396 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi032.front.sepia.ceph.com 2024-08-03T21:00:04.396 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-08-03T21:00:04.396 DEBUG:teuthology.orchestra.run.smithi032:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-08-03T21:00:04.432 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-08-03T21:00:04.701 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:04.699+0000 7f92d4200640 1 -- 172.21.15.32:0/2288325501 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f92cc15f610 msgr2=0x7f92cc1523f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:00:04.702 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:04.700+0000 7f92d4200640 1 --2- 172.21.15.32:0/2288325501 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f92cc15f610 0x7f92cc1523f0 secure :-1 s=READY pgs=2017 cs=0 l=1 rev1=1 crypto rx=0x7f92b8009830 tx=0x7f92b802a480 comp rx=0 tx=0).stop 2024-08-03T21:00:04.702 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:04.700+0000 7f92d4200640 1 -- 172.21.15.32:0/2288325501 shutdown_connections 2024-08-03T21:00:04.702 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:04.700+0000 7f92d4200640 1 --2- 172.21.15.32:0/2288325501 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f92cc15f610 0x7f92cc1523f0 unknown :-1 s=CLOSED pgs=2017 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:04.702 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:04.700+0000 7f92d4200640 1 -- 172.21.15.32:0/2288325501 >> 172.21.15.32:0/2288325501 conn(0x7f92cc0b1bd0 msgr2=0x7f92cc0b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T21:00:04.702 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:04.700+0000 7f92d4200640 1 -- 172.21.15.32:0/2288325501 shutdown_connections 2024-08-03T21:00:04.702 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:04.700+0000 7f92d4200640 1 -- 172.21.15.32:0/2288325501 wait complete. 2024-08-03T21:00:04.703 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:04.702+0000 7f92d4200640 1 Processor -- start 2024-08-03T21:00:04.703 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:04.702+0000 7f92d4200640 1 -- start start 2024-08-03T21:00:04.706 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:04.704+0000 7f92d4200640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f92cc0b38b0 0x7f92cc1f3930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:00:04.706 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:04.704+0000 7f92d4200640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f92cc0b4ce0 0x7f92cc158070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:00:04.706 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:04.704+0000 7f92d4200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f92cc1585b0 0x7f92cc14e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:00:04.706 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:04.704+0000 7f92d4200640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f92cc0cec50 con 0x7f92cc1585b0 2024-08-03T21:00:04.706 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:04.704+0000 7f92d4200640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f92cc0cead0 con 0x7f92cc0b38b0 2024-08-03T21:00:04.706 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:04.704+0000 7f92d4200640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7f92cc0cedd0 con 0x7f92cc0b4ce0 2024-08-03T21:00:04.706 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:04.704+0000 7f92d2200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f92cc1585b0 0x7f92cc14e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:00:04.707 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:04.704+0000 7f92d2200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f92cc1585b0 0x7f92cc14e770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:39654/0 (socket says 172.21.15.32:39654) 2024-08-03T21:00:04.707 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:04.704+0000 7f92d2200640 1 -- 172.21.15.32:0/4110632908 learned_addr learned my addr 172.21.15.32:0/4110632908 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T21:00:04.707 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:04.704+0000 7f92d1800640 1 --2- 172.21.15.32:0/4110632908 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f92cc0b38b0 0x7f92cc1f3930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:00:04.707 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:04.704+0000 7f92d0e00640 1 --2- 172.21.15.32:0/4110632908 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f92cc0b4ce0 0x7f92cc158070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:00:04.707 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:04.704+0000 7f92d2200640 1 -- 172.21.15.32:0/4110632908 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f92cc0b4ce0 msgr2=0x7f92cc158070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:00:04.707 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:04.704+0000 7f92d2200640 1 --2- 172.21.15.32:0/4110632908 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f92cc0b4ce0 0x7f92cc158070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:04.707 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:04.704+0000 7f92d2200640 1 -- 172.21.15.32:0/4110632908 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f92cc0b38b0 msgr2=0x7f92cc1f3930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:00:04.707 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:04.704+0000 7f92d2200640 1 --2- 172.21.15.32:0/4110632908 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f92cc0b38b0 0x7f92cc1f3930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:04.707 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:04.704+0000 7f92d2200640 1 -- 172.21.15.32:0/4110632908 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f92cc156770 con 0x7f92cc1585b0 2024-08-03T21:00:04.707 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:04.705+0000 7f92d2200640 1 --2- 172.21.15.32:0/4110632908 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f92cc1585b0 0x7f92cc14e770 secure :-1 s=READY pgs=2018 cs=0 l=1 rev1=1 crypto rx=0x7f92c400b970 tx=0x7f92c400be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T21:00:04.708 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:04.705+0000 7f92c9600640 1 -- 172.21.15.32:0/4110632908 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92c4019070 con 0x7f92cc1585b0 2024-08-03T21:00:04.708 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:04.705+0000 7f92c9600640 1 -- 172.21.15.32:0/4110632908 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f92c400cc30 con 0x7f92cc1585b0 2024-08-03T21:00:04.708 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:04.705+0000 7f92c9600640 1 -- 172.21.15.32:0/4110632908 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92c40146a0 con 0x7f92cc1585b0 2024-08-03T21:00:04.708 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:04.705+0000 7f92d4200640 1 -- 172.21.15.32:0/4110632908 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f92cc156a50 con 0x7f92cc1585b0 2024-08-03T21:00:04.708 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:04.705+0000 7f92d4200640 1 -- 172.21.15.32:0/4110632908 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f92cc157010 con 0x7f92cc1585b0 2024-08-03T21:00:04.708 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:04.706+0000 7f92d4200640 1 -- 172.21.15.32:0/4110632908 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f92cc1564e0 con 0x7f92cc1585b0 2024-08-03T21:00:04.709 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:04.708+0000 7f92c9600640 1 -- 172.21.15.32:0/4110632908 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f92c401c070 con 0x7f92cc1585b0 2024-08-03T21:00:04.710 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:04.709+0000 7f92c9600640 1 --2- 172.21.15.32:0/4110632908 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f92ac077010 0x7f92ac079520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:00:04.710 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:04.709+0000 7f92c9600640 1 -- 172.21.15.32:0/4110632908 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(121..121 src has 1..121) ==== 8950+0+0 (secure 0 0 0) 0x7f92c4099280 con 0x7f92cc1585b0 2024-08-03T21:00:04.710 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:04.709+0000 7f92d1800640 1 --2- 172.21.15.32:0/4110632908 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f92ac077010 0x7f92ac079520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:00:04.711 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:04.710+0000 7f92d1800640 1 --2- 172.21.15.32:0/4110632908 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f92ac077010 0x7f92ac079520 secure :-1 s=READY pgs=806 cs=0 l=1 rev1=1 crypto rx=0x7f92c0007fd0 tx=0x7f92c0009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T21:00:04.715 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:04.714+0000 7f92c9600640 1 -- 172.21.15.32:0/4110632908 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7f92c4061810 con 0x7f92cc1585b0 2024-08-03T21:00:04.959 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:04.957+0000 7f92d4200640 1 -- 172.21.15.32:0/4110632908 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7f92cc1523f0 con 0x7f92cc1585b0 2024-08-03T21:00:04.959 INFO:tasks.ceph.mon.a.smithi032.stderr:2024-08-03T21:00:04.958+0000 7f4e6d600640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-08-03T21:00:04.961 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:04.960+0000 7f92c9600640 1 -- 172.21.15.32:0/4110632908 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v309) ==== 161+0+0 (secure 0 0 0) 0x7f92c4061110 con 0x7f92cc1585b0 2024-08-03T21:00:04.962 INFO:teuthology.orchestra.run.smithi032.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-08-03T21:00:04.965 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:04.964+0000 7f92d4200640 1 -- 172.21.15.32:0/4110632908 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f92ac077010 msgr2=0x7f92ac079520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:00:04.965 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:04.964+0000 7f92d4200640 1 --2- 172.21.15.32:0/4110632908 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f92ac077010 0x7f92ac079520 secure :-1 s=READY pgs=806 cs=0 l=1 rev1=1 crypto rx=0x7f92c0007fd0 tx=0x7f92c0009040 comp rx=0 tx=0).stop 2024-08-03T21:00:04.965 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:04.964+0000 7f92d4200640 1 -- 172.21.15.32:0/4110632908 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f92cc1585b0 msgr2=0x7f92cc14e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:00:04.966 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:04.964+0000 7f92d4200640 1 --2- 172.21.15.32:0/4110632908 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f92cc1585b0 0x7f92cc14e770 secure :-1 s=READY pgs=2018 cs=0 l=1 rev1=1 crypto rx=0x7f92c400b970 tx=0x7f92c400be30 comp rx=0 tx=0).stop 2024-08-03T21:00:04.966 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:04.964+0000 7f92d4200640 1 -- 172.21.15.32:0/4110632908 shutdown_connections 2024-08-03T21:00:04.966 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:04.964+0000 7f92d4200640 1 --2- 172.21.15.32:0/4110632908 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f92ac077010 0x7f92ac079520 unknown :-1 s=CLOSED pgs=806 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:04.966 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:04.964+0000 7f92d4200640 1 --2- 172.21.15.32:0/4110632908 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f92cc1585b0 0x7f92cc14e770 unknown :-1 s=CLOSED pgs=2018 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:04.966 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:04.964+0000 7f92d4200640 1 --2- 172.21.15.32:0/4110632908 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f92cc0b4ce0 0x7f92cc158070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:04.966 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:04.964+0000 7f92d4200640 1 --2- 172.21.15.32:0/4110632908 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f92cc0b38b0 0x7f92cc1f3930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:04.966 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:04.964+0000 7f92d4200640 1 -- 172.21.15.32:0/4110632908 >> 172.21.15.32:0/4110632908 conn(0x7f92cc0b1bd0 msgr2=0x7f92cc073e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T21:00:04.966 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:04.965+0000 7f92d4200640 1 -- 172.21.15.32:0/4110632908 shutdown_connections 2024-08-03T21:00:04.967 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:04.965+0000 7f92d4200640 1 -- 172.21.15.32:0/4110632908 wait complete. 2024-08-03T21:00:04.967 INFO:tasks.ceph.mds.a.smithi032.stderr: -1909> 2024-08-03T20:59:54.800+0000 7fc55e800640 -1 mds.0.openfiles _load_finish: corrupted header/values: End of buffer [buffer:2] 2024-08-03T21:00:04.970 INFO:tasks.ceph.mds.a.smithi032.stderr: -649> 2024-08-03T21:00:01.778+0000 7fc563800640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-08-03T21:00:04.992 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mds repaired 0 2024-08-03T21:00:05.087 INFO:tasks.ceph.mds.a.smithi032.stdout:starting mds.a at 2024-08-03T21:00:05.225 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:05.223+0000 7f7613200640 1 -- 172.21.15.32:0/2682342997 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f760c0b52a0 msgr2=0x7f760c0b56b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:00:05.225 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:05.223+0000 7f7613200640 1 --2- 172.21.15.32:0/2682342997 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f760c0b52a0 0x7f760c0b56b0 secure :-1 s=READY pgs=2021 cs=0 l=1 rev1=1 crypto rx=0x7f760c073f50 tx=0x7f760802a980 comp rx=0 tx=0).stop 2024-08-03T21:00:05.225 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:05.223+0000 7f7613200640 1 -- 172.21.15.32:0/2682342997 shutdown_connections 2024-08-03T21:00:05.225 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:05.223+0000 7f7613200640 1 --2- 172.21.15.32:0/2682342997 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f760c0b52a0 0x7f760c0b56b0 unknown :-1 s=CLOSED pgs=2021 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:05.225 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:05.223+0000 7f7613200640 1 --2- 172.21.15.32:0/2682342997 >> v2:172.21.15.118:3300/0 conn(0x7f760c161d20 0x7f760c0cedc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:05.225 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:05.223+0000 7f7613200640 1 --2- 172.21.15.32:0/2682342997 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f760c0b74c0 0x7f760c0ce880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:05.225 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:05.223+0000 7f7613200640 1 -- 172.21.15.32:0/2682342997 >> 172.21.15.32:0/2682342997 conn(0x7f760c0d1e90 msgr2=0x7f760c0d22a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T21:00:05.225 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:05.223+0000 7f7613200640 1 -- 172.21.15.32:0/2682342997 shutdown_connections 2024-08-03T21:00:05.226 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:05.223+0000 7f7613200640 1 -- 172.21.15.32:0/2682342997 wait complete. 2024-08-03T21:00:05.226 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:05.225+0000 7f7613200640 1 Processor -- start 2024-08-03T21:00:05.227 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:05.225+0000 7f7613200640 1 -- start start 2024-08-03T21:00:05.230 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:05.227+0000 7f7613200640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f760c0b52a0 0x7f760c157c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:00:05.230 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:05.227+0000 7f7613200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f760c0b74c0 0x7f760c1f9160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:00:05.230 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:05.227+0000 7f7613200640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f760c161d20 0x7f760c1f5b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:00:05.230 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:05.227+0000 7f7613200640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f760c0b5f80 con 0x7f760c0b74c0 2024-08-03T21:00:05.230 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:05.227+0000 7f7613200640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f760c0b5e00 con 0x7f760c0b52a0 2024-08-03T21:00:05.230 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:05.227+0000 7f7613200640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7f760c0b6100 con 0x7f760c161d20 2024-08-03T21:00:05.230 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:05.227+0000 7f7611400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f760c0b74c0 0x7f760c1f9160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:00:05.230 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:05.227+0000 7f7611400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f760c0b74c0 0x7f760c1f9160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:39682/0 (socket says 172.21.15.32:39682) 2024-08-03T21:00:05.230 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:05.227+0000 7f7611400640 1 -- 172.21.15.32:0/951612081 learned_addr learned my addr 172.21.15.32:0/951612081 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T21:00:05.230 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:05.227+0000 7f7612800640 1 --2- 172.21.15.32:0/951612081 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f760c161d20 0x7f760c1f5b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:00:05.231 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:05.227+0000 7f7611e00640 1 --2- 172.21.15.32:0/951612081 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f760c0b52a0 0x7f760c157c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:00:05.231 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:05.227+0000 7f7611400640 1 -- 172.21.15.32:0/951612081 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f760c161d20 msgr2=0x7f760c1f5b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:00:05.231 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:05.228+0000 7f7611400640 1 --2- 172.21.15.32:0/951612081 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f760c161d20 0x7f760c1f5b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:05.231 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:05.228+0000 7f7611400640 1 -- 172.21.15.32:0/951612081 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f760c0b52a0 msgr2=0x7f760c157c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:00:05.231 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:05.228+0000 7f7611400640 1 --2- 172.21.15.32:0/951612081 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f760c0b52a0 0x7f760c157c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:05.231 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:05.228+0000 7f7611400640 1 -- 172.21.15.32:0/951612081 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f760c09cf00 con 0x7f760c0b74c0 2024-08-03T21:00:05.231 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:05.228+0000 7f7611e00640 1 --2- 172.21.15.32:0/951612081 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f760c0b52a0 0x7f760c157c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-03T21:00:05.231 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:05.228+0000 7f7611400640 1 --2- 172.21.15.32:0/951612081 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f760c0b74c0 0x7f760c1f9160 secure :-1 s=READY pgs=2022 cs=0 l=1 rev1=1 crypto rx=0x7f75fc00c9a0 tx=0x7f75fc00ce60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T21:00:05.231 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:05.228+0000 7f7606000640 1 -- 172.21.15.32:0/951612081 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75fc007b50 con 0x7f760c0b74c0 2024-08-03T21:00:05.231 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:05.228+0000 7f7606000640 1 -- 172.21.15.32:0/951612081 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f75fc007de0 con 0x7f760c0b74c0 2024-08-03T21:00:05.231 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:05.228+0000 7f7613200640 1 -- 172.21.15.32:0/951612081 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f760c09d1e0 con 0x7f760c0b74c0 2024-08-03T21:00:05.231 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:05.228+0000 7f7606000640 1 -- 172.21.15.32:0/951612081 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75fc00f6b0 con 0x7f760c0b74c0 2024-08-03T21:00:05.232 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:05.228+0000 7f7613200640 1 -- 172.21.15.32:0/951612081 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f760c09f3f0 con 0x7f760c0b74c0 2024-08-03T21:00:05.232 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:05.229+0000 7f7613200640 1 -- 172.21.15.32:0/951612081 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f760c09cc70 con 0x7f760c0b74c0 2024-08-03T21:00:05.232 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:05.231+0000 7f7606000640 1 -- 172.21.15.32:0/951612081 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f75fc00f850 con 0x7f760c0b74c0 2024-08-03T21:00:05.233 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:05.232+0000 7f7606000640 1 --2- 172.21.15.32:0/951612081 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f75f0077010 0x7f75f0079520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:00:05.233 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:05.232+0000 7f7606000640 1 -- 172.21.15.32:0/951612081 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(122..122 src has 1..122) ==== 9036+0+0 (secure 0 0 0) 0x7f75fc099de0 con 0x7f760c0b74c0 2024-08-03T21:00:05.234 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:05.232+0000 7f7611e00640 1 --2- 172.21.15.32:0/951612081 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f75f0077010 0x7f75f0079520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:00:05.234 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:05.233+0000 7f7611e00640 1 --2- 172.21.15.32:0/951612081 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f75f0077010 0x7f75f0079520 secure :-1 s=READY pgs=807 cs=0 l=1 rev1=1 crypto rx=0x7f7600009a20 tx=0x7f7600005d30 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T21:00:05.237 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:05.236+0000 7f7606000640 1 -- 172.21.15.32:0/951612081 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7f75fc014030 con 0x7f760c0b74c0 2024-08-03T21:00:05.524 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:05.522+0000 7f7613200640 1 -- 172.21.15.32:0/951612081 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "mds repaired", "role": "0"} v 0) -- 0x7f760c0b56b0 con 0x7f760c0b74c0 2024-08-03T21:00:05.964 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:05.962+0000 7f7606000640 1 -- 172.21.15.32:0/951612081 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "mds repaired", "role": "0"}]=0 nothing to do: rank is not damaged v310) ==== 107+0+0 (secure 0 0 0) 0x7f75fc062230 con 0x7f760c0b74c0 2024-08-03T21:00:05.964 INFO:teuthology.orchestra.run.smithi032.stderr:nothing to do: rank is not damaged 2024-08-03T21:00:05.966 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:05.965+0000 7f7613200640 1 -- 172.21.15.32:0/951612081 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f75f0077010 msgr2=0x7f75f0079520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:00:05.967 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:05.965+0000 7f7613200640 1 --2- 172.21.15.32:0/951612081 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f75f0077010 0x7f75f0079520 secure :-1 s=READY pgs=807 cs=0 l=1 rev1=1 crypto rx=0x7f7600009a20 tx=0x7f7600005d30 comp rx=0 tx=0).stop 2024-08-03T21:00:05.967 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:05.965+0000 7f7613200640 1 -- 172.21.15.32:0/951612081 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f760c0b74c0 msgr2=0x7f760c1f9160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:00:05.967 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:05.965+0000 7f7613200640 1 --2- 172.21.15.32:0/951612081 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f760c0b74c0 0x7f760c1f9160 secure :-1 s=READY pgs=2022 cs=0 l=1 rev1=1 crypto rx=0x7f75fc00c9a0 tx=0x7f75fc00ce60 comp rx=0 tx=0).stop 2024-08-03T21:00:05.967 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:05.965+0000 7f7613200640 1 -- 172.21.15.32:0/951612081 shutdown_connections 2024-08-03T21:00:05.967 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:05.965+0000 7f7613200640 1 --2- 172.21.15.32:0/951612081 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f75f0077010 0x7f75f0079520 unknown :-1 s=CLOSED pgs=807 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:05.967 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:05.965+0000 7f7613200640 1 --2- 172.21.15.32:0/951612081 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f760c161d20 0x7f760c1f5b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:05.967 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:05.965+0000 7f7613200640 1 --2- 172.21.15.32:0/951612081 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f760c0b74c0 0x7f760c1f9160 unknown :-1 s=CLOSED pgs=2022 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:05.967 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:05.965+0000 7f7613200640 1 --2- 172.21.15.32:0/951612081 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f760c0b52a0 0x7f760c157c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:05.967 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:05.965+0000 7f7613200640 1 -- 172.21.15.32:0/951612081 >> 172.21.15.32:0/951612081 conn(0x7f760c0d1e90 msgr2=0x7f760c15f760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T21:00:05.967 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:05.965+0000 7f7613200640 1 -- 172.21.15.32:0/951612081 shutdown_connections 2024-08-03T21:00:05.967 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:05.965+0000 7f7613200640 1 -- 172.21.15.32:0/951612081 wait complete. 2024-08-03T21:00:05.994 DEBUG:teuthology.orchestra.run.smithi032:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata import - 2024-08-03T21:00:06.140 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.138+0000 7f654cb20f40 1 Processor -- start 2024-08-03T21:00:06.140 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.138+0000 7f654cb20f40 1 -- start start 2024-08-03T21:00:06.142 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.139+0000 7f654cb20f40 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x5646a0de7010 0x5646a0f001b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:00:06.142 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.139+0000 7f654cb20f40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5646a0de8f50 0x5646a0e734d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:00:06.142 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.140+0000 7f654cb20f40 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5646a0e71820 0x5646a0e73a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:00:06.142 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.140+0000 7f654cb20f40 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x5646a0cede30 con 0x5646a0e71820 2024-08-03T21:00:06.142 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.140+0000 7f654cb20f40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x5646a0d4bbf0 con 0x5646a0de8f50 2024-08-03T21:00:06.143 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.140+0000 7f654cb20f40 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x5646a0de76f0 con 0x5646a0de7010 2024-08-03T21:00:06.143 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.140+0000 7f6545e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5646a0e71820 0x5646a0e73a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:00:06.143 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.140+0000 7f6545e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5646a0e71820 0x5646a0e73a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:39694/0 (socket says 172.21.15.32:39694) 2024-08-03T21:00:06.143 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.140+0000 7f6545e00640 1 -- 172.21.15.32:0/4269949007 learned_addr learned my addr 172.21.15.32:0/4269949007 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T21:00:06.143 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.140+0000 7f6544a00640 1 --2- 172.21.15.32:0/4269949007 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5646a0de8f50 0x5646a0e734d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:00:06.143 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.140+0000 7f6545400640 1 --2- 172.21.15.32:0/4269949007 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x5646a0de7010 0x5646a0f001b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:00:06.143 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.140+0000 7f6545e00640 1 -- 172.21.15.32:0/4269949007 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x5646a0de7010 msgr2=0x5646a0f001b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T21:00:06.143 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.140+0000 7f6545e00640 1 --2- 172.21.15.32:0/4269949007 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x5646a0de7010 0x5646a0f001b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:06.143 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.140+0000 7f6545e00640 1 -- 172.21.15.32:0/4269949007 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5646a0de8f50 msgr2=0x5646a0e734d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T21:00:06.143 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.140+0000 7f6545e00640 1 --2- 172.21.15.32:0/4269949007 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5646a0de8f50 0x5646a0e734d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:06.145 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.140+0000 7f6545e00640 1 -- 172.21.15.32:0/4269949007 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5646a0d9d510 con 0x5646a0e71820 2024-08-03T21:00:06.145 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.140+0000 7f6544a00640 1 --2- 172.21.15.32:0/4269949007 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5646a0de8f50 0x5646a0e734d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-03T21:00:06.145 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.140+0000 7f6545400640 1 --2- 172.21.15.32:0/4269949007 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x5646a0de7010 0x5646a0f001b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-03T21:00:06.145 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.141+0000 7f6545e00640 1 --2- 172.21.15.32:0/4269949007 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5646a0e71820 0x5646a0e73a10 secure :-1 s=READY pgs=2024 cs=0 l=1 rev1=1 crypto rx=0x7f65400015e0 tx=0x7f65400019e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=19bcedfd1e65dfd5 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T21:00:06.145 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.141+0000 7f653f400640 1 -- 172.21.15.32:0/4269949007 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6540016070 con 0x5646a0e71820 2024-08-03T21:00:06.145 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.141+0000 7f653f400640 1 -- 172.21.15.32:0/4269949007 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f654000f800 con 0x5646a0e71820 2024-08-03T21:00:06.146 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.141+0000 7f653f400640 1 -- 172.21.15.32:0/4269949007 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f654000fc50 con 0x5646a0e71820 2024-08-03T21:00:06.155 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.142+0000 7f654cb20f40 1 -- 172.21.15.32:0/4269949007 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5646a0e71820 msgr2=0x5646a0e73a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:00:06.155 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.142+0000 7f654cb20f40 1 --2- 172.21.15.32:0/4269949007 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5646a0e71820 0x5646a0e73a10 secure :-1 s=READY pgs=2024 cs=0 l=1 rev1=1 crypto rx=0x7f65400015e0 tx=0x7f65400019e0 comp rx=0 tx=0).stop 2024-08-03T21:00:06.155 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.142+0000 7f654cb20f40 1 -- 172.21.15.32:0/4269949007 shutdown_connections 2024-08-03T21:00:06.155 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.142+0000 7f654cb20f40 1 --2- 172.21.15.32:0/4269949007 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5646a0de8f50 0x5646a0e734d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:06.155 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.142+0000 7f654cb20f40 1 --2- 172.21.15.32:0/4269949007 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5646a0e71820 0x5646a0e73a10 unknown :-1 s=CLOSED pgs=2024 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:06.155 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.142+0000 7f654cb20f40 1 --2- 172.21.15.32:0/4269949007 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x5646a0de7010 0x5646a0f001b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:06.155 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.142+0000 7f654cb20f40 1 -- 172.21.15.32:0/4269949007 >> 172.21.15.32:0/4269949007 conn(0x5646a0cbfac0 msgr2=0x5646a0dca120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T21:00:06.155 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.142+0000 7f654cb20f40 1 -- 172.21.15.32:0/4269949007 shutdown_connections 2024-08-03T21:00:06.155 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.142+0000 7f654cb20f40 1 -- 172.21.15.32:0/4269949007 wait complete. 2024-08-03T21:00:06.155 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.143+0000 7f654cb20f40 1 Processor -- start 2024-08-03T21:00:06.155 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.143+0000 7f654cb20f40 1 -- start start 2024-08-03T21:00:06.156 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.144+0000 7f654cb20f40 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5646a0de7010 0x5646a0e65a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:00:06.156 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.144+0000 7f654cb20f40 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x5646a0de8f50 0x5646a0e740a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:00:06.156 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.144+0000 7f654cb20f40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5646a0e71820 0x5646a0e745e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:00:06.156 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.144+0000 7f654cb20f40 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x5646a0e6d4a0 con 0x5646a0de7010 2024-08-03T21:00:06.156 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.144+0000 7f654cb20f40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x5646a0e6d780 con 0x5646a0e71820 2024-08-03T21:00:06.156 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.144+0000 7f654cb20f40 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x5646a0de4bc0 con 0x5646a0de8f50 2024-08-03T21:00:06.156 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.145+0000 7f6545400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5646a0de7010 0x5646a0e65a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:00:06.156 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.145+0000 7f6545400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5646a0de7010 0x5646a0e65a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:39696/0 (socket says 172.21.15.32:39696) 2024-08-03T21:00:06.156 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.145+0000 7f6545400640 1 -- 172.21.15.32:0/3921196523 learned_addr learned my addr 172.21.15.32:0/3921196523 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T21:00:06.156 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.145+0000 7f6545e00640 1 --2- 172.21.15.32:0/3921196523 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5646a0e71820 0x5646a0e745e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:00:06.156 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.145+0000 7f6544a00640 1 --2- 172.21.15.32:0/3921196523 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x5646a0de8f50 0x5646a0e740a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:00:06.156 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.145+0000 7f6545400640 1 -- 172.21.15.32:0/3921196523 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x5646a0de8f50 msgr2=0x5646a0e740a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:00:06.156 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.145+0000 7f6545400640 1 --2- 172.21.15.32:0/3921196523 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x5646a0de8f50 0x5646a0e740a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:06.157 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.145+0000 7f6545400640 1 -- 172.21.15.32:0/3921196523 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5646a0e71820 msgr2=0x5646a0e745e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:00:06.157 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.145+0000 7f6545400640 1 --2- 172.21.15.32:0/3921196523 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5646a0e71820 0x5646a0e745e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:06.157 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.145+0000 7f6545400640 1 -- 172.21.15.32:0/3921196523 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5646a0f02f20 con 0x5646a0de7010 2024-08-03T21:00:06.157 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.145+0000 7f6545400640 1 --2- 172.21.15.32:0/3921196523 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5646a0de7010 0x5646a0e65a00 secure :-1 s=READY pgs=2025 cs=0 l=1 rev1=1 crypto rx=0x7f653800a6b0 tx=0x7f653800ab70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T21:00:06.157 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.146+0000 7f653e000640 1 -- 172.21.15.32:0/3921196523 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6538008990 con 0x5646a0de7010 2024-08-03T21:00:06.157 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.146+0000 7f653e000640 1 -- 172.21.15.32:0/3921196523 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f6538008b30 con 0x5646a0de7010 2024-08-03T21:00:06.157 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.146+0000 7f653e000640 1 -- 172.21.15.32:0/3921196523 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6538016420 con 0x5646a0de7010 2024-08-03T21:00:06.157 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.146+0000 7f654cb20f40 1 -- 172.21.15.32:0/3921196523 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x5646a0f03200 con 0x5646a0de7010 2024-08-03T21:00:06.157 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.146+0000 7f654cb20f40 1 -- 172.21.15.32:0/3921196523 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5646a0f00450 con 0x5646a0de7010 2024-08-03T21:00:06.157 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.148+0000 7f653e000640 1 -- 172.21.15.32:0/3921196523 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f653801f440 con 0x5646a0de7010 2024-08-03T21:00:06.157 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.149+0000 7f653e000640 1 --2- 172.21.15.32:0/3921196523 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f65280759f0 0x7f6528077f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:00:06.157 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.149+0000 7f653e000640 1 -- 172.21.15.32:0/3921196523 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(122..122 src has 1..122) ==== 9036+0+0 (secure 0 0 0) 0x7f6538098320 con 0x5646a0de7010 2024-08-03T21:00:06.158 INFO:teuthology.orchestra.run.smithi032.stdout:Importing pool 2024-08-03T21:00:06.159 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.149+0000 7f654cb20f40 1 --2- 172.21.15.32:0/3921196523 >> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] conn(0x5646a0e69d20 0x5646a0dc9fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:00:06.159 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.149+0000 7f654cb20f40 1 -- 172.21.15.32:0/3921196523 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:1 12.10 12:09401a81:::601.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122) -- 0x5646a0dca4f0 con 0x5646a0e69d20 2024-08-03T21:00:06.159 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.149+0000 7f6544a00640 1 --2- 172.21.15.32:0/3921196523 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f65280759f0 0x7f6528077f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:00:06.159 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.150+0000 7f6545e00640 1 --2- 172.21.15.32:0/3921196523 >> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] conn(0x5646a0e69d20 0x5646a0dc9fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:00:06.159 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.150+0000 7f6545e00640 1 --2- 172.21.15.32:0/3921196523 >> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] conn(0x5646a0e69d20 0x5646a0dc9fb0 crc :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T21:00:06.159 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.150+0000 7f6544a00640 1 --2- 172.21.15.32:0/3921196523 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f65280759f0 0x7f6528077f00 secure :-1 s=READY pgs=809 cs=0 l=1 rev1=1 crypto rx=0x7f6534055000 tx=0x7f6534003020 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T21:00:06.160 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.153+0000 7f6545e00640 1 -- 172.21.15.32:0/3921196523 <== osd.0 v2:172.21.15.32:6816/3066156116 1 ==== osd_op_reply(1 601.00000000 [create] v122'1128 uv1102 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f6540019040 con 0x5646a0e69d20 2024-08-03T21:00:06.160 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.153+0000 7f654cb20f40 1 -- 172.21.15.32:0/3921196523 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:2 12.10 12:09401a81:::601.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e122) -- 0x5646a0dca8d0 con 0x5646a0e69d20 2024-08-03T21:00:06.160 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.155+0000 7f6545e00640 1 -- 172.21.15.32:0/3921196523 <== osd.0 v2:172.21.15.32:6816/3066156116 2 ==== osd_op_reply(2 601.00000000 [delete] v122'1129 uv1129 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f6540019040 con 0x5646a0e69d20 2024-08-03T21:00:06.160 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.155+0000 7f654cb20f40 1 -- 172.21.15.32:0/3921196523 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:3 12.10 12:09401a81:::601.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122) -- 0x5646a0f149e0 con 0x5646a0e69d20 2024-08-03T21:00:06.160 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::601.00000000:head# 2024-08-03T21:00:06.161 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.157+0000 7f6545e00640 1 -- 172.21.15.32:0/3921196523 <== osd.0 v2:172.21.15.32:6816/3066156116 3 ==== osd_op_reply(3 601.00000000 [create] v122'1130 uv1130 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f6540019040 con 0x5646a0e69d20 2024-08-03T21:00:06.161 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.157+0000 7f654cb20f40 1 -- 172.21.15.32:0/3921196523 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:4 12.10 12:09401a81:::601.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122) -- 0x5646a0f15d90 con 0x5646a0e69d20 2024-08-03T21:00:06.161 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.158+0000 7f6545e00640 1 -- 172.21.15.32:0/3921196523 <== osd.0 v2:172.21.15.32:6816/3066156116 4 ==== osd_op_reply(4 601.00000000 [setxattr (30)] v122'1131 uv1131 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f6540019040 con 0x5646a0e69d20 2024-08-03T21:00:06.162 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.158+0000 7f654cb20f40 1 -- 172.21.15.32:0/3921196523 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:5 12.10 12:09401a81:::601.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122) -- 0x5646a0f16130 con 0x5646a0e69d20 2024-08-03T21:00:06.162 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.160+0000 7f6545e00640 1 -- 172.21.15.32:0/3921196523 <== osd.0 v2:172.21.15.32:6816/3066156116 5 ==== osd_op_reply(5 601.00000000 [setxattr (62)] v122'1132 uv1132 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f6540019040 con 0x5646a0e69d20 2024-08-03T21:00:06.162 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.160+0000 7f654cb20f40 1 -- 172.21.15.32:0/3921196523 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:6 12.10 12:09401a81:::601.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122) -- 0x5646a0f14d80 con 0x5646a0e69d20 2024-08-03T21:00:06.164 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.162+0000 7f6545e00640 1 -- 172.21.15.32:0/3921196523 <== osd.0 v2:172.21.15.32:6816/3066156116 6 ==== osd_op_reply(6 601.00000000 [omap-set-header] v122'1133 uv1133 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f6540019040 con 0x5646a0e69d20 2024-08-03T21:00:06.164 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.163+0000 7f654cb20f40 1 -- 172.21.15.32:0/3921196523 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:7 12.10 12:0bd6d154:::602.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122) -- 0x5646a0f15120 con 0x5646a0e69d20 2024-08-03T21:00:06.166 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.165+0000 7f6545e00640 1 -- 172.21.15.32:0/3921196523 <== osd.0 v2:172.21.15.32:6816/3066156116 7 ==== osd_op_reply(7 602.00000000 [create] v122'1134 uv1108 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f6540019040 con 0x5646a0e69d20 2024-08-03T21:00:06.166 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.165+0000 7f654cb20f40 1 -- 172.21.15.32:0/3921196523 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:8 12.10 12:0bd6d154:::602.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e122) -- 0x5646a0f154c0 con 0x5646a0e69d20 2024-08-03T21:00:06.168 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.166+0000 7f6545e00640 1 -- 172.21.15.32:0/3921196523 <== osd.0 v2:172.21.15.32:6816/3066156116 8 ==== osd_op_reply(8 602.00000000 [delete] v122'1135 uv1135 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f6540019040 con 0x5646a0e69d20 2024-08-03T21:00:06.168 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.166+0000 7f654cb20f40 1 -- 172.21.15.32:0/3921196523 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:9 12.10 12:0bd6d154:::602.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122) -- 0x5646a0f154c0 con 0x5646a0e69d20 2024-08-03T21:00:06.170 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::602.00000000:head# 2024-08-03T21:00:06.170 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.168+0000 7f6545e00640 1 -- 172.21.15.32:0/3921196523 <== osd.0 v2:172.21.15.32:6816/3066156116 9 ==== osd_op_reply(9 602.00000000 [create] v122'1136 uv1136 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f6540019040 con 0x5646a0e69d20 2024-08-03T21:00:06.170 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.168+0000 7f654cb20f40 1 -- 172.21.15.32:0/3921196523 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:10 12.10 12:0bd6d154:::602.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122) -- 0x5646a0f154c0 con 0x5646a0e69d20 2024-08-03T21:00:06.172 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.171+0000 7f6545e00640 1 -- 172.21.15.32:0/3921196523 <== osd.0 v2:172.21.15.32:6816/3066156116 10 ==== osd_op_reply(10 602.00000000 [setxattr (30)] v122'1137 uv1137 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f6540019040 con 0x5646a0e69d20 2024-08-03T21:00:06.172 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.171+0000 7f654cb20f40 1 -- 172.21.15.32:0/3921196523 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:11 12.10 12:0bd6d154:::602.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122) -- 0x5646a0f154c0 con 0x5646a0e69d20 2024-08-03T21:00:06.174 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.172+0000 7f6545e00640 1 -- 172.21.15.32:0/3921196523 <== osd.0 v2:172.21.15.32:6816/3066156116 11 ==== osd_op_reply(11 602.00000000 [setxattr (62)] v122'1138 uv1138 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f6540019040 con 0x5646a0e69d20 2024-08-03T21:00:06.174 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.172+0000 7f654cb20f40 1 -- 172.21.15.32:0/3921196523 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:12 12.10 12:0bd6d154:::602.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122) -- 0x5646a0f154c0 con 0x5646a0e69d20 2024-08-03T21:00:06.176 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.175+0000 7f6545e00640 1 -- 172.21.15.32:0/3921196523 <== osd.0 v2:172.21.15.32:6816/3066156116 12 ==== osd_op_reply(12 602.00000000 [omap-set-header] v122'1139 uv1139 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f6540019040 con 0x5646a0e69d20 2024-08-03T21:00:06.176 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.175+0000 7f654cb20f40 1 -- 172.21.15.32:0/3921196523 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:13 12.10 12:0cf6ad0f:::10000000000.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122) -- 0x5646a0f154c0 con 0x5646a0e69d20 2024-08-03T21:00:06.178 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.176+0000 7f6545e00640 1 -- 172.21.15.32:0/3921196523 <== osd.0 v2:172.21.15.32:6816/3066156116 13 ==== osd_op_reply(13 10000000000.00000000 [create] v122'1140 uv1115 ondisk = -17 ((17) File exists)) ==== 164+0+0 (crc 0 0 0) 0x7f6540019040 con 0x5646a0e69d20 2024-08-03T21:00:06.178 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.177+0000 7f654cb20f40 1 -- 172.21.15.32:0/3921196523 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:14 12.10 12:0cf6ad0f:::10000000000.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e122) -- 0x5646a0f154c0 con 0x5646a0e69d20 2024-08-03T21:00:06.179 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.178+0000 7f6545e00640 1 -- 172.21.15.32:0/3921196523 <== osd.0 v2:172.21.15.32:6816/3066156116 14 ==== osd_op_reply(14 10000000000.00000000 [delete] v122'1141 uv1141 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f6540019040 con 0x5646a0e69d20 2024-08-03T21:00:06.180 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.178+0000 7f654cb20f40 1 -- 172.21.15.32:0/3921196523 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:15 12.10 12:0cf6ad0f:::10000000000.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122) -- 0x5646a0f154c0 con 0x5646a0e69d20 2024-08-03T21:00:06.182 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.180+0000 7f6545e00640 1 -- 172.21.15.32:0/3921196523 <== osd.0 v2:172.21.15.32:6816/3066156116 15 ==== osd_op_reply(15 10000000000.00000000 [create] v122'1142 uv1142 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f6540019040 con 0x5646a0e69d20 2024-08-03T21:00:06.182 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::10000000000.00000000:head# 2024-08-03T21:00:06.182 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.180+0000 7f654cb20f40 1 -- 172.21.15.32:0/3921196523 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:16 12.10 12:0cf6ad0f:::10000000000.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122) -- 0x5646a0f154c0 con 0x5646a0e69d20 2024-08-03T21:00:06.183 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.182+0000 7f6545e00640 1 -- 172.21.15.32:0/3921196523 <== osd.0 v2:172.21.15.32:6816/3066156116 16 ==== osd_op_reply(16 10000000000.00000000 [setxattr (30)] v122'1143 uv1143 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f6540019040 con 0x5646a0e69d20 2024-08-03T21:00:06.184 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.182+0000 7f654cb20f40 1 -- 172.21.15.32:0/3921196523 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:17 12.10 12:0cf6ad0f:::10000000000.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122) -- 0x5646a0f154c0 con 0x5646a0e69d20 2024-08-03T21:00:06.185 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.184+0000 7f6545e00640 1 -- 172.21.15.32:0/3921196523 <== osd.0 v2:172.21.15.32:6816/3066156116 17 ==== osd_op_reply(17 10000000000.00000000 [setxattr (62)] v122'1144 uv1144 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f6540019040 con 0x5646a0e69d20 2024-08-03T21:00:06.185 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.184+0000 7f654cb20f40 1 -- 172.21.15.32:0/3921196523 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:18 12.10 12:0cf6ad0f:::10000000000.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122) -- 0x5646a0f154c0 con 0x5646a0e69d20 2024-08-03T21:00:06.187 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.186+0000 7f6545e00640 1 -- 172.21.15.32:0/3921196523 <== osd.0 v2:172.21.15.32:6816/3066156116 18 ==== osd_op_reply(18 10000000000.00000000 [omap-set-header] v122'1145 uv1145 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f6540019040 con 0x5646a0e69d20 2024-08-03T21:00:06.187 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.186+0000 7f654cb20f40 1 -- 172.21.15.32:0/3921196523 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:19 12.10 12:0cf6ad0f:::10000000000.00000000:head [omap-set-vals in=593b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122) -- 0x5646a0f154c0 con 0x5646a0e69d20 2024-08-03T21:00:06.189 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.188+0000 7f6545e00640 1 -- 172.21.15.32:0/3921196523 <== osd.0 v2:172.21.15.32:6816/3066156116 19 ==== osd_op_reply(19 10000000000.00000000 [omap-set-vals] v122'1146 uv1146 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f6540019040 con 0x5646a0e69d20 2024-08-03T21:00:06.189 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.188+0000 7f654cb20f40 1 -- 172.21.15.32:0/3921196523 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:20 12.10 12:0d82a743:::600.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122) -- 0x5646a0f154c0 con 0x5646a0e69d20 2024-08-03T21:00:06.191 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.190+0000 7f6545e00640 1 -- 172.21.15.32:0/3921196523 <== osd.0 v2:172.21.15.32:6816/3066156116 20 ==== osd_op_reply(20 600.00000000 [create] v122'1147 uv1121 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f6540019040 con 0x5646a0e69d20 2024-08-03T21:00:06.191 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.190+0000 7f654cb20f40 1 -- 172.21.15.32:0/3921196523 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:21 12.10 12:0d82a743:::600.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e122) -- 0x5646a0f154c0 con 0x5646a0e69d20 2024-08-03T21:00:06.193 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.191+0000 7f6545e00640 1 -- 172.21.15.32:0/3921196523 <== osd.0 v2:172.21.15.32:6816/3066156116 21 ==== osd_op_reply(21 600.00000000 [delete] v122'1148 uv1148 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f6540019040 con 0x5646a0e69d20 2024-08-03T21:00:06.193 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.191+0000 7f654cb20f40 1 -- 172.21.15.32:0/3921196523 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:22 12.10 12:0d82a743:::600.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122) -- 0x5646a0f154c0 con 0x5646a0e69d20 2024-08-03T21:00:06.194 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::600.00000000:head# 2024-08-03T21:00:06.195 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.193+0000 7f6545e00640 1 -- 172.21.15.32:0/3921196523 <== osd.0 v2:172.21.15.32:6816/3066156116 22 ==== osd_op_reply(22 600.00000000 [create] v122'1149 uv1149 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f6540019040 con 0x5646a0e69d20 2024-08-03T21:00:06.195 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.193+0000 7f654cb20f40 1 -- 172.21.15.32:0/3921196523 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:23 12.10 12:0d82a743:::600.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122) -- 0x5646a0f154c0 con 0x5646a0e69d20 2024-08-03T21:00:06.196 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.195+0000 7f6545e00640 1 -- 172.21.15.32:0/3921196523 <== osd.0 v2:172.21.15.32:6816/3066156116 23 ==== osd_op_reply(23 600.00000000 [setxattr (30)] v122'1150 uv1150 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f6540019040 con 0x5646a0e69d20 2024-08-03T21:00:06.196 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.195+0000 7f654cb20f40 1 -- 172.21.15.32:0/3921196523 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:24 12.10 12:0d82a743:::600.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122) -- 0x5646a0f154c0 con 0x5646a0e69d20 2024-08-03T21:00:06.198 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.197+0000 7f6545e00640 1 -- 172.21.15.32:0/3921196523 <== osd.0 v2:172.21.15.32:6816/3066156116 24 ==== osd_op_reply(24 600.00000000 [setxattr (62)] v122'1151 uv1151 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f6540019040 con 0x5646a0e69d20 2024-08-03T21:00:06.198 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.197+0000 7f654cb20f40 1 -- 172.21.15.32:0/3921196523 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:25 12.10 12:0d82a743:::600.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122) -- 0x5646a0f154c0 con 0x5646a0e69d20 2024-08-03T21:00:06.200 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.198+0000 7f6545e00640 1 -- 172.21.15.32:0/3921196523 <== osd.0 v2:172.21.15.32:6816/3066156116 25 ==== osd_op_reply(25 600.00000000 [omap-set-header] v122'1152 uv1152 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f6540019040 con 0x5646a0e69d20 2024-08-03T21:00:06.200 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.198+0000 7f654cb20f40 1 -- 172.21.15.32:0/3921196523 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:26 12.10 12:0d89b25e:::603.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122) -- 0x5646a0f154c0 con 0x5646a0e69d20 2024-08-03T21:00:06.202 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.200+0000 7f6545e00640 1 -- 172.21.15.32:0/3921196523 <== osd.0 v2:172.21.15.32:6816/3066156116 26 ==== osd_op_reply(26 603.00000000 [create] v122'1153 uv1127 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f6540019040 con 0x5646a0e69d20 2024-08-03T21:00:06.202 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.200+0000 7f654cb20f40 1 -- 172.21.15.32:0/3921196523 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:27 12.10 12:0d89b25e:::603.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e122) -- 0x5646a0f154c0 con 0x5646a0e69d20 2024-08-03T21:00:06.203 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.202+0000 7f6545e00640 1 -- 172.21.15.32:0/3921196523 <== osd.0 v2:172.21.15.32:6816/3066156116 27 ==== osd_op_reply(27 603.00000000 [delete] v122'1154 uv1154 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f6540019040 con 0x5646a0e69d20 2024-08-03T21:00:06.204 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.202+0000 7f654cb20f40 1 -- 172.21.15.32:0/3921196523 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:28 12.10 12:0d89b25e:::603.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122) -- 0x5646a0f154c0 con 0x5646a0e69d20 2024-08-03T21:00:06.205 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::603.00000000:head# 2024-08-03T21:00:06.205 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.204+0000 7f6545e00640 1 -- 172.21.15.32:0/3921196523 <== osd.0 v2:172.21.15.32:6816/3066156116 28 ==== osd_op_reply(28 603.00000000 [create] v122'1155 uv1155 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f6540019040 con 0x5646a0e69d20 2024-08-03T21:00:06.205 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.204+0000 7f654cb20f40 1 -- 172.21.15.32:0/3921196523 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:29 12.10 12:0d89b25e:::603.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122) -- 0x5646a0f154c0 con 0x5646a0e69d20 2024-08-03T21:00:06.207 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.205+0000 7f6545e00640 1 -- 172.21.15.32:0/3921196523 <== osd.0 v2:172.21.15.32:6816/3066156116 29 ==== osd_op_reply(29 603.00000000 [setxattr (30)] v122'1156 uv1156 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f6540019040 con 0x5646a0e69d20 2024-08-03T21:00:06.207 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.205+0000 7f654cb20f40 1 -- 172.21.15.32:0/3921196523 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:30 12.10 12:0d89b25e:::603.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122) -- 0x5646a0f154c0 con 0x5646a0e69d20 2024-08-03T21:00:06.208 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.207+0000 7f6545e00640 1 -- 172.21.15.32:0/3921196523 <== osd.0 v2:172.21.15.32:6816/3066156116 30 ==== osd_op_reply(30 603.00000000 [setxattr (62)] v122'1157 uv1157 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f6540019040 con 0x5646a0e69d20 2024-08-03T21:00:06.208 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.207+0000 7f654cb20f40 1 -- 172.21.15.32:0/3921196523 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:31 12.10 12:0d89b25e:::603.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122) -- 0x5646a0f154c0 con 0x5646a0e69d20 2024-08-03T21:00:06.210 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.208+0000 7f6545e00640 1 -- 172.21.15.32:0/3921196523 <== osd.0 v2:172.21.15.32:6816/3066156116 31 ==== osd_op_reply(31 603.00000000 [omap-set-header] v122'1158 uv1158 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f6540019040 con 0x5646a0e69d20 2024-08-03T21:00:06.210 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.208+0000 7f654cb20f40 1 --2- 172.21.15.32:0/3921196523 >> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] conn(0x5646a0f158a0 0x5646a0f189f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:00:06.210 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.208+0000 7f654cb20f40 1 -- 172.21.15.32:0/3921196523 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:32 12.14 12:287034c4:::1.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122) -- 0x5646a0f18f30 con 0x5646a0f158a0 2024-08-03T21:00:06.210 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.209+0000 7f6545400640 1 --2- 172.21.15.32:0/3921196523 >> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] conn(0x5646a0f158a0 0x5646a0f189f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:00:06.211 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.209+0000 7f6545400640 1 --2- 172.21.15.32:0/3921196523 >> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] conn(0x5646a0f158a0 0x5646a0f189f0 crc :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T21:00:06.213 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.212+0000 7f6545400640 1 -- 172.21.15.32:0/3921196523 <== osd.5 v2:172.21.15.118:6810/2724891770 1 ==== osd_op_reply(32 1.00000000.inode [create] v122'824 uv803 ondisk = -17 ((17) File exists)) ==== 160+0+0 (crc 0 0 0) 0x7f653801c020 con 0x5646a0f158a0 2024-08-03T21:00:06.213 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.212+0000 7f654cb20f40 1 -- 172.21.15.32:0/3921196523 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:33 12.14 12:287034c4:::1.00000000.inode:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e122) -- 0x5646a0f192d0 con 0x5646a0f158a0 2024-08-03T21:00:06.215 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.214+0000 7f6545400640 1 -- 172.21.15.32:0/3921196523 <== osd.5 v2:172.21.15.118:6810/2724891770 2 ==== osd_op_reply(33 1.00000000.inode [delete] v122'825 uv825 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f653801c020 con 0x5646a0f158a0 2024-08-03T21:00:06.215 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.214+0000 7f654cb20f40 1 -- 172.21.15.32:0/3921196523 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:34 12.14 12:287034c4:::1.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122) -- 0x5646a0f19670 con 0x5646a0f158a0 2024-08-03T21:00:06.217 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::1.00000000.inode:head# 2024-08-03T21:00:06.218 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.216+0000 7f6545400640 1 -- 172.21.15.32:0/3921196523 <== osd.5 v2:172.21.15.118:6810/2724891770 3 ==== osd_op_reply(34 1.00000000.inode [create] v122'826 uv826 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f653801c020 con 0x5646a0f158a0 2024-08-03T21:00:06.218 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.216+0000 7f654cb20f40 1 -- 172.21.15.32:0/3921196523 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:35 12.14 12:287034c4:::1.00000000.inode:head [write 0~582 in=582b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122) -- 0x5646a0f19a10 con 0x5646a0f158a0 2024-08-03T21:00:06.219 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.218+0000 7f6545400640 1 -- 172.21.15.32:0/3921196523 <== osd.5 v2:172.21.15.118:6810/2724891770 4 ==== osd_op_reply(35 1.00000000.inode [write 0~582] v122'827 uv827 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f653801c020 con 0x5646a0f158a0 2024-08-03T21:00:06.219 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.218+0000 7f654cb20f40 1 -- 172.21.15.32:0/3921196523 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:36 12.14 12:287034c4:::1.00000000.inode:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122) -- 0x5646a0f19db0 con 0x5646a0f158a0 2024-08-03T21:00:06.221 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.220+0000 7f6545400640 1 -- 172.21.15.32:0/3921196523 <== osd.5 v2:172.21.15.118:6810/2724891770 5 ==== osd_op_reply(36 1.00000000.inode [omap-set-header] v122'828 uv828 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f653801c020 con 0x5646a0f158a0 2024-08-03T21:00:06.221 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.220+0000 7f654cb20f40 1 -- 172.21.15.32:0/3921196523 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:37 12.14 12:292cf221:::200.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122) -- 0x5646a0f1a150 con 0x5646a0f158a0 2024-08-03T21:00:06.223 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.221+0000 7f6545400640 1 -- 172.21.15.32:0/3921196523 <== osd.5 v2:172.21.15.118:6810/2724891770 6 ==== osd_op_reply(37 200.00000000 [create] v122'829 uv822 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f653801c020 con 0x5646a0f158a0 2024-08-03T21:00:06.223 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.221+0000 7f654cb20f40 1 -- 172.21.15.32:0/3921196523 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:38 12.14 12:292cf221:::200.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e122) -- 0x5646a0f1a4f0 con 0x5646a0f158a0 2024-08-03T21:00:06.225 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.223+0000 7f6545400640 1 -- 172.21.15.32:0/3921196523 <== osd.5 v2:172.21.15.118:6810/2724891770 7 ==== osd_op_reply(38 200.00000000 [delete] v122'830 uv830 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f653801c020 con 0x5646a0f158a0 2024-08-03T21:00:06.225 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.223+0000 7f654cb20f40 1 -- 172.21.15.32:0/3921196523 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:39 12.14 12:292cf221:::200.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122) -- 0x5646a0f1a4f0 con 0x5646a0f158a0 2024-08-03T21:00:06.226 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::200.00000000:head# 2024-08-03T21:00:06.227 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.225+0000 7f6545400640 1 -- 172.21.15.32:0/3921196523 <== osd.5 v2:172.21.15.118:6810/2724891770 8 ==== osd_op_reply(39 200.00000000 [create] v122'831 uv831 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f653801c020 con 0x5646a0f158a0 2024-08-03T21:00:06.227 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.225+0000 7f654cb20f40 1 -- 172.21.15.32:0/3921196523 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:40 12.14 12:292cf221:::200.00000000:head [write 0~90 in=90b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122) -- 0x5646a0f1a4f0 con 0x5646a0f158a0 2024-08-03T21:00:06.228 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.227+0000 7f6545400640 1 -- 172.21.15.32:0/3921196523 <== osd.5 v2:172.21.15.118:6810/2724891770 9 ==== osd_op_reply(40 200.00000000 [write 0~90] v122'832 uv832 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f653801c020 con 0x5646a0f158a0 2024-08-03T21:00:06.228 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.227+0000 7f654cb20f40 1 -- 172.21.15.32:0/3921196523 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:41 12.14 12:292cf221:::200.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122) -- 0x5646a0f1a4f0 con 0x5646a0f158a0 2024-08-03T21:00:06.230 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.228+0000 7f6545400640 1 -- 172.21.15.32:0/3921196523 <== osd.5 v2:172.21.15.118:6810/2724891770 10 ==== osd_op_reply(41 200.00000000 [omap-set-header] v122'833 uv833 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f653801c020 con 0x5646a0f158a0 2024-08-03T21:00:06.230 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.228+0000 7f654cb20f40 1 -- 172.21.15.32:0/3921196523 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:42 12.14 12:2e2fa760:::200.00000001:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122) -- 0x5646a0f1a4f0 con 0x5646a0f158a0 2024-08-03T21:00:06.231 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.230+0000 7f6545400640 1 -- 172.21.15.32:0/3921196523 <== osd.5 v2:172.21.15.118:6810/2724891770 11 ==== osd_op_reply(42 200.00000001 [create] v122'834 uv823 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f653801c020 con 0x5646a0f158a0 2024-08-03T21:00:06.232 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.230+0000 7f654cb20f40 1 -- 172.21.15.32:0/3921196523 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:43 12.14 12:2e2fa760:::200.00000001:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e122) -- 0x5646a0f1a4f0 con 0x5646a0f158a0 2024-08-03T21:00:06.233 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.232+0000 7f6545400640 1 -- 172.21.15.32:0/3921196523 <== osd.5 v2:172.21.15.118:6810/2724891770 12 ==== osd_op_reply(43 200.00000001 [delete] v122'835 uv835 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f653801c020 con 0x5646a0f158a0 2024-08-03T21:00:06.233 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.232+0000 7f654cb20f40 1 -- 172.21.15.32:0/3921196523 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:44 12.14 12:2e2fa760:::200.00000001:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122) -- 0x5646a0f1a4f0 con 0x5646a0f158a0 2024-08-03T21:00:06.235 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::200.00000001:head# 2024-08-03T21:00:06.235 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.233+0000 7f6545400640 1 -- 172.21.15.32:0/3921196523 <== osd.5 v2:172.21.15.118:6810/2724891770 13 ==== osd_op_reply(44 200.00000001 [create] v122'836 uv836 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f653801c020 con 0x5646a0f158a0 2024-08-03T21:00:06.235 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.233+0000 7f654cb20f40 1 -- 172.21.15.32:0/3921196523 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:45 12.14 12:2e2fa760:::200.00000001:head [write 0~18118 in=18118b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122) -- 0x5646a0f1ebe0 con 0x5646a0f158a0 2024-08-03T21:00:06.237 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.235+0000 7f6545400640 1 -- 172.21.15.32:0/3921196523 <== osd.5 v2:172.21.15.118:6810/2724891770 14 ==== osd_op_reply(45 200.00000001 [write 0~18118] v122'837 uv837 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f653801c020 con 0x5646a0f158a0 2024-08-03T21:00:06.237 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.235+0000 7f654cb20f40 1 -- 172.21.15.32:0/3921196523 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:46 12.14 12:2e2fa760:::200.00000001:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122) -- 0x5646a0f1ebe0 con 0x5646a0f158a0 2024-08-03T21:00:06.239 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.237+0000 7f6545400640 1 -- 172.21.15.32:0/3921196523 <== osd.5 v2:172.21.15.118:6810/2724891770 15 ==== osd_op_reply(46 200.00000001 [omap-set-header] v122'838 uv838 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f653801c020 con 0x5646a0f158a0 2024-08-03T21:00:06.240 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.237+0000 7f654cb20f40 1 -- 172.21.15.32:0/3921196523 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:47 12.14 12:2f55791f:::606.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122) -- 0x5646a0f1a4f0 con 0x5646a0f158a0 2024-08-03T21:00:06.240 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.239+0000 7f6545400640 1 -- 172.21.15.32:0/3921196523 <== osd.5 v2:172.21.15.118:6810/2724891770 16 ==== osd_op_reply(47 606.00000000 [create] v122'839 uv819 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f653801c020 con 0x5646a0f158a0 2024-08-03T21:00:06.240 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.239+0000 7f654cb20f40 1 -- 172.21.15.32:0/3921196523 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:48 12.14 12:2f55791f:::606.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e122) -- 0x5646a0f1a4f0 con 0x5646a0f158a0 2024-08-03T21:00:06.242 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.241+0000 7f6545400640 1 -- 172.21.15.32:0/3921196523 <== osd.5 v2:172.21.15.118:6810/2724891770 17 ==== osd_op_reply(48 606.00000000 [delete] v122'840 uv840 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f653801c020 con 0x5646a0f158a0 2024-08-03T21:00:06.242 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.241+0000 7f654cb20f40 1 -- 172.21.15.32:0/3921196523 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:49 12.14 12:2f55791f:::606.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122) -- 0x5646a0f1a4f0 con 0x5646a0f158a0 2024-08-03T21:00:06.244 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::606.00000000:head# 2024-08-03T21:00:06.244 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.242+0000 7f6545400640 1 -- 172.21.15.32:0/3921196523 <== osd.5 v2:172.21.15.118:6810/2724891770 18 ==== osd_op_reply(49 606.00000000 [create] v122'841 uv841 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f653801c020 con 0x5646a0f158a0 2024-08-03T21:00:06.244 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.243+0000 7f654cb20f40 1 -- 172.21.15.32:0/3921196523 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:50 12.14 12:2f55791f:::606.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122) -- 0x5646a0f1b500 con 0x5646a0f158a0 2024-08-03T21:00:06.246 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.244+0000 7f6545400640 1 -- 172.21.15.32:0/3921196523 <== osd.5 v2:172.21.15.118:6810/2724891770 19 ==== osd_op_reply(50 606.00000000 [setxattr (30)] v122'842 uv842 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f653801c020 con 0x5646a0f158a0 2024-08-03T21:00:06.246 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.244+0000 7f654cb20f40 1 -- 172.21.15.32:0/3921196523 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:51 12.14 12:2f55791f:::606.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122) -- 0x5646a0f1b500 con 0x5646a0f158a0 2024-08-03T21:00:06.247 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.246+0000 7f6545400640 1 -- 172.21.15.32:0/3921196523 <== osd.5 v2:172.21.15.118:6810/2724891770 20 ==== osd_op_reply(51 606.00000000 [setxattr (62)] v122'843 uv843 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f653801c020 con 0x5646a0f158a0 2024-08-03T21:00:06.247 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.246+0000 7f654cb20f40 1 -- 172.21.15.32:0/3921196523 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:52 12.14 12:2f55791f:::606.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122) -- 0x5646a0f1a4f0 con 0x5646a0f158a0 2024-08-03T21:00:06.249 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.248+0000 7f6545400640 1 -- 172.21.15.32:0/3921196523 <== osd.5 v2:172.21.15.118:6810/2724891770 21 ==== osd_op_reply(52 606.00000000 [omap-set-header] v122'844 uv844 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f653801c020 con 0x5646a0f158a0 2024-08-03T21:00:06.249 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.248+0000 7f654cb20f40 1 -- 172.21.15.32:0/3921196523 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:53 12.1c 12:3ed09add:::607.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122) -- 0x5646a0f1a4f0 con 0x5646a0e69d20 2024-08-03T21:00:06.250 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.249+0000 7f6545e00640 1 -- 172.21.15.32:0/3921196523 <== osd.0 v2:172.21.15.32:6816/3066156116 32 ==== osd_op_reply(53 607.00000000 [create] v122'219 uv218 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f6540019040 con 0x5646a0e69d20 2024-08-03T21:00:06.251 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.249+0000 7f654cb20f40 1 -- 172.21.15.32:0/3921196523 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:54 12.1c 12:3ed09add:::607.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e122) -- 0x5646a0f1a4f0 con 0x5646a0e69d20 2024-08-03T21:00:06.252 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.250+0000 7f6545e00640 1 -- 172.21.15.32:0/3921196523 <== osd.0 v2:172.21.15.32:6816/3066156116 33 ==== osd_op_reply(54 607.00000000 [delete] v122'220 uv220 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f6540019040 con 0x5646a0e69d20 2024-08-03T21:00:06.252 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.250+0000 7f654cb20f40 1 -- 172.21.15.32:0/3921196523 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:55 12.1c 12:3ed09add:::607.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122) -- 0x5646a0f1a4f0 con 0x5646a0e69d20 2024-08-03T21:00:06.253 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::607.00000000:head# 2024-08-03T21:00:06.254 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.252+0000 7f6545e00640 1 -- 172.21.15.32:0/3921196523 <== osd.0 v2:172.21.15.32:6816/3066156116 34 ==== osd_op_reply(55 607.00000000 [create] v122'221 uv221 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f6540019040 con 0x5646a0e69d20 2024-08-03T21:00:06.254 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.252+0000 7f654cb20f40 1 -- 172.21.15.32:0/3921196523 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:56 12.1c 12:3ed09add:::607.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122) -- 0x5646a0f1b500 con 0x5646a0e69d20 2024-08-03T21:00:06.255 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.253+0000 7f6545e00640 1 -- 172.21.15.32:0/3921196523 <== osd.0 v2:172.21.15.32:6816/3066156116 35 ==== osd_op_reply(56 607.00000000 [setxattr (30)] v122'222 uv222 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f6540019040 con 0x5646a0e69d20 2024-08-03T21:00:06.255 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.253+0000 7f654cb20f40 1 -- 172.21.15.32:0/3921196523 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:57 12.1c 12:3ed09add:::607.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122) -- 0x5646a0f1b500 con 0x5646a0e69d20 2024-08-03T21:00:06.256 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.254+0000 7f6545e00640 1 -- 172.21.15.32:0/3921196523 <== osd.0 v2:172.21.15.32:6816/3066156116 36 ==== osd_op_reply(57 607.00000000 [setxattr (62)] v122'223 uv223 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f6540019040 con 0x5646a0e69d20 2024-08-03T21:00:06.256 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.254+0000 7f654cb20f40 1 -- 172.21.15.32:0/3921196523 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:58 12.1c 12:3ed09add:::607.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122) -- 0x5646a0f1a4f0 con 0x5646a0e69d20 2024-08-03T21:00:06.257 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.256+0000 7f6545e00640 1 -- 172.21.15.32:0/3921196523 <== osd.0 v2:172.21.15.32:6816/3066156116 37 ==== osd_op_reply(58 607.00000000 [omap-set-header] v122'224 uv224 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f6540019040 con 0x5646a0e69d20 2024-08-03T21:00:06.257 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.256+0000 7f654cb20f40 1 --2- 172.21.15.32:0/3921196523 >> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] conn(0x5646a0f1ab70 0x5646a0f1cf90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:00:06.257 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.256+0000 7f654cb20f40 1 -- 172.21.15.32:0/3921196523 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:59 12.6 12:654134d2:::mds0_openfiles.0:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122) -- 0x5646a0f1d560 con 0x5646a0f1ab70 2024-08-03T21:00:06.258 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.256+0000 7f6544a00640 1 --2- 172.21.15.32:0/3921196523 >> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] conn(0x5646a0f1ab70 0x5646a0f1cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:00:06.258 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.257+0000 7f6544a00640 1 --2- 172.21.15.32:0/3921196523 >> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] conn(0x5646a0f1ab70 0x5646a0f1cf90 crc :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T21:00:06.259 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.258+0000 7f6544a00640 1 -- 172.21.15.32:0/3921196523 <== osd.1 v2:172.21.15.32:6824/1551241759 1 ==== osd_op_reply(59 mds0_openfiles.0 [create] v122'176 uv175 ondisk = -17 ((17) File exists)) ==== 160+0+0 (crc 0 0 0) 0x7f6534056870 con 0x5646a0f1ab70 2024-08-03T21:00:06.259 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.258+0000 7f654cb20f40 1 -- 172.21.15.32:0/3921196523 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:60 12.6 12:654134d2:::mds0_openfiles.0:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e122) -- 0x5646a0f1da70 con 0x5646a0f1ab70 2024-08-03T21:00:06.261 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.259+0000 7f6544a00640 1 -- 172.21.15.32:0/3921196523 <== osd.1 v2:172.21.15.32:6824/1551241759 2 ==== osd_op_reply(60 mds0_openfiles.0 [delete] v122'177 uv177 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f6534056870 con 0x5646a0f1ab70 2024-08-03T21:00:06.261 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.259+0000 7f654cb20f40 1 -- 172.21.15.32:0/3921196523 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:61 12.6 12:654134d2:::mds0_openfiles.0:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122) -- 0x5646a0f1df70 con 0x5646a0f1ab70 2024-08-03T21:00:06.262 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.261+0000 7f6544a00640 1 -- 172.21.15.32:0/3921196523 <== osd.1 v2:172.21.15.32:6824/1551241759 3 ==== osd_op_reply(61 mds0_openfiles.0 [create] v122'178 uv178 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f6534056870 con 0x5646a0f1ab70 2024-08-03T21:00:06.262 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds0_openfiles.0:head# 2024-08-03T21:00:06.263 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.261+0000 7f654cb20f40 1 -- 172.21.15.32:0/3921196523 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:62 12.6 12:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122) -- 0x5646a0f1e3e0 con 0x5646a0f1ab70 2024-08-03T21:00:06.263 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.262+0000 7f6544a00640 1 -- 172.21.15.32:0/3921196523 <== osd.1 v2:172.21.15.32:6824/1551241759 4 ==== osd_op_reply(62 mds0_openfiles.0 [omap-set-header] v122'179 uv179 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f6534056870 con 0x5646a0f1ab70 2024-08-03T21:00:06.264 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.262+0000 7f654cb20f40 1 -- 172.21.15.32:0/3921196523 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:63 12.e 12:717a0223:::608.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122) -- 0x5646a0f1e810 con 0x5646a0e69d20 2024-08-03T21:00:06.265 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.264+0000 7f6545e00640 1 -- 172.21.15.32:0/3921196523 <== osd.0 v2:172.21.15.32:6816/3066156116 38 ==== osd_op_reply(63 608.00000000 [create] v122'219 uv218 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f6540019040 con 0x5646a0e69d20 2024-08-03T21:00:06.265 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.264+0000 7f654cb20f40 1 -- 172.21.15.32:0/3921196523 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:64 12.e 12:717a0223:::608.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e122) -- 0x5646a0f1e810 con 0x5646a0e69d20 2024-08-03T21:00:06.267 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.265+0000 7f6545e00640 1 -- 172.21.15.32:0/3921196523 <== osd.0 v2:172.21.15.32:6816/3066156116 39 ==== osd_op_reply(64 608.00000000 [delete] v122'220 uv220 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f6540019040 con 0x5646a0e69d20 2024-08-03T21:00:06.267 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.265+0000 7f654cb20f40 1 -- 172.21.15.32:0/3921196523 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:65 12.e 12:717a0223:::608.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122) -- 0x5646a0f1e810 con 0x5646a0e69d20 2024-08-03T21:00:06.268 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::608.00000000:head# 2024-08-03T21:00:06.269 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.267+0000 7f6545e00640 1 -- 172.21.15.32:0/3921196523 <== osd.0 v2:172.21.15.32:6816/3066156116 40 ==== osd_op_reply(65 608.00000000 [create] v122'221 uv221 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f6540019040 con 0x5646a0e69d20 2024-08-03T21:00:06.269 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.267+0000 7f654cb20f40 1 -- 172.21.15.32:0/3921196523 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:66 12.e 12:717a0223:::608.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122) -- 0x5646a0f1f820 con 0x5646a0e69d20 2024-08-03T21:00:06.270 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.269+0000 7f6545e00640 1 -- 172.21.15.32:0/3921196523 <== osd.0 v2:172.21.15.32:6816/3066156116 41 ==== osd_op_reply(66 608.00000000 [setxattr (30)] v122'222 uv222 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f6540019040 con 0x5646a0e69d20 2024-08-03T21:00:06.270 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.269+0000 7f654cb20f40 1 -- 172.21.15.32:0/3921196523 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:67 12.e 12:717a0223:::608.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122) -- 0x5646a0f1f820 con 0x5646a0e69d20 2024-08-03T21:00:06.272 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.270+0000 7f6545e00640 1 -- 172.21.15.32:0/3921196523 <== osd.0 v2:172.21.15.32:6816/3066156116 42 ==== osd_op_reply(67 608.00000000 [setxattr (62)] v122'223 uv223 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f6540019040 con 0x5646a0e69d20 2024-08-03T21:00:06.272 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.270+0000 7f654cb20f40 1 -- 172.21.15.32:0/3921196523 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:68 12.e 12:717a0223:::608.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122) -- 0x5646a0f1e810 con 0x5646a0e69d20 2024-08-03T21:00:06.273 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.272+0000 7f6545e00640 1 -- 172.21.15.32:0/3921196523 <== osd.0 v2:172.21.15.32:6816/3066156116 43 ==== osd_op_reply(68 608.00000000 [omap-set-header] v122'224 uv224 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f6540019040 con 0x5646a0e69d20 2024-08-03T21:00:06.273 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.272+0000 7f654cb20f40 1 --2- 172.21.15.32:0/3921196523 >> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] conn(0x5646a0f1f0e0 0x5646a0f1f4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:00:06.273 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.272+0000 7f654cb20f40 1 -- 172.21.15.32:0/3921196523 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:69 12.15 12:a93a17c2:::604.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122) -- 0x5646a0f21cc0 con 0x5646a0f1f0e0 2024-08-03T21:00:06.274 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.272+0000 7f6545e00640 1 --2- 172.21.15.32:0/3921196523 >> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] conn(0x5646a0f1f0e0 0x5646a0f1f4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:00:06.274 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.273+0000 7f6545e00640 1 --2- 172.21.15.32:0/3921196523 >> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] conn(0x5646a0f1f0e0 0x5646a0f1f4f0 crc :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T21:00:06.276 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.275+0000 7f6545e00640 1 -- 172.21.15.32:0/3921196523 <== osd.6 v2:172.21.15.118:6825/801947386 1 ==== osd_op_reply(69 604.00000000 [create] v122'404 uv398 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f6540019040 con 0x5646a0f1f0e0 2024-08-03T21:00:06.276 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.275+0000 7f654cb20f40 1 -- 172.21.15.32:0/3921196523 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:70 12.15 12:a93a17c2:::604.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e122) -- 0x5646a0f21cc0 con 0x5646a0f1f0e0 2024-08-03T21:00:06.278 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.276+0000 7f6545e00640 1 -- 172.21.15.32:0/3921196523 <== osd.6 v2:172.21.15.118:6825/801947386 2 ==== osd_op_reply(70 604.00000000 [delete] v122'405 uv405 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f6540019040 con 0x5646a0f1f0e0 2024-08-03T21:00:06.278 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.276+0000 7f654cb20f40 1 -- 172.21.15.32:0/3921196523 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:71 12.15 12:a93a17c2:::604.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122) -- 0x5646a0f21cc0 con 0x5646a0f1f0e0 2024-08-03T21:00:06.279 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.278+0000 7f6545e00640 1 -- 172.21.15.32:0/3921196523 <== osd.6 v2:172.21.15.118:6825/801947386 3 ==== osd_op_reply(71 604.00000000 [create] v122'406 uv406 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f6540019040 con 0x5646a0f1f0e0 2024-08-03T21:00:06.279 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::604.00000000:head# 2024-08-03T21:00:06.279 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.278+0000 7f654cb20f40 1 -- 172.21.15.32:0/3921196523 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:72 12.15 12:a93a17c2:::604.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122) -- 0x5646a0f22d30 con 0x5646a0f1f0e0 2024-08-03T21:00:06.281 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.279+0000 7f6545e00640 1 -- 172.21.15.32:0/3921196523 <== osd.6 v2:172.21.15.118:6825/801947386 4 ==== osd_op_reply(72 604.00000000 [setxattr (30)] v122'407 uv407 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f6540019040 con 0x5646a0f1f0e0 2024-08-03T21:00:06.281 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.279+0000 7f654cb20f40 1 -- 172.21.15.32:0/3921196523 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:73 12.15 12:a93a17c2:::604.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122) -- 0x5646a0f22cf0 con 0x5646a0f1f0e0 2024-08-03T21:00:06.282 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.281+0000 7f6545e00640 1 -- 172.21.15.32:0/3921196523 <== osd.6 v2:172.21.15.118:6825/801947386 5 ==== osd_op_reply(73 604.00000000 [setxattr (62)] v122'408 uv408 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f6540019040 con 0x5646a0f1f0e0 2024-08-03T21:00:06.282 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.281+0000 7f654cb20f40 1 -- 172.21.15.32:0/3921196523 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:74 12.15 12:a93a17c2:::604.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122) -- 0x5646a0f22cf0 con 0x5646a0f1f0e0 2024-08-03T21:00:06.284 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.282+0000 7f6545e00640 1 -- 172.21.15.32:0/3921196523 <== osd.6 v2:172.21.15.118:6825/801947386 6 ==== osd_op_reply(74 604.00000000 [omap-set-header] v122'409 uv409 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f6540019040 con 0x5646a0f1f0e0 2024-08-03T21:00:06.284 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.282+0000 7f654cb20f40 1 -- 172.21.15.32:0/3921196523 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:75 12.15 12:aa448500:::500.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122) -- 0x5646a0f22cf0 con 0x5646a0f1f0e0 2024-08-03T21:00:06.285 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.284+0000 7f6545e00640 1 -- 172.21.15.32:0/3921196523 <== osd.6 v2:172.21.15.118:6825/801947386 7 ==== osd_op_reply(75 500.00000000 [create] v122'410 uv403 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f6540019040 con 0x5646a0f1f0e0 2024-08-03T21:00:06.285 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.284+0000 7f654cb20f40 1 -- 172.21.15.32:0/3921196523 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:76 12.15 12:aa448500:::500.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e122) -- 0x5646a0f22cf0 con 0x5646a0f1f0e0 2024-08-03T21:00:06.287 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.285+0000 7f6545e00640 1 -- 172.21.15.32:0/3921196523 <== osd.6 v2:172.21.15.118:6825/801947386 8 ==== osd_op_reply(76 500.00000000 [delete] v122'411 uv411 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f6540019040 con 0x5646a0f1f0e0 2024-08-03T21:00:06.287 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.285+0000 7f654cb20f40 1 -- 172.21.15.32:0/3921196523 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:77 12.15 12:aa448500:::500.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122) -- 0x5646a0f22cf0 con 0x5646a0f1f0e0 2024-08-03T21:00:06.288 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.287+0000 7f6545e00640 1 -- 172.21.15.32:0/3921196523 <== osd.6 v2:172.21.15.118:6825/801947386 9 ==== osd_op_reply(77 500.00000000 [create] v122'412 uv412 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f6540019040 con 0x5646a0f1f0e0 2024-08-03T21:00:06.288 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::500.00000000:head# 2024-08-03T21:00:06.289 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.287+0000 7f654cb20f40 1 -- 172.21.15.32:0/3921196523 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:78 12.15 12:aa448500:::500.00000000:head [write 0~90 in=90b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122) -- 0x5646a0f22cf0 con 0x5646a0f1f0e0 2024-08-03T21:00:06.289 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.288+0000 7f6545e00640 1 -- 172.21.15.32:0/3921196523 <== osd.6 v2:172.21.15.118:6825/801947386 10 ==== osd_op_reply(78 500.00000000 [write 0~90] v122'413 uv413 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f6540019040 con 0x5646a0f1f0e0 2024-08-03T21:00:06.289 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.288+0000 7f654cb20f40 1 -- 172.21.15.32:0/3921196523 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:79 12.15 12:aa448500:::500.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122) -- 0x5646a0f22cf0 con 0x5646a0f1f0e0 2024-08-03T21:00:06.291 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.289+0000 7f6545e00640 1 -- 172.21.15.32:0/3921196523 <== osd.6 v2:172.21.15.118:6825/801947386 11 ==== osd_op_reply(79 500.00000000 [omap-set-header] v122'414 uv414 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f6540019040 con 0x5646a0f1f0e0 2024-08-03T21:00:06.291 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.289+0000 7f654cb20f40 1 --2- 172.21.15.32:0/3921196523 >> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] conn(0x5646a0f22240 0x5646a0f22650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:00:06.291 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.289+0000 7f654cb20f40 1 -- 172.21.15.32:0/3921196523 --> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] -- osd_op(unknown.0.0:80 12.d 12:b50e409b:::mds_snaptable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122) -- 0x5646a0f25120 con 0x5646a0f22240 2024-08-03T21:00:06.291 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.290+0000 7f6545400640 1 --2- 172.21.15.32:0/3921196523 >> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] conn(0x5646a0f22240 0x5646a0f22650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:00:06.292 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.290+0000 7f6545400640 1 --2- 172.21.15.32:0/3921196523 >> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] conn(0x5646a0f22240 0x5646a0f22650 crc :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T21:00:06.293 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.292+0000 7f6545400640 1 -- 172.21.15.32:0/3921196523 <== osd.7 v2:172.21.15.118:6818/653000048 1 ==== osd_op_reply(80 mds_snaptable [create] v122'183 uv182 ondisk = -17 ((17) File exists)) ==== 157+0+0 (crc 0 0 0) 0x7f653801c020 con 0x5646a0f22240 2024-08-03T21:00:06.293 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.292+0000 7f654cb20f40 1 -- 172.21.15.32:0/3921196523 --> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] -- osd_op(unknown.0.0:81 12.d 12:b50e409b:::mds_snaptable:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e122) -- 0x5646a0f25120 con 0x5646a0f22240 2024-08-03T21:00:06.295 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.293+0000 7f6545400640 1 -- 172.21.15.32:0/3921196523 <== osd.7 v2:172.21.15.118:6818/653000048 2 ==== osd_op_reply(81 mds_snaptable [delete] v122'184 uv184 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f653801c020 con 0x5646a0f22240 2024-08-03T21:00:06.295 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.293+0000 7f654cb20f40 1 -- 172.21.15.32:0/3921196523 --> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] -- osd_op(unknown.0.0:82 12.d 12:b50e409b:::mds_snaptable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122) -- 0x5646a0f25510 con 0x5646a0f22240 2024-08-03T21:00:06.296 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.295+0000 7f6545400640 1 -- 172.21.15.32:0/3921196523 <== osd.7 v2:172.21.15.118:6818/653000048 3 ==== osd_op_reply(82 mds_snaptable [create] v122'185 uv185 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f653801c020 con 0x5646a0f22240 2024-08-03T21:00:06.296 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds_snaptable:head# 2024-08-03T21:00:06.296 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.295+0000 7f654cb20f40 1 -- 172.21.15.32:0/3921196523 --> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] -- osd_op(unknown.0.0:83 12.d 12:b50e409b:::mds_snaptable:head [write 0~70 in=70b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122) -- 0x5646a0f25510 con 0x5646a0f22240 2024-08-03T21:00:06.298 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.296+0000 7f6545400640 1 -- 172.21.15.32:0/3921196523 <== osd.7 v2:172.21.15.118:6818/653000048 4 ==== osd_op_reply(83 mds_snaptable [write 0~70] v122'186 uv186 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f653801c020 con 0x5646a0f22240 2024-08-03T21:00:06.298 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.296+0000 7f654cb20f40 1 -- 172.21.15.32:0/3921196523 --> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] -- osd_op(unknown.0.0:84 12.d 12:b50e409b:::mds_snaptable:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122) -- 0x5646a0f25510 con 0x5646a0f22240 2024-08-03T21:00:06.299 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.297+0000 7f6545400640 1 -- 172.21.15.32:0/3921196523 <== osd.7 v2:172.21.15.118:6818/653000048 5 ==== osd_op_reply(84 mds_snaptable [omap-set-header] v122'187 uv187 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f653801c020 con 0x5646a0f22240 2024-08-03T21:00:06.299 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.298+0000 7f654cb20f40 1 -- 172.21.15.32:0/3921196523 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:85 12.1d 12:b871830b:::605.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122) -- 0x5646a0f25510 con 0x5646a0e69d20 2024-08-03T21:00:06.300 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.299+0000 7f6545e00640 1 -- 172.21.15.32:0/3921196523 <== osd.0 v2:172.21.15.32:6816/3066156116 44 ==== osd_op_reply(85 605.00000000 [create] v122'219 uv218 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f6540019040 con 0x5646a0e69d20 2024-08-03T21:00:06.300 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.299+0000 7f654cb20f40 1 -- 172.21.15.32:0/3921196523 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:86 12.1d 12:b871830b:::605.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e122) -- 0x5646a0f25510 con 0x5646a0e69d20 2024-08-03T21:00:06.302 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.300+0000 7f6545e00640 1 -- 172.21.15.32:0/3921196523 <== osd.0 v2:172.21.15.32:6816/3066156116 45 ==== osd_op_reply(86 605.00000000 [delete] v122'220 uv220 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f6540019040 con 0x5646a0e69d20 2024-08-03T21:00:06.302 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.300+0000 7f654cb20f40 1 -- 172.21.15.32:0/3921196523 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:87 12.1d 12:b871830b:::605.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122) -- 0x5646a0f25510 con 0x5646a0e69d20 2024-08-03T21:00:06.303 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::605.00000000:head# 2024-08-03T21:00:06.303 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.302+0000 7f6545e00640 1 -- 172.21.15.32:0/3921196523 <== osd.0 v2:172.21.15.32:6816/3066156116 46 ==== osd_op_reply(87 605.00000000 [create] v122'221 uv221 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f6540019040 con 0x5646a0e69d20 2024-08-03T21:00:06.303 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.302+0000 7f654cb20f40 1 -- 172.21.15.32:0/3921196523 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:88 12.1d 12:b871830b:::605.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122) -- 0x5646a0f26520 con 0x5646a0e69d20 2024-08-03T21:00:06.304 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.303+0000 7f6545e00640 1 -- 172.21.15.32:0/3921196523 <== osd.0 v2:172.21.15.32:6816/3066156116 47 ==== osd_op_reply(88 605.00000000 [setxattr (30)] v122'222 uv222 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f6540019040 con 0x5646a0e69d20 2024-08-03T21:00:06.304 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.303+0000 7f654cb20f40 1 -- 172.21.15.32:0/3921196523 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:89 12.1d 12:b871830b:::605.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122) -- 0x5646a0f26520 con 0x5646a0e69d20 2024-08-03T21:00:06.306 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.304+0000 7f6545e00640 1 -- 172.21.15.32:0/3921196523 <== osd.0 v2:172.21.15.32:6816/3066156116 48 ==== osd_op_reply(89 605.00000000 [setxattr (62)] v122'223 uv223 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f6540019040 con 0x5646a0e69d20 2024-08-03T21:00:06.306 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.304+0000 7f654cb20f40 1 -- 172.21.15.32:0/3921196523 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:90 12.1d 12:b871830b:::605.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122) -- 0x5646a0f25510 con 0x5646a0e69d20 2024-08-03T21:00:06.307 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.306+0000 7f6545e00640 1 -- 172.21.15.32:0/3921196523 <== osd.0 v2:172.21.15.32:6816/3066156116 49 ==== osd_op_reply(90 605.00000000 [omap-set-header] v122'224 uv224 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f6540019040 con 0x5646a0e69d20 2024-08-03T21:00:06.307 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.306+0000 7f654cb20f40 1 -- 172.21.15.32:0/3921196523 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:91 12.13 12:c91d4a1d:::mds0_inotable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122) -- 0x5646a0f25510 con 0x5646a0f1ab70 2024-08-03T21:00:06.308 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.307+0000 7f6544a00640 1 -- 172.21.15.32:0/3921196523 <== osd.1 v2:172.21.15.32:6824/1551241759 5 ==== osd_op_reply(91 mds0_inotable [create] v122'365 uv359 ondisk = -17 ((17) File exists)) ==== 157+0+0 (crc 0 0 0) 0x7f6534056870 con 0x5646a0f1ab70 2024-08-03T21:00:06.309 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.307+0000 7f654cb20f40 1 -- 172.21.15.32:0/3921196523 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:92 12.13 12:c91d4a1d:::mds0_inotable:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e122) -- 0x5646a0f258b0 con 0x5646a0f1ab70 2024-08-03T21:00:06.310 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.308+0000 7f6544a00640 1 -- 172.21.15.32:0/3921196523 <== osd.1 v2:172.21.15.32:6824/1551241759 6 ==== osd_op_reply(92 mds0_inotable [delete] v122'366 uv366 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f6534056870 con 0x5646a0f1ab70 2024-08-03T21:00:06.310 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.308+0000 7f654cb20f40 1 -- 172.21.15.32:0/3921196523 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:93 12.13 12:c91d4a1d:::mds0_inotable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122) -- 0x5646a0f25ca0 con 0x5646a0f1ab70 2024-08-03T21:00:06.311 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds0_inotable:head# 2024-08-03T21:00:06.311 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.310+0000 7f6544a00640 1 -- 172.21.15.32:0/3921196523 <== osd.1 v2:172.21.15.32:6824/1551241759 7 ==== osd_op_reply(93 mds0_inotable [create] v122'367 uv367 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f6534056870 con 0x5646a0f1ab70 2024-08-03T21:00:06.311 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.310+0000 7f654cb20f40 1 -- 172.21.15.32:0/3921196523 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:94 12.13 12:c91d4a1d:::mds0_inotable:head [write 0~34 in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122) -- 0x5646a0f26090 con 0x5646a0f1ab70 2024-08-03T21:00:06.313 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.311+0000 7f6544a00640 1 -- 172.21.15.32:0/3921196523 <== osd.1 v2:172.21.15.32:6824/1551241759 8 ==== osd_op_reply(94 mds0_inotable [write 0~34] v122'368 uv368 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f6534056870 con 0x5646a0f1ab70 2024-08-03T21:00:06.313 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.311+0000 7f654cb20f40 1 -- 172.21.15.32:0/3921196523 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:95 12.13 12:c91d4a1d:::mds0_inotable:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122) -- 0x5646a0f26090 con 0x5646a0f1ab70 2024-08-03T21:00:06.314 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.313+0000 7f6544a00640 1 -- 172.21.15.32:0/3921196523 <== osd.1 v2:172.21.15.32:6824/1551241759 9 ==== osd_op_reply(95 mds0_inotable [omap-set-header] v122'369 uv369 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f6534056870 con 0x5646a0f1ab70 2024-08-03T21:00:06.314 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.313+0000 7f654cb20f40 1 -- 172.21.15.32:0/3921196523 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:96 12.13 12:cd5a64a3:::100.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122) -- 0x5646a0f25db0 con 0x5646a0f1ab70 2024-08-03T21:00:06.315 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.314+0000 7f6544a00640 1 -- 172.21.15.32:0/3921196523 <== osd.1 v2:172.21.15.32:6824/1551241759 10 ==== osd_op_reply(96 100.00000000 [create] v122'370 uv364 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f6534056870 con 0x5646a0f1ab70 2024-08-03T21:00:06.315 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.314+0000 7f654cb20f40 1 -- 172.21.15.32:0/3921196523 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:97 12.13 12:cd5a64a3:::100.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e122) -- 0x5646a0f25d60 con 0x5646a0f1ab70 2024-08-03T21:00:06.317 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.315+0000 7f6544a00640 1 -- 172.21.15.32:0/3921196523 <== osd.1 v2:172.21.15.32:6824/1551241759 11 ==== osd_op_reply(97 100.00000000 [delete] v122'371 uv371 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f6534056870 con 0x5646a0f1ab70 2024-08-03T21:00:06.317 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.315+0000 7f654cb20f40 1 -- 172.21.15.32:0/3921196523 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:98 12.13 12:cd5a64a3:::100.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122) -- 0x5646a0f25d60 con 0x5646a0f1ab70 2024-08-03T21:00:06.318 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.317+0000 7f6544a00640 1 -- 172.21.15.32:0/3921196523 <== osd.1 v2:172.21.15.32:6824/1551241759 12 ==== osd_op_reply(98 100.00000000 [create] v122'372 uv372 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f6534056870 con 0x5646a0f1ab70 2024-08-03T21:00:06.318 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::100.00000000:head# 2024-08-03T21:00:06.318 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.317+0000 7f654cb20f40 1 -- 172.21.15.32:0/3921196523 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:99 12.13 12:cd5a64a3:::100.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122) -- 0x5646a0f26150 con 0x5646a0f1ab70 2024-08-03T21:00:06.319 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.318+0000 7f6544a00640 1 -- 172.21.15.32:0/3921196523 <== osd.1 v2:172.21.15.32:6824/1551241759 13 ==== osd_op_reply(99 100.00000000 [omap-set-header] v122'373 uv373 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f6534056870 con 0x5646a0f1ab70 2024-08-03T21:00:06.319 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.318+0000 7f654cb20f40 1 -- 172.21.15.32:0/3921196523 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:100 12.13 12:cd5a64a3:::100.00000000:head [omap-set-vals in=5194b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122) -- 0x5646a0f28b90 con 0x5646a0f1ab70 2024-08-03T21:00:06.321 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.319+0000 7f6544a00640 1 -- 172.21.15.32:0/3921196523 <== osd.1 v2:172.21.15.32:6824/1551241759 14 ==== osd_op_reply(100 100.00000000 [omap-set-vals] v122'374 uv374 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f6534056870 con 0x5646a0f1ab70 2024-08-03T21:00:06.321 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.319+0000 7f654cb20f40 1 --2- 172.21.15.32:0/3921196523 >> [v2:172.21.15.32:6800/9088303,v1:172.21.15.32:6801/9088303] conn(0x5646a0f26870 0x5646a0f28c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:00:06.321 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.319+0000 7f654cb20f40 1 -- 172.21.15.32:0/3921196523 --> [v2:172.21.15.32:6800/9088303,v1:172.21.15.32:6801/9088303] -- osd_op(unknown.0.0:101 12.b 12:d0630e4c:::mds0_sessionmap:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122) -- 0x5646a0f29250 con 0x5646a0f26870 2024-08-03T21:00:06.321 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.320+0000 7f6544a00640 1 --2- 172.21.15.32:0/3921196523 >> [v2:172.21.15.32:6800/9088303,v1:172.21.15.32:6801/9088303] conn(0x5646a0f26870 0x5646a0f28c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:00:06.321 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.320+0000 7f6544a00640 1 --2- 172.21.15.32:0/3921196523 >> [v2:172.21.15.32:6800/9088303,v1:172.21.15.32:6801/9088303] conn(0x5646a0f26870 0x5646a0f28c90 crc :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T21:00:06.323 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.321+0000 7f6544a00640 1 -- 172.21.15.32:0/3921196523 <== osd.3 v2:172.21.15.32:6800/9088303 1 ==== osd_op_reply(101 mds0_sessionmap [create] v122'162 uv160 ondisk = -17 ((17) File exists)) ==== 159+0+0 (crc 0 0 0) 0x7f6534056870 con 0x5646a0f26870 2024-08-03T21:00:06.323 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.322+0000 7f654cb20f40 1 -- 172.21.15.32:0/3921196523 --> [v2:172.21.15.32:6800/9088303,v1:172.21.15.32:6801/9088303] -- osd_op(unknown.0.0:102 12.b 12:d0630e4c:::mds0_sessionmap:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e122) -- 0x5646a0f29250 con 0x5646a0f26870 2024-08-03T21:00:06.324 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.323+0000 7f6544a00640 1 -- 172.21.15.32:0/3921196523 <== osd.3 v2:172.21.15.32:6800/9088303 2 ==== osd_op_reply(102 mds0_sessionmap [delete] v122'163 uv163 ondisk = 0) ==== 159+0+0 (crc 0 0 0) 0x7f6534056870 con 0x5646a0f26870 2024-08-03T21:00:06.325 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.323+0000 7f654cb20f40 1 -- 172.21.15.32:0/3921196523 --> [v2:172.21.15.32:6800/9088303,v1:172.21.15.32:6801/9088303] -- osd_op(unknown.0.0:103 12.b 12:d0630e4c:::mds0_sessionmap:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122) -- 0x5646a0f29250 con 0x5646a0f26870 2024-08-03T21:00:06.325 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.324+0000 7f6544a00640 1 -- 172.21.15.32:0/3921196523 <== osd.3 v2:172.21.15.32:6800/9088303 3 ==== osd_op_reply(103 mds0_sessionmap [create] v122'164 uv164 ondisk = 0) ==== 159+0+0 (crc 0 0 0) 0x7f6534056870 con 0x5646a0f26870 2024-08-03T21:00:06.326 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds0_sessionmap:head# 2024-08-03T21:00:06.326 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.324+0000 7f654cb20f40 1 -- 172.21.15.32:0/3921196523 --> [v2:172.21.15.32:6800/9088303,v1:172.21.15.32:6801/9088303] -- osd_op(unknown.0.0:104 12.b 12:d0630e4c:::mds0_sessionmap:head [omap-set-header in=14b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122) -- 0x5646a0f29640 con 0x5646a0f26870 2024-08-03T21:00:06.327 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.325+0000 7f6544a00640 1 -- 172.21.15.32:0/3921196523 <== osd.3 v2:172.21.15.32:6800/9088303 4 ==== osd_op_reply(104 mds0_sessionmap [omap-set-header] v122'165 uv165 ondisk = 0) ==== 159+0+0 (crc 0 0 0) 0x7f6534056870 con 0x5646a0f26870 2024-08-03T21:00:06.327 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.325+0000 7f654cb20f40 1 -- 172.21.15.32:0/3921196523 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:105 12.7 12:e0b41d06:::609.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122) -- 0x5646a0f29640 con 0x5646a0f1ab70 2024-08-03T21:00:06.328 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.327+0000 7f6544a00640 1 -- 172.21.15.32:0/3921196523 <== osd.1 v2:172.21.15.32:6824/1551241759 15 ==== osd_op_reply(105 609.00000000 [create] v122'219 uv218 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f6534056870 con 0x5646a0f1ab70 2024-08-03T21:00:06.329 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.327+0000 7f654cb20f40 1 -- 172.21.15.32:0/3921196523 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:106 12.7 12:e0b41d06:::609.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e122) -- 0x5646a0f29280 con 0x5646a0f1ab70 2024-08-03T21:00:06.330 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.329+0000 7f6544a00640 1 -- 172.21.15.32:0/3921196523 <== osd.1 v2:172.21.15.32:6824/1551241759 16 ==== osd_op_reply(106 609.00000000 [delete] v122'220 uv220 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f6534056870 con 0x5646a0f1ab70 2024-08-03T21:00:06.330 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.329+0000 7f654cb20f40 1 -- 172.21.15.32:0/3921196523 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:107 12.7 12:e0b41d06:::609.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122) -- 0x5646a0f29280 con 0x5646a0f1ab70 2024-08-03T21:00:06.331 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::609.00000000:head# 2024-08-03T21:00:06.332 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.330+0000 7f6544a00640 1 -- 172.21.15.32:0/3921196523 <== osd.1 v2:172.21.15.32:6824/1551241759 17 ==== osd_op_reply(107 609.00000000 [create] v122'221 uv221 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f6534056870 con 0x5646a0f1ab70 2024-08-03T21:00:06.332 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.330+0000 7f654cb20f40 1 -- 172.21.15.32:0/3921196523 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:108 12.7 12:e0b41d06:::609.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122) -- 0x5646a0f2a290 con 0x5646a0f1ab70 2024-08-03T21:00:06.333 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.332+0000 7f6544a00640 1 -- 172.21.15.32:0/3921196523 <== osd.1 v2:172.21.15.32:6824/1551241759 18 ==== osd_op_reply(108 609.00000000 [setxattr (30)] v122'222 uv222 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f6534056870 con 0x5646a0f1ab70 2024-08-03T21:00:06.333 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.332+0000 7f654cb20f40 1 -- 172.21.15.32:0/3921196523 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:109 12.7 12:e0b41d06:::609.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122) -- 0x5646a0f2a290 con 0x5646a0f1ab70 2024-08-03T21:00:06.334 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.333+0000 7f6544a00640 1 -- 172.21.15.32:0/3921196523 <== osd.1 v2:172.21.15.32:6824/1551241759 19 ==== osd_op_reply(109 609.00000000 [setxattr (62)] v122'223 uv223 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f6534056870 con 0x5646a0f1ab70 2024-08-03T21:00:06.334 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.333+0000 7f654cb20f40 1 -- 172.21.15.32:0/3921196523 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:110 12.7 12:e0b41d06:::609.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122) -- 0x5646a0f29280 con 0x5646a0f1ab70 2024-08-03T21:00:06.335 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.334+0000 7f6544a00640 1 -- 172.21.15.32:0/3921196523 <== osd.1 v2:172.21.15.32:6824/1551241759 20 ==== osd_op_reply(110 609.00000000 [omap-set-header] v122'224 uv224 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f6534056870 con 0x5646a0f1ab70 2024-08-03T21:00:06.335 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.334+0000 7f654cb20f40 1 -- 172.21.15.32:0/3921196523 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:111 12.f 12:f1f69726:::400.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122) -- 0x5646a0f29280 con 0x5646a0f1f0e0 2024-08-03T21:00:06.337 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.336+0000 7f6545e00640 1 -- 172.21.15.32:0/3921196523 <== osd.6 v2:172.21.15.118:6825/801947386 12 ==== osd_op_reply(111 400.00000000 [create] v122'186 uv185 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f6540019040 con 0x5646a0f1f0e0 2024-08-03T21:00:06.337 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.336+0000 7f654cb20f40 1 -- 172.21.15.32:0/3921196523 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:112 12.f 12:f1f69726:::400.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e122) -- 0x5646a0f29280 con 0x5646a0f1f0e0 2024-08-03T21:00:06.338 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.337+0000 7f6545e00640 1 -- 172.21.15.32:0/3921196523 <== osd.6 v2:172.21.15.118:6825/801947386 13 ==== osd_op_reply(112 400.00000000 [delete] v122'187 uv187 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f6540019040 con 0x5646a0f1f0e0 2024-08-03T21:00:06.338 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.337+0000 7f654cb20f40 1 -- 172.21.15.32:0/3921196523 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:113 12.f 12:f1f69726:::400.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122) -- 0x5646a0f29280 con 0x5646a0f1f0e0 2024-08-03T21:00:06.340 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::400.00000000:head# 2024-08-03T21:00:06.340 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.339+0000 7f6545e00640 1 -- 172.21.15.32:0/3921196523 <== osd.6 v2:172.21.15.118:6825/801947386 14 ==== osd_op_reply(113 400.00000000 [create] v122'188 uv188 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f6540019040 con 0x5646a0f1f0e0 2024-08-03T21:00:06.340 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.339+0000 7f654cb20f40 1 -- 172.21.15.32:0/3921196523 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:114 12.f 12:f1f69726:::400.00000000:head [write 0~22 in=22b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122) -- 0x5646a0f29280 con 0x5646a0f1f0e0 2024-08-03T21:00:06.342 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.340+0000 7f6545e00640 1 -- 172.21.15.32:0/3921196523 <== osd.6 v2:172.21.15.118:6825/801947386 15 ==== osd_op_reply(114 400.00000000 [write 0~22] v122'189 uv189 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f6540019040 con 0x5646a0f1f0e0 2024-08-03T21:00:06.342 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.341+0000 7f654cb20f40 1 -- 172.21.15.32:0/3921196523 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:115 12.f 12:f1f69726:::400.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122) -- 0x5646a0f29280 con 0x5646a0f1f0e0 2024-08-03T21:00:06.343 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.342+0000 7f6545e00640 1 -- 172.21.15.32:0/3921196523 <== osd.6 v2:172.21.15.118:6825/801947386 16 ==== osd_op_reply(115 400.00000000 [omap-set-header] v122'190 uv190 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f6540019040 con 0x5646a0f1f0e0 2024-08-03T21:00:06.344 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.342+0000 7f654cb20f40 1 -- 172.21.15.32:0/3921196523 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:116 12.1f 12:fe07bba1:::100.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122) -- 0x5646a0f29280 con 0x5646a0f1f0e0 2024-08-03T21:00:06.345 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.344+0000 7f6545e00640 1 -- 172.21.15.32:0/3921196523 <== osd.6 v2:172.21.15.118:6825/801947386 17 ==== osd_op_reply(116 100.00000000.inode [create] v122'455 uv446 ondisk = -17 ((17) File exists)) ==== 162+0+0 (crc 0 0 0) 0x7f6540019040 con 0x5646a0f1f0e0 2024-08-03T21:00:06.345 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.344+0000 7f654cb20f40 1 -- 172.21.15.32:0/3921196523 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:117 12.1f 12:fe07bba1:::100.00000000.inode:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e122) -- 0x5646a0f29280 con 0x5646a0f1f0e0 2024-08-03T21:00:06.347 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.346+0000 7f6545e00640 1 -- 172.21.15.32:0/3921196523 <== osd.6 v2:172.21.15.118:6825/801947386 18 ==== osd_op_reply(117 100.00000000.inode [delete] v122'456 uv456 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7f6540019040 con 0x5646a0f1f0e0 2024-08-03T21:00:06.347 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.346+0000 7f654cb20f40 1 -- 172.21.15.32:0/3921196523 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:118 12.1f 12:fe07bba1:::100.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122) -- 0x5646a0f29280 con 0x5646a0f1f0e0 2024-08-03T21:00:06.349 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::100.00000000.inode:head# 2024-08-03T21:00:06.349 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.347+0000 7f6545e00640 1 -- 172.21.15.32:0/3921196523 <== osd.6 v2:172.21.15.118:6825/801947386 19 ==== osd_op_reply(118 100.00000000.inode [create] v122'457 uv457 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7f6540019040 con 0x5646a0f1f0e0 2024-08-03T21:00:06.349 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.347+0000 7f654cb20f40 1 -- 172.21.15.32:0/3921196523 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:119 12.1f 12:fe07bba1:::100.00000000.inode:head [write 0~582 in=582b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122) -- 0x5646a0f29280 con 0x5646a0f1f0e0 2024-08-03T21:00:06.350 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.349+0000 7f6545e00640 1 -- 172.21.15.32:0/3921196523 <== osd.6 v2:172.21.15.118:6825/801947386 20 ==== osd_op_reply(119 100.00000000.inode [write 0~582] v122'458 uv458 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7f6540019040 con 0x5646a0f1f0e0 2024-08-03T21:00:06.350 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.349+0000 7f654cb20f40 1 -- 172.21.15.32:0/3921196523 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:120 12.1f 12:fe07bba1:::100.00000000.inode:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122) -- 0x5646a0f29280 con 0x5646a0f1f0e0 2024-08-03T21:00:06.352 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.351+0000 7f6545e00640 1 -- 172.21.15.32:0/3921196523 <== osd.6 v2:172.21.15.118:6825/801947386 21 ==== osd_op_reply(120 100.00000000.inode [omap-set-header] v122'459 uv459 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7f6540019040 con 0x5646a0f1f0e0 2024-08-03T21:00:06.352 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.351+0000 7f654cb20f40 1 -- 172.21.15.32:0/3921196523 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:121 12.1f 12:ff5b34d6:::1.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122) -- 0x5646a0f29280 con 0x5646a0f1f0e0 2024-08-03T21:00:06.353 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.352+0000 7f6545e00640 1 -- 172.21.15.32:0/3921196523 <== osd.6 v2:172.21.15.118:6825/801947386 22 ==== osd_op_reply(121 1.00000000 [create] v122'460 uv453 ondisk = -17 ((17) File exists)) ==== 154+0+0 (crc 0 0 0) 0x7f6540019040 con 0x5646a0f1f0e0 2024-08-03T21:00:06.354 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.352+0000 7f654cb20f40 1 -- 172.21.15.32:0/3921196523 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:122 12.1f 12:ff5b34d6:::1.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e122) -- 0x5646a0f29280 con 0x5646a0f1f0e0 2024-08-03T21:00:06.355 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.354+0000 7f6545e00640 1 -- 172.21.15.32:0/3921196523 <== osd.6 v2:172.21.15.118:6825/801947386 23 ==== osd_op_reply(122 1.00000000 [delete] v122'461 uv461 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7f6540019040 con 0x5646a0f1f0e0 2024-08-03T21:00:06.355 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.354+0000 7f654cb20f40 1 -- 172.21.15.32:0/3921196523 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:123 12.1f 12:ff5b34d6:::1.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122) -- 0x5646a0f29280 con 0x5646a0f1f0e0 2024-08-03T21:00:06.356 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.355+0000 7f6545e00640 1 -- 172.21.15.32:0/3921196523 <== osd.6 v2:172.21.15.118:6825/801947386 24 ==== osd_op_reply(123 1.00000000 [create] v122'462 uv462 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7f6540019040 con 0x5646a0f1f0e0 2024-08-03T21:00:06.357 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::1.00000000:head# 2024-08-03T21:00:06.357 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.355+0000 7f654cb20f40 1 -- 172.21.15.32:0/3921196523 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:124 12.1f 12:ff5b34d6:::1.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122) -- 0x5646a0f29280 con 0x5646a0f1f0e0 2024-08-03T21:00:06.358 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.357+0000 7f6545e00640 1 -- 172.21.15.32:0/3921196523 <== osd.6 v2:172.21.15.118:6825/801947386 25 ==== osd_op_reply(124 1.00000000 [setxattr (30)] v122'463 uv463 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7f6540019040 con 0x5646a0f1f0e0 2024-08-03T21:00:06.358 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.357+0000 7f654cb20f40 1 -- 172.21.15.32:0/3921196523 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:125 12.1f 12:ff5b34d6:::1.00000000:head [setxattr parent (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122) -- 0x5646a0f29280 con 0x5646a0f1f0e0 2024-08-03T21:00:06.359 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.358+0000 7f6545e00640 1 -- 172.21.15.32:0/3921196523 <== osd.6 v2:172.21.15.118:6825/801947386 26 ==== osd_op_reply(125 1.00000000 [setxattr (30)] v122'464 uv464 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7f6540019040 con 0x5646a0f1f0e0 2024-08-03T21:00:06.360 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.358+0000 7f654cb20f40 1 -- 172.21.15.32:0/3921196523 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:126 12.1f 12:ff5b34d6:::1.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122) -- 0x5646a0f29280 con 0x5646a0f1f0e0 2024-08-03T21:00:06.361 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.360+0000 7f6545e00640 1 -- 172.21.15.32:0/3921196523 <== osd.6 v2:172.21.15.118:6825/801947386 27 ==== osd_op_reply(126 1.00000000 [omap-set-header] v122'465 uv465 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7f6540019040 con 0x5646a0f1f0e0 2024-08-03T21:00:06.362 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.360+0000 7f654cb20f40 1 -- 172.21.15.32:0/3921196523 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:127 12.1f 12:ff5b34d6:::1.00000000:head [omap-set-vals in=584b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122) -- 0x5646a0f29b20 con 0x5646a0f1f0e0 2024-08-03T21:00:06.364 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.362+0000 7f6545e00640 1 -- 172.21.15.32:0/3921196523 <== osd.6 v2:172.21.15.118:6825/801947386 28 ==== osd_op_reply(127 1.00000000 [omap-set-vals] v122'466 uv466 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7f6540019040 con 0x5646a0f1f0e0 2024-08-03T21:00:06.364 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.362+0000 7f654cb20f40 1 -- 172.21.15.32:0/3921196523 >> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] conn(0x5646a0e69d20 msgr2=0x5646a0dc9fb0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:00:06.364 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.362+0000 7f654cb20f40 1 --2- 172.21.15.32:0/3921196523 >> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] conn(0x5646a0e69d20 0x5646a0dc9fb0 crc :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:06.364 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.362+0000 7f654cb20f40 1 -- 172.21.15.32:0/3921196523 >> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] conn(0x5646a0f1ab70 msgr2=0x5646a0f1cf90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:00:06.364 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.362+0000 7f654cb20f40 1 --2- 172.21.15.32:0/3921196523 >> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] conn(0x5646a0f1ab70 0x5646a0f1cf90 crc :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:06.365 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.362+0000 7f654cb20f40 1 -- 172.21.15.32:0/3921196523 >> [v2:172.21.15.32:6800/9088303,v1:172.21.15.32:6801/9088303] conn(0x5646a0f26870 msgr2=0x5646a0f28c90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:00:06.365 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.362+0000 7f654cb20f40 1 --2- 172.21.15.32:0/3921196523 >> [v2:172.21.15.32:6800/9088303,v1:172.21.15.32:6801/9088303] conn(0x5646a0f26870 0x5646a0f28c90 crc :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:06.365 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.362+0000 7f654cb20f40 1 -- 172.21.15.32:0/3921196523 >> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] conn(0x5646a0f158a0 msgr2=0x5646a0f189f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:00:06.365 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.362+0000 7f654cb20f40 1 --2- 172.21.15.32:0/3921196523 >> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] conn(0x5646a0f158a0 0x5646a0f189f0 crc :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:06.365 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.362+0000 7f654cb20f40 1 -- 172.21.15.32:0/3921196523 >> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] conn(0x5646a0f1f0e0 msgr2=0x5646a0f1f4f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:00:06.365 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.362+0000 7f654cb20f40 1 --2- 172.21.15.32:0/3921196523 >> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] conn(0x5646a0f1f0e0 0x5646a0f1f4f0 crc :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:06.365 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.362+0000 7f654cb20f40 1 -- 172.21.15.32:0/3921196523 >> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] conn(0x5646a0f22240 msgr2=0x5646a0f22650 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:00:06.365 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.362+0000 7f6545e00640 1 -- 172.21.15.32:0/3921196523 reap_dead start 2024-08-03T21:00:06.365 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.362+0000 7f654cb20f40 1 --2- 172.21.15.32:0/3921196523 >> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] conn(0x5646a0f22240 0x5646a0f22650 crc :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:06.365 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.362+0000 7f6545e00640 1 -- 172.21.15.32:0/3921196523 reap_dead start 2024-08-03T21:00:06.365 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.362+0000 7f654cb20f40 1 -- 172.21.15.32:0/3921196523 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f65280759f0 msgr2=0x7f6528077f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:00:06.365 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.362+0000 7f654cb20f40 1 --2- 172.21.15.32:0/3921196523 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f65280759f0 0x7f6528077f00 secure :-1 s=READY pgs=809 cs=0 l=1 rev1=1 crypto rx=0x7f6534055000 tx=0x7f6534003020 comp rx=0 tx=0).stop 2024-08-03T21:00:06.366 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.362+0000 7f654cb20f40 1 -- 172.21.15.32:0/3921196523 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5646a0de7010 msgr2=0x5646a0e65a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:00:06.366 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.362+0000 7f654cb20f40 1 --2- 172.21.15.32:0/3921196523 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5646a0de7010 0x5646a0e65a00 secure :-1 s=READY pgs=2025 cs=0 l=1 rev1=1 crypto rx=0x7f653800a6b0 tx=0x7f653800ab70 comp rx=0 tx=0).stop 2024-08-03T21:00:06.366 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.362+0000 7f654cb20f40 1 -- 172.21.15.32:0/3921196523 shutdown_connections 2024-08-03T21:00:06.366 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.362+0000 7f654cb20f40 1 --2- 172.21.15.32:0/3921196523 >> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] conn(0x5646a0f22240 0x5646a0f22650 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:06.366 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.362+0000 7f654cb20f40 1 --2- 172.21.15.32:0/3921196523 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f65280759f0 0x7f6528077f00 unknown :-1 s=CLOSED pgs=809 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:06.366 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.362+0000 7f654cb20f40 1 --2- 172.21.15.32:0/3921196523 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5646a0de7010 0x5646a0e65a00 unknown :-1 s=CLOSED pgs=2025 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:06.366 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.362+0000 7f654cb20f40 1 -- 172.21.15.32:0/3921196523 >> 172.21.15.32:0/3921196523 conn(0x5646a0cbfac0 msgr2=0x5646a0e6e220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T21:00:06.366 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.362+0000 7f654cb20f40 1 -- 172.21.15.32:0/3921196523 shutdown_connections 2024-08-03T21:00:06.366 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.362+0000 7f654cb20f40 1 -- 172.21.15.32:0/3921196523 wait complete. 2024-08-03T21:00:06.378 DEBUG:teuthology.orchestra.run.smithi032:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata setomapheader 100.00000000 deadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeef 2024-08-03T21:00:06.523 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.521+0000 7fbc3bbfef40 1 Processor -- start 2024-08-03T21:00:06.523 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.521+0000 7fbc3bbfef40 1 -- start start 2024-08-03T21:00:06.524 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.523+0000 7fbc3bbfef40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5642752b5870 0x56427529e3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:00:06.524 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.523+0000 7fbc3bbfef40 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5642752b7370 0x56427532b480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:00:06.524 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.523+0000 7fbc3bbfef40 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x56427532c270 0x56427533a0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:00:06.525 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.523+0000 7fbc3bbfef40 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x5642751bbe30 con 0x5642752b7370 2024-08-03T21:00:06.525 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.523+0000 7fbc3bbfef40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x564275219bf0 con 0x5642752b5870 2024-08-03T21:00:06.525 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.523+0000 7fbc3bbfef40 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x5642752b8da0 con 0x56427532c270 2024-08-03T21:00:06.525 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.523+0000 7fbc2fe00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5642752b7370 0x56427532b480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:00:06.525 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.523+0000 7fbc2fe00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5642752b7370 0x56427532b480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:39704/0 (socket says 172.21.15.32:39704) 2024-08-03T21:00:06.525 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.523+0000 7fbc2fe00640 1 -- 172.21.15.32:0/3700175169 learned_addr learned my addr 172.21.15.32:0/3700175169 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T21:00:06.526 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.523+0000 7fbc2fe00640 1 -- 172.21.15.32:0/3700175169 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x56427532c270 msgr2=0x56427533a0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T21:00:06.526 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.523+0000 7fbc2fe00640 1 --2- 172.21.15.32:0/3700175169 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x56427532c270 0x56427533a0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:06.527 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.523+0000 7fbc2fe00640 1 -- 172.21.15.32:0/3700175169 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5642752b5870 msgr2=0x56427529e3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T21:00:06.527 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.523+0000 7fbc2fe00640 1 --2- 172.21.15.32:0/3700175169 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5642752b5870 0x56427529e3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:06.527 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.523+0000 7fbc2fe00640 1 -- 172.21.15.32:0/3700175169 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5642753400d0 con 0x5642752b7370 2024-08-03T21:00:06.527 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.524+0000 7fbc2fe00640 1 --2- 172.21.15.32:0/3700175169 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5642752b7370 0x56427532b480 secure :-1 s=READY pgs=2027 cs=0 l=1 rev1=1 crypto rx=0x7fbc20001e30 tx=0x7fbc2000b730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f184fd155df6f297 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T21:00:06.527 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.524+0000 7fbc2ea00640 1 -- 172.21.15.32:0/3700175169 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc20012070 con 0x5642752b7370 2024-08-03T21:00:06.527 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.524+0000 7fbc2ea00640 1 -- 172.21.15.32:0/3700175169 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fbc20006050 con 0x5642752b7370 2024-08-03T21:00:06.527 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.524+0000 7fbc2ea00640 1 -- 172.21.15.32:0/3700175169 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc2000ebe0 con 0x5642752b7370 2024-08-03T21:00:06.527 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.524+0000 7fbc3bbfef40 1 -- 172.21.15.32:0/3700175169 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5642752b7370 msgr2=0x56427532b480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:00:06.527 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.524+0000 7fbc3bbfef40 1 --2- 172.21.15.32:0/3700175169 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5642752b7370 0x56427532b480 secure :-1 s=READY pgs=2027 cs=0 l=1 rev1=1 crypto rx=0x7fbc20001e30 tx=0x7fbc2000b730 comp rx=0 tx=0).stop 2024-08-03T21:00:06.527 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.524+0000 7fbc3bbfef40 1 -- 172.21.15.32:0/3700175169 shutdown_connections 2024-08-03T21:00:06.527 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.524+0000 7fbc3bbfef40 1 --2- 172.21.15.32:0/3700175169 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x56427532c270 0x56427533a0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:06.527 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.524+0000 7fbc3bbfef40 1 --2- 172.21.15.32:0/3700175169 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5642752b7370 0x56427532b480 unknown :-1 s=CLOSED pgs=2027 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:06.528 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.524+0000 7fbc3bbfef40 1 --2- 172.21.15.32:0/3700175169 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5642752b5870 0x56427529e3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:06.528 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.524+0000 7fbc3bbfef40 1 -- 172.21.15.32:0/3700175169 >> 172.21.15.32:0/3700175169 conn(0x56427518dac0 msgr2=0x56427526cf10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T21:00:06.528 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.525+0000 7fbc3bbfef40 1 -- 172.21.15.32:0/3700175169 shutdown_connections 2024-08-03T21:00:06.528 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.525+0000 7fbc3bbfef40 1 -- 172.21.15.32:0/3700175169 wait complete. 2024-08-03T21:00:06.528 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.526+0000 7fbc3bbfef40 1 Processor -- start 2024-08-03T21:00:06.528 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.526+0000 7fbc3bbfef40 1 -- start start 2024-08-03T21:00:06.530 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.527+0000 7fbc3bbfef40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5642752b5870 0x56427532a3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:00:06.530 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.527+0000 7fbc3bbfef40 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56427532c270 0x56427526e390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:00:06.530 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.527+0000 7fbc3bbfef40 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x56427532aa80 0x56427533c260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:00:06.530 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.527+0000 7fbc3bbfef40 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x56427529d930 con 0x56427532c270 2024-08-03T21:00:06.530 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.527+0000 7fbc3bbfef40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x564275340370 con 0x5642752b5870 2024-08-03T21:00:06.530 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.527+0000 7fbc3bbfef40 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x56427529dbf0 con 0x56427532aa80 2024-08-03T21:00:06.530 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.528+0000 7fbc2fe00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56427532c270 0x56427526e390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:00:06.530 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.528+0000 7fbc2fe00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56427532c270 0x56427526e390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:39718/0 (socket says 172.21.15.32:39718) 2024-08-03T21:00:06.531 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.528+0000 7fbc2fe00640 1 -- 172.21.15.32:0/3358230370 learned_addr learned my addr 172.21.15.32:0/3358230370 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T21:00:06.531 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.528+0000 7fbc35800640 1 --2- 172.21.15.32:0/3358230370 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x56427532aa80 0x56427533c260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:00:06.531 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.528+0000 7fbc34e00640 1 --2- 172.21.15.32:0/3358230370 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5642752b5870 0x56427532a3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:00:06.531 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.528+0000 7fbc2fe00640 1 -- 172.21.15.32:0/3358230370 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x56427532aa80 msgr2=0x56427533c260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:00:06.531 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.528+0000 7fbc2fe00640 1 --2- 172.21.15.32:0/3358230370 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x56427532aa80 0x56427533c260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:06.531 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.528+0000 7fbc2fe00640 1 -- 172.21.15.32:0/3358230370 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5642752b5870 msgr2=0x56427532a3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:00:06.531 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.528+0000 7fbc2fe00640 1 --2- 172.21.15.32:0/3358230370 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5642752b5870 0x56427532a3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:06.531 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.528+0000 7fbc2fe00640 1 -- 172.21.15.32:0/3358230370 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56427526c7e0 con 0x56427532c270 2024-08-03T21:00:06.531 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.528+0000 7fbc35800640 1 --2- 172.21.15.32:0/3358230370 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x56427532aa80 0x56427533c260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-03T21:00:06.531 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.528+0000 7fbc34e00640 1 --2- 172.21.15.32:0/3358230370 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5642752b5870 0x56427532a3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-03T21:00:06.531 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.528+0000 7fbc2fe00640 1 --2- 172.21.15.32:0/3358230370 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56427532c270 0x56427526e390 secure :-1 s=READY pgs=2028 cs=0 l=1 rev1=1 crypto rx=0x564275342870 tx=0x7fbc2000ebd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T21:00:06.532 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.529+0000 7fbc2d600640 1 -- 172.21.15.32:0/3358230370 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc20012070 con 0x56427532c270 2024-08-03T21:00:06.532 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.529+0000 7fbc2d600640 1 -- 172.21.15.32:0/3358230370 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fbc20006040 con 0x56427532c270 2024-08-03T21:00:06.532 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.529+0000 7fbc2d600640 1 -- 172.21.15.32:0/3358230370 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc20017910 con 0x56427532c270 2024-08-03T21:00:06.532 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.529+0000 7fbc3bbfef40 1 -- 172.21.15.32:0/3358230370 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x5642753d42f0 con 0x56427532c270 2024-08-03T21:00:06.532 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.529+0000 7fbc3bbfef40 1 -- 172.21.15.32:0/3358230370 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56427526ebc0 con 0x56427532c270 2024-08-03T21:00:06.533 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.531+0000 7fbc2d600640 1 -- 172.21.15.32:0/3358230370 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7fbc20015070 con 0x56427532c270 2024-08-03T21:00:06.533 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.532+0000 7fbc2d600640 1 --2- 172.21.15.32:0/3358230370 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fbc140759f0 0x7fbc14077f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:00:06.533 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.532+0000 7fbc2d600640 1 -- 172.21.15.32:0/3358230370 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(122..122 src has 1..122) ==== 9036+0+0 (secure 0 0 0) 0x7fbc20099300 con 0x56427532c270 2024-08-03T21:00:06.534 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.532+0000 7fbc3bbfef40 1 --2- 172.21.15.32:0/3358230370 >> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] conn(0x5642753e8c30 0x5642753eb050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:00:06.534 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.532+0000 7fbc3bbfef40 1 -- 172.21.15.32:0/3358230370 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:1 12.13 12:cd5a64a3:::100.00000000:head [omap-set-header in=80b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122) -- 0x5642753d16d0 con 0x5642753e8c30 2024-08-03T21:00:06.534 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.533+0000 7fbc34e00640 1 --2- 172.21.15.32:0/3358230370 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fbc140759f0 0x7fbc14077f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:00:06.534 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.533+0000 7fbc35800640 1 --2- 172.21.15.32:0/3358230370 >> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] conn(0x5642753e8c30 0x5642753eb050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:00:06.535 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.533+0000 7fbc35800640 1 --2- 172.21.15.32:0/3358230370 >> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] conn(0x5642753e8c30 0x5642753eb050 crc :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T21:00:06.535 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.533+0000 7fbc34e00640 1 --2- 172.21.15.32:0/3358230370 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fbc140759f0 0x7fbc14077f00 secure :-1 s=READY pgs=810 cs=0 l=1 rev1=1 crypto rx=0x7fbc28055000 tx=0x7fbc28058040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T21:00:06.537 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.535+0000 7fbc35800640 1 -- 172.21.15.32:0/3358230370 <== osd.1 v2:172.21.15.32:6824/1551241759 1 ==== osd_op_reply(1 100.00000000 [omap-set-header] v122'375 uv375 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fbc3000b040 con 0x5642753e8c30 2024-08-03T21:00:06.537 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.535+0000 7fbc3bbfef40 1 -- 172.21.15.32:0/3358230370 >> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] conn(0x5642753e8c30 msgr2=0x5642753eb050 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:00:06.537 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.535+0000 7fbc3bbfef40 1 --2- 172.21.15.32:0/3358230370 >> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] conn(0x5642753e8c30 0x5642753eb050 crc :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:06.537 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.535+0000 7fbc3bbfef40 1 -- 172.21.15.32:0/3358230370 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fbc140759f0 msgr2=0x7fbc14077f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:00:06.537 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.535+0000 7fbc3bbfef40 1 --2- 172.21.15.32:0/3358230370 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fbc140759f0 0x7fbc14077f00 secure :-1 s=READY pgs=810 cs=0 l=1 rev1=1 crypto rx=0x7fbc28055000 tx=0x7fbc28058040 comp rx=0 tx=0).stop 2024-08-03T21:00:06.537 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.535+0000 7fbc3bbfef40 1 -- 172.21.15.32:0/3358230370 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56427532c270 msgr2=0x56427526e390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:00:06.537 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.535+0000 7fbc3bbfef40 1 --2- 172.21.15.32:0/3358230370 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56427532c270 0x56427526e390 secure :-1 s=READY pgs=2028 cs=0 l=1 rev1=1 crypto rx=0x564275342870 tx=0x7fbc2000ebd0 comp rx=0 tx=0).stop 2024-08-03T21:00:06.537 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.535+0000 7fbc35800640 1 -- 172.21.15.32:0/3358230370 reap_dead start 2024-08-03T21:00:06.537 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.535+0000 7fbc3bbfef40 1 -- 172.21.15.32:0/3358230370 shutdown_connections 2024-08-03T21:00:06.538 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.535+0000 7fbc3bbfef40 1 -- 172.21.15.32:0/3358230370 >> 172.21.15.32:0/3358230370 conn(0x56427518dac0 msgr2=0x56427532b4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T21:00:06.538 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.535+0000 7fbc3bbfef40 1 -- 172.21.15.32:0/3358230370 shutdown_connections 2024-08-03T21:00:06.538 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.535+0000 7fbc3bbfef40 1 -- 172.21.15.32:0/3358230370 wait complete. 2024-08-03T21:00:06.547 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs joinable true 2024-08-03T21:00:06.780 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.778+0000 7fba1b200640 1 -- 172.21.15.32:0/2551849635 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fba1c160350 msgr2=0x7fba1c06e630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:00:06.780 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.778+0000 7fba1b200640 1 --2- 172.21.15.32:0/2551849635 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fba1c160350 0x7fba1c06e630 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fba14005fd0 tx=0x7fba1402a8a0 comp rx=0 tx=0).stop 2024-08-03T21:00:06.780 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.778+0000 7fba1b200640 1 -- 172.21.15.32:0/2551849635 shutdown_connections 2024-08-03T21:00:06.780 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.778+0000 7fba1b200640 1 --2- 172.21.15.32:0/2551849635 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fba1c160350 0x7fba1c06e630 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:06.780 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.778+0000 7fba1b200640 1 --2- 172.21.15.32:0/2551849635 >> v2:172.21.15.118:3300/0 conn(0x7fba1c159ff0 0x7fba1c15fe10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:06.780 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.778+0000 7fba1b200640 1 -- 172.21.15.32:0/2551849635 >> 172.21.15.32:0/2551849635 conn(0x7fba1c0cfd80 msgr2=0x7fba1c0d0190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T21:00:06.781 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.778+0000 7fba1b200640 1 -- 172.21.15.32:0/2551849635 shutdown_connections 2024-08-03T21:00:06.781 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.778+0000 7fba1b200640 1 -- 172.21.15.32:0/2551849635 wait complete. 2024-08-03T21:00:06.782 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.780+0000 7fba1b200640 1 Processor -- start 2024-08-03T21:00:06.782 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.780+0000 7fba1b200640 1 -- start start 2024-08-03T21:00:06.783 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.782+0000 7fba1b200640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fba1c159ff0 0x7fba1c1ef180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:00:06.784 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.782+0000 7fba1b200640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fba1c160350 0x7fba1c0c5540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:00:06.784 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.782+0000 7fba1b200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fba1c1609d0 0x7fba1c0c5a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:00:06.784 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.782+0000 7fba1b200640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fba1c06ef00 con 0x7fba1c1609d0 2024-08-03T21:00:06.784 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.782+0000 7fba1b200640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7fba1c06ed80 con 0x7fba1c160350 2024-08-03T21:00:06.784 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.782+0000 7fba1b200640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7fba1c06f080 con 0x7fba1c159ff0 2024-08-03T21:00:06.784 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.782+0000 7fba1a800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fba1c1609d0 0x7fba1c0c5a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:00:06.784 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.782+0000 7fba1a800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fba1c1609d0 0x7fba1c0c5a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:39724/0 (socket says 172.21.15.32:39724) 2024-08-03T21:00:06.784 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.782+0000 7fba1a800640 1 -- 172.21.15.32:0/3437759919 learned_addr learned my addr 172.21.15.32:0/3437759919 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T21:00:06.785 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.783+0000 7fba13e00640 1 --2- 172.21.15.32:0/3437759919 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fba1c160350 0x7fba1c0c5540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:00:06.785 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.783+0000 7fba19e00640 1 --2- 172.21.15.32:0/3437759919 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fba1c159ff0 0x7fba1c1ef180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:00:06.786 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.783+0000 7fba1a800640 1 -- 172.21.15.32:0/3437759919 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fba1c159ff0 msgr2=0x7fba1c1ef180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:00:06.786 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.783+0000 7fba1a800640 1 --2- 172.21.15.32:0/3437759919 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fba1c159ff0 0x7fba1c1ef180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:06.786 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.783+0000 7fba1a800640 1 -- 172.21.15.32:0/3437759919 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fba1c160350 msgr2=0x7fba1c0c5540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:00:06.786 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.783+0000 7fba1a800640 1 --2- 172.21.15.32:0/3437759919 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fba1c160350 0x7fba1c0c5540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:06.786 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.783+0000 7fba1a800640 1 -- 172.21.15.32:0/3437759919 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fba1c0b5200 con 0x7fba1c1609d0 2024-08-03T21:00:06.786 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.783+0000 7fba13e00640 1 --2- 172.21.15.32:0/3437759919 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fba1c160350 0x7fba1c0c5540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-03T21:00:06.786 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.783+0000 7fba1a800640 1 --2- 172.21.15.32:0/3437759919 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fba1c1609d0 0x7fba1c0c5a80 secure :-1 s=READY pgs=2030 cs=0 l=1 rev1=1 crypto rx=0x7fba14009cd0 tx=0x7fba14032db0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T21:00:06.786 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.783+0000 7fba12000640 1 -- 172.21.15.32:0/3437759919 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba14041070 con 0x7fba1c1609d0 2024-08-03T21:00:06.786 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.783+0000 7fba12000640 1 -- 172.21.15.32:0/3437759919 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fba14004450 con 0x7fba1c1609d0 2024-08-03T21:00:06.786 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.783+0000 7fba12000640 1 -- 172.21.15.32:0/3437759919 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba14031430 con 0x7fba1c1609d0 2024-08-03T21:00:06.786 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.783+0000 7fba1b200640 1 -- 172.21.15.32:0/3437759919 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fba1c0b54a0 con 0x7fba1c1609d0 2024-08-03T21:00:06.786 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.784+0000 7fba1b200640 1 -- 172.21.15.32:0/3437759919 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fba1c0cdcb0 con 0x7fba1c1609d0 2024-08-03T21:00:06.787 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.785+0000 7fba1b200640 1 -- 172.21.15.32:0/3437759919 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fba1c0b59c0 con 0x7fba1c1609d0 2024-08-03T21:00:06.787 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.786+0000 7fba12000640 1 -- 172.21.15.32:0/3437759919 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7fba14044070 con 0x7fba1c1609d0 2024-08-03T21:00:06.789 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.787+0000 7fba12000640 1 --2- 172.21.15.32:0/3437759919 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fb9f8077010 0x7fb9f8079520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:00:06.789 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.788+0000 7fba12000640 1 -- 172.21.15.32:0/3437759919 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(122..122 src has 1..122) ==== 9036+0+0 (secure 0 0 0) 0x7fba1402e020 con 0x7fba1c1609d0 2024-08-03T21:00:06.789 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.788+0000 7fba19e00640 1 --2- 172.21.15.32:0/3437759919 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fb9f8077010 0x7fb9f8079520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:00:06.790 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.788+0000 7fba19e00640 1 --2- 172.21.15.32:0/3437759919 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fb9f8077010 0x7fb9f8079520 secure :-1 s=READY pgs=811 cs=0 l=1 rev1=1 crypto rx=0x7fba0c0013f0 tx=0x7fba0c009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T21:00:06.793 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:06.792+0000 7fba12000640 1 -- 172.21.15.32:0/3437759919 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7fba1c0b59c0 con 0x7fba1c1609d0 2024-08-03T21:00:07.038 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:07.037+0000 7fba1b200640 1 -- 172.21.15.32:0/3437759919 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "joinable", "val": "true"} v 0) -- 0x7fba1c06f5a0 con 0x7fba1c1609d0 2024-08-03T21:00:07.978 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:07.976+0000 7fba12000640 1 -- 172.21.15.32:0/3437759919 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "joinable", "val": "true"}]=0 cephfs marked joinable; MDS may join as newly active. v311) ==== 162+0+0 (secure 0 0 0) 0x7fba1404b030 con 0x7fba1c1609d0 2024-08-03T21:00:07.978 INFO:teuthology.orchestra.run.smithi032.stderr:cephfs marked joinable; MDS may join as newly active. 2024-08-03T21:00:07.981 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:07.979+0000 7fba1b200640 1 -- 172.21.15.32:0/3437759919 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fb9f8077010 msgr2=0x7fb9f8079520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:00:07.981 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:07.979+0000 7fba1b200640 1 --2- 172.21.15.32:0/3437759919 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fb9f8077010 0x7fb9f8079520 secure :-1 s=READY pgs=811 cs=0 l=1 rev1=1 crypto rx=0x7fba0c0013f0 tx=0x7fba0c009040 comp rx=0 tx=0).stop 2024-08-03T21:00:07.982 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:07.979+0000 7fba1b200640 1 -- 172.21.15.32:0/3437759919 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fba1c1609d0 msgr2=0x7fba1c0c5a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:00:07.982 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:07.979+0000 7fba1b200640 1 --2- 172.21.15.32:0/3437759919 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fba1c1609d0 0x7fba1c0c5a80 secure :-1 s=READY pgs=2030 cs=0 l=1 rev1=1 crypto rx=0x7fba14009cd0 tx=0x7fba14032db0 comp rx=0 tx=0).stop 2024-08-03T21:00:07.982 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:07.979+0000 7fba1b200640 1 -- 172.21.15.32:0/3437759919 shutdown_connections 2024-08-03T21:00:07.982 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:07.979+0000 7fba1b200640 1 --2- 172.21.15.32:0/3437759919 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fb9f8077010 0x7fb9f8079520 unknown :-1 s=CLOSED pgs=811 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:07.982 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:07.979+0000 7fba1b200640 1 --2- 172.21.15.32:0/3437759919 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fba1c160350 0x7fba1c0c5540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:07.982 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:07.979+0000 7fba1b200640 1 --2- 172.21.15.32:0/3437759919 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fba1c1609d0 0x7fba1c0c5a80 unknown :-1 s=CLOSED pgs=2030 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:07.982 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:07.979+0000 7fba1b200640 1 --2- 172.21.15.32:0/3437759919 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fba1c159ff0 0x7fba1c1ef180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:07.982 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:07.979+0000 7fba1b200640 1 -- 172.21.15.32:0/3437759919 >> 172.21.15.32:0/3437759919 conn(0x7fba1c0cfd80 msgr2=0x7fba1c0cc800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T21:00:07.982 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:07.980+0000 7fba1b200640 1 -- 172.21.15.32:0/3437759919 shutdown_connections 2024-08-03T21:00:07.982 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:07.980+0000 7fba1b200640 1 -- 172.21.15.32:0/3437759919 wait complete. 2024-08-03T21:00:08.010 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-08-03T21:00:08.244 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:08.242+0000 7f6e13000640 1 -- 172.21.15.32:0/368522651 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6df4004000 msgr2=0x7f6df4004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:00:08.244 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:08.242+0000 7f6e13000640 1 --2- 172.21.15.32:0/368522651 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6df4004000 0x7f6df4004800 secure :-1 s=READY pgs=2032 cs=0 l=1 rev1=1 crypto rx=0x7f6df8009b50 tx=0x7f6df4005830 comp rx=0 tx=0).stop 2024-08-03T21:00:08.244 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:08.242+0000 7f6e12600640 1 -- 172.21.15.32:0/368522651 reap_dead start 2024-08-03T21:00:08.244 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:08.242+0000 7f6e13000640 1 -- 172.21.15.32:0/368522651 shutdown_connections 2024-08-03T21:00:08.244 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:08.242+0000 7f6e13000640 1 -- 172.21.15.32:0/368522651 >> 172.21.15.32:0/368522651 conn(0x7f6e0c08d7a0 msgr2=0x7f6e0c09d3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T21:00:08.244 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:08.242+0000 7f6e13000640 1 -- 172.21.15.32:0/368522651 shutdown_connections 2024-08-03T21:00:08.244 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:08.242+0000 7f6e13000640 1 -- 172.21.15.32:0/368522651 wait complete. 2024-08-03T21:00:08.246 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:08.244+0000 7f6e13000640 1 Processor -- start 2024-08-03T21:00:08.246 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:08.244+0000 7f6e13000640 1 -- start start 2024-08-03T21:00:08.249 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:08.246+0000 7f6e13000640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f6df4004000 0x7f6e0c155500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:00:08.249 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:08.246+0000 7f6e13000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6e0c0cdc30 0x7f6e0c0b1620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:00:08.249 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:08.246+0000 7f6e13000640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f6e0c15d8e0 0x7f6e0c0b1b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:00:08.249 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:08.246+0000 7f6e13000640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f6e0c0b4220 con 0x7f6e0c0cdc30 2024-08-03T21:00:08.249 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:08.246+0000 7f6e13000640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f6e0c0b40a0 con 0x7f6e0c15d8e0 2024-08-03T21:00:08.249 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:08.246+0000 7f6e13000640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7f6e0c0d3f80 con 0x7f6df4004000 2024-08-03T21:00:08.249 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:08.246+0000 7f6e11200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6e0c0cdc30 0x7f6e0c0b1620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:00:08.249 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:08.246+0000 7f6e11200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6e0c0cdc30 0x7f6e0c0b1620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:39750/0 (socket says 172.21.15.32:39750) 2024-08-03T21:00:08.249 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:08.246+0000 7f6e11200640 1 -- 172.21.15.32:0/1074763086 learned_addr learned my addr 172.21.15.32:0/1074763086 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T21:00:08.249 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:08.246+0000 7f6e11c00640 1 --2- 172.21.15.32:0/1074763086 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f6df4004000 0x7f6e0c155500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:00:08.249 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:08.246+0000 7f6e12600640 1 --2- 172.21.15.32:0/1074763086 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f6e0c15d8e0 0x7f6e0c0b1b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:00:08.249 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:08.247+0000 7f6e11200640 1 -- 172.21.15.32:0/1074763086 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f6df4004000 msgr2=0x7f6e0c155500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:00:08.250 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:08.247+0000 7f6e11200640 1 --2- 172.21.15.32:0/1074763086 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f6df4004000 0x7f6e0c155500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:08.250 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:08.247+0000 7f6e11200640 1 -- 172.21.15.32:0/1074763086 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f6e0c15d8e0 msgr2=0x7f6e0c0b1b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:00:08.250 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:08.247+0000 7f6e11200640 1 --2- 172.21.15.32:0/1074763086 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f6e0c15d8e0 0x7f6e0c0b1b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:08.250 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:08.247+0000 7f6e11200640 1 -- 172.21.15.32:0/1074763086 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6e0c1f8c20 con 0x7f6e0c0cdc30 2024-08-03T21:00:08.250 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:08.247+0000 7f6e11200640 1 --2- 172.21.15.32:0/1074763086 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6e0c0cdc30 0x7f6e0c0b1620 secure :-1 s=READY pgs=2033 cs=0 l=1 rev1=1 crypto rx=0x7f6df8036040 tx=0x7f6df8033670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T21:00:08.250 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:08.247+0000 7f6e09600640 1 -- 172.21.15.32:0/1074763086 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6df8038040 con 0x7f6e0c0cdc30 2024-08-03T21:00:08.250 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:08.247+0000 7f6e09600640 1 -- 172.21.15.32:0/1074763086 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f6df8041070 con 0x7f6e0c0cdc30 2024-08-03T21:00:08.250 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:08.247+0000 7f6e09600640 1 -- 172.21.15.32:0/1074763086 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6df8033d50 con 0x7f6e0c0cdc30 2024-08-03T21:00:08.250 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:08.247+0000 7f6e13000640 1 -- 172.21.15.32:0/1074763086 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6e0c1f8f00 con 0x7f6e0c0cdc30 2024-08-03T21:00:08.250 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:08.247+0000 7f6e13000640 1 -- 172.21.15.32:0/1074763086 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6e0c1f94c0 con 0x7f6e0c0cdc30 2024-08-03T21:00:08.251 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:08.248+0000 7f6e13000640 1 -- 172.21.15.32:0/1074763086 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6e0c0ce110 con 0x7f6e0c0cdc30 2024-08-03T21:00:08.251 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:08.250+0000 7f6e09600640 1 -- 172.21.15.32:0/1074763086 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f6df8046460 con 0x7f6e0c0cdc30 2024-08-03T21:00:08.252 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:08.251+0000 7f6e09600640 1 --2- 172.21.15.32:0/1074763086 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f6dec077040 0x7f6dec079550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:00:08.252 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:08.251+0000 7f6e09600640 1 -- 172.21.15.32:0/1074763086 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(122..122 src has 1..122) ==== 9036+0+0 (secure 0 0 0) 0x7f6df80ba0c0 con 0x7f6e0c0cdc30 2024-08-03T21:00:08.252 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:08.251+0000 7f6e11c00640 1 --2- 172.21.15.32:0/1074763086 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f6dec077040 0x7f6dec079550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:00:08.253 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:08.252+0000 7f6e11c00640 1 --2- 172.21.15.32:0/1074763086 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f6dec077040 0x7f6dec079550 secure :-1 s=READY pgs=812 cs=0 l=1 rev1=1 crypto rx=0x7f6e00006b50 tx=0x7f6e00006720 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T21:00:08.256 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:08.255+0000 7f6e09600640 1 -- 172.21.15.32:0/1074763086 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7f6df8081ec0 con 0x7f6e0c0cdc30 2024-08-03T21:00:08.505 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:08.504+0000 7f6e13000640 1 -- 172.21.15.32:0/1074763086 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f6e0c0ce410 con 0x7f6e0c0cdc30 2024-08-03T21:00:08.506 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:08.504+0000 7f6e09600640 1 -- 172.21.15.32:0/1074763086 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 312 v312) ==== 95+0+5220 (secure 0 0 0) 0x7f6e0c0ce410 con 0x7f6e0c0cdc30 2024-08-03T21:00:08.506 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-03T21:00:08.506 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":312,"btime":"2024-08-03T21:00:07:980896+0000","default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6439,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6835/531080160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6834","nonce":531080160},{"type":"v1","addr":"172.21.15.118:6835","nonce":531080160}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":84},{"gid":6454,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6837/2796882319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6836","nonce":2796882319},{"type":"v1","addr":"172.21.15.118:6837","nonce":2796882319}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":86},{"gid":6813,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6834/2686933890","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":2686933890},{"type":"v1","addr":"172.21.15.32:6834","nonce":2686933890}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":85}],"filesystems":[{"mdsmap":{"epoch":312,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-08-03T20:54:26.950518+0000","modified":"2024-08-03T21:00:07.980869+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":122,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":9283},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9283":{"gid":9283,"name":"a","rank":0,"incarnation":312,"state":"up:replay","state_seq":1,"addr":"172.21.15.32:6835/73253044","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6833","nonce":73253044},{"type":"v1","addr":"172.21.15.32:6835","nonce":73253044}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":6}]} 2024-08-03T21:00:08.506 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 312 2024-08-03T21:00:08.509 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:08.507+0000 7f6e13000640 1 -- 172.21.15.32:0/1074763086 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f6dec077040 msgr2=0x7f6dec079550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:00:08.509 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:08.507+0000 7f6e13000640 1 --2- 172.21.15.32:0/1074763086 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f6dec077040 0x7f6dec079550 secure :-1 s=READY pgs=812 cs=0 l=1 rev1=1 crypto rx=0x7f6e00006b50 tx=0x7f6e00006720 comp rx=0 tx=0).stop 2024-08-03T21:00:08.509 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:08.507+0000 7f6e13000640 1 -- 172.21.15.32:0/1074763086 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6e0c0cdc30 msgr2=0x7f6e0c0b1620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:00:08.509 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:08.507+0000 7f6e13000640 1 --2- 172.21.15.32:0/1074763086 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6e0c0cdc30 0x7f6e0c0b1620 secure :-1 s=READY pgs=2033 cs=0 l=1 rev1=1 crypto rx=0x7f6df8036040 tx=0x7f6df8033670 comp rx=0 tx=0).stop 2024-08-03T21:00:08.509 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:08.507+0000 7f6e13000640 1 -- 172.21.15.32:0/1074763086 shutdown_connections 2024-08-03T21:00:08.509 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:08.507+0000 7f6e13000640 1 --2- 172.21.15.32:0/1074763086 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f6dec077040 0x7f6dec079550 unknown :-1 s=CLOSED pgs=812 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:08.509 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:08.507+0000 7f6e13000640 1 --2- 172.21.15.32:0/1074763086 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f6e0c15d8e0 0x7f6e0c0b1b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:08.509 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:08.507+0000 7f6e13000640 1 --2- 172.21.15.32:0/1074763086 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6e0c0cdc30 0x7f6e0c0b1620 unknown :-1 s=CLOSED pgs=2033 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:08.509 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:08.507+0000 7f6e13000640 1 --2- 172.21.15.32:0/1074763086 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f6df4004000 0x7f6e0c155500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:08.509 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:08.507+0000 7f6e13000640 1 -- 172.21.15.32:0/1074763086 >> 172.21.15.32:0/1074763086 conn(0x7f6e0c08d7a0 msgr2=0x7f6e0c0b3030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T21:00:08.509 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:08.507+0000 7f6e13000640 1 -- 172.21.15.32:0/1074763086 shutdown_connections 2024-08-03T21:00:08.510 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:08.507+0000 7f6e13000640 1 -- 172.21.15.32:0/1074763086 wait complete. 2024-08-03T21:00:08.530 DEBUG:tasks.ceph_test_case:wait_until_true: waiting (timeout=60 retry_count=0)... 2024-08-03T21:00:09.039 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-08-03T21:00:09.043 DEBUG:teuthology.orchestra.run.smithi118:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-08-03T21:00:10.995 INFO:tasks.ceph.mds.a.smithi032.stderr:2024-08-03T21:00:10.993+0000 7f75d2400640 -1 mds.0.cache.dir(0x100) Corrupt fnode in dirfrag 0x100: Decoder at 'void fnode_t::decode(ceph::buffer::v15_2_0::list::const_iterator&)' v=3 cannot decode v=100 minimal_decoder=101: Malformed input [buffer:3] 2024-08-03T21:00:10.998 INFO:tasks.ceph.mon.a.smithi032.stderr:2024-08-03T21:00:10.996+0000 7f4e6d600640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-08-03T21:00:10.998 INFO:tasks.ceph.mon.a.smithi032.stderr:2024-08-03T21:00:10.996+0000 7f4e6d600640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 mds daemon damaged (MDS_DAMAGE) 2024-08-03T21:00:13.532 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-08-03T21:00:13.772 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:13.769+0000 7f19d3c00640 1 -- 172.21.15.32:0/1441562833 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f19b4004d40 msgr2=0x7f19b4005150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:00:13.772 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:13.769+0000 7f19d3c00640 1 --2- 172.21.15.32:0/1441562833 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f19b4004d40 0x7f19b4005150 secure :-1 s=READY pgs=2035 cs=0 l=1 rev1=1 crypto rx=0x7f19c4005fd0 tx=0x7f19b4005830 comp rx=0 tx=0).stop 2024-08-03T21:00:13.772 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:13.770+0000 7f19d1c00640 1 -- 172.21.15.32:0/1441562833 reap_dead start 2024-08-03T21:00:13.772 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:13.770+0000 7f19d3c00640 1 -- 172.21.15.32:0/1441562833 shutdown_connections 2024-08-03T21:00:13.772 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:13.770+0000 7f19d3c00640 1 -- 172.21.15.32:0/1441562833 >> 172.21.15.32:0/1441562833 conn(0x7f19cc0b1bd0 msgr2=0x7f19cc0b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T21:00:13.772 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:13.770+0000 7f19d3c00640 1 -- 172.21.15.32:0/1441562833 shutdown_connections 2024-08-03T21:00:13.772 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:13.770+0000 7f19d3c00640 1 -- 172.21.15.32:0/1441562833 wait complete. 2024-08-03T21:00:13.773 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:13.772+0000 7f19d3c00640 1 Processor -- start 2024-08-03T21:00:13.774 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:13.772+0000 7f19d3c00640 1 -- start start 2024-08-03T21:00:13.777 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:13.774+0000 7f19d3c00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f19b4004d40 0x7f19cc15f710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:00:13.777 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:13.774+0000 7f19d3c00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f19cc15b940 0x7f19cc13ea70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:00:13.777 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:13.774+0000 7f19d3c00640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f19cc1f34e0 0x7f19cc1f38f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:00:13.777 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:13.774+0000 7f19d3c00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f19cc0cf840 con 0x7f19b4004d40 2024-08-03T21:00:13.777 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:13.774+0000 7f19d3c00640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f19cc145f30 con 0x7f19cc15b940 2024-08-03T21:00:13.777 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:13.774+0000 7f19d3c00640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7f19cc0cf9c0 con 0x7f19cc1f34e0 2024-08-03T21:00:13.778 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:13.774+0000 7f19d1200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f19b4004d40 0x7f19cc15f710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:00:13.778 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:13.774+0000 7f19d1200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f19b4004d40 0x7f19cc15f710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:43930/0 (socket says 172.21.15.32:43930) 2024-08-03T21:00:13.778 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:13.774+0000 7f19d1200640 1 -- 172.21.15.32:0/2968121407 learned_addr learned my addr 172.21.15.32:0/2968121407 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T21:00:13.778 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:13.774+0000 7f19cbe00640 1 --2- 172.21.15.32:0/2968121407 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f19cc15b940 0x7f19cc13ea70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:00:13.778 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:13.774+0000 7f19d1c00640 1 --2- 172.21.15.32:0/2968121407 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f19cc1f34e0 0x7f19cc1f38f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:00:13.778 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:13.775+0000 7f19d1200640 1 -- 172.21.15.32:0/2968121407 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f19cc1f34e0 msgr2=0x7f19cc1f38f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:00:13.778 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:13.775+0000 7f19d1200640 1 --2- 172.21.15.32:0/2968121407 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f19cc1f34e0 0x7f19cc1f38f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:13.778 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:13.775+0000 7f19d1200640 1 -- 172.21.15.32:0/2968121407 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f19cc15b940 msgr2=0x7f19cc13ea70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:00:13.778 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:13.775+0000 7f19d1200640 1 --2- 172.21.15.32:0/2968121407 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f19cc15b940 0x7f19cc13ea70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:13.778 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:13.775+0000 7f19d1200640 1 -- 172.21.15.32:0/2968121407 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f19cc15e670 con 0x7f19b4004d40 2024-08-03T21:00:13.778 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:13.775+0000 7f19cbe00640 1 --2- 172.21.15.32:0/2968121407 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f19cc15b940 0x7f19cc13ea70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-03T21:00:13.778 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:13.775+0000 7f19d1c00640 1 --2- 172.21.15.32:0/2968121407 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f19cc1f34e0 0x7f19cc1f38f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-03T21:00:13.779 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:13.775+0000 7f19d1200640 1 --2- 172.21.15.32:0/2968121407 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f19b4004d40 0x7f19cc15f710 secure :-1 s=READY pgs=2036 cs=0 l=1 rev1=1 crypto rx=0x7f19c0006610 tx=0x7f19c00039c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T21:00:13.779 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:13.775+0000 7f19c8c00640 1 -- 172.21.15.32:0/2968121407 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19c0015070 con 0x7f19b4004d40 2024-08-03T21:00:13.783 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:13.775+0000 7f19c8c00640 1 -- 172.21.15.32:0/2968121407 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f19c000f960 con 0x7f19b4004d40 2024-08-03T21:00:13.783 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:13.775+0000 7f19d3c00640 1 -- 172.21.15.32:0/2968121407 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f19cc15e950 con 0x7f19b4004d40 2024-08-03T21:00:13.783 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:13.775+0000 7f19c8c00640 1 -- 172.21.15.32:0/2968121407 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19c000fd90 con 0x7f19b4004d40 2024-08-03T21:00:13.783 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:13.775+0000 7f19d3c00640 1 -- 172.21.15.32:0/2968121407 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f19cc15ef10 con 0x7f19b4004d40 2024-08-03T21:00:13.785 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:13.776+0000 7f19d3c00640 1 -- 172.21.15.32:0/2968121407 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f19cc074780 con 0x7f19b4004d40 2024-08-03T21:00:13.785 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:13.778+0000 7f19c8c00640 1 -- 172.21.15.32:0/2968121407 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f19c0011430 con 0x7f19b4004d40 2024-08-03T21:00:13.785 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:13.778+0000 7f19c8c00640 1 --2- 172.21.15.32:0/2968121407 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f19ac077010 0x7f19ac079520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:00:13.785 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:13.779+0000 7f19c8c00640 1 -- 172.21.15.32:0/2968121407 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(123..123 src has 1..123) ==== 9122+0+0 (secure 0 0 0) 0x7f19c009a3e0 con 0x7f19b4004d40 2024-08-03T21:00:13.785 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:13.779+0000 7f19cbe00640 1 --2- 172.21.15.32:0/2968121407 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f19ac077010 0x7f19ac079520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:00:13.785 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:13.780+0000 7f19cbe00640 1 --2- 172.21.15.32:0/2968121407 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f19ac077010 0x7f19ac079520 secure :-1 s=READY pgs=813 cs=0 l=1 rev1=1 crypto rx=0x7f19cc15b8f0 tx=0x7f19b8005f90 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T21:00:13.785 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:13.783+0000 7f19c8c00640 1 -- 172.21.15.32:0/2968121407 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7f19c00628d0 con 0x7f19b4004d40 2024-08-03T21:00:14.029 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:14.027+0000 7f19d3c00640 1 -- 172.21.15.32:0/2968121407 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f19cc145740 con 0x7f19b4004d40 2024-08-03T21:00:14.029 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:14.028+0000 7f19c8c00640 1 -- 172.21.15.32:0/2968121407 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 316 v316) ==== 95+0+4391 (secure 0 0 0) 0x7f19c00621d0 con 0x7f19b4004d40 2024-08-03T21:00:14.030 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-03T21:00:14.030 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":316,"btime":"2024-08-03T21:00:10:997611+0000","default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6439,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6835/531080160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6834","nonce":531080160},{"type":"v1","addr":"172.21.15.118:6835","nonce":531080160}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":84},{"gid":6454,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6837/2796882319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6836","nonce":2796882319},{"type":"v1","addr":"172.21.15.118:6837","nonce":2796882319}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":86},{"gid":6813,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6834/2686933890","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":2686933890},{"type":"v1","addr":"172.21.15.32:6834","nonce":2686933890}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":85}],"filesystems":[{"mdsmap":{"epoch":316,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-08-03T20:54:26.950518+0000","modified":"2024-08-03T21:00:10.997604+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":123,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":6}]} 2024-08-03T21:00:14.030 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 316 2024-08-03T21:00:14.032 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:14.030+0000 7f19d3c00640 1 -- 172.21.15.32:0/2968121407 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f19ac077010 msgr2=0x7f19ac079520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:00:14.032 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:14.030+0000 7f19d3c00640 1 --2- 172.21.15.32:0/2968121407 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f19ac077010 0x7f19ac079520 secure :-1 s=READY pgs=813 cs=0 l=1 rev1=1 crypto rx=0x7f19cc15b8f0 tx=0x7f19b8005f90 comp rx=0 tx=0).stop 2024-08-03T21:00:14.032 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:14.030+0000 7f19d3c00640 1 -- 172.21.15.32:0/2968121407 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f19b4004d40 msgr2=0x7f19cc15f710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:00:14.032 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:14.030+0000 7f19d3c00640 1 --2- 172.21.15.32:0/2968121407 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f19b4004d40 0x7f19cc15f710 secure :-1 s=READY pgs=2036 cs=0 l=1 rev1=1 crypto rx=0x7f19c0006610 tx=0x7f19c00039c0 comp rx=0 tx=0).stop 2024-08-03T21:00:14.032 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:14.030+0000 7f19d3c00640 1 -- 172.21.15.32:0/2968121407 shutdown_connections 2024-08-03T21:00:14.033 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:14.030+0000 7f19d3c00640 1 --2- 172.21.15.32:0/2968121407 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f19ac077010 0x7f19ac079520 unknown :-1 s=CLOSED pgs=813 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:14.033 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:14.030+0000 7f19d3c00640 1 --2- 172.21.15.32:0/2968121407 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f19cc15b940 0x7f19cc13ea70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:14.033 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:14.030+0000 7f19d3c00640 1 --2- 172.21.15.32:0/2968121407 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f19cc1f34e0 0x7f19cc1f38f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:14.033 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:14.030+0000 7f19d3c00640 1 --2- 172.21.15.32:0/2968121407 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f19b4004d40 0x7f19cc15f710 unknown :-1 s=CLOSED pgs=2036 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:14.033 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:14.030+0000 7f19d3c00640 1 -- 172.21.15.32:0/2968121407 >> 172.21.15.32:0/2968121407 conn(0x7f19cc0b1bd0 msgr2=0x7f19cc143220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T21:00:14.033 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:14.031+0000 7f19d3c00640 1 -- 172.21.15.32:0/2968121407 shutdown_connections 2024-08-03T21:00:14.033 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:14.031+0000 7f19d3c00640 1 -- 172.21.15.32:0/2968121407 wait complete. 2024-08-03T21:00:14.058 DEBUG:tasks.ceph_test_case:wait_until_true: success in 5s and 0 retries 2024-08-03T21:00:14.058 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-08-03T21:00:14.287 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:14.285+0000 7f5413800640 1 -- 172.21.15.32:0/968327117 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f540c0747b0 msgr2=0x7f540c0cc8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:00:14.288 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:14.285+0000 7f5413800640 1 --2- 172.21.15.32:0/968327117 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f540c0747b0 0x7f540c0cc8e0 secure :-1 s=READY pgs=2037 cs=0 l=1 rev1=1 crypto rx=0x7f540c071700 tx=0x7f540402aa30 comp rx=0 tx=0).stop 2024-08-03T21:00:14.288 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:14.285+0000 7f5413800640 1 -- 172.21.15.32:0/968327117 shutdown_connections 2024-08-03T21:00:14.288 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:14.285+0000 7f5413800640 1 --2- 172.21.15.32:0/968327117 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f540c0747b0 0x7f540c0cc8e0 unknown :-1 s=CLOSED pgs=2037 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:14.288 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:14.285+0000 7f5413800640 1 --2- 172.21.15.32:0/968327117 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f540c073e60 0x7f540c074270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:14.288 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:14.285+0000 7f5413800640 1 --2- 172.21.15.32:0/968327117 >> v2:172.21.15.118:3300/0 conn(0x7f540c06e930 0x7f540c144f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:14.288 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:14.285+0000 7f5413800640 1 -- 172.21.15.32:0/968327117 >> 172.21.15.32:0/968327117 conn(0x7f540c0b1bd0 msgr2=0x7f540c0b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T21:00:14.288 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:14.286+0000 7f5413800640 1 -- 172.21.15.32:0/968327117 shutdown_connections 2024-08-03T21:00:14.288 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:14.286+0000 7f5413800640 1 -- 172.21.15.32:0/968327117 wait complete. 2024-08-03T21:00:14.289 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:14.288+0000 7f5413800640 1 Processor -- start 2024-08-03T21:00:14.289 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:14.288+0000 7f5413800640 1 -- start start 2024-08-03T21:00:14.292 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:14.289+0000 7f5413800640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f540c06e930 0x7f540c1e9a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:00:14.292 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:14.290+0000 7f5413800640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f540c073e60 0x7f540c1e8620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:00:14.292 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:14.290+0000 7f5413800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f540c0747b0 0x7f540c1e8b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:00:14.292 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:14.290+0000 7f5413800640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f540c0cf9c0 con 0x7f540c0747b0 2024-08-03T21:00:14.292 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:14.290+0000 7f5413800640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f540c0cf840 con 0x7f540c073e60 2024-08-03T21:00:14.292 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:14.290+0000 7f5413800640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7f540c0cce20 con 0x7f540c06e930 2024-08-03T21:00:14.292 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:14.290+0000 7f5411800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f540c0747b0 0x7f540c1e8b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:00:14.292 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:14.290+0000 7f5411800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f540c0747b0 0x7f540c1e8b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:43960/0 (socket says 172.21.15.32:43960) 2024-08-03T21:00:14.292 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:14.290+0000 7f5411800640 1 -- 172.21.15.32:0/1079353476 learned_addr learned my addr 172.21.15.32:0/1079353476 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T21:00:14.292 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:14.290+0000 7f5410e00640 1 --2- 172.21.15.32:0/1079353476 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f540c06e930 0x7f540c1e9a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:00:14.292 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:14.290+0000 7f540be00640 1 --2- 172.21.15.32:0/1079353476 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f540c073e60 0x7f540c1e8620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:00:14.292 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:14.290+0000 7f5411800640 1 -- 172.21.15.32:0/1079353476 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f540c06e930 msgr2=0x7f540c1e9a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:00:14.293 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:14.290+0000 7f5411800640 1 --2- 172.21.15.32:0/1079353476 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f540c06e930 0x7f540c1e9a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:14.293 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:14.290+0000 7f5411800640 1 -- 172.21.15.32:0/1079353476 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f540c073e60 msgr2=0x7f540c1e8620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:00:14.293 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:14.290+0000 7f5411800640 1 --2- 172.21.15.32:0/1079353476 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f540c073e60 0x7f540c1e8620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:14.293 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:14.290+0000 7f5411800640 1 -- 172.21.15.32:0/1079353476 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f540c15f530 con 0x7f540c0747b0 2024-08-03T21:00:14.294 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:14.291+0000 7f5411800640 1 --2- 172.21.15.32:0/1079353476 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f540c0747b0 0x7f540c1e8b60 secure :-1 s=READY pgs=2038 cs=0 l=1 rev1=1 crypto rx=0x7f540402b020 tx=0x7f5404003f10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T21:00:14.294 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:14.291+0000 7f5408c00640 1 -- 172.21.15.32:0/1079353476 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5404033050 con 0x7f540c0747b0 2024-08-03T21:00:14.294 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:14.291+0000 7f5408c00640 1 -- 172.21.15.32:0/1079353476 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f5404004450 con 0x7f540c0747b0 2024-08-03T21:00:14.294 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:14.291+0000 7f5408c00640 1 -- 172.21.15.32:0/1079353476 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5404038450 con 0x7f540c0747b0 2024-08-03T21:00:14.294 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:14.291+0000 7f5413800640 1 -- 172.21.15.32:0/1079353476 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f540c141680 con 0x7f540c0747b0 2024-08-03T21:00:14.294 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:14.291+0000 7f5413800640 1 -- 172.21.15.32:0/1079353476 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f540c143fe0 con 0x7f540c0747b0 2024-08-03T21:00:14.294 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:14.292+0000 7f5413800640 1 -- 172.21.15.32:0/1079353476 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f540c074270 con 0x7f540c0747b0 2024-08-03T21:00:14.295 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:14.293+0000 7f5408c00640 1 -- 172.21.15.32:0/1079353476 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f5404036070 con 0x7f540c0747b0 2024-08-03T21:00:14.296 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:14.294+0000 7f5408c00640 1 --2- 172.21.15.32:0/1079353476 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f53ec077010 0x7f53ec079520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:00:14.296 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:14.294+0000 7f5408c00640 1 -- 172.21.15.32:0/1079353476 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(123..123 src has 1..123) ==== 9122+0+0 (secure 0 0 0) 0x7f54040ba8c0 con 0x7f540c0747b0 2024-08-03T21:00:14.296 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:14.295+0000 7f5410e00640 1 --2- 172.21.15.32:0/1079353476 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f53ec077010 0x7f53ec079520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:00:14.297 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:14.295+0000 7f5410e00640 1 --2- 172.21.15.32:0/1079353476 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f53ec077010 0x7f53ec079520 secure :-1 s=READY pgs=814 cs=0 l=1 rev1=1 crypto rx=0x7f5400009b30 tx=0x7f5400005f90 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T21:00:14.300 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:14.299+0000 7f5408c00640 1 -- 172.21.15.32:0/1079353476 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7f5404082700 con 0x7f540c0747b0 2024-08-03T21:00:14.547 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:14.546+0000 7f5413800640 1 -- 172.21.15.32:0/1079353476 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f540c15f700 con 0x7f540c0747b0 2024-08-03T21:00:14.548 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:14.546+0000 7f5408c00640 1 -- 172.21.15.32:0/1079353476 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 316 v316) ==== 95+0+4391 (secure 0 0 0) 0x7f5404086c50 con 0x7f540c0747b0 2024-08-03T21:00:14.548 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-03T21:00:14.548 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":316,"btime":"2024-08-03T21:00:10:997611+0000","default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6439,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6835/531080160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6834","nonce":531080160},{"type":"v1","addr":"172.21.15.118:6835","nonce":531080160}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":84},{"gid":6454,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6837/2796882319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6836","nonce":2796882319},{"type":"v1","addr":"172.21.15.118:6837","nonce":2796882319}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":86},{"gid":6813,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6834/2686933890","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":2686933890},{"type":"v1","addr":"172.21.15.32:6834","nonce":2686933890}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":85}],"filesystems":[{"mdsmap":{"epoch":316,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-08-03T20:54:26.950518+0000","modified":"2024-08-03T21:00:10.997604+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":123,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":6}]} 2024-08-03T21:00:14.548 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 316 2024-08-03T21:00:14.550 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:14.549+0000 7f5413800640 1 -- 172.21.15.32:0/1079353476 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f53ec077010 msgr2=0x7f53ec079520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:00:14.551 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:14.549+0000 7f5413800640 1 --2- 172.21.15.32:0/1079353476 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f53ec077010 0x7f53ec079520 secure :-1 s=READY pgs=814 cs=0 l=1 rev1=1 crypto rx=0x7f5400009b30 tx=0x7f5400005f90 comp rx=0 tx=0).stop 2024-08-03T21:00:14.551 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:14.549+0000 7f5413800640 1 -- 172.21.15.32:0/1079353476 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f540c0747b0 msgr2=0x7f540c1e8b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:00:14.551 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:14.549+0000 7f5413800640 1 --2- 172.21.15.32:0/1079353476 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f540c0747b0 0x7f540c1e8b60 secure :-1 s=READY pgs=2038 cs=0 l=1 rev1=1 crypto rx=0x7f540402b020 tx=0x7f5404003f10 comp rx=0 tx=0).stop 2024-08-03T21:00:14.551 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:14.549+0000 7f5413800640 1 -- 172.21.15.32:0/1079353476 shutdown_connections 2024-08-03T21:00:14.551 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:14.549+0000 7f5413800640 1 --2- 172.21.15.32:0/1079353476 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f53ec077010 0x7f53ec079520 unknown :-1 s=CLOSED pgs=814 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:14.551 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:14.549+0000 7f5413800640 1 --2- 172.21.15.32:0/1079353476 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f540c073e60 0x7f540c1e8620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:14.551 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:14.549+0000 7f5413800640 1 --2- 172.21.15.32:0/1079353476 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f540c0747b0 0x7f540c1e8b60 unknown :-1 s=CLOSED pgs=2038 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:14.551 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:14.549+0000 7f5413800640 1 --2- 172.21.15.32:0/1079353476 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f540c06e930 0x7f540c1e9a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:14.551 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:14.549+0000 7f5413800640 1 -- 172.21.15.32:0/1079353476 >> 172.21.15.32:0/1079353476 conn(0x7f540c0b1bd0 msgr2=0x7f540c1e63f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T21:00:14.551 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:14.549+0000 7f5413800640 1 -- 172.21.15.32:0/1079353476 shutdown_connections 2024-08-03T21:00:14.551 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:14.549+0000 7f5413800640 1 -- 172.21.15.32:0/1079353476 wait complete. 2024-08-03T21:00:14.577 DEBUG:tasks.ceph_test_case:wait_until_true: success in 0s and 0 retries 2024-08-03T21:00:14.577 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-08-03T21:00:14.809 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:14.807+0000 7f7fb1000640 1 -- 172.21.15.32:0/2558593344 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7f90003a00 msgr2=0x7f7f90003e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:00:14.809 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:14.807+0000 7f7fb1000640 1 --2- 172.21.15.32:0/2558593344 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7f90003a00 0x7f7f90003e70 secure :-1 s=READY pgs=2040 cs=0 l=1 rev1=1 crypto rx=0x7f7f9c009b50 tx=0x7f7f90005830 comp rx=0 tx=0).stop 2024-08-03T21:00:14.809 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:14.807+0000 7f7fabe00640 1 -- 172.21.15.32:0/2558593344 reap_dead start 2024-08-03T21:00:14.809 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:14.807+0000 7f7fb1000640 1 -- 172.21.15.32:0/2558593344 shutdown_connections 2024-08-03T21:00:14.809 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:14.807+0000 7f7fb1000640 1 -- 172.21.15.32:0/2558593344 >> 172.21.15.32:0/2558593344 conn(0x7f7fac0b2120 msgr2=0x7f7fac0cfd60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T21:00:14.810 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:14.807+0000 7f7fb1000640 1 -- 172.21.15.32:0/2558593344 shutdown_connections 2024-08-03T21:00:14.810 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:14.807+0000 7f7fb1000640 1 -- 172.21.15.32:0/2558593344 wait complete. 2024-08-03T21:00:14.811 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:14.809+0000 7f7fb1000640 1 Processor -- start 2024-08-03T21:00:14.811 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:14.810+0000 7f7fb1000640 1 -- start start 2024-08-03T21:00:14.813 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:14.811+0000 7f7fb1000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7f90003a00 0x7f7fac0b5640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:00:14.813 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:14.811+0000 7f7fb1000640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f7fac155f80 0x7f7fac15a8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:00:14.813 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:14.811+0000 7f7fb1000640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f7fac15adf0 0x7f7fac1f4ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:00:14.813 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:14.811+0000 7f7fb1000640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f7fac15c440 con 0x7f7f90003a00 2024-08-03T21:00:14.813 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:14.811+0000 7f7fb1000640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f7fac15c2c0 con 0x7f7fac155f80 2024-08-03T21:00:14.813 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:14.811+0000 7f7fb1000640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7f7fac06f9c0 con 0x7f7fac15adf0 2024-08-03T21:00:14.814 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:14.811+0000 7f7fab400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7f90003a00 0x7f7fac0b5640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:00:14.814 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:14.811+0000 7f7fab400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7f90003a00 0x7f7fac0b5640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:43982/0 (socket says 172.21.15.32:43982) 2024-08-03T21:00:14.814 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:14.811+0000 7f7fab400640 1 -- 172.21.15.32:0/1605306187 learned_addr learned my addr 172.21.15.32:0/1605306187 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T21:00:14.814 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:14.811+0000 7f7fabe00640 1 --2- 172.21.15.32:0/1605306187 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f7fac15adf0 0x7f7fac1f4ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:00:14.814 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:14.811+0000 7f7fa2a00640 1 --2- 172.21.15.32:0/1605306187 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f7fac155f80 0x7f7fac15a8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:00:14.814 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:14.812+0000 7f7fab400640 1 -- 172.21.15.32:0/1605306187 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f7fac15adf0 msgr2=0x7f7fac1f4ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:00:14.814 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:14.812+0000 7f7fab400640 1 --2- 172.21.15.32:0/1605306187 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f7fac15adf0 0x7f7fac1f4ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:14.814 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:14.812+0000 7f7fab400640 1 -- 172.21.15.32:0/1605306187 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f7fac155f80 msgr2=0x7f7fac15a8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:00:14.814 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:14.812+0000 7f7fab400640 1 --2- 172.21.15.32:0/1605306187 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f7fac155f80 0x7f7fac15a8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:14.814 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:14.812+0000 7f7fab400640 1 -- 172.21.15.32:0/1605306187 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7fac15b200 con 0x7f7f90003a00 2024-08-03T21:00:14.815 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:14.812+0000 7f7fab400640 1 --2- 172.21.15.32:0/1605306187 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7f90003a00 0x7f7fac0b5640 secure :-1 s=READY pgs=2041 cs=0 l=1 rev1=1 crypto rx=0x7f7f9c009c70 tx=0x7f7f9c033670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T21:00:14.815 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:14.812+0000 7f7fa3e00640 1 -- 172.21.15.32:0/1605306187 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f9c038050 con 0x7f7f90003a00 2024-08-03T21:00:14.815 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:14.812+0000 7f7fa3e00640 1 -- 172.21.15.32:0/1605306187 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f7f9c039040 con 0x7f7f90003a00 2024-08-03T21:00:14.815 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:14.812+0000 7f7fb1000640 1 -- 172.21.15.32:0/1605306187 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7fac09aaf0 con 0x7f7f90003a00 2024-08-03T21:00:14.816 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:14.812+0000 7f7fa3e00640 1 -- 172.21.15.32:0/1605306187 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f9c0339f0 con 0x7f7f90003a00 2024-08-03T21:00:14.816 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:14.813+0000 7f7fb1000640 1 -- 172.21.15.32:0/1605306187 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7fac09b0b0 con 0x7f7f90003a00 2024-08-03T21:00:14.816 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:14.814+0000 7f7fb1000640 1 -- 172.21.15.32:0/1605306187 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7fac0caf80 con 0x7f7f90003a00 2024-08-03T21:00:14.816 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:14.815+0000 7f7fa3e00640 1 -- 172.21.15.32:0/1605306187 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f7f9c046020 con 0x7f7f90003a00 2024-08-03T21:00:14.818 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:14.816+0000 7f7fa3e00640 1 --2- 172.21.15.32:0/1605306187 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f7f88077040 0x7f7f88079550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:00:14.818 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:14.817+0000 7f7fa3e00640 1 -- 172.21.15.32:0/1605306187 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(123..123 src has 1..123) ==== 9122+0+0 (secure 0 0 0) 0x7f7f9c0ba0a0 con 0x7f7f90003a00 2024-08-03T21:00:14.818 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:14.817+0000 7f7fa2a00640 1 --2- 172.21.15.32:0/1605306187 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f7f88077040 0x7f7f88079550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:00:14.819 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:14.818+0000 7f7fa2a00640 1 --2- 172.21.15.32:0/1605306187 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f7f88077040 0x7f7f88079550 secure :-1 s=READY pgs=815 cs=0 l=1 rev1=1 crypto rx=0x7f7fac0cdff0 tx=0x7f7f94005f90 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T21:00:14.822 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:14.821+0000 7f7fa3e00640 1 -- 172.21.15.32:0/1605306187 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7f7fac0caf80 con 0x7f7f90003a00 2024-08-03T21:00:15.067 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:15.065+0000 7f7fb1000640 1 -- 172.21.15.32:0/1605306187 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f7fac0afe20 con 0x7f7f90003a00 2024-08-03T21:00:15.067 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:15.066+0000 7f7fa3e00640 1 -- 172.21.15.32:0/1605306187 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 316 v316) ==== 95+0+4391 (secure 0 0 0) 0x7f7fac0afe20 con 0x7f7f90003a00 2024-08-03T21:00:15.067 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-03T21:00:15.067 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":316,"btime":"2024-08-03T21:00:10:997611+0000","default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6439,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6835/531080160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6834","nonce":531080160},{"type":"v1","addr":"172.21.15.118:6835","nonce":531080160}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":84},{"gid":6454,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6837/2796882319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6836","nonce":2796882319},{"type":"v1","addr":"172.21.15.118:6837","nonce":2796882319}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":86},{"gid":6813,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6834/2686933890","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":2686933890},{"type":"v1","addr":"172.21.15.32:6834","nonce":2686933890}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":85}],"filesystems":[{"mdsmap":{"epoch":316,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-08-03T20:54:26.950518+0000","modified":"2024-08-03T21:00:10.997604+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":123,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":6}]} 2024-08-03T21:00:15.068 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 316 2024-08-03T21:00:15.070 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:15.068+0000 7f7fb1000640 1 -- 172.21.15.32:0/1605306187 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f7f88077040 msgr2=0x7f7f88079550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:00:15.070 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:15.068+0000 7f7fb1000640 1 --2- 172.21.15.32:0/1605306187 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f7f88077040 0x7f7f88079550 secure :-1 s=READY pgs=815 cs=0 l=1 rev1=1 crypto rx=0x7f7fac0cdff0 tx=0x7f7f94005f90 comp rx=0 tx=0).stop 2024-08-03T21:00:15.070 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:15.068+0000 7f7fb1000640 1 -- 172.21.15.32:0/1605306187 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7f90003a00 msgr2=0x7f7fac0b5640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:00:15.070 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:15.068+0000 7f7fb1000640 1 --2- 172.21.15.32:0/1605306187 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7f90003a00 0x7f7fac0b5640 secure :-1 s=READY pgs=2041 cs=0 l=1 rev1=1 crypto rx=0x7f7f9c009c70 tx=0x7f7f9c033670 comp rx=0 tx=0).stop 2024-08-03T21:00:15.070 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:15.068+0000 7f7fb1000640 1 -- 172.21.15.32:0/1605306187 shutdown_connections 2024-08-03T21:00:15.070 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:15.068+0000 7f7fb1000640 1 --2- 172.21.15.32:0/1605306187 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f7f88077040 0x7f7f88079550 unknown :-1 s=CLOSED pgs=815 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:15.071 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:15.068+0000 7f7fb1000640 1 --2- 172.21.15.32:0/1605306187 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f7fac155f80 0x7f7fac15a8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:15.071 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:15.068+0000 7f7fb1000640 1 --2- 172.21.15.32:0/1605306187 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f7fac15adf0 0x7f7fac1f4ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:15.071 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:15.068+0000 7f7fb1000640 1 --2- 172.21.15.32:0/1605306187 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7f90003a00 0x7f7fac0b5640 unknown :-1 s=CLOSED pgs=2041 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:15.071 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:15.068+0000 7f7fb1000640 1 -- 172.21.15.32:0/1605306187 >> 172.21.15.32:0/1605306187 conn(0x7f7fac0b2120 msgr2=0x7f7fac073720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T21:00:15.071 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:15.068+0000 7f7fb1000640 1 -- 172.21.15.32:0/1605306187 shutdown_connections 2024-08-03T21:00:15.071 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:15.068+0000 7f7fb1000640 1 -- 172.21.15.32:0/1605306187 wait complete. 2024-08-03T21:00:15.092 INFO:tasks.cephfs.test_damage:Result: Mutation 'Corrupt omap header on 100.00000000' led to DAMAGED state 2024-08-03T21:00:15.092 INFO:tasks.cephfs.test_damage:Applying mutation 'Corrupt omap header on mds0_sessionmap' 2024-08-03T21:00:15.092 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-08-03T21:00:15.093 DEBUG:teuthology.orchestra.run.smithi032:> dd if=/proc/self/mounts of=/dev/stdout 2024-08-03T21:00:15.128 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053453,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6499208k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 none /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249604k,nr_inodes=812401,mode=700,uid=1000,gid=1316 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6499208k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-08-03T21:00:15.128 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi032.front.sepia.ceph.com 2024-08-03T21:00:15.128 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-08-03T21:00:15.128 DEBUG:teuthology.orchestra.run.smithi032:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-08-03T21:00:15.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-03T21:00:15.204 INFO:teuthology.orchestra.run.smithi032.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.0': No such file or directory 2024-08-03T21:00:15.204 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-08-03T21:00:15.473 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:15.471+0000 7fae03200640 1 -- 172.21.15.32:0/2232809846 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fae040d26e0 msgr2=0x7fae04161b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:00:15.474 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:15.471+0000 7fae03200640 1 --2- 172.21.15.32:0/2232809846 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fae040d26e0 0x7fae04161b80 secure :-1 s=READY pgs=2042 cs=0 l=1 rev1=1 crypto rx=0x7fae0408e370 tx=0x7fadf402a560 comp rx=0 tx=0).stop 2024-08-03T21:00:15.474 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:15.471+0000 7fae03200640 1 -- 172.21.15.32:0/2232809846 shutdown_connections 2024-08-03T21:00:15.474 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:15.471+0000 7fae03200640 1 --2- 172.21.15.32:0/2232809846 >> v2:172.21.15.118:3300/0 conn(0x7fae040b43d0 0x7fae040cf510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:15.475 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:15.471+0000 7fae03200640 1 --2- 172.21.15.32:0/2232809846 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fae040cffe0 0x7fae040b3e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:15.475 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:15.471+0000 7fae03200640 1 --2- 172.21.15.32:0/2232809846 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fae040d26e0 0x7fae04161b80 unknown :-1 s=CLOSED pgs=2042 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:15.475 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:15.471+0000 7fae03200640 1 -- 172.21.15.32:0/2232809846 >> 172.21.15.32:0/2232809846 conn(0x7fae0408d7a0 msgr2=0x7fae0409d3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T21:00:15.475 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:15.471+0000 7fae03200640 1 -- 172.21.15.32:0/2232809846 shutdown_connections 2024-08-03T21:00:15.476 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:15.471+0000 7fae03200640 1 -- 172.21.15.32:0/2232809846 wait complete. 2024-08-03T21:00:15.480 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:15.473+0000 7fae03200640 1 Processor -- start 2024-08-03T21:00:15.480 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:15.473+0000 7fae03200640 1 -- start start 2024-08-03T21:00:15.480 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:15.475+0000 7fae03200640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fae040b43d0 0x7fae0415d680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:00:15.480 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:15.475+0000 7fae03200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fae040cffe0 0x7fae041f56e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:00:15.480 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:15.475+0000 7fae03200640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fae040d26e0 0x7fae041f5c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:00:15.481 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:15.475+0000 7fae03200640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fae040d3f80 con 0x7fae040cffe0 2024-08-03T21:00:15.481 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:15.475+0000 7fae03200640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7fae040cfda0 con 0x7fae040b43d0 2024-08-03T21:00:15.481 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:15.475+0000 7fae03200640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7fae040d4100 con 0x7fae040d26e0 2024-08-03T21:00:15.481 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:15.475+0000 7fadfbe00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fae040cffe0 0x7fae041f56e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:00:15.481 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:15.475+0000 7fadfbe00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fae040cffe0 0x7fae041f56e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:43996/0 (socket says 172.21.15.32:43996) 2024-08-03T21:00:15.481 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:15.475+0000 7fadfbe00640 1 -- 172.21.15.32:0/3812672492 learned_addr learned my addr 172.21.15.32:0/3812672492 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T21:00:15.481 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:15.475+0000 7fae01e00640 1 --2- 172.21.15.32:0/3812672492 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fae040b43d0 0x7fae0415d680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:00:15.481 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:15.475+0000 7fae02800640 1 --2- 172.21.15.32:0/3812672492 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fae040d26e0 0x7fae041f5c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:00:15.482 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:15.476+0000 7fadfbe00640 1 -- 172.21.15.32:0/3812672492 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fae040d26e0 msgr2=0x7fae041f5c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:00:15.482 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:15.476+0000 7fadfbe00640 1 --2- 172.21.15.32:0/3812672492 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fae040d26e0 0x7fae041f5c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:15.482 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:15.476+0000 7fadfbe00640 1 -- 172.21.15.32:0/3812672492 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fae040b43d0 msgr2=0x7fae0415d680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:00:15.482 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:15.476+0000 7fadfbe00640 1 --2- 172.21.15.32:0/3812672492 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fae040b43d0 0x7fae0415d680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:15.482 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:15.476+0000 7fadfbe00640 1 -- 172.21.15.32:0/3812672492 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fae040c98c0 con 0x7fae040cffe0 2024-08-03T21:00:15.482 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:15.476+0000 7fadfbe00640 1 --2- 172.21.15.32:0/3812672492 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fae040cffe0 0x7fae041f56e0 secure :-1 s=READY pgs=2043 cs=0 l=1 rev1=1 crypto rx=0x7fadec00d6d0 tx=0x7fadec00db90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T21:00:15.482 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:15.476+0000 7fadfa000640 1 -- 172.21.15.32:0/3812672492 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fadec017070 con 0x7fae040cffe0 2024-08-03T21:00:15.482 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:15.476+0000 7fadfa000640 1 -- 172.21.15.32:0/3812672492 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fadec005330 con 0x7fae040cffe0 2024-08-03T21:00:15.482 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:15.476+0000 7fadfa000640 1 -- 172.21.15.32:0/3812672492 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fadec0097c0 con 0x7fae040cffe0 2024-08-03T21:00:15.482 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:15.476+0000 7fae03200640 1 -- 172.21.15.32:0/3812672492 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fae040ca380 con 0x7fae040cffe0 2024-08-03T21:00:15.483 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:15.476+0000 7fae03200640 1 -- 172.21.15.32:0/3812672492 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fae040a1840 con 0x7fae040cffe0 2024-08-03T21:00:15.483 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:15.477+0000 7fae03200640 1 -- 172.21.15.32:0/3812672492 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fae040b3e90 con 0x7fae040cffe0 2024-08-03T21:00:15.483 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:15.478+0000 7fadfa000640 1 -- 172.21.15.32:0/3812672492 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7fadec009b70 con 0x7fae040cffe0 2024-08-03T21:00:15.484 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:15.479+0000 7fadfa000640 1 --2- 172.21.15.32:0/3812672492 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fade0077010 0x7fade0079520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:00:15.484 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:15.480+0000 7fadfa000640 1 -- 172.21.15.32:0/3812672492 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(123..123 src has 1..123) ==== 9122+0+0 (secure 0 0 0) 0x7fadec099920 con 0x7fae040cffe0 2024-08-03T21:00:15.484 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:15.480+0000 7fae01e00640 1 --2- 172.21.15.32:0/3812672492 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fade0077010 0x7fade0079520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:00:15.484 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:15.481+0000 7fae01e00640 1 --2- 172.21.15.32:0/3812672492 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fade0077010 0x7fade0079520 secure :-1 s=READY pgs=816 cs=0 l=1 rev1=1 crypto rx=0x7fadf4033000 tx=0x7fadf4004730 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T21:00:15.485 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:15.484+0000 7fadfa000640 1 -- 172.21.15.32:0/3812672492 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7fadec0616c0 con 0x7fae040cffe0 2024-08-03T21:00:15.734 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:15.732+0000 7fae03200640 1 -- 172.21.15.32:0/3812672492 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7fae040b3520 con 0x7fae040cffe0 2024-08-03T21:00:15.749 INFO:tasks.ceph.mds.a.smithi032.stderr: -49> 2024-08-03T21:00:10.993+0000 7f75d2400640 -1 mds.0.cache.dir(0x100) Corrupt fnode in dirfrag 0x100: Decoder at 'void fnode_t::decode(ceph::buffer::v15_2_0::list::const_iterator&)' v=3 cannot decode v=100 minimal_decoder=101: Malformed input [buffer:3] 2024-08-03T21:00:15.863 INFO:tasks.ceph.mds.a.smithi032.stdout:starting mds.a at 2024-08-03T21:00:16.022 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:16.019+0000 7fadfa000640 1 -- 172.21.15.32:0/3812672492 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v317) ==== 161+0+0 (secure 0 0 0) 0x7fadec065c10 con 0x7fae040cffe0 2024-08-03T21:00:16.022 INFO:teuthology.orchestra.run.smithi032.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-08-03T21:00:16.027 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:16.022+0000 7fae03200640 1 -- 172.21.15.32:0/3812672492 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fade0077010 msgr2=0x7fade0079520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:00:16.027 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:16.022+0000 7fae03200640 1 --2- 172.21.15.32:0/3812672492 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fade0077010 0x7fade0079520 secure :-1 s=READY pgs=816 cs=0 l=1 rev1=1 crypto rx=0x7fadf4033000 tx=0x7fadf4004730 comp rx=0 tx=0).stop 2024-08-03T21:00:16.027 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:16.022+0000 7fae03200640 1 -- 172.21.15.32:0/3812672492 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fae040cffe0 msgr2=0x7fae041f56e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:00:16.027 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:16.022+0000 7fae03200640 1 --2- 172.21.15.32:0/3812672492 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fae040cffe0 0x7fae041f56e0 secure :-1 s=READY pgs=2043 cs=0 l=1 rev1=1 crypto rx=0x7fadec00d6d0 tx=0x7fadec00db90 comp rx=0 tx=0).stop 2024-08-03T21:00:16.028 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:16.023+0000 7fae03200640 1 -- 172.21.15.32:0/3812672492 shutdown_connections 2024-08-03T21:00:16.028 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:16.023+0000 7fae03200640 1 --2- 172.21.15.32:0/3812672492 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fade0077010 0x7fade0079520 unknown :-1 s=CLOSED pgs=816 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:16.028 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:16.023+0000 7fae03200640 1 --2- 172.21.15.32:0/3812672492 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fae040d26e0 0x7fae041f5c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:16.028 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:16.023+0000 7fae03200640 1 --2- 172.21.15.32:0/3812672492 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fae040cffe0 0x7fae041f56e0 unknown :-1 s=CLOSED pgs=2043 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:16.028 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:16.023+0000 7fae03200640 1 --2- 172.21.15.32:0/3812672492 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fae040b43d0 0x7fae0415d680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:16.028 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:16.023+0000 7fae03200640 1 -- 172.21.15.32:0/3812672492 >> 172.21.15.32:0/3812672492 conn(0x7fae0408d7a0 msgr2=0x7fae0406ed30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T21:00:16.029 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:16.023+0000 7fae03200640 1 -- 172.21.15.32:0/3812672492 shutdown_connections 2024-08-03T21:00:16.029 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:16.023+0000 7fae03200640 1 -- 172.21.15.32:0/3812672492 wait complete. 2024-08-03T21:00:16.048 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mds repaired 0 2024-08-03T21:00:16.279 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:16.276+0000 7f9b27800640 1 -- 172.21.15.32:0/685438671 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9b20144f90 msgr2=0x7f9b20074570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:00:16.279 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:16.276+0000 7f9b27800640 1 --2- 172.21.15.32:0/685438671 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9b20144f90 0x7f9b20074570 secure :-1 s=READY pgs=2047 cs=0 l=1 rev1=1 crypto rx=0x7f9b20073b50 tx=0x7f9b0c02a660 comp rx=0 tx=0).stop 2024-08-03T21:00:16.279 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:16.276+0000 7f9b27800640 1 -- 172.21.15.32:0/685438671 shutdown_connections 2024-08-03T21:00:16.279 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:16.276+0000 7f9b27800640 1 --2- 172.21.15.32:0/685438671 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9b20144f90 0x7f9b20074570 unknown :-1 s=CLOSED pgs=2047 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:16.280 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:16.276+0000 7f9b27800640 1 -- 172.21.15.32:0/685438671 >> 172.21.15.32:0/685438671 conn(0x7f9b200b1bd0 msgr2=0x7f9b200b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T21:00:16.280 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:16.276+0000 7f9b27800640 1 -- 172.21.15.32:0/685438671 shutdown_connections 2024-08-03T21:00:16.280 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:16.276+0000 7f9b27800640 1 -- 172.21.15.32:0/685438671 wait complete. 2024-08-03T21:00:16.281 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:16.278+0000 7f9b27800640 1 Processor -- start 2024-08-03T21:00:16.281 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:16.279+0000 7f9b27800640 1 -- start start 2024-08-03T21:00:16.285 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:16.280+0000 7f9b27800640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f9b2006e930 0x7f9b20143c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:00:16.285 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:16.280+0000 7f9b27800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9b20144f90 0x7f9b201f8cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:00:16.285 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:16.280+0000 7f9b27800640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f9b2015fff0 0x7f9b201e9f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:00:16.285 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:16.280+0000 7f9b27800640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f9b200cf840 con 0x7f9b20144f90 2024-08-03T21:00:16.285 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:16.280+0000 7f9b27800640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f9b20145f30 con 0x7f9b2015fff0 2024-08-03T21:00:16.286 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:16.280+0000 7f9b27800640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7f9b200cf9c0 con 0x7f9b2006e930 2024-08-03T21:00:16.286 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:16.280+0000 7f9b1fe00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9b20144f90 0x7f9b201f8cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:00:16.286 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:16.280+0000 7f9b1fe00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9b20144f90 0x7f9b201f8cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:44008/0 (socket says 172.21.15.32:44008) 2024-08-03T21:00:16.286 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:16.280+0000 7f9b1fe00640 1 -- 172.21.15.32:0/2293219983 learned_addr learned my addr 172.21.15.32:0/2293219983 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T21:00:16.286 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:16.280+0000 7f9b25800640 1 --2- 172.21.15.32:0/2293219983 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f9b2015fff0 0x7f9b201e9f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:00:16.286 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:16.280+0000 7f9b24e00640 1 --2- 172.21.15.32:0/2293219983 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f9b2006e930 0x7f9b20143c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:00:16.287 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:16.281+0000 7f9b1fe00640 1 -- 172.21.15.32:0/2293219983 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f9b2006e930 msgr2=0x7f9b20143c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:00:16.287 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:16.281+0000 7f9b1fe00640 1 --2- 172.21.15.32:0/2293219983 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f9b2006e930 0x7f9b20143c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:16.287 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:16.281+0000 7f9b1fe00640 1 -- 172.21.15.32:0/2293219983 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f9b2015fff0 msgr2=0x7f9b201e9f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:00:16.287 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:16.281+0000 7f9b1fe00640 1 --2- 172.21.15.32:0/2293219983 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f9b2015fff0 0x7f9b201e9f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:16.287 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:16.281+0000 7f9b1fe00640 1 -- 172.21.15.32:0/2293219983 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9b201f99c0 con 0x7f9b20144f90 2024-08-03T21:00:16.287 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:16.281+0000 7f9b25800640 1 --2- 172.21.15.32:0/2293219983 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f9b2015fff0 0x7f9b201e9f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-03T21:00:16.287 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:16.281+0000 7f9b1fe00640 1 --2- 172.21.15.32:0/2293219983 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9b20144f90 0x7f9b201f8cd0 secure :-1 s=READY pgs=2048 cs=0 l=1 rev1=1 crypto rx=0x7f9b0c034040 tx=0x7f9b0c0039e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T21:00:16.288 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:16.281+0000 7f9b1cc00640 1 -- 172.21.15.32:0/2293219983 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b0c03e030 con 0x7f9b20144f90 2024-08-03T21:00:16.288 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:16.281+0000 7f9b1cc00640 1 -- 172.21.15.32:0/2293219983 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f9b0c0047a0 con 0x7f9b20144f90 2024-08-03T21:00:16.288 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:16.281+0000 7f9b27800640 1 -- 172.21.15.32:0/2293219983 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9b2014b6b0 con 0x7f9b20144f90 2024-08-03T21:00:16.288 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:16.281+0000 7f9b1cc00640 1 -- 172.21.15.32:0/2293219983 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b0c035040 con 0x7f9b20144f90 2024-08-03T21:00:16.288 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:16.282+0000 7f9b27800640 1 -- 172.21.15.32:0/2293219983 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9b2014bc70 con 0x7f9b20144f90 2024-08-03T21:00:16.288 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:16.283+0000 7f9b27800640 1 -- 172.21.15.32:0/2293219983 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9b201f9730 con 0x7f9b20144f90 2024-08-03T21:00:16.289 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:16.284+0000 7f9b1cc00640 1 -- 172.21.15.32:0/2293219983 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f9b0c041030 con 0x7f9b20144f90 2024-08-03T21:00:16.290 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:16.285+0000 7f9b1cc00640 1 --2- 172.21.15.32:0/2293219983 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f9b00077010 0x7f9b00079520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:00:16.290 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:16.285+0000 7f9b1cc00640 1 -- 172.21.15.32:0/2293219983 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(123..123 src has 1..123) ==== 9122+0+0 (secure 0 0 0) 0x7f9b0c0b9f80 con 0x7f9b20144f90 2024-08-03T21:00:16.290 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:16.286+0000 7f9b24e00640 1 --2- 172.21.15.32:0/2293219983 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f9b00077010 0x7f9b00079520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:00:16.290 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:16.287+0000 7f9b24e00640 1 --2- 172.21.15.32:0/2293219983 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f9b00077010 0x7f9b00079520 secure :-1 s=READY pgs=818 cs=0 l=1 rev1=1 crypto rx=0x7f9b140015e0 tx=0x7f9b14009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T21:00:16.291 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:16.290+0000 7f9b1cc00640 1 -- 172.21.15.32:0/2293219983 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7f9b0c0823d0 con 0x7f9b20144f90 2024-08-03T21:00:16.571 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:16.570+0000 7f9b27800640 1 -- 172.21.15.32:0/2293219983 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "mds repaired", "role": "0"} v 0) -- 0x7f9b20160400 con 0x7f9b20144f90 2024-08-03T21:00:17.029 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.027+0000 7f9b1cc00640 1 -- 172.21.15.32:0/2293219983 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "mds repaired", "role": "0"}]=0 repaired: restoring rank 6:0 v318) ==== 101+0+0 (secure 0 0 0) 0x7f9b0c081cd0 con 0x7f9b20144f90 2024-08-03T21:00:17.030 INFO:teuthology.orchestra.run.smithi032.stderr:repaired: restoring rank 6:0 2024-08-03T21:00:17.033 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.031+0000 7f9b27800640 1 -- 172.21.15.32:0/2293219983 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f9b00077010 msgr2=0x7f9b00079520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:00:17.033 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.031+0000 7f9b27800640 1 --2- 172.21.15.32:0/2293219983 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f9b00077010 0x7f9b00079520 secure :-1 s=READY pgs=818 cs=0 l=1 rev1=1 crypto rx=0x7f9b140015e0 tx=0x7f9b14009040 comp rx=0 tx=0).stop 2024-08-03T21:00:17.033 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.031+0000 7f9b27800640 1 -- 172.21.15.32:0/2293219983 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9b20144f90 msgr2=0x7f9b201f8cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:00:17.033 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.031+0000 7f9b27800640 1 --2- 172.21.15.32:0/2293219983 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9b20144f90 0x7f9b201f8cd0 secure :-1 s=READY pgs=2048 cs=0 l=1 rev1=1 crypto rx=0x7f9b0c034040 tx=0x7f9b0c0039e0 comp rx=0 tx=0).stop 2024-08-03T21:00:17.033 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.031+0000 7f9b27800640 1 -- 172.21.15.32:0/2293219983 shutdown_connections 2024-08-03T21:00:17.033 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.031+0000 7f9b27800640 1 --2- 172.21.15.32:0/2293219983 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f9b00077010 0x7f9b00079520 unknown :-1 s=CLOSED pgs=818 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:17.033 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.031+0000 7f9b27800640 1 --2- 172.21.15.32:0/2293219983 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f9b2015fff0 0x7f9b201e9f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:17.034 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.031+0000 7f9b27800640 1 --2- 172.21.15.32:0/2293219983 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9b20144f90 0x7f9b201f8cd0 unknown :-1 s=CLOSED pgs=2048 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:17.034 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.031+0000 7f9b27800640 1 --2- 172.21.15.32:0/2293219983 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f9b2006e930 0x7f9b20143c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:17.034 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.031+0000 7f9b27800640 1 -- 172.21.15.32:0/2293219983 >> 172.21.15.32:0/2293219983 conn(0x7f9b200b1bd0 msgr2=0x7f9b200b0120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T21:00:17.034 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.031+0000 7f9b27800640 1 -- 172.21.15.32:0/2293219983 shutdown_connections 2024-08-03T21:00:17.034 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.031+0000 7f9b27800640 1 -- 172.21.15.32:0/2293219983 wait complete. 2024-08-03T21:00:17.057 DEBUG:teuthology.orchestra.run.smithi032:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata import - 2024-08-03T21:00:17.199 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.198+0000 7f62839e0f40 1 Processor -- start 2024-08-03T21:00:17.200 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.198+0000 7f62839e0f40 1 -- start start 2024-08-03T21:00:17.201 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.199+0000 7f62839e0f40 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5563ef2ad360 0x5563ef2c5900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:00:17.201 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.199+0000 7f62839e0f40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5563ef2aeea0 0x5563ef1f4da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:00:17.202 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.199+0000 7f62839e0f40 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x5563ef2c4330 0x5563ef1f64f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:00:17.202 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.199+0000 7f62839e0f40 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x5563ef144e30 con 0x5563ef2ad360 2024-08-03T21:00:17.202 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.199+0000 7f62839e0f40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x5563ef1a2bf0 con 0x5563ef2aeea0 2024-08-03T21:00:17.202 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.199+0000 7f62839e0f40 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x5563ef2ae440 con 0x5563ef2c4330 2024-08-03T21:00:17.202 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.199+0000 7f627cc00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5563ef2ad360 0x5563ef2c5900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:00:17.202 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.200+0000 7f627cc00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5563ef2ad360 0x5563ef2c5900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:44028/0 (socket says 172.21.15.32:44028) 2024-08-03T21:00:17.202 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.200+0000 7f627cc00640 1 -- 172.21.15.32:0/1490644686 learned_addr learned my addr 172.21.15.32:0/1490644686 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T21:00:17.202 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.200+0000 7f6277e00640 1 --2- 172.21.15.32:0/1490644686 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5563ef2aeea0 0x5563ef1f4da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:00:17.202 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.200+0000 7f627d600640 1 --2- 172.21.15.32:0/1490644686 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x5563ef2c4330 0x5563ef1f64f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:00:17.202 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.200+0000 7f627cc00640 1 -- 172.21.15.32:0/1490644686 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x5563ef2c4330 msgr2=0x5563ef1f64f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T21:00:17.202 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.200+0000 7f627cc00640 1 --2- 172.21.15.32:0/1490644686 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x5563ef2c4330 0x5563ef1f64f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:17.203 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.200+0000 7f627cc00640 1 -- 172.21.15.32:0/1490644686 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5563ef2aeea0 msgr2=0x5563ef1f4da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T21:00:17.203 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.200+0000 7f627cc00640 1 --2- 172.21.15.32:0/1490644686 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5563ef2aeea0 0x5563ef1f4da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:17.203 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.200+0000 7f627cc00640 1 -- 172.21.15.32:0/1490644686 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5563ef2c5e40 con 0x5563ef2ad360 2024-08-03T21:00:17.204 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.200+0000 7f627cc00640 1 --2- 172.21.15.32:0/1490644686 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5563ef2ad360 0x5563ef2c5900 secure :-1 s=READY pgs=2050 cs=0 l=1 rev1=1 crypto rx=0x7f6270001cf0 tx=0x7f627000b5b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3ddc4fea9e951daf server_cookie=0 in_seq=0 out_seq=0 2024-08-03T21:00:17.204 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.201+0000 7f6276a00640 1 -- 172.21.15.32:0/1490644686 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f627000bd80 con 0x5563ef2ad360 2024-08-03T21:00:17.204 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.201+0000 7f6276a00640 1 -- 172.21.15.32:0/1490644686 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f627000f400 con 0x5563ef2ad360 2024-08-03T21:00:17.204 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.201+0000 7f6276a00640 1 -- 172.21.15.32:0/1490644686 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f627000f890 con 0x5563ef2ad360 2024-08-03T21:00:17.204 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.201+0000 7f62839e0f40 1 -- 172.21.15.32:0/1490644686 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5563ef2ad360 msgr2=0x5563ef2c5900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:00:17.204 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.201+0000 7f62839e0f40 1 --2- 172.21.15.32:0/1490644686 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5563ef2ad360 0x5563ef2c5900 secure :-1 s=READY pgs=2050 cs=0 l=1 rev1=1 crypto rx=0x7f6270001cf0 tx=0x7f627000b5b0 comp rx=0 tx=0).stop 2024-08-03T21:00:17.204 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.201+0000 7f62839e0f40 1 -- 172.21.15.32:0/1490644686 shutdown_connections 2024-08-03T21:00:17.204 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.201+0000 7f62839e0f40 1 --2- 172.21.15.32:0/1490644686 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5563ef2aeea0 0x5563ef1f4da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:17.205 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.201+0000 7f62839e0f40 1 --2- 172.21.15.32:0/1490644686 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x5563ef2c4330 0x5563ef1f64f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:17.205 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.201+0000 7f62839e0f40 1 --2- 172.21.15.32:0/1490644686 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5563ef2ad360 0x5563ef2c5900 unknown :-1 s=CLOSED pgs=2050 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:17.205 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.201+0000 7f62839e0f40 1 -- 172.21.15.32:0/1490644686 >> 172.21.15.32:0/1490644686 conn(0x5563ef116ac0 msgr2=0x5563ef2c6ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T21:00:17.205 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.201+0000 7f62839e0f40 1 -- 172.21.15.32:0/1490644686 shutdown_connections 2024-08-03T21:00:17.205 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.202+0000 7f62839e0f40 1 -- 172.21.15.32:0/1490644686 wait complete. 2024-08-03T21:00:17.205 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.203+0000 7f62839e0f40 1 Processor -- start 2024-08-03T21:00:17.205 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.203+0000 7f62839e0f40 1 -- start start 2024-08-03T21:00:17.206 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.204+0000 7f62839e0f40 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5563ef2ad360 0x5563ef2399d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:00:17.206 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.204+0000 7f62839e0f40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5563ef2aeea0 0x5563ef2ab700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:00:17.206 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.204+0000 7f62839e0f40 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x5563ef2c4330 0x5563ef23e070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:00:17.206 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.204+0000 7f62839e0f40 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x5563ef23ffa0 con 0x5563ef2ad360 2024-08-03T21:00:17.206 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.204+0000 7f62839e0f40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x5563ef2c9ba0 con 0x5563ef2aeea0 2024-08-03T21:00:17.206 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.204+0000 7f62839e0f40 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x5563ef3632e0 con 0x5563ef2c4330 2024-08-03T21:00:17.206 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.204+0000 7f627cc00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5563ef2ad360 0x5563ef2399d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:00:17.206 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.205+0000 7f627cc00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5563ef2ad360 0x5563ef2399d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:44040/0 (socket says 172.21.15.32:44040) 2024-08-03T21:00:17.207 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.205+0000 7f627cc00640 1 -- 172.21.15.32:0/1475946011 learned_addr learned my addr 172.21.15.32:0/1475946011 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T21:00:17.208 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.205+0000 7f6277e00640 1 --2- 172.21.15.32:0/1475946011 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5563ef2aeea0 0x5563ef2ab700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:00:17.208 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.205+0000 7f627d600640 1 --2- 172.21.15.32:0/1475946011 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x5563ef2c4330 0x5563ef23e070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:00:17.208 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.205+0000 7f627cc00640 1 -- 172.21.15.32:0/1475946011 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x5563ef2c4330 msgr2=0x5563ef23e070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:00:17.208 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.205+0000 7f627cc00640 1 --2- 172.21.15.32:0/1475946011 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x5563ef2c4330 0x5563ef23e070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:17.208 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.205+0000 7f627cc00640 1 -- 172.21.15.32:0/1475946011 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5563ef2aeea0 msgr2=0x5563ef2ab700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:00:17.208 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.205+0000 7f627cc00640 1 --2- 172.21.15.32:0/1475946011 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5563ef2aeea0 0x5563ef2ab700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:17.208 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.205+0000 7f627cc00640 1 -- 172.21.15.32:0/1475946011 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5563ef2abc40 con 0x5563ef2ad360 2024-08-03T21:00:17.208 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.205+0000 7f6277e00640 1 --2- 172.21.15.32:0/1475946011 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5563ef2aeea0 0x5563ef2ab700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-03T21:00:17.208 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.205+0000 7f627d600640 1 --2- 172.21.15.32:0/1475946011 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x5563ef2c4330 0x5563ef23e070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-03T21:00:17.208 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.205+0000 7f627cc00640 1 --2- 172.21.15.32:0/1475946011 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5563ef2ad360 0x5563ef2399d0 secure :-1 s=READY pgs=2051 cs=0 l=1 rev1=1 crypto rx=0x7f627000bab0 tx=0x7f627000fdd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T21:00:17.208 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.205+0000 7f6275600640 1 -- 172.21.15.32:0/1475946011 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f627001b430 con 0x5563ef2ad360 2024-08-03T21:00:17.209 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.205+0000 7f6275600640 1 -- 172.21.15.32:0/1475946011 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f6270020020 con 0x5563ef2ad360 2024-08-03T21:00:17.209 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.206+0000 7f62839e0f40 1 -- 172.21.15.32:0/1475946011 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x5563ef2c75c0 con 0x5563ef2ad360 2024-08-03T21:00:17.209 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.206+0000 7f6275600640 1 -- 172.21.15.32:0/1475946011 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f627001b760 con 0x5563ef2ad360 2024-08-03T21:00:17.209 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.206+0000 7f62839e0f40 1 -- 172.21.15.32:0/1475946011 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5563ef23f230 con 0x5563ef2ad360 2024-08-03T21:00:17.209 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.208+0000 7f6275600640 1 -- 172.21.15.32:0/1475946011 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f6270017070 con 0x5563ef2ad360 2024-08-03T21:00:17.210 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.209+0000 7f6275600640 1 --2- 172.21.15.32:0/1475946011 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f625c0759f0 0x7f625c077f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:00:17.210 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.209+0000 7f6275600640 1 -- 172.21.15.32:0/1475946011 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(123..123 src has 1..123) ==== 9122+0+0 (secure 0 0 0) 0x7f62700994f0 con 0x5563ef2ad360 2024-08-03T21:00:17.211 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.209+0000 7f6277e00640 1 --2- 172.21.15.32:0/1475946011 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f625c0759f0 0x7f625c077f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:00:17.211 INFO:teuthology.orchestra.run.smithi032.stdout:Importing pool 2024-08-03T21:00:17.212 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.209+0000 7f62839e0f40 1 --2- 172.21.15.32:0/1475946011 >> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] conn(0x5563ef23f6a0 0x5563ef363c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:00:17.212 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.209+0000 7f62839e0f40 1 -- 172.21.15.32:0/1475946011 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:1 12.10 12:09401a81:::601.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123) -- 0x5563ef2c5a70 con 0x5563ef23f6a0 2024-08-03T21:00:17.212 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.210+0000 7f627d600640 1 --2- 172.21.15.32:0/1475946011 >> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] conn(0x5563ef23f6a0 0x5563ef363c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:00:17.212 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.210+0000 7f6277e00640 1 --2- 172.21.15.32:0/1475946011 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f625c0759f0 0x7f625c077f00 secure :-1 s=READY pgs=819 cs=0 l=1 rev1=1 crypto rx=0x5563ef2aeaa0 tx=0x7f6268004930 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T21:00:17.212 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.210+0000 7f627d600640 1 --2- 172.21.15.32:0/1475946011 >> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] conn(0x5563ef23f6a0 0x5563ef363c50 crc :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T21:00:17.214 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.213+0000 7f627d600640 1 -- 172.21.15.32:0/1475946011 <== osd.0 v2:172.21.15.32:6816/3066156116 1 ==== osd_op_reply(1 601.00000000 [create] v123'1159 uv1133 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f627800b040 con 0x5563ef23f6a0 2024-08-03T21:00:17.214 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.213+0000 7f62839e0f40 1 -- 172.21.15.32:0/1475946011 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:2 12.10 12:09401a81:::601.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e123) -- 0x5563ef36fa90 con 0x5563ef23f6a0 2024-08-03T21:00:17.216 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.215+0000 7f627d600640 1 -- 172.21.15.32:0/1475946011 <== osd.0 v2:172.21.15.32:6816/3066156116 2 ==== osd_op_reply(2 601.00000000 [delete] v123'1160 uv1160 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f627800b040 con 0x5563ef23f6a0 2024-08-03T21:00:17.216 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.215+0000 7f62839e0f40 1 -- 172.21.15.32:0/1475946011 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:3 12.10 12:09401a81:::601.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123) -- 0x5563ef36fe30 con 0x5563ef23f6a0 2024-08-03T21:00:17.218 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.216+0000 7f627d600640 1 -- 172.21.15.32:0/1475946011 <== osd.0 v2:172.21.15.32:6816/3066156116 3 ==== osd_op_reply(3 601.00000000 [create] v123'1161 uv1161 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f627800b040 con 0x5563ef23f6a0 2024-08-03T21:00:17.218 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::601.00000000:head# 2024-08-03T21:00:17.218 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.216+0000 7f62839e0f40 1 -- 172.21.15.32:0/1475946011 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:4 12.10 12:09401a81:::601.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123) -- 0x5563ef3711e0 con 0x5563ef23f6a0 2024-08-03T21:00:17.220 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.218+0000 7f627d600640 1 -- 172.21.15.32:0/1475946011 <== osd.0 v2:172.21.15.32:6816/3066156116 4 ==== osd_op_reply(4 601.00000000 [setxattr (30)] v123'1162 uv1162 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f627800b040 con 0x5563ef23f6a0 2024-08-03T21:00:17.220 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.218+0000 7f62839e0f40 1 -- 172.21.15.32:0/1475946011 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:5 12.10 12:09401a81:::601.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123) -- 0x5563ef371580 con 0x5563ef23f6a0 2024-08-03T21:00:17.222 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.220+0000 7f627d600640 1 -- 172.21.15.32:0/1475946011 <== osd.0 v2:172.21.15.32:6816/3066156116 5 ==== osd_op_reply(5 601.00000000 [setxattr (62)] v123'1163 uv1163 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f627800b040 con 0x5563ef23f6a0 2024-08-03T21:00:17.222 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.220+0000 7f62839e0f40 1 -- 172.21.15.32:0/1475946011 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:6 12.10 12:09401a81:::601.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123) -- 0x5563ef3701d0 con 0x5563ef23f6a0 2024-08-03T21:00:17.223 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.222+0000 7f627d600640 1 -- 172.21.15.32:0/1475946011 <== osd.0 v2:172.21.15.32:6816/3066156116 6 ==== osd_op_reply(6 601.00000000 [omap-set-header] v123'1164 uv1164 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f627800b040 con 0x5563ef23f6a0 2024-08-03T21:00:17.223 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.222+0000 7f62839e0f40 1 -- 172.21.15.32:0/1475946011 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:7 12.10 12:0bd6d154:::602.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123) -- 0x5563ef3705c0 con 0x5563ef23f6a0 2024-08-03T21:00:17.225 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.224+0000 7f627d600640 1 -- 172.21.15.32:0/1475946011 <== osd.0 v2:172.21.15.32:6816/3066156116 7 ==== osd_op_reply(7 602.00000000 [create] v123'1165 uv1139 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f627800b040 con 0x5563ef23f6a0 2024-08-03T21:00:17.225 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.224+0000 7f62839e0f40 1 -- 172.21.15.32:0/1475946011 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:8 12.10 12:0bd6d154:::602.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e123) -- 0x5563ef3709b0 con 0x5563ef23f6a0 2024-08-03T21:00:17.227 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.226+0000 7f627d600640 1 -- 172.21.15.32:0/1475946011 <== osd.0 v2:172.21.15.32:6816/3066156116 8 ==== osd_op_reply(8 602.00000000 [delete] v123'1166 uv1166 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f627800b040 con 0x5563ef23f6a0 2024-08-03T21:00:17.227 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.226+0000 7f62839e0f40 1 -- 172.21.15.32:0/1475946011 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:9 12.10 12:0bd6d154:::602.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123) -- 0x5563ef3709b0 con 0x5563ef23f6a0 2024-08-03T21:00:17.229 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::602.00000000:head# 2024-08-03T21:00:17.230 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.228+0000 7f627d600640 1 -- 172.21.15.32:0/1475946011 <== osd.0 v2:172.21.15.32:6816/3066156116 9 ==== osd_op_reply(9 602.00000000 [create] v123'1167 uv1167 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f627800b040 con 0x5563ef23f6a0 2024-08-03T21:00:17.230 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.228+0000 7f62839e0f40 1 -- 172.21.15.32:0/1475946011 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:10 12.10 12:0bd6d154:::602.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123) -- 0x5563ef370960 con 0x5563ef23f6a0 2024-08-03T21:00:17.231 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.230+0000 7f627d600640 1 -- 172.21.15.32:0/1475946011 <== osd.0 v2:172.21.15.32:6816/3066156116 10 ==== osd_op_reply(10 602.00000000 [setxattr (30)] v123'1168 uv1168 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f627800b040 con 0x5563ef23f6a0 2024-08-03T21:00:17.231 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.230+0000 7f62839e0f40 1 -- 172.21.15.32:0/1475946011 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:11 12.10 12:0bd6d154:::602.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123) -- 0x5563ef370960 con 0x5563ef23f6a0 2024-08-03T21:00:17.233 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.231+0000 7f627d600640 1 -- 172.21.15.32:0/1475946011 <== osd.0 v2:172.21.15.32:6816/3066156116 11 ==== osd_op_reply(11 602.00000000 [setxattr (62)] v123'1169 uv1169 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f627800b040 con 0x5563ef23f6a0 2024-08-03T21:00:17.233 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.231+0000 7f62839e0f40 1 -- 172.21.15.32:0/1475946011 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:12 12.10 12:0bd6d154:::602.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123) -- 0x5563ef370960 con 0x5563ef23f6a0 2024-08-03T21:00:17.235 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.233+0000 7f627d600640 1 -- 172.21.15.32:0/1475946011 <== osd.0 v2:172.21.15.32:6816/3066156116 12 ==== osd_op_reply(12 602.00000000 [omap-set-header] v123'1170 uv1170 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f627800b040 con 0x5563ef23f6a0 2024-08-03T21:00:17.235 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.233+0000 7f62839e0f40 1 -- 172.21.15.32:0/1475946011 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:13 12.10 12:0cf6ad0f:::10000000000.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123) -- 0x5563ef370960 con 0x5563ef23f6a0 2024-08-03T21:00:17.237 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.235+0000 7f627d600640 1 -- 172.21.15.32:0/1475946011 <== osd.0 v2:172.21.15.32:6816/3066156116 13 ==== osd_op_reply(13 10000000000.00000000 [create] v123'1171 uv1146 ondisk = -17 ((17) File exists)) ==== 164+0+0 (crc 0 0 0) 0x7f627800b040 con 0x5563ef23f6a0 2024-08-03T21:00:17.237 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.235+0000 7f62839e0f40 1 -- 172.21.15.32:0/1475946011 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:14 12.10 12:0cf6ad0f:::10000000000.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e123) -- 0x5563ef370960 con 0x5563ef23f6a0 2024-08-03T21:00:17.238 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.237+0000 7f627d600640 1 -- 172.21.15.32:0/1475946011 <== osd.0 v2:172.21.15.32:6816/3066156116 14 ==== osd_op_reply(14 10000000000.00000000 [delete] v123'1172 uv1172 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f627800b040 con 0x5563ef23f6a0 2024-08-03T21:00:17.239 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.237+0000 7f62839e0f40 1 -- 172.21.15.32:0/1475946011 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:15 12.10 12:0cf6ad0f:::10000000000.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123) -- 0x5563ef370960 con 0x5563ef23f6a0 2024-08-03T21:00:17.240 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::10000000000.00000000:head# 2024-08-03T21:00:17.241 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.239+0000 7f627d600640 1 -- 172.21.15.32:0/1475946011 <== osd.0 v2:172.21.15.32:6816/3066156116 15 ==== osd_op_reply(15 10000000000.00000000 [create] v123'1173 uv1173 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f627800b040 con 0x5563ef23f6a0 2024-08-03T21:00:17.241 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.239+0000 7f62839e0f40 1 -- 172.21.15.32:0/1475946011 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:16 12.10 12:0cf6ad0f:::10000000000.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123) -- 0x5563ef370960 con 0x5563ef23f6a0 2024-08-03T21:00:17.242 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.241+0000 7f627d600640 1 -- 172.21.15.32:0/1475946011 <== osd.0 v2:172.21.15.32:6816/3066156116 16 ==== osd_op_reply(16 10000000000.00000000 [setxattr (30)] v123'1174 uv1174 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f627800b040 con 0x5563ef23f6a0 2024-08-03T21:00:17.242 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.241+0000 7f62839e0f40 1 -- 172.21.15.32:0/1475946011 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:17 12.10 12:0cf6ad0f:::10000000000.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123) -- 0x5563ef370960 con 0x5563ef23f6a0 2024-08-03T21:00:17.244 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.243+0000 7f627d600640 1 -- 172.21.15.32:0/1475946011 <== osd.0 v2:172.21.15.32:6816/3066156116 17 ==== osd_op_reply(17 10000000000.00000000 [setxattr (62)] v123'1175 uv1175 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f627800b040 con 0x5563ef23f6a0 2024-08-03T21:00:17.244 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.243+0000 7f62839e0f40 1 -- 172.21.15.32:0/1475946011 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:18 12.10 12:0cf6ad0f:::10000000000.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123) -- 0x5563ef370960 con 0x5563ef23f6a0 2024-08-03T21:00:17.246 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.244+0000 7f627d600640 1 -- 172.21.15.32:0/1475946011 <== osd.0 v2:172.21.15.32:6816/3066156116 18 ==== osd_op_reply(18 10000000000.00000000 [omap-set-header] v123'1176 uv1176 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f627800b040 con 0x5563ef23f6a0 2024-08-03T21:00:17.246 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.245+0000 7f62839e0f40 1 -- 172.21.15.32:0/1475946011 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:19 12.10 12:0cf6ad0f:::10000000000.00000000:head [omap-set-vals in=593b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123) -- 0x5563ef23e5b0 con 0x5563ef23f6a0 2024-08-03T21:00:17.248 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.246+0000 7f627d600640 1 -- 172.21.15.32:0/1475946011 <== osd.0 v2:172.21.15.32:6816/3066156116 19 ==== osd_op_reply(19 10000000000.00000000 [omap-set-vals] v123'1177 uv1177 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f627800b040 con 0x5563ef23f6a0 2024-08-03T21:00:17.248 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.246+0000 7f62839e0f40 1 -- 172.21.15.32:0/1475946011 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:20 12.10 12:0d82a743:::600.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123) -- 0x5563ef23e5b0 con 0x5563ef23f6a0 2024-08-03T21:00:17.250 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.248+0000 7f627d600640 1 -- 172.21.15.32:0/1475946011 <== osd.0 v2:172.21.15.32:6816/3066156116 20 ==== osd_op_reply(20 600.00000000 [create] v123'1178 uv1152 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f627800b040 con 0x5563ef23f6a0 2024-08-03T21:00:17.250 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.248+0000 7f62839e0f40 1 -- 172.21.15.32:0/1475946011 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:21 12.10 12:0d82a743:::600.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e123) -- 0x5563ef23e5b0 con 0x5563ef23f6a0 2024-08-03T21:00:17.254 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.250+0000 7f627d600640 1 -- 172.21.15.32:0/1475946011 <== osd.0 v2:172.21.15.32:6816/3066156116 21 ==== osd_op_reply(21 600.00000000 [delete] v123'1179 uv1179 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f627800b040 con 0x5563ef23f6a0 2024-08-03T21:00:17.255 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.250+0000 7f62839e0f40 1 -- 172.21.15.32:0/1475946011 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:22 12.10 12:0d82a743:::600.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123) -- 0x5563ef23e5b0 con 0x5563ef23f6a0 2024-08-03T21:00:17.255 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::600.00000000:head# 2024-08-03T21:00:17.256 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.252+0000 7f627d600640 1 -- 172.21.15.32:0/1475946011 <== osd.0 v2:172.21.15.32:6816/3066156116 22 ==== osd_op_reply(22 600.00000000 [create] v123'1180 uv1180 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f627800b040 con 0x5563ef23f6a0 2024-08-03T21:00:17.256 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.252+0000 7f62839e0f40 1 -- 172.21.15.32:0/1475946011 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:23 12.10 12:0d82a743:::600.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123) -- 0x5563ef23e5b0 con 0x5563ef23f6a0 2024-08-03T21:00:17.256 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.254+0000 7f627d600640 1 -- 172.21.15.32:0/1475946011 <== osd.0 v2:172.21.15.32:6816/3066156116 23 ==== osd_op_reply(23 600.00000000 [setxattr (30)] v123'1181 uv1181 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f627800b040 con 0x5563ef23f6a0 2024-08-03T21:00:17.256 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.254+0000 7f62839e0f40 1 -- 172.21.15.32:0/1475946011 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:24 12.10 12:0d82a743:::600.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123) -- 0x5563ef23e5b0 con 0x5563ef23f6a0 2024-08-03T21:00:17.257 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.255+0000 7f627d600640 1 -- 172.21.15.32:0/1475946011 <== osd.0 v2:172.21.15.32:6816/3066156116 24 ==== osd_op_reply(24 600.00000000 [setxattr (62)] v123'1182 uv1182 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f627800b040 con 0x5563ef23f6a0 2024-08-03T21:00:17.257 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.255+0000 7f62839e0f40 1 -- 172.21.15.32:0/1475946011 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:25 12.10 12:0d82a743:::600.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123) -- 0x5563ef23e5b0 con 0x5563ef23f6a0 2024-08-03T21:00:17.259 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.257+0000 7f627d600640 1 -- 172.21.15.32:0/1475946011 <== osd.0 v2:172.21.15.32:6816/3066156116 25 ==== osd_op_reply(25 600.00000000 [omap-set-header] v123'1183 uv1183 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f627800b040 con 0x5563ef23f6a0 2024-08-03T21:00:17.259 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.257+0000 7f62839e0f40 1 -- 172.21.15.32:0/1475946011 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:26 12.10 12:0d89b25e:::603.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123) -- 0x5563ef23e5b0 con 0x5563ef23f6a0 2024-08-03T21:00:17.260 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.259+0000 7f627d600640 1 -- 172.21.15.32:0/1475946011 <== osd.0 v2:172.21.15.32:6816/3066156116 26 ==== osd_op_reply(26 603.00000000 [create] v123'1184 uv1158 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f627800b040 con 0x5563ef23f6a0 2024-08-03T21:00:17.260 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.259+0000 7f62839e0f40 1 -- 172.21.15.32:0/1475946011 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:27 12.10 12:0d89b25e:::603.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e123) -- 0x5563ef23e5b0 con 0x5563ef23f6a0 2024-08-03T21:00:17.262 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.260+0000 7f627d600640 1 -- 172.21.15.32:0/1475946011 <== osd.0 v2:172.21.15.32:6816/3066156116 27 ==== osd_op_reply(27 603.00000000 [delete] v123'1185 uv1185 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f627800b040 con 0x5563ef23f6a0 2024-08-03T21:00:17.262 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.261+0000 7f62839e0f40 1 -- 172.21.15.32:0/1475946011 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:28 12.10 12:0d89b25e:::603.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123) -- 0x5563ef23e5b0 con 0x5563ef23f6a0 2024-08-03T21:00:17.263 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::603.00000000:head# 2024-08-03T21:00:17.264 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.262+0000 7f627d600640 1 -- 172.21.15.32:0/1475946011 <== osd.0 v2:172.21.15.32:6816/3066156116 28 ==== osd_op_reply(28 603.00000000 [create] v123'1186 uv1186 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f627800b040 con 0x5563ef23f6a0 2024-08-03T21:00:17.264 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.262+0000 7f62839e0f40 1 -- 172.21.15.32:0/1475946011 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:29 12.10 12:0d89b25e:::603.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123) -- 0x5563ef23e5b0 con 0x5563ef23f6a0 2024-08-03T21:00:17.265 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.264+0000 7f627d600640 1 -- 172.21.15.32:0/1475946011 <== osd.0 v2:172.21.15.32:6816/3066156116 29 ==== osd_op_reply(29 603.00000000 [setxattr (30)] v123'1187 uv1187 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f627800b040 con 0x5563ef23f6a0 2024-08-03T21:00:17.265 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.264+0000 7f62839e0f40 1 -- 172.21.15.32:0/1475946011 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:30 12.10 12:0d89b25e:::603.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123) -- 0x5563ef23e5b0 con 0x5563ef23f6a0 2024-08-03T21:00:17.267 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.265+0000 7f627d600640 1 -- 172.21.15.32:0/1475946011 <== osd.0 v2:172.21.15.32:6816/3066156116 30 ==== osd_op_reply(30 603.00000000 [setxattr (62)] v123'1188 uv1188 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f627800b040 con 0x5563ef23f6a0 2024-08-03T21:00:17.267 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.265+0000 7f62839e0f40 1 -- 172.21.15.32:0/1475946011 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:31 12.10 12:0d89b25e:::603.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123) -- 0x5563ef23e5b0 con 0x5563ef23f6a0 2024-08-03T21:00:17.269 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.267+0000 7f627d600640 1 -- 172.21.15.32:0/1475946011 <== osd.0 v2:172.21.15.32:6816/3066156116 31 ==== osd_op_reply(31 603.00000000 [omap-set-header] v123'1189 uv1189 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f627800b040 con 0x5563ef23f6a0 2024-08-03T21:00:17.269 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.267+0000 7f62839e0f40 1 --2- 172.21.15.32:0/1475946011 >> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] conn(0x5563ef371e30 0x5563ef374250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:00:17.269 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.267+0000 7f62839e0f40 1 -- 172.21.15.32:0/1475946011 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:32 12.14 12:287034c4:::1.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123) -- 0x5563ef23e5b0 con 0x5563ef371e30 2024-08-03T21:00:17.269 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.268+0000 7f627cc00640 1 --2- 172.21.15.32:0/1475946011 >> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] conn(0x5563ef371e30 0x5563ef374250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:00:17.270 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.268+0000 7f627cc00640 1 --2- 172.21.15.32:0/1475946011 >> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] conn(0x5563ef371e30 0x5563ef374250 crc :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T21:00:17.272 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.270+0000 7f627cc00640 1 -- 172.21.15.32:0/1475946011 <== osd.5 v2:172.21.15.118:6810/2724891770 1 ==== osd_op_reply(32 1.00000000.inode [create] v123'846 uv828 ondisk = -17 ((17) File exists)) ==== 160+0+0 (crc 0 0 0) 0x7f627005d590 con 0x5563ef371e30 2024-08-03T21:00:17.272 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.270+0000 7f62839e0f40 1 -- 172.21.15.32:0/1475946011 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:33 12.14 12:287034c4:::1.00000000.inode:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e123) -- 0x5563ef374790 con 0x5563ef371e30 2024-08-03T21:00:17.273 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.272+0000 7f627cc00640 1 -- 172.21.15.32:0/1475946011 <== osd.5 v2:172.21.15.118:6810/2724891770 2 ==== osd_op_reply(33 1.00000000.inode [delete] v123'847 uv847 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f627005d590 con 0x5563ef371e30 2024-08-03T21:00:17.273 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.272+0000 7f62839e0f40 1 -- 172.21.15.32:0/1475946011 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:34 12.14 12:287034c4:::1.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123) -- 0x5563ef374b30 con 0x5563ef371e30 2024-08-03T21:00:17.275 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::1.00000000.inode:head# 2024-08-03T21:00:17.275 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.273+0000 7f627cc00640 1 -- 172.21.15.32:0/1475946011 <== osd.5 v2:172.21.15.118:6810/2724891770 3 ==== osd_op_reply(34 1.00000000.inode [create] v123'848 uv848 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f627005d590 con 0x5563ef371e30 2024-08-03T21:00:17.275 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.273+0000 7f62839e0f40 1 -- 172.21.15.32:0/1475946011 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:35 12.14 12:287034c4:::1.00000000.inode:head [write 0~582 in=582b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123) -- 0x5563ef374ed0 con 0x5563ef371e30 2024-08-03T21:00:17.277 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.275+0000 7f627cc00640 1 -- 172.21.15.32:0/1475946011 <== osd.5 v2:172.21.15.118:6810/2724891770 4 ==== osd_op_reply(35 1.00000000.inode [write 0~582] v123'849 uv849 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f627005d590 con 0x5563ef371e30 2024-08-03T21:00:17.277 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.275+0000 7f62839e0f40 1 -- 172.21.15.32:0/1475946011 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:36 12.14 12:287034c4:::1.00000000.inode:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123) -- 0x5563ef375270 con 0x5563ef371e30 2024-08-03T21:00:17.278 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.277+0000 7f627cc00640 1 -- 172.21.15.32:0/1475946011 <== osd.5 v2:172.21.15.118:6810/2724891770 5 ==== osd_op_reply(36 1.00000000.inode [omap-set-header] v123'850 uv850 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f627005d590 con 0x5563ef371e30 2024-08-03T21:00:17.279 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.277+0000 7f62839e0f40 1 -- 172.21.15.32:0/1475946011 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:37 12.14 12:292cf221:::200.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123) -- 0x5563ef375610 con 0x5563ef371e30 2024-08-03T21:00:17.280 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.279+0000 7f627cc00640 1 -- 172.21.15.32:0/1475946011 <== osd.5 v2:172.21.15.118:6810/2724891770 6 ==== osd_op_reply(37 200.00000000 [create] v123'851 uv833 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f627005d590 con 0x5563ef371e30 2024-08-03T21:00:17.280 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.279+0000 7f62839e0f40 1 -- 172.21.15.32:0/1475946011 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:38 12.14 12:292cf221:::200.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e123) -- 0x5563ef3759b0 con 0x5563ef371e30 2024-08-03T21:00:17.282 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.280+0000 7f627cc00640 1 -- 172.21.15.32:0/1475946011 <== osd.5 v2:172.21.15.118:6810/2724891770 7 ==== osd_op_reply(38 200.00000000 [delete] v123'852 uv852 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f627005d590 con 0x5563ef371e30 2024-08-03T21:00:17.282 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.280+0000 7f62839e0f40 1 -- 172.21.15.32:0/1475946011 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:39 12.14 12:292cf221:::200.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123) -- 0x5563ef375d50 con 0x5563ef371e30 2024-08-03T21:00:17.283 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::200.00000000:head# 2024-08-03T21:00:17.284 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.282+0000 7f627cc00640 1 -- 172.21.15.32:0/1475946011 <== osd.5 v2:172.21.15.118:6810/2724891770 8 ==== osd_op_reply(39 200.00000000 [create] v123'853 uv853 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f627005d590 con 0x5563ef371e30 2024-08-03T21:00:17.284 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.282+0000 7f62839e0f40 1 -- 172.21.15.32:0/1475946011 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:40 12.14 12:292cf221:::200.00000000:head [write 0~90 in=90b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123) -- 0x5563ef375d50 con 0x5563ef371e30 2024-08-03T21:00:17.285 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.284+0000 7f627cc00640 1 -- 172.21.15.32:0/1475946011 <== osd.5 v2:172.21.15.118:6810/2724891770 9 ==== osd_op_reply(40 200.00000000 [write 0~90] v123'854 uv854 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f627005d590 con 0x5563ef371e30 2024-08-03T21:00:17.285 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.284+0000 7f62839e0f40 1 -- 172.21.15.32:0/1475946011 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:41 12.14 12:292cf221:::200.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123) -- 0x5563ef375d50 con 0x5563ef371e30 2024-08-03T21:00:17.287 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.285+0000 7f627cc00640 1 -- 172.21.15.32:0/1475946011 <== osd.5 v2:172.21.15.118:6810/2724891770 10 ==== osd_op_reply(41 200.00000000 [omap-set-header] v123'855 uv855 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f627005d590 con 0x5563ef371e30 2024-08-03T21:00:17.287 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.285+0000 7f62839e0f40 1 -- 172.21.15.32:0/1475946011 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:42 12.14 12:2e2fa760:::200.00000001:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123) -- 0x5563ef375d50 con 0x5563ef371e30 2024-08-03T21:00:17.288 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.287+0000 7f627cc00640 1 -- 172.21.15.32:0/1475946011 <== osd.5 v2:172.21.15.118:6810/2724891770 11 ==== osd_op_reply(42 200.00000001 [create] v123'856 uv838 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f627005d590 con 0x5563ef371e30 2024-08-03T21:00:17.289 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.287+0000 7f62839e0f40 1 -- 172.21.15.32:0/1475946011 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:43 12.14 12:2e2fa760:::200.00000001:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e123) -- 0x5563ef375d50 con 0x5563ef371e30 2024-08-03T21:00:17.290 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.289+0000 7f627cc00640 1 -- 172.21.15.32:0/1475946011 <== osd.5 v2:172.21.15.118:6810/2724891770 12 ==== osd_op_reply(43 200.00000001 [delete] v123'857 uv857 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f627005d590 con 0x5563ef371e30 2024-08-03T21:00:17.290 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.289+0000 7f62839e0f40 1 -- 172.21.15.32:0/1475946011 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:44 12.14 12:2e2fa760:::200.00000001:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123) -- 0x5563ef375d50 con 0x5563ef371e30 2024-08-03T21:00:17.292 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::200.00000001:head# 2024-08-03T21:00:17.292 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.290+0000 7f627cc00640 1 -- 172.21.15.32:0/1475946011 <== osd.5 v2:172.21.15.118:6810/2724891770 13 ==== osd_op_reply(44 200.00000001 [create] v123'858 uv858 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f627005d590 con 0x5563ef371e30 2024-08-03T21:00:17.292 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.291+0000 7f62839e0f40 1 -- 172.21.15.32:0/1475946011 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:45 12.14 12:2e2fa760:::200.00000001:head [write 0~18118 in=18118b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123) -- 0x5563ef37a440 con 0x5563ef371e30 2024-08-03T21:00:17.294 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.292+0000 7f627cc00640 1 -- 172.21.15.32:0/1475946011 <== osd.5 v2:172.21.15.118:6810/2724891770 14 ==== osd_op_reply(45 200.00000001 [write 0~18118] v123'859 uv859 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f627005d590 con 0x5563ef371e30 2024-08-03T21:00:17.294 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.292+0000 7f62839e0f40 1 -- 172.21.15.32:0/1475946011 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:46 12.14 12:2e2fa760:::200.00000001:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123) -- 0x5563ef37a440 con 0x5563ef371e30 2024-08-03T21:00:17.296 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.294+0000 7f627cc00640 1 -- 172.21.15.32:0/1475946011 <== osd.5 v2:172.21.15.118:6810/2724891770 15 ==== osd_op_reply(46 200.00000001 [omap-set-header] v123'860 uv860 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f627005d590 con 0x5563ef371e30 2024-08-03T21:00:17.296 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.294+0000 7f62839e0f40 1 -- 172.21.15.32:0/1475946011 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:47 12.14 12:2f55791f:::606.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123) -- 0x5563ef375d50 con 0x5563ef371e30 2024-08-03T21:00:17.298 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.296+0000 7f627cc00640 1 -- 172.21.15.32:0/1475946011 <== osd.5 v2:172.21.15.118:6810/2724891770 16 ==== osd_op_reply(47 606.00000000 [create] v123'861 uv844 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f627005d590 con 0x5563ef371e30 2024-08-03T21:00:17.298 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.296+0000 7f62839e0f40 1 -- 172.21.15.32:0/1475946011 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:48 12.14 12:2f55791f:::606.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e123) -- 0x5563ef375d50 con 0x5563ef371e30 2024-08-03T21:00:17.300 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.298+0000 7f627cc00640 1 -- 172.21.15.32:0/1475946011 <== osd.5 v2:172.21.15.118:6810/2724891770 17 ==== osd_op_reply(48 606.00000000 [delete] v123'862 uv862 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f627005d590 con 0x5563ef371e30 2024-08-03T21:00:17.300 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.298+0000 7f62839e0f40 1 -- 172.21.15.32:0/1475946011 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:49 12.14 12:2f55791f:::606.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123) -- 0x5563ef375d50 con 0x5563ef371e30 2024-08-03T21:00:17.301 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.299+0000 7f627cc00640 1 -- 172.21.15.32:0/1475946011 <== osd.5 v2:172.21.15.118:6810/2724891770 18 ==== osd_op_reply(49 606.00000000 [create] v123'863 uv863 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f627005d590 con 0x5563ef371e30 2024-08-03T21:00:17.301 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::606.00000000:head# 2024-08-03T21:00:17.301 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.299+0000 7f62839e0f40 1 -- 172.21.15.32:0/1475946011 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:50 12.14 12:2f55791f:::606.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123) -- 0x5563ef376d60 con 0x5563ef371e30 2024-08-03T21:00:17.303 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.301+0000 7f627cc00640 1 -- 172.21.15.32:0/1475946011 <== osd.5 v2:172.21.15.118:6810/2724891770 19 ==== osd_op_reply(50 606.00000000 [setxattr (30)] v123'864 uv864 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f627005d590 con 0x5563ef371e30 2024-08-03T21:00:17.303 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.301+0000 7f62839e0f40 1 -- 172.21.15.32:0/1475946011 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:51 12.14 12:2f55791f:::606.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123) -- 0x5563ef376d60 con 0x5563ef371e30 2024-08-03T21:00:17.304 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.303+0000 7f627cc00640 1 -- 172.21.15.32:0/1475946011 <== osd.5 v2:172.21.15.118:6810/2724891770 20 ==== osd_op_reply(51 606.00000000 [setxattr (62)] v123'865 uv865 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f627005d590 con 0x5563ef371e30 2024-08-03T21:00:17.304 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.303+0000 7f62839e0f40 1 -- 172.21.15.32:0/1475946011 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:52 12.14 12:2f55791f:::606.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123) -- 0x5563ef375d50 con 0x5563ef371e30 2024-08-03T21:00:17.306 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.304+0000 7f627cc00640 1 -- 172.21.15.32:0/1475946011 <== osd.5 v2:172.21.15.118:6810/2724891770 21 ==== osd_op_reply(52 606.00000000 [omap-set-header] v123'866 uv866 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f627005d590 con 0x5563ef371e30 2024-08-03T21:00:17.306 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.305+0000 7f62839e0f40 1 -- 172.21.15.32:0/1475946011 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:53 12.1c 12:3ed09add:::607.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123) -- 0x5563ef375d50 con 0x5563ef23f6a0 2024-08-03T21:00:17.308 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.306+0000 7f627d600640 1 -- 172.21.15.32:0/1475946011 <== osd.0 v2:172.21.15.32:6816/3066156116 32 ==== osd_op_reply(53 607.00000000 [create] v123'225 uv224 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f627800b040 con 0x5563ef23f6a0 2024-08-03T21:00:17.308 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.306+0000 7f62839e0f40 1 -- 172.21.15.32:0/1475946011 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:54 12.1c 12:3ed09add:::607.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e123) -- 0x5563ef375d50 con 0x5563ef23f6a0 2024-08-03T21:00:17.309 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.308+0000 7f627d600640 1 -- 172.21.15.32:0/1475946011 <== osd.0 v2:172.21.15.32:6816/3066156116 33 ==== osd_op_reply(54 607.00000000 [delete] v123'226 uv226 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f627800b040 con 0x5563ef23f6a0 2024-08-03T21:00:17.309 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.308+0000 7f62839e0f40 1 -- 172.21.15.32:0/1475946011 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:55 12.1c 12:3ed09add:::607.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123) -- 0x5563ef375d50 con 0x5563ef23f6a0 2024-08-03T21:00:17.310 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.309+0000 7f627d600640 1 -- 172.21.15.32:0/1475946011 <== osd.0 v2:172.21.15.32:6816/3066156116 34 ==== osd_op_reply(55 607.00000000 [create] v123'227 uv227 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f627800b040 con 0x5563ef23f6a0 2024-08-03T21:00:17.311 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::607.00000000:head# 2024-08-03T21:00:17.311 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.309+0000 7f62839e0f40 1 -- 172.21.15.32:0/1475946011 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:56 12.1c 12:3ed09add:::607.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123) -- 0x5563ef375d50 con 0x5563ef23f6a0 2024-08-03T21:00:17.312 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.310+0000 7f627d600640 1 -- 172.21.15.32:0/1475946011 <== osd.0 v2:172.21.15.32:6816/3066156116 35 ==== osd_op_reply(56 607.00000000 [setxattr (30)] v123'228 uv228 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f627800b040 con 0x5563ef23f6a0 2024-08-03T21:00:17.312 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.310+0000 7f62839e0f40 1 -- 172.21.15.32:0/1475946011 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:57 12.1c 12:3ed09add:::607.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123) -- 0x5563ef375d50 con 0x5563ef23f6a0 2024-08-03T21:00:17.313 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.312+0000 7f627d600640 1 -- 172.21.15.32:0/1475946011 <== osd.0 v2:172.21.15.32:6816/3066156116 36 ==== osd_op_reply(57 607.00000000 [setxattr (62)] v123'229 uv229 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f627800b040 con 0x5563ef23f6a0 2024-08-03T21:00:17.313 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.312+0000 7f62839e0f40 1 -- 172.21.15.32:0/1475946011 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:58 12.1c 12:3ed09add:::607.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123) -- 0x5563ef375d50 con 0x5563ef23f6a0 2024-08-03T21:00:17.315 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.313+0000 7f627d600640 1 -- 172.21.15.32:0/1475946011 <== osd.0 v2:172.21.15.32:6816/3066156116 37 ==== osd_op_reply(58 607.00000000 [omap-set-header] v123'230 uv230 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f627800b040 con 0x5563ef23f6a0 2024-08-03T21:00:17.315 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.313+0000 7f62839e0f40 1 --2- 172.21.15.32:0/1475946011 >> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] conn(0x5563ef376660 0x5563ef376a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:00:17.315 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.313+0000 7f62839e0f40 1 -- 172.21.15.32:0/1475946011 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:59 12.6 12:654134d2:::mds0_openfiles.0:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123) -- 0x5563ef379130 con 0x5563ef376660 2024-08-03T21:00:17.315 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.313+0000 7f6277e00640 1 --2- 172.21.15.32:0/1475946011 >> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] conn(0x5563ef376660 0x5563ef376a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:00:17.315 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.314+0000 7f6277e00640 1 --2- 172.21.15.32:0/1475946011 >> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] conn(0x5563ef376660 0x5563ef376a70 crc :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T21:00:17.317 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.315+0000 7f6277e00640 1 -- 172.21.15.32:0/1475946011 <== osd.1 v2:172.21.15.32:6824/1551241759 1 ==== osd_op_reply(59 mds0_openfiles.0 [create] v123'180 uv179 ondisk = -17 ((17) File exists)) ==== 160+0+0 (crc 0 0 0) 0x7f626800a5e0 con 0x5563ef376660 2024-08-03T21:00:17.317 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.315+0000 7f62839e0f40 1 -- 172.21.15.32:0/1475946011 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:60 12.6 12:654134d2:::mds0_openfiles.0:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e123) -- 0x5563ef3794d0 con 0x5563ef376660 2024-08-03T21:00:17.318 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.317+0000 7f6277e00640 1 -- 172.21.15.32:0/1475946011 <== osd.1 v2:172.21.15.32:6824/1551241759 2 ==== osd_op_reply(60 mds0_openfiles.0 [delete] v123'181 uv181 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f626800a5e0 con 0x5563ef376660 2024-08-03T21:00:17.318 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.317+0000 7f62839e0f40 1 -- 172.21.15.32:0/1475946011 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:61 12.6 12:654134d2:::mds0_openfiles.0:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123) -- 0x5563ef379da0 con 0x5563ef376660 2024-08-03T21:00:17.319 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.318+0000 7f6277e00640 1 -- 172.21.15.32:0/1475946011 <== osd.1 v2:172.21.15.32:6824/1551241759 3 ==== osd_op_reply(61 mds0_openfiles.0 [create] v123'182 uv182 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f626800a5e0 con 0x5563ef376660 2024-08-03T21:00:17.320 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds0_openfiles.0:head# 2024-08-03T21:00:17.320 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.318+0000 7f62839e0f40 1 -- 172.21.15.32:0/1475946011 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:62 12.6 12:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123) -- 0x5563ef37a140 con 0x5563ef376660 2024-08-03T21:00:17.320 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.319+0000 7f6277e00640 1 -- 172.21.15.32:0/1475946011 <== osd.1 v2:172.21.15.32:6824/1551241759 4 ==== osd_op_reply(62 mds0_openfiles.0 [omap-set-header] v123'183 uv183 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f626800a5e0 con 0x5563ef376660 2024-08-03T21:00:17.321 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.319+0000 7f62839e0f40 1 -- 172.21.15.32:0/1475946011 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:63 12.e 12:717a0223:::608.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123) -- 0x5563ef37a4e0 con 0x5563ef23f6a0 2024-08-03T21:00:17.322 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.321+0000 7f627d600640 1 -- 172.21.15.32:0/1475946011 <== osd.0 v2:172.21.15.32:6816/3066156116 38 ==== osd_op_reply(63 608.00000000 [create] v123'225 uv224 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f627800b040 con 0x5563ef23f6a0 2024-08-03T21:00:17.322 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.321+0000 7f62839e0f40 1 -- 172.21.15.32:0/1475946011 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:64 12.e 12:717a0223:::608.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e123) -- 0x5563ef37a4e0 con 0x5563ef23f6a0 2024-08-03T21:00:17.324 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.322+0000 7f627d600640 1 -- 172.21.15.32:0/1475946011 <== osd.0 v2:172.21.15.32:6816/3066156116 39 ==== osd_op_reply(64 608.00000000 [delete] v123'226 uv226 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f627800b040 con 0x5563ef23f6a0 2024-08-03T21:00:17.324 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.322+0000 7f62839e0f40 1 -- 172.21.15.32:0/1475946011 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:65 12.e 12:717a0223:::608.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123) -- 0x5563ef37a4e0 con 0x5563ef23f6a0 2024-08-03T21:00:17.325 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::608.00000000:head# 2024-08-03T21:00:17.325 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.324+0000 7f627d600640 1 -- 172.21.15.32:0/1475946011 <== osd.0 v2:172.21.15.32:6816/3066156116 40 ==== osd_op_reply(65 608.00000000 [create] v123'227 uv227 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f627800b040 con 0x5563ef23f6a0 2024-08-03T21:00:17.325 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.324+0000 7f62839e0f40 1 -- 172.21.15.32:0/1475946011 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:66 12.e 12:717a0223:::608.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123) -- 0x5563ef37b4f0 con 0x5563ef23f6a0 2024-08-03T21:00:17.326 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.325+0000 7f627d600640 1 -- 172.21.15.32:0/1475946011 <== osd.0 v2:172.21.15.32:6816/3066156116 41 ==== osd_op_reply(66 608.00000000 [setxattr (30)] v123'228 uv228 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f627800b040 con 0x5563ef23f6a0 2024-08-03T21:00:17.326 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.325+0000 7f62839e0f40 1 -- 172.21.15.32:0/1475946011 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:67 12.e 12:717a0223:::608.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123) -- 0x5563ef37b4f0 con 0x5563ef23f6a0 2024-08-03T21:00:17.328 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.326+0000 7f627d600640 1 -- 172.21.15.32:0/1475946011 <== osd.0 v2:172.21.15.32:6816/3066156116 42 ==== osd_op_reply(67 608.00000000 [setxattr (62)] v123'229 uv229 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f627800b040 con 0x5563ef23f6a0 2024-08-03T21:00:17.328 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.326+0000 7f62839e0f40 1 -- 172.21.15.32:0/1475946011 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:68 12.e 12:717a0223:::608.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123) -- 0x5563ef37a4e0 con 0x5563ef23f6a0 2024-08-03T21:00:17.329 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.328+0000 7f627d600640 1 -- 172.21.15.32:0/1475946011 <== osd.0 v2:172.21.15.32:6816/3066156116 43 ==== osd_op_reply(68 608.00000000 [omap-set-header] v123'230 uv230 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f627800b040 con 0x5563ef23f6a0 2024-08-03T21:00:17.329 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.328+0000 7f62839e0f40 1 --2- 172.21.15.32:0/1475946011 >> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] conn(0x5563ef37ab60 0x5563ef37cf80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:00:17.330 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.328+0000 7f62839e0f40 1 -- 172.21.15.32:0/1475946011 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:69 12.15 12:a93a17c2:::604.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123) -- 0x5563ef37d4c0 con 0x5563ef37ab60 2024-08-03T21:00:17.330 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.328+0000 7f627d600640 1 --2- 172.21.15.32:0/1475946011 >> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] conn(0x5563ef37ab60 0x5563ef37cf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:00:17.330 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.329+0000 7f627d600640 1 --2- 172.21.15.32:0/1475946011 >> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] conn(0x5563ef37ab60 0x5563ef37cf80 crc :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T21:00:17.332 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.331+0000 7f627d600640 1 -- 172.21.15.32:0/1475946011 <== osd.6 v2:172.21.15.118:6825/801947386 1 ==== osd_op_reply(69 604.00000000 [create] v123'415 uv409 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f627800b040 con 0x5563ef37ab60 2024-08-03T21:00:17.332 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.331+0000 7f62839e0f40 1 -- 172.21.15.32:0/1475946011 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:70 12.15 12:a93a17c2:::604.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e123) -- 0x5563ef37d890 con 0x5563ef37ab60 2024-08-03T21:00:17.334 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.332+0000 7f627d600640 1 -- 172.21.15.32:0/1475946011 <== osd.6 v2:172.21.15.118:6825/801947386 2 ==== osd_op_reply(70 604.00000000 [delete] v123'416 uv416 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f627800b040 con 0x5563ef37ab60 2024-08-03T21:00:17.334 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.332+0000 7f62839e0f40 1 -- 172.21.15.32:0/1475946011 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:71 12.15 12:a93a17c2:::604.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123) -- 0x5563ef37d890 con 0x5563ef37ab60 2024-08-03T21:00:17.335 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::604.00000000:head# 2024-08-03T21:00:17.336 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.334+0000 7f627d600640 1 -- 172.21.15.32:0/1475946011 <== osd.6 v2:172.21.15.118:6825/801947386 3 ==== osd_op_reply(71 604.00000000 [create] v123'417 uv417 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f627800b040 con 0x5563ef37ab60 2024-08-03T21:00:17.336 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.334+0000 7f62839e0f40 1 -- 172.21.15.32:0/1475946011 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:72 12.15 12:a93a17c2:::604.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123) -- 0x5563ef37e8a0 con 0x5563ef37ab60 2024-08-03T21:00:17.337 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.335+0000 7f627d600640 1 -- 172.21.15.32:0/1475946011 <== osd.6 v2:172.21.15.118:6825/801947386 4 ==== osd_op_reply(72 604.00000000 [setxattr (30)] v123'418 uv418 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f627800b040 con 0x5563ef37ab60 2024-08-03T21:00:17.337 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.335+0000 7f62839e0f40 1 -- 172.21.15.32:0/1475946011 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:73 12.15 12:a93a17c2:::604.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123) -- 0x5563ef37e8a0 con 0x5563ef37ab60 2024-08-03T21:00:17.339 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.337+0000 7f627d600640 1 -- 172.21.15.32:0/1475946011 <== osd.6 v2:172.21.15.118:6825/801947386 5 ==== osd_op_reply(73 604.00000000 [setxattr (62)] v123'419 uv419 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f627800b040 con 0x5563ef37ab60 2024-08-03T21:00:17.339 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.337+0000 7f62839e0f40 1 -- 172.21.15.32:0/1475946011 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:74 12.15 12:a93a17c2:::604.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123) -- 0x5563ef37d890 con 0x5563ef37ab60 2024-08-03T21:00:17.340 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.338+0000 7f627d600640 1 -- 172.21.15.32:0/1475946011 <== osd.6 v2:172.21.15.118:6825/801947386 6 ==== osd_op_reply(74 604.00000000 [omap-set-header] v123'420 uv420 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f627800b040 con 0x5563ef37ab60 2024-08-03T21:00:17.340 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.338+0000 7f62839e0f40 1 -- 172.21.15.32:0/1475946011 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:75 12.15 12:aa448500:::500.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123) -- 0x5563ef37d890 con 0x5563ef37ab60 2024-08-03T21:00:17.341 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.340+0000 7f627d600640 1 -- 172.21.15.32:0/1475946011 <== osd.6 v2:172.21.15.118:6825/801947386 7 ==== osd_op_reply(75 500.00000000 [create] v123'421 uv414 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f627800b040 con 0x5563ef37ab60 2024-08-03T21:00:17.341 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.340+0000 7f62839e0f40 1 -- 172.21.15.32:0/1475946011 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:76 12.15 12:aa448500:::500.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e123) -- 0x5563ef37d890 con 0x5563ef37ab60 2024-08-03T21:00:17.343 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.341+0000 7f627d600640 1 -- 172.21.15.32:0/1475946011 <== osd.6 v2:172.21.15.118:6825/801947386 8 ==== osd_op_reply(76 500.00000000 [delete] v123'422 uv422 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f627800b040 con 0x5563ef37ab60 2024-08-03T21:00:17.343 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.341+0000 7f62839e0f40 1 -- 172.21.15.32:0/1475946011 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:77 12.15 12:aa448500:::500.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123) -- 0x5563ef37d890 con 0x5563ef37ab60 2024-08-03T21:00:17.345 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::500.00000000:head# 2024-08-03T21:00:17.345 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.343+0000 7f627d600640 1 -- 172.21.15.32:0/1475946011 <== osd.6 v2:172.21.15.118:6825/801947386 9 ==== osd_op_reply(77 500.00000000 [create] v123'423 uv423 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f627800b040 con 0x5563ef37ab60 2024-08-03T21:00:17.345 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.343+0000 7f62839e0f40 1 -- 172.21.15.32:0/1475946011 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:78 12.15 12:aa448500:::500.00000000:head [write 0~90 in=90b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123) -- 0x5563ef37d890 con 0x5563ef37ab60 2024-08-03T21:00:17.346 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.345+0000 7f627d600640 1 -- 172.21.15.32:0/1475946011 <== osd.6 v2:172.21.15.118:6825/801947386 10 ==== osd_op_reply(78 500.00000000 [write 0~90] v123'424 uv424 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f627800b040 con 0x5563ef37ab60 2024-08-03T21:00:17.347 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.345+0000 7f62839e0f40 1 -- 172.21.15.32:0/1475946011 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:79 12.15 12:aa448500:::500.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123) -- 0x5563ef37d890 con 0x5563ef37ab60 2024-08-03T21:00:17.348 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.347+0000 7f627d600640 1 -- 172.21.15.32:0/1475946011 <== osd.6 v2:172.21.15.118:6825/801947386 11 ==== osd_op_reply(79 500.00000000 [omap-set-header] v123'425 uv425 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f627800b040 con 0x5563ef37ab60 2024-08-03T21:00:17.348 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.347+0000 7f62839e0f40 1 --2- 172.21.15.32:0/1475946011 >> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] conn(0x5563ef37df10 0x5563ef380330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:00:17.348 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.347+0000 7f62839e0f40 1 -- 172.21.15.32:0/1475946011 --> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] -- osd_op(unknown.0.0:80 12.d 12:b50e409b:::mds_snaptable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123) -- 0x5563ef380a30 con 0x5563ef37df10 2024-08-03T21:00:17.348 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.347+0000 7f627cc00640 1 --2- 172.21.15.32:0/1475946011 >> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] conn(0x5563ef37df10 0x5563ef380330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:00:17.349 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.348+0000 7f627cc00640 1 --2- 172.21.15.32:0/1475946011 >> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] conn(0x5563ef37df10 0x5563ef380330 crc :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T21:00:17.350 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.349+0000 7f627cc00640 1 -- 172.21.15.32:0/1475946011 <== osd.7 v2:172.21.15.118:6818/653000048 1 ==== osd_op_reply(80 mds_snaptable [create] v123'188 uv187 ondisk = -17 ((17) File exists)) ==== 157+0+0 (crc 0 0 0) 0x7f627005d590 con 0x5563ef37df10 2024-08-03T21:00:17.351 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.349+0000 7f62839e0f40 1 -- 172.21.15.32:0/1475946011 --> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] -- osd_op(unknown.0.0:81 12.d 12:b50e409b:::mds_snaptable:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e123) -- 0x5563ef380e00 con 0x5563ef37df10 2024-08-03T21:00:17.352 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.351+0000 7f627cc00640 1 -- 172.21.15.32:0/1475946011 <== osd.7 v2:172.21.15.118:6818/653000048 2 ==== osd_op_reply(81 mds_snaptable [delete] v123'189 uv189 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f627005d590 con 0x5563ef37df10 2024-08-03T21:00:17.352 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.351+0000 7f62839e0f40 1 -- 172.21.15.32:0/1475946011 --> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] -- osd_op(unknown.0.0:82 12.d 12:b50e409b:::mds_snaptable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123) -- 0x5563ef380e00 con 0x5563ef37df10 2024-08-03T21:00:17.353 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds_snaptable:head# 2024-08-03T21:00:17.354 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.352+0000 7f627cc00640 1 -- 172.21.15.32:0/1475946011 <== osd.7 v2:172.21.15.118:6818/653000048 3 ==== osd_op_reply(82 mds_snaptable [create] v123'190 uv190 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f627005d590 con 0x5563ef37df10 2024-08-03T21:00:17.354 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.352+0000 7f62839e0f40 1 -- 172.21.15.32:0/1475946011 --> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] -- osd_op(unknown.0.0:83 12.d 12:b50e409b:::mds_snaptable:head [write 0~70 in=70b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123) -- 0x5563ef380e00 con 0x5563ef37df10 2024-08-03T21:00:17.355 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.353+0000 7f627cc00640 1 -- 172.21.15.32:0/1475946011 <== osd.7 v2:172.21.15.118:6818/653000048 4 ==== osd_op_reply(83 mds_snaptable [write 0~70] v123'191 uv191 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f627005d590 con 0x5563ef37df10 2024-08-03T21:00:17.355 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.353+0000 7f62839e0f40 1 -- 172.21.15.32:0/1475946011 --> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] -- osd_op(unknown.0.0:84 12.d 12:b50e409b:::mds_snaptable:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123) -- 0x5563ef380e00 con 0x5563ef37df10 2024-08-03T21:00:17.356 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.355+0000 7f627cc00640 1 -- 172.21.15.32:0/1475946011 <== osd.7 v2:172.21.15.118:6818/653000048 5 ==== osd_op_reply(84 mds_snaptable [omap-set-header] v123'192 uv192 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f627005d590 con 0x5563ef37df10 2024-08-03T21:00:17.356 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.355+0000 7f62839e0f40 1 -- 172.21.15.32:0/1475946011 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:85 12.1d 12:b871830b:::605.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123) -- 0x5563ef380e00 con 0x5563ef23f6a0 2024-08-03T21:00:17.358 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.356+0000 7f627d600640 1 -- 172.21.15.32:0/1475946011 <== osd.0 v2:172.21.15.32:6816/3066156116 44 ==== osd_op_reply(85 605.00000000 [create] v123'225 uv224 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f627800b040 con 0x5563ef23f6a0 2024-08-03T21:00:17.358 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.356+0000 7f62839e0f40 1 -- 172.21.15.32:0/1475946011 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:86 12.1d 12:b871830b:::605.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e123) -- 0x5563ef380e00 con 0x5563ef23f6a0 2024-08-03T21:00:17.359 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.358+0000 7f627d600640 1 -- 172.21.15.32:0/1475946011 <== osd.0 v2:172.21.15.32:6816/3066156116 45 ==== osd_op_reply(86 605.00000000 [delete] v123'226 uv226 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f627800b040 con 0x5563ef23f6a0 2024-08-03T21:00:17.359 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.358+0000 7f62839e0f40 1 -- 172.21.15.32:0/1475946011 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:87 12.1d 12:b871830b:::605.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123) -- 0x5563ef380e00 con 0x5563ef23f6a0 2024-08-03T21:00:17.360 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.359+0000 7f627d600640 1 -- 172.21.15.32:0/1475946011 <== osd.0 v2:172.21.15.32:6816/3066156116 46 ==== osd_op_reply(87 605.00000000 [create] v123'227 uv227 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f627800b040 con 0x5563ef23f6a0 2024-08-03T21:00:17.360 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::605.00000000:head# 2024-08-03T21:00:17.361 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.359+0000 7f62839e0f40 1 -- 172.21.15.32:0/1475946011 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:88 12.1d 12:b871830b:::605.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123) -- 0x5563ef380e00 con 0x5563ef23f6a0 2024-08-03T21:00:17.361 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.360+0000 7f627d600640 1 -- 172.21.15.32:0/1475946011 <== osd.0 v2:172.21.15.32:6816/3066156116 47 ==== osd_op_reply(88 605.00000000 [setxattr (30)] v123'228 uv228 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f627800b040 con 0x5563ef23f6a0 2024-08-03T21:00:17.362 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.360+0000 7f62839e0f40 1 -- 172.21.15.32:0/1475946011 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:89 12.1d 12:b871830b:::605.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123) -- 0x5563ef381e10 con 0x5563ef23f6a0 2024-08-03T21:00:17.363 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.362+0000 7f627d600640 1 -- 172.21.15.32:0/1475946011 <== osd.0 v2:172.21.15.32:6816/3066156116 48 ==== osd_op_reply(89 605.00000000 [setxattr (62)] v123'229 uv229 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f627800b040 con 0x5563ef23f6a0 2024-08-03T21:00:17.363 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.362+0000 7f62839e0f40 1 -- 172.21.15.32:0/1475946011 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:90 12.1d 12:b871830b:::605.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123) -- 0x5563ef380e00 con 0x5563ef23f6a0 2024-08-03T21:00:17.365 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.363+0000 7f627d600640 1 -- 172.21.15.32:0/1475946011 <== osd.0 v2:172.21.15.32:6816/3066156116 49 ==== osd_op_reply(90 605.00000000 [omap-set-header] v123'230 uv230 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f627800b040 con 0x5563ef23f6a0 2024-08-03T21:00:17.365 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.363+0000 7f62839e0f40 1 -- 172.21.15.32:0/1475946011 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:91 12.13 12:c91d4a1d:::mds0_inotable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123) -- 0x5563ef380e00 con 0x5563ef376660 2024-08-03T21:00:17.366 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.364+0000 7f6277e00640 1 -- 172.21.15.32:0/1475946011 <== osd.1 v2:172.21.15.32:6824/1551241759 5 ==== osd_op_reply(91 mds0_inotable [create] v123'376 uv369 ondisk = -17 ((17) File exists)) ==== 157+0+0 (crc 0 0 0) 0x7f626800a5e0 con 0x5563ef376660 2024-08-03T21:00:17.366 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.365+0000 7f62839e0f40 1 -- 172.21.15.32:0/1475946011 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:92 12.13 12:c91d4a1d:::mds0_inotable:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e123) -- 0x5563ef3811a0 con 0x5563ef376660 2024-08-03T21:00:17.367 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.366+0000 7f6277e00640 1 -- 172.21.15.32:0/1475946011 <== osd.1 v2:172.21.15.32:6824/1551241759 6 ==== osd_op_reply(92 mds0_inotable [delete] v123'377 uv377 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f626800a5e0 con 0x5563ef376660 2024-08-03T21:00:17.367 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.366+0000 7f62839e0f40 1 -- 172.21.15.32:0/1475946011 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:93 12.13 12:c91d4a1d:::mds0_inotable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123) -- 0x5563ef381590 con 0x5563ef376660 2024-08-03T21:00:17.369 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.367+0000 7f6277e00640 1 -- 172.21.15.32:0/1475946011 <== osd.1 v2:172.21.15.32:6824/1551241759 7 ==== osd_op_reply(93 mds0_inotable [create] v123'378 uv378 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f626800a5e0 con 0x5563ef376660 2024-08-03T21:00:17.369 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds0_inotable:head# 2024-08-03T21:00:17.369 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.367+0000 7f62839e0f40 1 -- 172.21.15.32:0/1475946011 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:94 12.13 12:c91d4a1d:::mds0_inotable:head [write 0~34 in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123) -- 0x5563ef381de0 con 0x5563ef376660 2024-08-03T21:00:17.370 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.369+0000 7f6277e00640 1 -- 172.21.15.32:0/1475946011 <== osd.1 v2:172.21.15.32:6824/1551241759 8 ==== osd_op_reply(94 mds0_inotable [write 0~34] v123'379 uv379 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f626800a5e0 con 0x5563ef376660 2024-08-03T21:00:17.370 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.369+0000 7f62839e0f40 1 -- 172.21.15.32:0/1475946011 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:95 12.13 12:c91d4a1d:::mds0_inotable:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123) -- 0x5563ef381de0 con 0x5563ef376660 2024-08-03T21:00:17.372 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.370+0000 7f6277e00640 1 -- 172.21.15.32:0/1475946011 <== osd.1 v2:172.21.15.32:6824/1551241759 9 ==== osd_op_reply(95 mds0_inotable [omap-set-header] v123'380 uv380 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f626800a5e0 con 0x5563ef376660 2024-08-03T21:00:17.372 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.370+0000 7f62839e0f40 1 -- 172.21.15.32:0/1475946011 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:96 12.13 12:cd5a64a3:::100.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123) -- 0x5563ef381de0 con 0x5563ef376660 2024-08-03T21:00:17.373 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.372+0000 7f6277e00640 1 -- 172.21.15.32:0/1475946011 <== osd.1 v2:172.21.15.32:6824/1551241759 10 ==== osd_op_reply(96 100.00000000 [create] v123'381 uv375 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f626800a5e0 con 0x5563ef376660 2024-08-03T21:00:17.373 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.372+0000 7f62839e0f40 1 -- 172.21.15.32:0/1475946011 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:97 12.13 12:cd5a64a3:::100.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e123) -- 0x5563ef381de0 con 0x5563ef376660 2024-08-03T21:00:17.374 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.373+0000 7f6277e00640 1 -- 172.21.15.32:0/1475946011 <== osd.1 v2:172.21.15.32:6824/1551241759 11 ==== osd_op_reply(97 100.00000000 [delete] v123'382 uv382 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f626800a5e0 con 0x5563ef376660 2024-08-03T21:00:17.375 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.373+0000 7f62839e0f40 1 -- 172.21.15.32:0/1475946011 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:98 12.13 12:cd5a64a3:::100.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123) -- 0x5563ef381de0 con 0x5563ef376660 2024-08-03T21:00:17.376 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::100.00000000:head# 2024-08-03T21:00:17.376 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.374+0000 7f6277e00640 1 -- 172.21.15.32:0/1475946011 <== osd.1 v2:172.21.15.32:6824/1551241759 12 ==== osd_op_reply(98 100.00000000 [create] v123'383 uv383 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f626800a5e0 con 0x5563ef376660 2024-08-03T21:00:17.376 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.374+0000 7f62839e0f40 1 -- 172.21.15.32:0/1475946011 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:99 12.13 12:cd5a64a3:::100.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123) -- 0x5563ef381de0 con 0x5563ef376660 2024-08-03T21:00:17.377 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.376+0000 7f6277e00640 1 -- 172.21.15.32:0/1475946011 <== osd.1 v2:172.21.15.32:6824/1551241759 13 ==== osd_op_reply(99 100.00000000 [omap-set-header] v123'384 uv384 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f626800a5e0 con 0x5563ef376660 2024-08-03T21:00:17.377 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.376+0000 7f62839e0f40 1 -- 172.21.15.32:0/1475946011 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:100 12.13 12:cd5a64a3:::100.00000000:head [omap-set-vals in=5194b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123) -- 0x5563ef384c90 con 0x5563ef376660 2024-08-03T21:00:17.379 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.377+0000 7f6277e00640 1 -- 172.21.15.32:0/1475946011 <== osd.1 v2:172.21.15.32:6824/1551241759 14 ==== osd_op_reply(100 100.00000000 [omap-set-vals] v123'385 uv385 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f626800a5e0 con 0x5563ef376660 2024-08-03T21:00:17.379 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.377+0000 7f62839e0f40 1 --2- 172.21.15.32:0/1475946011 >> [v2:172.21.15.32:6800/9088303,v1:172.21.15.32:6801/9088303] conn(0x5563ef382840 0x5563ef384c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:00:17.379 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.377+0000 7f62839e0f40 1 -- 172.21.15.32:0/1475946011 --> [v2:172.21.15.32:6800/9088303,v1:172.21.15.32:6801/9088303] -- osd_op(unknown.0.0:101 12.b 12:d0630e4c:::mds0_sessionmap:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123) -- 0x5563ef3851a0 con 0x5563ef382840 2024-08-03T21:00:17.379 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.377+0000 7f6277e00640 1 --2- 172.21.15.32:0/1475946011 >> [v2:172.21.15.32:6800/9088303,v1:172.21.15.32:6801/9088303] conn(0x5563ef382840 0x5563ef384c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:00:17.379 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.378+0000 7f6277e00640 1 --2- 172.21.15.32:0/1475946011 >> [v2:172.21.15.32:6800/9088303,v1:172.21.15.32:6801/9088303] conn(0x5563ef382840 0x5563ef384c60 crc :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T21:00:17.381 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.379+0000 7f6277e00640 1 -- 172.21.15.32:0/1475946011 <== osd.3 v2:172.21.15.32:6800/9088303 1 ==== osd_op_reply(101 mds0_sessionmap [create] v123'167 uv165 ondisk = -17 ((17) File exists)) ==== 159+0+0 (crc 0 0 0) 0x7f626800a5e0 con 0x5563ef382840 2024-08-03T21:00:17.381 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.379+0000 7f62839e0f40 1 -- 172.21.15.32:0/1475946011 --> [v2:172.21.15.32:6800/9088303,v1:172.21.15.32:6801/9088303] -- osd_op(unknown.0.0:102 12.b 12:d0630e4c:::mds0_sessionmap:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e123) -- 0x5563ef3851a0 con 0x5563ef382840 2024-08-03T21:00:17.382 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.381+0000 7f6277e00640 1 -- 172.21.15.32:0/1475946011 <== osd.3 v2:172.21.15.32:6800/9088303 2 ==== osd_op_reply(102 mds0_sessionmap [delete] v123'168 uv168 ondisk = 0) ==== 159+0+0 (crc 0 0 0) 0x7f626800a5e0 con 0x5563ef382840 2024-08-03T21:00:17.382 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.381+0000 7f62839e0f40 1 -- 172.21.15.32:0/1475946011 --> [v2:172.21.15.32:6800/9088303,v1:172.21.15.32:6801/9088303] -- osd_op(unknown.0.0:103 12.b 12:d0630e4c:::mds0_sessionmap:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123) -- 0x5563ef3851a0 con 0x5563ef382840 2024-08-03T21:00:17.383 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.382+0000 7f6277e00640 1 -- 172.21.15.32:0/1475946011 <== osd.3 v2:172.21.15.32:6800/9088303 3 ==== osd_op_reply(103 mds0_sessionmap [create] v123'169 uv169 ondisk = 0) ==== 159+0+0 (crc 0 0 0) 0x7f626800a5e0 con 0x5563ef382840 2024-08-03T21:00:17.384 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds0_sessionmap:head# 2024-08-03T21:00:17.384 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.382+0000 7f62839e0f40 1 -- 172.21.15.32:0/1475946011 --> [v2:172.21.15.32:6800/9088303,v1:172.21.15.32:6801/9088303] -- osd_op(unknown.0.0:104 12.b 12:d0630e4c:::mds0_sessionmap:head [omap-set-header in=14b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123) -- 0x5563ef3851a0 con 0x5563ef382840 2024-08-03T21:00:17.385 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.383+0000 7f6277e00640 1 -- 172.21.15.32:0/1475946011 <== osd.3 v2:172.21.15.32:6800/9088303 4 ==== osd_op_reply(104 mds0_sessionmap [omap-set-header] v123'170 uv170 ondisk = 0) ==== 159+0+0 (crc 0 0 0) 0x7f626800a5e0 con 0x5563ef382840 2024-08-03T21:00:17.385 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.383+0000 7f62839e0f40 1 -- 172.21.15.32:0/1475946011 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:105 12.7 12:e0b41d06:::609.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123) -- 0x5563ef3851a0 con 0x5563ef376660 2024-08-03T21:00:17.386 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.385+0000 7f6277e00640 1 -- 172.21.15.32:0/1475946011 <== osd.1 v2:172.21.15.32:6824/1551241759 15 ==== osd_op_reply(105 609.00000000 [create] v123'225 uv224 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f626800a5e0 con 0x5563ef376660 2024-08-03T21:00:17.386 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.385+0000 7f62839e0f40 1 -- 172.21.15.32:0/1475946011 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:106 12.7 12:e0b41d06:::609.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e123) -- 0x5563ef3851a0 con 0x5563ef376660 2024-08-03T21:00:17.387 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.386+0000 7f6277e00640 1 -- 172.21.15.32:0/1475946011 <== osd.1 v2:172.21.15.32:6824/1551241759 16 ==== osd_op_reply(106 609.00000000 [delete] v123'226 uv226 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f626800a5e0 con 0x5563ef376660 2024-08-03T21:00:17.388 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.386+0000 7f62839e0f40 1 -- 172.21.15.32:0/1475946011 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:107 12.7 12:e0b41d06:::609.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123) -- 0x5563ef3851a0 con 0x5563ef376660 2024-08-03T21:00:17.389 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::609.00000000:head# 2024-08-03T21:00:17.389 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.388+0000 7f6277e00640 1 -- 172.21.15.32:0/1475946011 <== osd.1 v2:172.21.15.32:6824/1551241759 17 ==== osd_op_reply(107 609.00000000 [create] v123'227 uv227 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f626800a5e0 con 0x5563ef376660 2024-08-03T21:00:17.389 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.388+0000 7f62839e0f40 1 -- 172.21.15.32:0/1475946011 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:108 12.7 12:e0b41d06:::609.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123) -- 0x5563ef3861b0 con 0x5563ef376660 2024-08-03T21:00:17.390 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.389+0000 7f6277e00640 1 -- 172.21.15.32:0/1475946011 <== osd.1 v2:172.21.15.32:6824/1551241759 18 ==== osd_op_reply(108 609.00000000 [setxattr (30)] v123'228 uv228 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f626800a5e0 con 0x5563ef376660 2024-08-03T21:00:17.390 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.389+0000 7f62839e0f40 1 -- 172.21.15.32:0/1475946011 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:109 12.7 12:e0b41d06:::609.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123) -- 0x5563ef3861b0 con 0x5563ef376660 2024-08-03T21:00:17.391 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.390+0000 7f6277e00640 1 -- 172.21.15.32:0/1475946011 <== osd.1 v2:172.21.15.32:6824/1551241759 19 ==== osd_op_reply(109 609.00000000 [setxattr (62)] v123'229 uv229 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f626800a5e0 con 0x5563ef376660 2024-08-03T21:00:17.392 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.390+0000 7f62839e0f40 1 -- 172.21.15.32:0/1475946011 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:110 12.7 12:e0b41d06:::609.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123) -- 0x5563ef3851a0 con 0x5563ef376660 2024-08-03T21:00:17.393 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.391+0000 7f6277e00640 1 -- 172.21.15.32:0/1475946011 <== osd.1 v2:172.21.15.32:6824/1551241759 20 ==== osd_op_reply(110 609.00000000 [omap-set-header] v123'230 uv230 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f626800a5e0 con 0x5563ef376660 2024-08-03T21:00:17.393 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.392+0000 7f62839e0f40 1 -- 172.21.15.32:0/1475946011 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:111 12.f 12:f1f69726:::400.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123) -- 0x5563ef3851a0 con 0x5563ef37ab60 2024-08-03T21:00:17.394 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.393+0000 7f627d600640 1 -- 172.21.15.32:0/1475946011 <== osd.6 v2:172.21.15.118:6825/801947386 12 ==== osd_op_reply(111 400.00000000 [create] v123'191 uv190 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f627800b040 con 0x5563ef37ab60 2024-08-03T21:00:17.394 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.393+0000 7f62839e0f40 1 -- 172.21.15.32:0/1475946011 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:112 12.f 12:f1f69726:::400.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e123) -- 0x5563ef3851a0 con 0x5563ef37ab60 2024-08-03T21:00:17.396 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.394+0000 7f627d600640 1 -- 172.21.15.32:0/1475946011 <== osd.6 v2:172.21.15.118:6825/801947386 13 ==== osd_op_reply(112 400.00000000 [delete] v123'192 uv192 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f627800b040 con 0x5563ef37ab60 2024-08-03T21:00:17.396 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.394+0000 7f62839e0f40 1 -- 172.21.15.32:0/1475946011 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:113 12.f 12:f1f69726:::400.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123) -- 0x5563ef3851a0 con 0x5563ef37ab60 2024-08-03T21:00:17.397 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.396+0000 7f627d600640 1 -- 172.21.15.32:0/1475946011 <== osd.6 v2:172.21.15.118:6825/801947386 14 ==== osd_op_reply(113 400.00000000 [create] v123'193 uv193 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f627800b040 con 0x5563ef37ab60 2024-08-03T21:00:17.397 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::400.00000000:head# 2024-08-03T21:00:17.398 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.396+0000 7f62839e0f40 1 -- 172.21.15.32:0/1475946011 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:114 12.f 12:f1f69726:::400.00000000:head [write 0~22 in=22b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123) -- 0x5563ef3851a0 con 0x5563ef37ab60 2024-08-03T21:00:17.399 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.397+0000 7f627d600640 1 -- 172.21.15.32:0/1475946011 <== osd.6 v2:172.21.15.118:6825/801947386 15 ==== osd_op_reply(114 400.00000000 [write 0~22] v123'194 uv194 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f627800b040 con 0x5563ef37ab60 2024-08-03T21:00:17.399 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.397+0000 7f62839e0f40 1 -- 172.21.15.32:0/1475946011 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:115 12.f 12:f1f69726:::400.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123) -- 0x5563ef3851a0 con 0x5563ef37ab60 2024-08-03T21:00:17.401 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.399+0000 7f627d600640 1 -- 172.21.15.32:0/1475946011 <== osd.6 v2:172.21.15.118:6825/801947386 16 ==== osd_op_reply(115 400.00000000 [omap-set-header] v123'195 uv195 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f627800b040 con 0x5563ef37ab60 2024-08-03T21:00:17.401 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.399+0000 7f62839e0f40 1 -- 172.21.15.32:0/1475946011 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:116 12.1f 12:fe07bba1:::100.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123) -- 0x5563ef3851a0 con 0x5563ef37ab60 2024-08-03T21:00:17.402 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.401+0000 7f627d600640 1 -- 172.21.15.32:0/1475946011 <== osd.6 v2:172.21.15.118:6825/801947386 17 ==== osd_op_reply(116 100.00000000.inode [create] v123'468 uv459 ondisk = -17 ((17) File exists)) ==== 162+0+0 (crc 0 0 0) 0x7f627800b040 con 0x5563ef37ab60 2024-08-03T21:00:17.403 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.401+0000 7f62839e0f40 1 -- 172.21.15.32:0/1475946011 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:117 12.1f 12:fe07bba1:::100.00000000.inode:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e123) -- 0x5563ef3851a0 con 0x5563ef37ab60 2024-08-03T21:00:17.404 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.402+0000 7f627d600640 1 -- 172.21.15.32:0/1475946011 <== osd.6 v2:172.21.15.118:6825/801947386 18 ==== osd_op_reply(117 100.00000000.inode [delete] v123'469 uv469 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7f627800b040 con 0x5563ef37ab60 2024-08-03T21:00:17.404 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.402+0000 7f62839e0f40 1 -- 172.21.15.32:0/1475946011 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:118 12.1f 12:fe07bba1:::100.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123) -- 0x5563ef3851a0 con 0x5563ef37ab60 2024-08-03T21:00:17.405 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.404+0000 7f627d600640 1 -- 172.21.15.32:0/1475946011 <== osd.6 v2:172.21.15.118:6825/801947386 19 ==== osd_op_reply(118 100.00000000.inode [create] v123'470 uv470 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7f627800b040 con 0x5563ef37ab60 2024-08-03T21:00:17.406 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::100.00000000.inode:head# 2024-08-03T21:00:17.406 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.404+0000 7f62839e0f40 1 -- 172.21.15.32:0/1475946011 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:119 12.1f 12:fe07bba1:::100.00000000.inode:head [write 0~582 in=582b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123) -- 0x5563ef3851a0 con 0x5563ef37ab60 2024-08-03T21:00:17.407 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.406+0000 7f627d600640 1 -- 172.21.15.32:0/1475946011 <== osd.6 v2:172.21.15.118:6825/801947386 20 ==== osd_op_reply(119 100.00000000.inode [write 0~582] v123'471 uv471 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7f627800b040 con 0x5563ef37ab60 2024-08-03T21:00:17.407 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.406+0000 7f62839e0f40 1 -- 172.21.15.32:0/1475946011 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:120 12.1f 12:fe07bba1:::100.00000000.inode:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123) -- 0x5563ef3851a0 con 0x5563ef37ab60 2024-08-03T21:00:17.408 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.407+0000 7f627d600640 1 -- 172.21.15.32:0/1475946011 <== osd.6 v2:172.21.15.118:6825/801947386 21 ==== osd_op_reply(120 100.00000000.inode [omap-set-header] v123'472 uv472 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7f627800b040 con 0x5563ef37ab60 2024-08-03T21:00:17.409 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.407+0000 7f62839e0f40 1 -- 172.21.15.32:0/1475946011 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:121 12.1f 12:ff5b34d6:::1.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123) -- 0x5563ef3851a0 con 0x5563ef37ab60 2024-08-03T21:00:17.409 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.408+0000 7f627d600640 1 -- 172.21.15.32:0/1475946011 <== osd.6 v2:172.21.15.118:6825/801947386 22 ==== osd_op_reply(121 1.00000000 [create] v123'473 uv466 ondisk = -17 ((17) File exists)) ==== 154+0+0 (crc 0 0 0) 0x7f627800b040 con 0x5563ef37ab60 2024-08-03T21:00:17.410 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.408+0000 7f62839e0f40 1 -- 172.21.15.32:0/1475946011 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:122 12.1f 12:ff5b34d6:::1.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e123) -- 0x5563ef3851a0 con 0x5563ef37ab60 2024-08-03T21:00:17.411 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.410+0000 7f627d600640 1 -- 172.21.15.32:0/1475946011 <== osd.6 v2:172.21.15.118:6825/801947386 23 ==== osd_op_reply(122 1.00000000 [delete] v123'474 uv474 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7f627800b040 con 0x5563ef37ab60 2024-08-03T21:00:17.411 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.410+0000 7f62839e0f40 1 -- 172.21.15.32:0/1475946011 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:123 12.1f 12:ff5b34d6:::1.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123) -- 0x5563ef3851a0 con 0x5563ef37ab60 2024-08-03T21:00:17.412 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.411+0000 7f627d600640 1 -- 172.21.15.32:0/1475946011 <== osd.6 v2:172.21.15.118:6825/801947386 24 ==== osd_op_reply(123 1.00000000 [create] v123'475 uv475 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7f627800b040 con 0x5563ef37ab60 2024-08-03T21:00:17.413 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::1.00000000:head# 2024-08-03T21:00:17.413 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.411+0000 7f62839e0f40 1 -- 172.21.15.32:0/1475946011 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:124 12.1f 12:ff5b34d6:::1.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123) -- 0x5563ef3861b0 con 0x5563ef37ab60 2024-08-03T21:00:17.414 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.412+0000 7f627d600640 1 -- 172.21.15.32:0/1475946011 <== osd.6 v2:172.21.15.118:6825/801947386 25 ==== osd_op_reply(124 1.00000000 [setxattr (30)] v123'476 uv476 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7f627800b040 con 0x5563ef37ab60 2024-08-03T21:00:17.414 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.412+0000 7f62839e0f40 1 -- 172.21.15.32:0/1475946011 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:125 12.1f 12:ff5b34d6:::1.00000000:head [setxattr parent (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123) -- 0x5563ef3861b0 con 0x5563ef37ab60 2024-08-03T21:00:17.415 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.414+0000 7f627d600640 1 -- 172.21.15.32:0/1475946011 <== osd.6 v2:172.21.15.118:6825/801947386 26 ==== osd_op_reply(125 1.00000000 [setxattr (30)] v123'477 uv477 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7f627800b040 con 0x5563ef37ab60 2024-08-03T21:00:17.415 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.414+0000 7f62839e0f40 1 -- 172.21.15.32:0/1475946011 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:126 12.1f 12:ff5b34d6:::1.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123) -- 0x5563ef3851a0 con 0x5563ef37ab60 2024-08-03T21:00:17.416 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.415+0000 7f627d600640 1 -- 172.21.15.32:0/1475946011 <== osd.6 v2:172.21.15.118:6825/801947386 27 ==== osd_op_reply(126 1.00000000 [omap-set-header] v123'478 uv478 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7f627800b040 con 0x5563ef37ab60 2024-08-03T21:00:17.417 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.415+0000 7f62839e0f40 1 -- 172.21.15.32:0/1475946011 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:127 12.1f 12:ff5b34d6:::1.00000000:head [omap-set-vals in=584b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123) -- 0x5563ef385a10 con 0x5563ef37ab60 2024-08-03T21:00:17.419 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.416+0000 7f627d600640 1 -- 172.21.15.32:0/1475946011 <== osd.6 v2:172.21.15.118:6825/801947386 28 ==== osd_op_reply(127 1.00000000 [omap-set-vals] v123'479 uv479 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7f627800b040 con 0x5563ef37ab60 2024-08-03T21:00:17.419 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.417+0000 7f62839e0f40 1 -- 172.21.15.32:0/1475946011 >> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] conn(0x5563ef23f6a0 msgr2=0x5563ef363c50 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:00:17.419 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.417+0000 7f62839e0f40 1 --2- 172.21.15.32:0/1475946011 >> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] conn(0x5563ef23f6a0 0x5563ef363c50 crc :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:17.419 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.417+0000 7f62839e0f40 1 -- 172.21.15.32:0/1475946011 >> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] conn(0x5563ef376660 msgr2=0x5563ef376a70 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:00:17.419 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.417+0000 7f62839e0f40 1 --2- 172.21.15.32:0/1475946011 >> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] conn(0x5563ef376660 0x5563ef376a70 crc :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:17.419 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.417+0000 7f62839e0f40 1 -- 172.21.15.32:0/1475946011 >> [v2:172.21.15.32:6800/9088303,v1:172.21.15.32:6801/9088303] conn(0x5563ef382840 msgr2=0x5563ef384c60 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:00:17.419 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.417+0000 7f62839e0f40 1 --2- 172.21.15.32:0/1475946011 >> [v2:172.21.15.32:6800/9088303,v1:172.21.15.32:6801/9088303] conn(0x5563ef382840 0x5563ef384c60 crc :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:17.420 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.417+0000 7f62839e0f40 1 -- 172.21.15.32:0/1475946011 >> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] conn(0x5563ef371e30 msgr2=0x5563ef374250 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:00:17.420 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.417+0000 7f62839e0f40 1 --2- 172.21.15.32:0/1475946011 >> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] conn(0x5563ef371e30 0x5563ef374250 crc :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:17.420 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.417+0000 7f627d600640 1 -- 172.21.15.32:0/1475946011 reap_dead start 2024-08-03T21:00:17.420 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.417+0000 7f62839e0f40 1 -- 172.21.15.32:0/1475946011 >> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] conn(0x5563ef37ab60 msgr2=0x5563ef37cf80 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:00:17.420 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.417+0000 7f62839e0f40 1 --2- 172.21.15.32:0/1475946011 >> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] conn(0x5563ef37ab60 0x5563ef37cf80 crc :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:17.420 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.417+0000 7f627d600640 1 -- 172.21.15.32:0/1475946011 reap_dead start 2024-08-03T21:00:17.420 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.417+0000 7f62839e0f40 1 -- 172.21.15.32:0/1475946011 >> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] conn(0x5563ef37df10 msgr2=0x5563ef380330 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:00:17.420 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.417+0000 7f62839e0f40 1 --2- 172.21.15.32:0/1475946011 >> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] conn(0x5563ef37df10 0x5563ef380330 crc :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:17.420 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.417+0000 7f62839e0f40 1 -- 172.21.15.32:0/1475946011 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f625c0759f0 msgr2=0x7f625c077f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:00:17.420 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.417+0000 7f62839e0f40 1 --2- 172.21.15.32:0/1475946011 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f625c0759f0 0x7f625c077f00 secure :-1 s=READY pgs=819 cs=0 l=1 rev1=1 crypto rx=0x5563ef2aeaa0 tx=0x7f6268004930 comp rx=0 tx=0).stop 2024-08-03T21:00:17.420 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.417+0000 7f62839e0f40 1 -- 172.21.15.32:0/1475946011 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5563ef2ad360 msgr2=0x5563ef2399d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:00:17.420 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.417+0000 7f62839e0f40 1 --2- 172.21.15.32:0/1475946011 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5563ef2ad360 0x5563ef2399d0 secure :-1 s=READY pgs=2051 cs=0 l=1 rev1=1 crypto rx=0x7f627000bab0 tx=0x7f627000fdd0 comp rx=0 tx=0).stop 2024-08-03T21:00:17.420 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.417+0000 7f62839e0f40 1 -- 172.21.15.32:0/1475946011 shutdown_connections 2024-08-03T21:00:17.420 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.417+0000 7f62839e0f40 1 --2- 172.21.15.32:0/1475946011 >> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] conn(0x5563ef37df10 0x5563ef380330 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:17.421 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.417+0000 7f62839e0f40 1 --2- 172.21.15.32:0/1475946011 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f625c0759f0 0x7f625c077f00 unknown :-1 s=CLOSED pgs=819 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:17.421 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.417+0000 7f62839e0f40 1 --2- 172.21.15.32:0/1475946011 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5563ef2ad360 0x5563ef2399d0 unknown :-1 s=CLOSED pgs=2051 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:17.421 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.417+0000 7f62839e0f40 1 -- 172.21.15.32:0/1475946011 >> 172.21.15.32:0/1475946011 conn(0x5563ef116ac0 msgr2=0x5563ef1f4da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T21:00:17.421 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.417+0000 7f62839e0f40 1 -- 172.21.15.32:0/1475946011 shutdown_connections 2024-08-03T21:00:17.421 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.417+0000 7f62839e0f40 1 -- 172.21.15.32:0/1475946011 wait complete. 2024-08-03T21:00:17.430 DEBUG:teuthology.orchestra.run.smithi032:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata setomapheader mds0_sessionmap deadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeef 2024-08-03T21:00:17.615 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.613+0000 7fa400dd7f40 1 Processor -- start 2024-08-03T21:00:17.615 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.613+0000 7fa400dd7f40 1 -- start start 2024-08-03T21:00:17.624 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.615+0000 7fa400dd7f40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x563ca65b1b40 0x563ca65b01f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:00:17.624 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.615+0000 7fa400dd7f40 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x563ca6624be0 0x563ca663ff20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:00:17.624 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.615+0000 7fa400dd7f40 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x563ca663fac0 0x563ca6641690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:00:17.624 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.615+0000 7fa400dd7f40 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x563ca6625510 con 0x563ca663fac0 2024-08-03T21:00:17.624 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.615+0000 7fa400dd7f40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x563ca64bae30 con 0x563ca65b1b40 2024-08-03T21:00:17.625 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.615+0000 7fa400dd7f40 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x563ca6625690 con 0x563ca6624be0 2024-08-03T21:00:17.625 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.615+0000 7fa3faa00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x563ca663fac0 0x563ca6641690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:00:17.625 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.615+0000 7fa3faa00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x563ca663fac0 0x563ca6641690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:44062/0 (socket says 172.21.15.32:44062) 2024-08-03T21:00:17.625 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.615+0000 7fa3faa00640 1 -- 172.21.15.32:0/3826650726 learned_addr learned my addr 172.21.15.32:0/3826650726 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T21:00:17.625 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.615+0000 7fa3f9600640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x563ca6624be0 0x563ca663ff20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:00:17.626 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.615+0000 7fa3fa000640 1 --2- 172.21.15.32:0/3826650726 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x563ca65b1b40 0x563ca65b01f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:00:17.626 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.615+0000 7fa3faa00640 1 -- 172.21.15.32:0/3826650726 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x563ca6624be0 msgr2=0x563ca663ff20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T21:00:17.626 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.615+0000 7fa3faa00640 1 --2- 172.21.15.32:0/3826650726 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x563ca6624be0 0x563ca663ff20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:17.626 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.615+0000 7fa3faa00640 1 -- 172.21.15.32:0/3826650726 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x563ca65b1b40 msgr2=0x563ca65b01f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T21:00:17.626 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.615+0000 7fa3faa00640 1 --2- 172.21.15.32:0/3826650726 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x563ca65b1b40 0x563ca65b01f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:17.627 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.615+0000 7fa3faa00640 1 -- 172.21.15.32:0/3826650726 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563ca6493f10 con 0x563ca663fac0 2024-08-03T21:00:17.627 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.616+0000 7fa3faa00640 1 --2- 172.21.15.32:0/3826650726 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x563ca663fac0 0x563ca6641690 secure :-1 s=READY pgs=2053 cs=0 l=1 rev1=1 crypto rx=0x7fa3f4057040 tx=0x7fa3f4003330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=167b13ddfabeb0e2 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T21:00:17.627 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.616+0000 7fa3f3e00640 1 -- 172.21.15.32:0/3826650726 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3f4064030 con 0x563ca663fac0 2024-08-03T21:00:17.627 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.616+0000 7fa3f3e00640 1 -- 172.21.15.32:0/3826650726 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fa3f4004020 con 0x563ca663fac0 2024-08-03T21:00:17.627 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.616+0000 7fa3f3e00640 1 -- 172.21.15.32:0/3826650726 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3f4004250 con 0x563ca663fac0 2024-08-03T21:00:17.628 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.617+0000 7fa400dd7f40 1 -- 172.21.15.32:0/3826650726 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x563ca663fac0 msgr2=0x563ca6641690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:00:17.628 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.617+0000 7fa400dd7f40 1 --2- 172.21.15.32:0/3826650726 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x563ca663fac0 0x563ca6641690 secure :-1 s=READY pgs=2053 cs=0 l=1 rev1=1 crypto rx=0x7fa3f4057040 tx=0x7fa3f4003330 comp rx=0 tx=0).stop 2024-08-03T21:00:17.628 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.617+0000 7fa400dd7f40 1 -- 172.21.15.32:0/3826650726 shutdown_connections 2024-08-03T21:00:17.628 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.617+0000 7fa400dd7f40 1 --2- 172.21.15.32:0/3826650726 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x563ca663fac0 0x563ca6641690 unknown :-1 s=CLOSED pgs=2053 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:17.628 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.617+0000 7fa400dd7f40 1 --2- 172.21.15.32:0/3826650726 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x563ca6624be0 0x563ca663ff20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:17.629 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.617+0000 7fa400dd7f40 1 --2- 172.21.15.32:0/3826650726 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x563ca65b1b40 0x563ca65b01f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:17.629 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.617+0000 7fa400dd7f40 1 -- 172.21.15.32:0/3826650726 >> 172.21.15.32:0/3826650726 conn(0x563ca648cac0 msgr2=0x563ca6644c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T21:00:17.629 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.617+0000 7fa400dd7f40 1 -- 172.21.15.32:0/3826650726 shutdown_connections 2024-08-03T21:00:17.629 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.617+0000 7fa400dd7f40 1 -- 172.21.15.32:0/3826650726 wait complete. 2024-08-03T21:00:17.629 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.618+0000 7fa400dd7f40 1 Processor -- start 2024-08-03T21:00:17.629 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.618+0000 7fa400dd7f40 1 -- start start 2024-08-03T21:00:17.630 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.619+0000 7fa400dd7f40 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x563ca65b1b40 0x563ca663f340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:00:17.630 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.619+0000 7fa400dd7f40 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x563ca6624be0 0x563ca6641690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:00:17.630 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.619+0000 7fa400dd7f40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x563ca663fac0 0x563ca6641bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:00:17.630 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.619+0000 7fa400dd7f40 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x563ca661aa80 con 0x563ca6624be0 2024-08-03T21:00:17.630 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.619+0000 7fa400dd7f40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x563ca66409b0 con 0x563ca663fac0 2024-08-03T21:00:17.630 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.619+0000 7fa400dd7f40 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x563ca661b550 con 0x563ca65b1b40 2024-08-03T21:00:17.631 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.620+0000 7fa3f9600640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x563ca6624be0 0x563ca6641690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:00:17.631 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.620+0000 7fa3f9600640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x563ca6624be0 0x563ca6641690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:44074/0 (socket says 172.21.15.32:44074) 2024-08-03T21:00:17.631 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.620+0000 7fa3f9600640 1 -- 172.21.15.32:0/4234520137 learned_addr learned my addr 172.21.15.32:0/4234520137 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T21:00:17.631 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.620+0000 7fa3faa00640 1 --2- 172.21.15.32:0/4234520137 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x563ca663fac0 0x563ca6641bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:00:17.631 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.620+0000 7fa3fa000640 1 --2- 172.21.15.32:0/4234520137 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x563ca65b1b40 0x563ca663f340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:00:17.632 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.620+0000 7fa3f9600640 1 -- 172.21.15.32:0/4234520137 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x563ca65b1b40 msgr2=0x563ca663f340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:00:17.632 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.620+0000 7fa3f9600640 1 --2- 172.21.15.32:0/4234520137 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x563ca65b1b40 0x563ca663f340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:17.632 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.620+0000 7fa3f9600640 1 -- 172.21.15.32:0/4234520137 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x563ca663fac0 msgr2=0x563ca6641bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:00:17.632 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.620+0000 7fa3f9600640 1 --2- 172.21.15.32:0/4234520137 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x563ca663fac0 0x563ca6641bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:17.632 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.620+0000 7fa3f9600640 1 -- 172.21.15.32:0/4234520137 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563ca6619860 con 0x563ca6624be0 2024-08-03T21:00:17.632 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.620+0000 7fa3f9600640 1 --2- 172.21.15.32:0/4234520137 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x563ca6624be0 0x563ca6641690 secure :-1 s=READY pgs=2054 cs=0 l=1 rev1=1 crypto rx=0x7fa3e8008720 tx=0x7fa3e8008810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T21:00:17.633 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.621+0000 7fa3f2a00640 1 -- 172.21.15.32:0/4234520137 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3e800a850 con 0x563ca6624be0 2024-08-03T21:00:17.633 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.621+0000 7fa3f2a00640 1 -- 172.21.15.32:0/4234520137 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fa3e800a9f0 con 0x563ca6624be0 2024-08-03T21:00:17.633 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.621+0000 7fa3f2a00640 1 -- 172.21.15.32:0/4234520137 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3e800ae20 con 0x563ca6624be0 2024-08-03T21:00:17.633 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.621+0000 7fa400dd7f40 1 -- 172.21.15.32:0/4234520137 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x563ca661ccd0 con 0x563ca6624be0 2024-08-03T21:00:17.633 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.621+0000 7fa400dd7f40 1 -- 172.21.15.32:0/4234520137 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563ca661d290 con 0x563ca6624be0 2024-08-03T21:00:17.636 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.623+0000 7fa3f2a00640 1 -- 172.21.15.32:0/4234520137 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7fa3e8016430 con 0x563ca6624be0 2024-08-03T21:00:17.636 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.624+0000 7fa3f2a00640 1 --2- 172.21.15.32:0/4234520137 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fa3dc0759f0 0x7fa3dc077f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:00:17.636 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.624+0000 7fa3f2a00640 1 -- 172.21.15.32:0/4234520137 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(123..123 src has 1..123) ==== 9122+0+0 (secure 0 0 0) 0x7fa3e8098ab0 con 0x563ca6624be0 2024-08-03T21:00:17.636 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.625+0000 7fa3fa000640 1 --2- 172.21.15.32:0/4234520137 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fa3dc0759f0 0x7fa3dc077f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:00:17.636 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.625+0000 7fa400dd7f40 1 --2- 172.21.15.32:0/4234520137 >> [v2:172.21.15.32:6800/9088303,v1:172.21.15.32:6801/9088303] conn(0x563ca66dcc00 0x563ca66dd010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:00:17.636 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.625+0000 7fa400dd7f40 1 -- 172.21.15.32:0/4234520137 --> [v2:172.21.15.32:6800/9088303,v1:172.21.15.32:6801/9088303] -- osd_op(unknown.0.0:1 12.b 12:d0630e4c:::mds0_sessionmap:head [omap-set-header in=80b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123) -- 0x563ca66e5780 con 0x563ca66dcc00 2024-08-03T21:00:17.636 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.625+0000 7fa3faa00640 1 --2- 172.21.15.32:0/4234520137 >> [v2:172.21.15.32:6800/9088303,v1:172.21.15.32:6801/9088303] conn(0x563ca66dcc00 0x563ca66dd010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:00:17.636 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.625+0000 7fa3fa000640 1 --2- 172.21.15.32:0/4234520137 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fa3dc0759f0 0x7fa3dc077f00 secure :-1 s=READY pgs=820 cs=0 l=1 rev1=1 crypto rx=0x7fa3ec00c6a0 tx=0x7fa3ec00e040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T21:00:17.636 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.625+0000 7fa3faa00640 1 --2- 172.21.15.32:0/4234520137 >> [v2:172.21.15.32:6800/9088303,v1:172.21.15.32:6801/9088303] conn(0x563ca66dcc00 0x563ca66dd010 crc :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T21:00:17.637 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.627+0000 7fa3faa00640 1 -- 172.21.15.32:0/4234520137 <== osd.3 v2:172.21.15.32:6800/9088303 1 ==== osd_op_reply(1 mds0_sessionmap [omap-set-header] v123'171 uv171 ondisk = 0) ==== 159+0+0 (crc 0 0 0) 0x7fa3f400b040 con 0x563ca66dcc00 2024-08-03T21:00:17.637 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.627+0000 7fa400dd7f40 1 -- 172.21.15.32:0/4234520137 >> [v2:172.21.15.32:6800/9088303,v1:172.21.15.32:6801/9088303] conn(0x563ca66dcc00 msgr2=0x563ca66dd010 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:00:17.637 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.627+0000 7fa400dd7f40 1 --2- 172.21.15.32:0/4234520137 >> [v2:172.21.15.32:6800/9088303,v1:172.21.15.32:6801/9088303] conn(0x563ca66dcc00 0x563ca66dd010 crc :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:17.637 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.627+0000 7fa400dd7f40 1 -- 172.21.15.32:0/4234520137 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fa3dc0759f0 msgr2=0x7fa3dc077f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:00:17.637 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.627+0000 7fa400dd7f40 1 --2- 172.21.15.32:0/4234520137 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fa3dc0759f0 0x7fa3dc077f00 secure :-1 s=READY pgs=820 cs=0 l=1 rev1=1 crypto rx=0x7fa3ec00c6a0 tx=0x7fa3ec00e040 comp rx=0 tx=0).stop 2024-08-03T21:00:17.637 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.627+0000 7fa400dd7f40 1 -- 172.21.15.32:0/4234520137 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x563ca6624be0 msgr2=0x563ca6641690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:00:17.637 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.627+0000 7fa400dd7f40 1 --2- 172.21.15.32:0/4234520137 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x563ca6624be0 0x563ca6641690 secure :-1 s=READY pgs=2054 cs=0 l=1 rev1=1 crypto rx=0x7fa3e8008720 tx=0x7fa3e8008810 comp rx=0 tx=0).stop 2024-08-03T21:00:17.637 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.627+0000 7fa3faa00640 1 -- 172.21.15.32:0/4234520137 reap_dead start 2024-08-03T21:00:17.637 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.627+0000 7fa400dd7f40 1 -- 172.21.15.32:0/4234520137 shutdown_connections 2024-08-03T21:00:17.637 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.627+0000 7fa400dd7f40 1 -- 172.21.15.32:0/4234520137 >> 172.21.15.32:0/4234520137 conn(0x563ca648cac0 msgr2=0x563ca656c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T21:00:17.637 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.627+0000 7fa400dd7f40 1 -- 172.21.15.32:0/4234520137 shutdown_connections 2024-08-03T21:00:17.637 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.628+0000 7fa400dd7f40 1 -- 172.21.15.32:0/4234520137 wait complete. 2024-08-03T21:00:17.642 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs joinable true 2024-08-03T21:00:17.874 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.871+0000 7ff181c00640 1 -- 172.21.15.32:0/1119825595 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff17c0cccf0 msgr2=0x7ff17c0cd100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:00:17.983 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.871+0000 7ff181c00640 1 --2- 172.21.15.32:0/1119825595 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff17c0cccf0 0x7ff17c0cd100 secure :-1 s=READY pgs=2056 cs=0 l=1 rev1=1 crypto rx=0x7ff17c074a30 tx=0x7ff16802a4a0 comp rx=0 tx=0).stop 2024-08-03T21:00:17.983 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.872+0000 7ff181c00640 1 -- 172.21.15.32:0/1119825595 shutdown_connections 2024-08-03T21:00:17.983 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.872+0000 7ff181c00640 1 --2- 172.21.15.32:0/1119825595 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff17c0cccf0 0x7ff17c0cd100 unknown :-1 s=CLOSED pgs=2056 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:17.984 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.872+0000 7ff181c00640 1 --2- 172.21.15.32:0/1119825595 >> v2:172.21.15.118:3300/0 conn(0x7ff17c145c50 0x7ff17c06e950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:17.984 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.872+0000 7ff181c00640 1 -- 172.21.15.32:0/1119825595 >> 172.21.15.32:0/1119825595 conn(0x7ff17c0b1bd0 msgr2=0x7ff17c0b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T21:00:17.984 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.872+0000 7ff181c00640 1 -- 172.21.15.32:0/1119825595 shutdown_connections 2024-08-03T21:00:17.984 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.872+0000 7ff181c00640 1 -- 172.21.15.32:0/1119825595 wait complete. 2024-08-03T21:00:17.987 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.874+0000 7ff181c00640 1 Processor -- start 2024-08-03T21:00:17.987 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.874+0000 7ff181c00640 1 -- start start 2024-08-03T21:00:17.987 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.875+0000 7ff181c00640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7ff17c0cccf0 0x7ff17c0cff10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:00:17.987 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.875+0000 7ff181c00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ff17c0ced30 0x7ff17c0d0450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:00:17.987 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.875+0000 7ff181c00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff17c145c50 0x7ff17c1f5640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:00:17.987 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.875+0000 7ff181c00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7ff17c0b40b0 con 0x7ff17c145c50 2024-08-03T21:00:17.987 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.875+0000 7ff181c00640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7ff17c0b3f30 con 0x7ff17c0ced30 2024-08-03T21:00:17.987 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.875+0000 7ff181c00640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7ff17c0b4230 con 0x7ff17c0cccf0 2024-08-03T21:00:17.988 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.876+0000 7ff17b400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff17c145c50 0x7ff17c1f5640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:00:17.988 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.876+0000 7ff17b400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff17c145c50 0x7ff17c1f5640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:44100/0 (socket says 172.21.15.32:44100) 2024-08-03T21:00:17.988 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.876+0000 7ff17b400640 1 -- 172.21.15.32:0/2149650785 learned_addr learned my addr 172.21.15.32:0/2149650785 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T21:00:17.988 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.876+0000 7ff17aa00640 1 --2- 172.21.15.32:0/2149650785 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7ff17c0cccf0 0x7ff17c0cff10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:00:17.988 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.876+0000 7ff17a000640 1 --2- 172.21.15.32:0/2149650785 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ff17c0ced30 0x7ff17c0d0450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:00:17.988 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.876+0000 7ff17b400640 1 -- 172.21.15.32:0/2149650785 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7ff17c0cccf0 msgr2=0x7ff17c0cff10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:00:17.988 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.876+0000 7ff17b400640 1 --2- 172.21.15.32:0/2149650785 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7ff17c0cccf0 0x7ff17c0cff10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:17.988 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.876+0000 7ff17b400640 1 -- 172.21.15.32:0/2149650785 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ff17c0ced30 msgr2=0x7ff17c0d0450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:00:17.988 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.876+0000 7ff17b400640 1 --2- 172.21.15.32:0/2149650785 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ff17c0ced30 0x7ff17c0d0450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:17.988 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.876+0000 7ff17b400640 1 -- 172.21.15.32:0/2149650785 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff17c0ca980 con 0x7ff17c145c50 2024-08-03T21:00:17.989 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.877+0000 7ff17b400640 1 --2- 172.21.15.32:0/2149650785 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff17c145c50 0x7ff17c1f5640 secure :-1 s=READY pgs=2057 cs=0 l=1 rev1=1 crypto rx=0x7ff17400c730 tx=0x7ff17400cb30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T21:00:17.989 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.877+0000 7ff172a00640 1 -- 172.21.15.32:0/2149650785 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff174003b70 con 0x7ff17c145c50 2024-08-03T21:00:17.989 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.877+0000 7ff172a00640 1 -- 172.21.15.32:0/2149650785 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7ff174003d10 con 0x7ff17c145c50 2024-08-03T21:00:17.989 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.877+0000 7ff172a00640 1 -- 172.21.15.32:0/2149650785 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1740052d0 con 0x7ff17c145c50 2024-08-03T21:00:17.989 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.877+0000 7ff181c00640 1 -- 172.21.15.32:0/2149650785 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff17c072ef0 con 0x7ff17c145c50 2024-08-03T21:00:17.989 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.877+0000 7ff181c00640 1 -- 172.21.15.32:0/2149650785 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff17c0734b0 con 0x7ff17c145c50 2024-08-03T21:00:17.989 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.878+0000 7ff181c00640 1 -- 172.21.15.32:0/2149650785 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff17c0cab50 con 0x7ff17c145c50 2024-08-03T21:00:17.989 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.880+0000 7ff172a00640 1 -- 172.21.15.32:0/2149650785 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7ff174005470 con 0x7ff17c145c50 2024-08-03T21:00:17.989 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.880+0000 7ff172a00640 1 --2- 172.21.15.32:0/2149650785 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7ff15c077040 0x7ff15c079550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:00:17.989 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.881+0000 7ff172a00640 1 -- 172.21.15.32:0/2149650785 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(123..123 src has 1..123) ==== 9122+0+0 (secure 0 0 0) 0x7ff1740994a0 con 0x7ff17c145c50 2024-08-03T21:00:17.989 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.881+0000 7ff17aa00640 1 --2- 172.21.15.32:0/2149650785 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7ff15c077040 0x7ff15c079550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:00:17.990 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.882+0000 7ff17aa00640 1 --2- 172.21.15.32:0/2149650785 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7ff15c077040 0x7ff15c079550 secure :-1 s=READY pgs=821 cs=0 l=1 rev1=1 crypto rx=0x7ff16c009b30 tx=0x7ff16c005f90 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T21:00:17.990 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:17.885+0000 7ff172a00640 1 -- 172.21.15.32:0/2149650785 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7ff1740611c0 con 0x7ff17c145c50 2024-08-03T21:00:18.132 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:18.131+0000 7ff181c00640 1 -- 172.21.15.32:0/2149650785 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "joinable", "val": "true"} v 0) -- 0x7ff17c072c60 con 0x7ff17c145c50 2024-08-03T21:00:19.038 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:19.037+0000 7ff172a00640 1 -- 172.21.15.32:0/2149650785 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "joinable", "val": "true"}]=0 cephfs marked joinable; MDS may join as newly active. v319) ==== 162+0+0 (secure 0 0 0) 0x7ff174065710 con 0x7ff17c145c50 2024-08-03T21:00:19.039 INFO:teuthology.orchestra.run.smithi032.stderr:cephfs marked joinable; MDS may join as newly active. 2024-08-03T21:00:19.042 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:19.040+0000 7ff181c00640 1 -- 172.21.15.32:0/2149650785 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7ff15c077040 msgr2=0x7ff15c079550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:00:19.042 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:19.040+0000 7ff181c00640 1 --2- 172.21.15.32:0/2149650785 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7ff15c077040 0x7ff15c079550 secure :-1 s=READY pgs=821 cs=0 l=1 rev1=1 crypto rx=0x7ff16c009b30 tx=0x7ff16c005f90 comp rx=0 tx=0).stop 2024-08-03T21:00:19.042 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:19.040+0000 7ff181c00640 1 -- 172.21.15.32:0/2149650785 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff17c145c50 msgr2=0x7ff17c1f5640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:00:19.042 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:19.040+0000 7ff181c00640 1 --2- 172.21.15.32:0/2149650785 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff17c145c50 0x7ff17c1f5640 secure :-1 s=READY pgs=2057 cs=0 l=1 rev1=1 crypto rx=0x7ff17400c730 tx=0x7ff17400cb30 comp rx=0 tx=0).stop 2024-08-03T21:00:19.042 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:19.040+0000 7ff181c00640 1 -- 172.21.15.32:0/2149650785 shutdown_connections 2024-08-03T21:00:19.042 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:19.040+0000 7ff181c00640 1 --2- 172.21.15.32:0/2149650785 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7ff15c077040 0x7ff15c079550 unknown :-1 s=CLOSED pgs=821 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:19.042 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:19.040+0000 7ff181c00640 1 --2- 172.21.15.32:0/2149650785 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ff17c0ced30 0x7ff17c0d0450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:19.042 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:19.040+0000 7ff181c00640 1 --2- 172.21.15.32:0/2149650785 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff17c145c50 0x7ff17c1f5640 unknown :-1 s=CLOSED pgs=2057 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:19.043 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:19.040+0000 7ff181c00640 1 --2- 172.21.15.32:0/2149650785 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7ff17c0cccf0 0x7ff17c0cff10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:19.043 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:19.040+0000 7ff181c00640 1 -- 172.21.15.32:0/2149650785 >> 172.21.15.32:0/2149650785 conn(0x7ff17c0b1bd0 msgr2=0x7ff17c0b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T21:00:19.043 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:19.040+0000 7ff181c00640 1 -- 172.21.15.32:0/2149650785 shutdown_connections 2024-08-03T21:00:19.043 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:19.040+0000 7ff181c00640 1 -- 172.21.15.32:0/2149650785 wait complete. 2024-08-03T21:00:19.050 INFO:tasks.ceph.mds.a.smithi032.stderr:2024-08-03T21:00:19.048+0000 7f23c4e00640 -1 log_channel(cluster) log [ERR] : corrupt sessionmap header: Decoder at 'virtual void SessionMapStore::decode_header(ceph::bufferlist&)' v=1 cannot decode v=100 minimal_decoder=101: Malformed input [buffer:3] 2024-08-03T21:00:19.051 INFO:tasks.ceph.mon.a.smithi032.stderr:2024-08-03T21:00:19.049+0000 7f4e6d600640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-08-03T21:00:19.051 INFO:tasks.ceph.mon.a.smithi032.stderr:2024-08-03T21:00:19.049+0000 7f4e6d600640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 mds daemon damaged (MDS_DAMAGE) 2024-08-03T21:00:19.072 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-08-03T21:00:19.306 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:19.304+0000 7f45c5000640 1 -- 172.21.15.32:0/3538411620 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f45c006e930 msgr2=0x7f45c0144f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:00:19.307 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:19.304+0000 7f45c5000640 1 --2- 172.21.15.32:0/3538411620 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f45c006e930 0x7f45c0144f90 secure :-1 s=READY pgs=2059 cs=0 l=1 rev1=1 crypto rx=0x7f45b0009830 tx=0x7f45b002a5c0 comp rx=0 tx=0).stop 2024-08-03T21:00:19.307 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:19.304+0000 7f45c5000640 1 -- 172.21.15.32:0/3538411620 shutdown_connections 2024-08-03T21:00:19.307 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:19.304+0000 7f45c5000640 1 --2- 172.21.15.32:0/3538411620 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f45c006e930 0x7f45c0144f90 unknown :-1 s=CLOSED pgs=2059 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:19.307 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:19.304+0000 7f45c5000640 1 -- 172.21.15.32:0/3538411620 >> 172.21.15.32:0/3538411620 conn(0x7f45c00b1bd0 msgr2=0x7f45c00b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T21:00:19.307 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:19.304+0000 7f45c5000640 1 -- 172.21.15.32:0/3538411620 shutdown_connections 2024-08-03T21:00:19.307 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:19.304+0000 7f45c5000640 1 -- 172.21.15.32:0/3538411620 wait complete. 2024-08-03T21:00:19.308 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:19.306+0000 7f45c5000640 1 Processor -- start 2024-08-03T21:00:19.308 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:19.307+0000 7f45c5000640 1 -- start start 2024-08-03T21:00:19.311 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:19.308+0000 7f45c5000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f45c006e930 0x7f45c013ed10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:00:19.311 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:19.308+0000 7f45c5000640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f45c0073e60 0x7f45c013f250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:00:19.311 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:19.308+0000 7f45c5000640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f45c015fff0 0x7f45c013c760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:00:19.311 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:19.308+0000 7f45c5000640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f45c00cf840 con 0x7f45c006e930 2024-08-03T21:00:19.311 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:19.308+0000 7f45c5000640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f45c0145f30 con 0x7f45c0073e60 2024-08-03T21:00:19.311 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:19.308+0000 7f45c5000640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7f45c00cf9c0 con 0x7f45c015fff0 2024-08-03T21:00:19.311 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:19.308+0000 7f45bde00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f45c006e930 0x7f45c013ed10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:00:19.311 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:19.308+0000 7f45bde00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f45c006e930 0x7f45c013ed10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:44114/0 (socket says 172.21.15.32:44114) 2024-08-03T21:00:19.311 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:19.308+0000 7f45bde00640 1 -- 172.21.15.32:0/212739022 learned_addr learned my addr 172.21.15.32:0/212739022 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T21:00:19.311 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:19.309+0000 7f45b7e00640 1 --2- 172.21.15.32:0/212739022 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f45c0073e60 0x7f45c013f250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:00:19.311 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:19.309+0000 7f45be800640 1 --2- 172.21.15.32:0/212739022 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f45c015fff0 0x7f45c013c760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:00:19.312 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:19.309+0000 7f45bde00640 1 -- 172.21.15.32:0/212739022 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f45c015fff0 msgr2=0x7f45c013c760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:00:19.312 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:19.309+0000 7f45bde00640 1 --2- 172.21.15.32:0/212739022 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f45c015fff0 0x7f45c013c760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:19.312 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:19.309+0000 7f45bde00640 1 -- 172.21.15.32:0/212739022 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f45c0073e60 msgr2=0x7f45c013f250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:00:19.312 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:19.309+0000 7f45bde00640 1 --2- 172.21.15.32:0/212739022 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f45c0073e60 0x7f45c013f250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:19.312 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:19.309+0000 7f45bde00640 1 -- 172.21.15.32:0/212739022 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f45c015d090 con 0x7f45c006e930 2024-08-03T21:00:19.312 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:19.309+0000 7f45bde00640 1 --2- 172.21.15.32:0/212739022 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f45c006e930 0x7f45c013ed10 secure :-1 s=READY pgs=2060 cs=0 l=1 rev1=1 crypto rx=0x7f45b0009950 tx=0x7f45b0033c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T21:00:19.313 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:19.310+0000 7f45b6000640 1 -- 172.21.15.32:0/212739022 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45b0031b60 con 0x7f45c006e930 2024-08-03T21:00:19.313 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:19.310+0000 7f45b6000640 1 -- 172.21.15.32:0/212739022 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f45b0031df0 con 0x7f45c006e930 2024-08-03T21:00:19.313 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:19.310+0000 7f45b6000640 1 -- 172.21.15.32:0/212739022 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45b0005280 con 0x7f45c006e930 2024-08-03T21:00:19.313 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:19.310+0000 7f45c5000640 1 -- 172.21.15.32:0/212739022 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f45c015d370 con 0x7f45c006e930 2024-08-03T21:00:19.313 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:19.310+0000 7f45c5000640 1 -- 172.21.15.32:0/212739022 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f45c015d930 con 0x7f45c006e930 2024-08-03T21:00:19.313 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:19.311+0000 7f45c5000640 1 -- 172.21.15.32:0/212739022 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f45c0141780 con 0x7f45c006e930 2024-08-03T21:00:19.314 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:19.312+0000 7f45b6000640 1 -- 172.21.15.32:0/212739022 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f45b0005420 con 0x7f45c006e930 2024-08-03T21:00:19.315 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:19.313+0000 7f45b6000640 1 --2- 172.21.15.32:0/212739022 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f459c077010 0x7f459c079520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:00:19.315 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:19.313+0000 7f45b6000640 1 -- 172.21.15.32:0/212739022 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(124..124 src has 1..124) ==== 9208+0+0 (secure 0 0 0) 0x7f45b00ba850 con 0x7f45c006e930 2024-08-03T21:00:19.315 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:19.314+0000 7f45b7e00640 1 --2- 172.21.15.32:0/212739022 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f459c077010 0x7f459c079520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:00:19.316 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:19.314+0000 7f45b7e00640 1 --2- 172.21.15.32:0/212739022 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f459c077010 0x7f459c079520 secure :-1 s=READY pgs=822 cs=0 l=1 rev1=1 crypto rx=0x7f45c0143cf0 tx=0x7f45a8009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T21:00:19.319 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:19.317+0000 7f45b6000640 1 -- 172.21.15.32:0/212739022 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7f45b003f030 con 0x7f45c006e930 2024-08-03T21:00:19.565 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:19.564+0000 7f45c5000640 1 -- 172.21.15.32:0/212739022 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f45c0160e20 con 0x7f45c006e930 2024-08-03T21:00:19.566 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:19.565+0000 7f45b6000640 1 -- 172.21.15.32:0/212739022 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 321 v321) ==== 95+0+4391 (secure 0 0 0) 0x7f45b0031770 con 0x7f45c006e930 2024-08-03T21:00:19.566 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-03T21:00:19.566 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":321,"btime":"2024-08-03T21:00:19:051094+0000","default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6439,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6835/531080160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6834","nonce":531080160},{"type":"v1","addr":"172.21.15.118:6835","nonce":531080160}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":84},{"gid":6454,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6837/2796882319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6836","nonce":2796882319},{"type":"v1","addr":"172.21.15.118:6837","nonce":2796882319}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":86},{"gid":6813,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6834/2686933890","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":2686933890},{"type":"v1","addr":"172.21.15.32:6834","nonce":2686933890}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":85}],"filesystems":[{"mdsmap":{"epoch":321,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-08-03T20:54:26.950518+0000","modified":"2024-08-03T21:00:19.051020+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":124,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":6}]} 2024-08-03T21:00:19.567 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 321 2024-08-03T21:00:19.569 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:19.567+0000 7f45c5000640 1 -- 172.21.15.32:0/212739022 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f459c077010 msgr2=0x7f459c079520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:00:19.569 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:19.567+0000 7f45c5000640 1 --2- 172.21.15.32:0/212739022 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f459c077010 0x7f459c079520 secure :-1 s=READY pgs=822 cs=0 l=1 rev1=1 crypto rx=0x7f45c0143cf0 tx=0x7f45a8009040 comp rx=0 tx=0).stop 2024-08-03T21:00:19.569 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:19.567+0000 7f45c5000640 1 -- 172.21.15.32:0/212739022 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f45c006e930 msgr2=0x7f45c013ed10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:00:19.569 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:19.567+0000 7f45c5000640 1 --2- 172.21.15.32:0/212739022 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f45c006e930 0x7f45c013ed10 secure :-1 s=READY pgs=2060 cs=0 l=1 rev1=1 crypto rx=0x7f45b0009950 tx=0x7f45b0033c10 comp rx=0 tx=0).stop 2024-08-03T21:00:19.569 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:19.567+0000 7f45c5000640 1 -- 172.21.15.32:0/212739022 shutdown_connections 2024-08-03T21:00:19.569 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:19.567+0000 7f45c5000640 1 --2- 172.21.15.32:0/212739022 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f459c077010 0x7f459c079520 unknown :-1 s=CLOSED pgs=822 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:19.569 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:19.567+0000 7f45c5000640 1 --2- 172.21.15.32:0/212739022 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f45c0073e60 0x7f45c013f250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:19.569 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:19.567+0000 7f45c5000640 1 --2- 172.21.15.32:0/212739022 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f45c015fff0 0x7f45c013c760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:19.569 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:19.567+0000 7f45c5000640 1 --2- 172.21.15.32:0/212739022 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f45c006e930 0x7f45c013ed10 unknown :-1 s=CLOSED pgs=2060 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:19.569 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:19.567+0000 7f45c5000640 1 -- 172.21.15.32:0/212739022 >> 172.21.15.32:0/212739022 conn(0x7f45c00b1bd0 msgr2=0x7f45c00b14b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T21:00:19.570 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:19.567+0000 7f45c5000640 1 -- 172.21.15.32:0/212739022 shutdown_connections 2024-08-03T21:00:19.570 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:19.567+0000 7f45c5000640 1 -- 172.21.15.32:0/212739022 wait complete. 2024-08-03T21:00:19.591 DEBUG:tasks.ceph_test_case:wait_until_true: success in 0s and 0 retries 2024-08-03T21:00:19.591 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-08-03T21:00:19.820 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:19.818+0000 7f4f4a200640 1 -- 172.21.15.32:0/3982111458 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4f440b38b0 msgr2=0x7f4f441523f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:00:19.820 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:19.818+0000 7f4f4a200640 1 --2- 172.21.15.32:0/3982111458 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4f440b38b0 0x7f4f441523f0 secure :-1 s=READY pgs=2062 cs=0 l=1 rev1=1 crypto rx=0x7f4f34009830 tx=0x7f4f3402a520 comp rx=0 tx=0).stop 2024-08-03T21:00:19.820 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:19.818+0000 7f4f4a200640 1 -- 172.21.15.32:0/3982111458 shutdown_connections 2024-08-03T21:00:19.821 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:19.818+0000 7f4f4a200640 1 --2- 172.21.15.32:0/3982111458 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4f440b38b0 0x7f4f441523f0 unknown :-1 s=CLOSED pgs=2062 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:19.821 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:19.819+0000 7f4f4a200640 1 -- 172.21.15.32:0/3982111458 >> 172.21.15.32:0/3982111458 conn(0x7f4f440b1bd0 msgr2=0x7f4f440b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T21:00:19.821 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:19.819+0000 7f4f4a200640 1 -- 172.21.15.32:0/3982111458 shutdown_connections 2024-08-03T21:00:19.821 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:19.819+0000 7f4f4a200640 1 -- 172.21.15.32:0/3982111458 wait complete. 2024-08-03T21:00:19.822 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:19.821+0000 7f4f4a200640 1 Processor -- start 2024-08-03T21:00:19.822 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:19.821+0000 7f4f4a200640 1 -- start start 2024-08-03T21:00:19.825 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:19.822+0000 7f4f4a200640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f4f440b38b0 0x7f4f44154d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:00:19.825 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:19.822+0000 7f4f4a200640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f4f440b4ce0 0x7f4f441444f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:00:19.825 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:19.823+0000 7f4f4a200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4f441f8780 0x7f4f441f8b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:00:19.826 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:19.823+0000 7f4f4a200640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f4f440cec50 con 0x7f4f441f8780 2024-08-03T21:00:19.826 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:19.823+0000 7f4f4a200640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f4f440cead0 con 0x7f4f440b4ce0 2024-08-03T21:00:19.826 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:19.823+0000 7f4f4a200640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7f4f440cedd0 con 0x7f4f440b38b0 2024-08-03T21:00:19.826 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:19.823+0000 7f4f43e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4f441f8780 0x7f4f441f8b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:00:19.826 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:19.823+0000 7f4f43e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4f441f8780 0x7f4f441f8b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:38002/0 (socket says 172.21.15.32:38002) 2024-08-03T21:00:19.826 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:19.823+0000 7f4f43e00640 1 -- 172.21.15.32:0/2086940841 learned_addr learned my addr 172.21.15.32:0/2086940841 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T21:00:19.826 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:19.823+0000 7f4f43400640 1 --2- 172.21.15.32:0/2086940841 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f4f440b38b0 0x7f4f44154d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:00:19.826 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:19.823+0000 7f4f3aa00640 1 --2- 172.21.15.32:0/2086940841 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f4f440b4ce0 0x7f4f441444f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:00:19.826 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:19.823+0000 7f4f43e00640 1 -- 172.21.15.32:0/2086940841 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f4f440b38b0 msgr2=0x7f4f44154d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:00:19.826 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:19.823+0000 7f4f43e00640 1 --2- 172.21.15.32:0/2086940841 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f4f440b38b0 0x7f4f44154d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:19.826 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:19.823+0000 7f4f43e00640 1 -- 172.21.15.32:0/2086940841 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f4f440b4ce0 msgr2=0x7f4f441444f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:00:19.826 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:19.823+0000 7f4f43e00640 1 --2- 172.21.15.32:0/2086940841 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f4f440b4ce0 0x7f4f441444f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:19.827 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:19.823+0000 7f4f43e00640 1 -- 172.21.15.32:0/2086940841 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4f44144a30 con 0x7f4f441f8780 2024-08-03T21:00:19.827 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:19.824+0000 7f4f43e00640 1 --2- 172.21.15.32:0/2086940841 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4f441f8780 0x7f4f441f8b90 secure :-1 s=READY pgs=2063 cs=0 l=1 rev1=1 crypto rx=0x7f4f3c00c3c0 tx=0x7f4f3c00c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T21:00:19.827 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:19.824+0000 7f4f3be00640 1 -- 172.21.15.32:0/2086940841 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f3c017070 con 0x7f4f441f8780 2024-08-03T21:00:19.827 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:19.824+0000 7f4f3be00640 1 -- 172.21.15.32:0/2086940841 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f4f3c0053c0 con 0x7f4f441f8780 2024-08-03T21:00:19.827 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:19.824+0000 7f4f3be00640 1 -- 172.21.15.32:0/2086940841 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f3c003850 con 0x7f4f441f8780 2024-08-03T21:00:19.827 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:19.824+0000 7f4f4a200640 1 -- 172.21.15.32:0/2086940841 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4f440727d0 con 0x7f4f441f8780 2024-08-03T21:00:19.827 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:19.824+0000 7f4f4a200640 1 -- 172.21.15.32:0/2086940841 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4f441558f0 con 0x7f4f441f8780 2024-08-03T21:00:19.827 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:19.825+0000 7f4f4a200640 1 -- 172.21.15.32:0/2086940841 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4f441523f0 con 0x7f4f441f8780 2024-08-03T21:00:19.828 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:19.826+0000 7f4f3be00640 1 -- 172.21.15.32:0/2086940841 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f4f3c01a070 con 0x7f4f441f8780 2024-08-03T21:00:19.829 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:19.827+0000 7f4f3be00640 1 --2- 172.21.15.32:0/2086940841 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f4f20077010 0x7f4f20079520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:00:19.829 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:19.827+0000 7f4f3be00640 1 -- 172.21.15.32:0/2086940841 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(124..124 src has 1..124) ==== 9208+0+0 (secure 0 0 0) 0x7f4f3c098b50 con 0x7f4f441f8780 2024-08-03T21:00:19.829 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:19.828+0000 7f4f43400640 1 --2- 172.21.15.32:0/2086940841 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f4f20077010 0x7f4f20079520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:00:19.830 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:19.829+0000 7f4f43400640 1 --2- 172.21.15.32:0/2086940841 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f4f20077010 0x7f4f20079520 secure :-1 s=READY pgs=823 cs=0 l=1 rev1=1 crypto rx=0x7f4f34034000 tx=0x7f4f34004730 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T21:00:19.833 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:19.832+0000 7f4f3be00640 1 -- 172.21.15.32:0/2086940841 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7f4f44144a30 con 0x7f4f441f8780 2024-08-03T21:00:20.083 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:20.081+0000 7f4f4a200640 1 -- 172.21.15.32:0/2086940841 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f4f441553a0 con 0x7f4f441f8780 2024-08-03T21:00:20.083 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:20.082+0000 7f4f3be00640 1 -- 172.21.15.32:0/2086940841 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 321 v321) ==== 95+0+4391 (secure 0 0 0) 0x7f4f3c060f80 con 0x7f4f441f8780 2024-08-03T21:00:20.084 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-03T21:00:20.084 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":321,"btime":"2024-08-03T21:00:19:051094+0000","default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6439,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6835/531080160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6834","nonce":531080160},{"type":"v1","addr":"172.21.15.118:6835","nonce":531080160}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":84},{"gid":6454,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6837/2796882319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6836","nonce":2796882319},{"type":"v1","addr":"172.21.15.118:6837","nonce":2796882319}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":86},{"gid":6813,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6834/2686933890","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":2686933890},{"type":"v1","addr":"172.21.15.32:6834","nonce":2686933890}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":85}],"filesystems":[{"mdsmap":{"epoch":321,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-08-03T20:54:26.950518+0000","modified":"2024-08-03T21:00:19.051020+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":124,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":6}]} 2024-08-03T21:00:20.084 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 321 2024-08-03T21:00:20.086 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:20.084+0000 7f4f4a200640 1 -- 172.21.15.32:0/2086940841 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f4f20077010 msgr2=0x7f4f20079520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:00:20.086 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:20.084+0000 7f4f4a200640 1 --2- 172.21.15.32:0/2086940841 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f4f20077010 0x7f4f20079520 secure :-1 s=READY pgs=823 cs=0 l=1 rev1=1 crypto rx=0x7f4f34034000 tx=0x7f4f34004730 comp rx=0 tx=0).stop 2024-08-03T21:00:20.086 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:20.084+0000 7f4f4a200640 1 -- 172.21.15.32:0/2086940841 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4f441f8780 msgr2=0x7f4f441f8b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:00:20.087 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:20.084+0000 7f4f4a200640 1 --2- 172.21.15.32:0/2086940841 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4f441f8780 0x7f4f441f8b90 secure :-1 s=READY pgs=2063 cs=0 l=1 rev1=1 crypto rx=0x7f4f3c00c3c0 tx=0x7f4f3c00c880 comp rx=0 tx=0).stop 2024-08-03T21:00:20.087 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:20.085+0000 7f4f4a200640 1 -- 172.21.15.32:0/2086940841 shutdown_connections 2024-08-03T21:00:20.087 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:20.085+0000 7f4f4a200640 1 --2- 172.21.15.32:0/2086940841 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f4f20077010 0x7f4f20079520 unknown :-1 s=CLOSED pgs=823 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:20.087 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:20.085+0000 7f4f4a200640 1 --2- 172.21.15.32:0/2086940841 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f4f440b4ce0 0x7f4f441444f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:20.087 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:20.085+0000 7f4f4a200640 1 --2- 172.21.15.32:0/2086940841 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4f441f8780 0x7f4f441f8b90 unknown :-1 s=CLOSED pgs=2063 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:20.087 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:20.085+0000 7f4f4a200640 1 --2- 172.21.15.32:0/2086940841 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f4f440b38b0 0x7f4f44154d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:20.087 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:20.085+0000 7f4f4a200640 1 -- 172.21.15.32:0/2086940841 >> 172.21.15.32:0/2086940841 conn(0x7f4f440b1bd0 msgr2=0x7f4f44159610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T21:00:20.087 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:20.085+0000 7f4f4a200640 1 -- 172.21.15.32:0/2086940841 shutdown_connections 2024-08-03T21:00:20.087 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:20.085+0000 7f4f4a200640 1 -- 172.21.15.32:0/2086940841 wait complete. 2024-08-03T21:00:20.112 DEBUG:tasks.ceph_test_case:wait_until_true: success in 0s and 0 retries 2024-08-03T21:00:20.113 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-08-03T21:00:20.340 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:20.338+0000 7f1bc9800640 1 -- 172.21.15.32:0/1793978426 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1bc406d9f0 msgr2=0x7f1bc406de00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:00:20.340 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:20.338+0000 7f1bc9800640 1 --2- 172.21.15.32:0/1793978426 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1bc406d9f0 0x7f1bc406de00 secure :-1 s=READY pgs=2065 cs=0 l=1 rev1=1 crypto rx=0x7f1bc4072a10 tx=0x7f1bbc02aac0 comp rx=0 tx=0).stop 2024-08-03T21:00:20.340 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:20.338+0000 7f1bc9800640 1 -- 172.21.15.32:0/1793978426 shutdown_connections 2024-08-03T21:00:20.340 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:20.338+0000 7f1bc9800640 1 --2- 172.21.15.32:0/1793978426 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1bc406d9f0 0x7f1bc406de00 unknown :-1 s=CLOSED pgs=2065 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:20.340 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:20.338+0000 7f1bc9800640 1 --2- 172.21.15.32:0/1793978426 >> v2:172.21.15.118:3300/0 conn(0x7f1bc4073e30 0x7f1bc40ce050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:20.340 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:20.338+0000 7f1bc9800640 1 -- 172.21.15.32:0/1793978426 >> 172.21.15.32:0/1793978426 conn(0x7f1bc4072e70 msgr2=0x7f1bc4073280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T21:00:20.341 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:20.338+0000 7f1bc9800640 1 -- 172.21.15.32:0/1793978426 shutdown_connections 2024-08-03T21:00:20.341 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:20.338+0000 7f1bc9800640 1 -- 172.21.15.32:0/1793978426 wait complete. 2024-08-03T21:00:20.342 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:20.340+0000 7f1bc9800640 1 Processor -- start 2024-08-03T21:00:20.342 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:20.341+0000 7f1bc9800640 1 -- start start 2024-08-03T21:00:20.344 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:20.342+0000 7f1bc9800640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f1bc406d9f0 0x7f1bc415e590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:00:20.344 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:20.342+0000 7f1bc9800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1bc4073e30 0x7f1bc4155680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:00:20.344 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:20.342+0000 7f1bc9800640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f1bc415b1b0 0x7f1bc4155bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:00:20.344 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:20.342+0000 7f1bc9800640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f1bc40b4dc0 con 0x7f1bc4073e30 2024-08-03T21:00:20.344 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:20.342+0000 7f1bc9800640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f1bc40b4c40 con 0x7f1bc415b1b0 2024-08-03T21:00:20.345 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:20.342+0000 7f1bc9800640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7f1bc40b4f40 con 0x7f1bc406d9f0 2024-08-03T21:00:20.345 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:20.342+0000 7f1bbb400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1bc4073e30 0x7f1bc4155680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:00:20.345 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:20.342+0000 7f1bbb400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1bc4073e30 0x7f1bc4155680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:38016/0 (socket says 172.21.15.32:38016) 2024-08-03T21:00:20.345 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:20.342+0000 7f1bbb400640 1 -- 172.21.15.32:0/118190287 learned_addr learned my addr 172.21.15.32:0/118190287 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T21:00:20.345 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:20.342+0000 7f1bc3e00640 1 --2- 172.21.15.32:0/118190287 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f1bc406d9f0 0x7f1bc415e590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:00:20.345 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:20.342+0000 7f1bc8e00640 1 --2- 172.21.15.32:0/118190287 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f1bc415b1b0 0x7f1bc4155bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:00:20.345 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:20.343+0000 7f1bbb400640 1 -- 172.21.15.32:0/118190287 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f1bc406d9f0 msgr2=0x7f1bc415e590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:00:20.345 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:20.343+0000 7f1bbb400640 1 --2- 172.21.15.32:0/118190287 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f1bc406d9f0 0x7f1bc415e590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:20.345 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:20.343+0000 7f1bbb400640 1 -- 172.21.15.32:0/118190287 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f1bc415b1b0 msgr2=0x7f1bc4155bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:00:20.345 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:20.343+0000 7f1bbb400640 1 --2- 172.21.15.32:0/118190287 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f1bc415b1b0 0x7f1bc4155bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:20.345 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:20.343+0000 7f1bbb400640 1 -- 172.21.15.32:0/118190287 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1bc415caf0 con 0x7f1bc4073e30 2024-08-03T21:00:20.346 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:20.343+0000 7f1bbb400640 1 --2- 172.21.15.32:0/118190287 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1bc4073e30 0x7f1bc4155680 secure :-1 s=READY pgs=2066 cs=0 l=1 rev1=1 crypto rx=0x7f1bac00c370 tx=0x7f1bac00c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T21:00:20.346 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:20.343+0000 7f1bc0c00640 1 -- 172.21.15.32:0/118190287 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1bac017070 con 0x7f1bc4073e30 2024-08-03T21:00:20.346 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:20.343+0000 7f1bc0c00640 1 -- 172.21.15.32:0/118190287 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f1bac0035f0 con 0x7f1bc4073e30 2024-08-03T21:00:20.347 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:20.343+0000 7f1bc9800640 1 -- 172.21.15.32:0/118190287 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1bc415cdd0 con 0x7f1bc4073e30 2024-08-03T21:00:20.347 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:20.344+0000 7f1bc0c00640 1 -- 172.21.15.32:0/118190287 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1bac003a40 con 0x7f1bc4073e30 2024-08-03T21:00:20.347 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:20.344+0000 7f1bc9800640 1 -- 172.21.15.32:0/118190287 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1bc4154e80 con 0x7f1bc4073e30 2024-08-03T21:00:20.347 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:20.345+0000 7f1bc9800640 1 -- 172.21.15.32:0/118190287 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1bc40b53f0 con 0x7f1bc4073e30 2024-08-03T21:00:20.347 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:20.346+0000 7f1bc0c00640 1 -- 172.21.15.32:0/118190287 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f1bac01a070 con 0x7f1bc4073e30 2024-08-03T21:00:20.348 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:20.347+0000 7f1bc0c00640 1 --2- 172.21.15.32:0/118190287 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f1ba0077040 0x7f1ba0079550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:00:20.348 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:20.347+0000 7f1bc0c00640 1 -- 172.21.15.32:0/118190287 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(124..124 src has 1..124) ==== 9208+0+0 (secure 0 0 0) 0x7f1bac098960 con 0x7f1bc4073e30 2024-08-03T21:00:20.348 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:20.347+0000 7f1bc3e00640 1 --2- 172.21.15.32:0/118190287 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f1ba0077040 0x7f1ba0079550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:00:20.349 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:20.348+0000 7f1bc3e00640 1 --2- 172.21.15.32:0/118190287 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f1ba0077040 0x7f1ba0079550 secure :-1 s=READY pgs=824 cs=0 l=1 rev1=1 crypto rx=0x7f1bb4009b30 tx=0x7f1bb4005f90 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T21:00:20.352 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:20.351+0000 7f1bc0c00640 1 -- 172.21.15.32:0/118190287 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7f1bc406e340 con 0x7f1bc4073e30 2024-08-03T21:00:20.595 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:20.593+0000 7f1bc9800640 1 -- 172.21.15.32:0/118190287 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f1bc415c880 con 0x7f1bc4073e30 2024-08-03T21:00:20.595 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:20.594+0000 7f1bc0c00640 1 -- 172.21.15.32:0/118190287 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 321 v321) ==== 95+0+4391 (secure 0 0 0) 0x7f1bac065e20 con 0x7f1bc4073e30 2024-08-03T21:00:20.596 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-03T21:00:20.596 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":321,"btime":"2024-08-03T21:00:19:051094+0000","default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6439,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6835/531080160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6834","nonce":531080160},{"type":"v1","addr":"172.21.15.118:6835","nonce":531080160}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":84},{"gid":6454,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6837/2796882319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6836","nonce":2796882319},{"type":"v1","addr":"172.21.15.118:6837","nonce":2796882319}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":86},{"gid":6813,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6834/2686933890","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":2686933890},{"type":"v1","addr":"172.21.15.32:6834","nonce":2686933890}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":85}],"filesystems":[{"mdsmap":{"epoch":321,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-08-03T20:54:26.950518+0000","modified":"2024-08-03T21:00:19.051020+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":124,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":6}]} 2024-08-03T21:00:20.596 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 321 2024-08-03T21:00:20.598 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:20.596+0000 7f1bc9800640 1 -- 172.21.15.32:0/118190287 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f1ba0077040 msgr2=0x7f1ba0079550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:00:20.598 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:20.596+0000 7f1bc9800640 1 --2- 172.21.15.32:0/118190287 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f1ba0077040 0x7f1ba0079550 secure :-1 s=READY pgs=824 cs=0 l=1 rev1=1 crypto rx=0x7f1bb4009b30 tx=0x7f1bb4005f90 comp rx=0 tx=0).stop 2024-08-03T21:00:20.598 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:20.596+0000 7f1bc9800640 1 -- 172.21.15.32:0/118190287 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1bc4073e30 msgr2=0x7f1bc4155680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:00:20.598 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:20.596+0000 7f1bc9800640 1 --2- 172.21.15.32:0/118190287 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1bc4073e30 0x7f1bc4155680 secure :-1 s=READY pgs=2066 cs=0 l=1 rev1=1 crypto rx=0x7f1bac00c370 tx=0x7f1bac00c830 comp rx=0 tx=0).stop 2024-08-03T21:00:20.598 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:20.596+0000 7f1bc9800640 1 -- 172.21.15.32:0/118190287 shutdown_connections 2024-08-03T21:00:20.598 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:20.596+0000 7f1bc9800640 1 --2- 172.21.15.32:0/118190287 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f1ba0077040 0x7f1ba0079550 unknown :-1 s=CLOSED pgs=824 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:20.598 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:20.596+0000 7f1bc9800640 1 --2- 172.21.15.32:0/118190287 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f1bc415b1b0 0x7f1bc4155bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:20.599 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:20.596+0000 7f1bc9800640 1 --2- 172.21.15.32:0/118190287 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1bc4073e30 0x7f1bc4155680 unknown :-1 s=CLOSED pgs=2066 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:20.599 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:20.596+0000 7f1bc9800640 1 --2- 172.21.15.32:0/118190287 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f1bc406d9f0 0x7f1bc415e590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:20.599 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:20.596+0000 7f1bc9800640 1 -- 172.21.15.32:0/118190287 >> 172.21.15.32:0/118190287 conn(0x7f1bc4072e70 msgr2=0x7f1bc406eed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T21:00:20.599 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:20.596+0000 7f1bc9800640 1 -- 172.21.15.32:0/118190287 shutdown_connections 2024-08-03T21:00:20.599 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:20.596+0000 7f1bc9800640 1 -- 172.21.15.32:0/118190287 wait complete. 2024-08-03T21:00:20.621 INFO:tasks.cephfs.test_damage:Result: Mutation 'Corrupt omap header on mds0_sessionmap' led to DAMAGED state 2024-08-03T21:00:20.621 INFO:tasks.cephfs.test_damage:Applying mutation 'Corrupt omap header on 1.00000000' 2024-08-03T21:00:20.621 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-08-03T21:00:20.621 DEBUG:teuthology.orchestra.run.smithi032:> dd if=/proc/self/mounts of=/dev/stdout 2024-08-03T21:00:20.657 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053453,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6499208k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 none /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249604k,nr_inodes=812401,mode=700,uid=1000,gid=1316 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6499208k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-08-03T21:00:20.657 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi032.front.sepia.ceph.com 2024-08-03T21:00:20.657 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-08-03T21:00:20.657 DEBUG:teuthology.orchestra.run.smithi032:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-08-03T21:00:20.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-03T21:00:20.734 INFO:teuthology.orchestra.run.smithi032.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.0': No such file or directory 2024-08-03T21:00:20.735 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-08-03T21:00:21.002 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:21.000+0000 7f856b200640 1 -- 172.21.15.32:0/4152975603 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f856c0acd80 msgr2=0x7f856c0ad190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:00:21.002 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:21.000+0000 7f856b200640 1 --2- 172.21.15.32:0/4152975603 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f856c0acd80 0x7f856c0ad190 secure :-1 s=READY pgs=2068 cs=0 l=1 rev1=1 crypto rx=0x7f8554009830 tx=0x7f855402a480 comp rx=0 tx=0).stop 2024-08-03T21:00:21.002 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:21.000+0000 7f856b200640 1 -- 172.21.15.32:0/4152975603 shutdown_connections 2024-08-03T21:00:21.002 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:21.000+0000 7f856b200640 1 --2- 172.21.15.32:0/4152975603 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f856c06f440 0x7f856c06f850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:21.002 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:21.000+0000 7f856b200640 1 --2- 172.21.15.32:0/4152975603 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f856c0acd80 0x7f856c0ad190 unknown :-1 s=CLOSED pgs=2068 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:21.002 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:21.000+0000 7f856b200640 1 -- 172.21.15.32:0/4152975603 >> 172.21.15.32:0/4152975603 conn(0x7f856c0b2120 msgr2=0x7f856c0cfd60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T21:00:21.002 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:21.000+0000 7f856b200640 1 -- 172.21.15.32:0/4152975603 shutdown_connections 2024-08-03T21:00:21.003 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:21.000+0000 7f856b200640 1 -- 172.21.15.32:0/4152975603 wait complete. 2024-08-03T21:00:21.004 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:21.002+0000 7f856b200640 1 Processor -- start 2024-08-03T21:00:21.004 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:21.002+0000 7f856b200640 1 -- start start 2024-08-03T21:00:21.007 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:21.004+0000 7f856b200640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f856c06f440 0x7f856c155350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:00:21.007 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:21.004+0000 7f856b200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f856c0acd80 0x7f856c15b360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:00:21.007 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:21.004+0000 7f856b200640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f856c0af020 0x7f856c15b8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:00:21.007 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:21.004+0000 7f856b200640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f856c1601e0 con 0x7f856c0acd80 2024-08-03T21:00:21.007 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:21.004+0000 7f856b200640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f856c160060 con 0x7f856c06f440 2024-08-03T21:00:21.007 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:21.004+0000 7f856b200640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7f856c160360 con 0x7f856c0af020 2024-08-03T21:00:21.007 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:21.004+0000 7f8563e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f856c0acd80 0x7f856c15b360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:00:21.007 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:21.004+0000 7f8563e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f856c0acd80 0x7f856c15b360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:38040/0 (socket says 172.21.15.32:38040) 2024-08-03T21:00:21.007 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:21.004+0000 7f8563e00640 1 -- 172.21.15.32:0/561070609 learned_addr learned my addr 172.21.15.32:0/561070609 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T21:00:21.007 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:21.004+0000 7f856a800640 1 --2- 172.21.15.32:0/561070609 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f856c0af020 0x7f856c15b8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:00:21.008 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:21.004+0000 7f8569e00640 1 --2- 172.21.15.32:0/561070609 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f856c06f440 0x7f856c155350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:00:21.008 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:21.004+0000 7f8563e00640 1 -- 172.21.15.32:0/561070609 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f856c0af020 msgr2=0x7f856c15b8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:00:21.008 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:21.004+0000 7f8563e00640 1 --2- 172.21.15.32:0/561070609 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f856c0af020 0x7f856c15b8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:21.008 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:21.005+0000 7f8563e00640 1 -- 172.21.15.32:0/561070609 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f856c06f440 msgr2=0x7f856c155350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:00:21.008 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:21.005+0000 7f8563e00640 1 --2- 172.21.15.32:0/561070609 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f856c06f440 0x7f856c155350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:21.008 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:21.005+0000 7f8563e00640 1 -- 172.21.15.32:0/561070609 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f856c155890 con 0x7f856c0acd80 2024-08-03T21:00:21.008 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:21.005+0000 7f8569e00640 1 --2- 172.21.15.32:0/561070609 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f856c06f440 0x7f856c155350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-03T21:00:21.008 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:21.005+0000 7f856a800640 1 --2- 172.21.15.32:0/561070609 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f856c0af020 0x7f856c15b8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-03T21:00:21.008 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:21.005+0000 7f8563e00640 1 --2- 172.21.15.32:0/561070609 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f856c0acd80 0x7f856c15b360 secure :-1 s=READY pgs=2069 cs=0 l=1 rev1=1 crypto rx=0x7f8554009950 tx=0x7f8554033c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T21:00:21.008 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:21.005+0000 7f8562000640 1 -- 172.21.15.32:0/561070609 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8554005c90 con 0x7f856c0acd80 2024-08-03T21:00:21.008 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:21.005+0000 7f8562000640 1 -- 172.21.15.32:0/561070609 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f8554031e50 con 0x7f856c0acd80 2024-08-03T21:00:21.008 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:21.005+0000 7f856b200640 1 -- 172.21.15.32:0/561070609 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f856c0b4ea0 con 0x7f856c0acd80 2024-08-03T21:00:21.008 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:21.005+0000 7f8562000640 1 -- 172.21.15.32:0/561070609 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8554005260 con 0x7f856c0acd80 2024-08-03T21:00:21.009 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:21.005+0000 7f856b200640 1 -- 172.21.15.32:0/561070609 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f856c0b5420 con 0x7f856c0acd80 2024-08-03T21:00:21.009 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:21.006+0000 7f856b200640 1 -- 172.21.15.32:0/561070609 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f856c073030 con 0x7f856c0acd80 2024-08-03T21:00:21.009 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:21.007+0000 7f8562000640 1 -- 172.21.15.32:0/561070609 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f8554005400 con 0x7f856c0acd80 2024-08-03T21:00:21.010 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:21.008+0000 7f8562000640 1 --2- 172.21.15.32:0/561070609 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f8548077010 0x7f8548079520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:00:21.010 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:21.009+0000 7f8562000640 1 -- 172.21.15.32:0/561070609 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(124..124 src has 1..124) ==== 9208+0+0 (secure 0 0 0) 0x7f85540ba8c0 con 0x7f856c0acd80 2024-08-03T21:00:21.010 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:21.009+0000 7f8569e00640 1 --2- 172.21.15.32:0/561070609 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f8548077010 0x7f8548079520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:00:21.011 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:21.010+0000 7f8569e00640 1 --2- 172.21.15.32:0/561070609 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f8548077010 0x7f8548079520 secure :-1 s=READY pgs=825 cs=0 l=1 rev1=1 crypto rx=0x7f855c000f60 tx=0x7f855c009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T21:00:21.014 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:21.013+0000 7f8562000640 1 -- 172.21.15.32:0/561070609 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7f856c073030 con 0x7f856c0acd80 2024-08-03T21:00:21.257 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:21.255+0000 7f856b200640 1 -- 172.21.15.32:0/561070609 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7f856c0ad190 con 0x7f856c0acd80 2024-08-03T21:00:22.048 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:22.047+0000 7f8562000640 1 -- 172.21.15.32:0/561070609 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v322) ==== 161+0+0 (secure 0 0 0) 0x7f856c155890 con 0x7f856c0acd80 2024-08-03T21:00:22.049 INFO:teuthology.orchestra.run.smithi032.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-08-03T21:00:22.052 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:22.050+0000 7f856b200640 1 -- 172.21.15.32:0/561070609 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f8548077010 msgr2=0x7f8548079520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:00:22.052 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:22.050+0000 7f856b200640 1 --2- 172.21.15.32:0/561070609 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f8548077010 0x7f8548079520 secure :-1 s=READY pgs=825 cs=0 l=1 rev1=1 crypto rx=0x7f855c000f60 tx=0x7f855c009040 comp rx=0 tx=0).stop 2024-08-03T21:00:22.052 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:22.050+0000 7f856b200640 1 -- 172.21.15.32:0/561070609 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f856c0acd80 msgr2=0x7f856c15b360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:00:22.052 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:22.050+0000 7f856b200640 1 --2- 172.21.15.32:0/561070609 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f856c0acd80 0x7f856c15b360 secure :-1 s=READY pgs=2069 cs=0 l=1 rev1=1 crypto rx=0x7f8554009950 tx=0x7f8554033c20 comp rx=0 tx=0).stop 2024-08-03T21:00:22.052 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:22.050+0000 7f856b200640 1 -- 172.21.15.32:0/561070609 shutdown_connections 2024-08-03T21:00:22.052 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:22.050+0000 7f856b200640 1 --2- 172.21.15.32:0/561070609 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f8548077010 0x7f8548079520 unknown :-1 s=CLOSED pgs=825 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:22.052 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:22.050+0000 7f856b200640 1 --2- 172.21.15.32:0/561070609 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f856c0af020 0x7f856c15b8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:22.053 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:22.050+0000 7f856b200640 1 --2- 172.21.15.32:0/561070609 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f856c0acd80 0x7f856c15b360 unknown :-1 s=CLOSED pgs=2069 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:22.053 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:22.050+0000 7f856b200640 1 --2- 172.21.15.32:0/561070609 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f856c06f440 0x7f856c155350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:22.053 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:22.050+0000 7f856b200640 1 -- 172.21.15.32:0/561070609 >> 172.21.15.32:0/561070609 conn(0x7f856c0b2120 msgr2=0x7f856c06f850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T21:00:22.053 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:22.050+0000 7f856b200640 1 -- 172.21.15.32:0/561070609 shutdown_connections 2024-08-03T21:00:22.053 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:22.051+0000 7f856b200640 1 -- 172.21.15.32:0/561070609 wait complete. 2024-08-03T21:00:22.078 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mds repaired 0 2024-08-03T21:00:22.311 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:22.309+0000 7fc5f2e00640 1 -- 172.21.15.32:0/3849512158 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fc5ec0cf510 msgr2=0x7fc5ec0cf920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:00:22.311 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:22.309+0000 7fc5f2e00640 1 --2- 172.21.15.32:0/3849512158 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fc5ec0cf510 0x7fc5ec0cf920 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fc5e4009be0 tx=0x7fc5e402a920 comp rx=0 tx=0).stop 2024-08-03T21:00:22.311 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:22.309+0000 7fc5f2e00640 1 -- 172.21.15.32:0/3849512158 shutdown_connections 2024-08-03T21:00:22.311 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:22.309+0000 7fc5f2e00640 1 --2- 172.21.15.32:0/3849512158 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fc5ec0cf510 0x7fc5ec0cf920 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:22.311 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:22.309+0000 7fc5f2e00640 1 -- 172.21.15.32:0/3849512158 >> 172.21.15.32:0/3849512158 conn(0x7fc5ec08d7a0 msgr2=0x7fc5ec09d3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T21:00:22.312 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:22.309+0000 7fc5f2e00640 1 -- 172.21.15.32:0/3849512158 shutdown_connections 2024-08-03T21:00:22.312 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:22.309+0000 7fc5f2e00640 1 -- 172.21.15.32:0/3849512158 wait complete. 2024-08-03T21:00:22.313 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:22.312+0000 7fc5f2e00640 1 Processor -- start 2024-08-03T21:00:22.313 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:22.312+0000 7fc5f2e00640 1 -- start start 2024-08-03T21:00:22.316 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:22.313+0000 7fc5f2e00640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fc5ec0cf510 0x7fc5ec15c350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:00:22.316 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:22.313+0000 7fc5f2e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc5ec0cffe0 0x7fc5ec14f740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:00:22.316 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:22.313+0000 7fc5f2e00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc5ec0d26e0 0x7fc5ec1f5460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:00:22.317 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:22.313+0000 7fc5f2e00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fc5ec0d4310 con 0x7fc5ec0cffe0 2024-08-03T21:00:22.317 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:22.313+0000 7fc5f2e00640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7fc5ec0d4190 con 0x7fc5ec0d26e0 2024-08-03T21:00:22.317 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:22.313+0000 7fc5f2e00640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7fc5ec0d4490 con 0x7fc5ec0cf510 2024-08-03T21:00:22.317 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:22.313+0000 7fc5f1000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc5ec0cffe0 0x7fc5ec14f740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:00:22.317 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:22.314+0000 7fc5f1000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc5ec0cffe0 0x7fc5ec14f740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:38050/0 (socket says 172.21.15.32:38050) 2024-08-03T21:00:22.317 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:22.314+0000 7fc5f1000640 1 -- 172.21.15.32:0/2373415728 learned_addr learned my addr 172.21.15.32:0/2373415728 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T21:00:22.317 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:22.314+0000 7fc5f1a00640 1 --2- 172.21.15.32:0/2373415728 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fc5ec0cf510 0x7fc5ec15c350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:00:22.317 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:22.314+0000 7fc5f2400640 1 --2- 172.21.15.32:0/2373415728 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc5ec0d26e0 0x7fc5ec1f5460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:00:22.317 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:22.314+0000 7fc5f1000640 1 -- 172.21.15.32:0/2373415728 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fc5ec0cf510 msgr2=0x7fc5ec15c350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:00:22.317 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:22.314+0000 7fc5f1000640 1 --2- 172.21.15.32:0/2373415728 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fc5ec0cf510 0x7fc5ec15c350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:22.317 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:22.314+0000 7fc5f1000640 1 -- 172.21.15.32:0/2373415728 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc5ec0d26e0 msgr2=0x7fc5ec1f5460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:00:22.317 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:22.314+0000 7fc5f1000640 1 --2- 172.21.15.32:0/2373415728 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc5ec0d26e0 0x7fc5ec1f5460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:22.317 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:22.314+0000 7fc5f1000640 1 -- 172.21.15.32:0/2373415728 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc5ec15b590 con 0x7fc5ec0cffe0 2024-08-03T21:00:22.318 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:22.314+0000 7fc5f1a00640 1 --2- 172.21.15.32:0/2373415728 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fc5ec0cf510 0x7fc5ec15c350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-03T21:00:22.318 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:22.314+0000 7fc5f1000640 1 --2- 172.21.15.32:0/2373415728 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc5ec0cffe0 0x7fc5ec14f740 secure :-1 s=READY pgs=2071 cs=0 l=1 rev1=1 crypto rx=0x7fc5d800c460 tx=0x7fc5d800c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T21:00:22.318 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:22.315+0000 7fc5e9600640 1 -- 172.21.15.32:0/2373415728 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5d8016020 con 0x7fc5ec0cffe0 2024-08-03T21:00:22.318 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:22.315+0000 7fc5e9600640 1 -- 172.21.15.32:0/2373415728 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fc5d8012940 con 0x7fc5ec0cffe0 2024-08-03T21:00:22.318 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:22.315+0000 7fc5e9600640 1 -- 172.21.15.32:0/2373415728 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5d8012d90 con 0x7fc5ec0cffe0 2024-08-03T21:00:22.318 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:22.315+0000 7fc5f2e00640 1 -- 172.21.15.32:0/2373415728 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc5ec15b870 con 0x7fc5ec0cffe0 2024-08-03T21:00:22.318 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:22.315+0000 7fc5f2e00640 1 -- 172.21.15.32:0/2373415728 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc5ec15be30 con 0x7fc5ec0cffe0 2024-08-03T21:00:22.319 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:22.316+0000 7fc5f2e00640 1 -- 172.21.15.32:0/2373415728 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc5ec1f5fa0 con 0x7fc5ec0cffe0 2024-08-03T21:00:22.319 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:22.317+0000 7fc5e9600640 1 -- 172.21.15.32:0/2373415728 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7fc5d8019020 con 0x7fc5ec0cffe0 2024-08-03T21:00:22.320 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:22.318+0000 7fc5e9600640 1 --2- 172.21.15.32:0/2373415728 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fc5cc077010 0x7fc5cc079520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:00:22.320 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:22.318+0000 7fc5e9600640 1 -- 172.21.15.32:0/2373415728 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(124..124 src has 1..124) ==== 9208+0+0 (secure 0 0 0) 0x7fc5d8099090 con 0x7fc5ec0cffe0 2024-08-03T21:00:22.320 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:22.319+0000 7fc5f1a00640 1 --2- 172.21.15.32:0/2373415728 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fc5cc077010 0x7fc5cc079520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:00:22.321 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:22.320+0000 7fc5f1a00640 1 --2- 172.21.15.32:0/2373415728 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fc5cc077010 0x7fc5cc079520 secure :-1 s=READY pgs=826 cs=0 l=1 rev1=1 crypto rx=0x7fc5e0007fd0 tx=0x7fc5e0009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T21:00:22.324 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:22.323+0000 7fc5e9600640 1 -- 172.21.15.32:0/2373415728 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7fc5d8020070 con 0x7fc5ec0cffe0 2024-08-03T21:00:22.613 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:22.612+0000 7fc5f2e00640 1 -- 172.21.15.32:0/2373415728 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "mds repaired", "role": "0"} v 0) -- 0x7fc5ec0d4940 con 0x7fc5ec0cffe0 2024-08-03T21:00:23.059 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.057+0000 7fc5e9600640 1 -- 172.21.15.32:0/2373415728 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "mds repaired", "role": "0"}]=0 repaired: restoring rank 6:0 v323) ==== 101+0+0 (secure 0 0 0) 0x7fc5d80614c0 con 0x7fc5ec0cffe0 2024-08-03T21:00:23.059 INFO:teuthology.orchestra.run.smithi032.stderr:repaired: restoring rank 6:0 2024-08-03T21:00:23.062 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.060+0000 7fc5f2e00640 1 -- 172.21.15.32:0/2373415728 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fc5cc077010 msgr2=0x7fc5cc079520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:00:23.063 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.061+0000 7fc5f2e00640 1 --2- 172.21.15.32:0/2373415728 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fc5cc077010 0x7fc5cc079520 secure :-1 s=READY pgs=826 cs=0 l=1 rev1=1 crypto rx=0x7fc5e0007fd0 tx=0x7fc5e0009040 comp rx=0 tx=0).stop 2024-08-03T21:00:23.063 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.061+0000 7fc5f2e00640 1 -- 172.21.15.32:0/2373415728 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc5ec0cffe0 msgr2=0x7fc5ec14f740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:00:23.063 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.061+0000 7fc5f2e00640 1 --2- 172.21.15.32:0/2373415728 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc5ec0cffe0 0x7fc5ec14f740 secure :-1 s=READY pgs=2071 cs=0 l=1 rev1=1 crypto rx=0x7fc5d800c460 tx=0x7fc5d800c920 comp rx=0 tx=0).stop 2024-08-03T21:00:23.063 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.061+0000 7fc5f2e00640 1 -- 172.21.15.32:0/2373415728 shutdown_connections 2024-08-03T21:00:23.063 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.061+0000 7fc5f2e00640 1 --2- 172.21.15.32:0/2373415728 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fc5cc077010 0x7fc5cc079520 unknown :-1 s=CLOSED pgs=826 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:23.063 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.061+0000 7fc5f2e00640 1 --2- 172.21.15.32:0/2373415728 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc5ec0d26e0 0x7fc5ec1f5460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:23.063 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.061+0000 7fc5f2e00640 1 --2- 172.21.15.32:0/2373415728 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc5ec0cffe0 0x7fc5ec14f740 unknown :-1 s=CLOSED pgs=2071 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:23.063 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.061+0000 7fc5f2e00640 1 --2- 172.21.15.32:0/2373415728 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fc5ec0cf510 0x7fc5ec15c350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:23.063 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.061+0000 7fc5f2e00640 1 -- 172.21.15.32:0/2373415728 >> 172.21.15.32:0/2373415728 conn(0x7fc5ec08d7a0 msgr2=0x7fc5ec0d1300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T21:00:23.064 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.061+0000 7fc5f2e00640 1 -- 172.21.15.32:0/2373415728 shutdown_connections 2024-08-03T21:00:23.064 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.061+0000 7fc5f2e00640 1 -- 172.21.15.32:0/2373415728 wait complete. 2024-08-03T21:00:23.086 DEBUG:teuthology.orchestra.run.smithi032:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata import - 2024-08-03T21:00:23.232 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.230+0000 7f91802b1f40 1 Processor -- start 2024-08-03T21:00:23.232 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.231+0000 7f91802b1f40 1 -- start start 2024-08-03T21:00:23.234 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.232+0000 7f91802b1f40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5562836dcb40 0x55628376dfb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:00:23.235 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.232+0000 7f91802b1f40 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55628374fbe0 0x556283750aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:00:23.235 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.232+0000 7f91802b1f40 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5562836956c0 0x556283744c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:00:23.235 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.232+0000 7f91802b1f40 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x556283643bf0 con 0x5562836956c0 2024-08-03T21:00:23.235 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.232+0000 7f91802b1f40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x5562836506b0 con 0x5562836dcb40 2024-08-03T21:00:23.235 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.232+0000 7f91802b1f40 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x5562835e5e30 con 0x55628374fbe0 2024-08-03T21:00:23.235 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.232+0000 7f9179e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5562836956c0 0x556283744c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:00:23.235 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.232+0000 7f9179e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5562836956c0 0x556283744c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:38066/0 (socket says 172.21.15.32:38066) 2024-08-03T21:00:23.235 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.232+0000 7f9179e00640 1 -- 172.21.15.32:0/4084276347 learned_addr learned my addr 172.21.15.32:0/4084276347 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T21:00:23.235 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.233+0000 7f9178a00640 1 --2- 172.21.15.32:0/4084276347 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55628374fbe0 0x556283750aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:00:23.235 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.233+0000 7f9179400640 1 --2- 172.21.15.32:0/4084276347 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5562836dcb40 0x55628376dfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:00:23.235 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.233+0000 7f9179e00640 1 -- 172.21.15.32:0/4084276347 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55628374fbe0 msgr2=0x556283750aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T21:00:23.235 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.233+0000 7f9179e00640 1 --2- 172.21.15.32:0/4084276347 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55628374fbe0 0x556283750aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:23.235 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.233+0000 7f9179e00640 1 -- 172.21.15.32:0/4084276347 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5562836dcb40 msgr2=0x55628376dfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T21:00:23.236 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.233+0000 7f9179e00640 1 --2- 172.21.15.32:0/4084276347 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5562836dcb40 0x55628376dfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:23.236 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.233+0000 7f9179e00640 1 -- 172.21.15.32:0/4084276347 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5562837457c0 con 0x5562836956c0 2024-08-03T21:00:23.236 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.233+0000 7f9179e00640 1 --2- 172.21.15.32:0/4084276347 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5562836956c0 0x556283744c70 secure :-1 s=READY pgs=2073 cs=0 l=1 rev1=1 crypto rx=0x7f9174025df0 tx=0x7f9174003330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=535c58ab5a12c02e server_cookie=0 in_seq=0 out_seq=0 2024-08-03T21:00:23.237 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.233+0000 7f9173400640 1 -- 172.21.15.32:0/4084276347 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9174064070 con 0x5562836956c0 2024-08-03T21:00:23.237 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.234+0000 7f9173400640 1 -- 172.21.15.32:0/4084276347 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f9174004020 con 0x5562836956c0 2024-08-03T21:00:23.237 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.234+0000 7f9173400640 1 -- 172.21.15.32:0/4084276347 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f917404dae0 con 0x5562836956c0 2024-08-03T21:00:23.237 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.234+0000 7f91802b1f40 1 -- 172.21.15.32:0/4084276347 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5562836956c0 msgr2=0x556283744c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:00:23.237 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.234+0000 7f91802b1f40 1 --2- 172.21.15.32:0/4084276347 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5562836956c0 0x556283744c70 secure :-1 s=READY pgs=2073 cs=0 l=1 rev1=1 crypto rx=0x7f9174025df0 tx=0x7f9174003330 comp rx=0 tx=0).stop 2024-08-03T21:00:23.237 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.234+0000 7f91802b1f40 1 -- 172.21.15.32:0/4084276347 shutdown_connections 2024-08-03T21:00:23.237 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.234+0000 7f91802b1f40 1 --2- 172.21.15.32:0/4084276347 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5562836956c0 0x556283744c70 unknown :-1 s=CLOSED pgs=2073 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:23.237 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.234+0000 7f91802b1f40 1 --2- 172.21.15.32:0/4084276347 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55628374fbe0 0x556283750aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:23.238 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.234+0000 7f91802b1f40 1 --2- 172.21.15.32:0/4084276347 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5562836dcb40 0x55628376dfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:23.238 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.234+0000 7f91802b1f40 1 -- 172.21.15.32:0/4084276347 >> 172.21.15.32:0/4084276347 conn(0x5562835b7ac0 msgr2=0x55628376fc10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T21:00:23.238 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.234+0000 7f91802b1f40 1 -- 172.21.15.32:0/4084276347 shutdown_connections 2024-08-03T21:00:23.238 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.234+0000 7f91802b1f40 1 -- 172.21.15.32:0/4084276347 wait complete. 2024-08-03T21:00:23.238 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.235+0000 7f91802b1f40 1 Processor -- start 2024-08-03T21:00:23.238 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.236+0000 7f91802b1f40 1 -- start start 2024-08-03T21:00:23.239 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.237+0000 7f91802b1f40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5562836956c0 0x556283745ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:00:23.240 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.237+0000 7f91802b1f40 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5562836dcb40 0x556283749fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:00:23.240 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.237+0000 7f91802b1f40 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55628374fbe0 0x55628373faf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:00:23.240 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.237+0000 7f91802b1f40 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x556283697da0 con 0x5562836dcb40 2024-08-03T21:00:23.240 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.237+0000 7f91802b1f40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x5562837517f0 con 0x5562836956c0 2024-08-03T21:00:23.240 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.237+0000 7f91802b1f40 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x55628374b340 con 0x55628374fbe0 2024-08-03T21:00:23.240 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.237+0000 7f9178a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5562836dcb40 0x556283749fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:00:23.240 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.237+0000 7f9178a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5562836dcb40 0x556283749fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:38080/0 (socket says 172.21.15.32:38080) 2024-08-03T21:00:23.240 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.237+0000 7f9178a00640 1 -- 172.21.15.32:0/40379518 learned_addr learned my addr 172.21.15.32:0/40379518 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T21:00:23.240 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.237+0000 7f9179e00640 1 --2- 172.21.15.32:0/40379518 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55628374fbe0 0x55628373faf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:00:23.240 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.237+0000 7f9179400640 1 --2- 172.21.15.32:0/40379518 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5562836956c0 0x556283745ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:00:23.240 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.238+0000 7f9178a00640 1 -- 172.21.15.32:0/40379518 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55628374fbe0 msgr2=0x55628373faf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:00:23.240 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.238+0000 7f9178a00640 1 --2- 172.21.15.32:0/40379518 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55628374fbe0 0x55628373faf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:23.240 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.238+0000 7f9178a00640 1 -- 172.21.15.32:0/40379518 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5562836956c0 msgr2=0x556283745ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:00:23.241 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.238+0000 7f9178a00640 1 --2- 172.21.15.32:0/40379518 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5562836956c0 0x556283745ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:23.241 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.238+0000 7f9178a00640 1 -- 172.21.15.32:0/40379518 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5562837476f0 con 0x5562836dcb40 2024-08-03T21:00:23.241 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.238+0000 7f9178a00640 1 --2- 172.21.15.32:0/40379518 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5562836dcb40 0x556283749fa0 secure :-1 s=READY pgs=2074 cs=0 l=1 rev1=1 crypto rx=0x7f916800dfa0 tx=0x7f916800b5b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T21:00:23.241 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.238+0000 7f9172000640 1 -- 172.21.15.32:0/40379518 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91680032d0 con 0x5562836dcb40 2024-08-03T21:00:23.241 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.238+0000 7f9172000640 1 -- 172.21.15.32:0/40379518 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f9168016070 con 0x5562836dcb40 2024-08-03T21:00:23.241 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.238+0000 7f9172000640 1 -- 172.21.15.32:0/40379518 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9168001800 con 0x5562836dcb40 2024-08-03T21:00:23.241 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.238+0000 7f91802b1f40 1 -- 172.21.15.32:0/40379518 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x5562837479d0 con 0x5562836dcb40 2024-08-03T21:00:23.241 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.238+0000 7f91802b1f40 1 -- 172.21.15.32:0/40379518 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556283740030 con 0x5562836dcb40 2024-08-03T21:00:23.242 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.241+0000 7f9172000640 1 -- 172.21.15.32:0/40379518 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f91680019a0 con 0x5562836dcb40 2024-08-03T21:00:23.243 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.242+0000 7f9172000640 1 --2- 172.21.15.32:0/40379518 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f915c0759f0 0x7f915c077f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:00:23.243 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.242+0000 7f9172000640 1 -- 172.21.15.32:0/40379518 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(124..124 src has 1..124) ==== 9208+0+0 (secure 0 0 0) 0x7f9168098330 con 0x5562836dcb40 2024-08-03T21:00:23.243 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.242+0000 7f9179400640 1 --2- 172.21.15.32:0/40379518 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f915c0759f0 0x7f915c077f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:00:23.244 INFO:teuthology.orchestra.run.smithi032.stdout:Importing pool 2024-08-03T21:00:23.244 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.242+0000 7f91802b1f40 1 --2- 172.21.15.32:0/40379518 >> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] conn(0x5562836df610 0x5562836dfa20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:00:23.244 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.242+0000 7f91802b1f40 1 -- 172.21.15.32:0/40379518 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:1 12.10 12:09401a81:::601.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e124) -- 0x5562838109c0 con 0x5562836df610 2024-08-03T21:00:23.245 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.242+0000 7f9179e00640 1 --2- 172.21.15.32:0/40379518 >> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] conn(0x5562836df610 0x5562836dfa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:00:23.245 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.243+0000 7f9179400640 1 --2- 172.21.15.32:0/40379518 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f915c0759f0 0x7f915c077f00 secure :-1 s=READY pgs=827 cs=0 l=1 rev1=1 crypto rx=0x7f916c000c00 tx=0x7f916c002550 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T21:00:23.245 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.243+0000 7f9179e00640 1 --2- 172.21.15.32:0/40379518 >> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] conn(0x5562836df610 0x5562836dfa20 crc :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T21:00:23.247 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.246+0000 7f9179e00640 1 -- 172.21.15.32:0/40379518 <== osd.0 v2:172.21.15.32:6816/3066156116 1 ==== osd_op_reply(1 601.00000000 [create] v124'1190 uv1164 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f917400b040 con 0x5562836df610 2024-08-03T21:00:23.247 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.246+0000 7f91802b1f40 1 -- 172.21.15.32:0/40379518 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:2 12.10 12:09401a81:::601.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e124) -- 0x556283810d60 con 0x5562836df610 2024-08-03T21:00:23.249 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.248+0000 7f9179e00640 1 -- 172.21.15.32:0/40379518 <== osd.0 v2:172.21.15.32:6816/3066156116 2 ==== osd_op_reply(2 601.00000000 [delete] v124'1191 uv1191 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f917400b040 con 0x5562836df610 2024-08-03T21:00:23.249 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.248+0000 7f91802b1f40 1 -- 172.21.15.32:0/40379518 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:3 12.10 12:09401a81:::601.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e124) -- 0x556283811100 con 0x5562836df610 2024-08-03T21:00:23.251 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.249+0000 7f9179e00640 1 -- 172.21.15.32:0/40379518 <== osd.0 v2:172.21.15.32:6816/3066156116 3 ==== osd_op_reply(3 601.00000000 [create] v124'1192 uv1192 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f917400b040 con 0x5562836df610 2024-08-03T21:00:23.251 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::601.00000000:head# 2024-08-03T21:00:23.251 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.250+0000 7f91802b1f40 1 -- 172.21.15.32:0/40379518 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:4 12.10 12:09401a81:::601.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e124) -- 0x5562838124b0 con 0x5562836df610 2024-08-03T21:00:23.253 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.252+0000 7f9179e00640 1 -- 172.21.15.32:0/40379518 <== osd.0 v2:172.21.15.32:6816/3066156116 4 ==== osd_op_reply(4 601.00000000 [setxattr (30)] v124'1193 uv1193 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f917400b040 con 0x5562836df610 2024-08-03T21:00:23.253 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.252+0000 7f91802b1f40 1 -- 172.21.15.32:0/40379518 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:5 12.10 12:09401a81:::601.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e124) -- 0x556283812850 con 0x5562836df610 2024-08-03T21:00:23.255 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.253+0000 7f9179e00640 1 -- 172.21.15.32:0/40379518 <== osd.0 v2:172.21.15.32:6816/3066156116 5 ==== osd_op_reply(5 601.00000000 [setxattr (62)] v124'1194 uv1194 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f917400b040 con 0x5562836df610 2024-08-03T21:00:23.255 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.253+0000 7f91802b1f40 1 -- 172.21.15.32:0/40379518 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:6 12.10 12:09401a81:::601.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e124) -- 0x5562838114a0 con 0x5562836df610 2024-08-03T21:00:23.256 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.255+0000 7f9179e00640 1 -- 172.21.15.32:0/40379518 <== osd.0 v2:172.21.15.32:6816/3066156116 6 ==== osd_op_reply(6 601.00000000 [omap-set-header] v124'1195 uv1195 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f917400b040 con 0x5562836df610 2024-08-03T21:00:23.257 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.255+0000 7f91802b1f40 1 -- 172.21.15.32:0/40379518 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:7 12.10 12:0bd6d154:::602.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e124) -- 0x556283811890 con 0x5562836df610 2024-08-03T21:00:23.259 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.257+0000 7f9179e00640 1 -- 172.21.15.32:0/40379518 <== osd.0 v2:172.21.15.32:6816/3066156116 7 ==== osd_op_reply(7 602.00000000 [create] v124'1196 uv1170 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f917400b040 con 0x5562836df610 2024-08-03T21:00:23.259 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.257+0000 7f91802b1f40 1 -- 172.21.15.32:0/40379518 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:8 12.10 12:0bd6d154:::602.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e124) -- 0x556283811c30 con 0x5562836df610 2024-08-03T21:00:23.260 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.259+0000 7f9179e00640 1 -- 172.21.15.32:0/40379518 <== osd.0 v2:172.21.15.32:6816/3066156116 8 ==== osd_op_reply(8 602.00000000 [delete] v124'1197 uv1197 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f917400b040 con 0x5562836df610 2024-08-03T21:00:23.260 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.259+0000 7f91802b1f40 1 -- 172.21.15.32:0/40379518 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:9 12.10 12:0bd6d154:::602.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e124) -- 0x556283811c30 con 0x5562836df610 2024-08-03T21:00:23.262 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::602.00000000:head# 2024-08-03T21:00:23.262 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.261+0000 7f9179e00640 1 -- 172.21.15.32:0/40379518 <== osd.0 v2:172.21.15.32:6816/3066156116 9 ==== osd_op_reply(9 602.00000000 [create] v124'1198 uv1198 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f917400b040 con 0x5562836df610 2024-08-03T21:00:23.263 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.261+0000 7f91802b1f40 1 -- 172.21.15.32:0/40379518 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:10 12.10 12:0bd6d154:::602.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e124) -- 0x556283811c30 con 0x5562836df610 2024-08-03T21:00:23.264 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.262+0000 7f9179e00640 1 -- 172.21.15.32:0/40379518 <== osd.0 v2:172.21.15.32:6816/3066156116 10 ==== osd_op_reply(10 602.00000000 [setxattr (30)] v124'1199 uv1199 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f917400b040 con 0x5562836df610 2024-08-03T21:00:23.264 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.262+0000 7f91802b1f40 1 -- 172.21.15.32:0/40379518 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:11 12.10 12:0bd6d154:::602.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e124) -- 0x556283811c30 con 0x5562836df610 2024-08-03T21:00:23.266 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.264+0000 7f9179e00640 1 -- 172.21.15.32:0/40379518 <== osd.0 v2:172.21.15.32:6816/3066156116 11 ==== osd_op_reply(11 602.00000000 [setxattr (62)] v124'1200 uv1200 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f917400b040 con 0x5562836df610 2024-08-03T21:00:23.266 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.264+0000 7f91802b1f40 1 -- 172.21.15.32:0/40379518 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:12 12.10 12:0bd6d154:::602.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e124) -- 0x556283811c30 con 0x5562836df610 2024-08-03T21:00:23.267 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.266+0000 7f9179e00640 1 -- 172.21.15.32:0/40379518 <== osd.0 v2:172.21.15.32:6816/3066156116 12 ==== osd_op_reply(12 602.00000000 [omap-set-header] v124'1201 uv1201 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f917400b040 con 0x5562836df610 2024-08-03T21:00:23.268 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.266+0000 7f91802b1f40 1 -- 172.21.15.32:0/40379518 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:13 12.10 12:0cf6ad0f:::10000000000.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e124) -- 0x556283811c30 con 0x5562836df610 2024-08-03T21:00:23.269 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.268+0000 7f9179e00640 1 -- 172.21.15.32:0/40379518 <== osd.0 v2:172.21.15.32:6816/3066156116 13 ==== osd_op_reply(13 10000000000.00000000 [create] v124'1202 uv1177 ondisk = -17 ((17) File exists)) ==== 164+0+0 (crc 0 0 0) 0x7f917400b040 con 0x5562836df610 2024-08-03T21:00:23.270 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.268+0000 7f91802b1f40 1 -- 172.21.15.32:0/40379518 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:14 12.10 12:0cf6ad0f:::10000000000.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e124) -- 0x556283811c30 con 0x5562836df610 2024-08-03T21:00:23.271 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.270+0000 7f9179e00640 1 -- 172.21.15.32:0/40379518 <== osd.0 v2:172.21.15.32:6816/3066156116 14 ==== osd_op_reply(14 10000000000.00000000 [delete] v124'1203 uv1203 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f917400b040 con 0x5562836df610 2024-08-03T21:00:23.271 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.270+0000 7f91802b1f40 1 -- 172.21.15.32:0/40379518 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:15 12.10 12:0cf6ad0f:::10000000000.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e124) -- 0x556283811c30 con 0x5562836df610 2024-08-03T21:00:23.273 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.272+0000 7f9179e00640 1 -- 172.21.15.32:0/40379518 <== osd.0 v2:172.21.15.32:6816/3066156116 15 ==== osd_op_reply(15 10000000000.00000000 [create] v124'1204 uv1204 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f917400b040 con 0x5562836df610 2024-08-03T21:00:23.274 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::10000000000.00000000:head# 2024-08-03T21:00:23.274 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.272+0000 7f91802b1f40 1 -- 172.21.15.32:0/40379518 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:16 12.10 12:0cf6ad0f:::10000000000.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e124) -- 0x556283811c30 con 0x5562836df610 2024-08-03T21:00:23.275 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.273+0000 7f9179e00640 1 -- 172.21.15.32:0/40379518 <== osd.0 v2:172.21.15.32:6816/3066156116 16 ==== osd_op_reply(16 10000000000.00000000 [setxattr (30)] v124'1205 uv1205 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f917400b040 con 0x5562836df610 2024-08-03T21:00:23.275 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.273+0000 7f91802b1f40 1 -- 172.21.15.32:0/40379518 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:17 12.10 12:0cf6ad0f:::10000000000.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e124) -- 0x556283811c30 con 0x5562836df610 2024-08-03T21:00:23.276 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.275+0000 7f9179e00640 1 -- 172.21.15.32:0/40379518 <== osd.0 v2:172.21.15.32:6816/3066156116 17 ==== osd_op_reply(17 10000000000.00000000 [setxattr (62)] v124'1206 uv1206 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f917400b040 con 0x5562836df610 2024-08-03T21:00:23.277 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.275+0000 7f91802b1f40 1 -- 172.21.15.32:0/40379518 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:18 12.10 12:0cf6ad0f:::10000000000.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e124) -- 0x556283811c30 con 0x5562836df610 2024-08-03T21:00:23.278 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.277+0000 7f9179e00640 1 -- 172.21.15.32:0/40379518 <== osd.0 v2:172.21.15.32:6816/3066156116 18 ==== osd_op_reply(18 10000000000.00000000 [omap-set-header] v124'1207 uv1207 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f917400b040 con 0x5562836df610 2024-08-03T21:00:23.278 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.277+0000 7f91802b1f40 1 -- 172.21.15.32:0/40379518 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:19 12.10 12:0cf6ad0f:::10000000000.00000000:head [omap-set-vals in=593b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e124) -- 0x556283812010 con 0x5562836df610 2024-08-03T21:00:23.280 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.278+0000 7f9179e00640 1 -- 172.21.15.32:0/40379518 <== osd.0 v2:172.21.15.32:6816/3066156116 19 ==== osd_op_reply(19 10000000000.00000000 [omap-set-vals] v124'1208 uv1208 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f917400b040 con 0x5562836df610 2024-08-03T21:00:23.280 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.279+0000 7f91802b1f40 1 -- 172.21.15.32:0/40379518 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:20 12.10 12:0d82a743:::600.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e124) -- 0x55628374a850 con 0x5562836df610 2024-08-03T21:00:23.282 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.280+0000 7f9179e00640 1 -- 172.21.15.32:0/40379518 <== osd.0 v2:172.21.15.32:6816/3066156116 20 ==== osd_op_reply(20 600.00000000 [create] v124'1209 uv1183 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f917400b040 con 0x5562836df610 2024-08-03T21:00:23.282 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.280+0000 7f91802b1f40 1 -- 172.21.15.32:0/40379518 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:21 12.10 12:0d82a743:::600.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e124) -- 0x55628374a850 con 0x5562836df610 2024-08-03T21:00:23.284 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.283+0000 7f9179e00640 1 -- 172.21.15.32:0/40379518 <== osd.0 v2:172.21.15.32:6816/3066156116 21 ==== osd_op_reply(21 600.00000000 [delete] v124'1210 uv1210 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f917400b040 con 0x5562836df610 2024-08-03T21:00:23.284 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.283+0000 7f91802b1f40 1 -- 172.21.15.32:0/40379518 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:22 12.10 12:0d82a743:::600.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e124) -- 0x55628374a850 con 0x5562836df610 2024-08-03T21:00:23.285 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::600.00000000:head# 2024-08-03T21:00:23.286 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.284+0000 7f9179e00640 1 -- 172.21.15.32:0/40379518 <== osd.0 v2:172.21.15.32:6816/3066156116 22 ==== osd_op_reply(22 600.00000000 [create] v124'1211 uv1211 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f917400b040 con 0x5562836df610 2024-08-03T21:00:23.286 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.284+0000 7f91802b1f40 1 -- 172.21.15.32:0/40379518 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:23 12.10 12:0d82a743:::600.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e124) -- 0x55628374a850 con 0x5562836df610 2024-08-03T21:00:23.287 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.286+0000 7f9179e00640 1 -- 172.21.15.32:0/40379518 <== osd.0 v2:172.21.15.32:6816/3066156116 23 ==== osd_op_reply(23 600.00000000 [setxattr (30)] v124'1212 uv1212 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f917400b040 con 0x5562836df610 2024-08-03T21:00:23.288 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.286+0000 7f91802b1f40 1 -- 172.21.15.32:0/40379518 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:24 12.10 12:0d82a743:::600.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e124) -- 0x55628374a850 con 0x5562836df610 2024-08-03T21:00:23.289 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.288+0000 7f9179e00640 1 -- 172.21.15.32:0/40379518 <== osd.0 v2:172.21.15.32:6816/3066156116 24 ==== osd_op_reply(24 600.00000000 [setxattr (62)] v124'1213 uv1213 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f917400b040 con 0x5562836df610 2024-08-03T21:00:23.289 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.288+0000 7f91802b1f40 1 -- 172.21.15.32:0/40379518 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:25 12.10 12:0d82a743:::600.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e124) -- 0x55628374a850 con 0x5562836df610 2024-08-03T21:00:23.291 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.289+0000 7f9179e00640 1 -- 172.21.15.32:0/40379518 <== osd.0 v2:172.21.15.32:6816/3066156116 25 ==== osd_op_reply(25 600.00000000 [omap-set-header] v124'1214 uv1214 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f917400b040 con 0x5562836df610 2024-08-03T21:00:23.291 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.289+0000 7f91802b1f40 1 -- 172.21.15.32:0/40379518 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:26 12.10 12:0d89b25e:::603.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e124) -- 0x55628374a850 con 0x5562836df610 2024-08-03T21:00:23.293 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.291+0000 7f9179e00640 1 -- 172.21.15.32:0/40379518 <== osd.0 v2:172.21.15.32:6816/3066156116 26 ==== osd_op_reply(26 603.00000000 [create] v124'1215 uv1189 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f917400b040 con 0x5562836df610 2024-08-03T21:00:23.293 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.291+0000 7f91802b1f40 1 -- 172.21.15.32:0/40379518 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:27 12.10 12:0d89b25e:::603.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e124) -- 0x55628374a850 con 0x5562836df610 2024-08-03T21:00:23.295 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.293+0000 7f9179e00640 1 -- 172.21.15.32:0/40379518 <== osd.0 v2:172.21.15.32:6816/3066156116 27 ==== osd_op_reply(27 603.00000000 [delete] v124'1216 uv1216 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f917400b040 con 0x5562836df610 2024-08-03T21:00:23.295 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.293+0000 7f91802b1f40 1 -- 172.21.15.32:0/40379518 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:28 12.10 12:0d89b25e:::603.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e124) -- 0x55628374a850 con 0x5562836df610 2024-08-03T21:00:23.296 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::603.00000000:head# 2024-08-03T21:00:23.297 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.295+0000 7f9179e00640 1 -- 172.21.15.32:0/40379518 <== osd.0 v2:172.21.15.32:6816/3066156116 28 ==== osd_op_reply(28 603.00000000 [create] v124'1217 uv1217 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f917400b040 con 0x5562836df610 2024-08-03T21:00:23.297 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.295+0000 7f91802b1f40 1 -- 172.21.15.32:0/40379518 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:29 12.10 12:0d89b25e:::603.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e124) -- 0x55628374a850 con 0x5562836df610 2024-08-03T21:00:23.298 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.297+0000 7f9179e00640 1 -- 172.21.15.32:0/40379518 <== osd.0 v2:172.21.15.32:6816/3066156116 29 ==== osd_op_reply(29 603.00000000 [setxattr (30)] v124'1218 uv1218 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f917400b040 con 0x5562836df610 2024-08-03T21:00:23.298 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.297+0000 7f91802b1f40 1 -- 172.21.15.32:0/40379518 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:30 12.10 12:0d89b25e:::603.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e124) -- 0x55628374a850 con 0x5562836df610 2024-08-03T21:00:23.300 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.298+0000 7f9179e00640 1 -- 172.21.15.32:0/40379518 <== osd.0 v2:172.21.15.32:6816/3066156116 30 ==== osd_op_reply(30 603.00000000 [setxattr (62)] v124'1219 uv1219 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f917400b040 con 0x5562836df610 2024-08-03T21:00:23.300 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.298+0000 7f91802b1f40 1 -- 172.21.15.32:0/40379518 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:31 12.10 12:0d89b25e:::603.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e124) -- 0x55628374a850 con 0x5562836df610 2024-08-03T21:00:23.302 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.300+0000 7f9179e00640 1 -- 172.21.15.32:0/40379518 <== osd.0 v2:172.21.15.32:6816/3066156116 31 ==== osd_op_reply(31 603.00000000 [omap-set-header] v124'1220 uv1220 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f917400b040 con 0x5562836df610 2024-08-03T21:00:23.302 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.300+0000 7f91802b1f40 1 --2- 172.21.15.32:0/40379518 >> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] conn(0x556283812bf0 0x556283815010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:00:23.302 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.300+0000 7f91802b1f40 1 -- 172.21.15.32:0/40379518 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:32 12.14 12:287034c4:::1.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e124) -- 0x556283815550 con 0x556283812bf0 2024-08-03T21:00:23.302 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.302+0000 7f9179400640 1 --2- 172.21.15.32:0/40379518 >> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] conn(0x556283812bf0 0x556283815010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:00:23.303 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.302+0000 7f9179400640 1 --2- 172.21.15.32:0/40379518 >> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] conn(0x556283812bf0 0x556283815010 crc :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T21:00:23.305 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.304+0000 7f9179400640 1 -- 172.21.15.32:0/40379518 <== osd.5 v2:172.21.15.118:6810/2724891770 1 ==== osd_op_reply(32 1.00000000.inode [create] v124'867 uv850 ondisk = -17 ((17) File exists)) ==== 160+0+0 (crc 0 0 0) 0x7f916c01c410 con 0x556283812bf0 2024-08-03T21:00:23.305 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.304+0000 7f91802b1f40 1 -- 172.21.15.32:0/40379518 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:33 12.14 12:287034c4:::1.00000000.inode:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e124) -- 0x5562838158f0 con 0x556283812bf0 2024-08-03T21:00:23.307 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.305+0000 7f9179400640 1 -- 172.21.15.32:0/40379518 <== osd.5 v2:172.21.15.118:6810/2724891770 2 ==== osd_op_reply(33 1.00000000.inode [delete] v124'868 uv868 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f916c01c410 con 0x556283812bf0 2024-08-03T21:00:23.307 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.305+0000 7f91802b1f40 1 -- 172.21.15.32:0/40379518 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:34 12.14 12:287034c4:::1.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e124) -- 0x556283815c90 con 0x556283812bf0 2024-08-03T21:00:23.308 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.307+0000 7f9179400640 1 -- 172.21.15.32:0/40379518 <== osd.5 v2:172.21.15.118:6810/2724891770 3 ==== osd_op_reply(34 1.00000000.inode [create] v124'869 uv869 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f916c01c410 con 0x556283812bf0 2024-08-03T21:00:23.309 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::1.00000000.inode:head# 2024-08-03T21:00:23.309 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.307+0000 7f91802b1f40 1 -- 172.21.15.32:0/40379518 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:35 12.14 12:287034c4:::1.00000000.inode:head [write 0~582 in=582b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e124) -- 0x556283816030 con 0x556283812bf0 2024-08-03T21:00:23.310 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.309+0000 7f9179400640 1 -- 172.21.15.32:0/40379518 <== osd.5 v2:172.21.15.118:6810/2724891770 4 ==== osd_op_reply(35 1.00000000.inode [write 0~582] v124'870 uv870 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f916c01c410 con 0x556283812bf0 2024-08-03T21:00:23.310 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.309+0000 7f91802b1f40 1 -- 172.21.15.32:0/40379518 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:36 12.14 12:287034c4:::1.00000000.inode:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e124) -- 0x5562838163d0 con 0x556283812bf0 2024-08-03T21:00:23.312 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.310+0000 7f9179400640 1 -- 172.21.15.32:0/40379518 <== osd.5 v2:172.21.15.118:6810/2724891770 5 ==== osd_op_reply(36 1.00000000.inode [omap-set-header] v124'871 uv871 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f916c01c410 con 0x556283812bf0 2024-08-03T21:00:23.312 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.310+0000 7f91802b1f40 1 -- 172.21.15.32:0/40379518 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:37 12.14 12:292cf221:::200.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e124) -- 0x556283816770 con 0x556283812bf0 2024-08-03T21:00:23.314 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.312+0000 7f9179400640 1 -- 172.21.15.32:0/40379518 <== osd.5 v2:172.21.15.118:6810/2724891770 6 ==== osd_op_reply(37 200.00000000 [create] v124'872 uv855 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f916c01c410 con 0x556283812bf0 2024-08-03T21:00:23.314 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.312+0000 7f91802b1f40 1 -- 172.21.15.32:0/40379518 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:38 12.14 12:292cf221:::200.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e124) -- 0x556283816b10 con 0x556283812bf0 2024-08-03T21:00:23.315 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.314+0000 7f9179400640 1 -- 172.21.15.32:0/40379518 <== osd.5 v2:172.21.15.118:6810/2724891770 7 ==== osd_op_reply(38 200.00000000 [delete] v124'873 uv873 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f916c01c410 con 0x556283812bf0 2024-08-03T21:00:23.316 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.314+0000 7f91802b1f40 1 -- 172.21.15.32:0/40379518 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:39 12.14 12:292cf221:::200.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e124) -- 0x556283816eb0 con 0x556283812bf0 2024-08-03T21:00:23.317 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.316+0000 7f9179400640 1 -- 172.21.15.32:0/40379518 <== osd.5 v2:172.21.15.118:6810/2724891770 8 ==== osd_op_reply(39 200.00000000 [create] v124'874 uv874 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f916c01c410 con 0x556283812bf0 2024-08-03T21:00:23.317 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::200.00000000:head# 2024-08-03T21:00:23.318 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.316+0000 7f91802b1f40 1 -- 172.21.15.32:0/40379518 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:40 12.14 12:292cf221:::200.00000000:head [write 0~90 in=90b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e124) -- 0x556283816eb0 con 0x556283812bf0 2024-08-03T21:00:23.319 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.318+0000 7f9179400640 1 -- 172.21.15.32:0/40379518 <== osd.5 v2:172.21.15.118:6810/2724891770 9 ==== osd_op_reply(40 200.00000000 [write 0~90] v124'875 uv875 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f916c01c410 con 0x556283812bf0 2024-08-03T21:00:23.319 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.318+0000 7f91802b1f40 1 -- 172.21.15.32:0/40379518 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:41 12.14 12:292cf221:::200.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e124) -- 0x556283816eb0 con 0x556283812bf0 2024-08-03T21:00:23.321 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.319+0000 7f9179400640 1 -- 172.21.15.32:0/40379518 <== osd.5 v2:172.21.15.118:6810/2724891770 10 ==== osd_op_reply(41 200.00000000 [omap-set-header] v124'876 uv876 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f916c01c410 con 0x556283812bf0 2024-08-03T21:00:23.321 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.319+0000 7f91802b1f40 1 -- 172.21.15.32:0/40379518 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:42 12.14 12:2e2fa760:::200.00000001:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e124) -- 0x556283816eb0 con 0x556283812bf0 2024-08-03T21:00:23.323 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.321+0000 7f9179400640 1 -- 172.21.15.32:0/40379518 <== osd.5 v2:172.21.15.118:6810/2724891770 11 ==== osd_op_reply(42 200.00000001 [create] v124'877 uv860 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f916c01c410 con 0x556283812bf0 2024-08-03T21:00:23.323 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.321+0000 7f91802b1f40 1 -- 172.21.15.32:0/40379518 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:43 12.14 12:2e2fa760:::200.00000001:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e124) -- 0x556283816eb0 con 0x556283812bf0 2024-08-03T21:00:23.324 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.323+0000 7f9179400640 1 -- 172.21.15.32:0/40379518 <== osd.5 v2:172.21.15.118:6810/2724891770 12 ==== osd_op_reply(43 200.00000001 [delete] v124'878 uv878 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f916c01c410 con 0x556283812bf0 2024-08-03T21:00:23.325 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.323+0000 7f91802b1f40 1 -- 172.21.15.32:0/40379518 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:44 12.14 12:2e2fa760:::200.00000001:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e124) -- 0x556283816eb0 con 0x556283812bf0 2024-08-03T21:00:23.326 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.325+0000 7f9179400640 1 -- 172.21.15.32:0/40379518 <== osd.5 v2:172.21.15.118:6810/2724891770 13 ==== osd_op_reply(44 200.00000001 [create] v124'879 uv879 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f916c01c410 con 0x556283812bf0 2024-08-03T21:00:23.327 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::200.00000001:head# 2024-08-03T21:00:23.327 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.325+0000 7f91802b1f40 1 -- 172.21.15.32:0/40379518 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:45 12.14 12:2e2fa760:::200.00000001:head [write 0~18118 in=18118b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e124) -- 0x55628381b5a0 con 0x556283812bf0 2024-08-03T21:00:23.328 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.327+0000 7f9179400640 1 -- 172.21.15.32:0/40379518 <== osd.5 v2:172.21.15.118:6810/2724891770 14 ==== osd_op_reply(45 200.00000001 [write 0~18118] v124'880 uv880 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f916c01c410 con 0x556283812bf0 2024-08-03T21:00:23.328 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.327+0000 7f91802b1f40 1 -- 172.21.15.32:0/40379518 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:46 12.14 12:2e2fa760:::200.00000001:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e124) -- 0x55628381b5a0 con 0x556283812bf0 2024-08-03T21:00:23.330 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.328+0000 7f9179400640 1 -- 172.21.15.32:0/40379518 <== osd.5 v2:172.21.15.118:6810/2724891770 15 ==== osd_op_reply(46 200.00000001 [omap-set-header] v124'881 uv881 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f916c01c410 con 0x556283812bf0 2024-08-03T21:00:23.330 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.329+0000 7f91802b1f40 1 -- 172.21.15.32:0/40379518 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:47 12.14 12:2f55791f:::606.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e124) -- 0x556283816eb0 con 0x556283812bf0 2024-08-03T21:00:23.332 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.330+0000 7f9179400640 1 -- 172.21.15.32:0/40379518 <== osd.5 v2:172.21.15.118:6810/2724891770 16 ==== osd_op_reply(47 606.00000000 [create] v124'882 uv866 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f916c01c410 con 0x556283812bf0 2024-08-03T21:00:23.332 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.331+0000 7f91802b1f40 1 -- 172.21.15.32:0/40379518 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:48 12.14 12:2f55791f:::606.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e124) -- 0x556283816eb0 con 0x556283812bf0 2024-08-03T21:00:23.334 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.332+0000 7f9179400640 1 -- 172.21.15.32:0/40379518 <== osd.5 v2:172.21.15.118:6810/2724891770 17 ==== osd_op_reply(48 606.00000000 [delete] v124'883 uv883 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f916c01c410 con 0x556283812bf0 2024-08-03T21:00:23.334 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.332+0000 7f91802b1f40 1 -- 172.21.15.32:0/40379518 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:49 12.14 12:2f55791f:::606.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e124) -- 0x556283816eb0 con 0x556283812bf0 2024-08-03T21:00:23.335 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.334+0000 7f9179400640 1 -- 172.21.15.32:0/40379518 <== osd.5 v2:172.21.15.118:6810/2724891770 18 ==== osd_op_reply(49 606.00000000 [create] v124'884 uv884 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f916c01c410 con 0x556283812bf0 2024-08-03T21:00:23.335 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::606.00000000:head# 2024-08-03T21:00:23.335 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.334+0000 7f91802b1f40 1 -- 172.21.15.32:0/40379518 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:50 12.14 12:2f55791f:::606.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e124) -- 0x556283817ec0 con 0x556283812bf0 2024-08-03T21:00:23.337 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.335+0000 7f9179400640 1 -- 172.21.15.32:0/40379518 <== osd.5 v2:172.21.15.118:6810/2724891770 19 ==== osd_op_reply(50 606.00000000 [setxattr (30)] v124'885 uv885 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f916c01c410 con 0x556283812bf0 2024-08-03T21:00:23.337 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.335+0000 7f91802b1f40 1 -- 172.21.15.32:0/40379518 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:51 12.14 12:2f55791f:::606.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e124) -- 0x556283817ec0 con 0x556283812bf0 2024-08-03T21:00:23.339 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.337+0000 7f9179400640 1 -- 172.21.15.32:0/40379518 <== osd.5 v2:172.21.15.118:6810/2724891770 20 ==== osd_op_reply(51 606.00000000 [setxattr (62)] v124'886 uv886 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f916c01c410 con 0x556283812bf0 2024-08-03T21:00:23.339 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.337+0000 7f91802b1f40 1 -- 172.21.15.32:0/40379518 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- osd_op(unknown.0.0:52 12.14 12:2f55791f:::606.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e124) -- 0x556283816eb0 con 0x556283812bf0 2024-08-03T21:00:23.340 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.339+0000 7f9179400640 1 -- 172.21.15.32:0/40379518 <== osd.5 v2:172.21.15.118:6810/2724891770 21 ==== osd_op_reply(52 606.00000000 [omap-set-header] v124'887 uv887 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f916c01c410 con 0x556283812bf0 2024-08-03T21:00:23.340 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.339+0000 7f91802b1f40 1 -- 172.21.15.32:0/40379518 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:53 12.1c 12:3ed09add:::607.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e124) -- 0x556283816eb0 con 0x5562836df610 2024-08-03T21:00:23.342 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.340+0000 7f9179e00640 1 -- 172.21.15.32:0/40379518 <== osd.0 v2:172.21.15.32:6816/3066156116 32 ==== osd_op_reply(53 607.00000000 [create] v124'231 uv230 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f917400b040 con 0x5562836df610 2024-08-03T21:00:23.342 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.340+0000 7f91802b1f40 1 -- 172.21.15.32:0/40379518 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:54 12.1c 12:3ed09add:::607.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e124) -- 0x556283816eb0 con 0x5562836df610 2024-08-03T21:00:23.343 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.341+0000 7f9179e00640 1 -- 172.21.15.32:0/40379518 <== osd.0 v2:172.21.15.32:6816/3066156116 33 ==== osd_op_reply(54 607.00000000 [delete] v124'232 uv232 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f917400b040 con 0x5562836df610 2024-08-03T21:00:23.343 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.342+0000 7f91802b1f40 1 -- 172.21.15.32:0/40379518 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:55 12.1c 12:3ed09add:::607.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e124) -- 0x556283816eb0 con 0x5562836df610 2024-08-03T21:00:23.344 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.343+0000 7f9179e00640 1 -- 172.21.15.32:0/40379518 <== osd.0 v2:172.21.15.32:6816/3066156116 34 ==== osd_op_reply(55 607.00000000 [create] v124'233 uv233 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f917400b040 con 0x5562836df610 2024-08-03T21:00:23.344 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::607.00000000:head# 2024-08-03T21:00:23.345 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.343+0000 7f91802b1f40 1 -- 172.21.15.32:0/40379518 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:56 12.1c 12:3ed09add:::607.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e124) -- 0x556283817ec0 con 0x5562836df610 2024-08-03T21:00:23.345 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.344+0000 7f9179e00640 1 -- 172.21.15.32:0/40379518 <== osd.0 v2:172.21.15.32:6816/3066156116 35 ==== osd_op_reply(56 607.00000000 [setxattr (30)] v124'234 uv234 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f917400b040 con 0x5562836df610 2024-08-03T21:00:23.345 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.344+0000 7f91802b1f40 1 -- 172.21.15.32:0/40379518 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:57 12.1c 12:3ed09add:::607.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e124) -- 0x556283817ec0 con 0x5562836df610 2024-08-03T21:00:23.347 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.345+0000 7f9179e00640 1 -- 172.21.15.32:0/40379518 <== osd.0 v2:172.21.15.32:6816/3066156116 36 ==== osd_op_reply(57 607.00000000 [setxattr (62)] v124'235 uv235 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f917400b040 con 0x5562836df610 2024-08-03T21:00:23.347 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.345+0000 7f91802b1f40 1 -- 172.21.15.32:0/40379518 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:58 12.1c 12:3ed09add:::607.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e124) -- 0x556283816eb0 con 0x5562836df610 2024-08-03T21:00:23.348 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.347+0000 7f9179e00640 1 -- 172.21.15.32:0/40379518 <== osd.0 v2:172.21.15.32:6816/3066156116 37 ==== osd_op_reply(58 607.00000000 [omap-set-header] v124'236 uv236 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f917400b040 con 0x5562836df610 2024-08-03T21:00:23.348 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.347+0000 7f91802b1f40 1 --2- 172.21.15.32:0/40379518 >> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] conn(0x5562838173c0 0x5562838197e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:00:23.348 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.347+0000 7f91802b1f40 1 -- 172.21.15.32:0/40379518 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:59 12.6 12:654134d2:::mds0_openfiles.0:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e124) -- 0x556283819e90 con 0x5562838173c0 2024-08-03T21:00:23.348 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.347+0000 7f9178a00640 1 --2- 172.21.15.32:0/40379518 >> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] conn(0x5562838173c0 0x5562838197e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:00:23.349 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.347+0000 7f9178a00640 1 --2- 172.21.15.32:0/40379518 >> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] conn(0x5562838173c0 0x5562838197e0 crc :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T21:00:23.350 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.349+0000 7f9178a00640 1 -- 172.21.15.32:0/40379518 <== osd.1 v2:172.21.15.32:6824/1551241759 1 ==== osd_op_reply(59 mds0_openfiles.0 [create] v124'184 uv183 ondisk = -17 ((17) File exists)) ==== 160+0+0 (crc 0 0 0) 0x7f916800e040 con 0x5562838173c0 2024-08-03T21:00:23.350 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.349+0000 7f91802b1f40 1 -- 172.21.15.32:0/40379518 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:60 12.6 12:654134d2:::mds0_openfiles.0:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e124) -- 0x55628381a3a0 con 0x5562838173c0 2024-08-03T21:00:23.352 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.350+0000 7f9178a00640 1 -- 172.21.15.32:0/40379518 <== osd.1 v2:172.21.15.32:6824/1551241759 2 ==== osd_op_reply(60 mds0_openfiles.0 [delete] v124'185 uv185 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f916800e040 con 0x5562838173c0 2024-08-03T21:00:23.352 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.350+0000 7f91802b1f40 1 -- 172.21.15.32:0/40379518 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:61 12.6 12:654134d2:::mds0_openfiles.0:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e124) -- 0x55628381a8a0 con 0x5562838173c0 2024-08-03T21:00:23.353 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.351+0000 7f9178a00640 1 -- 172.21.15.32:0/40379518 <== osd.1 v2:172.21.15.32:6824/1551241759 3 ==== osd_op_reply(61 mds0_openfiles.0 [create] v124'186 uv186 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f916800e040 con 0x5562838173c0 2024-08-03T21:00:23.353 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds0_openfiles.0:head# 2024-08-03T21:00:23.353 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.352+0000 7f91802b1f40 1 -- 172.21.15.32:0/40379518 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:62 12.6 12:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e124) -- 0x556283811fd0 con 0x5562838173c0 2024-08-03T21:00:23.354 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.353+0000 7f9178a00640 1 -- 172.21.15.32:0/40379518 <== osd.1 v2:172.21.15.32:6824/1551241759 4 ==== osd_op_reply(62 mds0_openfiles.0 [omap-set-header] v124'187 uv187 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f916800e040 con 0x5562838173c0 2024-08-03T21:00:23.354 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.353+0000 7f91802b1f40 1 -- 172.21.15.32:0/40379518 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:63 12.e 12:717a0223:::608.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e124) -- 0x55628381b1d0 con 0x5562836df610 2024-08-03T21:00:23.356 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.355+0000 7f9179e00640 1 -- 172.21.15.32:0/40379518 <== osd.0 v2:172.21.15.32:6816/3066156116 38 ==== osd_op_reply(63 608.00000000 [create] v124'231 uv230 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f917400b040 con 0x5562836df610 2024-08-03T21:00:23.356 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.355+0000 7f91802b1f40 1 -- 172.21.15.32:0/40379518 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:64 12.e 12:717a0223:::608.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e124) -- 0x55628381b1d0 con 0x5562836df610 2024-08-03T21:00:23.358 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.356+0000 7f9179e00640 1 -- 172.21.15.32:0/40379518 <== osd.0 v2:172.21.15.32:6816/3066156116 39 ==== osd_op_reply(64 608.00000000 [delete] v124'232 uv232 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f917400b040 con 0x5562836df610 2024-08-03T21:00:23.358 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.356+0000 7f91802b1f40 1 -- 172.21.15.32:0/40379518 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:65 12.e 12:717a0223:::608.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e124) -- 0x55628381b1d0 con 0x5562836df610 2024-08-03T21:00:23.359 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::608.00000000:head# 2024-08-03T21:00:23.359 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.358+0000 7f9179e00640 1 -- 172.21.15.32:0/40379518 <== osd.0 v2:172.21.15.32:6816/3066156116 40 ==== osd_op_reply(65 608.00000000 [create] v124'233 uv233 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f917400b040 con 0x5562836df610 2024-08-03T21:00:23.359 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.358+0000 7f91802b1f40 1 -- 172.21.15.32:0/40379518 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:66 12.e 12:717a0223:::608.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e124) -- 0x55628381c1e0 con 0x5562836df610 2024-08-03T21:00:23.360 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.359+0000 7f9179e00640 1 -- 172.21.15.32:0/40379518 <== osd.0 v2:172.21.15.32:6816/3066156116 41 ==== osd_op_reply(66 608.00000000 [setxattr (30)] v124'234 uv234 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f917400b040 con 0x5562836df610 2024-08-03T21:00:23.361 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.359+0000 7f91802b1f40 1 -- 172.21.15.32:0/40379518 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:67 12.e 12:717a0223:::608.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e124) -- 0x55628381c1e0 con 0x5562836df610 2024-08-03T21:00:23.362 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.360+0000 7f9179e00640 1 -- 172.21.15.32:0/40379518 <== osd.0 v2:172.21.15.32:6816/3066156116 42 ==== osd_op_reply(67 608.00000000 [setxattr (62)] v124'235 uv235 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f917400b040 con 0x5562836df610 2024-08-03T21:00:23.362 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.361+0000 7f91802b1f40 1 -- 172.21.15.32:0/40379518 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:68 12.e 12:717a0223:::608.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e124) -- 0x55628381b1d0 con 0x5562836df610 2024-08-03T21:00:23.364 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.362+0000 7f9179e00640 1 -- 172.21.15.32:0/40379518 <== osd.0 v2:172.21.15.32:6816/3066156116 43 ==== osd_op_reply(68 608.00000000 [omap-set-header] v124'236 uv236 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f917400b040 con 0x5562836df610 2024-08-03T21:00:23.364 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.362+0000 7f91802b1f40 1 --2- 172.21.15.32:0/40379518 >> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] conn(0x55628381baa0 0x55628381dec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:00:23.364 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.362+0000 7f91802b1f40 1 -- 172.21.15.32:0/40379518 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:69 12.15 12:a93a17c2:::604.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e124) -- 0x55628381e400 con 0x55628381baa0 2024-08-03T21:00:23.364 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.363+0000 7f9179e00640 1 --2- 172.21.15.32:0/40379518 >> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] conn(0x55628381baa0 0x55628381dec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:00:23.365 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.364+0000 7f9179e00640 1 --2- 172.21.15.32:0/40379518 >> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] conn(0x55628381baa0 0x55628381dec0 crc :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T21:00:23.367 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.365+0000 7f9179e00640 1 -- 172.21.15.32:0/40379518 <== osd.6 v2:172.21.15.118:6825/801947386 1 ==== osd_op_reply(69 604.00000000 [create] v124'426 uv420 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f917400b040 con 0x55628381baa0 2024-08-03T21:00:23.367 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.366+0000 7f91802b1f40 1 -- 172.21.15.32:0/40379518 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:70 12.15 12:a93a17c2:::604.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e124) -- 0x55628381e400 con 0x55628381baa0 2024-08-03T21:00:23.368 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.367+0000 7f9179e00640 1 -- 172.21.15.32:0/40379518 <== osd.6 v2:172.21.15.118:6825/801947386 2 ==== osd_op_reply(70 604.00000000 [delete] v124'427 uv427 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f917400b040 con 0x55628381baa0 2024-08-03T21:00:23.368 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.367+0000 7f91802b1f40 1 -- 172.21.15.32:0/40379518 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:71 12.15 12:a93a17c2:::604.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e124) -- 0x55628381e820 con 0x55628381baa0 2024-08-03T21:00:23.370 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::604.00000000:head# 2024-08-03T21:00:23.370 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.368+0000 7f9179e00640 1 -- 172.21.15.32:0/40379518 <== osd.6 v2:172.21.15.118:6825/801947386 3 ==== osd_op_reply(71 604.00000000 [create] v124'428 uv428 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f917400b040 con 0x55628381baa0 2024-08-03T21:00:23.371 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.368+0000 7f91802b1f40 1 -- 172.21.15.32:0/40379518 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:72 12.15 12:a93a17c2:::604.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e124) -- 0x55628381f830 con 0x55628381baa0 2024-08-03T21:00:23.371 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.370+0000 7f9179e00640 1 -- 172.21.15.32:0/40379518 <== osd.6 v2:172.21.15.118:6825/801947386 4 ==== osd_op_reply(72 604.00000000 [setxattr (30)] v124'429 uv429 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f917400b040 con 0x55628381baa0 2024-08-03T21:00:23.371 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.370+0000 7f91802b1f40 1 -- 172.21.15.32:0/40379518 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:73 12.15 12:a93a17c2:::604.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e124) -- 0x55628381f830 con 0x55628381baa0 2024-08-03T21:00:23.373 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.371+0000 7f9179e00640 1 -- 172.21.15.32:0/40379518 <== osd.6 v2:172.21.15.118:6825/801947386 5 ==== osd_op_reply(73 604.00000000 [setxattr (62)] v124'430 uv430 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f917400b040 con 0x55628381baa0 2024-08-03T21:00:23.373 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.371+0000 7f91802b1f40 1 -- 172.21.15.32:0/40379518 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:74 12.15 12:a93a17c2:::604.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e124) -- 0x55628381e820 con 0x55628381baa0 2024-08-03T21:00:23.374 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.373+0000 7f9179e00640 1 -- 172.21.15.32:0/40379518 <== osd.6 v2:172.21.15.118:6825/801947386 6 ==== osd_op_reply(74 604.00000000 [omap-set-header] v124'431 uv431 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f917400b040 con 0x55628381baa0 2024-08-03T21:00:23.375 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.373+0000 7f91802b1f40 1 -- 172.21.15.32:0/40379518 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:75 12.15 12:aa448500:::500.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e124) -- 0x55628381e820 con 0x55628381baa0 2024-08-03T21:00:23.376 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.375+0000 7f9179e00640 1 -- 172.21.15.32:0/40379518 <== osd.6 v2:172.21.15.118:6825/801947386 7 ==== osd_op_reply(75 500.00000000 [create] v124'432 uv425 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f917400b040 con 0x55628381baa0 2024-08-03T21:00:23.376 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.375+0000 7f91802b1f40 1 -- 172.21.15.32:0/40379518 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:76 12.15 12:aa448500:::500.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e124) -- 0x55628381e820 con 0x55628381baa0 2024-08-03T21:00:23.378 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.376+0000 7f9179e00640 1 -- 172.21.15.32:0/40379518 <== osd.6 v2:172.21.15.118:6825/801947386 8 ==== osd_op_reply(76 500.00000000 [delete] v124'433 uv433 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f917400b040 con 0x55628381baa0 2024-08-03T21:00:23.378 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.376+0000 7f91802b1f40 1 -- 172.21.15.32:0/40379518 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:77 12.15 12:aa448500:::500.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e124) -- 0x55628381e820 con 0x55628381baa0 2024-08-03T21:00:23.379 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::500.00000000:head# 2024-08-03T21:00:23.380 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.378+0000 7f9179e00640 1 -- 172.21.15.32:0/40379518 <== osd.6 v2:172.21.15.118:6825/801947386 9 ==== osd_op_reply(77 500.00000000 [create] v124'434 uv434 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f917400b040 con 0x55628381baa0 2024-08-03T21:00:23.380 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.378+0000 7f91802b1f40 1 -- 172.21.15.32:0/40379518 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:78 12.15 12:aa448500:::500.00000000:head [write 0~90 in=90b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e124) -- 0x55628381e820 con 0x55628381baa0 2024-08-03T21:00:23.381 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.380+0000 7f9179e00640 1 -- 172.21.15.32:0/40379518 <== osd.6 v2:172.21.15.118:6825/801947386 10 ==== osd_op_reply(78 500.00000000 [write 0~90] v124'435 uv435 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f917400b040 con 0x55628381baa0 2024-08-03T21:00:23.381 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.380+0000 7f91802b1f40 1 -- 172.21.15.32:0/40379518 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:79 12.15 12:aa448500:::500.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e124) -- 0x55628381e820 con 0x55628381baa0 2024-08-03T21:00:23.383 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.381+0000 7f9179e00640 1 -- 172.21.15.32:0/40379518 <== osd.6 v2:172.21.15.118:6825/801947386 11 ==== osd_op_reply(79 500.00000000 [omap-set-header] v124'436 uv436 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f917400b040 con 0x55628381baa0 2024-08-03T21:00:23.383 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.381+0000 7f91802b1f40 1 --2- 172.21.15.32:0/40379518 >> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] conn(0x55628381ed30 0x5562838211d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:00:23.383 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.381+0000 7f91802b1f40 1 -- 172.21.15.32:0/40379518 --> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] -- osd_op(unknown.0.0:80 12.d 12:b50e409b:::mds_snaptable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e124) -- 0x5562838218d0 con 0x55628381ed30 2024-08-03T21:00:23.383 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.382+0000 7f9179400640 1 --2- 172.21.15.32:0/40379518 >> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] conn(0x55628381ed30 0x5562838211d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:00:23.383 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.382+0000 7f9179400640 1 --2- 172.21.15.32:0/40379518 >> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] conn(0x55628381ed30 0x5562838211d0 crc :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T21:00:23.385 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.384+0000 7f9179400640 1 -- 172.21.15.32:0/40379518 <== osd.7 v2:172.21.15.118:6818/653000048 1 ==== osd_op_reply(80 mds_snaptable [create] v124'193 uv192 ondisk = -17 ((17) File exists)) ==== 157+0+0 (crc 0 0 0) 0x7f916c01c410 con 0x55628381ed30 2024-08-03T21:00:23.386 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.384+0000 7f91802b1f40 1 -- 172.21.15.32:0/40379518 --> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] -- osd_op(unknown.0.0:81 12.d 12:b50e409b:::mds_snaptable:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e124) -- 0x556283821ca0 con 0x55628381ed30 2024-08-03T21:00:23.387 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.385+0000 7f9179400640 1 -- 172.21.15.32:0/40379518 <== osd.7 v2:172.21.15.118:6818/653000048 2 ==== osd_op_reply(81 mds_snaptable [delete] v124'194 uv194 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f916c01c410 con 0x55628381ed30 2024-08-03T21:00:23.387 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.385+0000 7f91802b1f40 1 -- 172.21.15.32:0/40379518 --> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] -- osd_op(unknown.0.0:82 12.d 12:b50e409b:::mds_snaptable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e124) -- 0x556283821ca0 con 0x55628381ed30 2024-08-03T21:00:23.388 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.387+0000 7f9179400640 1 -- 172.21.15.32:0/40379518 <== osd.7 v2:172.21.15.118:6818/653000048 3 ==== osd_op_reply(82 mds_snaptable [create] v124'195 uv195 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f916c01c410 con 0x55628381ed30 2024-08-03T21:00:23.389 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds_snaptable:head# 2024-08-03T21:00:23.389 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.387+0000 7f91802b1f40 1 -- 172.21.15.32:0/40379518 --> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] -- osd_op(unknown.0.0:83 12.d 12:b50e409b:::mds_snaptable:head [write 0~70 in=70b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e124) -- 0x556283821ca0 con 0x55628381ed30 2024-08-03T21:00:23.391 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.389+0000 7f9179400640 1 -- 172.21.15.32:0/40379518 <== osd.7 v2:172.21.15.118:6818/653000048 4 ==== osd_op_reply(83 mds_snaptable [write 0~70] v124'196 uv196 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f916c01c410 con 0x55628381ed30 2024-08-03T21:00:23.391 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.389+0000 7f91802b1f40 1 -- 172.21.15.32:0/40379518 --> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] -- osd_op(unknown.0.0:84 12.d 12:b50e409b:::mds_snaptable:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e124) -- 0x556283821ca0 con 0x55628381ed30 2024-08-03T21:00:23.392 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.390+0000 7f9179400640 1 -- 172.21.15.32:0/40379518 <== osd.7 v2:172.21.15.118:6818/653000048 5 ==== osd_op_reply(84 mds_snaptable [omap-set-header] v124'197 uv197 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f916c01c410 con 0x55628381ed30 2024-08-03T21:00:23.392 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.390+0000 7f91802b1f40 1 -- 172.21.15.32:0/40379518 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:85 12.1d 12:b871830b:::605.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e124) -- 0x556283821ca0 con 0x5562836df610 2024-08-03T21:00:23.393 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.392+0000 7f9179e00640 1 -- 172.21.15.32:0/40379518 <== osd.0 v2:172.21.15.32:6816/3066156116 44 ==== osd_op_reply(85 605.00000000 [create] v124'231 uv230 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f917400b040 con 0x5562836df610 2024-08-03T21:00:23.393 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.392+0000 7f91802b1f40 1 -- 172.21.15.32:0/40379518 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:86 12.1d 12:b871830b:::605.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e124) -- 0x556283821ca0 con 0x5562836df610 2024-08-03T21:00:23.398 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.393+0000 7f9179e00640 1 -- 172.21.15.32:0/40379518 <== osd.0 v2:172.21.15.32:6816/3066156116 45 ==== osd_op_reply(86 605.00000000 [delete] v124'232 uv232 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f917400b040 con 0x5562836df610 2024-08-03T21:00:23.398 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.393+0000 7f91802b1f40 1 -- 172.21.15.32:0/40379518 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:87 12.1d 12:b871830b:::605.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e124) -- 0x556283821ca0 con 0x5562836df610 2024-08-03T21:00:23.398 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::605.00000000:head# 2024-08-03T21:00:23.399 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.395+0000 7f9179e00640 1 -- 172.21.15.32:0/40379518 <== osd.0 v2:172.21.15.32:6816/3066156116 46 ==== osd_op_reply(87 605.00000000 [create] v124'233 uv233 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f917400b040 con 0x5562836df610 2024-08-03T21:00:23.400 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.395+0000 7f91802b1f40 1 -- 172.21.15.32:0/40379518 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:88 12.1d 12:b871830b:::605.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e124) -- 0x5562838230a0 con 0x5562836df610 2024-08-03T21:00:23.400 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.396+0000 7f9179e00640 1 -- 172.21.15.32:0/40379518 <== osd.0 v2:172.21.15.32:6816/3066156116 47 ==== osd_op_reply(88 605.00000000 [setxattr (30)] v124'234 uv234 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f917400b040 con 0x5562836df610 2024-08-03T21:00:23.400 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.396+0000 7f91802b1f40 1 -- 172.21.15.32:0/40379518 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:89 12.1d 12:b871830b:::605.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e124) -- 0x5562838230a0 con 0x5562836df610 2024-08-03T21:00:23.400 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.397+0000 7f9179e00640 1 -- 172.21.15.32:0/40379518 <== osd.0 v2:172.21.15.32:6816/3066156116 48 ==== osd_op_reply(89 605.00000000 [setxattr (62)] v124'235 uv235 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f917400b040 con 0x5562836df610 2024-08-03T21:00:23.400 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.397+0000 7f91802b1f40 1 -- 172.21.15.32:0/40379518 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:90 12.1d 12:b871830b:::605.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e124) -- 0x556283822090 con 0x5562836df610 2024-08-03T21:00:23.401 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.399+0000 7f9179e00640 1 -- 172.21.15.32:0/40379518 <== osd.0 v2:172.21.15.32:6816/3066156116 49 ==== osd_op_reply(90 605.00000000 [omap-set-header] v124'236 uv236 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f917400b040 con 0x5562836df610 2024-08-03T21:00:23.401 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.399+0000 7f91802b1f40 1 -- 172.21.15.32:0/40379518 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:91 12.13 12:c91d4a1d:::mds0_inotable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e124) -- 0x556283822090 con 0x5562838173c0 2024-08-03T21:00:23.402 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.400+0000 7f9178a00640 1 -- 172.21.15.32:0/40379518 <== osd.1 v2:172.21.15.32:6824/1551241759 5 ==== osd_op_reply(91 mds0_inotable [create] v124'386 uv380 ondisk = -17 ((17) File exists)) ==== 157+0+0 (crc 0 0 0) 0x7f916800e040 con 0x5562838173c0 2024-08-03T21:00:23.402 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.400+0000 7f91802b1f40 1 -- 172.21.15.32:0/40379518 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:92 12.13 12:c91d4a1d:::mds0_inotable:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e124) -- 0x556283822430 con 0x5562838173c0 2024-08-03T21:00:23.403 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.401+0000 7f9178a00640 1 -- 172.21.15.32:0/40379518 <== osd.1 v2:172.21.15.32:6824/1551241759 6 ==== osd_op_reply(92 mds0_inotable [delete] v124'387 uv387 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f916800e040 con 0x5562838173c0 2024-08-03T21:00:23.403 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.401+0000 7f91802b1f40 1 -- 172.21.15.32:0/40379518 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:93 12.13 12:c91d4a1d:::mds0_inotable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e124) -- 0x5562838227d0 con 0x5562838173c0 2024-08-03T21:00:23.404 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds0_inotable:head# 2024-08-03T21:00:23.404 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.403+0000 7f9178a00640 1 -- 172.21.15.32:0/40379518 <== osd.1 v2:172.21.15.32:6824/1551241759 7 ==== osd_op_reply(93 mds0_inotable [create] v124'388 uv388 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f916800e040 con 0x5562838173c0 2024-08-03T21:00:23.405 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.403+0000 7f91802b1f40 1 -- 172.21.15.32:0/40379518 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:94 12.13 12:c91d4a1d:::mds0_inotable:head [write 0~34 in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e124) -- 0x556283822b70 con 0x5562838173c0 2024-08-03T21:00:23.405 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.404+0000 7f9178a00640 1 -- 172.21.15.32:0/40379518 <== osd.1 v2:172.21.15.32:6824/1551241759 8 ==== osd_op_reply(94 mds0_inotable [write 0~34] v124'389 uv389 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f916800e040 con 0x5562838173c0 2024-08-03T21:00:23.406 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.404+0000 7f91802b1f40 1 -- 172.21.15.32:0/40379518 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:95 12.13 12:c91d4a1d:::mds0_inotable:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e124) -- 0x556283822b70 con 0x5562838173c0 2024-08-03T21:00:23.407 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.405+0000 7f9178a00640 1 -- 172.21.15.32:0/40379518 <== osd.1 v2:172.21.15.32:6824/1551241759 9 ==== osd_op_reply(95 mds0_inotable [omap-set-header] v124'390 uv390 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f916800e040 con 0x5562838173c0 2024-08-03T21:00:23.407 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.405+0000 7f91802b1f40 1 -- 172.21.15.32:0/40379518 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:96 12.13 12:cd5a64a3:::100.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e124) -- 0x556283822b70 con 0x5562838173c0 2024-08-03T21:00:23.408 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.407+0000 7f9178a00640 1 -- 172.21.15.32:0/40379518 <== osd.1 v2:172.21.15.32:6824/1551241759 10 ==== osd_op_reply(96 100.00000000 [create] v124'391 uv385 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f916800e040 con 0x5562838173c0 2024-08-03T21:00:23.408 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.407+0000 7f91802b1f40 1 -- 172.21.15.32:0/40379518 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:97 12.13 12:cd5a64a3:::100.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e124) -- 0x556283822b70 con 0x5562838173c0 2024-08-03T21:00:23.409 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.408+0000 7f9178a00640 1 -- 172.21.15.32:0/40379518 <== osd.1 v2:172.21.15.32:6824/1551241759 11 ==== osd_op_reply(97 100.00000000 [delete] v124'392 uv392 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f916800e040 con 0x5562838173c0 2024-08-03T21:00:23.410 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.408+0000 7f91802b1f40 1 -- 172.21.15.32:0/40379518 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:98 12.13 12:cd5a64a3:::100.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e124) -- 0x556283822b70 con 0x5562838173c0 2024-08-03T21:00:23.411 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.409+0000 7f9178a00640 1 -- 172.21.15.32:0/40379518 <== osd.1 v2:172.21.15.32:6824/1551241759 12 ==== osd_op_reply(98 100.00000000 [create] v124'393 uv393 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f916800e040 con 0x5562838173c0 2024-08-03T21:00:23.411 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::100.00000000:head# 2024-08-03T21:00:23.412 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.409+0000 7f91802b1f40 1 -- 172.21.15.32:0/40379518 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:99 12.13 12:cd5a64a3:::100.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e124) -- 0x556283822b70 con 0x5562838173c0 2024-08-03T21:00:23.412 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.411+0000 7f9178a00640 1 -- 172.21.15.32:0/40379518 <== osd.1 v2:172.21.15.32:6824/1551241759 13 ==== osd_op_reply(99 100.00000000 [omap-set-header] v124'394 uv394 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f916800e040 con 0x5562838173c0 2024-08-03T21:00:23.412 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.411+0000 7f91802b1f40 1 -- 172.21.15.32:0/40379518 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:100 12.13 12:cd5a64a3:::100.00000000:head [omap-set-vals in=5194b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e124) -- 0x556283825980 con 0x5562838173c0 2024-08-03T21:00:23.414 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.412+0000 7f9178a00640 1 -- 172.21.15.32:0/40379518 <== osd.1 v2:172.21.15.32:6824/1551241759 14 ==== osd_op_reply(100 100.00000000 [omap-set-vals] v124'395 uv395 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f916800e040 con 0x5562838173c0 2024-08-03T21:00:23.414 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.412+0000 7f91802b1f40 1 --2- 172.21.15.32:0/40379518 >> [v2:172.21.15.32:6800/9088303,v1:172.21.15.32:6801/9088303] conn(0x556283823530 0x556283825e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:00:23.414 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.412+0000 7f91802b1f40 1 -- 172.21.15.32:0/40379518 --> [v2:172.21.15.32:6800/9088303,v1:172.21.15.32:6801/9088303] -- osd_op(unknown.0.0:101 12.b 12:d0630e4c:::mds0_sessionmap:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e124) -- 0x556283825950 con 0x556283823530 2024-08-03T21:00:23.415 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.413+0000 7f9178a00640 1 --2- 172.21.15.32:0/40379518 >> [v2:172.21.15.32:6800/9088303,v1:172.21.15.32:6801/9088303] conn(0x556283823530 0x556283825e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:00:23.415 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.413+0000 7f9178a00640 1 --2- 172.21.15.32:0/40379518 >> [v2:172.21.15.32:6800/9088303,v1:172.21.15.32:6801/9088303] conn(0x556283823530 0x556283825e40 crc :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T21:00:23.416 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.415+0000 7f9178a00640 1 -- 172.21.15.32:0/40379518 <== osd.3 v2:172.21.15.32:6800/9088303 1 ==== osd_op_reply(101 mds0_sessionmap [create] v124'172 uv171 ondisk = -17 ((17) File exists)) ==== 159+0+0 (crc 0 0 0) 0x7f916800e040 con 0x556283823530 2024-08-03T21:00:23.416 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.415+0000 7f91802b1f40 1 -- 172.21.15.32:0/40379518 --> [v2:172.21.15.32:6800/9088303,v1:172.21.15.32:6801/9088303] -- osd_op(unknown.0.0:102 12.b 12:d0630e4c:::mds0_sessionmap:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e124) -- 0x556283825950 con 0x556283823530 2024-08-03T21:00:23.417 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.416+0000 7f9178a00640 1 -- 172.21.15.32:0/40379518 <== osd.3 v2:172.21.15.32:6800/9088303 2 ==== osd_op_reply(102 mds0_sessionmap [delete] v124'173 uv173 ondisk = 0) ==== 159+0+0 (crc 0 0 0) 0x7f916800e040 con 0x556283823530 2024-08-03T21:00:23.418 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.416+0000 7f91802b1f40 1 -- 172.21.15.32:0/40379518 --> [v2:172.21.15.32:6800/9088303,v1:172.21.15.32:6801/9088303] -- osd_op(unknown.0.0:103 12.b 12:d0630e4c:::mds0_sessionmap:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e124) -- 0x556283825950 con 0x556283823530 2024-08-03T21:00:23.419 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.417+0000 7f9178a00640 1 -- 172.21.15.32:0/40379518 <== osd.3 v2:172.21.15.32:6800/9088303 3 ==== osd_op_reply(103 mds0_sessionmap [create] v124'174 uv174 ondisk = 0) ==== 159+0+0 (crc 0 0 0) 0x7f916800e040 con 0x556283823530 2024-08-03T21:00:23.419 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds0_sessionmap:head# 2024-08-03T21:00:23.419 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.417+0000 7f91802b1f40 1 -- 172.21.15.32:0/40379518 --> [v2:172.21.15.32:6800/9088303,v1:172.21.15.32:6801/9088303] -- osd_op(unknown.0.0:104 12.b 12:d0630e4c:::mds0_sessionmap:head [omap-set-header in=14b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e124) -- 0x556283825950 con 0x556283823530 2024-08-03T21:00:23.420 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.419+0000 7f9178a00640 1 -- 172.21.15.32:0/40379518 <== osd.3 v2:172.21.15.32:6800/9088303 4 ==== osd_op_reply(104 mds0_sessionmap [omap-set-header] v124'175 uv175 ondisk = 0) ==== 159+0+0 (crc 0 0 0) 0x7f916800e040 con 0x556283823530 2024-08-03T21:00:23.420 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.419+0000 7f91802b1f40 1 -- 172.21.15.32:0/40379518 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:105 12.7 12:e0b41d06:::609.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e124) -- 0x556283825950 con 0x5562838173c0 2024-08-03T21:00:23.422 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.421+0000 7f9178a00640 1 -- 172.21.15.32:0/40379518 <== osd.1 v2:172.21.15.32:6824/1551241759 15 ==== osd_op_reply(105 609.00000000 [create] v124'231 uv230 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f916800e040 con 0x5562838173c0 2024-08-03T21:00:23.423 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.421+0000 7f91802b1f40 1 -- 172.21.15.32:0/40379518 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:106 12.7 12:e0b41d06:::609.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e124) -- 0x556283825950 con 0x5562838173c0 2024-08-03T21:00:23.423 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.422+0000 7f9178a00640 1 -- 172.21.15.32:0/40379518 <== osd.1 v2:172.21.15.32:6824/1551241759 16 ==== osd_op_reply(106 609.00000000 [delete] v124'232 uv232 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f916800e040 con 0x5562838173c0 2024-08-03T21:00:23.423 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.422+0000 7f91802b1f40 1 -- 172.21.15.32:0/40379518 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:107 12.7 12:e0b41d06:::609.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e124) -- 0x556283825950 con 0x5562838173c0 2024-08-03T21:00:23.424 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.423+0000 7f9178a00640 1 -- 172.21.15.32:0/40379518 <== osd.1 v2:172.21.15.32:6824/1551241759 17 ==== osd_op_reply(107 609.00000000 [create] v124'233 uv233 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f916800e040 con 0x5562838173c0 2024-08-03T21:00:23.425 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::609.00000000:head# 2024-08-03T21:00:23.425 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.423+0000 7f91802b1f40 1 -- 172.21.15.32:0/40379518 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:108 12.7 12:e0b41d06:::609.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e124) -- 0x556283825950 con 0x5562838173c0 2024-08-03T21:00:23.426 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.425+0000 7f9178a00640 1 -- 172.21.15.32:0/40379518 <== osd.1 v2:172.21.15.32:6824/1551241759 18 ==== osd_op_reply(108 609.00000000 [setxattr (30)] v124'234 uv234 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f916800e040 con 0x5562838173c0 2024-08-03T21:00:23.426 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.425+0000 7f91802b1f40 1 -- 172.21.15.32:0/40379518 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:109 12.7 12:e0b41d06:::609.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e124) -- 0x556283825950 con 0x5562838173c0 2024-08-03T21:00:23.427 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.426+0000 7f9178a00640 1 -- 172.21.15.32:0/40379518 <== osd.1 v2:172.21.15.32:6824/1551241759 19 ==== osd_op_reply(109 609.00000000 [setxattr (62)] v124'235 uv235 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f916800e040 con 0x5562838173c0 2024-08-03T21:00:23.427 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.426+0000 7f91802b1f40 1 -- 172.21.15.32:0/40379518 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:110 12.7 12:e0b41d06:::609.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e124) -- 0x556283825950 con 0x5562838173c0 2024-08-03T21:00:23.429 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.427+0000 7f9178a00640 1 -- 172.21.15.32:0/40379518 <== osd.1 v2:172.21.15.32:6824/1551241759 20 ==== osd_op_reply(110 609.00000000 [omap-set-header] v124'236 uv236 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f916800e040 con 0x5562838173c0 2024-08-03T21:00:23.429 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.427+0000 7f91802b1f40 1 -- 172.21.15.32:0/40379518 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:111 12.f 12:f1f69726:::400.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e124) -- 0x556283825950 con 0x55628381baa0 2024-08-03T21:00:23.430 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.430+0000 7f9179e00640 1 -- 172.21.15.32:0/40379518 <== osd.6 v2:172.21.15.118:6825/801947386 12 ==== osd_op_reply(111 400.00000000 [create] v124'196 uv195 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f917400b040 con 0x55628381baa0 2024-08-03T21:00:23.430 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.430+0000 7f91802b1f40 1 -- 172.21.15.32:0/40379518 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:112 12.f 12:f1f69726:::400.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e124) -- 0x556283825950 con 0x55628381baa0 2024-08-03T21:00:23.432 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.431+0000 7f9179e00640 1 -- 172.21.15.32:0/40379518 <== osd.6 v2:172.21.15.118:6825/801947386 13 ==== osd_op_reply(112 400.00000000 [delete] v124'197 uv197 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f917400b040 con 0x55628381baa0 2024-08-03T21:00:23.432 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.431+0000 7f91802b1f40 1 -- 172.21.15.32:0/40379518 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:113 12.f 12:f1f69726:::400.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e124) -- 0x556283825950 con 0x55628381baa0 2024-08-03T21:00:23.434 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::400.00000000:head# 2024-08-03T21:00:23.434 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.432+0000 7f9179e00640 1 -- 172.21.15.32:0/40379518 <== osd.6 v2:172.21.15.118:6825/801947386 14 ==== osd_op_reply(113 400.00000000 [create] v124'198 uv198 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f917400b040 con 0x55628381baa0 2024-08-03T21:00:23.434 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.433+0000 7f91802b1f40 1 -- 172.21.15.32:0/40379518 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:114 12.f 12:f1f69726:::400.00000000:head [write 0~22 in=22b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e124) -- 0x556283825950 con 0x55628381baa0 2024-08-03T21:00:23.436 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.434+0000 7f9179e00640 1 -- 172.21.15.32:0/40379518 <== osd.6 v2:172.21.15.118:6825/801947386 15 ==== osd_op_reply(114 400.00000000 [write 0~22] v124'199 uv199 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f917400b040 con 0x55628381baa0 2024-08-03T21:00:23.436 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.434+0000 7f91802b1f40 1 -- 172.21.15.32:0/40379518 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:115 12.f 12:f1f69726:::400.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e124) -- 0x556283825950 con 0x55628381baa0 2024-08-03T21:00:23.437 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.436+0000 7f9179e00640 1 -- 172.21.15.32:0/40379518 <== osd.6 v2:172.21.15.118:6825/801947386 16 ==== osd_op_reply(115 400.00000000 [omap-set-header] v124'200 uv200 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f917400b040 con 0x55628381baa0 2024-08-03T21:00:23.437 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.436+0000 7f91802b1f40 1 -- 172.21.15.32:0/40379518 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:116 12.1f 12:fe07bba1:::100.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e124) -- 0x556283825950 con 0x55628381baa0 2024-08-03T21:00:23.439 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.438+0000 7f9179e00640 1 -- 172.21.15.32:0/40379518 <== osd.6 v2:172.21.15.118:6825/801947386 17 ==== osd_op_reply(116 100.00000000.inode [create] v124'480 uv472 ondisk = -17 ((17) File exists)) ==== 162+0+0 (crc 0 0 0) 0x7f917400b040 con 0x55628381baa0 2024-08-03T21:00:23.439 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.438+0000 7f91802b1f40 1 -- 172.21.15.32:0/40379518 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:117 12.1f 12:fe07bba1:::100.00000000.inode:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e124) -- 0x556283825950 con 0x55628381baa0 2024-08-03T21:00:23.440 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.439+0000 7f9179e00640 1 -- 172.21.15.32:0/40379518 <== osd.6 v2:172.21.15.118:6825/801947386 18 ==== osd_op_reply(117 100.00000000.inode [delete] v124'481 uv481 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7f917400b040 con 0x55628381baa0 2024-08-03T21:00:23.441 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.439+0000 7f91802b1f40 1 -- 172.21.15.32:0/40379518 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:118 12.1f 12:fe07bba1:::100.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e124) -- 0x556283825950 con 0x55628381baa0 2024-08-03T21:00:23.442 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.441+0000 7f9179e00640 1 -- 172.21.15.32:0/40379518 <== osd.6 v2:172.21.15.118:6825/801947386 19 ==== osd_op_reply(118 100.00000000.inode [create] v124'482 uv482 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7f917400b040 con 0x55628381baa0 2024-08-03T21:00:23.442 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::100.00000000.inode:head# 2024-08-03T21:00:23.442 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.441+0000 7f91802b1f40 1 -- 172.21.15.32:0/40379518 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:119 12.1f 12:fe07bba1:::100.00000000.inode:head [write 0~582 in=582b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e124) -- 0x556283825950 con 0x55628381baa0 2024-08-03T21:00:23.443 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.442+0000 7f9179e00640 1 -- 172.21.15.32:0/40379518 <== osd.6 v2:172.21.15.118:6825/801947386 20 ==== osd_op_reply(119 100.00000000.inode [write 0~582] v124'483 uv483 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7f917400b040 con 0x55628381baa0 2024-08-03T21:00:23.444 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.442+0000 7f91802b1f40 1 -- 172.21.15.32:0/40379518 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:120 12.1f 12:fe07bba1:::100.00000000.inode:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e124) -- 0x556283825950 con 0x55628381baa0 2024-08-03T21:00:23.445 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.443+0000 7f9179e00640 1 -- 172.21.15.32:0/40379518 <== osd.6 v2:172.21.15.118:6825/801947386 21 ==== osd_op_reply(120 100.00000000.inode [omap-set-header] v124'484 uv484 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7f917400b040 con 0x55628381baa0 2024-08-03T21:00:23.445 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.443+0000 7f91802b1f40 1 -- 172.21.15.32:0/40379518 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:121 12.1f 12:ff5b34d6:::1.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e124) -- 0x556283825950 con 0x55628381baa0 2024-08-03T21:00:23.446 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.445+0000 7f9179e00640 1 -- 172.21.15.32:0/40379518 <== osd.6 v2:172.21.15.118:6825/801947386 22 ==== osd_op_reply(121 1.00000000 [create] v124'485 uv479 ondisk = -17 ((17) File exists)) ==== 154+0+0 (crc 0 0 0) 0x7f917400b040 con 0x55628381baa0 2024-08-03T21:00:23.447 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.445+0000 7f91802b1f40 1 -- 172.21.15.32:0/40379518 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:122 12.1f 12:ff5b34d6:::1.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e124) -- 0x556283825950 con 0x55628381baa0 2024-08-03T21:00:23.448 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.447+0000 7f9179e00640 1 -- 172.21.15.32:0/40379518 <== osd.6 v2:172.21.15.118:6825/801947386 23 ==== osd_op_reply(122 1.00000000 [delete] v124'486 uv486 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7f917400b040 con 0x55628381baa0 2024-08-03T21:00:23.448 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.447+0000 7f91802b1f40 1 -- 172.21.15.32:0/40379518 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:123 12.1f 12:ff5b34d6:::1.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e124) -- 0x556283825950 con 0x55628381baa0 2024-08-03T21:00:23.449 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.448+0000 7f9179e00640 1 -- 172.21.15.32:0/40379518 <== osd.6 v2:172.21.15.118:6825/801947386 24 ==== osd_op_reply(123 1.00000000 [create] v124'487 uv487 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7f917400b040 con 0x55628381baa0 2024-08-03T21:00:23.450 INFO:teuthology.orchestra.run.smithi032.stdout:***Overwrite*** #-9223372036854775808:00000000:::1.00000000:head# 2024-08-03T21:00:23.450 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.448+0000 7f91802b1f40 1 -- 172.21.15.32:0/40379518 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:124 12.1f 12:ff5b34d6:::1.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e124) -- 0x556283825950 con 0x55628381baa0 2024-08-03T21:00:23.451 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.449+0000 7f9179e00640 1 -- 172.21.15.32:0/40379518 <== osd.6 v2:172.21.15.118:6825/801947386 25 ==== osd_op_reply(124 1.00000000 [setxattr (30)] v124'488 uv488 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7f917400b040 con 0x55628381baa0 2024-08-03T21:00:23.451 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.450+0000 7f91802b1f40 1 -- 172.21.15.32:0/40379518 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:125 12.1f 12:ff5b34d6:::1.00000000:head [setxattr parent (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e124) -- 0x556283825950 con 0x55628381baa0 2024-08-03T21:00:23.452 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.451+0000 7f9179e00640 1 -- 172.21.15.32:0/40379518 <== osd.6 v2:172.21.15.118:6825/801947386 26 ==== osd_op_reply(125 1.00000000 [setxattr (30)] v124'489 uv489 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7f917400b040 con 0x55628381baa0 2024-08-03T21:00:23.452 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.451+0000 7f91802b1f40 1 -- 172.21.15.32:0/40379518 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:126 12.1f 12:ff5b34d6:::1.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e124) -- 0x556283825950 con 0x55628381baa0 2024-08-03T21:00:23.454 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.452+0000 7f9179e00640 1 -- 172.21.15.32:0/40379518 <== osd.6 v2:172.21.15.118:6825/801947386 27 ==== osd_op_reply(126 1.00000000 [omap-set-header] v124'490 uv490 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7f917400b040 con 0x55628381baa0 2024-08-03T21:00:23.454 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.452+0000 7f91802b1f40 1 -- 172.21.15.32:0/40379518 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:127 12.1f 12:ff5b34d6:::1.00000000:head [omap-set-vals in=584b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e124) -- 0x556283826830 con 0x55628381baa0 2024-08-03T21:00:23.456 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.454+0000 7f9179e00640 1 -- 172.21.15.32:0/40379518 <== osd.6 v2:172.21.15.118:6825/801947386 28 ==== osd_op_reply(127 1.00000000 [omap-set-vals] v124'491 uv491 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7f917400b040 con 0x55628381baa0 2024-08-03T21:00:23.456 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.454+0000 7f91802b1f40 1 -- 172.21.15.32:0/40379518 >> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] conn(0x5562836df610 msgr2=0x5562836dfa20 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:00:23.456 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.454+0000 7f91802b1f40 1 --2- 172.21.15.32:0/40379518 >> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] conn(0x5562836df610 0x5562836dfa20 crc :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:23.456 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.454+0000 7f91802b1f40 1 -- 172.21.15.32:0/40379518 >> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] conn(0x5562838173c0 msgr2=0x5562838197e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:00:23.456 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.454+0000 7f91802b1f40 1 --2- 172.21.15.32:0/40379518 >> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] conn(0x5562838173c0 0x5562838197e0 crc :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:23.456 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.454+0000 7f91802b1f40 1 -- 172.21.15.32:0/40379518 >> [v2:172.21.15.32:6800/9088303,v1:172.21.15.32:6801/9088303] conn(0x556283823530 msgr2=0x556283825e40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:00:23.457 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.454+0000 7f91802b1f40 1 --2- 172.21.15.32:0/40379518 >> [v2:172.21.15.32:6800/9088303,v1:172.21.15.32:6801/9088303] conn(0x556283823530 0x556283825e40 crc :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:23.457 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.454+0000 7f91802b1f40 1 -- 172.21.15.32:0/40379518 >> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] conn(0x556283812bf0 msgr2=0x556283815010 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:00:23.457 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.454+0000 7f91802b1f40 1 --2- 172.21.15.32:0/40379518 >> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] conn(0x556283812bf0 0x556283815010 crc :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:23.457 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.454+0000 7f91802b1f40 1 -- 172.21.15.32:0/40379518 >> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] conn(0x55628381baa0 msgr2=0x55628381dec0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:00:23.457 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.454+0000 7f91802b1f40 1 --2- 172.21.15.32:0/40379518 >> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] conn(0x55628381baa0 0x55628381dec0 crc :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:23.457 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.454+0000 7f91802b1f40 1 -- 172.21.15.32:0/40379518 >> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] conn(0x55628381ed30 msgr2=0x5562838211d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:00:23.457 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.454+0000 7f91802b1f40 1 --2- 172.21.15.32:0/40379518 >> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] conn(0x55628381ed30 0x5562838211d0 crc :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:23.457 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.454+0000 7f9179e00640 1 -- 172.21.15.32:0/40379518 reap_dead start 2024-08-03T21:00:23.457 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.454+0000 7f9179e00640 1 -- 172.21.15.32:0/40379518 reap_dead start 2024-08-03T21:00:23.457 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.454+0000 7f9179e00640 1 -- 172.21.15.32:0/40379518 reap_dead start 2024-08-03T21:00:23.457 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.454+0000 7f91802b1f40 1 -- 172.21.15.32:0/40379518 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f915c0759f0 msgr2=0x7f915c077f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:00:23.457 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.454+0000 7f91802b1f40 1 --2- 172.21.15.32:0/40379518 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f915c0759f0 0x7f915c077f00 secure :-1 s=READY pgs=827 cs=0 l=1 rev1=1 crypto rx=0x7f916c000c00 tx=0x7f916c002550 comp rx=0 tx=0).stop 2024-08-03T21:00:23.457 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.454+0000 7f91802b1f40 1 -- 172.21.15.32:0/40379518 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5562836dcb40 msgr2=0x556283749fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:00:23.457 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.454+0000 7f91802b1f40 1 --2- 172.21.15.32:0/40379518 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5562836dcb40 0x556283749fa0 secure :-1 s=READY pgs=2074 cs=0 l=1 rev1=1 crypto rx=0x7f916800dfa0 tx=0x7f916800b5b0 comp rx=0 tx=0).stop 2024-08-03T21:00:23.458 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.454+0000 7f91802b1f40 1 -- 172.21.15.32:0/40379518 shutdown_connections 2024-08-03T21:00:23.458 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.454+0000 7f91802b1f40 1 --2- 172.21.15.32:0/40379518 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f915c0759f0 0x7f915c077f00 unknown :-1 s=CLOSED pgs=827 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:23.458 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.454+0000 7f91802b1f40 1 --2- 172.21.15.32:0/40379518 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5562836dcb40 0x556283749fa0 unknown :-1 s=CLOSED pgs=2074 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:23.458 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.454+0000 7f91802b1f40 1 -- 172.21.15.32:0/40379518 >> 172.21.15.32:0/40379518 conn(0x5562835b7ac0 msgr2=0x5562836da680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T21:00:23.458 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.454+0000 7f91802b1f40 1 -- 172.21.15.32:0/40379518 shutdown_connections 2024-08-03T21:00:23.458 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.454+0000 7f91802b1f40 1 -- 172.21.15.32:0/40379518 wait complete. 2024-08-03T21:00:23.469 DEBUG:teuthology.orchestra.run.smithi032:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata setomapheader 1.00000000 deadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeef 2024-08-03T21:00:23.613 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.611+0000 7ff8e6bdbf40 1 Processor -- start 2024-08-03T21:00:23.613 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.611+0000 7ff8e6bdbf40 1 -- start start 2024-08-03T21:00:23.615 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.612+0000 7ff8e6bdbf40 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x558cbcfea010 0x558cbd075020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:00:23.615 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.612+0000 7ff8e6bdbf40 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558cbd074820 0x558cbcfcb480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:00:23.616 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.612+0000 7ff8e6bdbf40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x558cbcfcc1f0 0x558cbcfa2480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:00:23.616 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.612+0000 7ff8e6bdbf40 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x558cbcf4ebf0 con 0x558cbd074820 2024-08-03T21:00:23.616 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.612+0000 7ff8e6bdbf40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x558cbcf5b6b0 con 0x558cbcfcc1f0 2024-08-03T21:00:23.616 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.612+0000 7ff8e6bdbf40 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x558cbcef0e30 con 0x558cbcfea010 2024-08-03T21:00:23.616 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.613+0000 7ff8df400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558cbd074820 0x558cbcfcb480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:00:23.616 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.613+0000 7ff8df400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558cbd074820 0x558cbcfcb480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:38084/0 (socket says 172.21.15.32:38084) 2024-08-03T21:00:23.616 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.613+0000 7ff8df400640 1 -- 172.21.15.32:0/1173460837 learned_addr learned my addr 172.21.15.32:0/1173460837 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T21:00:23.616 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.613+0000 7ff8df400640 1 -- 172.21.15.32:0/1173460837 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x558cbcfea010 msgr2=0x558cbd075020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T21:00:23.616 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.613+0000 7ff8df400640 1 --2- 172.21.15.32:0/1173460837 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x558cbcfea010 0x558cbd075020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:23.616 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.613+0000 7ff8df400640 1 -- 172.21.15.32:0/1173460837 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x558cbcfcc1f0 msgr2=0x558cbcfa2480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T21:00:23.616 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.613+0000 7ff8e0800640 1 --2- 172.21.15.32:0/1173460837 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x558cbcfcc1f0 0x558cbcfa2480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:00:23.616 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.613+0000 7ff8df400640 1 --2- 172.21.15.32:0/1173460837 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x558cbcfcc1f0 0x558cbcfa2480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:23.617 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.613+0000 7ff8df400640 1 -- 172.21.15.32:0/1173460837 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558cbd0773b0 con 0x558cbd074820 2024-08-03T21:00:23.617 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.613+0000 7ff8e0800640 1 --2- 172.21.15.32:0/1173460837 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x558cbcfcc1f0 0x558cbcfa2480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-03T21:00:23.617 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.613+0000 7ff8df400640 1 --2- 172.21.15.32:0/1173460837 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558cbd074820 0x558cbcfcb480 secure :-1 s=READY pgs=2076 cs=0 l=1 rev1=1 crypto rx=0x7ff8cc046600 tx=0x7ff8cc002fb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=21783f12b509587a server_cookie=0 in_seq=0 out_seq=0 2024-08-03T21:00:23.617 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.614+0000 7ff8de000640 1 -- 172.21.15.32:0/1173460837 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8cc066070 con 0x558cbd074820 2024-08-03T21:00:23.617 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.614+0000 7ff8de000640 1 -- 172.21.15.32:0/1173460837 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7ff8cc056560 con 0x558cbd074820 2024-08-03T21:00:23.617 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.614+0000 7ff8de000640 1 -- 172.21.15.32:0/1173460837 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8cc056910 con 0x558cbd074820 2024-08-03T21:00:23.618 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.614+0000 7ff8e6bdbf40 1 -- 172.21.15.32:0/1173460837 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558cbd074820 msgr2=0x558cbcfcb480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:00:23.618 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.614+0000 7ff8e6bdbf40 1 --2- 172.21.15.32:0/1173460837 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558cbd074820 0x558cbcfcb480 secure :-1 s=READY pgs=2076 cs=0 l=1 rev1=1 crypto rx=0x7ff8cc046600 tx=0x7ff8cc002fb0 comp rx=0 tx=0).stop 2024-08-03T21:00:23.618 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.614+0000 7ff8e6bdbf40 1 -- 172.21.15.32:0/1173460837 shutdown_connections 2024-08-03T21:00:23.618 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.614+0000 7ff8e6bdbf40 1 --2- 172.21.15.32:0/1173460837 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x558cbcfcc1f0 0x558cbcfa2480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:23.618 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.614+0000 7ff8e6bdbf40 1 --2- 172.21.15.32:0/1173460837 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558cbd074820 0x558cbcfcb480 unknown :-1 s=CLOSED pgs=2076 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:23.618 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.614+0000 7ff8e6bdbf40 1 --2- 172.21.15.32:0/1173460837 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x558cbcfea010 0x558cbd075020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:23.618 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.614+0000 7ff8e6bdbf40 1 -- 172.21.15.32:0/1173460837 >> 172.21.15.32:0/1173460837 conn(0x558cbcec2ac0 msgr2=0x558cbcfeba10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T21:00:23.619 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.615+0000 7ff8e6bdbf40 1 -- 172.21.15.32:0/1173460837 shutdown_connections 2024-08-03T21:00:23.619 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.615+0000 7ff8e6bdbf40 1 -- 172.21.15.32:0/1173460837 wait complete. 2024-08-03T21:00:23.619 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.616+0000 7ff8e6bdbf40 1 Processor -- start 2024-08-03T21:00:23.619 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.616+0000 7ff8e6bdbf40 1 -- start start 2024-08-03T21:00:23.621 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.617+0000 7ff8e6bdbf40 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x558cbcfcc1f0 0x558cbd06c7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:00:23.621 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.617+0000 7ff8e6bdbf40 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558cbcfea010 0x558cbd070fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:00:23.621 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.617+0000 7ff8e6bdbf40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x558cbd074820 0x558cbd0714f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:00:23.621 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.617+0000 7ff8e6bdbf40 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x558cbd077860 con 0x558cbcfea010 2024-08-03T21:00:23.621 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.617+0000 7ff8e6bdbf40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x558cbcfec120 con 0x558cbd074820 2024-08-03T21:00:23.621 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.617+0000 7ff8e6bdbf40 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x558cbd06bdd0 con 0x558cbcfcc1f0 2024-08-03T21:00:23.621 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.617+0000 7ff8df400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558cbcfea010 0x558cbd070fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:00:23.621 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.617+0000 7ff8df400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558cbcfea010 0x558cbd070fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:38086/0 (socket says 172.21.15.32:38086) 2024-08-03T21:00:23.621 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.617+0000 7ff8df400640 1 -- 172.21.15.32:0/589113463 learned_addr learned my addr 172.21.15.32:0/589113463 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T21:00:23.621 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.618+0000 7ff8dfe00640 1 --2- 172.21.15.32:0/589113463 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x558cbcfcc1f0 0x558cbd06c7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:00:23.622 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.618+0000 7ff8e0800640 1 --2- 172.21.15.32:0/589113463 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x558cbd074820 0x558cbd0714f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:00:23.622 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.618+0000 7ff8df400640 1 -- 172.21.15.32:0/589113463 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x558cbcfcc1f0 msgr2=0x558cbd06c7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:00:23.622 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.618+0000 7ff8df400640 1 --2- 172.21.15.32:0/589113463 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x558cbcfcc1f0 0x558cbd06c7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:23.622 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.618+0000 7ff8df400640 1 -- 172.21.15.32:0/589113463 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x558cbd074820 msgr2=0x558cbd0714f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:00:23.622 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.618+0000 7ff8df400640 1 --2- 172.21.15.32:0/589113463 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x558cbd074820 0x558cbd0714f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:23.622 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.618+0000 7ff8df400640 1 -- 172.21.15.32:0/589113463 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558cbd06edb0 con 0x558cbcfea010 2024-08-03T21:00:23.622 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.618+0000 7ff8dfe00640 1 --2- 172.21.15.32:0/589113463 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x558cbcfcc1f0 0x558cbd06c7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-03T21:00:23.622 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.618+0000 7ff8df400640 1 --2- 172.21.15.32:0/589113463 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558cbcfea010 0x558cbd070fb0 secure :-1 s=READY pgs=2077 cs=0 l=1 rev1=1 crypto rx=0x7ff8cc046600 tx=0x7ff8cc056c90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T21:00:23.622 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.619+0000 7ff8dcc00640 1 -- 172.21.15.32:0/589113463 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8cc066030 con 0x558cbcfea010 2024-08-03T21:00:23.622 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.619+0000 7ff8dcc00640 1 -- 172.21.15.32:0/589113463 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7ff8cc067040 con 0x558cbcfea010 2024-08-03T21:00:23.622 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.619+0000 7ff8dcc00640 1 -- 172.21.15.32:0/589113463 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8cc06c990 con 0x558cbcfea010 2024-08-03T21:00:23.622 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.619+0000 7ff8e6bdbf40 1 -- 172.21.15.32:0/589113463 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x558cbd06f090 con 0x558cbcfea010 2024-08-03T21:00:23.623 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.619+0000 7ff8e6bdbf40 1 -- 172.21.15.32:0/589113463 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558cbd06f650 con 0x558cbcfea010 2024-08-03T21:00:23.623 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.621+0000 7ff8dcc00640 1 -- 172.21.15.32:0/589113463 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7ff8cc06cb30 con 0x558cbcfea010 2024-08-03T21:00:23.623 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.622+0000 7ff8dcc00640 1 --2- 172.21.15.32:0/589113463 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7ff8c0075a20 0x7ff8c0077f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:00:23.624 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.622+0000 7ff8dcc00640 1 -- 172.21.15.32:0/589113463 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(124..124 src has 1..124) ==== 9208+0+0 (secure 0 0 0) 0x7ff8cc0ddf30 con 0x558cbcfea010 2024-08-03T21:00:23.625 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.622+0000 7ff8dfe00640 1 --2- 172.21.15.32:0/589113463 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7ff8c0075a20 0x7ff8c0077f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:00:23.625 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.622+0000 7ff8e6bdbf40 1 --2- 172.21.15.32:0/589113463 >> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] conn(0x558cbd112950 0x558cbd112d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:00:23.625 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.622+0000 7ff8e6bdbf40 1 -- 172.21.15.32:0/589113463 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- osd_op(unknown.0.0:1 12.1f 12:ff5b34d6:::1.00000000:head [omap-set-header in=80b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e124) -- 0x558cbd11b4d0 con 0x558cbd112950 2024-08-03T21:00:23.625 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.623+0000 7ff8e0800640 1 --2- 172.21.15.32:0/589113463 >> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] conn(0x558cbd112950 0x558cbd112d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:00:23.625 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.623+0000 7ff8dfe00640 1 --2- 172.21.15.32:0/589113463 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7ff8c0075a20 0x7ff8c0077f30 secure :-1 s=READY pgs=828 cs=0 l=1 rev1=1 crypto rx=0x7ff8d000d3e0 tx=0x7ff8d0002500 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T21:00:23.625 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.623+0000 7ff8e0800640 1 --2- 172.21.15.32:0/589113463 >> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] conn(0x558cbd112950 0x558cbd112d60 crc :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T21:00:23.627 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.625+0000 7ff8e0800640 1 -- 172.21.15.32:0/589113463 <== osd.6 v2:172.21.15.118:6825/801947386 1 ==== osd_op_reply(1 1.00000000 [omap-set-header] v124'492 uv492 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7ff8d80037f0 con 0x558cbd112950 2024-08-03T21:00:23.627 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.625+0000 7ff8e6bdbf40 1 -- 172.21.15.32:0/589113463 >> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] conn(0x558cbd112950 msgr2=0x558cbd112d60 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:00:23.628 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.625+0000 7ff8e6bdbf40 1 --2- 172.21.15.32:0/589113463 >> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] conn(0x558cbd112950 0x558cbd112d60 crc :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:23.628 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.625+0000 7ff8e6bdbf40 1 -- 172.21.15.32:0/589113463 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7ff8c0075a20 msgr2=0x7ff8c0077f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:00:23.628 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.625+0000 7ff8e6bdbf40 1 --2- 172.21.15.32:0/589113463 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7ff8c0075a20 0x7ff8c0077f30 secure :-1 s=READY pgs=828 cs=0 l=1 rev1=1 crypto rx=0x7ff8d000d3e0 tx=0x7ff8d0002500 comp rx=0 tx=0).stop 2024-08-03T21:00:23.628 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.625+0000 7ff8e6bdbf40 1 -- 172.21.15.32:0/589113463 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558cbcfea010 msgr2=0x558cbd070fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:00:23.628 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.625+0000 7ff8e6bdbf40 1 --2- 172.21.15.32:0/589113463 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558cbcfea010 0x558cbd070fb0 secure :-1 s=READY pgs=2077 cs=0 l=1 rev1=1 crypto rx=0x7ff8cc046600 tx=0x7ff8cc056c90 comp rx=0 tx=0).stop 2024-08-03T21:00:23.628 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.625+0000 7ff8e0800640 1 -- 172.21.15.32:0/589113463 reap_dead start 2024-08-03T21:00:23.628 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.626+0000 7ff8e6bdbf40 1 -- 172.21.15.32:0/589113463 shutdown_connections 2024-08-03T21:00:23.628 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.626+0000 7ff8e6bdbf40 1 -- 172.21.15.32:0/589113463 >> 172.21.15.32:0/589113463 conn(0x558cbcec2ac0 msgr2=0x558cbcfcb3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T21:00:23.628 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.626+0000 7ff8e6bdbf40 1 -- 172.21.15.32:0/589113463 shutdown_connections 2024-08-03T21:00:23.628 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.626+0000 7ff8e6bdbf40 1 -- 172.21.15.32:0/589113463 wait complete. 2024-08-03T21:00:23.638 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs joinable true 2024-08-03T21:00:23.803 INFO:tasks.ceph.mds.a.smithi032.stderr: -51> 2024-08-03T21:00:19.048+0000 7f23c4e00640 -1 log_channel(cluster) log [ERR] : corrupt sessionmap header: Decoder at 'virtual void SessionMapStore::decode_header(ceph::bufferlist&)' v=1 cannot decode v=100 minimal_decoder=101: Malformed input [buffer:3] 2024-08-03T21:00:23.873 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.871+0000 7fe149400640 1 -- 172.21.15.32:0/4004569342 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fe1440cfac0 msgr2=0x7fe1440ceb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:00:23.873 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.871+0000 7fe149400640 1 --2- 172.21.15.32:0/4004569342 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fe1440cfac0 0x7fe1440ceb10 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fe12c009830 tx=0x7fe12c02a510 comp rx=0 tx=0).stop 2024-08-03T21:00:23.874 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.871+0000 7fe149400640 1 -- 172.21.15.32:0/4004569342 shutdown_connections 2024-08-03T21:00:23.874 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.871+0000 7fe149400640 1 --2- 172.21.15.32:0/4004569342 >> v2:172.21.15.32:3300/0 conn(0x7fe14415ba50 0x7fe14415be60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:23.874 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.871+0000 7fe149400640 1 --2- 172.21.15.32:0/4004569342 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fe1440cfac0 0x7fe1440ceb10 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:23.874 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.871+0000 7fe149400640 1 -- 172.21.15.32:0/4004569342 >> 172.21.15.32:0/4004569342 conn(0x7fe1440b1bd0 msgr2=0x7fe1440b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T21:00:23.874 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.871+0000 7fe149400640 1 -- 172.21.15.32:0/4004569342 shutdown_connections 2024-08-03T21:00:23.874 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.872+0000 7fe149400640 1 -- 172.21.15.32:0/4004569342 wait complete. 2024-08-03T21:00:23.875 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.874+0000 7fe149400640 1 Processor -- start 2024-08-03T21:00:23.875 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.874+0000 7fe149400640 1 -- start start 2024-08-03T21:00:23.877 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.875+0000 7fe149400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe144155920 0x7fe1440cc190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:00:23.877 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.875+0000 7fe149400640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fe14415ba50 0x7fe1441571e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:00:23.877 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.875+0000 7fe149400640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fe1441f8e70 0x7fe144157720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:00:23.878 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.875+0000 7fe149400640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fe1440b3260 con 0x7fe144155920 2024-08-03T21:00:23.878 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.875+0000 7fe149400640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7fe1440b30e0 con 0x7fe1441f8e70 2024-08-03T21:00:23.878 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.875+0000 7fe149400640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7fe1440b33e0 con 0x7fe14415ba50 2024-08-03T21:00:23.878 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.876+0000 7fe142800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe144155920 0x7fe1440cc190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:00:23.878 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.876+0000 7fe142800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe144155920 0x7fe1440cc190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:38100/0 (socket says 172.21.15.32:38100) 2024-08-03T21:00:23.878 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.876+0000 7fe142800640 1 -- 172.21.15.32:0/1818915093 learned_addr learned my addr 172.21.15.32:0/1818915093 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T21:00:23.878 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.876+0000 7fe13be00640 1 --2- 172.21.15.32:0/1818915093 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fe14415ba50 0x7fe1441571e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:00:23.878 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.876+0000 7fe143200640 1 --2- 172.21.15.32:0/1818915093 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fe1441f8e70 0x7fe144157720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:00:23.879 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.876+0000 7fe142800640 1 -- 172.21.15.32:0/1818915093 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fe14415ba50 msgr2=0x7fe1441571e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:00:23.879 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.876+0000 7fe142800640 1 --2- 172.21.15.32:0/1818915093 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fe14415ba50 0x7fe1441571e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:23.879 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.876+0000 7fe142800640 1 -- 172.21.15.32:0/1818915093 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fe1441f8e70 msgr2=0x7fe144157720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:00:23.879 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.876+0000 7fe142800640 1 --2- 172.21.15.32:0/1818915093 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fe1441f8e70 0x7fe144157720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:23.880 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.876+0000 7fe142800640 1 -- 172.21.15.32:0/1818915093 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe144156140 con 0x7fe144155920 2024-08-03T21:00:23.880 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.876+0000 7fe13be00640 1 --2- 172.21.15.32:0/1818915093 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fe14415ba50 0x7fe1441571e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-03T21:00:23.880 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.876+0000 7fe143200640 1 --2- 172.21.15.32:0/1818915093 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fe1441f8e70 0x7fe144157720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-03T21:00:23.880 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.877+0000 7fe142800640 1 --2- 172.21.15.32:0/1818915093 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe144155920 0x7fe1440cc190 secure :-1 s=READY pgs=2079 cs=0 l=1 rev1=1 crypto rx=0x7fe13400c3a0 tx=0x7fe13400c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T21:00:23.880 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.877+0000 7fe13aa00640 1 -- 172.21.15.32:0/1818915093 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe134017070 con 0x7fe144155920 2024-08-03T21:00:23.880 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.877+0000 7fe13aa00640 1 -- 172.21.15.32:0/1818915093 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fe1340053c0 con 0x7fe144155920 2024-08-03T21:00:23.880 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.877+0000 7fe13aa00640 1 -- 172.21.15.32:0/1818915093 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe134003850 con 0x7fe144155920 2024-08-03T21:00:23.880 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.877+0000 7fe149400640 1 -- 172.21.15.32:0/1818915093 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe144156420 con 0x7fe144155920 2024-08-03T21:00:23.880 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.877+0000 7fe149400640 1 -- 172.21.15.32:0/1818915093 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe1441569e0 con 0x7fe144155920 2024-08-03T21:00:23.881 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.878+0000 7fe149400640 1 -- 172.21.15.32:0/1818915093 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe1440b38c0 con 0x7fe144155920 2024-08-03T21:00:23.881 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.880+0000 7fe13aa00640 1 -- 172.21.15.32:0/1818915093 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7fe13401a070 con 0x7fe144155920 2024-08-03T21:00:23.882 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.880+0000 7fe13aa00640 1 --2- 172.21.15.32:0/1818915093 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fe120077010 0x7fe120079520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:00:23.882 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.881+0000 7fe13aa00640 1 -- 172.21.15.32:0/1818915093 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(124..124 src has 1..124) ==== 9208+0+0 (secure 0 0 0) 0x7fe134098a40 con 0x7fe144155920 2024-08-03T21:00:23.882 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.881+0000 7fe13be00640 1 --2- 172.21.15.32:0/1818915093 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fe120077010 0x7fe120079520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:00:23.883 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.882+0000 7fe13be00640 1 --2- 172.21.15.32:0/1818915093 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fe120077010 0x7fe120079520 secure :-1 s=READY pgs=829 cs=0 l=1 rev1=1 crypto rx=0x7fe12c034000 tx=0x7fe12c004730 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T21:00:23.887 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:23.885+0000 7fe13aa00640 1 -- 172.21.15.32:0/1818915093 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7fe134060880 con 0x7fe144155920 2024-08-03T21:00:23.920 INFO:tasks.ceph.mds.a.smithi032.stdout:starting mds.a at 2024-08-03T21:00:24.141 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:24.139+0000 7fe149400640 1 -- 172.21.15.32:0/1818915093 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "joinable", "val": "true"} v 0) -- 0x7fe14415cbd0 con 0x7fe144155920 2024-08-03T21:00:25.074 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:25.073+0000 7fe13aa00640 1 -- 172.21.15.32:0/1818915093 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "joinable", "val": "true"}]=0 cephfs marked joinable; MDS may join as newly active. v325) ==== 162+0+0 (secure 0 0 0) 0x7fe134098e20 con 0x7fe144155920 2024-08-03T21:00:25.074 INFO:teuthology.orchestra.run.smithi032.stderr:cephfs marked joinable; MDS may join as newly active. 2024-08-03T21:00:25.078 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:25.077+0000 7fe149400640 1 -- 172.21.15.32:0/1818915093 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fe120077010 msgr2=0x7fe120079520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:00:25.078 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:25.077+0000 7fe149400640 1 --2- 172.21.15.32:0/1818915093 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fe120077010 0x7fe120079520 secure :-1 s=READY pgs=829 cs=0 l=1 rev1=1 crypto rx=0x7fe12c034000 tx=0x7fe12c004730 comp rx=0 tx=0).stop 2024-08-03T21:00:25.078 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:25.077+0000 7fe149400640 1 -- 172.21.15.32:0/1818915093 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe144155920 msgr2=0x7fe1440cc190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:00:25.078 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:25.077+0000 7fe149400640 1 --2- 172.21.15.32:0/1818915093 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe144155920 0x7fe1440cc190 secure :-1 s=READY pgs=2079 cs=0 l=1 rev1=1 crypto rx=0x7fe13400c3a0 tx=0x7fe13400c860 comp rx=0 tx=0).stop 2024-08-03T21:00:25.078 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:25.077+0000 7fe149400640 1 -- 172.21.15.32:0/1818915093 shutdown_connections 2024-08-03T21:00:25.078 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:25.077+0000 7fe149400640 1 --2- 172.21.15.32:0/1818915093 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fe120077010 0x7fe120079520 unknown :-1 s=CLOSED pgs=829 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:25.078 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:25.077+0000 7fe149400640 1 --2- 172.21.15.32:0/1818915093 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fe1441f8e70 0x7fe144157720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:25.078 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:25.077+0000 7fe149400640 1 --2- 172.21.15.32:0/1818915093 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fe14415ba50 0x7fe1441571e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:25.079 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:25.077+0000 7fe149400640 1 --2- 172.21.15.32:0/1818915093 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe144155920 0x7fe1440cc190 unknown :-1 s=CLOSED pgs=2079 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:25.079 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:25.077+0000 7fe149400640 1 -- 172.21.15.32:0/1818915093 >> 172.21.15.32:0/1818915093 conn(0x7fe1440b1bd0 msgr2=0x7fe1440b4890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T21:00:25.079 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:25.077+0000 7fe149400640 1 -- 172.21.15.32:0/1818915093 shutdown_connections 2024-08-03T21:00:25.079 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:25.077+0000 7fe149400640 1 -- 172.21.15.32:0/1818915093 wait complete. 2024-08-03T21:00:25.111 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-08-03T21:00:25.343 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:25.342+0000 7fd755200640 1 -- 172.21.15.32:0/4039290425 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd7500ade80 msgr2=0x7fd7500ae290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:00:25.343 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:25.342+0000 7fd755200640 1 --2- 172.21.15.32:0/4039290425 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd7500ade80 0x7fd7500ae290 secure :-1 s=READY pgs=2083 cs=0 l=1 rev1=1 crypto rx=0x7fd7500b2840 tx=0x7fd74802aac0 comp rx=0 tx=0).stop 2024-08-03T21:00:25.343 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:25.342+0000 7fd755200640 1 -- 172.21.15.32:0/4039290425 shutdown_connections 2024-08-03T21:00:25.343 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:25.342+0000 7fd755200640 1 --2- 172.21.15.32:0/4039290425 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd7500ade80 0x7fd7500ae290 unknown :-1 s=CLOSED pgs=2083 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:25.344 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:25.342+0000 7fd755200640 1 --2- 172.21.15.32:0/4039290425 >> v2:172.21.15.118:3300/0 conn(0x7fd75006f720 0x7fd75006fb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:25.344 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:25.342+0000 7fd755200640 1 -- 172.21.15.32:0/4039290425 >> 172.21.15.32:0/4039290425 conn(0x7fd7500b1bd0 msgr2=0x7fd7500b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T21:00:25.344 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:25.342+0000 7fd755200640 1 -- 172.21.15.32:0/4039290425 shutdown_connections 2024-08-03T21:00:25.344 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:25.342+0000 7fd755200640 1 -- 172.21.15.32:0/4039290425 wait complete. 2024-08-03T21:00:25.345 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:25.345+0000 7fd755200640 1 Processor -- start 2024-08-03T21:00:25.345 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:25.345+0000 7fd755200640 1 -- start start 2024-08-03T21:00:25.348 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:25.346+0000 7fd755200640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fd75006f720 0x7fd75014e840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:00:25.348 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:25.346+0000 7fd755200640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fd7500ade80 0x7fd750158110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:00:25.348 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:25.346+0000 7fd755200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd7500b39a0 0x7fd750158650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:00:25.348 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:25.346+0000 7fd755200640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fd7500aeb60 con 0x7fd7500b39a0 2024-08-03T21:00:25.348 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:25.346+0000 7fd755200640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7fd7500ae9e0 con 0x7fd7500ade80 2024-08-03T21:00:25.348 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:25.346+0000 7fd755200640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7fd7500aece0 con 0x7fd75006f720 2024-08-03T21:00:25.348 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:25.346+0000 7fd74e800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd7500b39a0 0x7fd750158650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:00:25.348 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:25.347+0000 7fd74e800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd7500b39a0 0x7fd750158650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:38126/0 (socket says 172.21.15.32:38126) 2024-08-03T21:00:25.348 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:25.347+0000 7fd74e800640 1 -- 172.21.15.32:0/1620537839 learned_addr learned my addr 172.21.15.32:0/1620537839 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T21:00:25.348 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:25.347+0000 7fd74d400640 1 --2- 172.21.15.32:0/1620537839 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fd7500ade80 0x7fd750158110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:00:25.348 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:25.347+0000 7fd74de00640 1 --2- 172.21.15.32:0/1620537839 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fd75006f720 0x7fd75014e840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:00:25.348 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:25.347+0000 7fd74e800640 1 -- 172.21.15.32:0/1620537839 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fd75006f720 msgr2=0x7fd75014e840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:00:25.348 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:25.347+0000 7fd74e800640 1 --2- 172.21.15.32:0/1620537839 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fd75006f720 0x7fd75014e840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:25.349 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:25.347+0000 7fd74e800640 1 -- 172.21.15.32:0/1620537839 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fd7500ade80 msgr2=0x7fd750158110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:00:25.349 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:25.347+0000 7fd74e800640 1 --2- 172.21.15.32:0/1620537839 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fd7500ade80 0x7fd750158110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:25.349 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:25.347+0000 7fd74e800640 1 -- 172.21.15.32:0/1620537839 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd7500ce490 con 0x7fd7500b39a0 2024-08-03T21:00:25.349 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:25.347+0000 7fd74e800640 1 --2- 172.21.15.32:0/1620537839 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd7500b39a0 0x7fd750158650 secure :-1 s=READY pgs=2084 cs=0 l=1 rev1=1 crypto rx=0x7fd74802b0b0 tx=0x7fd748003ce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T21:00:25.349 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:25.348+0000 7fd746000640 1 -- 172.21.15.32:0/1620537839 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd748003e50 con 0x7fd7500b39a0 2024-08-03T21:00:25.350 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:25.348+0000 7fd746000640 1 -- 172.21.15.32:0/1620537839 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fd748004250 con 0x7fd7500b39a0 2024-08-03T21:00:25.350 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:25.348+0000 7fd746000640 1 -- 172.21.15.32:0/1620537839 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd748004680 con 0x7fd7500b39a0 2024-08-03T21:00:25.350 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:25.348+0000 7fd755200640 1 -- 172.21.15.32:0/1620537839 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd7500ce770 con 0x7fd7500b39a0 2024-08-03T21:00:25.350 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:25.348+0000 7fd755200640 1 -- 172.21.15.32:0/1620537839 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd7500ced30 con 0x7fd7500b39a0 2024-08-03T21:00:25.350 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:25.349+0000 7fd755200640 1 -- 172.21.15.32:0/1620537839 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd7500afaf0 con 0x7fd7500b39a0 2024-08-03T21:00:25.350 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:25.350+0000 7fd746000640 1 -- 172.21.15.32:0/1620537839 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7fd748049430 con 0x7fd7500b39a0 2024-08-03T21:00:25.351 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:25.351+0000 7fd746000640 1 --2- 172.21.15.32:0/1620537839 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fd7300771e0 0x7fd7300796f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:00:25.351 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:25.351+0000 7fd746000640 1 -- 172.21.15.32:0/1620537839 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(124..124 src has 1..124) ==== 9208+0+0 (secure 0 0 0) 0x7fd7480ba310 con 0x7fd7500b39a0 2024-08-03T21:00:25.352 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:25.351+0000 7fd74de00640 1 --2- 172.21.15.32:0/1620537839 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fd7300771e0 0x7fd7300796f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:00:25.352 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:25.352+0000 7fd74de00640 1 --2- 172.21.15.32:0/1620537839 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fd7300771e0 0x7fd7300796f0 secure :-1 s=READY pgs=831 cs=0 l=1 rev1=1 crypto rx=0x7fd740009b30 tx=0x7fd740005f90 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T21:00:25.356 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:25.355+0000 7fd746000640 1 -- 172.21.15.32:0/1620537839 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7fd7500afaf0 con 0x7fd7500b39a0 2024-08-03T21:00:25.600 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:25.600+0000 7fd755200640 1 -- 172.21.15.32:0/1620537839 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd7500b4e80 con 0x7fd7500b39a0 2024-08-03T21:00:25.601 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:25.600+0000 7fd746000640 1 -- 172.21.15.32:0/1620537839 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 326 v326) ==== 95+0+5229 (secure 0 0 0) 0x7fd7500b4e80 con 0x7fd7500b39a0 2024-08-03T21:00:25.601 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-03T21:00:25.601 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":326,"btime":"2024-08-03T21:00:25:078190+0000","default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6439,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6835/531080160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6834","nonce":531080160},{"type":"v1","addr":"172.21.15.118:6835","nonce":531080160}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":84},{"gid":6454,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6837/2796882319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6836","nonce":2796882319},{"type":"v1","addr":"172.21.15.118:6837","nonce":2796882319}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":86},{"gid":6813,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6834/2686933890","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":2686933890},{"type":"v1","addr":"172.21.15.32:6834","nonce":2686933890}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":85}],"filesystems":[{"mdsmap":{"epoch":326,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-08-03T20:54:26.950518+0000","modified":"2024-08-03T21:00:25.078161+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":124,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":10320},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10320":{"gid":10320,"name":"a","rank":0,"incarnation":326,"state":"up:replay","state_seq":1,"addr":"172.21.15.32:6835/1660875176","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6833","nonce":1660875176},{"type":"v1","addr":"172.21.15.32:6835","nonce":1660875176}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":6}]} 2024-08-03T21:00:25.601 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 326 2024-08-03T21:00:25.603 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:25.602+0000 7fd755200640 1 -- 172.21.15.32:0/1620537839 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fd7300771e0 msgr2=0x7fd7300796f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:00:25.603 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:25.602+0000 7fd755200640 1 --2- 172.21.15.32:0/1620537839 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fd7300771e0 0x7fd7300796f0 secure :-1 s=READY pgs=831 cs=0 l=1 rev1=1 crypto rx=0x7fd740009b30 tx=0x7fd740005f90 comp rx=0 tx=0).stop 2024-08-03T21:00:25.603 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:25.602+0000 7fd755200640 1 -- 172.21.15.32:0/1620537839 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd7500b39a0 msgr2=0x7fd750158650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:00:25.603 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:25.602+0000 7fd755200640 1 --2- 172.21.15.32:0/1620537839 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd7500b39a0 0x7fd750158650 secure :-1 s=READY pgs=2084 cs=0 l=1 rev1=1 crypto rx=0x7fd74802b0b0 tx=0x7fd748003ce0 comp rx=0 tx=0).stop 2024-08-03T21:00:25.604 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:25.602+0000 7fd755200640 1 -- 172.21.15.32:0/1620537839 shutdown_connections 2024-08-03T21:00:25.604 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:25.602+0000 7fd755200640 1 --2- 172.21.15.32:0/1620537839 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fd7300771e0 0x7fd7300796f0 unknown :-1 s=CLOSED pgs=831 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:25.604 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:25.602+0000 7fd755200640 1 --2- 172.21.15.32:0/1620537839 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fd7500ade80 0x7fd750158110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:25.604 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:25.602+0000 7fd755200640 1 --2- 172.21.15.32:0/1620537839 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd7500b39a0 0x7fd750158650 unknown :-1 s=CLOSED pgs=2084 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:25.604 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:25.602+0000 7fd755200640 1 --2- 172.21.15.32:0/1620537839 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fd75006f720 0x7fd75014e840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:25.604 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:25.602+0000 7fd755200640 1 -- 172.21.15.32:0/1620537839 >> 172.21.15.32:0/1620537839 conn(0x7fd7500b1bd0 msgr2=0x7fd7500b2ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T21:00:25.604 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:25.603+0000 7fd755200640 1 -- 172.21.15.32:0/1620537839 shutdown_connections 2024-08-03T21:00:25.604 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:25.603+0000 7fd755200640 1 -- 172.21.15.32:0/1620537839 wait complete. 2024-08-03T21:00:25.625 DEBUG:tasks.ceph_test_case:wait_until_true: waiting (timeout=60 retry_count=0)... 2024-08-03T21:00:28.099 INFO:tasks.ceph.mds.a.smithi032.stderr:2024-08-03T21:00:28.098+0000 7f0a6ce00640 -1 mds.0.cache.dir(0x1) Corrupt fnode in dirfrag 0x1: Decoder at 'void fnode_t::decode(ceph::buffer::v15_2_0::list::const_iterator&)' v=3 cannot decode v=100 minimal_decoder=101: Malformed input [buffer:3] 2024-08-03T21:00:28.099 INFO:tasks.ceph.mds.a.smithi032.stderr:2024-08-03T21:00:28.098+0000 7f0a6ce00640 -1 mds.0.damage notify_dirfrag Damage to fragment * of ino 0x1 is fatal because it is a system directory for this rank 2024-08-03T21:00:28.100 INFO:tasks.ceph.mon.a.smithi032.stderr:2024-08-03T21:00:28.099+0000 7f4e6d600640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-08-03T21:00:28.100 INFO:tasks.ceph.mon.a.smithi032.stderr:2024-08-03T21:00:28.099+0000 7f4e6d600640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 mds daemon damaged (MDS_DAMAGE) 2024-08-03T21:00:29.071 INFO:tasks.ceph.mds.a.smithi032.stderr:2024-08-03T21:00:29.070+0000 7f0a71e00640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-08-03T21:00:30.626 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-08-03T21:00:30.867 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:30.866+0000 7feadbc00640 1 -- 172.21.15.32:0/2228841141 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fead4144f90 msgr2=0x7fead4074570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:00:30.867 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:30.866+0000 7feadbc00640 1 --2- 172.21.15.32:0/2228841141 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fead4144f90 0x7fead4074570 secure :-1 s=READY pgs=2086 cs=0 l=1 rev1=1 crypto rx=0x7fead4073b50 tx=0x7feac002a510 comp rx=0 tx=0).stop 2024-08-03T21:00:30.867 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:30.866+0000 7feadbc00640 1 -- 172.21.15.32:0/2228841141 shutdown_connections 2024-08-03T21:00:30.867 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:30.866+0000 7feadbc00640 1 --2- 172.21.15.32:0/2228841141 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fead4144f90 0x7fead4074570 unknown :-1 s=CLOSED pgs=2086 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:30.867 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:30.866+0000 7feadbc00640 1 -- 172.21.15.32:0/2228841141 >> 172.21.15.32:0/2228841141 conn(0x7fead40b1bd0 msgr2=0x7fead40b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T21:00:30.867 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:30.866+0000 7feadbc00640 1 -- 172.21.15.32:0/2228841141 shutdown_connections 2024-08-03T21:00:30.867 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:30.866+0000 7feadbc00640 1 -- 172.21.15.32:0/2228841141 wait complete. 2024-08-03T21:00:30.869 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:30.868+0000 7feadbc00640 1 Processor -- start 2024-08-03T21:00:30.869 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:30.868+0000 7feadbc00640 1 -- start start 2024-08-03T21:00:30.871 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:30.870+0000 7feadbc00640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fead406e930 0x7fead415d430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:00:30.872 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:30.870+0000 7feadbc00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fead4144f90 0x7fead4151810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:00:30.872 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:30.870+0000 7feadbc00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fead415fff0 0x7fead41f5340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:00:30.872 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:30.870+0000 7feadbc00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fead40cf840 con 0x7fead4144f90 2024-08-03T21:00:30.872 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:30.870+0000 7feadbc00640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7fead4145f30 con 0x7fead415fff0 2024-08-03T21:00:30.872 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:30.870+0000 7feadbc00640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7fead40cf9c0 con 0x7fead406e930 2024-08-03T21:00:30.872 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:30.870+0000 7fead3e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fead4144f90 0x7fead4151810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:00:30.872 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:30.870+0000 7fead3e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fead4144f90 0x7fead4151810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:52694/0 (socket says 172.21.15.32:52694) 2024-08-03T21:00:30.872 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:30.870+0000 7fead3e00640 1 -- 172.21.15.32:0/548945480 learned_addr learned my addr 172.21.15.32:0/548945480 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T21:00:30.872 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:30.870+0000 7fead9200640 1 --2- 172.21.15.32:0/548945480 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fead406e930 0x7fead415d430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:00:30.872 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:30.870+0000 7fead9c00640 1 --2- 172.21.15.32:0/548945480 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fead415fff0 0x7fead41f5340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:00:30.872 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:30.871+0000 7fead3e00640 1 -- 172.21.15.32:0/548945480 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fead406e930 msgr2=0x7fead415d430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:00:30.873 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:30.871+0000 7fead3e00640 1 --2- 172.21.15.32:0/548945480 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fead406e930 0x7fead415d430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:30.873 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:30.871+0000 7fead3e00640 1 -- 172.21.15.32:0/548945480 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fead415fff0 msgr2=0x7fead41f5340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:00:30.873 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:30.871+0000 7fead3e00640 1 --2- 172.21.15.32:0/548945480 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fead415fff0 0x7fead41f5340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:30.873 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:30.871+0000 7fead3e00640 1 -- 172.21.15.32:0/548945480 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fead413d530 con 0x7fead4144f90 2024-08-03T21:00:30.873 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:30.871+0000 7fead3e00640 1 --2- 172.21.15.32:0/548945480 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fead4144f90 0x7fead4151810 secure :-1 s=READY pgs=2087 cs=0 l=1 rev1=1 crypto rx=0x7feac00098f0 tx=0x7feac0033c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T21:00:30.874 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:30.871+0000 7fead0c00640 1 -- 172.21.15.32:0/548945480 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feac0005c90 con 0x7fead4144f90 2024-08-03T21:00:30.874 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:30.871+0000 7fead0c00640 1 -- 172.21.15.32:0/548945480 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7feac0031e50 con 0x7fead4144f90 2024-08-03T21:00:30.874 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:30.871+0000 7feadbc00640 1 -- 172.21.15.32:0/548945480 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fead413d810 con 0x7fead4144f90 2024-08-03T21:00:30.874 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:30.872+0000 7fead0c00640 1 -- 172.21.15.32:0/548945480 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feac0005260 con 0x7fead4144f90 2024-08-03T21:00:30.874 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:30.872+0000 7feadbc00640 1 -- 172.21.15.32:0/548945480 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fead41512f0 con 0x7fead4144f90 2024-08-03T21:00:30.874 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:30.873+0000 7feadbc00640 1 -- 172.21.15.32:0/548945480 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fead4160400 con 0x7fead4144f90 2024-08-03T21:00:30.874 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:30.874+0000 7fead0c00640 1 -- 172.21.15.32:0/548945480 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7feac0005400 con 0x7fead4144f90 2024-08-03T21:00:30.875 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:30.875+0000 7fead0c00640 1 --2- 172.21.15.32:0/548945480 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7feab4077040 0x7feab4079550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:00:30.875 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:30.875+0000 7fead0c00640 1 -- 172.21.15.32:0/548945480 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(125..125 src has 1..125) ==== 9294+0+0 (secure 0 0 0) 0x7feac00ba6d0 con 0x7fead4144f90 2024-08-03T21:00:30.876 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:30.875+0000 7fead9200640 1 --2- 172.21.15.32:0/548945480 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7feab4077040 0x7feab4079550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:00:30.876 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:30.876+0000 7fead9200640 1 --2- 172.21.15.32:0/548945480 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7feab4077040 0x7feab4079550 secure :-1 s=READY pgs=832 cs=0 l=1 rev1=1 crypto rx=0x7feac4007fd0 tx=0x7feac4001370 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T21:00:30.880 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:30.879+0000 7fead0c00640 1 -- 172.21.15.32:0/548945480 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7feac003f030 con 0x7fead4144f90 2024-08-03T21:00:31.127 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:31.126+0000 7feadbc00640 1 -- 172.21.15.32:0/548945480 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fead415da50 con 0x7fead4144f90 2024-08-03T21:00:31.127 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:31.127+0000 7fead0c00640 1 -- 172.21.15.32:0/548945480 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 330 v330) ==== 95+0+4391 (secure 0 0 0) 0x7feac0082460 con 0x7fead4144f90 2024-08-03T21:00:31.128 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-03T21:00:31.128 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":330,"btime":"2024-08-03T21:00:28:099623+0000","default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6439,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6835/531080160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6834","nonce":531080160},{"type":"v1","addr":"172.21.15.118:6835","nonce":531080160}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":84},{"gid":6454,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6837/2796882319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6836","nonce":2796882319},{"type":"v1","addr":"172.21.15.118:6837","nonce":2796882319}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":86},{"gid":6813,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6834/2686933890","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":2686933890},{"type":"v1","addr":"172.21.15.32:6834","nonce":2686933890}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":85}],"filesystems":[{"mdsmap":{"epoch":330,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-08-03T20:54:26.950518+0000","modified":"2024-08-03T21:00:28.099619+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":125,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":6}]} 2024-08-03T21:00:31.128 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 330 2024-08-03T21:00:31.130 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:31.129+0000 7feadbc00640 1 -- 172.21.15.32:0/548945480 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7feab4077040 msgr2=0x7feab4079550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:00:31.130 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:31.129+0000 7feadbc00640 1 --2- 172.21.15.32:0/548945480 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7feab4077040 0x7feab4079550 secure :-1 s=READY pgs=832 cs=0 l=1 rev1=1 crypto rx=0x7feac4007fd0 tx=0x7feac4001370 comp rx=0 tx=0).stop 2024-08-03T21:00:31.131 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:31.129+0000 7feadbc00640 1 -- 172.21.15.32:0/548945480 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fead4144f90 msgr2=0x7fead4151810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:00:31.131 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:31.129+0000 7feadbc00640 1 --2- 172.21.15.32:0/548945480 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fead4144f90 0x7fead4151810 secure :-1 s=READY pgs=2087 cs=0 l=1 rev1=1 crypto rx=0x7feac00098f0 tx=0x7feac0033c20 comp rx=0 tx=0).stop 2024-08-03T21:00:31.131 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:31.129+0000 7feadbc00640 1 -- 172.21.15.32:0/548945480 shutdown_connections 2024-08-03T21:00:31.131 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:31.129+0000 7feadbc00640 1 --2- 172.21.15.32:0/548945480 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7feab4077040 0x7feab4079550 unknown :-1 s=CLOSED pgs=832 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:31.131 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:31.129+0000 7feadbc00640 1 --2- 172.21.15.32:0/548945480 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fead415fff0 0x7fead41f5340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:31.131 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:31.129+0000 7feadbc00640 1 --2- 172.21.15.32:0/548945480 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fead4144f90 0x7fead4151810 unknown :-1 s=CLOSED pgs=2087 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:31.131 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:31.129+0000 7feadbc00640 1 --2- 172.21.15.32:0/548945480 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fead406e930 0x7fead415d430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:31.131 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:31.129+0000 7feadbc00640 1 -- 172.21.15.32:0/548945480 >> 172.21.15.32:0/548945480 conn(0x7fead40b1bd0 msgr2=0x7fead40b0320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T21:00:31.131 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:31.129+0000 7feadbc00640 1 -- 172.21.15.32:0/548945480 shutdown_connections 2024-08-03T21:00:31.131 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:31.129+0000 7feadbc00640 1 -- 172.21.15.32:0/548945480 wait complete. 2024-08-03T21:00:31.156 DEBUG:tasks.ceph_test_case:wait_until_true: success in 5s and 0 retries 2024-08-03T21:00:31.156 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-08-03T21:00:31.385 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:31.384+0000 7fd92ae00640 1 -- 172.21.15.32:0/2579735931 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd9240d0ff0 msgr2=0x7fd9240b73a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:00:31.385 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:31.384+0000 7fd92ae00640 1 --2- 172.21.15.32:0/2579735931 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd9240d0ff0 0x7fd9240b73a0 secure :-1 s=READY pgs=2088 cs=0 l=1 rev1=1 crypto rx=0x7fd9240d4ff0 tx=0x7fd91c02a980 comp rx=0 tx=0).stop 2024-08-03T21:00:31.385 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:31.384+0000 7fd92ae00640 1 -- 172.21.15.32:0/2579735931 shutdown_connections 2024-08-03T21:00:31.385 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:31.384+0000 7fd92ae00640 1 --2- 172.21.15.32:0/2579735931 >> v2:172.21.15.118:3300/0 conn(0x7fd9240b94f0 0x7fd9240d0ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:31.385 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:31.384+0000 7fd92ae00640 1 --2- 172.21.15.32:0/2579735931 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd9240d0ff0 0x7fd9240b73a0 unknown :-1 s=CLOSED pgs=2088 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:31.385 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:31.384+0000 7fd92ae00640 1 -- 172.21.15.32:0/2579735931 >> 172.21.15.32:0/2579735931 conn(0x7fd9240d40c0 msgr2=0x7fd9240d44d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T21:00:31.385 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:31.384+0000 7fd92ae00640 1 -- 172.21.15.32:0/2579735931 shutdown_connections 2024-08-03T21:00:31.385 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:31.384+0000 7fd92ae00640 1 -- 172.21.15.32:0/2579735931 wait complete. 2024-08-03T21:00:31.387 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:31.386+0000 7fd92ae00640 1 Processor -- start 2024-08-03T21:00:31.387 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:31.386+0000 7fd92ae00640 1 -- start start 2024-08-03T21:00:31.389 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:31.388+0000 7fd92ae00640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fd9240b94f0 0x7fd9240b4620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:00:31.389 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:31.388+0000 7fd92ae00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd9240d0ff0 0x7fd924158320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:00:31.389 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:31.388+0000 7fd92ae00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fd9240d4be0 0x7fd924158860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:00:31.390 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:31.388+0000 7fd92ae00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fd9240b7c70 con 0x7fd9240d0ff0 2024-08-03T21:00:31.390 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:31.388+0000 7fd92ae00640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7fd9240b7af0 con 0x7fd9240d4be0 2024-08-03T21:00:31.390 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:31.388+0000 7fd92ae00640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7fd9240b7df0 con 0x7fd9240b94f0 2024-08-03T21:00:31.390 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:31.388+0000 7fd929000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd9240d0ff0 0x7fd924158320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:00:31.390 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:31.388+0000 7fd929000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd9240d0ff0 0x7fd924158320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:52722/0 (socket says 172.21.15.32:52722) 2024-08-03T21:00:31.390 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:31.388+0000 7fd929000640 1 -- 172.21.15.32:0/68980944 learned_addr learned my addr 172.21.15.32:0/68980944 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T21:00:31.390 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:31.388+0000 7fd92a400640 1 --2- 172.21.15.32:0/68980944 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fd9240d4be0 0x7fd924158860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:00:31.390 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:31.388+0000 7fd929a00640 1 --2- 172.21.15.32:0/68980944 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fd9240b94f0 0x7fd9240b4620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:00:31.390 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:31.389+0000 7fd929000640 1 -- 172.21.15.32:0/68980944 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fd9240b94f0 msgr2=0x7fd9240b4620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:00:31.390 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:31.389+0000 7fd929000640 1 --2- 172.21.15.32:0/68980944 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fd9240b94f0 0x7fd9240b4620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:31.391 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:31.389+0000 7fd929000640 1 -- 172.21.15.32:0/68980944 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fd9240d4be0 msgr2=0x7fd924158860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:00:31.391 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:31.389+0000 7fd929000640 1 --2- 172.21.15.32:0/68980944 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fd9240d4be0 0x7fd924158860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:31.391 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:31.389+0000 7fd929000640 1 -- 172.21.15.32:0/68980944 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd9241f5a60 con 0x7fd9240d0ff0 2024-08-03T21:00:31.391 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:31.389+0000 7fd929000640 1 --2- 172.21.15.32:0/68980944 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd9240d0ff0 0x7fd924158320 secure :-1 s=READY pgs=2089 cs=0 l=1 rev1=1 crypto rx=0x7fd91000d6d0 tx=0x7fd91000db90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T21:00:31.392 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:31.389+0000 7fd921600640 1 -- 172.21.15.32:0/68980944 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd910017070 con 0x7fd9240d0ff0 2024-08-03T21:00:31.392 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:31.389+0000 7fd921600640 1 -- 172.21.15.32:0/68980944 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fd910005330 con 0x7fd9240d0ff0 2024-08-03T21:00:31.392 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:31.389+0000 7fd921600640 1 -- 172.21.15.32:0/68980944 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd910009820 con 0x7fd9240d0ff0 2024-08-03T21:00:31.392 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:31.389+0000 7fd92ae00640 1 -- 172.21.15.32:0/68980944 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd9241f5d40 con 0x7fd9240d0ff0 2024-08-03T21:00:31.392 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:31.389+0000 7fd92ae00640 1 -- 172.21.15.32:0/68980944 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd9241f6300 con 0x7fd9240d0ff0 2024-08-03T21:00:31.392 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:31.390+0000 7fd92ae00640 1 -- 172.21.15.32:0/68980944 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd9240d0ab0 con 0x7fd9240d0ff0 2024-08-03T21:00:31.392 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:31.392+0000 7fd921600640 1 -- 172.21.15.32:0/68980944 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7fd910009bd0 con 0x7fd9240d0ff0 2024-08-03T21:00:31.393 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:31.393+0000 7fd921600640 1 --2- 172.21.15.32:0/68980944 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fd904077010 0x7fd904079520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:00:31.393 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:31.393+0000 7fd921600640 1 -- 172.21.15.32:0/68980944 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(125..125 src has 1..125) ==== 9294+0+0 (secure 0 0 0) 0x7fd910099990 con 0x7fd9240d0ff0 2024-08-03T21:00:31.394 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:31.393+0000 7fd929a00640 1 --2- 172.21.15.32:0/68980944 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fd904077010 0x7fd904079520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:00:31.394 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:31.394+0000 7fd929a00640 1 --2- 172.21.15.32:0/68980944 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fd904077010 0x7fd904079520 secure :-1 s=READY pgs=833 cs=0 l=1 rev1=1 crypto rx=0x7fd9180098c0 tx=0x7fd918009060 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T21:00:31.398 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:31.398+0000 7fd921600640 1 -- 172.21.15.32:0/68980944 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7fd910061690 con 0x7fd9240d0ff0 2024-08-03T21:00:31.645 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:31.645+0000 7fd92ae00640 1 -- 172.21.15.32:0/68980944 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd92406dff0 con 0x7fd9240d0ff0 2024-08-03T21:00:31.646 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:31.645+0000 7fd921600640 1 -- 172.21.15.32:0/68980944 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 330 v330) ==== 95+0+4391 (secure 0 0 0) 0x7fd910065be0 con 0x7fd9240d0ff0 2024-08-03T21:00:31.646 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-03T21:00:31.646 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":330,"btime":"2024-08-03T21:00:28:099623+0000","default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6439,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6835/531080160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6834","nonce":531080160},{"type":"v1","addr":"172.21.15.118:6835","nonce":531080160}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":84},{"gid":6454,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6837/2796882319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6836","nonce":2796882319},{"type":"v1","addr":"172.21.15.118:6837","nonce":2796882319}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":86},{"gid":6813,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6834/2686933890","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":2686933890},{"type":"v1","addr":"172.21.15.32:6834","nonce":2686933890}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":85}],"filesystems":[{"mdsmap":{"epoch":330,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-08-03T20:54:26.950518+0000","modified":"2024-08-03T21:00:28.099619+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":125,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":6}]} 2024-08-03T21:00:31.647 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 330 2024-08-03T21:00:31.649 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:31.648+0000 7fd92ae00640 1 -- 172.21.15.32:0/68980944 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fd904077010 msgr2=0x7fd904079520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:00:31.649 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:31.648+0000 7fd92ae00640 1 --2- 172.21.15.32:0/68980944 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fd904077010 0x7fd904079520 secure :-1 s=READY pgs=833 cs=0 l=1 rev1=1 crypto rx=0x7fd9180098c0 tx=0x7fd918009060 comp rx=0 tx=0).stop 2024-08-03T21:00:31.649 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:31.648+0000 7fd92ae00640 1 -- 172.21.15.32:0/68980944 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd9240d0ff0 msgr2=0x7fd924158320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:00:31.649 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:31.648+0000 7fd92ae00640 1 --2- 172.21.15.32:0/68980944 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd9240d0ff0 0x7fd924158320 secure :-1 s=READY pgs=2089 cs=0 l=1 rev1=1 crypto rx=0x7fd91000d6d0 tx=0x7fd91000db90 comp rx=0 tx=0).stop 2024-08-03T21:00:31.649 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:31.648+0000 7fd92ae00640 1 -- 172.21.15.32:0/68980944 shutdown_connections 2024-08-03T21:00:31.649 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:31.648+0000 7fd92ae00640 1 --2- 172.21.15.32:0/68980944 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fd904077010 0x7fd904079520 unknown :-1 s=CLOSED pgs=833 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:31.649 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:31.648+0000 7fd92ae00640 1 --2- 172.21.15.32:0/68980944 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fd9240d4be0 0x7fd924158860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:31.649 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:31.648+0000 7fd92ae00640 1 --2- 172.21.15.32:0/68980944 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd9240d0ff0 0x7fd924158320 unknown :-1 s=CLOSED pgs=2089 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:31.649 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:31.648+0000 7fd92ae00640 1 --2- 172.21.15.32:0/68980944 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fd9240b94f0 0x7fd9240b4620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:31.649 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:31.648+0000 7fd92ae00640 1 -- 172.21.15.32:0/68980944 >> 172.21.15.32:0/68980944 conn(0x7fd9240d40c0 msgr2=0x7fd9240b5bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T21:00:31.649 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:31.648+0000 7fd92ae00640 1 -- 172.21.15.32:0/68980944 shutdown_connections 2024-08-03T21:00:31.649 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:31.648+0000 7fd92ae00640 1 -- 172.21.15.32:0/68980944 wait complete. 2024-08-03T21:00:31.676 DEBUG:tasks.ceph_test_case:wait_until_true: success in 0s and 0 retries 2024-08-03T21:00:31.676 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-08-03T21:00:31.906 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:31.905+0000 7f00f2c00640 1 -- 172.21.15.32:0/807392174 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f00ec0d4bb0 msgr2=0x7f00ec0d1c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:00:31.906 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:31.905+0000 7f00f2c00640 1 --2- 172.21.15.32:0/807392174 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f00ec0d4bb0 0x7f00ec0d1c60 secure :-1 s=READY pgs=2091 cs=0 l=1 rev1=1 crypto rx=0x7f00e0009830 tx=0x7f00e002a480 comp rx=0 tx=0).stop 2024-08-03T21:00:31.906 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:31.905+0000 7f00f2c00640 1 -- 172.21.15.32:0/807392174 shutdown_connections 2024-08-03T21:00:31.906 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:31.905+0000 7f00f2c00640 1 --2- 172.21.15.32:0/807392174 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f00ec0cec30 0x7f00ec0cf040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:31.906 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:31.905+0000 7f00f2c00640 1 --2- 172.21.15.32:0/807392174 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f00ec0d4bb0 0x7f00ec0d1c60 unknown :-1 s=CLOSED pgs=2091 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:31.906 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:31.905+0000 7f00f2c00640 1 -- 172.21.15.32:0/807392174 >> 172.21.15.32:0/807392174 conn(0x7f00ec0d4090 msgr2=0x7f00ec0d44a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T21:00:31.906 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:31.905+0000 7f00f2c00640 1 -- 172.21.15.32:0/807392174 shutdown_connections 2024-08-03T21:00:31.906 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:31.905+0000 7f00f2c00640 1 -- 172.21.15.32:0/807392174 wait complete. 2024-08-03T21:00:31.908 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:31.907+0000 7f00f2c00640 1 Processor -- start 2024-08-03T21:00:31.908 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:31.908+0000 7f00f2c00640 1 -- start start 2024-08-03T21:00:31.911 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:31.909+0000 7f00f2c00640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f00ec0cec30 0x7f00ec15c690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:00:31.911 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:31.909+0000 7f00f2c00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f00ec0d2290 0x7f00ec0b7260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:00:31.911 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:31.909+0000 7f00f2c00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f00ec0d4bb0 0x7f00ec0b77a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:00:31.911 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:31.909+0000 7f00f2c00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f00ec0cfad0 con 0x7f00ec0d2290 2024-08-03T21:00:31.911 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:31.909+0000 7f00f2c00640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f00ec0cf950 con 0x7f00ec0d4bb0 2024-08-03T21:00:31.912 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:31.909+0000 7f00f2c00640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7f00ec0cfc50 con 0x7f00ec0cec30 2024-08-03T21:00:31.912 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:31.909+0000 7f00f0e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f00ec0d2290 0x7f00ec0b7260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:00:31.912 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:31.909+0000 7f00f0e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f00ec0d2290 0x7f00ec0b7260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:52740/0 (socket says 172.21.15.32:52740) 2024-08-03T21:00:31.912 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:31.909+0000 7f00f0e00640 1 -- 172.21.15.32:0/2565263327 learned_addr learned my addr 172.21.15.32:0/2565263327 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T21:00:31.912 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:31.909+0000 7f00f1800640 1 --2- 172.21.15.32:0/2565263327 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f00ec0cec30 0x7f00ec15c690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:00:31.912 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:31.909+0000 7f00f2200640 1 --2- 172.21.15.32:0/2565263327 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f00ec0d4bb0 0x7f00ec0b77a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:00:31.912 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:31.910+0000 7f00f0e00640 1 -- 172.21.15.32:0/2565263327 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f00ec0cec30 msgr2=0x7f00ec15c690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:00:31.912 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:31.910+0000 7f00f0e00640 1 --2- 172.21.15.32:0/2565263327 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f00ec0cec30 0x7f00ec15c690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:31.912 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:31.910+0000 7f00f0e00640 1 -- 172.21.15.32:0/2565263327 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f00ec0d4bb0 msgr2=0x7f00ec0b77a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:00:31.912 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:31.910+0000 7f00f0e00640 1 --2- 172.21.15.32:0/2565263327 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f00ec0d4bb0 0x7f00ec0b77a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:31.912 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:31.910+0000 7f00f0e00640 1 -- 172.21.15.32:0/2565263327 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f00ec166920 con 0x7f00ec0d2290 2024-08-03T21:00:31.912 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:31.910+0000 7f00f0e00640 1 --2- 172.21.15.32:0/2565263327 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f00ec0d2290 0x7f00ec0b7260 secure :-1 s=READY pgs=2092 cs=0 l=1 rev1=1 crypto rx=0x7f00d800c3f0 tx=0x7f00d800c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T21:00:31.912 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:31.910+0000 7f00e9600640 1 -- 172.21.15.32:0/2565263327 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00d8017070 con 0x7f00ec0d2290 2024-08-03T21:00:31.913 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:31.910+0000 7f00e9600640 1 -- 172.21.15.32:0/2565263327 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f00d80035f0 con 0x7f00ec0d2290 2024-08-03T21:00:31.913 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:31.910+0000 7f00e9600640 1 -- 172.21.15.32:0/2565263327 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00d8003a40 con 0x7f00ec0d2290 2024-08-03T21:00:31.913 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:31.910+0000 7f00f2c00640 1 -- 172.21.15.32:0/2565263327 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f00ec166be0 con 0x7f00ec0d2290 2024-08-03T21:00:31.913 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:31.911+0000 7f00f2c00640 1 -- 172.21.15.32:0/2565263327 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f00ec1671a0 con 0x7f00ec0d2290 2024-08-03T21:00:31.913 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:31.912+0000 7f00f2c00640 1 -- 172.21.15.32:0/2565263327 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f00ec0cf040 con 0x7f00ec0d2290 2024-08-03T21:00:31.914 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:31.913+0000 7f00e9600640 1 -- 172.21.15.32:0/2565263327 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f00d801a070 con 0x7f00ec0d2290 2024-08-03T21:00:31.914 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:31.914+0000 7f00e9600640 1 --2- 172.21.15.32:0/2565263327 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f00cc077010 0x7f00cc079520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:00:31.914 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:31.914+0000 7f00e9600640 1 -- 172.21.15.32:0/2565263327 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(125..125 src has 1..125) ==== 9294+0+0 (secure 0 0 0) 0x7f00d80989a0 con 0x7f00ec0d2290 2024-08-03T21:00:31.915 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:31.914+0000 7f00f1800640 1 --2- 172.21.15.32:0/2565263327 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f00cc077010 0x7f00cc079520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:00:31.915 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:31.915+0000 7f00f1800640 1 --2- 172.21.15.32:0/2565263327 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f00cc077010 0x7f00cc079520 secure :-1 s=READY pgs=834 cs=0 l=1 rev1=1 crypto rx=0x7f00e0034000 tx=0x7f00e0004730 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T21:00:31.919 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:31.918+0000 7f00e9600640 1 -- 172.21.15.32:0/2565263327 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7f00d8060790 con 0x7f00ec0d2290 2024-08-03T21:00:32.168 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:32.168+0000 7f00f2c00640 1 -- 172.21.15.32:0/2565263327 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f00ec08c100 con 0x7f00ec0d2290 2024-08-03T21:00:32.169 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:32.169+0000 7f00e9600640 1 -- 172.21.15.32:0/2565263327 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 330 v330) ==== 95+0+4391 (secure 0 0 0) 0x7f00d8064ce0 con 0x7f00ec0d2290 2024-08-03T21:00:32.169 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-03T21:00:32.169 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":330,"btime":"2024-08-03T21:00:28:099623+0000","default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6439,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6835/531080160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6834","nonce":531080160},{"type":"v1","addr":"172.21.15.118:6835","nonce":531080160}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":84},{"gid":6454,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6837/2796882319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6836","nonce":2796882319},{"type":"v1","addr":"172.21.15.118:6837","nonce":2796882319}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":86},{"gid":6813,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6834/2686933890","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":2686933890},{"type":"v1","addr":"172.21.15.32:6834","nonce":2686933890}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":85}],"filesystems":[{"mdsmap":{"epoch":330,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-08-03T20:54:26.950518+0000","modified":"2024-08-03T21:00:28.099619+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":125,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":6}]} 2024-08-03T21:00:32.170 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 330 2024-08-03T21:00:32.172 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:32.171+0000 7f00f2c00640 1 -- 172.21.15.32:0/2565263327 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f00cc077010 msgr2=0x7f00cc079520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:00:32.172 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:32.171+0000 7f00f2c00640 1 --2- 172.21.15.32:0/2565263327 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f00cc077010 0x7f00cc079520 secure :-1 s=READY pgs=834 cs=0 l=1 rev1=1 crypto rx=0x7f00e0034000 tx=0x7f00e0004730 comp rx=0 tx=0).stop 2024-08-03T21:00:32.172 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:32.171+0000 7f00f2c00640 1 -- 172.21.15.32:0/2565263327 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f00ec0d2290 msgr2=0x7f00ec0b7260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:00:32.172 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:32.171+0000 7f00f2c00640 1 --2- 172.21.15.32:0/2565263327 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f00ec0d2290 0x7f00ec0b7260 secure :-1 s=READY pgs=2092 cs=0 l=1 rev1=1 crypto rx=0x7f00d800c3f0 tx=0x7f00d800c8b0 comp rx=0 tx=0).stop 2024-08-03T21:00:32.172 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:32.171+0000 7f00f2c00640 1 -- 172.21.15.32:0/2565263327 shutdown_connections 2024-08-03T21:00:32.172 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:32.171+0000 7f00f2c00640 1 --2- 172.21.15.32:0/2565263327 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f00cc077010 0x7f00cc079520 unknown :-1 s=CLOSED pgs=834 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:32.172 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:32.171+0000 7f00f2c00640 1 --2- 172.21.15.32:0/2565263327 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f00ec0d4bb0 0x7f00ec0b77a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:32.172 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:32.171+0000 7f00f2c00640 1 --2- 172.21.15.32:0/2565263327 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f00ec0d2290 0x7f00ec0b7260 unknown :-1 s=CLOSED pgs=2092 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:32.172 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:32.171+0000 7f00f2c00640 1 --2- 172.21.15.32:0/2565263327 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f00ec0cec30 0x7f00ec15c690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:32.173 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:32.171+0000 7f00f2c00640 1 -- 172.21.15.32:0/2565263327 >> 172.21.15.32:0/2565263327 conn(0x7f00ec0d4090 msgr2=0x7f00ec0b4e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T21:00:32.173 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:32.171+0000 7f00f2c00640 1 -- 172.21.15.32:0/2565263327 shutdown_connections 2024-08-03T21:00:32.173 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:32.171+0000 7f00f2c00640 1 -- 172.21.15.32:0/2565263327 wait complete. 2024-08-03T21:00:32.194 INFO:tasks.cephfs.test_damage:Result: Mutation 'Corrupt omap header on 1.00000000' led to DAMAGED state 2024-08-03T21:00:32.194 INFO:tasks.cephfs.test_damage:All 39 mutations had expected outcomes 2024-08-03T21:00:32.194 INFO:teuthology.nuke:Clearing teuthology firewall rules... 2024-08-03T21:00:32.195 DEBUG:teuthology.orchestra.run.smithi032:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-08-03T21:00:32.280 DEBUG:teuthology.orchestra.run.smithi118:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-08-03T21:00:32.384 INFO:teuthology.nuke:Cleared teuthology firewall rules. 2024-08-03T21:00:32.384 INFO:tasks.cephfs.mount:Terminating background process 2024-08-03T21:00:32.384 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-08-03T21:00:32.385 DEBUG:teuthology.orchestra.run.smithi032:> dd if=/proc/self/mounts of=/dev/stdout 2024-08-03T21:00:32.420 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053453,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6499208k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 none /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249604k,nr_inodes=812401,mode=700,uid=1000,gid=1316 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6499208k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-08-03T21:00:32.420 INFO:tasks.cephfs.mount:Terminating background process 2024-08-03T21:00:32.421 DEBUG:teuthology.orchestra.run.smithi118:> set -ex 2024-08-03T21:00:32.421 DEBUG:teuthology.orchestra.run.smithi118:> dd if=/proc/self/mounts of=/dev/stdout 2024-08-03T21:00:32.458 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052663,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497944k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248972k,nr_inodes=812243,mode=700,uid=1000,gid=1316 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497944k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-08-03T21:00:32.458 INFO:tasks.cephfs.mount:Terminating background process 2024-08-03T21:00:32.458 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-08-03T21:00:32.458 DEBUG:teuthology.orchestra.run.smithi032:> dd if=/proc/self/mounts of=/dev/stdout 2024-08-03T21:00:32.495 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053453,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6499208k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 none /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249604k,nr_inodes=812401,mode=700,uid=1000,gid=1316 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6499208k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-08-03T21:00:32.495 INFO:tasks.cephfs.mount:Terminating background process 2024-08-03T21:00:32.495 DEBUG:teuthology.orchestra.run.smithi118:> set -ex 2024-08-03T21:00:32.496 DEBUG:teuthology.orchestra.run.smithi118:> dd if=/proc/self/mounts of=/dev/stdout 2024-08-03T21:00:32.534 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052663,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497944k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248972k,nr_inodes=812243,mode=700,uid=1000,gid=1316 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497944k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-08-03T21:00:32.534 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-08-03T21:00:32.763 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:32.762+0000 7f63c1600640 1 -- 172.21.15.32:0/4258509259 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f63bc0b4ce0 msgr2=0x7f63bc15ef00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:00:32.763 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:32.762+0000 7f63c1600640 1 --2- 172.21.15.32:0/4258509259 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f63bc0b4ce0 0x7f63bc15ef00 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7f63a4009830 tx=0x7f63a402a510 comp rx=0 tx=0).stop 2024-08-03T21:00:32.763 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:32.762+0000 7f63c1600640 1 -- 172.21.15.32:0/4258509259 shutdown_connections 2024-08-03T21:00:32.763 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:32.762+0000 7f63c1600640 1 --2- 172.21.15.32:0/4258509259 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f63bc0b4ce0 0x7f63bc15ef00 unknown :-1 s=CLOSED pgs=623 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:32.763 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:32.762+0000 7f63c1600640 1 --2- 172.21.15.32:0/4258509259 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f63bc0b38b0 0x7f63bc1523f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:32.764 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:32.762+0000 7f63c1600640 1 -- 172.21.15.32:0/4258509259 >> 172.21.15.32:0/4258509259 conn(0x7f63bc0b1bd0 msgr2=0x7f63bc0b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T21:00:32.764 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:32.762+0000 7f63c1600640 1 -- 172.21.15.32:0/4258509259 shutdown_connections 2024-08-03T21:00:32.764 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:32.762+0000 7f63c1600640 1 -- 172.21.15.32:0/4258509259 wait complete. 2024-08-03T21:00:32.765 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:32.764+0000 7f63c1600640 1 Processor -- start 2024-08-03T21:00:32.765 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:32.764+0000 7f63c1600640 1 -- start start 2024-08-03T21:00:32.767 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:32.766+0000 7f63c1600640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f63bc0b38b0 0x7f63bc151c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:00:32.767 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:32.766+0000 7f63c1600640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f63bc0b4ce0 0x7f63bc155650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:00:32.767 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:32.766+0000 7f63c1600640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f63bc15f440 0x7f63bc15d0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:00:32.767 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:32.766+0000 7f63c1600640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f63bc15c020 con 0x7f63bc0b4ce0 2024-08-03T21:00:32.767 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:32.766+0000 7f63c1600640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f63bc0cf240 con 0x7f63bc15f440 2024-08-03T21:00:32.767 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:32.766+0000 7f63c1600640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7f63bc15c1a0 con 0x7f63bc0b38b0 2024-08-03T21:00:32.767 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:32.766+0000 7f63b2000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f63bc0b4ce0 0x7f63bc155650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:00:32.767 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:32.766+0000 7f63b2000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f63bc0b4ce0 0x7f63bc155650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:52742/0 (socket says 172.21.15.32:52742) 2024-08-03T21:00:32.767 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:32.766+0000 7f63b2000640 1 -- 172.21.15.32:0/3114734420 learned_addr learned my addr 172.21.15.32:0/3114734420 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T21:00:32.767 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:32.766+0000 7f63baa00640 1 --2- 172.21.15.32:0/3114734420 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f63bc0b38b0 0x7f63bc151c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:00:32.767 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:32.766+0000 7f63bb400640 1 --2- 172.21.15.32:0/3114734420 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f63bc15f440 0x7f63bc15d0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:00:32.769 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:32.767+0000 7f63b2000640 1 -- 172.21.15.32:0/3114734420 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f63bc0b38b0 msgr2=0x7f63bc151c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:00:32.769 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:32.767+0000 7f63b2000640 1 --2- 172.21.15.32:0/3114734420 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f63bc0b38b0 0x7f63bc151c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:32.769 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:32.767+0000 7f63b2000640 1 -- 172.21.15.32:0/3114734420 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f63bc15f440 msgr2=0x7f63bc15d0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:00:32.769 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:32.767+0000 7f63b2000640 1 --2- 172.21.15.32:0/3114734420 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f63bc15f440 0x7f63bc15d0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:32.769 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:32.767+0000 7f63b2000640 1 -- 172.21.15.32:0/3114734420 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f63bc1f8510 con 0x7f63bc0b4ce0 2024-08-03T21:00:32.769 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:32.767+0000 7f63b2000640 1 --2- 172.21.15.32:0/3114734420 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f63bc0b4ce0 0x7f63bc155650 secure :-1 s=READY pgs=2093 cs=0 l=1 rev1=1 crypto rx=0x7f63a4009950 tx=0x7f63a4033c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T21:00:32.769 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:32.767+0000 7f63b3400640 1 -- 172.21.15.32:0/3114734420 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63a4031d60 con 0x7f63bc0b4ce0 2024-08-03T21:00:32.770 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:32.767+0000 7f63b3400640 1 -- 172.21.15.32:0/3114734420 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f63a4033e10 con 0x7f63bc0b4ce0 2024-08-03T21:00:32.770 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:32.767+0000 7f63b3400640 1 -- 172.21.15.32:0/3114734420 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63a4005260 con 0x7f63bc0b4ce0 2024-08-03T21:00:32.770 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:32.767+0000 7f63c1600640 1 -- 172.21.15.32:0/3114734420 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f63bc1f8a10 con 0x7f63bc0b4ce0 2024-08-03T21:00:32.770 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:32.768+0000 7f63c1600640 1 -- 172.21.15.32:0/3114734420 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f63bc1f8fd0 con 0x7f63bc0b4ce0 2024-08-03T21:00:32.770 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:32.769+0000 7f63c1600640 1 -- 172.21.15.32:0/3114734420 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f63bc1f8780 con 0x7f63bc0b4ce0 2024-08-03T21:00:32.770 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:32.770+0000 7f63b3400640 1 -- 172.21.15.32:0/3114734420 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f63a4005400 con 0x7f63bc0b4ce0 2024-08-03T21:00:32.772 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:32.771+0000 7f63b3400640 1 --2- 172.21.15.32:0/3114734420 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f6398077040 0x7f6398079550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:00:32.772 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:32.772+0000 7f63b3400640 1 -- 172.21.15.32:0/3114734420 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(125..125 src has 1..125) ==== 9294+0+0 (secure 0 0 0) 0x7f63a40ba680 con 0x7f63bc0b4ce0 2024-08-03T21:00:32.772 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:32.772+0000 7f63baa00640 1 --2- 172.21.15.32:0/3114734420 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f6398077040 0x7f6398079550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:00:32.773 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:32.773+0000 7f63baa00640 1 --2- 172.21.15.32:0/3114734420 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f6398077040 0x7f6398079550 secure :-1 s=READY pgs=835 cs=0 l=1 rev1=1 crypto rx=0x7f63ac009530 tx=0x7f63ac009260 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T21:00:32.777 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:32.776+0000 7f63b3400640 1 -- 172.21.15.32:0/3114734420 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7f63a403f030 con 0x7f63bc0b4ce0 2024-08-03T21:00:32.853 INFO:tasks.ceph.mds.a.smithi032.stderr: -54> 2024-08-03T21:00:28.098+0000 7f0a6ce00640 -1 mds.0.cache.dir(0x1) Corrupt fnode in dirfrag 0x1: Decoder at 'void fnode_t::decode(ceph::buffer::v15_2_0::list::const_iterator&)' v=3 cannot decode v=100 minimal_decoder=101: Malformed input [buffer:3] 2024-08-03T21:00:32.853 INFO:tasks.ceph.mds.a.smithi032.stderr: -51> 2024-08-03T21:00:28.098+0000 7f0a6ce00640 -1 mds.0.damage notify_dirfrag Damage to fragment * of ino 0x1 is fatal because it is a system directory for this rank 2024-08-03T21:00:32.853 INFO:tasks.ceph.mds.a.smithi032.stderr: -31> 2024-08-03T21:00:29.070+0000 7f0a71e00640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-08-03T21:00:32.966 INFO:tasks.ceph.mds.a.smithi032.stdout:starting mds.a at 2024-08-03T21:00:33.023 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:33.023+0000 7f63c1600640 1 -- 172.21.15.32:0/3114734420 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f63bc15c650 con 0x7f63bc0b4ce0 2024-08-03T21:00:33.024 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:33.024+0000 7f63b3400640 1 -- 172.21.15.32:0/3114734420 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 330 v330) ==== 95+0+4391 (secure 0 0 0) 0x7f63a4082470 con 0x7f63bc0b4ce0 2024-08-03T21:00:33.024 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-03T21:00:33.025 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":330,"btime":"2024-08-03T21:00:28:099623+0000","default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6439,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6835/531080160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6834","nonce":531080160},{"type":"v1","addr":"172.21.15.118:6835","nonce":531080160}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":84},{"gid":6454,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6837/2796882319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6836","nonce":2796882319},{"type":"v1","addr":"172.21.15.118:6837","nonce":2796882319}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":86},{"gid":6813,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6834/2686933890","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":2686933890},{"type":"v1","addr":"172.21.15.32:6834","nonce":2686933890}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":85}],"filesystems":[{"mdsmap":{"epoch":330,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-08-03T20:54:26.950518+0000","modified":"2024-08-03T21:00:28.099619+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":125,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":6}]} 2024-08-03T21:00:33.025 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 330 2024-08-03T21:00:33.027 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:33.026+0000 7f63c1600640 1 -- 172.21.15.32:0/3114734420 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f6398077040 msgr2=0x7f6398079550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:00:33.027 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:33.026+0000 7f63c1600640 1 --2- 172.21.15.32:0/3114734420 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f6398077040 0x7f6398079550 secure :-1 s=READY pgs=835 cs=0 l=1 rev1=1 crypto rx=0x7f63ac009530 tx=0x7f63ac009260 comp rx=0 tx=0).stop 2024-08-03T21:00:33.027 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:33.026+0000 7f63c1600640 1 -- 172.21.15.32:0/3114734420 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f63bc0b4ce0 msgr2=0x7f63bc155650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:00:33.027 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:33.026+0000 7f63c1600640 1 --2- 172.21.15.32:0/3114734420 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f63bc0b4ce0 0x7f63bc155650 secure :-1 s=READY pgs=2093 cs=0 l=1 rev1=1 crypto rx=0x7f63a4009950 tx=0x7f63a4033c20 comp rx=0 tx=0).stop 2024-08-03T21:00:33.027 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:33.026+0000 7f63c1600640 1 -- 172.21.15.32:0/3114734420 shutdown_connections 2024-08-03T21:00:33.027 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:33.026+0000 7f63c1600640 1 --2- 172.21.15.32:0/3114734420 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f6398077040 0x7f6398079550 unknown :-1 s=CLOSED pgs=835 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:33.027 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:33.026+0000 7f63c1600640 1 --2- 172.21.15.32:0/3114734420 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f63bc15f440 0x7f63bc15d0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:33.027 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:33.026+0000 7f63c1600640 1 --2- 172.21.15.32:0/3114734420 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f63bc0b4ce0 0x7f63bc155650 unknown :-1 s=CLOSED pgs=2093 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:33.027 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:33.026+0000 7f63c1600640 1 --2- 172.21.15.32:0/3114734420 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f63bc0b38b0 0x7f63bc151c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:33.028 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:33.026+0000 7f63c1600640 1 -- 172.21.15.32:0/3114734420 >> 172.21.15.32:0/3114734420 conn(0x7f63bc0b1bd0 msgr2=0x7f63bc159a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T21:00:33.028 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:33.026+0000 7f63c1600640 1 -- 172.21.15.32:0/3114734420 shutdown_connections 2024-08-03T21:00:33.028 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:33.026+0000 7f63c1600640 1 -- 172.21.15.32:0/3114734420 wait complete. 2024-08-03T21:00:33.051 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-08-03T21:00:33.281 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:33.280+0000 7f51ffa00640 1 -- 172.21.15.32:0/1505754654 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f51f806e930 msgr2=0x7f51f8144f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:00:33.281 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:33.280+0000 7f51ffa00640 1 --2- 172.21.15.32:0/1505754654 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f51f806e930 0x7f51f8144f90 secure :-1 s=READY pgs=2097 cs=0 l=1 rev1=1 crypto rx=0x7f51f80afaf0 tx=0x7f51ec02a560 comp rx=0 tx=0).stop 2024-08-03T21:00:33.281 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:33.280+0000 7f51ffa00640 1 -- 172.21.15.32:0/1505754654 shutdown_connections 2024-08-03T21:00:33.281 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:33.280+0000 7f51ffa00640 1 --2- 172.21.15.32:0/1505754654 >> v2:172.21.15.118:3300/0 conn(0x7f51f8073e60 0x7f51f8074270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:33.281 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:33.280+0000 7f51ffa00640 1 --2- 172.21.15.32:0/1505754654 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f51f806e930 0x7f51f8144f90 unknown :-1 s=CLOSED pgs=2097 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:33.281 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:33.280+0000 7f51ffa00640 1 -- 172.21.15.32:0/1505754654 >> 172.21.15.32:0/1505754654 conn(0x7f51f80b1bd0 msgr2=0x7f51f80b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T21:00:33.281 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:33.280+0000 7f51ffa00640 1 -- 172.21.15.32:0/1505754654 shutdown_connections 2024-08-03T21:00:33.282 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:33.280+0000 7f51ffa00640 1 -- 172.21.15.32:0/1505754654 wait complete. 2024-08-03T21:00:33.282 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:33.282+0000 7f51ffa00640 1 Processor -- start 2024-08-03T21:00:33.283 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:33.282+0000 7f51ffa00640 1 -- start start 2024-08-03T21:00:33.286 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:33.284+0000 7f51ffa00640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f51f806e930 0x7f51f815bb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:00:33.286 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:33.284+0000 7f51ffa00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f51f8073e60 0x7f51f80aac20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:00:33.286 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:33.284+0000 7f51ffa00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f51f80747b0 0x7f51f813f500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:00:33.286 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:33.284+0000 7f51ffa00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f51f80cf9c0 con 0x7f51f8073e60 2024-08-03T21:00:33.286 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:33.284+0000 7f51ffa00640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f51f80cf840 con 0x7f51f80747b0 2024-08-03T21:00:33.286 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:33.284+0000 7f51ffa00640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7f51f80ccff0 con 0x7f51f806e930 2024-08-03T21:00:33.286 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:33.284+0000 7f51f7e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f51f8073e60 0x7f51f80aac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:00:33.287 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:33.284+0000 7f51f7e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f51f8073e60 0x7f51f80aac20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:52778/0 (socket says 172.21.15.32:52778) 2024-08-03T21:00:33.287 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:33.284+0000 7f51f7e00640 1 -- 172.21.15.32:0/2037336735 learned_addr learned my addr 172.21.15.32:0/2037336735 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T21:00:33.287 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:33.284+0000 7f51fd000640 1 --2- 172.21.15.32:0/2037336735 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f51f806e930 0x7f51f815bb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:00:33.287 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:33.284+0000 7f51fda00640 1 --2- 172.21.15.32:0/2037336735 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f51f80747b0 0x7f51f813f500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:00:33.287 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:33.285+0000 7f51f7e00640 1 -- 172.21.15.32:0/2037336735 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f51f806e930 msgr2=0x7f51f815bb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:00:33.287 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:33.285+0000 7f51f7e00640 1 --2- 172.21.15.32:0/2037336735 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f51f806e930 0x7f51f815bb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:33.287 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:33.285+0000 7f51f7e00640 1 -- 172.21.15.32:0/2037336735 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f51f80747b0 msgr2=0x7f51f813f500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:00:33.287 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:33.285+0000 7f51f7e00640 1 --2- 172.21.15.32:0/2037336735 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f51f80747b0 0x7f51f813f500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:33.287 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:33.285+0000 7f51f7e00640 1 -- 172.21.15.32:0/2037336735 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f51f81f2f00 con 0x7f51f8073e60 2024-08-03T21:00:33.287 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:33.285+0000 7f51fda00640 1 --2- 172.21.15.32:0/2037336735 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f51f80747b0 0x7f51f813f500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-03T21:00:33.287 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:33.285+0000 7f51f7e00640 1 --2- 172.21.15.32:0/2037336735 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f51f8073e60 0x7f51f80aac20 secure :-1 s=READY pgs=2098 cs=0 l=1 rev1=1 crypto rx=0x7f51e400e7c0 tx=0x7f51e400ec80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T21:00:33.287 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:33.285+0000 7f51f4c00640 1 -- 172.21.15.32:0/2037336735 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51e400cc60 con 0x7f51f8073e60 2024-08-03T21:00:33.287 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:33.285+0000 7f51f4c00640 1 -- 172.21.15.32:0/2037336735 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f51e400ce00 con 0x7f51f8073e60 2024-08-03T21:00:33.288 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:33.285+0000 7f51f4c00640 1 -- 172.21.15.32:0/2037336735 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51e4004280 con 0x7f51f8073e60 2024-08-03T21:00:33.288 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:33.285+0000 7f51ffa00640 1 -- 172.21.15.32:0/2037336735 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f51f815c070 con 0x7f51f8073e60 2024-08-03T21:00:33.288 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:33.286+0000 7f51ffa00640 1 -- 172.21.15.32:0/2037336735 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f51f8143f70 con 0x7f51f8073e60 2024-08-03T21:00:33.288 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:33.287+0000 7f51ffa00640 1 -- 172.21.15.32:0/2037336735 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f51f80ab6c0 con 0x7f51f8073e60 2024-08-03T21:00:33.288 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:33.288+0000 7f51f4c00640 1 -- 172.21.15.32:0/2037336735 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f51e40035f0 con 0x7f51f8073e60 2024-08-03T21:00:33.289 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:33.289+0000 7f51f4c00640 1 --2- 172.21.15.32:0/2037336735 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f51d8077010 0x7f51d8079520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:00:33.289 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:33.289+0000 7f51f4c00640 1 -- 172.21.15.32:0/2037336735 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(125..125 src has 1..125) ==== 9294+0+0 (secure 0 0 0) 0x7f51e4099eb0 con 0x7f51f8073e60 2024-08-03T21:00:33.290 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:33.289+0000 7f51fd000640 1 --2- 172.21.15.32:0/2037336735 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f51d8077010 0x7f51d8079520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:00:33.290 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:33.290+0000 7f51fd000640 1 --2- 172.21.15.32:0/2037336735 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f51d8077010 0x7f51d8079520 secure :-1 s=READY pgs=837 cs=0 l=1 rev1=1 crypto rx=0x7f51ec033000 tx=0x7f51ec004730 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T21:00:33.293 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:33.293+0000 7f51f4c00640 1 -- 172.21.15.32:0/2037336735 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7f51f815c070 con 0x7f51f8073e60 2024-08-03T21:00:33.537 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:33.537+0000 7f51ffa00640 1 -- 172.21.15.32:0/2037336735 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f51f80cd580 con 0x7f51f8073e60 2024-08-03T21:00:33.538 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:33.537+0000 7f51f4c00640 1 -- 172.21.15.32:0/2037336735 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 331 v331) ==== 95+0+5218 (secure 0 0 0) 0x7f51e4062320 con 0x7f51f8073e60 2024-08-03T21:00:33.538 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-03T21:00:33.538 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":331,"btime":"2024-08-03T21:00:33:109208+0000","default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6439,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6835/531080160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6834","nonce":531080160},{"type":"v1","addr":"172.21.15.118:6835","nonce":531080160}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":84},{"gid":6454,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6837/2796882319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6836","nonce":2796882319},{"type":"v1","addr":"172.21.15.118:6837","nonce":2796882319}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":86},{"gid":6813,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6834/2686933890","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":2686933890},{"type":"v1","addr":"172.21.15.32:6834","nonce":2686933890}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":85},{"gid":10359,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6835/3212405498","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6833","nonce":3212405498},{"type":"v1","addr":"172.21.15.32:6835","nonce":3212405498}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":331}],"filesystems":[{"mdsmap":{"epoch":330,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-08-03T20:54:26.950518+0000","modified":"2024-08-03T21:00:28.099619+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":125,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":6}]} 2024-08-03T21:00:33.539 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 331 2024-08-03T21:00:33.541 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:33.540+0000 7f51ffa00640 1 -- 172.21.15.32:0/2037336735 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f51d8077010 msgr2=0x7f51d8079520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:00:33.541 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:33.540+0000 7f51ffa00640 1 --2- 172.21.15.32:0/2037336735 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f51d8077010 0x7f51d8079520 secure :-1 s=READY pgs=837 cs=0 l=1 rev1=1 crypto rx=0x7f51ec033000 tx=0x7f51ec004730 comp rx=0 tx=0).stop 2024-08-03T21:00:33.541 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:33.540+0000 7f51ffa00640 1 -- 172.21.15.32:0/2037336735 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f51f8073e60 msgr2=0x7f51f80aac20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:00:33.541 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:33.540+0000 7f51ffa00640 1 --2- 172.21.15.32:0/2037336735 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f51f8073e60 0x7f51f80aac20 secure :-1 s=READY pgs=2098 cs=0 l=1 rev1=1 crypto rx=0x7f51e400e7c0 tx=0x7f51e400ec80 comp rx=0 tx=0).stop 2024-08-03T21:00:33.541 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:33.540+0000 7f51ffa00640 1 -- 172.21.15.32:0/2037336735 shutdown_connections 2024-08-03T21:00:33.541 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:33.540+0000 7f51ffa00640 1 --2- 172.21.15.32:0/2037336735 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f51d8077010 0x7f51d8079520 unknown :-1 s=CLOSED pgs=837 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:33.541 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:33.540+0000 7f51ffa00640 1 --2- 172.21.15.32:0/2037336735 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f51f80747b0 0x7f51f813f500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:33.541 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:33.540+0000 7f51ffa00640 1 --2- 172.21.15.32:0/2037336735 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f51f8073e60 0x7f51f80aac20 unknown :-1 s=CLOSED pgs=2098 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:33.541 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:33.540+0000 7f51ffa00640 1 --2- 172.21.15.32:0/2037336735 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f51f806e930 0x7f51f815bb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:33.542 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:33.540+0000 7f51ffa00640 1 -- 172.21.15.32:0/2037336735 >> 172.21.15.32:0/2037336735 conn(0x7f51f80b1bd0 msgr2=0x7f51f806ef70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T21:00:33.542 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:33.540+0000 7f51ffa00640 1 -- 172.21.15.32:0/2037336735 shutdown_connections 2024-08-03T21:00:33.542 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:33.540+0000 7f51ffa00640 1 -- 172.21.15.32:0/2037336735 wait complete. 2024-08-03T21:00:33.563 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-08-03T21:00:33.792 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:33.791+0000 7f2157e00640 1 -- 172.21.15.32:0/411648193 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f21580d1c60 msgr2=0x7f21580cf340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:00:33.792 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:33.791+0000 7f2157e00640 1 --2- 172.21.15.32:0/411648193 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f21580d1c60 0x7f21580cf340 secure :-1 s=READY pgs=2100 cs=0 l=1 rev1=1 crypto rx=0x7f2150005fd0 tx=0x7f215002a8b0 comp rx=0 tx=0).stop 2024-08-03T21:00:33.792 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:33.791+0000 7f2157e00640 1 -- 172.21.15.32:0/411648193 shutdown_connections 2024-08-03T21:00:33.792 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:33.791+0000 7f2157e00640 1 --2- 172.21.15.32:0/411648193 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f21580d1c60 0x7f21580cf340 unknown :-1 s=CLOSED pgs=2100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:33.793 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:33.791+0000 7f2157e00640 1 -- 172.21.15.32:0/411648193 >> 172.21.15.32:0/411648193 conn(0x7f21580d4090 msgr2=0x7f21580d44a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T21:00:33.793 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:33.791+0000 7f2157e00640 1 -- 172.21.15.32:0/411648193 shutdown_connections 2024-08-03T21:00:33.793 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:33.791+0000 7f2157e00640 1 -- 172.21.15.32:0/411648193 wait complete. 2024-08-03T21:00:33.794 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:33.793+0000 7f2157e00640 1 Processor -- start 2024-08-03T21:00:33.794 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:33.793+0000 7f2157e00640 1 -- start start 2024-08-03T21:00:33.797 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:33.795+0000 7f2157e00640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f21580d1c60 0x7f21580cd2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:00:33.797 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:33.795+0000 7f2157e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f21580d4bb0 0x7f21580cd820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:00:33.797 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:33.795+0000 7f2157e00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f21581f60a0 0x7f21581f8e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:00:33.797 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:33.795+0000 7f2157e00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f21580cfcf0 con 0x7f21580d4bb0 2024-08-03T21:00:33.797 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:33.795+0000 7f2157e00640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f21580cfb70 con 0x7f21581f60a0 2024-08-03T21:00:33.797 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:33.795+0000 7f2157e00640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7f21580cfe70 con 0x7f21580d1c60 2024-08-03T21:00:33.797 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:33.795+0000 7f214e000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f21580d4bb0 0x7f21580cd820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:00:33.797 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:33.795+0000 7f214e000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f21580d4bb0 0x7f21580cd820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:52796/0 (socket says 172.21.15.32:52796) 2024-08-03T21:00:33.798 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:33.795+0000 7f214e000640 1 -- 172.21.15.32:0/4132389288 learned_addr learned my addr 172.21.15.32:0/4132389288 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T21:00:33.798 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:33.795+0000 7f2156a00640 1 --2- 172.21.15.32:0/4132389288 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f21580d1c60 0x7f21580cd2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:00:33.798 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:33.795+0000 7f2157400640 1 --2- 172.21.15.32:0/4132389288 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f21581f60a0 0x7f21581f8e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:00:33.798 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:33.796+0000 7f214e000640 1 -- 172.21.15.32:0/4132389288 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f21580d1c60 msgr2=0x7f21580cd2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:00:33.798 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:33.796+0000 7f214e000640 1 --2- 172.21.15.32:0/4132389288 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f21580d1c60 0x7f21580cd2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:33.798 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:33.796+0000 7f214e000640 1 -- 172.21.15.32:0/4132389288 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f21581f60a0 msgr2=0x7f21581f8e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:00:33.798 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:33.796+0000 7f214e000640 1 --2- 172.21.15.32:0/4132389288 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f21581f60a0 0x7f21581f8e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:33.798 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:33.796+0000 7f214e000640 1 -- 172.21.15.32:0/4132389288 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2158166a60 con 0x7f21580d4bb0 2024-08-03T21:00:33.798 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:33.796+0000 7f214e000640 1 --2- 172.21.15.32:0/4132389288 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f21580d4bb0 0x7f21580cd820 secure :-1 s=READY pgs=2101 cs=0 l=1 rev1=1 crypto rx=0x7f214000c430 tx=0x7f214000c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T21:00:33.798 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:33.796+0000 7f214f400640 1 -- 172.21.15.32:0/4132389288 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2140011790 con 0x7f21580d4bb0 2024-08-03T21:00:33.799 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:33.796+0000 7f214f400640 1 -- 172.21.15.32:0/4132389288 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f2140017070 con 0x7f21580d4bb0 2024-08-03T21:00:33.799 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:33.796+0000 7f2157e00640 1 -- 172.21.15.32:0/4132389288 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f21581f55c0 con 0x7f21580d4bb0 2024-08-03T21:00:33.799 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:33.796+0000 7f214f400640 1 -- 172.21.15.32:0/4132389288 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2140011ce0 con 0x7f21580d4bb0 2024-08-03T21:00:33.799 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:33.796+0000 7f2157e00640 1 -- 172.21.15.32:0/4132389288 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f21581f5b80 con 0x7f21580d4bb0 2024-08-03T21:00:33.800 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:33.797+0000 7f2157e00640 1 -- 172.21.15.32:0/4132389288 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f21581f53d0 con 0x7f21580d4bb0 2024-08-03T21:00:33.800 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:33.799+0000 7f214f400640 1 -- 172.21.15.32:0/4132389288 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f214001c460 con 0x7f21580d4bb0 2024-08-03T21:00:33.800 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:33.800+0000 7f214f400640 1 --2- 172.21.15.32:0/4132389288 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f2134077010 0x7f2134079520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:00:33.800 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:33.800+0000 7f214f400640 1 -- 172.21.15.32:0/4132389288 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(125..125 src has 1..125) ==== 9294+0+0 (secure 0 0 0) 0x7f21400994e0 con 0x7f21580d4bb0 2024-08-03T21:00:33.801 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:33.800+0000 7f2156a00640 1 --2- 172.21.15.32:0/4132389288 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f2134077010 0x7f2134079520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:00:33.801 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:33.801+0000 7f2156a00640 1 --2- 172.21.15.32:0/4132389288 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f2134077010 0x7f2134079520 secure :-1 s=READY pgs=838 cs=0 l=1 rev1=1 crypto rx=0x7f2148001720 tx=0x7f2148009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T21:00:33.805 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:33.804+0000 7f214f400640 1 -- 172.21.15.32:0/4132389288 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7f2140061240 con 0x7f21580d4bb0 2024-08-03T21:00:34.010 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:34.009+0000 7f2157e00640 1 -- 172.21.15.32:0/4132389288 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f21581667f0 con 0x7f21580d4bb0 2024-08-03T21:00:34.011 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:34.011+0000 7f214f400640 1 -- 172.21.15.32:0/4132389288 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v125) ==== 74+0+22023 (secure 0 0 0) 0x7f2140065790 con 0x7f21580d4bb0 2024-08-03T21:00:34.012 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-03T21:00:34.012 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":125,"fsid":"40ab141a-01d7-49b9-ae3c-2edec9876e95","created":"2024-08-03T20:47:14.957362+0000","modified":"2024-08-03T21:00:28.099308+0000","last_up_change":"2024-08-03T20:47:28.874451+0000","last_in_change":"2024-08-03T20:47:20.079442+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":13,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-08-03T20:47:28.261300+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_metadata","create_time":"2024-08-03T20:54:24.911555+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"82","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs_data","create_time":"2024-08-03T20:54:25.917207+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"83","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5a83e759-9035-438b-889a-0a6af1f0a0d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6816","nonce":3066156116},{"type":"v1","addr":"172.21.15.32:6817","nonce":3066156116}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6818","nonce":3066156116},{"type":"v1","addr":"172.21.15.32:6819","nonce":3066156116}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6822","nonce":3066156116},{"type":"v1","addr":"172.21.15.32:6823","nonce":3066156116}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6820","nonce":3066156116},{"type":"v1","addr":"172.21.15.32:6821","nonce":3066156116}]},"public_addr":"172.21.15.32:6817/3066156116","cluster_addr":"172.21.15.32:6819/3066156116","heartbeat_back_addr":"172.21.15.32:6823/3066156116","heartbeat_front_addr":"172.21.15.32:6821/3066156116","state":["exists","up"]},{"osd":1,"uuid":"d7c5c210-da4f-49f9-b9e7-1dfb02daa4bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6824","nonce":1551241759},{"type":"v1","addr":"172.21.15.32:6825","nonce":1551241759}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6826","nonce":1551241759},{"type":"v1","addr":"172.21.15.32:6827","nonce":1551241759}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6830","nonce":1551241759},{"type":"v1","addr":"172.21.15.32:6831","nonce":1551241759}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6828","nonce":1551241759},{"type":"v1","addr":"172.21.15.32:6829","nonce":1551241759}]},"public_addr":"172.21.15.32:6825/1551241759","cluster_addr":"172.21.15.32:6827/1551241759","heartbeat_back_addr":"172.21.15.32:6831/1551241759","heartbeat_front_addr":"172.21.15.32:6829/1551241759","state":["exists","up"]},{"osd":2,"uuid":"57f2e6de-a201-45c2-8981-8e56a4875ebd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6808","nonce":3806708944},{"type":"v1","addr":"172.21.15.32:6809","nonce":3806708944}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6810","nonce":3806708944},{"type":"v1","addr":"172.21.15.32:6811","nonce":3806708944}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6814","nonce":3806708944},{"type":"v1","addr":"172.21.15.32:6815","nonce":3806708944}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6812","nonce":3806708944},{"type":"v1","addr":"172.21.15.32:6813","nonce":3806708944}]},"public_addr":"172.21.15.32:6809/3806708944","cluster_addr":"172.21.15.32:6811/3806708944","heartbeat_back_addr":"172.21.15.32:6815/3806708944","heartbeat_front_addr":"172.21.15.32:6813/3806708944","state":["exists","up"]},{"osd":3,"uuid":"6439a280-da45-4148-8130-f16323935ff3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6800","nonce":9088303},{"type":"v1","addr":"172.21.15.32:6801","nonce":9088303}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6802","nonce":9088303},{"type":"v1","addr":"172.21.15.32:6803","nonce":9088303}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6806","nonce":9088303},{"type":"v1","addr":"172.21.15.32:6807","nonce":9088303}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6804","nonce":9088303},{"type":"v1","addr":"172.21.15.32:6805","nonce":9088303}]},"public_addr":"172.21.15.32:6801/9088303","cluster_addr":"172.21.15.32:6803/9088303","heartbeat_back_addr":"172.21.15.32:6807/9088303","heartbeat_front_addr":"172.21.15.32:6805/9088303","state":["exists","up"]},{"osd":4,"uuid":"4f902eff-1e18-418d-82c9-2151c69524fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6802","nonce":864752275},{"type":"v1","addr":"172.21.15.118:6803","nonce":864752275}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6804","nonce":864752275},{"type":"v1","addr":"172.21.15.118:6805","nonce":864752275}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6808","nonce":864752275},{"type":"v1","addr":"172.21.15.118:6809","nonce":864752275}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6806","nonce":864752275},{"type":"v1","addr":"172.21.15.118:6807","nonce":864752275}]},"public_addr":"172.21.15.118:6803/864752275","cluster_addr":"172.21.15.118:6805/864752275","heartbeat_back_addr":"172.21.15.118:6809/864752275","heartbeat_front_addr":"172.21.15.118:6807/864752275","state":["exists","up"]},{"osd":5,"uuid":"bfd9feac-dd06-458c-91b9-af7f175f612c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6810","nonce":2724891770},{"type":"v1","addr":"172.21.15.118:6811","nonce":2724891770}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6812","nonce":2724891770},{"type":"v1","addr":"172.21.15.118:6813","nonce":2724891770}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6816","nonce":2724891770},{"type":"v1","addr":"172.21.15.118:6817","nonce":2724891770}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6814","nonce":2724891770},{"type":"v1","addr":"172.21.15.118:6815","nonce":2724891770}]},"public_addr":"172.21.15.118:6811/2724891770","cluster_addr":"172.21.15.118:6813/2724891770","heartbeat_back_addr":"172.21.15.118:6817/2724891770","heartbeat_front_addr":"172.21.15.118:6815/2724891770","state":["exists","up"]},{"osd":6,"uuid":"5ed5a279-9d8e-416d-bbc7-7edd3e6d5705","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6825","nonce":801947386},{"type":"v1","addr":"172.21.15.118:6827","nonce":801947386}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6828","nonce":801947386},{"type":"v1","addr":"172.21.15.118:6829","nonce":801947386}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6832","nonce":801947386},{"type":"v1","addr":"172.21.15.118:6833","nonce":801947386}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6830","nonce":801947386},{"type":"v1","addr":"172.21.15.118:6831","nonce":801947386}]},"public_addr":"172.21.15.118:6827/801947386","cluster_addr":"172.21.15.118:6829/801947386","heartbeat_back_addr":"172.21.15.118:6833/801947386","heartbeat_front_addr":"172.21.15.118:6831/801947386","state":["exists","up"]},{"osd":7,"uuid":"f2086c47-875e-4451-a436-2008d1ea16af","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6818","nonce":653000048},{"type":"v1","addr":"172.21.15.118:6819","nonce":653000048}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6820","nonce":653000048},{"type":"v1","addr":"172.21.15.118:6821","nonce":653000048}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6824","nonce":653000048},{"type":"v1","addr":"172.21.15.118:6826","nonce":653000048}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6822","nonce":653000048},{"type":"v1","addr":"172.21.15.118:6823","nonce":653000048}]},"public_addr":"172.21.15.118:6819/653000048","cluster_addr":"172.21.15.118:6821/653000048","heartbeat_back_addr":"172.21.15.118:6826/653000048","heartbeat_front_addr":"172.21.15.118:6823/653000048","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-03T20:47:25.491994+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-03T20:47:25.587358+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-03T20:47:25.454175+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-03T20:47:25.377734+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-03T20:47:25.412351+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-03T20:47:25.499815+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-03T20:47:25.575044+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-03T20:47:25.552383+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"13.8","mappings":[{"from":2,"to":3}]},{"pgid":"13.14","mappings":[{"from":6,"to":5}]},{"pgid":"13.23","mappings":[{"from":1,"to":5}]},{"pgid":"13.32","mappings":[{"from":1,"to":5}]},{"pgid":"13.34","mappings":[{"from":6,"to":5}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.32:6835/1660875176":"2024-08-04T21:00:28.099287+0000","172.21.15.32:6833/1660875176":"2024-08-04T21:00:28.099287+0000","172.21.15.32:6835/2824868449":"2024-08-04T21:00:19.050715+0000","172.21.15.32:6835/73253044":"2024-08-04T21:00:10.997224+0000","172.21.15.32:6835/2017655932":"2024-08-04T21:00:04.959586+0000","172.21.15.32:6833/2017655932":"2024-08-04T21:00:04.959586+0000","172.21.15.32:6835/3127944613":"2024-08-04T20:59:46.766712+0000","172.21.15.32:6835/2106229503":"2024-08-04T20:59:40.719664+0000","172.21.15.32:6833/2106229503":"2024-08-04T20:59:40.719664+0000","172.21.15.32:6835/2173042128":"2024-08-04T20:59:28.193772+0000","172.21.15.32:6833/2173042128":"2024-08-04T20:59:28.193772+0000","172.21.15.32:6835/506775245":"2024-08-04T20:59:03.076256+0000","172.21.15.32:6833/506775245":"2024-08-04T20:59:03.076256+0000","172.21.15.32:6833/556650445":"2024-08-04T20:58:57.063754+0000","172.21.15.32:6835/2156596385":"2024-08-04T20:58:35.073838+0000","172.21.15.32:6833/2156596385":"2024-08-04T20:58:35.073838+0000","172.21.15.32:6835/3669493978":"2024-08-04T20:58:30.064676+0000","172.21.15.32:6833/2178506907":"2024-08-04T20:56:40.045549+0000","172.21.15.32:6835/556650445":"2024-08-04T20:58:57.063754+0000","172.21.15.32:6835/2278473729":"2024-08-04T20:56:13.041693+0000","172.21.15.32:6835/2345472481":"2024-08-04T20:57:25.968590+0000","172.21.15.32:6833/3940515773":"2024-08-04T20:58:24.290690+0000","172.21.15.32:6833/2600386707":"2024-08-04T20:57:50.063964+0000","172.21.15.32:6835/2178506907":"2024-08-04T20:56:40.045549+0000","172.21.15.32:6833/73253044":"2024-08-04T21:00:10.997224+0000","172.21.15.32:6833/2292690833":"2024-08-04T20:55:20.021329+0000","172.21.15.32:6835/2803901497":"2024-08-04T20:55:55.181144+0000","172.21.15.32:6835/2194902489":"2024-08-04T20:55:00.359126+0000","172.21.15.32:6833/2824868449":"2024-08-04T21:00:19.050715+0000","172.21.15.32:6833/3127944613":"2024-08-04T20:59:46.766712+0000","172.21.15.32:6833/2803901497":"2024-08-04T20:55:55.181144+0000","172.21.15.32:6835/3191654276":"2024-08-04T20:55:42.433364+0000","172.21.15.32:6833/2242533881":"2024-08-04T20:56:48.048099+0000","172.21.15.32:6833/2283924374":"2024-08-04T20:57:34.396782+0000","172.21.15.32:6835/1172050551":"2024-08-04T20:55:26.214113+0000","172.21.15.32:6835/1016338829":"2024-08-04T20:55:06.071485+0000","172.21.15.32:6833/1474984145":"2024-08-04T20:57:40.052808+0000","172.21.15.32:6833/3669493978":"2024-08-04T20:58:30.064676+0000","172.21.15.32:6835/756683988":"2024-08-04T20:55:14.119228+0000","172.21.15.32:6833/2194902489":"2024-08-04T20:55:00.359126+0000","172.21.15.32:6835/2806168786":"2024-08-04T20:56:37.279160+0000","172.21.15.32:6833/2278473729":"2024-08-04T20:56:13.041693+0000","172.21.15.32:6833/3191654276":"2024-08-04T20:55:42.433364+0000","172.21.15.32:6835/3431300225":"2024-08-04T20:56:21.088927+0000","172.21.15.32:6833/4172198298":"2024-08-04T20:56:04.256182+0000","172.21.15.32:6833/3431300225":"2024-08-04T20:56:21.088927+0000","172.21.15.32:6835/2292690833":"2024-08-04T20:55:20.021329+0000","172.21.15.32:6833/2806168786":"2024-08-04T20:56:37.279160+0000","172.21.15.32:6833/1172050551":"2024-08-04T20:55:26.214113+0000","172.21.15.32:6835/4172198298":"2024-08-04T20:56:04.256182+0000","172.21.15.32:6835/3409752696":"2024-08-04T20:58:18.243140+0000","172.21.15.32:6833/2274054575":"2024-08-04T20:57:55.062490+0000","172.21.15.32:6833/1016338829":"2024-08-04T20:55:06.071485+0000","172.21.15.32:6833/1707454554":"2024-08-04T20:54:38.409411+0000","172.21.15.32:6835/2242533881":"2024-08-04T20:56:48.048099+0000","172.21.15.32:6833/739874257":"2024-08-04T20:56:56.101271+0000","172.21.15.32:6835/739874257":"2024-08-04T20:56:56.101271+0000","172.21.15.32:6833/2362145329":"2024-08-04T20:57:02.150199+0000","172.21.15.32:6833/3771304550":"2024-08-04T20:55:58.212752+0000","172.21.15.32:6835/2362145329":"2024-08-04T20:57:02.150199+0000","172.21.15.32:6835/3940515773":"2024-08-04T20:58:24.290690+0000","172.21.15.32:6833/1885034124":"2024-08-04T20:57:08.190707+0000","172.21.15.32:6833/511221267":"2024-08-04T20:57:45.062448+0000","172.21.15.32:6835/3771304550":"2024-08-04T20:55:58.212752+0000","172.21.15.32:6835/1885034124":"2024-08-04T20:57:08.190707+0000","172.21.15.32:6833/2345472481":"2024-08-04T20:57:25.968590+0000","172.21.15.32:6833/265163622":"2024-08-04T20:57:28.354509+0000","172.21.15.32:6835/265163622":"2024-08-04T20:57:28.354509+0000","172.21.15.32:6835/2274054575":"2024-08-04T20:57:55.062490+0000","172.21.15.32:6835/2283924374":"2024-08-04T20:57:34.396782+0000","172.21.15.32:6833/756683988":"2024-08-04T20:55:14.119228+0000","172.21.15.32:6835/1474984145":"2024-08-04T20:57:40.052808+0000","172.21.15.32:6835/511221267":"2024-08-04T20:57:45.062448+0000","172.21.15.32:6835/1707454554":"2024-08-04T20:54:38.409411+0000","172.21.15.32:6835/2600386707":"2024-08-04T20:57:50.063964+0000","172.21.15.32:6833/675070591":"2024-08-04T20:58:00.068114+0000","172.21.15.32:6833/3340816283":"2024-08-04T20:58:15.214542+0000","172.21.15.32:6835/3340816283":"2024-08-04T20:58:15.214542+0000","172.21.15.32:6835/675070591":"2024-08-04T20:58:00.068114+0000","172.21.15.32:6833/3409752696":"2024-08-04T20:58:18.243140+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-08-03T21:00:34.014 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:34.013+0000 7f2157e00640 1 -- 172.21.15.32:0/4132389288 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f2134077010 msgr2=0x7f2134079520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:00:34.014 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:34.013+0000 7f2157e00640 1 --2- 172.21.15.32:0/4132389288 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f2134077010 0x7f2134079520 secure :-1 s=READY pgs=838 cs=0 l=1 rev1=1 crypto rx=0x7f2148001720 tx=0x7f2148009040 comp rx=0 tx=0).stop 2024-08-03T21:00:34.014 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:34.013+0000 7f2157e00640 1 -- 172.21.15.32:0/4132389288 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f21580d4bb0 msgr2=0x7f21580cd820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:00:34.014 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:34.013+0000 7f2157e00640 1 --2- 172.21.15.32:0/4132389288 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f21580d4bb0 0x7f21580cd820 secure :-1 s=READY pgs=2101 cs=0 l=1 rev1=1 crypto rx=0x7f214000c430 tx=0x7f214000c8f0 comp rx=0 tx=0).stop 2024-08-03T21:00:34.015 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:34.013+0000 7f2157e00640 1 -- 172.21.15.32:0/4132389288 shutdown_connections 2024-08-03T21:00:34.015 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:34.013+0000 7f2157e00640 1 --2- 172.21.15.32:0/4132389288 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f2134077010 0x7f2134079520 unknown :-1 s=CLOSED pgs=838 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:34.015 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:34.013+0000 7f2157e00640 1 --2- 172.21.15.32:0/4132389288 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f21581f60a0 0x7f21581f8e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:34.015 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:34.013+0000 7f2157e00640 1 --2- 172.21.15.32:0/4132389288 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f21580d4bb0 0x7f21580cd820 unknown :-1 s=CLOSED pgs=2101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:34.015 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:34.013+0000 7f2157e00640 1 --2- 172.21.15.32:0/4132389288 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f21580d1c60 0x7f21580cd2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:34.015 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:34.013+0000 7f2157e00640 1 -- 172.21.15.32:0/4132389288 >> 172.21.15.32:0/4132389288 conn(0x7f21580d4090 msgr2=0x7f215806de40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T21:00:34.015 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:34.013+0000 7f2157e00640 1 -- 172.21.15.32:0/4132389288 shutdown_connections 2024-08-03T21:00:34.015 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:34.013+0000 7f2157e00640 1 -- 172.21.15.32:0/4132389288 wait complete. 2024-08-03T21:00:34.038 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-08-03T21:00:34.038 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-08-03T21:00:34.268 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:34.267+0000 7f6861600640 1 -- 172.21.15.32:0/310345745 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6840005250 msgr2=0x7f684000b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:00:34.268 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:34.267+0000 7f6861600640 1 --2- 172.21.15.32:0/310345745 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6840005250 0x7f684000b9c0 secure :-1 s=READY pgs=2103 cs=0 l=1 rev1=1 crypto rx=0x7f68540016c0 tx=0x7f6840005830 comp rx=0 tx=0).stop 2024-08-03T21:00:34.268 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:34.267+0000 7f6861600640 1 -- 172.21.15.32:0/310345745 shutdown_connections 2024-08-03T21:00:34.268 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:34.267+0000 7f6861600640 1 --2- 172.21.15.32:0/310345745 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6840005250 0x7f684000b9c0 unknown :-1 s=CLOSED pgs=2103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:34.268 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:34.267+0000 7f6861600640 1 -- 172.21.15.32:0/310345745 >> 172.21.15.32:0/310345745 conn(0x7f685c0b1bd0 msgr2=0x7f685c0b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T21:00:34.268 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:34.267+0000 7f6861600640 1 -- 172.21.15.32:0/310345745 shutdown_connections 2024-08-03T21:00:34.268 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:34.267+0000 7f6861600640 1 -- 172.21.15.32:0/310345745 wait complete. 2024-08-03T21:00:34.270 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:34.269+0000 7f6861600640 1 Processor -- start 2024-08-03T21:00:34.270 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:34.270+0000 7f6861600640 1 -- start start 2024-08-03T21:00:34.273 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:34.271+0000 7f6861600640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f6840005250 0x7f685c15d410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:00:34.273 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:34.271+0000 7f6861600640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f685c0cc720 0x7f685c0ccb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:00:34.273 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:34.271+0000 7f6861600640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f685c15bae0 0x7f685c15bef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:00:34.273 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:34.271+0000 7f6861600640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f685c145110 con 0x7f685c0cc720 2024-08-03T21:00:34.274 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:34.271+0000 7f6861600640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f685c144f90 con 0x7f685c15bae0 2024-08-03T21:00:34.274 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:34.271+0000 7f6861600640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7f685c145290 con 0x7f6840005250 2024-08-03T21:00:34.274 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:34.271+0000 7f6852000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f685c0cc720 0x7f685c0ccb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:00:34.274 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:34.271+0000 7f6852000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f685c0cc720 0x7f685c0ccb30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:52818/0 (socket says 172.21.15.32:52818) 2024-08-03T21:00:34.274 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:34.271+0000 7f6852000640 1 -- 172.21.15.32:0/4004103554 learned_addr learned my addr 172.21.15.32:0/4004103554 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T21:00:34.274 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:34.271+0000 7f685aa00640 1 --2- 172.21.15.32:0/4004103554 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f6840005250 0x7f685c15d410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:00:34.274 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:34.271+0000 7f685b400640 1 --2- 172.21.15.32:0/4004103554 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f685c15bae0 0x7f685c15bef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:00:34.274 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:34.272+0000 7f6852000640 1 -- 172.21.15.32:0/4004103554 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f6840005250 msgr2=0x7f685c15d410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:00:34.274 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:34.272+0000 7f6852000640 1 --2- 172.21.15.32:0/4004103554 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f6840005250 0x7f685c15d410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:34.274 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:34.272+0000 7f6852000640 1 -- 172.21.15.32:0/4004103554 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f685c15bae0 msgr2=0x7f685c15bef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:00:34.274 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:34.272+0000 7f6852000640 1 --2- 172.21.15.32:0/4004103554 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f685c15bae0 0x7f685c15bef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:34.274 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:34.272+0000 7f6852000640 1 -- 172.21.15.32:0/4004103554 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f685c0cb7f0 con 0x7f685c0cc720 2024-08-03T21:00:34.275 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:34.272+0000 7f685aa00640 1 --2- 172.21.15.32:0/4004103554 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f6840005250 0x7f685c15d410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-03T21:00:34.275 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:34.272+0000 7f6852000640 1 --2- 172.21.15.32:0/4004103554 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f685c0cc720 0x7f685c0ccb30 secure :-1 s=READY pgs=2104 cs=0 l=1 rev1=1 crypto rx=0x7f684400e590 tx=0x7f684400ea50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T21:00:34.275 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:34.272+0000 7f6853400640 1 -- 172.21.15.32:0/4004103554 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6844015070 con 0x7f685c0cc720 2024-08-03T21:00:34.275 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:34.272+0000 7f6853400640 1 -- 172.21.15.32:0/4004103554 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f6844010980 con 0x7f685c0cc720 2024-08-03T21:00:34.275 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:34.272+0000 7f6861600640 1 -- 172.21.15.32:0/4004103554 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f685c15c430 con 0x7f685c0cc720 2024-08-03T21:00:34.275 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:34.272+0000 7f6853400640 1 -- 172.21.15.32:0/4004103554 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6844010db0 con 0x7f685c0cc720 2024-08-03T21:00:34.275 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:34.272+0000 7f6861600640 1 -- 172.21.15.32:0/4004103554 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f685c0cba90 con 0x7f685c0cc720 2024-08-03T21:00:34.275 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:34.273+0000 7f6861600640 1 -- 172.21.15.32:0/4004103554 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f685c074350 con 0x7f685c0cc720 2024-08-03T21:00:34.275 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:34.275+0000 7f6853400640 1 -- 172.21.15.32:0/4004103554 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f6844018070 con 0x7f685c0cc720 2024-08-03T21:00:34.276 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:34.276+0000 7f6853400640 1 --2- 172.21.15.32:0/4004103554 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f6838077010 0x7f6838079520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:00:34.276 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:34.276+0000 7f6853400640 1 -- 172.21.15.32:0/4004103554 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(125..125 src has 1..125) ==== 9294+0+0 (secure 0 0 0) 0x7f6844099b40 con 0x7f685c0cc720 2024-08-03T21:00:34.277 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:34.276+0000 7f685aa00640 1 --2- 172.21.15.32:0/4004103554 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f6838077010 0x7f6838079520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:00:34.277 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:34.277+0000 7f685aa00640 1 --2- 172.21.15.32:0/4004103554 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f6838077010 0x7f6838079520 secure :-1 s=READY pgs=839 cs=0 l=1 rev1=1 crypto rx=0x7f684c005fd0 tx=0x7f684c009500 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T21:00:34.280 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:34.280+0000 7f6853400640 1 -- 172.21.15.32:0/4004103554 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7f6844061e90 con 0x7f685c0cc720 2024-08-03T21:00:34.526 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:34.525+0000 7f6861600640 1 -- 172.21.15.32:0/4004103554 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f685c0c9e40 con 0x7f685c0cc720 2024-08-03T21:00:34.526 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:34.526+0000 7f6853400640 1 -- 172.21.15.32:0/4004103554 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 331 v331) ==== 95+0+5218 (secure 0 0 0) 0x7f684406a160 con 0x7f685c0cc720 2024-08-03T21:00:34.527 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-03T21:00:34.527 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":331,"btime":"2024-08-03T21:00:33:109208+0000","default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6439,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6835/531080160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6834","nonce":531080160},{"type":"v1","addr":"172.21.15.118:6835","nonce":531080160}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":84},{"gid":6454,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6837/2796882319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6836","nonce":2796882319},{"type":"v1","addr":"172.21.15.118:6837","nonce":2796882319}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":86},{"gid":6813,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6834/2686933890","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":2686933890},{"type":"v1","addr":"172.21.15.32:6834","nonce":2686933890}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":85},{"gid":10359,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6835/3212405498","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6833","nonce":3212405498},{"type":"v1","addr":"172.21.15.32:6835","nonce":3212405498}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":331}],"filesystems":[{"mdsmap":{"epoch":330,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-08-03T20:54:26.950518+0000","modified":"2024-08-03T21:00:28.099619+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":125,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":6}]} 2024-08-03T21:00:34.527 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 331 2024-08-03T21:00:34.529 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:34.528+0000 7f6861600640 1 -- 172.21.15.32:0/4004103554 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f6838077010 msgr2=0x7f6838079520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:00:34.529 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:34.528+0000 7f6861600640 1 --2- 172.21.15.32:0/4004103554 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f6838077010 0x7f6838079520 secure :-1 s=READY pgs=839 cs=0 l=1 rev1=1 crypto rx=0x7f684c005fd0 tx=0x7f684c009500 comp rx=0 tx=0).stop 2024-08-03T21:00:34.529 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:34.528+0000 7f6861600640 1 -- 172.21.15.32:0/4004103554 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f685c0cc720 msgr2=0x7f685c0ccb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:00:34.529 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:34.528+0000 7f6861600640 1 --2- 172.21.15.32:0/4004103554 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f685c0cc720 0x7f685c0ccb30 secure :-1 s=READY pgs=2104 cs=0 l=1 rev1=1 crypto rx=0x7f684400e590 tx=0x7f684400ea50 comp rx=0 tx=0).stop 2024-08-03T21:00:34.529 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:34.528+0000 7f6861600640 1 -- 172.21.15.32:0/4004103554 shutdown_connections 2024-08-03T21:00:34.530 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:34.528+0000 7f6861600640 1 --2- 172.21.15.32:0/4004103554 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f6838077010 0x7f6838079520 unknown :-1 s=CLOSED pgs=839 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:34.530 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:34.528+0000 7f6861600640 1 --2- 172.21.15.32:0/4004103554 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f685c15bae0 0x7f685c15bef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:34.530 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:34.528+0000 7f6861600640 1 --2- 172.21.15.32:0/4004103554 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f685c0cc720 0x7f685c0ccb30 unknown :-1 s=CLOSED pgs=2104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:34.530 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:34.528+0000 7f6861600640 1 --2- 172.21.15.32:0/4004103554 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f6840005250 0x7f685c15d410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:34.530 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:34.528+0000 7f6861600640 1 -- 172.21.15.32:0/4004103554 >> 172.21.15.32:0/4004103554 conn(0x7f685c0b1bd0 msgr2=0x7f685c1408a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T21:00:34.530 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:34.528+0000 7f6861600640 1 -- 172.21.15.32:0/4004103554 shutdown_connections 2024-08-03T21:00:34.530 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:34.528+0000 7f6861600640 1 -- 172.21.15.32:0/4004103554 wait complete. 2024-08-03T21:00:34.551 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-08-03T21:00:34.781 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:34.780+0000 7f4612e00640 1 -- 172.21.15.32:0/1029420561 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f45f0004d40 msgr2=0x7f45f0005150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:00:34.781 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:34.780+0000 7f4612e00640 1 --2- 172.21.15.32:0/1029420561 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f45f0004d40 0x7f45f0005150 secure :-1 s=READY pgs=2106 cs=0 l=1 rev1=1 crypto rx=0x7f4604009ed0 tx=0x7f45f0005830 comp rx=0 tx=0).stop 2024-08-03T21:00:34.781 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:34.780+0000 7f4610e00640 1 -- 172.21.15.32:0/1029420561 reap_dead start 2024-08-03T21:00:34.781 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:34.780+0000 7f4612e00640 1 -- 172.21.15.32:0/1029420561 shutdown_connections 2024-08-03T21:00:34.782 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:34.780+0000 7f4612e00640 1 -- 172.21.15.32:0/1029420561 >> 172.21.15.32:0/1029420561 conn(0x7f460c0b1bd0 msgr2=0x7f460c0b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T21:00:34.782 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:34.780+0000 7f4612e00640 1 -- 172.21.15.32:0/1029420561 shutdown_connections 2024-08-03T21:00:34.782 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:34.780+0000 7f4612e00640 1 -- 172.21.15.32:0/1029420561 wait complete. 2024-08-03T21:00:34.783 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:34.783+0000 7f4612e00640 1 Processor -- start 2024-08-03T21:00:34.783 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:34.783+0000 7f4612e00640 1 -- start start 2024-08-03T21:00:34.785 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:34.784+0000 7f4612e00640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f45f0004d40 0x7f460c1f5c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:00:34.785 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:34.784+0000 7f4612e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f460c0c4e40 0x7f460c138dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:00:34.786 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:34.784+0000 7f4612e00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f460c1429c0 0x7f460c142dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:00:34.786 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:34.784+0000 7f4612e00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f460c0cf840 con 0x7f460c0c4e40 2024-08-03T21:00:34.786 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:34.784+0000 7f4612e00640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f460c145f30 con 0x7f460c1429c0 2024-08-03T21:00:34.786 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:34.784+0000 7f4612e00640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7f460c0cf9c0 con 0x7f45f0004d40 2024-08-03T21:00:34.786 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:34.785+0000 7f4603400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f460c0c4e40 0x7f460c138dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:00:34.786 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:34.785+0000 7f4603400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f460c0c4e40 0x7f460c138dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:52832/0 (socket says 172.21.15.32:52832) 2024-08-03T21:00:34.786 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:34.785+0000 7f4603400640 1 -- 172.21.15.32:0/2341019920 learned_addr learned my addr 172.21.15.32:0/2341019920 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T21:00:34.786 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:34.785+0000 7f460be00640 1 --2- 172.21.15.32:0/2341019920 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f45f0004d40 0x7f460c1f5c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:00:34.786 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:34.785+0000 7f4610e00640 1 --2- 172.21.15.32:0/2341019920 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f460c1429c0 0x7f460c142dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:00:34.786 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:34.785+0000 7f4603400640 1 -- 172.21.15.32:0/2341019920 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f45f0004d40 msgr2=0x7f460c1f5c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:00:34.786 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:34.785+0000 7f4603400640 1 --2- 172.21.15.32:0/2341019920 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f45f0004d40 0x7f460c1f5c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:34.787 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:34.785+0000 7f4603400640 1 -- 172.21.15.32:0/2341019920 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f460c1429c0 msgr2=0x7f460c142dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:00:34.787 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:34.785+0000 7f4603400640 1 --2- 172.21.15.32:0/2341019920 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f460c1429c0 0x7f460c142dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:34.787 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:34.785+0000 7f4603400640 1 -- 172.21.15.32:0/2341019920 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f460c143310 con 0x7f460c0c4e40 2024-08-03T21:00:34.788 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:34.785+0000 7f4603400640 1 --2- 172.21.15.32:0/2341019920 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f460c0c4e40 0x7f460c138dd0 secure :-1 s=READY pgs=2107 cs=0 l=1 rev1=1 crypto rx=0x7f45f4001ad0 tx=0x7f45f40039c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T21:00:34.788 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:34.786+0000 7f4608c00640 1 -- 172.21.15.32:0/2341019920 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45f4015070 con 0x7f460c0c4e40 2024-08-03T21:00:34.788 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:34.786+0000 7f4608c00640 1 -- 172.21.15.32:0/2341019920 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f45f4011920 con 0x7f460c0c4e40 2024-08-03T21:00:34.788 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:34.786+0000 7f4608c00640 1 -- 172.21.15.32:0/2341019920 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45f4011d70 con 0x7f460c0c4e40 2024-08-03T21:00:34.788 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:34.786+0000 7f4612e00640 1 -- 172.21.15.32:0/2341019920 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f460c1f6300 con 0x7f460c0c4e40 2024-08-03T21:00:34.788 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:34.786+0000 7f4612e00640 1 -- 172.21.15.32:0/2341019920 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f460c1f6880 con 0x7f460c0c4e40 2024-08-03T21:00:34.788 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:34.787+0000 7f4612e00640 1 -- 172.21.15.32:0/2341019920 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f460c144f90 con 0x7f460c0c4e40 2024-08-03T21:00:34.788 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:34.788+0000 7f4608c00640 1 -- 172.21.15.32:0/2341019920 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f45f4018070 con 0x7f460c0c4e40 2024-08-03T21:00:34.790 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:34.789+0000 7f4608c00640 1 --2- 172.21.15.32:0/2341019920 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f45e8077040 0x7f45e8079550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:00:34.790 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:34.789+0000 7f4608c00640 1 -- 172.21.15.32:0/2341019920 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(125..125 src has 1..125) ==== 9294+0+0 (secure 0 0 0) 0x7f45f409a450 con 0x7f460c0c4e40 2024-08-03T21:00:34.790 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:34.790+0000 7f460be00640 1 --2- 172.21.15.32:0/2341019920 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f45e8077040 0x7f45e8079550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:00:34.791 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:34.790+0000 7f460be00640 1 --2- 172.21.15.32:0/2341019920 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f45e8077040 0x7f45e8079550 secure :-1 s=READY pgs=840 cs=0 l=1 rev1=1 crypto rx=0x7f45fc009cd0 tx=0x7f45fc005f90 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T21:00:34.794 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:34.794+0000 7f4608c00640 1 -- 172.21.15.32:0/2341019920 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7f460c143310 con 0x7f460c0c4e40 2024-08-03T21:00:35.035 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:35.034+0000 7f4612e00640 1 -- 172.21.15.32:0/2341019920 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f460c0aff50 con 0x7f460c0c4e40 2024-08-03T21:00:35.035 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:35.035+0000 7f4608c00640 1 -- 172.21.15.32:0/2341019920 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 331 v331) ==== 95+0+5218 (secure 0 0 0) 0x7f45f40621f0 con 0x7f460c0c4e40 2024-08-03T21:00:35.036 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-03T21:00:35.036 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":331,"btime":"2024-08-03T21:00:33:109208+0000","default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6439,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6835/531080160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6834","nonce":531080160},{"type":"v1","addr":"172.21.15.118:6835","nonce":531080160}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":84},{"gid":6454,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6837/2796882319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6836","nonce":2796882319},{"type":"v1","addr":"172.21.15.118:6837","nonce":2796882319}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":86},{"gid":6813,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6834/2686933890","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":2686933890},{"type":"v1","addr":"172.21.15.32:6834","nonce":2686933890}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":85},{"gid":10359,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6835/3212405498","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6833","nonce":3212405498},{"type":"v1","addr":"172.21.15.32:6835","nonce":3212405498}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":331}],"filesystems":[{"mdsmap":{"epoch":330,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-08-03T20:54:26.950518+0000","modified":"2024-08-03T21:00:28.099619+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":125,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":6}]} 2024-08-03T21:00:35.036 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 331 2024-08-03T21:00:35.038 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:35.037+0000 7f4612e00640 1 -- 172.21.15.32:0/2341019920 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f45e8077040 msgr2=0x7f45e8079550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:00:35.038 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:35.037+0000 7f4612e00640 1 --2- 172.21.15.32:0/2341019920 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f45e8077040 0x7f45e8079550 secure :-1 s=READY pgs=840 cs=0 l=1 rev1=1 crypto rx=0x7f45fc009cd0 tx=0x7f45fc005f90 comp rx=0 tx=0).stop 2024-08-03T21:00:35.038 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:35.037+0000 7f4612e00640 1 -- 172.21.15.32:0/2341019920 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f460c0c4e40 msgr2=0x7f460c138dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:00:35.038 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:35.037+0000 7f4612e00640 1 --2- 172.21.15.32:0/2341019920 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f460c0c4e40 0x7f460c138dd0 secure :-1 s=READY pgs=2107 cs=0 l=1 rev1=1 crypto rx=0x7f45f4001ad0 tx=0x7f45f40039c0 comp rx=0 tx=0).stop 2024-08-03T21:00:35.038 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:35.037+0000 7f4612e00640 1 -- 172.21.15.32:0/2341019920 shutdown_connections 2024-08-03T21:00:35.039 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:35.037+0000 7f4612e00640 1 --2- 172.21.15.32:0/2341019920 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f45e8077040 0x7f45e8079550 unknown :-1 s=CLOSED pgs=840 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:35.039 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:35.037+0000 7f4612e00640 1 --2- 172.21.15.32:0/2341019920 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f460c1429c0 0x7f460c142dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:35.039 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:35.037+0000 7f4612e00640 1 --2- 172.21.15.32:0/2341019920 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f460c0c4e40 0x7f460c138dd0 unknown :-1 s=CLOSED pgs=2107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:35.039 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:35.037+0000 7f4612e00640 1 --2- 172.21.15.32:0/2341019920 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f45f0004d40 0x7f460c1f5c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:35.039 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:35.037+0000 7f4612e00640 1 -- 172.21.15.32:0/2341019920 >> 172.21.15.32:0/2341019920 conn(0x7f460c0b1bd0 msgr2=0x7f460c0b0e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T21:00:35.039 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:35.038+0000 7f4612e00640 1 -- 172.21.15.32:0/2341019920 shutdown_connections 2024-08-03T21:00:35.039 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:35.038+0000 7f4612e00640 1 -- 172.21.15.32:0/2341019920 wait complete. 2024-08-03T21:00:35.064 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-08-03T21:00:35.297 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:35.296+0000 7f5439200640 1 -- 172.21.15.32:0/4171087408 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f54340747b0 msgr2=0x7f54340cc8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:00:35.298 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:35.296+0000 7f5439200640 1 --2- 172.21.15.32:0/4171087408 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f54340747b0 0x7f54340cc8e0 secure :-1 s=READY pgs=2108 cs=0 l=1 rev1=1 crypto rx=0x7f54340b2860 tx=0x7f542c02aa30 comp rx=0 tx=0).stop 2024-08-03T21:00:35.298 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:35.297+0000 7f5439200640 1 -- 172.21.15.32:0/4171087408 shutdown_connections 2024-08-03T21:00:35.298 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:35.297+0000 7f5439200640 1 --2- 172.21.15.32:0/4171087408 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f54340747b0 0x7f54340cc8e0 unknown :-1 s=CLOSED pgs=2108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:35.298 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:35.297+0000 7f5439200640 1 --2- 172.21.15.32:0/4171087408 >> v2:172.21.15.118:3300/0 conn(0x7f5434073e60 0x7f5434074270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:35.298 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:35.297+0000 7f5439200640 1 --2- 172.21.15.32:0/4171087408 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f543406e930 0x7f5434144f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:35.298 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:35.297+0000 7f5439200640 1 -- 172.21.15.32:0/4171087408 >> 172.21.15.32:0/4171087408 conn(0x7f54340b1bd0 msgr2=0x7f54340b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T21:00:35.298 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:35.297+0000 7f5439200640 1 -- 172.21.15.32:0/4171087408 shutdown_connections 2024-08-03T21:00:35.298 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:35.297+0000 7f5439200640 1 -- 172.21.15.32:0/4171087408 wait complete. 2024-08-03T21:00:35.299 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:35.299+0000 7f5439200640 1 Processor -- start 2024-08-03T21:00:35.299 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:35.299+0000 7f5439200640 1 -- start start 2024-08-03T21:00:35.302 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:35.300+0000 7f5439200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f543406e930 0x7f54341f5f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:00:35.302 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:35.300+0000 7f5439200640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f5434073e60 0x7f54341ebee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:00:35.302 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:35.300+0000 7f5439200640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f54340747b0 0x7f54340b4020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:00:35.302 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:35.301+0000 7f5439200640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f54340cf9c0 con 0x7f543406e930 2024-08-03T21:00:35.302 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:35.301+0000 7f5439200640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f54340cf840 con 0x7f5434073e60 2024-08-03T21:00:35.302 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:35.301+0000 7f5439200640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7f54340cce20 con 0x7f54340747b0 2024-08-03T21:00:35.302 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:35.301+0000 7f5431e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f543406e930 0x7f54341f5f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:00:35.302 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:35.301+0000 7f5431e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f543406e930 0x7f54341f5f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:52856/0 (socket says 172.21.15.32:52856) 2024-08-03T21:00:35.302 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:35.301+0000 7f5431e00640 1 -- 172.21.15.32:0/2949218715 learned_addr learned my addr 172.21.15.32:0/2949218715 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T21:00:35.302 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:35.301+0000 7f5431400640 1 --2- 172.21.15.32:0/2949218715 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f5434073e60 0x7f54341ebee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:00:35.303 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:35.301+0000 7f5432800640 1 --2- 172.21.15.32:0/2949218715 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f54340747b0 0x7f54340b4020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:00:35.303 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:35.301+0000 7f5431e00640 1 -- 172.21.15.32:0/2949218715 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f54340747b0 msgr2=0x7f54340b4020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:00:35.303 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:35.301+0000 7f5431e00640 1 --2- 172.21.15.32:0/2949218715 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f54340747b0 0x7f54340b4020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:35.303 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:35.301+0000 7f5431e00640 1 -- 172.21.15.32:0/2949218715 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f5434073e60 msgr2=0x7f54341ebee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:00:35.303 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:35.301+0000 7f5431e00640 1 --2- 172.21.15.32:0/2949218715 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f5434073e60 0x7f54341ebee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:35.303 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:35.301+0000 7f5431e00640 1 -- 172.21.15.32:0/2949218715 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f54341ec420 con 0x7f543406e930 2024-08-03T21:00:35.303 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:35.301+0000 7f5431e00640 1 --2- 172.21.15.32:0/2949218715 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f543406e930 0x7f54341f5f80 secure :-1 s=READY pgs=2109 cs=0 l=1 rev1=1 crypto rx=0x7f542400d7a0 tx=0x7f542400dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T21:00:35.304 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:35.302+0000 7f542a000640 1 -- 172.21.15.32:0/2949218715 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5424017070 con 0x7f543406e930 2024-08-03T21:00:35.304 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:35.302+0000 7f542a000640 1 -- 172.21.15.32:0/2949218715 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f5424009d20 con 0x7f543406e930 2024-08-03T21:00:35.304 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:35.302+0000 7f542a000640 1 -- 172.21.15.32:0/2949218715 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5424004280 con 0x7f543406e930 2024-08-03T21:00:35.304 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:35.302+0000 7f5439200640 1 -- 172.21.15.32:0/2949218715 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5434141a10 con 0x7f543406e930 2024-08-03T21:00:35.304 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:35.302+0000 7f5439200640 1 -- 172.21.15.32:0/2949218715 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5434141fd0 con 0x7f543406e930 2024-08-03T21:00:35.304 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:35.303+0000 7f5439200640 1 -- 172.21.15.32:0/2949218715 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5434074270 con 0x7f543406e930 2024-08-03T21:00:35.305 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:35.304+0000 7f542a000640 1 -- 172.21.15.32:0/2949218715 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f5424005020 con 0x7f543406e930 2024-08-03T21:00:35.306 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:35.305+0000 7f542a000640 1 --2- 172.21.15.32:0/2949218715 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f5414077010 0x7f5414079520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:00:35.306 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:35.305+0000 7f542a000640 1 -- 172.21.15.32:0/2949218715 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(125..125 src has 1..125) ==== 9294+0+0 (secure 0 0 0) 0x7f5424099ae0 con 0x7f543406e930 2024-08-03T21:00:35.306 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:35.306+0000 7f5431400640 1 --2- 172.21.15.32:0/2949218715 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f5414077010 0x7f5414079520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:00:35.307 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:35.307+0000 7f5431400640 1 --2- 172.21.15.32:0/2949218715 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f5414077010 0x7f5414079520 secure :-1 s=READY pgs=841 cs=0 l=1 rev1=1 crypto rx=0x7f54340b0b80 tx=0x7f5420005f90 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T21:00:35.310 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:35.310+0000 7f542a000640 1 -- 172.21.15.32:0/2949218715 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7f5424020020 con 0x7f543406e930 2024-08-03T21:00:35.524 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:35.523+0000 7f5439200640 1 -- 172.21.15.32:0/2949218715 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f54341457d0 con 0x7f543406e930 2024-08-03T21:00:35.525 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:35.524+0000 7f542a000640 1 -- 172.21.15.32:0/2949218715 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v125) ==== 74+0+22023 (secure 0 0 0) 0x7f5424061ec0 con 0x7f543406e930 2024-08-03T21:00:35.525 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-03T21:00:35.526 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":125,"fsid":"40ab141a-01d7-49b9-ae3c-2edec9876e95","created":"2024-08-03T20:47:14.957362+0000","modified":"2024-08-03T21:00:28.099308+0000","last_up_change":"2024-08-03T20:47:28.874451+0000","last_in_change":"2024-08-03T20:47:20.079442+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":13,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-08-03T20:47:28.261300+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_metadata","create_time":"2024-08-03T20:54:24.911555+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"82","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs_data","create_time":"2024-08-03T20:54:25.917207+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"83","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5a83e759-9035-438b-889a-0a6af1f0a0d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6816","nonce":3066156116},{"type":"v1","addr":"172.21.15.32:6817","nonce":3066156116}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6818","nonce":3066156116},{"type":"v1","addr":"172.21.15.32:6819","nonce":3066156116}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6822","nonce":3066156116},{"type":"v1","addr":"172.21.15.32:6823","nonce":3066156116}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6820","nonce":3066156116},{"type":"v1","addr":"172.21.15.32:6821","nonce":3066156116}]},"public_addr":"172.21.15.32:6817/3066156116","cluster_addr":"172.21.15.32:6819/3066156116","heartbeat_back_addr":"172.21.15.32:6823/3066156116","heartbeat_front_addr":"172.21.15.32:6821/3066156116","state":["exists","up"]},{"osd":1,"uuid":"d7c5c210-da4f-49f9-b9e7-1dfb02daa4bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6824","nonce":1551241759},{"type":"v1","addr":"172.21.15.32:6825","nonce":1551241759}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6826","nonce":1551241759},{"type":"v1","addr":"172.21.15.32:6827","nonce":1551241759}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6830","nonce":1551241759},{"type":"v1","addr":"172.21.15.32:6831","nonce":1551241759}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6828","nonce":1551241759},{"type":"v1","addr":"172.21.15.32:6829","nonce":1551241759}]},"public_addr":"172.21.15.32:6825/1551241759","cluster_addr":"172.21.15.32:6827/1551241759","heartbeat_back_addr":"172.21.15.32:6831/1551241759","heartbeat_front_addr":"172.21.15.32:6829/1551241759","state":["exists","up"]},{"osd":2,"uuid":"57f2e6de-a201-45c2-8981-8e56a4875ebd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6808","nonce":3806708944},{"type":"v1","addr":"172.21.15.32:6809","nonce":3806708944}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6810","nonce":3806708944},{"type":"v1","addr":"172.21.15.32:6811","nonce":3806708944}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6814","nonce":3806708944},{"type":"v1","addr":"172.21.15.32:6815","nonce":3806708944}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6812","nonce":3806708944},{"type":"v1","addr":"172.21.15.32:6813","nonce":3806708944}]},"public_addr":"172.21.15.32:6809/3806708944","cluster_addr":"172.21.15.32:6811/3806708944","heartbeat_back_addr":"172.21.15.32:6815/3806708944","heartbeat_front_addr":"172.21.15.32:6813/3806708944","state":["exists","up"]},{"osd":3,"uuid":"6439a280-da45-4148-8130-f16323935ff3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6800","nonce":9088303},{"type":"v1","addr":"172.21.15.32:6801","nonce":9088303}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6802","nonce":9088303},{"type":"v1","addr":"172.21.15.32:6803","nonce":9088303}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6806","nonce":9088303},{"type":"v1","addr":"172.21.15.32:6807","nonce":9088303}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6804","nonce":9088303},{"type":"v1","addr":"172.21.15.32:6805","nonce":9088303}]},"public_addr":"172.21.15.32:6801/9088303","cluster_addr":"172.21.15.32:6803/9088303","heartbeat_back_addr":"172.21.15.32:6807/9088303","heartbeat_front_addr":"172.21.15.32:6805/9088303","state":["exists","up"]},{"osd":4,"uuid":"4f902eff-1e18-418d-82c9-2151c69524fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6802","nonce":864752275},{"type":"v1","addr":"172.21.15.118:6803","nonce":864752275}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6804","nonce":864752275},{"type":"v1","addr":"172.21.15.118:6805","nonce":864752275}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6808","nonce":864752275},{"type":"v1","addr":"172.21.15.118:6809","nonce":864752275}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6806","nonce":864752275},{"type":"v1","addr":"172.21.15.118:6807","nonce":864752275}]},"public_addr":"172.21.15.118:6803/864752275","cluster_addr":"172.21.15.118:6805/864752275","heartbeat_back_addr":"172.21.15.118:6809/864752275","heartbeat_front_addr":"172.21.15.118:6807/864752275","state":["exists","up"]},{"osd":5,"uuid":"bfd9feac-dd06-458c-91b9-af7f175f612c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6810","nonce":2724891770},{"type":"v1","addr":"172.21.15.118:6811","nonce":2724891770}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6812","nonce":2724891770},{"type":"v1","addr":"172.21.15.118:6813","nonce":2724891770}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6816","nonce":2724891770},{"type":"v1","addr":"172.21.15.118:6817","nonce":2724891770}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6814","nonce":2724891770},{"type":"v1","addr":"172.21.15.118:6815","nonce":2724891770}]},"public_addr":"172.21.15.118:6811/2724891770","cluster_addr":"172.21.15.118:6813/2724891770","heartbeat_back_addr":"172.21.15.118:6817/2724891770","heartbeat_front_addr":"172.21.15.118:6815/2724891770","state":["exists","up"]},{"osd":6,"uuid":"5ed5a279-9d8e-416d-bbc7-7edd3e6d5705","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6825","nonce":801947386},{"type":"v1","addr":"172.21.15.118:6827","nonce":801947386}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6828","nonce":801947386},{"type":"v1","addr":"172.21.15.118:6829","nonce":801947386}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6832","nonce":801947386},{"type":"v1","addr":"172.21.15.118:6833","nonce":801947386}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6830","nonce":801947386},{"type":"v1","addr":"172.21.15.118:6831","nonce":801947386}]},"public_addr":"172.21.15.118:6827/801947386","cluster_addr":"172.21.15.118:6829/801947386","heartbeat_back_addr":"172.21.15.118:6833/801947386","heartbeat_front_addr":"172.21.15.118:6831/801947386","state":["exists","up"]},{"osd":7,"uuid":"f2086c47-875e-4451-a436-2008d1ea16af","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6818","nonce":653000048},{"type":"v1","addr":"172.21.15.118:6819","nonce":653000048}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6820","nonce":653000048},{"type":"v1","addr":"172.21.15.118:6821","nonce":653000048}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6824","nonce":653000048},{"type":"v1","addr":"172.21.15.118:6826","nonce":653000048}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6822","nonce":653000048},{"type":"v1","addr":"172.21.15.118:6823","nonce":653000048}]},"public_addr":"172.21.15.118:6819/653000048","cluster_addr":"172.21.15.118:6821/653000048","heartbeat_back_addr":"172.21.15.118:6826/653000048","heartbeat_front_addr":"172.21.15.118:6823/653000048","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-03T20:47:25.491994+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-03T20:47:25.587358+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-03T20:47:25.454175+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-03T20:47:25.377734+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-03T20:47:25.412351+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-03T20:47:25.499815+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-03T20:47:25.575044+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-03T20:47:25.552383+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"13.8","mappings":[{"from":2,"to":3}]},{"pgid":"13.14","mappings":[{"from":6,"to":5}]},{"pgid":"13.23","mappings":[{"from":1,"to":5}]},{"pgid":"13.32","mappings":[{"from":1,"to":5}]},{"pgid":"13.34","mappings":[{"from":6,"to":5}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.32:6835/1660875176":"2024-08-04T21:00:28.099287+0000","172.21.15.32:6833/1660875176":"2024-08-04T21:00:28.099287+0000","172.21.15.32:6835/2824868449":"2024-08-04T21:00:19.050715+0000","172.21.15.32:6835/73253044":"2024-08-04T21:00:10.997224+0000","172.21.15.32:6835/2017655932":"2024-08-04T21:00:04.959586+0000","172.21.15.32:6833/2017655932":"2024-08-04T21:00:04.959586+0000","172.21.15.32:6835/3127944613":"2024-08-04T20:59:46.766712+0000","172.21.15.32:6835/2106229503":"2024-08-04T20:59:40.719664+0000","172.21.15.32:6833/2106229503":"2024-08-04T20:59:40.719664+0000","172.21.15.32:6835/2173042128":"2024-08-04T20:59:28.193772+0000","172.21.15.32:6833/2173042128":"2024-08-04T20:59:28.193772+0000","172.21.15.32:6835/506775245":"2024-08-04T20:59:03.076256+0000","172.21.15.32:6833/506775245":"2024-08-04T20:59:03.076256+0000","172.21.15.32:6833/556650445":"2024-08-04T20:58:57.063754+0000","172.21.15.32:6835/2156596385":"2024-08-04T20:58:35.073838+0000","172.21.15.32:6833/2156596385":"2024-08-04T20:58:35.073838+0000","172.21.15.32:6835/3669493978":"2024-08-04T20:58:30.064676+0000","172.21.15.32:6833/2178506907":"2024-08-04T20:56:40.045549+0000","172.21.15.32:6835/556650445":"2024-08-04T20:58:57.063754+0000","172.21.15.32:6835/2278473729":"2024-08-04T20:56:13.041693+0000","172.21.15.32:6835/2345472481":"2024-08-04T20:57:25.968590+0000","172.21.15.32:6833/3940515773":"2024-08-04T20:58:24.290690+0000","172.21.15.32:6833/2600386707":"2024-08-04T20:57:50.063964+0000","172.21.15.32:6835/2178506907":"2024-08-04T20:56:40.045549+0000","172.21.15.32:6833/73253044":"2024-08-04T21:00:10.997224+0000","172.21.15.32:6833/2292690833":"2024-08-04T20:55:20.021329+0000","172.21.15.32:6835/2803901497":"2024-08-04T20:55:55.181144+0000","172.21.15.32:6835/2194902489":"2024-08-04T20:55:00.359126+0000","172.21.15.32:6833/2824868449":"2024-08-04T21:00:19.050715+0000","172.21.15.32:6833/3127944613":"2024-08-04T20:59:46.766712+0000","172.21.15.32:6833/2803901497":"2024-08-04T20:55:55.181144+0000","172.21.15.32:6835/3191654276":"2024-08-04T20:55:42.433364+0000","172.21.15.32:6833/2242533881":"2024-08-04T20:56:48.048099+0000","172.21.15.32:6833/2283924374":"2024-08-04T20:57:34.396782+0000","172.21.15.32:6835/1172050551":"2024-08-04T20:55:26.214113+0000","172.21.15.32:6835/1016338829":"2024-08-04T20:55:06.071485+0000","172.21.15.32:6833/1474984145":"2024-08-04T20:57:40.052808+0000","172.21.15.32:6833/3669493978":"2024-08-04T20:58:30.064676+0000","172.21.15.32:6835/756683988":"2024-08-04T20:55:14.119228+0000","172.21.15.32:6833/2194902489":"2024-08-04T20:55:00.359126+0000","172.21.15.32:6835/2806168786":"2024-08-04T20:56:37.279160+0000","172.21.15.32:6833/2278473729":"2024-08-04T20:56:13.041693+0000","172.21.15.32:6833/3191654276":"2024-08-04T20:55:42.433364+0000","172.21.15.32:6835/3431300225":"2024-08-04T20:56:21.088927+0000","172.21.15.32:6833/4172198298":"2024-08-04T20:56:04.256182+0000","172.21.15.32:6833/3431300225":"2024-08-04T20:56:21.088927+0000","172.21.15.32:6835/2292690833":"2024-08-04T20:55:20.021329+0000","172.21.15.32:6833/2806168786":"2024-08-04T20:56:37.279160+0000","172.21.15.32:6833/1172050551":"2024-08-04T20:55:26.214113+0000","172.21.15.32:6835/4172198298":"2024-08-04T20:56:04.256182+0000","172.21.15.32:6835/3409752696":"2024-08-04T20:58:18.243140+0000","172.21.15.32:6833/2274054575":"2024-08-04T20:57:55.062490+0000","172.21.15.32:6833/1016338829":"2024-08-04T20:55:06.071485+0000","172.21.15.32:6833/1707454554":"2024-08-04T20:54:38.409411+0000","172.21.15.32:6835/2242533881":"2024-08-04T20:56:48.048099+0000","172.21.15.32:6833/739874257":"2024-08-04T20:56:56.101271+0000","172.21.15.32:6835/739874257":"2024-08-04T20:56:56.101271+0000","172.21.15.32:6833/2362145329":"2024-08-04T20:57:02.150199+0000","172.21.15.32:6833/3771304550":"2024-08-04T20:55:58.212752+0000","172.21.15.32:6835/2362145329":"2024-08-04T20:57:02.150199+0000","172.21.15.32:6835/3940515773":"2024-08-04T20:58:24.290690+0000","172.21.15.32:6833/1885034124":"2024-08-04T20:57:08.190707+0000","172.21.15.32:6833/511221267":"2024-08-04T20:57:45.062448+0000","172.21.15.32:6835/3771304550":"2024-08-04T20:55:58.212752+0000","172.21.15.32:6835/1885034124":"2024-08-04T20:57:08.190707+0000","172.21.15.32:6833/2345472481":"2024-08-04T20:57:25.968590+0000","172.21.15.32:6833/265163622":"2024-08-04T20:57:28.354509+0000","172.21.15.32:6835/265163622":"2024-08-04T20:57:28.354509+0000","172.21.15.32:6835/2274054575":"2024-08-04T20:57:55.062490+0000","172.21.15.32:6835/2283924374":"2024-08-04T20:57:34.396782+0000","172.21.15.32:6833/756683988":"2024-08-04T20:55:14.119228+0000","172.21.15.32:6835/1474984145":"2024-08-04T20:57:40.052808+0000","172.21.15.32:6835/511221267":"2024-08-04T20:57:45.062448+0000","172.21.15.32:6835/1707454554":"2024-08-04T20:54:38.409411+0000","172.21.15.32:6835/2600386707":"2024-08-04T20:57:50.063964+0000","172.21.15.32:6833/675070591":"2024-08-04T20:58:00.068114+0000","172.21.15.32:6833/3340816283":"2024-08-04T20:58:15.214542+0000","172.21.15.32:6835/3340816283":"2024-08-04T20:58:15.214542+0000","172.21.15.32:6835/675070591":"2024-08-04T20:58:00.068114+0000","172.21.15.32:6833/3409752696":"2024-08-04T20:58:18.243140+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-08-03T21:00:35.527 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:35.527+0000 7f5439200640 1 -- 172.21.15.32:0/2949218715 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f5414077010 msgr2=0x7f5414079520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:00:35.528 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:35.527+0000 7f5439200640 1 --2- 172.21.15.32:0/2949218715 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f5414077010 0x7f5414079520 secure :-1 s=READY pgs=841 cs=0 l=1 rev1=1 crypto rx=0x7f54340b0b80 tx=0x7f5420005f90 comp rx=0 tx=0).stop 2024-08-03T21:00:35.528 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:35.527+0000 7f5439200640 1 -- 172.21.15.32:0/2949218715 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f543406e930 msgr2=0x7f54341f5f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:00:35.528 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:35.527+0000 7f5439200640 1 --2- 172.21.15.32:0/2949218715 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f543406e930 0x7f54341f5f80 secure :-1 s=READY pgs=2109 cs=0 l=1 rev1=1 crypto rx=0x7f542400d7a0 tx=0x7f542400dc60 comp rx=0 tx=0).stop 2024-08-03T21:00:35.528 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:35.527+0000 7f5439200640 1 -- 172.21.15.32:0/2949218715 shutdown_connections 2024-08-03T21:00:35.528 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:35.527+0000 7f5439200640 1 --2- 172.21.15.32:0/2949218715 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f5414077010 0x7f5414079520 unknown :-1 s=CLOSED pgs=841 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:35.528 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:35.527+0000 7f5439200640 1 --2- 172.21.15.32:0/2949218715 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f5434073e60 0x7f54341ebee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:35.528 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:35.527+0000 7f5439200640 1 --2- 172.21.15.32:0/2949218715 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f54340747b0 0x7f54340b4020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:35.528 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:35.527+0000 7f5439200640 1 --2- 172.21.15.32:0/2949218715 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f543406e930 0x7f54341f5f80 unknown :-1 s=CLOSED pgs=2109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:35.528 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:35.527+0000 7f5439200640 1 -- 172.21.15.32:0/2949218715 >> 172.21.15.32:0/2949218715 conn(0x7f54340b1bd0 msgr2=0x7f54340cbac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T21:00:35.528 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:35.527+0000 7f5439200640 1 -- 172.21.15.32:0/2949218715 shutdown_connections 2024-08-03T21:00:35.528 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:35.527+0000 7f5439200640 1 -- 172.21.15.32:0/2949218715 wait complete. 2024-08-03T21:00:35.551 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-08-03T21:00:35.778 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:35.777+0000 7ffab1000640 1 -- 172.21.15.32:0/2847346053 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ffaac15fff0 msgr2=0x7ffaac160400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:00:35.779 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:35.777+0000 7ffab1000640 1 --2- 172.21.15.32:0/2847346053 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ffaac15fff0 0x7ffaac160400 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7ffaa4005fd0 tx=0x7ffaa402aaa0 comp rx=0 tx=0).stop 2024-08-03T21:00:35.779 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:35.778+0000 7ffab1000640 1 -- 172.21.15.32:0/2847346053 shutdown_connections 2024-08-03T21:00:35.779 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:35.778+0000 7ffab1000640 1 --2- 172.21.15.32:0/2847346053 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ffaac15fff0 0x7ffaac160400 unknown :-1 s=CLOSED pgs=628 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:35.779 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:35.778+0000 7ffab1000640 1 --2- 172.21.15.32:0/2847346053 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7ffaac06e930 0x7ffaac144f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:35.779 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:35.778+0000 7ffab1000640 1 -- 172.21.15.32:0/2847346053 >> 172.21.15.32:0/2847346053 conn(0x7ffaac0b1bd0 msgr2=0x7ffaac0b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T21:00:35.779 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:35.778+0000 7ffab1000640 1 -- 172.21.15.32:0/2847346053 shutdown_connections 2024-08-03T21:00:35.780 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:35.778+0000 7ffab1000640 1 -- 172.21.15.32:0/2847346053 wait complete. 2024-08-03T21:00:35.781 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:35.780+0000 7ffab1000640 1 Processor -- start 2024-08-03T21:00:35.781 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:35.780+0000 7ffab1000640 1 -- start start 2024-08-03T21:00:35.783 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:35.781+0000 7ffab1000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ffaac06e930 0x7ffaac0b44c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:00:35.784 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:35.781+0000 7ffab1000640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7ffaac073e60 0x7ffaac130580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:00:35.784 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:35.781+0000 7ffab1000640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ffaac15fff0 0x7ffaac15ca70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:00:35.784 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:35.781+0000 7ffab1000640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7ffaac0cf840 con 0x7ffaac06e930 2024-08-03T21:00:35.784 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:35.781+0000 7ffab1000640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7ffaac145f30 con 0x7ffaac15fff0 2024-08-03T21:00:35.784 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:35.781+0000 7ffab1000640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7ffaac0cf9c0 con 0x7ffaac073e60 2024-08-03T21:00:35.784 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:35.782+0000 7ffaa9e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ffaac06e930 0x7ffaac0b44c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:00:35.784 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:35.782+0000 7ffaa9e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ffaac06e930 0x7ffaac0b44c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:52860/0 (socket says 172.21.15.32:52860) 2024-08-03T21:00:35.784 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:35.782+0000 7ffaa9e00640 1 -- 172.21.15.32:0/428093661 learned_addr learned my addr 172.21.15.32:0/428093661 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T21:00:35.784 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:35.782+0000 7ffaaa800640 1 --2- 172.21.15.32:0/428093661 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ffaac15fff0 0x7ffaac15ca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:00:35.784 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:35.782+0000 7ffaa1400640 1 --2- 172.21.15.32:0/428093661 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7ffaac073e60 0x7ffaac130580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:00:35.784 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:35.782+0000 7ffaa9e00640 1 -- 172.21.15.32:0/428093661 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7ffaac073e60 msgr2=0x7ffaac130580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:00:35.784 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:35.782+0000 7ffaa9e00640 1 --2- 172.21.15.32:0/428093661 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7ffaac073e60 0x7ffaac130580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:35.784 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:35.782+0000 7ffaa9e00640 1 -- 172.21.15.32:0/428093661 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ffaac15fff0 msgr2=0x7ffaac15ca70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:00:35.785 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:35.782+0000 7ffaa9e00640 1 --2- 172.21.15.32:0/428093661 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ffaac15fff0 0x7ffaac15ca70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:35.785 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:35.782+0000 7ffaa9e00640 1 -- 172.21.15.32:0/428093661 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffaac0b4a00 con 0x7ffaac06e930 2024-08-03T21:00:35.785 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:35.782+0000 7ffaa1400640 1 --2- 172.21.15.32:0/428093661 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7ffaac073e60 0x7ffaac130580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-03T21:00:35.785 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:35.783+0000 7ffaa9e00640 1 --2- 172.21.15.32:0/428093661 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ffaac06e930 0x7ffaac0b44c0 secure :-1 s=READY pgs=2110 cs=0 l=1 rev1=1 crypto rx=0x7ffa9c00e790 tx=0x7ffa9c00ec50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T21:00:35.785 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:35.783+0000 7ffaa2a00640 1 -- 172.21.15.32:0/428093661 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa9c00eea0 con 0x7ffaac06e930 2024-08-03T21:00:35.785 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:35.783+0000 7ffaa2a00640 1 -- 172.21.15.32:0/428093661 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7ffa9c00cdb0 con 0x7ffaac06e930 2024-08-03T21:00:35.785 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:35.783+0000 7ffaa2a00640 1 -- 172.21.15.32:0/428093661 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa9c010690 con 0x7ffaac06e930 2024-08-03T21:00:35.785 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:35.783+0000 7ffab1000640 1 -- 172.21.15.32:0/428093661 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffaac13d8f0 con 0x7ffaac06e930 2024-08-03T21:00:35.785 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:35.783+0000 7ffab1000640 1 -- 172.21.15.32:0/428093661 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffaac0b3700 con 0x7ffaac06e930 2024-08-03T21:00:35.786 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:35.784+0000 7ffab1000640 1 -- 172.21.15.32:0/428093661 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ffaac160b70 con 0x7ffaac06e930 2024-08-03T21:00:35.786 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:35.785+0000 7ffaa2a00640 1 -- 172.21.15.32:0/428093661 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7ffa9c010830 con 0x7ffaac06e930 2024-08-03T21:00:35.787 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:35.786+0000 7ffaa2a00640 1 --2- 172.21.15.32:0/428093661 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7ffa88077010 0x7ffa88079520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:00:35.787 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:35.786+0000 7ffaa2a00640 1 -- 172.21.15.32:0/428093661 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(125..125 src has 1..125) ==== 9294+0+0 (secure 0 0 0) 0x7ffa9c09a100 con 0x7ffaac06e930 2024-08-03T21:00:35.787 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:35.787+0000 7ffaa1400640 1 --2- 172.21.15.32:0/428093661 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7ffa88077010 0x7ffa88079520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:00:35.788 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:35.787+0000 7ffaa1400640 1 --2- 172.21.15.32:0/428093661 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7ffa88077010 0x7ffa88079520 secure :-1 s=READY pgs=842 cs=0 l=1 rev1=1 crypto rx=0x7ffaac070510 tx=0x7ffa94009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T21:00:35.791 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:35.791+0000 7ffaa2a00640 1 -- 172.21.15.32:0/428093661 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7ffa9c0624e0 con 0x7ffaac06e930 2024-08-03T21:00:36.034 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:36.034+0000 7ffab1000640 1 -- 172.21.15.32:0/428093661 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7ffaac0b40a0 con 0x7ffaac06e930 2024-08-03T21:00:36.124 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:36.123+0000 7ffaa2a00640 1 -- 172.21.15.32:0/428093661 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v332) ==== 161+0+0 (secure 0 0 0) 0x7ffa9c061de0 con 0x7ffaac06e930 2024-08-03T21:00:36.124 INFO:teuthology.orchestra.run.smithi032.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-08-03T21:00:36.127 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:36.126+0000 7ffab1000640 1 -- 172.21.15.32:0/428093661 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7ffa88077010 msgr2=0x7ffa88079520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:00:36.127 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:36.126+0000 7ffab1000640 1 --2- 172.21.15.32:0/428093661 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7ffa88077010 0x7ffa88079520 secure :-1 s=READY pgs=842 cs=0 l=1 rev1=1 crypto rx=0x7ffaac070510 tx=0x7ffa94009040 comp rx=0 tx=0).stop 2024-08-03T21:00:36.127 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:36.126+0000 7ffab1000640 1 -- 172.21.15.32:0/428093661 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ffaac06e930 msgr2=0x7ffaac0b44c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:00:36.127 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:36.126+0000 7ffab1000640 1 --2- 172.21.15.32:0/428093661 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ffaac06e930 0x7ffaac0b44c0 secure :-1 s=READY pgs=2110 cs=0 l=1 rev1=1 crypto rx=0x7ffa9c00e790 tx=0x7ffa9c00ec50 comp rx=0 tx=0).stop 2024-08-03T21:00:36.127 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:36.126+0000 7ffab1000640 1 -- 172.21.15.32:0/428093661 shutdown_connections 2024-08-03T21:00:36.128 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:36.126+0000 7ffab1000640 1 --2- 172.21.15.32:0/428093661 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7ffa88077010 0x7ffa88079520 unknown :-1 s=CLOSED pgs=842 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:36.128 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:36.126+0000 7ffab1000640 1 --2- 172.21.15.32:0/428093661 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ffaac15fff0 0x7ffaac15ca70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:36.128 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:36.126+0000 7ffab1000640 1 --2- 172.21.15.32:0/428093661 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7ffaac073e60 0x7ffaac130580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:36.128 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:36.126+0000 7ffab1000640 1 --2- 172.21.15.32:0/428093661 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ffaac06e930 0x7ffaac0b44c0 unknown :-1 s=CLOSED pgs=2110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:36.128 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:36.126+0000 7ffab1000640 1 -- 172.21.15.32:0/428093661 >> 172.21.15.32:0/428093661 conn(0x7ffaac0b1bd0 msgr2=0x7ffaac0b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T21:00:36.128 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:36.126+0000 7ffab1000640 1 -- 172.21.15.32:0/428093661 shutdown_connections 2024-08-03T21:00:36.128 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:36.126+0000 7ffab1000640 1 -- 172.21.15.32:0/428093661 wait complete. 2024-08-03T21:00:36.153 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-08-03T21:00:36.387 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:36.383+0000 7f37f3600640 1 -- 172.21.15.32:0/1329992470 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f37ec09b040 msgr2=0x7f37ec0cd730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:00:36.387 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:36.383+0000 7f37f3600640 1 --2- 172.21.15.32:0/1329992470 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f37ec09b040 0x7f37ec0cd730 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7f37d8009830 tx=0x7f37d802a520 comp rx=0 tx=0).stop 2024-08-03T21:00:36.387 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:36.383+0000 7f37f3600640 1 -- 172.21.15.32:0/1329992470 shutdown_connections 2024-08-03T21:00:36.387 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:36.383+0000 7f37f3600640 1 --2- 172.21.15.32:0/1329992470 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f37ec09b040 0x7f37ec0cd730 unknown :-1 s=CLOSED pgs=630 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:36.387 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:36.383+0000 7f37f3600640 1 -- 172.21.15.32:0/1329992470 >> 172.21.15.32:0/1329992470 conn(0x7f37ec0b3c00 msgr2=0x7f37ec0b4010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T21:00:36.387 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:36.383+0000 7f37f3600640 1 -- 172.21.15.32:0/1329992470 shutdown_connections 2024-08-03T21:00:36.387 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:36.384+0000 7f37f3600640 1 -- 172.21.15.32:0/1329992470 wait complete. 2024-08-03T21:00:36.387 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:36.386+0000 7f37f3600640 1 Processor -- start 2024-08-03T21:00:36.388 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:36.386+0000 7f37f3600640 1 -- start start 2024-08-03T21:00:36.389 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:36.388+0000 7f37f3600640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f37ec09ac30 0x7f37ec1df360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:00:36.389 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:36.388+0000 7f37f3600640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f37ec09b040 0x7f37ec1dce20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:00:36.390 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:36.388+0000 7f37f3600640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f37ec0cdc70 0x7f37ec0c91a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:00:36.390 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:36.388+0000 7f37f3600640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f37ec0c9020 con 0x7f37ec09ac30 2024-08-03T21:00:36.390 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:36.388+0000 7f37f3600640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f37ec0c8ea0 con 0x7f37ec0cdc70 2024-08-03T21:00:36.390 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:36.388+0000 7f37f3600640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7f37ec0b7a20 con 0x7f37ec09b040 2024-08-03T21:00:36.390 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:36.388+0000 7f37f0c00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f37ec09ac30 0x7f37ec1df360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:00:36.390 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:36.388+0000 7f37f0c00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f37ec09ac30 0x7f37ec1df360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:52862/0 (socket says 172.21.15.32:52862) 2024-08-03T21:00:36.390 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:36.388+0000 7f37f0c00640 1 -- 172.21.15.32:0/500027728 learned_addr learned my addr 172.21.15.32:0/500027728 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T21:00:36.390 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:36.388+0000 7f37f1600640 1 --2- 172.21.15.32:0/500027728 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f37ec0cdc70 0x7f37ec0c91a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:00:36.390 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:36.388+0000 7f37ebe00640 1 --2- 172.21.15.32:0/500027728 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f37ec09b040 0x7f37ec1dce20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:00:36.390 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:36.388+0000 7f37f0c00640 1 -- 172.21.15.32:0/500027728 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f37ec09b040 msgr2=0x7f37ec1dce20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:00:36.390 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:36.388+0000 7f37f0c00640 1 --2- 172.21.15.32:0/500027728 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f37ec09b040 0x7f37ec1dce20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:36.391 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:36.388+0000 7f37f0c00640 1 -- 172.21.15.32:0/500027728 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f37ec0cdc70 msgr2=0x7f37ec0c91a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:00:36.391 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:36.388+0000 7f37f0c00640 1 --2- 172.21.15.32:0/500027728 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f37ec0cdc70 0x7f37ec0c91a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:36.391 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:36.388+0000 7f37f0c00640 1 -- 172.21.15.32:0/500027728 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f37ec1e00e0 con 0x7f37ec09ac30 2024-08-03T21:00:36.391 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:36.388+0000 7f37f1600640 1 --2- 172.21.15.32:0/500027728 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f37ec0cdc70 0x7f37ec0c91a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-03T21:00:36.391 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:36.389+0000 7f37f0c00640 1 --2- 172.21.15.32:0/500027728 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f37ec09ac30 0x7f37ec1df360 secure :-1 s=READY pgs=2112 cs=0 l=1 rev1=1 crypto rx=0x7f37e0001220 tx=0x7f37e000c890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T21:00:36.391 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:36.389+0000 7f37e8c00640 1 -- 172.21.15.32:0/500027728 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37e0011790 con 0x7f37ec09ac30 2024-08-03T21:00:36.391 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:36.389+0000 7f37e8c00640 1 -- 172.21.15.32:0/500027728 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f37e0017070 con 0x7f37ec09ac30 2024-08-03T21:00:36.391 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:36.389+0000 7f37e8c00640 1 -- 172.21.15.32:0/500027728 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37e0011cc0 con 0x7f37ec09ac30 2024-08-03T21:00:36.391 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:36.389+0000 7f37f3600640 1 -- 172.21.15.32:0/500027728 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f37ec1de880 con 0x7f37ec09ac30 2024-08-03T21:00:36.391 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:36.389+0000 7f37f3600640 1 -- 172.21.15.32:0/500027728 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f37ec0c9c00 con 0x7f37ec09ac30 2024-08-03T21:00:36.392 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:36.390+0000 7f37f3600640 1 -- 172.21.15.32:0/500027728 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f37ec0b2d80 con 0x7f37ec09ac30 2024-08-03T21:00:36.392 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:36.391+0000 7f37e8c00640 1 -- 172.21.15.32:0/500027728 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f37e001c460 con 0x7f37ec09ac30 2024-08-03T21:00:36.392 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:36.392+0000 7f37e8c00640 1 --2- 172.21.15.32:0/500027728 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f37cc077010 0x7f37cc079520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:00:36.393 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:36.392+0000 7f37e8c00640 1 -- 172.21.15.32:0/500027728 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(125..125 src has 1..125) ==== 9294+0+0 (secure 0 0 0) 0x7f37e00994e0 con 0x7f37ec09ac30 2024-08-03T21:00:36.393 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:36.393+0000 7f37ebe00640 1 --2- 172.21.15.32:0/500027728 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f37cc077010 0x7f37cc079520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:00:36.394 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:36.393+0000 7f37ebe00640 1 --2- 172.21.15.32:0/500027728 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f37cc077010 0x7f37cc079520 secure :-1 s=READY pgs=843 cs=0 l=1 rev1=1 crypto rx=0x7f37d8034000 tx=0x7f37d8004730 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T21:00:36.397 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:36.396+0000 7f37e8c00640 1 -- 172.21.15.32:0/500027728 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7f37e0061950 con 0x7f37ec09ac30 2024-08-03T21:00:36.639 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:36.639+0000 7f37f3600640 1 -- 172.21.15.32:0/500027728 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) -- 0x7f37ec0b6850 con 0x7f37ec09ac30 2024-08-03T21:00:37.135 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:37.135+0000 7f37e8c00640 1 -- 172.21.15.32:0/500027728 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v333) ==== 104+0+0 (secure 0 0 0) 0x7f37e0061250 con 0x7f37ec09ac30 2024-08-03T21:00:37.138 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:37.137+0000 7f37f3600640 1 -- 172.21.15.32:0/500027728 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f37cc077010 msgr2=0x7f37cc079520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:00:37.138 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:37.137+0000 7f37f3600640 1 --2- 172.21.15.32:0/500027728 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f37cc077010 0x7f37cc079520 secure :-1 s=READY pgs=843 cs=0 l=1 rev1=1 crypto rx=0x7f37d8034000 tx=0x7f37d8004730 comp rx=0 tx=0).stop 2024-08-03T21:00:37.138 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:37.137+0000 7f37f3600640 1 -- 172.21.15.32:0/500027728 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f37ec09ac30 msgr2=0x7f37ec1df360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:00:37.139 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:37.137+0000 7f37f3600640 1 --2- 172.21.15.32:0/500027728 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f37ec09ac30 0x7f37ec1df360 secure :-1 s=READY pgs=2112 cs=0 l=1 rev1=1 crypto rx=0x7f37e0001220 tx=0x7f37e000c890 comp rx=0 tx=0).stop 2024-08-03T21:00:37.139 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:37.137+0000 7f37f3600640 1 -- 172.21.15.32:0/500027728 shutdown_connections 2024-08-03T21:00:37.139 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:37.137+0000 7f37f3600640 1 --2- 172.21.15.32:0/500027728 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f37cc077010 0x7f37cc079520 unknown :-1 s=CLOSED pgs=843 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:37.139 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:37.137+0000 7f37f3600640 1 --2- 172.21.15.32:0/500027728 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f37ec0cdc70 0x7f37ec0c91a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:37.139 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:37.137+0000 7f37f3600640 1 --2- 172.21.15.32:0/500027728 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f37ec09b040 0x7f37ec1dce20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:37.139 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:37.137+0000 7f37f3600640 1 --2- 172.21.15.32:0/500027728 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f37ec09ac30 0x7f37ec1df360 unknown :-1 s=CLOSED pgs=2112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:37.139 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:37.137+0000 7f37f3600640 1 -- 172.21.15.32:0/500027728 >> 172.21.15.32:0/500027728 conn(0x7f37ec0b3c00 msgr2=0x7f37ec0b57e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T21:00:37.139 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:37.138+0000 7f37f3600640 1 -- 172.21.15.32:0/500027728 shutdown_connections 2024-08-03T21:00:37.139 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:37.138+0000 7f37f3600640 1 -- 172.21.15.32:0/500027728 wait complete. 2024-08-03T21:00:37.162 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-08-03T21:00:37.397 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:37.396+0000 7f450b200640 1 -- 172.21.15.32:0/2380136167 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f44e8004800 msgr2=0x7f44e8004c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:00:37.397 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:37.396+0000 7f450b200640 1 --2- 172.21.15.32:0/2380136167 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f44e8004800 0x7f44e8004c10 secure :-1 s=READY pgs=2114 cs=0 l=1 rev1=1 crypto rx=0x7f44ec009b20 tx=0x7f44e8005830 comp rx=0 tx=0).stop 2024-08-03T21:00:37.398 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:37.396+0000 7f450b200640 1 -- 172.21.15.32:0/2380136167 shutdown_connections 2024-08-03T21:00:37.398 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:37.396+0000 7f450b200640 1 --2- 172.21.15.32:0/2380136167 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f44e8004800 0x7f44e8004c10 unknown :-1 s=CLOSED pgs=2114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:37.398 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:37.396+0000 7f450b200640 1 -- 172.21.15.32:0/2380136167 >> 172.21.15.32:0/2380136167 conn(0x7f45040b1bd0 msgr2=0x7f45040b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T21:00:37.398 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:37.397+0000 7f450b200640 1 -- 172.21.15.32:0/2380136167 shutdown_connections 2024-08-03T21:00:37.398 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:37.397+0000 7f450b200640 1 -- 172.21.15.32:0/2380136167 wait complete. 2024-08-03T21:00:37.399 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:37.399+0000 7f450b200640 1 Processor -- start 2024-08-03T21:00:37.399 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:37.399+0000 7f450b200640 1 -- start start 2024-08-03T21:00:37.402 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:37.400+0000 7f450b200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f44e8004800 0x7f45041f93f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:00:37.402 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:37.400+0000 7f450b200640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f45041f60d0 0x7f45040cb870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:00:37.402 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:37.400+0000 7f450b200640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f45040cbdb0 0x7f45041f54c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:00:37.403 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:37.400+0000 7f450b200640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f450406ead0 con 0x7f44e8004800 2024-08-03T21:00:37.403 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:37.400+0000 7f450b200640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f450406e950 con 0x7f45040cbdb0 2024-08-03T21:00:37.403 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:37.400+0000 7f450b200640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7f450406ec50 con 0x7f45041f60d0 2024-08-03T21:00:37.403 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:37.401+0000 7f4503e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f44e8004800 0x7f45041f93f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:00:37.403 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:37.401+0000 7f4503e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f44e8004800 0x7f45041f93f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:52886/0 (socket says 172.21.15.32:52886) 2024-08-03T21:00:37.403 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:37.401+0000 7f4503e00640 1 -- 172.21.15.32:0/3818297210 learned_addr learned my addr 172.21.15.32:0/3818297210 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T21:00:37.403 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:37.401+0000 7f44fb400640 1 --2- 172.21.15.32:0/3818297210 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f45041f60d0 0x7f45040cb870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:00:37.403 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:37.401+0000 7f4503e00640 1 -- 172.21.15.32:0/3818297210 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f45041f60d0 msgr2=0x7f45040cb870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:00:37.403 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:37.401+0000 7f4509200640 1 --2- 172.21.15.32:0/3818297210 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f45040cbdb0 0x7f45041f54c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:00:37.403 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:37.401+0000 7f4503e00640 1 --2- 172.21.15.32:0/3818297210 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f45041f60d0 0x7f45040cb870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:37.403 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:37.401+0000 7f4503e00640 1 -- 172.21.15.32:0/3818297210 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f45040cbdb0 msgr2=0x7f45041f54c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:00:37.403 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:37.401+0000 7f4503e00640 1 --2- 172.21.15.32:0/3818297210 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f45040cbdb0 0x7f45041f54c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:37.403 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:37.401+0000 7f4503e00640 1 -- 172.21.15.32:0/3818297210 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f45040a95e0 con 0x7f44e8004800 2024-08-03T21:00:37.404 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:37.401+0000 7f44fb400640 1 --2- 172.21.15.32:0/3818297210 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f45041f60d0 0x7f45040cb870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-03T21:00:37.404 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:37.401+0000 7f4509200640 1 --2- 172.21.15.32:0/3818297210 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f45040cbdb0 0x7f45041f54c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-03T21:00:37.404 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:37.401+0000 7f4503e00640 1 --2- 172.21.15.32:0/3818297210 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f44e8004800 0x7f45041f93f0 secure :-1 s=READY pgs=2115 cs=0 l=1 rev1=1 crypto rx=0x7f44f400e5c0 tx=0x7f44f400ea80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T21:00:37.404 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:37.402+0000 7f4500c00640 1 -- 172.21.15.32:0/3818297210 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44f4015020 con 0x7f44e8004800 2024-08-03T21:00:37.404 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:37.402+0000 7f4500c00640 1 -- 172.21.15.32:0/3818297210 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f44f4010980 con 0x7f44e8004800 2024-08-03T21:00:37.404 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:37.402+0000 7f4500c00640 1 -- 172.21.15.32:0/3818297210 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44f4010dd0 con 0x7f44e8004800 2024-08-03T21:00:37.404 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:37.402+0000 7f450b200640 1 -- 172.21.15.32:0/3818297210 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f45040a98c0 con 0x7f44e8004800 2024-08-03T21:00:37.404 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:37.402+0000 7f450b200640 1 -- 172.21.15.32:0/3818297210 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f45041f5a00 con 0x7f44e8004800 2024-08-03T21:00:37.404 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:37.403+0000 7f450b200640 1 -- 172.21.15.32:0/3818297210 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f450406f820 con 0x7f44e8004800 2024-08-03T21:00:37.405 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:37.404+0000 7f4500c00640 1 -- 172.21.15.32:0/3818297210 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f44f4018020 con 0x7f44e8004800 2024-08-03T21:00:37.405 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:37.405+0000 7f4500c00640 1 --2- 172.21.15.32:0/3818297210 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f44e0077010 0x7f44e0079520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:00:37.405 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:37.405+0000 7f4500c00640 1 -- 172.21.15.32:0/3818297210 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(125..125 src has 1..125) ==== 9294+0+0 (secure 0 0 0) 0x7f44f4099bc0 con 0x7f44e8004800 2024-08-03T21:00:37.406 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:37.405+0000 7f44fb400640 1 --2- 172.21.15.32:0/3818297210 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f44e0077010 0x7f44e0079520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:00:37.406 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:37.406+0000 7f44fb400640 1 --2- 172.21.15.32:0/3818297210 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f44e0077010 0x7f44e0079520 secure :-1 s=READY pgs=844 cs=0 l=1 rev1=1 crypto rx=0x7f44ec035000 tx=0x7f44ec0046c0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T21:00:37.410 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:37.410+0000 7f4500c00640 1 -- 172.21.15.32:0/3818297210 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7f450406f820 con 0x7f44e8004800 2024-08-03T21:00:37.613 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:37.613+0000 7f450b200640 1 -- 172.21.15.32:0/3818297210 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) -- 0x7f45040cf140 con 0x7f44e8004800 2024-08-03T21:00:38.138 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:38.137+0000 7f4500c00640 1 -- 172.21.15.32:0/3818297210 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v126) ==== 181+0+0 (secure 0 0 0) 0x7f44f401f020 con 0x7f44e8004800 2024-08-03T21:00:38.138 INFO:teuthology.orchestra.run.smithi032.stderr:pool 'cephfs_metadata' removed 2024-08-03T21:00:38.142 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:38.141+0000 7f450b200640 1 -- 172.21.15.32:0/3818297210 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f44e0077010 msgr2=0x7f44e0079520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:00:38.142 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:38.141+0000 7f450b200640 1 --2- 172.21.15.32:0/3818297210 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f44e0077010 0x7f44e0079520 secure :-1 s=READY pgs=844 cs=0 l=1 rev1=1 crypto rx=0x7f44ec035000 tx=0x7f44ec0046c0 comp rx=0 tx=0).stop 2024-08-03T21:00:38.142 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:38.141+0000 7f450b200640 1 -- 172.21.15.32:0/3818297210 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f44e8004800 msgr2=0x7f45041f93f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:00:38.142 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:38.141+0000 7f450b200640 1 --2- 172.21.15.32:0/3818297210 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f44e8004800 0x7f45041f93f0 secure :-1 s=READY pgs=2115 cs=0 l=1 rev1=1 crypto rx=0x7f44f400e5c0 tx=0x7f44f400ea80 comp rx=0 tx=0).stop 2024-08-03T21:00:38.142 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:38.141+0000 7f450b200640 1 -- 172.21.15.32:0/3818297210 shutdown_connections 2024-08-03T21:00:38.142 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:38.141+0000 7f450b200640 1 --2- 172.21.15.32:0/3818297210 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f44e0077010 0x7f44e0079520 unknown :-1 s=CLOSED pgs=844 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:38.143 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:38.141+0000 7f450b200640 1 --2- 172.21.15.32:0/3818297210 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f45040cbdb0 0x7f45041f54c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:38.143 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:38.141+0000 7f450b200640 1 --2- 172.21.15.32:0/3818297210 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f45041f60d0 0x7f45040cb870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:38.143 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:38.141+0000 7f450b200640 1 --2- 172.21.15.32:0/3818297210 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f44e8004800 0x7f45041f93f0 unknown :-1 s=CLOSED pgs=2115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:38.143 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:38.141+0000 7f450b200640 1 -- 172.21.15.32:0/3818297210 >> 172.21.15.32:0/3818297210 conn(0x7f45040b1bd0 msgr2=0x7f45040b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T21:00:38.143 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:38.141+0000 7f450b200640 1 -- 172.21.15.32:0/3818297210 shutdown_connections 2024-08-03T21:00:38.143 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:38.141+0000 7f450b200640 1 -- 172.21.15.32:0/3818297210 wait complete. 2024-08-03T21:00:38.166 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-08-03T21:00:38.398 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:38.397+0000 7f67ff800640 1 -- 172.21.15.32:0/2180240265 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f67f8145c50 msgr2=0x7f67f806e950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:00:38.399 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:38.397+0000 7f67ff800640 1 --2- 172.21.15.32:0/2180240265 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f67f8145c50 0x7f67f806e950 secure :-1 s=READY pgs=2116 cs=0 l=1 rev1=1 crypto rx=0x7f67f815fea0 tx=0x7f67ec02a580 comp rx=0 tx=0).stop 2024-08-03T21:00:38.399 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:38.397+0000 7f67ff800640 1 -- 172.21.15.32:0/2180240265 shutdown_connections 2024-08-03T21:00:38.399 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:38.397+0000 7f67ff800640 1 --2- 172.21.15.32:0/2180240265 >> v2:172.21.15.118:3300/0 conn(0x7f67f80ced30 0x7f67f80cf140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:38.399 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:38.397+0000 7f67ff800640 1 --2- 172.21.15.32:0/2180240265 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f67f80cccf0 0x7f67f80cd100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:38.399 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:38.397+0000 7f67ff800640 1 --2- 172.21.15.32:0/2180240265 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f67f8145c50 0x7f67f806e950 unknown :-1 s=CLOSED pgs=2116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:38.399 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:38.397+0000 7f67ff800640 1 -- 172.21.15.32:0/2180240265 >> 172.21.15.32:0/2180240265 conn(0x7f67f80b1bd0 msgr2=0x7f67f80b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T21:00:38.399 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:38.398+0000 7f67ff800640 1 -- 172.21.15.32:0/2180240265 shutdown_connections 2024-08-03T21:00:38.399 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:38.398+0000 7f67ff800640 1 -- 172.21.15.32:0/2180240265 wait complete. 2024-08-03T21:00:38.400 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:38.400+0000 7f67ff800640 1 Processor -- start 2024-08-03T21:00:38.400 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:38.400+0000 7f67ff800640 1 -- start start 2024-08-03T21:00:38.403 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:38.401+0000 7f67ff800640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f67f80cccf0 0x7f67f814dbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:00:38.403 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:38.401+0000 7f67ff800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f67f80ced30 0x7f67f81f8820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:00:38.404 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:38.401+0000 7f67ff800640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f67f8145c50 0x7f67f81f8d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:00:38.404 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:38.401+0000 7f67ff800640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f67f80b3e60 con 0x7f67f80ced30 2024-08-03T21:00:38.404 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:38.401+0000 7f67ff800640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f67f80b3ce0 con 0x7f67f80cccf0 2024-08-03T21:00:38.404 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:38.401+0000 7f67ff800640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7f67f80b3fe0 con 0x7f67f8145c50 2024-08-03T21:00:38.404 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:38.402+0000 7f67f7e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f67f80ced30 0x7f67f81f8820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:00:38.404 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:38.402+0000 7f67f7e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f67f80ced30 0x7f67f81f8820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:52902/0 (socket says 172.21.15.32:52902) 2024-08-03T21:00:38.404 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:38.402+0000 7f67f7e00640 1 -- 172.21.15.32:0/1307113967 learned_addr learned my addr 172.21.15.32:0/1307113967 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T21:00:38.404 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:38.402+0000 7f67fce00640 1 --2- 172.21.15.32:0/1307113967 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f67f80cccf0 0x7f67f814dbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:00:38.404 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:38.402+0000 7f67fd800640 1 --2- 172.21.15.32:0/1307113967 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f67f8145c50 0x7f67f81f8d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:00:38.404 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:38.402+0000 7f67f7e00640 1 -- 172.21.15.32:0/1307113967 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f67f8145c50 msgr2=0x7f67f81f8d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:00:38.404 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:38.402+0000 7f67f7e00640 1 --2- 172.21.15.32:0/1307113967 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f67f8145c50 0x7f67f81f8d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:38.404 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:38.402+0000 7f67f7e00640 1 -- 172.21.15.32:0/1307113967 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f67f80cccf0 msgr2=0x7f67f814dbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:00:38.404 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:38.402+0000 7f67f7e00640 1 --2- 172.21.15.32:0/1307113967 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f67f80cccf0 0x7f67f814dbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:38.404 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:38.402+0000 7f67f7e00640 1 -- 172.21.15.32:0/1307113967 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f67f813e2d0 con 0x7f67f80ced30 2024-08-03T21:00:38.405 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:38.402+0000 7f67fce00640 1 --2- 172.21.15.32:0/1307113967 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f67f80cccf0 0x7f67f814dbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-03T21:00:38.405 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:38.402+0000 7f67f7e00640 1 --2- 172.21.15.32:0/1307113967 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f67f80ced30 0x7f67f81f8820 secure :-1 s=READY pgs=2117 cs=0 l=1 rev1=1 crypto rx=0x7f67e400c9a0 tx=0x7f67e400ce60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T21:00:38.405 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:38.403+0000 7f67f4c00640 1 -- 172.21.15.32:0/1307113967 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67e4007bb0 con 0x7f67f80ced30 2024-08-03T21:00:38.405 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:38.403+0000 7f67f4c00640 1 -- 172.21.15.32:0/1307113967 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f67e4007e40 con 0x7f67f80ced30 2024-08-03T21:00:38.405 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:38.403+0000 7f67f4c00640 1 -- 172.21.15.32:0/1307113967 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67e400f690 con 0x7f67f80ced30 2024-08-03T21:00:38.405 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:38.403+0000 7f67ff800640 1 -- 172.21.15.32:0/1307113967 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f67f813eb00 con 0x7f67f80ced30 2024-08-03T21:00:38.405 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:38.403+0000 7f67ff800640 1 -- 172.21.15.32:0/1307113967 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f67f813e820 con 0x7f67f80ced30 2024-08-03T21:00:38.405 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:38.404+0000 7f67ff800640 1 -- 172.21.15.32:0/1307113967 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f67f8140080 con 0x7f67f80ced30 2024-08-03T21:00:38.406 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:38.405+0000 7f67f4c00640 1 -- 172.21.15.32:0/1307113967 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f67e400f830 con 0x7f67f80ced30 2024-08-03T21:00:38.407 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:38.406+0000 7f67f4c00640 1 --2- 172.21.15.32:0/1307113967 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f67d8077010 0x7f67d8079520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:00:38.407 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:38.406+0000 7f67f4c00640 1 -- 172.21.15.32:0/1307113967 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(126..126 src has 1..126) ==== 8856+0+0 (secure 0 0 0) 0x7f67e4099dc0 con 0x7f67f80ced30 2024-08-03T21:00:38.407 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:38.407+0000 7f67fce00640 1 --2- 172.21.15.32:0/1307113967 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f67d8077010 0x7f67d8079520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:00:38.408 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:38.408+0000 7f67fce00640 1 --2- 172.21.15.32:0/1307113967 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f67d8077010 0x7f67d8079520 secure :-1 s=READY pgs=845 cs=0 l=1 rev1=1 crypto rx=0x7f67ec033000 tx=0x7f67ec004730 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T21:00:38.411 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:38.411+0000 7f67f4c00640 1 -- 172.21.15.32:0/1307113967 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7f67e4014030 con 0x7f67f80ced30 2024-08-03T21:00:38.615 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:38.614+0000 7f67ff800640 1 -- 172.21.15.32:0/1307113967 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) -- 0x7f67f80b4500 con 0x7f67f80ced30 2024-08-03T21:00:39.148 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:39.143+0000 7f67f4c00640 1 -- 172.21.15.32:0/1307113967 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v127) ==== 169+0+0 (secure 0 0 0) 0x7f67e4062350 con 0x7f67f80ced30 2024-08-03T21:00:39.148 INFO:teuthology.orchestra.run.smithi032.stderr:pool 'cephfs_data' removed 2024-08-03T21:00:39.148 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:39.146+0000 7f67ff800640 1 -- 172.21.15.32:0/1307113967 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f67d8077010 msgr2=0x7f67d8079520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:00:39.148 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:39.146+0000 7f67ff800640 1 --2- 172.21.15.32:0/1307113967 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f67d8077010 0x7f67d8079520 secure :-1 s=READY pgs=845 cs=0 l=1 rev1=1 crypto rx=0x7f67ec033000 tx=0x7f67ec004730 comp rx=0 tx=0).stop 2024-08-03T21:00:39.149 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:39.146+0000 7f67ff800640 1 -- 172.21.15.32:0/1307113967 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f67f80ced30 msgr2=0x7f67f81f8820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:00:39.149 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:39.146+0000 7f67ff800640 1 --2- 172.21.15.32:0/1307113967 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f67f80ced30 0x7f67f81f8820 secure :-1 s=READY pgs=2117 cs=0 l=1 rev1=1 crypto rx=0x7f67e400c9a0 tx=0x7f67e400ce60 comp rx=0 tx=0).stop 2024-08-03T21:00:39.149 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:39.146+0000 7f67ff800640 1 -- 172.21.15.32:0/1307113967 shutdown_connections 2024-08-03T21:00:39.149 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:39.146+0000 7f67ff800640 1 --2- 172.21.15.32:0/1307113967 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f67d8077010 0x7f67d8079520 unknown :-1 s=CLOSED pgs=845 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:39.149 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:39.146+0000 7f67ff800640 1 --2- 172.21.15.32:0/1307113967 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f67f8145c50 0x7f67f81f8d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:39.149 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:39.146+0000 7f67ff800640 1 --2- 172.21.15.32:0/1307113967 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f67f80ced30 0x7f67f81f8820 unknown :-1 s=CLOSED pgs=2117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:39.149 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:39.146+0000 7f67ff800640 1 --2- 172.21.15.32:0/1307113967 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f67f80cccf0 0x7f67f814dbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:39.149 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:39.146+0000 7f67ff800640 1 -- 172.21.15.32:0/1307113967 >> 172.21.15.32:0/1307113967 conn(0x7f67f80b1bd0 msgr2=0x7f67f806f890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T21:00:39.149 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:39.146+0000 7f67ff800640 1 -- 172.21.15.32:0/1307113967 shutdown_connections 2024-08-03T21:00:39.149 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:39.146+0000 7f67ff800640 1 -- 172.21.15.32:0/1307113967 wait complete. 2024-08-03T21:00:39.174 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config reset 27 2024-08-03T21:00:39.178 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-08-03T21:00:39.182 DEBUG:teuthology.orchestra.run.smithi118:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-08-03T21:00:39.405 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:39.404+0000 7fe2aa600640 1 -- 172.21.15.32:0/2613192890 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fe2a406fb90 msgr2=0x7fe2a40ca3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:00:39.405 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:39.404+0000 7fe2aa600640 1 --2- 172.21.15.32:0/2613192890 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fe2a406fb90 0x7fe2a40ca3b0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fe29c009be0 tx=0x7fe29c02a920 comp rx=0 tx=0).stop 2024-08-03T21:00:39.405 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:39.404+0000 7fe2aa600640 1 -- 172.21.15.32:0/2613192890 shutdown_connections 2024-08-03T21:00:39.405 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:39.404+0000 7fe2aa600640 1 --2- 172.21.15.32:0/2613192890 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fe2a406fb90 0x7fe2a40ca3b0 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:39.405 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:39.404+0000 7fe2aa600640 1 -- 172.21.15.32:0/2613192890 >> 172.21.15.32:0/2613192890 conn(0x7fe2a40b2120 msgr2=0x7fe2a40cfd60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T21:00:39.406 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:39.404+0000 7fe2aa600640 1 -- 172.21.15.32:0/2613192890 shutdown_connections 2024-08-03T21:00:39.406 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:39.404+0000 7fe2aa600640 1 -- 172.21.15.32:0/2613192890 wait complete. 2024-08-03T21:00:39.407 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:39.406+0000 7fe2aa600640 1 Processor -- start 2024-08-03T21:00:39.407 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:39.407+0000 7fe2aa600640 1 -- start start 2024-08-03T21:00:39.410 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:39.408+0000 7fe2aa600640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fe2a40af040 0x7fe2a4157fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:00:39.410 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:39.408+0000 7fe2aa600640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fe2a40cee10 0x7fe2a415d860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:00:39.410 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:39.408+0000 7fe2aa600640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe2a409b3b0 0x7fe2a415dda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:00:39.410 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:39.408+0000 7fe2aa600640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fe2a40caa70 con 0x7fe2a409b3b0 2024-08-03T21:00:39.411 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:39.408+0000 7fe2aa600640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7fe2a40ca8f0 con 0x7fe2a40af040 2024-08-03T21:00:39.411 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:39.408+0000 7fe2aa600640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7fe2a40cabf0 con 0x7fe2a40cee10 2024-08-03T21:00:39.411 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:39.408+0000 7fe2a9c00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe2a409b3b0 0x7fe2a415dda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:00:39.411 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:39.408+0000 7fe2a9c00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe2a409b3b0 0x7fe2a415dda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:52918/0 (socket says 172.21.15.32:52918) 2024-08-03T21:00:39.411 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:39.408+0000 7fe2a9c00640 1 -- 172.21.15.32:0/1253698240 learned_addr learned my addr 172.21.15.32:0/1253698240 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T21:00:39.411 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:39.408+0000 7fe2a3e00640 1 --2- 172.21.15.32:0/1253698240 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fe2a40cee10 0x7fe2a415d860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:00:39.411 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:39.408+0000 7fe2a9200640 1 --2- 172.21.15.32:0/1253698240 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fe2a40af040 0x7fe2a4157fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:00:39.411 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:39.409+0000 7fe2a9c00640 1 -- 172.21.15.32:0/1253698240 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fe2a40cee10 msgr2=0x7fe2a415d860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:00:39.411 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:39.409+0000 7fe2a9c00640 1 --2- 172.21.15.32:0/1253698240 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fe2a40cee10 0x7fe2a415d860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:39.411 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:39.409+0000 7fe2a9c00640 1 -- 172.21.15.32:0/1253698240 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fe2a40af040 msgr2=0x7fe2a4157fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:00:39.411 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:39.409+0000 7fe2a9c00640 1 --2- 172.21.15.32:0/1253698240 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fe2a40af040 0x7fe2a4157fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:39.411 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:39.409+0000 7fe2a9c00640 1 -- 172.21.15.32:0/1253698240 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe2a41f5d30 con 0x7fe2a409b3b0 2024-08-03T21:00:39.412 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:39.409+0000 7fe2a9c00640 1 --2- 172.21.15.32:0/1253698240 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe2a409b3b0 0x7fe2a415dda0 secure :-1 s=READY pgs=2119 cs=0 l=1 rev1=1 crypto rx=0x7fe29c009be0 tx=0x7fe29c003ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T21:00:39.412 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:39.409+0000 7fe2a0c00640 1 -- 172.21.15.32:0/1253698240 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe29c030c50 con 0x7fe2a409b3b0 2024-08-03T21:00:39.412 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:39.409+0000 7fe2aa600640 1 -- 172.21.15.32:0/1253698240 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe2a41f4f70 con 0x7fe2a409b3b0 2024-08-03T21:00:39.412 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:39.409+0000 7fe2a0c00640 1 -- 172.21.15.32:0/1253698240 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fe29c03e070 con 0x7fe2a409b3b0 2024-08-03T21:00:39.412 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:39.409+0000 7fe2a0c00640 1 -- 172.21.15.32:0/1253698240 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe29c0042a0 con 0x7fe2a409b3b0 2024-08-03T21:00:39.412 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:39.410+0000 7fe2aa600640 1 -- 172.21.15.32:0/1253698240 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe2a41f5530 con 0x7fe2a409b3b0 2024-08-03T21:00:39.412 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:39.411+0000 7fe2aa600640 1 -- 172.21.15.32:0/1253698240 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe2a40ca3b0 con 0x7fe2a409b3b0 2024-08-03T21:00:39.412 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:39.412+0000 7fe2a0c00640 1 -- 172.21.15.32:0/1253698240 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7fe29c004440 con 0x7fe2a409b3b0 2024-08-03T21:00:39.413 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:39.413+0000 7fe2a0c00640 1 --2- 172.21.15.32:0/1253698240 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fe284077040 0x7fe284079550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:00:39.413 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:39.413+0000 7fe2a0c00640 1 -- 172.21.15.32:0/1253698240 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(127..127 src has 1..127) ==== 8297+0+0 (secure 0 0 0) 0x7fe29c0baaf0 con 0x7fe2a409b3b0 2024-08-03T21:00:39.414 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:39.413+0000 7fe2a9200640 1 --2- 172.21.15.32:0/1253698240 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fe284077040 0x7fe284079550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:00:39.414 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:39.414+0000 7fe2a9200640 1 --2- 172.21.15.32:0/1253698240 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fe284077040 0x7fe284079550 secure :-1 s=READY pgs=846 cs=0 l=1 rev1=1 crypto rx=0x7fe298007fd0 tx=0x7fe298007eb0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T21:00:39.417 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:39.417+0000 7fe2a0c00640 1 -- 172.21.15.32:0/1253698240 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7fe29c02d070 con 0x7fe2a409b3b0 2024-08-03T21:00:39.623 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:39.622+0000 7fe2aa600640 1 -- 172.21.15.32:0/1253698240 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "config reset", "num": 27} v 0) -- 0x7fe2a40afeb0 con 0x7fe2a409b3b0 2024-08-03T21:00:39.627 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:39.627+0000 7fe2a0c00640 1 -- 172.21.15.32:0/1253698240 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "config reset", "num": 27}]=0 v44) ==== 71+0+0 (secure 0 0 0) 0x7fe29c082c70 con 0x7fe2a409b3b0 2024-08-03T21:00:39.630 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:39.629+0000 7fe2aa600640 1 -- 172.21.15.32:0/1253698240 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fe284077040 msgr2=0x7fe284079550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:00:39.630 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:39.629+0000 7fe2aa600640 1 --2- 172.21.15.32:0/1253698240 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fe284077040 0x7fe284079550 secure :-1 s=READY pgs=846 cs=0 l=1 rev1=1 crypto rx=0x7fe298007fd0 tx=0x7fe298007eb0 comp rx=0 tx=0).stop 2024-08-03T21:00:39.630 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:39.629+0000 7fe2aa600640 1 -- 172.21.15.32:0/1253698240 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe2a409b3b0 msgr2=0x7fe2a415dda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:00:39.630 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:39.629+0000 7fe2aa600640 1 --2- 172.21.15.32:0/1253698240 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe2a409b3b0 0x7fe2a415dda0 secure :-1 s=READY pgs=2119 cs=0 l=1 rev1=1 crypto rx=0x7fe29c009be0 tx=0x7fe29c003ba0 comp rx=0 tx=0).stop 2024-08-03T21:00:39.631 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:39.629+0000 7fe2aa600640 1 -- 172.21.15.32:0/1253698240 shutdown_connections 2024-08-03T21:00:39.631 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:39.629+0000 7fe2aa600640 1 --2- 172.21.15.32:0/1253698240 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fe284077040 0x7fe284079550 unknown :-1 s=CLOSED pgs=846 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:39.631 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:39.629+0000 7fe2aa600640 1 --2- 172.21.15.32:0/1253698240 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe2a409b3b0 0x7fe2a415dda0 unknown :-1 s=CLOSED pgs=2119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:39.631 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:39.629+0000 7fe2aa600640 1 --2- 172.21.15.32:0/1253698240 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fe2a40cee10 0x7fe2a415d860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:39.631 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:39.629+0000 7fe2aa600640 1 --2- 172.21.15.32:0/1253698240 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fe2a40af040 0x7fe2a4157fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:39.631 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:39.629+0000 7fe2aa600640 1 -- 172.21.15.32:0/1253698240 >> 172.21.15.32:0/1253698240 conn(0x7fe2a40b2120 msgr2=0x7fe2a40731d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T21:00:39.631 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:39.629+0000 7fe2aa600640 1 -- 172.21.15.32:0/1253698240 shutdown_connections 2024-08-03T21:00:39.631 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:39.630+0000 7fe2aa600640 1 -- 172.21.15.32:0/1253698240 wait complete. 2024-08-03T21:00:39.652 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_damage.TestDamage.test_object_deletion' 2024-08-03T21:00:39.890 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:39.889+0000 7fdb77e00640 1 -- 172.21.15.32:0/2717765032 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fdb780d11f0 msgr2=0x7fdb780b32a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:00:39.890 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:39.889+0000 7fdb77e00640 1 --2- 172.21.15.32:0/2717765032 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fdb780d11f0 0x7fdb780b32a0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fdb64005fd0 tx=0x7fdb6402a4b0 comp rx=0 tx=0).stop 2024-08-03T21:00:39.890 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:39.889+0000 7fdb77e00640 1 -- 172.21.15.32:0/2717765032 shutdown_connections 2024-08-03T21:00:39.890 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:39.889+0000 7fdb77e00640 1 --2- 172.21.15.32:0/2717765032 >> v2:172.21.15.118:3300/0 conn(0x7fdb780b94d0 0x7fdb780b3f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:39.891 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:39.889+0000 7fdb77e00640 1 --2- 172.21.15.32:0/2717765032 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fdb780d11f0 0x7fdb780b32a0 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:39.891 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:39.889+0000 7fdb77e00640 1 -- 172.21.15.32:0/2717765032 >> 172.21.15.32:0/2717765032 conn(0x7fdb780d4180 msgr2=0x7fdb780d4590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T21:00:39.891 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:39.889+0000 7fdb77e00640 1 -- 172.21.15.32:0/2717765032 shutdown_connections 2024-08-03T21:00:39.891 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:39.889+0000 7fdb77e00640 1 -- 172.21.15.32:0/2717765032 wait complete. 2024-08-03T21:00:39.892 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:39.891+0000 7fdb77e00640 1 Processor -- start 2024-08-03T21:00:39.892 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:39.892+0000 7fdb77e00640 1 -- start start 2024-08-03T21:00:39.895 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:39.893+0000 7fdb77e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdb780b94d0 0x7fdb7814deb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:00:39.896 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:39.893+0000 7fdb77e00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fdb780d4ca0 0x7fdb780c7c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:00:39.896 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:39.893+0000 7fdb77e00640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fdb780b6fb0 0x7fdb780b73c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:00:39.896 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:39.893+0000 7fdb77e00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fdb780b4980 con 0x7fdb780b94d0 2024-08-03T21:00:39.896 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:39.893+0000 7fdb77e00640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7fdb780b4800 con 0x7fdb780d4ca0 2024-08-03T21:00:39.896 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:39.893+0000 7fdb77e00640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7fdb780b4b00 con 0x7fdb780b6fb0 2024-08-03T21:00:39.896 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:39.893+0000 7fdb76a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdb780b94d0 0x7fdb7814deb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:00:39.896 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:39.893+0000 7fdb76a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdb780b94d0 0x7fdb7814deb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:56874/0 (socket says 172.21.15.32:56874) 2024-08-03T21:00:39.896 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:39.893+0000 7fdb76a00640 1 -- 172.21.15.32:0/91326304 learned_addr learned my addr 172.21.15.32:0/91326304 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T21:00:39.896 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:39.894+0000 7fdb77400640 1 --2- 172.21.15.32:0/91326304 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fdb780b6fb0 0x7fdb780b73c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:00:39.896 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:39.894+0000 7fdb76000640 1 --2- 172.21.15.32:0/91326304 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fdb780d4ca0 0x7fdb780c7c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:00:39.896 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:39.894+0000 7fdb76a00640 1 -- 172.21.15.32:0/91326304 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fdb780b6fb0 msgr2=0x7fdb780b73c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:00:39.897 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:39.894+0000 7fdb76a00640 1 --2- 172.21.15.32:0/91326304 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fdb780b6fb0 0x7fdb780b73c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:39.897 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:39.894+0000 7fdb76a00640 1 -- 172.21.15.32:0/91326304 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fdb780d4ca0 msgr2=0x7fdb780c7c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:00:39.897 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:39.894+0000 7fdb76a00640 1 --2- 172.21.15.32:0/91326304 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fdb780d4ca0 0x7fdb780c7c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:39.897 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:39.894+0000 7fdb76a00640 1 -- 172.21.15.32:0/91326304 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdb780b7900 con 0x7fdb780b94d0 2024-08-03T21:00:39.897 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:39.894+0000 7fdb77400640 1 --2- 172.21.15.32:0/91326304 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fdb780b6fb0 0x7fdb780b73c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-03T21:00:39.897 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:39.894+0000 7fdb76a00640 1 --2- 172.21.15.32:0/91326304 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdb780b94d0 0x7fdb7814deb0 secure :-1 s=READY pgs=2120 cs=0 l=1 rev1=1 crypto rx=0x7fdb6800c390 tx=0x7fdb6800c850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T21:00:39.897 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:39.895+0000 7fdb6ea00640 1 -- 172.21.15.32:0/91326304 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb68017070 con 0x7fdb780b94d0 2024-08-03T21:00:39.897 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:39.895+0000 7fdb77e00640 1 -- 172.21.15.32:0/91326304 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdb780b7f40 con 0x7fdb780b94d0 2024-08-03T21:00:39.897 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:39.895+0000 7fdb6ea00640 1 -- 172.21.15.32:0/91326304 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fdb680053c0 con 0x7fdb780b94d0 2024-08-03T21:00:39.897 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:39.895+0000 7fdb6ea00640 1 -- 172.21.15.32:0/91326304 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb68003880 con 0x7fdb780b94d0 2024-08-03T21:00:39.897 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:39.895+0000 7fdb77e00640 1 -- 172.21.15.32:0/91326304 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdb780ca420 con 0x7fdb780b94d0 2024-08-03T21:00:39.898 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:39.896+0000 7fdb77e00640 1 -- 172.21.15.32:0/91326304 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdb780b4fb0 con 0x7fdb780b94d0 2024-08-03T21:00:39.898 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:39.897+0000 7fdb6ea00640 1 -- 172.21.15.32:0/91326304 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7fdb6801a070 con 0x7fdb780b94d0 2024-08-03T21:00:39.898 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:39.898+0000 7fdb6ea00640 1 --2- 172.21.15.32:0/91326304 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fdb58077010 0x7fdb58079520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:00:39.899 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:39.898+0000 7fdb6ea00640 1 -- 172.21.15.32:0/91326304 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(127..127 src has 1..127) ==== 8297+0+0 (secure 0 0 0) 0x7fdb68098790 con 0x7fdb780b94d0 2024-08-03T21:00:39.899 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:39.898+0000 7fdb76000640 1 --2- 172.21.15.32:0/91326304 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fdb58077010 0x7fdb58079520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:00:39.899 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:39.899+0000 7fdb76000640 1 --2- 172.21.15.32:0/91326304 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fdb58077010 0x7fdb58079520 secure :-1 s=READY pgs=847 cs=0 l=1 rev1=1 crypto rx=0x7fdb64033000 tx=0x7fdb64004730 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T21:00:39.903 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:39.902+0000 7fdb6ea00640 1 -- 172.21.15.32:0/91326304 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7fdb780b7900 con 0x7fdb780b94d0 2024-08-03T21:00:40.188 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:40.187+0000 7fdb77e00640 1 -- 172.21.15.32:0/91326304 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_damage.TestDamage.test_object_deletion"]} v 0) -- 0x7fdb7806e5f0 con 0x7fdb780b94d0 2024-08-03T21:00:41.152 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:41.151+0000 7fdb6ea00640 1 -- 172.21.15.32:0/91326304 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_damage.TestDamage.test_object_deletion"]}]=0 v680) ==== 135+0+0 (secure 0 0 0) 0x7fdb680691a0 con 0x7fdb780b94d0 2024-08-03T21:00:41.155 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:41.154+0000 7fdb77e00640 1 -- 172.21.15.32:0/91326304 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fdb58077010 msgr2=0x7fdb58079520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:00:41.155 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:41.154+0000 7fdb77e00640 1 --2- 172.21.15.32:0/91326304 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fdb58077010 0x7fdb58079520 secure :-1 s=READY pgs=847 cs=0 l=1 rev1=1 crypto rx=0x7fdb64033000 tx=0x7fdb64004730 comp rx=0 tx=0).stop 2024-08-03T21:00:41.156 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:41.154+0000 7fdb77e00640 1 -- 172.21.15.32:0/91326304 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdb780b94d0 msgr2=0x7fdb7814deb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:00:41.156 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:41.154+0000 7fdb77e00640 1 --2- 172.21.15.32:0/91326304 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdb780b94d0 0x7fdb7814deb0 secure :-1 s=READY pgs=2120 cs=0 l=1 rev1=1 crypto rx=0x7fdb6800c390 tx=0x7fdb6800c850 comp rx=0 tx=0).stop 2024-08-03T21:00:41.156 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:41.154+0000 7fdb77e00640 1 -- 172.21.15.32:0/91326304 shutdown_connections 2024-08-03T21:00:41.156 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:41.154+0000 7fdb77e00640 1 --2- 172.21.15.32:0/91326304 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fdb58077010 0x7fdb58079520 unknown :-1 s=CLOSED pgs=847 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:41.156 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:41.154+0000 7fdb77e00640 1 --2- 172.21.15.32:0/91326304 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fdb780d4ca0 0x7fdb780c7c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:41.156 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:41.154+0000 7fdb77e00640 1 --2- 172.21.15.32:0/91326304 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fdb780b6fb0 0x7fdb780b73c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:41.156 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:41.154+0000 7fdb77e00640 1 --2- 172.21.15.32:0/91326304 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdb780b94d0 0x7fdb7814deb0 unknown :-1 s=CLOSED pgs=2120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:41.156 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:41.155+0000 7fdb77e00640 1 -- 172.21.15.32:0/91326304 >> 172.21.15.32:0/91326304 conn(0x7fdb780d4180 msgr2=0x7fdb780d1d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T21:00:41.156 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:41.155+0000 7fdb77e00640 1 -- 172.21.15.32:0/91326304 shutdown_connections 2024-08-03T21:00:41.156 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:41.155+0000 7fdb77e00640 1 -- 172.21.15.32:0/91326304 wait complete. 2024-08-03T21:00:41.180 INFO:tasks.cephfs_test_runner:That the MDS has a clean 'damaged' response to loss of any single metadata object ... ok 2024-08-03T21:00:41.180 INFO:tasks.cephfs_test_runner:Starting test: test_open_ino_errors (tasks.cephfs.test_damage.TestDamage) That errors encountered during opening inos are properly propagated 2024-08-03T21:00:41.180 INFO:tasks.cephfs_test_runner:test_open_ino_errors (tasks.cephfs.test_damage.TestDamage) 2024-08-03T21:00:41.181 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-08-03T21:00:41.413 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:41.412+0000 7f33bb000640 1 -- 172.21.15.32:0/3676712615 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f33b406e930 msgr2=0x7f33b4144f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:00:41.414 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:41.412+0000 7f33bb000640 1 --2- 172.21.15.32:0/3676712615 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f33b406e930 0x7f33b4144f90 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f33a4009830 tx=0x7f33a402a510 comp rx=0 tx=0).stop 2024-08-03T21:00:41.414 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:41.413+0000 7f33bb000640 1 -- 172.21.15.32:0/3676712615 shutdown_connections 2024-08-03T21:00:41.414 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:41.413+0000 7f33bb000640 1 --2- 172.21.15.32:0/3676712615 >> v2:172.21.15.118:3300/0 conn(0x7f33b415fff0 0x7f33b4160400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:41.414 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:41.413+0000 7f33bb000640 1 --2- 172.21.15.32:0/3676712615 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f33b406e930 0x7f33b4144f90 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:41.414 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:41.413+0000 7f33bb000640 1 -- 172.21.15.32:0/3676712615 >> 172.21.15.32:0/3676712615 conn(0x7f33b40b1bd0 msgr2=0x7f33b40b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T21:00:41.414 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:41.413+0000 7f33bb000640 1 -- 172.21.15.32:0/3676712615 shutdown_connections 2024-08-03T21:00:41.414 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:41.413+0000 7f33bb000640 1 -- 172.21.15.32:0/3676712615 wait complete. 2024-08-03T21:00:41.415 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:41.415+0000 7f33bb000640 1 Processor -- start 2024-08-03T21:00:41.416 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:41.415+0000 7f33bb000640 1 -- start start 2024-08-03T21:00:41.419 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:41.417+0000 7f33bb000640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f33b406e930 0x7f33b40b4170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:00:41.419 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:41.417+0000 7f33bb000640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f33b4073e60 0x7f33b41386d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:00:41.419 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:41.417+0000 7f33bb000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f33b415fff0 0x7f33b41436b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:00:41.420 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:41.417+0000 7f33bb000640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f33b40cf840 con 0x7f33b415fff0 2024-08-03T21:00:41.420 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:41.417+0000 7f33bb000640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f33b4145f30 con 0x7f33b4073e60 2024-08-03T21:00:41.420 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:41.417+0000 7f33bb000640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7f33b40cf9c0 con 0x7f33b406e930 2024-08-03T21:00:41.420 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:41.417+0000 7f33b9000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f33b415fff0 0x7f33b41436b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:00:41.420 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:41.417+0000 7f33b9000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f33b415fff0 0x7f33b41436b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:56886/0 (socket says 172.21.15.32:56886) 2024-08-03T21:00:41.420 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:41.417+0000 7f33b9000640 1 -- 172.21.15.32:0/3784941736 learned_addr learned my addr 172.21.15.32:0/3784941736 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T21:00:41.420 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:41.417+0000 7f33abe00640 1 --2- 172.21.15.32:0/3784941736 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f33b4073e60 0x7f33b41386d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:00:41.420 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:41.417+0000 7f33b3e00640 1 --2- 172.21.15.32:0/3784941736 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f33b406e930 0x7f33b40b4170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:00:41.420 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:41.418+0000 7f33b9000640 1 -- 172.21.15.32:0/3784941736 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f33b406e930 msgr2=0x7f33b40b4170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:00:41.420 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:41.418+0000 7f33b9000640 1 --2- 172.21.15.32:0/3784941736 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f33b406e930 0x7f33b40b4170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:41.420 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:41.418+0000 7f33b9000640 1 -- 172.21.15.32:0/3784941736 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f33b4073e60 msgr2=0x7f33b41386d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:00:41.420 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:41.418+0000 7f33b9000640 1 --2- 172.21.15.32:0/3784941736 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f33b4073e60 0x7f33b41386d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:41.421 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:41.418+0000 7f33b9000640 1 -- 172.21.15.32:0/3784941736 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f33b41f8880 con 0x7f33b415fff0 2024-08-03T21:00:41.421 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:41.418+0000 7f33b3e00640 1 --2- 172.21.15.32:0/3784941736 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f33b406e930 0x7f33b40b4170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-03T21:00:41.421 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:41.418+0000 7f33b9000640 1 --2- 172.21.15.32:0/3784941736 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f33b415fff0 0x7f33b41436b0 secure :-1 s=READY pgs=2121 cs=0 l=1 rev1=1 crypto rx=0x7f33ac00ebf0 tx=0x7f33ac00c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T21:00:41.421 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:41.418+0000 7f33b0c00640 1 -- 172.21.15.32:0/3784941736 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33ac010630 con 0x7f33b415fff0 2024-08-03T21:00:41.421 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:41.418+0000 7f33bb000640 1 -- 172.21.15.32:0/3784941736 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f33b40b4c30 con 0x7f33b415fff0 2024-08-03T21:00:41.421 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:41.418+0000 7f33bb000640 1 -- 172.21.15.32:0/3784941736 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f33b41f8e00 con 0x7f33b415fff0 2024-08-03T21:00:41.421 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:41.418+0000 7f33b0c00640 1 -- 172.21.15.32:0/3784941736 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f33ac0107d0 con 0x7f33b415fff0 2024-08-03T21:00:41.421 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:41.418+0000 7f33b0c00640 1 -- 172.21.15.32:0/3784941736 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33ac010be0 con 0x7f33b415fff0 2024-08-03T21:00:41.422 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:41.420+0000 7f33bb000640 1 -- 172.21.15.32:0/3784941736 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f33b40b49a0 con 0x7f33b415fff0 2024-08-03T21:00:41.422 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:41.421+0000 7f33b0c00640 1 -- 172.21.15.32:0/3784941736 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f33ac021430 con 0x7f33b415fff0 2024-08-03T21:00:41.422 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:41.422+0000 7f33b0c00640 1 --2- 172.21.15.32:0/3784941736 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f3390077010 0x7f3390079520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:00:41.423 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:41.422+0000 7f33b0c00640 1 -- 172.21.15.32:0/3784941736 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(127..127 src has 1..127) ==== 8297+0+0 (secure 0 0 0) 0x7f33ac09a180 con 0x7f33b415fff0 2024-08-03T21:00:41.423 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:41.422+0000 7f33b3e00640 1 --2- 172.21.15.32:0/3784941736 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f3390077010 0x7f3390079520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:00:41.423 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:41.423+0000 7f33b3e00640 1 --2- 172.21.15.32:0/3784941736 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f3390077010 0x7f3390079520 secure :-1 s=READY pgs=848 cs=0 l=1 rev1=1 crypto rx=0x7f33a4034000 tx=0x7f33a4004730 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T21:00:41.427 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:41.426+0000 7f33b0c00640 1 -- 172.21.15.32:0/3784941736 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7f33ac062940 con 0x7f33b415fff0 2024-08-03T21:00:41.635 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:41.634+0000 7f33bb000640 1 -- 172.21.15.32:0/3784941736 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f33b4160bc0 con 0x7f33b415fff0 2024-08-03T21:00:41.636 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:41.635+0000 7f33b0c00640 1 -- 172.21.15.32:0/3784941736 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v127) ==== 74+0+18014 (secure 0 0 0) 0x7f33ac062240 con 0x7f33b415fff0 2024-08-03T21:00:41.636 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-03T21:00:41.636 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":127,"fsid":"40ab141a-01d7-49b9-ae3c-2edec9876e95","created":"2024-08-03T20:47:14.957362+0000","modified":"2024-08-03T21:00:39.138135+0000","last_up_change":"2024-08-03T20:47:28.874451+0000","last_in_change":"2024-08-03T20:47:20.079442+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":13,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-08-03T20:47:28.261300+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5a83e759-9035-438b-889a-0a6af1f0a0d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6816","nonce":3066156116},{"type":"v1","addr":"172.21.15.32:6817","nonce":3066156116}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6818","nonce":3066156116},{"type":"v1","addr":"172.21.15.32:6819","nonce":3066156116}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6822","nonce":3066156116},{"type":"v1","addr":"172.21.15.32:6823","nonce":3066156116}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6820","nonce":3066156116},{"type":"v1","addr":"172.21.15.32:6821","nonce":3066156116}]},"public_addr":"172.21.15.32:6817/3066156116","cluster_addr":"172.21.15.32:6819/3066156116","heartbeat_back_addr":"172.21.15.32:6823/3066156116","heartbeat_front_addr":"172.21.15.32:6821/3066156116","state":["exists","up"]},{"osd":1,"uuid":"d7c5c210-da4f-49f9-b9e7-1dfb02daa4bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6824","nonce":1551241759},{"type":"v1","addr":"172.21.15.32:6825","nonce":1551241759}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6826","nonce":1551241759},{"type":"v1","addr":"172.21.15.32:6827","nonce":1551241759}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6830","nonce":1551241759},{"type":"v1","addr":"172.21.15.32:6831","nonce":1551241759}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6828","nonce":1551241759},{"type":"v1","addr":"172.21.15.32:6829","nonce":1551241759}]},"public_addr":"172.21.15.32:6825/1551241759","cluster_addr":"172.21.15.32:6827/1551241759","heartbeat_back_addr":"172.21.15.32:6831/1551241759","heartbeat_front_addr":"172.21.15.32:6829/1551241759","state":["exists","up"]},{"osd":2,"uuid":"57f2e6de-a201-45c2-8981-8e56a4875ebd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6808","nonce":3806708944},{"type":"v1","addr":"172.21.15.32:6809","nonce":3806708944}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6810","nonce":3806708944},{"type":"v1","addr":"172.21.15.32:6811","nonce":3806708944}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6814","nonce":3806708944},{"type":"v1","addr":"172.21.15.32:6815","nonce":3806708944}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6812","nonce":3806708944},{"type":"v1","addr":"172.21.15.32:6813","nonce":3806708944}]},"public_addr":"172.21.15.32:6809/3806708944","cluster_addr":"172.21.15.32:6811/3806708944","heartbeat_back_addr":"172.21.15.32:6815/3806708944","heartbeat_front_addr":"172.21.15.32:6813/3806708944","state":["exists","up"]},{"osd":3,"uuid":"6439a280-da45-4148-8130-f16323935ff3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6800","nonce":9088303},{"type":"v1","addr":"172.21.15.32:6801","nonce":9088303}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6802","nonce":9088303},{"type":"v1","addr":"172.21.15.32:6803","nonce":9088303}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6806","nonce":9088303},{"type":"v1","addr":"172.21.15.32:6807","nonce":9088303}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6804","nonce":9088303},{"type":"v1","addr":"172.21.15.32:6805","nonce":9088303}]},"public_addr":"172.21.15.32:6801/9088303","cluster_addr":"172.21.15.32:6803/9088303","heartbeat_back_addr":"172.21.15.32:6807/9088303","heartbeat_front_addr":"172.21.15.32:6805/9088303","state":["exists","up"]},{"osd":4,"uuid":"4f902eff-1e18-418d-82c9-2151c69524fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6802","nonce":864752275},{"type":"v1","addr":"172.21.15.118:6803","nonce":864752275}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6804","nonce":864752275},{"type":"v1","addr":"172.21.15.118:6805","nonce":864752275}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6808","nonce":864752275},{"type":"v1","addr":"172.21.15.118:6809","nonce":864752275}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6806","nonce":864752275},{"type":"v1","addr":"172.21.15.118:6807","nonce":864752275}]},"public_addr":"172.21.15.118:6803/864752275","cluster_addr":"172.21.15.118:6805/864752275","heartbeat_back_addr":"172.21.15.118:6809/864752275","heartbeat_front_addr":"172.21.15.118:6807/864752275","state":["exists","up"]},{"osd":5,"uuid":"bfd9feac-dd06-458c-91b9-af7f175f612c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6810","nonce":2724891770},{"type":"v1","addr":"172.21.15.118:6811","nonce":2724891770}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6812","nonce":2724891770},{"type":"v1","addr":"172.21.15.118:6813","nonce":2724891770}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6816","nonce":2724891770},{"type":"v1","addr":"172.21.15.118:6817","nonce":2724891770}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6814","nonce":2724891770},{"type":"v1","addr":"172.21.15.118:6815","nonce":2724891770}]},"public_addr":"172.21.15.118:6811/2724891770","cluster_addr":"172.21.15.118:6813/2724891770","heartbeat_back_addr":"172.21.15.118:6817/2724891770","heartbeat_front_addr":"172.21.15.118:6815/2724891770","state":["exists","up"]},{"osd":6,"uuid":"5ed5a279-9d8e-416d-bbc7-7edd3e6d5705","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6825","nonce":801947386},{"type":"v1","addr":"172.21.15.118:6827","nonce":801947386}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6828","nonce":801947386},{"type":"v1","addr":"172.21.15.118:6829","nonce":801947386}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6832","nonce":801947386},{"type":"v1","addr":"172.21.15.118:6833","nonce":801947386}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6830","nonce":801947386},{"type":"v1","addr":"172.21.15.118:6831","nonce":801947386}]},"public_addr":"172.21.15.118:6827/801947386","cluster_addr":"172.21.15.118:6829/801947386","heartbeat_back_addr":"172.21.15.118:6833/801947386","heartbeat_front_addr":"172.21.15.118:6831/801947386","state":["exists","up"]},{"osd":7,"uuid":"f2086c47-875e-4451-a436-2008d1ea16af","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6818","nonce":653000048},{"type":"v1","addr":"172.21.15.118:6819","nonce":653000048}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6820","nonce":653000048},{"type":"v1","addr":"172.21.15.118:6821","nonce":653000048}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6824","nonce":653000048},{"type":"v1","addr":"172.21.15.118:6826","nonce":653000048}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6822","nonce":653000048},{"type":"v1","addr":"172.21.15.118:6823","nonce":653000048}]},"public_addr":"172.21.15.118:6819/653000048","cluster_addr":"172.21.15.118:6821/653000048","heartbeat_back_addr":"172.21.15.118:6826/653000048","heartbeat_front_addr":"172.21.15.118:6823/653000048","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-03T20:47:25.491994+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-03T20:47:25.587358+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-03T20:47:25.454175+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-03T20:47:25.377734+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-03T20:47:25.412351+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-03T20:47:25.499815+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-03T20:47:25.575044+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-03T20:47:25.552383+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.32:6835/1660875176":"2024-08-04T21:00:28.099287+0000","172.21.15.32:6833/1660875176":"2024-08-04T21:00:28.099287+0000","172.21.15.32:6835/2824868449":"2024-08-04T21:00:19.050715+0000","172.21.15.32:6835/73253044":"2024-08-04T21:00:10.997224+0000","172.21.15.32:6835/2017655932":"2024-08-04T21:00:04.959586+0000","172.21.15.32:6833/2017655932":"2024-08-04T21:00:04.959586+0000","172.21.15.32:6835/3127944613":"2024-08-04T20:59:46.766712+0000","172.21.15.32:6835/2106229503":"2024-08-04T20:59:40.719664+0000","172.21.15.32:6833/2106229503":"2024-08-04T20:59:40.719664+0000","172.21.15.32:6835/2173042128":"2024-08-04T20:59:28.193772+0000","172.21.15.32:6833/2173042128":"2024-08-04T20:59:28.193772+0000","172.21.15.32:6835/506775245":"2024-08-04T20:59:03.076256+0000","172.21.15.32:6833/506775245":"2024-08-04T20:59:03.076256+0000","172.21.15.32:6833/556650445":"2024-08-04T20:58:57.063754+0000","172.21.15.32:6835/2156596385":"2024-08-04T20:58:35.073838+0000","172.21.15.32:6833/2156596385":"2024-08-04T20:58:35.073838+0000","172.21.15.32:6835/3669493978":"2024-08-04T20:58:30.064676+0000","172.21.15.32:6833/2178506907":"2024-08-04T20:56:40.045549+0000","172.21.15.32:6835/556650445":"2024-08-04T20:58:57.063754+0000","172.21.15.32:6835/2278473729":"2024-08-04T20:56:13.041693+0000","172.21.15.32:6835/2345472481":"2024-08-04T20:57:25.968590+0000","172.21.15.32:6833/3940515773":"2024-08-04T20:58:24.290690+0000","172.21.15.32:6833/2600386707":"2024-08-04T20:57:50.063964+0000","172.21.15.32:6835/2178506907":"2024-08-04T20:56:40.045549+0000","172.21.15.32:6833/73253044":"2024-08-04T21:00:10.997224+0000","172.21.15.32:6833/2292690833":"2024-08-04T20:55:20.021329+0000","172.21.15.32:6835/2803901497":"2024-08-04T20:55:55.181144+0000","172.21.15.32:6835/2194902489":"2024-08-04T20:55:00.359126+0000","172.21.15.32:6833/2824868449":"2024-08-04T21:00:19.050715+0000","172.21.15.32:6833/3127944613":"2024-08-04T20:59:46.766712+0000","172.21.15.32:6833/2803901497":"2024-08-04T20:55:55.181144+0000","172.21.15.32:6835/3191654276":"2024-08-04T20:55:42.433364+0000","172.21.15.32:6833/2242533881":"2024-08-04T20:56:48.048099+0000","172.21.15.32:6833/2283924374":"2024-08-04T20:57:34.396782+0000","172.21.15.32:6835/1172050551":"2024-08-04T20:55:26.214113+0000","172.21.15.32:6835/1016338829":"2024-08-04T20:55:06.071485+0000","172.21.15.32:6833/1474984145":"2024-08-04T20:57:40.052808+0000","172.21.15.32:6833/3669493978":"2024-08-04T20:58:30.064676+0000","172.21.15.32:6835/756683988":"2024-08-04T20:55:14.119228+0000","172.21.15.32:6833/2194902489":"2024-08-04T20:55:00.359126+0000","172.21.15.32:6835/2806168786":"2024-08-04T20:56:37.279160+0000","172.21.15.32:6833/2278473729":"2024-08-04T20:56:13.041693+0000","172.21.15.32:6833/3191654276":"2024-08-04T20:55:42.433364+0000","172.21.15.32:6835/3431300225":"2024-08-04T20:56:21.088927+0000","172.21.15.32:6833/4172198298":"2024-08-04T20:56:04.256182+0000","172.21.15.32:6833/3431300225":"2024-08-04T20:56:21.088927+0000","172.21.15.32:6835/2292690833":"2024-08-04T20:55:20.021329+0000","172.21.15.32:6833/2806168786":"2024-08-04T20:56:37.279160+0000","172.21.15.32:6833/1172050551":"2024-08-04T20:55:26.214113+0000","172.21.15.32:6835/4172198298":"2024-08-04T20:56:04.256182+0000","172.21.15.32:6835/3409752696":"2024-08-04T20:58:18.243140+0000","172.21.15.32:6833/2274054575":"2024-08-04T20:57:55.062490+0000","172.21.15.32:6833/1016338829":"2024-08-04T20:55:06.071485+0000","172.21.15.32:6833/1707454554":"2024-08-04T20:54:38.409411+0000","172.21.15.32:6835/2242533881":"2024-08-04T20:56:48.048099+0000","172.21.15.32:6833/739874257":"2024-08-04T20:56:56.101271+0000","172.21.15.32:6835/739874257":"2024-08-04T20:56:56.101271+0000","172.21.15.32:6833/2362145329":"2024-08-04T20:57:02.150199+0000","172.21.15.32:6833/3771304550":"2024-08-04T20:55:58.212752+0000","172.21.15.32:6835/2362145329":"2024-08-04T20:57:02.150199+0000","172.21.15.32:6835/3940515773":"2024-08-04T20:58:24.290690+0000","172.21.15.32:6833/1885034124":"2024-08-04T20:57:08.190707+0000","172.21.15.32:6833/511221267":"2024-08-04T20:57:45.062448+0000","172.21.15.32:6835/3771304550":"2024-08-04T20:55:58.212752+0000","172.21.15.32:6835/1885034124":"2024-08-04T20:57:08.190707+0000","172.21.15.32:6833/2345472481":"2024-08-04T20:57:25.968590+0000","172.21.15.32:6833/265163622":"2024-08-04T20:57:28.354509+0000","172.21.15.32:6835/265163622":"2024-08-04T20:57:28.354509+0000","172.21.15.32:6835/2274054575":"2024-08-04T20:57:55.062490+0000","172.21.15.32:6835/2283924374":"2024-08-04T20:57:34.396782+0000","172.21.15.32:6833/756683988":"2024-08-04T20:55:14.119228+0000","172.21.15.32:6835/1474984145":"2024-08-04T20:57:40.052808+0000","172.21.15.32:6835/511221267":"2024-08-04T20:57:45.062448+0000","172.21.15.32:6835/1707454554":"2024-08-04T20:54:38.409411+0000","172.21.15.32:6835/2600386707":"2024-08-04T20:57:50.063964+0000","172.21.15.32:6833/675070591":"2024-08-04T20:58:00.068114+0000","172.21.15.32:6833/3340816283":"2024-08-04T20:58:15.214542+0000","172.21.15.32:6835/3340816283":"2024-08-04T20:58:15.214542+0000","172.21.15.32:6835/675070591":"2024-08-04T20:58:00.068114+0000","172.21.15.32:6833/3409752696":"2024-08-04T20:58:18.243140+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-08-03T21:00:41.639 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:41.638+0000 7f33bb000640 1 -- 172.21.15.32:0/3784941736 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f3390077010 msgr2=0x7f3390079520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:00:41.639 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:41.638+0000 7f33bb000640 1 --2- 172.21.15.32:0/3784941736 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f3390077010 0x7f3390079520 secure :-1 s=READY pgs=848 cs=0 l=1 rev1=1 crypto rx=0x7f33a4034000 tx=0x7f33a4004730 comp rx=0 tx=0).stop 2024-08-03T21:00:41.639 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:41.638+0000 7f33bb000640 1 -- 172.21.15.32:0/3784941736 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f33b415fff0 msgr2=0x7f33b41436b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:00:41.639 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:41.638+0000 7f33bb000640 1 --2- 172.21.15.32:0/3784941736 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f33b415fff0 0x7f33b41436b0 secure :-1 s=READY pgs=2121 cs=0 l=1 rev1=1 crypto rx=0x7f33ac00ebf0 tx=0x7f33ac00c6a0 comp rx=0 tx=0).stop 2024-08-03T21:00:41.639 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:41.638+0000 7f33bb000640 1 -- 172.21.15.32:0/3784941736 shutdown_connections 2024-08-03T21:00:41.639 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:41.638+0000 7f33bb000640 1 --2- 172.21.15.32:0/3784941736 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f3390077010 0x7f3390079520 unknown :-1 s=CLOSED pgs=848 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:41.639 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:41.638+0000 7f33bb000640 1 --2- 172.21.15.32:0/3784941736 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f33b4073e60 0x7f33b41386d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:41.639 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:41.638+0000 7f33bb000640 1 --2- 172.21.15.32:0/3784941736 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f33b415fff0 0x7f33b41436b0 unknown :-1 s=CLOSED pgs=2121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:41.639 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:41.638+0000 7f33bb000640 1 --2- 172.21.15.32:0/3784941736 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f33b406e930 0x7f33b40b4170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:41.639 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:41.638+0000 7f33bb000640 1 -- 172.21.15.32:0/3784941736 >> 172.21.15.32:0/3784941736 conn(0x7f33b40b1bd0 msgr2=0x7f33b415ca20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T21:00:41.640 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:41.638+0000 7f33bb000640 1 -- 172.21.15.32:0/3784941736 shutdown_connections 2024-08-03T21:00:41.640 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:41.638+0000 7f33bb000640 1 -- 172.21.15.32:0/3784941736 wait complete. 2024-08-03T21:00:41.661 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-08-03T20:47:28.261300+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-08-03T21:00:41.661 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-08-03T21:00:41.893 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:41.891+0000 7f2ee6000640 1 -- 172.21.15.32:0/3175353743 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2ee00ced30 msgr2=0x7f2ee00cf140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:00:41.893 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:41.891+0000 7f2ee6000640 1 --2- 172.21.15.32:0/3175353743 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2ee00ced30 0x7f2ee00cf140 secure :-1 s=READY pgs=2123 cs=0 l=1 rev1=1 crypto rx=0x7f2ed8005fd0 tx=0x7f2ed802a5c0 comp rx=0 tx=0).stop 2024-08-03T21:00:41.893 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:41.891+0000 7f2ee6000640 1 -- 172.21.15.32:0/3175353743 shutdown_connections 2024-08-03T21:00:41.893 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:41.891+0000 7f2ee6000640 1 --2- 172.21.15.32:0/3175353743 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2ee00ced30 0x7f2ee00cf140 unknown :-1 s=CLOSED pgs=2123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:41.893 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:41.891+0000 7f2ee6000640 1 --2- 172.21.15.32:0/3175353743 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f2ee00cccf0 0x7f2ee00cd100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:41.893 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:41.891+0000 7f2ee6000640 1 -- 172.21.15.32:0/3175353743 >> 172.21.15.32:0/3175353743 conn(0x7f2ee00b1bd0 msgr2=0x7f2ee00b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T21:00:41.893 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:41.891+0000 7f2ee6000640 1 -- 172.21.15.32:0/3175353743 shutdown_connections 2024-08-03T21:00:41.893 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:41.891+0000 7f2ee6000640 1 -- 172.21.15.32:0/3175353743 wait complete. 2024-08-03T21:00:41.894 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:41.894+0000 7f2ee6000640 1 Processor -- start 2024-08-03T21:00:41.894 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:41.894+0000 7f2ee6000640 1 -- start start 2024-08-03T21:00:41.897 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:41.895+0000 7f2ee6000640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f2ee00cccf0 0x7f2ee013e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:00:41.898 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:41.895+0000 7f2ee6000640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f2ee00ced30 0x7f2ee015bc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:00:41.898 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:41.895+0000 7f2ee6000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2ee0145c50 0x7f2ee00cb260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:00:41.898 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:41.895+0000 7f2ee6000640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f2ee00b4070 con 0x7f2ee0145c50 2024-08-03T21:00:41.898 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:41.895+0000 7f2ee6000640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f2ee00b3ef0 con 0x7f2ee00ced30 2024-08-03T21:00:41.898 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:41.895+0000 7f2ee6000640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7f2ee00b41f0 con 0x7f2ee00cccf0 2024-08-03T21:00:41.898 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:41.895+0000 7f2edfe00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2ee0145c50 0x7f2ee00cb260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:00:41.898 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:41.896+0000 7f2edfe00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2ee0145c50 0x7f2ee00cb260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:56904/0 (socket says 172.21.15.32:56904) 2024-08-03T21:00:41.898 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:41.896+0000 7f2edfe00640 1 -- 172.21.15.32:0/3726164403 learned_addr learned my addr 172.21.15.32:0/3726164403 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T21:00:41.898 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:41.896+0000 7f2edf400640 1 --2- 172.21.15.32:0/3726164403 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f2ee00cccf0 0x7f2ee013e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:00:41.899 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:41.896+0000 7f2ed6a00640 1 --2- 172.21.15.32:0/3726164403 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f2ee00ced30 0x7f2ee015bc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:00:41.899 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:41.896+0000 7f2edfe00640 1 -- 172.21.15.32:0/3726164403 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f2ee00cccf0 msgr2=0x7f2ee013e090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:00:41.899 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:41.896+0000 7f2edfe00640 1 --2- 172.21.15.32:0/3726164403 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f2ee00cccf0 0x7f2ee013e090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:41.899 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:41.896+0000 7f2edfe00640 1 -- 172.21.15.32:0/3726164403 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f2ee00ced30 msgr2=0x7f2ee015bc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:00:41.899 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:41.896+0000 7f2edfe00640 1 --2- 172.21.15.32:0/3726164403 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f2ee00ced30 0x7f2ee015bc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:41.899 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:41.896+0000 7f2edfe00640 1 -- 172.21.15.32:0/3726164403 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2ee013e5d0 con 0x7f2ee0145c50 2024-08-03T21:00:41.899 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:41.896+0000 7f2ed6a00640 1 --2- 172.21.15.32:0/3726164403 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f2ee00ced30 0x7f2ee015bc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-03T21:00:41.899 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:41.896+0000 7f2edfe00640 1 --2- 172.21.15.32:0/3726164403 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2ee0145c50 0x7f2ee00cb260 secure :-1 s=READY pgs=2124 cs=0 l=1 rev1=1 crypto rx=0x7f2ed8006010 tx=0x7f2ed8003890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T21:00:41.899 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:41.897+0000 7f2ed7e00640 1 -- 172.21.15.32:0/3726164403 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ed803e030 con 0x7f2ee0145c50 2024-08-03T21:00:41.899 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:41.897+0000 7f2ed7e00640 1 -- 172.21.15.32:0/3726164403 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f2ed8003d80 con 0x7f2ee0145c50 2024-08-03T21:00:41.899 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:41.897+0000 7f2ed7e00640 1 -- 172.21.15.32:0/3726164403 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ed8035040 con 0x7f2ee0145c50 2024-08-03T21:00:41.899 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:41.897+0000 7f2ee6000640 1 -- 172.21.15.32:0/3726164403 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2ee01f5780 con 0x7f2ee0145c50 2024-08-03T21:00:41.900 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:41.897+0000 7f2ee6000640 1 -- 172.21.15.32:0/3726164403 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2ee01f5d40 con 0x7f2ee0145c50 2024-08-03T21:00:41.901 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:41.898+0000 7f2ee6000640 1 -- 172.21.15.32:0/3726164403 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2ee00b46a0 con 0x7f2ee0145c50 2024-08-03T21:00:41.901 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:41.899+0000 7f2ed7e00640 1 -- 172.21.15.32:0/3726164403 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f2ed8041030 con 0x7f2ee0145c50 2024-08-03T21:00:41.901 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:41.900+0000 7f2ed7e00640 1 --2- 172.21.15.32:0/3726164403 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f2ebc077010 0x7f2ebc079520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:00:41.901 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:41.900+0000 7f2ed7e00640 1 -- 172.21.15.32:0/3726164403 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(127..127 src has 1..127) ==== 8297+0+0 (secure 0 0 0) 0x7f2ed80b9df0 con 0x7f2ee0145c50 2024-08-03T21:00:41.901 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:41.901+0000 7f2edf400640 1 --2- 172.21.15.32:0/3726164403 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f2ebc077010 0x7f2ebc079520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:00:41.902 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:41.901+0000 7f2edf400640 1 --2- 172.21.15.32:0/3726164403 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f2ebc077010 0x7f2ebc079520 secure :-1 s=READY pgs=849 cs=0 l=1 rev1=1 crypto rx=0x7f2ed0001e20 tx=0x7f2ed0009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T21:00:41.905 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:41.905+0000 7f2ed7e00640 1 -- 172.21.15.32:0/3726164403 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7f2ed8081f60 con 0x7f2ee0145c50 2024-08-03T21:00:42.108 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:42.108+0000 7f2ee6000640 1 -- 172.21.15.32:0/3726164403 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f2ee013e870 con 0x7f2ee0145c50 2024-08-03T21:00:42.109 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:42.109+0000 7f2ed7e00640 1 -- 172.21.15.32:0/3726164403 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v127) ==== 93+0+10 (secure 0 0 0) 0x7f2ed80864b0 con 0x7f2ee0145c50 2024-08-03T21:00:42.109 INFO:teuthology.orchestra.run.smithi032.stdout:pg_num: 1 2024-08-03T21:00:42.112 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:42.111+0000 7f2ee6000640 1 -- 172.21.15.32:0/3726164403 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f2ebc077010 msgr2=0x7f2ebc079520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:00:42.112 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:42.111+0000 7f2ee6000640 1 --2- 172.21.15.32:0/3726164403 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f2ebc077010 0x7f2ebc079520 secure :-1 s=READY pgs=849 cs=0 l=1 rev1=1 crypto rx=0x7f2ed0001e20 tx=0x7f2ed0009040 comp rx=0 tx=0).stop 2024-08-03T21:00:42.112 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:42.111+0000 7f2ee6000640 1 -- 172.21.15.32:0/3726164403 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2ee0145c50 msgr2=0x7f2ee00cb260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:00:42.112 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:42.111+0000 7f2ee6000640 1 --2- 172.21.15.32:0/3726164403 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2ee0145c50 0x7f2ee00cb260 secure :-1 s=READY pgs=2124 cs=0 l=1 rev1=1 crypto rx=0x7f2ed8006010 tx=0x7f2ed8003890 comp rx=0 tx=0).stop 2024-08-03T21:00:42.112 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:42.111+0000 7f2ee6000640 1 -- 172.21.15.32:0/3726164403 shutdown_connections 2024-08-03T21:00:42.112 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:42.111+0000 7f2ee6000640 1 --2- 172.21.15.32:0/3726164403 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f2ebc077010 0x7f2ebc079520 unknown :-1 s=CLOSED pgs=849 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:42.113 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:42.111+0000 7f2ee6000640 1 --2- 172.21.15.32:0/3726164403 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f2ee00ced30 0x7f2ee015bc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:42.113 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:42.111+0000 7f2ee6000640 1 --2- 172.21.15.32:0/3726164403 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2ee0145c50 0x7f2ee00cb260 unknown :-1 s=CLOSED pgs=2124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:42.113 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:42.111+0000 7f2ee6000640 1 --2- 172.21.15.32:0/3726164403 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f2ee00cccf0 0x7f2ee013e090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:42.113 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:42.111+0000 7f2ee6000640 1 -- 172.21.15.32:0/3726164403 >> 172.21.15.32:0/3726164403 conn(0x7f2ee00b1bd0 msgr2=0x7f2ee00727f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T21:00:42.113 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:42.111+0000 7f2ee6000640 1 -- 172.21.15.32:0/3726164403 shutdown_connections 2024-08-03T21:00:42.113 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:42.111+0000 7f2ee6000640 1 -- 172.21.15.32:0/3726164403 wait complete. 2024-08-03T21:00:42.139 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_damage.TestDamage.test_open_ino_errors' 2024-08-03T21:00:42.368 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:42.368+0000 7f2ca7e00640 1 -- 172.21.15.32:0/29922519 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2ca015e170 msgr2=0x7f2ca015e580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:00:42.369 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:42.368+0000 7f2ca7e00640 1 --2- 172.21.15.32:0/29922519 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2ca015e170 0x7f2ca015e580 secure :-1 s=READY pgs=2125 cs=0 l=1 rev1=1 crypto rx=0x7f2ca0072280 tx=0x7f2c9802a660 comp rx=0 tx=0).stop 2024-08-03T21:00:42.369 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:42.368+0000 7f2ca7e00640 1 -- 172.21.15.32:0/29922519 shutdown_connections 2024-08-03T21:00:42.369 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:42.368+0000 7f2ca7e00640 1 --2- 172.21.15.32:0/29922519 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2ca015e170 0x7f2ca015e580 unknown :-1 s=CLOSED pgs=2125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:42.369 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:42.368+0000 7f2ca7e00640 1 --2- 172.21.15.32:0/29922519 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f2ca00733f0 0x7f2ca0073800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:42.369 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:42.368+0000 7f2ca7e00640 1 --2- 172.21.15.32:0/29922519 >> v2:172.21.15.118:3300/0 conn(0x7f2ca00b4de0 0x7f2ca01515c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:42.369 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:42.368+0000 7f2ca7e00640 1 -- 172.21.15.32:0/29922519 >> 172.21.15.32:0/29922519 conn(0x7f2ca00b1bd0 msgr2=0x7f2ca00b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T21:00:42.369 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:42.368+0000 7f2ca7e00640 1 -- 172.21.15.32:0/29922519 shutdown_connections 2024-08-03T21:00:42.369 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:42.368+0000 7f2ca7e00640 1 -- 172.21.15.32:0/29922519 wait complete. 2024-08-03T21:00:42.370 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:42.370+0000 7f2ca7e00640 1 Processor -- start 2024-08-03T21:00:42.371 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:42.370+0000 7f2ca7e00640 1 -- start start 2024-08-03T21:00:42.373 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:42.372+0000 7f2ca7e00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f2ca00733f0 0x7f2ca00a9ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:00:42.373 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:42.372+0000 7f2ca7e00640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f2ca00b4de0 0x7f2ca0153c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:00:42.374 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:42.372+0000 7f2ca7e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2ca015e170 0x7f2ca014e5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:00:42.374 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:42.372+0000 7f2ca7e00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f2ca006e930 con 0x7f2ca015e170 2024-08-03T21:00:42.374 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:42.372+0000 7f2ca7e00640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f2ca0070ad0 con 0x7f2ca00733f0 2024-08-03T21:00:42.374 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:42.372+0000 7f2ca7e00640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7f2ca006eab0 con 0x7f2ca00b4de0 2024-08-03T21:00:42.374 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:42.372+0000 7f2ca5e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2ca015e170 0x7f2ca014e5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:00:42.374 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:42.372+0000 7f2ca5e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2ca015e170 0x7f2ca014e5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:56930/0 (socket says 172.21.15.32:56930) 2024-08-03T21:00:42.374 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:42.372+0000 7f2ca5e00640 1 -- 172.21.15.32:0/3933879515 learned_addr learned my addr 172.21.15.32:0/3933879515 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T21:00:42.374 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:42.372+0000 7f2ca5400640 1 --2- 172.21.15.32:0/3933879515 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f2ca00733f0 0x7f2ca00a9ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:00:42.374 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:42.372+0000 7f2ca4a00640 1 --2- 172.21.15.32:0/3933879515 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f2ca00b4de0 0x7f2ca0153c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:00:42.374 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:42.373+0000 7f2ca5e00640 1 -- 172.21.15.32:0/3933879515 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f2ca00b4de0 msgr2=0x7f2ca0153c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:00:42.374 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:42.373+0000 7f2ca5e00640 1 --2- 172.21.15.32:0/3933879515 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f2ca00b4de0 0x7f2ca0153c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:42.374 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:42.373+0000 7f2ca5e00640 1 -- 172.21.15.32:0/3933879515 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f2ca00733f0 msgr2=0x7f2ca00a9ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:00:42.375 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:42.373+0000 7f2ca5e00640 1 --2- 172.21.15.32:0/3933879515 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f2ca00733f0 0x7f2ca00a9ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:42.375 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:42.373+0000 7f2ca5e00640 1 -- 172.21.15.32:0/3933879515 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2ca00aa420 con 0x7f2ca015e170 2024-08-03T21:00:42.375 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:42.373+0000 7f2ca4a00640 1 --2- 172.21.15.32:0/3933879515 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f2ca00b4de0 0x7f2ca0153c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-03T21:00:42.375 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:42.373+0000 7f2ca5400640 1 --2- 172.21.15.32:0/3933879515 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f2ca00733f0 0x7f2ca00a9ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-03T21:00:42.375 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:42.373+0000 7f2ca5e00640 1 --2- 172.21.15.32:0/3933879515 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2ca015e170 0x7f2ca014e5c0 secure :-1 s=READY pgs=2126 cs=0 l=1 rev1=1 crypto rx=0x7f2c980098c0 tx=0x7f2c98003ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T21:00:42.376 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:42.373+0000 7f2c9d600640 1 -- 172.21.15.32:0/3933879515 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c98004240 con 0x7f2ca015e170 2024-08-03T21:00:42.376 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:42.373+0000 7f2c9d600640 1 -- 172.21.15.32:0/3933879515 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f2c980044d0 con 0x7f2ca015e170 2024-08-03T21:00:42.376 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:42.373+0000 7f2c9d600640 1 -- 172.21.15.32:0/3933879515 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c9803b410 con 0x7f2ca015e170 2024-08-03T21:00:42.376 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:42.373+0000 7f2ca7e00640 1 -- 172.21.15.32:0/3933879515 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2ca00aa8f0 con 0x7f2ca015e170 2024-08-03T21:00:42.376 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:42.374+0000 7f2ca7e00640 1 -- 172.21.15.32:0/3933879515 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2ca00cf7f0 con 0x7f2ca015e170 2024-08-03T21:00:42.376 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:42.375+0000 7f2ca7e00640 1 -- 172.21.15.32:0/3933879515 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2ca01515c0 con 0x7f2ca015e170 2024-08-03T21:00:42.376 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:42.376+0000 7f2c9d600640 1 -- 172.21.15.32:0/3933879515 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f2c9803b5b0 con 0x7f2ca015e170 2024-08-03T21:00:42.378 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:42.377+0000 7f2c9d600640 1 --2- 172.21.15.32:0/3933879515 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f2c80077010 0x7f2c80079520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:00:42.378 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:42.377+0000 7f2c9d600640 1 -- 172.21.15.32:0/3933879515 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(127..127 src has 1..127) ==== 8297+0+0 (secure 0 0 0) 0x7f2c980ba910 con 0x7f2ca015e170 2024-08-03T21:00:42.378 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:42.378+0000 7f2ca5400640 1 --2- 172.21.15.32:0/3933879515 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f2c80077010 0x7f2c80079520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:00:42.379 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:42.379+0000 7f2ca5400640 1 --2- 172.21.15.32:0/3933879515 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f2c80077010 0x7f2c80079520 secure :-1 s=READY pgs=850 cs=0 l=1 rev1=1 crypto rx=0x7f2c94005fd0 tx=0x7f2c94005c70 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T21:00:42.382 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:42.382+0000 7f2c9d600640 1 -- 172.21.15.32:0/3933879515 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7f2c9803f030 con 0x7f2ca015e170 2024-08-03T21:00:42.667 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:42.666+0000 7f2ca7e00640 1 -- 172.21.15.32:0/3933879515 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_damage.TestDamage.test_open_ino_errors"]} v 0) -- 0x7f2ca00b3680 con 0x7f2ca015e170 2024-08-03T21:00:43.159 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:43.158+0000 7f2c9d600640 1 -- 172.21.15.32:0/3933879515 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_damage.TestDamage.test_open_ino_errors"]}]=0 v682) ==== 138+0+0 (secure 0 0 0) 0x7f2c9802f880 con 0x7f2ca015e170 2024-08-03T21:00:43.162 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:43.161+0000 7f2ca7e00640 1 -- 172.21.15.32:0/3933879515 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f2c80077010 msgr2=0x7f2c80079520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:00:43.162 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:43.161+0000 7f2ca7e00640 1 --2- 172.21.15.32:0/3933879515 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f2c80077010 0x7f2c80079520 secure :-1 s=READY pgs=850 cs=0 l=1 rev1=1 crypto rx=0x7f2c94005fd0 tx=0x7f2c94005c70 comp rx=0 tx=0).stop 2024-08-03T21:00:43.162 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:43.161+0000 7f2ca7e00640 1 -- 172.21.15.32:0/3933879515 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2ca015e170 msgr2=0x7f2ca014e5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:00:43.162 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:43.161+0000 7f2ca7e00640 1 --2- 172.21.15.32:0/3933879515 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2ca015e170 0x7f2ca014e5c0 secure :-1 s=READY pgs=2126 cs=0 l=1 rev1=1 crypto rx=0x7f2c980098c0 tx=0x7f2c98003ba0 comp rx=0 tx=0).stop 2024-08-03T21:00:43.162 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:43.161+0000 7f2ca7e00640 1 -- 172.21.15.32:0/3933879515 shutdown_connections 2024-08-03T21:00:43.162 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:43.161+0000 7f2ca7e00640 1 --2- 172.21.15.32:0/3933879515 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f2c80077010 0x7f2c80079520 unknown :-1 s=CLOSED pgs=850 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:43.163 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:43.161+0000 7f2ca7e00640 1 --2- 172.21.15.32:0/3933879515 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2ca015e170 0x7f2ca014e5c0 unknown :-1 s=CLOSED pgs=2126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:43.163 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:43.161+0000 7f2ca7e00640 1 --2- 172.21.15.32:0/3933879515 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f2ca00b4de0 0x7f2ca0153c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:43.163 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:43.161+0000 7f2ca7e00640 1 --2- 172.21.15.32:0/3933879515 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f2ca00733f0 0x7f2ca00a9ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:43.163 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:43.161+0000 7f2ca7e00640 1 -- 172.21.15.32:0/3933879515 >> 172.21.15.32:0/3933879515 conn(0x7f2ca00b1bd0 msgr2=0x7f2ca01595c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T21:00:43.163 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:43.161+0000 7f2ca7e00640 1 -- 172.21.15.32:0/3933879515 shutdown_connections 2024-08-03T21:00:43.163 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:43.162+0000 7f2ca7e00640 1 -- 172.21.15.32:0/3933879515 wait complete. 2024-08-03T21:00:43.188 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config dump 2024-08-03T21:00:43.418 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:43.417+0000 7f9d96000640 1 -- 172.21.15.32:0/2956145784 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9d9015fff0 msgr2=0x7f9d90160400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:00:43.418 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:43.417+0000 7f9d96000640 1 --2- 172.21.15.32:0/2956145784 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9d9015fff0 0x7f9d90160400 secure :-1 s=READY pgs=2127 cs=0 l=1 rev1=1 crypto rx=0x7f9d900afaf0 tx=0x7f9d8802aac0 comp rx=0 tx=0).stop 2024-08-03T21:00:43.418 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:43.417+0000 7f9d96000640 1 -- 172.21.15.32:0/2956145784 shutdown_connections 2024-08-03T21:00:43.418 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:43.417+0000 7f9d96000640 1 --2- 172.21.15.32:0/2956145784 >> v2:172.21.15.118:3300/0 conn(0x7f9d90144f90 0x7f9d90074570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:43.418 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:43.417+0000 7f9d96000640 1 --2- 172.21.15.32:0/2956145784 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9d9015fff0 0x7f9d90160400 unknown :-1 s=CLOSED pgs=2127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:43.419 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:43.417+0000 7f9d96000640 1 -- 172.21.15.32:0/2956145784 >> 172.21.15.32:0/2956145784 conn(0x7f9d900b1bd0 msgr2=0x7f9d900b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T21:00:43.419 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:43.417+0000 7f9d96000640 1 -- 172.21.15.32:0/2956145784 shutdown_connections 2024-08-03T21:00:43.419 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:43.417+0000 7f9d96000640 1 -- 172.21.15.32:0/2956145784 wait complete. 2024-08-03T21:00:43.420 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:43.420+0000 7f9d96000640 1 Processor -- start 2024-08-03T21:00:43.420 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:43.420+0000 7f9d96000640 1 -- start start 2024-08-03T21:00:43.423 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:43.421+0000 7f9d96000640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f9d9006e930 0x7f9d9015ca40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:00:43.423 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:43.421+0000 7f9d96000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9d90144f90 0x7f9d901f82a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:00:43.423 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:43.421+0000 7f9d96000640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f9d9015fff0 0x7f9d901f9070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:00:43.423 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:43.421+0000 7f9d96000640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f9d900cf840 con 0x7f9d90144f90 2024-08-03T21:00:43.423 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:43.421+0000 7f9d96000640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f9d90145f30 con 0x7f9d9006e930 2024-08-03T21:00:43.423 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:43.421+0000 7f9d96000640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7f9d900cf9c0 con 0x7f9d9015fff0 2024-08-03T21:00:43.424 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:43.421+0000 7f9d87e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9d90144f90 0x7f9d901f82a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:00:43.424 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:43.421+0000 7f9d87e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9d90144f90 0x7f9d901f82a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:56956/0 (socket says 172.21.15.32:56956) 2024-08-03T21:00:43.424 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:43.421+0000 7f9d87e00640 1 -- 172.21.15.32:0/4287095897 learned_addr learned my addr 172.21.15.32:0/4287095897 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T21:00:43.424 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:43.422+0000 7f9d8fe00640 1 --2- 172.21.15.32:0/4287095897 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f9d9015fff0 0x7f9d901f9070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:00:43.424 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:43.422+0000 7f9d8f400640 1 --2- 172.21.15.32:0/4287095897 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f9d9006e930 0x7f9d9015ca40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:00:43.424 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:43.422+0000 7f9d87e00640 1 -- 172.21.15.32:0/4287095897 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f9d9015fff0 msgr2=0x7f9d901f9070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:00:43.424 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:43.422+0000 7f9d87e00640 1 --2- 172.21.15.32:0/4287095897 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f9d9015fff0 0x7f9d901f9070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:43.424 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:43.422+0000 7f9d87e00640 1 -- 172.21.15.32:0/4287095897 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f9d9006e930 msgr2=0x7f9d9015ca40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:00:43.424 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:43.422+0000 7f9d87e00640 1 --2- 172.21.15.32:0/4287095897 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f9d9006e930 0x7f9d9015ca40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:43.424 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:43.422+0000 7f9d87e00640 1 -- 172.21.15.32:0/4287095897 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9d90135410 con 0x7f9d90144f90 2024-08-03T21:00:43.424 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:43.422+0000 7f9d8fe00640 1 --2- 172.21.15.32:0/4287095897 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f9d9015fff0 0x7f9d901f9070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-03T21:00:43.424 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:43.422+0000 7f9d8f400640 1 --2- 172.21.15.32:0/4287095897 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f9d9006e930 0x7f9d9015ca40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-03T21:00:43.425 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:43.422+0000 7f9d87e00640 1 --2- 172.21.15.32:0/4287095897 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9d90144f90 0x7f9d901f82a0 secure :-1 s=READY pgs=2128 cs=0 l=1 rev1=1 crypto rx=0x7f9d7800e7f0 tx=0x7f9d7800ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T21:00:43.425 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:43.422+0000 7f9d87400640 1 -- 172.21.15.32:0/4287095897 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d7800eea0 con 0x7f9d90144f90 2024-08-03T21:00:43.425 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:43.422+0000 7f9d87400640 1 -- 172.21.15.32:0/4287095897 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f9d7800ccc0 con 0x7f9d90144f90 2024-08-03T21:00:43.425 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:43.422+0000 7f9d87400640 1 -- 172.21.15.32:0/4287095897 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d78004280 con 0x7f9d90144f90 2024-08-03T21:00:43.425 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:43.422+0000 7f9d96000640 1 -- 172.21.15.32:0/4287095897 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9d901356f0 con 0x7f9d90144f90 2024-08-03T21:00:43.425 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:43.423+0000 7f9d96000640 1 -- 172.21.15.32:0/4287095897 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9d901f8b50 con 0x7f9d90144f90 2024-08-03T21:00:43.425 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:43.424+0000 7f9d96000640 1 -- 172.21.15.32:0/4287095897 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9d90135180 con 0x7f9d90144f90 2024-08-03T21:00:43.426 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:43.425+0000 7f9d87400640 1 -- 172.21.15.32:0/4287095897 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f9d780035f0 con 0x7f9d90144f90 2024-08-03T21:00:43.426 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:43.426+0000 7f9d87400640 1 --2- 172.21.15.32:0/4287095897 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f9d6c077010 0x7f9d6c079520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:00:43.426 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:43.426+0000 7f9d87400640 1 -- 172.21.15.32:0/4287095897 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(127..127 src has 1..127) ==== 8297+0+0 (secure 0 0 0) 0x7f9d78099cd0 con 0x7f9d90144f90 2024-08-03T21:00:43.426 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:43.426+0000 7f9d8f400640 1 --2- 172.21.15.32:0/4287095897 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f9d6c077010 0x7f9d6c079520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:00:43.427 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:43.427+0000 7f9d8f400640 1 --2- 172.21.15.32:0/4287095897 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f9d6c077010 0x7f9d6c079520 secure :-1 s=READY pgs=851 cs=0 l=1 rev1=1 crypto rx=0x7f9d800016a0 tx=0x7f9d80009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T21:00:43.430 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:43.430+0000 7f9d87400640 1 -- 172.21.15.32:0/4287095897 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7f9d78062400 con 0x7f9d90144f90 2024-08-03T21:00:43.634 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:43.633+0000 7f9d96000640 1 -- 172.21.15.32:0/4287095897 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "config dump"} v 0) -- 0x7f9d90135c10 con 0x7f9d90144f90 2024-08-03T21:00:43.634 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:43.634+0000 7f9d87400640 1 -- 172.21.15.32:0/4287095897 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "config dump"}]=0 v44) ==== 59+0+1998 (secure 0 0 0) 0x7f9d90135c10 con 0x7f9d90144f90 2024-08-03T21:00:43.635 INFO:teuthology.orchestra.run.smithi032.stdout:WHO MASK LEVEL OPTION VALUE RO 2024-08-03T21:00:43.635 INFO:teuthology.orchestra.run.smithi032.stdout:mon advanced mon_op_complaint_time 120 2024-08-03T21:00:43.635 INFO:teuthology.orchestra.run.smithi032.stdout:mon dev mon_warn_on_pool_no_app false 2024-08-03T21:00:43.635 INFO:teuthology.orchestra.run.smithi032.stdout:osd advanced osd_op_complaint_time 180.000000 2024-08-03T21:00:43.635 INFO:teuthology.orchestra.run.smithi032.stdout:osd.0 basic osd_mclock_max_capacity_iops_ssd 4313.565729 2024-08-03T21:00:43.635 INFO:teuthology.orchestra.run.smithi032.stdout:osd.1 basic osd_mclock_max_capacity_iops_ssd 4272.517979 2024-08-03T21:00:43.635 INFO:teuthology.orchestra.run.smithi032.stdout:osd.2 basic osd_mclock_max_capacity_iops_ssd 4131.201546 2024-08-03T21:00:43.635 INFO:teuthology.orchestra.run.smithi032.stdout:osd.3 basic osd_mclock_max_capacity_iops_ssd 4155.765914 2024-08-03T21:00:43.635 INFO:teuthology.orchestra.run.smithi032.stdout:osd.4 basic osd_mclock_max_capacity_iops_ssd 5744.667532 2024-08-03T21:00:43.635 INFO:teuthology.orchestra.run.smithi032.stdout:osd.5 basic osd_mclock_max_capacity_iops_ssd 3764.678226 2024-08-03T21:00:43.635 INFO:teuthology.orchestra.run.smithi032.stdout:osd.6 basic osd_mclock_max_capacity_iops_ssd 3927.989098 2024-08-03T21:00:43.635 INFO:teuthology.orchestra.run.smithi032.stdout:osd.7 basic osd_mclock_max_capacity_iops_ssd 3760.968870 2024-08-03T21:00:43.636 INFO:teuthology.orchestra.run.smithi032.stdout:mds advanced debug_mds 20/20 2024-08-03T21:00:43.636 INFO:teuthology.orchestra.run.smithi032.stdout:mds advanced debug_mds_balancer 20/20 2024-08-03T21:00:43.636 INFO:teuthology.orchestra.run.smithi032.stdout:mds advanced debug_ms 2/2 2024-08-03T21:00:43.636 INFO:teuthology.orchestra.run.smithi032.stdout:mds dev mds_debug_frag true 2024-08-03T21:00:43.636 INFO:teuthology.orchestra.run.smithi032.stdout:mds dev mds_debug_scatterstat true 2024-08-03T21:00:43.636 INFO:teuthology.orchestra.run.smithi032.stdout:mds advanced mds_op_complaint_time 180.000000 2024-08-03T21:00:43.636 INFO:teuthology.orchestra.run.smithi032.stdout:mds dev mds_verify_scatter true 2024-08-03T21:00:43.636 INFO:teuthology.orchestra.run.smithi032.stdout:mds advanced osd_op_complaint_time 180.000000 2024-08-03T21:00:43.636 INFO:teuthology.orchestra.run.smithi032.stdout:mds advanced rados_mon_op_timeout 900 2024-08-03T21:00:43.636 INFO:teuthology.orchestra.run.smithi032.stdout:mds advanced rados_osd_op_timeout 900 2024-08-03T21:00:43.636 INFO:teuthology.orchestra.run.smithi032.stdout:client advanced client_mount_timeout 600 2024-08-03T21:00:43.636 INFO:teuthology.orchestra.run.smithi032.stdout:client advanced debug_client 20/20 2024-08-03T21:00:43.636 INFO:teuthology.orchestra.run.smithi032.stdout:client advanced debug_ms 1/1 2024-08-03T21:00:43.637 INFO:teuthology.orchestra.run.smithi032.stdout:client advanced rados_mon_op_timeout 900 2024-08-03T21:00:43.637 INFO:teuthology.orchestra.run.smithi032.stdout:client advanced rados_osd_op_timeout 900 2024-08-03T21:00:43.638 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:43.636+0000 7f9d96000640 1 -- 172.21.15.32:0/4287095897 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f9d6c077010 msgr2=0x7f9d6c079520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:00:43.638 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:43.636+0000 7f9d96000640 1 --2- 172.21.15.32:0/4287095897 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f9d6c077010 0x7f9d6c079520 secure :-1 s=READY pgs=851 cs=0 l=1 rev1=1 crypto rx=0x7f9d800016a0 tx=0x7f9d80009040 comp rx=0 tx=0).stop 2024-08-03T21:00:43.638 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:43.636+0000 7f9d96000640 1 -- 172.21.15.32:0/4287095897 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9d90144f90 msgr2=0x7f9d901f82a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:00:43.638 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:43.636+0000 7f9d96000640 1 --2- 172.21.15.32:0/4287095897 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9d90144f90 0x7f9d901f82a0 secure :-1 s=READY pgs=2128 cs=0 l=1 rev1=1 crypto rx=0x7f9d7800e7f0 tx=0x7f9d7800ecb0 comp rx=0 tx=0).stop 2024-08-03T21:00:43.638 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:43.636+0000 7f9d96000640 1 -- 172.21.15.32:0/4287095897 shutdown_connections 2024-08-03T21:00:43.638 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:43.636+0000 7f9d96000640 1 --2- 172.21.15.32:0/4287095897 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f9d6c077010 0x7f9d6c079520 unknown :-1 s=CLOSED pgs=851 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:43.638 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:43.636+0000 7f9d96000640 1 --2- 172.21.15.32:0/4287095897 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f9d9015fff0 0x7f9d901f9070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:43.638 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:43.636+0000 7f9d96000640 1 --2- 172.21.15.32:0/4287095897 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9d90144f90 0x7f9d901f82a0 unknown :-1 s=CLOSED pgs=2128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:43.638 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:43.636+0000 7f9d96000640 1 --2- 172.21.15.32:0/4287095897 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f9d9006e930 0x7f9d9015ca40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:43.638 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:43.636+0000 7f9d96000640 1 -- 172.21.15.32:0/4287095897 >> 172.21.15.32:0/4287095897 conn(0x7f9d900b1bd0 msgr2=0x7f9d90142310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T21:00:43.638 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:43.636+0000 7f9d96000640 1 -- 172.21.15.32:0/4287095897 shutdown_connections 2024-08-03T21:00:43.639 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:43.636+0000 7f9d96000640 1 -- 172.21.15.32:0/4287095897 wait complete. 2024-08-03T21:00:43.659 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-08-03T21:00:43.891 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:43.890+0000 7f830f000640 1 -- 172.21.15.32:0/412927065 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f83080cecc0 msgr2=0x7f83080b4f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:00:43.891 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:43.890+0000 7f830f000640 1 --2- 172.21.15.32:0/412927065 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f83080cecc0 0x7f83080b4f10 secure :-1 s=READY pgs=2129 cs=0 l=1 rev1=1 crypto rx=0x7f82f8005fd0 tx=0x7f82f802a530 comp rx=0 tx=0).stop 2024-08-03T21:00:43.891 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:43.890+0000 7f830f000640 1 -- 172.21.15.32:0/412927065 shutdown_connections 2024-08-03T21:00:43.892 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:43.890+0000 7f830f000640 1 --2- 172.21.15.32:0/412927065 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f83081606f0 0x7f83081450f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:43.892 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:43.890+0000 7f830f000640 1 --2- 172.21.15.32:0/412927065 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f83080cecc0 0x7f83080b4f10 unknown :-1 s=CLOSED pgs=2129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:43.892 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:43.890+0000 7f830f000640 1 -- 172.21.15.32:0/412927065 >> 172.21.15.32:0/412927065 conn(0x7f83080b1bd0 msgr2=0x7f83080b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T21:00:43.892 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:43.890+0000 7f830f000640 1 -- 172.21.15.32:0/412927065 shutdown_connections 2024-08-03T21:00:43.892 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:43.890+0000 7f830f000640 1 -- 172.21.15.32:0/412927065 wait complete. 2024-08-03T21:00:43.893 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:43.893+0000 7f830f000640 1 Processor -- start 2024-08-03T21:00:43.893 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:43.893+0000 7f830f000640 1 -- start start 2024-08-03T21:00:43.896 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:43.894+0000 7f830f000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f83080cd210 0x7f8308073fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:00:43.896 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:43.894+0000 7f830f000640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f83080cecc0 0x7f83080727f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:00:43.896 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:43.894+0000 7f830f000640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8308073520 0x7f8308133ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:00:43.896 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:43.894+0000 7f830f000640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f8308145ad0 con 0x7f83080cd210 2024-08-03T21:00:43.896 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:43.894+0000 7f830f000640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f8308145950 con 0x7f8308073520 2024-08-03T21:00:43.896 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:43.894+0000 7f830f000640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7f8308145c50 con 0x7f83080cecc0 2024-08-03T21:00:43.896 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:43.895+0000 7f8307e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f83080cd210 0x7f8308073fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:00:43.896 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:43.895+0000 7f8307e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f83080cd210 0x7f8308073fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:56982/0 (socket says 172.21.15.32:56982) 2024-08-03T21:00:43.896 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:43.895+0000 7f8307e00640 1 -- 172.21.15.32:0/1558124946 learned_addr learned my addr 172.21.15.32:0/1558124946 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T21:00:43.896 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:43.895+0000 7f82ff400640 1 --2- 172.21.15.32:0/1558124946 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f83080cecc0 0x7f83080727f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:00:43.896 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:43.895+0000 7f830d000640 1 --2- 172.21.15.32:0/1558124946 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8308073520 0x7f8308133ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:00:43.896 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:43.895+0000 7f8307e00640 1 -- 172.21.15.32:0/1558124946 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f83080cecc0 msgr2=0x7f83080727f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:00:43.897 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:43.895+0000 7f8307e00640 1 --2- 172.21.15.32:0/1558124946 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f83080cecc0 0x7f83080727f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:43.897 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:43.895+0000 7f8307e00640 1 -- 172.21.15.32:0/1558124946 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8308073520 msgr2=0x7f8308133ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:00:43.897 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:43.895+0000 7f8307e00640 1 --2- 172.21.15.32:0/1558124946 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8308073520 0x7f8308133ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:43.897 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:43.895+0000 7f8307e00640 1 -- 172.21.15.32:0/1558124946 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8308156910 con 0x7f83080cd210 2024-08-03T21:00:43.897 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:43.895+0000 7f8307e00640 1 --2- 172.21.15.32:0/1558124946 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f83080cd210 0x7f8308073fd0 secure :-1 s=READY pgs=2130 cs=0 l=1 rev1=1 crypto rx=0x7f82f8006010 tx=0x7f82f80038a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T21:00:43.898 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:43.896+0000 7f8304c00640 1 -- 172.21.15.32:0/1558124946 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82f803e030 con 0x7f83080cd210 2024-08-03T21:00:43.898 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:43.896+0000 7f8304c00640 1 -- 172.21.15.32:0/1558124946 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f82f8003d90 con 0x7f83080cd210 2024-08-03T21:00:43.898 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:43.896+0000 7f8304c00640 1 -- 172.21.15.32:0/1558124946 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82f8035040 con 0x7f83080cd210 2024-08-03T21:00:43.898 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:43.896+0000 7f830f000640 1 -- 172.21.15.32:0/1558124946 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f83080c96e0 con 0x7f83080cd210 2024-08-03T21:00:43.898 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:43.896+0000 7f830f000640 1 -- 172.21.15.32:0/1558124946 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f83080cbae0 con 0x7f83080cd210 2024-08-03T21:00:43.898 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:43.897+0000 7f830f000640 1 -- 172.21.15.32:0/1558124946 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f83080b4f10 con 0x7f83080cd210 2024-08-03T21:00:43.898 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:43.898+0000 7f8304c00640 1 -- 172.21.15.32:0/1558124946 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f82f8041030 con 0x7f83080cd210 2024-08-03T21:00:43.899 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:43.899+0000 7f8304c00640 1 --2- 172.21.15.32:0/1558124946 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f82e4077040 0x7f82e4079550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:00:43.899 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:43.899+0000 7f8304c00640 1 -- 172.21.15.32:0/1558124946 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(127..127 src has 1..127) ==== 8297+0+0 (secure 0 0 0) 0x7f82f80b9b20 con 0x7f83080cd210 2024-08-03T21:00:43.900 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:43.900+0000 7f82ff400640 1 --2- 172.21.15.32:0/1558124946 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f82e4077040 0x7f82e4079550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:00:43.900 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:43.900+0000 7f82ff400640 1 --2- 172.21.15.32:0/1558124946 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f82e4077040 0x7f82e4079550 secure :-1 s=READY pgs=852 cs=0 l=1 rev1=1 crypto rx=0x7f82f0006fd0 tx=0x7f82f0008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T21:00:43.904 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:43.903+0000 7f8304c00640 1 -- 172.21.15.32:0/1558124946 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7f82f80822e0 con 0x7f83080cd210 2024-08-03T21:00:44.114 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:44.112+0000 7f830f000640 1 -- 172.21.15.32:0/1558124946 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) -- 0x7f830806fb20 con 0x7f83080cd210 2024-08-03T21:00:44.118 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:44.117+0000 7f8304c00640 1 -- 172.21.15.32:0/1558124946 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v45)=0 v45) ==== 122+0+0 (secure 0 0 0) 0x7f82f8081be0 con 0x7f83080cd210 2024-08-03T21:00:44.121 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:44.119+0000 7f830f000640 1 -- 172.21.15.32:0/1558124946 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f82e4077040 msgr2=0x7f82e4079550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:00:44.121 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:44.119+0000 7f830f000640 1 --2- 172.21.15.32:0/1558124946 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f82e4077040 0x7f82e4079550 secure :-1 s=READY pgs=852 cs=0 l=1 rev1=1 crypto rx=0x7f82f0006fd0 tx=0x7f82f0008040 comp rx=0 tx=0).stop 2024-08-03T21:00:44.121 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:44.119+0000 7f830f000640 1 -- 172.21.15.32:0/1558124946 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f83080cd210 msgr2=0x7f8308073fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:00:44.121 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:44.119+0000 7f830f000640 1 --2- 172.21.15.32:0/1558124946 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f83080cd210 0x7f8308073fd0 secure :-1 s=READY pgs=2130 cs=0 l=1 rev1=1 crypto rx=0x7f82f8006010 tx=0x7f82f80038a0 comp rx=0 tx=0).stop 2024-08-03T21:00:44.121 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:44.119+0000 7f830f000640 1 -- 172.21.15.32:0/1558124946 shutdown_connections 2024-08-03T21:00:44.121 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:44.119+0000 7f830f000640 1 --2- 172.21.15.32:0/1558124946 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f82e4077040 0x7f82e4079550 unknown :-1 s=CLOSED pgs=852 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:44.122 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:44.119+0000 7f830f000640 1 --2- 172.21.15.32:0/1558124946 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8308073520 0x7f8308133ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:44.122 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:44.119+0000 7f830f000640 1 --2- 172.21.15.32:0/1558124946 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f83080cecc0 0x7f83080727f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:44.122 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:44.119+0000 7f830f000640 1 --2- 172.21.15.32:0/1558124946 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f83080cd210 0x7f8308073fd0 unknown :-1 s=CLOSED pgs=2130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:44.122 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:44.119+0000 7f830f000640 1 -- 172.21.15.32:0/1558124946 >> 172.21.15.32:0/1558124946 conn(0x7f83080b1bd0 msgr2=0x7f8308144b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T21:00:44.122 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:44.119+0000 7f830f000640 1 -- 172.21.15.32:0/1558124946 shutdown_connections 2024-08-03T21:00:44.122 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:44.120+0000 7f830f000640 1 -- 172.21.15.32:0/1558124946 wait complete. 2024-08-03T21:00:44.144 INFO:teuthology.nuke:Clearing teuthology firewall rules... 2024-08-03T21:00:44.144 DEBUG:teuthology.orchestra.run.smithi032:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-08-03T21:00:44.248 DEBUG:teuthology.orchestra.run.smithi118:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-08-03T21:00:44.338 INFO:teuthology.nuke:Cleared teuthology firewall rules. 2024-08-03T21:00:44.338 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-08-03T21:00:44.339 DEBUG:teuthology.orchestra.run.smithi032:> dd if=/proc/self/mounts of=/dev/stdout 2024-08-03T21:00:44.374 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053453,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6499208k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 none /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249604k,nr_inodes=812401,mode=700,uid=1000,gid=1316 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6499208k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-08-03T21:00:44.374 DEBUG:teuthology.orchestra.run.smithi118:> set -ex 2024-08-03T21:00:44.374 DEBUG:teuthology.orchestra.run.smithi118:> dd if=/proc/self/mounts of=/dev/stdout 2024-08-03T21:00:44.411 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052663,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497944k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248972k,nr_inodes=812243,mode=700,uid=1000,gid=1316 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497944k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-08-03T21:00:44.411 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-08-03T21:00:44.411 DEBUG:teuthology.orchestra.run.smithi032:> dd if=/proc/self/mounts of=/dev/stdout 2024-08-03T21:00:44.450 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053453,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6499208k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 none /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249604k,nr_inodes=812401,mode=700,uid=1000,gid=1316 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6499208k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-08-03T21:00:44.450 DEBUG:teuthology.orchestra.run.smithi118:> set -ex 2024-08-03T21:00:44.450 DEBUG:teuthology.orchestra.run.smithi118:> dd if=/proc/self/mounts of=/dev/stdout 2024-08-03T21:00:44.488 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052663,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497944k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248972k,nr_inodes=812243,mode=700,uid=1000,gid=1316 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497944k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-08-03T21:00:44.488 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-08-03T21:00:44.489 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-08-03T21:00:44.718 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:44.716+0000 7f3717200640 1 -- 172.21.15.32:0/1839696338 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f37180d1c60 msgr2=0x7f37180cf340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:00:44.719 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:44.716+0000 7f3717200640 1 --2- 172.21.15.32:0/1839696338 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f37180d1c60 0x7f37180cf340 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f3710009be0 tx=0x7f371002a890 comp rx=0 tx=0).stop 2024-08-03T21:00:44.719 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:44.716+0000 7f3717200640 1 -- 172.21.15.32:0/1839696338 shutdown_connections 2024-08-03T21:00:44.719 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:44.716+0000 7f3717200640 1 --2- 172.21.15.32:0/1839696338 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f37180d1c60 0x7f37180cf340 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:44.719 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:44.716+0000 7f3717200640 1 -- 172.21.15.32:0/1839696338 >> 172.21.15.32:0/1839696338 conn(0x7f37180d4090 msgr2=0x7f37180d44a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T21:00:44.719 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:44.717+0000 7f3717200640 1 -- 172.21.15.32:0/1839696338 shutdown_connections 2024-08-03T21:00:44.719 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:44.717+0000 7f3717200640 1 -- 172.21.15.32:0/1839696338 wait complete. 2024-08-03T21:00:44.720 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:44.719+0000 7f3717200640 1 Processor -- start 2024-08-03T21:00:44.720 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:44.719+0000 7f3717200640 1 -- start start 2024-08-03T21:00:44.723 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:44.720+0000 7f3717200640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f37180d1c60 0x7f3718166b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:00:44.723 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:44.720+0000 7f3717200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f37180d4bb0 0x7f37180b80f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:00:44.723 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:44.720+0000 7f3717200640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f37180b8630 0x7f37181611d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:00:44.723 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:44.720+0000 7f3717200640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f37180cfd50 con 0x7f37180d4bb0 2024-08-03T21:00:44.723 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:44.720+0000 7f3717200640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f37180cfbd0 con 0x7f37180d1c60 2024-08-03T21:00:44.723 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:44.720+0000 7f3717200640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7f37180cfed0 con 0x7f37180b8630 2024-08-03T21:00:44.723 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:44.721+0000 7f370d400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f37180d4bb0 0x7f37180b80f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:00:44.723 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:44.721+0000 7f370d400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f37180d4bb0 0x7f37180b80f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:56996/0 (socket says 172.21.15.32:56996) 2024-08-03T21:00:44.723 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:44.721+0000 7f370d400640 1 -- 172.21.15.32:0/1554837354 learned_addr learned my addr 172.21.15.32:0/1554837354 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T21:00:44.723 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:44.721+0000 7f3715e00640 1 --2- 172.21.15.32:0/1554837354 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f37180d1c60 0x7f3718166b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:00:44.724 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:44.721+0000 7f3716800640 1 --2- 172.21.15.32:0/1554837354 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f37180b8630 0x7f37181611d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:00:44.724 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:44.721+0000 7f370d400640 1 -- 172.21.15.32:0/1554837354 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f37180b8630 msgr2=0x7f37181611d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:00:44.724 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:44.721+0000 7f370d400640 1 --2- 172.21.15.32:0/1554837354 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f37180b8630 0x7f37181611d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:44.724 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:44.721+0000 7f370d400640 1 -- 172.21.15.32:0/1554837354 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f37180d1c60 msgr2=0x7f3718166b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:00:44.724 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:44.721+0000 7f370d400640 1 --2- 172.21.15.32:0/1554837354 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f37180d1c60 0x7f3718166b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:44.724 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:44.721+0000 7f370d400640 1 -- 172.21.15.32:0/1554837354 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f37181698c0 con 0x7f37180d4bb0 2024-08-03T21:00:44.725 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:44.722+0000 7f370d400640 1 --2- 172.21.15.32:0/1554837354 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f37180d4bb0 0x7f37180b80f0 secure :-1 s=READY pgs=2132 cs=0 l=1 rev1=1 crypto rx=0x7f370000c430 tx=0x7f370000c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T21:00:44.725 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:44.722+0000 7f370ea00640 1 -- 172.21.15.32:0/1554837354 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3700017070 con 0x7f37180d4bb0 2024-08-03T21:00:44.725 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:44.722+0000 7f370ea00640 1 -- 172.21.15.32:0/1554837354 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f37000053e0 con 0x7f37180d4bb0 2024-08-03T21:00:44.725 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:44.722+0000 7f370ea00640 1 -- 172.21.15.32:0/1554837354 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3700003850 con 0x7f37180d4bb0 2024-08-03T21:00:44.725 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:44.722+0000 7f3717200640 1 -- 172.21.15.32:0/1554837354 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f37181673a0 con 0x7f37180d4bb0 2024-08-03T21:00:44.725 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:44.722+0000 7f3717200640 1 -- 172.21.15.32:0/1554837354 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f37181690c0 con 0x7f37180d4bb0 2024-08-03T21:00:44.725 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:44.723+0000 7f3717200640 1 -- 172.21.15.32:0/1554837354 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3718050ba0 con 0x7f37180d4bb0 2024-08-03T21:00:44.726 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:44.725+0000 7f370ea00640 1 -- 172.21.15.32:0/1554837354 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f370001a070 con 0x7f37180d4bb0 2024-08-03T21:00:44.727 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:44.726+0000 7f370ea00640 1 --2- 172.21.15.32:0/1554837354 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f36f4077040 0x7f36f4079550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:00:44.727 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:44.726+0000 7f370ea00640 1 -- 172.21.15.32:0/1554837354 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(127..127 src has 1..127) ==== 8297+0+0 (secure 0 0 0) 0x7f3700098750 con 0x7f37180d4bb0 2024-08-03T21:00:44.727 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:44.726+0000 7f3715e00640 1 --2- 172.21.15.32:0/1554837354 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f36f4077040 0x7f36f4079550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:00:44.728 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:44.727+0000 7f3715e00640 1 --2- 172.21.15.32:0/1554837354 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f36f4077040 0x7f36f4079550 secure :-1 s=READY pgs=853 cs=0 l=1 rev1=1 crypto rx=0x7f3708007fd0 tx=0x7f3708007eb0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T21:00:44.731 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:44.730+0000 7f370ea00640 1 -- 172.21.15.32:0/1554837354 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7f3718050ba0 con 0x7f37180d4bb0 2024-08-03T21:00:44.977 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:44.975+0000 7f3717200640 1 -- 172.21.15.32:0/1554837354 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f37180d2290 con 0x7f37180d4bb0 2024-08-03T21:00:44.978 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:44.976+0000 7f370ea00640 1 -- 172.21.15.32:0/1554837354 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 333 v333) ==== 95+0+3917 (secure 0 0 0) 0x7f3700069150 con 0x7f37180d4bb0 2024-08-03T21:00:44.978 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-03T21:00:44.978 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":333,"btime":"2024-08-03T21:00:37:129156+0000","default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6439,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6835/531080160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6834","nonce":531080160},{"type":"v1","addr":"172.21.15.118:6835","nonce":531080160}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":84},{"gid":6454,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6837/2796882319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6836","nonce":2796882319},{"type":"v1","addr":"172.21.15.118:6837","nonce":2796882319}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":86},{"gid":6813,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6834/2686933890","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":2686933890},{"type":"v1","addr":"172.21.15.32:6834","nonce":2686933890}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":85},{"gid":10359,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6835/3212405498","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6833","nonce":3212405498},{"type":"v1","addr":"172.21.15.32:6835","nonce":3212405498}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":331}],"filesystems":[]} 2024-08-03T21:00:44.978 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 333 2024-08-03T21:00:44.980 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:44.978+0000 7f3717200640 1 -- 172.21.15.32:0/1554837354 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f36f4077040 msgr2=0x7f36f4079550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:00:44.980 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:44.978+0000 7f3717200640 1 --2- 172.21.15.32:0/1554837354 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f36f4077040 0x7f36f4079550 secure :-1 s=READY pgs=853 cs=0 l=1 rev1=1 crypto rx=0x7f3708007fd0 tx=0x7f3708007eb0 comp rx=0 tx=0).stop 2024-08-03T21:00:44.980 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:44.978+0000 7f3717200640 1 -- 172.21.15.32:0/1554837354 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f37180d4bb0 msgr2=0x7f37180b80f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:00:44.980 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:44.978+0000 7f3717200640 1 --2- 172.21.15.32:0/1554837354 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f37180d4bb0 0x7f37180b80f0 secure :-1 s=READY pgs=2132 cs=0 l=1 rev1=1 crypto rx=0x7f370000c430 tx=0x7f370000c8f0 comp rx=0 tx=0).stop 2024-08-03T21:00:44.981 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:44.978+0000 7f3717200640 1 -- 172.21.15.32:0/1554837354 shutdown_connections 2024-08-03T21:00:44.981 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:44.978+0000 7f3717200640 1 --2- 172.21.15.32:0/1554837354 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f36f4077040 0x7f36f4079550 unknown :-1 s=CLOSED pgs=853 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:44.981 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:44.978+0000 7f3717200640 1 --2- 172.21.15.32:0/1554837354 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f37180b8630 0x7f37181611d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:44.981 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:44.978+0000 7f3717200640 1 --2- 172.21.15.32:0/1554837354 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f37180d4bb0 0x7f37180b80f0 unknown :-1 s=CLOSED pgs=2132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:44.981 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:44.978+0000 7f3717200640 1 --2- 172.21.15.32:0/1554837354 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f37180d1c60 0x7f3718166b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:44.981 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:44.978+0000 7f3717200640 1 -- 172.21.15.32:0/1554837354 >> 172.21.15.32:0/1554837354 conn(0x7f37180d4090 msgr2=0x7f371806ee90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T21:00:44.981 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:44.978+0000 7f3717200640 1 -- 172.21.15.32:0/1554837354 shutdown_connections 2024-08-03T21:00:44.981 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:44.978+0000 7f3717200640 1 -- 172.21.15.32:0/1554837354 wait complete. 2024-08-03T21:00:45.006 INFO:tasks.ceph.mds.a:Restarting daemon 2024-08-03T21:00:45.006 INFO:tasks.ceph.mds.a:Stopping old one... 2024-08-03T21:00:45.006 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-08-03T21:00:45.006 INFO:teuthology.orchestra.run:waiting for 300 2024-08-03T21:00:45.007 INFO:tasks.ceph.mds.c:Restarting daemon 2024-08-03T21:00:45.007 INFO:tasks.ceph.mds.c:Stopping old one... 2024-08-03T21:00:45.007 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-08-03T21:00:45.007 INFO:teuthology.orchestra.run:waiting for 300 2024-08-03T21:00:45.007 INFO:tasks.ceph.mds.b:Restarting daemon 2024-08-03T21:00:45.007 INFO:tasks.ceph.mds.b:Stopping old one... 2024-08-03T21:00:45.007 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-08-03T21:00:45.007 INFO:teuthology.orchestra.run:waiting for 300 2024-08-03T21:00:45.008 INFO:tasks.ceph.mds.d:Restarting daemon 2024-08-03T21:00:45.008 INFO:tasks.ceph.mds.d:Stopping old one... 2024-08-03T21:00:45.008 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-08-03T21:00:45.008 INFO:teuthology.orchestra.run:waiting for 300 2024-08-03T21:00:45.029 INFO:tasks.ceph.mds.a:Stopped 2024-08-03T21:00:45.029 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-08-03T21:00:45.031 INFO:tasks.ceph.mds.b:Stopped 2024-08-03T21:00:45.031 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-08-03T21:00:45.032 INFO:tasks.ceph.mds.d:Stopped 2024-08-03T21:00:45.032 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-08-03T21:00:45.033 INFO:tasks.ceph.mds.c:Stopped 2024-08-03T21:00:45.033 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-08-03T21:00:45.076 INFO:tasks.ceph.mds.b:Started 2024-08-03T21:00:45.076 DEBUG:teuthology.parallel:result is None 2024-08-03T21:00:45.077 INFO:tasks.ceph.mds.a:Started 2024-08-03T21:00:45.078 DEBUG:teuthology.parallel:result is None 2024-08-03T21:00:45.079 INFO:tasks.ceph.mds.d:Started 2024-08-03T21:00:45.079 DEBUG:teuthology.parallel:result is None 2024-08-03T21:00:45.081 INFO:tasks.ceph.mds.c:Started 2024-08-03T21:00:45.081 DEBUG:teuthology.parallel:result is None 2024-08-03T21:00:45.082 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-08-03T21:00:45.275 INFO:tasks.ceph.mds.d.smithi118.stdout:starting mds.d at 2024-08-03T21:00:45.276 INFO:tasks.ceph.mds.b.smithi118.stdout:starting mds.b at 2024-08-03T21:00:45.278 INFO:tasks.ceph.mds.a.smithi032.stdout:starting mds.a at 2024-08-03T21:00:45.278 INFO:tasks.ceph.mds.c.smithi032.stdout:starting mds.c at 2024-08-03T21:00:45.324 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:45.322+0000 7efe92800640 1 -- 172.21.15.32:0/2846191939 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efe8c15fff0 msgr2=0x7efe8c160400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:00:45.324 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:45.322+0000 7efe92800640 1 --2- 172.21.15.32:0/2846191939 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efe8c15fff0 0x7efe8c160400 secure :-1 s=READY pgs=2138 cs=0 l=1 rev1=1 crypto rx=0x7efe8c071700 tx=0x7efe8402a5f0 comp rx=0 tx=0).stop 2024-08-03T21:00:45.324 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:45.322+0000 7efe92800640 1 -- 172.21.15.32:0/2846191939 shutdown_connections 2024-08-03T21:00:45.325 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:45.322+0000 7efe92800640 1 --2- 172.21.15.32:0/2846191939 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efe8c15fff0 0x7efe8c160400 unknown :-1 s=CLOSED pgs=2138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:45.325 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:45.322+0000 7efe92800640 1 --2- 172.21.15.32:0/2846191939 >> v2:172.21.15.118:3300/0 conn(0x7efe8c06e930 0x7efe8c144f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:45.325 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:45.322+0000 7efe92800640 1 -- 172.21.15.32:0/2846191939 >> 172.21.15.32:0/2846191939 conn(0x7efe8c0b1bd0 msgr2=0x7efe8c0b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T21:00:45.325 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:45.322+0000 7efe92800640 1 -- 172.21.15.32:0/2846191939 shutdown_connections 2024-08-03T21:00:45.325 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:45.322+0000 7efe92800640 1 -- 172.21.15.32:0/2846191939 wait complete. 2024-08-03T21:00:45.326 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:45.325+0000 7efe92800640 1 Processor -- start 2024-08-03T21:00:45.326 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:45.325+0000 7efe92800640 1 -- start start 2024-08-03T21:00:45.328 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:45.326+0000 7efe92800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efe8c06e930 0x7efe8c0b2fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:00:45.328 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:45.326+0000 7efe92800640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7efe8c073e60 0x7efe8c15b8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:00:45.329 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:45.326+0000 7efe92800640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7efe8c15fff0 0x7efe8c0b0900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:00:45.329 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:45.326+0000 7efe92800640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7efe8c0cf840 con 0x7efe8c06e930 2024-08-03T21:00:45.329 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:45.326+0000 7efe92800640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7efe8c145f30 con 0x7efe8c15fff0 2024-08-03T21:00:45.329 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:45.326+0000 7efe92800640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7efe8c0cf9c0 con 0x7efe8c073e60 2024-08-03T21:00:45.329 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:45.326+0000 7efe8b400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efe8c06e930 0x7efe8c0b2fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:00:45.329 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:45.326+0000 7efe8b400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efe8c06e930 0x7efe8c0b2fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:57036/0 (socket says 172.21.15.32:57036) 2024-08-03T21:00:45.329 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:45.326+0000 7efe8b400640 1 -- 172.21.15.32:0/1782809552 learned_addr learned my addr 172.21.15.32:0/1782809552 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T21:00:45.329 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:45.326+0000 7efe82a00640 1 --2- 172.21.15.32:0/1782809552 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7efe8c073e60 0x7efe8c15b8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:00:45.329 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:45.327+0000 7efe8be00640 1 --2- 172.21.15.32:0/1782809552 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7efe8c15fff0 0x7efe8c0b0900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:00:45.329 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:45.327+0000 7efe8b400640 1 -- 172.21.15.32:0/1782809552 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7efe8c073e60 msgr2=0x7efe8c15b8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:00:45.329 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:45.327+0000 7efe8b400640 1 --2- 172.21.15.32:0/1782809552 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7efe8c073e60 0x7efe8c15b8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:45.329 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:45.327+0000 7efe8b400640 1 -- 172.21.15.32:0/1782809552 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7efe8c15fff0 msgr2=0x7efe8c0b0900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:00:45.329 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:45.327+0000 7efe8b400640 1 --2- 172.21.15.32:0/1782809552 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7efe8c15fff0 0x7efe8c0b0900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:45.329 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:45.327+0000 7efe8b400640 1 -- 172.21.15.32:0/1782809552 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efe8c15bdf0 con 0x7efe8c06e930 2024-08-03T21:00:45.330 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:45.327+0000 7efe8b400640 1 --2- 172.21.15.32:0/1782809552 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efe8c06e930 0x7efe8c0b2fd0 secure :-1 s=READY pgs=2139 cs=0 l=1 rev1=1 crypto rx=0x7efe7c00eb60 tx=0x7efe7c00c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T21:00:45.330 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:45.327+0000 7efe83e00640 1 -- 172.21.15.32:0/1782809552 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe7c010630 con 0x7efe8c06e930 2024-08-03T21:00:45.331 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:45.327+0000 7efe83e00640 1 -- 172.21.15.32:0/1782809552 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7efe7c0108c0 con 0x7efe8c06e930 2024-08-03T21:00:45.331 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:45.328+0000 7efe92800640 1 -- 172.21.15.32:0/1782809552 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efe8c1f9050 con 0x7efe8c06e930 2024-08-03T21:00:45.331 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:45.328+0000 7efe83e00640 1 -- 172.21.15.32:0/1782809552 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe7c010d10 con 0x7efe8c06e930 2024-08-03T21:00:45.331 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:45.328+0000 7efe92800640 1 -- 172.21.15.32:0/1782809552 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efe8c1f9610 con 0x7efe8c06e930 2024-08-03T21:00:45.331 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:45.329+0000 7efe92800640 1 -- 172.21.15.32:0/1782809552 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efe8c06a430 con 0x7efe8c06e930 2024-08-03T21:00:45.331 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:45.330+0000 7efe83e00640 1 -- 172.21.15.32:0/1782809552 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7efe7c018420 con 0x7efe8c06e930 2024-08-03T21:00:45.332 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:45.331+0000 7efe83e00640 1 --2- 172.21.15.32:0/1782809552 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7efe68077010 0x7efe68079520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:00:45.332 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:45.331+0000 7efe83e00640 1 -- 172.21.15.32:0/1782809552 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(127..127 src has 1..127) ==== 8297+0+0 (secure 0 0 0) 0x7efe7c099fd0 con 0x7efe8c06e930 2024-08-03T21:00:45.333 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:45.331+0000 7efe82a00640 1 --2- 172.21.15.32:0/1782809552 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7efe68077010 0x7efe68079520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:00:45.333 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:45.332+0000 7efe82a00640 1 --2- 172.21.15.32:0/1782809552 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7efe68077010 0x7efe68079520 secure :-1 s=READY pgs=854 cs=0 l=1 rev1=1 crypto rx=0x7efe8c1441c0 tx=0x7efe74009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T21:00:45.337 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:45.335+0000 7efe83e00640 1 -- 172.21.15.32:0/1782809552 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7efe8c06a430 con 0x7efe8c06e930 2024-08-03T21:00:45.554 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:45.553+0000 7efe92800640 1 -- 172.21.15.32:0/1782809552 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) -- 0x7efe8c160c30 con 0x7efe8c06e930 2024-08-03T21:00:46.181 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:46.179+0000 7efe83e00640 1 -- 172.21.15.32:0/1782809552 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v129) ==== 97+0+0 (secure 0 0 0) 0x7efe7c00ee60 con 0x7efe8c06e930 2024-08-03T21:00:46.181 INFO:teuthology.orchestra.run.smithi032.stderr: removed all blocklist entries 2024-08-03T21:00:46.184 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:46.182+0000 7efe92800640 1 -- 172.21.15.32:0/1782809552 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7efe68077010 msgr2=0x7efe68079520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:00:46.184 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:46.182+0000 7efe92800640 1 --2- 172.21.15.32:0/1782809552 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7efe68077010 0x7efe68079520 secure :-1 s=READY pgs=854 cs=0 l=1 rev1=1 crypto rx=0x7efe8c1441c0 tx=0x7efe74009040 comp rx=0 tx=0).stop 2024-08-03T21:00:46.184 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:46.182+0000 7efe92800640 1 -- 172.21.15.32:0/1782809552 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efe8c06e930 msgr2=0x7efe8c0b2fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:00:46.184 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:46.182+0000 7efe92800640 1 --2- 172.21.15.32:0/1782809552 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efe8c06e930 0x7efe8c0b2fd0 secure :-1 s=READY pgs=2139 cs=0 l=1 rev1=1 crypto rx=0x7efe7c00eb60 tx=0x7efe7c00c6a0 comp rx=0 tx=0).stop 2024-08-03T21:00:46.185 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:46.182+0000 7efe92800640 1 -- 172.21.15.32:0/1782809552 shutdown_connections 2024-08-03T21:00:46.185 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:46.182+0000 7efe92800640 1 --2- 172.21.15.32:0/1782809552 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7efe68077010 0x7efe68079520 unknown :-1 s=CLOSED pgs=854 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:46.185 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:46.182+0000 7efe92800640 1 --2- 172.21.15.32:0/1782809552 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7efe8c15fff0 0x7efe8c0b0900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:46.185 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:46.182+0000 7efe92800640 1 --2- 172.21.15.32:0/1782809552 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7efe8c073e60 0x7efe8c15b8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:46.185 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:46.182+0000 7efe92800640 1 --2- 172.21.15.32:0/1782809552 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efe8c06e930 0x7efe8c0b2fd0 unknown :-1 s=CLOSED pgs=2139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:46.185 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:46.182+0000 7efe92800640 1 -- 172.21.15.32:0/1782809552 >> 172.21.15.32:0/1782809552 conn(0x7efe8c0b1bd0 msgr2=0x7efe8c15ca40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T21:00:46.185 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:46.183+0000 7efe92800640 1 -- 172.21.15.32:0/1782809552 shutdown_connections 2024-08-03T21:00:46.185 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:46.183+0000 7efe92800640 1 -- 172.21.15.32:0/1782809552 wait complete. 2024-08-03T21:00:46.210 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-08-03T21:00:46.440 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:46.438+0000 7fb8ee200640 1 -- 172.21.15.32:0/2251788596 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb8e80d4bb0 msgr2=0x7fb8e80d1c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:00:46.440 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:46.438+0000 7fb8ee200640 1 --2- 172.21.15.32:0/2251788596 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb8e80d4bb0 0x7fb8e80d1c60 secure :-1 s=READY pgs=2141 cs=0 l=1 rev1=1 crypto rx=0x7fb8dc005fd0 tx=0x7fb8dc02a540 comp rx=0 tx=0).stop 2024-08-03T21:00:46.440 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:46.438+0000 7fb8ee200640 1 -- 172.21.15.32:0/2251788596 shutdown_connections 2024-08-03T21:00:46.440 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:46.438+0000 7fb8ee200640 1 --2- 172.21.15.32:0/2251788596 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fb8e80cec30 0x7fb8e80cf040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:46.440 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:46.438+0000 7fb8ee200640 1 --2- 172.21.15.32:0/2251788596 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb8e80d4bb0 0x7fb8e80d1c60 unknown :-1 s=CLOSED pgs=2141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:46.440 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:46.438+0000 7fb8ee200640 1 -- 172.21.15.32:0/2251788596 >> 172.21.15.32:0/2251788596 conn(0x7fb8e80d4090 msgr2=0x7fb8e80d44a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T21:00:46.440 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:46.438+0000 7fb8ee200640 1 -- 172.21.15.32:0/2251788596 shutdown_connections 2024-08-03T21:00:46.440 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:46.438+0000 7fb8ee200640 1 -- 172.21.15.32:0/2251788596 wait complete. 2024-08-03T21:00:46.442 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:46.440+0000 7fb8ee200640 1 Processor -- start 2024-08-03T21:00:46.442 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:46.440+0000 7fb8ee200640 1 -- start start 2024-08-03T21:00:46.444 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:46.442+0000 7fb8ee200640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb8e80cec30 0x7fb8e815c570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:00:46.444 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:46.442+0000 7fb8ee200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb8e80d2290 0x7fb8e80cb990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:00:46.444 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:46.442+0000 7fb8ee200640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fb8e80d4bb0 0x7fb8e80cbed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:00:46.444 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:46.442+0000 7fb8ee200640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fb8e80cfad0 con 0x7fb8e80d2290 2024-08-03T21:00:46.444 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:46.442+0000 7fb8ee200640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7fb8e80cf950 con 0x7fb8e80cec30 2024-08-03T21:00:46.444 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:46.442+0000 7fb8ee200640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7fb8e80cfc50 con 0x7fb8e80d4bb0 2024-08-03T21:00:46.444 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:46.442+0000 7fb8e7e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb8e80d2290 0x7fb8e80cb990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:00:46.444 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:46.442+0000 7fb8e7e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb8e80d2290 0x7fb8e80cb990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:57060/0 (socket says 172.21.15.32:57060) 2024-08-03T21:00:46.444 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:46.442+0000 7fb8e7e00640 1 -- 172.21.15.32:0/181399801 learned_addr learned my addr 172.21.15.32:0/181399801 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T21:00:46.446 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:46.443+0000 7fb8ed800640 1 --2- 172.21.15.32:0/181399801 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fb8e80d4bb0 0x7fb8e80cbed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:00:46.446 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:46.443+0000 7fb8ece00640 1 --2- 172.21.15.32:0/181399801 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb8e80cec30 0x7fb8e815c570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:00:46.446 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:46.443+0000 7fb8e7e00640 1 -- 172.21.15.32:0/181399801 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fb8e80d4bb0 msgr2=0x7fb8e80cbed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:00:46.446 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:46.443+0000 7fb8e7e00640 1 --2- 172.21.15.32:0/181399801 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fb8e80d4bb0 0x7fb8e80cbed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:46.446 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:46.443+0000 7fb8e7e00640 1 -- 172.21.15.32:0/181399801 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb8e80cec30 msgr2=0x7fb8e815c570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:00:46.446 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:46.443+0000 7fb8e7e00640 1 --2- 172.21.15.32:0/181399801 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb8e80cec30 0x7fb8e815c570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:46.446 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:46.443+0000 7fb8e7e00640 1 -- 172.21.15.32:0/181399801 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb8e80cdf60 con 0x7fb8e80d2290 2024-08-03T21:00:46.446 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:46.443+0000 7fb8ed800640 1 --2- 172.21.15.32:0/181399801 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fb8e80d4bb0 0x7fb8e80cbed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-03T21:00:46.446 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:46.443+0000 7fb8e7e00640 1 --2- 172.21.15.32:0/181399801 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb8e80d2290 0x7fb8e80cb990 secure :-1 s=READY pgs=2142 cs=0 l=1 rev1=1 crypto rx=0x7fb8d400c3c0 tx=0x7fb8d400c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T21:00:46.446 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:46.443+0000 7fb8e4c00640 1 -- 172.21.15.32:0/181399801 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8d4017070 con 0x7fb8e80d2290 2024-08-03T21:00:46.447 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:46.443+0000 7fb8e4c00640 1 -- 172.21.15.32:0/181399801 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fb8d40035f0 con 0x7fb8e80d2290 2024-08-03T21:00:46.447 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:46.443+0000 7fb8e4c00640 1 -- 172.21.15.32:0/181399801 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8d4003a80 con 0x7fb8e80d2290 2024-08-03T21:00:46.447 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:46.443+0000 7fb8ee200640 1 -- 172.21.15.32:0/181399801 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb8e80b7970 con 0x7fb8e80d2290 2024-08-03T21:00:46.447 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:46.444+0000 7fb8ee200640 1 -- 172.21.15.32:0/181399801 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb8e815fe90 con 0x7fb8e80d2290 2024-08-03T21:00:46.447 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:46.445+0000 7fb8ee200640 1 -- 172.21.15.32:0/181399801 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb8e815cf20 con 0x7fb8e80d2290 2024-08-03T21:00:46.447 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:46.446+0000 7fb8e4c00640 1 -- 172.21.15.32:0/181399801 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7fb8d401a070 con 0x7fb8e80d2290 2024-08-03T21:00:46.449 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:46.447+0000 7fb8e4c00640 1 --2- 172.21.15.32:0/181399801 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fb8c8077010 0x7fb8c8079520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:00:46.449 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:46.447+0000 7fb8e4c00640 1 -- 172.21.15.32:0/181399801 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(131..131 src has 1..131) ==== 4857+0+0 (secure 0 0 0) 0x7fb8d4097900 con 0x7fb8e80d2290 2024-08-03T21:00:46.449 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:46.448+0000 7fb8ece00640 1 --2- 172.21.15.32:0/181399801 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fb8c8077010 0x7fb8c8079520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:00:46.450 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:46.448+0000 7fb8ece00640 1 --2- 172.21.15.32:0/181399801 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fb8c8077010 0x7fb8c8079520 secure :-1 s=READY pgs=856 cs=0 l=1 rev1=1 crypto rx=0x7fb8dc033000 tx=0x7fb8dc004730 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T21:00:46.453 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:46.452+0000 7fb8e4c00640 1 -- 172.21.15.32:0/181399801 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7fb8d4060ed0 con 0x7fb8e80d2290 2024-08-03T21:00:46.731 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:46.729+0000 7fb8ee200640 1 -- 172.21.15.32:0/181399801 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) -- 0x7fb8e8166790 con 0x7fb8e80d2290 2024-08-03T21:00:46.731 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:46.730+0000 7fb8e4c00640 1 -- 172.21.15.32:0/181399801 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v17) ==== 80+0+5304 (secure 0 0 0) 0x7fb8e8166790 con 0x7fb8e80d2290 2024-08-03T21:00:46.732 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-03T21:00:46.732 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-08-03T21:00:46.732 INFO:teuthology.orchestra.run.smithi032.stdout: "auth_dump": [ 2024-08-03T21:00:46.732 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-03T21:00:46.732 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "mds.a", 2024-08-03T21:00:46.732 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQAvl65mfaKbOhAAUgCH82U0SB7owF5Cds3XYA==", 2024-08-03T21:00:46.732 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2024-08-03T21:00:46.732 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": "allow", 2024-08-03T21:00:46.732 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow *", 2024-08-03T21:00:46.732 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow *", 2024-08-03T21:00:46.732 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2024-08-03T21:00:46.733 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-03T21:00:46.733 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-03T21:00:46.733 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-03T21:00:46.733 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "mds.b", 2024-08-03T21:00:46.733 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQAwl65mNcoVGxAAlRXNjZhxVu8Vjslk9Msrxw==", 2024-08-03T21:00:46.733 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2024-08-03T21:00:46.733 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": "allow", 2024-08-03T21:00:46.733 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow *", 2024-08-03T21:00:46.733 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow *", 2024-08-03T21:00:46.733 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2024-08-03T21:00:46.733 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-03T21:00:46.733 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-03T21:00:46.734 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-03T21:00:46.734 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "mds.c", 2024-08-03T21:00:46.734 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQAwl65mEUAFDRAAGdw1cymfvfKxGeJS/Od6Rw==", 2024-08-03T21:00:46.734 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2024-08-03T21:00:46.734 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": "allow", 2024-08-03T21:00:46.734 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow *", 2024-08-03T21:00:46.734 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow *", 2024-08-03T21:00:46.734 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2024-08-03T21:00:46.734 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-03T21:00:46.734 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-03T21:00:46.734 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-03T21:00:46.734 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "mds.d", 2024-08-03T21:00:46.734 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQAwl65mjUHZKBAAqOCnIopBK8Rhk36hrlNnzw==", 2024-08-03T21:00:46.735 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2024-08-03T21:00:46.735 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": "allow", 2024-08-03T21:00:46.735 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow *", 2024-08-03T21:00:46.735 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow *", 2024-08-03T21:00:46.735 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2024-08-03T21:00:46.735 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-03T21:00:46.735 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-03T21:00:46.735 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-03T21:00:46.735 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "osd.0", 2024-08-03T21:00:46.735 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQAyl65m4kIyLxAAdCh0D2MjJNPPu5b7jNkrbw==", 2024-08-03T21:00:46.735 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2024-08-03T21:00:46.735 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow profile osd", 2024-08-03T21:00:46.735 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow profile osd", 2024-08-03T21:00:46.736 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2024-08-03T21:00:46.736 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-03T21:00:46.736 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-03T21:00:46.736 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-03T21:00:46.736 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "osd.1", 2024-08-03T21:00:46.736 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQA1l65mLzVOOhAA081V1cUyYFPz17L0nny7gA==", 2024-08-03T21:00:46.736 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2024-08-03T21:00:46.736 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow profile osd", 2024-08-03T21:00:46.736 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow profile osd", 2024-08-03T21:00:46.736 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2024-08-03T21:00:46.736 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-03T21:00:46.736 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-03T21:00:46.736 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-03T21:00:46.736 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "osd.2", 2024-08-03T21:00:46.737 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQA5l65mBnXDBxAA9/lZVkwD2ykMTOIysoYODg==", 2024-08-03T21:00:46.737 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2024-08-03T21:00:46.737 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow profile osd", 2024-08-03T21:00:46.737 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow profile osd", 2024-08-03T21:00:46.737 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2024-08-03T21:00:46.737 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-03T21:00:46.737 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-03T21:00:46.737 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-03T21:00:46.737 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "osd.3", 2024-08-03T21:00:46.737 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQA8l65mAriiERAAfzV8An67+PnA3EzeAutIbA==", 2024-08-03T21:00:46.737 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2024-08-03T21:00:46.737 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow profile osd", 2024-08-03T21:00:46.737 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow profile osd", 2024-08-03T21:00:46.737 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2024-08-03T21:00:46.738 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-03T21:00:46.738 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-03T21:00:46.738 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-03T21:00:46.738 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "osd.4", 2024-08-03T21:00:46.738 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQBAl65m5uJSJRAA8S2IjJpxuqHY8YK4tTDMPg==", 2024-08-03T21:00:46.738 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2024-08-03T21:00:46.738 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow profile osd", 2024-08-03T21:00:46.738 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow profile osd", 2024-08-03T21:00:46.738 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2024-08-03T21:00:46.738 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-03T21:00:46.738 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-03T21:00:46.738 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-03T21:00:46.739 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "osd.5", 2024-08-03T21:00:46.739 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQBDl65mBP3JLBAADgK2X3zNyfnjjZADX1PFcA==", 2024-08-03T21:00:46.739 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2024-08-03T21:00:46.739 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow profile osd", 2024-08-03T21:00:46.739 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow profile osd", 2024-08-03T21:00:46.739 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2024-08-03T21:00:46.739 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-03T21:00:46.739 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-03T21:00:46.739 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-03T21:00:46.739 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "osd.6", 2024-08-03T21:00:46.739 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQBGl65m8DBNNRAANeOUfW70S5JfqFlpuMF9KQ==", 2024-08-03T21:00:46.739 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2024-08-03T21:00:46.739 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow profile osd", 2024-08-03T21:00:46.740 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow profile osd", 2024-08-03T21:00:46.740 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2024-08-03T21:00:46.740 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-03T21:00:46.740 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-03T21:00:46.740 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-03T21:00:46.740 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "osd.7", 2024-08-03T21:00:46.740 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQBKl65mUnOcABAA41wylzC1Ec9Fj01Ke0HhTw==", 2024-08-03T21:00:46.740 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2024-08-03T21:00:46.740 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow profile osd", 2024-08-03T21:00:46.740 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow profile osd", 2024-08-03T21:00:46.740 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2024-08-03T21:00:46.740 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-03T21:00:46.740 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-03T21:00:46.741 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-03T21:00:46.741 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "client.0", 2024-08-03T21:00:46.741 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQAwl65msgO3NRAAHG/WuMTK9c4lDXocXvqWnA==", 2024-08-03T21:00:46.741 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2024-08-03T21:00:46.741 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": "allow", 2024-08-03T21:00:46.741 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow r", 2024-08-03T21:00:46.741 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-08-03T21:00:46.741 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-03T21:00:46.741 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-03T21:00:46.741 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-03T21:00:46.741 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "client.1", 2024-08-03T21:00:46.741 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQAxl65mbe19DxAA+sepc9ucXk2Bqy7C90mfbA==", 2024-08-03T21:00:46.741 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2024-08-03T21:00:46.742 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": "allow", 2024-08-03T21:00:46.742 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow r", 2024-08-03T21:00:46.742 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow rw", 2024-08-03T21:00:46.742 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow rwx" 2024-08-03T21:00:46.742 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-03T21:00:46.742 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-03T21:00:46.742 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-03T21:00:46.742 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "client.2", 2024-08-03T21:00:46.742 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQAxl65mBnbkBBAAjor60WWLTKXidiBMgtKuEA==", 2024-08-03T21:00:46.742 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2024-08-03T21:00:46.742 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": "allow", 2024-08-03T21:00:46.743 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow r", 2024-08-03T21:00:46.743 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow rw", 2024-08-03T21:00:46.743 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow rwx" 2024-08-03T21:00:46.743 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-03T21:00:46.743 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-03T21:00:46.743 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-03T21:00:46.743 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "client.3", 2024-08-03T21:00:46.743 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQAxl65munz8GRAA/cRl3ghwQE42FOizurcW3g==", 2024-08-03T21:00:46.743 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2024-08-03T21:00:46.743 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": "allow", 2024-08-03T21:00:46.743 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow r", 2024-08-03T21:00:46.743 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow rw", 2024-08-03T21:00:46.743 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow rwx" 2024-08-03T21:00:46.743 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-03T21:00:46.744 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-03T21:00:46.744 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-03T21:00:46.744 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "client.admin", 2024-08-03T21:00:46.744 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQAul65mB8VcOhAA105JQ2FX1vhtaGj3Fer8mg==", 2024-08-03T21:00:46.744 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2024-08-03T21:00:46.744 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": "allow *", 2024-08-03T21:00:46.744 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow *", 2024-08-03T21:00:46.744 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow *", 2024-08-03T21:00:46.744 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2024-08-03T21:00:46.744 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-03T21:00:46.744 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-03T21:00:46.745 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-03T21:00:46.745 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "mgr.x", 2024-08-03T21:00:46.745 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQAvl65mUaPKLxAAnll74Gmk8nSuN/HuX6rTxA==", 2024-08-03T21:00:46.745 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2024-08-03T21:00:46.745 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": "allow *", 2024-08-03T21:00:46.745 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow profile mgr", 2024-08-03T21:00:46.745 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2024-08-03T21:00:46.745 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-03T21:00:46.745 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-08-03T21:00:46.745 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-03T21:00:46.745 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "mgr.y", 2024-08-03T21:00:46.745 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQAvl65m5Gq4JBAA306Ge1zHF3Z0WIzUrset4A==", 2024-08-03T21:00:46.745 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2024-08-03T21:00:46.746 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": "allow *", 2024-08-03T21:00:46.746 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow profile mgr", 2024-08-03T21:00:46.746 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2024-08-03T21:00:46.746 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-03T21:00:46.746 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-03T21:00:46.746 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-08-03T21:00:46.746 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-08-03T21:00:46.747 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:46.732+0000 7fb8ee200640 1 -- 172.21.15.32:0/181399801 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fb8c8077010 msgr2=0x7fb8c8079520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:00:46.747 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:46.732+0000 7fb8ee200640 1 --2- 172.21.15.32:0/181399801 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fb8c8077010 0x7fb8c8079520 secure :-1 s=READY pgs=856 cs=0 l=1 rev1=1 crypto rx=0x7fb8dc033000 tx=0x7fb8dc004730 comp rx=0 tx=0).stop 2024-08-03T21:00:46.747 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:46.732+0000 7fb8ee200640 1 -- 172.21.15.32:0/181399801 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb8e80d2290 msgr2=0x7fb8e80cb990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:00:46.747 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:46.732+0000 7fb8ee200640 1 --2- 172.21.15.32:0/181399801 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb8e80d2290 0x7fb8e80cb990 secure :-1 s=READY pgs=2142 cs=0 l=1 rev1=1 crypto rx=0x7fb8d400c3c0 tx=0x7fb8d400c880 comp rx=0 tx=0).stop 2024-08-03T21:00:46.747 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:46.732+0000 7fb8ee200640 1 -- 172.21.15.32:0/181399801 shutdown_connections 2024-08-03T21:00:46.747 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:46.732+0000 7fb8ee200640 1 --2- 172.21.15.32:0/181399801 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fb8c8077010 0x7fb8c8079520 unknown :-1 s=CLOSED pgs=856 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:46.747 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:46.732+0000 7fb8ee200640 1 --2- 172.21.15.32:0/181399801 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fb8e80d4bb0 0x7fb8e80cbed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:46.747 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:46.732+0000 7fb8ee200640 1 --2- 172.21.15.32:0/181399801 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb8e80d2290 0x7fb8e80cb990 unknown :-1 s=CLOSED pgs=2142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:46.748 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:46.732+0000 7fb8ee200640 1 --2- 172.21.15.32:0/181399801 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb8e80cec30 0x7fb8e815c570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:46.748 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:46.732+0000 7fb8ee200640 1 -- 172.21.15.32:0/181399801 >> 172.21.15.32:0/181399801 conn(0x7fb8e80d4090 msgr2=0x7fb8e80d44a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T21:00:46.748 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:46.732+0000 7fb8ee200640 1 -- 172.21.15.32:0/181399801 shutdown_connections 2024-08-03T21:00:46.748 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:46.732+0000 7fb8ee200640 1 -- 172.21.15.32:0/181399801 wait complete. 2024-08-03T21:00:46.761 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-08-03T21:00:46.989 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:46.987+0000 7fdc0e400640 1 -- 172.21.15.32:0/1043388199 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdc080af020 msgr2=0x7fdc080742b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:00:46.989 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:46.987+0000 7fdc0e400640 1 --2- 172.21.15.32:0/1043388199 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdc080af020 0x7fdc080742b0 secure :-1 s=READY pgs=2143 cs=0 l=1 rev1=1 crypto rx=0x7fdc08160f00 tx=0x7fdbfc02a580 comp rx=0 tx=0).stop 2024-08-03T21:00:46.989 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:46.987+0000 7fdc0e400640 1 -- 172.21.15.32:0/1043388199 shutdown_connections 2024-08-03T21:00:46.989 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:46.987+0000 7fdc0e400640 1 --2- 172.21.15.32:0/1043388199 >> v2:172.21.15.118:3300/0 conn(0x7fdc080acd80 0x7fdc080ad190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:46.989 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:46.987+0000 7fdc0e400640 1 --2- 172.21.15.32:0/1043388199 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdc080af020 0x7fdc080742b0 unknown :-1 s=CLOSED pgs=2143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:46.989 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:46.987+0000 7fdc0e400640 1 -- 172.21.15.32:0/1043388199 >> 172.21.15.32:0/1043388199 conn(0x7fdc080b2120 msgr2=0x7fdc080cfd60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T21:00:46.989 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:46.987+0000 7fdc0e400640 1 -- 172.21.15.32:0/1043388199 shutdown_connections 2024-08-03T21:00:46.990 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:46.987+0000 7fdc0e400640 1 -- 172.21.15.32:0/1043388199 wait complete. 2024-08-03T21:00:46.991 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:46.989+0000 7fdc0e400640 1 Processor -- start 2024-08-03T21:00:46.991 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:46.989+0000 7fdc0e400640 1 -- start start 2024-08-03T21:00:46.995 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:46.991+0000 7fdc0e400640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fdc0806f440 0x7fdc080c9ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:00:46.995 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:46.991+0000 7fdc0e400640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fdc080acd80 0x7fdc08155130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:00:46.995 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:46.991+0000 7fdc0e400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdc080af020 0x7fdc0815ea80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:00:46.996 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:46.991+0000 7fdc0e400640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fdc081601e0 con 0x7fdc080af020 2024-08-03T21:00:46.996 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:46.991+0000 7fdc0e400640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7fdc08160060 con 0x7fdc080acd80 2024-08-03T21:00:46.996 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:46.991+0000 7fdc0e400640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7fdc08160360 con 0x7fdc0806f440 2024-08-03T21:00:46.996 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:46.991+0000 7fdc0da00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdc080af020 0x7fdc0815ea80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:00:46.996 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:46.991+0000 7fdc0da00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdc080af020 0x7fdc0815ea80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:57078/0 (socket says 172.21.15.32:57078) 2024-08-03T21:00:46.996 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:46.991+0000 7fdc0da00640 1 -- 172.21.15.32:0/3653762900 learned_addr learned my addr 172.21.15.32:0/3653762900 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T21:00:46.996 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:46.991+0000 7fdc0d000640 1 --2- 172.21.15.32:0/3653762900 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fdc0806f440 0x7fdc080c9ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:00:46.996 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:46.991+0000 7fdc07e00640 1 --2- 172.21.15.32:0/3653762900 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fdc080acd80 0x7fdc08155130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:00:46.997 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:46.992+0000 7fdc0da00640 1 -- 172.21.15.32:0/3653762900 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fdc0806f440 msgr2=0x7fdc080c9ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:00:46.997 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:46.992+0000 7fdc0da00640 1 --2- 172.21.15.32:0/3653762900 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fdc0806f440 0x7fdc080c9ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:46.997 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:46.992+0000 7fdc0da00640 1 -- 172.21.15.32:0/3653762900 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fdc080acd80 msgr2=0x7fdc08155130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:00:46.997 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:46.992+0000 7fdc0da00640 1 --2- 172.21.15.32:0/3653762900 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fdc080acd80 0x7fdc08155130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:46.997 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:46.992+0000 7fdc0da00640 1 -- 172.21.15.32:0/3653762900 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdc080b3550 con 0x7fdc080af020 2024-08-03T21:00:46.997 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:46.992+0000 7fdc0da00640 1 --2- 172.21.15.32:0/3653762900 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdc080af020 0x7fdc0815ea80 secure :-1 s=READY pgs=2144 cs=0 l=1 rev1=1 crypto rx=0x7fdc0000c760 tx=0x7fdc0000cc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T21:00:46.997 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:46.992+0000 7fdc04c00640 1 -- 172.21.15.32:0/3653762900 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc00003b70 con 0x7fdc080af020 2024-08-03T21:00:46.997 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:46.992+0000 7fdc04c00640 1 -- 172.21.15.32:0/3653762900 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fdc00003d10 con 0x7fdc080af020 2024-08-03T21:00:46.997 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:46.992+0000 7fdc0e400640 1 -- 172.21.15.32:0/3653762900 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdc080b37f0 con 0x7fdc080af020 2024-08-03T21:00:46.997 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:46.992+0000 7fdc04c00640 1 -- 172.21.15.32:0/3653762900 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc0000aab0 con 0x7fdc080af020 2024-08-03T21:00:46.997 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:46.993+0000 7fdc0e400640 1 -- 172.21.15.32:0/3653762900 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdc080c7220 con 0x7fdc080af020 2024-08-03T21:00:46.997 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:46.993+0000 7fdc0e400640 1 -- 172.21.15.32:0/3653762900 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdc080ad190 con 0x7fdc080af020 2024-08-03T21:00:46.998 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:46.995+0000 7fdc04c00640 1 -- 172.21.15.32:0/3653762900 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7fdc0000ac50 con 0x7fdc080af020 2024-08-03T21:00:46.998 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:46.996+0000 7fdc04c00640 1 --2- 172.21.15.32:0/3653762900 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fdbe8077040 0x7fdbe8079550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:00:46.998 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:46.996+0000 7fdc04c00640 1 -- 172.21.15.32:0/3653762900 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(131..131 src has 1..131) ==== 4857+0+0 (secure 0 0 0) 0x7fdc00098200 con 0x7fdc080af020 2024-08-03T21:00:46.999 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:46.996+0000 7fdc0d000640 1 --2- 172.21.15.32:0/3653762900 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fdbe8077040 0x7fdbe8079550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:00:46.999 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:46.997+0000 7fdc0d000640 1 --2- 172.21.15.32:0/3653762900 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fdbe8077040 0x7fdbe8079550 secure :-1 s=READY pgs=857 cs=0 l=1 rev1=1 crypto rx=0x7fdbfc033000 tx=0x7fdbfc004730 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T21:00:47.001 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:47.000+0000 7fdc04c00640 1 -- 172.21.15.32:0/3653762900 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7fdc000610f0 con 0x7fdc080af020 2024-08-03T21:00:47.211 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:47.210+0000 7fdc0e400640 1 -- 172.21.15.32:0/3653762900 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) -- 0x7fdc080d0930 con 0x7fdc080af020 2024-08-03T21:00:47.211 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:47.210+0000 7fdc04c00640 1 -- 172.21.15.32:0/3653762900 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v131) ==== 84+0+65 (secure 0 0 0) 0x7fdc00065640 con 0x7fdc080af020 2024-08-03T21:00:47.212 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-03T21:00:47.212 INFO:teuthology.orchestra.run.smithi032.stdout:[ 2024-08-03T21:00:47.212 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-08-03T21:00:47.212 INFO:teuthology.orchestra.run.smithi032.stdout: "poolnum": 1, 2024-08-03T21:00:47.212 INFO:teuthology.orchestra.run.smithi032.stdout: "poolname": ".mgr" 2024-08-03T21:00:47.212 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-08-03T21:00:47.212 INFO:teuthology.orchestra.run.smithi032.stdout:] 2024-08-03T21:00:47.214 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:47.212+0000 7fdc0e400640 1 -- 172.21.15.32:0/3653762900 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fdbe8077040 msgr2=0x7fdbe8079550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:00:47.215 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:47.212+0000 7fdc0e400640 1 --2- 172.21.15.32:0/3653762900 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fdbe8077040 0x7fdbe8079550 secure :-1 s=READY pgs=857 cs=0 l=1 rev1=1 crypto rx=0x7fdbfc033000 tx=0x7fdbfc004730 comp rx=0 tx=0).stop 2024-08-03T21:00:47.215 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:47.212+0000 7fdc0e400640 1 -- 172.21.15.32:0/3653762900 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdc080af020 msgr2=0x7fdc0815ea80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:00:47.215 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:47.212+0000 7fdc0e400640 1 --2- 172.21.15.32:0/3653762900 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdc080af020 0x7fdc0815ea80 secure :-1 s=READY pgs=2144 cs=0 l=1 rev1=1 crypto rx=0x7fdc0000c760 tx=0x7fdc0000cc20 comp rx=0 tx=0).stop 2024-08-03T21:00:47.215 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:47.213+0000 7fdc0e400640 1 -- 172.21.15.32:0/3653762900 shutdown_connections 2024-08-03T21:00:47.215 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:47.213+0000 7fdc0e400640 1 --2- 172.21.15.32:0/3653762900 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fdbe8077040 0x7fdbe8079550 unknown :-1 s=CLOSED pgs=857 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:47.215 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:47.213+0000 7fdc0e400640 1 --2- 172.21.15.32:0/3653762900 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fdc080acd80 0x7fdc08155130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:47.215 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:47.213+0000 7fdc0e400640 1 --2- 172.21.15.32:0/3653762900 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdc080af020 0x7fdc0815ea80 unknown :-1 s=CLOSED pgs=2144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:47.215 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:47.213+0000 7fdc0e400640 1 --2- 172.21.15.32:0/3653762900 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fdc0806f440 0x7fdc080c9ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:47.215 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:47.213+0000 7fdc0e400640 1 -- 172.21.15.32:0/3653762900 >> 172.21.15.32:0/3653762900 conn(0x7fdc080b2120 msgr2=0x7fdc0815bbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T21:00:47.215 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:47.213+0000 7fdc0e400640 1 -- 172.21.15.32:0/3653762900 shutdown_connections 2024-08-03T21:00:47.215 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:47.213+0000 7fdc0e400640 1 -- 172.21.15.32:0/3653762900 wait complete. 2024-08-03T21:00:47.238 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-08-03T21:00:47.238 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-08-03T21:00:47.474 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:47.472+0000 7f86cfe00640 1 -- 172.21.15.32:0/3473335571 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f86c80b3e80 msgr2=0x7f86c80b4290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:00:47.474 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:47.472+0000 7f86cfe00640 1 --2- 172.21.15.32:0/3473335571 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f86c80b3e80 0x7f86c80b4290 secure :-1 s=READY pgs=2146 cs=0 l=1 rev1=1 crypto rx=0x7f86c815fea0 tx=0x7f86c002a8b0 comp rx=0 tx=0).stop 2024-08-03T21:00:47.474 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:47.472+0000 7f86cfe00640 1 -- 172.21.15.32:0/3473335571 shutdown_connections 2024-08-03T21:00:47.474 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:47.472+0000 7f86cfe00640 1 --2- 172.21.15.32:0/3473335571 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f86c80b3e80 0x7f86c80b4290 unknown :-1 s=CLOSED pgs=2146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:47.475 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:47.472+0000 7f86cfe00640 1 --2- 172.21.15.32:0/3473335571 >> v2:172.21.15.118:3300/0 conn(0x7f86c806e950 0x7f86c806ed60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:47.475 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:47.472+0000 7f86cfe00640 1 -- 172.21.15.32:0/3473335571 >> 172.21.15.32:0/3473335571 conn(0x7f86c80b1bd0 msgr2=0x7f86c80b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T21:00:47.475 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:47.472+0000 7f86cfe00640 1 -- 172.21.15.32:0/3473335571 shutdown_connections 2024-08-03T21:00:47.475 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:47.472+0000 7f86cfe00640 1 -- 172.21.15.32:0/3473335571 wait complete. 2024-08-03T21:00:47.476 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:47.474+0000 7f86cfe00640 1 Processor -- start 2024-08-03T21:00:47.476 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:47.475+0000 7f86cfe00640 1 -- start start 2024-08-03T21:00:47.478 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:47.476+0000 7f86cfe00640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f86c806e950 0x7f86c815f870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:00:47.479 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:47.476+0000 7f86cfe00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f86c80b3e80 0x7f86c8073890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:00:47.479 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:47.476+0000 7f86cfe00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f86c815d2b0 0x7f86c8073dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:00:47.479 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:47.476+0000 7f86cfe00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f86c80cd010 con 0x7f86c815d2b0 2024-08-03T21:00:47.479 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:47.476+0000 7f86cfe00640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f86c80cce90 con 0x7f86c80b3e80 2024-08-03T21:00:47.479 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:47.476+0000 7f86cfe00640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7f86c80cd190 con 0x7f86c806e950 2024-08-03T21:00:47.479 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:47.476+0000 7f86cde00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f86c815d2b0 0x7f86c8073dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:00:47.479 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:47.476+0000 7f86cde00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f86c815d2b0 0x7f86c8073dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:57090/0 (socket says 172.21.15.32:57090) 2024-08-03T21:00:47.479 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:47.477+0000 7f86cde00640 1 -- 172.21.15.32:0/2565176565 learned_addr learned my addr 172.21.15.32:0/2565176565 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T21:00:47.479 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:47.477+0000 7f86cca00640 1 --2- 172.21.15.32:0/2565176565 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f86c80b3e80 0x7f86c8073890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:00:47.479 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:47.477+0000 7f86cd400640 1 --2- 172.21.15.32:0/2565176565 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f86c806e950 0x7f86c815f870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:00:47.479 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:47.477+0000 7f86cde00640 1 -- 172.21.15.32:0/2565176565 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f86c806e950 msgr2=0x7f86c815f870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:00:47.479 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:47.477+0000 7f86cde00640 1 --2- 172.21.15.32:0/2565176565 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f86c806e950 0x7f86c815f870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:47.480 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:47.477+0000 7f86cde00640 1 -- 172.21.15.32:0/2565176565 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f86c80b3e80 msgr2=0x7f86c8073890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:00:47.480 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:47.477+0000 7f86cde00640 1 --2- 172.21.15.32:0/2565176565 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f86c80b3e80 0x7f86c8073890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:47.480 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:47.477+0000 7f86cde00640 1 -- 172.21.15.32:0/2565176565 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f86c8153a00 con 0x7f86c815d2b0 2024-08-03T21:00:47.480 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:47.477+0000 7f86cde00640 1 --2- 172.21.15.32:0/2565176565 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f86c815d2b0 0x7f86c8073dd0 secure :-1 s=READY pgs=2147 cs=0 l=1 rev1=1 crypto rx=0x7f86c0005fd0 tx=0x7f86c00316d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T21:00:47.481 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:47.478+0000 7f86c5600640 1 -- 172.21.15.32:0/2565176565 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f86c0031990 con 0x7f86c815d2b0 2024-08-03T21:00:47.481 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:47.478+0000 7f86c5600640 1 -- 172.21.15.32:0/2565176565 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f86c0031c20 con 0x7f86c815d2b0 2024-08-03T21:00:47.481 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:47.478+0000 7f86cfe00640 1 -- 172.21.15.32:0/2565176565 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f86c8153f00 con 0x7f86c815d2b0 2024-08-03T21:00:47.481 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:47.478+0000 7f86c5600640 1 -- 172.21.15.32:0/2565176565 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f86c00330d0 con 0x7f86c815d2b0 2024-08-03T21:00:47.481 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:47.478+0000 7f86cfe00640 1 -- 172.21.15.32:0/2565176565 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f86c81544c0 con 0x7f86c815d2b0 2024-08-03T21:00:47.481 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:47.479+0000 7f86cfe00640 1 -- 172.21.15.32:0/2565176565 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f86c8153c70 con 0x7f86c815d2b0 2024-08-03T21:00:47.481 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:47.480+0000 7f86c5600640 1 -- 172.21.15.32:0/2565176565 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f86c0033270 con 0x7f86c815d2b0 2024-08-03T21:00:47.482 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:47.481+0000 7f86c5600640 1 --2- 172.21.15.32:0/2565176565 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f86a8077040 0x7f86a8079550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:00:47.482 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:47.481+0000 7f86c5600640 1 -- 172.21.15.32:0/2565176565 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(131..131 src has 1..131) ==== 4857+0+0 (secure 0 0 0) 0x7f86c00b98b0 con 0x7f86c815d2b0 2024-08-03T21:00:47.483 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:47.482+0000 7f86cd400640 1 --2- 172.21.15.32:0/2565176565 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f86a8077040 0x7f86a8079550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:00:47.484 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:47.482+0000 7f86cd400640 1 --2- 172.21.15.32:0/2565176565 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f86a8077040 0x7f86a8079550 secure :-1 s=READY pgs=861 cs=0 l=1 rev1=1 crypto rx=0x7f86bc000f60 tx=0x7f86bc009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T21:00:47.486 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:47.485+0000 7f86c5600640 1 -- 172.21.15.32:0/2565176565 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7f86c003f030 con 0x7f86c815d2b0 2024-08-03T21:00:47.692 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:47.691+0000 7f86cfe00640 1 -- 172.21.15.32:0/2565176565 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) -- 0x7f86c806ed90 con 0x7f86c815d2b0 2024-08-03T21:00:48.195 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:48.193+0000 7f86c5600640 1 -- 172.21.15.32:0/2565176565 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v132) ==== 138+0+0 (secure 0 0 0) 0x7f86c806ed90 con 0x7f86c815d2b0 2024-08-03T21:00:48.195 INFO:teuthology.orchestra.run.smithi032.stderr:pool 'cephfs_metadata' created 2024-08-03T21:00:48.199 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:48.197+0000 7f86cfe00640 1 -- 172.21.15.32:0/2565176565 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f86a8077040 msgr2=0x7f86a8079550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:00:48.199 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:48.197+0000 7f86cfe00640 1 --2- 172.21.15.32:0/2565176565 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f86a8077040 0x7f86a8079550 secure :-1 s=READY pgs=861 cs=0 l=1 rev1=1 crypto rx=0x7f86bc000f60 tx=0x7f86bc009040 comp rx=0 tx=0).stop 2024-08-03T21:00:48.200 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:48.197+0000 7f86cfe00640 1 -- 172.21.15.32:0/2565176565 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f86c815d2b0 msgr2=0x7f86c8073dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:00:48.200 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:48.197+0000 7f86cfe00640 1 --2- 172.21.15.32:0/2565176565 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f86c815d2b0 0x7f86c8073dd0 secure :-1 s=READY pgs=2147 cs=0 l=1 rev1=1 crypto rx=0x7f86c0005fd0 tx=0x7f86c00316d0 comp rx=0 tx=0).stop 2024-08-03T21:00:48.200 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:48.197+0000 7f86cfe00640 1 -- 172.21.15.32:0/2565176565 shutdown_connections 2024-08-03T21:00:48.200 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:48.197+0000 7f86cfe00640 1 --2- 172.21.15.32:0/2565176565 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f86a8077040 0x7f86a8079550 unknown :-1 s=CLOSED pgs=861 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:48.200 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:48.197+0000 7f86cfe00640 1 --2- 172.21.15.32:0/2565176565 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f86c80b3e80 0x7f86c8073890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:48.201 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:48.197+0000 7f86cfe00640 1 --2- 172.21.15.32:0/2565176565 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f86c815d2b0 0x7f86c8073dd0 unknown :-1 s=CLOSED pgs=2147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:48.201 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:48.197+0000 7f86cfe00640 1 --2- 172.21.15.32:0/2565176565 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f86c806e950 0x7f86c815f870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:48.201 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:48.197+0000 7f86cfe00640 1 -- 172.21.15.32:0/2565176565 >> 172.21.15.32:0/2565176565 conn(0x7f86c80b1bd0 msgr2=0x7f86c8068940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T21:00:48.201 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:48.197+0000 7f86cfe00640 1 -- 172.21.15.32:0/2565176565 shutdown_connections 2024-08-03T21:00:48.201 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:48.197+0000 7f86cfe00640 1 -- 172.21.15.32:0/2565176565 wait complete. 2024-08-03T21:00:48.231 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-08-03T21:00:48.467 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:48.465+0000 7fb0bf200640 1 -- 172.21.15.32:0/3706506955 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb0c0159ff0 msgr2=0x7fb0c015fe10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:00:48.468 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:48.465+0000 7fb0bf200640 1 --2- 172.21.15.32:0/3706506955 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb0c0159ff0 0x7fb0c015fe10 secure :-1 s=READY pgs=2149 cs=0 l=1 rev1=1 crypto rx=0x7fb0ac009830 tx=0x7fb0ac02a5c0 comp rx=0 tx=0).stop 2024-08-03T21:00:48.468 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:48.466+0000 7fb0bf200640 1 -- 172.21.15.32:0/3706506955 shutdown_connections 2024-08-03T21:00:48.468 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:48.466+0000 7fb0bf200640 1 --2- 172.21.15.32:0/3706506955 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb0c0159ff0 0x7fb0c015fe10 unknown :-1 s=CLOSED pgs=2149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:48.468 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:48.466+0000 7fb0bf200640 1 -- 172.21.15.32:0/3706506955 >> 172.21.15.32:0/3706506955 conn(0x7fb0c00cfd80 msgr2=0x7fb0c00d0190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T21:00:48.468 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:48.466+0000 7fb0bf200640 1 -- 172.21.15.32:0/3706506955 shutdown_connections 2024-08-03T21:00:48.468 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:48.466+0000 7fb0bf200640 1 -- 172.21.15.32:0/3706506955 wait complete. 2024-08-03T21:00:48.469 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:48.468+0000 7fb0bf200640 1 Processor -- start 2024-08-03T21:00:48.469 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:48.468+0000 7fb0bf200640 1 -- start start 2024-08-03T21:00:48.473 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:48.469+0000 7fb0bf200640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb0c0159ff0 0x7fb0c01f3130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:00:48.473 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:48.469+0000 7fb0bf200640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fb0c0160350 0x7fb0c00c5630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:00:48.473 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:48.469+0000 7fb0bf200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb0c01609d0 0x7fb0c00c5b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:00:48.473 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:48.469+0000 7fb0bf200640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fb0c006f0d0 con 0x7fb0c01609d0 2024-08-03T21:00:48.473 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:48.469+0000 7fb0bf200640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7fb0c006ef50 con 0x7fb0c0159ff0 2024-08-03T21:00:48.473 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:48.469+0000 7fb0bf200640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7fb0c006f250 con 0x7fb0c0160350 2024-08-03T21:00:48.474 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:48.470+0000 7fb0be800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb0c01609d0 0x7fb0c00c5b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:00:48.474 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:48.470+0000 7fb0be800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb0c01609d0 0x7fb0c00c5b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:57122/0 (socket says 172.21.15.32:57122) 2024-08-03T21:00:48.474 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:48.470+0000 7fb0be800640 1 -- 172.21.15.32:0/3568407595 learned_addr learned my addr 172.21.15.32:0/3568407595 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T21:00:48.474 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:48.470+0000 7fb0bde00640 1 --2- 172.21.15.32:0/3568407595 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb0c0159ff0 0x7fb0c01f3130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:00:48.474 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:48.470+0000 7fb0bd400640 1 --2- 172.21.15.32:0/3568407595 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fb0c0160350 0x7fb0c00c5630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:00:48.474 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:48.470+0000 7fb0be800640 1 -- 172.21.15.32:0/3568407595 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fb0c0160350 msgr2=0x7fb0c00c5630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:00:48.475 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:48.470+0000 7fb0be800640 1 --2- 172.21.15.32:0/3568407595 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fb0c0160350 0x7fb0c00c5630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:48.475 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:48.470+0000 7fb0be800640 1 -- 172.21.15.32:0/3568407595 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb0c0159ff0 msgr2=0x7fb0c01f3130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:00:48.475 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:48.470+0000 7fb0be800640 1 --2- 172.21.15.32:0/3568407595 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb0c0159ff0 0x7fb0c01f3130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:48.475 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:48.470+0000 7fb0be800640 1 -- 172.21.15.32:0/3568407595 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb0c00b4710 con 0x7fb0c01609d0 2024-08-03T21:00:48.475 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:48.471+0000 7fb0be800640 1 --2- 172.21.15.32:0/3568407595 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb0c01609d0 0x7fb0c00c5b70 secure :-1 s=READY pgs=2150 cs=0 l=1 rev1=1 crypto rx=0x7fb0b800cad0 tx=0x7fb0b800cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T21:00:48.475 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:48.471+0000 7fb0b6000640 1 -- 172.21.15.32:0/3568407595 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0b8016020 con 0x7fb0c01609d0 2024-08-03T21:00:48.475 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:48.471+0000 7fb0b6000640 1 -- 172.21.15.32:0/3568407595 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fb0b800ad20 con 0x7fb0c01609d0 2024-08-03T21:00:48.476 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:48.471+0000 7fb0b6000640 1 -- 172.21.15.32:0/3568407595 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0b8012680 con 0x7fb0c01609d0 2024-08-03T21:00:48.476 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:48.471+0000 7fb0bf200640 1 -- 172.21.15.32:0/3568407595 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb0c00b49f0 con 0x7fb0c01609d0 2024-08-03T21:00:48.476 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:48.471+0000 7fb0bf200640 1 -- 172.21.15.32:0/3568407595 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb0c00c4e30 con 0x7fb0c01609d0 2024-08-03T21:00:48.477 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:48.472+0000 7fb0bf200640 1 -- 172.21.15.32:0/3568407595 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb0c00b4480 con 0x7fb0c01609d0 2024-08-03T21:00:48.478 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:48.474+0000 7fb0b6000640 1 -- 172.21.15.32:0/3568407595 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7fb0b8019020 con 0x7fb0c01609d0 2024-08-03T21:00:48.478 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:48.475+0000 7fb0b6000640 1 --2- 172.21.15.32:0/3568407595 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fb0a0077010 0x7fb0a0079520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:00:48.478 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:48.475+0000 7fb0b6000640 1 -- 172.21.15.32:0/3568407595 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(132..132 src has 1..132) ==== 5227+0+0 (secure 0 0 0) 0x7fb0b8099360 con 0x7fb0c01609d0 2024-08-03T21:00:48.478 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:48.476+0000 7fb0bde00640 1 --2- 172.21.15.32:0/3568407595 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fb0a0077010 0x7fb0a0079520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:00:48.478 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:48.476+0000 7fb0bde00640 1 --2- 172.21.15.32:0/3568407595 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fb0a0077010 0x7fb0a0079520 secure :-1 s=READY pgs=862 cs=0 l=1 rev1=1 crypto rx=0x7fb0b0001720 tx=0x7fb0b0009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T21:00:48.481 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:48.480+0000 7fb0b6000640 1 -- 172.21.15.32:0/3568407595 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7fb0b8020020 con 0x7fb0c01609d0 2024-08-03T21:00:48.690 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:48.688+0000 7fb0bf200640 1 -- 172.21.15.32:0/3568407595 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) -- 0x7fb0c015fe10 con 0x7fb0c01609d0 2024-08-03T21:00:49.203 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:49.201+0000 7fb0b6000640 1 -- 172.21.15.32:0/3568407595 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v133) ==== 170+0+0 (secure 0 0 0) 0x7fb0c0160760 con 0x7fb0c01609d0 2024-08-03T21:00:49.203 INFO:teuthology.orchestra.run.smithi032.stderr:pool 'cephfs_data' created 2024-08-03T21:00:49.206 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:49.204+0000 7fb0bf200640 1 -- 172.21.15.32:0/3568407595 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fb0a0077010 msgr2=0x7fb0a0079520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:00:49.206 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:49.204+0000 7fb0bf200640 1 --2- 172.21.15.32:0/3568407595 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fb0a0077010 0x7fb0a0079520 secure :-1 s=READY pgs=862 cs=0 l=1 rev1=1 crypto rx=0x7fb0b0001720 tx=0x7fb0b0009040 comp rx=0 tx=0).stop 2024-08-03T21:00:49.207 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:49.204+0000 7fb0bf200640 1 -- 172.21.15.32:0/3568407595 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb0c01609d0 msgr2=0x7fb0c00c5b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:00:49.207 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:49.204+0000 7fb0bf200640 1 --2- 172.21.15.32:0/3568407595 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb0c01609d0 0x7fb0c00c5b70 secure :-1 s=READY pgs=2150 cs=0 l=1 rev1=1 crypto rx=0x7fb0b800cad0 tx=0x7fb0b800cf90 comp rx=0 tx=0).stop 2024-08-03T21:00:49.207 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:49.204+0000 7fb0bf200640 1 -- 172.21.15.32:0/3568407595 shutdown_connections 2024-08-03T21:00:49.207 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:49.204+0000 7fb0bf200640 1 --2- 172.21.15.32:0/3568407595 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fb0a0077010 0x7fb0a0079520 unknown :-1 s=CLOSED pgs=862 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:49.207 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:49.204+0000 7fb0bf200640 1 --2- 172.21.15.32:0/3568407595 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb0c01609d0 0x7fb0c00c5b70 unknown :-1 s=CLOSED pgs=2150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:49.207 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:49.204+0000 7fb0bf200640 1 --2- 172.21.15.32:0/3568407595 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fb0c0160350 0x7fb0c00c5630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:49.207 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:49.204+0000 7fb0bf200640 1 --2- 172.21.15.32:0/3568407595 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb0c0159ff0 0x7fb0c01f3130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:49.207 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:49.204+0000 7fb0bf200640 1 -- 172.21.15.32:0/3568407595 >> 172.21.15.32:0/3568407595 conn(0x7fb0c00cfd80 msgr2=0x7fb0c00b0a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T21:00:49.207 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:49.204+0000 7fb0bf200640 1 -- 172.21.15.32:0/3568407595 shutdown_connections 2024-08-03T21:00:49.208 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:49.205+0000 7fb0bf200640 1 -- 172.21.15.32:0/3568407595 wait complete. 2024-08-03T21:00:49.247 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-08-03T21:00:49.480 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:49.478+0000 7fc920000640 1 -- 172.21.15.32:0/3424301841 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc9180cccf0 msgr2=0x7fc9180cd100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:00:49.481 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:49.478+0000 7fc920000640 1 --2- 172.21.15.32:0/3424301841 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc9180cccf0 0x7fc9180cd100 secure :-1 s=READY pgs=2151 cs=0 l=1 rev1=1 crypto rx=0x7fc904005fd0 tx=0x7fc90402a5c0 comp rx=0 tx=0).stop 2024-08-03T21:00:49.481 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:49.478+0000 7fc920000640 1 -- 172.21.15.32:0/3424301841 shutdown_connections 2024-08-03T21:00:49.481 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:49.478+0000 7fc920000640 1 --2- 172.21.15.32:0/3424301841 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fc9180ced30 0x7fc9180cf140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:49.481 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:49.478+0000 7fc920000640 1 --2- 172.21.15.32:0/3424301841 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc9180cccf0 0x7fc9180cd100 unknown :-1 s=CLOSED pgs=2151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:49.481 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:49.478+0000 7fc920000640 1 -- 172.21.15.32:0/3424301841 >> 172.21.15.32:0/3424301841 conn(0x7fc9180b1bd0 msgr2=0x7fc9180b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T21:00:49.481 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:49.478+0000 7fc920000640 1 -- 172.21.15.32:0/3424301841 shutdown_connections 2024-08-03T21:00:49.482 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:49.479+0000 7fc920000640 1 -- 172.21.15.32:0/3424301841 wait complete. 2024-08-03T21:00:49.482 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:49.481+0000 7fc920000640 1 Processor -- start 2024-08-03T21:00:49.482 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:49.481+0000 7fc920000640 1 -- start start 2024-08-03T21:00:49.485 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:49.482+0000 7fc920000640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc9180cccf0 0x7fc9181418b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:00:49.485 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:49.482+0000 7fc920000640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fc9180ced30 0x7fc91813bb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:00:49.485 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:49.482+0000 7fc920000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc918145c50 0x7fc91815e0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:00:49.485 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:49.482+0000 7fc920000640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fc9180b4070 con 0x7fc918145c50 2024-08-03T21:00:49.485 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:49.482+0000 7fc920000640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7fc9180b3ef0 con 0x7fc9180cccf0 2024-08-03T21:00:49.485 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:49.482+0000 7fc920000640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7fc9180b41f0 con 0x7fc9180ced30 2024-08-03T21:00:49.485 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:49.483+0000 7fc91e000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc918145c50 0x7fc91815e0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:00:49.485 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:49.483+0000 7fc91e000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc918145c50 0x7fc91815e0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:57146/0 (socket says 172.21.15.32:57146) 2024-08-03T21:00:49.485 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:49.483+0000 7fc91e000640 1 -- 172.21.15.32:0/2756028826 learned_addr learned my addr 172.21.15.32:0/2756028826 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T21:00:49.485 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:49.483+0000 7fc91d600640 1 --2- 172.21.15.32:0/2756028826 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc9180cccf0 0x7fc9181418b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:00:49.485 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:49.483+0000 7fc91cc00640 1 --2- 172.21.15.32:0/2756028826 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fc9180ced30 0x7fc91813bb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:00:49.486 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:49.483+0000 7fc91e000640 1 -- 172.21.15.32:0/2756028826 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fc9180ced30 msgr2=0x7fc91813bb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:00:49.486 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:49.483+0000 7fc91e000640 1 --2- 172.21.15.32:0/2756028826 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fc9180ced30 0x7fc91813bb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:49.486 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:49.483+0000 7fc91e000640 1 -- 172.21.15.32:0/2756028826 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc9180cccf0 msgr2=0x7fc9181418b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:00:49.486 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:49.483+0000 7fc91e000640 1 --2- 172.21.15.32:0/2756028826 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc9180cccf0 0x7fc9181418b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:49.486 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:49.483+0000 7fc91e000640 1 -- 172.21.15.32:0/2756028826 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc9181f87e0 con 0x7fc918145c50 2024-08-03T21:00:49.486 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:49.483+0000 7fc91e000640 1 --2- 172.21.15.32:0/2756028826 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc918145c50 0x7fc91815e0f0 secure :-1 s=READY pgs=2152 cs=0 l=1 rev1=1 crypto rx=0x7fc91000ea10 tx=0x7fc91000eed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T21:00:49.487 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:49.484+0000 7fc915600640 1 -- 172.21.15.32:0/2756028826 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc91000cde0 con 0x7fc918145c50 2024-08-03T21:00:49.487 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:49.484+0000 7fc915600640 1 -- 172.21.15.32:0/2756028826 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fc910004c10 con 0x7fc918145c50 2024-08-03T21:00:49.487 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:49.484+0000 7fc915600640 1 -- 172.21.15.32:0/2756028826 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc910005020 con 0x7fc918145c50 2024-08-03T21:00:49.487 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:49.484+0000 7fc920000640 1 -- 172.21.15.32:0/2756028826 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc9181f8ac0 con 0x7fc918145c50 2024-08-03T21:00:49.487 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:49.484+0000 7fc920000640 1 -- 172.21.15.32:0/2756028826 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc9181f9080 con 0x7fc918145c50 2024-08-03T21:00:49.488 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:49.485+0000 7fc920000640 1 -- 172.21.15.32:0/2756028826 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc9180b46e0 con 0x7fc918145c50 2024-08-03T21:00:49.488 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:49.486+0000 7fc915600640 1 -- 172.21.15.32:0/2756028826 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7fc9100051c0 con 0x7fc918145c50 2024-08-03T21:00:49.489 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:49.487+0000 7fc915600640 1 --2- 172.21.15.32:0/2756028826 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fc8f8077010 0x7fc8f8079520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:00:49.489 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:49.487+0000 7fc915600640 1 -- 172.21.15.32:0/2756028826 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(133..133 src has 1..133) ==== 5609+0+0 (secure 0 0 0) 0x7fc910099790 con 0x7fc918145c50 2024-08-03T21:00:49.489 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:49.488+0000 7fc91d600640 1 --2- 172.21.15.32:0/2756028826 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fc8f8077010 0x7fc8f8079520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:00:49.490 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:49.488+0000 7fc91d600640 1 --2- 172.21.15.32:0/2756028826 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fc8f8077010 0x7fc8f8079520 secure :-1 s=READY pgs=863 cs=0 l=1 rev1=1 crypto rx=0x7fc90c0015e0 tx=0x7fc90c009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T21:00:49.494 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:49.492+0000 7fc915600640 1 -- 172.21.15.32:0/2756028826 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7fc910062950 con 0x7fc918145c50 2024-08-03T21:00:49.742 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:49.741+0000 7fc920000640 1 -- 172.21.15.32:0/2756028826 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) -- 0x7fc91806d9f0 con 0x7fc918145c50 2024-08-03T21:00:50.205 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:50.204+0000 7fc915600640 1 -- 172.21.15.32:0/2756028826 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 14 and data pool 15 v336) ==== 174+0+0 (secure 0 0 0) 0x7fc910062250 con 0x7fc918145c50 2024-08-03T21:00:50.206 INFO:teuthology.orchestra.run.smithi032.stderr:new fs with metadata pool 14 and data pool 15 2024-08-03T21:00:50.210 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:50.208+0000 7fc920000640 1 -- 172.21.15.32:0/2756028826 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fc8f8077010 msgr2=0x7fc8f8079520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:00:50.210 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:50.208+0000 7fc920000640 1 --2- 172.21.15.32:0/2756028826 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fc8f8077010 0x7fc8f8079520 secure :-1 s=READY pgs=863 cs=0 l=1 rev1=1 crypto rx=0x7fc90c0015e0 tx=0x7fc90c009040 comp rx=0 tx=0).stop 2024-08-03T21:00:50.210 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:50.208+0000 7fc920000640 1 -- 172.21.15.32:0/2756028826 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc918145c50 msgr2=0x7fc91815e0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:00:50.210 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:50.208+0000 7fc920000640 1 --2- 172.21.15.32:0/2756028826 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc918145c50 0x7fc91815e0f0 secure :-1 s=READY pgs=2152 cs=0 l=1 rev1=1 crypto rx=0x7fc91000ea10 tx=0x7fc91000eed0 comp rx=0 tx=0).stop 2024-08-03T21:00:50.210 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:50.208+0000 7fc920000640 1 -- 172.21.15.32:0/2756028826 shutdown_connections 2024-08-03T21:00:50.210 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:50.208+0000 7fc920000640 1 --2- 172.21.15.32:0/2756028826 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fc8f8077010 0x7fc8f8079520 unknown :-1 s=CLOSED pgs=863 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:50.210 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:50.208+0000 7fc920000640 1 --2- 172.21.15.32:0/2756028826 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc918145c50 0x7fc91815e0f0 unknown :-1 s=CLOSED pgs=2152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:50.211 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:50.208+0000 7fc920000640 1 --2- 172.21.15.32:0/2756028826 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fc9180ced30 0x7fc91813bb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:50.211 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:50.208+0000 7fc920000640 1 --2- 172.21.15.32:0/2756028826 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc9180cccf0 0x7fc9181418b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:50.211 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:50.208+0000 7fc920000640 1 -- 172.21.15.32:0/2756028826 >> 172.21.15.32:0/2756028826 conn(0x7fc9180b1bd0 msgr2=0x7fc9180b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T21:00:50.211 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:50.209+0000 7fc920000640 1 -- 172.21.15.32:0/2756028826 shutdown_connections 2024-08-03T21:00:50.211 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:50.209+0000 7fc920000640 1 -- 172.21.15.32:0/2756028826 wait complete. 2024-08-03T21:00:50.265 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-08-03T21:00:50.496 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:50.494+0000 7fa09de00640 1 -- 172.21.15.32:0/2666191954 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa07c004800 msgr2=0x7fa07c004c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:00:50.496 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:50.494+0000 7fa09de00640 1 --2- 172.21.15.32:0/2666191954 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa07c004800 0x7fa07c004c10 secure :-1 s=READY pgs=2154 cs=0 l=1 rev1=1 crypto rx=0x7fa080006f90 tx=0x7fa07c005830 comp rx=0 tx=0).stop 2024-08-03T21:00:50.496 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:50.494+0000 7fa09de00640 1 -- 172.21.15.32:0/2666191954 shutdown_connections 2024-08-03T21:00:50.496 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:50.494+0000 7fa09de00640 1 --2- 172.21.15.32:0/2666191954 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa07c004800 0x7fa07c004c10 unknown :-1 s=CLOSED pgs=2154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:50.496 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:50.494+0000 7fa09de00640 1 -- 172.21.15.32:0/2666191954 >> 172.21.15.32:0/2666191954 conn(0x7fa0980b2120 msgr2=0x7fa0980cfd60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T21:00:50.496 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:50.494+0000 7fa09de00640 1 -- 172.21.15.32:0/2666191954 shutdown_connections 2024-08-03T21:00:50.496 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:50.494+0000 7fa09de00640 1 -- 172.21.15.32:0/2666191954 wait complete. 2024-08-03T21:00:50.497 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:50.496+0000 7fa09de00640 1 Processor -- start 2024-08-03T21:00:50.498 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:50.496+0000 7fa09de00640 1 -- start start 2024-08-03T21:00:50.499 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:50.498+0000 7fa09de00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa07c004800 0x7fa0980b5110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:00:50.500 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:50.498+0000 7fa09de00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa09814e250 0x7fa098072cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:00:50.500 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:50.498+0000 7fa09de00640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fa0980731f0 0x7fa0981f6b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:00:50.500 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:50.498+0000 7fa09de00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fa0980ca7c0 con 0x7fa07c004800 2024-08-03T21:00:50.500 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:50.498+0000 7fa09de00640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7fa0980708f0 con 0x7fa09814e250 2024-08-03T21:00:50.500 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:50.498+0000 7fa09de00640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7fa0980ca940 con 0x7fa0980731f0 2024-08-03T21:00:50.500 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:50.498+0000 7fa09ca00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa07c004800 0x7fa0980b5110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:00:50.500 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:50.498+0000 7fa09ca00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa07c004800 0x7fa0980b5110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:49988/0 (socket says 172.21.15.32:49988) 2024-08-03T21:00:50.500 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:50.498+0000 7fa09ca00640 1 -- 172.21.15.32:0/3761585617 learned_addr learned my addr 172.21.15.32:0/3761585617 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T21:00:50.500 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:50.498+0000 7fa08fe00640 1 --2- 172.21.15.32:0/3761585617 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa09814e250 0x7fa098072cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:00:50.501 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:50.498+0000 7fa09d400640 1 --2- 172.21.15.32:0/3761585617 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fa0980731f0 0x7fa0981f6b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:00:50.502 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:50.498+0000 7fa09ca00640 1 -- 172.21.15.32:0/3761585617 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fa0980731f0 msgr2=0x7fa0981f6b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:00:50.502 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:50.498+0000 7fa09ca00640 1 --2- 172.21.15.32:0/3761585617 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fa0980731f0 0x7fa0981f6b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:50.502 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:50.498+0000 7fa09ca00640 1 -- 172.21.15.32:0/3761585617 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa09814e250 msgr2=0x7fa098072cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:00:50.502 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:50.499+0000 7fa09ca00640 1 --2- 172.21.15.32:0/3761585617 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa09814e250 0x7fa098072cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:50.502 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:50.499+0000 7fa09ca00640 1 -- 172.21.15.32:0/3761585617 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa098073600 con 0x7fa07c004800 2024-08-03T21:00:50.502 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:50.499+0000 7fa09d400640 1 --2- 172.21.15.32:0/3761585617 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fa0980731f0 0x7fa0981f6b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-03T21:00:50.502 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:50.499+0000 7fa08fe00640 1 --2- 172.21.15.32:0/3761585617 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa09814e250 0x7fa098072cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-03T21:00:50.502 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:50.499+0000 7fa09ca00640 1 --2- 172.21.15.32:0/3761585617 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa07c004800 0x7fa0980b5110 secure :-1 s=READY pgs=2155 cs=0 l=1 rev1=1 crypto rx=0x7fa08800ea20 tx=0x7fa08800eee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T21:00:50.502 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:50.500+0000 7fa095600640 1 -- 172.21.15.32:0/3761585617 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa08800dc10 con 0x7fa07c004800 2024-08-03T21:00:50.502 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:50.500+0000 7fa095600640 1 -- 172.21.15.32:0/3761585617 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fa08800ddb0 con 0x7fa07c004800 2024-08-03T21:00:50.502 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:50.500+0000 7fa095600640 1 -- 172.21.15.32:0/3761585617 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0880042b0 con 0x7fa07c004800 2024-08-03T21:00:50.503 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:50.500+0000 7fa09de00640 1 -- 172.21.15.32:0/3761585617 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa0980b3890 con 0x7fa07c004800 2024-08-03T21:00:50.503 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:50.500+0000 7fa09de00640 1 -- 172.21.15.32:0/3761585617 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa09809aaf0 con 0x7fa07c004800 2024-08-03T21:00:50.503 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:50.501+0000 7fa09de00640 1 -- 172.21.15.32:0/3761585617 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa0980aefb0 con 0x7fa07c004800 2024-08-03T21:00:50.504 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:50.502+0000 7fa095600640 1 -- 172.21.15.32:0/3761585617 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7fa0880035f0 con 0x7fa07c004800 2024-08-03T21:00:50.504 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:50.503+0000 7fa095600640 1 --2- 172.21.15.32:0/3761585617 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fa074077010 0x7fa074079520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:00:50.505 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:50.503+0000 7fa095600640 1 -- 172.21.15.32:0/3761585617 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(134..134 src has 1..134) ==== 5709+0+0 (secure 0 0 0) 0x7fa088099350 con 0x7fa07c004800 2024-08-03T21:00:50.505 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:50.504+0000 7fa08fe00640 1 --2- 172.21.15.32:0/3761585617 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fa074077010 0x7fa074079520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:00:50.506 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:50.505+0000 7fa08fe00640 1 --2- 172.21.15.32:0/3761585617 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fa074077010 0x7fa074079520 secure :-1 s=READY pgs=864 cs=0 l=1 rev1=1 crypto rx=0x7fa080033000 tx=0x7fa09806f3d0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T21:00:50.509 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:50.508+0000 7fa095600640 1 -- 172.21.15.32:0/3761585617 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7fa098073600 con 0x7fa07c004800 2024-08-03T21:00:50.712 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:50.711+0000 7fa09de00640 1 -- 172.21.15.32:0/3761585617 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fa0980b4ed0 con 0x7fa07c004800 2024-08-03T21:00:50.714 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:50.712+0000 7fa095600640 1 -- 172.21.15.32:0/3761585617 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v134) ==== 74+0+16572 (secure 0 0 0) 0x7fa088062550 con 0x7fa07c004800 2024-08-03T21:00:50.714 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-03T21:00:50.715 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":134,"fsid":"40ab141a-01d7-49b9-ae3c-2edec9876e95","created":"2024-08-03T20:47:14.957362+0000","modified":"2024-08-03T21:00:49.742894+0000","last_up_change":"2024-08-03T20:47:28.874451+0000","last_in_change":"2024-08-03T20:47:20.079442+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":15,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-08-03T20:47:28.261300+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":14,"pool_name":"cephfs_metadata","create_time":"2024-08-03T21:00:47.705081+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"134","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":15,"pool_name":"cephfs_data","create_time":"2024-08-03T21:00:48.703409+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"134","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5a83e759-9035-438b-889a-0a6af1f0a0d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":133,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6816","nonce":3066156116},{"type":"v1","addr":"172.21.15.32:6817","nonce":3066156116}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6818","nonce":3066156116},{"type":"v1","addr":"172.21.15.32:6819","nonce":3066156116}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6822","nonce":3066156116},{"type":"v1","addr":"172.21.15.32:6823","nonce":3066156116}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6820","nonce":3066156116},{"type":"v1","addr":"172.21.15.32:6821","nonce":3066156116}]},"public_addr":"172.21.15.32:6817/3066156116","cluster_addr":"172.21.15.32:6819/3066156116","heartbeat_back_addr":"172.21.15.32:6823/3066156116","heartbeat_front_addr":"172.21.15.32:6821/3066156116","state":["exists","up"]},{"osd":1,"uuid":"d7c5c210-da4f-49f9-b9e7-1dfb02daa4bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":133,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6824","nonce":1551241759},{"type":"v1","addr":"172.21.15.32:6825","nonce":1551241759}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6826","nonce":1551241759},{"type":"v1","addr":"172.21.15.32:6827","nonce":1551241759}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6830","nonce":1551241759},{"type":"v1","addr":"172.21.15.32:6831","nonce":1551241759}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6828","nonce":1551241759},{"type":"v1","addr":"172.21.15.32:6829","nonce":1551241759}]},"public_addr":"172.21.15.32:6825/1551241759","cluster_addr":"172.21.15.32:6827/1551241759","heartbeat_back_addr":"172.21.15.32:6831/1551241759","heartbeat_front_addr":"172.21.15.32:6829/1551241759","state":["exists","up"]},{"osd":2,"uuid":"57f2e6de-a201-45c2-8981-8e56a4875ebd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":133,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6808","nonce":3806708944},{"type":"v1","addr":"172.21.15.32:6809","nonce":3806708944}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6810","nonce":3806708944},{"type":"v1","addr":"172.21.15.32:6811","nonce":3806708944}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6814","nonce":3806708944},{"type":"v1","addr":"172.21.15.32:6815","nonce":3806708944}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6812","nonce":3806708944},{"type":"v1","addr":"172.21.15.32:6813","nonce":3806708944}]},"public_addr":"172.21.15.32:6809/3806708944","cluster_addr":"172.21.15.32:6811/3806708944","heartbeat_back_addr":"172.21.15.32:6815/3806708944","heartbeat_front_addr":"172.21.15.32:6813/3806708944","state":["exists","up"]},{"osd":3,"uuid":"6439a280-da45-4148-8130-f16323935ff3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":133,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6800","nonce":9088303},{"type":"v1","addr":"172.21.15.32:6801","nonce":9088303}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6802","nonce":9088303},{"type":"v1","addr":"172.21.15.32:6803","nonce":9088303}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6806","nonce":9088303},{"type":"v1","addr":"172.21.15.32:6807","nonce":9088303}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6804","nonce":9088303},{"type":"v1","addr":"172.21.15.32:6805","nonce":9088303}]},"public_addr":"172.21.15.32:6801/9088303","cluster_addr":"172.21.15.32:6803/9088303","heartbeat_back_addr":"172.21.15.32:6807/9088303","heartbeat_front_addr":"172.21.15.32:6805/9088303","state":["exists","up"]},{"osd":4,"uuid":"4f902eff-1e18-418d-82c9-2151c69524fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":133,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6802","nonce":864752275},{"type":"v1","addr":"172.21.15.118:6803","nonce":864752275}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6804","nonce":864752275},{"type":"v1","addr":"172.21.15.118:6805","nonce":864752275}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6808","nonce":864752275},{"type":"v1","addr":"172.21.15.118:6809","nonce":864752275}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6806","nonce":864752275},{"type":"v1","addr":"172.21.15.118:6807","nonce":864752275}]},"public_addr":"172.21.15.118:6803/864752275","cluster_addr":"172.21.15.118:6805/864752275","heartbeat_back_addr":"172.21.15.118:6809/864752275","heartbeat_front_addr":"172.21.15.118:6807/864752275","state":["exists","up"]},{"osd":5,"uuid":"bfd9feac-dd06-458c-91b9-af7f175f612c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":133,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6810","nonce":2724891770},{"type":"v1","addr":"172.21.15.118:6811","nonce":2724891770}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6812","nonce":2724891770},{"type":"v1","addr":"172.21.15.118:6813","nonce":2724891770}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6816","nonce":2724891770},{"type":"v1","addr":"172.21.15.118:6817","nonce":2724891770}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6814","nonce":2724891770},{"type":"v1","addr":"172.21.15.118:6815","nonce":2724891770}]},"public_addr":"172.21.15.118:6811/2724891770","cluster_addr":"172.21.15.118:6813/2724891770","heartbeat_back_addr":"172.21.15.118:6817/2724891770","heartbeat_front_addr":"172.21.15.118:6815/2724891770","state":["exists","up"]},{"osd":6,"uuid":"5ed5a279-9d8e-416d-bbc7-7edd3e6d5705","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":133,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6825","nonce":801947386},{"type":"v1","addr":"172.21.15.118:6827","nonce":801947386}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6828","nonce":801947386},{"type":"v1","addr":"172.21.15.118:6829","nonce":801947386}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6832","nonce":801947386},{"type":"v1","addr":"172.21.15.118:6833","nonce":801947386}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6830","nonce":801947386},{"type":"v1","addr":"172.21.15.118:6831","nonce":801947386}]},"public_addr":"172.21.15.118:6827/801947386","cluster_addr":"172.21.15.118:6829/801947386","heartbeat_back_addr":"172.21.15.118:6833/801947386","heartbeat_front_addr":"172.21.15.118:6831/801947386","state":["exists","up"]},{"osd":7,"uuid":"f2086c47-875e-4451-a436-2008d1ea16af","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":133,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6818","nonce":653000048},{"type":"v1","addr":"172.21.15.118:6819","nonce":653000048}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6820","nonce":653000048},{"type":"v1","addr":"172.21.15.118:6821","nonce":653000048}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6824","nonce":653000048},{"type":"v1","addr":"172.21.15.118:6826","nonce":653000048}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6822","nonce":653000048},{"type":"v1","addr":"172.21.15.118:6823","nonce":653000048}]},"public_addr":"172.21.15.118:6819/653000048","cluster_addr":"172.21.15.118:6821/653000048","heartbeat_back_addr":"172.21.15.118:6826/653000048","heartbeat_front_addr":"172.21.15.118:6823/653000048","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-03T20:47:25.491994+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-03T20:47:25.587358+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-03T20:47:25.454175+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-03T20:47:25.377734+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-03T20:47:25.412351+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-03T20:47:25.499815+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-03T20:47:25.575044+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-03T20:47:25.552383+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-08-03T21:00:50.716 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:50.714+0000 7fa09de00640 1 -- 172.21.15.32:0/3761585617 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fa074077010 msgr2=0x7fa074079520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:00:50.716 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:50.715+0000 7fa09de00640 1 --2- 172.21.15.32:0/3761585617 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fa074077010 0x7fa074079520 secure :-1 s=READY pgs=864 cs=0 l=1 rev1=1 crypto rx=0x7fa080033000 tx=0x7fa09806f3d0 comp rx=0 tx=0).stop 2024-08-03T21:00:50.717 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:50.715+0000 7fa09de00640 1 -- 172.21.15.32:0/3761585617 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa07c004800 msgr2=0x7fa0980b5110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:00:50.717 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:50.715+0000 7fa09de00640 1 --2- 172.21.15.32:0/3761585617 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa07c004800 0x7fa0980b5110 secure :-1 s=READY pgs=2155 cs=0 l=1 rev1=1 crypto rx=0x7fa08800ea20 tx=0x7fa08800eee0 comp rx=0 tx=0).stop 2024-08-03T21:00:50.717 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:50.715+0000 7fa09de00640 1 -- 172.21.15.32:0/3761585617 shutdown_connections 2024-08-03T21:00:50.717 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:50.715+0000 7fa09de00640 1 --2- 172.21.15.32:0/3761585617 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fa074077010 0x7fa074079520 unknown :-1 s=CLOSED pgs=864 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:50.717 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:50.715+0000 7fa09de00640 1 --2- 172.21.15.32:0/3761585617 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa09814e250 0x7fa098072cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:50.717 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:50.715+0000 7fa09de00640 1 --2- 172.21.15.32:0/3761585617 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fa0980731f0 0x7fa0981f6b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:50.717 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:50.715+0000 7fa09de00640 1 --2- 172.21.15.32:0/3761585617 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa07c004800 0x7fa0980b5110 unknown :-1 s=CLOSED pgs=2155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:50.717 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:50.715+0000 7fa09de00640 1 -- 172.21.15.32:0/3761585617 >> 172.21.15.32:0/3761585617 conn(0x7fa0980b2120 msgr2=0x7fa098157320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T21:00:50.717 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:50.715+0000 7fa09de00640 1 -- 172.21.15.32:0/3761585617 shutdown_connections 2024-08-03T21:00:50.717 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:50.715+0000 7fa09de00640 1 -- 172.21.15.32:0/3761585617 wait complete. 2024-08-03T21:00:50.740 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-08-03T21:00:50.969 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:50.967+0000 7fcc4cc00640 1 -- 172.21.15.32:0/1637460286 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcc4806fb90 msgr2=0x7fcc480ca3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:00:50.969 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:50.967+0000 7fcc4cc00640 1 --2- 172.21.15.32:0/1637460286 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcc4806fb90 0x7fcc480ca3b0 secure :-1 s=READY pgs=2157 cs=0 l=1 rev1=1 crypto rx=0x7fcc40005fd0 tx=0x7fcc4002ab40 comp rx=0 tx=0).stop 2024-08-03T21:00:50.969 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:50.967+0000 7fcc4cc00640 1 -- 172.21.15.32:0/1637460286 shutdown_connections 2024-08-03T21:00:50.969 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:50.967+0000 7fcc4cc00640 1 --2- 172.21.15.32:0/1637460286 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcc4806fb90 0x7fcc480ca3b0 unknown :-1 s=CLOSED pgs=2157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:50.969 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:50.967+0000 7fcc4cc00640 1 -- 172.21.15.32:0/1637460286 >> 172.21.15.32:0/1637460286 conn(0x7fcc480b2120 msgr2=0x7fcc480cfd60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T21:00:50.969 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:50.967+0000 7fcc4cc00640 1 -- 172.21.15.32:0/1637460286 shutdown_connections 2024-08-03T21:00:50.969 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:50.967+0000 7fcc4cc00640 1 -- 172.21.15.32:0/1637460286 wait complete. 2024-08-03T21:00:50.970 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:50.969+0000 7fcc4cc00640 1 Processor -- start 2024-08-03T21:00:50.971 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:50.969+0000 7fcc4cc00640 1 -- start start 2024-08-03T21:00:50.973 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:50.971+0000 7fcc4cc00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fcc480af040 0x7fcc4815dd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:00:50.973 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:50.971+0000 7fcc4cc00640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fcc480cee10 0x7fcc4814c210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:00:50.973 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:50.971+0000 7fcc4cc00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcc48159c50 0x7fcc480adf40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:00:50.973 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:50.971+0000 7fcc4cc00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fcc480caa70 con 0x7fcc48159c50 2024-08-03T21:00:50.973 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:50.971+0000 7fcc4cc00640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7fcc480ca8f0 con 0x7fcc480af040 2024-08-03T21:00:50.973 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:50.971+0000 7fcc4cc00640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7fcc480cabf0 con 0x7fcc480cee10 2024-08-03T21:00:50.974 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:50.971+0000 7fcc47e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcc48159c50 0x7fcc480adf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:00:50.974 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:50.971+0000 7fcc47e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcc48159c50 0x7fcc480adf40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:49992/0 (socket says 172.21.15.32:49992) 2024-08-03T21:00:50.974 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:50.971+0000 7fcc47e00640 1 -- 172.21.15.32:0/3115932896 learned_addr learned my addr 172.21.15.32:0/3115932896 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T21:00:50.974 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:50.971+0000 7fcc46a00640 1 --2- 172.21.15.32:0/3115932896 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fcc480cee10 0x7fcc4814c210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:00:50.974 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:50.971+0000 7fcc47400640 1 --2- 172.21.15.32:0/3115932896 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fcc480af040 0x7fcc4815dd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:00:50.974 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:50.971+0000 7fcc47e00640 1 -- 172.21.15.32:0/3115932896 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fcc480cee10 msgr2=0x7fcc4814c210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:00:50.974 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:50.971+0000 7fcc47e00640 1 --2- 172.21.15.32:0/3115932896 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fcc480cee10 0x7fcc4814c210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:50.974 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:50.971+0000 7fcc47e00640 1 -- 172.21.15.32:0/3115932896 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fcc480af040 msgr2=0x7fcc4815dd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:00:50.974 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:50.971+0000 7fcc47e00640 1 --2- 172.21.15.32:0/3115932896 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fcc480af040 0x7fcc4815dd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:50.974 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:50.971+0000 7fcc47e00640 1 -- 172.21.15.32:0/3115932896 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcc48154d20 con 0x7fcc48159c50 2024-08-03T21:00:50.974 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:50.972+0000 7fcc47e00640 1 --2- 172.21.15.32:0/3115932896 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcc48159c50 0x7fcc480adf40 secure :-1 s=READY pgs=2158 cs=0 l=1 rev1=1 crypto rx=0x7fcc40009d90 tx=0x7fcc4002c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T21:00:50.975 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:50.972+0000 7fcc3f400640 1 -- 172.21.15.32:0/3115932896 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc40044070 con 0x7fcc48159c50 2024-08-03T21:00:50.975 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:50.972+0000 7fcc3f400640 1 -- 172.21.15.32:0/3115932896 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fcc40004250 con 0x7fcc48159c50 2024-08-03T21:00:50.975 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:50.972+0000 7fcc4cc00640 1 -- 172.21.15.32:0/3115932896 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcc4815a460 con 0x7fcc48159c50 2024-08-03T21:00:50.975 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:50.972+0000 7fcc3f400640 1 -- 172.21.15.32:0/3115932896 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc400046a0 con 0x7fcc48159c50 2024-08-03T21:00:50.975 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:50.972+0000 7fcc4cc00640 1 -- 172.21.15.32:0/3115932896 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcc481e4bc0 con 0x7fcc48159c50 2024-08-03T21:00:50.976 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:50.973+0000 7fcc4cc00640 1 -- 172.21.15.32:0/3115932896 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcc48154ef0 con 0x7fcc48159c50 2024-08-03T21:00:50.976 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:50.975+0000 7fcc3f400640 1 -- 172.21.15.32:0/3115932896 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7fcc4003f410 con 0x7fcc48159c50 2024-08-03T21:00:50.977 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:50.976+0000 7fcc3f400640 1 --2- 172.21.15.32:0/3115932896 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fcc28077010 0x7fcc28079520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:00:50.977 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:50.976+0000 7fcc3f400640 1 -- 172.21.15.32:0/3115932896 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(134..134 src has 1..134) ==== 5709+0+0 (secure 0 0 0) 0x7fcc400b9520 con 0x7fcc48159c50 2024-08-03T21:00:50.978 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:50.976+0000 7fcc47400640 1 --2- 172.21.15.32:0/3115932896 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fcc28077010 0x7fcc28079520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:00:50.978 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:50.977+0000 7fcc47400640 1 --2- 172.21.15.32:0/3115932896 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fcc28077010 0x7fcc28079520 secure :-1 s=READY pgs=865 cs=0 l=1 rev1=1 crypto rx=0x7fcc38000f60 tx=0x7fcc38009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T21:00:50.980 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:50.979+0000 7fcc3f400640 1 -- 172.21.15.32:0/3115932896 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7fcc40032030 con 0x7fcc48159c50 2024-08-03T21:00:51.186 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:51.185+0000 7fcc4cc00640 1 -- 172.21.15.32:0/3115932896 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fcc480ca3b0 con 0x7fcc48159c50 2024-08-03T21:00:51.187 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:51.186+0000 7fcc3f400640 1 -- 172.21.15.32:0/3115932896 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v134) ==== 74+0+16572 (secure 0 0 0) 0x7fcc40082790 con 0x7fcc48159c50 2024-08-03T21:00:51.188 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-03T21:00:51.189 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":134,"fsid":"40ab141a-01d7-49b9-ae3c-2edec9876e95","created":"2024-08-03T20:47:14.957362+0000","modified":"2024-08-03T21:00:49.742894+0000","last_up_change":"2024-08-03T20:47:28.874451+0000","last_in_change":"2024-08-03T20:47:20.079442+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":15,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-08-03T20:47:28.261300+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":14,"pool_name":"cephfs_metadata","create_time":"2024-08-03T21:00:47.705081+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"134","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":15,"pool_name":"cephfs_data","create_time":"2024-08-03T21:00:48.703409+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"134","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5a83e759-9035-438b-889a-0a6af1f0a0d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":133,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6816","nonce":3066156116},{"type":"v1","addr":"172.21.15.32:6817","nonce":3066156116}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6818","nonce":3066156116},{"type":"v1","addr":"172.21.15.32:6819","nonce":3066156116}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6822","nonce":3066156116},{"type":"v1","addr":"172.21.15.32:6823","nonce":3066156116}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6820","nonce":3066156116},{"type":"v1","addr":"172.21.15.32:6821","nonce":3066156116}]},"public_addr":"172.21.15.32:6817/3066156116","cluster_addr":"172.21.15.32:6819/3066156116","heartbeat_back_addr":"172.21.15.32:6823/3066156116","heartbeat_front_addr":"172.21.15.32:6821/3066156116","state":["exists","up"]},{"osd":1,"uuid":"d7c5c210-da4f-49f9-b9e7-1dfb02daa4bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":133,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6824","nonce":1551241759},{"type":"v1","addr":"172.21.15.32:6825","nonce":1551241759}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6826","nonce":1551241759},{"type":"v1","addr":"172.21.15.32:6827","nonce":1551241759}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6830","nonce":1551241759},{"type":"v1","addr":"172.21.15.32:6831","nonce":1551241759}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6828","nonce":1551241759},{"type":"v1","addr":"172.21.15.32:6829","nonce":1551241759}]},"public_addr":"172.21.15.32:6825/1551241759","cluster_addr":"172.21.15.32:6827/1551241759","heartbeat_back_addr":"172.21.15.32:6831/1551241759","heartbeat_front_addr":"172.21.15.32:6829/1551241759","state":["exists","up"]},{"osd":2,"uuid":"57f2e6de-a201-45c2-8981-8e56a4875ebd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":133,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6808","nonce":3806708944},{"type":"v1","addr":"172.21.15.32:6809","nonce":3806708944}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6810","nonce":3806708944},{"type":"v1","addr":"172.21.15.32:6811","nonce":3806708944}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6814","nonce":3806708944},{"type":"v1","addr":"172.21.15.32:6815","nonce":3806708944}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6812","nonce":3806708944},{"type":"v1","addr":"172.21.15.32:6813","nonce":3806708944}]},"public_addr":"172.21.15.32:6809/3806708944","cluster_addr":"172.21.15.32:6811/3806708944","heartbeat_back_addr":"172.21.15.32:6815/3806708944","heartbeat_front_addr":"172.21.15.32:6813/3806708944","state":["exists","up"]},{"osd":3,"uuid":"6439a280-da45-4148-8130-f16323935ff3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":133,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6800","nonce":9088303},{"type":"v1","addr":"172.21.15.32:6801","nonce":9088303}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6802","nonce":9088303},{"type":"v1","addr":"172.21.15.32:6803","nonce":9088303}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6806","nonce":9088303},{"type":"v1","addr":"172.21.15.32:6807","nonce":9088303}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6804","nonce":9088303},{"type":"v1","addr":"172.21.15.32:6805","nonce":9088303}]},"public_addr":"172.21.15.32:6801/9088303","cluster_addr":"172.21.15.32:6803/9088303","heartbeat_back_addr":"172.21.15.32:6807/9088303","heartbeat_front_addr":"172.21.15.32:6805/9088303","state":["exists","up"]},{"osd":4,"uuid":"4f902eff-1e18-418d-82c9-2151c69524fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":133,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6802","nonce":864752275},{"type":"v1","addr":"172.21.15.118:6803","nonce":864752275}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6804","nonce":864752275},{"type":"v1","addr":"172.21.15.118:6805","nonce":864752275}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6808","nonce":864752275},{"type":"v1","addr":"172.21.15.118:6809","nonce":864752275}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6806","nonce":864752275},{"type":"v1","addr":"172.21.15.118:6807","nonce":864752275}]},"public_addr":"172.21.15.118:6803/864752275","cluster_addr":"172.21.15.118:6805/864752275","heartbeat_back_addr":"172.21.15.118:6809/864752275","heartbeat_front_addr":"172.21.15.118:6807/864752275","state":["exists","up"]},{"osd":5,"uuid":"bfd9feac-dd06-458c-91b9-af7f175f612c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":133,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6810","nonce":2724891770},{"type":"v1","addr":"172.21.15.118:6811","nonce":2724891770}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6812","nonce":2724891770},{"type":"v1","addr":"172.21.15.118:6813","nonce":2724891770}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6816","nonce":2724891770},{"type":"v1","addr":"172.21.15.118:6817","nonce":2724891770}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6814","nonce":2724891770},{"type":"v1","addr":"172.21.15.118:6815","nonce":2724891770}]},"public_addr":"172.21.15.118:6811/2724891770","cluster_addr":"172.21.15.118:6813/2724891770","heartbeat_back_addr":"172.21.15.118:6817/2724891770","heartbeat_front_addr":"172.21.15.118:6815/2724891770","state":["exists","up"]},{"osd":6,"uuid":"5ed5a279-9d8e-416d-bbc7-7edd3e6d5705","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":133,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6825","nonce":801947386},{"type":"v1","addr":"172.21.15.118:6827","nonce":801947386}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6828","nonce":801947386},{"type":"v1","addr":"172.21.15.118:6829","nonce":801947386}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6832","nonce":801947386},{"type":"v1","addr":"172.21.15.118:6833","nonce":801947386}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6830","nonce":801947386},{"type":"v1","addr":"172.21.15.118:6831","nonce":801947386}]},"public_addr":"172.21.15.118:6827/801947386","cluster_addr":"172.21.15.118:6829/801947386","heartbeat_back_addr":"172.21.15.118:6833/801947386","heartbeat_front_addr":"172.21.15.118:6831/801947386","state":["exists","up"]},{"osd":7,"uuid":"f2086c47-875e-4451-a436-2008d1ea16af","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":133,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6818","nonce":653000048},{"type":"v1","addr":"172.21.15.118:6819","nonce":653000048}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6820","nonce":653000048},{"type":"v1","addr":"172.21.15.118:6821","nonce":653000048}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6824","nonce":653000048},{"type":"v1","addr":"172.21.15.118:6826","nonce":653000048}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6822","nonce":653000048},{"type":"v1","addr":"172.21.15.118:6823","nonce":653000048}]},"public_addr":"172.21.15.118:6819/653000048","cluster_addr":"172.21.15.118:6821/653000048","heartbeat_back_addr":"172.21.15.118:6826/653000048","heartbeat_front_addr":"172.21.15.118:6823/653000048","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-03T20:47:25.491994+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-03T20:47:25.587358+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-03T20:47:25.454175+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-03T20:47:25.377734+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-03T20:47:25.412351+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-03T20:47:25.499815+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-03T20:47:25.575044+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-03T20:47:25.552383+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-08-03T21:00:51.190 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:51.188+0000 7fcc4cc00640 1 -- 172.21.15.32:0/3115932896 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fcc28077010 msgr2=0x7fcc28079520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:00:51.191 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:51.188+0000 7fcc4cc00640 1 --2- 172.21.15.32:0/3115932896 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fcc28077010 0x7fcc28079520 secure :-1 s=READY pgs=865 cs=0 l=1 rev1=1 crypto rx=0x7fcc38000f60 tx=0x7fcc38009040 comp rx=0 tx=0).stop 2024-08-03T21:00:51.191 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:51.188+0000 7fcc4cc00640 1 -- 172.21.15.32:0/3115932896 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcc48159c50 msgr2=0x7fcc480adf40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:00:51.191 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:51.188+0000 7fcc4cc00640 1 --2- 172.21.15.32:0/3115932896 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcc48159c50 0x7fcc480adf40 secure :-1 s=READY pgs=2158 cs=0 l=1 rev1=1 crypto rx=0x7fcc40009d90 tx=0x7fcc4002c8b0 comp rx=0 tx=0).stop 2024-08-03T21:00:51.191 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:51.188+0000 7fcc4cc00640 1 -- 172.21.15.32:0/3115932896 shutdown_connections 2024-08-03T21:00:51.191 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:51.188+0000 7fcc4cc00640 1 --2- 172.21.15.32:0/3115932896 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fcc28077010 0x7fcc28079520 unknown :-1 s=CLOSED pgs=865 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:51.191 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:51.188+0000 7fcc4cc00640 1 --2- 172.21.15.32:0/3115932896 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcc48159c50 0x7fcc480adf40 unknown :-1 s=CLOSED pgs=2158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:51.191 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:51.188+0000 7fcc4cc00640 1 --2- 172.21.15.32:0/3115932896 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fcc480cee10 0x7fcc4814c210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:51.191 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:51.188+0000 7fcc4cc00640 1 --2- 172.21.15.32:0/3115932896 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fcc480af040 0x7fcc4815dd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:51.192 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:51.188+0000 7fcc4cc00640 1 -- 172.21.15.32:0/3115932896 >> 172.21.15.32:0/3115932896 conn(0x7fcc480b2120 msgr2=0x7fcc4815f460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T21:00:51.192 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:51.188+0000 7fcc4cc00640 1 -- 172.21.15.32:0/3115932896 shutdown_connections 2024-08-03T21:00:51.192 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:51.188+0000 7fcc4cc00640 1 -- 172.21.15.32:0/3115932896 wait complete. 2024-08-03T21:00:51.218 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-08-03T21:00:51.451 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:51.449+0000 7f2793c00640 1 -- 172.21.15.32:0/239008483 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f278c145c50 msgr2=0x7f278c06e950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:00:51.451 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:51.449+0000 7f2793c00640 1 --2- 172.21.15.32:0/239008483 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f278c145c50 0x7f278c06e950 secure :-1 s=READY pgs=2160 cs=0 l=1 rev1=1 crypto rx=0x7f277c005fd0 tx=0x7f277c02a540 comp rx=0 tx=0).stop 2024-08-03T21:00:51.451 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:51.449+0000 7f2793c00640 1 -- 172.21.15.32:0/239008483 shutdown_connections 2024-08-03T21:00:51.451 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:51.449+0000 7f2793c00640 1 --2- 172.21.15.32:0/239008483 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f278c145c50 0x7f278c06e950 unknown :-1 s=CLOSED pgs=2160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:51.451 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:51.449+0000 7f2793c00640 1 -- 172.21.15.32:0/239008483 >> 172.21.15.32:0/239008483 conn(0x7f278c0b1bd0 msgr2=0x7f278c0b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T21:00:51.452 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:51.449+0000 7f2793c00640 1 -- 172.21.15.32:0/239008483 shutdown_connections 2024-08-03T21:00:51.452 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:51.449+0000 7f2793c00640 1 -- 172.21.15.32:0/239008483 wait complete. 2024-08-03T21:00:51.453 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:51.451+0000 7f2793c00640 1 Processor -- start 2024-08-03T21:00:51.453 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:51.452+0000 7f2793c00640 1 -- start start 2024-08-03T21:00:51.455 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:51.453+0000 7f2793c00640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f278c0cccf0 0x7f278c13bc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:00:51.455 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:51.453+0000 7f2793c00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f278c0ced30 0x7f278c157a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:00:51.455 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:51.453+0000 7f2793c00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f278c145c50 0x7f278c1e94e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:00:51.456 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:51.453+0000 7f2793c00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f278c0b4210 con 0x7f278c145c50 2024-08-03T21:00:51.456 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:51.453+0000 7f2793c00640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f278c0b4090 con 0x7f278c0ced30 2024-08-03T21:00:51.456 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:51.453+0000 7f2793c00640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7f278c0b4390 con 0x7f278c0cccf0 2024-08-03T21:00:51.456 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:51.453+0000 7f2791200640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f278c0cccf0 0x7f278c13bc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:00:51.456 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:51.453+0000 7f2791c00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f278c145c50 0x7f278c1e94e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:00:51.456 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:51.453+0000 7f2791200640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f278c0cccf0 0x7f278c13bc60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3301/0 says I am v2:172.21.15.32:46334/0 (socket says 172.21.15.32:46334) 2024-08-03T21:00:51.456 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:51.453+0000 7f2791200640 1 -- 172.21.15.32:0/3752173822 learned_addr learned my addr 172.21.15.32:0/3752173822 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T21:00:51.456 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:51.453+0000 7f2783e00640 1 --2- 172.21.15.32:0/3752173822 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f278c0ced30 0x7f278c157a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:00:51.456 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:51.454+0000 7f2791c00640 1 -- 172.21.15.32:0/3752173822 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f278c0cccf0 msgr2=0x7f278c13bc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:00:51.456 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:51.454+0000 7f2791c00640 1 --2- 172.21.15.32:0/3752173822 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f278c0cccf0 0x7f278c13bc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:51.456 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:51.454+0000 7f2791c00640 1 -- 172.21.15.32:0/3752173822 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f278c0ced30 msgr2=0x7f278c157a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:00:51.456 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:51.454+0000 7f2791c00640 1 --2- 172.21.15.32:0/3752173822 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f278c0ced30 0x7f278c157a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:51.456 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:51.454+0000 7f2791c00640 1 -- 172.21.15.32:0/3752173822 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f278c13c1a0 con 0x7f278c145c50 2024-08-03T21:00:51.456 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:51.454+0000 7f2791c00640 1 --2- 172.21.15.32:0/3752173822 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f278c145c50 0x7f278c1e94e0 secure :-1 s=READY pgs=2161 cs=0 l=1 rev1=1 crypto rx=0x7f278400c760 tx=0x7f278400cc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T21:00:51.457 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:51.454+0000 7f2789600640 1 -- 172.21.15.32:0/3752173822 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2784003b70 con 0x7f278c145c50 2024-08-03T21:00:51.457 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:51.454+0000 7f2789600640 1 -- 172.21.15.32:0/3752173822 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f2784003d10 con 0x7f278c145c50 2024-08-03T21:00:51.457 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:51.454+0000 7f2789600640 1 -- 172.21.15.32:0/3752173822 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2784005330 con 0x7f278c145c50 2024-08-03T21:00:51.457 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:51.454+0000 7f2793c00640 1 -- 172.21.15.32:0/3752173822 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f278c1e8440 con 0x7f278c145c50 2024-08-03T21:00:51.458 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:51.455+0000 7f2793c00640 1 -- 172.21.15.32:0/3752173822 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f278c1e8a00 con 0x7f278c145c50 2024-08-03T21:00:51.458 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:51.456+0000 7f2793c00640 1 -- 172.21.15.32:0/3752173822 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f278c0cdab0 con 0x7f278c145c50 2024-08-03T21:00:51.458 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:51.457+0000 7f2789600640 1 -- 172.21.15.32:0/3752173822 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f27840054d0 con 0x7f278c145c50 2024-08-03T21:00:51.460 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:51.458+0000 7f2789600640 1 --2- 172.21.15.32:0/3752173822 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f2768077010 0x7f2768079520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:00:51.460 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:51.459+0000 7f2789600640 1 -- 172.21.15.32:0/3752173822 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(135..135 src has 1..135) ==== 5709+0+0 (secure 0 0 0) 0x7f27840985b0 con 0x7f278c145c50 2024-08-03T21:00:51.460 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:51.459+0000 7f2791200640 1 --2- 172.21.15.32:0/3752173822 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f2768077010 0x7f2768079520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:00:51.461 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:51.460+0000 7f2791200640 1 --2- 172.21.15.32:0/3752173822 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f2768077010 0x7f2768079520 secure :-1 s=READY pgs=866 cs=0 l=1 rev1=1 crypto rx=0x7f277c009230 tx=0x7f277c004730 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T21:00:51.464 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:51.463+0000 7f2789600640 1 -- 172.21.15.32:0/3752173822 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7f278c0cdab0 con 0x7f278c145c50 2024-08-03T21:00:51.711 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:51.709+0000 7f2793c00640 1 -- 172.21.15.32:0/3752173822 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) -- 0x7f278c06e950 con 0x7f278c145c50 2024-08-03T21:00:52.192 INFO:tasks.ceph.mds.a.smithi032.stderr:2024-08-03T21:00:52.190+0000 7ffaa7000640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-08-03T21:00:52.231 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:52.230+0000 7f2789600640 1 -- 172.21.15.32:0/3752173822 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v339) ==== 118+0+0 (secure 0 0 0) 0x7f27840610c0 con 0x7f278c145c50 2024-08-03T21:00:52.234 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:52.232+0000 7f2793c00640 1 -- 172.21.15.32:0/3752173822 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f2768077010 msgr2=0x7f2768079520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:00:52.235 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:52.232+0000 7f2793c00640 1 --2- 172.21.15.32:0/3752173822 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f2768077010 0x7f2768079520 secure :-1 s=READY pgs=866 cs=0 l=1 rev1=1 crypto rx=0x7f277c009230 tx=0x7f277c004730 comp rx=0 tx=0).stop 2024-08-03T21:00:52.235 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:52.232+0000 7f2793c00640 1 -- 172.21.15.32:0/3752173822 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f278c145c50 msgr2=0x7f278c1e94e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:00:52.235 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:52.232+0000 7f2793c00640 1 --2- 172.21.15.32:0/3752173822 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f278c145c50 0x7f278c1e94e0 secure :-1 s=READY pgs=2161 cs=0 l=1 rev1=1 crypto rx=0x7f278400c760 tx=0x7f278400cc20 comp rx=0 tx=0).stop 2024-08-03T21:00:52.235 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:52.233+0000 7f2793c00640 1 -- 172.21.15.32:0/3752173822 shutdown_connections 2024-08-03T21:00:52.235 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:52.233+0000 7f2793c00640 1 --2- 172.21.15.32:0/3752173822 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f2768077010 0x7f2768079520 unknown :-1 s=CLOSED pgs=866 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:52.235 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:52.233+0000 7f2793c00640 1 --2- 172.21.15.32:0/3752173822 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f278c0ced30 0x7f278c157a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:52.235 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:52.233+0000 7f2793c00640 1 --2- 172.21.15.32:0/3752173822 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f278c145c50 0x7f278c1e94e0 unknown :-1 s=CLOSED pgs=2161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:52.235 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:52.233+0000 7f2793c00640 1 --2- 172.21.15.32:0/3752173822 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f278c0cccf0 0x7f278c13bc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:52.235 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:52.233+0000 7f2793c00640 1 -- 172.21.15.32:0/3752173822 >> 172.21.15.32:0/3752173822 conn(0x7f278c0b1bd0 msgr2=0x7f278c15c9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T21:00:52.235 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:52.233+0000 7f2793c00640 1 -- 172.21.15.32:0/3752173822 shutdown_connections 2024-08-03T21:00:52.235 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:52.233+0000 7f2793c00640 1 -- 172.21.15.32:0/3752173822 wait complete. 2024-08-03T21:00:52.259 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-08-03T21:00:52.259 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-08-03T21:00:52.492 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:52.490+0000 7f95ca800640 1 -- 172.21.15.32:0/638279749 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f95c41456f0 msgr2=0x7f95c4160860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:00:52.492 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:52.490+0000 7f95ca800640 1 --2- 172.21.15.32:0/638279749 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f95c41456f0 0x7f95c4160860 secure :-1 s=READY pgs=2163 cs=0 l=1 rev1=1 crypto rx=0x7f95c40b6470 tx=0x7f95b002a410 comp rx=0 tx=0).stop 2024-08-03T21:00:52.492 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:52.490+0000 7f95ca800640 1 -- 172.21.15.32:0/638279749 shutdown_connections 2024-08-03T21:00:52.492 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:52.490+0000 7f95ca800640 1 --2- 172.21.15.32:0/638279749 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f95c41456f0 0x7f95c4160860 unknown :-1 s=CLOSED pgs=2163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:52.492 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:52.490+0000 7f95ca800640 1 --2- 172.21.15.32:0/638279749 >> v2:172.21.15.118:3300/0 conn(0x7f95c4145cd0 0x7f95c4145100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:52.492 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:52.490+0000 7f95ca800640 1 -- 172.21.15.32:0/638279749 >> 172.21.15.32:0/638279749 conn(0x7f95c40b32f0 msgr2=0x7f95c40d1050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T21:00:52.492 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:52.490+0000 7f95ca800640 1 -- 172.21.15.32:0/638279749 shutdown_connections 2024-08-03T21:00:52.492 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:52.490+0000 7f95ca800640 1 -- 172.21.15.32:0/638279749 wait complete. 2024-08-03T21:00:52.493 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:52.492+0000 7f95ca800640 1 Processor -- start 2024-08-03T21:00:52.494 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:52.492+0000 7f95ca800640 1 -- start start 2024-08-03T21:00:52.496 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:52.494+0000 7f95ca800640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f95c406c9e0 0x7f95c40cc130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:00:52.496 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:52.494+0000 7f95ca800640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f95c41456f0 0x7f95c41f46b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:00:52.496 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:52.494+0000 7f95ca800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f95c4145cd0 0x7f95c415bc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:00:52.496 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:52.494+0000 7f95ca800640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f95c40cffc0 con 0x7f95c4145cd0 2024-08-03T21:00:52.496 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:52.494+0000 7f95ca800640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f95c4161010 con 0x7f95c406c9e0 2024-08-03T21:00:52.496 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:52.494+0000 7f95ca800640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7f95c40d0140 con 0x7f95c41456f0 2024-08-03T21:00:52.496 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:52.494+0000 7f95c3e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f95c4145cd0 0x7f95c415bc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:00:52.496 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:52.494+0000 7f95c3e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f95c4145cd0 0x7f95c415bc70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:50036/0 (socket says 172.21.15.32:50036) 2024-08-03T21:00:52.496 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:52.494+0000 7f95c3e00640 1 -- 172.21.15.32:0/128054689 learned_addr learned my addr 172.21.15.32:0/128054689 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T21:00:52.496 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:52.494+0000 7f95c2a00640 1 --2- 172.21.15.32:0/128054689 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f95c41456f0 0x7f95c41f46b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:00:52.496 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:52.494+0000 7f95c3400640 1 --2- 172.21.15.32:0/128054689 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f95c406c9e0 0x7f95c40cc130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:00:52.497 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:52.495+0000 7f95c3e00640 1 -- 172.21.15.32:0/128054689 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f95c41456f0 msgr2=0x7f95c41f46b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:00:52.497 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:52.495+0000 7f95c3e00640 1 --2- 172.21.15.32:0/128054689 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f95c41456f0 0x7f95c41f46b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:52.497 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:52.495+0000 7f95c3e00640 1 -- 172.21.15.32:0/128054689 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f95c406c9e0 msgr2=0x7f95c40cc130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:00:52.498 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:52.495+0000 7f95c3e00640 1 --2- 172.21.15.32:0/128054689 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f95c406c9e0 0x7f95c40cc130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:52.498 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:52.495+0000 7f95c3e00640 1 -- 172.21.15.32:0/128054689 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f95c41f53f0 con 0x7f95c4145cd0 2024-08-03T21:00:52.498 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:52.495+0000 7f95c3e00640 1 --2- 172.21.15.32:0/128054689 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f95c4145cd0 0x7f95c415bc70 secure :-1 s=READY pgs=2164 cs=0 l=1 rev1=1 crypto rx=0x7f95bc00c6c0 tx=0x7f95bc00cb80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T21:00:52.498 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:52.495+0000 7f95bb400640 1 -- 172.21.15.32:0/128054689 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95bc017070 con 0x7f95c4145cd0 2024-08-03T21:00:52.498 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:52.495+0000 7f95bb400640 1 -- 172.21.15.32:0/128054689 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f95bc003c60 con 0x7f95c4145cd0 2024-08-03T21:00:52.498 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:52.495+0000 7f95bb400640 1 -- 172.21.15.32:0/128054689 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95bc0052d0 con 0x7f95c4145cd0 2024-08-03T21:00:52.498 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:52.495+0000 7f95ca800640 1 -- 172.21.15.32:0/128054689 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f95c41f56d0 con 0x7f95c4145cd0 2024-08-03T21:00:52.498 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:52.495+0000 7f95ca800640 1 -- 172.21.15.32:0/128054689 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f95c41f5c90 con 0x7f95c4145cd0 2024-08-03T21:00:52.499 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:52.497+0000 7f95ca800640 1 -- 172.21.15.32:0/128054689 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f95c41f5160 con 0x7f95c4145cd0 2024-08-03T21:00:52.499 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:52.498+0000 7f95bb400640 1 -- 172.21.15.32:0/128054689 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f95bc01a070 con 0x7f95c4145cd0 2024-08-03T21:00:52.500 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:52.499+0000 7f95bb400640 1 --2- 172.21.15.32:0/128054689 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f95a4077010 0x7f95a4079520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:00:52.500 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:52.499+0000 7f95bb400640 1 -- 172.21.15.32:0/128054689 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(135..135 src has 1..135) ==== 5709+0+0 (secure 0 0 0) 0x7f95bc097fe0 con 0x7f95c4145cd0 2024-08-03T21:00:52.501 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:52.500+0000 7f95c3400640 1 --2- 172.21.15.32:0/128054689 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f95a4077010 0x7f95a4079520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:00:52.502 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:52.501+0000 7f95c3400640 1 --2- 172.21.15.32:0/128054689 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f95a4077010 0x7f95a4079520 secure :-1 s=READY pgs=867 cs=0 l=1 rev1=1 crypto rx=0x7f95b400bb50 tx=0x7f95b400b2e0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T21:00:52.505 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:52.503+0000 7f95bb400640 1 -- 172.21.15.32:0/128054689 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7f95bc060ab0 con 0x7f95c4145cd0 2024-08-03T21:00:52.750 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:52.749+0000 7f95ca800640 1 -- 172.21.15.32:0/128054689 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) -- 0x7f95c40d05f0 con 0x7f95c4145cd0 2024-08-03T21:00:53.238 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:53.236+0000 7f95bb400640 1 -- 172.21.15.32:0/128054689 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v340) ==== 122+0+0 (secure 0 0 0) 0x7f95bc0693d0 con 0x7f95c4145cd0 2024-08-03T21:00:53.241 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:53.239+0000 7f95ca800640 1 -- 172.21.15.32:0/128054689 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f95a4077010 msgr2=0x7f95a4079520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:00:53.242 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:53.239+0000 7f95ca800640 1 --2- 172.21.15.32:0/128054689 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f95a4077010 0x7f95a4079520 secure :-1 s=READY pgs=867 cs=0 l=1 rev1=1 crypto rx=0x7f95b400bb50 tx=0x7f95b400b2e0 comp rx=0 tx=0).stop 2024-08-03T21:00:53.242 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:53.239+0000 7f95ca800640 1 -- 172.21.15.32:0/128054689 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f95c4145cd0 msgr2=0x7f95c415bc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:00:53.242 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:53.239+0000 7f95ca800640 1 --2- 172.21.15.32:0/128054689 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f95c4145cd0 0x7f95c415bc70 secure :-1 s=READY pgs=2164 cs=0 l=1 rev1=1 crypto rx=0x7f95bc00c6c0 tx=0x7f95bc00cb80 comp rx=0 tx=0).stop 2024-08-03T21:00:53.242 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:53.239+0000 7f95ca800640 1 -- 172.21.15.32:0/128054689 shutdown_connections 2024-08-03T21:00:53.242 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:53.239+0000 7f95ca800640 1 --2- 172.21.15.32:0/128054689 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f95a4077010 0x7f95a4079520 unknown :-1 s=CLOSED pgs=867 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:53.242 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:53.239+0000 7f95ca800640 1 --2- 172.21.15.32:0/128054689 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f95c4145cd0 0x7f95c415bc70 unknown :-1 s=CLOSED pgs=2164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:53.242 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:53.239+0000 7f95ca800640 1 --2- 172.21.15.32:0/128054689 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f95c41456f0 0x7f95c41f46b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:53.243 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:53.239+0000 7f95ca800640 1 --2- 172.21.15.32:0/128054689 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f95c406c9e0 0x7f95c40cc130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:53.243 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:53.239+0000 7f95ca800640 1 -- 172.21.15.32:0/128054689 >> 172.21.15.32:0/128054689 conn(0x7f95c40b32f0 msgr2=0x7f95c40b0e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T21:00:53.243 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:53.240+0000 7f95ca800640 1 -- 172.21.15.32:0/128054689 shutdown_connections 2024-08-03T21:00:53.243 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:53.240+0000 7f95ca800640 1 -- 172.21.15.32:0/128054689 wait complete. 2024-08-03T21:00:53.270 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-08-03T21:00:53.270 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-08-03T21:00:53.499 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:53.497+0000 7fe877000640 1 -- 172.21.15.32:0/3631019563 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe870144f90 msgr2=0x7fe870074570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:00:53.500 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:53.497+0000 7fe877000640 1 --2- 172.21.15.32:0/3631019563 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe870144f90 0x7fe870074570 secure :-1 s=READY pgs=2165 cs=0 l=1 rev1=1 crypto rx=0x7fe870072d80 tx=0x7fe85802a5f0 comp rx=0 tx=0).stop 2024-08-03T21:00:53.500 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:53.498+0000 7fe877000640 1 -- 172.21.15.32:0/3631019563 shutdown_connections 2024-08-03T21:00:53.500 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:53.498+0000 7fe877000640 1 --2- 172.21.15.32:0/3631019563 >> v2:172.21.15.118:3300/0 conn(0x7fe87015fff0 0x7fe870160400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:53.500 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:53.498+0000 7fe877000640 1 --2- 172.21.15.32:0/3631019563 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe870144f90 0x7fe870074570 unknown :-1 s=CLOSED pgs=2165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:53.500 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:53.498+0000 7fe877000640 1 -- 172.21.15.32:0/3631019563 >> 172.21.15.32:0/3631019563 conn(0x7fe8700b1bd0 msgr2=0x7fe8700b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T21:00:53.500 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:53.498+0000 7fe877000640 1 -- 172.21.15.32:0/3631019563 shutdown_connections 2024-08-03T21:00:53.501 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:53.498+0000 7fe877000640 1 -- 172.21.15.32:0/3631019563 wait complete. 2024-08-03T21:00:53.501 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:53.500+0000 7fe877000640 1 Processor -- start 2024-08-03T21:00:53.501 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:53.500+0000 7fe877000640 1 -- start start 2024-08-03T21:00:53.503 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:53.501+0000 7fe877000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe87006e930 0x7fe8701f9a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:00:53.503 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:53.501+0000 7fe877000640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fe870144f90 0x7fe87013dcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:00:53.503 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:53.501+0000 7fe877000640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fe87015fff0 0x7fe870142c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:00:53.504 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:53.501+0000 7fe877000640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fe8700cf840 con 0x7fe87006e930 2024-08-03T21:00:53.504 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:53.502+0000 7fe877000640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7fe870145f30 con 0x7fe87015fff0 2024-08-03T21:00:53.504 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:53.502+0000 7fe877000640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7fe8700cf9c0 con 0x7fe870144f90 2024-08-03T21:00:53.504 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:53.502+0000 7fe86fe00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe87006e930 0x7fe8701f9a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:00:53.504 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:53.502+0000 7fe86fe00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe87006e930 0x7fe8701f9a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:50046/0 (socket says 172.21.15.32:50046) 2024-08-03T21:00:53.504 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:53.502+0000 7fe86fe00640 1 -- 172.21.15.32:0/1519126924 learned_addr learned my addr 172.21.15.32:0/1519126924 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T21:00:53.504 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:53.502+0000 7fe867e00640 1 --2- 172.21.15.32:0/1519126924 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fe870144f90 0x7fe87013dcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:00:53.504 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:53.502+0000 7fe875000640 1 --2- 172.21.15.32:0/1519126924 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fe87015fff0 0x7fe870142c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:00:53.505 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:53.502+0000 7fe86fe00640 1 -- 172.21.15.32:0/1519126924 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fe870144f90 msgr2=0x7fe87013dcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:00:53.505 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:53.502+0000 7fe86fe00640 1 --2- 172.21.15.32:0/1519126924 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fe870144f90 0x7fe87013dcf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:53.505 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:53.502+0000 7fe86fe00640 1 -- 172.21.15.32:0/1519126924 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fe87015fff0 msgr2=0x7fe870142c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:00:53.505 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:53.502+0000 7fe86fe00640 1 --2- 172.21.15.32:0/1519126924 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fe87015fff0 0x7fe870142c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:53.505 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:53.502+0000 7fe86fe00640 1 -- 172.21.15.32:0/1519126924 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe870141730 con 0x7fe87006e930 2024-08-03T21:00:53.505 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:53.503+0000 7fe86fe00640 1 --2- 172.21.15.32:0/1519126924 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe87006e930 0x7fe8701f9a00 secure :-1 s=READY pgs=2166 cs=0 l=1 rev1=1 crypto rx=0x7fe86000c3a0 tx=0x7fe86000c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T21:00:53.506 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:53.503+0000 7fe86cc00640 1 -- 172.21.15.32:0/1519126924 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe860017070 con 0x7fe87006e930 2024-08-03T21:00:53.506 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:53.503+0000 7fe86cc00640 1 -- 172.21.15.32:0/1519126924 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fe8600053c0 con 0x7fe87006e930 2024-08-03T21:00:53.506 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:53.503+0000 7fe877000640 1 -- 172.21.15.32:0/1519126924 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe870141a10 con 0x7fe87006e930 2024-08-03T21:00:53.506 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:53.503+0000 7fe86cc00640 1 -- 172.21.15.32:0/1519126924 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe860003850 con 0x7fe87006e930 2024-08-03T21:00:53.506 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:53.503+0000 7fe877000640 1 -- 172.21.15.32:0/1519126924 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe870141fd0 con 0x7fe87006e930 2024-08-03T21:00:53.506 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:53.504+0000 7fe877000640 1 -- 172.21.15.32:0/1519126924 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe870160c30 con 0x7fe87006e930 2024-08-03T21:00:53.507 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:53.506+0000 7fe86cc00640 1 -- 172.21.15.32:0/1519126924 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7fe86001a070 con 0x7fe87006e930 2024-08-03T21:00:53.508 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:53.507+0000 7fe86cc00640 1 --2- 172.21.15.32:0/1519126924 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fe84c077010 0x7fe84c079520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:00:53.508 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:53.507+0000 7fe86cc00640 1 -- 172.21.15.32:0/1519126924 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(135..135 src has 1..135) ==== 5709+0+0 (secure 0 0 0) 0x7fe860097d50 con 0x7fe87006e930 2024-08-03T21:00:53.508 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:53.507+0000 7fe867e00640 1 --2- 172.21.15.32:0/1519126924 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fe84c077010 0x7fe84c079520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:00:53.509 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:53.508+0000 7fe867e00640 1 --2- 172.21.15.32:0/1519126924 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fe84c077010 0x7fe84c079520 secure :-1 s=READY pgs=868 cs=0 l=1 rev1=1 crypto rx=0x7fe858033000 tx=0x7fe858004730 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T21:00:53.512 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:53.511+0000 7fe86cc00640 1 -- 172.21.15.32:0/1519126924 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7fe860060860 con 0x7fe87006e930 2024-08-03T21:00:53.754 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:53.752+0000 7fe877000640 1 -- 172.21.15.32:0/1519126924 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe870145720 con 0x7fe87006e930 2024-08-03T21:00:53.755 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:53.753+0000 7fe86cc00640 1 -- 172.21.15.32:0/1519126924 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 340 v340) ==== 95+0+5234 (secure 0 0 0) 0x7fe860069180 con 0x7fe87006e930 2024-08-03T21:00:53.756 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-03T21:00:53.756 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":340,"btime":"2024-08-03T21:00:53:231957+0000","default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8519,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6836/3249267506","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6834","nonce":3249267506},{"type":"v1","addr":"172.21.15.118:6836","nonce":3249267506}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":334},{"gid":8522,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6837/408542078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6835","nonce":408542078},{"type":"v1","addr":"172.21.15.118:6837","nonce":408542078}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":335},{"gid":9589,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6834/3440267920","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":3440267920},{"type":"v1","addr":"172.21.15.32:6834","nonce":3440267920}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":335}],"filesystems":[{"mdsmap":{"epoch":340,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-08-03T21:00:49.742850+0000","modified":"2024-08-03T21:00:52.750915+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":9592},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9592":{"gid":9592,"name":"a","rank":0,"incarnation":336,"state":"up:active","state_seq":3,"addr":"172.21.15.32:6835/4032603982","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6833","nonce":4032603982},{"type":"v1","addr":"172.21.15.32:6835","nonce":4032603982}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9592,"qdb_cluster":[9592]},"id":7}]} 2024-08-03T21:00:53.756 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 340 2024-08-03T21:00:53.758 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:53.755+0000 7fe877000640 1 -- 172.21.15.32:0/1519126924 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fe84c077010 msgr2=0x7fe84c079520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:00:53.758 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:53.755+0000 7fe877000640 1 --2- 172.21.15.32:0/1519126924 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fe84c077010 0x7fe84c079520 secure :-1 s=READY pgs=868 cs=0 l=1 rev1=1 crypto rx=0x7fe858033000 tx=0x7fe858004730 comp rx=0 tx=0).stop 2024-08-03T21:00:53.758 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:53.755+0000 7fe877000640 1 -- 172.21.15.32:0/1519126924 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe87006e930 msgr2=0x7fe8701f9a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:00:53.758 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:53.755+0000 7fe877000640 1 --2- 172.21.15.32:0/1519126924 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe87006e930 0x7fe8701f9a00 secure :-1 s=READY pgs=2166 cs=0 l=1 rev1=1 crypto rx=0x7fe86000c3a0 tx=0x7fe86000c860 comp rx=0 tx=0).stop 2024-08-03T21:00:53.759 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:53.755+0000 7fe877000640 1 -- 172.21.15.32:0/1519126924 shutdown_connections 2024-08-03T21:00:53.759 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:53.755+0000 7fe877000640 1 --2- 172.21.15.32:0/1519126924 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fe84c077010 0x7fe84c079520 unknown :-1 s=CLOSED pgs=868 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:53.759 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:53.756+0000 7fe877000640 1 --2- 172.21.15.32:0/1519126924 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fe87015fff0 0x7fe870142c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:53.759 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:53.756+0000 7fe877000640 1 --2- 172.21.15.32:0/1519126924 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fe870144f90 0x7fe87013dcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:53.759 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:53.756+0000 7fe877000640 1 --2- 172.21.15.32:0/1519126924 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe87006e930 0x7fe8701f9a00 unknown :-1 s=CLOSED pgs=2166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:53.759 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:53.756+0000 7fe877000640 1 -- 172.21.15.32:0/1519126924 >> 172.21.15.32:0/1519126924 conn(0x7fe8700b1bd0 msgr2=0x7fe8700b01b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T21:00:53.760 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:53.756+0000 7fe877000640 1 -- 172.21.15.32:0/1519126924 shutdown_connections 2024-08-03T21:00:53.760 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:53.756+0000 7fe877000640 1 -- 172.21.15.32:0/1519126924 wait complete. 2024-08-03T21:00:53.785 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-08-03T21:00:54.014 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:54.012+0000 7fd95e600640 1 -- 172.21.15.32:0/538676997 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fd9580ceef0 msgr2=0x7fd9580b72e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:00:54.014 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:54.012+0000 7fd95e600640 1 --2- 172.21.15.32:0/538676997 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fd9580ceef0 0x7fd9580b72e0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fd940005fd0 tx=0x7fd94002a540 comp rx=0 tx=0).stop 2024-08-03T21:00:54.014 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:54.012+0000 7fd95e600640 1 -- 172.21.15.32:0/538676997 shutdown_connections 2024-08-03T21:00:54.014 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:54.012+0000 7fd95e600640 1 --2- 172.21.15.32:0/538676997 >> v2:172.21.15.118:3300/0 conn(0x7fd9580d29b0 0x7fd9580b1f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:54.014 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:54.012+0000 7fd95e600640 1 --2- 172.21.15.32:0/538676997 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fd9580ceef0 0x7fd9580b72e0 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:54.014 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:54.012+0000 7fd95e600640 1 -- 172.21.15.32:0/538676997 >> 172.21.15.32:0/538676997 conn(0x7fd9580d1e90 msgr2=0x7fd9580d22a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T21:00:54.014 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:54.012+0000 7fd95e600640 1 -- 172.21.15.32:0/538676997 shutdown_connections 2024-08-03T21:00:54.014 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:54.012+0000 7fd95e600640 1 -- 172.21.15.32:0/538676997 wait complete. 2024-08-03T21:00:54.015 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:54.014+0000 7fd95e600640 1 Processor -- start 2024-08-03T21:00:54.016 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:54.014+0000 7fd95e600640 1 -- start start 2024-08-03T21:00:54.018 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:54.016+0000 7fd95e600640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fd9580b1300 0x7fd9580b3d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:00:54.018 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:54.016+0000 7fd95e600640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fd9580d29b0 0x7fd958071d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:00:54.018 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:54.016+0000 7fd95e600640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd9580722b0 0x7fd9581e6bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:00:54.018 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:54.016+0000 7fd95e600640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fd9580b29f0 con 0x7fd9580722b0 2024-08-03T21:00:54.018 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:54.016+0000 7fd95e600640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7fd9580b2870 con 0x7fd9580b1300 2024-08-03T21:00:54.018 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:54.016+0000 7fd95e600640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7fd9580b2b70 con 0x7fd9580d29b0 2024-08-03T21:00:54.018 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:54.016+0000 7fd95dc00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd9580722b0 0x7fd9581e6bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:00:54.018 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:54.016+0000 7fd95dc00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd9580722b0 0x7fd9581e6bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:50048/0 (socket says 172.21.15.32:50048) 2024-08-03T21:00:54.018 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:54.016+0000 7fd95dc00640 1 -- 172.21.15.32:0/2312389028 learned_addr learned my addr 172.21.15.32:0/2312389028 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T21:00:54.018 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:54.016+0000 7fd94fe00640 1 --2- 172.21.15.32:0/2312389028 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fd9580d29b0 0x7fd958071d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:00:54.018 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:54.016+0000 7fd95d200640 1 --2- 172.21.15.32:0/2312389028 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fd9580b1300 0x7fd9580b3d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:00:54.019 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:54.017+0000 7fd95dc00640 1 -- 172.21.15.32:0/2312389028 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fd9580d29b0 msgr2=0x7fd958071d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:00:54.020 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:54.017+0000 7fd95dc00640 1 --2- 172.21.15.32:0/2312389028 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fd9580d29b0 0x7fd958071d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:54.020 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:54.017+0000 7fd95dc00640 1 -- 172.21.15.32:0/2312389028 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fd9580b1300 msgr2=0x7fd9580b3d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:00:54.020 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:54.017+0000 7fd95dc00640 1 --2- 172.21.15.32:0/2312389028 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fd9580b1300 0x7fd9580b3d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:54.020 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:54.017+0000 7fd95dc00640 1 -- 172.21.15.32:0/2312389028 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd9581f8bb0 con 0x7fd9580722b0 2024-08-03T21:00:54.020 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:54.017+0000 7fd94fe00640 1 --2- 172.21.15.32:0/2312389028 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fd9580d29b0 0x7fd958071d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-03T21:00:54.020 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:54.017+0000 7fd95dc00640 1 --2- 172.21.15.32:0/2312389028 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd9580722b0 0x7fd9581e6bd0 secure :-1 s=READY pgs=2167 cs=0 l=1 rev1=1 crypto rx=0x7fd95000ea40 tx=0x7fd95000ef00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T21:00:54.020 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:54.017+0000 7fd955600640 1 -- 172.21.15.32:0/2312389028 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd95000ce40 con 0x7fd9580722b0 2024-08-03T21:00:54.020 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:54.017+0000 7fd955600640 1 -- 172.21.15.32:0/2312389028 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fd950004c10 con 0x7fd9580722b0 2024-08-03T21:00:54.020 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:54.017+0000 7fd95e600640 1 -- 172.21.15.32:0/2312389028 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd9581f8e90 con 0x7fd9580722b0 2024-08-03T21:00:54.020 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:54.017+0000 7fd955600640 1 -- 172.21.15.32:0/2312389028 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd950005020 con 0x7fd9580722b0 2024-08-03T21:00:54.020 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:54.017+0000 7fd95e600640 1 -- 172.21.15.32:0/2312389028 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd95815ec50 con 0x7fd9580722b0 2024-08-03T21:00:54.021 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:54.018+0000 7fd95e600640 1 -- 172.21.15.32:0/2312389028 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd9581f8920 con 0x7fd9580722b0 2024-08-03T21:00:54.021 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:54.020+0000 7fd955600640 1 -- 172.21.15.32:0/2312389028 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7fd9500051c0 con 0x7fd9580722b0 2024-08-03T21:00:54.022 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:54.021+0000 7fd955600640 1 --2- 172.21.15.32:0/2312389028 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fd934077010 0x7fd934079520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:00:54.022 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:54.021+0000 7fd955600640 1 -- 172.21.15.32:0/2312389028 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(135..135 src has 1..135) ==== 5709+0+0 (secure 0 0 0) 0x7fd950099650 con 0x7fd9580722b0 2024-08-03T21:00:54.023 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:54.021+0000 7fd95d200640 1 --2- 172.21.15.32:0/2312389028 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fd934077010 0x7fd934079520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:00:54.023 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:54.022+0000 7fd95d200640 1 --2- 172.21.15.32:0/2312389028 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fd934077010 0x7fd934079520 secure :-1 s=READY pgs=869 cs=0 l=1 rev1=1 crypto rx=0x7fd948001720 tx=0x7fd948009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T21:00:54.026 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:54.025+0000 7fd955600640 1 -- 172.21.15.32:0/2312389028 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7fd950062160 con 0x7fd9580722b0 2024-08-03T21:00:54.234 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:54.232+0000 7fd95e600640 1 -- 172.21.15.32:0/2312389028 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fd9580699c0 con 0x7fd9580722b0 2024-08-03T21:00:54.235 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:54.233+0000 7fd955600640 1 -- 172.21.15.32:0/2312389028 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v135) ==== 74+0+16559 (secure 0 0 0) 0x7fd9500666b0 con 0x7fd9580722b0 2024-08-03T21:00:54.235 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-03T21:00:54.236 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":135,"fsid":"40ab141a-01d7-49b9-ae3c-2edec9876e95","created":"2024-08-03T20:47:14.957362+0000","modified":"2024-08-03T21:00:51.217135+0000","last_up_change":"2024-08-03T20:47:28.874451+0000","last_in_change":"2024-08-03T20:47:20.079442+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":15,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-08-03T20:47:28.261300+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":14,"pool_name":"cephfs_metadata","create_time":"2024-08-03T21:00:47.705081+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"134","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":15,"pool_name":"cephfs_data","create_time":"2024-08-03T21:00:48.703409+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"135","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5a83e759-9035-438b-889a-0a6af1f0a0d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":133,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6816","nonce":3066156116},{"type":"v1","addr":"172.21.15.32:6817","nonce":3066156116}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6818","nonce":3066156116},{"type":"v1","addr":"172.21.15.32:6819","nonce":3066156116}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6822","nonce":3066156116},{"type":"v1","addr":"172.21.15.32:6823","nonce":3066156116}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6820","nonce":3066156116},{"type":"v1","addr":"172.21.15.32:6821","nonce":3066156116}]},"public_addr":"172.21.15.32:6817/3066156116","cluster_addr":"172.21.15.32:6819/3066156116","heartbeat_back_addr":"172.21.15.32:6823/3066156116","heartbeat_front_addr":"172.21.15.32:6821/3066156116","state":["exists","up"]},{"osd":1,"uuid":"d7c5c210-da4f-49f9-b9e7-1dfb02daa4bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":133,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6824","nonce":1551241759},{"type":"v1","addr":"172.21.15.32:6825","nonce":1551241759}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6826","nonce":1551241759},{"type":"v1","addr":"172.21.15.32:6827","nonce":1551241759}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6830","nonce":1551241759},{"type":"v1","addr":"172.21.15.32:6831","nonce":1551241759}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6828","nonce":1551241759},{"type":"v1","addr":"172.21.15.32:6829","nonce":1551241759}]},"public_addr":"172.21.15.32:6825/1551241759","cluster_addr":"172.21.15.32:6827/1551241759","heartbeat_back_addr":"172.21.15.32:6831/1551241759","heartbeat_front_addr":"172.21.15.32:6829/1551241759","state":["exists","up"]},{"osd":2,"uuid":"57f2e6de-a201-45c2-8981-8e56a4875ebd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":133,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6808","nonce":3806708944},{"type":"v1","addr":"172.21.15.32:6809","nonce":3806708944}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6810","nonce":3806708944},{"type":"v1","addr":"172.21.15.32:6811","nonce":3806708944}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6814","nonce":3806708944},{"type":"v1","addr":"172.21.15.32:6815","nonce":3806708944}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6812","nonce":3806708944},{"type":"v1","addr":"172.21.15.32:6813","nonce":3806708944}]},"public_addr":"172.21.15.32:6809/3806708944","cluster_addr":"172.21.15.32:6811/3806708944","heartbeat_back_addr":"172.21.15.32:6815/3806708944","heartbeat_front_addr":"172.21.15.32:6813/3806708944","state":["exists","up"]},{"osd":3,"uuid":"6439a280-da45-4148-8130-f16323935ff3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":133,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6800","nonce":9088303},{"type":"v1","addr":"172.21.15.32:6801","nonce":9088303}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6802","nonce":9088303},{"type":"v1","addr":"172.21.15.32:6803","nonce":9088303}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6806","nonce":9088303},{"type":"v1","addr":"172.21.15.32:6807","nonce":9088303}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6804","nonce":9088303},{"type":"v1","addr":"172.21.15.32:6805","nonce":9088303}]},"public_addr":"172.21.15.32:6801/9088303","cluster_addr":"172.21.15.32:6803/9088303","heartbeat_back_addr":"172.21.15.32:6807/9088303","heartbeat_front_addr":"172.21.15.32:6805/9088303","state":["exists","up"]},{"osd":4,"uuid":"4f902eff-1e18-418d-82c9-2151c69524fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":133,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6802","nonce":864752275},{"type":"v1","addr":"172.21.15.118:6803","nonce":864752275}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6804","nonce":864752275},{"type":"v1","addr":"172.21.15.118:6805","nonce":864752275}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6808","nonce":864752275},{"type":"v1","addr":"172.21.15.118:6809","nonce":864752275}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6806","nonce":864752275},{"type":"v1","addr":"172.21.15.118:6807","nonce":864752275}]},"public_addr":"172.21.15.118:6803/864752275","cluster_addr":"172.21.15.118:6805/864752275","heartbeat_back_addr":"172.21.15.118:6809/864752275","heartbeat_front_addr":"172.21.15.118:6807/864752275","state":["exists","up"]},{"osd":5,"uuid":"bfd9feac-dd06-458c-91b9-af7f175f612c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":133,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6810","nonce":2724891770},{"type":"v1","addr":"172.21.15.118:6811","nonce":2724891770}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6812","nonce":2724891770},{"type":"v1","addr":"172.21.15.118:6813","nonce":2724891770}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6816","nonce":2724891770},{"type":"v1","addr":"172.21.15.118:6817","nonce":2724891770}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6814","nonce":2724891770},{"type":"v1","addr":"172.21.15.118:6815","nonce":2724891770}]},"public_addr":"172.21.15.118:6811/2724891770","cluster_addr":"172.21.15.118:6813/2724891770","heartbeat_back_addr":"172.21.15.118:6817/2724891770","heartbeat_front_addr":"172.21.15.118:6815/2724891770","state":["exists","up"]},{"osd":6,"uuid":"5ed5a279-9d8e-416d-bbc7-7edd3e6d5705","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":133,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6825","nonce":801947386},{"type":"v1","addr":"172.21.15.118:6827","nonce":801947386}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6828","nonce":801947386},{"type":"v1","addr":"172.21.15.118:6829","nonce":801947386}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6832","nonce":801947386},{"type":"v1","addr":"172.21.15.118:6833","nonce":801947386}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6830","nonce":801947386},{"type":"v1","addr":"172.21.15.118:6831","nonce":801947386}]},"public_addr":"172.21.15.118:6827/801947386","cluster_addr":"172.21.15.118:6829/801947386","heartbeat_back_addr":"172.21.15.118:6833/801947386","heartbeat_front_addr":"172.21.15.118:6831/801947386","state":["exists","up"]},{"osd":7,"uuid":"f2086c47-875e-4451-a436-2008d1ea16af","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":133,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6818","nonce":653000048},{"type":"v1","addr":"172.21.15.118:6819","nonce":653000048}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6820","nonce":653000048},{"type":"v1","addr":"172.21.15.118:6821","nonce":653000048}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6824","nonce":653000048},{"type":"v1","addr":"172.21.15.118:6826","nonce":653000048}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6822","nonce":653000048},{"type":"v1","addr":"172.21.15.118:6823","nonce":653000048}]},"public_addr":"172.21.15.118:6819/653000048","cluster_addr":"172.21.15.118:6821/653000048","heartbeat_back_addr":"172.21.15.118:6826/653000048","heartbeat_front_addr":"172.21.15.118:6823/653000048","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-03T20:47:25.491994+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-03T20:47:25.587358+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-03T20:47:25.454175+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-03T20:47:25.377734+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-03T20:47:25.412351+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-03T20:47:25.499815+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-03T20:47:25.575044+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-03T20:47:25.552383+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-08-03T21:00:54.238 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:54.236+0000 7fd95e600640 1 -- 172.21.15.32:0/2312389028 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fd934077010 msgr2=0x7fd934079520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:00:54.238 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:54.236+0000 7fd95e600640 1 --2- 172.21.15.32:0/2312389028 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fd934077010 0x7fd934079520 secure :-1 s=READY pgs=869 cs=0 l=1 rev1=1 crypto rx=0x7fd948001720 tx=0x7fd948009040 comp rx=0 tx=0).stop 2024-08-03T21:00:54.238 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:54.236+0000 7fd95e600640 1 -- 172.21.15.32:0/2312389028 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd9580722b0 msgr2=0x7fd9581e6bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:00:54.238 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:54.236+0000 7fd95e600640 1 --2- 172.21.15.32:0/2312389028 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd9580722b0 0x7fd9581e6bd0 secure :-1 s=READY pgs=2167 cs=0 l=1 rev1=1 crypto rx=0x7fd95000ea40 tx=0x7fd95000ef00 comp rx=0 tx=0).stop 2024-08-03T21:00:54.238 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:54.236+0000 7fd95e600640 1 -- 172.21.15.32:0/2312389028 shutdown_connections 2024-08-03T21:00:54.238 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:54.236+0000 7fd95e600640 1 --2- 172.21.15.32:0/2312389028 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fd934077010 0x7fd934079520 unknown :-1 s=CLOSED pgs=869 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:54.238 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:54.236+0000 7fd95e600640 1 --2- 172.21.15.32:0/2312389028 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd9580722b0 0x7fd9581e6bd0 unknown :-1 s=CLOSED pgs=2167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:54.238 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:54.236+0000 7fd95e600640 1 --2- 172.21.15.32:0/2312389028 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fd9580d29b0 0x7fd958071d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:54.238 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:54.236+0000 7fd95e600640 1 --2- 172.21.15.32:0/2312389028 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fd9580b1300 0x7fd9580b3d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:54.238 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:54.236+0000 7fd95e600640 1 -- 172.21.15.32:0/2312389028 >> 172.21.15.32:0/2312389028 conn(0x7fd9580d1e90 msgr2=0x7fd95806daf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T21:00:54.238 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:54.236+0000 7fd95e600640 1 -- 172.21.15.32:0/2312389028 shutdown_connections 2024-08-03T21:00:54.239 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:54.236+0000 7fd95e600640 1 -- 172.21.15.32:0/2312389028 wait complete. 2024-08-03T21:00:54.265 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-08-03T21:00:54.265 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-08-03T21:00:54.494 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:54.492+0000 7fd523e00640 1 -- 172.21.15.32:0/1308224670 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd508005320 msgr2=0x7fd50800b9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:00:54.494 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:54.492+0000 7fd523e00640 1 --2- 172.21.15.32:0/1308224670 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd508005320 0x7fd50800b9e0 secure :-1 s=READY pgs=2169 cs=0 l=1 rev1=1 crypto rx=0x7fd51c00c610 tx=0x7fd508005830 comp rx=0 tx=0).stop 2024-08-03T21:00:54.494 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:54.492+0000 7fd523e00640 1 -- 172.21.15.32:0/1308224670 shutdown_connections 2024-08-03T21:00:54.494 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:54.492+0000 7fd523e00640 1 --2- 172.21.15.32:0/1308224670 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd508005320 0x7fd50800b9e0 unknown :-1 s=CLOSED pgs=2169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:54.494 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:54.492+0000 7fd523e00640 1 -- 172.21.15.32:0/1308224670 >> 172.21.15.32:0/1308224670 conn(0x7fd5240d4090 msgr2=0x7fd5240d44a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T21:00:54.494 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:54.492+0000 7fd523e00640 1 -- 172.21.15.32:0/1308224670 shutdown_connections 2024-08-03T21:00:54.494 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:54.492+0000 7fd523e00640 1 -- 172.21.15.32:0/1308224670 wait complete. 2024-08-03T21:00:54.496 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:54.494+0000 7fd523e00640 1 Processor -- start 2024-08-03T21:00:54.496 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:54.495+0000 7fd523e00640 1 -- start start 2024-08-03T21:00:54.499 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:54.496+0000 7fd523e00640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fd508005320 0x7fd5241e52b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:00:54.499 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:54.496+0000 7fd523e00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fd5241e57f0 0x7fd5240cd670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:00:54.499 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:54.496+0000 7fd523e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd5240cdbb0 0x7fd5241e3ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:00:54.499 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:54.496+0000 7fd523e00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fd5240d2410 con 0x7fd5240cdbb0 2024-08-03T21:00:54.499 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:54.496+0000 7fd523e00640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7fd5240d2290 con 0x7fd5241e57f0 2024-08-03T21:00:54.499 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:54.496+0000 7fd523e00640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7fd5240d2590 con 0x7fd508005320 2024-08-03T21:00:54.499 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:54.496+0000 7fd523400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd5240cdbb0 0x7fd5241e3ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:00:54.500 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:54.496+0000 7fd523400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd5240cdbb0 0x7fd5241e3ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:50060/0 (socket says 172.21.15.32:50060) 2024-08-03T21:00:54.500 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:54.496+0000 7fd523400640 1 -- 172.21.15.32:0/3699658630 learned_addr learned my addr 172.21.15.32:0/3699658630 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T21:00:54.500 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:54.497+0000 7fd522a00640 1 --2- 172.21.15.32:0/3699658630 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fd508005320 0x7fd5241e52b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:00:54.500 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:54.497+0000 7fd51be00640 1 --2- 172.21.15.32:0/3699658630 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fd5241e57f0 0x7fd5240cd670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:00:54.500 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:54.497+0000 7fd523400640 1 -- 172.21.15.32:0/3699658630 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fd508005320 msgr2=0x7fd5241e52b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:00:54.500 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:54.497+0000 7fd523400640 1 --2- 172.21.15.32:0/3699658630 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fd508005320 0x7fd5241e52b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:54.500 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:54.497+0000 7fd523400640 1 -- 172.21.15.32:0/3699658630 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fd5241e57f0 msgr2=0x7fd5240cd670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:00:54.500 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:54.497+0000 7fd523400640 1 --2- 172.21.15.32:0/3699658630 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fd5241e57f0 0x7fd5240cd670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:54.500 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:54.497+0000 7fd523400640 1 -- 172.21.15.32:0/3699658630 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd5240cc470 con 0x7fd5240cdbb0 2024-08-03T21:00:54.500 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:54.497+0000 7fd523400640 1 --2- 172.21.15.32:0/3699658630 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd5240cdbb0 0x7fd5241e3ec0 secure :-1 s=READY pgs=2170 cs=0 l=1 rev1=1 crypto rx=0x7fd51c00c730 tx=0x7fd51c0039f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T21:00:54.500 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:54.497+0000 7fd51aa00640 1 -- 172.21.15.32:0/3699658630 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd51c02b720 con 0x7fd5240cdbb0 2024-08-03T21:00:54.500 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:54.497+0000 7fd51aa00640 1 -- 172.21.15.32:0/3699658630 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fd51c03a070 con 0x7fd5240cdbb0 2024-08-03T21:00:54.500 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:54.498+0000 7fd51aa00640 1 -- 172.21.15.32:0/3699658630 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd51c02bc70 con 0x7fd5240cdbb0 2024-08-03T21:00:54.501 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:54.498+0000 7fd523e00640 1 -- 172.21.15.32:0/3699658630 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd5240cc750 con 0x7fd5240cdbb0 2024-08-03T21:00:54.501 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:54.498+0000 7fd523e00640 1 -- 172.21.15.32:0/3699658630 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd5240ccd10 con 0x7fd5240cdbb0 2024-08-03T21:00:54.501 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:54.499+0000 7fd523e00640 1 -- 172.21.15.32:0/3699658630 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd5240d2af0 con 0x7fd5240cdbb0 2024-08-03T21:00:54.501 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:54.500+0000 7fd51aa00640 1 -- 172.21.15.32:0/3699658630 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7fd51c03c020 con 0x7fd5240cdbb0 2024-08-03T21:00:54.503 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:54.501+0000 7fd51aa00640 1 --2- 172.21.15.32:0/3699658630 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fd500077010 0x7fd500079520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:00:54.506 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:54.502+0000 7fd51aa00640 1 -- 172.21.15.32:0/3699658630 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(135..135 src has 1..135) ==== 5709+0+0 (secure 0 0 0) 0x7fd51c0b7cc0 con 0x7fd5240cdbb0 2024-08-03T21:00:54.506 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:54.502+0000 7fd522a00640 1 --2- 172.21.15.32:0/3699658630 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fd500077010 0x7fd500079520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:00:54.506 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:54.503+0000 7fd522a00640 1 --2- 172.21.15.32:0/3699658630 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fd500077010 0x7fd500079520 secure :-1 s=READY pgs=870 cs=0 l=1 rev1=1 crypto rx=0x7fd514000c00 tx=0x7fd514006b10 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T21:00:54.507 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:54.506+0000 7fd51aa00640 1 -- 172.21.15.32:0/3699658630 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7fd51c0be080 con 0x7fd5240cdbb0 2024-08-03T21:00:54.710 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:54.707+0000 7fd523e00640 1 -- 172.21.15.32:0/3699658630 --> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fd5240cc200 con 0x7fd500077010 2024-08-03T21:00:54.714 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:54.713+0000 7fd51aa00640 1 -- 172.21.15.32:0/3699658630 <== mgr.4106 v2:172.21.15.118:6800/4205950699 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+249456 (secure 0 0 0) 0x7fd5240cc200 con 0x7fd500077010 2024-08-03T21:00:54.717 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-03T21:00:54.719 INFO:teuthology.orchestra.run.smithi032.stderr:dumped all 2024-08-03T21:00:54.720 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:54.717+0000 7fd523e00640 1 -- 172.21.15.32:0/3699658630 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fd500077010 msgr2=0x7fd500079520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:00:54.720 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:54.717+0000 7fd523e00640 1 --2- 172.21.15.32:0/3699658630 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fd500077010 0x7fd500079520 secure :-1 s=READY pgs=870 cs=0 l=1 rev1=1 crypto rx=0x7fd514000c00 tx=0x7fd514006b10 comp rx=0 tx=0).stop 2024-08-03T21:00:54.721 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:54.717+0000 7fd523e00640 1 -- 172.21.15.32:0/3699658630 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd5240cdbb0 msgr2=0x7fd5241e3ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:00:54.721 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:54.717+0000 7fd523e00640 1 --2- 172.21.15.32:0/3699658630 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd5240cdbb0 0x7fd5241e3ec0 secure :-1 s=READY pgs=2170 cs=0 l=1 rev1=1 crypto rx=0x7fd51c00c730 tx=0x7fd51c0039f0 comp rx=0 tx=0).stop 2024-08-03T21:00:54.721 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:54.717+0000 7fd523e00640 1 -- 172.21.15.32:0/3699658630 shutdown_connections 2024-08-03T21:00:54.721 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:54.717+0000 7fd523e00640 1 --2- 172.21.15.32:0/3699658630 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fd500077010 0x7fd500079520 unknown :-1 s=CLOSED pgs=870 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:54.721 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:54.717+0000 7fd523e00640 1 --2- 172.21.15.32:0/3699658630 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fd5241e57f0 0x7fd5240cd670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:54.721 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:54.717+0000 7fd523e00640 1 --2- 172.21.15.32:0/3699658630 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd5240cdbb0 0x7fd5241e3ec0 unknown :-1 s=CLOSED pgs=2170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:54.721 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:54.717+0000 7fd523e00640 1 --2- 172.21.15.32:0/3699658630 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fd508005320 0x7fd5241e52b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:54.721 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:54.717+0000 7fd523e00640 1 -- 172.21.15.32:0/3699658630 >> 172.21.15.32:0/3699658630 conn(0x7fd5240d4090 msgr2=0x7fd52406de10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T21:00:54.721 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:54.717+0000 7fd523e00640 1 -- 172.21.15.32:0/3699658630 shutdown_connections 2024-08-03T21:00:54.721 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:54.717+0000 7fd523e00640 1 -- 172.21.15.32:0/3699658630 wait complete. 2024-08-03T21:00:54.741 INFO:teuthology.orchestra.run.smithi032.stdout:{"pg_ready":true,"pg_map":{"version":531,"stamp":"2024-08-03T21:00:54.454153+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592014,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":113,"num_write_kb":1638,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":82,"ondisk_log_size":82,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":81,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":301392,"kb_used_data":14492,"kb_used_omap":12,"kb_used_meta":286771,"kb_avail":754673328,"statfs":{"total":773094113280,"available":772785487872,"internally_reserved":0,"allocated":14839808,"data_stored":7734630,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":293654065},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":20,"apply_latency_ms":20,"commit_latency_ns":20000000,"apply_latency_ns":20000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":1646,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001779"},"pg_stats":[{"pgid":"15.3f","version":"0'0","reported_seq":1,"reported_epoch":133,"state":"creating+peering","last_fresh":"2024-08-03T21:00:49.214906+0000","last_change":"2024-08-03T21:00:49.214906+0000","last_active":"2024-08-03T21:00:49.195479+0000","last_peered":"2024-08-03T21:00:49.195479+0000","last_clean":"2024-08-03T21:00:49.195479+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-08-03T21:00:49.214906+0000","last_undegraded":"2024-08-03T21:00:49.214906+0000","last_fullsized":"2024-08-03T21:00:49.214906+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_clean_scrub_stamp":"2024-08-03T21:00:49.195479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.3e","version":"0'0","reported_seq":1,"reported_epoch":133,"state":"creating+peering","last_fresh":"2024-08-03T21:00:49.221252+0000","last_change":"2024-08-03T21:00:49.221252+0000","last_active":"2024-08-03T21:00:49.195479+0000","last_peered":"2024-08-03T21:00:49.195479+0000","last_clean":"2024-08-03T21:00:49.195479+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-08-03T21:00:49.221252+0000","last_undegraded":"2024-08-03T21:00:49.221252+0000","last_fullsized":"2024-08-03T21:00:49.221252+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_clean_scrub_stamp":"2024-08-03T21:00:49.195479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.3d","version":"0'0","reported_seq":1,"reported_epoch":133,"state":"creating+peering","last_fresh":"2024-08-03T21:00:49.214132+0000","last_change":"2024-08-03T21:00:49.214132+0000","last_active":"2024-08-03T21:00:49.195479+0000","last_peered":"2024-08-03T21:00:49.195479+0000","last_clean":"2024-08-03T21:00:49.195479+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-08-03T21:00:49.214132+0000","last_undegraded":"2024-08-03T21:00:49.214132+0000","last_fullsized":"2024-08-03T21:00:49.214132+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_clean_scrub_stamp":"2024-08-03T21:00:49.195479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.3c","version":"0'0","reported_seq":13,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-08-03T21:00:51.225682+0000","last_change":"2024-08-03T21:00:50.243269+0000","last_active":"2024-08-03T21:00:51.225682+0000","last_peered":"2024-08-03T21:00:51.225682+0000","last_clean":"2024-08-03T21:00:51.225682+0000","last_became_active":"2024-08-03T21:00:50.242985+0000","last_became_peered":"2024-08-03T21:00:50.242985+0000","last_unstale":"2024-08-03T21:00:51.225682+0000","last_undegraded":"2024-08-03T21:00:51.225682+0000","last_fullsized":"2024-08-03T21:00:51.225682+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_clean_scrub_stamp":"2024-08-03T21:00:49.195479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T03:32:19.871676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.3b","version":"0'0","reported_seq":1,"reported_epoch":133,"state":"creating+peering","last_fresh":"2024-08-03T21:00:49.212243+0000","last_change":"2024-08-03T21:00:49.212243+0000","last_active":"2024-08-03T21:00:49.195479+0000","last_peered":"2024-08-03T21:00:49.195479+0000","last_clean":"2024-08-03T21:00:49.195479+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-08-03T21:00:49.212243+0000","last_undegraded":"2024-08-03T21:00:49.212243+0000","last_fullsized":"2024-08-03T21:00:49.212243+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_clean_scrub_stamp":"2024-08-03T21:00:49.195479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.3a","version":"0'0","reported_seq":1,"reported_epoch":133,"state":"creating+peering","last_fresh":"2024-08-03T21:00:49.215377+0000","last_change":"2024-08-03T21:00:49.215377+0000","last_active":"2024-08-03T21:00:49.195479+0000","last_peered":"2024-08-03T21:00:49.195479+0000","last_clean":"2024-08-03T21:00:49.195479+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-08-03T21:00:49.215377+0000","last_undegraded":"2024-08-03T21:00:49.215377+0000","last_fullsized":"2024-08-03T21:00:49.215377+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_clean_scrub_stamp":"2024-08-03T21:00:49.195479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.39","version":"0'0","reported_seq":1,"reported_epoch":133,"state":"creating+peering","last_fresh":"2024-08-03T21:00:49.213231+0000","last_change":"2024-08-03T21:00:49.213231+0000","last_active":"2024-08-03T21:00:49.195479+0000","last_peered":"2024-08-03T21:00:49.195479+0000","last_clean":"2024-08-03T21:00:49.195479+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-08-03T21:00:49.213231+0000","last_undegraded":"2024-08-03T21:00:49.213231+0000","last_fullsized":"2024-08-03T21:00:49.213231+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_clean_scrub_stamp":"2024-08-03T21:00:49.195479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.38","version":"0'0","reported_seq":1,"reported_epoch":133,"state":"creating+peering","last_fresh":"2024-08-03T21:00:49.214019+0000","last_change":"2024-08-03T21:00:49.214019+0000","last_active":"2024-08-03T21:00:49.195479+0000","last_peered":"2024-08-03T21:00:49.195479+0000","last_clean":"2024-08-03T21:00:49.195479+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-08-03T21:00:49.214019+0000","last_undegraded":"2024-08-03T21:00:49.214019+0000","last_fullsized":"2024-08-03T21:00:49.214019+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_clean_scrub_stamp":"2024-08-03T21:00:49.195479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.37","version":"0'0","reported_seq":13,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-08-03T21:00:51.939823+0000","last_change":"2024-08-03T21:00:50.223154+0000","last_active":"2024-08-03T21:00:51.939823+0000","last_peered":"2024-08-03T21:00:51.939823+0000","last_clean":"2024-08-03T21:00:51.939823+0000","last_became_active":"2024-08-03T21:00:50.222779+0000","last_became_peered":"2024-08-03T21:00:50.222779+0000","last_unstale":"2024-08-03T21:00:51.939823+0000","last_undegraded":"2024-08-03T21:00:51.939823+0000","last_fullsized":"2024-08-03T21:00:51.939823+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_clean_scrub_stamp":"2024-08-03T21:00:49.195479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T06:34:19.632267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.36","version":"0'0","reported_seq":1,"reported_epoch":133,"state":"creating+peering","last_fresh":"2024-08-03T21:00:49.211444+0000","last_change":"2024-08-03T21:00:49.211444+0000","last_active":"2024-08-03T21:00:49.195479+0000","last_peered":"2024-08-03T21:00:49.195479+0000","last_clean":"2024-08-03T21:00:49.195479+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-08-03T21:00:49.211444+0000","last_undegraded":"2024-08-03T21:00:49.211444+0000","last_fullsized":"2024-08-03T21:00:49.211444+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_clean_scrub_stamp":"2024-08-03T21:00:49.195479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.35","version":"0'0","reported_seq":1,"reported_epoch":133,"state":"creating+peering","last_fresh":"2024-08-03T21:00:49.217444+0000","last_change":"2024-08-03T21:00:49.217444+0000","last_active":"2024-08-03T21:00:49.195479+0000","last_peered":"2024-08-03T21:00:49.195479+0000","last_clean":"2024-08-03T21:00:49.195479+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-08-03T21:00:49.217444+0000","last_undegraded":"2024-08-03T21:00:49.217444+0000","last_fullsized":"2024-08-03T21:00:49.217444+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_clean_scrub_stamp":"2024-08-03T21:00:49.195479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.34","version":"0'0","reported_seq":13,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-08-03T21:00:51.939323+0000","last_change":"2024-08-03T21:00:50.229025+0000","last_active":"2024-08-03T21:00:51.939323+0000","last_peered":"2024-08-03T21:00:51.939323+0000","last_clean":"2024-08-03T21:00:51.939323+0000","last_became_active":"2024-08-03T21:00:50.228698+0000","last_became_peered":"2024-08-03T21:00:50.228698+0000","last_unstale":"2024-08-03T21:00:51.939323+0000","last_undegraded":"2024-08-03T21:00:51.939323+0000","last_fullsized":"2024-08-03T21:00:51.939323+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_clean_scrub_stamp":"2024-08-03T21:00:49.195479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T06:51:09.079780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.33","version":"0'0","reported_seq":13,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-08-03T21:00:51.225611+0000","last_change":"2024-08-03T21:00:50.224093+0000","last_active":"2024-08-03T21:00:51.225611+0000","last_peered":"2024-08-03T21:00:51.225611+0000","last_clean":"2024-08-03T21:00:51.225611+0000","last_became_active":"2024-08-03T21:00:50.222966+0000","last_became_peered":"2024-08-03T21:00:50.222966+0000","last_unstale":"2024-08-03T21:00:51.225611+0000","last_undegraded":"2024-08-03T21:00:51.225611+0000","last_fullsized":"2024-08-03T21:00:51.225611+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_clean_scrub_stamp":"2024-08-03T21:00:49.195479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T04:11:36.038037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.32","version":"0'0","reported_seq":13,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-08-03T21:00:51.937848+0000","last_change":"2024-08-03T21:00:50.230657+0000","last_active":"2024-08-03T21:00:51.937848+0000","last_peered":"2024-08-03T21:00:51.937848+0000","last_clean":"2024-08-03T21:00:51.937848+0000","last_became_active":"2024-08-03T21:00:50.230356+0000","last_became_peered":"2024-08-03T21:00:50.230356+0000","last_unstale":"2024-08-03T21:00:51.937848+0000","last_undegraded":"2024-08-03T21:00:51.937848+0000","last_fullsized":"2024-08-03T21:00:51.937848+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_clean_scrub_stamp":"2024-08-03T21:00:49.195479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-04T21:20:29.060152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.31","version":"0'0","reported_seq":13,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-08-03T21:00:51.225596+0000","last_change":"2024-08-03T21:00:50.236123+0000","last_active":"2024-08-03T21:00:51.225596+0000","last_peered":"2024-08-03T21:00:51.225596+0000","last_clean":"2024-08-03T21:00:51.225596+0000","last_became_active":"2024-08-03T21:00:50.235747+0000","last_became_peered":"2024-08-03T21:00:50.235747+0000","last_unstale":"2024-08-03T21:00:51.225596+0000","last_undegraded":"2024-08-03T21:00:51.225596+0000","last_fullsized":"2024-08-03T21:00:51.225596+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_clean_scrub_stamp":"2024-08-03T21:00:49.195479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T05:25:30.476710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.30","version":"0'0","reported_seq":1,"reported_epoch":133,"state":"creating+peering","last_fresh":"2024-08-03T21:00:49.214934+0000","last_change":"2024-08-03T21:00:49.214934+0000","last_active":"2024-08-03T21:00:49.195479+0000","last_peered":"2024-08-03T21:00:49.195479+0000","last_clean":"2024-08-03T21:00:49.195479+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-08-03T21:00:49.214934+0000","last_undegraded":"2024-08-03T21:00:49.214934+0000","last_fullsized":"2024-08-03T21:00:49.214934+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_clean_scrub_stamp":"2024-08-03T21:00:49.195479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.2f","version":"0'0","reported_seq":1,"reported_epoch":133,"state":"creating+peering","last_fresh":"2024-08-03T21:00:49.220256+0000","last_change":"2024-08-03T21:00:49.220256+0000","last_active":"2024-08-03T21:00:49.195479+0000","last_peered":"2024-08-03T21:00:49.195479+0000","last_clean":"2024-08-03T21:00:49.195479+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-08-03T21:00:49.220256+0000","last_undegraded":"2024-08-03T21:00:49.220256+0000","last_fullsized":"2024-08-03T21:00:49.220256+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_clean_scrub_stamp":"2024-08-03T21:00:49.195479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.2e","version":"0'0","reported_seq":13,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-08-03T21:00:51.225830+0000","last_change":"2024-08-03T21:00:50.241634+0000","last_active":"2024-08-03T21:00:51.225830+0000","last_peered":"2024-08-03T21:00:51.225830+0000","last_clean":"2024-08-03T21:00:51.225830+0000","last_became_active":"2024-08-03T21:00:50.241263+0000","last_became_peered":"2024-08-03T21:00:50.241263+0000","last_unstale":"2024-08-03T21:00:51.225830+0000","last_undegraded":"2024-08-03T21:00:51.225830+0000","last_fullsized":"2024-08-03T21:00:51.225830+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_clean_scrub_stamp":"2024-08-03T21:00:49.195479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T02:07:46.123334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.2d","version":"0'0","reported_seq":13,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-08-03T21:00:51.938842+0000","last_change":"2024-08-03T21:00:50.243910+0000","last_active":"2024-08-03T21:00:51.938842+0000","last_peered":"2024-08-03T21:00:51.938842+0000","last_clean":"2024-08-03T21:00:51.938842+0000","last_became_active":"2024-08-03T21:00:50.243054+0000","last_became_peered":"2024-08-03T21:00:50.243054+0000","last_unstale":"2024-08-03T21:00:51.938842+0000","last_undegraded":"2024-08-03T21:00:51.938842+0000","last_fullsized":"2024-08-03T21:00:51.938842+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_clean_scrub_stamp":"2024-08-03T21:00:49.195479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T08:31:27.530602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.2c","version":"0'0","reported_seq":13,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-08-03T21:00:51.939427+0000","last_change":"2024-08-03T21:00:50.223678+0000","last_active":"2024-08-03T21:00:51.939427+0000","last_peered":"2024-08-03T21:00:51.939427+0000","last_clean":"2024-08-03T21:00:51.939427+0000","last_became_active":"2024-08-03T21:00:50.223357+0000","last_became_peered":"2024-08-03T21:00:50.223357+0000","last_unstale":"2024-08-03T21:00:51.939427+0000","last_undegraded":"2024-08-03T21:00:51.939427+0000","last_fullsized":"2024-08-03T21:00:51.939427+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_clean_scrub_stamp":"2024-08-03T21:00:49.195479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T07:48:35.449963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.2b","version":"0'0","reported_seq":13,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-08-03T21:00:51.939465+0000","last_change":"2024-08-03T21:00:50.243413+0000","last_active":"2024-08-03T21:00:51.939465+0000","last_peered":"2024-08-03T21:00:51.939465+0000","last_clean":"2024-08-03T21:00:51.939465+0000","last_became_active":"2024-08-03T21:00:50.243004+0000","last_became_peered":"2024-08-03T21:00:50.243004+0000","last_unstale":"2024-08-03T21:00:51.939465+0000","last_undegraded":"2024-08-03T21:00:51.939465+0000","last_fullsized":"2024-08-03T21:00:51.939465+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_clean_scrub_stamp":"2024-08-03T21:00:49.195479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-04T23:21:29.132988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.2a","version":"0'0","reported_seq":1,"reported_epoch":133,"state":"creating+peering","last_fresh":"2024-08-03T21:00:49.215684+0000","last_change":"2024-08-03T21:00:49.215684+0000","last_active":"2024-08-03T21:00:49.195479+0000","last_peered":"2024-08-03T21:00:49.195479+0000","last_clean":"2024-08-03T21:00:49.195479+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-08-03T21:00:49.215684+0000","last_undegraded":"2024-08-03T21:00:49.215684+0000","last_fullsized":"2024-08-03T21:00:49.215684+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_clean_scrub_stamp":"2024-08-03T21:00:49.195479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.29","version":"0'0","reported_seq":1,"reported_epoch":133,"state":"creating+peering","last_fresh":"2024-08-03T21:00:49.214421+0000","last_change":"2024-08-03T21:00:49.214421+0000","last_active":"2024-08-03T21:00:49.195479+0000","last_peered":"2024-08-03T21:00:49.195479+0000","last_clean":"2024-08-03T21:00:49.195479+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-08-03T21:00:49.214421+0000","last_undegraded":"2024-08-03T21:00:49.214421+0000","last_fullsized":"2024-08-03T21:00:49.214421+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_clean_scrub_stamp":"2024-08-03T21:00:49.195479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.28","version":"0'0","reported_seq":1,"reported_epoch":133,"state":"creating+peering","last_fresh":"2024-08-03T21:00:49.214489+0000","last_change":"2024-08-03T21:00:49.214489+0000","last_active":"2024-08-03T21:00:49.195479+0000","last_peered":"2024-08-03T21:00:49.195479+0000","last_clean":"2024-08-03T21:00:49.195479+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-08-03T21:00:49.214489+0000","last_undegraded":"2024-08-03T21:00:49.214489+0000","last_fullsized":"2024-08-03T21:00:49.214489+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_clean_scrub_stamp":"2024-08-03T21:00:49.195479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.27","version":"0'0","reported_seq":1,"reported_epoch":133,"state":"creating+peering","last_fresh":"2024-08-03T21:00:49.211525+0000","last_change":"2024-08-03T21:00:49.211525+0000","last_active":"2024-08-03T21:00:49.195479+0000","last_peered":"2024-08-03T21:00:49.195479+0000","last_clean":"2024-08-03T21:00:49.195479+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-08-03T21:00:49.211525+0000","last_undegraded":"2024-08-03T21:00:49.211525+0000","last_fullsized":"2024-08-03T21:00:49.211525+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_clean_scrub_stamp":"2024-08-03T21:00:49.195479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.26","version":"0'0","reported_seq":1,"reported_epoch":133,"state":"creating+peering","last_fresh":"2024-08-03T21:00:49.220518+0000","last_change":"2024-08-03T21:00:49.220518+0000","last_active":"2024-08-03T21:00:49.195479+0000","last_peered":"2024-08-03T21:00:49.195479+0000","last_clean":"2024-08-03T21:00:49.195479+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-08-03T21:00:49.220518+0000","last_undegraded":"2024-08-03T21:00:49.220518+0000","last_fullsized":"2024-08-03T21:00:49.220518+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_clean_scrub_stamp":"2024-08-03T21:00:49.195479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.25","version":"0'0","reported_seq":1,"reported_epoch":133,"state":"creating+peering","last_fresh":"2024-08-03T21:00:49.216934+0000","last_change":"2024-08-03T21:00:49.216934+0000","last_active":"2024-08-03T21:00:49.195479+0000","last_peered":"2024-08-03T21:00:49.195479+0000","last_clean":"2024-08-03T21:00:49.195479+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-08-03T21:00:49.216934+0000","last_undegraded":"2024-08-03T21:00:49.216934+0000","last_fullsized":"2024-08-03T21:00:49.216934+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_clean_scrub_stamp":"2024-08-03T21:00:49.195479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.24","version":"0'0","reported_seq":1,"reported_epoch":133,"state":"creating+peering","last_fresh":"2024-08-03T21:00:49.212944+0000","last_change":"2024-08-03T21:00:49.212944+0000","last_active":"2024-08-03T21:00:49.195479+0000","last_peered":"2024-08-03T21:00:49.195479+0000","last_clean":"2024-08-03T21:00:49.195479+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-08-03T21:00:49.212944+0000","last_undegraded":"2024-08-03T21:00:49.212944+0000","last_fullsized":"2024-08-03T21:00:49.212944+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_clean_scrub_stamp":"2024-08-03T21:00:49.195479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.23","version":"0'0","reported_seq":1,"reported_epoch":133,"state":"creating+peering","last_fresh":"2024-08-03T21:00:49.219217+0000","last_change":"2024-08-03T21:00:49.219217+0000","last_active":"2024-08-03T21:00:49.195479+0000","last_peered":"2024-08-03T21:00:49.195479+0000","last_clean":"2024-08-03T21:00:49.195479+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-08-03T21:00:49.219217+0000","last_undegraded":"2024-08-03T21:00:49.219217+0000","last_fullsized":"2024-08-03T21:00:49.219217+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_clean_scrub_stamp":"2024-08-03T21:00:49.195479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.22","version":"0'0","reported_seq":13,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-08-03T21:00:51.938224+0000","last_change":"2024-08-03T21:00:50.243735+0000","last_active":"2024-08-03T21:00:51.938224+0000","last_peered":"2024-08-03T21:00:51.938224+0000","last_clean":"2024-08-03T21:00:51.938224+0000","last_became_active":"2024-08-03T21:00:50.243333+0000","last_became_peered":"2024-08-03T21:00:50.243333+0000","last_unstale":"2024-08-03T21:00:51.938224+0000","last_undegraded":"2024-08-03T21:00:51.938224+0000","last_fullsized":"2024-08-03T21:00:51.938224+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_clean_scrub_stamp":"2024-08-03T21:00:49.195479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T08:50:30.366065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.21","version":"0'0","reported_seq":13,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-08-03T21:00:51.225546+0000","last_change":"2024-08-03T21:00:50.229431+0000","last_active":"2024-08-03T21:00:51.225546+0000","last_peered":"2024-08-03T21:00:51.225546+0000","last_clean":"2024-08-03T21:00:51.225546+0000","last_became_active":"2024-08-03T21:00:50.228793+0000","last_became_peered":"2024-08-03T21:00:50.228793+0000","last_unstale":"2024-08-03T21:00:51.225546+0000","last_undegraded":"2024-08-03T21:00:51.225546+0000","last_fullsized":"2024-08-03T21:00:51.225546+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_clean_scrub_stamp":"2024-08-03T21:00:49.195479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T03:21:17.046927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.20","version":"0'0","reported_seq":1,"reported_epoch":133,"state":"creating+peering","last_fresh":"2024-08-03T21:00:49.213774+0000","last_change":"2024-08-03T21:00:49.213774+0000","last_active":"2024-08-03T21:00:49.195479+0000","last_peered":"2024-08-03T21:00:49.195479+0000","last_clean":"2024-08-03T21:00:49.195479+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-08-03T21:00:49.213774+0000","last_undegraded":"2024-08-03T21:00:49.213774+0000","last_fullsized":"2024-08-03T21:00:49.213774+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_clean_scrub_stamp":"2024-08-03T21:00:49.195479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.1e","version":"0'0","reported_seq":13,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-08-03T21:00:51.939274+0000","last_change":"2024-08-03T21:00:50.235759+0000","last_active":"2024-08-03T21:00:51.939274+0000","last_peered":"2024-08-03T21:00:51.939274+0000","last_clean":"2024-08-03T21:00:51.939274+0000","last_became_active":"2024-08-03T21:00:50.235440+0000","last_became_peered":"2024-08-03T21:00:50.235440+0000","last_unstale":"2024-08-03T21:00:51.939274+0000","last_undegraded":"2024-08-03T21:00:51.939274+0000","last_fullsized":"2024-08-03T21:00:51.939274+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_clean_scrub_stamp":"2024-08-03T21:00:49.195479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-04T21:51:33.128276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.1f","version":"0'0","reported_seq":10,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-08-03T21:00:49.232640+0000","last_change":"2024-08-03T21:00:49.231401+0000","last_active":"2024-08-03T21:00:49.232640+0000","last_peered":"2024-08-03T21:00:49.232640+0000","last_clean":"2024-08-03T21:00:49.232640+0000","last_became_active":"2024-08-03T21:00:49.231089+0000","last_became_peered":"2024-08-03T21:00:49.231089+0000","last_unstale":"2024-08-03T21:00:49.232640+0000","last_undegraded":"2024-08-03T21:00:49.232640+0000","last_fullsized":"2024-08-03T21:00:49.232640+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:48.192422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:48.192422+0000","last_clean_scrub_stamp":"2024-08-03T21:00:48.192422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T01:29:44.046851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.1f","version":"0'0","reported_seq":13,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-08-03T21:00:51.225857+0000","last_change":"2024-08-03T21:00:50.226490+0000","last_active":"2024-08-03T21:00:51.225857+0000","last_peered":"2024-08-03T21:00:51.225857+0000","last_clean":"2024-08-03T21:00:51.225857+0000","last_became_active":"2024-08-03T21:00:50.226103+0000","last_became_peered":"2024-08-03T21:00:50.226103+0000","last_unstale":"2024-08-03T21:00:51.225857+0000","last_undegraded":"2024-08-03T21:00:51.225857+0000","last_fullsized":"2024-08-03T21:00:51.225857+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_clean_scrub_stamp":"2024-08-03T21:00:49.195479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T04:51:31.368175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.1e","version":"0'0","reported_seq":10,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-08-03T21:00:49.228209+0000","last_change":"2024-08-03T21:00:49.227358+0000","last_active":"2024-08-03T21:00:49.228209+0000","last_peered":"2024-08-03T21:00:49.228209+0000","last_clean":"2024-08-03T21:00:49.228209+0000","last_became_active":"2024-08-03T21:00:49.226904+0000","last_became_peered":"2024-08-03T21:00:49.226904+0000","last_unstale":"2024-08-03T21:00:49.228209+0000","last_undegraded":"2024-08-03T21:00:49.228209+0000","last_fullsized":"2024-08-03T21:00:49.228209+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:48.192422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:48.192422+0000","last_clean_scrub_stamp":"2024-08-03T21:00:48.192422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T07:44:13.158341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.1c","version":"0'0","reported_seq":1,"reported_epoch":133,"state":"creating+peering","last_fresh":"2024-08-03T21:00:49.214791+0000","last_change":"2024-08-03T21:00:49.214791+0000","last_active":"2024-08-03T21:00:49.195479+0000","last_peered":"2024-08-03T21:00:49.195479+0000","last_clean":"2024-08-03T21:00:49.195479+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-08-03T21:00:49.214791+0000","last_undegraded":"2024-08-03T21:00:49.214791+0000","last_fullsized":"2024-08-03T21:00:49.214791+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_clean_scrub_stamp":"2024-08-03T21:00:49.195479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.1d","version":"0'0","reported_seq":10,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-08-03T21:00:49.238854+0000","last_change":"2024-08-03T21:00:49.234514+0000","last_active":"2024-08-03T21:00:49.238854+0000","last_peered":"2024-08-03T21:00:49.238854+0000","last_clean":"2024-08-03T21:00:49.238854+0000","last_became_active":"2024-08-03T21:00:49.233091+0000","last_became_peered":"2024-08-03T21:00:49.233091+0000","last_unstale":"2024-08-03T21:00:49.238854+0000","last_undegraded":"2024-08-03T21:00:49.238854+0000","last_fullsized":"2024-08-03T21:00:49.238854+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:48.192422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:48.192422+0000","last_clean_scrub_stamp":"2024-08-03T21:00:48.192422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T02:44:09.712432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.1d","version":"0'0","reported_seq":13,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-08-03T21:00:51.939823+0000","last_change":"2024-08-03T21:00:50.235731+0000","last_active":"2024-08-03T21:00:51.939823+0000","last_peered":"2024-08-03T21:00:51.939823+0000","last_clean":"2024-08-03T21:00:51.939823+0000","last_became_active":"2024-08-03T21:00:50.234805+0000","last_became_peered":"2024-08-03T21:00:50.234805+0000","last_unstale":"2024-08-03T21:00:51.939823+0000","last_undegraded":"2024-08-03T21:00:51.939823+0000","last_fullsized":"2024-08-03T21:00:51.939823+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_clean_scrub_stamp":"2024-08-03T21:00:49.195479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T05:40:57.488842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.1c","version":"0'0","reported_seq":10,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-08-03T21:00:49.230596+0000","last_change":"2024-08-03T21:00:49.227739+0000","last_active":"2024-08-03T21:00:49.230596+0000","last_peered":"2024-08-03T21:00:49.230596+0000","last_clean":"2024-08-03T21:00:49.230596+0000","last_became_active":"2024-08-03T21:00:49.226569+0000","last_became_peered":"2024-08-03T21:00:49.226569+0000","last_unstale":"2024-08-03T21:00:49.230596+0000","last_undegraded":"2024-08-03T21:00:49.230596+0000","last_fullsized":"2024-08-03T21:00:49.230596+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:48.192422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:48.192422+0000","last_clean_scrub_stamp":"2024-08-03T21:00:48.192422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T08:47:16.511690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.1a","version":"0'0","reported_seq":1,"reported_epoch":133,"state":"creating+peering","last_fresh":"2024-08-03T21:00:49.211222+0000","last_change":"2024-08-03T21:00:49.211222+0000","last_active":"2024-08-03T21:00:49.195479+0000","last_peered":"2024-08-03T21:00:49.195479+0000","last_clean":"2024-08-03T21:00:49.195479+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-08-03T21:00:49.211222+0000","last_undegraded":"2024-08-03T21:00:49.211222+0000","last_fullsized":"2024-08-03T21:00:49.211222+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_clean_scrub_stamp":"2024-08-03T21:00:49.195479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.1b","version":"0'0","reported_seq":10,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-08-03T21:00:49.220521+0000","last_change":"2024-08-03T21:00:49.217597+0000","last_active":"2024-08-03T21:00:49.220521+0000","last_peered":"2024-08-03T21:00:49.220521+0000","last_clean":"2024-08-03T21:00:49.220521+0000","last_became_active":"2024-08-03T21:00:49.217212+0000","last_became_peered":"2024-08-03T21:00:49.217212+0000","last_unstale":"2024-08-03T21:00:49.220521+0000","last_undegraded":"2024-08-03T21:00:49.220521+0000","last_fullsized":"2024-08-03T21:00:49.220521+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:48.192422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:48.192422+0000","last_clean_scrub_stamp":"2024-08-03T21:00:48.192422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-04T22:22:37.333147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.1b","version":"0'0","reported_seq":1,"reported_epoch":133,"state":"creating+peering","last_fresh":"2024-08-03T21:00:49.215473+0000","last_change":"2024-08-03T21:00:49.215473+0000","last_active":"2024-08-03T21:00:49.195479+0000","last_peered":"2024-08-03T21:00:49.195479+0000","last_clean":"2024-08-03T21:00:49.195479+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-08-03T21:00:49.215473+0000","last_undegraded":"2024-08-03T21:00:49.215473+0000","last_fullsized":"2024-08-03T21:00:49.215473+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_clean_scrub_stamp":"2024-08-03T21:00:49.195479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.1a","version":"0'0","reported_seq":10,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-08-03T21:00:49.239275+0000","last_change":"2024-08-03T21:00:49.238475+0000","last_active":"2024-08-03T21:00:49.239275+0000","last_peered":"2024-08-03T21:00:49.239275+0000","last_clean":"2024-08-03T21:00:49.239275+0000","last_became_active":"2024-08-03T21:00:49.238001+0000","last_became_peered":"2024-08-03T21:00:49.238001+0000","last_unstale":"2024-08-03T21:00:49.239275+0000","last_undegraded":"2024-08-03T21:00:49.239275+0000","last_fullsized":"2024-08-03T21:00:49.239275+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:48.192422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:48.192422+0000","last_clean_scrub_stamp":"2024-08-03T21:00:48.192422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T07:44:13.158341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.18","version":"0'0","reported_seq":13,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-08-03T21:00:51.225613+0000","last_change":"2024-08-03T21:00:50.228305+0000","last_active":"2024-08-03T21:00:51.225613+0000","last_peered":"2024-08-03T21:00:51.225613+0000","last_clean":"2024-08-03T21:00:51.225613+0000","last_became_active":"2024-08-03T21:00:50.228009+0000","last_became_peered":"2024-08-03T21:00:50.228009+0000","last_unstale":"2024-08-03T21:00:51.225613+0000","last_undegraded":"2024-08-03T21:00:51.225613+0000","last_fullsized":"2024-08-03T21:00:51.225613+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_clean_scrub_stamp":"2024-08-03T21:00:49.195479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-04T23:22:25.975528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.19","version":"0'0","reported_seq":10,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-08-03T21:00:49.229663+0000","last_change":"2024-08-03T21:00:49.228986+0000","last_active":"2024-08-03T21:00:49.229663+0000","last_peered":"2024-08-03T21:00:49.229663+0000","last_clean":"2024-08-03T21:00:49.229663+0000","last_became_active":"2024-08-03T21:00:49.228761+0000","last_became_peered":"2024-08-03T21:00:49.228761+0000","last_unstale":"2024-08-03T21:00:49.229663+0000","last_undegraded":"2024-08-03T21:00:49.229663+0000","last_fullsized":"2024-08-03T21:00:49.229663+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:48.192422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:48.192422+0000","last_clean_scrub_stamp":"2024-08-03T21:00:48.192422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T02:44:18.436142+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.19","version":"0'0","reported_seq":13,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-08-03T21:00:51.940174+0000","last_change":"2024-08-03T21:00:50.235335+0000","last_active":"2024-08-03T21:00:51.940174+0000","last_peered":"2024-08-03T21:00:51.940174+0000","last_clean":"2024-08-03T21:00:51.940174+0000","last_became_active":"2024-08-03T21:00:50.234940+0000","last_became_peered":"2024-08-03T21:00:50.234940+0000","last_unstale":"2024-08-03T21:00:51.940174+0000","last_undegraded":"2024-08-03T21:00:51.940174+0000","last_fullsized":"2024-08-03T21:00:51.940174+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_clean_scrub_stamp":"2024-08-03T21:00:49.195479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T00:40:44.403536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.18","version":"0'0","reported_seq":10,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-08-03T21:00:49.237874+0000","last_change":"2024-08-03T21:00:49.236558+0000","last_active":"2024-08-03T21:00:49.237874+0000","last_peered":"2024-08-03T21:00:49.237874+0000","last_clean":"2024-08-03T21:00:49.237874+0000","last_became_active":"2024-08-03T21:00:49.236201+0000","last_became_peered":"2024-08-03T21:00:49.236201+0000","last_unstale":"2024-08-03T21:00:49.237874+0000","last_undegraded":"2024-08-03T21:00:49.237874+0000","last_fullsized":"2024-08-03T21:00:49.237874+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:48.192422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:48.192422+0000","last_clean_scrub_stamp":"2024-08-03T21:00:48.192422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-04T23:01:53.386452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.16","version":"0'0","reported_seq":1,"reported_epoch":133,"state":"creating+peering","last_fresh":"2024-08-03T21:00:49.219922+0000","last_change":"2024-08-03T21:00:49.219922+0000","last_active":"2024-08-03T21:00:49.195479+0000","last_peered":"2024-08-03T21:00:49.195479+0000","last_clean":"2024-08-03T21:00:49.195479+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-08-03T21:00:49.219922+0000","last_undegraded":"2024-08-03T21:00:49.219922+0000","last_fullsized":"2024-08-03T21:00:49.219922+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_clean_scrub_stamp":"2024-08-03T21:00:49.195479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.17","version":"0'0","reported_seq":10,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-08-03T21:00:49.229166+0000","last_change":"2024-08-03T21:00:49.226248+0000","last_active":"2024-08-03T21:00:49.229166+0000","last_peered":"2024-08-03T21:00:49.229166+0000","last_clean":"2024-08-03T21:00:49.229166+0000","last_became_active":"2024-08-03T21:00:49.224012+0000","last_became_peered":"2024-08-03T21:00:49.224012+0000","last_unstale":"2024-08-03T21:00:49.229166+0000","last_undegraded":"2024-08-03T21:00:49.229166+0000","last_fullsized":"2024-08-03T21:00:49.229166+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:48.192422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:48.192422+0000","last_clean_scrub_stamp":"2024-08-03T21:00:48.192422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T02:58:35.310272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.17","version":"0'0","reported_seq":13,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-08-03T21:00:51.940036+0000","last_change":"2024-08-03T21:00:50.244964+0000","last_active":"2024-08-03T21:00:51.940036+0000","last_peered":"2024-08-03T21:00:51.940036+0000","last_clean":"2024-08-03T21:00:51.940036+0000","last_became_active":"2024-08-03T21:00:50.244323+0000","last_became_peered":"2024-08-03T21:00:50.244323+0000","last_unstale":"2024-08-03T21:00:51.940036+0000","last_undegraded":"2024-08-03T21:00:51.940036+0000","last_fullsized":"2024-08-03T21:00:51.940036+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_clean_scrub_stamp":"2024-08-03T21:00:49.195479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-04T21:46:43.719103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.16","version":"0'0","reported_seq":15,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-08-03T21:00:51.225728+0000","last_change":"2024-08-03T21:00:49.233357+0000","last_active":"2024-08-03T21:00:51.225728+0000","last_peered":"2024-08-03T21:00:51.225728+0000","last_clean":"2024-08-03T21:00:51.225728+0000","last_became_active":"2024-08-03T21:00:49.232477+0000","last_became_peered":"2024-08-03T21:00:49.232477+0000","last_unstale":"2024-08-03T21:00:51.225728+0000","last_undegraded":"2024-08-03T21:00:51.225728+0000","last_fullsized":"2024-08-03T21:00:51.225728+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:48.192422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:48.192422+0000","last_clean_scrub_stamp":"2024-08-03T21:00:48.192422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T05:40:56.485784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.14","version":"0'0","reported_seq":1,"reported_epoch":133,"state":"creating+peering","last_fresh":"2024-08-03T21:00:49.210789+0000","last_change":"2024-08-03T21:00:49.210789+0000","last_active":"2024-08-03T21:00:49.195479+0000","last_peered":"2024-08-03T21:00:49.195479+0000","last_clean":"2024-08-03T21:00:49.195479+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-08-03T21:00:49.210789+0000","last_undegraded":"2024-08-03T21:00:49.210789+0000","last_fullsized":"2024-08-03T21:00:49.210789+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_clean_scrub_stamp":"2024-08-03T21:00:49.195479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.15","version":"0'0","reported_seq":10,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-08-03T21:00:49.225338+0000","last_change":"2024-08-03T21:00:49.224739+0000","last_active":"2024-08-03T21:00:49.225338+0000","last_peered":"2024-08-03T21:00:49.225338+0000","last_clean":"2024-08-03T21:00:49.225338+0000","last_became_active":"2024-08-03T21:00:49.224342+0000","last_became_peered":"2024-08-03T21:00:49.224342+0000","last_unstale":"2024-08-03T21:00:49.225338+0000","last_undegraded":"2024-08-03T21:00:49.225338+0000","last_fullsized":"2024-08-03T21:00:49.225338+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:48.192422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:48.192422+0000","last_clean_scrub_stamp":"2024-08-03T21:00:48.192422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-04T23:53:41.240790+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.15","version":"0'0","reported_seq":1,"reported_epoch":133,"state":"creating+peering","last_fresh":"2024-08-03T21:00:49.216465+0000","last_change":"2024-08-03T21:00:49.216465+0000","last_active":"2024-08-03T21:00:49.195479+0000","last_peered":"2024-08-03T21:00:49.195479+0000","last_clean":"2024-08-03T21:00:49.195479+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-08-03T21:00:49.216465+0000","last_undegraded":"2024-08-03T21:00:49.216465+0000","last_fullsized":"2024-08-03T21:00:49.216465+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_clean_scrub_stamp":"2024-08-03T21:00:49.195479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.14","version":"134'6","reported_seq":21,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-08-03T21:00:51.939402+0000","last_change":"2024-08-03T21:00:49.235410+0000","last_active":"2024-08-03T21:00:51.939402+0000","last_peered":"2024-08-03T21:00:51.939402+0000","last_clean":"2024-08-03T21:00:51.939402+0000","last_became_active":"2024-08-03T21:00:49.234999+0000","last_became_peered":"2024-08-03T21:00:49.234999+0000","last_unstale":"2024-08-03T21:00:51.939402+0000","last_undegraded":"2024-08-03T21:00:51.939402+0000","last_fullsized":"2024-08-03T21:00:51.939402+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:48.192422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:48.192422+0000","last_clean_scrub_stamp":"2024-08-03T21:00:48.192422+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T02:55:58.470527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.12","version":"0'0","reported_seq":13,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-08-03T21:00:51.225979+0000","last_change":"2024-08-03T21:00:50.218292+0000","last_active":"2024-08-03T21:00:51.225979+0000","last_peered":"2024-08-03T21:00:51.225979+0000","last_clean":"2024-08-03T21:00:51.225979+0000","last_became_active":"2024-08-03T21:00:50.217979+0000","last_became_peered":"2024-08-03T21:00:50.217979+0000","last_unstale":"2024-08-03T21:00:51.225979+0000","last_undegraded":"2024-08-03T21:00:51.225979+0000","last_fullsized":"2024-08-03T21:00:51.225979+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_clean_scrub_stamp":"2024-08-03T21:00:49.195479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T06:55:48.820648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.13","version":"134'2","reported_seq":17,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-08-03T21:00:51.939709+0000","last_change":"2024-08-03T21:00:49.224867+0000","last_active":"2024-08-03T21:00:51.939709+0000","last_peered":"2024-08-03T21:00:51.939709+0000","last_clean":"2024-08-03T21:00:51.939709+0000","last_became_active":"2024-08-03T21:00:49.224555+0000","last_became_peered":"2024-08-03T21:00:49.224555+0000","last_unstale":"2024-08-03T21:00:51.939709+0000","last_undegraded":"2024-08-03T21:00:51.939709+0000","last_fullsized":"2024-08-03T21:00:51.939709+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:48.192422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:48.192422+0000","last_clean_scrub_stamp":"2024-08-03T21:00:48.192422+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T07:55:25.225889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.13","version":"0'0","reported_seq":1,"reported_epoch":133,"state":"creating+peering","last_fresh":"2024-08-03T21:00:49.211586+0000","last_change":"2024-08-03T21:00:49.211586+0000","last_active":"2024-08-03T21:00:49.195479+0000","last_peered":"2024-08-03T21:00:49.195479+0000","last_clean":"2024-08-03T21:00:49.195479+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-08-03T21:00:49.211586+0000","last_undegraded":"2024-08-03T21:00:49.211586+0000","last_fullsized":"2024-08-03T21:00:49.211586+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_clean_scrub_stamp":"2024-08-03T21:00:49.195479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.12","version":"0'0","reported_seq":15,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-08-03T21:00:51.938928+0000","last_change":"2024-08-03T21:00:49.224078+0000","last_active":"2024-08-03T21:00:51.938928+0000","last_peered":"2024-08-03T21:00:51.938928+0000","last_clean":"2024-08-03T21:00:51.938928+0000","last_became_active":"2024-08-03T21:00:49.223717+0000","last_became_peered":"2024-08-03T21:00:49.223717+0000","last_unstale":"2024-08-03T21:00:51.938928+0000","last_undegraded":"2024-08-03T21:00:51.938928+0000","last_fullsized":"2024-08-03T21:00:51.938928+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:48.192422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:48.192422+0000","last_clean_scrub_stamp":"2024-08-03T21:00:48.192422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T04:56:22.864880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.10","version":"0'0","reported_seq":13,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-08-03T21:00:51.939036+0000","last_change":"2024-08-03T21:00:50.231767+0000","last_active":"2024-08-03T21:00:51.939036+0000","last_peered":"2024-08-03T21:00:51.939036+0000","last_clean":"2024-08-03T21:00:51.939036+0000","last_became_active":"2024-08-03T21:00:50.229726+0000","last_became_peered":"2024-08-03T21:00:50.229726+0000","last_unstale":"2024-08-03T21:00:51.939036+0000","last_undegraded":"2024-08-03T21:00:51.939036+0000","last_fullsized":"2024-08-03T21:00:51.939036+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_clean_scrub_stamp":"2024-08-03T21:00:49.195479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T00:14:18.409160+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.11","version":"0'0","reported_seq":15,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-08-03T21:00:51.939969+0000","last_change":"2024-08-03T21:00:49.219205+0000","last_active":"2024-08-03T21:00:51.939969+0000","last_peered":"2024-08-03T21:00:51.939969+0000","last_clean":"2024-08-03T21:00:51.939969+0000","last_became_active":"2024-08-03T21:00:49.218855+0000","last_became_peered":"2024-08-03T21:00:49.218855+0000","last_unstale":"2024-08-03T21:00:51.939969+0000","last_undegraded":"2024-08-03T21:00:51.939969+0000","last_fullsized":"2024-08-03T21:00:51.939969+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:48.192422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:48.192422+0000","last_clean_scrub_stamp":"2024-08-03T21:00:48.192422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T00:24:17.141394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.11","version":"0'0","reported_seq":13,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-08-03T21:00:51.939150+0000","last_change":"2024-08-03T21:00:50.230042+0000","last_active":"2024-08-03T21:00:51.939150+0000","last_peered":"2024-08-03T21:00:51.939150+0000","last_clean":"2024-08-03T21:00:51.939150+0000","last_became_active":"2024-08-03T21:00:50.229716+0000","last_became_peered":"2024-08-03T21:00:50.229716+0000","last_unstale":"2024-08-03T21:00:51.939150+0000","last_undegraded":"2024-08-03T21:00:51.939150+0000","last_fullsized":"2024-08-03T21:00:51.939150+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_clean_scrub_stamp":"2024-08-03T21:00:49.195479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T08:05:48.492097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.10","version":"0'0","reported_seq":10,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-08-03T21:00:49.222553+0000","last_change":"2024-08-03T21:00:49.219610+0000","last_active":"2024-08-03T21:00:49.222553+0000","last_peered":"2024-08-03T21:00:49.222553+0000","last_clean":"2024-08-03T21:00:49.222553+0000","last_became_active":"2024-08-03T21:00:49.219289+0000","last_became_peered":"2024-08-03T21:00:49.219289+0000","last_unstale":"2024-08-03T21:00:49.222553+0000","last_undegraded":"2024-08-03T21:00:49.222553+0000","last_fullsized":"2024-08-03T21:00:49.222553+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:48.192422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:48.192422+0000","last_clean_scrub_stamp":"2024-08-03T21:00:48.192422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-04T22:10:58.211954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.f","version":"0'0","reported_seq":1,"reported_epoch":133,"state":"creating+peering","last_fresh":"2024-08-03T21:00:49.209245+0000","last_change":"2024-08-03T21:00:49.209245+0000","last_active":"2024-08-03T21:00:49.195479+0000","last_peered":"2024-08-03T21:00:49.195479+0000","last_clean":"2024-08-03T21:00:49.195479+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-08-03T21:00:49.209245+0000","last_undegraded":"2024-08-03T21:00:49.209245+0000","last_fullsized":"2024-08-03T21:00:49.209245+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_clean_scrub_stamp":"2024-08-03T21:00:49.195479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.e","version":"134'2","reported_seq":17,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-08-03T21:00:51.939110+0000","last_change":"2024-08-03T21:00:49.221709+0000","last_active":"2024-08-03T21:00:51.939110+0000","last_peered":"2024-08-03T21:00:51.939110+0000","last_clean":"2024-08-03T21:00:51.939110+0000","last_became_active":"2024-08-03T21:00:49.221146+0000","last_became_peered":"2024-08-03T21:00:49.221146+0000","last_unstale":"2024-08-03T21:00:51.939110+0000","last_undegraded":"2024-08-03T21:00:51.939110+0000","last_fullsized":"2024-08-03T21:00:51.939110+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:48.192422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:48.192422+0000","last_clean_scrub_stamp":"2024-08-03T21:00:48.192422+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T02:29:53.585049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.c","version":"0'0","reported_seq":1,"reported_epoch":133,"state":"creating+peering","last_fresh":"2024-08-03T21:00:49.211956+0000","last_change":"2024-08-03T21:00:49.211956+0000","last_active":"2024-08-03T21:00:49.195479+0000","last_peered":"2024-08-03T21:00:49.195479+0000","last_clean":"2024-08-03T21:00:49.195479+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-08-03T21:00:49.211956+0000","last_undegraded":"2024-08-03T21:00:49.211956+0000","last_fullsized":"2024-08-03T21:00:49.211956+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_clean_scrub_stamp":"2024-08-03T21:00:49.195479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.d","version":"0'0","reported_seq":10,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-08-03T21:00:49.216969+0000","last_change":"2024-08-03T21:00:49.214725+0000","last_active":"2024-08-03T21:00:49.216969+0000","last_peered":"2024-08-03T21:00:49.216969+0000","last_clean":"2024-08-03T21:00:49.216969+0000","last_became_active":"2024-08-03T21:00:49.214155+0000","last_became_peered":"2024-08-03T21:00:49.214155+0000","last_unstale":"2024-08-03T21:00:49.216969+0000","last_undegraded":"2024-08-03T21:00:49.216969+0000","last_fullsized":"2024-08-03T21:00:49.216969+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:48.192422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:48.192422+0000","last_clean_scrub_stamp":"2024-08-03T21:00:48.192422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T07:43:13.286648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.d","version":"0'0","reported_seq":1,"reported_epoch":133,"state":"creating+peering","last_fresh":"2024-08-03T21:00:49.213316+0000","last_change":"2024-08-03T21:00:49.213316+0000","last_active":"2024-08-03T21:00:49.195479+0000","last_peered":"2024-08-03T21:00:49.195479+0000","last_clean":"2024-08-03T21:00:49.195479+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-08-03T21:00:49.213316+0000","last_undegraded":"2024-08-03T21:00:49.213316+0000","last_fullsized":"2024-08-03T21:00:49.213316+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_clean_scrub_stamp":"2024-08-03T21:00:49.195479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.c","version":"0'0","reported_seq":10,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-08-03T21:00:49.230569+0000","last_change":"2024-08-03T21:00:49.225810+0000","last_active":"2024-08-03T21:00:49.230569+0000","last_peered":"2024-08-03T21:00:49.230569+0000","last_clean":"2024-08-03T21:00:49.230569+0000","last_became_active":"2024-08-03T21:00:49.225501+0000","last_became_peered":"2024-08-03T21:00:49.225501+0000","last_unstale":"2024-08-03T21:00:49.230569+0000","last_undegraded":"2024-08-03T21:00:49.230569+0000","last_fullsized":"2024-08-03T21:00:49.230569+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:48.192422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:48.192422+0000","last_clean_scrub_stamp":"2024-08-03T21:00:48.192422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-04T23:49:05.539690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.a","version":"0'0","reported_seq":13,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-08-03T21:00:51.938230+0000","last_change":"2024-08-03T21:00:50.238896+0000","last_active":"2024-08-03T21:00:51.938230+0000","last_peered":"2024-08-03T21:00:51.938230+0000","last_clean":"2024-08-03T21:00:51.938230+0000","last_became_active":"2024-08-03T21:00:50.238557+0000","last_became_peered":"2024-08-03T21:00:50.238557+0000","last_unstale":"2024-08-03T21:00:51.938230+0000","last_undegraded":"2024-08-03T21:00:51.938230+0000","last_fullsized":"2024-08-03T21:00:51.938230+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_clean_scrub_stamp":"2024-08-03T21:00:49.195479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-04T22:10:38.257913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.b","version":"0'0","reported_seq":10,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-08-03T21:00:49.238315+0000","last_change":"2024-08-03T21:00:49.233291+0000","last_active":"2024-08-03T21:00:49.238315+0000","last_peered":"2024-08-03T21:00:49.238315+0000","last_clean":"2024-08-03T21:00:49.238315+0000","last_became_active":"2024-08-03T21:00:49.232973+0000","last_became_peered":"2024-08-03T21:00:49.232973+0000","last_unstale":"2024-08-03T21:00:49.238315+0000","last_undegraded":"2024-08-03T21:00:49.238315+0000","last_fullsized":"2024-08-03T21:00:49.238315+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:48.192422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:48.192422+0000","last_clean_scrub_stamp":"2024-08-03T21:00:48.192422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T02:44:18.436142+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.b","version":"0'0","reported_seq":13,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-08-03T21:00:51.939647+0000","last_change":"2024-08-03T21:00:50.242588+0000","last_active":"2024-08-03T21:00:51.939647+0000","last_peered":"2024-08-03T21:00:51.939647+0000","last_clean":"2024-08-03T21:00:51.939647+0000","last_became_active":"2024-08-03T21:00:50.239795+0000","last_became_peered":"2024-08-03T21:00:50.239795+0000","last_unstale":"2024-08-03T21:00:51.939647+0000","last_undegraded":"2024-08-03T21:00:51.939647+0000","last_fullsized":"2024-08-03T21:00:51.939647+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_clean_scrub_stamp":"2024-08-03T21:00:49.195479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T06:53:11.234357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.a","version":"0'0","reported_seq":10,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-08-03T21:00:49.223212+0000","last_change":"2024-08-03T21:00:49.222066+0000","last_active":"2024-08-03T21:00:49.223212+0000","last_peered":"2024-08-03T21:00:49.223212+0000","last_clean":"2024-08-03T21:00:49.223212+0000","last_became_active":"2024-08-03T21:00:49.221757+0000","last_became_peered":"2024-08-03T21:00:49.221757+0000","last_unstale":"2024-08-03T21:00:49.223212+0000","last_undegraded":"2024-08-03T21:00:49.223212+0000","last_fullsized":"2024-08-03T21:00:49.223212+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:48.192422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:48.192422+0000","last_clean_scrub_stamp":"2024-08-03T21:00:48.192422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T08:47:16.511690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.8","version":"0'0","reported_seq":13,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-08-03T21:00:51.937406+0000","last_change":"2024-08-03T21:00:50.240625+0000","last_active":"2024-08-03T21:00:51.937406+0000","last_peered":"2024-08-03T21:00:51.937406+0000","last_clean":"2024-08-03T21:00:51.937406+0000","last_became_active":"2024-08-03T21:00:50.240139+0000","last_became_peered":"2024-08-03T21:00:50.240139+0000","last_unstale":"2024-08-03T21:00:51.937406+0000","last_undegraded":"2024-08-03T21:00:51.937406+0000","last_fullsized":"2024-08-03T21:00:51.937406+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_clean_scrub_stamp":"2024-08-03T21:00:49.195479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T06:58:48.625406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.9","version":"0'0","reported_seq":10,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-08-03T21:00:49.236203+0000","last_change":"2024-08-03T21:00:49.235212+0000","last_active":"2024-08-03T21:00:49.236203+0000","last_peered":"2024-08-03T21:00:49.236203+0000","last_clean":"2024-08-03T21:00:49.236203+0000","last_became_active":"2024-08-03T21:00:49.234793+0000","last_became_peered":"2024-08-03T21:00:49.234793+0000","last_unstale":"2024-08-03T21:00:49.236203+0000","last_undegraded":"2024-08-03T21:00:49.236203+0000","last_fullsized":"2024-08-03T21:00:49.236203+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:48.192422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:48.192422+0000","last_clean_scrub_stamp":"2024-08-03T21:00:48.192422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T07:43:13.286648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.9","version":"0'0","reported_seq":1,"reported_epoch":133,"state":"creating+peering","last_fresh":"2024-08-03T21:00:49.213171+0000","last_change":"2024-08-03T21:00:49.213171+0000","last_active":"2024-08-03T21:00:49.195479+0000","last_peered":"2024-08-03T21:00:49.195479+0000","last_clean":"2024-08-03T21:00:49.195479+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-08-03T21:00:49.213171+0000","last_undegraded":"2024-08-03T21:00:49.213171+0000","last_fullsized":"2024-08-03T21:00:49.213171+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_clean_scrub_stamp":"2024-08-03T21:00:49.195479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.8","version":"0'0","reported_seq":10,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-08-03T21:00:49.234238+0000","last_change":"2024-08-03T21:00:49.230326+0000","last_active":"2024-08-03T21:00:49.234238+0000","last_peered":"2024-08-03T21:00:49.234238+0000","last_clean":"2024-08-03T21:00:49.234238+0000","last_became_active":"2024-08-03T21:00:49.229867+0000","last_became_peered":"2024-08-03T21:00:49.229867+0000","last_unstale":"2024-08-03T21:00:49.234238+0000","last_undegraded":"2024-08-03T21:00:49.234238+0000","last_fullsized":"2024-08-03T21:00:49.234238+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:48.192422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:48.192422+0000","last_clean_scrub_stamp":"2024-08-03T21:00:48.192422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T05:44:38.082933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.6","version":"0'0","reported_seq":1,"reported_epoch":133,"state":"creating+peering","last_fresh":"2024-08-03T21:00:49.217620+0000","last_change":"2024-08-03T21:00:49.217620+0000","last_active":"2024-08-03T21:00:49.195479+0000","last_peered":"2024-08-03T21:00:49.195479+0000","last_clean":"2024-08-03T21:00:49.195479+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-08-03T21:00:49.217620+0000","last_undegraded":"2024-08-03T21:00:49.217620+0000","last_fullsized":"2024-08-03T21:00:49.217620+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_clean_scrub_stamp":"2024-08-03T21:00:49.195479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.7","version":"0'0","reported_seq":10,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-08-03T21:00:49.223032+0000","last_change":"2024-08-03T21:00:49.220966+0000","last_active":"2024-08-03T21:00:49.223032+0000","last_peered":"2024-08-03T21:00:49.223032+0000","last_clean":"2024-08-03T21:00:49.223032+0000","last_became_active":"2024-08-03T21:00:49.220433+0000","last_became_peered":"2024-08-03T21:00:49.220433+0000","last_unstale":"2024-08-03T21:00:49.223032+0000","last_undegraded":"2024-08-03T21:00:49.223032+0000","last_fullsized":"2024-08-03T21:00:49.223032+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:48.192422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:48.192422+0000","last_clean_scrub_stamp":"2024-08-03T21:00:48.192422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-04T23:49:05.539690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.7","version":"0'0","reported_seq":1,"reported_epoch":133,"state":"creating+peering","last_fresh":"2024-08-03T21:00:49.212025+0000","last_change":"2024-08-03T21:00:49.212025+0000","last_active":"2024-08-03T21:00:49.195479+0000","last_peered":"2024-08-03T21:00:49.195479+0000","last_clean":"2024-08-03T21:00:49.195479+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-08-03T21:00:49.212025+0000","last_undegraded":"2024-08-03T21:00:49.212025+0000","last_fullsized":"2024-08-03T21:00:49.212025+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_clean_scrub_stamp":"2024-08-03T21:00:49.195479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.6","version":"0'0","reported_seq":10,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-08-03T21:00:49.238693+0000","last_change":"2024-08-03T21:00:49.235126+0000","last_active":"2024-08-03T21:00:49.238693+0000","last_peered":"2024-08-03T21:00:49.238693+0000","last_clean":"2024-08-03T21:00:49.238693+0000","last_became_active":"2024-08-03T21:00:49.233960+0000","last_became_peered":"2024-08-03T21:00:49.233960+0000","last_unstale":"2024-08-03T21:00:49.238693+0000","last_undegraded":"2024-08-03T21:00:49.238693+0000","last_fullsized":"2024-08-03T21:00:49.238693+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:48.192422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:48.192422+0000","last_clean_scrub_stamp":"2024-08-03T21:00:48.192422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T07:27:35.968034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.4","version":"0'0","reported_seq":1,"reported_epoch":133,"state":"creating+peering","last_fresh":"2024-08-03T21:00:49.212551+0000","last_change":"2024-08-03T21:00:49.212551+0000","last_active":"2024-08-03T21:00:49.195479+0000","last_peered":"2024-08-03T21:00:49.195479+0000","last_clean":"2024-08-03T21:00:49.195479+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-08-03T21:00:49.212551+0000","last_undegraded":"2024-08-03T21:00:49.212551+0000","last_fullsized":"2024-08-03T21:00:49.212551+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_clean_scrub_stamp":"2024-08-03T21:00:49.195479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.5","version":"0'0","reported_seq":10,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-08-03T21:00:49.230593+0000","last_change":"2024-08-03T21:00:49.227879+0000","last_active":"2024-08-03T21:00:49.230593+0000","last_peered":"2024-08-03T21:00:49.230593+0000","last_clean":"2024-08-03T21:00:49.230593+0000","last_became_active":"2024-08-03T21:00:49.227574+0000","last_became_peered":"2024-08-03T21:00:49.227574+0000","last_unstale":"2024-08-03T21:00:49.230593+0000","last_undegraded":"2024-08-03T21:00:49.230593+0000","last_fullsized":"2024-08-03T21:00:49.230593+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:48.192422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:48.192422+0000","last_clean_scrub_stamp":"2024-08-03T21:00:48.192422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T07:43:13.286648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.5","version":"0'0","reported_seq":1,"reported_epoch":133,"state":"creating+peering","last_fresh":"2024-08-03T21:00:49.209221+0000","last_change":"2024-08-03T21:00:49.209221+0000","last_active":"2024-08-03T21:00:49.195479+0000","last_peered":"2024-08-03T21:00:49.195479+0000","last_clean":"2024-08-03T21:00:49.195479+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-08-03T21:00:49.209221+0000","last_undegraded":"2024-08-03T21:00:49.209221+0000","last_fullsized":"2024-08-03T21:00:49.209221+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_clean_scrub_stamp":"2024-08-03T21:00:49.195479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.4","version":"0'0","reported_seq":10,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-08-03T21:00:49.224181+0000","last_change":"2024-08-03T21:00:49.221928+0000","last_active":"2024-08-03T21:00:49.224181+0000","last_peered":"2024-08-03T21:00:49.224181+0000","last_clean":"2024-08-03T21:00:49.224181+0000","last_became_active":"2024-08-03T21:00:49.221623+0000","last_became_peered":"2024-08-03T21:00:49.221623+0000","last_unstale":"2024-08-03T21:00:49.224181+0000","last_undegraded":"2024-08-03T21:00:49.224181+0000","last_fullsized":"2024-08-03T21:00:49.224181+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:48.192422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:48.192422+0000","last_clean_scrub_stamp":"2024-08-03T21:00:48.192422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T01:29:44.046851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.2","version":"0'0","reported_seq":13,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-08-03T21:00:51.938470+0000","last_change":"2024-08-03T21:00:50.232514+0000","last_active":"2024-08-03T21:00:51.938470+0000","last_peered":"2024-08-03T21:00:51.938470+0000","last_clean":"2024-08-03T21:00:51.938470+0000","last_became_active":"2024-08-03T21:00:50.232148+0000","last_became_peered":"2024-08-03T21:00:50.232148+0000","last_unstale":"2024-08-03T21:00:51.938470+0000","last_undegraded":"2024-08-03T21:00:51.938470+0000","last_fullsized":"2024-08-03T21:00:51.938470+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_clean_scrub_stamp":"2024-08-03T21:00:49.195479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-04T23:42:52.232285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.3","version":"0'0","reported_seq":10,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-08-03T21:00:49.238736+0000","last_change":"2024-08-03T21:00:49.235356+0000","last_active":"2024-08-03T21:00:49.238736+0000","last_peered":"2024-08-03T21:00:49.238736+0000","last_clean":"2024-08-03T21:00:49.238736+0000","last_became_active":"2024-08-03T21:00:49.234801+0000","last_became_peered":"2024-08-03T21:00:49.234801+0000","last_unstale":"2024-08-03T21:00:49.238736+0000","last_undegraded":"2024-08-03T21:00:49.238736+0000","last_fullsized":"2024-08-03T21:00:49.238736+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:48.192422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:48.192422+0000","last_clean_scrub_stamp":"2024-08-03T21:00:48.192422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-04T23:53:41.240790+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.3","version":"0'0","reported_seq":13,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-08-03T21:00:51.939532+0000","last_change":"2024-08-03T21:00:50.233609+0000","last_active":"2024-08-03T21:00:51.939532+0000","last_peered":"2024-08-03T21:00:51.939532+0000","last_clean":"2024-08-03T21:00:51.939532+0000","last_became_active":"2024-08-03T21:00:50.231678+0000","last_became_peered":"2024-08-03T21:00:50.231678+0000","last_unstale":"2024-08-03T21:00:51.939532+0000","last_undegraded":"2024-08-03T21:00:51.939532+0000","last_fullsized":"2024-08-03T21:00:51.939532+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_clean_scrub_stamp":"2024-08-03T21:00:49.195479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T02:34:58.346689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.2","version":"0'0","reported_seq":10,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-08-03T21:00:49.219683+0000","last_change":"2024-08-03T21:00:49.218982+0000","last_active":"2024-08-03T21:00:49.219683+0000","last_peered":"2024-08-03T21:00:49.219683+0000","last_clean":"2024-08-03T21:00:49.219683+0000","last_became_active":"2024-08-03T21:00:49.217673+0000","last_became_peered":"2024-08-03T21:00:49.217673+0000","last_unstale":"2024-08-03T21:00:49.219683+0000","last_undegraded":"2024-08-03T21:00:49.219683+0000","last_fullsized":"2024-08-03T21:00:49.219683+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:48.192422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:48.192422+0000","last_clean_scrub_stamp":"2024-08-03T21:00:48.192422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T05:44:38.082933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.0","version":"0'0","reported_seq":13,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-08-03T21:00:51.938141+0000","last_change":"2024-08-03T21:00:50.234121+0000","last_active":"2024-08-03T21:00:51.938141+0000","last_peered":"2024-08-03T21:00:51.938141+0000","last_clean":"2024-08-03T21:00:51.938141+0000","last_became_active":"2024-08-03T21:00:50.233758+0000","last_became_peered":"2024-08-03T21:00:50.233758+0000","last_unstale":"2024-08-03T21:00:51.938141+0000","last_undegraded":"2024-08-03T21:00:51.938141+0000","last_fullsized":"2024-08-03T21:00:51.938141+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_clean_scrub_stamp":"2024-08-03T21:00:49.195479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T05:28:59.859843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.1","version":"0'0","reported_seq":15,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-08-03T21:00:51.225900+0000","last_change":"2024-08-03T21:00:49.230098+0000","last_active":"2024-08-03T21:00:51.225900+0000","last_peered":"2024-08-03T21:00:51.225900+0000","last_clean":"2024-08-03T21:00:51.225900+0000","last_became_active":"2024-08-03T21:00:49.229777+0000","last_became_peered":"2024-08-03T21:00:49.229777+0000","last_unstale":"2024-08-03T21:00:51.225900+0000","last_undegraded":"2024-08-03T21:00:51.225900+0000","last_fullsized":"2024-08-03T21:00:51.225900+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:48.192422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:48.192422+0000","last_clean_scrub_stamp":"2024-08-03T21:00:48.192422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T06:34:18.629210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.1","version":"0'0","reported_seq":1,"reported_epoch":133,"state":"creating+peering","last_fresh":"2024-08-03T21:00:49.210235+0000","last_change":"2024-08-03T21:00:49.210235+0000","last_active":"2024-08-03T21:00:49.195479+0000","last_peered":"2024-08-03T21:00:49.195479+0000","last_clean":"2024-08-03T21:00:49.195479+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-08-03T21:00:49.210235+0000","last_undegraded":"2024-08-03T21:00:49.210235+0000","last_fullsized":"2024-08-03T21:00:49.210235+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_clean_scrub_stamp":"2024-08-03T21:00:49.195479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.0","version":"0'0","reported_seq":15,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-08-03T21:00:51.938216+0000","last_change":"2024-08-03T21:00:49.231962+0000","last_active":"2024-08-03T21:00:51.938216+0000","last_peered":"2024-08-03T21:00:51.938216+0000","last_clean":"2024-08-03T21:00:51.938216+0000","last_became_active":"2024-08-03T21:00:49.231607+0000","last_became_peered":"2024-08-03T21:00:49.231607+0000","last_unstale":"2024-08-03T21:00:51.938216+0000","last_undegraded":"2024-08-03T21:00:51.938216+0000","last_fullsized":"2024-08-03T21:00:51.938216+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:48.192422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:48.192422+0000","last_clean_scrub_stamp":"2024-08-03T21:00:48.192422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T08:45:01.729460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.e","version":"0'0","reported_seq":13,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-08-03T21:00:51.939469+0000","last_change":"2024-08-03T21:00:50.249570+0000","last_active":"2024-08-03T21:00:51.939469+0000","last_peered":"2024-08-03T21:00:51.939469+0000","last_clean":"2024-08-03T21:00:51.939469+0000","last_became_active":"2024-08-03T21:00:50.240733+0000","last_became_peered":"2024-08-03T21:00:50.240733+0000","last_unstale":"2024-08-03T21:00:51.939469+0000","last_undegraded":"2024-08-03T21:00:51.939469+0000","last_fullsized":"2024-08-03T21:00:51.939469+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_clean_scrub_stamp":"2024-08-03T21:00:49.195479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T08:43:59.747967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.f","version":"0'0","reported_seq":10,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-08-03T21:00:49.235936+0000","last_change":"2024-08-03T21:00:49.234956+0000","last_active":"2024-08-03T21:00:49.235936+0000","last_peered":"2024-08-03T21:00:49.235936+0000","last_clean":"2024-08-03T21:00:49.235936+0000","last_became_active":"2024-08-03T21:00:49.234351+0000","last_became_peered":"2024-08-03T21:00:49.234351+0000","last_unstale":"2024-08-03T21:00:49.235936+0000","last_undegraded":"2024-08-03T21:00:49.235936+0000","last_fullsized":"2024-08-03T21:00:49.235936+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:48.192422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:48.192422+0000","last_clean_scrub_stamp":"2024-08-03T21:00:48.192422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-04T22:46:01.864033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"16'72","reported_seq":327,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-08-03T21:00:49.218344+0000","last_change":"2024-08-03T20:47:30.547632+0000","last_active":"2024-08-03T21:00:49.218344+0000","last_peered":"2024-08-03T21:00:49.218344+0000","last_clean":"2024-08-03T21:00:49.218344+0000","last_became_active":"2024-08-03T20:47:30.546255+0000","last_became_peered":"2024-08-03T20:47:30.546255+0000","last_unstale":"2024-08-03T21:00:49.218344+0000","last_undegraded":"2024-08-03T21:00:49.218344+0000","last_fullsized":"2024-08-03T21:00:49.218344+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:47:28.874451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:47:28.874451+0000","last_clean_scrub_stamp":"2024-08-03T20:47:28.874451+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T00:19:40.548812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":15,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":14,"num_pg":32,"stat_sum":{"num_bytes":1646,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":3404,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":10,"ondisk_log_size":10,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509603,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41732,"kb_used_data":2228,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330108,"statfs":{"total":96636764160,"available":96594030592,"internally_reserved":0,"allocated":2281472,"data_stored":1401923,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":40434122},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":6,"up_from":15,"seq":64424509604,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36988,"kb_used_data":1648,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334852,"statfs":{"total":96636764160,"available":96598888448,"internally_reserved":0,"allocated":1687552,"data_stored":811555,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":15,"seq":64424509604,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36988,"kb_used_data":1648,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334852,"statfs":{"total":96636764160,"available":96598888448,"internally_reserved":0,"allocated":1687552,"data_stored":811555,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509603,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41148,"kb_used_data":1648,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330692,"statfs":{"total":96636764160,"available":96594628608,"internally_reserved":0,"allocated":1687552,"data_stored":811555,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509605,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":29252,"kb_used_data":2228,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94342588,"statfs":{"total":96636764160,"available":96606810112,"internally_reserved":0,"allocated":2281472,"data_stored":1401923,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542309,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37048,"kb_used_data":1704,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334792,"statfs":{"total":96636764160,"available":96598827008,"internally_reserved":0,"allocated":1744896,"data_stored":832641,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":3,"up_from":14,"seq":60129542307,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37044,"kb_used_data":1700,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334796,"statfs":{"total":96636764160,"available":96598831104,"internally_reserved":0,"allocated":1740800,"data_stored":832539,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":2,"up_from":14,"seq":60129542308,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41192,"kb_used_data":1688,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330648,"statfs":{"total":96636764160,"available":96594583552,"internally_reserved":0,"allocated":1728512,"data_stored":830939,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-08-03T21:00:54.746 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-08-03T21:00:55.011 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:55.009+0000 7f8275600640 1 -- 172.21.15.32:0/2723913821 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8254004800 msgr2=0x7f8254004c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:00:55.011 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:55.009+0000 7f8275600640 1 --2- 172.21.15.32:0/2723913821 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8254004800 0x7f8254004c10 secure :-1 s=READY pgs=2172 cs=0 l=1 rev1=1 crypto rx=0x7f82700cc4d0 tx=0x7f8254005830 comp rx=0 tx=0).stop 2024-08-03T21:00:55.011 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:55.009+0000 7f826f400640 1 -- 172.21.15.32:0/2723913821 reap_dead start 2024-08-03T21:00:55.011 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:55.009+0000 7f8275600640 1 -- 172.21.15.32:0/2723913821 shutdown_connections 2024-08-03T21:00:55.011 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:55.009+0000 7f8275600640 1 -- 172.21.15.32:0/2723913821 >> 172.21.15.32:0/2723913821 conn(0x7f82700b1bd0 msgr2=0x7f82700b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T21:00:55.012 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:55.009+0000 7f8275600640 1 -- 172.21.15.32:0/2723913821 shutdown_connections 2024-08-03T21:00:55.012 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:55.009+0000 7f8275600640 1 -- 172.21.15.32:0/2723913821 wait complete. 2024-08-03T21:00:55.013 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:55.011+0000 7f8275600640 1 Processor -- start 2024-08-03T21:00:55.013 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:55.011+0000 7f8275600640 1 -- start start 2024-08-03T21:00:55.016 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:55.013+0000 7f8275600640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8254004800 0x7f82700b2fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:00:55.016 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:55.013+0000 7f8275600640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f82701429c0 0x7f82700b4760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:00:55.016 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:55.013+0000 7f8275600640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f8270142dd0 0x7f8270141780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:00:55.016 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:55.013+0000 7f8275600640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f82700cf840 con 0x7f82701429c0 2024-08-03T21:00:55.016 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:55.013+0000 7f8275600640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f8270145f30 con 0x7f8254004800 2024-08-03T21:00:55.016 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:55.013+0000 7f8275600640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7f82700cf9c0 con 0x7f8270142dd0 2024-08-03T21:00:55.016 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:55.013+0000 7f8267e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f82701429c0 0x7f82700b4760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:00:55.016 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:55.013+0000 7f8267e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f82701429c0 0x7f82700b4760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:50078/0 (socket says 172.21.15.32:50078) 2024-08-03T21:00:55.016 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:55.013+0000 7f8267e00640 1 -- 172.21.15.32:0/1749291073 learned_addr learned my addr 172.21.15.32:0/1749291073 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T21:00:55.017 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:55.013+0000 7f826ea00640 1 --2- 172.21.15.32:0/1749291073 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8254004800 0x7f82700b2fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:00:55.017 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:55.013+0000 7f826f400640 1 --2- 172.21.15.32:0/1749291073 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f8270142dd0 0x7f8270141780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:00:55.017 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:55.014+0000 7f8267e00640 1 -- 172.21.15.32:0/1749291073 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f8270142dd0 msgr2=0x7f8270141780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:00:55.017 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:55.014+0000 7f8267e00640 1 --2- 172.21.15.32:0/1749291073 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f8270142dd0 0x7f8270141780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:55.017 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:55.014+0000 7f8267e00640 1 -- 172.21.15.32:0/1749291073 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8254004800 msgr2=0x7f82700b2fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:00:55.017 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:55.014+0000 7f8267e00640 1 --2- 172.21.15.32:0/1749291073 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8254004800 0x7f82700b2fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:55.017 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:55.014+0000 7f8267e00640 1 -- 172.21.15.32:0/1749291073 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f82701f9160 con 0x7f82701429c0 2024-08-03T21:00:55.017 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:55.014+0000 7f8267e00640 1 --2- 172.21.15.32:0/1749291073 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f82701429c0 0x7f82700b4760 secure :-1 s=READY pgs=2173 cs=0 l=1 rev1=1 crypto rx=0x7f825800c3a0 tx=0x7f8258006d90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T21:00:55.017 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:55.014+0000 7f8266a00640 1 -- 172.21.15.32:0/1749291073 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8258031040 con 0x7f82701429c0 2024-08-03T21:00:55.017 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:55.014+0000 7f8266a00640 1 -- 172.21.15.32:0/1749291073 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f82580065a0 con 0x7f82701429c0 2024-08-03T21:00:55.017 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:55.014+0000 7f8275600640 1 -- 172.21.15.32:0/1749291073 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f82701f9440 con 0x7f82701429c0 2024-08-03T21:00:55.017 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:55.014+0000 7f8266a00640 1 -- 172.21.15.32:0/1749291073 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8258003a60 con 0x7f82701429c0 2024-08-03T21:00:55.017 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:55.014+0000 7f8275600640 1 -- 172.21.15.32:0/1749291073 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f82701f9a00 con 0x7f82701429c0 2024-08-03T21:00:55.018 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:55.015+0000 7f8275600640 1 -- 172.21.15.32:0/1749291073 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8270070880 con 0x7f82701429c0 2024-08-03T21:00:55.018 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:55.017+0000 7f8266a00640 1 -- 172.21.15.32:0/1749291073 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f8258034070 con 0x7f82701429c0 2024-08-03T21:00:55.019 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:55.018+0000 7f8266a00640 1 --2- 172.21.15.32:0/1749291073 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f824c077010 0x7f824c079520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:00:55.019 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:55.018+0000 7f8266a00640 1 -- 172.21.15.32:0/1749291073 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(135..135 src has 1..135) ==== 5709+0+0 (secure 0 0 0) 0x7f82580b87a0 con 0x7f82701429c0 2024-08-03T21:00:55.020 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:55.018+0000 7f826ea00640 1 --2- 172.21.15.32:0/1749291073 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f824c077010 0x7f824c079520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:00:55.020 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:55.019+0000 7f826ea00640 1 --2- 172.21.15.32:0/1749291073 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f824c077010 0x7f824c079520 secure :-1 s=READY pgs=871 cs=0 l=1 rev1=1 crypto rx=0x7f8260009c50 tx=0x7f8260005f90 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T21:00:55.023 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:55.022+0000 7f8266a00640 1 -- 172.21.15.32:0/1749291073 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7f8258081990 con 0x7f82701429c0 2024-08-03T21:00:55.227 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:55.225+0000 7f8275600640 1 -- 172.21.15.32:0/1749291073 --> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f82700b0430 con 0x7f824c077010 2024-08-03T21:00:55.232 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:55.230+0000 7f8266a00640 1 -- 172.21.15.32:0/1749291073 <== mgr.4106 v2:172.21.15.118:6800/4205950699 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+249456 (secure 0 0 0) 0x7f82700b0430 con 0x7f824c077010 2024-08-03T21:00:55.236 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-03T21:00:55.238 INFO:teuthology.orchestra.run.smithi032.stderr:dumped all 2024-08-03T21:00:55.239 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:55.235+0000 7f8275600640 1 -- 172.21.15.32:0/1749291073 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f824c077010 msgr2=0x7f824c079520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:00:55.239 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:55.235+0000 7f8275600640 1 --2- 172.21.15.32:0/1749291073 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f824c077010 0x7f824c079520 secure :-1 s=READY pgs=871 cs=0 l=1 rev1=1 crypto rx=0x7f8260009c50 tx=0x7f8260005f90 comp rx=0 tx=0).stop 2024-08-03T21:00:55.239 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:55.235+0000 7f8275600640 1 -- 172.21.15.32:0/1749291073 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f82701429c0 msgr2=0x7f82700b4760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:00:55.239 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:55.235+0000 7f8275600640 1 --2- 172.21.15.32:0/1749291073 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f82701429c0 0x7f82700b4760 secure :-1 s=READY pgs=2173 cs=0 l=1 rev1=1 crypto rx=0x7f825800c3a0 tx=0x7f8258006d90 comp rx=0 tx=0).stop 2024-08-03T21:00:55.239 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:55.235+0000 7f8275600640 1 -- 172.21.15.32:0/1749291073 shutdown_connections 2024-08-03T21:00:55.240 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:55.235+0000 7f8275600640 1 --2- 172.21.15.32:0/1749291073 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f824c077010 0x7f824c079520 unknown :-1 s=CLOSED pgs=871 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:55.240 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:55.235+0000 7f8275600640 1 --2- 172.21.15.32:0/1749291073 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f8270142dd0 0x7f8270141780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:55.240 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:55.235+0000 7f8275600640 1 --2- 172.21.15.32:0/1749291073 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f82701429c0 0x7f82700b4760 unknown :-1 s=CLOSED pgs=2173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:55.240 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:55.235+0000 7f8275600640 1 --2- 172.21.15.32:0/1749291073 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8254004800 0x7f82700b2fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:55.240 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:55.235+0000 7f8275600640 1 -- 172.21.15.32:0/1749291073 >> 172.21.15.32:0/1749291073 conn(0x7f82700b1bd0 msgr2=0x7f82700b11d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T21:00:55.240 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:55.235+0000 7f8275600640 1 -- 172.21.15.32:0/1749291073 shutdown_connections 2024-08-03T21:00:55.240 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:55.235+0000 7f8275600640 1 -- 172.21.15.32:0/1749291073 wait complete. 2024-08-03T21:00:55.262 INFO:teuthology.orchestra.run.smithi032.stdout:{"pg_ready":true,"pg_map":{"version":531,"stamp":"2024-08-03T21:00:54.454153+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592014,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":113,"num_write_kb":1638,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":82,"ondisk_log_size":82,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":81,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":301392,"kb_used_data":14492,"kb_used_omap":12,"kb_used_meta":286771,"kb_avail":754673328,"statfs":{"total":773094113280,"available":772785487872,"internally_reserved":0,"allocated":14839808,"data_stored":7734630,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":293654065},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":20,"apply_latency_ms":20,"commit_latency_ns":20000000,"apply_latency_ns":20000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":1646,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001779"},"pg_stats":[{"pgid":"15.3f","version":"0'0","reported_seq":1,"reported_epoch":133,"state":"creating+peering","last_fresh":"2024-08-03T21:00:49.214906+0000","last_change":"2024-08-03T21:00:49.214906+0000","last_active":"2024-08-03T21:00:49.195479+0000","last_peered":"2024-08-03T21:00:49.195479+0000","last_clean":"2024-08-03T21:00:49.195479+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-08-03T21:00:49.214906+0000","last_undegraded":"2024-08-03T21:00:49.214906+0000","last_fullsized":"2024-08-03T21:00:49.214906+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_clean_scrub_stamp":"2024-08-03T21:00:49.195479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.3e","version":"0'0","reported_seq":1,"reported_epoch":133,"state":"creating+peering","last_fresh":"2024-08-03T21:00:49.221252+0000","last_change":"2024-08-03T21:00:49.221252+0000","last_active":"2024-08-03T21:00:49.195479+0000","last_peered":"2024-08-03T21:00:49.195479+0000","last_clean":"2024-08-03T21:00:49.195479+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-08-03T21:00:49.221252+0000","last_undegraded":"2024-08-03T21:00:49.221252+0000","last_fullsized":"2024-08-03T21:00:49.221252+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_clean_scrub_stamp":"2024-08-03T21:00:49.195479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.3d","version":"0'0","reported_seq":1,"reported_epoch":133,"state":"creating+peering","last_fresh":"2024-08-03T21:00:49.214132+0000","last_change":"2024-08-03T21:00:49.214132+0000","last_active":"2024-08-03T21:00:49.195479+0000","last_peered":"2024-08-03T21:00:49.195479+0000","last_clean":"2024-08-03T21:00:49.195479+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-08-03T21:00:49.214132+0000","last_undegraded":"2024-08-03T21:00:49.214132+0000","last_fullsized":"2024-08-03T21:00:49.214132+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_clean_scrub_stamp":"2024-08-03T21:00:49.195479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.3c","version":"0'0","reported_seq":13,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-08-03T21:00:51.225682+0000","last_change":"2024-08-03T21:00:50.243269+0000","last_active":"2024-08-03T21:00:51.225682+0000","last_peered":"2024-08-03T21:00:51.225682+0000","last_clean":"2024-08-03T21:00:51.225682+0000","last_became_active":"2024-08-03T21:00:50.242985+0000","last_became_peered":"2024-08-03T21:00:50.242985+0000","last_unstale":"2024-08-03T21:00:51.225682+0000","last_undegraded":"2024-08-03T21:00:51.225682+0000","last_fullsized":"2024-08-03T21:00:51.225682+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_clean_scrub_stamp":"2024-08-03T21:00:49.195479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T03:32:19.871676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.3b","version":"0'0","reported_seq":1,"reported_epoch":133,"state":"creating+peering","last_fresh":"2024-08-03T21:00:49.212243+0000","last_change":"2024-08-03T21:00:49.212243+0000","last_active":"2024-08-03T21:00:49.195479+0000","last_peered":"2024-08-03T21:00:49.195479+0000","last_clean":"2024-08-03T21:00:49.195479+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-08-03T21:00:49.212243+0000","last_undegraded":"2024-08-03T21:00:49.212243+0000","last_fullsized":"2024-08-03T21:00:49.212243+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_clean_scrub_stamp":"2024-08-03T21:00:49.195479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.3a","version":"0'0","reported_seq":1,"reported_epoch":133,"state":"creating+peering","last_fresh":"2024-08-03T21:00:49.215377+0000","last_change":"2024-08-03T21:00:49.215377+0000","last_active":"2024-08-03T21:00:49.195479+0000","last_peered":"2024-08-03T21:00:49.195479+0000","last_clean":"2024-08-03T21:00:49.195479+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-08-03T21:00:49.215377+0000","last_undegraded":"2024-08-03T21:00:49.215377+0000","last_fullsized":"2024-08-03T21:00:49.215377+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_clean_scrub_stamp":"2024-08-03T21:00:49.195479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.39","version":"0'0","reported_seq":1,"reported_epoch":133,"state":"creating+peering","last_fresh":"2024-08-03T21:00:49.213231+0000","last_change":"2024-08-03T21:00:49.213231+0000","last_active":"2024-08-03T21:00:49.195479+0000","last_peered":"2024-08-03T21:00:49.195479+0000","last_clean":"2024-08-03T21:00:49.195479+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-08-03T21:00:49.213231+0000","last_undegraded":"2024-08-03T21:00:49.213231+0000","last_fullsized":"2024-08-03T21:00:49.213231+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_clean_scrub_stamp":"2024-08-03T21:00:49.195479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.38","version":"0'0","reported_seq":1,"reported_epoch":133,"state":"creating+peering","last_fresh":"2024-08-03T21:00:49.214019+0000","last_change":"2024-08-03T21:00:49.214019+0000","last_active":"2024-08-03T21:00:49.195479+0000","last_peered":"2024-08-03T21:00:49.195479+0000","last_clean":"2024-08-03T21:00:49.195479+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-08-03T21:00:49.214019+0000","last_undegraded":"2024-08-03T21:00:49.214019+0000","last_fullsized":"2024-08-03T21:00:49.214019+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_clean_scrub_stamp":"2024-08-03T21:00:49.195479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.37","version":"0'0","reported_seq":13,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-08-03T21:00:51.939823+0000","last_change":"2024-08-03T21:00:50.223154+0000","last_active":"2024-08-03T21:00:51.939823+0000","last_peered":"2024-08-03T21:00:51.939823+0000","last_clean":"2024-08-03T21:00:51.939823+0000","last_became_active":"2024-08-03T21:00:50.222779+0000","last_became_peered":"2024-08-03T21:00:50.222779+0000","last_unstale":"2024-08-03T21:00:51.939823+0000","last_undegraded":"2024-08-03T21:00:51.939823+0000","last_fullsized":"2024-08-03T21:00:51.939823+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_clean_scrub_stamp":"2024-08-03T21:00:49.195479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T06:34:19.632267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.36","version":"0'0","reported_seq":1,"reported_epoch":133,"state":"creating+peering","last_fresh":"2024-08-03T21:00:49.211444+0000","last_change":"2024-08-03T21:00:49.211444+0000","last_active":"2024-08-03T21:00:49.195479+0000","last_peered":"2024-08-03T21:00:49.195479+0000","last_clean":"2024-08-03T21:00:49.195479+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-08-03T21:00:49.211444+0000","last_undegraded":"2024-08-03T21:00:49.211444+0000","last_fullsized":"2024-08-03T21:00:49.211444+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_clean_scrub_stamp":"2024-08-03T21:00:49.195479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.35","version":"0'0","reported_seq":1,"reported_epoch":133,"state":"creating+peering","last_fresh":"2024-08-03T21:00:49.217444+0000","last_change":"2024-08-03T21:00:49.217444+0000","last_active":"2024-08-03T21:00:49.195479+0000","last_peered":"2024-08-03T21:00:49.195479+0000","last_clean":"2024-08-03T21:00:49.195479+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-08-03T21:00:49.217444+0000","last_undegraded":"2024-08-03T21:00:49.217444+0000","last_fullsized":"2024-08-03T21:00:49.217444+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_clean_scrub_stamp":"2024-08-03T21:00:49.195479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.34","version":"0'0","reported_seq":13,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-08-03T21:00:51.939323+0000","last_change":"2024-08-03T21:00:50.229025+0000","last_active":"2024-08-03T21:00:51.939323+0000","last_peered":"2024-08-03T21:00:51.939323+0000","last_clean":"2024-08-03T21:00:51.939323+0000","last_became_active":"2024-08-03T21:00:50.228698+0000","last_became_peered":"2024-08-03T21:00:50.228698+0000","last_unstale":"2024-08-03T21:00:51.939323+0000","last_undegraded":"2024-08-03T21:00:51.939323+0000","last_fullsized":"2024-08-03T21:00:51.939323+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_clean_scrub_stamp":"2024-08-03T21:00:49.195479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T06:51:09.079780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.33","version":"0'0","reported_seq":13,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-08-03T21:00:51.225611+0000","last_change":"2024-08-03T21:00:50.224093+0000","last_active":"2024-08-03T21:00:51.225611+0000","last_peered":"2024-08-03T21:00:51.225611+0000","last_clean":"2024-08-03T21:00:51.225611+0000","last_became_active":"2024-08-03T21:00:50.222966+0000","last_became_peered":"2024-08-03T21:00:50.222966+0000","last_unstale":"2024-08-03T21:00:51.225611+0000","last_undegraded":"2024-08-03T21:00:51.225611+0000","last_fullsized":"2024-08-03T21:00:51.225611+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_clean_scrub_stamp":"2024-08-03T21:00:49.195479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T04:11:36.038037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.32","version":"0'0","reported_seq":13,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-08-03T21:00:51.937848+0000","last_change":"2024-08-03T21:00:50.230657+0000","last_active":"2024-08-03T21:00:51.937848+0000","last_peered":"2024-08-03T21:00:51.937848+0000","last_clean":"2024-08-03T21:00:51.937848+0000","last_became_active":"2024-08-03T21:00:50.230356+0000","last_became_peered":"2024-08-03T21:00:50.230356+0000","last_unstale":"2024-08-03T21:00:51.937848+0000","last_undegraded":"2024-08-03T21:00:51.937848+0000","last_fullsized":"2024-08-03T21:00:51.937848+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_clean_scrub_stamp":"2024-08-03T21:00:49.195479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-04T21:20:29.060152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.31","version":"0'0","reported_seq":13,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-08-03T21:00:51.225596+0000","last_change":"2024-08-03T21:00:50.236123+0000","last_active":"2024-08-03T21:00:51.225596+0000","last_peered":"2024-08-03T21:00:51.225596+0000","last_clean":"2024-08-03T21:00:51.225596+0000","last_became_active":"2024-08-03T21:00:50.235747+0000","last_became_peered":"2024-08-03T21:00:50.235747+0000","last_unstale":"2024-08-03T21:00:51.225596+0000","last_undegraded":"2024-08-03T21:00:51.225596+0000","last_fullsized":"2024-08-03T21:00:51.225596+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_clean_scrub_stamp":"2024-08-03T21:00:49.195479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T05:25:30.476710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.30","version":"0'0","reported_seq":1,"reported_epoch":133,"state":"creating+peering","last_fresh":"2024-08-03T21:00:49.214934+0000","last_change":"2024-08-03T21:00:49.214934+0000","last_active":"2024-08-03T21:00:49.195479+0000","last_peered":"2024-08-03T21:00:49.195479+0000","last_clean":"2024-08-03T21:00:49.195479+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-08-03T21:00:49.214934+0000","last_undegraded":"2024-08-03T21:00:49.214934+0000","last_fullsized":"2024-08-03T21:00:49.214934+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_clean_scrub_stamp":"2024-08-03T21:00:49.195479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.2f","version":"0'0","reported_seq":1,"reported_epoch":133,"state":"creating+peering","last_fresh":"2024-08-03T21:00:49.220256+0000","last_change":"2024-08-03T21:00:49.220256+0000","last_active":"2024-08-03T21:00:49.195479+0000","last_peered":"2024-08-03T21:00:49.195479+0000","last_clean":"2024-08-03T21:00:49.195479+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-08-03T21:00:49.220256+0000","last_undegraded":"2024-08-03T21:00:49.220256+0000","last_fullsized":"2024-08-03T21:00:49.220256+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_clean_scrub_stamp":"2024-08-03T21:00:49.195479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.2e","version":"0'0","reported_seq":13,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-08-03T21:00:51.225830+0000","last_change":"2024-08-03T21:00:50.241634+0000","last_active":"2024-08-03T21:00:51.225830+0000","last_peered":"2024-08-03T21:00:51.225830+0000","last_clean":"2024-08-03T21:00:51.225830+0000","last_became_active":"2024-08-03T21:00:50.241263+0000","last_became_peered":"2024-08-03T21:00:50.241263+0000","last_unstale":"2024-08-03T21:00:51.225830+0000","last_undegraded":"2024-08-03T21:00:51.225830+0000","last_fullsized":"2024-08-03T21:00:51.225830+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_clean_scrub_stamp":"2024-08-03T21:00:49.195479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T02:07:46.123334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.2d","version":"0'0","reported_seq":13,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-08-03T21:00:51.938842+0000","last_change":"2024-08-03T21:00:50.243910+0000","last_active":"2024-08-03T21:00:51.938842+0000","last_peered":"2024-08-03T21:00:51.938842+0000","last_clean":"2024-08-03T21:00:51.938842+0000","last_became_active":"2024-08-03T21:00:50.243054+0000","last_became_peered":"2024-08-03T21:00:50.243054+0000","last_unstale":"2024-08-03T21:00:51.938842+0000","last_undegraded":"2024-08-03T21:00:51.938842+0000","last_fullsized":"2024-08-03T21:00:51.938842+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_clean_scrub_stamp":"2024-08-03T21:00:49.195479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T08:31:27.530602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.2c","version":"0'0","reported_seq":13,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-08-03T21:00:51.939427+0000","last_change":"2024-08-03T21:00:50.223678+0000","last_active":"2024-08-03T21:00:51.939427+0000","last_peered":"2024-08-03T21:00:51.939427+0000","last_clean":"2024-08-03T21:00:51.939427+0000","last_became_active":"2024-08-03T21:00:50.223357+0000","last_became_peered":"2024-08-03T21:00:50.223357+0000","last_unstale":"2024-08-03T21:00:51.939427+0000","last_undegraded":"2024-08-03T21:00:51.939427+0000","last_fullsized":"2024-08-03T21:00:51.939427+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_clean_scrub_stamp":"2024-08-03T21:00:49.195479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T07:48:35.449963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.2b","version":"0'0","reported_seq":13,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-08-03T21:00:51.939465+0000","last_change":"2024-08-03T21:00:50.243413+0000","last_active":"2024-08-03T21:00:51.939465+0000","last_peered":"2024-08-03T21:00:51.939465+0000","last_clean":"2024-08-03T21:00:51.939465+0000","last_became_active":"2024-08-03T21:00:50.243004+0000","last_became_peered":"2024-08-03T21:00:50.243004+0000","last_unstale":"2024-08-03T21:00:51.939465+0000","last_undegraded":"2024-08-03T21:00:51.939465+0000","last_fullsized":"2024-08-03T21:00:51.939465+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_clean_scrub_stamp":"2024-08-03T21:00:49.195479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-04T23:21:29.132988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.2a","version":"0'0","reported_seq":1,"reported_epoch":133,"state":"creating+peering","last_fresh":"2024-08-03T21:00:49.215684+0000","last_change":"2024-08-03T21:00:49.215684+0000","last_active":"2024-08-03T21:00:49.195479+0000","last_peered":"2024-08-03T21:00:49.195479+0000","last_clean":"2024-08-03T21:00:49.195479+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-08-03T21:00:49.215684+0000","last_undegraded":"2024-08-03T21:00:49.215684+0000","last_fullsized":"2024-08-03T21:00:49.215684+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_clean_scrub_stamp":"2024-08-03T21:00:49.195479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.29","version":"0'0","reported_seq":1,"reported_epoch":133,"state":"creating+peering","last_fresh":"2024-08-03T21:00:49.214421+0000","last_change":"2024-08-03T21:00:49.214421+0000","last_active":"2024-08-03T21:00:49.195479+0000","last_peered":"2024-08-03T21:00:49.195479+0000","last_clean":"2024-08-03T21:00:49.195479+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-08-03T21:00:49.214421+0000","last_undegraded":"2024-08-03T21:00:49.214421+0000","last_fullsized":"2024-08-03T21:00:49.214421+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_clean_scrub_stamp":"2024-08-03T21:00:49.195479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.28","version":"0'0","reported_seq":1,"reported_epoch":133,"state":"creating+peering","last_fresh":"2024-08-03T21:00:49.214489+0000","last_change":"2024-08-03T21:00:49.214489+0000","last_active":"2024-08-03T21:00:49.195479+0000","last_peered":"2024-08-03T21:00:49.195479+0000","last_clean":"2024-08-03T21:00:49.195479+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-08-03T21:00:49.214489+0000","last_undegraded":"2024-08-03T21:00:49.214489+0000","last_fullsized":"2024-08-03T21:00:49.214489+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_clean_scrub_stamp":"2024-08-03T21:00:49.195479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.27","version":"0'0","reported_seq":1,"reported_epoch":133,"state":"creating+peering","last_fresh":"2024-08-03T21:00:49.211525+0000","last_change":"2024-08-03T21:00:49.211525+0000","last_active":"2024-08-03T21:00:49.195479+0000","last_peered":"2024-08-03T21:00:49.195479+0000","last_clean":"2024-08-03T21:00:49.195479+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-08-03T21:00:49.211525+0000","last_undegraded":"2024-08-03T21:00:49.211525+0000","last_fullsized":"2024-08-03T21:00:49.211525+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_clean_scrub_stamp":"2024-08-03T21:00:49.195479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.26","version":"0'0","reported_seq":1,"reported_epoch":133,"state":"creating+peering","last_fresh":"2024-08-03T21:00:49.220518+0000","last_change":"2024-08-03T21:00:49.220518+0000","last_active":"2024-08-03T21:00:49.195479+0000","last_peered":"2024-08-03T21:00:49.195479+0000","last_clean":"2024-08-03T21:00:49.195479+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-08-03T21:00:49.220518+0000","last_undegraded":"2024-08-03T21:00:49.220518+0000","last_fullsized":"2024-08-03T21:00:49.220518+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_clean_scrub_stamp":"2024-08-03T21:00:49.195479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.25","version":"0'0","reported_seq":1,"reported_epoch":133,"state":"creating+peering","last_fresh":"2024-08-03T21:00:49.216934+0000","last_change":"2024-08-03T21:00:49.216934+0000","last_active":"2024-08-03T21:00:49.195479+0000","last_peered":"2024-08-03T21:00:49.195479+0000","last_clean":"2024-08-03T21:00:49.195479+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-08-03T21:00:49.216934+0000","last_undegraded":"2024-08-03T21:00:49.216934+0000","last_fullsized":"2024-08-03T21:00:49.216934+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_clean_scrub_stamp":"2024-08-03T21:00:49.195479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.24","version":"0'0","reported_seq":1,"reported_epoch":133,"state":"creating+peering","last_fresh":"2024-08-03T21:00:49.212944+0000","last_change":"2024-08-03T21:00:49.212944+0000","last_active":"2024-08-03T21:00:49.195479+0000","last_peered":"2024-08-03T21:00:49.195479+0000","last_clean":"2024-08-03T21:00:49.195479+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-08-03T21:00:49.212944+0000","last_undegraded":"2024-08-03T21:00:49.212944+0000","last_fullsized":"2024-08-03T21:00:49.212944+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_clean_scrub_stamp":"2024-08-03T21:00:49.195479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.23","version":"0'0","reported_seq":1,"reported_epoch":133,"state":"creating+peering","last_fresh":"2024-08-03T21:00:49.219217+0000","last_change":"2024-08-03T21:00:49.219217+0000","last_active":"2024-08-03T21:00:49.195479+0000","last_peered":"2024-08-03T21:00:49.195479+0000","last_clean":"2024-08-03T21:00:49.195479+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-08-03T21:00:49.219217+0000","last_undegraded":"2024-08-03T21:00:49.219217+0000","last_fullsized":"2024-08-03T21:00:49.219217+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_clean_scrub_stamp":"2024-08-03T21:00:49.195479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.22","version":"0'0","reported_seq":13,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-08-03T21:00:51.938224+0000","last_change":"2024-08-03T21:00:50.243735+0000","last_active":"2024-08-03T21:00:51.938224+0000","last_peered":"2024-08-03T21:00:51.938224+0000","last_clean":"2024-08-03T21:00:51.938224+0000","last_became_active":"2024-08-03T21:00:50.243333+0000","last_became_peered":"2024-08-03T21:00:50.243333+0000","last_unstale":"2024-08-03T21:00:51.938224+0000","last_undegraded":"2024-08-03T21:00:51.938224+0000","last_fullsized":"2024-08-03T21:00:51.938224+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_clean_scrub_stamp":"2024-08-03T21:00:49.195479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T08:50:30.366065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.21","version":"0'0","reported_seq":13,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-08-03T21:00:51.225546+0000","last_change":"2024-08-03T21:00:50.229431+0000","last_active":"2024-08-03T21:00:51.225546+0000","last_peered":"2024-08-03T21:00:51.225546+0000","last_clean":"2024-08-03T21:00:51.225546+0000","last_became_active":"2024-08-03T21:00:50.228793+0000","last_became_peered":"2024-08-03T21:00:50.228793+0000","last_unstale":"2024-08-03T21:00:51.225546+0000","last_undegraded":"2024-08-03T21:00:51.225546+0000","last_fullsized":"2024-08-03T21:00:51.225546+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_clean_scrub_stamp":"2024-08-03T21:00:49.195479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T03:21:17.046927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.20","version":"0'0","reported_seq":1,"reported_epoch":133,"state":"creating+peering","last_fresh":"2024-08-03T21:00:49.213774+0000","last_change":"2024-08-03T21:00:49.213774+0000","last_active":"2024-08-03T21:00:49.195479+0000","last_peered":"2024-08-03T21:00:49.195479+0000","last_clean":"2024-08-03T21:00:49.195479+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-08-03T21:00:49.213774+0000","last_undegraded":"2024-08-03T21:00:49.213774+0000","last_fullsized":"2024-08-03T21:00:49.213774+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_clean_scrub_stamp":"2024-08-03T21:00:49.195479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.1e","version":"0'0","reported_seq":13,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-08-03T21:00:51.939274+0000","last_change":"2024-08-03T21:00:50.235759+0000","last_active":"2024-08-03T21:00:51.939274+0000","last_peered":"2024-08-03T21:00:51.939274+0000","last_clean":"2024-08-03T21:00:51.939274+0000","last_became_active":"2024-08-03T21:00:50.235440+0000","last_became_peered":"2024-08-03T21:00:50.235440+0000","last_unstale":"2024-08-03T21:00:51.939274+0000","last_undegraded":"2024-08-03T21:00:51.939274+0000","last_fullsized":"2024-08-03T21:00:51.939274+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_clean_scrub_stamp":"2024-08-03T21:00:49.195479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-04T21:51:33.128276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.1f","version":"0'0","reported_seq":10,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-08-03T21:00:49.232640+0000","last_change":"2024-08-03T21:00:49.231401+0000","last_active":"2024-08-03T21:00:49.232640+0000","last_peered":"2024-08-03T21:00:49.232640+0000","last_clean":"2024-08-03T21:00:49.232640+0000","last_became_active":"2024-08-03T21:00:49.231089+0000","last_became_peered":"2024-08-03T21:00:49.231089+0000","last_unstale":"2024-08-03T21:00:49.232640+0000","last_undegraded":"2024-08-03T21:00:49.232640+0000","last_fullsized":"2024-08-03T21:00:49.232640+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:48.192422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:48.192422+0000","last_clean_scrub_stamp":"2024-08-03T21:00:48.192422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T01:29:44.046851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.1f","version":"0'0","reported_seq":13,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-08-03T21:00:51.225857+0000","last_change":"2024-08-03T21:00:50.226490+0000","last_active":"2024-08-03T21:00:51.225857+0000","last_peered":"2024-08-03T21:00:51.225857+0000","last_clean":"2024-08-03T21:00:51.225857+0000","last_became_active":"2024-08-03T21:00:50.226103+0000","last_became_peered":"2024-08-03T21:00:50.226103+0000","last_unstale":"2024-08-03T21:00:51.225857+0000","last_undegraded":"2024-08-03T21:00:51.225857+0000","last_fullsized":"2024-08-03T21:00:51.225857+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_clean_scrub_stamp":"2024-08-03T21:00:49.195479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T04:51:31.368175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.1e","version":"0'0","reported_seq":10,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-08-03T21:00:49.228209+0000","last_change":"2024-08-03T21:00:49.227358+0000","last_active":"2024-08-03T21:00:49.228209+0000","last_peered":"2024-08-03T21:00:49.228209+0000","last_clean":"2024-08-03T21:00:49.228209+0000","last_became_active":"2024-08-03T21:00:49.226904+0000","last_became_peered":"2024-08-03T21:00:49.226904+0000","last_unstale":"2024-08-03T21:00:49.228209+0000","last_undegraded":"2024-08-03T21:00:49.228209+0000","last_fullsized":"2024-08-03T21:00:49.228209+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:48.192422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:48.192422+0000","last_clean_scrub_stamp":"2024-08-03T21:00:48.192422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T07:44:13.158341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.1c","version":"0'0","reported_seq":1,"reported_epoch":133,"state":"creating+peering","last_fresh":"2024-08-03T21:00:49.214791+0000","last_change":"2024-08-03T21:00:49.214791+0000","last_active":"2024-08-03T21:00:49.195479+0000","last_peered":"2024-08-03T21:00:49.195479+0000","last_clean":"2024-08-03T21:00:49.195479+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-08-03T21:00:49.214791+0000","last_undegraded":"2024-08-03T21:00:49.214791+0000","last_fullsized":"2024-08-03T21:00:49.214791+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_clean_scrub_stamp":"2024-08-03T21:00:49.195479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.1d","version":"0'0","reported_seq":10,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-08-03T21:00:49.238854+0000","last_change":"2024-08-03T21:00:49.234514+0000","last_active":"2024-08-03T21:00:49.238854+0000","last_peered":"2024-08-03T21:00:49.238854+0000","last_clean":"2024-08-03T21:00:49.238854+0000","last_became_active":"2024-08-03T21:00:49.233091+0000","last_became_peered":"2024-08-03T21:00:49.233091+0000","last_unstale":"2024-08-03T21:00:49.238854+0000","last_undegraded":"2024-08-03T21:00:49.238854+0000","last_fullsized":"2024-08-03T21:00:49.238854+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:48.192422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:48.192422+0000","last_clean_scrub_stamp":"2024-08-03T21:00:48.192422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T02:44:09.712432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.1d","version":"0'0","reported_seq":13,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-08-03T21:00:51.939823+0000","last_change":"2024-08-03T21:00:50.235731+0000","last_active":"2024-08-03T21:00:51.939823+0000","last_peered":"2024-08-03T21:00:51.939823+0000","last_clean":"2024-08-03T21:00:51.939823+0000","last_became_active":"2024-08-03T21:00:50.234805+0000","last_became_peered":"2024-08-03T21:00:50.234805+0000","last_unstale":"2024-08-03T21:00:51.939823+0000","last_undegraded":"2024-08-03T21:00:51.939823+0000","last_fullsized":"2024-08-03T21:00:51.939823+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_clean_scrub_stamp":"2024-08-03T21:00:49.195479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T05:40:57.488842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.1c","version":"0'0","reported_seq":10,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-08-03T21:00:49.230596+0000","last_change":"2024-08-03T21:00:49.227739+0000","last_active":"2024-08-03T21:00:49.230596+0000","last_peered":"2024-08-03T21:00:49.230596+0000","last_clean":"2024-08-03T21:00:49.230596+0000","last_became_active":"2024-08-03T21:00:49.226569+0000","last_became_peered":"2024-08-03T21:00:49.226569+0000","last_unstale":"2024-08-03T21:00:49.230596+0000","last_undegraded":"2024-08-03T21:00:49.230596+0000","last_fullsized":"2024-08-03T21:00:49.230596+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:48.192422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:48.192422+0000","last_clean_scrub_stamp":"2024-08-03T21:00:48.192422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T08:47:16.511690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.1a","version":"0'0","reported_seq":1,"reported_epoch":133,"state":"creating+peering","last_fresh":"2024-08-03T21:00:49.211222+0000","last_change":"2024-08-03T21:00:49.211222+0000","last_active":"2024-08-03T21:00:49.195479+0000","last_peered":"2024-08-03T21:00:49.195479+0000","last_clean":"2024-08-03T21:00:49.195479+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-08-03T21:00:49.211222+0000","last_undegraded":"2024-08-03T21:00:49.211222+0000","last_fullsized":"2024-08-03T21:00:49.211222+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_clean_scrub_stamp":"2024-08-03T21:00:49.195479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.1b","version":"0'0","reported_seq":10,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-08-03T21:00:49.220521+0000","last_change":"2024-08-03T21:00:49.217597+0000","last_active":"2024-08-03T21:00:49.220521+0000","last_peered":"2024-08-03T21:00:49.220521+0000","last_clean":"2024-08-03T21:00:49.220521+0000","last_became_active":"2024-08-03T21:00:49.217212+0000","last_became_peered":"2024-08-03T21:00:49.217212+0000","last_unstale":"2024-08-03T21:00:49.220521+0000","last_undegraded":"2024-08-03T21:00:49.220521+0000","last_fullsized":"2024-08-03T21:00:49.220521+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:48.192422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:48.192422+0000","last_clean_scrub_stamp":"2024-08-03T21:00:48.192422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-04T22:22:37.333147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.1b","version":"0'0","reported_seq":1,"reported_epoch":133,"state":"creating+peering","last_fresh":"2024-08-03T21:00:49.215473+0000","last_change":"2024-08-03T21:00:49.215473+0000","last_active":"2024-08-03T21:00:49.195479+0000","last_peered":"2024-08-03T21:00:49.195479+0000","last_clean":"2024-08-03T21:00:49.195479+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-08-03T21:00:49.215473+0000","last_undegraded":"2024-08-03T21:00:49.215473+0000","last_fullsized":"2024-08-03T21:00:49.215473+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_clean_scrub_stamp":"2024-08-03T21:00:49.195479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.1a","version":"0'0","reported_seq":10,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-08-03T21:00:49.239275+0000","last_change":"2024-08-03T21:00:49.238475+0000","last_active":"2024-08-03T21:00:49.239275+0000","last_peered":"2024-08-03T21:00:49.239275+0000","last_clean":"2024-08-03T21:00:49.239275+0000","last_became_active":"2024-08-03T21:00:49.238001+0000","last_became_peered":"2024-08-03T21:00:49.238001+0000","last_unstale":"2024-08-03T21:00:49.239275+0000","last_undegraded":"2024-08-03T21:00:49.239275+0000","last_fullsized":"2024-08-03T21:00:49.239275+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:48.192422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:48.192422+0000","last_clean_scrub_stamp":"2024-08-03T21:00:48.192422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T07:44:13.158341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.18","version":"0'0","reported_seq":13,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-08-03T21:00:51.225613+0000","last_change":"2024-08-03T21:00:50.228305+0000","last_active":"2024-08-03T21:00:51.225613+0000","last_peered":"2024-08-03T21:00:51.225613+0000","last_clean":"2024-08-03T21:00:51.225613+0000","last_became_active":"2024-08-03T21:00:50.228009+0000","last_became_peered":"2024-08-03T21:00:50.228009+0000","last_unstale":"2024-08-03T21:00:51.225613+0000","last_undegraded":"2024-08-03T21:00:51.225613+0000","last_fullsized":"2024-08-03T21:00:51.225613+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_clean_scrub_stamp":"2024-08-03T21:00:49.195479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-04T23:22:25.975528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.19","version":"0'0","reported_seq":10,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-08-03T21:00:49.229663+0000","last_change":"2024-08-03T21:00:49.228986+0000","last_active":"2024-08-03T21:00:49.229663+0000","last_peered":"2024-08-03T21:00:49.229663+0000","last_clean":"2024-08-03T21:00:49.229663+0000","last_became_active":"2024-08-03T21:00:49.228761+0000","last_became_peered":"2024-08-03T21:00:49.228761+0000","last_unstale":"2024-08-03T21:00:49.229663+0000","last_undegraded":"2024-08-03T21:00:49.229663+0000","last_fullsized":"2024-08-03T21:00:49.229663+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:48.192422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:48.192422+0000","last_clean_scrub_stamp":"2024-08-03T21:00:48.192422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T02:44:18.436142+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.19","version":"0'0","reported_seq":13,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-08-03T21:00:51.940174+0000","last_change":"2024-08-03T21:00:50.235335+0000","last_active":"2024-08-03T21:00:51.940174+0000","last_peered":"2024-08-03T21:00:51.940174+0000","last_clean":"2024-08-03T21:00:51.940174+0000","last_became_active":"2024-08-03T21:00:50.234940+0000","last_became_peered":"2024-08-03T21:00:50.234940+0000","last_unstale":"2024-08-03T21:00:51.940174+0000","last_undegraded":"2024-08-03T21:00:51.940174+0000","last_fullsized":"2024-08-03T21:00:51.940174+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_clean_scrub_stamp":"2024-08-03T21:00:49.195479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T00:40:44.403536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.18","version":"0'0","reported_seq":10,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-08-03T21:00:49.237874+0000","last_change":"2024-08-03T21:00:49.236558+0000","last_active":"2024-08-03T21:00:49.237874+0000","last_peered":"2024-08-03T21:00:49.237874+0000","last_clean":"2024-08-03T21:00:49.237874+0000","last_became_active":"2024-08-03T21:00:49.236201+0000","last_became_peered":"2024-08-03T21:00:49.236201+0000","last_unstale":"2024-08-03T21:00:49.237874+0000","last_undegraded":"2024-08-03T21:00:49.237874+0000","last_fullsized":"2024-08-03T21:00:49.237874+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:48.192422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:48.192422+0000","last_clean_scrub_stamp":"2024-08-03T21:00:48.192422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-04T23:01:53.386452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.16","version":"0'0","reported_seq":1,"reported_epoch":133,"state":"creating+peering","last_fresh":"2024-08-03T21:00:49.219922+0000","last_change":"2024-08-03T21:00:49.219922+0000","last_active":"2024-08-03T21:00:49.195479+0000","last_peered":"2024-08-03T21:00:49.195479+0000","last_clean":"2024-08-03T21:00:49.195479+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-08-03T21:00:49.219922+0000","last_undegraded":"2024-08-03T21:00:49.219922+0000","last_fullsized":"2024-08-03T21:00:49.219922+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_clean_scrub_stamp":"2024-08-03T21:00:49.195479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.17","version":"0'0","reported_seq":10,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-08-03T21:00:49.229166+0000","last_change":"2024-08-03T21:00:49.226248+0000","last_active":"2024-08-03T21:00:49.229166+0000","last_peered":"2024-08-03T21:00:49.229166+0000","last_clean":"2024-08-03T21:00:49.229166+0000","last_became_active":"2024-08-03T21:00:49.224012+0000","last_became_peered":"2024-08-03T21:00:49.224012+0000","last_unstale":"2024-08-03T21:00:49.229166+0000","last_undegraded":"2024-08-03T21:00:49.229166+0000","last_fullsized":"2024-08-03T21:00:49.229166+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:48.192422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:48.192422+0000","last_clean_scrub_stamp":"2024-08-03T21:00:48.192422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T02:58:35.310272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.17","version":"0'0","reported_seq":13,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-08-03T21:00:51.940036+0000","last_change":"2024-08-03T21:00:50.244964+0000","last_active":"2024-08-03T21:00:51.940036+0000","last_peered":"2024-08-03T21:00:51.940036+0000","last_clean":"2024-08-03T21:00:51.940036+0000","last_became_active":"2024-08-03T21:00:50.244323+0000","last_became_peered":"2024-08-03T21:00:50.244323+0000","last_unstale":"2024-08-03T21:00:51.940036+0000","last_undegraded":"2024-08-03T21:00:51.940036+0000","last_fullsized":"2024-08-03T21:00:51.940036+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_clean_scrub_stamp":"2024-08-03T21:00:49.195479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-04T21:46:43.719103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.16","version":"0'0","reported_seq":15,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-08-03T21:00:51.225728+0000","last_change":"2024-08-03T21:00:49.233357+0000","last_active":"2024-08-03T21:00:51.225728+0000","last_peered":"2024-08-03T21:00:51.225728+0000","last_clean":"2024-08-03T21:00:51.225728+0000","last_became_active":"2024-08-03T21:00:49.232477+0000","last_became_peered":"2024-08-03T21:00:49.232477+0000","last_unstale":"2024-08-03T21:00:51.225728+0000","last_undegraded":"2024-08-03T21:00:51.225728+0000","last_fullsized":"2024-08-03T21:00:51.225728+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:48.192422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:48.192422+0000","last_clean_scrub_stamp":"2024-08-03T21:00:48.192422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T05:40:56.485784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.14","version":"0'0","reported_seq":1,"reported_epoch":133,"state":"creating+peering","last_fresh":"2024-08-03T21:00:49.210789+0000","last_change":"2024-08-03T21:00:49.210789+0000","last_active":"2024-08-03T21:00:49.195479+0000","last_peered":"2024-08-03T21:00:49.195479+0000","last_clean":"2024-08-03T21:00:49.195479+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-08-03T21:00:49.210789+0000","last_undegraded":"2024-08-03T21:00:49.210789+0000","last_fullsized":"2024-08-03T21:00:49.210789+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_clean_scrub_stamp":"2024-08-03T21:00:49.195479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.15","version":"0'0","reported_seq":10,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-08-03T21:00:49.225338+0000","last_change":"2024-08-03T21:00:49.224739+0000","last_active":"2024-08-03T21:00:49.225338+0000","last_peered":"2024-08-03T21:00:49.225338+0000","last_clean":"2024-08-03T21:00:49.225338+0000","last_became_active":"2024-08-03T21:00:49.224342+0000","last_became_peered":"2024-08-03T21:00:49.224342+0000","last_unstale":"2024-08-03T21:00:49.225338+0000","last_undegraded":"2024-08-03T21:00:49.225338+0000","last_fullsized":"2024-08-03T21:00:49.225338+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:48.192422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:48.192422+0000","last_clean_scrub_stamp":"2024-08-03T21:00:48.192422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-04T23:53:41.240790+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.15","version":"0'0","reported_seq":1,"reported_epoch":133,"state":"creating+peering","last_fresh":"2024-08-03T21:00:49.216465+0000","last_change":"2024-08-03T21:00:49.216465+0000","last_active":"2024-08-03T21:00:49.195479+0000","last_peered":"2024-08-03T21:00:49.195479+0000","last_clean":"2024-08-03T21:00:49.195479+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-08-03T21:00:49.216465+0000","last_undegraded":"2024-08-03T21:00:49.216465+0000","last_fullsized":"2024-08-03T21:00:49.216465+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_clean_scrub_stamp":"2024-08-03T21:00:49.195479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.14","version":"134'6","reported_seq":21,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-08-03T21:00:51.939402+0000","last_change":"2024-08-03T21:00:49.235410+0000","last_active":"2024-08-03T21:00:51.939402+0000","last_peered":"2024-08-03T21:00:51.939402+0000","last_clean":"2024-08-03T21:00:51.939402+0000","last_became_active":"2024-08-03T21:00:49.234999+0000","last_became_peered":"2024-08-03T21:00:49.234999+0000","last_unstale":"2024-08-03T21:00:51.939402+0000","last_undegraded":"2024-08-03T21:00:51.939402+0000","last_fullsized":"2024-08-03T21:00:51.939402+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:48.192422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:48.192422+0000","last_clean_scrub_stamp":"2024-08-03T21:00:48.192422+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T02:55:58.470527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.12","version":"0'0","reported_seq":13,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-08-03T21:00:51.225979+0000","last_change":"2024-08-03T21:00:50.218292+0000","last_active":"2024-08-03T21:00:51.225979+0000","last_peered":"2024-08-03T21:00:51.225979+0000","last_clean":"2024-08-03T21:00:51.225979+0000","last_became_active":"2024-08-03T21:00:50.217979+0000","last_became_peered":"2024-08-03T21:00:50.217979+0000","last_unstale":"2024-08-03T21:00:51.225979+0000","last_undegraded":"2024-08-03T21:00:51.225979+0000","last_fullsized":"2024-08-03T21:00:51.225979+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_clean_scrub_stamp":"2024-08-03T21:00:49.195479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T06:55:48.820648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.13","version":"134'2","reported_seq":17,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-08-03T21:00:51.939709+0000","last_change":"2024-08-03T21:00:49.224867+0000","last_active":"2024-08-03T21:00:51.939709+0000","last_peered":"2024-08-03T21:00:51.939709+0000","last_clean":"2024-08-03T21:00:51.939709+0000","last_became_active":"2024-08-03T21:00:49.224555+0000","last_became_peered":"2024-08-03T21:00:49.224555+0000","last_unstale":"2024-08-03T21:00:51.939709+0000","last_undegraded":"2024-08-03T21:00:51.939709+0000","last_fullsized":"2024-08-03T21:00:51.939709+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:48.192422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:48.192422+0000","last_clean_scrub_stamp":"2024-08-03T21:00:48.192422+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T07:55:25.225889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.13","version":"0'0","reported_seq":1,"reported_epoch":133,"state":"creating+peering","last_fresh":"2024-08-03T21:00:49.211586+0000","last_change":"2024-08-03T21:00:49.211586+0000","last_active":"2024-08-03T21:00:49.195479+0000","last_peered":"2024-08-03T21:00:49.195479+0000","last_clean":"2024-08-03T21:00:49.195479+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-08-03T21:00:49.211586+0000","last_undegraded":"2024-08-03T21:00:49.211586+0000","last_fullsized":"2024-08-03T21:00:49.211586+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_clean_scrub_stamp":"2024-08-03T21:00:49.195479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.12","version":"0'0","reported_seq":15,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-08-03T21:00:51.938928+0000","last_change":"2024-08-03T21:00:49.224078+0000","last_active":"2024-08-03T21:00:51.938928+0000","last_peered":"2024-08-03T21:00:51.938928+0000","last_clean":"2024-08-03T21:00:51.938928+0000","last_became_active":"2024-08-03T21:00:49.223717+0000","last_became_peered":"2024-08-03T21:00:49.223717+0000","last_unstale":"2024-08-03T21:00:51.938928+0000","last_undegraded":"2024-08-03T21:00:51.938928+0000","last_fullsized":"2024-08-03T21:00:51.938928+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:48.192422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:48.192422+0000","last_clean_scrub_stamp":"2024-08-03T21:00:48.192422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T04:56:22.864880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.10","version":"0'0","reported_seq":13,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-08-03T21:00:51.939036+0000","last_change":"2024-08-03T21:00:50.231767+0000","last_active":"2024-08-03T21:00:51.939036+0000","last_peered":"2024-08-03T21:00:51.939036+0000","last_clean":"2024-08-03T21:00:51.939036+0000","last_became_active":"2024-08-03T21:00:50.229726+0000","last_became_peered":"2024-08-03T21:00:50.229726+0000","last_unstale":"2024-08-03T21:00:51.939036+0000","last_undegraded":"2024-08-03T21:00:51.939036+0000","last_fullsized":"2024-08-03T21:00:51.939036+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_clean_scrub_stamp":"2024-08-03T21:00:49.195479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T00:14:18.409160+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.11","version":"0'0","reported_seq":15,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-08-03T21:00:51.939969+0000","last_change":"2024-08-03T21:00:49.219205+0000","last_active":"2024-08-03T21:00:51.939969+0000","last_peered":"2024-08-03T21:00:51.939969+0000","last_clean":"2024-08-03T21:00:51.939969+0000","last_became_active":"2024-08-03T21:00:49.218855+0000","last_became_peered":"2024-08-03T21:00:49.218855+0000","last_unstale":"2024-08-03T21:00:51.939969+0000","last_undegraded":"2024-08-03T21:00:51.939969+0000","last_fullsized":"2024-08-03T21:00:51.939969+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:48.192422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:48.192422+0000","last_clean_scrub_stamp":"2024-08-03T21:00:48.192422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T00:24:17.141394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.11","version":"0'0","reported_seq":13,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-08-03T21:00:51.939150+0000","last_change":"2024-08-03T21:00:50.230042+0000","last_active":"2024-08-03T21:00:51.939150+0000","last_peered":"2024-08-03T21:00:51.939150+0000","last_clean":"2024-08-03T21:00:51.939150+0000","last_became_active":"2024-08-03T21:00:50.229716+0000","last_became_peered":"2024-08-03T21:00:50.229716+0000","last_unstale":"2024-08-03T21:00:51.939150+0000","last_undegraded":"2024-08-03T21:00:51.939150+0000","last_fullsized":"2024-08-03T21:00:51.939150+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_clean_scrub_stamp":"2024-08-03T21:00:49.195479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T08:05:48.492097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.10","version":"0'0","reported_seq":10,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-08-03T21:00:49.222553+0000","last_change":"2024-08-03T21:00:49.219610+0000","last_active":"2024-08-03T21:00:49.222553+0000","last_peered":"2024-08-03T21:00:49.222553+0000","last_clean":"2024-08-03T21:00:49.222553+0000","last_became_active":"2024-08-03T21:00:49.219289+0000","last_became_peered":"2024-08-03T21:00:49.219289+0000","last_unstale":"2024-08-03T21:00:49.222553+0000","last_undegraded":"2024-08-03T21:00:49.222553+0000","last_fullsized":"2024-08-03T21:00:49.222553+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:48.192422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:48.192422+0000","last_clean_scrub_stamp":"2024-08-03T21:00:48.192422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-04T22:10:58.211954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.f","version":"0'0","reported_seq":1,"reported_epoch":133,"state":"creating+peering","last_fresh":"2024-08-03T21:00:49.209245+0000","last_change":"2024-08-03T21:00:49.209245+0000","last_active":"2024-08-03T21:00:49.195479+0000","last_peered":"2024-08-03T21:00:49.195479+0000","last_clean":"2024-08-03T21:00:49.195479+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-08-03T21:00:49.209245+0000","last_undegraded":"2024-08-03T21:00:49.209245+0000","last_fullsized":"2024-08-03T21:00:49.209245+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_clean_scrub_stamp":"2024-08-03T21:00:49.195479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.e","version":"134'2","reported_seq":17,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-08-03T21:00:51.939110+0000","last_change":"2024-08-03T21:00:49.221709+0000","last_active":"2024-08-03T21:00:51.939110+0000","last_peered":"2024-08-03T21:00:51.939110+0000","last_clean":"2024-08-03T21:00:51.939110+0000","last_became_active":"2024-08-03T21:00:49.221146+0000","last_became_peered":"2024-08-03T21:00:49.221146+0000","last_unstale":"2024-08-03T21:00:51.939110+0000","last_undegraded":"2024-08-03T21:00:51.939110+0000","last_fullsized":"2024-08-03T21:00:51.939110+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:48.192422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:48.192422+0000","last_clean_scrub_stamp":"2024-08-03T21:00:48.192422+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T02:29:53.585049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.c","version":"0'0","reported_seq":1,"reported_epoch":133,"state":"creating+peering","last_fresh":"2024-08-03T21:00:49.211956+0000","last_change":"2024-08-03T21:00:49.211956+0000","last_active":"2024-08-03T21:00:49.195479+0000","last_peered":"2024-08-03T21:00:49.195479+0000","last_clean":"2024-08-03T21:00:49.195479+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-08-03T21:00:49.211956+0000","last_undegraded":"2024-08-03T21:00:49.211956+0000","last_fullsized":"2024-08-03T21:00:49.211956+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_clean_scrub_stamp":"2024-08-03T21:00:49.195479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.d","version":"0'0","reported_seq":10,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-08-03T21:00:49.216969+0000","last_change":"2024-08-03T21:00:49.214725+0000","last_active":"2024-08-03T21:00:49.216969+0000","last_peered":"2024-08-03T21:00:49.216969+0000","last_clean":"2024-08-03T21:00:49.216969+0000","last_became_active":"2024-08-03T21:00:49.214155+0000","last_became_peered":"2024-08-03T21:00:49.214155+0000","last_unstale":"2024-08-03T21:00:49.216969+0000","last_undegraded":"2024-08-03T21:00:49.216969+0000","last_fullsized":"2024-08-03T21:00:49.216969+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:48.192422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:48.192422+0000","last_clean_scrub_stamp":"2024-08-03T21:00:48.192422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T07:43:13.286648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.d","version":"0'0","reported_seq":1,"reported_epoch":133,"state":"creating+peering","last_fresh":"2024-08-03T21:00:49.213316+0000","last_change":"2024-08-03T21:00:49.213316+0000","last_active":"2024-08-03T21:00:49.195479+0000","last_peered":"2024-08-03T21:00:49.195479+0000","last_clean":"2024-08-03T21:00:49.195479+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-08-03T21:00:49.213316+0000","last_undegraded":"2024-08-03T21:00:49.213316+0000","last_fullsized":"2024-08-03T21:00:49.213316+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_clean_scrub_stamp":"2024-08-03T21:00:49.195479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.c","version":"0'0","reported_seq":10,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-08-03T21:00:49.230569+0000","last_change":"2024-08-03T21:00:49.225810+0000","last_active":"2024-08-03T21:00:49.230569+0000","last_peered":"2024-08-03T21:00:49.230569+0000","last_clean":"2024-08-03T21:00:49.230569+0000","last_became_active":"2024-08-03T21:00:49.225501+0000","last_became_peered":"2024-08-03T21:00:49.225501+0000","last_unstale":"2024-08-03T21:00:49.230569+0000","last_undegraded":"2024-08-03T21:00:49.230569+0000","last_fullsized":"2024-08-03T21:00:49.230569+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:48.192422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:48.192422+0000","last_clean_scrub_stamp":"2024-08-03T21:00:48.192422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-04T23:49:05.539690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.a","version":"0'0","reported_seq":13,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-08-03T21:00:51.938230+0000","last_change":"2024-08-03T21:00:50.238896+0000","last_active":"2024-08-03T21:00:51.938230+0000","last_peered":"2024-08-03T21:00:51.938230+0000","last_clean":"2024-08-03T21:00:51.938230+0000","last_became_active":"2024-08-03T21:00:50.238557+0000","last_became_peered":"2024-08-03T21:00:50.238557+0000","last_unstale":"2024-08-03T21:00:51.938230+0000","last_undegraded":"2024-08-03T21:00:51.938230+0000","last_fullsized":"2024-08-03T21:00:51.938230+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_clean_scrub_stamp":"2024-08-03T21:00:49.195479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-04T22:10:38.257913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.b","version":"0'0","reported_seq":10,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-08-03T21:00:49.238315+0000","last_change":"2024-08-03T21:00:49.233291+0000","last_active":"2024-08-03T21:00:49.238315+0000","last_peered":"2024-08-03T21:00:49.238315+0000","last_clean":"2024-08-03T21:00:49.238315+0000","last_became_active":"2024-08-03T21:00:49.232973+0000","last_became_peered":"2024-08-03T21:00:49.232973+0000","last_unstale":"2024-08-03T21:00:49.238315+0000","last_undegraded":"2024-08-03T21:00:49.238315+0000","last_fullsized":"2024-08-03T21:00:49.238315+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:48.192422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:48.192422+0000","last_clean_scrub_stamp":"2024-08-03T21:00:48.192422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T02:44:18.436142+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.b","version":"0'0","reported_seq":13,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-08-03T21:00:51.939647+0000","last_change":"2024-08-03T21:00:50.242588+0000","last_active":"2024-08-03T21:00:51.939647+0000","last_peered":"2024-08-03T21:00:51.939647+0000","last_clean":"2024-08-03T21:00:51.939647+0000","last_became_active":"2024-08-03T21:00:50.239795+0000","last_became_peered":"2024-08-03T21:00:50.239795+0000","last_unstale":"2024-08-03T21:00:51.939647+0000","last_undegraded":"2024-08-03T21:00:51.939647+0000","last_fullsized":"2024-08-03T21:00:51.939647+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_clean_scrub_stamp":"2024-08-03T21:00:49.195479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T06:53:11.234357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.a","version":"0'0","reported_seq":10,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-08-03T21:00:49.223212+0000","last_change":"2024-08-03T21:00:49.222066+0000","last_active":"2024-08-03T21:00:49.223212+0000","last_peered":"2024-08-03T21:00:49.223212+0000","last_clean":"2024-08-03T21:00:49.223212+0000","last_became_active":"2024-08-03T21:00:49.221757+0000","last_became_peered":"2024-08-03T21:00:49.221757+0000","last_unstale":"2024-08-03T21:00:49.223212+0000","last_undegraded":"2024-08-03T21:00:49.223212+0000","last_fullsized":"2024-08-03T21:00:49.223212+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:48.192422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:48.192422+0000","last_clean_scrub_stamp":"2024-08-03T21:00:48.192422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T08:47:16.511690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.8","version":"0'0","reported_seq":13,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-08-03T21:00:51.937406+0000","last_change":"2024-08-03T21:00:50.240625+0000","last_active":"2024-08-03T21:00:51.937406+0000","last_peered":"2024-08-03T21:00:51.937406+0000","last_clean":"2024-08-03T21:00:51.937406+0000","last_became_active":"2024-08-03T21:00:50.240139+0000","last_became_peered":"2024-08-03T21:00:50.240139+0000","last_unstale":"2024-08-03T21:00:51.937406+0000","last_undegraded":"2024-08-03T21:00:51.937406+0000","last_fullsized":"2024-08-03T21:00:51.937406+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_clean_scrub_stamp":"2024-08-03T21:00:49.195479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T06:58:48.625406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.9","version":"0'0","reported_seq":10,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-08-03T21:00:49.236203+0000","last_change":"2024-08-03T21:00:49.235212+0000","last_active":"2024-08-03T21:00:49.236203+0000","last_peered":"2024-08-03T21:00:49.236203+0000","last_clean":"2024-08-03T21:00:49.236203+0000","last_became_active":"2024-08-03T21:00:49.234793+0000","last_became_peered":"2024-08-03T21:00:49.234793+0000","last_unstale":"2024-08-03T21:00:49.236203+0000","last_undegraded":"2024-08-03T21:00:49.236203+0000","last_fullsized":"2024-08-03T21:00:49.236203+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:48.192422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:48.192422+0000","last_clean_scrub_stamp":"2024-08-03T21:00:48.192422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T07:43:13.286648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.9","version":"0'0","reported_seq":1,"reported_epoch":133,"state":"creating+peering","last_fresh":"2024-08-03T21:00:49.213171+0000","last_change":"2024-08-03T21:00:49.213171+0000","last_active":"2024-08-03T21:00:49.195479+0000","last_peered":"2024-08-03T21:00:49.195479+0000","last_clean":"2024-08-03T21:00:49.195479+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-08-03T21:00:49.213171+0000","last_undegraded":"2024-08-03T21:00:49.213171+0000","last_fullsized":"2024-08-03T21:00:49.213171+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_clean_scrub_stamp":"2024-08-03T21:00:49.195479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.8","version":"0'0","reported_seq":10,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-08-03T21:00:49.234238+0000","last_change":"2024-08-03T21:00:49.230326+0000","last_active":"2024-08-03T21:00:49.234238+0000","last_peered":"2024-08-03T21:00:49.234238+0000","last_clean":"2024-08-03T21:00:49.234238+0000","last_became_active":"2024-08-03T21:00:49.229867+0000","last_became_peered":"2024-08-03T21:00:49.229867+0000","last_unstale":"2024-08-03T21:00:49.234238+0000","last_undegraded":"2024-08-03T21:00:49.234238+0000","last_fullsized":"2024-08-03T21:00:49.234238+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:48.192422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:48.192422+0000","last_clean_scrub_stamp":"2024-08-03T21:00:48.192422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T05:44:38.082933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.6","version":"0'0","reported_seq":1,"reported_epoch":133,"state":"creating+peering","last_fresh":"2024-08-03T21:00:49.217620+0000","last_change":"2024-08-03T21:00:49.217620+0000","last_active":"2024-08-03T21:00:49.195479+0000","last_peered":"2024-08-03T21:00:49.195479+0000","last_clean":"2024-08-03T21:00:49.195479+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-08-03T21:00:49.217620+0000","last_undegraded":"2024-08-03T21:00:49.217620+0000","last_fullsized":"2024-08-03T21:00:49.217620+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_clean_scrub_stamp":"2024-08-03T21:00:49.195479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.7","version":"0'0","reported_seq":10,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-08-03T21:00:49.223032+0000","last_change":"2024-08-03T21:00:49.220966+0000","last_active":"2024-08-03T21:00:49.223032+0000","last_peered":"2024-08-03T21:00:49.223032+0000","last_clean":"2024-08-03T21:00:49.223032+0000","last_became_active":"2024-08-03T21:00:49.220433+0000","last_became_peered":"2024-08-03T21:00:49.220433+0000","last_unstale":"2024-08-03T21:00:49.223032+0000","last_undegraded":"2024-08-03T21:00:49.223032+0000","last_fullsized":"2024-08-03T21:00:49.223032+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:48.192422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:48.192422+0000","last_clean_scrub_stamp":"2024-08-03T21:00:48.192422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-04T23:49:05.539690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.7","version":"0'0","reported_seq":1,"reported_epoch":133,"state":"creating+peering","last_fresh":"2024-08-03T21:00:49.212025+0000","last_change":"2024-08-03T21:00:49.212025+0000","last_active":"2024-08-03T21:00:49.195479+0000","last_peered":"2024-08-03T21:00:49.195479+0000","last_clean":"2024-08-03T21:00:49.195479+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-08-03T21:00:49.212025+0000","last_undegraded":"2024-08-03T21:00:49.212025+0000","last_fullsized":"2024-08-03T21:00:49.212025+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_clean_scrub_stamp":"2024-08-03T21:00:49.195479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.6","version":"0'0","reported_seq":10,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-08-03T21:00:49.238693+0000","last_change":"2024-08-03T21:00:49.235126+0000","last_active":"2024-08-03T21:00:49.238693+0000","last_peered":"2024-08-03T21:00:49.238693+0000","last_clean":"2024-08-03T21:00:49.238693+0000","last_became_active":"2024-08-03T21:00:49.233960+0000","last_became_peered":"2024-08-03T21:00:49.233960+0000","last_unstale":"2024-08-03T21:00:49.238693+0000","last_undegraded":"2024-08-03T21:00:49.238693+0000","last_fullsized":"2024-08-03T21:00:49.238693+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:48.192422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:48.192422+0000","last_clean_scrub_stamp":"2024-08-03T21:00:48.192422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T07:27:35.968034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.4","version":"0'0","reported_seq":1,"reported_epoch":133,"state":"creating+peering","last_fresh":"2024-08-03T21:00:49.212551+0000","last_change":"2024-08-03T21:00:49.212551+0000","last_active":"2024-08-03T21:00:49.195479+0000","last_peered":"2024-08-03T21:00:49.195479+0000","last_clean":"2024-08-03T21:00:49.195479+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-08-03T21:00:49.212551+0000","last_undegraded":"2024-08-03T21:00:49.212551+0000","last_fullsized":"2024-08-03T21:00:49.212551+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_clean_scrub_stamp":"2024-08-03T21:00:49.195479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.5","version":"0'0","reported_seq":10,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-08-03T21:00:49.230593+0000","last_change":"2024-08-03T21:00:49.227879+0000","last_active":"2024-08-03T21:00:49.230593+0000","last_peered":"2024-08-03T21:00:49.230593+0000","last_clean":"2024-08-03T21:00:49.230593+0000","last_became_active":"2024-08-03T21:00:49.227574+0000","last_became_peered":"2024-08-03T21:00:49.227574+0000","last_unstale":"2024-08-03T21:00:49.230593+0000","last_undegraded":"2024-08-03T21:00:49.230593+0000","last_fullsized":"2024-08-03T21:00:49.230593+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:48.192422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:48.192422+0000","last_clean_scrub_stamp":"2024-08-03T21:00:48.192422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T07:43:13.286648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.5","version":"0'0","reported_seq":1,"reported_epoch":133,"state":"creating+peering","last_fresh":"2024-08-03T21:00:49.209221+0000","last_change":"2024-08-03T21:00:49.209221+0000","last_active":"2024-08-03T21:00:49.195479+0000","last_peered":"2024-08-03T21:00:49.195479+0000","last_clean":"2024-08-03T21:00:49.195479+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-08-03T21:00:49.209221+0000","last_undegraded":"2024-08-03T21:00:49.209221+0000","last_fullsized":"2024-08-03T21:00:49.209221+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_clean_scrub_stamp":"2024-08-03T21:00:49.195479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.4","version":"0'0","reported_seq":10,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-08-03T21:00:49.224181+0000","last_change":"2024-08-03T21:00:49.221928+0000","last_active":"2024-08-03T21:00:49.224181+0000","last_peered":"2024-08-03T21:00:49.224181+0000","last_clean":"2024-08-03T21:00:49.224181+0000","last_became_active":"2024-08-03T21:00:49.221623+0000","last_became_peered":"2024-08-03T21:00:49.221623+0000","last_unstale":"2024-08-03T21:00:49.224181+0000","last_undegraded":"2024-08-03T21:00:49.224181+0000","last_fullsized":"2024-08-03T21:00:49.224181+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:48.192422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:48.192422+0000","last_clean_scrub_stamp":"2024-08-03T21:00:48.192422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T01:29:44.046851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.2","version":"0'0","reported_seq":13,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-08-03T21:00:51.938470+0000","last_change":"2024-08-03T21:00:50.232514+0000","last_active":"2024-08-03T21:00:51.938470+0000","last_peered":"2024-08-03T21:00:51.938470+0000","last_clean":"2024-08-03T21:00:51.938470+0000","last_became_active":"2024-08-03T21:00:50.232148+0000","last_became_peered":"2024-08-03T21:00:50.232148+0000","last_unstale":"2024-08-03T21:00:51.938470+0000","last_undegraded":"2024-08-03T21:00:51.938470+0000","last_fullsized":"2024-08-03T21:00:51.938470+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_clean_scrub_stamp":"2024-08-03T21:00:49.195479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-04T23:42:52.232285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.3","version":"0'0","reported_seq":10,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-08-03T21:00:49.238736+0000","last_change":"2024-08-03T21:00:49.235356+0000","last_active":"2024-08-03T21:00:49.238736+0000","last_peered":"2024-08-03T21:00:49.238736+0000","last_clean":"2024-08-03T21:00:49.238736+0000","last_became_active":"2024-08-03T21:00:49.234801+0000","last_became_peered":"2024-08-03T21:00:49.234801+0000","last_unstale":"2024-08-03T21:00:49.238736+0000","last_undegraded":"2024-08-03T21:00:49.238736+0000","last_fullsized":"2024-08-03T21:00:49.238736+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:48.192422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:48.192422+0000","last_clean_scrub_stamp":"2024-08-03T21:00:48.192422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-04T23:53:41.240790+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.3","version":"0'0","reported_seq":13,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-08-03T21:00:51.939532+0000","last_change":"2024-08-03T21:00:50.233609+0000","last_active":"2024-08-03T21:00:51.939532+0000","last_peered":"2024-08-03T21:00:51.939532+0000","last_clean":"2024-08-03T21:00:51.939532+0000","last_became_active":"2024-08-03T21:00:50.231678+0000","last_became_peered":"2024-08-03T21:00:50.231678+0000","last_unstale":"2024-08-03T21:00:51.939532+0000","last_undegraded":"2024-08-03T21:00:51.939532+0000","last_fullsized":"2024-08-03T21:00:51.939532+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_clean_scrub_stamp":"2024-08-03T21:00:49.195479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T02:34:58.346689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.2","version":"0'0","reported_seq":10,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-08-03T21:00:49.219683+0000","last_change":"2024-08-03T21:00:49.218982+0000","last_active":"2024-08-03T21:00:49.219683+0000","last_peered":"2024-08-03T21:00:49.219683+0000","last_clean":"2024-08-03T21:00:49.219683+0000","last_became_active":"2024-08-03T21:00:49.217673+0000","last_became_peered":"2024-08-03T21:00:49.217673+0000","last_unstale":"2024-08-03T21:00:49.219683+0000","last_undegraded":"2024-08-03T21:00:49.219683+0000","last_fullsized":"2024-08-03T21:00:49.219683+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:48.192422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:48.192422+0000","last_clean_scrub_stamp":"2024-08-03T21:00:48.192422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T05:44:38.082933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.0","version":"0'0","reported_seq":13,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-08-03T21:00:51.938141+0000","last_change":"2024-08-03T21:00:50.234121+0000","last_active":"2024-08-03T21:00:51.938141+0000","last_peered":"2024-08-03T21:00:51.938141+0000","last_clean":"2024-08-03T21:00:51.938141+0000","last_became_active":"2024-08-03T21:00:50.233758+0000","last_became_peered":"2024-08-03T21:00:50.233758+0000","last_unstale":"2024-08-03T21:00:51.938141+0000","last_undegraded":"2024-08-03T21:00:51.938141+0000","last_fullsized":"2024-08-03T21:00:51.938141+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_clean_scrub_stamp":"2024-08-03T21:00:49.195479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T05:28:59.859843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.1","version":"0'0","reported_seq":15,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-08-03T21:00:51.225900+0000","last_change":"2024-08-03T21:00:49.230098+0000","last_active":"2024-08-03T21:00:51.225900+0000","last_peered":"2024-08-03T21:00:51.225900+0000","last_clean":"2024-08-03T21:00:51.225900+0000","last_became_active":"2024-08-03T21:00:49.229777+0000","last_became_peered":"2024-08-03T21:00:49.229777+0000","last_unstale":"2024-08-03T21:00:51.225900+0000","last_undegraded":"2024-08-03T21:00:51.225900+0000","last_fullsized":"2024-08-03T21:00:51.225900+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:48.192422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:48.192422+0000","last_clean_scrub_stamp":"2024-08-03T21:00:48.192422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T06:34:18.629210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.1","version":"0'0","reported_seq":1,"reported_epoch":133,"state":"creating+peering","last_fresh":"2024-08-03T21:00:49.210235+0000","last_change":"2024-08-03T21:00:49.210235+0000","last_active":"2024-08-03T21:00:49.195479+0000","last_peered":"2024-08-03T21:00:49.195479+0000","last_clean":"2024-08-03T21:00:49.195479+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-08-03T21:00:49.210235+0000","last_undegraded":"2024-08-03T21:00:49.210235+0000","last_fullsized":"2024-08-03T21:00:49.210235+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_clean_scrub_stamp":"2024-08-03T21:00:49.195479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.0","version":"0'0","reported_seq":15,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-08-03T21:00:51.938216+0000","last_change":"2024-08-03T21:00:49.231962+0000","last_active":"2024-08-03T21:00:51.938216+0000","last_peered":"2024-08-03T21:00:51.938216+0000","last_clean":"2024-08-03T21:00:51.938216+0000","last_became_active":"2024-08-03T21:00:49.231607+0000","last_became_peered":"2024-08-03T21:00:49.231607+0000","last_unstale":"2024-08-03T21:00:51.938216+0000","last_undegraded":"2024-08-03T21:00:51.938216+0000","last_fullsized":"2024-08-03T21:00:51.938216+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:48.192422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:48.192422+0000","last_clean_scrub_stamp":"2024-08-03T21:00:48.192422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T08:45:01.729460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.e","version":"0'0","reported_seq":13,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-08-03T21:00:51.939469+0000","last_change":"2024-08-03T21:00:50.249570+0000","last_active":"2024-08-03T21:00:51.939469+0000","last_peered":"2024-08-03T21:00:51.939469+0000","last_clean":"2024-08-03T21:00:51.939469+0000","last_became_active":"2024-08-03T21:00:50.240733+0000","last_became_peered":"2024-08-03T21:00:50.240733+0000","last_unstale":"2024-08-03T21:00:51.939469+0000","last_undegraded":"2024-08-03T21:00:51.939469+0000","last_fullsized":"2024-08-03T21:00:51.939469+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_clean_scrub_stamp":"2024-08-03T21:00:49.195479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T08:43:59.747967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.f","version":"0'0","reported_seq":10,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-08-03T21:00:49.235936+0000","last_change":"2024-08-03T21:00:49.234956+0000","last_active":"2024-08-03T21:00:49.235936+0000","last_peered":"2024-08-03T21:00:49.235936+0000","last_clean":"2024-08-03T21:00:49.235936+0000","last_became_active":"2024-08-03T21:00:49.234351+0000","last_became_peered":"2024-08-03T21:00:49.234351+0000","last_unstale":"2024-08-03T21:00:49.235936+0000","last_undegraded":"2024-08-03T21:00:49.235936+0000","last_fullsized":"2024-08-03T21:00:49.235936+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:48.192422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:48.192422+0000","last_clean_scrub_stamp":"2024-08-03T21:00:48.192422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-04T22:46:01.864033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"16'72","reported_seq":327,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-08-03T21:00:49.218344+0000","last_change":"2024-08-03T20:47:30.547632+0000","last_active":"2024-08-03T21:00:49.218344+0000","last_peered":"2024-08-03T21:00:49.218344+0000","last_clean":"2024-08-03T21:00:49.218344+0000","last_became_active":"2024-08-03T20:47:30.546255+0000","last_became_peered":"2024-08-03T20:47:30.546255+0000","last_unstale":"2024-08-03T21:00:49.218344+0000","last_undegraded":"2024-08-03T21:00:49.218344+0000","last_fullsized":"2024-08-03T21:00:49.218344+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:47:28.874451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:47:28.874451+0000","last_clean_scrub_stamp":"2024-08-03T20:47:28.874451+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T00:19:40.548812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":15,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":14,"num_pg":32,"stat_sum":{"num_bytes":1646,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":3404,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":10,"ondisk_log_size":10,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509603,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41732,"kb_used_data":2228,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330108,"statfs":{"total":96636764160,"available":96594030592,"internally_reserved":0,"allocated":2281472,"data_stored":1401923,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":40434122},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":6,"up_from":15,"seq":64424509604,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36988,"kb_used_data":1648,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334852,"statfs":{"total":96636764160,"available":96598888448,"internally_reserved":0,"allocated":1687552,"data_stored":811555,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":15,"seq":64424509604,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36988,"kb_used_data":1648,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334852,"statfs":{"total":96636764160,"available":96598888448,"internally_reserved":0,"allocated":1687552,"data_stored":811555,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509603,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41148,"kb_used_data":1648,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330692,"statfs":{"total":96636764160,"available":96594628608,"internally_reserved":0,"allocated":1687552,"data_stored":811555,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509605,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":29252,"kb_used_data":2228,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94342588,"statfs":{"total":96636764160,"available":96606810112,"internally_reserved":0,"allocated":2281472,"data_stored":1401923,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542309,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37048,"kb_used_data":1704,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334792,"statfs":{"total":96636764160,"available":96598827008,"internally_reserved":0,"allocated":1744896,"data_stored":832641,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":3,"up_from":14,"seq":60129542307,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37044,"kb_used_data":1700,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334796,"statfs":{"total":96636764160,"available":96598831104,"internally_reserved":0,"allocated":1740800,"data_stored":832539,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":2,"up_from":14,"seq":60129542308,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41192,"kb_used_data":1688,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330648,"statfs":{"total":96636764160,"available":96594583552,"internally_reserved":0,"allocated":1728512,"data_stored":830939,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-08-03T21:00:55.266 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-08-03T21:00:55.531 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:55.529+0000 7f72a0200640 1 -- 172.21.15.32:0/1707077684 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f7298165160 msgr2=0x7f7298167580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:00:55.532 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:55.529+0000 7f72a0200640 1 --2- 172.21.15.32:0/1707077684 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f7298165160 0x7f7298167580 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7f7290005fd0 tx=0x7f729002a840 comp rx=0 tx=0).stop 2024-08-03T21:00:55.532 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:55.529+0000 7f72a0200640 1 -- 172.21.15.32:0/1707077684 shutdown_connections 2024-08-03T21:00:55.532 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:55.529+0000 7f72a0200640 1 --2- 172.21.15.32:0/1707077684 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f7298165160 0x7f7298167580 unknown :-1 s=CLOSED pgs=654 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:55.532 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:55.529+0000 7f72a0200640 1 -- 172.21.15.32:0/1707077684 >> 172.21.15.32:0/1707077684 conn(0x7f72980b1bd0 msgr2=0x7f72980b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T21:00:55.532 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:55.530+0000 7f72a0200640 1 -- 172.21.15.32:0/1707077684 shutdown_connections 2024-08-03T21:00:55.532 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:55.530+0000 7f72a0200640 1 -- 172.21.15.32:0/1707077684 wait complete. 2024-08-03T21:00:55.533 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:55.532+0000 7f72a0200640 1 Processor -- start 2024-08-03T21:00:55.533 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:55.532+0000 7f72a0200640 1 -- start start 2024-08-03T21:00:55.536 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:55.533+0000 7f72a0200640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f729806e930 0x7f7298156390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:00:55.536 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:55.533+0000 7f72a0200640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f7298144f90 0x7f7298137f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:00:55.536 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:55.533+0000 7f72a0200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7298165160 0x7f729813bbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:00:55.536 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:55.533+0000 7f72a0200640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f72980cf9c0 con 0x7f7298165160 2024-08-03T21:00:55.537 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:55.533+0000 7f72a0200640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f72980cf840 con 0x7f729806e930 2024-08-03T21:00:55.537 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:55.533+0000 7f72a0200640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7f7298167ac0 con 0x7f7298144f90 2024-08-03T21:00:55.537 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:55.533+0000 7f729e200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7298165160 0x7f729813bbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:00:55.537 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:55.534+0000 7f729e200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7298165160 0x7f729813bbf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:50090/0 (socket says 172.21.15.32:50090) 2024-08-03T21:00:55.537 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:55.534+0000 7f729e200640 1 -- 172.21.15.32:0/3341571672 learned_addr learned my addr 172.21.15.32:0/3341571672 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T21:00:55.537 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:55.534+0000 7f729d800640 1 --2- 172.21.15.32:0/3341571672 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f729806e930 0x7f7298156390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:00:55.537 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:55.534+0000 7f729ce00640 1 --2- 172.21.15.32:0/3341571672 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f7298144f90 0x7f7298137f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:00:55.537 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:55.534+0000 7f729e200640 1 -- 172.21.15.32:0/3341571672 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f7298144f90 msgr2=0x7f7298137f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:00:55.537 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:55.534+0000 7f729e200640 1 --2- 172.21.15.32:0/3341571672 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f7298144f90 0x7f7298137f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:55.537 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:55.534+0000 7f729e200640 1 -- 172.21.15.32:0/3341571672 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f729806e930 msgr2=0x7f7298156390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:00:55.537 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:55.534+0000 7f729e200640 1 --2- 172.21.15.32:0/3341571672 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f729806e930 0x7f7298156390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:55.537 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:55.534+0000 7f729e200640 1 -- 172.21.15.32:0/3341571672 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7298138440 con 0x7f7298165160 2024-08-03T21:00:55.538 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:55.534+0000 7f729e200640 1 --2- 172.21.15.32:0/3341571672 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7298165160 0x7f729813bbf0 secure :-1 s=READY pgs=2175 cs=0 l=1 rev1=1 crypto rx=0x7f7290009cd0 tx=0x7f7290032c70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T21:00:55.538 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:55.535+0000 7f7295600640 1 -- 172.21.15.32:0/3341571672 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7290040070 con 0x7f7298165160 2024-08-03T21:00:55.538 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:55.535+0000 7f7295600640 1 -- 172.21.15.32:0/3341571672 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f7290004540 con 0x7f7298165160 2024-08-03T21:00:55.538 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:55.535+0000 7f7295600640 1 -- 172.21.15.32:0/3341571672 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7290003e60 con 0x7f7298165160 2024-08-03T21:00:55.538 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:55.535+0000 7f72a0200640 1 -- 172.21.15.32:0/3341571672 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7298138c50 con 0x7f7298165160 2024-08-03T21:00:55.538 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:55.535+0000 7f72a0200640 1 -- 172.21.15.32:0/3341571672 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f729813e450 con 0x7f7298165160 2024-08-03T21:00:55.538 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:55.536+0000 7f72a0200640 1 -- 172.21.15.32:0/3341571672 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7298073e60 con 0x7f7298165160 2024-08-03T21:00:55.539 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:55.537+0000 7f7295600640 1 -- 172.21.15.32:0/3341571672 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f7290043070 con 0x7f7298165160 2024-08-03T21:00:55.539 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:55.538+0000 7f7295600640 1 --2- 172.21.15.32:0/3341571672 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f7278077010 0x7f7278079520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:00:55.540 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:55.538+0000 7f7295600640 1 -- 172.21.15.32:0/3341571672 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(135..135 src has 1..135) ==== 5709+0+0 (secure 0 0 0) 0x7f72900b9b50 con 0x7f7298165160 2024-08-03T21:00:55.540 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:55.539+0000 7f729d800640 1 --2- 172.21.15.32:0/3341571672 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f7278077010 0x7f7278079520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:00:55.541 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:55.539+0000 7f729d800640 1 --2- 172.21.15.32:0/3341571672 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f7278077010 0x7f7278079520 secure :-1 s=READY pgs=872 cs=0 l=1 rev1=1 crypto rx=0x7f728c001120 tx=0x7f728c009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T21:00:55.544 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:55.543+0000 7f7295600640 1 -- 172.21.15.32:0/3341571672 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7f7298138440 con 0x7f7298165160 2024-08-03T21:00:55.746 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:55.745+0000 7f72a0200640 1 -- 172.21.15.32:0/3341571672 --> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f7298167350 con 0x7f7278077010 2024-08-03T21:00:55.751 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:55.750+0000 7f7295600640 1 -- 172.21.15.32:0/3341571672 <== mgr.4106 v2:172.21.15.118:6800/4205950699 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+249456 (secure 0 0 0) 0x7f7298167350 con 0x7f7278077010 2024-08-03T21:00:55.755 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-03T21:00:55.756 INFO:teuthology.orchestra.run.smithi032.stderr:dumped all 2024-08-03T21:00:55.758 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:55.754+0000 7f72a0200640 1 -- 172.21.15.32:0/3341571672 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f7278077010 msgr2=0x7f7278079520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:00:55.758 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:55.754+0000 7f72a0200640 1 --2- 172.21.15.32:0/3341571672 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f7278077010 0x7f7278079520 secure :-1 s=READY pgs=872 cs=0 l=1 rev1=1 crypto rx=0x7f728c001120 tx=0x7f728c009040 comp rx=0 tx=0).stop 2024-08-03T21:00:55.758 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:55.754+0000 7f72a0200640 1 -- 172.21.15.32:0/3341571672 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7298165160 msgr2=0x7f729813bbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:00:55.758 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:55.754+0000 7f72a0200640 1 --2- 172.21.15.32:0/3341571672 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7298165160 0x7f729813bbf0 secure :-1 s=READY pgs=2175 cs=0 l=1 rev1=1 crypto rx=0x7f7290009cd0 tx=0x7f7290032c70 comp rx=0 tx=0).stop 2024-08-03T21:00:55.758 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:55.754+0000 7f72a0200640 1 -- 172.21.15.32:0/3341571672 shutdown_connections 2024-08-03T21:00:55.758 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:55.754+0000 7f72a0200640 1 --2- 172.21.15.32:0/3341571672 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f7278077010 0x7f7278079520 unknown :-1 s=CLOSED pgs=872 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:55.758 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:55.754+0000 7f72a0200640 1 --2- 172.21.15.32:0/3341571672 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7298165160 0x7f729813bbf0 unknown :-1 s=CLOSED pgs=2175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:55.758 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:55.754+0000 7f72a0200640 1 --2- 172.21.15.32:0/3341571672 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f7298144f90 0x7f7298137f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:55.758 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:55.754+0000 7f72a0200640 1 --2- 172.21.15.32:0/3341571672 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f729806e930 0x7f7298156390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:55.758 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:55.754+0000 7f72a0200640 1 -- 172.21.15.32:0/3341571672 >> 172.21.15.32:0/3341571672 conn(0x7f72980b1bd0 msgr2=0x7f7298165f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T21:00:55.759 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:55.754+0000 7f72a0200640 1 -- 172.21.15.32:0/3341571672 shutdown_connections 2024-08-03T21:00:55.759 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:55.754+0000 7f72a0200640 1 -- 172.21.15.32:0/3341571672 wait complete. 2024-08-03T21:00:55.776 INFO:teuthology.orchestra.run.smithi032.stdout:{"pg_ready":true,"pg_map":{"version":531,"stamp":"2024-08-03T21:00:54.454153+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592014,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":113,"num_write_kb":1638,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":82,"ondisk_log_size":82,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":81,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":301392,"kb_used_data":14492,"kb_used_omap":12,"kb_used_meta":286771,"kb_avail":754673328,"statfs":{"total":773094113280,"available":772785487872,"internally_reserved":0,"allocated":14839808,"data_stored":7734630,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":293654065},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":20,"apply_latency_ms":20,"commit_latency_ns":20000000,"apply_latency_ns":20000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":1646,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001779"},"pg_stats":[{"pgid":"15.3f","version":"0'0","reported_seq":1,"reported_epoch":133,"state":"creating+peering","last_fresh":"2024-08-03T21:00:49.214906+0000","last_change":"2024-08-03T21:00:49.214906+0000","last_active":"2024-08-03T21:00:49.195479+0000","last_peered":"2024-08-03T21:00:49.195479+0000","last_clean":"2024-08-03T21:00:49.195479+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-08-03T21:00:49.214906+0000","last_undegraded":"2024-08-03T21:00:49.214906+0000","last_fullsized":"2024-08-03T21:00:49.214906+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_clean_scrub_stamp":"2024-08-03T21:00:49.195479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.3e","version":"0'0","reported_seq":1,"reported_epoch":133,"state":"creating+peering","last_fresh":"2024-08-03T21:00:49.221252+0000","last_change":"2024-08-03T21:00:49.221252+0000","last_active":"2024-08-03T21:00:49.195479+0000","last_peered":"2024-08-03T21:00:49.195479+0000","last_clean":"2024-08-03T21:00:49.195479+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-08-03T21:00:49.221252+0000","last_undegraded":"2024-08-03T21:00:49.221252+0000","last_fullsized":"2024-08-03T21:00:49.221252+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_clean_scrub_stamp":"2024-08-03T21:00:49.195479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.3d","version":"0'0","reported_seq":1,"reported_epoch":133,"state":"creating+peering","last_fresh":"2024-08-03T21:00:49.214132+0000","last_change":"2024-08-03T21:00:49.214132+0000","last_active":"2024-08-03T21:00:49.195479+0000","last_peered":"2024-08-03T21:00:49.195479+0000","last_clean":"2024-08-03T21:00:49.195479+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-08-03T21:00:49.214132+0000","last_undegraded":"2024-08-03T21:00:49.214132+0000","last_fullsized":"2024-08-03T21:00:49.214132+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_clean_scrub_stamp":"2024-08-03T21:00:49.195479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.3c","version":"0'0","reported_seq":13,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-08-03T21:00:51.225682+0000","last_change":"2024-08-03T21:00:50.243269+0000","last_active":"2024-08-03T21:00:51.225682+0000","last_peered":"2024-08-03T21:00:51.225682+0000","last_clean":"2024-08-03T21:00:51.225682+0000","last_became_active":"2024-08-03T21:00:50.242985+0000","last_became_peered":"2024-08-03T21:00:50.242985+0000","last_unstale":"2024-08-03T21:00:51.225682+0000","last_undegraded":"2024-08-03T21:00:51.225682+0000","last_fullsized":"2024-08-03T21:00:51.225682+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_clean_scrub_stamp":"2024-08-03T21:00:49.195479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T03:32:19.871676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.3b","version":"0'0","reported_seq":1,"reported_epoch":133,"state":"creating+peering","last_fresh":"2024-08-03T21:00:49.212243+0000","last_change":"2024-08-03T21:00:49.212243+0000","last_active":"2024-08-03T21:00:49.195479+0000","last_peered":"2024-08-03T21:00:49.195479+0000","last_clean":"2024-08-03T21:00:49.195479+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-08-03T21:00:49.212243+0000","last_undegraded":"2024-08-03T21:00:49.212243+0000","last_fullsized":"2024-08-03T21:00:49.212243+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_clean_scrub_stamp":"2024-08-03T21:00:49.195479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.3a","version":"0'0","reported_seq":1,"reported_epoch":133,"state":"creating+peering","last_fresh":"2024-08-03T21:00:49.215377+0000","last_change":"2024-08-03T21:00:49.215377+0000","last_active":"2024-08-03T21:00:49.195479+0000","last_peered":"2024-08-03T21:00:49.195479+0000","last_clean":"2024-08-03T21:00:49.195479+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-08-03T21:00:49.215377+0000","last_undegraded":"2024-08-03T21:00:49.215377+0000","last_fullsized":"2024-08-03T21:00:49.215377+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_clean_scrub_stamp":"2024-08-03T21:00:49.195479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.39","version":"0'0","reported_seq":1,"reported_epoch":133,"state":"creating+peering","last_fresh":"2024-08-03T21:00:49.213231+0000","last_change":"2024-08-03T21:00:49.213231+0000","last_active":"2024-08-03T21:00:49.195479+0000","last_peered":"2024-08-03T21:00:49.195479+0000","last_clean":"2024-08-03T21:00:49.195479+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-08-03T21:00:49.213231+0000","last_undegraded":"2024-08-03T21:00:49.213231+0000","last_fullsized":"2024-08-03T21:00:49.213231+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_clean_scrub_stamp":"2024-08-03T21:00:49.195479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.38","version":"0'0","reported_seq":1,"reported_epoch":133,"state":"creating+peering","last_fresh":"2024-08-03T21:00:49.214019+0000","last_change":"2024-08-03T21:00:49.214019+0000","last_active":"2024-08-03T21:00:49.195479+0000","last_peered":"2024-08-03T21:00:49.195479+0000","last_clean":"2024-08-03T21:00:49.195479+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-08-03T21:00:49.214019+0000","last_undegraded":"2024-08-03T21:00:49.214019+0000","last_fullsized":"2024-08-03T21:00:49.214019+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_clean_scrub_stamp":"2024-08-03T21:00:49.195479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.37","version":"0'0","reported_seq":13,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-08-03T21:00:51.939823+0000","last_change":"2024-08-03T21:00:50.223154+0000","last_active":"2024-08-03T21:00:51.939823+0000","last_peered":"2024-08-03T21:00:51.939823+0000","last_clean":"2024-08-03T21:00:51.939823+0000","last_became_active":"2024-08-03T21:00:50.222779+0000","last_became_peered":"2024-08-03T21:00:50.222779+0000","last_unstale":"2024-08-03T21:00:51.939823+0000","last_undegraded":"2024-08-03T21:00:51.939823+0000","last_fullsized":"2024-08-03T21:00:51.939823+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_clean_scrub_stamp":"2024-08-03T21:00:49.195479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T06:34:19.632267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.36","version":"0'0","reported_seq":1,"reported_epoch":133,"state":"creating+peering","last_fresh":"2024-08-03T21:00:49.211444+0000","last_change":"2024-08-03T21:00:49.211444+0000","last_active":"2024-08-03T21:00:49.195479+0000","last_peered":"2024-08-03T21:00:49.195479+0000","last_clean":"2024-08-03T21:00:49.195479+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-08-03T21:00:49.211444+0000","last_undegraded":"2024-08-03T21:00:49.211444+0000","last_fullsized":"2024-08-03T21:00:49.211444+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_clean_scrub_stamp":"2024-08-03T21:00:49.195479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.35","version":"0'0","reported_seq":1,"reported_epoch":133,"state":"creating+peering","last_fresh":"2024-08-03T21:00:49.217444+0000","last_change":"2024-08-03T21:00:49.217444+0000","last_active":"2024-08-03T21:00:49.195479+0000","last_peered":"2024-08-03T21:00:49.195479+0000","last_clean":"2024-08-03T21:00:49.195479+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-08-03T21:00:49.217444+0000","last_undegraded":"2024-08-03T21:00:49.217444+0000","last_fullsized":"2024-08-03T21:00:49.217444+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_clean_scrub_stamp":"2024-08-03T21:00:49.195479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.34","version":"0'0","reported_seq":13,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-08-03T21:00:51.939323+0000","last_change":"2024-08-03T21:00:50.229025+0000","last_active":"2024-08-03T21:00:51.939323+0000","last_peered":"2024-08-03T21:00:51.939323+0000","last_clean":"2024-08-03T21:00:51.939323+0000","last_became_active":"2024-08-03T21:00:50.228698+0000","last_became_peered":"2024-08-03T21:00:50.228698+0000","last_unstale":"2024-08-03T21:00:51.939323+0000","last_undegraded":"2024-08-03T21:00:51.939323+0000","last_fullsized":"2024-08-03T21:00:51.939323+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_clean_scrub_stamp":"2024-08-03T21:00:49.195479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T06:51:09.079780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.33","version":"0'0","reported_seq":13,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-08-03T21:00:51.225611+0000","last_change":"2024-08-03T21:00:50.224093+0000","last_active":"2024-08-03T21:00:51.225611+0000","last_peered":"2024-08-03T21:00:51.225611+0000","last_clean":"2024-08-03T21:00:51.225611+0000","last_became_active":"2024-08-03T21:00:50.222966+0000","last_became_peered":"2024-08-03T21:00:50.222966+0000","last_unstale":"2024-08-03T21:00:51.225611+0000","last_undegraded":"2024-08-03T21:00:51.225611+0000","last_fullsized":"2024-08-03T21:00:51.225611+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_clean_scrub_stamp":"2024-08-03T21:00:49.195479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T04:11:36.038037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.32","version":"0'0","reported_seq":13,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-08-03T21:00:51.937848+0000","last_change":"2024-08-03T21:00:50.230657+0000","last_active":"2024-08-03T21:00:51.937848+0000","last_peered":"2024-08-03T21:00:51.937848+0000","last_clean":"2024-08-03T21:00:51.937848+0000","last_became_active":"2024-08-03T21:00:50.230356+0000","last_became_peered":"2024-08-03T21:00:50.230356+0000","last_unstale":"2024-08-03T21:00:51.937848+0000","last_undegraded":"2024-08-03T21:00:51.937848+0000","last_fullsized":"2024-08-03T21:00:51.937848+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_clean_scrub_stamp":"2024-08-03T21:00:49.195479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-04T21:20:29.060152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.31","version":"0'0","reported_seq":13,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-08-03T21:00:51.225596+0000","last_change":"2024-08-03T21:00:50.236123+0000","last_active":"2024-08-03T21:00:51.225596+0000","last_peered":"2024-08-03T21:00:51.225596+0000","last_clean":"2024-08-03T21:00:51.225596+0000","last_became_active":"2024-08-03T21:00:50.235747+0000","last_became_peered":"2024-08-03T21:00:50.235747+0000","last_unstale":"2024-08-03T21:00:51.225596+0000","last_undegraded":"2024-08-03T21:00:51.225596+0000","last_fullsized":"2024-08-03T21:00:51.225596+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_clean_scrub_stamp":"2024-08-03T21:00:49.195479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T05:25:30.476710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.30","version":"0'0","reported_seq":1,"reported_epoch":133,"state":"creating+peering","last_fresh":"2024-08-03T21:00:49.214934+0000","last_change":"2024-08-03T21:00:49.214934+0000","last_active":"2024-08-03T21:00:49.195479+0000","last_peered":"2024-08-03T21:00:49.195479+0000","last_clean":"2024-08-03T21:00:49.195479+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-08-03T21:00:49.214934+0000","last_undegraded":"2024-08-03T21:00:49.214934+0000","last_fullsized":"2024-08-03T21:00:49.214934+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_clean_scrub_stamp":"2024-08-03T21:00:49.195479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.2f","version":"0'0","reported_seq":1,"reported_epoch":133,"state":"creating+peering","last_fresh":"2024-08-03T21:00:49.220256+0000","last_change":"2024-08-03T21:00:49.220256+0000","last_active":"2024-08-03T21:00:49.195479+0000","last_peered":"2024-08-03T21:00:49.195479+0000","last_clean":"2024-08-03T21:00:49.195479+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-08-03T21:00:49.220256+0000","last_undegraded":"2024-08-03T21:00:49.220256+0000","last_fullsized":"2024-08-03T21:00:49.220256+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_clean_scrub_stamp":"2024-08-03T21:00:49.195479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.2e","version":"0'0","reported_seq":13,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-08-03T21:00:51.225830+0000","last_change":"2024-08-03T21:00:50.241634+0000","last_active":"2024-08-03T21:00:51.225830+0000","last_peered":"2024-08-03T21:00:51.225830+0000","last_clean":"2024-08-03T21:00:51.225830+0000","last_became_active":"2024-08-03T21:00:50.241263+0000","last_became_peered":"2024-08-03T21:00:50.241263+0000","last_unstale":"2024-08-03T21:00:51.225830+0000","last_undegraded":"2024-08-03T21:00:51.225830+0000","last_fullsized":"2024-08-03T21:00:51.225830+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_clean_scrub_stamp":"2024-08-03T21:00:49.195479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T02:07:46.123334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.2d","version":"0'0","reported_seq":13,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-08-03T21:00:51.938842+0000","last_change":"2024-08-03T21:00:50.243910+0000","last_active":"2024-08-03T21:00:51.938842+0000","last_peered":"2024-08-03T21:00:51.938842+0000","last_clean":"2024-08-03T21:00:51.938842+0000","last_became_active":"2024-08-03T21:00:50.243054+0000","last_became_peered":"2024-08-03T21:00:50.243054+0000","last_unstale":"2024-08-03T21:00:51.938842+0000","last_undegraded":"2024-08-03T21:00:51.938842+0000","last_fullsized":"2024-08-03T21:00:51.938842+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_clean_scrub_stamp":"2024-08-03T21:00:49.195479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T08:31:27.530602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.2c","version":"0'0","reported_seq":13,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-08-03T21:00:51.939427+0000","last_change":"2024-08-03T21:00:50.223678+0000","last_active":"2024-08-03T21:00:51.939427+0000","last_peered":"2024-08-03T21:00:51.939427+0000","last_clean":"2024-08-03T21:00:51.939427+0000","last_became_active":"2024-08-03T21:00:50.223357+0000","last_became_peered":"2024-08-03T21:00:50.223357+0000","last_unstale":"2024-08-03T21:00:51.939427+0000","last_undegraded":"2024-08-03T21:00:51.939427+0000","last_fullsized":"2024-08-03T21:00:51.939427+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_clean_scrub_stamp":"2024-08-03T21:00:49.195479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T07:48:35.449963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.2b","version":"0'0","reported_seq":13,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-08-03T21:00:51.939465+0000","last_change":"2024-08-03T21:00:50.243413+0000","last_active":"2024-08-03T21:00:51.939465+0000","last_peered":"2024-08-03T21:00:51.939465+0000","last_clean":"2024-08-03T21:00:51.939465+0000","last_became_active":"2024-08-03T21:00:50.243004+0000","last_became_peered":"2024-08-03T21:00:50.243004+0000","last_unstale":"2024-08-03T21:00:51.939465+0000","last_undegraded":"2024-08-03T21:00:51.939465+0000","last_fullsized":"2024-08-03T21:00:51.939465+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_clean_scrub_stamp":"2024-08-03T21:00:49.195479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-04T23:21:29.132988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.2a","version":"0'0","reported_seq":1,"reported_epoch":133,"state":"creating+peering","last_fresh":"2024-08-03T21:00:49.215684+0000","last_change":"2024-08-03T21:00:49.215684+0000","last_active":"2024-08-03T21:00:49.195479+0000","last_peered":"2024-08-03T21:00:49.195479+0000","last_clean":"2024-08-03T21:00:49.195479+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-08-03T21:00:49.215684+0000","last_undegraded":"2024-08-03T21:00:49.215684+0000","last_fullsized":"2024-08-03T21:00:49.215684+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_clean_scrub_stamp":"2024-08-03T21:00:49.195479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.29","version":"0'0","reported_seq":1,"reported_epoch":133,"state":"creating+peering","last_fresh":"2024-08-03T21:00:49.214421+0000","last_change":"2024-08-03T21:00:49.214421+0000","last_active":"2024-08-03T21:00:49.195479+0000","last_peered":"2024-08-03T21:00:49.195479+0000","last_clean":"2024-08-03T21:00:49.195479+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-08-03T21:00:49.214421+0000","last_undegraded":"2024-08-03T21:00:49.214421+0000","last_fullsized":"2024-08-03T21:00:49.214421+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_clean_scrub_stamp":"2024-08-03T21:00:49.195479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.28","version":"0'0","reported_seq":1,"reported_epoch":133,"state":"creating+peering","last_fresh":"2024-08-03T21:00:49.214489+0000","last_change":"2024-08-03T21:00:49.214489+0000","last_active":"2024-08-03T21:00:49.195479+0000","last_peered":"2024-08-03T21:00:49.195479+0000","last_clean":"2024-08-03T21:00:49.195479+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-08-03T21:00:49.214489+0000","last_undegraded":"2024-08-03T21:00:49.214489+0000","last_fullsized":"2024-08-03T21:00:49.214489+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_clean_scrub_stamp":"2024-08-03T21:00:49.195479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.27","version":"0'0","reported_seq":1,"reported_epoch":133,"state":"creating+peering","last_fresh":"2024-08-03T21:00:49.211525+0000","last_change":"2024-08-03T21:00:49.211525+0000","last_active":"2024-08-03T21:00:49.195479+0000","last_peered":"2024-08-03T21:00:49.195479+0000","last_clean":"2024-08-03T21:00:49.195479+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-08-03T21:00:49.211525+0000","last_undegraded":"2024-08-03T21:00:49.211525+0000","last_fullsized":"2024-08-03T21:00:49.211525+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_clean_scrub_stamp":"2024-08-03T21:00:49.195479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.26","version":"0'0","reported_seq":1,"reported_epoch":133,"state":"creating+peering","last_fresh":"2024-08-03T21:00:49.220518+0000","last_change":"2024-08-03T21:00:49.220518+0000","last_active":"2024-08-03T21:00:49.195479+0000","last_peered":"2024-08-03T21:00:49.195479+0000","last_clean":"2024-08-03T21:00:49.195479+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-08-03T21:00:49.220518+0000","last_undegraded":"2024-08-03T21:00:49.220518+0000","last_fullsized":"2024-08-03T21:00:49.220518+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_clean_scrub_stamp":"2024-08-03T21:00:49.195479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.25","version":"0'0","reported_seq":1,"reported_epoch":133,"state":"creating+peering","last_fresh":"2024-08-03T21:00:49.216934+0000","last_change":"2024-08-03T21:00:49.216934+0000","last_active":"2024-08-03T21:00:49.195479+0000","last_peered":"2024-08-03T21:00:49.195479+0000","last_clean":"2024-08-03T21:00:49.195479+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-08-03T21:00:49.216934+0000","last_undegraded":"2024-08-03T21:00:49.216934+0000","last_fullsized":"2024-08-03T21:00:49.216934+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_clean_scrub_stamp":"2024-08-03T21:00:49.195479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.24","version":"0'0","reported_seq":1,"reported_epoch":133,"state":"creating+peering","last_fresh":"2024-08-03T21:00:49.212944+0000","last_change":"2024-08-03T21:00:49.212944+0000","last_active":"2024-08-03T21:00:49.195479+0000","last_peered":"2024-08-03T21:00:49.195479+0000","last_clean":"2024-08-03T21:00:49.195479+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-08-03T21:00:49.212944+0000","last_undegraded":"2024-08-03T21:00:49.212944+0000","last_fullsized":"2024-08-03T21:00:49.212944+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_clean_scrub_stamp":"2024-08-03T21:00:49.195479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.23","version":"0'0","reported_seq":1,"reported_epoch":133,"state":"creating+peering","last_fresh":"2024-08-03T21:00:49.219217+0000","last_change":"2024-08-03T21:00:49.219217+0000","last_active":"2024-08-03T21:00:49.195479+0000","last_peered":"2024-08-03T21:00:49.195479+0000","last_clean":"2024-08-03T21:00:49.195479+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-08-03T21:00:49.219217+0000","last_undegraded":"2024-08-03T21:00:49.219217+0000","last_fullsized":"2024-08-03T21:00:49.219217+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_clean_scrub_stamp":"2024-08-03T21:00:49.195479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.22","version":"0'0","reported_seq":13,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-08-03T21:00:51.938224+0000","last_change":"2024-08-03T21:00:50.243735+0000","last_active":"2024-08-03T21:00:51.938224+0000","last_peered":"2024-08-03T21:00:51.938224+0000","last_clean":"2024-08-03T21:00:51.938224+0000","last_became_active":"2024-08-03T21:00:50.243333+0000","last_became_peered":"2024-08-03T21:00:50.243333+0000","last_unstale":"2024-08-03T21:00:51.938224+0000","last_undegraded":"2024-08-03T21:00:51.938224+0000","last_fullsized":"2024-08-03T21:00:51.938224+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_clean_scrub_stamp":"2024-08-03T21:00:49.195479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T08:50:30.366065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.21","version":"0'0","reported_seq":13,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-08-03T21:00:51.225546+0000","last_change":"2024-08-03T21:00:50.229431+0000","last_active":"2024-08-03T21:00:51.225546+0000","last_peered":"2024-08-03T21:00:51.225546+0000","last_clean":"2024-08-03T21:00:51.225546+0000","last_became_active":"2024-08-03T21:00:50.228793+0000","last_became_peered":"2024-08-03T21:00:50.228793+0000","last_unstale":"2024-08-03T21:00:51.225546+0000","last_undegraded":"2024-08-03T21:00:51.225546+0000","last_fullsized":"2024-08-03T21:00:51.225546+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_clean_scrub_stamp":"2024-08-03T21:00:49.195479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T03:21:17.046927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.20","version":"0'0","reported_seq":1,"reported_epoch":133,"state":"creating+peering","last_fresh":"2024-08-03T21:00:49.213774+0000","last_change":"2024-08-03T21:00:49.213774+0000","last_active":"2024-08-03T21:00:49.195479+0000","last_peered":"2024-08-03T21:00:49.195479+0000","last_clean":"2024-08-03T21:00:49.195479+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-08-03T21:00:49.213774+0000","last_undegraded":"2024-08-03T21:00:49.213774+0000","last_fullsized":"2024-08-03T21:00:49.213774+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_clean_scrub_stamp":"2024-08-03T21:00:49.195479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.1e","version":"0'0","reported_seq":13,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-08-03T21:00:51.939274+0000","last_change":"2024-08-03T21:00:50.235759+0000","last_active":"2024-08-03T21:00:51.939274+0000","last_peered":"2024-08-03T21:00:51.939274+0000","last_clean":"2024-08-03T21:00:51.939274+0000","last_became_active":"2024-08-03T21:00:50.235440+0000","last_became_peered":"2024-08-03T21:00:50.235440+0000","last_unstale":"2024-08-03T21:00:51.939274+0000","last_undegraded":"2024-08-03T21:00:51.939274+0000","last_fullsized":"2024-08-03T21:00:51.939274+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_clean_scrub_stamp":"2024-08-03T21:00:49.195479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-04T21:51:33.128276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.1f","version":"0'0","reported_seq":10,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-08-03T21:00:49.232640+0000","last_change":"2024-08-03T21:00:49.231401+0000","last_active":"2024-08-03T21:00:49.232640+0000","last_peered":"2024-08-03T21:00:49.232640+0000","last_clean":"2024-08-03T21:00:49.232640+0000","last_became_active":"2024-08-03T21:00:49.231089+0000","last_became_peered":"2024-08-03T21:00:49.231089+0000","last_unstale":"2024-08-03T21:00:49.232640+0000","last_undegraded":"2024-08-03T21:00:49.232640+0000","last_fullsized":"2024-08-03T21:00:49.232640+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:48.192422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:48.192422+0000","last_clean_scrub_stamp":"2024-08-03T21:00:48.192422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T01:29:44.046851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.1f","version":"0'0","reported_seq":13,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-08-03T21:00:51.225857+0000","last_change":"2024-08-03T21:00:50.226490+0000","last_active":"2024-08-03T21:00:51.225857+0000","last_peered":"2024-08-03T21:00:51.225857+0000","last_clean":"2024-08-03T21:00:51.225857+0000","last_became_active":"2024-08-03T21:00:50.226103+0000","last_became_peered":"2024-08-03T21:00:50.226103+0000","last_unstale":"2024-08-03T21:00:51.225857+0000","last_undegraded":"2024-08-03T21:00:51.225857+0000","last_fullsized":"2024-08-03T21:00:51.225857+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_clean_scrub_stamp":"2024-08-03T21:00:49.195479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T04:51:31.368175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.1e","version":"0'0","reported_seq":10,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-08-03T21:00:49.228209+0000","last_change":"2024-08-03T21:00:49.227358+0000","last_active":"2024-08-03T21:00:49.228209+0000","last_peered":"2024-08-03T21:00:49.228209+0000","last_clean":"2024-08-03T21:00:49.228209+0000","last_became_active":"2024-08-03T21:00:49.226904+0000","last_became_peered":"2024-08-03T21:00:49.226904+0000","last_unstale":"2024-08-03T21:00:49.228209+0000","last_undegraded":"2024-08-03T21:00:49.228209+0000","last_fullsized":"2024-08-03T21:00:49.228209+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:48.192422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:48.192422+0000","last_clean_scrub_stamp":"2024-08-03T21:00:48.192422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T07:44:13.158341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.1c","version":"0'0","reported_seq":1,"reported_epoch":133,"state":"creating+peering","last_fresh":"2024-08-03T21:00:49.214791+0000","last_change":"2024-08-03T21:00:49.214791+0000","last_active":"2024-08-03T21:00:49.195479+0000","last_peered":"2024-08-03T21:00:49.195479+0000","last_clean":"2024-08-03T21:00:49.195479+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-08-03T21:00:49.214791+0000","last_undegraded":"2024-08-03T21:00:49.214791+0000","last_fullsized":"2024-08-03T21:00:49.214791+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_clean_scrub_stamp":"2024-08-03T21:00:49.195479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.1d","version":"0'0","reported_seq":10,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-08-03T21:00:49.238854+0000","last_change":"2024-08-03T21:00:49.234514+0000","last_active":"2024-08-03T21:00:49.238854+0000","last_peered":"2024-08-03T21:00:49.238854+0000","last_clean":"2024-08-03T21:00:49.238854+0000","last_became_active":"2024-08-03T21:00:49.233091+0000","last_became_peered":"2024-08-03T21:00:49.233091+0000","last_unstale":"2024-08-03T21:00:49.238854+0000","last_undegraded":"2024-08-03T21:00:49.238854+0000","last_fullsized":"2024-08-03T21:00:49.238854+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:48.192422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:48.192422+0000","last_clean_scrub_stamp":"2024-08-03T21:00:48.192422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T02:44:09.712432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.1d","version":"0'0","reported_seq":13,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-08-03T21:00:51.939823+0000","last_change":"2024-08-03T21:00:50.235731+0000","last_active":"2024-08-03T21:00:51.939823+0000","last_peered":"2024-08-03T21:00:51.939823+0000","last_clean":"2024-08-03T21:00:51.939823+0000","last_became_active":"2024-08-03T21:00:50.234805+0000","last_became_peered":"2024-08-03T21:00:50.234805+0000","last_unstale":"2024-08-03T21:00:51.939823+0000","last_undegraded":"2024-08-03T21:00:51.939823+0000","last_fullsized":"2024-08-03T21:00:51.939823+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_clean_scrub_stamp":"2024-08-03T21:00:49.195479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T05:40:57.488842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.1c","version":"0'0","reported_seq":10,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-08-03T21:00:49.230596+0000","last_change":"2024-08-03T21:00:49.227739+0000","last_active":"2024-08-03T21:00:49.230596+0000","last_peered":"2024-08-03T21:00:49.230596+0000","last_clean":"2024-08-03T21:00:49.230596+0000","last_became_active":"2024-08-03T21:00:49.226569+0000","last_became_peered":"2024-08-03T21:00:49.226569+0000","last_unstale":"2024-08-03T21:00:49.230596+0000","last_undegraded":"2024-08-03T21:00:49.230596+0000","last_fullsized":"2024-08-03T21:00:49.230596+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:48.192422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:48.192422+0000","last_clean_scrub_stamp":"2024-08-03T21:00:48.192422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T08:47:16.511690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.1a","version":"0'0","reported_seq":1,"reported_epoch":133,"state":"creating+peering","last_fresh":"2024-08-03T21:00:49.211222+0000","last_change":"2024-08-03T21:00:49.211222+0000","last_active":"2024-08-03T21:00:49.195479+0000","last_peered":"2024-08-03T21:00:49.195479+0000","last_clean":"2024-08-03T21:00:49.195479+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-08-03T21:00:49.211222+0000","last_undegraded":"2024-08-03T21:00:49.211222+0000","last_fullsized":"2024-08-03T21:00:49.211222+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_clean_scrub_stamp":"2024-08-03T21:00:49.195479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.1b","version":"0'0","reported_seq":10,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-08-03T21:00:49.220521+0000","last_change":"2024-08-03T21:00:49.217597+0000","last_active":"2024-08-03T21:00:49.220521+0000","last_peered":"2024-08-03T21:00:49.220521+0000","last_clean":"2024-08-03T21:00:49.220521+0000","last_became_active":"2024-08-03T21:00:49.217212+0000","last_became_peered":"2024-08-03T21:00:49.217212+0000","last_unstale":"2024-08-03T21:00:49.220521+0000","last_undegraded":"2024-08-03T21:00:49.220521+0000","last_fullsized":"2024-08-03T21:00:49.220521+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:48.192422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:48.192422+0000","last_clean_scrub_stamp":"2024-08-03T21:00:48.192422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-04T22:22:37.333147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.1b","version":"0'0","reported_seq":1,"reported_epoch":133,"state":"creating+peering","last_fresh":"2024-08-03T21:00:49.215473+0000","last_change":"2024-08-03T21:00:49.215473+0000","last_active":"2024-08-03T21:00:49.195479+0000","last_peered":"2024-08-03T21:00:49.195479+0000","last_clean":"2024-08-03T21:00:49.195479+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-08-03T21:00:49.215473+0000","last_undegraded":"2024-08-03T21:00:49.215473+0000","last_fullsized":"2024-08-03T21:00:49.215473+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_clean_scrub_stamp":"2024-08-03T21:00:49.195479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.1a","version":"0'0","reported_seq":10,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-08-03T21:00:49.239275+0000","last_change":"2024-08-03T21:00:49.238475+0000","last_active":"2024-08-03T21:00:49.239275+0000","last_peered":"2024-08-03T21:00:49.239275+0000","last_clean":"2024-08-03T21:00:49.239275+0000","last_became_active":"2024-08-03T21:00:49.238001+0000","last_became_peered":"2024-08-03T21:00:49.238001+0000","last_unstale":"2024-08-03T21:00:49.239275+0000","last_undegraded":"2024-08-03T21:00:49.239275+0000","last_fullsized":"2024-08-03T21:00:49.239275+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:48.192422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:48.192422+0000","last_clean_scrub_stamp":"2024-08-03T21:00:48.192422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T07:44:13.158341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.18","version":"0'0","reported_seq":13,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-08-03T21:00:51.225613+0000","last_change":"2024-08-03T21:00:50.228305+0000","last_active":"2024-08-03T21:00:51.225613+0000","last_peered":"2024-08-03T21:00:51.225613+0000","last_clean":"2024-08-03T21:00:51.225613+0000","last_became_active":"2024-08-03T21:00:50.228009+0000","last_became_peered":"2024-08-03T21:00:50.228009+0000","last_unstale":"2024-08-03T21:00:51.225613+0000","last_undegraded":"2024-08-03T21:00:51.225613+0000","last_fullsized":"2024-08-03T21:00:51.225613+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_clean_scrub_stamp":"2024-08-03T21:00:49.195479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-04T23:22:25.975528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.19","version":"0'0","reported_seq":10,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-08-03T21:00:49.229663+0000","last_change":"2024-08-03T21:00:49.228986+0000","last_active":"2024-08-03T21:00:49.229663+0000","last_peered":"2024-08-03T21:00:49.229663+0000","last_clean":"2024-08-03T21:00:49.229663+0000","last_became_active":"2024-08-03T21:00:49.228761+0000","last_became_peered":"2024-08-03T21:00:49.228761+0000","last_unstale":"2024-08-03T21:00:49.229663+0000","last_undegraded":"2024-08-03T21:00:49.229663+0000","last_fullsized":"2024-08-03T21:00:49.229663+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:48.192422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:48.192422+0000","last_clean_scrub_stamp":"2024-08-03T21:00:48.192422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T02:44:18.436142+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.19","version":"0'0","reported_seq":13,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-08-03T21:00:51.940174+0000","last_change":"2024-08-03T21:00:50.235335+0000","last_active":"2024-08-03T21:00:51.940174+0000","last_peered":"2024-08-03T21:00:51.940174+0000","last_clean":"2024-08-03T21:00:51.940174+0000","last_became_active":"2024-08-03T21:00:50.234940+0000","last_became_peered":"2024-08-03T21:00:50.234940+0000","last_unstale":"2024-08-03T21:00:51.940174+0000","last_undegraded":"2024-08-03T21:00:51.940174+0000","last_fullsized":"2024-08-03T21:00:51.940174+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_clean_scrub_stamp":"2024-08-03T21:00:49.195479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T00:40:44.403536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.18","version":"0'0","reported_seq":10,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-08-03T21:00:49.237874+0000","last_change":"2024-08-03T21:00:49.236558+0000","last_active":"2024-08-03T21:00:49.237874+0000","last_peered":"2024-08-03T21:00:49.237874+0000","last_clean":"2024-08-03T21:00:49.237874+0000","last_became_active":"2024-08-03T21:00:49.236201+0000","last_became_peered":"2024-08-03T21:00:49.236201+0000","last_unstale":"2024-08-03T21:00:49.237874+0000","last_undegraded":"2024-08-03T21:00:49.237874+0000","last_fullsized":"2024-08-03T21:00:49.237874+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:48.192422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:48.192422+0000","last_clean_scrub_stamp":"2024-08-03T21:00:48.192422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-04T23:01:53.386452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.16","version":"0'0","reported_seq":1,"reported_epoch":133,"state":"creating+peering","last_fresh":"2024-08-03T21:00:49.219922+0000","last_change":"2024-08-03T21:00:49.219922+0000","last_active":"2024-08-03T21:00:49.195479+0000","last_peered":"2024-08-03T21:00:49.195479+0000","last_clean":"2024-08-03T21:00:49.195479+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-08-03T21:00:49.219922+0000","last_undegraded":"2024-08-03T21:00:49.219922+0000","last_fullsized":"2024-08-03T21:00:49.219922+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_clean_scrub_stamp":"2024-08-03T21:00:49.195479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.17","version":"0'0","reported_seq":10,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-08-03T21:00:49.229166+0000","last_change":"2024-08-03T21:00:49.226248+0000","last_active":"2024-08-03T21:00:49.229166+0000","last_peered":"2024-08-03T21:00:49.229166+0000","last_clean":"2024-08-03T21:00:49.229166+0000","last_became_active":"2024-08-03T21:00:49.224012+0000","last_became_peered":"2024-08-03T21:00:49.224012+0000","last_unstale":"2024-08-03T21:00:49.229166+0000","last_undegraded":"2024-08-03T21:00:49.229166+0000","last_fullsized":"2024-08-03T21:00:49.229166+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:48.192422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:48.192422+0000","last_clean_scrub_stamp":"2024-08-03T21:00:48.192422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T02:58:35.310272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.17","version":"0'0","reported_seq":13,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-08-03T21:00:51.940036+0000","last_change":"2024-08-03T21:00:50.244964+0000","last_active":"2024-08-03T21:00:51.940036+0000","last_peered":"2024-08-03T21:00:51.940036+0000","last_clean":"2024-08-03T21:00:51.940036+0000","last_became_active":"2024-08-03T21:00:50.244323+0000","last_became_peered":"2024-08-03T21:00:50.244323+0000","last_unstale":"2024-08-03T21:00:51.940036+0000","last_undegraded":"2024-08-03T21:00:51.940036+0000","last_fullsized":"2024-08-03T21:00:51.940036+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_clean_scrub_stamp":"2024-08-03T21:00:49.195479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-04T21:46:43.719103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.16","version":"0'0","reported_seq":15,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-08-03T21:00:51.225728+0000","last_change":"2024-08-03T21:00:49.233357+0000","last_active":"2024-08-03T21:00:51.225728+0000","last_peered":"2024-08-03T21:00:51.225728+0000","last_clean":"2024-08-03T21:00:51.225728+0000","last_became_active":"2024-08-03T21:00:49.232477+0000","last_became_peered":"2024-08-03T21:00:49.232477+0000","last_unstale":"2024-08-03T21:00:51.225728+0000","last_undegraded":"2024-08-03T21:00:51.225728+0000","last_fullsized":"2024-08-03T21:00:51.225728+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:48.192422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:48.192422+0000","last_clean_scrub_stamp":"2024-08-03T21:00:48.192422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T05:40:56.485784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.14","version":"0'0","reported_seq":1,"reported_epoch":133,"state":"creating+peering","last_fresh":"2024-08-03T21:00:49.210789+0000","last_change":"2024-08-03T21:00:49.210789+0000","last_active":"2024-08-03T21:00:49.195479+0000","last_peered":"2024-08-03T21:00:49.195479+0000","last_clean":"2024-08-03T21:00:49.195479+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-08-03T21:00:49.210789+0000","last_undegraded":"2024-08-03T21:00:49.210789+0000","last_fullsized":"2024-08-03T21:00:49.210789+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_clean_scrub_stamp":"2024-08-03T21:00:49.195479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.15","version":"0'0","reported_seq":10,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-08-03T21:00:49.225338+0000","last_change":"2024-08-03T21:00:49.224739+0000","last_active":"2024-08-03T21:00:49.225338+0000","last_peered":"2024-08-03T21:00:49.225338+0000","last_clean":"2024-08-03T21:00:49.225338+0000","last_became_active":"2024-08-03T21:00:49.224342+0000","last_became_peered":"2024-08-03T21:00:49.224342+0000","last_unstale":"2024-08-03T21:00:49.225338+0000","last_undegraded":"2024-08-03T21:00:49.225338+0000","last_fullsized":"2024-08-03T21:00:49.225338+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:48.192422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:48.192422+0000","last_clean_scrub_stamp":"2024-08-03T21:00:48.192422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-04T23:53:41.240790+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.15","version":"0'0","reported_seq":1,"reported_epoch":133,"state":"creating+peering","last_fresh":"2024-08-03T21:00:49.216465+0000","last_change":"2024-08-03T21:00:49.216465+0000","last_active":"2024-08-03T21:00:49.195479+0000","last_peered":"2024-08-03T21:00:49.195479+0000","last_clean":"2024-08-03T21:00:49.195479+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-08-03T21:00:49.216465+0000","last_undegraded":"2024-08-03T21:00:49.216465+0000","last_fullsized":"2024-08-03T21:00:49.216465+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_clean_scrub_stamp":"2024-08-03T21:00:49.195479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.14","version":"134'6","reported_seq":21,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-08-03T21:00:51.939402+0000","last_change":"2024-08-03T21:00:49.235410+0000","last_active":"2024-08-03T21:00:51.939402+0000","last_peered":"2024-08-03T21:00:51.939402+0000","last_clean":"2024-08-03T21:00:51.939402+0000","last_became_active":"2024-08-03T21:00:49.234999+0000","last_became_peered":"2024-08-03T21:00:49.234999+0000","last_unstale":"2024-08-03T21:00:51.939402+0000","last_undegraded":"2024-08-03T21:00:51.939402+0000","last_fullsized":"2024-08-03T21:00:51.939402+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:48.192422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:48.192422+0000","last_clean_scrub_stamp":"2024-08-03T21:00:48.192422+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T02:55:58.470527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.12","version":"0'0","reported_seq":13,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-08-03T21:00:51.225979+0000","last_change":"2024-08-03T21:00:50.218292+0000","last_active":"2024-08-03T21:00:51.225979+0000","last_peered":"2024-08-03T21:00:51.225979+0000","last_clean":"2024-08-03T21:00:51.225979+0000","last_became_active":"2024-08-03T21:00:50.217979+0000","last_became_peered":"2024-08-03T21:00:50.217979+0000","last_unstale":"2024-08-03T21:00:51.225979+0000","last_undegraded":"2024-08-03T21:00:51.225979+0000","last_fullsized":"2024-08-03T21:00:51.225979+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_clean_scrub_stamp":"2024-08-03T21:00:49.195479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T06:55:48.820648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.13","version":"134'2","reported_seq":17,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-08-03T21:00:51.939709+0000","last_change":"2024-08-03T21:00:49.224867+0000","last_active":"2024-08-03T21:00:51.939709+0000","last_peered":"2024-08-03T21:00:51.939709+0000","last_clean":"2024-08-03T21:00:51.939709+0000","last_became_active":"2024-08-03T21:00:49.224555+0000","last_became_peered":"2024-08-03T21:00:49.224555+0000","last_unstale":"2024-08-03T21:00:51.939709+0000","last_undegraded":"2024-08-03T21:00:51.939709+0000","last_fullsized":"2024-08-03T21:00:51.939709+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:48.192422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:48.192422+0000","last_clean_scrub_stamp":"2024-08-03T21:00:48.192422+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T07:55:25.225889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.13","version":"0'0","reported_seq":1,"reported_epoch":133,"state":"creating+peering","last_fresh":"2024-08-03T21:00:49.211586+0000","last_change":"2024-08-03T21:00:49.211586+0000","last_active":"2024-08-03T21:00:49.195479+0000","last_peered":"2024-08-03T21:00:49.195479+0000","last_clean":"2024-08-03T21:00:49.195479+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-08-03T21:00:49.211586+0000","last_undegraded":"2024-08-03T21:00:49.211586+0000","last_fullsized":"2024-08-03T21:00:49.211586+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_clean_scrub_stamp":"2024-08-03T21:00:49.195479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.12","version":"0'0","reported_seq":15,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-08-03T21:00:51.938928+0000","last_change":"2024-08-03T21:00:49.224078+0000","last_active":"2024-08-03T21:00:51.938928+0000","last_peered":"2024-08-03T21:00:51.938928+0000","last_clean":"2024-08-03T21:00:51.938928+0000","last_became_active":"2024-08-03T21:00:49.223717+0000","last_became_peered":"2024-08-03T21:00:49.223717+0000","last_unstale":"2024-08-03T21:00:51.938928+0000","last_undegraded":"2024-08-03T21:00:51.938928+0000","last_fullsized":"2024-08-03T21:00:51.938928+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:48.192422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:48.192422+0000","last_clean_scrub_stamp":"2024-08-03T21:00:48.192422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T04:56:22.864880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.10","version":"0'0","reported_seq":13,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-08-03T21:00:51.939036+0000","last_change":"2024-08-03T21:00:50.231767+0000","last_active":"2024-08-03T21:00:51.939036+0000","last_peered":"2024-08-03T21:00:51.939036+0000","last_clean":"2024-08-03T21:00:51.939036+0000","last_became_active":"2024-08-03T21:00:50.229726+0000","last_became_peered":"2024-08-03T21:00:50.229726+0000","last_unstale":"2024-08-03T21:00:51.939036+0000","last_undegraded":"2024-08-03T21:00:51.939036+0000","last_fullsized":"2024-08-03T21:00:51.939036+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_clean_scrub_stamp":"2024-08-03T21:00:49.195479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T00:14:18.409160+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.11","version":"0'0","reported_seq":15,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-08-03T21:00:51.939969+0000","last_change":"2024-08-03T21:00:49.219205+0000","last_active":"2024-08-03T21:00:51.939969+0000","last_peered":"2024-08-03T21:00:51.939969+0000","last_clean":"2024-08-03T21:00:51.939969+0000","last_became_active":"2024-08-03T21:00:49.218855+0000","last_became_peered":"2024-08-03T21:00:49.218855+0000","last_unstale":"2024-08-03T21:00:51.939969+0000","last_undegraded":"2024-08-03T21:00:51.939969+0000","last_fullsized":"2024-08-03T21:00:51.939969+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:48.192422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:48.192422+0000","last_clean_scrub_stamp":"2024-08-03T21:00:48.192422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T00:24:17.141394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.11","version":"0'0","reported_seq":13,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-08-03T21:00:51.939150+0000","last_change":"2024-08-03T21:00:50.230042+0000","last_active":"2024-08-03T21:00:51.939150+0000","last_peered":"2024-08-03T21:00:51.939150+0000","last_clean":"2024-08-03T21:00:51.939150+0000","last_became_active":"2024-08-03T21:00:50.229716+0000","last_became_peered":"2024-08-03T21:00:50.229716+0000","last_unstale":"2024-08-03T21:00:51.939150+0000","last_undegraded":"2024-08-03T21:00:51.939150+0000","last_fullsized":"2024-08-03T21:00:51.939150+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_clean_scrub_stamp":"2024-08-03T21:00:49.195479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T08:05:48.492097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.10","version":"0'0","reported_seq":10,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-08-03T21:00:49.222553+0000","last_change":"2024-08-03T21:00:49.219610+0000","last_active":"2024-08-03T21:00:49.222553+0000","last_peered":"2024-08-03T21:00:49.222553+0000","last_clean":"2024-08-03T21:00:49.222553+0000","last_became_active":"2024-08-03T21:00:49.219289+0000","last_became_peered":"2024-08-03T21:00:49.219289+0000","last_unstale":"2024-08-03T21:00:49.222553+0000","last_undegraded":"2024-08-03T21:00:49.222553+0000","last_fullsized":"2024-08-03T21:00:49.222553+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:48.192422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:48.192422+0000","last_clean_scrub_stamp":"2024-08-03T21:00:48.192422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-04T22:10:58.211954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.f","version":"0'0","reported_seq":1,"reported_epoch":133,"state":"creating+peering","last_fresh":"2024-08-03T21:00:49.209245+0000","last_change":"2024-08-03T21:00:49.209245+0000","last_active":"2024-08-03T21:00:49.195479+0000","last_peered":"2024-08-03T21:00:49.195479+0000","last_clean":"2024-08-03T21:00:49.195479+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-08-03T21:00:49.209245+0000","last_undegraded":"2024-08-03T21:00:49.209245+0000","last_fullsized":"2024-08-03T21:00:49.209245+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_clean_scrub_stamp":"2024-08-03T21:00:49.195479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.e","version":"134'2","reported_seq":17,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-08-03T21:00:51.939110+0000","last_change":"2024-08-03T21:00:49.221709+0000","last_active":"2024-08-03T21:00:51.939110+0000","last_peered":"2024-08-03T21:00:51.939110+0000","last_clean":"2024-08-03T21:00:51.939110+0000","last_became_active":"2024-08-03T21:00:49.221146+0000","last_became_peered":"2024-08-03T21:00:49.221146+0000","last_unstale":"2024-08-03T21:00:51.939110+0000","last_undegraded":"2024-08-03T21:00:51.939110+0000","last_fullsized":"2024-08-03T21:00:51.939110+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:48.192422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:48.192422+0000","last_clean_scrub_stamp":"2024-08-03T21:00:48.192422+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T02:29:53.585049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.c","version":"0'0","reported_seq":1,"reported_epoch":133,"state":"creating+peering","last_fresh":"2024-08-03T21:00:49.211956+0000","last_change":"2024-08-03T21:00:49.211956+0000","last_active":"2024-08-03T21:00:49.195479+0000","last_peered":"2024-08-03T21:00:49.195479+0000","last_clean":"2024-08-03T21:00:49.195479+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-08-03T21:00:49.211956+0000","last_undegraded":"2024-08-03T21:00:49.211956+0000","last_fullsized":"2024-08-03T21:00:49.211956+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_clean_scrub_stamp":"2024-08-03T21:00:49.195479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.d","version":"0'0","reported_seq":10,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-08-03T21:00:49.216969+0000","last_change":"2024-08-03T21:00:49.214725+0000","last_active":"2024-08-03T21:00:49.216969+0000","last_peered":"2024-08-03T21:00:49.216969+0000","last_clean":"2024-08-03T21:00:49.216969+0000","last_became_active":"2024-08-03T21:00:49.214155+0000","last_became_peered":"2024-08-03T21:00:49.214155+0000","last_unstale":"2024-08-03T21:00:49.216969+0000","last_undegraded":"2024-08-03T21:00:49.216969+0000","last_fullsized":"2024-08-03T21:00:49.216969+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:48.192422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:48.192422+0000","last_clean_scrub_stamp":"2024-08-03T21:00:48.192422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T07:43:13.286648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.d","version":"0'0","reported_seq":1,"reported_epoch":133,"state":"creating+peering","last_fresh":"2024-08-03T21:00:49.213316+0000","last_change":"2024-08-03T21:00:49.213316+0000","last_active":"2024-08-03T21:00:49.195479+0000","last_peered":"2024-08-03T21:00:49.195479+0000","last_clean":"2024-08-03T21:00:49.195479+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-08-03T21:00:49.213316+0000","last_undegraded":"2024-08-03T21:00:49.213316+0000","last_fullsized":"2024-08-03T21:00:49.213316+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_clean_scrub_stamp":"2024-08-03T21:00:49.195479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.c","version":"0'0","reported_seq":10,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-08-03T21:00:49.230569+0000","last_change":"2024-08-03T21:00:49.225810+0000","last_active":"2024-08-03T21:00:49.230569+0000","last_peered":"2024-08-03T21:00:49.230569+0000","last_clean":"2024-08-03T21:00:49.230569+0000","last_became_active":"2024-08-03T21:00:49.225501+0000","last_became_peered":"2024-08-03T21:00:49.225501+0000","last_unstale":"2024-08-03T21:00:49.230569+0000","last_undegraded":"2024-08-03T21:00:49.230569+0000","last_fullsized":"2024-08-03T21:00:49.230569+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:48.192422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:48.192422+0000","last_clean_scrub_stamp":"2024-08-03T21:00:48.192422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-04T23:49:05.539690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.a","version":"0'0","reported_seq":13,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-08-03T21:00:51.938230+0000","last_change":"2024-08-03T21:00:50.238896+0000","last_active":"2024-08-03T21:00:51.938230+0000","last_peered":"2024-08-03T21:00:51.938230+0000","last_clean":"2024-08-03T21:00:51.938230+0000","last_became_active":"2024-08-03T21:00:50.238557+0000","last_became_peered":"2024-08-03T21:00:50.238557+0000","last_unstale":"2024-08-03T21:00:51.938230+0000","last_undegraded":"2024-08-03T21:00:51.938230+0000","last_fullsized":"2024-08-03T21:00:51.938230+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_clean_scrub_stamp":"2024-08-03T21:00:49.195479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-04T22:10:38.257913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.b","version":"0'0","reported_seq":10,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-08-03T21:00:49.238315+0000","last_change":"2024-08-03T21:00:49.233291+0000","last_active":"2024-08-03T21:00:49.238315+0000","last_peered":"2024-08-03T21:00:49.238315+0000","last_clean":"2024-08-03T21:00:49.238315+0000","last_became_active":"2024-08-03T21:00:49.232973+0000","last_became_peered":"2024-08-03T21:00:49.232973+0000","last_unstale":"2024-08-03T21:00:49.238315+0000","last_undegraded":"2024-08-03T21:00:49.238315+0000","last_fullsized":"2024-08-03T21:00:49.238315+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:48.192422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:48.192422+0000","last_clean_scrub_stamp":"2024-08-03T21:00:48.192422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T02:44:18.436142+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.b","version":"0'0","reported_seq":13,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-08-03T21:00:51.939647+0000","last_change":"2024-08-03T21:00:50.242588+0000","last_active":"2024-08-03T21:00:51.939647+0000","last_peered":"2024-08-03T21:00:51.939647+0000","last_clean":"2024-08-03T21:00:51.939647+0000","last_became_active":"2024-08-03T21:00:50.239795+0000","last_became_peered":"2024-08-03T21:00:50.239795+0000","last_unstale":"2024-08-03T21:00:51.939647+0000","last_undegraded":"2024-08-03T21:00:51.939647+0000","last_fullsized":"2024-08-03T21:00:51.939647+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_clean_scrub_stamp":"2024-08-03T21:00:49.195479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T06:53:11.234357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.a","version":"0'0","reported_seq":10,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-08-03T21:00:49.223212+0000","last_change":"2024-08-03T21:00:49.222066+0000","last_active":"2024-08-03T21:00:49.223212+0000","last_peered":"2024-08-03T21:00:49.223212+0000","last_clean":"2024-08-03T21:00:49.223212+0000","last_became_active":"2024-08-03T21:00:49.221757+0000","last_became_peered":"2024-08-03T21:00:49.221757+0000","last_unstale":"2024-08-03T21:00:49.223212+0000","last_undegraded":"2024-08-03T21:00:49.223212+0000","last_fullsized":"2024-08-03T21:00:49.223212+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:48.192422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:48.192422+0000","last_clean_scrub_stamp":"2024-08-03T21:00:48.192422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T08:47:16.511690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.8","version":"0'0","reported_seq":13,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-08-03T21:00:51.937406+0000","last_change":"2024-08-03T21:00:50.240625+0000","last_active":"2024-08-03T21:00:51.937406+0000","last_peered":"2024-08-03T21:00:51.937406+0000","last_clean":"2024-08-03T21:00:51.937406+0000","last_became_active":"2024-08-03T21:00:50.240139+0000","last_became_peered":"2024-08-03T21:00:50.240139+0000","last_unstale":"2024-08-03T21:00:51.937406+0000","last_undegraded":"2024-08-03T21:00:51.937406+0000","last_fullsized":"2024-08-03T21:00:51.937406+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_clean_scrub_stamp":"2024-08-03T21:00:49.195479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T06:58:48.625406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.9","version":"0'0","reported_seq":10,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-08-03T21:00:49.236203+0000","last_change":"2024-08-03T21:00:49.235212+0000","last_active":"2024-08-03T21:00:49.236203+0000","last_peered":"2024-08-03T21:00:49.236203+0000","last_clean":"2024-08-03T21:00:49.236203+0000","last_became_active":"2024-08-03T21:00:49.234793+0000","last_became_peered":"2024-08-03T21:00:49.234793+0000","last_unstale":"2024-08-03T21:00:49.236203+0000","last_undegraded":"2024-08-03T21:00:49.236203+0000","last_fullsized":"2024-08-03T21:00:49.236203+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:48.192422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:48.192422+0000","last_clean_scrub_stamp":"2024-08-03T21:00:48.192422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T07:43:13.286648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.9","version":"0'0","reported_seq":1,"reported_epoch":133,"state":"creating+peering","last_fresh":"2024-08-03T21:00:49.213171+0000","last_change":"2024-08-03T21:00:49.213171+0000","last_active":"2024-08-03T21:00:49.195479+0000","last_peered":"2024-08-03T21:00:49.195479+0000","last_clean":"2024-08-03T21:00:49.195479+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-08-03T21:00:49.213171+0000","last_undegraded":"2024-08-03T21:00:49.213171+0000","last_fullsized":"2024-08-03T21:00:49.213171+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_clean_scrub_stamp":"2024-08-03T21:00:49.195479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.8","version":"0'0","reported_seq":10,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-08-03T21:00:49.234238+0000","last_change":"2024-08-03T21:00:49.230326+0000","last_active":"2024-08-03T21:00:49.234238+0000","last_peered":"2024-08-03T21:00:49.234238+0000","last_clean":"2024-08-03T21:00:49.234238+0000","last_became_active":"2024-08-03T21:00:49.229867+0000","last_became_peered":"2024-08-03T21:00:49.229867+0000","last_unstale":"2024-08-03T21:00:49.234238+0000","last_undegraded":"2024-08-03T21:00:49.234238+0000","last_fullsized":"2024-08-03T21:00:49.234238+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:48.192422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:48.192422+0000","last_clean_scrub_stamp":"2024-08-03T21:00:48.192422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T05:44:38.082933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.6","version":"0'0","reported_seq":1,"reported_epoch":133,"state":"creating+peering","last_fresh":"2024-08-03T21:00:49.217620+0000","last_change":"2024-08-03T21:00:49.217620+0000","last_active":"2024-08-03T21:00:49.195479+0000","last_peered":"2024-08-03T21:00:49.195479+0000","last_clean":"2024-08-03T21:00:49.195479+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-08-03T21:00:49.217620+0000","last_undegraded":"2024-08-03T21:00:49.217620+0000","last_fullsized":"2024-08-03T21:00:49.217620+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_clean_scrub_stamp":"2024-08-03T21:00:49.195479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.7","version":"0'0","reported_seq":10,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-08-03T21:00:49.223032+0000","last_change":"2024-08-03T21:00:49.220966+0000","last_active":"2024-08-03T21:00:49.223032+0000","last_peered":"2024-08-03T21:00:49.223032+0000","last_clean":"2024-08-03T21:00:49.223032+0000","last_became_active":"2024-08-03T21:00:49.220433+0000","last_became_peered":"2024-08-03T21:00:49.220433+0000","last_unstale":"2024-08-03T21:00:49.223032+0000","last_undegraded":"2024-08-03T21:00:49.223032+0000","last_fullsized":"2024-08-03T21:00:49.223032+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:48.192422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:48.192422+0000","last_clean_scrub_stamp":"2024-08-03T21:00:48.192422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-04T23:49:05.539690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.7","version":"0'0","reported_seq":1,"reported_epoch":133,"state":"creating+peering","last_fresh":"2024-08-03T21:00:49.212025+0000","last_change":"2024-08-03T21:00:49.212025+0000","last_active":"2024-08-03T21:00:49.195479+0000","last_peered":"2024-08-03T21:00:49.195479+0000","last_clean":"2024-08-03T21:00:49.195479+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-08-03T21:00:49.212025+0000","last_undegraded":"2024-08-03T21:00:49.212025+0000","last_fullsized":"2024-08-03T21:00:49.212025+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_clean_scrub_stamp":"2024-08-03T21:00:49.195479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.6","version":"0'0","reported_seq":10,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-08-03T21:00:49.238693+0000","last_change":"2024-08-03T21:00:49.235126+0000","last_active":"2024-08-03T21:00:49.238693+0000","last_peered":"2024-08-03T21:00:49.238693+0000","last_clean":"2024-08-03T21:00:49.238693+0000","last_became_active":"2024-08-03T21:00:49.233960+0000","last_became_peered":"2024-08-03T21:00:49.233960+0000","last_unstale":"2024-08-03T21:00:49.238693+0000","last_undegraded":"2024-08-03T21:00:49.238693+0000","last_fullsized":"2024-08-03T21:00:49.238693+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:48.192422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:48.192422+0000","last_clean_scrub_stamp":"2024-08-03T21:00:48.192422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T07:27:35.968034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.4","version":"0'0","reported_seq":1,"reported_epoch":133,"state":"creating+peering","last_fresh":"2024-08-03T21:00:49.212551+0000","last_change":"2024-08-03T21:00:49.212551+0000","last_active":"2024-08-03T21:00:49.195479+0000","last_peered":"2024-08-03T21:00:49.195479+0000","last_clean":"2024-08-03T21:00:49.195479+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-08-03T21:00:49.212551+0000","last_undegraded":"2024-08-03T21:00:49.212551+0000","last_fullsized":"2024-08-03T21:00:49.212551+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_clean_scrub_stamp":"2024-08-03T21:00:49.195479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.5","version":"0'0","reported_seq":10,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-08-03T21:00:49.230593+0000","last_change":"2024-08-03T21:00:49.227879+0000","last_active":"2024-08-03T21:00:49.230593+0000","last_peered":"2024-08-03T21:00:49.230593+0000","last_clean":"2024-08-03T21:00:49.230593+0000","last_became_active":"2024-08-03T21:00:49.227574+0000","last_became_peered":"2024-08-03T21:00:49.227574+0000","last_unstale":"2024-08-03T21:00:49.230593+0000","last_undegraded":"2024-08-03T21:00:49.230593+0000","last_fullsized":"2024-08-03T21:00:49.230593+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:48.192422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:48.192422+0000","last_clean_scrub_stamp":"2024-08-03T21:00:48.192422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T07:43:13.286648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.5","version":"0'0","reported_seq":1,"reported_epoch":133,"state":"creating+peering","last_fresh":"2024-08-03T21:00:49.209221+0000","last_change":"2024-08-03T21:00:49.209221+0000","last_active":"2024-08-03T21:00:49.195479+0000","last_peered":"2024-08-03T21:00:49.195479+0000","last_clean":"2024-08-03T21:00:49.195479+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-08-03T21:00:49.209221+0000","last_undegraded":"2024-08-03T21:00:49.209221+0000","last_fullsized":"2024-08-03T21:00:49.209221+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_clean_scrub_stamp":"2024-08-03T21:00:49.195479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.4","version":"0'0","reported_seq":10,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-08-03T21:00:49.224181+0000","last_change":"2024-08-03T21:00:49.221928+0000","last_active":"2024-08-03T21:00:49.224181+0000","last_peered":"2024-08-03T21:00:49.224181+0000","last_clean":"2024-08-03T21:00:49.224181+0000","last_became_active":"2024-08-03T21:00:49.221623+0000","last_became_peered":"2024-08-03T21:00:49.221623+0000","last_unstale":"2024-08-03T21:00:49.224181+0000","last_undegraded":"2024-08-03T21:00:49.224181+0000","last_fullsized":"2024-08-03T21:00:49.224181+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:48.192422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:48.192422+0000","last_clean_scrub_stamp":"2024-08-03T21:00:48.192422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T01:29:44.046851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.2","version":"0'0","reported_seq":13,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-08-03T21:00:51.938470+0000","last_change":"2024-08-03T21:00:50.232514+0000","last_active":"2024-08-03T21:00:51.938470+0000","last_peered":"2024-08-03T21:00:51.938470+0000","last_clean":"2024-08-03T21:00:51.938470+0000","last_became_active":"2024-08-03T21:00:50.232148+0000","last_became_peered":"2024-08-03T21:00:50.232148+0000","last_unstale":"2024-08-03T21:00:51.938470+0000","last_undegraded":"2024-08-03T21:00:51.938470+0000","last_fullsized":"2024-08-03T21:00:51.938470+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_clean_scrub_stamp":"2024-08-03T21:00:49.195479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-04T23:42:52.232285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.3","version":"0'0","reported_seq":10,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-08-03T21:00:49.238736+0000","last_change":"2024-08-03T21:00:49.235356+0000","last_active":"2024-08-03T21:00:49.238736+0000","last_peered":"2024-08-03T21:00:49.238736+0000","last_clean":"2024-08-03T21:00:49.238736+0000","last_became_active":"2024-08-03T21:00:49.234801+0000","last_became_peered":"2024-08-03T21:00:49.234801+0000","last_unstale":"2024-08-03T21:00:49.238736+0000","last_undegraded":"2024-08-03T21:00:49.238736+0000","last_fullsized":"2024-08-03T21:00:49.238736+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:48.192422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:48.192422+0000","last_clean_scrub_stamp":"2024-08-03T21:00:48.192422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-04T23:53:41.240790+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.3","version":"0'0","reported_seq":13,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-08-03T21:00:51.939532+0000","last_change":"2024-08-03T21:00:50.233609+0000","last_active":"2024-08-03T21:00:51.939532+0000","last_peered":"2024-08-03T21:00:51.939532+0000","last_clean":"2024-08-03T21:00:51.939532+0000","last_became_active":"2024-08-03T21:00:50.231678+0000","last_became_peered":"2024-08-03T21:00:50.231678+0000","last_unstale":"2024-08-03T21:00:51.939532+0000","last_undegraded":"2024-08-03T21:00:51.939532+0000","last_fullsized":"2024-08-03T21:00:51.939532+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_clean_scrub_stamp":"2024-08-03T21:00:49.195479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T02:34:58.346689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.2","version":"0'0","reported_seq":10,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-08-03T21:00:49.219683+0000","last_change":"2024-08-03T21:00:49.218982+0000","last_active":"2024-08-03T21:00:49.219683+0000","last_peered":"2024-08-03T21:00:49.219683+0000","last_clean":"2024-08-03T21:00:49.219683+0000","last_became_active":"2024-08-03T21:00:49.217673+0000","last_became_peered":"2024-08-03T21:00:49.217673+0000","last_unstale":"2024-08-03T21:00:49.219683+0000","last_undegraded":"2024-08-03T21:00:49.219683+0000","last_fullsized":"2024-08-03T21:00:49.219683+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:48.192422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:48.192422+0000","last_clean_scrub_stamp":"2024-08-03T21:00:48.192422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T05:44:38.082933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.0","version":"0'0","reported_seq":13,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-08-03T21:00:51.938141+0000","last_change":"2024-08-03T21:00:50.234121+0000","last_active":"2024-08-03T21:00:51.938141+0000","last_peered":"2024-08-03T21:00:51.938141+0000","last_clean":"2024-08-03T21:00:51.938141+0000","last_became_active":"2024-08-03T21:00:50.233758+0000","last_became_peered":"2024-08-03T21:00:50.233758+0000","last_unstale":"2024-08-03T21:00:51.938141+0000","last_undegraded":"2024-08-03T21:00:51.938141+0000","last_fullsized":"2024-08-03T21:00:51.938141+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_clean_scrub_stamp":"2024-08-03T21:00:49.195479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T05:28:59.859843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.1","version":"0'0","reported_seq":15,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-08-03T21:00:51.225900+0000","last_change":"2024-08-03T21:00:49.230098+0000","last_active":"2024-08-03T21:00:51.225900+0000","last_peered":"2024-08-03T21:00:51.225900+0000","last_clean":"2024-08-03T21:00:51.225900+0000","last_became_active":"2024-08-03T21:00:49.229777+0000","last_became_peered":"2024-08-03T21:00:49.229777+0000","last_unstale":"2024-08-03T21:00:51.225900+0000","last_undegraded":"2024-08-03T21:00:51.225900+0000","last_fullsized":"2024-08-03T21:00:51.225900+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:48.192422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:48.192422+0000","last_clean_scrub_stamp":"2024-08-03T21:00:48.192422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T06:34:18.629210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.1","version":"0'0","reported_seq":1,"reported_epoch":133,"state":"creating+peering","last_fresh":"2024-08-03T21:00:49.210235+0000","last_change":"2024-08-03T21:00:49.210235+0000","last_active":"2024-08-03T21:00:49.195479+0000","last_peered":"2024-08-03T21:00:49.195479+0000","last_clean":"2024-08-03T21:00:49.195479+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-08-03T21:00:49.210235+0000","last_undegraded":"2024-08-03T21:00:49.210235+0000","last_fullsized":"2024-08-03T21:00:49.210235+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_clean_scrub_stamp":"2024-08-03T21:00:49.195479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.0","version":"0'0","reported_seq":15,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-08-03T21:00:51.938216+0000","last_change":"2024-08-03T21:00:49.231962+0000","last_active":"2024-08-03T21:00:51.938216+0000","last_peered":"2024-08-03T21:00:51.938216+0000","last_clean":"2024-08-03T21:00:51.938216+0000","last_became_active":"2024-08-03T21:00:49.231607+0000","last_became_peered":"2024-08-03T21:00:49.231607+0000","last_unstale":"2024-08-03T21:00:51.938216+0000","last_undegraded":"2024-08-03T21:00:51.938216+0000","last_fullsized":"2024-08-03T21:00:51.938216+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:48.192422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:48.192422+0000","last_clean_scrub_stamp":"2024-08-03T21:00:48.192422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T08:45:01.729460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.e","version":"0'0","reported_seq":13,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-08-03T21:00:51.939469+0000","last_change":"2024-08-03T21:00:50.249570+0000","last_active":"2024-08-03T21:00:51.939469+0000","last_peered":"2024-08-03T21:00:51.939469+0000","last_clean":"2024-08-03T21:00:51.939469+0000","last_became_active":"2024-08-03T21:00:50.240733+0000","last_became_peered":"2024-08-03T21:00:50.240733+0000","last_unstale":"2024-08-03T21:00:51.939469+0000","last_undegraded":"2024-08-03T21:00:51.939469+0000","last_fullsized":"2024-08-03T21:00:51.939469+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_clean_scrub_stamp":"2024-08-03T21:00:49.195479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T08:43:59.747967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.f","version":"0'0","reported_seq":10,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-08-03T21:00:49.235936+0000","last_change":"2024-08-03T21:00:49.234956+0000","last_active":"2024-08-03T21:00:49.235936+0000","last_peered":"2024-08-03T21:00:49.235936+0000","last_clean":"2024-08-03T21:00:49.235936+0000","last_became_active":"2024-08-03T21:00:49.234351+0000","last_became_peered":"2024-08-03T21:00:49.234351+0000","last_unstale":"2024-08-03T21:00:49.235936+0000","last_undegraded":"2024-08-03T21:00:49.235936+0000","last_fullsized":"2024-08-03T21:00:49.235936+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:48.192422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:48.192422+0000","last_clean_scrub_stamp":"2024-08-03T21:00:48.192422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-04T22:46:01.864033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"16'72","reported_seq":327,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-08-03T21:00:49.218344+0000","last_change":"2024-08-03T20:47:30.547632+0000","last_active":"2024-08-03T21:00:49.218344+0000","last_peered":"2024-08-03T21:00:49.218344+0000","last_clean":"2024-08-03T21:00:49.218344+0000","last_became_active":"2024-08-03T20:47:30.546255+0000","last_became_peered":"2024-08-03T20:47:30.546255+0000","last_unstale":"2024-08-03T21:00:49.218344+0000","last_undegraded":"2024-08-03T21:00:49.218344+0000","last_fullsized":"2024-08-03T21:00:49.218344+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:47:28.874451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:47:28.874451+0000","last_clean_scrub_stamp":"2024-08-03T20:47:28.874451+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T00:19:40.548812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":15,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":14,"num_pg":32,"stat_sum":{"num_bytes":1646,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":3404,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":10,"ondisk_log_size":10,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509603,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41732,"kb_used_data":2228,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330108,"statfs":{"total":96636764160,"available":96594030592,"internally_reserved":0,"allocated":2281472,"data_stored":1401923,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":40434122},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":6,"up_from":15,"seq":64424509604,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36988,"kb_used_data":1648,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334852,"statfs":{"total":96636764160,"available":96598888448,"internally_reserved":0,"allocated":1687552,"data_stored":811555,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":15,"seq":64424509604,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36988,"kb_used_data":1648,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334852,"statfs":{"total":96636764160,"available":96598888448,"internally_reserved":0,"allocated":1687552,"data_stored":811555,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509603,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41148,"kb_used_data":1648,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330692,"statfs":{"total":96636764160,"available":96594628608,"internally_reserved":0,"allocated":1687552,"data_stored":811555,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509605,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":29252,"kb_used_data":2228,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94342588,"statfs":{"total":96636764160,"available":96606810112,"internally_reserved":0,"allocated":2281472,"data_stored":1401923,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542309,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37048,"kb_used_data":1704,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334792,"statfs":{"total":96636764160,"available":96598827008,"internally_reserved":0,"allocated":1744896,"data_stored":832641,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":3,"up_from":14,"seq":60129542307,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37044,"kb_used_data":1700,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334796,"statfs":{"total":96636764160,"available":96598831104,"internally_reserved":0,"allocated":1740800,"data_stored":832539,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":2,"up_from":14,"seq":60129542308,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41192,"kb_used_data":1688,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330648,"statfs":{"total":96636764160,"available":96594583552,"internally_reserved":0,"allocated":1728512,"data_stored":830939,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-08-03T21:00:55.780 INFO:tasks.ceph.ceph_manager.ceph:PG 15.3f is not active+clean 2024-08-03T21:00:55.781 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '15.3f', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 133, 'state': 'creating+peering', 'last_fresh': '2024-08-03T21:00:49.214906+0000', 'last_change': '2024-08-03T21:00:49.214906+0000', 'last_active': '2024-08-03T21:00:49.195479+0000', 'last_peered': '2024-08-03T21:00:49.195479+0000', 'last_clean': '2024-08-03T21:00:49.195479+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-08-03T21:00:49.214906+0000', 'last_undegraded': '2024-08-03T21:00:49.214906+0000', 'last_fullsized': '2024-08-03T21:00:49.214906+0000', 'mapping_epoch': 133, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 133, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-08-03T21:00:49.195479+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-08-03T21:00:49.195479+0000', 'last_clean_scrub_stamp': '2024-08-03T21:00:49.195479+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 5], 'acting': [1, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-08-03T21:00:55.781 INFO:tasks.ceph.ceph_manager.ceph:PG 15.3e is not active+clean 2024-08-03T21:00:55.781 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '15.3e', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 133, 'state': 'creating+peering', 'last_fresh': '2024-08-03T21:00:49.221252+0000', 'last_change': '2024-08-03T21:00:49.221252+0000', 'last_active': '2024-08-03T21:00:49.195479+0000', 'last_peered': '2024-08-03T21:00:49.195479+0000', 'last_clean': '2024-08-03T21:00:49.195479+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-08-03T21:00:49.221252+0000', 'last_undegraded': '2024-08-03T21:00:49.221252+0000', 'last_fullsized': '2024-08-03T21:00:49.221252+0000', 'mapping_epoch': 133, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 133, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-08-03T21:00:49.195479+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-08-03T21:00:49.195479+0000', 'last_clean_scrub_stamp': '2024-08-03T21:00:49.195479+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 5], 'acting': [0, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-08-03T21:00:55.781 INFO:tasks.ceph.ceph_manager.ceph:PG 15.3d is not active+clean 2024-08-03T21:00:55.781 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '15.3d', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 133, 'state': 'creating+peering', 'last_fresh': '2024-08-03T21:00:49.214132+0000', 'last_change': '2024-08-03T21:00:49.214132+0000', 'last_active': '2024-08-03T21:00:49.195479+0000', 'last_peered': '2024-08-03T21:00:49.195479+0000', 'last_clean': '2024-08-03T21:00:49.195479+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-08-03T21:00:49.214132+0000', 'last_undegraded': '2024-08-03T21:00:49.214132+0000', 'last_fullsized': '2024-08-03T21:00:49.214132+0000', 'mapping_epoch': 133, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 133, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-08-03T21:00:49.195479+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-08-03T21:00:49.195479+0000', 'last_clean_scrub_stamp': '2024-08-03T21:00:49.195479+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 5], 'acting': [1, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-08-03T21:00:55.781 INFO:tasks.ceph.ceph_manager.ceph:PG 15.3b is not active+clean 2024-08-03T21:00:55.781 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '15.3b', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 133, 'state': 'creating+peering', 'last_fresh': '2024-08-03T21:00:49.212243+0000', 'last_change': '2024-08-03T21:00:49.212243+0000', 'last_active': '2024-08-03T21:00:49.195479+0000', 'last_peered': '2024-08-03T21:00:49.195479+0000', 'last_clean': '2024-08-03T21:00:49.195479+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-08-03T21:00:49.212243+0000', 'last_undegraded': '2024-08-03T21:00:49.212243+0000', 'last_fullsized': '2024-08-03T21:00:49.212243+0000', 'mapping_epoch': 133, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 133, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-08-03T21:00:49.195479+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-08-03T21:00:49.195479+0000', 'last_clean_scrub_stamp': '2024-08-03T21:00:49.195479+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 3], 'acting': [6, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-08-03T21:00:55.781 INFO:tasks.ceph.ceph_manager.ceph:PG 15.3a is not active+clean 2024-08-03T21:00:55.781 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '15.3a', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 133, 'state': 'creating+peering', 'last_fresh': '2024-08-03T21:00:49.215377+0000', 'last_change': '2024-08-03T21:00:49.215377+0000', 'last_active': '2024-08-03T21:00:49.195479+0000', 'last_peered': '2024-08-03T21:00:49.195479+0000', 'last_clean': '2024-08-03T21:00:49.195479+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-08-03T21:00:49.215377+0000', 'last_undegraded': '2024-08-03T21:00:49.215377+0000', 'last_fullsized': '2024-08-03T21:00:49.215377+0000', 'mapping_epoch': 133, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 133, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-08-03T21:00:49.195479+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-08-03T21:00:49.195479+0000', 'last_clean_scrub_stamp': '2024-08-03T21:00:49.195479+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 5], 'acting': [7, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-08-03T21:00:55.782 INFO:tasks.ceph.ceph_manager.ceph:PG 15.39 is not active+clean 2024-08-03T21:00:55.782 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '15.39', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 133, 'state': 'creating+peering', 'last_fresh': '2024-08-03T21:00:49.213231+0000', 'last_change': '2024-08-03T21:00:49.213231+0000', 'last_active': '2024-08-03T21:00:49.195479+0000', 'last_peered': '2024-08-03T21:00:49.195479+0000', 'last_clean': '2024-08-03T21:00:49.195479+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-08-03T21:00:49.213231+0000', 'last_undegraded': '2024-08-03T21:00:49.213231+0000', 'last_fullsized': '2024-08-03T21:00:49.213231+0000', 'mapping_epoch': 133, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 133, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-08-03T21:00:49.195479+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-08-03T21:00:49.195479+0000', 'last_clean_scrub_stamp': '2024-08-03T21:00:49.195479+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 5], 'acting': [7, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-08-03T21:00:55.782 INFO:tasks.ceph.ceph_manager.ceph:PG 15.38 is not active+clean 2024-08-03T21:00:55.782 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '15.38', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 133, 'state': 'creating+peering', 'last_fresh': '2024-08-03T21:00:49.214019+0000', 'last_change': '2024-08-03T21:00:49.214019+0000', 'last_active': '2024-08-03T21:00:49.195479+0000', 'last_peered': '2024-08-03T21:00:49.195479+0000', 'last_clean': '2024-08-03T21:00:49.195479+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-08-03T21:00:49.214019+0000', 'last_undegraded': '2024-08-03T21:00:49.214019+0000', 'last_fullsized': '2024-08-03T21:00:49.214019+0000', 'mapping_epoch': 133, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 133, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-08-03T21:00:49.195479+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-08-03T21:00:49.195479+0000', 'last_clean_scrub_stamp': '2024-08-03T21:00:49.195479+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 7], 'acting': [0, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-08-03T21:00:55.782 INFO:tasks.ceph.ceph_manager.ceph:PG 15.36 is not active+clean 2024-08-03T21:00:55.782 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '15.36', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 133, 'state': 'creating+peering', 'last_fresh': '2024-08-03T21:00:49.211444+0000', 'last_change': '2024-08-03T21:00:49.211444+0000', 'last_active': '2024-08-03T21:00:49.195479+0000', 'last_peered': '2024-08-03T21:00:49.195479+0000', 'last_clean': '2024-08-03T21:00:49.195479+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-08-03T21:00:49.211444+0000', 'last_undegraded': '2024-08-03T21:00:49.211444+0000', 'last_fullsized': '2024-08-03T21:00:49.211444+0000', 'mapping_epoch': 133, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 133, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-08-03T21:00:49.195479+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-08-03T21:00:49.195479+0000', 'last_clean_scrub_stamp': '2024-08-03T21:00:49.195479+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 1], 'acting': [6, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-08-03T21:00:55.782 INFO:tasks.ceph.ceph_manager.ceph:PG 15.35 is not active+clean 2024-08-03T21:00:55.782 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '15.35', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 133, 'state': 'creating+peering', 'last_fresh': '2024-08-03T21:00:49.217444+0000', 'last_change': '2024-08-03T21:00:49.217444+0000', 'last_active': '2024-08-03T21:00:49.195479+0000', 'last_peered': '2024-08-03T21:00:49.195479+0000', 'last_clean': '2024-08-03T21:00:49.195479+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-08-03T21:00:49.217444+0000', 'last_undegraded': '2024-08-03T21:00:49.217444+0000', 'last_fullsized': '2024-08-03T21:00:49.217444+0000', 'mapping_epoch': 133, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 133, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-08-03T21:00:49.195479+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-08-03T21:00:49.195479+0000', 'last_clean_scrub_stamp': '2024-08-03T21:00:49.195479+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 4], 'acting': [7, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-08-03T21:00:55.782 INFO:tasks.ceph.ceph_manager.ceph:PG 15.30 is not active+clean 2024-08-03T21:00:55.782 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '15.30', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 133, 'state': 'creating+peering', 'last_fresh': '2024-08-03T21:00:49.214934+0000', 'last_change': '2024-08-03T21:00:49.214934+0000', 'last_active': '2024-08-03T21:00:49.195479+0000', 'last_peered': '2024-08-03T21:00:49.195479+0000', 'last_clean': '2024-08-03T21:00:49.195479+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-08-03T21:00:49.214934+0000', 'last_undegraded': '2024-08-03T21:00:49.214934+0000', 'last_fullsized': '2024-08-03T21:00:49.214934+0000', 'mapping_epoch': 133, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 133, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-08-03T21:00:49.195479+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-08-03T21:00:49.195479+0000', 'last_clean_scrub_stamp': '2024-08-03T21:00:49.195479+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 0], 'acting': [7, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-08-03T21:00:55.782 INFO:tasks.ceph.ceph_manager.ceph:PG 15.2f is not active+clean 2024-08-03T21:00:55.783 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '15.2f', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 133, 'state': 'creating+peering', 'last_fresh': '2024-08-03T21:00:49.220256+0000', 'last_change': '2024-08-03T21:00:49.220256+0000', 'last_active': '2024-08-03T21:00:49.195479+0000', 'last_peered': '2024-08-03T21:00:49.195479+0000', 'last_clean': '2024-08-03T21:00:49.195479+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-08-03T21:00:49.220256+0000', 'last_undegraded': '2024-08-03T21:00:49.220256+0000', 'last_fullsized': '2024-08-03T21:00:49.220256+0000', 'mapping_epoch': 133, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 133, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-08-03T21:00:49.195479+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-08-03T21:00:49.195479+0000', 'last_clean_scrub_stamp': '2024-08-03T21:00:49.195479+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3], 'acting': [0, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-08-03T21:00:55.783 INFO:tasks.ceph.ceph_manager.ceph:PG 15.2a is not active+clean 2024-08-03T21:00:55.783 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '15.2a', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 133, 'state': 'creating+peering', 'last_fresh': '2024-08-03T21:00:49.215684+0000', 'last_change': '2024-08-03T21:00:49.215684+0000', 'last_active': '2024-08-03T21:00:49.195479+0000', 'last_peered': '2024-08-03T21:00:49.195479+0000', 'last_clean': '2024-08-03T21:00:49.195479+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-08-03T21:00:49.215684+0000', 'last_undegraded': '2024-08-03T21:00:49.215684+0000', 'last_fullsized': '2024-08-03T21:00:49.215684+0000', 'mapping_epoch': 133, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 133, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-08-03T21:00:49.195479+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-08-03T21:00:49.195479+0000', 'last_clean_scrub_stamp': '2024-08-03T21:00:49.195479+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 5], 'acting': [0, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-08-03T21:00:55.783 INFO:tasks.ceph.ceph_manager.ceph:PG 15.29 is not active+clean 2024-08-03T21:00:55.783 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '15.29', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 133, 'state': 'creating+peering', 'last_fresh': '2024-08-03T21:00:49.214421+0000', 'last_change': '2024-08-03T21:00:49.214421+0000', 'last_active': '2024-08-03T21:00:49.195479+0000', 'last_peered': '2024-08-03T21:00:49.195479+0000', 'last_clean': '2024-08-03T21:00:49.195479+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-08-03T21:00:49.214421+0000', 'last_undegraded': '2024-08-03T21:00:49.214421+0000', 'last_fullsized': '2024-08-03T21:00:49.214421+0000', 'mapping_epoch': 133, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 133, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-08-03T21:00:49.195479+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-08-03T21:00:49.195479+0000', 'last_clean_scrub_stamp': '2024-08-03T21:00:49.195479+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 5], 'acting': [0, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-08-03T21:00:55.783 INFO:tasks.ceph.ceph_manager.ceph:PG 15.28 is not active+clean 2024-08-03T21:00:55.783 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '15.28', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 133, 'state': 'creating+peering', 'last_fresh': '2024-08-03T21:00:49.214489+0000', 'last_change': '2024-08-03T21:00:49.214489+0000', 'last_active': '2024-08-03T21:00:49.195479+0000', 'last_peered': '2024-08-03T21:00:49.195479+0000', 'last_clean': '2024-08-03T21:00:49.195479+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-08-03T21:00:49.214489+0000', 'last_undegraded': '2024-08-03T21:00:49.214489+0000', 'last_fullsized': '2024-08-03T21:00:49.214489+0000', 'mapping_epoch': 133, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 133, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-08-03T21:00:49.195479+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-08-03T21:00:49.195479+0000', 'last_clean_scrub_stamp': '2024-08-03T21:00:49.195479+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 1], 'acting': [7, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-08-03T21:00:55.783 INFO:tasks.ceph.ceph_manager.ceph:PG 15.27 is not active+clean 2024-08-03T21:00:55.783 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '15.27', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 133, 'state': 'creating+peering', 'last_fresh': '2024-08-03T21:00:49.211525+0000', 'last_change': '2024-08-03T21:00:49.211525+0000', 'last_active': '2024-08-03T21:00:49.195479+0000', 'last_peered': '2024-08-03T21:00:49.195479+0000', 'last_clean': '2024-08-03T21:00:49.195479+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-08-03T21:00:49.211525+0000', 'last_undegraded': '2024-08-03T21:00:49.211525+0000', 'last_fullsized': '2024-08-03T21:00:49.211525+0000', 'mapping_epoch': 133, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 133, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-08-03T21:00:49.195479+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-08-03T21:00:49.195479+0000', 'last_clean_scrub_stamp': '2024-08-03T21:00:49.195479+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 0], 'acting': [6, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-08-03T21:00:55.783 INFO:tasks.ceph.ceph_manager.ceph:PG 15.26 is not active+clean 2024-08-03T21:00:55.783 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '15.26', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 133, 'state': 'creating+peering', 'last_fresh': '2024-08-03T21:00:49.220518+0000', 'last_change': '2024-08-03T21:00:49.220518+0000', 'last_active': '2024-08-03T21:00:49.195479+0000', 'last_peered': '2024-08-03T21:00:49.195479+0000', 'last_clean': '2024-08-03T21:00:49.195479+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-08-03T21:00:49.220518+0000', 'last_undegraded': '2024-08-03T21:00:49.220518+0000', 'last_fullsized': '2024-08-03T21:00:49.220518+0000', 'mapping_epoch': 133, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 133, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-08-03T21:00:49.195479+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-08-03T21:00:49.195479+0000', 'last_clean_scrub_stamp': '2024-08-03T21:00:49.195479+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-08-03T21:00:55.784 INFO:tasks.ceph.ceph_manager.ceph:PG 15.25 is not active+clean 2024-08-03T21:00:55.784 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '15.25', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 133, 'state': 'creating+peering', 'last_fresh': '2024-08-03T21:00:49.216934+0000', 'last_change': '2024-08-03T21:00:49.216934+0000', 'last_active': '2024-08-03T21:00:49.195479+0000', 'last_peered': '2024-08-03T21:00:49.195479+0000', 'last_clean': '2024-08-03T21:00:49.195479+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-08-03T21:00:49.216934+0000', 'last_undegraded': '2024-08-03T21:00:49.216934+0000', 'last_fullsized': '2024-08-03T21:00:49.216934+0000', 'mapping_epoch': 133, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 133, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-08-03T21:00:49.195479+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-08-03T21:00:49.195479+0000', 'last_clean_scrub_stamp': '2024-08-03T21:00:49.195479+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 2], 'acting': [7, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-08-03T21:00:55.784 INFO:tasks.ceph.ceph_manager.ceph:PG 15.24 is not active+clean 2024-08-03T21:00:55.784 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '15.24', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 133, 'state': 'creating+peering', 'last_fresh': '2024-08-03T21:00:49.212944+0000', 'last_change': '2024-08-03T21:00:49.212944+0000', 'last_active': '2024-08-03T21:00:49.195479+0000', 'last_peered': '2024-08-03T21:00:49.195479+0000', 'last_clean': '2024-08-03T21:00:49.195479+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-08-03T21:00:49.212944+0000', 'last_undegraded': '2024-08-03T21:00:49.212944+0000', 'last_fullsized': '2024-08-03T21:00:49.212944+0000', 'mapping_epoch': 133, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 133, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-08-03T21:00:49.195479+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-08-03T21:00:49.195479+0000', 'last_clean_scrub_stamp': '2024-08-03T21:00:49.195479+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 3], 'acting': [6, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-08-03T21:00:55.784 INFO:tasks.ceph.ceph_manager.ceph:PG 15.23 is not active+clean 2024-08-03T21:00:55.784 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '15.23', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 133, 'state': 'creating+peering', 'last_fresh': '2024-08-03T21:00:49.219217+0000', 'last_change': '2024-08-03T21:00:49.219217+0000', 'last_active': '2024-08-03T21:00:49.195479+0000', 'last_peered': '2024-08-03T21:00:49.195479+0000', 'last_clean': '2024-08-03T21:00:49.195479+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-08-03T21:00:49.219217+0000', 'last_undegraded': '2024-08-03T21:00:49.219217+0000', 'last_fullsized': '2024-08-03T21:00:49.219217+0000', 'mapping_epoch': 133, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 133, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-08-03T21:00:49.195479+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-08-03T21:00:49.195479+0000', 'last_clean_scrub_stamp': '2024-08-03T21:00:49.195479+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-08-03T21:00:55.784 INFO:tasks.ceph.ceph_manager.ceph:PG 15.20 is not active+clean 2024-08-03T21:00:55.784 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '15.20', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 133, 'state': 'creating+peering', 'last_fresh': '2024-08-03T21:00:49.213774+0000', 'last_change': '2024-08-03T21:00:49.213774+0000', 'last_active': '2024-08-03T21:00:49.195479+0000', 'last_peered': '2024-08-03T21:00:49.195479+0000', 'last_clean': '2024-08-03T21:00:49.195479+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-08-03T21:00:49.213774+0000', 'last_undegraded': '2024-08-03T21:00:49.213774+0000', 'last_fullsized': '2024-08-03T21:00:49.213774+0000', 'mapping_epoch': 133, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 133, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-08-03T21:00:49.195479+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-08-03T21:00:49.195479+0000', 'last_clean_scrub_stamp': '2024-08-03T21:00:49.195479+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [1, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-08-03T21:00:55.784 INFO:tasks.ceph.ceph_manager.ceph:PG 15.1c is not active+clean 2024-08-03T21:00:55.784 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '15.1c', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 133, 'state': 'creating+peering', 'last_fresh': '2024-08-03T21:00:49.214791+0000', 'last_change': '2024-08-03T21:00:49.214791+0000', 'last_active': '2024-08-03T21:00:49.195479+0000', 'last_peered': '2024-08-03T21:00:49.195479+0000', 'last_clean': '2024-08-03T21:00:49.195479+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-08-03T21:00:49.214791+0000', 'last_undegraded': '2024-08-03T21:00:49.214791+0000', 'last_fullsized': '2024-08-03T21:00:49.214791+0000', 'mapping_epoch': 133, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 133, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-08-03T21:00:49.195479+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-08-03T21:00:49.195479+0000', 'last_clean_scrub_stamp': '2024-08-03T21:00:49.195479+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 4], 'acting': [1, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-08-03T21:00:55.784 INFO:tasks.ceph.ceph_manager.ceph:PG 15.1a is not active+clean 2024-08-03T21:00:55.785 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '15.1a', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 133, 'state': 'creating+peering', 'last_fresh': '2024-08-03T21:00:49.211222+0000', 'last_change': '2024-08-03T21:00:49.211222+0000', 'last_active': '2024-08-03T21:00:49.195479+0000', 'last_peered': '2024-08-03T21:00:49.195479+0000', 'last_clean': '2024-08-03T21:00:49.195479+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-08-03T21:00:49.211222+0000', 'last_undegraded': '2024-08-03T21:00:49.211222+0000', 'last_fullsized': '2024-08-03T21:00:49.211222+0000', 'mapping_epoch': 133, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 133, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-08-03T21:00:49.195479+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-08-03T21:00:49.195479+0000', 'last_clean_scrub_stamp': '2024-08-03T21:00:49.195479+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 4], 'acting': [5, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-08-03T21:00:55.785 INFO:tasks.ceph.ceph_manager.ceph:PG 15.1b is not active+clean 2024-08-03T21:00:55.785 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '15.1b', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 133, 'state': 'creating+peering', 'last_fresh': '2024-08-03T21:00:49.215473+0000', 'last_change': '2024-08-03T21:00:49.215473+0000', 'last_active': '2024-08-03T21:00:49.195479+0000', 'last_peered': '2024-08-03T21:00:49.195479+0000', 'last_clean': '2024-08-03T21:00:49.195479+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-08-03T21:00:49.215473+0000', 'last_undegraded': '2024-08-03T21:00:49.215473+0000', 'last_fullsized': '2024-08-03T21:00:49.215473+0000', 'mapping_epoch': 133, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 133, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-08-03T21:00:49.195479+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-08-03T21:00:49.195479+0000', 'last_clean_scrub_stamp': '2024-08-03T21:00:49.195479+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 6], 'acting': [7, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-08-03T21:00:55.785 INFO:tasks.ceph.ceph_manager.ceph:PG 15.16 is not active+clean 2024-08-03T21:00:55.785 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '15.16', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 133, 'state': 'creating+peering', 'last_fresh': '2024-08-03T21:00:49.219922+0000', 'last_change': '2024-08-03T21:00:49.219922+0000', 'last_active': '2024-08-03T21:00:49.195479+0000', 'last_peered': '2024-08-03T21:00:49.195479+0000', 'last_clean': '2024-08-03T21:00:49.195479+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-08-03T21:00:49.219922+0000', 'last_undegraded': '2024-08-03T21:00:49.219922+0000', 'last_fullsized': '2024-08-03T21:00:49.219922+0000', 'mapping_epoch': 133, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 133, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-08-03T21:00:49.195479+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-08-03T21:00:49.195479+0000', 'last_clean_scrub_stamp': '2024-08-03T21:00:49.195479+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 4], 'acting': [0, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-08-03T21:00:55.785 INFO:tasks.ceph.ceph_manager.ceph:PG 15.14 is not active+clean 2024-08-03T21:00:55.785 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '15.14', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 133, 'state': 'creating+peering', 'last_fresh': '2024-08-03T21:00:49.210789+0000', 'last_change': '2024-08-03T21:00:49.210789+0000', 'last_active': '2024-08-03T21:00:49.195479+0000', 'last_peered': '2024-08-03T21:00:49.195479+0000', 'last_clean': '2024-08-03T21:00:49.195479+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-08-03T21:00:49.210789+0000', 'last_undegraded': '2024-08-03T21:00:49.210789+0000', 'last_fullsized': '2024-08-03T21:00:49.210789+0000', 'mapping_epoch': 133, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 133, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-08-03T21:00:49.195479+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-08-03T21:00:49.195479+0000', 'last_clean_scrub_stamp': '2024-08-03T21:00:49.195479+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 0], 'acting': [6, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-08-03T21:00:55.785 INFO:tasks.ceph.ceph_manager.ceph:PG 15.15 is not active+clean 2024-08-03T21:00:55.785 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '15.15', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 133, 'state': 'creating+peering', 'last_fresh': '2024-08-03T21:00:49.216465+0000', 'last_change': '2024-08-03T21:00:49.216465+0000', 'last_active': '2024-08-03T21:00:49.195479+0000', 'last_peered': '2024-08-03T21:00:49.195479+0000', 'last_clean': '2024-08-03T21:00:49.195479+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-08-03T21:00:49.216465+0000', 'last_undegraded': '2024-08-03T21:00:49.216465+0000', 'last_fullsized': '2024-08-03T21:00:49.216465+0000', 'mapping_epoch': 133, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 133, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-08-03T21:00:49.195479+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-08-03T21:00:49.195479+0000', 'last_clean_scrub_stamp': '2024-08-03T21:00:49.195479+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 0], 'acting': [7, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-08-03T21:00:55.785 INFO:tasks.ceph.ceph_manager.ceph:PG 15.13 is not active+clean 2024-08-03T21:00:55.785 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '15.13', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 133, 'state': 'creating+peering', 'last_fresh': '2024-08-03T21:00:49.211586+0000', 'last_change': '2024-08-03T21:00:49.211586+0000', 'last_active': '2024-08-03T21:00:49.195479+0000', 'last_peered': '2024-08-03T21:00:49.195479+0000', 'last_clean': '2024-08-03T21:00:49.195479+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-08-03T21:00:49.211586+0000', 'last_undegraded': '2024-08-03T21:00:49.211586+0000', 'last_fullsized': '2024-08-03T21:00:49.211586+0000', 'mapping_epoch': 133, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 133, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-08-03T21:00:49.195479+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-08-03T21:00:49.195479+0000', 'last_clean_scrub_stamp': '2024-08-03T21:00:49.195479+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [1, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-08-03T21:00:55.786 INFO:tasks.ceph.ceph_manager.ceph:PG 15.f is not active+clean 2024-08-03T21:00:55.786 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '15.f', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 133, 'state': 'creating+peering', 'last_fresh': '2024-08-03T21:00:49.209245+0000', 'last_change': '2024-08-03T21:00:49.209245+0000', 'last_active': '2024-08-03T21:00:49.195479+0000', 'last_peered': '2024-08-03T21:00:49.195479+0000', 'last_clean': '2024-08-03T21:00:49.195479+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-08-03T21:00:49.209245+0000', 'last_undegraded': '2024-08-03T21:00:49.209245+0000', 'last_fullsized': '2024-08-03T21:00:49.209245+0000', 'mapping_epoch': 133, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 133, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-08-03T21:00:49.195479+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-08-03T21:00:49.195479+0000', 'last_clean_scrub_stamp': '2024-08-03T21:00:49.195479+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 4], 'acting': [6, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-08-03T21:00:55.786 INFO:tasks.ceph.ceph_manager.ceph:PG 15.c is not active+clean 2024-08-03T21:00:55.786 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '15.c', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 133, 'state': 'creating+peering', 'last_fresh': '2024-08-03T21:00:49.211956+0000', 'last_change': '2024-08-03T21:00:49.211956+0000', 'last_active': '2024-08-03T21:00:49.195479+0000', 'last_peered': '2024-08-03T21:00:49.195479+0000', 'last_clean': '2024-08-03T21:00:49.195479+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-08-03T21:00:49.211956+0000', 'last_undegraded': '2024-08-03T21:00:49.211956+0000', 'last_fullsized': '2024-08-03T21:00:49.211956+0000', 'mapping_epoch': 133, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 133, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-08-03T21:00:49.195479+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-08-03T21:00:49.195479+0000', 'last_clean_scrub_stamp': '2024-08-03T21:00:49.195479+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 6], 'acting': [5, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-08-03T21:00:55.786 INFO:tasks.ceph.ceph_manager.ceph:PG 15.d is not active+clean 2024-08-03T21:00:55.786 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '15.d', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 133, 'state': 'creating+peering', 'last_fresh': '2024-08-03T21:00:49.213316+0000', 'last_change': '2024-08-03T21:00:49.213316+0000', 'last_active': '2024-08-03T21:00:49.195479+0000', 'last_peered': '2024-08-03T21:00:49.195479+0000', 'last_clean': '2024-08-03T21:00:49.195479+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-08-03T21:00:49.213316+0000', 'last_undegraded': '2024-08-03T21:00:49.213316+0000', 'last_fullsized': '2024-08-03T21:00:49.213316+0000', 'mapping_epoch': 133, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 133, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-08-03T21:00:49.195479+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-08-03T21:00:49.195479+0000', 'last_clean_scrub_stamp': '2024-08-03T21:00:49.195479+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 4], 'acting': [7, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-08-03T21:00:55.786 INFO:tasks.ceph.ceph_manager.ceph:PG 15.9 is not active+clean 2024-08-03T21:00:55.786 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '15.9', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 133, 'state': 'creating+peering', 'last_fresh': '2024-08-03T21:00:49.213171+0000', 'last_change': '2024-08-03T21:00:49.213171+0000', 'last_active': '2024-08-03T21:00:49.195479+0000', 'last_peered': '2024-08-03T21:00:49.195479+0000', 'last_clean': '2024-08-03T21:00:49.195479+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-08-03T21:00:49.213171+0000', 'last_undegraded': '2024-08-03T21:00:49.213171+0000', 'last_fullsized': '2024-08-03T21:00:49.213171+0000', 'mapping_epoch': 133, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 133, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-08-03T21:00:49.195479+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-08-03T21:00:49.195479+0000', 'last_clean_scrub_stamp': '2024-08-03T21:00:49.195479+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 1], 'acting': [6, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-08-03T21:00:55.786 INFO:tasks.ceph.ceph_manager.ceph:PG 15.6 is not active+clean 2024-08-03T21:00:55.786 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '15.6', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 133, 'state': 'creating+peering', 'last_fresh': '2024-08-03T21:00:49.217620+0000', 'last_change': '2024-08-03T21:00:49.217620+0000', 'last_active': '2024-08-03T21:00:49.195479+0000', 'last_peered': '2024-08-03T21:00:49.195479+0000', 'last_clean': '2024-08-03T21:00:49.195479+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-08-03T21:00:49.217620+0000', 'last_undegraded': '2024-08-03T21:00:49.217620+0000', 'last_fullsized': '2024-08-03T21:00:49.217620+0000', 'mapping_epoch': 133, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 133, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-08-03T21:00:49.195479+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-08-03T21:00:49.195479+0000', 'last_clean_scrub_stamp': '2024-08-03T21:00:49.195479+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 5], 'acting': [0, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-08-03T21:00:55.786 INFO:tasks.ceph.ceph_manager.ceph:PG 15.7 is not active+clean 2024-08-03T21:00:55.787 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '15.7', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 133, 'state': 'creating+peering', 'last_fresh': '2024-08-03T21:00:49.212025+0000', 'last_change': '2024-08-03T21:00:49.212025+0000', 'last_active': '2024-08-03T21:00:49.195479+0000', 'last_peered': '2024-08-03T21:00:49.195479+0000', 'last_clean': '2024-08-03T21:00:49.195479+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-08-03T21:00:49.212025+0000', 'last_undegraded': '2024-08-03T21:00:49.212025+0000', 'last_fullsized': '2024-08-03T21:00:49.212025+0000', 'mapping_epoch': 133, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 133, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-08-03T21:00:49.195479+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-08-03T21:00:49.195479+0000', 'last_clean_scrub_stamp': '2024-08-03T21:00:49.195479+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [1, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-08-03T21:00:55.787 INFO:tasks.ceph.ceph_manager.ceph:PG 15.4 is not active+clean 2024-08-03T21:00:55.787 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '15.4', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 133, 'state': 'creating+peering', 'last_fresh': '2024-08-03T21:00:49.212551+0000', 'last_change': '2024-08-03T21:00:49.212551+0000', 'last_active': '2024-08-03T21:00:49.195479+0000', 'last_peered': '2024-08-03T21:00:49.195479+0000', 'last_clean': '2024-08-03T21:00:49.195479+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-08-03T21:00:49.212551+0000', 'last_undegraded': '2024-08-03T21:00:49.212551+0000', 'last_fullsized': '2024-08-03T21:00:49.212551+0000', 'mapping_epoch': 133, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 133, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-08-03T21:00:49.195479+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-08-03T21:00:49.195479+0000', 'last_clean_scrub_stamp': '2024-08-03T21:00:49.195479+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 6], 'acting': [1, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-08-03T21:00:55.787 INFO:tasks.ceph.ceph_manager.ceph:PG 15.5 is not active+clean 2024-08-03T21:00:55.787 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '15.5', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 133, 'state': 'creating+peering', 'last_fresh': '2024-08-03T21:00:49.209221+0000', 'last_change': '2024-08-03T21:00:49.209221+0000', 'last_active': '2024-08-03T21:00:49.195479+0000', 'last_peered': '2024-08-03T21:00:49.195479+0000', 'last_clean': '2024-08-03T21:00:49.195479+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-08-03T21:00:49.209221+0000', 'last_undegraded': '2024-08-03T21:00:49.209221+0000', 'last_fullsized': '2024-08-03T21:00:49.209221+0000', 'mapping_epoch': 133, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 133, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-08-03T21:00:49.195479+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-08-03T21:00:49.195479+0000', 'last_clean_scrub_stamp': '2024-08-03T21:00:49.195479+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 3], 'acting': [6, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-08-03T21:00:55.787 INFO:tasks.ceph.ceph_manager.ceph:PG 15.1 is not active+clean 2024-08-03T21:00:55.787 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '15.1', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 133, 'state': 'creating+peering', 'last_fresh': '2024-08-03T21:00:49.210235+0000', 'last_change': '2024-08-03T21:00:49.210235+0000', 'last_active': '2024-08-03T21:00:49.195479+0000', 'last_peered': '2024-08-03T21:00:49.195479+0000', 'last_clean': '2024-08-03T21:00:49.195479+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-08-03T21:00:49.210235+0000', 'last_undegraded': '2024-08-03T21:00:49.210235+0000', 'last_fullsized': '2024-08-03T21:00:49.210235+0000', 'mapping_epoch': 133, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 133, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-08-03T21:00:49.195479+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-08-03T21:00:49.195479+0000', 'last_clean_scrub_stamp': '2024-08-03T21:00:49.195479+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 3], 'acting': [6, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-08-03T21:00:55.788 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-08-03T21:00:56.044 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:56.042+0000 7f5e0ea00640 1 -- 172.21.15.32:0/1264478265 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f5e080d26e0 msgr2=0x7f5e08161b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:00:56.044 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:56.042+0000 7f5e0ea00640 1 --2- 172.21.15.32:0/1264478265 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f5e080d26e0 0x7f5e08161b80 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f5dfc009860 tx=0x7f5dfc02a5a0 comp rx=0 tx=0).stop 2024-08-03T21:00:56.044 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:56.042+0000 7f5e0ea00640 1 -- 172.21.15.32:0/1264478265 shutdown_connections 2024-08-03T21:00:56.044 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:56.042+0000 7f5e0ea00640 1 --2- 172.21.15.32:0/1264478265 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f5e080d26e0 0x7f5e08161b80 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:56.044 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:56.042+0000 7f5e0ea00640 1 -- 172.21.15.32:0/1264478265 >> 172.21.15.32:0/1264478265 conn(0x7f5e0808d7a0 msgr2=0x7f5e0809d3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T21:00:56.044 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:56.042+0000 7f5e0ea00640 1 -- 172.21.15.32:0/1264478265 shutdown_connections 2024-08-03T21:00:56.044 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:56.042+0000 7f5e0ea00640 1 -- 172.21.15.32:0/1264478265 wait complete. 2024-08-03T21:00:56.046 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:56.045+0000 7f5e0ea00640 1 Processor -- start 2024-08-03T21:00:56.046 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:56.045+0000 7f5e0ea00640 1 -- start start 2024-08-03T21:00:56.048 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:56.046+0000 7f5e0ea00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5e080cf510 0x7f5e080cd860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:00:56.049 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:56.046+0000 7f5e0ea00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f5e080cffe0 0x7f5e080ca4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:00:56.049 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:56.046+0000 7f5e0ea00640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f5e080d26e0 0x7f5e080b8540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:00:56.049 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:56.046+0000 7f5e0ea00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f5e080cf920 con 0x7f5e080cf510 2024-08-03T21:00:56.049 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:56.046+0000 7f5e0ea00640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f5e080d48a0 con 0x7f5e080cffe0 2024-08-03T21:00:56.049 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:56.046+0000 7f5e0ea00640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7f5e080cfaa0 con 0x7f5e080d26e0 2024-08-03T21:00:56.049 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:56.046+0000 7f5e0d600640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5e080cf510 0x7f5e080cd860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:00:56.049 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:56.047+0000 7f5e0d600640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5e080cf510 0x7f5e080cd860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:50100/0 (socket says 172.21.15.32:50100) 2024-08-03T21:00:56.049 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:56.047+0000 7f5e0d600640 1 -- 172.21.15.32:0/1121594347 learned_addr learned my addr 172.21.15.32:0/1121594347 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T21:00:56.049 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:56.047+0000 7f5e0cc00640 1 --2- 172.21.15.32:0/1121594347 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f5e080cffe0 0x7f5e080ca4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:00:56.049 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:56.047+0000 7f5e0e000640 1 --2- 172.21.15.32:0/1121594347 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f5e080d26e0 0x7f5e080b8540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:00:56.049 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:56.047+0000 7f5e0d600640 1 -- 172.21.15.32:0/1121594347 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f5e080d26e0 msgr2=0x7f5e080b8540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:00:56.049 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:56.047+0000 7f5e0d600640 1 --2- 172.21.15.32:0/1121594347 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f5e080d26e0 0x7f5e080b8540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:56.050 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:56.047+0000 7f5e0d600640 1 -- 172.21.15.32:0/1121594347 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f5e080cffe0 msgr2=0x7f5e080ca4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:00:56.050 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:56.047+0000 7f5e0d600640 1 --2- 172.21.15.32:0/1121594347 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f5e080cffe0 0x7f5e080ca4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:56.050 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:56.047+0000 7f5e0d600640 1 -- 172.21.15.32:0/1121594347 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5e081f5300 con 0x7f5e080cf510 2024-08-03T21:00:56.050 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:56.047+0000 7f5e0cc00640 1 --2- 172.21.15.32:0/1121594347 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f5e080cffe0 0x7f5e080ca4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-03T21:00:56.050 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:56.047+0000 7f5e0e000640 1 --2- 172.21.15.32:0/1121594347 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f5e080d26e0 0x7f5e080b8540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-03T21:00:56.050 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:56.047+0000 7f5e0d600640 1 --2- 172.21.15.32:0/1121594347 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5e080cf510 0x7f5e080cd860 secure :-1 s=READY pgs=2177 cs=0 l=1 rev1=1 crypto rx=0x7f5dfc02ab90 tx=0x7f5dfc033c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T21:00:56.051 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:56.047+0000 7f5e05600640 1 -- 172.21.15.32:0/1121594347 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5dfc031d60 con 0x7f5e080cf510 2024-08-03T21:00:56.051 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:56.048+0000 7f5e05600640 1 -- 172.21.15.32:0/1121594347 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f5dfc033e10 con 0x7f5e080cf510 2024-08-03T21:00:56.051 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:56.048+0000 7f5e05600640 1 -- 172.21.15.32:0/1121594347 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5dfc005260 con 0x7f5e080cf510 2024-08-03T21:00:56.051 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:56.048+0000 7f5e0ea00640 1 -- 172.21.15.32:0/1121594347 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5e081f5800 con 0x7f5e080cf510 2024-08-03T21:00:56.051 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:56.048+0000 7f5e0ea00640 1 -- 172.21.15.32:0/1121594347 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5e081f5dc0 con 0x7f5e080cf510 2024-08-03T21:00:56.051 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:56.049+0000 7f5e0ea00640 1 -- 172.21.15.32:0/1121594347 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5e080d3f80 con 0x7f5e080cf510 2024-08-03T21:00:56.051 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:56.050+0000 7f5e05600640 1 -- 172.21.15.32:0/1121594347 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f5dfc005400 con 0x7f5e080cf510 2024-08-03T21:00:56.052 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:56.051+0000 7f5e05600640 1 --2- 172.21.15.32:0/1121594347 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f5de8077040 0x7f5de8079550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:00:56.052 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:56.051+0000 7f5e05600640 1 -- 172.21.15.32:0/1121594347 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(135..135 src has 1..135) ==== 5709+0+0 (secure 0 0 0) 0x7f5dfc0b9a30 con 0x7f5e080cf510 2024-08-03T21:00:56.053 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:56.051+0000 7f5e0cc00640 1 --2- 172.21.15.32:0/1121594347 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f5de8077040 0x7f5de8079550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:00:56.054 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:56.052+0000 7f5e0cc00640 1 --2- 172.21.15.32:0/1121594347 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f5de8077040 0x7f5de8079550 secure :-1 s=READY pgs=873 cs=0 l=1 rev1=1 crypto rx=0x7f5e08161fb0 tx=0x7f5df4009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T21:00:56.057 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:56.055+0000 7f5e05600640 1 -- 172.21.15.32:0/1121594347 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7f5dfc03f030 con 0x7f5e080cf510 2024-08-03T21:00:56.337 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:56.336+0000 7f5e0ea00640 1 -- 172.21.15.32:0/1121594347 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "status", "format": "json"} v 0) -- 0x7f5e080cdda0 con 0x7f5e080cf510 2024-08-03T21:00:56.338 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:56.336+0000 7f5e05600640 1 -- 172.21.15.32:0/1121594347 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json"}]=0 v0) ==== 72+0+1143 (secure 0 0 0) 0x7f5dfc082b90 con 0x7f5e080cf510 2024-08-03T21:00:56.338 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-03T21:00:56.338 INFO:teuthology.orchestra.run.smithi032.stdout:{"fsid":"40ab141a-01d7-49b9-ae3c-2edec9876e95","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":821,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":135,"num_osds":8,"num_up_osds":8,"osd_up_since":1722718048,"num_in_osds":8,"osd_in_since":1722718040,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":61},{"state_name":"creating+peering","count":36}],"num_pgs":97,"num_pools":3,"num_objects":9,"data_bytes":592014,"bytes_used":308625408,"bytes_avail":772785487872,"bytes_total":773094113280,"inactive_pgs_ratio":0.37113401293754578,"write_bytes_sec":1706,"read_op_per_sec":0,"write_op_per_sec":1},"fsmap":{"epoch":340,"btime":"2024-08-03T21:00:53:231957+0000","id":7,"up":1,"in":1,"max":1,"by_rank":[{"filesystem_id":7,"rank":0,"name":"a","status":"up:active","gid":9592}],"up:standby":3},"mgrmap":{"available":true,"num_standbys":1,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":52,"modified":"2024-08-03T21:00:28.447311+0000","services":{}},"progress_events":{}} 2024-08-03T21:00:56.340 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:56.338+0000 7f5e0ea00640 1 -- 172.21.15.32:0/1121594347 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f5de8077040 msgr2=0x7f5de8079550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:00:56.340 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:56.338+0000 7f5e0ea00640 1 --2- 172.21.15.32:0/1121594347 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f5de8077040 0x7f5de8079550 secure :-1 s=READY pgs=873 cs=0 l=1 rev1=1 crypto rx=0x7f5e08161fb0 tx=0x7f5df4009040 comp rx=0 tx=0).stop 2024-08-03T21:00:56.340 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:56.338+0000 7f5e0ea00640 1 -- 172.21.15.32:0/1121594347 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5e080cf510 msgr2=0x7f5e080cd860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:00:56.340 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:56.338+0000 7f5e0ea00640 1 --2- 172.21.15.32:0/1121594347 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5e080cf510 0x7f5e080cd860 secure :-1 s=READY pgs=2177 cs=0 l=1 rev1=1 crypto rx=0x7f5dfc02ab90 tx=0x7f5dfc033c20 comp rx=0 tx=0).stop 2024-08-03T21:00:56.340 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:56.338+0000 7f5e0ea00640 1 -- 172.21.15.32:0/1121594347 shutdown_connections 2024-08-03T21:00:56.340 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:56.338+0000 7f5e0ea00640 1 --2- 172.21.15.32:0/1121594347 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f5de8077040 0x7f5de8079550 unknown :-1 s=CLOSED pgs=873 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:56.341 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:56.338+0000 7f5e0ea00640 1 --2- 172.21.15.32:0/1121594347 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f5e080cffe0 0x7f5e080ca4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:56.341 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:56.338+0000 7f5e0ea00640 1 --2- 172.21.15.32:0/1121594347 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f5e080d26e0 0x7f5e080b8540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:56.341 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:56.338+0000 7f5e0ea00640 1 --2- 172.21.15.32:0/1121594347 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5e080cf510 0x7f5e080cd860 unknown :-1 s=CLOSED pgs=2177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:56.341 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:56.338+0000 7f5e0ea00640 1 -- 172.21.15.32:0/1121594347 >> 172.21.15.32:0/1121594347 conn(0x7f5e0808d7a0 msgr2=0x7f5e080b6010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T21:00:56.341 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:56.338+0000 7f5e0ea00640 1 -- 172.21.15.32:0/1121594347 shutdown_connections 2024-08-03T21:00:56.341 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:56.339+0000 7f5e0ea00640 1 -- 172.21.15.32:0/1121594347 wait complete. 2024-08-03T21:00:56.362 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2024-08-03T21:00:56.362 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-08-03T21:00:56.589 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:56.587+0000 7f65a2e00640 1 -- 172.21.15.32:0/797815784 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f659c15fff0 msgr2=0x7f659c160400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:00:56.589 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:56.587+0000 7f65a2e00640 1 --2- 172.21.15.32:0/797815784 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f659c15fff0 0x7f659c160400 secure :-1 s=READY pgs=2178 cs=0 l=1 rev1=1 crypto rx=0x7f6594005fd0 tx=0x7f659402aaa0 comp rx=0 tx=0).stop 2024-08-03T21:00:56.589 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:56.587+0000 7f65a2e00640 1 -- 172.21.15.32:0/797815784 shutdown_connections 2024-08-03T21:00:56.589 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:56.587+0000 7f65a2e00640 1 --2- 172.21.15.32:0/797815784 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f659c15fff0 0x7f659c160400 unknown :-1 s=CLOSED pgs=2178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:56.589 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:56.587+0000 7f65a2e00640 1 --2- 172.21.15.32:0/797815784 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f659c06e930 0x7f659c144f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:56.589 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:56.587+0000 7f65a2e00640 1 -- 172.21.15.32:0/797815784 >> 172.21.15.32:0/797815784 conn(0x7f659c0b1bd0 msgr2=0x7f659c0b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T21:00:56.589 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:56.587+0000 7f65a2e00640 1 -- 172.21.15.32:0/797815784 shutdown_connections 2024-08-03T21:00:56.589 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:56.587+0000 7f65a2e00640 1 -- 172.21.15.32:0/797815784 wait complete. 2024-08-03T21:00:56.591 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:56.589+0000 7f65a2e00640 1 Processor -- start 2024-08-03T21:00:56.591 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:56.590+0000 7f65a2e00640 1 -- start start 2024-08-03T21:00:56.593 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:56.591+0000 7f65a2e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f659c06e930 0x7f659c13a670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:00:56.593 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:56.591+0000 7f65a2e00640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f659c073e60 0x7f659c13d970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:00:56.593 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:56.591+0000 7f65a2e00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f659c15fff0 0x7f659c13deb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:00:56.593 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:56.591+0000 7f65a2e00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f659c0cf840 con 0x7f659c06e930 2024-08-03T21:00:56.593 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:56.591+0000 7f65a2e00640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f659c145f30 con 0x7f659c15fff0 2024-08-03T21:00:56.593 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:56.591+0000 7f65a2e00640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7f659c0cf9c0 con 0x7f659c073e60 2024-08-03T21:00:56.593 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:56.591+0000 7f659be00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f659c06e930 0x7f659c13a670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:00:56.593 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:56.591+0000 7f659be00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f659c06e930 0x7f659c13a670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:50122/0 (socket says 172.21.15.32:50122) 2024-08-03T21:00:56.593 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:56.591+0000 7f659be00640 1 -- 172.21.15.32:0/3648003166 learned_addr learned my addr 172.21.15.32:0/3648003166 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T21:00:56.594 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:56.591+0000 7f6593e00640 1 --2- 172.21.15.32:0/3648003166 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f659c073e60 0x7f659c13d970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:00:56.594 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:56.592+0000 7f65a0e00640 1 --2- 172.21.15.32:0/3648003166 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f659c15fff0 0x7f659c13deb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:00:56.595 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:56.592+0000 7f659be00640 1 -- 172.21.15.32:0/3648003166 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f659c073e60 msgr2=0x7f659c13d970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:00:56.595 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:56.592+0000 7f659be00640 1 --2- 172.21.15.32:0/3648003166 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f659c073e60 0x7f659c13d970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:56.595 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:56.592+0000 7f659be00640 1 -- 172.21.15.32:0/3648003166 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f659c15fff0 msgr2=0x7f659c13deb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:00:56.595 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:56.592+0000 7f659be00640 1 --2- 172.21.15.32:0/3648003166 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f659c15fff0 0x7f659c13deb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:56.595 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:56.592+0000 7f659be00640 1 -- 172.21.15.32:0/3648003166 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f659c13c8d0 con 0x7f659c06e930 2024-08-03T21:00:56.595 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:56.592+0000 7f659be00640 1 --2- 172.21.15.32:0/3648003166 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f659c06e930 0x7f659c13a670 secure :-1 s=READY pgs=2179 cs=0 l=1 rev1=1 crypto rx=0x7f658c00e7c0 tx=0x7f658c00ec80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T21:00:56.595 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:56.593+0000 7f6598c00640 1 -- 172.21.15.32:0/3648003166 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f658c00ccc0 con 0x7f659c06e930 2024-08-03T21:00:56.595 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:56.593+0000 7f6598c00640 1 -- 172.21.15.32:0/3648003166 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f658c010430 con 0x7f659c06e930 2024-08-03T21:00:56.595 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:56.593+0000 7f6598c00640 1 -- 172.21.15.32:0/3648003166 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f658c010860 con 0x7f659c06e930 2024-08-03T21:00:56.595 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:56.593+0000 7f65a2e00640 1 -- 172.21.15.32:0/3648003166 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f659c13cbb0 con 0x7f659c06e930 2024-08-03T21:00:56.596 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:56.593+0000 7f65a2e00640 1 -- 172.21.15.32:0/3648003166 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f659c13d170 con 0x7f659c06e930 2024-08-03T21:00:56.596 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:56.594+0000 7f65a2e00640 1 -- 172.21.15.32:0/3648003166 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f659c160b70 con 0x7f659c06e930 2024-08-03T21:00:56.596 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:56.595+0000 7f6598c00640 1 -- 172.21.15.32:0/3648003166 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f658c010a00 con 0x7f659c06e930 2024-08-03T21:00:56.597 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:56.596+0000 7f6598c00640 1 --2- 172.21.15.32:0/3648003166 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f6578077040 0x7f6578079550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:00:56.598 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:56.596+0000 7f6598c00640 1 -- 172.21.15.32:0/3648003166 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(135..135 src has 1..135) ==== 5709+0+0 (secure 0 0 0) 0x7f658c0992a0 con 0x7f659c06e930 2024-08-03T21:00:56.598 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:56.597+0000 7f6593e00640 1 --2- 172.21.15.32:0/3648003166 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f6578077040 0x7f6578079550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:00:56.599 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:56.597+0000 7f6593e00640 1 --2- 172.21.15.32:0/3648003166 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f6578077040 0x7f6578079550 secure :-1 s=READY pgs=874 cs=0 l=1 rev1=1 crypto rx=0x7f659c15c0d0 tx=0x7f6584009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T21:00:56.602 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:56.600+0000 7f6598c00640 1 -- 172.21.15.32:0/3648003166 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7f658c061db0 con 0x7f659c06e930 2024-08-03T21:00:56.805 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:56.804+0000 7f65a2e00640 1 -- 172.21.15.32:0/3648003166 --> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f659c145a00 con 0x7f6578077040 2024-08-03T21:00:56.810 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:56.809+0000 7f6598c00640 1 -- 172.21.15.32:0/3648003166 <== mgr.4106 v2:172.21.15.118:6800/4205950699 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+252476 (secure 0 0 0) 0x7f659c145a00 con 0x7f6578077040 2024-08-03T21:00:56.813 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-03T21:00:56.815 INFO:teuthology.orchestra.run.smithi032.stderr:dumped all 2024-08-03T21:00:56.816 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:56.813+0000 7f65a2e00640 1 -- 172.21.15.32:0/3648003166 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f6578077040 msgr2=0x7f6578079550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:00:56.816 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:56.813+0000 7f65a2e00640 1 --2- 172.21.15.32:0/3648003166 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f6578077040 0x7f6578079550 secure :-1 s=READY pgs=874 cs=0 l=1 rev1=1 crypto rx=0x7f659c15c0d0 tx=0x7f6584009040 comp rx=0 tx=0).stop 2024-08-03T21:00:56.816 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:56.813+0000 7f65a2e00640 1 -- 172.21.15.32:0/3648003166 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f659c06e930 msgr2=0x7f659c13a670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:00:56.816 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:56.813+0000 7f65a2e00640 1 --2- 172.21.15.32:0/3648003166 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f659c06e930 0x7f659c13a670 secure :-1 s=READY pgs=2179 cs=0 l=1 rev1=1 crypto rx=0x7f658c00e7c0 tx=0x7f658c00ec80 comp rx=0 tx=0).stop 2024-08-03T21:00:56.816 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:56.813+0000 7f65a2e00640 1 -- 172.21.15.32:0/3648003166 shutdown_connections 2024-08-03T21:00:56.816 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:56.813+0000 7f65a2e00640 1 --2- 172.21.15.32:0/3648003166 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f6578077040 0x7f6578079550 unknown :-1 s=CLOSED pgs=874 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:56.816 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:56.813+0000 7f65a2e00640 1 --2- 172.21.15.32:0/3648003166 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f659c15fff0 0x7f659c13deb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:56.817 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:56.813+0000 7f65a2e00640 1 --2- 172.21.15.32:0/3648003166 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f659c073e60 0x7f659c13d970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:56.817 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:56.813+0000 7f65a2e00640 1 --2- 172.21.15.32:0/3648003166 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f659c06e930 0x7f659c13a670 unknown :-1 s=CLOSED pgs=2179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:00:56.817 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:56.813+0000 7f65a2e00640 1 -- 172.21.15.32:0/3648003166 >> 172.21.15.32:0/3648003166 conn(0x7f659c0b1bd0 msgr2=0x7f659c15ad30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T21:00:56.817 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:56.813+0000 7f65a2e00640 1 -- 172.21.15.32:0/3648003166 shutdown_connections 2024-08-03T21:00:56.817 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:00:56.813+0000 7f65a2e00640 1 -- 172.21.15.32:0/3648003166 wait complete. 2024-08-03T21:00:56.837 INFO:teuthology.orchestra.run.smithi032.stdout:{"pg_ready":true,"pg_map":{"version":532,"stamp":"2024-08-03T21:00:56.454859+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":146,"num_write_kb":1642,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":111,"ondisk_log_size":111,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":301596,"kb_used_data":14696,"kb_used_omap":12,"kb_used_meta":286771,"kb_avail":754673124,"statfs":{"total":773094113280,"available":772785278976,"internally_reserved":0,"allocated":15048704,"data_stored":7832796,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":293654065},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":35,"apply_latency_ms":35,"commit_latency_ns":35000000,"apply_latency_ns":35000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.251104"},"pg_stats":[{"pgid":"15.3f","version":"0'0","reported_seq":13,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-08-03T21:00:51.937735+0000","last_change":"2024-08-03T21:00:50.248721+0000","last_active":"2024-08-03T21:00:51.937735+0000","last_peered":"2024-08-03T21:00:51.937735+0000","last_clean":"2024-08-03T21:00:51.937735+0000","last_became_active":"2024-08-03T21:00:50.242246+0000","last_became_peered":"2024-08-03T21:00:50.242246+0000","last_unstale":"2024-08-03T21:00:51.937735+0000","last_undegraded":"2024-08-03T21:00:51.937735+0000","last_fullsized":"2024-08-03T21:00:51.937735+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_clean_scrub_stamp":"2024-08-03T21:00:49.195479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-04T23:49:06.542747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.3e","version":"0'0","reported_seq":13,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-08-03T21:00:51.939015+0000","last_change":"2024-08-03T21:00:50.245164+0000","last_active":"2024-08-03T21:00:51.939015+0000","last_peered":"2024-08-03T21:00:51.939015+0000","last_clean":"2024-08-03T21:00:51.939015+0000","last_became_active":"2024-08-03T21:00:50.244799+0000","last_became_peered":"2024-08-03T21:00:50.244799+0000","last_unstale":"2024-08-03T21:00:51.939015+0000","last_undegraded":"2024-08-03T21:00:51.939015+0000","last_fullsized":"2024-08-03T21:00:51.939015+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_clean_scrub_stamp":"2024-08-03T21:00:49.195479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T00:55:29.579986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.3d","version":"0'0","reported_seq":13,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-08-03T21:00:51.937833+0000","last_change":"2024-08-03T21:00:50.246785+0000","last_active":"2024-08-03T21:00:51.937833+0000","last_peered":"2024-08-03T21:00:51.937833+0000","last_clean":"2024-08-03T21:00:51.937833+0000","last_became_active":"2024-08-03T21:00:50.241990+0000","last_became_peered":"2024-08-03T21:00:50.241990+0000","last_unstale":"2024-08-03T21:00:51.937833+0000","last_undegraded":"2024-08-03T21:00:51.937833+0000","last_fullsized":"2024-08-03T21:00:51.937833+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_clean_scrub_stamp":"2024-08-03T21:00:49.195479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T08:47:17.514748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.3c","version":"0'0","reported_seq":13,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-08-03T21:00:51.225682+0000","last_change":"2024-08-03T21:00:50.243269+0000","last_active":"2024-08-03T21:00:51.225682+0000","last_peered":"2024-08-03T21:00:51.225682+0000","last_clean":"2024-08-03T21:00:51.225682+0000","last_became_active":"2024-08-03T21:00:50.242985+0000","last_became_peered":"2024-08-03T21:00:50.242985+0000","last_unstale":"2024-08-03T21:00:51.225682+0000","last_undegraded":"2024-08-03T21:00:51.225682+0000","last_fullsized":"2024-08-03T21:00:51.225682+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_clean_scrub_stamp":"2024-08-03T21:00:49.195479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T03:32:19.871676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.3b","version":"0'0","reported_seq":13,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-08-03T21:00:51.938596+0000","last_change":"2024-08-03T21:00:50.222469+0000","last_active":"2024-08-03T21:00:51.938596+0000","last_peered":"2024-08-03T21:00:51.938596+0000","last_clean":"2024-08-03T21:00:51.938596+0000","last_became_active":"2024-08-03T21:00:50.222097+0000","last_became_peered":"2024-08-03T21:00:50.222097+0000","last_unstale":"2024-08-03T21:00:51.938596+0000","last_undegraded":"2024-08-03T21:00:51.938596+0000","last_fullsized":"2024-08-03T21:00:51.938596+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_clean_scrub_stamp":"2024-08-03T21:00:49.195479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T05:22:18.315672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.3a","version":"0'0","reported_seq":13,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-08-03T21:00:51.226404+0000","last_change":"2024-08-03T21:00:50.240363+0000","last_active":"2024-08-03T21:00:51.226404+0000","last_peered":"2024-08-03T21:00:51.226404+0000","last_clean":"2024-08-03T21:00:51.226404+0000","last_became_active":"2024-08-03T21:00:50.239995+0000","last_became_peered":"2024-08-03T21:00:50.239995+0000","last_unstale":"2024-08-03T21:00:51.226404+0000","last_undegraded":"2024-08-03T21:00:51.226404+0000","last_fullsized":"2024-08-03T21:00:51.226404+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_clean_scrub_stamp":"2024-08-03T21:00:49.195479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T03:52:56.414014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.39","version":"0'0","reported_seq":13,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-08-03T21:00:51.226291+0000","last_change":"2024-08-03T21:00:50.230272+0000","last_active":"2024-08-03T21:00:51.226291+0000","last_peered":"2024-08-03T21:00:51.226291+0000","last_clean":"2024-08-03T21:00:51.226291+0000","last_became_active":"2024-08-03T21:00:50.229940+0000","last_became_peered":"2024-08-03T21:00:50.229940+0000","last_unstale":"2024-08-03T21:00:51.226291+0000","last_undegraded":"2024-08-03T21:00:51.226291+0000","last_fullsized":"2024-08-03T21:00:51.226291+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_clean_scrub_stamp":"2024-08-03T21:00:49.195479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-04T21:28:54.164554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.38","version":"0'0","reported_seq":13,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-08-03T21:00:51.937383+0000","last_change":"2024-08-03T21:00:50.233757+0000","last_active":"2024-08-03T21:00:51.937383+0000","last_peered":"2024-08-03T21:00:51.937383+0000","last_clean":"2024-08-03T21:00:51.937383+0000","last_became_active":"2024-08-03T21:00:50.233470+0000","last_became_peered":"2024-08-03T21:00:50.233470+0000","last_unstale":"2024-08-03T21:00:51.937383+0000","last_undegraded":"2024-08-03T21:00:51.937383+0000","last_fullsized":"2024-08-03T21:00:51.937383+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_clean_scrub_stamp":"2024-08-03T21:00:49.195479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T04:02:53.255557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.37","version":"0'0","reported_seq":13,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-08-03T21:00:51.939823+0000","last_change":"2024-08-03T21:00:50.223154+0000","last_active":"2024-08-03T21:00:51.939823+0000","last_peered":"2024-08-03T21:00:51.939823+0000","last_clean":"2024-08-03T21:00:51.939823+0000","last_became_active":"2024-08-03T21:00:50.222779+0000","last_became_peered":"2024-08-03T21:00:50.222779+0000","last_unstale":"2024-08-03T21:00:51.939823+0000","last_undegraded":"2024-08-03T21:00:51.939823+0000","last_fullsized":"2024-08-03T21:00:51.939823+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_clean_scrub_stamp":"2024-08-03T21:00:49.195479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T06:34:19.632267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.36","version":"0'0","reported_seq":13,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-08-03T21:00:51.939105+0000","last_change":"2024-08-03T21:00:50.238935+0000","last_active":"2024-08-03T21:00:51.939105+0000","last_peered":"2024-08-03T21:00:51.939105+0000","last_clean":"2024-08-03T21:00:51.939105+0000","last_became_active":"2024-08-03T21:00:50.238549+0000","last_became_peered":"2024-08-03T21:00:50.238549+0000","last_unstale":"2024-08-03T21:00:51.939105+0000","last_undegraded":"2024-08-03T21:00:51.939105+0000","last_fullsized":"2024-08-03T21:00:51.939105+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_clean_scrub_stamp":"2024-08-03T21:00:49.195479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T00:17:50.223702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.35","version":"0'0","reported_seq":13,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-08-03T21:00:51.226808+0000","last_change":"2024-08-03T21:00:50.229274+0000","last_active":"2024-08-03T21:00:51.226808+0000","last_peered":"2024-08-03T21:00:51.226808+0000","last_clean":"2024-08-03T21:00:51.226808+0000","last_became_active":"2024-08-03T21:00:50.228900+0000","last_became_peered":"2024-08-03T21:00:50.228900+0000","last_unstale":"2024-08-03T21:00:51.226808+0000","last_undegraded":"2024-08-03T21:00:51.226808+0000","last_fullsized":"2024-08-03T21:00:51.226808+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_clean_scrub_stamp":"2024-08-03T21:00:49.195479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T04:02:53.255557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.34","version":"0'0","reported_seq":13,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-08-03T21:00:51.939323+0000","last_change":"2024-08-03T21:00:50.229025+0000","last_active":"2024-08-03T21:00:51.939323+0000","last_peered":"2024-08-03T21:00:51.939323+0000","last_clean":"2024-08-03T21:00:51.939323+0000","last_became_active":"2024-08-03T21:00:50.228698+0000","last_became_peered":"2024-08-03T21:00:50.228698+0000","last_unstale":"2024-08-03T21:00:51.939323+0000","last_undegraded":"2024-08-03T21:00:51.939323+0000","last_fullsized":"2024-08-03T21:00:51.939323+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_clean_scrub_stamp":"2024-08-03T21:00:49.195479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T06:51:09.079780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.33","version":"0'0","reported_seq":13,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-08-03T21:00:51.225611+0000","last_change":"2024-08-03T21:00:50.224093+0000","last_active":"2024-08-03T21:00:51.225611+0000","last_peered":"2024-08-03T21:00:51.225611+0000","last_clean":"2024-08-03T21:00:51.225611+0000","last_became_active":"2024-08-03T21:00:50.222966+0000","last_became_peered":"2024-08-03T21:00:50.222966+0000","last_unstale":"2024-08-03T21:00:51.225611+0000","last_undegraded":"2024-08-03T21:00:51.225611+0000","last_fullsized":"2024-08-03T21:00:51.225611+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_clean_scrub_stamp":"2024-08-03T21:00:49.195479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T04:11:36.038037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.32","version":"0'0","reported_seq":13,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-08-03T21:00:51.937848+0000","last_change":"2024-08-03T21:00:50.230657+0000","last_active":"2024-08-03T21:00:51.937848+0000","last_peered":"2024-08-03T21:00:51.937848+0000","last_clean":"2024-08-03T21:00:51.937848+0000","last_became_active":"2024-08-03T21:00:50.230356+0000","last_became_peered":"2024-08-03T21:00:50.230356+0000","last_unstale":"2024-08-03T21:00:51.937848+0000","last_undegraded":"2024-08-03T21:00:51.937848+0000","last_fullsized":"2024-08-03T21:00:51.937848+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_clean_scrub_stamp":"2024-08-03T21:00:49.195479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-04T21:20:29.060152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.31","version":"0'0","reported_seq":13,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-08-03T21:00:51.225596+0000","last_change":"2024-08-03T21:00:50.236123+0000","last_active":"2024-08-03T21:00:51.225596+0000","last_peered":"2024-08-03T21:00:51.225596+0000","last_clean":"2024-08-03T21:00:51.225596+0000","last_became_active":"2024-08-03T21:00:50.235747+0000","last_became_peered":"2024-08-03T21:00:50.235747+0000","last_unstale":"2024-08-03T21:00:51.225596+0000","last_undegraded":"2024-08-03T21:00:51.225596+0000","last_fullsized":"2024-08-03T21:00:51.225596+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_clean_scrub_stamp":"2024-08-03T21:00:49.195479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T05:25:30.476710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.30","version":"0'0","reported_seq":13,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-08-03T21:00:51.226256+0000","last_change":"2024-08-03T21:00:50.242724+0000","last_active":"2024-08-03T21:00:51.226256+0000","last_peered":"2024-08-03T21:00:51.226256+0000","last_clean":"2024-08-03T21:00:51.226256+0000","last_became_active":"2024-08-03T21:00:50.239291+0000","last_became_peered":"2024-08-03T21:00:50.239291+0000","last_unstale":"2024-08-03T21:00:51.226256+0000","last_undegraded":"2024-08-03T21:00:51.226256+0000","last_fullsized":"2024-08-03T21:00:51.226256+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_clean_scrub_stamp":"2024-08-03T21:00:49.195479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T04:09:17.948462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.2f","version":"0'0","reported_seq":13,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-08-03T21:00:51.939095+0000","last_change":"2024-08-03T21:00:50.244406+0000","last_active":"2024-08-03T21:00:51.939095+0000","last_peered":"2024-08-03T21:00:51.939095+0000","last_clean":"2024-08-03T21:00:51.939095+0000","last_became_active":"2024-08-03T21:00:50.239150+0000","last_became_peered":"2024-08-03T21:00:50.239150+0000","last_unstale":"2024-08-03T21:00:51.939095+0000","last_undegraded":"2024-08-03T21:00:51.939095+0000","last_fullsized":"2024-08-03T21:00:51.939095+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_clean_scrub_stamp":"2024-08-03T21:00:49.195479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T02:24:57.063186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.2e","version":"0'0","reported_seq":13,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-08-03T21:00:51.225830+0000","last_change":"2024-08-03T21:00:50.241634+0000","last_active":"2024-08-03T21:00:51.225830+0000","last_peered":"2024-08-03T21:00:51.225830+0000","last_clean":"2024-08-03T21:00:51.225830+0000","last_became_active":"2024-08-03T21:00:50.241263+0000","last_became_peered":"2024-08-03T21:00:50.241263+0000","last_unstale":"2024-08-03T21:00:51.225830+0000","last_undegraded":"2024-08-03T21:00:51.225830+0000","last_fullsized":"2024-08-03T21:00:51.225830+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_clean_scrub_stamp":"2024-08-03T21:00:49.195479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T02:07:46.123334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.2d","version":"0'0","reported_seq":13,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-08-03T21:00:51.938842+0000","last_change":"2024-08-03T21:00:50.243910+0000","last_active":"2024-08-03T21:00:51.938842+0000","last_peered":"2024-08-03T21:00:51.938842+0000","last_clean":"2024-08-03T21:00:51.938842+0000","last_became_active":"2024-08-03T21:00:50.243054+0000","last_became_peered":"2024-08-03T21:00:50.243054+0000","last_unstale":"2024-08-03T21:00:51.938842+0000","last_undegraded":"2024-08-03T21:00:51.938842+0000","last_fullsized":"2024-08-03T21:00:51.938842+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_clean_scrub_stamp":"2024-08-03T21:00:49.195479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T08:31:27.530602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.2c","version":"0'0","reported_seq":13,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-08-03T21:00:51.939427+0000","last_change":"2024-08-03T21:00:50.223678+0000","last_active":"2024-08-03T21:00:51.939427+0000","last_peered":"2024-08-03T21:00:51.939427+0000","last_clean":"2024-08-03T21:00:51.939427+0000","last_became_active":"2024-08-03T21:00:50.223357+0000","last_became_peered":"2024-08-03T21:00:50.223357+0000","last_unstale":"2024-08-03T21:00:51.939427+0000","last_undegraded":"2024-08-03T21:00:51.939427+0000","last_fullsized":"2024-08-03T21:00:51.939427+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_clean_scrub_stamp":"2024-08-03T21:00:49.195479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T07:48:35.449963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.2b","version":"0'0","reported_seq":13,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-08-03T21:00:51.939465+0000","last_change":"2024-08-03T21:00:50.243413+0000","last_active":"2024-08-03T21:00:51.939465+0000","last_peered":"2024-08-03T21:00:51.939465+0000","last_clean":"2024-08-03T21:00:51.939465+0000","last_became_active":"2024-08-03T21:00:50.243004+0000","last_became_peered":"2024-08-03T21:00:50.243004+0000","last_unstale":"2024-08-03T21:00:51.939465+0000","last_undegraded":"2024-08-03T21:00:51.939465+0000","last_fullsized":"2024-08-03T21:00:51.939465+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_clean_scrub_stamp":"2024-08-03T21:00:49.195479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-04T23:21:29.132988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.2a","version":"0'0","reported_seq":13,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-08-03T21:00:51.938006+0000","last_change":"2024-08-03T21:00:50.241385+0000","last_active":"2024-08-03T21:00:51.938006+0000","last_peered":"2024-08-03T21:00:51.938006+0000","last_clean":"2024-08-03T21:00:51.938006+0000","last_became_active":"2024-08-03T21:00:50.241070+0000","last_became_peered":"2024-08-03T21:00:50.241070+0000","last_unstale":"2024-08-03T21:00:51.938006+0000","last_undegraded":"2024-08-03T21:00:51.938006+0000","last_fullsized":"2024-08-03T21:00:51.938006+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_clean_scrub_stamp":"2024-08-03T21:00:49.195479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T05:41:07.172696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.29","version":"0'0","reported_seq":13,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-08-03T21:00:51.937830+0000","last_change":"2024-08-03T21:00:50.248216+0000","last_active":"2024-08-03T21:00:51.937830+0000","last_peered":"2024-08-03T21:00:51.937830+0000","last_clean":"2024-08-03T21:00:51.937830+0000","last_became_active":"2024-08-03T21:00:50.247907+0000","last_became_peered":"2024-08-03T21:00:50.247907+0000","last_unstale":"2024-08-03T21:00:51.937830+0000","last_undegraded":"2024-08-03T21:00:51.937830+0000","last_fullsized":"2024-08-03T21:00:51.937830+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_clean_scrub_stamp":"2024-08-03T21:00:49.195479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T02:13:04.485780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.28","version":"0'0","reported_seq":13,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-08-03T21:00:51.226317+0000","last_change":"2024-08-03T21:00:50.243021+0000","last_active":"2024-08-03T21:00:51.226317+0000","last_peered":"2024-08-03T21:00:51.226317+0000","last_clean":"2024-08-03T21:00:51.226317+0000","last_became_active":"2024-08-03T21:00:50.242046+0000","last_became_peered":"2024-08-03T21:00:50.242046+0000","last_unstale":"2024-08-03T21:00:51.226317+0000","last_undegraded":"2024-08-03T21:00:51.226317+0000","last_fullsized":"2024-08-03T21:00:51.226317+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_clean_scrub_stamp":"2024-08-03T21:00:49.195479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-04T23:18:44.490549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.27","version":"0'0","reported_seq":13,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-08-03T21:00:51.939134+0000","last_change":"2024-08-03T21:00:50.247527+0000","last_active":"2024-08-03T21:00:51.939134+0000","last_peered":"2024-08-03T21:00:51.939134+0000","last_clean":"2024-08-03T21:00:51.939134+0000","last_became_active":"2024-08-03T21:00:50.247153+0000","last_became_peered":"2024-08-03T21:00:50.247153+0000","last_unstale":"2024-08-03T21:00:51.939134+0000","last_undegraded":"2024-08-03T21:00:51.939134+0000","last_fullsized":"2024-08-03T21:00:51.939134+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_clean_scrub_stamp":"2024-08-03T21:00:49.195479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T04:09:17.948462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.26","version":"0'0","reported_seq":13,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-08-03T21:00:51.939190+0000","last_change":"2024-08-03T21:00:50.245129+0000","last_active":"2024-08-03T21:00:51.939190+0000","last_peered":"2024-08-03T21:00:51.939190+0000","last_clean":"2024-08-03T21:00:51.939190+0000","last_became_active":"2024-08-03T21:00:50.244410+0000","last_became_peered":"2024-08-03T21:00:50.244410+0000","last_unstale":"2024-08-03T21:00:51.939190+0000","last_undegraded":"2024-08-03T21:00:51.939190+0000","last_fullsized":"2024-08-03T21:00:51.939190+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_clean_scrub_stamp":"2024-08-03T21:00:49.195479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T08:18:17.429255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.25","version":"0'0","reported_seq":13,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-08-03T21:00:51.226949+0000","last_change":"2024-08-03T21:00:50.237747+0000","last_active":"2024-08-03T21:00:51.226949+0000","last_peered":"2024-08-03T21:00:51.226949+0000","last_clean":"2024-08-03T21:00:51.226949+0000","last_became_active":"2024-08-03T21:00:50.237417+0000","last_became_peered":"2024-08-03T21:00:50.237417+0000","last_unstale":"2024-08-03T21:00:51.226949+0000","last_undegraded":"2024-08-03T21:00:51.226949+0000","last_fullsized":"2024-08-03T21:00:51.226949+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_clean_scrub_stamp":"2024-08-03T21:00:49.195479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T02:13:04.485780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.24","version":"0'0","reported_seq":13,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-08-03T21:00:51.939440+0000","last_change":"2024-08-03T21:00:50.225198+0000","last_active":"2024-08-03T21:00:51.939440+0000","last_peered":"2024-08-03T21:00:51.939440+0000","last_clean":"2024-08-03T21:00:51.939440+0000","last_became_active":"2024-08-03T21:00:50.221390+0000","last_became_peered":"2024-08-03T21:00:50.221390+0000","last_unstale":"2024-08-03T21:00:51.939440+0000","last_undegraded":"2024-08-03T21:00:51.939440+0000","last_fullsized":"2024-08-03T21:00:51.939440+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_clean_scrub_stamp":"2024-08-03T21:00:49.195479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-04T21:28:54.164554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.23","version":"0'0","reported_seq":13,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-08-03T21:00:51.938541+0000","last_change":"2024-08-03T21:00:50.238959+0000","last_active":"2024-08-03T21:00:51.938541+0000","last_peered":"2024-08-03T21:00:51.938541+0000","last_clean":"2024-08-03T21:00:51.938541+0000","last_became_active":"2024-08-03T21:00:50.238586+0000","last_became_peered":"2024-08-03T21:00:50.238586+0000","last_unstale":"2024-08-03T21:00:51.938541+0000","last_undegraded":"2024-08-03T21:00:51.938541+0000","last_fullsized":"2024-08-03T21:00:51.938541+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_clean_scrub_stamp":"2024-08-03T21:00:49.195479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T03:49:21.574825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.22","version":"0'0","reported_seq":13,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-08-03T21:00:51.938224+0000","last_change":"2024-08-03T21:00:50.243735+0000","last_active":"2024-08-03T21:00:51.938224+0000","last_peered":"2024-08-03T21:00:51.938224+0000","last_clean":"2024-08-03T21:00:51.938224+0000","last_became_active":"2024-08-03T21:00:50.243333+0000","last_became_peered":"2024-08-03T21:00:50.243333+0000","last_unstale":"2024-08-03T21:00:51.938224+0000","last_undegraded":"2024-08-03T21:00:51.938224+0000","last_fullsized":"2024-08-03T21:00:51.938224+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_clean_scrub_stamp":"2024-08-03T21:00:49.195479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T08:50:30.366065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.21","version":"0'0","reported_seq":13,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-08-03T21:00:51.225546+0000","last_change":"2024-08-03T21:00:50.229431+0000","last_active":"2024-08-03T21:00:51.225546+0000","last_peered":"2024-08-03T21:00:51.225546+0000","last_clean":"2024-08-03T21:00:51.225546+0000","last_became_active":"2024-08-03T21:00:50.228793+0000","last_became_peered":"2024-08-03T21:00:50.228793+0000","last_unstale":"2024-08-03T21:00:51.225546+0000","last_undegraded":"2024-08-03T21:00:51.225546+0000","last_fullsized":"2024-08-03T21:00:51.225546+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_clean_scrub_stamp":"2024-08-03T21:00:49.195479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T03:21:17.046927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.20","version":"0'0","reported_seq":13,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-08-03T21:00:51.939588+0000","last_change":"2024-08-03T21:00:50.242219+0000","last_active":"2024-08-03T21:00:51.939588+0000","last_peered":"2024-08-03T21:00:51.939588+0000","last_clean":"2024-08-03T21:00:51.939588+0000","last_became_active":"2024-08-03T21:00:50.241899+0000","last_became_peered":"2024-08-03T21:00:50.241899+0000","last_unstale":"2024-08-03T21:00:51.939588+0000","last_undegraded":"2024-08-03T21:00:51.939588+0000","last_fullsized":"2024-08-03T21:00:51.939588+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_clean_scrub_stamp":"2024-08-03T21:00:49.195479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T02:44:19.439200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.1e","version":"0'0","reported_seq":13,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-08-03T21:00:51.939274+0000","last_change":"2024-08-03T21:00:50.235759+0000","last_active":"2024-08-03T21:00:51.939274+0000","last_peered":"2024-08-03T21:00:51.939274+0000","last_clean":"2024-08-03T21:00:51.939274+0000","last_became_active":"2024-08-03T21:00:50.235440+0000","last_became_peered":"2024-08-03T21:00:50.235440+0000","last_unstale":"2024-08-03T21:00:51.939274+0000","last_undegraded":"2024-08-03T21:00:51.939274+0000","last_fullsized":"2024-08-03T21:00:51.939274+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_clean_scrub_stamp":"2024-08-03T21:00:49.195479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-04T21:51:33.128276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.1f","version":"134'4","reported_seq":19,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-08-03T21:00:51.939213+0000","last_change":"2024-08-03T21:00:49.231401+0000","last_active":"2024-08-03T21:00:51.939213+0000","last_peered":"2024-08-03T21:00:51.939213+0000","last_clean":"2024-08-03T21:00:51.939213+0000","last_became_active":"2024-08-03T21:00:49.231089+0000","last_became_peered":"2024-08-03T21:00:49.231089+0000","last_unstale":"2024-08-03T21:00:51.939213+0000","last_undegraded":"2024-08-03T21:00:51.939213+0000","last_fullsized":"2024-08-03T21:00:51.939213+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:48.192422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:48.192422+0000","last_clean_scrub_stamp":"2024-08-03T21:00:48.192422+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T03:51:00.885909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.1f","version":"0'0","reported_seq":13,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-08-03T21:00:51.225857+0000","last_change":"2024-08-03T21:00:50.226490+0000","last_active":"2024-08-03T21:00:51.225857+0000","last_peered":"2024-08-03T21:00:51.225857+0000","last_clean":"2024-08-03T21:00:51.225857+0000","last_became_active":"2024-08-03T21:00:50.226103+0000","last_became_peered":"2024-08-03T21:00:50.226103+0000","last_unstale":"2024-08-03T21:00:51.225857+0000","last_undegraded":"2024-08-03T21:00:51.225857+0000","last_fullsized":"2024-08-03T21:00:51.225857+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_clean_scrub_stamp":"2024-08-03T21:00:49.195479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T04:51:31.368175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.1e","version":"134'1","reported_seq":16,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-08-03T21:00:51.226663+0000","last_change":"2024-08-03T21:00:49.227358+0000","last_active":"2024-08-03T21:00:51.226663+0000","last_peered":"2024-08-03T21:00:51.226663+0000","last_clean":"2024-08-03T21:00:51.226663+0000","last_became_active":"2024-08-03T21:00:49.226904+0000","last_became_peered":"2024-08-03T21:00:49.226904+0000","last_unstale":"2024-08-03T21:00:51.226663+0000","last_undegraded":"2024-08-03T21:00:51.226663+0000","last_fullsized":"2024-08-03T21:00:51.226663+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:48.192422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:48.192422+0000","last_clean_scrub_stamp":"2024-08-03T21:00:48.192422+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T05:45:41.794416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.1c","version":"0'0","reported_seq":13,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-08-03T21:00:51.937638+0000","last_change":"2024-08-03T21:00:50.236287+0000","last_active":"2024-08-03T21:00:51.937638+0000","last_peered":"2024-08-03T21:00:51.937638+0000","last_clean":"2024-08-03T21:00:51.937638+0000","last_became_active":"2024-08-03T21:00:50.234964+0000","last_became_peered":"2024-08-03T21:00:50.234964+0000","last_unstale":"2024-08-03T21:00:51.937638+0000","last_undegraded":"2024-08-03T21:00:51.937638+0000","last_fullsized":"2024-08-03T21:00:51.937638+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_clean_scrub_stamp":"2024-08-03T21:00:49.195479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T07:20:57.267245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.1d","version":"134'2","reported_seq":17,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-08-03T21:00:51.938671+0000","last_change":"2024-08-03T21:00:49.234514+0000","last_active":"2024-08-03T21:00:51.938671+0000","last_peered":"2024-08-03T21:00:51.938671+0000","last_clean":"2024-08-03T21:00:51.938671+0000","last_became_active":"2024-08-03T21:00:49.233091+0000","last_became_peered":"2024-08-03T21:00:49.233091+0000","last_unstale":"2024-08-03T21:00:51.938671+0000","last_undegraded":"2024-08-03T21:00:51.938671+0000","last_fullsized":"2024-08-03T21:00:51.938671+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:48.192422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:48.192422+0000","last_clean_scrub_stamp":"2024-08-03T21:00:48.192422+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T00:43:49.078349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.1d","version":"0'0","reported_seq":13,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-08-03T21:00:51.939823+0000","last_change":"2024-08-03T21:00:50.235731+0000","last_active":"2024-08-03T21:00:51.939823+0000","last_peered":"2024-08-03T21:00:51.939823+0000","last_clean":"2024-08-03T21:00:51.939823+0000","last_became_active":"2024-08-03T21:00:50.234805+0000","last_became_peered":"2024-08-03T21:00:50.234805+0000","last_unstale":"2024-08-03T21:00:51.939823+0000","last_undegraded":"2024-08-03T21:00:51.939823+0000","last_fullsized":"2024-08-03T21:00:51.939823+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_clean_scrub_stamp":"2024-08-03T21:00:49.195479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T05:40:57.488842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.1c","version":"134'2","reported_seq":17,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-08-03T21:00:51.938470+0000","last_change":"2024-08-03T21:00:49.227739+0000","last_active":"2024-08-03T21:00:51.938470+0000","last_peered":"2024-08-03T21:00:51.938470+0000","last_clean":"2024-08-03T21:00:51.938470+0000","last_became_active":"2024-08-03T21:00:49.226569+0000","last_became_peered":"2024-08-03T21:00:49.226569+0000","last_unstale":"2024-08-03T21:00:51.938470+0000","last_undegraded":"2024-08-03T21:00:51.938470+0000","last_fullsized":"2024-08-03T21:00:51.938470+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:48.192422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:48.192422+0000","last_clean_scrub_stamp":"2024-08-03T21:00:48.192422+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-04T22:46:01.864033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.1a","version":"0'0","reported_seq":13,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-08-03T21:00:51.940112+0000","last_change":"2024-08-03T21:00:50.230345+0000","last_active":"2024-08-03T21:00:51.940112+0000","last_peered":"2024-08-03T21:00:51.940112+0000","last_clean":"2024-08-03T21:00:51.940112+0000","last_became_active":"2024-08-03T21:00:50.230001+0000","last_became_peered":"2024-08-03T21:00:50.230001+0000","last_unstale":"2024-08-03T21:00:51.940112+0000","last_undegraded":"2024-08-03T21:00:51.940112+0000","last_fullsized":"2024-08-03T21:00:51.940112+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_clean_scrub_stamp":"2024-08-03T21:00:49.195479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-04T23:53:42.243847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.1b","version":"134'1","reported_seq":16,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-08-03T21:00:51.938855+0000","last_change":"2024-08-03T21:00:49.217597+0000","last_active":"2024-08-03T21:00:51.938855+0000","last_peered":"2024-08-03T21:00:51.938855+0000","last_clean":"2024-08-03T21:00:51.938855+0000","last_became_active":"2024-08-03T21:00:49.217212+0000","last_became_peered":"2024-08-03T21:00:49.217212+0000","last_unstale":"2024-08-03T21:00:51.938855+0000","last_undegraded":"2024-08-03T21:00:51.938855+0000","last_fullsized":"2024-08-03T21:00:51.938855+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:48.192422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:48.192422+0000","last_clean_scrub_stamp":"2024-08-03T21:00:48.192422+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T00:07:15.964072+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.1b","version":"0'0","reported_seq":13,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-08-03T21:00:51.226661+0000","last_change":"2024-08-03T21:00:50.225540+0000","last_active":"2024-08-03T21:00:51.226661+0000","last_peered":"2024-08-03T21:00:51.226661+0000","last_clean":"2024-08-03T21:00:51.226661+0000","last_became_active":"2024-08-03T21:00:50.225186+0000","last_became_peered":"2024-08-03T21:00:50.225186+0000","last_unstale":"2024-08-03T21:00:51.226661+0000","last_undegraded":"2024-08-03T21:00:51.226661+0000","last_fullsized":"2024-08-03T21:00:51.226661+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_clean_scrub_stamp":"2024-08-03T21:00:49.195479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T02:52:49.825428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.1a","version":"134'1","reported_seq":16,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-08-03T21:00:51.938252+0000","last_change":"2024-08-03T21:00:49.238475+0000","last_active":"2024-08-03T21:00:51.938252+0000","last_peered":"2024-08-03T21:00:51.938252+0000","last_clean":"2024-08-03T21:00:51.938252+0000","last_became_active":"2024-08-03T21:00:49.238001+0000","last_became_peered":"2024-08-03T21:00:49.238001+0000","last_unstale":"2024-08-03T21:00:51.938252+0000","last_undegraded":"2024-08-03T21:00:51.938252+0000","last_fullsized":"2024-08-03T21:00:51.938252+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:48.192422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:48.192422+0000","last_clean_scrub_stamp":"2024-08-03T21:00:48.192422+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-04T23:43:28.912246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.18","version":"0'0","reported_seq":13,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-08-03T21:00:51.225613+0000","last_change":"2024-08-03T21:00:50.228305+0000","last_active":"2024-08-03T21:00:51.225613+0000","last_peered":"2024-08-03T21:00:51.225613+0000","last_clean":"2024-08-03T21:00:51.225613+0000","last_became_active":"2024-08-03T21:00:50.228009+0000","last_became_peered":"2024-08-03T21:00:50.228009+0000","last_unstale":"2024-08-03T21:00:51.225613+0000","last_undegraded":"2024-08-03T21:00:51.225613+0000","last_fullsized":"2024-08-03T21:00:51.225613+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_clean_scrub_stamp":"2024-08-03T21:00:49.195479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-04T23:22:25.975528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.19","version":"0'0","reported_seq":15,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-08-03T21:00:51.938739+0000","last_change":"2024-08-03T21:00:49.228986+0000","last_active":"2024-08-03T21:00:51.938739+0000","last_peered":"2024-08-03T21:00:51.938739+0000","last_clean":"2024-08-03T21:00:51.938739+0000","last_became_active":"2024-08-03T21:00:49.228761+0000","last_became_peered":"2024-08-03T21:00:49.228761+0000","last_unstale":"2024-08-03T21:00:51.938739+0000","last_undegraded":"2024-08-03T21:00:51.938739+0000","last_fullsized":"2024-08-03T21:00:51.938739+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:48.192422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:48.192422+0000","last_clean_scrub_stamp":"2024-08-03T21:00:48.192422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-04T23:53:41.240790+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.19","version":"0'0","reported_seq":13,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-08-03T21:00:51.940174+0000","last_change":"2024-08-03T21:00:50.235335+0000","last_active":"2024-08-03T21:00:51.940174+0000","last_peered":"2024-08-03T21:00:51.940174+0000","last_clean":"2024-08-03T21:00:51.940174+0000","last_became_active":"2024-08-03T21:00:50.234940+0000","last_became_peered":"2024-08-03T21:00:50.234940+0000","last_unstale":"2024-08-03T21:00:51.940174+0000","last_undegraded":"2024-08-03T21:00:51.940174+0000","last_fullsized":"2024-08-03T21:00:51.940174+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_clean_scrub_stamp":"2024-08-03T21:00:49.195479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T00:40:44.403536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.18","version":"0'0","reported_seq":15,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-08-03T21:00:51.937778+0000","last_change":"2024-08-03T21:00:49.236558+0000","last_active":"2024-08-03T21:00:51.937778+0000","last_peered":"2024-08-03T21:00:51.937778+0000","last_clean":"2024-08-03T21:00:51.937778+0000","last_became_active":"2024-08-03T21:00:49.236201+0000","last_became_peered":"2024-08-03T21:00:49.236201+0000","last_unstale":"2024-08-03T21:00:51.937778+0000","last_undegraded":"2024-08-03T21:00:51.937778+0000","last_fullsized":"2024-08-03T21:00:51.937778+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:48.192422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:48.192422+0000","last_clean_scrub_stamp":"2024-08-03T21:00:48.192422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T08:11:10.436293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.16","version":"0'0","reported_seq":13,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-08-03T21:00:51.939374+0000","last_change":"2024-08-03T21:00:50.244082+0000","last_active":"2024-08-03T21:00:51.939374+0000","last_peered":"2024-08-03T21:00:51.939374+0000","last_clean":"2024-08-03T21:00:51.939374+0000","last_became_active":"2024-08-03T21:00:50.239536+0000","last_became_peered":"2024-08-03T21:00:50.239536+0000","last_unstale":"2024-08-03T21:00:51.939374+0000","last_undegraded":"2024-08-03T21:00:51.939374+0000","last_fullsized":"2024-08-03T21:00:51.939374+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_clean_scrub_stamp":"2024-08-03T21:00:49.195479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T06:38:59.823707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.17","version":"0'0","reported_seq":15,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-08-03T21:00:51.939140+0000","last_change":"2024-08-03T21:00:49.226248+0000","last_active":"2024-08-03T21:00:51.939140+0000","last_peered":"2024-08-03T21:00:51.939140+0000","last_clean":"2024-08-03T21:00:51.939140+0000","last_became_active":"2024-08-03T21:00:49.224012+0000","last_became_peered":"2024-08-03T21:00:49.224012+0000","last_unstale":"2024-08-03T21:00:51.939140+0000","last_undegraded":"2024-08-03T21:00:51.939140+0000","last_fullsized":"2024-08-03T21:00:51.939140+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:48.192422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:48.192422+0000","last_clean_scrub_stamp":"2024-08-03T21:00:48.192422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T01:08:20.166085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.17","version":"0'0","reported_seq":13,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-08-03T21:00:51.940036+0000","last_change":"2024-08-03T21:00:50.244964+0000","last_active":"2024-08-03T21:00:51.940036+0000","last_peered":"2024-08-03T21:00:51.940036+0000","last_clean":"2024-08-03T21:00:51.940036+0000","last_became_active":"2024-08-03T21:00:50.244323+0000","last_became_peered":"2024-08-03T21:00:50.244323+0000","last_unstale":"2024-08-03T21:00:51.940036+0000","last_undegraded":"2024-08-03T21:00:51.940036+0000","last_fullsized":"2024-08-03T21:00:51.940036+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_clean_scrub_stamp":"2024-08-03T21:00:49.195479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-04T21:46:43.719103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.16","version":"0'0","reported_seq":15,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-08-03T21:00:51.225728+0000","last_change":"2024-08-03T21:00:49.233357+0000","last_active":"2024-08-03T21:00:51.225728+0000","last_peered":"2024-08-03T21:00:51.225728+0000","last_clean":"2024-08-03T21:00:51.225728+0000","last_became_active":"2024-08-03T21:00:49.232477+0000","last_became_peered":"2024-08-03T21:00:49.232477+0000","last_unstale":"2024-08-03T21:00:51.225728+0000","last_undegraded":"2024-08-03T21:00:51.225728+0000","last_fullsized":"2024-08-03T21:00:51.225728+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:48.192422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:48.192422+0000","last_clean_scrub_stamp":"2024-08-03T21:00:48.192422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T05:40:56.485784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.14","version":"0'0","reported_seq":13,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-08-03T21:00:51.939653+0000","last_change":"2024-08-03T21:00:50.247566+0000","last_active":"2024-08-03T21:00:51.939653+0000","last_peered":"2024-08-03T21:00:51.939653+0000","last_clean":"2024-08-03T21:00:51.939653+0000","last_became_active":"2024-08-03T21:00:50.247175+0000","last_became_peered":"2024-08-03T21:00:50.247175+0000","last_unstale":"2024-08-03T21:00:51.939653+0000","last_undegraded":"2024-08-03T21:00:51.939653+0000","last_fullsized":"2024-08-03T21:00:51.939653+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_clean_scrub_stamp":"2024-08-03T21:00:49.195479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T03:52:56.414014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.15","version":"134'3","reported_seq":18,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-08-03T21:00:51.227301+0000","last_change":"2024-08-03T21:00:49.224739+0000","last_active":"2024-08-03T21:00:51.227301+0000","last_peered":"2024-08-03T21:00:51.227301+0000","last_clean":"2024-08-03T21:00:51.227301+0000","last_became_active":"2024-08-03T21:00:49.224342+0000","last_became_peered":"2024-08-03T21:00:49.224342+0000","last_unstale":"2024-08-03T21:00:51.227301+0000","last_undegraded":"2024-08-03T21:00:51.227301+0000","last_fullsized":"2024-08-03T21:00:51.227301+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:48.192422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:48.192422+0000","last_clean_scrub_stamp":"2024-08-03T21:00:48.192422+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T06:55:23.113259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.15","version":"0'0","reported_seq":13,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-08-03T21:00:51.227168+0000","last_change":"2024-08-03T21:00:50.236337+0000","last_active":"2024-08-03T21:00:51.227168+0000","last_peered":"2024-08-03T21:00:51.227168+0000","last_clean":"2024-08-03T21:00:51.227168+0000","last_became_active":"2024-08-03T21:00:50.235968+0000","last_became_peered":"2024-08-03T21:00:50.235968+0000","last_unstale":"2024-08-03T21:00:51.227168+0000","last_undegraded":"2024-08-03T21:00:51.227168+0000","last_fullsized":"2024-08-03T21:00:51.227168+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_clean_scrub_stamp":"2024-08-03T21:00:49.195479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T05:41:07.172696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.14","version":"134'6","reported_seq":21,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-08-03T21:00:51.939402+0000","last_change":"2024-08-03T21:00:49.235410+0000","last_active":"2024-08-03T21:00:51.939402+0000","last_peered":"2024-08-03T21:00:51.939402+0000","last_clean":"2024-08-03T21:00:51.939402+0000","last_became_active":"2024-08-03T21:00:49.234999+0000","last_became_peered":"2024-08-03T21:00:49.234999+0000","last_unstale":"2024-08-03T21:00:51.939402+0000","last_undegraded":"2024-08-03T21:00:51.939402+0000","last_fullsized":"2024-08-03T21:00:51.939402+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:48.192422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:48.192422+0000","last_clean_scrub_stamp":"2024-08-03T21:00:48.192422+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T02:55:58.470527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.12","version":"0'0","reported_seq":13,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-08-03T21:00:51.225979+0000","last_change":"2024-08-03T21:00:50.218292+0000","last_active":"2024-08-03T21:00:51.225979+0000","last_peered":"2024-08-03T21:00:51.225979+0000","last_clean":"2024-08-03T21:00:51.225979+0000","last_became_active":"2024-08-03T21:00:50.217979+0000","last_became_peered":"2024-08-03T21:00:50.217979+0000","last_unstale":"2024-08-03T21:00:51.225979+0000","last_undegraded":"2024-08-03T21:00:51.225979+0000","last_fullsized":"2024-08-03T21:00:51.225979+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_clean_scrub_stamp":"2024-08-03T21:00:49.195479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T06:55:48.820648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.13","version":"134'2","reported_seq":17,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-08-03T21:00:51.939709+0000","last_change":"2024-08-03T21:00:49.224867+0000","last_active":"2024-08-03T21:00:51.939709+0000","last_peered":"2024-08-03T21:00:51.939709+0000","last_clean":"2024-08-03T21:00:51.939709+0000","last_became_active":"2024-08-03T21:00:49.224555+0000","last_became_peered":"2024-08-03T21:00:49.224555+0000","last_unstale":"2024-08-03T21:00:51.939709+0000","last_undegraded":"2024-08-03T21:00:51.939709+0000","last_fullsized":"2024-08-03T21:00:51.939709+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:48.192422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:48.192422+0000","last_clean_scrub_stamp":"2024-08-03T21:00:48.192422+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T07:55:25.225889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.13","version":"0'0","reported_seq":13,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-08-03T21:00:51.938720+0000","last_change":"2024-08-03T21:00:50.247701+0000","last_active":"2024-08-03T21:00:51.938720+0000","last_peered":"2024-08-03T21:00:51.938720+0000","last_clean":"2024-08-03T21:00:51.938720+0000","last_became_active":"2024-08-03T21:00:50.247414+0000","last_became_peered":"2024-08-03T21:00:50.247414+0000","last_unstale":"2024-08-03T21:00:51.938720+0000","last_undegraded":"2024-08-03T21:00:51.938720+0000","last_fullsized":"2024-08-03T21:00:51.938720+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_clean_scrub_stamp":"2024-08-03T21:00:49.195479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T05:44:39.085990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.12","version":"0'0","reported_seq":15,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-08-03T21:00:51.938928+0000","last_change":"2024-08-03T21:00:49.224078+0000","last_active":"2024-08-03T21:00:51.938928+0000","last_peered":"2024-08-03T21:00:51.938928+0000","last_clean":"2024-08-03T21:00:51.938928+0000","last_became_active":"2024-08-03T21:00:49.223717+0000","last_became_peered":"2024-08-03T21:00:49.223717+0000","last_unstale":"2024-08-03T21:00:51.938928+0000","last_undegraded":"2024-08-03T21:00:51.938928+0000","last_fullsized":"2024-08-03T21:00:51.938928+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:48.192422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:48.192422+0000","last_clean_scrub_stamp":"2024-08-03T21:00:48.192422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T04:56:22.864880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.10","version":"0'0","reported_seq":13,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-08-03T21:00:51.939036+0000","last_change":"2024-08-03T21:00:50.231767+0000","last_active":"2024-08-03T21:00:51.939036+0000","last_peered":"2024-08-03T21:00:51.939036+0000","last_clean":"2024-08-03T21:00:51.939036+0000","last_became_active":"2024-08-03T21:00:50.229726+0000","last_became_peered":"2024-08-03T21:00:50.229726+0000","last_unstale":"2024-08-03T21:00:51.939036+0000","last_undegraded":"2024-08-03T21:00:51.939036+0000","last_fullsized":"2024-08-03T21:00:51.939036+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_clean_scrub_stamp":"2024-08-03T21:00:49.195479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T00:14:18.409160+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.11","version":"0'0","reported_seq":15,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-08-03T21:00:51.939969+0000","last_change":"2024-08-03T21:00:49.219205+0000","last_active":"2024-08-03T21:00:51.939969+0000","last_peered":"2024-08-03T21:00:51.939969+0000","last_clean":"2024-08-03T21:00:51.939969+0000","last_became_active":"2024-08-03T21:00:49.218855+0000","last_became_peered":"2024-08-03T21:00:49.218855+0000","last_unstale":"2024-08-03T21:00:51.939969+0000","last_undegraded":"2024-08-03T21:00:51.939969+0000","last_fullsized":"2024-08-03T21:00:51.939969+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:48.192422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:48.192422+0000","last_clean_scrub_stamp":"2024-08-03T21:00:48.192422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T00:24:17.141394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.11","version":"0'0","reported_seq":13,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-08-03T21:00:51.939150+0000","last_change":"2024-08-03T21:00:50.230042+0000","last_active":"2024-08-03T21:00:51.939150+0000","last_peered":"2024-08-03T21:00:51.939150+0000","last_clean":"2024-08-03T21:00:51.939150+0000","last_became_active":"2024-08-03T21:00:50.229716+0000","last_became_peered":"2024-08-03T21:00:50.229716+0000","last_unstale":"2024-08-03T21:00:51.939150+0000","last_undegraded":"2024-08-03T21:00:51.939150+0000","last_fullsized":"2024-08-03T21:00:51.939150+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_clean_scrub_stamp":"2024-08-03T21:00:49.195479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T08:05:48.492097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.10","version":"134'8","reported_seq":23,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-08-03T21:00:51.937904+0000","last_change":"2024-08-03T21:00:49.219610+0000","last_active":"2024-08-03T21:00:51.937904+0000","last_peered":"2024-08-03T21:00:51.937904+0000","last_clean":"2024-08-03T21:00:51.937904+0000","last_became_active":"2024-08-03T21:00:49.219289+0000","last_became_peered":"2024-08-03T21:00:49.219289+0000","last_unstale":"2024-08-03T21:00:51.937904+0000","last_undegraded":"2024-08-03T21:00:51.937904+0000","last_fullsized":"2024-08-03T21:00:51.937904+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:48.192422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:48.192422+0000","last_clean_scrub_stamp":"2024-08-03T21:00:48.192422+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T01:32:44.783771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.f","version":"0'0","reported_seq":13,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-08-03T21:00:51.939337+0000","last_change":"2024-08-03T21:00:50.236210+0000","last_active":"2024-08-03T21:00:51.939337+0000","last_peered":"2024-08-03T21:00:51.939337+0000","last_clean":"2024-08-03T21:00:51.939337+0000","last_became_active":"2024-08-03T21:00:50.233503+0000","last_became_peered":"2024-08-03T21:00:50.233503+0000","last_unstale":"2024-08-03T21:00:51.939337+0000","last_undegraded":"2024-08-03T21:00:51.939337+0000","last_fullsized":"2024-08-03T21:00:51.939337+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_clean_scrub_stamp":"2024-08-03T21:00:49.195479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-04T23:18:44.490549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.e","version":"134'2","reported_seq":17,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-08-03T21:00:51.939110+0000","last_change":"2024-08-03T21:00:49.221709+0000","last_active":"2024-08-03T21:00:51.939110+0000","last_peered":"2024-08-03T21:00:51.939110+0000","last_clean":"2024-08-03T21:00:51.939110+0000","last_became_active":"2024-08-03T21:00:49.221146+0000","last_became_peered":"2024-08-03T21:00:49.221146+0000","last_unstale":"2024-08-03T21:00:51.939110+0000","last_undegraded":"2024-08-03T21:00:51.939110+0000","last_fullsized":"2024-08-03T21:00:51.939110+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:48.192422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:48.192422+0000","last_clean_scrub_stamp":"2024-08-03T21:00:48.192422+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T02:29:53.585049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.c","version":"0'0","reported_seq":13,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-08-03T21:00:51.940242+0000","last_change":"2024-08-03T21:00:50.240090+0000","last_active":"2024-08-03T21:00:51.940242+0000","last_peered":"2024-08-03T21:00:51.940242+0000","last_clean":"2024-08-03T21:00:51.940242+0000","last_became_active":"2024-08-03T21:00:50.239718+0000","last_became_peered":"2024-08-03T21:00:50.239718+0000","last_unstale":"2024-08-03T21:00:51.940242+0000","last_undegraded":"2024-08-03T21:00:51.940242+0000","last_fullsized":"2024-08-03T21:00:51.940242+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_clean_scrub_stamp":"2024-08-03T21:00:49.195479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T07:44:14.161399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.d","version":"134'1","reported_seq":16,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-08-03T21:00:51.939184+0000","last_change":"2024-08-03T21:00:49.214725+0000","last_active":"2024-08-03T21:00:51.939184+0000","last_peered":"2024-08-03T21:00:51.939184+0000","last_clean":"2024-08-03T21:00:51.939184+0000","last_became_active":"2024-08-03T21:00:49.214155+0000","last_became_peered":"2024-08-03T21:00:49.214155+0000","last_unstale":"2024-08-03T21:00:51.939184+0000","last_undegraded":"2024-08-03T21:00:51.939184+0000","last_fullsized":"2024-08-03T21:00:51.939184+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:48.192422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:48.192422+0000","last_clean_scrub_stamp":"2024-08-03T21:00:48.192422+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T07:44:13.158341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.d","version":"0'0","reported_seq":13,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-08-03T21:00:51.227187+0000","last_change":"2024-08-03T21:00:50.228406+0000","last_active":"2024-08-03T21:00:51.227187+0000","last_peered":"2024-08-03T21:00:51.227187+0000","last_clean":"2024-08-03T21:00:51.227187+0000","last_became_active":"2024-08-03T21:00:50.227965+0000","last_became_peered":"2024-08-03T21:00:50.227965+0000","last_unstale":"2024-08-03T21:00:51.227187+0000","last_undegraded":"2024-08-03T21:00:51.227187+0000","last_fullsized":"2024-08-03T21:00:51.227187+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_clean_scrub_stamp":"2024-08-03T21:00:49.195479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T03:49:21.574825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.c","version":"0'0","reported_seq":15,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-08-03T21:00:51.938518+0000","last_change":"2024-08-03T21:00:49.225810+0000","last_active":"2024-08-03T21:00:51.938518+0000","last_peered":"2024-08-03T21:00:51.938518+0000","last_clean":"2024-08-03T21:00:51.938518+0000","last_became_active":"2024-08-03T21:00:49.225501+0000","last_became_peered":"2024-08-03T21:00:49.225501+0000","last_unstale":"2024-08-03T21:00:51.938518+0000","last_undegraded":"2024-08-03T21:00:51.938518+0000","last_fullsized":"2024-08-03T21:00:51.938518+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:48.192422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:48.192422+0000","last_clean_scrub_stamp":"2024-08-03T21:00:48.192422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T06:55:23.113259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.a","version":"0'0","reported_seq":13,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-08-03T21:00:51.938230+0000","last_change":"2024-08-03T21:00:50.238896+0000","last_active":"2024-08-03T21:00:51.938230+0000","last_peered":"2024-08-03T21:00:51.938230+0000","last_clean":"2024-08-03T21:00:51.938230+0000","last_became_active":"2024-08-03T21:00:50.238557+0000","last_became_peered":"2024-08-03T21:00:50.238557+0000","last_unstale":"2024-08-03T21:00:51.938230+0000","last_undegraded":"2024-08-03T21:00:51.938230+0000","last_fullsized":"2024-08-03T21:00:51.938230+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_clean_scrub_stamp":"2024-08-03T21:00:49.195479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-04T22:10:38.257913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.b","version":"134'2","reported_seq":17,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-08-03T21:00:51.938675+0000","last_change":"2024-08-03T21:00:49.233291+0000","last_active":"2024-08-03T21:00:51.938675+0000","last_peered":"2024-08-03T21:00:51.938675+0000","last_clean":"2024-08-03T21:00:51.938675+0000","last_became_active":"2024-08-03T21:00:49.232973+0000","last_became_peered":"2024-08-03T21:00:49.232973+0000","last_unstale":"2024-08-03T21:00:51.938675+0000","last_undegraded":"2024-08-03T21:00:51.938675+0000","last_fullsized":"2024-08-03T21:00:51.938675+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:48.192422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:48.192422+0000","last_clean_scrub_stamp":"2024-08-03T21:00:48.192422+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T05:33:33.972534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.b","version":"0'0","reported_seq":13,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-08-03T21:00:51.939647+0000","last_change":"2024-08-03T21:00:50.242588+0000","last_active":"2024-08-03T21:00:51.939647+0000","last_peered":"2024-08-03T21:00:51.939647+0000","last_clean":"2024-08-03T21:00:51.939647+0000","last_became_active":"2024-08-03T21:00:50.239795+0000","last_became_peered":"2024-08-03T21:00:50.239795+0000","last_unstale":"2024-08-03T21:00:51.939647+0000","last_undegraded":"2024-08-03T21:00:51.939647+0000","last_fullsized":"2024-08-03T21:00:51.939647+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_clean_scrub_stamp":"2024-08-03T21:00:49.195479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T06:53:11.234357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.a","version":"0'0","reported_seq":15,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-08-03T21:00:51.940019+0000","last_change":"2024-08-03T21:00:49.222066+0000","last_active":"2024-08-03T21:00:51.940019+0000","last_peered":"2024-08-03T21:00:51.940019+0000","last_clean":"2024-08-03T21:00:51.940019+0000","last_became_active":"2024-08-03T21:00:49.221757+0000","last_became_peered":"2024-08-03T21:00:49.221757+0000","last_unstale":"2024-08-03T21:00:51.940019+0000","last_undegraded":"2024-08-03T21:00:51.940019+0000","last_fullsized":"2024-08-03T21:00:51.940019+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:48.192422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:48.192422+0000","last_clean_scrub_stamp":"2024-08-03T21:00:48.192422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T01:29:44.046851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.8","version":"0'0","reported_seq":13,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-08-03T21:00:51.937406+0000","last_change":"2024-08-03T21:00:50.240625+0000","last_active":"2024-08-03T21:00:51.937406+0000","last_peered":"2024-08-03T21:00:51.937406+0000","last_clean":"2024-08-03T21:00:51.937406+0000","last_became_active":"2024-08-03T21:00:50.240139+0000","last_became_peered":"2024-08-03T21:00:50.240139+0000","last_unstale":"2024-08-03T21:00:51.937406+0000","last_undegraded":"2024-08-03T21:00:51.937406+0000","last_fullsized":"2024-08-03T21:00:51.937406+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_clean_scrub_stamp":"2024-08-03T21:00:49.195479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T06:58:48.625406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.9","version":"0'0","reported_seq":15,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-08-03T21:00:51.938935+0000","last_change":"2024-08-03T21:00:49.235212+0000","last_active":"2024-08-03T21:00:51.938935+0000","last_peered":"2024-08-03T21:00:51.938935+0000","last_clean":"2024-08-03T21:00:51.938935+0000","last_became_active":"2024-08-03T21:00:49.234793+0000","last_became_peered":"2024-08-03T21:00:49.234793+0000","last_unstale":"2024-08-03T21:00:51.938935+0000","last_undegraded":"2024-08-03T21:00:51.938935+0000","last_fullsized":"2024-08-03T21:00:51.938935+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:48.192422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:48.192422+0000","last_clean_scrub_stamp":"2024-08-03T21:00:48.192422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T05:44:38.082933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.9","version":"0'0","reported_seq":13,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-08-03T21:00:51.938543+0000","last_change":"2024-08-03T21:00:50.226039+0000","last_active":"2024-08-03T21:00:51.938543+0000","last_peered":"2024-08-03T21:00:51.938543+0000","last_clean":"2024-08-03T21:00:51.938543+0000","last_became_active":"2024-08-03T21:00:50.224881+0000","last_became_peered":"2024-08-03T21:00:50.224881+0000","last_unstale":"2024-08-03T21:00:51.938543+0000","last_undegraded":"2024-08-03T21:00:51.938543+0000","last_fullsized":"2024-08-03T21:00:51.938543+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_clean_scrub_stamp":"2024-08-03T21:00:49.195479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T03:51:01.888967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.8","version":"0'0","reported_seq":15,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-08-03T21:00:51.937513+0000","last_change":"2024-08-03T21:00:49.230326+0000","last_active":"2024-08-03T21:00:51.937513+0000","last_peered":"2024-08-03T21:00:51.937513+0000","last_clean":"2024-08-03T21:00:51.937513+0000","last_became_active":"2024-08-03T21:00:49.229867+0000","last_became_peered":"2024-08-03T21:00:49.229867+0000","last_unstale":"2024-08-03T21:00:51.937513+0000","last_undegraded":"2024-08-03T21:00:51.937513+0000","last_fullsized":"2024-08-03T21:00:51.937513+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:48.192422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:48.192422+0000","last_clean_scrub_stamp":"2024-08-03T21:00:48.192422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T01:53:59.097881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.6","version":"0'0","reported_seq":13,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-08-03T21:00:51.939434+0000","last_change":"2024-08-03T21:00:50.245369+0000","last_active":"2024-08-03T21:00:51.939434+0000","last_peered":"2024-08-03T21:00:51.939434+0000","last_clean":"2024-08-03T21:00:51.939434+0000","last_became_active":"2024-08-03T21:00:50.244895+0000","last_became_peered":"2024-08-03T21:00:50.244895+0000","last_unstale":"2024-08-03T21:00:51.939434+0000","last_undegraded":"2024-08-03T21:00:51.939434+0000","last_fullsized":"2024-08-03T21:00:51.939434+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_clean_scrub_stamp":"2024-08-03T21:00:49.195479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T08:08:18.583196+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.7","version":"134'2","reported_seq":17,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-08-03T21:00:51.940273+0000","last_change":"2024-08-03T21:00:49.220966+0000","last_active":"2024-08-03T21:00:51.940273+0000","last_peered":"2024-08-03T21:00:51.940273+0000","last_clean":"2024-08-03T21:00:51.940273+0000","last_became_active":"2024-08-03T21:00:49.220433+0000","last_became_peered":"2024-08-03T21:00:49.220433+0000","last_unstale":"2024-08-03T21:00:51.940273+0000","last_undegraded":"2024-08-03T21:00:51.940273+0000","last_fullsized":"2024-08-03T21:00:51.940273+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:48.192422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:48.192422+0000","last_clean_scrub_stamp":"2024-08-03T21:00:48.192422+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T02:44:18.436142+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.7","version":"0'0","reported_seq":13,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-08-03T21:00:51.939167+0000","last_change":"2024-08-03T21:00:50.232102+0000","last_active":"2024-08-03T21:00:51.939167+0000","last_peered":"2024-08-03T21:00:51.939167+0000","last_clean":"2024-08-03T21:00:51.939167+0000","last_became_active":"2024-08-03T21:00:50.231742+0000","last_became_peered":"2024-08-03T21:00:50.231742+0000","last_unstale":"2024-08-03T21:00:51.939167+0000","last_undegraded":"2024-08-03T21:00:51.939167+0000","last_fullsized":"2024-08-03T21:00:51.939167+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_clean_scrub_stamp":"2024-08-03T21:00:49.195479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T01:29:45.049908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.6","version":"135'1","reported_seq":16,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-08-03T21:00:51.939543+0000","last_change":"2024-08-03T21:00:49.235126+0000","last_active":"2024-08-03T21:00:51.939543+0000","last_peered":"2024-08-03T21:00:51.939543+0000","last_clean":"2024-08-03T21:00:51.939543+0000","last_became_active":"2024-08-03T21:00:49.233960+0000","last_became_peered":"2024-08-03T21:00:49.233960+0000","last_unstale":"2024-08-03T21:00:51.939543+0000","last_undegraded":"2024-08-03T21:00:51.939543+0000","last_fullsized":"2024-08-03T21:00:51.939543+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:48.192422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:48.192422+0000","last_clean_scrub_stamp":"2024-08-03T21:00:48.192422+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T06:11:55.068764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.4","version":"0'0","reported_seq":13,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-08-03T21:00:51.937871+0000","last_change":"2024-08-03T21:00:50.237044+0000","last_active":"2024-08-03T21:00:51.937871+0000","last_peered":"2024-08-03T21:00:51.937871+0000","last_clean":"2024-08-03T21:00:51.937871+0000","last_became_active":"2024-08-03T21:00:50.236738+0000","last_became_peered":"2024-08-03T21:00:50.236738+0000","last_unstale":"2024-08-03T21:00:51.937871+0000","last_undegraded":"2024-08-03T21:00:51.937871+0000","last_fullsized":"2024-08-03T21:00:51.937871+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_clean_scrub_stamp":"2024-08-03T21:00:49.195479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T07:43:14.289706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.5","version":"0'0","reported_seq":15,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-08-03T21:00:51.938775+0000","last_change":"2024-08-03T21:00:49.227879+0000","last_active":"2024-08-03T21:00:51.938775+0000","last_peered":"2024-08-03T21:00:51.938775+0000","last_clean":"2024-08-03T21:00:51.938775+0000","last_became_active":"2024-08-03T21:00:49.227574+0000","last_became_peered":"2024-08-03T21:00:49.227574+0000","last_unstale":"2024-08-03T21:00:51.938775+0000","last_undegraded":"2024-08-03T21:00:51.938775+0000","last_fullsized":"2024-08-03T21:00:51.938775+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:48.192422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:48.192422+0000","last_clean_scrub_stamp":"2024-08-03T21:00:48.192422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T08:56:28.307290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.5","version":"0'0","reported_seq":13,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-08-03T21:00:51.938535+0000","last_change":"2024-08-03T21:00:50.228194+0000","last_active":"2024-08-03T21:00:51.938535+0000","last_peered":"2024-08-03T21:00:51.938535+0000","last_clean":"2024-08-03T21:00:51.938535+0000","last_became_active":"2024-08-03T21:00:50.226123+0000","last_became_peered":"2024-08-03T21:00:50.226123+0000","last_unstale":"2024-08-03T21:00:51.938535+0000","last_undegraded":"2024-08-03T21:00:51.938535+0000","last_fullsized":"2024-08-03T21:00:51.938535+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_clean_scrub_stamp":"2024-08-03T21:00:49.195479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T08:22:58.480315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.4","version":"0'0","reported_seq":15,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-08-03T21:00:51.939696+0000","last_change":"2024-08-03T21:00:49.221928+0000","last_active":"2024-08-03T21:00:51.939696+0000","last_peered":"2024-08-03T21:00:51.939696+0000","last_clean":"2024-08-03T21:00:51.939696+0000","last_became_active":"2024-08-03T21:00:49.221623+0000","last_became_peered":"2024-08-03T21:00:49.221623+0000","last_unstale":"2024-08-03T21:00:51.939696+0000","last_undegraded":"2024-08-03T21:00:51.939696+0000","last_fullsized":"2024-08-03T21:00:51.939696+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:48.192422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:48.192422+0000","last_clean_scrub_stamp":"2024-08-03T21:00:48.192422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T07:27:35.968034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.2","version":"0'0","reported_seq":13,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-08-03T21:00:51.938470+0000","last_change":"2024-08-03T21:00:50.232514+0000","last_active":"2024-08-03T21:00:51.938470+0000","last_peered":"2024-08-03T21:00:51.938470+0000","last_clean":"2024-08-03T21:00:51.938470+0000","last_became_active":"2024-08-03T21:00:50.232148+0000","last_became_peered":"2024-08-03T21:00:50.232148+0000","last_unstale":"2024-08-03T21:00:51.938470+0000","last_undegraded":"2024-08-03T21:00:51.938470+0000","last_fullsized":"2024-08-03T21:00:51.938470+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_clean_scrub_stamp":"2024-08-03T21:00:49.195479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-04T23:42:52.232285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.3","version":"0'0","reported_seq":15,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-08-03T21:00:51.938780+0000","last_change":"2024-08-03T21:00:49.235356+0000","last_active":"2024-08-03T21:00:51.938780+0000","last_peered":"2024-08-03T21:00:51.938780+0000","last_clean":"2024-08-03T21:00:51.938780+0000","last_became_active":"2024-08-03T21:00:49.234801+0000","last_became_peered":"2024-08-03T21:00:49.234801+0000","last_unstale":"2024-08-03T21:00:51.938780+0000","last_undegraded":"2024-08-03T21:00:51.938780+0000","last_fullsized":"2024-08-03T21:00:51.938780+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:48.192422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:48.192422+0000","last_clean_scrub_stamp":"2024-08-03T21:00:48.192422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T05:45:41.794416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.3","version":"0'0","reported_seq":13,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-08-03T21:00:51.939532+0000","last_change":"2024-08-03T21:00:50.233609+0000","last_active":"2024-08-03T21:00:51.939532+0000","last_peered":"2024-08-03T21:00:51.939532+0000","last_clean":"2024-08-03T21:00:51.939532+0000","last_became_active":"2024-08-03T21:00:50.231678+0000","last_became_peered":"2024-08-03T21:00:50.231678+0000","last_unstale":"2024-08-03T21:00:51.939532+0000","last_undegraded":"2024-08-03T21:00:51.939532+0000","last_fullsized":"2024-08-03T21:00:51.939532+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_clean_scrub_stamp":"2024-08-03T21:00:49.195479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T02:34:58.346689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.2","version":"0'0","reported_seq":15,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-08-03T21:00:51.938710+0000","last_change":"2024-08-03T21:00:49.218982+0000","last_active":"2024-08-03T21:00:51.938710+0000","last_peered":"2024-08-03T21:00:51.938710+0000","last_clean":"2024-08-03T21:00:51.938710+0000","last_became_active":"2024-08-03T21:00:49.217673+0000","last_became_peered":"2024-08-03T21:00:49.217673+0000","last_unstale":"2024-08-03T21:00:51.938710+0000","last_undegraded":"2024-08-03T21:00:51.938710+0000","last_fullsized":"2024-08-03T21:00:51.938710+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:48.192422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:48.192422+0000","last_clean_scrub_stamp":"2024-08-03T21:00:48.192422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T02:44:09.712432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.0","version":"0'0","reported_seq":13,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-08-03T21:00:51.938141+0000","last_change":"2024-08-03T21:00:50.234121+0000","last_active":"2024-08-03T21:00:51.938141+0000","last_peered":"2024-08-03T21:00:51.938141+0000","last_clean":"2024-08-03T21:00:51.938141+0000","last_became_active":"2024-08-03T21:00:50.233758+0000","last_became_peered":"2024-08-03T21:00:50.233758+0000","last_unstale":"2024-08-03T21:00:51.938141+0000","last_undegraded":"2024-08-03T21:00:51.938141+0000","last_fullsized":"2024-08-03T21:00:51.938141+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_clean_scrub_stamp":"2024-08-03T21:00:49.195479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T05:28:59.859843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.1","version":"0'0","reported_seq":15,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-08-03T21:00:51.225900+0000","last_change":"2024-08-03T21:00:49.230098+0000","last_active":"2024-08-03T21:00:51.225900+0000","last_peered":"2024-08-03T21:00:51.225900+0000","last_clean":"2024-08-03T21:00:51.225900+0000","last_became_active":"2024-08-03T21:00:49.229777+0000","last_became_peered":"2024-08-03T21:00:49.229777+0000","last_unstale":"2024-08-03T21:00:51.225900+0000","last_undegraded":"2024-08-03T21:00:51.225900+0000","last_fullsized":"2024-08-03T21:00:51.225900+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:48.192422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:48.192422+0000","last_clean_scrub_stamp":"2024-08-03T21:00:48.192422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T06:34:18.629210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.1","version":"0'0","reported_seq":13,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-08-03T21:00:51.938573+0000","last_change":"2024-08-03T21:00:50.234536+0000","last_active":"2024-08-03T21:00:51.938573+0000","last_peered":"2024-08-03T21:00:51.938573+0000","last_clean":"2024-08-03T21:00:51.938573+0000","last_became_active":"2024-08-03T21:00:50.233723+0000","last_became_peered":"2024-08-03T21:00:50.233723+0000","last_unstale":"2024-08-03T21:00:51.938573+0000","last_undegraded":"2024-08-03T21:00:51.938573+0000","last_fullsized":"2024-08-03T21:00:51.938573+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_clean_scrub_stamp":"2024-08-03T21:00:49.195479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T00:02:57.081976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.0","version":"0'0","reported_seq":15,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-08-03T21:00:51.938216+0000","last_change":"2024-08-03T21:00:49.231962+0000","last_active":"2024-08-03T21:00:51.938216+0000","last_peered":"2024-08-03T21:00:51.938216+0000","last_clean":"2024-08-03T21:00:51.938216+0000","last_became_active":"2024-08-03T21:00:49.231607+0000","last_became_peered":"2024-08-03T21:00:49.231607+0000","last_unstale":"2024-08-03T21:00:51.938216+0000","last_undegraded":"2024-08-03T21:00:51.938216+0000","last_fullsized":"2024-08-03T21:00:51.938216+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:48.192422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:48.192422+0000","last_clean_scrub_stamp":"2024-08-03T21:00:48.192422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T08:45:01.729460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.e","version":"0'0","reported_seq":13,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-08-03T21:00:51.939469+0000","last_change":"2024-08-03T21:00:50.249570+0000","last_active":"2024-08-03T21:00:51.939469+0000","last_peered":"2024-08-03T21:00:51.939469+0000","last_clean":"2024-08-03T21:00:51.939469+0000","last_became_active":"2024-08-03T21:00:50.240733+0000","last_became_peered":"2024-08-03T21:00:50.240733+0000","last_unstale":"2024-08-03T21:00:51.939469+0000","last_undegraded":"2024-08-03T21:00:51.939469+0000","last_fullsized":"2024-08-03T21:00:51.939469+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_clean_scrub_stamp":"2024-08-03T21:00:49.195479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T08:43:59.747967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.f","version":"134'1","reported_seq":16,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-08-03T21:00:51.226571+0000","last_change":"2024-08-03T21:00:49.234956+0000","last_active":"2024-08-03T21:00:51.226571+0000","last_peered":"2024-08-03T21:00:51.226571+0000","last_clean":"2024-08-03T21:00:51.226571+0000","last_became_active":"2024-08-03T21:00:49.234351+0000","last_became_peered":"2024-08-03T21:00:49.234351+0000","last_unstale":"2024-08-03T21:00:51.226571+0000","last_undegraded":"2024-08-03T21:00:51.226571+0000","last_fullsized":"2024-08-03T21:00:51.226571+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:48.192422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:48.192422+0000","last_clean_scrub_stamp":"2024-08-03T21:00:48.192422+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T00:43:49.078349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"16'72","reported_seq":331,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-08-03T21:00:51.226485+0000","last_change":"2024-08-03T20:47:30.547632+0000","last_active":"2024-08-03T21:00:51.226485+0000","last_peered":"2024-08-03T21:00:51.226485+0000","last_clean":"2024-08-03T21:00:51.226485+0000","last_became_active":"2024-08-03T20:47:30.546255+0000","last_became_peered":"2024-08-03T20:47:30.546255+0000","last_unstale":"2024-08-03T21:00:51.226485+0000","last_undegraded":"2024-08-03T21:00:51.226485+0000","last_fullsized":"2024-08-03T21:00:51.226485+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:47:28.874451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:47:28.874451+0000","last_clean_scrub_stamp":"2024-08-03T20:47:28.874451+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T00:19:40.548812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":15,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":14,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509604,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41776,"kb_used_data":2272,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330064,"statfs":{"total":96636764160,"available":96593985536,"internally_reserved":0,"allocated":2326528,"data_stored":1421385,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":40434122},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":6,"up_from":15,"seq":64424509605,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37032,"kb_used_data":1692,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334808,"statfs":{"total":96636764160,"available":96598843392,"internally_reserved":0,"allocated":1732608,"data_stored":831557,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":5,"up_from":15,"seq":64424509605,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37028,"kb_used_data":1688,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334812,"statfs":{"total":96636764160,"available":96598847488,"internally_reserved":0,"allocated":1728512,"data_stored":830975,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509604,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41184,"kb_used_data":1684,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330656,"statfs":{"total":96636764160,"available":96594591744,"internally_reserved":0,"allocated":1724416,"data_stored":830905,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509606,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":29292,"kb_used_data":2268,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94342548,"statfs":{"total":96636764160,"available":96606769152,"internally_reserved":0,"allocated":2322432,"data_stored":1421855,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542309,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37048,"kb_used_data":1704,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334792,"statfs":{"total":96636764160,"available":96598827008,"internally_reserved":0,"allocated":1744896,"data_stored":832641,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":3,"up_from":14,"seq":60129542307,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37044,"kb_used_data":1700,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334796,"statfs":{"total":96636764160,"available":96598831104,"internally_reserved":0,"allocated":1740800,"data_stored":832539,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":2,"up_from":14,"seq":60129542308,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41192,"kb_used_data":1688,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330648,"statfs":{"total":96636764160,"available":96594583552,"internally_reserved":0,"allocated":1728512,"data_stored":830939,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-08-03T21:00:59.843 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-08-03T21:01:00.082 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:00.080+0000 7fd1bb200640 1 -- 172.21.15.32:0/436713786 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd1bc06d9f0 msgr2=0x7fd1bc06de00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:01:00.082 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:00.080+0000 7fd1bb200640 1 --2- 172.21.15.32:0/436713786 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd1bc06d9f0 0x7fd1bc06de00 secure :-1 s=READY pgs=2180 cs=0 l=1 rev1=1 crypto rx=0x7fd1bc06e4f0 tx=0x7fd1b402a980 comp rx=0 tx=0).stop 2024-08-03T21:01:00.082 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:00.080+0000 7fd1bb200640 1 -- 172.21.15.32:0/436713786 shutdown_connections 2024-08-03T21:01:00.082 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:00.080+0000 7fd1bb200640 1 --2- 172.21.15.32:0/436713786 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd1bc06d9f0 0x7fd1bc06de00 unknown :-1 s=CLOSED pgs=2180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:01:00.083 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:00.080+0000 7fd1bb200640 1 --2- 172.21.15.32:0/436713786 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fd1bc153270 0x7fd1bc15d230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:01:00.083 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:00.080+0000 7fd1bb200640 1 --2- 172.21.15.32:0/436713786 >> v2:172.21.15.118:3300/0 conn(0x7fd1bc073990 0x7fd1bc0cd860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:01:00.083 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:00.080+0000 7fd1bb200640 1 -- 172.21.15.32:0/436713786 >> 172.21.15.32:0/436713786 conn(0x7fd1bc072e70 msgr2=0x7fd1bc073280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T21:01:00.083 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:00.080+0000 7fd1bb200640 1 -- 172.21.15.32:0/436713786 shutdown_connections 2024-08-03T21:01:00.083 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:00.080+0000 7fd1bb200640 1 -- 172.21.15.32:0/436713786 wait complete. 2024-08-03T21:01:00.084 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:00.083+0000 7fd1bb200640 1 Processor -- start 2024-08-03T21:01:00.084 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:00.083+0000 7fd1bb200640 1 -- start start 2024-08-03T21:01:00.086 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:00.084+0000 7fd1bb200640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fd1bc06d9f0 0x7fd1bc156370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:01:00.086 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:00.084+0000 7fd1bb200640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fd1bc073990 0x7fd1bc142920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:01:00.086 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:00.084+0000 7fd1bb200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd1bc153270 0x7fd1bc152230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:01:00.086 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:00.084+0000 7fd1bb200640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fd1bc160130 con 0x7fd1bc153270 2024-08-03T21:01:00.087 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:00.084+0000 7fd1bb200640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7fd1bc15ffb0 con 0x7fd1bc06d9f0 2024-08-03T21:01:00.087 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:00.084+0000 7fd1bb200640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7fd1bc1602b0 con 0x7fd1bc073990 2024-08-03T21:01:00.087 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:00.084+0000 7fd1ba800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd1bc153270 0x7fd1bc152230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:01:00.087 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:00.084+0000 7fd1ba800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd1bc153270 0x7fd1bc152230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:50404/0 (socket says 172.21.15.32:50404) 2024-08-03T21:01:00.087 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:00.084+0000 7fd1ba800640 1 -- 172.21.15.32:0/1001440673 learned_addr learned my addr 172.21.15.32:0/1001440673 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T21:01:00.087 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:00.084+0000 7fd1b3e00640 1 --2- 172.21.15.32:0/1001440673 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fd1bc073990 0x7fd1bc142920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:01:00.087 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:00.084+0000 7fd1b9e00640 1 --2- 172.21.15.32:0/1001440673 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fd1bc06d9f0 0x7fd1bc156370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:01:00.087 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:00.085+0000 7fd1ba800640 1 -- 172.21.15.32:0/1001440673 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fd1bc073990 msgr2=0x7fd1bc142920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:01:00.087 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:00.085+0000 7fd1ba800640 1 --2- 172.21.15.32:0/1001440673 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fd1bc073990 0x7fd1bc142920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:01:00.087 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:00.085+0000 7fd1ba800640 1 -- 172.21.15.32:0/1001440673 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fd1bc06d9f0 msgr2=0x7fd1bc156370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:01:00.087 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:00.085+0000 7fd1ba800640 1 --2- 172.21.15.32:0/1001440673 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fd1bc06d9f0 0x7fd1bc156370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:01:00.087 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:00.085+0000 7fd1ba800640 1 -- 172.21.15.32:0/1001440673 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd1bc146130 con 0x7fd1bc153270 2024-08-03T21:01:00.088 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:00.085+0000 7fd1ba800640 1 --2- 172.21.15.32:0/1001440673 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd1bc153270 0x7fd1bc152230 secure :-1 s=READY pgs=2181 cs=0 l=1 rev1=1 crypto rx=0x7fd1b4033040 tx=0x7fd1b4003f10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T21:01:00.088 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:00.085+0000 7fd1b2000640 1 -- 172.21.15.32:0/1001440673 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1b4034040 con 0x7fd1bc153270 2024-08-03T21:01:00.088 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:00.086+0000 7fd1b2000640 1 -- 172.21.15.32:0/1001440673 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fd1b4004540 con 0x7fd1bc153270 2024-08-03T21:01:00.088 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:00.086+0000 7fd1b2000640 1 -- 172.21.15.32:0/1001440673 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1b4045480 con 0x7fd1bc153270 2024-08-03T21:01:00.088 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:00.086+0000 7fd1bb200640 1 -- 172.21.15.32:0/1001440673 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd1bc1f8af0 con 0x7fd1bc153270 2024-08-03T21:01:00.089 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:00.086+0000 7fd1bb200640 1 -- 172.21.15.32:0/1001440673 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd1bc1f90b0 con 0x7fd1bc153270 2024-08-03T21:01:00.089 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:00.087+0000 7fd1bb200640 1 -- 172.21.15.32:0/1001440673 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd1bc06de00 con 0x7fd1bc153270 2024-08-03T21:01:00.089 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:00.088+0000 7fd1b2000640 1 -- 172.21.15.32:0/1001440673 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7fd1b4037020 con 0x7fd1bc153270 2024-08-03T21:01:00.091 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:00.089+0000 7fd1b2000640 1 --2- 172.21.15.32:0/1001440673 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fd198077010 0x7fd198079520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:01:00.091 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:00.089+0000 7fd1b2000640 1 -- 172.21.15.32:0/1001440673 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(135..135 src has 1..135) ==== 5709+0+0 (secure 0 0 0) 0x7fd1b40ba140 con 0x7fd1bc153270 2024-08-03T21:01:00.091 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:00.090+0000 7fd1b9e00640 1 --2- 172.21.15.32:0/1001440673 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fd198077010 0x7fd198079520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:01:00.092 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:00.090+0000 7fd1b9e00640 1 --2- 172.21.15.32:0/1001440673 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fd198077010 0x7fd198079520 secure :-1 s=READY pgs=875 cs=0 l=1 rev1=1 crypto rx=0x7fd1bc0cc8b0 tx=0x7fd1ac00a2d0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T21:01:00.096 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:00.094+0000 7fd1b2000640 1 -- 172.21.15.32:0/1001440673 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7fd1bc06de00 con 0x7fd1bc153270 2024-08-03T21:01:00.296 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:00.295+0000 7fd1bb200640 1 -- 172.21.15.32:0/1001440673 --> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fd1bc0b3cb0 con 0x7fd198077010 2024-08-03T21:01:00.301 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:00.299+0000 7fd1b2000640 1 -- 172.21.15.32:0/1001440673 <== mgr.4106 v2:172.21.15.118:6800/4205950699 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+252476 (secure 0 0 0) 0x7fd1bc0b3cb0 con 0x7fd198077010 2024-08-03T21:01:00.304 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-03T21:01:00.305 INFO:teuthology.orchestra.run.smithi032.stderr:dumped all 2024-08-03T21:01:00.307 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:00.304+0000 7fd1bb200640 1 -- 172.21.15.32:0/1001440673 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fd198077010 msgr2=0x7fd198079520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:01:00.307 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:00.304+0000 7fd1bb200640 1 --2- 172.21.15.32:0/1001440673 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fd198077010 0x7fd198079520 secure :-1 s=READY pgs=875 cs=0 l=1 rev1=1 crypto rx=0x7fd1bc0cc8b0 tx=0x7fd1ac00a2d0 comp rx=0 tx=0).stop 2024-08-03T21:01:00.307 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:00.304+0000 7fd1bb200640 1 -- 172.21.15.32:0/1001440673 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd1bc153270 msgr2=0x7fd1bc152230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:01:00.307 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:00.304+0000 7fd1bb200640 1 --2- 172.21.15.32:0/1001440673 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd1bc153270 0x7fd1bc152230 secure :-1 s=READY pgs=2181 cs=0 l=1 rev1=1 crypto rx=0x7fd1b4033040 tx=0x7fd1b4003f10 comp rx=0 tx=0).stop 2024-08-03T21:01:00.307 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:00.304+0000 7fd1bb200640 1 -- 172.21.15.32:0/1001440673 shutdown_connections 2024-08-03T21:01:00.307 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:00.304+0000 7fd1bb200640 1 --2- 172.21.15.32:0/1001440673 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fd198077010 0x7fd198079520 unknown :-1 s=CLOSED pgs=875 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:01:00.307 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:00.304+0000 7fd1bb200640 1 --2- 172.21.15.32:0/1001440673 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd1bc153270 0x7fd1bc152230 unknown :-1 s=CLOSED pgs=2181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:01:00.307 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:00.304+0000 7fd1bb200640 1 --2- 172.21.15.32:0/1001440673 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fd1bc073990 0x7fd1bc142920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:01:00.307 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:00.304+0000 7fd1bb200640 1 --2- 172.21.15.32:0/1001440673 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fd1bc06d9f0 0x7fd1bc156370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:01:00.307 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:00.304+0000 7fd1bb200640 1 -- 172.21.15.32:0/1001440673 >> 172.21.15.32:0/1001440673 conn(0x7fd1bc072e70 msgr2=0x7fd1bc0b0f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T21:01:00.307 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:00.304+0000 7fd1bb200640 1 -- 172.21.15.32:0/1001440673 shutdown_connections 2024-08-03T21:01:00.307 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:00.304+0000 7fd1bb200640 1 -- 172.21.15.32:0/1001440673 wait complete. 2024-08-03T21:01:00.328 INFO:teuthology.orchestra.run.smithi032.stdout:{"pg_ready":true,"pg_map":{"version":533,"stamp":"2024-08-03T21:00:58.455604+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":146,"num_write_kb":1642,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":111,"ondisk_log_size":111,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":301596,"kb_used_data":14696,"kb_used_omap":12,"kb_used_meta":286771,"kb_avail":754673124,"statfs":{"total":773094113280,"available":772785278976,"internally_reserved":0,"allocated":15048704,"data_stored":7832796,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":293654065},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":35,"apply_latency_ms":35,"commit_latency_ns":35000000,"apply_latency_ns":35000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.249301"},"pg_stats":[{"pgid":"15.3f","version":"0'0","reported_seq":13,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-08-03T21:00:51.937735+0000","last_change":"2024-08-03T21:00:50.248721+0000","last_active":"2024-08-03T21:00:51.937735+0000","last_peered":"2024-08-03T21:00:51.937735+0000","last_clean":"2024-08-03T21:00:51.937735+0000","last_became_active":"2024-08-03T21:00:50.242246+0000","last_became_peered":"2024-08-03T21:00:50.242246+0000","last_unstale":"2024-08-03T21:00:51.937735+0000","last_undegraded":"2024-08-03T21:00:51.937735+0000","last_fullsized":"2024-08-03T21:00:51.937735+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_clean_scrub_stamp":"2024-08-03T21:00:49.195479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-04T23:49:06.542747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.3e","version":"0'0","reported_seq":13,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-08-03T21:00:51.939015+0000","last_change":"2024-08-03T21:00:50.245164+0000","last_active":"2024-08-03T21:00:51.939015+0000","last_peered":"2024-08-03T21:00:51.939015+0000","last_clean":"2024-08-03T21:00:51.939015+0000","last_became_active":"2024-08-03T21:00:50.244799+0000","last_became_peered":"2024-08-03T21:00:50.244799+0000","last_unstale":"2024-08-03T21:00:51.939015+0000","last_undegraded":"2024-08-03T21:00:51.939015+0000","last_fullsized":"2024-08-03T21:00:51.939015+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_clean_scrub_stamp":"2024-08-03T21:00:49.195479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T00:55:29.579986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.3d","version":"0'0","reported_seq":13,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-08-03T21:00:51.937833+0000","last_change":"2024-08-03T21:00:50.246785+0000","last_active":"2024-08-03T21:00:51.937833+0000","last_peered":"2024-08-03T21:00:51.937833+0000","last_clean":"2024-08-03T21:00:51.937833+0000","last_became_active":"2024-08-03T21:00:50.241990+0000","last_became_peered":"2024-08-03T21:00:50.241990+0000","last_unstale":"2024-08-03T21:00:51.937833+0000","last_undegraded":"2024-08-03T21:00:51.937833+0000","last_fullsized":"2024-08-03T21:00:51.937833+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_clean_scrub_stamp":"2024-08-03T21:00:49.195479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T08:47:17.514748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.3c","version":"0'0","reported_seq":13,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-08-03T21:00:51.225682+0000","last_change":"2024-08-03T21:00:50.243269+0000","last_active":"2024-08-03T21:00:51.225682+0000","last_peered":"2024-08-03T21:00:51.225682+0000","last_clean":"2024-08-03T21:00:51.225682+0000","last_became_active":"2024-08-03T21:00:50.242985+0000","last_became_peered":"2024-08-03T21:00:50.242985+0000","last_unstale":"2024-08-03T21:00:51.225682+0000","last_undegraded":"2024-08-03T21:00:51.225682+0000","last_fullsized":"2024-08-03T21:00:51.225682+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_clean_scrub_stamp":"2024-08-03T21:00:49.195479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T03:32:19.871676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.3b","version":"0'0","reported_seq":13,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-08-03T21:00:51.938596+0000","last_change":"2024-08-03T21:00:50.222469+0000","last_active":"2024-08-03T21:00:51.938596+0000","last_peered":"2024-08-03T21:00:51.938596+0000","last_clean":"2024-08-03T21:00:51.938596+0000","last_became_active":"2024-08-03T21:00:50.222097+0000","last_became_peered":"2024-08-03T21:00:50.222097+0000","last_unstale":"2024-08-03T21:00:51.938596+0000","last_undegraded":"2024-08-03T21:00:51.938596+0000","last_fullsized":"2024-08-03T21:00:51.938596+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_clean_scrub_stamp":"2024-08-03T21:00:49.195479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T05:22:18.315672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.3a","version":"0'0","reported_seq":13,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-08-03T21:00:51.226404+0000","last_change":"2024-08-03T21:00:50.240363+0000","last_active":"2024-08-03T21:00:51.226404+0000","last_peered":"2024-08-03T21:00:51.226404+0000","last_clean":"2024-08-03T21:00:51.226404+0000","last_became_active":"2024-08-03T21:00:50.239995+0000","last_became_peered":"2024-08-03T21:00:50.239995+0000","last_unstale":"2024-08-03T21:00:51.226404+0000","last_undegraded":"2024-08-03T21:00:51.226404+0000","last_fullsized":"2024-08-03T21:00:51.226404+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_clean_scrub_stamp":"2024-08-03T21:00:49.195479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T03:52:56.414014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.39","version":"0'0","reported_seq":13,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-08-03T21:00:51.226291+0000","last_change":"2024-08-03T21:00:50.230272+0000","last_active":"2024-08-03T21:00:51.226291+0000","last_peered":"2024-08-03T21:00:51.226291+0000","last_clean":"2024-08-03T21:00:51.226291+0000","last_became_active":"2024-08-03T21:00:50.229940+0000","last_became_peered":"2024-08-03T21:00:50.229940+0000","last_unstale":"2024-08-03T21:00:51.226291+0000","last_undegraded":"2024-08-03T21:00:51.226291+0000","last_fullsized":"2024-08-03T21:00:51.226291+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_clean_scrub_stamp":"2024-08-03T21:00:49.195479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-04T21:28:54.164554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.38","version":"0'0","reported_seq":13,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-08-03T21:00:51.937383+0000","last_change":"2024-08-03T21:00:50.233757+0000","last_active":"2024-08-03T21:00:51.937383+0000","last_peered":"2024-08-03T21:00:51.937383+0000","last_clean":"2024-08-03T21:00:51.937383+0000","last_became_active":"2024-08-03T21:00:50.233470+0000","last_became_peered":"2024-08-03T21:00:50.233470+0000","last_unstale":"2024-08-03T21:00:51.937383+0000","last_undegraded":"2024-08-03T21:00:51.937383+0000","last_fullsized":"2024-08-03T21:00:51.937383+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_clean_scrub_stamp":"2024-08-03T21:00:49.195479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T04:02:53.255557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.37","version":"0'0","reported_seq":13,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-08-03T21:00:51.939823+0000","last_change":"2024-08-03T21:00:50.223154+0000","last_active":"2024-08-03T21:00:51.939823+0000","last_peered":"2024-08-03T21:00:51.939823+0000","last_clean":"2024-08-03T21:00:51.939823+0000","last_became_active":"2024-08-03T21:00:50.222779+0000","last_became_peered":"2024-08-03T21:00:50.222779+0000","last_unstale":"2024-08-03T21:00:51.939823+0000","last_undegraded":"2024-08-03T21:00:51.939823+0000","last_fullsized":"2024-08-03T21:00:51.939823+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_clean_scrub_stamp":"2024-08-03T21:00:49.195479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T06:34:19.632267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.36","version":"0'0","reported_seq":13,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-08-03T21:00:51.939105+0000","last_change":"2024-08-03T21:00:50.238935+0000","last_active":"2024-08-03T21:00:51.939105+0000","last_peered":"2024-08-03T21:00:51.939105+0000","last_clean":"2024-08-03T21:00:51.939105+0000","last_became_active":"2024-08-03T21:00:50.238549+0000","last_became_peered":"2024-08-03T21:00:50.238549+0000","last_unstale":"2024-08-03T21:00:51.939105+0000","last_undegraded":"2024-08-03T21:00:51.939105+0000","last_fullsized":"2024-08-03T21:00:51.939105+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_clean_scrub_stamp":"2024-08-03T21:00:49.195479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T00:17:50.223702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.35","version":"0'0","reported_seq":13,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-08-03T21:00:51.226808+0000","last_change":"2024-08-03T21:00:50.229274+0000","last_active":"2024-08-03T21:00:51.226808+0000","last_peered":"2024-08-03T21:00:51.226808+0000","last_clean":"2024-08-03T21:00:51.226808+0000","last_became_active":"2024-08-03T21:00:50.228900+0000","last_became_peered":"2024-08-03T21:00:50.228900+0000","last_unstale":"2024-08-03T21:00:51.226808+0000","last_undegraded":"2024-08-03T21:00:51.226808+0000","last_fullsized":"2024-08-03T21:00:51.226808+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_clean_scrub_stamp":"2024-08-03T21:00:49.195479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T04:02:53.255557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.34","version":"0'0","reported_seq":13,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-08-03T21:00:51.939323+0000","last_change":"2024-08-03T21:00:50.229025+0000","last_active":"2024-08-03T21:00:51.939323+0000","last_peered":"2024-08-03T21:00:51.939323+0000","last_clean":"2024-08-03T21:00:51.939323+0000","last_became_active":"2024-08-03T21:00:50.228698+0000","last_became_peered":"2024-08-03T21:00:50.228698+0000","last_unstale":"2024-08-03T21:00:51.939323+0000","last_undegraded":"2024-08-03T21:00:51.939323+0000","last_fullsized":"2024-08-03T21:00:51.939323+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_clean_scrub_stamp":"2024-08-03T21:00:49.195479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T06:51:09.079780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.33","version":"0'0","reported_seq":13,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-08-03T21:00:51.225611+0000","last_change":"2024-08-03T21:00:50.224093+0000","last_active":"2024-08-03T21:00:51.225611+0000","last_peered":"2024-08-03T21:00:51.225611+0000","last_clean":"2024-08-03T21:00:51.225611+0000","last_became_active":"2024-08-03T21:00:50.222966+0000","last_became_peered":"2024-08-03T21:00:50.222966+0000","last_unstale":"2024-08-03T21:00:51.225611+0000","last_undegraded":"2024-08-03T21:00:51.225611+0000","last_fullsized":"2024-08-03T21:00:51.225611+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_clean_scrub_stamp":"2024-08-03T21:00:49.195479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T04:11:36.038037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.32","version":"0'0","reported_seq":13,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-08-03T21:00:51.937848+0000","last_change":"2024-08-03T21:00:50.230657+0000","last_active":"2024-08-03T21:00:51.937848+0000","last_peered":"2024-08-03T21:00:51.937848+0000","last_clean":"2024-08-03T21:00:51.937848+0000","last_became_active":"2024-08-03T21:00:50.230356+0000","last_became_peered":"2024-08-03T21:00:50.230356+0000","last_unstale":"2024-08-03T21:00:51.937848+0000","last_undegraded":"2024-08-03T21:00:51.937848+0000","last_fullsized":"2024-08-03T21:00:51.937848+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_clean_scrub_stamp":"2024-08-03T21:00:49.195479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-04T21:20:29.060152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.31","version":"0'0","reported_seq":13,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-08-03T21:00:51.225596+0000","last_change":"2024-08-03T21:00:50.236123+0000","last_active":"2024-08-03T21:00:51.225596+0000","last_peered":"2024-08-03T21:00:51.225596+0000","last_clean":"2024-08-03T21:00:51.225596+0000","last_became_active":"2024-08-03T21:00:50.235747+0000","last_became_peered":"2024-08-03T21:00:50.235747+0000","last_unstale":"2024-08-03T21:00:51.225596+0000","last_undegraded":"2024-08-03T21:00:51.225596+0000","last_fullsized":"2024-08-03T21:00:51.225596+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_clean_scrub_stamp":"2024-08-03T21:00:49.195479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T05:25:30.476710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.30","version":"0'0","reported_seq":13,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-08-03T21:00:51.226256+0000","last_change":"2024-08-03T21:00:50.242724+0000","last_active":"2024-08-03T21:00:51.226256+0000","last_peered":"2024-08-03T21:00:51.226256+0000","last_clean":"2024-08-03T21:00:51.226256+0000","last_became_active":"2024-08-03T21:00:50.239291+0000","last_became_peered":"2024-08-03T21:00:50.239291+0000","last_unstale":"2024-08-03T21:00:51.226256+0000","last_undegraded":"2024-08-03T21:00:51.226256+0000","last_fullsized":"2024-08-03T21:00:51.226256+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_clean_scrub_stamp":"2024-08-03T21:00:49.195479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T04:09:17.948462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.2f","version":"0'0","reported_seq":13,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-08-03T21:00:51.939095+0000","last_change":"2024-08-03T21:00:50.244406+0000","last_active":"2024-08-03T21:00:51.939095+0000","last_peered":"2024-08-03T21:00:51.939095+0000","last_clean":"2024-08-03T21:00:51.939095+0000","last_became_active":"2024-08-03T21:00:50.239150+0000","last_became_peered":"2024-08-03T21:00:50.239150+0000","last_unstale":"2024-08-03T21:00:51.939095+0000","last_undegraded":"2024-08-03T21:00:51.939095+0000","last_fullsized":"2024-08-03T21:00:51.939095+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_clean_scrub_stamp":"2024-08-03T21:00:49.195479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T02:24:57.063186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.2e","version":"0'0","reported_seq":13,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-08-03T21:00:51.225830+0000","last_change":"2024-08-03T21:00:50.241634+0000","last_active":"2024-08-03T21:00:51.225830+0000","last_peered":"2024-08-03T21:00:51.225830+0000","last_clean":"2024-08-03T21:00:51.225830+0000","last_became_active":"2024-08-03T21:00:50.241263+0000","last_became_peered":"2024-08-03T21:00:50.241263+0000","last_unstale":"2024-08-03T21:00:51.225830+0000","last_undegraded":"2024-08-03T21:00:51.225830+0000","last_fullsized":"2024-08-03T21:00:51.225830+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_clean_scrub_stamp":"2024-08-03T21:00:49.195479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T02:07:46.123334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.2d","version":"0'0","reported_seq":13,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-08-03T21:00:51.938842+0000","last_change":"2024-08-03T21:00:50.243910+0000","last_active":"2024-08-03T21:00:51.938842+0000","last_peered":"2024-08-03T21:00:51.938842+0000","last_clean":"2024-08-03T21:00:51.938842+0000","last_became_active":"2024-08-03T21:00:50.243054+0000","last_became_peered":"2024-08-03T21:00:50.243054+0000","last_unstale":"2024-08-03T21:00:51.938842+0000","last_undegraded":"2024-08-03T21:00:51.938842+0000","last_fullsized":"2024-08-03T21:00:51.938842+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_clean_scrub_stamp":"2024-08-03T21:00:49.195479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T08:31:27.530602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.2c","version":"0'0","reported_seq":13,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-08-03T21:00:51.939427+0000","last_change":"2024-08-03T21:00:50.223678+0000","last_active":"2024-08-03T21:00:51.939427+0000","last_peered":"2024-08-03T21:00:51.939427+0000","last_clean":"2024-08-03T21:00:51.939427+0000","last_became_active":"2024-08-03T21:00:50.223357+0000","last_became_peered":"2024-08-03T21:00:50.223357+0000","last_unstale":"2024-08-03T21:00:51.939427+0000","last_undegraded":"2024-08-03T21:00:51.939427+0000","last_fullsized":"2024-08-03T21:00:51.939427+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_clean_scrub_stamp":"2024-08-03T21:00:49.195479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T07:48:35.449963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.2b","version":"0'0","reported_seq":13,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-08-03T21:00:51.939465+0000","last_change":"2024-08-03T21:00:50.243413+0000","last_active":"2024-08-03T21:00:51.939465+0000","last_peered":"2024-08-03T21:00:51.939465+0000","last_clean":"2024-08-03T21:00:51.939465+0000","last_became_active":"2024-08-03T21:00:50.243004+0000","last_became_peered":"2024-08-03T21:00:50.243004+0000","last_unstale":"2024-08-03T21:00:51.939465+0000","last_undegraded":"2024-08-03T21:00:51.939465+0000","last_fullsized":"2024-08-03T21:00:51.939465+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_clean_scrub_stamp":"2024-08-03T21:00:49.195479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-04T23:21:29.132988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.2a","version":"0'0","reported_seq":13,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-08-03T21:00:51.938006+0000","last_change":"2024-08-03T21:00:50.241385+0000","last_active":"2024-08-03T21:00:51.938006+0000","last_peered":"2024-08-03T21:00:51.938006+0000","last_clean":"2024-08-03T21:00:51.938006+0000","last_became_active":"2024-08-03T21:00:50.241070+0000","last_became_peered":"2024-08-03T21:00:50.241070+0000","last_unstale":"2024-08-03T21:00:51.938006+0000","last_undegraded":"2024-08-03T21:00:51.938006+0000","last_fullsized":"2024-08-03T21:00:51.938006+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_clean_scrub_stamp":"2024-08-03T21:00:49.195479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T05:41:07.172696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.29","version":"0'0","reported_seq":13,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-08-03T21:00:51.937830+0000","last_change":"2024-08-03T21:00:50.248216+0000","last_active":"2024-08-03T21:00:51.937830+0000","last_peered":"2024-08-03T21:00:51.937830+0000","last_clean":"2024-08-03T21:00:51.937830+0000","last_became_active":"2024-08-03T21:00:50.247907+0000","last_became_peered":"2024-08-03T21:00:50.247907+0000","last_unstale":"2024-08-03T21:00:51.937830+0000","last_undegraded":"2024-08-03T21:00:51.937830+0000","last_fullsized":"2024-08-03T21:00:51.937830+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_clean_scrub_stamp":"2024-08-03T21:00:49.195479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T02:13:04.485780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.28","version":"0'0","reported_seq":13,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-08-03T21:00:51.226317+0000","last_change":"2024-08-03T21:00:50.243021+0000","last_active":"2024-08-03T21:00:51.226317+0000","last_peered":"2024-08-03T21:00:51.226317+0000","last_clean":"2024-08-03T21:00:51.226317+0000","last_became_active":"2024-08-03T21:00:50.242046+0000","last_became_peered":"2024-08-03T21:00:50.242046+0000","last_unstale":"2024-08-03T21:00:51.226317+0000","last_undegraded":"2024-08-03T21:00:51.226317+0000","last_fullsized":"2024-08-03T21:00:51.226317+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_clean_scrub_stamp":"2024-08-03T21:00:49.195479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-04T23:18:44.490549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.27","version":"0'0","reported_seq":13,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-08-03T21:00:51.939134+0000","last_change":"2024-08-03T21:00:50.247527+0000","last_active":"2024-08-03T21:00:51.939134+0000","last_peered":"2024-08-03T21:00:51.939134+0000","last_clean":"2024-08-03T21:00:51.939134+0000","last_became_active":"2024-08-03T21:00:50.247153+0000","last_became_peered":"2024-08-03T21:00:50.247153+0000","last_unstale":"2024-08-03T21:00:51.939134+0000","last_undegraded":"2024-08-03T21:00:51.939134+0000","last_fullsized":"2024-08-03T21:00:51.939134+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_clean_scrub_stamp":"2024-08-03T21:00:49.195479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T04:09:17.948462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.26","version":"0'0","reported_seq":13,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-08-03T21:00:51.939190+0000","last_change":"2024-08-03T21:00:50.245129+0000","last_active":"2024-08-03T21:00:51.939190+0000","last_peered":"2024-08-03T21:00:51.939190+0000","last_clean":"2024-08-03T21:00:51.939190+0000","last_became_active":"2024-08-03T21:00:50.244410+0000","last_became_peered":"2024-08-03T21:00:50.244410+0000","last_unstale":"2024-08-03T21:00:51.939190+0000","last_undegraded":"2024-08-03T21:00:51.939190+0000","last_fullsized":"2024-08-03T21:00:51.939190+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_clean_scrub_stamp":"2024-08-03T21:00:49.195479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T08:18:17.429255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.25","version":"0'0","reported_seq":13,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-08-03T21:00:51.226949+0000","last_change":"2024-08-03T21:00:50.237747+0000","last_active":"2024-08-03T21:00:51.226949+0000","last_peered":"2024-08-03T21:00:51.226949+0000","last_clean":"2024-08-03T21:00:51.226949+0000","last_became_active":"2024-08-03T21:00:50.237417+0000","last_became_peered":"2024-08-03T21:00:50.237417+0000","last_unstale":"2024-08-03T21:00:51.226949+0000","last_undegraded":"2024-08-03T21:00:51.226949+0000","last_fullsized":"2024-08-03T21:00:51.226949+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_clean_scrub_stamp":"2024-08-03T21:00:49.195479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T02:13:04.485780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.24","version":"0'0","reported_seq":13,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-08-03T21:00:51.939440+0000","last_change":"2024-08-03T21:00:50.225198+0000","last_active":"2024-08-03T21:00:51.939440+0000","last_peered":"2024-08-03T21:00:51.939440+0000","last_clean":"2024-08-03T21:00:51.939440+0000","last_became_active":"2024-08-03T21:00:50.221390+0000","last_became_peered":"2024-08-03T21:00:50.221390+0000","last_unstale":"2024-08-03T21:00:51.939440+0000","last_undegraded":"2024-08-03T21:00:51.939440+0000","last_fullsized":"2024-08-03T21:00:51.939440+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_clean_scrub_stamp":"2024-08-03T21:00:49.195479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-04T21:28:54.164554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.23","version":"0'0","reported_seq":13,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-08-03T21:00:51.938541+0000","last_change":"2024-08-03T21:00:50.238959+0000","last_active":"2024-08-03T21:00:51.938541+0000","last_peered":"2024-08-03T21:00:51.938541+0000","last_clean":"2024-08-03T21:00:51.938541+0000","last_became_active":"2024-08-03T21:00:50.238586+0000","last_became_peered":"2024-08-03T21:00:50.238586+0000","last_unstale":"2024-08-03T21:00:51.938541+0000","last_undegraded":"2024-08-03T21:00:51.938541+0000","last_fullsized":"2024-08-03T21:00:51.938541+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_clean_scrub_stamp":"2024-08-03T21:00:49.195479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T03:49:21.574825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.22","version":"0'0","reported_seq":13,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-08-03T21:00:51.938224+0000","last_change":"2024-08-03T21:00:50.243735+0000","last_active":"2024-08-03T21:00:51.938224+0000","last_peered":"2024-08-03T21:00:51.938224+0000","last_clean":"2024-08-03T21:00:51.938224+0000","last_became_active":"2024-08-03T21:00:50.243333+0000","last_became_peered":"2024-08-03T21:00:50.243333+0000","last_unstale":"2024-08-03T21:00:51.938224+0000","last_undegraded":"2024-08-03T21:00:51.938224+0000","last_fullsized":"2024-08-03T21:00:51.938224+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_clean_scrub_stamp":"2024-08-03T21:00:49.195479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T08:50:30.366065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.21","version":"0'0","reported_seq":13,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-08-03T21:00:51.225546+0000","last_change":"2024-08-03T21:00:50.229431+0000","last_active":"2024-08-03T21:00:51.225546+0000","last_peered":"2024-08-03T21:00:51.225546+0000","last_clean":"2024-08-03T21:00:51.225546+0000","last_became_active":"2024-08-03T21:00:50.228793+0000","last_became_peered":"2024-08-03T21:00:50.228793+0000","last_unstale":"2024-08-03T21:00:51.225546+0000","last_undegraded":"2024-08-03T21:00:51.225546+0000","last_fullsized":"2024-08-03T21:00:51.225546+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_clean_scrub_stamp":"2024-08-03T21:00:49.195479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T03:21:17.046927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.20","version":"0'0","reported_seq":13,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-08-03T21:00:51.939588+0000","last_change":"2024-08-03T21:00:50.242219+0000","last_active":"2024-08-03T21:00:51.939588+0000","last_peered":"2024-08-03T21:00:51.939588+0000","last_clean":"2024-08-03T21:00:51.939588+0000","last_became_active":"2024-08-03T21:00:50.241899+0000","last_became_peered":"2024-08-03T21:00:50.241899+0000","last_unstale":"2024-08-03T21:00:51.939588+0000","last_undegraded":"2024-08-03T21:00:51.939588+0000","last_fullsized":"2024-08-03T21:00:51.939588+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_clean_scrub_stamp":"2024-08-03T21:00:49.195479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T02:44:19.439200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.1e","version":"0'0","reported_seq":13,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-08-03T21:00:51.939274+0000","last_change":"2024-08-03T21:00:50.235759+0000","last_active":"2024-08-03T21:00:51.939274+0000","last_peered":"2024-08-03T21:00:51.939274+0000","last_clean":"2024-08-03T21:00:51.939274+0000","last_became_active":"2024-08-03T21:00:50.235440+0000","last_became_peered":"2024-08-03T21:00:50.235440+0000","last_unstale":"2024-08-03T21:00:51.939274+0000","last_undegraded":"2024-08-03T21:00:51.939274+0000","last_fullsized":"2024-08-03T21:00:51.939274+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_clean_scrub_stamp":"2024-08-03T21:00:49.195479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-04T21:51:33.128276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.1f","version":"134'4","reported_seq":19,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-08-03T21:00:51.939213+0000","last_change":"2024-08-03T21:00:49.231401+0000","last_active":"2024-08-03T21:00:51.939213+0000","last_peered":"2024-08-03T21:00:51.939213+0000","last_clean":"2024-08-03T21:00:51.939213+0000","last_became_active":"2024-08-03T21:00:49.231089+0000","last_became_peered":"2024-08-03T21:00:49.231089+0000","last_unstale":"2024-08-03T21:00:51.939213+0000","last_undegraded":"2024-08-03T21:00:51.939213+0000","last_fullsized":"2024-08-03T21:00:51.939213+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:48.192422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:48.192422+0000","last_clean_scrub_stamp":"2024-08-03T21:00:48.192422+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T03:51:00.885909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.1f","version":"0'0","reported_seq":13,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-08-03T21:00:51.225857+0000","last_change":"2024-08-03T21:00:50.226490+0000","last_active":"2024-08-03T21:00:51.225857+0000","last_peered":"2024-08-03T21:00:51.225857+0000","last_clean":"2024-08-03T21:00:51.225857+0000","last_became_active":"2024-08-03T21:00:50.226103+0000","last_became_peered":"2024-08-03T21:00:50.226103+0000","last_unstale":"2024-08-03T21:00:51.225857+0000","last_undegraded":"2024-08-03T21:00:51.225857+0000","last_fullsized":"2024-08-03T21:00:51.225857+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_clean_scrub_stamp":"2024-08-03T21:00:49.195479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T04:51:31.368175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.1e","version":"134'1","reported_seq":16,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-08-03T21:00:51.226663+0000","last_change":"2024-08-03T21:00:49.227358+0000","last_active":"2024-08-03T21:00:51.226663+0000","last_peered":"2024-08-03T21:00:51.226663+0000","last_clean":"2024-08-03T21:00:51.226663+0000","last_became_active":"2024-08-03T21:00:49.226904+0000","last_became_peered":"2024-08-03T21:00:49.226904+0000","last_unstale":"2024-08-03T21:00:51.226663+0000","last_undegraded":"2024-08-03T21:00:51.226663+0000","last_fullsized":"2024-08-03T21:00:51.226663+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:48.192422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:48.192422+0000","last_clean_scrub_stamp":"2024-08-03T21:00:48.192422+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T05:45:41.794416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.1c","version":"0'0","reported_seq":13,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-08-03T21:00:51.937638+0000","last_change":"2024-08-03T21:00:50.236287+0000","last_active":"2024-08-03T21:00:51.937638+0000","last_peered":"2024-08-03T21:00:51.937638+0000","last_clean":"2024-08-03T21:00:51.937638+0000","last_became_active":"2024-08-03T21:00:50.234964+0000","last_became_peered":"2024-08-03T21:00:50.234964+0000","last_unstale":"2024-08-03T21:00:51.937638+0000","last_undegraded":"2024-08-03T21:00:51.937638+0000","last_fullsized":"2024-08-03T21:00:51.937638+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_clean_scrub_stamp":"2024-08-03T21:00:49.195479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T07:20:57.267245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.1d","version":"134'2","reported_seq":17,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-08-03T21:00:51.938671+0000","last_change":"2024-08-03T21:00:49.234514+0000","last_active":"2024-08-03T21:00:51.938671+0000","last_peered":"2024-08-03T21:00:51.938671+0000","last_clean":"2024-08-03T21:00:51.938671+0000","last_became_active":"2024-08-03T21:00:49.233091+0000","last_became_peered":"2024-08-03T21:00:49.233091+0000","last_unstale":"2024-08-03T21:00:51.938671+0000","last_undegraded":"2024-08-03T21:00:51.938671+0000","last_fullsized":"2024-08-03T21:00:51.938671+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:48.192422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:48.192422+0000","last_clean_scrub_stamp":"2024-08-03T21:00:48.192422+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T00:43:49.078349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.1d","version":"0'0","reported_seq":13,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-08-03T21:00:51.939823+0000","last_change":"2024-08-03T21:00:50.235731+0000","last_active":"2024-08-03T21:00:51.939823+0000","last_peered":"2024-08-03T21:00:51.939823+0000","last_clean":"2024-08-03T21:00:51.939823+0000","last_became_active":"2024-08-03T21:00:50.234805+0000","last_became_peered":"2024-08-03T21:00:50.234805+0000","last_unstale":"2024-08-03T21:00:51.939823+0000","last_undegraded":"2024-08-03T21:00:51.939823+0000","last_fullsized":"2024-08-03T21:00:51.939823+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_clean_scrub_stamp":"2024-08-03T21:00:49.195479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T05:40:57.488842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.1c","version":"134'2","reported_seq":17,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-08-03T21:00:51.938470+0000","last_change":"2024-08-03T21:00:49.227739+0000","last_active":"2024-08-03T21:00:51.938470+0000","last_peered":"2024-08-03T21:00:51.938470+0000","last_clean":"2024-08-03T21:00:51.938470+0000","last_became_active":"2024-08-03T21:00:49.226569+0000","last_became_peered":"2024-08-03T21:00:49.226569+0000","last_unstale":"2024-08-03T21:00:51.938470+0000","last_undegraded":"2024-08-03T21:00:51.938470+0000","last_fullsized":"2024-08-03T21:00:51.938470+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:48.192422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:48.192422+0000","last_clean_scrub_stamp":"2024-08-03T21:00:48.192422+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-04T22:46:01.864033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.1a","version":"0'0","reported_seq":13,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-08-03T21:00:51.940112+0000","last_change":"2024-08-03T21:00:50.230345+0000","last_active":"2024-08-03T21:00:51.940112+0000","last_peered":"2024-08-03T21:00:51.940112+0000","last_clean":"2024-08-03T21:00:51.940112+0000","last_became_active":"2024-08-03T21:00:50.230001+0000","last_became_peered":"2024-08-03T21:00:50.230001+0000","last_unstale":"2024-08-03T21:00:51.940112+0000","last_undegraded":"2024-08-03T21:00:51.940112+0000","last_fullsized":"2024-08-03T21:00:51.940112+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_clean_scrub_stamp":"2024-08-03T21:00:49.195479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-04T23:53:42.243847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.1b","version":"134'1","reported_seq":16,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-08-03T21:00:51.938855+0000","last_change":"2024-08-03T21:00:49.217597+0000","last_active":"2024-08-03T21:00:51.938855+0000","last_peered":"2024-08-03T21:00:51.938855+0000","last_clean":"2024-08-03T21:00:51.938855+0000","last_became_active":"2024-08-03T21:00:49.217212+0000","last_became_peered":"2024-08-03T21:00:49.217212+0000","last_unstale":"2024-08-03T21:00:51.938855+0000","last_undegraded":"2024-08-03T21:00:51.938855+0000","last_fullsized":"2024-08-03T21:00:51.938855+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:48.192422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:48.192422+0000","last_clean_scrub_stamp":"2024-08-03T21:00:48.192422+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T00:07:15.964072+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.1b","version":"0'0","reported_seq":13,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-08-03T21:00:51.226661+0000","last_change":"2024-08-03T21:00:50.225540+0000","last_active":"2024-08-03T21:00:51.226661+0000","last_peered":"2024-08-03T21:00:51.226661+0000","last_clean":"2024-08-03T21:00:51.226661+0000","last_became_active":"2024-08-03T21:00:50.225186+0000","last_became_peered":"2024-08-03T21:00:50.225186+0000","last_unstale":"2024-08-03T21:00:51.226661+0000","last_undegraded":"2024-08-03T21:00:51.226661+0000","last_fullsized":"2024-08-03T21:00:51.226661+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_clean_scrub_stamp":"2024-08-03T21:00:49.195479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T02:52:49.825428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.1a","version":"134'1","reported_seq":16,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-08-03T21:00:51.938252+0000","last_change":"2024-08-03T21:00:49.238475+0000","last_active":"2024-08-03T21:00:51.938252+0000","last_peered":"2024-08-03T21:00:51.938252+0000","last_clean":"2024-08-03T21:00:51.938252+0000","last_became_active":"2024-08-03T21:00:49.238001+0000","last_became_peered":"2024-08-03T21:00:49.238001+0000","last_unstale":"2024-08-03T21:00:51.938252+0000","last_undegraded":"2024-08-03T21:00:51.938252+0000","last_fullsized":"2024-08-03T21:00:51.938252+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:48.192422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:48.192422+0000","last_clean_scrub_stamp":"2024-08-03T21:00:48.192422+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-04T23:43:28.912246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.18","version":"0'0","reported_seq":13,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-08-03T21:00:51.225613+0000","last_change":"2024-08-03T21:00:50.228305+0000","last_active":"2024-08-03T21:00:51.225613+0000","last_peered":"2024-08-03T21:00:51.225613+0000","last_clean":"2024-08-03T21:00:51.225613+0000","last_became_active":"2024-08-03T21:00:50.228009+0000","last_became_peered":"2024-08-03T21:00:50.228009+0000","last_unstale":"2024-08-03T21:00:51.225613+0000","last_undegraded":"2024-08-03T21:00:51.225613+0000","last_fullsized":"2024-08-03T21:00:51.225613+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_clean_scrub_stamp":"2024-08-03T21:00:49.195479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-04T23:22:25.975528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.19","version":"0'0","reported_seq":15,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-08-03T21:00:51.938739+0000","last_change":"2024-08-03T21:00:49.228986+0000","last_active":"2024-08-03T21:00:51.938739+0000","last_peered":"2024-08-03T21:00:51.938739+0000","last_clean":"2024-08-03T21:00:51.938739+0000","last_became_active":"2024-08-03T21:00:49.228761+0000","last_became_peered":"2024-08-03T21:00:49.228761+0000","last_unstale":"2024-08-03T21:00:51.938739+0000","last_undegraded":"2024-08-03T21:00:51.938739+0000","last_fullsized":"2024-08-03T21:00:51.938739+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:48.192422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:48.192422+0000","last_clean_scrub_stamp":"2024-08-03T21:00:48.192422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-04T23:53:41.240790+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.19","version":"0'0","reported_seq":13,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-08-03T21:00:51.940174+0000","last_change":"2024-08-03T21:00:50.235335+0000","last_active":"2024-08-03T21:00:51.940174+0000","last_peered":"2024-08-03T21:00:51.940174+0000","last_clean":"2024-08-03T21:00:51.940174+0000","last_became_active":"2024-08-03T21:00:50.234940+0000","last_became_peered":"2024-08-03T21:00:50.234940+0000","last_unstale":"2024-08-03T21:00:51.940174+0000","last_undegraded":"2024-08-03T21:00:51.940174+0000","last_fullsized":"2024-08-03T21:00:51.940174+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_clean_scrub_stamp":"2024-08-03T21:00:49.195479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T00:40:44.403536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.18","version":"0'0","reported_seq":15,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-08-03T21:00:51.937778+0000","last_change":"2024-08-03T21:00:49.236558+0000","last_active":"2024-08-03T21:00:51.937778+0000","last_peered":"2024-08-03T21:00:51.937778+0000","last_clean":"2024-08-03T21:00:51.937778+0000","last_became_active":"2024-08-03T21:00:49.236201+0000","last_became_peered":"2024-08-03T21:00:49.236201+0000","last_unstale":"2024-08-03T21:00:51.937778+0000","last_undegraded":"2024-08-03T21:00:51.937778+0000","last_fullsized":"2024-08-03T21:00:51.937778+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:48.192422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:48.192422+0000","last_clean_scrub_stamp":"2024-08-03T21:00:48.192422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T08:11:10.436293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.16","version":"0'0","reported_seq":13,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-08-03T21:00:51.939374+0000","last_change":"2024-08-03T21:00:50.244082+0000","last_active":"2024-08-03T21:00:51.939374+0000","last_peered":"2024-08-03T21:00:51.939374+0000","last_clean":"2024-08-03T21:00:51.939374+0000","last_became_active":"2024-08-03T21:00:50.239536+0000","last_became_peered":"2024-08-03T21:00:50.239536+0000","last_unstale":"2024-08-03T21:00:51.939374+0000","last_undegraded":"2024-08-03T21:00:51.939374+0000","last_fullsized":"2024-08-03T21:00:51.939374+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_clean_scrub_stamp":"2024-08-03T21:00:49.195479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T06:38:59.823707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.17","version":"0'0","reported_seq":15,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-08-03T21:00:51.939140+0000","last_change":"2024-08-03T21:00:49.226248+0000","last_active":"2024-08-03T21:00:51.939140+0000","last_peered":"2024-08-03T21:00:51.939140+0000","last_clean":"2024-08-03T21:00:51.939140+0000","last_became_active":"2024-08-03T21:00:49.224012+0000","last_became_peered":"2024-08-03T21:00:49.224012+0000","last_unstale":"2024-08-03T21:00:51.939140+0000","last_undegraded":"2024-08-03T21:00:51.939140+0000","last_fullsized":"2024-08-03T21:00:51.939140+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:48.192422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:48.192422+0000","last_clean_scrub_stamp":"2024-08-03T21:00:48.192422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T01:08:20.166085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.17","version":"0'0","reported_seq":13,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-08-03T21:00:51.940036+0000","last_change":"2024-08-03T21:00:50.244964+0000","last_active":"2024-08-03T21:00:51.940036+0000","last_peered":"2024-08-03T21:00:51.940036+0000","last_clean":"2024-08-03T21:00:51.940036+0000","last_became_active":"2024-08-03T21:00:50.244323+0000","last_became_peered":"2024-08-03T21:00:50.244323+0000","last_unstale":"2024-08-03T21:00:51.940036+0000","last_undegraded":"2024-08-03T21:00:51.940036+0000","last_fullsized":"2024-08-03T21:00:51.940036+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_clean_scrub_stamp":"2024-08-03T21:00:49.195479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-04T21:46:43.719103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.16","version":"0'0","reported_seq":15,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-08-03T21:00:51.225728+0000","last_change":"2024-08-03T21:00:49.233357+0000","last_active":"2024-08-03T21:00:51.225728+0000","last_peered":"2024-08-03T21:00:51.225728+0000","last_clean":"2024-08-03T21:00:51.225728+0000","last_became_active":"2024-08-03T21:00:49.232477+0000","last_became_peered":"2024-08-03T21:00:49.232477+0000","last_unstale":"2024-08-03T21:00:51.225728+0000","last_undegraded":"2024-08-03T21:00:51.225728+0000","last_fullsized":"2024-08-03T21:00:51.225728+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:48.192422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:48.192422+0000","last_clean_scrub_stamp":"2024-08-03T21:00:48.192422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T05:40:56.485784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.14","version":"0'0","reported_seq":13,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-08-03T21:00:51.939653+0000","last_change":"2024-08-03T21:00:50.247566+0000","last_active":"2024-08-03T21:00:51.939653+0000","last_peered":"2024-08-03T21:00:51.939653+0000","last_clean":"2024-08-03T21:00:51.939653+0000","last_became_active":"2024-08-03T21:00:50.247175+0000","last_became_peered":"2024-08-03T21:00:50.247175+0000","last_unstale":"2024-08-03T21:00:51.939653+0000","last_undegraded":"2024-08-03T21:00:51.939653+0000","last_fullsized":"2024-08-03T21:00:51.939653+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_clean_scrub_stamp":"2024-08-03T21:00:49.195479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T03:52:56.414014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.15","version":"134'3","reported_seq":18,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-08-03T21:00:51.227301+0000","last_change":"2024-08-03T21:00:49.224739+0000","last_active":"2024-08-03T21:00:51.227301+0000","last_peered":"2024-08-03T21:00:51.227301+0000","last_clean":"2024-08-03T21:00:51.227301+0000","last_became_active":"2024-08-03T21:00:49.224342+0000","last_became_peered":"2024-08-03T21:00:49.224342+0000","last_unstale":"2024-08-03T21:00:51.227301+0000","last_undegraded":"2024-08-03T21:00:51.227301+0000","last_fullsized":"2024-08-03T21:00:51.227301+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:48.192422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:48.192422+0000","last_clean_scrub_stamp":"2024-08-03T21:00:48.192422+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T06:55:23.113259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.15","version":"0'0","reported_seq":13,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-08-03T21:00:51.227168+0000","last_change":"2024-08-03T21:00:50.236337+0000","last_active":"2024-08-03T21:00:51.227168+0000","last_peered":"2024-08-03T21:00:51.227168+0000","last_clean":"2024-08-03T21:00:51.227168+0000","last_became_active":"2024-08-03T21:00:50.235968+0000","last_became_peered":"2024-08-03T21:00:50.235968+0000","last_unstale":"2024-08-03T21:00:51.227168+0000","last_undegraded":"2024-08-03T21:00:51.227168+0000","last_fullsized":"2024-08-03T21:00:51.227168+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_clean_scrub_stamp":"2024-08-03T21:00:49.195479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T05:41:07.172696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.14","version":"134'6","reported_seq":21,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-08-03T21:00:51.939402+0000","last_change":"2024-08-03T21:00:49.235410+0000","last_active":"2024-08-03T21:00:51.939402+0000","last_peered":"2024-08-03T21:00:51.939402+0000","last_clean":"2024-08-03T21:00:51.939402+0000","last_became_active":"2024-08-03T21:00:49.234999+0000","last_became_peered":"2024-08-03T21:00:49.234999+0000","last_unstale":"2024-08-03T21:00:51.939402+0000","last_undegraded":"2024-08-03T21:00:51.939402+0000","last_fullsized":"2024-08-03T21:00:51.939402+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:48.192422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:48.192422+0000","last_clean_scrub_stamp":"2024-08-03T21:00:48.192422+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T02:55:58.470527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.12","version":"0'0","reported_seq":13,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-08-03T21:00:51.225979+0000","last_change":"2024-08-03T21:00:50.218292+0000","last_active":"2024-08-03T21:00:51.225979+0000","last_peered":"2024-08-03T21:00:51.225979+0000","last_clean":"2024-08-03T21:00:51.225979+0000","last_became_active":"2024-08-03T21:00:50.217979+0000","last_became_peered":"2024-08-03T21:00:50.217979+0000","last_unstale":"2024-08-03T21:00:51.225979+0000","last_undegraded":"2024-08-03T21:00:51.225979+0000","last_fullsized":"2024-08-03T21:00:51.225979+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_clean_scrub_stamp":"2024-08-03T21:00:49.195479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T06:55:48.820648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.13","version":"134'2","reported_seq":17,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-08-03T21:00:51.939709+0000","last_change":"2024-08-03T21:00:49.224867+0000","last_active":"2024-08-03T21:00:51.939709+0000","last_peered":"2024-08-03T21:00:51.939709+0000","last_clean":"2024-08-03T21:00:51.939709+0000","last_became_active":"2024-08-03T21:00:49.224555+0000","last_became_peered":"2024-08-03T21:00:49.224555+0000","last_unstale":"2024-08-03T21:00:51.939709+0000","last_undegraded":"2024-08-03T21:00:51.939709+0000","last_fullsized":"2024-08-03T21:00:51.939709+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:48.192422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:48.192422+0000","last_clean_scrub_stamp":"2024-08-03T21:00:48.192422+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T07:55:25.225889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.13","version":"0'0","reported_seq":13,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-08-03T21:00:51.938720+0000","last_change":"2024-08-03T21:00:50.247701+0000","last_active":"2024-08-03T21:00:51.938720+0000","last_peered":"2024-08-03T21:00:51.938720+0000","last_clean":"2024-08-03T21:00:51.938720+0000","last_became_active":"2024-08-03T21:00:50.247414+0000","last_became_peered":"2024-08-03T21:00:50.247414+0000","last_unstale":"2024-08-03T21:00:51.938720+0000","last_undegraded":"2024-08-03T21:00:51.938720+0000","last_fullsized":"2024-08-03T21:00:51.938720+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_clean_scrub_stamp":"2024-08-03T21:00:49.195479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T05:44:39.085990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.12","version":"0'0","reported_seq":15,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-08-03T21:00:51.938928+0000","last_change":"2024-08-03T21:00:49.224078+0000","last_active":"2024-08-03T21:00:51.938928+0000","last_peered":"2024-08-03T21:00:51.938928+0000","last_clean":"2024-08-03T21:00:51.938928+0000","last_became_active":"2024-08-03T21:00:49.223717+0000","last_became_peered":"2024-08-03T21:00:49.223717+0000","last_unstale":"2024-08-03T21:00:51.938928+0000","last_undegraded":"2024-08-03T21:00:51.938928+0000","last_fullsized":"2024-08-03T21:00:51.938928+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:48.192422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:48.192422+0000","last_clean_scrub_stamp":"2024-08-03T21:00:48.192422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T04:56:22.864880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.10","version":"0'0","reported_seq":13,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-08-03T21:00:51.939036+0000","last_change":"2024-08-03T21:00:50.231767+0000","last_active":"2024-08-03T21:00:51.939036+0000","last_peered":"2024-08-03T21:00:51.939036+0000","last_clean":"2024-08-03T21:00:51.939036+0000","last_became_active":"2024-08-03T21:00:50.229726+0000","last_became_peered":"2024-08-03T21:00:50.229726+0000","last_unstale":"2024-08-03T21:00:51.939036+0000","last_undegraded":"2024-08-03T21:00:51.939036+0000","last_fullsized":"2024-08-03T21:00:51.939036+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_clean_scrub_stamp":"2024-08-03T21:00:49.195479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T00:14:18.409160+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.11","version":"0'0","reported_seq":15,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-08-03T21:00:51.939969+0000","last_change":"2024-08-03T21:00:49.219205+0000","last_active":"2024-08-03T21:00:51.939969+0000","last_peered":"2024-08-03T21:00:51.939969+0000","last_clean":"2024-08-03T21:00:51.939969+0000","last_became_active":"2024-08-03T21:00:49.218855+0000","last_became_peered":"2024-08-03T21:00:49.218855+0000","last_unstale":"2024-08-03T21:00:51.939969+0000","last_undegraded":"2024-08-03T21:00:51.939969+0000","last_fullsized":"2024-08-03T21:00:51.939969+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:48.192422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:48.192422+0000","last_clean_scrub_stamp":"2024-08-03T21:00:48.192422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T00:24:17.141394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.11","version":"0'0","reported_seq":13,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-08-03T21:00:51.939150+0000","last_change":"2024-08-03T21:00:50.230042+0000","last_active":"2024-08-03T21:00:51.939150+0000","last_peered":"2024-08-03T21:00:51.939150+0000","last_clean":"2024-08-03T21:00:51.939150+0000","last_became_active":"2024-08-03T21:00:50.229716+0000","last_became_peered":"2024-08-03T21:00:50.229716+0000","last_unstale":"2024-08-03T21:00:51.939150+0000","last_undegraded":"2024-08-03T21:00:51.939150+0000","last_fullsized":"2024-08-03T21:00:51.939150+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_clean_scrub_stamp":"2024-08-03T21:00:49.195479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T08:05:48.492097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.10","version":"134'8","reported_seq":23,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-08-03T21:00:51.937904+0000","last_change":"2024-08-03T21:00:49.219610+0000","last_active":"2024-08-03T21:00:51.937904+0000","last_peered":"2024-08-03T21:00:51.937904+0000","last_clean":"2024-08-03T21:00:51.937904+0000","last_became_active":"2024-08-03T21:00:49.219289+0000","last_became_peered":"2024-08-03T21:00:49.219289+0000","last_unstale":"2024-08-03T21:00:51.937904+0000","last_undegraded":"2024-08-03T21:00:51.937904+0000","last_fullsized":"2024-08-03T21:00:51.937904+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:48.192422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:48.192422+0000","last_clean_scrub_stamp":"2024-08-03T21:00:48.192422+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T01:32:44.783771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.f","version":"0'0","reported_seq":13,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-08-03T21:00:51.939337+0000","last_change":"2024-08-03T21:00:50.236210+0000","last_active":"2024-08-03T21:00:51.939337+0000","last_peered":"2024-08-03T21:00:51.939337+0000","last_clean":"2024-08-03T21:00:51.939337+0000","last_became_active":"2024-08-03T21:00:50.233503+0000","last_became_peered":"2024-08-03T21:00:50.233503+0000","last_unstale":"2024-08-03T21:00:51.939337+0000","last_undegraded":"2024-08-03T21:00:51.939337+0000","last_fullsized":"2024-08-03T21:00:51.939337+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_clean_scrub_stamp":"2024-08-03T21:00:49.195479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-04T23:18:44.490549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.e","version":"134'2","reported_seq":17,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-08-03T21:00:51.939110+0000","last_change":"2024-08-03T21:00:49.221709+0000","last_active":"2024-08-03T21:00:51.939110+0000","last_peered":"2024-08-03T21:00:51.939110+0000","last_clean":"2024-08-03T21:00:51.939110+0000","last_became_active":"2024-08-03T21:00:49.221146+0000","last_became_peered":"2024-08-03T21:00:49.221146+0000","last_unstale":"2024-08-03T21:00:51.939110+0000","last_undegraded":"2024-08-03T21:00:51.939110+0000","last_fullsized":"2024-08-03T21:00:51.939110+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:48.192422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:48.192422+0000","last_clean_scrub_stamp":"2024-08-03T21:00:48.192422+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T02:29:53.585049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.c","version":"0'0","reported_seq":13,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-08-03T21:00:51.940242+0000","last_change":"2024-08-03T21:00:50.240090+0000","last_active":"2024-08-03T21:00:51.940242+0000","last_peered":"2024-08-03T21:00:51.940242+0000","last_clean":"2024-08-03T21:00:51.940242+0000","last_became_active":"2024-08-03T21:00:50.239718+0000","last_became_peered":"2024-08-03T21:00:50.239718+0000","last_unstale":"2024-08-03T21:00:51.940242+0000","last_undegraded":"2024-08-03T21:00:51.940242+0000","last_fullsized":"2024-08-03T21:00:51.940242+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_clean_scrub_stamp":"2024-08-03T21:00:49.195479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T07:44:14.161399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.d","version":"134'1","reported_seq":16,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-08-03T21:00:51.939184+0000","last_change":"2024-08-03T21:00:49.214725+0000","last_active":"2024-08-03T21:00:51.939184+0000","last_peered":"2024-08-03T21:00:51.939184+0000","last_clean":"2024-08-03T21:00:51.939184+0000","last_became_active":"2024-08-03T21:00:49.214155+0000","last_became_peered":"2024-08-03T21:00:49.214155+0000","last_unstale":"2024-08-03T21:00:51.939184+0000","last_undegraded":"2024-08-03T21:00:51.939184+0000","last_fullsized":"2024-08-03T21:00:51.939184+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:48.192422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:48.192422+0000","last_clean_scrub_stamp":"2024-08-03T21:00:48.192422+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T07:44:13.158341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.d","version":"0'0","reported_seq":13,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-08-03T21:00:51.227187+0000","last_change":"2024-08-03T21:00:50.228406+0000","last_active":"2024-08-03T21:00:51.227187+0000","last_peered":"2024-08-03T21:00:51.227187+0000","last_clean":"2024-08-03T21:00:51.227187+0000","last_became_active":"2024-08-03T21:00:50.227965+0000","last_became_peered":"2024-08-03T21:00:50.227965+0000","last_unstale":"2024-08-03T21:00:51.227187+0000","last_undegraded":"2024-08-03T21:00:51.227187+0000","last_fullsized":"2024-08-03T21:00:51.227187+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_clean_scrub_stamp":"2024-08-03T21:00:49.195479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T03:49:21.574825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.c","version":"0'0","reported_seq":15,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-08-03T21:00:51.938518+0000","last_change":"2024-08-03T21:00:49.225810+0000","last_active":"2024-08-03T21:00:51.938518+0000","last_peered":"2024-08-03T21:00:51.938518+0000","last_clean":"2024-08-03T21:00:51.938518+0000","last_became_active":"2024-08-03T21:00:49.225501+0000","last_became_peered":"2024-08-03T21:00:49.225501+0000","last_unstale":"2024-08-03T21:00:51.938518+0000","last_undegraded":"2024-08-03T21:00:51.938518+0000","last_fullsized":"2024-08-03T21:00:51.938518+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:48.192422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:48.192422+0000","last_clean_scrub_stamp":"2024-08-03T21:00:48.192422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T06:55:23.113259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.a","version":"0'0","reported_seq":13,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-08-03T21:00:51.938230+0000","last_change":"2024-08-03T21:00:50.238896+0000","last_active":"2024-08-03T21:00:51.938230+0000","last_peered":"2024-08-03T21:00:51.938230+0000","last_clean":"2024-08-03T21:00:51.938230+0000","last_became_active":"2024-08-03T21:00:50.238557+0000","last_became_peered":"2024-08-03T21:00:50.238557+0000","last_unstale":"2024-08-03T21:00:51.938230+0000","last_undegraded":"2024-08-03T21:00:51.938230+0000","last_fullsized":"2024-08-03T21:00:51.938230+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_clean_scrub_stamp":"2024-08-03T21:00:49.195479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-04T22:10:38.257913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.b","version":"134'2","reported_seq":17,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-08-03T21:00:51.938675+0000","last_change":"2024-08-03T21:00:49.233291+0000","last_active":"2024-08-03T21:00:51.938675+0000","last_peered":"2024-08-03T21:00:51.938675+0000","last_clean":"2024-08-03T21:00:51.938675+0000","last_became_active":"2024-08-03T21:00:49.232973+0000","last_became_peered":"2024-08-03T21:00:49.232973+0000","last_unstale":"2024-08-03T21:00:51.938675+0000","last_undegraded":"2024-08-03T21:00:51.938675+0000","last_fullsized":"2024-08-03T21:00:51.938675+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:48.192422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:48.192422+0000","last_clean_scrub_stamp":"2024-08-03T21:00:48.192422+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T05:33:33.972534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.b","version":"0'0","reported_seq":13,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-08-03T21:00:51.939647+0000","last_change":"2024-08-03T21:00:50.242588+0000","last_active":"2024-08-03T21:00:51.939647+0000","last_peered":"2024-08-03T21:00:51.939647+0000","last_clean":"2024-08-03T21:00:51.939647+0000","last_became_active":"2024-08-03T21:00:50.239795+0000","last_became_peered":"2024-08-03T21:00:50.239795+0000","last_unstale":"2024-08-03T21:00:51.939647+0000","last_undegraded":"2024-08-03T21:00:51.939647+0000","last_fullsized":"2024-08-03T21:00:51.939647+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_clean_scrub_stamp":"2024-08-03T21:00:49.195479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T06:53:11.234357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.a","version":"0'0","reported_seq":15,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-08-03T21:00:51.940019+0000","last_change":"2024-08-03T21:00:49.222066+0000","last_active":"2024-08-03T21:00:51.940019+0000","last_peered":"2024-08-03T21:00:51.940019+0000","last_clean":"2024-08-03T21:00:51.940019+0000","last_became_active":"2024-08-03T21:00:49.221757+0000","last_became_peered":"2024-08-03T21:00:49.221757+0000","last_unstale":"2024-08-03T21:00:51.940019+0000","last_undegraded":"2024-08-03T21:00:51.940019+0000","last_fullsized":"2024-08-03T21:00:51.940019+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:48.192422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:48.192422+0000","last_clean_scrub_stamp":"2024-08-03T21:00:48.192422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T01:29:44.046851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.8","version":"0'0","reported_seq":13,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-08-03T21:00:51.937406+0000","last_change":"2024-08-03T21:00:50.240625+0000","last_active":"2024-08-03T21:00:51.937406+0000","last_peered":"2024-08-03T21:00:51.937406+0000","last_clean":"2024-08-03T21:00:51.937406+0000","last_became_active":"2024-08-03T21:00:50.240139+0000","last_became_peered":"2024-08-03T21:00:50.240139+0000","last_unstale":"2024-08-03T21:00:51.937406+0000","last_undegraded":"2024-08-03T21:00:51.937406+0000","last_fullsized":"2024-08-03T21:00:51.937406+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_clean_scrub_stamp":"2024-08-03T21:00:49.195479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T06:58:48.625406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.9","version":"0'0","reported_seq":15,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-08-03T21:00:51.938935+0000","last_change":"2024-08-03T21:00:49.235212+0000","last_active":"2024-08-03T21:00:51.938935+0000","last_peered":"2024-08-03T21:00:51.938935+0000","last_clean":"2024-08-03T21:00:51.938935+0000","last_became_active":"2024-08-03T21:00:49.234793+0000","last_became_peered":"2024-08-03T21:00:49.234793+0000","last_unstale":"2024-08-03T21:00:51.938935+0000","last_undegraded":"2024-08-03T21:00:51.938935+0000","last_fullsized":"2024-08-03T21:00:51.938935+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:48.192422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:48.192422+0000","last_clean_scrub_stamp":"2024-08-03T21:00:48.192422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T05:44:38.082933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.9","version":"0'0","reported_seq":13,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-08-03T21:00:51.938543+0000","last_change":"2024-08-03T21:00:50.226039+0000","last_active":"2024-08-03T21:00:51.938543+0000","last_peered":"2024-08-03T21:00:51.938543+0000","last_clean":"2024-08-03T21:00:51.938543+0000","last_became_active":"2024-08-03T21:00:50.224881+0000","last_became_peered":"2024-08-03T21:00:50.224881+0000","last_unstale":"2024-08-03T21:00:51.938543+0000","last_undegraded":"2024-08-03T21:00:51.938543+0000","last_fullsized":"2024-08-03T21:00:51.938543+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_clean_scrub_stamp":"2024-08-03T21:00:49.195479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T03:51:01.888967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.8","version":"0'0","reported_seq":15,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-08-03T21:00:51.937513+0000","last_change":"2024-08-03T21:00:49.230326+0000","last_active":"2024-08-03T21:00:51.937513+0000","last_peered":"2024-08-03T21:00:51.937513+0000","last_clean":"2024-08-03T21:00:51.937513+0000","last_became_active":"2024-08-03T21:00:49.229867+0000","last_became_peered":"2024-08-03T21:00:49.229867+0000","last_unstale":"2024-08-03T21:00:51.937513+0000","last_undegraded":"2024-08-03T21:00:51.937513+0000","last_fullsized":"2024-08-03T21:00:51.937513+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:48.192422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:48.192422+0000","last_clean_scrub_stamp":"2024-08-03T21:00:48.192422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T01:53:59.097881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.6","version":"0'0","reported_seq":13,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-08-03T21:00:51.939434+0000","last_change":"2024-08-03T21:00:50.245369+0000","last_active":"2024-08-03T21:00:51.939434+0000","last_peered":"2024-08-03T21:00:51.939434+0000","last_clean":"2024-08-03T21:00:51.939434+0000","last_became_active":"2024-08-03T21:00:50.244895+0000","last_became_peered":"2024-08-03T21:00:50.244895+0000","last_unstale":"2024-08-03T21:00:51.939434+0000","last_undegraded":"2024-08-03T21:00:51.939434+0000","last_fullsized":"2024-08-03T21:00:51.939434+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_clean_scrub_stamp":"2024-08-03T21:00:49.195479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T08:08:18.583196+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.7","version":"134'2","reported_seq":17,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-08-03T21:00:51.940273+0000","last_change":"2024-08-03T21:00:49.220966+0000","last_active":"2024-08-03T21:00:51.940273+0000","last_peered":"2024-08-03T21:00:51.940273+0000","last_clean":"2024-08-03T21:00:51.940273+0000","last_became_active":"2024-08-03T21:00:49.220433+0000","last_became_peered":"2024-08-03T21:00:49.220433+0000","last_unstale":"2024-08-03T21:00:51.940273+0000","last_undegraded":"2024-08-03T21:00:51.940273+0000","last_fullsized":"2024-08-03T21:00:51.940273+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:48.192422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:48.192422+0000","last_clean_scrub_stamp":"2024-08-03T21:00:48.192422+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T02:44:18.436142+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.7","version":"0'0","reported_seq":13,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-08-03T21:00:51.939167+0000","last_change":"2024-08-03T21:00:50.232102+0000","last_active":"2024-08-03T21:00:51.939167+0000","last_peered":"2024-08-03T21:00:51.939167+0000","last_clean":"2024-08-03T21:00:51.939167+0000","last_became_active":"2024-08-03T21:00:50.231742+0000","last_became_peered":"2024-08-03T21:00:50.231742+0000","last_unstale":"2024-08-03T21:00:51.939167+0000","last_undegraded":"2024-08-03T21:00:51.939167+0000","last_fullsized":"2024-08-03T21:00:51.939167+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_clean_scrub_stamp":"2024-08-03T21:00:49.195479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T01:29:45.049908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.6","version":"135'1","reported_seq":16,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-08-03T21:00:51.939543+0000","last_change":"2024-08-03T21:00:49.235126+0000","last_active":"2024-08-03T21:00:51.939543+0000","last_peered":"2024-08-03T21:00:51.939543+0000","last_clean":"2024-08-03T21:00:51.939543+0000","last_became_active":"2024-08-03T21:00:49.233960+0000","last_became_peered":"2024-08-03T21:00:49.233960+0000","last_unstale":"2024-08-03T21:00:51.939543+0000","last_undegraded":"2024-08-03T21:00:51.939543+0000","last_fullsized":"2024-08-03T21:00:51.939543+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:48.192422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:48.192422+0000","last_clean_scrub_stamp":"2024-08-03T21:00:48.192422+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T06:11:55.068764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.4","version":"0'0","reported_seq":13,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-08-03T21:00:51.937871+0000","last_change":"2024-08-03T21:00:50.237044+0000","last_active":"2024-08-03T21:00:51.937871+0000","last_peered":"2024-08-03T21:00:51.937871+0000","last_clean":"2024-08-03T21:00:51.937871+0000","last_became_active":"2024-08-03T21:00:50.236738+0000","last_became_peered":"2024-08-03T21:00:50.236738+0000","last_unstale":"2024-08-03T21:00:51.937871+0000","last_undegraded":"2024-08-03T21:00:51.937871+0000","last_fullsized":"2024-08-03T21:00:51.937871+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_clean_scrub_stamp":"2024-08-03T21:00:49.195479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T07:43:14.289706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.5","version":"0'0","reported_seq":15,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-08-03T21:00:51.938775+0000","last_change":"2024-08-03T21:00:49.227879+0000","last_active":"2024-08-03T21:00:51.938775+0000","last_peered":"2024-08-03T21:00:51.938775+0000","last_clean":"2024-08-03T21:00:51.938775+0000","last_became_active":"2024-08-03T21:00:49.227574+0000","last_became_peered":"2024-08-03T21:00:49.227574+0000","last_unstale":"2024-08-03T21:00:51.938775+0000","last_undegraded":"2024-08-03T21:00:51.938775+0000","last_fullsized":"2024-08-03T21:00:51.938775+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:48.192422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:48.192422+0000","last_clean_scrub_stamp":"2024-08-03T21:00:48.192422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T08:56:28.307290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.5","version":"0'0","reported_seq":13,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-08-03T21:00:51.938535+0000","last_change":"2024-08-03T21:00:50.228194+0000","last_active":"2024-08-03T21:00:51.938535+0000","last_peered":"2024-08-03T21:00:51.938535+0000","last_clean":"2024-08-03T21:00:51.938535+0000","last_became_active":"2024-08-03T21:00:50.226123+0000","last_became_peered":"2024-08-03T21:00:50.226123+0000","last_unstale":"2024-08-03T21:00:51.938535+0000","last_undegraded":"2024-08-03T21:00:51.938535+0000","last_fullsized":"2024-08-03T21:00:51.938535+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_clean_scrub_stamp":"2024-08-03T21:00:49.195479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T08:22:58.480315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.4","version":"0'0","reported_seq":15,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-08-03T21:00:51.939696+0000","last_change":"2024-08-03T21:00:49.221928+0000","last_active":"2024-08-03T21:00:51.939696+0000","last_peered":"2024-08-03T21:00:51.939696+0000","last_clean":"2024-08-03T21:00:51.939696+0000","last_became_active":"2024-08-03T21:00:49.221623+0000","last_became_peered":"2024-08-03T21:00:49.221623+0000","last_unstale":"2024-08-03T21:00:51.939696+0000","last_undegraded":"2024-08-03T21:00:51.939696+0000","last_fullsized":"2024-08-03T21:00:51.939696+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:48.192422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:48.192422+0000","last_clean_scrub_stamp":"2024-08-03T21:00:48.192422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T07:27:35.968034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.2","version":"0'0","reported_seq":13,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-08-03T21:00:51.938470+0000","last_change":"2024-08-03T21:00:50.232514+0000","last_active":"2024-08-03T21:00:51.938470+0000","last_peered":"2024-08-03T21:00:51.938470+0000","last_clean":"2024-08-03T21:00:51.938470+0000","last_became_active":"2024-08-03T21:00:50.232148+0000","last_became_peered":"2024-08-03T21:00:50.232148+0000","last_unstale":"2024-08-03T21:00:51.938470+0000","last_undegraded":"2024-08-03T21:00:51.938470+0000","last_fullsized":"2024-08-03T21:00:51.938470+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_clean_scrub_stamp":"2024-08-03T21:00:49.195479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-04T23:42:52.232285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.3","version":"0'0","reported_seq":15,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-08-03T21:00:51.938780+0000","last_change":"2024-08-03T21:00:49.235356+0000","last_active":"2024-08-03T21:00:51.938780+0000","last_peered":"2024-08-03T21:00:51.938780+0000","last_clean":"2024-08-03T21:00:51.938780+0000","last_became_active":"2024-08-03T21:00:49.234801+0000","last_became_peered":"2024-08-03T21:00:49.234801+0000","last_unstale":"2024-08-03T21:00:51.938780+0000","last_undegraded":"2024-08-03T21:00:51.938780+0000","last_fullsized":"2024-08-03T21:00:51.938780+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:48.192422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:48.192422+0000","last_clean_scrub_stamp":"2024-08-03T21:00:48.192422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T05:45:41.794416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.3","version":"0'0","reported_seq":13,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-08-03T21:00:51.939532+0000","last_change":"2024-08-03T21:00:50.233609+0000","last_active":"2024-08-03T21:00:51.939532+0000","last_peered":"2024-08-03T21:00:51.939532+0000","last_clean":"2024-08-03T21:00:51.939532+0000","last_became_active":"2024-08-03T21:00:50.231678+0000","last_became_peered":"2024-08-03T21:00:50.231678+0000","last_unstale":"2024-08-03T21:00:51.939532+0000","last_undegraded":"2024-08-03T21:00:51.939532+0000","last_fullsized":"2024-08-03T21:00:51.939532+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_clean_scrub_stamp":"2024-08-03T21:00:49.195479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T02:34:58.346689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.2","version":"0'0","reported_seq":15,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-08-03T21:00:51.938710+0000","last_change":"2024-08-03T21:00:49.218982+0000","last_active":"2024-08-03T21:00:51.938710+0000","last_peered":"2024-08-03T21:00:51.938710+0000","last_clean":"2024-08-03T21:00:51.938710+0000","last_became_active":"2024-08-03T21:00:49.217673+0000","last_became_peered":"2024-08-03T21:00:49.217673+0000","last_unstale":"2024-08-03T21:00:51.938710+0000","last_undegraded":"2024-08-03T21:00:51.938710+0000","last_fullsized":"2024-08-03T21:00:51.938710+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:48.192422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:48.192422+0000","last_clean_scrub_stamp":"2024-08-03T21:00:48.192422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T02:44:09.712432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.0","version":"0'0","reported_seq":13,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-08-03T21:00:51.938141+0000","last_change":"2024-08-03T21:00:50.234121+0000","last_active":"2024-08-03T21:00:51.938141+0000","last_peered":"2024-08-03T21:00:51.938141+0000","last_clean":"2024-08-03T21:00:51.938141+0000","last_became_active":"2024-08-03T21:00:50.233758+0000","last_became_peered":"2024-08-03T21:00:50.233758+0000","last_unstale":"2024-08-03T21:00:51.938141+0000","last_undegraded":"2024-08-03T21:00:51.938141+0000","last_fullsized":"2024-08-03T21:00:51.938141+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_clean_scrub_stamp":"2024-08-03T21:00:49.195479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T05:28:59.859843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.1","version":"0'0","reported_seq":15,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-08-03T21:00:51.225900+0000","last_change":"2024-08-03T21:00:49.230098+0000","last_active":"2024-08-03T21:00:51.225900+0000","last_peered":"2024-08-03T21:00:51.225900+0000","last_clean":"2024-08-03T21:00:51.225900+0000","last_became_active":"2024-08-03T21:00:49.229777+0000","last_became_peered":"2024-08-03T21:00:49.229777+0000","last_unstale":"2024-08-03T21:00:51.225900+0000","last_undegraded":"2024-08-03T21:00:51.225900+0000","last_fullsized":"2024-08-03T21:00:51.225900+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:48.192422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:48.192422+0000","last_clean_scrub_stamp":"2024-08-03T21:00:48.192422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T06:34:18.629210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.1","version":"0'0","reported_seq":13,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-08-03T21:00:51.938573+0000","last_change":"2024-08-03T21:00:50.234536+0000","last_active":"2024-08-03T21:00:51.938573+0000","last_peered":"2024-08-03T21:00:51.938573+0000","last_clean":"2024-08-03T21:00:51.938573+0000","last_became_active":"2024-08-03T21:00:50.233723+0000","last_became_peered":"2024-08-03T21:00:50.233723+0000","last_unstale":"2024-08-03T21:00:51.938573+0000","last_undegraded":"2024-08-03T21:00:51.938573+0000","last_fullsized":"2024-08-03T21:00:51.938573+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_clean_scrub_stamp":"2024-08-03T21:00:49.195479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T00:02:57.081976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.0","version":"0'0","reported_seq":15,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-08-03T21:00:51.938216+0000","last_change":"2024-08-03T21:00:49.231962+0000","last_active":"2024-08-03T21:00:51.938216+0000","last_peered":"2024-08-03T21:00:51.938216+0000","last_clean":"2024-08-03T21:00:51.938216+0000","last_became_active":"2024-08-03T21:00:49.231607+0000","last_became_peered":"2024-08-03T21:00:49.231607+0000","last_unstale":"2024-08-03T21:00:51.938216+0000","last_undegraded":"2024-08-03T21:00:51.938216+0000","last_fullsized":"2024-08-03T21:00:51.938216+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:48.192422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:48.192422+0000","last_clean_scrub_stamp":"2024-08-03T21:00:48.192422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T08:45:01.729460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.e","version":"0'0","reported_seq":13,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-08-03T21:00:51.939469+0000","last_change":"2024-08-03T21:00:50.249570+0000","last_active":"2024-08-03T21:00:51.939469+0000","last_peered":"2024-08-03T21:00:51.939469+0000","last_clean":"2024-08-03T21:00:51.939469+0000","last_became_active":"2024-08-03T21:00:50.240733+0000","last_became_peered":"2024-08-03T21:00:50.240733+0000","last_unstale":"2024-08-03T21:00:51.939469+0000","last_undegraded":"2024-08-03T21:00:51.939469+0000","last_fullsized":"2024-08-03T21:00:51.939469+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:49.195479+0000","last_clean_scrub_stamp":"2024-08-03T21:00:49.195479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T08:43:59.747967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.f","version":"134'1","reported_seq":16,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-08-03T21:00:51.226571+0000","last_change":"2024-08-03T21:00:49.234956+0000","last_active":"2024-08-03T21:00:51.226571+0000","last_peered":"2024-08-03T21:00:51.226571+0000","last_clean":"2024-08-03T21:00:51.226571+0000","last_became_active":"2024-08-03T21:00:49.234351+0000","last_became_peered":"2024-08-03T21:00:49.234351+0000","last_unstale":"2024-08-03T21:00:51.226571+0000","last_undegraded":"2024-08-03T21:00:51.226571+0000","last_fullsized":"2024-08-03T21:00:51.226571+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T21:00:48.192422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T21:00:48.192422+0000","last_clean_scrub_stamp":"2024-08-03T21:00:48.192422+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T00:43:49.078349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"16'72","reported_seq":331,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-08-03T21:00:51.226485+0000","last_change":"2024-08-03T20:47:30.547632+0000","last_active":"2024-08-03T21:00:51.226485+0000","last_peered":"2024-08-03T21:00:51.226485+0000","last_clean":"2024-08-03T21:00:51.226485+0000","last_became_active":"2024-08-03T20:47:30.546255+0000","last_became_peered":"2024-08-03T20:47:30.546255+0000","last_unstale":"2024-08-03T21:00:51.226485+0000","last_undegraded":"2024-08-03T21:00:51.226485+0000","last_fullsized":"2024-08-03T21:00:51.226485+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:47:28.874451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:47:28.874451+0000","last_clean_scrub_stamp":"2024-08-03T20:47:28.874451+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T00:19:40.548812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":15,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":14,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509604,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41776,"kb_used_data":2272,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330064,"statfs":{"total":96636764160,"available":96593985536,"internally_reserved":0,"allocated":2326528,"data_stored":1421385,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":40434122},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":6,"up_from":15,"seq":64424509605,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37032,"kb_used_data":1692,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334808,"statfs":{"total":96636764160,"available":96598843392,"internally_reserved":0,"allocated":1732608,"data_stored":831557,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":5,"up_from":15,"seq":64424509605,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37028,"kb_used_data":1688,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334812,"statfs":{"total":96636764160,"available":96598847488,"internally_reserved":0,"allocated":1728512,"data_stored":830975,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509604,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41184,"kb_used_data":1684,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330656,"statfs":{"total":96636764160,"available":96594591744,"internally_reserved":0,"allocated":1724416,"data_stored":830905,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509606,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":29292,"kb_used_data":2268,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94342548,"statfs":{"total":96636764160,"available":96606769152,"internally_reserved":0,"allocated":2322432,"data_stored":1421855,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542309,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37048,"kb_used_data":1704,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334792,"statfs":{"total":96636764160,"available":96598827008,"internally_reserved":0,"allocated":1744896,"data_stored":832641,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":3,"up_from":14,"seq":60129542307,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37044,"kb_used_data":1700,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334796,"statfs":{"total":96636764160,"available":96598831104,"internally_reserved":0,"allocated":1740800,"data_stored":832539,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":2,"up_from":14,"seq":60129542308,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41192,"kb_used_data":1688,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330648,"statfs":{"total":96636764160,"available":96594583552,"internally_reserved":0,"allocated":1728512,"data_stored":830939,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-08-03T21:01:00.332 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-08-03T21:01:00.332 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-08-03T21:01:00.604 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:00.602+0000 7f828ce00640 1 -- 172.21.15.32:0/4160395601 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f82880ced30 msgr2=0x7f82880cf140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:01:00.604 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:00.602+0000 7f828ce00640 1 --2- 172.21.15.32:0/4160395601 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f82880ced30 0x7f82880cf140 secure :-1 s=READY pgs=2183 cs=0 l=1 rev1=1 crypto rx=0x7f8280005fd0 tx=0x7f828002a8b0 comp rx=0 tx=0).stop 2024-08-03T21:01:00.604 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:00.602+0000 7f828ce00640 1 -- 172.21.15.32:0/4160395601 shutdown_connections 2024-08-03T21:01:00.604 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:00.602+0000 7f828ce00640 1 --2- 172.21.15.32:0/4160395601 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f82880ced30 0x7f82880cf140 unknown :-1 s=CLOSED pgs=2183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:01:00.605 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:00.602+0000 7f828ce00640 1 -- 172.21.15.32:0/4160395601 >> 172.21.15.32:0/4160395601 conn(0x7f82880b1bd0 msgr2=0x7f82880b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T21:01:00.605 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:00.602+0000 7f828ce00640 1 -- 172.21.15.32:0/4160395601 shutdown_connections 2024-08-03T21:01:00.605 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:00.602+0000 7f828ce00640 1 -- 172.21.15.32:0/4160395601 wait complete. 2024-08-03T21:01:00.606 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:00.604+0000 7f828ce00640 1 Processor -- start 2024-08-03T21:01:00.606 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:00.604+0000 7f828ce00640 1 -- start start 2024-08-03T21:01:00.608 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:00.606+0000 7f828ce00640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f82880cccf0 0x7f8288074020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:01:00.609 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:00.606+0000 7f828ce00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f82880ced30 0x7f82881e8f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:01:00.609 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:00.606+0000 7f828ce00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8288145c50 0x7f82881e9450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:01:00.609 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:00.606+0000 7f828ce00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f82880b4210 con 0x7f8288145c50 2024-08-03T21:01:00.609 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:00.606+0000 7f828ce00640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f82880b4090 con 0x7f82880ced30 2024-08-03T21:01:00.609 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:00.606+0000 7f828ce00640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7f82880b4390 con 0x7f82880cccf0 2024-08-03T21:01:00.609 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:00.606+0000 7f8286800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8288145c50 0x7f82881e9450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:01:00.609 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:00.606+0000 7f8286800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8288145c50 0x7f82881e9450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:50430/0 (socket says 172.21.15.32:50430) 2024-08-03T21:01:00.609 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:00.606+0000 7f8286800640 1 -- 172.21.15.32:0/3285612655 learned_addr learned my addr 172.21.15.32:0/3285612655 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T21:01:00.609 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:00.606+0000 7f827fe00640 1 --2- 172.21.15.32:0/3285612655 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f82880ced30 0x7f82881e8f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:01:00.609 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:00.607+0000 7f8285e00640 1 --2- 172.21.15.32:0/3285612655 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f82880cccf0 0x7f8288074020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:01:00.609 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:00.607+0000 7f8286800640 1 -- 172.21.15.32:0/3285612655 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f82880cccf0 msgr2=0x7f8288074020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:01:00.609 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:00.607+0000 7f8286800640 1 --2- 172.21.15.32:0/3285612655 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f82880cccf0 0x7f8288074020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:01:00.610 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:00.607+0000 7f8286800640 1 -- 172.21.15.32:0/3285612655 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f82880ced30 msgr2=0x7f82881e8f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:01:00.610 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:00.607+0000 7f8286800640 1 --2- 172.21.15.32:0/3285612655 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f82880ced30 0x7f82881e8f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:01:00.610 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:00.607+0000 7f8286800640 1 -- 172.21.15.32:0/3285612655 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8288072b20 con 0x7f8288145c50 2024-08-03T21:01:00.610 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:00.607+0000 7f8285e00640 1 --2- 172.21.15.32:0/3285612655 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f82880cccf0 0x7f8288074020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-03T21:01:00.610 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:00.607+0000 7f8286800640 1 --2- 172.21.15.32:0/3285612655 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8288145c50 0x7f82881e9450 secure :-1 s=READY pgs=2184 cs=0 l=1 rev1=1 crypto rx=0x7f8280006010 tx=0x7f8280003ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T21:01:00.611 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:00.607+0000 7f827e000640 1 -- 172.21.15.32:0/3285612655 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8280031b90 con 0x7f8288145c50 2024-08-03T21:01:00.611 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:00.607+0000 7f827e000640 1 -- 172.21.15.32:0/3285612655 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f8280031e20 con 0x7f8288145c50 2024-08-03T21:01:00.611 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:00.608+0000 7f828ce00640 1 -- 172.21.15.32:0/3285612655 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8288072e00 con 0x7f8288145c50 2024-08-03T21:01:00.611 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:00.608+0000 7f827e000640 1 -- 172.21.15.32:0/3285612655 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82800332f0 con 0x7f8288145c50 2024-08-03T21:01:00.611 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:00.608+0000 7f828ce00640 1 -- 172.21.15.32:0/3285612655 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f82880733c0 con 0x7f8288145c50 2024-08-03T21:01:00.611 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:00.609+0000 7f828ce00640 1 -- 172.21.15.32:0/3285612655 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f82880cf140 con 0x7f8288145c50 2024-08-03T21:01:00.611 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:00.610+0000 7f827e000640 1 -- 172.21.15.32:0/3285612655 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f8280031660 con 0x7f8288145c50 2024-08-03T21:01:00.612 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:00.611+0000 7f827e000640 1 --2- 172.21.15.32:0/3285612655 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f8264077040 0x7f8264079550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:01:00.612 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:00.611+0000 7f827e000640 1 -- 172.21.15.32:0/3285612655 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(135..135 src has 1..135) ==== 5709+0+0 (secure 0 0 0) 0x7f82800b9ec0 con 0x7f8288145c50 2024-08-03T21:01:00.613 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:00.611+0000 7f8285e00640 1 --2- 172.21.15.32:0/3285612655 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f8264077040 0x7f8264079550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:01:00.613 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:00.612+0000 7f8285e00640 1 --2- 172.21.15.32:0/3285612655 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f8264077040 0x7f8264079550 secure :-1 s=READY pgs=876 cs=0 l=1 rev1=1 crypto rx=0x7f82780046b0 tx=0x7f8278009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T21:01:00.617 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:00.615+0000 7f827e000640 1 -- 172.21.15.32:0/3285612655 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7f828003f030 con 0x7f8288145c50 2024-08-03T21:01:00.899 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:00.897+0000 7f828ce00640 1 -- 172.21.15.32:0/3285612655 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) -- 0x7f82880b23f0 con 0x7f8288145c50 2024-08-03T21:01:00.902 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:00.901+0000 7f827e000640 1 -- 172.21.15.32:0/3285612655 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v18) ==== 190+0+147 (secure 0 0 0) 0x7f8280083020 con 0x7f8288145c50 2024-08-03T21:01:00.902 INFO:teuthology.orchestra.run.smithi032.stdout:[client.0] 2024-08-03T21:01:00.902 INFO:teuthology.orchestra.run.smithi032.stdout: key = AQAwl65msgO3NRAAHG/WuMTK9c4lDXocXvqWnA== 2024-08-03T21:01:00.903 INFO:teuthology.orchestra.run.smithi032.stdout: caps mds = "allow" 2024-08-03T21:01:00.903 INFO:teuthology.orchestra.run.smithi032.stdout: caps mon = "allow r" 2024-08-03T21:01:00.903 INFO:teuthology.orchestra.run.smithi032.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2024-08-03T21:01:00.903 INFO:teuthology.orchestra.run.smithi032.stderr:updated caps for client.0 2024-08-03T21:01:00.905 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:00.903+0000 7f828ce00640 1 -- 172.21.15.32:0/3285612655 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f8264077040 msgr2=0x7f8264079550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:01:00.905 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:00.903+0000 7f828ce00640 1 --2- 172.21.15.32:0/3285612655 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f8264077040 0x7f8264079550 secure :-1 s=READY pgs=876 cs=0 l=1 rev1=1 crypto rx=0x7f82780046b0 tx=0x7f8278009040 comp rx=0 tx=0).stop 2024-08-03T21:01:00.905 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:00.903+0000 7f828ce00640 1 -- 172.21.15.32:0/3285612655 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8288145c50 msgr2=0x7f82881e9450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:01:00.905 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:00.903+0000 7f828ce00640 1 --2- 172.21.15.32:0/3285612655 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8288145c50 0x7f82881e9450 secure :-1 s=READY pgs=2184 cs=0 l=1 rev1=1 crypto rx=0x7f8280006010 tx=0x7f8280003ba0 comp rx=0 tx=0).stop 2024-08-03T21:01:00.905 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:00.903+0000 7f828ce00640 1 -- 172.21.15.32:0/3285612655 shutdown_connections 2024-08-03T21:01:00.905 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:00.903+0000 7f828ce00640 1 --2- 172.21.15.32:0/3285612655 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f8264077040 0x7f8264079550 unknown :-1 s=CLOSED pgs=876 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:01:00.905 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:00.903+0000 7f828ce00640 1 --2- 172.21.15.32:0/3285612655 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f82880ced30 0x7f82881e8f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:01:00.905 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:00.903+0000 7f828ce00640 1 --2- 172.21.15.32:0/3285612655 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8288145c50 0x7f82881e9450 unknown :-1 s=CLOSED pgs=2184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:01:00.906 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:00.903+0000 7f828ce00640 1 --2- 172.21.15.32:0/3285612655 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f82880cccf0 0x7f8288074020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:01:00.906 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:00.903+0000 7f828ce00640 1 -- 172.21.15.32:0/3285612655 >> 172.21.15.32:0/3285612655 conn(0x7f82880b1bd0 msgr2=0x7f82880cfea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T21:01:00.906 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:00.903+0000 7f828ce00640 1 -- 172.21.15.32:0/3285612655 shutdown_connections 2024-08-03T21:01:00.906 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:00.903+0000 7f828ce00640 1 -- 172.21.15.32:0/3285612655 wait complete. 2024-08-03T21:01:00.931 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-08-03T21:01:01.160 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:01.158+0000 7fde8d200640 1 -- 172.21.15.32:0/2300502468 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fde8806e930 msgr2=0x7fde88144f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:01:01.161 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:01.158+0000 7fde8d200640 1 --2- 172.21.15.32:0/2300502468 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fde8806e930 0x7fde88144f90 secure :-1 s=READY pgs=2185 cs=0 l=1 rev1=1 crypto rx=0x7fde78005fd0 tx=0x7fde7802a5d0 comp rx=0 tx=0).stop 2024-08-03T21:01:01.161 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:01.159+0000 7fde8d200640 1 -- 172.21.15.32:0/2300502468 shutdown_connections 2024-08-03T21:01:01.161 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:01.159+0000 7fde8d200640 1 --2- 172.21.15.32:0/2300502468 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fde8815fff0 0x7fde88160400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:01:01.161 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:01.159+0000 7fde8d200640 1 --2- 172.21.15.32:0/2300502468 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fde8806e930 0x7fde88144f90 unknown :-1 s=CLOSED pgs=2185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:01:01.161 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:01.159+0000 7fde8d200640 1 -- 172.21.15.32:0/2300502468 >> 172.21.15.32:0/2300502468 conn(0x7fde880b1bd0 msgr2=0x7fde880b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T21:01:01.161 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:01.159+0000 7fde8d200640 1 -- 172.21.15.32:0/2300502468 shutdown_connections 2024-08-03T21:01:01.161 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:01.159+0000 7fde8d200640 1 -- 172.21.15.32:0/2300502468 wait complete. 2024-08-03T21:01:01.162 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:01.161+0000 7fde8d200640 1 Processor -- start 2024-08-03T21:01:01.162 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:01.161+0000 7fde8d200640 1 -- start start 2024-08-03T21:01:01.165 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:01.162+0000 7fde8d200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fde8806e930 0x7fde881f9540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:01:01.165 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:01.162+0000 7fde8d200640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fde88073e60 0x7fde8815bd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:01:01.165 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:01.162+0000 7fde8d200640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fde8815fff0 0x7fde880b3350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:01:01.165 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:01.162+0000 7fde8d200640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fde880cf840 con 0x7fde8806e930 2024-08-03T21:01:01.165 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:01.162+0000 7fde8d200640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7fde88145f30 con 0x7fde8815fff0 2024-08-03T21:01:01.165 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:01.162+0000 7fde8d200640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7fde880cf9c0 con 0x7fde88073e60 2024-08-03T21:01:01.166 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:01.163+0000 7fde85e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fde8806e930 0x7fde881f9540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:01:01.166 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:01.163+0000 7fde85e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fde8806e930 0x7fde881f9540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:50448/0 (socket says 172.21.15.32:50448) 2024-08-03T21:01:01.166 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:01.163+0000 7fde85e00640 1 -- 172.21.15.32:0/1732732760 learned_addr learned my addr 172.21.15.32:0/1732732760 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T21:01:01.166 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:01.163+0000 7fde86800640 1 --2- 172.21.15.32:0/1732732760 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fde8815fff0 0x7fde880b3350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:01:01.166 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:01.163+0000 7fde7d400640 1 --2- 172.21.15.32:0/1732732760 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fde88073e60 0x7fde8815bd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:01:01.166 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:01.163+0000 7fde85e00640 1 -- 172.21.15.32:0/1732732760 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fde88073e60 msgr2=0x7fde8815bd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:01:01.166 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:01.163+0000 7fde85e00640 1 --2- 172.21.15.32:0/1732732760 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fde88073e60 0x7fde8815bd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:01:01.166 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:01.163+0000 7fde85e00640 1 -- 172.21.15.32:0/1732732760 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fde8815fff0 msgr2=0x7fde880b3350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:01:01.166 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:01.163+0000 7fde85e00640 1 --2- 172.21.15.32:0/1732732760 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fde8815fff0 0x7fde880b3350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:01:01.167 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:01.163+0000 7fde85e00640 1 -- 172.21.15.32:0/1732732760 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fde880b3890 con 0x7fde8806e930 2024-08-03T21:01:01.167 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:01.163+0000 7fde7d400640 1 --2- 172.21.15.32:0/1732732760 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fde88073e60 0x7fde8815bd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-03T21:01:01.167 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:01.164+0000 7fde85e00640 1 --2- 172.21.15.32:0/1732732760 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fde8806e930 0x7fde881f9540 secure :-1 s=READY pgs=2186 cs=0 l=1 rev1=1 crypto rx=0x7fde78034040 tx=0x7fde780038a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T21:01:01.167 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:01.164+0000 7fde7ea00640 1 -- 172.21.15.32:0/1732732760 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde7803e030 con 0x7fde8806e930 2024-08-03T21:01:01.168 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:01.164+0000 7fde7ea00640 1 -- 172.21.15.32:0/1732732760 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fde78003ca0 con 0x7fde8806e930 2024-08-03T21:01:01.168 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:01.164+0000 7fde8d200640 1 -- 172.21.15.32:0/1732732760 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fde88144a00 con 0x7fde8806e930 2024-08-03T21:01:01.168 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:01.164+0000 7fde7ea00640 1 -- 172.21.15.32:0/1732732760 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde78035040 con 0x7fde8806e930 2024-08-03T21:01:01.168 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:01.164+0000 7fde8d200640 1 -- 172.21.15.32:0/1732732760 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fde881e5f10 con 0x7fde8806e930 2024-08-03T21:01:01.168 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:01.165+0000 7fde8d200640 1 -- 172.21.15.32:0/1732732760 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fde880b3a60 con 0x7fde8806e930 2024-08-03T21:01:01.168 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:01.166+0000 7fde7ea00640 1 -- 172.21.15.32:0/1732732760 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7fde78041030 con 0x7fde8806e930 2024-08-03T21:01:01.169 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:01.167+0000 7fde7ea00640 1 --2- 172.21.15.32:0/1732732760 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fde64077040 0x7fde64079550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:01:01.169 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:01.167+0000 7fde7ea00640 1 -- 172.21.15.32:0/1732732760 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(135..135 src has 1..135) ==== 5709+0+0 (secure 0 0 0) 0x7fde780b8ee0 con 0x7fde8806e930 2024-08-03T21:01:01.169 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:01.168+0000 7fde7d400640 1 --2- 172.21.15.32:0/1732732760 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fde64077040 0x7fde64079550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:01:01.170 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:01.168+0000 7fde7d400640 1 --2- 172.21.15.32:0/1732732760 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fde64077040 0x7fde64079550 secure :-1 s=READY pgs=877 cs=0 l=1 rev1=1 crypto rx=0x7fde880cbfd0 tx=0x7fde70009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T21:01:01.173 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:01.172+0000 7fde7ea00640 1 -- 172.21.15.32:0/1732732760 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7fde780820e0 con 0x7fde8806e930 2024-08-03T21:01:01.417 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:01.415+0000 7fde8d200640 1 -- 172.21.15.32:0/1732732760 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fde88160c70 con 0x7fde8806e930 2024-08-03T21:01:01.418 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:01.416+0000 7fde7ea00640 1 -- 172.21.15.32:0/1732732760 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 340 v340) ==== 95+0+5234 (secure 0 0 0) 0x7fde7808a3b0 con 0x7fde8806e930 2024-08-03T21:01:01.418 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-03T21:01:01.418 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":340,"btime":"2024-08-03T21:00:53:231957+0000","default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8519,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6836/3249267506","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6834","nonce":3249267506},{"type":"v1","addr":"172.21.15.118:6836","nonce":3249267506}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":334},{"gid":8522,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6837/408542078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6835","nonce":408542078},{"type":"v1","addr":"172.21.15.118:6837","nonce":408542078}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":335},{"gid":9589,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6834/3440267920","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":3440267920},{"type":"v1","addr":"172.21.15.32:6834","nonce":3440267920}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":335}],"filesystems":[{"mdsmap":{"epoch":340,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-08-03T21:00:49.742850+0000","modified":"2024-08-03T21:00:52.750915+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":9592},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9592":{"gid":9592,"name":"a","rank":0,"incarnation":336,"state":"up:active","state_seq":3,"addr":"172.21.15.32:6835/4032603982","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6833","nonce":4032603982},{"type":"v1","addr":"172.21.15.32:6835","nonce":4032603982}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9592,"qdb_cluster":[9592]},"id":7}]} 2024-08-03T21:01:01.418 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 340 2024-08-03T21:01:01.420 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:01.418+0000 7fde8d200640 1 -- 172.21.15.32:0/1732732760 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fde64077040 msgr2=0x7fde64079550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:01:01.420 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:01.418+0000 7fde8d200640 1 --2- 172.21.15.32:0/1732732760 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fde64077040 0x7fde64079550 secure :-1 s=READY pgs=877 cs=0 l=1 rev1=1 crypto rx=0x7fde880cbfd0 tx=0x7fde70009040 comp rx=0 tx=0).stop 2024-08-03T21:01:01.420 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:01.418+0000 7fde8d200640 1 -- 172.21.15.32:0/1732732760 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fde8806e930 msgr2=0x7fde881f9540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:01:01.420 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:01.418+0000 7fde8d200640 1 --2- 172.21.15.32:0/1732732760 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fde8806e930 0x7fde881f9540 secure :-1 s=READY pgs=2186 cs=0 l=1 rev1=1 crypto rx=0x7fde78034040 tx=0x7fde780038a0 comp rx=0 tx=0).stop 2024-08-03T21:01:01.421 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:01.418+0000 7fde8d200640 1 -- 172.21.15.32:0/1732732760 shutdown_connections 2024-08-03T21:01:01.421 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:01.418+0000 7fde8d200640 1 --2- 172.21.15.32:0/1732732760 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fde64077040 0x7fde64079550 unknown :-1 s=CLOSED pgs=877 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:01:01.421 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:01.418+0000 7fde8d200640 1 --2- 172.21.15.32:0/1732732760 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fde8815fff0 0x7fde880b3350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:01:01.421 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:01.418+0000 7fde8d200640 1 --2- 172.21.15.32:0/1732732760 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fde88073e60 0x7fde8815bd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:01:01.421 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:01.418+0000 7fde8d200640 1 --2- 172.21.15.32:0/1732732760 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fde8806e930 0x7fde881f9540 unknown :-1 s=CLOSED pgs=2186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:01:01.421 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:01.418+0000 7fde8d200640 1 -- 172.21.15.32:0/1732732760 >> 172.21.15.32:0/1732732760 conn(0x7fde880b1bd0 msgr2=0x7fde8806f930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T21:01:01.421 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:01.419+0000 7fde8d200640 1 -- 172.21.15.32:0/1732732760 shutdown_connections 2024-08-03T21:01:01.421 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:01.419+0000 7fde8d200640 1 -- 172.21.15.32:0/1732732760 wait complete. 2024-08-03T21:01:01.848 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 340, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-08-03T21:00:49.742850+0000', 'modified': '2024-08-03T21:00:52.750915+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 9592}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9592': {'gid': 9592, 'name': 'a', 'rank': 0, 'incarnation': 336, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.32:6835/4032603982', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6833', 'nonce': 4032603982}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 4032603982}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [15], 'metadata_pool': 14, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9592, 'qdb_cluster': [9592]} 2024-08-03T21:01:01.848 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-08-03T21:01:01.848 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2024-08-03T21:01:02.080 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:02.079+0000 7f4c6be00640 1 -- 172.21.15.32:0/3064306842 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4c6c073990 msgr2=0x7f4c6c0cd860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:01:02.081 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:02.079+0000 7f4c6be00640 1 --2- 172.21.15.32:0/3064306842 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4c6c073990 0x7f4c6c0cd860 secure :-1 s=READY pgs=2187 cs=0 l=1 rev1=1 crypto rx=0x7f4c6c06e4f0 tx=0x7f4c5c02a4b0 comp rx=0 tx=0).stop 2024-08-03T21:01:02.081 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:02.079+0000 7f4c6be00640 1 -- 172.21.15.32:0/3064306842 shutdown_connections 2024-08-03T21:01:02.081 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:02.079+0000 7f4c6be00640 1 --2- 172.21.15.32:0/3064306842 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f4c6c06d9f0 0x7f4c6c06de00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:01:02.081 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:02.079+0000 7f4c6be00640 1 --2- 172.21.15.32:0/3064306842 >> v2:172.21.15.118:3300/0 conn(0x7f4c6c153270 0x7f4c6c15d230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:01:02.081 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:02.079+0000 7f4c6be00640 1 --2- 172.21.15.32:0/3064306842 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4c6c073990 0x7f4c6c0cd860 unknown :-1 s=CLOSED pgs=2187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:01:02.081 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:02.079+0000 7f4c6be00640 1 -- 172.21.15.32:0/3064306842 >> 172.21.15.32:0/3064306842 conn(0x7f4c6c072e70 msgr2=0x7f4c6c073280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T21:01:02.081 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:02.079+0000 7f4c6be00640 1 -- 172.21.15.32:0/3064306842 shutdown_connections 2024-08-03T21:01:02.081 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:02.079+0000 7f4c6be00640 1 -- 172.21.15.32:0/3064306842 wait complete. 2024-08-03T21:01:02.082 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:02.081+0000 7f4c6be00640 1 Processor -- start 2024-08-03T21:01:02.082 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:02.081+0000 7f4c6be00640 1 -- start start 2024-08-03T21:01:02.085 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:02.083+0000 7f4c6be00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f4c6c06d9f0 0x7f4c6c1f6950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:01:02.086 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:02.083+0000 7f4c6be00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4c6c073990 0x7f4c6c15bca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:01:02.086 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:02.083+0000 7f4c6be00640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f4c6c153270 0x7f4c6c1f3360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:01:02.086 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:02.083+0000 7f4c6be00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f4c6c160240 con 0x7f4c6c073990 2024-08-03T21:01:02.086 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:02.083+0000 7f4c6be00640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f4c6c1600c0 con 0x7f4c6c06d9f0 2024-08-03T21:01:02.086 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:02.083+0000 7f4c6be00640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7f4c6c1603c0 con 0x7f4c6c153270 2024-08-03T21:01:02.086 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:02.083+0000 7f4c63e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4c6c073990 0x7f4c6c15bca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:01:02.086 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:02.083+0000 7f4c63e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4c6c073990 0x7f4c6c15bca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:50458/0 (socket says 172.21.15.32:50458) 2024-08-03T21:01:02.086 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:02.083+0000 7f4c63e00640 1 -- 172.21.15.32:0/783010170 learned_addr learned my addr 172.21.15.32:0/783010170 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T21:01:02.086 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:02.083+0000 7f4c6b400640 1 --2- 172.21.15.32:0/783010170 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f4c6c153270 0x7f4c6c1f3360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:01:02.086 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:02.083+0000 7f4c6aa00640 1 --2- 172.21.15.32:0/783010170 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f4c6c06d9f0 0x7f4c6c1f6950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:01:02.086 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:02.083+0000 7f4c63e00640 1 -- 172.21.15.32:0/783010170 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f4c6c153270 msgr2=0x7f4c6c1f3360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:01:02.086 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:02.083+0000 7f4c63e00640 1 --2- 172.21.15.32:0/783010170 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f4c6c153270 0x7f4c6c1f3360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:01:02.087 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:02.083+0000 7f4c63e00640 1 -- 172.21.15.32:0/783010170 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f4c6c06d9f0 msgr2=0x7f4c6c1f6950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:01:02.087 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:02.083+0000 7f4c63e00640 1 --2- 172.21.15.32:0/783010170 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f4c6c06d9f0 0x7f4c6c1f6950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:01:02.087 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:02.083+0000 7f4c63e00640 1 -- 172.21.15.32:0/783010170 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4c6c1f7530 con 0x7f4c6c073990 2024-08-03T21:01:02.087 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:02.083+0000 7f4c6aa00640 1 --2- 172.21.15.32:0/783010170 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f4c6c06d9f0 0x7f4c6c1f6950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-03T21:01:02.087 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:02.083+0000 7f4c6b400640 1 --2- 172.21.15.32:0/783010170 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f4c6c153270 0x7f4c6c1f3360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-03T21:01:02.087 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:02.084+0000 7f4c63e00640 1 --2- 172.21.15.32:0/783010170 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4c6c073990 0x7f4c6c15bca0 secure :-1 s=READY pgs=2188 cs=0 l=1 rev1=1 crypto rx=0x7f4c5400d6a0 tx=0x7f4c5400db60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T21:01:02.087 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:02.084+0000 7f4c62a00640 1 -- 172.21.15.32:0/783010170 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c54017070 con 0x7f4c6c073990 2024-08-03T21:01:02.087 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:02.084+0000 7f4c62a00640 1 -- 172.21.15.32:0/783010170 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f4c54004310 con 0x7f4c6c073990 2024-08-03T21:01:02.087 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:02.084+0000 7f4c62a00640 1 -- 172.21.15.32:0/783010170 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c54009a20 con 0x7f4c6c073990 2024-08-03T21:01:02.087 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:02.084+0000 7f4c6be00640 1 -- 172.21.15.32:0/783010170 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4c6c1f7810 con 0x7f4c6c073990 2024-08-03T21:01:02.087 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:02.084+0000 7f4c6be00640 1 -- 172.21.15.32:0/783010170 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4c6c154ca0 con 0x7f4c6c073990 2024-08-03T21:01:02.088 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:02.087+0000 7f4c62a00640 1 -- 172.21.15.32:0/783010170 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f4c54009bc0 con 0x7f4c6c073990 2024-08-03T21:01:02.089 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:02.087+0000 7f4c62a00640 1 --2- 172.21.15.32:0/783010170 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f4c48077010 0x7f4c48079520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:01:02.089 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:02.088+0000 7f4c62a00640 1 -- 172.21.15.32:0/783010170 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(135..135 src has 1..135) ==== 5709+0+0 (secure 0 0 0) 0x7f4c540988b0 con 0x7f4c6c073990 2024-08-03T21:01:02.089 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:02.088+0000 7f4c78b6c740 1 Processor -- start 2024-08-03T21:01:02.090 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:02.088+0000 7f4c6aa00640 1 --2- 172.21.15.32:0/783010170 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f4c48077010 0x7f4c48079520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:01:02.090 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:02.088+0000 7f4c78b6c740 1 -- start start 2024-08-03T21:01:02.090 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:02.089+0000 7f4c6aa00640 1 --2- 172.21.15.32:0/783010170 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f4c48077010 0x7f4c48079520 secure :-1 s=READY pgs=878 cs=0 l=1 rev1=1 crypto rx=0x7f4c5c033000 tx=0x7f4c5c004730 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T21:01:02.092 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:02.089+0000 7f4c78b6c740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555bc697c270 0x555bc6d692c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:01:02.092 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:02.089+0000 7f4c78b6c740 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x555bc69533b0 0x555bc6d1ce00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:01:02.092 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:02.090+0000 7f4c78b6c740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x555bc695b590 0x555bc6d1d340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:01:02.092 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:02.090+0000 7f4c78b6c740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x555bc6c94a60 con 0x555bc697c270 2024-08-03T21:01:02.092 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:02.090+0000 7f4c78b6c740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x555bc6c83f60 con 0x555bc695b590 2024-08-03T21:01:02.092 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:02.090+0000 7f4c78b6c740 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x555bc6c87fa0 con 0x555bc69533b0 2024-08-03T21:01:02.092 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:02.090+0000 7f4c6aa00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555bc697c270 0x555bc6d692c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:01:02.093 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:02.090+0000 7f4c6aa00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555bc697c270 0x555bc6d692c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:50462/0 (socket says 172.21.15.32:50462) 2024-08-03T21:01:02.093 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:02.090+0000 7f4c6aa00640 1 -- 172.21.15.32:0/3011525765 learned_addr learned my addr 172.21.15.32:0/3011525765 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T21:01:02.093 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:02.090+0000 7f4c6b400640 1 --2- 172.21.15.32:0/3011525765 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x555bc695b590 0x555bc6d1d340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:01:02.093 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:02.090+0000 7f4c63e00640 1 --2- 172.21.15.32:0/3011525765 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x555bc69533b0 0x555bc6d1ce00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:01:02.093 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:02.090+0000 7f4c6aa00640 1 -- 172.21.15.32:0/3011525765 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x555bc69533b0 msgr2=0x555bc6d1ce00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T21:01:02.093 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:02.090+0000 7f4c6aa00640 1 --2- 172.21.15.32:0/3011525765 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x555bc69533b0 0x555bc6d1ce00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:01:02.093 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:02.090+0000 7f4c6aa00640 1 -- 172.21.15.32:0/3011525765 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x555bc695b590 msgr2=0x555bc6d1d340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T21:01:02.093 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:02.090+0000 7f4c6aa00640 1 --2- 172.21.15.32:0/3011525765 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x555bc695b590 0x555bc6d1d340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:01:02.093 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:02.090+0000 7f4c6aa00640 1 -- 172.21.15.32:0/3011525765 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555bc69e7850 con 0x555bc697c270 2024-08-03T21:01:02.094 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:02.090+0000 7f4c6b400640 1 --2- 172.21.15.32:0/3011525765 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x555bc695b590 0x555bc6d1d340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-03T21:01:02.094 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:02.091+0000 7f4c6aa00640 1 --2- 172.21.15.32:0/3011525765 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555bc697c270 0x555bc6d692c0 secure :-1 s=READY pgs=2189 cs=0 l=1 rev1=1 crypto rx=0x7f4c6c15c7b0 tx=0x7f4c5c03c000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5f806b6ad218367d server_cookie=0 in_seq=0 out_seq=0 2024-08-03T21:01:02.095 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:02.091+0000 7f4c5b400640 1 -- 172.21.15.32:0/3011525765 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c5c035040 con 0x555bc697c270 2024-08-03T21:01:02.095 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:02.091+0000 7f4c5b400640 1 -- 172.21.15.32:0/3011525765 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f4c5c04d070 con 0x555bc697c270 2024-08-03T21:01:02.095 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:02.091+0000 7f4c5b400640 1 -- 172.21.15.32:0/3011525765 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c5c003cc0 con 0x555bc697c270 2024-08-03T21:01:02.095 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:02.092+0000 7f4c78b6c740 1 -- 172.21.15.32:0/3011525765 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555bc697c270 msgr2=0x555bc6d692c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:01:02.095 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:02.092+0000 7f4c78b6c740 1 --2- 172.21.15.32:0/3011525765 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555bc697c270 0x555bc6d692c0 secure :-1 s=READY pgs=2189 cs=0 l=1 rev1=1 crypto rx=0x7f4c6c15c7b0 tx=0x7f4c5c03c000 comp rx=0 tx=0).stop 2024-08-03T21:01:02.096 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:02.092+0000 7f4c78b6c740 1 -- 172.21.15.32:0/3011525765 shutdown_connections 2024-08-03T21:01:02.096 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:02.092+0000 7f4c78b6c740 1 --2- 172.21.15.32:0/3011525765 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x555bc695b590 0x555bc6d1d340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:01:02.096 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:02.092+0000 7f4c78b6c740 1 --2- 172.21.15.32:0/3011525765 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x555bc69533b0 0x555bc6d1ce00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:01:02.096 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:02.092+0000 7f4c78b6c740 1 --2- 172.21.15.32:0/3011525765 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555bc697c270 0x555bc6d692c0 unknown :-1 s=CLOSED pgs=2189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:01:02.096 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:02.092+0000 7f4c78b6c740 1 -- 172.21.15.32:0/3011525765 >> 172.21.15.32:0/3011525765 conn(0x555bc69703c0 msgr2=0x555bc6d221b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T21:01:02.096 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:02.092+0000 7f4c78b6c740 1 -- 172.21.15.32:0/3011525765 shutdown_connections 2024-08-03T21:01:02.096 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:02.092+0000 7f4c78b6c740 1 -- 172.21.15.32:0/3011525765 wait complete. 2024-08-03T21:01:02.096 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:02.093+0000 7f4c78b6c740 1 -- start start 2024-08-03T21:01:02.096 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:02.093+0000 7f4c78b6c740 1 Processor -- start 2024-08-03T21:01:02.099 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:02.094+0000 7f4c78b6c740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555bc69533b0 0x555bc6d4d360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:01:02.099 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:02.095+0000 7f4c78b6c740 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x555bc695b590 0x555bc6d4b710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:01:02.099 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:02.095+0000 7f4c78b6c740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x555bc697c270 0x555bc6d78490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:01:02.099 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:02.095+0000 7f4c78b6c740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x555bc69f6620 con 0x555bc69533b0 2024-08-03T21:01:02.099 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:02.095+0000 7f4c78b6c740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x555bc6c21160 con 0x555bc697c270 2024-08-03T21:01:02.099 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:02.095+0000 7f4c78b6c740 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x555bc6d4dbb0 con 0x555bc695b590 2024-08-03T21:01:02.099 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:02.095+0000 7f4c78b6c740 10 client.0 _ll_register_callbacks cb 0x555bc6c306b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-03T21:01:02.099 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:02.095+0000 7f4c6aa00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555bc69533b0 0x555bc6d4d360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:01:02.100 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:02.095+0000 7f4c6aa00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555bc69533b0 0x555bc6d4d360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:50470/0 (socket says 172.21.15.32:50470) 2024-08-03T21:01:02.100 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:02.095+0000 7f4c6aa00640 1 -- 172.21.15.32:0/3542940804 learned_addr learned my addr 172.21.15.32:0/3542940804 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T21:01:02.100 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:02.095+0000 7f4c63e00640 1 --2- 172.21.15.32:0/3542940804 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x555bc695b590 0x555bc6d4b710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:01:02.100 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:02.095+0000 7f4c6b400640 1 --2- 172.21.15.32:0/3542940804 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x555bc697c270 0x555bc6d78490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:01:02.100 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:02.095+0000 7f4c6aa00640 1 -- 172.21.15.32:0/3542940804 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x555bc695b590 msgr2=0x555bc6d4b710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T21:01:02.100 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:02.095+0000 7f4c6aa00640 1 --2- 172.21.15.32:0/3542940804 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x555bc695b590 0x555bc6d4b710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:01:02.100 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:02.095+0000 7f4c6aa00640 1 -- 172.21.15.32:0/3542940804 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x555bc697c270 msgr2=0x555bc6d78490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T21:01:02.100 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:02.095+0000 7f4c6aa00640 1 --2- 172.21.15.32:0/3542940804 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x555bc697c270 0x555bc6d78490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:01:02.100 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:02.095+0000 7f4c6aa00640 1 -- 172.21.15.32:0/3542940804 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555bc6a5e6b0 con 0x555bc69533b0 2024-08-03T21:01:02.100 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:02.095+0000 7f4c6b400640 1 --2- 172.21.15.32:0/3542940804 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x555bc697c270 0x555bc6d78490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-03T21:01:02.100 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:02.096+0000 7f4c6aa00640 1 --2- 172.21.15.32:0/3542940804 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555bc69533b0 0x555bc6d4d360 secure :-1 s=READY pgs=2190 cs=0 l=1 rev1=1 crypto rx=0x7f4c5c04a040 tx=0x7f4c5c0037b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3d45f3e78208b991 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T21:01:02.100 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:02.096+0000 7f4c4fe00640 10 client.0 ms_handle_connect on v2:172.21.15.32:3300/0 2024-08-03T21:01:02.100 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:02.096+0000 7f4c4fe00640 1 -- 172.21.15.32:0/3542940804 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c5c035040 con 0x555bc69533b0 2024-08-03T21:01:02.101 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:02.096+0000 7f4c4fe00640 1 -- 172.21.15.32:0/3542940804 <== mon.0 v2:172.21.15.32:3300/0 2 ==== osd_map(135..135 src has 1..135) ==== 5709+0+0 (secure 0 0 0) 0x7f4c5c004cf0 con 0x555bc69533b0 2024-08-03T21:01:02.101 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:02.096+0000 7f4c78b6c740 1 -- 172.21.15.32:0/3542940804 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4c5c04d070 con 0x555bc69533b0 2024-08-03T21:01:02.101 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:02.096+0000 7f4c4fe00640 1 -- 172.21.15.32:0/3542940804 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_get_version_reply(handle=1 version=340) ==== 24+0+0 (secure 0 0 0) 0x7f4c5c064050 con 0x555bc69533b0 2024-08-03T21:01:02.101 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:02.096+0000 7f4c78b6c740 10 client.10641 fetch_fsmap learned FSMap version 340 2024-08-03T21:01:02.101 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:02.096+0000 7f4c78b6c740 1 -- 172.21.15.32:0/3542940804 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,fsmap=340,monmap=2+}) -- 0x555bc6d623e0 con 0x555bc69533b0 2024-08-03T21:01:02.101 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:02.096+0000 7f4c4fe00640 1 -- 172.21.15.32:0/3542940804 <== mon.0 v2:172.21.15.32:3300/0 4 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f4c5c04d070 con 0x555bc69533b0 2024-08-03T21:01:02.101 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:02.097+0000 7f4c4fe00640 1 -- 172.21.15.32:0/3542940804 <== mon.0 v2:172.21.15.32:3300/0 5 ==== fsmap(e 340) ==== 3502+0+0 (secure 0 0 0) 0x7f4c5c03a040 con 0x555bc69533b0 2024-08-03T21:01:02.101 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:02.097+0000 7f4c78b6c740 10 client.10641 fetch_fsmap finished waiting for FSMap version 340 2024-08-03T21:01:02.101 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:02.097+0000 7f4c78b6c740 10 client.10641 resolve_mds: resolved name 'a' to daemon mds.a 2024-08-03T21:01:02.101 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:02.097+0000 7f4c78b6c740 20 client.10641 populate_metadata read hostname 'smithi032' 2024-08-03T21:01:02.101 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:02.097+0000 7f4c78b6c740 1 --2- 172.21.15.32:0/3542940804 >> [v2:172.21.15.32:6833/4032603982,v1:172.21.15.32:6835/4032603982] conn(0x555bc693ae40 0x555bc6d82060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:01:02.101 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:02.097+0000 7f4c78b6c740 4 client.10641 mds_command: new command op to 9592 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-08-03T21:01:02.101 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:02.097+0000 7f4c78b6c740 1 -- 172.21.15.32:0/3542940804 --> [v2:172.21.15.32:6833/4032603982,v1:172.21.15.32:6835/4032603982] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x555bc6c969b0 con 0x555bc693ae40 2024-08-03T21:01:02.101 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:02.097+0000 7f4c63e00640 1 --2- 172.21.15.32:0/3542940804 >> [v2:172.21.15.32:6833/4032603982,v1:172.21.15.32:6835/4032603982] conn(0x555bc693ae40 0x555bc6d82060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:01:02.102 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:02.098+0000 7f4c63e00640 1 --2- 172.21.15.32:0/3542940804 >> [v2:172.21.15.32:6833/4032603982,v1:172.21.15.32:6835/4032603982] conn(0x555bc693ae40 0x555bc6d82060 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cc6b925522fd2dbe server_cookie=d4268d090d53c9b4 in_seq=0 out_seq=0 2024-08-03T21:01:02.102 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:02.098+0000 7f4c4fe00640 10 client.10641 ms_handle_connect on v2:172.21.15.32:6833/4032603982 2024-08-03T21:01:02.103 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:02.098+0000 7f4c4fe00640 1 -- 172.21.15.32:0/3542940804 <== mds.0 v2:172.21.15.32:6833/4032603982 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x555bc6c969b0 con 0x555bc693ae40 2024-08-03T21:01:02.103 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:02.098+0000 7f4c4fe00640 10 client.10641 handle_command_reply: tid=0 2024-08-03T21:01:02.103 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:02.098+0000 7f4c78b6c740 1 client.10641 shutdown 2024-08-03T21:01:02.103 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:02.099+0000 7f4c78b6c740 1 -- 172.21.15.32:0/3542940804 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555bc69533b0 msgr2=0x555bc6d4d360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:01:02.103 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:02.099+0000 7f4c78b6c740 1 --2- 172.21.15.32:0/3542940804 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555bc69533b0 0x555bc6d4d360 secure :-1 s=READY pgs=2190 cs=0 l=1 rev1=1 crypto rx=0x7f4c5c04a040 tx=0x7f4c5c0037b0 comp rx=0 tx=0).stop 2024-08-03T21:01:02.103 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:02.099+0000 7f4c78b6c740 1 -- 172.21.15.32:0/3542940804 shutdown_connections 2024-08-03T21:01:02.103 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:02.099+0000 7f4c78b6c740 1 --2- 172.21.15.32:0/3542940804 >> [v2:172.21.15.32:6833/4032603982,v1:172.21.15.32:6835/4032603982] conn(0x555bc693ae40 0x555bc6d82060 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:01:02.103 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:02.099+0000 7f4c78b6c740 1 --2- 172.21.15.32:0/3542940804 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x555bc697c270 0x555bc6d78490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:01:02.104 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:02.099+0000 7f4c78b6c740 1 --2- 172.21.15.32:0/3542940804 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x555bc695b590 0x555bc6d4b710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:01:02.104 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:02.099+0000 7f4c78b6c740 1 --2- 172.21.15.32:0/3542940804 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555bc69533b0 0x555bc6d4d360 unknown :-1 s=CLOSED pgs=2190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:01:02.104 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:02.099+0000 7f4c4fe00640 0 client.10641 ms_handle_reset on v2:172.21.15.32:6833/4032603982 2024-08-03T21:01:02.104 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:02.099+0000 7f4c78b6c740 1 -- 172.21.15.32:0/3542940804 >> 172.21.15.32:0/3542940804 conn(0x555bc69703c0 msgr2=0x555bc6d5d300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T21:01:02.104 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:02.099+0000 7f4c78b6c740 1 -- 172.21.15.32:0/3542940804 shutdown_connections 2024-08-03T21:01:02.104 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:02.099+0000 7f4c78b6c740 1 -- 172.21.15.32:0/3542940804 wait complete. 2024-08-03T21:01:02.104 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:02.099+0000 7f4c78b6c740 20 client.10641 trim_cache size 0 max 16384 2024-08-03T21:01:02.121 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:02.119+0000 7f4c78b6c740 1 Processor -- start 2024-08-03T21:01:02.121 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:02.119+0000 7f4c78b6c740 1 -- start start 2024-08-03T21:01:02.123 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:02.121+0000 7f4c78b6c740 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x555bc69533b0 0x555bc6d1cf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:01:02.123 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:02.121+0000 7f4c78b6c740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555bc695b590 0x555bc6d68ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:01:02.123 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:02.121+0000 7f4c78b6c740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x555bc697c270 0x555bc6d69010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:01:02.123 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:02.121+0000 7f4c78b6c740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f4c6c06f0d0 con 0x555bc695b590 2024-08-03T21:01:02.123 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:02.121+0000 7f4c78b6c740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f4c6c13f810 con 0x555bc697c270 2024-08-03T21:01:02.123 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:02.121+0000 7f4c78b6c740 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7f4c6c13fe60 con 0x555bc69533b0 2024-08-03T21:01:02.124 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:02.121+0000 7f4c63e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555bc695b590 0x555bc6d68ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:01:02.124 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:02.121+0000 7f4c63e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555bc695b590 0x555bc6d68ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:50472/0 (socket says 172.21.15.32:50472) 2024-08-03T21:01:02.124 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:02.121+0000 7f4c63e00640 1 -- 172.21.15.32:0/3654864230 learned_addr learned my addr 172.21.15.32:0/3654864230 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T21:01:02.124 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:02.121+0000 7f4c6b400640 1 --2- 172.21.15.32:0/3654864230 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x555bc697c270 0x555bc6d69010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:01:02.124 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:02.121+0000 7f4c6aa00640 1 --2- 172.21.15.32:0/3654864230 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x555bc69533b0 0x555bc6d1cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:01:02.124 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:02.121+0000 7f4c63e00640 1 -- 172.21.15.32:0/3654864230 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x555bc69533b0 msgr2=0x555bc6d1cf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T21:01:02.124 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:02.121+0000 7f4c63e00640 1 --2- 172.21.15.32:0/3654864230 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x555bc69533b0 0x555bc6d1cf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:01:02.124 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:02.121+0000 7f4c63e00640 1 -- 172.21.15.32:0/3654864230 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x555bc697c270 msgr2=0x555bc6d69010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T21:01:02.124 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:02.121+0000 7f4c63e00640 1 --2- 172.21.15.32:0/3654864230 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x555bc697c270 0x555bc6d69010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:01:02.125 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:02.121+0000 7f4c63e00640 1 -- 172.21.15.32:0/3654864230 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555bc6d4ca00 con 0x555bc695b590 2024-08-03T21:01:02.125 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:02.121+0000 7f4c6aa00640 1 --2- 172.21.15.32:0/3654864230 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x555bc69533b0 0x555bc6d1cf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-03T21:01:02.125 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:02.122+0000 7f4c63e00640 1 --2- 172.21.15.32:0/3654864230 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555bc695b590 0x555bc6d68ad0 secure :-1 s=READY pgs=2191 cs=0 l=1 rev1=1 crypto rx=0x7f4c54096ce0 tx=0x7f4c54096c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=18974c5487cd32c server_cookie=0 in_seq=0 out_seq=0 2024-08-03T21:01:02.125 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:02.122+0000 7f4c5aa00640 1 -- 172.21.15.32:0/3654864230 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c54020040 con 0x555bc695b590 2024-08-03T21:01:02.125 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:02.122+0000 7f4c5aa00640 1 -- 172.21.15.32:0/3654864230 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f4c54093070 con 0x555bc695b590 2024-08-03T21:01:02.125 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:02.122+0000 7f4c5aa00640 1 -- 172.21.15.32:0/3654864230 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c54065910 con 0x555bc695b590 2024-08-03T21:01:02.126 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:02.122+0000 7f4c78b6c740 1 -- 172.21.15.32:0/3654864230 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555bc695b590 msgr2=0x555bc6d68ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:01:02.126 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:02.122+0000 7f4c78b6c740 1 --2- 172.21.15.32:0/3654864230 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555bc695b590 0x555bc6d68ad0 secure :-1 s=READY pgs=2191 cs=0 l=1 rev1=1 crypto rx=0x7f4c54096ce0 tx=0x7f4c54096c10 comp rx=0 tx=0).stop 2024-08-03T21:01:02.126 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:02.123+0000 7f4c78b6c740 1 -- 172.21.15.32:0/3654864230 shutdown_connections 2024-08-03T21:01:02.126 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:02.123+0000 7f4c78b6c740 1 --2- 172.21.15.32:0/3654864230 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x555bc697c270 0x555bc6d69010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:01:02.126 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:02.123+0000 7f4c78b6c740 1 --2- 172.21.15.32:0/3654864230 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555bc695b590 0x555bc6d68ad0 unknown :-1 s=CLOSED pgs=2191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:01:02.126 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:02.123+0000 7f4c78b6c740 1 --2- 172.21.15.32:0/3654864230 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x555bc69533b0 0x555bc6d1cf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:01:02.126 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:02.123+0000 7f4c78b6c740 1 -- 172.21.15.32:0/3654864230 >> 172.21.15.32:0/3654864230 conn(0x555bc69703c0 msgr2=0x555bc6d17c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T21:01:02.127 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:02.123+0000 7f4c78b6c740 1 -- 172.21.15.32:0/3654864230 shutdown_connections 2024-08-03T21:01:02.127 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:02.123+0000 7f4c78b6c740 1 -- 172.21.15.32:0/3654864230 wait complete. 2024-08-03T21:01:02.127 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:02.124+0000 7f4c78b6c740 1 -- start start 2024-08-03T21:01:02.127 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:02.124+0000 7f4c78b6c740 1 Processor -- start 2024-08-03T21:01:02.129 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:02.125+0000 7f4c78b6c740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555bc69533b0 0x555bc6df0790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:01:02.129 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:02.125+0000 7f4c78b6c740 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x555bc695b590 0x555bc6dec830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:01:02.129 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:02.125+0000 7f4c78b6c740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x555bc697c270 0x555bc6de30f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:01:02.130 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:02.125+0000 7f4c78b6c740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x555bc6d4d9e0 con 0x555bc69533b0 2024-08-03T21:01:02.130 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:02.125+0000 7f4c78b6c740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x555bc69f6620 con 0x555bc697c270 2024-08-03T21:01:02.130 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:02.125+0000 7f4c78b6c740 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x555bc6d668e0 con 0x555bc695b590 2024-08-03T21:01:02.130 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:02.125+0000 7f4c78b6c740 10 client.0 _ll_register_callbacks cb 0x7f4c5c003ea0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-03T21:01:02.130 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:02.126+0000 7f4c6aa00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555bc69533b0 0x555bc6df0790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:01:02.130 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:02.126+0000 7f4c6aa00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555bc69533b0 0x555bc6df0790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:50482/0 (socket says 172.21.15.32:50482) 2024-08-03T21:01:02.130 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:02.126+0000 7f4c6aa00640 1 -- 172.21.15.32:0/3815205457 learned_addr learned my addr 172.21.15.32:0/3815205457 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T21:01:02.130 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:02.126+0000 7f4c6b400640 1 --2- 172.21.15.32:0/3815205457 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x555bc697c270 0x555bc6de30f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:01:02.130 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:02.126+0000 7f4c63e00640 1 --2- 172.21.15.32:0/3815205457 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x555bc695b590 0x555bc6dec830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:01:02.130 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:02.126+0000 7f4c6aa00640 1 -- 172.21.15.32:0/3815205457 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x555bc695b590 msgr2=0x555bc6dec830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T21:01:02.130 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:02.126+0000 7f4c6aa00640 1 --2- 172.21.15.32:0/3815205457 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x555bc695b590 0x555bc6dec830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:01:02.130 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:02.126+0000 7f4c6aa00640 1 -- 172.21.15.32:0/3815205457 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x555bc697c270 msgr2=0x555bc6de30f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T21:01:02.131 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:02.126+0000 7f4c6aa00640 1 --2- 172.21.15.32:0/3815205457 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x555bc697c270 0x555bc6de30f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:01:02.131 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:02.126+0000 7f4c6aa00640 1 -- 172.21.15.32:0/3815205457 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555bc6decd70 con 0x555bc69533b0 2024-08-03T21:01:02.131 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:02.126+0000 7f4c6b400640 1 --2- 172.21.15.32:0/3815205457 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x555bc697c270 0x555bc6de30f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-03T21:01:02.131 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:02.126+0000 7f4c63e00640 1 --2- 172.21.15.32:0/3815205457 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x555bc695b590 0x555bc6dec830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-03T21:01:02.131 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:02.126+0000 7f4c6aa00640 1 --2- 172.21.15.32:0/3815205457 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555bc69533b0 0x555bc6df0790 secure :-1 s=READY pgs=2192 cs=0 l=1 rev1=1 crypto rx=0x7f4c5c004110 tx=0x7f4c5c004da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d3918b5a2264bf85 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T21:01:02.131 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:02.126+0000 7f4c4fe00640 10 client.0 ms_handle_connect on v2:172.21.15.32:3300/0 2024-08-03T21:01:02.131 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:02.126+0000 7f4c4fe00640 1 -- 172.21.15.32:0/3815205457 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c5c035070 con 0x555bc69533b0 2024-08-03T21:01:02.131 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:02.127+0000 7f4c4fe00640 1 -- 172.21.15.32:0/3815205457 <== mon.0 v2:172.21.15.32:3300/0 2 ==== osd_map(135..135 src has 1..135) ==== 5709+0+0 (secure 0 0 0) 0x7f4c5c03dcc0 con 0x555bc69533b0 2024-08-03T21:01:02.132 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:02.127+0000 7f4c78b6c740 1 -- 172.21.15.32:0/3815205457 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4c54093070 con 0x555bc69533b0 2024-08-03T21:01:02.132 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:02.127+0000 7f4c4fe00640 1 -- 172.21.15.32:0/3815205457 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_get_version_reply(handle=1 version=340) ==== 24+0+0 (secure 0 0 0) 0x7f4c5c048870 con 0x555bc69533b0 2024-08-03T21:01:02.132 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:02.127+0000 7f4c78b6c740 10 client.10647 fetch_fsmap learned FSMap version 340 2024-08-03T21:01:02.132 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:02.127+0000 7f4c78b6c740 1 -- 172.21.15.32:0/3815205457 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,fsmap=340,monmap=2+}) -- 0x555bc6decfe0 con 0x555bc69533b0 2024-08-03T21:01:02.132 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:02.127+0000 7f4c4fe00640 1 -- 172.21.15.32:0/3815205457 <== mon.0 v2:172.21.15.32:3300/0 4 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f4c54093070 con 0x555bc69533b0 2024-08-03T21:01:02.132 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:02.127+0000 7f4c4fe00640 1 -- 172.21.15.32:0/3815205457 <== mon.0 v2:172.21.15.32:3300/0 5 ==== fsmap(e 340) ==== 3502+0+0 (secure 0 0 0) 0x7f4c5c03a030 con 0x555bc69533b0 2024-08-03T21:01:02.132 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:02.127+0000 7f4c78b6c740 10 client.10647 fetch_fsmap finished waiting for FSMap version 340 2024-08-03T21:01:02.132 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:02.127+0000 7f4c78b6c740 10 client.10647 resolve_mds: resolved name 'a' to daemon mds.a 2024-08-03T21:01:02.132 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:02.127+0000 7f4c78b6c740 20 client.10647 populate_metadata read hostname 'smithi032' 2024-08-03T21:01:02.132 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:02.127+0000 7f4c78b6c740 1 --2- 172.21.15.32:0/3815205457 >> [v2:172.21.15.32:6833/4032603982,v1:172.21.15.32:6835/4032603982] conn(0x555bc6d773f0 0x555bc6d77800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:01:02.132 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:02.127+0000 7f4c78b6c740 4 client.10647 mds_command: new command op to 9592 tid=0 multi_id=0 [{"prefix": "status"}] 2024-08-03T21:01:02.132 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:02.127+0000 7f4c78b6c740 1 -- 172.21.15.32:0/3815205457 --> [v2:172.21.15.32:6833/4032603982,v1:172.21.15.32:6835/4032603982] -- command(tid 0: {"prefix": "status"}) -- 0x555bc6c92450 con 0x555bc6d773f0 2024-08-03T21:01:02.133 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:02.128+0000 7f4c63e00640 1 --2- 172.21.15.32:0/3815205457 >> [v2:172.21.15.32:6833/4032603982,v1:172.21.15.32:6835/4032603982] conn(0x555bc6d773f0 0x555bc6d77800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:01:02.133 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:02.128+0000 7f4c63e00640 1 --2- 172.21.15.32:0/3815205457 >> [v2:172.21.15.32:6833/4032603982,v1:172.21.15.32:6835/4032603982] conn(0x555bc6d773f0 0x555bc6d77800 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=32507994397ad956 server_cookie=b0c2503d873d6cc7 in_seq=0 out_seq=0 2024-08-03T21:01:02.133 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:02.128+0000 7f4c4fe00640 10 client.10647 ms_handle_connect on v2:172.21.15.32:6833/4032603982 2024-08-03T21:01:02.134 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:02.128+0000 7f4c4fe00640 1 -- 172.21.15.32:0/3815205457 <== mds.0 v2:172.21.15.32:6833/4032603982 1 ==== command_reply(tid 0: 0 ) ==== 8+0+328 (crc 0 0 0) 0x555bc6c92450 con 0x555bc6d773f0 2024-08-03T21:01:02.134 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:02.128+0000 7f4c4fe00640 10 client.10647 handle_command_reply: tid=0 2024-08-03T21:01:02.134 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:02.128+0000 7f4c78b6c740 1 client.10647 shutdown 2024-08-03T21:01:02.135 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:02.128+0000 7f4c78b6c740 1 -- 172.21.15.32:0/3815205457 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555bc69533b0 msgr2=0x555bc6df0790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:01:02.135 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:02.128+0000 7f4c78b6c740 1 --2- 172.21.15.32:0/3815205457 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555bc69533b0 0x555bc6df0790 secure :-1 s=READY pgs=2192 cs=0 l=1 rev1=1 crypto rx=0x7f4c5c004110 tx=0x7f4c5c004da0 comp rx=0 tx=0).stop 2024-08-03T21:01:02.135 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:02.129+0000 7f4c78b6c740 1 -- 172.21.15.32:0/3815205457 shutdown_connections 2024-08-03T21:01:02.135 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:02.129+0000 7f4c78b6c740 1 --2- 172.21.15.32:0/3815205457 >> [v2:172.21.15.32:6833/4032603982,v1:172.21.15.32:6835/4032603982] conn(0x555bc6d773f0 0x555bc6d77800 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:01:02.135 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:02.129+0000 7f4c78b6c740 1 --2- 172.21.15.32:0/3815205457 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x555bc697c270 0x555bc6de30f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:01:02.135 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:02.129+0000 7f4c78b6c740 1 --2- 172.21.15.32:0/3815205457 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x555bc695b590 0x555bc6dec830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:01:02.135 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:02.129+0000 7f4c78b6c740 1 --2- 172.21.15.32:0/3815205457 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555bc69533b0 0x555bc6df0790 unknown :-1 s=CLOSED pgs=2192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:01:02.135 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:02.129+0000 7f4c78b6c740 1 -- 172.21.15.32:0/3815205457 >> 172.21.15.32:0/3815205457 conn(0x555bc69703c0 msgr2=0x555bc6d48990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T21:01:02.135 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:02.129+0000 7f4c4fe00640 0 client.10647 ms_handle_reset on v2:172.21.15.32:6833/4032603982 2024-08-03T21:01:02.135 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:02.129+0000 7f4c78b6c740 1 -- 172.21.15.32:0/3815205457 shutdown_connections 2024-08-03T21:01:02.135 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:02.129+0000 7f4c78b6c740 1 -- 172.21.15.32:0/3815205457 wait complete. 2024-08-03T21:01:02.136 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:02.129+0000 7f4c78b6c740 20 client.10647 trim_cache size 0 max 16384 2024-08-03T21:01:02.136 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-08-03T21:01:02.136 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_fsid": "40ab141a-01d7-49b9-ae3c-2edec9876e95", 2024-08-03T21:01:02.136 INFO:teuthology.orchestra.run.smithi032.stdout: "whoami": 0, 2024-08-03T21:01:02.136 INFO:teuthology.orchestra.run.smithi032.stdout: "id": 9592, 2024-08-03T21:01:02.137 INFO:teuthology.orchestra.run.smithi032.stdout: "want_state": "up:active", 2024-08-03T21:01:02.137 INFO:teuthology.orchestra.run.smithi032.stdout: "state": "up:active", 2024-08-03T21:01:02.137 INFO:teuthology.orchestra.run.smithi032.stdout: "fs_name": "cephfs", 2024-08-03T21:01:02.137 INFO:teuthology.orchestra.run.smithi032.stdout: "rank_uptime": 11.923102749, 2024-08-03T21:01:02.137 INFO:teuthology.orchestra.run.smithi032.stdout: "mdsmap_epoch": 340, 2024-08-03T21:01:02.137 INFO:teuthology.orchestra.run.smithi032.stdout: "osdmap_epoch": 135, 2024-08-03T21:01:02.137 INFO:teuthology.orchestra.run.smithi032.stdout: "osdmap_epoch_barrier": 135, 2024-08-03T21:01:02.137 INFO:teuthology.orchestra.run.smithi032.stdout: "uptime": 16.846408304000001 2024-08-03T21:01:02.137 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-08-03T21:01:02.138 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:02.129+0000 7f4c6be00640 1 -- 172.21.15.32:0/783010170 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f4c48077010 msgr2=0x7f4c48079520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:01:02.138 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:02.129+0000 7f4c6be00640 1 --2- 172.21.15.32:0/783010170 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f4c48077010 0x7f4c48079520 secure :-1 s=READY pgs=878 cs=0 l=1 rev1=1 crypto rx=0x7f4c5c033000 tx=0x7f4c5c004730 comp rx=0 tx=0).stop 2024-08-03T21:01:02.138 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:02.129+0000 7f4c6be00640 1 -- 172.21.15.32:0/783010170 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4c6c073990 msgr2=0x7f4c6c15bca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:01:02.138 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:02.129+0000 7f4c6be00640 1 --2- 172.21.15.32:0/783010170 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4c6c073990 0x7f4c6c15bca0 secure :-1 s=READY pgs=2188 cs=0 l=1 rev1=1 crypto rx=0x7f4c5400d6a0 tx=0x7f4c5400db60 comp rx=0 tx=0).stop 2024-08-03T21:01:02.138 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:02.130+0000 7f4c6be00640 1 -- 172.21.15.32:0/783010170 shutdown_connections 2024-08-03T21:01:02.139 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:02.130+0000 7f4c6be00640 1 --2- 172.21.15.32:0/783010170 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f4c48077010 0x7f4c48079520 unknown :-1 s=CLOSED pgs=878 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:01:02.139 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:02.130+0000 7f4c6be00640 1 --2- 172.21.15.32:0/783010170 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f4c6c153270 0x7f4c6c1f3360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:01:02.139 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:02.130+0000 7f4c6be00640 1 --2- 172.21.15.32:0/783010170 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4c6c073990 0x7f4c6c15bca0 unknown :-1 s=CLOSED pgs=2188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:01:02.139 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:02.130+0000 7f4c6be00640 1 --2- 172.21.15.32:0/783010170 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f4c6c06d9f0 0x7f4c6c1f6950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:01:02.139 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:02.130+0000 7f4c6be00640 1 -- 172.21.15.32:0/783010170 >> 172.21.15.32:0/783010170 conn(0x7f4c6c072e70 msgr2=0x7f4c6c152900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T21:01:02.139 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:02.130+0000 7f4c6be00640 1 -- 172.21.15.32:0/783010170 shutdown_connections 2024-08-03T21:01:02.139 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:02.130+0000 7f4c6be00640 1 -- 172.21.15.32:0/783010170 wait complete. 2024-08-03T21:01:02.155 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-08-03T21:01:02.384 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:02.382+0000 7fa76fe00640 1 -- 172.21.15.32:0/1972726240 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa77006e100 msgr2=0x7fa770153270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:01:02.384 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:02.382+0000 7fa76fe00640 1 --2- 172.21.15.32:0/1972726240 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa77006e100 0x7fa770153270 secure :-1 s=READY pgs=2194 cs=0 l=1 rev1=1 crypto rx=0x7fa768009be0 tx=0x7fa76802a840 comp rx=0 tx=0).stop 2024-08-03T21:01:02.384 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:02.382+0000 7fa76fe00640 1 -- 172.21.15.32:0/1972726240 shutdown_connections 2024-08-03T21:01:02.384 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:02.382+0000 7fa76fe00640 1 --2- 172.21.15.32:0/1972726240 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa77006e100 0x7fa770153270 unknown :-1 s=CLOSED pgs=2194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:01:02.384 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:02.382+0000 7fa76fe00640 1 -- 172.21.15.32:0/1972726240 >> 172.21.15.32:0/1972726240 conn(0x7fa770072e70 msgr2=0x7fa770073280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T21:01:02.384 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:02.382+0000 7fa76fe00640 1 -- 172.21.15.32:0/1972726240 shutdown_connections 2024-08-03T21:01:02.384 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:02.382+0000 7fa76fe00640 1 -- 172.21.15.32:0/1972726240 wait complete. 2024-08-03T21:01:02.386 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:02.384+0000 7fa76fe00640 1 Processor -- start 2024-08-03T21:01:02.386 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:02.384+0000 7fa76fe00640 1 -- start start 2024-08-03T21:01:02.388 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:02.386+0000 7fa76fe00640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fa770073990 0x7fa77014f2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:01:02.388 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:02.386+0000 7fa76fe00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa7700cd860 0x7fa7701560e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:01:02.388 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:02.386+0000 7fa76fe00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa77015d890 0x7fa77014f970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:01:02.388 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:02.386+0000 7fa76fe00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fa7701676e0 con 0x7fa7700cd860 2024-08-03T21:01:02.388 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:02.386+0000 7fa76fe00640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7fa770167560 con 0x7fa77015d890 2024-08-03T21:01:02.388 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:02.386+0000 7fa76fe00640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7fa770167860 con 0x7fa770073990 2024-08-03T21:01:02.388 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:02.386+0000 7fa767e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa7700cd860 0x7fa7701560e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:01:02.388 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:02.386+0000 7fa767e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa7700cd860 0x7fa7701560e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:50496/0 (socket says 172.21.15.32:50496) 2024-08-03T21:01:02.388 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:02.386+0000 7fa767e00640 1 -- 172.21.15.32:0/3482667058 learned_addr learned my addr 172.21.15.32:0/3482667058 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T21:01:02.389 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:02.386+0000 7fa76f400640 1 --2- 172.21.15.32:0/3482667058 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa77015d890 0x7fa77014f970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:01:02.389 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:02.386+0000 7fa76ea00640 1 --2- 172.21.15.32:0/3482667058 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fa770073990 0x7fa77014f2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:01:02.390 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:02.387+0000 7fa767e00640 1 -- 172.21.15.32:0/3482667058 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fa770073990 msgr2=0x7fa77014f2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:01:02.390 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:02.387+0000 7fa767e00640 1 --2- 172.21.15.32:0/3482667058 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fa770073990 0x7fa77014f2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:01:02.390 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:02.387+0000 7fa767e00640 1 -- 172.21.15.32:0/3482667058 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa77015d890 msgr2=0x7fa77014f970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:01:02.390 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:02.387+0000 7fa767e00640 1 --2- 172.21.15.32:0/3482667058 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa77015d890 0x7fa77014f970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:01:02.390 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:02.387+0000 7fa767e00640 1 -- 172.21.15.32:0/3482667058 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa7701f8460 con 0x7fa7700cd860 2024-08-03T21:01:02.390 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:02.387+0000 7fa767e00640 1 --2- 172.21.15.32:0/3482667058 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa7700cd860 0x7fa7701560e0 secure :-1 s=READY pgs=2195 cs=0 l=1 rev1=1 crypto rx=0x7fa75800c3a0 tx=0x7fa75800c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T21:01:02.390 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:02.387+0000 7fa766a00640 1 -- 172.21.15.32:0/3482667058 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa758017070 con 0x7fa7700cd860 2024-08-03T21:01:02.390 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:02.387+0000 7fa766a00640 1 -- 172.21.15.32:0/3482667058 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fa7580053c0 con 0x7fa7700cd860 2024-08-03T21:01:02.390 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:02.387+0000 7fa766a00640 1 -- 172.21.15.32:0/3482667058 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa758003850 con 0x7fa7700cd860 2024-08-03T21:01:02.390 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:02.387+0000 7fa76fe00640 1 -- 172.21.15.32:0/3482667058 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa7701569a0 con 0x7fa7700cd860 2024-08-03T21:01:02.390 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:02.388+0000 7fa76fe00640 1 -- 172.21.15.32:0/3482667058 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa77015ea60 con 0x7fa7700cd860 2024-08-03T21:01:02.391 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:02.389+0000 7fa76fe00640 1 -- 172.21.15.32:0/3482667058 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa770153270 con 0x7fa7700cd860 2024-08-03T21:01:02.391 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:02.390+0000 7fa766a00640 1 -- 172.21.15.32:0/3482667058 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7fa75801a070 con 0x7fa7700cd860 2024-08-03T21:01:02.392 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:02.391+0000 7fa766a00640 1 --2- 172.21.15.32:0/3482667058 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fa74c077010 0x7fa74c079520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:01:02.392 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:02.391+0000 7fa766a00640 1 -- 172.21.15.32:0/3482667058 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(135..135 src has 1..135) ==== 5709+0+0 (secure 0 0 0) 0x7fa758097d00 con 0x7fa7700cd860 2024-08-03T21:01:02.393 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:02.391+0000 7fa76ea00640 1 --2- 172.21.15.32:0/3482667058 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fa74c077010 0x7fa74c079520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:01:02.394 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:02.392+0000 7fa76ea00640 1 --2- 172.21.15.32:0/3482667058 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fa74c077010 0x7fa74c079520 secure :-1 s=READY pgs=879 cs=0 l=1 rev1=1 crypto rx=0x7fa760000f60 tx=0x7fa760009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T21:01:02.397 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:02.395+0000 7fa766a00640 1 -- 172.21.15.32:0/3482667058 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7fa758060790 con 0x7fa7700cd860 2024-08-03T21:01:02.641 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:02.640+0000 7fa76fe00640 1 -- 172.21.15.32:0/3482667058 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7fa77015f680 con 0x7fa7700cd860 2024-08-03T21:01:02.642 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:02.640+0000 7fa766a00640 1 -- 172.21.15.32:0/3482667058 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v340) ==== 53+0+73 (secure 0 0 0) 0x7fa7580690b0 con 0x7fa7700cd860 2024-08-03T21:01:02.642 INFO:teuthology.orchestra.run.smithi032.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-08-03T21:01:02.644 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:02.642+0000 7fa76fe00640 1 -- 172.21.15.32:0/3482667058 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fa74c077010 msgr2=0x7fa74c079520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:01:02.644 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:02.642+0000 7fa76fe00640 1 --2- 172.21.15.32:0/3482667058 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fa74c077010 0x7fa74c079520 secure :-1 s=READY pgs=879 cs=0 l=1 rev1=1 crypto rx=0x7fa760000f60 tx=0x7fa760009040 comp rx=0 tx=0).stop 2024-08-03T21:01:02.644 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:02.642+0000 7fa76fe00640 1 -- 172.21.15.32:0/3482667058 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa7700cd860 msgr2=0x7fa7701560e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:01:02.645 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:02.642+0000 7fa76fe00640 1 --2- 172.21.15.32:0/3482667058 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa7700cd860 0x7fa7701560e0 secure :-1 s=READY pgs=2195 cs=0 l=1 rev1=1 crypto rx=0x7fa75800c3a0 tx=0x7fa75800c860 comp rx=0 tx=0).stop 2024-08-03T21:01:02.645 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:02.642+0000 7fa76fe00640 1 -- 172.21.15.32:0/3482667058 shutdown_connections 2024-08-03T21:01:02.645 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:02.642+0000 7fa76fe00640 1 --2- 172.21.15.32:0/3482667058 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fa74c077010 0x7fa74c079520 unknown :-1 s=CLOSED pgs=879 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:01:02.645 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:02.642+0000 7fa76fe00640 1 --2- 172.21.15.32:0/3482667058 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa77015d890 0x7fa77014f970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:01:02.645 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:02.642+0000 7fa76fe00640 1 --2- 172.21.15.32:0/3482667058 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa7700cd860 0x7fa7701560e0 unknown :-1 s=CLOSED pgs=2195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:01:02.645 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:02.642+0000 7fa76fe00640 1 --2- 172.21.15.32:0/3482667058 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fa770073990 0x7fa77014f2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:01:02.645 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:02.643+0000 7fa76fe00640 1 -- 172.21.15.32:0/3482667058 >> 172.21.15.32:0/3482667058 conn(0x7fa770072e70 msgr2=0x7fa77006dd40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T21:01:02.645 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:02.643+0000 7fa76fe00640 1 -- 172.21.15.32:0/3482667058 shutdown_connections 2024-08-03T21:01:02.645 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:02.643+0000 7fa76fe00640 1 -- 172.21.15.32:0/3482667058 wait complete. 2024-08-03T21:01:02.667 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-08-03T21:01:02.667 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-08-03T21:01:02.668 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi032.front.sepia.ceph.com 2024-08-03T21:01:02.668 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-08-03T21:01:02.668 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-08-03T21:01:02.668 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-08-03T21:01:02.668 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2024-08-03T21:01:02.668 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-08-03T21:01:02.668 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-08-03T21:01:02.668 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-08-03T21:01:02.668 DEBUG:teuthology.orchestra.run.smithi032:> ip addr 2024-08-03T21:01:02.703 INFO:teuthology.orchestra.run.smithi032.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-08-03T21:01:02.703 INFO:teuthology.orchestra.run.smithi032.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-08-03T21:01:02.703 INFO:teuthology.orchestra.run.smithi032.stdout: inet 127.0.0.1/8 scope host lo 2024-08-03T21:01:02.704 INFO:teuthology.orchestra.run.smithi032.stdout: valid_lft forever preferred_lft forever 2024-08-03T21:01:02.704 INFO:teuthology.orchestra.run.smithi032.stdout: inet6 ::1/128 scope host 2024-08-03T21:01:02.704 INFO:teuthology.orchestra.run.smithi032.stdout: valid_lft forever preferred_lft forever 2024-08-03T21:01:02.704 INFO:teuthology.orchestra.run.smithi032.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-08-03T21:01:02.704 INFO:teuthology.orchestra.run.smithi032.stdout: link/ether 0c:c4:7a:6b:fd:0c brd ff:ff:ff:ff:ff:ff 2024-08-03T21:01:02.704 INFO:teuthology.orchestra.run.smithi032.stdout: altname enp7s0f0 2024-08-03T21:01:02.704 INFO:teuthology.orchestra.run.smithi032.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-08-03T21:01:02.704 INFO:teuthology.orchestra.run.smithi032.stdout: link/ether 0c:c4:7a:6b:fd:0d brd ff:ff:ff:ff:ff:ff 2024-08-03T21:01:02.704 INFO:teuthology.orchestra.run.smithi032.stdout: altname enp7s0f1 2024-08-03T21:01:02.704 INFO:teuthology.orchestra.run.smithi032.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-08-03T21:01:02.704 INFO:teuthology.orchestra.run.smithi032.stdout: link/ether ac:1f:6b:f8:26:02 brd ff:ff:ff:ff:ff:ff 2024-08-03T21:01:02.704 INFO:teuthology.orchestra.run.smithi032.stdout: altname enp1s0f0 2024-08-03T21:01:02.704 INFO:teuthology.orchestra.run.smithi032.stdout: inet 172.21.15.32/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-08-03T21:01:02.704 INFO:teuthology.orchestra.run.smithi032.stdout: valid_lft 41693sec preferred_lft 41693sec 2024-08-03T21:01:02.704 INFO:teuthology.orchestra.run.smithi032.stdout: inet6 fe80::ae1f:6bff:fef8:2602/64 scope link noprefixroute 2024-08-03T21:01:02.705 INFO:teuthology.orchestra.run.smithi032.stdout: valid_lft forever preferred_lft forever 2024-08-03T21:01:02.705 INFO:teuthology.orchestra.run.smithi032.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-08-03T21:01:02.705 INFO:teuthology.orchestra.run.smithi032.stdout: link/ether ac:1f:6b:f8:26:03 brd ff:ff:ff:ff:ff:ff 2024-08-03T21:01:02.705 INFO:teuthology.orchestra.run.smithi032.stdout: altname enp1s0f1 2024-08-03T21:01:02.705 INFO:teuthology.orchestra.run.smithi032.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-08-03T21:01:02.705 INFO:teuthology.orchestra.run.smithi032.stdout: link/ether 0a:23:2b:ca:2a:93 brd ff:ff:ff:ff:ff:ff 2024-08-03T21:01:02.705 INFO:teuthology.orchestra.run.smithi032.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-08-03T21:01:02.705 INFO:teuthology.orchestra.run.smithi032.stdout: valid_lft forever preferred_lft forever 2024-08-03T21:01:02.705 INFO:teuthology.orchestra.run.smithi032.stdout: inet6 fe80::d804:7cff:fed9:7ba4/64 scope link 2024-08-03T21:01:02.705 INFO:teuthology.orchestra.run.smithi032.stdout: valid_lft forever preferred_lft forever 2024-08-03T21:01:02.705 INFO:teuthology.orchestra.run.smithi032.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-08-03T21:01:02.705 INFO:teuthology.orchestra.run.smithi032.stdout: link/ether 22:86:28:30:4e:62 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-08-03T21:01:02.705 INFO:teuthology.orchestra.run.smithi032.stdout: inet6 fe80::2086:28ff:fe30:4e62/64 scope link 2024-08-03T21:01:02.705 INFO:teuthology.orchestra.run.smithi032.stdout: valid_lft forever preferred_lft forever 2024-08-03T21:01:02.706 INFO:teuthology.orchestra.run.smithi032.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-08-03T21:01:02.706 INFO:teuthology.orchestra.run.smithi032.stdout: link/ether 0a:23:2b:ca:2a:93 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-08-03T21:01:02.706 INFO:teuthology.orchestra.run.smithi032.stdout: inet6 fe80::823:2bff:feca:2a93/64 scope link 2024-08-03T21:01:02.706 INFO:teuthology.orchestra.run.smithi032.stdout: valid_lft forever preferred_lft forever 2024-08-03T21:01:02.707 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-08-03T21:01:02.707 DEBUG:teuthology.orchestra.run.smithi032:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-08-03T21:01:02.801 INFO:teuthology.orchestra.run.smithi032.stdout:1 2024-08-03T21:01:02.804 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-08-03T21:01:02.805 DEBUG:teuthology.orchestra.run.smithi032:> route 2024-08-03T21:01:02.841 INFO:teuthology.orchestra.run.smithi032.stdout:Kernel IP routing table 2024-08-03T21:01:02.841 INFO:teuthology.orchestra.run.smithi032.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-08-03T21:01:02.842 INFO:teuthology.orchestra.run.smithi032.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-08-03T21:01:02.842 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-08-03T21:01:02.842 INFO:teuthology.orchestra.run.smithi032.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-08-03T21:01:02.842 DEBUG:teuthology.orchestra.run.smithi032:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-08-03T21:01:02.842 DEBUG:teuthology.orchestra.run.smithi032:> set -e 2024-08-03T21:01:02.842 DEBUG:teuthology.orchestra.run.smithi032:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-08-03T21:01:02.843 DEBUG:teuthology.orchestra.run.smithi032:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-08-03T21:01:02.843 DEBUG:teuthology.orchestra.run.smithi032:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-08-03T21:01:02.843 DEBUG:teuthology.orchestra.run.smithi032:> ') 2024-08-03T21:01:02.960 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:02 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-08-03T21:01:03.031 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:03 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-08-03T21:01:03.039 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-08-03T21:01:03.039 DEBUG:teuthology.orchestra.run.smithi032:> ip netns list 2024-08-03T21:01:03.075 INFO:teuthology.orchestra.run.smithi032.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-08-03T21:01:03.075 INFO:teuthology.orchestra.run.smithi032.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-08-03T21:01:03.076 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-08-03T21:01:03.076 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-08-03T21:01:03.076 DEBUG:teuthology.orchestra.run.smithi032:> sudo ip link set brx.0 up 2024-08-03T21:01:03.172 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-08-03T21:01:03.172 DEBUG:teuthology.orchestra.run.smithi032:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-08-03T21:01:03.207 INFO:teuthology.orchestra.run.smithi032.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-08-03T21:01:03.208 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-08-03T21:01:03.208 DEBUG:teuthology.orchestra.run.smithi032:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-08-03T21:01:03.282 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2024-08-03T21:01:03.283 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-08-03T21:01:03.283 DEBUG:teuthology.orchestra.run.smithi032:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-08-03T21:01:03.480 INFO:teuthology.orchestra.run.smithi032.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-08-03T21:01:03.481 INFO:teuthology.orchestra.run.smithi032.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-08-03T21:01:03.481 INFO:teuthology.orchestra.run.smithi032.stdout:invalid new device string format 2024-08-03T21:01:03.501 INFO:teuthology.orchestra.run.smithi032.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-08-03T21:01:03.501 INFO:teuthology.orchestra.run.smithi032.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-08-03T21:01:03.501 INFO:teuthology.orchestra.run.smithi032.stdout:invalid new device string format 2024-08-03T21:01:03.501 INFO:teuthology.orchestra.run.smithi032.stdout:mount.ceph: resolved to: "172.21.15.32:3300,172.21.15.118:3300,172.21.15.118:3301" 2024-08-03T21:01:03.501 INFO:teuthology.orchestra.run.smithi032.stdout:mount.ceph: switching to using v1 address with old syntax 2024-08-03T21:01:03.534 INFO:teuthology.orchestra.run.smithi032.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-08-03T21:01:03.534 INFO:teuthology.orchestra.run.smithi032.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-08-03T21:01:03.534 INFO:teuthology.orchestra.run.smithi032.stdout:invalid new device string format 2024-08-03T21:01:03.535 INFO:teuthology.orchestra.run.smithi032.stdout:mount.ceph: resolved to: "172.21.15.32:3300,172.21.15.118:3300,172.21.15.118:3301" 2024-08-03T21:01:03.535 INFO:teuthology.orchestra.run.smithi032.stdout:mount.ceph: switching to using v1 address with old syntax 2024-08-03T21:01:03.535 INFO:teuthology.orchestra.run.smithi032.stdout:mount.ceph: resolved to: "172.21.15.32:6789,172.21.15.118:6789,172.21.15.118:6790" 2024-08-03T21:01:03.535 INFO:teuthology.orchestra.run.smithi032.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.32:6789,172.21.15.118:6789,172.21.15.118:6790:/ 2024-08-03T21:01:03.535 INFO:teuthology.orchestra.run.smithi032.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,key=0,fsid=40ab141a-01d7-49b9-ae3c-2edec9876e95" will pass to kernel 2024-08-03T21:01:03.536 INFO:teuthology.orchestra.run.smithi032.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2024-08-03T21:01:03.536 INFO:teuthology.orchestra.run.smithi032.stdout: You just mounted a file system that supports labels which does not 2024-08-03T21:01:03.536 INFO:teuthology.orchestra.run.smithi032.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-08-03T21:01:03.536 INFO:teuthology.orchestra.run.smithi032.stdout: applications will generate AVC messages and not be allowed access to 2024-08-03T21:01:03.537 INFO:teuthology.orchestra.run.smithi032.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-08-03T21:01:03.539 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-08-03T21:01:03.540 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-08-03T21:01:03.540 DEBUG:teuthology.orchestra.run.smithi032:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-08-03T21:01:03.597 DEBUG:teuthology.orchestra.run.smithi032:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-08-03T21:01:03.678 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:03 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-08-03T21:01:03.688 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:03 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-08-03T21:01:03.696 INFO:teuthology.orchestra.run.smithi032.stdout:client9745 2024-08-03T21:01:03.697 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-08-03T21:01:03.697 DEBUG:teuthology.orchestra.run.smithi032:> dd if=/proc/self/mounts of=/dev/stdout 2024-08-03T21:01:03.733 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-08-03T21:01:03.733 DEBUG:teuthology.orchestra.run.smithi032:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir dir1) 2024-08-03T21:01:03.815 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-08-03T21:01:03.815 DEBUG:teuthology.orchestra.run.smithi032:> (cd /home/ubuntu/cephtest/mnt.0 && exec touch dir1/file1) 2024-08-03T21:01:03.894 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-08-03T21:01:03.894 DEBUG:teuthology.orchestra.run.smithi032:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir dir2) 2024-08-03T21:01:03.973 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-08-03T21:01:03.973 DEBUG:teuthology.orchestra.run.smithi032:> (cd /home/ubuntu/cephtest/mnt.0 && exec touch dir2/file2) 2024-08-03T21:01:04.054 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-08-03T21:01:04.054 DEBUG:teuthology.orchestra.run.smithi032:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir testdir) 2024-08-03T21:01:04.133 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-08-03T21:01:04.134 DEBUG:teuthology.orchestra.run.smithi032:> (cd /home/ubuntu/cephtest/mnt.0 && exec ln dir1/file1 testdir/hardlink1) 2024-08-03T21:01:04.213 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-08-03T21:01:04.214 DEBUG:teuthology.orchestra.run.smithi032:> (cd /home/ubuntu/cephtest/mnt.0 && exec ln dir2/file2 testdir/hardlink2) 2024-08-03T21:01:04.292 DEBUG:teuthology.orchestra.run.smithi032:> stdin-killer -- python3 -c ' 2024-08-03T21:01:04.293 DEBUG:teuthology.orchestra.run.smithi032:> import os 2024-08-03T21:01:04.293 DEBUG:teuthology.orchestra.run.smithi032:> import stat 2024-08-03T21:01:04.293 DEBUG:teuthology.orchestra.run.smithi032:> 2024-08-03T21:01:04.293 DEBUG:teuthology.orchestra.run.smithi032:> print(os.stat("/home/ubuntu/cephtest/mnt.0/dir1/file1").st_ino) 2024-08-03T21:01:04.293 DEBUG:teuthology.orchestra.run.smithi032:> ' 2024-08-03T21:01:04.433 INFO:teuthology.orchestra.run.smithi032.stdout:1099511627777 2024-08-03T21:01:04.436 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:04 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-08-03T21:01:04.444 DEBUG:teuthology.orchestra.run.smithi032:> stdin-killer -- python3 -c ' 2024-08-03T21:01:04.445 DEBUG:teuthology.orchestra.run.smithi032:> import os 2024-08-03T21:01:04.445 DEBUG:teuthology.orchestra.run.smithi032:> import stat 2024-08-03T21:01:04.445 DEBUG:teuthology.orchestra.run.smithi032:> 2024-08-03T21:01:04.445 DEBUG:teuthology.orchestra.run.smithi032:> print(os.stat("/home/ubuntu/cephtest/mnt.0/dir2/file2").st_ino) 2024-08-03T21:01:04.445 DEBUG:teuthology.orchestra.run.smithi032:> ' 2024-08-03T21:01:04.585 INFO:teuthology.orchestra.run.smithi032.stdout:1099511628279 2024-08-03T21:01:04.588 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:04 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-08-03T21:01:04.596 DEBUG:teuthology.orchestra.run.smithi032:> stdin-killer -- python3 -c ' 2024-08-03T21:01:04.597 DEBUG:teuthology.orchestra.run.smithi032:> import os 2024-08-03T21:01:04.597 DEBUG:teuthology.orchestra.run.smithi032:> import stat 2024-08-03T21:01:04.597 DEBUG:teuthology.orchestra.run.smithi032:> 2024-08-03T21:01:04.597 DEBUG:teuthology.orchestra.run.smithi032:> print(os.stat("/home/ubuntu/cephtest/mnt.0/dir2").st_ino) 2024-08-03T21:01:04.597 DEBUG:teuthology.orchestra.run.smithi032:> ' 2024-08-03T21:01:04.737 INFO:teuthology.orchestra.run.smithi032.stdout:1099511628278 2024-08-03T21:01:04.740 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:04 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-08-03T21:01:04.748 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-08-03T21:01:04.748 DEBUG:teuthology.orchestra.run.smithi032:> dd if=/proc/self/mounts of=/dev/stdout 2024-08-03T21:01:04.824 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-08-03T21:01:04.824 DEBUG:teuthology.orchestra.run.smithi032:> dd if=/proc/self/mounts of=/dev/stdout 2024-08-03T21:01:04.900 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-08-03T21:01:05.169 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:05.167+0000 7fae66a00640 1 -- 172.21.15.32:0/2219727241 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fae60073e60 msgr2=0x7fae60074270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:01:05.169 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:05.167+0000 7fae66a00640 1 --2- 172.21.15.32:0/2219727241 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fae60073e60 0x7fae60074270 secure :-1 s=READY pgs=2196 cs=0 l=1 rev1=1 crypto rx=0x7fae60071700 tx=0x7fae4802a560 comp rx=0 tx=0).stop 2024-08-03T21:01:05.170 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:05.167+0000 7fae66a00640 1 -- 172.21.15.32:0/2219727241 shutdown_connections 2024-08-03T21:01:05.170 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:05.167+0000 7fae66a00640 1 --2- 172.21.15.32:0/2219727241 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fae600747b0 0x7fae600cc8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:01:05.170 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:05.167+0000 7fae66a00640 1 --2- 172.21.15.32:0/2219727241 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fae60073e60 0x7fae60074270 unknown :-1 s=CLOSED pgs=2196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:01:05.170 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:05.167+0000 7fae66a00640 1 --2- 172.21.15.32:0/2219727241 >> v2:172.21.15.118:3300/0 conn(0x7fae6006e930 0x7fae60144f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:01:05.170 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:05.167+0000 7fae66a00640 1 -- 172.21.15.32:0/2219727241 >> 172.21.15.32:0/2219727241 conn(0x7fae600b1bd0 msgr2=0x7fae600b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T21:01:05.170 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:05.167+0000 7fae66a00640 1 -- 172.21.15.32:0/2219727241 shutdown_connections 2024-08-03T21:01:05.170 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:05.167+0000 7fae66a00640 1 -- 172.21.15.32:0/2219727241 wait complete. 2024-08-03T21:01:05.171 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:05.170+0000 7fae66a00640 1 Processor -- start 2024-08-03T21:01:05.171 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:05.170+0000 7fae66a00640 1 -- start start 2024-08-03T21:01:05.173 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:05.171+0000 7fae66a00640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fae6006e930 0x7fae6013cfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:01:05.173 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:05.171+0000 7fae66a00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fae60073e60 0x7fae601f4e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:01:05.173 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:05.171+0000 7fae66a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fae600747b0 0x7fae60143aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:01:05.173 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:05.171+0000 7fae66a00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fae600cf9c0 con 0x7fae600747b0 2024-08-03T21:01:05.174 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:05.171+0000 7fae66a00640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7fae600cf840 con 0x7fae60073e60 2024-08-03T21:01:05.174 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:05.171+0000 7fae66a00640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7fae600cce20 con 0x7fae6006e930 2024-08-03T21:01:05.174 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:05.171+0000 7fae64a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fae600747b0 0x7fae60143aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:01:05.174 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:05.172+0000 7fae64a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fae600747b0 0x7fae60143aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:50514/0 (socket says 172.21.15.32:50514) 2024-08-03T21:01:05.174 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:05.172+0000 7fae64a00640 1 -- 172.21.15.32:0/3408626150 learned_addr learned my addr 172.21.15.32:0/3408626150 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T21:01:05.174 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:05.172+0000 7fae57e00640 1 --2- 172.21.15.32:0/3408626150 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fae60073e60 0x7fae601f4e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:01:05.174 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:05.172+0000 7fae5fe00640 1 --2- 172.21.15.32:0/3408626150 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fae6006e930 0x7fae6013cfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:01:05.174 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:05.172+0000 7fae64a00640 1 -- 172.21.15.32:0/3408626150 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fae6006e930 msgr2=0x7fae6013cfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:01:05.174 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:05.172+0000 7fae64a00640 1 --2- 172.21.15.32:0/3408626150 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fae6006e930 0x7fae6013cfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:01:05.174 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:05.172+0000 7fae64a00640 1 -- 172.21.15.32:0/3408626150 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fae60073e60 msgr2=0x7fae601f4e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:01:05.174 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:05.172+0000 7fae64a00640 1 --2- 172.21.15.32:0/3408626150 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fae60073e60 0x7fae601f4e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:01:05.174 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:05.172+0000 7fae64a00640 1 -- 172.21.15.32:0/3408626150 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fae601f8980 con 0x7fae600747b0 2024-08-03T21:01:05.175 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:05.172+0000 7fae64a00640 1 --2- 172.21.15.32:0/3408626150 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fae600747b0 0x7fae60143aa0 secure :-1 s=READY pgs=2197 cs=0 l=1 rev1=1 crypto rx=0x7fae5800e990 tx=0x7fae5800ee50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T21:01:05.175 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:05.172+0000 7fae5cc00640 1 -- 172.21.15.32:0/3408626150 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae58004d20 con 0x7fae600747b0 2024-08-03T21:01:05.176 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:05.173+0000 7fae5cc00640 1 -- 172.21.15.32:0/3408626150 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fae5800cb00 con 0x7fae600747b0 2024-08-03T21:01:05.176 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:05.173+0000 7fae5cc00640 1 -- 172.21.15.32:0/3408626150 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae58005020 con 0x7fae600747b0 2024-08-03T21:01:05.176 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:05.173+0000 7fae66a00640 1 -- 172.21.15.32:0/3408626150 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fae601f6230 con 0x7fae600747b0 2024-08-03T21:01:05.176 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:05.173+0000 7fae66a00640 1 -- 172.21.15.32:0/3408626150 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fae601f8f20 con 0x7fae600747b0 2024-08-03T21:01:05.176 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:05.174+0000 7fae66a00640 1 -- 172.21.15.32:0/3408626150 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fae60074270 con 0x7fae600747b0 2024-08-03T21:01:05.176 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:05.175+0000 7fae5cc00640 1 -- 172.21.15.32:0/3408626150 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7fae580051c0 con 0x7fae600747b0 2024-08-03T21:01:05.177 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:05.176+0000 7fae5cc00640 1 --2- 172.21.15.32:0/3408626150 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fae3c077010 0x7fae3c079520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:01:05.178 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:05.176+0000 7fae5cc00640 1 -- 172.21.15.32:0/3408626150 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(135..135 src has 1..135) ==== 5709+0+0 (secure 0 0 0) 0x7fae58098fe0 con 0x7fae600747b0 2024-08-03T21:01:05.178 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:05.177+0000 7fae5fe00640 1 --2- 172.21.15.32:0/3408626150 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fae3c077010 0x7fae3c079520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:01:05.179 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:05.177+0000 7fae5fe00640 1 --2- 172.21.15.32:0/3408626150 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fae3c077010 0x7fae3c079520 secure :-1 s=READY pgs=880 cs=0 l=1 rev1=1 crypto rx=0x7fae5000bb50 tx=0x7fae5000b2e0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T21:01:05.182 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:05.181+0000 7fae5cc00640 1 -- 172.21.15.32:0/3408626150 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7fae58016030 con 0x7fae600747b0 2024-08-03T21:01:05.407 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:05.405+0000 7fae66a00640 1 -- 172.21.15.32:0/3408626150 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7fae601f5fc0 con 0x7fae600747b0 2024-08-03T21:01:05.408 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:05.406+0000 7fae5cc00640 1 -- 172.21.15.32:0/3408626150 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v135) ==== 80+0+0 (secure 0 0 0) 0x7fae58061bc0 con 0x7fae600747b0 2024-08-03T21:01:05.408 INFO:teuthology.orchestra.run.smithi032.stderr:listed 0 entries 2024-08-03T21:01:05.410 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:05.409+0000 7fae66a00640 1 -- 172.21.15.32:0/3408626150 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fae3c077010 msgr2=0x7fae3c079520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:01:05.411 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:05.409+0000 7fae66a00640 1 --2- 172.21.15.32:0/3408626150 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fae3c077010 0x7fae3c079520 secure :-1 s=READY pgs=880 cs=0 l=1 rev1=1 crypto rx=0x7fae5000bb50 tx=0x7fae5000b2e0 comp rx=0 tx=0).stop 2024-08-03T21:01:05.411 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:05.409+0000 7fae66a00640 1 -- 172.21.15.32:0/3408626150 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fae600747b0 msgr2=0x7fae60143aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:01:05.411 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:05.409+0000 7fae66a00640 1 --2- 172.21.15.32:0/3408626150 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fae600747b0 0x7fae60143aa0 secure :-1 s=READY pgs=2197 cs=0 l=1 rev1=1 crypto rx=0x7fae5800e990 tx=0x7fae5800ee50 comp rx=0 tx=0).stop 2024-08-03T21:01:05.411 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:05.409+0000 7fae66a00640 1 -- 172.21.15.32:0/3408626150 shutdown_connections 2024-08-03T21:01:05.411 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:05.409+0000 7fae66a00640 1 --2- 172.21.15.32:0/3408626150 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fae3c077010 0x7fae3c079520 unknown :-1 s=CLOSED pgs=880 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:01:05.411 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:05.409+0000 7fae66a00640 1 --2- 172.21.15.32:0/3408626150 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fae60073e60 0x7fae601f4e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:01:05.411 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:05.409+0000 7fae66a00640 1 --2- 172.21.15.32:0/3408626150 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fae600747b0 0x7fae60143aa0 unknown :-1 s=CLOSED pgs=2197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:01:05.411 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:05.409+0000 7fae66a00640 1 --2- 172.21.15.32:0/3408626150 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fae6006e930 0x7fae6013cfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:01:05.411 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:05.409+0000 7fae66a00640 1 -- 172.21.15.32:0/3408626150 >> 172.21.15.32:0/3408626150 conn(0x7fae600b1bd0 msgr2=0x7fae6015cb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T21:01:05.411 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:05.409+0000 7fae66a00640 1 -- 172.21.15.32:0/3408626150 shutdown_connections 2024-08-03T21:01:05.411 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:05.409+0000 7fae66a00640 1 -- 172.21.15.32:0/3408626150 wait complete. 2024-08-03T21:01:05.434 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-08-03T21:01:05.435 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-08-03T21:01:05.435 DEBUG:teuthology.orchestra.run.smithi032:> sudo umount /home/ubuntu/cephtest/mnt.0 2024-08-03T21:01:05.527 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi032.front.sepia.ceph.com 2024-08-03T21:01:05.528 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-08-03T21:01:05.528 DEBUG:teuthology.orchestra.run.smithi032:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-08-03T21:01:05.584 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-08-03T21:01:05.815 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:05.813+0000 7fe984a00640 1 -- 172.21.15.32:0/2932683865 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe9800cec30 msgr2=0x7fe9800cf040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:01:05.815 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:05.813+0000 7fe984a00640 1 --2- 172.21.15.32:0/2932683865 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe9800cec30 0x7fe9800cf040 secure :-1 s=READY pgs=2199 cs=0 l=1 rev1=1 crypto rx=0x7fe978005fd0 tx=0x7fe97802a820 comp rx=0 tx=0).stop 2024-08-03T21:01:05.815 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:05.813+0000 7fe984a00640 1 -- 172.21.15.32:0/2932683865 shutdown_connections 2024-08-03T21:01:05.815 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:05.813+0000 7fe984a00640 1 --2- 172.21.15.32:0/2932683865 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe9800cec30 0x7fe9800cf040 unknown :-1 s=CLOSED pgs=2199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:01:05.815 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:05.813+0000 7fe984a00640 1 --2- 172.21.15.32:0/2932683865 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fe9800d29a0 0x7fe9800d1c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:01:05.815 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:05.813+0000 7fe984a00640 1 -- 172.21.15.32:0/2932683865 >> 172.21.15.32:0/2932683865 conn(0x7fe9800d4090 msgr2=0x7fe9800d44a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T21:01:05.815 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:05.813+0000 7fe984a00640 1 -- 172.21.15.32:0/2932683865 shutdown_connections 2024-08-03T21:01:05.816 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:05.813+0000 7fe984a00640 1 -- 172.21.15.32:0/2932683865 wait complete. 2024-08-03T21:01:05.817 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:05.816+0000 7fe984a00640 1 Processor -- start 2024-08-03T21:01:05.817 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:05.816+0000 7fe984a00640 1 -- start start 2024-08-03T21:01:05.819 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:05.817+0000 7fe984a00640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fe9800cec30 0x7fe9801d5420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:01:05.820 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:05.817+0000 7fe984a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe9800d4bb0 0x7fe9800c9570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:01:05.820 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:05.817+0000 7fe984a00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fe9800c9ab0 0x7fe9801f4ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:01:05.820 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:05.817+0000 7fe984a00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fe9800cfaa0 con 0x7fe9800d4bb0 2024-08-03T21:01:05.820 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:05.817+0000 7fe984a00640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7fe9800cf920 con 0x7fe9800c9ab0 2024-08-03T21:01:05.820 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:05.817+0000 7fe984a00640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7fe9800cfc20 con 0x7fe9800cec30 2024-08-03T21:01:05.820 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:05.817+0000 7fe975e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe9800d4bb0 0x7fe9800c9570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:01:05.820 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:05.818+0000 7fe975e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe9800d4bb0 0x7fe9800c9570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:50528/0 (socket says 172.21.15.32:50528) 2024-08-03T21:01:05.820 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:05.818+0000 7fe975e00640 1 -- 172.21.15.32:0/4225784967 learned_addr learned my addr 172.21.15.32:0/4225784967 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T21:01:05.821 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:05.818+0000 7fe97e800640 1 --2- 172.21.15.32:0/4225784967 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fe9800cec30 0x7fe9801d5420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:01:05.821 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:05.818+0000 7fe97f200640 1 --2- 172.21.15.32:0/4225784967 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fe9800c9ab0 0x7fe9801f4ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:01:05.821 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:05.818+0000 7fe975e00640 1 -- 172.21.15.32:0/4225784967 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fe9800cec30 msgr2=0x7fe9801d5420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:01:05.821 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:05.818+0000 7fe975e00640 1 --2- 172.21.15.32:0/4225784967 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fe9800cec30 0x7fe9801d5420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:01:05.821 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:05.818+0000 7fe975e00640 1 -- 172.21.15.32:0/4225784967 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fe9800c9ab0 msgr2=0x7fe9801f4ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:01:05.821 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:05.818+0000 7fe975e00640 1 --2- 172.21.15.32:0/4225784967 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fe9800c9ab0 0x7fe9801f4ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:01:05.821 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:05.818+0000 7fe975e00640 1 -- 172.21.15.32:0/4225784967 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe9801d5960 con 0x7fe9800d4bb0 2024-08-03T21:01:05.821 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:05.818+0000 7fe975e00640 1 --2- 172.21.15.32:0/4225784967 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe9800d4bb0 0x7fe9800c9570 secure :-1 s=READY pgs=2200 cs=0 l=1 rev1=1 crypto rx=0x7fe96800e790 tx=0x7fe96800ec50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T21:01:05.822 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:05.818+0000 7fe977400640 1 -- 172.21.15.32:0/4225784967 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe96800ca60 con 0x7fe9800d4bb0 2024-08-03T21:01:05.822 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:05.819+0000 7fe977400640 1 -- 172.21.15.32:0/4225784967 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fe96800ccf0 con 0x7fe9800d4bb0 2024-08-03T21:01:05.822 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:05.819+0000 7fe984a00640 1 -- 172.21.15.32:0/4225784967 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe9801f5bf0 con 0x7fe9800d4bb0 2024-08-03T21:01:05.822 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:05.819+0000 7fe977400640 1 -- 172.21.15.32:0/4225784967 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe968010690 con 0x7fe9800d4bb0 2024-08-03T21:01:05.822 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:05.819+0000 7fe984a00640 1 -- 172.21.15.32:0/4225784967 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe9801f5eb0 con 0x7fe9800d4bb0 2024-08-03T21:01:05.822 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:05.820+0000 7fe984a00640 1 -- 172.21.15.32:0/4225784967 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe9801f65d0 con 0x7fe9800d4bb0 2024-08-03T21:01:05.823 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:05.821+0000 7fe977400640 1 -- 172.21.15.32:0/4225784967 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7fe968010830 con 0x7fe9800d4bb0 2024-08-03T21:01:05.824 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:05.822+0000 7fe977400640 1 --2- 172.21.15.32:0/4225784967 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fe95c077010 0x7fe95c079520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:01:05.824 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:05.822+0000 7fe977400640 1 -- 172.21.15.32:0/4225784967 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(135..135 src has 1..135) ==== 5709+0+0 (secure 0 0 0) 0x7fe9680992a0 con 0x7fe9800d4bb0 2024-08-03T21:01:05.824 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:05.823+0000 7fe97e800640 1 --2- 172.21.15.32:0/4225784967 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fe95c077010 0x7fe95c079520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:01:05.825 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:05.824+0000 7fe97e800640 1 --2- 172.21.15.32:0/4225784967 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fe95c077010 0x7fe95c079520 secure :-1 s=READY pgs=881 cs=0 l=1 rev1=1 crypto rx=0x7fe970001610 tx=0x7fe970009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T21:01:05.828 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:05.826+0000 7fe977400640 1 -- 172.21.15.32:0/4225784967 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7fe968062490 con 0x7fe9800d4bb0 2024-08-03T21:01:06.073 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:06.071+0000 7fe984a00640 1 -- 172.21.15.32:0/4225784967 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe9800cf040 con 0x7fe9800d4bb0 2024-08-03T21:01:06.073 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:06.072+0000 7fe977400640 1 -- 172.21.15.32:0/4225784967 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 340 v340) ==== 95+0+5234 (secure 0 0 0) 0x7fe968061d90 con 0x7fe9800d4bb0 2024-08-03T21:01:06.073 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-03T21:01:06.074 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":340,"btime":"2024-08-03T21:00:53:231957+0000","default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8519,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6836/3249267506","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6834","nonce":3249267506},{"type":"v1","addr":"172.21.15.118:6836","nonce":3249267506}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":334},{"gid":8522,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6837/408542078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6835","nonce":408542078},{"type":"v1","addr":"172.21.15.118:6837","nonce":408542078}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":335},{"gid":9589,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6834/3440267920","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":3440267920},{"type":"v1","addr":"172.21.15.32:6834","nonce":3440267920}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":335}],"filesystems":[{"mdsmap":{"epoch":340,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-08-03T21:00:49.742850+0000","modified":"2024-08-03T21:00:52.750915+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":9592},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9592":{"gid":9592,"name":"a","rank":0,"incarnation":336,"state":"up:active","state_seq":3,"addr":"172.21.15.32:6835/4032603982","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6833","nonce":4032603982},{"type":"v1","addr":"172.21.15.32:6835","nonce":4032603982}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9592,"qdb_cluster":[9592]},"id":7}]} 2024-08-03T21:01:06.074 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 340 2024-08-03T21:01:06.076 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:06.074+0000 7fe984a00640 1 -- 172.21.15.32:0/4225784967 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fe95c077010 msgr2=0x7fe95c079520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:01:06.076 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:06.074+0000 7fe984a00640 1 --2- 172.21.15.32:0/4225784967 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fe95c077010 0x7fe95c079520 secure :-1 s=READY pgs=881 cs=0 l=1 rev1=1 crypto rx=0x7fe970001610 tx=0x7fe970009040 comp rx=0 tx=0).stop 2024-08-03T21:01:06.076 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:06.074+0000 7fe984a00640 1 -- 172.21.15.32:0/4225784967 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe9800d4bb0 msgr2=0x7fe9800c9570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:01:06.076 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:06.074+0000 7fe984a00640 1 --2- 172.21.15.32:0/4225784967 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe9800d4bb0 0x7fe9800c9570 secure :-1 s=READY pgs=2200 cs=0 l=1 rev1=1 crypto rx=0x7fe96800e790 tx=0x7fe96800ec50 comp rx=0 tx=0).stop 2024-08-03T21:01:06.076 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:06.074+0000 7fe984a00640 1 -- 172.21.15.32:0/4225784967 shutdown_connections 2024-08-03T21:01:06.076 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:06.074+0000 7fe984a00640 1 --2- 172.21.15.32:0/4225784967 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fe95c077010 0x7fe95c079520 unknown :-1 s=CLOSED pgs=881 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:01:06.076 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:06.074+0000 7fe984a00640 1 --2- 172.21.15.32:0/4225784967 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fe9800c9ab0 0x7fe9801f4ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:01:06.076 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:06.074+0000 7fe984a00640 1 --2- 172.21.15.32:0/4225784967 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe9800d4bb0 0x7fe9800c9570 unknown :-1 s=CLOSED pgs=2200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:01:06.076 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:06.074+0000 7fe984a00640 1 --2- 172.21.15.32:0/4225784967 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fe9800cec30 0x7fe9801d5420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:01:06.076 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:06.074+0000 7fe984a00640 1 -- 172.21.15.32:0/4225784967 >> 172.21.15.32:0/4225784967 conn(0x7fe9800d4090 msgr2=0x7fe9800b64d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T21:01:06.076 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:06.074+0000 7fe984a00640 1 -- 172.21.15.32:0/4225784967 shutdown_connections 2024-08-03T21:01:06.077 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:06.074+0000 7fe984a00640 1 -- 172.21.15.32:0/4225784967 wait complete. 2024-08-03T21:01:06.102 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json flush journal 2024-08-03T21:01:06.363 INFO:teuthology.orchestra.run.smithi032.stdout:{"message":"","return_code":0} 2024-08-03T21:01:06.364 DEBUG:tasks.cephfs.filesystem:_json_asok output { "message": "", "return_code": 0 } 2024-08-03T21:01:06.364 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-08-03T21:01:06.594 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:06.592+0000 7f73dde00640 1 -- 172.21.15.32:0/4054419007 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f73d8165160 msgr2=0x7f73d8167580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:01:06.595 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:06.592+0000 7f73dde00640 1 --2- 172.21.15.32:0/4054419007 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f73d8165160 0x7f73d8167580 secure :-1 s=READY pgs=2202 cs=0 l=1 rev1=1 crypto rx=0x7f73d0005fd0 tx=0x7f73d002a820 comp rx=0 tx=0).stop 2024-08-03T21:01:06.595 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:06.593+0000 7f73dde00640 1 -- 172.21.15.32:0/4054419007 shutdown_connections 2024-08-03T21:01:06.595 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:06.593+0000 7f73dde00640 1 --2- 172.21.15.32:0/4054419007 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f73d8165160 0x7f73d8167580 unknown :-1 s=CLOSED pgs=2202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:01:06.595 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:06.593+0000 7f73dde00640 1 -- 172.21.15.32:0/4054419007 >> 172.21.15.32:0/4054419007 conn(0x7f73d80b1bd0 msgr2=0x7f73d80b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T21:01:06.595 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:06.593+0000 7f73dde00640 1 -- 172.21.15.32:0/4054419007 shutdown_connections 2024-08-03T21:01:06.595 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:06.593+0000 7f73dde00640 1 -- 172.21.15.32:0/4054419007 wait complete. 2024-08-03T21:01:06.596 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:06.595+0000 7f73dde00640 1 Processor -- start 2024-08-03T21:01:06.596 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:06.595+0000 7f73dde00640 1 -- start start 2024-08-03T21:01:06.598 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:06.596+0000 7f73dde00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f73d806e930 0x7f73d8158d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:01:06.599 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:06.596+0000 7f73dde00640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f73d8144f90 0x7f73d813b7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:01:06.599 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:06.596+0000 7f73dde00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f73d8165160 0x7f73d813bce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:01:06.599 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:06.596+0000 7f73dde00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f73d80cf9c0 con 0x7f73d806e930 2024-08-03T21:01:06.599 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:06.596+0000 7f73dde00640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f73d80cf840 con 0x7f73d8165160 2024-08-03T21:01:06.599 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:06.596+0000 7f73dde00640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7f73d8167ac0 con 0x7f73d8144f90 2024-08-03T21:01:06.599 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:06.597+0000 7f73d6a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f73d806e930 0x7f73d8158d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:01:06.599 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:06.597+0000 7f73d6a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f73d806e930 0x7f73d8158d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:50546/0 (socket says 172.21.15.32:50546) 2024-08-03T21:01:06.599 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:06.597+0000 7f73d6a00640 1 -- 172.21.15.32:0/3932226632 learned_addr learned my addr 172.21.15.32:0/3932226632 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T21:01:06.599 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:06.597+0000 7f73d7400640 1 --2- 172.21.15.32:0/3932226632 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f73d8165160 0x7f73d813bce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:01:06.599 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:06.597+0000 7f73d6000640 1 --2- 172.21.15.32:0/3932226632 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f73d8144f90 0x7f73d813b7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:01:06.600 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:06.597+0000 7f73d6a00640 1 -- 172.21.15.32:0/3932226632 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f73d8144f90 msgr2=0x7f73d813b7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:01:06.600 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:06.597+0000 7f73d6a00640 1 --2- 172.21.15.32:0/3932226632 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f73d8144f90 0x7f73d813b7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:01:06.600 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:06.597+0000 7f73d6a00640 1 -- 172.21.15.32:0/3932226632 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f73d8165160 msgr2=0x7f73d813bce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:01:06.600 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:06.597+0000 7f73d6a00640 1 --2- 172.21.15.32:0/3932226632 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f73d8165160 0x7f73d813bce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:01:06.600 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:06.597+0000 7f73d6a00640 1 -- 172.21.15.32:0/3932226632 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f73d813e7f0 con 0x7f73d806e930 2024-08-03T21:01:06.601 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:06.597+0000 7f73d6a00640 1 --2- 172.21.15.32:0/3932226632 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f73d806e930 0x7f73d8158d50 secure :-1 s=READY pgs=2203 cs=0 l=1 rev1=1 crypto rx=0x7f73c800b3f0 tx=0x7f73c800b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T21:01:06.601 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:06.598+0000 7f73cf400640 1 -- 172.21.15.32:0/3932226632 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73c8015020 con 0x7f73d806e930 2024-08-03T21:01:06.601 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:06.598+0000 7f73cf400640 1 -- 172.21.15.32:0/3932226632 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f73c8011920 con 0x7f73d806e930 2024-08-03T21:01:06.601 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:06.598+0000 7f73dde00640 1 -- 172.21.15.32:0/3932226632 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f73d813ead0 con 0x7f73d806e930 2024-08-03T21:01:06.601 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:06.598+0000 7f73cf400640 1 -- 172.21.15.32:0/3932226632 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73c8011d70 con 0x7f73d806e930 2024-08-03T21:01:06.601 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:06.598+0000 7f73dde00640 1 -- 172.21.15.32:0/3932226632 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f73d813f090 con 0x7f73d806e930 2024-08-03T21:01:06.601 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:06.599+0000 7f73dde00640 1 -- 172.21.15.32:0/3932226632 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f73d8073e60 con 0x7f73d806e930 2024-08-03T21:01:06.602 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:06.600+0000 7f73cf400640 1 -- 172.21.15.32:0/3932226632 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f73c8018020 con 0x7f73d806e930 2024-08-03T21:01:06.602 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:06.601+0000 7f73cf400640 1 --2- 172.21.15.32:0/3932226632 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f73b8077010 0x7f73b8079520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:01:06.603 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:06.601+0000 7f73cf400640 1 -- 172.21.15.32:0/3932226632 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(135..135 src has 1..135) ==== 5709+0+0 (secure 0 0 0) 0x7f73c8098740 con 0x7f73d806e930 2024-08-03T21:01:06.603 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:06.602+0000 7f73d6000640 1 --2- 172.21.15.32:0/3932226632 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f73b8077010 0x7f73b8079520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:01:06.604 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:06.602+0000 7f73d6000640 1 --2- 172.21.15.32:0/3932226632 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f73b8077010 0x7f73b8079520 secure :-1 s=READY pgs=882 cs=0 l=1 rev1=1 crypto rx=0x7f73d8165110 tx=0x7f73c4009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T21:01:06.607 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:06.605+0000 7f73cf400640 1 -- 172.21.15.32:0/3932226632 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7f73c801f020 con 0x7f73d806e930 2024-08-03T21:01:06.854 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:06.852+0000 7f73dde00640 1 -- 172.21.15.32:0/3932226632 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7f73d81457e0 con 0x7f73d806e930 2024-08-03T21:01:06.854 INFO:tasks.ceph.mon.a.smithi032.stderr:2024-08-03T21:01:06.853+0000 7f4e6d600640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-08-03T21:01:06.856 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:06.855+0000 7f73cf400640 1 -- 172.21.15.32:0/3932226632 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v341) ==== 161+0+0 (secure 0 0 0) 0x7f73c8061370 con 0x7f73d806e930 2024-08-03T21:01:06.857 INFO:teuthology.orchestra.run.smithi032.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-08-03T21:01:06.859 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:06.857+0000 7f73dde00640 1 -- 172.21.15.32:0/3932226632 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f73b8077010 msgr2=0x7f73b8079520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:01:06.859 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:06.857+0000 7f73dde00640 1 --2- 172.21.15.32:0/3932226632 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f73b8077010 0x7f73b8079520 secure :-1 s=READY pgs=882 cs=0 l=1 rev1=1 crypto rx=0x7f73d8165110 tx=0x7f73c4009040 comp rx=0 tx=0).stop 2024-08-03T21:01:06.860 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:06.857+0000 7f73dde00640 1 -- 172.21.15.32:0/3932226632 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f73d806e930 msgr2=0x7f73d8158d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:01:06.860 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:06.857+0000 7f73dde00640 1 --2- 172.21.15.32:0/3932226632 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f73d806e930 0x7f73d8158d50 secure :-1 s=READY pgs=2203 cs=0 l=1 rev1=1 crypto rx=0x7f73c800b3f0 tx=0x7f73c800b8b0 comp rx=0 tx=0).stop 2024-08-03T21:01:06.860 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:06.858+0000 7f73dde00640 1 -- 172.21.15.32:0/3932226632 shutdown_connections 2024-08-03T21:01:06.860 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:06.858+0000 7f73dde00640 1 --2- 172.21.15.32:0/3932226632 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f73b8077010 0x7f73b8079520 unknown :-1 s=CLOSED pgs=882 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:01:06.860 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:06.858+0000 7f73dde00640 1 --2- 172.21.15.32:0/3932226632 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f73d8165160 0x7f73d813bce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:01:06.860 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:06.858+0000 7f73dde00640 1 --2- 172.21.15.32:0/3932226632 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f73d8144f90 0x7f73d813b7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:01:06.860 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:06.858+0000 7f73dde00640 1 --2- 172.21.15.32:0/3932226632 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f73d806e930 0x7f73d8158d50 unknown :-1 s=CLOSED pgs=2203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:01:06.860 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:06.858+0000 7f73dde00640 1 -- 172.21.15.32:0/3932226632 >> 172.21.15.32:0/3932226632 conn(0x7f73d80b1bd0 msgr2=0x7f73d806ef30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T21:01:06.860 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:06.858+0000 7f73dde00640 1 -- 172.21.15.32:0/3932226632 shutdown_connections 2024-08-03T21:01:06.860 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:06.858+0000 7f73dde00640 1 -- 172.21.15.32:0/3932226632 wait complete. 2024-08-03T21:01:06.871 INFO:tasks.ceph.mds.a.smithi032.stderr: -7206> 2024-08-03T21:00:52.190+0000 7ffaa7000640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-08-03T21:01:06.882 DEBUG:teuthology.orchestra.run.smithi032:> cephfs-journal-tool --debug-mds=20 --debug-ms=1 --debug-objecter=1 --rank cephfs:0 journal reset --yes-i-really-really-mean-it 2024-08-03T21:01:06.994 INFO:tasks.ceph.mds.a.smithi032.stdout:starting mds.a at 2024-08-03T21:01:07.027 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:07.025+0000 7f39c760e280 1 -- start start 2024-08-03T21:01:07.027 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:07.026+0000 7f39c760e280 1 Processor -- start 2024-08-03T21:01:07.030 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:07.027+0000 7f39c760e280 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x564f6d7cf680 0x564f6e4ee000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:01:07.030 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:07.027+0000 7f39c760e280 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x564f6d7cfb00 0x564f6e4ee580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:01:07.031 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:07.027+0000 7f39c760e280 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x564f6d7d0400 0x564f6e4eeb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:01:07.031 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:07.027+0000 7f39c760e280 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x564f6e4fe600 con 0x564f6d7d0400 2024-08-03T21:01:07.031 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:07.027+0000 7f39c760e280 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x564f6e4fe480 con 0x564f6d7cfb00 2024-08-03T21:01:07.031 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:07.027+0000 7f39c760e280 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x564f6e4fe300 con 0x564f6d7cf680 2024-08-03T21:01:07.031 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:07.027+0000 7f39c5400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x564f6d7d0400 0x564f6e4eeb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:01:07.031 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:07.027+0000 7f39c5400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x564f6d7d0400 0x564f6e4eeb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:50580/0 (socket says 172.21.15.32:50580) 2024-08-03T21:01:07.031 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:07.027+0000 7f39c5400640 1 -- 172.21.15.32:0/1141641759 learned_addr learned my addr 172.21.15.32:0/1141641759 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T21:01:07.031 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:07.028+0000 7f39c4000640 1 --2- 172.21.15.32:0/1141641759 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x564f6d7cfb00 0x564f6e4ee580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:01:07.031 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:07.028+0000 7f39c4a00640 1 --2- 172.21.15.32:0/1141641759 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x564f6d7cf680 0x564f6e4ee000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:01:07.031 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:07.028+0000 7f39c5400640 1 -- 172.21.15.32:0/1141641759 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x564f6d7cf680 msgr2=0x564f6e4ee000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T21:01:07.031 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:07.028+0000 7f39c5400640 1 --2- 172.21.15.32:0/1141641759 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x564f6d7cf680 0x564f6e4ee000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:01:07.031 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:07.028+0000 7f39c5400640 1 -- 172.21.15.32:0/1141641759 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x564f6d7cfb00 msgr2=0x564f6e4ee580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T21:01:07.031 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:07.028+0000 7f39c5400640 1 --2- 172.21.15.32:0/1141641759 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x564f6d7cfb00 0x564f6e4ee580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:01:07.032 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:07.028+0000 7f39c5400640 1 -- 172.21.15.32:0/1141641759 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564f6971a780 con 0x564f6d7d0400 2024-08-03T21:01:07.032 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:07.028+0000 7f39c4000640 1 --2- 172.21.15.32:0/1141641759 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x564f6d7cfb00 0x564f6e4ee580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-03T21:01:07.032 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:07.028+0000 7f39c4a00640 1 --2- 172.21.15.32:0/1141641759 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x564f6d7cf680 0x564f6e4ee000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-03T21:01:07.032 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:07.028+0000 7f39c5400640 1 --2- 172.21.15.32:0/1141641759 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x564f6d7d0400 0x564f6e4eeb00 secure :-1 s=READY pgs=2207 cs=0 l=1 rev1=1 crypto rx=0x564f6e537da0 tx=0x564f6d8344e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9e4a0d9ea662badf server_cookie=0 in_seq=0 out_seq=0 2024-08-03T21:01:07.032 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:07.028+0000 7f39c1800640 1 -- 172.21.15.32:0/1141641759 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x564f69709520 con 0x564f6d7d0400 2024-08-03T21:01:07.032 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:07.028+0000 7f39c1800640 1 -- 172.21.15.32:0/1141641759 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x564f6970efc0 con 0x564f6d7d0400 2024-08-03T21:01:07.032 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:07.028+0000 7f39c1800640 1 -- 172.21.15.32:0/1141641759 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x564f697089c0 con 0x564f6d7d0400 2024-08-03T21:01:07.032 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:07.028+0000 7f39c760e280 1 -- 172.21.15.32:0/1141641759 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564f6971a3c0 con 0x564f6d7d0400 2024-08-03T21:01:07.032 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:07.028+0000 7f39c760e280 1 -- 172.21.15.32:0/1141641759 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({fsmap=0}) -- 0x564f6971b680 con 0x564f6d7d0400 2024-08-03T21:01:07.032 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:07.028+0000 7f39c760e280 4 waiting for MDS map... 2024-08-03T21:01:07.032 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:07.029+0000 7f39c1800640 1 -- 172.21.15.32:0/1141641759 <== mon.0 v2:172.21.15.32:3300/0 4 ==== osd_map(136..136 src has 1..136) ==== 5795+0+0 (secure 0 0 0) 0x564f6e5f6600 con 0x564f6d7d0400 2024-08-03T21:01:07.032 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:07.029+0000 7f39c1800640 1 -- 172.21.15.32:0/1141641759 <== mon.0 v2:172.21.15.32:3300/0 5 ==== fsmap(e 341) ==== 2893+0+0 (secure 0 0 0) 0x564f6d76d180 con 0x564f6d7d0400 2024-08-03T21:01:07.032 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:07.029+0000 7f39c760e280 4 Got MDS map 341 2024-08-03T21:01:07.033 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:07.029+0000 7f39c760e280 10 main: JournalTool::main 2024-08-03T21:01:07.033 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:07.029+0000 7f39c760e280 4 main: JournalTool: connecting to RADOS... 2024-08-03T21:01:07.034 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:07.030+0000 7f39c760e280 1 Processor -- start 2024-08-03T21:01:07.034 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:07.030+0000 7f39c760e280 1 -- start start 2024-08-03T21:01:07.034 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:07.031+0000 7f39c760e280 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x564f6d7ced80 0x564f6e4ef080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:01:07.034 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:07.031+0000 7f39c760e280 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x564f6d7ce900 0x564f6e4ef600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:01:07.035 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:07.031+0000 7f39c760e280 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x564f6d7ce480 0x564f6e4efb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:01:07.035 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:07.031+0000 7f39c760e280 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x564f6e4fe180 con 0x564f6d7ced80 2024-08-03T21:01:07.035 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:07.031+0000 7f39c760e280 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x564f6d775380 con 0x564f6d7ce480 2024-08-03T21:01:07.035 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:07.031+0000 7f39c760e280 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x564f6d775800 con 0x564f6d7ce900 2024-08-03T21:01:07.035 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:07.032+0000 7f39c4000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x564f6d7ced80 0x564f6e4ef080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:01:07.035 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:07.032+0000 7f39c4000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x564f6d7ced80 0x564f6e4ef080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:50596/0 (socket says 172.21.15.32:50596) 2024-08-03T21:01:07.035 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:07.032+0000 7f39c4000640 1 -- 172.21.15.32:0/682105847 learned_addr learned my addr 172.21.15.32:0/682105847 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T21:01:07.035 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:07.032+0000 7f39c4a00640 1 --2- 172.21.15.32:0/682105847 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x564f6d7ce900 0x564f6e4ef600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:01:07.035 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:07.032+0000 7f39c4000640 1 --2- 172.21.15.32:0/682105847 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x564f6d7ce480 0x564f6e4efb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:01:07.035 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:07.032+0000 7f39c4000640 1 -- 172.21.15.32:0/682105847 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x564f6d7ce900 msgr2=0x564f6e4ef600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T21:01:07.035 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:07.032+0000 7f39c4000640 1 --2- 172.21.15.32:0/682105847 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x564f6d7ce900 0x564f6e4ef600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:01:07.035 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:07.032+0000 7f39c4000640 1 -- 172.21.15.32:0/682105847 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x564f6d7ce480 msgr2=0x564f6e4efb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T21:01:07.035 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:07.032+0000 7f39c4000640 1 --2- 172.21.15.32:0/682105847 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x564f6d7ce480 0x564f6e4efb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:01:07.035 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:07.032+0000 7f39c4000640 1 -- 172.21.15.32:0/682105847 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564f6971b4a0 con 0x564f6d7ced80 2024-08-03T21:01:07.036 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:07.033+0000 7f39c4000640 1 --2- 172.21.15.32:0/682105847 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x564f6d7ced80 0x564f6e4ef080 secure :-1 s=READY pgs=2208 cs=0 l=1 rev1=1 crypto rx=0x564f6e611110 tx=0x564f6e66a120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=27ac869bc0c12771 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T21:01:07.036 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:07.033+0000 7f39bf000640 1 -- 172.21.15.32:0/682105847 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x564f69709520 con 0x564f6d7ced80 2024-08-03T21:01:07.036 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:07.033+0000 7f39bf000640 1 -- 172.21.15.32:0/682105847 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x564f6970e380 con 0x564f6d7ced80 2024-08-03T21:01:07.036 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:07.033+0000 7f39bf000640 1 -- 172.21.15.32:0/682105847 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x564f697089c0 con 0x564f6d7ced80 2024-08-03T21:01:07.037 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:07.034+0000 7f39c760e280 1 -- 172.21.15.32:0/682105847 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x564f6d7ced80 msgr2=0x564f6e4ef080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:01:07.037 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:07.034+0000 7f39c760e280 1 --2- 172.21.15.32:0/682105847 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x564f6d7ced80 0x564f6e4ef080 secure :-1 s=READY pgs=2208 cs=0 l=1 rev1=1 crypto rx=0x564f6e611110 tx=0x564f6e66a120 comp rx=0 tx=0).stop 2024-08-03T21:01:07.037 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:07.034+0000 7f39c760e280 1 -- 172.21.15.32:0/682105847 shutdown_connections 2024-08-03T21:01:07.037 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:07.034+0000 7f39c760e280 1 --2- 172.21.15.32:0/682105847 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x564f6d7ce480 0x564f6e4efb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:01:07.037 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:07.034+0000 7f39c760e280 1 --2- 172.21.15.32:0/682105847 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x564f6d7ce900 0x564f6e4ef600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:01:07.037 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:07.034+0000 7f39c760e280 1 --2- 172.21.15.32:0/682105847 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x564f6d7ced80 0x564f6e4ef080 secure :-1 s=CLOSED pgs=2208 cs=0 l=1 rev1=1 crypto rx=0x564f6e611110 tx=0x564f6e66a120 comp rx=0 tx=0).stop 2024-08-03T21:01:07.037 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:07.034+0000 7f39c760e280 1 -- 172.21.15.32:0/682105847 >> 172.21.15.32:0/682105847 conn(0x564f6d7cf200 msgr2=0x564f6d83b800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T21:01:07.037 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:07.034+0000 7f39c760e280 1 -- 172.21.15.32:0/682105847 shutdown_connections 2024-08-03T21:01:07.037 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:07.034+0000 7f39c760e280 1 -- 172.21.15.32:0/682105847 wait complete. 2024-08-03T21:01:07.037 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:07.035+0000 7f39c760e280 1 Processor -- start 2024-08-03T21:01:07.038 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:07.035+0000 7f39c760e280 1 -- start start 2024-08-03T21:01:07.040 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:07.037+0000 7f39c760e280 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x564f6d7ce480 0x564f6e4efb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:01:07.040 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:07.037+0000 7f39c760e280 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x564f6d7ce900 0x564f6e4ef600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:01:07.040 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:07.037+0000 7f39c760e280 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x564f6d7ce000 0x564f6e4f0100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:01:07.040 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:07.037+0000 7f39c760e280 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x564f6e4ff380 con 0x564f6d7ce480 2024-08-03T21:01:07.040 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:07.037+0000 7f39c760e280 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x564f6e4fe180 con 0x564f6d7ce900 2024-08-03T21:01:07.040 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:07.037+0000 7f39c760e280 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x564f6e4ff500 con 0x564f6d7ce000 2024-08-03T21:01:07.040 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:07.037+0000 7f39c4000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x564f6d7ce480 0x564f6e4efb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:01:07.040 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:07.037+0000 7f39c4000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x564f6d7ce480 0x564f6e4efb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:50604/0 (socket says 172.21.15.32:50604) 2024-08-03T21:01:07.040 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:07.037+0000 7f39c4000640 1 -- 172.21.15.32:0/1680063217 learned_addr learned my addr 172.21.15.32:0/1680063217 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T21:01:07.041 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:07.037+0000 7f39c4a00640 1 --2- 172.21.15.32:0/1680063217 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x564f6d7ce900 0x564f6e4ef600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:01:07.041 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:07.037+0000 7f39c4000640 1 --2- 172.21.15.32:0/1680063217 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x564f6d7ce000 0x564f6e4f0100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:01:07.041 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:07.038+0000 7f39c4000640 1 -- 172.21.15.32:0/1680063217 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x564f6d7ce000 msgr2=0x564f6e4f0100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:01:07.041 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:07.038+0000 7f39c4000640 1 --2- 172.21.15.32:0/1680063217 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x564f6d7ce000 0x564f6e4f0100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:01:07.041 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:07.038+0000 7f39c4000640 1 -- 172.21.15.32:0/1680063217 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x564f6d7ce900 msgr2=0x564f6e4ef600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:01:07.041 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:07.038+0000 7f39c4000640 1 --2- 172.21.15.32:0/1680063217 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x564f6d7ce900 0x564f6e4ef600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:01:07.041 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:07.038+0000 7f39c4000640 1 -- 172.21.15.32:0/1680063217 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564f6971a780 con 0x564f6d7ce480 2024-08-03T21:01:07.041 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:07.038+0000 7f39c4a00640 1 --2- 172.21.15.32:0/1680063217 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x564f6d7ce900 0x564f6e4ef600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-03T21:01:07.041 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:07.038+0000 7f39c4000640 1 --2- 172.21.15.32:0/1680063217 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x564f6d7ce480 0x564f6e4efb80 secure :-1 s=READY pgs=2209 cs=0 l=1 rev1=1 crypto rx=0x564f6e611d40 tx=0x564f6e66a300 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T21:01:07.041 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:07.038+0000 7f39bd200640 1 -- 172.21.15.32:0/1680063217 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x564f69708820 con 0x564f6d7ce480 2024-08-03T21:01:07.041 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:07.038+0000 7f39bd200640 1 -- 172.21.15.32:0/1680063217 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x564f6970ee00 con 0x564f6d7ce480 2024-08-03T21:01:07.041 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:07.038+0000 7f39bd200640 1 -- 172.21.15.32:0/1680063217 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x564f69709520 con 0x564f6d7ce480 2024-08-03T21:01:07.041 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:07.038+0000 7f39c760e280 1 -- 172.21.15.32:0/1680063217 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x564f6971a3c0 con 0x564f6d7ce480 2024-08-03T21:01:07.042 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:07.038+0000 7f39c760e280 1 -- 172.21.15.32:0/1680063217 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564f6971a000 con 0x564f6d7ce480 2024-08-03T21:01:07.042 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:07.038+0000 7f39c760e280 4 main: JournalTool: resolving pool 14 2024-08-03T21:01:07.043 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:07.040+0000 7f39bd200640 1 -- 172.21.15.32:0/1680063217 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x564f6e67cc00 con 0x564f6d7ce480 2024-08-03T21:01:07.043 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:07.041+0000 7f39bd200640 1 --2- 172.21.15.32:0/1680063217 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x564f6d7d1180 0x564f6e4f1700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:01:07.043 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:07.041+0000 7f39bd200640 1 -- 172.21.15.32:0/1680063217 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(136..136 src has 1..136) ==== 5795+0+0 (secure 0 0 0) 0x564f6e504a00 con 0x564f6d7ce480 2024-08-03T21:01:07.043 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:07.041+0000 7f39c760e280 4 main: JournalTool: creating IoCtx.. 2024-08-03T21:01:07.043 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:07.041+0000 7f39c760e280 4 main: Executing for rank 0 2024-08-03T21:01:07.043 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:07.041+0000 7f39c4a00640 1 --2- 172.21.15.32:0/1680063217 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x564f6d7d1180 0x564f6e4f1700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:01:07.044 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:07.042+0000 7f39c4a00640 1 --2- 172.21.15.32:0/1680063217 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x564f6d7d1180 0x564f6e4f1700 secure :-1 s=READY pgs=883 cs=0 l=1 rev1=1 crypto rx=0x564f6d818a50 tx=0x564f6db26c00 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T21:01:07.044 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:07.042+0000 7f39c760e280 1 -- start start 2024-08-03T21:01:07.044 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:07.042+0000 7f39c760e280 1 Processor -- start 2024-08-03T21:01:07.047 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:07.044+0000 7f39c760e280 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x564f6d7d0d00 0x564f6e4f0680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:01:07.047 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:07.044+0000 7f39c760e280 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x564f6d7d1600 0x564f6e4f0c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:01:07.047 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:07.044+0000 7f39c760e280 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x564f6d7d1a80 0x564f6e4f1180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:01:07.047 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:07.044+0000 7f39c760e280 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x564f6e4ff800 con 0x564f6d7d1600 2024-08-03T21:01:07.047 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:07.044+0000 7f39c760e280 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x564f6e4ff980 con 0x564f6d7d0d00 2024-08-03T21:01:07.047 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:07.044+0000 7f39c760e280 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x564f6d775800 con 0x564f6d7d1a80 2024-08-03T21:01:07.047 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:07.044+0000 7f39c4a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x564f6d7d1600 0x564f6e4f0c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:01:07.047 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:07.044+0000 7f39c4a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x564f6d7d1600 0x564f6e4f0c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:50608/0 (socket says 172.21.15.32:50608) 2024-08-03T21:01:07.047 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:07.044+0000 7f39c4a00640 1 -- 172.21.15.32:0/1741253874 learned_addr learned my addr 172.21.15.32:0/1741253874 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T21:01:07.048 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:07.044+0000 7f39c4000640 1 --2- 172.21.15.32:0/1741253874 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x564f6d7d1a80 0x564f6e4f1180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:01:07.048 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:07.044+0000 7f39c5400640 1 --2- 172.21.15.32:0/1741253874 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x564f6d7d0d00 0x564f6e4f0680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:01:07.048 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:07.045+0000 7f39c4a00640 1 -- 172.21.15.32:0/1741253874 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x564f6d7d1a80 msgr2=0x564f6e4f1180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T21:01:07.048 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:07.045+0000 7f39c4a00640 1 --2- 172.21.15.32:0/1741253874 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x564f6d7d1a80 0x564f6e4f1180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:01:07.048 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:07.045+0000 7f39c4a00640 1 -- 172.21.15.32:0/1741253874 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x564f6d7d0d00 msgr2=0x564f6e4f0680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T21:01:07.048 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:07.045+0000 7f39c4a00640 1 --2- 172.21.15.32:0/1741253874 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x564f6d7d0d00 0x564f6e4f0680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:01:07.048 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:07.045+0000 7f39c4a00640 1 -- 172.21.15.32:0/1741253874 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564f6971a960 con 0x564f6d7d1600 2024-08-03T21:01:07.048 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:07.045+0000 7f39c4a00640 1 --2- 172.21.15.32:0/1741253874 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x564f6d7d1600 0x564f6e4f0c00 secure :-1 s=READY pgs=2210 cs=0 l=1 rev1=1 crypto rx=0x564f6e60faa0 tx=0x564f6db26d20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=53e446040fd1a5ce server_cookie=0 in_seq=0 out_seq=0 2024-08-03T21:01:07.048 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:07.045+0000 7f39b9600640 1 -- 172.21.15.32:0/1741253874 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x564f69709380 con 0x564f6d7d1600 2024-08-03T21:01:07.048 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:07.045+0000 7f39b9600640 1 -- 172.21.15.32:0/1741253874 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x564f6970e700 con 0x564f6d7d1600 2024-08-03T21:01:07.048 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:07.045+0000 7f39b9600640 1 -- 172.21.15.32:0/1741253874 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x564f6da54680 con 0x564f6d7d1600 2024-08-03T21:01:07.048 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:07.045+0000 7f39c760e280 1 -- 172.21.15.32:0/1741253874 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564f6971b4a0 con 0x564f6d7d1600 2024-08-03T21:01:07.048 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:07.045+0000 7f39c760e280 1 -- 172.21.15.32:0/1741253874 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({fsmap=0}) -- 0x564f6971a780 con 0x564f6d7d1600 2024-08-03T21:01:07.049 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:07.045+0000 7f39c760e280 4 waiting for MDS map... 2024-08-03T21:01:07.049 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:07.045+0000 7f39b9600640 1 -- 172.21.15.32:0/1741253874 <== mon.0 v2:172.21.15.32:3300/0 4 ==== osd_map(136..136 src has 1..136) ==== 5795+0+0 (secure 0 0 0) 0x564f6db72000 con 0x564f6d7d1600 2024-08-03T21:01:07.049 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:07.046+0000 7f39b9600640 1 -- 172.21.15.32:0/1741253874 <== mon.0 v2:172.21.15.32:3300/0 5 ==== fsmap(e 341) ==== 2893+0+0 (secure 0 0 0) 0x564f6d76d500 con 0x564f6d7d1600 2024-08-03T21:01:07.049 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:07.046+0000 7f39c760e280 4 Got MDS map 341 2024-08-03T21:01:07.050 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:07.046+0000 7f39c760e280 1 --2- 172.21.15.32:0/1741253874 >> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] conn(0x564f6db82000 0x564f6db8a000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:01:07.050 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:07.046+0000 7f39c760e280 1 -- 172.21.15.32:0/1741253874 --> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] -- osd_op(unknown.0.0:1 14.f 14:f1f69726:::400.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e136) -- 0x564f6db8e000 con 0x564f6db82000 2024-08-03T21:01:07.050 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:07.046+0000 7f39c5400640 1 --2- 172.21.15.32:0/1741253874 >> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] conn(0x564f6db82000 0x564f6db8a000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:01:07.050 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:07.047+0000 7f39c5400640 1 --2- 172.21.15.32:0/1741253874 >> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] conn(0x564f6db82000 0x564f6db8a000 crc :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=d4b2ea9209c3eede server_cookie=0 in_seq=0 out_seq=0 2024-08-03T21:01:07.052 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:07.050+0000 7f39c5400640 1 -- 172.21.15.32:0/1741253874 <== osd.7 v2:172.21.15.118:6818/653000048 1 ==== osd_op_reply(1 400.00000000 [read 0~22 out=22b] v0'0 uv1 ondisk = 0) ==== 156+0+22 (crc 0 0 0) 0x564f6d84c000 con 0x564f6db82000 2024-08-03T21:01:07.052 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:07.050+0000 7f39c760e280 1 --2- 172.21.15.32:0/1741253874 >> [v2:172.21.15.118:6802/864752275,v1:172.21.15.118:6803/864752275] conn(0x564f6db82480 0x564f6db8a580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:01:07.052 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:07.050+0000 7f39c760e280 1 -- 172.21.15.32:0/1741253874 --> [v2:172.21.15.118:6802/864752275,v1:172.21.15.118:6803/864752275] -- osd_op(unknown.0.0:2 14.14 14:292cf221:::200.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e136) -- 0x564f6db8e400 con 0x564f6db82480 2024-08-03T21:01:07.052 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:07.051+0000 7f39c4000640 1 --2- 172.21.15.32:0/1741253874 >> [v2:172.21.15.118:6802/864752275,v1:172.21.15.118:6803/864752275] conn(0x564f6db82480 0x564f6db8a580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:01:07.053 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:07.051+0000 7f39c4000640 1 --2- 172.21.15.32:0/1741253874 >> [v2:172.21.15.118:6802/864752275,v1:172.21.15.118:6803/864752275] conn(0x564f6db82480 0x564f6db8a580 crc :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=b7c54c97153c3257 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T21:01:07.057 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:07.055+0000 7f39c4000640 1 -- 172.21.15.32:0/1741253874 <== osd.4 v2:172.21.15.118:6802/864752275 1 ==== osd_op_reply(2 200.00000000 [read 0~90 out=90b] v0'0 uv23 ondisk = 0) ==== 156+0+90 (crc 0 0 0) 0x564f6d84c500 con 0x564f6db82480 2024-08-03T21:01:07.057 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:07.055+0000 7f39b7800640 1 -- 172.21.15.32:0/1741253874 --> [v2:172.21.15.118:6802/864752275,v1:172.21.15.118:6803/864752275] -- osd_op(unknown.0.0:3 14.14 14:2e2fa760:::200.00000001:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e136) -- 0x564f6db8e800 con 0x564f6db82480 2024-08-03T21:01:07.057 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:07.055+0000 7f39b7800640 1 --2- 172.21.15.32:0/1741253874 >> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] conn(0x564f6db85600 0x564f6db8bb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:01:07.057 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:07.055+0000 7f39b7800640 1 -- 172.21.15.32:0/1741253874 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:4 14.1b 14:ddb4e4d7:::200.00000002:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e136) -- 0x564f6db8ec00 con 0x564f6db85600 2024-08-03T21:01:07.058 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:07.055+0000 7f39c5400640 1 --2- 172.21.15.32:0/1741253874 >> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] conn(0x564f6db85600 0x564f6db8bb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:01:07.058 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:07.055+0000 7f39c4000640 1 -- 172.21.15.32:0/1741253874 <== osd.4 v2:172.21.15.118:6802/864752275 2 ==== osd_op_reply(3 200.00000001 [stat out=16b] v0'0 uv20 ondisk = 0) ==== 156+0+16 (crc 0 0 0) 0x564f6d84c500 con 0x564f6db82480 2024-08-03T21:01:07.058 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:07.056+0000 7f39c5400640 1 --2- 172.21.15.32:0/1741253874 >> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] conn(0x564f6db85600 0x564f6db8bb80 crc :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=f3eeb4eea99a24ca server_cookie=0 in_seq=0 out_seq=0 2024-08-03T21:01:07.060 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:07.058+0000 7f39c5400640 1 -- 172.21.15.32:0/1741253874 <== osd.1 v2:172.21.15.32:6824/1551241759 1 ==== osd_op_reply(4 200.00000002 [stat] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 156+0+0 (crc 0 0 0) 0x564f6d84c000 con 0x564f6db85600 2024-08-03T21:01:07.060 INFO:teuthology.orchestra.run.smithi032.stdout:old journal was 4226680~2030 2024-08-03T21:01:07.060 INFO:teuthology.orchestra.run.smithi032.stdout:new journal start will be 8388608 (4159898 bytes past old end) 2024-08-03T21:01:07.060 INFO:teuthology.orchestra.run.smithi032.stdout:writing journal head 2024-08-03T21:01:07.060 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:07.058+0000 7f39c760e280 1 -- 172.21.15.32:0/1741253874 --> [v2:172.21.15.118:6802/864752275,v1:172.21.15.118:6803/864752275] -- osd_op(unknown.0.0:5 14.14 14:292cf221:::200.00000000:head [writefull 0~90 in=90b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e136) -- 0x564f6db8f400 con 0x564f6db82480 2024-08-03T21:01:07.062 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:07.060+0000 7f39c4000640 1 -- 172.21.15.32:0/1741253874 <== osd.4 v2:172.21.15.118:6802/864752275 3 ==== osd_op_reply(5 200.00000000 [writefull 0~90] v136'24 uv24 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x564f6d84c500 con 0x564f6db82480 2024-08-03T21:01:07.062 INFO:teuthology.orchestra.run.smithi032.stdout:writing EResetJournal entry 2024-08-03T21:01:07.063 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:07.060+0000 7f39c760e280 1 -- 172.21.15.32:0/1741253874 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:6 14.1b 14:ddb4e4d7:::200.00000002:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e136) -- 0x564f6db8f800 con 0x564f6db85600 2024-08-03T21:01:07.063 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:07.060+0000 7f39c760e280 1 --2- 172.21.15.32:0/1741253874 >> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] conn(0x564f6db82900 0x564f6db8ab00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:01:07.063 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:07.061+0000 7f39c760e280 1 -- 172.21.15.32:0/1741253874 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:7 14.b 14:d5c7a900:::200.00000003:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e136) -- 0x564f6db8fc00 con 0x564f6db82900 2024-08-03T21:01:07.063 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:07.061+0000 7f39c760e280 1 -- 172.21.15.32:0/1741253874 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:8 14.1f 14:f83d65d8:::200.00000004:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e136) -- 0x564f6dba8000 con 0x564f6db82900 2024-08-03T21:01:07.063 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:07.061+0000 7f39c760e280 1 -- 172.21.15.32:0/1741253874 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:9 14.1a 14:5e41ab69:::200.00000005:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e136) -- 0x564f6dba8400 con 0x564f6db82900 2024-08-03T21:01:07.064 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:07.061+0000 7f39c760e280 1 -- 172.21.15.32:0/1741253874 --> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] -- osd_op(unknown.0.0:10 14.1e 14:7bf7e7b5:::200.00000006:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e136) -- 0x564f6dba8800 con 0x564f6db82000 2024-08-03T21:01:07.064 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:07.061+0000 7f39c760e280 1 -- 172.21.15.32:0/1741253874 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- osd_op(unknown.0.0:11 14.1d 14:bec1251c:::200.00000007:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e136) -- 0x564f6dba8c00 con 0x564f6db82900 2024-08-03T21:01:07.064 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:07.061+0000 7f39c4a00640 1 --2- 172.21.15.32:0/1741253874 >> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] conn(0x564f6db82900 0x564f6db8ab00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:01:07.064 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:07.062+0000 7f39c4a00640 1 --2- 172.21.15.32:0/1741253874 >> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] conn(0x564f6db82900 0x564f6db8ab00 crc :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=8938f93bd203d5ff server_cookie=0 in_seq=0 out_seq=0 2024-08-03T21:01:07.064 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:07.062+0000 7f39c5400640 1 -- 172.21.15.32:0/1741253874 <== osd.1 v2:172.21.15.32:6824/1551241759 2 ==== osd_op_reply(6 200.00000002 [delete] v136'2 uv0 ondisk = -2 ((2) No such file or directory)) ==== 156+0+0 (crc 0 0 0) 0x564f6d84c000 con 0x564f6db85600 2024-08-03T21:01:07.064 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:07.062+0000 7f39c5400640 1 -- 172.21.15.32:0/1741253874 <== osd.7 v2:172.21.15.118:6818/653000048 2 ==== osd_op_reply(10 200.00000006 [delete] v136'2 uv0 ondisk = -2 ((2) No such file or directory)) ==== 156+0+0 (crc 0 0 0) 0x564f6d84c000 con 0x564f6db82000 2024-08-03T21:01:07.065 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:07.064+0000 7f39c4a00640 1 -- 172.21.15.32:0/1741253874 <== osd.0 v2:172.21.15.32:6816/3066156116 1 ==== osd_op_reply(9 200.00000005 [delete] v136'2 uv0 ondisk = -2 ((2) No such file or directory)) ==== 156+0+0 (crc 0 0 0) 0x564f6d84c780 con 0x564f6db82900 2024-08-03T21:01:07.065 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:07.064+0000 7f39c4a00640 1 -- 172.21.15.32:0/1741253874 <== osd.0 v2:172.21.15.32:6816/3066156116 2 ==== osd_op_reply(7 200.00000003 [delete] v136'4 uv3 ondisk = -2 ((2) No such file or directory)) ==== 156+0+0 (crc 0 0 0) 0x564f6d84c780 con 0x564f6db82900 2024-08-03T21:01:07.066 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:07.064+0000 7f39c4a00640 1 -- 172.21.15.32:0/1741253874 <== osd.0 v2:172.21.15.32:6816/3066156116 3 ==== osd_op_reply(8 200.00000004 [delete] v136'7 uv6 ondisk = -2 ((2) No such file or directory)) ==== 156+0+0 (crc 0 0 0) 0x564f6d84c780 con 0x564f6db82900 2024-08-03T21:01:07.066 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:07.064+0000 7f39b7800640 1 -- 172.21.15.32:0/1741253874 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:12 14.1b 14:ddb4e4d7:::200.00000002:head [write 0~48 in=48b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e136) -- 0x564f6db8f000 con 0x564f6db85600 2024-08-03T21:01:07.066 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:07.064+0000 7f39c4a00640 1 -- 172.21.15.32:0/1741253874 <== osd.0 v2:172.21.15.32:6816/3066156116 4 ==== osd_op_reply(11 200.00000007 [delete] v136'3 uv2 ondisk = -2 ((2) No such file or directory)) ==== 156+0+0 (crc 0 0 0) 0x564f6d84c780 con 0x564f6db82900 2024-08-03T21:01:07.067 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:07.066+0000 7f39c5400640 1 -- 172.21.15.32:0/1741253874 <== osd.1 v2:172.21.15.32:6824/1551241759 3 ==== osd_op_reply(12 200.00000002 [write 0~48] v136'3 uv3 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x564f6d84c000 con 0x564f6db85600 2024-08-03T21:01:07.067 INFO:teuthology.orchestra.run.smithi032.stdout:done 2024-08-03T21:01:07.069 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:07.066+0000 7f39c760e280 1 -- 172.21.15.32:0/1741253874 >> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] conn(0x564f6db82900 msgr2=0x564f6db8ab00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:01:07.069 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:07.066+0000 7f39c760e280 1 --2- 172.21.15.32:0/1741253874 >> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] conn(0x564f6db82900 0x564f6db8ab00 crc :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:01:07.069 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:07.066+0000 7f39c760e280 1 -- 172.21.15.32:0/1741253874 >> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] conn(0x564f6db85600 msgr2=0x564f6db8bb80 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:01:07.069 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:07.066+0000 7f39c760e280 1 --2- 172.21.15.32:0/1741253874 >> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] conn(0x564f6db85600 0x564f6db8bb80 crc :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:01:07.069 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:07.066+0000 7f39c760e280 1 -- 172.21.15.32:0/1741253874 >> [v2:172.21.15.118:6802/864752275,v1:172.21.15.118:6803/864752275] conn(0x564f6db82480 msgr2=0x564f6db8a580 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:01:07.070 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:07.066+0000 7f39c760e280 1 --2- 172.21.15.32:0/1741253874 >> [v2:172.21.15.118:6802/864752275,v1:172.21.15.118:6803/864752275] conn(0x564f6db82480 0x564f6db8a580 crc :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:01:07.070 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:07.066+0000 7f39c760e280 1 -- 172.21.15.32:0/1741253874 >> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] conn(0x564f6db82000 msgr2=0x564f6db8a000 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:01:07.070 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:07.066+0000 7f39c760e280 1 --2- 172.21.15.32:0/1741253874 >> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] conn(0x564f6db82000 0x564f6db8a000 crc :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:01:07.070 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:07.066+0000 7f39c4000640 1 -- 172.21.15.32:0/1741253874 reap_dead start 2024-08-03T21:01:07.070 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:07.066+0000 7f39c760e280 1 -- 172.21.15.32:0/1741253874 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x564f6d7d1600 msgr2=0x564f6e4f0c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:01:07.070 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:07.066+0000 7f39c760e280 1 --2- 172.21.15.32:0/1741253874 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x564f6d7d1600 0x564f6e4f0c00 secure :-1 s=READY pgs=2210 cs=0 l=1 rev1=1 crypto rx=0x564f6e60faa0 tx=0x564f6db26d20 comp rx=0 tx=0).stop 2024-08-03T21:01:07.070 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:07.066+0000 7f39c4000640 1 -- 172.21.15.32:0/1741253874 reap_dead start 2024-08-03T21:01:07.070 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:07.066+0000 7f39c760e280 1 -- 172.21.15.32:0/1741253874 shutdown_connections 2024-08-03T21:01:07.070 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:07.066+0000 7f39c760e280 1 --2- 172.21.15.32:0/1741253874 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x564f6d7d1600 0x564f6e4f0c00 unknown :-1 s=CLOSED pgs=2210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:01:07.070 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:07.066+0000 7f39c760e280 1 -- 172.21.15.32:0/1741253874 >> 172.21.15.32:0/1741253874 conn(0x564f6d7d0880 msgr2=0x564f6db20800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T21:01:07.070 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:07.066+0000 7f39c760e280 1 -- 172.21.15.32:0/1741253874 shutdown_connections 2024-08-03T21:01:07.070 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:07.066+0000 7f39c760e280 1 -- 172.21.15.32:0/1741253874 wait complete. 2024-08-03T21:01:07.071 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:07.067+0000 7f39c760e280 1 -- 172.21.15.32:0/1680063217 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x564f6d7d1180 msgr2=0x564f6e4f1700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:01:07.071 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:07.067+0000 7f39c760e280 1 --2- 172.21.15.32:0/1680063217 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x564f6d7d1180 0x564f6e4f1700 secure :-1 s=READY pgs=883 cs=0 l=1 rev1=1 crypto rx=0x564f6d818a50 tx=0x564f6db26c00 comp rx=0 tx=0).stop 2024-08-03T21:01:07.071 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:07.067+0000 7f39c760e280 1 -- 172.21.15.32:0/1680063217 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x564f6d7ce480 msgr2=0x564f6e4efb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:01:07.071 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:07.067+0000 7f39c760e280 1 --2- 172.21.15.32:0/1680063217 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x564f6d7ce480 0x564f6e4efb80 secure :-1 s=READY pgs=2209 cs=0 l=1 rev1=1 crypto rx=0x564f6e611d40 tx=0x564f6e66a300 comp rx=0 tx=0).stop 2024-08-03T21:01:07.071 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:07.067+0000 7f39c760e280 1 -- 172.21.15.32:0/1680063217 shutdown_connections 2024-08-03T21:01:07.071 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:07.067+0000 7f39c760e280 1 --2- 172.21.15.32:0/1680063217 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x564f6d7d1180 0x564f6e4f1700 unknown :-1 s=CLOSED pgs=883 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:01:07.071 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:07.067+0000 7f39c760e280 1 --2- 172.21.15.32:0/1680063217 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x564f6d7ce900 0x564f6e4ef600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:01:07.071 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:07.067+0000 7f39c760e280 1 --2- 172.21.15.32:0/1680063217 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x564f6d7ce000 0x564f6e4f0100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:01:07.071 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:07.067+0000 7f39c760e280 1 --2- 172.21.15.32:0/1680063217 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x564f6d7ce480 0x564f6e4efb80 unknown :-1 s=CLOSED pgs=2209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:01:07.071 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:07.067+0000 7f39c760e280 1 -- 172.21.15.32:0/1680063217 >> 172.21.15.32:0/1680063217 conn(0x564f6d7cf200 msgr2=0x564f6d83b800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T21:01:07.071 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:07.067+0000 7f39c760e280 1 -- 172.21.15.32:0/1680063217 shutdown_connections 2024-08-03T21:01:07.071 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:07.067+0000 7f39c760e280 1 -- 172.21.15.32:0/1680063217 wait complete. 2024-08-03T21:01:07.071 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:07.067+0000 7f39c760e280 1 -- 172.21.15.32:0/1141641759 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x564f6d7d0400 msgr2=0x564f6e4eeb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:01:07.071 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:07.067+0000 7f39c760e280 1 --2- 172.21.15.32:0/1141641759 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x564f6d7d0400 0x564f6e4eeb00 secure :-1 s=READY pgs=2207 cs=0 l=1 rev1=1 crypto rx=0x564f6e537da0 tx=0x564f6d8344e0 comp rx=0 tx=0).stop 2024-08-03T21:01:07.072 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:07.067+0000 7f39c760e280 1 -- 172.21.15.32:0/1141641759 shutdown_connections 2024-08-03T21:01:07.072 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:07.067+0000 7f39c760e280 1 --2- 172.21.15.32:0/1141641759 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x564f6d7cfb00 0x564f6e4ee580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:01:07.072 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:07.067+0000 7f39c760e280 1 --2- 172.21.15.32:0/1141641759 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x564f6d7d0400 0x564f6e4eeb00 unknown :-1 s=CLOSED pgs=2207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:01:07.072 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:07.067+0000 7f39c760e280 1 --2- 172.21.15.32:0/1141641759 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x564f6d7cf680 0x564f6e4ee000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:01:07.072 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:07.067+0000 7f39c760e280 1 -- 172.21.15.32:0/1141641759 >> 172.21.15.32:0/1141641759 conn(0x564f6d7cff80 msgr2=0x564f6d839800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T21:01:07.072 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:07.068+0000 7f39c760e280 1 -- 172.21.15.32:0/1141641759 shutdown_connections 2024-08-03T21:01:07.072 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:07.068+0000 7f39c760e280 1 -- 172.21.15.32:0/1141641759 wait complete. 2024-08-03T21:01:07.083 DEBUG:tasks.cephfs.filesystem:Ran ['cephfs-journal-tool', '--debug-mds=20', '--debug-ms=1', '--debug-objecter=1', '--rank', 'cephfs:0', 'journal', 'reset', '--yes-i-really-really-mean-it'] in time 0:00:00.201001, result: old journal was 4226680~2030 new journal start will be 8388608 (4159898 bytes past old end) writing journal head writing EResetJournal entry done 2024-08-03T21:01:07.083 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs joinable true 2024-08-03T21:01:07.314 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:07.312+0000 7ff82f600640 1 -- 172.21.15.32:0/4072495904 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff8280cccf0 msgr2=0x7ff8280cd100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:01:07.314 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:07.312+0000 7ff82f600640 1 --2- 172.21.15.32:0/4072495904 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff8280cccf0 0x7ff8280cd100 secure :-1 s=READY pgs=2212 cs=0 l=1 rev1=1 crypto rx=0x7ff828074a30 tx=0x7ff81402a4a0 comp rx=0 tx=0).stop 2024-08-03T21:01:07.314 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:07.312+0000 7ff82f600640 1 -- 172.21.15.32:0/4072495904 shutdown_connections 2024-08-03T21:01:07.314 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:07.312+0000 7ff82f600640 1 --2- 172.21.15.32:0/4072495904 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff8280cccf0 0x7ff8280cd100 unknown :-1 s=CLOSED pgs=2212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:01:07.315 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:07.312+0000 7ff82f600640 1 --2- 172.21.15.32:0/4072495904 >> v2:172.21.15.118:3300/0 conn(0x7ff828145c50 0x7ff82806e950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:01:07.315 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:07.312+0000 7ff82f600640 1 -- 172.21.15.32:0/4072495904 >> 172.21.15.32:0/4072495904 conn(0x7ff8280b1bd0 msgr2=0x7ff8280b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T21:01:07.315 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:07.312+0000 7ff82f600640 1 -- 172.21.15.32:0/4072495904 shutdown_connections 2024-08-03T21:01:07.315 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:07.312+0000 7ff82f600640 1 -- 172.21.15.32:0/4072495904 wait complete. 2024-08-03T21:01:07.316 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:07.315+0000 7ff82f600640 1 Processor -- start 2024-08-03T21:01:07.316 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:07.315+0000 7ff82f600640 1 -- start start 2024-08-03T21:01:07.319 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:07.316+0000 7ff82f600640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff8280cccf0 0x7ff8280c9250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:01:07.319 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:07.316+0000 7ff82f600640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ff8280ced30 0x7ff828153c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:01:07.319 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:07.316+0000 7ff82f600640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7ff828145c50 0x7ff82813e700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:01:07.319 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:07.316+0000 7ff82f600640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7ff8280b40b0 con 0x7ff8280cccf0 2024-08-03T21:01:07.320 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:07.316+0000 7ff82f600640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7ff8280b3f30 con 0x7ff8280ced30 2024-08-03T21:01:07.320 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:07.316+0000 7ff82f600640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7ff8280b4230 con 0x7ff828145c50 2024-08-03T21:01:07.320 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:07.316+0000 7ff82cc00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff8280cccf0 0x7ff8280c9250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:01:07.320 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:07.317+0000 7ff82cc00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff8280cccf0 0x7ff8280c9250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:50622/0 (socket says 172.21.15.32:50622) 2024-08-03T21:01:07.320 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:07.317+0000 7ff82cc00640 1 -- 172.21.15.32:0/790734766 learned_addr learned my addr 172.21.15.32:0/790734766 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T21:01:07.320 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:07.317+0000 7ff827e00640 1 --2- 172.21.15.32:0/790734766 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ff8280ced30 0x7ff828153c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:01:07.320 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:07.317+0000 7ff82d600640 1 --2- 172.21.15.32:0/790734766 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7ff828145c50 0x7ff82813e700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:01:07.320 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:07.317+0000 7ff82cc00640 1 -- 172.21.15.32:0/790734766 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7ff828145c50 msgr2=0x7ff82813e700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:01:07.320 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:07.317+0000 7ff82cc00640 1 --2- 172.21.15.32:0/790734766 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7ff828145c50 0x7ff82813e700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:01:07.320 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:07.317+0000 7ff82cc00640 1 -- 172.21.15.32:0/790734766 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ff8280ced30 msgr2=0x7ff828153c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:01:07.320 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:07.317+0000 7ff82cc00640 1 --2- 172.21.15.32:0/790734766 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ff8280ced30 0x7ff828153c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:01:07.320 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:07.317+0000 7ff82cc00640 1 -- 172.21.15.32:0/790734766 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff8281f9670 con 0x7ff8280cccf0 2024-08-03T21:01:07.320 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:07.317+0000 7ff827e00640 1 --2- 172.21.15.32:0/790734766 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ff8280ced30 0x7ff828153c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-03T21:01:07.321 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:07.317+0000 7ff82cc00640 1 --2- 172.21.15.32:0/790734766 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff8280cccf0 0x7ff8280c9250 secure :-1 s=READY pgs=2213 cs=0 l=1 rev1=1 crypto rx=0x7ff81c00d740 tx=0x7ff81c00dc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T21:01:07.321 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:07.318+0000 7ff824c00640 1 -- 172.21.15.32:0/790734766 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff81c017070 con 0x7ff8280cccf0 2024-08-03T21:01:07.321 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:07.318+0000 7ff824c00640 1 -- 172.21.15.32:0/790734766 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7ff81c009d20 con 0x7ff8280cccf0 2024-08-03T21:01:07.321 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:07.318+0000 7ff824c00640 1 -- 172.21.15.32:0/790734766 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff81c0106c0 con 0x7ff8280cccf0 2024-08-03T21:01:07.321 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:07.318+0000 7ff82f600640 1 -- 172.21.15.32:0/790734766 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff82813dc60 con 0x7ff8280cccf0 2024-08-03T21:01:07.321 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:07.318+0000 7ff82f600640 1 -- 172.21.15.32:0/790734766 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff82813df00 con 0x7ff8280cccf0 2024-08-03T21:01:07.321 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:07.319+0000 7ff82f600640 1 -- 172.21.15.32:0/790734766 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff8280b4750 con 0x7ff8280cccf0 2024-08-03T21:01:07.321 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:07.320+0000 7ff824c00640 1 -- 172.21.15.32:0/790734766 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7ff81c01a070 con 0x7ff8280cccf0 2024-08-03T21:01:07.322 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:07.321+0000 7ff824c00640 1 --2- 172.21.15.32:0/790734766 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7ff808077010 0x7ff808079520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:01:07.323 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:07.321+0000 7ff824c00640 1 -- 172.21.15.32:0/790734766 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(136..136 src has 1..136) ==== 5795+0+0 (secure 0 0 0) 0x7ff81c098bb0 con 0x7ff8280cccf0 2024-08-03T21:01:07.323 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:07.322+0000 7ff827e00640 1 --2- 172.21.15.32:0/790734766 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7ff808077010 0x7ff808079520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:01:07.324 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:07.322+0000 7ff827e00640 1 --2- 172.21.15.32:0/790734766 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7ff808077010 0x7ff808079520 secure :-1 s=READY pgs=884 cs=0 l=1 rev1=1 crypto rx=0x7ff814034000 tx=0x7ff814004730 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T21:01:07.327 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:07.326+0000 7ff824c00640 1 -- 172.21.15.32:0/790734766 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7ff81c061cc0 con 0x7ff8280cccf0 2024-08-03T21:01:07.570 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:07.568+0000 7ff82f600640 1 -- 172.21.15.32:0/790734766 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "joinable", "val": "true"} v 0) -- 0x7ff82813f9a0 con 0x7ff8280cccf0 2024-08-03T21:01:07.859 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:07.857+0000 7ff824c00640 1 -- 172.21.15.32:0/790734766 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "joinable", "val": "true"}]=0 cephfs marked joinable; MDS may join as newly active. v342) ==== 162+0+0 (secure 0 0 0) 0x7ff81c0615c0 con 0x7ff8280cccf0 2024-08-03T21:01:07.859 INFO:teuthology.orchestra.run.smithi032.stderr:cephfs marked joinable; MDS may join as newly active. 2024-08-03T21:01:07.861 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:07.859+0000 7ff82f600640 1 -- 172.21.15.32:0/790734766 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7ff808077010 msgr2=0x7ff808079520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:01:07.861 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:07.859+0000 7ff82f600640 1 --2- 172.21.15.32:0/790734766 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7ff808077010 0x7ff808079520 secure :-1 s=READY pgs=884 cs=0 l=1 rev1=1 crypto rx=0x7ff814034000 tx=0x7ff814004730 comp rx=0 tx=0).stop 2024-08-03T21:01:07.861 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:07.859+0000 7ff82f600640 1 -- 172.21.15.32:0/790734766 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff8280cccf0 msgr2=0x7ff8280c9250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:01:07.862 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:07.859+0000 7ff82f600640 1 --2- 172.21.15.32:0/790734766 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff8280cccf0 0x7ff8280c9250 secure :-1 s=READY pgs=2213 cs=0 l=1 rev1=1 crypto rx=0x7ff81c00d740 tx=0x7ff81c00dc00 comp rx=0 tx=0).stop 2024-08-03T21:01:07.862 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:07.860+0000 7ff82f600640 1 -- 172.21.15.32:0/790734766 shutdown_connections 2024-08-03T21:01:07.862 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:07.860+0000 7ff82f600640 1 --2- 172.21.15.32:0/790734766 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7ff808077010 0x7ff808079520 unknown :-1 s=CLOSED pgs=884 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:01:07.862 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:07.860+0000 7ff82f600640 1 --2- 172.21.15.32:0/790734766 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ff8280ced30 0x7ff828153c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:01:07.862 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:07.860+0000 7ff82f600640 1 --2- 172.21.15.32:0/790734766 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7ff828145c50 0x7ff82813e700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:01:07.862 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:07.860+0000 7ff82f600640 1 --2- 172.21.15.32:0/790734766 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff8280cccf0 0x7ff8280c9250 unknown :-1 s=CLOSED pgs=2213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:01:07.862 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:07.860+0000 7ff82f600640 1 -- 172.21.15.32:0/790734766 >> 172.21.15.32:0/790734766 conn(0x7ff8280b1bd0 msgr2=0x7ff8280b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T21:01:07.862 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:07.860+0000 7ff82f600640 1 -- 172.21.15.32:0/790734766 shutdown_connections 2024-08-03T21:01:07.862 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:07.860+0000 7ff82f600640 1 -- 172.21.15.32:0/790734766 wait complete. 2024-08-03T21:01:07.890 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-08-03T21:01:08.118 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:08.115+0000 7fe8a6a00640 1 -- 172.21.15.32:0/670279054 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe8a006fb50 msgr2=0x7fe8a00ad190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:01:08.118 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:08.115+0000 7fe8a6a00640 1 --2- 172.21.15.32:0/670279054 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe8a006fb50 0x7fe8a00ad190 secure :-1 s=READY pgs=2215 cs=0 l=1 rev1=1 crypto rx=0x7fe8a00720a0 tx=0x7fe89802aa30 comp rx=0 tx=0).stop 2024-08-03T21:01:08.118 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:08.116+0000 7fe8a6a00640 1 -- 172.21.15.32:0/670279054 shutdown_connections 2024-08-03T21:01:08.118 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:08.116+0000 7fe8a6a00640 1 --2- 172.21.15.32:0/670279054 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe8a006fb50 0x7fe8a00ad190 unknown :-1 s=CLOSED pgs=2215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:01:08.118 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:08.116+0000 7fe8a6a00640 1 --2- 172.21.15.32:0/670279054 >> v2:172.21.15.118:3300/0 conn(0x7fe8a00af020 0x7fe8a00742b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:01:08.118 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:08.116+0000 7fe8a6a00640 1 -- 172.21.15.32:0/670279054 >> 172.21.15.32:0/670279054 conn(0x7fe8a00b2120 msgr2=0x7fe8a00cfd60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T21:01:08.118 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:08.116+0000 7fe8a6a00640 1 -- 172.21.15.32:0/670279054 shutdown_connections 2024-08-03T21:01:08.119 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:08.116+0000 7fe8a6a00640 1 -- 172.21.15.32:0/670279054 wait complete. 2024-08-03T21:01:08.119 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:08.118+0000 7fe8a6a00640 1 Processor -- start 2024-08-03T21:01:08.120 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:08.118+0000 7fe8a6a00640 1 -- start start 2024-08-03T21:01:08.123 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:08.119+0000 7fe8a6a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe8a006fb50 0x7fe8a009ba50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:01:08.123 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:08.119+0000 7fe8a6a00640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fe8a00acd80 0x7fe8a00ce980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:01:08.123 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:08.119+0000 7fe8a6a00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fe8a00af020 0x7fe8a00ceec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:01:08.123 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:08.120+0000 7fe8a6a00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fe8a01601e0 con 0x7fe8a006fb50 2024-08-03T21:01:08.124 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:08.120+0000 7fe8a6a00640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7fe8a0160060 con 0x7fe8a00af020 2024-08-03T21:01:08.124 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:08.120+0000 7fe8a6a00640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7fe8a0160360 con 0x7fe8a00acd80 2024-08-03T21:01:08.124 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:08.120+0000 7fe8a5600640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe8a006fb50 0x7fe8a009ba50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:01:08.124 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:08.120+0000 7fe8a5600640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe8a006fb50 0x7fe8a009ba50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:50630/0 (socket says 172.21.15.32:50630) 2024-08-03T21:01:08.124 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:08.120+0000 7fe8a5600640 1 -- 172.21.15.32:0/936476538 learned_addr learned my addr 172.21.15.32:0/936476538 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T21:01:08.124 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:08.120+0000 7fe8a6000640 1 --2- 172.21.15.32:0/936476538 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fe8a00af020 0x7fe8a00ceec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:01:08.124 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:08.120+0000 7fe8a4c00640 1 --2- 172.21.15.32:0/936476538 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fe8a00acd80 0x7fe8a00ce980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:01:08.124 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:08.120+0000 7fe8a5600640 1 -- 172.21.15.32:0/936476538 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fe8a00acd80 msgr2=0x7fe8a00ce980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:01:08.125 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:08.120+0000 7fe8a5600640 1 --2- 172.21.15.32:0/936476538 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fe8a00acd80 0x7fe8a00ce980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:01:08.125 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:08.120+0000 7fe8a5600640 1 -- 172.21.15.32:0/936476538 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fe8a00af020 msgr2=0x7fe8a00ceec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:01:08.125 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:08.120+0000 7fe8a5600640 1 --2- 172.21.15.32:0/936476538 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fe8a00af020 0x7fe8a00ceec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:01:08.125 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:08.120+0000 7fe8a5600640 1 -- 172.21.15.32:0/936476538 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe8a00c5bd0 con 0x7fe8a006fb50 2024-08-03T21:01:08.125 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:08.120+0000 7fe8a6000640 1 --2- 172.21.15.32:0/936476538 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fe8a00af020 0x7fe8a00ceec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-03T21:01:08.125 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:08.121+0000 7fe8a5600640 1 --2- 172.21.15.32:0/936476538 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe8a006fb50 0x7fe8a009ba50 secure :-1 s=READY pgs=2216 cs=0 l=1 rev1=1 crypto rx=0x7fe89400d770 tx=0x7fe89400dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T21:01:08.125 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:08.121+0000 7fe89d600640 1 -- 172.21.15.32:0/936476538 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe894017070 con 0x7fe8a006fb50 2024-08-03T21:01:08.125 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:08.121+0000 7fe89d600640 1 -- 172.21.15.32:0/936476538 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fe894009d20 con 0x7fe8a006fb50 2024-08-03T21:01:08.125 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:08.121+0000 7fe89d600640 1 -- 172.21.15.32:0/936476538 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe8940106c0 con 0x7fe8a006fb50 2024-08-03T21:01:08.126 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:08.121+0000 7fe8a6a00640 1 -- 172.21.15.32:0/936476538 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe8a00c5e90 con 0x7fe8a006fb50 2024-08-03T21:01:08.126 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:08.121+0000 7fe8a6a00640 1 -- 172.21.15.32:0/936476538 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe8a00c6450 con 0x7fe8a006fb50 2024-08-03T21:01:08.127 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:08.122+0000 7fe8a6a00640 1 -- 172.21.15.32:0/936476538 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe8a00ad190 con 0x7fe8a006fb50 2024-08-03T21:01:08.127 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:08.123+0000 7fe89d600640 1 -- 172.21.15.32:0/936476538 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7fe89401a070 con 0x7fe8a006fb50 2024-08-03T21:01:08.127 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:08.124+0000 7fe89d600640 1 --2- 172.21.15.32:0/936476538 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fe880077010 0x7fe880079520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:01:08.127 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:08.124+0000 7fe89d600640 1 -- 172.21.15.32:0/936476538 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(136..136 src has 1..136) ==== 5795+0+0 (secure 0 0 0) 0x7fe8940989a0 con 0x7fe8a006fb50 2024-08-03T21:01:08.127 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:08.125+0000 7fe8a4c00640 1 --2- 172.21.15.32:0/936476538 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fe880077010 0x7fe880079520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:01:08.128 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:08.125+0000 7fe8a4c00640 1 --2- 172.21.15.32:0/936476538 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fe880077010 0x7fe880079520 secure :-1 s=READY pgs=886 cs=0 l=1 rev1=1 crypto rx=0x7fe8a0160c70 tx=0x7fe88c009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T21:01:08.130 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:08.129+0000 7fe89d600640 1 -- 172.21.15.32:0/936476538 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7fe894061bd0 con 0x7fe8a006fb50 2024-08-03T21:01:08.371 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:08.370+0000 7fe8a6a00640 1 -- 172.21.15.32:0/936476538 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe8a00d0930 con 0x7fe8a006fb50 2024-08-03T21:01:08.372 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:08.371+0000 7fe89d600640 1 -- 172.21.15.32:0/936476538 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 343 v343) ==== 95+0+5229 (secure 0 0 0) 0x7fe8940614d0 con 0x7fe8a006fb50 2024-08-03T21:01:08.372 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-03T21:01:08.372 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":343,"btime":"2024-08-03T21:01:07:861875+0000","default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8519,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6836/3249267506","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6834","nonce":3249267506},{"type":"v1","addr":"172.21.15.118:6836","nonce":3249267506}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":334},{"gid":8522,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6837/408542078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6835","nonce":408542078},{"type":"v1","addr":"172.21.15.118:6837","nonce":408542078}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":335},{"gid":8648,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6835/1693401067","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6833","nonce":1693401067},{"type":"v1","addr":"172.21.15.32:6835","nonce":1693401067}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":342}],"filesystems":[{"mdsmap":{"epoch":343,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-08-03T21:00:49.742850+0000","modified":"2024-08-03T21:01:07.861859+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":136,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":9589},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9589":{"gid":9589,"name":"c","rank":0,"incarnation":343,"state":"up:replay","state_seq":1,"addr":"172.21.15.32:6834/3440267920","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":3440267920},{"type":"v1","addr":"172.21.15.32:6834","nonce":3440267920}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":7}]} 2024-08-03T21:01:08.373 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 343 2024-08-03T21:01:08.375 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:08.373+0000 7fe8a6a00640 1 -- 172.21.15.32:0/936476538 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fe880077010 msgr2=0x7fe880079520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:01:08.375 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:08.373+0000 7fe8a6a00640 1 --2- 172.21.15.32:0/936476538 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fe880077010 0x7fe880079520 secure :-1 s=READY pgs=886 cs=0 l=1 rev1=1 crypto rx=0x7fe8a0160c70 tx=0x7fe88c009040 comp rx=0 tx=0).stop 2024-08-03T21:01:08.375 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:08.373+0000 7fe8a6a00640 1 -- 172.21.15.32:0/936476538 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe8a006fb50 msgr2=0x7fe8a009ba50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:01:08.375 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:08.373+0000 7fe8a6a00640 1 --2- 172.21.15.32:0/936476538 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe8a006fb50 0x7fe8a009ba50 secure :-1 s=READY pgs=2216 cs=0 l=1 rev1=1 crypto rx=0x7fe89400d770 tx=0x7fe89400dc30 comp rx=0 tx=0).stop 2024-08-03T21:01:08.375 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:08.373+0000 7fe8a6a00640 1 -- 172.21.15.32:0/936476538 shutdown_connections 2024-08-03T21:01:08.375 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:08.373+0000 7fe8a6a00640 1 --2- 172.21.15.32:0/936476538 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fe880077010 0x7fe880079520 unknown :-1 s=CLOSED pgs=886 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:01:08.376 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:08.373+0000 7fe8a6a00640 1 --2- 172.21.15.32:0/936476538 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fe8a00af020 0x7fe8a00ceec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:01:08.376 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:08.373+0000 7fe8a6a00640 1 --2- 172.21.15.32:0/936476538 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fe8a00acd80 0x7fe8a00ce980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:01:08.376 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:08.373+0000 7fe8a6a00640 1 --2- 172.21.15.32:0/936476538 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe8a006fb50 0x7fe8a009ba50 unknown :-1 s=CLOSED pgs=2216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:01:08.376 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:08.373+0000 7fe8a6a00640 1 -- 172.21.15.32:0/936476538 >> 172.21.15.32:0/936476538 conn(0x7fe8a00b2120 msgr2=0x7fe8a0068940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T21:01:08.376 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:08.373+0000 7fe8a6a00640 1 -- 172.21.15.32:0/936476538 shutdown_connections 2024-08-03T21:01:08.376 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:08.373+0000 7fe8a6a00640 1 -- 172.21.15.32:0/936476538 wait complete. 2024-08-03T21:01:08.799 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 343, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-08-03T21:00:49.742850+0000', 'modified': '2024-08-03T21:01:07.861859+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 136, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 9589}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9589': {'gid': 9589, 'name': 'c', 'rank': 0, 'incarnation': 343, 'state': 'up:replay', 'state_seq': 1, 'addr': '172.21.15.32:6834/3440267920', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6832', 'nonce': 3440267920}, {'type': 'v1', 'addr': '172.21.15.32:6834', 'nonce': 3440267920}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [15], 'metadata_pool': 14, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 0, 'qdb_cluster': []} 2024-08-03T21:01:08.800 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9589:up:replay 2024-08-03T21:01:09.287 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-08-03T21:01:09.292 DEBUG:teuthology.orchestra.run.smithi118:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-08-03T21:01:09.801 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-08-03T21:01:10.041 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:10.039+0000 7f495ce00640 1 -- 172.21.15.32:0/3182272434 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f495815f610 msgr2=0x7f49580cdf70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:01:10.042 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:10.039+0000 7f495ce00640 1 --2- 172.21.15.32:0/3182272434 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f495815f610 0x7f49580cdf70 secure :-1 s=READY pgs=2217 cs=0 l=1 rev1=1 crypto rx=0x7f49580d1030 tx=0x7f495002aa30 comp rx=0 tx=0).stop 2024-08-03T21:01:10.042 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:10.040+0000 7f495ce00640 1 -- 172.21.15.32:0/3182272434 shutdown_connections 2024-08-03T21:01:10.042 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:10.040+0000 7f495ce00640 1 --2- 172.21.15.32:0/3182272434 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f495815f610 0x7f49580cdf70 unknown :-1 s=CLOSED pgs=2217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:01:10.042 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:10.040+0000 7f495ce00640 1 --2- 172.21.15.32:0/3182272434 >> v2:172.21.15.118:3300/0 conn(0x7f49580b38b0 0x7f49581523f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:01:10.042 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:10.040+0000 7f495ce00640 1 -- 172.21.15.32:0/3182272434 >> 172.21.15.32:0/3182272434 conn(0x7f49580b1bd0 msgr2=0x7f49580b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T21:01:10.042 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:10.040+0000 7f495ce00640 1 -- 172.21.15.32:0/3182272434 shutdown_connections 2024-08-03T21:01:10.042 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:10.040+0000 7f495ce00640 1 -- 172.21.15.32:0/3182272434 wait complete. 2024-08-03T21:01:10.043 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:10.042+0000 7f495ce00640 1 Processor -- start 2024-08-03T21:01:10.044 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:10.042+0000 7f495ce00640 1 -- start start 2024-08-03T21:01:10.047 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:10.044+0000 7f495ce00640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f49580b38b0 0x7f495814ea80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:01:10.047 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:10.044+0000 7f495ce00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f49580b4ce0 0x7f4958156d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:01:10.047 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:10.044+0000 7f495ce00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f495814d8e0 0x7f495814d110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:01:10.047 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:10.044+0000 7f495ce00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f49580cea30 con 0x7f49580b4ce0 2024-08-03T21:01:10.047 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:10.044+0000 7f495ce00640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f49580ce8b0 con 0x7f495814d8e0 2024-08-03T21:01:10.047 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:10.044+0000 7f495ce00640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7f49580cebb0 con 0x7f49580b38b0 2024-08-03T21:01:10.047 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:10.044+0000 7f494fe00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f49580b4ce0 0x7f4958156d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:01:10.047 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:10.044+0000 7f494fe00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f49580b4ce0 0x7f4958156d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:58738/0 (socket says 172.21.15.32:58738) 2024-08-03T21:01:10.047 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:10.044+0000 7f494fe00640 1 -- 172.21.15.32:0/1011170124 learned_addr learned my addr 172.21.15.32:0/1011170124 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T21:01:10.047 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:10.044+0000 7f4955e00640 1 --2- 172.21.15.32:0/1011170124 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f49580b38b0 0x7f495814ea80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:01:10.047 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:10.044+0000 7f4956800640 1 --2- 172.21.15.32:0/1011170124 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f495814d8e0 0x7f495814d110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:01:10.048 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:10.044+0000 7f494fe00640 1 -- 172.21.15.32:0/1011170124 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f49580b38b0 msgr2=0x7f495814ea80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:01:10.048 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:10.044+0000 7f494fe00640 1 --2- 172.21.15.32:0/1011170124 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f49580b38b0 0x7f495814ea80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:01:10.048 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:10.044+0000 7f494fe00640 1 -- 172.21.15.32:0/1011170124 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f495814d8e0 msgr2=0x7f495814d110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:01:10.048 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:10.044+0000 7f494fe00640 1 --2- 172.21.15.32:0/1011170124 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f495814d8e0 0x7f495814d110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:01:10.048 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:10.045+0000 7f494fe00640 1 -- 172.21.15.32:0/1011170124 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f495814f2d0 con 0x7f49580b4ce0 2024-08-03T21:01:10.048 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:10.045+0000 7f494fe00640 1 --2- 172.21.15.32:0/1011170124 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f49580b4ce0 0x7f4958156d00 secure :-1 s=READY pgs=2218 cs=0 l=1 rev1=1 crypto rx=0x7f494000c3d0 tx=0x7f494000c890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T21:01:10.048 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:10.045+0000 7f494e000640 1 -- 172.21.15.32:0/1011170124 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4940017070 con 0x7f49580b4ce0 2024-08-03T21:01:10.048 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:10.045+0000 7f494e000640 1 -- 172.21.15.32:0/1011170124 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f49400053c0 con 0x7f49580b4ce0 2024-08-03T21:01:10.048 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:10.045+0000 7f494e000640 1 -- 172.21.15.32:0/1011170124 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4940003850 con 0x7f49580b4ce0 2024-08-03T21:01:10.048 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:10.045+0000 7f495ce00640 1 -- 172.21.15.32:0/1011170124 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4958156220 con 0x7f49580b4ce0 2024-08-03T21:01:10.048 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:10.045+0000 7f495ce00640 1 -- 172.21.15.32:0/1011170124 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f495814d650 con 0x7f49580b4ce0 2024-08-03T21:01:10.049 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:10.046+0000 7f495ce00640 1 -- 172.21.15.32:0/1011170124 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f495815b0d0 con 0x7f49580b4ce0 2024-08-03T21:01:10.049 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:10.048+0000 7f494e000640 1 -- 172.21.15.32:0/1011170124 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f494001a070 con 0x7f49580b4ce0 2024-08-03T21:01:10.050 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:10.048+0000 7f494e000640 1 --2- 172.21.15.32:0/1011170124 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f4934077040 0x7f4934079550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:01:10.050 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:10.049+0000 7f494e000640 1 -- 172.21.15.32:0/1011170124 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(136..136 src has 1..136) ==== 5795+0+0 (secure 0 0 0) 0x7f4940097d50 con 0x7f49580b4ce0 2024-08-03T21:01:10.050 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:10.049+0000 7f4955e00640 1 --2- 172.21.15.32:0/1011170124 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f4934077040 0x7f4934079550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:01:10.051 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:10.050+0000 7f4955e00640 1 --2- 172.21.15.32:0/1011170124 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f4934077040 0x7f4934079550 secure :-1 s=READY pgs=887 cs=0 l=1 rev1=1 crypto rx=0x7f4948004630 tx=0x7f494800b2e0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T21:01:10.055 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:10.053+0000 7f494e000640 1 -- 172.21.15.32:0/1011170124 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7f495815b0d0 con 0x7f49580b4ce0 2024-08-03T21:01:10.303 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:10.302+0000 7f495ce00640 1 -- 172.21.15.32:0/1011170124 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f495814f060 con 0x7f49580b4ce0 2024-08-03T21:01:10.304 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:10.302+0000 7f494e000640 1 -- 172.21.15.32:0/1011170124 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 345 v345) ==== 95+0+5229 (secure 0 0 0) 0x7f4940069120 con 0x7f49580b4ce0 2024-08-03T21:01:10.304 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-03T21:01:10.304 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":345,"btime":"2024-08-03T21:01:09:869947+0000","default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8519,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6836/3249267506","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6834","nonce":3249267506},{"type":"v1","addr":"172.21.15.118:6836","nonce":3249267506}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":334},{"gid":8522,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6837/408542078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6835","nonce":408542078},{"type":"v1","addr":"172.21.15.118:6837","nonce":408542078}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":335},{"gid":8648,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6835/1693401067","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6833","nonce":1693401067},{"type":"v1","addr":"172.21.15.32:6835","nonce":1693401067}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":342}],"filesystems":[{"mdsmap":{"epoch":345,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-08-03T21:00:49.742850+0000","modified":"2024-08-03T21:01:08.873134+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":136,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":9589},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9589":{"gid":9589,"name":"c","rank":0,"incarnation":343,"state":"up:rejoin","state_seq":8,"addr":"172.21.15.32:6834/3440267920","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":3440267920},{"type":"v1","addr":"172.21.15.32:6834","nonce":3440267920}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":7}]} 2024-08-03T21:01:10.305 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 345 2024-08-03T21:01:10.307 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:10.305+0000 7f495ce00640 1 -- 172.21.15.32:0/1011170124 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f4934077040 msgr2=0x7f4934079550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:01:10.307 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:10.305+0000 7f495ce00640 1 --2- 172.21.15.32:0/1011170124 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f4934077040 0x7f4934079550 secure :-1 s=READY pgs=887 cs=0 l=1 rev1=1 crypto rx=0x7f4948004630 tx=0x7f494800b2e0 comp rx=0 tx=0).stop 2024-08-03T21:01:10.307 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:10.305+0000 7f495ce00640 1 -- 172.21.15.32:0/1011170124 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f49580b4ce0 msgr2=0x7f4958156d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:01:10.307 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:10.305+0000 7f495ce00640 1 --2- 172.21.15.32:0/1011170124 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f49580b4ce0 0x7f4958156d00 secure :-1 s=READY pgs=2218 cs=0 l=1 rev1=1 crypto rx=0x7f494000c3d0 tx=0x7f494000c890 comp rx=0 tx=0).stop 2024-08-03T21:01:10.307 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:10.305+0000 7f495ce00640 1 -- 172.21.15.32:0/1011170124 shutdown_connections 2024-08-03T21:01:10.307 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:10.305+0000 7f495ce00640 1 --2- 172.21.15.32:0/1011170124 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f4934077040 0x7f4934079550 unknown :-1 s=CLOSED pgs=887 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:01:10.307 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:10.305+0000 7f495ce00640 1 --2- 172.21.15.32:0/1011170124 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f495814d8e0 0x7f495814d110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:01:10.307 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:10.305+0000 7f495ce00640 1 --2- 172.21.15.32:0/1011170124 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f49580b4ce0 0x7f4958156d00 unknown :-1 s=CLOSED pgs=2218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:01:10.307 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:10.305+0000 7f495ce00640 1 --2- 172.21.15.32:0/1011170124 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f49580b38b0 0x7f495814ea80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:01:10.307 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:10.305+0000 7f495ce00640 1 -- 172.21.15.32:0/1011170124 >> 172.21.15.32:0/1011170124 conn(0x7f49580b1bd0 msgr2=0x7f4958150bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T21:01:10.307 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:10.305+0000 7f495ce00640 1 -- 172.21.15.32:0/1011170124 shutdown_connections 2024-08-03T21:01:10.307 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:10.305+0000 7f495ce00640 1 -- 172.21.15.32:0/1011170124 wait complete. 2024-08-03T21:01:10.731 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 345, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-08-03T21:00:49.742850+0000', 'modified': '2024-08-03T21:01:08.873134+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 136, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 9589}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9589': {'gid': 9589, 'name': 'c', 'rank': 0, 'incarnation': 343, 'state': 'up:rejoin', 'state_seq': 8, 'addr': '172.21.15.32:6834/3440267920', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6832', 'nonce': 3440267920}, {'type': 'v1', 'addr': '172.21.15.32:6834', 'nonce': 3440267920}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [15], 'metadata_pool': 14, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 0, 'qdb_cluster': []} 2024-08-03T21:01:10.731 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9589:up:rejoin 2024-08-03T21:01:11.732 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-08-03T21:01:11.973 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:11.970+0000 7f8cbf800640 1 -- 172.21.15.32:0/2328664022 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8cb80b38b0 msgr2=0x7f8cb81523f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:01:11.973 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:11.970+0000 7f8cbf800640 1 --2- 172.21.15.32:0/2328664022 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8cb80b38b0 0x7f8cb81523f0 secure :-1 s=READY pgs=2219 cs=0 l=1 rev1=1 crypto rx=0x7f8cb80cd6d0 tx=0x7f8cac02a560 comp rx=0 tx=0).stop 2024-08-03T21:01:11.973 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:11.971+0000 7f8cbf800640 1 -- 172.21.15.32:0/2328664022 shutdown_connections 2024-08-03T21:01:11.973 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:11.971+0000 7f8cbf800640 1 --2- 172.21.15.32:0/2328664022 >> v2:172.21.15.118:3300/0 conn(0x7f8cb80b4ce0 0x7f8cb815ef00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:01:11.973 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:11.971+0000 7f8cbf800640 1 --2- 172.21.15.32:0/2328664022 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8cb80b38b0 0x7f8cb81523f0 unknown :-1 s=CLOSED pgs=2219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:01:11.973 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:11.971+0000 7f8cbf800640 1 -- 172.21.15.32:0/2328664022 >> 172.21.15.32:0/2328664022 conn(0x7f8cb80b1bd0 msgr2=0x7f8cb80b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T21:01:11.973 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:11.971+0000 7f8cbf800640 1 -- 172.21.15.32:0/2328664022 shutdown_connections 2024-08-03T21:01:11.973 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:11.971+0000 7f8cbf800640 1 -- 172.21.15.32:0/2328664022 wait complete. 2024-08-03T21:01:11.974 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:11.973+0000 7f8cbf800640 1 Processor -- start 2024-08-03T21:01:11.974 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:11.973+0000 7f8cbf800640 1 -- start start 2024-08-03T21:01:11.977 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:11.975+0000 7f8cbf800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8cb80b38b0 0x7f8cb813e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:01:11.977 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:11.975+0000 7f8cbf800640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8cb80b4ce0 0x7f8cb813b6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:01:11.977 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:11.975+0000 7f8cbf800640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f8cb815f440 0x7f8cb8073a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:01:11.977 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:11.975+0000 7f8cbf800640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f8cb815c020 con 0x7f8cb80b38b0 2024-08-03T21:01:11.978 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:11.975+0000 7f8cbf800640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f8cb80cf190 con 0x7f8cb80b4ce0 2024-08-03T21:01:11.978 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:11.975+0000 7f8cbf800640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7f8cb815c1a0 con 0x7f8cb815f440 2024-08-03T21:01:11.978 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:11.975+0000 7f8cbce00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8cb80b38b0 0x7f8cb813e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:01:11.978 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:11.975+0000 7f8cbce00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8cb80b38b0 0x7f8cb813e2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:58756/0 (socket says 172.21.15.32:58756) 2024-08-03T21:01:11.978 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:11.975+0000 7f8cbce00640 1 -- 172.21.15.32:0/725966915 learned_addr learned my addr 172.21.15.32:0/725966915 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T21:01:11.978 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:11.975+0000 7f8cb7e00640 1 --2- 172.21.15.32:0/725966915 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8cb80b4ce0 0x7f8cb813b6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:01:11.978 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:11.975+0000 7f8cbd800640 1 --2- 172.21.15.32:0/725966915 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f8cb815f440 0x7f8cb8073a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:01:11.978 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:11.975+0000 7f8cbce00640 1 -- 172.21.15.32:0/725966915 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f8cb815f440 msgr2=0x7f8cb8073a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:01:11.978 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:11.975+0000 7f8cbce00640 1 --2- 172.21.15.32:0/725966915 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f8cb815f440 0x7f8cb8073a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:01:11.978 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:11.976+0000 7f8cbce00640 1 -- 172.21.15.32:0/725966915 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8cb80b4ce0 msgr2=0x7f8cb813b6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:01:11.978 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:11.976+0000 7f8cbce00640 1 --2- 172.21.15.32:0/725966915 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8cb80b4ce0 0x7f8cb813b6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:01:11.978 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:11.976+0000 7f8cbce00640 1 -- 172.21.15.32:0/725966915 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8cb8073280 con 0x7f8cb80b38b0 2024-08-03T21:01:11.979 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:11.976+0000 7f8cb7e00640 1 --2- 172.21.15.32:0/725966915 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8cb80b4ce0 0x7f8cb813b6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-03T21:01:11.979 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:11.976+0000 7f8cbce00640 1 --2- 172.21.15.32:0/725966915 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8cb80b38b0 0x7f8cb813e2f0 secure :-1 s=READY pgs=2220 cs=0 l=1 rev1=1 crypto rx=0x7f8cac034040 tx=0x7f8cac003910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T21:01:11.980 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:11.976+0000 7f8cb4c00640 1 -- 172.21.15.32:0/725966915 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8cac03e030 con 0x7f8cb80b38b0 2024-08-03T21:01:11.980 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:11.976+0000 7f8cb4c00640 1 -- 172.21.15.32:0/725966915 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f8cac003e00 con 0x7f8cb80b38b0 2024-08-03T21:01:11.980 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:11.976+0000 7f8cbf800640 1 -- 172.21.15.32:0/725966915 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8cb8155aa0 con 0x7f8cb80b38b0 2024-08-03T21:01:11.980 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:11.976+0000 7f8cb4c00640 1 -- 172.21.15.32:0/725966915 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8cac035040 con 0x7f8cb80b38b0 2024-08-03T21:01:11.980 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:11.976+0000 7f8cbf800640 1 -- 172.21.15.32:0/725966915 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8cb8073520 con 0x7f8cb80b38b0 2024-08-03T21:01:11.980 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:11.978+0000 7f8cbf800640 1 -- 172.21.15.32:0/725966915 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8cb815c730 con 0x7f8cb80b38b0 2024-08-03T21:01:11.980 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:11.979+0000 7f8cb4c00640 1 -- 172.21.15.32:0/725966915 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f8cac041030 con 0x7f8cb80b38b0 2024-08-03T21:01:11.981 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:11.980+0000 7f8cb4c00640 1 --2- 172.21.15.32:0/725966915 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f8c98077010 0x7f8c98079520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:01:11.981 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:11.980+0000 7f8cb4c00640 1 -- 172.21.15.32:0/725966915 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(136..136 src has 1..136) ==== 5795+0+0 (secure 0 0 0) 0x7f8cac0b9170 con 0x7f8cb80b38b0 2024-08-03T21:01:11.982 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:11.980+0000 7f8cb7e00640 1 --2- 172.21.15.32:0/725966915 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f8c98077010 0x7f8c98079520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:01:11.982 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:11.981+0000 7f8cb7e00640 1 --2- 172.21.15.32:0/725966915 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f8c98077010 0x7f8c98079520 secure :-1 s=READY pgs=888 cs=0 l=1 rev1=1 crypto rx=0x7f8cb80b1300 tx=0x7f8ca4005f10 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T21:01:11.986 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:11.984+0000 7f8cb4c00640 1 -- 172.21.15.32:0/725966915 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7f8cac082270 con 0x7f8cb80b38b0 2024-08-03T21:01:12.193 INFO:tasks.ceph.mds.c.smithi032.stderr:2024-08-03T21:01:12.192+0000 7fe797400640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-08-03T21:01:12.231 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:12.229+0000 7f8cbf800640 1 -- 172.21.15.32:0/725966915 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f8cb80b3440 con 0x7f8cb80b38b0 2024-08-03T21:01:12.231 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:12.230+0000 7f8cb4c00640 1 -- 172.21.15.32:0/725966915 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 346 v346) ==== 95+0+5236 (secure 0 0 0) 0x7f8cac081b70 con 0x7f8cb80b38b0 2024-08-03T21:01:12.232 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-03T21:01:12.232 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":346,"btime":"2024-08-03T21:01:10:877273+0000","default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8519,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6836/3249267506","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6834","nonce":3249267506},{"type":"v1","addr":"172.21.15.118:6836","nonce":3249267506}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":334},{"gid":8522,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6837/408542078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6835","nonce":408542078},{"type":"v1","addr":"172.21.15.118:6837","nonce":408542078}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":335},{"gid":8648,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6835/1693401067","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6833","nonce":1693401067},{"type":"v1","addr":"172.21.15.32:6835","nonce":1693401067}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":342}],"filesystems":[{"mdsmap":{"epoch":346,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-08-03T21:00:49.742850+0000","modified":"2024-08-03T21:01:10.877262+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":136,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":9589},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9589":{"gid":9589,"name":"c","rank":0,"incarnation":343,"state":"up:active","state_seq":9,"addr":"172.21.15.32:6834/3440267920","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":3440267920},{"type":"v1","addr":"172.21.15.32:6834","nonce":3440267920}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9589,"qdb_cluster":[9589]},"id":7}]} 2024-08-03T21:01:12.232 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 346 2024-08-03T21:01:12.234 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:12.232+0000 7f8cbf800640 1 -- 172.21.15.32:0/725966915 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f8c98077010 msgr2=0x7f8c98079520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:01:12.234 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:12.232+0000 7f8cbf800640 1 --2- 172.21.15.32:0/725966915 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f8c98077010 0x7f8c98079520 secure :-1 s=READY pgs=888 cs=0 l=1 rev1=1 crypto rx=0x7f8cb80b1300 tx=0x7f8ca4005f10 comp rx=0 tx=0).stop 2024-08-03T21:01:12.234 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:12.232+0000 7f8cbf800640 1 -- 172.21.15.32:0/725966915 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8cb80b38b0 msgr2=0x7f8cb813e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:01:12.234 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:12.232+0000 7f8cbf800640 1 --2- 172.21.15.32:0/725966915 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8cb80b38b0 0x7f8cb813e2f0 secure :-1 s=READY pgs=2220 cs=0 l=1 rev1=1 crypto rx=0x7f8cac034040 tx=0x7f8cac003910 comp rx=0 tx=0).stop 2024-08-03T21:01:12.234 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:12.232+0000 7f8cbf800640 1 -- 172.21.15.32:0/725966915 shutdown_connections 2024-08-03T21:01:12.235 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:12.232+0000 7f8cbf800640 1 --2- 172.21.15.32:0/725966915 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f8c98077010 0x7f8c98079520 unknown :-1 s=CLOSED pgs=888 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:01:12.235 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:12.232+0000 7f8cbf800640 1 --2- 172.21.15.32:0/725966915 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8cb80b4ce0 0x7f8cb813b6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:01:12.235 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:12.232+0000 7f8cbf800640 1 --2- 172.21.15.32:0/725966915 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f8cb815f440 0x7f8cb8073a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:01:12.235 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:12.232+0000 7f8cbf800640 1 --2- 172.21.15.32:0/725966915 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8cb80b38b0 0x7f8cb813e2f0 unknown :-1 s=CLOSED pgs=2220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:01:12.235 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:12.232+0000 7f8cbf800640 1 -- 172.21.15.32:0/725966915 >> 172.21.15.32:0/725966915 conn(0x7f8cb80b1bd0 msgr2=0x7f8cb806fbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T21:01:12.235 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:12.232+0000 7f8cbf800640 1 -- 172.21.15.32:0/725966915 shutdown_connections 2024-08-03T21:01:12.235 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:12.232+0000 7f8cbf800640 1 -- 172.21.15.32:0/725966915 wait complete. 2024-08-03T21:01:12.663 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 346, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-08-03T21:00:49.742850+0000', 'modified': '2024-08-03T21:01:10.877262+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 136, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 9589}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9589': {'gid': 9589, 'name': 'c', 'rank': 0, 'incarnation': 343, 'state': 'up:active', 'state_seq': 9, 'addr': '172.21.15.32:6834/3440267920', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6832', 'nonce': 3440267920}, {'type': 'v1', 'addr': '172.21.15.32:6834', 'nonce': 3440267920}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [15], 'metadata_pool': 14, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9589, 'qdb_cluster': [9589]} 2024-08-03T21:01:12.663 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-08-03T21:01:12.663 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-08-03T21:01:12.895 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:12.893+0000 7fed97e00640 1 -- 172.21.15.32:0/1963286731 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fed980cffe0 msgr2=0x7fed980b3e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:01:12.896 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:12.893+0000 7fed97e00640 1 --2- 172.21.15.32:0/1963286731 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fed980cffe0 0x7fed980b3e90 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fed80009860 tx=0x7fed8002a5a0 comp rx=0 tx=0).stop 2024-08-03T21:01:12.896 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:12.894+0000 7fed97e00640 1 -- 172.21.15.32:0/1963286731 shutdown_connections 2024-08-03T21:01:12.896 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:12.894+0000 7fed97e00640 1 --2- 172.21.15.32:0/1963286731 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fed980cffe0 0x7fed980b3e90 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:01:12.896 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:12.894+0000 7fed97e00640 1 -- 172.21.15.32:0/1963286731 >> 172.21.15.32:0/1963286731 conn(0x7fed9808d7a0 msgr2=0x7fed9809d3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T21:01:12.896 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:12.894+0000 7fed97e00640 1 -- 172.21.15.32:0/1963286731 shutdown_connections 2024-08-03T21:01:12.896 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:12.894+0000 7fed97e00640 1 -- 172.21.15.32:0/1963286731 wait complete. 2024-08-03T21:01:12.897 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:12.897+0000 7fed97e00640 1 Processor -- start 2024-08-03T21:01:12.897 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:12.897+0000 7fed97e00640 1 -- start start 2024-08-03T21:01:12.900 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:12.898+0000 7fed97e00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fed980b43d0 0x7fed980c7c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:01:12.900 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:12.898+0000 7fed97e00640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fed980cffe0 0x7fed98148e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:01:12.900 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:12.898+0000 7fed97e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fed980d26e0 0x7fed981f9590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:01:12.900 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:12.898+0000 7fed97e00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fed980d4310 con 0x7fed980d26e0 2024-08-03T21:01:12.900 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:12.898+0000 7fed97e00640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7fed980d4190 con 0x7fed980b43d0 2024-08-03T21:01:12.900 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:12.898+0000 7fed97e00640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7fed980d4490 con 0x7fed980cffe0 2024-08-03T21:01:12.900 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:12.898+0000 7fed97400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fed980d26e0 0x7fed981f9590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:01:12.901 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:12.898+0000 7fed97400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fed980d26e0 0x7fed981f9590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:58764/0 (socket says 172.21.15.32:58764) 2024-08-03T21:01:12.901 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:12.898+0000 7fed97400640 1 -- 172.21.15.32:0/3589889497 learned_addr learned my addr 172.21.15.32:0/3589889497 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T21:01:12.901 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:12.898+0000 7fed8fe00640 1 --2- 172.21.15.32:0/3589889497 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fed980cffe0 0x7fed98148e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:01:12.901 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:12.898+0000 7fed96a00640 1 --2- 172.21.15.32:0/3589889497 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fed980b43d0 0x7fed980c7c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:01:12.901 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:12.898+0000 7fed97400640 1 -- 172.21.15.32:0/3589889497 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fed980cffe0 msgr2=0x7fed98148e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:01:12.901 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:12.898+0000 7fed97400640 1 --2- 172.21.15.32:0/3589889497 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fed980cffe0 0x7fed98148e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:01:12.901 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:12.898+0000 7fed97400640 1 -- 172.21.15.32:0/3589889497 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fed980b43d0 msgr2=0x7fed980c7c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:01:12.901 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:12.898+0000 7fed97400640 1 --2- 172.21.15.32:0/3589889497 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fed980b43d0 0x7fed980c7c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:01:12.901 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:12.898+0000 7fed97400640 1 -- 172.21.15.32:0/3589889497 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fed98149350 con 0x7fed980d26e0 2024-08-03T21:01:12.901 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:12.899+0000 7fed97400640 1 --2- 172.21.15.32:0/3589889497 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fed980d26e0 0x7fed981f9590 secure :-1 s=READY pgs=2222 cs=0 l=1 rev1=1 crypto rx=0x7fed9000c6e0 tx=0x7fed9000cba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T21:01:12.902 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:12.899+0000 7fed8ea00640 1 -- 172.21.15.32:0/3589889497 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed90003b70 con 0x7fed980d26e0 2024-08-03T21:01:12.902 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:12.899+0000 7fed8ea00640 1 -- 172.21.15.32:0/3589889497 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fed90003d10 con 0x7fed980d26e0 2024-08-03T21:01:12.902 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:12.899+0000 7fed97e00640 1 -- 172.21.15.32:0/3589889497 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fed980b7080 con 0x7fed980d26e0 2024-08-03T21:01:12.902 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:12.899+0000 7fed8ea00640 1 -- 172.21.15.32:0/3589889497 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed90005330 con 0x7fed980d26e0 2024-08-03T21:01:12.902 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:12.899+0000 7fed97e00640 1 -- 172.21.15.32:0/3589889497 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fed980b7640 con 0x7fed980d26e0 2024-08-03T21:01:12.903 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:12.902+0000 7fed8ea00640 1 -- 172.21.15.32:0/3589889497 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7fed900054d0 con 0x7fed980d26e0 2024-08-03T21:01:12.904 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:12.902+0000 7fed8ea00640 1 --2- 172.21.15.32:0/3589889497 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fed74077040 0x7fed74079550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:01:12.904 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:12.902+0000 7fed8ea00640 1 -- 172.21.15.32:0/3589889497 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(136..136 src has 1..136) ==== 5795+0+0 (secure 0 0 0) 0x7fed90098680 con 0x7fed980d26e0 2024-08-03T21:01:12.904 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:12.903+0000 7feda4bd1740 1 Processor -- start 2024-08-03T21:01:12.905 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:12.903+0000 7fed96a00640 1 --2- 172.21.15.32:0/3589889497 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fed74077040 0x7fed74079550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:01:12.905 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:12.903+0000 7feda4bd1740 1 -- start start 2024-08-03T21:01:12.905 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:12.904+0000 7fed96a00640 1 --2- 172.21.15.32:0/3589889497 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fed74077040 0x7fed74079550 secure :-1 s=READY pgs=889 cs=0 l=1 rev1=1 crypto rx=0x7fed880046b0 tx=0x7fed88007e90 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T21:01:12.907 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:12.904+0000 7feda4bd1740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x556ff613d270 0x556ff6528080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:01:12.907 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:12.904+0000 7feda4bd1740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x556ff61143b0 0x556ff64fced0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:01:12.907 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:12.904+0000 7feda4bd1740 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x556ff611c590 0x556ff64fd410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:01:12.907 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:12.904+0000 7feda4bd1740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x556ff6455a60 con 0x556ff61143b0 2024-08-03T21:01:12.907 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:12.904+0000 7feda4bd1740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x556ff6444f60 con 0x556ff613d270 2024-08-03T21:01:12.907 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:12.904+0000 7feda4bd1740 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x556ff6448fa0 con 0x556ff611c590 2024-08-03T21:01:12.907 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:12.905+0000 7fed8fe00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x556ff61143b0 0x556ff64fced0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:01:12.907 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:12.905+0000 7fed8fe00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x556ff61143b0 0x556ff64fced0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:58774/0 (socket says 172.21.15.32:58774) 2024-08-03T21:01:12.907 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:12.905+0000 7fed8fe00640 1 -- 172.21.15.32:0/2782404533 learned_addr learned my addr 172.21.15.32:0/2782404533 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T21:01:12.907 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:12.905+0000 7fed96a00640 1 --2- 172.21.15.32:0/2782404533 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x556ff613d270 0x556ff6528080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:01:12.908 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:12.905+0000 7fed97400640 1 --2- 172.21.15.32:0/2782404533 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x556ff611c590 0x556ff64fd410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:01:12.908 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:12.905+0000 7fed8fe00640 1 -- 172.21.15.32:0/2782404533 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x556ff611c590 msgr2=0x556ff64fd410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T21:01:12.908 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:12.905+0000 7fed8fe00640 1 --2- 172.21.15.32:0/2782404533 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x556ff611c590 0x556ff64fd410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:01:12.908 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:12.905+0000 7fed8fe00640 1 -- 172.21.15.32:0/2782404533 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x556ff613d270 msgr2=0x556ff6528080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T21:01:12.908 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:12.905+0000 7fed8fe00640 1 --2- 172.21.15.32:0/2782404533 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x556ff613d270 0x556ff6528080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:01:12.908 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:12.905+0000 7fed8fe00640 1 -- 172.21.15.32:0/2782404533 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556ff61a8850 con 0x556ff61143b0 2024-08-03T21:01:12.908 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:12.905+0000 7fed96a00640 1 --2- 172.21.15.32:0/2782404533 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x556ff613d270 0x556ff6528080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-03T21:01:12.908 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:12.905+0000 7fed97400640 1 --2- 172.21.15.32:0/2782404533 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x556ff611c590 0x556ff64fd410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-03T21:01:12.908 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:12.905+0000 7fed8fe00640 1 --2- 172.21.15.32:0/2782404533 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x556ff61143b0 0x556ff64fced0 secure :-1 s=READY pgs=2223 cs=0 l=1 rev1=1 crypto rx=0x7fed8002ab90 tx=0x7fed800046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9e173609935f2a99 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T21:01:12.909 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:12.906+0000 7fed87400640 1 -- 172.21.15.32:0/2782404533 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed80005220 con 0x556ff61143b0 2024-08-03T21:01:12.910 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:12.906+0000 7fed87400640 1 -- 172.21.15.32:0/2782404533 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fed80040030 con 0x556ff61143b0 2024-08-03T21:01:12.910 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:12.906+0000 7fed87400640 1 -- 172.21.15.32:0/2782404533 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed800047a0 con 0x556ff61143b0 2024-08-03T21:01:12.910 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:12.907+0000 7feda4bd1740 1 -- 172.21.15.32:0/2782404533 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x556ff61143b0 msgr2=0x556ff64fced0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:01:12.910 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:12.907+0000 7feda4bd1740 1 --2- 172.21.15.32:0/2782404533 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x556ff61143b0 0x556ff64fced0 secure :-1 s=READY pgs=2223 cs=0 l=1 rev1=1 crypto rx=0x7fed8002ab90 tx=0x7fed800046c0 comp rx=0 tx=0).stop 2024-08-03T21:01:12.910 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:12.907+0000 7feda4bd1740 1 -- 172.21.15.32:0/2782404533 shutdown_connections 2024-08-03T21:01:12.910 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:12.907+0000 7feda4bd1740 1 --2- 172.21.15.32:0/2782404533 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x556ff611c590 0x556ff64fd410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:01:12.910 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:12.907+0000 7feda4bd1740 1 --2- 172.21.15.32:0/2782404533 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x556ff61143b0 0x556ff64fced0 unknown :-1 s=CLOSED pgs=2223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:01:12.910 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:12.907+0000 7feda4bd1740 1 --2- 172.21.15.32:0/2782404533 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x556ff613d270 0x556ff6528080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:01:12.910 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:12.907+0000 7feda4bd1740 1 -- 172.21.15.32:0/2782404533 >> 172.21.15.32:0/2782404533 conn(0x556ff61313c0 msgr2=0x556ff64e3370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T21:01:12.910 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:12.907+0000 7feda4bd1740 1 -- 172.21.15.32:0/2782404533 shutdown_connections 2024-08-03T21:01:12.910 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:12.907+0000 7feda4bd1740 1 -- 172.21.15.32:0/2782404533 wait complete. 2024-08-03T21:01:12.910 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:12.909+0000 7feda4bd1740 1 -- start start 2024-08-03T21:01:12.911 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:12.909+0000 7feda4bd1740 1 Processor -- start 2024-08-03T21:01:12.913 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:12.909+0000 7feda4bd1740 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x556ff61143b0 0x556ff65379b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:01:12.913 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:12.909+0000 7feda4bd1740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x556ff611c590 0x556ff65225c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:01:12.914 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:12.909+0000 7feda4bd1740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x556ff613d270 0x556ff64fba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:01:12.914 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:12.909+0000 7feda4bd1740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x556ff61b7620 con 0x556ff611c590 2024-08-03T21:01:12.914 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:12.909+0000 7feda4bd1740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x556ff63e2160 con 0x556ff613d270 2024-08-03T21:01:12.914 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:12.909+0000 7feda4bd1740 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x556ff6522d10 con 0x556ff61143b0 2024-08-03T21:01:12.914 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:12.910+0000 7feda4bd1740 10 client.0 _ll_register_callbacks cb 0x556ff63f16b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-03T21:01:12.914 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:12.910+0000 7fed8fe00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x556ff611c590 0x556ff65225c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:01:12.914 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:12.910+0000 7fed8fe00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x556ff611c590 0x556ff65225c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:58784/0 (socket says 172.21.15.32:58784) 2024-08-03T21:01:12.914 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:12.910+0000 7fed8fe00640 1 -- 172.21.15.32:0/789350563 learned_addr learned my addr 172.21.15.32:0/789350563 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T21:01:12.914 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:12.910+0000 7fed97400640 1 --2- 172.21.15.32:0/789350563 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x556ff613d270 0x556ff64fba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:01:12.914 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:12.910+0000 7fed96a00640 1 --2- 172.21.15.32:0/789350563 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x556ff61143b0 0x556ff65379b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:01:12.914 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:12.910+0000 7fed8fe00640 1 -- 172.21.15.32:0/789350563 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x556ff61143b0 msgr2=0x556ff65379b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T21:01:12.914 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:12.910+0000 7fed8fe00640 1 --2- 172.21.15.32:0/789350563 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x556ff61143b0 0x556ff65379b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:01:12.915 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:12.910+0000 7fed8fe00640 1 -- 172.21.15.32:0/789350563 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x556ff613d270 msgr2=0x556ff64fba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T21:01:12.915 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:12.910+0000 7fed8fe00640 1 --2- 172.21.15.32:0/789350563 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x556ff613d270 0x556ff64fba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:01:12.915 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:12.910+0000 7fed8fe00640 1 -- 172.21.15.32:0/789350563 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556ff621f6b0 con 0x556ff611c590 2024-08-03T21:01:12.915 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:12.910+0000 7fed8fe00640 1 --2- 172.21.15.32:0/789350563 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x556ff611c590 0x556ff65225c0 secure :-1 s=READY pgs=2224 cs=0 l=1 rev1=1 crypto rx=0x7fed800054d0 tx=0x7fed80005500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a71844a7382805dc server_cookie=0 in_seq=0 out_seq=0 2024-08-03T21:01:12.915 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:12.911+0000 7fed7be00640 10 client.0 ms_handle_connect on v2:172.21.15.32:3300/0 2024-08-03T21:01:12.915 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:12.911+0000 7fed7be00640 1 -- 172.21.15.32:0/789350563 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed80045680 con 0x556ff611c590 2024-08-03T21:01:12.915 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:12.911+0000 7fed7be00640 1 -- 172.21.15.32:0/789350563 <== mon.0 v2:172.21.15.32:3300/0 2 ==== osd_map(136..136 src has 1..136) ==== 5795+0+0 (secure 0 0 0) 0x7fed80045820 con 0x556ff611c590 2024-08-03T21:01:12.915 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:12.911+0000 7feda4bd1740 1 -- 172.21.15.32:0/789350563 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fed80040030 con 0x556ff611c590 2024-08-03T21:01:12.915 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:12.911+0000 7fed7be00640 1 -- 172.21.15.32:0/789350563 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_get_version_reply(handle=1 version=346) ==== 24+0+0 (secure 0 0 0) 0x7fed80056050 con 0x556ff611c590 2024-08-03T21:01:12.915 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:12.911+0000 7feda4bd1740 10 client.10743 fetch_fsmap learned FSMap version 346 2024-08-03T21:01:12.915 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:12.911+0000 7feda4bd1740 1 -- 172.21.15.32:0/789350563 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,fsmap=346,monmap=2+}) -- 0x556ff64fc010 con 0x556ff611c590 2024-08-03T21:01:12.915 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:12.911+0000 7fed7be00640 1 -- 172.21.15.32:0/789350563 <== mon.0 v2:172.21.15.32:3300/0 4 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fed80040030 con 0x556ff611c590 2024-08-03T21:01:12.915 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:12.911+0000 7fed7be00640 1 -- 172.21.15.32:0/789350563 <== mon.0 v2:172.21.15.32:3300/0 5 ==== fsmap(e 346) ==== 3502+0+0 (secure 0 0 0) 0x7fed80059020 con 0x556ff611c590 2024-08-03T21:01:12.916 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:12.911+0000 7feda4bd1740 10 client.10743 fetch_fsmap finished waiting for FSMap version 346 2024-08-03T21:01:12.916 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:12.911+0000 7feda4bd1740 10 client.10743 resolve_mds: resolved name 'c' to daemon mds.c 2024-08-03T21:01:12.916 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:12.911+0000 7feda4bd1740 20 client.10743 populate_metadata read hostname 'smithi032' 2024-08-03T21:01:12.916 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:12.912+0000 7feda4bd1740 1 --2- 172.21.15.32:0/789350563 >> [v2:172.21.15.32:6832/3440267920,v1:172.21.15.32:6834/3440267920] conn(0x556ff60fbe40 0x556ff653a6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:01:12.916 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:12.912+0000 7feda4bd1740 4 client.10743 mds_command: new command op to 9589 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-08-03T21:01:12.916 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:12.912+0000 7feda4bd1740 1 -- 172.21.15.32:0/789350563 --> [v2:172.21.15.32:6832/3440267920,v1:172.21.15.32:6834/3440267920] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x556ff64579b0 con 0x556ff60fbe40 2024-08-03T21:01:12.916 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:12.912+0000 7fed96a00640 1 --2- 172.21.15.32:0/789350563 >> [v2:172.21.15.32:6832/3440267920,v1:172.21.15.32:6834/3440267920] conn(0x556ff60fbe40 0x556ff653a6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:01:12.917 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:12.912+0000 7fed96a00640 1 --2- 172.21.15.32:0/789350563 >> [v2:172.21.15.32:6832/3440267920,v1:172.21.15.32:6834/3440267920] conn(0x556ff60fbe40 0x556ff653a6a0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f4b6d7d54d50715e server_cookie=95db5dc092650067 in_seq=0 out_seq=0 2024-08-03T21:01:12.918 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:12.912+0000 7fed7be00640 10 client.10743 ms_handle_connect on v2:172.21.15.32:6832/3440267920 2024-08-03T21:01:12.918 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:12.913+0000 7fed7be00640 1 -- 172.21.15.32:0/789350563 <== mds.0 v2:172.21.15.32:6832/3440267920 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x556ff64579b0 con 0x556ff60fbe40 2024-08-03T21:01:12.918 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:12.913+0000 7fed7be00640 10 client.10743 handle_command_reply: tid=0 2024-08-03T21:01:12.918 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:12.913+0000 7feda4bd1740 1 client.10743 shutdown 2024-08-03T21:01:12.918 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:12.914+0000 7feda4bd1740 1 -- 172.21.15.32:0/789350563 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x556ff611c590 msgr2=0x556ff65225c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:01:12.918 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:12.914+0000 7feda4bd1740 1 --2- 172.21.15.32:0/789350563 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x556ff611c590 0x556ff65225c0 secure :-1 s=READY pgs=2224 cs=0 l=1 rev1=1 crypto rx=0x7fed800054d0 tx=0x7fed80005500 comp rx=0 tx=0).stop 2024-08-03T21:01:12.918 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:12.914+0000 7feda4bd1740 1 -- 172.21.15.32:0/789350563 shutdown_connections 2024-08-03T21:01:12.918 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:12.914+0000 7feda4bd1740 1 --2- 172.21.15.32:0/789350563 >> [v2:172.21.15.32:6832/3440267920,v1:172.21.15.32:6834/3440267920] conn(0x556ff60fbe40 0x556ff653a6a0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:01:12.918 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:12.914+0000 7feda4bd1740 1 --2- 172.21.15.32:0/789350563 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x556ff613d270 0x556ff64fba10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:01:12.918 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:12.914+0000 7feda4bd1740 1 --2- 172.21.15.32:0/789350563 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x556ff611c590 0x556ff65225c0 unknown :-1 s=CLOSED pgs=2224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:01:12.918 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:12.914+0000 7fed7be00640 0 client.10743 ms_handle_reset on v2:172.21.15.32:6832/3440267920 2024-08-03T21:01:12.919 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:12.914+0000 7feda4bd1740 1 --2- 172.21.15.32:0/789350563 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x556ff61143b0 0x556ff65379b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:01:12.919 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:12.914+0000 7feda4bd1740 1 -- 172.21.15.32:0/789350563 >> 172.21.15.32:0/789350563 conn(0x556ff61313c0 msgr2=0x556ff64fd4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T21:01:12.919 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:12.914+0000 7feda4bd1740 1 -- 172.21.15.32:0/789350563 shutdown_connections 2024-08-03T21:01:12.919 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:12.914+0000 7feda4bd1740 1 -- 172.21.15.32:0/789350563 wait complete. 2024-08-03T21:01:12.919 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:12.914+0000 7feda4bd1740 20 client.10743 trim_cache size 0 max 16384 2024-08-03T21:01:12.940 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:12.934+0000 7feda4bd1740 1 Processor -- start 2024-08-03T21:01:12.940 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:12.934+0000 7feda4bd1740 1 -- start start 2024-08-03T21:01:12.940 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:12.935+0000 7feda4bd1740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x556ff61143b0 0x556ff65374e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:01:12.940 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:12.935+0000 7feda4bd1740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x556ff611c590 0x556ff650a380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:01:12.940 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:12.936+0000 7feda4bd1740 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x556ff613d270 0x556ff650a8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:01:12.940 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:12.936+0000 7feda4bd1740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fed981f67b0 con 0x556ff611c590 2024-08-03T21:01:12.940 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:12.936+0000 7feda4bd1740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7fed980d14d0 con 0x556ff61143b0 2024-08-03T21:01:12.941 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:12.936+0000 7feda4bd1740 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7fed981f6160 con 0x556ff613d270 2024-08-03T21:01:12.941 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:12.936+0000 7fed8fe00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x556ff611c590 0x556ff650a380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:01:12.941 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:12.936+0000 7fed8fe00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x556ff611c590 0x556ff650a380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:58792/0 (socket says 172.21.15.32:58792) 2024-08-03T21:01:12.941 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:12.936+0000 7fed8fe00640 1 -- 172.21.15.32:0/4132601753 learned_addr learned my addr 172.21.15.32:0/4132601753 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T21:01:12.941 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:12.936+0000 7fed96a00640 1 --2- 172.21.15.32:0/4132601753 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x556ff61143b0 0x556ff65374e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:01:12.941 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:12.936+0000 7fed97400640 1 --2- 172.21.15.32:0/4132601753 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x556ff613d270 0x556ff650a8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:01:12.941 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:12.936+0000 7fed8fe00640 1 -- 172.21.15.32:0/4132601753 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x556ff613d270 msgr2=0x556ff650a8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T21:01:12.941 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:12.936+0000 7fed8fe00640 1 --2- 172.21.15.32:0/4132601753 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x556ff613d270 0x556ff650a8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:01:12.941 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:12.936+0000 7fed8fe00640 1 -- 172.21.15.32:0/4132601753 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x556ff61143b0 msgr2=0x556ff65374e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T21:01:12.941 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:12.936+0000 7fed8fe00640 1 --2- 172.21.15.32:0/4132601753 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x556ff61143b0 0x556ff65374e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:01:12.941 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:12.936+0000 7fed8fe00640 1 -- 172.21.15.32:0/4132601753 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556ff6522930 con 0x556ff611c590 2024-08-03T21:01:12.941 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:12.936+0000 7fed97400640 1 --2- 172.21.15.32:0/4132601753 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x556ff613d270 0x556ff650a8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-03T21:01:12.942 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:12.937+0000 7fed8fe00640 1 --2- 172.21.15.32:0/4132601753 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x556ff611c590 0x556ff650a380 secure :-1 s=READY pgs=2225 cs=0 l=1 rev1=1 crypto rx=0x7fed80005de0 tx=0x7fed80045ae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=74e471402ffe7e4b server_cookie=0 in_seq=0 out_seq=0 2024-08-03T21:01:12.942 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:12.937+0000 7fed86a00640 1 -- 172.21.15.32:0/4132601753 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed80045680 con 0x556ff611c590 2024-08-03T21:01:12.942 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:12.937+0000 7fed86a00640 1 -- 172.21.15.32:0/4132601753 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fed80040070 con 0x556ff611c590 2024-08-03T21:01:12.942 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:12.937+0000 7fed86a00640 1 -- 172.21.15.32:0/4132601753 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed8003c660 con 0x556ff611c590 2024-08-03T21:01:12.942 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:12.937+0000 7feda4bd1740 1 -- 172.21.15.32:0/4132601753 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x556ff611c590 msgr2=0x556ff650a380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:01:12.942 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:12.937+0000 7feda4bd1740 1 --2- 172.21.15.32:0/4132601753 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x556ff611c590 0x556ff650a380 secure :-1 s=READY pgs=2225 cs=0 l=1 rev1=1 crypto rx=0x7fed80005de0 tx=0x7fed80045ae0 comp rx=0 tx=0).stop 2024-08-03T21:01:12.942 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:12.938+0000 7feda4bd1740 1 -- 172.21.15.32:0/4132601753 shutdown_connections 2024-08-03T21:01:12.942 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:12.938+0000 7feda4bd1740 1 --2- 172.21.15.32:0/4132601753 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x556ff613d270 0x556ff650a8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:01:12.942 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:12.938+0000 7feda4bd1740 1 --2- 172.21.15.32:0/4132601753 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x556ff611c590 0x556ff650a380 unknown :-1 s=CLOSED pgs=2225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:01:12.942 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:12.938+0000 7feda4bd1740 1 --2- 172.21.15.32:0/4132601753 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x556ff61143b0 0x556ff65374e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:01:12.942 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:12.938+0000 7feda4bd1740 1 -- 172.21.15.32:0/4132601753 >> 172.21.15.32:0/4132601753 conn(0x556ff61313c0 msgr2=0x556ff65361b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T21:01:12.943 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:12.938+0000 7feda4bd1740 1 -- 172.21.15.32:0/4132601753 shutdown_connections 2024-08-03T21:01:12.943 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:12.938+0000 7feda4bd1740 1 -- 172.21.15.32:0/4132601753 wait complete. 2024-08-03T21:01:12.947 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:12.939+0000 7feda4bd1740 1 -- start start 2024-08-03T21:01:12.947 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:12.939+0000 7feda4bd1740 1 Processor -- start 2024-08-03T21:01:12.947 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:12.940+0000 7feda4bd1740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x556ff61143b0 0x556ff6526250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:01:12.948 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:12.940+0000 7feda4bd1740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x556ff611c590 0x556ff65a2330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:01:12.948 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:12.940+0000 7feda4bd1740 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x556ff613d270 0x556ff65a8380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:01:12.948 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:12.940+0000 7feda4bd1740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x556ff6520340 con 0x556ff611c590 2024-08-03T21:01:12.948 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:12.940+0000 7feda4bd1740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x556ff651fec0 con 0x556ff61143b0 2024-08-03T21:01:12.948 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:12.940+0000 7feda4bd1740 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x556ff6520fe0 con 0x556ff613d270 2024-08-03T21:01:12.948 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:12.941+0000 7feda4bd1740 10 client.0 _ll_register_callbacks cb 0x7fed80004540 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-03T21:01:12.948 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:12.941+0000 7fed8fe00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x556ff611c590 0x556ff65a2330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:01:12.948 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:12.941+0000 7fed8fe00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x556ff611c590 0x556ff65a2330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:58798/0 (socket says 172.21.15.32:58798) 2024-08-03T21:01:12.948 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:12.941+0000 7fed8fe00640 1 -- 172.21.15.32:0/1744847991 learned_addr learned my addr 172.21.15.32:0/1744847991 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T21:01:12.948 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:12.941+0000 7fed96a00640 1 --2- 172.21.15.32:0/1744847991 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x556ff61143b0 0x556ff6526250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:01:12.948 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:12.941+0000 7fed97400640 1 --2- 172.21.15.32:0/1744847991 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x556ff613d270 0x556ff65a8380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:01:12.948 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:12.941+0000 7fed8fe00640 1 -- 172.21.15.32:0/1744847991 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x556ff613d270 msgr2=0x556ff65a8380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T21:01:12.949 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:12.941+0000 7fed8fe00640 1 --2- 172.21.15.32:0/1744847991 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x556ff613d270 0x556ff65a8380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:01:12.949 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:12.941+0000 7fed8fe00640 1 -- 172.21.15.32:0/1744847991 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x556ff61143b0 msgr2=0x556ff6526250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T21:01:12.949 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:12.941+0000 7fed8fe00640 1 --2- 172.21.15.32:0/1744847991 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x556ff61143b0 0x556ff6526250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:01:12.949 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:12.941+0000 7fed8fe00640 1 -- 172.21.15.32:0/1744847991 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556ff650ad10 con 0x556ff611c590 2024-08-03T21:01:12.949 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:12.942+0000 7fed8fe00640 1 --2- 172.21.15.32:0/1744847991 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x556ff611c590 0x556ff65a2330 secure :-1 s=READY pgs=2226 cs=0 l=1 rev1=1 crypto rx=0x7fed80009340 tx=0x7fed80031a90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a4ed616125757f67 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T21:01:12.949 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:12.942+0000 7fed7be00640 10 client.0 ms_handle_connect on v2:172.21.15.32:3300/0 2024-08-03T21:01:12.949 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:12.942+0000 7fed7be00640 1 -- 172.21.15.32:0/1744847991 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed80068060 con 0x556ff611c590 2024-08-03T21:01:12.949 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:12.942+0000 7fed7be00640 1 -- 172.21.15.32:0/1744847991 <== mon.0 v2:172.21.15.32:3300/0 2 ==== osd_map(136..136 src has 1..136) ==== 5795+0+0 (secure 0 0 0) 0x7fed80048740 con 0x556ff611c590 2024-08-03T21:01:12.949 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:12.942+0000 7feda4bd1740 1 -- 172.21.15.32:0/1744847991 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fed80040070 con 0x556ff611c590 2024-08-03T21:01:12.949 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:12.942+0000 7fed7be00640 1 -- 172.21.15.32:0/1744847991 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_get_version_reply(handle=1 version=346) ==== 24+0+0 (secure 0 0 0) 0x7fed80045680 con 0x556ff611c590 2024-08-03T21:01:12.949 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:12.942+0000 7feda4bd1740 10 client.10749 fetch_fsmap learned FSMap version 346 2024-08-03T21:01:12.950 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:12.942+0000 7feda4bd1740 1 -- 172.21.15.32:0/1744847991 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,fsmap=346,monmap=2+}) -- 0x556ff65a8e50 con 0x556ff611c590 2024-08-03T21:01:12.950 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:12.942+0000 7fed7be00640 1 -- 172.21.15.32:0/1744847991 <== mon.0 v2:172.21.15.32:3300/0 4 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fed80040070 con 0x556ff611c590 2024-08-03T21:01:12.950 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:12.942+0000 7fed7be00640 1 -- 172.21.15.32:0/1744847991 <== mon.0 v2:172.21.15.32:3300/0 5 ==== fsmap(e 346) ==== 3502+0+0 (secure 0 0 0) 0x7fed80059070 con 0x556ff611c590 2024-08-03T21:01:12.950 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:12.943+0000 7feda4bd1740 10 client.10749 fetch_fsmap finished waiting for FSMap version 346 2024-08-03T21:01:12.950 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:12.943+0000 7feda4bd1740 10 client.10749 resolve_mds: resolved name 'c' to daemon mds.c 2024-08-03T21:01:12.950 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:12.943+0000 7feda4bd1740 20 client.10749 populate_metadata read hostname 'smithi032' 2024-08-03T21:01:12.950 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:12.943+0000 7feda4bd1740 1 --2- 172.21.15.32:0/1744847991 >> [v2:172.21.15.32:6832/3440267920,v1:172.21.15.32:6834/3440267920] conn(0x556ff64e3190 0x556ff64ddb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:01:12.950 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:12.943+0000 7feda4bd1740 4 client.10749 mds_command: new command op to 9589 tid=0 multi_id=0 [{"prefix": "status"}] 2024-08-03T21:01:12.951 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:12.943+0000 7feda4bd1740 1 -- 172.21.15.32:0/1744847991 --> [v2:172.21.15.32:6832/3440267920,v1:172.21.15.32:6834/3440267920] -- command(tid 0: {"prefix": "status"}) -- 0x556ff6453450 con 0x556ff64e3190 2024-08-03T21:01:12.951 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:12.943+0000 7fed96a00640 1 --2- 172.21.15.32:0/1744847991 >> [v2:172.21.15.32:6832/3440267920,v1:172.21.15.32:6834/3440267920] conn(0x556ff64e3190 0x556ff64ddb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:01:12.951 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:12.943+0000 7fed96a00640 1 --2- 172.21.15.32:0/1744847991 >> [v2:172.21.15.32:6832/3440267920,v1:172.21.15.32:6834/3440267920] conn(0x556ff64e3190 0x556ff64ddb70 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a08998acd0e8d7a server_cookie=7bc77826da26777c in_seq=0 out_seq=0 2024-08-03T21:01:12.951 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:12.943+0000 7fed7be00640 10 client.10749 ms_handle_connect on v2:172.21.15.32:6832/3440267920 2024-08-03T21:01:12.951 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:12.944+0000 7fed7be00640 1 -- 172.21.15.32:0/1744847991 <== mds.0 v2:172.21.15.32:6832/3440267920 1 ==== command_reply(tid 0: 0 ) ==== 8+0+327 (crc 0 0 0) 0x556ff6453450 con 0x556ff64e3190 2024-08-03T21:01:12.951 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:12.944+0000 7fed7be00640 10 client.10749 handle_command_reply: tid=0 2024-08-03T21:01:12.952 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:12.944+0000 7feda4bd1740 1 client.10749 shutdown 2024-08-03T21:01:12.952 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:12.944+0000 7feda4bd1740 1 -- 172.21.15.32:0/1744847991 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x556ff611c590 msgr2=0x556ff65a2330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:01:12.952 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:12.944+0000 7feda4bd1740 1 --2- 172.21.15.32:0/1744847991 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x556ff611c590 0x556ff65a2330 secure :-1 s=READY pgs=2226 cs=0 l=1 rev1=1 crypto rx=0x7fed80009340 tx=0x7fed80031a90 comp rx=0 tx=0).stop 2024-08-03T21:01:12.952 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:12.944+0000 7feda4bd1740 1 -- 172.21.15.32:0/1744847991 shutdown_connections 2024-08-03T21:01:12.952 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:12.944+0000 7feda4bd1740 1 --2- 172.21.15.32:0/1744847991 >> [v2:172.21.15.32:6832/3440267920,v1:172.21.15.32:6834/3440267920] conn(0x556ff64e3190 0x556ff64ddb70 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:01:12.952 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:12.944+0000 7feda4bd1740 1 --2- 172.21.15.32:0/1744847991 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x556ff613d270 0x556ff65a8380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:01:12.952 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:12.944+0000 7feda4bd1740 1 --2- 172.21.15.32:0/1744847991 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x556ff611c590 0x556ff65a2330 unknown :-1 s=CLOSED pgs=2226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:01:12.952 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:12.944+0000 7fed7be00640 0 client.10749 ms_handle_reset on v2:172.21.15.32:6832/3440267920 2024-08-03T21:01:12.953 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:12.944+0000 7feda4bd1740 1 --2- 172.21.15.32:0/1744847991 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x556ff61143b0 0x556ff6526250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:01:12.953 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:12.944+0000 7feda4bd1740 1 -- 172.21.15.32:0/1744847991 >> 172.21.15.32:0/1744847991 conn(0x556ff61313c0 msgr2=0x556ff6521950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T21:01:12.953 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:12.944+0000 7feda4bd1740 1 -- 172.21.15.32:0/1744847991 shutdown_connections 2024-08-03T21:01:12.953 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:12.944+0000 7feda4bd1740 1 -- 172.21.15.32:0/1744847991 wait complete. 2024-08-03T21:01:12.953 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:12.944+0000 7feda4bd1740 20 client.10749 trim_cache size 0 max 16384 2024-08-03T21:01:12.953 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-08-03T21:01:12.954 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_fsid": "40ab141a-01d7-49b9-ae3c-2edec9876e95", 2024-08-03T21:01:12.954 INFO:teuthology.orchestra.run.smithi032.stdout: "whoami": 0, 2024-08-03T21:01:12.954 INFO:teuthology.orchestra.run.smithi032.stdout: "id": 9589, 2024-08-03T21:01:12.954 INFO:teuthology.orchestra.run.smithi032.stdout: "want_state": "up:active", 2024-08-03T21:01:12.955 INFO:teuthology.orchestra.run.smithi032.stdout: "state": "up:active", 2024-08-03T21:01:12.955 INFO:teuthology.orchestra.run.smithi032.stdout: "fs_name": "cephfs", 2024-08-03T21:01:12.955 INFO:teuthology.orchestra.run.smithi032.stdout: "rank_uptime": 5.080416638, 2024-08-03T21:01:12.955 INFO:teuthology.orchestra.run.smithi032.stdout: "mdsmap_epoch": 346, 2024-08-03T21:01:12.955 INFO:teuthology.orchestra.run.smithi032.stdout: "osdmap_epoch": 136, 2024-08-03T21:01:12.955 INFO:teuthology.orchestra.run.smithi032.stdout: "osdmap_epoch_barrier": 136, 2024-08-03T21:01:12.955 INFO:teuthology.orchestra.run.smithi032.stdout: "uptime": 27.663180440000001 2024-08-03T21:01:12.955 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-08-03T21:01:12.956 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:12.945+0000 7fed97e00640 1 -- 172.21.15.32:0/3589889497 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fed74077040 msgr2=0x7fed74079550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:01:12.956 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:12.945+0000 7fed97e00640 1 --2- 172.21.15.32:0/3589889497 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fed74077040 0x7fed74079550 secure :-1 s=READY pgs=889 cs=0 l=1 rev1=1 crypto rx=0x7fed880046b0 tx=0x7fed88007e90 comp rx=0 tx=0).stop 2024-08-03T21:01:12.956 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:12.945+0000 7fed97e00640 1 -- 172.21.15.32:0/3589889497 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fed980d26e0 msgr2=0x7fed981f9590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:01:12.956 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:12.945+0000 7fed97e00640 1 --2- 172.21.15.32:0/3589889497 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fed980d26e0 0x7fed981f9590 secure :-1 s=READY pgs=2222 cs=0 l=1 rev1=1 crypto rx=0x7fed9000c6e0 tx=0x7fed9000cba0 comp rx=0 tx=0).stop 2024-08-03T21:01:12.957 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:12.945+0000 7fed97e00640 1 -- 172.21.15.32:0/3589889497 shutdown_connections 2024-08-03T21:01:12.957 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:12.945+0000 7fed97e00640 1 --2- 172.21.15.32:0/3589889497 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fed74077040 0x7fed74079550 unknown :-1 s=CLOSED pgs=889 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:01:12.957 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:12.945+0000 7fed97e00640 1 --2- 172.21.15.32:0/3589889497 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fed980d26e0 0x7fed981f9590 unknown :-1 s=CLOSED pgs=2222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:01:12.957 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:12.945+0000 7fed97e00640 1 --2- 172.21.15.32:0/3589889497 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fed980cffe0 0x7fed98148e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:01:12.957 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:12.945+0000 7fed97e00640 1 --2- 172.21.15.32:0/3589889497 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fed980b43d0 0x7fed980c7c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:01:12.957 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:12.945+0000 7fed97e00640 1 -- 172.21.15.32:0/3589889497 >> 172.21.15.32:0/3589889497 conn(0x7fed9808d7a0 msgr2=0x7fed980b27a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T21:01:12.957 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:12.945+0000 7fed97e00640 1 -- 172.21.15.32:0/3589889497 shutdown_connections 2024-08-03T21:01:12.957 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:12.945+0000 7fed97e00640 1 -- 172.21.15.32:0/3589889497 wait complete. 2024-08-03T21:01:12.972 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-08-03T21:01:13.200 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:13.198+0000 7f4aaae00640 1 -- 172.21.15.32:0/2752726337 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f4aa406e930 msgr2=0x7f4aa4144f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:01:13.200 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:13.198+0000 7f4aaae00640 1 --2- 172.21.15.32:0/2752726337 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f4aa406e930 0x7f4aa4144f90 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f4a94009860 tx=0x7f4a9402a5a0 comp rx=0 tx=0).stop 2024-08-03T21:01:13.200 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:13.198+0000 7f4aaae00640 1 -- 172.21.15.32:0/2752726337 shutdown_connections 2024-08-03T21:01:13.200 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:13.198+0000 7f4aaae00640 1 --2- 172.21.15.32:0/2752726337 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f4aa406e930 0x7f4aa4144f90 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:01:13.200 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:13.198+0000 7f4aaae00640 1 -- 172.21.15.32:0/2752726337 >> 172.21.15.32:0/2752726337 conn(0x7f4aa40b1bd0 msgr2=0x7f4aa40b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T21:01:13.200 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:13.198+0000 7f4aaae00640 1 -- 172.21.15.32:0/2752726337 shutdown_connections 2024-08-03T21:01:13.200 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:13.198+0000 7f4aaae00640 1 -- 172.21.15.32:0/2752726337 wait complete. 2024-08-03T21:01:13.201 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:13.201+0000 7f4aaae00640 1 Processor -- start 2024-08-03T21:01:13.202 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:13.201+0000 7f4aaae00640 1 -- start start 2024-08-03T21:01:13.204 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:13.202+0000 7f4aaae00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f4aa406e930 0x7f4aa415f870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:01:13.204 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:13.202+0000 7f4aaae00640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f4aa4073e60 0x7f4aa41539d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:01:13.204 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:13.202+0000 7f4aaae00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4aa415fff0 0x7f4aa41f5470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:01:13.204 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:13.202+0000 7f4aaae00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f4aa40cf840 con 0x7f4aa415fff0 2024-08-03T21:01:13.204 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:13.202+0000 7f4aaae00640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f4aa4145f30 con 0x7f4aa406e930 2024-08-03T21:01:13.204 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:13.202+0000 7f4aaae00640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7f4aa40cf9c0 con 0x7f4aa4073e60 2024-08-03T21:01:13.204 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:13.202+0000 7f4aa8e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4aa415fff0 0x7f4aa41f5470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:01:13.204 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:13.202+0000 7f4aa8e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4aa415fff0 0x7f4aa41f5470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:58800/0 (socket says 172.21.15.32:58800) 2024-08-03T21:01:13.204 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:13.202+0000 7f4aa8e00640 1 -- 172.21.15.32:0/2684368920 learned_addr learned my addr 172.21.15.32:0/2684368920 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T21:01:13.205 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:13.202+0000 7f4aa3e00640 1 --2- 172.21.15.32:0/2684368920 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f4aa406e930 0x7f4aa415f870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:01:13.205 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:13.202+0000 7f4a9be00640 1 --2- 172.21.15.32:0/2684368920 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f4aa4073e60 0x7f4aa41539d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:01:13.205 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:13.202+0000 7f4aa8e00640 1 -- 172.21.15.32:0/2684368920 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f4aa4073e60 msgr2=0x7f4aa41539d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:01:13.205 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:13.202+0000 7f4aa8e00640 1 --2- 172.21.15.32:0/2684368920 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f4aa4073e60 0x7f4aa41539d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:01:13.205 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:13.202+0000 7f4aa8e00640 1 -- 172.21.15.32:0/2684368920 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f4aa406e930 msgr2=0x7f4aa415f870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:01:13.205 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:13.202+0000 7f4aa8e00640 1 --2- 172.21.15.32:0/2684368920 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f4aa406e930 0x7f4aa415f870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:01:13.205 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:13.202+0000 7f4aa8e00640 1 -- 172.21.15.32:0/2684368920 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4aa415cdc0 con 0x7f4aa415fff0 2024-08-03T21:01:13.205 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:13.202+0000 7f4a9be00640 1 --2- 172.21.15.32:0/2684368920 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f4aa4073e60 0x7f4aa41539d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-03T21:01:13.205 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:13.203+0000 7f4aa8e00640 1 --2- 172.21.15.32:0/2684368920 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4aa415fff0 0x7f4aa41f5470 secure :-1 s=READY pgs=2228 cs=0 l=1 rev1=1 crypto rx=0x7f4a9c00c730 tx=0x7f4a9c00cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T21:01:13.207 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:13.203+0000 7f4aa0c00640 1 -- 172.21.15.32:0/2684368920 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a9c003b70 con 0x7f4aa415fff0 2024-08-03T21:01:13.207 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:13.203+0000 7f4aa0c00640 1 -- 172.21.15.32:0/2684368920 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f4a9c003d30 con 0x7f4aa415fff0 2024-08-03T21:01:13.207 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:13.203+0000 7f4aaae00640 1 -- 172.21.15.32:0/2684368920 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4aa415d0a0 con 0x7f4aa415fff0 2024-08-03T21:01:13.207 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:13.203+0000 7f4aa0c00640 1 -- 172.21.15.32:0/2684368920 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a9c005330 con 0x7f4aa415fff0 2024-08-03T21:01:13.207 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:13.203+0000 7f4aaae00640 1 -- 172.21.15.32:0/2684368920 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4aa41f8de0 con 0x7f4aa415fff0 2024-08-03T21:01:13.207 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:13.204+0000 7f4aaae00640 1 -- 172.21.15.32:0/2684368920 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4aa4160db0 con 0x7f4aa415fff0 2024-08-03T21:01:13.207 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:13.205+0000 7f4aa0c00640 1 -- 172.21.15.32:0/2684368920 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f4a9c0056e0 con 0x7f4aa415fff0 2024-08-03T21:01:13.208 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:13.206+0000 7f4aa0c00640 1 --2- 172.21.15.32:0/2684368920 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f4a80077010 0x7f4a80079520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:01:13.208 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:13.206+0000 7f4aa0c00640 1 -- 172.21.15.32:0/2684368920 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(136..136 src has 1..136) ==== 5795+0+0 (secure 0 0 0) 0x7f4a9c098970 con 0x7f4aa415fff0 2024-08-03T21:01:13.208 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:13.207+0000 7f4aa3e00640 1 --2- 172.21.15.32:0/2684368920 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f4a80077010 0x7f4a80079520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:01:13.209 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:13.208+0000 7f4aa3e00640 1 --2- 172.21.15.32:0/2684368920 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f4a80077010 0x7f4a80079520 secure :-1 s=READY pgs=890 cs=0 l=1 rev1=1 crypto rx=0x7f4a94034000 tx=0x7f4a94004730 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T21:01:13.212 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:13.211+0000 7f4aa0c00640 1 -- 172.21.15.32:0/2684368920 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7f4a9c061ba0 con 0x7f4aa415fff0 2024-08-03T21:01:13.463 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:13.462+0000 7f4aaae00640 1 -- 172.21.15.32:0/2684368920 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f4aa4050ba0 con 0x7f4aa415fff0 2024-08-03T21:01:13.464 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:13.462+0000 7f4aa0c00640 1 -- 172.21.15.32:0/2684368920 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v346) ==== 53+0+73 (secure 0 0 0) 0x7f4a9c0614a0 con 0x7f4aa415fff0 2024-08-03T21:01:13.464 INFO:teuthology.orchestra.run.smithi032.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-08-03T21:01:13.466 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:13.465+0000 7f4aaae00640 1 -- 172.21.15.32:0/2684368920 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f4a80077010 msgr2=0x7f4a80079520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:01:13.467 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:13.465+0000 7f4aaae00640 1 --2- 172.21.15.32:0/2684368920 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f4a80077010 0x7f4a80079520 secure :-1 s=READY pgs=890 cs=0 l=1 rev1=1 crypto rx=0x7f4a94034000 tx=0x7f4a94004730 comp rx=0 tx=0).stop 2024-08-03T21:01:13.467 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:13.465+0000 7f4aaae00640 1 -- 172.21.15.32:0/2684368920 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4aa415fff0 msgr2=0x7f4aa41f5470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:01:13.467 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:13.465+0000 7f4aaae00640 1 --2- 172.21.15.32:0/2684368920 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4aa415fff0 0x7f4aa41f5470 secure :-1 s=READY pgs=2228 cs=0 l=1 rev1=1 crypto rx=0x7f4a9c00c730 tx=0x7f4a9c00cbf0 comp rx=0 tx=0).stop 2024-08-03T21:01:13.467 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:13.465+0000 7f4aaae00640 1 -- 172.21.15.32:0/2684368920 shutdown_connections 2024-08-03T21:01:13.467 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:13.465+0000 7f4aaae00640 1 --2- 172.21.15.32:0/2684368920 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f4a80077010 0x7f4a80079520 unknown :-1 s=CLOSED pgs=890 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:01:13.467 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:13.465+0000 7f4aaae00640 1 --2- 172.21.15.32:0/2684368920 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4aa415fff0 0x7f4aa41f5470 unknown :-1 s=CLOSED pgs=2228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:01:13.467 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:13.465+0000 7f4aaae00640 1 --2- 172.21.15.32:0/2684368920 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f4aa4073e60 0x7f4aa41539d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:01:13.467 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:13.465+0000 7f4aaae00640 1 --2- 172.21.15.32:0/2684368920 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f4aa406e930 0x7f4aa415f870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:01:13.467 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:13.465+0000 7f4aaae00640 1 -- 172.21.15.32:0/2684368920 >> 172.21.15.32:0/2684368920 conn(0x7f4aa40b1bd0 msgr2=0x7f4aa4160400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T21:01:13.467 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:13.465+0000 7f4aaae00640 1 -- 172.21.15.32:0/2684368920 shutdown_connections 2024-08-03T21:01:13.468 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:13.465+0000 7f4aaae00640 1 -- 172.21.15.32:0/2684368920 wait complete. 2024-08-03T21:01:13.489 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-08-03T21:01:13.490 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-08-03T21:01:13.490 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi032.front.sepia.ceph.com 2024-08-03T21:01:13.490 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-08-03T21:01:13.490 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-08-03T21:01:13.490 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-08-03T21:01:13.490 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2024-08-03T21:01:13.490 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-08-03T21:01:13.490 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-08-03T21:01:13.490 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-08-03T21:01:13.491 DEBUG:teuthology.orchestra.run.smithi032:> ip addr 2024-08-03T21:01:13.527 INFO:teuthology.orchestra.run.smithi032.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-08-03T21:01:13.527 INFO:teuthology.orchestra.run.smithi032.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-08-03T21:01:13.527 INFO:teuthology.orchestra.run.smithi032.stdout: inet 127.0.0.1/8 scope host lo 2024-08-03T21:01:13.527 INFO:teuthology.orchestra.run.smithi032.stdout: valid_lft forever preferred_lft forever 2024-08-03T21:01:13.527 INFO:teuthology.orchestra.run.smithi032.stdout: inet6 ::1/128 scope host 2024-08-03T21:01:13.527 INFO:teuthology.orchestra.run.smithi032.stdout: valid_lft forever preferred_lft forever 2024-08-03T21:01:13.527 INFO:teuthology.orchestra.run.smithi032.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-08-03T21:01:13.527 INFO:teuthology.orchestra.run.smithi032.stdout: link/ether 0c:c4:7a:6b:fd:0c brd ff:ff:ff:ff:ff:ff 2024-08-03T21:01:13.527 INFO:teuthology.orchestra.run.smithi032.stdout: altname enp7s0f0 2024-08-03T21:01:13.527 INFO:teuthology.orchestra.run.smithi032.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-08-03T21:01:13.528 INFO:teuthology.orchestra.run.smithi032.stdout: link/ether 0c:c4:7a:6b:fd:0d brd ff:ff:ff:ff:ff:ff 2024-08-03T21:01:13.528 INFO:teuthology.orchestra.run.smithi032.stdout: altname enp7s0f1 2024-08-03T21:01:13.528 INFO:teuthology.orchestra.run.smithi032.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-08-03T21:01:13.528 INFO:teuthology.orchestra.run.smithi032.stdout: link/ether ac:1f:6b:f8:26:02 brd ff:ff:ff:ff:ff:ff 2024-08-03T21:01:13.528 INFO:teuthology.orchestra.run.smithi032.stdout: altname enp1s0f0 2024-08-03T21:01:13.528 INFO:teuthology.orchestra.run.smithi032.stdout: inet 172.21.15.32/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-08-03T21:01:13.528 INFO:teuthology.orchestra.run.smithi032.stdout: valid_lft 41683sec preferred_lft 41683sec 2024-08-03T21:01:13.528 INFO:teuthology.orchestra.run.smithi032.stdout: inet6 fe80::ae1f:6bff:fef8:2602/64 scope link noprefixroute 2024-08-03T21:01:13.528 INFO:teuthology.orchestra.run.smithi032.stdout: valid_lft forever preferred_lft forever 2024-08-03T21:01:13.528 INFO:teuthology.orchestra.run.smithi032.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-08-03T21:01:13.528 INFO:teuthology.orchestra.run.smithi032.stdout: link/ether ac:1f:6b:f8:26:03 brd ff:ff:ff:ff:ff:ff 2024-08-03T21:01:13.529 INFO:teuthology.orchestra.run.smithi032.stdout: altname enp1s0f1 2024-08-03T21:01:13.529 INFO:teuthology.orchestra.run.smithi032.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-08-03T21:01:13.529 INFO:teuthology.orchestra.run.smithi032.stdout: link/ether 0a:23:2b:ca:2a:93 brd ff:ff:ff:ff:ff:ff 2024-08-03T21:01:13.529 INFO:teuthology.orchestra.run.smithi032.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-08-03T21:01:13.529 INFO:teuthology.orchestra.run.smithi032.stdout: valid_lft forever preferred_lft forever 2024-08-03T21:01:13.529 INFO:teuthology.orchestra.run.smithi032.stdout: inet6 fe80::d804:7cff:fed9:7ba4/64 scope link 2024-08-03T21:01:13.529 INFO:teuthology.orchestra.run.smithi032.stdout: valid_lft forever preferred_lft forever 2024-08-03T21:01:13.529 INFO:teuthology.orchestra.run.smithi032.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-08-03T21:01:13.529 INFO:teuthology.orchestra.run.smithi032.stdout: link/ether 22:86:28:30:4e:62 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-08-03T21:01:13.529 INFO:teuthology.orchestra.run.smithi032.stdout: inet6 fe80::2086:28ff:fe30:4e62/64 scope link 2024-08-03T21:01:13.529 INFO:teuthology.orchestra.run.smithi032.stdout: valid_lft forever preferred_lft forever 2024-08-03T21:01:13.530 INFO:teuthology.orchestra.run.smithi032.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-08-03T21:01:13.530 INFO:teuthology.orchestra.run.smithi032.stdout: link/ether 0a:23:2b:ca:2a:93 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-08-03T21:01:13.530 INFO:teuthology.orchestra.run.smithi032.stdout: inet6 fe80::823:2bff:feca:2a93/64 scope link 2024-08-03T21:01:13.530 INFO:teuthology.orchestra.run.smithi032.stdout: valid_lft forever preferred_lft forever 2024-08-03T21:01:13.530 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-08-03T21:01:13.530 DEBUG:teuthology.orchestra.run.smithi032:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-08-03T21:01:13.621 INFO:teuthology.orchestra.run.smithi032.stdout:1 2024-08-03T21:01:13.624 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-08-03T21:01:13.624 DEBUG:teuthology.orchestra.run.smithi032:> route 2024-08-03T21:01:13.660 INFO:teuthology.orchestra.run.smithi032.stdout:Kernel IP routing table 2024-08-03T21:01:13.660 INFO:teuthology.orchestra.run.smithi032.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-08-03T21:01:13.660 INFO:teuthology.orchestra.run.smithi032.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-08-03T21:01:13.660 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-08-03T21:01:13.660 INFO:teuthology.orchestra.run.smithi032.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-08-03T21:01:13.661 DEBUG:teuthology.orchestra.run.smithi032:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-08-03T21:01:13.661 DEBUG:teuthology.orchestra.run.smithi032:> set -e 2024-08-03T21:01:13.661 DEBUG:teuthology.orchestra.run.smithi032:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-08-03T21:01:13.661 DEBUG:teuthology.orchestra.run.smithi032:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-08-03T21:01:13.661 DEBUG:teuthology.orchestra.run.smithi032:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-08-03T21:01:13.661 DEBUG:teuthology.orchestra.run.smithi032:> ') 2024-08-03T21:01:13.779 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:13 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-08-03T21:01:13.851 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:13 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-08-03T21:01:13.859 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-08-03T21:01:13.859 DEBUG:teuthology.orchestra.run.smithi032:> ip netns list 2024-08-03T21:01:13.895 INFO:teuthology.orchestra.run.smithi032.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-08-03T21:01:13.895 INFO:teuthology.orchestra.run.smithi032.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-08-03T21:01:13.895 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-08-03T21:01:13.896 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-08-03T21:01:13.896 DEBUG:teuthology.orchestra.run.smithi032:> sudo ip link set brx.0 up 2024-08-03T21:01:13.992 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-08-03T21:01:13.992 DEBUG:teuthology.orchestra.run.smithi032:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-08-03T21:01:14.027 INFO:teuthology.orchestra.run.smithi032.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-08-03T21:01:14.027 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-08-03T21:01:14.028 DEBUG:teuthology.orchestra.run.smithi032:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-08-03T21:01:14.102 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2024-08-03T21:01:14.102 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-08-03T21:01:14.102 DEBUG:teuthology.orchestra.run.smithi032:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-08-03T21:01:14.302 INFO:teuthology.orchestra.run.smithi032.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-08-03T21:01:14.302 INFO:teuthology.orchestra.run.smithi032.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-08-03T21:01:14.302 INFO:teuthology.orchestra.run.smithi032.stdout:invalid new device string format 2024-08-03T21:01:14.322 INFO:teuthology.orchestra.run.smithi032.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-08-03T21:01:14.323 INFO:teuthology.orchestra.run.smithi032.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-08-03T21:01:14.323 INFO:teuthology.orchestra.run.smithi032.stdout:invalid new device string format 2024-08-03T21:01:14.323 INFO:teuthology.orchestra.run.smithi032.stdout:mount.ceph: resolved to: "172.21.15.32:3300,172.21.15.118:3300,172.21.15.118:3301" 2024-08-03T21:01:14.323 INFO:teuthology.orchestra.run.smithi032.stdout:mount.ceph: switching to using v1 address with old syntax 2024-08-03T21:01:14.353 INFO:teuthology.orchestra.run.smithi032.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-08-03T21:01:14.353 INFO:teuthology.orchestra.run.smithi032.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-08-03T21:01:14.353 INFO:teuthology.orchestra.run.smithi032.stdout:invalid new device string format 2024-08-03T21:01:14.353 INFO:teuthology.orchestra.run.smithi032.stdout:mount.ceph: resolved to: "172.21.15.32:3300,172.21.15.118:3300,172.21.15.118:3301" 2024-08-03T21:01:14.353 INFO:teuthology.orchestra.run.smithi032.stdout:mount.ceph: switching to using v1 address with old syntax 2024-08-03T21:01:14.353 INFO:teuthology.orchestra.run.smithi032.stdout:mount.ceph: resolved to: "172.21.15.32:6789,172.21.15.118:6789,172.21.15.118:6790" 2024-08-03T21:01:14.353 INFO:teuthology.orchestra.run.smithi032.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.32:6789,172.21.15.118:6789,172.21.15.118:6790:/ 2024-08-03T21:01:14.353 INFO:teuthology.orchestra.run.smithi032.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,key=0,fsid=40ab141a-01d7-49b9-ae3c-2edec9876e95" will pass to kernel 2024-08-03T21:01:14.355 INFO:teuthology.orchestra.run.smithi032.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2024-08-03T21:01:14.355 INFO:teuthology.orchestra.run.smithi032.stdout: You just mounted a file system that supports labels which does not 2024-08-03T21:01:14.355 INFO:teuthology.orchestra.run.smithi032.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-08-03T21:01:14.355 INFO:teuthology.orchestra.run.smithi032.stdout: applications will generate AVC messages and not be allowed access to 2024-08-03T21:01:14.355 INFO:teuthology.orchestra.run.smithi032.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-08-03T21:01:14.358 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-08-03T21:01:14.358 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-08-03T21:01:14.358 DEBUG:teuthology.orchestra.run.smithi032:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-08-03T21:01:14.414 DEBUG:teuthology.orchestra.run.smithi032:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-08-03T21:01:14.494 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:14 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-08-03T21:01:14.497 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:14 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-08-03T21:01:14.505 INFO:teuthology.orchestra.run.smithi032.stdout:client9826 2024-08-03T21:01:14.505 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-08-03T21:01:14.505 DEBUG:teuthology.orchestra.run.smithi032:> dd if=/proc/self/mounts of=/dev/stdout 2024-08-03T21:01:14.541 DEBUG:teuthology.orchestra.run.smithi032:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_data getxattr 10000000001.00000000 parent 2024-08-03T21:01:14.724 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:14.724+0000 7fc9f7a18f40 1 Processor -- start 2024-08-03T21:01:14.725 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:14.724+0000 7fc9f7a18f40 1 -- start start 2024-08-03T21:01:14.727 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:14.725+0000 7fc9f7a18f40 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55f47e4bc450 0x55f47e4bfa00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:01:14.727 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:14.725+0000 7fc9f7a18f40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55f47e4bc860 0x55f47e4bce30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:01:14.727 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:14.725+0000 7fc9f7a18f40 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55f47e4bd370 0x55f47e4c2cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:01:14.727 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:14.725+0000 7fc9f7a18f40 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55f47e33be30 con 0x55f47e4bc450 2024-08-03T21:01:14.727 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:14.725+0000 7fc9f7a18f40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x55f47e399bf0 con 0x55f47e4bc860 2024-08-03T21:01:14.727 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:14.725+0000 7fc9f7a18f40 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x55f47e4be6f0 con 0x55f47e4bd370 2024-08-03T21:01:14.727 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:14.725+0000 7fc9f0000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55f47e4bc450 0x55f47e4bfa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:01:14.727 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:14.725+0000 7fc9f0000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55f47e4bc450 0x55f47e4bfa00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:58820/0 (socket says 172.21.15.32:58820) 2024-08-03T21:01:14.727 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:14.725+0000 7fc9f0000640 1 -- 172.21.15.32:0/1279702279 learned_addr learned my addr 172.21.15.32:0/1279702279 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T21:01:14.727 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:14.726+0000 7fc9f0a00640 1 --2- 172.21.15.32:0/1279702279 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55f47e4bd370 0x55f47e4c2cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:01:14.727 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:14.726+0000 7fc9ef600640 1 --2- 172.21.15.32:0/1279702279 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55f47e4bc860 0x55f47e4bce30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:01:14.727 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:14.726+0000 7fc9f0000640 1 -- 172.21.15.32:0/1279702279 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55f47e4bd370 msgr2=0x55f47e4c2cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T21:01:14.728 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:14.726+0000 7fc9f0000640 1 --2- 172.21.15.32:0/1279702279 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55f47e4bd370 0x55f47e4c2cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:01:14.728 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:14.726+0000 7fc9f0000640 1 -- 172.21.15.32:0/1279702279 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55f47e4bc860 msgr2=0x55f47e4bce30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-03T21:01:14.728 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:14.726+0000 7fc9f0000640 1 --2- 172.21.15.32:0/1279702279 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55f47e4bc860 0x55f47e4bce30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:01:14.728 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:14.726+0000 7fc9f0000640 1 -- 172.21.15.32:0/1279702279 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f47e437050 con 0x55f47e4bc450 2024-08-03T21:01:14.728 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:14.726+0000 7fc9f0000640 1 --2- 172.21.15.32:0/1279702279 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55f47e4bc450 0x55f47e4bfa00 secure :-1 s=READY pgs=2231 cs=0 l=1 rev1=1 crypto rx=0x7fc9e0046820 tx=0x7fc9e0001a30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5e68018c02dc7de7 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T21:01:14.728 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:14.726+0000 7fc9ee200640 1 -- 172.21.15.32:0/1279702279 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9e0063020 con 0x55f47e4bc450 2024-08-03T21:01:14.728 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:14.726+0000 7fc9ee200640 1 -- 172.21.15.32:0/1279702279 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fc9e0001850 con 0x55f47e4bc450 2024-08-03T21:01:14.728 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:14.726+0000 7fc9ee200640 1 -- 172.21.15.32:0/1279702279 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9e004dbe0 con 0x55f47e4bc450 2024-08-03T21:01:14.729 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:14.727+0000 7fc9f7a18f40 1 -- 172.21.15.32:0/1279702279 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55f47e4bc450 msgr2=0x55f47e4bfa00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:01:14.729 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:14.727+0000 7fc9f7a18f40 1 --2- 172.21.15.32:0/1279702279 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55f47e4bc450 0x55f47e4bfa00 secure :-1 s=READY pgs=2231 cs=0 l=1 rev1=1 crypto rx=0x7fc9e0046820 tx=0x7fc9e0001a30 comp rx=0 tx=0).stop 2024-08-03T21:01:14.729 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:14.727+0000 7fc9f7a18f40 1 -- 172.21.15.32:0/1279702279 shutdown_connections 2024-08-03T21:01:14.729 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:14.727+0000 7fc9f7a18f40 1 --2- 172.21.15.32:0/1279702279 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55f47e4bc860 0x55f47e4bce30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:01:14.729 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:14.727+0000 7fc9f7a18f40 1 --2- 172.21.15.32:0/1279702279 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55f47e4bd370 0x55f47e4c2cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:01:14.729 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:14.727+0000 7fc9f7a18f40 1 --2- 172.21.15.32:0/1279702279 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55f47e4bc450 0x55f47e4bfa00 unknown :-1 s=CLOSED pgs=2231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:01:14.730 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:14.727+0000 7fc9f7a18f40 1 -- 172.21.15.32:0/1279702279 >> 172.21.15.32:0/1279702279 conn(0x55f47e30dac0 msgr2=0x55f47e4bbae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T21:01:14.730 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:14.727+0000 7fc9f7a18f40 1 -- 172.21.15.32:0/1279702279 shutdown_connections 2024-08-03T21:01:14.730 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:14.727+0000 7fc9f7a18f40 1 -- 172.21.15.32:0/1279702279 wait complete. 2024-08-03T21:01:14.730 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:14.728+0000 7fc9f7a18f40 1 Processor -- start 2024-08-03T21:01:14.730 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:14.728+0000 7fc9f7a18f40 1 -- start start 2024-08-03T21:01:14.732 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:14.730+0000 7fc9f7a18f40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55f47e4bc450 0x55f47e42ddb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:01:14.732 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:14.730+0000 7fc9f7a18f40 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55f47e4bc860 0x55f47e4c2660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:01:14.733 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:14.730+0000 7fc9f7a18f40 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55f47e4bd370 0x55f47e4c2ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:01:14.733 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:14.730+0000 7fc9f7a18f40 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55f47e436d70 con 0x55f47e4bc860 2024-08-03T21:01:14.733 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:14.730+0000 7fc9f7a18f40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x55f47e436a90 con 0x55f47e4bc450 2024-08-03T21:01:14.733 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:14.730+0000 7fc9f7a18f40 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x55f47e4be0a0 con 0x55f47e4bd370 2024-08-03T21:01:14.733 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:14.730+0000 7fc9ef600640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55f47e4bc860 0x55f47e4c2660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:01:14.733 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:14.730+0000 7fc9ef600640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55f47e4bc860 0x55f47e4c2660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:58826/0 (socket says 172.21.15.32:58826) 2024-08-03T21:01:14.733 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:14.730+0000 7fc9ef600640 1 -- 172.21.15.32:0/2323379704 learned_addr learned my addr 172.21.15.32:0/2323379704 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T21:01:14.734 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:14.730+0000 7fc9f0000640 1 --2- 172.21.15.32:0/2323379704 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55f47e4bc450 0x55f47e42ddb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:01:14.734 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:14.730+0000 7fc9f0a00640 1 --2- 172.21.15.32:0/2323379704 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55f47e4bd370 0x55f47e4c2ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:01:14.734 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:14.731+0000 7fc9ef600640 1 -- 172.21.15.32:0/2323379704 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55f47e4bd370 msgr2=0x55f47e4c2ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:01:14.734 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:14.731+0000 7fc9ef600640 1 --2- 172.21.15.32:0/2323379704 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x55f47e4bd370 0x55f47e4c2ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:01:14.734 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:14.731+0000 7fc9ef600640 1 -- 172.21.15.32:0/2323379704 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55f47e4bc450 msgr2=0x55f47e42ddb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:01:14.734 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:14.731+0000 7fc9ef600640 1 --2- 172.21.15.32:0/2323379704 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55f47e4bc450 0x55f47e42ddb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:01:14.734 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:14.731+0000 7fc9ef600640 1 -- 172.21.15.32:0/2323379704 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f47e436410 con 0x55f47e4bc860 2024-08-03T21:01:14.735 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:14.731+0000 7fc9ef600640 1 --2- 172.21.15.32:0/2323379704 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55f47e4bc860 0x55f47e4c2660 secure :-1 s=READY pgs=2232 cs=0 l=1 rev1=1 crypto rx=0x7fc9dc00a710 tx=0x7fc9dc00abd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T21:01:14.735 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:14.731+0000 7fc9ece00640 1 -- 172.21.15.32:0/2323379704 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9dc009d30 con 0x55f47e4bc860 2024-08-03T21:01:14.735 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:14.731+0000 7fc9ece00640 1 -- 172.21.15.32:0/2323379704 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fc9dc01b070 con 0x55f47e4bc860 2024-08-03T21:01:14.735 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:14.731+0000 7fc9ece00640 1 -- 172.21.15.32:0/2323379704 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9dc0166d0 con 0x55f47e4bc860 2024-08-03T21:01:14.735 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:14.731+0000 7fc9f7a18f40 1 -- 172.21.15.32:0/2323379704 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x55f47e49d4a0 con 0x55f47e4bc860 2024-08-03T21:01:14.735 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:14.731+0000 7fc9f7a18f40 1 -- 172.21.15.32:0/2323379704 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f47e4a35f0 con 0x55f47e4bc860 2024-08-03T21:01:14.737 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:14.733+0000 7fc9ece00640 1 -- 172.21.15.32:0/2323379704 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7fc9dc016870 con 0x55f47e4bc860 2024-08-03T21:01:14.737 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:14.734+0000 7fc9ece00640 1 --2- 172.21.15.32:0/2323379704 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fc9d0075a20 0x7fc9d0077f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:01:14.737 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:14.734+0000 7fc9ece00640 1 -- 172.21.15.32:0/2323379704 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(136..136 src has 1..136) ==== 5795+0+0 (secure 0 0 0) 0x7fc9dc098830 con 0x55f47e4bc860 2024-08-03T21:01:14.737 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:14.735+0000 7fc9f7a18f40 1 --2- 172.21.15.32:0/2323379704 >> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] conn(0x55f47e49ce40 0x55f47e42d6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:01:14.738 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:14.735+0000 7fc9f0000640 1 --2- 172.21.15.32:0/2323379704 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fc9d0075a20 0x7fc9d0077f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:01:14.738 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:14.735+0000 7fc9f7a18f40 1 -- 172.21.15.32:0/2323379704 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- osd_op(unknown.0.0:1 15.7 15:e2e07930:::10000000001.00000000:head [getxattr parent in=6b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e136) -- 0x55f47e5674c0 con 0x55f47e49ce40 2024-08-03T21:01:14.738 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:14.735+0000 7fc9f0a00640 1 --2- 172.21.15.32:0/2323379704 >> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] conn(0x55f47e49ce40 0x55f47e42d6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:01:14.738 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:14.735+0000 7fc9f0000640 1 --2- 172.21.15.32:0/2323379704 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fc9d0075a20 0x7fc9d0077f30 secure :-1 s=READY pgs=891 cs=0 l=1 rev1=1 crypto rx=0x7fc9e0064000 tx=0x7fc9e00092a0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T21:01:14.738 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:14.736+0000 7fc9f0a00640 1 --2- 172.21.15.32:0/2323379704 >> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] conn(0x55f47e49ce40 0x55f47e42d6d0 crc :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T21:01:14.738 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:14.736+0000 7fc9f0a00640 1 -- 172.21.15.32:0/2323379704 <== osd.1 v2:172.21.15.32:6824/1551241759 1 ==== osd_op_reply(1 10000000001.00000000 [getxattr (91) out=91b] v0'0 uv6 ondisk = 0) ==== 164+0+91 (crc 0 0 0) 0x7fc9e8003040 con 0x55f47e49ce40 2024-08-03T21:01:14.739 INFO:teuthology.orchestra.run.smithi032.stdout:Ufile1dir1 2024-08-03T21:01:14.740 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:14.736+0000 7fc9f7a18f40 1 -- 172.21.15.32:0/2323379704 >> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] conn(0x55f47e49ce40 msgr2=0x55f47e42d6d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:01:14.740 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:14.736+0000 7fc9f7a18f40 1 --2- 172.21.15.32:0/2323379704 >> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] conn(0x55f47e49ce40 0x55f47e42d6d0 crc :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:01:14.740 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:14.736+0000 7fc9f7a18f40 1 -- 172.21.15.32:0/2323379704 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fc9d0075a20 msgr2=0x7fc9d0077f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:01:14.740 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:14.736+0000 7fc9f7a18f40 1 --2- 172.21.15.32:0/2323379704 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fc9d0075a20 0x7fc9d0077f30 secure :-1 s=READY pgs=891 cs=0 l=1 rev1=1 crypto rx=0x7fc9e0064000 tx=0x7fc9e00092a0 comp rx=0 tx=0).stop 2024-08-03T21:01:14.741 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:14.736+0000 7fc9f7a18f40 1 -- 172.21.15.32:0/2323379704 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55f47e4bc860 msgr2=0x55f47e4c2660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:01:14.741 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:14.736+0000 7fc9f7a18f40 1 --2- 172.21.15.32:0/2323379704 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55f47e4bc860 0x55f47e4c2660 secure :-1 s=READY pgs=2232 cs=0 l=1 rev1=1 crypto rx=0x7fc9dc00a710 tx=0x7fc9dc00abd0 comp rx=0 tx=0).stop 2024-08-03T21:01:14.741 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:14.736+0000 7fc9f0a00640 1 -- 172.21.15.32:0/2323379704 reap_dead start 2024-08-03T21:01:14.741 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:14.736+0000 7fc9f7a18f40 1 -- 172.21.15.32:0/2323379704 shutdown_connections 2024-08-03T21:01:14.741 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:14.736+0000 7fc9f7a18f40 1 -- 172.21.15.32:0/2323379704 >> 172.21.15.32:0/2323379704 conn(0x55f47e30dac0 msgr2=0x55f47e4bd780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T21:01:14.741 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:14.737+0000 7fc9f7a18f40 1 -- 172.21.15.32:0/2323379704 shutdown_connections 2024-08-03T21:01:14.741 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:14.737+0000 7fc9f7a18f40 1 -- 172.21.15.32:0/2323379704 wait complete. 2024-08-03T21:01:14.749 DEBUG:teuthology.orchestra.run.smithi032:> ceph-dencoder type inode_backtrace_t import - decode dump_json 2024-08-03T21:01:14.944 INFO:teuthology.orchestra.run.smithi032.stderr:error: stray data at end of buffer, offset 91 2024-08-03T21:01:14.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-03T21:01:14.960 INFO:teuthology.nuke:Clearing teuthology firewall rules... 2024-08-03T21:01:14.960 DEBUG:teuthology.orchestra.run.smithi032:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-08-03T21:01:15.052 DEBUG:teuthology.orchestra.run.smithi118:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-08-03T21:01:15.144 INFO:teuthology.nuke:Cleared teuthology firewall rules. 2024-08-03T21:01:15.144 INFO:tasks.cephfs.mount:Terminating background process 2024-08-03T21:01:15.145 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-08-03T21:01:15.145 DEBUG:teuthology.orchestra.run.smithi032:> dd if=/proc/self/mounts of=/dev/stdout 2024-08-03T21:01:15.180 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-08-03T21:01:15.180 DEBUG:teuthology.orchestra.run.smithi032:> dd if=/proc/self/mounts of=/dev/stdout 2024-08-03T21:01:15.256 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-08-03T21:01:15.527 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:15.526+0000 7fa1e2200640 1 -- 172.21.15.32:0/2256976188 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa1dc15fff0 msgr2=0x7fa1dc160400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:01:15.528 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:15.526+0000 7fa1e2200640 1 --2- 172.21.15.32:0/2256976188 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa1dc15fff0 0x7fa1dc160400 secure :-1 s=READY pgs=2234 cs=0 l=1 rev1=1 crypto rx=0x7fa1d4005fd0 tx=0x7fa1d402a8b0 comp rx=0 tx=0).stop 2024-08-03T21:01:15.528 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:15.526+0000 7fa1e2200640 1 -- 172.21.15.32:0/2256976188 shutdown_connections 2024-08-03T21:01:15.528 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:15.526+0000 7fa1e2200640 1 --2- 172.21.15.32:0/2256976188 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa1dc15fff0 0x7fa1dc160400 unknown :-1 s=CLOSED pgs=2234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:01:15.528 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:15.526+0000 7fa1e2200640 1 --2- 172.21.15.32:0/2256976188 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fa1dc144f90 0x7fa1dc074570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:01:15.528 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:15.526+0000 7fa1e2200640 1 -- 172.21.15.32:0/2256976188 >> 172.21.15.32:0/2256976188 conn(0x7fa1dc0b1bd0 msgr2=0x7fa1dc0b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T21:01:15.528 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:15.526+0000 7fa1e2200640 1 -- 172.21.15.32:0/2256976188 shutdown_connections 2024-08-03T21:01:15.528 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:15.526+0000 7fa1e2200640 1 -- 172.21.15.32:0/2256976188 wait complete. 2024-08-03T21:01:15.529 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:15.529+0000 7fa1e2200640 1 Processor -- start 2024-08-03T21:01:15.529 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:15.529+0000 7fa1e2200640 1 -- start start 2024-08-03T21:01:15.532 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:15.530+0000 7fa1e2200640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fa1dc06e930 0x7fa1dc13c980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:01:15.533 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:15.530+0000 7fa1e2200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa1dc144f90 0x7fa1dc1427f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:01:15.533 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:15.530+0000 7fa1e2200640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa1dc15fff0 0x7fa1dc13d100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:01:15.533 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:15.530+0000 7fa1e2200640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fa1dc0cf840 con 0x7fa1dc144f90 2024-08-03T21:01:15.533 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:15.530+0000 7fa1e2200640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7fa1dc145f30 con 0x7fa1dc15fff0 2024-08-03T21:01:15.533 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:15.530+0000 7fa1e2200640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7fa1dc0cf9c0 con 0x7fa1dc06e930 2024-08-03T21:01:15.533 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:15.531+0000 7fa1d3e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa1dc144f90 0x7fa1dc1427f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:01:15.533 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:15.531+0000 7fa1d3e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa1dc144f90 0x7fa1dc1427f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:58838/0 (socket says 172.21.15.32:58838) 2024-08-03T21:01:15.533 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:15.531+0000 7fa1d3e00640 1 -- 172.21.15.32:0/3986670279 learned_addr learned my addr 172.21.15.32:0/3986670279 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T21:01:15.533 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:15.531+0000 7fa1db400640 1 --2- 172.21.15.32:0/3986670279 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fa1dc06e930 0x7fa1dc13c980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:01:15.533 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:15.531+0000 7fa1dbe00640 1 --2- 172.21.15.32:0/3986670279 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa1dc15fff0 0x7fa1dc13d100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:01:15.533 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:15.531+0000 7fa1d3e00640 1 -- 172.21.15.32:0/3986670279 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fa1dc06e930 msgr2=0x7fa1dc13c980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:01:15.533 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:15.531+0000 7fa1d3e00640 1 --2- 172.21.15.32:0/3986670279 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fa1dc06e930 0x7fa1dc13c980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:01:15.533 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:15.531+0000 7fa1d3e00640 1 -- 172.21.15.32:0/3986670279 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa1dc15fff0 msgr2=0x7fa1dc13d100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:01:15.534 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:15.531+0000 7fa1d3e00640 1 --2- 172.21.15.32:0/3986670279 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa1dc15fff0 0x7fa1dc13d100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:01:15.534 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:15.531+0000 7fa1d3e00640 1 -- 172.21.15.32:0/3986670279 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa1dc13cec0 con 0x7fa1dc144f90 2024-08-03T21:01:15.534 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:15.531+0000 7fa1d3e00640 1 --2- 172.21.15.32:0/3986670279 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa1dc144f90 0x7fa1dc1427f0 secure :-1 s=READY pgs=2235 cs=0 l=1 rev1=1 crypto rx=0x7fa1c400d770 tx=0x7fa1c400dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T21:01:15.534 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:15.532+0000 7fa1d3400640 1 -- 172.21.15.32:0/3986670279 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1c4017070 con 0x7fa1dc144f90 2024-08-03T21:01:15.534 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:15.532+0000 7fa1d3400640 1 -- 172.21.15.32:0/3986670279 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fa1c4009d20 con 0x7fa1dc144f90 2024-08-03T21:01:15.534 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:15.532+0000 7fa1e2200640 1 -- 172.21.15.32:0/3986670279 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa1dc143270 con 0x7fa1dc144f90 2024-08-03T21:01:15.534 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:15.532+0000 7fa1d3400640 1 -- 172.21.15.32:0/3986670279 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1c4004280 con 0x7fa1dc144f90 2024-08-03T21:01:15.534 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:15.532+0000 7fa1e2200640 1 -- 172.21.15.32:0/3986670279 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa1dc143830 con 0x7fa1dc144f90 2024-08-03T21:01:15.535 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:15.533+0000 7fa1e2200640 1 -- 172.21.15.32:0/3986670279 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa1dc160400 con 0x7fa1dc144f90 2024-08-03T21:01:15.535 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:15.534+0000 7fa1d3400640 1 -- 172.21.15.32:0/3986670279 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7fa1c4005020 con 0x7fa1dc144f90 2024-08-03T21:01:15.536 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:15.535+0000 7fa1d3400640 1 --2- 172.21.15.32:0/3986670279 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fa1b8077010 0x7fa1b8079520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:01:15.536 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:15.535+0000 7fa1d3400640 1 -- 172.21.15.32:0/3986670279 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(136..136 src has 1..136) ==== 5795+0+0 (secure 0 0 0) 0x7fa1c4098ce0 con 0x7fa1dc144f90 2024-08-03T21:01:15.536 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:15.536+0000 7fa1db400640 1 --2- 172.21.15.32:0/3986670279 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fa1b8077010 0x7fa1b8079520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:01:15.537 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:15.536+0000 7fa1db400640 1 --2- 172.21.15.32:0/3986670279 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fa1b8077010 0x7fa1b8079520 secure :-1 s=READY pgs=892 cs=0 l=1 rev1=1 crypto rx=0x7fa1cc007fd0 tx=0x7fa1cc009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T21:01:15.540 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:15.540+0000 7fa1d3400640 1 -- 172.21.15.32:0/3986670279 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7fa1dc160400 con 0x7fa1dc144f90 2024-08-03T21:01:15.757 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:15.757+0000 7fa1e2200640 1 -- 172.21.15.32:0/3986670279 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7fa1dc143000 con 0x7fa1dc144f90 2024-08-03T21:01:15.758 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:15.757+0000 7fa1d3400640 1 -- 172.21.15.32:0/3986670279 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 2 entries v136) ==== 80+0+122 (secure 0 0 0) 0x7fa1c4020020 con 0x7fa1dc144f90 2024-08-03T21:01:15.758 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6835/4032603982 2024-08-04T21:01:06.854369+0000 2024-08-03T21:01:15.758 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6833/4032603982 2024-08-04T21:01:06.854369+0000 2024-08-03T21:01:15.759 INFO:teuthology.orchestra.run.smithi032.stderr:listed 2 entries 2024-08-03T21:01:15.760 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:15.760+0000 7fa1e2200640 1 -- 172.21.15.32:0/3986670279 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fa1b8077010 msgr2=0x7fa1b8079520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:01:15.761 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:15.760+0000 7fa1e2200640 1 --2- 172.21.15.32:0/3986670279 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fa1b8077010 0x7fa1b8079520 secure :-1 s=READY pgs=892 cs=0 l=1 rev1=1 crypto rx=0x7fa1cc007fd0 tx=0x7fa1cc009040 comp rx=0 tx=0).stop 2024-08-03T21:01:15.761 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:15.760+0000 7fa1e2200640 1 -- 172.21.15.32:0/3986670279 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa1dc144f90 msgr2=0x7fa1dc1427f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:01:15.761 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:15.760+0000 7fa1e2200640 1 --2- 172.21.15.32:0/3986670279 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa1dc144f90 0x7fa1dc1427f0 secure :-1 s=READY pgs=2235 cs=0 l=1 rev1=1 crypto rx=0x7fa1c400d770 tx=0x7fa1c400dc30 comp rx=0 tx=0).stop 2024-08-03T21:01:15.761 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:15.760+0000 7fa1e2200640 1 -- 172.21.15.32:0/3986670279 shutdown_connections 2024-08-03T21:01:15.761 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:15.760+0000 7fa1e2200640 1 --2- 172.21.15.32:0/3986670279 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fa1b8077010 0x7fa1b8079520 unknown :-1 s=CLOSED pgs=892 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:01:15.761 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:15.760+0000 7fa1e2200640 1 --2- 172.21.15.32:0/3986670279 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa1dc15fff0 0x7fa1dc13d100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:01:15.761 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:15.760+0000 7fa1e2200640 1 --2- 172.21.15.32:0/3986670279 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa1dc144f90 0x7fa1dc1427f0 unknown :-1 s=CLOSED pgs=2235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:01:15.761 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:15.760+0000 7fa1e2200640 1 --2- 172.21.15.32:0/3986670279 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fa1dc06e930 0x7fa1dc13c980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:01:15.761 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:15.760+0000 7fa1e2200640 1 -- 172.21.15.32:0/3986670279 >> 172.21.15.32:0/3986670279 conn(0x7fa1dc0b1bd0 msgr2=0x7fa1dc0d0920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T21:01:15.761 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:15.760+0000 7fa1e2200640 1 -- 172.21.15.32:0/3986670279 shutdown_connections 2024-08-03T21:01:15.761 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:15.760+0000 7fa1e2200640 1 -- 172.21.15.32:0/3986670279 wait complete. 2024-08-03T21:01:15.784 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-08-03T21:01:15.785 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-08-03T21:01:15.785 DEBUG:teuthology.orchestra.run.smithi032:> sudo umount /home/ubuntu/cephtest/mnt.0 2024-08-03T21:01:15.847 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi032.front.sepia.ceph.com 2024-08-03T21:01:15.847 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-08-03T21:01:15.847 DEBUG:teuthology.orchestra.run.smithi032:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-08-03T21:01:15.884 INFO:tasks.cephfs.mount:Terminating background process 2024-08-03T21:01:15.884 DEBUG:teuthology.orchestra.run.smithi118:> set -ex 2024-08-03T21:01:15.884 DEBUG:teuthology.orchestra.run.smithi118:> dd if=/proc/self/mounts of=/dev/stdout 2024-08-03T21:01:15.925 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052663,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497944k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248972k,nr_inodes=812243,mode=700,uid=1000,gid=1316 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497944k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-08-03T21:01:15.926 INFO:tasks.cephfs.mount:Terminating background process 2024-08-03T21:01:15.926 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-08-03T21:01:15.926 DEBUG:teuthology.orchestra.run.smithi032:> dd if=/proc/self/mounts of=/dev/stdout 2024-08-03T21:01:15.961 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053453,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6499208k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 none /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249604k,nr_inodes=812401,mode=700,uid=1000,gid=1316 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6499208k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-08-03T21:01:15.961 INFO:tasks.cephfs.mount:Terminating background process 2024-08-03T21:01:15.961 DEBUG:teuthology.orchestra.run.smithi118:> set -ex 2024-08-03T21:01:15.961 DEBUG:teuthology.orchestra.run.smithi118:> dd if=/proc/self/mounts of=/dev/stdout 2024-08-03T21:01:16.004 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052663,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497944k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248972k,nr_inodes=812243,mode=700,uid=1000,gid=1316 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497944k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-08-03T21:01:16.004 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-08-03T21:01:16.232 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:16.231+0000 7f4085800640 1 -- 172.21.15.32:0/586756049 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f40800cd860 msgr2=0x7f4080153270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:01:16.233 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:16.231+0000 7f4085800640 1 --2- 172.21.15.32:0/586756049 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f40800cd860 0x7f4080153270 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7f4068009830 tx=0x7f406802a490 comp rx=0 tx=0).stop 2024-08-03T21:01:16.233 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:16.231+0000 7f4085800640 1 -- 172.21.15.32:0/586756049 shutdown_connections 2024-08-03T21:01:16.233 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:16.231+0000 7f4085800640 1 --2- 172.21.15.32:0/586756049 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f40800cd860 0x7f4080153270 unknown :-1 s=CLOSED pgs=670 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:01:16.233 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:16.231+0000 7f4085800640 1 -- 172.21.15.32:0/586756049 >> 172.21.15.32:0/586756049 conn(0x7f4080072e70 msgr2=0x7f4080073280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T21:01:16.233 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:16.231+0000 7f4085800640 1 -- 172.21.15.32:0/586756049 shutdown_connections 2024-08-03T21:01:16.233 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:16.232+0000 7f4085800640 1 -- 172.21.15.32:0/586756049 wait complete. 2024-08-03T21:01:16.234 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:16.234+0000 7f4085800640 1 Processor -- start 2024-08-03T21:01:16.234 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:16.234+0000 7f4085800640 1 -- start start 2024-08-03T21:01:16.237 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:16.235+0000 7f4085800640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f408006d9f0 0x7f4080141ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:01:16.237 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:16.235+0000 7f4085800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4080073990 0x7f4080155300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:01:16.237 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:16.235+0000 7f4085800640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f40800cd860 0x7f408015e820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:01:16.237 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:16.235+0000 7f4085800640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f4080167730 con 0x7f4080073990 2024-08-03T21:01:16.237 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:16.235+0000 7f4085800640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f40801675b0 con 0x7f408006d9f0 2024-08-03T21:01:16.237 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:16.235+0000 7f4085800640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7f40801678b0 con 0x7f40800cd860 2024-08-03T21:01:16.237 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:16.236+0000 7f4077e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4080073990 0x7f4080155300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:01:16.237 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:16.236+0000 7f4077e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4080073990 0x7f4080155300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:58848/0 (socket says 172.21.15.32:58848) 2024-08-03T21:01:16.237 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:16.236+0000 7f4077e00640 1 -- 172.21.15.32:0/1500928694 learned_addr learned my addr 172.21.15.32:0/1500928694 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T21:01:16.238 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:16.236+0000 7f407fe00640 1 --2- 172.21.15.32:0/1500928694 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f408006d9f0 0x7f4080141ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:01:16.238 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:16.236+0000 7f4084e00640 1 --2- 172.21.15.32:0/1500928694 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f40800cd860 0x7f408015e820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:01:16.238 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:16.236+0000 7f4077e00640 1 -- 172.21.15.32:0/1500928694 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f40800cd860 msgr2=0x7f408015e820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:01:16.238 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:16.236+0000 7f4077e00640 1 --2- 172.21.15.32:0/1500928694 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f40800cd860 0x7f408015e820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:01:16.238 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:16.236+0000 7f4077e00640 1 -- 172.21.15.32:0/1500928694 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f408006d9f0 msgr2=0x7f4080141ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:01:16.238 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:16.236+0000 7f4077e00640 1 --2- 172.21.15.32:0/1500928694 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f408006d9f0 0x7f4080141ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:01:16.238 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:16.236+0000 7f4077e00640 1 -- 172.21.15.32:0/1500928694 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4080154380 con 0x7f4080073990 2024-08-03T21:01:16.238 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:16.236+0000 7f4077e00640 1 --2- 172.21.15.32:0/1500928694 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4080073990 0x7f4080155300 secure :-1 s=READY pgs=2237 cs=0 l=1 rev1=1 crypto rx=0x7f4068009950 tx=0x7f4068033c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T21:01:16.239 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:16.237+0000 7f407cc00640 1 -- 172.21.15.32:0/1500928694 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4068031d60 con 0x7f4080073990 2024-08-03T21:01:16.239 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:16.237+0000 7f407cc00640 1 -- 172.21.15.32:0/1500928694 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f4068033e10 con 0x7f4080073990 2024-08-03T21:01:16.239 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:16.237+0000 7f407cc00640 1 -- 172.21.15.32:0/1500928694 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4068005260 con 0x7f4080073990 2024-08-03T21:01:16.240 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:16.237+0000 7f4085800640 1 -- 172.21.15.32:0/1500928694 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4080154660 con 0x7f4080073990 2024-08-03T21:01:16.240 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:16.237+0000 7f4085800640 1 -- 172.21.15.32:0/1500928694 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4080154c20 con 0x7f4080073990 2024-08-03T21:01:16.240 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:16.238+0000 7f4085800640 1 -- 172.21.15.32:0/1500928694 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4080161730 con 0x7f4080073990 2024-08-03T21:01:16.240 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:16.239+0000 7f407cc00640 1 -- 172.21.15.32:0/1500928694 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f4068005400 con 0x7f4080073990 2024-08-03T21:01:16.241 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:16.240+0000 7f407cc00640 1 --2- 172.21.15.32:0/1500928694 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f405c077040 0x7f405c079550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:01:16.241 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:16.240+0000 7f407cc00640 1 -- 172.21.15.32:0/1500928694 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(136..136 src has 1..136) ==== 5795+0+0 (secure 0 0 0) 0x7f40680b99a0 con 0x7f4080073990 2024-08-03T21:01:16.241 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:16.241+0000 7f407fe00640 1 --2- 172.21.15.32:0/1500928694 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f405c077040 0x7f405c079550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:01:16.242 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:16.242+0000 7f407fe00640 1 --2- 172.21.15.32:0/1500928694 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f405c077040 0x7f405c079550 secure :-1 s=READY pgs=893 cs=0 l=1 rev1=1 crypto rx=0x7f4070007fd0 tx=0x7f4070001370 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T21:01:16.245 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:16.245+0000 7f407cc00640 1 -- 172.21.15.32:0/1500928694 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7f406803f030 con 0x7f4080073990 2024-08-03T21:01:16.488 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:16.488+0000 7f4085800640 1 -- 172.21.15.32:0/1500928694 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f4080153270 con 0x7f4080073990 2024-08-03T21:01:16.489 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:16.488+0000 7f407cc00640 1 -- 172.21.15.32:0/1500928694 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 346 v346) ==== 95+0+5236 (secure 0 0 0) 0x7f40680824e0 con 0x7f4080073990 2024-08-03T21:01:16.490 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-03T21:01:16.490 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":346,"btime":"2024-08-03T21:01:10:877273+0000","default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8519,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6836/3249267506","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6834","nonce":3249267506},{"type":"v1","addr":"172.21.15.118:6836","nonce":3249267506}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":334},{"gid":8522,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6837/408542078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6835","nonce":408542078},{"type":"v1","addr":"172.21.15.118:6837","nonce":408542078}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":335},{"gid":8648,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6835/1693401067","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6833","nonce":1693401067},{"type":"v1","addr":"172.21.15.32:6835","nonce":1693401067}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":342}],"filesystems":[{"mdsmap":{"epoch":346,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-08-03T21:00:49.742850+0000","modified":"2024-08-03T21:01:10.877262+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":136,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":9589},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9589":{"gid":9589,"name":"c","rank":0,"incarnation":343,"state":"up:active","state_seq":9,"addr":"172.21.15.32:6834/3440267920","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":3440267920},{"type":"v1","addr":"172.21.15.32:6834","nonce":3440267920}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9589,"qdb_cluster":[9589]},"id":7}]} 2024-08-03T21:01:16.490 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 346 2024-08-03T21:01:16.491 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:16.491+0000 7f4085800640 1 -- 172.21.15.32:0/1500928694 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f405c077040 msgr2=0x7f405c079550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:01:16.491 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:16.491+0000 7f4085800640 1 --2- 172.21.15.32:0/1500928694 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f405c077040 0x7f405c079550 secure :-1 s=READY pgs=893 cs=0 l=1 rev1=1 crypto rx=0x7f4070007fd0 tx=0x7f4070001370 comp rx=0 tx=0).stop 2024-08-03T21:01:16.492 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:16.491+0000 7f4085800640 1 -- 172.21.15.32:0/1500928694 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4080073990 msgr2=0x7f4080155300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:01:16.492 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:16.491+0000 7f4085800640 1 --2- 172.21.15.32:0/1500928694 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4080073990 0x7f4080155300 secure :-1 s=READY pgs=2237 cs=0 l=1 rev1=1 crypto rx=0x7f4068009950 tx=0x7f4068033c20 comp rx=0 tx=0).stop 2024-08-03T21:01:16.492 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:16.491+0000 7f4085800640 1 -- 172.21.15.32:0/1500928694 shutdown_connections 2024-08-03T21:01:16.492 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:16.491+0000 7f4085800640 1 --2- 172.21.15.32:0/1500928694 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f405c077040 0x7f405c079550 unknown :-1 s=CLOSED pgs=893 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:01:16.492 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:16.491+0000 7f4085800640 1 --2- 172.21.15.32:0/1500928694 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f40800cd860 0x7f408015e820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:01:16.492 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:16.491+0000 7f4085800640 1 --2- 172.21.15.32:0/1500928694 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4080073990 0x7f4080155300 unknown :-1 s=CLOSED pgs=2237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:01:16.492 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:16.491+0000 7f4085800640 1 --2- 172.21.15.32:0/1500928694 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f408006d9f0 0x7f4080141ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:01:16.492 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:16.491+0000 7f4085800640 1 -- 172.21.15.32:0/1500928694 >> 172.21.15.32:0/1500928694 conn(0x7f4080072e70 msgr2=0x7f408015fbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T21:01:16.492 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:16.491+0000 7f4085800640 1 -- 172.21.15.32:0/1500928694 shutdown_connections 2024-08-03T21:01:16.492 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:16.491+0000 7f4085800640 1 -- 172.21.15.32:0/1500928694 wait complete. 2024-08-03T21:01:16.515 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-08-03T21:01:16.742 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:16.741+0000 7eff69400640 1 -- 172.21.15.32:0/2827487116 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7eff640b4de0 msgr2=0x7eff641515c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:01:16.743 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:16.741+0000 7eff69400640 1 --2- 172.21.15.32:0/2827487116 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7eff640b4de0 0x7eff641515c0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7eff4c009860 tx=0x7eff4c02a510 comp rx=0 tx=0).stop 2024-08-03T21:01:16.743 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:16.741+0000 7eff69400640 1 -- 172.21.15.32:0/2827487116 shutdown_connections 2024-08-03T21:01:16.743 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:16.741+0000 7eff69400640 1 --2- 172.21.15.32:0/2827487116 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7eff640b4de0 0x7eff641515c0 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:01:16.743 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:16.741+0000 7eff69400640 1 -- 172.21.15.32:0/2827487116 >> 172.21.15.32:0/2827487116 conn(0x7eff640b1bd0 msgr2=0x7eff640b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T21:01:16.743 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:16.742+0000 7eff69400640 1 -- 172.21.15.32:0/2827487116 shutdown_connections 2024-08-03T21:01:16.743 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:16.742+0000 7eff69400640 1 -- 172.21.15.32:0/2827487116 wait complete. 2024-08-03T21:01:16.744 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:16.744+0000 7eff69400640 1 Processor -- start 2024-08-03T21:01:16.744 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:16.744+0000 7eff69400640 1 -- start start 2024-08-03T21:01:16.749 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:16.745+0000 7eff69400640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7eff640708c0 0x7eff641f5ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:01:16.749 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:16.745+0000 7eff69400640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7eff640b4de0 0x7eff6414ec90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:01:16.749 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:16.745+0000 7eff69400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7eff6415e170 0x7eff641f88f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:01:16.749 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:16.745+0000 7eff69400640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7eff6406ecc0 con 0x7eff6415e170 2024-08-03T21:01:16.750 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:16.745+0000 7eff69400640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7eff6406eb40 con 0x7eff640708c0 2024-08-03T21:01:16.750 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:16.745+0000 7eff69400640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7eff6406ee40 con 0x7eff640b4de0 2024-08-03T21:01:16.750 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:16.746+0000 7eff63200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7eff6415e170 0x7eff641f88f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:01:16.750 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:16.746+0000 7eff63200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7eff6415e170 0x7eff641f88f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:58862/0 (socket says 172.21.15.32:58862) 2024-08-03T21:01:16.750 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:16.746+0000 7eff63200640 1 -- 172.21.15.32:0/2425657904 learned_addr learned my addr 172.21.15.32:0/2425657904 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T21:01:16.750 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:16.746+0000 7eff59e00640 1 --2- 172.21.15.32:0/2425657904 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7eff640b4de0 0x7eff6414ec90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:01:16.751 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:16.746+0000 7eff62800640 1 --2- 172.21.15.32:0/2425657904 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7eff640708c0 0x7eff641f5ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:01:16.751 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:16.746+0000 7eff63200640 1 -- 172.21.15.32:0/2425657904 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7eff640b4de0 msgr2=0x7eff6414ec90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:01:16.751 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:16.746+0000 7eff63200640 1 --2- 172.21.15.32:0/2425657904 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7eff640b4de0 0x7eff6414ec90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:01:16.751 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:16.746+0000 7eff63200640 1 -- 172.21.15.32:0/2425657904 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7eff640708c0 msgr2=0x7eff641f5ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:01:16.751 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:16.746+0000 7eff63200640 1 --2- 172.21.15.32:0/2425657904 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7eff640708c0 0x7eff641f5ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:01:16.751 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:16.746+0000 7eff63200640 1 -- 172.21.15.32:0/2425657904 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7eff641f61e0 con 0x7eff6415e170 2024-08-03T21:01:16.752 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:16.747+0000 7eff63200640 1 --2- 172.21.15.32:0/2425657904 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7eff6415e170 0x7eff641f88f0 secure :-1 s=READY pgs=2239 cs=0 l=1 rev1=1 crypto rx=0x7eff5c00c730 tx=0x7eff5c00cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T21:01:16.753 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:16.747+0000 7eff5b400640 1 -- 172.21.15.32:0/2425657904 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff5c003b70 con 0x7eff6415e170 2024-08-03T21:01:16.753 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:16.747+0000 7eff5b400640 1 -- 172.21.15.32:0/2425657904 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7eff5c003d30 con 0x7eff6415e170 2024-08-03T21:01:16.753 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:16.747+0000 7eff69400640 1 -- 172.21.15.32:0/2425657904 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7eff641f9220 con 0x7eff6415e170 2024-08-03T21:01:16.753 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:16.747+0000 7eff5b400640 1 -- 172.21.15.32:0/2425657904 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff5c0052d0 con 0x7eff6415e170 2024-08-03T21:01:16.753 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:16.747+0000 7eff69400640 1 -- 172.21.15.32:0/2425657904 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7eff641f54a0 con 0x7eff6415e170 2024-08-03T21:01:16.753 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:16.748+0000 7eff69400640 1 -- 172.21.15.32:0/2425657904 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7eff6406f310 con 0x7eff6415e170 2024-08-03T21:01:16.754 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:16.749+0000 7eff5b400640 1 -- 172.21.15.32:0/2425657904 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7eff5c005680 con 0x7eff6415e170 2024-08-03T21:01:16.754 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:16.750+0000 7eff5b400640 1 --2- 172.21.15.32:0/2425657904 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7eff40077040 0x7eff40079550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:01:16.754 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:16.750+0000 7eff5b400640 1 -- 172.21.15.32:0/2425657904 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(136..136 src has 1..136) ==== 5795+0+0 (secure 0 0 0) 0x7eff5c098710 con 0x7eff6415e170 2024-08-03T21:01:16.754 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:16.751+0000 7eff62800640 1 --2- 172.21.15.32:0/2425657904 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7eff40077040 0x7eff40079550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:01:16.754 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:16.751+0000 7eff62800640 1 --2- 172.21.15.32:0/2425657904 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7eff40077040 0x7eff40079550 secure :-1 s=READY pgs=894 cs=0 l=1 rev1=1 crypto rx=0x7eff54007d60 tx=0x7eff54007cf0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T21:01:16.755 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:16.755+0000 7eff5b400640 1 -- 172.21.15.32:0/2425657904 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7eff5c0612f0 con 0x7eff6415e170 2024-08-03T21:01:17.006 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:17.006+0000 7eff69400640 1 -- 172.21.15.32:0/2425657904 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7eff6406fe80 con 0x7eff6415e170 2024-08-03T21:01:17.007 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:17.006+0000 7eff5b400640 1 -- 172.21.15.32:0/2425657904 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 346 v346) ==== 95+0+5236 (secure 0 0 0) 0x7eff5c065840 con 0x7eff6415e170 2024-08-03T21:01:17.007 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-03T21:01:17.007 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":346,"btime":"2024-08-03T21:01:10:877273+0000","default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8519,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6836/3249267506","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6834","nonce":3249267506},{"type":"v1","addr":"172.21.15.118:6836","nonce":3249267506}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":334},{"gid":8522,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6837/408542078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6835","nonce":408542078},{"type":"v1","addr":"172.21.15.118:6837","nonce":408542078}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":335},{"gid":8648,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6835/1693401067","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6833","nonce":1693401067},{"type":"v1","addr":"172.21.15.32:6835","nonce":1693401067}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":342}],"filesystems":[{"mdsmap":{"epoch":346,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-08-03T21:00:49.742850+0000","modified":"2024-08-03T21:01:10.877262+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":136,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":9589},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9589":{"gid":9589,"name":"c","rank":0,"incarnation":343,"state":"up:active","state_seq":9,"addr":"172.21.15.32:6834/3440267920","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":3440267920},{"type":"v1","addr":"172.21.15.32:6834","nonce":3440267920}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9589,"qdb_cluster":[9589]},"id":7}]} 2024-08-03T21:01:17.008 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 346 2024-08-03T21:01:17.009 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:17.009+0000 7eff69400640 1 -- 172.21.15.32:0/2425657904 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7eff40077040 msgr2=0x7eff40079550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:01:17.009 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:17.009+0000 7eff69400640 1 --2- 172.21.15.32:0/2425657904 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7eff40077040 0x7eff40079550 secure :-1 s=READY pgs=894 cs=0 l=1 rev1=1 crypto rx=0x7eff54007d60 tx=0x7eff54007cf0 comp rx=0 tx=0).stop 2024-08-03T21:01:17.010 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:17.009+0000 7eff69400640 1 -- 172.21.15.32:0/2425657904 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7eff6415e170 msgr2=0x7eff641f88f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:01:17.010 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:17.009+0000 7eff69400640 1 --2- 172.21.15.32:0/2425657904 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7eff6415e170 0x7eff641f88f0 secure :-1 s=READY pgs=2239 cs=0 l=1 rev1=1 crypto rx=0x7eff5c00c730 tx=0x7eff5c00cbf0 comp rx=0 tx=0).stop 2024-08-03T21:01:17.010 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:17.009+0000 7eff69400640 1 -- 172.21.15.32:0/2425657904 shutdown_connections 2024-08-03T21:01:17.010 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:17.009+0000 7eff69400640 1 --2- 172.21.15.32:0/2425657904 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7eff40077040 0x7eff40079550 unknown :-1 s=CLOSED pgs=894 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:01:17.010 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:17.009+0000 7eff69400640 1 --2- 172.21.15.32:0/2425657904 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7eff6415e170 0x7eff641f88f0 unknown :-1 s=CLOSED pgs=2239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:01:17.010 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:17.009+0000 7eff69400640 1 --2- 172.21.15.32:0/2425657904 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7eff640b4de0 0x7eff6414ec90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:01:17.010 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:17.009+0000 7eff69400640 1 --2- 172.21.15.32:0/2425657904 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7eff640708c0 0x7eff641f5ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:01:17.010 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:17.009+0000 7eff69400640 1 -- 172.21.15.32:0/2425657904 >> 172.21.15.32:0/2425657904 conn(0x7eff640b1bd0 msgr2=0x7eff6415bc00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T21:01:17.010 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:17.009+0000 7eff69400640 1 -- 172.21.15.32:0/2425657904 shutdown_connections 2024-08-03T21:01:17.010 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:17.009+0000 7eff69400640 1 -- 172.21.15.32:0/2425657904 wait complete. 2024-08-03T21:01:17.033 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-08-03T21:01:17.263 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:17.262+0000 7f1f1fc00640 1 -- 172.21.15.32:0/3945696194 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f1f180b40b0 msgr2=0x7f1f1806fb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:01:17.263 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:17.262+0000 7f1f1fc00640 1 --2- 172.21.15.32:0/3945696194 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f1f180b40b0 0x7f1f1806fb30 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f1f04009830 tx=0x7f1f0402a510 comp rx=0 tx=0).stop 2024-08-03T21:01:17.263 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:17.262+0000 7f1f1fc00640 1 -- 172.21.15.32:0/3945696194 shutdown_connections 2024-08-03T21:01:17.263 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:17.262+0000 7f1f1fc00640 1 --2- 172.21.15.32:0/3945696194 >> v2:172.21.15.118:3300/0 conn(0x7f1f180ade80 0x7f1f180ae290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:01:17.263 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:17.262+0000 7f1f1fc00640 1 --2- 172.21.15.32:0/3945696194 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f1f180b40b0 0x7f1f1806fb30 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:01:17.263 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:17.262+0000 7f1f1fc00640 1 -- 172.21.15.32:0/3945696194 >> 172.21.15.32:0/3945696194 conn(0x7f1f180b1bd0 msgr2=0x7f1f180b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T21:01:17.263 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:17.262+0000 7f1f1fc00640 1 -- 172.21.15.32:0/3945696194 shutdown_connections 2024-08-03T21:01:17.263 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:17.262+0000 7f1f1fc00640 1 -- 172.21.15.32:0/3945696194 wait complete. 2024-08-03T21:01:17.264 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:17.264+0000 7f1f1fc00640 1 Processor -- start 2024-08-03T21:01:17.264 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:17.264+0000 7f1f1fc00640 1 -- start start 2024-08-03T21:01:17.266 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:17.266+0000 7f1f1fc00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f1f1806f720 0x7f1f18148a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:01:17.266 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:17.266+0000 7f1f1fc00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1f180ade80 0x7f1f181f59a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:01:17.267 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:17.266+0000 7f1f1fc00640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f1f180b40b0 0x7f1f180cc970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:01:17.267 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:17.266+0000 7f1f1fc00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f1f180aebe0 con 0x7f1f180ade80 2024-08-03T21:01:17.267 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:17.266+0000 7f1f1fc00640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f1f180aea60 con 0x7f1f1806f720 2024-08-03T21:01:17.267 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:17.266+0000 7f1f1fc00640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7f1f180aed60 con 0x7f1f180b40b0 2024-08-03T21:01:17.267 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:17.266+0000 7f1f17e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1f180ade80 0x7f1f181f59a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:01:17.267 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:17.266+0000 7f1f17e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1f180ade80 0x7f1f181f59a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:58870/0 (socket says 172.21.15.32:58870) 2024-08-03T21:01:17.267 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:17.266+0000 7f1f17e00640 1 -- 172.21.15.32:0/3568793727 learned_addr learned my addr 172.21.15.32:0/3568793727 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T21:01:17.267 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:17.266+0000 7f1f1d200640 1 --2- 172.21.15.32:0/3568793727 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f1f1806f720 0x7f1f18148a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:01:17.267 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:17.266+0000 7f1f1dc00640 1 --2- 172.21.15.32:0/3568793727 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f1f180b40b0 0x7f1f180cc970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:01:17.268 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:17.266+0000 7f1f17e00640 1 -- 172.21.15.32:0/3568793727 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f1f180b40b0 msgr2=0x7f1f180cc970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:01:17.268 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:17.266+0000 7f1f17e00640 1 --2- 172.21.15.32:0/3568793727 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f1f180b40b0 0x7f1f180cc970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:01:17.268 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:17.266+0000 7f1f17e00640 1 -- 172.21.15.32:0/3568793727 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f1f1806f720 msgr2=0x7f1f18148a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:01:17.268 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:17.266+0000 7f1f17e00640 1 --2- 172.21.15.32:0/3568793727 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f1f1806f720 0x7f1f18148a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:01:17.269 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:17.267+0000 7f1f17e00640 1 -- 172.21.15.32:0/3568793727 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1f18148f50 con 0x7f1f180ade80 2024-08-03T21:01:17.269 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:17.267+0000 7f1f17e00640 1 --2- 172.21.15.32:0/3568793727 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1f180ade80 0x7f1f181f59a0 secure :-1 s=READY pgs=2240 cs=0 l=1 rev1=1 crypto rx=0x7f1f04000c00 tx=0x7f1f04033c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T21:01:17.269 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:17.267+0000 7f1f14c00640 1 -- 172.21.15.32:0/3568793727 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f04031d60 con 0x7f1f180ade80 2024-08-03T21:01:17.269 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:17.267+0000 7f1f14c00640 1 -- 172.21.15.32:0/3568793727 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f1f04033e10 con 0x7f1f180ade80 2024-08-03T21:01:17.269 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:17.267+0000 7f1f1fc00640 1 -- 172.21.15.32:0/3568793727 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1f181f63a0 con 0x7f1f180ade80 2024-08-03T21:01:17.269 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:17.267+0000 7f1f14c00640 1 -- 172.21.15.32:0/3568793727 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f04033310 con 0x7f1f180ade80 2024-08-03T21:01:17.269 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:17.267+0000 7f1f1fc00640 1 -- 172.21.15.32:0/3568793727 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1f181577e0 con 0x7f1f180ade80 2024-08-03T21:01:17.269 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:17.268+0000 7f1f1fc00640 1 -- 172.21.15.32:0/3568793727 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1f180ae290 con 0x7f1f180ade80 2024-08-03T21:01:17.270 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:17.270+0000 7f1f14c00640 1 -- 172.21.15.32:0/3568793727 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f1f040334b0 con 0x7f1f180ade80 2024-08-03T21:01:17.271 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:17.270+0000 7f1f14c00640 1 --2- 172.21.15.32:0/3568793727 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f1ef8077010 0x7f1ef8079520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:01:17.271 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:17.271+0000 7f1f14c00640 1 -- 172.21.15.32:0/3568793727 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(136..136 src has 1..136) ==== 5795+0+0 (secure 0 0 0) 0x7f1f040b9c50 con 0x7f1f180ade80 2024-08-03T21:01:17.271 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:17.271+0000 7f1f1d200640 1 --2- 172.21.15.32:0/3568793727 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f1ef8077010 0x7f1ef8079520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:01:17.272 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:17.272+0000 7f1f1d200640 1 --2- 172.21.15.32:0/3568793727 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f1ef8077010 0x7f1ef8079520 secure :-1 s=READY pgs=895 cs=0 l=1 rev1=1 crypto rx=0x7f1f0c001720 tx=0x7f1f0c009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T21:01:17.275 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:17.275+0000 7f1f14c00640 1 -- 172.21.15.32:0/3568793727 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7f1f18148f50 con 0x7f1f180ade80 2024-08-03T21:01:17.482 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:17.482+0000 7f1f1fc00640 1 -- 172.21.15.32:0/3568793727 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f1f18155060 con 0x7f1f180ade80 2024-08-03T21:01:17.483 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:17.483+0000 7f1f14c00640 1 -- 172.21.15.32:0/3568793727 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v136) ==== 74+0+16688 (secure 0 0 0) 0x7f1f0403f030 con 0x7f1f180ade80 2024-08-03T21:01:17.484 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-03T21:01:17.484 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":136,"fsid":"40ab141a-01d7-49b9-ae3c-2edec9876e95","created":"2024-08-03T20:47:14.957362+0000","modified":"2024-08-03T21:01:06.854411+0000","last_up_change":"2024-08-03T20:47:28.874451+0000","last_in_change":"2024-08-03T20:47:20.079442+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":15,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-08-03T20:47:28.261300+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":14,"pool_name":"cephfs_metadata","create_time":"2024-08-03T21:00:47.705081+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"134","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":15,"pool_name":"cephfs_data","create_time":"2024-08-03T21:00:48.703409+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"135","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5a83e759-9035-438b-889a-0a6af1f0a0d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":133,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6816","nonce":3066156116},{"type":"v1","addr":"172.21.15.32:6817","nonce":3066156116}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6818","nonce":3066156116},{"type":"v1","addr":"172.21.15.32:6819","nonce":3066156116}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6822","nonce":3066156116},{"type":"v1","addr":"172.21.15.32:6823","nonce":3066156116}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6820","nonce":3066156116},{"type":"v1","addr":"172.21.15.32:6821","nonce":3066156116}]},"public_addr":"172.21.15.32:6817/3066156116","cluster_addr":"172.21.15.32:6819/3066156116","heartbeat_back_addr":"172.21.15.32:6823/3066156116","heartbeat_front_addr":"172.21.15.32:6821/3066156116","state":["exists","up"]},{"osd":1,"uuid":"d7c5c210-da4f-49f9-b9e7-1dfb02daa4bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":133,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6824","nonce":1551241759},{"type":"v1","addr":"172.21.15.32:6825","nonce":1551241759}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6826","nonce":1551241759},{"type":"v1","addr":"172.21.15.32:6827","nonce":1551241759}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6830","nonce":1551241759},{"type":"v1","addr":"172.21.15.32:6831","nonce":1551241759}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6828","nonce":1551241759},{"type":"v1","addr":"172.21.15.32:6829","nonce":1551241759}]},"public_addr":"172.21.15.32:6825/1551241759","cluster_addr":"172.21.15.32:6827/1551241759","heartbeat_back_addr":"172.21.15.32:6831/1551241759","heartbeat_front_addr":"172.21.15.32:6829/1551241759","state":["exists","up"]},{"osd":2,"uuid":"57f2e6de-a201-45c2-8981-8e56a4875ebd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":133,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6808","nonce":3806708944},{"type":"v1","addr":"172.21.15.32:6809","nonce":3806708944}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6810","nonce":3806708944},{"type":"v1","addr":"172.21.15.32:6811","nonce":3806708944}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6814","nonce":3806708944},{"type":"v1","addr":"172.21.15.32:6815","nonce":3806708944}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6812","nonce":3806708944},{"type":"v1","addr":"172.21.15.32:6813","nonce":3806708944}]},"public_addr":"172.21.15.32:6809/3806708944","cluster_addr":"172.21.15.32:6811/3806708944","heartbeat_back_addr":"172.21.15.32:6815/3806708944","heartbeat_front_addr":"172.21.15.32:6813/3806708944","state":["exists","up"]},{"osd":3,"uuid":"6439a280-da45-4148-8130-f16323935ff3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":133,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6800","nonce":9088303},{"type":"v1","addr":"172.21.15.32:6801","nonce":9088303}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6802","nonce":9088303},{"type":"v1","addr":"172.21.15.32:6803","nonce":9088303}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6806","nonce":9088303},{"type":"v1","addr":"172.21.15.32:6807","nonce":9088303}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6804","nonce":9088303},{"type":"v1","addr":"172.21.15.32:6805","nonce":9088303}]},"public_addr":"172.21.15.32:6801/9088303","cluster_addr":"172.21.15.32:6803/9088303","heartbeat_back_addr":"172.21.15.32:6807/9088303","heartbeat_front_addr":"172.21.15.32:6805/9088303","state":["exists","up"]},{"osd":4,"uuid":"4f902eff-1e18-418d-82c9-2151c69524fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":133,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6802","nonce":864752275},{"type":"v1","addr":"172.21.15.118:6803","nonce":864752275}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6804","nonce":864752275},{"type":"v1","addr":"172.21.15.118:6805","nonce":864752275}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6808","nonce":864752275},{"type":"v1","addr":"172.21.15.118:6809","nonce":864752275}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6806","nonce":864752275},{"type":"v1","addr":"172.21.15.118:6807","nonce":864752275}]},"public_addr":"172.21.15.118:6803/864752275","cluster_addr":"172.21.15.118:6805/864752275","heartbeat_back_addr":"172.21.15.118:6809/864752275","heartbeat_front_addr":"172.21.15.118:6807/864752275","state":["exists","up"]},{"osd":5,"uuid":"bfd9feac-dd06-458c-91b9-af7f175f612c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":133,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6810","nonce":2724891770},{"type":"v1","addr":"172.21.15.118:6811","nonce":2724891770}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6812","nonce":2724891770},{"type":"v1","addr":"172.21.15.118:6813","nonce":2724891770}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6816","nonce":2724891770},{"type":"v1","addr":"172.21.15.118:6817","nonce":2724891770}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6814","nonce":2724891770},{"type":"v1","addr":"172.21.15.118:6815","nonce":2724891770}]},"public_addr":"172.21.15.118:6811/2724891770","cluster_addr":"172.21.15.118:6813/2724891770","heartbeat_back_addr":"172.21.15.118:6817/2724891770","heartbeat_front_addr":"172.21.15.118:6815/2724891770","state":["exists","up"]},{"osd":6,"uuid":"5ed5a279-9d8e-416d-bbc7-7edd3e6d5705","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":133,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6825","nonce":801947386},{"type":"v1","addr":"172.21.15.118:6827","nonce":801947386}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6828","nonce":801947386},{"type":"v1","addr":"172.21.15.118:6829","nonce":801947386}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6832","nonce":801947386},{"type":"v1","addr":"172.21.15.118:6833","nonce":801947386}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6830","nonce":801947386},{"type":"v1","addr":"172.21.15.118:6831","nonce":801947386}]},"public_addr":"172.21.15.118:6827/801947386","cluster_addr":"172.21.15.118:6829/801947386","heartbeat_back_addr":"172.21.15.118:6833/801947386","heartbeat_front_addr":"172.21.15.118:6831/801947386","state":["exists","up"]},{"osd":7,"uuid":"f2086c47-875e-4451-a436-2008d1ea16af","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":133,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6818","nonce":653000048},{"type":"v1","addr":"172.21.15.118:6819","nonce":653000048}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6820","nonce":653000048},{"type":"v1","addr":"172.21.15.118:6821","nonce":653000048}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6824","nonce":653000048},{"type":"v1","addr":"172.21.15.118:6826","nonce":653000048}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6822","nonce":653000048},{"type":"v1","addr":"172.21.15.118:6823","nonce":653000048}]},"public_addr":"172.21.15.118:6819/653000048","cluster_addr":"172.21.15.118:6821/653000048","heartbeat_back_addr":"172.21.15.118:6826/653000048","heartbeat_front_addr":"172.21.15.118:6823/653000048","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-03T20:47:25.491994+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-03T20:47:25.587358+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-03T20:47:25.454175+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-03T20:47:25.377734+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-03T20:47:25.412351+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-03T20:47:25.499815+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-03T20:47:25.575044+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-03T20:47:25.552383+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.32:6835/4032603982":"2024-08-04T21:01:06.854369+0000","172.21.15.32:6833/4032603982":"2024-08-04T21:01:06.854369+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-08-03T21:01:17.486 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:17.485+0000 7f1f1fc00640 1 -- 172.21.15.32:0/3568793727 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f1ef8077010 msgr2=0x7f1ef8079520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:01:17.486 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:17.485+0000 7f1f1fc00640 1 --2- 172.21.15.32:0/3568793727 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f1ef8077010 0x7f1ef8079520 secure :-1 s=READY pgs=895 cs=0 l=1 rev1=1 crypto rx=0x7f1f0c001720 tx=0x7f1f0c009040 comp rx=0 tx=0).stop 2024-08-03T21:01:17.486 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:17.485+0000 7f1f1fc00640 1 -- 172.21.15.32:0/3568793727 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1f180ade80 msgr2=0x7f1f181f59a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:01:17.486 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:17.485+0000 7f1f1fc00640 1 --2- 172.21.15.32:0/3568793727 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1f180ade80 0x7f1f181f59a0 secure :-1 s=READY pgs=2240 cs=0 l=1 rev1=1 crypto rx=0x7f1f04000c00 tx=0x7f1f04033c20 comp rx=0 tx=0).stop 2024-08-03T21:01:17.486 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:17.485+0000 7f1f1fc00640 1 -- 172.21.15.32:0/3568793727 shutdown_connections 2024-08-03T21:01:17.486 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:17.485+0000 7f1f1fc00640 1 --2- 172.21.15.32:0/3568793727 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f1ef8077010 0x7f1ef8079520 unknown :-1 s=CLOSED pgs=895 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:01:17.487 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:17.485+0000 7f1f1fc00640 1 --2- 172.21.15.32:0/3568793727 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f1f180b40b0 0x7f1f180cc970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:01:17.487 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:17.485+0000 7f1f1fc00640 1 --2- 172.21.15.32:0/3568793727 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1f180ade80 0x7f1f181f59a0 unknown :-1 s=CLOSED pgs=2240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:01:17.487 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:17.485+0000 7f1f1fc00640 1 --2- 172.21.15.32:0/3568793727 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f1f1806f720 0x7f1f18148a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:01:17.487 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:17.485+0000 7f1f1fc00640 1 -- 172.21.15.32:0/3568793727 >> 172.21.15.32:0/3568793727 conn(0x7f1f180b1bd0 msgr2=0x7f1f1815c7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T21:01:17.487 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:17.485+0000 7f1f1fc00640 1 -- 172.21.15.32:0/3568793727 shutdown_connections 2024-08-03T21:01:17.487 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:17.486+0000 7f1f1fc00640 1 -- 172.21.15.32:0/3568793727 wait complete. 2024-08-03T21:01:17.508 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-08-03T21:01:17.508 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-08-03T21:01:17.738 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:17.737+0000 7f5764000640 1 -- 172.21.15.32:0/1158736224 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5744004b40 msgr2=0x7f5744004fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:01:17.738 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:17.737+0000 7f5764000640 1 --2- 172.21.15.32:0/1158736224 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5744004b40 0x7f5744004fb0 secure :-1 s=READY pgs=2242 cs=0 l=1 rev1=1 crypto rx=0x7f575000c610 tx=0x7f5744006940 comp rx=0 tx=0).stop 2024-08-03T21:01:17.738 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:17.737+0000 7f5762000640 1 -- 172.21.15.32:0/1158736224 reap_dead start 2024-08-03T21:01:17.738 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:17.737+0000 7f5764000640 1 -- 172.21.15.32:0/1158736224 shutdown_connections 2024-08-03T21:01:17.738 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:17.737+0000 7f5764000640 1 -- 172.21.15.32:0/1158736224 >> 172.21.15.32:0/1158736224 conn(0x7f575c0b1bd0 msgr2=0x7f575c0b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T21:01:17.738 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:17.737+0000 7f5764000640 1 -- 172.21.15.32:0/1158736224 shutdown_connections 2024-08-03T21:01:17.739 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:17.737+0000 7f5764000640 1 -- 172.21.15.32:0/1158736224 wait complete. 2024-08-03T21:01:17.740 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:17.739+0000 7f5764000640 1 Processor -- start 2024-08-03T21:01:17.740 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:17.740+0000 7f5764000640 1 -- start start 2024-08-03T21:01:17.742 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:17.741+0000 7f5764000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5744004b40 0x7f575c1e45f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:01:17.742 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:17.741+0000 7f5764000640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f575c1f5dc0 0x7f575c1f8c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:01:17.742 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:17.741+0000 7f5764000640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f575c1f91d0 0x7f575c1595c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:01:17.742 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:17.741+0000 7f5764000640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f575c15ea00 con 0x7f5744004b40 2024-08-03T21:01:17.743 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:17.741+0000 7f5764000640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f575c15e880 con 0x7f575c1f5dc0 2024-08-03T21:01:17.743 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:17.741+0000 7f5764000640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7f575c0708c0 con 0x7f575c1f91d0 2024-08-03T21:01:17.743 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:17.741+0000 7f5761600640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5744004b40 0x7f575c1e45f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:01:17.743 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:17.741+0000 7f5761600640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5744004b40 0x7f575c1e45f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:58888/0 (socket says 172.21.15.32:58888) 2024-08-03T21:01:17.743 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:17.741+0000 7f5761600640 1 -- 172.21.15.32:0/780321584 learned_addr learned my addr 172.21.15.32:0/780321584 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T21:01:17.743 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:17.741+0000 7f5760c00640 1 --2- 172.21.15.32:0/780321584 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f575c1f5dc0 0x7f575c1f8c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:01:17.743 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:17.741+0000 7f5762000640 1 --2- 172.21.15.32:0/780321584 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f575c1f91d0 0x7f575c1595c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:01:17.743 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:17.742+0000 7f5761600640 1 -- 172.21.15.32:0/780321584 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f575c1f91d0 msgr2=0x7f575c1595c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:01:17.743 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:17.742+0000 7f5761600640 1 --2- 172.21.15.32:0/780321584 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f575c1f91d0 0x7f575c1595c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:01:17.743 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:17.742+0000 7f5761600640 1 -- 172.21.15.32:0/780321584 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f575c1f5dc0 msgr2=0x7f575c1f8c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:01:17.743 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:17.742+0000 7f5761600640 1 --2- 172.21.15.32:0/780321584 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f575c1f5dc0 0x7f575c1f8c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:01:17.743 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:17.742+0000 7f5761600640 1 -- 172.21.15.32:0/780321584 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f575c1f5060 con 0x7f5744004b40 2024-08-03T21:01:17.744 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:17.742+0000 7f5761600640 1 --2- 172.21.15.32:0/780321584 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5744004b40 0x7f575c1e45f0 secure :-1 s=READY pgs=2243 cs=0 l=1 rev1=1 crypto rx=0x7f575000c730 tx=0x7f5750006310 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T21:01:17.744 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:17.742+0000 7f5759600640 1 -- 172.21.15.32:0/780321584 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5750039070 con 0x7f5744004b40 2024-08-03T21:01:17.744 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:17.743+0000 7f5759600640 1 -- 172.21.15.32:0/780321584 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f5750006710 con 0x7f5744004b40 2024-08-03T21:01:17.744 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:17.743+0000 7f5764000640 1 -- 172.21.15.32:0/780321584 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f575c1e4d50 con 0x7f5744004b40 2024-08-03T21:01:17.744 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:17.743+0000 7f5759600640 1 -- 172.21.15.32:0/780321584 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f575002bba0 con 0x7f5744004b40 2024-08-03T21:01:17.745 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:17.743+0000 7f5764000640 1 -- 172.21.15.32:0/780321584 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f575c1f55c0 con 0x7f5744004b40 2024-08-03T21:01:17.745 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:17.744+0000 7f5764000640 1 -- 172.21.15.32:0/780321584 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f575c0733f0 con 0x7f5744004b40 2024-08-03T21:01:17.745 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:17.745+0000 7f5759600640 1 -- 172.21.15.32:0/780321584 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f5750044020 con 0x7f5744004b40 2024-08-03T21:01:17.747 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:17.746+0000 7f5759600640 1 --2- 172.21.15.32:0/780321584 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f573c077010 0x7f573c079520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:01:17.747 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:17.747+0000 7f5759600640 1 -- 172.21.15.32:0/780321584 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(136..136 src has 1..136) ==== 5795+0+0 (secure 0 0 0) 0x7f57500b8900 con 0x7f5744004b40 2024-08-03T21:01:17.748 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:17.747+0000 7f5760c00640 1 --2- 172.21.15.32:0/780321584 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f573c077010 0x7f573c079520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:01:17.748 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:17.748+0000 7f5760c00640 1 --2- 172.21.15.32:0/780321584 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f573c077010 0x7f573c079520 secure :-1 s=READY pgs=896 cs=0 l=1 rev1=1 crypto rx=0x7f575c1e7fc0 tx=0x7f5748006b70 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T21:01:17.751 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:17.751+0000 7f5759600640 1 -- 172.21.15.32:0/780321584 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7f575c1f5060 con 0x7f5744004b40 2024-08-03T21:01:18.000 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:18.000+0000 7f5764000640 1 -- 172.21.15.32:0/780321584 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f575c0af7a0 con 0x7f5744004b40 2024-08-03T21:01:18.001 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:18.000+0000 7f5759600640 1 -- 172.21.15.32:0/780321584 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 346 v346) ==== 95+0+5236 (secure 0 0 0) 0x7f575c0b51f0 con 0x7f5744004b40 2024-08-03T21:01:18.001 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-03T21:01:18.001 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":346,"btime":"2024-08-03T21:01:10:877273+0000","default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8519,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6836/3249267506","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6834","nonce":3249267506},{"type":"v1","addr":"172.21.15.118:6836","nonce":3249267506}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":334},{"gid":8522,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6837/408542078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6835","nonce":408542078},{"type":"v1","addr":"172.21.15.118:6837","nonce":408542078}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":335},{"gid":8648,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6835/1693401067","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6833","nonce":1693401067},{"type":"v1","addr":"172.21.15.32:6835","nonce":1693401067}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":342}],"filesystems":[{"mdsmap":{"epoch":346,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-08-03T21:00:49.742850+0000","modified":"2024-08-03T21:01:10.877262+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":136,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":9589},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9589":{"gid":9589,"name":"c","rank":0,"incarnation":343,"state":"up:active","state_seq":9,"addr":"172.21.15.32:6834/3440267920","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":3440267920},{"type":"v1","addr":"172.21.15.32:6834","nonce":3440267920}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9589,"qdb_cluster":[9589]},"id":7}]} 2024-08-03T21:01:18.001 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 346 2024-08-03T21:01:18.004 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:18.003+0000 7f5764000640 1 -- 172.21.15.32:0/780321584 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f573c077010 msgr2=0x7f573c079520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:01:18.004 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:18.003+0000 7f5764000640 1 --2- 172.21.15.32:0/780321584 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f573c077010 0x7f573c079520 secure :-1 s=READY pgs=896 cs=0 l=1 rev1=1 crypto rx=0x7f575c1e7fc0 tx=0x7f5748006b70 comp rx=0 tx=0).stop 2024-08-03T21:01:18.004 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:18.003+0000 7f5764000640 1 -- 172.21.15.32:0/780321584 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5744004b40 msgr2=0x7f575c1e45f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:01:18.004 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:18.003+0000 7f5764000640 1 --2- 172.21.15.32:0/780321584 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5744004b40 0x7f575c1e45f0 secure :-1 s=READY pgs=2243 cs=0 l=1 rev1=1 crypto rx=0x7f575000c730 tx=0x7f5750006310 comp rx=0 tx=0).stop 2024-08-03T21:01:18.004 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:18.003+0000 7f5764000640 1 -- 172.21.15.32:0/780321584 shutdown_connections 2024-08-03T21:01:18.004 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:18.003+0000 7f5764000640 1 --2- 172.21.15.32:0/780321584 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f573c077010 0x7f573c079520 unknown :-1 s=CLOSED pgs=896 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:01:18.004 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:18.003+0000 7f5764000640 1 --2- 172.21.15.32:0/780321584 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f575c1f5dc0 0x7f575c1f8c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:01:18.004 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:18.003+0000 7f5764000640 1 --2- 172.21.15.32:0/780321584 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f575c1f91d0 0x7f575c1595c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:01:18.005 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:18.003+0000 7f5764000640 1 --2- 172.21.15.32:0/780321584 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5744004b40 0x7f575c1e45f0 unknown :-1 s=CLOSED pgs=2243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:01:18.005 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:18.003+0000 7f5764000640 1 -- 172.21.15.32:0/780321584 >> 172.21.15.32:0/780321584 conn(0x7f575c0b1bd0 msgr2=0x7f575c0b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T21:01:18.005 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:18.003+0000 7f5764000640 1 -- 172.21.15.32:0/780321584 shutdown_connections 2024-08-03T21:01:18.005 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:18.003+0000 7f5764000640 1 -- 172.21.15.32:0/780321584 wait complete. 2024-08-03T21:01:18.031 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-08-03T21:01:18.266 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:18.258+0000 7f7ea2200640 1 -- 172.21.15.32:0/2508417500 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7e9c144f90 msgr2=0x7f7e9c074570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:01:18.267 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:18.258+0000 7f7ea2200640 1 --2- 172.21.15.32:0/2508417500 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7e9c144f90 0x7f7e9c074570 secure :-1 s=READY pgs=2244 cs=0 l=1 rev1=1 crypto rx=0x7f7e84005fd0 tx=0x7f7e8402a530 comp rx=0 tx=0).stop 2024-08-03T21:01:18.267 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:18.258+0000 7f7ea2200640 1 -- 172.21.15.32:0/2508417500 shutdown_connections 2024-08-03T21:01:18.267 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:18.258+0000 7f7ea2200640 1 --2- 172.21.15.32:0/2508417500 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f7e9c15fff0 0x7f7e9c160400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:01:18.268 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:18.258+0000 7f7ea2200640 1 --2- 172.21.15.32:0/2508417500 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7e9c144f90 0x7f7e9c074570 unknown :-1 s=CLOSED pgs=2244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:01:18.268 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:18.258+0000 7f7ea2200640 1 -- 172.21.15.32:0/2508417500 >> 172.21.15.32:0/2508417500 conn(0x7f7e9c0b1bd0 msgr2=0x7f7e9c0b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T21:01:18.268 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:18.258+0000 7f7ea2200640 1 -- 172.21.15.32:0/2508417500 shutdown_connections 2024-08-03T21:01:18.268 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:18.258+0000 7f7ea2200640 1 -- 172.21.15.32:0/2508417500 wait complete. 2024-08-03T21:01:18.268 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:18.260+0000 7f7ea2200640 1 Processor -- start 2024-08-03T21:01:18.268 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:18.260+0000 7f7ea2200640 1 -- start start 2024-08-03T21:01:18.269 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:18.262+0000 7f7ea2200640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f7e9c06e930 0x7f7e9c15bd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:01:18.269 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:18.262+0000 7f7ea2200640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f7e9c144f90 0x7f7e9c15ca30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:01:18.269 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:18.262+0000 7f7ea2200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7e9c15fff0 0x7f7e9c1e67d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:01:18.269 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:18.262+0000 7f7ea2200640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f7e9c0cf840 con 0x7f7e9c15fff0 2024-08-03T21:01:18.269 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:18.262+0000 7f7ea2200640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f7e9c145f30 con 0x7f7e9c06e930 2024-08-03T21:01:18.269 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:18.262+0000 7f7ea2200640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7f7e9c0cf9c0 con 0x7f7e9c144f90 2024-08-03T21:01:18.270 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:18.262+0000 7f7e9be00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7e9c15fff0 0x7f7e9c1e67d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:01:18.270 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:18.262+0000 7f7e9be00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7e9c15fff0 0x7f7e9c1e67d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:58902/0 (socket says 172.21.15.32:58902) 2024-08-03T21:01:18.270 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:18.262+0000 7f7e9be00640 1 -- 172.21.15.32:0/1618188700 learned_addr learned my addr 172.21.15.32:0/1618188700 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T21:01:18.270 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:18.262+0000 7f7e93e00640 1 --2- 172.21.15.32:0/1618188700 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f7e9c144f90 0x7f7e9c15ca30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:01:18.270 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:18.262+0000 7f7e9b400640 1 --2- 172.21.15.32:0/1618188700 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f7e9c06e930 0x7f7e9c15bd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:01:18.270 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:18.263+0000 7f7e9be00640 1 -- 172.21.15.32:0/1618188700 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f7e9c144f90 msgr2=0x7f7e9c15ca30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:01:18.271 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:18.263+0000 7f7e9be00640 1 --2- 172.21.15.32:0/1618188700 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f7e9c144f90 0x7f7e9c15ca30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:01:18.271 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:18.263+0000 7f7e9be00640 1 -- 172.21.15.32:0/1618188700 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f7e9c06e930 msgr2=0x7f7e9c15bd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:01:18.271 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:18.263+0000 7f7e9be00640 1 --2- 172.21.15.32:0/1618188700 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f7e9c06e930 0x7f7e9c15bd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:01:18.271 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:18.263+0000 7f7e9be00640 1 -- 172.21.15.32:0/1618188700 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7e9c1f6260 con 0x7f7e9c15fff0 2024-08-03T21:01:18.271 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:18.263+0000 7f7e9be00640 1 --2- 172.21.15.32:0/1618188700 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7e9c15fff0 0x7f7e9c1e67d0 secure :-1 s=READY pgs=2245 cs=0 l=1 rev1=1 crypto rx=0x7f7e9400e960 tx=0x7f7e9400ee20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T21:01:18.271 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:18.263+0000 7f7e93400640 1 -- 172.21.15.32:0/1618188700 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e94004d20 con 0x7f7e9c15fff0 2024-08-03T21:01:18.272 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:18.263+0000 7f7e93400640 1 -- 172.21.15.32:0/1618188700 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f7e9400cb20 con 0x7f7e9c15fff0 2024-08-03T21:01:18.272 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:18.263+0000 7f7ea2200640 1 -- 172.21.15.32:0/1618188700 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7e9c1e5d30 con 0x7f7e9c15fff0 2024-08-03T21:01:18.272 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:18.263+0000 7f7e93400640 1 -- 172.21.15.32:0/1618188700 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e94005020 con 0x7f7e9c15fff0 2024-08-03T21:01:18.272 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:18.263+0000 7f7ea2200640 1 -- 172.21.15.32:0/1618188700 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7e9c1e5fd0 con 0x7f7e9c15fff0 2024-08-03T21:01:18.272 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:18.264+0000 7f7ea2200640 1 -- 172.21.15.32:0/1618188700 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7e9c160ce0 con 0x7f7e9c15fff0 2024-08-03T21:01:18.272 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:18.266+0000 7f7e93400640 1 -- 172.21.15.32:0/1618188700 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f7e940051c0 con 0x7f7e9c15fff0 2024-08-03T21:01:18.274 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:18.267+0000 7f7e93400640 1 --2- 172.21.15.32:0/1618188700 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f7e78077010 0x7f7e78079520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:01:18.274 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:18.267+0000 7f7e93400640 1 -- 172.21.15.32:0/1618188700 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(136..136 src has 1..136) ==== 5795+0+0 (secure 0 0 0) 0x7f7e94099190 con 0x7f7e9c15fff0 2024-08-03T21:01:18.274 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:18.267+0000 7f7e9b400640 1 --2- 172.21.15.32:0/1618188700 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f7e78077010 0x7f7e78079520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:01:18.274 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:18.268+0000 7f7e9b400640 1 --2- 172.21.15.32:0/1618188700 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f7e78077010 0x7f7e78079520 secure :-1 s=READY pgs=897 cs=0 l=1 rev1=1 crypto rx=0x7f7e8c006fd0 tx=0x7f7e8c009060 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T21:01:18.274 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:18.271+0000 7f7e93400640 1 -- 172.21.15.32:0/1618188700 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7f7e94016030 con 0x7f7e9c15fff0 2024-08-03T21:01:18.515 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:18.515+0000 7f7ea2200640 1 -- 172.21.15.32:0/1618188700 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f7e9c070140 con 0x7f7e9c15fff0 2024-08-03T21:01:18.516 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:18.515+0000 7f7e93400640 1 -- 172.21.15.32:0/1618188700 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 346 v346) ==== 95+0+5236 (secure 0 0 0) 0x7f7e94061d60 con 0x7f7e9c15fff0 2024-08-03T21:01:18.516 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-03T21:01:18.516 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":346,"btime":"2024-08-03T21:01:10:877273+0000","default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8519,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6836/3249267506","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6834","nonce":3249267506},{"type":"v1","addr":"172.21.15.118:6836","nonce":3249267506}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":334},{"gid":8522,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6837/408542078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6835","nonce":408542078},{"type":"v1","addr":"172.21.15.118:6837","nonce":408542078}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":335},{"gid":8648,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6835/1693401067","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6833","nonce":1693401067},{"type":"v1","addr":"172.21.15.32:6835","nonce":1693401067}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":342}],"filesystems":[{"mdsmap":{"epoch":346,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-08-03T21:00:49.742850+0000","modified":"2024-08-03T21:01:10.877262+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":136,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":9589},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9589":{"gid":9589,"name":"c","rank":0,"incarnation":343,"state":"up:active","state_seq":9,"addr":"172.21.15.32:6834/3440267920","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":3440267920},{"type":"v1","addr":"172.21.15.32:6834","nonce":3440267920}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9589,"qdb_cluster":[9589]},"id":7}]} 2024-08-03T21:01:18.517 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 346 2024-08-03T21:01:18.519 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:18.518+0000 7f7ea2200640 1 -- 172.21.15.32:0/1618188700 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f7e78077010 msgr2=0x7f7e78079520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:01:18.519 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:18.518+0000 7f7ea2200640 1 --2- 172.21.15.32:0/1618188700 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f7e78077010 0x7f7e78079520 secure :-1 s=READY pgs=897 cs=0 l=1 rev1=1 crypto rx=0x7f7e8c006fd0 tx=0x7f7e8c009060 comp rx=0 tx=0).stop 2024-08-03T21:01:18.519 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:18.518+0000 7f7ea2200640 1 -- 172.21.15.32:0/1618188700 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7e9c15fff0 msgr2=0x7f7e9c1e67d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:01:18.519 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:18.518+0000 7f7ea2200640 1 --2- 172.21.15.32:0/1618188700 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7e9c15fff0 0x7f7e9c1e67d0 secure :-1 s=READY pgs=2245 cs=0 l=1 rev1=1 crypto rx=0x7f7e9400e960 tx=0x7f7e9400ee20 comp rx=0 tx=0).stop 2024-08-03T21:01:18.519 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:18.518+0000 7f7ea2200640 1 -- 172.21.15.32:0/1618188700 shutdown_connections 2024-08-03T21:01:18.519 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:18.518+0000 7f7ea2200640 1 --2- 172.21.15.32:0/1618188700 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f7e78077010 0x7f7e78079520 unknown :-1 s=CLOSED pgs=897 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:01:18.520 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:18.518+0000 7f7ea2200640 1 --2- 172.21.15.32:0/1618188700 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7e9c15fff0 0x7f7e9c1e67d0 unknown :-1 s=CLOSED pgs=2245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:01:18.520 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:18.518+0000 7f7ea2200640 1 --2- 172.21.15.32:0/1618188700 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f7e9c144f90 0x7f7e9c15ca30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:01:18.520 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:18.518+0000 7f7ea2200640 1 --2- 172.21.15.32:0/1618188700 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f7e9c06e930 0x7f7e9c15bd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:01:18.520 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:18.518+0000 7f7ea2200640 1 -- 172.21.15.32:0/1618188700 >> 172.21.15.32:0/1618188700 conn(0x7f7e9c0b1bd0 msgr2=0x7f7e9c0b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T21:01:18.520 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:18.518+0000 7f7ea2200640 1 -- 172.21.15.32:0/1618188700 shutdown_connections 2024-08-03T21:01:18.520 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:18.518+0000 7f7ea2200640 1 -- 172.21.15.32:0/1618188700 wait complete. 2024-08-03T21:01:18.541 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-08-03T21:01:18.768 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:18.767+0000 7f7a91a00640 1 -- 172.21.15.32:0/3908363511 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7a8c0b4de0 msgr2=0x7f7a8c1515c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:01:18.768 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:18.767+0000 7f7a91a00640 1 --2- 172.21.15.32:0/3908363511 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7a8c0b4de0 0x7f7a8c1515c0 secure :-1 s=READY pgs=2247 cs=0 l=1 rev1=1 crypto rx=0x7f7a7c00a940 tx=0x7f7a7c02b850 comp rx=0 tx=0).stop 2024-08-03T21:01:18.769 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:18.767+0000 7f7a91a00640 1 -- 172.21.15.32:0/3908363511 shutdown_connections 2024-08-03T21:01:18.769 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:18.767+0000 7f7a91a00640 1 --2- 172.21.15.32:0/3908363511 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7a8c0b4de0 0x7f7a8c1515c0 unknown :-1 s=CLOSED pgs=2247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:01:18.769 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:18.767+0000 7f7a91a00640 1 -- 172.21.15.32:0/3908363511 >> 172.21.15.32:0/3908363511 conn(0x7f7a8c0b1bd0 msgr2=0x7f7a8c0b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T21:01:18.769 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:18.767+0000 7f7a91a00640 1 -- 172.21.15.32:0/3908363511 shutdown_connections 2024-08-03T21:01:18.769 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:18.768+0000 7f7a91a00640 1 -- 172.21.15.32:0/3908363511 wait complete. 2024-08-03T21:01:18.770 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:18.770+0000 7f7a91a00640 1 Processor -- start 2024-08-03T21:01:18.771 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:18.770+0000 7f7a91a00640 1 -- start start 2024-08-03T21:01:18.773 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:18.771+0000 7f7a91a00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f7a8c0733f0 0x7f7a8c1459e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:01:18.773 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:18.771+0000 7f7a91a00640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f7a8c0b4de0 0x7f7a8c0cda00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:01:18.773 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:18.771+0000 7f7a91a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7a8c14eb10 0x7f7a8c0cdf40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:01:18.773 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:18.771+0000 7f7a91a00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f7a8c073980 con 0x7f7a8c14eb10 2024-08-03T21:01:18.773 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:18.771+0000 7f7a91a00640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f7a8c073800 con 0x7f7a8c0733f0 2024-08-03T21:01:18.773 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:18.771+0000 7f7a91a00640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7f7a8c073b00 con 0x7f7a8c0b4de0 2024-08-03T21:01:18.773 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:18.772+0000 7f7a8b400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7a8c14eb10 0x7f7a8c0cdf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:01:18.773 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:18.772+0000 7f7a8b400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7a8c14eb10 0x7f7a8c0cdf40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:58920/0 (socket says 172.21.15.32:58920) 2024-08-03T21:01:18.773 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:18.772+0000 7f7a8b400640 1 -- 172.21.15.32:0/871860713 learned_addr learned my addr 172.21.15.32:0/871860713 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T21:01:18.773 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:18.772+0000 7f7a8aa00640 1 --2- 172.21.15.32:0/871860713 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f7a8c0733f0 0x7f7a8c1459e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:01:18.773 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:18.772+0000 7f7a8a000640 1 --2- 172.21.15.32:0/871860713 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f7a8c0b4de0 0x7f7a8c0cda00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:01:18.773 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:18.772+0000 7f7a8b400640 1 -- 172.21.15.32:0/871860713 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f7a8c0b4de0 msgr2=0x7f7a8c0cda00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:01:18.773 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:18.772+0000 7f7a8b400640 1 --2- 172.21.15.32:0/871860713 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f7a8c0b4de0 0x7f7a8c0cda00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:01:18.774 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:18.772+0000 7f7a8b400640 1 -- 172.21.15.32:0/871860713 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f7a8c0733f0 msgr2=0x7f7a8c1459e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:01:18.774 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:18.772+0000 7f7a8b400640 1 --2- 172.21.15.32:0/871860713 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f7a8c0733f0 0x7f7a8c1459e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:01:18.774 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:18.772+0000 7f7a8b400640 1 -- 172.21.15.32:0/871860713 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7a8c152070 con 0x7f7a8c14eb10 2024-08-03T21:01:18.774 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:18.772+0000 7f7a8aa00640 1 --2- 172.21.15.32:0/871860713 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f7a8c0733f0 0x7f7a8c1459e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-03T21:01:18.775 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:18.773+0000 7f7a8b400640 1 --2- 172.21.15.32:0/871860713 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7a8c14eb10 0x7f7a8c0cdf40 secure :-1 s=READY pgs=2248 cs=0 l=1 rev1=1 crypto rx=0x7f7a8400cb10 tx=0x7f7a8400a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T21:01:18.775 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:18.773+0000 7f7a82a00640 1 -- 172.21.15.32:0/871860713 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a84016020 con 0x7f7a8c14eb10 2024-08-03T21:01:18.775 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:18.773+0000 7f7a82a00640 1 -- 172.21.15.32:0/871860713 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f7a8400ad00 con 0x7f7a8c14eb10 2024-08-03T21:01:18.775 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:18.773+0000 7f7a91a00640 1 -- 172.21.15.32:0/871860713 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7a8c1f8c60 con 0x7f7a8c14eb10 2024-08-03T21:01:18.775 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:18.773+0000 7f7a82a00640 1 -- 172.21.15.32:0/871860713 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a84012680 con 0x7f7a8c14eb10 2024-08-03T21:01:18.775 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:18.773+0000 7f7a91a00640 1 -- 172.21.15.32:0/871860713 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7a8c1f9440 con 0x7f7a8c14eb10 2024-08-03T21:01:18.775 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:18.774+0000 7f7a91a00640 1 -- 172.21.15.32:0/871860713 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7a8c1f8ed0 con 0x7f7a8c14eb10 2024-08-03T21:01:18.776 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:18.775+0000 7f7a82a00640 1 -- 172.21.15.32:0/871860713 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f7a84019020 con 0x7f7a8c14eb10 2024-08-03T21:01:18.777 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:18.776+0000 7f7a82a00640 1 --2- 172.21.15.32:0/871860713 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f7a6c077010 0x7f7a6c079520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:01:18.777 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:18.776+0000 7f7a82a00640 1 -- 172.21.15.32:0/871860713 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(136..136 src has 1..136) ==== 5795+0+0 (secure 0 0 0) 0x7f7a84098d80 con 0x7f7a8c14eb10 2024-08-03T21:01:18.777 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:18.777+0000 7f7a8aa00640 1 --2- 172.21.15.32:0/871860713 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f7a6c077010 0x7f7a6c079520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:01:18.778 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:18.778+0000 7f7a8aa00640 1 --2- 172.21.15.32:0/871860713 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f7a6c077010 0x7f7a6c079520 secure :-1 s=READY pgs=898 cs=0 l=1 rev1=1 crypto rx=0x7f7a7c036000 tx=0x7f7a7c004730 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T21:01:18.781 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:18.781+0000 7f7a82a00640 1 -- 172.21.15.32:0/871860713 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7f7a84020030 con 0x7f7a8c14eb10 2024-08-03T21:01:18.986 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:18.986+0000 7f7a91a00640 1 -- 172.21.15.32:0/871860713 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f7a8c152240 con 0x7f7a8c14eb10 2024-08-03T21:01:18.988 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:18.987+0000 7f7a82a00640 1 -- 172.21.15.32:0/871860713 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v136) ==== 74+0+16688 (secure 0 0 0) 0x7f7a8c0b51f0 con 0x7f7a8c14eb10 2024-08-03T21:01:18.988 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-03T21:01:18.988 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":136,"fsid":"40ab141a-01d7-49b9-ae3c-2edec9876e95","created":"2024-08-03T20:47:14.957362+0000","modified":"2024-08-03T21:01:06.854411+0000","last_up_change":"2024-08-03T20:47:28.874451+0000","last_in_change":"2024-08-03T20:47:20.079442+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":15,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-08-03T20:47:28.261300+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":14,"pool_name":"cephfs_metadata","create_time":"2024-08-03T21:00:47.705081+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"134","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":15,"pool_name":"cephfs_data","create_time":"2024-08-03T21:00:48.703409+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"135","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5a83e759-9035-438b-889a-0a6af1f0a0d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":133,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6816","nonce":3066156116},{"type":"v1","addr":"172.21.15.32:6817","nonce":3066156116}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6818","nonce":3066156116},{"type":"v1","addr":"172.21.15.32:6819","nonce":3066156116}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6822","nonce":3066156116},{"type":"v1","addr":"172.21.15.32:6823","nonce":3066156116}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6820","nonce":3066156116},{"type":"v1","addr":"172.21.15.32:6821","nonce":3066156116}]},"public_addr":"172.21.15.32:6817/3066156116","cluster_addr":"172.21.15.32:6819/3066156116","heartbeat_back_addr":"172.21.15.32:6823/3066156116","heartbeat_front_addr":"172.21.15.32:6821/3066156116","state":["exists","up"]},{"osd":1,"uuid":"d7c5c210-da4f-49f9-b9e7-1dfb02daa4bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":133,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6824","nonce":1551241759},{"type":"v1","addr":"172.21.15.32:6825","nonce":1551241759}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6826","nonce":1551241759},{"type":"v1","addr":"172.21.15.32:6827","nonce":1551241759}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6830","nonce":1551241759},{"type":"v1","addr":"172.21.15.32:6831","nonce":1551241759}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6828","nonce":1551241759},{"type":"v1","addr":"172.21.15.32:6829","nonce":1551241759}]},"public_addr":"172.21.15.32:6825/1551241759","cluster_addr":"172.21.15.32:6827/1551241759","heartbeat_back_addr":"172.21.15.32:6831/1551241759","heartbeat_front_addr":"172.21.15.32:6829/1551241759","state":["exists","up"]},{"osd":2,"uuid":"57f2e6de-a201-45c2-8981-8e56a4875ebd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":133,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6808","nonce":3806708944},{"type":"v1","addr":"172.21.15.32:6809","nonce":3806708944}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6810","nonce":3806708944},{"type":"v1","addr":"172.21.15.32:6811","nonce":3806708944}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6814","nonce":3806708944},{"type":"v1","addr":"172.21.15.32:6815","nonce":3806708944}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6812","nonce":3806708944},{"type":"v1","addr":"172.21.15.32:6813","nonce":3806708944}]},"public_addr":"172.21.15.32:6809/3806708944","cluster_addr":"172.21.15.32:6811/3806708944","heartbeat_back_addr":"172.21.15.32:6815/3806708944","heartbeat_front_addr":"172.21.15.32:6813/3806708944","state":["exists","up"]},{"osd":3,"uuid":"6439a280-da45-4148-8130-f16323935ff3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":133,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6800","nonce":9088303},{"type":"v1","addr":"172.21.15.32:6801","nonce":9088303}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6802","nonce":9088303},{"type":"v1","addr":"172.21.15.32:6803","nonce":9088303}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6806","nonce":9088303},{"type":"v1","addr":"172.21.15.32:6807","nonce":9088303}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6804","nonce":9088303},{"type":"v1","addr":"172.21.15.32:6805","nonce":9088303}]},"public_addr":"172.21.15.32:6801/9088303","cluster_addr":"172.21.15.32:6803/9088303","heartbeat_back_addr":"172.21.15.32:6807/9088303","heartbeat_front_addr":"172.21.15.32:6805/9088303","state":["exists","up"]},{"osd":4,"uuid":"4f902eff-1e18-418d-82c9-2151c69524fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":133,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6802","nonce":864752275},{"type":"v1","addr":"172.21.15.118:6803","nonce":864752275}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6804","nonce":864752275},{"type":"v1","addr":"172.21.15.118:6805","nonce":864752275}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6808","nonce":864752275},{"type":"v1","addr":"172.21.15.118:6809","nonce":864752275}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6806","nonce":864752275},{"type":"v1","addr":"172.21.15.118:6807","nonce":864752275}]},"public_addr":"172.21.15.118:6803/864752275","cluster_addr":"172.21.15.118:6805/864752275","heartbeat_back_addr":"172.21.15.118:6809/864752275","heartbeat_front_addr":"172.21.15.118:6807/864752275","state":["exists","up"]},{"osd":5,"uuid":"bfd9feac-dd06-458c-91b9-af7f175f612c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":133,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6810","nonce":2724891770},{"type":"v1","addr":"172.21.15.118:6811","nonce":2724891770}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6812","nonce":2724891770},{"type":"v1","addr":"172.21.15.118:6813","nonce":2724891770}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6816","nonce":2724891770},{"type":"v1","addr":"172.21.15.118:6817","nonce":2724891770}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6814","nonce":2724891770},{"type":"v1","addr":"172.21.15.118:6815","nonce":2724891770}]},"public_addr":"172.21.15.118:6811/2724891770","cluster_addr":"172.21.15.118:6813/2724891770","heartbeat_back_addr":"172.21.15.118:6817/2724891770","heartbeat_front_addr":"172.21.15.118:6815/2724891770","state":["exists","up"]},{"osd":6,"uuid":"5ed5a279-9d8e-416d-bbc7-7edd3e6d5705","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":133,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6825","nonce":801947386},{"type":"v1","addr":"172.21.15.118:6827","nonce":801947386}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6828","nonce":801947386},{"type":"v1","addr":"172.21.15.118:6829","nonce":801947386}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6832","nonce":801947386},{"type":"v1","addr":"172.21.15.118:6833","nonce":801947386}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6830","nonce":801947386},{"type":"v1","addr":"172.21.15.118:6831","nonce":801947386}]},"public_addr":"172.21.15.118:6827/801947386","cluster_addr":"172.21.15.118:6829/801947386","heartbeat_back_addr":"172.21.15.118:6833/801947386","heartbeat_front_addr":"172.21.15.118:6831/801947386","state":["exists","up"]},{"osd":7,"uuid":"f2086c47-875e-4451-a436-2008d1ea16af","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":133,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6818","nonce":653000048},{"type":"v1","addr":"172.21.15.118:6819","nonce":653000048}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6820","nonce":653000048},{"type":"v1","addr":"172.21.15.118:6821","nonce":653000048}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6824","nonce":653000048},{"type":"v1","addr":"172.21.15.118:6826","nonce":653000048}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6822","nonce":653000048},{"type":"v1","addr":"172.21.15.118:6823","nonce":653000048}]},"public_addr":"172.21.15.118:6819/653000048","cluster_addr":"172.21.15.118:6821/653000048","heartbeat_back_addr":"172.21.15.118:6826/653000048","heartbeat_front_addr":"172.21.15.118:6823/653000048","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-03T20:47:25.491994+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-03T20:47:25.587358+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-03T20:47:25.454175+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-03T20:47:25.377734+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-03T20:47:25.412351+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-03T20:47:25.499815+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-03T20:47:25.575044+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-03T20:47:25.552383+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.32:6835/4032603982":"2024-08-04T21:01:06.854369+0000","172.21.15.32:6833/4032603982":"2024-08-04T21:01:06.854369+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-08-03T21:01:18.990 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:18.989+0000 7f7a91a00640 1 -- 172.21.15.32:0/871860713 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f7a6c077010 msgr2=0x7f7a6c079520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:01:18.990 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:18.989+0000 7f7a91a00640 1 --2- 172.21.15.32:0/871860713 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f7a6c077010 0x7f7a6c079520 secure :-1 s=READY pgs=898 cs=0 l=1 rev1=1 crypto rx=0x7f7a7c036000 tx=0x7f7a7c004730 comp rx=0 tx=0).stop 2024-08-03T21:01:18.991 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:18.989+0000 7f7a91a00640 1 -- 172.21.15.32:0/871860713 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7a8c14eb10 msgr2=0x7f7a8c0cdf40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:01:18.991 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:18.989+0000 7f7a91a00640 1 --2- 172.21.15.32:0/871860713 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7a8c14eb10 0x7f7a8c0cdf40 secure :-1 s=READY pgs=2248 cs=0 l=1 rev1=1 crypto rx=0x7f7a8400cb10 tx=0x7f7a8400a060 comp rx=0 tx=0).stop 2024-08-03T21:01:18.991 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:18.990+0000 7f7a91a00640 1 -- 172.21.15.32:0/871860713 shutdown_connections 2024-08-03T21:01:18.991 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:18.990+0000 7f7a91a00640 1 --2- 172.21.15.32:0/871860713 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f7a6c077010 0x7f7a6c079520 unknown :-1 s=CLOSED pgs=898 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:01:18.991 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:18.990+0000 7f7a91a00640 1 --2- 172.21.15.32:0/871860713 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7a8c14eb10 0x7f7a8c0cdf40 unknown :-1 s=CLOSED pgs=2248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:01:18.991 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:18.990+0000 7f7a91a00640 1 --2- 172.21.15.32:0/871860713 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f7a8c0b4de0 0x7f7a8c0cda00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:01:18.991 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:18.990+0000 7f7a91a00640 1 --2- 172.21.15.32:0/871860713 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f7a8c0733f0 0x7f7a8c1459e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:01:18.991 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:18.990+0000 7f7a91a00640 1 -- 172.21.15.32:0/871860713 >> 172.21.15.32:0/871860713 conn(0x7f7a8c0b1bd0 msgr2=0x7f7a8c150ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T21:01:18.991 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:18.990+0000 7f7a91a00640 1 -- 172.21.15.32:0/871860713 shutdown_connections 2024-08-03T21:01:18.991 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:18.990+0000 7f7a91a00640 1 -- 172.21.15.32:0/871860713 wait complete. 2024-08-03T21:01:19.017 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-08-03T21:01:19.245 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:19.244+0000 7fae94a00640 1 -- 172.21.15.32:0/2220022133 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fae900d26e0 msgr2=0x7fae90161b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:01:19.245 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:19.244+0000 7fae94a00640 1 --2- 172.21.15.32:0/2220022133 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fae900d26e0 0x7fae90161b80 secure :-1 s=READY pgs=2249 cs=0 l=1 rev1=1 crypto rx=0x7fae900d3730 tx=0x7fae8002a5f0 comp rx=0 tx=0).stop 2024-08-03T21:01:19.245 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:19.244+0000 7fae94a00640 1 -- 172.21.15.32:0/2220022133 shutdown_connections 2024-08-03T21:01:19.245 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:19.244+0000 7fae94a00640 1 --2- 172.21.15.32:0/2220022133 >> v2:172.21.15.118:3300/0 conn(0x7fae900cffe0 0x7fae900b3e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:01:19.245 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:19.244+0000 7fae94a00640 1 --2- 172.21.15.32:0/2220022133 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fae900d26e0 0x7fae90161b80 unknown :-1 s=CLOSED pgs=2249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:01:19.246 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:19.244+0000 7fae94a00640 1 -- 172.21.15.32:0/2220022133 >> 172.21.15.32:0/2220022133 conn(0x7fae9008d7a0 msgr2=0x7fae9009d3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T21:01:19.246 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:19.244+0000 7fae94a00640 1 -- 172.21.15.32:0/2220022133 shutdown_connections 2024-08-03T21:01:19.246 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:19.244+0000 7fae94a00640 1 -- 172.21.15.32:0/2220022133 wait complete. 2024-08-03T21:01:19.247 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:19.246+0000 7fae94a00640 1 Processor -- start 2024-08-03T21:01:19.247 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:19.246+0000 7fae94a00640 1 -- start start 2024-08-03T21:01:19.249 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:19.248+0000 7fae94a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fae900b43d0 0x7fae900ca2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:01:19.249 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:19.248+0000 7fae94a00640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fae900cffe0 0x7fae9015dad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:01:19.249 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:19.248+0000 7fae94a00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fae900d26e0 0x7fae900c8390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:01:19.249 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:19.248+0000 7fae94a00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fae900d4100 con 0x7fae900b43d0 2024-08-03T21:01:19.249 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:19.248+0000 7fae94a00640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7fae900d3f80 con 0x7fae900d26e0 2024-08-03T21:01:19.249 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:19.248+0000 7fae94a00640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7fae900d4280 con 0x7fae900cffe0 2024-08-03T21:01:19.249 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:19.248+0000 7fae8f400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fae900b43d0 0x7fae900ca2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:01:19.249 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:19.248+0000 7fae8f400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fae900b43d0 0x7fae900ca2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:58938/0 (socket says 172.21.15.32:58938) 2024-08-03T21:01:19.249 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:19.248+0000 7fae8f400640 1 -- 172.21.15.32:0/1758260712 learned_addr learned my addr 172.21.15.32:0/1758260712 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T21:01:19.250 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:19.248+0000 7fae86a00640 1 --2- 172.21.15.32:0/1758260712 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fae900cffe0 0x7fae9015dad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:01:19.250 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:19.248+0000 7fae8fe00640 1 --2- 172.21.15.32:0/1758260712 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fae900d26e0 0x7fae900c8390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:01:19.251 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:19.249+0000 7fae8f400640 1 -- 172.21.15.32:0/1758260712 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fae900cffe0 msgr2=0x7fae9015dad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:01:19.251 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:19.249+0000 7fae8f400640 1 --2- 172.21.15.32:0/1758260712 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fae900cffe0 0x7fae9015dad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:01:19.251 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:19.249+0000 7fae8f400640 1 -- 172.21.15.32:0/1758260712 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fae900d26e0 msgr2=0x7fae900c8390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:01:19.251 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:19.249+0000 7fae8f400640 1 --2- 172.21.15.32:0/1758260712 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fae900d26e0 0x7fae900c8390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:01:19.251 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:19.249+0000 7fae8f400640 1 -- 172.21.15.32:0/1758260712 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fae900ca830 con 0x7fae900b43d0 2024-08-03T21:01:19.251 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:19.249+0000 7fae8f400640 1 --2- 172.21.15.32:0/1758260712 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fae900b43d0 0x7fae900ca2f0 secure :-1 s=READY pgs=2250 cs=0 l=1 rev1=1 crypto rx=0x7fae80000c00 tx=0x7fae80003820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T21:01:19.251 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:19.249+0000 7fae87e00640 1 -- 172.21.15.32:0/1758260712 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae8003e030 con 0x7fae900b43d0 2024-08-03T21:01:19.251 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:19.249+0000 7fae87e00640 1 -- 172.21.15.32:0/1758260712 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fae80003d10 con 0x7fae900b43d0 2024-08-03T21:01:19.251 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:19.249+0000 7fae87e00640 1 -- 172.21.15.32:0/1758260712 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae80035040 con 0x7fae900b43d0 2024-08-03T21:01:19.251 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:19.249+0000 7fae94a00640 1 -- 172.21.15.32:0/1758260712 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fae900cad40 con 0x7fae900b43d0 2024-08-03T21:01:19.252 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:19.250+0000 7fae94a00640 1 -- 172.21.15.32:0/1758260712 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fae901f8a20 con 0x7fae900b43d0 2024-08-03T21:01:19.253 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:19.251+0000 7fae94a00640 1 -- 172.21.15.32:0/1758260712 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fae900d4730 con 0x7fae900b43d0 2024-08-03T21:01:19.253 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:19.252+0000 7fae87e00640 1 -- 172.21.15.32:0/1758260712 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7fae80041030 con 0x7fae900b43d0 2024-08-03T21:01:19.254 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:19.253+0000 7fae87e00640 1 --2- 172.21.15.32:0/1758260712 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fae6c077010 0x7fae6c079520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:01:19.254 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:19.254+0000 7fae87e00640 1 -- 172.21.15.32:0/1758260712 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(136..136 src has 1..136) ==== 5795+0+0 (secure 0 0 0) 0x7fae800b8f50 con 0x7fae900b43d0 2024-08-03T21:01:19.255 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:19.254+0000 7fae86a00640 1 --2- 172.21.15.32:0/1758260712 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fae6c077010 0x7fae6c079520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:01:19.255 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:19.255+0000 7fae86a00640 1 --2- 172.21.15.32:0/1758260712 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fae6c077010 0x7fae6c079520 secure :-1 s=READY pgs=899 cs=0 l=1 rev1=1 crypto rx=0x7fae90162030 tx=0x7fae78005f90 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T21:01:19.259 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:19.259+0000 7fae87e00640 1 -- 172.21.15.32:0/1758260712 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7fae80081b20 con 0x7fae900b43d0 2024-08-03T21:01:19.501 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:19.500+0000 7fae94a00640 1 -- 172.21.15.32:0/1758260712 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7fae9015e010 con 0x7fae900b43d0 2024-08-03T21:01:19.502 INFO:tasks.ceph.mon.a.smithi032.stderr:2024-08-03T21:01:19.501+0000 7f4e6d600640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-08-03T21:01:19.504 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:19.503+0000 7fae87e00640 1 -- 172.21.15.32:0/1758260712 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v347) ==== 161+0+0 (secure 0 0 0) 0x7fae8008a3b0 con 0x7fae900b43d0 2024-08-03T21:01:19.504 INFO:teuthology.orchestra.run.smithi032.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-08-03T21:01:19.507 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:19.506+0000 7fae94a00640 1 -- 172.21.15.32:0/1758260712 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fae6c077010 msgr2=0x7fae6c079520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:01:19.508 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:19.506+0000 7fae94a00640 1 --2- 172.21.15.32:0/1758260712 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fae6c077010 0x7fae6c079520 secure :-1 s=READY pgs=899 cs=0 l=1 rev1=1 crypto rx=0x7fae90162030 tx=0x7fae78005f90 comp rx=0 tx=0).stop 2024-08-03T21:01:19.508 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:19.506+0000 7fae94a00640 1 -- 172.21.15.32:0/1758260712 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fae900b43d0 msgr2=0x7fae900ca2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:01:19.508 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:19.506+0000 7fae94a00640 1 --2- 172.21.15.32:0/1758260712 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fae900b43d0 0x7fae900ca2f0 secure :-1 s=READY pgs=2250 cs=0 l=1 rev1=1 crypto rx=0x7fae80000c00 tx=0x7fae80003820 comp rx=0 tx=0).stop 2024-08-03T21:01:19.508 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:19.507+0000 7fae94a00640 1 -- 172.21.15.32:0/1758260712 shutdown_connections 2024-08-03T21:01:19.508 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:19.507+0000 7fae94a00640 1 --2- 172.21.15.32:0/1758260712 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fae6c077010 0x7fae6c079520 unknown :-1 s=CLOSED pgs=899 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:01:19.508 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:19.507+0000 7fae94a00640 1 --2- 172.21.15.32:0/1758260712 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fae900d26e0 0x7fae900c8390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:01:19.508 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:19.507+0000 7fae94a00640 1 --2- 172.21.15.32:0/1758260712 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fae900cffe0 0x7fae9015dad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:01:19.508 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:19.507+0000 7fae94a00640 1 --2- 172.21.15.32:0/1758260712 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fae900b43d0 0x7fae900ca2f0 unknown :-1 s=CLOSED pgs=2250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:01:19.508 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:19.507+0000 7fae94a00640 1 -- 172.21.15.32:0/1758260712 >> 172.21.15.32:0/1758260712 conn(0x7fae9008d7a0 msgr2=0x7fae900b3030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T21:01:19.508 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:19.507+0000 7fae94a00640 1 -- 172.21.15.32:0/1758260712 shutdown_connections 2024-08-03T21:01:19.508 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:19.507+0000 7fae94a00640 1 -- 172.21.15.32:0/1758260712 wait complete. 2024-08-03T21:01:19.513 INFO:tasks.ceph.mds.c.smithi032.stderr: -529> 2024-08-03T21:01:12.192+0000 7fe797400640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-08-03T21:01:19.535 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-08-03T21:01:19.626 INFO:tasks.ceph.mds.c.smithi032.stdout:starting mds.c at 2024-08-03T21:01:19.772 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:19.771+0000 7fdc8ac00640 1 -- 172.21.15.32:0/3691152302 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdc6c0051a0 msgr2=0x7fdc6c00b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:01:19.772 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:19.771+0000 7fdc8ac00640 1 --2- 172.21.15.32:0/3691152302 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdc6c0051a0 0x7fdc6c00b9c0 secure :-1 s=READY pgs=2255 cs=0 l=1 rev1=1 crypto rx=0x7fdc84074940 tx=0x7fdc6c005830 comp rx=0 tx=0).stop 2024-08-03T21:01:19.772 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:19.771+0000 7fdc8a200640 1 -- 172.21.15.32:0/3691152302 reap_dead start 2024-08-03T21:01:19.772 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:19.771+0000 7fdc8ac00640 1 -- 172.21.15.32:0/3691152302 shutdown_connections 2024-08-03T21:01:19.772 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:19.771+0000 7fdc8ac00640 1 -- 172.21.15.32:0/3691152302 >> 172.21.15.32:0/3691152302 conn(0x7fdc840d0dd0 msgr2=0x7fdc840d11e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T21:01:19.772 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:19.771+0000 7fdc8ac00640 1 -- 172.21.15.32:0/3691152302 shutdown_connections 2024-08-03T21:01:19.772 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:19.771+0000 7fdc8ac00640 1 -- 172.21.15.32:0/3691152302 wait complete. 2024-08-03T21:01:19.773 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:19.773+0000 7fdc8ac00640 1 Processor -- start 2024-08-03T21:01:19.774 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:19.773+0000 7fdc8ac00640 1 -- start start 2024-08-03T21:01:19.777 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:19.775+0000 7fdc8ac00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdc6c0051a0 0x7fdc84156e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:01:19.777 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:19.775+0000 7fdc8ac00640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fdc84151f50 0x7fdc840c3ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:01:19.777 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:19.775+0000 7fdc8ac00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fdc8415b670 0x7fdc8415ba80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:01:19.777 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:19.775+0000 7fdc8ac00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fdc841607f0 con 0x7fdc6c0051a0 2024-08-03T21:01:19.777 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:19.775+0000 7fdc8ac00640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7fdc84156c50 con 0x7fdc8415b670 2024-08-03T21:01:19.777 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:19.775+0000 7fdc8ac00640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7fdc84160970 con 0x7fdc84151f50 2024-08-03T21:01:19.777 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:19.775+0000 7fdc89800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdc6c0051a0 0x7fdc84156e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:01:19.777 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:19.775+0000 7fdc89800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdc6c0051a0 0x7fdc84156e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:40110/0 (socket says 172.21.15.32:40110) 2024-08-03T21:01:19.777 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:19.775+0000 7fdc89800640 1 -- 172.21.15.32:0/765330368 learned_addr learned my addr 172.21.15.32:0/765330368 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T21:01:19.777 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:19.775+0000 7fdc88e00640 1 --2- 172.21.15.32:0/765330368 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fdc84151f50 0x7fdc840c3ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:01:19.778 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:19.775+0000 7fdc8a200640 1 --2- 172.21.15.32:0/765330368 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fdc8415b670 0x7fdc8415ba80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:01:19.778 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:19.775+0000 7fdc89800640 1 -- 172.21.15.32:0/765330368 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fdc84151f50 msgr2=0x7fdc840c3ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:01:19.778 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:19.775+0000 7fdc89800640 1 --2- 172.21.15.32:0/765330368 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fdc84151f50 0x7fdc840c3ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:01:19.778 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:19.775+0000 7fdc89800640 1 -- 172.21.15.32:0/765330368 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fdc8415b670 msgr2=0x7fdc8415ba80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:01:19.778 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:19.775+0000 7fdc89800640 1 --2- 172.21.15.32:0/765330368 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fdc8415b670 0x7fdc8415ba80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:01:19.778 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:19.775+0000 7fdc89800640 1 -- 172.21.15.32:0/765330368 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdc8415bfc0 con 0x7fdc6c0051a0 2024-08-03T21:01:19.778 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:19.776+0000 7fdc89800640 1 --2- 172.21.15.32:0/765330368 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdc6c0051a0 0x7fdc84156e20 secure :-1 s=READY pgs=2256 cs=0 l=1 rev1=1 crypto rx=0x7fdc78003f80 tx=0x7fdc780016f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T21:01:19.778 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:19.776+0000 7fdc81600640 1 -- 172.21.15.32:0/765330368 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc78016030 con 0x7fdc6c0051a0 2024-08-03T21:01:19.778 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:19.776+0000 7fdc81600640 1 -- 172.21.15.32:0/765330368 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fdc78011700 con 0x7fdc6c0051a0 2024-08-03T21:01:19.778 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:19.776+0000 7fdc81600640 1 -- 172.21.15.32:0/765330368 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc78011b30 con 0x7fdc6c0051a0 2024-08-03T21:01:19.778 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:19.776+0000 7fdc8ac00640 1 -- 172.21.15.32:0/765330368 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdc84142090 con 0x7fdc6c0051a0 2024-08-03T21:01:19.778 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:19.776+0000 7fdc8ac00640 1 -- 172.21.15.32:0/765330368 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdc84142650 con 0x7fdc6c0051a0 2024-08-03T21:01:19.779 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:19.777+0000 7fdc8ac00640 1 -- 172.21.15.32:0/765330368 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdc8406f660 con 0x7fdc6c0051a0 2024-08-03T21:01:19.779 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:19.778+0000 7fdc81600640 1 -- 172.21.15.32:0/765330368 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7fdc78019070 con 0x7fdc6c0051a0 2024-08-03T21:01:19.780 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:19.779+0000 7fdc81600640 1 --2- 172.21.15.32:0/765330368 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fdc64077010 0x7fdc64079520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:01:19.780 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:19.780+0000 7fdc81600640 1 -- 172.21.15.32:0/765330368 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(137..137 src has 1..137) ==== 5881+0+0 (secure 0 0 0) 0x7fdc780990d0 con 0x7fdc6c0051a0 2024-08-03T21:01:19.780 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:19.780+0000 7fdc88e00640 1 --2- 172.21.15.32:0/765330368 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fdc64077010 0x7fdc64079520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:01:19.781 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:19.781+0000 7fdc88e00640 1 --2- 172.21.15.32:0/765330368 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fdc64077010 0x7fdc64079520 secure :-1 s=READY pgs=900 cs=0 l=1 rev1=1 crypto rx=0x7fdc841597d0 tx=0x7fdc70005f90 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T21:01:19.784 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:19.784+0000 7fdc81600640 1 -- 172.21.15.32:0/765330368 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7fdc8415bfc0 con 0x7fdc6c0051a0 2024-08-03T21:01:20.036 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:20.036+0000 7fdc8ac00640 1 -- 172.21.15.32:0/765330368 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) -- 0x7fdc84157550 con 0x7fdc6c0051a0 2024-08-03T21:01:20.506 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:20.505+0000 7fdc81600640 1 -- 172.21.15.32:0/765330368 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v348) ==== 104+0+0 (secure 0 0 0) 0x7fdc78062210 con 0x7fdc6c0051a0 2024-08-03T21:01:20.509 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:20.508+0000 7fdc8ac00640 1 -- 172.21.15.32:0/765330368 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fdc64077010 msgr2=0x7fdc64079520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:01:20.509 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:20.508+0000 7fdc8ac00640 1 --2- 172.21.15.32:0/765330368 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fdc64077010 0x7fdc64079520 secure :-1 s=READY pgs=900 cs=0 l=1 rev1=1 crypto rx=0x7fdc841597d0 tx=0x7fdc70005f90 comp rx=0 tx=0).stop 2024-08-03T21:01:20.509 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:20.508+0000 7fdc8ac00640 1 -- 172.21.15.32:0/765330368 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdc6c0051a0 msgr2=0x7fdc84156e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:01:20.509 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:20.508+0000 7fdc8ac00640 1 --2- 172.21.15.32:0/765330368 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdc6c0051a0 0x7fdc84156e20 secure :-1 s=READY pgs=2256 cs=0 l=1 rev1=1 crypto rx=0x7fdc78003f80 tx=0x7fdc780016f0 comp rx=0 tx=0).stop 2024-08-03T21:01:20.509 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:20.508+0000 7fdc8ac00640 1 -- 172.21.15.32:0/765330368 shutdown_connections 2024-08-03T21:01:20.509 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:20.508+0000 7fdc8ac00640 1 --2- 172.21.15.32:0/765330368 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fdc64077010 0x7fdc64079520 unknown :-1 s=CLOSED pgs=900 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:01:20.509 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:20.508+0000 7fdc8ac00640 1 --2- 172.21.15.32:0/765330368 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fdc8415b670 0x7fdc8415ba80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:01:20.509 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:20.508+0000 7fdc8ac00640 1 --2- 172.21.15.32:0/765330368 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fdc84151f50 0x7fdc840c3ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:01:20.510 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:20.508+0000 7fdc8ac00640 1 --2- 172.21.15.32:0/765330368 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdc6c0051a0 0x7fdc84156e20 unknown :-1 s=CLOSED pgs=2256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:01:20.510 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:20.508+0000 7fdc8ac00640 1 -- 172.21.15.32:0/765330368 >> 172.21.15.32:0/765330368 conn(0x7fdc840d0dd0 msgr2=0x7fdc84070b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T21:01:20.510 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:20.508+0000 7fdc8ac00640 1 -- 172.21.15.32:0/765330368 shutdown_connections 2024-08-03T21:01:20.510 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:20.508+0000 7fdc8ac00640 1 -- 172.21.15.32:0/765330368 wait complete. 2024-08-03T21:01:20.533 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-08-03T21:01:20.767 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:20.766+0000 7f5960c00640 1 -- 172.21.15.32:0/1423462410 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f595c0cccf0 msgr2=0x7f595c0cd100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:01:20.768 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:20.766+0000 7f5960c00640 1 --2- 172.21.15.32:0/1423462410 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f595c0cccf0 0x7f595c0cd100 secure :-1 s=READY pgs=2257 cs=0 l=1 rev1=1 crypto rx=0x7f595c15a560 tx=0x7f594402a580 comp rx=0 tx=0).stop 2024-08-03T21:01:20.768 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:20.766+0000 7f5960c00640 1 -- 172.21.15.32:0/1423462410 shutdown_connections 2024-08-03T21:01:20.768 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:20.766+0000 7f5960c00640 1 --2- 172.21.15.32:0/1423462410 >> v2:172.21.15.118:3300/0 conn(0x7f595c0ced30 0x7f595c0cf140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:01:20.768 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:20.766+0000 7f5960c00640 1 --2- 172.21.15.32:0/1423462410 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f595c0cccf0 0x7f595c0cd100 unknown :-1 s=CLOSED pgs=2257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:01:20.768 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:20.766+0000 7f5960c00640 1 --2- 172.21.15.32:0/1423462410 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f595c145c50 0x7f595c06e950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:01:20.768 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:20.766+0000 7f5960c00640 1 -- 172.21.15.32:0/1423462410 >> 172.21.15.32:0/1423462410 conn(0x7f595c0b1bd0 msgr2=0x7f595c0b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T21:01:20.768 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:20.767+0000 7f5960c00640 1 -- 172.21.15.32:0/1423462410 shutdown_connections 2024-08-03T21:01:20.768 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:20.767+0000 7f5960c00640 1 -- 172.21.15.32:0/1423462410 wait complete. 2024-08-03T21:01:20.769 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:20.769+0000 7f5960c00640 1 Processor -- start 2024-08-03T21:01:20.769 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:20.769+0000 7f5960c00640 1 -- start start 2024-08-03T21:01:20.772 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:20.770+0000 7f5960c00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f595c0cccf0 0x7f595c0732c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:01:20.772 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:20.770+0000 7f5960c00640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f595c0ced30 0x7f595c073800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:01:20.772 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:20.770+0000 7f5960c00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f595c145c50 0x7f595c0c98a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:01:20.772 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:20.770+0000 7f5960c00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f595c0b3e60 con 0x7f595c145c50 2024-08-03T21:01:20.772 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:20.770+0000 7f5960c00640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f595c0b3ce0 con 0x7f595c0cccf0 2024-08-03T21:01:20.772 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:20.770+0000 7f5960c00640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7f595c0b3fe0 con 0x7f595c0ced30 2024-08-03T21:01:20.772 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:20.771+0000 7f595a800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f595c145c50 0x7f595c0c98a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:01:20.772 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:20.771+0000 7f595a800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f595c145c50 0x7f595c0c98a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:40138/0 (socket says 172.21.15.32:40138) 2024-08-03T21:01:20.772 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:20.771+0000 7f595a800640 1 -- 172.21.15.32:0/922038949 learned_addr learned my addr 172.21.15.32:0/922038949 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T21:01:20.772 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:20.771+0000 7f5959e00640 1 --2- 172.21.15.32:0/922038949 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f595c0cccf0 0x7f595c0732c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:01:20.773 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:20.771+0000 7f5953e00640 1 --2- 172.21.15.32:0/922038949 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f595c0ced30 0x7f595c073800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:01:20.773 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:20.771+0000 7f595a800640 1 -- 172.21.15.32:0/922038949 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f595c0ced30 msgr2=0x7f595c073800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:01:20.773 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:20.771+0000 7f595a800640 1 --2- 172.21.15.32:0/922038949 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f595c0ced30 0x7f595c073800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:01:20.773 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:20.771+0000 7f595a800640 1 -- 172.21.15.32:0/922038949 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f595c0cccf0 msgr2=0x7f595c0732c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:01:20.773 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:20.771+0000 7f595a800640 1 --2- 172.21.15.32:0/922038949 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f595c0cccf0 0x7f595c0732c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:01:20.773 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:20.771+0000 7f595a800640 1 -- 172.21.15.32:0/922038949 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f595c1f9070 con 0x7f595c145c50 2024-08-03T21:01:20.773 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:20.771+0000 7f595a800640 1 --2- 172.21.15.32:0/922038949 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f595c145c50 0x7f595c0c98a0 secure :-1 s=READY pgs=2258 cs=0 l=1 rev1=1 crypto rx=0x7f595400ea70 tx=0x7f595400ef30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T21:01:20.774 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:20.772+0000 7f5952000640 1 -- 172.21.15.32:0/922038949 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f595400ce40 con 0x7f595c145c50 2024-08-03T21:01:20.774 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:20.772+0000 7f5952000640 1 -- 172.21.15.32:0/922038949 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f5954004c10 con 0x7f595c145c50 2024-08-03T21:01:20.774 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:20.772+0000 7f5952000640 1 -- 172.21.15.32:0/922038949 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5954005020 con 0x7f595c145c50 2024-08-03T21:01:20.774 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:20.772+0000 7f5960c00640 1 -- 172.21.15.32:0/922038949 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f595c1f9350 con 0x7f595c145c50 2024-08-03T21:01:20.774 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:20.772+0000 7f5960c00640 1 -- 172.21.15.32:0/922038949 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f595c074000 con 0x7f595c145c50 2024-08-03T21:01:20.774 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:20.773+0000 7f5960c00640 1 -- 172.21.15.32:0/922038949 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f595c0b4500 con 0x7f595c145c50 2024-08-03T21:01:20.775 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:20.774+0000 7f5952000640 1 -- 172.21.15.32:0/922038949 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f59540051c0 con 0x7f595c145c50 2024-08-03T21:01:20.776 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:20.775+0000 7f5952000640 1 --2- 172.21.15.32:0/922038949 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f5938077010 0x7f5938079520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:01:20.776 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:20.776+0000 7f5952000640 1 -- 172.21.15.32:0/922038949 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(137..137 src has 1..137) ==== 5881+0+0 (secure 0 0 0) 0x7f5954099680 con 0x7f595c145c50 2024-08-03T21:01:20.776 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:20.776+0000 7f5959e00640 1 --2- 172.21.15.32:0/922038949 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f5938077010 0x7f5938079520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:01:20.777 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:20.777+0000 7f5959e00640 1 --2- 172.21.15.32:0/922038949 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f5938077010 0x7f5938079520 secure :-1 s=READY pgs=902 cs=0 l=1 rev1=1 crypto rx=0x7f594c009ab0 tx=0x7f594c005f90 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T21:01:20.780 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:20.780+0000 7f5952000640 1 -- 172.21.15.32:0/922038949 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7f5954062170 con 0x7f595c145c50 2024-08-03T21:01:20.985 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:20.985+0000 7f5960c00640 1 -- 172.21.15.32:0/922038949 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) -- 0x7f595c15e000 con 0x7f595c145c50 2024-08-03T21:01:21.930 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:21.930+0000 7f5952000640 1 -- 172.21.15.32:0/922038949 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v138) ==== 181+0+0 (secure 0 0 0) 0x7f59540666c0 con 0x7f595c145c50 2024-08-03T21:01:21.931 INFO:teuthology.orchestra.run.smithi032.stderr:pool 'cephfs_metadata' removed 2024-08-03T21:01:21.934 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:21.933+0000 7f5960c00640 1 -- 172.21.15.32:0/922038949 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f5938077010 msgr2=0x7f5938079520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:01:21.934 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:21.933+0000 7f5960c00640 1 --2- 172.21.15.32:0/922038949 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f5938077010 0x7f5938079520 secure :-1 s=READY pgs=902 cs=0 l=1 rev1=1 crypto rx=0x7f594c009ab0 tx=0x7f594c005f90 comp rx=0 tx=0).stop 2024-08-03T21:01:21.934 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:21.933+0000 7f5960c00640 1 -- 172.21.15.32:0/922038949 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f595c145c50 msgr2=0x7f595c0c98a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:01:21.934 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:21.933+0000 7f5960c00640 1 --2- 172.21.15.32:0/922038949 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f595c145c50 0x7f595c0c98a0 secure :-1 s=READY pgs=2258 cs=0 l=1 rev1=1 crypto rx=0x7f595400ea70 tx=0x7f595400ef30 comp rx=0 tx=0).stop 2024-08-03T21:01:21.934 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:21.933+0000 7f5960c00640 1 -- 172.21.15.32:0/922038949 shutdown_connections 2024-08-03T21:01:21.934 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:21.933+0000 7f5960c00640 1 --2- 172.21.15.32:0/922038949 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f5938077010 0x7f5938079520 unknown :-1 s=CLOSED pgs=902 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:01:21.934 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:21.933+0000 7f5960c00640 1 --2- 172.21.15.32:0/922038949 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f595c145c50 0x7f595c0c98a0 unknown :-1 s=CLOSED pgs=2258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:01:21.934 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:21.933+0000 7f5960c00640 1 --2- 172.21.15.32:0/922038949 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f595c0ced30 0x7f595c073800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:01:21.934 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:21.933+0000 7f5960c00640 1 --2- 172.21.15.32:0/922038949 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f595c0cccf0 0x7f595c0732c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:01:21.935 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:21.933+0000 7f5960c00640 1 -- 172.21.15.32:0/922038949 >> 172.21.15.32:0/922038949 conn(0x7f595c0b1bd0 msgr2=0x7f595c15e750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T21:01:21.935 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:21.933+0000 7f5960c00640 1 -- 172.21.15.32:0/922038949 shutdown_connections 2024-08-03T21:01:21.935 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:21.933+0000 7f5960c00640 1 -- 172.21.15.32:0/922038949 wait complete. 2024-08-03T21:01:21.964 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-08-03T21:01:22.204 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:22.203+0000 7fe1df200640 1 -- 172.21.15.32:0/2559807857 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe1d80d28d0 msgr2=0x7fe1d8161bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:01:22.204 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:22.203+0000 7fe1df200640 1 --2- 172.21.15.32:0/2559807857 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe1d80d28d0 0x7fe1d8161bf0 secure :-1 s=READY pgs=2260 cs=0 l=1 rev1=1 crypto rx=0x7fe1cc009830 tx=0x7fe1cc02a480 comp rx=0 tx=0).stop 2024-08-03T21:01:22.204 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:22.203+0000 7fe1df200640 1 -- 172.21.15.32:0/2559807857 shutdown_connections 2024-08-03T21:01:22.204 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:22.203+0000 7fe1df200640 1 --2- 172.21.15.32:0/2559807857 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fe1d80d4820 0x7fe1d80b5480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:01:22.204 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:22.203+0000 7fe1df200640 1 --2- 172.21.15.32:0/2559807857 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe1d80d28d0 0x7fe1d8161bf0 unknown :-1 s=CLOSED pgs=2260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:01:22.205 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:22.203+0000 7fe1df200640 1 -- 172.21.15.32:0/2559807857 >> 172.21.15.32:0/2559807857 conn(0x7fe1d808d980 msgr2=0x7fe1d809d5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T21:01:22.205 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:22.203+0000 7fe1df200640 1 -- 172.21.15.32:0/2559807857 shutdown_connections 2024-08-03T21:01:22.205 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:22.203+0000 7fe1df200640 1 -- 172.21.15.32:0/2559807857 wait complete. 2024-08-03T21:01:22.205 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:22.205+0000 7fe1df200640 1 Processor -- start 2024-08-03T21:01:22.206 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:22.205+0000 7fe1df200640 1 -- start start 2024-08-03T21:01:22.209 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:22.207+0000 7fe1df200640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fe1d80cec90 0x7fe1d8155680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:01:22.209 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:22.207+0000 7fe1df200640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fe1d80d28d0 0x7fe1d81e89d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:01:22.209 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:22.207+0000 7fe1df200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe1d80d4820 0x7fe1d81e8f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:01:22.209 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:22.207+0000 7fe1df200640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fe1d80b8af0 con 0x7fe1d80d4820 2024-08-03T21:01:22.209 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:22.207+0000 7fe1df200640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7fe1d80b8970 con 0x7fe1d80cec90 2024-08-03T21:01:22.209 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:22.207+0000 7fe1df200640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7fe1d80b8c70 con 0x7fe1d80d28d0 2024-08-03T21:01:22.209 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:22.207+0000 7fe1de800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe1d80d4820 0x7fe1d81e8f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:01:22.209 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:22.207+0000 7fe1de800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe1d80d4820 0x7fe1d81e8f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:40146/0 (socket says 172.21.15.32:40146) 2024-08-03T21:01:22.210 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:22.207+0000 7fe1de800640 1 -- 172.21.15.32:0/502949260 learned_addr learned my addr 172.21.15.32:0/502949260 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T21:01:22.210 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:22.207+0000 7fe1dde00640 1 --2- 172.21.15.32:0/502949260 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fe1d80cec90 0x7fe1d8155680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:01:22.210 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:22.207+0000 7fe1dd400640 1 --2- 172.21.15.32:0/502949260 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fe1d80d28d0 0x7fe1d81e89d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:01:22.210 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:22.207+0000 7fe1de800640 1 -- 172.21.15.32:0/502949260 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fe1d80d28d0 msgr2=0x7fe1d81e89d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:01:22.210 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:22.207+0000 7fe1de800640 1 --2- 172.21.15.32:0/502949260 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fe1d80d28d0 0x7fe1d81e89d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:01:22.210 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:22.207+0000 7fe1de800640 1 -- 172.21.15.32:0/502949260 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fe1d80cec90 msgr2=0x7fe1d8155680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:01:22.210 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:22.207+0000 7fe1de800640 1 --2- 172.21.15.32:0/502949260 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fe1d80cec90 0x7fe1d8155680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:01:22.210 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:22.207+0000 7fe1de800640 1 -- 172.21.15.32:0/502949260 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe1d80b29f0 con 0x7fe1d80d4820 2024-08-03T21:01:22.210 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:22.207+0000 7fe1dd400640 1 --2- 172.21.15.32:0/502949260 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fe1d80d28d0 0x7fe1d81e89d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-03T21:01:22.210 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:22.208+0000 7fe1de800640 1 --2- 172.21.15.32:0/502949260 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe1d80d4820 0x7fe1d81e8f10 secure :-1 s=READY pgs=2261 cs=0 l=1 rev1=1 crypto rx=0x7fe1d400e990 tx=0x7fe1d400ee50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T21:01:22.210 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:22.208+0000 7fe1d2000640 1 -- 172.21.15.32:0/502949260 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1d4004d20 con 0x7fe1d80d4820 2024-08-03T21:01:22.210 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:22.208+0000 7fe1d2000640 1 -- 172.21.15.32:0/502949260 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fe1d400caa0 con 0x7fe1d80d4820 2024-08-03T21:01:22.210 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:22.208+0000 7fe1d2000640 1 -- 172.21.15.32:0/502949260 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1d4005020 con 0x7fe1d80d4820 2024-08-03T21:01:22.211 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:22.208+0000 7fe1df200640 1 -- 172.21.15.32:0/502949260 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe1d80b1e70 con 0x7fe1d80d4820 2024-08-03T21:01:22.211 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:22.208+0000 7fe1df200640 1 -- 172.21.15.32:0/502949260 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe1d80b2430 con 0x7fe1d80d4820 2024-08-03T21:01:22.211 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:22.209+0000 7fe1df200640 1 -- 172.21.15.32:0/502949260 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe1d80b9120 con 0x7fe1d80d4820 2024-08-03T21:01:22.211 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:22.211+0000 7fe1d2000640 1 -- 172.21.15.32:0/502949260 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7fe1d40051c0 con 0x7fe1d80d4820 2024-08-03T21:01:22.212 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:22.212+0000 7fe1d2000640 1 --2- 172.21.15.32:0/502949260 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fe1bc077010 0x7fe1bc079520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:01:22.212 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:22.212+0000 7fe1d2000640 1 -- 172.21.15.32:0/502949260 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(138..138 src has 1..138) ==== 5443+0+0 (secure 0 0 0) 0x7fe1d40990a0 con 0x7fe1d80d4820 2024-08-03T21:01:22.213 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:22.212+0000 7fe1dde00640 1 --2- 172.21.15.32:0/502949260 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fe1bc077010 0x7fe1bc079520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:01:22.213 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:22.213+0000 7fe1dde00640 1 --2- 172.21.15.32:0/502949260 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fe1bc077010 0x7fe1bc079520 secure :-1 s=READY pgs=903 cs=0 l=1 rev1=1 crypto rx=0x7fe1cc034000 tx=0x7fe1cc004730 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T21:01:22.216 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:22.216+0000 7fe1d2000640 1 -- 172.21.15.32:0/502949260 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7fe1d4016030 con 0x7fe1d80d4820 2024-08-03T21:01:22.426 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:22.425+0000 7fe1df200640 1 -- 172.21.15.32:0/502949260 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) -- 0x7fe1d80b48f0 con 0x7fe1d80d4820 2024-08-03T21:01:22.937 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:22.936+0000 7fe1d2000640 1 -- 172.21.15.32:0/502949260 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v139) ==== 169+0+0 (secure 0 0 0) 0x7fe1d4062300 con 0x7fe1d80d4820 2024-08-03T21:01:22.937 INFO:teuthology.orchestra.run.smithi032.stderr:pool 'cephfs_data' removed 2024-08-03T21:01:22.941 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:22.941+0000 7fe1df200640 1 -- 172.21.15.32:0/502949260 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fe1bc077010 msgr2=0x7fe1bc079520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:01:22.941 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:22.941+0000 7fe1df200640 1 --2- 172.21.15.32:0/502949260 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fe1bc077010 0x7fe1bc079520 secure :-1 s=READY pgs=903 cs=0 l=1 rev1=1 crypto rx=0x7fe1cc034000 tx=0x7fe1cc004730 comp rx=0 tx=0).stop 2024-08-03T21:01:22.942 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:22.941+0000 7fe1df200640 1 -- 172.21.15.32:0/502949260 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe1d80d4820 msgr2=0x7fe1d81e8f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:01:22.942 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:22.941+0000 7fe1df200640 1 --2- 172.21.15.32:0/502949260 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe1d80d4820 0x7fe1d81e8f10 secure :-1 s=READY pgs=2261 cs=0 l=1 rev1=1 crypto rx=0x7fe1d400e990 tx=0x7fe1d400ee50 comp rx=0 tx=0).stop 2024-08-03T21:01:22.942 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:22.941+0000 7fe1df200640 1 -- 172.21.15.32:0/502949260 shutdown_connections 2024-08-03T21:01:22.942 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:22.941+0000 7fe1df200640 1 --2- 172.21.15.32:0/502949260 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fe1bc077010 0x7fe1bc079520 unknown :-1 s=CLOSED pgs=903 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:01:22.942 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:22.941+0000 7fe1df200640 1 --2- 172.21.15.32:0/502949260 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe1d80d4820 0x7fe1d81e8f10 unknown :-1 s=CLOSED pgs=2261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:01:22.943 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:22.941+0000 7fe1df200640 1 --2- 172.21.15.32:0/502949260 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fe1d80d28d0 0x7fe1d81e89d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:01:22.943 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:22.941+0000 7fe1df200640 1 --2- 172.21.15.32:0/502949260 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fe1d80cec90 0x7fe1d8155680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:01:22.943 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:22.941+0000 7fe1df200640 1 -- 172.21.15.32:0/502949260 >> 172.21.15.32:0/502949260 conn(0x7fe1d808d980 msgr2=0x7fe1d80d1390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T21:01:22.943 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:22.942+0000 7fe1df200640 1 -- 172.21.15.32:0/502949260 shutdown_connections 2024-08-03T21:01:22.943 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:22.942+0000 7fe1df200640 1 -- 172.21.15.32:0/502949260 wait complete. 2024-08-03T21:01:22.974 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config reset 27 2024-08-03T21:01:23.206 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:23.205+0000 7f1952600640 1 -- 172.21.15.32:0/1212105392 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f194c06e930 msgr2=0x7f194c144f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:01:23.206 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:23.205+0000 7f1952600640 1 --2- 172.21.15.32:0/1212105392 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f194c06e930 0x7f194c144f90 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f193c005fd0 tx=0x7f193c02a5d0 comp rx=0 tx=0).stop 2024-08-03T21:01:23.207 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:23.205+0000 7f1952600640 1 -- 172.21.15.32:0/1212105392 shutdown_connections 2024-08-03T21:01:23.207 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:23.205+0000 7f1952600640 1 --2- 172.21.15.32:0/1212105392 >> v2:172.21.15.118:3300/0 conn(0x7f194c15fff0 0x7f194c160400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:01:23.207 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:23.205+0000 7f1952600640 1 --2- 172.21.15.32:0/1212105392 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f194c06e930 0x7f194c144f90 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:01:23.207 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:23.205+0000 7f1952600640 1 -- 172.21.15.32:0/1212105392 >> 172.21.15.32:0/1212105392 conn(0x7f194c0b1bd0 msgr2=0x7f194c0b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T21:01:23.207 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:23.205+0000 7f1952600640 1 -- 172.21.15.32:0/1212105392 shutdown_connections 2024-08-03T21:01:23.207 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:23.205+0000 7f1952600640 1 -- 172.21.15.32:0/1212105392 wait complete. 2024-08-03T21:01:23.208 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:23.207+0000 7f1952600640 1 Processor -- start 2024-08-03T21:01:23.208 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:23.207+0000 7f1952600640 1 -- start start 2024-08-03T21:01:23.210 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:23.209+0000 7f1952600640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f194c06e930 0x7f194c15b7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:01:23.210 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:23.209+0000 7f1952600640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f194c073e60 0x7f194c0a9ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:01:23.210 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:23.209+0000 7f1952600640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f194c15fff0 0x7f194c15ebf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:01:23.210 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:23.209+0000 7f1952600640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f194c0cf840 con 0x7f194c06e930 2024-08-03T21:01:23.210 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:23.209+0000 7f1952600640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f194c145f30 con 0x7f194c15fff0 2024-08-03T21:01:23.210 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:23.209+0000 7f1952600640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7f194c0cf9c0 con 0x7f194c073e60 2024-08-03T21:01:23.210 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:23.209+0000 7f194b400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f194c06e930 0x7f194c15b7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:01:23.210 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:23.209+0000 7f194b400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f194c06e930 0x7f194c15b7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:40156/0 (socket says 172.21.15.32:40156) 2024-08-03T21:01:23.210 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:23.209+0000 7f194b400640 1 -- 172.21.15.32:0/3747974031 learned_addr learned my addr 172.21.15.32:0/3747974031 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T21:01:23.211 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:23.209+0000 7f194aa00640 1 --2- 172.21.15.32:0/3747974031 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f194c073e60 0x7f194c0a9ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:01:23.211 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:23.209+0000 7f194be00640 1 --2- 172.21.15.32:0/3747974031 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f194c15fff0 0x7f194c15ebf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:01:23.212 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:23.210+0000 7f194b400640 1 -- 172.21.15.32:0/3747974031 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f194c073e60 msgr2=0x7f194c0a9ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:01:23.212 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:23.210+0000 7f194b400640 1 --2- 172.21.15.32:0/3747974031 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f194c073e60 0x7f194c0a9ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:01:23.212 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:23.210+0000 7f194b400640 1 -- 172.21.15.32:0/3747974031 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f194c15fff0 msgr2=0x7f194c15ebf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:01:23.212 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:23.210+0000 7f194b400640 1 --2- 172.21.15.32:0/3747974031 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f194c15fff0 0x7f194c15ebf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:01:23.212 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:23.210+0000 7f194b400640 1 -- 172.21.15.32:0/3747974031 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f194c134180 con 0x7f194c06e930 2024-08-03T21:01:23.212 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:23.210+0000 7f194b400640 1 --2- 172.21.15.32:0/3747974031 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f194c06e930 0x7f194c15b7b0 secure :-1 s=READY pgs=2262 cs=0 l=1 rev1=1 crypto rx=0x7f193c034040 tx=0x7f193c0035f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T21:01:23.212 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:23.210+0000 7f1943400640 1 -- 172.21.15.32:0/3747974031 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f193c03e030 con 0x7f194c06e930 2024-08-03T21:01:23.213 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:23.210+0000 7f1952600640 1 -- 172.21.15.32:0/3747974031 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f194c15bcf0 con 0x7f194c06e930 2024-08-03T21:01:23.213 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:23.210+0000 7f1943400640 1 -- 172.21.15.32:0/3747974031 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f193c003ae0 con 0x7f194c06e930 2024-08-03T21:01:23.213 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:23.211+0000 7f1943400640 1 -- 172.21.15.32:0/3747974031 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f193c035040 con 0x7f194c06e930 2024-08-03T21:01:23.213 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:23.211+0000 7f1952600640 1 -- 172.21.15.32:0/3747974031 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f194c1343f0 con 0x7f194c06e930 2024-08-03T21:01:23.213 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:23.212+0000 7f1952600640 1 -- 172.21.15.32:0/3747974031 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f194c160bc0 con 0x7f194c06e930 2024-08-03T21:01:23.213 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:23.213+0000 7f1943400640 1 -- 172.21.15.32:0/3747974031 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f193c041030 con 0x7f194c06e930 2024-08-03T21:01:23.214 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:23.214+0000 7f1943400640 1 --2- 172.21.15.32:0/3747974031 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f192c077010 0x7f192c079520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:01:23.215 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:23.214+0000 7f1943400640 1 -- 172.21.15.32:0/3747974031 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(139..139 src has 1..139) ==== 5029+0+0 (secure 0 0 0) 0x7f193c0b8be0 con 0x7f194c06e930 2024-08-03T21:01:23.215 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:23.214+0000 7f194aa00640 1 --2- 172.21.15.32:0/3747974031 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f192c077010 0x7f192c079520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:01:23.216 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:23.215+0000 7f194aa00640 1 --2- 172.21.15.32:0/3747974031 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f192c077010 0x7f192c079520 secure :-1 s=READY pgs=904 cs=0 l=1 rev1=1 crypto rx=0x7f194c13fc60 tx=0x7f1938009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T21:01:23.219 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:23.218+0000 7f1943400640 1 -- 172.21.15.32:0/3747974031 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7f193c081910 con 0x7f194c06e930 2024-08-03T21:01:23.423 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:23.423+0000 7f1952600640 1 -- 172.21.15.32:0/3747974031 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "config reset", "num": 27} v 0) -- 0x7f194c145920 con 0x7f194c06e930 2024-08-03T21:01:23.427 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:23.427+0000 7f1943400640 1 -- 172.21.15.32:0/3747974031 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "config reset", "num": 27}]=0 v46) ==== 71+0+0 (secure 0 0 0) 0x7f193c085e60 con 0x7f194c06e930 2024-08-03T21:01:23.431 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:23.430+0000 7f1952600640 1 -- 172.21.15.32:0/3747974031 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f192c077010 msgr2=0x7f192c079520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:01:23.431 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:23.430+0000 7f1952600640 1 --2- 172.21.15.32:0/3747974031 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f192c077010 0x7f192c079520 secure :-1 s=READY pgs=904 cs=0 l=1 rev1=1 crypto rx=0x7f194c13fc60 tx=0x7f1938009040 comp rx=0 tx=0).stop 2024-08-03T21:01:23.431 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:23.430+0000 7f1952600640 1 -- 172.21.15.32:0/3747974031 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f194c06e930 msgr2=0x7f194c15b7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:01:23.431 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:23.430+0000 7f1952600640 1 --2- 172.21.15.32:0/3747974031 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f194c06e930 0x7f194c15b7b0 secure :-1 s=READY pgs=2262 cs=0 l=1 rev1=1 crypto rx=0x7f193c034040 tx=0x7f193c0035f0 comp rx=0 tx=0).stop 2024-08-03T21:01:23.431 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:23.430+0000 7f1952600640 1 -- 172.21.15.32:0/3747974031 shutdown_connections 2024-08-03T21:01:23.431 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:23.430+0000 7f1952600640 1 --2- 172.21.15.32:0/3747974031 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f192c077010 0x7f192c079520 unknown :-1 s=CLOSED pgs=904 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:01:23.431 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:23.430+0000 7f1952600640 1 --2- 172.21.15.32:0/3747974031 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f194c15fff0 0x7f194c15ebf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:01:23.431 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:23.430+0000 7f1952600640 1 --2- 172.21.15.32:0/3747974031 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f194c073e60 0x7f194c0a9ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:01:23.432 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:23.430+0000 7f1952600640 1 --2- 172.21.15.32:0/3747974031 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f194c06e930 0x7f194c15b7b0 unknown :-1 s=CLOSED pgs=2262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:01:23.432 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:23.430+0000 7f1952600640 1 -- 172.21.15.32:0/3747974031 >> 172.21.15.32:0/3747974031 conn(0x7f194c0b1bd0 msgr2=0x7f194c1434f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T21:01:23.432 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:23.430+0000 7f1952600640 1 -- 172.21.15.32:0/3747974031 shutdown_connections 2024-08-03T21:01:23.432 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:23.430+0000 7f1952600640 1 -- 172.21.15.32:0/3747974031 wait complete. 2024-08-03T21:01:23.455 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_damage.TestDamage.test_open_ino_errors' 2024-08-03T21:01:23.684 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:23.683+0000 7f902ac00640 1 -- 172.21.15.32:0/2270952052 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f90240ade80 msgr2=0x7f90240ae290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:01:23.685 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:23.683+0000 7f902ac00640 1 --2- 172.21.15.32:0/2270952052 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f90240ade80 0x7f90240ae290 secure :-1 s=READY pgs=2263 cs=0 l=1 rev1=1 crypto rx=0x7f9024071cc0 tx=0x7f901c02aa30 comp rx=0 tx=0).stop 2024-08-03T21:01:23.685 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:23.683+0000 7f902ac00640 1 -- 172.21.15.32:0/2270952052 shutdown_connections 2024-08-03T21:01:23.685 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:23.683+0000 7f902ac00640 1 --2- 172.21.15.32:0/2270952052 >> v2:172.21.15.118:3300/0 conn(0x7f90240b40b0 0x7f902406fb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:01:23.685 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:23.683+0000 7f902ac00640 1 --2- 172.21.15.32:0/2270952052 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f90240ade80 0x7f90240ae290 unknown :-1 s=CLOSED pgs=2263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:01:23.685 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:23.683+0000 7f902ac00640 1 -- 172.21.15.32:0/2270952052 >> 172.21.15.32:0/2270952052 conn(0x7f90240b1bd0 msgr2=0x7f90240b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T21:01:23.685 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:23.683+0000 7f902ac00640 1 -- 172.21.15.32:0/2270952052 shutdown_connections 2024-08-03T21:01:23.685 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:23.684+0000 7f902ac00640 1 -- 172.21.15.32:0/2270952052 wait complete. 2024-08-03T21:01:23.686 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:23.686+0000 7f902ac00640 1 Processor -- start 2024-08-03T21:01:23.686 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:23.686+0000 7f902ac00640 1 -- start start 2024-08-03T21:01:23.690 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:23.687+0000 7f902ac00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f902406f720 0x7f902414a370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:01:23.690 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:23.687+0000 7f902ac00640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f90240ade80 0x7f9024155e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:01:23.690 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:23.687+0000 7f902ac00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f90240b40b0 0x7f90241563a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:01:23.691 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:23.687+0000 7f902ac00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f90240aeb60 con 0x7f90240b40b0 2024-08-03T21:01:23.691 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:23.687+0000 7f902ac00640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f90240ae9e0 con 0x7f902406f720 2024-08-03T21:01:23.691 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:23.687+0000 7f902ac00640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7f90240aece0 con 0x7f90240ade80 2024-08-03T21:01:23.691 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:23.688+0000 7f9028c00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f90240b40b0 0x7f90241563a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:01:23.691 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:23.688+0000 7f9028c00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f90240b40b0 0x7f90241563a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:40166/0 (socket says 172.21.15.32:40166) 2024-08-03T21:01:23.691 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:23.688+0000 7f9028c00640 1 -- 172.21.15.32:0/2925854874 learned_addr learned my addr 172.21.15.32:0/2925854874 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T21:01:23.691 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:23.688+0000 7f9023e00640 1 --2- 172.21.15.32:0/2925854874 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f902406f720 0x7f902414a370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:01:23.691 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:23.688+0000 7f901be00640 1 --2- 172.21.15.32:0/2925854874 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f90240ade80 0x7f9024155e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:01:23.691 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:23.688+0000 7f9028c00640 1 -- 172.21.15.32:0/2925854874 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f90240ade80 msgr2=0x7f9024155e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:01:23.691 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:23.688+0000 7f9028c00640 1 --2- 172.21.15.32:0/2925854874 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f90240ade80 0x7f9024155e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:01:23.691 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:23.688+0000 7f9028c00640 1 -- 172.21.15.32:0/2925854874 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f902406f720 msgr2=0x7f902414a370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:01:23.692 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:23.688+0000 7f9028c00640 1 --2- 172.21.15.32:0/2925854874 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f902406f720 0x7f902414a370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:01:23.692 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:23.688+0000 7f9028c00640 1 -- 172.21.15.32:0/2925854874 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9024161900 con 0x7f90240b40b0 2024-08-03T21:01:23.692 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:23.688+0000 7f901be00640 1 --2- 172.21.15.32:0/2925854874 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f90240ade80 0x7f9024155e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-03T21:01:23.692 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:23.689+0000 7f9028c00640 1 --2- 172.21.15.32:0/2925854874 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f90240b40b0 0x7f90241563a0 secure :-1 s=READY pgs=2264 cs=0 l=1 rev1=1 crypto rx=0x7f901c02b020 tx=0x7f901c003f10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T21:01:23.692 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:23.689+0000 7f9020c00640 1 -- 172.21.15.32:0/2925854874 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f901c033050 con 0x7f90240b40b0 2024-08-03T21:01:23.692 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:23.689+0000 7f9020c00640 1 -- 172.21.15.32:0/2925854874 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f901c004450 con 0x7f90240b40b0 2024-08-03T21:01:23.692 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:23.689+0000 7f9020c00640 1 -- 172.21.15.32:0/2925854874 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f901c038450 con 0x7f90240b40b0 2024-08-03T21:01:23.692 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:23.689+0000 7f902ac00640 1 -- 172.21.15.32:0/2925854874 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f902415fdd0 con 0x7f90240b40b0 2024-08-03T21:01:23.692 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:23.689+0000 7f902ac00640 1 -- 172.21.15.32:0/2925854874 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9024156ee0 con 0x7f90240b40b0 2024-08-03T21:01:23.693 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:23.690+0000 7f902ac00640 1 -- 172.21.15.32:0/2925854874 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9024159120 con 0x7f90240b40b0 2024-08-03T21:01:23.693 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:23.691+0000 7f9020c00640 1 -- 172.21.15.32:0/2925854874 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f901c036070 con 0x7f90240b40b0 2024-08-03T21:01:23.693 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:23.692+0000 7f9020c00640 1 --2- 172.21.15.32:0/2925854874 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f9000077010 0x7f9000079520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:01:23.693 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:23.692+0000 7f9020c00640 1 -- 172.21.15.32:0/2925854874 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(139..139 src has 1..139) ==== 5029+0+0 (secure 0 0 0) 0x7f901c0b9aa0 con 0x7f90240b40b0 2024-08-03T21:01:23.693 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:23.693+0000 7f9023e00640 1 --2- 172.21.15.32:0/2925854874 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f9000077010 0x7f9000079520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:01:23.694 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:23.694+0000 7f9023e00640 1 --2- 172.21.15.32:0/2925854874 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f9000077010 0x7f9000079520 secure :-1 s=READY pgs=905 cs=0 l=1 rev1=1 crypto rx=0x7f9014006fd0 tx=0x7f9014009060 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T21:01:23.697 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:23.697+0000 7f9020c00640 1 -- 172.21.15.32:0/2925854874 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7f901c082f30 con 0x7f90240b40b0 2024-08-03T21:01:23.979 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:23.979+0000 7f902ac00640 1 -- 172.21.15.32:0/2925854874 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_damage.TestDamage.test_open_ino_errors"]} v 0) -- 0x7f9024161690 con 0x7f90240b40b0 2024-08-03T21:01:24.946 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:24.945+0000 7f9020c00640 1 -- 172.21.15.32:0/2925854874 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_damage.TestDamage.test_open_ino_errors"]}]=0 v718) ==== 135+0+0 (secure 0 0 0) 0x7f901c082830 con 0x7f90240b40b0 2024-08-03T21:01:24.949 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:24.948+0000 7f902ac00640 1 -- 172.21.15.32:0/2925854874 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f9000077010 msgr2=0x7f9000079520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:01:24.949 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:24.948+0000 7f902ac00640 1 --2- 172.21.15.32:0/2925854874 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f9000077010 0x7f9000079520 secure :-1 s=READY pgs=905 cs=0 l=1 rev1=1 crypto rx=0x7f9014006fd0 tx=0x7f9014009060 comp rx=0 tx=0).stop 2024-08-03T21:01:24.949 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:24.948+0000 7f902ac00640 1 -- 172.21.15.32:0/2925854874 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f90240b40b0 msgr2=0x7f90241563a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:01:24.949 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:24.948+0000 7f902ac00640 1 --2- 172.21.15.32:0/2925854874 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f90240b40b0 0x7f90241563a0 secure :-1 s=READY pgs=2264 cs=0 l=1 rev1=1 crypto rx=0x7f901c02b020 tx=0x7f901c003f10 comp rx=0 tx=0).stop 2024-08-03T21:01:24.949 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:24.948+0000 7f902ac00640 1 -- 172.21.15.32:0/2925854874 shutdown_connections 2024-08-03T21:01:24.950 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:24.948+0000 7f902ac00640 1 --2- 172.21.15.32:0/2925854874 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f9000077010 0x7f9000079520 unknown :-1 s=CLOSED pgs=905 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:01:24.950 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:24.948+0000 7f902ac00640 1 --2- 172.21.15.32:0/2925854874 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f90240b40b0 0x7f90241563a0 unknown :-1 s=CLOSED pgs=2264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:01:24.950 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:24.948+0000 7f902ac00640 1 --2- 172.21.15.32:0/2925854874 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f90240ade80 0x7f9024155e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:01:24.950 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:24.948+0000 7f902ac00640 1 --2- 172.21.15.32:0/2925854874 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f902406f720 0x7f902414a370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:01:24.950 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:24.948+0000 7f902ac00640 1 -- 172.21.15.32:0/2925854874 >> 172.21.15.32:0/2925854874 conn(0x7f90240b1bd0 msgr2=0x7f90240afb80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T21:01:24.950 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:24.948+0000 7f902ac00640 1 -- 172.21.15.32:0/2925854874 shutdown_connections 2024-08-03T21:01:24.950 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:24.949+0000 7f902ac00640 1 -- 172.21.15.32:0/2925854874 wait complete. 2024-08-03T21:01:24.976 INFO:tasks.cephfs_test_runner:That errors encountered during opening inos are properly propagated ... ERROR 2024-08-03T21:01:24.976 INFO:tasks.cephfs_test_runner: 2024-08-03T21:01:24.976 INFO:tasks.cephfs_test_runner:====================================================================== 2024-08-03T21:01:24.976 INFO:tasks.cephfs_test_runner:ERROR: test_open_ino_errors (tasks.cephfs.test_damage.TestDamage) 2024-08-03T21:01:24.976 INFO:tasks.cephfs_test_runner:That errors encountered during opening inos are properly propagated 2024-08-03T21:01:24.976 INFO:tasks.cephfs_test_runner:---------------------------------------------------------------------- 2024-08-03T21:01:24.977 INFO:tasks.cephfs_test_runner:Traceback (most recent call last): 2024-08-03T21:01:24.977 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_ceph-c_3792a0a4a40f645dee271188eab388b1c22c10f8/qa/tasks/cephfs/test_damage.py", line 510, in test_open_ino_errors 2024-08-03T21:01:24.977 INFO:tasks.cephfs_test_runner: self.fs.read_backtrace(file1_ino) 2024-08-03T21:01:24.977 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_ceph-c_3792a0a4a40f645dee271188eab388b1c22c10f8/qa/tasks/cephfs/filesystem.py", line 1545, in read_backtrace 2024-08-03T21:01:24.977 INFO:tasks.cephfs_test_runner: return self._read_data_xattr(ino_no, "parent", "inode_backtrace_t", pool) 2024-08-03T21:01:24.977 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_ceph-c_3792a0a4a40f645dee271188eab388b1c22c10f8/qa/tasks/cephfs/filesystem.py", line 1501, in _read_data_xattr 2024-08-03T21:01:24.977 INFO:tasks.cephfs_test_runner: return json.loads(self.dencoder(obj_type, obj_blob).strip()) 2024-08-03T21:01:24.977 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_ceph-c_3792a0a4a40f645dee271188eab388b1c22c10f8/qa/tasks/cephfs/filesystem.py", line 1304, in dencoder 2024-08-03T21:01:24.977 INFO:tasks.cephfs_test_runner: p = self.mon_manager.controller.run(args=args, stdin=BytesIO(obj_blob), stdout=BytesIO()) 2024-08-03T21:01:24.977 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_teuthology_7d7f5fe79c7b6b5080402153d07fb1acc4ce00a2/teuthology/orchestra/remote.py", line 523, in run 2024-08-03T21:01:24.977 INFO:tasks.cephfs_test_runner: r = self._runner(client=self.ssh, name=self.shortname, **kwargs) 2024-08-03T21:01:24.977 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_teuthology_7d7f5fe79c7b6b5080402153d07fb1acc4ce00a2/teuthology/orchestra/run.py", line 455, in run 2024-08-03T21:01:24.977 INFO:tasks.cephfs_test_runner: r.wait() 2024-08-03T21:01:24.977 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_teuthology_7d7f5fe79c7b6b5080402153d07fb1acc4ce00a2/teuthology/orchestra/run.py", line 161, in wait 2024-08-03T21:01:24.977 INFO:tasks.cephfs_test_runner: self._raise_for_status() 2024-08-03T21:01:24.978 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_teuthology_7d7f5fe79c7b6b5080402153d07fb1acc4ce00a2/teuthology/orchestra/run.py", line 181, in _raise_for_status 2024-08-03T21:01:24.978 INFO:tasks.cephfs_test_runner: raise CommandFailedError( 2024-08-03T21:01:24.978 INFO:tasks.cephfs_test_runner:teuthology.exceptions.CommandFailedError: Command failed on smithi032 with status 1: 'ceph-dencoder type inode_backtrace_t import - decode dump_json' 2024-08-03T21:01:24.978 INFO:tasks.cephfs_test_runner: 2024-08-03T21:01:24.978 INFO:tasks.cephfs_test_runner:---------------------------------------------------------------------- 2024-08-03T21:01:24.978 INFO:tasks.cephfs_test_runner:Ran 6 tests in 800.403s 2024-08-03T21:01:24.978 INFO:tasks.cephfs_test_runner: 2024-08-03T21:01:24.978 INFO:tasks.cephfs_test_runner:FAILED (errors=1) 2024-08-03T21:01:24.978 INFO:tasks.cephfs_test_runner: 2024-08-03T21:01:24.978 INFO:tasks.cephfs_test_runner:====================================================================== 2024-08-03T21:01:24.978 INFO:tasks.cephfs_test_runner:ERROR: test_open_ino_errors (tasks.cephfs.test_damage.TestDamage) 2024-08-03T21:01:24.978 INFO:tasks.cephfs_test_runner:That errors encountered during opening inos are properly propagated 2024-08-03T21:01:24.978 INFO:tasks.cephfs_test_runner:---------------------------------------------------------------------- 2024-08-03T21:01:24.979 INFO:tasks.cephfs_test_runner:Traceback (most recent call last): 2024-08-03T21:01:24.979 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_ceph-c_3792a0a4a40f645dee271188eab388b1c22c10f8/qa/tasks/cephfs/test_damage.py", line 510, in test_open_ino_errors 2024-08-03T21:01:24.979 INFO:tasks.cephfs_test_runner: self.fs.read_backtrace(file1_ino) 2024-08-03T21:01:24.979 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_ceph-c_3792a0a4a40f645dee271188eab388b1c22c10f8/qa/tasks/cephfs/filesystem.py", line 1545, in read_backtrace 2024-08-03T21:01:24.979 INFO:tasks.cephfs_test_runner: return self._read_data_xattr(ino_no, "parent", "inode_backtrace_t", pool) 2024-08-03T21:01:24.979 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_ceph-c_3792a0a4a40f645dee271188eab388b1c22c10f8/qa/tasks/cephfs/filesystem.py", line 1501, in _read_data_xattr 2024-08-03T21:01:24.979 INFO:tasks.cephfs_test_runner: return json.loads(self.dencoder(obj_type, obj_blob).strip()) 2024-08-03T21:01:24.979 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_ceph-c_3792a0a4a40f645dee271188eab388b1c22c10f8/qa/tasks/cephfs/filesystem.py", line 1304, in dencoder 2024-08-03T21:01:24.979 INFO:tasks.cephfs_test_runner: p = self.mon_manager.controller.run(args=args, stdin=BytesIO(obj_blob), stdout=BytesIO()) 2024-08-03T21:01:24.979 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_teuthology_7d7f5fe79c7b6b5080402153d07fb1acc4ce00a2/teuthology/orchestra/remote.py", line 523, in run 2024-08-03T21:01:24.979 INFO:tasks.cephfs_test_runner: r = self._runner(client=self.ssh, name=self.shortname, **kwargs) 2024-08-03T21:01:24.979 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_teuthology_7d7f5fe79c7b6b5080402153d07fb1acc4ce00a2/teuthology/orchestra/run.py", line 455, in run 2024-08-03T21:01:24.979 INFO:tasks.cephfs_test_runner: r.wait() 2024-08-03T21:01:24.979 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_teuthology_7d7f5fe79c7b6b5080402153d07fb1acc4ce00a2/teuthology/orchestra/run.py", line 161, in wait 2024-08-03T21:01:24.979 INFO:tasks.cephfs_test_runner: self._raise_for_status() 2024-08-03T21:01:24.979 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_teuthology_7d7f5fe79c7b6b5080402153d07fb1acc4ce00a2/teuthology/orchestra/run.py", line 181, in _raise_for_status 2024-08-03T21:01:24.980 INFO:tasks.cephfs_test_runner: raise CommandFailedError( 2024-08-03T21:01:24.980 INFO:tasks.cephfs_test_runner:teuthology.exceptions.CommandFailedError: Command failed on smithi032 with status 1: 'ceph-dencoder type inode_backtrace_t import - decode dump_json' 2024-08-03T21:01:24.980 INFO:tasks.cephfs_test_runner: 2024-08-03T21:01:24.980 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_7d7f5fe79c7b6b5080402153d07fb1acc4ce00a2/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.10/contextlib.py", line 135, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_3792a0a4a40f645dee271188eab388b1c22c10f8/qa/tasks/cephfs_test_runner.py", line 211, in task raise RuntimeError("Test failure: {0}".format(", ".join(bad_tests))) RuntimeError: Test failure: test_open_ino_errors (tasks.cephfs.test_damage.TestDamage) 2024-08-03T21:01:25.279 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=bf524262d9714b4388c6610345d2bc69 Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_7d7f5fe79c7b6b5080402153d07fb1acc4ce00a2/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.10/contextlib.py", line 135, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_3792a0a4a40f645dee271188eab388b1c22c10f8/qa/tasks/cephfs_test_runner.py", line 211, in task raise RuntimeError("Test failure: {0}".format(", ".join(bad_tests))) RuntimeError: Test failure: test_open_ino_errors (tasks.cephfs.test_damage.TestDamage) 2024-08-03T21:01:25.282 DEBUG:teuthology.run_tasks:Unwinding manager cephfs_test_runner 2024-08-03T21:01:25.291 DEBUG:teuthology.run_tasks:Unwinding manager kclient 2024-08-03T21:01:25.298 INFO:tasks.kclient:Unmounting kernel clients... 2024-08-03T21:01:25.299 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-08-03T21:01:25.299 DEBUG:teuthology.orchestra.run.smithi032:> dd if=/proc/self/mounts of=/dev/stdout 2024-08-03T21:01:25.341 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053453,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6499208k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 none /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249604k,nr_inodes=812401,mode=700,uid=1000,gid=1316 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6499208k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-08-03T21:01:25.341 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-08-03T21:01:25.342 DEBUG:teuthology.orchestra.run.smithi032:> dd if=/proc/self/mounts of=/dev/stdout 2024-08-03T21:01:25.419 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053453,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6499208k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 none /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249604k,nr_inodes=812401,mode=700,uid=1000,gid=1316 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6499208k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-08-03T21:01:25.419 DEBUG:teuthology.orchestra.run.smithi118:> set -ex 2024-08-03T21:01:25.420 DEBUG:teuthology.orchestra.run.smithi118:> dd if=/proc/self/mounts of=/dev/stdout 2024-08-03T21:01:25.460 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052663,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497944k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248972k,nr_inodes=812243,mode=700,uid=1000,gid=1316 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497944k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-08-03T21:01:25.460 DEBUG:teuthology.orchestra.run.smithi118:> set -ex 2024-08-03T21:01:25.460 DEBUG:teuthology.orchestra.run.smithi118:> dd if=/proc/self/mounts of=/dev/stdout 2024-08-03T21:01:25.538 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052663,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497944k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248972k,nr_inodes=812243,mode=700,uid=1000,gid=1316 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497944k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-08-03T21:01:25.538 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-08-03T21:01:25.538 DEBUG:teuthology.orchestra.run.smithi032:> ip netns list 2024-08-03T21:01:25.576 INFO:teuthology.orchestra.run.smithi032.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-08-03T21:01:25.576 INFO:teuthology.orchestra.run.smithi032.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-08-03T21:01:25.576 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-08-03T21:01:25.577 DEBUG:teuthology.orchestra.run.smithi032:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.0 2024-08-03T21:01:25.677 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-08-03T21:01:25.677 DEBUG:teuthology.orchestra.run.smithi032:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.2 2024-08-03T21:01:25.945 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-08-03T21:01:25.945 DEBUG:teuthology.orchestra.run.smithi032:> sudo ip link delete ceph-brx 2024-08-03T21:01:26.262 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-08-03T21:01:26.262 DEBUG:teuthology.orchestra.run.smithi032:> ip netns list 2024-08-03T21:01:26.298 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-08-03T21:01:26.298 DEBUG:teuthology.orchestra.run.smithi032:> sudo ip link delete ceph-brx 2024-08-03T21:01:26.397 INFO:teuthology.orchestra.run.smithi032.stderr:Cannot find device "ceph-brx" 2024-08-03T21:01:26.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-03T21:01:26.400 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-08-03T21:01:26.401 DEBUG:teuthology.orchestra.run.smithi118:> ip netns list 2024-08-03T21:01:26.441 INFO:teuthology.orchestra.run.smithi118.stdout:ceph-ns--home-ubuntu-cephtest-mnt.1 (id: 0) 2024-08-03T21:01:26.441 INFO:teuthology.orchestra.run.smithi118.stdout:ceph-ns--home-ubuntu-cephtest-mnt.3 (id: 1) 2024-08-03T21:01:26.442 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-08-03T21:01:26.442 DEBUG:teuthology.orchestra.run.smithi118:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.1 2024-08-03T21:01:26.543 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-08-03T21:01:26.544 DEBUG:teuthology.orchestra.run.smithi118:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.3 2024-08-03T21:01:26.819 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-08-03T21:01:26.819 DEBUG:teuthology.orchestra.run.smithi118:> sudo ip link delete ceph-brx 2024-08-03T21:01:27.112 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-08-03T21:01:27.112 DEBUG:teuthology.orchestra.run.smithi118:> ip netns list 2024-08-03T21:01:27.147 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-08-03T21:01:27.148 DEBUG:teuthology.orchestra.run.smithi118:> sudo ip link delete ceph-brx 2024-08-03T21:01:27.247 INFO:teuthology.orchestra.run.smithi118.stderr:Cannot find device "ceph-brx" 2024-08-03T21:01:27.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-03T21:01:27.251 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-08-03T21:01:27.260 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-08-03T21:01:27.260 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-08-03T21:01:27.499 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:27.498+0000 7f4636200640 1 -- 172.21.15.32:0/456876140 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f46300ade80 msgr2=0x7f46300ae290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:01:27.500 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:27.498+0000 7f4636200640 1 --2- 172.21.15.32:0/456876140 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f46300ade80 0x7f46300ae290 secure :-1 s=READY pgs=2265 cs=0 l=1 rev1=1 crypto rx=0x7f46300b2840 tx=0x7f462802aac0 comp rx=0 tx=0).stop 2024-08-03T21:01:27.500 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:27.498+0000 7f4636200640 1 -- 172.21.15.32:0/456876140 shutdown_connections 2024-08-03T21:01:27.500 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:27.498+0000 7f4636200640 1 --2- 172.21.15.32:0/456876140 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f46300ade80 0x7f46300ae290 unknown :-1 s=CLOSED pgs=2265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:01:27.500 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:27.498+0000 7f4636200640 1 --2- 172.21.15.32:0/456876140 >> v2:172.21.15.118:3300/0 conn(0x7f463006f720 0x7f463006fb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:01:27.501 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:27.498+0000 7f4636200640 1 -- 172.21.15.32:0/456876140 >> 172.21.15.32:0/456876140 conn(0x7f46300b1bd0 msgr2=0x7f46300b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T21:01:27.501 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:27.499+0000 7f4636200640 1 -- 172.21.15.32:0/456876140 shutdown_connections 2024-08-03T21:01:27.501 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:27.499+0000 7f4636200640 1 -- 172.21.15.32:0/456876140 wait complete. 2024-08-03T21:01:27.502 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:27.501+0000 7f4636200640 1 Processor -- start 2024-08-03T21:01:27.502 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:27.501+0000 7f4636200640 1 -- start start 2024-08-03T21:01:27.503 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:27.503+0000 7f4636200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f463006f720 0x7f4630150be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:01:27.504 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:27.503+0000 7f4636200640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f46300ade80 0x7f4630146960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:01:27.504 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:27.503+0000 7f4636200640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f46300b39a0 0x7f4630158750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:01:27.504 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:27.503+0000 7f4636200640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f46300aeb60 con 0x7f463006f720 2024-08-03T21:01:27.504 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:27.503+0000 7f4636200640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f46300ae9e0 con 0x7f46300b39a0 2024-08-03T21:01:27.504 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:27.503+0000 7f4636200640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7f46300aece0 con 0x7f46300ade80 2024-08-03T21:01:27.504 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:27.503+0000 7f462f400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f463006f720 0x7f4630150be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:01:27.504 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:27.503+0000 7f462f400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f463006f720 0x7f4630150be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:40182/0 (socket says 172.21.15.32:40182) 2024-08-03T21:01:27.505 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:27.503+0000 7f462f400640 1 -- 172.21.15.32:0/2983818151 learned_addr learned my addr 172.21.15.32:0/2983818151 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T21:01:27.505 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:27.503+0000 7f462fe00640 1 --2- 172.21.15.32:0/2983818151 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f46300b39a0 0x7f4630158750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:01:27.505 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:27.503+0000 7f4627e00640 1 --2- 172.21.15.32:0/2983818151 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f46300ade80 0x7f4630146960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:01:27.506 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:27.503+0000 7f462f400640 1 -- 172.21.15.32:0/2983818151 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f46300ade80 msgr2=0x7f4630146960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:01:27.506 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:27.503+0000 7f462f400640 1 --2- 172.21.15.32:0/2983818151 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f46300ade80 0x7f4630146960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:01:27.506 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:27.503+0000 7f462f400640 1 -- 172.21.15.32:0/2983818151 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f46300b39a0 msgr2=0x7f4630158750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:01:27.506 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:27.503+0000 7f462f400640 1 --2- 172.21.15.32:0/2983818151 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f46300b39a0 0x7f4630158750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:01:27.507 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:27.503+0000 7f462f400640 1 -- 172.21.15.32:0/2983818151 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4630146ea0 con 0x7f463006f720 2024-08-03T21:01:27.507 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:27.504+0000 7f462f400640 1 --2- 172.21.15.32:0/2983818151 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f463006f720 0x7f4630150be0 secure :-1 s=READY pgs=2266 cs=0 l=1 rev1=1 crypto rx=0x7f462000d740 tx=0x7f462000dc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T21:01:27.507 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:27.504+0000 7f4627400640 1 -- 172.21.15.32:0/2983818151 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4620017070 con 0x7f463006f720 2024-08-03T21:01:27.507 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:27.504+0000 7f4627400640 1 -- 172.21.15.32:0/2983818151 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f4620009d20 con 0x7f463006f720 2024-08-03T21:01:27.507 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:27.504+0000 7f4636200640 1 -- 172.21.15.32:0/2983818151 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4630156580 con 0x7f463006f720 2024-08-03T21:01:27.507 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:27.504+0000 7f4627400640 1 -- 172.21.15.32:0/2983818151 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46200106c0 con 0x7f463006f720 2024-08-03T21:01:27.507 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:27.504+0000 7f4636200640 1 -- 172.21.15.32:0/2983818151 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f46301f9b40 con 0x7f463006f720 2024-08-03T21:01:27.507 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:27.506+0000 7f4636200640 1 -- 172.21.15.32:0/2983818151 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4630074a90 con 0x7f463006f720 2024-08-03T21:01:27.508 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:27.506+0000 7f4627400640 1 -- 172.21.15.32:0/2983818151 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f462001a070 con 0x7f463006f720 2024-08-03T21:01:27.508 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:27.507+0000 7f4627400640 1 --2- 172.21.15.32:0/2983818151 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f460c077010 0x7f460c079520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:01:27.508 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:27.508+0000 7f4627400640 1 -- 172.21.15.32:0/2983818151 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(139..139 src has 1..139) ==== 5029+0+0 (secure 0 0 0) 0x7f46200987a0 con 0x7f463006f720 2024-08-03T21:01:27.509 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:27.508+0000 7f4627e00640 1 --2- 172.21.15.32:0/2983818151 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f460c077010 0x7f460c079520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:01:27.509 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:27.509+0000 7f4627e00640 1 --2- 172.21.15.32:0/2983818151 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f460c077010 0x7f460c079520 secure :-1 s=READY pgs=906 cs=0 l=1 rev1=1 crypto rx=0x7f46300cc250 tx=0x7f4618009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T21:01:27.513 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:27.512+0000 7f4627400640 1 -- 172.21.15.32:0/2983818151 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7f46200614d0 con 0x7f463006f720 2024-08-03T21:01:27.716 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:27.716+0000 7f4636200640 1 -- 172.21.15.32:0/2983818151 --> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f46300738d0 con 0x7f460c077010 2024-08-03T21:01:27.717 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:27.717+0000 7f4627400640 1 -- 172.21.15.32:0/2983818151 <== mgr.4106 v2:172.21.15.118:6800/4205950699 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+13471 (secure 0 0 0) 0x7f46300738d0 con 0x7f460c077010 2024-08-03T21:01:27.718 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-03T21:01:27.718 INFO:teuthology.orchestra.run.smithi032.stderr:dumped all 2024-08-03T21:01:27.720 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:27.719+0000 7f4636200640 1 -- 172.21.15.32:0/2983818151 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f460c077010 msgr2=0x7f460c079520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:01:27.720 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:27.719+0000 7f4636200640 1 --2- 172.21.15.32:0/2983818151 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f460c077010 0x7f460c079520 secure :-1 s=READY pgs=906 cs=0 l=1 rev1=1 crypto rx=0x7f46300cc250 tx=0x7f4618009040 comp rx=0 tx=0).stop 2024-08-03T21:01:27.720 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:27.719+0000 7f4636200640 1 -- 172.21.15.32:0/2983818151 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f463006f720 msgr2=0x7f4630150be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:01:27.720 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:27.719+0000 7f4636200640 1 --2- 172.21.15.32:0/2983818151 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f463006f720 0x7f4630150be0 secure :-1 s=READY pgs=2266 cs=0 l=1 rev1=1 crypto rx=0x7f462000d740 tx=0x7f462000dc00 comp rx=0 tx=0).stop 2024-08-03T21:01:27.720 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:27.719+0000 7f4636200640 1 -- 172.21.15.32:0/2983818151 shutdown_connections 2024-08-03T21:01:27.721 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:27.719+0000 7f4636200640 1 --2- 172.21.15.32:0/2983818151 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f460c077010 0x7f460c079520 unknown :-1 s=CLOSED pgs=906 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:01:27.721 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:27.719+0000 7f4636200640 1 --2- 172.21.15.32:0/2983818151 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f46300b39a0 0x7f4630158750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:01:27.721 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:27.719+0000 7f4636200640 1 --2- 172.21.15.32:0/2983818151 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f46300ade80 0x7f4630146960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:01:27.721 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:27.719+0000 7f4636200640 1 --2- 172.21.15.32:0/2983818151 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f463006f720 0x7f4630150be0 unknown :-1 s=CLOSED pgs=2266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:01:27.721 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:27.719+0000 7f4636200640 1 -- 172.21.15.32:0/2983818151 >> 172.21.15.32:0/2983818151 conn(0x7f46300b1bd0 msgr2=0x7f4630154f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T21:01:27.721 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:27.719+0000 7f4636200640 1 -- 172.21.15.32:0/2983818151 shutdown_connections 2024-08-03T21:01:27.721 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:27.719+0000 7f4636200640 1 -- 172.21.15.32:0/2983818151 wait complete. 2024-08-03T21:01:27.746 INFO:teuthology.orchestra.run.smithi032.stdout:{"pg_ready":true,"pg_map":{"version":551,"stamp":"2024-08-03T21:01:26.463910+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":177,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":3,"kb":754974720,"kb_used":310176,"kb_used_data":14968,"kb_used_omap":12,"kb_used_meta":295091,"kb_avail":754664544,"statfs":{"total":773094113280,"available":772776493056,"internally_reserved":0,"allocated":15327232,"data_stored":7980323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":302173745},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-36199,"num_objects":-28,"num_object_clones":0,"num_object_copies":-56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-28,"num_whiteouts":0,"num_read":-83,"num_read_kb":-85,"num_write":-95,"num_write_kb":-70,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":-17,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.959100"},"pg_stats":[{"pgid":"1.0","version":"16'72","reported_seq":339,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-08-03T21:01:24.550640+0000","last_change":"2024-08-03T20:47:30.547632+0000","last_active":"2024-08-03T21:01:24.550640+0000","last_peered":"2024-08-03T21:01:24.550640+0000","last_clean":"2024-08-03T21:01:24.550640+0000","last_became_active":"2024-08-03T20:47:30.546255+0000","last_became_peered":"2024-08-03T20:47:30.546255+0000","last_unstale":"2024-08-03T21:01:24.550640+0000","last_undegraded":"2024-08-03T21:01:24.550640+0000","last_fullsized":"2024-08-03T21:01:24.550640+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:47:28.874451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:47:28.874451+0000","last_clean_scrub_stamp":"2024-08-03T20:47:28.874451+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T00:19:40.548812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509610,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41804,"kb_used_data":2304,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330036,"statfs":{"total":96636764160,"available":96593956864,"internally_reserved":0,"allocated":2359296,"data_stored":1439610,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":40434122},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":15,"seq":64424509611,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":37064,"kb_used_data":1724,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334776,"statfs":{"total":96636764160,"available":96598810624,"internally_reserved":0,"allocated":1765376,"data_stored":849242,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":15,"seq":64424509611,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":41244,"kb_used_data":1740,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330596,"statfs":{"total":96636764160,"available":96594530304,"internally_reserved":0,"allocated":1781760,"data_stored":854893,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509610,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":41224,"kb_used_data":1724,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330616,"statfs":{"total":96636764160,"available":96594550784,"internally_reserved":0,"allocated":1765376,"data_stored":849242,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509612,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":29324,"kb_used_data":2304,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94342516,"statfs":{"total":96636764160,"available":96606736384,"internally_reserved":0,"allocated":2359296,"data_stored":1439610,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542315,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":37064,"kb_used_data":1724,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334776,"statfs":{"total":96636764160,"available":96598810624,"internally_reserved":0,"allocated":1765376,"data_stored":849242,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":14,"seq":60129542313,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":41224,"kb_used_data":1724,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330616,"statfs":{"total":96636764160,"available":96594550784,"internally_reserved":0,"allocated":1765376,"data_stored":849242,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":40434119},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":14,"seq":60129542314,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41228,"kb_used_data":1724,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330612,"statfs":{"total":96636764160,"available":96594546688,"internally_reserved":0,"allocated":1765376,"data_stored":849242,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-08-03T21:01:27.747 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-08-03T21:01:27.976 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:27.975+0000 7f2b4da00640 1 -- 172.21.15.32:0/1708847302 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2b480cfd30 msgr2=0x7f2b480d0140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:01:27.976 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:27.975+0000 7f2b4da00640 1 --2- 172.21.15.32:0/1708847302 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2b480cfd30 0x7f2b480d0140 secure :-1 s=READY pgs=2268 cs=0 l=1 rev1=1 crypto rx=0x7f2b48072af0 tx=0x7f2b3402a530 comp rx=0 tx=0).stop 2024-08-03T21:01:27.976 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:27.975+0000 7f2b4da00640 1 -- 172.21.15.32:0/1708847302 shutdown_connections 2024-08-03T21:01:27.976 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:27.975+0000 7f2b4da00640 1 --2- 172.21.15.32:0/1708847302 >> v2:172.21.15.118:3300/0 conn(0x7f2b48073ea0 0x7f2b480742b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:01:27.976 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:27.975+0000 7f2b4da00640 1 --2- 172.21.15.32:0/1708847302 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2b480cfd30 0x7f2b480d0140 unknown :-1 s=CLOSED pgs=2268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:01:27.977 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:27.975+0000 7f2b4da00640 1 -- 172.21.15.32:0/1708847302 >> 172.21.15.32:0/1708847302 conn(0x7f2b480b34b0 msgr2=0x7f2b480d1050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T21:01:27.977 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:27.975+0000 7f2b4da00640 1 -- 172.21.15.32:0/1708847302 shutdown_connections 2024-08-03T21:01:27.977 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:27.975+0000 7f2b4da00640 1 -- 172.21.15.32:0/1708847302 wait complete. 2024-08-03T21:01:27.978 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:27.977+0000 7f2b4da00640 1 Processor -- start 2024-08-03T21:01:27.978 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:27.977+0000 7f2b4da00640 1 -- start start 2024-08-03T21:01:27.980 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:27.979+0000 7f2b4da00640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f2b48073ea0 0x7f2b4813ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:01:27.980 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:27.979+0000 7f2b4da00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f2b480cfd30 0x7f2b4815e2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:01:27.980 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:27.979+0000 7f2b4da00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2b4815ffa0 0x7f2b481f5350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:01:27.981 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:27.979+0000 7f2b4da00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f2b4806cd70 con 0x7f2b4815ffa0 2024-08-03T21:01:27.981 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:27.979+0000 7f2b4da00640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f2b4806cbf0 con 0x7f2b480cfd30 2024-08-03T21:01:27.981 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:27.979+0000 7f2b4da00640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7f2b4806cef0 con 0x7f2b48073ea0 2024-08-03T21:01:27.981 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:27.979+0000 7f2b47400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2b4815ffa0 0x7f2b481f5350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:01:27.981 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:27.979+0000 7f2b47400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2b4815ffa0 0x7f2b481f5350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:40194/0 (socket says 172.21.15.32:40194) 2024-08-03T21:01:27.981 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:27.979+0000 7f2b47400640 1 -- 172.21.15.32:0/2698371166 learned_addr learned my addr 172.21.15.32:0/2698371166 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T21:01:27.981 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:27.979+0000 7f2b46000640 1 --2- 172.21.15.32:0/2698371166 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f2b480cfd30 0x7f2b4815e2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:01:27.981 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:27.979+0000 7f2b46a00640 1 --2- 172.21.15.32:0/2698371166 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f2b48073ea0 0x7f2b4813ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:01:27.982 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:27.980+0000 7f2b47400640 1 -- 172.21.15.32:0/2698371166 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f2b48073ea0 msgr2=0x7f2b4813ede0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:01:27.982 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:27.980+0000 7f2b47400640 1 --2- 172.21.15.32:0/2698371166 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f2b48073ea0 0x7f2b4813ede0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:01:27.982 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:27.980+0000 7f2b47400640 1 -- 172.21.15.32:0/2698371166 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f2b480cfd30 msgr2=0x7f2b4815e2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:01:27.982 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:27.980+0000 7f2b47400640 1 --2- 172.21.15.32:0/2698371166 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f2b480cfd30 0x7f2b4815e2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:01:27.982 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:27.980+0000 7f2b47400640 1 -- 172.21.15.32:0/2698371166 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2b480b5610 con 0x7f2b4815ffa0 2024-08-03T21:01:27.982 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:27.980+0000 7f2b46a00640 1 --2- 172.21.15.32:0/2698371166 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f2b48073ea0 0x7f2b4813ede0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-03T21:01:27.982 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:27.980+0000 7f2b47400640 1 --2- 172.21.15.32:0/2698371166 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2b4815ffa0 0x7f2b481f5350 secure :-1 s=READY pgs=2269 cs=0 l=1 rev1=1 crypto rx=0x7f2b4000ea40 tx=0x7f2b4000ef00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T21:01:27.984 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:27.980+0000 7f2b3ea00640 1 -- 172.21.15.32:0/2698371166 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b4000ce40 con 0x7f2b4815ffa0 2024-08-03T21:01:27.984 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:27.980+0000 7f2b3ea00640 1 -- 172.21.15.32:0/2698371166 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f2b40004c10 con 0x7f2b4815ffa0 2024-08-03T21:01:27.984 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:27.980+0000 7f2b3ea00640 1 -- 172.21.15.32:0/2698371166 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b40005020 con 0x7f2b4815ffa0 2024-08-03T21:01:27.984 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:27.980+0000 7f2b4da00640 1 -- 172.21.15.32:0/2698371166 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2b4815d560 con 0x7f2b4815ffa0 2024-08-03T21:01:27.984 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:27.980+0000 7f2b4da00640 1 -- 172.21.15.32:0/2698371166 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2b4815dae0 con 0x7f2b4815ffa0 2024-08-03T21:01:27.984 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:27.981+0000 7f2b4da00640 1 -- 172.21.15.32:0/2698371166 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2b480742b0 con 0x7f2b4815ffa0 2024-08-03T21:01:27.985 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:27.983+0000 7f2b3ea00640 1 -- 172.21.15.32:0/2698371166 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f2b400051c0 con 0x7f2b4815ffa0 2024-08-03T21:01:27.985 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:27.984+0000 7f2b3ea00640 1 --2- 172.21.15.32:0/2698371166 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f2b28077010 0x7f2b28079520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:01:27.985 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:27.984+0000 7f2b3ea00640 1 -- 172.21.15.32:0/2698371166 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(139..139 src has 1..139) ==== 5029+0+0 (secure 0 0 0) 0x7f2b40099320 con 0x7f2b4815ffa0 2024-08-03T21:01:27.986 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:27.984+0000 7f2b46a00640 1 --2- 172.21.15.32:0/2698371166 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f2b28077010 0x7f2b28079520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:01:27.986 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:27.985+0000 7f2b46a00640 1 --2- 172.21.15.32:0/2698371166 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f2b28077010 0x7f2b28079520 secure :-1 s=READY pgs=907 cs=0 l=1 rev1=1 crypto rx=0x7f2b38000f60 tx=0x7f2b38009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T21:01:27.988 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:27.988+0000 7f2b3ea00640 1 -- 172.21.15.32:0/2698371166 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7f2b40062850 con 0x7f2b4815ffa0 2024-08-03T21:01:28.190 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:28.190+0000 7f2b4da00640 1 -- 172.21.15.32:0/2698371166 --> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f2b480b53a0 con 0x7f2b28077010 2024-08-03T21:01:28.191 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:28.191+0000 7f2b3ea00640 1 -- 172.21.15.32:0/2698371166 <== mgr.4106 v2:172.21.15.118:6800/4205950699 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+13471 (secure 0 0 0) 0x7f2b480b53a0 con 0x7f2b28077010 2024-08-03T21:01:28.192 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-03T21:01:28.192 INFO:teuthology.orchestra.run.smithi032.stderr:dumped all 2024-08-03T21:01:28.195 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:28.193+0000 7f2b4da00640 1 -- 172.21.15.32:0/2698371166 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f2b28077010 msgr2=0x7f2b28079520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:01:28.195 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:28.193+0000 7f2b4da00640 1 --2- 172.21.15.32:0/2698371166 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f2b28077010 0x7f2b28079520 secure :-1 s=READY pgs=907 cs=0 l=1 rev1=1 crypto rx=0x7f2b38000f60 tx=0x7f2b38009040 comp rx=0 tx=0).stop 2024-08-03T21:01:28.195 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:28.193+0000 7f2b4da00640 1 -- 172.21.15.32:0/2698371166 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2b4815ffa0 msgr2=0x7f2b481f5350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:01:28.195 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:28.193+0000 7f2b4da00640 1 --2- 172.21.15.32:0/2698371166 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2b4815ffa0 0x7f2b481f5350 secure :-1 s=READY pgs=2269 cs=0 l=1 rev1=1 crypto rx=0x7f2b4000ea40 tx=0x7f2b4000ef00 comp rx=0 tx=0).stop 2024-08-03T21:01:28.195 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:28.193+0000 7f2b4da00640 1 -- 172.21.15.32:0/2698371166 shutdown_connections 2024-08-03T21:01:28.195 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:28.193+0000 7f2b4da00640 1 --2- 172.21.15.32:0/2698371166 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f2b28077010 0x7f2b28079520 unknown :-1 s=CLOSED pgs=907 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:01:28.195 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:28.193+0000 7f2b4da00640 1 --2- 172.21.15.32:0/2698371166 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f2b480cfd30 0x7f2b4815e2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:01:28.195 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:28.193+0000 7f2b4da00640 1 --2- 172.21.15.32:0/2698371166 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2b4815ffa0 0x7f2b481f5350 unknown :-1 s=CLOSED pgs=2269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:01:28.196 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:28.193+0000 7f2b4da00640 1 --2- 172.21.15.32:0/2698371166 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f2b48073ea0 0x7f2b4813ede0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:01:28.196 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:28.193+0000 7f2b4da00640 1 -- 172.21.15.32:0/2698371166 >> 172.21.15.32:0/2698371166 conn(0x7f2b480b34b0 msgr2=0x7f2b480afa00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T21:01:28.196 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:28.193+0000 7f2b4da00640 1 -- 172.21.15.32:0/2698371166 shutdown_connections 2024-08-03T21:01:28.196 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:28.193+0000 7f2b4da00640 1 -- 172.21.15.32:0/2698371166 wait complete. 2024-08-03T21:01:28.216 INFO:teuthology.orchestra.run.smithi032.stdout:{"pg_ready":true,"pg_map":{"version":551,"stamp":"2024-08-03T21:01:26.463910+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":177,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":3,"kb":754974720,"kb_used":310176,"kb_used_data":14968,"kb_used_omap":12,"kb_used_meta":295091,"kb_avail":754664544,"statfs":{"total":773094113280,"available":772776493056,"internally_reserved":0,"allocated":15327232,"data_stored":7980323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":302173745},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-36199,"num_objects":-28,"num_object_clones":0,"num_object_copies":-56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-28,"num_whiteouts":0,"num_read":-83,"num_read_kb":-85,"num_write":-95,"num_write_kb":-70,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":-17,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.959100"},"pg_stats":[{"pgid":"1.0","version":"16'72","reported_seq":339,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-08-03T21:01:24.550640+0000","last_change":"2024-08-03T20:47:30.547632+0000","last_active":"2024-08-03T21:01:24.550640+0000","last_peered":"2024-08-03T21:01:24.550640+0000","last_clean":"2024-08-03T21:01:24.550640+0000","last_became_active":"2024-08-03T20:47:30.546255+0000","last_became_peered":"2024-08-03T20:47:30.546255+0000","last_unstale":"2024-08-03T21:01:24.550640+0000","last_undegraded":"2024-08-03T21:01:24.550640+0000","last_fullsized":"2024-08-03T21:01:24.550640+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:47:28.874451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:47:28.874451+0000","last_clean_scrub_stamp":"2024-08-03T20:47:28.874451+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T00:19:40.548812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509610,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41804,"kb_used_data":2304,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330036,"statfs":{"total":96636764160,"available":96593956864,"internally_reserved":0,"allocated":2359296,"data_stored":1439610,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":40434122},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":15,"seq":64424509611,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":37064,"kb_used_data":1724,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334776,"statfs":{"total":96636764160,"available":96598810624,"internally_reserved":0,"allocated":1765376,"data_stored":849242,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":15,"seq":64424509611,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":41244,"kb_used_data":1740,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330596,"statfs":{"total":96636764160,"available":96594530304,"internally_reserved":0,"allocated":1781760,"data_stored":854893,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509610,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":41224,"kb_used_data":1724,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330616,"statfs":{"total":96636764160,"available":96594550784,"internally_reserved":0,"allocated":1765376,"data_stored":849242,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509612,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":29324,"kb_used_data":2304,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94342516,"statfs":{"total":96636764160,"available":96606736384,"internally_reserved":0,"allocated":2359296,"data_stored":1439610,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542315,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":37064,"kb_used_data":1724,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334776,"statfs":{"total":96636764160,"available":96598810624,"internally_reserved":0,"allocated":1765376,"data_stored":849242,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":14,"seq":60129542313,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":41224,"kb_used_data":1724,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330616,"statfs":{"total":96636764160,"available":96594550784,"internally_reserved":0,"allocated":1765376,"data_stored":849242,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":40434119},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":14,"seq":60129542314,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41228,"kb_used_data":1724,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330612,"statfs":{"total":96636764160,"available":96594546688,"internally_reserved":0,"allocated":1765376,"data_stored":849242,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-08-03T21:01:28.217 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-08-03T21:01:28.217 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-08-03T21:01:28.445 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:28.444+0000 7fbb3b200640 1 -- 172.21.15.32:0/2111096185 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbb3c06e100 msgr2=0x7fbb3c153270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:01:28.445 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:28.444+0000 7fbb3b200640 1 --2- 172.21.15.32:0/2111096185 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbb3c06e100 0x7fbb3c153270 secure :-1 s=READY pgs=2271 cs=0 l=1 rev1=1 crypto rx=0x7fbb34009ca0 tx=0x7fbb3402aa50 comp rx=0 tx=0).stop 2024-08-03T21:01:28.445 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:28.444+0000 7fbb3b200640 1 -- 172.21.15.32:0/2111096185 shutdown_connections 2024-08-03T21:01:28.445 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:28.444+0000 7fbb3b200640 1 --2- 172.21.15.32:0/2111096185 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbb3c06e100 0x7fbb3c153270 unknown :-1 s=CLOSED pgs=2271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:01:28.445 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:28.444+0000 7fbb3b200640 1 -- 172.21.15.32:0/2111096185 >> 172.21.15.32:0/2111096185 conn(0x7fbb3c072e70 msgr2=0x7fbb3c073280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T21:01:28.446 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:28.445+0000 7fbb3b200640 1 -- 172.21.15.32:0/2111096185 shutdown_connections 2024-08-03T21:01:28.446 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:28.445+0000 7fbb3b200640 1 -- 172.21.15.32:0/2111096185 wait complete. 2024-08-03T21:01:28.447 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:28.447+0000 7fbb3b200640 1 Processor -- start 2024-08-03T21:01:28.447 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:28.447+0000 7fbb3b200640 1 -- start start 2024-08-03T21:01:28.450 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:28.448+0000 7fbb3b200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbb3c073990 0x7fbb3c1f3aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:01:28.450 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:28.448+0000 7fbb3b200640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fbb3c0cd860 0x7fbb3c155690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:01:28.450 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:28.448+0000 7fbb3b200640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fbb3c0b4770 0x7fbb3c0b4b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:01:28.450 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:28.448+0000 7fbb3b200640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fbb3c167730 con 0x7fbb3c073990 2024-08-03T21:01:28.450 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:28.448+0000 7fbb3b200640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7fbb3c1675b0 con 0x7fbb3c0b4770 2024-08-03T21:01:28.450 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:28.448+0000 7fbb3b200640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7fbb3c1678b0 con 0x7fbb3c0cd860 2024-08-03T21:01:28.451 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:28.449+0000 7fbb39e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbb3c073990 0x7fbb3c1f3aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:01:28.451 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:28.449+0000 7fbb39e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbb3c073990 0x7fbb3c1f3aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:40218/0 (socket says 172.21.15.32:40218) 2024-08-03T21:01:28.451 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:28.449+0000 7fbb39e00640 1 -- 172.21.15.32:0/587559823 learned_addr learned my addr 172.21.15.32:0/587559823 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T21:01:28.451 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:28.449+0000 7fbb31400640 1 --2- 172.21.15.32:0/587559823 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fbb3c0cd860 0x7fbb3c155690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:01:28.451 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:28.449+0000 7fbb3a800640 1 --2- 172.21.15.32:0/587559823 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fbb3c0b4770 0x7fbb3c0b4b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:01:28.451 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:28.449+0000 7fbb39e00640 1 -- 172.21.15.32:0/587559823 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fbb3c0cd860 msgr2=0x7fbb3c155690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:01:28.451 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:28.449+0000 7fbb39e00640 1 --2- 172.21.15.32:0/587559823 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fbb3c0cd860 0x7fbb3c155690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:01:28.452 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:28.449+0000 7fbb39e00640 1 -- 172.21.15.32:0/587559823 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fbb3c0b4770 msgr2=0x7fbb3c0b4b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:01:28.452 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:28.449+0000 7fbb39e00640 1 --2- 172.21.15.32:0/587559823 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fbb3c0b4770 0x7fbb3c0b4b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:01:28.452 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:28.449+0000 7fbb39e00640 1 -- 172.21.15.32:0/587559823 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbb3c154130 con 0x7fbb3c073990 2024-08-03T21:01:28.452 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:28.449+0000 7fbb39e00640 1 --2- 172.21.15.32:0/587559823 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbb3c073990 0x7fbb3c1f3aa0 secure :-1 s=READY pgs=2272 cs=0 l=1 rev1=1 crypto rx=0x7fbb2c00c340 tx=0x7fbb2c00c800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T21:01:28.453 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:28.450+0000 7fbb32a00640 1 -- 172.21.15.32:0/587559823 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb2c017070 con 0x7fbb3c073990 2024-08-03T21:01:28.454 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:28.450+0000 7fbb32a00640 1 -- 172.21.15.32:0/587559823 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fbb2c0035f0 con 0x7fbb3c073990 2024-08-03T21:01:28.454 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:28.450+0000 7fbb32a00640 1 -- 172.21.15.32:0/587559823 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb2c003a80 con 0x7fbb3c073990 2024-08-03T21:01:28.454 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:28.450+0000 7fbb3b200640 1 -- 172.21.15.32:0/587559823 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbb3c0b50c0 con 0x7fbb3c073990 2024-08-03T21:01:28.454 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:28.450+0000 7fbb3b200640 1 -- 172.21.15.32:0/587559823 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbb3c153b70 con 0x7fbb3c073990 2024-08-03T21:01:28.454 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:28.451+0000 7fbb3b200640 1 -- 172.21.15.32:0/587559823 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbb3c153270 con 0x7fbb3c073990 2024-08-03T21:01:28.454 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:28.452+0000 7fbb32a00640 1 -- 172.21.15.32:0/587559823 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7fbb2c01a070 con 0x7fbb3c073990 2024-08-03T21:01:28.454 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:28.453+0000 7fbb32a00640 1 --2- 172.21.15.32:0/587559823 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fbb18077040 0x7fbb18079550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:01:28.455 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:28.453+0000 7fbb32a00640 1 -- 172.21.15.32:0/587559823 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(139..139 src has 1..139) ==== 5029+0+0 (secure 0 0 0) 0x7fbb2c0979e0 con 0x7fbb3c073990 2024-08-03T21:01:28.455 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:28.454+0000 7fbb31400640 1 --2- 172.21.15.32:0/587559823 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fbb18077040 0x7fbb18079550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:01:28.455 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:28.454+0000 7fbb31400640 1 --2- 172.21.15.32:0/587559823 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fbb18077040 0x7fbb18079550 secure :-1 s=READY pgs=908 cs=0 l=1 rev1=1 crypto rx=0x7fbb3c0cc880 tx=0x7fbb24009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T21:01:28.458 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:28.458+0000 7fbb32a00640 1 -- 172.21.15.32:0/587559823 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7fbb3c0b50c0 con 0x7fbb3c073990 2024-08-03T21:01:28.661 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:28.661+0000 7fbb3b200640 1 -- 172.21.15.32:0/587559823 --> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fbb3c06d770 con 0x7fbb18077040 2024-08-03T21:01:28.663 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:28.662+0000 7fbb32a00640 1 -- 172.21.15.32:0/587559823 <== mgr.4106 v2:172.21.15.118:6800/4205950699 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+13471 (secure 0 0 0) 0x7fbb3c06d770 con 0x7fbb18077040 2024-08-03T21:01:28.663 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-03T21:01:28.663 INFO:teuthology.orchestra.run.smithi032.stderr:dumped all 2024-08-03T21:01:28.665 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:28.664+0000 7fbb3b200640 1 -- 172.21.15.32:0/587559823 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fbb18077040 msgr2=0x7fbb18079550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:01:28.665 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:28.664+0000 7fbb3b200640 1 --2- 172.21.15.32:0/587559823 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fbb18077040 0x7fbb18079550 secure :-1 s=READY pgs=908 cs=0 l=1 rev1=1 crypto rx=0x7fbb3c0cc880 tx=0x7fbb24009040 comp rx=0 tx=0).stop 2024-08-03T21:01:28.665 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:28.664+0000 7fbb3b200640 1 -- 172.21.15.32:0/587559823 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbb3c073990 msgr2=0x7fbb3c1f3aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:01:28.665 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:28.664+0000 7fbb3b200640 1 --2- 172.21.15.32:0/587559823 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbb3c073990 0x7fbb3c1f3aa0 secure :-1 s=READY pgs=2272 cs=0 l=1 rev1=1 crypto rx=0x7fbb2c00c340 tx=0x7fbb2c00c800 comp rx=0 tx=0).stop 2024-08-03T21:01:28.666 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:28.664+0000 7fbb3b200640 1 -- 172.21.15.32:0/587559823 shutdown_connections 2024-08-03T21:01:28.666 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:28.664+0000 7fbb3b200640 1 --2- 172.21.15.32:0/587559823 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fbb18077040 0x7fbb18079550 unknown :-1 s=CLOSED pgs=908 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:01:28.666 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:28.664+0000 7fbb3b200640 1 --2- 172.21.15.32:0/587559823 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fbb3c0b4770 0x7fbb3c0b4b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:01:28.666 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:28.664+0000 7fbb3b200640 1 --2- 172.21.15.32:0/587559823 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fbb3c0cd860 0x7fbb3c155690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:01:28.666 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:28.664+0000 7fbb3b200640 1 --2- 172.21.15.32:0/587559823 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbb3c073990 0x7fbb3c1f3aa0 unknown :-1 s=CLOSED pgs=2272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:01:28.666 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:28.664+0000 7fbb3b200640 1 -- 172.21.15.32:0/587559823 >> 172.21.15.32:0/587559823 conn(0x7fbb3c072e70 msgr2=0x7fbb3c0cf0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T21:01:28.666 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:28.665+0000 7fbb3b200640 1 -- 172.21.15.32:0/587559823 shutdown_connections 2024-08-03T21:01:28.666 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:28.665+0000 7fbb3b200640 1 -- 172.21.15.32:0/587559823 wait complete. 2024-08-03T21:01:28.690 INFO:teuthology.orchestra.run.smithi032.stdout:{"pg_ready":true,"pg_map":{"version":552,"stamp":"2024-08-03T21:01:28.464293+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":177,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":3,"kb":754974720,"kb_used":310176,"kb_used_data":14968,"kb_used_omap":12,"kb_used_meta":295091,"kb_avail":754664544,"statfs":{"total":773094113280,"available":772776493056,"internally_reserved":0,"allocated":15327232,"data_stored":7980323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":302173745},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-36340,"num_objects":-28,"num_object_clones":0,"num_object_copies":-56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-28,"num_whiteouts":0,"num_read":-84,"num_read_kb":-86,"num_write":-96,"num_write_kb":-71,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":-17,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002044"},"pg_stats":[{"pgid":"1.0","version":"16'72","reported_seq":339,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-08-03T21:01:24.550640+0000","last_change":"2024-08-03T20:47:30.547632+0000","last_active":"2024-08-03T21:01:24.550640+0000","last_peered":"2024-08-03T21:01:24.550640+0000","last_clean":"2024-08-03T21:01:24.550640+0000","last_became_active":"2024-08-03T20:47:30.546255+0000","last_became_peered":"2024-08-03T20:47:30.546255+0000","last_unstale":"2024-08-03T21:01:24.550640+0000","last_undegraded":"2024-08-03T21:01:24.550640+0000","last_fullsized":"2024-08-03T21:01:24.550640+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:47:28.874451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:47:28.874451+0000","last_clean_scrub_stamp":"2024-08-03T20:47:28.874451+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T00:19:40.548812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509610,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41804,"kb_used_data":2304,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330036,"statfs":{"total":96636764160,"available":96593956864,"internally_reserved":0,"allocated":2359296,"data_stored":1439610,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":40434122},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":15,"seq":64424509611,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":37064,"kb_used_data":1724,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334776,"statfs":{"total":96636764160,"available":96598810624,"internally_reserved":0,"allocated":1765376,"data_stored":849242,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":15,"seq":64424509611,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":41244,"kb_used_data":1740,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330596,"statfs":{"total":96636764160,"available":96594530304,"internally_reserved":0,"allocated":1781760,"data_stored":854893,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509610,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":41224,"kb_used_data":1724,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330616,"statfs":{"total":96636764160,"available":96594550784,"internally_reserved":0,"allocated":1765376,"data_stored":849242,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509612,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":29324,"kb_used_data":2304,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94342516,"statfs":{"total":96636764160,"available":96606736384,"internally_reserved":0,"allocated":2359296,"data_stored":1439610,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542315,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":37064,"kb_used_data":1724,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334776,"statfs":{"total":96636764160,"available":96598810624,"internally_reserved":0,"allocated":1765376,"data_stored":849242,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":14,"seq":60129542313,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":41224,"kb_used_data":1724,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330616,"statfs":{"total":96636764160,"available":96594550784,"internally_reserved":0,"allocated":1765376,"data_stored":849242,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":40434119},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":14,"seq":60129542314,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41228,"kb_used_data":1724,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330612,"statfs":{"total":96636764160,"available":96594546688,"internally_reserved":0,"allocated":1765376,"data_stored":849242,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-08-03T21:01:28.691 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-08-03T21:01:28.922 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:28.921+0000 7f4d79800640 1 -- 172.21.15.32:0/2257968822 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4d58003c30 msgr2=0x7f4d58004080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:01:28.922 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:28.921+0000 7f4d79800640 1 --2- 172.21.15.32:0/2257968822 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4d58003c30 0x7f4d58004080 secure :-1 s=READY pgs=2274 cs=0 l=1 rev1=1 crypto rx=0x7f4d64001830 tx=0x7f4d58005830 comp rx=0 tx=0).stop 2024-08-03T21:01:28.923 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:28.921+0000 7f4d79800640 1 -- 172.21.15.32:0/2257968822 shutdown_connections 2024-08-03T21:01:28.923 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:28.921+0000 7f4d79800640 1 --2- 172.21.15.32:0/2257968822 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4d58003c30 0x7f4d58004080 unknown :-1 s=CLOSED pgs=2274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:01:28.923 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:28.921+0000 7f4d79800640 1 -- 172.21.15.32:0/2257968822 >> 172.21.15.32:0/2257968822 conn(0x7f4d7408d7a0 msgr2=0x7f4d7409d3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T21:01:28.923 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:28.921+0000 7f4d79800640 1 -- 172.21.15.32:0/2257968822 shutdown_connections 2024-08-03T21:01:28.923 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:28.921+0000 7f4d79800640 1 -- 172.21.15.32:0/2257968822 wait complete. 2024-08-03T21:01:28.924 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:28.923+0000 7f4d79800640 1 Processor -- start 2024-08-03T21:01:28.924 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:28.924+0000 7f4d79800640 1 -- start start 2024-08-03T21:01:28.926 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:28.925+0000 7f4d79800640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f4d58003c30 0x7f4d740b4af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:01:28.926 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:28.925+0000 7f4d79800640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f4d740b5030 0x7f4d741f8ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:01:28.927 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:28.925+0000 7f4d79800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4d741f90e0 0x7f4d7414beb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:01:28.927 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:28.925+0000 7f4d79800640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f4d74161b80 con 0x7f4d741f90e0 2024-08-03T21:01:28.927 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:28.925+0000 7f4d79800640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f4d740cfe00 con 0x7f4d58003c30 2024-08-03T21:01:28.927 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:28.925+0000 7f4d79800640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7f4d74161d00 con 0x7f4d740b5030 2024-08-03T21:01:28.927 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:28.925+0000 7f4d78e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4d741f90e0 0x7f4d7414beb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:01:28.927 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:28.925+0000 7f4d78e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4d741f90e0 0x7f4d7414beb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:40238/0 (socket says 172.21.15.32:40238) 2024-08-03T21:01:28.927 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:28.925+0000 7f4d78e00640 1 -- 172.21.15.32:0/4218543462 learned_addr learned my addr 172.21.15.32:0/4218543462 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T21:01:28.927 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:28.925+0000 7f4d6be00640 1 --2- 172.21.15.32:0/4218543462 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f4d740b5030 0x7f4d741f8ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:01:28.928 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:28.925+0000 7f4d73e00640 1 --2- 172.21.15.32:0/4218543462 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f4d58003c30 0x7f4d740b4af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:01:28.928 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:28.926+0000 7f4d73e00640 1 -- 172.21.15.32:0/4218543462 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f4d740b5030 msgr2=0x7f4d741f8ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:01:28.928 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:28.926+0000 7f4d73e00640 1 --2- 172.21.15.32:0/4218543462 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f4d740b5030 0x7f4d741f8ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:01:28.928 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:28.926+0000 7f4d73e00640 1 -- 172.21.15.32:0/4218543462 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4d741f90e0 msgr2=0x7f4d7414beb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:01:28.928 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:28.926+0000 7f4d73e00640 1 --2- 172.21.15.32:0/4218543462 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4d741f90e0 0x7f4d7414beb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:01:28.928 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:28.926+0000 7f4d73e00640 1 -- 172.21.15.32:0/4218543462 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4d740b5440 con 0x7f4d58003c30 2024-08-03T21:01:28.928 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:28.926+0000 7f4d6be00640 1 --2- 172.21.15.32:0/4218543462 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f4d740b5030 0x7f4d741f8ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-03T21:01:28.928 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:28.926+0000 7f4d78e00640 1 --2- 172.21.15.32:0/4218543462 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4d741f90e0 0x7f4d7414beb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-03T21:01:28.928 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:28.926+0000 7f4d73e00640 1 --2- 172.21.15.32:0/4218543462 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f4d58003c30 0x7f4d740b4af0 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7f4d64004000 tx=0x7f4d640061b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T21:01:28.929 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:28.926+0000 7f4d70c00640 1 -- 172.21.15.32:0/4218543462 <== mon.1 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d64043070 con 0x7f4d58003c30 2024-08-03T21:01:28.930 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:28.926+0000 7f4d70c00640 1 -- 172.21.15.32:0/4218543462 <== mon.1 v2:172.21.15.118:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f4d64003af0 con 0x7f4d58003c30 2024-08-03T21:01:28.930 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:28.926+0000 7f4d70c00640 1 -- 172.21.15.32:0/4218543462 <== mon.1 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d64033480 con 0x7f4d58003c30 2024-08-03T21:01:28.930 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:28.926+0000 7f4d79800640 1 -- 172.21.15.32:0/4218543462 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4d740ca660 con 0x7f4d58003c30 2024-08-03T21:01:28.930 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:28.927+0000 7f4d79800640 1 -- 172.21.15.32:0/4218543462 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4d741f5790 con 0x7f4d58003c30 2024-08-03T21:01:28.930 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:28.927+0000 7f4d79800640 1 -- 172.21.15.32:0/4218543462 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4d740b3e90 con 0x7f4d58003c30 2024-08-03T21:01:28.930 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:28.929+0000 7f4d70c00640 1 -- 172.21.15.32:0/4218543462 <== mon.1 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f4d64033620 con 0x7f4d58003c30 2024-08-03T21:01:28.931 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:28.930+0000 7f4d70c00640 1 --2- 172.21.15.32:0/4218543462 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f4d50077040 0x7f4d50079550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:01:28.931 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:28.930+0000 7f4d70c00640 1 -- 172.21.15.32:0/4218543462 <== mon.1 v2:172.21.15.118:3300/0 5 ==== osd_map(139..139 src has 1..139) ==== 5029+0+0 (secure 0 0 0) 0x7f4d640b8390 con 0x7f4d58003c30 2024-08-03T21:01:28.931 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:28.930+0000 7f4d6be00640 1 --2- 172.21.15.32:0/4218543462 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f4d50077040 0x7f4d50079550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:01:28.932 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:28.931+0000 7f4d6be00640 1 --2- 172.21.15.32:0/4218543462 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f4d50077040 0x7f4d50079550 secure :-1 s=READY pgs=909 cs=0 l=1 rev1=1 crypto rx=0x7f4d5c0011a0 tx=0x7f4d5c001110 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T21:01:28.933 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:28.933+0000 7f4d70c00640 1 -- 172.21.15.32:0/4218543462 <== mon.1 v2:172.21.15.118:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7f4d740b5440 con 0x7f4d58003c30 2024-08-03T21:01:29.141 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:29.140+0000 7f4d79800640 1 -- 172.21.15.32:0/4218543462 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f4d740b6640 con 0x7f4d58003c30 2024-08-03T21:01:29.142 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:29.141+0000 7f4d70c00640 1 -- 172.21.15.32:0/4218543462 <== mon.1 v2:172.21.15.118:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v139) ==== 74+0+13100 (secure 0 0 0) 0x7f4d64081310 con 0x7f4d58003c30 2024-08-03T21:01:29.142 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-03T21:01:29.142 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":139,"fsid":"40ab141a-01d7-49b9-ae3c-2edec9876e95","created":"2024-08-03T20:47:14.957362+0000","modified":"2024-08-03T21:01:22.930784+0000","last_up_change":"2024-08-03T20:47:28.874451+0000","last_in_change":"2024-08-03T20:47:20.079442+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":15,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-08-03T20:47:28.261300+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5a83e759-9035-438b-889a-0a6af1f0a0d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":133,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6816","nonce":3066156116},{"type":"v1","addr":"172.21.15.32:6817","nonce":3066156116}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6818","nonce":3066156116},{"type":"v1","addr":"172.21.15.32:6819","nonce":3066156116}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6822","nonce":3066156116},{"type":"v1","addr":"172.21.15.32:6823","nonce":3066156116}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6820","nonce":3066156116},{"type":"v1","addr":"172.21.15.32:6821","nonce":3066156116}]},"public_addr":"172.21.15.32:6817/3066156116","cluster_addr":"172.21.15.32:6819/3066156116","heartbeat_back_addr":"172.21.15.32:6823/3066156116","heartbeat_front_addr":"172.21.15.32:6821/3066156116","state":["exists","up"]},{"osd":1,"uuid":"d7c5c210-da4f-49f9-b9e7-1dfb02daa4bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":133,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6824","nonce":1551241759},{"type":"v1","addr":"172.21.15.32:6825","nonce":1551241759}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6826","nonce":1551241759},{"type":"v1","addr":"172.21.15.32:6827","nonce":1551241759}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6830","nonce":1551241759},{"type":"v1","addr":"172.21.15.32:6831","nonce":1551241759}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6828","nonce":1551241759},{"type":"v1","addr":"172.21.15.32:6829","nonce":1551241759}]},"public_addr":"172.21.15.32:6825/1551241759","cluster_addr":"172.21.15.32:6827/1551241759","heartbeat_back_addr":"172.21.15.32:6831/1551241759","heartbeat_front_addr":"172.21.15.32:6829/1551241759","state":["exists","up"]},{"osd":2,"uuid":"57f2e6de-a201-45c2-8981-8e56a4875ebd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":133,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6808","nonce":3806708944},{"type":"v1","addr":"172.21.15.32:6809","nonce":3806708944}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6810","nonce":3806708944},{"type":"v1","addr":"172.21.15.32:6811","nonce":3806708944}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6814","nonce":3806708944},{"type":"v1","addr":"172.21.15.32:6815","nonce":3806708944}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6812","nonce":3806708944},{"type":"v1","addr":"172.21.15.32:6813","nonce":3806708944}]},"public_addr":"172.21.15.32:6809/3806708944","cluster_addr":"172.21.15.32:6811/3806708944","heartbeat_back_addr":"172.21.15.32:6815/3806708944","heartbeat_front_addr":"172.21.15.32:6813/3806708944","state":["exists","up"]},{"osd":3,"uuid":"6439a280-da45-4148-8130-f16323935ff3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":133,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6800","nonce":9088303},{"type":"v1","addr":"172.21.15.32:6801","nonce":9088303}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6802","nonce":9088303},{"type":"v1","addr":"172.21.15.32:6803","nonce":9088303}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6806","nonce":9088303},{"type":"v1","addr":"172.21.15.32:6807","nonce":9088303}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6804","nonce":9088303},{"type":"v1","addr":"172.21.15.32:6805","nonce":9088303}]},"public_addr":"172.21.15.32:6801/9088303","cluster_addr":"172.21.15.32:6803/9088303","heartbeat_back_addr":"172.21.15.32:6807/9088303","heartbeat_front_addr":"172.21.15.32:6805/9088303","state":["exists","up"]},{"osd":4,"uuid":"4f902eff-1e18-418d-82c9-2151c69524fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":133,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6802","nonce":864752275},{"type":"v1","addr":"172.21.15.118:6803","nonce":864752275}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6804","nonce":864752275},{"type":"v1","addr":"172.21.15.118:6805","nonce":864752275}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6808","nonce":864752275},{"type":"v1","addr":"172.21.15.118:6809","nonce":864752275}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6806","nonce":864752275},{"type":"v1","addr":"172.21.15.118:6807","nonce":864752275}]},"public_addr":"172.21.15.118:6803/864752275","cluster_addr":"172.21.15.118:6805/864752275","heartbeat_back_addr":"172.21.15.118:6809/864752275","heartbeat_front_addr":"172.21.15.118:6807/864752275","state":["exists","up"]},{"osd":5,"uuid":"bfd9feac-dd06-458c-91b9-af7f175f612c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":133,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6810","nonce":2724891770},{"type":"v1","addr":"172.21.15.118:6811","nonce":2724891770}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6812","nonce":2724891770},{"type":"v1","addr":"172.21.15.118:6813","nonce":2724891770}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6816","nonce":2724891770},{"type":"v1","addr":"172.21.15.118:6817","nonce":2724891770}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6814","nonce":2724891770},{"type":"v1","addr":"172.21.15.118:6815","nonce":2724891770}]},"public_addr":"172.21.15.118:6811/2724891770","cluster_addr":"172.21.15.118:6813/2724891770","heartbeat_back_addr":"172.21.15.118:6817/2724891770","heartbeat_front_addr":"172.21.15.118:6815/2724891770","state":["exists","up"]},{"osd":6,"uuid":"5ed5a279-9d8e-416d-bbc7-7edd3e6d5705","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":133,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6825","nonce":801947386},{"type":"v1","addr":"172.21.15.118:6827","nonce":801947386}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6828","nonce":801947386},{"type":"v1","addr":"172.21.15.118:6829","nonce":801947386}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6832","nonce":801947386},{"type":"v1","addr":"172.21.15.118:6833","nonce":801947386}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6830","nonce":801947386},{"type":"v1","addr":"172.21.15.118:6831","nonce":801947386}]},"public_addr":"172.21.15.118:6827/801947386","cluster_addr":"172.21.15.118:6829/801947386","heartbeat_back_addr":"172.21.15.118:6833/801947386","heartbeat_front_addr":"172.21.15.118:6831/801947386","state":["exists","up"]},{"osd":7,"uuid":"f2086c47-875e-4451-a436-2008d1ea16af","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":133,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6818","nonce":653000048},{"type":"v1","addr":"172.21.15.118:6819","nonce":653000048}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6820","nonce":653000048},{"type":"v1","addr":"172.21.15.118:6821","nonce":653000048}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6824","nonce":653000048},{"type":"v1","addr":"172.21.15.118:6826","nonce":653000048}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6822","nonce":653000048},{"type":"v1","addr":"172.21.15.118:6823","nonce":653000048}]},"public_addr":"172.21.15.118:6819/653000048","cluster_addr":"172.21.15.118:6821/653000048","heartbeat_back_addr":"172.21.15.118:6826/653000048","heartbeat_front_addr":"172.21.15.118:6823/653000048","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-03T20:47:25.491994+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-03T20:47:25.587358+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-03T20:47:25.454175+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-03T20:47:25.377734+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-03T20:47:25.412351+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-03T20:47:25.499815+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-03T20:47:25.575044+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-03T20:47:25.552383+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.32:6834/3440267920":"2024-08-04T21:01:19.501719+0000","172.21.15.32:6832/3440267920":"2024-08-04T21:01:19.501719+0000","172.21.15.32:6835/4032603982":"2024-08-04T21:01:06.854369+0000","172.21.15.32:6833/4032603982":"2024-08-04T21:01:06.854369+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-08-03T21:01:29.145 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:29.144+0000 7f4d79800640 1 -- 172.21.15.32:0/4218543462 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f4d50077040 msgr2=0x7f4d50079550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:01:29.145 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:29.144+0000 7f4d79800640 1 --2- 172.21.15.32:0/4218543462 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f4d50077040 0x7f4d50079550 secure :-1 s=READY pgs=909 cs=0 l=1 rev1=1 crypto rx=0x7f4d5c0011a0 tx=0x7f4d5c001110 comp rx=0 tx=0).stop 2024-08-03T21:01:29.145 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:29.144+0000 7f4d79800640 1 -- 172.21.15.32:0/4218543462 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f4d58003c30 msgr2=0x7f4d740b4af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:01:29.145 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:29.144+0000 7f4d79800640 1 --2- 172.21.15.32:0/4218543462 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f4d58003c30 0x7f4d740b4af0 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7f4d64004000 tx=0x7f4d640061b0 comp rx=0 tx=0).stop 2024-08-03T21:01:29.145 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:29.144+0000 7f4d79800640 1 -- 172.21.15.32:0/4218543462 shutdown_connections 2024-08-03T21:01:29.146 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:29.144+0000 7f4d79800640 1 --2- 172.21.15.32:0/4218543462 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f4d50077040 0x7f4d50079550 unknown :-1 s=CLOSED pgs=909 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:01:29.146 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:29.144+0000 7f4d79800640 1 --2- 172.21.15.32:0/4218543462 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4d741f90e0 0x7f4d7414beb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:01:29.146 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:29.144+0000 7f4d79800640 1 --2- 172.21.15.32:0/4218543462 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f4d740b5030 0x7f4d741f8ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:01:29.146 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:29.144+0000 7f4d79800640 1 --2- 172.21.15.32:0/4218543462 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f4d58003c30 0x7f4d740b4af0 unknown :-1 s=CLOSED pgs=683 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:01:29.146 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:29.144+0000 7f4d79800640 1 -- 172.21.15.32:0/4218543462 >> 172.21.15.32:0/4218543462 conn(0x7f4d7408d7a0 msgr2=0x7f4d740d1cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T21:01:29.146 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:29.144+0000 7f4d79800640 1 -- 172.21.15.32:0/4218543462 shutdown_connections 2024-08-03T21:01:29.146 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:29.144+0000 7f4d79800640 1 -- 172.21.15.32:0/4218543462 wait complete. 2024-08-03T21:01:30.172 INFO:tasks.ceph:Scrubbing osd.0 2024-08-03T21:01:30.172 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 config set osd_debug_deep_scrub_sleep 0 2024-08-03T21:01:30.415 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:30.413+0000 7f8f56000640 1 -- 172.21.15.32:0/1955702434 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8f500cccf0 msgr2=0x7f8f500cd100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:01:30.415 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:30.413+0000 7f8f56000640 1 --2- 172.21.15.32:0/1955702434 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8f500cccf0 0x7f8f500cd100 secure :-1 s=READY pgs=2275 cs=0 l=1 rev1=1 crypto rx=0x7f8f3c005fd0 tx=0x7f8f3c02a5c0 comp rx=0 tx=0).stop 2024-08-03T21:01:30.415 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:30.414+0000 7f8f56000640 1 -- 172.21.15.32:0/1955702434 shutdown_connections 2024-08-03T21:01:30.415 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:30.414+0000 7f8f56000640 1 --2- 172.21.15.32:0/1955702434 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8f500cccf0 0x7f8f500cd100 unknown :-1 s=CLOSED pgs=2275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:01:30.415 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:30.414+0000 7f8f56000640 1 --2- 172.21.15.32:0/1955702434 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f8f50145c50 0x7f8f5006e950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:01:30.415 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:30.414+0000 7f8f56000640 1 -- 172.21.15.32:0/1955702434 >> 172.21.15.32:0/1955702434 conn(0x7f8f500b1bd0 msgr2=0x7f8f500b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T21:01:30.416 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:30.414+0000 7f8f56000640 1 -- 172.21.15.32:0/1955702434 shutdown_connections 2024-08-03T21:01:30.416 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:30.414+0000 7f8f56000640 1 -- 172.21.15.32:0/1955702434 wait complete. 2024-08-03T21:01:30.416 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:30.416+0000 7f8f56000640 1 Processor -- start 2024-08-03T21:01:30.417 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:30.416+0000 7f8f56000640 1 -- start start 2024-08-03T21:01:30.419 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:30.418+0000 7f8f56000640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8f500cccf0 0x7f8f50141000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:01:30.419 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:30.418+0000 7f8f56000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8f500ced30 0x7f8f500ca480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:01:30.419 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:30.418+0000 7f8f56000640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f8f50145c50 0x7f8f501597f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:01:30.420 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:30.418+0000 7f8f56000640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f8f500b4070 con 0x7f8f500ced30 2024-08-03T21:01:30.420 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:30.418+0000 7f8f56000640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f8f500b3ef0 con 0x7f8f500cccf0 2024-08-03T21:01:30.420 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:30.418+0000 7f8f56000640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7f8f500b41f0 con 0x7f8f50145c50 2024-08-03T21:01:30.420 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:30.418+0000 7f8f4ea00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8f500ced30 0x7f8f500ca480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:01:30.420 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:30.418+0000 7f8f4ea00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8f500ced30 0x7f8f500ca480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:39456/0 (socket says 172.21.15.32:39456) 2024-08-03T21:01:30.420 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:30.418+0000 7f8f4ea00640 1 -- 172.21.15.32:0/332857151 learned_addr learned my addr 172.21.15.32:0/332857151 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T21:01:30.420 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:30.418+0000 7f8f4fe00640 1 --2- 172.21.15.32:0/332857151 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f8f50145c50 0x7f8f501597f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:01:30.420 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:30.418+0000 7f8f4f400640 1 --2- 172.21.15.32:0/332857151 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8f500cccf0 0x7f8f50141000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:01:30.421 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:30.419+0000 7f8f4ea00640 1 -- 172.21.15.32:0/332857151 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f8f50145c50 msgr2=0x7f8f501597f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:01:30.421 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:30.419+0000 7f8f4ea00640 1 --2- 172.21.15.32:0/332857151 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f8f50145c50 0x7f8f501597f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:01:30.421 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:30.419+0000 7f8f4ea00640 1 -- 172.21.15.32:0/332857151 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8f500cccf0 msgr2=0x7f8f50141000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:01:30.421 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:30.419+0000 7f8f4ea00640 1 --2- 172.21.15.32:0/332857151 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8f500cccf0 0x7f8f50141000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:01:30.421 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:30.419+0000 7f8f4ea00640 1 -- 172.21.15.32:0/332857151 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8f501f8dd0 con 0x7f8f500ced30 2024-08-03T21:01:30.422 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:30.419+0000 7f8f4ea00640 1 --2- 172.21.15.32:0/332857151 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8f500ced30 0x7f8f500ca480 secure :-1 s=READY pgs=2276 cs=0 l=1 rev1=1 crypto rx=0x7f8f3c000c00 tx=0x7f8f3c0038a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T21:01:30.422 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:30.419+0000 7f8f47400640 1 -- 172.21.15.32:0/332857151 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f3c03e030 con 0x7f8f500ced30 2024-08-03T21:01:30.422 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:30.419+0000 7f8f47400640 1 -- 172.21.15.32:0/332857151 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f8f3c003db0 con 0x7f8f500ced30 2024-08-03T21:01:30.423 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:30.419+0000 7f8f47400640 1 -- 172.21.15.32:0/332857151 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f3c035040 con 0x7f8f500ced30 2024-08-03T21:01:30.423 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:30.419+0000 7f8f56000640 1 -- 172.21.15.32:0/332857151 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8f501f90b0 con 0x7f8f500ced30 2024-08-03T21:01:30.423 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:30.419+0000 7f8f56000640 1 -- 172.21.15.32:0/332857151 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8f501f9670 con 0x7f8f500ced30 2024-08-03T21:01:30.423 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:30.420+0000 7f8f56000640 1 -- 172.21.15.32:0/332857151 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f8f501f98b0 con 0x7f8f500ced30 2024-08-03T21:01:30.423 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:30.422+0000 7f8f47400640 1 -- 172.21.15.32:0/332857151 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f8f3c041030 con 0x7f8f500ced30 2024-08-03T21:01:30.424 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:30.423+0000 7f8f47400640 1 --2- 172.21.15.32:0/332857151 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f8f30077010 0x7f8f30079520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:01:30.424 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:30.423+0000 7f8f47400640 1 -- 172.21.15.32:0/332857151 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(139..139 src has 1..139) ==== 5029+0+0 (secure 0 0 0) 0x7f8f3c0b8d90 con 0x7f8f500ced30 2024-08-03T21:01:30.425 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:30.423+0000 7f8f47400640 1 --2- 172.21.15.32:0/332857151 >> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] conn(0x7f8f3007cfc0 0x7f8f3007f410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:01:30.425 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:30.423+0000 7f8f47400640 1 -- 172.21.15.32:0/332857151 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f8f3c00b0b0 con 0x7f8f3007cfc0 2024-08-03T21:01:30.425 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:30.423+0000 7f8f47400640 1 -- 172.21.15.32:0/332857151 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_get_version_reply(handle=1 version=139) ==== 24+0+0 (secure 0 0 0) 0x7f8f3c0bb3b0 con 0x7f8f500ced30 2024-08-03T21:01:30.425 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:30.423+0000 7f8f4fe00640 1 --2- 172.21.15.32:0/332857151 >> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] conn(0x7f8f3007cfc0 0x7f8f3007f410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:01:30.425 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:30.423+0000 7f8f4f400640 1 --2- 172.21.15.32:0/332857151 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f8f30077010 0x7f8f30079520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:01:30.425 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:30.424+0000 7f8f4fe00640 1 --2- 172.21.15.32:0/332857151 >> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] conn(0x7f8f3007cfc0 0x7f8f3007f410 crc :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T21:01:30.425 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:30.424+0000 7f8f4f400640 1 --2- 172.21.15.32:0/332857151 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f8f30077010 0x7f8f30079520 secure :-1 s=READY pgs=910 cs=0 l=1 rev1=1 crypto rx=0x7f8f40009b30 tx=0x7f8f40005f90 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T21:01:30.426 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:30.425+0000 7f8f47400640 1 -- 172.21.15.32:0/332857151 <== osd.0 v2:172.21.15.32:6816/3066156116 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f8f3c00b0b0 con 0x7f8f3007cfc0 2024-08-03T21:01:30.439 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:30.438+0000 7f8f56000640 1 -- 172.21.15.32:0/332857151 --> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7f8f501f8b40 con 0x7f8f3007cfc0 2024-08-03T21:01:30.439 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:30.439+0000 7f8f47400640 1 -- 172.21.15.32:0/332857151 <== osd.0 v2:172.21.15.32:6816/3066156116 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7f8f501f8b40 con 0x7f8f3007cfc0 2024-08-03T21:01:30.439 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-08-03T21:01:30.440 INFO:teuthology.orchestra.run.smithi032.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-08-03T21:01:30.440 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-08-03T21:01:30.441 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:30.439+0000 7f8f56000640 1 -- 172.21.15.32:0/332857151 >> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] conn(0x7f8f3007cfc0 msgr2=0x7f8f3007f410 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:01:30.441 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:30.439+0000 7f8f56000640 1 --2- 172.21.15.32:0/332857151 >> [v2:172.21.15.32:6816/3066156116,v1:172.21.15.32:6817/3066156116] conn(0x7f8f3007cfc0 0x7f8f3007f410 crc :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:01:30.441 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:30.439+0000 7f8f56000640 1 -- 172.21.15.32:0/332857151 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f8f30077010 msgr2=0x7f8f30079520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:01:30.441 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:30.439+0000 7f8f56000640 1 --2- 172.21.15.32:0/332857151 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f8f30077010 0x7f8f30079520 secure :-1 s=READY pgs=910 cs=0 l=1 rev1=1 crypto rx=0x7f8f40009b30 tx=0x7f8f40005f90 comp rx=0 tx=0).stop 2024-08-03T21:01:30.441 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:30.439+0000 7f8f56000640 1 -- 172.21.15.32:0/332857151 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8f500ced30 msgr2=0x7f8f500ca480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:01:30.441 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:30.439+0000 7f8f56000640 1 --2- 172.21.15.32:0/332857151 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8f500ced30 0x7f8f500ca480 secure :-1 s=READY pgs=2276 cs=0 l=1 rev1=1 crypto rx=0x7f8f3c000c00 tx=0x7f8f3c0038a0 comp rx=0 tx=0).stop 2024-08-03T21:01:30.442 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:30.439+0000 7f8f4fe00640 1 -- 172.21.15.32:0/332857151 reap_dead start 2024-08-03T21:01:30.442 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:30.439+0000 7f8f56000640 1 -- 172.21.15.32:0/332857151 shutdown_connections 2024-08-03T21:01:30.442 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:30.439+0000 7f8f56000640 1 -- 172.21.15.32:0/332857151 >> 172.21.15.32:0/332857151 conn(0x7f8f500b1bd0 msgr2=0x7f8f5006fb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T21:01:30.442 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:30.439+0000 7f8f56000640 1 -- 172.21.15.32:0/332857151 shutdown_connections 2024-08-03T21:01:30.442 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:30.439+0000 7f8f56000640 1 -- 172.21.15.32:0/332857151 wait complete. 2024-08-03T21:01:30.465 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 0 2024-08-03T21:01:30.694 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:30.693+0000 7fe690c00640 1 -- 172.21.15.32:0/3628780855 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe68c160740 msgr2=0x7fe68c0cf970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:01:30.694 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:30.693+0000 7fe690c00640 1 --2- 172.21.15.32:0/3628780855 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe68c160740 0x7fe68c0cf970 secure :-1 s=READY pgs=2278 cs=0 l=1 rev1=1 crypto rx=0x7fe684005fd0 tx=0x7fe68402a8b0 comp rx=0 tx=0).stop 2024-08-03T21:01:30.694 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:30.693+0000 7fe690c00640 1 -- 172.21.15.32:0/3628780855 shutdown_connections 2024-08-03T21:01:30.694 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:30.693+0000 7fe690c00640 1 --2- 172.21.15.32:0/3628780855 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe68c160740 0x7fe68c0cf970 unknown :-1 s=CLOSED pgs=2278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:01:30.694 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:30.693+0000 7fe690c00640 1 --2- 172.21.15.32:0/3628780855 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fe68c0cc830 0x7fe68c0cf430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:01:30.695 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:30.693+0000 7fe690c00640 1 -- 172.21.15.32:0/3628780855 >> 172.21.15.32:0/3628780855 conn(0x7fe68c0b1c40 msgr2=0x7fe68c0b2050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T21:01:30.695 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:30.693+0000 7fe690c00640 1 -- 172.21.15.32:0/3628780855 shutdown_connections 2024-08-03T21:01:30.695 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:30.693+0000 7fe690c00640 1 -- 172.21.15.32:0/3628780855 wait complete. 2024-08-03T21:01:30.695 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:30.695+0000 7fe690c00640 1 Processor -- start 2024-08-03T21:01:30.696 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:30.695+0000 7fe690c00640 1 -- start start 2024-08-03T21:01:30.698 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:30.697+0000 7fe690c00640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fe68c0cc830 0x7fe68c130250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:01:30.698 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:30.697+0000 7fe690c00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fe68c0ccc40 0x7fe68c1f3620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:01:30.698 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:30.697+0000 7fe690c00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe68c1f3b60 0x7fe68c0b4300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:01:30.698 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:30.697+0000 7fe690c00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fe68c06fd30 con 0x7fe68c1f3b60 2024-08-03T21:01:30.698 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:30.697+0000 7fe690c00640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7fe68c06fbb0 con 0x7fe68c0ccc40 2024-08-03T21:01:30.698 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:30.697+0000 7fe690c00640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7fe68c06feb0 con 0x7fe68c0cc830 2024-08-03T21:01:30.698 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:30.697+0000 7fe68a800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe68c1f3b60 0x7fe68c0b4300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:01:30.699 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:30.697+0000 7fe68a800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe68c1f3b60 0x7fe68c0b4300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:39478/0 (socket says 172.21.15.32:39478) 2024-08-03T21:01:30.699 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:30.697+0000 7fe68a800640 1 -- 172.21.15.32:0/2634930101 learned_addr learned my addr 172.21.15.32:0/2634930101 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T21:01:30.699 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:30.697+0000 7fe689e00640 1 --2- 172.21.15.32:0/2634930101 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fe68c0cc830 0x7fe68c130250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:01:30.699 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:30.697+0000 7fe689400640 1 --2- 172.21.15.32:0/2634930101 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fe68c0ccc40 0x7fe68c1f3620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:01:30.700 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:30.697+0000 7fe68a800640 1 -- 172.21.15.32:0/2634930101 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fe68c0cc830 msgr2=0x7fe68c130250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:01:30.700 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:30.697+0000 7fe68a800640 1 --2- 172.21.15.32:0/2634930101 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fe68c0cc830 0x7fe68c130250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:01:30.701 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:30.698+0000 7fe68a800640 1 -- 172.21.15.32:0/2634930101 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fe68c0ccc40 msgr2=0x7fe68c1f3620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:01:30.701 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:30.698+0000 7fe68a800640 1 --2- 172.21.15.32:0/2634930101 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fe68c0ccc40 0x7fe68c1f3620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:01:30.701 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:30.698+0000 7fe68a800640 1 -- 172.21.15.32:0/2634930101 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe68c13ff50 con 0x7fe68c1f3b60 2024-08-03T21:01:30.701 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:30.698+0000 7fe68a800640 1 --2- 172.21.15.32:0/2634930101 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe68c1f3b60 0x7fe68c0b4300 secure :-1 s=READY pgs=2279 cs=0 l=1 rev1=1 crypto rx=0x7fe68402aea0 tx=0x7fe6840316f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T21:01:30.701 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:30.698+0000 7fe682000640 1 -- 172.21.15.32:0/2634930101 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe684031da0 con 0x7fe68c1f3b60 2024-08-03T21:01:30.701 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:30.698+0000 7fe682000640 1 -- 172.21.15.32:0/2634930101 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fe684003a60 con 0x7fe68c1f3b60 2024-08-03T21:01:30.701 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:30.698+0000 7fe690c00640 1 -- 172.21.15.32:0/2634930101 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe68c140230 con 0x7fe68c1f3b60 2024-08-03T21:01:30.701 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:30.698+0000 7fe682000640 1 -- 172.21.15.32:0/2634930101 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6840330d0 con 0x7fe68c1f3b60 2024-08-03T21:01:30.702 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:30.698+0000 7fe690c00640 1 -- 172.21.15.32:0/2634930101 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe68c12fa50 con 0x7fe68c1f3b60 2024-08-03T21:01:30.702 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:30.699+0000 7fe690c00640 1 -- 172.21.15.32:0/2634930101 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe68c0703d0 con 0x7fe68c1f3b60 2024-08-03T21:01:30.703 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:30.701+0000 7fe682000640 1 -- 172.21.15.32:0/2634930101 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7fe684033270 con 0x7fe68c1f3b60 2024-08-03T21:01:30.703 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:30.702+0000 7fe682000640 1 --2- 172.21.15.32:0/2634930101 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fe66c077040 0x7fe66c079550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:01:30.703 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:30.702+0000 7fe682000640 1 -- 172.21.15.32:0/2634930101 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(139..139 src has 1..139) ==== 5029+0+0 (secure 0 0 0) 0x7fe6840b9920 con 0x7fe68c1f3b60 2024-08-03T21:01:30.703 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:30.702+0000 7fe689e00640 1 --2- 172.21.15.32:0/2634930101 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fe66c077040 0x7fe66c079550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:01:30.703 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:30.703+0000 7fe689e00640 1 --2- 172.21.15.32:0/2634930101 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fe66c077040 0x7fe66c079550 secure :-1 s=READY pgs=911 cs=0 l=1 rev1=1 crypto rx=0x7fe67c009ac0 tx=0x7fe67c005f90 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T21:01:30.707 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:30.706+0000 7fe682000640 1 -- 172.21.15.32:0/2634930101 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7fe68403f030 con 0x7fe68c1f3b60 2024-08-03T21:01:30.914 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:30.914+0000 7fe690c00640 1 -- 172.21.15.32:0/2634930101 --> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "0", "target": ["mon-mgr", ""]}) -- 0x7fe68c15d6a0 con 0x7fe66c077040 2024-08-03T21:01:30.915 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:30.914+0000 7fe682000640 1 -- 172.21.15.32:0/2634930101 <== mgr.4106 v2:172.21.15.118:6800/4205950699 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 0 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7fe68c15d6a0 con 0x7fe66c077040 2024-08-03T21:01:30.915 INFO:teuthology.orchestra.run.smithi032.stderr:instructed osd(s) 0 to deep-scrub 2024-08-03T21:01:30.918 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:30.917+0000 7fe690c00640 1 -- 172.21.15.32:0/2634930101 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fe66c077040 msgr2=0x7fe66c079550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:01:30.918 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:30.917+0000 7fe690c00640 1 --2- 172.21.15.32:0/2634930101 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fe66c077040 0x7fe66c079550 secure :-1 s=READY pgs=911 cs=0 l=1 rev1=1 crypto rx=0x7fe67c009ac0 tx=0x7fe67c005f90 comp rx=0 tx=0).stop 2024-08-03T21:01:30.918 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:30.917+0000 7fe690c00640 1 -- 172.21.15.32:0/2634930101 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe68c1f3b60 msgr2=0x7fe68c0b4300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:01:30.918 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:30.917+0000 7fe690c00640 1 --2- 172.21.15.32:0/2634930101 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe68c1f3b60 0x7fe68c0b4300 secure :-1 s=READY pgs=2279 cs=0 l=1 rev1=1 crypto rx=0x7fe68402aea0 tx=0x7fe6840316f0 comp rx=0 tx=0).stop 2024-08-03T21:01:30.918 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:30.917+0000 7fe690c00640 1 -- 172.21.15.32:0/2634930101 shutdown_connections 2024-08-03T21:01:30.918 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:30.917+0000 7fe690c00640 1 --2- 172.21.15.32:0/2634930101 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fe66c077040 0x7fe66c079550 unknown :-1 s=CLOSED pgs=911 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:01:30.919 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:30.917+0000 7fe690c00640 1 --2- 172.21.15.32:0/2634930101 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fe68c0ccc40 0x7fe68c1f3620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:01:30.919 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:30.917+0000 7fe690c00640 1 --2- 172.21.15.32:0/2634930101 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe68c1f3b60 0x7fe68c0b4300 unknown :-1 s=CLOSED pgs=2279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:01:30.919 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:30.917+0000 7fe690c00640 1 --2- 172.21.15.32:0/2634930101 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fe68c0cc830 0x7fe68c130250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:01:30.919 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:30.917+0000 7fe690c00640 1 -- 172.21.15.32:0/2634930101 >> 172.21.15.32:0/2634930101 conn(0x7fe68c0b1c40 msgr2=0x7fe68c145890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T21:01:30.919 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:30.917+0000 7fe690c00640 1 -- 172.21.15.32:0/2634930101 shutdown_connections 2024-08-03T21:01:30.919 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:30.917+0000 7fe690c00640 1 -- 172.21.15.32:0/2634930101 wait complete. 2024-08-03T21:01:30.945 INFO:tasks.ceph:Scrubbing osd.1 2024-08-03T21:01:30.945 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 config set osd_debug_deep_scrub_sleep 0 2024-08-03T21:01:31.177 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:31.176+0000 7f82fba00640 1 -- 172.21.15.32:0/4292320955 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f82f4160520 msgr2=0x7f82f41450f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:01:31.178 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:31.176+0000 7f82fba00640 1 --2- 172.21.15.32:0/4292320955 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f82f4160520 0x7f82f41450f0 secure :-1 s=READY pgs=2281 cs=0 l=1 rev1=1 crypto rx=0x7f82ec005fd0 tx=0x7f82ec02aa10 comp rx=0 tx=0).stop 2024-08-03T21:01:31.178 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:31.176+0000 7f82fba00640 1 -- 172.21.15.32:0/4292320955 shutdown_connections 2024-08-03T21:01:31.178 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:31.177+0000 7f82fba00640 1 --2- 172.21.15.32:0/4292320955 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f82f4160520 0x7f82f41450f0 unknown :-1 s=CLOSED pgs=2281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:01:31.178 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:31.177+0000 7f82fba00640 1 --2- 172.21.15.32:0/4292320955 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f82f40cc850 0x7f82f415ffe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:01:31.178 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:31.177+0000 7f82fba00640 1 -- 172.21.15.32:0/4292320955 >> 172.21.15.32:0/4292320955 conn(0x7f82f40b1bd0 msgr2=0x7f82f40b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T21:01:31.178 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:31.177+0000 7f82fba00640 1 -- 172.21.15.32:0/4292320955 shutdown_connections 2024-08-03T21:01:31.178 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:31.177+0000 7f82fba00640 1 -- 172.21.15.32:0/4292320955 wait complete. 2024-08-03T21:01:31.179 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:31.179+0000 7f82fba00640 1 Processor -- start 2024-08-03T21:01:31.179 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:31.179+0000 7f82fba00640 1 -- start start 2024-08-03T21:01:31.182 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:31.180+0000 7f82fba00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f82f40cc850 0x7f82f41404c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:01:31.183 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:31.180+0000 7f82fba00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f82f40cf8d0 0x7f82f41f88a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:01:31.183 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:31.180+0000 7f82fba00640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f82f4160520 0x7f82f41f8de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:01:31.183 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:31.180+0000 7f82fba00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f82f41459c0 con 0x7f82f40cf8d0 2024-08-03T21:01:31.183 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:31.180+0000 7f82fba00640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f82f4145840 con 0x7f82f40cc850 2024-08-03T21:01:31.183 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:31.180+0000 7f82fba00640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7f82f4145b40 con 0x7f82f4160520 2024-08-03T21:01:31.183 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:31.181+0000 7f82f3e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f82f40cf8d0 0x7f82f41f88a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:01:31.183 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:31.181+0000 7f82f3e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f82f40cf8d0 0x7f82f41f88a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:39506/0 (socket says 172.21.15.32:39506) 2024-08-03T21:01:31.184 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:31.181+0000 7f82f3e00640 1 -- 172.21.15.32:0/1186148123 learned_addr learned my addr 172.21.15.32:0/1186148123 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T21:01:31.184 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:31.181+0000 7f82f9000640 1 --2- 172.21.15.32:0/1186148123 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f82f40cc850 0x7f82f41404c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:01:31.184 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:31.181+0000 7f82f9a00640 1 --2- 172.21.15.32:0/1186148123 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f82f4160520 0x7f82f41f8de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:01:31.184 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:31.181+0000 7f82f3e00640 1 -- 172.21.15.32:0/1186148123 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f82f4160520 msgr2=0x7f82f41f8de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:01:31.184 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:31.181+0000 7f82f3e00640 1 --2- 172.21.15.32:0/1186148123 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f82f4160520 0x7f82f41f8de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:01:31.184 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:31.181+0000 7f82f3e00640 1 -- 172.21.15.32:0/1186148123 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f82f40cc850 msgr2=0x7f82f41404c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:01:31.184 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:31.181+0000 7f82f3e00640 1 --2- 172.21.15.32:0/1186148123 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f82f40cc850 0x7f82f41404c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:01:31.184 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:31.181+0000 7f82f3e00640 1 -- 172.21.15.32:0/1186148123 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f82f4136250 con 0x7f82f40cf8d0 2024-08-03T21:01:31.184 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:31.181+0000 7f82f9a00640 1 --2- 172.21.15.32:0/1186148123 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f82f4160520 0x7f82f41f8de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-03T21:01:31.185 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:31.181+0000 7f82f9000640 1 --2- 172.21.15.32:0/1186148123 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f82f40cc850 0x7f82f41404c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-03T21:01:31.185 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:31.182+0000 7f82f3e00640 1 --2- 172.21.15.32:0/1186148123 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f82f40cf8d0 0x7f82f41f88a0 secure :-1 s=READY pgs=2282 cs=0 l=1 rev1=1 crypto rx=0x7f82e000c970 tx=0x7f82e000ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T21:01:31.185 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:31.182+0000 7f82f0c00640 1 -- 172.21.15.32:0/1186148123 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82e0007bb0 con 0x7f82f40cf8d0 2024-08-03T21:01:31.185 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:31.182+0000 7f82f0c00640 1 -- 172.21.15.32:0/1186148123 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f82e0007e40 con 0x7f82f40cf8d0 2024-08-03T21:01:31.185 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:31.182+0000 7f82fba00640 1 -- 172.21.15.32:0/1186148123 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f82f4136530 con 0x7f82f40cf8d0 2024-08-03T21:01:31.185 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:31.182+0000 7f82f0c00640 1 -- 172.21.15.32:0/1186148123 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82e0005280 con 0x7f82f40cf8d0 2024-08-03T21:01:31.185 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:31.182+0000 7f82fba00640 1 -- 172.21.15.32:0/1186148123 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f82f41f9600 con 0x7f82f40cf8d0 2024-08-03T21:01:31.186 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:31.183+0000 7f82fba00640 1 -- 172.21.15.32:0/1186148123 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f82f4156160 con 0x7f82f40cf8d0 2024-08-03T21:01:31.187 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:31.184+0000 7f82f0c00640 1 -- 172.21.15.32:0/1186148123 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f82e0005420 con 0x7f82f40cf8d0 2024-08-03T21:01:31.187 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:31.185+0000 7f82f0c00640 1 --2- 172.21.15.32:0/1186148123 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f82d4077010 0x7f82d4079520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:01:31.187 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:31.185+0000 7f82f0c00640 1 -- 172.21.15.32:0/1186148123 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(139..139 src has 1..139) ==== 5029+0+0 (secure 0 0 0) 0x7f82e0098fe0 con 0x7f82f40cf8d0 2024-08-03T21:01:31.187 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:31.186+0000 7f82f0c00640 1 --2- 172.21.15.32:0/1186148123 >> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] conn(0x7f82d407cfc0 0x7f82d407f410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:01:31.187 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:31.186+0000 7f82f0c00640 1 -- 172.21.15.32:0/1186148123 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f82e0016ec0 con 0x7f82d407cfc0 2024-08-03T21:01:31.187 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:31.186+0000 7f82f0c00640 1 -- 172.21.15.32:0/1186148123 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_get_version_reply(handle=1 version=139) ==== 24+0+0 (secure 0 0 0) 0x7f82e00993d0 con 0x7f82f40cf8d0 2024-08-03T21:01:31.187 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:31.186+0000 7f82f9a00640 1 --2- 172.21.15.32:0/1186148123 >> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] conn(0x7f82d407cfc0 0x7f82d407f410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:01:31.188 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:31.186+0000 7f82f9000640 1 --2- 172.21.15.32:0/1186148123 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f82d4077010 0x7f82d4079520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:01:31.188 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:31.186+0000 7f82f9a00640 1 --2- 172.21.15.32:0/1186148123 >> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] conn(0x7f82d407cfc0 0x7f82d407f410 crc :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T21:01:31.188 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:31.187+0000 7f82f9000640 1 --2- 172.21.15.32:0/1186148123 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f82d4077010 0x7f82d4079520 secure :-1 s=READY pgs=912 cs=0 l=1 rev1=1 crypto rx=0x7f82e8007fd0 tx=0x7f82e8009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T21:01:31.188 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:31.187+0000 7f82f0c00640 1 -- 172.21.15.32:0/1186148123 <== osd.1 v2:172.21.15.32:6824/1551241759 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f82e0016ec0 con 0x7f82d407cfc0 2024-08-03T21:01:31.202 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:31.201+0000 7f82fba00640 1 -- 172.21.15.32:0/1186148123 --> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7f82f415da30 con 0x7f82d407cfc0 2024-08-03T21:01:31.202 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:31.202+0000 7f82f0c00640 1 -- 172.21.15.32:0/1186148123 <== osd.1 v2:172.21.15.32:6824/1551241759 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7f82f415da30 con 0x7f82d407cfc0 2024-08-03T21:01:31.203 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-08-03T21:01:31.203 INFO:teuthology.orchestra.run.smithi032.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-08-03T21:01:31.203 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-08-03T21:01:31.204 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:31.202+0000 7f82fba00640 1 -- 172.21.15.32:0/1186148123 >> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] conn(0x7f82d407cfc0 msgr2=0x7f82d407f410 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:01:31.204 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:31.202+0000 7f82fba00640 1 --2- 172.21.15.32:0/1186148123 >> [v2:172.21.15.32:6824/1551241759,v1:172.21.15.32:6825/1551241759] conn(0x7f82d407cfc0 0x7f82d407f410 crc :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:01:31.204 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:31.202+0000 7f82fba00640 1 -- 172.21.15.32:0/1186148123 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f82d4077010 msgr2=0x7f82d4079520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:01:31.204 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:31.202+0000 7f82fba00640 1 --2- 172.21.15.32:0/1186148123 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f82d4077010 0x7f82d4079520 secure :-1 s=READY pgs=912 cs=0 l=1 rev1=1 crypto rx=0x7f82e8007fd0 tx=0x7f82e8009040 comp rx=0 tx=0).stop 2024-08-03T21:01:31.204 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:31.202+0000 7f82fba00640 1 -- 172.21.15.32:0/1186148123 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f82f40cf8d0 msgr2=0x7f82f41f88a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:01:31.204 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:31.202+0000 7f82fba00640 1 --2- 172.21.15.32:0/1186148123 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f82f40cf8d0 0x7f82f41f88a0 secure :-1 s=READY pgs=2282 cs=0 l=1 rev1=1 crypto rx=0x7f82e000c970 tx=0x7f82e000ce30 comp rx=0 tx=0).stop 2024-08-03T21:01:31.204 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:31.202+0000 7f82f9a00640 1 -- 172.21.15.32:0/1186148123 reap_dead start 2024-08-03T21:01:31.205 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:31.202+0000 7f82fba00640 1 -- 172.21.15.32:0/1186148123 shutdown_connections 2024-08-03T21:01:31.205 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:31.202+0000 7f82fba00640 1 -- 172.21.15.32:0/1186148123 >> 172.21.15.32:0/1186148123 conn(0x7f82f40b1bd0 msgr2=0x7f82f40d04e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T21:01:31.205 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:31.202+0000 7f82fba00640 1 -- 172.21.15.32:0/1186148123 shutdown_connections 2024-08-03T21:01:31.205 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:31.202+0000 7f82fba00640 1 -- 172.21.15.32:0/1186148123 wait complete. 2024-08-03T21:01:31.224 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 1 2024-08-03T21:01:31.454 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:31.453+0000 7fe091a00640 1 -- 172.21.15.32:0/407910470 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fe08c06e930 msgr2=0x7fe08c144f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:01:31.454 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:31.453+0000 7fe091a00640 1 --2- 172.21.15.32:0/407910470 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fe08c06e930 0x7fe08c144f90 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fe07c009830 tx=0x7fe07c02a520 comp rx=0 tx=0).stop 2024-08-03T21:01:31.454 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:31.453+0000 7fe091a00640 1 -- 172.21.15.32:0/407910470 shutdown_connections 2024-08-03T21:01:31.454 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:31.453+0000 7fe091a00640 1 --2- 172.21.15.32:0/407910470 >> v2:172.21.15.118:3300/0 conn(0x7fe08c15fff0 0x7fe08c160400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:01:31.455 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:31.453+0000 7fe091a00640 1 --2- 172.21.15.32:0/407910470 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fe08c06e930 0x7fe08c144f90 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:01:31.455 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:31.453+0000 7fe091a00640 1 -- 172.21.15.32:0/407910470 >> 172.21.15.32:0/407910470 conn(0x7fe08c0b1bd0 msgr2=0x7fe08c0b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T21:01:31.455 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:31.453+0000 7fe091a00640 1 -- 172.21.15.32:0/407910470 shutdown_connections 2024-08-03T21:01:31.455 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:31.453+0000 7fe091a00640 1 -- 172.21.15.32:0/407910470 wait complete. 2024-08-03T21:01:31.456 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:31.455+0000 7fe091a00640 1 Processor -- start 2024-08-03T21:01:31.456 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:31.455+0000 7fe091a00640 1 -- start start 2024-08-03T21:01:31.458 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:31.457+0000 7fe091a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe08c06e930 0x7fe08c1414e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:01:31.458 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:31.457+0000 7fe091a00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fe08c073e60 0x7fe08c0a9e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:01:31.459 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:31.457+0000 7fe091a00640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fe08c15fff0 0x7fe08c0aafe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:01:31.459 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:31.457+0000 7fe091a00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fe08c0cf840 con 0x7fe08c06e930 2024-08-03T21:01:31.459 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:31.457+0000 7fe091a00640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7fe08c145f30 con 0x7fe08c073e60 2024-08-03T21:01:31.459 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:31.457+0000 7fe091a00640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7fe08c0cf9c0 con 0x7fe08c15fff0 2024-08-03T21:01:31.459 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:31.457+0000 7fe08aa00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe08c06e930 0x7fe08c1414e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:01:31.459 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:31.457+0000 7fe08aa00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe08c06e930 0x7fe08c1414e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:39508/0 (socket says 172.21.15.32:39508) 2024-08-03T21:01:31.459 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:31.457+0000 7fe08aa00640 1 -- 172.21.15.32:0/3291239721 learned_addr learned my addr 172.21.15.32:0/3291239721 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T21:01:31.459 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:31.458+0000 7fe082000640 1 --2- 172.21.15.32:0/3291239721 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fe08c073e60 0x7fe08c0a9e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:01:31.460 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:31.458+0000 7fe08b400640 1 --2- 172.21.15.32:0/3291239721 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fe08c15fff0 0x7fe08c0aafe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:01:31.460 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:31.458+0000 7fe08aa00640 1 -- 172.21.15.32:0/3291239721 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fe08c15fff0 msgr2=0x7fe08c0aafe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:01:31.460 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:31.458+0000 7fe08aa00640 1 --2- 172.21.15.32:0/3291239721 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fe08c15fff0 0x7fe08c0aafe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:01:31.460 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:31.458+0000 7fe08aa00640 1 -- 172.21.15.32:0/3291239721 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fe08c073e60 msgr2=0x7fe08c0a9e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:01:31.460 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:31.458+0000 7fe08aa00640 1 --2- 172.21.15.32:0/3291239721 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fe08c073e60 0x7fe08c0a9e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:01:31.460 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:31.458+0000 7fe08aa00640 1 -- 172.21.15.32:0/3291239721 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe08c144100 con 0x7fe08c06e930 2024-08-03T21:01:31.460 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:31.458+0000 7fe08aa00640 1 --2- 172.21.15.32:0/3291239721 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe08c06e930 0x7fe08c1414e0 secure :-1 s=READY pgs=2283 cs=0 l=1 rev1=1 crypto rx=0x7fe07c000c00 tx=0x7fe07c033c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T21:01:31.462 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:31.458+0000 7fe083400640 1 -- 172.21.15.32:0/3291239721 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe07c031d60 con 0x7fe08c06e930 2024-08-03T21:01:31.462 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:31.458+0000 7fe083400640 1 -- 172.21.15.32:0/3291239721 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fe07c033e10 con 0x7fe08c06e930 2024-08-03T21:01:31.462 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:31.458+0000 7fe091a00640 1 -- 172.21.15.32:0/3291239721 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe08c1443e0 con 0x7fe08c06e930 2024-08-03T21:01:31.462 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:31.458+0000 7fe083400640 1 -- 172.21.15.32:0/3291239721 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe07c033310 con 0x7fe08c06e930 2024-08-03T21:01:31.462 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:31.458+0000 7fe091a00640 1 -- 172.21.15.32:0/3291239721 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe08c0a9930 con 0x7fe08c06e930 2024-08-03T21:01:31.462 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:31.459+0000 7fe091a00640 1 -- 172.21.15.32:0/3291239721 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe08c06fc80 con 0x7fe08c06e930 2024-08-03T21:01:31.462 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:31.461+0000 7fe083400640 1 -- 172.21.15.32:0/3291239721 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7fe07c0334b0 con 0x7fe08c06e930 2024-08-03T21:01:31.463 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:31.462+0000 7fe083400640 1 --2- 172.21.15.32:0/3291239721 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fe068077040 0x7fe068079550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:01:31.463 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:31.462+0000 7fe083400640 1 -- 172.21.15.32:0/3291239721 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(139..139 src has 1..139) ==== 5029+0+0 (secure 0 0 0) 0x7fe07c0b9a60 con 0x7fe08c06e930 2024-08-03T21:01:31.463 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:31.462+0000 7fe082000640 1 --2- 172.21.15.32:0/3291239721 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fe068077040 0x7fe068079550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:01:31.464 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:31.463+0000 7fe082000640 1 --2- 172.21.15.32:0/3291239721 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fe068077040 0x7fe068079550 secure :-1 s=READY pgs=913 cs=0 l=1 rev1=1 crypto rx=0x7fe08c0acd20 tx=0x7fe074009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T21:01:31.466 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:31.466+0000 7fe083400640 1 -- 172.21.15.32:0/3291239721 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7fe08c06fc80 con 0x7fe08c06e930 2024-08-03T21:01:31.670 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:31.670+0000 7fe091a00640 1 -- 172.21.15.32:0/3291239721 --> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "1", "target": ["mon-mgr", ""]}) -- 0x7fe08c160bc0 con 0x7fe068077040 2024-08-03T21:01:31.671 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:31.671+0000 7fe083400640 1 -- 172.21.15.32:0/3291239721 <== mgr.4106 v2:172.21.15.118:6800/4205950699 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 1 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7fe08c160bc0 con 0x7fe068077040 2024-08-03T21:01:31.671 INFO:teuthology.orchestra.run.smithi032.stderr:instructed osd(s) 1 to deep-scrub 2024-08-03T21:01:31.674 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:31.673+0000 7fe091a00640 1 -- 172.21.15.32:0/3291239721 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fe068077040 msgr2=0x7fe068079550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:01:31.674 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:31.673+0000 7fe091a00640 1 --2- 172.21.15.32:0/3291239721 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fe068077040 0x7fe068079550 secure :-1 s=READY pgs=913 cs=0 l=1 rev1=1 crypto rx=0x7fe08c0acd20 tx=0x7fe074009040 comp rx=0 tx=0).stop 2024-08-03T21:01:31.674 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:31.673+0000 7fe091a00640 1 -- 172.21.15.32:0/3291239721 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe08c06e930 msgr2=0x7fe08c1414e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:01:31.674 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:31.673+0000 7fe091a00640 1 --2- 172.21.15.32:0/3291239721 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe08c06e930 0x7fe08c1414e0 secure :-1 s=READY pgs=2283 cs=0 l=1 rev1=1 crypto rx=0x7fe07c000c00 tx=0x7fe07c033c20 comp rx=0 tx=0).stop 2024-08-03T21:01:31.674 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:31.673+0000 7fe091a00640 1 -- 172.21.15.32:0/3291239721 shutdown_connections 2024-08-03T21:01:31.675 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:31.673+0000 7fe091a00640 1 --2- 172.21.15.32:0/3291239721 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fe068077040 0x7fe068079550 unknown :-1 s=CLOSED pgs=913 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:01:31.675 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:31.673+0000 7fe091a00640 1 --2- 172.21.15.32:0/3291239721 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fe08c073e60 0x7fe08c0a9e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:01:31.675 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:31.673+0000 7fe091a00640 1 --2- 172.21.15.32:0/3291239721 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fe08c15fff0 0x7fe08c0aafe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:01:31.675 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:31.673+0000 7fe091a00640 1 --2- 172.21.15.32:0/3291239721 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe08c06e930 0x7fe08c1414e0 unknown :-1 s=CLOSED pgs=2283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:01:31.675 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:31.673+0000 7fe091a00640 1 -- 172.21.15.32:0/3291239721 >> 172.21.15.32:0/3291239721 conn(0x7fe08c0b1bd0 msgr2=0x7fe08c074270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T21:01:31.675 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:31.673+0000 7fe091a00640 1 -- 172.21.15.32:0/3291239721 shutdown_connections 2024-08-03T21:01:31.675 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:31.673+0000 7fe091a00640 1 -- 172.21.15.32:0/3291239721 wait complete. 2024-08-03T21:01:31.696 INFO:tasks.ceph:Scrubbing osd.2 2024-08-03T21:01:31.696 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 config set osd_debug_deep_scrub_sleep 0 2024-08-03T21:01:31.929 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:31.928+0000 7fb9f3c00640 1 -- 172.21.15.32:0/1126897917 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb9ec0cc850 msgr2=0x7fb9ec15ffe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:01:31.929 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:31.928+0000 7fb9f3c00640 1 --2- 172.21.15.32:0/1126897917 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb9ec0cc850 0x7fb9ec15ffe0 secure :-1 s=READY pgs=2285 cs=0 l=1 rev1=1 crypto rx=0x7fb9d8009830 tx=0x7fb9d802a490 comp rx=0 tx=0).stop 2024-08-03T21:01:31.930 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:31.928+0000 7fb9f3c00640 1 -- 172.21.15.32:0/1126897917 shutdown_connections 2024-08-03T21:01:31.930 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:31.928+0000 7fb9f3c00640 1 --2- 172.21.15.32:0/1126897917 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb9ec0cc850 0x7fb9ec15ffe0 unknown :-1 s=CLOSED pgs=2285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:01:31.930 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:31.928+0000 7fb9f3c00640 1 -- 172.21.15.32:0/1126897917 >> 172.21.15.32:0/1126897917 conn(0x7fb9ec0b1bd0 msgr2=0x7fb9ec0b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T21:01:31.930 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:31.928+0000 7fb9f3c00640 1 -- 172.21.15.32:0/1126897917 shutdown_connections 2024-08-03T21:01:31.930 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:31.928+0000 7fb9f3c00640 1 -- 172.21.15.32:0/1126897917 wait complete. 2024-08-03T21:01:31.931 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:31.930+0000 7fb9f3c00640 1 Processor -- start 2024-08-03T21:01:31.931 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:31.931+0000 7fb9f3c00640 1 -- start start 2024-08-03T21:01:31.934 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:31.932+0000 7fb9f3c00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb9ec0cc850 0x7fb9ec13dfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:01:31.934 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:31.932+0000 7fb9f3c00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb9ec0cf8d0 0x7fb9ec1f8f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:01:31.934 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:31.932+0000 7fb9f3c00640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fb9ec160520 0x7fb9ec1f9480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:01:31.934 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:31.932+0000 7fb9f3c00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fb9ec145800 con 0x7fb9ec0cc850 2024-08-03T21:01:31.934 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:31.932+0000 7fb9f3c00640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7fb9ec070270 con 0x7fb9ec0cf8d0 2024-08-03T21:01:31.935 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:31.932+0000 7fb9f3c00640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7fb9ec145980 con 0x7fb9ec160520 2024-08-03T21:01:31.935 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:31.932+0000 7fb9f1200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb9ec0cc850 0x7fb9ec13dfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:01:31.935 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:31.932+0000 7fb9f1200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb9ec0cc850 0x7fb9ec13dfb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:39520/0 (socket says 172.21.15.32:39520) 2024-08-03T21:01:31.935 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:31.932+0000 7fb9f1200640 1 -- 172.21.15.32:0/1932931764 learned_addr learned my addr 172.21.15.32:0/1932931764 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T21:01:31.935 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:31.933+0000 7fb9ebe00640 1 --2- 172.21.15.32:0/1932931764 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb9ec0cf8d0 0x7fb9ec1f8f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:01:31.935 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:31.933+0000 7fb9f1c00640 1 --2- 172.21.15.32:0/1932931764 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fb9ec160520 0x7fb9ec1f9480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:01:31.935 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:31.933+0000 7fb9f1200640 1 -- 172.21.15.32:0/1932931764 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fb9ec160520 msgr2=0x7fb9ec1f9480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:01:31.935 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:31.933+0000 7fb9f1200640 1 --2- 172.21.15.32:0/1932931764 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fb9ec160520 0x7fb9ec1f9480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:01:31.936 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:31.933+0000 7fb9f1200640 1 -- 172.21.15.32:0/1932931764 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb9ec0cf8d0 msgr2=0x7fb9ec1f8f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:01:31.936 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:31.933+0000 7fb9f1200640 1 --2- 172.21.15.32:0/1932931764 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb9ec0cf8d0 0x7fb9ec1f8f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:01:31.936 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:31.933+0000 7fb9f1200640 1 -- 172.21.15.32:0/1932931764 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb9ec1f9cc0 con 0x7fb9ec0cc850 2024-08-03T21:01:31.936 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:31.933+0000 7fb9f1200640 1 --2- 172.21.15.32:0/1932931764 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb9ec0cc850 0x7fb9ec13dfb0 secure :-1 s=READY pgs=2286 cs=0 l=1 rev1=1 crypto rx=0x7fb9e000c3a0 tx=0x7fb9e000c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T21:01:31.936 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:31.933+0000 7fb9e8c00640 1 -- 172.21.15.32:0/1932931764 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9e0017070 con 0x7fb9ec0cc850 2024-08-03T21:01:31.936 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:31.933+0000 7fb9e8c00640 1 -- 172.21.15.32:0/1932931764 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fb9e00053c0 con 0x7fb9ec0cc850 2024-08-03T21:01:31.936 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:31.933+0000 7fb9e8c00640 1 -- 172.21.15.32:0/1932931764 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9e0003850 con 0x7fb9ec0cc850 2024-08-03T21:01:31.936 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:31.933+0000 7fb9f3c00640 1 -- 172.21.15.32:0/1932931764 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb9ec1417c0 con 0x7fb9ec0cc850 2024-08-03T21:01:31.937 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:31.934+0000 7fb9f3c00640 1 -- 172.21.15.32:0/1932931764 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb9ec141490 con 0x7fb9ec0cc850 2024-08-03T21:01:31.938 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:31.934+0000 7fb9f3c00640 1 -- 172.21.15.32:0/1932931764 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fb9ec145ea0 con 0x7fb9ec0cc850 2024-08-03T21:01:31.938 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:31.936+0000 7fb9e8c00640 1 -- 172.21.15.32:0/1932931764 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7fb9e001a070 con 0x7fb9ec0cc850 2024-08-03T21:01:31.938 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:31.937+0000 7fb9e8c00640 1 --2- 172.21.15.32:0/1932931764 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fb9cc077010 0x7fb9cc079520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:01:31.938 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:31.937+0000 7fb9e8c00640 1 -- 172.21.15.32:0/1932931764 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(139..139 src has 1..139) ==== 5029+0+0 (secure 0 0 0) 0x7fb9e0097a20 con 0x7fb9ec0cc850 2024-08-03T21:01:31.938 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:31.937+0000 7fb9e8c00640 1 --2- 172.21.15.32:0/1932931764 >> [v2:172.21.15.32:6808/3806708944,v1:172.21.15.32:6809/3806708944] conn(0x7fb9cc07cfc0 0x7fb9cc07f410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:01:31.938 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:31.937+0000 7fb9e8c00640 1 -- 172.21.15.32:0/1932931764 --> [v2:172.21.15.32:6808/3806708944,v1:172.21.15.32:6809/3806708944] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fb9cc07fae0 con 0x7fb9cc07cfc0 2024-08-03T21:01:31.939 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:31.937+0000 7fb9e8c00640 1 -- 172.21.15.32:0/1932931764 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_get_version_reply(handle=1 version=139) ==== 24+0+0 (secure 0 0 0) 0x7fb9e009c2a0 con 0x7fb9ec0cc850 2024-08-03T21:01:31.939 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:31.937+0000 7fb9ebe00640 1 --2- 172.21.15.32:0/1932931764 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fb9cc077010 0x7fb9cc079520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:01:31.939 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:31.937+0000 7fb9f1c00640 1 --2- 172.21.15.32:0/1932931764 >> [v2:172.21.15.32:6808/3806708944,v1:172.21.15.32:6809/3806708944] conn(0x7fb9cc07cfc0 0x7fb9cc07f410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:01:31.939 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:31.938+0000 7fb9ebe00640 1 --2- 172.21.15.32:0/1932931764 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fb9cc077010 0x7fb9cc079520 secure :-1 s=READY pgs=914 cs=0 l=1 rev1=1 crypto rx=0x7fb9d8034000 tx=0x7fb9d8004730 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T21:01:31.940 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:31.938+0000 7fb9f1c00640 1 --2- 172.21.15.32:0/1932931764 >> [v2:172.21.15.32:6808/3806708944,v1:172.21.15.32:6809/3806708944] conn(0x7fb9cc07cfc0 0x7fb9cc07f410 crc :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T21:01:31.940 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:31.939+0000 7fb9e8c00640 1 -- 172.21.15.32:0/1932931764 <== osd.2 v2:172.21.15.32:6808/3806708944 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fb9cc07fae0 con 0x7fb9cc07cfc0 2024-08-03T21:01:31.953 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:31.953+0000 7fb9f3c00640 1 -- 172.21.15.32:0/1932931764 --> [v2:172.21.15.32:6808/3806708944,v1:172.21.15.32:6809/3806708944] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7fb9ec156680 con 0x7fb9cc07cfc0 2024-08-03T21:01:31.954 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:31.953+0000 7fb9e8c00640 1 -- 172.21.15.32:0/1932931764 <== osd.2 v2:172.21.15.32:6808/3806708944 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7fb9ec156680 con 0x7fb9cc07cfc0 2024-08-03T21:01:31.954 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-08-03T21:01:31.954 INFO:teuthology.orchestra.run.smithi032.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-08-03T21:01:31.954 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-08-03T21:01:31.955 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:31.953+0000 7fb9f3c00640 1 -- 172.21.15.32:0/1932931764 >> [v2:172.21.15.32:6808/3806708944,v1:172.21.15.32:6809/3806708944] conn(0x7fb9cc07cfc0 msgr2=0x7fb9cc07f410 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:01:31.955 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:31.953+0000 7fb9f3c00640 1 --2- 172.21.15.32:0/1932931764 >> [v2:172.21.15.32:6808/3806708944,v1:172.21.15.32:6809/3806708944] conn(0x7fb9cc07cfc0 0x7fb9cc07f410 crc :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:01:31.956 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:31.954+0000 7fb9f3c00640 1 -- 172.21.15.32:0/1932931764 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fb9cc077010 msgr2=0x7fb9cc079520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:01:31.956 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:31.954+0000 7fb9f3c00640 1 --2- 172.21.15.32:0/1932931764 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fb9cc077010 0x7fb9cc079520 secure :-1 s=READY pgs=914 cs=0 l=1 rev1=1 crypto rx=0x7fb9d8034000 tx=0x7fb9d8004730 comp rx=0 tx=0).stop 2024-08-03T21:01:31.956 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:31.954+0000 7fb9f3c00640 1 -- 172.21.15.32:0/1932931764 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb9ec0cc850 msgr2=0x7fb9ec13dfb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:01:31.956 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:31.954+0000 7fb9f3c00640 1 --2- 172.21.15.32:0/1932931764 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb9ec0cc850 0x7fb9ec13dfb0 secure :-1 s=READY pgs=2286 cs=0 l=1 rev1=1 crypto rx=0x7fb9e000c3a0 tx=0x7fb9e000c860 comp rx=0 tx=0).stop 2024-08-03T21:01:31.956 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:31.954+0000 7fb9f1c00640 1 -- 172.21.15.32:0/1932931764 reap_dead start 2024-08-03T21:01:31.956 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:31.954+0000 7fb9f3c00640 1 -- 172.21.15.32:0/1932931764 shutdown_connections 2024-08-03T21:01:31.956 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:31.954+0000 7fb9f3c00640 1 -- 172.21.15.32:0/1932931764 >> 172.21.15.32:0/1932931764 conn(0x7fb9ec0b1bd0 msgr2=0x7fb9ec15f980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T21:01:31.956 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:31.954+0000 7fb9f3c00640 1 -- 172.21.15.32:0/1932931764 shutdown_connections 2024-08-03T21:01:31.956 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:31.954+0000 7fb9f3c00640 1 -- 172.21.15.32:0/1932931764 wait complete. 2024-08-03T21:01:31.975 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 2 2024-08-03T21:01:32.206 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:32.205+0000 7fb289600640 1 -- 172.21.15.32:0/3873120414 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb284160030 msgr2=0x7fb284160440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:01:32.206 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:32.205+0000 7fb289600640 1 --2- 172.21.15.32:0/3873120414 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb284160030 0x7fb284160440 secure :-1 s=READY pgs=2288 cs=0 l=1 rev1=1 crypto rx=0x7fb2840cc400 tx=0x7fb26c02a4a0 comp rx=0 tx=0).stop 2024-08-03T21:01:32.206 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:32.205+0000 7fb289600640 1 -- 172.21.15.32:0/3873120414 shutdown_connections 2024-08-03T21:01:32.206 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:32.205+0000 7fb289600640 1 --2- 172.21.15.32:0/3873120414 >> v2:172.21.15.118:3300/0 conn(0x7fb28406f360 0x7fb28406f770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:01:32.207 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:32.205+0000 7fb289600640 1 --2- 172.21.15.32:0/3873120414 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb284160030 0x7fb284160440 unknown :-1 s=CLOSED pgs=2288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:01:32.207 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:32.205+0000 7fb289600640 1 -- 172.21.15.32:0/3873120414 >> 172.21.15.32:0/3873120414 conn(0x7fb2840b1c40 msgr2=0x7fb2840b2050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T21:01:32.207 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:32.205+0000 7fb289600640 1 -- 172.21.15.32:0/3873120414 shutdown_connections 2024-08-03T21:01:32.207 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:32.205+0000 7fb289600640 1 -- 172.21.15.32:0/3873120414 wait complete. 2024-08-03T21:01:32.207 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:32.207+0000 7fb289600640 1 Processor -- start 2024-08-03T21:01:32.208 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:32.207+0000 7fb289600640 1 -- start start 2024-08-03T21:01:32.211 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:32.209+0000 7fb289600640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb28406f360 0x7fb2841e93c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:01:32.211 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:32.209+0000 7fb289600640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fb284145bd0 0x7fb28412ef00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:01:32.211 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:32.209+0000 7fb289600640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb284160030 0x7fb28412f440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:01:32.211 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:32.209+0000 7fb289600640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fb28406fe30 con 0x7fb28406f360 2024-08-03T21:01:32.211 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:32.209+0000 7fb289600640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7fb28406fcb0 con 0x7fb284160030 2024-08-03T21:01:32.211 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:32.209+0000 7fb289600640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7fb28406ffb0 con 0x7fb284145bd0 2024-08-03T21:01:32.211 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:32.209+0000 7fb282a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb28406f360 0x7fb2841e93c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:01:32.212 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:32.209+0000 7fb282a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb28406f360 0x7fb2841e93c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:39528/0 (socket says 172.21.15.32:39528) 2024-08-03T21:01:32.212 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:32.209+0000 7fb282a00640 1 -- 172.21.15.32:0/2739733667 learned_addr learned my addr 172.21.15.32:0/2739733667 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T21:01:32.212 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:32.209+0000 7fb283400640 1 --2- 172.21.15.32:0/2739733667 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb284160030 0x7fb28412f440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:01:32.212 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:32.209+0000 7fb27be00640 1 --2- 172.21.15.32:0/2739733667 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fb284145bd0 0x7fb28412ef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:01:32.212 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:32.210+0000 7fb282a00640 1 -- 172.21.15.32:0/2739733667 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fb284145bd0 msgr2=0x7fb28412ef00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:01:32.212 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:32.210+0000 7fb282a00640 1 --2- 172.21.15.32:0/2739733667 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fb284145bd0 0x7fb28412ef00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:01:32.213 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:32.210+0000 7fb282a00640 1 -- 172.21.15.32:0/2739733667 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb284160030 msgr2=0x7fb28412f440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:01:32.213 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:32.210+0000 7fb282a00640 1 --2- 172.21.15.32:0/2739733667 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb284160030 0x7fb28412f440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:01:32.213 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:32.210+0000 7fb282a00640 1 -- 172.21.15.32:0/2739733667 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb28415bca0 con 0x7fb28406f360 2024-08-03T21:01:32.213 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:32.210+0000 7fb27be00640 1 --2- 172.21.15.32:0/2739733667 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fb284145bd0 0x7fb28412ef00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-03T21:01:32.213 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:32.210+0000 7fb282a00640 1 --2- 172.21.15.32:0/2739733667 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb28406f360 0x7fb2841e93c0 secure :-1 s=READY pgs=2289 cs=0 l=1 rev1=1 crypto rx=0x7fb27400c3f0 tx=0x7fb27400c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T21:01:32.213 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:32.210+0000 7fb27aa00640 1 -- 172.21.15.32:0/2739733667 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb274017070 con 0x7fb28406f360 2024-08-03T21:01:32.213 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:32.210+0000 7fb27aa00640 1 -- 172.21.15.32:0/2739733667 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fb2740035f0 con 0x7fb28406f360 2024-08-03T21:01:32.213 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:32.210+0000 7fb27aa00640 1 -- 172.21.15.32:0/2739733667 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb274003a40 con 0x7fb28406f360 2024-08-03T21:01:32.214 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:32.210+0000 7fb289600640 1 -- 172.21.15.32:0/2739733667 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb28412e420 con 0x7fb28406f360 2024-08-03T21:01:32.214 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:32.210+0000 7fb289600640 1 -- 172.21.15.32:0/2739733667 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb2841e9900 con 0x7fb28406f360 2024-08-03T21:01:32.215 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:32.212+0000 7fb289600640 1 -- 172.21.15.32:0/2739733667 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb2840b4ff0 con 0x7fb28406f360 2024-08-03T21:01:32.215 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:32.213+0000 7fb27aa00640 1 -- 172.21.15.32:0/2739733667 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7fb27401a070 con 0x7fb28406f360 2024-08-03T21:01:32.215 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:32.213+0000 7fb27aa00640 1 --2- 172.21.15.32:0/2739733667 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fb260077010 0x7fb260079520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:01:32.215 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:32.214+0000 7fb27aa00640 1 -- 172.21.15.32:0/2739733667 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(139..139 src has 1..139) ==== 5029+0+0 (secure 0 0 0) 0x7fb274097a80 con 0x7fb28406f360 2024-08-03T21:01:32.215 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:32.214+0000 7fb27be00640 1 --2- 172.21.15.32:0/2739733667 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fb260077010 0x7fb260079520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:01:32.216 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:32.215+0000 7fb27be00640 1 --2- 172.21.15.32:0/2739733667 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fb260077010 0x7fb260079520 secure :-1 s=READY pgs=915 cs=0 l=1 rev1=1 crypto rx=0x7fb26c034000 tx=0x7fb26c004730 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T21:01:32.218 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:32.218+0000 7fb27aa00640 1 -- 172.21.15.32:0/2739733667 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7fb2840b4ff0 con 0x7fb28406f360 2024-08-03T21:01:32.424 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:32.424+0000 7fb289600640 1 -- 172.21.15.32:0/2739733667 --> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "2", "target": ["mon-mgr", ""]}) -- 0x7fb2840704d0 con 0x7fb260077010 2024-08-03T21:01:32.425 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:32.425+0000 7fb27aa00640 1 -- 172.21.15.32:0/2739733667 <== mgr.4106 v2:172.21.15.118:6800/4205950699 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 2 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7fb2840704d0 con 0x7fb260077010 2024-08-03T21:01:32.425 INFO:teuthology.orchestra.run.smithi032.stderr:instructed osd(s) 2 to deep-scrub 2024-08-03T21:01:32.428 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:32.427+0000 7fb289600640 1 -- 172.21.15.32:0/2739733667 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fb260077010 msgr2=0x7fb260079520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:01:32.428 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:32.427+0000 7fb289600640 1 --2- 172.21.15.32:0/2739733667 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fb260077010 0x7fb260079520 secure :-1 s=READY pgs=915 cs=0 l=1 rev1=1 crypto rx=0x7fb26c034000 tx=0x7fb26c004730 comp rx=0 tx=0).stop 2024-08-03T21:01:32.428 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:32.427+0000 7fb289600640 1 -- 172.21.15.32:0/2739733667 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb28406f360 msgr2=0x7fb2841e93c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:01:32.428 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:32.427+0000 7fb289600640 1 --2- 172.21.15.32:0/2739733667 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb28406f360 0x7fb2841e93c0 secure :-1 s=READY pgs=2289 cs=0 l=1 rev1=1 crypto rx=0x7fb27400c3f0 tx=0x7fb27400c8b0 comp rx=0 tx=0).stop 2024-08-03T21:01:32.428 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:32.427+0000 7fb289600640 1 -- 172.21.15.32:0/2739733667 shutdown_connections 2024-08-03T21:01:32.429 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:32.427+0000 7fb289600640 1 --2- 172.21.15.32:0/2739733667 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fb260077010 0x7fb260079520 unknown :-1 s=CLOSED pgs=915 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:01:32.429 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:32.427+0000 7fb289600640 1 --2- 172.21.15.32:0/2739733667 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb284160030 0x7fb28412f440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:01:32.429 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:32.427+0000 7fb289600640 1 --2- 172.21.15.32:0/2739733667 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fb284145bd0 0x7fb28412ef00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:01:32.429 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:32.427+0000 7fb289600640 1 --2- 172.21.15.32:0/2739733667 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb28406f360 0x7fb2841e93c0 unknown :-1 s=CLOSED pgs=2289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:01:32.429 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:32.427+0000 7fb289600640 1 -- 172.21.15.32:0/2739733667 >> 172.21.15.32:0/2739733667 conn(0x7fb2840b1c40 msgr2=0x7fb28415dd70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T21:01:32.429 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:32.427+0000 7fb289600640 1 -- 172.21.15.32:0/2739733667 shutdown_connections 2024-08-03T21:01:32.429 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:32.427+0000 7fb289600640 1 -- 172.21.15.32:0/2739733667 wait complete. 2024-08-03T21:01:32.455 INFO:tasks.ceph:Scrubbing osd.3 2024-08-03T21:01:32.455 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 config set osd_debug_deep_scrub_sleep 0 2024-08-03T21:01:32.686 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:32.684+0000 7f2322e00640 1 -- 172.21.15.32:0/447738038 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f231c0cccf0 msgr2=0x7f231c0cd100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:01:32.686 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:32.684+0000 7f2322e00640 1 --2- 172.21.15.32:0/447738038 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f231c0cccf0 0x7f231c0cd100 secure :-1 s=READY pgs=2290 cs=0 l=1 rev1=1 crypto rx=0x7f231c074a30 tx=0x7f230402a4d0 comp rx=0 tx=0).stop 2024-08-03T21:01:32.687 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:32.684+0000 7f2322e00640 1 -- 172.21.15.32:0/447738038 shutdown_connections 2024-08-03T21:01:32.687 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:32.684+0000 7f2322e00640 1 --2- 172.21.15.32:0/447738038 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f231c0cccf0 0x7f231c0cd100 unknown :-1 s=CLOSED pgs=2290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:01:32.687 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:32.684+0000 7f2322e00640 1 --2- 172.21.15.32:0/447738038 >> v2:172.21.15.118:3300/0 conn(0x7f231c145c50 0x7f231c06e950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:01:32.687 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:32.684+0000 7f2322e00640 1 -- 172.21.15.32:0/447738038 >> 172.21.15.32:0/447738038 conn(0x7f231c0b1bd0 msgr2=0x7f231c0b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T21:01:32.687 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:32.685+0000 7f2322e00640 1 -- 172.21.15.32:0/447738038 shutdown_connections 2024-08-03T21:01:32.687 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:32.685+0000 7f2322e00640 1 -- 172.21.15.32:0/447738038 wait complete. 2024-08-03T21:01:32.688 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:32.687+0000 7f2322e00640 1 Processor -- start 2024-08-03T21:01:32.688 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:32.687+0000 7f2322e00640 1 -- start start 2024-08-03T21:01:32.691 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:32.688+0000 7f2322e00640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f231c0cccf0 0x7f231c141aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:01:32.691 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:32.688+0000 7f2322e00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f231c0ced30 0x7f231c13ef10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:01:32.691 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:32.688+0000 7f2322e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f231c145c50 0x7f231c0cb670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:01:32.692 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:32.688+0000 7f2322e00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f231c0b40b0 con 0x7f231c145c50 2024-08-03T21:01:32.692 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:32.688+0000 7f2322e00640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f231c0b3f30 con 0x7f231c0ced30 2024-08-03T21:01:32.692 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:32.688+0000 7f2322e00640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7f231c0b4230 con 0x7f231c0cccf0 2024-08-03T21:01:32.692 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:32.689+0000 7f2320e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f231c145c50 0x7f231c0cb670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:01:32.692 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:32.689+0000 7f2320e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f231c145c50 0x7f231c0cb670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:39544/0 (socket says 172.21.15.32:39544) 2024-08-03T21:01:32.692 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:32.689+0000 7f2320e00640 1 -- 172.21.15.32:0/1945424355 learned_addr learned my addr 172.21.15.32:0/1945424355 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T21:01:32.692 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:32.689+0000 7f2313400640 1 --2- 172.21.15.32:0/1945424355 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f231c0ced30 0x7f231c13ef10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:01:32.693 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:32.689+0000 7f231be00640 1 --2- 172.21.15.32:0/1945424355 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f231c0cccf0 0x7f231c141aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:01:32.693 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:32.689+0000 7f2320e00640 1 -- 172.21.15.32:0/1945424355 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f231c0cccf0 msgr2=0x7f231c141aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:01:32.693 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:32.689+0000 7f2320e00640 1 --2- 172.21.15.32:0/1945424355 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f231c0cccf0 0x7f231c141aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:01:32.693 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:32.689+0000 7f2320e00640 1 -- 172.21.15.32:0/1945424355 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f231c0ced30 msgr2=0x7f231c13ef10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:01:32.693 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:32.689+0000 7f2320e00640 1 --2- 172.21.15.32:0/1945424355 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f231c0ced30 0x7f231c13ef10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:01:32.693 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:32.689+0000 7f2320e00640 1 -- 172.21.15.32:0/1945424355 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f231c13e190 con 0x7f231c145c50 2024-08-03T21:01:32.693 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:32.689+0000 7f2320e00640 1 --2- 172.21.15.32:0/1945424355 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f231c145c50 0x7f231c0cb670 secure :-1 s=READY pgs=2291 cs=0 l=1 rev1=1 crypto rx=0x7f231400b9c0 tx=0x7f231400be80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T21:01:32.693 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:32.690+0000 7f2318c00640 1 -- 172.21.15.32:0/1945424355 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2314019070 con 0x7f231c145c50 2024-08-03T21:01:32.694 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:32.690+0000 7f2318c00640 1 -- 172.21.15.32:0/1945424355 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f231400cc30 con 0x7f231c145c50 2024-08-03T21:01:32.694 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:32.690+0000 7f2318c00640 1 -- 172.21.15.32:0/1945424355 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f23140146a0 con 0x7f231c145c50 2024-08-03T21:01:32.694 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:32.690+0000 7f2322e00640 1 -- 172.21.15.32:0/1945424355 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f231c0cbbb0 con 0x7f231c145c50 2024-08-03T21:01:32.694 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:32.690+0000 7f2322e00640 1 -- 172.21.15.32:0/1945424355 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f231c13e430 con 0x7f231c145c50 2024-08-03T21:01:32.695 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:32.691+0000 7f2322e00640 1 -- 172.21.15.32:0/1945424355 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f231c0b47d0 con 0x7f231c145c50 2024-08-03T21:01:32.695 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:32.692+0000 7f2318c00640 1 -- 172.21.15.32:0/1945424355 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f231401c070 con 0x7f231c145c50 2024-08-03T21:01:32.695 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:32.693+0000 7f2318c00640 1 --2- 172.21.15.32:0/1945424355 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f22f8077010 0x7f22f8079520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:01:32.696 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:32.693+0000 7f2318c00640 1 -- 172.21.15.32:0/1945424355 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(139..139 src has 1..139) ==== 5029+0+0 (secure 0 0 0) 0x7f2314098550 con 0x7f231c145c50 2024-08-03T21:01:32.696 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:32.693+0000 7f2318c00640 1 --2- 172.21.15.32:0/1945424355 >> [v2:172.21.15.32:6800/9088303,v1:172.21.15.32:6801/9088303] conn(0x7f22f807cfc0 0x7f22f807f410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:01:32.696 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:32.694+0000 7f2318c00640 1 -- 172.21.15.32:0/1945424355 --> [v2:172.21.15.32:6800/9088303,v1:172.21.15.32:6801/9088303] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f2314018ec0 con 0x7f22f807cfc0 2024-08-03T21:01:32.696 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:32.694+0000 7f2318c00640 1 -- 172.21.15.32:0/1945424355 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_get_version_reply(handle=1 version=139) ==== 24+0+0 (secure 0 0 0) 0x7f231409f050 con 0x7f231c145c50 2024-08-03T21:01:32.696 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:32.694+0000 7f231be00640 1 --2- 172.21.15.32:0/1945424355 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f22f8077010 0x7f22f8079520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:01:32.696 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:32.694+0000 7f2313400640 1 --2- 172.21.15.32:0/1945424355 >> [v2:172.21.15.32:6800/9088303,v1:172.21.15.32:6801/9088303] conn(0x7f22f807cfc0 0x7f22f807f410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:01:32.697 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:32.694+0000 7f231be00640 1 --2- 172.21.15.32:0/1945424355 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f22f8077010 0x7f22f8079520 secure :-1 s=READY pgs=916 cs=0 l=1 rev1=1 crypto rx=0x7f230c00b810 tx=0x7f230c008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T21:01:32.697 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:32.694+0000 7f2313400640 1 --2- 172.21.15.32:0/1945424355 >> [v2:172.21.15.32:6800/9088303,v1:172.21.15.32:6801/9088303] conn(0x7f22f807cfc0 0x7f22f807f410 crc :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T21:01:32.697 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:32.695+0000 7f2318c00640 1 -- 172.21.15.32:0/1945424355 <== osd.3 v2:172.21.15.32:6800/9088303 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f2314018ec0 con 0x7f22f807cfc0 2024-08-03T21:01:32.710 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:32.709+0000 7f2322e00640 1 -- 172.21.15.32:0/1945424355 --> [v2:172.21.15.32:6800/9088303,v1:172.21.15.32:6801/9088303] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7f231c1f6060 con 0x7f22f807cfc0 2024-08-03T21:01:32.711 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:32.709+0000 7f2318c00640 1 -- 172.21.15.32:0/1945424355 <== osd.3 v2:172.21.15.32:6800/9088303 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7f231c1f6060 con 0x7f22f807cfc0 2024-08-03T21:01:32.711 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-08-03T21:01:32.712 INFO:teuthology.orchestra.run.smithi032.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-08-03T21:01:32.712 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-08-03T21:01:32.713 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:32.710+0000 7f2322e00640 1 -- 172.21.15.32:0/1945424355 >> [v2:172.21.15.32:6800/9088303,v1:172.21.15.32:6801/9088303] conn(0x7f22f807cfc0 msgr2=0x7f22f807f410 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:01:32.713 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:32.710+0000 7f2322e00640 1 --2- 172.21.15.32:0/1945424355 >> [v2:172.21.15.32:6800/9088303,v1:172.21.15.32:6801/9088303] conn(0x7f22f807cfc0 0x7f22f807f410 crc :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:01:32.713 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:32.710+0000 7f2322e00640 1 -- 172.21.15.32:0/1945424355 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f22f8077010 msgr2=0x7f22f8079520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:01:32.713 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:32.710+0000 7f2322e00640 1 --2- 172.21.15.32:0/1945424355 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f22f8077010 0x7f22f8079520 secure :-1 s=READY pgs=916 cs=0 l=1 rev1=1 crypto rx=0x7f230c00b810 tx=0x7f230c008040 comp rx=0 tx=0).stop 2024-08-03T21:01:32.713 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:32.710+0000 7f2322e00640 1 -- 172.21.15.32:0/1945424355 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f231c145c50 msgr2=0x7f231c0cb670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:01:32.713 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:32.710+0000 7f2322e00640 1 --2- 172.21.15.32:0/1945424355 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f231c145c50 0x7f231c0cb670 secure :-1 s=READY pgs=2291 cs=0 l=1 rev1=1 crypto rx=0x7f231400b9c0 tx=0x7f231400be80 comp rx=0 tx=0).stop 2024-08-03T21:01:32.713 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:32.710+0000 7f2320e00640 1 -- 172.21.15.32:0/1945424355 reap_dead start 2024-08-03T21:01:32.713 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:32.710+0000 7f2322e00640 1 -- 172.21.15.32:0/1945424355 shutdown_connections 2024-08-03T21:01:32.714 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:32.710+0000 7f2322e00640 1 -- 172.21.15.32:0/1945424355 >> 172.21.15.32:0/1945424355 conn(0x7f231c0b1bd0 msgr2=0x7f231c15f3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T21:01:32.714 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:32.710+0000 7f2322e00640 1 -- 172.21.15.32:0/1945424355 shutdown_connections 2024-08-03T21:01:32.714 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:32.710+0000 7f2322e00640 1 -- 172.21.15.32:0/1945424355 wait complete. 2024-08-03T21:01:32.733 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 3 2024-08-03T21:01:32.967 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:32.965+0000 7ff8db800640 1 -- 172.21.15.32:0/1849951237 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff8b8003a50 msgr2=0x7ff8b8003ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:01:32.968 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:32.965+0000 7ff8db800640 1 --2- 172.21.15.32:0/1849951237 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff8b8003a50 0x7ff8b8003ec0 secure :-1 s=READY pgs=2293 cs=0 l=1 rev1=1 crypto rx=0x7ff8c400bbb0 tx=0x7ff8b8005830 comp rx=0 tx=0).stop 2024-08-03T21:01:32.968 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:32.965+0000 7ff8db800640 1 -- 172.21.15.32:0/1849951237 shutdown_connections 2024-08-03T21:01:32.968 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:32.965+0000 7ff8db800640 1 --2- 172.21.15.32:0/1849951237 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff8b8003a50 0x7ff8b8003ec0 unknown :-1 s=CLOSED pgs=2293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:01:32.968 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:32.965+0000 7ff8db800640 1 -- 172.21.15.32:0/1849951237 >> 172.21.15.32:0/1849951237 conn(0x7ff8d40b1c40 msgr2=0x7ff8d40b2050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T21:01:32.968 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:32.965+0000 7ff8db800640 1 -- 172.21.15.32:0/1849951237 shutdown_connections 2024-08-03T21:01:32.968 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:32.965+0000 7ff8db800640 1 -- 172.21.15.32:0/1849951237 wait complete. 2024-08-03T21:01:32.969 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:32.967+0000 7ff8db800640 1 Processor -- start 2024-08-03T21:01:32.969 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:32.968+0000 7ff8db800640 1 -- start start 2024-08-03T21:01:32.971 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:32.969+0000 7ff8db800640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7ff8b8003a50 0x7ff8d41f5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:01:32.972 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:32.969+0000 7ff8db800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff8d415f8b0 0x7ff8d4157070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:01:32.972 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:32.969+0000 7ff8db800640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ff8d415e320 0x7ff8d4157f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:01:32.972 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:32.969+0000 7ff8db800640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7ff8d40ccbc0 con 0x7ff8d415f8b0 2024-08-03T21:01:32.972 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:32.969+0000 7ff8db800640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7ff8d40cca40 con 0x7ff8d415e320 2024-08-03T21:01:32.972 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:32.969+0000 7ff8db800640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7ff8d4160030 con 0x7ff8b8003a50 2024-08-03T21:01:32.972 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:32.969+0000 7ff8cbe00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff8d415f8b0 0x7ff8d4157070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:01:32.972 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:32.969+0000 7ff8cbe00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff8d415f8b0 0x7ff8d4157070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:39570/0 (socket says 172.21.15.32:39570) 2024-08-03T21:01:32.973 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:32.969+0000 7ff8cbe00640 1 -- 172.21.15.32:0/2602040033 learned_addr learned my addr 172.21.15.32:0/2602040033 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T21:01:32.973 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:32.969+0000 7ff8d8e00640 1 --2- 172.21.15.32:0/2602040033 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7ff8b8003a50 0x7ff8d41f5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:01:32.973 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:32.970+0000 7ff8d9800640 1 --2- 172.21.15.32:0/2602040033 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ff8d415e320 0x7ff8d4157f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:01:32.973 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:32.970+0000 7ff8cbe00640 1 -- 172.21.15.32:0/2602040033 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7ff8b8003a50 msgr2=0x7ff8d41f5170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:01:32.973 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:32.970+0000 7ff8cbe00640 1 --2- 172.21.15.32:0/2602040033 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7ff8b8003a50 0x7ff8d41f5170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:01:32.973 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:32.970+0000 7ff8cbe00640 1 -- 172.21.15.32:0/2602040033 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ff8d415e320 msgr2=0x7ff8d4157f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:01:32.973 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:32.970+0000 7ff8cbe00640 1 --2- 172.21.15.32:0/2602040033 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ff8d415e320 0x7ff8d4157f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:01:32.973 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:32.970+0000 7ff8cbe00640 1 -- 172.21.15.32:0/2602040033 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff8d41f6030 con 0x7ff8d415f8b0 2024-08-03T21:01:32.974 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:32.970+0000 7ff8cbe00640 1 --2- 172.21.15.32:0/2602040033 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff8d415f8b0 0x7ff8d4157070 secure :-1 s=READY pgs=2294 cs=0 l=1 rev1=1 crypto rx=0x7ff8bc006630 tx=0x7ff8bc0039c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T21:01:32.975 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:32.970+0000 7ff8d1600640 1 -- 172.21.15.32:0/2602040033 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8bc015070 con 0x7ff8d415f8b0 2024-08-03T21:01:32.975 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:32.970+0000 7ff8d1600640 1 -- 172.21.15.32:0/2602040033 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7ff8bc011900 con 0x7ff8d415f8b0 2024-08-03T21:01:32.975 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:32.970+0000 7ff8db800640 1 -- 172.21.15.32:0/2602040033 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff8d41f5a70 con 0x7ff8d415f8b0 2024-08-03T21:01:32.975 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:32.970+0000 7ff8d1600640 1 -- 172.21.15.32:0/2602040033 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8bc011d70 con 0x7ff8d415f8b0 2024-08-03T21:01:32.975 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:32.971+0000 7ff8db800640 1 -- 172.21.15.32:0/2602040033 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff8d415db20 con 0x7ff8d415f8b0 2024-08-03T21:01:32.975 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:32.972+0000 7ff8db800640 1 -- 172.21.15.32:0/2602040033 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff8d41f57e0 con 0x7ff8d415f8b0 2024-08-03T21:01:32.975 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:32.973+0000 7ff8d1600640 1 -- 172.21.15.32:0/2602040033 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7ff8bc018070 con 0x7ff8d415f8b0 2024-08-03T21:01:32.976 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:32.974+0000 7ff8d1600640 1 --2- 172.21.15.32:0/2602040033 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7ff8b0077010 0x7ff8b0079520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:01:32.976 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:32.974+0000 7ff8d1600640 1 -- 172.21.15.32:0/2602040033 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(139..139 src has 1..139) ==== 5029+0+0 (secure 0 0 0) 0x7ff8bc099210 con 0x7ff8d415f8b0 2024-08-03T21:01:32.976 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:32.974+0000 7ff8d8e00640 1 --2- 172.21.15.32:0/2602040033 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7ff8b0077010 0x7ff8b0079520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:01:32.977 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:32.975+0000 7ff8d8e00640 1 --2- 172.21.15.32:0/2602040033 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7ff8b0077010 0x7ff8b0079520 secure :-1 s=READY pgs=917 cs=0 l=1 rev1=1 crypto rx=0x7ff8c4036000 tx=0x7ff8c403c040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T21:01:32.980 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:32.978+0000 7ff8d1600640 1 -- 172.21.15.32:0/2602040033 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7ff8bc062060 con 0x7ff8d415f8b0 2024-08-03T21:01:33.183 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:33.182+0000 7ff8db800640 1 -- 172.21.15.32:0/2602040033 --> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "3", "target": ["mon-mgr", ""]}) -- 0x7ff8d4160590 con 0x7ff8b0077010 2024-08-03T21:01:33.184 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:33.182+0000 7ff8d1600640 1 -- 172.21.15.32:0/2602040033 <== mgr.4106 v2:172.21.15.118:6800/4205950699 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 3 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7ff8d4160590 con 0x7ff8b0077010 2024-08-03T21:01:33.184 INFO:teuthology.orchestra.run.smithi032.stderr:instructed osd(s) 3 to deep-scrub 2024-08-03T21:01:33.186 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:33.184+0000 7ff8db800640 1 -- 172.21.15.32:0/2602040033 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7ff8b0077010 msgr2=0x7ff8b0079520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:01:33.187 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:33.184+0000 7ff8db800640 1 --2- 172.21.15.32:0/2602040033 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7ff8b0077010 0x7ff8b0079520 secure :-1 s=READY pgs=917 cs=0 l=1 rev1=1 crypto rx=0x7ff8c4036000 tx=0x7ff8c403c040 comp rx=0 tx=0).stop 2024-08-03T21:01:33.187 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:33.184+0000 7ff8db800640 1 -- 172.21.15.32:0/2602040033 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff8d415f8b0 msgr2=0x7ff8d4157070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:01:33.187 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:33.184+0000 7ff8db800640 1 --2- 172.21.15.32:0/2602040033 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff8d415f8b0 0x7ff8d4157070 secure :-1 s=READY pgs=2294 cs=0 l=1 rev1=1 crypto rx=0x7ff8bc006630 tx=0x7ff8bc0039c0 comp rx=0 tx=0).stop 2024-08-03T21:01:33.187 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:33.185+0000 7ff8db800640 1 -- 172.21.15.32:0/2602040033 shutdown_connections 2024-08-03T21:01:33.187 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:33.185+0000 7ff8db800640 1 --2- 172.21.15.32:0/2602040033 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7ff8b0077010 0x7ff8b0079520 unknown :-1 s=CLOSED pgs=917 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:01:33.187 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:33.185+0000 7ff8db800640 1 --2- 172.21.15.32:0/2602040033 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ff8d415e320 0x7ff8d4157f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:01:33.187 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:33.185+0000 7ff8db800640 1 --2- 172.21.15.32:0/2602040033 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff8d415f8b0 0x7ff8d4157070 unknown :-1 s=CLOSED pgs=2294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:01:33.188 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:33.185+0000 7ff8db800640 1 --2- 172.21.15.32:0/2602040033 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7ff8b8003a50 0x7ff8d41f5170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:01:33.188 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:33.185+0000 7ff8db800640 1 -- 172.21.15.32:0/2602040033 >> 172.21.15.32:0/2602040033 conn(0x7ff8d40b1c40 msgr2=0x7ff8d415a690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T21:01:33.188 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:33.185+0000 7ff8db800640 1 -- 172.21.15.32:0/2602040033 shutdown_connections 2024-08-03T21:01:33.188 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:33.185+0000 7ff8db800640 1 -- 172.21.15.32:0/2602040033 wait complete. 2024-08-03T21:01:33.208 INFO:tasks.ceph:Scrubbing osd.4 2024-08-03T21:01:33.208 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 config set osd_debug_deep_scrub_sleep 0 2024-08-03T21:01:33.442 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:33.440+0000 7f9c8de00640 1 -- 172.21.15.32:0/3701298055 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f9c88145c50 msgr2=0x7f9c8806e950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:01:33.442 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:33.440+0000 7f9c8de00640 1 --2- 172.21.15.32:0/3701298055 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f9c88145c50 0x7f9c8806e950 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f9c78009860 tx=0x7f9c7802a530 comp rx=0 tx=0).stop 2024-08-03T21:01:33.443 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:33.440+0000 7f9c8de00640 1 -- 172.21.15.32:0/3701298055 shutdown_connections 2024-08-03T21:01:33.443 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:33.440+0000 7f9c8de00640 1 --2- 172.21.15.32:0/3701298055 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f9c88145c50 0x7f9c8806e950 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:01:33.443 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:33.440+0000 7f9c8de00640 1 -- 172.21.15.32:0/3701298055 >> 172.21.15.32:0/3701298055 conn(0x7f9c880b1bd0 msgr2=0x7f9c880b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T21:01:33.443 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:33.440+0000 7f9c8de00640 1 -- 172.21.15.32:0/3701298055 shutdown_connections 2024-08-03T21:01:33.443 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:33.440+0000 7f9c8de00640 1 -- 172.21.15.32:0/3701298055 wait complete. 2024-08-03T21:01:33.444 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:33.443+0000 7f9c8de00640 1 Processor -- start 2024-08-03T21:01:33.444 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:33.443+0000 7f9c8de00640 1 -- start start 2024-08-03T21:01:33.447 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:33.444+0000 7f9c8de00640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f9c880cccf0 0x7f9c8813db30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:01:33.447 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:33.444+0000 7f9c8de00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f9c880ced30 0x7f9c8813e070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:01:33.447 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:33.444+0000 7f9c8de00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9c88145c50 0x7f9c881f6b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:01:33.447 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:33.444+0000 7f9c8de00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f9c880b4230 con 0x7f9c88145c50 2024-08-03T21:01:33.447 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:33.444+0000 7f9c8de00640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f9c880b40b0 con 0x7f9c880ced30 2024-08-03T21:01:33.447 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:33.444+0000 7f9c8de00640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7f9c880b43b0 con 0x7f9c880cccf0 2024-08-03T21:01:33.447 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:33.445+0000 7f9c87400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9c88145c50 0x7f9c881f6b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:01:33.448 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:33.445+0000 7f9c87400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9c88145c50 0x7f9c881f6b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:39580/0 (socket says 172.21.15.32:39580) 2024-08-03T21:01:33.448 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:33.445+0000 7f9c87400640 1 -- 172.21.15.32:0/2263544367 learned_addr learned my addr 172.21.15.32:0/2263544367 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T21:01:33.448 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:33.445+0000 7f9c86000640 1 --2- 172.21.15.32:0/2263544367 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f9c880ced30 0x7f9c8813e070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:01:33.448 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:33.445+0000 7f9c86a00640 1 --2- 172.21.15.32:0/2263544367 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f9c880cccf0 0x7f9c8813db30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:01:33.448 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:33.445+0000 7f9c87400640 1 -- 172.21.15.32:0/2263544367 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f9c880cccf0 msgr2=0x7f9c8813db30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:01:33.448 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:33.445+0000 7f9c87400640 1 --2- 172.21.15.32:0/2263544367 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f9c880cccf0 0x7f9c8813db30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:01:33.448 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:33.445+0000 7f9c87400640 1 -- 172.21.15.32:0/2263544367 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f9c880ced30 msgr2=0x7f9c8813e070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:01:33.448 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:33.445+0000 7f9c87400640 1 --2- 172.21.15.32:0/2263544367 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f9c880ced30 0x7f9c8813e070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:01:33.449 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:33.445+0000 7f9c87400640 1 -- 172.21.15.32:0/2263544367 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9c8813e5b0 con 0x7f9c88145c50 2024-08-03T21:01:33.449 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:33.446+0000 7f9c87400640 1 --2- 172.21.15.32:0/2263544367 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9c88145c50 0x7f9c881f6b90 secure :-1 s=READY pgs=2296 cs=0 l=1 rev1=1 crypto rx=0x7f9c8000c730 tx=0x7f9c8000cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T21:01:33.450 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:33.446+0000 7f9c7f400640 1 -- 172.21.15.32:0/2263544367 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c80003b70 con 0x7f9c88145c50 2024-08-03T21:01:33.450 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:33.446+0000 7f9c7f400640 1 -- 172.21.15.32:0/2263544367 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f9c80003d10 con 0x7f9c88145c50 2024-08-03T21:01:33.450 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:33.446+0000 7f9c8de00640 1 -- 172.21.15.32:0/2263544367 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9c881e6d20 con 0x7f9c88145c50 2024-08-03T21:01:33.450 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:33.446+0000 7f9c7f400640 1 -- 172.21.15.32:0/2263544367 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c8000aab0 con 0x7f9c88145c50 2024-08-03T21:01:33.450 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:33.446+0000 7f9c8de00640 1 -- 172.21.15.32:0/2263544367 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9c881e72e0 con 0x7f9c88145c50 2024-08-03T21:01:33.450 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:33.447+0000 7f9c8de00640 1 -- 172.21.15.32:0/2263544367 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f9c8806e950 con 0x7f9c88145c50 2024-08-03T21:01:33.451 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:33.448+0000 7f9c7f400640 1 -- 172.21.15.32:0/2263544367 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f9c8000ac50 con 0x7f9c88145c50 2024-08-03T21:01:33.451 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:33.449+0000 7f9c7f400640 1 --2- 172.21.15.32:0/2263544367 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f9c68077010 0x7f9c68079520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:01:33.452 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:33.449+0000 7f9c7f400640 1 -- 172.21.15.32:0/2263544367 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(139..139 src has 1..139) ==== 5029+0+0 (secure 0 0 0) 0x7f9c800983a0 con 0x7f9c88145c50 2024-08-03T21:01:33.452 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:33.449+0000 7f9c7f400640 1 --2- 172.21.15.32:0/2263544367 >> [v2:172.21.15.118:6802/864752275,v1:172.21.15.118:6803/864752275] conn(0x7f9c6807cfc0 0x7f9c6807f410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:01:33.452 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:33.449+0000 7f9c7f400640 1 -- 172.21.15.32:0/2263544367 --> [v2:172.21.15.118:6802/864752275,v1:172.21.15.118:6803/864752275] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f9c80011e50 con 0x7f9c6807cfc0 2024-08-03T21:01:33.452 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:33.449+0000 7f9c7f400640 1 -- 172.21.15.32:0/2263544367 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_get_version_reply(handle=1 version=139) ==== 24+0+0 (secure 0 0 0) 0x7f9c8009f050 con 0x7f9c88145c50 2024-08-03T21:01:33.452 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:33.450+0000 7f9c86a00640 1 --2- 172.21.15.32:0/2263544367 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f9c68077010 0x7f9c68079520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:01:33.452 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:33.450+0000 7f9c86000640 1 --2- 172.21.15.32:0/2263544367 >> [v2:172.21.15.118:6802/864752275,v1:172.21.15.118:6803/864752275] conn(0x7f9c6807cfc0 0x7f9c6807f410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:01:33.453 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:33.450+0000 7f9c86a00640 1 --2- 172.21.15.32:0/2263544367 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f9c68077010 0x7f9c68079520 secure :-1 s=READY pgs=918 cs=0 l=1 rev1=1 crypto rx=0x7f9c78034000 tx=0x7f9c78004730 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T21:01:33.453 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:33.451+0000 7f9c86000640 1 --2- 172.21.15.32:0/2263544367 >> [v2:172.21.15.118:6802/864752275,v1:172.21.15.118:6803/864752275] conn(0x7f9c6807cfc0 0x7f9c6807f410 crc :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T21:01:33.453 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:33.452+0000 7f9c7f400640 1 -- 172.21.15.32:0/2263544367 <== osd.4 v2:172.21.15.118:6802/864752275 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f9c80011e50 con 0x7f9c6807cfc0 2024-08-03T21:01:33.467 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:33.466+0000 7f9c8de00640 1 -- 172.21.15.32:0/2263544367 --> [v2:172.21.15.118:6802/864752275,v1:172.21.15.118:6803/864752275] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7f9c881e6ad0 con 0x7f9c6807cfc0 2024-08-03T21:01:33.468 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:33.467+0000 7f9c7f400640 1 -- 172.21.15.32:0/2263544367 <== osd.4 v2:172.21.15.118:6802/864752275 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7f9c881e6ad0 con 0x7f9c6807cfc0 2024-08-03T21:01:33.468 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-08-03T21:01:33.469 INFO:teuthology.orchestra.run.smithi032.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-08-03T21:01:33.469 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-08-03T21:01:33.470 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:33.467+0000 7f9c8de00640 1 -- 172.21.15.32:0/2263544367 >> [v2:172.21.15.118:6802/864752275,v1:172.21.15.118:6803/864752275] conn(0x7f9c6807cfc0 msgr2=0x7f9c6807f410 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:01:33.470 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:33.467+0000 7f9c8de00640 1 --2- 172.21.15.32:0/2263544367 >> [v2:172.21.15.118:6802/864752275,v1:172.21.15.118:6803/864752275] conn(0x7f9c6807cfc0 0x7f9c6807f410 crc :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:01:33.470 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:33.467+0000 7f9c8de00640 1 -- 172.21.15.32:0/2263544367 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f9c68077010 msgr2=0x7f9c68079520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:01:33.470 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:33.467+0000 7f9c8de00640 1 --2- 172.21.15.32:0/2263544367 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f9c68077010 0x7f9c68079520 secure :-1 s=READY pgs=918 cs=0 l=1 rev1=1 crypto rx=0x7f9c78034000 tx=0x7f9c78004730 comp rx=0 tx=0).stop 2024-08-03T21:01:33.470 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:33.467+0000 7f9c8de00640 1 -- 172.21.15.32:0/2263544367 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9c88145c50 msgr2=0x7f9c881f6b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:01:33.470 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:33.467+0000 7f9c8de00640 1 --2- 172.21.15.32:0/2263544367 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9c88145c50 0x7f9c881f6b90 secure :-1 s=READY pgs=2296 cs=0 l=1 rev1=1 crypto rx=0x7f9c8000c730 tx=0x7f9c8000cbf0 comp rx=0 tx=0).stop 2024-08-03T21:01:33.471 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:33.467+0000 7f9c87400640 1 -- 172.21.15.32:0/2263544367 reap_dead start 2024-08-03T21:01:33.471 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:33.467+0000 7f9c8de00640 1 -- 172.21.15.32:0/2263544367 shutdown_connections 2024-08-03T21:01:33.471 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:33.467+0000 7f9c8de00640 1 -- 172.21.15.32:0/2263544367 >> 172.21.15.32:0/2263544367 conn(0x7f9c880b1bd0 msgr2=0x7f9c881452e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T21:01:33.471 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:33.467+0000 7f9c8de00640 1 -- 172.21.15.32:0/2263544367 shutdown_connections 2024-08-03T21:01:33.471 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:33.467+0000 7f9c8de00640 1 -- 172.21.15.32:0/2263544367 wait complete. 2024-08-03T21:01:33.492 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 4 2024-08-03T21:01:33.721 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:33.718+0000 7f1efd400640 1 -- 172.21.15.32:0/81709166 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1ef8160010 msgr2=0x7f1ef8160420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:01:33.721 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:33.718+0000 7f1efd400640 1 --2- 172.21.15.32:0/81709166 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1ef8160010 0x7f1ef8160420 secure :-1 s=READY pgs=2297 cs=0 l=1 rev1=1 crypto rx=0x7f1ef8153b10 tx=0x7f1ee002a560 comp rx=0 tx=0).stop 2024-08-03T21:01:33.721 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:33.718+0000 7f1efd400640 1 -- 172.21.15.32:0/81709166 shutdown_connections 2024-08-03T21:01:33.721 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:33.718+0000 7f1efd400640 1 --2- 172.21.15.32:0/81709166 >> v2:172.21.15.118:3300/0 conn(0x7f1ef806f3d0 0x7f1ef806f7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:01:33.722 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:33.718+0000 7f1efd400640 1 --2- 172.21.15.32:0/81709166 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1ef8160010 0x7f1ef8160420 unknown :-1 s=CLOSED pgs=2297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:01:33.722 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:33.718+0000 7f1efd400640 1 -- 172.21.15.32:0/81709166 >> 172.21.15.32:0/81709166 conn(0x7f1ef80b1c40 msgr2=0x7f1ef80b2050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T21:01:33.722 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:33.719+0000 7f1efd400640 1 -- 172.21.15.32:0/81709166 shutdown_connections 2024-08-03T21:01:33.722 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:33.719+0000 7f1efd400640 1 -- 172.21.15.32:0/81709166 wait complete. 2024-08-03T21:01:33.722 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:33.721+0000 7f1efd400640 1 Processor -- start 2024-08-03T21:01:33.723 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:33.721+0000 7f1efd400640 1 -- start start 2024-08-03T21:01:33.725 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:33.722+0000 7f1efd400640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f1ef806f3d0 0x7f1ef80ca100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:01:33.725 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:33.722+0000 7f1efd400640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f1ef8145b40 0x7f1ef81f5650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:01:33.726 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:33.722+0000 7f1efd400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1ef8160010 0x7f1ef81597d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:01:33.726 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:33.722+0000 7f1efd400640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f1ef806fd20 con 0x7f1ef8160010 2024-08-03T21:01:33.726 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:33.723+0000 7f1efd400640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f1ef80cd080 con 0x7f1ef8145b40 2024-08-03T21:01:33.726 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:33.723+0000 7f1efd400640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7f1ef806fea0 con 0x7f1ef806f3d0 2024-08-03T21:01:33.726 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:33.723+0000 7f1ef7200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1ef8160010 0x7f1ef81597d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:01:33.726 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:33.723+0000 7f1ef7200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1ef8160010 0x7f1ef81597d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:39598/0 (socket says 172.21.15.32:39598) 2024-08-03T21:01:33.726 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:33.723+0000 7f1ef7200640 1 -- 172.21.15.32:0/4082945284 learned_addr learned my addr 172.21.15.32:0/4082945284 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T21:01:33.727 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:33.723+0000 7f1eede00640 1 --2- 172.21.15.32:0/4082945284 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f1ef8145b40 0x7f1ef81f5650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:01:33.727 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:33.723+0000 7f1ef6800640 1 --2- 172.21.15.32:0/4082945284 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f1ef806f3d0 0x7f1ef80ca100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:01:33.727 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:33.723+0000 7f1ef7200640 1 -- 172.21.15.32:0/4082945284 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f1ef806f3d0 msgr2=0x7f1ef80ca100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:01:33.727 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:33.723+0000 7f1ef7200640 1 --2- 172.21.15.32:0/4082945284 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f1ef806f3d0 0x7f1ef80ca100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:01:33.727 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:33.723+0000 7f1ef7200640 1 -- 172.21.15.32:0/4082945284 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f1ef8145b40 msgr2=0x7f1ef81f5650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:01:33.727 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:33.723+0000 7f1ef7200640 1 --2- 172.21.15.32:0/4082945284 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f1ef8145b40 0x7f1ef81f5650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:01:33.727 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:33.723+0000 7f1ef7200640 1 -- 172.21.15.32:0/4082945284 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1ef8072c60 con 0x7f1ef8160010 2024-08-03T21:01:33.728 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:33.724+0000 7f1ef7200640 1 --2- 172.21.15.32:0/4082945284 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1ef8160010 0x7f1ef81597d0 secure :-1 s=READY pgs=2298 cs=0 l=1 rev1=1 crypto rx=0x7f1ef000e960 tx=0x7f1ef000ee20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T21:01:33.728 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:33.724+0000 7f1eef400640 1 -- 172.21.15.32:0/4082945284 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ef0004d20 con 0x7f1ef8160010 2024-08-03T21:01:33.728 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:33.724+0000 7f1eef400640 1 -- 172.21.15.32:0/4082945284 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f1ef000cb20 con 0x7f1ef8160010 2024-08-03T21:01:33.728 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:33.724+0000 7f1eef400640 1 -- 172.21.15.32:0/4082945284 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ef0005020 con 0x7f1ef8160010 2024-08-03T21:01:33.728 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:33.724+0000 7f1efd400640 1 -- 172.21.15.32:0/4082945284 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1ef81f6120 con 0x7f1ef8160010 2024-08-03T21:01:33.728 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:33.724+0000 7f1efd400640 1 -- 172.21.15.32:0/4082945284 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1ef8154950 con 0x7f1ef8160010 2024-08-03T21:01:33.729 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:33.725+0000 7f1efd400640 1 -- 172.21.15.32:0/4082945284 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1ef8072e30 con 0x7f1ef8160010 2024-08-03T21:01:33.729 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:33.726+0000 7f1eef400640 1 -- 172.21.15.32:0/4082945284 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f1ef00051c0 con 0x7f1ef8160010 2024-08-03T21:01:33.729 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:33.727+0000 7f1eef400640 1 --2- 172.21.15.32:0/4082945284 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f1ed4077010 0x7f1ed4079520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:01:33.729 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:33.727+0000 7f1eef400640 1 -- 172.21.15.32:0/4082945284 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(139..139 src has 1..139) ==== 5029+0+0 (secure 0 0 0) 0x7f1ef0098e80 con 0x7f1ef8160010 2024-08-03T21:01:33.730 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:33.727+0000 7f1ef6800640 1 --2- 172.21.15.32:0/4082945284 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f1ed4077010 0x7f1ed4079520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:01:33.730 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:33.728+0000 7f1ef6800640 1 --2- 172.21.15.32:0/4082945284 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f1ed4077010 0x7f1ed4079520 secure :-1 s=READY pgs=919 cs=0 l=1 rev1=1 crypto rx=0x7f1ee8006fd0 tx=0x7f1ee8009060 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T21:01:33.733 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:33.731+0000 7f1eef400640 1 -- 172.21.15.32:0/4082945284 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7f1ef0016030 con 0x7f1ef8160010 2024-08-03T21:01:33.937 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:33.935+0000 7f1efd400640 1 -- 172.21.15.32:0/4082945284 --> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "4", "target": ["mon-mgr", ""]}) -- 0x7f1ef8154770 con 0x7f1ed4077010 2024-08-03T21:01:33.938 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:33.936+0000 7f1eef400640 1 -- 172.21.15.32:0/4082945284 <== mgr.4106 v2:172.21.15.118:6800/4205950699 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 4 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7f1ef8154770 con 0x7f1ed4077010 2024-08-03T21:01:33.938 INFO:teuthology.orchestra.run.smithi032.stderr:instructed osd(s) 4 to deep-scrub 2024-08-03T21:01:33.940 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:33.938+0000 7f1efd400640 1 -- 172.21.15.32:0/4082945284 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f1ed4077010 msgr2=0x7f1ed4079520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:01:33.940 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:33.938+0000 7f1efd400640 1 --2- 172.21.15.32:0/4082945284 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f1ed4077010 0x7f1ed4079520 secure :-1 s=READY pgs=919 cs=0 l=1 rev1=1 crypto rx=0x7f1ee8006fd0 tx=0x7f1ee8009060 comp rx=0 tx=0).stop 2024-08-03T21:01:33.940 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:33.938+0000 7f1efd400640 1 -- 172.21.15.32:0/4082945284 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1ef8160010 msgr2=0x7f1ef81597d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:01:33.940 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:33.938+0000 7f1efd400640 1 --2- 172.21.15.32:0/4082945284 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1ef8160010 0x7f1ef81597d0 secure :-1 s=READY pgs=2298 cs=0 l=1 rev1=1 crypto rx=0x7f1ef000e960 tx=0x7f1ef000ee20 comp rx=0 tx=0).stop 2024-08-03T21:01:33.941 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:33.938+0000 7f1efd400640 1 -- 172.21.15.32:0/4082945284 shutdown_connections 2024-08-03T21:01:33.941 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:33.938+0000 7f1efd400640 1 --2- 172.21.15.32:0/4082945284 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f1ed4077010 0x7f1ed4079520 unknown :-1 s=CLOSED pgs=919 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:01:33.941 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:33.938+0000 7f1efd400640 1 --2- 172.21.15.32:0/4082945284 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f1ef8145b40 0x7f1ef81f5650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:01:33.941 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:33.938+0000 7f1efd400640 1 --2- 172.21.15.32:0/4082945284 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1ef8160010 0x7f1ef81597d0 unknown :-1 s=CLOSED pgs=2298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:01:33.941 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:33.938+0000 7f1efd400640 1 --2- 172.21.15.32:0/4082945284 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f1ef806f3d0 0x7f1ef80ca100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:01:33.941 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:33.938+0000 7f1efd400640 1 -- 172.21.15.32:0/4082945284 >> 172.21.15.32:0/4082945284 conn(0x7f1ef80b1c40 msgr2=0x7f1ef80b2050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T21:01:33.941 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:33.938+0000 7f1efd400640 1 -- 172.21.15.32:0/4082945284 shutdown_connections 2024-08-03T21:01:33.941 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:33.938+0000 7f1efd400640 1 -- 172.21.15.32:0/4082945284 wait complete. 2024-08-03T21:01:33.966 INFO:tasks.ceph:Scrubbing osd.5 2024-08-03T21:01:33.966 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 config set osd_debug_deep_scrub_sleep 0 2024-08-03T21:01:34.199 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:34.197+0000 7f6c2b200640 1 -- 172.21.15.32:0/2522054871 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6c2c0ce700 msgr2=0x7f6c2c0ceb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:01:34.200 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:34.197+0000 7f6c2b200640 1 --2- 172.21.15.32:0/2522054871 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6c2c0ce700 0x7f6c2c0ceb10 secure :-1 s=READY pgs=2299 cs=0 l=1 rev1=1 crypto rx=0x7f6c2c073d70 tx=0x7f6c2402aac0 comp rx=0 tx=0).stop 2024-08-03T21:01:34.200 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:34.197+0000 7f6c2b200640 1 -- 172.21.15.32:0/2522054871 shutdown_connections 2024-08-03T21:01:34.200 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:34.197+0000 7f6c2b200640 1 --2- 172.21.15.32:0/2522054871 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6c2c0ce700 0x7f6c2c0ceb10 unknown :-1 s=CLOSED pgs=2299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:01:34.200 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:34.197+0000 7f6c2b200640 1 --2- 172.21.15.32:0/2522054871 >> v2:172.21.15.118:3300/0 conn(0x7f6c2c0cd5f0 0x7f6c2c0d29a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:01:34.200 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:34.197+0000 7f6c2b200640 1 -- 172.21.15.32:0/2522054871 >> 172.21.15.32:0/2522054871 conn(0x7f6c2c0d1e80 msgr2=0x7f6c2c0d2290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T21:01:34.200 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:34.198+0000 7f6c2b200640 1 -- 172.21.15.32:0/2522054871 shutdown_connections 2024-08-03T21:01:34.200 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:34.198+0000 7f6c2b200640 1 -- 172.21.15.32:0/2522054871 wait complete. 2024-08-03T21:01:34.201 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:34.200+0000 7f6c2b200640 1 Processor -- start 2024-08-03T21:01:34.201 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:34.200+0000 7f6c2b200640 1 -- start start 2024-08-03T21:01:34.204 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:34.201+0000 7f6c2b200640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f6c2c0b7420 0x7f6c2c1f5fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:01:34.204 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:34.201+0000 7f6c2b200640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f6c2c0cd5f0 0x7f6c2c156340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:01:34.204 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:34.201+0000 7f6c2b200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6c2c0ce700 0x7f6c2c1f8ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:01:34.204 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:34.201+0000 7f6c2b200640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f6c2c0d09f0 con 0x7f6c2c0ce700 2024-08-03T21:01:34.204 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:34.201+0000 7f6c2b200640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f6c2c0d0870 con 0x7f6c2c0b7420 2024-08-03T21:01:34.204 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:34.201+0000 7f6c2b200640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7f6c2c0d0b70 con 0x7f6c2c0cd5f0 2024-08-03T21:01:34.204 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:34.202+0000 7f6c2a800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6c2c0ce700 0x7f6c2c1f8ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:01:34.205 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:34.202+0000 7f6c29e00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f6c2c0b7420 0x7f6c2c1f5fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:01:34.205 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:34.202+0000 7f6c2a800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6c2c0ce700 0x7f6c2c1f8ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:39614/0 (socket says 172.21.15.32:39614) 2024-08-03T21:01:34.205 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:34.202+0000 7f6c2a800640 1 -- 172.21.15.32:0/3950198978 learned_addr learned my addr 172.21.15.32:0/3950198978 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T21:01:34.205 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:34.202+0000 7f6c29400640 1 --2- 172.21.15.32:0/3950198978 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f6c2c0cd5f0 0x7f6c2c156340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:01:34.205 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:34.202+0000 7f6c2a800640 1 -- 172.21.15.32:0/3950198978 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f6c2c0cd5f0 msgr2=0x7f6c2c156340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:01:34.205 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:34.202+0000 7f6c2a800640 1 --2- 172.21.15.32:0/3950198978 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f6c2c0cd5f0 0x7f6c2c156340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:01:34.205 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:34.202+0000 7f6c2a800640 1 -- 172.21.15.32:0/3950198978 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f6c2c0b7420 msgr2=0x7f6c2c1f5fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:01:34.205 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:34.202+0000 7f6c2a800640 1 --2- 172.21.15.32:0/3950198978 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f6c2c0b7420 0x7f6c2c1f5fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:01:34.206 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:34.202+0000 7f6c2a800640 1 -- 172.21.15.32:0/3950198978 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6c2c1f9560 con 0x7f6c2c0ce700 2024-08-03T21:01:34.206 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:34.202+0000 7f6c29e00640 1 --2- 172.21.15.32:0/3950198978 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f6c2c0b7420 0x7f6c2c1f5fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-03T21:01:34.207 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:34.203+0000 7f6c2a800640 1 --2- 172.21.15.32:0/3950198978 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6c2c0ce700 0x7f6c2c1f8ac0 secure :-1 s=READY pgs=2300 cs=0 l=1 rev1=1 crypto rx=0x7f6c2402b0b0 tx=0x7f6c24003ce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T21:01:34.207 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:34.203+0000 7f6c22000640 1 -- 172.21.15.32:0/3950198978 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c24004250 con 0x7f6c2c0ce700 2024-08-03T21:01:34.207 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:34.203+0000 7f6c22000640 1 -- 172.21.15.32:0/3950198978 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f6c240043f0 con 0x7f6c2c0ce700 2024-08-03T21:01:34.207 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:34.203+0000 7f6c2b200640 1 -- 172.21.15.32:0/3950198978 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6c2c1f9840 con 0x7f6c2c0ce700 2024-08-03T21:01:34.207 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:34.203+0000 7f6c22000640 1 -- 172.21.15.32:0/3950198978 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c24004840 con 0x7f6c2c0ce700 2024-08-03T21:01:34.207 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:34.203+0000 7f6c2b200640 1 -- 172.21.15.32:0/3950198978 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6c2c156880 con 0x7f6c2c0ce700 2024-08-03T21:01:34.208 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:34.204+0000 7f6c2b200640 1 -- 172.21.15.32:0/3950198978 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f6c2c0d1100 con 0x7f6c2c0ce700 2024-08-03T21:01:34.208 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:34.205+0000 7f6c22000640 1 -- 172.21.15.32:0/3950198978 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f6c24049430 con 0x7f6c2c0ce700 2024-08-03T21:01:34.209 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:34.206+0000 7f6c22000640 1 --2- 172.21.15.32:0/3950198978 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f6c0c077010 0x7f6c0c079520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:01:34.209 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:34.206+0000 7f6c22000640 1 -- 172.21.15.32:0/3950198978 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(139..139 src has 1..139) ==== 5029+0+0 (secure 0 0 0) 0x7f6c240b9970 con 0x7f6c2c0ce700 2024-08-03T21:01:34.209 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:34.207+0000 7f6c22000640 1 --2- 172.21.15.32:0/3950198978 >> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] conn(0x7f6c0c07cfc0 0x7f6c0c07f410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:01:34.209 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:34.207+0000 7f6c22000640 1 -- 172.21.15.32:0/3950198978 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f6c240059c0 con 0x7f6c0c07cfc0 2024-08-03T21:01:34.209 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:34.207+0000 7f6c22000640 1 -- 172.21.15.32:0/3950198978 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_get_version_reply(handle=1 version=139) ==== 24+0+0 (secure 0 0 0) 0x7f6c240b9d20 con 0x7f6c2c0ce700 2024-08-03T21:01:34.209 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:34.207+0000 7f6c29e00640 1 --2- 172.21.15.32:0/3950198978 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f6c0c077010 0x7f6c0c079520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:01:34.210 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:34.207+0000 7f6c29400640 1 --2- 172.21.15.32:0/3950198978 >> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] conn(0x7f6c0c07cfc0 0x7f6c0c07f410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:01:34.210 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:34.208+0000 7f6c29e00640 1 --2- 172.21.15.32:0/3950198978 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f6c0c077010 0x7f6c0c079520 secure :-1 s=READY pgs=920 cs=0 l=1 rev1=1 crypto rx=0x7f6c1c00bf40 tx=0x7f6c1c00b2e0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T21:01:34.210 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:34.208+0000 7f6c29400640 1 --2- 172.21.15.32:0/3950198978 >> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] conn(0x7f6c0c07cfc0 0x7f6c0c07f410 crc :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T21:01:34.211 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:34.209+0000 7f6c22000640 1 -- 172.21.15.32:0/3950198978 <== osd.5 v2:172.21.15.118:6810/2724891770 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f6c240059c0 con 0x7f6c0c07cfc0 2024-08-03T21:01:34.224 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:34.223+0000 7f6c2b200640 1 -- 172.21.15.32:0/3950198978 --> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7f6c2c1f9310 con 0x7f6c0c07cfc0 2024-08-03T21:01:34.225 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:34.223+0000 7f6c22000640 1 -- 172.21.15.32:0/3950198978 <== osd.5 v2:172.21.15.118:6810/2724891770 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7f6c2c1f9310 con 0x7f6c0c07cfc0 2024-08-03T21:01:34.225 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-08-03T21:01:34.225 INFO:teuthology.orchestra.run.smithi032.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-08-03T21:01:34.226 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-08-03T21:01:34.226 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:34.224+0000 7f6c2b200640 1 -- 172.21.15.32:0/3950198978 >> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] conn(0x7f6c0c07cfc0 msgr2=0x7f6c0c07f410 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:01:34.227 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:34.224+0000 7f6c2b200640 1 --2- 172.21.15.32:0/3950198978 >> [v2:172.21.15.118:6810/2724891770,v1:172.21.15.118:6811/2724891770] conn(0x7f6c0c07cfc0 0x7f6c0c07f410 crc :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:01:34.227 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:34.224+0000 7f6c2b200640 1 -- 172.21.15.32:0/3950198978 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f6c0c077010 msgr2=0x7f6c0c079520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:01:34.227 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:34.224+0000 7f6c2b200640 1 --2- 172.21.15.32:0/3950198978 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f6c0c077010 0x7f6c0c079520 secure :-1 s=READY pgs=920 cs=0 l=1 rev1=1 crypto rx=0x7f6c1c00bf40 tx=0x7f6c1c00b2e0 comp rx=0 tx=0).stop 2024-08-03T21:01:34.227 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:34.224+0000 7f6c2b200640 1 -- 172.21.15.32:0/3950198978 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6c2c0ce700 msgr2=0x7f6c2c1f8ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:01:34.227 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:34.224+0000 7f6c2b200640 1 --2- 172.21.15.32:0/3950198978 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6c2c0ce700 0x7f6c2c1f8ac0 secure :-1 s=READY pgs=2300 cs=0 l=1 rev1=1 crypto rx=0x7f6c2402b0b0 tx=0x7f6c24003ce0 comp rx=0 tx=0).stop 2024-08-03T21:01:34.227 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:34.224+0000 7f6c2a800640 1 -- 172.21.15.32:0/3950198978 reap_dead start 2024-08-03T21:01:34.227 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:34.224+0000 7f6c2b200640 1 -- 172.21.15.32:0/3950198978 shutdown_connections 2024-08-03T21:01:34.228 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:34.224+0000 7f6c2b200640 1 -- 172.21.15.32:0/3950198978 >> 172.21.15.32:0/3950198978 conn(0x7f6c2c0d1e80 msgr2=0x7f6c2c070d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T21:01:34.228 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:34.224+0000 7f6c2b200640 1 -- 172.21.15.32:0/3950198978 shutdown_connections 2024-08-03T21:01:34.228 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:34.224+0000 7f6c2b200640 1 -- 172.21.15.32:0/3950198978 wait complete. 2024-08-03T21:01:34.247 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 5 2024-08-03T21:01:34.476 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:34.475+0000 7fcc30a00640 1 -- 172.21.15.32:0/4274330967 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcc2c0d0af0 msgr2=0x7fcc2c0d0f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:01:34.477 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:34.475+0000 7fcc30a00640 1 --2- 172.21.15.32:0/4274330967 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcc2c0d0af0 0x7fcc2c0d0f00 secure :-1 s=READY pgs=2302 cs=0 l=1 rev1=1 crypto rx=0x7fcc18005fd0 tx=0x7fcc1802a5f0 comp rx=0 tx=0).stop 2024-08-03T21:01:34.477 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:34.475+0000 7fcc30a00640 1 -- 172.21.15.32:0/4274330967 shutdown_connections 2024-08-03T21:01:34.477 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:34.475+0000 7fcc30a00640 1 --2- 172.21.15.32:0/4274330967 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcc2c0d0af0 0x7fcc2c0d0f00 unknown :-1 s=CLOSED pgs=2302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:01:34.477 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:34.475+0000 7fcc30a00640 1 -- 172.21.15.32:0/4274330967 >> 172.21.15.32:0/4274330967 conn(0x7fcc2c0d4120 msgr2=0x7fcc2c0d4530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T21:01:34.477 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:34.475+0000 7fcc30a00640 1 -- 172.21.15.32:0/4274330967 shutdown_connections 2024-08-03T21:01:34.477 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:34.475+0000 7fcc30a00640 1 -- 172.21.15.32:0/4274330967 wait complete. 2024-08-03T21:01:34.478 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:34.477+0000 7fcc30a00640 1 Processor -- start 2024-08-03T21:01:34.479 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:34.477+0000 7fcc30a00640 1 -- start start 2024-08-03T21:01:34.482 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:34.479+0000 7fcc30a00640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fcc2c0b93d0 0x7fcc2c0b0cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:01:34.482 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:34.479+0000 7fcc30a00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fcc2c0d0af0 0x7fcc2c15d170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:01:34.482 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:34.479+0000 7fcc30a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcc2c0d1cf0 0x7fcc2c167c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:01:34.482 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:34.479+0000 7fcc30a00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fcc2c0b4e30 con 0x7fcc2c0d1cf0 2024-08-03T21:01:34.482 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:34.479+0000 7fcc30a00640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7fcc2c0b4cb0 con 0x7fcc2c0d0af0 2024-08-03T21:01:34.483 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:34.479+0000 7fcc30a00640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7fcc2c169e00 con 0x7fcc2c0b93d0 2024-08-03T21:01:34.483 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:34.479+0000 7fcc2be00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcc2c0d1cf0 0x7fcc2c167c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:01:34.483 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:34.479+0000 7fcc2be00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcc2c0d1cf0 0x7fcc2c167c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:39630/0 (socket says 172.21.15.32:39630) 2024-08-03T21:01:34.483 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:34.479+0000 7fcc2be00640 1 -- 172.21.15.32:0/3051529327 learned_addr learned my addr 172.21.15.32:0/3051529327 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T21:01:34.483 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:34.479+0000 7fcc2aa00640 1 --2- 172.21.15.32:0/3051529327 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fcc2c0d0af0 0x7fcc2c15d170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:01:34.483 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:34.479+0000 7fcc2b400640 1 --2- 172.21.15.32:0/3051529327 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fcc2c0b93d0 0x7fcc2c0b0cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:01:34.484 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:34.480+0000 7fcc2be00640 1 -- 172.21.15.32:0/3051529327 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fcc2c0b93d0 msgr2=0x7fcc2c0b0cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:01:34.484 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:34.480+0000 7fcc2be00640 1 --2- 172.21.15.32:0/3051529327 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fcc2c0b93d0 0x7fcc2c0b0cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:01:34.484 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:34.480+0000 7fcc2be00640 1 -- 172.21.15.32:0/3051529327 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fcc2c0d0af0 msgr2=0x7fcc2c15d170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:01:34.484 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:34.480+0000 7fcc2be00640 1 --2- 172.21.15.32:0/3051529327 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fcc2c0d0af0 0x7fcc2c15d170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:01:34.484 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:34.480+0000 7fcc2be00640 1 -- 172.21.15.32:0/3051529327 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcc2c0b2ad0 con 0x7fcc2c0d1cf0 2024-08-03T21:01:34.484 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:34.480+0000 7fcc2be00640 1 --2- 172.21.15.32:0/3051529327 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcc2c0d1cf0 0x7fcc2c167c70 secure :-1 s=READY pgs=2303 cs=0 l=1 rev1=1 crypto rx=0x7fcc24002230 tx=0x7fcc2400cb90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T21:01:34.484 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:34.480+0000 7fcc23400640 1 -- 172.21.15.32:0/3051529327 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc24003b70 con 0x7fcc2c0d1cf0 2024-08-03T21:01:34.484 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:34.480+0000 7fcc23400640 1 -- 172.21.15.32:0/3051529327 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fcc24003d10 con 0x7fcc2c0d1cf0 2024-08-03T21:01:34.485 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:34.480+0000 7fcc23400640 1 -- 172.21.15.32:0/3051529327 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc24005330 con 0x7fcc2c0d1cf0 2024-08-03T21:01:34.485 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:34.480+0000 7fcc30a00640 1 -- 172.21.15.32:0/3051529327 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcc2c0b2230 con 0x7fcc2c0d1cf0 2024-08-03T21:01:34.485 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:34.480+0000 7fcc30a00640 1 -- 172.21.15.32:0/3051529327 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcc2c0b27f0 con 0x7fcc2c0d1cf0 2024-08-03T21:01:34.486 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:34.481+0000 7fcc30a00640 1 -- 172.21.15.32:0/3051529327 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcc2c0d0f00 con 0x7fcc2c0d1cf0 2024-08-03T21:01:34.486 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:34.483+0000 7fcc23400640 1 -- 172.21.15.32:0/3051529327 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7fcc240056e0 con 0x7fcc2c0d1cf0 2024-08-03T21:01:34.486 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:34.484+0000 7fcc23400640 1 --2- 172.21.15.32:0/3051529327 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fcc0c077010 0x7fcc0c079520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:01:34.486 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:34.484+0000 7fcc23400640 1 -- 172.21.15.32:0/3051529327 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(139..139 src has 1..139) ==== 5029+0+0 (secure 0 0 0) 0x7fcc240984e0 con 0x7fcc2c0d1cf0 2024-08-03T21:01:34.486 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:34.484+0000 7fcc2b400640 1 --2- 172.21.15.32:0/3051529327 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fcc0c077010 0x7fcc0c079520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:01:34.487 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:34.485+0000 7fcc2b400640 1 --2- 172.21.15.32:0/3051529327 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fcc0c077010 0x7fcc0c079520 secure :-1 s=READY pgs=921 cs=0 l=1 rev1=1 crypto rx=0x7fcc1c0016b0 tx=0x7fcc1c009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T21:01:34.489 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:34.488+0000 7fcc23400640 1 -- 172.21.15.32:0/3051529327 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7fcc240613c0 con 0x7fcc2c0d1cf0 2024-08-03T21:01:34.692 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:34.690+0000 7fcc30a00640 1 -- 172.21.15.32:0/3051529327 --> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "5", "target": ["mon-mgr", ""]}) -- 0x7fcc2c15d6b0 con 0x7fcc0c077010 2024-08-03T21:01:34.692 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:34.691+0000 7fcc23400640 1 -- 172.21.15.32:0/3051529327 <== mgr.4106 v2:172.21.15.118:6800/4205950699 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 5 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7fcc1c001c90 con 0x7fcc0c077010 2024-08-03T21:01:34.693 INFO:teuthology.orchestra.run.smithi032.stderr:instructed osd(s) 5 to deep-scrub 2024-08-03T21:01:34.695 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:34.693+0000 7fcc30a00640 1 -- 172.21.15.32:0/3051529327 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fcc0c077010 msgr2=0x7fcc0c079520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:01:34.695 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:34.693+0000 7fcc30a00640 1 --2- 172.21.15.32:0/3051529327 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fcc0c077010 0x7fcc0c079520 secure :-1 s=READY pgs=921 cs=0 l=1 rev1=1 crypto rx=0x7fcc1c0016b0 tx=0x7fcc1c009040 comp rx=0 tx=0).stop 2024-08-03T21:01:34.695 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:34.693+0000 7fcc30a00640 1 -- 172.21.15.32:0/3051529327 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcc2c0d1cf0 msgr2=0x7fcc2c167c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:01:34.696 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:34.693+0000 7fcc30a00640 1 --2- 172.21.15.32:0/3051529327 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcc2c0d1cf0 0x7fcc2c167c70 secure :-1 s=READY pgs=2303 cs=0 l=1 rev1=1 crypto rx=0x7fcc24002230 tx=0x7fcc2400cb90 comp rx=0 tx=0).stop 2024-08-03T21:01:34.696 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:34.693+0000 7fcc30a00640 1 -- 172.21.15.32:0/3051529327 shutdown_connections 2024-08-03T21:01:34.696 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:34.693+0000 7fcc30a00640 1 --2- 172.21.15.32:0/3051529327 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fcc0c077010 0x7fcc0c079520 unknown :-1 s=CLOSED pgs=921 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:01:34.696 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:34.693+0000 7fcc30a00640 1 --2- 172.21.15.32:0/3051529327 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fcc2c0d0af0 0x7fcc2c15d170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:01:34.696 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:34.693+0000 7fcc30a00640 1 --2- 172.21.15.32:0/3051529327 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcc2c0d1cf0 0x7fcc2c167c70 unknown :-1 s=CLOSED pgs=2303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:01:34.696 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:34.693+0000 7fcc30a00640 1 --2- 172.21.15.32:0/3051529327 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fcc2c0b93d0 0x7fcc2c0b0cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:01:34.696 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:34.693+0000 7fcc30a00640 1 -- 172.21.15.32:0/3051529327 >> 172.21.15.32:0/3051529327 conn(0x7fcc2c0d4120 msgr2=0x7fcc2c0b7960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T21:01:34.697 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:34.694+0000 7fcc30a00640 1 -- 172.21.15.32:0/3051529327 shutdown_connections 2024-08-03T21:01:34.697 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:34.694+0000 7fcc30a00640 1 -- 172.21.15.32:0/3051529327 wait complete. 2024-08-03T21:01:34.719 INFO:tasks.ceph:Scrubbing osd.6 2024-08-03T21:01:34.719 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 config set osd_debug_deep_scrub_sleep 0 2024-08-03T21:01:34.952 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:34.950+0000 7fafbaa00640 1 -- 172.21.15.32:0/3806430070 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fafb4145c50 msgr2=0x7fafb406e950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:01:34.952 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:34.950+0000 7fafbaa00640 1 --2- 172.21.15.32:0/3806430070 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fafb4145c50 0x7fafb406e950 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fafa4009860 tx=0x7fafa402a5a0 comp rx=0 tx=0).stop 2024-08-03T21:01:34.953 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:34.950+0000 7fafbaa00640 1 -- 172.21.15.32:0/3806430070 shutdown_connections 2024-08-03T21:01:34.953 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:34.950+0000 7fafbaa00640 1 --2- 172.21.15.32:0/3806430070 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fafb4145c50 0x7fafb406e950 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:01:34.953 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:34.950+0000 7fafbaa00640 1 -- 172.21.15.32:0/3806430070 >> 172.21.15.32:0/3806430070 conn(0x7fafb40b1bd0 msgr2=0x7fafb40b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T21:01:34.953 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:34.950+0000 7fafbaa00640 1 -- 172.21.15.32:0/3806430070 shutdown_connections 2024-08-03T21:01:34.953 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:34.950+0000 7fafbaa00640 1 -- 172.21.15.32:0/3806430070 wait complete. 2024-08-03T21:01:34.954 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:34.953+0000 7fafbaa00640 1 Processor -- start 2024-08-03T21:01:34.954 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:34.953+0000 7fafbaa00640 1 -- start start 2024-08-03T21:01:34.956 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:34.954+0000 7fafbaa00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fafb40cccf0 0x7fafb413b5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:01:34.957 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:34.954+0000 7fafbaa00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fafb40ced30 0x7fafb4073a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:01:34.957 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:34.954+0000 7fafbaa00640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fafb4145c50 0x7fafb4073fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:01:34.957 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:34.954+0000 7fafbaa00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fafb40b4230 con 0x7fafb40cccf0 2024-08-03T21:01:34.957 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:34.954+0000 7fafbaa00640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7fafb40b40b0 con 0x7fafb40ced30 2024-08-03T21:01:34.957 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:34.954+0000 7fafbaa00640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7fafb40b43b0 con 0x7fafb4145c50 2024-08-03T21:01:34.957 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:34.954+0000 7fafb3e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fafb40cccf0 0x7fafb413b5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:01:34.957 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:34.954+0000 7fafb3e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fafb40cccf0 0x7fafb413b5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:39646/0 (socket says 172.21.15.32:39646) 2024-08-03T21:01:34.958 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:34.954+0000 7fafb3e00640 1 -- 172.21.15.32:0/2503913596 learned_addr learned my addr 172.21.15.32:0/2503913596 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T21:01:34.958 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:34.955+0000 7fafab400640 1 --2- 172.21.15.32:0/2503913596 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fafb40ced30 0x7fafb4073a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:01:34.958 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:34.955+0000 7fafb8a00640 1 --2- 172.21.15.32:0/2503913596 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fafb4145c50 0x7fafb4073fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:01:34.958 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:34.955+0000 7fafb3e00640 1 -- 172.21.15.32:0/2503913596 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fafb4145c50 msgr2=0x7fafb4073fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:01:34.958 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:34.955+0000 7fafb3e00640 1 --2- 172.21.15.32:0/2503913596 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fafb4145c50 0x7fafb4073fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:01:34.958 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:34.955+0000 7fafb3e00640 1 -- 172.21.15.32:0/2503913596 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fafb40ced30 msgr2=0x7fafb4073a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:01:34.958 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:34.955+0000 7fafb3e00640 1 --2- 172.21.15.32:0/2503913596 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fafb40ced30 0x7fafb4073a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:01:34.958 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:34.955+0000 7fafb3e00640 1 -- 172.21.15.32:0/2503913596 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fafb41f54d0 con 0x7fafb40cccf0 2024-08-03T21:01:34.959 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:34.955+0000 7fafab400640 1 --2- 172.21.15.32:0/2503913596 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fafb40ced30 0x7fafb4073a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-03T21:01:34.959 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:34.955+0000 7fafb3e00640 1 --2- 172.21.15.32:0/2503913596 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fafb40cccf0 0x7fafb413b5f0 secure :-1 s=READY pgs=2305 cs=0 l=1 rev1=1 crypto rx=0x7fafa4009980 tx=0x7fafa4033ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T21:01:34.960 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:34.955+0000 7fafb0c00640 1 -- 172.21.15.32:0/2503913596 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fafa4031b50 con 0x7fafb40cccf0 2024-08-03T21:01:34.960 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:34.956+0000 7fafb0c00640 1 -- 172.21.15.32:0/2503913596 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fafa4031de0 con 0x7fafb40cccf0 2024-08-03T21:01:34.960 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:34.956+0000 7fafb0c00640 1 -- 172.21.15.32:0/2503913596 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fafa4033330 con 0x7fafb40cccf0 2024-08-03T21:01:34.960 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:34.956+0000 7fafbaa00640 1 -- 172.21.15.32:0/2503913596 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fafb41e5790 con 0x7fafb40cccf0 2024-08-03T21:01:34.960 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:34.956+0000 7fafbaa00640 1 -- 172.21.15.32:0/2503913596 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fafb41e5a30 con 0x7fafb40cccf0 2024-08-03T21:01:34.960 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:34.957+0000 7fafbaa00640 1 -- 172.21.15.32:0/2503913596 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fafb406e950 con 0x7fafb40cccf0 2024-08-03T21:01:34.960 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:34.958+0000 7fafb0c00640 1 -- 172.21.15.32:0/2503913596 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7fafa40334d0 con 0x7fafb40cccf0 2024-08-03T21:01:34.962 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:34.959+0000 7fafb0c00640 1 --2- 172.21.15.32:0/2503913596 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7faf90077010 0x7faf90079520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:01:34.962 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:34.959+0000 7fafb0c00640 1 -- 172.21.15.32:0/2503913596 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(139..139 src has 1..139) ==== 5029+0+0 (secure 0 0 0) 0x7fafa40b97e0 con 0x7fafb40cccf0 2024-08-03T21:01:34.962 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:34.959+0000 7fafb0c00640 1 --2- 172.21.15.32:0/2503913596 >> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] conn(0x7faf9007cfc0 0x7faf9007f410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:01:34.962 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:34.959+0000 7fafb0c00640 1 -- 172.21.15.32:0/2503913596 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fafa4032e50 con 0x7faf9007cfc0 2024-08-03T21:01:34.962 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:34.959+0000 7fafb0c00640 1 -- 172.21.15.32:0/2503913596 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_get_version_reply(handle=1 version=139) ==== 24+0+0 (secure 0 0 0) 0x7fafa40c0050 con 0x7fafb40cccf0 2024-08-03T21:01:34.962 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:34.959+0000 7fafab400640 1 --2- 172.21.15.32:0/2503913596 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7faf90077010 0x7faf90079520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:01:34.962 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:34.960+0000 7fafb8a00640 1 --2- 172.21.15.32:0/2503913596 >> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] conn(0x7faf9007cfc0 0x7faf9007f410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:01:34.963 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:34.960+0000 7fafab400640 1 --2- 172.21.15.32:0/2503913596 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7faf90077010 0x7faf90079520 secure :-1 s=READY pgs=922 cs=0 l=1 rev1=1 crypto rx=0x7fafb40cf770 tx=0x7faf9c009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T21:01:34.963 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:34.960+0000 7fafb8a00640 1 --2- 172.21.15.32:0/2503913596 >> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] conn(0x7faf9007cfc0 0x7faf9007f410 crc :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T21:01:34.963 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:34.961+0000 7fafb0c00640 1 -- 172.21.15.32:0/2503913596 <== osd.6 v2:172.21.15.118:6825/801947386 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fafa4032e50 con 0x7faf9007cfc0 2024-08-03T21:01:34.981 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:34.980+0000 7fafbaa00640 1 -- 172.21.15.32:0/2503913596 --> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7fafb41e7870 con 0x7faf9007cfc0 2024-08-03T21:01:34.982 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:34.981+0000 7fafb0c00640 1 -- 172.21.15.32:0/2503913596 <== osd.6 v2:172.21.15.118:6825/801947386 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7fafb41e7870 con 0x7faf9007cfc0 2024-08-03T21:01:34.982 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-08-03T21:01:34.983 INFO:teuthology.orchestra.run.smithi032.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-08-03T21:01:34.983 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-08-03T21:01:34.984 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:34.981+0000 7fafbaa00640 1 -- 172.21.15.32:0/2503913596 >> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] conn(0x7faf9007cfc0 msgr2=0x7faf9007f410 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:01:34.984 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:34.981+0000 7fafbaa00640 1 --2- 172.21.15.32:0/2503913596 >> [v2:172.21.15.118:6825/801947386,v1:172.21.15.118:6827/801947386] conn(0x7faf9007cfc0 0x7faf9007f410 crc :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:01:34.984 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:34.981+0000 7fafbaa00640 1 -- 172.21.15.32:0/2503913596 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7faf90077010 msgr2=0x7faf90079520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:01:34.984 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:34.981+0000 7fafbaa00640 1 --2- 172.21.15.32:0/2503913596 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7faf90077010 0x7faf90079520 secure :-1 s=READY pgs=922 cs=0 l=1 rev1=1 crypto rx=0x7fafb40cf770 tx=0x7faf9c009040 comp rx=0 tx=0).stop 2024-08-03T21:01:34.985 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:34.981+0000 7fafbaa00640 1 -- 172.21.15.32:0/2503913596 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fafb40cccf0 msgr2=0x7fafb413b5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:01:34.985 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:34.981+0000 7fafbaa00640 1 --2- 172.21.15.32:0/2503913596 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fafb40cccf0 0x7fafb413b5f0 secure :-1 s=READY pgs=2305 cs=0 l=1 rev1=1 crypto rx=0x7fafa4009980 tx=0x7fafa4033ba0 comp rx=0 tx=0).stop 2024-08-03T21:01:34.985 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:34.981+0000 7fafb8a00640 1 -- 172.21.15.32:0/2503913596 reap_dead start 2024-08-03T21:01:34.985 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:34.982+0000 7fafbaa00640 1 -- 172.21.15.32:0/2503913596 shutdown_connections 2024-08-03T21:01:34.985 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:34.982+0000 7fafbaa00640 1 -- 172.21.15.32:0/2503913596 >> 172.21.15.32:0/2503913596 conn(0x7fafb40b1bd0 msgr2=0x7fafb40cff60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T21:01:34.985 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:34.982+0000 7fafbaa00640 1 -- 172.21.15.32:0/2503913596 shutdown_connections 2024-08-03T21:01:34.985 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:34.982+0000 7fafbaa00640 1 -- 172.21.15.32:0/2503913596 wait complete. 2024-08-03T21:01:35.008 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 6 2024-08-03T21:01:35.239 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:35.236+0000 7fa2ab800640 1 -- 172.21.15.32:0/2355442090 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa28c004800 msgr2=0x7fa28c004c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:01:35.239 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:35.236+0000 7fa2ab800640 1 --2- 172.21.15.32:0/2355442090 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa28c004800 0x7fa28c004c10 secure :-1 s=READY pgs=2307 cs=0 l=1 rev1=1 crypto rx=0x7fa2a40cc490 tx=0x7fa28c005830 comp rx=0 tx=0).stop 2024-08-03T21:01:35.239 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:35.236+0000 7fa2a9800640 1 -- 172.21.15.32:0/2355442090 reap_dead start 2024-08-03T21:01:35.239 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:35.237+0000 7fa2ab800640 1 -- 172.21.15.32:0/2355442090 shutdown_connections 2024-08-03T21:01:35.239 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:35.237+0000 7fa2ab800640 1 -- 172.21.15.32:0/2355442090 >> 172.21.15.32:0/2355442090 conn(0x7fa2a40b1c40 msgr2=0x7fa2a40b2050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T21:01:35.239 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:35.237+0000 7fa2ab800640 1 -- 172.21.15.32:0/2355442090 shutdown_connections 2024-08-03T21:01:35.240 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:35.237+0000 7fa2ab800640 1 -- 172.21.15.32:0/2355442090 wait complete. 2024-08-03T21:01:35.240 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:35.239+0000 7fa2ab800640 1 Processor -- start 2024-08-03T21:01:35.240 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:35.239+0000 7fa2ab800640 1 -- start start 2024-08-03T21:01:35.243 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:35.240+0000 7fa2ab800640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fa28c004800 0x7fa2a41f1da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:01:35.244 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:35.240+0000 7fa2ab800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa2a4157a80 0x7fa2a41f22e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:01:35.244 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:35.241+0000 7fa2ab800640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa2a40aeba0 0x7fa2a40aefb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:01:35.244 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:35.241+0000 7fa2ab800640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fa2a406fac0 con 0x7fa2a4157a80 2024-08-03T21:01:35.244 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:35.241+0000 7fa2ab800640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7fa2a406f940 con 0x7fa2a40aeba0 2024-08-03T21:01:35.244 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:35.241+0000 7fa2ab800640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7fa2a406fc40 con 0x7fa28c004800 2024-08-03T21:01:35.244 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:35.241+0000 7fa2a3e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa2a4157a80 0x7fa2a41f22e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:01:35.244 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:35.241+0000 7fa2a3e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa2a4157a80 0x7fa2a41f22e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:39670/0 (socket says 172.21.15.32:39670) 2024-08-03T21:01:35.244 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:35.241+0000 7fa2a3e00640 1 -- 172.21.15.32:0/3801345060 learned_addr learned my addr 172.21.15.32:0/3801345060 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T21:01:35.245 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:35.241+0000 7fa2a9800640 1 --2- 172.21.15.32:0/3801345060 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa2a40aeba0 0x7fa2a40aefb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:01:35.245 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:35.241+0000 7fa2a8e00640 1 --2- 172.21.15.32:0/3801345060 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fa28c004800 0x7fa2a41f1da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:01:35.245 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:35.241+0000 7fa2a3e00640 1 -- 172.21.15.32:0/3801345060 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fa28c004800 msgr2=0x7fa2a41f1da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:01:35.245 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:35.241+0000 7fa2a3e00640 1 --2- 172.21.15.32:0/3801345060 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fa28c004800 0x7fa2a41f1da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:01:35.245 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:35.241+0000 7fa2a3e00640 1 -- 172.21.15.32:0/3801345060 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa2a40aeba0 msgr2=0x7fa2a40aefb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:01:35.245 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:35.241+0000 7fa2a3e00640 1 --2- 172.21.15.32:0/3801345060 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa2a40aeba0 0x7fa2a40aefb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:01:35.245 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:35.241+0000 7fa2a3e00640 1 -- 172.21.15.32:0/3801345060 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa2a4158120 con 0x7fa2a4157a80 2024-08-03T21:01:35.245 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:35.241+0000 7fa2a9800640 1 --2- 172.21.15.32:0/3801345060 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa2a40aeba0 0x7fa2a40aefb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-03T21:01:35.245 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:35.242+0000 7fa2a3e00640 1 --2- 172.21.15.32:0/3801345060 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa2a4157a80 0x7fa2a41f22e0 secure :-1 s=READY pgs=2308 cs=0 l=1 rev1=1 crypto rx=0x7fa29002aa40 tx=0x7fa2900060e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T21:01:35.246 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:35.242+0000 7fa2a0c00640 1 -- 172.21.15.32:0/3801345060 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa290037040 con 0x7fa2a4157a80 2024-08-03T21:01:35.246 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:35.242+0000 7fa2a0c00640 1 -- 172.21.15.32:0/3801345060 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fa290041070 con 0x7fa2a4157a80 2024-08-03T21:01:35.246 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:35.242+0000 7fa2a0c00640 1 -- 172.21.15.32:0/3801345060 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa290003850 con 0x7fa2a4157a80 2024-08-03T21:01:35.246 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:35.242+0000 7fa2ab800640 1 -- 172.21.15.32:0/3801345060 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa2a4158400 con 0x7fa2a4157a80 2024-08-03T21:01:35.246 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:35.242+0000 7fa2ab800640 1 -- 172.21.15.32:0/3801345060 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa2a41589c0 con 0x7fa2a4157a80 2024-08-03T21:01:35.247 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:35.243+0000 7fa2ab800640 1 -- 172.21.15.32:0/3801345060 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa2a40701d0 con 0x7fa2a4157a80 2024-08-03T21:01:35.247 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:35.244+0000 7fa2a0c00640 1 -- 172.21.15.32:0/3801345060 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7fa2900039f0 con 0x7fa2a4157a80 2024-08-03T21:01:35.247 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:35.245+0000 7fa2a0c00640 1 --2- 172.21.15.32:0/3801345060 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fa284077010 0x7fa284079520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:01:35.247 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:35.245+0000 7fa2a0c00640 1 -- 172.21.15.32:0/3801345060 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(139..139 src has 1..139) ==== 5029+0+0 (secure 0 0 0) 0x7fa2900b8b70 con 0x7fa2a4157a80 2024-08-03T21:01:35.248 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:35.246+0000 7fa2a8e00640 1 --2- 172.21.15.32:0/3801345060 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fa284077010 0x7fa284079520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:01:35.248 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:35.247+0000 7fa2a8e00640 1 --2- 172.21.15.32:0/3801345060 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fa284077010 0x7fa284079520 secure :-1 s=READY pgs=923 cs=0 l=1 rev1=1 crypto rx=0x7fa298009b20 tx=0x7fa298005f90 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T21:01:35.251 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:35.250+0000 7fa2a0c00640 1 -- 172.21.15.32:0/3801345060 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7fa290081f70 con 0x7fa2a4157a80 2024-08-03T21:01:35.458 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:35.457+0000 7fa2ab800640 1 -- 172.21.15.32:0/3801345060 --> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "6", "target": ["mon-mgr", ""]}) -- 0x7fa2a4157eb0 con 0x7fa284077010 2024-08-03T21:01:35.459 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:35.457+0000 7fa2a0c00640 1 -- 172.21.15.32:0/3801345060 <== mgr.4106 v2:172.21.15.118:6800/4205950699 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 6 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7fa2a4157eb0 con 0x7fa284077010 2024-08-03T21:01:35.459 INFO:teuthology.orchestra.run.smithi032.stderr:instructed osd(s) 6 to deep-scrub 2024-08-03T21:01:35.461 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:35.459+0000 7fa2ab800640 1 -- 172.21.15.32:0/3801345060 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fa284077010 msgr2=0x7fa284079520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:01:35.461 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:35.459+0000 7fa2ab800640 1 --2- 172.21.15.32:0/3801345060 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fa284077010 0x7fa284079520 secure :-1 s=READY pgs=923 cs=0 l=1 rev1=1 crypto rx=0x7fa298009b20 tx=0x7fa298005f90 comp rx=0 tx=0).stop 2024-08-03T21:01:35.462 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:35.459+0000 7fa2ab800640 1 -- 172.21.15.32:0/3801345060 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa2a4157a80 msgr2=0x7fa2a41f22e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:01:35.462 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:35.459+0000 7fa2ab800640 1 --2- 172.21.15.32:0/3801345060 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa2a4157a80 0x7fa2a41f22e0 secure :-1 s=READY pgs=2308 cs=0 l=1 rev1=1 crypto rx=0x7fa29002aa40 tx=0x7fa2900060e0 comp rx=0 tx=0).stop 2024-08-03T21:01:35.462 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:35.460+0000 7fa2ab800640 1 -- 172.21.15.32:0/3801345060 shutdown_connections 2024-08-03T21:01:35.462 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:35.460+0000 7fa2ab800640 1 --2- 172.21.15.32:0/3801345060 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fa284077010 0x7fa284079520 unknown :-1 s=CLOSED pgs=923 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:01:35.462 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:35.460+0000 7fa2ab800640 1 --2- 172.21.15.32:0/3801345060 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa2a40aeba0 0x7fa2a40aefb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:01:35.462 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:35.460+0000 7fa2ab800640 1 --2- 172.21.15.32:0/3801345060 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa2a4157a80 0x7fa2a41f22e0 unknown :-1 s=CLOSED pgs=2308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:01:35.462 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:35.460+0000 7fa2ab800640 1 --2- 172.21.15.32:0/3801345060 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fa28c004800 0x7fa2a41f1da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:01:35.462 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:35.460+0000 7fa2ab800640 1 -- 172.21.15.32:0/3801345060 >> 172.21.15.32:0/3801345060 conn(0x7fa2a40b1c40 msgr2=0x7fa2a40b1270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T21:01:35.463 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:35.460+0000 7fa2ab800640 1 -- 172.21.15.32:0/3801345060 shutdown_connections 2024-08-03T21:01:35.463 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:35.460+0000 7fa2ab800640 1 -- 172.21.15.32:0/3801345060 wait complete. 2024-08-03T21:01:35.483 INFO:tasks.ceph:Scrubbing osd.7 2024-08-03T21:01:35.483 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 config set osd_debug_deep_scrub_sleep 0 2024-08-03T21:01:35.717 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:35.715+0000 7f9fbbe00640 1 -- 172.21.15.32:0/713161348 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9fa0004d40 msgr2=0x7f9fa00051f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:01:35.717 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:35.715+0000 7f9fbbe00640 1 --2- 172.21.15.32:0/713161348 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9fa0004d40 0x7f9fa00051f0 secure :-1 s=READY pgs=2310 cs=0 l=1 rev1=1 crypto rx=0x7f9fb4001d90 tx=0x7f9fa0005830 comp rx=0 tx=0).stop 2024-08-03T21:01:35.717 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:35.715+0000 7f9fbb400640 1 -- 172.21.15.32:0/713161348 reap_dead start 2024-08-03T21:01:35.718 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:35.715+0000 7f9fbbe00640 1 -- 172.21.15.32:0/713161348 shutdown_connections 2024-08-03T21:01:35.718 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:35.715+0000 7f9fbbe00640 1 -- 172.21.15.32:0/713161348 >> 172.21.15.32:0/713161348 conn(0x7f9fbc0cfd80 msgr2=0x7f9fbc0d0190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T21:01:35.718 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:35.715+0000 7f9fbbe00640 1 -- 172.21.15.32:0/713161348 shutdown_connections 2024-08-03T21:01:35.718 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:35.715+0000 7f9fbbe00640 1 -- 172.21.15.32:0/713161348 wait complete. 2024-08-03T21:01:35.719 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:35.718+0000 7f9fbbe00640 1 Processor -- start 2024-08-03T21:01:35.719 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:35.718+0000 7f9fbbe00640 1 -- start start 2024-08-03T21:01:35.721 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:35.719+0000 7f9fbbe00640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f9fa0004d40 0x7f9fbc0aed80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:01:35.722 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:35.719+0000 7f9fbbe00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f9fbc156530 0x7f9fbc148ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:01:35.722 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:35.719+0000 7f9fbbe00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9fbc149010 0x7f9fbc149420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:01:35.722 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:35.719+0000 7f9fbbe00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f9fbc1601a0 con 0x7f9fbc149010 2024-08-03T21:01:35.722 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:35.719+0000 7f9fbbe00640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f9fbc160020 con 0x7f9fbc156530 2024-08-03T21:01:35.722 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:35.719+0000 7f9fbbe00640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7f9fbc06ed40 con 0x7f9fa0004d40 2024-08-03T21:01:35.722 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:35.720+0000 7f9fbb400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9fbc149010 0x7f9fbc149420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:01:35.722 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:35.720+0000 7f9fbaa00640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f9fa0004d40 0x7f9fbc0aed80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:01:35.722 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:35.720+0000 7f9fbb400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9fbc149010 0x7f9fbc149420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:39686/0 (socket says 172.21.15.32:39686) 2024-08-03T21:01:35.723 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:35.720+0000 7f9fbb400640 1 -- 172.21.15.32:0/1859223170 learned_addr learned my addr 172.21.15.32:0/1859223170 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T21:01:35.723 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:35.720+0000 7f9fb3e00640 1 --2- 172.21.15.32:0/1859223170 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f9fbc156530 0x7f9fbc148ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:01:35.723 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:35.720+0000 7f9fbb400640 1 -- 172.21.15.32:0/1859223170 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f9fa0004d40 msgr2=0x7f9fbc0aed80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:01:35.723 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:35.720+0000 7f9fbb400640 1 --2- 172.21.15.32:0/1859223170 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f9fa0004d40 0x7f9fbc0aed80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:01:35.723 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:35.720+0000 7f9fbb400640 1 -- 172.21.15.32:0/1859223170 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f9fbc156530 msgr2=0x7f9fbc148ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:01:35.724 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:35.720+0000 7f9fbb400640 1 --2- 172.21.15.32:0/1859223170 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f9fbc156530 0x7f9fbc148ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:01:35.724 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:35.720+0000 7f9fbb400640 1 -- 172.21.15.32:0/1859223170 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9fbc15b5b0 con 0x7f9fbc149010 2024-08-03T21:01:35.725 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:35.720+0000 7f9fbb400640 1 --2- 172.21.15.32:0/1859223170 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9fbc149010 0x7f9fbc149420 secure :-1 s=READY pgs=2311 cs=0 l=1 rev1=1 crypto rx=0x7f9fb402ac80 tx=0x7f9fb4001b10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T21:01:35.725 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:35.721+0000 7f9fb2a00640 1 -- 172.21.15.32:0/1859223170 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9fb40060b0 con 0x7f9fbc149010 2024-08-03T21:01:35.725 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:35.721+0000 7f9fb2a00640 1 -- 172.21.15.32:0/1859223170 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f9fb4006250 con 0x7f9fbc149010 2024-08-03T21:01:35.725 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:35.721+0000 7f9fbbe00640 1 -- 172.21.15.32:0/1859223170 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9fbc15b890 con 0x7f9fbc149010 2024-08-03T21:01:35.725 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:35.721+0000 7f9fb2a00640 1 -- 172.21.15.32:0/1859223170 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9fb40066a0 con 0x7f9fbc149010 2024-08-03T21:01:35.725 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:35.721+0000 7f9fbbe00640 1 -- 172.21.15.32:0/1859223170 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9fbc1e7320 con 0x7f9fbc149010 2024-08-03T21:01:35.725 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:35.722+0000 7f9fbbe00640 1 -- 172.21.15.32:0/1859223170 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f9fbc1e5e80 con 0x7f9fbc149010 2024-08-03T21:01:35.725 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:35.723+0000 7f9fb2a00640 1 -- 172.21.15.32:0/1859223170 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f9fb4033480 con 0x7f9fbc149010 2024-08-03T21:01:35.726 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:35.724+0000 7f9fb2a00640 1 --2- 172.21.15.32:0/1859223170 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f9f98077010 0x7f9f98079520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:01:35.727 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:35.724+0000 7f9fb2a00640 1 -- 172.21.15.32:0/1859223170 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(139..139 src has 1..139) ==== 5029+0+0 (secure 0 0 0) 0x7f9fb40b83b0 con 0x7f9fbc149010 2024-08-03T21:01:35.727 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:35.724+0000 7f9fb2a00640 1 --2- 172.21.15.32:0/1859223170 >> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] conn(0x7f9f9807cfc0 0x7f9f9807f410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:01:35.727 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:35.724+0000 7f9fb2a00640 1 -- 172.21.15.32:0/1859223170 --> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f9fb4003b90 con 0x7f9f9807cfc0 2024-08-03T21:01:35.727 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:35.725+0000 7f9fb2a00640 1 -- 172.21.15.32:0/1859223170 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_get_version_reply(handle=1 version=139) ==== 24+0+0 (secure 0 0 0) 0x7f9fb40bf050 con 0x7f9fbc149010 2024-08-03T21:01:35.727 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:35.725+0000 7f9fbaa00640 1 --2- 172.21.15.32:0/1859223170 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f9f98077010 0x7f9f98079520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:01:35.727 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:35.725+0000 7f9fb3e00640 1 --2- 172.21.15.32:0/1859223170 >> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] conn(0x7f9f9807cfc0 0x7f9f9807f410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:01:35.728 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:35.725+0000 7f9fbaa00640 1 --2- 172.21.15.32:0/1859223170 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f9f98077010 0x7f9f98079520 secure :-1 s=READY pgs=924 cs=0 l=1 rev1=1 crypto rx=0x7f9fac006fd0 tx=0x7f9fac006970 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T21:01:35.728 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:35.726+0000 7f9fb3e00640 1 --2- 172.21.15.32:0/1859223170 >> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] conn(0x7f9f9807cfc0 0x7f9f9807f410 crc :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T21:01:35.728 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:35.727+0000 7f9fb2a00640 1 -- 172.21.15.32:0/1859223170 <== osd.7 v2:172.21.15.118:6818/653000048 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f9fb4003b90 con 0x7f9f9807cfc0 2024-08-03T21:01:35.742 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:35.740+0000 7f9fbbe00640 1 -- 172.21.15.32:0/1859223170 --> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7f9fbc0d0ed0 con 0x7f9f9807cfc0 2024-08-03T21:01:35.743 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:35.741+0000 7f9fb2a00640 1 -- 172.21.15.32:0/1859223170 <== osd.7 v2:172.21.15.118:6818/653000048 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7f9fbc0d0ed0 con 0x7f9f9807cfc0 2024-08-03T21:01:35.743 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-08-03T21:01:35.744 INFO:teuthology.orchestra.run.smithi032.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-08-03T21:01:35.744 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-08-03T21:01:35.745 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:35.742+0000 7f9fbbe00640 1 -- 172.21.15.32:0/1859223170 >> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] conn(0x7f9f9807cfc0 msgr2=0x7f9f9807f410 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:01:35.745 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:35.742+0000 7f9fbbe00640 1 --2- 172.21.15.32:0/1859223170 >> [v2:172.21.15.118:6818/653000048,v1:172.21.15.118:6819/653000048] conn(0x7f9f9807cfc0 0x7f9f9807f410 crc :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:01:35.745 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:35.742+0000 7f9fbbe00640 1 -- 172.21.15.32:0/1859223170 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f9f98077010 msgr2=0x7f9f98079520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:01:35.745 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:35.742+0000 7f9fbbe00640 1 --2- 172.21.15.32:0/1859223170 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f9f98077010 0x7f9f98079520 secure :-1 s=READY pgs=924 cs=0 l=1 rev1=1 crypto rx=0x7f9fac006fd0 tx=0x7f9fac006970 comp rx=0 tx=0).stop 2024-08-03T21:01:35.745 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:35.742+0000 7f9fbbe00640 1 -- 172.21.15.32:0/1859223170 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9fbc149010 msgr2=0x7f9fbc149420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:01:35.745 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:35.742+0000 7f9fbbe00640 1 --2- 172.21.15.32:0/1859223170 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9fbc149010 0x7f9fbc149420 secure :-1 s=READY pgs=2311 cs=0 l=1 rev1=1 crypto rx=0x7f9fb402ac80 tx=0x7f9fb4001b10 comp rx=0 tx=0).stop 2024-08-03T21:01:35.745 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:35.742+0000 7f9fbb400640 1 -- 172.21.15.32:0/1859223170 reap_dead start 2024-08-03T21:01:35.746 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:35.742+0000 7f9fbbe00640 1 -- 172.21.15.32:0/1859223170 shutdown_connections 2024-08-03T21:01:35.746 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:35.742+0000 7f9fbbe00640 1 -- 172.21.15.32:0/1859223170 >> 172.21.15.32:0/1859223170 conn(0x7f9fbc0cfd80 msgr2=0x7f9fbc0af740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T21:01:35.746 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:35.743+0000 7f9fbbe00640 1 -- 172.21.15.32:0/1859223170 shutdown_connections 2024-08-03T21:01:35.746 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:35.743+0000 7f9fbbe00640 1 -- 172.21.15.32:0/1859223170 wait complete. 2024-08-03T21:01:35.765 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 7 2024-08-03T21:01:35.994 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:35.992+0000 7fe30a400640 1 -- 172.21.15.32:0/3722907763 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe3040b6c10 msgr2=0x7fe3040b2720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:01:35.994 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:35.992+0000 7fe30a400640 1 --2- 172.21.15.32:0/3722907763 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe3040b6c10 0x7fe3040b2720 secure :-1 s=READY pgs=2312 cs=0 l=1 rev1=1 crypto rx=0x7fe3040cc840 tx=0x7fe2fc02aa70 comp rx=0 tx=0).stop 2024-08-03T21:01:35.994 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:35.992+0000 7fe30a400640 1 -- 172.21.15.32:0/3722907763 shutdown_connections 2024-08-03T21:01:35.994 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:35.992+0000 7fe30a400640 1 --2- 172.21.15.32:0/3722907763 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fe30409b160 0x7fe3040b21e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:01:35.994 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:35.992+0000 7fe30a400640 1 --2- 172.21.15.32:0/3722907763 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe3040b6c10 0x7fe3040b2720 unknown :-1 s=CLOSED pgs=2312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:01:35.995 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:35.992+0000 7fe30a400640 1 -- 172.21.15.32:0/3722907763 >> 172.21.15.32:0/3722907763 conn(0x7fe3040d2160 msgr2=0x7fe3040d2570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T21:01:35.995 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:35.992+0000 7fe30a400640 1 -- 172.21.15.32:0/3722907763 shutdown_connections 2024-08-03T21:01:35.995 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:35.992+0000 7fe30a400640 1 -- 172.21.15.32:0/3722907763 wait complete. 2024-08-03T21:01:35.996 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:35.994+0000 7fe30a400640 1 Processor -- start 2024-08-03T21:01:35.996 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:35.994+0000 7fe30a400640 1 -- start start 2024-08-03T21:01:35.998 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:35.996+0000 7fe30a400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe3040b6c10 0x7fe30415ddd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:01:35.998 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:35.996+0000 7fe30a400640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fe3040cf540 0x7fe3041f5ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:01:35.998 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:35.996+0000 7fe30a400640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fe30415e540 0x7fe30409e250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:01:35.999 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:35.996+0000 7fe30a400640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fe3040b2ff0 con 0x7fe3040b6c10 2024-08-03T21:01:35.999 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:35.996+0000 7fe30a400640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7fe3040b2e70 con 0x7fe3040cf540 2024-08-03T21:01:35.999 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:35.996+0000 7fe30a400640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7fe3040b3170 con 0x7fe30415e540 2024-08-03T21:01:35.999 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:35.996+0000 7fe303400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe3040b6c10 0x7fe30415ddd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:01:35.999 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:35.996+0000 7fe303400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe3040b6c10 0x7fe30415ddd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:39698/0 (socket says 172.21.15.32:39698) 2024-08-03T21:01:35.999 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:35.996+0000 7fe303400640 1 -- 172.21.15.32:0/1448091705 learned_addr learned my addr 172.21.15.32:0/1448091705 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T21:01:35.999 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:35.996+0000 7fe2faa00640 1 --2- 172.21.15.32:0/1448091705 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fe3040cf540 0x7fe3041f5ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:01:35.999 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:35.996+0000 7fe303e00640 1 --2- 172.21.15.32:0/1448091705 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fe30415e540 0x7fe30409e250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:01:35.999 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:35.997+0000 7fe303400640 1 -- 172.21.15.32:0/1448091705 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fe30415e540 msgr2=0x7fe30409e250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:01:36.000 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:35.997+0000 7fe303400640 1 --2- 172.21.15.32:0/1448091705 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fe30415e540 0x7fe30409e250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:01:36.000 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:35.997+0000 7fe303400640 1 -- 172.21.15.32:0/1448091705 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fe3040cf540 msgr2=0x7fe3041f5ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:01:36.000 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:35.997+0000 7fe303400640 1 --2- 172.21.15.32:0/1448091705 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fe3040cf540 0x7fe3041f5ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:01:36.000 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:35.997+0000 7fe303400640 1 -- 172.21.15.32:0/1448091705 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe30415e310 con 0x7fe3040b6c10 2024-08-03T21:01:36.000 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:35.997+0000 7fe303400640 1 --2- 172.21.15.32:0/1448091705 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe3040b6c10 0x7fe30415ddd0 secure :-1 s=READY pgs=2313 cs=0 l=1 rev1=1 crypto rx=0x7fe2f400b3c0 tx=0x7fe2f400b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T21:01:36.001 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:35.997+0000 7fe2fbe00640 1 -- 172.21.15.32:0/1448091705 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2f4011830 con 0x7fe3040b6c10 2024-08-03T21:01:36.001 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:35.997+0000 7fe2fbe00640 1 -- 172.21.15.32:0/1448091705 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fe2f4016070 con 0x7fe3040b6c10 2024-08-03T21:01:36.001 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:35.997+0000 7fe2fbe00640 1 -- 172.21.15.32:0/1448091705 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2f4011d60 con 0x7fe3040b6c10 2024-08-03T21:01:36.001 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:35.997+0000 7fe30a400640 1 -- 172.21.15.32:0/1448091705 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe30409c970 con 0x7fe3040b6c10 2024-08-03T21:01:36.001 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:35.997+0000 7fe30a400640 1 -- 172.21.15.32:0/1448091705 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe3041f59a0 con 0x7fe3040b6c10 2024-08-03T21:01:36.001 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:35.999+0000 7fe30a400640 1 -- 172.21.15.32:0/1448091705 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe3040d32b0 con 0x7fe3040b6c10 2024-08-03T21:01:36.002 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:36.000+0000 7fe2fbe00640 1 -- 172.21.15.32:0/1448091705 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7fe2f401b460 con 0x7fe3040b6c10 2024-08-03T21:01:36.002 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:36.001+0000 7fe2fbe00640 1 --2- 172.21.15.32:0/1448091705 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fe2e0077010 0x7fe2e0079520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:01:36.002 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:36.001+0000 7fe2fbe00640 1 -- 172.21.15.32:0/1448091705 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(139..139 src has 1..139) ==== 5029+0+0 (secure 0 0 0) 0x7fe2f4097fb0 con 0x7fe3040b6c10 2024-08-03T21:01:36.003 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:36.001+0000 7fe2faa00640 1 --2- 172.21.15.32:0/1448091705 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fe2e0077010 0x7fe2e0079520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:01:36.003 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:36.002+0000 7fe2faa00640 1 --2- 172.21.15.32:0/1448091705 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fe2e0077010 0x7fe2e0079520 secure :-1 s=READY pgs=925 cs=0 l=1 rev1=1 crypto rx=0x7fe3040cc6d0 tx=0x7fe2ec00b2e0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T21:01:36.006 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:36.005+0000 7fe2fbe00640 1 -- 172.21.15.32:0/1448091705 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7fe2f4020020 con 0x7fe3040b6c10 2024-08-03T21:01:36.214 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:36.212+0000 7fe30a400640 1 -- 172.21.15.32:0/1448091705 --> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "7", "target": ["mon-mgr", ""]}) -- 0x7fe3040b36d0 con 0x7fe2e0077010 2024-08-03T21:01:36.215 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:36.213+0000 7fe2fbe00640 1 -- 172.21.15.32:0/1448091705 <== mgr.4106 v2:172.21.15.118:6800/4205950699 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 7 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7fe3040b36d0 con 0x7fe2e0077010 2024-08-03T21:01:36.215 INFO:teuthology.orchestra.run.smithi032.stderr:instructed osd(s) 7 to deep-scrub 2024-08-03T21:01:36.218 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:36.216+0000 7fe30a400640 1 -- 172.21.15.32:0/1448091705 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fe2e0077010 msgr2=0x7fe2e0079520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:01:36.218 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:36.216+0000 7fe30a400640 1 --2- 172.21.15.32:0/1448091705 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fe2e0077010 0x7fe2e0079520 secure :-1 s=READY pgs=925 cs=0 l=1 rev1=1 crypto rx=0x7fe3040cc6d0 tx=0x7fe2ec00b2e0 comp rx=0 tx=0).stop 2024-08-03T21:01:36.218 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:36.216+0000 7fe30a400640 1 -- 172.21.15.32:0/1448091705 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe3040b6c10 msgr2=0x7fe30415ddd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:01:36.218 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:36.216+0000 7fe30a400640 1 --2- 172.21.15.32:0/1448091705 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe3040b6c10 0x7fe30415ddd0 secure :-1 s=READY pgs=2313 cs=0 l=1 rev1=1 crypto rx=0x7fe2f400b3c0 tx=0x7fe2f400b880 comp rx=0 tx=0).stop 2024-08-03T21:01:36.218 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:36.216+0000 7fe30a400640 1 -- 172.21.15.32:0/1448091705 shutdown_connections 2024-08-03T21:01:36.218 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:36.216+0000 7fe30a400640 1 --2- 172.21.15.32:0/1448091705 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fe2e0077010 0x7fe2e0079520 unknown :-1 s=CLOSED pgs=925 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:01:36.218 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:36.216+0000 7fe30a400640 1 --2- 172.21.15.32:0/1448091705 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fe3040cf540 0x7fe3041f5ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:01:36.219 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:36.216+0000 7fe30a400640 1 --2- 172.21.15.32:0/1448091705 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fe30415e540 0x7fe30409e250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:01:36.219 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:36.216+0000 7fe30a400640 1 --2- 172.21.15.32:0/1448091705 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe3040b6c10 0x7fe30415ddd0 unknown :-1 s=CLOSED pgs=2313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:01:36.219 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:36.216+0000 7fe30a400640 1 -- 172.21.15.32:0/1448091705 >> 172.21.15.32:0/1448091705 conn(0x7fe3040d2160 msgr2=0x7fe304072270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T21:01:36.219 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:36.216+0000 7fe30a400640 1 -- 172.21.15.32:0/1448091705 shutdown_connections 2024-08-03T21:01:36.219 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:36.216+0000 7fe30a400640 1 -- 172.21.15.32:0/1448091705 wait complete. 2024-08-03T21:01:36.242 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-08-03T21:01:36.472 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:36.469+0000 7f01a7600640 1 -- 172.21.15.32:0/1308706194 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f01a0073e60 msgr2=0x7f01a0074270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:01:36.472 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:36.469+0000 7f01a7600640 1 --2- 172.21.15.32:0/1308706194 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f01a0073e60 0x7f01a0074270 secure :-1 s=READY pgs=2314 cs=0 l=1 rev1=1 crypto rx=0x7f01a00b2f80 tx=0x7f018c02a560 comp rx=0 tx=0).stop 2024-08-03T21:01:36.472 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:36.470+0000 7f01a7600640 1 -- 172.21.15.32:0/1308706194 shutdown_connections 2024-08-03T21:01:36.472 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:36.470+0000 7f01a7600640 1 --2- 172.21.15.32:0/1308706194 >> v2:172.21.15.118:3300/0 conn(0x7f01a00747b0 0x7f01a00cc8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:01:36.472 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:36.470+0000 7f01a7600640 1 --2- 172.21.15.32:0/1308706194 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f01a0073e60 0x7f01a0074270 unknown :-1 s=CLOSED pgs=2314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:01:36.472 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:36.470+0000 7f01a7600640 1 --2- 172.21.15.32:0/1308706194 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f01a006e930 0x7f01a0144f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:01:36.473 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:36.470+0000 7f01a7600640 1 -- 172.21.15.32:0/1308706194 >> 172.21.15.32:0/1308706194 conn(0x7f01a00b1bd0 msgr2=0x7f01a00b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T21:01:36.473 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:36.470+0000 7f01a7600640 1 -- 172.21.15.32:0/1308706194 shutdown_connections 2024-08-03T21:01:36.473 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:36.470+0000 7f01a7600640 1 -- 172.21.15.32:0/1308706194 wait complete. 2024-08-03T21:01:36.473 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:36.472+0000 7f01a7600640 1 Processor -- start 2024-08-03T21:01:36.473 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:36.472+0000 7f01a7600640 1 -- start start 2024-08-03T21:01:36.476 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:36.474+0000 7f01a7600640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f01a006e930 0x7f01a0139650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:01:36.476 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:36.474+0000 7f01a7600640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f01a0073e60 0x7f01a00c4e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:01:36.476 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:36.474+0000 7f01a7600640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f01a00747b0 0x7f01a0143130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:01:36.476 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:36.474+0000 7f01a7600640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f01a00cf9c0 con 0x7f01a00747b0 2024-08-03T21:01:36.476 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:36.474+0000 7f01a7600640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f01a00cf840 con 0x7f01a0073e60 2024-08-03T21:01:36.477 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:36.474+0000 7f01a7600640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7f01a00cce20 con 0x7f01a006e930 2024-08-03T21:01:36.477 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:36.474+0000 7f01a5600640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f01a00747b0 0x7f01a0143130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:01:36.477 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:36.474+0000 7f01a5600640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f01a00747b0 0x7f01a0143130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:39722/0 (socket says 172.21.15.32:39722) 2024-08-03T21:01:36.477 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:36.474+0000 7f01a5600640 1 -- 172.21.15.32:0/1927878125 learned_addr learned my addr 172.21.15.32:0/1927878125 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T21:01:36.477 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:36.474+0000 7f01a4c00640 1 --2- 172.21.15.32:0/1927878125 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f01a006e930 0x7f01a0139650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:01:36.477 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:36.474+0000 7f019fe00640 1 --2- 172.21.15.32:0/1927878125 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f01a0073e60 0x7f01a00c4e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:01:36.477 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:36.475+0000 7f01a5600640 1 -- 172.21.15.32:0/1927878125 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f01a006e930 msgr2=0x7f01a0139650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:01:36.477 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:36.475+0000 7f01a5600640 1 --2- 172.21.15.32:0/1927878125 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f01a006e930 0x7f01a0139650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:01:36.478 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:36.475+0000 7f01a5600640 1 -- 172.21.15.32:0/1927878125 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f01a0073e60 msgr2=0x7f01a00c4e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:01:36.478 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:36.475+0000 7f01a5600640 1 --2- 172.21.15.32:0/1927878125 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f01a0073e60 0x7f01a00c4e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:01:36.478 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:36.475+0000 7f01a5600640 1 -- 172.21.15.32:0/1927878125 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f01a0139b90 con 0x7f01a00747b0 2024-08-03T21:01:36.479 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:36.475+0000 7f01a5600640 1 --2- 172.21.15.32:0/1927878125 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f01a00747b0 0x7f01a0143130 secure :-1 s=READY pgs=2315 cs=0 l=1 rev1=1 crypto rx=0x7f019800e9c0 tx=0x7f019800ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T21:01:36.479 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:36.475+0000 7f019cc00640 1 -- 172.21.15.32:0/1927878125 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0198004d20 con 0x7f01a00747b0 2024-08-03T21:01:36.479 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:36.475+0000 7f019cc00640 1 -- 172.21.15.32:0/1927878125 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f019800cb20 con 0x7f01a00747b0 2024-08-03T21:01:36.479 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:36.475+0000 7f01a7600640 1 -- 172.21.15.32:0/1927878125 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f01a00c5380 con 0x7f01a00747b0 2024-08-03T21:01:36.479 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:36.475+0000 7f019cc00640 1 -- 172.21.15.32:0/1927878125 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0198005020 con 0x7f01a00747b0 2024-08-03T21:01:36.479 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:36.475+0000 7f01a7600640 1 -- 172.21.15.32:0/1927878125 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f01a015d050 con 0x7f01a00747b0 2024-08-03T21:01:36.480 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:36.476+0000 7f01a7600640 1 -- 172.21.15.32:0/1927878125 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f01a015cae0 con 0x7f01a00747b0 2024-08-03T21:01:36.480 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:36.478+0000 7f019cc00640 1 -- 172.21.15.32:0/1927878125 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f01980051c0 con 0x7f01a00747b0 2024-08-03T21:01:36.480 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:36.479+0000 7f019cc00640 1 --2- 172.21.15.32:0/1927878125 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f0180077040 0x7f0180079550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:01:36.480 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:36.479+0000 7f019cc00640 1 -- 172.21.15.32:0/1927878125 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(139..139 src has 1..139) ==== 5029+0+0 (secure 0 0 0) 0x7f0198098f00 con 0x7f01a00747b0 2024-08-03T21:01:36.481 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:36.479+0000 7f01a4c00640 1 --2- 172.21.15.32:0/1927878125 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f0180077040 0x7f0180079550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:01:36.481 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:36.480+0000 7f01a4c00640 1 --2- 172.21.15.32:0/1927878125 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f0180077040 0x7f0180079550 secure :-1 s=READY pgs=926 cs=0 l=1 rev1=1 crypto rx=0x7f0194004630 tx=0x7f019400b2e0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T21:01:36.485 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:36.483+0000 7f019cc00640 1 -- 172.21.15.32:0/1927878125 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7f0198016030 con 0x7f01a00747b0 2024-08-03T21:01:36.687 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:36.686+0000 7f01a7600640 1 -- 172.21.15.32:0/1927878125 --> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f01a00728b0 con 0x7f0180077040 2024-08-03T21:01:36.688 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:36.687+0000 7f019cc00640 1 -- 172.21.15.32:0/1927878125 <== mgr.4106 v2:172.21.15.118:6800/4205950699 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+13357 (secure 0 0 0) 0x7f01a00728b0 con 0x7f0180077040 2024-08-03T21:01:36.688 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-03T21:01:36.689 INFO:teuthology.orchestra.run.smithi032.stderr:dumped all 2024-08-03T21:01:36.691 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:36.689+0000 7f01a7600640 1 -- 172.21.15.32:0/1927878125 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f0180077040 msgr2=0x7f0180079550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:01:36.691 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:36.689+0000 7f01a7600640 1 --2- 172.21.15.32:0/1927878125 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f0180077040 0x7f0180079550 secure :-1 s=READY pgs=926 cs=0 l=1 rev1=1 crypto rx=0x7f0194004630 tx=0x7f019400b2e0 comp rx=0 tx=0).stop 2024-08-03T21:01:36.691 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:36.689+0000 7f01a7600640 1 -- 172.21.15.32:0/1927878125 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f01a00747b0 msgr2=0x7f01a0143130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:01:36.691 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:36.689+0000 7f01a7600640 1 --2- 172.21.15.32:0/1927878125 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f01a00747b0 0x7f01a0143130 secure :-1 s=READY pgs=2315 cs=0 l=1 rev1=1 crypto rx=0x7f019800e9c0 tx=0x7f019800ee80 comp rx=0 tx=0).stop 2024-08-03T21:01:36.692 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:36.689+0000 7f01a7600640 1 -- 172.21.15.32:0/1927878125 shutdown_connections 2024-08-03T21:01:36.692 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:36.689+0000 7f01a7600640 1 --2- 172.21.15.32:0/1927878125 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f0180077040 0x7f0180079550 unknown :-1 s=CLOSED pgs=926 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:01:36.692 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:36.689+0000 7f01a7600640 1 --2- 172.21.15.32:0/1927878125 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f01a0073e60 0x7f01a00c4e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:01:36.692 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:36.689+0000 7f01a7600640 1 --2- 172.21.15.32:0/1927878125 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f01a00747b0 0x7f01a0143130 unknown :-1 s=CLOSED pgs=2315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:01:36.692 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:36.689+0000 7f01a7600640 1 --2- 172.21.15.32:0/1927878125 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f01a006e930 0x7f01a0139650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:01:36.692 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:36.689+0000 7f01a7600640 1 -- 172.21.15.32:0/1927878125 >> 172.21.15.32:0/1927878125 conn(0x7f01a00b1bd0 msgr2=0x7f01a006f930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T21:01:36.692 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:36.689+0000 7f01a7600640 1 -- 172.21.15.32:0/1927878125 shutdown_connections 2024-08-03T21:01:36.692 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:36.689+0000 7f01a7600640 1 -- 172.21.15.32:0/1927878125 wait complete. 2024-08-03T21:01:36.716 INFO:teuthology.orchestra.run.smithi032.stdout:{"pg_ready":true,"pg_map":{"version":556,"stamp":"2024-08-03T21:01:36.466148+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":2,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":2,"kb":754974720,"kb_used":310292,"kb_used_data":15080,"kb_used_omap":12,"kb_used_meta":295091,"kb_avail":754664428,"statfs":{"total":773094113280,"available":772776374272,"internally_reserved":0,"allocated":15441920,"data_stored":8019880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":302173745},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002670"},"pg_stats":[{"pgid":"1.0","version":"16'72","reported_seq":340,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-08-03T21:01:24.550724+0000","last_change":"2024-08-03T20:47:30.547632+0000","last_active":"2024-08-03T21:01:24.550724+0000","last_peered":"2024-08-03T21:01:24.550724+0000","last_clean":"2024-08-03T21:01:24.550724+0000","last_became_active":"2024-08-03T20:47:30.546255+0000","last_became_peered":"2024-08-03T20:47:30.546255+0000","last_unstale":"2024-08-03T21:01:24.550724+0000","last_undegraded":"2024-08-03T21:01:24.550724+0000","last_fullsized":"2024-08-03T21:01:24.550724+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-03T20:47:28.874451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-03T20:47:28.874451+0000","last_clean_scrub_stamp":"2024-08-03T20:47:28.874451+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T00:19:40.548812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509612,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41820,"kb_used_data":2320,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330020,"statfs":{"total":96636764160,"available":96593940480,"internally_reserved":0,"allocated":2375680,"data_stored":1445261,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":40434122},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":15,"seq":64424509613,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":37084,"kb_used_data":1740,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334756,"statfs":{"total":96636764160,"available":96598790144,"internally_reserved":0,"allocated":1781760,"data_stored":854893,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":15,"seq":64424509613,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":41244,"kb_used_data":1740,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330596,"statfs":{"total":96636764160,"available":96594530304,"internally_reserved":0,"allocated":1781760,"data_stored":854893,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509612,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":41240,"kb_used_data":1740,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330600,"statfs":{"total":96636764160,"available":96594534400,"internally_reserved":0,"allocated":1781760,"data_stored":854893,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509614,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":29340,"kb_used_data":2320,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94342500,"statfs":{"total":96636764160,"available":96606720000,"internally_reserved":0,"allocated":2375680,"data_stored":1445261,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542317,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":37084,"kb_used_data":1740,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334756,"statfs":{"total":96636764160,"available":96598790144,"internally_reserved":0,"allocated":1781760,"data_stored":854893,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":14,"seq":60129542315,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":41240,"kb_used_data":1740,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330600,"statfs":{"total":96636764160,"available":96594534400,"internally_reserved":0,"allocated":1781760,"data_stored":854893,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":40434119},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":14,"seq":60129542316,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":41240,"kb_used_data":1740,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330600,"statfs":{"total":96636764160,"available":96594534400,"internally_reserved":0,"allocated":1781760,"data_stored":854893,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-08-03T21:01:36.719 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2024-08-03T20:47:28.874451+0000 time.struct_time(tm_year=2024, tm_mon=8, tm_mday=3, tm_hour=20, tm_min=47, tm_sec=28, tm_wday=5, tm_yday=216, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=8, tm_mday=3, tm_hour=21, tm_min=1, tm_sec=29, tm_wday=5, tm_yday=216, tm_isdst=0) 2024-08-03T21:01:36.719 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2024-08-03T21:01:39.391 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-08-03T21:01:39.396 DEBUG:teuthology.orchestra.run.smithi118:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-08-03T21:01:56.721 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-08-03T21:01:56.962 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:56.960+0000 7fc52d400640 1 -- 172.21.15.32:0/3415607251 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc52806f440 msgr2=0x7fc52806f850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:01:56.962 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:56.960+0000 7fc52d400640 1 --2- 172.21.15.32:0/3415607251 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc52806f440 0x7fc52806f850 secure :-1 s=READY pgs=2316 cs=0 l=1 rev1=1 crypto rx=0x7fc528160f00 tx=0x7fc52002aa30 comp rx=0 tx=0).stop 2024-08-03T21:01:56.962 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:56.960+0000 7fc52d400640 1 -- 172.21.15.32:0/3415607251 shutdown_connections 2024-08-03T21:01:56.962 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:56.960+0000 7fc52d400640 1 --2- 172.21.15.32:0/3415607251 >> v2:172.21.15.118:3300/0 conn(0x7fc5280acd80 0x7fc5280ad190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:01:56.962 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:56.960+0000 7fc52d400640 1 --2- 172.21.15.32:0/3415607251 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc52806f440 0x7fc52806f850 unknown :-1 s=CLOSED pgs=2316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:01:56.963 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:56.960+0000 7fc52d400640 1 -- 172.21.15.32:0/3415607251 >> 172.21.15.32:0/3415607251 conn(0x7fc5280b2120 msgr2=0x7fc5280cfd60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T21:01:56.963 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:56.960+0000 7fc52d400640 1 -- 172.21.15.32:0/3415607251 shutdown_connections 2024-08-03T21:01:56.963 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:56.960+0000 7fc52d400640 1 -- 172.21.15.32:0/3415607251 wait complete. 2024-08-03T21:01:56.963 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:56.962+0000 7fc52d400640 1 Processor -- start 2024-08-03T21:01:56.963 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:56.962+0000 7fc52d400640 1 -- start start 2024-08-03T21:01:56.966 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:56.963+0000 7fc52d400640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fc52806f440 0x7fc5280c8e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:01:56.966 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:56.963+0000 7fc52d400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc5280acd80 0x7fc5280c9380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:01:56.966 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:56.963+0000 7fc52d400640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc5280af020 0x7fc5280c4460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:01:56.966 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:56.963+0000 7fc52d400640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fc5281601e0 con 0x7fc5280acd80 2024-08-03T21:01:56.966 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:56.963+0000 7fc52d400640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7fc528160060 con 0x7fc5280af020 2024-08-03T21:01:56.966 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:56.963+0000 7fc52d400640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7fc528160360 con 0x7fc52806f440 2024-08-03T21:01:56.966 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:56.964+0000 7fc527400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc5280acd80 0x7fc5280c9380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:01:56.967 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:56.964+0000 7fc527400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc5280acd80 0x7fc5280c9380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:37894/0 (socket says 172.21.15.32:37894) 2024-08-03T21:01:56.967 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:56.964+0000 7fc527400640 1 -- 172.21.15.32:0/631062933 learned_addr learned my addr 172.21.15.32:0/631062933 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T21:01:56.967 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:56.964+0000 7fc52ca00640 1 --2- 172.21.15.32:0/631062933 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc5280af020 0x7fc5280c4460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:01:56.967 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:56.964+0000 7fc527e00640 1 --2- 172.21.15.32:0/631062933 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fc52806f440 0x7fc5280c8e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:01:56.967 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:56.964+0000 7fc527400640 1 -- 172.21.15.32:0/631062933 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fc52806f440 msgr2=0x7fc5280c8e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:01:56.967 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:56.964+0000 7fc527400640 1 --2- 172.21.15.32:0/631062933 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fc52806f440 0x7fc5280c8e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:01:56.967 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:56.964+0000 7fc527400640 1 -- 172.21.15.32:0/631062933 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc5280af020 msgr2=0x7fc5280c4460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:01:56.967 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:56.964+0000 7fc527400640 1 --2- 172.21.15.32:0/631062933 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc5280af020 0x7fc5280c4460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:01:56.968 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:56.964+0000 7fc527400640 1 -- 172.21.15.32:0/631062933 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc5280c76a0 con 0x7fc5280acd80 2024-08-03T21:01:56.968 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:56.964+0000 7fc52ca00640 1 --2- 172.21.15.32:0/631062933 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc5280af020 0x7fc5280c4460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-03T21:01:56.969 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:56.965+0000 7fc527400640 1 --2- 172.21.15.32:0/631062933 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc5280acd80 0x7fc5280c9380 secure :-1 s=READY pgs=2317 cs=0 l=1 rev1=1 crypto rx=0x7fc51400d740 tx=0x7fc51400dc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T21:01:56.969 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:56.965+0000 7fc51fe00640 1 -- 172.21.15.32:0/631062933 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc514017070 con 0x7fc5280acd80 2024-08-03T21:01:56.969 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:56.965+0000 7fc51fe00640 1 -- 172.21.15.32:0/631062933 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fc514009d20 con 0x7fc5280acd80 2024-08-03T21:01:56.969 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:56.965+0000 7fc52d400640 1 -- 172.21.15.32:0/631062933 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc5280c9d20 con 0x7fc5280acd80 2024-08-03T21:01:56.969 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:56.965+0000 7fc51fe00640 1 -- 172.21.15.32:0/631062933 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5140106c0 con 0x7fc5280acd80 2024-08-03T21:01:56.969 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:56.965+0000 7fc52d400640 1 -- 172.21.15.32:0/631062933 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc5280c6eb0 con 0x7fc5280acd80 2024-08-03T21:01:56.969 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:56.966+0000 7fc52d400640 1 -- 172.21.15.32:0/631062933 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc5280b57b0 con 0x7fc5280acd80 2024-08-03T21:01:56.970 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:56.967+0000 7fc51fe00640 1 -- 172.21.15.32:0/631062933 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7fc51401a070 con 0x7fc5280acd80 2024-08-03T21:01:56.970 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:56.968+0000 7fc51fe00640 1 --2- 172.21.15.32:0/631062933 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fc508077040 0x7fc508079550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:01:56.970 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:56.968+0000 7fc51fe00640 1 -- 172.21.15.32:0/631062933 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(139..139 src has 1..139) ==== 5029+0+0 (secure 0 0 0) 0x7fc514098870 con 0x7fc5280acd80 2024-08-03T21:01:56.971 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:56.969+0000 7fc527e00640 1 --2- 172.21.15.32:0/631062933 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fc508077040 0x7fc508079550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:01:56.971 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:56.970+0000 7fc527e00640 1 --2- 172.21.15.32:0/631062933 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fc508077040 0x7fc508079550 secure :-1 s=READY pgs=927 cs=0 l=1 rev1=1 crypto rx=0x7fc5180015e0 tx=0x7fc518009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T21:01:56.974 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:56.973+0000 7fc51fe00640 1 -- 172.21.15.32:0/631062933 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7fc514061c70 con 0x7fc5280acd80 2024-08-03T21:01:57.178 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:57.177+0000 7fc52d400640 1 -- 172.21.15.32:0/631062933 --> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fc5280d0930 con 0x7fc508077040 2024-08-03T21:01:57.180 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:57.179+0000 7fc51fe00640 1 -- 172.21.15.32:0/631062933 <== mgr.4106 v2:172.21.15.118:6800/4205950699 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+13364 (secure 0 0 0) 0x7fc5280d0930 con 0x7fc508077040 2024-08-03T21:01:57.180 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-03T21:01:57.180 INFO:teuthology.orchestra.run.smithi032.stderr:dumped all 2024-08-03T21:01:57.182 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:57.180+0000 7fc52d400640 1 -- 172.21.15.32:0/631062933 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fc508077040 msgr2=0x7fc508079550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:01:57.182 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:57.180+0000 7fc52d400640 1 --2- 172.21.15.32:0/631062933 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fc508077040 0x7fc508079550 secure :-1 s=READY pgs=927 cs=0 l=1 rev1=1 crypto rx=0x7fc5180015e0 tx=0x7fc518009040 comp rx=0 tx=0).stop 2024-08-03T21:01:57.183 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:57.180+0000 7fc52d400640 1 -- 172.21.15.32:0/631062933 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc5280acd80 msgr2=0x7fc5280c9380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:01:57.183 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:57.180+0000 7fc52d400640 1 --2- 172.21.15.32:0/631062933 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc5280acd80 0x7fc5280c9380 secure :-1 s=READY pgs=2317 cs=0 l=1 rev1=1 crypto rx=0x7fc51400d740 tx=0x7fc51400dc00 comp rx=0 tx=0).stop 2024-08-03T21:01:57.183 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:57.181+0000 7fc52d400640 1 -- 172.21.15.32:0/631062933 shutdown_connections 2024-08-03T21:01:57.183 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:57.181+0000 7fc52d400640 1 --2- 172.21.15.32:0/631062933 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7fc508077040 0x7fc508079550 unknown :-1 s=CLOSED pgs=927 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:01:57.183 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:57.181+0000 7fc52d400640 1 --2- 172.21.15.32:0/631062933 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc5280af020 0x7fc5280c4460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:01:57.183 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:57.181+0000 7fc52d400640 1 --2- 172.21.15.32:0/631062933 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc5280acd80 0x7fc5280c9380 unknown :-1 s=CLOSED pgs=2317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:01:57.183 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:57.181+0000 7fc52d400640 1 --2- 172.21.15.32:0/631062933 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7fc52806f440 0x7fc5280c8e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:01:57.183 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:57.181+0000 7fc52d400640 1 -- 172.21.15.32:0/631062933 >> 172.21.15.32:0/631062933 conn(0x7fc5280b2120 msgr2=0x7fc5280afbf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T21:01:57.184 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:57.181+0000 7fc52d400640 1 -- 172.21.15.32:0/631062933 shutdown_connections 2024-08-03T21:01:57.184 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:57.181+0000 7fc52d400640 1 -- 172.21.15.32:0/631062933 wait complete. 2024-08-03T21:01:57.208 INFO:teuthology.orchestra.run.smithi032.stdout:{"pg_ready":true,"pg_map":{"version":566,"stamp":"2024-08-03T21:01:56.470573+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":74,"ondisk_log_size":74,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":2,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":2,"kb":754974720,"kb_used":310284,"kb_used_data":15080,"kb_used_omap":12,"kb_used_meta":295091,"kb_avail":754664436,"statfs":{"total":773094113280,"available":772776382464,"internally_reserved":0,"allocated":15441920,"data_stored":8019880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":302173745},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002485"},"pg_stats":[{"pgid":"1.0","version":"139'74","reported_seq":351,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-08-03T21:01:36.740619+0000","last_change":"2024-08-03T21:01:36.740662+0000","last_active":"2024-08-03T21:01:36.740619+0000","last_peered":"2024-08-03T21:01:36.740619+0000","last_clean":"2024-08-03T21:01:36.740619+0000","last_became_active":"2024-08-03T20:47:30.546255+0000","last_became_peered":"2024-08-03T20:47:30.546255+0000","last_unstale":"2024-08-03T21:01:36.740619+0000","last_undegraded":"2024-08-03T21:01:36.740619+0000","last_fullsized":"2024-08-03T21:01:36.740619+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"139'74","last_scrub_stamp":"2024-08-03T21:01:36.740523+0000","last_deep_scrub":"139'74","last_deep_scrub_stamp":"2024-08-03T21:01:36.740523+0000","last_clean_scrub_stamp":"2024-08-03T21:01:36.740523+0000","objects_scrubbed":2,"log_size":74,"log_dups_size":0,"ondisk_log_size":74,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-05T03:55:41.232894+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":74,"ondisk_log_size":74,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509616,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41816,"kb_used_data":2320,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330024,"statfs":{"total":96636764160,"available":96593944576,"internally_reserved":0,"allocated":2375680,"data_stored":1445261,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":40434122},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":15,"seq":64424509617,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":37084,"kb_used_data":1740,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334756,"statfs":{"total":96636764160,"available":96598790144,"internally_reserved":0,"allocated":1781760,"data_stored":854893,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":15,"seq":64424509617,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":41244,"kb_used_data":1740,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330596,"statfs":{"total":96636764160,"available":96594530304,"internally_reserved":0,"allocated":1781760,"data_stored":854893,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509616,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":41240,"kb_used_data":1740,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330600,"statfs":{"total":96636764160,"available":96594534400,"internally_reserved":0,"allocated":1781760,"data_stored":854893,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509618,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":29336,"kb_used_data":2320,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94342504,"statfs":{"total":96636764160,"available":96606724096,"internally_reserved":0,"allocated":2375680,"data_stored":1445261,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542321,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":37084,"kb_used_data":1740,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334756,"statfs":{"total":96636764160,"available":96598790144,"internally_reserved":0,"allocated":1781760,"data_stored":854893,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":14,"seq":60129542319,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":41240,"kb_used_data":1740,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330600,"statfs":{"total":96636764160,"available":96594534400,"internally_reserved":0,"allocated":1781760,"data_stored":854893,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":40434119},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":14,"seq":60129542320,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":41240,"kb_used_data":1740,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330600,"statfs":{"total":96636764160,"available":96594534400,"internally_reserved":0,"allocated":1781760,"data_stored":854893,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-08-03T21:01:57.209 DEBUG:teuthology.orchestra.run.smithi032:> sudo ceph --cluster ceph config set global mon_health_to_clog false 2024-08-03T21:01:57.435 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:57.433+0000 7ff694c00640 1 -- 172.21.15.32:0/1479254044 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff6740052e0 msgr2=0x7ff67400b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:01:57.435 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:57.433+0000 7ff694c00640 1 --2- 172.21.15.32:0/1479254044 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff6740052e0 0x7ff67400b9c0 secure :-1 s=READY pgs=2319 cs=0 l=1 rev1=1 crypto rx=0x7ff688009ed0 tx=0x7ff674005830 comp rx=0 tx=0).stop 2024-08-03T21:01:57.435 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:57.433+0000 7ff694c00640 1 -- 172.21.15.32:0/1479254044 shutdown_connections 2024-08-03T21:01:57.435 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:57.433+0000 7ff694c00640 1 --2- 172.21.15.32:0/1479254044 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff6740052e0 0x7ff67400b9c0 unknown :-1 s=CLOSED pgs=2319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:01:57.436 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:57.433+0000 7ff694c00640 1 -- 172.21.15.32:0/1479254044 >> 172.21.15.32:0/1479254044 conn(0x7ff6900cd1e0 msgr2=0x7ff6900cd5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T21:01:57.436 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:57.433+0000 7ff694c00640 1 -- 172.21.15.32:0/1479254044 shutdown_connections 2024-08-03T21:01:57.436 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:57.433+0000 7ff694c00640 1 -- 172.21.15.32:0/1479254044 wait complete. 2024-08-03T21:01:57.437 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:57.435+0000 7ff694c00640 1 Processor -- start 2024-08-03T21:01:57.437 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:57.435+0000 7ff694c00640 1 -- start start 2024-08-03T21:01:57.440 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:57.437+0000 7ff694c00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ff6740052e0 0x7ff6901541a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:01:57.440 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:57.437+0000 7ff694c00640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7ff6900a94c0 0x7ff6900b5600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:01:57.440 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:57.437+0000 7ff694c00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff6900b5b40 0x7ff6901427e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:01:57.440 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:57.437+0000 7ff694c00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7ff690156b70 con 0x7ff6900b5b40 2024-08-03T21:01:57.440 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:57.437+0000 7ff694c00640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7ff6901569f0 con 0x7ff6740052e0 2024-08-03T21:01:57.440 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:57.437+0000 7ff694c00640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7ff690156cf0 con 0x7ff6900a94c0 2024-08-03T21:01:57.440 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:57.437+0000 7ff68e800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff6900b5b40 0x7ff6901427e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:01:57.440 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:57.437+0000 7ff68e800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff6900b5b40 0x7ff6901427e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:37924/0 (socket says 172.21.15.32:37924) 2024-08-03T21:01:57.441 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:57.437+0000 7ff68e800640 1 -- 172.21.15.32:0/1553250775 learned_addr learned my addr 172.21.15.32:0/1553250775 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T21:01:57.441 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:57.437+0000 7ff687e00640 1 --2- 172.21.15.32:0/1553250775 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7ff6900a94c0 0x7ff6900b5600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:01:57.441 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:57.437+0000 7ff68de00640 1 --2- 172.21.15.32:0/1553250775 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ff6740052e0 0x7ff6901541a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:01:57.441 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:57.438+0000 7ff68e800640 1 -- 172.21.15.32:0/1553250775 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7ff6900a94c0 msgr2=0x7ff6900b5600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:01:57.441 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:57.438+0000 7ff68e800640 1 --2- 172.21.15.32:0/1553250775 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7ff6900a94c0 0x7ff6900b5600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:01:57.441 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:57.438+0000 7ff68e800640 1 -- 172.21.15.32:0/1553250775 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ff6740052e0 msgr2=0x7ff6901541a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:01:57.441 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:57.438+0000 7ff68e800640 1 --2- 172.21.15.32:0/1553250775 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ff6740052e0 0x7ff6901541a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:01:57.441 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:57.438+0000 7ff68e800640 1 -- 172.21.15.32:0/1553250775 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff6900b5f50 con 0x7ff6900b5b40 2024-08-03T21:01:57.442 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:57.438+0000 7ff68de00640 1 --2- 172.21.15.32:0/1553250775 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ff6740052e0 0x7ff6901541a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-03T21:01:57.442 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:57.438+0000 7ff68e800640 1 --2- 172.21.15.32:0/1553250775 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff6900b5b40 0x7ff6901427e0 secure :-1 s=READY pgs=2320 cs=0 l=1 rev1=1 crypto rx=0x7ff688009ed0 tx=0x7ff6880042d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T21:01:57.442 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:57.438+0000 7ff686000640 1 -- 172.21.15.32:0/1553250775 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6880046d0 con 0x7ff6900b5b40 2024-08-03T21:01:57.442 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:57.438+0000 7ff686000640 1 -- 172.21.15.32:0/1553250775 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7ff68803c480 con 0x7ff6900b5b40 2024-08-03T21:01:57.442 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:57.438+0000 7ff686000640 1 -- 172.21.15.32:0/1553250775 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff68803c890 con 0x7ff6900b5b40 2024-08-03T21:01:57.442 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:57.438+0000 7ff694c00640 1 -- 172.21.15.32:0/1553250775 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff690140900 con 0x7ff6900b5b40 2024-08-03T21:01:57.442 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:57.438+0000 7ff694c00640 1 -- 172.21.15.32:0/1553250775 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff6900aa1f0 con 0x7ff6900b5b40 2024-08-03T21:01:57.443 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:57.439+0000 7ff694c00640 1 -- 172.21.15.32:0/1553250775 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff6900afbb0 con 0x7ff6900b5b40 2024-08-03T21:01:57.443 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:57.441+0000 7ff686000640 1 -- 172.21.15.32:0/1553250775 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7ff688044020 con 0x7ff6900b5b40 2024-08-03T21:01:57.443 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:57.441+0000 7ff686000640 1 --2- 172.21.15.32:0/1553250775 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7ff66c077010 0x7ff66c079520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:01:57.444 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:57.442+0000 7ff686000640 1 -- 172.21.15.32:0/1553250775 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(139..139 src has 1..139) ==== 5029+0+0 (secure 0 0 0) 0x7ff6880b9740 con 0x7ff6900b5b40 2024-08-03T21:01:57.444 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:57.442+0000 7ff68de00640 1 --2- 172.21.15.32:0/1553250775 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7ff66c077010 0x7ff66c079520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:01:57.444 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:57.443+0000 7ff68de00640 1 --2- 172.21.15.32:0/1553250775 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7ff66c077010 0x7ff66c079520 secure :-1 s=READY pgs=928 cs=0 l=1 rev1=1 crypto rx=0x7ff680006520 tx=0x7ff680006b10 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T21:01:57.447 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:57.446+0000 7ff686000640 1 -- 172.21.15.32:0/1553250775 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7ff68804b020 con 0x7ff6900b5b40 2024-08-03T21:01:57.655 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:57.653+0000 7ff694c00640 1 -- 172.21.15.32:0/1553250775 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command([{prefix=config set, name=mon_health_to_clog}] v 0) -- 0x7ff6900a9ca0 con 0x7ff6900b5b40 2024-08-03T21:01:57.659 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:57.658+0000 7ff686000640 1 -- 172.21.15.32:0/1553250775 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_health_to_clog}]=0 v47)=0 v47) ==== 123+0+0 (secure 0 0 0) 0x7ff6900b5f50 con 0x7ff6900b5b40 2024-08-03T21:01:57.661 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:57.660+0000 7ff686000640 1 -- 172.21.15.32:0/1553250775 <== mon.0 v2:172.21.15.32:3300/0 8 ==== config(6 keys) ==== 172+0+0 (secure 0 0 0) 0x7ff688081fe0 con 0x7ff6900b5b40 2024-08-03T21:01:57.663 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:57.661+0000 7ff694c00640 1 -- 172.21.15.32:0/1553250775 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7ff66c077010 msgr2=0x7ff66c079520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:01:57.663 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:57.661+0000 7ff694c00640 1 --2- 172.21.15.32:0/1553250775 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7ff66c077010 0x7ff66c079520 secure :-1 s=READY pgs=928 cs=0 l=1 rev1=1 crypto rx=0x7ff680006520 tx=0x7ff680006b10 comp rx=0 tx=0).stop 2024-08-03T21:01:57.663 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:57.661+0000 7ff694c00640 1 -- 172.21.15.32:0/1553250775 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff6900b5b40 msgr2=0x7ff6901427e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:01:57.663 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:57.661+0000 7ff694c00640 1 --2- 172.21.15.32:0/1553250775 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff6900b5b40 0x7ff6901427e0 secure :-1 s=READY pgs=2320 cs=0 l=1 rev1=1 crypto rx=0x7ff688009ed0 tx=0x7ff6880042d0 comp rx=0 tx=0).stop 2024-08-03T21:01:57.664 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:57.661+0000 7ff694c00640 1 -- 172.21.15.32:0/1553250775 shutdown_connections 2024-08-03T21:01:57.664 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:57.661+0000 7ff694c00640 1 --2- 172.21.15.32:0/1553250775 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7ff66c077010 0x7ff66c079520 unknown :-1 s=CLOSED pgs=928 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:01:57.664 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:57.661+0000 7ff694c00640 1 --2- 172.21.15.32:0/1553250775 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff6900b5b40 0x7ff6901427e0 unknown :-1 s=CLOSED pgs=2320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:01:57.664 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:57.661+0000 7ff694c00640 1 --2- 172.21.15.32:0/1553250775 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7ff6900a94c0 0x7ff6900b5600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:01:57.664 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:57.661+0000 7ff694c00640 1 --2- 172.21.15.32:0/1553250775 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ff6740052e0 0x7ff6901541a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:01:57.664 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:57.661+0000 7ff694c00640 1 -- 172.21.15.32:0/1553250775 >> 172.21.15.32:0/1553250775 conn(0x7ff6900cd1e0 msgr2=0x7ff69015b4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T21:01:57.664 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:57.662+0000 7ff694c00640 1 -- 172.21.15.32:0/1553250775 shutdown_connections 2024-08-03T21:01:57.664 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:57.662+0000 7ff694c00640 1 -- 172.21.15.32:0/1553250775 wait complete. 2024-08-03T21:01:57.685 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-08-03T21:01:57.686 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-08-03T21:01:57.916 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:57.914+0000 7f06b9c00640 1 -- 172.21.15.32:0/3567313096 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0698004d40 msgr2=0x7f0698005150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:01:57.917 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:57.914+0000 7f06b9c00640 1 --2- 172.21.15.32:0/3567313096 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0698004d40 0x7f0698005150 secure :-1 s=READY pgs=2322 cs=0 l=1 rev1=1 crypto rx=0x7f06b40afb10 tx=0x7f0698005830 comp rx=0 tx=0).stop 2024-08-03T21:01:57.917 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:57.914+0000 7f06b3400640 1 -- 172.21.15.32:0/3567313096 reap_dead start 2024-08-03T21:01:57.917 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:57.915+0000 7f06b9c00640 1 -- 172.21.15.32:0/3567313096 shutdown_connections 2024-08-03T21:01:57.917 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:57.915+0000 7f06b9c00640 1 -- 172.21.15.32:0/3567313096 >> 172.21.15.32:0/3567313096 conn(0x7f06b40b1bd0 msgr2=0x7f06b40b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T21:01:57.917 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:57.915+0000 7f06b9c00640 1 -- 172.21.15.32:0/3567313096 shutdown_connections 2024-08-03T21:01:57.917 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:57.915+0000 7f06b9c00640 1 -- 172.21.15.32:0/3567313096 wait complete. 2024-08-03T21:01:57.918 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:57.917+0000 7f06b9c00640 1 Processor -- start 2024-08-03T21:01:57.919 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:57.917+0000 7f06b9c00640 1 -- start start 2024-08-03T21:01:57.922 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:57.919+0000 7f06b9c00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f06b415ca00 0x7f06b4144210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:01:57.922 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:57.919+0000 7f06b9c00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f06b414e940 0x7f06b413e830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:01:57.922 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:57.919+0000 7f06b9c00640 1 --2- >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f06b414ed50 0x7f06b413ed70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:01:57.922 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:57.919+0000 7f06b9c00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f06b40cf9c0 con 0x7f06b415ca00 2024-08-03T21:01:57.922 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:57.919+0000 7f06b9c00640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f06b40cf840 con 0x7f06b414e940 2024-08-03T21:01:57.922 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:57.919+0000 7f06b9c00640 1 -- --> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] -- mon_getmap magic: 0 -- 0x7f06b4074270 con 0x7f06b414ed50 2024-08-03T21:01:57.923 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:57.919+0000 7f06b2a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f06b415ca00 0x7f06b4144210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:01:57.923 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:57.919+0000 7f06b2a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f06b415ca00 0x7f06b4144210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:37936/0 (socket says 172.21.15.32:37936) 2024-08-03T21:01:57.923 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:57.919+0000 7f06b2a00640 1 -- 172.21.15.32:0/304670761 learned_addr learned my addr 172.21.15.32:0/304670761 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-08-03T21:01:57.923 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:57.919+0000 7f06b3400640 1 --2- 172.21.15.32:0/304670761 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f06b414ed50 0x7f06b413ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:01:57.923 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:57.919+0000 7f06b2000640 1 --2- 172.21.15.32:0/304670761 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f06b414e940 0x7f06b413e830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:01:57.923 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:57.919+0000 7f06b2a00640 1 -- 172.21.15.32:0/304670761 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f06b414ed50 msgr2=0x7f06b413ed70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:01:57.923 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:57.919+0000 7f06b2a00640 1 --2- 172.21.15.32:0/304670761 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f06b414ed50 0x7f06b413ed70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:01:57.923 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:57.919+0000 7f06b2a00640 1 -- 172.21.15.32:0/304670761 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f06b414e940 msgr2=0x7f06b413e830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:01:57.923 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:57.919+0000 7f06b2a00640 1 --2- 172.21.15.32:0/304670761 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f06b414e940 0x7f06b413e830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:01:57.924 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:57.919+0000 7f06b2a00640 1 -- 172.21.15.32:0/304670761 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f06b415fd70 con 0x7f06b415ca00 2024-08-03T21:01:57.924 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:57.920+0000 7f06b2a00640 1 --2- 172.21.15.32:0/304670761 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f06b415ca00 0x7f06b4144210 secure :-1 s=READY pgs=2323 cs=0 l=1 rev1=1 crypto rx=0x7f06a4011b10 tx=0x7f06a4009f10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T21:01:57.924 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:57.920+0000 7f06aaa00640 1 -- 172.21.15.32:0/304670761 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06a400fce0 con 0x7f06b415ca00 2024-08-03T21:01:57.924 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:57.920+0000 7f06aaa00640 1 -- 172.21.15.32:0/304670761 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(6 keys) ==== 172+0+0 (secure 0 0 0) 0x7f06a4004020 con 0x7f06b415ca00 2024-08-03T21:01:57.924 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:57.920+0000 7f06b9c00640 1 -- 172.21.15.32:0/304670761 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f06b415ce10 con 0x7f06b415ca00 2024-08-03T21:01:57.924 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:57.920+0000 7f06aaa00640 1 -- 172.21.15.32:0/304670761 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06a4004490 con 0x7f06b415ca00 2024-08-03T21:01:57.924 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:57.920+0000 7f06b9c00640 1 -- 172.21.15.32:0/304670761 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f06b415f7b0 con 0x7f06b415ca00 2024-08-03T21:01:57.925 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:57.921+0000 7f06b9c00640 1 -- 172.21.15.32:0/304670761 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f06b413f2b0 con 0x7f06b415ca00 2024-08-03T21:01:57.925 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:57.922+0000 7f06aaa00640 1 -- 172.21.15.32:0/304670761 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 99139+0+0 (secure 0 0 0) 0x7f06a40035f0 con 0x7f06b415ca00 2024-08-03T21:01:57.926 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:57.923+0000 7f06aaa00640 1 --2- 172.21.15.32:0/304670761 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f06940771b0 0x7f06940796c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-03T21:01:57.926 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:57.923+0000 7f06aaa00640 1 -- 172.21.15.32:0/304670761 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(139..139 src has 1..139) ==== 5029+0+0 (secure 0 0 0) 0x7f06a40993b0 con 0x7f06b415ca00 2024-08-03T21:01:57.926 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:57.924+0000 7f06b2000640 1 --2- 172.21.15.32:0/304670761 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f06940771b0 0x7f06940796c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-03T21:01:57.926 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:57.925+0000 7f06b2000640 1 --2- 172.21.15.32:0/304670761 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f06940771b0 0x7f06940796c0 secure :-1 s=READY pgs=929 cs=0 l=1 rev1=1 crypto rx=0x7f06a0009b90 tx=0x7f06a0005f90 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-03T21:01:57.929 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:57.928+0000 7f06aaa00640 1 -- 172.21.15.32:0/304670761 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194029 (secure 0 0 0) 0x7f06b415ce10 con 0x7f06b415ca00 2024-08-03T21:01:58.172 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:58.170+0000 7f06b9c00640 1 -- 172.21.15.32:0/304670761 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f06b4144f90 con 0x7f06b415ca00 2024-08-03T21:01:58.172 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:58.171+0000 7f06aaa00640 1 -- 172.21.15.32:0/304670761 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 348 v348) ==== 95+0+3917 (secure 0 0 0) 0x7f06a40627b0 con 0x7f06b415ca00 2024-08-03T21:01:58.173 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-08-03T21:01:58.173 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":348,"btime":"2024-08-03T21:01:20:503933+0000","default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8519,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6836/3249267506","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6834","nonce":3249267506},{"type":"v1","addr":"172.21.15.118:6836","nonce":3249267506}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":334},{"gid":8522,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6837/408542078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6835","nonce":408542078},{"type":"v1","addr":"172.21.15.118:6837","nonce":408542078}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":335},{"gid":8648,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6835/1693401067","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6833","nonce":1693401067},{"type":"v1","addr":"172.21.15.32:6835","nonce":1693401067}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":342},{"gid":10824,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6834/380281788","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":380281788},{"type":"v1","addr":"172.21.15.32:6834","nonce":380281788}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":348}],"filesystems":[]} 2024-08-03T21:01:58.173 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 348 2024-08-03T21:01:58.175 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:58.173+0000 7f06b9c00640 1 -- 172.21.15.32:0/304670761 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f06940771b0 msgr2=0x7f06940796c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:01:58.175 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:58.173+0000 7f06b9c00640 1 --2- 172.21.15.32:0/304670761 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f06940771b0 0x7f06940796c0 secure :-1 s=READY pgs=929 cs=0 l=1 rev1=1 crypto rx=0x7f06a0009b90 tx=0x7f06a0005f90 comp rx=0 tx=0).stop 2024-08-03T21:01:58.175 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:58.173+0000 7f06b9c00640 1 -- 172.21.15.32:0/304670761 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f06b415ca00 msgr2=0x7f06b4144210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-03T21:01:58.175 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:58.173+0000 7f06b9c00640 1 --2- 172.21.15.32:0/304670761 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f06b415ca00 0x7f06b4144210 secure :-1 s=READY pgs=2323 cs=0 l=1 rev1=1 crypto rx=0x7f06a4011b10 tx=0x7f06a4009f10 comp rx=0 tx=0).stop 2024-08-03T21:01:58.175 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:58.173+0000 7f06b9c00640 1 -- 172.21.15.32:0/304670761 shutdown_connections 2024-08-03T21:01:58.175 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:58.173+0000 7f06b9c00640 1 --2- 172.21.15.32:0/304670761 >> [v2:172.21.15.118:6800/4205950699,v1:172.21.15.118:6801/4205950699] conn(0x7f06940771b0 0x7f06940796c0 unknown :-1 s=CLOSED pgs=929 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:01:58.176 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:58.173+0000 7f06b9c00640 1 --2- 172.21.15.32:0/304670761 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f06b414e940 0x7f06b413e830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:01:58.176 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:58.173+0000 7f06b9c00640 1 --2- 172.21.15.32:0/304670761 >> [v2:172.21.15.118:3301/0,v1:172.21.15.118:6790/0] conn(0x7f06b414ed50 0x7f06b413ed70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:01:58.176 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:58.173+0000 7f06b9c00640 1 --2- 172.21.15.32:0/304670761 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f06b415ca00 0x7f06b4144210 unknown :-1 s=CLOSED pgs=2323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-03T21:01:58.176 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:58.173+0000 7f06b9c00640 1 -- 172.21.15.32:0/304670761 >> 172.21.15.32:0/304670761 conn(0x7f06b40b1bd0 msgr2=0x7f06b40afb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-03T21:01:58.176 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:58.173+0000 7f06b9c00640 1 -- 172.21.15.32:0/304670761 shutdown_connections 2024-08-03T21:01:58.176 INFO:teuthology.orchestra.run.smithi032.stderr:2024-08-03T21:01:58.173+0000 7f06b9c00640 1 -- 172.21.15.32:0/304670761 wait complete. 2024-08-03T21:01:58.199 DEBUG:tasks.cephfs.filesystem:already dead... 2024-08-03T21:01:58.200 INFO:teuthology.misc:Shutting down mds daemons... 2024-08-03T21:01:58.200 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-08-03T21:01:58.200 INFO:teuthology.orchestra.run:waiting for 300 2024-08-03T21:01:58.220 INFO:tasks.ceph.mds.a:Stopped 2024-08-03T21:01:58.220 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-08-03T21:01:58.220 INFO:teuthology.orchestra.run:waiting for 300 2024-08-03T21:01:58.285 INFO:tasks.ceph.mds.c:Stopped 2024-08-03T21:01:58.285 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-08-03T21:01:58.285 INFO:teuthology.orchestra.run:waiting for 300 2024-08-03T21:01:58.310 INFO:tasks.ceph.mds.b:Stopped 2024-08-03T21:01:58.310 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-08-03T21:01:58.310 INFO:teuthology.orchestra.run:waiting for 300 2024-08-03T21:01:58.374 INFO:tasks.ceph.mds.d:Stopped 2024-08-03T21:01:58.375 INFO:teuthology.misc:Shutting down osd daemons... 2024-08-03T21:01:58.375 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2024-08-03T21:01:58.375 INFO:teuthology.orchestra.run:waiting for 300 2024-08-03T21:01:58.426 INFO:tasks.ceph.osd.0:Stopped 2024-08-03T21:01:58.426 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2024-08-03T21:01:58.426 INFO:teuthology.orchestra.run:waiting for 300 2024-08-03T21:01:58.497 INFO:tasks.ceph.osd.1:Stopped 2024-08-03T21:01:58.498 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2024-08-03T21:01:58.498 INFO:teuthology.orchestra.run:waiting for 300 2024-08-03T21:01:58.561 INFO:tasks.ceph.osd.2:Stopped 2024-08-03T21:01:58.562 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2024-08-03T21:01:58.562 INFO:teuthology.orchestra.run:waiting for 300 2024-08-03T21:01:58.628 INFO:tasks.ceph.osd.3:Stopped 2024-08-03T21:01:58.629 DEBUG:tasks.ceph.osd.4:waiting for process to exit 2024-08-03T21:01:58.629 INFO:teuthology.orchestra.run:waiting for 300 2024-08-03T21:01:58.683 INFO:tasks.ceph.osd.4:Stopped 2024-08-03T21:01:58.684 DEBUG:tasks.ceph.osd.5:waiting for process to exit 2024-08-03T21:01:58.684 INFO:teuthology.orchestra.run:waiting for 300 2024-08-03T21:01:58.744 INFO:tasks.ceph.osd.5:Stopped 2024-08-03T21:01:58.744 DEBUG:tasks.ceph.osd.6:waiting for process to exit 2024-08-03T21:01:58.744 INFO:teuthology.orchestra.run:waiting for 300 2024-08-03T21:01:58.803 INFO:tasks.ceph.osd.6:Stopped 2024-08-03T21:01:58.803 DEBUG:tasks.ceph.osd.7:waiting for process to exit 2024-08-03T21:01:58.803 INFO:teuthology.orchestra.run:waiting for 300 2024-08-03T21:01:58.870 INFO:tasks.ceph.osd.7:Stopped 2024-08-03T21:01:58.871 INFO:teuthology.misc:Shutting down mgr daemons... 2024-08-03T21:01:58.871 DEBUG:tasks.ceph.mgr.y:waiting for process to exit 2024-08-03T21:01:58.871 INFO:teuthology.orchestra.run:waiting for 300 2024-08-03T21:01:58.944 INFO:tasks.ceph.mgr.y:Stopped 2024-08-03T21:01:58.944 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2024-08-03T21:01:58.944 INFO:teuthology.orchestra.run:waiting for 300 2024-08-03T21:01:59.020 INFO:tasks.ceph.mgr.x:Stopped 2024-08-03T21:01:59.020 INFO:teuthology.misc:Shutting down mon daemons... 2024-08-03T21:01:59.021 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-08-03T21:01:59.021 INFO:teuthology.orchestra.run:waiting for 300 2024-08-03T21:01:59.051 INFO:tasks.ceph.mon.a:Stopped 2024-08-03T21:01:59.052 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-08-03T21:01:59.052 INFO:teuthology.orchestra.run:waiting for 300 2024-08-03T21:01:59.078 INFO:tasks.ceph.mon.b:Stopped 2024-08-03T21:01:59.079 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-08-03T21:01:59.079 INFO:teuthology.orchestra.run:waiting for 300 2024-08-03T21:01:59.145 INFO:tasks.ceph.mon.c:Stopped 2024-08-03T21:01:59.146 INFO:tasks.ceph:Checking cluster log for badness... 2024-08-03T21:01:59.146 DEBUG:teuthology.orchestra.run.smithi032:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v FS_DEGRADED | egrep -v FS_INLINE_DATA_DEPRECATED | egrep -v FS_WITH_FAILED_MDS | egrep -v MDS_ALL_DOWN | egrep -v MDS_DAMAGE | egrep -v MDS_DEGRADED | egrep -v MDS_FAILED | egrep -v MDS_INSUFFICIENT_STANDBY | egrep -v MDS_UP_LESS_THAN_MAX | egrep -v 'online, but wants' | egrep -v 'filesystem is online with fewer MDS than max_mds' | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'do not have an application enabled' | egrep -v 'overall HEALTH_' | egrep -v 'Replacing daemon' | egrep -v 'deprecated feature inline_data' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | egrep -v PG_AVAILABILITY | egrep -v PG_DEGRADED | egrep -v 'Reduced data availability' | egrep -v 'Degraded data redundancy' | egrep -v 'bad backtrace' | egrep -v 'object missing on disk' | egrep -v 'error reading table object' | egrep -v 'error reading sessionmap' | egrep -v 'Error loading MDS rank' | egrep -v 'missing journal object' | egrep -v 'Error recovering journal' | egrep -v 'error decoding table object' | egrep -v 'failed to read JournalPointer' | egrep -v 'Corrupt directory entry' | egrep -v 'Corrupt fnode header' | egrep -v 'corrupt sessionmap header' | egrep -v 'Corrupt dentry' | egrep -v 'Scrub error on inode' | egrep -v 'Metadata damage detected' | egrep -v MDS_READ_ONLY | egrep -v 'force file system read-only' | egrep -v 'with standby daemon mds' | egrep -v 'MDS abort because newly corrupt dentry' | head -n 1 2024-08-03T21:01:59.238 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi032.front.sepia.ceph.com 2024-08-03T21:01:59.239 DEBUG:teuthology.orchestra.run.smithi032:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2024-08-03T21:01:59.456 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi032.front.sepia.ceph.com 2024-08-03T21:01:59.456 DEBUG:teuthology.orchestra.run.smithi032:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2024-08-03T21:01:59.558 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi032.front.sepia.ceph.com 2024-08-03T21:01:59.558 DEBUG:teuthology.orchestra.run.smithi032:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2024-08-03T21:01:59.674 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi032.front.sepia.ceph.com 2024-08-03T21:01:59.675 DEBUG:teuthology.orchestra.run.smithi032:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2024-08-03T21:01:59.776 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-4 on ubuntu@smithi118.front.sepia.ceph.com 2024-08-03T21:01:59.776 DEBUG:teuthology.orchestra.run.smithi118:> sync && sudo umount -f /var/lib/ceph/osd/ceph-4 2024-08-03T21:02:00.085 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-5 on ubuntu@smithi118.front.sepia.ceph.com 2024-08-03T21:02:00.085 DEBUG:teuthology.orchestra.run.smithi118:> sync && sudo umount -f /var/lib/ceph/osd/ceph-5 2024-08-03T21:02:00.186 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-6 on ubuntu@smithi118.front.sepia.ceph.com 2024-08-03T21:02:00.187 DEBUG:teuthology.orchestra.run.smithi118:> sync && sudo umount -f /var/lib/ceph/osd/ceph-6 2024-08-03T21:02:00.285 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-7 on ubuntu@smithi118.front.sepia.ceph.com 2024-08-03T21:02:00.285 DEBUG:teuthology.orchestra.run.smithi118:> sync && sudo umount -f /var/lib/ceph/osd/ceph-7 2024-08-03T21:02:00.395 INFO:tasks.ceph:Archiving mon data... 2024-08-03T21:02:00.397 DEBUG:teuthology.misc:Transferring archived files from smithi032:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/vshankar-2024-08-02_18:35:56-fs-wip-vshankar-testing-20240802.091913-debug-testing-default-smithi/7833836/data/mon.a.tgz 2024-08-03T21:02:00.397 DEBUG:teuthology.orchestra.run.smithi032:> mktemp 2024-08-03T21:02:00.434 INFO:teuthology.orchestra.run.smithi032.stdout:/tmp/tmp.BePqMBHIWI 2024-08-03T21:02:00.434 DEBUG:teuthology.orchestra.run.smithi032:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.BePqMBHIWI 2024-08-03T21:02:00.677 DEBUG:teuthology.orchestra.run.smithi032:> sudo chmod 0666 /tmp/tmp.BePqMBHIWI 2024-08-03T21:02:00.771 DEBUG:teuthology.orchestra.remote:smithi032:/tmp/tmp.BePqMBHIWI is 663KB 2024-08-03T21:02:00.867 DEBUG:teuthology.orchestra.run.smithi032:> rm -fr /tmp/tmp.BePqMBHIWI 2024-08-03T21:02:00.902 DEBUG:teuthology.misc:Transferring archived files from smithi118:/var/lib/ceph/mon/ceph-b to /home/teuthworker/archive/vshankar-2024-08-02_18:35:56-fs-wip-vshankar-testing-20240802.091913-debug-testing-default-smithi/7833836/data/mon.b.tgz 2024-08-03T21:02:00.902 DEBUG:teuthology.orchestra.run.smithi118:> mktemp 2024-08-03T21:02:00.937 INFO:teuthology.orchestra.run.smithi118.stdout:/tmp/tmp.uB7qfu32Hx 2024-08-03T21:02:00.938 DEBUG:teuthology.orchestra.run.smithi118:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-b -- . > /tmp/tmp.uB7qfu32Hx 2024-08-03T21:02:01.198 DEBUG:teuthology.orchestra.run.smithi118:> sudo chmod 0666 /tmp/tmp.uB7qfu32Hx 2024-08-03T21:02:01.291 DEBUG:teuthology.orchestra.remote:smithi118:/tmp/tmp.uB7qfu32Hx is 700KB 2024-08-03T21:02:01.385 DEBUG:teuthology.orchestra.run.smithi118:> rm -fr /tmp/tmp.uB7qfu32Hx 2024-08-03T21:02:01.420 DEBUG:teuthology.misc:Transferring archived files from smithi118:/var/lib/ceph/mon/ceph-c to /home/teuthworker/archive/vshankar-2024-08-02_18:35:56-fs-wip-vshankar-testing-20240802.091913-debug-testing-default-smithi/7833836/data/mon.c.tgz 2024-08-03T21:02:01.420 DEBUG:teuthology.orchestra.run.smithi118:> mktemp 2024-08-03T21:02:01.495 INFO:teuthology.orchestra.run.smithi118.stdout:/tmp/tmp.53gAOIkYOe 2024-08-03T21:02:01.495 DEBUG:teuthology.orchestra.run.smithi118:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-c -- . > /tmp/tmp.53gAOIkYOe 2024-08-03T21:02:01.732 DEBUG:teuthology.orchestra.run.smithi118:> sudo chmod 0666 /tmp/tmp.53gAOIkYOe 2024-08-03T21:02:01.825 DEBUG:teuthology.orchestra.remote:smithi118:/tmp/tmp.53gAOIkYOe is 700KB 2024-08-03T21:02:01.918 DEBUG:teuthology.orchestra.run.smithi118:> rm -fr /tmp/tmp.53gAOIkYOe 2024-08-03T21:02:01.952 INFO:tasks.ceph:Cleaning ceph cluster... 2024-08-03T21:02:01.953 DEBUG:teuthology.orchestra.run.smithi032:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-08-03T21:02:01.957 DEBUG:teuthology.orchestra.run.smithi118:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-08-03T21:02:02.085 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-08-03T21:02:02.120 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-08-03T21:02:02.121 INFO:tasks.ceph:Archiving crash dumps... 2024-08-03T21:02:02.122 DEBUG:teuthology.misc:Transferring archived files from smithi032:/var/lib/ceph/crash to /home/teuthworker/archive/vshankar-2024-08-02_18:35:56-fs-wip-vshankar-testing-20240802.091913-debug-testing-default-smithi/7833836/remote/smithi032/crash 2024-08-03T21:02:02.123 DEBUG:teuthology.orchestra.run.smithi032:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-08-03T21:02:02.220 DEBUG:teuthology.misc:Transferring archived files from smithi118:/var/lib/ceph/crash to /home/teuthworker/archive/vshankar-2024-08-02_18:35:56-fs-wip-vshankar-testing-20240802.091913-debug-testing-default-smithi/7833836/remote/smithi118/crash 2024-08-03T21:02:02.221 DEBUG:teuthology.orchestra.run.smithi118:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-08-03T21:02:02.274 INFO:tasks.ceph:Shutting down logrotate 2024-08-03T21:02:02.275 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-08-03T21:02:02.279 DEBUG:teuthology.orchestra.run.smithi118:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-08-03T21:02:02.380 DEBUG:teuthology.orchestra.run.smithi032:> sudo rm /etc/logrotate.d/ceph-test.conf 2024-08-03T21:02:02.436 DEBUG:teuthology.orchestra.run.smithi118:> sudo rm /etc/logrotate.d/ceph-test.conf 2024-08-03T21:02:02.491 INFO:tasks.ceph:Compressing logs... 2024-08-03T21:02:02.491 DEBUG:teuthology.orchestra.run.smithi032:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-08-03T21:02:02.496 DEBUG:teuthology.orchestra.run.smithi118:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-08-03T21:02:02.546 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84390.log 2024-08-03T21:02:02.547 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73919.log 2024-08-03T21:02:02.548 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.84390.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72506.log 2024-08-03T21:02:02.548 INFO:teuthology.orchestra.run.smithi032.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.84390.log.gz 2024-08-03T21:02:02.548 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.73919.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72608.log 2024-08-03T21:02:02.549 INFO:teuthology.orchestra.run.smithi032.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.73919.log.gz 2024-08-03T21:02:02.549 INFO:teuthology.orchestra.run.smithi118.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-08-03T21:02:02.550 INFO:teuthology.orchestra.run.smithi118.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37658.log 2024-08-03T21:02:02.551 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.72506.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59605.log 2024-08-03T21:02:02.551 INFO:teuthology.orchestra.run.smithi032.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.72506.log.gz 2024-08-03T21:02:02.551 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.72608.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72866.log 2024-08-03T21:02:02.551 INFO:teuthology.orchestra.run.smithi032.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.72608.log.gz 2024-08-03T21:02:02.555 INFO:teuthology.orchestra.run.smithi118.stderr:/var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-08-03T21:02:02.556 INFO:teuthology.orchestra.run.smithi118.stderr:/var/log/ceph/ceph-client.admin.37658.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37698.log 2024-08-03T21:02:02.556 INFO:teuthology.orchestra.run.smithi118.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.37658.log.gz 2024-08-03T21:02:02.556 INFO:teuthology.orchestra.run.smithi118.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.7.log 2024-08-03T21:02:02.556 INFO:teuthology.orchestra.run.smithi118.stderr:/var/log/ceph/ceph-client.admin.37698.log: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.37698.log.gz 2024-08-03T21:02:02.556 INFO:teuthology.orchestra.run.smithi118.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mds.b.log 2024-08-03T21:02:02.556 INFO:teuthology.orchestra.run.smithi118.stderr:/var/log/ceph/ceph-osd.7.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.5.log 2024-08-03T21:02:02.557 INFO:teuthology.orchestra.run.smithi118.stderr:/var/log/ceph/ceph-mds.b.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37938.log 2024-08-03T21:02:02.562 INFO:teuthology.orchestra.run.smithi118.stderr: 92.8% -- replaced with /var/log/ceph/ceph.log.gz 2024-08-03T21:02:02.562 INFO:teuthology.orchestra.run.smithi118.stderr:/var/log/ceph/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.4.log 2024-08-03T21:02:02.562 INFO:teuthology.orchestra.run.smithi118.stderr: 94.1% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-08-03T21:02:02.562 INFO:teuthology.orchestra.run.smithi118.stderr:/var/log/ceph/ceph-client.admin.37938.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.1.42142.log 2024-08-03T21:02:02.562 INFO:teuthology.orchestra.run.smithi118.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.37938.log.gz 2024-08-03T21:02:02.563 INFO:teuthology.orchestra.run.smithi118.stderr:/var/log/ceph/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2024-08-03T21:02:02.563 INFO:teuthology.orchestra.run.smithi118.stderr:/var/log/ceph/ceph-client.1.42142.log: 0.0% -- replaced with /var/log/ceph/ceph-client.1.42142.log.gz 2024-08-03T21:02:02.563 INFO:teuthology.orchestra.run.smithi118.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.c.log 2024-08-03T21:02:02.563 INFO:teuthology.orchestra.run.smithi118.stderr:/var/log/ceph/ceph-mgr.x.log: 93.2% -- replaced with /var/log/ceph/ceph-mds.b.log.gz 2024-08-03T21:02:02.563 INFO:teuthology.orchestra.run.smithi118.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.1.42161.log 2024-08-03T21:02:02.566 INFO:teuthology.orchestra.run.smithi118.stderr:/var/log/ceph/ceph-mon.c.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.6.log 2024-08-03T21:02:02.566 INFO:teuthology.orchestra.run.smithi118.stderr:/var/log/ceph/ceph-client.1.42161.log: 0.0%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37898.log 2024-08-03T21:02:02.566 INFO:teuthology.orchestra.run.smithi118.stderr: -- replaced with /var/log/ceph/ceph-client.1.42161.log.gz 2024-08-03T21:02:02.567 INFO:teuthology.orchestra.run.smithi118.stderr:/var/log/ceph/ceph-osd.6.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37858.log 2024-08-03T21:02:02.568 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.59605.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88659.log 2024-08-03T21:02:02.568 INFO:teuthology.orchestra.run.smithi032.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.59605.log.gz 2024-08-03T21:02:02.569 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.72866.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83234.log 2024-08-03T21:02:02.569 INFO:teuthology.orchestra.run.smithi032.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.72866.log.gz 2024-08-03T21:02:02.569 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.88659.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85690.log 2024-08-03T21:02:02.569 INFO:teuthology.orchestra.run.smithi032.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.88659.log.gz 2024-08-03T21:02:02.569 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.83234.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44190.log 2024-08-03T21:02:02.569 INFO:teuthology.orchestra.run.smithi032.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.83234.log.gz 2024-08-03T21:02:02.569 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.85690.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45985.log 2024-08-03T21:02:02.569 INFO:teuthology.orchestra.run.smithi032.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.85690.log.gz 2024-08-03T21:02:02.570 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.44190.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79422.log 2024-08-03T21:02:02.570 INFO:teuthology.orchestra.run.smithi032.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.44190.log.gz 2024-08-03T21:02:02.570 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.45985.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68156.log 2024-08-03T21:02:02.570 INFO:teuthology.orchestra.run.smithi032.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.45985.log.gz 2024-08-03T21:02:02.570 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.79422.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76394.log 2024-08-03T21:02:02.570 INFO:teuthology.orchestra.run.smithi032.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.79422.log.gz 2024-08-03T21:02:02.570 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.68156.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53874.log 2024-08-03T21:02:02.570 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.76394.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50263.log 2024-08-03T21:02:02.571 INFO:teuthology.orchestra.run.smithi032.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.76394.log.gz 2024-08-03T21:02:02.571 INFO:teuthology.orchestra.run.smithi032.stderr: 92.5% -- replaced with /var/log/ceph/ceph-client.admin.68156.log.gz 2024-08-03T21:02:02.571 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.53874.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75041.log 2024-08-03T21:02:02.571 INFO:teuthology.orchestra.run.smithi032.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.53874.log.gz 2024-08-03T21:02:02.571 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.50263.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67610.log 2024-08-03T21:02:02.571 INFO:teuthology.orchestra.run.smithi032.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.50263.log.gz 2024-08-03T21:02:02.571 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.75041.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52605.log 2024-08-03T21:02:02.571 INFO:teuthology.orchestra.run.smithi032.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.75041.log.gz 2024-08-03T21:02:02.572 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.67610.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60852.log 2024-08-03T21:02:02.572 INFO:teuthology.orchestra.run.smithi032.stderr: 90.9% -- replaced with /var/log/ceph/ceph-client.admin.67610.log.gz 2024-08-03T21:02:02.572 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.52605.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58709.log 2024-08-03T21:02:02.572 INFO:teuthology.orchestra.run.smithi032.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.52605.log.gz 2024-08-03T21:02:02.572 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.60852.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81998.log 2024-08-03T21:02:02.572 INFO:teuthology.orchestra.run.smithi032.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.60852.log.gz 2024-08-03T21:02:02.572 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.58709.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83505.log 2024-08-03T21:02:02.572 INFO:teuthology.orchestra.run.smithi032.stderr: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.58709.log.gz 2024-08-03T21:02:02.573 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.81998.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71813.log 2024-08-03T21:02:02.573 INFO:teuthology.orchestra.run.smithi032.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.81998.log.gz 2024-08-03T21:02:02.573 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.83505.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.2.45375.log 2024-08-03T21:02:02.573 INFO:teuthology.orchestra.run.smithi032.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.83505.log.gz 2024-08-03T21:02:02.573 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.71813.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69543.log 2024-08-03T21:02:02.574 INFO:teuthology.orchestra.run.smithi032.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.71813.log.gz 2024-08-03T21:02:02.574 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.2.45375.log: 0.0% -- replaced with /var/log/ceph/ceph-client.2.45375.log.gz 2024-08-03T21:02:02.574 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62951.log 2024-08-03T21:02:02.574 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.69543.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63395.log 2024-08-03T21:02:02.574 INFO:teuthology.orchestra.run.smithi032.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.69543.log.gz 2024-08-03T21:02:02.585 INFO:teuthology.orchestra.run.smithi118.stderr:/var/log/ceph/ceph-client.admin.37898.log: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.37898.log.gz 2024-08-03T21:02:02.585 INFO:teuthology.orchestra.run.smithi118.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37738.log 2024-08-03T21:02:02.585 INFO:teuthology.orchestra.run.smithi118.stderr:/var/log/ceph/ceph-client.admin.37858.log: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.37858.log.gz 2024-08-03T21:02:02.586 INFO:teuthology.orchestra.run.smithi118.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37778.log 2024-08-03T21:02:02.586 INFO:teuthology.orchestra.run.smithi118.stderr:/var/log/ceph/ceph-client.admin.37738.log: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.37738.log.gz 2024-08-03T21:02:02.586 INFO:teuthology.orchestra.run.smithi118.stderr:gzip -5 --verbose/var/log/ceph/ceph-client.admin.37778.log: -- /var/log/ceph/ceph-client.3.42560.log 2024-08-03T21:02:02.586 INFO:teuthology.orchestra.run.smithi118.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.37778.log.gz 2024-08-03T21:02:02.586 INFO:teuthology.orchestra.run.smithi118.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37818.log 2024-08-03T21:02:02.586 INFO:teuthology.orchestra.run.smithi118.stderr:/var/log/ceph/ceph-client.3.42560.log: 0.0% -- replaced with /var/log/ceph/ceph-client.3.42560.log.gz 2024-08-03T21:02:02.586 INFO:teuthology.orchestra.run.smithi118.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.3.42556.log 2024-08-03T21:02:02.587 INFO:teuthology.orchestra.run.smithi118.stderr:/var/log/ceph/ceph-client.admin.37818.log: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.37818.log.gz 2024-08-03T21:02:02.587 INFO:teuthology.orchestra.run.smithi118.stderr:/var/log/ceph/ceph-client.3.42556.log: 0.0% -- replaced with /var/log/ceph/ceph-client.3.42556.log.gz 2024-08-03T21:02:02.587 INFO:teuthology.orchestra.run.smithi118.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mds.d.log 2024-08-03T21:02:02.587 INFO:teuthology.orchestra.run.smithi118.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.b.log 2024-08-03T21:02:02.600 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.62951.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65169.log 2024-08-03T21:02:02.600 INFO:teuthology.orchestra.run.smithi032.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.62951.log.gz 2024-08-03T21:02:02.600 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.63395.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73658.log 2024-08-03T21:02:02.600 INFO:teuthology.orchestra.run.smithi032.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.63395.log.gz 2024-08-03T21:02:02.601 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.65169.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89024.log 2024-08-03T21:02:02.601 INFO:teuthology.orchestra.run.smithi032.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.65169.log.gz 2024-08-03T21:02:02.601 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.73658.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47306.log 2024-08-03T21:02:02.601 INFO:teuthology.orchestra.run.smithi032.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.73658.log.gz 2024-08-03T21:02:02.601 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.89024.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37716.log 2024-08-03T21:02:02.601 INFO:teuthology.orchestra.run.smithi032.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.89024.log.gz 2024-08-03T21:02:02.601 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.47306.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74653.log 2024-08-03T21:02:02.602 INFO:teuthology.orchestra.run.smithi032.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.47306.log.gz 2024-08-03T21:02:02.602 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.37716.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37716.log.gz 2024-08-03T21:02:02.602 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87752.log 2024-08-03T21:02:02.602 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.74653.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83724.log 2024-08-03T21:02:02.602 INFO:teuthology.orchestra.run.smithi032.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.74653.log.gz 2024-08-03T21:02:02.602 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.87752.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73016.log 2024-08-03T21:02:02.602 INFO:teuthology.orchestra.run.smithi032.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.87752.log.gz 2024-08-03T21:02:02.603 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.83724.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69196.log 2024-08-03T21:02:02.603 INFO:teuthology.orchestra.run.smithi032.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.83724.log.gz 2024-08-03T21:02:02.603 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.73016.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.73016.log.gz 2024-08-03T21:02:02.603 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60395.log 2024-08-03T21:02:02.603 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.69196.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43614.log 2024-08-03T21:02:02.603 INFO:teuthology.orchestra.run.smithi032.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.69196.log.gz 2024-08-03T21:02:02.603 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.60395.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72736.log 2024-08-03T21:02:02.603 INFO:teuthology.orchestra.run.smithi032.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.60395.log.gz 2024-08-03T21:02:02.603 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.43614.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82781.log 2024-08-03T21:02:02.604 INFO:teuthology.orchestra.run.smithi032.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.43614.log.gz 2024-08-03T21:02:02.604 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.72736.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83159.log 2024-08-03T21:02:02.604 INFO:teuthology.orchestra.run.smithi032.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.72736.log.gz 2024-08-03T21:02:02.604 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.82781.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43280.log 2024-08-03T21:02:02.604 INFO:teuthology.orchestra.run.smithi032.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.82781.log.gz 2024-08-03T21:02:02.604 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.83159.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74046.log 2024-08-03T21:02:02.604 INFO:teuthology.orchestra.run.smithi032.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.83159.log.gz 2024-08-03T21:02:02.605 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.43280.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57366.log 2024-08-03T21:02:02.605 INFO:teuthology.orchestra.run.smithi032.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.43280.log.gz 2024-08-03T21:02:02.605 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.74046.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.82576.log 2024-08-03T21:02:02.605 INFO:teuthology.orchestra.run.smithi032.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.74046.log.gz 2024-08-03T21:02:02.605 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.57366.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42257.log 2024-08-03T21:02:02.605 INFO:teuthology.orchestra.run.smithi032.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.57366.log.gz 2024-08-03T21:02:02.605 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.0.82576.log: 0.0% -- replaced with /var/log/ceph/ceph-client.0.82576.log.gz 2024-08-03T21:02:02.605 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72257.log 2024-08-03T21:02:02.606 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.42257.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60642.log 2024-08-03T21:02:02.606 INFO:teuthology.orchestra.run.smithi032.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.42257.log.gz 2024-08-03T21:02:02.606 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.72257.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66169.log 2024-08-03T21:02:02.606 INFO:teuthology.orchestra.run.smithi032.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.72257.log.gz 2024-08-03T21:02:02.606 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.60642.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49016.log 2024-08-03T21:02:02.606 INFO:teuthology.orchestra.run.smithi032.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.60642.log.gz 2024-08-03T21:02:02.607 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.66169.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58457.log 2024-08-03T21:02:02.607 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.49016.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47688.log 2024-08-03T21:02:02.607 INFO:teuthology.orchestra.run.smithi032.stderr: 92.5% -- replaced with /var/log/ceph/ceph-client.admin.66169.log.gz 2024-08-03T21:02:02.607 INFO:teuthology.orchestra.run.smithi032.stderr: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.49016.log.gz 2024-08-03T21:02:02.607 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.58457.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74924.log 2024-08-03T21:02:02.607 INFO:teuthology.orchestra.run.smithi032.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.58457.log.gz 2024-08-03T21:02:02.607 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.47688.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78851.log 2024-08-03T21:02:02.608 INFO:teuthology.orchestra.run.smithi032.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.47688.log.gz 2024-08-03T21:02:02.608 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.74924.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.74924.log.gz 2024-08-03T21:02:02.608 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75101.log 2024-08-03T21:02:02.608 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.78851.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66895.log 2024-08-03T21:02:02.608 INFO:teuthology.orchestra.run.smithi032.stderr: 90.9% -- replaced with /var/log/ceph/ceph-client.admin.78851.log.gz 2024-08-03T21:02:02.608 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.75101.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66723.log 2024-08-03T21:02:02.608 INFO:teuthology.orchestra.run.smithi032.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.75101.log.gz 2024-08-03T21:02:02.608 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.66895.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73550.log 2024-08-03T21:02:02.609 INFO:teuthology.orchestra.run.smithi032.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.66895.log.gz 2024-08-03T21:02:02.609 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.66723.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63988.log 2024-08-03T21:02:02.609 INFO:teuthology.orchestra.run.smithi032.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.66723.log.gz 2024-08-03T21:02:02.609 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.73550.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82318.log 2024-08-03T21:02:02.609 INFO:teuthology.orchestra.run.smithi032.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.73550.log.gz 2024-08-03T21:02:02.609 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.63988.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86860.log 2024-08-03T21:02:02.609 INFO:teuthology.orchestra.run.smithi032.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.63988.log.gz 2024-08-03T21:02:02.610 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.82318.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64179.log 2024-08-03T21:02:02.610 INFO:teuthology.orchestra.run.smithi032.stderr: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.82318.log.gz 2024-08-03T21:02:02.610 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.86860.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78127.log 2024-08-03T21:02:02.610 INFO:teuthology.orchestra.run.smithi032.stderr: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.86860.log.gz 2024-08-03T21:02:02.611 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.64179.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83463.log 2024-08-03T21:02:02.618 INFO:teuthology.orchestra.run.smithi118.stderr:/var/log/ceph/ceph-mds.d.log: /var/log/ceph/ceph-mon.b.log: 92.1% -- replaced with /var/log/ceph/ceph-mds.d.log.gz 2024-08-03T21:02:02.624 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.78127.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67995.log 2024-08-03T21:02:02.624 INFO:teuthology.orchestra.run.smithi032.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.78127.log.gz 2024-08-03T21:02:02.624 INFO:teuthology.orchestra.run.smithi032.stderr: 92.7% -- replaced with /var/log/ceph/ceph-client.admin.64179.log.gz 2024-08-03T21:02:02.624 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.83463.log: gzip -5 --verbose 87.3% -- -- replaced with /var/log/ceph/ceph-client.admin.83463.log.gz /var/log/ceph/ceph-client.admin.42888.log 2024-08-03T21:02:02.624 INFO:teuthology.orchestra.run.smithi032.stderr: 2024-08-03T21:02:02.625 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.67995.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55901.log 2024-08-03T21:02:02.625 INFO:teuthology.orchestra.run.smithi032.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.67995.log.gz 2024-08-03T21:02:02.625 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.42888.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55635.log 2024-08-03T21:02:02.625 INFO:teuthology.orchestra.run.smithi032.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.42888.log.gz 2024-08-03T21:02:02.625 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.55901.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38221.log 2024-08-03T21:02:02.625 INFO:teuthology.orchestra.run.smithi032.stderr: 92.1% -- replaced with /var/log/ceph/ceph-client.admin.55901.log.gz/var/log/ceph/ceph-client.admin.55635.log: 2024-08-03T21:02:02.625 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46566.log 2024-08-03T21:02:02.625 INFO:teuthology.orchestra.run.smithi032.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.55635.log.gz 2024-08-03T21:02:02.626 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.38221.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81148.log 2024-08-03T21:02:02.626 INFO:teuthology.orchestra.run.smithi032.stderr: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.38221.log.gz 2024-08-03T21:02:02.626 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.46566.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83295.log 2024-08-03T21:02:02.627 INFO:teuthology.orchestra.run.smithi032.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.46566.log.gz 2024-08-03T21:02:02.627 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.81148.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65105.log 2024-08-03T21:02:02.627 INFO:teuthology.orchestra.run.smithi032.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.81148.log.gz 2024-08-03T21:02:02.627 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.83295.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44274.log 2024-08-03T21:02:02.627 INFO:teuthology.orchestra.run.smithi032.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.83295.log.gz 2024-08-03T21:02:02.627 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.65105.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79740.log 2024-08-03T21:02:02.628 INFO:teuthology.orchestra.run.smithi032.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.65105.log.gz 2024-08-03T21:02:02.628 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.44274.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51061.log 2024-08-03T21:02:02.628 INFO:teuthology.orchestra.run.smithi032.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.44274.log.gz 2024-08-03T21:02:02.628 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.79740.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.79740.log.gz 2024-08-03T21:02:02.628 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62311.log 2024-08-03T21:02:02.628 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.51061.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69978.log 2024-08-03T21:02:02.628 INFO:teuthology.orchestra.run.smithi032.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.51061.log.gz 2024-08-03T21:02:02.629 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.62311.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38239.log 2024-08-03T21:02:02.629 INFO:teuthology.orchestra.run.smithi032.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.62311.log.gz 2024-08-03T21:02:02.629 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.69978.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66829.log 2024-08-03T21:02:02.629 INFO:teuthology.orchestra.run.smithi032.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.69978.log.gz 2024-08-03T21:02:02.629 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.38239.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63728.log 2024-08-03T21:02:02.629 INFO:teuthology.orchestra.run.smithi032.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.38239.log.gz 2024-08-03T21:02:02.629 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.66829.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73513.log 2024-08-03T21:02:02.629 INFO:teuthology.orchestra.run.smithi032.stderr: 90.9% -- replaced with /var/log/ceph/ceph-client.admin.66829.log.gz 2024-08-03T21:02:02.629 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.63728.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88196.log 2024-08-03T21:02:02.630 INFO:teuthology.orchestra.run.smithi032.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.63728.log.gz 2024-08-03T21:02:02.630 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.73513.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.67087.log 2024-08-03T21:02:02.630 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.88196.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88533.log 2024-08-03T21:02:02.630 INFO:teuthology.orchestra.run.smithi032.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.88196.log.gz 2024-08-03T21:02:02.630 INFO:teuthology.orchestra.run.smithi032.stderr: 92.5% -- replaced with /var/log/ceph/ceph-client.admin.73513.log.gz 2024-08-03T21:02:02.630 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.0.67087.log: 0.0% -- replaced with /var/log/ceph/ceph-client.0.67087.log.gz 2024-08-03T21:02:02.630 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56613.log 2024-08-03T21:02:02.631 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.88533.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85354.log 2024-08-03T21:02:02.631 INFO:teuthology.orchestra.run.smithi032.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.88533.log.gz 2024-08-03T21:02:02.631 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.56613.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.86127.log 2024-08-03T21:02:02.631 INFO:teuthology.orchestra.run.smithi032.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.56613.log.gz 2024-08-03T21:02:02.631 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.85354.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85308.log 2024-08-03T21:02:02.631 INFO:teuthology.orchestra.run.smithi032.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.85354.log.gz 2024-08-03T21:02:02.631 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.0.86127.log: 0.0% -- replaced with /var/log/ceph/ceph-client.0.86127.log.gz 2024-08-03T21:02:02.631 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72778.log 2024-08-03T21:02:02.632 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.85308.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52206.log 2024-08-03T21:02:02.632 INFO:teuthology.orchestra.run.smithi032.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.85308.log.gz 2024-08-03T21:02:02.632 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.72778.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57721.log 2024-08-03T21:02:02.632 INFO:teuthology.orchestra.run.smithi032.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.72778.log.gz 2024-08-03T21:02:02.639 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.52206.log: gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-08-03T21:02:02.639 INFO:teuthology.orchestra.run.smithi032.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.52206.log.gz 2024-08-03T21:02:02.639 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.57721.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64479.log 2024-08-03T21:02:02.640 INFO:teuthology.orchestra.run.smithi032.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.57721.log.gz 2024-08-03T21:02:02.640 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66787.log 2024-08-03T21:02:02.640 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.64479.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60091.log 2024-08-03T21:02:02.640 INFO:teuthology.orchestra.run.smithi032.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.64479.log.gz 2024-08-03T21:02:02.640 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.66787.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55360.log 2024-08-03T21:02:02.641 INFO:teuthology.orchestra.run.smithi032.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.66787.log.gz 2024-08-03T21:02:02.641 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.60091.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61151.log 2024-08-03T21:02:02.641 INFO:teuthology.orchestra.run.smithi032.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.60091.log.gz 2024-08-03T21:02:02.641 INFO:teuthology.orchestra.run.smithi032.stderr: 87.4% -- replaced with /var/log/ceph/ceph.log.gz 2024-08-03T21:02:02.641 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.55360.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86411.log 2024-08-03T21:02:02.641 INFO:teuthology.orchestra.run.smithi032.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.55360.log.gz 2024-08-03T21:02:02.641 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.61151.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84137.log 2024-08-03T21:02:02.641 INFO:teuthology.orchestra.run.smithi032.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.61151.log.gz 2024-08-03T21:02:02.642 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.86411.log: gzip -5 --verbose 87.7% -- replaced with /var/log/ceph/ceph-client.admin.86411.log.gz 2024-08-03T21:02:02.642 INFO:teuthology.orchestra.run.smithi032.stderr: -- /var/log/ceph/ceph-client.admin.38139.log 2024-08-03T21:02:02.642 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.84137.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.86131.log 2024-08-03T21:02:02.642 INFO:teuthology.orchestra.run.smithi032.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.84137.log.gz 2024-08-03T21:02:02.642 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.38139.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73606.log 2024-08-03T21:02:02.642 INFO:teuthology.orchestra.run.smithi032.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.38139.log.gz 2024-08-03T21:02:02.643 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.0.86131.log: 0.0% -- replaced with /var/log/ceph/ceph-client.0.86131.log.gz 2024-08-03T21:02:02.643 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77841.log 2024-08-03T21:02:02.643 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.73606.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74769.log 2024-08-03T21:02:02.643 INFO:teuthology.orchestra.run.smithi032.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.73606.log.gz 2024-08-03T21:02:02.643 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.77841.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74130.log 2024-08-03T21:02:02.643 INFO:teuthology.orchestra.run.smithi032.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.77841.log.gz 2024-08-03T21:02:02.643 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.74769.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83355.log 2024-08-03T21:02:02.643 INFO:teuthology.orchestra.run.smithi032.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.74769.log.gz 2024-08-03T21:02:02.644 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.74130.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38495.log 2024-08-03T21:02:02.644 INFO:teuthology.orchestra.run.smithi032.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.74130.log.gz 2024-08-03T21:02:02.644 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.83355.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52927.log 2024-08-03T21:02:02.644 INFO:teuthology.orchestra.run.smithi032.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.83355.log.gz 2024-08-03T21:02:02.644 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.38495.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50660.log 2024-08-03T21:02:02.644 INFO:teuthology.orchestra.run.smithi032.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.38495.log.gz 2024-08-03T21:02:02.644 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.52927.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87691.log 2024-08-03T21:02:02.644 INFO:teuthology.orchestra.run.smithi032.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.52927.log.gz 2024-08-03T21:02:02.649 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.50660.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62914.log 2024-08-03T21:02:02.650 INFO:teuthology.orchestra.run.smithi032.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.50660.log.gz 2024-08-03T21:02:02.650 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.87691.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83622.log 2024-08-03T21:02:02.650 INFO:teuthology.orchestra.run.smithi032.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.87691.log.gz 2024-08-03T21:02:02.650 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.62914.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78763.log 2024-08-03T21:02:02.650 INFO:teuthology.orchestra.run.smithi032.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.62914.log.gz 2024-08-03T21:02:02.650 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.83622.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46669.log 2024-08-03T21:02:02.650 INFO:teuthology.orchestra.run.smithi032.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.83622.log.gz 2024-08-03T21:02:02.651 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.78763.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43277.log 2024-08-03T21:02:02.651 INFO:teuthology.orchestra.run.smithi032.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.78763.log.gz 2024-08-03T21:02:02.651 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.46669.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69744.log 2024-08-03T21:02:02.651 INFO:teuthology.orchestra.run.smithi032.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.46669.log.gz 2024-08-03T21:02:02.651 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.43277.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2024-08-03T21:02:02.651 INFO:teuthology.orchestra.run.smithi032.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.43277.log.gz 2024-08-03T21:02:02.651 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.69744.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65253.log 2024-08-03T21:02:02.651 INFO:teuthology.orchestra.run.smithi032.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.69744.log.gz 2024-08-03T21:02:02.652 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73859.log 2024-08-03T21:02:02.652 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.65253.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87396.log 2024-08-03T21:02:02.652 INFO:teuthology.orchestra.run.smithi032.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.65253.log.gz 2024-08-03T21:02:02.652 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.73859.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.81354.log 2024-08-03T21:02:02.652 INFO:teuthology.orchestra.run.smithi032.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.73859.log.gz 2024-08-03T21:02:02.652 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.87396.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68385.log 2024-08-03T21:02:02.652 INFO:teuthology.orchestra.run.smithi032.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.87396.log.gz 2024-08-03T21:02:02.655 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.0.81354.log: 0.0% -- replaced with /var/log/ceph/ceph-client.0.81354.log.gz 2024-08-03T21:02:02.655 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59736.log 2024-08-03T21:02:02.655 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.68385.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83117.log 2024-08-03T21:02:02.655 INFO:teuthology.orchestra.run.smithi032.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.68385.log.gz 2024-08-03T21:02:02.655 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.59736.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79652.log 2024-08-03T21:02:02.656 INFO:teuthology.orchestra.run.smithi032.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.59736.log.gz 2024-08-03T21:02:02.656 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.83117.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52128.log 2024-08-03T21:02:02.656 INFO:teuthology.orchestra.run.smithi032.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.83117.log.gz 2024-08-03T21:02:02.656 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.79652.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.79652.log.gz 2024-08-03T21:02:02.656 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74999.log 2024-08-03T21:02:02.656 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.52128.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.55248.log 2024-08-03T21:02:02.658 INFO:teuthology.orchestra.run.smithi032.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.52128.log.gz 2024-08-03T21:02:02.659 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.74999.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38210.log 2024-08-03T21:02:02.659 INFO:teuthology.orchestra.run.smithi032.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.74999.log.gz 2024-08-03T21:02:02.659 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.0.55248.log: gzip 0.0% -5 -- replaced with /var/log/ceph/ceph-client.0.55248.log.gz --verbose 2024-08-03T21:02:02.659 INFO:teuthology.orchestra.run.smithi032.stderr: -- /var/log/ceph/ceph-client.admin.49308.log 2024-08-03T21:02:02.659 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.38210.log: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.38210.log.gz 2024-08-03T21:02:02.659 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72824.log 2024-08-03T21:02:02.660 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.49308.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58499.log 2024-08-03T21:02:02.660 INFO:teuthology.orchestra.run.smithi032.stderr: 90.9% -- replaced with /var/log/ceph/ceph-client.admin.49308.log.gz 2024-08-03T21:02:02.661 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.72824.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50534.log 2024-08-03T21:02:02.661 INFO:teuthology.orchestra.run.smithi032.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.72824.log.gz 2024-08-03T21:02:02.661 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.58499.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55056.log 2024-08-03T21:02:02.661 INFO:teuthology.orchestra.run.smithi032.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.58499.log.gz 2024-08-03T21:02:02.662 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.50534.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53618.log 2024-08-03T21:02:02.662 INFO:teuthology.orchestra.run.smithi032.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.50534.log.gz 2024-08-03T21:02:02.663 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.55056.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58146.log 2024-08-03T21:02:02.663 INFO:teuthology.orchestra.run.smithi032.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.55056.log.gz 2024-08-03T21:02:02.663 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.53618.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.53618.log.gz 2024-08-03T21:02:02.663 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86550.log 2024-08-03T21:02:02.664 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.58146.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53053.log 2024-08-03T21:02:02.664 INFO:teuthology.orchestra.run.smithi032.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.58146.log.gz 2024-08-03T21:02:02.665 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.86550.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73352.log 2024-08-03T21:02:02.665 INFO:teuthology.orchestra.run.smithi032.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.86550.log.gz 2024-08-03T21:02:02.666 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.53053.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66651.log 2024-08-03T21:02:02.666 INFO:teuthology.orchestra.run.smithi032.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.53053.log.gz 2024-08-03T21:02:02.667 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.73352.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.73352.log.gz 2024-08-03T21:02:02.667 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75877.log 2024-08-03T21:02:02.667 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.66651.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53095.log 2024-08-03T21:02:02.668 INFO:teuthology.orchestra.run.smithi032.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.66651.log.gz 2024-08-03T21:02:02.668 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.75877.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87178.log 2024-08-03T21:02:02.668 INFO:teuthology.orchestra.run.smithi032.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.75877.log.gz 2024-08-03T21:02:02.669 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.53095.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47041.log 2024-08-03T21:02:02.669 INFO:teuthology.orchestra.run.smithi032.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.53095.log.gz 2024-08-03T21:02:02.670 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.87178.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46871.log 2024-08-03T21:02:02.670 INFO:teuthology.orchestra.run.smithi032.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.87178.log.gz 2024-08-03T21:02:02.671 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.47041.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63877.log 2024-08-03T21:02:02.671 INFO:teuthology.orchestra.run.smithi032.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.47041.log.gz 2024-08-03T21:02:02.675 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.46871.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74882.log 2024-08-03T21:02:02.675 INFO:teuthology.orchestra.run.smithi032.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.46871.log.gz 2024-08-03T21:02:02.675 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.63877.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60437.log 2024-08-03T21:02:02.675 INFO:teuthology.orchestra.run.smithi032.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.63877.log.gz 2024-08-03T21:02:02.675 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.74882.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83940.log 2024-08-03T21:02:02.675 INFO:teuthology.orchestra.run.smithi032.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.74882.log.gz 2024-08-03T21:02:02.675 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.60437.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48974.log 2024-08-03T21:02:02.675 INFO:teuthology.orchestra.run.smithi032.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.60437.log.gz 2024-08-03T21:02:02.678 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.83940.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51372.log 2024-08-03T21:02:02.678 INFO:teuthology.orchestra.run.smithi032.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.83940.log.gz 2024-08-03T21:02:02.678 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.48974.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64798.log 2024-08-03T21:02:02.678 INFO:teuthology.orchestra.run.smithi032.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.48974.log.gz 2024-08-03T21:02:02.678 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.51372.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82082.log 2024-08-03T21:02:02.678 INFO:teuthology.orchestra.run.smithi032.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.51372.log.gz 2024-08-03T21:02:02.679 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.64798.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68460.log 2024-08-03T21:02:02.680 INFO:teuthology.orchestra.run.smithi032.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.64798.log.gz 2024-08-03T21:02:02.680 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.82082.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37734.log 2024-08-03T21:02:02.681 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.68460.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80764.log 2024-08-03T21:02:02.681 INFO:teuthology.orchestra.run.smithi032.stderr: 92.4% -- replaced with /var/log/ceph/ceph-client.admin.82082.log.gz 2024-08-03T21:02:02.681 INFO:teuthology.orchestra.run.smithi032.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.68460.log.gz 2024-08-03T21:02:02.682 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.37734.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48670.log 2024-08-03T21:02:02.682 INFO:teuthology.orchestra.run.smithi032.stderr: 76.3% -- replaced with /var/log/ceph/ceph-client.admin.37734.log.gz 2024-08-03T21:02:02.683 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.80764.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.55244.log 2024-08-03T21:02:02.683 INFO:teuthology.orchestra.run.smithi032.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.80764.log.gz 2024-08-03T21:02:02.684 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.48670.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42551.log 2024-08-03T21:02:02.684 INFO:teuthology.orchestra.run.smithi032.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.48670.log.gz 2024-08-03T21:02:02.684 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.0.55244.log: 0.0% -- replaced with /var/log/ceph/ceph-client.0.55244.log.gz 2024-08-03T21:02:02.684 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52284.log 2024-08-03T21:02:02.685 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.42551.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78052.log 2024-08-03T21:02:02.685 INFO:teuthology.orchestra.run.smithi032.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.42551.log.gz 2024-08-03T21:02:02.685 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.52284.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61109.log 2024-08-03T21:02:02.686 INFO:teuthology.orchestra.run.smithi032.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.52284.log.gz 2024-08-03T21:02:02.687 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.78052.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88238.log 2024-08-03T21:02:02.687 INFO:teuthology.orchestra.run.smithi032.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.78052.log.gz 2024-08-03T21:02:02.687 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.61109.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65758.log 2024-08-03T21:02:02.688 INFO:teuthology.orchestra.run.smithi032.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.61109.log.gz 2024-08-03T21:02:02.688 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.88238.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86488.log 2024-08-03T21:02:02.688 INFO:teuthology.orchestra.run.smithi032.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.88238.log.gz 2024-08-03T21:02:02.689 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.65758.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69627.log 2024-08-03T21:02:02.690 INFO:teuthology.orchestra.run.smithi032.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.65758.log.gz 2024-08-03T21:02:02.690 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.86488.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.64671.log 2024-08-03T21:02:02.690 INFO:teuthology.orchestra.run.smithi032.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.86488.log.gz 2024-08-03T21:02:02.691 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.69627.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84534.log 2024-08-03T21:02:02.691 INFO:teuthology.orchestra.run.smithi032.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.69627.log.gz 2024-08-03T21:02:02.691 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.0.64671.log: 0.0% -- replaced with /var/log/ceph/ceph-client.0.64671.log.gz 2024-08-03T21:02:02.692 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-08-03T21:02:02.692 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.84534.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70729.log 2024-08-03T21:02:02.693 INFO:teuthology.orchestra.run.smithi032.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.84534.log.gz 2024-08-03T21:02:02.693 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51147.log 2024-08-03T21:02:02.694 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.70729.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57282.log 2024-08-03T21:02:02.694 INFO:teuthology.orchestra.run.smithi032.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.70729.log.gz 2024-08-03T21:02:02.695 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.51147.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.51147.log.gz 2024-08-03T21:02:02.695 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53282.log 2024-08-03T21:02:02.696 INFO:teuthology.orchestra.run.smithi032.stderr: 89.6% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-08-03T21:02:02.696 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.57282.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69025.log 2024-08-03T21:02:02.696 INFO:teuthology.orchestra.run.smithi032.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.57282.log.gz 2024-08-03T21:02:02.697 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.53282.log: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.53282.log.gz 2024-08-03T21:02:02.697 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55479.log 2024-08-03T21:02:02.699 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.69025.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46314.log 2024-08-03T21:02:02.699 INFO:teuthology.orchestra.run.smithi032.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.69025.log.gz 2024-08-03T21:02:02.699 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.55479.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82175.log 2024-08-03T21:02:02.699 INFO:teuthology.orchestra.run.smithi032.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.55479.log.gz 2024-08-03T21:02:02.700 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.46314.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83898.log 2024-08-03T21:02:02.701 INFO:teuthology.orchestra.run.smithi032.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.46314.log.gz 2024-08-03T21:02:02.701 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.82175.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.82175.log.gz 2024-08-03T21:02:02.701 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63543.log 2024-08-03T21:02:02.702 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.83898.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87438.log 2024-08-03T21:02:02.702 INFO:teuthology.orchestra.run.smithi032.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.83898.log.gz 2024-08-03T21:02:02.703 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.63543.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78010.log 2024-08-03T21:02:02.703 INFO:teuthology.orchestra.run.smithi032.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.63543.log.gz 2024-08-03T21:02:02.703 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.87438.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46627.log 2024-08-03T21:02:02.704 INFO:teuthology.orchestra.run.smithi032.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.87438.log.gz 2024-08-03T21:02:02.704 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.78010.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.78010.log.gz 2024-08-03T21:02:02.704 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43269.log 2024-08-03T21:02:02.705 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.46627.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57661.log 2024-08-03T21:02:02.705 INFO:teuthology.orchestra.run.smithi032.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.46627.log.gz 2024-08-03T21:02:02.706 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.43269.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47478.log 2024-08-03T21:02:02.707 INFO:teuthology.orchestra.run.smithi032.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.43269.log.gz 2024-08-03T21:02:02.707 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.57661.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72664.log 2024-08-03T21:02:02.707 INFO:teuthology.orchestra.run.smithi032.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.57661.log.gz 2024-08-03T21:02:02.708 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.47478.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68072.log 2024-08-03T21:02:02.708 INFO:teuthology.orchestra.run.smithi032.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.47478.log.gz 2024-08-03T21:02:02.708 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.72664.log: gzip 87.2% -- replaced with /var/log/ceph/ceph-client.admin.72664.log.gz 2024-08-03T21:02:02.708 INFO:teuthology.orchestra.run.smithi032.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.43278.log 2024-08-03T21:02:02.709 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.68072.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71552.log 2024-08-03T21:02:02.710 INFO:teuthology.orchestra.run.smithi032.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.68072.log.gz 2024-08-03T21:02:02.710 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.43278.log: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.43278.log.gz 2024-08-03T21:02:02.710 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.67864.log 2024-08-03T21:02:02.711 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.71552.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69767.log 2024-08-03T21:02:02.711 INFO:teuthology.orchestra.run.smithi032.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.71552.log.gz 2024-08-03T21:02:02.712 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.0.67864.log: 0.0% -- replaced with /var/log/ceph/ceph-client.0.67864.log.gz 2024-08-03T21:02:02.712 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64025.log 2024-08-03T21:02:02.713 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.69767.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66493.log 2024-08-03T21:02:02.713 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.64025.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48323.log 2024-08-03T21:02:02.714 INFO:teuthology.orchestra.run.smithi032.stderr: 92.4% -- replaced with /var/log/ceph/ceph-client.admin.69767.log.gz 89.3% 2024-08-03T21:02:02.714 INFO:teuthology.orchestra.run.smithi032.stderr: -- replaced with /var/log/ceph/ceph-client.admin.64025.log.gz 2024-08-03T21:02:02.715 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.66493.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75312.log 2024-08-03T21:02:02.715 INFO:teuthology.orchestra.run.smithi032.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.66493.log.gz 2024-08-03T21:02:02.716 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.48323.log: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.48323.log.gz 2024-08-03T21:02:02.716 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47436.log 2024-08-03T21:02:02.716 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.75312.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79543.log 2024-08-03T21:02:02.717 INFO:teuthology.orchestra.run.smithi032.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.75312.log.gz 2024-08-03T21:02:02.718 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.47436.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66595.log 2024-08-03T21:02:02.718 INFO:teuthology.orchestra.run.smithi032.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.47436.log.gz 2024-08-03T21:02:02.718 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.79543.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85215.log 2024-08-03T21:02:02.718 INFO:teuthology.orchestra.run.smithi032.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.79543.log.gz 2024-08-03T21:02:02.719 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.66595.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47562.log 2024-08-03T21:02:02.719 INFO:teuthology.orchestra.run.smithi032.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.66595.log.gz 2024-08-03T21:02:02.720 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.85215.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77897.log 2024-08-03T21:02:02.721 INFO:teuthology.orchestra.run.smithi032.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.85215.log.gz 2024-08-03T21:02:02.721 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.47562.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.47562.log.gz 2024-08-03T21:02:02.721 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82119.log 2024-08-03T21:02:02.722 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.77897.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85522.log 2024-08-03T21:02:02.722 INFO:teuthology.orchestra.run.smithi032.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.77897.log.gz 2024-08-03T21:02:02.722 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.82119.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47083.log 2024-08-03T21:02:02.723 INFO:teuthology.orchestra.run.smithi032.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.82119.log.gz 2024-08-03T21:02:02.723 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.85522.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81881.log 2024-08-03T21:02:02.724 INFO:teuthology.orchestra.run.smithi032.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.85522.log.gz 2024-08-03T21:02:02.724 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.47083.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.47083.log.gz 2024-08-03T21:02:02.724 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78557.log 2024-08-03T21:02:02.725 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.81881.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68301.log 2024-08-03T21:02:02.725 INFO:teuthology.orchestra.run.smithi032.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.81881.log.gz 2024-08-03T21:02:02.726 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.78557.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59213.log 2024-08-03T21:02:02.726 INFO:teuthology.orchestra.run.smithi032.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.78557.log.gz 2024-08-03T21:02:02.727 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.68301.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61865.log 2024-08-03T21:02:02.727 INFO:teuthology.orchestra.run.smithi032.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.68301.log.gz 2024-08-03T21:02:02.730 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.59213.log: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.59213.log.gz 2024-08-03T21:02:02.730 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.47988.log 2024-08-03T21:02:02.730 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.61865.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43598.log 2024-08-03T21:02:02.731 INFO:teuthology.orchestra.run.smithi032.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.61865.log.gz 2024-08-03T21:02:02.731 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.0.47988.log: 0.0% -- replaced with /var/log/ceph/ceph-client.0.47988.log.gz 2024-08-03T21:02:02.731 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56409.log 2024-08-03T21:02:02.731 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.43598.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63321.log 2024-08-03T21:02:02.731 INFO:teuthology.orchestra.run.smithi032.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.43598.log.gz 2024-08-03T21:02:02.733 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.56409.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53450.log 2024-08-03T21:02:02.733 INFO:teuthology.orchestra.run.smithi032.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.56409.log.gz 2024-08-03T21:02:02.733 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.63321.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57535.log 2024-08-03T21:02:02.733 INFO:teuthology.orchestra.run.smithi032.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.63321.log.gz 2024-08-03T21:02:02.733 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.53450.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43948.log 2024-08-03T21:02:02.733 INFO:teuthology.orchestra.run.smithi032.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.53450.log.gz 2024-08-03T21:02:02.734 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.57535.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70813.log 2024-08-03T21:02:02.735 INFO:teuthology.orchestra.run.smithi032.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.57535.log.gz 2024-08-03T21:02:02.735 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.43948.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65052.log 2024-08-03T21:02:02.735 INFO:teuthology.orchestra.run.smithi032.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.43948.log.gz 2024-08-03T21:02:02.736 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.70813.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51284.log 2024-08-03T21:02:02.736 INFO:teuthology.orchestra.run.smithi032.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.70813.log.gz 2024-08-03T21:02:02.737 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.65052.log: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.65052.log.gz 2024-08-03T21:02:02.737 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79698.log 2024-08-03T21:02:02.737 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.51284.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.49650.log 2024-08-03T21:02:02.737 INFO:teuthology.orchestra.run.smithi032.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.51284.log.gz 2024-08-03T21:02:02.739 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.79698.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44442.log 2024-08-03T21:02:02.739 INFO:teuthology.orchestra.run.smithi032.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.79698.log.gz 2024-08-03T21:02:02.739 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.0.49650.log: 0.0% -- replaced with /var/log/ceph/ceph-client.0.49650.log.gz 2024-08-03T21:02:02.739 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51103.log 2024-08-03T21:02:02.740 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.44442.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71046.log 2024-08-03T21:02:02.740 INFO:teuthology.orchestra.run.smithi032.stderr: 90.9% -- replaced with /var/log/ceph/ceph-client.admin.44442.log.gz 2024-08-03T21:02:02.741 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.51103.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.51103.log.gz 2024-08-03T21:02:02.741 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63099.log 2024-08-03T21:02:02.742 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.71046.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57021.log 2024-08-03T21:02:02.743 INFO:teuthology.orchestra.run.smithi032.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.71046.log.gz 2024-08-03T21:02:02.743 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.63099.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.3.log 2024-08-03T21:02:02.743 INFO:teuthology.orchestra.run.smithi032.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.63099.log.gz 2024-08-03T21:02:02.744 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.57021.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44400.log 2024-08-03T21:02:02.744 INFO:teuthology.orchestra.run.smithi032.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.57021.log.gz 2024-08-03T21:02:02.744 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-osd.3.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62840.log 2024-08-03T21:02:02.745 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.44400.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65449.log 2024-08-03T21:02:02.745 INFO:teuthology.orchestra.run.smithi032.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.44400.log.gz 2024-08-03T21:02:02.746 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.62840.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69154.log 2024-08-03T21:02:02.747 INFO:teuthology.orchestra.run.smithi032.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.62840.log.gz 2024-08-03T21:02:02.748 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.65449.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79464.log 2024-08-03T21:02:02.748 INFO:teuthology.orchestra.run.smithi032.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.65449.log.gz 2024-08-03T21:02:02.750 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.69154.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88364.log 2024-08-03T21:02:02.751 INFO:teuthology.orchestra.run.smithi032.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.69154.log.gz 2024-08-03T21:02:02.751 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.79464.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.79464.log.gz 2024-08-03T21:02:02.751 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62692.log 2024-08-03T21:02:02.751 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.88364.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.2.45371.log 2024-08-03T21:02:02.752 INFO:teuthology.orchestra.run.smithi032.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.88364.log.gz 2024-08-03T21:02:02.753 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.62692.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.62692.log.gz 2024-08-03T21:02:02.754 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52388.log 2024-08-03T21:02:02.754 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.2.45371.log: 0.0% -- replaced with /var/log/ceph/ceph-client.2.45371.log.gz 2024-08-03T21:02:02.754 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71594.log 2024-08-03T21:02:02.754 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.52388.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.52388.log.gz 2024-08-03T21:02:02.754 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.48862.log 2024-08-03T21:02:02.755 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.71594.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76266.log 2024-08-03T21:02:02.756 INFO:teuthology.orchestra.run.smithi032.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.71594.log.gz 2024-08-03T21:02:02.756 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.0.48862.log: 0.0% -- replaced with /var/log/ceph/ceph-client.0.48862.log.gz 2024-08-03T21:02:02.756 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56185.log 2024-08-03T21:02:02.759 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.76266.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58415.log 2024-08-03T21:02:02.759 INFO:teuthology.orchestra.run.smithi032.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.76266.log.gz 2024-08-03T21:02:02.759 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.56185.log: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.56185.log.gz 2024-08-03T21:02:02.759 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87480.log 2024-08-03T21:02:02.759 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.58415.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68525.log 2024-08-03T21:02:02.759 INFO:teuthology.orchestra.run.smithi032.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.58415.log.gz 2024-08-03T21:02:02.760 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.87480.log: gzip -5 87.4% --verbose -- -- replaced with /var/log/ceph/ceph-client.admin.87480.log.gz /var/log/ceph/ceph-client.admin.45067.log 2024-08-03T21:02:02.760 INFO:teuthology.orchestra.run.smithi032.stderr: 2024-08-03T21:02:02.760 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.68525.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.49646.log 2024-08-03T21:02:02.761 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.45067.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.45067.log.gz 2024-08-03T21:02:02.762 INFO:teuthology.orchestra.run.smithi032.stderr: 92.4% -- replaced with /var/log/ceph/ceph-client.admin.68525.log.gz 2024-08-03T21:02:02.762 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46063.log 2024-08-03T21:02:02.762 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.0.49646.log: 0.0% -- replaced with /var/log/ceph/ceph-client.0.49646.log.gz 2024-08-03T21:02:02.763 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51414.log 2024-08-03T21:02:02.764 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.46063.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37892.log 2024-08-03T21:02:02.764 INFO:teuthology.orchestra.run.smithi032.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.46063.log.gz 2024-08-03T21:02:02.765 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.51414.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.51414.log.gz 2024-08-03T21:02:02.765 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83318.log 2024-08-03T21:02:02.766 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.37892.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54003.log 2024-08-03T21:02:02.766 INFO:teuthology.orchestra.run.smithi032.stderr: 85.8% -- replaced with /var/log/ceph/ceph-client.admin.37892.log.gz 2024-08-03T21:02:02.767 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.83318.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66535.log 2024-08-03T21:02:02.767 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.54003.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.54003.log.gz 2024-08-03T21:02:02.768 INFO:teuthology.orchestra.run.smithi032.stderr: 92.5% -- replaced with /var/log/ceph/ceph-client.admin.83318.log.gz 2024-08-03T21:02:02.768 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50744.log 2024-08-03T21:02:02.768 INFO:teuthology.orchestra.run.smithi118.stderr: 91.4% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2024-08-03T21:02:02.769 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.66535.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42762.log 2024-08-03T21:02:02.769 INFO:teuthology.orchestra.run.smithi032.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.66535.log.gz 2024-08-03T21:02:02.770 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.50744.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80259.log 2024-08-03T21:02:02.770 INFO:teuthology.orchestra.run.smithi032.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.50744.log.gz 2024-08-03T21:02:02.771 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.42762.log: 87.6%gzip -- replaced with /var/log/ceph/ceph-client.admin.42762.log.gz 2024-08-03T21:02:02.771 INFO:teuthology.orchestra.run.smithi032.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.59293.log 2024-08-03T21:02:02.772 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.80259.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81923.log 2024-08-03T21:02:02.772 INFO:teuthology.orchestra.run.smithi032.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.80259.log.gz 2024-08-03T21:02:02.772 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.59293.log: 85.4% -- replaced with /var/log/ceph/ceph-client.admin.59293.log.gzgzip 2024-08-03T21:02:02.772 INFO:teuthology.orchestra.run.smithi032.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.42463.log 2024-08-03T21:02:02.773 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.81923.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85041.log 2024-08-03T21:02:02.774 INFO:teuthology.orchestra.run.smithi032.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.81923.log.gz 2024-08-03T21:02:02.774 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.42463.log: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.42463.log.gz 2024-08-03T21:02:02.775 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64413.log 2024-08-03T21:02:02.775 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.85041.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74452.log 2024-08-03T21:02:02.775 INFO:teuthology.orchestra.run.smithi032.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.85041.log.gz 2024-08-03T21:02:02.776 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.64413.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53221.log 2024-08-03T21:02:02.776 INFO:teuthology.orchestra.run.smithi032.stderr: 90.9% -- replaced with /var/log/ceph/ceph-client.admin.64413.log.gz 2024-08-03T21:02:02.777 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.74452.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.74452.log.gz 2024-08-03T21:02:02.777 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73700.log 2024-08-03T21:02:02.778 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.53221.log: gzip -5 87.2% --verbose -- replaced with /var/log/ceph/ceph-client.admin.53221.log.gz -- 2024-08-03T21:02:02.778 INFO:teuthology.orchestra.run.smithi032.stderr: /var/log/ceph/ceph-client.admin.63617.log 2024-08-03T21:02:02.779 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.73700.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67521.log 2024-08-03T21:02:02.779 INFO:teuthology.orchestra.run.smithi032.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.73700.log.gz 2024-08-03T21:02:02.780 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.63617.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66029.log 2024-08-03T21:02:02.780 INFO:teuthology.orchestra.run.smithi032.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.63617.log.gz 2024-08-03T21:02:02.781 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.67521.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60894.log 2024-08-03T21:02:02.781 INFO:teuthology.orchestra.run.smithi032.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.67521.log.gz 2024-08-03T21:02:02.783 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.66029.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58285.log 2024-08-03T21:02:02.783 INFO:teuthology.orchestra.run.smithi032.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.66029.log.gz 2024-08-03T21:02:02.783 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.60894.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74611.log 2024-08-03T21:02:02.783 INFO:teuthology.orchestra.run.smithi032.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.60894.log.gz 2024-08-03T21:02:02.784 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.58285.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.58285.log.gz 2024-08-03T21:02:02.784 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84618.log 2024-08-03T21:02:02.785 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.74611.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56902.log 2024-08-03T21:02:02.785 INFO:teuthology.orchestra.run.smithi032.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.74611.log.gz 2024-08-03T21:02:02.785 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.84618.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.84618.log.gz 2024-08-03T21:02:02.786 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64272.log 2024-08-03T21:02:02.787 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.56902.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70953.log 2024-08-03T21:02:02.787 INFO:teuthology.orchestra.run.smithi032.stderr: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.56902.log.gz 2024-08-03T21:02:02.787 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.64272.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.64272.log.gz 2024-08-03T21:02:02.788 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71869.log 2024-08-03T21:02:02.788 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.70953.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45874.log 2024-08-03T21:02:02.790 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.71869.log: 92.4% -- replaced with /var/log/ceph/ceph-client.admin.70953.log.gz 2024-08-03T21:02:02.790 INFO:teuthology.orchestra.run.smithi032.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.71869.log.gz 2024-08-03T21:02:02.790 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42846.log 2024-08-03T21:02:02.791 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.45874.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63580.log 2024-08-03T21:02:02.791 INFO:teuthology.orchestra.run.smithi032.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.45874.log.gz 2024-08-03T21:02:02.792 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.42846.log: gzip -5 --verbose -- /var/log/ceph/ceph-mds.c.log 2024-08-03T21:02:02.792 INFO:teuthology.orchestra.run.smithi032.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.42846.log.gz 2024-08-03T21:02:02.793 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.63580.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70086.log 2024-08-03T21:02:02.794 INFO:teuthology.orchestra.run.smithi032.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.63580.log.gz 2024-08-03T21:02:02.794 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-mds.c.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.73204.log 2024-08-03T21:02:02.795 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.70086.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61589.log 2024-08-03T21:02:02.795 INFO:teuthology.orchestra.run.smithi032.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.70086.log.gz 2024-08-03T21:02:02.796 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.0.73204.log: 0.0% -- replaced with /var/log/ceph/ceph-client.0.73204.log.gz 2024-08-03T21:02:02.796 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43947.log 2024-08-03T21:02:02.797 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.61589.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85873.log 2024-08-03T21:02:02.797 INFO:teuthology.orchestra.run.smithi032.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.61589.log.gz 2024-08-03T21:02:02.799 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.43947.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87794.log 2024-08-03T21:02:02.799 INFO:teuthology.orchestra.run.smithi032.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.43947.log.gz 2024-08-03T21:02:02.800 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.85873.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50408.log 2024-08-03T21:02:02.800 INFO:teuthology.orchestra.run.smithi032.stderr: 91.2% -- replaced with /var/log/ceph/ceph-client.admin.85873.log.gz 2024-08-03T21:02:02.801 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.87794.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75157.log 2024-08-03T21:02:02.801 INFO:teuthology.orchestra.run.smithi032.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.87794.log.gz 2024-08-03T21:02:02.801 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.50408.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.50408.log.gz 2024-08-03T21:02:02.802 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73471.log 2024-08-03T21:02:02.803 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.75157.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48166.log 2024-08-03T21:02:02.803 INFO:teuthology.orchestra.run.smithi032.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.75157.log.gz 2024-08-03T21:02:02.804 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.73471.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.73471.log.gz 2024-08-03T21:02:02.804 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49165.log 2024-08-03T21:02:02.805 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.48166.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77622.log 2024-08-03T21:02:02.805 INFO:teuthology.orchestra.run.smithi032.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.48166.log.gz 2024-08-03T21:02:02.806 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.49165.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38799.log 2024-08-03T21:02:02.806 INFO:teuthology.orchestra.run.smithi032.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.49165.log.gz 2024-08-03T21:02:02.807 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.77622.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.77622.log.gz 2024-08-03T21:02:02.807 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63284.log 2024-08-03T21:02:02.808 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.38799.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56347.log 2024-08-03T21:02:02.809 INFO:teuthology.orchestra.run.smithi032.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.38799.log.gz 2024-08-03T21:02:02.809 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.63284.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63654.log 2024-08-03T21:02:02.810 INFO:teuthology.orchestra.run.smithi032.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.63284.log.gz 2024-08-03T21:02:02.811 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.56347.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71206.log 2024-08-03T21:02:02.811 INFO:teuthology.orchestra.run.smithi032.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.56347.log.gz 2024-08-03T21:02:02.812 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.63654.log: gzip -5 89.0% --verbose -- -- replaced with /var/log/ceph/ceph-client.admin.63654.log.gz 2024-08-03T21:02:02.812 INFO:teuthology.orchestra.run.smithi032.stderr: /var/log/ceph/ceph-client.admin.43907.log 2024-08-03T21:02:02.813 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.71206.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50618.log 2024-08-03T21:02:02.813 INFO:teuthology.orchestra.run.smithi032.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.71206.log.gz 2024-08-03T21:02:02.814 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.43907.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.43907.log.gz 2024-08-03T21:02:02.814 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86818.log 2024-08-03T21:02:02.815 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.50618.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.50618.log.gz 2024-08-03T21:02:02.815 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58327.log 2024-08-03T21:02:02.816 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69313.log 2024-08-03T21:02:02.817 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.86818.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.86818.log.gz 2024-08-03T21:02:02.817 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.58327.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81683.log 87.7% 2024-08-03T21:02:02.818 INFO:teuthology.orchestra.run.smithi032.stderr: -- replaced with /var/log/ceph/ceph-client.admin.58327.log.gz 2024-08-03T21:02:02.819 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.69313.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73975.log 2024-08-03T21:02:02.819 INFO:teuthology.orchestra.run.smithi032.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.69313.log.gz 2024-08-03T21:02:02.820 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.81683.log: gzip -5 --verbose -- 89.2% /var/log/ceph/ceph-client.admin.74088.log 2024-08-03T21:02:02.820 INFO:teuthology.orchestra.run.smithi032.stderr: -- replaced with /var/log/ceph/ceph-client.admin.81683.log.gz 2024-08-03T21:02:02.821 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.73975.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57156.log 2024-08-03T21:02:02.821 INFO:teuthology.orchestra.run.smithi032.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.73975.log.gz 2024-08-03T21:02:02.822 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.74088.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74381.log 2024-08-03T21:02:02.822 INFO:teuthology.orchestra.run.smithi032.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.74088.log.gz 2024-08-03T21:02:02.823 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.57156.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.57156.log.gz 2024-08-03T21:02:02.823 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84222.log 2024-08-03T21:02:02.824 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.74381.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62617.log 2024-08-03T21:02:02.825 INFO:teuthology.orchestra.run.smithi032.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.74381.log.gz 2024-08-03T21:02:02.825 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.84222.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.84222.log.gz 2024-08-03T21:02:02.825 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84348.log 2024-08-03T21:02:02.827 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.62617.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84786.log 2024-08-03T21:02:02.827 INFO:teuthology.orchestra.run.smithi032.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.62617.log.gz 2024-08-03T21:02:02.828 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.84348.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64959.log 2024-08-03T21:02:02.828 INFO:teuthology.orchestra.run.smithi032.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.84348.log.gz 2024-08-03T21:02:02.829 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.84786.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71248.log 2024-08-03T21:02:02.829 INFO:teuthology.orchestra.run.smithi032.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.84786.log.gz 2024-08-03T21:02:02.830 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.64959.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71940.log 2024-08-03T21:02:02.831 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.71248.log: 92.6% -- replaced with /var/log/ceph/ceph-client.admin.64959.log.gzgzip 2024-08-03T21:02:02.831 INFO:teuthology.orchestra.run.smithi032.stderr: -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2024-08-03T21:02:02.832 INFO:teuthology.orchestra.run.smithi032.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.71248.log.gz 2024-08-03T21:02:02.832 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.71940.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73882.log 2024-08-03T21:02:02.833 INFO:teuthology.orchestra.run.smithi032.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.71940.log.gz 2024-08-03T21:02:02.834 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43098.log 2024-08-03T21:02:02.835 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.73882.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.62057.log 2024-08-03T21:02:02.836 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.43098.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.y.log 2024-08-03T21:02:02.836 INFO:teuthology.orchestra.run.smithi032.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.43098.log.gz 2024-08-03T21:02:02.836 INFO:teuthology.orchestra.run.smithi032.stderr: 92.4% -- replaced with /var/log/ceph/ceph-client.admin.73882.log.gz 2024-08-03T21:02:02.837 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.0.62057.log: 0.0% -- replaced with /var/log/ceph/ceph-client.0.62057.log.gz 2024-08-03T21:02:02.837 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50450.log 2024-08-03T21:02:02.839 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-mgr.y.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87836.log 2024-08-03T21:02:02.840 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.50450.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76164.log 2024-08-03T21:02:02.840 INFO:teuthology.orchestra.run.smithi032.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.50450.log.gz 2024-08-03T21:02:02.841 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.87836.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67333.log 2024-08-03T21:02:02.841 INFO:teuthology.orchestra.run.smithi032.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.87836.log.gz 2024-08-03T21:02:02.842 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.76164.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50839.log 2024-08-03T21:02:02.843 INFO:teuthology.orchestra.run.smithi032.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.76164.log.gz 2024-08-03T21:02:02.844 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.67333.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65716.log 2024-08-03T21:02:02.844 INFO:teuthology.orchestra.run.smithi032.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.67333.log.gz 2024-08-03T21:02:02.845 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.50839.log: 87.5%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52721.log 2024-08-03T21:02:02.845 INFO:teuthology.orchestra.run.smithi032.stderr: -- replaced with /var/log/ceph/ceph-client.admin.50839.log.gz 2024-08-03T21:02:02.846 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.65716.log: 87.1%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66262.log 2024-08-03T21:02:02.847 INFO:teuthology.orchestra.run.smithi032.stderr: -- replaced with /var/log/ceph/ceph-client.admin.65716.log.gz 2024-08-03T21:02:02.848 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.52721.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69374.log 2024-08-03T21:02:02.848 INFO:teuthology.orchestra.run.smithi032.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.52721.log.gz 2024-08-03T21:02:02.849 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.66262.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.66262.log.gz 2024-08-03T21:02:02.849 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37802.log 2024-08-03T21:02:02.851 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.69374.log: gzip 87.5% -- replaced with /var/log/ceph/ceph-client.admin.69374.log.gz 2024-08-03T21:02:02.851 INFO:teuthology.orchestra.run.smithi032.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.62391.log 2024-08-03T21:02:02.851 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.37802.log: 77.8% -- replaced with /var/log/ceph/ceph-client.admin.37802.log.gz 2024-08-03T21:02:02.852 INFO:teuthology.orchestra.run.smithi032.stderr: 94.6% -- replaced with /var/log/ceph/ceph-mgr.y.log.gz 2024-08-03T21:02:02.852 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80848.log 2024-08-03T21:02:02.853 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.62391.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76089.log 2024-08-03T21:02:02.853 INFO:teuthology.orchestra.run.smithi032.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.62391.log.gz 2024-08-03T21:02:02.854 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.80848.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54522.log 2024-08-03T21:02:02.856 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.76089.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64371.log 2024-08-03T21:02:02.856 INFO:teuthology.orchestra.run.smithi032.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.76089.log.gz 2024-08-03T21:02:02.856 INFO:teuthology.orchestra.run.smithi032.stderr: 92.5% -- replaced with /var/log/ceph/ceph-client.admin.80848.log.gz 2024-08-03T21:02:02.857 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.54522.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63210.log 2024-08-03T21:02:02.858 INFO:teuthology.orchestra.run.smithi032.stderr: 90.9% -- replaced with /var/log/ceph/ceph-client.admin.54522.log.gz 2024-08-03T21:02:02.858 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.64371.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67568.log 2024-08-03T21:02:02.859 INFO:teuthology.orchestra.run.smithi032.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.64371.log.gz 2024-08-03T21:02:02.859 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.63210.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62469.log 2024-08-03T21:02:02.860 INFO:teuthology.orchestra.run.smithi032.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.63210.log.gz 2024-08-03T21:02:02.861 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.67568.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2024-08-03T21:02:02.861 INFO:teuthology.orchestra.run.smithi032.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.67568.log.gz 2024-08-03T21:02:02.862 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.62469.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76206.log 2024-08-03T21:02:02.863 INFO:teuthology.orchestra.run.smithi032.stderr: 91.8% -- replaced with /var/log/ceph/ceph-client.admin.62469.log.gz 2024-08-03T21:02:02.863 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37870.log 2024-08-03T21:02:02.864 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.76206.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42663.log 2024-08-03T21:02:02.865 INFO:teuthology.orchestra.run.smithi032.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.76206.log.gz 2024-08-03T21:02:02.866 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.37870.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69490.log 2024-08-03T21:02:02.866 INFO:teuthology.orchestra.run.smithi032.stderr: 78.9% -- replaced with /var/log/ceph/ceph-client.admin.37870.log.gz 2024-08-03T21:02:02.867 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.42663.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.42663.log.gz 2024-08-03T21:02:02.867 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57198.log 2024-08-03T21:02:02.868 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.69490.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.69490.log.gz 2024-08-03T21:02:02.868 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71776.log 2024-08-03T21:02:02.870 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.57198.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65945.log 2024-08-03T21:02:02.870 INFO:teuthology.orchestra.run.smithi032.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.57198.log.gz 2024-08-03T21:02:02.871 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.71776.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51456.log 2024-08-03T21:02:02.873 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.65945.log: 92.5% 87.1% -- replaced with /var/log/ceph/ceph-client.admin.65945.log.gz 2024-08-03T21:02:02.873 INFO:teuthology.orchestra.run.smithi032.stderr: -- replaced with /var/log/ceph/ceph-client.admin.71776.log.gz 2024-08-03T21:02:02.873 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78229.log 2024-08-03T21:02:02.874 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.51456.log: 87.5%gzip -5 --verbose -- /var/log/ceph/ceph-client.0.67083.log 2024-08-03T21:02:02.874 INFO:teuthology.orchestra.run.smithi032.stderr: -- replaced with /var/log/ceph/ceph-client.admin.51456.log.gz 2024-08-03T21:02:02.875 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.78229.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77234.log 2024-08-03T21:02:02.876 INFO:teuthology.orchestra.run.smithi032.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.78229.log.gz 2024-08-03T21:02:02.877 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.0.67083.log: gzip -5 --verbose -- 0.0% /var/log/ceph/ceph-client.admin.54396.log -- replaced with /var/log/ceph/ceph-client.0.67083.log.gz 2024-08-03T21:02:02.877 INFO:teuthology.orchestra.run.smithi032.stderr: 2024-08-03T21:02:02.878 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.77234.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59479.log 2024-08-03T21:02:02.878 INFO:teuthology.orchestra.run.smithi032.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.77234.log.gz 2024-08-03T21:02:02.879 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.54396.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55681.log 2024-08-03T21:02:02.879 INFO:teuthology.orchestra.run.smithi032.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.54396.log.gz 2024-08-03T21:02:02.880 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.59479.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84576.log 2024-08-03T21:02:02.881 INFO:teuthology.orchestra.run.smithi032.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.59479.log.gz 2024-08-03T21:02:02.882 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.55681.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42717.log 2024-08-03T21:02:02.882 INFO:teuthology.orchestra.run.smithi032.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.55681.log.gz 2024-08-03T21:02:02.883 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.84576.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45676.log 2024-08-03T21:02:02.883 INFO:teuthology.orchestra.run.smithi032.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.84576.log.gz 2024-08-03T21:02:02.884 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.42717.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88785.log 2024-08-03T21:02:02.884 INFO:teuthology.orchestra.run.smithi032.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.42717.log.gz 2024-08-03T21:02:02.886 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.45676.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48474.log 2024-08-03T21:02:02.886 INFO:teuthology.orchestra.run.smithi032.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.45676.log.gz 2024-08-03T21:02:02.887 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.88785.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56041.log 2024-08-03T21:02:02.887 INFO:teuthology.orchestra.run.smithi032.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.88785.log.gz 2024-08-03T21:02:02.888 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.48474.log: 87.4%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47125.log 2024-08-03T21:02:02.888 INFO:teuthology.orchestra.run.smithi032.stderr: -- replaced with /var/log/ceph/ceph-client.admin.48474.log.gz 2024-08-03T21:02:02.889 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.56041.log: 87.5%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83664.log 2024-08-03T21:02:02.889 INFO:teuthology.orchestra.run.smithi032.stderr: -- replaced with /var/log/ceph/ceph-client.admin.56041.log.gz 2024-08-03T21:02:02.891 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.47125.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52490.log 2024-08-03T21:02:02.891 INFO:teuthology.orchestra.run.smithi032.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.47125.log.gz 2024-08-03T21:02:02.892 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.83664.log: gzip -5 --verbose -- 87.3% -- replaced with /var/log/ceph/ceph-client.admin.83664.log.gz 2024-08-03T21:02:02.892 INFO:teuthology.orchestra.run.smithi032.stderr: /var/log/ceph/ceph-client.admin.64875.log 2024-08-03T21:02:02.893 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.52490.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.52490.log.gz 2024-08-03T21:02:02.894 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86712.log 2024-08-03T21:02:02.895 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.64875.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.64875.log.gz 2024-08-03T21:02:02.896 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80994.log 2024-08-03T21:02:02.897 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.86712.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.86712.log.gz 2024-08-03T21:02:02.897 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77005.log 2024-08-03T21:02:02.898 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.80994.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47520.log 2024-08-03T21:02:02.898 INFO:teuthology.orchestra.run.smithi032.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.80994.log.gz 2024-08-03T21:02:02.899 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.77005.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.77005.log.gz 2024-08-03T21:02:02.900 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38252.log 2024-08-03T21:02:02.900 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.47520.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.47520.log.gz 2024-08-03T21:02:02.901 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45592.log 2024-08-03T21:02:02.902 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.38252.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58102.log 2024-08-03T21:02:02.902 INFO:teuthology.orchestra.run.smithi032.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.38252.log.gz 2024-08-03T21:02:02.904 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.45592.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42421.log 2024-08-03T21:02:02.904 INFO:teuthology.orchestra.run.smithi032.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.45592.log.gz 2024-08-03T21:02:02.905 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.58102.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59087.log 2024-08-03T21:02:02.905 INFO:teuthology.orchestra.run.smithi032.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.58102.log.gz 2024-08-03T21:02:02.906 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.42421.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83852.log 2024-08-03T21:02:02.906 INFO:teuthology.orchestra.run.smithi032.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.42421.log.gz 2024-08-03T21:02:02.907 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.59087.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45550.log 2024-08-03T21:02:02.908 INFO:teuthology.orchestra.run.smithi032.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.59087.log.gz 2024-08-03T21:02:02.908 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.83852.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.83852.log.gz 2024-08-03T21:02:02.909 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58234.log 2024-08-03T21:02:02.909 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.45550.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.45550.log.gz 2024-08-03T21:02:02.910 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88112.log 2024-08-03T21:02:02.911 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.58234.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61631.log 2024-08-03T21:02:02.912 INFO:teuthology.orchestra.run.smithi032.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.58234.log.gz 2024-08-03T21:02:02.913 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.88112.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52020.log 2024-08-03T21:02:02.913 INFO:teuthology.orchestra.run.smithi032.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.88112.log.gz 2024-08-03T21:02:02.914 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.61631.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.61631.log.gz 2024-08-03T21:02:02.914 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81739.log 2024-08-03T21:02:02.915 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.52020.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43265.log 2024-08-03T21:02:02.916 INFO:teuthology.orchestra.run.smithi032.stderr: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.52020.log.gz 2024-08-03T21:02:02.917 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.81739.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78515.log 2024-08-03T21:02:02.917 INFO:teuthology.orchestra.run.smithi032.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.81739.log.gz 2024-08-03T21:02:02.920 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.43265.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77047.log 2024-08-03T21:02:02.920 INFO:teuthology.orchestra.run.smithi032.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.43265.log.gz 2024-08-03T21:02:02.920 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.78515.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.78515.log.gz 2024-08-03T21:02:02.920 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88827.log 2024-08-03T21:02:02.921 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.77047.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49392.log 2024-08-03T21:02:02.923 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.88827.log: 86.9% 92.4% -- replaced with /var/log/ceph/ceph-client.admin.77047.log.gz 2024-08-03T21:02:02.923 INFO:teuthology.orchestra.run.smithi032.stderr: -- replaced with /var/log/ceph/ceph-client.admin.88827.log.gz 2024-08-03T21:02:02.923 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose --/var/log/ceph/ceph-client.admin.49392.log: /var/log/ceph/ceph-client.admin.59147.log 2024-08-03T21:02:02.924 INFO:teuthology.orchestra.run.smithi032.stderr:gzip 90.9% -- replaced with /var/log/ceph/ceph-client.admin.49392.log.gz 2024-08-03T21:02:02.924 INFO:teuthology.orchestra.run.smithi032.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.84077.log 2024-08-03T21:02:02.926 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.59147.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42930.log 2024-08-03T21:02:02.926 INFO:teuthology.orchestra.run.smithi032.stderr: 90.9% -- replaced with /var/log/ceph/ceph-client.admin.59147.log.gz 2024-08-03T21:02:02.928 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.84077.log: 87.6%gzip -- replaced with /var/log/ceph/ceph-client.admin.84077.log.gz 2024-08-03T21:02:02.928 INFO:teuthology.orchestra.run.smithi032.stderr: -5 --verbose -- /var/log/ceph/ceph-client.0.73208.log 2024-08-03T21:02:02.929 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.42930.log: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.42930.log.gz 2024-08-03T21:02:02.929 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54354.log 2024-08-03T21:02:02.930 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.0.73208.log: 0.0% -- replaced with /var/log/ceph/ceph-client.0.73208.log.gz 2024-08-03T21:02:02.931 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78356.log 2024-08-03T21:02:02.932 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.54354.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87564.log 2024-08-03T21:02:02.932 INFO:teuthology.orchestra.run.smithi032.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.54354.log.gz 2024-08-03T21:02:02.933 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.78356.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64996.log 2024-08-03T21:02:02.933 INFO:teuthology.orchestra.run.smithi032.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.78356.log.gz 2024-08-03T21:02:02.934 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.87564.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44316.log 2024-08-03T21:02:02.934 INFO:teuthology.orchestra.run.smithi032.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.87564.log.gz 2024-08-03T21:02:02.936 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.64996.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.64996.log.gz 2024-08-03T21:02:02.936 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64114.log 2024-08-03T21:02:02.937 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.44316.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.44316.log.gz 2024-08-03T21:02:02.938 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78285.log 2024-08-03T21:02:02.939 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.64114.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59376.log 2024-08-03T21:02:02.939 INFO:teuthology.orchestra.run.smithi032.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.64114.log.gz 2024-08-03T21:02:02.941 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.78285.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.78285.log.gz 2024-08-03T21:02:02.942 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80193.log/var/log/ceph/ceph-client.admin.59376.log: 2024-08-03T21:02:02.942 INFO:teuthology.orchestra.run.smithi032.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.59376.log.gz 2024-08-03T21:02:02.944 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61193.log 2024-08-03T21:02:02.944 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.80193.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45779.log 2024-08-03T21:02:02.945 INFO:teuthology.orchestra.run.smithi032.stderr: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.80193.log.gz 2024-08-03T21:02:02.945 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.61193.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.61193.log.gz 2024-08-03T21:02:02.947 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.48858.log 2024-08-03T21:02:02.947 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.45779.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.45779.log.gz 2024-08-03T21:02:02.948 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77276.log 2024-08-03T21:02:02.948 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.0.48858.log: 0.0%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65781.log 2024-08-03T21:02:02.948 INFO:teuthology.orchestra.run.smithi032.stderr: -- replaced with /var/log/ceph/ceph-client.0.48858.log.gz 2024-08-03T21:02:02.949 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.77276.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46482.log 2024-08-03T21:02:02.949 INFO:teuthology.orchestra.run.smithi032.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.77276.log.gz 2024-08-03T21:02:02.951 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.65781.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51191.log 2024-08-03T21:02:02.952 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.46482.log: 92.5% -- replaced with /var/log/ceph/ceph-client.admin.65781.log.gz 2024-08-03T21:02:02.952 INFO:teuthology.orchestra.run.smithi032.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.46482.log.gz 2024-08-03T21:02:02.953 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80151.log 2024-08-03T21:02:02.953 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.51191.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.51191.log.gz 2024-08-03T21:02:02.953 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79204.log 2024-08-03T21:02:02.954 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.80151.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49242.log 2024-08-03T21:02:02.954 INFO:teuthology.orchestra.run.smithi032.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.80151.log.gz 2024-08-03T21:02:02.955 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.79204.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48261.log 2024-08-03T21:02:02.955 INFO:teuthology.orchestra.run.smithi032.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.79204.log.gz 2024-08-03T21:02:02.957 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.49242.log: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.49242.log.gz 2024-08-03T21:02:02.957 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76322.log 2024-08-03T21:02:02.957 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.48261.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.48261.log.gz 2024-08-03T21:02:02.958 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66418.log 2024-08-03T21:02:02.960 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.76322.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.62053.log 2024-08-03T21:02:02.960 INFO:teuthology.orchestra.run.smithi032.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.76322.log.gz 2024-08-03T21:02:02.961 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.66418.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.66418.log.gz 2024-08-03T21:02:02.961 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83547.log 2024-08-03T21:02:02.962 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.0.62053.log: 0.0% -- replaced with /var/log/ceph/ceph-client.0.62053.log.gz 2024-08-03T21:02:02.963 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46915.log 2024-08-03T21:02:02.964 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.83547.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54228.log 2024-08-03T21:02:02.964 INFO:teuthology.orchestra.run.smithi032.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.83547.log.gz 2024-08-03T21:02:02.965 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.46915.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67412.log 2024-08-03T21:02:02.965 INFO:teuthology.orchestra.run.smithi032.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.46915.log.gz 2024-08-03T21:02:02.967 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.54228.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43605.log 2024-08-03T21:02:02.967 INFO:teuthology.orchestra.run.smithi032.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.54228.log.gz 2024-08-03T21:02:02.968 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.67412.log: gzip 89.3% -5 --verbose -- replaced with /var/log/ceph/ceph-client.admin.67412.log.gz -- 2024-08-03T21:02:02.968 INFO:teuthology.orchestra.run.smithi032.stderr: /var/log/ceph/ceph-client.admin.76229.log 2024-08-03T21:02:02.969 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.43605.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.43605.log.gz 2024-08-03T21:02:02.971 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.76229.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.76755.log 2024-08-03T21:02:02.972 INFO:teuthology.orchestra.run.smithi032.stderr: 92.5% -- replaced with /var/log/ceph/ceph-client.admin.76229.log.gz 2024-08-03T21:02:02.973 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77509.log 2024-08-03T21:02:02.975 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose/var/log/ceph/ceph-client.0.76755.log: -- /var/log/ceph/ceph-client.admin.68968.log 92.8% -- replaced with /var/log/ceph/ceph-mds.c.log.gz 2024-08-03T21:02:02.975 INFO:teuthology.orchestra.run.smithi032.stderr: 2024-08-03T21:02:02.975 INFO:teuthology.orchestra.run.smithi032.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.0.76755.log.gz 2024-08-03T21:02:02.976 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.77509.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80105.log 2024-08-03T21:02:02.976 INFO:teuthology.orchestra.run.smithi032.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.77509.log.gz 2024-08-03T21:02:02.976 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.68968.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.58963.log 2024-08-03T21:02:02.977 INFO:teuthology.orchestra.run.smithi032.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.68968.log.gz 2024-08-03T21:02:02.977 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.80105.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82384.log 2024-08-03T21:02:02.978 INFO:teuthology.orchestra.run.smithi032.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.80105.log.gz 2024-08-03T21:02:02.978 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.0.58963.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47796.log 2024-08-03T21:02:02.979 INFO:teuthology.orchestra.run.smithi032.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.0.58963.log.gz 2024-08-03T21:02:02.980 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.82384.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43276.log 2024-08-03T21:02:02.980 INFO:teuthology.orchestra.run.smithi032.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.82384.log.gz 2024-08-03T21:02:02.981 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.47796.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.47796.log.gz 2024-08-03T21:02:02.981 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78169.log 2024-08-03T21:02:02.982 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.43276.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69397.log 2024-08-03T21:02:02.982 INFO:teuthology.orchestra.run.smithi032.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.43276.log.gz 2024-08-03T21:02:02.983 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.78169.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43952.log 2024-08-03T21:02:02.983 INFO:teuthology.orchestra.run.smithi032.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.78169.log.gz 2024-08-03T21:02:02.984 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.69397.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75270.log 2024-08-03T21:02:02.986 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.43952.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.79105.log 2024-08-03T21:02:02.986 INFO:teuthology.orchestra.run.smithi032.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.43952.log.gz 2024-08-03T21:02:02.986 INFO:teuthology.orchestra.run.smithi032.stderr: 92.4% -- replaced with /var/log/ceph/ceph-client.admin.69397.log.gz 2024-08-03T21:02:02.987 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.75270.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85648.log 2024-08-03T21:02:02.987 INFO:teuthology.orchestra.run.smithi032.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.75270.log.gz 2024-08-03T21:02:02.988 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.0.79105.log: 0.0% -- replaced with /var/log/ceph/ceph-client.0.79105.log.gz 2024-08-03T21:02:02.988 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77664.log 2024-08-03T21:02:02.988 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50702.log 2024-08-03T21:02:02.989 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.85648.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.85648.log.gz 2024-08-03T21:02:02.990 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.77664.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37774.log 2024-08-03T21:02:02.990 INFO:teuthology.orchestra.run.smithi032.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.77664.log.gz 2024-08-03T21:02:02.991 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.50702.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.50702.log.gz 2024-08-03T21:02:02.991 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89066.log 2024-08-03T21:02:02.992 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58625.log 2024-08-03T21:02:02.993 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.37774.log: 77.8% -- replaced with /var/log/ceph/ceph-client.admin.37774.log.gz 2024-08-03T21:02:02.993 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.89066.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72141.log 2024-08-03T21:02:02.993 INFO:teuthology.orchestra.run.smithi032.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.89066.log.gz 2024-08-03T21:02:02.994 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.58625.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.58625.log.gz 2024-08-03T21:02:02.995 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66376.log 2024-08-03T21:02:02.996 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.72141.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43056.log 2024-08-03T21:02:02.996 INFO:teuthology.orchestra.run.smithi032.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.72141.log.gz 2024-08-03T21:02:02.997 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.66376.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.66376.log.gz 2024-08-03T21:02:02.997 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50305.log 2024-08-03T21:02:02.997 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47255.log 2024-08-03T21:02:02.998 INFO:teuthology.orchestra.run.smithi118.stderr: 91.3% -- replaced with /var/log/ceph/ceph-mon.c.log.gz 2024-08-03T21:02:02.998 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.43056.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.43056.log.gz 2024-08-03T21:02:02.999 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.50305.log: 87.1%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84180.log 2024-08-03T21:02:03.000 INFO:teuthology.orchestra.run.smithi032.stderr: -- replaced with /var/log/ceph/ceph-client.admin.50305.log.gz 2024-08-03T21:02:03.000 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.47255.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38082.log 2024-08-03T21:02:03.000 INFO:teuthology.orchestra.run.smithi032.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.47255.log.gz 2024-08-03T21:02:03.001 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.84180.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.84180.log.gz 2024-08-03T21:02:03.001 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60540.log 2024-08-03T21:02:03.002 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.38082.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51242.log 2024-08-03T21:02:03.003 INFO:teuthology.orchestra.run.smithi032.stderr: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.38082.log.gz 2024-08-03T21:02:03.004 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.60540.log: 87.3%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61279.log 2024-08-03T21:02:03.004 INFO:teuthology.orchestra.run.smithi032.stderr: -- replaced with /var/log/ceph/ceph-client.admin.60540.log.gz 2024-08-03T21:02:03.005 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.51242.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81604.log 2024-08-03T21:02:03.005 INFO:teuthology.orchestra.run.smithi032.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.51242.log.gz 2024-08-03T21:02:03.006 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.61279.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43594.log 2024-08-03T21:02:03.006 INFO:teuthology.orchestra.run.smithi032.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.61279.log.gz 2024-08-03T21:02:03.007 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.81604.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.81604.log.gz 2024-08-03T21:02:03.007 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62988.log 2024-08-03T21:02:03.009 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.43594.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2024-08-03T21:02:03.009 INFO:teuthology.orchestra.run.smithi032.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.43594.log.gz 2024-08-03T21:02:03.009 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.62988.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83982.log 2024-08-03T21:02:03.009 INFO:teuthology.orchestra.run.smithi032.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.62988.log.gz 2024-08-03T21:02:03.010 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42804.log 2024-08-03T21:02:03.012 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.83982.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54047.log 2024-08-03T21:02:03.012 INFO:teuthology.orchestra.run.smithi032.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.83982.log.gz 2024-08-03T21:02:03.013 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.42804.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53576.log 2024-08-03T21:02:03.013 INFO:teuthology.orchestra.run.smithi032.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.42804.log.gz 2024-08-03T21:02:03.014 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.54047.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.54047.log.gz 2024-08-03T21:02:03.015 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81082.log 2024-08-03T21:02:03.015 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.53576.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.53576.log.gz 2024-08-03T21:02:03.016 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43612.log 2024-08-03T21:02:03.017 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.81082.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68773.log 2024-08-03T21:02:03.018 INFO:teuthology.orchestra.run.smithi032.stderr: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.81082.log.gz 2024-08-03T21:02:03.018 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.43612.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78809.log 2024-08-03T21:02:03.018 INFO:teuthology.orchestra.run.smithi032.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.43612.log.gz 2024-08-03T21:02:03.019 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.68773.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.68773.log.gz 2024-08-03T21:02:03.020 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60269.log 2024-08-03T21:02:03.020 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.78809.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.78809.log.gz 2024-08-03T21:02:03.021 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68249.log 2024-08-03T21:02:03.022 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.60269.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83075.log 2024-08-03T21:02:03.022 INFO:teuthology.orchestra.run.smithi032.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.60269.log.gz 2024-08-03T21:02:03.023 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.68249.log: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.68249.log.gz 2024-08-03T21:02:03.024 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54438.log 2024-08-03T21:02:03.025 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.83075.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.83075.log.gz 2024-08-03T21:02:03.026 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53179.log 2024-08-03T21:02:03.026 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.54438.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88984.log 2024-08-03T21:02:03.027 INFO:teuthology.orchestra.run.smithi032.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.54438.log.gz 2024-08-03T21:02:03.028 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.53179.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.53179.log.gz 2024-08-03T21:02:03.029 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.88984.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.88984.log.gz 2024-08-03T21:02:03.029 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43014.log 2024-08-03T21:02:03.031 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61417.log 2024-08-03T21:02:03.032 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.43014.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77739.log 2024-08-03T21:02:03.032 INFO:teuthology.orchestra.run.smithi032.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.43014.log.gz 2024-08-03T21:02:03.033 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.61417.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.61417.log.gz 2024-08-03T21:02:03.033 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74840.log 2024-08-03T21:02:03.035 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.77739.log: gzip -5 --verbose 87.3% -- replaced with /var/log/ceph/ceph-client.admin.77739.log.gz 2024-08-03T21:02:03.035 INFO:teuthology.orchestra.run.smithi032.stderr: -- /var/log/ceph/ceph-client.admin.74494.log 2024-08-03T21:02:03.036 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.74840.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.74840.log.gz 2024-08-03T21:02:03.037 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58373.log 2024-08-03T21:02:03.038 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.74494.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76459.log 2024-08-03T21:02:03.038 INFO:teuthology.orchestra.run.smithi032.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.74494.log.gz 2024-08-03T21:02:03.039 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.58373.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46957.log 2024-08-03T21:02:03.039 INFO:teuthology.orchestra.run.smithi032.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.58373.log.gz 2024-08-03T21:02:03.040 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.76459.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81485.log 2024-08-03T21:02:03.040 INFO:teuthology.orchestra.run.smithi032.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.76459.log.gz 2024-08-03T21:02:03.042 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.46957.log: gzip -5 87.6% -- replaced with /var/log/ceph/ceph-client.admin.46957.log.gz 2024-08-03T21:02:03.043 INFO:teuthology.orchestra.run.smithi032.stderr: --verbose -- /var/log/ceph/ceph-client.admin.54270.log 2024-08-03T21:02:03.043 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.81485.log: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.81485.log.gz 2024-08-03T21:02:03.043 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76963.log 2024-08-03T21:02:03.046 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.54270.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37954.log 2024-08-03T21:02:03.046 INFO:teuthology.orchestra.run.smithi032.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.54270.log.gz 2024-08-03T21:02:03.046 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.76963.log: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.76963.log.gz 2024-08-03T21:02:03.046 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87303.log 2024-08-03T21:02:03.047 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.37954.log: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.37954.log.gz 2024-08-03T21:02:03.047 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57240.log 2024-08-03T21:02:03.048 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.87303.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85606.log 2024-08-03T21:02:03.048 INFO:teuthology.orchestra.run.smithi032.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.87303.log.gz 2024-08-03T21:02:03.049 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.57240.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88869.log 2024-08-03T21:02:03.050 INFO:teuthology.orchestra.run.smithi032.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.57240.log.gz 2024-08-03T21:02:03.051 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.85606.log: gzip -5 --verbose -- 87.1% /var/log/ceph/ceph-client.admin.72908.log 2024-08-03T21:02:03.051 INFO:teuthology.orchestra.run.smithi032.stderr: -- replaced with /var/log/ceph/ceph-client.admin.85606.log.gz 2024-08-03T21:02:03.052 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.88869.log: gzip 87.0% -- replaced with /var/log/ceph/ceph-client.admin.88869.log.gz -5 2024-08-03T21:02:03.052 INFO:teuthology.orchestra.run.smithi032.stderr: --verbose -- /var/log/ceph/ceph-client.admin.46230.log 2024-08-03T21:02:03.053 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.72908.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68114.log 2024-08-03T21:02:03.054 INFO:teuthology.orchestra.run.smithi032.stderr: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.72908.log.gz 2024-08-03T21:02:03.054 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.46230.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.46230.log.gz 2024-08-03T21:02:03.055 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75775.log 2024-08-03T21:02:03.056 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.68114.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.68114.log.gz 2024-08-03T21:02:03.056 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55831.log 2024-08-03T21:02:03.057 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.75775.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69107.log 2024-08-03T21:02:03.058 INFO:teuthology.orchestra.run.smithi032.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.75775.log.gz 2024-08-03T21:02:03.059 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.55831.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74713.log 2024-08-03T21:02:03.059 INFO:teuthology.orchestra.run.smithi032.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.55831.log.gz 2024-08-03T21:02:03.060 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.69107.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62543.log 2024-08-03T21:02:03.060 INFO:teuthology.orchestra.run.smithi032.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.69107.log.gz 2024-08-03T21:02:03.061 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.74713.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.74713.log.gz 2024-08-03T21:02:03.062 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72571.log 2024-08-03T21:02:03.062 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.62543.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.62543.log.gz 2024-08-03T21:02:03.063 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70482.log 2024-08-03T21:02:03.064 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.72571.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50977.log 2024-08-03T21:02:03.066 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.70482.log: 86.7% 92.5% -- replaced with /var/log/ceph/ceph-client.admin.72571.log.gz 2024-08-03T21:02:03.066 INFO:teuthology.orchestra.run.smithi032.stderr: -- replaced with /var/log/ceph/ceph-client.admin.70482.log.gz 2024-08-03T21:02:03.067 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5/var/log/ceph/ceph-client.admin.50977.log: --verbose -- /var/log/ceph/ceph-client.0.58967.log 2024-08-03T21:02:03.067 INFO:teuthology.orchestra.run.smithi032.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.50977.log.gz 2024-08-03T21:02:03.068 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77192.log 2024-08-03T21:02:03.069 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.0.58967.log: gzip 0.0% -- replaced with /var/log/ceph/ceph-client.0.58967.log.gz 2024-08-03T21:02:03.069 INFO:teuthology.orchestra.run.smithi032.stderr: -5 --verbose -- /var/log/ceph/ceph-client.0.87119.log 2024-08-03T21:02:03.070 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.77192.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.77192.log.gz 2024-08-03T21:02:03.070 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.44934.log 2024-08-03T21:02:03.071 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.0.87119.log: 0.0% -- replaced with /var/log/ceph/ceph-client.0.87119.log.gz 2024-08-03T21:02:03.072 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70930.log 2024-08-03T21:02:03.073 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.0.44934.log: 0.0% -- replaced with /var/log/ceph/ceph-client.0.44934.log.gz 2024-08-03T21:02:03.073 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68502.log 2024-08-03T21:02:03.074 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.70930.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.70930.log.gz 2024-08-03T21:02:03.075 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46356.log 2024-08-03T21:02:03.076 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.68502.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58667.log 2024-08-03T21:02:03.076 INFO:teuthology.orchestra.run.smithi032.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.68502.log.gz 2024-08-03T21:02:03.077 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.46356.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61673.log 2024-08-03T21:02:03.077 INFO:teuthology.orchestra.run.smithi032.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.46356.log.gz 2024-08-03T21:02:03.078 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.58667.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69932.log 2024-08-03T21:02:03.079 INFO:teuthology.orchestra.run.smithi032.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.58667.log.gz 2024-08-03T21:02:03.080 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.61673.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53324.log 2024-08-03T21:02:03.080 INFO:teuthology.orchestra.run.smithi032.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.61673.log.gz 2024-08-03T21:02:03.081 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.69932.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52533.log 2024-08-03T21:02:03.081 INFO:teuthology.orchestra.run.smithi032.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.69932.log.gz 2024-08-03T21:02:03.082 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.53324.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85816.log 2024-08-03T21:02:03.082 INFO:teuthology.orchestra.run.smithi032.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.53324.log.gz 2024-08-03T21:02:03.083 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.52533.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68731.log 2024-08-03T21:02:03.083 INFO:teuthology.orchestra.run.smithi032.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.52533.log.gz 2024-08-03T21:02:03.085 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.85816.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70888.log 2024-08-03T21:02:03.085 INFO:teuthology.orchestra.run.smithi032.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.85816.log.gz 2024-08-03T21:02:03.086 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.68731.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78691.log 2024-08-03T21:02:03.087 INFO:teuthology.orchestra.run.smithi032.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.68731.log.gz 2024-08-03T21:02:03.087 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.70888.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82865.log 2024-08-03T21:02:03.088 INFO:teuthology.orchestra.run.smithi032.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.70888.log.gz 2024-08-03T21:02:03.088 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.78691.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.76759.log 2024-08-03T21:02:03.089 INFO:teuthology.orchestra.run.smithi032.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.78691.log.gz 2024-08-03T21:02:03.092 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.82865.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62729.log 2024-08-03T21:02:03.093 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.0.76759.log: 0.0% -- replaced with /var/log/ceph/ceph-client.0.76759.log.gz 92.5% -- replaced with /var/log/ceph/ceph-client.admin.82865.log.gz 2024-08-03T21:02:03.093 INFO:teuthology.orchestra.run.smithi032.stderr: 2024-08-03T21:02:03.093 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68343.log 2024-08-03T21:02:03.094 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.62729.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69434.log 2024-08-03T21:02:03.094 INFO:teuthology.orchestra.run.smithi032.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.62729.log.gz 2024-08-03T21:02:03.094 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.68343.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.68343.log.gz 2024-08-03T21:02:03.096 INFO:teuthology.orchestra.run.smithi032.stderr:gzip/var/log/ceph/ceph-client.admin.69434.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.53408.log 2024-08-03T21:02:03.096 INFO:teuthology.orchestra.run.smithi032.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.69434.log.gz 2024-08-03T21:02:03.096 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65389.log 2024-08-03T21:02:03.098 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.53408.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43264.log 2024-08-03T21:02:03.098 INFO:teuthology.orchestra.run.smithi032.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.53408.log.gz 2024-08-03T21:02:03.099 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.65389.log: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.65389.log.gz 2024-08-03T21:02:03.100 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61505.log 2024-08-03T21:02:03.100 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.43264.log: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.43264.log.gz 2024-08-03T21:02:03.101 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68689.log 2024-08-03T21:02:03.102 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.61505.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64325.log 2024-08-03T21:02:03.102 INFO:teuthology.orchestra.run.smithi032.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.61505.log.gz 2024-08-03T21:02:03.104 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.68689.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.68689.log.gz 2024-08-03T21:02:03.105 INFO:teuthology.orchestra.run.smithi032.stderr:gzip/var/log/ceph/ceph-client.admin.64325.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.57805.log 2024-08-03T21:02:03.105 INFO:teuthology.orchestra.run.smithi032.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.64325.log.gz 2024-08-03T21:02:03.105 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50347.log 2024-08-03T21:02:03.107 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.57805.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84744.log 2024-08-03T21:02:03.107 INFO:teuthology.orchestra.run.smithi032.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.57805.log.gz 2024-08-03T21:02:03.108 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.50347.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.50347.log.gz 2024-08-03T21:02:03.109 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38118.log 2024-08-03T21:02:03.109 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.84744.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.84744.log.gz 2024-08-03T21:02:03.110 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88911.log 2024-08-03T21:02:03.111 INFO:teuthology.orchestra.run.smithi118.stderr: 90.8% -- replaced with /var/log/ceph/ceph-mon.b.log.gz 2024-08-03T21:02:03.111 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.38118.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.80447.log 2024-08-03T21:02:03.112 INFO:teuthology.orchestra.run.smithi032.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.38118.log.gz 2024-08-03T21:02:03.113 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.88911.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.88911.log.gz 2024-08-03T21:02:03.114 INFO:teuthology.orchestra.run.smithi032.stderr:gzip/var/log/ceph/ceph-client.0.80447.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.60989.log 0.0% -- replaced with /var/log/ceph/ceph-client.0.80447.log.gz 2024-08-03T21:02:03.114 INFO:teuthology.orchestra.run.smithi032.stderr: 2024-08-03T21:02:03.115 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82228.log 2024-08-03T21:02:03.116 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.60989.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88617.log 2024-08-03T21:02:03.117 INFO:teuthology.orchestra.run.smithi032.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.60989.log.gz 2024-08-03T21:02:03.117 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.82228.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.82228.log.gz 2024-08-03T21:02:03.118 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70405.log 2024-08-03T21:02:03.119 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.88617.log: 87.1%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72389.log 2024-08-03T21:02:03.119 INFO:teuthology.orchestra.run.smithi032.stderr: -- replaced with /var/log/ceph/ceph-client.admin.88617.log.gz 2024-08-03T21:02:03.120 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.70405.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79599.log 2024-08-03T21:02:03.121 INFO:teuthology.orchestra.run.smithi032.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.70405.log.gz 2024-08-03T21:02:03.122 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.72389.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.54780.log 2024-08-03T21:02:03.122 INFO:teuthology.orchestra.run.smithi032.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.72389.log.gz 2024-08-03T21:02:03.123 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.79599.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71711.log 2024-08-03T21:02:03.123 INFO:teuthology.orchestra.run.smithi032.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.79599.log.gz 2024-08-03T21:02:03.124 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.0.54780.log: 0.0%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53366.log 2024-08-03T21:02:03.124 INFO:teuthology.orchestra.run.smithi032.stderr: -- replaced with /var/log/ceph/ceph-client.0.54780.log.gz 2024-08-03T21:02:03.125 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.71711.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47394.log 2024-08-03T21:02:03.126 INFO:teuthology.orchestra.run.smithi032.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.71711.log.gz 2024-08-03T21:02:03.127 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.53366.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.53366.log.gz 2024-08-03T21:02:03.127 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82275.log 2024-08-03T21:02:03.128 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.47394.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.47394.log.gz 2024-08-03T21:02:03.128 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56493.log 2024-08-03T21:02:03.130 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.82275.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47646.log 2024-08-03T21:02:03.130 INFO:teuthology.orchestra.run.smithi032.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.82275.log.gz 2024-08-03T21:02:03.131 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.56493.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43839.log 2024-08-03T21:02:03.132 INFO:teuthology.orchestra.run.smithi032.stderr: 90.9% -- replaced with /var/log/ceph/ceph-client.admin.56493.log.gz 2024-08-03T21:02:03.133 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.47646.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85083.log 2024-08-03T21:02:03.133 INFO:teuthology.orchestra.run.smithi032.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.47646.log.gz 2024-08-03T21:02:03.134 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.43839.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85396.log 2024-08-03T21:02:03.134 INFO:teuthology.orchestra.run.smithi032.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.43839.log.gz 2024-08-03T21:02:03.135 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.85083.log: gzip -5 --verbose 87.4% -- /var/log/ceph/ceph-client.admin.63914.log -- replaced with /var/log/ceph/ceph-client.admin.85083.log.gz 2024-08-03T21:02:03.135 INFO:teuthology.orchestra.run.smithi032.stderr: 2024-08-03T21:02:03.136 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.85396.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71118.log 2024-08-03T21:02:03.137 INFO:teuthology.orchestra.run.smithi032.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.85396.log.gz 2024-08-03T21:02:03.138 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.63914.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49782.log 2024-08-03T21:02:03.138 INFO:teuthology.orchestra.run.smithi032.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.63914.log.gz 2024-08-03T21:02:03.139 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.71118.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46440.log 2024-08-03T21:02:03.139 INFO:teuthology.orchestra.run.smithi032.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.71118.log.gz 2024-08-03T21:02:03.140 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.49782.log: gzip 87.3% -- replaced with /var/log/ceph/ceph-client.admin.49782.log.gz -5 2024-08-03T21:02:03.141 INFO:teuthology.orchestra.run.smithi032.stderr: --verbose -- /var/log/ceph/ceph-client.admin.57763.log 2024-08-03T21:02:03.142 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.46440.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52651.log 2024-08-03T21:02:03.142 INFO:teuthology.orchestra.run.smithi032.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.46440.log.gz 2024-08-03T21:02:03.143 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.57763.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59563.log 2024-08-03T21:02:03.143 INFO:teuthology.orchestra.run.smithi032.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.57763.log.gz 2024-08-03T21:02:03.144 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.52651.log: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.52651.log.gz 2024-08-03T21:02:03.145 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62234.log 2024-08-03T21:02:03.145 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.59563.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.59563.log.gz 2024-08-03T21:02:03.146 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71323.log 2024-08-03T21:02:03.147 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.62234.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.44915.log 2024-08-03T21:02:03.148 INFO:teuthology.orchestra.run.smithi032.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.62234.log.gz 2024-08-03T21:02:03.149 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.71323.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54588.log 2024-08-03T21:02:03.149 INFO:teuthology.orchestra.run.smithi032.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.71323.log.gz 2024-08-03T21:02:03.150 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.0.44915.log: 0.0% -- replaced with /var/log/ceph/ceph-client.0.44915.log.gz 2024-08-03T21:02:03.150 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70621.log 2024-08-03T21:02:03.151 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.54588.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66334.log 2024-08-03T21:02:03.152 INFO:teuthology.orchestra.run.smithi032.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.54588.log.gz 2024-08-03T21:02:03.153 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.70621.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56227.log 2024-08-03T21:02:03.153 INFO:teuthology.orchestra.run.smithi032.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.70621.log.gz 2024-08-03T21:02:03.155 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.66334.log: gzip 87.6% -- replaced with /var/log/ceph/ceph-client.admin.66334.log.gz 2024-08-03T21:02:03.155 INFO:teuthology.orchestra.run.smithi032.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.51330.log 2024-08-03T21:02:03.155 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.56227.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.56227.log.gz 2024-08-03T21:02:03.156 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61375.log 2024-08-03T21:02:03.157 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.51330.log: gzip -5 --verbose -- 87.5% -- replaced with /var/log/ceph/ceph-client.admin.51330.log.gz 2024-08-03T21:02:03.157 INFO:teuthology.orchestra.run.smithi032.stderr: /var/log/ceph/ceph-client.admin.87522.log 2024-08-03T21:02:03.158 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.61375.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.61375.log.gz 2024-08-03T21:02:03.159 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.80451.log 2024-08-03T21:02:03.160 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.87522.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46153.log 2024-08-03T21:02:03.160 INFO:teuthology.orchestra.run.smithi032.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.87522.log.gz 2024-08-03T21:02:03.162 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.0.80451.log: 0.0% -- replaced with /var/log/ceph/ceph-client.0.80451.log.gz 2024-08-03T21:02:03.162 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50137.log 2024-08-03T21:02:03.162 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.46153.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.46153.log.gz 2024-08-03T21:02:03.163 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85564.log 2024-08-03T21:02:03.164 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.50137.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85480.log 2024-08-03T21:02:03.165 INFO:teuthology.orchestra.run.smithi032.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.50137.log.gz 2024-08-03T21:02:03.166 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.85564.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60810.log 2024-08-03T21:02:03.166 INFO:teuthology.orchestra.run.smithi032.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.85564.log.gz 2024-08-03T21:02:03.167 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.85480.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88448.log 2024-08-03T21:02:03.167 INFO:teuthology.orchestra.run.smithi032.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.85480.log.gz 2024-08-03T21:02:03.168 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.60810.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.60810.log.gz 2024-08-03T21:02:03.169 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88701.log 2024-08-03T21:02:03.169 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.88448.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.88448.log.gz 2024-08-03T21:02:03.170 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74325.log 2024-08-03T21:02:03.171 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.88701.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49824.log 2024-08-03T21:02:03.171 INFO:teuthology.orchestra.run.smithi032.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.88701.log.gz 2024-08-03T21:02:03.173 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.74325.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70020.log 2024-08-03T21:02:03.173 INFO:teuthology.orchestra.run.smithi032.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.74325.log.gz 2024-08-03T21:02:03.174 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.49824.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53534.log 2024-08-03T21:02:03.174 INFO:teuthology.orchestra.run.smithi032.stderr: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.49824.log.gz 2024-08-03T21:02:03.175 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.70020.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51624.log 2024-08-03T21:02:03.176 INFO:teuthology.orchestra.run.smithi032.stderr: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.70020.log.gz 2024-08-03T21:02:03.177 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.53534.log: gzip -5 --verbose -- 87.3% -- replaced with /var/log/ceph/ceph-client.admin.53534.log.gz 2024-08-03T21:02:03.177 INFO:teuthology.orchestra.run.smithi032.stderr: /var/log/ceph/ceph-client.admin.66558.log 2024-08-03T21:02:03.178 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.51624.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.51624.log.gz 2024-08-03T21:02:03.178 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67676.log 2024-08-03T21:02:03.179 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.66558.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78917.log 2024-08-03T21:02:03.181 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.67676.log: 92.5% 87.8% -- replaced with /var/log/ceph/ceph-client.admin.67676.log.gz 2024-08-03T21:02:03.182 INFO:teuthology.orchestra.run.smithi032.stderr: -- replaced with /var/log/ceph/ceph-client.admin.66558.log.gz 2024-08-03T21:02:03.182 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68890.log/var/log/ceph/ceph-client.admin.78917.log: 2024-08-03T21:02:03.182 INFO:teuthology.orchestra.run.smithi032.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.78917.log.gz 2024-08-03T21:02:03.183 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71164.log 2024-08-03T21:02:03.184 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.68890.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85774.log 2024-08-03T21:02:03.184 INFO:teuthology.orchestra.run.smithi032.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.68890.log.gz 2024-08-03T21:02:03.186 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.71164.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.71164.log.gz 2024-08-03T21:02:03.186 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51019.log 2024-08-03T21:02:03.187 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.85774.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.85774.log.gz 2024-08-03T21:02:03.187 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64156.log 2024-08-03T21:02:03.188 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.51019.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86772.log 2024-08-03T21:02:03.189 INFO:teuthology.orchestra.run.smithi032.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.51019.log.gz 2024-08-03T21:02:03.190 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.64156.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.64156.log.gz 2024-08-03T21:02:03.190 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80941.log 2024-08-03T21:02:03.192 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.86772.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.86772.log.gz 2024-08-03T21:02:03.192 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52086.log 2024-08-03T21:02:03.193 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.80941.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.80941.log.gz 2024-08-03T21:02:03.193 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.79109.log 2024-08-03T21:02:03.194 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.52086.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63951.log 2024-08-03T21:02:03.194 INFO:teuthology.orchestra.run.smithi032.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.52086.log.gz 2024-08-03T21:02:03.196 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.0.79109.log: gzip 0.0% -- replaced with /var/log/ceph/ceph-client.0.79109.log.gz 2024-08-03T21:02:03.196 INFO:teuthology.orchestra.run.smithi032.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.79246.log 2024-08-03T21:02:03.197 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.63951.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.63951.log.gz 2024-08-03T21:02:03.197 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51732.log 2024-08-03T21:02:03.198 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.79246.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65328.log 2024-08-03T21:02:03.199 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.51732.log: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.79246.log.gz 2024-08-03T21:02:03.200 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.56801.log 2024-08-03T21:02:03.200 INFO:teuthology.orchestra.run.smithi032.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.51732.log.gz 2024-08-03T21:02:03.201 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.65328.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.65328.log.gz 2024-08-03T21:02:03.202 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose/var/log/ceph/ceph-client.0.56801.log: -- /var/log/ceph/ceph-client.admin.58583.log 2024-08-03T21:02:03.202 INFO:teuthology.orchestra.run.smithi032.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.0.56801.log.gz 2024-08-03T21:02:03.203 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43602.log 2024-08-03T21:02:03.205 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.58583.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63766.log 2024-08-03T21:02:03.205 INFO:teuthology.orchestra.run.smithi032.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.58583.log.gz 2024-08-03T21:02:03.205 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.43602.log: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.43602.log.gz 2024-08-03T21:02:03.206 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61323.log 2024-08-03T21:02:03.207 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.63766.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53713.log 2024-08-03T21:02:03.208 INFO:teuthology.orchestra.run.smithi032.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.63766.log.gz 2024-08-03T21:02:03.209 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.61323.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.61323.log.gz 2024-08-03T21:02:03.209 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46999.log 2024-08-03T21:02:03.210 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.53713.log: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.53713.log.gz 2024-08-03T21:02:03.211 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43905.log 2024-08-03T21:02:03.211 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.46999.log: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.46999.log.gz 2024-08-03T21:02:03.212 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79506.log 2024-08-03T21:02:03.213 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.43905.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74205.log 2024-08-03T21:02:03.213 INFO:teuthology.orchestra.run.smithi032.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.43905.log.gz 2024-08-03T21:02:03.216 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.79506.log: gzip -5/var/log/ceph/ceph-client.admin.74205.log: --verbose -- /var/log/ceph/ceph-client.admin.51498.log 2024-08-03T21:02:03.216 INFO:teuthology.orchestra.run.smithi032.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.74205.log.gz 92.5% -- replaced with /var/log/ceph/ceph-client.admin.79506.log.gz 2024-08-03T21:02:03.216 INFO:teuthology.orchestra.run.smithi032.stderr: 2024-08-03T21:02:03.217 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45718.log 2024-08-03T21:02:03.218 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.51498.log: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.51498.log.gz 2024-08-03T21:02:03.219 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60684.log 2024-08-03T21:02:03.220 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.45718.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61463.log 2024-08-03T21:02:03.220 INFO:teuthology.orchestra.run.smithi032.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.45718.log.gz 2024-08-03T21:02:03.221 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.60684.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44580.log 2024-08-03T21:02:03.221 INFO:teuthology.orchestra.run.smithi032.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.60684.log.gz 2024-08-03T21:02:03.222 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.61463.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66104.log 2024-08-03T21:02:03.223 INFO:teuthology.orchestra.run.smithi032.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.61463.log.gz 2024-08-03T21:02:03.224 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.44580.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71982.log 2024-08-03T21:02:03.224 INFO:teuthology.orchestra.run.smithi032.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.44580.log.gz 2024-08-03T21:02:03.225 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.66104.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60600.log 2024-08-03T21:02:03.225 INFO:teuthology.orchestra.run.smithi032.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.66104.log.gz 2024-08-03T21:02:03.226 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.71982.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68193.log 2024-08-03T21:02:03.226 INFO:teuthology.orchestra.run.smithi032.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.71982.log.gz 2024-08-03T21:02:03.227 INFO:teuthology.orchestra.run.smithi118.stderr: 93.8% -- replaced with /var/log/ceph/ceph-osd.7.log.gz 2024-08-03T21:02:03.227 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.60600.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43616.log 2024-08-03T21:02:03.228 INFO:teuthology.orchestra.run.smithi032.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.60600.log.gz 2024-08-03T21:02:03.229 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.68193.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.87115.log 2024-08-03T21:02:03.229 INFO:teuthology.orchestra.run.smithi032.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.68193.log.gz 2024-08-03T21:02:03.230 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.43616.log: gzip 87.5% -- replaced with /var/log/ceph/ceph-client.admin.43616.log.gz 2024-08-03T21:02:03.231 INFO:teuthology.orchestra.run.smithi032.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.61715.log 2024-08-03T21:02:03.231 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.0.87115.log: 0.0% -- replaced with /var/log/ceph/ceph-client.0.87115.log.gz 2024-08-03T21:02:03.231 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77968.log 2024-08-03T21:02:03.233 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.61715.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80588.log 2024-08-03T21:02:03.233 INFO:teuthology.orchestra.run.smithi032.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.61715.log.gz 2024-08-03T21:02:03.234 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.77968.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.77968.log.gz 2024-08-03T21:02:03.235 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50492.log 2024-08-03T21:02:03.236 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.80588.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59521.log 2024-08-03T21:02:03.236 INFO:teuthology.orchestra.run.smithi032.stderr: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.80588.log.gz 2024-08-03T21:02:03.237 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.50492.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63469.log 2024-08-03T21:02:03.237 INFO:teuthology.orchestra.run.smithi032.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.50492.log.gz 2024-08-03T21:02:03.239 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.59521.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.59521.log.gz 2024-08-03T21:02:03.239 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70990.log 2024-08-03T21:02:03.240 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.63469.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.63469.log.gz 2024-08-03T21:02:03.241 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68848.log 2024-08-03T21:02:03.242 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.70990.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48516.log 2024-08-03T21:02:03.242 INFO:teuthology.orchestra.run.smithi032.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.70990.log.gz 2024-08-03T21:02:03.243 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.68848.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50053.log 2024-08-03T21:02:03.243 INFO:teuthology.orchestra.run.smithi032.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.68848.log.gz 2024-08-03T21:02:03.244 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.48516.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.48516.log.gz 2024-08-03T21:02:03.245 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70771.log 2024-08-03T21:02:03.246 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.50053.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.50053.log.gz 2024-08-03T21:02:03.247 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83687.log 2024-08-03T21:02:03.250 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.70771.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.70771.log.gz 2024-08-03T21:02:03.250 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66206.log 2024-08-03T21:02:03.250 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.83687.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57619.log 2024-08-03T21:02:03.251 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.66206.log: 92.4% -- replaced with /var/log/ceph/ceph-client.admin.83687.log.gz 2024-08-03T21:02:03.251 INFO:teuthology.orchestra.run.smithi032.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.66206.log.gz 2024-08-03T21:02:03.251 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59845.log 2024-08-03T21:02:03.252 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.57619.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.57619.log.gz 2024-08-03T21:02:03.252 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47211.log 2024-08-03T21:02:03.254 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.59845.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78635.log 2024-08-03T21:02:03.254 INFO:teuthology.orchestra.run.smithi032.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.59845.log.gz 2024-08-03T21:02:03.254 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.47211.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.47211.log.gz 2024-08-03T21:02:03.255 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51582.log 2024-08-03T21:02:03.256 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.78635.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65412.log 2024-08-03T21:02:03.256 INFO:teuthology.orchestra.run.smithi032.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.78635.log.gz 2024-08-03T21:02:03.257 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.51582.log: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.51582.log.gz 2024-08-03T21:02:03.258 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60353.log 2024-08-03T21:02:03.259 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.65412.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65557.log 2024-08-03T21:02:03.260 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.60353.log: 92.6% -- replaced with /var/log/ceph/ceph-client.admin.65412.log.gz 2024-08-03T21:02:03.261 INFO:teuthology.orchestra.run.smithi032.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.60353.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88575.log 2024-08-03T21:02:03.261 INFO:teuthology.orchestra.run.smithi032.stderr: 2024-08-03T21:02:03.261 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.65557.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.65557.log.gz 2024-08-03T21:02:03.262 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68618.log 2024-08-03T21:02:03.263 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.88575.log: 87.3%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82958.log 2024-08-03T21:02:03.263 INFO:teuthology.orchestra.run.smithi032.stderr: -- replaced with /var/log/ceph/ceph-client.admin.88575.log.gz 2024-08-03T21:02:03.265 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.68618.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65818.log 2024-08-03T21:02:03.265 INFO:teuthology.orchestra.run.smithi032.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.68618.log.gz 2024-08-03T21:02:03.266 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.82958.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.82958.log.gz 2024-08-03T21:02:03.266 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.67868.log 2024-08-03T21:02:03.267 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.65818.log: 89.5%gzip -- replaced with /var/log/ceph/ceph-client.admin.65818.log.gz 2024-08-03T21:02:03.268 INFO:teuthology.orchestra.run.smithi032.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.59972.log 2024-08-03T21:02:03.268 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.0.67868.log: 0.0% -- replaced with /var/log/ceph/ceph-client.0.67868.log.gz 2024-08-03T21:02:03.269 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88154.log 2024-08-03T21:02:03.270 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.59972.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67375.log 2024-08-03T21:02:03.271 INFO:teuthology.orchestra.run.smithi032.stderr: 90.9% -- replaced with /var/log/ceph/ceph-client.admin.59972.log.gz 2024-08-03T21:02:03.271 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.88154.log: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.88154.log.gz 2024-08-03T21:02:03.272 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57324.log 2024-08-03T21:02:03.275 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.67375.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77580.log 2024-08-03T21:02:03.275 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.57324.log: 92.5% -- replaced with /var/log/ceph/ceph-client.admin.67375.log.gz 2024-08-03T21:02:03.275 INFO:teuthology.orchestra.run.smithi032.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.57324.log.gz 2024-08-03T21:02:03.275 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54098.log 2024-08-03T21:02:03.276 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.77580.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.77580.log.gz 2024-08-03T21:02:03.277 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63803.log 2024-08-03T21:02:03.277 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.54098.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.54098.log.gz 2024-08-03T21:02:03.277 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47167.log 2024-08-03T21:02:03.279 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.63803.log: gzip -5 --verbose 89.4% -- replaced with /var/log/ceph/ceph-client.admin.63803.log.gz 2024-08-03T21:02:03.279 INFO:teuthology.orchestra.run.smithi032.stderr: -- /var/log/ceph/ceph-client.admin.85127.log 2024-08-03T21:02:03.280 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.47167.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.47167.log.gz 2024-08-03T21:02:03.281 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70566.log 2024-08-03T21:02:03.282 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.85127.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63062.log 2024-08-03T21:02:03.282 INFO:teuthology.orchestra.run.smithi032.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.85127.log.gz 2024-08-03T21:02:03.283 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.70566.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44358.log 2024-08-03T21:02:03.285 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.63062.log: 92.5% -- replaced with /var/log/ceph/ceph-client.admin.70566.log.gz 2024-08-03T21:02:03.285 INFO:teuthology.orchestra.run.smithi032.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.63062.log.gz 2024-08-03T21:02:03.285 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42379.log 2024-08-03T21:02:03.286 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.44358.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.44358.log.gz 2024-08-03T21:02:03.286 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75933.log 2024-08-03T21:02:03.287 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.42379.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63247.log 2024-08-03T21:02:03.288 INFO:teuthology.orchestra.run.smithi032.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.42379.log.gz 2024-08-03T21:02:03.288 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.75933.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.75933.log.gz 2024-08-03T21:02:03.289 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71407.log 2024-08-03T21:02:03.290 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.63247.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.63247.log.gz 2024-08-03T21:02:03.290 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65641.log 2024-08-03T21:02:03.291 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.71407.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51540.log 2024-08-03T21:02:03.293 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.65641.log: 92.4% -- replaced with /var/log/ceph/ceph-client.admin.71407.log.gz 2024-08-03T21:02:03.294 INFO:teuthology.orchestra.run.smithi032.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.65641.log.gz 2024-08-03T21:02:03.294 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42507.log 2024-08-03T21:02:03.294 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.51540.log: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.51540.log.gz 2024-08-03T21:02:03.295 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74536.log 2024-08-03T21:02:03.296 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.42507.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85438.log 2024-08-03T21:02:03.296 INFO:teuthology.orchestra.run.smithi032.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.42507.log.gz 2024-08-03T21:02:03.297 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.74536.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.74536.log.gz 2024-08-03T21:02:03.299 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5/var/log/ceph/ceph-client.admin.85438.log: --verbose -- /var/log/ceph/ceph-client.admin.72024.log 2024-08-03T21:02:03.299 INFO:teuthology.orchestra.run.smithi032.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.85438.log.gz 2024-08-03T21:02:03.299 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64216.log 2024-08-03T21:02:03.301 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.72024.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55437.log 2024-08-03T21:02:03.301 INFO:teuthology.orchestra.run.smithi032.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.72024.log.gz 2024-08-03T21:02:03.302 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.64216.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53492.log 2024-08-03T21:02:03.303 INFO:teuthology.orchestra.run.smithi032.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.64216.log.gz 2024-08-03T21:02:03.303 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.55437.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.55437.log.gz 2024-08-03T21:02:03.305 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.53492.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.53492.log.gz 2024-08-03T21:02:03.305 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79857.log 2024-08-03T21:02:03.306 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.47984.log 2024-08-03T21:02:03.308 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.79857.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54140.log 2024-08-03T21:02:03.308 INFO:teuthology.orchestra.run.smithi032.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.79857.log.gz 2024-08-03T21:02:03.308 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.0.47984.log: 0.0% -- replaced with /var/log/ceph/ceph-client.0.47984.log.gz 2024-08-03T21:02:03.309 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74270.log 2024-08-03T21:02:03.310 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.54140.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60311.log 2024-08-03T21:02:03.310 INFO:teuthology.orchestra.run.smithi032.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.54140.log.gz 2024-08-03T21:02:03.311 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.74270.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84474.log 2024-08-03T21:02:03.313 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.60311.log: 92.5% 87.5% -- replaced with /var/log/ceph/ceph-client.admin.74270.log.gz 2024-08-03T21:02:03.313 INFO:teuthology.orchestra.run.smithi032.stderr: -- replaced with /var/log/ceph/ceph-client.admin.60311.log.gz 2024-08-03T21:02:03.314 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67291.log 2024-08-03T21:02:03.315 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.84474.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.84474.log.gz 2024-08-03T21:02:03.315 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75489.log 2024-08-03T21:02:03.316 INFO:teuthology.orchestra.run.smithi118.stderr: 93.5% -- replaced with /var/log/ceph/ceph-osd.4.log.gz 2024-08-03T21:02:03.316 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.67291.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.67291.log.gz 2024-08-03T21:02:03.316 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74676.log 2024-08-03T21:02:03.318 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.75489.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80806.log 2024-08-03T21:02:03.318 INFO:teuthology.orchestra.run.smithi032.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.75489.log.gz 2024-08-03T21:02:03.321 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.74676.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75658.log 2024-08-03T21:02:03.321 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.80806.log: 92.6% 87.3% -- replaced with /var/log/ceph/ceph-client.admin.80806.log.gz 2024-08-03T21:02:03.321 INFO:teuthology.orchestra.run.smithi032.stderr: -- replaced with /var/log/ceph/ceph-client.admin.74676.log.gz 2024-08-03T21:02:03.321 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63691.log 2024-08-03T21:02:03.322 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.75658.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.75658.log.gz 2024-08-03T21:02:03.322 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75817.log 2024-08-03T21:02:03.323 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.63691.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84961.log 2024-08-03T21:02:03.323 INFO:teuthology.orchestra.run.smithi032.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.63691.log.gz 2024-08-03T21:02:03.325 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.75817.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.75817.log.gz 2024-08-03T21:02:03.325 INFO:teuthology.orchestra.run.smithi032.stderr:gzip/var/log/ceph/ceph-client.admin.84961.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.72329.log 2024-08-03T21:02:03.326 INFO:teuthology.orchestra.run.smithi032.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.84961.log.gz 2024-08-03T21:02:03.326 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82902.log 2024-08-03T21:02:03.328 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.72329.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38226.log 2024-08-03T21:02:03.328 INFO:teuthology.orchestra.run.smithi032.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.72329.log.gz 2024-08-03T21:02:03.329 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.82902.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.82902.log.gz 2024-08-03T21:02:03.330 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81040.log 2024-08-03T21:02:03.331 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.38226.log: gzip 86.6% -- replaced with /var/log/ceph/ceph-client.admin.38226.log.gz -5 2024-08-03T21:02:03.331 INFO:teuthology.orchestra.run.smithi032.stderr: --verbose -- /var/log/ceph/ceph-client.admin.57980.log 2024-08-03T21:02:03.332 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.81040.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.81040.log.gz 2024-08-03T21:02:03.333 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62766.log 2024-08-03T21:02:03.334 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.57980.log: 87.7%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72548.log 2024-08-03T21:02:03.334 INFO:teuthology.orchestra.run.smithi032.stderr: -- replaced with /var/log/ceph/ceph-client.admin.57980.log.gz 2024-08-03T21:02:03.335 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.62766.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.62766.log.gz 2024-08-03T21:02:03.336 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.72548.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.72548.log.gz 2024-08-03T21:02:03.336 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45634.log 2024-08-03T21:02:03.338 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76567.log 2024-08-03T21:02:03.339 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.45634.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73742.log 2024-08-03T21:02:03.339 INFO:teuthology.orchestra.run.smithi032.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.45634.log.gz 2024-08-03T21:02:03.340 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.76567.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84660.log 2024-08-03T21:02:03.341 INFO:teuthology.orchestra.run.smithi032.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.76567.log.gz 2024-08-03T21:02:03.342 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.73742.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.73742.log.gz 2024-08-03T21:02:03.342 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79782.log 2024-08-03T21:02:03.343 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.84660.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.84660.log.gz 2024-08-03T21:02:03.343 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84432.log 2024-08-03T21:02:03.345 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.79782.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55723.log 2024-08-03T21:02:03.345 INFO:teuthology.orchestra.run.smithi032.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.79782.log.gz 2024-08-03T21:02:03.346 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.84432.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81562.log 2024-08-03T21:02:03.347 INFO:teuthology.orchestra.run.smithi032.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.84432.log.gz 2024-08-03T21:02:03.348 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.55723.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80686.log 2024-08-03T21:02:03.348 INFO:teuthology.orchestra.run.smithi032.stderr: 90.9% -- replaced with /var/log/ceph/ceph-client.admin.55723.log.gz 2024-08-03T21:02:03.349 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.81562.log: gzip 87.2% -- replaced with /var/log/ceph/ceph-client.admin.81562.log.gz 2024-08-03T21:02:03.349 INFO:teuthology.orchestra.run.smithi032.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.46398.log 2024-08-03T21:02:03.350 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.80686.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.80686.log.gz 2024-08-03T21:02:03.350 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77393.log 2024-08-03T21:02:03.352 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.46398.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83011.log 2024-08-03T21:02:03.352 INFO:teuthology.orchestra.run.smithi032.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.46398.log.gz 2024-08-03T21:02:03.353 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.77393.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43920.log 2024-08-03T21:02:03.353 INFO:teuthology.orchestra.run.smithi032.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.77393.log.gz 2024-08-03T21:02:03.354 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.83011.log: 93.7% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2024-08-03T21:02:03.355 INFO:teuthology.orchestra.run.smithi032.stderr: 87.1%gzip -- replaced with /var/log/ceph/ceph-client.admin.83011.log.gz 2024-08-03T21:02:03.355 INFO:teuthology.orchestra.run.smithi032.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.88070.log 2024-08-03T21:02:03.355 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.43920.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.43920.log.gz 2024-08-03T21:02:03.356 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83411.log 2024-08-03T21:02:03.356 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.70274.log 2024-08-03T21:02:03.357 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.88070.log: /var/log/ceph/ceph-client.admin.83411.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.88070.log.gz 2024-08-03T21:02:03.358 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose 87.2% -- -- replaced with /var/log/ceph/ceph-client.admin.83411.log.gz /var/log/ceph/ceph-client.admin.53137.log 2024-08-03T21:02:03.358 INFO:teuthology.orchestra.run.smithi032.stderr: 2024-08-03T21:02:03.359 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.0.70274.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46829.log 2024-08-03T21:02:03.359 INFO:teuthology.orchestra.run.smithi032.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.0.70274.log.gz 2024-08-03T21:02:03.360 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.53137.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.53137.log.gz 2024-08-03T21:02:03.360 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75429.log 2024-08-03T21:02:03.361 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.46829.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85266.log 2024-08-03T21:02:03.361 INFO:teuthology.orchestra.run.smithi032.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.46829.log.gz 2024-08-03T21:02:03.362 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.75429.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.75429.log.gz 2024-08-03T21:02:03.362 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44148.log 2024-08-03T21:02:03.364 INFO:teuthology.orchestra.run.smithi118.stderr: 93.8% -- replaced with /var/log/ceph/ceph-osd.5.log.gz 2024-08-03T21:02:03.364 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.85266.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56083.log 2024-08-03T21:02:03.365 INFO:teuthology.orchestra.run.smithi032.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.85266.log.gz 2024-08-03T21:02:03.365 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.44148.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.44148.log.gz 2024-08-03T21:02:03.365 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84702.log 2024-08-03T21:02:03.365 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.56083.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52765.log 2024-08-03T21:02:03.366 INFO:teuthology.orchestra.run.smithi032.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.56083.log.gz 2024-08-03T21:02:03.367 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.84702.log: 87.7%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88280.log 2024-08-03T21:02:03.367 INFO:teuthology.orchestra.run.smithi032.stderr: -- replaced with /var/log/ceph/ceph-client.admin.84702.log.gz 2024-08-03T21:02:03.368 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.52765.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69804.log 2024-08-03T21:02:03.368 INFO:teuthology.orchestra.run.smithi032.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.52765.log.gz 2024-08-03T21:02:03.369 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.88280.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.88280.log.gz 2024-08-03T21:02:03.369 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79996.log 2024-08-03T21:02:03.370 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78398.log 2024-08-03T21:02:03.370 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.69804.log: /var/log/ceph/ceph-client.admin.79996.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.69804.log.gz 2024-08-03T21:02:03.371 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88491.log 2024-08-03T21:02:03.371 INFO:teuthology.orchestra.run.smithi032.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.79996.log.gz 2024-08-03T21:02:03.372 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.78398.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45508.log 2024-08-03T21:02:03.372 INFO:teuthology.orchestra.run.smithi032.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.78398.log.gz 2024-08-03T21:02:03.374 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.88491.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84306.log 2024-08-03T21:02:03.374 INFO:teuthology.orchestra.run.smithi032.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.88491.log.gz 2024-08-03T21:02:03.375 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.45508.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.45508.log.gz 2024-08-03T21:02:03.375 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60726.log 2024-08-03T21:02:03.375 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.84306.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85732.log 2024-08-03T21:02:03.376 INFO:teuthology.orchestra.run.smithi032.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.84306.log.gz 2024-08-03T21:02:03.377 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.60726.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74247.log 2024-08-03T21:02:03.377 INFO:teuthology.orchestra.run.smithi032.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.60726.log.gz 2024-08-03T21:02:03.378 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.85732.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.85732.log.gz 2024-08-03T21:02:03.378 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62803.log 2024-08-03T21:02:03.379 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.74247.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78192.log 2024-08-03T21:02:03.380 INFO:teuthology.orchestra.run.smithi032.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.74247.log.gz 2024-08-03T21:02:03.380 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.62803.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.62803.log.gz 2024-08-03T21:02:03.381 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38005.log 2024-08-03T21:02:03.381 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.78192.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69585.log 2024-08-03T21:02:03.383 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.38005.log: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.38005.log.gz 2024-08-03T21:02:03.383 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49458.log 2024-08-03T21:02:03.383 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.69585.log: 92.4% -- replaced with /var/log/ceph/ceph-client.admin.78192.log.gz 2024-08-03T21:02:03.384 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52326.log 2024-08-03T21:02:03.384 INFO:teuthology.orchestra.run.smithi032.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.69585.log.gz 2024-08-03T21:02:03.385 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.49458.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.49458.log.gz 2024-08-03T21:02:03.385 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65505.log 2024-08-03T21:02:03.386 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.52326.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72431.log 2024-08-03T21:02:03.386 INFO:teuthology.orchestra.run.smithi032.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.52326.log.gz 2024-08-03T21:02:03.387 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.65505.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.65505.log.gz 2024-08-03T21:02:03.387 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63432.log 2024-08-03T21:02:03.388 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.72431.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56451.log 2024-08-03T21:02:03.388 INFO:teuthology.orchestra.run.smithi032.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.72431.log.gz 2024-08-03T21:02:03.389 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.63432.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.63432.log.gz 2024-08-03T21:02:03.389 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65987.log 2024-08-03T21:02:03.390 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.56451.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77453.log 2024-08-03T21:02:03.390 INFO:teuthology.orchestra.run.smithi032.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.56451.log.gz 2024-08-03T21:02:03.391 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.65987.log: 87.2%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63173.log 2024-08-03T21:02:03.392 INFO:teuthology.orchestra.run.smithi118.stderr: 94.0% -- replaced with /var/log/ceph/ceph-osd.6.log.gz 2024-08-03T21:02:03.392 INFO:teuthology.orchestra.run.smithi032.stderr: -- replaced with /var/log/ceph/ceph-client.admin.65987.log.gz 2024-08-03T21:02:03.393 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.77453.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53793.log 2024-08-03T21:02:03.393 INFO:teuthology.orchestra.run.smithi032.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.77453.log.gz 2024-08-03T21:02:03.394 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.63173.log: 89.3%gzip -- replaced with /var/log/ceph/ceph-client.admin.63173.log.gz -5 2024-08-03T21:02:03.394 INFO:teuthology.orchestra.run.smithi032.stderr: --verbose -- /var/log/ceph/ceph-client.admin.44232.log 2024-08-03T21:02:03.395 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.53793.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59887.log 2024-08-03T21:02:03.396 INFO:teuthology.orchestra.run.smithi032.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.53793.log.gz 2024-08-03T21:02:03.396 INFO:teuthology.orchestra.run.smithi118.stderr: 2024-08-03T21:02:03.396 INFO:teuthology.orchestra.run.smithi118.stderr:real 0m0.867s 2024-08-03T21:02:03.396 INFO:teuthology.orchestra.run.smithi118.stderr:user 0m3.949s 2024-08-03T21:02:03.396 INFO:teuthology.orchestra.run.smithi118.stderr:sys 0m0.445s 2024-08-03T21:02:03.397 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.44232.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.44232.log.gz 2024-08-03T21:02:03.397 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43615.log 2024-08-03T21:02:03.397 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose --/var/log/ceph/ceph-client.admin.59887.log: /var/log/ceph/ceph-client.admin.46524.log 2024-08-03T21:02:03.398 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.43615.log: 90.9% -- replaced with /var/log/ceph/ceph-client.admin.59887.log.gz 2024-08-03T21:02:03.398 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68562.log 2024-08-03T21:02:03.398 INFO:teuthology.orchestra.run.smithi032.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.43615.log.gz 2024-08-03T21:02:03.399 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.46524.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86658.log 2024-08-03T21:02:03.399 INFO:teuthology.orchestra.run.smithi032.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.46524.log.gz 2024-08-03T21:02:03.400 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.68562.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.68562.log.gz 2024-08-03T21:02:03.400 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54990.log 2024-08-03T21:02:03.401 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.86658.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53916.log 2024-08-03T21:02:03.402 INFO:teuthology.orchestra.run.smithi032.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.86658.log.gz 2024-08-03T21:02:03.403 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.54990.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61235.log 2024-08-03T21:02:03.403 INFO:teuthology.orchestra.run.smithi032.stderr: 90.9% -- replaced with /var/log/ceph/ceph-client.admin.54990.log.gz 2024-08-03T21:02:03.404 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.53916.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.53916.log.gz 2024-08-03T21:02:03.404 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55789.log 2024-08-03T21:02:03.405 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.61235.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50221.log 2024-08-03T21:02:03.405 INFO:teuthology.orchestra.run.smithi032.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.61235.log.gz 2024-08-03T21:02:03.406 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.55789.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77140.log 2024-08-03T21:02:03.406 INFO:teuthology.orchestra.run.smithi032.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.55789.log.gz 2024-08-03T21:02:03.407 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.50221.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.50221.log.gz 2024-08-03T21:02:03.407 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79344.log 2024-08-03T21:02:03.408 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67468.log 2024-08-03T21:02:03.409 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.77140.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.77140.log.gz 2024-08-03T21:02:03.409 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.79344.log: gzip -5 --verbose -- 87.5% /var/log/ceph/ceph-client.admin.38035.log 2024-08-03T21:02:03.410 INFO:teuthology.orchestra.run.smithi032.stderr: -- replaced with /var/log/ceph/ceph-client.admin.79344.log.gz 2024-08-03T21:02:03.411 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.67468.log: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.67468.log.gz 2024-08-03T21:02:03.411 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82059.log 2024-08-03T21:02:03.412 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.38035.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68931.log 2024-08-03T21:02:03.412 INFO:teuthology.orchestra.run.smithi032.stderr: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.38035.log.gz 2024-08-03T21:02:03.413 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.82059.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43913.log 2024-08-03T21:02:03.413 INFO:teuthology.orchestra.run.smithi032.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.82059.log.gz 2024-08-03T21:02:03.414 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.68931.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76886.log 2024-08-03T21:02:03.415 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.43913.log: 92.4% -- replaced with /var/log/ceph/ceph-client.admin.68931.log.gz 2024-08-03T21:02:03.416 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.51924.log 2024-08-03T21:02:03.416 INFO:teuthology.orchestra.run.smithi032.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.43913.log.gz 2024-08-03T21:02:03.417 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.76886.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55999.log 2024-08-03T21:02:03.417 INFO:teuthology.orchestra.run.smithi032.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.76886.log.gz 2024-08-03T21:02:03.418 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.0.51924.log: 0.0% -- replaced with /var/log/ceph/ceph-client.0.51924.log.gz 2024-08-03T21:02:03.418 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45832.log 2024-08-03T21:02:03.419 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.55999.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75840.log 2024-08-03T21:02:03.419 INFO:teuthology.orchestra.run.smithi032.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.55999.log.gz 2024-08-03T21:02:03.420 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.45832.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.45832.log.gz 2024-08-03T21:02:03.420 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57114.log 2024-08-03T21:02:03.421 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.75840.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77416.log 2024-08-03T21:02:03.425 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.57114.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.57114.log.gz 2024-08-03T21:02:03.425 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47604.log 2024-08-03T21:02:03.425 INFO:teuthology.orchestra.run.smithi032.stderr: 92.5% -- replaced with /var/log/ceph/ceph-client.admin.75840.log.gz 2024-08-03T21:02:03.425 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61799.log 2024-08-03T21:02:03.425 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.77416.log: /var/log/ceph/ceph-client.admin.47604.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.47604.log.gz 2024-08-03T21:02:03.426 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69860.log 2024-08-03T21:02:03.426 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- 92.5% /var/log/ceph/ceph-client.admin.80546.log 2024-08-03T21:02:03.426 INFO:teuthology.orchestra.run.smithi032.stderr: -- replaced with /var/log/ceph/ceph-client.admin.77416.log.gz 2024-08-03T21:02:03.427 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.61799.log: /var/log/ceph/ceph-client.admin.69860.log: 90.9% -- replaced with /var/log/ceph/ceph-client.admin.61799.log.gz 2024-08-03T21:02:03.427 INFO:teuthology.orchestra.run.smithi032.stderr: 87.2%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80052.log 2024-08-03T21:02:03.427 INFO:teuthology.orchestra.run.smithi032.stderr: -- replaced with /var/log/ceph/ceph-client.admin.69860.log.gz 2024-08-03T21:02:03.428 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.80546.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46272.log 2024-08-03T21:02:03.428 INFO:teuthology.orchestra.run.smithi032.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.80546.log.gz 2024-08-03T21:02:03.429 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.80052.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75700.log 2024-08-03T21:02:03.429 INFO:teuthology.orchestra.run.smithi032.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.80052.log.gz 2024-08-03T21:02:03.430 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.46272.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.46272.log.gz 2024-08-03T21:02:03.430 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76047.log 2024-08-03T21:02:03.431 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.75700.log: gzip -5 --verbose -- /var/log/ceph/ceph-mds.a.log 2024-08-03T21:02:03.432 INFO:teuthology.orchestra.run.smithi032.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.75700.log.gz 2024-08-03T21:02:03.432 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.76047.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.76047.log.gz 2024-08-03T21:02:03.433 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71500.log 2024-08-03T21:02:03.433 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-mds.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82704.log 2024-08-03T21:02:03.435 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.71500.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.71500.log.gz 2024-08-03T21:02:03.435 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.51920.log 2024-08-03T21:02:03.436 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.82704.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81792.log 2024-08-03T21:02:03.436 INFO:teuthology.orchestra.run.smithi032.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.82704.log.gz 2024-08-03T21:02:03.437 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.0.51920.log: 0.0% -- replaced with /var/log/ceph/ceph-client.0.51920.log.gz 2024-08-03T21:02:03.437 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88406.log 2024-08-03T21:02:03.438 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.81792.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.81792.log.gz 2024-08-03T21:02:03.438 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63025.log 2024-08-03T21:02:03.440 INFO:teuthology.orchestra.run.smithi032.stderr:gzip/var/log/ceph/ceph-client.admin.88406.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.79959.log 2024-08-03T21:02:03.440 INFO:teuthology.orchestra.run.smithi032.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.88406.log.gz 2024-08-03T21:02:03.441 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.63025.log: gzip 89.1% -5 -- replaced with /var/log/ceph/ceph-client.admin.63025.log.gz --verbose 2024-08-03T21:02:03.442 INFO:teuthology.orchestra.run.smithi032.stderr: -- /var/log/ceph/ceph-client.admin.88322.log 2024-08-03T21:02:03.442 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.79959.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56143.log 2024-08-03T21:02:03.444 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.88322.log: 87.4% 92.5% -- replaced with /var/log/ceph/ceph-client.admin.88322.log.gz -- replaced with /var/log/ceph/ceph-client.admin.79959.log.gz 2024-08-03T21:02:03.444 INFO:teuthology.orchestra.run.smithi032.stderr: 2024-08-03T21:02:03.444 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37752.log 2024-08-03T21:02:03.445 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.56143.log: 87.5%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80885.log 2024-08-03T21:02:03.445 INFO:teuthology.orchestra.run.smithi032.stderr: -- replaced with /var/log/ceph/ceph-client.admin.56143.log.gz 2024-08-03T21:02:03.447 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.37752.log: gzip -5 --verbose 77.6% -- replaced with /var/log/ceph/ceph-client.admin.37752.log.gz 2024-08-03T21:02:03.447 INFO:teuthology.orchestra.run.smithi032.stderr: -- /var/log/ceph/ceph-client.admin.60227.log 2024-08-03T21:02:03.448 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.80885.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.80885.log.gz 2024-08-03T21:02:03.448 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51666.log 2024-08-03T21:02:03.449 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.60227.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.60227.log.gz 2024-08-03T21:02:03.449 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82823.log 2024-08-03T21:02:03.454 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63358.log 2024-08-03T21:02:03.455 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.51666.log: /var/log/ceph/ceph-client.admin.82823.log: 90.9% -- replaced with /var/log/ceph/ceph-client.admin.51666.log.gz 2024-08-03T21:02:03.455 INFO:teuthology.orchestra.run.smithi032.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.82823.log.gz 2024-08-03T21:02:03.455 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48420.log 2024-08-03T21:02:03.455 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.63358.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.63358.log.gz 2024-08-03T21:02:03.455 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54480.log 2024-08-03T21:02:03.455 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77781.log 2024-08-03T21:02:03.456 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.48420.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.48420.log.gz 2024-08-03T21:02:03.456 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.54480.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.54480.log.gz 2024-08-03T21:02:03.456 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63136.log 2024-08-03T21:02:03.457 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.77781.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.77781.log.gz 2024-08-03T21:02:03.457 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46749.log 2024-08-03T21:02:03.458 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.31696.log 2024-08-03T21:02:03.459 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.63136.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.63136.log.gz 2024-08-03T21:02:03.459 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.46749.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.46749.log.gz 2024-08-03T21:02:03.461 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42161.log 2024-08-03T21:02:03.461 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph.tmp-client.admin.31696.log: 0.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.31696.log.gz 2024-08-03T21:02:03.462 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57450.log 2024-08-03T21:02:03.462 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50095.log/var/log/ceph/ceph-client.admin.42161.log: 2024-08-03T21:02:03.463 INFO:teuthology.orchestra.run.smithi032.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.42161.log.gz 2024-08-03T21:02:03.463 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.57450.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.57450.log.gz 2024-08-03T21:02:03.464 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72974.log 2024-08-03T21:02:03.465 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.50095.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.50095.log.gz 2024-08-03T21:02:03.465 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73817.log 2024-08-03T21:02:03.466 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.72974.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.72974.log.gz 2024-08-03T21:02:03.467 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49726.log 2024-08-03T21:02:03.467 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.73817.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47348.log 2024-08-03T21:02:03.468 INFO:teuthology.orchestra.run.smithi032.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.73817.log.gz 2024-08-03T21:02:03.469 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.49726.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.49726.log.gz 2024-08-03T21:02:03.470 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84881.log 2024-08-03T21:02:03.470 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.47348.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.47348.log.gz 2024-08-03T21:02:03.471 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78440.log 2024-08-03T21:02:03.472 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5/var/log/ceph/ceph-client.admin.84881.log: --verbose -- /var/log/ceph/ceph-client.admin.58060.log 2024-08-03T21:02:03.472 INFO:teuthology.orchestra.run.smithi032.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.84881.log.gz 2024-08-03T21:02:03.473 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.78440.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.78440.log.gz 2024-08-03T21:02:03.474 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.54776.log 2024-08-03T21:02:03.475 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.58060.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.58060.log.gz 2024-08-03T21:02:03.475 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69238.log 2024-08-03T21:02:03.476 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60185.log 2024-08-03T21:02:03.476 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.0.54776.log: 0.0% -- replaced with /var/log/ceph/ceph-client.0.54776.log.gz 2024-08-03T21:02:03.477 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.69238.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.69238.log.gz 2024-08-03T21:02:03.478 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75616.log 2024-08-03T21:02:03.478 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.60185.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.60185.log.gz 2024-08-03T21:02:03.479 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72099.log 2024-08-03T21:02:03.480 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73429.log 2024-08-03T21:02:03.481 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.75616.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.75616.log.gz 2024-08-03T21:02:03.481 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.72099.log: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.72099.log.gz 2024-08-03T21:02:03.482 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53959.log 2024-08-03T21:02:03.482 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.73429.log: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.73429.log.gz 2024-08-03T21:02:03.483 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76501.log 2024-08-03T21:02:03.484 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88743.log 2024-08-03T21:02:03.484 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.53959.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.53959.log.gz 2024-08-03T21:02:03.485 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.76501.log: 90.9% -- replaced with /var/log/ceph/ceph-client.admin.76501.log.gz 2024-08-03T21:02:03.486 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67214.log 2024-08-03T21:02:03.487 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.88743.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.88743.log.gz 2024-08-03T21:02:03.487 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72201.log 2024-08-03T21:02:03.488 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55540.log 2024-08-03T21:02:03.489 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.67214.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.67214.log.gz 2024-08-03T21:02:03.489 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.72201.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.72201.log.gz 2024-08-03T21:02:03.490 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62580.log 2024-08-03T21:02:03.491 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.55540.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.55540.log.gz 2024-08-03T21:02:03.491 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54312.log 2024-08-03T21:02:03.492 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47730.log 2024-08-03T21:02:03.493 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.62580.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.62580.log.gz 2024-08-03T21:02:03.494 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.54312.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.54312.log.gz 2024-08-03T21:02:03.494 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49960.log 2024-08-03T21:02:03.495 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.47730.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63506.log 2024-08-03T21:02:03.496 INFO:teuthology.orchestra.run.smithi032.stderr: 90.9% -- replaced with /var/log/ceph/ceph-client.admin.47730.log.gz 2024-08-03T21:02:03.497 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.49960.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59798.log 2024-08-03T21:02:03.497 INFO:teuthology.orchestra.run.smithi032.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.49960.log.gz 2024-08-03T21:02:03.498 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.63506.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.63506.log.gz 2024-08-03T21:02:03.499 INFO:teuthology.orchestra.run.smithi032.stderr:gzip/var/log/ceph/ceph-client.admin.59798.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.57408.log 2024-08-03T21:02:03.500 INFO:teuthology.orchestra.run.smithi032.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.59798.log.gz 2024-08-03T21:02:03.500 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55593.log 2024-08-03T21:02:03.501 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.57408.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.57408.log.gz 2024-08-03T21:02:03.501 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43279.log 2024-08-03T21:02:03.503 INFO:teuthology.orchestra.run.smithi032.stderr:gzip/var/log/ceph/ceph-client.admin.55593.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.63840.log 2024-08-03T21:02:03.503 INFO:teuthology.orchestra.run.smithi032.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.55593.log.gz 2024-08-03T21:02:03.504 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.43279.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.43279.log.gz 2024-08-03T21:02:03.504 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75387.log 2024-08-03T21:02:03.505 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.63840.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.63840.log.gz 2024-08-03T21:02:03.506 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70524.log 2024-08-03T21:02:03.507 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.75387.log: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.75387.log.gz 2024-08-03T21:02:03.507 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81646.log 2024-08-03T21:02:03.508 INFO:teuthology.orchestra.run.smithi032.stderr:gzip/var/log/ceph/ceph-client.admin.70524.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.48346.log 2024-08-03T21:02:03.508 INFO:teuthology.orchestra.run.smithi032.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.70524.log.gz 2024-08-03T21:02:03.512 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.81646.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81839.log 2024-08-03T21:02:03.512 INFO:teuthology.orchestra.run.smithi032.stderr: 92.5% -- replaced with /var/log/ceph/ceph-client.admin.81646.log.gz 2024-08-03T21:02:03.512 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.48346.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71753.log 2024-08-03T21:02:03.512 INFO:teuthology.orchestra.run.smithi032.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.48346.log.gz 2024-08-03T21:02:03.513 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.81839.log: gzip 87.4% -5 --verbose -- replaced with /var/log/ceph/ceph-client.admin.81839.log.gz -- 2024-08-03T21:02:03.513 INFO:teuthology.orchestra.run.smithi032.stderr: /var/log/ceph/ceph-client.0.56805.log 2024-08-03T21:02:03.514 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.71753.log: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.71753.log.gz 2024-08-03T21:02:03.514 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52969.log 2024-08-03T21:02:03.515 INFO:teuthology.orchestra.run.smithi032.stderr:gzip/var/log/ceph/ceph-client.0.56805.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.52808.log 0.0% -- replaced with /var/log/ceph/ceph-client.0.56805.log.gz 2024-08-03T21:02:03.516 INFO:teuthology.orchestra.run.smithi032.stderr: 2024-08-03T21:02:03.516 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.52969.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.52969.log.gz 2024-08-03T21:02:03.517 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59334.log 2024-08-03T21:02:03.518 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65211.log 2024-08-03T21:02:03.518 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.52808.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.52808.log.gz 2024-08-03T21:02:03.519 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.59334.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.59334.log.gz 2024-08-03T21:02:03.520 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52430.log 2024-08-03T21:02:03.520 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.65211.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.65211.log.gz 2024-08-03T21:02:03.521 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75452.log 2024-08-03T21:02:03.522 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48562.log 2024-08-03T21:02:03.522 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.52430.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.52430.log.gz 2024-08-03T21:02:03.524 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.75452.log: /var/log/ceph/ceph-client.admin.48562.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.48562.log.gz 2024-08-03T21:02:03.524 INFO:teuthology.orchestra.run.smithi032.stderr: 92.5% -- replaced with /var/log/ceph/ceph-client.admin.75452.log.gz 2024-08-03T21:02:03.525 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84264.log 2024-08-03T21:02:03.525 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71636.log 2024-08-03T21:02:03.526 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.84264.log: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.84264.log.gz 2024-08-03T21:02:03.527 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54900.log 2024-08-03T21:02:03.529 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.71636.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57577.log 2024-08-03T21:02:03.529 INFO:teuthology.orchestra.run.smithi032.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.71636.log.gz 2024-08-03T21:02:03.529 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.54900.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.54900.log.gz 2024-08-03T21:02:03.530 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66146.log 2024-08-03T21:02:03.530 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.57577.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.57577.log.gz 2024-08-03T21:02:03.531 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38697.log 2024-08-03T21:02:03.532 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87649.log 2024-08-03T21:02:03.532 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.66146.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.66146.log.gz 2024-08-03T21:02:03.533 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.38697.log: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.38697.log.gz 2024-08-03T21:02:03.533 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59436.log 2024-08-03T21:02:03.534 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.87649.log: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.87649.log.gz 2024-08-03T21:02:03.535 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.70278.log 2024-08-03T21:02:03.536 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.59436.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.59436.log.gz 2024-08-03T21:02:03.536 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86926.log 2024-08-03T21:02:03.537 INFO:teuthology.orchestra.run.smithi032.stderr:gzip/var/log/ceph/ceph-client.0.70278.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.78580.log 0.0% -- replaced with /var/log/ceph/ceph-client.0.70278.log.gz 2024-08-03T21:02:03.537 INFO:teuthology.orchestra.run.smithi032.stderr: 2024-08-03T21:02:03.538 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.86926.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.86926.log.gz 2024-08-03T21:02:03.538 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.81358.log 2024-08-03T21:02:03.539 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.78580.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65874.log 2024-08-03T21:02:03.541 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.0.81358.log: 0.0% -- replaced with /var/log/ceph/ceph-client.0.81358.log.gz 2024-08-03T21:02:03.541 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76005.log 2024-08-03T21:02:03.542 INFO:teuthology.orchestra.run.smithi032.stderr: 92.4% -- replaced with /var/log/ceph/ceph-client.admin.78580.log.gz 2024-08-03T21:02:03.542 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.65874.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.65874.log.gz 2024-08-03T21:02:03.543 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55910.log 2024-08-03T21:02:03.544 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.76005.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.76005.log.gz 2024-08-03T21:02:03.544 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65599.log 2024-08-03T21:02:03.545 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.55910.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62506.log 2024-08-03T21:02:03.546 INFO:teuthology.orchestra.run.smithi032.stderr: 85.7% -- replaced with /var/log/ceph/ceph-client.admin.55910.log.gz 2024-08-03T21:02:03.546 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.65599.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.65599.log.gz 2024-08-03T21:02:03.547 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60768.log 2024-08-03T21:02:03.547 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.62506.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.62506.log.gz 2024-08-03T21:02:03.548 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50179.log 2024-08-03T21:02:03.549 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.82572.log 2024-08-03T21:02:03.550 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.60768.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.60768.log.gz 2024-08-03T21:02:03.550 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.50179.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.50179.log.gz 2024-08-03T21:02:03.551 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61757.log 2024-08-03T21:02:03.551 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.0.82572.log: 0.0% -- replaced with /var/log/ceph/ceph-client.0.82572.log.gz 2024-08-03T21:02:03.552 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79918.log 2024-08-03T21:02:03.553 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60479.log 2024-08-03T21:02:03.554 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.61757.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.61757.log.gz 2024-08-03T21:02:03.554 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.79918.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.79918.log.gz 2024-08-03T21:02:03.555 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54186.log 2024-08-03T21:02:03.555 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.60479.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.60479.log.gz 2024-08-03T21:02:03.556 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58775.log 2024-08-03T21:02:03.557 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.54186.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.54186.log.gz 2024-08-03T21:02:03.557 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72164.log 2024-08-03T21:02:03.559 INFO:teuthology.orchestra.run.smithi032.stderr:gzip/var/log/ceph/ceph-client.admin.58775.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.62414.log 2024-08-03T21:02:03.559 INFO:teuthology.orchestra.run.smithi032.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.58775.log.gz 2024-08-03T21:02:03.561 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.72164.log: /var/log/ceph/ceph-client.admin.62414.log: 92.5% -- replaced with /var/log/ceph/ceph-client.admin.72164.log.gz 2024-08-03T21:02:03.562 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77084.log 2024-08-03T21:02:03.562 INFO:teuthology.orchestra.run.smithi032.stderr: 92.7% -- replaced with /var/log/ceph/ceph-client.admin.62414.log.gz 2024-08-03T21:02:03.563 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85171.log 2024-08-03T21:02:03.564 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71384.log 2024-08-03T21:02:03.564 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.77084.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.77084.log.gz 2024-08-03T21:02:03.565 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.85171.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.85171.log.gz 2024-08-03T21:02:03.566 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58541.log 2024-08-03T21:02:03.566 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.71384.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.71384.log.gz 2024-08-03T21:02:03.567 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83780.log 2024-08-03T21:02:03.568 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62655.log 2024-08-03T21:02:03.568 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.58541.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.58541.log.gz 2024-08-03T21:02:03.569 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.83780.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.83780.log.gz 2024-08-03T21:02:03.569 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42972.log 2024-08-03T21:02:03.570 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.62655.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.62655.log.gz 2024-08-03T21:02:03.571 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50576.log 2024-08-03T21:02:03.572 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57900.log 2024-08-03T21:02:03.572 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.42972.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.42972.log.gz 2024-08-03T21:02:03.573 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.50576.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.50576.log.gz 2024-08-03T21:02:03.573 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62877.log 2024-08-03T21:02:03.574 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.57900.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.57900.log.gz 2024-08-03T21:02:03.574 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77351.log 2024-08-03T21:02:03.576 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75545.log 2024-08-03T21:02:03.576 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.62877.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.62877.log.gz 2024-08-03T21:02:03.577 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.77351.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.77351.log.gz 2024-08-03T21:02:03.578 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.64667.log 2024-08-03T21:02:03.578 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.75545.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.75545.log.gz 2024-08-03T21:02:03.579 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42602.log 2024-08-03T21:02:03.580 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75064.log 2024-08-03T21:02:03.580 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.0.64667.log: 0.0% -- replaced with /var/log/ceph/ceph-client.0.64667.log.gz 2024-08-03T21:02:03.581 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.42602.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.42602.log.gz 2024-08-03T21:02:03.582 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5/var/log/ceph/ceph-client.admin.75064.log: --verbose -- /var/log/ceph/ceph-client.admin.59045.log 2024-08-03T21:02:03.584 INFO:teuthology.orchestra.run.smithi032.stderr:gzip 92.5% -- replaced with /var/log/ceph/ceph-client.admin.75064.log.gz 2024-08-03T21:02:03.584 INFO:teuthology.orchestra.run.smithi032.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.70677.log 2024-08-03T21:02:03.585 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.59045.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87607.log 2024-08-03T21:02:03.585 INFO:teuthology.orchestra.run.smithi032.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.59045.log.gz 2024-08-03T21:02:03.586 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.70677.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.70677.log.gz 2024-08-03T21:02:03.586 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64917.log 2024-08-03T21:02:03.587 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.87607.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.87607.log.gz 2024-08-03T21:02:03.588 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87878.log 2024-08-03T21:02:03.589 INFO:teuthology.orchestra.run.smithi032.stderr:gzip/var/log/ceph/ceph-client.admin.64917.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.75228.log 2024-08-03T21:02:03.589 INFO:teuthology.orchestra.run.smithi032.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.64917.log.gz 2024-08-03T21:02:03.590 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.87878.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.87878.log.gz 2024-08-03T21:02:03.591 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57493.log 2024-08-03T21:02:03.591 INFO:teuthology.orchestra.run.smithi032.stderr: 93.6% -- replaced with /var/log/ceph/ceph-osd.3.log.gz 2024-08-03T21:02:03.592 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.75228.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71444.log 2024-08-03T21:02:03.592 INFO:teuthology.orchestra.run.smithi032.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.75228.log.gz 2024-08-03T21:02:03.593 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.57493.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.57493.log.gzgzip 2024-08-03T21:02:03.593 INFO:teuthology.orchestra.run.smithi032.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.77804.log 2024-08-03T21:02:03.593 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.71444.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61547.log 2024-08-03T21:02:03.594 INFO:teuthology.orchestra.run.smithi032.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.71444.log.gz 2024-08-03T21:02:03.595 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.77804.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69702.log 2024-08-03T21:02:03.596 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.61547.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.61547.log.gz 2024-08-03T21:02:03.596 INFO:teuthology.orchestra.run.smithi032.stderr: 92.5% -- replaced with /var/log/ceph/ceph-client.admin.77804.log.gz 2024-08-03T21:02:03.597 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53011.log 2024-08-03T21:02:03.597 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.69702.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58190.log 2024-08-03T21:02:03.598 INFO:teuthology.orchestra.run.smithi032.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.69702.log.gz 2024-08-03T21:02:03.598 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.53011.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86573.log 2024-08-03T21:02:03.598 INFO:teuthology.orchestra.run.smithi032.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.53011.log.gz 2024-08-03T21:02:03.599 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.58190.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85939.log 2024-08-03T21:02:03.599 INFO:teuthology.orchestra.run.smithi032.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.58190.log.gz 2024-08-03T21:02:03.600 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.86573.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48604.log 2024-08-03T21:02:03.601 INFO:teuthology.orchestra.run.smithi032.stderr: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.86573.log.gz 2024-08-03T21:02:03.601 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.85939.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.85939.log.gz 2024-08-03T21:02:03.603 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.48604.log: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.48604.log.gz 2024-08-03T21:02:03.821 INFO:teuthology.orchestra.run.smithi032.stderr: 91.4% -- replaced with /var/log/ceph/ceph-mds.a.log.gz 2024-08-03T21:02:03.887 INFO:teuthology.orchestra.run.smithi032.stderr: 93.8% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2024-08-03T21:02:04.089 INFO:teuthology.orchestra.run.smithi032.stderr: 90.2% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2024-08-03T21:02:04.113 INFO:teuthology.orchestra.run.smithi032.stderr: 94.3% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2024-08-03T21:02:04.116 INFO:teuthology.orchestra.run.smithi032.stderr: 2024-08-03T21:02:04.116 INFO:teuthology.orchestra.run.smithi032.stderr:real 0m1.592s 2024-08-03T21:02:04.116 INFO:teuthology.orchestra.run.smithi032.stderr:user 0m5.404s 2024-08-03T21:02:04.116 INFO:teuthology.orchestra.run.smithi032.stderr:sys 0m2.585s 2024-08-03T21:02:04.117 INFO:tasks.ceph:Archiving logs... 2024-08-03T21:02:04.118 DEBUG:teuthology.misc:Transferring archived files from smithi032:/var/log/ceph to /home/teuthworker/archive/vshankar-2024-08-02_18:35:56-fs-wip-vshankar-testing-20240802.091913-debug-testing-default-smithi/7833836/remote/smithi032/log 2024-08-03T21:02:04.119 DEBUG:teuthology.orchestra.run.smithi032:> sudo tar c -f - -C /var/log/ceph -- . 2024-08-03T21:02:05.496 DEBUG:teuthology.misc:Transferring archived files from smithi118:/var/log/ceph to /home/teuthworker/archive/vshankar-2024-08-02_18:35:56-fs-wip-vshankar-testing-20240802.091913-debug-testing-default-smithi/7833836/remote/smithi118/log 2024-08-03T21:02:05.497 DEBUG:teuthology.orchestra.run.smithi118:> sudo tar c -f - -C /var/log/ceph -- . 2024-08-03T21:02:06.080 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-08-03T21:02:06.092 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-08-03T21:02:06.092 DEBUG:teuthology.orchestra.run.smithi032:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-08-03T21:02:06.097 DEBUG:teuthology.orchestra.run.smithi118:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-08-03T21:02:06.203 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-08-03T21:02:06.213 INFO:teuthology.task.clock:Checking final clock skew... 2024-08-03T21:02:06.213 DEBUG:teuthology.orchestra.run.smithi032:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-08-03T21:02:06.217 DEBUG:teuthology.orchestra.run.smithi118:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-08-03T21:02:06.249 INFO:teuthology.orchestra.run.smithi032.stderr:bash: line 1: ntpq: command not found 2024-08-03T21:02:06.251 INFO:teuthology.orchestra.run.smithi118.stderr:bash: line 1: ntpq: command not found 2024-08-03T21:02:06.256 INFO:teuthology.orchestra.run.smithi032.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-08-03T21:02:06.256 INFO:teuthology.orchestra.run.smithi032.stdout:=============================================================================== 2024-08-03T21:02:06.256 INFO:teuthology.orchestra.run.smithi032.stdout:^+ hv01.front.sepia.ceph.com 3 6 377 1 +666us[ +666us] +/- 38ms 2024-08-03T21:02:06.257 INFO:teuthology.orchestra.run.smithi032.stdout:^* hv02.front.sepia.ceph.com 3 6 377 2 -1232us[-1231us] +/- 40ms 2024-08-03T21:02:06.257 INFO:teuthology.orchestra.run.smithi032.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 4 +838us[ +838us] +/- 72ms 2024-08-03T21:02:06.257 INFO:teuthology.orchestra.run.smithi032.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-08-03T21:02:06.258 INFO:teuthology.orchestra.run.smithi118.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-08-03T21:02:06.258 INFO:teuthology.orchestra.run.smithi118.stdout:=============================================================================== 2024-08-03T21:02:06.258 INFO:teuthology.orchestra.run.smithi118.stdout:^+ hv01.front.sepia.ceph.com 3 6 377 6 +665us[ +674us] +/- 38ms 2024-08-03T21:02:06.258 INFO:teuthology.orchestra.run.smithi118.stdout:^* hv02.front.sepia.ceph.com 3 6 377 4 -1216us[-1207us] +/- 40ms 2024-08-03T21:02:06.258 INFO:teuthology.orchestra.run.smithi118.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 3 +843us[ +843us] +/- 72ms 2024-08-03T21:02:06.259 INFO:teuthology.orchestra.run.smithi118.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-08-03T21:02:06.259 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-08-03T21:02:06.268 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-08-03T21:02:06.269 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-08-03T21:02:06.289 DEBUG:teuthology.orchestra.run.smithi032:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-08-03T21:02:06.445 DEBUG:teuthology.orchestra.run.smithi118:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-08-03T21:02:06.559 DEBUG:teuthology.orchestra.run.smithi032:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-08-03T21:02:06.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-03T21:02:06.622 DEBUG:teuthology.orchestra.run.smithi118:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-08-03T21:02:06.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-03T21:02:06.681 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-08-03T21:02:06.690 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-08-03T21:02:06.704 INFO:teuthology.task.internal:Duration was 1380.249141 seconds 2024-08-03T21:02:06.704 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-08-03T21:02:06.718 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-08-03T21:02:06.719 DEBUG:teuthology.orchestra.run.smithi032:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-08-03T21:02:06.723 DEBUG:teuthology.orchestra.run.smithi118:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-08-03T21:02:06.807 INFO:teuthology.orchestra.run.smithi032.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-08-03T21:02:06.810 INFO:teuthology.orchestra.run.smithi118.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-08-03T21:02:07.176 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-08-03T21:02:07.176 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi032.front.sepia.ceph.com 2024-08-03T21:02:07.177 DEBUG:teuthology.orchestra.run.smithi032:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-08-03T21:02:07.227 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi118.front.sepia.ceph.com 2024-08-03T21:02:07.228 DEBUG:teuthology.orchestra.run.smithi118:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-08-03T21:02:07.279 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-08-03T21:02:07.280 DEBUG:teuthology.orchestra.run.smithi032:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-08-03T21:02:07.285 DEBUG:teuthology.orchestra.run.smithi118:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-08-03T21:02:07.346 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-08-03T21:02:07.346 DEBUG:teuthology.orchestra.run.smithi032:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-08-03T21:02:07.351 DEBUG:teuthology.orchestra.run.smithi118:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-08-03T21:02:07.526 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-08-03T21:02:07.536 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-08-03T21:02:07.537 DEBUG:teuthology.orchestra.run.smithi032:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-08-03T21:02:07.592 DEBUG:teuthology.orchestra.run.smithi118:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-08-03T21:02:07.649 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-08-03T21:02:07.659 DEBUG:teuthology.orchestra.run.smithi032:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-08-03T21:02:07.663 DEBUG:teuthology.orchestra.run.smithi118:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-08-03T21:02:07.731 INFO:teuthology.orchestra.run.smithi032.stdout:kernel.core_pattern = core 2024-08-03T21:02:07.734 INFO:teuthology.orchestra.run.smithi118.stdout:kernel.core_pattern = core 2024-08-03T21:02:07.764 DEBUG:teuthology.orchestra.run.smithi032:> test -e /home/ubuntu/cephtest/archive/coredump 2024-08-03T21:02:07.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-03T21:02:07.797 DEBUG:teuthology.orchestra.run.smithi118:> test -e /home/ubuntu/cephtest/archive/coredump 2024-08-03T21:02:07.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-03T21:02:07.831 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-08-03T21:02:07.841 INFO:teuthology.task.internal:Transferring archived files... 2024-08-03T21:02:07.842 DEBUG:teuthology.misc:Transferring archived files from smithi032:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/vshankar-2024-08-02_18:35:56-fs-wip-vshankar-testing-20240802.091913-debug-testing-default-smithi/7833836/remote/smithi032 2024-08-03T21:02:07.842 DEBUG:teuthology.orchestra.run.smithi032:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-08-03T21:02:07.908 DEBUG:teuthology.misc:Transferring archived files from smithi118:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/vshankar-2024-08-02_18:35:56-fs-wip-vshankar-testing-20240802.091913-debug-testing-default-smithi/7833836/remote/smithi118 2024-08-03T21:02:07.908 DEBUG:teuthology.orchestra.run.smithi118:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-08-03T21:02:07.970 INFO:teuthology.task.internal:Removing archive directory... 2024-08-03T21:02:07.971 DEBUG:teuthology.orchestra.run.smithi032:> rm -rf -- /home/ubuntu/cephtest/archive 2024-08-03T21:02:07.975 DEBUG:teuthology.orchestra.run.smithi118:> rm -rf -- /home/ubuntu/cephtest/archive 2024-08-03T21:02:08.010 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-08-03T21:02:08.023 INFO:teuthology.task.internal:Not uploading archives. 2024-08-03T21:02:08.023 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-08-03T21:02:08.034 INFO:teuthology.task.internal:Tidying up after the test... 2024-08-03T21:02:08.034 DEBUG:teuthology.orchestra.run.smithi032:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-08-03T21:02:08.050 DEBUG:teuthology.orchestra.run.smithi118:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-08-03T21:02:08.081 INFO:teuthology.orchestra.run.smithi032.stdout: 393229 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Aug 3 21:02 /home/ubuntu/cephtest 2024-08-03T21:02:08.085 INFO:teuthology.orchestra.run.smithi118.stdout: 393730 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Aug 3 21:02 /home/ubuntu/cephtest 2024-08-03T21:02:08.087 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-08-03T21:02:08.097 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-08-03T21:02:08.120 INFO:teuthology.run:Summary data: client.0-kernel-sha1: ef231aaae597a12bd3e2479b633f31ed1fcc23a8 client.1-kernel-sha1: ef231aaae597a12bd3e2479b633f31ed1fcc23a8 description: fs/functional/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-4c-client conf/{client mds mgr mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} subvol_versions/create_subvol_version_v1 tasks/damage} duration: 1380.2491405010223 failure_reason: 'Test failure: test_open_ino_errors (tasks.cephfs.test_damage.TestDamage)' flavor: default owner: scheduled_vshankar@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=bf524262d9714b4388c6610345d2bc69 status: fail success: false 2024-08-03T21:02:08.121 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-08-03T21:02:08.208 INFO:teuthology.run:FAIL